2020-02-05T16:28:09.284 INFO:root:teuthology version: 1.0.0-95fed4d 2020-02-05T16:28:09.284 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.s0TU5s.tmp -- --verbose --lock --description fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_8.yaml} tasks/cfuse_workunit_kernel_untar_build.yaml} --name zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi --block --owner scheduled_zyan@teuthology --archive /home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997 2020-02-05T16:28:09.326 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-05T16:28:09.391 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997 branch: wip-async-dirops description: fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_8.yaml} tasks/cfuse_workunit_kernel_untar_build.yaml} email: null first_in_suite: false job_id: '4734997' kernel: flavor: basic kdb: true sha1: a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: rhel os_version: '8.0' overrides: admin_socket: branch: wip-async-dirops ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 fuse_default_permissions: 0 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd_max_omap_entries_per_request: 10000 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: ecafd7283ef390be73a605986afe2a0f325a3255 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: sha1: ecafd7283ef390be73a605986afe2a0f325a3255 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-async-dirops sha1: ecafd7283ef390be73a605986afe2a0f325a3255 owner: scheduled_zyan@teuthology priority: 50 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: ecafd7283ef390be73a605986afe2a0f325a3255 suite: fs suite_branch: wip-async-dirops suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: ecafd7283ef390be73a605986afe2a0f325a3255 tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - ceph: null - ceph-fuse: null - check-counter: counters: mds: - mds.dir_split - workunit: clients: all: - kernel_untar_build.sh teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.2502 2020-02-05T16:28:09.392 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa; will attempt to use it 2020-02-05T16:28:09.392 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa/tasks 2020-02-05T16:28:09.392 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-02-05T16:28:09.394 INFO:teuthology.task.internal:Checking packages... 2020-02-05T16:28:09.407 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'basic' and ceph hash 'ecafd7283ef390be73a605986afe2a0f325a3255' 2020-02-05T16:28:09.408 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-02-05T16:28:09.408 INFO:teuthology.packaging:ref: None 2020-02-05T16:28:09.408 INFO:teuthology.packaging:tag: None 2020-02-05T16:28:09.408 INFO:teuthology.packaging:branch: wip-async-dirops 2020-02-05T16:28:09.408 INFO:teuthology.packaging:sha1: ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:28:09.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-async-dirops 2020-02-05T16:28:09.682 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-9851.gecafd72 2020-02-05T16:28:09.682 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-02-05T16:28:09.693 INFO:teuthology.task.internal:no buildpackages task found 2020-02-05T16:28:09.693 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-02-05T16:28:09.739 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-02-05T16:28:09.739 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-05T16:28:09.799 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:28:19.818 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:28:29.834 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:28:39.850 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:28:49.869 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:28:59.889 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:29:09.906 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:29:19.924 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:29:29.942 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:29:39.962 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:29:50.068 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:00.086 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:10.102 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:20.112 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:30.131 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:40.149 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:30:50.162 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:31:00.176 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:31:10.193 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 1)... 2020-02-05T16:31:20.205 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:31:30.218 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:31:40.236 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:31:50.252 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:00.271 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:10.290 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:20.307 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:30.337 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:40.355 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:32:50.377 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:00.388 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:10.407 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:20.426 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:30.444 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:40.462 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:33:50.472 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:34:00.524 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 3)... 2020-02-05T16:34:10.542 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_zyan@teuthology', 'description': '/home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997', 'machine_type': 'smithi'} 2020-02-05T16:34:10.578 DEBUG:teuthology.lock.ops:locked ubuntu@smithi008.front.sepia.ceph.com, ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:34:10.632 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-05T16:34:10.871 INFO:teuthology.provision.fog.smithi008:Scheduling deploy of rhel 8.0 2020-02-05T16:34:10.874 INFO:teuthology.provision.fog.smithi164:Scheduling deploy of rhel 8.0 2020-02-05T16:34:11.170 INFO:teuthology.orchestra.console:Power off smithi164 2020-02-05T16:34:11.171 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-05T16:34:11.199 INFO:teuthology.orchestra.console:Power off smithi008 2020-02-05T16:34:11.200 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-02-05T16:34:11.207 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:11.219 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:15.230 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:15.246 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:19.343 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:19.364 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:19.555 INFO:teuthology.orchestra.console:Power off for smithi164 completed 2020-02-05T16:34:19.575 INFO:teuthology.orchestra.console:Power off for smithi008 completed 2020-02-05T16:34:19.656 INFO:teuthology.orchestra.console:Power on smithi164 2020-02-05T16:34:19.656 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-05T16:34:19.677 INFO:teuthology.orchestra.console:Power on smithi008 2020-02-05T16:34:19.678 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-02-05T16:34:19.688 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:19.741 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:23.714 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:23.763 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:27.827 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:27.873 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi008.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-02-05T16:34:28.084 INFO:teuthology.orchestra.console:Power on for smithi008 completed 2020-02-05T16:34:28.107 INFO:teuthology.orchestra.console:Power on for smithi164 completed 2020-02-05T16:34:28.184 INFO:teuthology.provision.fog.smithi008:Waiting for deploy to finish 2020-02-05T16:34:28.207 INFO:teuthology.provision.fog.smithi164:Waiting for deploy to finish 2020-02-05T16:36:44.114 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:08.567 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:14.301 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:17.623 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:26.681 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:35.739 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:38.775 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:47.839 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:37:56.925 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:06.014 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:07.558 INFO:teuthology.orchestra.run.smithi008:> hostname 2020-02-05T16:38:07.918 INFO:teuthology.orchestra.run.smithi008.stdout:smithi143 2020-02-05T16:38:07.919 INFO:teuthology.orchestra.run.smithi008:> grep smithi143 /etc/hosts 2020-02-05T16:38:07.973 INFO:teuthology.orchestra.run.smithi008.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143 2020-02-05T16:38:07.973 INFO:teuthology.orchestra.run.smithi008:> sudo hostname smithi008 2020-02-05T16:38:08.102 INFO:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/smithi143/smithi008/g' /etc/hosts 2020-02-05T16:38:08.134 INFO:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/172.21.15.143/172.21.15.8/g' /etc/hosts 2020-02-05T16:38:08.205 INFO:teuthology.orchestra.run.smithi008:> sudo sed -i -e 's/smithi143/smithi008/g' /etc/hostname 2020-02-05T16:38:08.276 INFO:teuthology.orchestra.run.smithi008:> sudo hostname smithi008 2020-02-05T16:38:08.346 INFO:teuthology.provision.fog.smithi008:Deploy complete! 2020-02-05T16:38:08.348 DEBUG:teuthology.parallel:result is None 2020-02-05T16:38:15.098 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:46.719 INFO:teuthology.orchestra.run.smithi164:> hostname 2020-02-05T16:38:47.062 INFO:teuthology.orchestra.run.smithi164.stdout:smithi143 2020-02-05T16:38:47.063 INFO:teuthology.orchestra.run.smithi164:> grep smithi143 /etc/hosts 2020-02-05T16:38:47.117 INFO:teuthology.orchestra.run.smithi164.stdout:172.21.15.143 smithi143.front.sepia.ceph.com smithi143 2020-02-05T16:38:47.117 INFO:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2020-02-05T16:38:47.235 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi143/smithi164/g' /etc/hosts 2020-02-05T16:38:47.268 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/172.21.15.143/172.21.15.164/g' /etc/hosts 2020-02-05T16:38:47.336 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i -e 's/smithi143/smithi164/g' /etc/hostname 2020-02-05T16:38:47.404 INFO:teuthology.orchestra.run.smithi164:> sudo hostname smithi164 2020-02-05T16:38:47.492 INFO:teuthology.provision.fog.smithi164:Deploy complete! 2020-02-05T16:38:47.494 DEBUG:teuthology.parallel:result is None 2020-02-05T16:38:47.613 INFO:teuthology.lock.ops:Checking smithi008.front.sepia.ceph.com 2020-02-05T16:38:47.614 INFO:teuthology.lock.ops:New key found. Updating... 2020-02-05T16:38:47.627 INFO:teuthology.lock.ops:Checking smithi164.front.sepia.ceph.com 2020-02-05T16:38:47.627 INFO:teuthology.lock.ops:New key found. Updating... 2020-02-05T16:38:47.642 INFO:teuthology.lock.ops:Updating [smithi008.front.sepia.ceph.com]: set os type and version on server 2020-02-05T16:38:47.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:38:47.643 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:47.714 INFO:teuthology.orchestra.run.smithi008:> true 2020-02-05T16:38:47.901 INFO:teuthology.orchestra.run.smithi008:> uname -m 2020-02-05T16:38:47.955 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2020-02-05T16:38:47.955 INFO:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2020-02-05T16:38:48.010 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Red Hat Enterprise Linux" 2020-02-05T16:38:48.010 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8.0 (Ootpa)" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:ID="rhel" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="fedora" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8.0" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2020-02-05T16:38:48.011 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2020-02-05T16:38:48.012 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2020-02-05T16:38:48.012 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.redhat.com/" 2020-02-05T16:38:48.012 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-02-05T16:38:48.012 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:38:48.012 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-02-05T16:38:48.013 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2020-02-05T16:38:48.013 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-02-05T16:38:48.013 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2020-02-05T16:38:48.014 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2020-02-05T16:38:48.036 INFO:teuthology.lock.ops:Updating [smithi164.front.sepia.ceph.com]: set os type and version on server 2020-02-05T16:38:48.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:38:48.036 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:48.104 INFO:teuthology.orchestra.run.smithi164:> true 2020-02-05T16:38:48.281 INFO:teuthology.orchestra.run.smithi164:> uname -m 2020-02-05T16:38:48.334 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2020-02-05T16:38:48.334 INFO:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2020-02-05T16:38:48.387 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Red Hat Enterprise Linux" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8.0 (Ootpa)" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:ID="rhel" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="fedora" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8.0" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2020-02-05T16:38:48.388 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.redhat.com/" 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2020-02-05T16:38:48.389 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-02-05T16:38:48.390 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2020-02-05T16:38:48.390 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2020-02-05T16:38:48.406 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2020-02-05T16:38:48.438 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi008.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn2A/ANXtv9dsa38HZFt8mns4dMjQq8EtRuB2FzX1Wo36HuBKjOR7Sxqo/o55ISzdqE9HkP0Tvsly6KvUHWgmHq6ryjp2FPcqt4FJ2CAgh3iG09PGnUkdyaTm2Xmig/i75PxOVFxu2hmUCf0NAcniGAUMQLpyIRK5qlWviz6h4GQGQD79ryFuGunpgesa70FZBor3VsINim1zLatIbb8SMj83S9K2KHb2EGKcSIcpKJoEdYPPL5JeT2YRcTV8yRI3yuHH8ThNzEw4OSvE2cC/e7vIYXoS3J18BP4BnjfUKicads3CwMarGdSWsCZulQq6vtrhx/FKXB1Xh8WVju3GF smithi164.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5XcYOcYXs6V5ZQMjnk27EqGyJkYMzBTSsMWrSQJA1Y+6RrGKAN6wCwrP4sL+6i3Q8ZUEDiYPNlJKteiqaljO0LFqqWXLFDAvbdYrmr1tGkXlgYuFPhec8u38De/5jF9aRHwce0vXJjz6NZy89XbqOsbpuEW02vWf0p/PmJtDK7Jqk4d9iI2Juseb+vNQTI+b2sPQdJlKkeZnA11UVRHW/emT35GTMvXYhDD3iIeIQF5EAFZs36wsDUmIxPnIsngImMAl7VSmJzfLWD2IW0dkLBZTWmL3za7otn8FykLxUIbMGsHDLw89MlZ2OcJCaq3q2VUPbfRC+/hh43dAcne8P 2020-02-05T16:38:48.438 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-02-05T16:38:48.490 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-02-05T16:38:48.498 INFO:teuthology.task.internal:Saving configuration 2020-02-05T16:38:48.536 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-02-05T16:38:48.544 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-02-05T16:38:48.564 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-02-05 16:34:10.549673', u'locked_by': u'scheduled_zyan@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi008.front.sepia.ceph.com', u'os_version': u'8.0', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'rhel', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOv5Uk3RhQFuprdk3Spuk1uBw37g3eqxlxX5wFqSklYXPWhf1aKaXbQnnpB639ZMOPn75pQGRBBPbQtYCSSTquE=', u'description': u'/home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997'} 2020-02-05T16:38:48.582 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-02-05 16:34:10.549706', u'locked_by': u'scheduled_zyan@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi164.front.sepia.ceph.com', u'os_version': u'8.0', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'rhel', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPkRQ4RSEzGwesIdpIPmiVYnD5j5OmbspelO419Jk7baqY0VZsZLisXS1O4iCiWYmf6jwDmE3+YgtAXLdGNyrw=', u'description': u'/home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997'} 2020-02-05T16:38:48.582 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-02-05T16:38:48.634 INFO:teuthology.task.internal:roles: ubuntu@smithi008.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2020-02-05T16:38:48.635 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2020-02-05T16:38:48.635 INFO:teuthology.run_tasks:Running task console_log... 2020-02-05T16:38:48.718 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-02-05T16:38:48.718 INFO:teuthology.run_tasks:Running task internal.connect... 2020-02-05T16:38:48.728 INFO:teuthology.task.internal:Opening connections... 2020-02-05T16:38:48.728 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:38:48.730 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:48.806 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:38:48.807 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:38:48.876 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-02-05T16:38:48.891 INFO:teuthology.orchestra.run.smithi008:> uname -m 2020-02-05T16:38:48.949 INFO:teuthology.orchestra.run.smithi008.stdout:x86_64 2020-02-05T16:38:48.950 INFO:teuthology.orchestra.run.smithi008:> cat /etc/os-release 2020-02-05T16:38:49.004 INFO:teuthology.orchestra.run.smithi008.stdout:NAME="Red Hat Enterprise Linux" 2020-02-05T16:38:49.004 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION="8.0 (Ootpa)" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:ID="rhel" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:ID_LIKE="fedora" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:VERSION_ID="8.0" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:PLATFORM_ID="platform:el8" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2020-02-05T16:38:49.005 INFO:teuthology.orchestra.run.smithi008.stdout:ANSI_COLOR="0;31" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:HOME_URL="https://www.redhat.com/" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-02-05T16:38:49.006 INFO:teuthology.orchestra.run.smithi008.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2020-02-05T16:38:49.007 INFO:teuthology.lock.ops:Updating smithi008.front.sepia.ceph.com on lock server 2020-02-05T16:38:49.028 INFO:teuthology.orchestra.run.smithi164:> uname -m 2020-02-05T16:38:49.043 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2020-02-05T16:38:49.043 INFO:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2020-02-05T16:38:49.097 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="Red Hat Enterprise Linux" 2020-02-05T16:38:49.097 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="8.0 (Ootpa)" 2020-02-05T16:38:49.097 INFO:teuthology.orchestra.run.smithi164.stdout:ID="rhel" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="fedora" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="8.0" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el8" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.0 (Ootpa)" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.0:GA" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://www.redhat.com/" 2020-02-05T16:38:49.098 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2020-02-05T16:38:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:38:49.099 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2020-02-05T16:38:49.099 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.0 2020-02-05T16:38:49.099 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2020-02-05T16:38:49.099 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.0" 2020-02-05T16:38:49.099 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2020-02-05T16:38:49.114 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-02-05T16:38:49.134 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-02-05T16:38:49.145 INFO:teuthology.task.internal:Checking for old test directory... 2020-02-05T16:38:49.145 INFO:teuthology.orchestra.run.smithi008:> test '!' -e /home/ubuntu/cephtest 2020-02-05T16:38:49.147 INFO:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2020-02-05T16:38:49.161 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-02-05T16:38:49.176 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-02-05T16:38:49.177 INFO:teuthology.orchestra.run.smithi008:> test -z $(ls -A /var/lib/ceph) 2020-02-05T16:38:49.202 INFO:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2020-02-05T16:38:49.224 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-02-05T16:38:49.267 INFO:teuthology.run_tasks:Running task kernel... 2020-02-05T16:38:49.284 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mon.c': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'client.0': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mds.a': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mds.b': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.1': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.0': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.3': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.2': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.5': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.4': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.7': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'osd.6': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mon.b': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mgr.y': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mgr.x': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}} 2020-02-05T16:38:49.284 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'mon.b': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}}, timeout 300 2020-02-05T16:38:49.285 INFO:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2020-02-05T16:38:49.339 INFO:teuthology.orchestra.run.smithi008.stdout:RedHatEnterprise 2020-02-05T16:38:49.340 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:38:49.341 DEBUG:teuthology.task.kernel:sha1 for mon.a is a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.341 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f"... 2020-02-05T16:38:49.341 INFO:teuthology.orchestra.run.smithi008:> uname -r 2020-02-05T16:38:49.355 INFO:teuthology.orchestra.run.smithi008.stdout:4.18.0-80.11.2.el8_0.x86_64 2020-02-05T16:38:49.356 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-80.11.2.el8_0.x86_64 vs a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.356 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2020-02-05T16:38:49.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.532 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-02-05T16:38:49.602 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2020-02-05T16:38:49.603 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:38:49.603 DEBUG:teuthology.task.kernel:sha1 for mon.b is a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.604 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f"... 2020-02-05T16:38:49.604 INFO:teuthology.orchestra.run.smithi164:> uname -r 2020-02-05T16:38:49.618 INFO:teuthology.orchestra.run.smithi164.stdout:4.18.0-80.11.2.el8_0.x86_64 2020-02-05T16:38:49.618 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-80.11.2.el8_0.x86_64 vs a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.618 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2020-02-05T16:38:49.618 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:38:49.781 INFO:teuthology.orchestra.run.smithi008:> sudo yum upgrade -y linux-firmware 2020-02-05T16:38:51.511 INFO:teuthology.orchestra.run.smithi008.stdout:Updating Subscription Management repositories. 2020-02-05T16:38:51.512 INFO:teuthology.orchestra.run.smithi008.stdout:Unable to read consumer identity 2020-02-05T16:38:51.618 INFO:teuthology.orchestra.run.smithi008.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-02-05T16:38:53.632 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 8 - x86_64 16 kB/s | 19 kB 00:01 2020-02-05T16:38:56.468 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 8 - x86_64 2.0 MB/s | 5.5 MB 00:02 2020-02-05T16:38:57.672 INFO:teuthology.orchestra.run.smithi008.stdout:Last metadata expiration check: 0:00:02 ago on Wed 05 Feb 2020 04:38:55 PM UTC. 2020-02-05T16:38:58.086 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2020-02-05T16:38:58.087 INFO:teuthology.orchestra.run.smithi008.stdout:Nothing to do. 2020-02-05T16:38:58.087 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2020-02-05T16:38:58.206 INFO:teuthology.orchestra.run.smithi164:> sudo yum upgrade -y linux-firmware 2020-02-05T16:38:59.814 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2020-02-05T16:38:59.814 INFO:teuthology.orchestra.run.smithi164.stdout:Unable to read consumer identity 2020-02-05T16:38:59.928 INFO:teuthology.orchestra.run.smithi164.stdout:This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 2020-02-05T16:39:00.988 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 8 - x86_64 119 kB/s | 19 kB 00:00 2020-02-05T16:39:01.567 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 8 - x86_64 9.6 MB/s | 5.5 MB 00:00 2020-02-05T16:39:02.683 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:01 ago on Wed 05 Feb 2020 04:39:01 PM UTC. 2020-02-05T16:39:03.131 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2020-02-05T16:39:03.132 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2020-02-05T16:39:03.132 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2020-02-05T16:39:03.259 INFO:teuthology.task.kernel:Downloading kernel a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f on mon.a... 2020-02-05T16:39:03.259 INFO:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2020-02-05T16:39:03.306 INFO:teuthology.orchestra.run.smithi008.stdout:RedHatEnterprise 2020-02-05T16:39:03.307 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:39:03.307 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:39:03.457 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/ 2020-02-05T16:39:03.458 INFO:teuthology.orchestra.run.smithi008:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.5.0_ceph_ga11f6ef5e0ea-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/ --input-file=- 2020-02-05T16:39:03.460 INFO:teuthology.task.kernel:Downloading kernel a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f on mon.b... 2020-02-05T16:39:03.460 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-02-05T16:39:03.499 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2020-02-05T16:39:03.500 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:39:03.501 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F8%2Fx86_64&sha1=a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f 2020-02-05T16:39:03.660 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/ 2020-02-05T16:39:03.660 INFO:teuthology.orchestra.run.smithi164:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-5.5.0_ceph_ga11f6ef5e0ea-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/ --input-file=- 2020-02-05T16:39:03.663 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi008.front.sepia.ceph.com to complete... 2020-02-05T16:39:41.377 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05 16:39:41 URL:https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/kernel-5.5.0_ceph_ga11f6ef5e0ea-1.x86_64.rpm [446376808/446376808] -> "/tmp/kernel.x86_64.rpm" [1] 2020-02-05T16:39:41.377 INFO:teuthology.orchestra.run.smithi008.stderr:FINISHED --2020-02-05 16:39:41-- 2020-02-05T16:39:41.377 INFO:teuthology.orchestra.run.smithi008.stderr:Total wall clock time: 38s 2020-02-05T16:39:41.377 INFO:teuthology.orchestra.run.smithi008.stderr:Downloaded: 1 files, 426M in 38s (11.3 MB/s) 2020-02-05T16:39:41.378 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi164.front.sepia.ceph.com to complete... 2020-02-05T16:39:41.467 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05 16:39:41 URL:https://4.chacra.ceph.com/r/kernel/testing/a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f/centos/8/flavors/default/x86_64/kernel-5.5.0_ceph_ga11f6ef5e0ea-1.x86_64.rpm [446376808/446376808] -> "/tmp/kernel.x86_64.rpm" [1] 2020-02-05T16:39:41.467 INFO:teuthology.orchestra.run.smithi164.stderr:FINISHED --2020-02-05 16:39:41-- 2020-02-05T16:39:41.467 INFO:teuthology.orchestra.run.smithi164.stderr:Total wall clock time: 38s 2020-02-05T16:39:41.467 INFO:teuthology.orchestra.run.smithi164.stderr:Downloaded: 1 files, 426M in 38s (11.3 MB/s) 2020-02-05T16:39:41.474 INFO:teuthology.task.kernel:Installing kernel a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f on mon.a... 2020-02-05T16:39:41.474 INFO:teuthology.orchestra.run.smithi008:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2020-02-05T16:39:43.345 INFO:teuthology.orchestra.run.smithi008.stdout:Verifying... ######################################## 2020-02-05T16:39:43.633 INFO:teuthology.orchestra.run.smithi008.stdout:Preparing... ######################################## 2020-02-05T16:39:43.672 INFO:teuthology.orchestra.run.smithi008.stdout:Updating / installing... 2020-02-05T16:40:18.982 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-5.5.0_ceph_ga11f6ef5e0ea-1 ######################################## 2020-02-05T16:40:40.462 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi008.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2020-02-05T16:40:40.463 INFO:teuthology.orchestra.run.smithi008:> rpm -qlp /tmp/kernel.x86_64.rpm 2020-02-05T16:40:40.511 INFO:teuthology.orchestra.run.smithi008.stdout:/boot/System.map-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.511 INFO:teuthology.orchestra.run.smithi008.stdout:/boot/config-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.511 INFO:teuthology.orchestra.run.smithi008.stdout:/boot/vmlinux-5.5.0-ceph-ga11f6ef5e0ea.bz2 2020-02-05T16:40:40.511 INFO:teuthology.orchestra.run.smithi008.stdout:/boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.511 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/blowfish-x86_64.ko 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2020-02-05T16:40:40.512 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-x86_64.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crc32-pclmul.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crc32c-intel.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2020-02-05T16:40:40.513 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-avx2.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha1-ssse3.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha256-ssse3.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha512-ssse3.ko 2020-02-05T16:40:40.514 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-x86_64.ko 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu/mce 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2020-02-05T16:40:40.515 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm-amd.ko 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm-intel.ko 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm.ko 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/oprofile 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/oprofile/oprofile.ko 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/ansi_cprng.ko 2020-02-05T16:40:40.516 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/anubis.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/arc4.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_memcpy.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_pq.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_raid6_recov.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_tx.ko 2020-02-05T16:40:40.517 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_xor.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/raid6test.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/authenc.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/authencesn.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blake2b_generic.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blowfish_common.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blowfish_generic.ko 2020-02-05T16:40:40.518 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/camellia_generic.ko 2020-02-05T16:40:40.519 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast5_generic.ko 2020-02-05T16:40:40.519 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast6_generic.ko 2020-02-05T16:40:40.519 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast_common.ko 2020-02-05T16:40:40.519 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/ccm.ko 2020-02-05T16:40:40.519 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cmac.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crc32_generic.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crypto_engine.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crypto_user.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cts.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/des_generic.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/dh_generic.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/echainiv.ko 2020-02-05T16:40:40.520 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/essiv.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/fcrypt.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/khazad.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/lz4.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/lz4hc.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/md4.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/michael_mic.ko 2020-02-05T16:40:40.521 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/pcbc.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/pcrypt.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd128.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd160.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd256.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd320.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/salsa20_generic.ko 2020-02-05T16:40:40.522 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/seed.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/serpent_generic.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/sha512_generic.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tcrypt.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tea.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tgr192.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/twofish_common.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/twofish_generic.ko 2020-02-05T16:40:40.523 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/vmac.ko 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/wp512.ko 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xcbc.ko 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xor.ko 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xxhash_generic.ko 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi 2020-02-05T16:40:40.524 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/acpi_ipmi.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/acpi_pad.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/custom_method.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/ec_sys.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/sbs.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/sbshc.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/video.ko 2020-02-05T16:40:40.525 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/acard-ahci.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/ahci_platform.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/ata_generic.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/libahci_platform.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_acpi.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ali.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_amd.ko 2020-02-05T16:40:40.526 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_artop.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_atiixp.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_atp867x.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cmd640.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cmd64x.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cypress.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_efar.ko 2020-02-05T16:40:40.527 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt366.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt37x.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt3x2n.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt3x3.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_it8213.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_it821x.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_jmicron.ko 2020-02-05T16:40:40.528 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_marvell.ko 2020-02-05T16:40:40.529 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_mpiix.ko 2020-02-05T16:40:40.529 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_netcell.ko 2020-02-05T16:40:40.529 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ninja32.ko 2020-02-05T16:40:40.529 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ns87410.ko 2020-02-05T16:40:40.529 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ns87415.ko 2020-02-05T16:40:40.530 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_oldpiix.ko 2020-02-05T16:40:40.530 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_opti.ko 2020-02-05T16:40:40.530 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_optidma.ko 2020-02-05T16:40:40.530 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pcmcia.ko 2020-02-05T16:40:40.530 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pdc2027x.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pdc202xx_old.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_piccolo.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_rdc.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sch.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_serverworks.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sil680.ko 2020-02-05T16:40:40.531 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sis.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sl82c105.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_triflex.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_via.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pdc_adma.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_inic162x.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_mv.ko 2020-02-05T16:40:40.532 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_nv.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_promise.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_qstor.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sil.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sil24.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sis.ko 2020-02-05T16:40:40.533 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_svw.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sx4.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_uli.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_via.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_vsc.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/auxdisplay 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/auxdisplay/ks0108.ko 2020-02-05T16:40:40.534 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/bcma 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/bcma/bcma.ko 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/aoe 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/aoe/aoe.ko 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/brd.ko 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/drbd 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/drbd/drbd.ko 2020-02-05T16:40:40.535 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/floppy.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/loop.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/mtip32xx 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/mtip32xx/mtip32xx.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/nbd.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/null_blk.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/pktcdvd.ko 2020-02-05T16:40:40.536 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/rbd.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/skd.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/sx8.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/umem.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/virtio_blk.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkback 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkback/xen-blkback.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkfront.ko 2020-02-05T16:40:40.537 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/zram 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/zram/zram.ko 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hangcheck-timer.ko 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/amd-rng.ko 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/intel-rng.ko 2020-02-05T16:40:40.538 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/timeriomem-rng.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/via-rng.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/virtio-rng.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_devintf.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2020-02-05T16:40:40.539 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_si.ko 2020-02-05T16:40:40.540 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_ssif.ko 2020-02-05T16:40:40.540 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2020-02-05T16:40:40.540 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/lp.ko 2020-02-05T16:40:40.540 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/mwave 2020-02-05T16:40:40.540 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/mwave/mwave.ko 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia/cm4000_cs.ko 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia/cm4040_cs.ko 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ppdev.ko 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tlclk.ko 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm 2020-02-05T16:40:40.541 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_atmel.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_infineon.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_nsc.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_tis.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_tis_core.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/uv_mmtimer.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/virtio_console.ko 2020-02-05T16:40:40.542 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/acpi-cpufreq.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/p4-clockmod.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/pcc-cpufreq.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/powernow-k8.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/speedstep-lib.ko 2020-02-05T16:40:40.543 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp/ccp-crypto.ko 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp/ccp.ko 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/padlock-aes.ko 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/padlock-sha.ko 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat 2020-02-05T16:40:40.544 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_common 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_dh895xcc 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/virtio 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/virtio/virtio_crypto.ko 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dca 2020-02-05T16:40:40.545 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dca/dca.ko 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/devfreq 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/devfreq/governor_simpleondemand.ko 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/dw 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/dw/dw_dmac.ko 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/ioat 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/ioat/ioatdma.ko 2020-02-05T16:40:40.546 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/mic_x100_dma.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/amd64_edac_mod.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/e752x_edac.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/edac_mce_amd.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i3000_edac.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i3200_edac.ko 2020-02-05T16:40:40.547 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5000_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5100_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5400_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i7300_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i7core_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i82975x_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/ie31200_edac.ko 2020-02-05T16:40:40.548 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/sb_edac.ko 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/x38_edac.ko 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware/edd.ko 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware/iscsi_ibft.ko 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio 2020-02-05T16:40:40.549 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio/gpio-ich.ko 2020-02-05T16:40:40.550 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio/gpio-viperboard.ko 2020-02-05T16:40:40.550 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid 2020-02-05T16:40:40.550 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-appleir.ko 2020-02-05T16:40:40.550 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-aureal.ko 2020-02-05T16:40:40.550 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-axff.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-dr.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-elecom.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-elo.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-emsff.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gaff.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gt683r.ko 2020-02-05T16:40:40.551 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gyration.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtek-kbd.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtek-mouse.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtekff.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-hyperv.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-icade.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-keytouch.ko 2020-02-05T16:40:40.552 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-kye.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-lcpower.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-led.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-lenovo.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-logitech-dj.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-logitech-hidpp.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-multitouch.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-ortek.ko 2020-02-05T16:40:40.553 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-penmount.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-petalynx.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-picolcd.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-pl.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-plantronics.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-primax.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-rmi.ko 2020-02-05T16:40:40.554 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-arvo.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-common.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-isku.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-kone.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-koneplus.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-konepure.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-kovaplus.ko 2020-02-05T16:40:40.555 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-lua.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-pyra.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-ryos.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-savu.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-saitek.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-samsung.ko 2020-02-05T16:40:40.556 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sensor-hub.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sjoy.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sony.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-speedlink.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-steelseries.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sunplus.ko 2020-02-05T16:40:40.557 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-tivo.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-tmff.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-topseed.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-twinhan.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-uclogic.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-waltop.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-wiimote.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-xinmo.ko 2020-02-05T16:40:40.558 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-zpff.ko 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-zydacron.ko 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/i2c-hid 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/i2c-hid/i2c-hid.ko 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/uhid.ko 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/wacom.ko 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv 2020-02-05T16:40:40.559 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_balloon.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_utils.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_vmbus.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/abituguru.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/abituguru3.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/acpi_power_meter.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ad7414.ko 2020-02-05T16:40:40.560 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ad7418.ko 2020-02-05T16:40:40.561 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adc128d818.ko 2020-02-05T16:40:40.561 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1021.ko 2020-02-05T16:40:40.561 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1025.ko 2020-02-05T16:40:40.561 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1026.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1029.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1031.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm9240.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ads7828.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7410.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7411.ko 2020-02-05T16:40:40.562 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7462.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7470.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7475.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7x10.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/amc6821.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/applesmc.ko 2020-02-05T16:40:40.563 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asb100.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asc7621.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asus_atk0110.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/atxp1.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/coretemp.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/dell-smm-hwmon.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/dme1737.ko 2020-02-05T16:40:40.564 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ds1621.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ds620.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/emc1403.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/emc6w201.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f71805f.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f71882fg.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f75375s.ko 2020-02-05T16:40:40.565 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/fam15h_power.ko 2020-02-05T16:40:40.566 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/fschmd.ko 2020-02-05T16:40:40.566 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/g760a.ko 2020-02-05T16:40:40.566 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/g762.ko 2020-02-05T16:40:40.566 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/gl518sm.ko 2020-02-05T16:40:40.566 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/gl520sm.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/hwmon-vid.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/i5500_temp.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/i5k_amb.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ibmaem.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ibmpex.ko 2020-02-05T16:40:40.567 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ina209.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ina2xx.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/it87.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/k10temp.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/k8temp.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lineage-pem.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm63.ko 2020-02-05T16:40:40.568 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm73.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm75.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm77.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm78.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm80.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm83.ko 2020-02-05T16:40:40.569 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm85.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm87.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm90.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm92.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm93.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95234.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95241.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95245.ko 2020-02-05T16:40:40.570 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc2945.ko 2020-02-05T16:40:40.571 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4151.ko 2020-02-05T16:40:40.571 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4215.ko 2020-02-05T16:40:40.571 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4222.ko 2020-02-05T16:40:40.571 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4245.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4260.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4261.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max16065.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max1619.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max1668.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max197.ko 2020-02-05T16:40:40.572 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6639.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6642.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6650.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6697.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/mcp3021.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct6683.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct6775.ko 2020-02-05T16:40:40.573 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct7802.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ntc_thermistor.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pc87360.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pc87427.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pcf8591.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/adm1275.ko 2020-02-05T16:40:40.574 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/lm25066.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ltc2978.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max16064.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max34440.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max8688.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/pmbus.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/tps40422.ko 2020-02-05T16:40:40.575 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ucd9000.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ucd9200.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/zl6100.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/powr1220.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch5627.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch5636.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch56xx-common.ko 2020-02-05T16:40:40.576 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sht15.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sht21.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/shtc1.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sis5595.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47b397.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47m1.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47m192.ko 2020-02-05T16:40:40.577 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/thmc50.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp102.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp103.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp401.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp421.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/via-cputemp.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/via686a.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/vt1211.ko 2020-02-05T16:40:40.578 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/vt8231.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83627ehf.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83627hf.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83781d.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83791d.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83792d.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83793.ko 2020-02-05T16:40:40.579 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83795.ko 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83l785ts.ko 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83l786ng.ko 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2020-02-05T16:40:40.580 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd756.ko 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd8111.ko 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-core.ko 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2020-02-05T16:40:40.581 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-i801.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-isch.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-ismt.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2020-02-05T16:40:40.582 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-nforce2.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-parport-light.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-parport.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-piix4.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-scmi.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-simtec.ko 2020-02-05T16:40:40.583 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-sis96x.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-via.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-viapro.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-viperboard.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-dev.ko 2020-02-05T16:40:40.584 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-mux.ko 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-slave-eeprom.ko 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-smbus.ko 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-stub.ko 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core 2020-02-05T16:40:40.585 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_cm.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_core.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_umad.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_uverbs.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/iw_cm.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/rdma_cm.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/rdma_ucm.ko 2020-02-05T16:40:40.586 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/cxgb4 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx4 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx5 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2020-02-05T16:40:40.587 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mthca 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/ipoib 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/iser 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2020-02-05T16:40:40.588 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/isert 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srp 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srpt 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input 2020-02-05T16:40:40.589 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/emu10k1-gp.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/fm801-gp.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/gameport.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/lightning.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/ns558.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/input-polldev.ko 2020-02-05T16:40:40.590 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/joydev.ko 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/keyboard 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/keyboard/gpio_keys.ko 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/appletouch.ko 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/bcm5974.ko 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/cyapatp.ko 2020-02-05T16:40:40.591 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/elan_i2c.ko 2020-02-05T16:40:40.592 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/sermouse.ko 2020-02-05T16:40:40.592 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/synaptics_i2c.ko 2020-02-05T16:40:40.592 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/synaptics_usb.ko 2020-02-05T16:40:40.592 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/vsxxxaa.ko 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/rmi4 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/rmi4/rmi_core.ko 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/altera_ps2.ko 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/arc_ps2.ko 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/hyperv-keyboard.ko 2020-02-05T16:40:40.593 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/serio_raw.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/sparse-keymap.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/iommu 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/iommu/amd_iommu_v2.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-blinkm.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-clevo-mail.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lm3530.ko 2020-02-05T16:40:40.594 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp3944.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5521.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5523.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5562.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp55xx-common.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-ss4200.ko 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger 2020-02-05T16:40:40.595 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-audio.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-camera.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2020-02-05T16:40:40.596 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-timer.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-transient.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/bcache 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/bcache/bcache.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-bio-prison.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-cache-smq.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-cache.ko 2020-02-05T16:40:40.597 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-crypt.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-delay.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-flakey.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-log-userspace.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-multipath.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-queue-length.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-raid.ko 2020-02-05T16:40:40.598 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-round-robin.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-service-time.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-switch.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-thin-pool.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-verity.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/faulty.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/linear.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/multipath.ko 2020-02-05T16:40:40.599 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/persistent-data 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid0.ko 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid1.ko 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid10.ko 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid456.ko 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message 2020-02-05T16:40:40.600 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptbase.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptctl.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptfc.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptlan.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptsas.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptscsih.ko 2020-02-05T16:40:40.601 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptspi.ko 2020-02-05T16:40:40.602 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd 2020-02-05T16:40:40.602 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/lpc_ich.ko 2020-02-05T16:40:40.602 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/lpc_sch.ko 2020-02-05T16:40:40.602 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/mfd-core.ko 2020-02-05T16:40:40.602 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/sm501.ko 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/viperboard.ko 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/vx855.ko 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/wl1273-core.ko 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/altera-stapl 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2020-02-05T16:40:40.603 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/apds9802als.ko 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/apds990x.ko 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/bh1770glc.ko 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/cb710 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/cb710/cb710.ko 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/at24.ko 2020-02-05T16:40:40.604 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/eeprom.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/max6875.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/enclosure.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/hpilo.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/isl29003.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/isl29020.ko 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d 2020-02-05T16:40:40.605 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei-me.ko 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei-txe.ko 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei.ko 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic 2020-02-05T16:40:40.606 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic/bus 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic/bus/mic_bus.ko 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/pvpanic.ko 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-gru 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-gru/gru.ko 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xp.ko 2020-02-05T16:40:40.607 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xpc.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xpnet.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tifm_7xx1.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tifm_core.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tsl2550.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_balloon.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_vmci 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2020-02-05T16:40:40.608 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/bonding 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/bonding/bonding.ko 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/b53 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/b53/b53_common.ko 2020-02-05T16:40:40.609 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/bcm-sf2.ko 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6060.ko 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6xxx 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dummy.ko 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/eql.ko 2020-02-05T16:40:40.610 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c59x.ko 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/typhoon.ko 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390 2020-02-05T16:40:40.611 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/8390.ko 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/adaptec 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/adaptec/starfire.ko 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/agere 2020-02-05T16:40:40.612 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/agere/et131x.ko 2020-02-05T16:40:40.613 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/alteon 2020-02-05T16:40:40.613 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/alteon/acenic.ko 2020-02-05T16:40:40.613 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/altera 2020-02-05T16:40:40.613 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/altera/altera_tse.ko 2020-02-05T16:40:40.613 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/amd8111e.ko 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/pcnet32.ko 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/alx 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2020-02-05T16:40:40.614 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1c 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1e 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2020-02-05T16:40:40.615 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/b44.ko 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2x 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/cnic.ko 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/genet 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2020-02-05T16:40:40.616 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/tg3.ko 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade/bna 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2020-02-05T16:40:40.617 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb3 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/libcxgb 2020-02-05T16:40:40.618 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco/enic 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2020-02-05T16:40:40.619 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink 2020-02-05T16:40:40.620 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink/dl2k.ko 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink/sundance.ko 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dnet.ko 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex/benet 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ethoc.ko 2020-02-05T16:40:40.621 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/fealnx.ko 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e100.ko 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000e 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2020-02-05T16:40:40.622 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/fm10k 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/i40e 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/iavf 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2020-02-05T16:40:40.623 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igb 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igb/igb.ko 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igbvf 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgb 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbe 2020-02-05T16:40:40.624 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbevf 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/jme.ko 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/skge.ko 2020-02-05T16:40:40.625 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/sky2.ko 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5/core 2020-02-05T16:40:40.626 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlxfw 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlxfw/mlxfw.ko 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/micrel 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom/myri10ge 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2020-02-05T16:40:40.627 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/s2io.ko 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/vxge 2020-02-05T16:40:40.628 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/nvidia 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2020-02-05T16:40:40.629 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/netxen 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qlcnic 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rdc 2020-02-05T16:40:40.630 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rdc/r6040.ko 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/8139cp.ko 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/8139too.ko 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/atp.ko 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/r8169.ko 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rocker 2020-02-05T16:40:40.631 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rocker/rocker.ko 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sfc 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sfc/sfc.ko 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/silan 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/silan/sc92031.ko 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis/sis190.ko 2020-02-05T16:40:40.632 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis/sis900.ko 2020-02-05T16:40:40.633 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc 2020-02-05T16:40:40.633 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/epic100.ko 2020-02-05T16:40:40.633 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2020-02-05T16:40:40.633 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2020-02-05T16:40:40.633 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro/stmmac 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/cassini.ko 2020-02-05T16:40:40.634 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/niu.ko 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/sungem.ko 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/sunhme.ko 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/tehuti 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ti 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ti/tlan.ko 2020-02-05T16:40:40.635 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via/via-rhine.ko 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via/via-velocity.ko 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet/w5100.ko 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet/w5300.ko 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/xircom 2020-02-05T16:40:40.636 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/geneve.ko 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/hyperv 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/hyperv/hv_netvsc.ko 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ieee802154 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ieee802154/fakelb.ko 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ifb.ko 2020-02-05T16:40:40.637 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ipvlan 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ipvlan/ipvlan.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/macvlan.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/macvtap.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/mdio.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/mii.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/net_failover.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/netconsole.ko 2020-02-05T16:40:40.638 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/nlmon.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/amd.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm-phy-lib.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm7xxx.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm87xx.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/broadcom.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/cicada.ko 2020-02-05T16:40:40.639 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/davicom.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/dp83640.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/et1011c.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/icplus.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/lxt.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/marvell.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/mdio-bcm-unimac.ko 2020-02-05T16:40:40.640 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/mdio-bitbang.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/micrel.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/national.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/phylink.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/qsemi.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/realtek.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/smsc.ko 2020-02-05T16:40:40.641 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/ste10Xp.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/vitesse.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/bsd_comp.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_async.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_deflate.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_generic.ko 2020-02-05T16:40:40.642 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_mppe.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_synctty.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pppoe.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pppox.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pptp.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip/slhc.ko 2020-02-05T16:40:40.643 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip/slip.ko 2020-02-05T16:40:40.644 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/sungem_phy.ko 2020-02-05T16:40:40.644 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/tap.ko 2020-02-05T16:40:40.644 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team 2020-02-05T16:40:40.644 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team.ko 2020-02-05T16:40:40.644 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_activebackup.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_broadcast.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_loadbalance.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_random.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_roundrobin.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/tun.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/veth.ko 2020-02-05T16:40:40.645 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/virtio_net.ko 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vmxnet3 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vmxnet3/vmxnet3.ko 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vxlan.ko 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netback 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netback/xen-netback.ko 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netfront.ko 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ntb 2020-02-05T16:40:40.646 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ntb/ntb.ko 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host/nvme-core.ko 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host/nvme.ko 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport.ko 2020-02-05T16:40:40.647 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_cs.ko 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_pc.ko 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_serial.ko 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/controller 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/controller/pci-hyperv-intf.ko 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/hotplug 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2020-02-05T16:40:40.648 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/pcie 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/pcie/aer_inject.ko 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/xen-pcifront.ko 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/i82092.ko 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/pd6729.ko 2020-02-05T16:40:40.649 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/yenta_socket.ko 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl/intel 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86 2020-02-05T16:40:40.650 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/acer-wmi.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/acerhdf.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/alienware-wmi.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/apple-gmux.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-laptop.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-nb-wmi.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-wmi.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/classmate-laptop.ko 2020-02-05T16:40:40.651 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dcdbas.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dell-smo8800.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dell-wmi-aio.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/eeepc-laptop.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/eeepc-wmi.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/fujitsu-laptop.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/fujitsu-tablet.ko 2020-02-05T16:40:40.652 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hdaps.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp-wireless.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp-wmi.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp_accel.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/intel-rst.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/intel_ips.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/msi-wmi.ko 2020-02-05T16:40:40.653 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/mxm-wmi.ko 2020-02-05T16:40:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/panasonic-laptop.ko 2020-02-05T16:40:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/samsung-laptop.ko 2020-02-05T16:40:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/samsung-q10.ko 2020-02-05T16:40:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/thinkpad_acpi.ko 2020-02-05T16:40:40.654 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/topstar-laptop.ko 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/toshiba_haps.ko 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/wmi-bmof.ko 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/wmi.ko 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power 2020-02-05T16:40:40.655 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power/supply 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power/supply/smb347-charger.ko 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap/intel_rapl_common.ko 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap/intel_rapl_msr.ko 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients 2020-02-05T16:40:40.656 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps-gpio.ko 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps-ldisc.ko 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps_parport.ko 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/pps_core.ko 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp/ptp.ko 2020-02-05T16:40:40.657 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp/ptp_kvm.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-bq32k.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-bq4802.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1286.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1307.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1374.ko 2020-02-05T16:40:40.658 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1511.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1553.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1672.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1742.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds2404.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds3232.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-em3027.ko 2020-02-05T16:40:40.659 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-fm3130.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-isl12022.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-isl1208.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m41t80.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m48t35.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m48t59.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-max6900.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-msm6242.ko 2020-02-05T16:40:40.660 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf2127.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf85063.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8523.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8563.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8583.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rp5c01.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rs5c372.ko 2020-02-05T16:40:40.661 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rv3029c2.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rx8025.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rx8581.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-stk17ta8.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-v3020.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-x1205.ko 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi 2020-02-05T16:40:40.662 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-9xxx.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-sas.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-xxxx.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/BusLogic.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/a100u2w.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aacraid 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aacraid/aacraid.ko 2020-02-05T16:40:40.663 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/advansys.ko 2020-02-05T16:40:40.664 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx 2020-02-05T16:40:40.664 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2020-02-05T16:40:40.664 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2020-02-05T16:40:40.664 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/am53c974.ko 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/arcmsr 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/arcmsr/arcmsr.ko 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/atp870u.ko 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/be2iscsi 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2020-02-05T16:40:40.665 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bfa 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bfa/bfa.ko 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2fc 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2i 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2i/bnx2i.ko 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ch.ko 2020-02-05T16:40:40.666 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/csiostor 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/csiostor/csiostor.ko 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb3i 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb4i 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2020-02-05T16:40:40.667 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/dc395x.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/dmx3191d.ko 2020-02-05T16:40:40.668 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esas2r 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esas2r/esas2r.ko 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esp_scsi.ko 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe/fcoe.ko 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe/libfcoe.ko 2020-02-05T16:40:40.669 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fnic 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fnic/fnic.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/gdth.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hpsa.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hptiop.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hv_storvsc.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/initio.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ipr.ko 2020-02-05T16:40:40.670 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ips.ko 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/isci 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/isci/isci.ko 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/iscsi_tcp.ko 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libfc 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libfc/libfc.ko 2020-02-05T16:40:40.671 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libiscsi.ko 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libiscsi_tcp.ko 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libsas 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libsas/libsas.ko 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/lpfc 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/lpfc/lpfc.ko 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid 2020-02-05T16:40:40.672 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid.ko 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mpt3sas 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvsas 2020-02-05T16:40:40.673 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvsas/mvsas.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvumi.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pm8001 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pm8001/pm80xx.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pmcraid.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla1280.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2020-02-05T16:40:40.674 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2020-02-05T16:40:40.675 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla4xxx 2020-02-05T16:40:40.675 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2020-02-05T16:40:40.675 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/raid_class.ko 2020-02-05T16:40:40.675 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_debug.ko 2020-02-05T16:40:40.675 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_fc.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_iscsi.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_sas.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_spi.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_srp.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ses.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/st.ko 2020-02-05T16:40:40.676 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/stex.ko 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/sym53c8xx_2 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs/ufshcd-core.ko 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/virtio_scsi.ko 2020-02-05T16:40:40.677 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/vmw_pvscsi.ko 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/wd719x.ko 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ssb 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ssb/ssb.ko 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/iscsi 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2020-02-05T16:40:40.678 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/loopback 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/loopback/tcm_loop.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_file.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_iblock.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_mod.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_pscsi.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_user.ko 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/tcm_fc 2020-02-05T16:40:40.679 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/tcm_fc/tcm_fc.ko 2020-02-05T16:40:40.680 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal 2020-02-05T16:40:40.680 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel 2020-02-05T16:40:40.680 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal 2020-02-05T16:40:40.680 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2020-02-05T16:40:40.680 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2020-02-05T16:40:40.682 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2020-02-05T16:40:40.682 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2020-02-05T16:40:40.683 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/cyclades.ko 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/ipwireless 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/ipwireless/ipwireless.ko 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/n_gsm.ko 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/n_hdlc.ko 2020-02-05T16:40:40.684 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/nozomi.ko 2020-02-05T16:40:40.685 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/rocket.ko 2020-02-05T16:40:40.685 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial 2020-02-05T16:40:40.685 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/8250 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/8250/serial_cs.ko 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/arc_uart.ko 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/jsm 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/jsm/jsm.ko 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclink.ko 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclink_gt.ko 2020-02-05T16:40:40.686 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclinkmp.ko 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio.ko 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_aec.ko 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_cif.ko 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_pci_generic.ko 2020-02-05T16:40:40.687 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_sercos3.ko 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/cdc-acm.ko 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/cdc-wdm.ko 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/usblp.ko 2020-02-05T16:40:40.688 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/usbtmc.ko 2020-02-05T16:40:40.689 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host 2020-02-05T16:40:40.689 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host/sl811-hcd.ko 2020-02-05T16:40:40.689 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host/u132-hcd.ko 2020-02-05T16:40:40.689 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image 2020-02-05T16:40:40.689 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image/mdc800.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image/microtek.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/adutux.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/appledisplay.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/emi26.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/emi62.ko 2020-02-05T16:40:40.690 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ezusb.ko 2020-02-05T16:40:40.691 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ftdi-elan.ko 2020-02-05T16:40:40.691 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/idmouse.ko 2020-02-05T16:40:40.691 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/iowarrior.ko 2020-02-05T16:40:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/isight_firmware.ko 2020-02-05T16:40:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ldusb.ko 2020-02-05T16:40:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/legousbtower.ko 2020-02-05T16:40:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/sisusbvga 2020-02-05T16:40:40.692 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2020-02-05T16:40:40.693 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/trancevibrator.ko 2020-02-05T16:40:40.693 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usb3503.ko 2020-02-05T16:40:40.693 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usblcd.ko 2020-02-05T16:40:40.693 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usbsevseg.ko 2020-02-05T16:40:40.693 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/uss720.ko 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/yurex.ko 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/phy 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/phy/phy-generic.ko 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/aircable.ko 2020-02-05T16:40:40.694 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ark3116.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/belkin_sa.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ch341.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cp210x.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cyberjack.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cypress_m8.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/digi_acceleport.ko 2020-02-05T16:40:40.695 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/empeg.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ftdi_sio.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/garmin_gps.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/io_edgeport.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/io_ti.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ipaq.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ipw.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ir-usb.ko 2020-02-05T16:40:40.696 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/iuu_phoenix.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/keyspan.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/keyspan_pda.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/kl5kusb105.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/kobil_sct.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mct_u232.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mos7720.ko 2020-02-05T16:40:40.697 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mos7840.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/navman.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/omninet.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/opticon.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/option.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/oti6858.ko 2020-02-05T16:40:40.698 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/pl2303.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/qcaux.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/qcserial.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/quatech2.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/safe_serial.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/sierra.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/spcp8x5.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ssu100.ko 2020-02-05T16:40:40.699 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/symbolserial.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb-serial-simple.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb_debug.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb_wwan.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/visor.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/whiteheat.ko 2020-02-05T16:40:40.700 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/xsens_mt.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/uas.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-alauda.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-cypress.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-datafab.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-eneub6250.ko 2020-02-05T16:40:40.701 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-freecom.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-isd200.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-jumpshot.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-karma.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-onetouch.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-realtek.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-sddr09.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-sddr55.ko 2020-02-05T16:40:40.702 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-usbat.ko 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/usb-storage.ko 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/usbip-core.ko 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/usbip-host.ko 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/vhci-hcd.ko 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio 2020-02-05T16:40:40.703 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/pci 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/pci/vfio-pci.ko 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio.ko 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio_iommu_type1.ko 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio_virqfd.ko 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost 2020-02-05T16:40:40.704 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost.ko 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost_net.ko 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost_scsi.ko 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/apple_bl.ko 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/lcd.ko 2020-02-05T16:40:40.705 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/platform_lcd.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_balloon.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_mmio.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_pci.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_ring.ko 2020-02-05T16:40:40.706 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds1wm.ko 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds2482.ko 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds2490.ko 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2406.ko 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2408.ko 2020-02-05T16:40:40.707 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2413.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2423.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2431.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2433.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2780.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2781.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds28e04.ko 2020-02-05T16:40:40.708 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_smem.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_therm.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/wire.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/alim1535_wdt.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/alim7101_wdt.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/f71808e_wdt.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/hpwdt.ko 2020-02-05T16:40:40.709 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/i6300esb.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/iTCO_vendor_support.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/iTCO_wdt.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ib700wdt.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ibmasr.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ie6xx_wdt.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/it8712f_wdt.ko 2020-02-05T16:40:40.710 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/it87_wdt.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/machzwd.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/nv_tco.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/pcwd_pci.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/pcwd_usb.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sch311x_wdt.ko 2020-02-05T16:40:40.711 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/softdog.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sp5100_tco.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/via_wdt.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83627hf_wdt.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83877f_wdt.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83977f_wdt.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/wdt_pci.ko 2020-02-05T16:40:40.712 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/xen_wdt.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-acpi-processor.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-evtchn.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-gntalloc.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-gntdev.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-pciback 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2020-02-05T16:40:40.713 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-privcmd.ko 2020-02-05T16:40:40.714 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xenfs 2020-02-05T16:40:40.714 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xenfs/xenfs.ko 2020-02-05T16:40:40.714 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs 2020-02-05T16:40:40.714 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/9p 2020-02-05T16:40:40.714 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/9p/9p.ko 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/affs 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/affs/affs.ko 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/befs 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/befs/befs.ko 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/binfmt_misc.ko 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/btrfs 2020-02-05T16:40:40.715 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/btrfs/btrfs.ko 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cachefiles 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cachefiles/cachefiles.ko 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ceph 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ceph/ceph.ko 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cifs 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cifs/cifs.ko 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/coda 2020-02-05T16:40:40.716 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/coda/coda.ko 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cramfs 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cramfs/cramfs.ko 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/dlm 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/dlm/dlm.ko 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ecryptfs 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ecryptfs/ecryptfs.ko 2020-02-05T16:40:40.717 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/f2fs 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/f2fs/f2fs.ko 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/fat.ko 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/msdos.ko 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/vfat.ko 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fscache 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fscache/fscache.ko 2020-02-05T16:40:40.718 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse/cuse.ko 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse/fuse.ko 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/gfs2 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/gfs2/gfs2.ko 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfs 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfs/hfs.ko 2020-02-05T16:40:40.719 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfsplus 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfsplus/hfsplus.ko 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/isofs 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/isofs/isofs.ko 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/jfs 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/jfs/jfs.ko 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/lockd 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/lockd/lockd.ko 2020-02-05T16:40:40.720 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/minix 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/minix/minix.ko 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/blocklayout 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/filelayout 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2020-02-05T16:40:40.721 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/flexfilelayout 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfs.ko 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfsv3.ko 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfsv4.ko 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common/grace.ko 2020-02-05T16:40:40.722 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common/nfs_acl.ko 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfsd 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfsd/nfsd.ko 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nilfs2 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nilfs2/nilfs2.ko 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-celtic.ko 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-centeuro.ko 2020-02-05T16:40:40.723 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-croatian.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-cyrillic.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-gaelic.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-greek.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-iceland.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-inuit.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-roman.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-romanian.ko 2020-02-05T16:40:40.724 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-turkish.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1250.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1251.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1255.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp737.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp775.ko 2020-02-05T16:40:40.725 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp850.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp852.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp855.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp857.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp860.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp861.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp862.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp863.ko 2020-02-05T16:40:40.726 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp864.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp865.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp866.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp869.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp874.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp932.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp936.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp949.ko 2020-02-05T16:40:40.727 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp950.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_euc-jp.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-1.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-13.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-14.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-15.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-2.ko 2020-02-05T16:40:40.728 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-3.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-4.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-5.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-6.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-7.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-9.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-r.ko 2020-02-05T16:40:40.729 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-ru.ko 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-u.ko 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_utf8.ko 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/cluster 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlm 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2020-02-05T16:40:40.730 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlmfs 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stack_user.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stackglue.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/overlayfs 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/overlayfs/overlay.ko 2020-02-05T16:40:40.731 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/pstore 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/pstore/ramoops.ko 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/reiserfs 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/reiserfs/reiserfs.ko 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/romfs 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/romfs/romfs.ko 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/squashfs 2020-02-05T16:40:40.732 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/squashfs/squashfs.ko 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/sysv 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/sysv/sysv.ko 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/udf 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/udf/udf.ko 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ufs 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ufs/ufs.ko 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/xfs 2020-02-05T16:40:40.733 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/xfs/xfs.ko 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/rcu 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/rcu/rcutorture.ko 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/torture.ko 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/trace 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/trace/ring_buffer_benchmark.ko 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib 2020-02-05T16:40:40.734 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc-itu-t.ko 2020-02-05T16:40:40.735 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc64.ko 2020-02-05T16:40:40.735 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc8.ko 2020-02-05T16:40:40.735 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto 2020-02-05T16:40:40.735 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto/libarc4.ko 2020-02-05T16:40:40.735 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto/libdes.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/libcrc32c.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lru_cache.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4/lz4_compress.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4/lz4hc_compress.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/math 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/math/cordic.ko 2020-02-05T16:40:40.736 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/raid6 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/raid6/raid6_pq.ko 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/reed_solomon 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/reed_solomon/reed_solomon.ko 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_bm.ko 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_fsm.ko 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_kmp.ko 2020-02-05T16:40:40.737 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd/zstd_compress.ko 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd/zstd_decompress.ko 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/mm 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/mm/hwpoison-inject.ko 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan 2020-02-05T16:40:40.738 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/6lowpan.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_dest.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_fragment.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_hop.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_ipv6.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_mobility.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_routing.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_udp.ko 2020-02-05T16:40:40.739 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/garp.ko 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/mrp.ko 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/p8022.ko 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/psnap.ko 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/stp.ko 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/8021q 2020-02-05T16:40:40.740 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/8021q/8021q.ko 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet.ko 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet_rdma.ko 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet_virtio.ko 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/batman-adv 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/batman-adv/batman-adv.ko 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge 2020-02-05T16:40:40.741 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/br_netfilter.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/bridge.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_802_3.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_among.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_arp.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_arpreply.ko 2020-02-05T16:40:40.742 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_dnat.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_ip.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_ip6.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_limit.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_log.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_mark.ko 2020-02-05T16:40:40.743 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_mark_m.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_nflog.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_pkttype.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_redirect.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_snat.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_stp.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_vlan.ko 2020-02-05T16:40:40.744 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_broute.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_filter.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_nat.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtables.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/nf_log_bridge.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2020-02-05T16:40:40.745 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ceph 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ceph/libceph.ko 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core/failover.ko 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core/pktgen.ko 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp 2020-02-05T16:40:40.746 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_diag.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_ipv4.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_ipv6.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dns_resolver 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dns_resolver/dns_resolver.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/dsa_core.ko 2020-02-05T16:40:40.747 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_brcm.ko 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_dsa.ko 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_edsa.ko 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_trailer.ko 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/6lowpan 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2020-02-05T16:40:40.748 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/ieee802154.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/ieee802154_socket.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ah4.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/esp4.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/fou.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/gre.ko 2020-02-05T16:40:40.749 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/inet_diag.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_gre.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_tunnel.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_vti.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ipcomp.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ipip.ko 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter 2020-02-05T16:40:40.750 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arp_tables.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arpt_mangle.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arptable_filter.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_ECN.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_ah.ko 2020-02-05T16:40:40.751 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_mangle.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_nat.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_raw.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_security.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2020-02-05T16:40:40.752 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_log_arp.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_log_ipv4.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2020-02-05T16:40:40.753 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_bic.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_dctcp.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_diag.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_highspeed.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_htcp.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_hybla.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_illinois.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_lp.ko 2020-02-05T16:40:40.754 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_scalable.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_vegas.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_veno.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_westwood.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_yeah.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tunnel4.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/udp_diag.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/udp_tunnel.ko 2020-02-05T16:40:40.755 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/xfrm4_tunnel.ko 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ah6.ko 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/esp6.ko 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/fou6.ko 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_tunnel.ko 2020-02-05T16:40:40.756 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_udp_tunnel.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_vti.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ipcomp6.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6_tables.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2020-02-05T16:40:40.757 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_ah.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_frag.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_mh.ko 2020-02-05T16:40:40.758 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_rt.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_filter.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_nat.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_raw.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_security.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2020-02-05T16:40:40.759 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_log_ipv6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/sit.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/tunnel6.ko 2020-02-05T16:40:40.760 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/xfrm6_tunnel.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/key 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/key/af_key.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_core.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_debugfs.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_eth.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ip.ko 2020-02-05T16:40:40.761 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ip6.ko 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_netlink.ko 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ppp.ko 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/llc 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/llc/llc.ko 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mac802154 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mac802154/mac802154.ko 2020-02-05T16:40:40.762 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mpls 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mpls/mpls_gso.ko 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set.ko 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2020-02-05T16:40:40.763 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2020-02-05T16:40:40.764 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_list_set.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2020-02-05T16:40:40.765 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2020-02-05T16:40:40.766 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conncount.ko 2020-02-05T16:40:40.767 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_amanda.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_broadcast.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_ftp.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_h323.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_irc.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2020-02-05T16:40:40.768 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_netlink.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_pptp.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_sane.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_sip.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_snmp.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_tftp.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_log_common.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat.ko 2020-02-05T16:40:40.769 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_amanda.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_ftp.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_irc.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_sip.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_tftp.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_synproxy_core.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_tables.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink.ko 2020-02-05T16:40:40.770 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_acct.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_log.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_osf.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_queue.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_compat.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_counter.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_ct.ko 2020-02-05T16:40:40.771 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_hash.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_limit.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_log.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_masq.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_queue.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_redir.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_reject.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_AUDIT.ko 2020-02-05T16:40:40.772 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CHECKSUM.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CLASSIFY.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CONNSECMARK.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CT.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_DSCP.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_HL.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_HMARK.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_IDLETIMER.ko 2020-02-05T16:40:40.773 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_LED.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_LOG.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_MASQUERADE.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NETMAP.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NFLOG.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NFQUEUE.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_RATEEST.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_REDIRECT.ko 2020-02-05T16:40:40.774 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_SECMARK.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TCPMSS.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TEE.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TPROXY.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TRACE.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_addrtype.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_bpf.ko 2020-02-05T16:40:40.775 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cgroup.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cluster.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_comment.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connbytes.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connlabel.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connlimit.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connmark.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_conntrack.ko 2020-02-05T16:40:40.776 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cpu.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_dccp.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_devgroup.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_dscp.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ecn.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_esp.ko 2020-02-05T16:40:40.777 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_hashlimit.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_helper.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_hl.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ipcomp.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_iprange.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ipvs.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_l2tp.ko 2020-02-05T16:40:40.778 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_length.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_limit.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_mac.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_mark.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_multiport.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_nat.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_nfacct.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_osf.ko 2020-02-05T16:40:40.779 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_owner.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_physdev.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_pkttype.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_policy.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_quota.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_rateest.ko 2020-02-05T16:40:40.780 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_realm.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_recent.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_sctp.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_set.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_socket.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_state.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_statistic.ko 2020-02-05T16:40:40.781 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_string.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_tcpmss.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_time.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_u32.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netlink 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netlink/netlink_diag.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/nsh 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/nsh/nsh.ko 2020-02-05T16:40:40.782 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/openvswitch.ko 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-geneve.ko 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-gre.ko 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-vxlan.ko 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/packet 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/packet/af_packet_diag.ko 2020-02-05T16:40:40.783 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds_rdma.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds_tcp.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_csum.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_gact.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_ipt.ko 2020-02-05T16:40:40.784 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_mirred.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_nat.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_pedit.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_police.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_simple.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_skbedit.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_vlan.ko 2020-02-05T16:40:40.785 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_basic.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_bpf.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_flow.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_fw.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_route.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_rsvp.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_rsvp6.ko 2020-02-05T16:40:40.786 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_tcindex.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_u32.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_cmp.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_ipset.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_meta.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_nbyte.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_text.ko 2020-02-05T16:40:40.787 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_u32.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_cbq.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_choke.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_codel.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_drr.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_dsmark.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_fq.ko 2020-02-05T16:40:40.788 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_gred.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_hfsc.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_hhf.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_htb.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_ingress.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_mqprio.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_multiq.ko 2020-02-05T16:40:40.789 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_netem.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_pie.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_plug.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_prio.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_qfq.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_red.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_sfb.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_sfq.ko 2020-02-05T16:40:40.790 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_tbf.ko 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_teql.ko 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp/sctp.ko 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp/sctp_diag.ko 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss 2020-02-05T16:40:40.791 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/sunrpc.ko 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/xprtrdma 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc/diag.ko 2020-02-05T16:40:40.792 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc/tipc.ko 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/unix 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/unix/unix_diag.ko 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vsock.ko 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vsock_diag.ko 2020-02-05T16:40:40.793 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/xfrm 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/xfrm/xfrm_ipcomp.ko 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/encrypted-keys 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/trusted-keys 2020-02-05T16:40:40.794 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/trusted-keys/trusted.ko 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt/lib 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt/lib/irqbypass.ko 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.alias 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.alias.bin 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin.bin 2020-02-05T16:40:40.795 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin.modinfo 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.dep 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.dep.bin 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.devname 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.order 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.softdep 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.symbols 2020-02-05T16:40:40.796 INFO:teuthology.orchestra.run.smithi008.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.symbols.bin 2020-02-05T16:40:40.797 DEBUG:teuthology.task.kernel:get_image_version: 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.797 INFO:teuthology.orchestra.run.smithi008:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2020-02-05T16:40:40.822 INFO:teuthology.orchestra.run.smithi008.stdout:postinstall scriptlet (using /bin/sh): 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea -a -r /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea ]; then 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:cp /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:cp /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:rm -f /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/installkernel 5.5.0-ceph-ga11f6ef5e0ea /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:rm -f /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:40:40.823 INFO:teuthology.orchestra.run.smithi008.stdout:fi 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:preuninstall scriptlet (using /bin/sh): 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:new-kernel-pkg --remove 5.5.0-ceph-ga11f6ef5e0ea --rminitrd --initrdfile=/boot/initramfs-5.5.0-ceph-ga11f6ef5e0ea.img 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:elif [ -x /usr/bin/kernel-install ]; then 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:kernel-install remove 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:fi 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:postuninstall scriptlet (using /bin/sh): 2020-02-05T16:40:40.824 INFO:teuthology.orchestra.run.smithi008.stdout:if [ -x /sbin/update-bootloader ]; then 2020-02-05T16:40:40.825 INFO:teuthology.orchestra.run.smithi008.stdout:/sbin/update-bootloader --remove 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.825 INFO:teuthology.orchestra.run.smithi008.stdout:fi 2020-02-05T16:40:40.826 INFO:teuthology.orchestra.run.smithi008:> sudo rpm -qi grub2-tools 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Name : grub2-tools 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Epoch : 1 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Version : 2.02 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Release : 66.el8 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Architecture: x86_64 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Install Date: Wed 11 Dec 2019 02:21:36 PM UTC 2020-02-05T16:40:40.908 INFO:teuthology.orchestra.run.smithi008.stdout:Group : System Environment/Base 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Size : 8937302 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:License : GPLv3+ 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Signature : RSA/SHA256, Wed 19 Dec 2018 07:38:39 PM UTC, Key ID 199e2f91fd431d51 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Source RPM : grub2-2.02-66.el8.src.rpm 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Build Date : Wed 19 Dec 2018 06:02:52 PM UTC 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Build Host : x86-vm-08.build.eng.bos.redhat.com 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Relocations : (not relocatable) 2020-02-05T16:40:40.909 INFO:teuthology.orchestra.run.smithi008.stdout:Packager : Red Hat, Inc. 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:Vendor : Red Hat, Inc. 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:URL : http://www.gnu.org/software/grub/ 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:Summary : Support tools for GRUB. 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:Description : 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-02-05T16:40:40.910 INFO:teuthology.orchestra.run.smithi008.stdout:customizable bootloader with modular architecture. It supports a rich 2020-02-05T16:40:40.911 INFO:teuthology.orchestra.run.smithi008.stdout:variety of kernel formats, file systems, computer architectures and 2020-02-05T16:40:40.911 INFO:teuthology.orchestra.run.smithi008.stdout:hardware devices. 2020-02-05T16:40:40.911 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:40:40.911 INFO:teuthology.orchestra.run.smithi008.stdout:This subpackage provides tools for support of all platforms. 2020-02-05T16:40:40.912 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-02-05T16:40:40.912 INFO:teuthology.task.kernel:Updating grub on smithi008 to boot 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:40.912 INFO:teuthology.orchestra.run.smithi008:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-02-05T16:40:45.088 INFO:teuthology.orchestra.run.smithi008.stderr:Generating grub configuration file ... 2020-02-05T16:40:47.191 INFO:teuthology.orchestra.run.smithi008.stderr:done 2020-02-05T16:40:47.193 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:40:47.210 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.gWTzBwnwaq 2020-02-05T16:40:47.211 INFO:teuthology.orchestra.run.smithi008:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gWTzBwnwaq 2020-02-05T16:40:47.286 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.gWTzBwnwaq 2020-02-05T16:40:47.443 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.gWTzBwnwaq is 4KB 2020-02-05T16:40:47.494 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.gWTzBwnwaq 2020-02-05T16:40:47.516 INFO:teuthology.orchestra.run.smithi008:> sudo /bin/ls /boot/loader/entries || true 2020-02-05T16:40:47.577 INFO:teuthology.orchestra.run.smithi008.stdout:af745df5f7934596ab03bf505e30b004-0-rescue.conf 2020-02-05T16:40:47.578 INFO:teuthology.orchestra.run.smithi008.stdout:af745df5f7934596ab03bf505e30b004-4.18.0-80.11.2.el8_0.x86_64.conf 2020-02-05T16:40:47.578 INFO:teuthology.orchestra.run.smithi008.stdout:af745df5f7934596ab03bf505e30b004-4.18.0-80.el8.x86_64.conf 2020-02-05T16:40:47.578 INFO:teuthology.orchestra.run.smithi008.stdout:af745df5f7934596ab03bf505e30b004-5.5.0-ceph-ga11f6ef5e0ea.conf 2020-02-05T16:40:47.579 INFO:teuthology.orchestra.run.smithi008:> sudo grub2-set-default af745df5f7934596ab03bf505e30b004-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:40:47.729 INFO:teuthology.orchestra.run.smithi008:> sudo shutdown -r now 2020-02-05T16:40:47.732 INFO:teuthology.task.kernel:Installing kernel a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f on mon.b... 2020-02-05T16:40:47.732 INFO:teuthology.orchestra.run.smithi164:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2020-02-05T16:40:49.520 INFO:teuthology.orchestra.run.smithi164.stdout:Verifying... ######################################## 2020-02-05T16:40:49.814 INFO:teuthology.orchestra.run.smithi164.stdout:Preparing... ######################################## 2020-02-05T16:40:49.850 INFO:teuthology.orchestra.run.smithi164.stdout:Updating / installing... 2020-02-05T16:41:22.972 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.5.0_ceph_ga11f6ef5e0ea-1 ######################################## 2020-02-05T16:41:42.915 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2020-02-05T16:41:42.916 INFO:teuthology.orchestra.run.smithi164:> rpm -qlp /tmp/kernel.x86_64.rpm 2020-02-05T16:41:42.965 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/System.map-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/config-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/vmlinux-5.5.0-ceph-ga11f6ef5e0ea.bz2 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch 2020-02-05T16:41:42.966 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/blowfish-x86_64.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/camellia-x86_64.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2020-02-05T16:41:42.967 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crc32-pclmul.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crc32c-intel.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-avx2.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2020-02-05T16:41:42.968 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha1-ssse3.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha256-ssse3.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/sha512-ssse3.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/crypto/twofish-x86_64.ko 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel 2020-02-05T16:41:42.969 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu/mce 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm-amd.ko 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm-intel.ko 2020-02-05T16:41:42.970 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/kvm/kvm.ko 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/oprofile 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/arch/x86/oprofile/oprofile.ko 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/ansi_cprng.ko 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/anubis.ko 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/arc4.ko 2020-02-05T16:41:42.971 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_memcpy.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_pq.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_raid6_recov.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_tx.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/async_xor.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/async_tx/raid6test.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/authenc.ko 2020-02-05T16:41:42.972 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/authencesn.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blake2b_generic.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blowfish_common.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/blowfish_generic.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/camellia_generic.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast5_generic.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast6_generic.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cast_common.ko 2020-02-05T16:41:42.973 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/ccm.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cmac.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crc32_generic.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crypto_engine.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/crypto_user.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/cts.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/des_generic.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/dh_generic.ko 2020-02-05T16:41:42.974 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/echainiv.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/essiv.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/fcrypt.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/khazad.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/lz4.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/lz4hc.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/md4.ko 2020-02-05T16:41:42.975 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/michael_mic.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/pcbc.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/pcrypt.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd128.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd160.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd256.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/rmd320.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/salsa20_generic.ko 2020-02-05T16:41:42.976 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/seed.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/serpent_generic.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/sha512_generic.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tcrypt.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tea.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/tgr192.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/twofish_common.ko 2020-02-05T16:41:42.977 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/twofish_generic.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/vmac.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/wp512.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xcbc.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xor.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/crypto/xxhash_generic.ko 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi 2020-02-05T16:41:42.978 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/acpi_ipmi.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/acpi_pad.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/custom_method.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/ec_sys.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/sbs.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/sbshc.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/acpi/video.ko 2020-02-05T16:41:42.979 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/acard-ahci.ko 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/ahci_platform.ko 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/ata_generic.ko 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/libahci_platform.ko 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_acpi.ko 2020-02-05T16:41:42.980 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ali.ko 2020-02-05T16:41:42.981 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_amd.ko 2020-02-05T16:41:42.981 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_artop.ko 2020-02-05T16:41:42.981 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_atiixp.ko 2020-02-05T16:41:42.981 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_atp867x.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cmd640.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cmd64x.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_cypress.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_efar.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt366.ko 2020-02-05T16:41:42.982 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt37x.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt3x2n.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_hpt3x3.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_it8213.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_it821x.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_jmicron.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_marvell.ko 2020-02-05T16:41:42.983 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_mpiix.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_netcell.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ninja32.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ns87410.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_ns87415.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_oldpiix.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_opti.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_optidma.ko 2020-02-05T16:41:42.984 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pcmcia.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pdc2027x.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_pdc202xx_old.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_piccolo.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_rdc.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sch.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_serverworks.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sil680.ko 2020-02-05T16:41:42.985 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sis.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_sl82c105.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_triflex.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pata_via.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/pdc_adma.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_inic162x.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_mv.ko 2020-02-05T16:41:42.986 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_nv.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_promise.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_qstor.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sil.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sil24.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sis.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_svw.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_sx4.ko 2020-02-05T16:41:42.987 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_uli.ko 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_via.ko 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ata/sata_vsc.ko 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/auxdisplay 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/auxdisplay/ks0108.ko 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/bcma 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/bcma/bcma.ko 2020-02-05T16:41:42.988 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/aoe 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/aoe/aoe.ko 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/brd.ko 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/drbd 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/drbd/drbd.ko 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/floppy.ko 2020-02-05T16:41:42.989 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/loop.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/mtip32xx 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/mtip32xx/mtip32xx.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/nbd.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/null_blk.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/pktcdvd.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/rbd.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/skd.ko 2020-02-05T16:41:42.990 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/sx8.ko 2020-02-05T16:41:42.991 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/umem.ko 2020-02-05T16:41:42.991 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/virtio_blk.ko 2020-02-05T16:41:42.991 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkback 2020-02-05T16:41:42.991 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkback/xen-blkback.ko 2020-02-05T16:41:42.991 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/xen-blkfront.ko 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/zram 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/block/zram/zram.ko 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hangcheck-timer.ko 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/amd-rng.ko 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/intel-rng.ko 2020-02-05T16:41:42.992 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/timeriomem-rng.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/via-rng.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/hw_random/virtio-rng.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_devintf.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2020-02-05T16:41:42.993 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_si.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_ssif.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/lp.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/mwave 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/mwave/mwave.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia/cm4000_cs.ko 2020-02-05T16:41:42.994 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/pcmcia/cm4040_cs.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/ppdev.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tlclk.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_atmel.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_infineon.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_nsc.ko 2020-02-05T16:41:42.995 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_tis.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/tpm/tpm_tis_core.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/uv_mmtimer.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/char/virtio_console.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/acpi-cpufreq.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2020-02-05T16:41:42.996 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/p4-clockmod.ko 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/pcc-cpufreq.ko 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/powernow-k8.ko 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/cpufreq/speedstep-lib.ko 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp/ccp-crypto.ko 2020-02-05T16:41:42.997 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/ccp/ccp.ko 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/padlock-aes.ko 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/padlock-sha.ko 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_common 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_common/intel_qat.ko 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_dh895xcc 2020-02-05T16:41:42.998 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/virtio 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/crypto/virtio/virtio_crypto.ko 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dca 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dca/dca.ko 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/devfreq 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/devfreq/governor_simpleondemand.ko 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma 2020-02-05T16:41:42.999 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/dw 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/dw/dw_dmac.ko 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/ioat 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/ioat/ioatdma.ko 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/dma/mic_x100_dma.ko 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/amd64_edac_mod.ko 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/e752x_edac.ko 2020-02-05T16:41:43.000 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/edac_mce_amd.ko 2020-02-05T16:41:43.001 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i3000_edac.ko 2020-02-05T16:41:43.001 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i3200_edac.ko 2020-02-05T16:41:43.001 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5000_edac.ko 2020-02-05T16:41:43.001 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5100_edac.ko 2020-02-05T16:41:43.001 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i5400_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i7300_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i7core_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/i82975x_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/ie31200_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/sb_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/edac/x38_edac.ko 2020-02-05T16:41:43.002 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware/edd.ko 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/firmware/iscsi_ibft.ko 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio/gpio-ich.ko 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/gpio/gpio-viperboard.ko 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-appleir.ko 2020-02-05T16:41:43.003 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-aureal.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-axff.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-dr.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-elecom.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-elo.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-emsff.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gaff.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gt683r.ko 2020-02-05T16:41:43.004 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-gyration.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtek-kbd.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtek-mouse.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-holtekff.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-hyperv.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-icade.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-keytouch.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-kye.ko 2020-02-05T16:41:43.005 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-lcpower.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-led.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-lenovo.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-logitech-dj.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-logitech-hidpp.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-multitouch.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-ortek.ko 2020-02-05T16:41:43.006 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-penmount.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-petalynx.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-picolcd.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-pl.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-plantronics.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-primax.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-rmi.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-arvo.ko 2020-02-05T16:41:43.007 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-common.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-isku.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-kone.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-koneplus.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-konepure.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-kovaplus.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-lua.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-pyra.ko 2020-02-05T16:41:43.008 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-ryos.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat-savu.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-roccat.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-saitek.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-samsung.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sensor-hub.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sjoy.ko 2020-02-05T16:41:43.009 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sony.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-speedlink.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-steelseries.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-sunplus.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-tivo.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-tmff.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-topseed.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-twinhan.ko 2020-02-05T16:41:43.010 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-uclogic.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-waltop.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-wiimote.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-xinmo.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-zpff.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/hid-zydacron.ko 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/i2c-hid 2020-02-05T16:41:43.011 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/i2c-hid/i2c-hid.ko 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/uhid.ko 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hid/wacom.ko 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_balloon.ko 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_utils.ko 2020-02-05T16:41:43.012 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hv/hv_vmbus.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/abituguru.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/abituguru3.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/acpi_power_meter.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ad7414.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ad7418.ko 2020-02-05T16:41:43.013 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adc128d818.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1021.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1025.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1026.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1029.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm1031.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adm9240.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ads7828.ko 2020-02-05T16:41:43.014 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7410.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7411.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7462.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7470.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7475.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/adt7x10.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/amc6821.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/applesmc.ko 2020-02-05T16:41:43.015 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asb100.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asc7621.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/asus_atk0110.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/atxp1.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/coretemp.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/dell-smm-hwmon.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/dme1737.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ds1621.ko 2020-02-05T16:41:43.016 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ds620.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/emc1403.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/emc6w201.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f71805f.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f71882fg.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/f75375s.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/fam15h_power.ko 2020-02-05T16:41:43.017 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/fschmd.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/g760a.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/g762.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/gl518sm.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/gl520sm.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/hwmon-vid.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/i5500_temp.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/i5k_amb.ko 2020-02-05T16:41:43.018 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ibmaem.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ibmpex.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ina209.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ina2xx.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/it87.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/k10temp.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/k8temp.ko 2020-02-05T16:41:43.019 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lineage-pem.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm63.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm73.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm75.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm77.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm78.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm80.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm83.ko 2020-02-05T16:41:43.020 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm85.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm87.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm90.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm92.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm93.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95234.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95241.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/lm95245.ko 2020-02-05T16:41:43.021 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc2945.ko 2020-02-05T16:41:43.022 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4151.ko 2020-02-05T16:41:43.022 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4215.ko 2020-02-05T16:41:43.022 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4222.ko 2020-02-05T16:41:43.022 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4245.ko 2020-02-05T16:41:43.022 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4260.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ltc4261.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max16065.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max1619.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max1668.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max197.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6639.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6642.ko 2020-02-05T16:41:43.023 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6650.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/max6697.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/mcp3021.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct6683.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct6775.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/nct7802.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/ntc_thermistor.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pc87360.ko 2020-02-05T16:41:43.024 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pc87427.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pcf8591.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/adm1275.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/lm25066.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ltc2978.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max16064.ko 2020-02-05T16:41:43.025 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max34440.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/max8688.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/pmbus.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/tps40422.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ucd9000.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/ucd9200.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/pmbus/zl6100.ko 2020-02-05T16:41:43.026 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/powr1220.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch5627.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch5636.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sch56xx-common.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sht15.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sht21.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/shtc1.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/sis5595.ko 2020-02-05T16:41:43.027 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47b397.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47m1.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/smsc47m192.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/thmc50.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp102.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp103.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp401.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/tmp421.ko 2020-02-05T16:41:43.028 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/via-cputemp.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/via686a.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/vt1211.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/vt8231.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83627ehf.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83627hf.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83781d.ko 2020-02-05T16:41:43.029 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83791d.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83792d.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83793.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83795.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83l785ts.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/hwmon/w83l786ng.ko 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c 2020-02-05T16:41:43.030 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd756.ko 2020-02-05T16:41:43.031 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-amd8111.ko 2020-02-05T16:41:43.032 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-core.ko 2020-02-05T16:41:43.032 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2020-02-05T16:41:43.032 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2020-02-05T16:41:43.032 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2020-02-05T16:41:43.032 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-i801.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-isch.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-ismt.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-nforce2.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-parport-light.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-parport.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2020-02-05T16:41:43.033 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-piix4.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-scmi.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-simtec.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-sis96x.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-via.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-viapro.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/busses/i2c-viperboard.ko 2020-02-05T16:41:43.034 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-dev.ko 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-mux.ko 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-slave-eeprom.ko 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-smbus.ko 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/i2c/i2c-stub.ko 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core 2020-02-05T16:41:43.035 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_cm.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_core.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_umad.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/ib_uverbs.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/iw_cm.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/rdma_cm.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/core/rdma_ucm.ko 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw 2020-02-05T16:41:43.036 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/cxgb4 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx4 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx5 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mthca 2020-02-05T16:41:43.037 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/ipoib 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/iser 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/isert 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2020-02-05T16:41:43.038 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srp 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srpt 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/emu10k1-gp.ko 2020-02-05T16:41:43.039 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/fm801-gp.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/gameport.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/lightning.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/gameport/ns558.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/input-polldev.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/joydev.ko 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/keyboard 2020-02-05T16:41:43.040 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/keyboard/gpio_keys.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/appletouch.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/bcm5974.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/cyapatp.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/elan_i2c.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/sermouse.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/synaptics_i2c.ko 2020-02-05T16:41:43.041 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/synaptics_usb.ko 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/mouse/vsxxxaa.ko 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/rmi4 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/rmi4/rmi_core.ko 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/altera_ps2.ko 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/arc_ps2.ko 2020-02-05T16:41:43.042 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/hyperv-keyboard.ko 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/serio/serio_raw.ko 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/input/sparse-keymap.ko 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/iommu 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/iommu/amd_iommu_v2.ko 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds 2020-02-05T16:41:43.043 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-blinkm.ko 2020-02-05T16:41:43.044 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-clevo-mail.ko 2020-02-05T16:41:43.044 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lm3530.ko 2020-02-05T16:41:43.044 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp3944.ko 2020-02-05T16:41:43.045 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5521.ko 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5523.ko 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp5562.ko 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-lp55xx-common.ko 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/leds-ss4200.ko 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger 2020-02-05T16:41:43.046 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-audio.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-camera.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2020-02-05T16:41:43.047 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-timer.ko 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/leds/trigger/ledtrig-transient.ko 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/bcache 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/bcache/bcache.ko 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-bio-prison.ko 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-cache-smq.ko 2020-02-05T16:41:43.048 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-cache.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-crypt.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-delay.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-flakey.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-log-userspace.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-multipath.ko 2020-02-05T16:41:43.049 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-queue-length.ko 2020-02-05T16:41:43.050 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-raid.ko 2020-02-05T16:41:43.050 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-round-robin.ko 2020-02-05T16:41:43.050 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-service-time.ko 2020-02-05T16:41:43.050 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-switch.ko 2020-02-05T16:41:43.050 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-thin-pool.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/dm-verity.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/faulty.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/linear.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/multipath.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/persistent-data 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2020-02-05T16:41:43.051 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid0.ko 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid1.ko 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid10.ko 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/md/raid456.ko 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptbase.ko 2020-02-05T16:41:43.052 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptctl.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptfc.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptlan.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptsas.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptscsih.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/message/fusion/mptspi.ko 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd 2020-02-05T16:41:43.053 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/lpc_ich.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/lpc_sch.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/mfd-core.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/sm501.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/viperboard.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/vx855.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/mfd/wl1273-core.ko 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc 2020-02-05T16:41:43.054 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/altera-stapl 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/apds9802als.ko 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/apds990x.ko 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/bh1770glc.ko 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/cb710 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/cb710/cb710.ko 2020-02-05T16:41:43.055 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/at24.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/eeprom.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/eeprom/max6875.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/enclosure.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/hpilo.ko 2020-02-05T16:41:43.056 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/isl29003.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/isl29020.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei-me.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei-txe.ko 2020-02-05T16:41:43.057 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mei/mei.ko 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic/bus 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/mic/bus/mic_bus.ko 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/pvpanic.ko 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-gru 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-gru/gru.ko 2020-02-05T16:41:43.058 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xp.ko 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xpc.ko 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/sgi-xp/xpnet.ko 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tifm_7xx1.ko 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tifm_core.ko 2020-02-05T16:41:43.059 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/tsl2550.ko 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_balloon.ko 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_vmci 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/bonding 2020-02-05T16:41:43.060 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/bonding/bonding.ko 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/b53 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/b53/b53_common.ko 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/bcm-sf2.ko 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6060.ko 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6xxx 2020-02-05T16:41:43.061 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/dummy.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/eql.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/3c59x.ko 2020-02-05T16:41:43.062 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/3com/typhoon.ko 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/8390.ko 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/adaptec 2020-02-05T16:41:43.063 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/adaptec/starfire.ko 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/agere 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/agere/et131x.ko 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/alteon 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/alteon/acenic.ko 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/altera 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/altera/altera_tse.ko 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd 2020-02-05T16:41:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/amd8111e.ko 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/amd/pcnet32.ko 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/alx 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1c 2020-02-05T16:41:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1e 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/b44.ko 2020-02-05T16:41:43.066 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2x 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/cnic.ko 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/genet 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/broadcom/tg3.ko 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade 2020-02-05T16:41:43.067 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade/bna 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb3 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4 2020-02-05T16:41:43.068 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/libcxgb 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco/enic 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2020-02-05T16:41:43.069 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec 2020-02-05T16:41:43.070 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip 2020-02-05T16:41:43.070 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2020-02-05T16:41:43.070 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/de4x5.ko 2020-02-05T16:41:43.070 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2020-02-05T16:41:43.070 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink/dl2k.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dlink/sundance.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/dnet.ko 2020-02-05T16:41:43.071 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex/benet 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ethoc.ko 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/fealnx.ko 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e100.ko 2020-02-05T16:41:43.072 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000e 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/fm10k 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/i40e 2020-02-05T16:41:43.073 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/iavf 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igb 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igb/igb.ko 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igbvf 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgb 2020-02-05T16:41:43.074 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgb/ixgb.ko 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbe 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbevf 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/jme.ko 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell 2020-02-05T16:41:43.075 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/skge.ko 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/marvell/sky2.ko 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5 2020-02-05T16:41:43.076 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5/core 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlxfw 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/mellanox/mlxfw/mlxfw.ko 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/micrel 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom 2020-02-05T16:41:43.077 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom/myri10ge 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/s2io.ko 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/vxge 2020-02-05T16:41:43.078 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/neterion/vxge/vxge.ko 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/nvidia 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic 2020-02-05T16:41:43.079 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/netxen 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qlcnic 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rdc 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rdc/r6040.ko 2020-02-05T16:41:43.080 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/8139cp.ko 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/8139too.ko 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/atp.ko 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/realtek/r8169.ko 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rocker 2020-02-05T16:41:43.081 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/rocker/rocker.ko 2020-02-05T16:41:43.082 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sfc 2020-02-05T16:41:43.082 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sfc/sfc.ko 2020-02-05T16:41:43.082 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/silan 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/silan/sc92031.ko 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis/sis190.ko 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sis/sis900.ko 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/epic100.ko 2020-02-05T16:41:43.083 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro/stmmac 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun 2020-02-05T16:41:43.084 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/cassini.ko 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/niu.ko 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/sungem.ko 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/sun/sunhme.ko 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/tehuti 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ti 2020-02-05T16:41:43.085 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/ti/tlan.ko 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via/via-rhine.ko 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/via/via-velocity.ko 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet/w5100.ko 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/wiznet/w5300.ko 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/xircom 2020-02-05T16:41:43.086 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/geneve.ko 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/hyperv 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/hyperv/hv_netvsc.ko 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ieee802154 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ieee802154/fakelb.ko 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ifb.ko 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ipvlan 2020-02-05T16:41:43.087 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ipvlan/ipvlan.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/macvlan.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/macvtap.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/mdio.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/mii.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/net_failover.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/netconsole.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/nlmon.ko 2020-02-05T16:41:43.088 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/amd.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm-phy-lib.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm7xxx.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/bcm87xx.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/broadcom.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/cicada.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/davicom.ko 2020-02-05T16:41:43.089 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/dp83640.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/et1011c.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/icplus.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/lxt.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/marvell.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/mdio-bcm-unimac.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/mdio-bitbang.ko 2020-02-05T16:41:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/micrel.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/national.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/phylink.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/qsemi.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/realtek.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/smsc.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/ste10Xp.ko 2020-02-05T16:41:43.091 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/phy/vitesse.ko 2020-02-05T16:41:43.092 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp 2020-02-05T16:41:43.092 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/bsd_comp.ko 2020-02-05T16:41:43.092 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_async.ko 2020-02-05T16:41:43.092 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_deflate.ko 2020-02-05T16:41:43.092 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_generic.ko 2020-02-05T16:41:43.093 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_mppe.ko 2020-02-05T16:41:43.093 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/ppp_synctty.ko 2020-02-05T16:41:43.093 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pppoe.ko 2020-02-05T16:41:43.093 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pppox.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/ppp/pptp.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip/slhc.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/slip/slip.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/sungem_phy.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/tap.ko 2020-02-05T16:41:43.094 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_activebackup.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_broadcast.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_loadbalance.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_random.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/team/team_mode_roundrobin.ko 2020-02-05T16:41:43.095 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/tun.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/veth.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/virtio_net.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vmxnet3 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vmxnet3/vmxnet3.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/vxlan.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netback 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netback/xen-netback.ko 2020-02-05T16:41:43.096 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/net/xen-netfront.ko 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ntb 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ntb/ntb.ko 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host/nvme-core.ko 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/nvme/host/nvme.ko 2020-02-05T16:41:43.097 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport.ko 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_cs.ko 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_pc.ko 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/parport/parport_serial.ko 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/controller 2020-02-05T16:41:43.098 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/controller/pci-hyperv-intf.ko 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/hotplug 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/pcie 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/pcie/aer_inject.ko 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pci/xen-pcifront.ko 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia 2020-02-05T16:41:43.099 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/i82092.ko 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/pd6729.ko 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pcmcia/yenta_socket.ko 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl/intel 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform 2020-02-05T16:41:43.100 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/acer-wmi.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/acerhdf.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/alienware-wmi.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/apple-gmux.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-laptop.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-nb-wmi.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/asus-wmi.ko 2020-02-05T16:41:43.101 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/classmate-laptop.ko 2020-02-05T16:41:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dcdbas.ko 2020-02-05T16:41:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dell-smo8800.ko 2020-02-05T16:41:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/dell-wmi-aio.ko 2020-02-05T16:41:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/eeepc-laptop.ko 2020-02-05T16:41:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/eeepc-wmi.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/fujitsu-laptop.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/fujitsu-tablet.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hdaps.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp-wireless.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp-wmi.ko 2020-02-05T16:41:43.103 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/hp_accel.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/intel-rst.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/intel_ips.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/msi-wmi.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/mxm-wmi.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/panasonic-laptop.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/samsung-laptop.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/samsung-q10.ko 2020-02-05T16:41:43.104 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/thinkpad_acpi.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/topstar-laptop.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/toshiba_haps.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/wmi-bmof.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/platform/x86/wmi.ko 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power 2020-02-05T16:41:43.105 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power/supply 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/power/supply/smb347-charger.ko 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap/intel_rapl_common.ko 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/powercap/intel_rapl_msr.ko 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients 2020-02-05T16:41:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps-gpio.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps-ldisc.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/clients/pps_parport.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/pps/pps_core.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp/ptp.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ptp/ptp_kvm.ko 2020-02-05T16:41:43.107 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-bq32k.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-bq4802.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1286.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1307.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1374.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1511.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1553.ko 2020-02-05T16:41:43.108 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1672.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds1742.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds2404.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-ds3232.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-em3027.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-fm3130.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-isl12022.ko 2020-02-05T16:41:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-isl1208.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m41t80.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m48t35.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-m48t59.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-max6900.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-msm6242.ko 2020-02-05T16:41:43.110 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf2127.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf85063.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8523.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8563.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-pcf8583.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rp5c01.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rs5c372.ko 2020-02-05T16:41:43.111 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rv3029c2.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rx8025.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-rx8581.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-stk17ta8.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-v3020.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/rtc/rtc-x1205.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-9xxx.ko 2020-02-05T16:41:43.112 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-sas.ko 2020-02-05T16:41:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/3w-xxxx.ko 2020-02-05T16:41:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/BusLogic.ko 2020-02-05T16:41:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/a100u2w.ko 2020-02-05T16:41:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aacraid 2020-02-05T16:41:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aacraid/aacraid.ko 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/advansys.ko 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/am53c974.ko 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/arcmsr 2020-02-05T16:41:43.114 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/arcmsr/arcmsr.ko 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/atp870u.ko 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/be2iscsi 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bfa 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bfa/bfa.ko 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2fc 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2020-02-05T16:41:43.115 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2i 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/bnx2i/bnx2i.ko 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ch.ko 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/csiostor 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/csiostor/csiostor.ko 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb3i 2020-02-05T16:41:43.116 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb4i 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/dc395x.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2020-02-05T16:41:43.117 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/dmx3191d.ko 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esas2r 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esas2r/esas2r.ko 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/esp_scsi.ko 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe 2020-02-05T16:41:43.118 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe/fcoe.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fcoe/libfcoe.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fnic 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/fnic/fnic.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/gdth.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hpsa.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hptiop.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/hv_storvsc.ko 2020-02-05T16:41:43.119 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/initio.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ipr.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ips.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/isci 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/isci/isci.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/iscsi_tcp.ko 2020-02-05T16:41:43.120 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libfc 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libfc/libfc.ko 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libiscsi.ko 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libiscsi_tcp.ko 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libsas 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/libsas/libsas.ko 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/lpfc 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/lpfc/lpfc.ko 2020-02-05T16:41:43.121 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid.ko 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mpt3sas 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2020-02-05T16:41:43.122 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvsas 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvsas/mvsas.ko 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/mvumi.ko 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pm8001 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pm8001/pm80xx.ko 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/pmcraid.ko 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla1280.ko 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx 2020-02-05T16:41:43.123 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla4xxx 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/raid_class.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_debug.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_fc.ko 2020-02-05T16:41:43.124 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_iscsi.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_sas.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_spi.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/scsi_transport_srp.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ses.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/st.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/stex.ko 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/sym53c8xx_2 2020-02-05T16:41:43.125 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs/ufshcd-core.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/ufs/ufshcd-pci.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/virtio_scsi.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/vmw_pvscsi.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/scsi/wd719x.ko 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ssb 2020-02-05T16:41:43.126 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/ssb/ssb.ko 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/iscsi 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/loopback 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/loopback/tcm_loop.ko 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_file.ko 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_iblock.ko 2020-02-05T16:41:43.127 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_mod.ko 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_pscsi.ko 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/target_core_user.ko 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/tcm_fc 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/target/tcm_fc/tcm_fc.ko 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal 2020-02-05T16:41:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2020-02-05T16:41:43.129 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/cyclades.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/ipwireless 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/ipwireless/ipwireless.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/n_gsm.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/n_hdlc.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/nozomi.ko 2020-02-05T16:41:43.130 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/rocket.ko 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/8250 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/8250/serial_cs.ko 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/arc_uart.ko 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/jsm 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/serial/jsm/jsm.ko 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclink.ko 2020-02-05T16:41:43.131 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclink_gt.ko 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/tty/synclinkmp.ko 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio.ko 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_aec.ko 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_cif.ko 2020-02-05T16:41:43.132 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_pci_generic.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/uio/uio_sercos3.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/cdc-acm.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/cdc-wdm.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/usblp.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/class/usbtmc.ko 2020-02-05T16:41:43.133 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host/sl811-hcd.ko 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/host/u132-hcd.ko 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image/mdc800.ko 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/image/microtek.ko 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc 2020-02-05T16:41:43.134 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/adutux.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/appledisplay.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/emi26.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/emi62.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ezusb.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ftdi-elan.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/idmouse.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/iowarrior.ko 2020-02-05T16:41:43.135 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/isight_firmware.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/ldusb.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/legousbtower.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/sisusbvga 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/trancevibrator.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usb3503.ko 2020-02-05T16:41:43.136 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usblcd.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/usbsevseg.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/uss720.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/misc/yurex.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/phy 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/phy/phy-generic.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/aircable.ko 2020-02-05T16:41:43.137 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ark3116.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/belkin_sa.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ch341.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cp210x.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cyberjack.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/cypress_m8.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/digi_acceleport.ko 2020-02-05T16:41:43.138 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/empeg.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ftdi_sio.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/garmin_gps.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/io_edgeport.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/io_ti.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ipaq.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ipw.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ir-usb.ko 2020-02-05T16:41:43.139 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/iuu_phoenix.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/keyspan.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/keyspan_pda.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/kl5kusb105.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/kobil_sct.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mct_u232.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mos7720.ko 2020-02-05T16:41:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/mos7840.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/navman.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/omninet.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/opticon.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/option.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/oti6858.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/pl2303.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/qcaux.ko 2020-02-05T16:41:43.141 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/qcserial.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/quatech2.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/safe_serial.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/sierra.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/spcp8x5.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ssu100.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/symbolserial.ko 2020-02-05T16:41:43.142 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb-serial-simple.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb_debug.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/usb_wwan.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/visor.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/whiteheat.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/serial/xsens_mt.ko 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage 2020-02-05T16:41:43.143 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/uas.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-alauda.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-cypress.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-datafab.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-eneub6250.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-freecom.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-isd200.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-jumpshot.ko 2020-02-05T16:41:43.144 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-karma.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-onetouch.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-realtek.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-sddr09.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-sddr55.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/ums-usbat.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/storage/usb-storage.ko 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip 2020-02-05T16:41:43.145 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/usbip-core.ko 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/usbip-host.ko 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/usb/usbip/vhci-hcd.ko 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/pci 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/pci/vfio-pci.ko 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio.ko 2020-02-05T16:41:43.146 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio_iommu_type1.ko 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vfio/vfio_virqfd.ko 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost.ko 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost_net.ko 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/vhost/vhost_scsi.ko 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight 2020-02-05T16:41:43.147 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/apple_bl.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/lcd.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/video/backlight/platform_lcd.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_balloon.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_mmio.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_pci.ko 2020-02-05T16:41:43.148 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/virtio/virtio_ring.ko 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds1wm.ko 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds2482.ko 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/masters/ds2490.ko 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves 2020-02-05T16:41:43.149 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2406.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2408.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2413.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2423.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2431.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2433.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2780.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds2781.ko 2020-02-05T16:41:43.150 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_ds28e04.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_smem.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/slaves/w1_therm.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/w1/wire.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/alim1535_wdt.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/alim7101_wdt.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/f71808e_wdt.ko 2020-02-05T16:41:43.151 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/hpwdt.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/i6300esb.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/iTCO_vendor_support.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/iTCO_wdt.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ib700wdt.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ibmasr.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/ie6xx_wdt.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/it8712f_wdt.ko 2020-02-05T16:41:43.152 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/it87_wdt.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/machzwd.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/nv_tco.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/pcwd_pci.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/pcwd_usb.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sch311x_wdt.ko 2020-02-05T16:41:43.153 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/softdog.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/sp5100_tco.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/via_wdt.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83627hf_wdt.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83877f_wdt.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/w83977f_wdt.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/wdt_pci.ko 2020-02-05T16:41:43.154 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/watchdog/xen_wdt.ko 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-acpi-processor.ko 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-evtchn.ko 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-gntalloc.ko 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-gntdev.ko 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-pciback 2020-02-05T16:41:43.155 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xen-privcmd.ko 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xenfs 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/drivers/xen/xenfs/xenfs.ko 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/9p 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/9p/9p.ko 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/affs 2020-02-05T16:41:43.156 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/affs/affs.ko 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/befs 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/befs/befs.ko 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/binfmt_misc.ko 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/btrfs 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/btrfs/btrfs.ko 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cachefiles 2020-02-05T16:41:43.157 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cachefiles/cachefiles.ko 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ceph 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ceph/ceph.ko 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cifs 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cifs/cifs.ko 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/coda 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/coda/coda.ko 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cramfs 2020-02-05T16:41:43.158 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/cramfs/cramfs.ko 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/dlm 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/dlm/dlm.ko 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ecryptfs 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ecryptfs/ecryptfs.ko 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/f2fs 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/f2fs/f2fs.ko 2020-02-05T16:41:43.159 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/fat.ko 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/msdos.ko 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fat/vfat.ko 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fscache 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fscache/fscache.ko 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse/cuse.ko 2020-02-05T16:41:43.160 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/fuse/fuse.ko 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/gfs2 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/gfs2/gfs2.ko 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfs 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfs/hfs.ko 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfsplus 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/hfsplus/hfsplus.ko 2020-02-05T16:41:43.161 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/isofs 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/isofs/isofs.ko 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/jfs 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/jfs/jfs.ko 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/lockd 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/lockd/lockd.ko 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/minix 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/minix/minix.ko 2020-02-05T16:41:43.162 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/blocklayout 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/filelayout 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/flexfilelayout 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2020-02-05T16:41:43.163 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfs.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfsv3.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs/nfsv4.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common/grace.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfs_common/nfs_acl.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfsd 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nfsd/nfsd.ko 2020-02-05T16:41:43.164 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nilfs2 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nilfs2/nilfs2.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-celtic.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-centeuro.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-croatian.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-cyrillic.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-gaelic.ko 2020-02-05T16:41:43.165 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-greek.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-iceland.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-inuit.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-roman.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-romanian.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/mac-turkish.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1250.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1251.ko 2020-02-05T16:41:43.166 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp1255.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp737.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp775.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp850.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp852.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp855.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp857.ko 2020-02-05T16:41:43.167 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp860.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp861.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp862.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp863.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp864.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp865.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp866.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp869.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp874.ko 2020-02-05T16:41:43.168 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp932.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp936.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp949.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_cp950.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_euc-jp.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-1.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-13.ko 2020-02-05T16:41:43.169 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-14.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-15.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-2.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-3.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-4.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-5.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-6.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-7.ko 2020-02-05T16:41:43.170 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_iso8859-9.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-r.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-ru.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_koi8-u.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/nls/nls_utf8.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/cluster 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2020-02-05T16:41:43.171 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlm 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlmfs 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stack_user.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ocfs2/ocfs2_stackglue.ko 2020-02-05T16:41:43.172 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/overlayfs 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/overlayfs/overlay.ko 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/pstore 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/pstore/ramoops.ko 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/reiserfs 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/reiserfs/reiserfs.ko 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/romfs 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/romfs/romfs.ko 2020-02-05T16:41:43.173 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/squashfs 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/squashfs/squashfs.ko 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/sysv 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/sysv/sysv.ko 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/udf 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/udf/udf.ko 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ufs 2020-02-05T16:41:43.174 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/ufs/ufs.ko 2020-02-05T16:41:43.175 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/xfs 2020-02-05T16:41:43.175 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/fs/xfs/xfs.ko 2020-02-05T16:41:43.175 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel 2020-02-05T16:41:43.175 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/rcu 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/rcu/rcutorture.ko 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/torture.ko 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/trace 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/kernel/trace/ring_buffer_benchmark.ko 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc-itu-t.ko 2020-02-05T16:41:43.176 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc64.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crc8.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto/libarc4.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/crypto/libdes.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/libcrc32c.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lru_cache.ko 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4 2020-02-05T16:41:43.177 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4/lz4_compress.ko 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/lz4/lz4hc_compress.ko 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/math 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/math/cordic.ko 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/raid6 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/raid6/raid6_pq.ko 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/reed_solomon 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/reed_solomon/reed_solomon.ko 2020-02-05T16:41:43.178 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_bm.ko 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_fsm.ko 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/ts_kmp.ko 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd/zstd_compress.ko 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/lib/zstd/zstd_decompress.ko 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/mm 2020-02-05T16:41:43.179 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/mm/hwpoison-inject.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/6lowpan.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_dest.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_fragment.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_hop.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_ipv6.ko 2020-02-05T16:41:43.180 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_mobility.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_routing.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/6lowpan/nhc_udp.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/garp.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/mrp.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/p8022.ko 2020-02-05T16:41:43.181 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/psnap.ko 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/802/stp.ko 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/8021q 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/8021q/8021q.ko 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet.ko 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet_rdma.ko 2020-02-05T16:41:43.182 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/9p/9pnet_virtio.ko 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/batman-adv 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/batman-adv/batman-adv.ko 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/br_netfilter.ko 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/bridge.ko 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_802_3.ko 2020-02-05T16:41:43.183 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_among.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_arp.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_arpreply.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_dnat.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_ip.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_ip6.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_limit.ko 2020-02-05T16:41:43.184 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_log.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_mark.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_mark_m.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_nflog.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_pkttype.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_redirect.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_snat.ko 2020-02-05T16:41:43.185 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_stp.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebt_vlan.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_broute.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_filter.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtable_nat.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/ebtables.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/nf_log_bridge.ko 2020-02-05T16:41:43.186 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ceph 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ceph/libceph.ko 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core/failover.ko 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/core/pktgen.ko 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp.ko 2020-02-05T16:41:43.187 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_diag.ko 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_ipv4.ko 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dccp/dccp_ipv6.ko 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dns_resolver 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dns_resolver/dns_resolver.ko 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/dsa_core.ko 2020-02-05T16:41:43.188 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_brcm.ko 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_dsa.ko 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_edsa.ko 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/dsa/tag_trailer.ko 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/6lowpan 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2020-02-05T16:41:43.189 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/ieee802154.ko 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ieee802154/ieee802154_socket.ko 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ah4.ko 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/esp4.ko 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/fou.ko 2020-02-05T16:41:43.190 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/gre.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/inet_diag.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_gre.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_tunnel.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ip_vti.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ipcomp.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/ipip.ko 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter 2020-02-05T16:41:43.191 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arp_tables.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arpt_mangle.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/arptable_filter.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_CLUSTERIP.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_ECN.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_ah.ko 2020-02-05T16:41:43.192 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_mangle.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_nat.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_raw.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/iptable_security.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2020-02-05T16:41:43.193 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_log_arp.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_log_ipv4.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2020-02-05T16:41:43.194 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_bic.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_dctcp.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_diag.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_highspeed.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_htcp.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_hybla.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_illinois.ko 2020-02-05T16:41:43.195 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_lp.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_scalable.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_vegas.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_veno.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_westwood.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tcp_yeah.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/tunnel4.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/udp_diag.ko 2020-02-05T16:41:43.196 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/udp_tunnel.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv4/xfrm4_tunnel.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ah6.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/esp6.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/fou6.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_tunnel.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_udp_tunnel.ko 2020-02-05T16:41:43.197 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ip6_vti.ko 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/ipcomp6.ko 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6_tables.ko 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2020-02-05T16:41:43.198 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_ah.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_frag.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_mh.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2020-02-05T16:41:43.199 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6t_rt.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_filter.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_nat.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_raw.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/ip6table_security.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2020-02-05T16:41:43.200 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_log_ipv6.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/sit.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/tunnel6.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/ipv6/xfrm6_tunnel.ko 2020-02-05T16:41:43.201 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/key 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/key/af_key.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_core.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_debugfs.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_eth.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ip.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ip6.ko 2020-02-05T16:41:43.202 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_netlink.ko 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/l2tp/l2tp_ppp.ko 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/llc 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/llc/llc.ko 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mac802154 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mac802154/mac802154.ko 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mpls 2020-02-05T16:41:43.203 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/mpls/mpls_gso.ko 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set.ko 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2020-02-05T16:41:43.204 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2020-02-05T16:41:43.205 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipset/ip_set_list_set.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2020-02-05T16:41:43.206 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2020-02-05T16:41:43.207 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conncount.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_amanda.ko 2020-02-05T16:41:43.208 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_broadcast.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_ftp.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_h323.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_irc.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_netlink.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_pptp.ko 2020-02-05T16:41:43.209 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_sane.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_sip.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_snmp.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_conntrack_tftp.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_log_common.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_amanda.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_ftp.ko 2020-02-05T16:41:43.210 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_irc.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_sip.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_nat_tftp.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_synproxy_core.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nf_tables.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink.ko 2020-02-05T16:41:43.211 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_acct.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_log.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_osf.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nfnetlink_queue.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_compat.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_counter.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_ct.ko 2020-02-05T16:41:43.212 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_hash.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_limit.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_log.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_masq.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_queue.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_redir.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/nft_reject.ko 2020-02-05T16:41:43.213 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_AUDIT.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CHECKSUM.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CLASSIFY.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CONNSECMARK.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_CT.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_DSCP.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_HL.ko 2020-02-05T16:41:43.214 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_HMARK.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_IDLETIMER.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_LED.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_LOG.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_MASQUERADE.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NETMAP.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NFLOG.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_NFQUEUE.ko 2020-02-05T16:41:43.215 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_RATEEST.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_REDIRECT.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_SECMARK.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TCPMSS.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TEE.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TPROXY.ko 2020-02-05T16:41:43.216 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_TRACE.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_addrtype.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_bpf.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cgroup.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cluster.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_comment.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connbytes.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connlabel.ko 2020-02-05T16:41:43.217 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connlimit.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_connmark.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_conntrack.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_cpu.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_dccp.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_devgroup.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_dscp.ko 2020-02-05T16:41:43.218 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ecn.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_esp.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_hashlimit.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_helper.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_hl.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ipcomp.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_iprange.ko 2020-02-05T16:41:43.219 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_ipvs.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_l2tp.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_length.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_limit.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_mac.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_mark.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_multiport.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_nat.ko 2020-02-05T16:41:43.220 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_nfacct.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_osf.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_owner.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_physdev.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_pkttype.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_policy.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_quota.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_rateest.ko 2020-02-05T16:41:43.221 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_realm.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_recent.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_sctp.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_set.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_socket.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_state.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_statistic.ko 2020-02-05T16:41:43.222 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_string.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_tcpmss.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_time.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netfilter/xt_u32.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netlink 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/netlink/netlink_diag.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/nsh 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/nsh/nsh.ko 2020-02-05T16:41:43.223 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/openvswitch.ko 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-geneve.ko 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-gre.ko 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/openvswitch/vport-vxlan.ko 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/packet 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/packet/af_packet_diag.ko 2020-02-05T16:41:43.224 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds.ko 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds_rdma.ko 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/rds/rds_tcp.ko 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_csum.ko 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_gact.ko 2020-02-05T16:41:43.225 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_ipt.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_mirred.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_nat.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_pedit.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_police.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_simple.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_skbedit.ko 2020-02-05T16:41:43.226 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/act_vlan.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_basic.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_bpf.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_flow.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_fw.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_route.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_rsvp.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_rsvp6.ko 2020-02-05T16:41:43.227 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_tcindex.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/cls_u32.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_cmp.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_ipset.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_meta.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_nbyte.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_text.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/em_u32.ko 2020-02-05T16:41:43.228 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_cbq.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_choke.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_codel.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_drr.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_dsmark.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_fq.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_gred.ko 2020-02-05T16:41:43.229 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_hfsc.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_hhf.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_htb.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_ingress.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_mqprio.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_multiq.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_netem.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_pie.ko 2020-02-05T16:41:43.230 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_plug.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_prio.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_qfq.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_red.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_sfb.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_sfq.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_tbf.ko 2020-02-05T16:41:43.231 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sched/sch_teql.ko 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp/sctp.ko 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sctp/sctp_diag.ko 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2020-02-05T16:41:43.232 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/sunrpc.ko 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/xprtrdma 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc/diag.ko 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/tipc/tipc.ko 2020-02-05T16:41:43.233 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/unix 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/unix/unix_diag.ko 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vsock.ko 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/vmw_vsock/vsock_diag.ko 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/xfrm 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/net/xfrm/xfrm_ipcomp.ko 2020-02-05T16:41:43.234 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/encrypted-keys 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/trusted-keys 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/security/keys/trusted-keys/trusted.ko 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt/lib 2020-02-05T16:41:43.235 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/kernel/virt/lib/irqbypass.ko 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.alias 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.alias.bin 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin.bin 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.builtin.modinfo 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.dep 2020-02-05T16:41:43.236 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.dep.bin 2020-02-05T16:41:43.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.devname 2020-02-05T16:41:43.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.order 2020-02-05T16:41:43.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.softdep 2020-02-05T16:41:43.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.symbols 2020-02-05T16:41:43.237 INFO:teuthology.orchestra.run.smithi164.stdout:/lib/modules/5.5.0-ceph-ga11f6ef5e0ea/modules.symbols.bin 2020-02-05T16:41:43.238 DEBUG:teuthology.task.kernel:get_image_version: 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:43.238 INFO:teuthology.orchestra.run.smithi164:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2020-02-05T16:41:43.261 INFO:teuthology.orchestra.run.smithi164.stdout:postinstall scriptlet (using /bin/sh): 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea -a -r /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea ]; then 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:cp /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:cp /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:rm -f /boot/vmlinuz-5.5.0-ceph-ga11f6ef5e0ea /boot/System.map-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:/sbin/installkernel 5.5.0-ceph-ga11f6ef5e0ea /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:rm -f /boot/.vmlinuz-5.5.0-ceph-ga11f6ef5e0ea-rpm /boot/.System.map-5.5.0-ceph-ga11f6ef5e0ea-rpm 2020-02-05T16:41:43.262 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:preuninstall scriptlet (using /bin/sh): 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:new-kernel-pkg --remove 5.5.0-ceph-ga11f6ef5e0ea --rminitrd --initrdfile=/boot/initramfs-5.5.0-ceph-ga11f6ef5e0ea.img 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:elif [ -x /usr/bin/kernel-install ]; then 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-install remove 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:postuninstall scriptlet (using /bin/sh): 2020-02-05T16:41:43.263 INFO:teuthology.orchestra.run.smithi164.stdout:if [ -x /sbin/update-bootloader ]; then 2020-02-05T16:41:43.264 INFO:teuthology.orchestra.run.smithi164.stdout:/sbin/update-bootloader --remove 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:43.264 INFO:teuthology.orchestra.run.smithi164.stdout:fi 2020-02-05T16:41:43.265 INFO:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2020-02-05T16:41:43.342 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2020-02-05T16:41:43.342 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2020-02-05T16:41:43.342 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.02 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 66.el8 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Wed 11 Dec 2019 02:21:36 PM UTC 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Group : System Environment/Base 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8937302 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2020-02-05T16:41:43.343 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Wed 19 Dec 2018 07:38:39 PM UTC, Key ID 199e2f91fd431d51 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.02-66.el8.src.rpm 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 19 Dec 2018 06:02:52 PM UTC 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-vm-08.build.eng.bos.redhat.com 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Relocations : (not relocatable) 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : Red Hat, Inc. 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : Red Hat, Inc. 2020-02-05T16:41:43.344 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2020-02-05T16:41:43.345 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2020-02-05T16:41:43.346 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:41:43.346 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2020-02-05T16:41:43.346 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-02-05T16:41:43.347 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:43.347 INFO:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-02-05T16:41:48.528 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2020-02-05T16:41:50.757 INFO:teuthology.orchestra.run.smithi164.stderr:done 2020-02-05T16:41:50.759 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:41:50.774 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.XJnbSBEudD 2020-02-05T16:41:50.774 INFO:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XJnbSBEudD 2020-02-05T16:41:50.852 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.XJnbSBEudD 2020-02-05T16:41:51.017 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.XJnbSBEudD is 4KB 2020-02-05T16:41:51.065 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.XJnbSBEudD 2020-02-05T16:41:51.079 INFO:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2020-02-05T16:41:51.143 INFO:teuthology.orchestra.run.smithi164.stdout:af745df5f7934596ab03bf505e30b004-0-rescue.conf 2020-02-05T16:41:51.144 INFO:teuthology.orchestra.run.smithi164.stdout:af745df5f7934596ab03bf505e30b004-4.18.0-80.11.2.el8_0.x86_64.conf 2020-02-05T16:41:51.144 INFO:teuthology.orchestra.run.smithi164.stdout:af745df5f7934596ab03bf505e30b004-4.18.0-80.el8.x86_64.conf 2020-02-05T16:41:51.144 INFO:teuthology.orchestra.run.smithi164.stdout:af745df5f7934596ab03bf505e30b004-5.5.0-ceph-ga11f6ef5e0ea.conf 2020-02-05T16:41:51.145 INFO:teuthology.orchestra.run.smithi164:> sudo grub2-set-default af745df5f7934596ab03bf505e30b004-5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:41:51.296 INFO:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2020-02-05T16:41:51.298 INFO:teuthology.misc:Re-opening connections... 2020-02-05T16:41:51.298 INFO:teuthology.misc:trying to connect to ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:41:51.299 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:41:51.299 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:09.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.8 2020-02-05T16:42:09.766 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:09.766 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:09.767 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:28.198 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:28.198 DEBUG:teuthology.misc:waited 36.8995070457 2020-02-05T16:42:29.198 INFO:teuthology.misc:trying to connect to ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:42:29.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:29.199 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:29.529 INFO:teuthology.orchestra.run.smithi008:> true 2020-02-05T16:42:29.892 DEBUG:teuthology.misc:waited 38.5932981968 2020-02-05T16:42:30.893 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:30.893 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:30.894 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:33.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:33.962 DEBUG:teuthology.misc:waited 42.6635551453 2020-02-05T16:42:34.963 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:34.963 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:34.964 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:37.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:37.030 DEBUG:teuthology.misc:waited 45.7314810753 2020-02-05T16:42:38.030 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:38.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:38.031 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:40.102 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:40.102 DEBUG:teuthology.misc:waited 48.8035151958 2020-02-05T16:42:41.103 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:41.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:41.104 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:43.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:43.174 DEBUG:teuthology.misc:waited 51.8755559921 2020-02-05T16:42:44.175 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:44.175 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:44.176 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:46.246 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:46.246 DEBUG:teuthology.misc:waited 54.9475810528 2020-02-05T16:42:47.247 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:47.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:47.248 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:49.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:49.318 DEBUG:teuthology.misc:waited 58.0198411942 2020-02-05T16:42:50.319 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:50.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:50.320 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:52.390 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:52.390 DEBUG:teuthology.misc:waited 61.0916190147 2020-02-05T16:42:53.390 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:53.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:53.391 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:55.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:55.462 DEBUG:teuthology.misc:waited 64.1635479927 2020-02-05T16:42:56.462 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:56.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:56.463 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:42:58.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:42:58.534 DEBUG:teuthology.misc:waited 67.2355430126 2020-02-05T16:42:59.534 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:42:59.535 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:42:59.535 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:43:01.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2020-02-05T16:43:01.606 DEBUG:teuthology.misc:waited 70.3077189922 2020-02-05T16:43:02.607 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:43:02.607 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:43:02.607 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:44:02.609 DEBUG:teuthology.orchestra.remote:timed out 2020-02-05T16:44:02.610 DEBUG:teuthology.misc:waited 131.311249018 2020-02-05T16:44:03.610 INFO:teuthology.misc:trying to connect to ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:44:03.611 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:44:03.611 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:44:03.960 INFO:teuthology.orchestra.run.smithi164:> true 2020-02-05T16:44:04.327 DEBUG:teuthology.misc:waited 133.028829098 2020-02-05T16:44:05.329 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-02-05T16:44:05.329 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.5.0-ceph-ga11f6ef5e0ea"... 2020-02-05T16:44:05.329 INFO:teuthology.orchestra.run.smithi008:> uname -r 2020-02-05T16:44:05.368 INFO:teuthology.orchestra.run.smithi008.stdout:5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:44:05.368 DEBUG:teuthology.task.kernel:current kernel version is 5.5.0-ceph-ga11f6ef5e0ea vs 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:44:05.368 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-02-05T16:44:05.368 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2020-02-05T16:44:05.368 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.5.0-ceph-ga11f6ef5e0ea"... 2020-02-05T16:44:05.369 INFO:teuthology.orchestra.run.smithi164:> uname -r 2020-02-05T16:44:05.397 INFO:teuthology.orchestra.run.smithi164.stdout:5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:44:05.397 DEBUG:teuthology.task.kernel:current kernel version is 5.5.0-ceph-ga11f6ef5e0ea vs 5.5.0-ceph-ga11f6ef5e0ea 2020-02-05T16:44:05.397 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-02-05T16:44:06.398 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-02-05T16:44:06.398 INFO:teuthology.orchestra.run.smithi008:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-02-05T16:44:06.573 INFO:teuthology.orchestra.run.smithi008.stdout:ttyS1 2020-02-05T16:44:06.579 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2020-02-05T16:44:06.579 INFO:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-02-05T16:44:06.714 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2020-02-05T16:44:06.729 INFO:teuthology.run_tasks:Running task internal.base... 2020-02-05T16:44:06.755 INFO:teuthology.task.internal:Creating test directory... 2020-02-05T16:44:06.755 INFO:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-02-05T16:44:06.758 INFO:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-02-05T16:44:06.786 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-02-05T16:44:06.802 INFO:teuthology.run_tasks:Running task internal.archive... 2020-02-05T16:44:06.821 INFO:teuthology.task.internal:Creating archive directory... 2020-02-05T16:44:06.822 INFO:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-02-05T16:44:06.829 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-02-05T16:44:06.862 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-02-05T16:44:06.873 INFO:teuthology.task.internal:Enabling coredump saving... 2020-02-05T16:44:06.873 INFO:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-02-05T16:44:06.876 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-02-05T16:44:06.929 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-02-05T16:44:06.931 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-02-05T16:44:06.955 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-02-05T16:44:06.956 INFO:teuthology.orchestra.run.smithi008.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-02-05T16:44:06.959 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-02-05T16:44:06.969 INFO:teuthology.task.internal:Configuring sudo... 2020-02-05T16:44:06.970 INFO:teuthology.orchestra.run.smithi008:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-02-05T16:44:06.973 INFO:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-02-05T16:44:07.023 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-02-05T16:44:07.035 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-02-05T16:44:07.035 INFO:teuthology.orchestra.run.smithi008:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-02-05T16:44:07.038 INFO:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-02-05T16:44:07.064 INFO:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-02-05T16:44:07.145 INFO:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-02-05T16:44:07.209 INFO:teuthology.orchestra.run.smithi008:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-02-05T16:44:07.255 INFO:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-02-05T16:44:07.305 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-02-05T16:44:07.357 INFO:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-02-05T16:44:07.403 INFO:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-02-05T16:44:07.461 INFO:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-02-05T16:44:07.506 INFO:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-02-05T16:44:07.580 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-02-05T16:44:07.631 INFO:teuthology.orchestra.run.smithi008:> sudo service rsyslog restart 2020-02-05T16:44:07.634 INFO:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2020-02-05T16:44:07.729 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-02-05T16:44:07.730 INFO:teuthology.orchestra.run.smithi008.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-02-05T16:44:08.131 INFO:teuthology.run_tasks:Running task internal.timer... 2020-02-05T16:44:08.143 INFO:teuthology.task.internal:Starting timer... 2020-02-05T16:44:08.143 INFO:teuthology.run_tasks:Running task pcp... 2020-02-05T16:44:08.192 INFO:teuthology.run_tasks:Running task selinux... 2020-02-05T16:44:08.204 INFO:teuthology.orchestra.run.smithi008:> sudo service auditd rotate 2020-02-05T16:44:08.305 INFO:teuthology.orchestra.run.smithi008.stdout:Rotating logs: [ OK ] 2020-02-05T16:44:08.308 INFO:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2020-02-05T16:44:08.399 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: [ OK ] 2020-02-05T16:44:08.402 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-02-05T16:44:08.402 INFO:teuthology.orchestra.run.smithi008:> /usr/sbin/getenforce 2020-02-05T16:44:08.445 INFO:teuthology.orchestra.run.smithi008.stdout:Permissive 2020-02-05T16:44:08.445 INFO:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2020-02-05T16:44:08.484 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2020-02-05T16:44:08.485 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi008.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi164.front.sepia.ceph.com': 'permissive'} 2020-02-05T16:44:08.485 INFO:teuthology.orchestra.run.smithi008:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\)' 2020-02-05T16:44:08.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-02-05T16:44:08.538 INFO:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\)' 2020-02-05T16:44:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-02-05T16:44:08.591 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-02-05T16:44:08.591 INFO:teuthology.orchestra.run.smithi008:> sudo /usr/sbin/setenforce permissive 2020-02-05T16:44:08.657 INFO:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2020-02-05T16:44:08.708 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-02-05T16:44:08.719 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-02-05T16:44:08.726 INFO:teuthology.repo_utils:Fetching master from origin 2020-02-05T16:44:08.776 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-02-05T16:44:08.790 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-02-05T16:44:08.793 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi008.front.sepia.ceph.com,smithi164.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-02-05T16:44:09.115 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-02-05T16:44:10.153 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-02-05T16:44:12.709 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-02-05T16:44:12.730 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:12 +0000 (0:00:00.060) 0:00:00.060 **** 2020-02-05T16:44:12.741 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:12 +0000 (0:00:00.010) 0:00:00.070 **** 2020-02-05T16:44:15.482 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi164.front.sepia.ceph.com] 2020-02-05T16:44:15.557 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] 2020-02-05T16:44:15.619 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:15 +0000 (0:00:02.878) 0:00:02.949 **** 2020-02-05T16:44:15.659 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:15 +0000 (0:00:00.039) 0:00:02.988 **** 2020-02-05T16:44:15.992 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-02-05T16:44:15.994 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-02-05T16:44:16.030 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.370) 0:00:03.359 **** 2020-02-05T16:44:16.060 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.030) 0:00:03.390 **** 2020-02-05T16:44:16.528 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-02-05T16:44:16.530 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-02-05T16:44:16.530 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-02-05T16:44:16.566 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.505) 0:00:03.895 **** 2020-02-05T16:44:16.596 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.030) 0:00:03.925 **** 2020-02-05T16:44:16.599 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-02-05T16:44:16.600 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:16.637 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.040) 0:00:03.966 **** 2020-02-05T16:44:16.642 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:16.678 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:16 +0000 (0:00:00.041) 0:00:04.008 **** 2020-02-05T16:44:17.002 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-02-05T16:44:17.003 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:44:17.039 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:17 +0000 (0:00:00.360) 0:00:04.368 **** 2020-02-05T16:44:17.069 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:17 +0000 (0:00:00.030) 0:00:04.399 **** 2020-02-05T16:44:18.041 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-02-05T16:44:18.042 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921057.08-171708779763288/source", "state": "file", "uid": 0} changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921057.05-112176705439238/source", "state": "file", "uid": 0} 2020-02-05T16:44:18.080 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:18 +0000 (0:00:01.009) 0:00:05.409 **** 2020-02-05T16:44:18.110 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:18 +0000 (0:00:00.030) 0:00:05.439 **** 2020-02-05T16:44:18.775 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-02-05T16:44:18.776 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-02-05T16:44:18.777 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-02-05T16:44:18.816 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-02-05T16:44:18.818 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-02-05T16:44:18.930 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-02-05T16:44:18.937 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-02-05T16:44:19.108 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:19 +0000 (0:00:00.997) 0:00:06.437 **** 2020-02-05T16:44:19.143 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:19 +0000 (0:00:00.035) 0:00:06.473 **** 2020-02-05T16:44:20.005 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2020-02-05T16:44:20.005 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] 2020-02-05T16:44:20.017 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] 2020-02-05T16:44:20.126 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-02-05T16:44:20.127 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.982) 0:00:07.456 **** 2020-02-05T16:44:20.162 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.035) 0:00:07.491 **** 2020-02-05T16:44:20.591 INFO:teuthology.task.ansible.out: 2020-02-05T16:44:20.593 INFO:teuthology.task.ansible.out:TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.214569", "end": "2020-02-05 16:44:20.572400", "rc": 0, "start": "2020-02-05 16:44:20.357831", "stderr": "", "stderr_lines": [], "stdout": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "stdout_lines": ["c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba"]} 2020-02-05T16:44:20.630 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.468) 0:00:07.960 **** 2020-02-05T16:44:20.636 INFO:teuthology.task.ansible.out: 2020-02-05T16:44:20.638 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.229850", "end": "2020-02-05 16:44:20.582375", "rc": 0, "start": "2020-02-05 16:44:20.352525", "stderr": "", "stderr_lines": [], "stdout": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "stdout_lines": ["c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba"]} 2020-02-05T16:44:20.674 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.043) 0:00:08.003 **** 2020-02-05T16:44:20.677 INFO:teuthology.task.ansible.out: 2020-02-05T16:44:20.877 INFO:teuthology.task.ansible.out: 2020-02-05T16:44:20.878 INFO:teuthology.task.ansible.out:TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009611", "end": "2020-02-05 16:44:20.850971", "failed_when_result": false, "rc": 0, "start": "2020-02-05 16:44:20.841360", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2020-02-05T16:44:20.924 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.242) 0:00:08.246 **** 2020-02-05T16:44:20.943 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-02-05T16:44:20.944 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:20.980 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:20 +0000 (0:00:00.063) 0:00:08.310 **** 2020-02-05T16:44:20.986 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-02-05T16:44:20.986 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009668", "end": "2020-02-05 16:44:20.931717", "failed_when_result": false, "rc": 0, "start": "2020-02-05 16:44:20.922049", "stderr": "", "stderr_lines": [], "stdout": "c84d080fc97e482f88d240859322d5765ac90366", "stdout_lines": ["c84d080fc97e482f88d240859322d5765ac90366"]} 2020-02-05T16:44:21.024 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.044) 0:00:08.354 **** 2020-02-05T16:44:21.031 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-02-05T16:44:21.032 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.067 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.043) 0:00:08.397 **** 2020-02-05T16:44:21.072 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-02-05T16:44:21.073 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.110 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.042) 0:00:08.439 **** 2020-02-05T16:44:21.124 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.158 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.048) 0:00:08.488 **** 2020-02-05T16:44:21.162 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-02-05T16:44:21.164 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.199 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.040) 0:00:08.528 **** 2020-02-05T16:44:21.203 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-02-05T16:44:21.204 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.241 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.041) 0:00:08.570 **** 2020-02-05T16:44:21.246 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-02-05T16:44:21.246 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.282 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.041) 0:00:08.612 **** 2020-02-05T16:44:21.287 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-02-05T16:44:21.288 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.324 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.041) 0:00:08.653 **** 2020-02-05T16:44:21.330 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-02-05T16:44:21.333 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.366 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.041) 0:00:08.695 **** 2020-02-05T16:44:21.392 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-02-05T16:44:21.393 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:21.428 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.061) 0:00:08.757 **** 2020-02-05T16:44:21.557 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-02-05T16:44:21.558 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:44:21.594 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.166) 0:00:08.924 **** 2020-02-05T16:44:21.672 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:44:21.708 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:21 +0000 (0:00:00.113) 0:00:09.037 **** 2020-02-05T16:44:21.889 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-02-05T16:44:21.890 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-02-05T16:44:22.004 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2020-02-05T16:44:22.140 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-02-05T16:44:22.254 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2020-02-05T16:44:22.391 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-02-05T16:44:22.517 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2020-02-05T16:44:22.642 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-02-05T16:44:22.769 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2020-02-05T16:44:22.894 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-02-05T16:44:23.030 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2020-02-05T16:44:23.146 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-02-05T16:44:23.282 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2020-02-05T16:44:23.408 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-02-05T16:44:23.543 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2020-02-05T16:44:23.658 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-02-05T16:44:23.794 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2020-02-05T16:44:23.909 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-02-05T16:44:24.045 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2020-02-05T16:44:24.160 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-02-05T16:44:24.316 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2020-02-05T16:44:24.411 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-02-05T16:44:24.580 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2020-02-05T16:44:24.667 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-02-05T16:44:24.825 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2020-02-05T16:44:24.932 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-02-05T16:44:25.091 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2020-02-05T16:44:25.197 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-02-05T16:44:25.354 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2020-02-05T16:44:25.450 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-02-05T16:44:25.607 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2020-02-05T16:44:25.695 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-02-05T16:44:25.873 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2020-02-05T16:44:25.947 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-02-05T16:44:26.151 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2020-02-05T16:44:26.205 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-02-05T16:44:26.415 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2020-02-05T16:44:26.460 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-02-05T16:44:26.664 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2020-02-05T16:44:26.719 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-02-05T16:44:26.932 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2020-02-05T16:44:26.965 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-02-05T16:44:27.214 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1122} 2020-02-05T16:44:27.237 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-02-05T16:44:27.464 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2020-02-05T16:44:27.487 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-02-05T16:44:27.727 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2020-02-05T16:44:27.739 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-02-05T16:44:27.989 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-02-05T16:44:28.000 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2020-02-05T16:44:28.239 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-02-05T16:44:28.251 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2020-02-05T16:44:28.493 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-02-05T16:44:28.515 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2020-02-05T16:44:28.755 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-02-05T16:44:28.767 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2020-02-05T16:44:29.007 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-02-05T16:44:29.029 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2020-02-05T16:44:29.271 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-02-05T16:44:29.283 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2020-02-05T16:44:29.523 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-02-05T16:44:29.534 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2020-02-05T16:44:29.774 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-02-05T16:44:29.808 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2020-02-05T16:44:30.037 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-02-05T16:44:30.059 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2020-02-05T16:44:30.298 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-02-05T16:44:30.321 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2020-02-05T16:44:30.554 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-02-05T16:44:30.583 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2020-02-05T16:44:30.805 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-02-05T16:44:30.838 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2020-02-05T16:44:31.066 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-02-05T16:44:31.099 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2020-02-05T16:44:31.311 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-02-05T16:44:31.354 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2020-02-05T16:44:31.574 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-02-05T16:44:31.613 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2020-02-05T16:44:31.821 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-02-05T16:44:31.864 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-02-05T16:44:32.073 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-02-05T16:44:32.126 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2020-02-05T16:44:32.324 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-02-05T16:44:32.393 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2020-02-05T16:44:32.572 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-02-05T16:44:32.657 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2020-02-05T16:44:32.824 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-02-05T16:44:32.909 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2020-02-05T16:44:33.079 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-02-05T16:44:33.177 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2020-02-05T16:44:33.325 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-02-05T16:44:33.430 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2020-02-05T16:44:33.577 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-02-05T16:44:33.693 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2020-02-05T16:44:33.860 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-02-05T16:44:33.955 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-02-05T16:44:34.127 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-02-05T16:44:34.211 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2020-02-05T16:44:34.378 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-02-05T16:44:34.477 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2020-02-05T16:44:34.633 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-02-05T16:44:34.740 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2020-02-05T16:44:34.885 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-02-05T16:44:35.000 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2020-02-05T16:44:35.136 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-02-05T16:44:35.255 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2020-02-05T16:44:35.395 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-02-05T16:44:35.512 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2020-02-05T16:44:35.637 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-02-05T16:44:35.773 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2020-02-05T16:44:35.889 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-02-05T16:44:36.028 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2020-02-05T16:44:36.136 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-02-05T16:44:36.275 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2020-02-05T16:44:36.393 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-02-05T16:44:36.542 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2020-02-05T16:44:36.640 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-02-05T16:44:36.818 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2020-02-05T16:44:36.902 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-02-05T16:44:37.078 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2020-02-05T16:44:37.152 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-02-05T16:44:37.330 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2020-02-05T16:44:37.414 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-02-05T16:44:37.585 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2020-02-05T16:44:37.669 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-02-05T16:44:37.847 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2020-02-05T16:44:37.932 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-02-05T16:44:38.106 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2020-02-05T16:44:38.193 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-02-05T16:44:38.362 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2020-02-05T16:44:38.444 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-02-05T16:44:38.624 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2020-02-05T16:44:38.688 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-02-05T16:44:38.887 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2020-02-05T16:44:38.942 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-02-05T16:44:39.140 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2020-02-05T16:44:39.225 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-02-05T16:44:39.402 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2020-02-05T16:44:39.445 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-02-05T16:44:39.659 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2020-02-05T16:44:39.702 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-02-05T16:44:39.910 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2020-02-05T16:44:39.943 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-02-05T16:44:40.182 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2020-02-05T16:44:40.194 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2020-02-05T16:44:40.434 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2020-02-05T16:44:40.445 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-02-05T16:44:40.703 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-02-05T16:44:40.949 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-02-05T16:44:40.950 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2020-02-05T16:44:41.206 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-02-05T16:44:41.208 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2020-02-05T16:44:41.467 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-02-05T16:44:41.468 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2020-02-05T16:44:41.717 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-02-05T16:44:41.718 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2020-02-05T16:44:41.979 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-02-05T16:44:41.980 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2020-02-05T16:44:42.231 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-02-05T16:44:42.232 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2020-02-05T16:44:42.497 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2020-02-05T16:44:42.498 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-02-05T16:44:42.747 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2020-02-05T16:44:42.759 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-02-05T16:44:43.009 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-02-05T16:44:43.010 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2020-02-05T16:44:43.259 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-02-05T16:44:43.270 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2020-02-05T16:44:43.511 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-02-05T16:44:43.523 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2020-02-05T16:44:43.766 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-02-05T16:44:43.779 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2020-02-05T16:44:44.019 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-02-05T16:44:44.052 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2020-02-05T16:44:44.272 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-02-05T16:44:44.317 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2020-02-05T16:44:44.529 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-02-05T16:44:44.571 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2020-02-05T16:44:44.770 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-02-05T16:44:44.834 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2020-02-05T16:44:45.023 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-02-05T16:44:45.088 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2020-02-05T16:44:45.264 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-02-05T16:44:45.350 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2020-02-05T16:44:45.527 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-02-05T16:44:45.601 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2020-02-05T16:44:45.767 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-02-05T16:44:45.862 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2020-02-05T16:44:46.019 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-02-05T16:44:46.114 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2020-02-05T16:44:46.273 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-02-05T16:44:46.368 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2020-02-05T16:44:46.525 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-02-05T16:44:46.629 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2020-02-05T16:44:46.779 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-02-05T16:44:46.883 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2020-02-05T16:44:47.028 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-02-05T16:44:47.133 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2020-02-05T16:44:47.270 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-02-05T16:44:47.402 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2020-02-05T16:44:47.528 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-02-05T16:44:47.654 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2020-02-05T16:44:47.780 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-02-05T16:44:47.915 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2020-02-05T16:44:48.031 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-02-05T16:44:48.167 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2020-02-05T16:44:48.282 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-02-05T16:44:48.428 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2020-02-05T16:44:48.677 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2020-02-05T16:44:48.937 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2020-02-05T16:44:49.936 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-02-05T16:44:50.662 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2020-02-05T16:44:51.410 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "name": "xiubli", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-02-05T16:44:52.227 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "name": "xiubli", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2020-02-05T16:44:52.837 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "name": "jafaj", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-02-05T16:44:53.768 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "name": "jafaj", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2020-02-05T16:44:54.314 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "name": "jschmid", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2020-02-05T16:44:55.241 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "name": "jschmid", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2020-02-05T16:44:55.808 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "name": "kachau", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2020-02-05T16:44:55.919 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:55 +0000 (0:00:34.211) 0:00:43.249 **** 2020-02-05T16:44:55.945 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-02-05T16:44:55.980 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:55 +0000 (0:00:00.060) 0:00:43.310 **** 2020-02-05T16:44:56.039 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-02-05T16:44:56.044 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-02-05T16:44:56.045 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-02-05T16:44:56.047 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-02-05T16:44:56.048 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-02-05T16:44:56.049 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c2 2020-02-05T16:44:56.050 INFO:teuthology.task.ansible.out:4e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-02-05T16:44:56.051 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-02-05T16:44:56.052 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-02-05T16:44:56.054 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-02-05T16:44:56.055 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029cc 2020-02-05T16:44:56.055 INFO:teuthology.task.ansible.out:dc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-02-05T16:44:56.057 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-02-05T16:44:56.058 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}]}, "changed": false} 2020-02-05T16:44:56.102 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:56 +0000 (0:00:00.121) 0:00:43.432 **** 2020-02-05T16:44:56.854 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-02-05T16:44:56.855 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10106, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "name": "kachau", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10106} 2020-02-05T16:44:56.975 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:56 +0000 (0:00:00.872) 0:00:44.305 **** 2020-02-05T16:44:57.001 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-02-05T16:44:57.035 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:57 +0000 (0:00:00.059) 0:00:44.365 **** 2020-02-05T16:44:57.040 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-02-05T16:44:57.040 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com -> localhost] => {"after": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "attempts": 1, "before": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "changed": false, "remote_url_changed": false} 2020-02-05T16:44:57.086 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:57 +0000 (0:00:00.050) 0:00:44.415 **** 2020-02-05T16:44:57.094 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-02-05T16:44:57.098 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ss 2020-02-05T16:44:57.100 INFO:teuthology.task.ansible.out:h-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN 2020-02-05T16:44:57.101 INFO:teuthology.task.ansible.out:2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499ea 2020-02-05T16:44:57.102 INFO:teuthology.task.ansible.out:f767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d 2020-02-05T16:44:57.104 INFO:teuthology.task.ansible.out:85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c2 2020-02-05T16:44:57.104 INFO:teuthology.task.ansible.out:4e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d 2020-02-05T16:44:57.105 INFO:teuthology.task.ansible.out:86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868" 2020-02-05T16:44:57.107 INFO:teuthology.task.ansible.out:}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad 2020-02-05T16:44:57.109 INFO:teuthology.task.ansible.out:7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280 2020-02-05T16:44:57.110 INFO:teuthology.task.ansible.out:b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029cc 2020-02-05T16:44:57.110 INFO:teuthology.task.ansible.out:dc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkr 2020-02-05T16:44:57.112 INFO:teuthology.task.ansible.out:Q 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, {"name": "toabctl", "ovpn": 2020-02-05T16:44:57.113 INFO:teuthology.task.ansible.out: "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}]}, "changed": false} 2020-02-05T16:44:57.150 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:57 +0000 (0:00:00.063) 0:00:44.479 **** 2020-02-05T16:44:57.487 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-02-05T16:44:57.488 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-02-05T16:44:57.768 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-02-05T16:44:57.769 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com -> localhost] => {"after": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "attempts": 1, "before": "c16f05bcbff1eb4abd4a6bef7adbfb3b945429ba", "changed": false, "remote_url_changed": false} 2020-02-05T16:44:57.805 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:44:57 +0000 (0:00:00.655) 0:00:45.135 **** 2020-02-05T16:44:57.822 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-02-05T16:44:57.824 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-02-05T16:44:58.168 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-02-05T16:44:58.221 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2020-02-05T16:44:58.510 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-02-05T16:44:58.512 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-02-05T16:44:58.573 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2020-02-05T16:44:58.869 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-02-05T16:44:58.922 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2020-02-05T16:44:59.208 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-02-05T16:44:59.210 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-02-05T16:44:59.261 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2020-02-05T16:44:59.262 INFO:teuthology.task.ansible.out:lidate_certs": true} 2020-02-05T16:44:59.542 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-02-05T16:44:59.618 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2020-02-05T16:44:59.879 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-02-05T16:44:59.890 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:44:59.974 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2020-02-05T16:44:59.977 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2020-02-05T16:45:00.234 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-02-05T16:45:00.235 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-02-05T16:45:00.245 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:00.256 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:00.267 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:00.331 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2020-02-05T16:45:00.633 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-02-05T16:45:00.676 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2020-02-05T16:45:00.687 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:00.977 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-02-05T16:45:00.979 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-02-05T16:45:01.051 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2020-02-05T16:45:01.053 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2020-02-05T16:45:01.062 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:01.063 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:01.074 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:01.313 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-02-05T16:45:01.417 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2020-02-05T16:45:01.646 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-02-05T16:45:01.762 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2020-02-05T16:45:01.764 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2020-02-05T16:45:01.981 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-02-05T16:45:02.106 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2020-02-05T16:45:02.325 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-02-05T16:45:02.458 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2020-02-05T16:45:02.656 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-02-05T16:45:02.802 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2020-02-05T16:45:02.991 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-02-05T16:45:03.157 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2020-02-05T16:45:03.334 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-02-05T16:45:03.336 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-02-05T16:45:03.511 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2020-02-05T16:45:03.679 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-02-05T16:45:03.680 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-02-05T16:45:03.867 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2020-02-05T16:45:04.004 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-02-05T16:45:04.213 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2020-02-05T16:45:04.214 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2020-02-05T16:45:04.349 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-02-05T16:45:04.558 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2020-02-05T16:45:04.560 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2020-02-05T16:45:04.684 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-02-05T16:45:04.685 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-02-05T16:45:04.903 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2020-02-05T16:45:05.018 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-02-05T16:45:05.257 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2020-02-05T16:45:05.351 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-02-05T16:45:05.612 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2020-02-05T16:45:05.613 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2020-02-05T16:45:05.685 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-02-05T16:45:05.977 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2020-02-05T16:45:06.030 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-02-05T16:45:06.321 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2020-02-05T16:45:06.377 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-02-05T16:45:06.689 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2020-02-05T16:45:06.711 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-02-05T16:45:06.712 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-02-05T16:45:07.043 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-02-05T16:45:07.044 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2020-02-05T16:45:07.378 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-02-05T16:45:07.401 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2020-02-05T16:45:07.734 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-02-05T16:45:07.745 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2020-02-05T16:45:07.747 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2020-02-05T16:45:08.070 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-02-05T16:45:08.103 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2020-02-05T16:45:08.405 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-02-05T16:45:08.448 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2020-02-05T16:45:08.750 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-02-05T16:45:08.793 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2020-02-05T16:45:09.083 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-02-05T16:45:09.161 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2020-02-05T16:45:09.420 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-02-05T16:45:09.506 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2020-02-05T16:45:09.746 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-02-05T16:45:09.861 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2020-02-05T16:45:10.092 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-02-05T16:45:10.093 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-02-05T16:45:10.208 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdP 2020-02-05T16:45:10.209 INFO:teuthology.task.ansible.out:Buld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-02-05T16:45:10.431 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-02-05T16:45:10.432 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-02-05T16:45:10.560 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2020-02-05T16:45:10.761 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-02-05T16:45:10.896 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2020-02-05T16:45:11.098 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-02-05T16:45:11.254 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2020-02-05T16:45:11.256 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2020-02-05T16:45:11.436 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-02-05T16:45:11.603 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2020-02-05T16:45:11.605 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2020-02-05T16:45:11.770 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-02-05T16:45:11.772 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-02-05T16:45:11.947 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2020-02-05T16:45:12.104 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-02-05T16:45:12.106 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-02-05T16:45:12.108 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-02-05T16:45:12.311 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2020-02-05T16:45:12.436 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-02-05T16:45:12.654 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2020-02-05T16:45:12.779 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-02-05T16:45:13.009 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2020-02-05T16:45:13.011 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-02-05T16:45:13.125 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-02-05T16:45:13.373 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2020-02-05T16:45:13.374 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2020-02-05T16:45:13.376 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2020-02-05T16:45:13.457 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-02-05T16:45:13.726 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2020-02-05T16:45:13.800 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-02-05T16:45:14.074 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2020-02-05T16:45:14.127 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-02-05T16:45:14.428 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2020-02-05T16:45:14.470 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-02-05T16:45:14.771 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2020-02-05T16:45:14.804 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-02-05T16:45:15.122 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2020-02-05T16:45:15.144 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-02-05T16:45:15.146 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2020-02-05T16:45:15.465 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2020-02-05T16:45:15.507 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-02-05T16:45:15.820 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2020-02-05T16:45:15.852 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-02-05T16:45:16.168 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2020-02-05T16:45:16.173 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:16.190 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-02-05T16:45:16.192 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-02-05T16:45:16.525 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2020-02-05T16:45:16.527 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-02-05T16:45:16.861 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-02-05T16:45:16.872 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2020-02-05T16:45:17.203 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-02-05T16:45:17.225 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2020-02-05T16:45:17.556 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-02-05T16:45:17.588 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2020-02-05T16:45:17.590 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2020-02-05T16:45:17.888 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-02-05T16:45:17.940 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2020-02-05T16:45:18.223 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-02-05T16:45:18.297 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2020-02-05T16:45:18.569 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-02-05T16:45:18.659 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2020-02-05T16:45:18.909 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-02-05T16:45:19.005 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2020-02-05T16:45:19.244 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-02-05T16:45:19.379 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2020-02-05T16:45:19.569 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-02-05T16:45:19.728 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2020-02-05T16:45:19.914 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-02-05T16:45:20.085 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2020-02-05T16:45:20.248 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-02-05T16:45:20.424 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2020-02-05T16:45:20.578 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-02-05T16:45:20.779 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2020-02-05T16:45:20.917 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-02-05T16:45:21.118 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2020-02-05T16:45:21.253 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-02-05T16:45:21.472 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2020-02-05T16:45:21.586 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-02-05T16:45:21.825 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2020-02-05T16:45:21.934 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-02-05T16:45:22.175 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2020-02-05T16:45:22.262 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-02-05T16:45:22.526 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2020-02-05T16:45:22.600 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-02-05T16:45:22.880 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2020-02-05T16:45:22.934 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-02-05T16:45:23.226 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2020-02-05T16:45:23.268 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-02-05T16:45:23.589 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2020-02-05T16:45:23.601 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-02-05T16:45:23.944 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2020-02-05T16:45:23.945 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-02-05T16:45:24.280 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-02-05T16:45:24.312 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2020-02-05T16:45:24.614 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-02-05T16:45:24.667 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2020-02-05T16:45:24.949 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-02-05T16:45:24.950 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-02-05T16:45:25.012 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2020-02-05T16:45:25.282 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-02-05T16:45:25.284 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-02-05T16:45:25.368 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2020-02-05T16:45:25.628 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-02-05T16:45:25.732 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2020-02-05T16:45:25.971 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-02-05T16:45:26.087 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2020-02-05T16:45:26.310 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-02-05T16:45:26.426 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2020-02-05T16:45:26.645 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-02-05T16:45:26.780 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2020-02-05T16:45:26.781 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2020-02-05T16:45:26.977 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-02-05T16:45:27.123 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2020-02-05T16:45:27.124 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2020-02-05T16:45:27.312 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-02-05T16:45:27.467 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2020-02-05T16:45:27.644 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-02-05T16:45:27.823 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2020-02-05T16:45:27.982 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-02-05T16:45:28.227 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2020-02-05T16:45:28.311 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-02-05T16:45:28.570 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2020-02-05T16:45:28.644 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-02-05T16:45:28.933 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2020-02-05T16:45:28.987 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-02-05T16:45:29.266 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2020-02-05T16:45:29.320 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2020-02-05T16:45:29.619 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2020-02-05T16:45:29.662 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-02-05T16:45:29.964 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2020-02-05T16:45:29.996 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-02-05T16:45:30.322 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2020-02-05T16:45:30.333 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-02-05T16:45:30.334 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-02-05T16:45:30.666 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2020-02-05T16:45:30.681 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-02-05T16:45:31.011 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-02-05T16:45:31.012 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2020-02-05T16:45:31.355 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-02-05T16:45:31.366 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validat 2020-02-05T16:45:31.368 INFO:teuthology.task.ansible.out:e_certs": true} 2020-02-05T16:45:31.690 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-02-05T16:45:31.702 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2020-02-05T16:45:32.030 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-02-05T16:45:32.074 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2020-02-05T16:45:32.368 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-02-05T16:45:32.421 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2020-02-05T16:45:32.422 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2020-02-05T16:45:32.711 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2020-02-05T16:45:32.775 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2020-02-05T16:45:33.039 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-02-05T16:45:33.139 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:33 +0000 (0:00:35.333) 0:01:20.469 **** 2020-02-05T16:45:33.145 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2020-02-05T16:45:33.171 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:33.173 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.183 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.194 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.195 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.206 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.217 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.228 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.228 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.466 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-02-05T16:45:33.468 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2020-02-05T16:45:33.581 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:33.583 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-02-05T16:45:33.593 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:33.823 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2020-02-05T16:45:33.825 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2020-02-05T16:45:33.919 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:33.921 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-02-05T16:45:34.169 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzVa77kg0qGGE3IpMZgSYJ+UwQiDEOg3Lji470V/sQ/R7g6/v7FZpaYqGn1wG3XeUp2gGACxNKLQfGV9EGU66NvfQYqn9A8dXRpTpZuGWN8EcKK98YrSAq0s0WzeiEzVT86RHzVHIqL6/Zn4B1p9sIeY9Hb+bfKOdtYkRBDoFELtiMRzHPFaiIJFnXl+EhH8wceYqXVNNvCuOUBMnOiDYLDtujvYdPt+ff7rvXuOzbwcAKoINFM1C4FtoUEYriKlsxJ2rr7AblfDm/WIrCZcqRBU4JcudtvZOtHhvN4uL6WgzuzuhIQFyePAeIHFUn6AiwKC/MApq7z1Qj+N3pN/Nz xiubli@fedora1", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2020-02-05T16:45:34.262 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:34.264 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-02-05T16:45:34.515 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2020-02-05T16:45:34.589 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-02-05T16:45:34.600 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.612 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.623 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.623 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.635 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.637 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.647 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.648 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.659 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.660 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.670 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d2 2020-02-05T16:45:34.672 INFO:teuthology.task.ansible.out:5807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.682 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.683 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.694 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.696 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.706 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.708 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.717 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.719 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.730 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.732 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.742 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.744 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.754 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.755 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.765 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.769 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.770 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.781 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.783 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.793 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.794 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.804 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.806 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:34.817 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.826 INFO:teuthology.task.ansible.out: skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.838 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.849 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.862 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.864 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.875 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.886 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.898 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.909 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.921 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.932 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.943 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.954 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.958 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.969 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.981 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:34.994 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.003 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.014 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.026 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.029 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.038 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.048 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.059 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.060 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.072 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.084 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.094 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.104 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.115 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.126 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.137 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.148 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.170 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.181 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.192 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.193 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.204 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.215 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** changed: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHZ739LVtNJVfcdnwYl7dwvPpC96dHxSn7UlRIzvSbBid0B+pMxE9OZv0UlyvDe737oPZthZYrAaeicKHO9p5aQwbhhIGImvm85sd1YxiQC6h6LxebJ6iFbtzMaEQ9SJUJFvUtItc1nQ8aPpY0CkDFWbzNn6ib8AKGTfcA+wsg08eBUJNSKexebHQCLC2c3HlDmRnjSTitl72h5pSgq7vcynhCJDlkJg49YesmNhWm9QNjwih3ITRzzUC5GaQ4uODlOETRmf84J9FDX8fMaRdPl4cZwhfI8q+0T/74Qrg1J82Ska+O5DhhbzFEqBkchYpbgetTPkUNHRbfL/GlK+7/8BKiUW+Fo7NkTDVpVYBaQOQSZEWWiPq04NMLORJ1HiHo7HsT7WI/HcVbt19YWZf7VZgB06p8LAaZsZxLBO/eqtjebOLFmMoEa4UaAepBXKo8O2hddaHjb4WVrEN+bTRUexuyTlZVbeukUky3gI2MfQirP4stmci02OKuvUuP1sM= karan@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2020-02-05T16:45:35.338 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:35.340 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.340 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.342 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.343 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.345 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.345 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:35.347 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.347 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.348 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.350 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.350 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.391 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:35 +0000 (0:00:02.251) 0:01:22.720 **** 2020-02-05T16:45:35.400 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.400 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.414 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.415 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.426 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.428 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.440 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.442 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.452 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.463 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.465 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.476 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.480 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.481 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.501 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.501 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.503 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.537 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:35 +0000 (0:00:00.146) 0:01:22.867 **** 2020-02-05T16:45:35.565 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.600 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:35 +0000 (0:00:00.063) 0:01:22.930 **** 2020-02-05T16:45:35.845 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:35.846 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2020-02-05T16:45:35.856 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:35.878 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:35.879 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-02-05T16:45:36.129 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-02-05T16:45:36.205 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:36.207 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2020-02-05T16:45:36.362 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:36.364 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-02-05T16:45:36.555 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2020-02-05T16:45:36.609 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:36.611 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-02-05T16:45:36.861 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-02-05T16:45:36.903 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:36.905 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2020-02-05T16:45:36.915 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.927 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.939 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.948 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.948 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.959 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.971 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.982 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:36.993 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.004 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.015 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.017 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.028 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.039 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.050 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.062 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.063 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.074 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.085 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.096 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.097 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.099 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi164.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-02-05T16:45:37.109 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-02-05T16:45:37.111 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.121 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.132 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.132 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.143 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.154 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.155 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.166 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.177 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.189 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.190 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.201 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.212 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.223 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.234 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.245 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.246 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.257 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.268 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.279 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.290 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.291 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.302 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.314 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.315 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.326 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.338 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.349 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.350 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:37.350 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-02-05T16:45:37.351 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.362 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.373 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.374 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.385 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.397 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.398 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.408 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.420 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.432 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.433 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.444 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.455 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.466 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.467 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.478 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.490 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.490 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.502 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.523 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.524 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.535 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.548 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.550 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.560 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.570 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.582 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.582 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.593 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:37.594 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.606 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.607 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.618 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.629 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.640 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.641 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.666 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.673 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.673 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.684 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.685 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.696 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.707 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.718 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 RYbu6kdXmsBtrtAkph0lIg b4a266ab604d320ab175070a427f4ce486f65cde25c5ac248e8316e3fd2f2435"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.834 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.835 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:37.836 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-02-05T16:45:37.837 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.848 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.914 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:37 +0000 (0:00:02.313) 0:01:25.243 **** 2020-02-05T16:45:37.940 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-02-05T16:45:37.941 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:37.976 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:37 +0000 (0:00:00.061) 0:01:25.305 **** 2020-02-05T16:45:38.088 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:38.088 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-02-05T16:45:38.266 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2020-02-05T16:45:38.340 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-02-05T16:45:38.517 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2020-02-05T16:45:38.584 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-02-05T16:45:38.773 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2020-02-05T16:45:38.827 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-02-05T16:45:39.026 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2020-02-05T16:45:39.079 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-02-05T16:45:39.266 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2020-02-05T16:45:39.340 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-02-05T16:45:39.516 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2020-02-05T16:45:39.590 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-02-05T16:45:39.779 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2020-02-05T16:45:39.831 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-02-05T16:45:40.018 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2020-02-05T16:45:40.081 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-02-05T16:45:40.268 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2020-02-05T16:45:40.321 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-02-05T16:45:40.529 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2020-02-05T16:45:40.571 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-02-05T16:45:40.780 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2020-02-05T16:45:40.812 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-02-05T16:45:41.037 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2020-02-05T16:45:41.048 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-02-05T16:45:41.289 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2020-02-05T16:45:41.300 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-02-05T16:45:41.542 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2020-02-05T16:45:41.543 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-02-05T16:45:41.782 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-02-05T16:45:41.798 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2020-02-05T16:45:42.028 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-02-05T16:45:42.040 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2020-02-05T16:45:42.270 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-02-05T16:45:42.292 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2020-02-05T16:45:42.515 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-02-05T16:45:42.558 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2020-02-05T16:45:42.769 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-02-05T16:45:42.807 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2020-02-05T16:45:43.015 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-02-05T16:45:43.058 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2020-02-05T16:45:43.265 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-02-05T16:45:43.307 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2020-02-05T16:45:43.515 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-02-05T16:45:43.568 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2020-02-05T16:45:43.747 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-02-05T16:45:43.821 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2020-02-05T16:45:44.069 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2020-02-05T16:45:44.330 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2020-02-05T16:45:44.581 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2020-02-05T16:45:44.839 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2020-02-05T16:45:45.088 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2020-02-05T16:45:45.254 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adeza", "name": "adeza", "remove": false, "state": "absent"} 2020-02-05T16:45:45.337 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2020-02-05T16:45:45.506 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-02-05T16:45:45.600 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2020-02-05T16:45:45.757 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-02-05T16:45:45.851 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2020-02-05T16:45:45.998 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-02-05T16:45:46.104 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2020-02-05T16:45:46.105 INFO:teuthology.task.ansible.out: 2020-02-05T16:45:46.239 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-02-05T16:45:46.354 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2020-02-05T16:45:46.490 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-02-05T16:45:46.728 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-02-05T16:45:46.976 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-02-05T16:45:47.225 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-02-05T16:45:47.277 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:47 +0000 (0:00:09.301) 0:01:34.607 **** 2020-02-05T16:45:47.896 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adeza", "name": "adeza", "remove": false, "state": "absent"} 2020-02-05T16:45:47.960 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-02-05T16:45:47.961 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "58dc04291ad37a0cd250b172c0413bd36a07c8f4", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "d5ad484ae002dd9ae71561ae66889ce9", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921147.3-115894274051408/source", "state": "file", "uid": 0} 2020-02-05T16:45:47.998 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:47 +0000 (0:00:00.720) 0:01:35.327 **** 2020-02-05T16:45:48.024 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-02-05T16:45:48.025 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => { "msg": "Host smithi164.front.sepia.ceph.com is running RedHat 8.0 (Ootpa)" } 2020-02-05T16:45:48.063 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.065) 0:01:35.393 **** 2020-02-05T16:45:48.100 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-02-05T16:45:48.102 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2020-02-05T16:45:48.138 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.075) 0:01:35.468 **** 2020-02-05T16:45:48.153 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:48.154 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2020-02-05T16:45:48.165 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-02-05T16:45:48.167 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.202 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.063) 0:01:35.532 **** 2020-02-05T16:45:48.406 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:48.406 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2020-02-05T16:45:48.522 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-02-05T16:45:48.524 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.086586", "end": "2020-02-05 16:45:48.499583", "rc": 0, "start": "2020-02-05 16:45:48.412997", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-02-05T16:45:48.560 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.357) 0:01:35.890 **** 2020-02-05T16:45:48.586 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-02-05T16:45:48.588 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.626 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.065) 0:01:35.955 **** 2020-02-05T16:45:48.651 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-02-05T16:45:48.652 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.686 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.060) 0:01:36.016 **** 2020-02-05T16:45:48.691 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:48.692 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2020-02-05T16:45:48.713 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-02-05T16:45:48.715 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.749 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.062) 0:01:36.079 **** 2020-02-05T16:45:48.774 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-02-05T16:45:48.775 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.809 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.059) 0:01:36.138 **** 2020-02-05T16:45:48.834 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-02-05T16:45:48.835 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:48.871 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.062) 0:01:36.201 **** 2020-02-05T16:45:48.908 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-02-05T16:45:48.910 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} TASK [users : Remove revoked users] ******************************************** ok: [smithi008.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2020-02-05T16:45:48.945 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:48 +0000 (0:00:00.073) 0:01:36.275 **** 2020-02-05T16:45:48.992 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-02-05T16:45:48.993 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-02-05T16:45:49.029 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:49 +0000 (0:00:00.083) 0:01:36.358 **** 2020-02-05T16:45:49.075 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-02-05T16:45:49.076 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2020-02-05T16:45:49.111 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:49 +0000 (0:00:00.082) 0:01:36.441 **** 2020-02-05T16:45:49.158 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2020-02-05T16:45:49.159 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2020-02-05T16:45:49.421 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2020-02-05T16:45:49.663 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2020-02-05T16:45:49.913 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2020-02-05T16:45:49.987 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:49 +0000 (0:00:00.875) 0:01:37.316 **** 2020-02-05T16:45:50.688 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-02-05T16:45:50.690 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "58dc04291ad37a0cd250b172c0413bd36a07c8f4", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "d5ad484ae002dd9ae71561ae66889ce9", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921150.01-175925414874359/source", "state": "file", "uid": 0} 2020-02-05T16:45:50.723 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:50 +0000 (0:00:00.735) 0:01:38.052 **** 2020-02-05T16:45:50.748 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-02-05T16:45:50.749 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => { "msg": "Host smithi008.front.sepia.ceph.com is running RedHat 8.0 (Ootpa)" } 2020-02-05T16:45:50.783 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:50 +0000 (0:00:00.060) 0:01:38.113 **** 2020-02-05T16:45:50.818 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-02-05T16:45:50.820 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"], "rhsm_repos": ["rhel-8-for-x86_64-baseos-rpms", "rhel-8-for-x86_64-appstream-rpms"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/redhat_8.yml"} 2020-02-05T16:45:50.853 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:50 +0000 (0:00:00.070) 0:01:38.183 **** 2020-02-05T16:45:50.879 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-02-05T16:45:50.880 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:50.915 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:50 +0000 (0:00:00.061) 0:01:38.244 **** 2020-02-05T16:45:51.254 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-02-05T16:45:51.254 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.096459", "end": "2020-02-05 16:45:51.229821", "rc": 0, "start": "2020-02-05 16:45:51.133362", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-02-05T16:45:51.290 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.374) 0:01:38.620 **** 2020-02-05T16:45:51.316 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-02-05T16:45:51.318 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:51.350 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.060) 0:01:38.680 **** 2020-02-05T16:45:51.376 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-02-05T16:45:51.377 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:51.410 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.060) 0:01:38.740 **** 2020-02-05T16:45:51.435 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-02-05T16:45:51.437 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:51.472 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.061) 0:01:38.801 **** 2020-02-05T16:45:51.496 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-02-05T16:45:51.498 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:51.532 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.060) 0:01:38.862 **** 2020-02-05T16:45:51.558 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-02-05T16:45:51.559 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:45:51.597 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.064) 0:01:38.926 **** 2020-02-05T16:45:51.632 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-02-05T16:45:51.634 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"entitlements_path": "/etc/ansible/secrets/entitlements.yml"}, "changed": false} 2020-02-05T16:45:51.669 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.071) 0:01:38.998 **** 2020-02-05T16:45:51.715 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-02-05T16:45:51.716 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ok: [smithi008.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-02-05T16:45:51.751 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.081) 0:01:39.080 **** 2020-02-05T16:45:51.789 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"have_entitlements": true}, "changed": false} 2020-02-05T16:45:51.827 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:51 +0000 (0:00:00.076) 0:01:39.156 **** 2020-02-05T16:45:52.736 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-02-05T16:45:52.738 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1580921152.542347-lz6epdfk/katello-ca-consumer-latest.noarchsvimrckb.rpm"]} 2020-02-05T16:45:52.773 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:52 +0000 (0:00:00.946) 0:01:40.102 **** 2020-02-05T16:45:53.822 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-02-05T16:45:53.824 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2020-02-05T16:45:53.860 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:53 +0000 (0:00:01.086) 0:01:41.189 **** 2020-02-05T16:45:53.896 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-02-05T16:45:53.898 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2020-02-05T16:45:53.938 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:53 +0000 (0:00:00.077) 0:01:41.267 **** 2020-02-05T16:45:54.635 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-02-05T16:45:54.637 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed /root/.ansible/tmp/ansible-moduletmp-1580921154.4272187-__23t9_j/katello-ca-consumer-latest.noarchg4sz2_bc.rpm"]} 2020-02-05T16:45:54.673 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:54 +0000 (0:00:00.735) 0:01:42.003 **** 2020-02-05T16:45:55.783 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-02-05T16:45:55.785 INFO:teuthology.task.ansible.out:fatal: [smithi008.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} ...ignoring 2020-02-05T16:45:55.820 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:55 +0000 (0:00:01.146) 0:01:43.150 **** 2020-02-05T16:45:55.859 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-02-05T16:45:55.861 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": false}, "changed": false} 2020-02-05T16:45:55.898 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:45:55 +0000 (0:00:00.077) 0:01:43.227 **** 2020-02-05T16:46:02.459 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-02-05T16:46:02.461 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:46:02.500 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:02 +0000 (0:00:06.601) 0:01:49.829 **** 2020-02-05T16:46:02.536 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-02-05T16:46:02.537 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2020-02-05T16:46:02.572 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:02 +0000 (0:00:00.072) 0:01:49.901 **** 2020-02-05T16:46:03.584 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-02-05T16:46:03.584 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.768172", "end": "2020-02-05 16:46:03.552508", "failed_when_result": false, "rc": 0, "start": "2020-02-05 16:46:02.784336", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7Server\n8\n8.0\n8.1", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7Server", "8", "8.0", "8.1"]} 2020-02-05T16:46:03.621 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:03 +0000 (0:00:01.049) 0:01:50.951 **** 2020-02-05T16:46:03.657 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-02-05T16:46:03.659 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:03.695 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:03 +0000 (0:00:00.073) 0:01:51.025 **** 2020-02-05T16:46:03.733 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-02-05T16:46:03.734 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:03.775 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:03 +0000 (0:00:00.079) 0:01:51.104 **** 2020-02-05T16:46:03.800 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-02-05T16:46:03.802 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-02-05T16:46:03.838 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:03 +0000 (0:00:00.062) 0:01:51.167 **** 2020-02-05T16:46:03.868 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:03.909 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:03 +0000 (0:00:00.071) 0:01:51.239 **** 2020-02-05T16:46:04.888 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-02-05T16:46:04.889 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"attempts": 1, "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:46:04.926 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:04 +0000 (0:00:01.016) 0:01:52.256 **** 2020-02-05T16:46:04.967 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-02-05T16:46:04.968 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"rhsm_registered": true}, "changed": false} 2020-02-05T16:46:05.008 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:05 +0000 (0:00:00.081) 0:01:52.337 **** 2020-02-05T16:46:06.041 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-02-05T16:46:06.042 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager release --list | grep -E '[0-9]'", "delta": "0:00:00.778228", "end": "2020-02-05 16:46:06.007682", "failed_when_result": false, "rc": 0, "start": "2020-02-05 16:46:05.229454", "stderr": "", "stderr_lines": [], "stdout": "7.4\n7.5\n7.6\n7.7\n7Server\n8\n8.0\n8.1", "stdout_lines": ["7.4", "7.5", "7.6", "7.7", "7Server", "8", "8.0", "8.1"]} 2020-02-05T16:46:06.080 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:06 +0000 (0:00:01.072) 0:01:53.410 **** 2020-02-05T16:46:06.107 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-02-05T16:46:06.108 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:06.142 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:06 +0000 (0:00:00.062) 0:01:53.472 **** 2020-02-05T16:46:06.178 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:06.209 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:06 +0000 (0:00:00.066) 0:01:53.539 **** 2020-02-05T16:46:06.238 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi008.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-02-05T16:46:06.276 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:06 +0000 (0:00:00.066) 0:01:53.605 **** 2020-02-05T16:46:06.313 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-02-05T16:46:06.314 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:06.353 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:06 +0000 (0:00:00.077) 0:01:53.682 **** 2020-02-05T16:46:14.583 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-02-05T16:46:14.583 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:10.426781", "end": "2020-02-05 16:46:14.554471", "rc": 0, "start": "2020-02-05 16:46:04.127690", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2020-02-05T16:46:14.630 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:14 +0000 (0:00:08.272) 0:02:01.955 **** 2020-02-05T16:46:14.662 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-02-05T16:46:14.663 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2020-02-05T16:46:14.699 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:14 +0000 (0:00:00.073) 0:02:02.028 **** 2020-02-05T16:46:14.735 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-02-05T16:46:14.736 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:14.773 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:14 +0000 (0:00:00.074) 0:02:02.102 **** 2020-02-05T16:46:14.809 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-02-05T16:46:14.811 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2020-02-05T16:46:14.851 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:14 +0000 (0:00:00.078) 0:02:02.181 **** 2020-02-05T16:46:14.888 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-02-05T16:46:14.889 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2020-02-05T16:46:14.925 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:14 +0000 (0:00:00.074) 0:02:02.255 **** 2020-02-05T16:46:17.658 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-02-05T16:46:17.659 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "cmd": "subscription-manager repos --list | grep -B4 'Enabled:.*1' | grep 'Repo ID:' | sed -e 's/Repo ID:\\s*\\(.*\\)/\\1/' | sort", "delta": "0:00:11.054506", "end": "2020-02-05 16:46:17.635188", "rc": 0, "start": "2020-02-05 16:46:06.580682", "stderr": "", "stderr_lines": [], "stdout": "rhel-8-for-x86_64-appstream-rpms\nrhel-8-for-x86_64-baseos-rpms", "stdout_lines": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]} 2020-02-05T16:46:17.695 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:17 +0000 (0:00:02.769) 0:02:05.025 **** 2020-02-05T16:46:17.734 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"repo_list": ["rhel-8-for-x86_64-appstream-rpms", "rhel-8-for-x86_64-baseos-rpms"]}, "changed": false} 2020-02-05T16:46:17.770 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:17 +0000 (0:00:00.074) 0:02:05.099 **** 2020-02-05T16:46:17.809 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:17.845 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:17 +0000 (0:00:00.075) 0:02:05.174 **** 2020-02-05T16:46:17.881 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2020-02-05T16:46:17.883 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": false}, "changed": false} 2020-02-05T16:46:17.918 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:17 +0000 (0:00:00.073) 0:02:05.247 **** 2020-02-05T16:46:17.955 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2020-02-05T16:46:17.957 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"replace_repos": true}, "changed": false} 2020-02-05T16:46:17.993 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:17 +0000 (0:00:00.075) 0:02:05.323 **** 2020-02-05T16:46:21.656 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2020-02-05T16:46:21.658 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:46:21.693 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:21 +0000 (0:00:03.699) 0:02:09.023 **** 2020-02-05T16:46:24.278 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:46:24.325 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:24 +0000 (0:00:02.631) 0:02:11.654 **** 2020-02-05T16:46:27.256 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-02-05T16:46:27.257 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-baseos-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-baseos-rpms"], "delta": "0:00:05.300937", "end": "2020-02-05 16:46:27.223784", "item": "rhel-8-for-x86_64-baseos-rpms", "rc": 0, "start": "2020-02-05 16:46:21.922847", "stderr": "", "stderr_lines": [], "stdout": "Internal Server Error): Required lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: update\nConflicts with tasks:\n https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/48bec96a d42c 4274 87ee bc79aab90031\n\nRepository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "stdout_lines": ["Internal Server Error): Required lock is already taken by other running tasks.", "Please inspect their state, fix their errors and resume them.", "", "Required lock: update", "Conflicts with tasks:", " https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/48bec96a d42c 4274 87ee bc79aab90031", "", "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system."]} 2020-02-05T16:46:29.724 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-baseos-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-baseos-rpms"], "delta": "0:00:05.133725", "end": "2020-02-05 16:46:29.694832", "item": "rhel-8-for-x86_64-baseos-rpms", "rc": 0, "start": "2020-02-05 16:46:24.561107", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-baseos-rpms' is enabled for this system."]} 2020-02-05T16:46:33.940 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-appstream-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:06.459242", "end": "2020-02-05 16:46:33.915525", "item": "rhel-8-for-x86_64-appstream-rpms", "rc": 0, "start": "2020-02-05 16:46:27.456283", "stderr": "", "stderr_lines": [], "stdout": "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2020-02-05T16:46:33.979 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:33 +0000 (0:00:09.653) 0:02:21.308 **** 2020-02-05T16:46:34.476 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-02-05T16:46:34.478 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2020-02-05T16:46:34.512 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:34 +0000 (0:00:00.533) 0:02:21.842 **** 2020-02-05T16:46:34.756 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-02-05T16:46:34.758 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:46:34.793 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:34 +0000 (0:00:00.280) 0:02:22.122 **** 2020-02-05T16:46:35.526 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-02-05T16:46:35.527 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "md5sum": "a8fc4394319cef97573dd9cbb76e901f", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921194.83-241697045609226/source", "state": "file", "uid": 0} 2020-02-05T16:46:35.695 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-02-05T16:46:35.697 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=rhel-8-for-x86_64-appstream-rpms) => {"ansible_loop_var": "item", "changed": true, "cmd": ["subscription-manager", "repos", "--enable", "rhel-8-for-x86_64-appstream-rpms"], "delta": "0:00:05.735609", "end": "2020-02-05 16:46:35.667290", "item": "rhel-8-for-x86_64-appstream-rpms", "rc": 0, "start": "2020-02-05 16:46:29.931681", "stderr": "", "stderr_lines": [], "stdout": "Internal Server Error): Required lock is already taken by other running tasks.\nPlease inspect their state, fix their errors and resume them.\n\nRequired lock: update\nConflicts with tasks:\n https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/6dc0ee4c f979 4a4e be73 7bbdfbeb4e81\n\nRepository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system.", "stdout_lines": ["Internal Server Error): Required lock is already taken by other running tasks.", "Please inspect their state, fix their errors and resume them.", "", "Required lock: update", "Conflicts with tasks:", " https://satellite.front.sepia.ceph.com/foreman_tasks/tasks/6dc0ee4c f979 4a4e be73 7bbdfbeb4e81", "", "Repository 'rhel-8-for-x86_64-appstream-rpms' is enabled for this system."]} 2020-02-05T16:46:35.758 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:35 +0000 (0:00:00.964) 0:02:23.087 **** 2020-02-05T16:46:36.016 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-02-05T16:46:36.017 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "path": "/etc/yum.repos.d/cd.repo", "state": "absent"} 2020-02-05T16:46:36.054 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:36 +0000 (0:00:00.296) 0:02:23.383 **** 2020-02-05T16:46:36.184 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-02-05T16:46:36.185 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "md5sum": "28027cc3431fdd12cf0ab83482917668", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921195.53-3179398190108/source", "state": "file", "uid": 0} 2020-02-05T16:46:36.224 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:36 +0000 (0:00:00.169) 0:02:23.553 **** 2020-02-05T16:46:36.302 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2020-02-05T16:46:36.304 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:46:36.339 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:36 +0000 (0:00:00.115) 0:02:23.669 **** 2020-02-05T16:46:36.962 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-02-05T16:46:36.963 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "c0b17108eae19ce4e019023f4d971436", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921196.26-213122131224819/source", "state": "file", "uid": 0} 2020-02-05T16:46:37.089 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-02-05T16:46:37.089 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "md5sum": "a8fc4394319cef97573dd9cbb76e901f", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921196.39-114919513148977/source", "state": "file", "uid": 0} 2020-02-05T16:46:37.675 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-02-05T16:46:37.676 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "8ea2607820b97348115878813225ac00", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921196.98-58584720882423/source", "state": "file", "uid": 0} 2020-02-05T16:46:37.720 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:37 +0000 (0:00:01.380) 0:02:25.050 **** 2020-02-05T16:46:37.799 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2020-02-05T16:46:37.800 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "md5sum": "28027cc3431fdd12cf0ab83482917668", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921197.1-58513688939006/source", "state": "file", "uid": 0} 2020-02-05T16:46:37.843 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:37 +0000 (0:00:00.122) 0:02:25.173 **** 2020-02-05T16:46:38.601 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-02-05T16:46:38.602 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "c0b17108eae19ce4e019023f4d971436", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921197.88-269265203502575/source", "state": "file", "uid": 0} 2020-02-05T16:46:39.155 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-02-05T16:46:39.157 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum module rather than running 'yum'. If you need to use command because yum is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:01.180920", "end": "2020-02-05 16:46:39.124109", "rc": 0, "start": "2020-02-05 16:46:37.943189", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\n195 files removed", "stdout_lines": ["Updating Subscription Management repositories.", "195 files removed"]} 2020-02-05T16:46:39.193 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:01.349) 0:02:26.522 **** 2020-02-05T16:46:39.218 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-02-05T16:46:39.220 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.254 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.060) 0:02:26.583 **** 2020-02-05T16:46:39.278 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-02-05T16:46:39.280 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.317 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.063) 0:02:26.647 **** 2020-02-05T16:46:39.324 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-02-05T16:46:39.325 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "8ea2607820b97348115878813225ac00", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921198.61-194025752200929/source", "state": "file", "uid": 0} 2020-02-05T16:46:39.366 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.048) 0:02:26.696 **** 2020-02-05T16:46:39.371 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-02-05T16:46:39.372 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.413 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.047) 0:02:26.743 **** 2020-02-05T16:46:39.440 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-02-05T16:46:39.442 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.475 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.061) 0:02:26.805 **** 2020-02-05T16:46:39.500 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-02-05T16:46:39.502 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.539 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.064) 0:02:26.869 **** 2020-02-05T16:46:39.565 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-02-05T16:46:39.567 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.602 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.062) 0:02:26.931 **** 2020-02-05T16:46:39.627 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-02-05T16:46:39.629 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.664 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.062) 0:02:26.993 **** 2020-02-05T16:46:39.689 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-02-05T16:46:39.691 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:39.725 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:39 +0000 (0:00:00.061) 0:02:27.055 **** 2020-02-05T16:46:40.531 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-02-05T16:46:40.532 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.571413", "end": "2020-02-05 16:46:40.500677", "rc": 0, "start": "2020-02-05 16:46:39.929264", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:46:40.570 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:40 +0000 (0:00:00.844) 0:02:27.900 **** 2020-02-05T16:46:40.721 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-02-05T16:46:40.722 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:01.106853", "end": "2020-02-05 16:46:40.695975", "rc": 0, "start": "2020-02-05 16:46:39.589122", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\n195 files removed", "stdout_lines": ["Updating Subscription Management repositories.", "195 files removed"]} 2020-02-05T16:46:40.760 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:40 +0000 (0:00:00.189) 0:02:28.089 **** 2020-02-05T16:46:40.787 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-02-05T16:46:40.788 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:40.829 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:40 +0000 (0:00:00.068) 0:02:28.158 **** 2020-02-05T16:46:40.856 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-02-05T16:46:40.858 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:40.900 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:40 +0000 (0:00:00.071) 0:02:28.230 **** 2020-02-05T16:46:40.927 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-02-05T16:46:40.928 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:40.963 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:40 +0000 (0:00:00.062) 0:02:28.292 **** 2020-02-05T16:46:40.990 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:41.027 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:41 +0000 (0:00:00.064) 0:02:28.356 **** 2020-02-05T16:46:41.053 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-02-05T16:46:41.054 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:41.095 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:41 +0000 (0:00:00.068) 0:02:28.425 **** 2020-02-05T16:46:41.121 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-02-05T16:46:41.123 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:41.159 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:41 +0000 (0:00:00.064) 0:02:28.489 **** 2020-02-05T16:46:41.185 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-02-05T16:46:41.187 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:41.223 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:41 +0000 (0:00:00.063) 0:02:28.552 **** 2020-02-05T16:46:41.249 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-02-05T16:46:41.251 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:41.290 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:41 +0000 (0:00:00.066) 0:02:28.619 **** 2020-02-05T16:46:42.530 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-02-05T16:46:42.532 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.991961", "end": "2020-02-05 16:46:42.500782", "rc": 0, "start": "2020-02-05 16:46:41.508821", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:46:42.568 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:42 +0000 (0:00:01.276) 0:02:29.895 **** 2020-02-05T16:46:56.146 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-02-05T16:46:56.147 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: krb5-workstation", "Installed: krb5-workstation-1.17-9.el8.x86_64", "Installed: krb5-libs-1.17-9.el8.x86_64", "Installed: libkadm5-1.17-9.el8.x86_64", "Removed: krb5-libs-1.16.1-22.el8.x86_64", "Removed: krb5-workstation-1.16.1-22.el8.x86_64", "Removed: libkadm5-1.16.1-22.el8.x86_64"]} 2020-02-05T16:46:56.185 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:56 +0000 (0:00:13.618) 0:02:43.514 **** 2020-02-05T16:46:56.221 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-02-05T16:46:56.223 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:56.257 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:56 +0000 (0:00:00.072) 0:02:43.587 **** 2020-02-05T16:46:56.303 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-02-05T16:46:56.305 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:56.348 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:56 +0000 (0:00:00.090) 0:02:43.677 **** 2020-02-05T16:46:56.375 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:56.411 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:56 +0000 (0:00:00.063) 0:02:43.741 **** 2020-02-05T16:46:56.552 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-02-05T16:46:56.553 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:56.697 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:56 +0000 (0:00:00.285) 0:02:44.026 **** 2020-02-05T16:46:57.317 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-02-05T16:46:57.318 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-02-05T16:46:57.377 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:57 +0000 (0:00:00.680) 0:02:44.706 **** 2020-02-05T16:46:58.660 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-02-05T16:46:58.661 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: krb5-workstation", "Installed: krb5-workstation-1.17-9.el8.x86_64", "Installed: krb5-libs-1.17-9.el8.x86_64", "Installed: libkadm5-1.17-9.el8.x86_64", "Removed: krb5-libs-1.16.1-22.el8.x86_64", "Removed: krb5-workstation-1.16.1-22.el8.x86_64", "Removed: libkadm5-1.16.1-22.el8.x86_64"]} 2020-02-05T16:46:58.721 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:58 +0000 (0:00:01.343) 0:02:46.050 **** 2020-02-05T16:46:58.747 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2020-02-05T16:46:58.748 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:58.783 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:58 +0000 (0:00:00.062) 0:02:46.113 **** 2020-02-05T16:46:58.809 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-02-05T16:46:58.810 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:58.845 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:58 +0000 (0:00:00.062) 0:02:46.175 **** 2020-02-05T16:46:58.870 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-02-05T16:46:58.872 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:58.908 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:58 +0000 (0:00:00.062) 0:02:46.237 **** 2020-02-05T16:46:58.933 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-02-05T16:46:58.935 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:46:58.971 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:58 +0000 (0:00:00.062) 0:02:46.300 **** 2020-02-05T16:46:59.453 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-02-05T16:46:59.454 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-02-05T16:46:59.490 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:59 +0000 (0:00:00.519) 0:02:46.820 **** 2020-02-05T16:46:59.588 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-02-05T16:46:59.589 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-02-05T16:46:59.629 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:46:59 +0000 (0:00:00.138) 0:02:46.958 **** 2020-02-05T16:47:00.324 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-02-05T16:47:00.324 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": true, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "md5sum": "da0dc6d43b293333920cda2d1fc220e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921219.51-222501674945745/source", "state": "file", "uid": 0} 2020-02-05T16:47:01.120 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": true, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "md5sum": "5e1d5cbeb339bbe2906752233e88486d", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921220.32-110184713525169/source", "state": "file", "uid": 0} 2020-02-05T16:47:01.815 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": true, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "md5sum": "f8f4b724d9a4fc98a48d3ba3cb2748e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921221.12-69270293370424/source", "state": "file", "uid": 0} 2020-02-05T16:47:01.855 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:01 +0000 (0:00:02.225) 0:02:49.184 **** 2020-02-05T16:47:02.067 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-02-05T16:47:02.069 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-02-05T16:47:02.107 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:02 +0000 (0:00:00.252) 0:02:49.437 **** 2020-02-05T16:47:02.113 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-02-05T16:47:02.113 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-02-05T16:47:02.152 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:02 +0000 (0:00:00.044) 0:02:49.481 **** 2020-02-05T16:47:02.864 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-02-05T16:47:02.866 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "a1185825c55c0301955d1c6f48afa459", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921222.17-55412833010040/source", "state": "file", "uid": 0} 2020-02-05T16:47:02.997 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** changed: [smithi008.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": true, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "md5sum": "da0dc6d43b293333920cda2d1fc220e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921222.13-245064938248247/source", "state": "file", "uid": 0} 2020-02-05T16:47:03.558 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** changed: [smithi164.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "md5sum": "2ff1d0b7420a46c24916a3f7cba75004", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921222.87-232291190539085/source", "state": "file", "uid": 0} 2020-02-05T16:47:03.787 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** changed: [smithi008.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": true, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "md5sum": "5e1d5cbeb339bbe2906752233e88486d", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921222.99-189611472699431/source", "state": "file", "uid": 0} 2020-02-05T16:47:04.214 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** changed: [smithi164.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "a826600dbfe6a128fb60ddc413ef1069", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921223.56-154093290438977/source", "state": "file", "uid": 0} 2020-02-05T16:47:04.267 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:02.115) 0:02:51.596 **** 2020-02-05T16:47:04.293 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-02-05T16:47:04.294 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:04.331 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.063) 0:02:51.660 **** 2020-02-05T16:47:04.365 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-02-05T16:47:04.367 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-02-05T16:47:04.402 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.071) 0:02:51.732 **** 2020-02-05T16:47:04.480 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-02-05T16:47:04.481 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": true, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "md5sum": "f8f4b724d9a4fc98a48d3ba3cb2748e2", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921223.78-188301432048105/source", "state": "file", "uid": 0} 2020-02-05T16:47:04.523 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.120) 0:02:51.852 **** 2020-02-05T16:47:04.646 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-02-05T16:47:04.648 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:47:04.684 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.161) 0:02:52.014 **** 2020-02-05T16:47:04.721 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-02-05T16:47:04.723 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:04.732 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:04.743 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:04.754 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:04.791 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.106) 0:02:52.121 **** 2020-02-05T16:47:04.802 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-02-05T16:47:04.832 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:04 +0000 (0:00:00.040) 0:02:52.161 **** 2020-02-05T16:47:05.530 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-02-05T16:47:05.531 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "a1185825c55c0301955d1c6f48afa459", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921224.86-238399125453861/source", "state": "file", "uid": 0} 2020-02-05T16:47:06.197 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "md5sum": "2ff1d0b7420a46c24916a3f7cba75004", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921225.53-275344749626330/source", "state": "file", "uid": 0} 2020-02-05T16:47:06.858 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": true, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "md5sum": "a826600dbfe6a128fb60ddc413ef1069", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921226.2-121852284938438/source", "state": "file", "uid": 0} 2020-02-05T16:47:06.897 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:06 +0000 (0:00:02.065) 0:02:54.226 **** 2020-02-05T16:47:06.922 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-02-05T16:47:06.924 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:06.959 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:06 +0000 (0:00:00.061) 0:02:54.288 **** 2020-02-05T16:47:06.995 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-02-05T16:47:06.996 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-02-05T16:47:07.032 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:07 +0000 (0:00:00.073) 0:02:54.362 **** 2020-02-05T16:47:07.285 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-02-05T16:47:07.286 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-02-05T16:47:07.321 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:07 +0000 (0:00:00.288) 0:02:54.651 **** 2020-02-05T16:47:07.367 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-02-05T16:47:07.369 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:07.378 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:07.389 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:07.400 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-02-05T16:47:07.437 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:07 +0000 (0:00:00.116) 0:02:54.767 **** 2020-02-05T16:47:16.251 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-02-05T16:47:16.251 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load", "Installed: nrpe-3.2.1-9.el8.x86_64", "Installed: nagios-common-4.4.5-1.el8.x86_64", "Installed: nagios-plugins-2.2.2-2.20190926git1b8ad57.el8.x86_64", "Installed: nagios-plugins-load-2.2.2-2.20190926git1b8ad57.el8.x86_64"]} 2020-02-05T16:47:16.288 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:16 +0000 (0:00:08.850) 0:03:03.617 **** 2020-02-05T16:47:16.324 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2020-02-05T16:47:16.326 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:16.360 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:16 +0000 (0:00:00.072) 0:03:03.690 **** 2020-02-05T16:47:17.082 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-02-05T16:47:17.083 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "md5sum": "acdfbe59732074480b7f015fa152a14e", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921236.39-181039455567296/source", "state": "file", "uid": 0} 2020-02-05T16:47:17.127 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:17 +0000 (0:00:00.766) 0:03:04.457 **** 2020-02-05T16:47:17.812 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-02-05T16:47:17.813 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "md5sum": "1f0ee88f0225fe8e005cd4f9fbc5e2d9", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921237.15-240582615038139/source", "state": "file", "uid": 0} 2020-02-05T16:47:17.852 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:17 +0000 (0:00:00.724) 0:03:05.182 **** 2020-02-05T16:47:17.878 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-02-05T16:47:17.880 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:17.916 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:17 +0000 (0:00:00.063) 0:03:05.245 **** 2020-02-05T16:47:18.171 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-02-05T16:47:18.172 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-02-05T16:47:18.210 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:18 +0000 (0:00:00.294) 0:03:05.540 **** 2020-02-05T16:47:18.508 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-02-05T16:47:18.509 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-02-05T16:47:18.545 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:18 +0000 (0:00:00.334) 0:03:05.875 **** 2020-02-05T16:47:18.592 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-02-05T16:47:18.593 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:18.630 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:18 +0000 (0:00:00.084) 0:03:05.960 **** 2020-02-05T16:47:19.372 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-02-05T16:47:19.373 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "md5sum": "a2f2d08aa8aa3493038c91c0fd4b7279", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921238.65-139848070014418/source", "state": "file", "uid": 0} 2020-02-05T16:47:19.409 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:19 +0000 (0:00:00.779) 0:03:06.739 **** 2020-02-05T16:47:19.966 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-02-05T16:47:19.967 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load", "Installed: nrpe-3.2.1-9.el8.x86_64", "Installed: nagios-common-4.4.5-1.el8.x86_64", "Installed: nagios-plugins-2.2.2-2.20190926git1b8ad57.el8.x86_64", "Installed: nagios-plugins-load-2.2.2-2.20190926git1b8ad57.el8.x86_64"]} 2020-02-05T16:47:20.003 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:20 +0000 (0:00:00.593) 0:03:07.333 **** 2020-02-05T16:47:20.030 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2020-02-05T16:47:20.031 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:20.066 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:20 +0000 (0:00:00.063) 0:03:07.396 **** 2020-02-05T16:47:20.800 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-02-05T16:47:20.802 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "md5sum": "acdfbe59732074480b7f015fa152a14e", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921240.1-158169154000607/source", "state": "file", "uid": 0} 2020-02-05T16:47:20.840 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:20 +0000 (0:00:00.773) 0:03:08.169 **** 2020-02-05T16:47:21.410 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-02-05T16:47:21.413 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-tmpfiles-setup.service -.mount nss-lookup.target local-fs.target basic.target tmp.mount systemd-journald.socket system.slice var-run.mount sysinit.target time-sync.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target getty@tty1.service xdm.service plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampM 2020-02-05T16:47:21.415 INFO:teuthology.task.ansible.out:onotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "[not set]", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127783", "LimitNPROCSoft": "127783", "LimitRSS": "infinity 2020-02-05T16:47:21.416 INFO:teuthology.task.ansible.out:", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127783", "LimitSIGPENDINGSoft": "127783", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", 2020-02-05T16:47:21.417 INFO:teuthology.task.ansible.out:"StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "User": "nrpe", "UtmpMode": "init", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-02-05T16:47:21.479 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:21 +0000 (0:00:00.637) 0:03:08.807 **** 2020-02-05T16:47:21.546 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-02-05T16:47:21.546 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "md5sum": "1f0ee88f0225fe8e005cd4f9fbc5e2d9", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921240.86-109816935987096/source", "state": "file", "uid": 0} 2020-02-05T16:47:21.585 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:21 +0000 (0:00:00.106) 0:03:08.914 **** 2020-02-05T16:47:21.611 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-02-05T16:47:21.613 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:21.649 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:21 +0000 (0:00:00.063) 0:03:08.978 **** 2020-02-05T16:47:21.727 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-02-05T16:47:21.729 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.003363", "end": "2020-02-05 16:47:21.702387", "rc": 0, "start": "2020-02-05 16:47:21.699024", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-02-05T16:47:21.770 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:21 +0000 (0:00:00.121) 0:03:09.099 **** 2020-02-05T16:47:21.900 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-02-05T16:47:21.902 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-02-05T16:47:21.938 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:21 +0000 (0:00:00.168) 0:03:09.268 **** 2020-02-05T16:47:22.249 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-02-05T16:47:22.250 INFO:teuthology.task.ansible.out:fatal: [smithi008.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-02-05T16:47:22.285 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:22 +0000 (0:00:00.347) 0:03:09.615 **** 2020-02-05T16:47:22.322 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:47:22.357 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:22 +0000 (0:00:00.071) 0:03:09.687 **** 2020-02-05T16:47:23.085 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2020-02-05T16:47:23.091 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "md5sum": "a2f2d08aa8aa3493038c91c0fd4b7279", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921242.38-153212623300853/source", "state": "file", "uid": 0} 2020-02-05T16:47:23.121 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:23 +0000 (0:00:00.763) 0:03:10.450 **** 2020-02-05T16:47:23.665 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-02-05T16:47:23.668 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket -.mount network.target var-run.mount nss-lookup.target system.slice tmp.mount sysinit.target systemd-tmpfiles-setup.service local-fs.target basic.target time-sync.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "getty@tty1.service xdm.service plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampM 2020-02-05T16:47:23.669 INFO:teuthology.task.ansible.out:onotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "[not set]", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127808", "LimitNPROCSoft": "127808", "LimitRSS": "infinity 2020-02-05T16:47:23.671 INFO:teuthology.task.ansible.out:", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127808", "LimitSIGPENDINGSoft": "127808", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", 2020-02-05T16:47:23.672 INFO:teuthology.task.ansible.out:"StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "disabled", "User": "nrpe", "UtmpMode": "init", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-02-05T16:47:23.715 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:23 +0000 (0:00:00.594) 0:03:11.044 **** 2020-02-05T16:47:23.960 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-02-05T16:47:23.962 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.003582", "end": "2020-02-05 16:47:23.935179", "rc": 0, "start": "2020-02-05 16:47:23.931597", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-02-05T16:47:24.004 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:24 +0000 (0:00:00.289) 0:03:11.333 **** 2020-02-05T16:47:32.971 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-02-05T16:47:32.973 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils", "Installed: policycoreutils-python-utils-2.9-3.el8_1.1.noarch", "Installed: libselinux-2.9-2.1.el8.x86_64", "Installed: libselinux-utils-2.9-2.1.el8.x86_64", "Installed: libsepol-2.9-1.el8.x86_64", "Installed: python3-policycoreutils-2.9-3.el8_1.1.noarch", "Installed: python3-libselinux-2.9-2.1.el8.x86_64", "Installed: policycoreutils-2.9-3.el8_1.1.x86_64", "Installed: python3-libsemanage-2.9-1.el8.x86_64", "Installed: libsemanage-2.9-1.el8.x86_64", "Removed: policycoreutils-2.8-16.1.el8.x86_64", "Removed: libselinux-2.8-6.el8.x86_64", "Removed: libselinux-utils-2.8-6.el8.x86_64", "Removed: libsemanage-2.8-5.el8.x86_64", "Removed: libsepol-2.8-2.el8.x86_64", "Removed: policycoreutils-python-utils-2.8-16.1.el8.noarch", "Removed: python3-libselinux-2.8-6.el8.x86_64", "Removed: python3-libsemanage-2.8-5.el8.x86_64", "Removed: python3-policycoreutils-2.8-16.1.el8.noarch"]} 2020-02-05T16:47:33.012 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:33 +0000 (0:00:09.008) 0:03:20.342 **** 2020-02-05T16:47:35.484 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils", "Installed: policycoreutils-python-utils-2.9-3.el8_1.1.noarch", "Installed: libselinux-2.9-2.1.el8.x86_64", "Installed: libselinux-utils-2.9-2.1.el8.x86_64", "Installed: libsepol-2.9-1.el8.x86_64", "Installed: python3-policycoreutils-2.9-3.el8_1.1.noarch", "Installed: python3-libselinux-2.9-2.1.el8.x86_64", "Installed: policycoreutils-2.9-3.el8_1.1.x86_64", "Installed: python3-libsemanage-2.9-1.el8.x86_64", "Installed: libsemanage-2.9-1.el8.x86_64", "Removed: policycoreutils-2.8-16.1.el8.x86_64", "Removed: libselinux-2.8-6.el8.x86_64", "Removed: libselinux-utils-2.8-6.el8.x86_64", "Removed: libsemanage-2.8-5.el8.x86_64", "Removed: libsepol-2.8-2.el8.x86_64", "Removed: policycoreutils-python-utils-2.8-16.1.el8.noarch", "Removed: python3-libselinux-2.8-6.el8.x86_64", "Removed: python3-libsemanage-2.8-5.el8.x86_64", "Removed: python3-policycoreutils-2.8-16.1.el8.noarch"]} 2020-02-05T16:47:35.527 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:47:35 +0000 (0:00:02.514) 0:03:22.856 **** 2020-02-05T16:48:32.239 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-02-05T16:48:32.239 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-targeted-3.14.3-20.el8.noarch", "Installed: selinux-policy-3.14.3-20.el8.noarch", "Removed: selinux-policy-3.14.1-61.el8_0.2.noarch", "Removed: selinux-policy-targeted-3.14.1-61.el8_0.2.noarch"]} 2020-02-05T16:48:32.274 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:32 +0000 (0:00:56.746) 0:04:19.603 **** 2020-02-05T16:48:33.755 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-02-05T16:48:33.757 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-02-05T16:48:33.813 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:33 +0000 (0:00:01.538) 0:04:21.142 **** 2020-02-05T16:48:34.322 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-02-05T16:48:34.323 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:00.265217", "end": "2020-02-05 16:48:34.295058", "failed_when_result": false, "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:48:34.029841", "stderr": "libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).\nsemodule: Failed!", "stderr_lines": ["libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).", "semodule: Failed!"], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:34.359 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:34 +0000 (0:00:00.546) 0:04:21.688 **** 2020-02-05T16:48:35.028 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-02-05T16:48:35.030 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921314.38-8380335080080/source", "state": "file", "uid": 0} 2020-02-05T16:48:35.065 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:35 +0000 (0:00:00.706) 0:04:22.394 **** 2020-02-05T16:48:35.329 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-02-05T16:48:35.330 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.024236", "end": "2020-02-05 16:48:35.304793", "rc": 0, "start": "2020-02-05 16:48:35.280557", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2020-02-05T16:48:35.366 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:35 +0000 (0:00:00.301) 0:04:22.696 **** 2020-02-05T16:48:35.610 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-02-05T16:48:35.611 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.003410", "end": "2020-02-05 16:48:35.582262", "rc": 0, "start": "2020-02-05 16:48:35.578852", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:35.648 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:35 +0000 (0:00:00.281) 0:04:22.978 **** 2020-02-05T16:48:37.576 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-02-05T16:48:37.578 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-targeted-3.14.3-20.el8.noarch", "Installed: selinux-policy-3.14.3-20.el8.noarch", "Removed: selinux-policy-3.14.1-61.el8_0.2.noarch", "Removed: selinux-policy-targeted-3.14.1-61.el8_0.2.noarch"]} 2020-02-05T16:48:37.625 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:37 +0000 (0:00:01.976) 0:04:24.954 **** 2020-02-05T16:48:38.983 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-02-05T16:48:38.985 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-02-05T16:48:39.027 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:39 +0000 (0:00:01.396) 0:04:26.351 **** 2020-02-05T16:48:39.549 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-02-05T16:48:39.550 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:00.280270", "end": "2020-02-05 16:48:39.523621", "failed_when_result": false, "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:48:39.243351", "stderr": "libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).\nsemodule: Failed!", "stderr_lines": ["libsemanage.semanage_direct_remove_key: Unable to remove module nrpe at priority 400. (No such file or directory).", "semodule: Failed!"], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:39.585 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:39 +0000 (0:00:00.563) 0:04:26.915 **** 2020-02-05T16:48:40.263 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-02-05T16:48:40.264 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "md5sum": "286c682fc73965c17876891a4581da3e", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921319.61-59660012339911/source", "state": "file", "uid": 0} 2020-02-05T16:48:40.299 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:40 +0000 (0:00:00.713) 0:04:27.629 **** 2020-02-05T16:48:40.563 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-02-05T16:48:40.563 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.015408", "end": "2020-02-05 16:48:40.538495", "rc": 0, "start": "2020-02-05 16:48:40.523087", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2020-02-05T16:48:40.600 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:40 +0000 (0:00:00.300) 0:04:27.929 **** 2020-02-05T16:48:40.842 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-02-05T16:48:40.843 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.003542", "end": "2020-02-05 16:48:40.821139", "rc": 0, "start": "2020-02-05 16:48:40.817597", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:40.879 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:40 +0000 (0:00:00.278) 0:04:28.208 **** 2020-02-05T16:48:51.777 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-02-05T16:48:51.783 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:15.888532", "end": "2020-02-05 16:48:51.749930", "rc": 0, "start": "2020-02-05 16:48:35.861398", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:51.818 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:51 +0000 (0:00:10.939) 0:04:39.148 **** 2020-02-05T16:48:52.073 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-02-05T16:48:52.075 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-02-05T16:48:52.111 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.292) 0:04:39.440 **** 2020-02-05T16:48:52.137 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-02-05T16:48:52.138 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-02-05T16:48:52.177 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.063) 0:04:39.504 **** 2020-02-05T16:48:52.211 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-02-05T16:48:52.213 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-02-05T16:48:52.252 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.076) 0:04:39.581 **** 2020-02-05T16:48:52.289 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-02-05T16:48:52.290 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2020-02-05T16:48:52.327 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.075) 0:04:39.657 **** 2020-02-05T16:48:52.363 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-02-05T16:48:52.365 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-02-05T16:48:52.405 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.073) 0:04:39.730 **** 2020-02-05T16:48:52.644 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-02-05T16:48:52.646 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-02-05T16:48:52.679 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.278) 0:04:40.009 **** 2020-02-05T16:48:52.932 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-02-05T16:48:52.934 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "name": "ubuntu", "state": "present", "system": false} 2020-02-05T16:48:52.968 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:52 +0000 (0:00:00.289) 0:04:40.298 **** 2020-02-05T16:48:52.993 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-02-05T16:48:52.995 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:48:53.028 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:53 +0000 (0:00:00.060) 0:04:40.358 **** 2020-02-05T16:48:53.054 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-02-05T16:48:53.056 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:48:53.090 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:53 +0000 (0:00:00.061) 0:04:40.420 **** 2020-02-05T16:48:53.355 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi164.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-02-05T16:48:53.406 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:53 +0000 (0:00:00.313) 0:04:40.733 **** 2020-02-05T16:48:54.114 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-02-05T16:48:54.116 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-02-05T16:48:54.157 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:54 +0000 (0:00:00.752) 0:04:41.486 **** 2020-02-05T16:48:54.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-02-05T16:48:54.198 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi164"}, "changed": false} 2020-02-05T16:48:54.232 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:54 +0000 (0:00:00.075) 0:04:41.562 **** 2020-02-05T16:48:55.150 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-02-05T16:48:55.152 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi164.front.sepia.ceph.com", "ansible_hostname": "smithi164", "ansible_nodename": "smithi164"}, "changed": true, "name": "smithi164"} 2020-02-05T16:48:55.189 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:55 +0000 (0:00:00.957) 0:04:42.519 **** 2020-02-05T16:48:55.795 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-02-05T16:48:55.796 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-02-05T16:48:55.840 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:55 +0000 (0:00:00.650) 0:04:43.169 **** 2020-02-05T16:48:56.461 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-02-05T16:48:56.462 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-02-05T16:48:56.508 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:56 +0000 (0:00:00.668) 0:04:43.837 **** 2020-02-05T16:48:57.332 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-02-05T16:48:57.334 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-02-05T16:48:57.372 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:57 +0000 (0:00:00.864) 0:04:44.702 **** 2020-02-05T16:48:57.697 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-02-05T16:48:57.699 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.569548", "end": "2020-02-05 16:48:57.665193", "rc": 0, "start": "2020-02-05 16:48:41.095645", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:48:57.737 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:57 +0000 (0:00:00.364) 0:04:45.066 **** 2020-02-05T16:48:57.999 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-02-05T16:48:58.000 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-02-05T16:48:58.001 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-02-05T16:48:58.003 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-02-05T16:48:58.042 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.305) 0:04:45.372 **** 2020-02-05T16:48:58.084 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.041) 0:04:45.413 **** 2020-02-05T16:48:58.090 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-02-05T16:48:58.091 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-02-05T16:48:58.131 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.047) 0:04:45.461 **** 2020-02-05T16:48:58.168 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-02-05T16:48:58.169 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-02-05T16:48:58.225 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.093) 0:04:45.554 **** 2020-02-05T16:48:58.272 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-02-05T16:48:58.273 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": ["dbench"], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "git-all", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "podman", "podman-docker", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "autoconf", "gdb"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/redhat_8.yml"} 2020-02-05T16:48:58.311 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.086) 0:04:45.641 **** 2020-02-05T16:48:58.329 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-02-05T16:48:58.330 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-02-05T16:48:58.372 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.060) 0:04:45.701 **** 2020-02-05T16:48:58.378 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-02-05T16:48:58.379 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-02-05T16:48:58.419 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.046) 0:04:45.748 **** 2020-02-05T16:48:58.622 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-02-05T16:48:58.624 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-02-05T16:48:58.676 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.256) 0:04:46.005 **** 2020-02-05T16:48:58.680 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-02-05T16:48:58.681 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-02-05T16:48:58.721 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.044) 0:04:46.050 **** 2020-02-05T16:48:58.947 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-02-05T16:48:58.947 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1205, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-02-05T16:48:58.990 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:58 +0000 (0:00:00.268) 0:04:46.319 **** 2020-02-05T16:48:58.996 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-02-05T16:48:58.998 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "name": "ubuntu", "state": "present", "system": false} 2020-02-05T16:48:59.042 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:59 +0000 (0:00:00.052) 0:04:46.371 **** 2020-02-05T16:48:59.070 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-02-05T16:48:59.072 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:48:59.112 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:59 +0000 (0:00:00.070) 0:04:46.442 **** 2020-02-05T16:48:59.138 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-02-05T16:48:59.140 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:48:59.189 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:59 +0000 (0:00:00.076) 0:04:46.518 **** 2020-02-05T16:48:59.495 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-02-05T16:48:59.497 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-02-05T16:48:59.549 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:59 +0000 (0:00:00.359) 0:04:46.878 **** 2020-02-05T16:48:59.617 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-02-05T16:48:59.618 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-02-05T16:48:59.656 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:48:59 +0000 (0:00:00.107) 0:04:46.986 **** 2020-02-05T16:49:00.256 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-02-05T16:49:00.257 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-02-05T16:49:00.299 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:00 +0000 (0:00:00.642) 0:04:47.629 **** 2020-02-05T16:49:00.305 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-02-05T16:49:00.306 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-02-05T16:49:00.356 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:00 +0000 (0:00:00.056) 0:04:47.685 **** 2020-02-05T16:49:00.362 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-02-05T16:49:00.362 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi008"}, "changed": false} 2020-02-05T16:49:00.403 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:00 +0000 (0:00:00.047) 0:04:47.733 **** 2020-02-05T16:49:00.893 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-02-05T16:49:00.895 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi008.front.sepia.ceph.com", "ansible_hostname": "smithi008", "ansible_nodename": "smithi008"}, "changed": true, "name": "smithi008"} 2020-02-05T16:49:00.931 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:00 +0000 (0:00:00.527) 0:04:48.260 **** 2020-02-05T16:49:01.555 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-02-05T16:49:01.556 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-02-05T16:49:01.593 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:01 +0000 (0:00:00.662) 0:04:48.922 **** 2020-02-05T16:49:02.223 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-02-05T16:49:02.224 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-02-05T16:49:02.262 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:02 +0000 (0:00:00.668) 0:04:49.591 **** 2020-02-05T16:49:02.570 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-02-05T16:49:02.571 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-02-05T16:49:02.572 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.964985", "end": "2020-02-05 16:49:02.542870", "rc": 0, "start": "2020-02-05 16:49:00.577885", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:49:02.610 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:02 +0000 (0:00:00.348) 0:04:49.940 **** 2020-02-05T16:49:03.024 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-02-05T16:49:03.026 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-02-05T16:49:03.065 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:03 +0000 (0:00:00.454) 0:04:50.394 **** 2020-02-05T16:49:03.342 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-02-05T16:49:03.343 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "md5sum": "fed359af26a9f925305897bc0401b3aa", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921342.64-167735837868017/source", "state": "file", "uid": 0} 2020-02-05T16:49:03.384 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:03 +0000 (0:00:00.319) 0:04:50.713 **** 2020-02-05T16:49:03.434 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-02-05T16:49:03.475 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:03 +0000 (0:00:00.090) 0:04:50.804 **** 2020-02-05T16:49:03.709 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-02-05T16:49:03.709 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-02-05T16:49:03.747 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:03 +0000 (0:00:00.272) 0:04:51.076 **** 2020-02-05T16:49:04.014 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-02-05T16:49:04.015 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-02-05T16:49:04.051 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:04 +0000 (0:00:00.304) 0:04:51.381 **** 2020-02-05T16:49:04.305 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-02-05T16:49:04.307 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-02-05T16:49:04.342 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:04 +0000 (0:00:00.291) 0:04:51.672 **** 2020-02-05T16:49:04.618 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-02-05T16:49:04.618 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1205, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-02-05T16:49:04.655 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:04 +0000 (0:00:00.312) 0:04:51.985 **** 2020-02-05T16:49:04.838 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-02-05T16:49:04.839 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:01.096090", "end": "2020-02-05 16:49:04.809267", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-02-05 16:49:03.713177", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-02-05T16:49:04.840 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-02-05T16:49:04.879 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:04 +0000 (0:00:00.223) 0:04:52.208 **** 2020-02-05T16:49:04.906 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-02-05T16:49:04.907 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:04.967 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:04 +0000 (0:00:00.088) 0:04:52.297 **** 2020-02-05T16:49:05.008 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-02-05T16:49:05.009 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:49:05.017 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:49:05.070 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:05 +0000 (0:00:00.102) 0:04:52.399 **** 2020-02-05T16:49:05.284 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-02-05T16:49:05.285 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-02-05T16:49:05.323 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:05 +0000 (0:00:00.253) 0:04:52.653 **** 2020-02-05T16:49:05.960 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-02-05T16:49:05.961 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-02-05T16:49:06.001 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:05 +0000 (0:00:00.677) 0:04:53.330 **** 2020-02-05T16:49:06.122 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-02-05T16:49:06.122 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.788398", "end": "2020-02-05 16:49:06.096936", "rc": 0, "start": "2020-02-05 16:49:05.308538", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\n45 files removed", "stdout_lines": ["Updating Subscription Management repositories.", "45 files removed"]} 2020-02-05T16:49:06.162 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.161) 0:04:53.492 **** 2020-02-05T16:49:06.190 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:06.226 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.063) 0:04:53.555 **** 2020-02-05T16:49:06.263 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-02-05T16:49:06.264 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:06.305 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.078) 0:04:53.634 **** 2020-02-05T16:49:06.342 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-02-05T16:49:06.344 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:06.381 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.075) 0:04:53.710 **** 2020-02-05T16:49:06.644 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-02-05T16:49:06.644 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.010693", "end": "2020-02-05 16:49:06.619429", "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:49:06.608736", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-02-05T16:49:06.684 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.302) 0:04:54.013 **** 2020-02-05T16:49:06.734 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-02-05T16:49:06.735 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:06.770 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:06 +0000 (0:00:00.086) 0:04:54.099 **** 2020-02-05T16:49:08.247 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-02-05T16:49:08.248 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.997603", "end": "2020-02-05 16:49:08.214822", "rc": 0, "start": "2020-02-05 16:49:06.217219", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:49:08.286 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:08 +0000 (0:00:01.515) 0:04:55.615 **** 2020-02-05T16:49:09.032 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-02-05T16:49:09.033 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": true, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "md5sum": "fed359af26a9f925305897bc0401b3aa", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1580921348.32-87499888626277/source", "state": "file", "uid": 0} 2020-02-05T16:49:09.073 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:09 +0000 (0:00:00.786) 0:04:56.402 **** 2020-02-05T16:49:09.112 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-02-05T16:49:09.151 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:09 +0000 (0:00:00.078) 0:04:56.481 **** 2020-02-05T16:49:10.545 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-02-05T16:49:10.546 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:01.132898", "end": "2020-02-05 16:49:10.517071", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-02-05 16:49:09.384173", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\nRepository successfully enabled.", "stdout_lines": ["Updating Subscription Management repositories.", "Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-02-05T16:49:10.585 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:10 +0000 (0:00:01.433) 0:04:57.914 **** 2020-02-05T16:49:10.622 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-02-05T16:49:10.623 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:10.666 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:10 +0000 (0:00:00.080) 0:04:57.995 **** 2020-02-05T16:49:10.703 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-02-05T16:49:10.704 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:49:10.741 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:10 +0000 (0:00:00.075) 0:04:58.070 **** 2020-02-05T16:49:11.873 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-02-05T16:49:11.874 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": "yum clean all", "delta": "0:00:00.878961", "end": "2020-02-05 16:49:11.848365", "rc": 0, "start": "2020-02-05 16:49:10.969404", "stderr": "", "stderr_lines": [], "stdout": "Updating Subscription Management repositories.\n45 files removed", "stdout_lines": ["Updating Subscription Management repositories.", "45 files removed"]} 2020-02-05T16:49:11.930 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:11 +0000 (0:00:01.189) 0:04:59.260 **** 2020-02-05T16:49:11.956 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:49:11.958 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:12.005 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:12 +0000 (0:00:00.074) 0:04:59.335 **** 2020-02-05T16:49:12.042 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-02-05T16:49:12.043 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:12.082 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:12 +0000 (0:00:00.076) 0:04:59.411 **** 2020-02-05T16:49:12.118 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-02-05T16:49:12.120 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:12.155 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:12 +0000 (0:00:00.073) 0:04:59.485 **** 2020-02-05T16:49:12.421 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-02-05T16:49:12.422 INFO:teuthology.task.ansible.out:fatal: [smithi008.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.010555", "end": "2020-02-05 16:49:12.391013", "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:49:12.380458", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-02-05T16:49:12.460 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:12 +0000 (0:00:00.304) 0:04:59.789 **** 2020-02-05T16:49:12.497 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-02-05T16:49:12.498 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:49:12.535 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:12 +0000 (0:00:00.074) 0:04:59.864 **** 2020-02-05T16:49:16.849 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-02-05T16:49:16.888 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:16 +0000 (0:00:04.353) 0:05:04.218 **** 2020-02-05T16:49:23.092 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-02-05T16:49:23.093 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.10.0-15.el8.noarch"]} 2020-02-05T16:49:23.129 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:23 +0000 (0:00:06.241) 0:05:10.459 **** 2020-02-05T16:49:23.709 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-02-05T16:49:23.711 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-02-05T16:49:23.747 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:23 +0000 (0:00:00.617) 0:05:11.076 **** 2020-02-05T16:49:30.650 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-02-05T16:49:30.652 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.10.0-15.el8.noarch"]} 2020-02-05T16:49:30.688 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:49:30 +0000 (0:00:06.941) 0:05:18.018 **** 2020-02-05T16:49:30.689 INFO:teuthology.task.ansible.out: 2020-02-05T16:52:14.687 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-02-05T16:52:14.695 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:02:51.199986", "end": "2020-02-05 16:52:14.626255", "rc": 0, "start": "2020-02-05 16:49:23.426269", "stderr": "warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/smartmontools-7.1-7.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY\nImporting GPG key 0x2A8A41EC:\n Userid : \"ktdreyer_ceph-el8 (None) \"\n Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC\n From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg", "stderr_lines": ["warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/smartmontools-7.1-7.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY", "Importing GPG key 0x2A8A41EC:", " Userid : \"ktdreyer_ceph-el8 (None) \"", " Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC", " From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg"], "stdout": "Updating Subscription Management repositories.\nCopr repo for ceph-el8 owned by ktdreyer 2.7 kB/s | 3.3 kB 00:01 \nExtra Packages for Enterprise Linux 154 kB/s | 5.3 kB 00:00 \nlab-extras 934 kB/s | 2.9 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 70 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling Groups:\n Base 2020-02-05T16:52:14.697 INFO:teuthology.task.ansible.out: \n\nTransaction Summary\n================================================================================\n\nComplete!\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - AppStre 68 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 57 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package ArchVersion Repository Size\n================================================================================\nUpgrading:\n smartmontools x86_641:7.1-7.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 546 k\n rsyslog x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 691 k\n insights-client noarch3.0.8-4.el8_1 rhel-8-for-x86_64-appstream-rpms 899 k\n wget x86_641.19.5-8.el8_1.1 rhel-8-for-x86_64-appstream-rpms 735 k\n rsyslog-gnutls x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 27 k\n vim-common x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 6.3 M\n vim-enhanced x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 1.4 M\n rsyslog-gssapi x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 31 k\n nmap-ncat x86_642:7.70-5.el8 rhel-8-for-x86_64-appstream-rpms 237 k\n plymouth x86_640.9.3-15.el8 rhel- 2020-02-05T16:52:14.699 INFO:teuthology.task.ansible.out:8-for-x86_64-appstream-rpms 114 k\n rsyslog-relp x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 29 k\n sssd-ldap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 205 k\n dbus-common noarch1:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 45 k\n ledmon x86_640.92-7.el8 rhel-8-for-x86_64-baseos-rpms 74 k\n cryptsetup x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 177 k\n microcode_ctl x86_644:20190618-1.20191115.3.el8_1 rhel-8-for-x86_64-baseos-rpms 2.5 M\n util-linux x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M\n lvm2 x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M\n util-linux-user x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 98 k\n kpatch noarch0.6.1-5.el8 rhel-8-for-x86_64-baseos-rpms 15 k\n dbus-daemon x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 241 k\n redhat-release x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 38 k\n device-mapper-event-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 265 k\n libmount x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 231 k\n libsmartcols x86_642.32.1-17.el8 r 2020-02-05T16:52:14.700 INFO:teuthology.task.ansible.out:hel-8-for-x86_64-baseos-rpms 175 k\n python3-sssdconfig noarch2.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 116 k\n subscription-manager-cockpit noarch1.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M\n python3-libstoragemgmt noarch1.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 171 k\n libstoragemgmt x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 266 k\n libuuid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 94 k\n dbus-libs x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 184 k\n rsync x86_643.1.3-6.el8 rhel-8-for-x86_64-baseos-rpms 404 k\n redhat-release-eula x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 16 k\n irqbalance x86_642:1.4.0-4.el8 rhel-8-for-x86_64-baseos-rpms 55 k\n ethtool x86_642:5.0-2.el8 rhel-8-for-x86_64-baseos-rpms 151 k\n libipa_hbac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 99 k\n device-mapper-event x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 266 k\n libteam x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 64 k\n sos noarch3.7-8.el8_1 rhel-8-for-x86_64-baseos-rpms 510 k\n sudo x86_641.8.25p1-8.el8_1 2020-02-05T16:52:14.701 INFO:teuthology.task.ansible.out: rhel-8-for-x86_64-baseos-rpms 875 k\n dbus x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 40 k\n sssd-ipa x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 323 k\n sssd x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 90 k\n dbus-tools x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 85 k\n sssd-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M\n mdadm x86_644.1-9.el8 rhel-8-for-x86_64-baseos-rpms 448 k\n libsss_idmap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 104 k\n teamd x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 129 k\n strace x86_644.24-5.el8 rhel-8-for-x86_64-baseos-rpms 971 k\n cockpit x86_64196.3-1.el8 rhel-8-for-x86_64-baseos-rpms 70 k\n python3-libstoragemgmt-clibs x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 27 k\n sssd-client x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 162 k\n cryptsetup-libs x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 422 k\n mcelog x86_643:162-2.el8 rhel-8-for-x86_64-baseos-rpms 79 k\n vdo x86_646.2.1.134-11.el8 2020-02-05T16:52:14.703 INFO:teuthology.task.ansible.out: rhel-8-for-x86_64-baseos-rpms 689 k\n sssd-common-pac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 161 k\n subscription-manager-rhsm-certificates x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 245 k\n libblkid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 215 k\n mtr 2020-02-05T16:52:14.703 INFO:teuthology.task.ansible.out: x86_642:0.92-3.el8 rhel-8-for-x86_64-baseos-rpms 96 k\n cyrus-sasl-gssapi x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 49 k\n cyrus-sasl-lib x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 124 k\n device-mapper x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 371 k\n blktrace x86_641.2.0-10.el8 rhel-8-for-x86_64-baseos-rpms 150 k\n libfdisk x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 249 k\n sssd-proxy x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k\n dnf-plugin-subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 272 k\n kmod-kvdo x86_646.2.1.138-57.el8 rhel-8-for-x86_64-baseos-rpms 355 k\n python3-subscription-manager-rhsm x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 338 k\n subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n sssd-kcm x86_642.2.0-19.el8 2020-02-05T16:52:14.705 INFO:teuthology.task.ansible.out: rhel-8-for-x86_64-baseos-rpms 213 k\n sssd-krb5-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 172 k\n chrony x86_643.5-1.el8 rhel-8-for-x86_64-baseos-rpms 271 k\n cyrus-sasl-plain x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k\n sssd-krb5 x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k\n sssd-ad x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 229 k\n lvm2-libs x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n subscription-manager-plugin-container x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 248 k\n device-mapper-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 403 k\n systemd-libs x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 560 k\n systemd-pam x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 230 k\n systemd x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 3.5 M\n bpftool x86_644.18.0-147.5.1.el8_1 rhel-8-for-x86_64-baseos-rpms 2.1 M\n systemd-udev x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n====================== 2020-02-05T16:52:14.706 INFO:teuthology.task.ansible.out:==========================================================\nUpgrade 83 Packages\n\nTotal download size: 42 M\nDownloading Packages:\n(1/83): rsyslog-8.37.0-13.el8.x86_64.rpm 11 MB/s | 691 kB 00:00 \n(2/83): insights-client-3.0.8-4.el8_1.noarch.rp 13 MB/s | 899 kB 00:00 \n(3/83): smartmontools-7.1-7.el8.x86_64.rpm 5.7 MB/s | 546 kB 00:00 \n(4/83): rsyslog-gnutls-8.37.0-13.el8.x86_64.rpm 760 kB/s | 27 kB 00:00 \n(5/83): wget-1.19.5-8.el8_1.1.x86_64.rpm 15 MB/s | 735 kB 00:00 \n(6/83): rsyslog-gssapi-8.37.0-13.el8.x86_64.rpm 1.0 MB/s | 31 kB 00:00 \n(7/83): vim-enhanced-8.0.1763-13.el8.x86_64.rpm 21 MB/s | 1.4 MB 00:00 \n(8/83): nmap-ncat-7.70-5.el8.x86_64.rpm 9.5 MB/s | 237 kB 00:00 \n(9/83): plymouth-0.9.3-15.el8.x86_64.rpm 3.5 MB/s | 114 kB 00:00 \n(10/83): rsyslog-relp-8.37.0-13.el8.x86_64.rpm 940 kB/s | 29 kB 00:00 \n(11/83): sssd-ldap-2.2.0-19.el8.x86_64.rpm 7.2 MB/s | 205 kB 00:00 \n(12/83): dbus-common-1.12.8-9.el8.noarch.rpm 1.8 MB/s | 45 kB 00:00 \n(13/83): ledmon-0.92-7.el8.x86_64.rpm 2.4 MB/s | 74 kB 00:00 \n(14/83): vim-common-8.0.1763-13.el8.x86_64.rpm 34 MB/s | 6.3 MB 00:00 \n(15/83): cryptsetup-2.2.0-2.el8.x86_64.rpm 3.4 MB/s | 177 kB 00:00 \n(16/83): microcode_ctl-20190618-1.20191115.3.el 28 MB/s | 2.5 MB 00:00 \n(17/83): lvm2-2.03.05-5.el8.x86_64.rpm 24 MB/s | 1.5 MB 00:00 \n(18/83): util-linux-2.32.1-17.el8.x86_64.rpm 28 MB/s | 2.5 MB 00:00 \n(19/83): util-linux-user-2.32.1-17.el8.x86_64.r 2.1 MB/s | 98 kB 00:00 \n(20/83): kpatch-0.6.1-5.el8.noarch.rpm 367 kB/s | 15 kB 00:00 \n(21/83): dbus-daemon-1.12.8-9.el8.x86_64.rpm 7.6 MB/s | 241 kB 00:00 \n(22/83): redhat-release-8.1-3.3.el8.x86_64.rpm 1.7 MB/s | 38 kB 00:00 \n(23/83): device-mapper-event-libs-1.02.163-5.el 7.0 MB/s | 265 kB 00:00 \n(24 2020-02-05T16:52:14.707 INFO:teuthology.task.ansible.out:/83): libmount-2.32.1-17.el8.x86_64.rpm 6.8 MB/s | 231 kB 00:00 \n(25/83): libsmartcols-2.32.1-17.el8.x86_64.rpm 6.0 MB/s | 175 kB 00:00 \n(26/83): python3-sssdconfig-2.2.0-19.el8.noarch 3.6 MB/s | 116 kB 00:00 \n(27/83): python3-libstoragemgmt-1.8.1-2.el8.noa 4.8 MB/s | 171 kB 00:00 \n(28/83): subscription-manager-cockpit-1.25.17-1 21 MB/s | 1.0 MB 00:00 \n(29/83): libstoragemgmt-1.8.1-2.el8.x86_64.rpm 8.0 MB/s | 266 kB 00:00 \n(30/83): libuuid-2.32.1-17.el8.x86_64.rpm 2.6 MB/s | 94 kB 00:00 \n(31/83): dbus-libs-1.12.8-9.el8.x86_64.rpm 4.6 MB/s | 184 kB 00:00 \n(32/83): redhat-release-eula-8.1-3.3.el8.x86_64 615 kB/s | 16 kB 00:00 \n(33/83): rsync-3.1.3-6.el8.x86_64.rpm 8.1 MB/s | 404 kB 00:00 \n(34/83): irqbalance-1.4.0-4.el8.x86_64.rpm 2.2 MB/s | 55 kB 00:00 \n(35/83): ethtool-5.0-2.el8.x86_64.rpm 4.3 MB/s | 151 kB 00:00 \n(36/83): libipa_hbac-2.2.0-19.el8.x86_64.rpm 2.9 MB/s | 99 kB 00:00 \n(37/83): device-mapper-event-1.02.163-5.el8.x86 7.4 MB/s | 266 kB 00:00 \n(38/83): libteam-1.28-4.el8.x86_64.rpm 1.8 MB/s | 64 kB 00:00 \n(39/83): sos-3.7-8.el8_1.noarch.rpm 12 MB/s | 510 kB 00:00 \n(40/83): dbus-1.12.8-9.el8.x86_64.rpm 1.2 MB/s | 40 kB 00:00 \n(41/83): sudo-1.8.25p1-8.el8_1.x86_64.rpm 18 MB/s | 875 kB 00:00 \n(42/83): sssd-ipa-2.2.0-19.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 \n(43/83): sssd-2.2.0-19.el8.x86_64.rpm 2.6 MB/s | 90 kB 00:00 \n(44/83): dbus-tools-1.12.8-9.el8.x86_64.rpm 2.6 MB/s | 85 kB 00:00 \n(45/83): mdadm-4.1-9.el8.x86_64.rpm 12 MB/s | 448 kB 00:00 \n(46/83): libsss_idmap-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 104 kB 00:00 \n(47/83): sssd-common-2.2.0-19.el8.x86_64.rpm 23 MB/s | 1.5 MB 00:00 \n(48/83): teamd-1.28-4.el8.x86_64.rpm 3.4 MB/s | 2020-02-05T16:52:14.709 INFO:teuthology.task.ansible.out:129 kB 00:00 \n(49/83): cockpit-196.3-1.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 \n(50/83): strace-4.24-5.el8.x86_64.rpm 16 MB/s | 971 kB 00:00 \n(51/83): python3-libstoragemgmt-clibs-1.8.1-2.e 860 kB/s | 27 kB 00:00 \n(52/83): sssd-client-2.2.0-19.el8.x86_64.rpm 5.1 MB/s | 162 kB 00:00 \n(53/83): mcelog-162-2.el8.x86_64.rpm 2.0 MB/s | 79 kB 00:00 \n(54/83): cryptsetup-libs-2.2.0-2.el8.x86_64.rpm 9.9 MB/s | 422 kB 00:00 \n(55/83): vdo-6.2.1.134-11.el8.x86_64.rpm 16 MB/s | 689 kB 00:00 \n(56/83): sssd-common-pac-2.2.0-19.el8.x86_64.rp 4.4 MB/s | 161 kB 00:00 \n(57/83): subscription-manager-rhsm-certificates 6.4 MB/s | 245 kB 00:00 \n(58/83): libblkid-2.32.1-17.el8.x86_64.rpm 6.3 MB/s | 215 kB 00:00 \n(59/83): mtr-0.92-3.el8.x86_64.rpm 2.8 MB/s | 96 kB 00:00 \n(60/83): cyrus-sasl-gssapi-2.1.27-1.el8.x86_64. 1.7 MB/s | 49 kB 00:00 \n(61/83): cyrus-sasl-lib-2.1.27-1.el8.x86_64.rpm 4.5 MB/s | 124 kB 00:00 \n(62/83): blktrace-1.2.0-10.el8.x86_64.rpm 4.1 MB/s | 150 kB 00:00 \n(63/83): device-mapper-1.02.163-5.el8.x86_64.rp 8.5 MB/s | 371 kB 00:00 \n(64/83): libfdisk-2.32.1-17.el8.x86_64.rpm 6.2 MB/s | 249 kB 00:00 \n(65/83): sssd-proxy-2.2.0-19.el8.x86_64.rpm 4.3 MB/s | 126 kB 00:00 \n(66/83): dnf-plugin-subscription-manager-1.25.1 7.6 MB/s | 272 kB 00:00 \n(67/83): kmod-kvdo-6.2.1.138-57.el8.x86_64.rpm 10 MB/s | 355 kB 00:00 \n(68/83): python3-subscription-manager-rhsm-1.25 9.7 MB/s | 338 kB 00:00 \n(69/83): sssd-kcm-2.2.0-19.el8.x86_64.rpm 7.3 MB/s | 213 kB 00:00 \n(70/83): subscription-manager-1.25.17-1.el8.x86 21 MB/s | 1.1 MB 00:00 \n(71/83): sssd-krb5-common-2.2.0-19.el8.x86_64.r 4.6 MB/s | 172 kB 00:00 \n(72/83): cyrus-sasl-plain-2.1.27-1.el8.x86_64.r 1.6 MB/s | 47 kB 00:00 \n(73/83): chrony-3.5-1.el8.x86_64.r 2020-02-05T16:52:14.711 INFO:teuthology.task.ansible.out:pm 7.6 MB/s | 271 kB 00:00 \n(74/83): sssd-krb5-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 126 kB 00:00 \n(75/83): sssd-ad-2.2.0-19.el8.x86_64.rpm 6.1 MB/s | 229 kB 00:00 \n(76/83): subscription-manager-plugin-container- 8.1 MB/s | 248 kB 00:00 \n(77/83): lvm2-libs-2.03.05-5.el8.x86_64.rpm 21 MB/s | 1.1 MB 00:00 \n(78/83): device-mapper-libs-1.02.163-5.el8.x86_ 11 MB/s | 403 kB 00:00 \n(79/83): systemd-libs-239-18.el8_1.2.x86_64.rpm 14 MB/s | 560 kB 00:00 \n(80/83): systemd-pam-239-18.el8_1.2.x86_64.rpm 5.9 MB/s | 230 kB 00:00 \n(81/83): systemd-udev-239-18.el8_1.2.x86_64.rpm 27 MB/s | 1.3 MB 00:00 \n(82/83): bpftool-4.18.0-147.5.1.el8_1.x86_64.rp 33 MB/s | 2.1 MB 00:00 \n(83/83): systemd-239-18.el8_1.2.x86_64.rpm 35 MB/s | 3.5 MB 00:00 \n--------------------------------------------------------------------------------\nTotal 33 MB/s | 42 MB 00:01 \nCopr repo for ceph-el8 owned by ktdreyer 954 B/s | 1.0 kB 00:01 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 1/1 \n Preparing : 1/1 \n Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/1 \n Upgrading : libuuid-2.32.1-17.el8.x86_64 1/166 \n Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/166 \n Upgrading : libblkid-2.32.1-17.el8.x86_64 2/166 \n Running scriptlet: libblkid-2.32.1-17.el8.x86_64 2/166 \n Upgrading : libsss_idmap-2.2.0-19.el8.x86_64 3/166 \n Running scriptlet: libsss_idmap-2.2.0-19.el8.x86_64 3/166 \n 2020-02-05T16:52:14.712 INFO:teuthology.task.ansible.out: Upgrading : libmount-2.32.1-17.el8.x86_64 4/166 \n Running scriptlet: libmount-2.32.1-17.el8.x86_64 4/166 \n Upgrading : systemd-libs-239-18.el8_1.2.x86_64 5/166 \n Running scriptlet: systemd-libs-239-18.el8_1.2.x86_64 5/166 \n Upgrading : dbus-libs-1:1.12.8-9.el8.x86_64 6/166 \n Running scriptlet: dbus-libs-1:1.12.8-9.el8.x86_64 6/166 \n Upgrading : cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 \n Running scriptlet: cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 \n Upgrading : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 8/166 \n Upgrading : dbus-tools-1:1.12.8-9.el8.x86_64 9/166 \n Upgrading : sssd-client-2.2.0-19.el8.x86_64 10/166 \n Running scriptlet: sssd-client-2.2.0-19.el8.x86_64 10/166 \n Upgrading : libfdisk-2.32.1-17.el8.x86_64 11/166 \n Running scriptlet: libfdisk-2.32.1-17.el8.x86_64 11/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Upgrading : kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Upgrading : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 13/166 \n Upgrading : subscription-manager-rhsm-certificates-1.25.17-1 14/ 2020-02-05T16:52:14.712 INFO:teuthology.task.ansible.out:166 \n Upgrading : python3-subscription-manager-rhsm-1.25.17-1.el8. 15/166 \n Upgrading : libteam-1.28-4.el8.x86_64 16/166 \n Running scriptlet: libteam-1.28-4.el8.x86_64 16/166 \n Upgrading : libipa_hbac-2.2.0-19.el8.x86_64 17/166 \n Running scriptlet: libipa_hbac-2.2.0-19.el8.x86_64 17/166 \n Upgrading : redhat-rele 2020-02-05T16:52:14.714 INFO:teuthology.task.ansible.out:ase-eula-8.1-3.3.el8.x86_64 18/166 \n Upgrading : python3-sssdconfig-2.2.0-19.el8.noarch 19/166 \n Upgrading : libsmartcols-2.32.1-17.el8.x86_64 20/166 \n Running scriptlet: libsmartcols-2.32.1-17.el8.x86_64 20/166 \n Upgrading : util-linux-2.32.1-17.el8.x86_64 21/166 \n Running scriptlet: util-linux-2.32.1-17.el8.x86_64 21/166 \n Upgrading : dbus-common-1:1.12.8-9.el8.noarch 22/166 \n Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Upgrading : dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Upgrading : device-mapper-8:1.02.163-5.el8.x86_64 24/166 \n Upgrading : device-mapper-libs-8:1.02.163-5.el8.x86_64 25/166 \n Upgrading : cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 \n Running scriptlet: cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 \n Upgrading : systemd-pam-239-18.el8_1.2.x86_64 27/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 \n Upgrading : systemd-239-18.el8_1.2.x86_64 28/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 \n Upgrading : dbus-1:1.12.8-9.el8.x86_64 29/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 \n Upgrading : sssd-common-2.2.0-19.el8.x86_64 30/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 \n Running scriptlet: sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 \n Upgrading : sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 \n Upgrading : device-mapper-event-libs-8:1.02.163-5.el8.x 2020-02-05T16:52:14.715 INFO:teuthology.task.ansible.out:86_64 32/166 \n Upgrading : rsyslog-8.37.0-13.el8.x86_64 33/166 \n Running scriptlet: rsyslog-8.37.0-13.el8.x86_64 33/166 \n Upgrading : sssd-common-pac-2.2.0-19.el8.x86_64 34/166 \n Upgrading : python3-libstoragemgmt-1.8.1-2.el8.noarch 35/166 \n Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Upgrading : libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Upgrading : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 37/166 \n Upgrading : subscription-manager-1.25.17-1.el8.x86_64 38/166 \n Running scriptlet: subscription-manager-1.25.17-1.el8.x86_64 38/166 \n Upgrading : subscription-manager-cockpit-1.25.17-1.el8.noarc 39/166 \n Running scriptlet: sssd-ipa-2.2.0-19.el8.x86_64 40/166 \n Upgrading : sssd-ipa-2.2.0-19.el8.x86_64 40/166 \n Upgrading : sssd-ad-2.2.0-19.el8.x86_64 41/166 \n Upgrading : device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 \n Running scriptlet: device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 \n Upgrading : lvm2-libs-8:2.03.05-5.el8.x86_64 43/166 \n Upgrading : lvm2-8:2.03.05-5.el8.x86_64 44/166 \n Running scriptlet: lvm2-8:2.03.05-5.el8.x86_64 44/166 \nwarning: /etc/lvm/lvm.conf created as /etc/lvm/lvm.conf.rpmnew\n\n Upgrading : sssd-ldap-2.2.0-19.el8.x86_64 45/166 \n Upgrading : sssd-krb5-2.2.0-19.el8.x86_64 46/166 \n Running scriptlet: sssd-proxy-2.2.0-19.el8.x86_64 47/166 \n Upgrading : sssd-proxy-2.2.0-19.el8.x86_64 47/166 \n Upgrading : vim-commo 2020-02-05T16:52:14.717 INFO:teuthology.task.ansible.out:n-2:8.0.1763-13.el8.x86_64 48/166 \n Upgrading : vim-enhanced-2:8.0.1763-13.el8.x86_64 49/166 \n Upgrading : sssd-2.2.0-19.el8.x86_64 50/166 \n Upgrading : vdo-6.2.1.134-11.el8.x86_64 51/166 \n Running scriptlet: vdo-6.2.1.134-11.el8.x86_64 51/166 \n Upgrading : cockpit-196.3-1.el8.x86_64 52/166 \n Upgrading : subscription-manager-plugin-container-1.25.17-1. 53/166 \n Running scriptlet: subscription-manager-plugin-container-1.25.17-1. 53/166 \n Upgrading : rsyslog-gnutls-8.37.0-13.el8.x86_64 54/166 \n Upgrading : rsyslog-gssapi-8.37.0-13.el8.x86_64 55/166 \n Upgrading : rsyslog-relp-8.37.0-13.el8.x86_64 56/166 \n Upgrading : sssd-kcm-2.2.0-19.el8.x86_64 57/166 \n Running scriptlet: sssd-kcm-2.2.0-19.el8.x86_64 57/166 \n Upgrading : smartmontools-1:7.1-7.el8.x86_64 58/166 \n Running scriptlet: smartmontools-1:7.1-7.el8.x86_64 58/166 \n Upgrading : insights-client-3.0.8-4.el8_1.noarch 59/166 \n Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 59/166 \n Upgrading : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 \n Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 \n Upgrading : mdadm-4.1-9.el8.x86_64 61/166 \n Running scriptlet: mdadm-4.1-9.el8.x86_64 61/166 \n Upgrading : mcelog-3:162-2.el8.x86_64 62/166 \n Running scriptlet: mcelog-3:162-2.el8.x86_64 62/166 \n Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 \n Upgrading : chrony-3.5-1.el8.x86_64 2020-02-05T16:52:14.718 INFO:teuthology.task.ansible.out: 63/166 \n Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 \n Upgrading : systemd-udev-239-18.el8_1.2.x86_64 64/166 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 64/166 \n Upgrading : cryptsetup-2.2.0-2.el8.x86_64 65/166 \n Upgrading : util-linux-user-2.32.1-17.el8.x86_64 66/166 \n Upgrading : redhat-release-8.1-3.3.el8.x86_64 67/166 \n Upgrading : teamd-1.28-4.el8.x86_64 68/166 \n Upgrading : cyrus-sasl-plain-2.1.27-1.el8.x86_64 69/166 \n Upgrading : plymouth-0.9.3-15.el8.x86_64 70/166 \n Upgrading : ledmon-0.92-7.el8.x86_64 71/166 \n Upgrading : wget-1.19.5-8.el8_1.1.x86_64 72/166 \n Running scriptlet: wget-1.19.5-8.el8_1.1.x86_64 72/166 \n Upgrading : bpftool-4.18.0-147.5.1.el8_1.x86_64 73/166 \n Upgrading : blktrace-1.2.0-10.el8.x86_64 74/166 \n Upgrading : mtr-2:0.92-3.el8.x86_64 75/166 \n Upgrading : strace-4.24-5.el8.x86_64 76/166 \n Upgrading : sudo-1.8.25p1-8.el8_1.x86_64 77/166 \n Running scriptlet: sudo-1.8.25p1-8.el8_1.x86_64 7 2020-02-05T16:52:14.723 INFO:teuthology.task.ansible.out:7/166 \n Upgrading : sos-3.7-8.el8_1.noarch 78/166 \n Upgrading : ethtool-2:5.0-2.el8.x86_64 79/166 \n Upgrading : irqbalance-2:1.4.0-4.el8.x86_64 80/166 \n Running scriptlet: irqbalance-2:1.4.0-4.el8.x86_64 80/166 \n Upgrading : rsync-3.1.3-6.el8.x86_64 81/166 \n Upgrading : kpatch-0.6.1-5.el8.noarch 82/166 \n Upgrading : nmap-ncat-2:7.70-5.el8.x86_64 83/166 \n Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 83/166 \n Cleanup : sssd-2.0.0-43.el8_0.3.x86_64 84/166 \n Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 \n Cleanup : insights-client-3.0.8-0.el8_0.noarch 85/166 \n Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 \n Cleanup : sssd-ipa-2.0.0-43.el8_0.3.x86_64 86/166 \n Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Cleanup : sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Cleanup : sssd-ad-2.0.0-43.el8_0.3.x86_64 88/166 \n Cleanup : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 89/166 \n Cleanup : sssd-ldap-2.0.0-43.el8_0.3.x86_64 90/166 \n Cleanup : systemd-udev-239-13.el8_0.5.x86_64 91/166 \n Running scriptlet: systemd-udev-239-13.el8_0.5.x86_64 91/166 \n Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Cleanup : vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Running scriptlet: lvm 2020-02-05T16:52:14.724 INFO:teuthology.task.ansible.out:2-8:2.03.02-6.el8.x86_64 93/166 \n Cleanup : lvm2-8:2.03.02-6.el8.x86_64 93/166 \n Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 93/166 \n Cleanup : lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 \n Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 \n Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 \n Cleanup : device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 \n Cleanup : sssd-krb5-2.0.0-43.el8_0.3.x86_64 96/166 \n Cleanup : sssd-proxy-2.0.0-43.el8_0.3.x86_64 97/166 \n Cleanup : cryptsetup-2.0.6-1.el8.x86_64 98/166 \n Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Cleanup : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Cleanup : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 100/166 \n Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Cleanup : sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Cleanup : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 \n Running scriptlet: device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 \n Cleanup : teamd-1.27-10.el8.x86_64 103/166 \n Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 \n Cleanup : chrony-3.3-3.el8.x86_64 104/166 \n Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 \n Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 \n Cleanup : mcelog-3:159-1.el8.x86_64 2020-02-05T16:52:14.726 INFO:teuthology.task.ansible.out: 105/166 \n Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 \n Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 \n Cleanup : mdadm-4.1-4.el8.x86_64 106/166 \n Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 \n Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 \n Cleanup : smartmontools-1:6.6-3.el8.x86_64 107/166 \n Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 \n Cleanup : python3-libstoragemgmt-1.6.2-9.el8.noarch 108/166 \n Cleanup : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 109/166 \n Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Cleanup : libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Cleanup : sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Cleanup : systemd-pam-239-13.el8_0.5.x86_64 112/166 \n Cleanup : ledmon-0.90-3.el8.x86_64 113/166 \n Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 114/166 \n Cleanup : wget-1.19.5-7.el8_0.1.x86_64 114/166 \n Cleanup : subscription-manager-plugin-container-1.23.8-35. 115/166 \n Cleanup : cockpit-185.1-1.el8_0.x86_64 116/166 \n Cleanup : subscription-manager-cockpit-1.23.8-35.el8.noarc 117/166 \n Cleanup : redhat-release-8.0-0.44.el8.x86_64 118/166 \n Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Clea 2020-02-05T16:52:14.727 INFO:teuthology.task.ansible.out:nup : subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Cleanup : python3-subscription-manager-rhsm-1.23.8-35.el8. 120/166 \n Cleanup : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 121/166 \n Cleanup : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 122/166 \n Cleanup : util-linux-user-2.32.1-8.el8.x86_64 123/166 \n Cleanup : rsyslog-relp-8.37.0-9.el8.x86_64 124/166 \n Cleanup : plymouth-0.9.3-12.el8.x86_64 125/166 \n Running scriptlet: plymouth-0.9.3-12.el8.x86_64 125/166 \n Cleanup : rsyslog-gssapi-8.37.0-9.el8.x86_64 126/166 \n Cleanup : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 127/166 \n Cleanup : rsyslog-gnutls-8.37.0-9.el8.x86_64 128/166 \n Cleanup : subscription-manager-rhsm-certificates-1.23.8-35 129/166 \n Cleanup : redhat-release-eula-8.0-0.44.el8.x86_64 130/166 \n Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Cleanup : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Cleanup : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 132/166 \n Cleanup : sos-3.6-10.el8_0.3.noarch 133/166 \n Cleanup : kpatch-0.6.1-1.el8.noarch 134/166 \n Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 \n Cleanup : rsyslog-8.37.0-9.el8.x86_64 135/166 \n Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 \n Cleanup : cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 \n Running scriptlet: cryptsetup-libs-2 2020-02-05T16:52:14.729 INFO:teuthology.task.ansible.out:.0.6-1.el8.x86_64 136/166 \n Cleanup : device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 \n Running scriptlet: device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 \n Cleanup : device-mapper-8:1.02.155-6.el8.x86_64 138/166 \n Cleanup : dbus-1:1.12.8-7.el8.x86_64 139/166 \n Running scriptlet: systemd-239-13.el8_0.5.x86_64 140/ 2020-02-05T16:52:14.729 INFO:teuthology.task.ansible.out:166 \n Cleanup : systemd-239-13.el8_0.5.x86_64 140/166 \n Cleanup : util-linux-2.32.1-8.el8.x86_64 141/166 \n Cleanup : libfdisk-2.32.1-8.el8.x86_64 142/166 \n Running scriptlet: libfdisk-2.32.1-8.el8.x86_64 142/166 \n Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Cleanup : dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Cleanup : dbus-tools-1:1.12.8-7.el8.x86_64 144/166 \n Cleanup : dbus-common-1:1.12.8-7.el8.noarch 145/166 \n Cleanup : dbus-libs-1:1.12.8-7.el8.x86_64 146/166 \n Running scriptlet: dbus-libs-1:1.12.8-7.el8.x86_64 146/166 \n Cleanup : systemd-libs-239-13.el8_0.5.x86_64 147/166 \n Cleanup : libmount-2.32.1-8.el8.x86_64 148/166 \n Running scriptlet: libmount-2.32.1-8.el8.x86_64 148/166 \n Cleanup : libblkid-2.32.1-8.el8.x86_64 149/166 \n Running scriptlet: libblkid-2.32.1-8.el8.x86_64 149/166 \n Cleanup : libuuid-2.32.1-8.el8.x86_64 150/166 \n Running scriptlet: libuuid-2.32.1-8.el8.x86_64 150/166 \n Cleanup : libsmartcols-2.32.1-8.el8.x86_64 2020-02-05T16:52:14.730 INFO:teuthology.task.ansible.out: 151/166 \n Running scriptlet: libsmartcols-2.32.1-8.el8.x86_64 151/166 \n Cleanup : vim-common-2:8.0.1763-11.el8_0.x86_64 152/166 \n Cleanup : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 \n Running scriptlet: cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 \n Cleanup : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 154/166 \n Cleanup : libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 \n Running scriptlet: libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 \n Cleanup : libteam-1.27-10.el8.x86_64 156/166 \n Running scriptlet: libteam-1.27-10.el8.x86_64 156/166 \n Cleanup : libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 \n Running scriptlet: libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 \n Cleanup : bpftool-4.18.0-80.11.2.el8_0.x86_64 158/166 \n Cleanup : blktrace-1.2.0-9.el8.x86_64 159/166 \n Cleanup : mtr-2:0.92-1.el8.x86_64 160/166 \n Cleanup : strace-4.24-3.el8.x86_64 161/166 \n Cleanup : sudo-1.8.25p1-4.el8_0.1.x86_64 162/166 \n Cleanup : ethtool-2:4.16-1.el8.x86_64 163/166 \n Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Cleanup : irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Cleanup : rsync-3.1.3-4.el8.x86_64 165/166 \n Cleanup : nmap-ncat-2:7.70-4.el8.x86_64 166/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 166/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 166/166 \n Running scriptlet: 2020-02-05T16:52:14.732 INFO:teuthology.task.ansible.out: insights-client-3.0.8-4.el8_1.noarch 166/166 \n Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 166/166 \n Running scriptlet: nmap-ncat-2:7.70-4.el8.x86_64 166/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 166/166 \n Running scriptlet: vim-common-2:8.0.1763-13.el8.x86_64 166/166 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 166/166 \n Verifying : smartmontools-1:7.1-7.el8.x86_64 1/166 \n Verifying : smartmontools-1:6.6-3.el8.x86_64 2/166 \n Verifying : rsyslog-8.37.0-13.el8.x86_64 3/166 \n Verifying : rsyslog-8.37.0-9.el8.x86_64 4/166 \n Verifying : insights-client-3.0.8-4.el8_1.noarch 5/166 \n Verifying : insights-client-3.0.8-0.el8_0.noarch 6/166 \n Verifying : wget-1.19.5-8.el8_1.1.x86_64 7/166 \n Verifying : wget-1.19.5-7.el8_0.1.x86_64 8/166 \n Verifying : rsyslog-gnutls-8.37.0-13.el8.x86_64 9/166 \n Verifying : rsyslog-gnutls-8.37.0-9.el8.x86_64 10/166 \n Verifying : vim-common-2:8.0.1763-13.el8.x86_64 11/166 \n Verifying : vim-common-2:8.0.1763-11.el8_0.x86_64 12/166 \n Verifying : vim-enhanced-2:8.0.1763-13.el8.x86_64 13/166 \n Verifying : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 14/166 \n Verifying : rsyslog-gssapi-8.37.0-13.el8.x86_64 15/166 \n Verifying : rsyslog-gssapi-8.37.0-9.el8.x86_64 16/166 \n Verifying : nmap-ncat-2:7.70-5.el8.x86_64 17/166 \n Verifying : nmap-ncat-2:7.70-4.el8.x86_64 18/166 \n Verifying : plymouth-0.9.3-15.el8.x86_64 2020-02-05T16:52:14.733 INFO:teuthology.task.ansible.out: 19/166 \n Verifying : plymouth-0.9.3-12.el8.x86_64 20/166 \n Verifying : rsyslog-relp-8.37.0-13.el8.x86_64 21/166 \n Verifying : rsyslog-relp-8.37.0-9.el8.x86_64 22/166 \n Verifying : sssd-ldap-2.2.0-19.el8.x86_64 23/166 \n Verifying : sssd-ldap-2.0.0-43.el8_0.3.x86_64 24/166 \n Verifying : dbus-common-1:1.12.8-9.el8.noarch 25/166 \n Verifying : dbus-common-1:1.12.8-7.el8.noarch 26/166 \n Verifying : ledmon-0.92-7.el8.x86_64 27/166 \n Verifying : ledmon-0.90-3.el8.x86_64 28/166 \n Verifying : cryptsetup-2.2.0-2.el8.x86_64 29/166 \n Verifying : cryptsetup-2.0.6-1.el8.x86_64 30/166 \n Verifying : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 31/166 \n Verifying : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 32/166 \n Verifying : util-linux-2.32.1-17.el8.x86_64 33/166 \n Verifying : util-linux-2.32.1-8.el8.x86_64 34/166 \n Verifying : lvm2-8:2.03.05-5.el8.x86_64 35/166 \n Verifying : lvm2-8:2.03.02-6.el8.x86_64 36/166 \n Verifying : util-linux-user-2.32.1-17.el8.x86_64 37/166 \n Verifying : util-linux-user-2.32.1-8.el8.x86_64 38/166 \n Verifying : kpatch-0.6.1-5.el8.noarch 39/166 \n Verifying : kpatch-0.6.1-1.el8.noarch 40/166 \n Verifying : dbus-daemon-1:1.12.8-9.el8.x86_64 41/166 \n Verifying : dbus-daemon-1:1.12.8-7.el8.x86_64 42/166 \n Verifying : redhat-release-8.1-3.3.el8.x86_64 43/166 \n 2020-02-05T16:52:14.735 INFO:teuthology.task.ansible.out:Verifying : redhat-release-8.0-0.44.el8.x86_64 44/166 \n Verifying : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 45/166 \n Verifying : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 46/166 \n Verifying : libmount-2.32.1-17.el8.x86_64 47/166 \n Verifying : libmount-2.32.1-8.el8.x86_64 2020-02-05T16:52:14.735 INFO:teuthology.task.ansible.out: 48/166 \n Verifying : libsmartcols-2.32.1-17.el8.x86_64 49/166 \n Verifying : libsmartcols-2.32.1-8.el8.x86_64 50/166 \n Verifying : python3-sssdconfig-2.2.0-19.el8.noarch 51/166 \n Verifying : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 52/166 \n Verifying : subscription-manager-cockpit-1.25.17-1.el8.noarc 53/166 \n Verifying : subscription-manager-cockpit-1.23.8-35.el8.noarc 54/166 \n Verifying : python3-libstoragemgmt-1.8.1-2.el8.noarch 55/166 \n Verifying : python3-libstoragemgmt-1.6.2-9.el8.noarch 56/166 \n Verifying : libstoragemgmt-1.8.1-2.el8.x86_64 57/166 \n Verifying : libstoragemgmt-1.6.2-9.el8.x86_64 58/166 \n Verifying : libuuid-2.32.1-17.el8.x86_64 59/166 \n Verifying : libuuid-2.32.1-8.el8.x86_64 60/166 \n Verifying : dbus-libs-1:1.12.8-9.el8.x86_64 61/166 \n Verifying : dbus-libs-1:1.12.8-7.el8.x86_64 62/166 \n Verifying : rsync-3.1.3-6.el8.x86_64 63/166 \n Verifying : rsync-3.1.3-4.el8.x86_64 64/166 \n Verifying : redhat-release-eula-8.1-3.3.el8.x86_64 65/166 \n Verifying : redhat-release-eula-8.0-0.44.el8.x86_64 66/166 \n Verifying : irqbalance-2:1.4.0-4.el8.x86_64 67/166 \n Verifying : irqbalance-2: 2020-02-05T16:52:14.736 INFO:teuthology.task.ansible.out:1.4.0-2.el8.x86_64 68/166 \n Verifying : ethtool-2:5.0-2.el8.x86_64 69/166 \n Verifying : ethtool-2:4.16-1.el8.x86_64 70/166 \n Verifying : libipa_hbac-2.2.0-19.el8.x86_64 71/166 \n Verifying : libipa_hbac-2.0.0-43.el8_0.3.x86_64 72/166 \n Verifying : device-mapper-event-8:1.02.163-5.el8.x86_64 73/166 \n Verifying : device-mapper-event-8:1.02.155-6.el8.x86_64 74/166 \n Verifying : libteam-1.28-4.el8.x86_64 75/166 \n Verifying : libteam-1.27-10.el8.x86_64 76/166 \n Verifying : sos-3.7-8.el8_1.noarch 77/166 \n Verifying : sos-3.6-10.el8_0.3.noarch 78/166 \n Verifying : sudo-1.8.25p1-8.el8_1.x86_64 79/166 \n Verifying : sudo-1.8.25p1-4.el8_0.1.x86_64 80/166 \n Verifying : dbus-1:1.12.8-9.el8.x86_64 81/166 \n Verifying : dbus-1:1.12.8-7.el8.x86_64 82/166 \n Verifying : sssd-ipa-2.2.0-19.el8.x86_64 83/166 \n Verifying : sssd-ipa-2.0.0-43.el8_0.3.x86_64 84/166 \n Verifying : sssd-2.2.0-19.el8.x86_64 85/166 \n Verifying : sssd-2.0.0-43.el8_0.3.x86_64 86/166 \n Verifying : dbus-tools-1:1.12.8-9.el8.x86_64 87/166 \n Verifying : dbus-tools-1:1.12.8-7.el8.x86_64 88/166 \n Verifying : sssd-common-2.2.0-19.el8.x86_64 89/166 \n Verifying : sssd-common-2.0.0-43.el8_0.3.x86_64 90/166 \n Verifying : mdadm-4.1-9.el8.x86_64 91/166 \n Verifying : mdadm-4.1-4.el8.x86_64 2020-02-05T16:52:14.738 INFO:teuthology.task.ansible.out: 92/166 \n Verifying : libsss_idmap-2.2.0-19.el8.x86_64 93/166 \n Verifying : libsss_idmap-2.0.0-43.el8_0.3.x86_64 94/166 \n Verifying : teamd-1.28-4.el8.x86_64 95/166 \n Verifying : teamd-1.27-10.el8.x86_64 96/166 \n Verifying : strace-4.24-5.el8.x86_64 97/166 \n Verifying : strace-4.24-3.el8.x86_64 98/166 \n Verifying : cockpit-196.3-1.el8.x86_64 99/166 \n Verifying : cockpit-185.1-1.el8_0.x86_64 100/166 \n Verifying : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 101/166 \n Verifying : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 102/166 \n Verifying : sssd-client-2.2.0-19.el8.x86_64 103/166 \n Verifying : sssd-client-2.0.0-43.el8_0.3.x86_64 104/166 \n Verifying : cryptsetup-libs-2.2.0-2.el8.x86_64 105/166 \n Verifying : cryptsetup-libs-2.0.6-1.el8.x86_64 106/166 \n Verifying : mcelog-3:162-2.el8.x86_64 107/166 \n Verifying : mcelog-3:159-1.el8.x86_64 108/166 \n Verifying : vdo-6.2.1.134-11.el8.x86_64 109/166 \n Verifying : vdo-6.2.0.298-10.el8_0.x86_64 110/166 \n Verifying : sssd-common-pac-2.2.0-19.el8.x86_64 111/166 \n Verifying : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 112/166 \n Verifying : subscription-manager-rhsm-certificates-1.25.17-1 113/166 \n Verifying : subscription-manager-rhsm-certificates-1.23.8-35 114/166 \n Verifying : libblkid-2.32.1-17.el8.x86_64 115/166 \n Verifying : libblkid-2.32.1-8.el8.x86_64 116/166 \n Verifying 2020-02-05T16:52:14.739 INFO:teuthology.task.ansible.out: : mtr-2:0.92-3.el8.x86_64 117/166 \n Verifying : mtr-2:0.92-1.el8.x86_64 118/166 \n Verifying : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 119/166 \n Verifying : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 120/166 \n Verifying : cyrus-sasl-lib-2.1.27-1.el8.x86_64 121/166 \n Verifying : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 122/166 \n Verifying : device-mapper-8:1.02.163-5.el8.x86_64 123/166 \n Verifying : device-mapper-8:1.02.155-6.el8.x86_64 124/166 \n Verifying : blktrace-1.2.0-10.el8.x86_64 125/166 \n Verifying : blktrace-1.2.0-9.el8.x86_64 126/166 \n Verifying : libfdisk-2.32.1-17.el8.x86_64 127/166 \n Verifying : libfdisk-2.32.1-8.el8.x86_64 128/166 \n Verifying : sssd-proxy-2.2.0-19.el8.x86_64 129/166 \n Verifying : sssd-proxy-2.0.0-43.el8_0.3.x86_64 130/166 \n Verifying : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 131/166 \n Verifying : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 132/166 \n Verifying : kmod-kvdo-6.2.1.138-57.el8.x86_64 133/166 \n Verifying : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 134/166 \n Verifying : python3-subscription-manager-rhsm-1.25.17-1.el8. 135/166 \n Verifying : python3-subscription-manager-rhsm-1.23.8-35.el8. 136/166 \n Verifying : subscription-manager-1.25.17-1.el8.x86_64 137/166 \n Verifying : subscription-manager-1.23.8-35.el8.x86_64 138/166 \n Verifying : sssd-kcm-2.2.0-19.el8.x86_64 139/166 \n Verifying : sssd-kcm-2.0.0-43.el8_0.3.x86_64 140/166 \n Verifying : sssd-krb5-common-2.2.0-19.e 2020-02-05T16:52:14.741 INFO:teuthology.task.ansible.out:l8.x86_64 141/166 \n Verifying : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 142/166 \n Verifying : chrony-3.5-1.el8.x86_64 143/166 \n Verifying : chrony-3.3-3.el8.x86_64 144/166 \n Verifying : cyrus-sasl-plain-2.1.27-1.el8.x86_64 145/166 \n Verifying : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 146/166 \n Verifying : sssd-krb5-2.2.0-19.el8.x86_64 147/166 \n Verifying : sssd-krb5-2.0.0-43.el8_0.3.x86_64 148/166 \n Verifying : sssd-ad-2.2.0-19.el8.x86_64 149/166 \n Verifying : sssd-ad-2.0.0-43.el8_0.3.x86_64 150/166 \n Verifying : lvm2-libs-8:2.03.05-5.el8.x86_64 151/166 \n Verifying : lvm2-libs-8:2.03.02-6.el8.x86_64 152/166 \n Verifying : subscription-manager-plugin-container-1.25.17-1. 153/166 \n Verifying : subscription-manager-plugin-container-1.23.8-35. 154/166 \n Verifying : device-mapper-libs-8:1.02.163-5.el8.x86_64 155/166 \n Verifying : device-mapper-libs-8:1.02.155-6.el8.x86_64 156/166 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 157/166 \n Verifying : systemd-libs-239-13.el8_0.5.x86_64 158/166 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 159/166 \n Verifying : systemd-pam-239-13.el8_0.5.x86_64 160/166 \n Verifying : systemd-239-18.el8_1.2.x86_64 161/166 \n Verifying : systemd-239-13.el8_0.5.x86_64 162/166 \n Verifying : bpftool-4.18.0-147.5.1.el8_1.x86_64 163/166 \n Verifying : bpftool-4.18.0-80.11.2.el8_0.x86_64 164/166 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 165/166 2020-02-05T16:52:14.742 INFO:teuthology.task.ansible.out:\n Verifying : systemd-udev-239-13.el8_0.5.x86_64 166/166 \nInstalled products updated.\n\nUpgraded:\n smartmontools-1:7.1-7.el8.x86_64 \n rsyslog-8.37.0-13.el8.x86_64 \n insights-client-3.0.8-4.el8_1.noarch \n wget-1.19.5-8.el8_1.1.x86_64 \n rsyslog-gnutls-8.37.0-13.el8.x86_64 \n vim-common-2:8.0.1763-13.el8.x86_64 \n vim-enhanced-2:8.0.1763-13.el8.x86_64 \n rsyslog-gssapi-8.37.0-13.el8.x86_64 \n nmap-ncat-2:7.70-5.el8.x86_64 \n plymouth-0.9.3-15.el8.x86_64 \n rsyslog-relp-8.37.0-13.el8.x86_64 \n sssd-ldap-2.2.0-19.el8.x86_64 \n dbus-common-1:1.12.8-9.el8.noarch \n ledmon-0.92-7.el8.x86_64 \n cryptsetup-2.2.0-2.el8.x86_64 \n microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_64 \n util-linux-2.32.1-17.el8.x86_64 \n lvm2-8:2.03.05-5.el8.x86_64 \n util-linux-user-2.32.1-17.el8.x86_64 \n kpatch-0.6.1-5.el8.noarch \n dbus-daemon-1:1.12.8-9.el8.x86_64 \n redhat-release-8.1-3.3.el8.x86_64 \n device-mapper-event-libs-8:1.02.163-5.el8.x86_64 2020-02-05T16:52:14.744 INFO:teuthology.task.ansible.out: \n libmount-2.32.1-17.el8.x86_64 \n libsmartcols-2.32.1-17.el8.x86_64 \n python3-sssdconfig-2.2.0-19.el8.noarch \n subscription-manager-cockpit-1.25.17-1.el8.noarch \n python3-libstoragemgmt-1.8.1-2.el8.noarch \n libstoragemgmt-1.8.1-2.el8.x86_64 \n libuuid-2.32.1-17.el8.x86_64 \n dbus-libs-1:1.12.8-9.el8.x86_64 \n rsync-3.1.3-6.el8.x86_64 \n redhat-release-eula-8.1-3.3.el8.x86_64 \n irqbalance-2:1.4.0-4.el8.x86_64 \n ethtool-2:5. 2020-02-05T16:52:14.744 INFO:teuthology.task.ansible.out:0-2.el8.x86_64 \n libipa_hbac-2.2.0-19.el8.x86_64 \n device-mapper-event-8:1.02.163-5.el8.x86_64 \n libteam-1.28-4.el8.x86_64 \n sos-3.7-8.el8_1.noarch \n sudo-1.8.25p1-8.el8_1.x86_64 \n dbus-1:1.12.8-9.el8.x86_64 \n sssd-ipa-2.2.0-19.el8.x86_64 \n sssd-2.2.0-19.el8.x86_64 \n dbus-tools-1:1.12.8-9.el8.x86_64 \n sssd-common-2.2.0-19.el8.x86_64 \n mdadm-4.1-9.el8.x86_64 \n libsss_idmap-2.2.0-19.el8.x86_64 \n teamd-1.28-4.el8.x 2020-02-05T16:52:14.745 INFO:teuthology.task.ansible.out:86_64 \n strace-4.24-5.el8.x86_64 \n cockpit-196.3-1.el8.x86_64 \n python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 \n sssd-client-2.2.0-19.el8.x86_64 \n cryptsetup-libs-2.2.0-2.el8.x86_64 \n mcelog-3:162-2.el8.x86_64 \n vdo-6.2.1.134-11.el8.x86_64 \n sssd-common-pac-2.2.0-19.el8.x86_64 \n subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64 \n libblkid-2.32.1-17.el8.x86_64 \n mtr-2:0.92-3.el8.x86_64 \n cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 \n cyrus-sasl-lib-2.1.27-1.el8.x86_64 \n device-mapper-8:1.02.163-5.el8.x86_64 \n blktrace-1.2.0-10.el8.x86_64 \n libfdisk-2.32.1-17.el8.x86_64 \n sssd-proxy-2.2.0-19.el8.x86_64 \n dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64 \n kmod-kvdo-6.2.1.138-57.el8.x86_64 \n python3-subscription-manager-rhsm-1.25.17-1.el8.x86_64 \n subscription-manager-1.25.17-1.el8.x86_64 \n sssd-kcm-2.2.0-19.el8.x86_64 \n sssd-krb5-common-2.2.0-19.el8.x86_64 \n chrony-3.5-1.el8.x86_64 2020-02-05T16:52:14.747 INFO:teuthology.task.ansible.out: \n cyrus-sasl-plain-2.1.27-1.el8.x86_64 \n sssd-krb5-2.2.0-19.el8.x86_64 \n sssd-ad-2.2.0-19.el8.x86_64 \n lvm2-libs-8:2.03.05-5.el8.x86_64 \n subscription-manager-plugin-container-1.25.17-1.el8.x86_64 \n device-mapper-libs-8:1.02.163-5.el8.x86_64 \n systemd-libs-239-18.el8_1.2.x86_64 \n systemd-pam-239-18.el8_1.2.x86_64 \n systemd-239-18.el8_1.2.x86_64 \n bpftool-4.18.0-147.5.1.el8_1.x86_64 \n systemd-udev-239-18.el8_1.2.x86_64 \n\nComplete!", "stdout_lines": ["Updating Subscription Management repositories.", "Copr repo for ceph-el8 owned by ktdreyer 2.7 kB/s | 3.3 kB 00:01 ", "Extra Packages for Enterprise Linux 154 kB/s | 5.3 kB 00:00 ", "lab-extras 934 kB/s | 2.9 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - AppStre 84 kB/s | 2.8 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - BaseOS 70 kB/s | 2.4 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Arch Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Updating Subscription Management repositories.", "Red Hat Ent 2020-02-05T16:52:14.748 INFO:teuthology.task.ansible.out:erprise Linux 8 for x86_64 - AppStre 68 kB/s | 2.8 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - BaseOS 57 kB/s | 2.4 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package ArchVersion Repository Size", "================================================================================", "Upgrading:", " smartmontools x86_641:7.1-7.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 546 k", " rsyslog x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 691 k", " insights-client noarch3.0.8-4.el8_1 rhel-8-for-x86_64-appstream-rpms 899 k", " wget x86_641.19.5-8.el8_1.1 rhel-8-for-x86_64-appstream-rpms 735 k", " rsyslog-gnutls x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 27 k", " vim-common x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 6.3 M", " vim-enhanced x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 1.4 M", " rsyslog-gssapi x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 31 k", " nmap-ncat x86_642:7.70-5.el8 rhel-8-for-x86_64-appstream-rpms 237 k", " plymouth x86_640.9.3-15.el8 rhel-8-for-x86_64-appstream-rpms 114 k", " rsyslog-relp x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 29 k", " sssd-ldap 2020-02-05T16:52:14.750 INFO:teuthology.task.ansible.out: x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 205 k", " dbus-common noarch1:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 45 k", " ledmon x86_640.92-7.el8 rhel-8-for-x86_64-baseos-rpms 74 k", " cryptsetup x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 177 k", " microcode_ctl x86_644:20190618-1.20191115.3.el8_1 rhel-8-for-x86_64-baseos-rpms 2.5 M", " util-linux x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M", " lvm2 x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M", " util-linux-user x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 98 k", " kpatch noarch0.6.1-5.el8 rhel-8-for-x86_64-baseos-rpms 15 k", " dbus-daemon x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 241 k", " redhat-release x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 38 k", " device-mapper-event-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 265 k", " libmount x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 231 k", " libsmartcols x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 175 k", " python3-sssdconfig noarch2.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 2020-02-05T16:52:14.751 INFO:teuthology.task.ansible.out: 116 k", " subscription-manager-cockpit noarch1.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M", " python3-libstoragemgmt noarch1.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 171 k", " libstoragemgmt x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 266 k", " libuuid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 94 k", " dbus-libs x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 184 k", " rsync x86_643.1.3-6.el8 rhel-8-for-x86_64-baseos-rpms 404 k", " redhat-release-eula x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 16 k", " irqbalance x86_642:1.4.0-4.el8 rhel-8-for-x86_64-baseos-rpms 55 k", " ethtool x86_642:5.0-2.el8 rhel-8-for-x86_64-baseos-rpms 151 k", " libipa_hbac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 99 k", " device-mapper-event x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 266 k", " libteam x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 64 k", " sos noarch3.7-8.el8_1 rhel-8-for-x86_64-baseos-rpms 510 k", " sudo x86_641.8.25p1-8.el8_1 rhel-8-for-x86_64-baseos-rpms 875 k", " dbus x86_641:1.12.8-9.el8 2020-02-05T16:52:14.753 INFO:teuthology.task.ansible.out: rhel-8-for-x86_64-baseos-rpms 40 k", " sssd-ipa x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 323 k", " sssd x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 90 k", " dbus-tools x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 85 k", " sssd-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M", " mdadm x86_644.1-9.el8 rhel-8-for-x86_64-baseos-rpms 448 k", " libsss_idmap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 104 k", " teamd x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 129 k", " strace x86_644.24-5.el8 rhel-8-for-x86_64-baseos-rpms 971 k", " cockpit x86_64196.3-1.el8 2020-02-05T16:52:14.753 INFO:teuthology.task.ansible.out: rhel-8-for-x86_64-baseos-rpms 70 k", " python3-libstoragemgmt-clibs x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 27 k", " sssd-client x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 162 k", " cryptsetup-libs x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 422 k", " mcelog x86_643:162-2.el8 rhel-8-for-x86_64-baseos-rpms 79 k", " vdo x86_646.2.1.134-11.el8 rhel-8-for-x86_64-baseos-rpms 689 k", " sssd-common-pac x 2020-02-05T16:52:14.754 INFO:teuthology.task.ansible.out:86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 161 k", " subscription-manager-rhsm-certificates x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 245 k", " libblkid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 215 k", " mtr x86_642:0.92-3.el8 rhel-8-for-x86_64-baseos-rpms 96 k", " cyrus-sasl-gssapi x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 49 k", " cyrus-sasl-lib x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 124 k", " device-mapper x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 371 k", " blktrace x86_641.2.0-10.el8 rhel-8-for-x86_64-baseos-rpms 150 k", " libfdisk x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 249 k", " sssd-proxy x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k", " dnf-plugin-subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 272 k", " kmod-kvdo x86_646.2.1.138-57.el8 rhel-8-for-x86_64-baseos-rpms 355 k", " python3-subscription-manager-rhsm x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 338 k", " subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M", " sssd-kcm x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 213 k", " sssd 2020-02-05T16:52:14.757 INFO:teuthology.task.ansible.out:-krb5-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 172 k", " chrony x86_643.5-1.el8 rhel-8-for-x86_64-baseos-rpms 271 k", " cyrus-sasl-plain x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k", " sssd-krb5 x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k", " sssd-ad x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 229 k", " lvm2-libs x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M", " subscription-manager-plugin-container x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 248 k", " device-mapper-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 403 k", " systemd-libs x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 560 k", " systemd-pam x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 230 k", " systemd x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 3.5 M", " bpftool x86_644.18.0-147.5.1.el8_1 rhel-8-for-x86_64-baseos-rpms 2.1 M", " systemd-udev x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================= 2020-02-05T16:52:14.758 INFO:teuthology.task.ansible.out:===============", "Upgrade 83 Packages", "", "Total download size: 42 M", "Downloading Packages:", "(1/83): rsyslog-8.37.0-13.el8.x86_64.rpm 11 MB/s | 691 kB 00:00 ", "(2/83): insights-client-3.0.8-4.el8_1.noarch.rp 13 MB/s | 899 kB 00:00 ", "(3/83): smartmontools-7.1-7.el8.x86_64.rpm 5.7 MB/s | 546 kB 00:00 ", "(4/83): rsyslog-gnutls-8.37.0-13.el8.x86_64.rpm 760 kB/s | 27 kB 00:00 ", "(5/83): wget-1.19.5-8.el8_1.1.x86_64.rpm 15 MB/s | 735 kB 00:00 ", "(6/83): rsyslog-gssapi-8.37.0-13.el8.x86_64.rpm 1.0 MB/s | 31 kB 00:00 ", "(7/83): vim-enhanced-8.0.1763-13.el8.x86_64.rpm 21 MB/s | 1.4 MB 00:00 ", "(8/83): nmap-ncat-7.70-5.el8.x86_64.rpm 9.5 MB/s | 237 kB 00:00 ", "(9/83): plymouth-0.9.3-15.el8.x86_64.rpm 3.5 MB/s | 114 kB 00:00 ", "(10/83): rsyslog-relp-8.37.0-13.el8.x86_64.rpm 940 kB/s | 29 kB 00:00 ", "(11/83): sssd-ldap-2.2.0-19.el8.x86_64.rpm 7.2 MB/s | 205 kB 00:00 ", "(12/83): dbus-common-1.12.8-9.el8.noarch.rpm 1.8 MB/s | 45 kB 00:00 ", "(13/83): ledmon-0.92-7.el8.x86_64.rpm 2.4 MB/s | 74 kB 00:00 ", "(14/83): vim-common-8.0.1763-13.el8.x86_64.rpm 34 MB/s | 6.3 MB 00:00 ", "(15/83): cryptsetup-2.2.0-2.el8.x86_64.rpm 3.4 MB/s | 177 kB 00:00 ", "(16/83): microcode_ctl-20190618-1.20191115.3.el 28 MB/s | 2.5 MB 00:00 ", "(17/83): lvm2-2.03.05-5.el8.x86_64.rpm 24 MB/s | 1.5 MB 00:00 ", "(18/83): util-linux-2.32.1-17.el8.x86_64.rpm 28 MB/s | 2.5 MB 00:00 ", "(19/83): util-linux-user-2.32.1-17.el8.x86_64.r 2.1 MB/s | 98 kB 00:00 ", "(20/83): kpatch-0.6.1-5.el8.noarch.rpm 367 kB/s | 15 kB 00:00 ", "(21/83): dbus-daemon-1.12.8-9.el8.x86_64.rpm 7.6 MB/s | 241 kB 00:00 ", "(22/83): redhat-release-8.1-3.3.el8.x86_64.rpm 1.7 MB/s | 38 kB 00:00 ", "(23/83): device-mapper-event-libs-1.02.163-5.el 7.0 MB/s | 265 kB 00: 2020-02-05T16:52:14.760 INFO:teuthology.task.ansible.out:00 ", "(24/83): libmount-2.32.1-17.el8.x86_64.rpm 6.8 MB/s | 231 kB 00:00 ", "(25/83): libsmartcols-2.32.1-17.el8.x86_64.rpm 6.0 MB/s | 175 kB 00:00 ", "(26/83): python3-sssdconfig-2.2.0-19.el8.noarch 3.6 MB/s | 116 kB 00:00 ", "(27/83): python3-libstoragemgmt-1.8.1-2.el8.noa 4.8 MB/s | 171 kB 00:00 ", "(28/83): subscription-manager-cockpit-1.25.17-1 21 MB/s | 1.0 MB 00:00 ", "(29/83): libstoragemgmt-1.8.1-2.el8.x86_64.rpm 8.0 MB/s | 266 kB 00:00 ", "(30/83): libuuid-2.32.1-17.el8.x86_64.rpm 2.6 MB/s | 94 kB 00:00 ", "(31/83): dbus-libs-1.12.8-9.el8.x86_64.rpm 4.6 MB/s | 184 kB 00:00 ", "(32/83): redhat-release-eula-8.1-3.3.el8.x86_64 615 kB/s | 16 kB 00:00 ", "(33/83): rsync-3.1.3-6.el8.x86_64.rpm 8.1 MB/s | 404 kB 00:00 ", "(34/83): irqbalance-1.4.0-4.el8.x86_64.rpm 2.2 MB/s | 55 kB 00:00 ", "(35/83): ethtool-5.0-2.el8.x86_64.rpm 4.3 MB/s | 151 kB 00:00 ", "(36/83): libipa_hbac-2.2.0-19.el8.x86_64.rpm 2.9 MB/s | 99 kB 00:00 ", "(37/83): device-mapper-event-1.02.163-5.el8.x86 7.4 MB/s | 266 kB 00:00 ", "(38/83): libteam-1.28-4.el8.x86_64.rpm 1.8 MB/s | 64 kB 00:00 ", "(39/83): sos-3.7-8.el8_1.noarch.rpm 12 MB/s | 510 kB 00:00 ", "(40/83): dbus-1.12.8-9.el8.x86_64.rpm 1.2 MB/s | 40 kB 00:00 ", "(41/83): sudo-1.8.25p1-8.el8_1.x86_64.rpm 18 MB/s | 875 kB 00:00 ", "(42/83): sssd-ipa-2.2.0-19.el8.x86_64.rpm 9.5 MB/s | 323 kB 00:00 ", "(43/83): sssd-2.2.0-19.el8.x86_64.rpm 2.6 MB/s | 90 kB 00:00 ", "(44/83): dbus-tools-1.12.8-9.el8.x86_64.rpm 2.6 MB/s | 85 kB 00:00 ", "(45/83): mdadm-4.1-9.el8.x86_64.rpm 12 MB/s | 448 kB 00:00 ", "(46/83): libsss_idmap-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 104 kB 00:00 ", "(47/83): sssd-common-2.2.0-19.el8.x86_64.rpm 23 MB/s | 1.5 MB 00:00 ", 2020-02-05T16:52:14.761 INFO:teuthology.task.ansible.out: "(48/83): teamd-1.28-4.el8.x86_64.rpm 3.4 MB/s | 129 kB 00:00 ", "(49/83): cockpit-196.3-1.el8.x86_64.rpm 2.0 MB/s | 70 kB 00:00 ", "(50/83): strace-4.24-5.el8.x86_64.rpm 16 MB/s | 971 kB 00:00 ", "(51/83): python3-libstoragemgmt-clibs-1.8.1-2.e 860 kB/s | 27 kB 00:00 ", "(52/83): sssd-client-2.2.0-19.el8.x86_64.rpm 5.1 MB/s | 162 kB 00:00 ", "(53/83): mcelog-162-2.el8.x86_64.rpm 2.0 MB/s | 79 kB 00:00 ", "(54/83): cryptsetup-libs-2.2.0-2.el8.x86_64.rpm 9.9 MB/s | 422 kB 00:00 ", "(55/83): vdo-6.2.1.134-11.el8.x86_64.rpm 16 MB/s | 689 kB 00:00 ", "(56/83): sssd-common-pac-2.2.0-19.el8.x86_64.rp 4.4 MB/s | 161 kB 00:00 ", "(57/83): subscription-manager-rhsm-certificates 6.4 MB/s | 245 kB 00:00 ", "(58/83): libblkid-2.32.1-17.el8.x86_64.rpm 6.3 MB/s | 215 kB 00:00 ", "(59/83): mtr-0.92-3.el8.x86_64.rpm 2.8 MB/s | 96 kB 00:00 ", "(60/83): cyrus-sasl-gssapi-2.1.27-1.el8.x86_64. 1.7 MB/s | 49 kB 00:00 ", "(61/83): cyrus-sasl-lib-2.1.27-1.el8.x86_64.rpm 4.5 MB/s | 124 kB 00:00 ", "(62/83): blktrace-1.2.0-10.el8.x86_64.rpm 4.1 MB/s | 150 kB 00:00 ", "(63/83): device-mapper-1.02.163-5.el8.x86_64.rp 8.5 MB/s | 371 kB 00:00 ", "(64/83): libfdisk-2.32.1-17.el8.x86_64.rpm 6.2 MB/s | 249 kB 00:00 ", "(65/83): sssd-proxy-2.2.0-19.el8.x86_64.rpm 4.3 MB/s | 126 kB 00:00 ", "(66/83): dnf-plugin-subscription-manager-1.25.1 7.6 MB/s | 272 kB 00:00 ", "(67/83): kmod-kvdo-6.2.1.138-57.el8.x86_64.rpm 10 MB/s | 355 kB 00:00 ", "(68/83): python3-subscription-manager-rhsm-1.25 9.7 MB/s | 338 kB 00:00 ", "(69/83): sssd-kcm-2.2.0-19.el8.x86_64.rpm 7.3 MB/s | 213 kB 00:00 ", "(70/83): subscription-manager-1.25.17-1.el8.x86 21 MB/s | 1.1 MB 00:00 ", "(71/83): sssd-krb5-common-2.2.0-19.el8.x86_64.r 4.6 MB/s | 172 kB 00:00 ", "(72/83 2020-02-05T16:52:14.763 INFO:teuthology.task.ansible.out:): cyrus-sasl-plain-2.1.27-1.el8.x86_64.r 1.6 MB/s | 47 kB 00:00 ", "(73/83): chrony-3.5-1.el8.x86_64.rpm 7.6 MB/s | 271 kB 00:00 ", "(74/83): sssd-krb5-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 126 kB 00:00 ", "(75/83): sssd-ad-2.2.0-19.el8.x86_64.rpm 6.1 MB/s | 229 kB 00:00 ", "(76/83): subscription-manager-plugin-container- 8.1 MB/s | 248 kB 00:00 ", "(77/83): lvm2-libs-2.03.05-5.el8.x86_64.rpm 21 MB/s | 1.1 MB 00:00 ", "(78/83): device-mapper-libs-1.02.163-5.el8.x86_ 11 MB/s | 403 kB 00:00 ", "(79/83): systemd-libs-239-18.el8_1.2.x86_64.rpm 14 MB/s | 560 kB 00:00 ", "(80/83): systemd-pam-239-18.el8_1.2.x86_64.rpm 5.9 MB/s | 230 kB 00:00 ", "(81/83): systemd-udev-239-18.el8_1.2.x86_64.rpm 27 MB/s | 1.3 MB 00:00 ", "(82/83): bpftool-4.18.0-147.5.1.el8_1.x86_64.rp 33 MB/s | 2.1 MB 00:00 ", "(83/83): systemd-239-18.el8_1.2.x86_64.rpm 35 MB/s | 3.5 MB 00:00 ", "--------------------------------------------------------------------------------", "Total 33 MB/s | 42 MB 00:01 ", "Copr repo for ceph-el8 owned by ktdreyer 954 B/s | 1.0 kB 00:01 ", "Key imported successfully", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 1/1 ", " Preparing : 1/1 ", " Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/1 ", " Upgrading : libuuid-2.32.1-17.el8.x86_64 1/166 ", " Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/166 ", " Upgrading : libblkid-2.32.1-17.el8.x86_64 2/166 ", " Running scriptlet: libblkid-2.32.1-17.el8.x86_64 2/166 ", 2020-02-05T16:52:14.764 INFO:teuthology.task.ansible.out:" Upgrading : libsss_idmap-2.2.0-19.el8.x86_64 3/166 ", " Running scriptlet: libsss_idmap-2.2.0-19.el8.x86_64 3/166 ", " Upgrading : libmount-2.32.1-17.el8.x86_64 4/166 ", " Running scriptlet: libmount-2.32.1-17.el8.x86_64 4/166 ", " Upgrading : systemd-libs-239-18.el8_1.2.x86_64 5/166 ", " Running scriptlet: systemd-libs-239-18.el8_1.2.x86_64 5/166 ", " Upgrading : dbus-libs-1:1.12.8-9.el8.x86_64 6/166 ", " Running scriptlet: dbus-libs-1:1.12.8-9.el8.x86_64 6/166 ", " Upgrading : cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 ", " Running scriptlet: cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 ", " Upgrading : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 8/166 ", " Upgrading : dbus-tools-1:1.12.8-9.el8.x86_64 9/166 ", " Upgrading : sssd-client-2.2.0-19.el8.x86_64 10/166 ", " Running scriptlet: sssd-client-2.2.0-19.el8.x86_64 10/166 ", " Upgrading : libfdisk-2.32.1-17.el8.x86_64 11/166 ", " Running scriptlet: libfdisk-2.32.1-17.el8.x86_64 11/166 ", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Upgrading : kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Upgrading : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 13/166 ", " Upgrading : subscription-manager-rhsm-certificates-1.25.17-1 14/166 ", " Upgrading : python3-subscription-manager-rhsm-1.25.17-1.el8. 15/166 ", " Upgrading : libteam-1.28-4.el8.x86_64 16/166 ", " Running scriptlet: libteam-1.28-4.el8.x86_64 2020-02-05T16:52:14.766 INFO:teuthology.task.ansible.out: 16/166 ", " Upgrading : libipa_hbac-2.2.0-19.el8.x86_64 17/166 ", " Running scriptlet: libipa_hbac-2.2.0-19.el8.x86_64 17/166 ", " Upgrading : redhat-release-eula-8.1-3.3.el8.x86_64 18/166 ", " Upgrading : python3-sssdconfig-2.2.0-19.el8.noarch 19/166 ", " Upgrading : libsmartcols-2.32.1-17.el8.x86_64 20/166 ", " Running scriptlet: libsmartcols-2.32.1-17.el8.x86_64 20/166 ", " Upgrading : util-linux-2.32.1-17.el8.x86_64 21/166 ", " Running scriptlet: util-linux-2.32.1-17.el8.x86_64 21/166 ", " Upgrading : dbus-common-1:1.12.8-9.el8.noarch 22/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Upgrading : dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Upgrading : device-mapper-8:1.02.163-5.el8.x86_64 24/166 ", " Upgrading : device-mapper-libs-8:1.02.163-5.el8.x86_64 25/166 ", " Upgrading : cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 ", " Running scriptlet: cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 ", " Upgrading : systemd-pam-239-18.el8_1.2.x86_64 27/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 ", " Upgrading : systemd-239-18.el8_1.2.x86_64 28/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 ", " Upgrading : dbus-1:1.12.8-9.el8.x86_64 29/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 ", " Upgrading : sssd-common-2.2.0-19.el8.x86_64 30/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_6 2020-02-05T16:52:14.767 INFO:teuthology.task.ansible.out:4 30/166 ", " Running scriptlet: sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 ", " Upgrading : sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 ", " Upgrading : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 32/166 ", " Upgrading : rsyslog-8.37.0-13.el8.x86_64 33/166 ", " Running scriptlet: rsyslog-8.37.0-13.el8.x86_64 33/166 ", " Upgrading : sssd-common-pac-2.2.0-19.el8.x86_64 34/166 ", " Upgrading : python3-libstoragemgmt-1.8.1-2.el8.noarch 35/166 ", " Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Upgrading : libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Upgrading : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 37/166 ", " Upgrading : subscription-manager-1.25.17-1.el8.x86_64 38/166 ", " Running scriptlet: subscription-manager-1.25.17-1.el8.x86_64 38/166 ", " Upgrading : subscription-manager-cockpit-1.25.17-1.el8.noarc 39/166 ", " Running scriptlet: sssd-ipa-2.2.0-19.el8.x86_64 40/166 ", " Upgrading : sssd-ipa-2.2.0-19.el8.x86_64 40/166 ", " Upgrading 2020-02-05T16:52:14.768 INFO:teuthology.task.ansible.out: : sssd-ad-2.2.0-19.el8.x86_64 41/166 ", " Upgrading : device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 ", " Running scriptlet: device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 ", " Upgrading : lvm2-libs-8:2.03.05-5.el8.x86_64 43/166 ", " Upgrading : lvm2-8:2.03.05-5.el8.x86_64 44/166 ", " Running scriptlet: lvm2-8:2.03.05-5.el8.x86_64 44/166 ", "warning: /etc/lvm/lvm.conf created as /etc/lvm/lvm.conf.rpmnew", "", " Upgrading : sssd-ldap-2.2.0-19.el8.x86_6 2020-02-05T16:52:14.769 INFO:teuthology.task.ansible.out:4 45/166 ", " Upgrading : sssd-krb5-2.2.0-19.el8.x86_64 46/166 ", " Running scriptlet: sssd-proxy-2.2.0-19.el8.x86_64 47/166 ", " Upgrading : sssd-proxy-2.2.0-19.el8.x86_64 47/166 ", " Upgrading : vim-common-2:8.0.1763-13.el8.x86_64 48/166 ", " Upgrading : vim-enhanced-2:8.0.1763-13.el8.x86_64 49/166 ", " Upgrading : sssd-2.2.0-19.el8.x86_64 50/166 ", " Upgrading : vdo-6.2.1.134-11.el8.x86_64 51/166 ", " Running scriptlet: vdo-6.2.1.134-11.el8.x86_64 51/166 ", " Upgrading : cockpit-196.3-1.el8.x86_64 52/166 ", " Upgrading : subscription-manager-plugin-container-1.25.17-1. 53/166 ", " Running scriptlet: subscription-manager-plugin-container-1.25.17-1. 53/166 ", " Upgrading : rsyslog-gnutls-8.37.0-13.el8.x86_64 54/166 ", " Upgrading : rsyslog-gssapi-8.37.0-13.el8.x86_64 55/166 ", " Upgrading : rsyslog-relp-8.37.0-13.el8.x86_64 56/166 ", " Upgrading : sssd-kcm-2.2.0-19.el8.x86_64 57/166 ", " Running scriptlet: sssd-kcm-2.2.0-19.el8.x86_64 57/166 ", " Upgrading : smartmontools-1:7.1-7.el8.x86_64 58/166 ", " Running scriptlet: smartmontools-1:7.1-7.el8.x86_64 58/166 ", " Upgrading : insights-client-3.0.8-4.el8_1.noarch 59/166 ", " Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 59/166 ", " Upgrading : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 ", " Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 ", " Upgrading : mdadm-4.1-9.el8.x86_64 61/166 ", " Running scriptlet: mdadm-4.1-9. 2020-02-05T16:52:14.771 INFO:teuthology.task.ansible.out:el8.x86_64 61/166 ", " Upgrading : mcelog-3:162-2.el8.x86_64 62/166 ", " Running scriptlet: mcelog-3:162-2.el8.x86_64 62/166 ", " Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 ", " Upgrading : chrony-3.5-1.el8.x86_64 63/166 ", " Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 ", " Upgrading : systemd-udev-239-18.el8_1.2.x86_64 64/166 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 64/166 ", " Upgrading : cryptsetup-2.2.0-2.el8.x86_64 65/166 ", " Upgrading : util-linux-user-2.32.1-17.el8.x86_64 66/166 ", " Upgrading : redhat-release-8.1-3.3.el8.x86_64 67/166 ", " Upgrading : teamd-1.28-4.el8.x86_64 68/166 ", " Upgrading : cyrus-sasl-plain-2.1.27-1.el8.x86_64 69/166 ", " Upgrading : plymouth-0.9.3-15.el8.x86_64 70/166 ", " Upgrading : ledmon-0.92-7.el8.x86_64 71/166 ", " Upgrading : wget-1.19.5-8.el8_1.1.x86_64 72/166 ", " Running scriptlet: wget-1.19.5-8.el8_1.1.x86_64 72/166 ", " Upgrading : bpftool-4.18.0-147.5.1.el8_1.x86_64 73/166 ", " Upgrading : blktrace-1.2.0-10.el8.x86_64 74/166 ", " Upgrading : mtr-2:0.92-3.el8.x86_64 75/166 ", " Upgrading : strace-4.24-5.el8.x86_64 76/166 ", " Upgrading : sudo-1.8.25p1-8.el8_1.x86_64 77/166 ", " Running scriptlet: sudo-1.8.25p1-8.el8_1.x86_64 77/166 ", " Upgrading : sos-3.7-8.el8_1.noarch 78/166 ", " Upgrading 2020-02-05T16:52:14.772 INFO:teuthology.task.ansible.out: : ethtool-2:5.0-2.el8.x86_64 79/166 ", " Upgrading : irqbalance-2:1.4.0-4.el8.x86_64 80/166 ", " Running scriptlet: irqbalance-2:1.4.0-4.el8.x86_64 80/166 ", " Upgrading : rsync-3.1.3-6.el8.x86_64 81/166 ", " Upgrading : kpatch-0.6.1-5.el8.noarch 82/166 ", " Upgrading : nmap-ncat-2:7.70-5.el8.x86_64 83/166 ", " Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 83/166 ", " Cleanup : sssd-2.0.0-43.el8_0.3.x86_64 84/166 ", " Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Cleanup : insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Cleanup : sssd-ipa-2.0.0-43.el8_0.3.x86_64 86/166 ", " Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Cleanup : sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Cleanup : sssd-ad-2.0.0-43.el8_0.3.x86_64 88/166 ", " Cleanup : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 89/166 ", " Cleanup : sssd-ldap-2.0.0-43.el8_0.3.x86_64 90/166 ", " Cleanup : systemd-udev-239-13.el8_0.5.x86_64 91/166 ", " Running scriptlet: systemd-udev-239-13.el8_0.5.x86_64 91/166 ", " Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Cleanup : vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 93/166 ", " 2020-02-05T16:52:14.774 INFO:teuthology.task.ansible.out: Cleanup : lvm2-8:2.03.02-6.el8.x86_64 93/166 ", " Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 93/166 ", " Cleanup : lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 ", " Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 ", " Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 ", " Cleanup : device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 ", " Cleanup : sssd-krb5-2.0.0-43.el8_0.3.x86_64 96/166 ", " Cleanup : sssd-proxy-2.0.0-43.el8_0.3.x86_64 97/166 ", " Cleanup : cryptsetup-2.0.6-1.el8.x86_64 98/166 ", " Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Cleanup : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Cleanup : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 100/166 ", " Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Cleanup : sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Cleanup : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 ", " Running scriptlet: device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 ", " Cleanup : teamd-1.27-10.el8.x86_64 103/166 ", " Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 ", " Cleanup : chrony-3.3-3.el8.x86_64 104/166 ", " Running scriptlet: chrony-3.3-3.el8.x86_64 104/1 2020-02-05T16:52:14.774 INFO:teuthology.task.ansible.out:66 ", " Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 ", " Cleanup : mcelog-3:159-1.el8.x86_64 2020-02-05T16:52:14.776 INFO:teuthology.task.ansible.out: 105/166 ", " Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 ", " Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 ", " Cleanup : mdadm-4.1-4.el8.x86_64 106/166 ", " Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 ", " Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Cleanup : smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Cleanup : python3-libstoragemgmt-1.6.2-9.el8.noarch 108/166 ", " Cleanup : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 109/166 ", " Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Cleanup : libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Cleanup : sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Cleanup : systemd-pam-239-13.el8_0.5.x86_64 112/166 ", " Cleanup : ledmon-0.90-3.el8.x86_64 113/166 ", " Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 114/166 ", " Cleanup : wget-1.19.5-7.el8_0.1.x86_64 114/166 ", " Cleanup : subscription-manager-plugin-container-1.23.8-35. 115/166 ", " Cleanup : cockpit-185.1-1.el8_0.x86_64 116/166 ", " Cleanup : subscription-manager-cockpit-1.23.8-35.el8.noarc 117/166 ", " Cleanup : redhat-release-8.0-0.44.el8.x86_64 118/166 ", " Running scriptlet: subscription-manager-1.23.8-35.e 2020-02-05T16:52:14.778 INFO:teuthology.task.ansible.out:l8.x86_64 119/166 ", " Cleanup : subscription-manager-1.23.8-35.el8.x86_64 119/166 ", " Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 ", " Cleanup : python3-subscription-manager-rhsm-1.23.8-35.el8. 120/166 ", " Cleanup : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 121/166 ", " Cleanup : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 122/166 ", " Cleanup : util-linux-user-2.32.1-8.el8.x86_64 123/166 ", " Cleanup : rsyslog-relp-8.37.0-9.el8.x86_64 124/166 ", " Cleanup : plymouth-0.9.3-12.el8.x86_64 125/166 ", " Running scriptlet: plymouth-0.9.3-12.el8.x86_64 125/166 ", " Cleanup : rsyslog-gssapi-8.37.0-9.el8.x86_64 126/166 ", " Cleanup : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 127/166 ", " Cleanup : rsyslog-gnutls-8.37.0-9.el8.x86_64 128/166 ", " Cleanup : subscription-manager-rhsm-certificates-1.23.8-35 129/166 ", " Cleanup : redhat-release-eula-8.0-0.44.el8.x86_64 130/166 ", " Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Cleanup : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Cleanup : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 132/166 ", " Cleanup : sos-3.6-10.el8_0.3.noarch 133/166 ", " Cleanup : kpatch-0.6.1-1.el8.noarch 134/166 ", " Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Cleanup : rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Cleanup : cryptsetup-libs- 2020-02-05T16:52:14.779 INFO:teuthology.task.ansible.out:2.0.6-1.el8.x86_64 136/166 ", " Running scriptlet: cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 ", " Cleanup : device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 ", " Running scriptlet: device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 ", " Cleanup : device-mapper-8:1.02.155-6.el8.x86_64 138/166 ", " Cleanup : dbus-1:1.12.8-7.el8.x86_64 139/166 ", " Running scriptlet: systemd-239-13.el8_0.5.x86_64 140/166 ", " Cleanup : systemd-239-13.el8_0.5.x86_64 140/166 ", " Cleanup : util-linux-2.32.1-8.el8.x86_64 141/166 ", " Cleanup : libfdisk-2.32.1-8.el8.x86_64 142/166 ", " Running scriptlet: libfdisk-2.32.1-8.el8.x86_64 142/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Cleanup : dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Cleanup : dbus-tools-1:1.12.8-7.el8.x86_64 144/166 ", " Cleanup : dbus-common-1:1.12.8-7.el8.noarch 145/166 ", " Cleanup : dbus-libs-1:1.12.8-7.el8.x86_64 146/166 ", " Running scriptlet: dbus-libs-1:1.12.8-7.el8.x86_64 146/166 ", " Cleanup : systemd-libs-239-13.el8_0.5.x86_64 147/166 ", " Cleanup : libmount-2.32.1-8.el8.x86_64 148/166 ", " Running scriptlet: libmount-2.32.1-8.el8.x86_64 148/166 ", " Cleanup : libblkid-2.32.1-8.el8.x86_64 149/166 ", " Running scriptlet: libblkid-2.32.1-8.el8.x86_64 149/166 ", " Cleanup : libuuid-2.32.1-8.el8.x86_64 150/166 ", " Running scriptlet: 2020-02-05T16:52:14.781 INFO:teuthology.task.ansible.out:libuuid-2.32.1-8.el8.x86_64 150/166 ", " Cleanup : libsmartcols-2.32.1-8.el8.x86_64 151/166 ", " Running scriptlet: libsmartcols-2.32.1-8.el8.x86_64 151/166 ", " Cleanup : vim-common-2:8.0.1763-11.el8_0.x86_64 152/166 ", " Cleanup : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 ", " Running scriptlet: cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 ", " Cleanup : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 154/166 ", " Cleanup : libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 ", " Running scriptlet: libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 ", " Cleanup : libteam-1.27-10.el8.x86_64 156/166 ", " Running scriptlet: libteam-1.27-10.el8.x86_64 156/166 ", " Cleanup : libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 ", " Running scriptlet: libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 ", " Cleanup : bpftool-4.18.0-80.11.2.el8_0.x86_64 158/166 ", " Cleanup : blktrace-1.2.0-9.el8.x86_64 159/166 ", " Cleanup : mtr-2:0.92-1.el8.x86_64 160/166 ", " Cleanup : strace-4.24-3.el8.x86_64 161/166 ", " Cleanup : sudo-1.8.25p1-4.el8_0.1.x86_64 162/166 ", " Cleanup : ethtool-2:4.16-1.el8.x86_64 163/166 ", " Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 ", " Cleanup : irqbalance-2:1.4.0-2.el8.x86_64 164/166 ", " Running scriptlet: irqbalance-2 2020-02-05T16:52:14.781 INFO:teuthology.task.ansible.out::1.4.0-2.el8.x86_64 164/166 ", " Cleanup : rsync-3.1.3-4.el8.x86_64 165/166 ", " Cleanup : nmap-ncat-2:7.70-4.el8.x86_64 166/166 ", " Run 2020-02-05T16:52:14.783 INFO:teuthology.task.ansible.out:ning scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 166/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 166/166 ", " Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 166/166 ", " Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 166/166 ", " Running scriptlet: nmap-ncat-2:7.70-4.el8.x86_64 166/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 166/166 ", " Running scriptlet: vim-common-2:8.0.1763-13.el8.x86_64 166/166 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 166/166 ", " Verifying : smartmontools-1:7.1-7.el8.x86_64 1/166 ", " Verifying : smartmontools-1:6.6-3.el8.x86_64 2/166 ", " Verifying : rsyslog-8.37.0-13.el8.x86_64 3/166 ", " Verifying : rsyslog-8.37.0-9.el8.x86_64 4/166 ", " Verifying : insights-client-3.0.8-4.el8_1.noarch 5/166 ", " Verifying : insights-client-3.0.8-0.el8_0.noarch 6/166 ", " Verifying : wget-1.19.5-8.el8_1.1.x86_64 7/166 ", " Verifying : wget-1.19.5-7.el8_0.1.x86_64 8/166 ", " Verifying : rsyslog-gnutls-8.37.0-13.el8.x86_64 9/166 ", " Verifying : rsyslog-gnutls-8.37.0-9.el8.x86_64 10/166 ", " Verifying : vim-common-2:8.0.1763-13.el8.x86_64 11/166 ", " Verifying : vim-common-2:8.0.1763-11.el8_0.x86_64 12/166 ", " Verifying : vim-enhanced-2:8.0.1763-13.el8.x86_64 13/166 ", " Verifying : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 14/166 ", " Verifying : rsyslog-gssapi-8.37.0-13.el8.x86_64 15/166 ", " Verifying : rsyslog-gssapi-8.37.0-9.el8.x86_64 2020-02-05T16:52:14.785 INFO:teuthology.task.ansible.out:16/166 ", " Verifying : nmap-ncat-2:7.70-5.el8.x86_64 17/166 ", " Verifying : nmap-ncat-2:7.70-4.el8.x86_64 18/166 ", " Verifying : plymouth-0.9.3-15.el8.x86_64 19/166 ", " Verifying : plymouth-0.9.3-12.el8.x86_64 20/166 ", " Verifying : rsyslog-relp-8.37.0-13.el8.x86_64 21/166 ", " Verifying : rsyslog-relp-8.37.0-9.el8.x86_64 22/166 ", " Verifying : sssd-ldap-2.2.0-19.el8.x86_64 23/166 ", " Verifying : sssd-ldap-2.0.0-43.el8_0.3.x86_64 24/166 ", " Verifying : dbus-common-1:1.12.8-9.el8.noarch 25/166 ", " Verifying : dbus-common-1:1.12.8-7.el8.noarch 26/166 ", " Verifying : ledmon-0.92-7.el8.x86_64 27/166 ", " Verifying : ledmon-0.90-3.el8.x86_64 28/166 ", " Verifying : cryptsetup-2.2.0-2.el8.x86_64 29/166 ", " Verifying : cryptsetup-2.0.6-1.el8.x86_64 30/166 ", " Verifying : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 31/166 ", " Verifying : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 32/166 ", " Verifying : util-linux-2.32.1-17.el8.x86_64 33/166 ", " Verifying : util-linux-2.32.1-8.el8.x86_64 34/166 ", " Verifying : lvm2-8:2.03.05-5.el8.x86_64 35/166 ", " Verifying : lvm2-8:2.03.02-6.el8.x86_64 36/166 ", " Verifying : util-linux-user-2.32.1-17.el8.x86_64 37/166 ", " Verifying : util-linux-user-2.32.1-8.el8.x86_64 38/166 ", " Verifying : kpatch-0.6.1-5.el8.noarch 39/166 ", " Verifying : kpatch-0.6.1-1.el8.noarch 2020-02-05T16:52:14.786 INFO:teuthology.task.ansible.out: 40/166 ", " Verifying : dbus-daemon-1:1.12.8-9.el8.x86_64 41/166 ", " Verifying : dbus-daemon-1:1.12.8-7.el8.x86_64 42/166 ", " Verifying : redhat-release-8.1-3.3.el8.x86_64 43/166 ", " Verifying : redhat-release-8.0-0.44.el8.x86_64 44/166 ", " Verifying : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 45/166 ", " Verifying : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 46/166 ", " Verifying : libmount-2.32.1-17.el8.x86_64 47/166 ", " Verifying : libmount-2.32.1-8.el8.x86_64 48/166 ", " Verifying : libsmartcols-2.32.1-17.el8.x86_64 49/166 ", " Verifying : libsmartcols-2.32.1-8.el8.x86_64 50/166 ", " Verifying : python3-sssdconfig-2.2.0-19.el8.noarch 51/166 ", " Verifying : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 52/166 ", " Verifying : subscription-manager-cockpit-1.25.17-1.el8.noarc 53/166 ", " Verifying : subscription-manager-cockpit-1.23.8-35.el8.noarc 54/166 ", " Verifying : python3-libstoragemgmt-1.8.1-2.el8.noarch 55/166 ", " Verifying : python3-libstoragemgmt-1.6.2-9.el8.noarch 56/166 ", " Verifying : libstoragemgmt-1.8.1-2.el8.x86_64 57/166 ", " Verifying : libstoragemgmt-1.6.2-9.el8.x86_64 58/166 ", " Verifying : libuuid-2.32.1-17.el8.x86_64 59/166 ", " Verifying : libuuid-2.32.1-8.el8.x86_64 60/166 ", " Verifying : dbus-libs-1:1.12.8-9.el8.x86_64 61/166 ", " Verifying : dbus-libs-1:1.12.8-7.el8.x86_64 62/166 ", " Verifying : rsync-3.1.3-6.el8.x86_64 63/166 ", " Verifying : rsync-3.1.3-4.el8.x8 2020-02-05T16:52:14.788 INFO:teuthology.task.ansible.out:6_64 64/166 ", " Verifying : redhat-release-eula-8.1-3.3.el8.x86_64 65/166 ", " Verifying : redhat-release-eula-8.0-0.44.el8.x86_64 66/166 ", " Verifying : irqbalance-2:1.4.0-4.el8.x86_64 67/166 ", " Verifying : irqbalance-2:1.4.0-2.el8.x86_64 68/166 ", " Verifying : ethtool-2:5.0-2.el8.x86_64 69/166 ", " Verifying : ethtool-2:4.16-1.el8.x86_64 70/166 ", " Verifying : libipa_hbac-2.2.0-19.el8.x86_64 71/166 ", " Verifying : libipa_hbac-2.0.0-43.el8_0.3.x86_64 72/166 ", " Verifying : device-mapper-event-8:1.02.163-5.el8.x86_64 73/166 ", " Verifying : device-mapper-event-8:1.02.155-6.el8.x86_64 74/166 ", " Verifying : libteam-1.28-4.el8.x86_64 75/166 ", " Verifying : libteam-1.27-10.el8.x86_64 76/166 ", " Verifying : sos-3.7-8.el8_1.noarch 77/166 ", " Verifying : sos-3.6-10.el8_0.3.noarch 78/166 ", " Verifying : sudo-1.8.25p1-8.el8_1.x86_64 79/166 ", " Verifying : sudo-1.8.25p1-4.el8_0.1.x86_64 80/166 ", " Verifying : dbus-1:1.12.8-9.el8.x86_64 81/166 ", " Verifying : dbus-1:1.12.8-7.el8.x86_64 82/166 ", " Verifying : sssd-ipa-2.2.0-19.el8.x86_64 83/166 ", " Verifying : sssd-ipa-2.0.0-43.el8_0.3.x86_64 84/166 ", " Verifying : sssd-2.2.0-19.el8.x86_64 85/166 ", " Verifying : sssd-2.0.0-43.el8_0.3.x86_64 86/166 ", " Verifying : dbus-tools-1:1.12.8-9.el8.x86_64 87/166 ", " Verifying : dbus 2020-02-05T16:52:14.789 INFO:teuthology.task.ansible.out:-tools-1:1.12.8-7.el8.x86_64 88/166 ", " Verifying : sssd-common-2.2.0-19.el8.x86_64 89/166 ", " Verifying : sssd-common-2.0.0-43.el8_0.3.x86_64 90/166 " 2020-02-05T16:52:14.790 INFO:teuthology.task.ansible.out:, " Verifying : mdadm-4.1-9.el8.x86_64 91/166 ", " Verifying : mdadm-4.1-4.el8.x86_64 92/166 ", " Verifying : libsss_idmap-2.2.0-19.el8.x86_64 93/166 ", " Verifying : libsss_idmap-2.0.0-43.el8_0.3.x86_64 94/166 ", " Verifying : teamd-1.28-4.el8.x86_64 95/166 ", " Verifying : teamd-1.27-10.el8.x86_64 96/166 ", " Verifying : strace-4.24-5.el8.x86_64 97/166 ", " Verifying : strace-4.24-3.el8.x86_64 98/166 ", " Verifying : cockpit-196.3-1.el8.x86_64 99/166 ", " Verifying : cockpit-185.1-1.el8_0.x86_64 100/166 ", " Verifying : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 101/166 ", " Verifying : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 102/166 ", " Verifying : sssd-client-2.2.0-19.el8.x86_64 103/166 ", " Verifying : sssd-client-2.0.0-43.el8_0.3.x86_64 104/166 ", " Verifying : cryptsetup-libs-2.2.0-2.el8.x86_64 105/166 ", " Verifying : cryptsetup-libs-2.0.6-1.el8.x86_64 106/166 ", " Verifying : mcelog-3:162-2.el8.x86_64 107/166 ", " Verifying : mcelog-3:159-1.el8.x86_64 108/166 ", " Verifying : vdo-6.2.1.134-11.el8.x86_64 109/166 ", " Verifying : vdo-6.2.0.298-10.el8_0.x86_64 110/166 ", " Verifying : sssd-common-pac-2.2.0-19.el8.x86_64 111/166 ", " Verifyi 2020-02-05T16:52:14.792 INFO:teuthology.task.ansible.out:ng : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 112/166 ", " Verifying : subscription-manager-rhsm-certificates-1.25.17-1 113/166 ", " Verifying : subscription-manager-rhsm-certificates-1.23.8-35 114/166 ", " Verifying : libblkid-2.32.1-17.el8.x86_64 115/166 ", " Verifying : libblkid-2.32.1-8.el8.x86_64 116/166 ", " Verifying : mtr-2:0.92-3.el8.x86_64 117/166 ", " Verifying : mtr-2:0.92-1.el8.x86_64 118/166 ", " Verifying : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 119/166 ", " Verifying : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 120/166 ", " Verifying : cyrus-sasl-lib-2.1.27-1.el8.x86_64 121/166 ", " Verifying : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 122/166 ", " Verifying : device-mapper-8:1.02.163-5.el8.x86_64 123/166 ", " Verifying : device-mapper-8:1.02.155-6.el8.x86_64 124/166 ", " Verifying : blktrace-1.2.0-10.el8.x86_64 125/166 ", " Verifying : blktrace-1.2.0-9.el8.x86_64 126/166 ", " Verifying : libfdisk-2.32.1-17.el8.x86_64 127/166 ", " Verifying : libfdisk-2.32.1-8.el8.x86_64 128/166 ", " Verifying : sssd-proxy-2.2.0-19.el8.x86_64 129/166 ", " Verifying : sssd-proxy-2.0.0-43.el8_0.3.x86_64 130/166 ", " Verifying : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 131/166 ", " Verifying : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 132/166 ", " Verifying : kmod-kvdo-6.2.1.138-57.el8.x86_64 133/166 ", " Verifying : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 134/166 ", " Verifying : python3-subscription-manager-rhsm-1.25.17-1.el8. 135/1 2020-02-05T16:52:14.793 INFO:teuthology.task.ansible.out:66 ", " Verifying : python3-subscription-manager-rhsm-1.23.8-35.el8. 136/166 ", " Verifying : subscription-manager-1.25.17-1.el8.x86_64 137/166 ", " Verifying : subscription-manager-1.23.8-35.el8.x86_64 138/166 ", " Verifying : sssd-kcm-2.2.0-19.el8.x86_64 139/166 ", " Verifying : sssd-kcm-2.0.0-43.el8_0.3.x86_64 140/166 ", " Verifying : sssd-krb5-common-2.2.0-19.el8.x86_64 141/166 ", " Verifying : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 142/166 ", " Verifying : chrony-3.5-1.el8.x86_64 143/166 ", " Verifying : chrony-3.3-3.el8.x86_64 144/166 ", " Verifying : cyrus-sasl-plain-2.1.27-1.el8.x86_64 145/166 ", " Verifying : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 146/166 ", " Verifying : sssd-krb5-2.2.0-19.el8.x86_64 147/166 ", " Verifying : sssd-krb5-2.0.0-43.el8_0.3.x86_64 148/166 ", " Verifying : sssd-ad-2.2.0-19.el8.x86_64 149/166 ", " Verifying : sssd-ad-2.0.0-43.el8_0.3.x86_64 150/166 ", " Verifying : lvm2-libs-8:2.03.05-5.el8.x86_64 151/166 ", " Verifying : lvm2-libs-8:2.03.02-6.el8.x86_64 152/166 ", " Verifying : subscription-manager-plugin-container-1.25.17-1. 153/166 ", " Verifying : subscription-manager-plugin-container-1.23.8-35. 154/166 ", " Verifying : device-mapper-libs-8:1.02.163-5.el8.x86_64 155/166 ", " Verifying : device-mapper-libs-8:1.02.155-6.el8.x86_64 156/166 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 157/166 ", " Verifying : systemd-libs-239-13.el8_0.5.x86_64 158/166 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 2020-02-05T16:52:14.795 INFO:teuthology.task.ansible.out: 159/166 ", " Verifying : systemd-pam-239-13.el8_0.5.x86_64 160/166 ", " Verifying : systemd-239-18.el8_1.2.x86_64 161/166 ", " Verifying : systemd-239-13.el8_0.5.x86_64 162/166 ", " Verifying : bpftool-4.18.0-147.5.1.el8_1.x86_64 163/166 ", " Verifying : bpftool-4.18.0-80.11.2.el8_0.x86_64 164/166 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 165/166 ", " Verifying : systemd-udev-239-13.el8_0.5.x86_64 166/166 ", "Installed products updated.", "", "Upgraded:", " smartmontools-1:7.1-7.el8.x86_64 ", " rsyslog-8.37.0-13.el8.x86_64 ", " insights-client-3.0.8-4.el8_1.noarch ", " wget-1.19.5-8.el8_1.1.x86_64 ", " rsyslog-gnutls-8.37.0-13.el8.x86_64 ", " vim-common-2:8.0.1763-13.el8.x86_64 ", " vim-enhanced-2:8.0.1763-13.el8.x86_64 ", " rsyslog-gssapi-8.37.0-13.el8.x86_64 ", " nmap-ncat-2:7.70-5.el8.x86_64 ", " plymouth-0.9.3-15.el8.x86_64 ", " rsyslog-relp-8.37.0-13.el8.x86_64 ", " sssd-ldap-2.2.0-19.el8.x86_64 ", " dbus-common-1:1.12.8-9.el8.noarch ", " ledmon-0.92-7.el8.x86_64 ", " cryptsetup-2.2.0-2.el8.x86_64 ", " microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_64 " 2020-02-05T16:52:14.797 INFO:teuthology.task.ansible.out:, " util-linux-2.32.1-17.el8.x86_64 ", " lvm2-8:2.03.05-5.el8.x86_64 2020-02-05T16:52:14.797 INFO:teuthology.task.ansible.out: ", " util-linux-user-2.32.1-17.el8.x86_64 ", " kpatch-0.6.1-5.el8.noarch ", " dbus-daemon-1:1.12.8-9.el8.x86_64 ", " redhat-release-8.1-3.3.el8.x86_64 ", " device-mapper-event-libs-8:1.02.163-5.el8.x86_64 ", " libmount-2.32.1-17.el8.x86_64 ", " libsmartcols-2.32.1-17.el8.x86_64 ", " python3-sssdconfig-2.2.0-19.el8.noarch ", " subscription-manager-cockpit-1.25.17-1.el8.noarch ", " python3-libstoragemgmt-1.8.1-2.el8.noarch ", " libstoragemgmt-1.8.1-2.el8.x86_64 ", " libuuid-2.32.1-17.el8.x86_64 ", " dbus-libs-1:1.12.8-9.el8.x86_64 ", " rsync-3.1.3-6.el8.x86_64 ", " redhat-release-eula-8.1-3.3.el8.x86_64 ", " irqbalance-2:1.4.0-4.el8.x86_64 ", " ethtool-2:5.0-2.el8.x86_64 ", " libipa_hbac-2.2.0-19.el8.x86_64 ", " device-mapper-event-8:1.02.163-5.el8.x86_64 ", " libteam-1.28-4.el8.x86_64 ", " sos-3.7-8.el8_1.noarch ", " sudo-1.8.25p1-8.el8_1.x86_64 2020-02-05T16:52:14.798 INFO:teuthology.task.ansible.out: ", " dbus-1:1.12.8-9.el8.x86_64 ", " sssd-ipa-2.2.0-19.el8.x86_64 ", " sssd-2.2.0-19.el8.x86_64 ", " dbus-tools-1:1.12.8-9.el8.x86_64 ", " sssd-common-2.2.0-19.el8.x86_64 ", " mdadm-4.1-9.el8.x86_64 ", " libsss_idmap-2.2.0-19.el8.x86_64 ", " teamd-1.28-4.el8.x86_64 ", " strace-4.24-5.el8.x86_64 ", " cockpit-196.3-1.el8.x86_64 ", " python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 ", " sssd-client-2.2.0-19.el8.x86_64 ", " cryptsetup-libs-2.2.0-2.el8.x86_64 ", " mcelog-3:162-2.el8.x86_64 ", " vdo-6.2.1.134-11.el8.x86_64 ", " sssd-common-pac-2.2.0-19.el8.x86_64 ", " subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64 ", " libblkid-2.32.1-17.el8.x86_64 ", " mtr-2:0.92-3.el8.x86_64 ", " cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 ", " cyrus-sasl-lib-2.1.27-1.el8.x86_64 ", " device-mapper-8:1.02.163-5.el8.x86_64 ", " blktrace-1.2.0-10.el8.x86_64 ", " libfdisk-2.32.1-17.el8.x86_64 2020-02-05T16:52:14.800 INFO:teuthology.task.ansible.out: ", " sssd-proxy-2.2.0-19.el8.x86_64 ", " dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64 ", " kmod-kvdo-6.2.1.138-57.el8.x86_64 ", " python3-subscription-manager-rhsm-1.25.17-1.el8.x86_64 ", " subscription-manager-1.25.17-1.el8.x86_64 ", " sssd-kcm-2.2.0-19.el8.x86_64 ", " sssd-krb5-common-2.2.0-19.el8.x86_64 ", " chrony-3.5-1.el8.x86_64 ", " cyrus-sasl-plain-2.1.27-1.el8.x86_64 ", " sssd-krb5-2.2.0-19.el8.x86_64 ", " sssd-ad-2.2.0-19.el8.x86_64 ", " lvm2-libs-8:2.03.05-5.el8.x86_64 ", " subscription-manager-plugin-container-1.25.17-1.el8.x86_64 ", " device-mapper-libs-8:1.02.163-5.el8.x86_64 ", " systemd-libs-239-18.el8_1.2.x86_64 ", " systemd-pam-239-18.el8_1.2.x86_64 ", " systemd-239-18.el8_1.2.x86_64 ", " bpftool-4.18.0-147.5.1.el8_1.x86_64 ", " systemd-udev-239-18.el8_1.2.x86_64 ", "", "Complete!"]} 2020-02-05T16:52:14.839 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:52:14 +0000 (0:02:44.151) 0:08:02.169 **** 2020-02-05T16:52:29.837 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:02:58.777342", "end": "2020-02-05 16:52:29.774925", "rc": 0, "start": "2020-02-05 16:49:30.997583", "stderr": "warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/smartmontools-7.1-7.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY\nImporting GPG key 0x2A8A41EC:\n Userid : \"ktdreyer_ceph-el8 (None) \"\n Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC\n From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg", "stderr_lines": ["warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8-0161c520b1c9fbf7/packages/smartmontools-7.1-7.el8.x86_64.rpm: Header V3 RSA/SHA1 Signature, key ID 2a8a41ec: NOKEY", "Importing GPG key 0x2A8A41EC:", " Userid : \"ktdreyer_ceph-el8 (None) \"", " Fingerprint: 64D3 346F FD63 5D2D B5D0 5873 1014 BDBE 2A8A 41EC", " From : https://download.copr.fedorainfracloud.org/results/ktdreyer/ceph-el8/pubkey.gpg"], "stdout": "Updating Subscription Management repositories.\nExtra Packages for Enterprise Linux 5.2 kB/s | 5.3 kB 00:01 \nlab-extras 856 kB/s | 2.9 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - AppStre 65 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 51 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n==== 2020-02-05T16:52:29.842 INFO:teuthology.task.ansible.out:============================================================================\n\nComplete!\nUpdating Subscription Management repositories.\nRed Hat Enterprise Linux 8 for x86_64 - AppStre 70 kB/s | 2.8 kB 00:00 \nRed Hat Enterprise Linux 8 for x86_64 - BaseOS 56 kB/s | 2.4 kB 00:00 \nDependencies resolved.\n================================================================================\n Package ArchVersion Repository Size\n================================================================================\nUpgrading:\n smartmontools x86_641:7.1-7.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 546 k\n rsyslog x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 691 k\n insights-client noarch3.0.8-4.el8_1 rhel-8-for-x86_64-appstream-rpms 899 k\n wget x86_641.19.5-8.el8_1.1 rhel-8-for-x86_64-appstream-rpms 735 k\n rsyslog-gnutls x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 27 k\n vim-common x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 6.3 M\n vim-enhanced x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 1.4 M\n rsyslog-gssapi x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 31 k\n nmap-ncat x86_642:7.70-5.el8 rhel-8-for-x86_64-appstream-rpms 237 k\n plymouth x86_640.9.3-15.el8 rhel-8-for-x86_64-appstream-rpms 114 k\n rsyslog-relp 2020-02-05T16:52:29.846 INFO:teuthology.task.ansible.out: x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 29 k\n sssd-ldap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 205 k\n dbus-common noarch1:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 45 k\n ledmon x86_640.92-7.el8 rhel-8-for-x86_64-baseos-rpms 74 k\n cryptsetup x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 177 k\n microcode_ctl x86_644:20190618-1.20191115.3.el8_1 rhel-8-for-x86_64-baseos-rpms 2.5 M\n util-linux x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M\n lvm2 x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M\n util-linux-user x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 98 k\n kpatch noarch0.6.1-5.el8 rhel-8-for-x86_64-baseos-rpms 15 k\n dbus-daemon x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 241 k\n redhat-release x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 38 k\n device-mapper-event-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 265 k\n libmount x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 231 k\n libsmartcols x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 175 k\n python3-sssdconfig 2020-02-05T16:52:29.848 INFO:teuthology.task.ansible.out: noarch2.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 116 k\n subscription-manager-cockpit noarch1.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M\n python3-libstoragemgmt noarch1.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 171 k\n libstoragemgmt x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 266 k\n libuuid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 94 k\n dbus-libs x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 184 k\n rsync x86_643.1.3-6.el8 rhel-8-for-x86_64-baseos-rpms 404 k\n redhat-release-eula x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 16 k\n irqbalance x86_642:1.4.0-4.el8 rhel-8-for-x86_64-baseos-rpms 55 k\n ethtool x86_642:5.0-2.el8 rhel-8-for-x86_64-baseos-rpms 151 k\n libipa_hbac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 99 k\n device-mapper-event x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 266 k\n libteam x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 64 k\n sos noarch3.7-8.el8_1 rhel-8-for-x86_64-baseos-rpms 510 k\n sudo x86_641.8.25p1-8.el8_1 rhel-8-for-x86_64-baseos-rpms 875 k\n dbus 2020-02-05T16:52:29.850 INFO:teuthology.task.ansible.out: x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 40 k\n sssd-ipa x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 323 k\n sssd x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 90 k\n dbus-tools x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 85 k\n sssd-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M\n mdadm x86_644.1-9.el8 rhel-8-for-x86_64-baseos-rpms 448 k\n libsss_idmap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 104 k\n teamd x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 129 k\n strace x86_644.24-5.el8 rhel-8-for-x86_64-baseos-rpms 971 k\n cockpit x86_64196.3-1.el8 rhel-8-for-x86_64-baseos-rpms 70 k\n python3-libstoragemgmt-clibs x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 27 k\n sssd-client x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 162 k\n cryptsetup-libs x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 422 k\n mcelog x86_643:162-2.el8 rhel-8-for-x86_64-baseos-rpms 79 k\n vdo x86_646.2.1.134-11.el8 rhel-8-for-x86_64-baseos-rpms 689 k\n sssd-common-pac 2020-02-05T16:52:29.851 INFO:teuthology.task.ansible.out: x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 161 k\n subscription-manager-rhsm-certificates x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 245 k\n libblkid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 215 k\n mtr x86_642:0.92-3.el8 rhel-8-for-x86_64-baseos-rpms 96 k\n cyrus-sasl-gssapi x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 49 k\n cyrus-sasl-lib x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 124 k\n device-mapper x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 371 k\n blktrace x86_641.2.0-10.el8 rhel-8-for-x86_64-baseos-rpms 150 k\n libfdisk x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 249 k\n sssd-proxy x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k\n dnf-plugin-subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 272 k\n kmod-kvdo x86_646.2.1.138-57.el8 rhel-8-for-x86_64-baseos-rpms 355 k\n python3-subscription-manager-rhsm x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 338 k\n subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n sssd-kcm x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 213 k\n sssd-krb 2020-02-05T16:52:29.853 INFO:teuthology.task.ansible.out:5-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 172 k\n chrony x86_643.5-1.el8 rhel-8-for-x86_64-baseos-rpms 271 k\n cyrus-sasl-plain x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k\n sssd-krb5 x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k\n sssd-ad x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 229 k\n lvm2-libs x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M\n subscription-manager-plugin-container x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 248 k\n device-mapper-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 403 k\n systemd-libs x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 560 k\n systemd-pam x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 2020-02-05T16:52:29.854 INFO:teuthology.task.ansible.out: 230 k\n systemd x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 3.5 M\n bpftool x86_644.18.0-147.5.1.el8_1 rhel-8-for-x86_64-baseos-rpms 2.1 M\n systemd-udev x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 1.3 M\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\nUpgrade 83 Packages\ 2020-02-05T16:52:29.855 INFO:teuthology.task.ansible.out:n\nTotal download size: 42 M\nDownloading Packages:\n(1/83): rsyslog-8.37.0-13.el8.x86_64.rpm 9.8 MB/s | 691 kB 00:00 \n(2/83): insights-client-3.0.8-4.el8_1.noarch.rp 9.1 MB/s | 899 kB 00:00 \n(3/83): smartmontools-7.1-7.el8.x86_64.rpm 5.0 MB/s | 546 kB 00:00 \n(4/83): wget-1.19.5-8.el8_1.1.x86_64.rpm 15 MB/s | 735 kB 00:00 \n(5/83): rsyslog-gnutls-8.37.0-13.el8.x86_64.rpm 525 kB/s | 27 kB 00:00 \n(6/83): vim-enhanced-8.0.1763-13.el8.x86_64.rpm 18 MB/s | 1.4 MB 00:00 \n(7/83): rsyslog-gssapi-8.37.0-13.el8.x86_64.rpm 592 kB/s | 31 kB 00:00 \n(8/83): nmap-ncat-7.70-5.el8.x86_64.rpm 5.8 MB/s | 237 kB 00:00 \n(9/83): plymouth-0.9.3-15.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 \n(10/83): rsyslog-relp-8.37.0-13.el8.x86_64.rpm 998 kB/s | 29 kB 00:00 \n(11/83): sssd-ldap-2.2.0-19.el8.x86_64.rpm 4.0 MB/s | 205 kB 00:00 \n(12/83): dbus-common-1.12.8-9.el8.noarch.rpm 977 kB/s | 45 kB 00:00 \n(13/83): ledmon-0.92-7.el8.x86_64.rpm 2.3 MB/s | 74 kB 00:00 \n(14/83): cryptsetup-2.2.0-2.el8.x86_64.rpm 4.7 MB/s | 177 kB 00:00 \n(15/83): vim-common-8.0.1763-13.el8.x86_64.rpm 23 MB/s | 6.3 MB 00:00 \n(16/83): microcode_ctl-20190618-1.20191115.3.el 21 MB/s | 2.5 MB 00:00 \n(17/83): lvm2-2.03.05-5.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 \n(18/83): util-linux-2.32.1-17.el8.x86_64.rpm 20 MB/s | 2.5 MB 00:00 \n(19/83): kpatch-0.6.1-5.el8.noarch.rpm 313 kB/s | 15 kB 00:00 \n(20/83): util-linux-user-2.32.1-17.el8.x86_64.r 1.7 MB/s | 98 kB 00:00 \n(21/83): dbus-daemon-1.12.8-9.el8.x86_64.rpm 3.7 MB/s | 241 kB 00:00 \n(22/83): device-mapper-event-libs-1.02.163-5.el 6.1 MB/s | 265 kB 00:00 \n(23/83): redhat-release-8.1-3.3.el8.x86_64.rpm 752 kB/s | 38 kB 00:00 \n(24/83): libmount-2.32.1-17.el8.x86_64.rpm 5.1 MB/s | 231 kB 00:00 \n(25 2020-02-05T16:52:29.856 INFO:teuthology.task.ansible.out:/83): libsmartcols-2.32.1-17.el8.x86_64.rpm 3.9 MB/s | 175 kB 00:00 \n(26/83): python3-sssdconfig-2.2.0-19.el8.noarch 2.1 MB/s | 116 kB 00:00 \n(27/83): python3-libstoragemgmt-1.8.1-2.el8.noa 4.4 MB/s | 171 kB 00:00 \n(28/83): subscription-manager-cockpit-1.25.17-1 16 MB/s | 1.0 MB 00:00 \n(29/83): libstoragemgmt-1.8.1-2.el8.x86_64.rpm 5.6 MB/s | 266 kB 00:00 \n(30/83): libuuid-2.32.1-17.el8.x86_64.rpm 2.7 MB/s | 94 kB 00:00 \n(31/83): dbus-libs-1.12.8-9.el8.x86_64.rpm 4.0 MB/s | 184 kB 00:00 \n(32/83): rsync-3.1.3-6.el8.x86_64.rpm 7.6 MB/s | 404 kB 00:00 \n(33/83): redhat-release-eula-8.1-3.3.el8.x86_64 303 kB/s | 16 kB 00:00 \n(34/83): irqbalance-1.4.0-4.el8.x86_64.rpm 1.8 MB/s | 55 kB 00:00 \n(35/83): ethtool-5.0-2.el8.x86_64.rpm 2.9 MB/s | 151 kB 00:00 \n(36/83): libipa_hbac-2.2.0-19.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 \n(37/83): device-mapper-event-1.02.163-5.el8.x86 4.9 MB/s | 266 kB 00:00 \n(38/83): libteam-1.28-4.el8.x86_64.rpm 2.2 MB/s | 64 kB 00:00 \n(39/83): sos-3.7-8.el8_1.noarch.rpm 8.0 MB/s | 510 kB 00:00 \n(40/83): dbus-1.12.8-9.el8.x86_64.rpm 753 kB/s | 40 kB 00:00 \n(41/83): sudo-1.8.25p1-8.el8_1.x86_64.rpm 14 MB/s | 875 kB 00:00 \n(42/83): sssd-ipa-2.2.0-19.el8.x86_64.rpm 6.4 MB/s | 323 kB 00:00 \n(43/83): sssd-2.2.0-19.el8.x86_64.rpm 1.8 MB/s | 90 kB 00:00 \n(44/83): dbus-tools-1.12.8-9.el8.x86_64.rpm 2.0 MB/s | 85 kB 00:00 \n(45/83): mdadm-4.1-9.el8.x86_64.rpm 8.0 MB/s | 448 kB 00:00 \n(46/83): libsss_idmap-2.2.0-19.el8.x86_64.rpm 2.1 MB/s | 104 kB 00:00 \n(47/83): teamd-1.28-4.el8.x86_64.rpm 4.3 MB/s | 129 kB 00:00 \n(48/83): sssd-common-2.2.0-19.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 \n(49/83): cockpit-196.3-1.el8.x86_64.rpm 1.7 MB/s | 2020-02-05T16:52:29.858 INFO:teuthology.task.ansible.out: 70 kB 00:00 \n(50/83): python3-libstoragemgmt-clibs-1.8.1-2.e 755 kB/s | 27 kB 00:00 \n(51/83): strace-4.24-5.el8.x86_64.rpm 12 MB/s | 971 kB 00:00 \n(52/83): sssd-client-2.2.0-19.el8.x86_64.rpm 4.0 MB/s | 162 kB 00:00 \n(53/83): cryptsetup-libs-2.2.0-2.el8.x86_64.rpm 7.4 MB/s | 422 kB 00:00 \n(54/83): mcelog-162-2.el8.x86_64.rpm 1.7 MB/s | 79 kB 00:00 \n(55/83): sssd-common-pac-2.2.0-19.el8.x86_64.rp 4.3 MB/s | 161 kB 00:00 \n(56/83): vdo-6.2.1.134-11.el8.x86_64.rpm 9.9 MB/s | 689 kB 00:00 \n(57/83): subscription-manager-rhsm-certificates 4.7 MB/s | 245 kB 00:00 \n(58/83): libblkid-2.32.1-17.el8.x86_64.rpm 6.0 MB/s | 215 kB 00:00 \n(59/83): mtr-0.92-3.el8.x86_64.rpm 2.5 MB/s | 96 kB 00:00 \n(60/83): cyrus-sasl-gssapi-2.1.27-1.el8.x86_64. 1.0 MB/s | 49 kB 00:00 \n(61/83): cyrus-sasl-lib-2.1.27-1.el8.x86_64.rpm 2.9 MB/s | 124 kB 00:00 \n(62/83): device-mapper-1.02.163-5.el8.x86_64.rp 5.6 MB/s | 371 kB 00:00 \n(63/83): blktrace-1.2.0-10.el8.x86_64.rpm 3.2 MB/s | 150 kB 00:00 \n(64/83): libfdisk-2.32.1-17.el8.x86_64.rpm 6.1 MB/s | 249 kB 00:00 \n(65/83): sssd-proxy-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 126 kB 00:00 \n(66/83): dnf-plugin-subscription-manager-1.25.1 4.4 MB/s | 272 kB 00:00 \n(67/83): kmod-kvdo-6.2.1.138-57.el8.x86_64.rpm 7.3 MB/s | 355 kB 00:00 \n(68/83): python3-subscription-manager-rhsm-1.25 5.3 MB/s | 338 kB 00:00 \n(69/83): subscription-manager-1.25.17-1.el8.x86 19 MB/s | 1.1 MB 00:00 \n(70/83): sssd-kcm-2.2.0-19.el8.x86_64.rpm 4.5 MB/s | 213 kB 00:00 \n(71/83): sssd-krb5-common-2.2.0-19.el8.x86_64.r 3.6 MB/s | 172 kB 00:00 \n(72/83): chrony-3.5-1.el8.x86_64.rpm 5.9 MB/s | 271 kB 00:00 \n(73/83): cyrus-sasl-plain-2.1.27-1.el8.x86_64.r 1.0 MB/s | 47 kB 00:00 \n(74/83): sssd-krb5-2.2.0-19.el8.x8 2020-02-05T16:52:29.859 INFO:teuthology.task.ansible.out:6_64.rpm 2.6 MB/s | 126 kB 00:00 \n(75/83): sssd-ad-2.2.0-19.el8.x86_64.rpm 4.8 MB/s | 229 kB 00:00 \n(76/83): subscription-manager-plugin-container- 6.3 MB/s | 248 kB 00:00 \n(77/83): lvm2-libs-2.03.05-5.el8.x86_64.rpm 13 MB/s | 1.1 MB 00:00 \n(78/83): device-mapper-libs-1.02.163-5.el8.x86_ 6.4 MB/s | 403 kB 00:00 \n(79/83): systemd-libs-239-18.el8_1.2.x86_64.rpm 10 MB/s | 560 kB 00:00 \n(80/83): systemd-pam-239-18.el8_1.2.x86_64.rpm 5.4 MB/s | 230 kB 00:00 \n(81/83): systemd-udev-239-18.el8_1.2.x86_64.rpm 19 MB/s | 1.3 MB 00:00 \n(82/83): bpftool-4.18.0-147.5.1.el8_1.x86_64.rp 24 MB/s | 2.1 MB 00:00 \n(83/83): systemd-239-18.el8_1.2.x86_64.rpm 28 MB/s | 3.5 MB 00:00 \n--------------------------------------------------------------------------------\nTotal 25 MB/s | 42 MB 00:01 \nCopr repo for ceph-el8 owned by ktdreyer 955 B/s | 1.0 kB 00:01 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 1/1 \n Preparing : 1/1 \n Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/1 \n Upgrading : libuuid-2.32.1-17.el8.x86_64 1/166 \n Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/166 \n Upgrading : libblkid-2.32.1-17.el8.x86_64 2/166 \n Running scriptlet: libblkid-2.32.1-17.el8.x86_64 2/166 \n Upgrading : libsss_idmap-2.2.0-19.el8.x86_64 3/166 \n Running scriptlet: libsss_idmap-2.2.0-19.el8.x86_64 3/166 \n Upgrading : libmount-2.32.1-17.el8.x86_64 4/166 \ 2020-02-05T16:52:29.861 INFO:teuthology.task.ansible.out:n Running scriptlet: libmount-2.32.1-17.el8.x86_64 4/166 \n Upgrading : systemd-libs-239-18.el8_1.2.x86_64 5/166 \n Running scriptlet: systemd-libs-239-18.el8_1.2.x86_64 5/166 \n Upgrading : dbus-libs-1:1.12.8-9.el8.x86_64 6/166 \n Running scriptlet: dbus-libs-1:1.12.8-9.el8.x86_64 6/166 \n Upgrading : cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 \n Running scriptlet: cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 \n Upgrading : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 8/166 \n Upgrading : dbus-tools-1:1.12.8-9.el8.x86_64 9/166 \n Upgrading : sssd-client-2.2.0-19.el8.x86_64 10/166 \n Running scriptlet: sssd-client-2.2.0-19.el8.x86_64 10/166 \n Upgrading : libfdisk-2.32.1-17.el8.x86_64 11/166 \n Running scriptlet: libfdisk-2.32.1-17.el8.x86_64 11/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Upgrading : kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 \n Upgrading : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 13/166 \n Upgrading : subscription-manager-rhsm-certificates-1.25.17-1 14/166 \n Upgrading : python3-subscription-manager-rhsm-1.25.17-1.el8. 15/166 \n Upgrading : libteam-1.28-4.el8.x86_64 16/166 \n Running scriptlet: libteam-1.28-4.el8.x86_64 16/166 \n Upgrading : libipa_hbac-2.2.0-19.el8.x86_64 17/166 \n Running scriptlet: libipa_hbac-2.2.0-19.el8.x86_64 17/166 \n Upgrading : redhat-release-eula-8.1-3.3.el8.x86_64 18/166 \n Upgrading : python3-ss 2020-02-05T16:52:29.862 INFO:teuthology.task.ansible.out:sdconfig-2.2.0-19.el8.noarch 19/166 \n Upgrading : libsmartcols-2.32.1-17.el8.x86_64 20/166 \n Running scriptlet: libsmartcols-2.32.1-17.el8.x86_64 20/166 \n Upgrading : util-linux-2.32.1-17.el8.x86_64 21/166 \n Running scriptlet: util-linux-2.32.1-17.el8.x86_64 21/166 \n Upgrading : dbus-common-1:1.12.8-9.el8.noarch 22/166 \n Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Upgrading : dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 \n Upgrading : device-mapper-8:1.02.163-5.el8.x86_64 24/166 \n Upgrading : device-mapper-libs-8:1.02.163-5.el8.x86_64 25/166 \n Upgrading : cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 \n Running scriptlet: cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 \n Upgrading : systemd-pam-239-18.el8_1.2.x86_64 27/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 \n Upgrading : systemd-239-18.el8_1.2.x86_64 28/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 \n Upgrading : dbus-1:1.12.8-9.el8.x86_64 29/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 \n Upgrading : sssd-common-2.2.0-19.el8.x86_64 2020-02-05T16:52:29.864 INFO:teuthology.task.ansible.out:30/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 \n Running scriptlet: sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 \n Upgrading : sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 \n Upgrading : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 32/166 \n Upgrading : rsyslog-8.37.0-13.el8.x86_64 33/166 \n Running scriptlet: rsyslog-8.37.0-13.el8.x86_64 33/166 \n Upgrading : sssd-common-pac-2.2.0-19.el8.x86_64 34/166 \n Upgrading : python3-libstoragemgmt-1.8.1-2.el8.noarch 35/166 \n Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Upgrading : libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 \n Upgrading : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 37/166 \n Upgrading : subscription-manager-1.25.17-1.el8.x86_64 38/166 \n Running scriptlet: subscription-manager-1.25.17-1.el8.x86_64 38/166 \n Upgrading : subscription-manager-cockpit-1.25.17-1.el8.noarc 39/166 \n Running scriptlet: sssd-ipa-2.2.0-19.el8.x86_64 40/166 \n Upgrading : sssd-ipa-2.2.0-19.el8.x86_64 40/166 \n Upgrading : sssd-ad-2.2.0-19.el8.x86_64 41/166 \n Upgrading : device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 \n Running scriptlet: device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 \n Upgrading : lvm2-libs-8:2.03.05-5.el8.x86_64 43/166 \n Upgrading : lvm2-8:2.03.05-5.el8.x86_64 44/166 \n Running scriptlet: lvm2-8:2.03.05-5.el8.x86_64 44/166 \nwarning: /etc/lvm/lvm.conf created as /etc/lvm/lvm.conf.rpmnew\n\n Upgrading : sssd-ldap-2.2.0-19 2020-02-05T16:52:29.866 INFO:teuthology.task.ansible.out:.el8.x86_64 45/166 \n Upgrading : sssd-krb5-2.2.0-19.el8.x86_64 46/166 \n Running scriptlet: sssd-proxy-2.2.0-19.el8.x86_64 47/166 \n Upgrading : sssd-proxy-2.2.0-19.el8.x86_64 47/166 \n Upgrading : vim-common-2:8.0.1763-13.el8.x86_64 48/166 \n Upgrading : vim-enhanced-2:8.0.1763-13.el8.x86_64 49/166 \n Upgrading : sssd-2.2.0-19.el8.x86_64 50/166 \n Upgrading : vdo-6.2.1.134-11.el8.x86_64 51/166 \n Running scriptlet: vdo-6.2.1.134-11.el8.x86_64 51/166 \n Upgrading : cockpit-196.3-1.el8.x86_64 52/166 \n Upgrading : subscription-manager-plugin-container-1.25.17-1. 53/166 \n Running scriptlet: subscription-manager-plugin-container-1.25.17-1. 53/166 \n Upgrading : rsyslog-gnutls-8.37.0-13.el8.x86_64 54/166 \n Upgrading : rsyslog-gssapi-8.37.0-13.el8.x86_64 55/166 \n Upgrading : rsyslog-relp-8.37.0-13.el8.x86_64 56/166 \n Upgrading : sssd-kcm-2.2.0-19.el8.x86_64 57/166 \n Running scriptlet: sssd-kcm-2.2.0-19.el8.x86_64 57/166 \n Upgrading : smartmontools-1:7.1-7.el8.x86_64 58/166 \n Running scriptlet: smartmontools-1:7.1-7.el8.x86_64 58/166 \n Upgrading : insights-client-3.0.8-4.el8_1.noarch 59/166 \n Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 59/166 \n Upgrading : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 \n Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 \n Upgrading : mdadm-4.1-9.el8.x86_64 61/166 \n Running scriptlet: mdadm-4.1-9.el8.x86_64 2020-02-05T16:52:29.867 INFO:teuthology.task.ansible.out: 61/166 \n Upgrading : mcelog-3:162-2.el8.x86_64 62/166 \n Running scriptlet: mcelog-3:162-2.el8.x86_64 62/166 \n Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 \n Upgrading : chrony-3.5-1.el8.x86_64 63/166 \n Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 \n Upgrading : systemd-udev-239-18.el8_1.2.x86_64 64/166 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 64/166 \n Upgrading : cryptsetup-2.2.0-2.el8.x86_64 65/166 \n Upgrading : util-linux-user-2.32.1-17.el8.x86_64 66/166 \n Upgrading : redhat-release-8.1-3.3.el8.x86_64 67/166 \n Upgrading : teamd-1.28-4.el8.x86_64 68/166 \n Upgrading : cyrus-sasl-plain-2.1.27-1.el8.x86_64 69/166 \n Upgrading : plymouth-0.9.3-15.el8.x86_64 70/166 \n Upgrading : ledmon-0.92-7.el8.x86_64 71/166 \n Upgrading : wget-1.19.5-8.el8_1.1.x86_64 72/166 \n Running scriptlet: wget-1.19.5-8.el8_1.1.x86_64 72/166 \n Upgrading : bpftool-4.18.0-147.5.1.el8_1.x86_64 73/166 \n Upgrading : blktrace-1.2.0-10.el8.x86_64 74/166 \n Upgrading : mtr-2:0.92-3.el8.x86_64 75/166 \n Upgrading : strace-4.24-5.el8.x86_64 76/166 \n Upgrading : sudo-1.8.25p1-8.el8_1.x86_64 77/166 \n Running scriptlet: sudo-1.8.25p1-8.el8_1.x86_64 77/166 \n Upgrading : sos-3.7-8.el8_1.noarch 78/166 \n Upgrading : ethtool-2:5.0-2.el8.x86_64 79/166 \n Upgrading : 2020-02-05T16:52:29.868 INFO:teuthology.task.ansible.out:irqbalance-2:1.4.0-4.el8.x86_64 80/166 \n Running scriptlet: irqbalance-2:1.4.0-4.el8.x86_64 80/166 \n Upgrading : rsync-3.1.3-6.el8.x86_64 81/166 \n Upgrading : kpatch-0.6.1-5.el8.noarch 82/166 \n Upgrading : nmap-ncat-2:7.70-5.el8.x86_64 83/166 \n Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 83/166 \n Cleanup : sssd-2.0.0-43.el8_0.3.x86_64 84/166 \n Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 \n Cleanup : insights-client-3.0.8-0.el8_0.noarch 85/166 \n Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 \n Cleanup : sssd-ipa-2.0.0-43.el8_0.3.x86_64 86/166 \n Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Cleanup : sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 \n Cleanup : sssd-ad-2.0.0-43.el8_0.3.x86_64 88/166 \n Cleanup : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 89/166 \n Cleanup : sssd-ldap-2.0.0-43.el8_0.3.x86_64 90/166 \n Cleanup : systemd-udev-239-13.el8_0.5.x86_64 91/166 \n Running scriptlet: systemd-udev-239-13.el8_0.5.x86_64 91/166 \n Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Cleanup : vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 \n Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 93/166 \n Cleanup : lvm2-8:2.03.02-6.el8.x86_64 93/166 \n Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 2020-02-05T16:52:29.870 INFO:teuthology.task.ansible.out: 93/166 \n Cleanup : lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 \n Running scriptlet: lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 \n Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 \n Cleanup : device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 \n Cleanup : sssd-krb5-2.0.0-43.el8_0.3.x86_64 96/166 \n Cleanup : sssd-proxy-2.0.0-43.el8_0.3.x86_64 97/166 \n Cleanup : cryptsetup-2.0.6-1.el8.x86_64 98/166 \n Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Cleanup : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 \n Cleanup : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 100/166 \n Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Cleanup : sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 \n Cleanup : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 \n Running scriptlet: device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 \n Cleanup : teamd-1.27-10.el8.x86_64 103/166 \n Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 \n Cleanup : chrony-3.3-3.el8.x86_64 104/166 \n Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 \n Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 \n Cleanup : mcelog-3:159-1.el8.x86_64 105/166 \n Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 \n Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 \n C 2020-02-05T16:52:29.871 INFO:teuthology.task.ansible.out:leanup : mdadm-4.1-4.el8.x86_64 106/166 \n Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 \n Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 \n Cleanup : smartmontools-1:6.6-3.el8.x86_64 107/166 \n Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 \n Cleanup : python3-libstoragemgmt-1.6.2-9.el8.noarch 108/166 \n Cleanup : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 109/166 \n Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Cleanup : libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 \n Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Cleanup : sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 \n Cleanup : systemd-pam-239-13.el8_0.5.x86_64 112/166 \n Cleanup : ledmon-0.90-3.el8.x86_64 113/166 \n Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 114/166 \n Cleanup : wget-1.19.5-7.el8_0.1.x86_64 114/166 \n Cleanup : subscription-manager-plugin-container-1.23.8-35. 115/166 \n Cleanup : cockpit-185.1-1.el8_0.x86_64 116/166 \n Cleanup : subscription-manager-cockpit-1.23.8-35.el8.noarc 117/166 \n Cleanup : redhat-release-8.0-0.44.el8.x86_64 118/166 \n Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Cleanup : subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 \n Cleanup : python3-subscr 2020-02-05T16:52:29.873 INFO:teuthology.task.ansible.out:iption-manager-rhsm-1.23.8-35.el8. 120/166 \n Cleanup : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 121/166 \n Cleanup : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 122/166 \n Cleanup : util-linux-user-2.32.1-8.el8.x86_64 123/166 \n Cleanup : rsyslog-relp-8.37.0-9.el8.x86_64 124/166 \n Cleanup : plymouth-0.9.3-12.el8.x86_64 125/166 \n Running scriptlet: plymouth-0.9.3-12.el8.x86_64 125/166 \n Cleanup : rsyslog-gssapi-8.37.0-9.el8.x86_64 126/166 \n Cleanup : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 127/166 \n Cleanup : rsyslog-gnutls-8.37.0-9.el8.x86_64 128/166 \n Cleanup : subscription-manager-rhsm-certificates-1.23.8-35 129/166 \n Cleanup : redhat-release-eula-8.0-0.44.el8.x86_64 130/166 \n Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Cleanup : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 \n Cleanup : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 132/166 \n Cleanup : sos-3.6-10.el8_0.3.noarch 133/166 \n Cleanup : kpatch-0.6.1-1.el8.noarch 134/166 \n Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 \n Cleanup : rsyslog-8.37.0-9.el8.x86_64 135/166 \n Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 \n Cleanup : cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 \n Running scriptlet: cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 \n Cleanup : device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 \n Running scriptlet: device-mapper-libs-8:1.02.155-6.el8.x86_64 2020-02-05T16:52:29.874 INFO:teuthology.task.ansible.out: 137/166 \n Cleanup : device-mapper-8:1.02.155-6.el8.x86_64 138/166 \n Cleanup : dbus-1:1.12.8-7.el8.x86_64 139/166 \n Running scriptlet: systemd-239-13.el8_0.5.x86_64 140/166 \n Cleanup : systemd-239-13.el8_0.5.x86_64 140/166 \n Cleanup : util-linux-2.32.1-8.el8.x86_64 141/166 \n Cleanup : libfdisk-2.32.1-8.el8.x86_64 142/166 \n Running scriptlet: libfdisk-2.32.1-8.el8.x86_64 142/166 \n Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Cleanup : dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 \n Cleanup : dbus-tools-1:1.12.8-7.el8.x86_64 144/166 \n Cleanup : dbus-common-1:1.12.8-7.el8.noarch 145/166 \n Cleanup : dbus-libs-1:1.12.8-7.el8.x86_64 146/166 \n Running scriptlet: dbus-libs-1:1.12.8-7.el8.x86_64 146/166 \n Cleanup : systemd-libs-239-13.el8_0.5.x86_64 147/166 \n Cleanup : libmount-2.32.1-8.el8.x86_64 148/166 \n Running scriptlet: libmount-2.32.1-8.el8.x86_64 148/166 \n Cleanup : libblkid-2.32.1-8.el8.x86_64 149/166 \n Running scriptlet: libblkid-2.32.1-8.el8.x86_64 149/166 \n Cleanup : libuuid-2.32.1-8.el8.x86_64 150/166 \n Running scriptlet: libuuid-2.32.1-8.el8.x86_64 150/166 \n Cleanup : libsmartcols-2.32.1-8.el8.x86_64 151/166 \n Running scriptlet: libsmartcols-2.32.1-8.el8.x86_64 151/166 \n Cleanup : vim-common-2:8.0.1763-11.el8_0.x86_64 152/166 \n Cleanup 2020-02-05T16:52:29.876 INFO:teuthology.task.ansible.out: : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 \n Running scriptlet: cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 \n Cleanup : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 154/166 \n Cleanup : libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 \n Running scriptlet: libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 \n Cleanup : libteam-1.27-10.el8.x86_64 156/166 \n Running scriptlet: libteam-1.27-10.el8.x86_64 156/166 \n Cleanup : libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 \n Running scriptlet: libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 \n Cleanup : bpftool-4.18.0-80.11.2.el8_0.x86_64 158/166 \n Cleanup : blktrace-1.2.0-9.el8.x86_64 159/166 \n Cleanup : mtr-2:0.92-1.el8.x86_64 160/166 \n Cleanup : strace-4.24-3.el8.x86_64 161/166 \n Cleanup : sudo-1.8.25p1-4.el8_0.1.x86_64 162/166 \n Cleanup : ethtool-2:4.16-1.el8.x86_64 163/166 \n Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Cleanup : irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 \n Cleanup : rsync-3.1.3-4.el8.x86_64 165/166 \n Cleanup : nmap-ncat-2:7.70-4.el8.x86_64 166/166 \n Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 166/166 \n Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 166/166 \n Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 166/166 \n Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 166/166 \n Running scriptlet: nmap-ncat-2:7.70-4.el8.x86_6 2020-02-05T16:52:29.877 INFO:teuthology.task.ansible.out:4 166/166 \n Running scriptlet: systemd-239-18.el8_1.2.x86_64 166/166 \n Running scriptlet: vim-common-2:8.0.1763-13.el8.x86_64 166/166 \n Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 166/166 \n Verifying : smartmontools-1:7.1-7.el8.x86_64 1/166 \n Verifying : smartmontools-1:6.6-3.el8.x86_64 2/166 \n Verifying : rsyslog-8.37.0-13.el8.x86_64 3/166 \n Verifying : rsyslog-8.37.0-9.el8.x86_64 4/166 \n Verifying : insights-client-3.0.8-4.el8_1.noarch 5/166 \n Verifying : insights-client-3.0.8-0.el8_0.noarch 6/166 \n Verifying : wget-1.19.5-8.el8_1.1.x86_64 7/166 \n Verifying : wget-1.19.5-7.el8_0.1.x86_64 8/166 \n Verifying : rsyslog-gnutls-8.37.0-13.el8.x86_64 9/166 \n Verifying : rsyslog-gnutls-8.37.0-9.el8.x86_64 10/166 \n Verifying : vim-common-2:8.0.1763-13.el8.x86_64 11/166 \n Verifying : vim-common-2:8.0.1763-11.el8_0.x86_64 12/166 \n Verifying : vim-enhanced-2:8.0.1763-13.el8.x86_64 13/166 \n Verifying : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 14/166 \n Verifying : rsyslog-gssapi-8.37.0-13.el8.x86_64 15/166 \n Verifying : rsyslog-gssapi-8.37.0-9.el8.x86_64 16/166 \n Verifying : nmap-ncat-2:7.70-5.el8.x86_64 17/166 \n Verifying : nmap-ncat-2:7.70-4.el8.x86_64 18/166 \n Verifying : plymouth-0.9.3-15.el8.x86_64 19/166 \n Verifying : plymouth-0.9.3-12.el8.x86_64 20/166 \n Verifying : rsyslog-relp-8.37.0-13.el8.x86_64 21/166 \ 2020-02-05T16:52:29.879 INFO:teuthology.task.ansible.out:n Verifying : rsyslog-relp-8.37.0-9.el8.x86_64 22/166 \n Verifying : sssd-ldap-2.2.0-19.el8.x86_64 23/166 \n Verifying : sssd-ldap-2.0.0-43.el8_0.3.x86_64 24/166 \n Verifying : dbus-common-1:1.12.8-9.el8.noarch 25/166 \n Verifying : dbus-common-1:1.12.8-7.el8.noarch 26/166 \n Verifying : ledmon-0.92-7.el8.x86_64 27/166 \n Verifying : ledmon-0.90-3.el8.x86_64 28/166 \n Verifying : cryptsetup-2.2.0-2.el8.x86_64 29/166 \n Verifying : cryptsetup-2.0.6-1.el8.x86_64 30/166 \n Verifying : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 31/166 \n Verifying : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 32/166 \n Verifying : util-linux-2.32.1-17.el8.x86_64 33/166 \n Verifying : util-linux-2.32.1-8.el8.x86_64 34/166 \n Verifying : lvm2-8:2.03.05-5.el8.x86_64 35/166 \n Verifying : lvm2-8:2.03.02-6.el8.x86_64 36/166 \n Verifying : util-linux-user-2.32.1-17.el8.x86_64 37/166 \n Verifying : util-linux-user-2.32.1-8.el8.x86_64 38/166 \n Verifying : kpatch-0.6.1-5.el8.noarch 39/166 \n Verifying : kpatch-0.6.1-1.el8.noarch 40/166 \n Verifying : dbus-daemon-1:1.12.8-9.el8.x86_64 41/166 \n Verifying : dbus-daemon-1:1.12.8-7.el8.x86_64 42/166 \n Verifying : redhat-release-8.1-3.3.el8.x86_64 43/166 \n Verifying : redhat-release-8.0-0.44.el8.x86_64 44/166 \n Verifying : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 45/166 \n Verifying : device-map 2020-02-05T16:52:29.880 INFO:teuthology.task.ansible.out:per-event-libs-8:1.02.155-6.el8.x86_64 46/166 \n Verifying : libmount-2.32.1-17.el8.x86_64 47/166 \n Verifying : libmount-2.32.1-8.el8.x86_64 48/166 \n Verifying : libsmartcols-2.32.1-17.el8.x86_64 49/166 \n Verifying : libsmartcols-2.32.1-8.el8.x86_64 50/166 \n Verifying : python3-sssdconfig-2.2.0-19.el8.noarch 51/166 \n Verifying : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 52/166 \n Verifying : subscription-manager-cockpit-1.25.17-1.el8.noarc 53/166 \n Verifying : subscription-manager-cockpit-1.23.8-35.el8.noarc 54/166 \n Verifying : python3-libstoragemgmt-1.8.1-2.el8.noarch 55/166 \n Verifying : python3-libstoragemgmt-1.6.2-9.el8.noarch 56/166 \n Verifying : libstoragemgmt-1.8.1-2.el8.x86_64 57/166 \n Verifying : libstoragemgmt-1.6.2-9.el8.x86_64 58/166 \n Verifying : libuuid-2.32.1-17.el8.x86_64 59/166 \n Verifying : libuuid-2.32.1-8.el8.x86_64 60/166 \n Verifying : dbus-libs-1:1.12.8-9.el8.x86_64 61/166 \n Verifying : dbus-libs-1:1.12.8-7.el8.x86_64 62/166 \n Verifying : rsync-3.1.3-6.el8.x86_64 63/166 \n Verifying : rsync-3.1.3-4.el8.x86_64 64/166 \n Verifying : redhat-release-eula-8.1-3.3.el8.x86_64 65/166 \n Verifying : redhat-release-eula-8.0-0.44.el8.x86_64 66/166 \n Verifying : irqbalance-2:1.4.0-4.el8.x86_64 67/166 \n Verifying : irqbalance-2:1.4.0-2.el8.x86_64 68/166 \n Verifying : ethtool-2:5.0-2.el8.x86_64 69/166 \n Verifying : ethtool-2:4.16-1.el8.x86_64 2020-02-05T16:52:29.882 INFO:teuthology.task.ansible.out: 70/166 \n Verifying : libipa_hbac-2.2.0-19.el8.x86_64 71/166 \n Verifying : libipa_hbac-2.0.0-43.el8_0.3.x86_64 72/166 \n Verifying : device-mapper-event-8:1.02.163-5.el8.x86_64 73/166 \n Verifying : device-mapper-event-8:1.02.155-6.el8.x86_64 74/166 \n Verifying : libteam-1.28-4.el8.x86_64 75/166 \n Verifying : libteam-1.27-10.el8.x86_64 76/166 \n Verifying : sos-3.7-8.el8_1.noarch 77/166 \n Verifying : sos-3.6-10.el8_0.3.noarch 78/166 \n Verifying : sudo-1.8.25p1-8.el8_1.x86_64 79/166 \n Verifying : sudo-1.8.25p1-4.el8_0.1.x86_64 80/166 \n Verifying : dbus-1:1.12.8-9.el8.x86_64 81/166 \n Verifying : dbus-1:1.12.8-7.el8.x86_64 82/166 \n Verifying : sssd-ipa-2.2.0-19.el8.x86_64 83/166 \n Verifying : sssd-ipa-2.0.0-43.el8_0.3.x86_64 84/166 \n Verifying : sssd-2.2.0-19.el8.x86_64 85/166 \n Verifying : sssd-2.0.0-43.el8_0.3.x86_64 86/166 \n Verifying : dbus-tools-1:1.12.8-9.el8.x86_64 87/166 \n Verifying : dbus-tools-1:1.12.8-7.el8.x86_64 88/166 \n Verifying : sssd-common-2.2.0-19.el8.x86_64 89/166 \n Verifying : sssd-common-2.0.0-43.el8_0.3.x86_64 90/166 \n Verifying : mdadm-4.1-9.el8.x86_64 91/166 \n Verifying : mdadm-4.1-4.el8.x86_64 92/166 \n Verifying : libsss_idmap-2.2.0-19.el8.x86_64 93/166 \n Verifying : libsss_idmap-2.0.0-43.el8_0.3.x86_64 94/166 \n Verifying 2020-02-05T16:52:29.883 INFO:teuthology.task.ansible.out: : teamd-1.28-4.el8.x86_64 95/166 \n Verifying : teamd-1.27-10.el8.x86_64 96/166 \n Verifying : strace-4.24-5.el8.x86_64 97/166 \n Verifying : strace-4.24-3.el8.x86_64 98/166 \n Verifying : cockpit-196.3-1.el8.x86_64 99/166 \n Verifying : cockpit-185.1-1.el8_0.x86_64 100/166 \n Verifying : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 101/166 \n Verifying : python3-libstoragemgmt-clibs-1.6.2-9.el8.x 2020-02-05T16:52:29.883 INFO:teuthology.task.ansible.out:86_64 102/166 \n Verifying : sssd-client-2.2.0-19.el8.x86_64 103/166 \n Verifying : sssd-client-2.0.0-43.el8_0.3.x86_64 104/166 \n Verifying : cryptsetup-libs-2.2.0-2.el8.x86_64 105/166 \n Verifying : cryptsetup-libs-2.0.6-1.el8.x86_64 106/166 \n Verifying : mcelog-3:162-2.el8.x86_64 107/166 \n Verifying : mcelog-3:159-1.el8.x86_64 108/166 \n Verifying : vdo-6.2.1.134-11.el8.x86_64 109/166 \n Verifying : vdo-6.2.0.298-10.el8_0.x86_64 110/166 \n Verifying : sssd-common-pac-2.2.0-19.el8.x86_64 111/166 \n Verifying : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 112/166 \n Verifying : subscription-manager-rhsm-certificates-1.25.17-1 113/166 \n Verifying : subscription-manager-rhsm-certificates-1.23.8-35 114/166 \n Verifying : libblkid-2.32.1-17.el8.x86_64 115/166 \n Verifying : libblkid-2.32.1-8.el8.x86_64 116/166 \n Verifying : mtr-2:0.92-3.el8.x86_64 117/166 \n Verifying : mtr-2:0.92-1.el8.x86_64 118/166 \n Verifying : cyrus-sasl-gssapi-2.1.27 2020-02-05T16:52:29.885 INFO:teuthology.task.ansible.out:-1.el8.x86_64 119/166 \n Verifying : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 120/166 \n Verifying : cyrus-sasl-lib-2.1.27-1.el8.x86_64 121/166 \n Verifying : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 122/166 \n Verifying : device-mapper-8:1.02.163-5.el8.x86_64 123/166 \n Verifying : device-mapper-8:1.02.155-6.el8.x86_64 124/166 \n Verifying : blktrace-1.2.0-10.el8.x86_64 125/166 \n Verifying : blktrace-1.2.0-9.el8.x86_64 126/166 \n Verifying : libfdisk-2.32.1-17.el8.x86_64 127/166 \n Verifying : libfdisk-2.32.1-8.el8.x86_64 128/166 \n Verifying : sssd-proxy-2.2.0-19.el8.x86_64 129/166 \n Verifying : sssd-proxy-2.0.0-43.el8_0.3.x86_64 130/166 \n Verifying : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 131/166 \n Verifying : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 132/166 \n Verifying : kmod-kvdo-6.2.1.138-57.el8.x86_64 133/166 \n Verifying : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 134/166 \n Verifying : python3-subscription-manager-rhsm-1.25.17-1.el8. 135/166 \n Verifying : python3-subscription-manager-rhsm-1.23.8-35.el8. 136/166 \n Verifying : subscription-manager-1.25.17-1.el8.x86_64 137/166 \n Verifying : subscription-manager-1.23.8-35.el8.x86_64 138/166 \n Verifying : sssd-kcm-2.2.0-19.el8.x86_64 139/166 \n Verifying : sssd-kcm-2.0.0-43.el8_0.3.x86_64 140/166 \n Verifying : sssd-krb5-common-2.2.0-19.el8.x86_64 141/166 \n Verifying : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 142/166 \n Verifying : chrony-3.5-1.el8.x86_64 143/1 2020-02-05T16:52:29.888 INFO:teuthology.task.ansible.out:66 \n Verifying : chrony-3.3-3.el8.x86_64 144/166 \n Verifying : cyrus-sasl-plain-2.1.27-1.el8.x86_64 145/166 \n Verifying : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 146/166 \n Verifying : sssd-krb5-2.2.0-19.el8.x86_64 147/166 \n Verifying : sssd-krb5-2.0.0-43.el8_0.3.x86_64 148/166 \n Verifying : sssd-ad-2.2.0-19.el8.x86_64 149/166 \n Verifying : sssd-ad-2.0.0-43.el8_0.3.x86_64 150/166 \n Verifying : lvm2-libs-8:2.03.05-5.el8.x86_64 151/166 \n Verifying : lvm2-libs-8:2.03.02-6.el8.x86_64 152/166 \n Verifying : subscription-manager-plugin-container-1.25.17-1. 153/166 \n Verifying : subscription-manager-plugin-container-1.23.8-35. 154/166 \n Verifying : device-mapper-libs-8:1.02.163-5.el8.x86_64 155/166 \n Verifying : device-mapper-libs-8:1.02.155-6.el8.x86_64 156/166 \n Verifying : systemd-libs-239-18.el8_1.2.x86_64 157/166 \n Verifying : systemd-libs-239-13.el8_0.5.x86_64 158/166 \n Verifying : systemd-pam-239-18.el8_1.2.x86_64 159/166 \n Verifying : systemd-pam-239-13.el8_0.5.x86_64 160/166 \n Verifying : systemd-239-18.el8_1.2.x86_64 161/166 \n Verifying : systemd-239-13.el8_0.5.x86_64 162/166 \n Verifying : bpftool-4.18.0-147.5.1.el8_1.x86_64 163/166 \n Verifying : bpftool-4.18.0-80.11.2.el8_0.x86_64 164/166 \n Verifying : systemd-udev-239-18.el8_1.2.x86_64 165/166 \n Verifying : systemd-udev-239-13.el8_0.5.x86_64 166/166 \nInstalled products updated.\n\nUpgraded:\n smartmontools-1:7.1-7.el8.x86_64 2020-02-05T16:52:29.890 INFO:teuthology.task.ansible.out: \n rsyslog-8.37.0-13.el8.x86_64 \n insights-client-3.0.8-4.el8_1.noarch \n wget-1.19.5-8.el8_1.1.x86_64 \n rsyslog-gnutls-8.37.0-13.el8.x86_64 \n vim-common-2:8.0.1763-13.el8.x86_64 \n vim-enhanced-2:8.0.1763-13.el8.x86_64 \n rsyslog-gssapi-8.37.0-13.el8.x86_64 \n nmap-ncat-2:7.70-5.el8.x86_64 \n plymouth-0.9.3-15.el8.x86_64 \n rsyslog-relp-8.37.0-13.el8.x86_64 \n sssd-ldap-2.2.0-19.el8.x86_64 \n dbus-common-1:1.12.8-9.el8.noarch \n ledmon-0.92-7.el8.x86_64 \n cryptsetup-2.2.0-2.el8.x86_64 \n microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_64 \n util-linux-2.32.1-17.el8.x86_64 \n lvm2-8:2.03.05-5.el8.x86_64 \n util-linux-user-2.32.1-17.el8.x86_64 \n kpatch-0.6.1-5.el8.noarch \n dbus-daemon-1:1.12.8-9.el8.x86_64 \n redhat-release-8.1-3.3.el8.x86_64 \n device-mapper-event-libs-8:1.02.163-5.el8.x86_64 \n libmount-2.32.1-17.el8.x86_64 \n libsmartcols-2.32.1-17.el8.x86_64 \n python3-sssdcon 2020-02-05T16:52:29.892 INFO:teuthology.task.ansible.out:fig-2.2.0-19.el8.noarch \n subscription-manager-cockpit-1.25.17-1.el8.noarch \n python3-libstoragemgmt-1.8.1-2.el8.noarch \n libstoragemgmt-1.8.1-2.el8.x86_64 \n libuuid-2.32.1-17.el8.x86_64 \n dbus-libs-1:1.12.8-9.el8.x86_64 \n rsync-3.1.3-6.el8.x86_64 \n redhat-release-eula-8.1-3.3.el8.x86_64 \n irqbalance-2:1.4.0-4.el8.x86_64 \n ethtool-2:5.0-2.el8.x86_64 \n libipa_hbac-2.2.0-19.el8.x86_64 \n device-mapper-event-8:1.02.163-5.el8.x86_64 \n libteam-1.28-4.el8.x86_64 \n sos-3.7-8.el8_1.noarch \n sudo-1.8.25p1-8.el8_1.x86_64 \n dbus-1:1.12.8-9.el8.x86_64 \n sssd-ipa-2.2.0-19.el8.x86_64 \n sssd-2.2.0-19.el8.x86_64 \n dbus-tools-1:1.12.8-9.el8.x86_64 \n sssd-common-2.2.0-19.el8.x86_64 \n mdadm-4.1-9.el8.x86_64 \n libsss_idmap-2.2.0-19.el8.x86_64 \n teamd-1.28-4.el8.x86_64 \n strace-4.24-5.el8.x86_64 \n cockpit-196.3-1.el8.x86_64 2020-02-05T16:52:29.893 INFO:teuthology.task.ansible.out: \n python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 \n sssd-client-2.2.0-19.el8.x86_64 \n cryptsetup-libs-2.2.0-2.el8.x86_64 \n mcelog-3:162-2.el8.x86_64 \n vdo-6.2.1.134-11.el8.x86_64 \n sssd-common-pac-2.2.0-19.el8.x86_64 \n subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64 \n libblkid-2.32.1-17.el8.x86_64 \n mtr-2:0.92-3.el8.x86_64 \n cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 \n cyrus-sasl-lib-2.1.27-1.el8.x86_64 \n device-mapper-8:1.02.163-5.el8.x86_64 \n blktrace-1.2.0-10.el8.x86_64 \n libfdisk-2.32.1-17.el8.x86_64 \n sssd-proxy-2.2.0-19.el8.x86_64 \n dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64 \n kmod-kvdo-6.2.1.138-57.el8.x86_64 \n python3-subscription-manager-rhsm-1.25.17-1.el8.x86_64 \n subscription-manager-1.25.17-1.el8.x86_64 \n sssd-kcm-2.2.0-19.el8.x86_64 \n sssd-krb5-common-2.2.0-19.el8.x86_64 \n chrony-3.5-1.el8.x86_64 \n cyrus-sasl-plain-2.1.27-1.el8.x86_64 \n sssd-krb5-2.2.0-19.el8.x86_64 \ 2020-02-05T16:52:29.895 INFO:teuthology.task.ansible.out:n sssd-ad-2.2.0-19.el8.x86_64 \n lvm2-libs-8:2.03.05-5.el8.x86_64 \n subscription-manager-plugin-container-1.25.17-1.el8.x86_64 \n device-mapper-libs-8:1.02.163-5.el8.x86_64 \n systemd-libs-239-18.el8_1.2.x86_64 \n systemd-pam-239-18.el8_1.2.x86_64 \n systemd-239-18.el8_1.2.x86_64 \n bpftool-4.18.0-147.5.1.el8_1.x86_64 \n systemd-udev-239-18.el8_1.2.x86_64 \n\nComplete!", "stdout_lines": ["Updating Subscription Management repositories.", "Extra Packages for Enterprise Linux 5.2 kB/s | 5.3 kB 00:01 ", "lab-extr 2020-02-05T16:52:29.895 INFO:teuthology.task.ansible.out:as 856 kB/s | 2.9 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - AppStre 65 kB/s | 2.8 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - BaseOS 51 kB/s | 2.4 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Arch Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Updating Subscription Management repositories.", "Red Hat Enterprise Linux 8 for x86_64 - AppStre 70 kB/s | 2.8 kB 00:00 ", "Red Hat Enterprise Linux 8 for x86_64 - BaseOS 56 kB/s | 2.4 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package 2020-02-05T16:52:29.896 INFO:teuthology.task.ansible.out: ArchVersion Repository Size", "================================================================================", "Upgrading:", " smartmontools x86_641:7.1-7.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 546 k", " rsyslog x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 691 k", " insights-client noarch3.0.8-4.el8_1 rhel-8-for-x86_64-appstream-rpms 899 k", " wget x86_641.19.5-8.el8_1.1 rhel-8-for-x86_64-appstream-rpms 735 k", " rsyslog-gnutls x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 27 k", " vim-common x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 6.3 M", " vim-enhanced x86_642:8.0.1763-13.el8 rhel-8-for-x86_64-appstream-rpms 1.4 M", " rsyslog-gssapi x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 31 k", " nmap-ncat x86_642:7.70-5.el8 rhel-8-for-x86_64-appstream-rpms 237 k", " plymouth x86_640.9.3-15.el8 rhel-8-for-x86_64-appstream-rpms 114 k", " rsyslog-relp x86_648.37.0-13.el8 rhel-8-for-x86_64-appstream-rpms 29 k", " sssd-ldap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 205 k", " dbus-common noarch1:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 45 k", " ledmon 2020-02-05T16:52:29.898 INFO:teuthology.task.ansible.out: x86_640.92-7.el8 rhel-8-for-x86_64-baseos-rpms 74 k", " cryptsetup x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 177 k", " microcode_ctl x86_644:20190618-1.20191115.3.el8_1 rhel-8-for-x86_64-baseos-rpms 2.5 M", " util-linux x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 2.5 M", " lvm2 x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M", " util-linux-user x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 98 k", " kpatch noarch0.6.1-5.el8 rhel-8-for-x86_64-baseos-rpms 15 k", " dbus-daemon x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 241 k", " redhat-release x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 38 k", " device-mapper-event-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 265 k", " libmount x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 231 k", " libsmartcols x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 175 k", " python3-sssdconfig noarch2.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 116 k", " subscription-manager-cockpit noarch1.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.0 M", " python3-libstoragemgmt noarch1.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 2020-02-05T16:52:29.899 INFO:teuthology.task.ansible.out: 171 k", " libstoragemgmt x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 266 k", " libuuid x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 94 k", " dbus-libs x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 184 k", " rsync x86_643.1.3-6.el8 rhel-8-for-x86_64-baseos-rpms 404 k", " redhat-release-eula x86_648.1-3.3.el8 rhel-8-for-x86_64-baseos-rpms 16 k", " irqbalance x86_642:1.4.0-4.el8 rhel-8-for-x86_64-baseos-rpms 55 k", " ethtool x86_642:5.0-2.el8 rhel-8-for-x86_64-baseos-rpms 151 k", " libipa_hbac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 99 k", " device-mapper-event x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 266 k", " libteam x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 64 k", " sos noarch3.7-8.el8_1 rhel-8-for-x86_64-baseos-rpms 510 k", " sudo x86_641.8.25p1-8.el8_1 rhel-8-for-x86_64-baseos-rpms 875 k", " dbus x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 40 k", " sssd-ipa x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 323 k", " sssd x86_642.2.0-19.el8 rhel 2020-02-05T16:52:29.901 INFO:teuthology.task.ansible.out:-8-for-x86_64-baseos-rpms 90 k", " dbus-tools x86_641:1.12.8-9.el8 rhel-8-for-x86_64-baseos-rpms 85 k", " sssd-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 1.5 M", " mdadm x86_644.1-9.el8 rhel-8-for-x86_64-baseos-rpms 448 k", " libsss_idmap x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 104 k", " teamd x86_641.28-4.el8 rhel-8-for-x86_64-baseos-rpms 129 k", " strace x86_644.24-5.el8 rhel-8-for-x86_64-baseos-rpms 2020-02-05T16:52:29.901 INFO:teuthology.task.ansible.out: 971 k", " cockpit x86_64196.3-1.el8 rhel-8-for-x86_64-baseos-rpms 70 k", " python3-libstoragemgmt-clibs x86_641.8.1-2.el8 rhel-8-for-x86_64-baseos-rpms 27 k", " sssd-client x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 162 k", " cryptsetup-libs x86_642.2.0-2.el8 rhel-8-for-x86_64-baseos-rpms 422 k", " mcelog x86_643:162-2.el8 rhel-8-for-x86_64-baseos-rpms 79 k", " vdo x86_646.2.1.134-11.el8 rhel-8-for-x86_64-baseos-rpms 689 k", " sssd-common-pac x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 161 k", " subscription-manager-rhsm-certificates x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 245 k", " libblkid x86_6 2020-02-05T16:52:29.902 INFO:teuthology.task.ansible.out:42.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 215 k", " mtr x86_642:0.92-3.el8 rhel-8-for-x86_64-baseos-rpms 96 k", " cyrus-sasl-gssapi x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 49 k", " cyrus-sasl-lib x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 124 k", " device-mapper x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 371 k", " blktrace x86_641.2.0-10.el8 rhel-8-for-x86_64-baseos-rpms 150 k", " libfdisk x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 249 k", " sssd-proxy x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k", " dnf-plugin-subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 272 k", " kmod-kvdo x86_646.2.1.138-57.el8 rhel-8-for-x86_64-baseos-rpms 355 k", " python3-subscription-manager-rhsm x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 338 k", " subscription-manager x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M", " sssd-kcm x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 213 k", " sssd-krb5-common x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 172 k", " chrony x86_643.5-1.el8 rhel-8-for-x86_64-baseos-rpms 271 k", " cyrus-sa 2020-02-05T16:52:29.904 INFO:teuthology.task.ansible.out:sl-plain x86_642.1.27-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k", " sssd-krb5 x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 126 k", " sssd-ad x86_642.2.0-19.el8 rhel-8-for-x86_64-baseos-rpms 229 k", " lvm2-libs x86_648:2.03.05-5.el8 rhel-8-for-x86_64-baseos-rpms 1.1 M", " subscription-manager-plugin-container x86_641.25.17-1.el8 rhel-8-for-x86_64-baseos-rpms 248 k", " device-mapper-libs x86_648:1.02.163-5.el8 rhel-8-for-x86_64-baseos-rpms 403 k", " systemd-libs x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 560 k", " systemd-pam x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 230 k", " systemd x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 3.5 M", " bpftool x86_644.18.0-147.5.1.el8_1 rhel-8-for-x86_64-baseos-rpms 2.1 M", " systemd-udev x86_64239-18.el8_1.2 rhel-8-for-x86_64-baseos-rpms 1.3 M", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "Upgrade 83 Packages", "", "Total download size: 42 M", "Downloading Packages:", "(1/83): rsyslog-8.37.0-13.el8.x86_64.rpm 9.8 MB/s | 691 kB 00:00 ", "(2/83): insights-client-3.0.8-4.el8_1.noarch.rp 9.1 MB/s | 899 kB 00:00 ", "(3/83): s 2020-02-05T16:52:29.905 INFO:teuthology.task.ansible.out:martmontools-7.1-7.el8.x86_64.rpm 5.0 MB/s | 546 kB 00:00 ", "(4/83): wget-1.19.5-8.el8_1.1.x86_64.rpm 15 MB/s | 735 kB 00:00 ", "(5/83): rsyslog-gnutls-8.37.0-13.el8.x86_64.rpm 525 kB/s | 27 kB 00:00 ", "(6/83): vim-enhanced-8.0.1763-13.el8.x86_64.rpm 18 MB/s | 1.4 MB 00:00 ", "(7/83): rsyslog-gssapi-8.37.0-13.el8.x86_64.rpm 592 kB/s | 31 kB 00:00 ", "(8/83): nmap-ncat-7.70-5.el8.x86_64.rpm 5.8 MB/s | 237 kB 00:00 ", "(9/83): plymouth-0.9.3-15.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 ", "(10/83): rsyslog-relp-8.37.0-13.el8.x86_64.rpm 998 kB/s | 29 kB 00:00 ", "(11/83): sssd-ldap-2.2.0-19.el8.x86_64.rpm 4.0 MB/s | 205 kB 00:00 ", "(12/83): dbus-common-1.12.8-9.el8.noarch.rpm 977 kB/s | 45 kB 00:00 ", "(13/83): ledmon-0.92-7.el8.x86_64.rpm 2.3 MB/s | 74 kB 00:00 ", "(14/83): cryptsetup-2.2.0-2.el8.x86_64.rpm 4.7 MB/s | 177 kB 00:00 ", "(15/83): vim-common-8.0.1763-13.el8.x86_64.rpm 23 MB/s | 6.3 MB 00:00 ", "(16/83): microcode_ctl-20190618-1.20191115.3.el 21 MB/s | 2.5 MB 00:00 ", "(17/83): lvm2-2.03.05-5.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 ", "(18/83): util-linux-2.32.1-17.el8.x86_64.rpm 20 MB/s | 2.5 MB 00:00 ", "(19/83): kpatch-0.6.1-5.el8.noarch.rpm 313 kB/s | 15 kB 00:00 ", "(20/83): util-linux-user-2.32.1-17.el8.x86_64.r 1.7 MB/s | 98 kB 00:00 ", "(21/83): dbus-daemon-1.12.8-9.el8.x86_64.rpm 3.7 MB/s | 241 kB 00:00 ", "(22/83): device-mapper-event-libs-1.02.163-5.el 6.1 MB/s | 265 kB 00:00 ", "(23/83): redhat-release-8.1-3.3.el8.x86_64.rpm 752 kB/s | 38 kB 00:00 ", "(24/83): libmount-2.32.1-17.el8.x86_64.rpm 5.1 MB/s | 231 kB 00:00 ", "(25/83): libsmartcols-2.32.1-17.el8.x86_64.rpm 3.9 MB/s | 175 kB 00:00 ", "(26/83): python3-sssdconfig-2.2.0-19.el8.noarch 2.1 MB/s | 116 kB 00:00 ", "(27/83): python3- 2020-02-05T16:52:29.907 INFO:teuthology.task.ansible.out:libstoragemgmt-1.8.1-2.el8.noa 4.4 MB/s | 171 kB 00:00 ", "(28/83): subscription-manager-cockpit-1.25.17-1 16 MB/s | 1.0 MB 00:00 ", "(29/83): libstoragemgmt-1.8.1-2.el8.x86_64.rpm 5.6 MB/s | 266 kB 00:00 ", "(30/83): libuuid-2.32.1-17.el8.x86_64.rpm 2.7 MB/s | 94 kB 00:00 ", "(31/83): dbus-libs-1.12.8-9.el8.x86_64.rpm 4.0 MB/s | 184 kB 00:00 ", "(32/83): rsync-3.1.3-6.el8.x86_64.rpm 7.6 MB/s | 404 kB 00:00 ", "(33/83): redhat-release-eula-8.1-3.3.el8.x86_64 303 kB/s | 16 kB 00:00 ", "(34/83): irqbalance-1.4.0-4.el8.x86_64.rpm 1.8 MB/s | 55 kB 00:00 ", "(35/83): ethtool-5.0-2.el8.x86_64.rpm 2.9 MB/s | 151 kB 00:00 ", "(36/83): libipa_hbac-2.2.0-19.el8.x86_64.rpm 1.9 MB/s | 99 kB 00:00 ", "(37/83): device-mapper-event-1.02.163-5.el8.x86 4.9 MB/s | 266 kB 00:00 ", "(38/83): libteam-1.28-4.el8.x86_64.rpm 2.2 MB/s | 64 kB 00:00 ", "(39/83): sos-3.7-8.el8_1.noarch.rpm 8.0 MB/s | 510 kB 00:00 ", "(40/83): dbus-1.12.8-9.el8.x86_64.rpm 753 kB/s | 40 kB 00:00 ", "(41/83): sudo-1.8.25p1-8.el8_1.x86_64.rpm 14 MB/s | 875 kB 00:00 ", "(42/83): sssd-ipa-2.2.0-19.el8.x86_64.rpm 6.4 MB/s | 323 kB 00:00 ", "(43/83): sssd-2.2.0-19.el8.x86_64.rpm 1.8 MB/s | 90 kB 00:00 ", "(44/83): dbus-tools-1.12.8-9.el8.x86_64.rpm 2.0 MB/s | 85 kB 00:00 ", "(45/83): mdadm-4.1-9.el8.x86_64.rpm 8.0 MB/s | 448 kB 00:00 ", "(46/83): libsss_idmap-2.2.0-19.el8.x86_64.rpm 2.1 MB/s | 104 kB 00:00 ", "(47/83): teamd-1.28-4.el8.x86_64.rpm 4.3 MB/s | 129 kB 00:00 ", "(48/83): sssd-common-2.2.0-19.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 ", "(49/83): cockpit-196.3-1.el8.x86_64.rpm 1.7 MB/s | 70 kB 00:00 ", "(50/83): python3-libstoragemgmt-clibs-1.8.1-2.e 755 kB/s | 27 kB 00:00 ", "(51/83): strace-4.24-5.el 2020-02-05T16:52:29.908 INFO:teuthology.task.ansible.out:8.x86_64.rpm 12 MB/s | 971 kB 00:00 ", "(52/83): sssd-client-2.2.0-19.el8.x86_64.rpm 4.0 MB/s | 162 kB 00:00 ", "(53/83): cryptsetup-libs-2.2.0-2.el8.x86_64.rpm 7.4 MB/s | 422 kB 00:00 ", "(54/83): mcelog-162-2.el8.x86_64.rpm 1.7 MB/s | 79 kB 00:00 ", "(55/83): sssd-common-pac-2.2.0-19.el8.x86_64.rp 4.3 MB/s | 161 kB 00:00 ", "(56/83): vdo-6.2.1.134-11.el8.x86_64.rpm 9.9 MB/s | 689 kB 00:00 ", "(57/83): subscription-manager-rhsm-certificates 4.7 MB/s | 245 kB 00:00 ", "(58/83): libblkid-2.32.1-17.el8.x86_64.rpm 6.0 MB/s | 215 kB 00:00 ", "(59/83): mtr-0.92-3.el8.x86_64.rpm 2.5 MB/s | 96 kB 00:00 ", "(60/83): cyrus-sasl-gssapi-2.1.27-1.el8.x86_64. 1.0 MB/s | 49 kB 00:00 ", "(61/83): cyrus-sasl-lib-2.1.27-1.el8.x86_64.rpm 2.9 MB/s | 124 kB 00:00 ", "(62/83): device-mapper-1.02.163-5.el8.x86_64.rp 5.6 MB/s | 371 kB 00:00 ", "(63/83): blktrace-1.2.0-10.el8.x86_64.rpm 3.2 MB/s | 150 kB 00:00 ", "(64/83): libfdisk-2.32.1-17.el8.x86_64.rpm 6.1 MB/s | 249 kB 00:00 ", "(65/83): sssd-proxy-2.2.0-19.el8.x86_64.rpm 3.9 MB/s | 126 kB 00:00 ", "(66/83): dnf-plugin-subscription-manager-1.25.1 4.4 MB/s | 272 kB 00:00 ", "(67/83): kmod-kvdo-6.2.1.138-57.el8.x86_64.rpm 7.3 MB/s | 355 kB 00:00 ", "(68/83): python3-subscription-manager-rhsm-1.25 5.3 MB/s | 338 kB 00:00 ", "(69/83): subscription-manager-1.25.17-1.el8.x86 19 MB/s | 1.1 MB 00:00 ", "(70/83): sssd-kcm-2.2.0-19.el8.x86_64.rpm 4.5 MB/s | 213 kB 00:00 ", "(71/83): sssd-krb5-common-2.2.0-19.el8.x86_64.r 3.6 MB/s | 172 kB 00:00 ", "(72/83): chrony-3.5-1.el8.x86_64.rpm 5.9 MB/s | 271 kB 00:00 ", "(73/83): cyrus-sasl-plain-2.1.27-1.el8.x86_64.r 1.0 MB/s | 47 kB 00:00 ", "(74/83): sssd-krb5-2.2.0-19.el8.x86_64.rpm 2.6 MB/s | 126 kB 00:00 ", "(75/83): sssd-ad-2.2.0-19.el8.x86 2020-02-05T16:52:29.909 INFO:teuthology.task.ansible.out:_64.rpm 4.8 MB/s | 229 kB 00:00 ", "(76/83): subscription-manager-plugin-container- 6.3 MB/s | 248 kB 00:00 ", "(77/83): lvm2-libs-2.03.05-5.el8.x86_64.rpm 13 MB/s | 1.1 MB 00:00 ", "(78/83): device-mapper-libs-1.02.163-5.el8.x86_ 6.4 MB/s | 403 kB 00:00 ", "(79/83): systemd-libs-239-18.el8_1.2.x86_64.rpm 10 MB/s | 560 kB 00:00 ", "(80/83): systemd-pam-239-18.el8_1.2.x86_64.rpm 5.4 MB/s | 230 kB 00:00 ", "(81/83): systemd-udev-239-18.el8_1.2.x86_64.rpm 19 MB/s | 1.3 MB 00:00 ", "(82/83): bpftool-4.18.0-147.5.1.el8_1.x86_64.rp 24 MB/s | 2.1 MB 00:00 ", "(83/83): systemd-239-18.el8_1.2.x86_64.rpm 28 MB/s | 3.5 MB 00:00 ", "--------------------------------------------------------------------------------", "Total 25 MB/s | 42 MB 00:01 ", "Copr repo for ceph-el8 owned by ktdreyer 955 B/s | 1.0 kB 00:01 ", "Key imported successfully", "Running transaction check", "Transaction check succeeded.", "Running transaction test", "Transaction test succeeded.", "Running transaction", 2020-02-05T16:52:29.916 INFO:teuthology.task.ansible.out:" Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 1/1 ", " Preparing : 1/1 ", " Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/1 ", " Upgrading : libuuid-2.32.1-17.el8.x86_64 1/166 ", " Running scriptlet: libuuid-2.32.1-17.el8.x86_64 1/166 ", " Upgrading : libblkid-2.32.1-17.el8.x86_64 2/166 ", " Running scriptlet: libblkid-2.32.1-17.el8.x86_64 2/166 ", " Upgrading : libsss_idmap-2.2.0-19.el8.x86_64 3/166 ", " Running scriptlet: libsss_idmap-2.2.0-19.el8.x86_64 3/166 ", " Upgrading : libmount-2.32.1-17.el8.x86_64 4/166 ", " Running scriptlet: li 2020-02-05T16:52:29.917 INFO:teuthology.task.ansible.out:bmount-2.32.1-17.el8.x86_64 4/166 ", " Upgrading : systemd-libs-239-18.el8_1.2.x86_64 5/166 ", " Running scriptlet: systemd-libs-239-18.el8_1.2.x86_64 5/166 ", " Upgrading : dbus-libs-1:1.12.8-9.el8.x86_64 6/166 ", " Running scriptlet: dbus-libs-1:1.12.8-9.el8.x86_64 6/166 ", " Upgrading : cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 ", " Running scriptlet: cyrus-sasl-lib-2.1.27-1.el8.x86_64 7/166 ", " Upgrading : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 8/166 ", " Upgrading : dbus-tools-1:1.12.8-9.el8.x86_64 9/166 ", " Upgrading : sssd-client-2.2.0-19.el8.x86_64 10/166 ", " Running scriptlet: sssd-client-2.2.0-19.el8.x86_64 10/166 ", " Upgrading : libfdisk-2.32.1-17.el8.x86_64 11/166 ", " Running scriptlet: libfdisk-2.32.1-17.el8.x86_64 11/166 ", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Upgrading : kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 12/166 ", " Upgrading : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 13/166 ", " Upgrading : subscription-manager-rhsm-certificates-1.25.17-1 14/166 ", " Upgrading : python3-subscription-manager-rhsm-1.25.17-1.el8. 15/166 ", " Upgrading : libteam-1.28-4.el8.x86_64 16/166 ", " Running scriptlet: libteam-1.28-4.el8.x86_64 16/166 ", " Upgrading : libipa_hbac-2.2.0-19.el8.x86_64 17/166 ", " Running scriptlet: libipa_hbac-2.2.0-19.el8.x86_64 17/166 ", " Upgrading : redhat-release-eula-8.1-3.3.el8.x86_64 18/166 ", " Upgra 2020-02-05T16:52:29.919 INFO:teuthology.task.ansible.out:ding : python3-sssdconfig-2.2.0-19.el8.noarch 19/166 ", " Upgrading : libsmartcols-2.32.1-17.el8.x86_64 20/166 ", " Running scriptlet: libsmartcols-2.32.1-17.el8.x86_64 20/166 ", " Upgrading : util-linux-2.32.1-17.el8.x86_64 21/166 ", " Running scriptlet: util-linux-2.32.1-17.el8.x86_64 21/166 ", " Upgrading : dbus-common-1:1.12.8-9.el8.noarch 22/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Upgrading : dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-9.el8.x86_64 23/166 ", " Upgrading : device-mapper-8:1.02.163-5.el8.x86_64 24/166 ", " Upgrading : device-mapper-libs-8:1.02.163-5.el8.x86_64 25/166 ", " Upgrading : cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 ", " Running scriptlet: cryptsetup-libs-2.2.0-2.el8.x86_64 26/166 ", " Upgrading : systemd-pam-239-18.el8_1.2.x86_64 27/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 ", " Upgrading : systemd-239-18.el8_1.2.x86_64 28/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 28/166 ", " Upgrading : dbus-1:1.12.8-9.el8.x86_64 29/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 ", " Upgrading : sssd-common-2.2.0-19.el8.x86_64 30/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 30/166 ", " Running scriptlet: sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 ", " Upgrading : sssd-krb5-common-2.2.0-19.el8.x86_64 31/166 ", " Upgrading : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 32 2020-02-05T16:52:29.920 INFO:teuthology.task.ansible.out:/166 ", " Upgrading : rsyslog-8.37.0-13.el8.x86_64 33/166 ", " Running scriptlet: rsyslog-8.37.0-13.el8.x86_64 33/166 ", " Upgrading : sssd-common-pac-2.2.0-19.el8.x86_64 34/166 ", " Upgrading : python3-libstoragemgmt-1.8.1-2.el8.noarch 35/166 ", " Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Upgrading : libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Running scriptlet: libstoragemgmt-1.8.1-2.el8.x86_64 36/166 ", " Upgrading : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 37/166 ", " Upgrading : subscription-manager-1.25.17-1.el8.x86_64 38/166 ", " Running scriptlet: subscription-manager-1.25.17-1.el8.x86_64 38/166 ", " Upgrading : subscription-manager-cockpit-1.25.17-1.el8.noarc 39/166 ", " Running scriptlet: sssd-ipa-2.2.0-19.el8.x86_64 40/166 ", " Upgrading : sssd-ipa-2.2.0-19.el8.x86_64 40/166 ", " Upgrading : sssd-ad-2.2.0-19.el8.x86_64 41/166 ", " Upgrading : device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 ", " Running scriptlet: device-mapper-event-8:1.02.163-5.el8.x86_64 42/166 ", " Upgrading : lvm2-libs-8:2.03.05-5.el8.x86_64 43/166 ", " Upgrading : lvm2-8:2.03.05-5.el8.x86_64 44/166 ", " Running scriptlet: lvm2-8:2.03.05-5.el8.x86_64 44/166 ", "warning: /etc/lvm/lvm.conf created as /etc/lvm/lvm.conf.rpmnew", "", " Upgrading : sssd-ldap-2.2.0-19.el8.x86_64 45/166 ", " Upgrading : sssd-krb5-2.2.0-19.el8.x86_64 46/166 ", " Running scriptlet: sssd-proxy-2.2.0-19.el8.x86_64 47/166 ", " Upgrading : sssd-proxy-2.2.0-19.el8.x86_64 2020-02-05T16:52:29.922 INFO:teuthology.task.ansible.out:47/166 ", " Upgrading : vim-common-2:8.0.1763-13.el8.x86_64 48/166 ", " Upgrading : vim-enhanced-2:8.0.1763-13.el8.x86_64 49/166 ", " Upgrading : sssd-2.2.0-19.el8.x86_64 50/166 ", " Upgrading : vdo-6.2.1.134-11.el8.x86_64 51/166 ", " Running scriptlet: vdo-6.2.1.134-11.el8.x86_64 51/166 ", " Upgrading : cockpit-196.3-1.el8.x86_64 52/166 ", " Upgrading : subscription-manager-plugin-container-1.25.17-1. 53/166 ", " Running scriptlet: subscription-manager-plugin-container-1.25.17-1. 53/166 ", " Upgrading : rsyslog-gnutls-8.37.0-13.el8.x86_64 54/166 ", " Upgrading : rsyslog-gssapi-8.37.0-13.el8.x86_64 55/166 ", " Upgrading : rsyslog-relp-8.37.0-13.el8.x86_64 56/166 ", " Upgrading : sssd-kcm-2.2.0-19.el8.x86_64 57/166 ", " Running scriptlet: sssd-kcm-2.2.0-19.el8.x86_64 57/166 ", " Upgrading : smartmontools-1:7.1-7.el8.x86_64 58/166 ", " Running scriptlet: smartmontools-1:7.1-7.el8.x86_64 58/166 ", " Upgrading : insights-client-3.0.8-4.el8_1.noarch 59/166 ", " Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 59/166 ", " Upgrading : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 ", " Running scriptlet: microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 60/166 ", " Upgrading : mdadm-4.1-9.el8.x86_64 61/166 ", " Running scriptlet: mdadm-4.1-9.el8.x86_64 61/166 ", " Upgrading : mcelog-3:162-2.el8.x86_64 62/166 ", " Running scriptlet: mcelog-3:162-2.el8.x86_64 62/166 ", " Running scriptlet: chrony-3.5-1.el8.x86_64 2020-02-05T16:52:29.923 INFO:teuthology.task.ansible.out: 63/166 ", " Upgrading : chrony-3.5-1.el8.x86_64 63/166 ", " Running scriptlet: chrony-3.5-1.el8.x86_64 63/166 ", " Upgrading : systemd-udev-239-18.el8_1.2.x86_64 64/166 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 64/166 ", " Upgrading : cryptsetup-2.2.0-2.el8.x86_64 65/166 ", " Upgrading : util-linux-user-2.32.1-17.el8.x86_64 66/166 ", " Upgrading : redhat-release-8.1-3.3.el8.x86_64 67/166 ", " Upgrading : teamd-1.28-4.el8.x86_64 68/166 ", " Upgrading : cyrus-sasl-plain-2.1.27-1.el8.x86_64 69/166 ", " Upgrading : plymouth-0.9.3-15.el8.x86_64 70/166 ", " Upgrading : ledmon-0.92-7.el8.x86_64 71/166 ", " Upgrading : wget-1.19.5-8.el8_1.1.x86_64 72/166 ", " Running scriptlet: wget-1.19.5-8.el8_1.1.x86_64 72/166 ", " Upgrading : bpftool-4.18.0-147.5.1.el8_1.x86_64 73/166 ", " Upgrading : blktrace-1.2.0-10.el8.x86_64 74/166 ", " Upgrading : mtr-2:0.92-3.el8.x86_64 75/166 ", " Upgrading : strace-4.24-5.el8.x86_64 76/166 ", " Upgrading : sudo-1.8.25p1-8.el8_1.x86_64 77/166 ", " Running scriptlet: sudo-1.8.25p1-8.el8_1.x86_64 77/166 ", " Upgrading : sos-3.7-8.el8_1.noarch 78/166 ", " Upgrading : ethtool-2:5.0-2.el8.x86_64 79/166 ", " Upgrading : irqbalance-2:1.4.0-4.el8.x86_64 80/166 ", " Running scriptlet: irqbalance-2:1.4.0-4.el8.x86_64 80/166 ", " Upgrading : rsync-3.1.3-6.el8.x8 2020-02-05T16:52:29.925 INFO:teuthology.task.ansible.out:6_64 81/166 ", " Upgrading : kpatch-0.6.1-5.el8.noarch 82/166 ", " Upgrading : nmap-ncat-2:7.70-5.el8.x86_64 83/166 ", " Running scriptlet: nmap-ncat-2:7.70-5.el8.x86_64 83/166 ", " Cleanup : sssd-2.0.0-43.el8_0.3.x86_64 84/166 ", " Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Cleanup : insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Running scriptlet: insights-client-3.0.8-0.el8_0.noarch 85/166 ", " Cleanup : sssd-ipa-2.0.0-43.el8_0.3.x86_64 86/166 ", " Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Cleanup : sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Running scriptlet: sssd-kcm-2.0.0-43.el8_0.3.x86_64 87/166 ", " Cleanup : sssd-ad-2.0.0-43.el8_0.3.x86_64 88/166 ", " Cleanup : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 89/166 ", " Cleanup : sssd-ldap-2.0.0-43.el8_0.3.x86_64 90/166 ", " Cleanup : systemd-udev-239-13.el8_0.5.x86_64 91/166 ", " Running scriptlet: systemd-udev-239-13.el8_0.5.x86_64 91/166 ", " Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Cleanup : vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Running scriptlet: vdo-6.2.0.298-10.el8_0.x86_64 92/166 ", " Running scriptlet: lvm2-8:2.03.02-6.el8 2020-02-05T16:52:29.925 INFO:teuthology.task.ansible.out:.x86_64 93/166 ", " Cleanup : lvm2-8:2.03.02-6.el8.x86_64 93/166 ", " Running scriptlet: lvm2-8:2.03.02-6.el8.x86_64 93/166 ", " Cleanup : lvm2-libs-8:2.03.02-6.el8.x86_64 94/166 ", " Running scriptlet: lvm2 2020-02-05T16:52:29.926 INFO:teuthology.task.ansible.out:-libs-8:2.03.02-6.el8.x86_64 94/166 ", " Running scriptlet: device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 ", " Cleanup : device-mapper-event-8:1.02.155-6.el8.x86_64 95/166 ", " Cleanup : sssd-krb5-2.0.0-43.el8_0.3.x86_64 96/166 ", " Cleanup : sssd-proxy-2.0.0-43.el8_0.3.x86_64 97/166 ", " Cleanup : cryptsetup-2.0.6-1.el8.x86_64 98/166 ", " Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Cleanup : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Running scriptlet: microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 99/166 ", " Cleanup : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 100/166 ", " Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Cleanup : sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Running scriptlet: sssd-common-2.0.0-43.el8_0.3.x86_64 101/166 ", " Cleanup : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 ", " Running scriptlet: device-mapper-event-libs-8:1.02.155-6.el8.x86_64 102/166 ", " Cleanup : teamd-1.27-10.el8.x86_64 103/166 ", " Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 ", " Cleanup : chrony-3.3-3.el8.x86_64 104/166 ", " Running scriptlet: chrony-3.3-3.el8.x86_64 104/166 ", " Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 ", " Cleanup : mcelog-3:159-1.el8.x86_64 105/166 ", " Running scriptlet: mcelog-3:159-1.el8.x86_64 105/166 ", " Running scriptlet: mdadm-4.1-4.el8.x86_64 106/166 ", " Cleanup : mdadm-4.1-4.el8.x86_64 106/166 ", " Running 2020-02-05T16:52:29.928 INFO:teuthology.task.ansible.out: scriptlet: mdadm-4.1-4.el8.x86_64 106/166 ", " Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Cleanup : smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Running scriptlet: smartmontools-1:6.6-3.el8.x86_64 107/166 ", " Cleanup : python3-libstoragemgmt-1.6.2-9.el8.noarch 108/166 ", " Cleanup : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 109/166 ", " Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Cleanup : libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Running scriptlet: libstoragemgmt-1.6.2-9.el8.x86_64 110/166 ", " Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Cleanup : sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Running scriptlet: sssd-client-2.0.0-43.el8_0.3.x86_64 111/166 ", " Cleanup : systemd-pam-239-13.el8_0.5.x86_64 112/166 ", " Cleanup : ledmon-0.90-3.el8.x86_64 113/166 ", " Running scriptlet: wget-1.19.5-7.el8_0.1.x86_64 114/166 ", " Cleanup : wget-1.19.5-7.el8_0.1.x86_64 114/166 ", " Cleanup : subscription-manager-plugin-container-1.23.8-35. 115/166 ", " Cleanup : cockpit-185.1-1.el8_0.x86_64 116/166 ", " Cleanup : subscription-manager-cockpit-1.23.8-35.el8.noarc 117/166 ", " Cleanup : redhat-release-8.0-0.44.el8.x86_64 118/166 ", " Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 ", " Cleanup : subscription-manager-1.23.8-35.el8.x86_64 119/166 ", " Running scriptlet: subscription-manager-1.23.8-35.el8.x86_64 119/166 ", " Cleanup : python3-subscription-manager-rhsm-1.23.8-35.el8. 120/1 2020-02-05T16:52:29.929 INFO:teuthology.task.ansible.out:66 ", " Cleanup : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 121/166 ", " Cleanup : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 122/166 ", " Cleanup : util-linux-user-2.32.1-8.el8.x86_64 123/166 ", " Cleanup : rsyslog-relp-8.37.0-9.el8.x86_64 124/166 ", " Cleanup : plymouth-0.9.3-12.el8.x86_64 125/166 ", " Running scriptlet: plymouth-0.9.3-12.el8.x86_64 125/166 ", " Cleanup : rsyslog-gssapi-8.37.0-9.el8.x86_64 126/166 ", " Cleanup : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 127/166 ", " Cleanup : rsyslog-gnutls-8.37.0-9.el8.x86_64 128/166 ", " Cleanup : subscription-manager-rhsm-certificates-1.23.8-35 129/166 ", " Cleanup : redhat-release-eula-8.0-0.44.el8.x86_64 130/166 ", " Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Cleanup : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Running scriptlet: kmod-kvdo-6.2.0.293-53.el8_0.x86_64 131/166 ", " Cleanup : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 132/166 ", " Cleanup : sos-3.6-10.el8_0.3.noarch 133/166 ", " Cleanup : kpatch-0.6.1-1.el8.noarch 134/166 ", " Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Cleanup : rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Running scriptlet: rsyslog-8.37.0-9.el8.x86_64 135/166 ", " Cleanup : cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 ", " Running scriptlet: cryptsetup-libs-2.0.6-1.el8.x86_64 136/166 ", " Cleanup : device-mapper-libs-8:1.02.155-6.el8.x86_64 137/166 ", " Running scriptlet: device-mapper-libs-8:1.02.155-6.el8.x86_ 2020-02-05T16:52:29.931 INFO:teuthology.task.ansible.out:64 137/166 ", " Cleanup : device-mapper-8:1.02.155-6.el8.x86_64 138/166 ", " Cleanup : dbus-1:1.12.8-7.el8.x86_64 139/166 ", " Running scriptlet: systemd-239-13.el8_0.5.x86_64 140/166 ", " Cleanup : systemd-239-13.el8_0.5.x86_64 140/166 ", " Cleanup : util-linux-2.32.1-8.el8.x86_64 141/166 ", " Cleanup : libfdisk-2.32.1-8.el8.x86_64 142/166 ", " Running scriptlet: libfdisk-2.32.1-8.el8.x86_64 142/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Cleanup : dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Running scriptlet: dbus-daemon-1:1.12.8-7.el8.x86_64 143/166 ", " Cleanup : dbus-tools-1:1.12.8-7.el8.x86_64 144/166 ", " Cleanup : dbus-common-1:1.12.8-7.el8.noarch 145/166 ", " Cleanup : dbus-libs-1:1.12.8-7.el8.x86_64 146/166 ", " Running scriptlet: dbus-libs-1:1.12.8-7.el8.x86_64 146/166 ", " Cleanup : systemd-libs-239-13.el8_0.5.x86_64 147/166 ", " Cleanup : libmount-2.32.1-8.el8.x86_64 148/166 ", " Running scriptlet: libmount-2.32.1-8.el8.x86_64 148/166 ", " Cleanup : libblkid-2.32.1-8.el8.x86_64 149/166 ", " Running scriptlet: libblkid-2.32.1-8.el8.x86_64 149/166 " 2020-02-05T16:52:29.931 INFO:teuthology.task.ansible.out:, " Cleanup : libuuid-2.32.1-8.el8.x86_64 150/166 ", " Running scriptlet: libuuid-2.32.1-8.el8.x86_64 150/166 ", " Cleanup : libsmartcols-2.32.1-8.el8.x86_64 151/166 ", " Running scriptlet: libsmartcols-2.32.1-8.el8.x86_64 151/166 ", " Cleanup : vim-common-2:8.0.1763-11 2020-02-05T16:52:29.932 INFO:teuthology.task.ansible.out:.el8_0.x86_64 152/166 ", " Cleanup : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 ", " Running scriptlet: cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 153/166 ", " Cleanup : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 154/166 ", " Cleanup : libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 ", " Running scriptlet: libsss_idmap-2.0.0-43.el8_0.3.x86_64 155/166 ", " Cleanup : libteam-1.27-10.el8.x86_64 156/166 ", " Running scriptlet: libteam-1.27-10.el8.x86_64 156/166 ", " Cleanup : libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 ", " Running scriptlet: libipa_hbac-2.0.0-43.el8_0.3.x86_64 157/166 ", " Cleanup : bpftool-4.18.0-80.11.2.el8_0.x86_64 158/166 ", " Cleanup : blktrace-1.2.0-9.el8.x86_64 159/166 ", " Cleanup : mtr-2:0.92-1.el8.x86_64 160/166 ", " Cleanup : strace-4.24-3.el8.x86_64 161/166 ", " Cleanup : sudo-1.8.25p1-4.el8_0.1.x86_64 162/166 ", " Cleanup : ethtool-2:4.16-1.el8.x86_64 163/166 ", " Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 ", " Cleanup : irqbalance-2:1.4.0-2.el8.x86_64 164/166 ", " Running scriptlet: irqbalance-2:1.4.0-2.el8.x86_64 164/166 ", " Cleanup : rsync-3.1.3-4.el8.x86_64 165/166 ", " Cleanup : nmap-ncat-2:7.70-4.el8.x86_64 166/166 ", " Running scriptlet: kmod-kvdo-6.2.1.138-57.el8.x86_64 166/166 ", " Running scriptlet: sssd-common-2.2.0-19.el8.x86_64 166/166 ", " Running scriptlet: insights-client-3.0.8-4.el8_1.noarch 166/166 ", " Running scriptlet: microcod 2020-02-05T16:52:29.934 INFO:teuthology.task.ansible.out:e_ctl-4:20190618-1.20191115.3.el8_1.x86_ 166/166 ", " Running scriptlet: nmap-ncat-2:7.70-4.el8.x86_64 166/166 ", " Running scriptlet: systemd-239-18.el8_1.2.x86_64 166/166 ", " Running scriptlet: vim-common-2:8.0.1763-13.el8.x86_64 166/166 ", " Running scriptlet: systemd-udev-239-18.el8_1.2.x86_64 166/166 ", " Verifying : smartmontools-1:7.1-7.el8.x86_64 1/166 ", " Verifying : smartmontools-1:6.6-3.el8.x86_64 2/166 ", " Verifying : rsyslog-8.37.0-13.el8.x86_64 3/166 ", " Verifying : rsyslog-8.37.0-9.el8.x86_64 4/166 ", " Verifying : insights-client-3.0.8-4.el8_1.noarch 5/166 ", " Verifying : insights-client-3.0.8-0.el8_0.noarch 6/166 ", " Verifying : wget-1.19.5-8.el8_1.1.x86_64 7/166 ", " Verifying : wget-1.19.5-7.el8_0.1.x86_64 8/166 ", " Verifying : rsyslog-gnutls-8.37.0-13.el8.x86_64 9/166 ", " Verifying : rsyslog-gnutls-8.37.0-9.el8.x86_64 10/166 ", " Verifying : vim-common-2:8.0.1763-13.el8.x86_64 11/166 ", " Verifying : vim-common-2:8.0.1763-11.el8_0.x86_64 12/166 ", " Verifying : vim-enhanced-2:8.0.1763-13.el8.x86_64 13/166 ", " Verifying : vim-enhanced-2:8.0.1763-11.el8_0.x86_64 14/166 ", " Verifying : rsyslog-gssapi-8.37.0-13.el8.x86_64 15/166 ", " Verifying : rsyslog-gssapi-8.37.0-9.el8.x86_64 16/166 ", " Verifying : nmap-ncat-2:7.70-5.el8.x86_64 17/166 ", " Verifying : nmap-ncat-2:7.70-4.el8.x86_64 18/166 ", " Verifying : plymouth-0.9.3-15.el8.x86_64 19/166 ", " Verifying 2020-02-05T16:52:29.935 INFO:teuthology.task.ansible.out: : plymouth-0.9.3-12.el8.x86_64 20/166 ", " Verifying : rsyslog-relp-8.37.0-13.el8.x86_64 21/166 ", " Verifying : rsyslog-relp-8.37.0-9.el8.x86_64 22/166 ", " Verifying : sssd-ldap-2.2.0-19.el8.x86_64 23/166 ", " Verifying : sssd-ldap-2.0.0-43.el8_0.3.x86_64 24/166 ", " Verifying : dbus-common-1:1.12.8-9.el8.noarch 25/166 ", " Verifying : dbus-common-1:1.12.8-7.el8.noarch 26/166 ", " Verifying : ledmon-0.92-7.el8.x86_64 27/166 ", " Verifying : ledmon-0.90-3.el8.x86_64 28/166 ", " Verifying : cryptsetup-2.2.0-2.el8.x86_64 29/166 ", " Verifying : cryptsetup-2.0.6-1.el8.x86_64 30/166 ", " Verifying : microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_ 31/166 ", " Verifying : microcode_ctl-4:20180807a-2.20190618.1.el8_0.x86 32/166 ", " Verifying : util-linux-2.32.1-17.el8.x86_64 33/166 ", " Verifying : util-linux-2.32.1-8.el8.x86_64 34/166 ", " Verifying : lvm2-8:2.03.05-5.el8.x86_64 35/166 ", " Verifying : lvm2-8:2.03.02-6.el8.x86_64 36/166 ", " Verifying : util-linux-user-2.32.1-17.el8.x86_64 37/166 ", " Verifying : util-linux-user-2.32.1-8.el8.x86_64 38/166 ", " Verifying : kpatch-0.6.1-5.el8.noarch 39/166 ", " Verifying : kpatch-0.6.1-1.el8.noarch 40/166 ", " Verifying : dbus-daemon-1:1.12.8-9.el8.x86_64 41/166 ", " Verifying : dbus-daemon-1:1.12.8-7.el8.x86_64 42/166 ", " Verifying : redhat-release-8.1-3.3.el8.x86_64 43/166 " 2020-02-05T16:52:29.937 INFO:teuthology.task.ansible.out:, " Verifying : redhat-release-8.0-0.44.el8.x86_64 44/166 ", " Verifying : device-mapper-event-libs-8:1.02.163-5.el8.x86_64 45/166 ", " Verifying : device-mapper-event-libs-8:1.02.155-6.el8.x86_64 46/166 ", " Verifying : libmount-2.32.1-17.el8.x86_64 47/166 ", " Verifying : libmount-2.32.1-8.el8.x86_64 48/166 ", " Verifying : libsmartcols-2.32.1-17.el8.x86_64 49/166 ", " Verifying : libsmartcols-2.32.1-8.el8.x86_64 50/166 ", " Verifying : python3-sssdconfig-2.2.0-19.el8.noarch 51/166 ", " Verifying : python3-sssdconfig-2.0.0-43.el8_0.3.noarch 52/166 ", " Verifying : subscription-manager-cockpit-1.25.17-1.el8.noarc 53/166 ", " Verifying : subscription-manager-cockpit-1.23.8-35.el8.noarc 54/166 ", " Verifying : python3-libstoragemgmt-1.8.1-2.el8.noarch 55/166 ", " Verifying : python3-libstoragemgmt-1.6.2-9.el8.noarch 56/166 ", " Verifying : libstoragemgmt-1.8.1-2.el8.x86_64 57/166 ", " Verifying : libstoragemgmt-1.6.2-9.el8.x86_64 58/166 ", " Verifying : libuuid-2.32.1-17.el8.x86_64 59/166 ", " Verifying : libuuid-2.32.1-8.el8.x86_64 60/166 ", " Verifying : dbus-libs-1:1.12.8-9.el8.x86_64 61/166 ", " Verifying : dbus-libs-1:1.12.8-7.el8.x86_64 62/166 ", " Verifying : rsync-3.1.3-6.el8.x86_64 63/166 ", " Verifying : rsync-3.1.3-4.el8.x86_64 64/166 ", " Verifying : redhat-release-eula-8.1-3.3.el8.x86_64 65/166 ", " Verifying : redhat-release-eula-8.0-0.44.el8.x86_64 66/166 ", " Verifying : irqbalance-2:1.4.0-4.el8.x86_64 2020-02-05T16:52:29.938 INFO:teuthology.task.ansible.out: 67/166 ", " Verifying : irqbalance-2:1.4.0-2.el8.x86_64 68/166 ", " Verifying : ethtool-2:5.0-2.el8.x86_64 69/166 ", " Verifying : ethtool-2:4.16-1.el8.x86_64 70/166 ", " Verifying : libipa_hbac-2.2.0-19.el8.x86_64 71/166 ", " Verifying : libipa_hbac-2.0.0-43.el8_0.3.x86_64 72/166 ", " Verifying : device-mapper-event-8:1.02.163-5.el8.x86_64 73/166 ", " Verifying : device-mapper-event-8:1.02.155-6.el8.x86_64 74/166 ", " Verifying : libteam-1.28-4.el8.x86_64 75/166 ", " Verifying : libteam-1.27-10.el8.x86_64 76/166 ", " Verifying : sos-3.7-8.el8_1.noarch 77/166 ", " Verifying : sos-3.6-10.el8_0.3.noarch 78/166 ", " Verifying : sudo-1.8.25p1-8.el8_1.x86_64 79/166 ", " Verifying : sudo-1.8.25p1-4.el8_0.1.x86_64 80/166 ", " Verifying : dbus-1:1.12.8-9.el8.x86_64 81/166 ", " Verifying : dbus-1:1.12.8-7.el8.x86_64 82/166 ", " Verifying : sssd-ipa-2.2.0-19.el8.x86_64 83/166 ", " Verifying : sssd-ipa-2.0.0-43.el8_0.3.x86_64 84/166 ", " Verifying : sssd-2.2.0-19.el8.x86_64 85/166 ", " Verifying : sssd-2.0.0-43.el8_0.3.x86_64 86/166 ", " Verifying : dbus-tools-1:1.12.8-9.el8.x86_64 87/166 ", " Verifying : dbus-tools-1:1.12.8-7.el8.x86_64 88/166 ", " Verifying : sssd-common-2.2.0-19.el8.x86_64 89/166 ", " Verifying : sssd-common-2.0.0-43.el8_0.3.x86_64 90/166 ", " Verifying : mdadm-4.1-9.el8.x86_64 2020-02-05T16:52:29.940 INFO:teuthology.task.ansible.out: 91/166 ", " Verifying : mdadm-4.1-4.el8.x86_64 92/166 ", " Verifying : libsss_idmap-2.2.0-19.el8.x86_64 93/166 ", " Verifying : libsss_idmap-2.0.0-43.el8_0.3.x86_64 94/166 ", " Verifying : teamd-1.28-4.el8.x86_64 95/166 ", " Verifying : teamd-1.27-10.el8.x86_64 96/166 ", " Verifying : strace-4.24-5.el8.x86_64 97/166 ", " Verifying : strace-4.24-3.el8.x86_64 98/166 ", " Verifying : cockpit-196.3-1.el8.x86_64 99/166 ", " Verifying : cockpit-185.1-1.el8_0.x86_64 100/166 ", " Verifying : python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 101/166 ", " Verifying : python3-libstoragemgmt-clibs-1.6.2-9.el8.x86_64 102/166 ", " Verifying : sssd-client-2.2.0-19.el8.x86_64 103/166 ", " Verifying : sssd-client-2.0.0-43.el8_0.3.x86_64 104/166 ", " Verifying : cryptsetup-libs-2.2.0-2.el8.x86_64 105/166 ", " Verifying : cryptsetup-libs-2.0.6-1.el8.x86_64 106/166 ", " Verifying : mcelog-3:162-2.el8.x86_64 107/166 ", " Verifying : mcelog-3:159-1.el8.x86_64 108/166 ", " Verifying : vdo-6.2.1.134-11.el8.x86_64 109/166 ", " Verifying : vdo-6.2.0.298-10.el8_0.x86_64 110/166 ", " Verifying : sssd-common-pac-2.2.0-19.el8.x86_64 111/166 ", " Verifying : sssd-common-pac-2.0.0-43.el8_0.3.x86_64 112/166 ", " Verifying : subscription-manager-rhsm-certificates-1.25.17-1 113/166 ", " Verifying : subscription-manager-rhsm-certificates-1.23.8-35 114/166 ", " Verifying : libblkid-2.3 2020-02-05T16:52:29.941 INFO:teuthology.task.ansible.out:2.1-17.el8.x86_64 115/166 ", " Verifying : libblkid-2.32.1-8.el8.x86_64 116/166 ", " Verifying : mtr- 2020-02-05T16:52:29.941 INFO:teuthology.task.ansible.out:2:0.92-3.el8.x86_64 117/166 ", " Verifying : mtr-2:0.92-1.el8.x86_64 118/166 ", " Verifying : cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 119/166 ", " Verifying : cyrus-sasl-gssapi-2.1.27-0.3rc7.el8.x86_64 120/166 ", " Verifying : cyrus-sasl-lib-2.1.27-1.el8.x86_64 121/166 ", " Verifying : cyrus-sasl-lib-2.1.27-0.3rc7.el8.x86_64 122/166 ", " Verifying : device-mapper-8:1.02.163-5.el8.x86_64 123/166 ", " Verifying : device-mapper-8:1.02.155-6.el8.x86_64 124/166 ", " Verifying : blktrace-1.2.0-10.el8.x86_64 125/166 ", " Verifying : blktrace-1.2.0-9.el8.x86_64 126/166 ", " Verifying : libfdisk-2.32.1-17.el8.x86_64 127/166 ", " Verifying : libfdisk-2.32.1-8.el8.x86_64 128/166 ", " Verifying : sssd-proxy-2.2.0-19.el8.x86_64 129/166 ", " Verifying : sssd-proxy-2.0.0-43.el8_0.3.x86_64 130/166 ", " Verifying : dnf-plugin-subscription-manager-1.25.17-1.el8.x8 131/166 ", " Verifying : dnf-plugin-subscription-manager-1.23.8-35.el8.x8 132/166 ", " Verifying : kmod-kvdo-6.2.1.138-57.el8.x86_64 133/166 ", " Verifying : kmod-kvdo-6.2.0.293-53.el8_0.x86_64 134/166 ", " Verifying : python3-subscription-manager-rhsm-1.25.17-1.el8. 135/166 ", " Verifying : python3-subscription-manager-rhsm-1.23.8-35.el8. 136/166 ", " Verifying : subscription-manager-1.25.17-1.el8.x86_64 137/166 ", " Verifying : subscription-manager-1.23.8-35.el8.x86_64 138/166 ", " Verifying 2020-02-05T16:52:29.943 INFO:teuthology.task.ansible.out: : sssd-kcm-2.2.0-19.el8.x86_64 139/166 ", " Verifying : sssd-kcm-2.0.0-43.el8_0.3.x86_64 140/166 ", " Verifying : sssd-krb5-common-2.2.0-19.el8.x86_64 141/166 ", " Verifying : sssd-krb5-common-2.0.0-43.el8_0.3.x86_64 142/166 ", " Verifying : chrony-3.5-1.el8.x86_64 143/166 ", " Verifying : chrony-3.3-3.el8.x86_64 144/166 ", " Verifying : cyrus-sasl-plain-2.1.27-1.el8.x86_64 145/166 ", " Verifying : cyrus-sasl-plain-2.1.27-0.3rc7.el8.x86_64 146/166 ", " Verifying : sssd-krb5-2.2.0-19.el8.x86_64 147/166 ", " Verifying : sssd-krb5-2.0.0-43.el8_0.3.x86_64 148/166 ", " Verifying : sssd-ad-2.2.0-19.el8.x86_64 149/166 ", " Verifying : sssd-ad-2.0.0-43.el8_0.3.x86_64 150/166 ", " Verifying : lvm2-libs-8:2.03.05-5.el8.x86_64 151/166 ", " Verifying : lvm2-libs-8:2.03.02-6.el8.x86_64 152/166 ", " Verifying : subscription-manager-plugin-container-1.25.17-1. 153/166 ", " Verifying : subscription-manager-plugin-container-1.23.8-35. 154/166 ", " Verifying : device-mapper-libs-8:1.02.163-5.el8.x86_64 155/166 ", " Verifying : device-mapper-libs-8:1.02.155-6.el8.x86_64 156/166 ", " Verifying : systemd-libs-239-18.el8_1.2.x86_64 157/166 ", " Verifying : systemd-libs-239-13.el8_0.5.x86_64 158/166 ", " Verifying : systemd-pam-239-18.el8_1.2.x86_64 159/166 ", " Verifying : systemd-pam-239-13.el8_0.5.x86_64 160/166 ", " Verifying : systemd-239-18.el8_1.2.x86_64 161/166 ", " Verifying : systemd-239-13.el8_0.5.x86_64 162/166 ", " 2020-02-05T16:52:29.944 INFO:teuthology.task.ansible.out: Verifying : bpftool-4.18.0-147.5.1.el8_1.x86_64 163/166 ", " Verifying : bpftool-4.18.0-80.11.2.el8_0.x86_64 164/166 ", " Verifying : systemd-udev-239-18.el8_1.2.x86_64 165/166 ", " Verifying : systemd-udev-239-13.el8_0.5.x86_64 166/166 ", "Installed products updated.", "", "Upgraded:", " smartmontools-1:7.1-7.el8.x86_64 ", " rsyslog-8.37.0-13.el8.x86_64 ", " insights-client-3.0.8-4.el8_1.noarch ", " wget-1.19.5-8.el8_1.1.x86_64 ", " rsyslog-gnutls-8.37.0-13.el8.x86_64 ", " vim-common-2:8.0.1763-13.el8.x86_64 ", " vim-enhanced-2:8.0.1763-13.el8.x86_64 ", " rsyslog-gssapi-8.37.0-13.el8.x86_64 ", " nmap-ncat-2:7.70-5.el8.x86_64 ", " plymouth-0.9.3-15.el8.x86_64 ", " rsyslog-relp-8.37.0-13.el8.x86_64 ", " sssd-ldap-2.2.0-19.el8.x86_64 ", " dbus-common-1:1.12.8-9.el8.noarch ", " ledmon-0.92-7.el8.x86_64 ", " cryptsetup-2.2.0-2.el8.x86_64 ", " microcode_ctl-4:20190618-1.20191115.3.el8_1.x86_64 ", " util-linux-2.32.1-17.el8.x86_64 ", " lvm2-8:2.03.05-5.el8.x86_64 ", " util-linux-user-2.32.1-17.el8.x86_64 ", " kpatch-0.6.1-5.el8. 2020-02-05T16:52:29.946 INFO:teuthology.task.ansible.out:noarch ", " dbus-daemon-1:1.12.8-9.el8.x86_64 ", " redhat-release-8.1-3.3.el8.x86_64 ", " device-mapper-event-libs-8:1.02.163-5.el8.x86_64 ", " libmount-2.32.1-17.el8.x86_64 ", " libsmartcols-2.32.1-17.el8.x86_64 ", " python3-sssdconfig-2.2.0-19.el8.noarch ", " subscription-manager-cockpit-1.25.17-1.el8.noarch ", " python3-libstoragemgmt-1.8.1-2.el8.noarch ", " libstoragemgmt-1.8.1-2.el8.x86_64 ", " libuuid-2.32.1-17.el8.x86_64 ", " dbus-libs-1:1.12.8-9.el8.x86_64 ", " rsync-3.1.3-6.el8.x86_64 ", " redhat-release-eula-8.1-3.3.el8.x86_64 ", " irqbalance-2:1.4.0-4.el8.x86_64 ", " ethtool-2:5.0-2.el8.x86_64 ", " libipa_hbac-2.2.0-19.el8.x86_64 ", " device-mapper-event-8:1.02.163-5.el8.x86_64 ", " libteam-1.28-4.el8.x86_64 ", " sos-3.7-8.el8_1.noarch ", " sudo-1.8.25p1-8.el8_1.x86_64 ", " dbus-1:1.12.8-9.el8.x86_64 ", " sssd-ipa-2.2.0-19.el8.x86_64 ", " sssd-2.2.0-19.el8.x86_64 ", " dbu 2020-02-05T16:52:29.947 INFO:teuthology.task.ansible.out:s-tools-1:1.12.8-9.el8.x86_64 ", " sssd-common-2.2.0-19.el8.x86_64 ", " mdadm-4.1-9.el8.x86_64 ", " libsss_idmap-2.2.0-19.el8.x86_64 ", " teamd-1.28-4.el8.x86_64 ", " strace-4.24-5.el8.x86_64 ", " cockpit-196.3-1.el8.x86_64 ", " python3-libstoragemgmt-clibs-1.8.1-2.el8.x86_64 ", " sssd-client-2.2.0-19.el8.x86_64 ", " cryptsetup-libs-2.2.0-2.el8.x86_64 ", " mcelog-3:162-2.el8.x86_64 ", " vdo-6.2.1.134-11.el8.x86_64 ", " sssd-common-pac-2.2.0-19.el8.x86_64 ", " subscription-manager-rhsm-certificates-1.25.17-1.el8.x86_64 ", " libblkid-2.32.1-17.el8.x86_64 ", " mtr-2:0.92-3.el8.x86_64 ", " cyrus-sasl-gssapi-2.1.27-1.el8.x86_64 ", " cyrus-sasl-lib-2.1.27-1.el8.x86_64 ", " device-mapper-8:1.02.163-5.el8.x86_64 ", " blktrace-1.2.0-10.el8.x86_64 ", " libfdisk-2.32.1-17.el8.x86_64 ", " sssd-proxy-2.2.0-19.el8.x86_64 ", " dnf-plugin-subscription-manager-1.25.17-1.el8.x86_64 ", " kmod-kvdo-6.2.1.138-57.el8.x86_64 2020-02-05T16:52:29.949 INFO:teuthology.task.ansible.out: ", " python3-subscription-manager-rhsm-1.25.17-1.el8.x86_64 ", " subscription-manager-1.25.17-1.el8.x86_64 ", " sssd-kcm-2.2.0-19.el8.x86_64 ", " sssd-krb5-common-2.2.0-19.el8.x86_64 ", " chrony-3.5-1.el8.x86_64 ", " cyrus-sasl-plain-2.1.27-1.el8.x86_64 ", " sssd-krb5-2.2.0-19.el8.x86_64 ", " sssd-ad-2.2.0-19.el8.x86_64 ", " lvm2-libs-8:2.03.05-5.el8.x86_64 ", " subscription-manager-plugin-container-1.25.17-1.el8.x86_64 ", " device-mapper-libs-8:1.02.163-5.el8.x86_64 ", " systemd-libs-239-18.el8_1.2.x86_64 ", " systemd-pam-239-18.el8_1.2.x86_64 ", " systemd-239-18.el8_1.2.x86_64 ", " bpftool-4.18.0-147.5.1.el8_1.x86_64 ", " systemd-udev-239-18.el8_1.2.x86_64 ", "", "Complete!"]} 2020-02-05T16:52:29.988 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:52:29 +0000 (0:00:15.148) 0:08:17.317 **** 2020-02-05T16:54:46.881 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-02-05T16:54:46.884 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: valgrind-1:3.15.0-9.el8.x86_64", "Installed: annobin-8.78-1.el8.x86_64", "Installed: celt051-0.5.1.3-15.el8.x86_64", "Installed: gdb-headless-8.2-6.el8_0.x86_64", "Installed: ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch", "Installed: python36-devel-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64", "Installed: platform-python-devel-3.6.8-1.el8.x86_64", "Installed: python3-rpm-4.14.2-25.el8.x86_64", "Installed: parted-3.2-38.el8.x86_64", "Installed: libpmem-1.6-2.el8.x86_64", "Installed: gc-7.6.4-3.el8.x86_64", "Installed: gitweb-2.18.2-1.el8_1.noarch", "Installed: rpm-4.14.2-25.el8.x86_64", "Installed: libss-1.44.6-3.el8.x86_64", "Installed: libacl-devel-2.2.53-1.el8.x86_64", "Installed: iwl2000-firmware-18.168.6.1-94.el8.1.noarch", "Installed: firewalld-0.7.0-5.el8.noarch", "Installed: boost-chrono-1.66.0-6.el8.x86_64", "Installed: gettext-0.19.8.1-17.el8.x86_64", "Installed: libbabeltrace-1.5.6-2.el8.x86_64", "Installed: lshw-B.02.18-21.el8.x86_64", "Installed: libglv 2020-02-05T16:54:46.885 INFO:teuthology.task.ansible.out:nd-egl-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: libwayland-egl-1.15.0-1.el8.x86_64", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: libglvnd-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: boost-atomic-1.66.0-6.el8.x86_64", "Installed: openssh-8.0p1-4.el8_1.x86_64", "Installed: glibc-devel-2.28-72.el8_1.1.x86_64", "Installed: authselect-1.1-2.el8.x86_64", "Installed: mesa-libglapi-19.1.4-3.el8_1.x86_64", "Installed: python3-virtualenv-15.1.0-19.module+el8.1.0+3514+14e79d98.noarch", "Installed: libdrm-2.4.98-2.el8.x86_64", "Installed: iwl5000-firmware-8.83.5.1_1-94.el8.1.noarch", "Installed: python3-firewall-0.7.0-5.el8.noarch", "Installed: setup-2.12.2-2.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: rdma-core-22.3-1.el8.x86_64", "Installed: gdb-8.2-6.el8_0.x86_64", "Installed: oddjob-mkhomedir-0.34.4-7.el8.x86_64", "Installed: perl-Git-SVN-2.18.2-1.el8_1.noarch", "Installed: libwayland-server-1.15.0-1.el8.x86_64", "Installed: libglvnd-gles-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: libogg-2:1.3.2-10.el8.x86_64", "Installed: dnf-plugin-spacewalk-2.8.5-11.module+el8.1.0+3455+3ddf2832.noarch", "Installed: libipt-1.6.1-8.el8.x86_64", "Installed: libtool-ltdl-2.4.6-25.el8.x86_64", "Installed: kexec-tools-2.0.19-12.el8_1.1.x86_64", "Installed: librdmacm-22.3-1.el8.x86_64", "Installed: libsolv-0.7.4-3.el8.x86_64", "Installed: audit-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: libgcrypt-1.8.3-4.el8.x86_64", "Installed: git-instaweb-2.18.2-1.el8_1.x86_64", "Installed: sg3_utils-libs-1.44-3.el8.x86_64", "Installed: isl-0.16.1-6.el8.x86_64", "Installed: git-email-2.18.2-1.el8_1.noarch", "Installed: mesa-libgbm-19.1.4-3.el8_1.x86_64", "Installed: qemu-img-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: iwl4965-firmware-228.61.2.24-94.el8.1.noarch", "Installed: openssl-libs-1:1.1.1c-2.el8.x86_64", "Installed: glibc-headers-2.28-72.el 2020-02-05T16:54:46.887 INFO:teuthology.task.ansible.out:8_1.1.x86_64", "Installed: seavgabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch", "Installed: qemu-kvm-common-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: genisoimage-1.1.11-39.el8.x86_64", "Installed: git-core-2.18.2-1.el8_1.x86_64", "Installed: qemu-kvm-core-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: vim-minimal-2:8.0.1763-13.el8.x86_64", "Installed: openssh-server-8.0p1-4.el8_1.x86_64", "Installed: kernel-tools-libs-4.18.0-147.5.1.el8_1.x86_64", "Installed: libatomic_ops-7.6.2-3.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: edk2-ovmf-20190308git89910a39dcfd-6.el8.noarch", "Installed: python3-rhn-setup-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: libpciaccess-0.14-1.el8.x86_64", "Installed: podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64", "Installed: libgomp-8.3.1-4.5.el8.x86_64", "Installed: shadow-utils-2:4.6-8.el8.x86_64", "Installed: openssh-clients-8.0p1-4.el8_1.x86_64", "Installed: libXv-1.0.11-7.el8.x86_64", "Installed: keyutils-1.5.10-6.el8.x86_64", "Installed: libwayland-client-1.15.0-1.el8.x86_64", "Installed: glibc-langpack-en-2.28-72.el8_1.1.x86_64", "Installed: slirp4netns-0.4.2-2.git21fdece.module+el8.1.1+5460+3ac089c3.x86_64", "Installed: iprutils-2.4.18.1-1.el8.x86_64", "Installed: python3-setuptools-wheel-39.2.0-5.el8.noarch", "Installed: gstreamer1-plugins-base-1.14.0-4.el8.x86_64", "Installed: python3-rhn-client-tools-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: python3-rhn-check-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: glibc-2.28-72.el8_1.1.x86_64", "Installed: dnf-plugins-core-4.0.8-3.el8.noarch", "Installed: glibc-common-2.28-72.el8_1.1.x86_64", "Installed: containernetworking-plugins-0.8.3-4.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: python3-dnf-4.2.7-7.el8_1.noarch", "Installed: rpm-plugin-systemd-inhibit-4.14.2-25.el8.x86_64", "Installed: libX11-xcb-1.6.7-1.el8.x86_64", 2020-02-05T16:54:46.889 INFO:teuthology.task.ansible.out:"Installed: python3-nose-1.3.7-30.module+el8.1.0+3334+5cb623d7.noarch", "Installed: kernel-headers-4.18.0-147.5.1.el8_1.x86_64", "Installed: e2fsprogs-libs-1.44.6-3.el8.x86_64", "Installed: boost-regex-1.66.0-6.el8.x86_64", "Installed: mesa-libGL-19.1.4-3.el8_1.x86_64", "Installed: libgcc-8.3.1-4.5.el8.x86_64", "Installed: libcurl-7.61.1-11.el8.x86_64", "Installed: gcc-gdb-plugin-8.3.1-4.5.el8.x86_64", "Installed: libusal-1.1.11-39.el8.x86_64", "Installed: conmon-2:2.0.6-1.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: dracut-config-rescue-049-27.git20190906.el8_1.1.x86_64", "Installed: oddjob-0.34.4-7.el8.x86_64", "Installed: podman-manpages-1.6.4-2.module+el8.1.1+5363+bf8ff1af.noarch", "Installed: device-mapper-multipath-libs-0.8.0-5.el8.x86_64", "Installed: mesa-libEGL-19.1.4-3.el8_1.x86_64", "Installed: libattr-devel-2.4.48-3.el8.x86_64", "Installed: linux-firmware-20190516-94.git711d3297.el8 2020-02-05T16:54:46.889 INFO:teuthology.task.ansible.out:.noarch", "Installed: libglvnd-glx-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: rpcbind-1.2.5-4.el8.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: dnf-data-4.2.7-7.el8_1.noarch", "Installed: runc-1.0.0-64.rc9.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: git-gui-2.18.2-1.el8_1.noarch", "Installed: sgabios-bin-1:0.20170427git-3.module+el8.1.0+4066+0f1aadab.noarch", "Installed: boost-iostreams-1.66.0-6.el8.x86_64", "Installed: xml-common-0.6.3-50.el8.noarch", "Installed: podman-docker-1.6.4-2.module+el8.1.1+5363+bf8ff1af.noarch", "Installed: kernel-tools-4.18.0-147.5.1.el8_1.x86_64", "Installed: iwl3160-firmware-1:25.30.13.0-94.el8.1.noarch", "Installed: curl-7.61.1-11.el8.x86_64", "Installed: iputils-20180629-2.el8.x86_64", "Installed: libcacard-3:2.7.0-2.el8_1.x86_64", "Installed: git-subtree-2.18.2-1.el8_1.x86_64", "Installed: cronie-anacron-1.5.2-4.el8.x86_64", "Installed: rhn-setup-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: libvisual-1:0.4.0-24.el8.x86_64", "Installed: rhn-check-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", " 2020-02-05T16:54:46.890 INFO:teuthology.task.ansible.out:Installed: rpm-build-libs-4.14.2-25.el8.x86_64", "Installed: rhn-client-tools-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: xfsprogs-5.0.0-1.el8.x86_64", "Installed: iwl7260-firmware-1:25.30.13.0-94.el8.1.noarch", "Installed: python-rpm-macros-3-37.el8.noarch", "Installed: libXxf86vm-1.1.4-9.el8.x86_64", "Installed: glusterfs-6.0-20.el8.x86_64", "Installed: libepoxy-1.5.2-1.el8.x86_64", "Installed: libvorbis-1:1.3.6-2.el8.x86_64", "Installed: iwl6000g2a-firmware-18.168.6.1-94.el8.1.noarch", "Installed: python3-dnf-plugin-spacewalk-2.8.5-11.module+el8.1.0+3455+3ddf2832.noarch", "Installed: python3-librepo-1.10.3-3.el8.x86_64", "Installed: libtheora-1:1.1.1-21.el8.x86_64", "Installed: orc-0.4.28-2.el8.x86_64", "Installed: seabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: nfs-utils-1:2.3.3-26.el8.x86_64", "Installed: authselect-compat-1.1-2.el8.x86_64", "Installed: glusterfs-libs-6.0-20.el8.x86_64", "Installed: boost-date-time-1.66.0-6.el8.x86_64", "Installed: rpm-plugin-selinux-4.14.2-25.el8.x86_64", "Installed: glusterfs-api-6.0-20.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: git-2.18.2-1.el8_1.x86_64", "Installed: bash-4.4.19-10.el8.x86_64", "Installed: python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: authselect-libs-1.1-2.el8.x86_64", "Installed: glusterfs-client-xlators-6.0-20.el8.x86_64", "Installed: iwl6050-firmware-41.28.5.1-94.el8.1.noarch", "Installed: sg3_utils-1.44-3.el8.x86_64", "Installed: iwl3945-firmware-15.32.2.9-94.el8.1.noarch", "Installed: dracut-network-049-27.git20190906.el8_1.1.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: gssproxy-0.8.0-14.el8.x86_64", "Installed: autoconf-2.69-27.el8.noarch", "Installed: python3-rpm-generators-5-4.el8.noarch", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: userspace-rcu-0.10.1-2.el8.x86_64", "Installed: cronie-1.5.2-4.el8.x86_64 2020-02-05T16:54:46.892 INFO:teuthology.task.ansible.out:", "Installed: git-core-doc-2.18.2-1.el8_1.noarch", "Installed: libxshmfence-1.3-2.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Installed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Installed: libwayland-cursor-1.15.0-1.el8.x86_64", "Installed: yum-4.2.7-7.el8_1.noarch", "Installed: iwl1000-firmware-1:39.31.5.1-94.el8.1.noarch", "Installed: dnf-4.2.7-7.el8_1.noarch", "Installed: iwl2030-firmware-18.168.6.1-94.el8.1.noarch", "Installed: iwl5150-firmware-8.24.2.2-94.el8.1.noarch", "Installed: initscripts-10.00.4-1.el8.x86_64", "Installed: firewalld-filesystem-0.7.0-5.el8.noarch", "Installed: libibumad-22.3-1.el8.x86_64", "Installed: grubby-8.40-37.el8.x86_64", "Installed: yum-utils-4.0.8-3.el8.noarch", "Installed: libxcrypt-devel-4.1.1-4.el8.x86_64", "Installed: python3-pip-wheel-9.0.3-15.el8.noarch", "Installed: dracut-squash-049-27.git20190906.el8_1.1.x86_64", "Installed: gitk-2.18.2-1.el8_1.noarch", "Installed: guile-5:2.0.14-7.el8.x86_64", "Installed: gstreamer1-1.14.0-3.el8.x86_64", "Installed: perl-Git-2.18.2-1.el8_1.noarch", "Installed: iwl100-firmware-39.31.5.1-94.el8.1.noarch", "Installed: libverto-libevent-0.3.0-5.el8.x86_64", "Installed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Installed: gettext-libs-0.19.8.1-17.el8.x86_64", "Installed: libcom_err-1.44.6-3.el8.x86_64", "Installed: opus-1.3-0.4.beta.el8.x86_64", "Installed: libdnf-0.35.1-9.el8_1.x86_64", "Installed: libibverbs-22.3-1.el8.x86_64", "Installed: iproute-4.18.0-15.el8.x86_64", "Installed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Installed: libseccomp-2.4.1-1.el8.x86_64", "Installed: openssl-1:1.1.1c-2.el8.x86_64", "Installed: dracut-049-27.git20190906.el8_1.1.x86_64", "Installed: rpm-libs-4.14.2-25.el8.x86_64", "Installed: e2fsprogs-1.44.6-3.el8.x86_64", "Installed: librepo-1.10.3-3.el8.x86_64", "Installed: iwl135-firmware-18.168.6.1-94.el8.1.noarch", "Installed: cpp-8.3.1-4.5.el8.x86_64", "Installed: python3-wheel-wheel-1:0.31.1-2.module+el8.1.0+3724+3c097090.noarch", "Installed: spic 2020-02-05T16:54:46.893 INFO:teuthology.task.ansible.out:e-server-0.14.2-1.el8.x86_64", "Installed: gcc-8.3.1-4.5.el8.x86_64", "Installed: git-svn-2.18.2-1.el8_1.x86_64", "Installed: libvarlink-18-3.el8.x86_64", "Installed: rhnlib-2.8.6-8.module+el8.1.0+3455+3ddf2832.noarch", "Installed: iwl6000-firmware-9.221.4.1-94.el8.1.noarch", "Installed: audit-libs-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: git-all-2.18.2-1.el8_1.noarch", "Installed: iso-codes-3.79-2.el8.noarch", "Installed: libmpc-1.0.2-9.el8.x86_64", "Installed: iwl105-firmware-18.168.6.1-94.el8.1.noarch", "Removed: NetworkManager-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-libnm-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-team-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-tui-1:1.14.0-14.el8.x86_64", "Removed: perl-Git-2.18.1-3.el8.noarch", "Removed: perl-Git-SVN-2.18.1-3.el8.noarch", "Removed: audit-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: audit-libs-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: authselect-1.0-13.el8.x86_64", "Removed: authselect-compat-1.0-13.el8.x86_64", "Removed: authselect-libs-1.0-13.el8.x86_64", "Removed: bash-4.4.19-8.el8_0.x86_64", "Removed: containernetworking-plugins-0.7.4-3.git9ebe139.module+el8+2769+577ad176.x86_64", "Removed: cronie-1.5.2-2.el8.x86_64", "Removed: cronie-anacron-1.5.2-2.el8.x86_64", "Removed: curl-7.61.1-8.el8.x86_64", "Removed: dnf-4.0.9.2-5.el8.noarch", "Removed: dnf-data-4.0.9.2-5.el8.noarch", "Removed: dnf-plugin-spacewalk-2.8.5-11.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: dnf-plugins-core-4.0.2.2-3.el8.noarch", "Removed: dnf-utils-4.0.2.2-3.el8.noarch", "Removed: dracut-049-10.git20190115.el8.x86_64", "Removed: dracut-config-rescue-049-10.git20190115.el8.x86_64", "Removed: dracut-network-049-10.git20190115.el8.x86_64", "Removed: dracut-squash-049-10.git20190115.el8.x86_64", "Removed: e2fsprogs-1.44.3-2.el8.x86_64", "Removed: e2fsprogs-libs-1.44.3-2.el8.x86_64", "Removed: firewalld-0.6.3-7.el8.noarch", "Removed: podman-1. 2020-02-05T16:54:46.895 INFO:teuthology.task.ansible.out:0.5-1.gitf604175.module+el8.0.0+4017+bbba319f.x86_64", "Removed: firewalld-filesystem-0.6.3-7.el8.noarch", "Removed: python3-audit-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: gettext-0.19.8.1-14.el8.x86_64", "Removed: gettext-libs-0.19.8.1-14.el8.x86_64", "Removed: git-2.18.1-3.el8.x86_64", "Removed: git-all-2.18.1-3.el8.noarch", "Removed: git-core-2.18.1-3.el8.x86_64", "Removed: git-core-doc-2.18.1-3.el8.noarch", "Removed: git-email-2.18.1-3.el8.noarch", "Removed: python3-dnf-plugin-spacewalk-2.8.5-11.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: git-instaweb-2.18.1-3.el8.x86_64", "Removed: git-subtree-2.18.1-3.el8.x86_64", "Removed: git-svn-2.18.1-3.el8.x86_64", "Removed: gitk-2.18.1-3.el8.noarch", "Removed: git-gui-2.18.1-3.el8.noarch", "Removed: gitweb-2.18.1-3.el8.noarch", "Removed: python3-hawkey-0.22.5-4.el8.x86_64", "Removed: python3-firewall-0.6.3-7.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.2.2-3.el8.noarch", "Removed: glibc-langpack-en-2.28-42.el8_0.1.x86_64", "Removed: glibc-2.28-42.el8_0.1.x86_64", "Removed: glibc-common-2.28-42.el8_0.1.x86_64", "Removed: python3-libdnf-0.22.5-4.el8.x86_64", "Removed: python3-librepo-1.9.2-1.el8.x86_64", "Removed: grubby-8.40-34.el8.x86_64", "Removed: python3-rhn-check-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rhn-client-tools-2.8.16-13.module+el8 2020-02-05T16:54:46.895 INFO:teuthology.task.ansible.out:.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rhn-setup-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rpm-4.14.2-11.el8_0.x86_64", "Removed: initscripts-10.00.1-1.el8_0.1.x86_64", "Removed: iproute-4.18.0-11.el8.x86_64", "Removed: iprutils-2.4.16.1-2.el8.x86_64", "Removed: iputils-20180629-1.el8.x86_64", "Removed: iwl100-firmware-39.31.5.1-92.el8.1.noarch", "Removed: iwl1000-firmware-1:39.31.5.1-92.el8.1.noarch", "Removed: iwl105-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl135-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl2000-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl2030-firmware-18.168 2020-02-05T16:54:46.896 INFO:teuthology.task.ansible.out:.6.1-92.el8.1.noarch", "Removed: iwl3160-firmware-1:25.30.13.0-92.el8.1.noarch", "Removed: iwl3945-firmware-15.32.2.9-92.el8.1.noarch", "Removed: iwl4965-firmware-228.61.2.24-92.el8.1.noarch", "Removed: iwl5000-firmware-8.83.5.1_1-92.el8.1.noarch", "Removed: iwl5150-firmware-8.24.2.2-92.el8.1.noarch", "Removed: iwl6000-firmware-9.221.4.1-92.el8.1.noarch", "Removed: iwl6000g2a-firmware-18.168.6.1-92.el8.1.noarch", "Removed: rhn-check-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhn-client-tools-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhn-setup-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhnlib-2.8.6-8.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: iwl6050-firmware-41.28.5.1-92.el8.1.noarch", "Removed: rpm-4.14.2-11.el8_0.x86_64", "Removed: rpm-build-libs-4.14.2-11.el8_0.x86_64", "Removed: rpm-libs-4.14.2-11.el8_0.x86_64", "Removed: iwl7260-firmware-1:25.30.13.0-92.el8.1.noarch", "Removed: rpm-plugin-systemd-inhibit-4.14.2-11.el8_0.x86_64", "Removed: rpm-plugin-selinux-4.14.2-11.el8_0.x86_64", "Removed: runc-1.0.0-56.rc8.dev.git425e105.module+el8.0.0+4017+bbba319f.x86_64", "Removed: kernel-tools-4.18.0-80.11.2.el8_0.x86_64", "Removed: kernel-tools-libs-4.18.0-80.11.2.el8_0.x86_64", "Removed: setup-2.12.2-2.el8.noarch", "Removed: kexec-tools-2.0.17-28.el8.x86_64", "Removed: sg3_utils-libs-1.44-2.el8.x86_64", "Removed: shadow-utils-2:4.6-7.el8.x86_64", "Removed: sg3_utils-1.44-2.el8.x86_64", "Removed: slirp4netns-0.1-2.dev.gitc4e1bc5.module+el8+2769+577ad176.x86_64", "Removed: python3-dnf-4.0.9.2-5.el8.noarch", "Removed: libcom_err-1.44.3-2.el8.x86_64", "Removed: libcurl-7.61.1-8.el8.x86_64", "Removed: libdnf-0.22.5-4.el8.x86_64", "Removed: libgcc-8.2.1-3.5.el8.x86_64", "Removed: libgcrypt-1.8.3-2.el8.x86_64", "Removed: libgomp-8.2.1-3.5.el8.x86_64", "Removed: vim-minimal-2:8.0.1763-11.el8_0.x86_64", "Removed: xfsprogs-4.19.0-2.el8.x86_64", "Removed: yum-4.0.9.2-5.el8.noarch", "Removed: librepo-1.9.2-1.el8.x86_64", " 2020-02-05T16:54:46.898 INFO:teuthology.task.ansible.out:Removed: libseccomp-2.3.3-3.el8.x86_64", "Removed: libsolv-0.6.35-6.el8.x86_64", "Removed: libss-1.44.3-2.el8.x86_64", "Removed: linux-firmware-20190111-92.gitd9fb2ee6.el8.noarch", "Removed: lshw-B.02.18-16.el8.x86_64", "Removed: openssh-7.8p1-4.el8.x86_64", "Removed: openssh-clients-7.8p1-4.el8.x86_64", "Removed: openssh-server-7.8p1-4.el8.x86_64", "Removed: openssl-1:1.1.1-8.el8.x86_64", "Removed: openssl-libs-1:1.1.1-8.el8.x86_64", "Removed: parted-3.2-36.el8.x86_64"]} 2020-02-05T16:54:46.954 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:46 +0000 (0:02:16.966) 0:10:34.284 **** 2020-02-05T16:54:53.500 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-02-05T16:54:53.502 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dbench", "Installed: dbench-4.0-20.el8.x86_64"]} 2020-02-05T16:54:53.541 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:53 +0000 (0:00:06.586) 0:10:40.870 **** 2020-02-05T16:54:56.042 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-02-05T16:54:56.043 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-02-05T16:54:56.093 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:56 +0000 (0:00:02.552) 0:10:43.423 **** 2020-02-05T16:54:56.130 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:54:56.131 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:56.166 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:56 +0000 (0:00:00.073) 0:10:43.496 **** 2020-02-05T16:54:56.647 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-02-05T16:54:56.649 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target system.slice dbus.socket dbus.service polkit.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service shutdown.target iptables.service ip6tables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-02-05T16:54:56.651 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127783", "LimitNPROCSoft": "127783", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127783", 2020-02-05T16:54:56.652 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127783", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-02-05T16:54:56.654 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-02-05T16:54:56.699 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:56 +0000 (0:00:00.532) 0:10:44.028 **** 2020-02-05T16:54:57.156 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-02-05T16:54:57.158 INFO:teuthology.task.ansible.out:fatal: [smithi164.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-02-05T16:54:57.201 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:57 +0000 (0:00:00.502) 0:10:44.531 **** 2020-02-05T16:54:58.078 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-02-05T16:54:58.080 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-02-05T16:54:58.118 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.916) 0:10:45.447 **** 2020-02-05T16:54:58.144 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-02-05T16:54:58.146 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.181 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.062) 0:10:45.510 **** 2020-02-05T16:54:58.207 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-02-05T16:54:58.243 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.062) 0:10:45.573 **** 2020-02-05T16:54:58.269 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-02-05T16:54:58.271 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.304 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.061) 0:10:45.634 **** 2020-02-05T16:54:58.331 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-02-05T16:54:58.333 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.369 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.064) 0:10:45.699 **** 2020-02-05T16:54:58.397 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* skipping: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.407 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.444 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.074) 0:10:45.773 **** 2020-02-05T16:54:58.470 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-02-05T16:54:58.471 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.507 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.063) 0:10:45.837 **** 2020-02-05T16:54:58.534 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-02-05T16:54:58.572 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.064) 0:10:45.901 **** 2020-02-05T16:54:58.598 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-02-05T16:54:58.599 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.636 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.064) 0:10:45.966 **** 2020-02-05T16:54:58.662 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-02-05T16:54:58.664 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.698 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.062) 0:10:46.028 **** 2020-02-05T16:54:58.727 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-02-05T16:54:58.734 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.781 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.082) 0:10:46.111 **** 2020-02-05T16:54:58.808 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:54:58.809 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.849 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.067) 0:10:46.178 **** 2020-02-05T16:54:58.878 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-02-05T16:54:58.879 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.924 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.074) 0:10:46.253 **** 2020-02-05T16:54:58.950 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-02-05T16:54:58.951 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:58.989 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:58 +0000 (0:00:00.065) 0:10:46.319 **** 2020-02-05T16:54:59.016 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-02-05T16:54:59.017 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.062 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.073) 0:10:46.392 **** 2020-02-05T16:54:59.088 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-02-05T16:54:59.090 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.127 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.065) 0:10:46.457 **** 2020-02-05T16:54:59.168 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-02-05T16:54:59.174 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.213 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.085) 0:10:46.543 **** 2020-02-05T16:54:59.244 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.278 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.064) 0:10:46.608 **** 2020-02-05T16:54:59.305 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-02-05T16:54:59.306 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.346 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.068) 0:10:46.676 **** 2020-02-05T16:54:59.374 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.419 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.072) 0:10:46.748 **** 2020-02-05T16:54:59.446 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-02-05T16:54:59.448 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.487 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.067) 0:10:46.816 **** 2020-02-05T16:54:59.515 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-02-05T16:54:59.517 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.559 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.072) 0:10:46.889 **** 2020-02-05T16:54:59.585 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-02-05T16:54:59.588 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.624 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.064) 0:10:46.953 **** 2020-02-05T16:54:59.654 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.693 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.066) 0:10:47.019 **** 2020-02-05T16:54:59.716 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-02-05T16:54:59.717 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.755 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.065) 0:10:47.085 **** 2020-02-05T16:54:59.782 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-02-05T16:54:59.783 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.820 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.064) 0:10:47.149 **** 2020-02-05T16:54:59.846 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-02-05T16:54:59.847 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.885 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.064) 0:10:47.214 **** 2020-02-05T16:54:59.911 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-02-05T16:54:59.912 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:54:59.949 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:54:59 +0000 (0:00:00.064) 0:10:47.278 **** 2020-02-05T16:54:59.975 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-02-05T16:54:59.977 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.014 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.065) 0:10:47.344 **** 2020-02-05T16:55:00.041 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-02-05T16:55:00.043 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.085 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.064) 0:10:47.409 **** 2020-02-05T16:55:00.106 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-02-05T16:55:00.108 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.144 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.065) 0:10:47.474 **** 2020-02-05T16:55:00.170 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-02-05T16:55:00.172 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.209 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.065) 0:10:47.539 **** 2020-02-05T16:55:00.236 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:55:00.237 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.276 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.066) 0:10:47.605 **** 2020-02-05T16:55:00.306 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.343 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.066) 0:10:47.672 **** 2020-02-05T16:55:00.379 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-02-05T16:55:00.380 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.420 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.077) 0:10:47.749 **** 2020-02-05T16:55:00.447 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-02-05T16:55:00.449 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.487 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.067) 0:10:47.816 **** 2020-02-05T16:55:00.513 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-02-05T16:55:00.515 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.560 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.072) 0:10:47.889 **** 2020-02-05T16:55:00.588 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.627 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.067) 0:10:47.956 **** 2020-02-05T16:55:00.655 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-02-05T16:55:00.656 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.696 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.069) 0:10:48.026 **** 2020-02-05T16:55:00.723 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-02-05T16:55:00.724 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.765 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.066) 0:10:48.092 **** 2020-02-05T16:55:00.799 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-02-05T16:55:00.801 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.844 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.081) 0:10:48.173 **** 2020-02-05T16:55:00.871 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-02-05T16:55:00.873 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.912 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.067) 0:10:48.241 **** 2020-02-05T16:55:00.938 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-02-05T16:55:00.940 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:00.977 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:00 +0000 (0:00:00.065) 0:10:48.307 **** 2020-02-05T16:55:01.005 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.044 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.066) 0:10:48.373 **** 2020-02-05T16:55:01.070 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-02-05T16:55:01.075 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.114 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.067) 0:10:48.441 **** 2020-02-05T16:55:01.138 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-02-05T16:55:01.139 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.176 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.064) 0:10:48.506 **** 2020-02-05T16:55:01.203 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-02-05T16:55:01.204 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.243 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.066) 0:10:48.572 **** 2020-02-05T16:55:01.269 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-02-05T16:55:01.270 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.307 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.064) 0:10:48.637 **** 2020-02-05T16:55:01.333 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-02-05T16:55:01.335 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.372 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.065) 0:10:48.702 **** 2020-02-05T16:55:01.399 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-02-05T16:55:01.400 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.442 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.069) 0:10:48.772 **** 2020-02-05T16:55:01.468 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-02-05T16:55:01.470 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.504 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.062) 0:10:48.834 **** 2020-02-05T16:55:01.530 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-02-05T16:55:01.532 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.579 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.061) 0:10:48.896 **** 2020-02-05T16:55:01.595 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.634 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.067) 0:10:48.963 **** 2020-02-05T16:55:01.660 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-02-05T16:55:01.662 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.697 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.063) 0:10:49.026 **** 2020-02-05T16:55:01.722 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-02-05T16:55:01.724 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.760 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.063) 0:10:49.089 **** 2020-02-05T16:55:01.784 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-02-05T16:55:01.786 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.821 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.060) 0:10:49.150 **** 2020-02-05T16:55:01.846 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-02-05T16:55:01.847 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.883 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.062) 0:10:49.213 **** 2020-02-05T16:55:01.909 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-02-05T16:55:01.911 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:01.947 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:01 +0000 (0:00:00.063) 0:10:49.276 **** 2020-02-05T16:55:01.972 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-02-05T16:55:01.974 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.010 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.063) 0:10:49.340 **** 2020-02-05T16:55:02.038 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:02.040 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-02-05T16:55:02.076 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.065) 0:10:49.405 **** 2020-02-05T16:55:02.112 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-02-05T16:55:02.114 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.139 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-02-05T16:55:02.145 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.156 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.157 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.160 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:02.171 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.182 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.225 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.149) 0:10:49.555 **** 2020-02-05T16:55:02.388 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-02-05T16:55:02.389 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003690", "end": "2020-02-05 16:55:02.367084", "rc": 0, "start": "2020-02-05 16:55:02.363394", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-02-05T16:55:02.432 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.206) 0:10:49.761 **** 2020-02-05T16:55:02.468 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-02-05T16:55:02.469 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.508 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.076) 0:10:49.837 **** 2020-02-05T16:55:02.534 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-02-05T16:55:02.535 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.569 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.060) 0:10:49.898 **** 2020-02-05T16:55:02.605 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-02-05T16:55:02.607 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.645 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.076) 0:10:49.975 **** 2020-02-05T16:55:02.671 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-02-05T16:55:02.673 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:02.710 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:02 +0000 (0:00:00.064) 0:10:50.039 **** 2020-02-05T16:55:04.918 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-02-05T16:55:04.920 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-02-05T16:55:04.957 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:04 +0000 (0:00:02.247) 0:10:52.286 **** 2020-02-05T16:55:05.004 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-02-05T16:55:05.005 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item={u'block_used': 13449165, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645943, u'size_available': 928341782528, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991236, u'device': u'/dev/sda1', u'inode_used': 63740, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645943, "block_size": 4096, "block_total": 240095108, "block_used": 13449165, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991236, "inode_total": 61054976, "inode_used": 63740, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341782528, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}} 2020-02-05T16:55:05.051 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:05 +0000 (0:00:00.094) 0:10:52.381 **** 2020-02-05T16:55:05.401 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-02-05T16:55:05.402 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.023622", "end": "2020-02-05 16:55:05.367729", "rc": 0, "start": "2020-02-05 16:55:05.344107", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:05.439 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:05 +0000 (0:00:00.387) 0:10:52.768 **** 2020-02-05T16:55:05.487 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-02-05T16:55:05.488 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'block_used': 13449165, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645943, u'size_available': 928341782528, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991236, u'device': u'/dev/sda1', u'inode_used': 63740, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645943, "block_size": 4096, "block_total": 240095108, "block_used": 13449165, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991236, "inode_total": 61054976, "inode_used": 63740, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341782528, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:05.529 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:05 +0000 (0:00:00.089) 0:10:52.858 **** 2020-02-05T16:55:05.554 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-02-05T16:55:05.592 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:05 +0000 (0:00:00.063) 0:10:52.922 **** 2020-02-05T16:55:05.632 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-02-05T16:55:05.634 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTQ7', u'wwn-0x5000c5009293f07d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XTQ7', u'wwn': u'0x5000c5009293f07d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', u'wwn-0x5000c5009293f07d-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vend 2020-02-05T16:55:05.635 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:07.307 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TGSOHB-zkys-GikI-ECEC-aJXK-yRDr-GcxNck', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', u'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.356595", "end": "2020-02-05 16:55:07.258333", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TGSOHB-zkys-GikI-ECEC-aJXK-yRDr-GcxNck", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:05.901738", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-02-05T16:55:07.309 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:08.674 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trA8nmcCg7YZNmdH886xDxqqZuecvNvw0s'], u'uuids': [u'd84e1482-b849-4e38-b7d3-eea0ca55af80']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.052598", "end": "2020-02-05 16:55:08.630296", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trA8nmcCg7YZNmdH886xDxqqZuecvNvw0s"], "labels": [], "masters": [], "uuids": ["d84e1482-b849-4e38-b7d3-eea0ca55af80"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:07.577698", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:10.062 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trEqs0f7NiuHlwLZj8CJQH8js4eaZdb6ti'], u'uuids': [u'd6cd7f0a-f585-4cfb-bf6d-dd4894000074']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.069861", "end": "2020-02-05 16:55:10.019424", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trEqs0f7NiuHlwLZj8CJQH8js4eaZdb6ti"], "labels": [], "masters": [], "uuids": ["d6cd7f0a-f585-4cfb-bf6d-dd4894000074"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:08.949563", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:11.442 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trM9Uhk4qDd3fzhk4Jdw6zqNIqxyAFtRdP'], u'uuids': [u'4a564494-9b48-4007-ab63-380635736df0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.068920", "end": "2020-02-05 16:55:11.404630", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trM9Uhk4qDd3fzhk4Jdw6zqNIqxyAFtRdP"], "labels": [], "masters": [], "uuids": ["4a564494-9b48-4007-ab63-380635736df0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:10.335710", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:12.828 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0try3EZ6EIorxI1qcep0ooNFJVwIYJEhwtZ'], u'uuids': [u'3918f874-2a8f-4390-a744-e72297ebee8c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.064741", "end": "2020-02-05 16:55:12.787542", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0try3EZ6EIorxI1qcep0ooNFJVwIYJEhwtZ"], "labels": [], "masters": [], "uuids": ["3918f874-2a8f-4390-a744-e72297ebee8c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:11.722801", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:14.218 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0truEEWqmmEAwxabu2DuNsw3ysdWYf3Rey4'], u'uuids': [u'7e9b7ba3-862f-4445-b1ee-0e273a1ec23b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.064401", "end": "2020-02-05 16:55:14.172246", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0truEEWqmmEAwxabu2DuNsw3ysdWYf3Rey4"], "labels": [], "masters": [], "uuids": ["7e9b7ba3-862f-4445-b1ee-0e273a1ec23b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:13.107845", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:14.260 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:14 +0000 (0:00:08.668) 0:11:01.590 **** 2020-02-05T16:55:14.298 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-02-05T16:55:14.299 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTQ7', u'wwn-0x5000c5009293f07d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XTQ7', u'wwn': u'0x5000c5009293f07d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1', u'wwn-0x5000c5009293f07d-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7", "wwn-0x5000c5009293f07d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XTQ7-part1", "wwn-0x5000c5009293f07d-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XTQ7", "size": "931.51 GB", "support_discard": "0", "vend 2020-02-05T16:55:14.300 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009293f07d"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:14.747 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TGSOHB-zkys-GikI-ECEC-aJXK-yRDr-GcxNck', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN', u'nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.130870", "end": "2020-02-05 16:55:14.707300", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TGSOHB-zkys-GikI-ECEC-aJXK-yRDr-GcxNck", "nvme-INTEL_SSDPEDMD400G4_CVFT623300H0400BGN", "nvme-nvme.8086-43564654363233333030483034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:14.576430", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-02-05T16:55:14.749 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.117279 s, 983 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.117279 s, 983 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:15.342 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trA8nmcCg7YZNmdH886xDxqqZuecvNvw0s'], u'uuids': [u'd84e1482-b849-4e38-b7d3-eea0ca55af80']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.295723", "end": "2020-02-05 16:55:15.298668", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trA8nmcCg7YZNmdH886xDxqqZuecvNvw0s"], "labels": [], "masters": [], "uuids": ["d84e1482-b849-4e38-b7d3-eea0ca55af80"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:15.002945", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.284106 s, 406 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.284106 s, 406 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:15.759 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-02-05T16:55:15.761 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: git-all", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: podman", "Installed: podman-docker", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: autoconf", "Installed: gdb", "Installed: valgrind-1:3.15.0-9.el8.x86_64", "Installed: annobin-8.78-1.el8.x86_64", "Installed: celt051-0.5.1.3-15.el8.x86_64", "Installed: gdb-headless-8.2-6.el8_0.x86_64", "Installed: ipxe-roms-qemu-20181214-3.git133f4c47.el8.noarch", "Installed: python36-devel-3.6.8-2.module+el8.1.0+3334+5cb623d7.x86_64", "Installed: platform-python-devel-3.6.8-1.el8.x86_64", "Installed: python3-rpm-4.14.2-25.el8.x86_64", "Installed: parted-3.2-38.el8.x86_64", "Installed: libpmem-1.6-2.el8.x86_64", "Installed: gc-7.6.4-3.el8.x86_64", "Installed: gitweb-2.18.2-1.el8_1.noarch", "Installed: rpm-4.14.2-25.el8.x86_64", "Installed: libss-1.44.6-3.el8.x86_64", "Installed: libacl-devel-2.2.53-1.el8.x86_64", "Installed: iwl2000-firmware-18.168.6.1-94.el8.1.noarch", "Installed: firewalld-0.7.0-5.el8.noarch", "Installed: boost-chrono-1.66.0-6.el8.x86_64", "Installed: gettext-0.19.8.1-17.el8.x86_64", "Installed: libbabeltrace-1.5.6-2.el8.x86_64", "Installed: lshw-B.02.18-21.el8.x86_64", "Installed: libglv 2020-02-05T16:55:15.763 INFO:teuthology.task.ansible.out:nd-egl-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: libwayland-egl-1.15.0-1.el8.x86_64", "Installed: NetworkManager-team-1:1.20.0-5.el8_1.x86_64", "Installed: libglvnd-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: boost-atomic-1.66.0-6.el8.x86_64", "Installed: openssh-8.0p1-4.el8_1.x86_64", "Installed: glibc-devel-2.28-72.el8_1.1.x86_64", "Installed: authselect-1.1-2.el8.x86_64", "Installed: mesa-libglapi-19.1.4-3.el8_1.x86_64", "Installed: python3-virtualenv-15.1.0-19.module+el8.1.0+3514+14e79d98.noarch", "Installed: libdrm-2.4.98-2.el8.x86_64", "Installed: iwl5000-firmware-8.83.5.1_1-94.el8.1.noarch", "Installed: python3-firewall-0.7.0-5.el8.noarch", "Installed: setup-2.12.2-2.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: rdma-core-22.3-1.el8.x86_64", "Installed: gdb-8.2-6.el8_0.x86_64", "Installed: oddjob-mkhomedir-0.34.4-7.el8.x86_64", "Installed: perl-Git-SVN-2.18.2-1.el8_1.noarch", "Installed: libwayland-server-1.15.0-1.el8.x86_64", "Installed: libglvnd-gles-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: NetworkManager-libnm-1:1.20.0-5.el8_1.x86_64", "Installed: libogg-2:1.3.2-10.el8.x86_64", "Installed: dnf-plugin-spacewalk-2.8.5-11.module+el8.1.0+3455+3ddf2832.noarch", "Installed: libipt-1.6.1-8.el8.x86_64", "Installed: libtool-ltdl-2.4.6-25.el8.x86_64", "Installed: kexec-tools-2.0.19-12.el8_1.1.x86_64", "Installed: librdmacm-22.3-1.el8.x86_64", "Installed: libsolv-0.7.4-3.el8.x86_64", "Installed: audit-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: libgcrypt-1.8.3-4.el8.x86_64", "Installed: git-instaweb-2.18.2-1.el8_1.x86_64", "Installed: sg3_utils-libs-1.44-3.el8.x86_64", "Installed: isl-0.16.1-6.el8.x86_64", "Installed: git-email-2.18.2-1.el8_1.noarch", "Installed: mesa-libgbm-19.1.4-3.el8_1.x86_64", "Installed: qemu-img-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: iwl4965-firmware-228.61.2.24-94.el8.1.noarch", "Installed: openssl-libs-1:1.1.1c-2.el8.x86_64", "Installed: glibc-headers-2.28-72.el 2020-02-05T16:55:15.764 INFO:teuthology.task.ansible.out:8_1.1.x86_64", "Installed: seavgabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch", "Installed: qemu-kvm-common-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: genisoimage-1.1.11-39.el8.x86_64", "Installed: git-core-2.18.2-1.el8_1.x86_64", "Installed: qemu-kvm-core-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: vim-minimal-2:8.0.1763-13.el8.x86_64", "Installed: openssh-server-8.0p1-4.el8_1.x86_64", "Installed: kernel-tools-libs-4.18.0-147.5.1.el8_1.x86_64", "Installed: libatomic_ops-7.6.2-3.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module+el8.1.0+5149+3ff2765e.2.x86_64", "Installed: edk2-ovmf-20190308git89910a39dcfd-6.el8.noarch", "Installed: python3-rhn-setup-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: libpciaccess-0.14-1.el8.x86_64", "Installed: podman-1.6.4-2.module+el8.1.1+5363+bf8ff1af.x86_64", "Installed: libgomp-8.3.1-4.5.el8.x86_64", "Installed: shadow-utils-2:4.6-8.el8.x86_64", "Installed: openssh-clients-8.0p1-4.el8_1.x86_64", "Installed: libXv-1.0.11-7.el8.x86_64", "Installed: keyutils-1.5.10-6.el8.x86_64", "Installed: libwayland-client-1.15.0-1.el8.x86_64", "Installed: glibc-langpack-en-2.28-72.el8_1.1.x86_64", "Installed: slirp4netns-0.4.2-2.git21fdece.module+el8.1.1+5460+3ac089c3.x86_64", "Installed: iprutils-2.4.18.1-1.el8.x86_64", "Installed: python3-setuptools-wheel-39.2.0-5.el8.noarch", "Installed: gstreamer1-plugins-base-1.14.0-4.el8.x86_64", "Installed: python3-rhn-client-tools-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: python3-rhn-check-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: glibc-2.28-72.el8_1.1.x86_64", "Installed: dnf-plugins-core-4.0.8-3.el8.noarch", "Installed: glibc-common-2.28-72.el8_1.1.x86_64", "Installed: containernetworking-plugins-0.8.3-4.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: python3-dnf-4.2.7-7.el8_1.noarch", "Installed: rpm-plugin-systemd-inhibit-4.14.2-25.el8.x86_64", "Installed: libX11-xcb-1.6.7-1.el8.x86_64", 2020-02-05T16:55:15.766 INFO:teuthology.task.ansible.out:"Installed: python3-nose-1.3.7-30.module+el8.1.0+3334+5cb623d7.noarch", "Installed: kernel-headers-4.18.0-147.5.1.el8_1.x86_64", "Installed: e2fsprogs-libs-1.44.6-3.el8.x86_64", "Installed: boost-regex-1.66.0-6.el8.x86_64", "Installed: mesa-libGL-19.1.4-3.el8_1.x86_64", "Installed: libgcc-8.3.1-4.5.el8.x86_64", "Installed: libcurl-7.61.1-11.el8.x86_64", "Installed: gcc-gdb-plugin-8.3.1-4.5.el8.x86_64", "Installed: libusal-1.1.11-39.el8.x86_64", "Installed: conmon-2:2.0.6-1.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: dracut-config-rescue-049-27.git20190906.el8_1.1.x86_64", "Installed: oddjob-0.34.4-7.el8.x86_64", "Installed: podman-manpages-1.6.4-2.module+el8.1.1+5363+bf8ff1af.noarch", "Installed: device-mapper-multipath-libs-0.8.0-5.el8.x86_64", "Installed: mesa-libEGL-19.1.4-3.el8_1.x86_64", "Installed: libattr-devel-2.4.48-3.el8.x86_64", "Installed: linux-firmware-20190516-94.git711d3297.el8.noarch", "Installed: libglvnd-glx-1:1.0.1-0.9.git5baa1e5.el8.x86_64", "Installed: rpcbind-1.2.5-4.el8.x86_64", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: dnf-data-4.2.7-7.el8_1.noarch", "Installed: runc-1.0.0-64.rc9.module+el8.1.1+5259+bcdd613a.x86_64", "Installed: git-gui-2.18.2-1.el8_1.noarch", "Installed: sgabios-bin-1:0.20170427git-3.module+el8.1.0+4066+0f1aadab.noarch", "Installed: boost-iostreams-1.66.0-6.el8.x86_64", "Installed: xml-common-0.6.3-50.el8.noarch", "Installed: podman-docker-1.6.4-2.module+el8.1.1+5363+bf8ff1af.noarch", "Installed: kernel-tools-4.18.0-147.5.1.el8_1.x86_64", "Installed: iwl3160-firmware-1:25.30.13.0-94.el8.1.noarch", "Installed: curl-7.61.1-11.el8.x86_64", "Installed: iputils-20180629-2.el8.x86_64", "Installed: libcacard-3:2.7.0-2.el8_1.x86_64", "Installed: git-subtree-2.18.2-1.el8_1.x86_64", "Installed: cronie-anacron-1.5.2-4.el8.x86_64", "Installed: rhn-setup-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: libvisual-1:0.4.0-24.el8.x86_64", "Installed: rhn-check-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", " 2020-02-05T16:55:15.767 INFO:teuthology.task.ansible.out:Installed: rpm-build-libs-4.14.2-25.el8.x86_64", "Installed: rhn-client-tools-2.8.16-13.module+el8.1.0+3455+3ddf2832.x86_64", "Installed: xfsprogs-5.0.0-1.el8.x86_64", "Installed: iwl7260-firmware-1:25.30.13.0-94.el8.1.noarch", "Installed: python-rpm-macros-3-37.el8.noarch", "Installed: libXxf86vm-1.1.4-9.el8.x86_64", "Installed: glusterfs-6.0-20.el8.x86_64", "Installed: libepoxy-1.5.2-1.el8.x86_64", "Installed: libvorbis-1:1.3.6-2.el8.x86_64", "Installed: iwl6000g2a-firmware-18.168.6.1-94.el8.1.noarch", "Installed: python3-dnf-plugin-spacewalk-2.8.5-11.module+el8.1.0+3455+3ddf2832.noarch", "Installed: python3-librepo-1.10.3-3.el8.x86_64", "Installed: libtheora-1:1.1.1-21.el8.x86_64", "Installed: orc-0.4.28-2.el8.x86_64", "Installed: seabios-bin-1.11.1-4.module+el8.1.0+4066+0f1aadab.noarch", "Installed: NetworkManager-tui-1:1.20.0-5.el8_1.x86_64", "Installed: nfs-utils-1:2.3.3-26.el8.x86_64", "Installed: authselect-compat-1.1-2.el8.x86_64", "Installed: glusterfs-libs-6.0-20.el8.x86_64", "Installed: boost-date-time-1.66.0-6.el8.x86_64", "Installed: rpm-plugin-selinux-4.14.2-25.el8.x86_64", "Installed: glusterfs-api-6.0-20.el8.x86_64", "Installed: NetworkManager-1:1.20.0-5.el8_1.x86_64", "Installed: git-2.18.2-1.el8_1.x86_64", "Installed: bash-4.4.19-10.el8.x86_64", "Installed: python3-audit-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: authselect-libs-1.1-2.el8.x86_64", "Installed: glusterfs-client-xlators-6.0-20.el8.x86_64", "Installed: iwl6050-firmware-41.28.5.1-94.el8.1.noarch", "Installed: sg3_utils-1.44-3.el8.x86_64", "Installed: iwl3945-firmware-15.32.2.9-94.el8.1.noarch", "Installed: dracut-network-049-27.git20190906.el8_1.1.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: gssproxy-0.8.0-14.el8.x86_64", "Installed: autoconf-2.69-27.el8.noarch", "Installed: python3-rpm-generators-5-4.el8.noarch", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: userspace-rcu-0.10.1-2.el8.x86_64", "Installed: cronie-1.5.2-4.el8.x86_64 2020-02-05T16:55:15.769 INFO:teuthology.task.ansible.out:", "Installed: git-core-doc-2.18.2-1.el8_1.noarch", "Installed: libxshmfence-1.3-2.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64", "Installed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Installed: libwayland-cursor-1.15.0-1.el8.x86_64", "Installed: yum-4.2.7-7.el8_1.noarch", "Installed: iwl1000-firmware-1:39.31.5.1-94.el8.1.noarch", "Installed: dnf-4.2.7-7.el8_1.noarch", "Installed: iwl2030-firmware-18.168.6.1-94.el8.1.noarch", "Installed: iwl5150-firmware-8.24.2.2-94.el8.1.noarch", "Installed: initscripts-10.00.4-1.el8.x86_64", "Installed: firewalld-filesystem-0.7.0-5.el8.noarch", "Installed: libibumad-22.3-1.el8.x86_64", "Installed: grubby-8.40-37.el8.x86_64", "Installed: yum-utils-4.0.8-3.el8.noarch", "Installed: libxcrypt-devel-4.1.1-4.el8.x86_64", "Installed: python3-pip-wheel-9.0.3-15.el8.noarch", "Installed: dracut-squash-049-27.git20190906.el8_1.1.x86_64", "Installed: gitk-2.18.2-1.el8_1.noarch", "Installed: guile-5:2.0.14-7.el8.x86_64", "Installed: gstreamer1-1.14.0-3.el8.x86_64", "Installed: perl-Git-2.18.2-1.el8_1.noarch", "Installed: iwl100-firmware-39.31.5.1-94.el8.1.noarch", "Installed: libverto-libevent-0.3.0-5.el8.x86_64", "Installed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Installed: gettext-libs-0.19.8.1-17.el8.x86_64", "Installed: libcom_err-1.44.6-3.el8.x86_64", "Installed: opus-1.3-0.4.beta.el8.x86_64", "Installed: libdnf-0.35.1-9.el8_1.x86_64", "Installed: libibverbs-22.3-1.el8.x86_64", "Installed: iproute-4.18.0-15.el8.x86_64", "Installed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Installed: libseccomp-2.4.1-1.el8.x86_64", "Installed: openssl-1:1.1.1c-2.el8.x86_64", "Installed: dracut-049-27.git20190906.el8_1.1.x86_64", "Installed: rpm-libs-4.14.2-25.el8.x86_64", "Installed: e2fsprogs-1.44.6-3.el8.x86_64", "Installed: librepo-1.10.3-3.el8.x86_64", "Installed: iwl135-firmware-18.168.6.1-94.el8.1.noarch", "Installed: cpp-8.3.1-4.5.el8.x86_64", "Installed: python3-wheel-wheel-1:0.31.1-2.module+el8.1.0+3724+3c097090.noarch", "Installed: spic 2020-02-05T16:55:15.770 INFO:teuthology.task.ansible.out:e-server-0.14.2-1.el8.x86_64", "Installed: gcc-8.3.1-4.5.el8.x86_64", "Installed: git-svn-2.18.2-1.el8_1.x86_64", "Installed: libvarlink-18-3.el8.x86_64", "Installed: rhnlib-2.8.6-8.module+el8.1.0+3455+3ddf2832.noarch", "Installed: iwl6000-firmware-9.221.4.1-94.el8.1.noarch", "Installed: audit-libs-3.0-0.13.20190507gitf58ec40.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: git-all-2.18.2-1.el8_1.noarch", "Installed: iso-codes-3.79-2.el8.noarch", "Installed: libmpc-1.0.2-9.el8.x86_64", "Installed: iwl105-firmware-18.168.6.1-94.el8.1.noarch", "Removed: NetworkManager-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-libnm-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-team-1:1.14.0-14.el8.x86_64", "Removed: NetworkManager-tui-1:1.14.0-14.el8.x86_64", "Removed: perl-Git-2.18.1-3.el8.noarch", "Removed: perl-Git-SVN-2.18.1-3.el8.noarch", "Removed: audit-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: audit-libs-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: authselect-1.0-13.el8.x86_64", "Removed: authselect-compat-1.0-13.el8.x86_64", "Removed: authselect-libs-1.0-13.el8.x86_64", "Removed: bash-4.4.19-8.el8_0.x86_64", "Removed: containernetworking-plugins-0.7.4-3.git9ebe139.module+el8+2769+577ad176.x86_64", "Removed: cronie-1.5.2-2.el8.x86_64", "Removed: cronie-anacron-1.5.2-2.el8.x86_64", "Removed: curl-7.61.1-8.el8.x86_64", "Removed: dnf-4.0.9.2-5.el8.noarch", "Removed: dnf-data-4.0.9.2-5.el8.noarch", "Removed: dnf-plugin-spacewalk-2.8.5-11.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: dnf-plugins-core-4.0.2.2-3.el8.noarch", "Removed: dnf-utils-4.0.2.2-3.el8.noarch", "Removed: dracut-049-10.git20190115.el8.x86_64", "Removed: dracut-config-rescue-049-10.git20190115.el8.x86_64", "Removed: dracut-network-049-10.git20190115.el8.x86_64", "Removed: dracut-squash-049-10.git20190115.el8.x86_64", "Removed: e2fsprogs-1.44.3-2.el8.x86_64", "Removed: e2fsprogs-libs-1.44.3-2.el8.x86_64", "Removed: firewalld-0.6.3-7.el8.noarch", "Removed: podman-1. 2020-02-05T16:55:15.771 INFO:teuthology.task.ansible.out:0.5-1.gitf604175.module+el8.0.0+4017+bbba319f.x86_64", "Removed: firewalld-filesystem-0.6.3-7.el8.noarch", "Removed: python3-audit-3.0-0.10.20180831git0047a6c.el8.x86_64", "Removed: gettext-0.19.8.1-14.el8.x86_64", "Removed: gettext-libs-0.19.8.1-14.el8.x86_64", "Removed: git-2.18.1-3.el8.x86_64", "Removed: git-all-2.18.1-3.el8.noarch", "Removed: git-core-2.18.1-3.el8.x86_64", "Removed: git-core-doc-2.18.1-3.el8.noarch", "Removed: git-email-2.18.1-3.el8.noarch", "Removed: python3-dnf-plugin-spacewalk-2.8.5-11.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: git-instaweb-2.18.1-3.el8.x86_64", "Removed: git-subtree-2.18.1-3.el8.x86_64", "Removed: git-svn-2.18.1-3.el8.x86_64", "Removed: gitk-2.18.1-3.el8.noarch", "Removed: git-gui-2.18.1-3.el8.noarch", "Removed: gitweb-2.18.1-3.el8.noarch", "Removed: python3-hawkey-0.22.5-4.el8.x86_64", "Removed: python3-firewall-0.6.3-7.el8.noarch", "Removed: python3-dnf-plugins-core-4.0.2.2-3.el8.noarch", "Removed: glibc-langpack-en-2.28-42.el8_0.1.x86_64", "Removed: glibc-2.28-42.el8_0.1.x86_64", "Removed: glibc-common-2.28-42.el8_0.1.x86_64", "Removed: python3-libdnf-0.22.5-4.el8.x86_64", "Removed: python3-librepo-1.9.2-1.el8.x86_64", "Removed: grubby-8.40-34.el8.x86_64", "Removed: python3-rhn-check-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rhn-client-tools-2.8.16-13.module+el8 2020-02-05T16:55:15.772 INFO:teuthology.task.ansible.out:.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rhn-setup-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: python3-rpm-4.14.2-11.el8_0.x86_64", "Removed: initscripts-10.00.1-1.el8_0.1.x86_64", "Removed: iproute-4.18.0-11.el8.x86_64", "Removed: iprutils-2.4.16.1-2.el8.x86_64", "Removed: iputils-20180629-1.el8.x86_64", "Removed: iwl100-firmware-39.31.5.1-92.el8.1.noarch", "Removed: iwl1000-firmware-1:39.31.5.1-92.el8.1.noarch", "Removed: iwl105-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl135-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl2000-firmware-18.168.6.1-92.el8.1.noarch", "Removed: iwl2030-firmware-18.168 2020-02-05T16:55:15.773 INFO:teuthology.task.ansible.out:.6.1-92.el8.1.noarch", "Removed: iwl3160-firmware-1:25.30.13.0-92.el8.1.noarch", "Removed: iwl3945-firmware-15.32.2.9-92.el8.1.noarch", "Removed: iwl4965-firmware-228.61.2.24-92.el8.1.noarch", "Removed: iwl5000-firmware-8.83.5.1_1-92.el8.1.noarch", "Removed: iwl5150-firmware-8.24.2.2-92.el8.1.noarch", "Removed: iwl6000-firmware-9.221.4.1-92.el8.1.noarch", "Removed: iwl6000g2a-firmware-18.168.6.1-92.el8.1.noarch", "Removed: rhn-check-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhn-client-tools-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhn-setup-2.8.16-13.module+el8.0.0.z+3494+e6f475f6.x86_64", "Removed: rhnlib-2.8.6-8.module+el8.0.0.z+3494+e6f475f6.noarch", "Removed: iwl6050-firmware-41.28.5.1-92.el8.1.noarch", "Removed: rpm-4.14.2-11.el8_0.x86_64", "Removed: rpm-build-libs-4.14.2-11.el8_0.x86_64", "Removed: rpm-libs-4.14.2-11.el8_0.x86_64", "Removed: iwl7260-firmware-1:25.30.13.0-92.el8.1.noarch", "Removed: rpm-plugin-systemd-inhibit-4.14.2-11.el8_0.x86_64", "Removed: rpm-plugin-selinux-4.14.2-11.el8_0.x86_64", "Removed: runc-1.0.0-56.rc8.dev.git425e105.module+el8.0.0+4017+bbba319f.x86_64", "Removed: kernel-tools-4.18.0-80.11.2.el8_0.x86_64", "Removed: kernel-tools-libs-4.18.0-80.11.2.el8_0.x86_64", "Removed: setup-2.12.2-2.el8.noarch", "Removed: kexec-tools-2.0.17-28.el8.x86_64", "Removed: sg3_utils-libs-1.44-2.el8.x86_64", "Removed: shadow-utils-2:4.6-7.el8.x86_64", "Removed: sg3_utils-1.44-2.el8.x86_64", "Removed: slirp4netns-0.1-2.dev.gitc4e1bc5.module+el8+2769+577ad176.x86_64", "Removed: python3-dnf-4.0.9.2-5.el8.noarch", "Removed: libcom_err-1.44.3-2.el8.x86_64", "Removed: libcurl-7.61.1-8.el8.x86_64", "Removed: libdnf-0.22.5-4.el8.x86_64", "Removed: libgcc-8.2.1-3.5.el8.x86_64", "Removed: libgcrypt-1.8.3-2.el8.x86_64", "Removed: libgomp-8.2.1-3.5.el8.x86_64", "Removed: vim-minimal-2:8.0.1763-11.el8_0.x86_64", "Removed: xfsprogs-4.19.0-2.el8.x86_64", "Removed: yum-4.0.9.2-5.el8.noarch", "Removed: librepo-1.9.2-1.el8.x86_64", " 2020-02-05T16:55:15.775 INFO:teuthology.task.ansible.out:Removed: libseccomp-2.3.3-3.el8.x86_64", "Removed: libsolv-0.6.35-6.el8.x86_64", "Removed: libss-1.44.3-2.el8.x86_64", "Removed: linux-firmware-20190111-92.gitd9fb2ee6.el8.noarch", "Removed: lshw-B.02.18-16.el8.x86_64", "Removed: openssh-7.8p1-4.el8.x86_64", "Removed: openssh-clients-7.8p1-4.el8.x86_64", "Removed: openssh-server-7.8p1-4.el8.x86_64", "Removed: openssl-1:1.1.1-8.el8.x86_64", "Removed: openssl-libs-1:1.1.1-8.el8.x86_64", "Removed: parted-3.2-36.el8.x86_64"]} 2020-02-05T16:55:15.820 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:15 +0000 (0:00:01.559) 0:11:03.149 **** 2020-02-05T16:55:15.945 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-02-05T16:55:15.946 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trEqs0f7NiuHlwLZj8CJQH8js4eaZdb6ti'], u'uuids': [u'd6cd7f0a-f585-4cfb-bf6d-dd4894000074']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.291916", "end": "2020-02-05 16:55:15.902816", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trEqs0f7NiuHlwLZj8CJQH8js4eaZdb6ti"], "labels": [], "masters": [], "uuids": ["d6cd7f0a-f585-4cfb-bf6d-dd4894000074"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:15.610900", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.280596 s, 411 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.280596 s, 411 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:16.554 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trM9Uhk4qDd3fzhk4Jdw6zqNIqxyAFtRdP'], u'uuids': [u'4a564494-9b48-4007-ab63-380635736df0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.296992", "end": "2020-02-05 16:55:16.507025", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0trM9Uhk4qDd3fzhk4Jdw6zqNIqxyAFtRdP"], "labels": [], "masters": [], "uuids": ["4a564494-9b48-4007-ab63-380635736df0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:16.210033", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.286325 s, 403 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.286325 s, 403 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:17.156 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0try3EZ6EIorxI1qcep0ooNFJVwIYJEhwtZ'], u'uuids': [u'3918f874-2a8f-4390-a744-e72297ebee8c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.285883", "end": "2020-02-05 16:55:17.116254", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0try3EZ6EIorxI1qcep0ooNFJVwIYJEhwtZ"], "labels": [], "masters": [], "uuids": ["3918f874-2a8f-4390-a744-e72297ebee8c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:16.830371", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.274135 s, 421 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.274135 s, 421 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:17.769 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0truEEWqmmEAwxabu2DuNsw3ysdWYf3Rey4'], u'uuids': [u'7e9b7ba3-862f-4445-b1ee-0e273a1ec23b']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.294269", "end": "2020-02-05 16:55:17.721210", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1TuTS9Jai17nDh0gDSmAfTRdF4Nhq0truEEWqmmEAwxabu2DuNsw3ysdWYf3Rey4"], "labels": [], "masters": [], "uuids": ["7e9b7ba3-862f-4445-b1ee-0e273a1ec23b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:17.426941", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.283758 s, 406 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.283758 s, 406 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:17.812 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:17 +0000 (0:00:01.992) 0:11:05.142 **** 2020-02-05T16:55:18.226 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-02-05T16:55:18.227 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.085256", "end": "2020-02-05 16:55:18.183446", "rc": 0, "start": "2020-02-05 16:55:18.098190", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:18.263 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.450) 0:11:05.593 **** 2020-02-05T16:55:18.679 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-02-05T16:55:18.681 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.075834", "end": "2020-02-05 16:55:18.630876", "rc": 0, "start": "2020-02-05 16:55:18.555042", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:18.718 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.454) 0:11:06.047 **** 2020-02-05T16:55:18.744 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-02-05T16:55:18.783 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.065) 0:11:06.112 **** 2020-02-05T16:55:18.818 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-02-05T16:55:18.820 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:18.861 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.078) 0:11:06.191 **** 2020-02-05T16:55:18.887 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-02-05T16:55:18.889 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:18.926 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.064) 0:11:06.255 **** 2020-02-05T16:55:18.953 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-02-05T16:55:18.955 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:18.993 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:18 +0000 (0:00:00.066) 0:11:06.322 **** 2020-02-05T16:55:19.019 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-02-05T16:55:19.020 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.060 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.066) 0:11:06.389 **** 2020-02-05T16:55:19.377 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-02-05T16:55:19.378 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:55:19.413 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.353) 0:11:06.742 **** 2020-02-05T16:55:19.462 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-02-05T16:55:19.464 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={u'block_used': 13449165, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645943, u'size_available': 928341782528, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991236, u'device': u'/dev/sda1', u'inode_used': 63740, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645943, "block_size": 4096, "block_total": 240095108, "block_used": 13449165, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991236, "inode_total": 61054976, "inode_used": 63740, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341782528, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.500 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.087) 0:11:06.829 **** 2020-02-05T16:55:19.536 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-02-05T16:55:19.557 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.571 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.070) 0:11:06.900 **** 2020-02-05T16:55:19.606 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:19.613 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.648 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.076) 0:11:06.977 **** 2020-02-05T16:55:19.674 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:19.675 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.710 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.062) 0:11:07.040 **** 2020-02-05T16:55:19.747 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-02-05T16:55:19.748 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.782 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.072) 0:11:07.112 **** 2020-02-05T16:55:19.808 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-02-05T16:55:19.810 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.844 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.061) 0:11:07.173 **** 2020-02-05T16:55:19.879 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-02-05T16:55:19.881 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:19.917 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:19 +0000 (0:00:00.072) 0:11:07.246 **** 2020-02-05T16:55:20.868 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-02-05T16:55:20.870 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-02-05T16:55:20.909 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:20 +0000 (0:00:00.992) 0:11:08.239 **** 2020-02-05T16:55:21.789 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-02-05T16:55:21.791 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:22.340 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:22.878 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:22.934 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-02-05T16:55:22.935 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dbench", "Installed: dbench-4.0-20.el8.x86_64"]} 2020-02-05T16:55:22.980 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:22 +0000 (0:00:02.070) 0:11:10.309 **** 2020-02-05T16:55:22.982 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:23.419 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-02-05T16:55:23.421 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:23.960 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:24.001 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:23 +0000 (0:00:01.020) 0:11:11.330 **** 2020-02-05T16:55:24.318 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-02-05T16:55:24.320 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-02-05T16:55:24.357 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:24 +0000 (0:00:00.356) 0:11:11.686 **** 2020-02-05T16:55:24.393 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-02-05T16:55:24.394 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:24.699 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:25.015 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:25.326 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:25.401 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-02-05T16:55:25.403 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-02-05T16:55:25.441 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:25 +0000 (0:00:01.084) 0:11:12.770 **** 2020-02-05T16:55:25.479 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:55:25.481 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:25.522 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:25 +0000 (0:00:00.081) 0:11:12.852 **** 2020-02-05T16:55:25.636 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-02-05T16:55:25.638 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:25.696 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:25 +0000 (0:00:00.173) 0:11:13.025 **** 2020-02-05T16:55:26.023 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-02-05T16:55:26.028 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-02-05T16:55:26.030 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-02-05T16:55:26.032 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice dbus.socket basic.target sysinit.target dbus.service polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ipset.service iptables.service ebtables.service shutdown.target ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-02-05T16:55:26.034 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127808", "LimitNPROCSoft": "127808", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127808", 2020-02-05T16:55:26.036 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127808", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket system.slice sysinit.target", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-02-05T16:55:26.038 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-02-05T16:55:26.072 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:26 +0000 (0:00:00.376) 0:11:13.402 **** 2020-02-05T16:55:26.105 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:26 +0000 (0:00:00.032) 0:11:13.434 **** 2020-02-05T16:55:26.131 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-02-05T16:55:26.132 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:26.170 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:26 +0000 (0:00:00.065) 0:11:13.500 **** 2020-02-05T16:55:26.531 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-02-05T16:55:26.532 INFO:teuthology.task.ansible.out:fatal: [smithi008.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-02-05T16:55:26.574 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:26 +0000 (0:00:00.403) 0:11:13.904 **** 2020-02-05T16:55:27.009 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-02-05T16:55:27.011 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-02-05T16:55:27.046 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.471) 0:11:14.375 **** 2020-02-05T16:55:27.072 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-02-05T16:55:27.073 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.108 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.062) 0:11:14.438 **** 2020-02-05T16:55:27.135 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-02-05T16:55:27.171 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.062) 0:11:14.501 **** 2020-02-05T16:55:27.197 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-02-05T16:55:27.199 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.234 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.062) 0:11:14.563 **** 2020-02-05T16:55:27.270 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-02-05T16:55:27.271 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.305 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.071) 0:11:14.635 **** 2020-02-05T16:55:27.342 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-02-05T16:55:27.344 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.379 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.074) 0:11:14.709 **** 2020-02-05T16:55:27.405 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-02-05T16:55:27.407 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.441 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.061) 0:11:14.771 **** 2020-02-05T16:55:27.468 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-02-05T16:55:27.505 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.063) 0:11:14.835 **** 2020-02-05T16:55:27.531 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-02-05T16:55:27.533 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.568 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.062) 0:11:14.897 **** 2020-02-05T16:55:27.605 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-02-05T16:55:27.607 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.649 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.080) 0:11:14.978 **** 2020-02-05T16:55:27.675 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-02-05T16:55:27.677 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.712 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.063) 0:11:15.042 **** 2020-02-05T16:55:27.739 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:55:27.740 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.777 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.064) 0:11:15.106 **** 2020-02-05T16:55:27.813 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-02-05T16:55:27.815 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.858 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.073) 0:11:15.179 **** 2020-02-05T16:55:27.893 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:27.929 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.079) 0:11:15.259 **** 2020-02-05T16:55:27.940 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi164.front.sepia.ceph.com] => {"changed": true} 2020-02-05T16:55:27.979 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:27 +0000 (0:00:00.049) 0:11:15.308 **** 2020-02-05T16:55:27.998 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.040 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.060) 0:11:15.369 **** 2020-02-05T16:55:28.069 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2020-02-05T16:55:28.071 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.130 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.090) 0:11:15.459 **** 2020-02-05T16:55:28.158 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-02-05T16:55:28.160 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.217 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.086) 0:11:15.546 **** 2020-02-05T16:55:28.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-02-05T16:55:28.247 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.285 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.067) 0:11:15.614 **** 2020-02-05T16:55:28.312 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.391 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.105) 0:11:15.720 **** 2020-02-05T16:55:28.419 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.458 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.066) 0:11:15.787 **** 2020-02-05T16:55:28.484 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-02-05T16:55:28.486 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.526 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.068) 0:11:15.856 **** 2020-02-05T16:55:28.554 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-02-05T16:55:28.555 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.598 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.071) 0:11:15.927 **** 2020-02-05T16:55:28.626 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.684 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.085) 0:11:16.013 **** 2020-02-05T16:55:28.713 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-02-05T16:55:28.714 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.782 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.088) 0:11:16.102 **** 2020-02-05T16:55:28.808 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.845 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.072) 0:11:16.174 **** 2020-02-05T16:55:28.882 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-02-05T16:55:28.883 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.919 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.073) 0:11:16.248 **** 2020-02-05T16:55:28.945 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-02-05T16:55:28.947 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:28.983 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:28 +0000 (0:00:00.064) 0:11:16.313 **** 2020-02-05T16:55:29.010 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.046 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.062) 0:11:16.375 **** 2020-02-05T16:55:29.073 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.110 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.063) 0:11:16.439 **** 2020-02-05T16:55:29.137 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-02-05T16:55:29.138 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.175 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.065) 0:11:16.505 **** 2020-02-05T16:55:29.182 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-02-05T16:55:29.185 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-02-05T16:55:29.227 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.051) 0:11:16.556 **** 2020-02-05T16:55:29.233 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-02-05T16:55:29.234 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.272 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.045) 0:11:16.601 **** 2020-02-05T16:55:29.277 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-02-05T16:55:29.279 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.324 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.051) 0:11:16.653 **** 2020-02-05T16:55:29.329 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-02-05T16:55:29.332 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.375 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.045) 0:11:16.699 **** TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.413 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.043) 0:11:16.743 **** 2020-02-05T16:55:29.419 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-02-05T16:55:29.421 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.462 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.048) 0:11:16.791 **** 2020-02-05T16:55:29.468 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-02-05T16:55:29.469 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.516 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.054) 0:11:16.845 **** 2020-02-05T16:55:29.521 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-02-05T16:55:29.522 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.567 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.051) 0:11:16.897 **** 2020-02-05T16:55:29.573 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-02-05T16:55:29.574 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.619 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.051) 0:11:16.949 **** 2020-02-05T16:55:29.624 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-02-05T16:55:29.625 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.663 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.043) 0:11:16.993 **** 2020-02-05T16:55:29.676 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.707 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.043) 0:11:17.036 **** 2020-02-05T16:55:29.714 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-02-05T16:55:29.715 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.753 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.046) 0:11:17.082 **** 2020-02-05T16:55:29.760 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-02-05T16:55:29.761 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.802 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.048) 0:11:17.131 **** 2020-02-05T16:55:29.807 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-02-05T16:55:29.808 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.849 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.047) 0:11:17.179 **** 2020-02-05T16:55:29.856 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-02-05T16:55:29.858 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.894 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.044) 0:11:17.223 **** 2020-02-05T16:55:29.899 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-02-05T16:55:29.901 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.938 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.043) 0:11:17.267 **** 2020-02-05T16:55:29.944 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-02-05T16:55:29.946 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:29.982 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:29 +0000 (0:00:00.044) 0:11:17.312 **** 2020-02-05T16:55:29.987 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-02-05T16:55:29.989 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.060 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.043) 0:11:17.355 **** TASK [testnode : Install ntp package on apt based systems.] ******************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.070 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.043) 0:11:17.399 **** 2020-02-05T16:55:30.075 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-02-05T16:55:30.075 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.126 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.056) 0:11:17.456 **** 2020-02-05T16:55:30.132 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-02-05T16:55:30.133 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.174 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.047) 0:11:17.503 **** 2020-02-05T16:55:30.180 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-02-05T16:55:30.180 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.219 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.044) 0:11:17.548 **** 2020-02-05T16:55:30.224 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-02-05T16:55:30.225 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.263 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.044) 0:11:17.593 **** 2020-02-05T16:55:30.271 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-02-05T16:55:30.272 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.317 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.050) 0:11:17.643 **** 2020-02-05T16:55:30.341 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-02-05T16:55:30.343 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.385 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.069) 0:11:17.713 **** 2020-02-05T16:55:30.412 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-02-05T16:55:30.413 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.455 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.071) 0:11:17.785 **** 2020-02-05T16:55:30.486 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.520 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.064) 0:11:17.849 **** 2020-02-05T16:55:30.551 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.589 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.069) 0:11:17.919 **** 2020-02-05T16:55:30.619 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.652 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.062) 0:11:17.981 **** 2020-02-05T16:55:30.688 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.735 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.078) 0:11:18.060 **** 2020-02-05T16:55:30.758 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.802 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.066) 0:11:18.127 **** 2020-02-05T16:55:30.824 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-02-05T16:55:30.826 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.865 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.067) 0:11:18.194 **** 2020-02-05T16:55:30.891 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-02-05T16:55:30.892 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:30.930 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:30 +0000 (0:00:00.064) 0:11:18.259 **** 2020-02-05T16:55:30.965 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.014 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.083) 0:11:18.343 **** 2020-02-05T16:55:31.050 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.079 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.065) 0:11:18.408 **** 2020-02-05T16:55:31.105 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.145 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.065) 0:11:18.474 **** 2020-02-05T16:55:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-02-05T16:55:31.173 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.208 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.063) 0:11:18.537 **** 2020-02-05T16:55:31.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-02-05T16:55:31.215 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-02-05T16:55:31.252 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.044) 0:11:18.581 **** 2020-02-05T16:55:31.257 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-02-05T16:55:31.258 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.296 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.044) 0:11:18.626 **** 2020-02-05T16:55:31.326 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-02-05T16:55:31.328 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.366 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.067) 0:11:18.694 **** 2020-02-05T16:55:31.390 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-02-05T16:55:31.392 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.427 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.063) 0:11:18.757 **** 2020-02-05T16:55:31.455 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-02-05T16:55:31.457 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.491 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.063) 0:11:18.820 **** 2020-02-05T16:55:31.519 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:31.520 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-02-05T16:55:31.559 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.067) 0:11:18.888 **** 2020-02-05T16:55:31.598 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-02-05T16:55:31.600 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.610 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-02-05T16:55:31.624 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.625 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.636 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.647 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.658 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.723 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.163) 0:11:19.052 **** 2020-02-05T16:55:31.752 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-02-05T16:55:31.754 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-02-05 16:50:23 UTC", "ActiveEnterTimestampMonotonic": "425256365", "ActiveExitTimestamp": "Wed 2020-02-05 16:50:23 UTC", "ActiveExitTimestampMonotonic": "425204009", "ActiveState": "active", "After": "ntpdate.service ntpd.service sntp.service systemd-tmpfiles-setup.service tmp.mount systemd-journald.socket -.mount sysinit.target system.slice basic.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-02-05 16:50:23 UTC", "AssertTimestampMonotonic": "425210724", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-02-05 16:50:23 UTC", "ConditionTimestampMonotonic": "425210642", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": " 2020-02-05T16:55:31.757 INFO:teuthology.task.ansible.out:0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "18997", "ExecMainStartTimestamp": "Wed 2020-02-05 16:50:23 UTC", "ExecMainStartTimestampMonotonic": "425236441", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Wed 2020-02-05 16:50:23 UTC", "InactiveEnterTimestampMonotonic": "425208064", "InactiveExitTimestamp": "Wed 2020-02-05 16:50:23 UTC", "InactiveExitTimestampMonotonic": "425213669", "InvocationID": "9331fdae241748eba3496b24d9840991", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "L 2020-02-05T16:55:31.758 INFO:teuthology.task.ansible.out:imitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127783", "LimitNPROCSoft": "127783", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127783", "LimitSIGPENDINGSoft": "127783", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "18997", "MemoryAccounting": "yes", "MemoryCurrent": "1937408", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": 2020-02-05T16:55:31.760 INFO:teuthology.task.ansible.out:"infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-02-05 16:50:23 UTC", "StateChangeTimestampMonotonic": "425256365", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-02-05 16:50:23 UTC", "WatchdogTimestampMonotonic": "425236485", "WatchdogUSec": "0"}} 2020-02-05T16:55:31.796 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.073) 0:11:19.125 **** 2020-02-05T16:55:31.823 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.866 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.065) 0:11:19.191 **** 2020-02-05T16:55:31.878 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-02-05T16:55:31.879 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003264", "end": "2020-02-05 16:55:31.855510", "rc": 0, "start": "2020-02-05 16:55:31.852246", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-02-05T16:55:31.942 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.080) 0:11:19.272 **** 2020-02-05T16:55:31.948 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-02-05T16:55:31.949 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:31.986 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:31 +0000 (0:00:00.043) 0:11:19.316 **** 2020-02-05T16:55:31.992 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:32.039 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.053) 0:11:19.369 **** 2020-02-05T16:55:32.046 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-02-05T16:55:32.048 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:32.104 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.065) 0:11:19.434 **** 2020-02-05T16:55:32.110 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-02-05T16:55:32.111 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:32.155 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.050) 0:11:19.484 **** 2020-02-05T16:55:32.195 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:32.231 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.075) 0:11:19.560 **** 2020-02-05T16:55:32.257 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-02-05T16:55:32.258 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:32.297 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.065) 0:11:19.626 **** 2020-02-05T16:55:32.436 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-02-05T16:55:32.437 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-02-05T16:55:32.478 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:32 +0000 (0:00:00.181) 0:11:19.807 **** 2020-02-05T16:55:33.389 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-02-05T16:55:33.390 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-02-05T16:55:33.432 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:33 +0000 (0:00:00.954) 0:11:20.762 **** 2020-02-05T16:55:34.752 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-02-05T16:55:34.754 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-02-05T16:55:34.795 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:34 +0000 (0:00:01.362) 0:11:22.124 **** 2020-02-05T16:55:34.844 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-02-05T16:55:34.844 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item={u'block_used': 13449180, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645928, u'size_available': 928341721088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991233, u'device': u'/dev/sda1', u'inode_used': 63743, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645928, "block_size": 4096, "block_total": 240095108, "block_used": 13449180, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991233, "inode_total": 61054976, "inode_used": 63743, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341721088, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}} 2020-02-05T16:55:34.890 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:34 +0000 (0:00:00.094) 0:11:22.219 **** 2020-02-05T16:55:35.237 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-02-05T16:55:35.238 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.023012", "end": "2020-02-05 16:55:35.199205", "rc": 0, "start": "2020-02-05 16:55:35.176193", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:35.300 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.410) 0:11:22.630 **** 2020-02-05T16:55:35.308 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-02-05T16:55:35.309 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-02-05T16:55:35.351 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.050) 0:11:22.680 **** 2020-02-05T16:55:35.352 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:35.358 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:35.360 INFO:teuthology.task.ansible.out:TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi008.front.sepia.ceph.com] => (item={u'block_used': 13449180, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645928, u'size_available': 928341721088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991233, u'device': u'/dev/sda1', u'inode_used': 63743, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645928, "block_size": 4096, "block_total": 240095108, "block_used": 13449180, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991233, "inode_total": 61054976, "inode_used": 63743, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341721088, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:35.401 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.050) 0:11:22.730 **** 2020-02-05T16:55:35.407 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-02-05T16:55:35.409 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:35.456 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.047) 0:11:22.778 **** TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-02-05T16:55:35.498 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.047) 0:11:22.826 **** 2020-02-05T16:55:35.502 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-02-05T16:55:35.506 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-02-05T16:55:35.548 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.052) 0:11:22.878 **** 2020-02-05T16:55:35.556 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-02-05T16:55:35.557 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4H3ZL', u'wwn-0x5000c500871b3d00'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4H3ZL', u'wwn': u'0x5000c500871b3d00', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4H3ZL-part1', u'wwn-0x5000c500871b3d00-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3ZL", "wwn-0x5000c500871b3d00"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3ZL-part1", "wwn-0x5000c500871b3d00-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H3ZL", "size": "931.51 GB", "support_discard": "0", "vend 2020-02-05T16:55:35.558 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871b3d00"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:35.865 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** ok: [smithi164.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-02-05T16:55:35.901 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.352) 0:11:23.230 **** 2020-02-05T16:55:35.929 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-02-05T16:55:35.931 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:35.969 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:35 +0000 (0:00:00.068) 0:11:23.299 **** 2020-02-05T16:55:35.995 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-02-05T16:55:35.997 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-02-05T16:55:36.033 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.063) 0:11:23.363 **** 2020-02-05T16:55:36.058 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-02-05T16:55:36.059 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.094 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.061) 0:11:23.424 **** 2020-02-05T16:55:36.410 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-02-05T16:55:36.411 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-02-05T16:55:36.457 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.362) 0:11:23.787 **** 2020-02-05T16:55:36.777 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-02-05T16:55:36.779 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-02-05T16:55:36.819 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.361) 0:11:24.148 **** 2020-02-05T16:55:36.845 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-02-05T16:55:36.847 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.884 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.065) 0:11:24.214 **** 2020-02-05T16:55:36.921 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-02-05T16:55:36.922 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi164.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.933 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.933 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.944 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-02-05T16:55:36.983 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:36 +0000 (0:00:00.098) 0:11:24.312 **** 2020-02-05T16:55:37.030 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-02-05T16:55:37.032 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.042 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.054 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.056 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.066 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.104 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.121) 0:11:24.434 **** 2020-02-05T16:55:37.130 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:37.132 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.171 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.066) 0:11:24.500 **** 2020-02-05T16:55:37.197 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-02-05T16:55:37.198 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.239 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.068) 0:11:24.569 **** 2020-02-05T16:55:37.265 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-02-05T16:55:37.267 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.305 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.065) 0:11:24.635 **** 2020-02-05T16:55:37.331 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-02-05T16:55:37.333 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.368 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.063) 0:11:24.698 **** 2020-02-05T16:55:37.394 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-02-05T16:55:37.396 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.434 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.065) 0:11:24.763 **** 2020-02-05T16:55:37.460 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.502 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.068) 0:11:24.831 **** 2020-02-05T16:55:37.538 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-02-05T16:55:37.540 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.585 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.082) 0:11:24.914 **** 2020-02-05T16:55:37.590 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-02-05T16:55:37.592 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Qskvcz-3lKA-NBPO-vT8D-MUjc-GxX6-u2GbqF', u'nvme-INTEL_SSDPEDMD400G4_CVFT5331002K400BGN', u'nvme-nvme.8086-43564654353333313030324b34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.694257", "end": "2020-02-05 16:55:37.520155", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Qskvcz-3lKA-NBPO-vT8D-MUjc-GxX6-u2GbqF", "nvme-INTEL_SSDPEDMD400G4_CVFT5331002K400BGN", "nvme-nvme.8086-43564654353333313030324b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:35.825898", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-02-05T16:55:37.594 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:37.614 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-02-05T16:55:37.616 INFO:teuthology.task.ansible.out:skipping: [smithi164.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:37.651 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:37 +0000 (0:00:00.066) 0:11:24.981 **** 2020-02-05T16:55:37.653 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:38.015 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-02-05T16:55:38.016 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.017075", "end": "2020-02-05 16:55:37.964131", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-02-05 16:55:37.947056", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:38.339 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.013542", "end": "2020-02-05 16:55:38.288901", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-02-05 16:55:38.275359", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:38.656 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.013281", "end": "2020-02-05 16:55:38.617670", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:55:38.604389", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:38.985 INFO:teuthology.task.ansible.out:ok: [smithi164.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.011548", "end": "2020-02-05 16:55:38.940760", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-02-05 16:55:38.929212", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:39.034 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:39 +0000 (0:00:01.379) 0:11:26.360 **** 2020-02-05T16:55:39.042 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-02-05T16:55:39.058 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL50Kmod319yZ5SVoyUEcmFv3uT9fZ54xad'], u'uuids': [u'64bfd9f2-671a-4b1f-b007-7798a5ba8125']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.106381", "end": "2020-02-05 16:55:38.970627", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL50Kmod319yZ5SVoyUEcmFv3uT9fZ54xad"], "labels": [], "masters": [], "uuids": ["64bfd9f2-671a-4b1f-b007-7798a5ba8125"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:37.864246", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:39.059 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-02-05T16:55:39.101 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:39 +0000 (0:00:00.068) 0:11:26.429 **** 2020-02-05T16:55:39.430 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-02-05T16:55:39.431 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-02-05T16:55:39.469 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:39 +0000 (0:00:00.369) 0:11:26.799 **** 2020-02-05T16:55:40.014 INFO:teuthology.task.ansible.out: RUNNING HANDLER [common : restart nagios-nrpe-server] ************************** 2020-02-05T16:55:40.017 INFO:teuthology.task.ansible.out:changed: [smithi164.front.sepia.ceph.com] => {"changed": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-02-05 16:47:21 UTC", "ActiveEnterTimestampMonotonic": "243084952", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-journald.socket basic.target time-sync.target sysinit.target systemd-tmpfiles-setup.service system.slice -.mount nss-lookup.target local-fs.target network.target tmp.mount var-run.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-02-05 16:47:21 UTC", "AssertTimestampMonotonic": "243083342", "Before": "multi-user.target plymouth-quit.service shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-02-05 16:47:21 UTC", "ConditionTimestampMonotonic": "243083341", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDepe 2020-02-05T16:55:40.018 INFO:teuthology.task.ansible.out:ndencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "11689", "ExecMainStartTimestamp": "Wed 2020-02-05 16:47:21 UTC", "ExecMainStartTimestampMonotonic": "243084841", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-02-05 16:47:21 UTC", "InactiveExitTimestampMonotonic": "243084952", "InvocationID": "2592b89a543d4d6e9dc6a0f505a097de", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft 2020-02-05T16:55:40.020 INFO:teuthology.task.ansible.out:": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127783", "LimitNPROCSoft": "127783", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127783", "LimitSIGPENDINGSoft": "127783", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "11689", "MemoryAccounting": "yes", "MemoryCurrent": "741376", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "075 2020-02-05T16:55:40.021 INFO:teuthology.task.ansible.out:5", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-02-05 16:47:21 UTC", "StateChangeTimestampMonotonic": "243084952", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-02-05 16:47:21 UTC", "WatchdogTimestampMonotonic": "243084951", "WatchdogUSec": "0"}} 2020-02-05T16:55:40.488 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5ro7mUjy4m45UbWlI2o0aNaiIW0ke8GyA'], u'uuids': [u'5fdb53b3-b9e9-4518-bd76-8c254ab00fe0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.118931", "end": "2020-02-05 16:55:40.441864", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5ro7mUjy4m45UbWlI2o0aNaiIW0ke8GyA"], "labels": [], "masters": [], "uuids": ["5fdb53b3-b9e9-4518-bd76-8c254ab00fe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:39.322933", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! 2020-02-05T16:55:40.489 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:41.878 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL53pPEGgEBkIfliBjuh6RpTds6eVTzqJFJ'], u'uuids': [u'0cd6aee8-e190-4735-a2ef-20a8ead51f1f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.066705", "end": "2020-02-05 16:55:41.835708", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL53pPEGgEBkIfliBjuh6RpTds6eVTzqJFJ"], "labels": [], "masters": [], "uuids": ["0cd6aee8-e190-4735-a2ef-20a8ead51f1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:40.769003", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:43.292 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5j4vFKCIaWoQfZBdWqOQX3enhtIeMfJiT'], u'uuids': [u'77a8b987-81e2-4813-b9c4-a6d95e93caaf']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.080547", "end": "2020-02-05 16:55:43.244326", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5j4vFKCIaWoQfZBdWqOQX3enhtIeMfJiT"], "labels": [], "masters": [], "uuids": ["77a8b987-81e2-4813-b9c4-a6d95e93caaf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:42.163779", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:44.695 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5il9YwWBRdsTRXaiVOVDi3isMsVTxdCGh'], u'uuids': [u'f4e53adc-76c1-433c-85bb-a9a222c85820']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.079566", "end": "2020-02-05 16:55:44.654316", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5il9YwWBRdsTRXaiVOVDi3isMsVTxdCGh"], "labels": [], "masters": [], "uuids": ["f4e53adc-76c1-433c-85bb-a9a222c85820"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:43.574750", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-02-05T16:55:44.738 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:44 +0000 (0:00:05.268) 0:11:32.067 **** 2020-02-05T16:55:44.778 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2020-02-05T16:55:44.780 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4H3ZL', u'wwn-0x5000c500871b3d00'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4H3ZL', u'wwn': u'0x5000c500871b3d00', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4H3ZL-part1', u'wwn-0x5000c500871b3d00-part1'], u'uuids': [u'cf568be9-1a39-43d2-a003-fbe6239693bc']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3ZL", "wwn-0x5000c500871b3d00"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H3ZL-part1", "wwn-0x5000c500871b3d00-part1"], "labels": [], "masters": [], "uuids": ["cf568be9-1a39-43d2-a003-fbe6239693bc"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H3ZL", "size": "931.51 GB", "support_discard": "0", "vend 2020-02-05T16:55:44.782 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c500871b3d00"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:46.278 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-Qskvcz-3lKA-NBPO-vT8D-MUjc-GxX6-u2GbqF', u'nvme-INTEL_SSDPEDMD400G4_CVFT5331002K400BGN', u'nvme-nvme.8086-43564654353333313030324b34303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:01.162499", "end": "2020-02-05 16:55:46.232059", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-Qskvcz-3lKA-NBPO-vT8D-MUjc-GxX6-u2GbqF", "nvme-INTEL_SSDPEDMD400G4_CVFT5331002K400BGN", "nvme-nvme.8086-43564654353333313030324b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:45.069560", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-02-05T16:55:46.280 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.125965 s, 916 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.125965 s, 916 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:46.904 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL50Kmod319yZ5SVoyUEcmFv3uT9fZ54xad'], u'uuids': [u'64bfd9f2-671a-4b1f-b007-7798a5ba8125']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.304018", "end": "2020-02-05 16:55:46.869792", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL50Kmod319yZ5SVoyUEcmFv3uT9fZ54xad"], "labels": [], "masters": [], "uuids": ["64bfd9f2-671a-4b1f-b007-7798a5ba8125"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:46.565774", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.292212 s, 395 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.292212 s, 395 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:47.542 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5ro7mUjy4m45UbWlI2o0aNaiIW0ke8GyA'], u'uuids': [u'5fdb53b3-b9e9-4518-bd76-8c254ab00fe0']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.302816", "end": "2020-02-05 16:55:47.490813", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5ro7mUjy4m45UbWlI2o0aNaiIW0ke8GyA"], "labels": [], "masters": [], "uuids": ["5fdb53b3-b9e9-4518-bd76-8c254ab00fe0"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:47.187997", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.290763 s, 397 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.290763 s, 397 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:48.151 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL53pPEGgEBkIfliBjuh6RpTds6eVTzqJFJ'], u'uuids': [u'0cd6aee8-e190-4735-a2ef-20a8ead51f1f']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.298655", "end": "2020-02-05 16:55:48.100063", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL53pPEGgEBkIfliBjuh6RpTds6eVTzqJFJ"], "labels": [], "masters": [], "uuids": ["0cd6aee8-e190-4735-a2ef-20a8ead51f1f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:47.801408", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.286778 s, 402 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.286778 s, 402 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:48.775 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5j4vFKCIaWoQfZBdWqOQX3enhtIeMfJiT'], u'uuids': [u'77a8b987-81e2-4813-b9c4-a6d95e93caaf']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.322609", "end": "2020-02-05 16:55:48.740850", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5j4vFKCIaWoQfZBdWqOQX3enhtIeMfJiT"], "labels": [], "masters": [], "uuids": ["77a8b987-81e2-4813-b9c4-a6d95e93caaf"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:48.418241", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.310986 s, 371 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.310986 s, 371 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:49.395 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5il9YwWBRdsTRXaiVOVDi3isMsVTxdCGh'], u'uuids': [u'f4e53adc-76c1-433c-85bb-a9a222c85820']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.301885", "end": "2020-02-05 16:55:49.357858", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-g81oWOUFAEYkyzCputX115JJMXGeohL5il9YwWBRdsTRXaiVOVDi3isMsVTxdCGh"], "labels": [], "masters": [], "uuids": ["f4e53adc-76c1-433c-85bb-a9a222c85820"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-02-05 16:55:49.055973", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.289924 s, 398 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.289924 s, 398 MB/s"], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:49.458 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:49 +0000 (0:00:04.720) 0:11:36.787 **** 2020-02-05T16:55:49.893 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-02-05T16:55:49.895 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.099630", "end": "2020-02-05 16:55:49.849845", "rc": 0, "start": "2020-02-05 16:55:49.750215", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:49.931 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:49 +0000 (0:00:00.472) 0:11:37.260 **** 2020-02-05T16:55:50.340 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-02-05T16:55:50.342 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.074734", "end": "2020-02-05 16:55:50.301144", "rc": 0, "start": "2020-02-05 16:55:50.226410", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:55:50.397 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.466) 0:11:37.727 **** 2020-02-05T16:55:50.424 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-02-05T16:55:50.463 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.066) 0:11:37.793 **** 2020-02-05T16:55:50.489 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-02-05T16:55:50.491 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:50.530 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.065) 0:11:37.859 **** 2020-02-05T16:55:50.556 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:50.594 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.063) 0:11:37.923 **** 2020-02-05T16:55:50.620 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-02-05T16:55:50.622 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:50.658 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.064) 0:11:37.988 **** 2020-02-05T16:55:50.684 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-02-05T16:55:50.686 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:50.720 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:50 +0000 (0:00:00.061) 0:11:38.049 **** 2020-02-05T16:55:51.047 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-02-05T16:55:51.049 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-02-05T16:55:51.084 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.364) 0:11:38.413 **** 2020-02-05T16:55:51.120 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-02-05T16:55:51.122 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={u'block_used': 13449180, u'uuid': u'cf568be9-1a39-43d2-a003-fbe6239693bc', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 226645928, u'size_available': 928341721088, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60991233, u'device': u'/dev/sda1', u'inode_used': 63743, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226645928, "block_size": 4096, "block_total": 240095108, "block_used": 13449180, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60991233, "inode_total": 61054976, "inode_used": 63743, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 928341721088, "size_total": 983429562368, "uuid": "cf568be9-1a39-43d2-a003-fbe6239693bc"}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.158 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.074) 0:11:38.488 **** 2020-02-05T16:55:51.194 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-02-05T16:55:51.196 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.231 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.072) 0:11:38.560 **** 2020-02-05T16:55:51.256 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-02-05T16:55:51.258 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.292 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.061) 0:11:38.622 **** 2020-02-05T16:55:51.329 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.364 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.072) 0:11:38.694 **** 2020-02-05T16:55:51.409 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-02-05T16:55:51.411 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.451 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.086) 0:11:38.780 **** 2020-02-05T16:55:51.477 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-02-05T16:55:51.479 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.514 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.063) 0:11:38.843 **** 2020-02-05T16:55:51.540 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-02-05T16:55:51.541 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:51.578 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:51 +0000 (0:00:00.064) 0:11:38.908 **** 2020-02-05T16:55:52.285 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-02-05T16:55:52.286 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-02-05T16:55:52.321 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:52 +0000 (0:00:00.742) 0:11:39.651 **** 2020-02-05T16:55:52.936 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-02-05T16:55:52.937 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:53.489 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:54.069 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:54.640 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:55.235 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-02-05T16:55:55.273 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:55 +0000 (0:00:02.951) 0:11:42.602 **** 2020-02-05T16:55:55.615 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-02-05T16:55:55.617 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-02-05T16:55:55.655 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:55 +0000 (0:00:00.381) 0:11:42.984 **** 2020-02-05T16:55:55.694 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-02-05T16:55:55.695 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:56.021 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:56.345 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:56.346 INFO:teuthology.task.ansible.out: 2020-02-05T16:55:56.696 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:57.007 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-02-05T16:55:57.054 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:57 +0000 (0:00:01.398) 0:11:44.383 **** 2020-02-05T16:55:57.392 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-02-05T16:55:57.393 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-02-05T16:55:57.441 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:57 +0000 (0:00:00.387) 0:11:44.770 **** 2020-02-05T16:55:57.467 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-02-05T16:55:57.469 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:57.507 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:57 +0000 (0:00:00.065) 0:11:44.836 **** 2020-02-05T16:55:57.970 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-02-05T16:55:57.972 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true} 2020-02-05T16:55:58.008 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:58 +0000 (0:00:00.500) 0:11:45.337 **** 2020-02-05T16:55:58.974 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-02-05T16:55:59.016 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:01.008) 0:11:46.345 **** 2020-02-05T16:55:59.043 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-02-05T16:55:59.045 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.086 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.070) 0:11:46.416 **** 2020-02-05T16:55:59.112 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-02-05T16:55:59.114 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.153 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.066) 0:11:46.482 **** 2020-02-05T16:55:59.176 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-02-05T16:55:59.178 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.236 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.083) 0:11:46.565 **** 2020-02-05T16:55:59.265 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-02-05T16:55:59.266 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.313 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.077) 0:11:46.643 **** 2020-02-05T16:55:59.340 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-02-05T16:55:59.342 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.378 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.064) 0:11:46.707 **** 2020-02-05T16:55:59.404 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-02-05T16:55:59.406 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.465 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.087) 0:11:46.795 **** 2020-02-05T16:55:59.493 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-02-05T16:55:59.494 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.533 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.067) 0:11:46.862 **** 2020-02-05T16:55:59.560 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-02-05T16:55:59.561 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.598 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.064) 0:11:46.927 **** 2020-02-05T16:55:59.623 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-02-05T16:55:59.625 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.661 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.063) 0:11:46.991 **** 2020-02-05T16:55:59.687 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-02-05T16:55:59.688 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.723 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.062) 0:11:47.053 **** 2020-02-05T16:55:59.749 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-02-05T16:55:59.750 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:55:59.787 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:55:59 +0000 (0:00:00.063) 0:11:47.116 **** 2020-02-05T16:56:00.707 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-02-05T16:56:00.745 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:00 +0000 (0:00:00.957) 0:11:48.074 **** 2020-02-05T16:56:01.233 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-02-05T16:56:01.235 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-02-05 16:50:35 UTC", "ActiveEnterTimestampMonotonic": "517085197", "ActiveExitTimestamp": "Wed 2020-02-05 16:50:35 UTC", "ActiveExitTimestampMonotonic": "517031448", "ActiveState": "active", "After": "-.mount sysinit.target ntpd.service system.slice tmp.mount sntp.service systemd-tmpfiles-setup.service systemd-journald.socket basic.target ntpdate.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-02-05 16:50:35 UTC", "AssertTimestampMonotonic": "517039359", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-02-05 16:50:35 UTC", "ConditionTimestampMonotonic": "517039249", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": " 2020-02-05T16:56:01.237 INFO:teuthology.task.ansible.out:0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "18968", "ExecMainStartTimestamp": "Wed 2020-02-05 16:50:35 UTC", "ExecMainStartTimestampMonotonic": "517063370", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestamp": "Wed 2020-02-05 16:50:35 UTC", "InactiveEnterTimestampMonotonic": "517035691", "InactiveExitTimestamp": "Wed 2020-02-05 16:50:35 UTC", "InactiveExitTimestampMonotonic": "517043090", "InvocationID": "e1c02936d47846919d473fc53acca854", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "L 2020-02-05T16:56:01.238 INFO:teuthology.task.ansible.out:imitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127808", "LimitNPROCSoft": "127808", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127808", "LimitSIGPENDINGSoft": "127808", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "18968", "MemoryAccounting": "yes", "MemoryCurrent": "1736704", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": 2020-02-05T16:56:01.240 INFO:teuthology.task.ansible.out:"infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-02-05 16:50:35 UTC", "StateChangeTimestampMonotonic": "517085197", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-02-05 16:50:35 UTC", "WatchdogTimestampMonotonic": "517063416", "WatchdogUSec": "0"}} 2020-02-05T16:56:01.274 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:01 +0000 (0:00:00.528) 0:11:48.603 **** 2020-02-05T16:56:01.299 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-02-05T16:56:01.301 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:01.340 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:01 +0000 (0:00:00.066) 0:11:48.670 **** 2020-02-05T16:56:01.370 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:01.406 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:01 +0000 (0:00:00.065) 0:11:48.735 **** 2020-02-05T16:56:01.442 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:01.481 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:01 +0000 (0:00:00.074) 0:11:48.810 **** 2020-02-05T16:56:01.822 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-02-05T16:56:01.823 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-02-05T16:56:01.869 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:01 +0000 (0:00:00.388) 0:11:49.199 **** 2020-02-05T16:56:02.838 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-02-05T16:56:02.839 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1205, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-02-05T16:56:02.883 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:02 +0000 (0:00:01.013) 0:11:50.213 **** 2020-02-05T16:56:04.483 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-02-05T16:56:04.543 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:04 +0000 (0:00:01.654) 0:11:51.867 **** 2020-02-05T16:56:04.575 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-02-05T16:56:04.577 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:04.625 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:04 +0000 (0:00:00.082) 0:11:51.949 **** 2020-02-05T16:56:04.647 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-02-05T16:56:04.649 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-02-05T16:56:04.688 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:04 +0000 (0:00:00.068) 0:11:52.017 **** 2020-02-05T16:56:05.019 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-02-05T16:56:05.026 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-02-05T16:56:05.061 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:05 +0000 (0:00:00.373) 0:11:52.391 **** 2020-02-05T16:56:05.088 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-02-05T16:56:05.090 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:05.156 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:05 +0000 (0:00:00.095) 0:11:52.486 **** 2020-02-05T16:56:05.193 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-02-05T16:56:05.194 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-02-05T16:56:05.241 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:05 +0000 (0:00:00.084) 0:11:52.570 **** 2020-02-05T16:56:05.269 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:05.306 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:05 +0000 (0:00:00.065) 0:11:52.636 **** 2020-02-05T16:56:05.644 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-02-05T16:56:05.645 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-02-05T16:56:05.683 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:05 +0000 (0:00:00.376) 0:11:53.013 **** 2020-02-05T16:56:06.017 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-02-05T16:56:06.018 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-02-05T16:56:06.061 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.378) 0:11:53.391 **** 2020-02-05T16:56:06.088 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-02-05T16:56:06.090 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.130 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.068) 0:11:53.459 **** 2020-02-05T16:56:06.171 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-02-05T16:56:06.173 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.184 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.185 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.228 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.097) 0:11:53.556 **** 2020-02-05T16:56:06.274 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi008.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.286 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi008.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.292 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.331 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.104) 0:11:53.661 **** 2020-02-05T16:56:06.370 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.414 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.082) 0:11:53.743 **** 2020-02-05T16:56:06.440 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-02-05T16:56:06.442 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.485 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.064) 0:11:53.808 **** 2020-02-05T16:56:06.506 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.551 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.071) 0:11:53.879 **** 2020-02-05T16:56:06.578 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-02-05T16:56:06.586 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.630 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.080) 0:11:53.960 **** 2020-02-05T16:56:06.656 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-02-05T16:56:06.658 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.706 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.075) 0:11:54.036 **** 2020-02-05T16:56:06.733 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-02-05T16:56:06.734 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.773 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.067) 0:11:54.103 **** 2020-02-05T16:56:06.810 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.845 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.071) 0:11:54.174 **** 2020-02-05T16:56:06.872 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-02-05T16:56:06.874 INFO:teuthology.task.ansible.out:skipping: [smithi008.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-02-05T16:56:06.923 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:06 +0000 (0:00:00.076) 0:11:54.251 **** 2020-02-05T16:56:07.292 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-02-05T16:56:07.293 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.016608", "end": "2020-02-05 16:56:07.241225", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-02-05 16:56:07.224617", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:56:07.618 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.013098", "end": "2020-02-05 16:56:07.575650", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-02-05 16:56:07.562552", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:56:07.975 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.013502", "end": "2020-02-05 16:56:07.909016", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-02-05 16:56:07.895514", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-02-05T16:56:08.314 INFO:teuthology.task.ansible.out:ok: [smithi008.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.013401", "end": "2020-02-05 16:56:08.264687", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-02-05 16:56:08.251286", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-02-05T16:56:08.357 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:08 +0000 (0:00:01.435) 0:11:55.686 **** 2020-02-05T16:56:08.383 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-02-05T16:56:08.429 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:08 +0000 (0:00:00.067) 0:11:55.754 **** 2020-02-05T16:56:08.766 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-02-05T16:56:08.767 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-02-05T16:56:08.813 INFO:teuthology.task.ansible.out:Wednesday 05 February 2020 16:56:08 +0000 (0:00:00.388) 0:11:56.143 **** 2020-02-05T16:56:09.373 INFO:teuthology.task.ansible.out: RUNNING HANDLER [common : restart nagios-nrpe-server] ************************** 2020-02-05T16:56:09.375 INFO:teuthology.task.ansible.out:changed: [smithi008.front.sepia.ceph.com] => {"changed": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-02-05 16:47:23 UTC", "ActiveEnterTimestampMonotonic": "325367312", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target time-sync.target nss-lookup.target basic.target var-run.mount systemd-journald.socket network.target systemd-tmpfiles-setup.service sysinit.target tmp.mount -.mount system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-02-05 16:47:23 UTC", "AssertTimestampMonotonic": "325365803", "Before": "shutdown.target multi-user.target xdm.service plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-02-05 16:47:23 UTC", "ConditionTimestampMonotonic": "325365803", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDepe 2020-02-05T16:56:09.377 INFO:teuthology.task.ansible.out:ndencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "11666", "ExecMainStartTimestamp": "Wed 2020-02-05 16:47:23 UTC", "ExecMainStartTimestampMonotonic": "325367262", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "985", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-02-05 16:47:23 UTC", "InactiveExitTimestampMonotonic": "325367312", "InvocationID": "32950c4dbcb44e0fa3083b4dc1095065", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft 2020-02-05T16:56:09.378 INFO:teuthology.task.ansible.out:": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127808", "LimitNPROCSoft": "127808", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127808", "LimitSIGPENDINGSoft": "127808", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "11666", "MemoryAccounting": "yes", "MemoryCurrent": "745472", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/run/nrpe /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "075 2020-02-05T16:56:09.380 INFO:teuthology.task.ansible.out:5", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-02-05 16:47:23 UTC", "StateChangeTimestampMonotonic": "325367312", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "989", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-02-05 16:47:23 UTC", "WatchdogTimestampMonotonic": "325367311", "WatchdogUSec": "0"}} 2020-02-05T16:56:09.448 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-02-05T16:56:09.450 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-02-05T16:56:09.452 INFO:teuthology.task.ansible.out: 2020-02-05T16:56:09.454 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi008.front.sepia.ceph.com : ok=130 changed=60 unreachable=0 failed=0 skipped=152 rescued=0 ignored=4 smithi164.front.sepia.ceph.com : ok=130 changed=60 unreachable=0 failed=0 skipped=152 rescued=0 ignored=4 Wednesday 05 February 2020 16:56:09 +0000 (0:00:00.640) 0:11:56.783 **** =============================================================================== 2020-02-05T16:56:09.455 INFO:teuthology.task.ansible.out:testnode : Base group workaround for BZ1782899 ------------------------ 164.15s testnode : Install packages ------------------------------------------- 136.97s common : nrpe - Ensure SELinux policy is up to date -------------------- 56.75s users : Update authorized_keys using the keys repo --------------------- 35.33s users : Create all admin users with sudo access. ----------------------- 34.21s common : Install Kerberos Packages (RedHat) ---------------------------- 13.62s common : nrpe - Load SELinux policy package ---------------------------- 10.94s common : Enable necessary rhsm repos ------------------------------------ 9.65s users : Remove revoked users -------------------------------------------- 9.30s common : nrpe - Install SELinux dependencies ---------------------------- 9.01s common : Install nrpe package and dependencies (RHEL/CentOS) ------------ 8.85s common : Get list of enabled RHSM repos --------------------------------- 8.27s testnode : Ensure ceph dependency packages are not present. ------------- 6.94s common : Register with subscription-manager. ---------------------------- 6.60s testnode : Blow away lingering OSD data and FSIDs ----------------------- 4.72s testnode : Ensure ceph packages are not present. ------------------------ 4.35s common : Disable all rhsm repos ----------------------------------------- 3.70s testnode : Create logical volume(s) ------------------------------------- 2.95s Gathering Facts --------------------------------------------------------- 2.88s common : Make sure smartmontools is installed --------------------------- 2.23s 2020-02-05T16:56:09.752 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi008.front.sepia.ceph.com'), Remote(name='ubuntu@smithi164.front.sepia.ceph.com')] 2020-02-05T16:56:09.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:56:09.755 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi008.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:56:09.826 INFO:teuthology.orchestra.run.smithi008:> true 2020-02-05T16:56:09.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-02-05T16:56:09.983 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi164.front.sepia.ceph.com', 'timeout': 60} 2020-02-05T16:56:10.044 INFO:teuthology.orchestra.run.smithi164:> true 2020-02-05T16:56:10.199 INFO:teuthology.run_tasks:Running task clock... 2020-02-05T16:56:10.207 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-02-05T16:56:10.208 INFO:teuthology.orchestra.run.smithi008:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-02-05T16:56:10.318 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-02-05T16:56:10.375 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-02-05T16:56:10.477 INFO:teuthology.orchestra.run.smithi008.stderr:sudo: ntpd: command not found 2020-02-05T16:56:10.525 INFO:teuthology.orchestra.run.smithi008.stdout:506 Cannot talk to daemon 2020-02-05T16:56:10.579 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-02-05T16:56:10.634 INFO:teuthology.orchestra.run.smithi008.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-02-05T16:56:10.755 INFO:teuthology.orchestra.run.smithi008.stderr:bash: ntpq: command not found 2020-02-05T16:56:10.762 INFO:teuthology.orchestra.run.smithi008.stdout:210 Number of sources = 4 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:=============================================================================== 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:10.763 INFO:teuthology.orchestra.run.smithi008.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:10.764 INFO:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-02-05T16:56:10.871 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-02-05T16:56:10.927 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-02-05T16:56:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2020-02-05T16:56:11.075 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2020-02-05T16:56:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-02-05T16:56:11.184 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-02-05T16:56:11.295 INFO:teuthology.orchestra.run.smithi164.stderr:bash: ntpq: command not found 2020-02-05T16:56:11.299 INFO:teuthology.orchestra.run.smithi164.stdout:210 Number of sources = 4 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:11.300 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-02-05T16:56:11.301 INFO:teuthology.run_tasks:Running task install... 2020-02-05T16:56:11.313 DEBUG:teuthology.task.install:project ceph 2020-02-05T16:56:11.313 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'}} 2020-02-05T16:56:11.313 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'} 2020-02-05T16:56:11.313 INFO:teuthology.task.install:Using flavor: basic 2020-02-05T16:56:11.320 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-02-05T16:56:11.320 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2020-02-05T16:56:11.320 INFO:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2020-02-05T16:56:11.456 INFO:teuthology.orchestra.run.smithi008.stdout:RedHatEnterprise 2020-02-05T16:56:11.459 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:56:11.459 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-02-05T16:56:11.460 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel']}, 'flavor': 'basic', 'extras': None} 2020-02-05T16:56:11.460 INFO:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2020-02-05T16:56:11.594 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2020-02-05T16:56:11.595 INFO:teuthology.orchestra.run.smithi008.stdout:RedHatEnterprise 2020-02-05T16:56:11.597 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:56:11.597 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel']}, 'flavor': 'basic', 'extras': None} 2020-02-05T16:56:11.598 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-02-05T16:56:11.599 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:56:11.599 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:56:11.731 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2020-02-05T16:56:11.734 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:56:11.735 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:56:11.797 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-async-dirops/ecafd7283ef390be73a605986afe2a0f325a3255/centos/8/flavors/default/ 2020-02-05T16:56:11.797 INFO:teuthology.task.install.rpm:Package version is 15.0.0-9851.gecafd72 2020-02-05T16:56:11.902 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-async-dirops/ecafd7283ef390be73a605986afe2a0f325a3255/centos/8/flavors/default/ 2020-02-05T16:56:11.903 INFO:teuthology.task.install.rpm:Package version is 15.0.0-9851.gecafd72 2020-02-05T16:56:11.925 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-02-05T16:56:12.027 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-02-05T16:56:12.036 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel on remote rpm x86_64 2020-02-05T16:56:12.036 INFO:teuthology.orchestra.run.smithi008:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecafd7283ef390be73a605986afe2a0f325a3255/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-02-05T16:56:12.127 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel on remote rpm x86_64 2020-02-05T16:56:12.128 INFO:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/ecafd7283ef390be73a605986afe2a0f325a3255/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-02-05T16:56:12.140 INFO:teuthology.orchestra.run.smithi008:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-02-05T16:56:12.224 INFO:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-02-05T16:56:12.290 INFO:teuthology.orchestra.run.smithi008:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-02-05T16:56:12.353 INFO:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-02-05T16:56:12.392 INFO:teuthology.orchestra.run.smithi008.stdout:check_obsoletes = 1 2020-02-05T16:56:12.397 INFO:teuthology.orchestra.run.smithi008:> sudo yum clean all 2020-02-05T16:56:12.442 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2020-02-05T16:56:12.447 INFO:teuthology.orchestra.run.smithi164:> sudo yum clean all 2020-02-05T16:56:12.907 INFO:teuthology.orchestra.run.smithi008.stdout:Updating Subscription Management repositories. 2020-02-05T16:56:12.940 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2020-02-05T16:56:13.530 INFO:teuthology.orchestra.run.smithi164.stdout:262 files removed 2020-02-05T16:56:13.561 INFO:teuthology.orchestra.run.smithi008.stdout:262 files removed 2020-02-05T16:56:13.588 INFO:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel 2020-02-05T16:56:13.620 INFO:teuthology.orchestra.run.smithi008:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel 2020-02-05T16:56:14.052 INFO:teuthology.orchestra.run.smithi164.stdout:Updating Subscription Management repositories. 2020-02-05T16:56:14.110 INFO:teuthology.orchestra.run.smithi008.stdout:Updating Subscription Management repositories. 2020-02-05T16:56:14.921 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 283 kB/s | 69 kB 00:00 2020-02-05T16:56:14.992 INFO:teuthology.orchestra.run.smithi008.stdout:ceph packages for x86_64 287 kB/s | 69 kB 00:00 2020-02-05T16:56:15.160 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 60 kB/s | 12 kB 00:00 2020-02-05T16:56:15.233 INFO:teuthology.orchestra.run.smithi008.stdout:ceph noarch packages 60 kB/s | 12 kB 00:00 2020-02-05T16:56:15.391 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 6.8 kB/s | 1.4 kB 00:00 2020-02-05T16:56:15.468 INFO:teuthology.orchestra.run.smithi008.stdout:ceph source packages 6.7 kB/s | 1.4 kB 00:00 2020-02-05T16:56:15.868 INFO:teuthology.orchestra.run.smithi164.stdout:Copr repo for ceph-el8 owned by ktdreyer 84 kB/s | 38 kB 00:00 2020-02-05T16:56:15.869 INFO:teuthology.orchestra.run.smithi008.stdout:Copr repo for ceph-el8 owned by ktdreyer 101 kB/s | 38 kB 00:00 2020-02-05T16:56:17.323 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 5.5 MB 00:01 2020-02-05T16:56:17.370 INFO:teuthology.orchestra.run.smithi008.stdout:Extra Packages for Enterprise Linux 3.8 MB/s | 5.5 MB 00:01 2020-02-05T16:56:18.503 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 5.4 MB/s | 33 kB 00:00 2020-02-05T16:56:18.596 INFO:teuthology.orchestra.run.smithi008.stdout:lab-extras 5.5 MB/s | 33 kB 00:00 2020-02-05T16:56:18.767 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 64 MB/s | 14 MB 00:00 2020-02-05T16:56:18.896 INFO:teuthology.orchestra.run.smithi008.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 54 MB/s | 14 MB 00:00 2020-02-05T16:56:21.136 INFO:teuthology.orchestra.run.smithi164.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 65 MB/s | 14 MB 00:00 2020-02-05T16:56:21.191 INFO:teuthology.orchestra.run.smithi008.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 14 MB 00:00 2020-02-05T16:56:24.380 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-02-05T16:56:24.381 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-02-05T16:56:24.386 INFO:teuthology.orchestra.run.smithi164.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-02-05T16:56:24.387 INFO:teuthology.orchestra.run.smithi164.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-02-05T16:56:24.388 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-02-05T16:56:24.389 INFO:teuthology.orchestra.run.smithi164.stdout:Package xfsprogs-5.0.0-1.el8.x86_64 is already installed. 2020-02-05T16:56:24.591 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2020-02-05T16:56:24.603 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2020-02-05T16:56:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: Package ArchVersion Repository Size 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_642:15.0.0-9851.gecafd72.el8 ceph 6.4 k 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_642:15.0.0-9851.gecafd72.el8 ceph 5.5 M 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_642:15.0.0-9851.gecafd72.el8 ceph 580 k 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_642:15.0.0-9851.gecafd72.el8 ceph 202 k 2020-02-05T16:56:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_642:15.0.0-9851.gecafd72.el8 ceph 1.6 M 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_642:15.0.0-9851.gecafd72.el8 ceph 6.4 M 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_642:15.0.0-9851.gecafd72.el8 ceph 40 M 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm x86_642:15.0.0-9851.gecafd72.el8 ceph 29 k 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_642:15.0.0-9851.gecafd72.el8 ceph 21 k 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_642:15.0.0-9851.gecafd72.el8 ceph 514 k 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_642:15.0.0-9851.gecafd72.el8 ceph 141 k 2020-02-05T16:56:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_642:15.0.0-9851.gecafd72.el8 ceph 149 k 2020-02-05T16:56:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_642:15.0.0-9851.gecafd72.el8 ceph 352 k 2020-02-05T16:56:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_642:15.0.0-9851.gecafd72.el8 ceph 309 k 2020-02-05T16:56:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_642:15.0.0-9851.gecafd72.el8 ceph 113 k 2020-02-05T16:56:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_642:15.0.0-9851.gecafd72.el8 ceph 72 k 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_642:15.0.0-9851.gecafd72.el8 ceph 2.6 M 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_642:15.0.0-9851.gecafd72.el8 ceph 141 k 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 27 k 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 3.8 M 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 8.6 M 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 16 k 2020-02-05T16:56:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: flex x86_642.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: bison x86_643.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: libtool x86_642.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel x86_641:1.1.1c-2.el8 rhel-8-for-x86_64-baseos-rpms 2.3 M 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 54 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel x86_645.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 69 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel x86_640.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2020-02-05T16:56:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 95 k 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_642:15.0.0-9851.gecafd72.el8 ceph 3.6 M 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_642:15.0.0-9851.gecafd72.el8 ceph 1.9 M 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 212 k 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 321 k 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2020-02-05T16:56:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_642:15.0.0-9851.gecafd72.el8 ceph 20 M 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_642:15.0.0-9851.gecafd72.el8 ceph 2.0 M 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_642:15.0.0-9851.gecafd72.el8 ceph 4.4 M 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_642:15.0.0-9851.gecafd72.el8 ceph 17 M 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_642:15.0.0-9851.gecafd72.el8 ceph 25 k 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_642:15.0.0-9851.gecafd72.el8 ceph 379 k 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_642:15.0.0-9851.gecafd72.el8 ceph 5.7 M 2020-02-05T16:56:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_642:15.0.0-9851.gecafd72.el8 ceph 40 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_642:15.0.0-9851.gecafd72.el8 ceph 46 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 22 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-02-05T16:56:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-02-05T16:56:24.612 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_642.7-6.el8 epel 315 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_642.6.2-3.el8 epel 59 k 2020-02-05T16:56:24.613 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_641.3.1-3.el8 epel 75 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch3.1.1-4.el8 epel 33 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch2018.10.15-7.el8 epel 16 k 2020-02-05T16:56:24.614 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch1:1.1.1-10.el8 epel 81 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch10.0.1-1.el8 epel 1.1 M 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch3.4.2-12.el8 epel 73 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch0.56.0-5.el8 epel 61 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: leveldb x86_641.21-2.el8 lab-extras 179 k 2020-02-05T16:56:24.615 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_642.8.1-11.el8 lab-extras 257 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_641.6.1-13.el8 lab-extras 69 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: automake noarch1.16.1-6.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webencodings noarch0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: python3-lxml x86_644.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: jq x86_641.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2020-02-05T16:56:24.616 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_641.7.3.2-6.el8 rhel-8-for-x86_64-appstream-rpms 298 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma x86_646.8.2-1.el8 rhel-8-for-x86_64-appstream-rpms 188 k 2020-02-05T16:56:24.617 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_640.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_640.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_641:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_641.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2020-02-05T16:56:24.618 INFO:teuthology.orchestra.run.smithi164.stdout: libverto-devel x86_640.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-utf32 x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel x86_641.2.11-10.el8 rhel-8-for-x86_64-baseos-rpms 56 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_642.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: libxslt x86_641.1.32-3.el8 rhel-8-for-x86_64-baseos-rpms 249 k 2020-02-05T16:56:24.619 INFO:teuthology.orchestra.run.smithi164.stdout: keyutils-libs-devel x86_641.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt noarch1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-devel x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_640.9.0-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-common x86_643.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-utf16 x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: libsepol-devel x86_642.9-1.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2020-02-05T16:56:24.620 INFO:teuthology.orchestra.run.smithi164.stdout: libselinux-devel x86_642.9-2.1.el8 rhel-8-for-x86_64-baseos-rpms 199 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_648.3.1-4.5.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: krb5-devel x86_641.17-9.el8 rhel-8-for-x86_64-baseos-rpms 549 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_648.3.1-4.5.el8 rhel-8-for-x86_64-baseos-rpms 638 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: libcom_err-devel x86_641.44.6-3.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-02-05T16:56:24.621 INFO:teuthology.orchestra.run.smithi164.stdout: python3-html5lib noarch1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout:Install 115 Packages 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 4 Packages 2020-02-05T16:56:24.622 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:56:24.623 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 170 M 2020-02-05T16:56:24.623 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2020-02-05T16:56:24.716 INFO:teuthology.orchestra.run.smithi164.stdout:(1/119): ceph-15.0.0-9851.gecafd72.el8.x86_64.r 67 kB/s | 6.4 kB 00:00 2020-02-05T16:56:24.846 INFO:teuthology.orchestra.run.smithi164.stdout:(2/119): ceph-fuse-15.0.0-9851.gecafd72.el8.x86 4.4 MB/s | 580 kB 00:00 2020-02-05T16:56:24.904 INFO:teuthology.orchestra.run.smithi164.stdout:(3/119): ceph-immutable-object-cache-15.0.0-985 3.8 MB/s | 202 kB 00:00 2020-02-05T16:56:24.920 INFO:teuthology.orchestra.run.smithi008.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2020-02-05T16:56:24.920 INFO:teuthology.orchestra.run.smithi008.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2020-02-05T16:56:24.926 INFO:teuthology.orchestra.run.smithi008.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2020-02-05T16:56:24.927 INFO:teuthology.orchestra.run.smithi008.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2020-02-05T16:56:24.928 INFO:teuthology.orchestra.run.smithi008.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2020-02-05T16:56:24.929 INFO:teuthology.orchestra.run.smithi008.stdout:Package xfsprogs-5.0.0-1.el8.x86_64 is already installed. 2020-02-05T16:56:25.142 INFO:teuthology.orchestra.run.smithi008.stdout:Dependencies resolved. 2020-02-05T16:56:25.156 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2020-02-05T16:56:25.156 INFO:teuthology.orchestra.run.smithi008.stdout: Package ArchVersion Repository Size 2020-02-05T16:56:25.156 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2020-02-05T16:56:25.156 INFO:teuthology.orchestra.run.smithi008.stdout:Installing: 2020-02-05T16:56:25.156 INFO:teuthology.orchestra.run.smithi008.stdout: ceph x86_642:15.0.0-9851.gecafd72.el8 ceph 6.4 k 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base x86_642:15.0.0-9851.gecafd72.el8 ceph 5.5 M 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse x86_642:15.0.0-9851.gecafd72.el8 ceph 580 k 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache x86_642:15.0.0-9851.gecafd72.el8 ceph 202 k 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr x86_642:15.0.0-9851.gecafd72.el8 ceph 1.6 M 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw x86_642:15.0.0-9851.gecafd72.el8 ceph 6.4 M 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test x86_642:15.0.0-9851.gecafd72.el8 ceph 40 M 2020-02-05T16:56:25.157 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm x86_642:15.0.0-9851.gecafd72.el8 ceph 29 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel x86_642:15.0.0-9851.gecafd72.el8 ceph 21 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2 x86_642:15.0.0-9851.gecafd72.el8 ceph 514 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel x86_642:15.0.0-9851.gecafd72.el8 ceph 141 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs x86_642:15.0.0-9851.gecafd72.el8 ceph 149 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados x86_642:15.0.0-9851.gecafd72.el8 ceph 352 k 2020-02-05T16:56:25.158 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd x86_642:15.0.0-9851.gecafd72.el8 ceph 309 k 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw x86_642:15.0.0-9851.gecafd72.el8 ceph 113 k 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse x86_642:15.0.0-9851.gecafd72.el8 ceph 72 k 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror x86_642:15.0.0-9851.gecafd72.el8 ceph 2.6 M 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd x86_642:15.0.0-9851.gecafd72.el8 ceph 141 k 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 27 k 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 3.8 M 2020-02-05T16:56:25.159 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 8.6 M 2020-02-05T16:56:25.160 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 16 k 2020-02-05T16:56:25.160 INFO:teuthology.orchestra.run.smithi008.stdout: openssl-devel x86_641:1.1.1c-2.el8 rhel-8-for-x86_64-baseos-rpms 2.3 M 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libelf-devel x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 54 k 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: xfsprogs-devel x86_645.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 69 k 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: libaio-devel x86_640.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: libuuid-devel x86_642.32.1-17.el8 rhel-8-for-x86_64-baseos-rpms 95 k 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: flex x86_642.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2020-02-05T16:56:25.165 INFO:teuthology.orchestra.run.smithi008.stdout: bison x86_643.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2020-02-05T16:56:25.166 INFO:teuthology.orchestra.run.smithi008.stdout: libtool x86_642.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2020-02-05T16:56:25.166 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrading: 2020-02-05T16:56:25.166 INFO:teuthology.orchestra.run.smithi008.stdout: librados2 x86_642:15.0.0-9851.gecafd72.el8 ceph 3.6 M 2020-02-05T16:56:25.166 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1 x86_642:15.0.0-9851.gecafd72.el8 ceph 1.9 M 2020-02-05T16:56:25.167 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libelf x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 212 k 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libs x86_640.176-5.el8 rhel-8-for-x86_64-baseos-rpms 321 k 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout:Installing dependencies: 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common x86_642:15.0.0-9851.gecafd72.el8 ceph 20 M 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds x86_642:15.0.0-9851.gecafd72.el8 ceph 2.0 M 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon x86_642:15.0.0-9851.gecafd72.el8 ceph 4.4 M 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd x86_642:15.0.0-9851.gecafd72.el8 ceph 17 M 2020-02-05T16:56:25.168 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux x86_642:15.0.0-9851.gecafd72.el8 ceph 25 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1 x86_642:15.0.0-9851.gecafd72.el8 ceph 379 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2 x86_642:15.0.0-9851.gecafd72.el8 ceph 5.7 M 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse x86_642:15.0.0-9851.gecafd72.el8 ceph 40 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common x86_642:15.0.0-9851.gecafd72.el8 ceph 46 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards noarch2:15.0.0-9851.gecafd72.el8 ceph-noarch 22 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-02-05T16:56:25.169 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-02-05T16:56:25.170 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-02-05T16:56:25.171 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-02-05T16:56:25.173 INFO:teuthology.orchestra.run.smithi008.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-02-05T16:56:25.173 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-02-05T16:56:25.174 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs x86_642.7-6.el8 epel 315 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: liboath x86_642.6.2-3.el8 epel 59 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind x86_641.3.1-3.el8 epel 75 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools noarch3.1.1-4.el8 epel 33 k 2020-02-05T16:56:25.175 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi noarch2018.10.15-7.el8 epel 16 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth noarch1:1.1.1-10.el8 epel 81 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes noarch10.0.1-1.el8 epel 1.1 M 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa noarch3.4.2-12.el8 epel 73 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client noarch0.56.0-5.el8 epel 61 k 2020-02-05T16:56:25.176 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-02-05T16:56:25.177 INFO:teuthology.orchestra.run.smithi008.stdout: leveldb x86_641.21-2.el8 lab-extras 179 k 2020-02-05T16:56:25.177 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust x86_642.8.1-11.el8 lab-extras 257 k 2020-02-05T16:56:25.177 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet x86_641.6.1-13.el8 lab-extras 69 k 2020-02-05T16:56:25.177 INFO:teuthology.orchestra.run.smithi008.stdout: libverto-devel x86_640.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2020-02-05T16:56:25.177 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-utf32 x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2020-02-05T16:56:25.178 INFO:teuthology.orchestra.run.smithi008.stdout: zlib-devel x86_641.2.11-10.el8 rhel-8-for-x86_64-baseos-rpms 56 k 2020-02-05T16:56:25.178 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib noarch1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-02-05T16:56:25.178 INFO:teuthology.orchestra.run.smithi008.stdout: fuse x86_642.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2020-02-05T16:56:25.178 INFO:teuthology.orchestra.run.smithi008.stdout: libxslt x86_641.1.32-3.el8 rhel-8-for-x86_64-baseos-rpms 249 k 2020-02-05T16:56:25.178 INFO:teuthology.orchestra.run.smithi008.stdout: keyutils-libs-devel x86_641.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jwt noarch1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-devel x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq x86_640.9.0-1.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib noarch2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: fuse-common x86_643.2.1-12.el8 rhel-8-for-x86_64-baseos-rpms 21 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-utf16 x86_6410.32-1.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2020-02-05T16:56:25.179 INFO:teuthology.orchestra.run.smithi008.stdout: libsepol-devel x86_642.9-1.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: libselinux-devel x86_642.9-2.1.el8 rhel-8-for-x86_64-baseos-rpms 199 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath x86_648.3.1-4.5.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: krb5-devel x86_641.17-9.el8 rhel-8-for-x86_64-baseos-rpms 549 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran x86_648.3.1-4.5.el8 rhel-8-for-x86_64-baseos-rpms 638 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: libcom_err-devel x86_641.44.6-3.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2020-02-05T16:56:25.180 INFO:teuthology.orchestra.run.smithi008.stdout: automake noarch1.16.1-6.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webencodings noarch0.5.1-6.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: python3-lxml x86_644.2.3-1.el8 rhel-8-for-x86_64-appstream-rpms 1.5 M 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: jq x86_641.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug noarch0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako noarch1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2020-02-05T16:56:25.181 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1 noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz noarch2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules noarch0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable noarch0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: socat x86_641.7.3.2-6.el8 rhel-8-for-x86_64-appstream-rpms 298 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-threads x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.4 M 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma x86_646.8.2-1.el8 rhel-8-for-x86_64-appstream-rpms 188 k 2020-02-05T16:56:25.182 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe x86_640.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka x86_640.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: openblas x86_640.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 4.3 M 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy x86_641:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy x86_641.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout:Installing weak dependencies: 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-02-05T16:56:25.183 INFO:teuthology.orchestra.run.smithi008.stdout: python3-html5lib noarch1:0.999999999-6.el8 rhel-8-for-x86_64-appstream-rpms 214 k 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction Summary 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout:================================================================================ 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout:Install 115 Packages 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout:Upgrade 4 Packages 2020-02-05T16:56:25.184 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:56:25.185 INFO:teuthology.orchestra.run.smithi164.stdout:(4/119): ceph-mds-15.0.0-9851.gecafd72.el8.x86_ 7.1 MB/s | 2.0 MB 00:00 2020-02-05T16:56:25.185 INFO:teuthology.orchestra.run.smithi008.stdout:Total download size: 170 M 2020-02-05T16:56:25.185 INFO:teuthology.orchestra.run.smithi008.stdout:Downloading Packages: 2020-02-05T16:56:25.250 INFO:teuthology.orchestra.run.smithi164.stdout:(5/119): ceph-base-15.0.0-9851.gecafd72.el8.x86 8.7 MB/s | 5.5 MB 00:00 2020-02-05T16:56:25.425 INFO:teuthology.orchestra.run.smithi008.stdout:(1/119): ceph-15.0.0-9851.gecafd72.el8.x86_64.r 26 kB/s | 6.4 kB 00:00 2020-02-05T16:56:25.433 INFO:teuthology.orchestra.run.smithi164.stdout:(6/119): ceph-mgr-15.0.0-9851.gecafd72.el8.x86_ 6.3 MB/s | 1.6 MB 00:00 2020-02-05T16:56:25.701 INFO:teuthology.orchestra.run.smithi164.stdout:(7/119): ceph-mon-15.0.0-9851.gecafd72.el8.x86_ 9.7 MB/s | 4.4 MB 00:00 2020-02-05T16:56:26.025 INFO:teuthology.orchestra.run.smithi008.stdout:(2/119): ceph-fuse-15.0.0-9851.gecafd72.el8.x86 967 kB/s | 580 kB 00:00 2020-02-05T16:56:26.181 INFO:teuthology.orchestra.run.smithi008.stdout:(3/119): ceph-immutable-object-cache-15.0.0-985 1.3 MB/s | 202 kB 00:00 2020-02-05T16:56:26.476 INFO:teuthology.orchestra.run.smithi164.stdout:(8/119): ceph-radosgw-15.0.0-9851.gecafd72.el8. 8.3 MB/s | 6.4 MB 00:00 2020-02-05T16:56:26.526 INFO:teuthology.orchestra.run.smithi164.stdout:(9/119): ceph-selinux-15.0.0-9851.gecafd72.el8. 497 kB/s | 25 kB 00:00 2020-02-05T16:56:26.797 INFO:teuthology.orchestra.run.smithi164.stdout:(10/119): ceph-common-15.0.0-9851.gecafd72.el8. 9.2 MB/s | 20 MB 00:02 2020-02-05T16:56:26.846 INFO:teuthology.orchestra.run.smithi164.stdout:(11/119): cephadm-15.0.0-9851.gecafd72.el8.x86_ 589 kB/s | 29 kB 00:00 2020-02-05T16:56:26.893 INFO:teuthology.orchestra.run.smithi164.stdout:(12/119): libcephfs-devel-15.0.0-9851.gecafd72. 459 kB/s | 21 kB 00:00 2020-02-05T16:56:26.962 INFO:teuthology.orchestra.run.smithi164.stdout:(13/119): libcephfs2-15.0.0-9851.gecafd72.el8.x 7.4 MB/s | 514 kB 00:00 2020-02-05T16:56:27.015 INFO:teuthology.orchestra.run.smithi164.stdout:(14/119): librados-devel-15.0.0-9851.gecafd72.e 2.6 MB/s | 141 kB 00:00 2020-02-05T16:56:27.080 INFO:teuthology.orchestra.run.smithi164.stdout:(15/119): libradosstriper1-15.0.0-9851.gecafd72 5.7 MB/s | 379 kB 00:00 2020-02-05T16:56:27.398 INFO:teuthology.orchestra.run.smithi008.stdout:(4/119): ceph-mds-15.0.0-9851.gecafd72.el8.x86_ 1.7 MB/s | 2.0 MB 00:01 2020-02-05T16:56:27.831 INFO:teuthology.orchestra.run.smithi008.stdout:(5/119): ceph-base-15.0.0-9851.gecafd72.el8.x86 2.1 MB/s | 5.5 MB 00:02 2020-02-05T16:56:27.834 INFO:teuthology.orchestra.run.smithi164.stdout:(16/119): ceph-osd-15.0.0-9851.gecafd72.el8.x86 6.9 MB/s | 17 MB 00:02 2020-02-05T16:56:27.858 INFO:teuthology.orchestra.run.smithi164.stdout:(17/119): librgw2-15.0.0-9851.gecafd72.el8.x86_ 7.3 MB/s | 5.7 MB 00:00 2020-02-05T16:56:27.876 INFO:teuthology.orchestra.run.smithi164.stdout:(18/119): python3-ceph-argparse-15.0.0-9851.gec 947 kB/s | 40 kB 00:00 2020-02-05T16:56:27.902 INFO:teuthology.orchestra.run.smithi164.stdout:(19/119): python3-ceph-common-15.0.0-9851.gecaf 1.1 MB/s | 46 kB 00:00 2020-02-05T16:56:27.926 INFO:teuthology.orchestra.run.smithi164.stdout:(20/119): python3-cephfs-15.0.0-9851.gecafd72.e 2.9 MB/s | 149 kB 00:00 2020-02-05T16:56:27.961 INFO:teuthology.orchestra.run.smithi164.stdout:(21/119): python3-rados-15.0.0-9851.gecafd72.el 5.8 MB/s | 352 kB 00:00 2020-02-05T16:56:27.992 INFO:teuthology.orchestra.run.smithi164.stdout:(22/119): python3-rbd-15.0.0-9851.gecafd72.el8. 4.6 MB/s | 309 kB 00:00 2020-02-05T16:56:28.018 INFO:teuthology.orchestra.run.smithi164.stdout:(23/119): python3-rgw-15.0.0-9851.gecafd72.el8. 1.9 MB/s | 113 kB 00:00 2020-02-05T16:56:28.040 INFO:teuthology.orchestra.run.smithi164.stdout:(24/119): rbd-fuse-15.0.0-9851.gecafd72.el8.x86 1.5 MB/s | 72 kB 00:00 2020-02-05T16:56:28.108 INFO:teuthology.orchestra.run.smithi164.stdout:(25/119): rbd-nbd-15.0.0-9851.gecafd72.el8.x86_ 2.1 MB/s | 141 kB 00:00 2020-02-05T16:56:28.171 INFO:teuthology.orchestra.run.smithi164.stdout:(26/119): ceph-grafana-dashboards-15.0.0-9851.g 356 kB/s | 22 kB 00:00 2020-02-05T16:56:28.234 INFO:teuthology.orchestra.run.smithi008.stdout:(6/119): ceph-mgr-15.0.0-9851.gecafd72.el8.x86_ 1.9 MB/s | 1.6 MB 00:00 2020-02-05T16:56:28.235 INFO:teuthology.orchestra.run.smithi164.stdout:(27/119): ceph-mgr-cephadm-15.0.0-9851.gecafd72 423 kB/s | 27 kB 00:00 2020-02-05T16:56:28.357 INFO:teuthology.orchestra.run.smithi164.stdout:(28/119): rbd-mirror-15.0.0-9851.gecafd72.el8.x 7.7 MB/s | 2.6 MB 00:00 2020-02-05T16:56:28.573 INFO:teuthology.orchestra.run.smithi008.stdout:(7/119): ceph-mon-15.0.0-9851.gecafd72.el8.x86_ 5.9 MB/s | 4.4 MB 00:00 2020-02-05T16:56:28.886 INFO:teuthology.orchestra.run.smithi164.stdout:(29/119): ceph-mgr-dashboard-15.0.0-9851.gecafd 5.8 MB/s | 3.8 MB 00:00 2020-02-05T16:56:28.944 INFO:teuthology.orchestra.run.smithi164.stdout:(30/119): ceph-mgr-rook-15.0.0-9851.gecafd72.el 287 kB/s | 16 kB 00:00 2020-02-05T16:56:29.002 INFO:teuthology.orchestra.run.smithi164.stdout:(31/119): python3-cheroot-8.2.1-1.el8.noarch.rp 2.5 MB/s | 145 kB 00:00 2020-02-05T16:56:29.025 INFO:teuthology.orchestra.run.smithi164.stdout:(32/119): python3-cherrypy-18.4.0-1.el8.noarch. 17 MB/s | 385 kB 00:00 2020-02-05T16:56:29.041 INFO:teuthology.orchestra.run.smithi164.stdout:(33/119): python3-execnet-1.7.1-1.el8.2.noarch. 22 MB/s | 356 kB 00:00 2020-02-05T16:56:29.050 INFO:teuthology.orchestra.run.smithi164.stdout:(34/119): python3-jaraco-6.2-6.el8.noarch.rpm 1.4 MB/s | 11 kB 00:00 2020-02-05T16:56:29.059 INFO:teuthology.orchestra.run.smithi164.stdout:(35/119): python3-jaraco-functools-2.0-4.el8.no 2.1 MB/s | 18 kB 00:00 2020-02-05T16:56:29.069 INFO:teuthology.orchestra.run.smithi164.stdout:(36/119): python3-logutils-0.3.5-11.el8.noarch. 4.9 MB/s | 49 kB 00:00 2020-02-05T16:56:29.080 INFO:teuthology.orchestra.run.smithi164.stdout:(37/119): python3-more-itertools-7.2.0-3.el8.no 5.7 MB/s | 59 kB 00:00 2020-02-05T16:56:29.095 INFO:teuthology.orchestra.run.smithi164.stdout:(38/119): python3-pecan-1.3.2-9.el8.noarch.rpm 18 MB/s | 283 kB 00:00 2020-02-05T16:56:29.105 INFO:teuthology.orchestra.run.smithi164.stdout:(39/119): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-02-05T16:56:29.114 INFO:teuthology.orchestra.run.smithi164.stdout:(40/119): python3-remoto-1.1.4-4.el8.noarch.rpm 5.2 MB/s | 48 kB 00:00 2020-02-05T16:56:29.123 INFO:teuthology.orchestra.run.smithi164.stdout:(41/119): python3-repoze-lru-0.7-6.el8.noarch.r 3.6 MB/s | 34 kB 00:00 2020-02-05T16:56:29.136 INFO:teuthology.orchestra.run.smithi164.stdout:(42/119): python3-routes-2.4.1-12.el8.noarch.rp 16 MB/s | 197 kB 00:00 2020-02-05T16:56:29.145 INFO:teuthology.orchestra.run.smithi164.stdout:(43/119): python3-simplegeneric-0.8.1-17.el8.no 2.1 MB/s | 19 kB 00:00 2020-02-05T16:56:29.158 INFO:teuthology.orchestra.run.smithi164.stdout:(44/119): python3-singledispatch-3.4.0.3-18.el8 2.0 MB/s | 24 kB 00:00 2020-02-05T16:56:29.167 INFO:teuthology.orchestra.run.smithi164.stdout:(45/119): python3-tempora-1.14.1-5.el8.noarch.r 3.3 MB/s | 30 kB 00:00 2020-02-05T16:56:29.176 INFO:teuthology.orchestra.run.smithi164.stdout:(46/119): python3-trustme-0.5.2-4.el8.noarch.rp 3.1 MB/s | 27 kB 00:00 2020-02-05T16:56:29.185 INFO:teuthology.orchestra.run.smithi164.stdout:(47/119): python3-zc-lockfile-2.0-2.el8.noarch. 2.6 MB/s | 23 kB 00:00 2020-02-05T16:56:29.316 INFO:teuthology.orchestra.run.smithi164.stdout:(48/119): gperftools-libs-2.7-6.el8.x86_64.rpm 2.4 MB/s | 315 kB 00:00 2020-02-05T16:56:29.352 INFO:teuthology.orchestra.run.smithi164.stdout:(49/119): ceph-mgr-diskprediction-local-15.0.0- 8.6 MB/s | 8.6 MB 00:00 2020-02-05T16:56:29.354 INFO:teuthology.orchestra.run.smithi164.stdout:(50/119): liboath-2.6.2-3.el8.x86_64.rpm 1.6 MB/s | 59 kB 00:00 2020-02-05T16:56:29.397 INFO:teuthology.orchestra.run.smithi164.stdout:(51/119): python3-apipkg-1.5-6.el8.noarch.rpm 490 kB/s | 21 kB 00:00 2020-02-05T16:56:29.419 INFO:teuthology.orchestra.run.smithi164.stdout:(52/119): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.0 MB/s | 44 kB 00:00 2020-02-05T16:56:29.430 INFO:teuthology.orchestra.run.smithi164.stdout:(53/119): libunwind-1.3.1-3.el8.x86_64.rpm 971 kB/s | 75 kB 00:00 2020-02-05T16:56:29.450 INFO:teuthology.orchestra.run.smithi164.stdout:(54/119): python3-cachetools-3.1.1-4.el8.noarch 1.6 MB/s | 33 kB 00:00 2020-02-05T16:56:29.459 INFO:teuthology.orchestra.run.smithi164.stdout:(55/119): python3-beautifulsoup4-4.6.3-2.el8.1. 4.5 MB/s | 185 kB 00:00 2020-02-05T16:56:29.470 INFO:teuthology.orchestra.run.smithi164.stdout:(56/119): python3-certifi-2018.10.15-7.el8.noar 838 kB/s | 16 kB 00:00 2020-02-05T16:56:29.480 INFO:teuthology.orchestra.run.smithi164.stdout:(57/119): python3-cssselect-0.9.2-10.el8.noarch 2.0 MB/s | 40 kB 00:00 2020-02-05T16:56:29.497 INFO:teuthology.orchestra.run.smithi164.stdout:(58/119): python3-google-auth-1.1.1-10.el8.noar 3.0 MB/s | 81 kB 00:00 2020-02-05T16:56:29.502 INFO:teuthology.orchestra.run.smithi008.stdout:(8/119): ceph-radosgw-15.0.0-9851.gecafd72.el8. 6.9 MB/s | 6.4 MB 00:00 2020-02-05T16:56:29.518 INFO:teuthology.orchestra.run.smithi164.stdout:(59/119): python3-rsa-3.4.2-12.el8.noarch.rpm 3.4 MB/s | 73 kB 00:00 2020-02-05T16:56:29.527 INFO:teuthology.orchestra.run.smithi008.stdout:(9/119): ceph-selinux-15.0.0-9851.gecafd72.el8. 1.0 MB/s | 25 kB 00:00 2020-02-05T16:56:29.560 INFO:teuthology.orchestra.run.smithi164.stdout:(60/119): python3-waitress-1.2.1-2.el8.1.noarch 5.7 MB/s | 241 kB 00:00 2020-02-05T16:56:29.603 INFO:teuthology.orchestra.run.smithi164.stdout:(61/119): python3-webob-1.8.5-1.el8.1.noarch.rp 5.8 MB/s | 251 kB 00:00 2020-02-05T16:56:29.624 INFO:teuthology.orchestra.run.smithi164.stdout:(62/119): python3-websocket-client-0.56.0-5.el8 2.9 MB/s | 61 kB 00:00 2020-02-05T16:56:29.638 INFO:teuthology.orchestra.run.smithi164.stdout:(63/119): python3-kubernetes-10.0.1-1.el8.noarc 7.1 MB/s | 1.1 MB 00:00 2020-02-05T16:56:29.644 INFO:teuthology.orchestra.run.smithi164.stdout:(64/119): leveldb-1.21-2.el8.x86_64.rpm 28 MB/s | 179 kB 00:00 2020-02-05T16:56:29.649 INFO:teuthology.orchestra.run.smithi164.stdout:(65/119): lttng-ust-2.8.1-11.el8.x86_64.rpm 59 MB/s | 257 kB 00:00 2020-02-05T16:56:29.652 INFO:teuthology.orchestra.run.smithi164.stdout:(66/119): xmlstarlet-1.6.1-13.el8.x86_64.rpm 34 MB/s | 69 kB 00:00 2020-02-05T16:56:29.652 INFO:teuthology.orchestra.run.smithi164.stdout:(67/119): python3-webtest-2.0.33-1.el8.noarch.r 2.9 MB/s | 85 kB 00:00 2020-02-05T16:56:29.701 INFO:teuthology.orchestra.run.smithi164.stdout:(68/119): automake-1.16.1-6.el8.noarch.rpm 14 MB/s | 713 kB 00:00 2020-02-05T16:56:29.702 INFO:teuthology.orchestra.run.smithi164.stdout:(69/119): python3-webencodings-0.5.1-6.el8.noar 551 kB/s | 27 kB 00:00 2020-02-05T16:56:29.743 INFO:teuthology.orchestra.run.smithi164.stdout:(70/119): jq-1.5-12.el8.x86_64.rpm 4.0 MB/s | 161 kB 00:00 2020-02-05T16:56:29.775 INFO:teuthology.orchestra.run.smithi164.stdout:(71/119): python3-lxml-4.2.3-1.el8.x86_64.rpm 21 MB/s | 1.5 MB 00:00 2020-02-05T16:56:29.785 INFO:teuthology.orchestra.run.smithi164.stdout:(72/119): python3-werkzeug-0.12.2-4.el8.noarch. 10 MB/s | 457 kB 00:00 2020-02-05T16:56:29.809 INFO:teuthology.orchestra.run.smithi164.stdout:(73/119): python3-mako-1.0.6-13.el8.noarch.rpm 4.5 MB/s | 157 kB 00:00 2020-02-05T16:56:29.820 INFO:teuthology.orchestra.run.smithi164.stdout:(74/119): python3-html5lib-0.999999999-6.el8.no 6.1 MB/s | 214 kB 00:00 2020-02-05T16:56:29.834 INFO:teuthology.orchestra.run.smithi164.stdout:(75/119): python3-pyasn1-0.3.7-6.el8.noarch.rpm 5.1 MB/s | 126 kB 00:00 2020-02-05T16:56:29.847 INFO:teuthology.orchestra.run.smithi164.stdout:(76/119): python3-pytz-2017.2-9.el8.noarch.rpm 2.4 MB/s | 54 kB 00:00 2020-02-05T16:56:29.861 INFO:teuthology.orchestra.run.smithi164.stdout:(77/119): python3-pyasn1-modules-0.3.7-6.el8.no 3.9 MB/s | 110 kB 00:00 2020-02-05T16:56:29.875 INFO:teuthology.orchestra.run.smithi164.stdout:(78/119): python3-prettytable-0.7.2-14.el8.noar 1.8 MB/s | 44 kB 00:00 2020-02-05T16:56:29.891 INFO:teuthology.orchestra.run.smithi164.stdout:(79/119): socat-1.7.3.2-6.el8.x86_64.rpm 9.8 MB/s | 298 kB 00:00 2020-02-05T16:56:29.921 INFO:teuthology.orchestra.run.smithi164.stdout:(80/119): oniguruma-6.8.2-1.el8.x86_64.rpm 7.3 MB/s | 188 kB 00:00 2020-02-05T16:56:29.943 INFO:teuthology.orchestra.run.smithi164.stdout:(81/119): python3-markupsafe-0.23-19.el8.x86_64 1.7 MB/s | 39 kB 00:00 2020-02-05T16:56:29.979 INFO:teuthology.orchestra.run.smithi164.stdout:(82/119): openblas-threads-0.3.3-2.el8.x86_64.r 42 MB/s | 4.4 MB 00:00 2020-02-05T16:56:29.984 INFO:teuthology.orchestra.run.smithi164.stdout:(83/119): flex-2.6.1-9.el8.x86_64.rpm 7.8 MB/s | 320 kB 00:00 2020-02-05T16:56:30.012 INFO:teuthology.orchestra.run.smithi164.stdout:(84/119): librdkafka-0.11.4-1.el8.x86_64.rpm 11 MB/s | 353 kB 00:00 2020-02-05T16:56:30.035 INFO:teuthology.orchestra.run.smithi164.stdout:(85/119): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2020-02-05T16:56:30.070 INFO:teuthology.orchestra.run.smithi164.stdout:(86/119): libtool-2.4.6-25.el8.x86_64.rpm 20 MB/s | 709 kB 00:00 2020-02-05T16:56:30.111 INFO:teuthology.orchestra.run.smithi164.stdout:(87/119): openblas-0.3.3-2.el8.x86_64.rpm 44 MB/s | 4.3 MB 00:00 2020-02-05T16:56:30.136 INFO:teuthology.orchestra.run.smithi164.stdout:(88/119): python3-numpy-f2py-1.14.3-9.el8.x86_6 8.7 MB/s | 225 kB 00:00 2020-02-05T16:56:30.174 INFO:teuthology.orchestra.run.smithi164.stdout:(89/119): python3-numpy-1.14.3-9.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2020-02-05T16:56:30.198 INFO:teuthology.orchestra.run.smithi164.stdout:(90/119): libverto-devel-0.3.0-5.el8.x86_64.rpm 757 kB/s | 18 kB 00:00 2020-02-05T16:56:30.228 INFO:teuthology.orchestra.run.smithi164.stdout:(91/119): pcre2-utf32-10.32-1.el8.x86_64.rpm 7.8 MB/s | 220 kB 00:00 2020-02-05T16:56:30.255 INFO:teuthology.orchestra.run.smithi164.stdout:(92/119): zlib-devel-1.2.11-10.el8.x86_64.rpm 2.0 MB/s | 56 kB 00:00 2020-02-05T16:56:30.283 INFO:teuthology.orchestra.run.smithi164.stdout:(93/119): python3-requests-oauthlib-1.0.0-1.el8 1.5 MB/s | 43 kB 00:00 2020-02-05T16:56:30.311 INFO:teuthology.orchestra.run.smithi164.stdout:(94/119): fuse-2.9.7-12.el8.x86_64.rpm 3.0 MB/s | 83 kB 00:00 2020-02-05T16:56:30.339 INFO:teuthology.orchestra.run.smithi164.stdout:(95/119): libxslt-1.1.32-3.el8.x86_64.rpm 8.7 MB/s | 249 kB 00:00 2020-02-05T16:56:30.362 INFO:teuthology.orchestra.run.smithi164.stdout:(96/119): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.1 MB/s | 48 kB 00:00 2020-02-05T16:56:30.416 INFO:teuthology.orchestra.run.smithi164.stdout:(97/119): python3-scipy-1.0.0-20.module+el8.1.0 52 MB/s | 15 MB 00:00 2020-02-05T16:56:30.441 INFO:teuthology.orchestra.run.smithi164.stdout:(98/119): python3-jwt-1.6.1-2.el8.noarch.rpm 542 kB/s | 43 kB 00:00 2020-02-05T16:56:30.468 INFO:teuthology.orchestra.run.smithi164.stdout:(99/119): pcre2-devel-10.32-1.el8.x86_64.rpm 13 MB/s | 605 kB 00:00 2020-02-05T16:56:30.470 INFO:teuthology.orchestra.run.smithi164.stdout:(100/119): librabbitmq-0.9.0-1.el8.x86_64.rpm 1.6 MB/s | 47 kB 00:00 2020-02-05T16:56:30.496 INFO:teuthology.orchestra.run.smithi164.stdout:(101/119): python3-oauthlib-2.1.0-1.el8.noarch. 5.3 MB/s | 155 kB 00:00 2020-02-05T16:56:30.504 INFO:teuthology.orchestra.run.smithi164.stdout:(102/119): fuse-common-3.2.1-12.el8.x86_64.rpm 631 kB/s | 21 kB 00:00 2020-02-05T16:56:30.521 INFO:teuthology.orchestra.run.smithi164.stdout:(103/119): pcre2-utf16-10.32-1.el8.x86_64.rpm 9.3 MB/s | 229 kB 00:00 2020-02-05T16:56:30.534 INFO:teuthology.orchestra.run.smithi164.stdout:(104/119): libsepol-devel-2.9-1.el8.x86_64.rpm 2.9 MB/s | 86 kB 00:00 2020-02-05T16:56:30.555 INFO:teuthology.orchestra.run.smithi164.stdout:(105/119): libselinux-devel-2.9-2.1.el8.x86_64. 6.6 MB/s | 199 kB 00:00 2020-02-05T16:56:30.564 INFO:teuthology.orchestra.run.smithi164.stdout:(106/119): libquadmath-8.3.1-4.5.el8.x86_64.rpm 5.5 MB/s | 169 kB 00:00 2020-02-05T16:56:30.588 INFO:teuthology.orchestra.run.smithi164.stdout:(107/119): krb5-devel-1.17-9.el8.x86_64.rpm 16 MB/s | 549 kB 00:00 2020-02-05T16:56:30.610 INFO:teuthology.orchestra.run.smithi164.stdout:(108/119): elfutils-libelf-devel-0.176-5.el8.x8 2.3 MB/s | 54 kB 00:00 2020-02-05T16:56:30.635 INFO:teuthology.orchestra.run.smithi164.stdout:(109/119): openssl-devel-1.1.1c-2.el8.x86_64.rp 32 MB/s | 2.3 MB 00:00 2020-02-05T16:56:30.637 INFO:teuthology.orchestra.run.smithi164.stdout:(110/119): xfsprogs-devel-5.0.0-1.el8.x86_64.rp 2.6 MB/s | 69 kB 00:00 2020-02-05T16:56:30.666 INFO:teuthology.orchestra.run.smithi164.stdout:(111/119): libaio-devel-0.3.112-1.el8.x86_64.rp 657 kB/s | 19 kB 00:00 2020-02-05T16:56:30.668 INFO:teuthology.orchestra.run.smithi164.stdout:(112/119): libgfortran-8.3.1-4.5.el8.x86_64.rpm 19 MB/s | 638 kB 00:00 2020-02-05T16:56:30.803 INFO:teuthology.orchestra.run.smithi164.stdout:(113/119): ceph-test-15.0.0-9851.gecafd72.el8.x 9.3 MB/s | 40 MB 00:04 2020-02-05T16:56:30.820 INFO:teuthology.orchestra.run.smithi164.stdout:(114/119): libcom_err-devel-1.44.6-3.el8.x86_64 245 kB/s | 38 kB 00:00 2020-02-05T16:56:30.825 INFO:teuthology.orchestra.run.smithi164.stdout:(115/119): libuuid-devel-2.32.1-17.el8.x86_64.r 607 kB/s | 95 kB 00:00 2020-02-05T16:56:30.856 INFO:teuthology.orchestra.run.smithi164.stdout:(116/119): elfutils-libelf-0.176-5.el8.x86_64.r 6.7 MB/s | 212 kB 00:00 2020-02-05T16:56:30.883 INFO:teuthology.orchestra.run.smithi164.stdout:(117/119): elfutils-libs-0.176-5.el8.x86_64.rpm 12 MB/s | 321 kB 00:00 2020-02-05T16:56:31.008 INFO:teuthology.orchestra.run.smithi164.stdout:(118/119): librbd1-15.0.0-9851.gecafd72.el8.x86 10 MB/s | 1.9 MB 00:00 2020-02-05T16:56:31.362 INFO:teuthology.orchestra.run.smithi164.stdout:(119/119): librados2-15.0.0-9851.gecafd72.el8.x 6.5 MB/s | 3.6 MB 00:00 2020-02-05T16:56:31.365 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2020-02-05T16:56:31.365 INFO:teuthology.orchestra.run.smithi164.stdout:Total 25 MB/s | 170 MB 00:06 2020-02-05T16:56:31.459 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2020-02-05T16:56:31.601 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2020-02-05T16:56:31.601 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2020-02-05T16:56:32.183 INFO:teuthology.orchestra.run.smithi008.stdout:(10/119): ceph-common-15.0.0-9851.gecafd72.el8. 2.9 MB/s | 20 MB 00:07 2020-02-05T16:56:32.216 INFO:teuthology.orchestra.run.smithi008.stdout:(11/119): cephadm-15.0.0-9851.gecafd72.el8.x86_ 851 kB/s | 29 kB 00:00 2020-02-05T16:56:32.240 INFO:teuthology.orchestra.run.smithi008.stdout:(12/119): libcephfs-devel-15.0.0-9851.gecafd72. 915 kB/s | 21 kB 00:00 2020-02-05T16:56:32.327 INFO:teuthology.orchestra.run.smithi008.stdout:(13/119): libcephfs2-15.0.0-9851.gecafd72.el8.x 5.8 MB/s | 514 kB 00:00 2020-02-05T16:56:32.355 INFO:teuthology.orchestra.run.smithi008.stdout:(14/119): librados-devel-15.0.0-9851.gecafd72.e 4.8 MB/s | 141 kB 00:00 2020-02-05T16:56:32.512 INFO:teuthology.orchestra.run.smithi008.stdout:(15/119): ceph-test-15.0.0-9851.gecafd72.el8.x8 13 MB/s | 40 MB 00:02 2020-02-05T16:56:32.518 INFO:teuthology.orchestra.run.smithi008.stdout:(16/119): libradosstriper1-15.0.0-9851.gecafd72 2.3 MB/s | 379 kB 00:00 2020-02-05T16:56:32.548 INFO:teuthology.orchestra.run.smithi008.stdout:(17/119): python3-ceph-argparse-15.0.0-9851.gec 1.3 MB/s | 40 kB 00:00 2020-02-05T16:56:32.563 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2020-02-05T16:56:32.563 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2020-02-05T16:56:32.572 INFO:teuthology.orchestra.run.smithi008.stdout:(18/119): python3-ceph-common-15.0.0-9851.gecaf 1.9 MB/s | 46 kB 00:00 2020-02-05T16:56:32.601 INFO:teuthology.orchestra.run.smithi008.stdout:(19/119): python3-cephfs-15.0.0-9851.gecafd72.e 5.1 MB/s | 149 kB 00:00 2020-02-05T16:56:32.660 INFO:teuthology.orchestra.run.smithi008.stdout:(20/119): python3-rados-15.0.0-9851.gecafd72.el 5.8 MB/s | 352 kB 00:00 2020-02-05T16:56:32.719 INFO:teuthology.orchestra.run.smithi008.stdout:(21/119): python3-rbd-15.0.0-9851.gecafd72.el8. 5.2 MB/s | 309 kB 00:00 2020-02-05T16:56:32.747 INFO:teuthology.orchestra.run.smithi008.stdout:(22/119): python3-rgw-15.0.0-9851.gecafd72.el8. 4.0 MB/s | 113 kB 00:00 2020-02-05T16:56:32.774 INFO:teuthology.orchestra.run.smithi008.stdout:(23/119): rbd-fuse-15.0.0-9851.gecafd72.el8.x86 2.7 MB/s | 72 kB 00:00 2020-02-05T16:56:32.847 INFO:teuthology.orchestra.run.smithi008.stdout:(24/119): librgw2-15.0.0-9851.gecafd72.el8.x86_ 17 MB/s | 5.7 MB 00:00 2020-02-05T16:56:32.876 INFO:teuthology.orchestra.run.smithi008.stdout:(25/119): rbd-nbd-15.0.0-9851.gecafd72.el8.x86_ 4.8 MB/s | 141 kB 00:00 2020-02-05T16:56:32.898 INFO:teuthology.orchestra.run.smithi008.stdout:(26/119): ceph-grafana-dashboards-15.0.0-9851.g 1.0 MB/s | 22 kB 00:00 2020-02-05T16:56:32.920 INFO:teuthology.orchestra.run.smithi008.stdout:(27/119): ceph-mgr-cephadm-15.0.0-9851.gecafd72 1.2 MB/s | 27 kB 00:00 2020-02-05T16:56:33.162 INFO:teuthology.orchestra.run.smithi008.stdout:(28/119): ceph-osd-15.0.0-9851.gecafd72.el8.x86 3.4 MB/s | 17 MB 00:04 2020-02-05T16:56:33.186 INFO:teuthology.orchestra.run.smithi008.stdout:(29/119): ceph-mgr-dashboard-15.0.0-9851.gecafd 14 MB/s | 3.8 MB 00:00 2020-02-05T16:56:33.196 INFO:teuthology.orchestra.run.smithi008.stdout:(30/119): rbd-mirror-15.0.0-9851.gecafd72.el8.x 6.1 MB/s | 2.6 MB 00:00 2020-02-05T16:56:33.213 INFO:teuthology.orchestra.run.smithi008.stdout:(31/119): ceph-mgr-rook-15.0.0-9851.gecafd72.el 629 kB/s | 16 kB 00:00 2020-02-05T16:56:33.262 INFO:teuthology.orchestra.run.smithi008.stdout:(32/119): python3-cheroot-8.2.1-1.el8.noarch.rp 2.2 MB/s | 145 kB 00:00 2020-02-05T16:56:33.285 INFO:teuthology.orchestra.run.smithi008.stdout:(33/119): python3-execnet-1.7.1-1.el8.2.noarch. 16 MB/s | 356 kB 00:00 2020-02-05T16:56:33.287 INFO:teuthology.orchestra.run.smithi008.stdout:(34/119): python3-cherrypy-18.4.0-1.el8.noarch. 5.1 MB/s | 385 kB 00:00 2020-02-05T16:56:33.293 INFO:teuthology.orchestra.run.smithi008.stdout:(35/119): python3-jaraco-6.2-6.el8.noarch.rpm 1.3 MB/s | 11 kB 00:00 2020-02-05T16:56:33.296 INFO:teuthology.orchestra.run.smithi008.stdout:(36/119): python3-jaraco-functools-2.0-4.el8.no 2.0 MB/s | 18 kB 00:00 2020-02-05T16:56:33.303 INFO:teuthology.orchestra.run.smithi008.stdout:(37/119): python3-logutils-0.3.5-11.el8.noarch. 5.1 MB/s | 49 kB 00:00 2020-02-05T16:56:33.306 INFO:teuthology.orchestra.run.smithi008.stdout:(38/119): python3-more-itertools-7.2.0-3.el8.no 5.8 MB/s | 59 kB 00:00 2020-02-05T16:56:33.315 INFO:teuthology.orchestra.run.smithi008.stdout:(39/119): python3-portend-2.6-1.el8.noarch.rpm 1.8 MB/s | 16 kB 00:00 2020-02-05T16:56:33.324 INFO:teuthology.orchestra.run.smithi008.stdout:(40/119): python3-pecan-1.3.2-9.el8.noarch.rpm 14 MB/s | 283 kB 00:00 2020-02-05T16:56:33.330 INFO:teuthology.orchestra.run.smithi008.stdout:(41/119): python3-remoto-1.1.4-4.el8.noarch.rpm 4.9 MB/s | 48 kB 00:00 2020-02-05T16:56:33.333 INFO:teuthology.orchestra.run.smithi008.stdout:(42/119): python3-repoze-lru-0.7-6.el8.noarch.r 3.7 MB/s | 34 kB 00:00 2020-02-05T16:56:33.338 INFO:teuthology.orchestra.run.smithi008.stdout:(43/119): python3-routes-2.4.1-12.el8.noarch.rp 15 MB/s | 197 kB 00:00 2020-02-05T16:56:33.342 INFO:teuthology.orchestra.run.smithi008.stdout:(44/119): python3-simplegeneric-0.8.1-17.el8.no 2.1 MB/s | 19 kB 00:00 2020-02-05T16:56:33.349 INFO:teuthology.orchestra.run.smithi008.stdout:(45/119): python3-singledispatch-3.4.0.3-18.el8 2.7 MB/s | 24 kB 00:00 2020-02-05T16:56:33.352 INFO:teuthology.orchestra.run.smithi008.stdout:(46/119): python3-tempora-1.14.1-5.el8.noarch.r 3.2 MB/s | 30 kB 00:00 2020-02-05T16:56:33.357 INFO:teuthology.orchestra.run.smithi008.stdout:(47/119): python3-trustme-0.5.2-4.el8.noarch.rp 3.0 MB/s | 27 kB 00:00 2020-02-05T16:56:33.361 INFO:teuthology.orchestra.run.smithi008.stdout:(48/119): python3-zc-lockfile-2.0-2.el8.noarch. 2.5 MB/s | 23 kB 00:00 2020-02-05T16:56:33.437 INFO:teuthology.orchestra.run.smithi008.stdout:(49/119): liboath-2.6.2-3.el8.x86_64.rpm 775 kB/s | 59 kB 00:00 2020-02-05T16:56:33.470 INFO:teuthology.orchestra.run.smithi008.stdout:(50/119): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-02-05T16:56:33.476 INFO:teuthology.orchestra.run.smithi008.stdout:(51/119): libunwind-1.3.1-3.el8.x86_64.rpm 1.9 MB/s | 75 kB 00:00 2020-02-05T16:56:33.486 INFO:teuthology.orchestra.run.smithi008.stdout:(52/119): python3-apipkg-1.5-6.el8.noarch.rpm 1.3 MB/s | 21 kB 00:00 2020-02-05T16:56:33.498 INFO:teuthology.orchestra.run.smithi008.stdout:(53/119): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.0 MB/s | 44 kB 00:00 2020-02-05T16:56:33.519 INFO:teuthology.orchestra.run.smithi008.stdout:(54/119): python3-cachetools-3.1.1-4.el8.noarch 1.6 MB/s | 33 kB 00:00 2020-02-05T16:56:33.521 INFO:teuthology.orchestra.run.smithi008.stdout:(55/119): python3-beautifulsoup4-4.6.3-2.el8.1. 5.2 MB/s | 185 kB 00:00 2020-02-05T16:56:33.538 INFO:teuthology.orchestra.run.smithi008.stdout:(56/119): python3-cssselect-0.9.2-10.el8.noarch 2.4 MB/s | 40 kB 00:00 2020-02-05T16:56:33.540 INFO:teuthology.orchestra.run.smithi008.stdout:(57/119): python3-certifi-2018.10.15-7.el8.noar 802 kB/s | 16 kB 00:00 2020-02-05T16:56:33.558 INFO:teuthology.orchestra.run.smithi008.stdout:(58/119): python3-google-auth-1.1.1-10.el8.noar 4.2 MB/s | 81 kB 00:00 2020-02-05T16:56:33.576 INFO:teuthology.orchestra.run.smithi008.stdout:(59/119): python3-rsa-3.4.2-12.el8.noarch.rpm 4.1 MB/s | 73 kB 00:00 2020-02-05T16:56:33.614 INFO:teuthology.orchestra.run.smithi008.stdout:(60/119): python3-waitress-1.2.1-2.el8.1.noarch 7.0 MB/s | 241 kB 00:00 2020-02-05T16:56:33.645 INFO:teuthology.orchestra.run.smithi008.stdout:(61/119): python3-webob-1.8.5-1.el8.1.noarch.rp 7.1 MB/s | 251 kB 00:00 2020-02-05T16:56:33.662 INFO:teuthology.orchestra.run.smithi008.stdout:(62/119): python3-websocket-client-0.56.0-5.el8 3.5 MB/s | 61 kB 00:00 2020-02-05T16:56:33.681 INFO:teuthology.orchestra.run.smithi008.stdout:(63/119): python3-webtest-2.0.33-1.el8.noarch.r 4.5 MB/s | 85 kB 00:00 2020-02-05T16:56:33.690 INFO:teuthology.orchestra.run.smithi008.stdout:(64/119): leveldb-1.21-2.el8.x86_64.rpm 23 MB/s | 179 kB 00:00 2020-02-05T16:56:33.695 INFO:teuthology.orchestra.run.smithi008.stdout:(65/119): lttng-ust-2.8.1-11.el8.x86_64.rpm 53 MB/s | 257 kB 00:00 2020-02-05T16:56:33.697 INFO:teuthology.orchestra.run.smithi008.stdout:(66/119): xmlstarlet-1.6.1-13.el8.x86_64.rpm 37 MB/s | 69 kB 00:00 2020-02-05T16:56:33.716 INFO:teuthology.orchestra.run.smithi008.stdout:(67/119): python3-kubernetes-10.0.1-1.el8.noarc 6.3 MB/s | 1.1 MB 00:00 2020-02-05T16:56:33.752 INFO:teuthology.orchestra.run.smithi008.stdout:(68/119): libverto-devel-0.3.0-5.el8.x86_64.rpm 325 kB/s | 18 kB 00:00 2020-02-05T16:56:33.782 INFO:teuthology.orchestra.run.smithi008.stdout:(69/119): pcre2-utf32-10.32-1.el8.x86_64.rpm 3.7 MB/s | 220 kB 00:00 2020-02-05T16:56:33.787 INFO:teuthology.orchestra.run.smithi008.stdout:(70/119): zlib-devel-1.2.11-10.el8.x86_64.rpm 1.6 MB/s | 56 kB 00:00 2020-02-05T16:56:33.821 INFO:teuthology.orchestra.run.smithi008.stdout:(71/119): python3-requests-oauthlib-1.0.0-1.el8 1.1 MB/s | 43 kB 00:00 2020-02-05T16:56:33.832 INFO:teuthology.orchestra.run.smithi008.stdout:(72/119): fuse-2.9.7-12.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2020-02-05T16:56:33.862 INFO:teuthology.orchestra.run.smithi008.stdout:(73/119): libxslt-1.1.32-3.el8.x86_64.rpm 5.9 MB/s | 249 kB 00:00 2020-02-05T16:56:33.868 INFO:teuthology.orchestra.run.smithi008.stdout:(74/119): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.3 MB/s | 48 kB 00:00 2020-02-05T16:56:33.905 INFO:teuthology.orchestra.run.smithi008.stdout:(75/119): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2020-02-05T16:56:33.931 INFO:teuthology.orchestra.run.smithi008.stdout:(76/119): pcre2-devel-10.32-1.el8.x86_64.rpm 9.5 MB/s | 605 kB 00:00 2020-02-05T16:56:33.941 INFO:teuthology.orchestra.run.smithi008.stdout:(77/119): librabbitmq-0.9.0-1.el8.x86_64.rpm 1.3 MB/s | 47 kB 00:00 2020-02-05T16:56:33.971 INFO:teuthology.orchestra.run.smithi008.stdout:(78/119): python3-oauthlib-2.1.0-1.el8.noarch.r 3.7 MB/s | 155 kB 00:00 2020-02-05T16:56:33.983 INFO:teuthology.orchestra.run.smithi008.stdout:(79/119): fuse-common-3.2.1-12.el8.x86_64.rpm 516 kB/s | 21 kB 00:00 2020-02-05T16:56:34.010 INFO:teuthology.orchestra.run.smithi008.stdout:(80/119): pcre2-utf16-10.32-1.el8.x86_64.rpm 5.8 MB/s | 229 kB 00:00 2020-02-05T16:56:34.019 INFO:teuthology.orchestra.run.smithi008.stdout:(81/119): libsepol-devel-2.9-1.el8.x86_64.rpm 2.3 MB/s | 86 kB 00:00 2020-02-05T16:56:34.075 INFO:teuthology.orchestra.run.smithi008.stdout:(82/119): libselinux-devel-2.9-2.1.el8.x86_64.r 3.0 MB/s | 199 kB 00:00 2020-02-05T16:56:34.078 INFO:teuthology.orchestra.run.smithi008.stdout:(83/119): libquadmath-8.3.1-4.5.el8.x86_64.rpm 3.5 MB/s | 169 kB 00:00 2020-02-05T16:56:34.117 INFO:teuthology.orchestra.run.smithi008.stdout:(84/119): krb5-devel-1.17-9.el8.x86_64.rpm 13 MB/s | 549 kB 00:00 2020-02-05T16:56:34.146 INFO:teuthology.orchestra.run.smithi008.stdout:(85/119): elfutils-libelf-devel-0.176-5.el8.x86 1.8 MB/s | 54 kB 00:00 2020-02-05T16:56:34.175 INFO:teuthology.orchestra.run.smithi008.stdout:(86/119): openssl-devel-1.1.1c-2.el8.x86_64.rpm 24 MB/s | 2.3 MB 00:00 2020-02-05T16:56:34.176 INFO:teuthology.orchestra.run.smithi008.stdout:(87/119): xfsprogs-devel-5.0.0-1.el8.x86_64.rpm 2.2 MB/s | 69 kB 00:00 2020-02-05T16:56:34.239 INFO:teuthology.orchestra.run.smithi008.stdout:(88/119): ceph-mgr-diskprediction-local-15.0.0- 8.0 MB/s | 8.6 MB 00:01 2020-02-05T16:56:34.241 INFO:teuthology.orchestra.run.smithi008.stdout:(89/119): libaio-devel-0.3.112-1.el8.x86_64.rpm 290 kB/s | 19 kB 00:00 2020-02-05T16:56:34.270 INFO:teuthology.orchestra.run.smithi008.stdout:(90/119): libgfortran-8.3.1-4.5.el8.x86_64.rpm 6.5 MB/s | 638 kB 00:00 2020-02-05T16:56:34.278 INFO:teuthology.orchestra.run.smithi008.stdout:(91/119): libuuid-devel-2.32.1-17.el8.x86_64.rp 2.5 MB/s | 95 kB 00:00 2020-02-05T16:56:34.302 INFO:teuthology.orchestra.run.smithi008.stdout:(92/119): libcom_err-devel-1.44.6-3.el8.x86_64. 604 kB/s | 38 kB 00:00 2020-02-05T16:56:34.327 INFO:teuthology.orchestra.run.smithi008.stdout:(93/119): python3-webencodings-0.5.1-6.el8.noar 560 kB/s | 27 kB 00:00 2020-02-05T16:56:34.347 INFO:teuthology.orchestra.run.smithi008.stdout:(94/119): automake-1.16.1-6.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2020-02-05T16:56:34.368 INFO:teuthology.orchestra.run.smithi008.stdout:(95/119): jq-1.5-12.el8.x86_64.rpm 3.8 MB/s | 161 kB 00:00 2020-02-05T16:56:34.403 INFO:teuthology.orchestra.run.smithi008.stdout:(96/119): python3-lxml-4.2.3-1.el8.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2020-02-05T16:56:34.406 INFO:teuthology.orchestra.run.smithi008.stdout:(97/119): python3-werkzeug-0.12.2-4.el8.noarch. 7.6 MB/s | 457 kB 00:00 2020-02-05T16:56:34.414 INFO:teuthology.orchestra.run.smithi008.stdout:(98/119): python3-mako-1.0.6-13.el8.noarch.rpm 3.9 MB/s | 157 kB 00:00 2020-02-05T16:56:34.454 INFO:teuthology.orchestra.run.smithi008.stdout:(99/119): python3-html5lib-0.999999999-6.el8.no 4.1 MB/s | 214 kB 00:00 2020-02-05T16:56:34.456 INFO:teuthology.orchestra.run.smithi008.stdout:(100/119): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.9 MB/s | 126 kB 00:00 2020-02-05T16:56:34.465 INFO:teuthology.orchestra.run.smithi008.stdout:(101/119): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2020-02-05T16:56:34.496 INFO:teuthology.orchestra.run.smithi008.stdout:(102/119): python3-prettytable-0.7.2-14.el8.noa 1.1 MB/s | 44 kB 00:00 2020-02-05T16:56:34.498 INFO:teuthology.orchestra.run.smithi008.stdout:(103/119): python3-pyasn1-modules-0.3.7-6.el8.n 2.5 MB/s | 110 kB 00:00 2020-02-05T16:56:34.530 INFO:teuthology.orchestra.run.smithi008.stdout:(104/119): socat-1.7.3.2-6.el8.x86_64.rpm 4.5 MB/s | 298 kB 00:00 2020-02-05T16:56:34.550 INFO:teuthology.orchestra.run.smithi008.stdout:(105/119): oniguruma-6.8.2-1.el8.x86_64.rpm 3.5 MB/s | 188 kB 00:00 2020-02-05T16:56:34.567 INFO:teuthology.orchestra.run.smithi008.stdout:(106/119): python3-markupsafe-0.23-19.el8.x86_6 1.2 MB/s | 39 kB 00:00 2020-02-05T16:56:34.596 INFO:teuthology.orchestra.run.smithi008.stdout:(107/119): flex-2.6.1-9.el8.x86_64.rpm 8.0 MB/s | 320 kB 00:00 2020-02-05T16:56:34.619 INFO:teuthology.orchestra.run.smithi008.stdout:(108/119): librdkafka-0.11.4-1.el8.x86_64.rpm 6.8 MB/s | 353 kB 00:00 2020-02-05T16:56:34.646 INFO:teuthology.orchestra.run.smithi008.stdout:(109/119): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2020-02-05T16:56:34.687 INFO:teuthology.orchestra.run.smithi008.stdout:(110/119): openblas-threads-0.3.3-2.el8.x86_64. 23 MB/s | 4.4 MB 00:00 2020-02-05T16:56:34.714 INFO:teuthology.orchestra.run.smithi008.stdout:(111/119): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2020-02-05T16:56:34.745 INFO:teuthology.orchestra.run.smithi008.stdout:(112/119): python3-numpy-f2py-1.14.3-9.el8.x86_ 7.2 MB/s | 225 kB 00:00 2020-02-05T16:56:34.791 INFO:teuthology.orchestra.run.smithi008.stdout:(113/119): openblas-0.3.3-2.el8.x86_64.rpm 25 MB/s | 4.3 MB 00:00 2020-02-05T16:56:34.816 INFO:teuthology.orchestra.run.smithi008.stdout:(114/119): python3-numpy-1.14.3-9.el8.x86_64.rp 29 MB/s | 3.7 MB 00:00 2020-02-05T16:56:34.958 INFO:teuthology.orchestra.run.smithi008.stdout:(115/119): python3-scipy-1.0.0-20.module+el8.1. 69 MB/s | 15 MB 00:00 2020-02-05T16:56:34.986 INFO:teuthology.orchestra.run.smithi008.stdout:(116/119): elfutils-libelf-0.176-5.el8.x86_64.r 7.2 MB/s | 212 kB 00:00 2020-02-05T16:56:35.014 INFO:teuthology.orchestra.run.smithi008.stdout:(117/119): elfutils-libs-0.176-5.el8.x86_64.rpm 11 MB/s | 321 kB 00:00 2020-02-05T16:56:35.051 INFO:teuthology.orchestra.run.smithi008.stdout:(118/119): librbd1-15.0.0-9851.gecafd72.el8.x86 8.0 MB/s | 1.9 MB 00:00 2020-02-05T16:56:35.254 INFO:teuthology.orchestra.run.smithi008.stdout:(119/119): librados2-15.0.0-9851.gecafd72.el8.x 7.7 MB/s | 3.6 MB 00:00 2020-02-05T16:56:35.258 INFO:teuthology.orchestra.run.smithi008.stdout:-------------------------------------------------------------------------------- 2020-02-05T16:56:35.258 INFO:teuthology.orchestra.run.smithi008.stdout:Total 17 MB/s | 170 MB 00:10 2020-02-05T16:56:35.356 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction check 2020-02-05T16:56:35.509 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction check succeeded. 2020-02-05T16:56:35.509 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction test 2020-02-05T16:56:36.536 INFO:teuthology.orchestra.run.smithi008.stdout:Transaction test succeeded. 2020-02-05T16:56:36.536 INFO:teuthology.orchestra.run.smithi008.stdout:Running transaction 2020-02-05T16:56:36.988 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2020-02-05T16:56:37.158 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/123 2020-02-05T16:56:37.365 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 2/123 2020-02-05T16:56:37.522 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/123 2020-02-05T16:56:37.663 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/123 2020-02-05T16:56:37.847 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/123 2020-02-05T16:56:38.291 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/123 2020-02-05T16:56:38.465 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:15.0.0-9851.gecafd72.el8.x86_64 5/123 2020-02-05T16:56:38.726 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:15.0.0-9851.gecafd72.el8.x86_64 5/123 2020-02-05T16:56:38.939 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:56:39.062 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:56:39.214 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:56:39.327 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:56:40.034 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 8/123 2020-02-05T16:56:40.172 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:56:40.333 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:56:40.558 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 10/123 2020-02-05T16:56:40.688 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 11/123 2020-02-05T16:56:40.804 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 11/123 2020-02-05T16:56:41.028 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libelf-0.176-5.el8.x86_64 12/123 2020-02-05T16:56:41.183 INFO:teuthology.orchestra.run.smithi008.stdout: Preparing : 1/1 2020-02-05T16:56:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 13/123 2020-02-05T16:56:41.375 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 13/123 2020-02-05T16:56:41.394 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/123 2020-02-05T16:56:41.530 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 14/123 2020-02-05T16:56:41.589 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/123 2020-02-05T16:56:41.683 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/123 2020-02-05T16:56:41.934 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 3/123 2020-02-05T16:56:42.148 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/123 2020-02-05T16:56:42.153 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 14/123 2020-02-05T16:56:42.288 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 15/123 2020-02-05T16:56:42.618 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/123 2020-02-05T16:56:42.839 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librados2-2:15.0.0-9851.gecafd72.el8.x86_64 5/123 2020-02-05T16:56:42.903 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 15/123 2020-02-05T16:56:43.079 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.3-2.el8.x86_64 16/123 2020-02-05T16:56:43.133 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-2:15.0.0-9851.gecafd72.el8.x86_64 5/123 2020-02-05T16:56:43.356 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:56:43.498 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:56:43.706 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:56:43.837 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:56:43.909 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 16/123 2020-02-05T16:56:44.204 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 17/123 2020-02-05T16:56:44.408 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 18/123 2020-02-05T16:56:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 19/123 2020-02-05T16:56:44.619 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 8/123 2020-02-05T16:56:44.818 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : zlib-devel-1.2.11-10.el8.x86_64 20/123 2020-02-05T16:56:44.822 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:56:44.981 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:56:45.095 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 21/123 2020-02-05T16:56:45.260 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 10/123 2020-02-05T16:56:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 22/123 2020-02-05T16:56:45.432 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 11/123 2020-02-05T16:56:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 23/123 2020-02-05T16:56:45.547 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8 24/123 2020-02-05T16:56:45.549 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 11/123 2020-02-05T16:56:45.766 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 25/123 2020-02-05T16:56:45.917 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : elfutils-libelf-0.176-5.el8.x86_64 12/123 2020-02-05T16:56:46.023 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-13.el8.x86_64 26/123 2020-02-05T16:56:46.087 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 13/123 2020-02-05T16:56:46.226 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 27/123 2020-02-05T16:56:46.297 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 14/123 2020-02-05T16:56:46.470 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 14/123 2020-02-05T16:56:46.478 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 28/123 2020-02-05T16:56:46.639 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 15/123 2020-02-05T16:56:46.718 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/123 2020-02-05T16:56:47.282 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 15/123 2020-02-05T16:56:47.456 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 16/123 2020-02-05T16:56:48.091 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 16/123 2020-02-05T16:56:48.313 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openblas-0.3.3-2.el8.x86_64 17/123 2020-02-05T16:56:48.952 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/123 2020-02-05T16:56:49.197 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 17/123 2020-02-05T16:56:49.278 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 30/123 2020-02-05T16:56:49.496 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 31/123 2020-02-05T16:56:49.517 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 18/123 2020-02-05T16:56:49.701 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 19/123 2020-02-05T16:56:49.879 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 20/123 2020-02-05T16:56:50.137 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : zlib-devel-1.2.11-10.el8.x86_64 21/123 2020-02-05T16:56:50.244 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 32/123 2020-02-05T16:56:50.435 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 33/123 2020-02-05T16:56:50.437 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 22/123 2020-02-05T16:56:50.620 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 34/123 2020-02-05T16:56:50.628 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 23/123 2020-02-05T16:56:50.758 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8 24/123 2020-02-05T16:56:50.788 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 35/123 2020-02-05T16:56:50.941 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libuuid-devel-2.32.1-17.el8.x86_64 36/123 2020-02-05T16:56:51.018 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 25/123 2020-02-05T16:56:51.111 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcom_err-devel-1.44.6-3.el8.x86_64 37/123 2020-02-05T16:56:51.326 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 38/123 2020-02-05T16:56:51.331 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : xmlstarlet-1.6.1-13.el8.x86_64 26/123 2020-02-05T16:56:51.486 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : pcre2-utf16-10.32-1.el8.x86_64 39/123 2020-02-05T16:56:51.486 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 40/123 2020-02-05T16:56:51.486 INFO:teuthology.orchestra.run.smithi164.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-02-05T16:56:51.487 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:56:51.746 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 27/123 2020-02-05T16:56:51.813 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.7-12.el8.x86_64 41/123 2020-02-05T16:56:52.018 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 42/123 2020-02-05T16:56:52.031 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 28/123 2020-02-05T16:56:52.238 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/123 2020-02-05T16:56:52.358 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : pcre2-utf32-10.32-1.el8.x86_64 43/123 2020-02-05T16:56:52.641 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : pcre2-devel-10.32-1.el8.x86_64 44/123 2020-02-05T16:56:52.844 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libselinux-devel-2.9-2.1.el8.x86_64 45/123 2020-02-05T16:56:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 46/123 2020-02-05T16:56:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : krb5-devel-1.17-9.el8.x86_64 47/123 2020-02-05T16:56:53.514 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 48/123 2020-02-05T16:56:53.797 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 49/123 2020-02-05T16:56:53.962 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 50/123 2020-02-05T16:56:54.055 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 50/123 2020-02-05T16:56:54.273 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : jq-1.5-12.el8.x86_64 51/123 2020-02-05T16:56:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 52/123 2020-02-05T16:56:54.630 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 29/123 2020-02-05T16:56:54.642 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 53/123 2020-02-05T16:56:54.819 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 54/123 2020-02-05T16:56:54.956 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 30/123 2020-02-05T16:56:55.071 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 55/123 2020-02-05T16:56:55.197 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 31/123 2020-02-05T16:56:55.434 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 32/123 2020-02-05T16:56:55.612 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 56/123 2020-02-05T16:56:55.663 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 33/123 2020-02-05T16:56:55.840 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : automake-1.16.1-6.el8.noarch 57/123 2020-02-05T16:56:55.896 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 34/123 2020-02-05T16:56:56.089 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 35/123 2020-02-05T16:56:56.158 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 58/123 2020-02-05T16:56:56.384 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 36/123 2020-02-05T16:56:56.392 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 59/123 2020-02-05T16:56:56.784 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 60/123 2020-02-05T16:56:57.109 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 37/123 2020-02-05T16:56:57.125 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 61/123 2020-02-05T16:56:57.330 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 38/123 2020-02-05T16:56:57.348 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 62/123 2020-02-05T16:56:57.478 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 38/123 2020-02-05T16:56:57.563 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 63/123 2020-02-05T16:56:57.708 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : jq-1.5-12.el8.x86_64 39/123 2020-02-05T16:56:57.733 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 64/123 2020-02-05T16:56:57.923 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 40/123 2020-02-05T16:56:57.966 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 65/123 2020-02-05T16:56:58.125 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/123 2020-02-05T16:56:58.354 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 42/123 2020-02-05T16:56:58.614 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 43/123 2020-02-05T16:56:58.927 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 44/123 2020-02-05T16:56:59.113 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : automake-1.16.1-6.el8.noarch 45/123 2020-02-05T16:56:59.298 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 66/123 2020-02-05T16:56:59.301 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libuuid-devel-2.32.1-17.el8.x86_64 46/123 2020-02-05T16:56:59.462 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcom_err-devel-1.44.6-3.el8.x86_64 47/123 2020-02-05T16:56:59.563 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-10.0.1-1.el8.noarch 67/123 2020-02-05T16:56:59.646 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libsepol-devel-2.9-1.el8.x86_64 48/123 2020-02-05T16:56:59.810 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 68/123 2020-02-05T16:56:59.855 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : pcre2-utf16-10.32-1.el8.x86_64 49/123 2020-02-05T16:56:59.855 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 50/123 2020-02-05T16:56:59.855 INFO:teuthology.orchestra.run.smithi008.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-02-05T16:56:59.856 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:57:00.094 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 69/123 2020-02-05T16:57:00.274 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/123 2020-02-05T16:57:00.349 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 70/123 2020-02-05T16:57:00.511 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 52/123 2020-02-05T16:57:00.580 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 71/123 2020-02-05T16:57:00.818 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 72/123 2020-02-05T16:57:00.902 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : pcre2-utf32-10.32-1.el8.x86_64 53/123 2020-02-05T16:57:01.046 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 73/123 2020-02-05T16:57:01.253 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : pcre2-devel-10.32-1.el8.x86_64 54/123 2020-02-05T16:57:01.283 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : leveldb-1.21-2.el8.x86_64 74/123 2020-02-05T16:57:01.445 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 75/123 2020-02-05T16:57:01.462 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libselinux-devel-2.9-2.1.el8.x86_64 55/123 2020-02-05T16:57:01.643 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 76/123 2020-02-05T16:57:01.655 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 56/123 2020-02-05T16:57:01.807 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 77/123 2020-02-05T16:57:01.958 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : krb5-devel-1.17-9.el8.x86_64 57/123 2020-02-05T16:57:02.007 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 78/123 2020-02-05T16:57:02.228 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 79/123 2020-02-05T16:57:02.305 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 58/123 2020-02-05T16:57:02.461 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 80/123 2020-02-05T16:57:02.560 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 59/123 2020-02-05T16:57:02.820 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 81/123 2020-02-05T16:57:03.006 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 60/123 2020-02-05T16:57:03.095 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/123 2020-02-05T16:57:03.289 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 83/123 2020-02-05T16:57:03.382 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 61/123 2020-02-05T16:57:03.514 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 84/123 2020-02-05T16:57:03.615 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 62/123 2020-02-05T16:57:03.816 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 85/123 2020-02-05T16:57:03.855 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 63/123 2020-02-05T16:57:04.041 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 86/123 2020-02-05T16:57:04.095 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 64/123 2020-02-05T16:57:04.349 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-2.6-1.el8.noarch 87/123 2020-02-05T16:57:04.369 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 65/123 2020-02-05T16:57:04.618 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 88/123 2020-02-05T16:57:04.812 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.e 89/123 2020-02-05T16:57:05.036 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x 90/123 2020-02-05T16:57:05.782 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 66/123 2020-02-05T16:57:06.090 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-kubernetes-10.0.1-1.el8.noarch 67/123 2020-02-05T16:57:06.345 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 68/123 2020-02-05T16:57:06.643 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 69/123 2020-02-05T16:57:06.948 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 70/123 2020-02-05T16:57:07.198 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 71/123 2020-02-05T16:57:07.473 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 72/123 2020-02-05T16:57:07.705 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 73/123 2020-02-05T16:57:07.947 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : leveldb-1.21-2.el8.x86_64 74/123 2020-02-05T16:57:08.156 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 75/123 2020-02-05T16:57:08.410 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 76/123 2020-02-05T16:57:08.576 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 77/123 2020-02-05T16:57:08.809 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 78/123 2020-02-05T16:57:08.848 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:09.065 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 79/123 2020-02-05T16:57:09.094 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:09.322 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 80/123 2020-02-05T16:57:09.758 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 81/123 2020-02-05T16:57:09.845 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:10.105 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/123 2020-02-05T16:57:10.185 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 92/123 2020-02-05T16:57:10.334 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 92/123 2020-02-05T16:57:10.334 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 83/123 2020-02-05T16:57:10.459 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 93/123 2020-02-05T16:57:10.635 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 84/123 2020-02-05T16:57:10.935 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 85/123 2020-02-05T16:57:11.209 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 86/123 2020-02-05T16:57:11.571 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-portend-2.6-1.el8.noarch 87/123 2020-02-05T16:57:11.879 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 88/123 2020-02-05T16:57:12.121 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.e 89/123 2020-02-05T16:57:12.412 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x 90/123 2020-02-05T16:57:16.687 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:16.928 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:17.731 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 91/123 2020-02-05T16:57:18.604 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 92/123 2020-02-05T16:57:18.747 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 92/123 2020-02-05T16:57:18.868 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 93/123 2020-02-05T16:57:27.785 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 93/123 2020-02-05T16:57:27.785 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2020-02-05T16:57:27.785 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /run 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2020-02-05T16:57:27.786 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2020-02-05T16:57:27.787 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /dev 2020-02-05T16:57:27.787 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:57:28.315 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 94/123 2020-02-05T16:57:28.586 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 94/123 2020-02-05T16:57:28.765 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 95/123 2020-02-05T16:57:29.244 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 95/123 2020-02-05T16:57:29.473 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 96/123 2020-02-05T16:57:31.016 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 96/123 2020-02-05T16:57:31.223 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 97/123 2020-02-05T16:57:31.294 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 97/123 2020-02-05T16:57:33.542 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 98/123 2020-02-05T16:57:33.694 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 98/123 2020-02-05T16:57:33.821 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 99/123 2020-02-05T16:57:33.863 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 99/123 2020-02-05T16:57:34.940 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:15.0.0-9851.gecafd72.el8.x86_64 100/123 2020-02-05T16:57:35.196 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 101/123 2020-02-05T16:57:36.380 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 101/123 2020-02-05T16:57:36.587 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 102/123 2020-02-05T16:57:36.628 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 102/123 2020-02-05T16:57:36.669 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 93/123 2020-02-05T16:57:36.669 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /sys 2020-02-05T16:57:36.669 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /proc 2020-02-05T16:57:36.669 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /dev 2020-02-05T16:57:36.669 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /run 2020-02-05T16:57:36.670 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /mnt 2020-02-05T16:57:36.670 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /var/tmp 2020-02-05T16:57:36.670 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /home 2020-02-05T16:57:36.670 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /tmp 2020-02-05T16:57:36.670 INFO:teuthology.orchestra.run.smithi008.stdout:skipping the directory /dev 2020-02-05T16:57:36.671 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:57:36.763 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 103/123 2020-02-05T16:57:37.225 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 94/123 2020-02-05T16:57:37.426 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 103/123 2020-02-05T16:57:37.495 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 94/123 2020-02-05T16:57:37.631 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 104/123 2020-02-05T16:57:37.659 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 95/123 2020-02-05T16:57:38.144 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 95/123 2020-02-05T16:57:38.254 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 104/123 2020-02-05T16:57:38.317 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 96/123 2020-02-05T16:57:38.497 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 105/123 2020-02-05T16:57:39.960 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 96/123 2020-02-05T16:57:40.183 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 97/123 2020-02-05T16:57:40.250 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 97/123 2020-02-05T16:57:42.317 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 98/123 2020-02-05T16:57:42.476 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 105/123 2020-02-05T16:57:42.496 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 98/123 2020-02-05T16:57:42.631 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 99/123 2020-02-05T16:57:42.674 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 99/123 2020-02-05T16:57:42.858 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 106/123 2020-02-05T16:57:43.595 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libtool-2.4.6-25.el8.x86_64 107/123 2020-02-05T16:57:43.861 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-2:15.0.0-9851.gecafd72.el8.x86_64 100/123 2020-02-05T16:57:44.114 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 101/123 2020-02-05T16:57:44.588 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 107/123 2020-02-05T16:57:44.932 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openssl-devel-1:1.1.1c-2.el8.x86_64 108/123 2020-02-05T16:57:45.152 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 109/123 2020-02-05T16:57:45.319 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xfsprogs-devel-5.0.0-1.el8.x86_64 110/123 2020-02-05T16:57:45.379 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 101/123 2020-02-05T16:57:45.480 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_6 111/123 2020-02-05T16:57:45.573 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 102/123 2020-02-05T16:57:45.620 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 102/123 2020-02-05T16:57:45.646 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : elfutils-libelf-devel-0.176-5.el8.x86_64 112/123 2020-02-05T16:57:45.764 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 103/123 2020-02-05T16:57:45.883 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : elfutils-libs-0.176-5.el8.x86_64 113/123 2020-02-05T16:57:46.111 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 114/123 2020-02-05T16:57:46.290 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 115/123 2020-02-05T16:57:46.454 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:15.0.0-9851.gecafd 116/123 2020-02-05T16:57:46.472 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 103/123 2020-02-05T16:57:46.523 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.0.0-9851.gecafd 116/123 2020-02-05T16:57:46.674 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 104/123 2020-02-05T16:57:46.757 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 117/123 2020-02-05T16:57:46.961 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/123 2020-02-05T16:57:47.055 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/123 2020-02-05T16:57:47.134 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 104/123 2020-02-05T16:57:47.211 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/123 2020-02-05T16:57:47.252 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/123 2020-02-05T16:57:47.307 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 105/123 2020-02-05T16:57:47.482 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libs-0.174-6.el8.x86_64 120/123 2020-02-05T16:57:47.483 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/123 2020-02-05T16:57:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/123 2020-02-05T16:57:47.766 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/123 2020-02-05T16:57:48.002 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/123 2020-02-05T16:57:48.229 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : elfutils-libelf-0.174-6.el8.x86_64 123/123 2020-02-05T16:57:49.065 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: elfutils-libelf-0.174-6.el8.x86_64 123/123 2020-02-05T16:57:49.065 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:15.0.0-9851.gecafd72.el8.x86_64 1/123 2020-02-05T16:57:49.065 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 2/123 2020-02-05T16:57:49.066 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 3/123 2020-02-05T16:57:49.066 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 4/123 2020-02-05T16:57:49.066 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:15.0.0-9851.gecafd 5/123 2020-02-05T16:57:49.066 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 8/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 10/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 11/123 2020-02-05T16:57:49.067 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 12/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 13/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_6 14/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 15/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 16/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 17/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 18/123 2020-02-05T16:57:49.068 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8 19/123 2020-02-05T16:57:49.069 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x 20/123 2020-02-05T16:57:49.069 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 21/123 2020-02-05T16:57:49.070 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 22/123 2020-02-05T16:57:49.070 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 23/123 2020-02-05T16:57:49.071 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 24/123 2020-02-05T16:57:49.071 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 25/123 2020-02-05T16:57:49.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 26/123 2020-02-05T16:57:49.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 27/123 2020-02-05T16:57:49.072 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.e 28/123 2020-02-05T16:57:49.080 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 29/123 2020-02-05T16:57:49.081 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 30/123 2020-02-05T16:57:49.081 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 31/123 2020-02-05T16:57:49.081 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 32/123 2020-02-05T16:57:49.081 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 33/123 2020-02-05T16:57:49.081 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 34/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 35/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 36/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 37/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 38/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 39/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 40/123 2020-02-05T16:57:49.082 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-2.6-1.el8.noarch 41/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 42/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 43/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 44/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 45/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 46/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 47/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 48/123 2020-02-05T16:57:49.083 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 49/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 50/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 51/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 52/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 53/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 54/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 55/123 2020-02-05T16:57:49.084 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 56/123 2020-02-05T16:57:49.085 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 57/123 2020-02-05T16:57:49.085 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 58/123 2020-02-05T16:57:49.085 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 59/123 2020-02-05T16:57:49.085 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-10.0.1-1.el8.noarch 60/123 2020-02-05T16:57:49.085 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 61/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 62/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 63/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 64/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 65/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : leveldb-1.21-2.el8.x86_64 66/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 67/123 2020-02-05T16:57:49.086 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-13.el8.x86_64 68/123 2020-02-05T16:57:49.087 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : automake-1.16.1-6.el8.noarch 69/123 2020-02-05T16:57:49.087 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 70/123 2020-02-05T16:57:49.087 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 71/123 2020-02-05T16:57:49.087 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : jq-1.5-12.el8.x86_64 72/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 73/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 74/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 75/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 76/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 77/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 78/123 2020-02-05T16:57:49.088 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 79/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 80/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 81/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 82/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 83/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flex-2.6.1-9.el8.x86_64 84/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 85/123 2020-02-05T16:57:49.089 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : bison-3.0.4-10.el8.x86_64 86/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 87/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 88/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 89/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 90/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 91/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 92/123 2020-02-05T16:57:49.090 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : pcre2-utf32-10.32-1.el8.x86_64 93/123 2020-02-05T16:57:49.091 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : zlib-devel-1.2.11-10.el8.x86_64 94/123 2020-02-05T16:57:49.091 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 95/123 2020-02-05T16:57:49.091 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 96/123 2020-02-05T16:57:49.094 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 97/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 98/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 99/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : pcre2-devel-10.32-1.el8.x86_64 100/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 101/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 102/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 103/123 2020-02-05T16:57:49.095 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : pcre2-utf16-10.32-1.el8.x86_64 104/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 105/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libselinux-devel-2.9-2.1.el8.x86_64 106/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 107/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : krb5-devel-1.17-9.el8.x86_64 108/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openssl-devel-1:1.1.1c-2.el8.x86_64 109/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-devel-0.176-5.el8.x86_64 110/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xfsprogs-devel-5.0.0-1.el8.x86_64 111/123 2020-02-05T16:57:49.096 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 112/123 2020-02-05T16:57:49.097 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/123 2020-02-05T16:57:49.097 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcom_err-devel-1.44.6-3.el8.x86_64 114/123 2020-02-05T16:57:49.097 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libuuid-devel-2.32.1-17.el8.x86_64 115/123 2020-02-05T16:57:49.097 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:15.0.0-9851.gecafd72.el8.x86_64 116/123 2020-02-05T16:57:49.097 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 117/123 2020-02-05T16:57:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 118/123 2020-02-05T16:57:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 119/123 2020-02-05T16:57:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.176-5.el8.x86_64 120/123 2020-02-05T16:57:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libelf-0.174-6.el8.x86_64 121/123 2020-02-05T16:57:49.099 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.176-5.el8.x86_64 122/123 2020-02-05T16:57:50.844 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : elfutils-libs-0.174-6.el8.x86_64 123/123 2020-02-05T16:57:50.845 INFO:teuthology.orchestra.run.smithi164.stdout:Installed products updated. 2020-02-05T16:57:51.568 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 105/123 2020-02-05T16:57:52.868 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 106/123 2020-02-05T16:57:53.212 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : openssl-devel-1:1.1.1c-2.el8.x86_64 107/123 2020-02-05T16:57:53.247 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:57:53.248 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2020-02-05T16:57:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.248 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-0.176-5.el8.x86_64 2020-02-05T16:57:53.249 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libs-0.176-5.el8.x86_64 2020-02-05T16:57:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:57:53.250 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2020-02-05T16:57:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.250 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.251 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.252 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:57:53.253 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: flex-2.6.1-9.el8.x86_64 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: bison-3.0.4-10.el8.x86_64 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: libtool-2.4.6-25.el8.x86_64 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: openssl-devel-1:1.1.1c-2.el8.x86_64 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: elfutils-libelf-devel-0.176-5.el8.x86_64 2020-02-05T16:57:53.254 INFO:teuthology.orchestra.run.smithi164.stdout: xfsprogs-devel-5.0.0-1.el8.x86_64 2020-02-05T16:57:53.255 INFO:teuthology.orchestra.run.smithi164.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-02-05T16:57:53.255 INFO:teuthology.orchestra.run.smithi164.stdout: libuuid-devel-2.32.1-17.el8.x86_64 2020-02-05T16:57:53.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-02-05T16:57:53.255 INFO:teuthology.orchestra.run.smithi164.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.256 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-6.2-6.el8.noarch 2020-02-05T16:57:53.257 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-02-05T16:57:53.258 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-02-05T16:57:53.258 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-02-05T16:57:53.258 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-02-05T16:57:53.258 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-2.6-1.el8.noarch 2020-02-05T16:57:53.258 INFO:teuthology.orchestra.run.smithi164.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-02-05T16:57:53.259 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-02-05T16:57:53.259 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.4.1-12.el8.noarch 2020-02-05T16:57:53.259 INFO:teuthology.orchestra.run.smithi164.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-02-05T16:57:53.259 INFO:teuthology.orchestra.run.smithi164.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-02-05T16:57:53.259 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-02-05T16:57:53.260 INFO:teuthology.orchestra.run.smithi164.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-02-05T16:57:53.260 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-02-05T16:57:53.260 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-02-05T16:57:53.260 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.2-3.el8.x86_64 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.3.1-3.el8.x86_64 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-apipkg-1.5-6.el8.noarch 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-02-05T16:57:53.261 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-02-05T16:57:53.262 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-10.0.1-1.el8.noarch 2020-02-05T16:57:53.262 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-02-05T16:57:53.262 INFO:teuthology.orchestra.run.smithi164.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-02-05T16:57:53.262 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: leveldb-1.21-2.el8.x86_64 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-13.el8.x86_64 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: automake-1.16.1-6.el8.noarch 2020-02-05T16:57:53.263 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: jq-1.5-12.el8.x86_64 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.0.6-13.el8.noarch 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2017.2-9.el8.noarch 2020-02-05T16:57:53.264 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.3.2-6.el8.x86_64 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: oniguruma-6.8.2-1.el8.x86_64 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-02-05T16:57:53.265 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-02-05T16:57:53.269 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.3-2.el8.x86_64 2020-02-05T16:57:53.269 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-utf32-10.32-1.el8.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: zlib-devel-1.2.11-10.el8.x86_64 2020-02-05T16:57:53.270 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-2.9.7-12.el8.x86_64 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: libxslt-1.1.32-3.el8.x86_64 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-devel-10.32-1.el8.x86_64 2020-02-05T16:57:53.271 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-02-05T16:57:53.272 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-02-05T16:57:53.272 INFO:teuthology.orchestra.run.smithi164.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-02-05T16:57:53.272 INFO:teuthology.orchestra.run.smithi164.stdout: pcre2-utf16-10.32-1.el8.x86_64 2020-02-05T16:57:53.272 INFO:teuthology.orchestra.run.smithi164.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-02-05T16:57:53.272 INFO:teuthology.orchestra.run.smithi164.stdout: libselinux-devel-2.9-2.1.el8.x86_64 2020-02-05T16:57:53.274 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-02-05T16:57:53.274 INFO:teuthology.orchestra.run.smithi164.stdout: krb5-devel-1.17-9.el8.x86_64 2020-02-05T16:57:53.274 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-02-05T16:57:53.274 INFO:teuthology.orchestra.run.smithi164.stdout: libcom_err-devel-1.44.6-3.el8.x86_64 2020-02-05T16:57:53.274 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:57:53.275 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2020-02-05T16:57:53.432 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 108/123 2020-02-05T16:57:53.563 DEBUG:teuthology.parallel:result is None 2020-02-05T16:57:53.730 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : xfsprogs-devel-5.0.0-1.el8.x86_64 109/123 2020-02-05T16:57:53.921 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libtool-2.4.6-25.el8.x86_64 110/123 2020-02-05T16:57:53.962 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 110/123 2020-02-05T16:57:54.290 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_6 111/123 2020-02-05T16:57:54.475 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : elfutils-libelf-devel-0.176-5.el8.x86_64 112/123 2020-02-05T16:57:54.704 INFO:teuthology.orchestra.run.smithi008.stdout: Upgrading : elfutils-libs-0.176-5.el8.x86_64 113/123 2020-02-05T16:57:54.928 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 114/123 2020-02-05T16:57:55.135 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 115/123 2020-02-05T16:57:55.290 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : ceph-immutable-object-cache-2:15.0.0-9851.gecafd 116/123 2020-02-05T16:57:55.446 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.0.0-9851.gecafd 116/123 2020-02-05T16:57:55.763 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : bison-3.0.4-10.el8.x86_64 117/123 2020-02-05T16:57:55.857 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 117/123 2020-02-05T16:57:56.048 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : flex-2.6.1-9.el8.x86_64 118/123 2020-02-05T16:57:56.082 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 118/123 2020-02-05T16:57:56.209 INFO:teuthology.orchestra.run.smithi008.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 119/123 2020-02-05T16:57:56.434 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : elfutils-libs-0.174-6.el8.x86_64 120/123 2020-02-05T16:57:56.435 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/123 2020-02-05T16:57:56.768 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/123 2020-02-05T16:57:56.769 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/123 2020-02-05T16:57:57.071 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/123 2020-02-05T16:57:57.281 INFO:teuthology.orchestra.run.smithi008.stdout: Cleanup : elfutils-libelf-0.174-6.el8.x86_64 123/123 2020-02-05T16:57:58.177 INFO:teuthology.orchestra.run.smithi008.stdout: Running scriptlet: elfutils-libelf-0.174-6.el8.x86_64 123/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-2:15.0.0-9851.gecafd72.el8.x86_64 1/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 2/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 3/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 4/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-immutable-object-cache-2:15.0.0-9851.gecafd 5/123 2020-02-05T16:57:58.178 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 6/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 7/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 8/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 9/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 10/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 11/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 12/123 2020-02-05T16:57:58.179 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 13/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_6 14/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 15/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 16/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_ 17/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 18/123 2020-02-05T16:57:58.180 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8 19/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x 20/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 21/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 22/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 23/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 24/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 25/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 26/123 2020-02-05T16:57:58.181 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 27/123 2020-02-05T16:57:58.182 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.e 28/123 2020-02-05T16:57:58.182 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noar 29/123 2020-02-05T16:57:58.182 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.no 30/123 2020-02-05T16:57:58.182 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.0.0-9851.geca 31/123 2020-02-05T16:57:58.189 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 32/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 33/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 34/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 35/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 36/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 37/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 38/123 2020-02-05T16:57:58.190 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 39/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 40/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-portend-2.6-1.el8.noarch 41/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 42/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 43/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 44/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 45/123 2020-02-05T16:57:58.191 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 46/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 47/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 48/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 49/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 50/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 51/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 52/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 53/123 2020-02-05T16:57:58.192 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 54/123 2020-02-05T16:57:58.193 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 55/123 2020-02-05T16:57:58.193 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 56/123 2020-02-05T16:57:58.193 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 57/123 2020-02-05T16:57:58.194 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 58/123 2020-02-05T16:57:58.194 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 59/123 2020-02-05T16:57:58.194 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-kubernetes-10.0.1-1.el8.noarch 60/123 2020-02-05T16:57:58.194 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 61/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 62/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 63/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 64/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 65/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : leveldb-1.21-2.el8.x86_64 66/123 2020-02-05T16:57:58.195 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 67/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : xmlstarlet-1.6.1-13.el8.x86_64 68/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 69/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : pcre2-utf32-10.32-1.el8.x86_64 70/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : zlib-devel-1.2.11-10.el8.x86_64 71/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 72/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 73/123 2020-02-05T16:57:58.196 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 74/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 75/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 76/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : pcre2-devel-10.32-1.el8.x86_64 77/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 78/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 79/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 80/123 2020-02-05T16:57:58.197 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : pcre2-utf16-10.32-1.el8.x86_64 81/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libsepol-devel-2.9-1.el8.x86_64 82/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libselinux-devel-2.9-2.1.el8.x86_64 83/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 84/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : krb5-devel-1.17-9.el8.x86_64 85/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openssl-devel-1:1.1.1c-2.el8.x86_64 86/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : elfutils-libelf-devel-0.176-5.el8.x86_64 87/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : xfsprogs-devel-5.0.0-1.el8.x86_64 88/123 2020-02-05T16:57:58.198 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 89/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 90/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libcom_err-devel-1.44.6-3.el8.x86_64 91/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libuuid-devel-2.32.1-17.el8.x86_64 92/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : automake-1.16.1-6.el8.noarch 93/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 94/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 95/123 2020-02-05T16:57:58.199 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : jq-1.5-12.el8.x86_64 96/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 97/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 98/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 99/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 100/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 101/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 102/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 103/123 2020-02-05T16:57:58.200 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 104/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 105/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 106/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 107/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : flex-2.6.1-9.el8.x86_64 108/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 109/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : bison-3.0.4-10.el8.x86_64 110/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 111/123 2020-02-05T16:57:58.201 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 112/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 113/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 114/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 115/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-2:15.0.0-9851.gecafd72.el8.x86_64 116/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 117/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 118/123 2020-02-05T16:57:58.202 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 119/123 2020-02-05T16:57:58.203 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : elfutils-libelf-0.176-5.el8.x86_64 120/123 2020-02-05T16:57:58.203 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : elfutils-libelf-0.174-6.el8.x86_64 121/123 2020-02-05T16:57:58.203 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : elfutils-libs-0.176-5.el8.x86_64 122/123 2020-02-05T16:58:00.057 INFO:teuthology.orchestra.run.smithi008.stdout: Verifying : elfutils-libs-0.174-6.el8.x86_64 123/123 2020-02-05T16:58:00.058 INFO:teuthology.orchestra.run.smithi008.stdout:Installed products updated. 2020-02-05T16:58:02.440 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:02.440 INFO:teuthology.orchestra.run.smithi008.stdout:Upgraded: 2020-02-05T16:58:02.441 INFO:teuthology.orchestra.run.smithi008.stdout: librados2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.441 INFO:teuthology.orchestra.run.smithi008.stdout: librbd1-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.441 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libelf-0.176-5.el8.x86_64 2020-02-05T16:58:02.441 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libs-0.176-5.el8.x86_64 2020-02-05T16:58:02.441 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout:Installed: 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-base-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-immutable-object-cache-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.442 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-radosgw-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-test-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: cephadm-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs-devel-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: libcephfs2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: librados-devel-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cephfs-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rados-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.443 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rbd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rgw-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-fuse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-mirror-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: rbd-nbd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-cephadm-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-dashboard-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:58:02.444 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-diskprediction-local-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:58:02.445 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mgr-rook-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:58:02.445 INFO:teuthology.orchestra.run.smithi008.stdout: openssl-devel-1:1.1.1c-2.el8.x86_64 2020-02-05T16:58:02.445 INFO:teuthology.orchestra.run.smithi008.stdout: elfutils-libelf-devel-0.176-5.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: xfsprogs-devel-5.0.0-1.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: libaio-devel-0.3.112-1.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: libuuid-devel-2.32.1-17.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: flex-2.6.1-9.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: bison-3.0.4-10.el8.x86_64 2020-02-05T16:58:02.446 INFO:teuthology.orchestra.run.smithi008.stdout: libtool-2.4.6-25.el8.x86_64 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mds-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-mon-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.447 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-osd-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-selinux-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: libradosstriper1-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: librgw2-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-argparse-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: python3-ceph-common-2:15.0.0-9851.gecafd72.el8.x86_64 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: ceph-grafana-dashboards-2:15.0.0-9851.gecafd72.el8.noarch 2020-02-05T16:58:02.448 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-6.2-6.el8.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-02-05T16:58:02.449 INFO:teuthology.orchestra.run.smithi008.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-portend-2.6-1.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-routes-2.4.1-12.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-02-05T16:58:02.450 INFO:teuthology.orchestra.run.smithi008.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-02-05T16:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-02-05T16:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-02-05T16:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-02-05T16:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout: liboath-2.6.2-3.el8.x86_64 2020-02-05T16:58:02.451 INFO:teuthology.orchestra.run.smithi008.stdout: libunwind-1.3.1-3.el8.x86_64 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-apipkg-1.5-6.el8.noarch 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-02-05T16:58:02.452 INFO:teuthology.orchestra.run.smithi008.stdout: python3-kubernetes-10.0.1-1.el8.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: leveldb-1.21-2.el8.x86_64 2020-02-05T16:58:02.453 INFO:teuthology.orchestra.run.smithi008.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: xmlstarlet-1.6.1-13.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: libverto-devel-0.3.0-5.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-utf32-10.32-1.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: zlib-devel-1.2.11-10.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: fuse-2.9.7-12.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: libxslt-1.1.32-3.el8.x86_64 2020-02-05T16:58:02.454 INFO:teuthology.orchestra.run.smithi008.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-devel-10.32-1.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: pcre2-utf16-10.32-1.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: libsepol-devel-2.9-1.el8.x86_64 2020-02-05T16:58:02.455 INFO:teuthology.orchestra.run.smithi008.stdout: libselinux-devel-2.9-2.1.el8.x86_64 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: krb5-devel-1.17-9.el8.x86_64 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: libcom_err-devel-1.44.6-3.el8.x86_64 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: automake-1.16.1-6.el8.noarch 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-02-05T16:58:02.456 INFO:teuthology.orchestra.run.smithi008.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: jq-1.5-12.el8.x86_64 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-mako-1.0.6-13.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pytz-2017.2-9.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-02-05T16:58:02.457 INFO:teuthology.orchestra.run.smithi008.stdout: socat-1.7.3.2-6.el8.x86_64 2020-02-05T16:58:02.458 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-02-05T16:58:02.458 INFO:teuthology.orchestra.run.smithi008.stdout: oniguruma-6.8.2-1.el8.x86_64 2020-02-05T16:58:02.458 INFO:teuthology.orchestra.run.smithi008.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-02-05T16:58:02.458 INFO:teuthology.orchestra.run.smithi008.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-02-05T16:58:02.458 INFO:teuthology.orchestra.run.smithi008.stdout: openblas-0.3.3-2.el8.x86_64 2020-02-05T16:58:02.459 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-02-05T16:58:02.459 INFO:teuthology.orchestra.run.smithi008.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-02-05T16:58:02.459 INFO:teuthology.orchestra.run.smithi008.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2020-02-05T16:58:02.459 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:02.459 INFO:teuthology.orchestra.run.smithi008.stdout:Complete! 2020-02-05T16:58:02.728 DEBUG:teuthology.parallel:result is None 2020-02-05T16:58:02.728 INFO:teuthology.orchestra.run.smithi008:> sudo lsb_release -is 2020-02-05T16:58:02.858 INFO:teuthology.orchestra.run.smithi008.stdout:RedHatEnterprise 2020-02-05T16:58:02.861 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:58:02.862 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:58:03.037 INFO:teuthology.orchestra.run.smithi008:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-02-05T16:58:03.101 INFO:teuthology.orchestra.run.smithi008.stdout:15.0.0-9851.gecafd72.el8 2020-02-05T16:58:03.102 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-9851.gecafd72.el8 2020-02-05T16:58:03.102 INFO:teuthology.task.install:The correct ceph version 15.0.0-9851.gecafd72 is installed. 2020-02-05T16:58:03.102 INFO:teuthology.orchestra.run.smithi164:> sudo lsb_release -is 2020-02-05T16:58:03.261 INFO:teuthology.orchestra.run.smithi164.stdout:RedHatEnterprise 2020-02-05T16:58:03.264 DEBUG:teuthology.misc:System to be installed: RedHatEnterprise 2020-02-05T16:58:03.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:58:03.439 INFO:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-02-05T16:58:03.508 INFO:teuthology.orchestra.run.smithi164.stdout:15.0.0-9851.gecafd72.el8 2020-02-05T16:58:03.508 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-9851.gecafd72.el8 2020-02-05T16:58:03.508 INFO:teuthology.task.install:The correct ceph version 15.0.0-9851.gecafd72 is installed. 2020-02-05T16:58:03.509 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-02-05T16:58:03.516 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-02-05T16:58:03.609 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-02-05T16:58:03.712 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-02-05T16:58:03.713 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-02-05T16:58:03.809 INFO:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-02-05T16:58:03.890 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-02-05T16:58:03.994 INFO:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-02-05T16:58:04.086 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-02-05T16:58:04.087 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-02-05T16:58:04.172 INFO:teuthology.orchestra.run.smithi008:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-02-05T16:58:04.258 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-02-05T16:58:04.362 INFO:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-02-05T16:58:04.461 INFO:teuthology.run_tasks:Running task ceph... 2020-02-05T16:58:04.548 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2020-02-05T16:58:04.549 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 777 /var/log/ceph 2020-02-05T16:58:04.554 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2020-02-05T16:58:04.650 INFO:tasks.ceph:Disabling ceph logrotate... 2020-02-05T16:58:04.650 INFO:teuthology.orchestra.run.smithi008:> sudo rm -f -- /etc/logrotate.d/ceph 2020-02-05T16:58:04.656 INFO:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2020-02-05T16:58:04.752 INFO:tasks.ceph:Creating extra log directories... 2020-02-05T16:58:04.752 INFO:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-02-05T16:58:04.756 INFO:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2020-02-05T16:58:04.855 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2020-02-05T16:58:04.855 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2020-02-05T16:58:04.855 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2020-02-05T16:58:04.855 INFO:teuthology.orchestra.run.smithi008:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-02-05T16:58:04.903 INFO:teuthology.orchestra.run.smithi008:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:05.093 INFO:teuthology.orchestra.run.smithi008:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:05.181 INFO:teuthology.orchestra.run.smithi164:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2020-02-05T16:58:05.239 INFO:teuthology.orchestra.run.smithi164:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:05.444 INFO:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:05.547 INFO:tasks.ceph:Creating ceph cluster ceph... 2020-02-05T16:58:05.548 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'fuse_default_permissions': 0, 'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10000, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd op complaint time': 180}}} 2020-02-05T16:58:05.548 INFO:tasks.ceph:ctx.config {'kernel': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'basic', 'sha1': 'a11f6ef5e0ea8c9afda0ced0ecea82f02ab60e9f', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'}}, {'ceph': None}, {'ceph-fuse': None}, {'check-counter': {'counters': {'mds': ['mds.dir_split']}}}, {'workunit': {'clients': {'all': ['kernel_untar_build.sh']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.2502', 'owner': 'scheduled_zyan@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'targets': {'smithi008.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCn2A/ANXtv9dsa38HZFt8mns4dMjQq8EtRuB2FzX1Wo36HuBKjOR7Sxqo/o55ISzdqE9HkP0Tvsly6KvUHWgmHq6ryjp2FPcqt4FJ2CAgh3iG09PGnUkdyaTm2Xmig/i75PxOVFxu2hmUCf0NAcniGAUMQLpyIRK5qlWviz6h4GQGQD79ryFuGunpgesa70FZBor3VsINim1zLatIbb8SMj83S9K2KHb2EGKcSIcpKJoEdYPPL5JeT2YRcTV8yRI3yuHH8ThNzEw4OSvE2cC/e7vIYXoS3J18BP4BnjfUKicads3CwMarGdSWsCZulQq6vtrhx/FKXB1Xh8WVju3GF', 'smithi164.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5XcYOcYXs6V5ZQMjnk27EqGyJkYMzBTSsMWrSQJA1Y+6RrGKAN6wCwrP4sL+6i3Q8ZUEDiYPNlJKteiqaljO0LFqqWXLFDAvbdYrmr1tGkXlgYuFPhec8u38De/5jF9aRHwce0vXJjz6NZy89XbqOsbpuEW02vWf0p/PmJtDK7Jqk4d9iI2Juseb+vNQTI+b2sPQdJlKkeZnA11UVRHW/emT35GTMvXYhDD3iIeIQF5EAFZs36wsDUmIxPnIsngImMAl7VSmJzfLWD2IW0dkLBZTWmL3za7otn8FykLxUIbMGsHDLw89MlZ2OcJCaq3q2VUPbfRC+/hh43dAcne8P'}, 'job_id': '4734997', 'tube': 'smithi', 'suite_branch': 'wip-async-dirops', 'priority': 50, 'os_version': '8.0', 'first_in_suite': False, 'branch': 'wip-async-dirops', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': None, 'archive_path': '/home/teuthworker/archive/zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi/4734997', 'description': 'fs/basic_workload/{begin.yaml clusters/fixed-2-ucephfs.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} inline/no.yaml mount/fuse.yaml objectstore-ec/bluestore-bitmap.yaml omap_limit/10000.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{rhel_8.yaml} tasks/cfuse_workunit_kernel_untar_build.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255', 'name': 'zyan-2020-02-05_12:49:30-fs-wip-async-dirops-testing-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'client': {'fuse_default_permissions': 0, 'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'osd_max_omap_entries_per_request': 10000, 'debug journal': 20, 'bluestore allocator': 'bitmap', 'debug rocksdb': 10, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'bdev async discard': True, 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'bdev enable discard': True, 'osd op complaint time': 180}}, 'fs': 'xfs', 'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'}, 'ceph-deploy': {'fs': 'xfs', 'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'mon osd full ratio': 0.9, 'mon osd backfillfull_ratio': 0.85, 'bluestore fsck on mount': True, 'mon osd nearfull ratio': 0.8, 'debug bluestore': 20, 'debug bluefs': 20, 'osd objectstore': 'bluestore', 'bluestore block size': 96636764160, 'debug rocksdb': 10, 'bdev enable discard': True, 'osd failsafe full ratio': 0.95, 'bdev async discard': True}}}, 'install': {'ceph': {'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'}}, 'admin_socket': {'branch': 'wip-async-dirops'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255', 'branch': 'wip-async-dirops'}}, 'os_type': 'rhel', 'suite_sha1': 'ecafd7283ef390be73a605986afe2a0f325a3255'} 2020-02-05T16:58:05.549 INFO:teuthology.orchestra.run.smithi008:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-02-05T16:58:05.555 INFO:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2020-02-05T16:58:05.613 INFO:teuthology.orchestra.run.smithi008:> sudo install -d -m0777 -- /var/run/ceph 2020-02-05T16:58:05.642 INFO:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2020-02-05T16:58:05.801 DEBUG:teuthology.orchestra.remote:smithi008:/scratch_devs is 72B 2020-02-05T16:58:05.852 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-02-05T16:58:05.853 INFO:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-02-05T16:58:05.907 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 6h/6d Inode: 135966 Links: 1 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2020-02-05 16:57:57.684454777 +0000 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2020-02-05 16:55:53.449862939 +0000 2020-02-05T16:58:05.908 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2020-02-05 16:55:53.449862939 +0000 2020-02-05T16:58:05.909 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2020-02-05T16:58:05.944 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2020-02-05T16:58:05.944 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2020-02-05T16:58:05.944 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000335191 s, 1.5 MB/s 2020-02-05T16:58:05.955 INFO:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 6h/6d Inode: 140299 Links: 1 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2020-02-05 16:57:57.684454777 +0000 2020-02-05T16:58:06.012 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2020-02-05 16:55:54.024856759 +0000 2020-02-05T16:58:06.013 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2020-02-05 16:55:54.024856759 +0000 2020-02-05T16:58:06.013 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2020-02-05T16:58:06.052 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2020-02-05T16:58:06.053 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2020-02-05T16:58:06.053 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000318816 s, 1.6 MB/s 2020-02-05T16:58:06.062 INFO:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-02-05T16:58:06.117 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-02-05T16:58:06.117 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.117 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 6h/6d Inode: 140335 Links: 1 2020-02-05T16:58:06.117 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.117 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.118 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2020-02-05 16:57:57.684454777 +0000 2020-02-05T16:58:06.118 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2020-02-05 16:55:54.601850558 +0000 2020-02-05T16:58:06.118 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2020-02-05 16:55:54.601850558 +0000 2020-02-05T16:58:06.118 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2020-02-05T16:58:06.155 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2020-02-05T16:58:06.156 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2020-02-05T16:58:06.156 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000315185 s, 1.6 MB/s 2020-02-05T16:58:06.164 INFO:teuthology.orchestra.run.smithi008:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-02-05T16:58:06.225 INFO:teuthology.orchestra.run.smithi008.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 6h/6d Inode: 137510 Links: 1 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 2020-02-05 16:57:57.684454777 +0000 2020-02-05T16:58:06.226 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2020-02-05 16:55:55.191844217 +0000 2020-02-05T16:58:06.227 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2020-02-05 16:55:55.191844217 +0000 2020-02-05T16:58:06.227 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2020-02-05T16:58:06.269 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records in 2020-02-05T16:58:06.269 INFO:teuthology.orchestra.run.smithi008.stderr:1+0 records out 2020-02-05T16:58:06.269 INFO:teuthology.orchestra.run.smithi008.stderr:512 bytes copied, 0.000363739 s, 1.4 MB/s 2020-02-05T16:58:06.277 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-02-05T16:58:06.277 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-02-05T16:58:06.277 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-02-05T16:58:06.277 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:06.341 DEBUG:teuthology.orchestra.remote:smithi164:/scratch_devs is 72B 2020-02-05T16:58:06.401 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-02-05T16:58:06.401 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2020-02-05T16:58:06.455 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-1 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 134021 Links: 1 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-02-05 16:57:48.601951184 +0000 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-02-05 16:55:22.302023865 +0000 2020-02-05T16:58:06.456 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-02-05 16:55:22.302023865 +0000 2020-02-05T16:58:06.457 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-02-05T16:58:06.499 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-02-05T16:58:06.500 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-02-05T16:58:06.500 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000333877 s, 1.5 MB/s 2020-02-05T16:58:06.508 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2020-02-05T16:58:06.563 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2020-02-05T16:58:06.563 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.563 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 137302 Links: 1 2020-02-05T16:58:06.563 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.564 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.564 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-02-05 16:57:48.601951184 +0000 2020-02-05T16:58:06.564 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-02-05 16:55:22.848014305 +0000 2020-02-05T16:58:06.564 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-02-05 16:55:22.848014305 +0000 2020-02-05T16:58:06.564 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-02-05T16:58:06.604 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-02-05T16:58:06.605 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-02-05T16:58:06.605 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000495759 s, 1.0 MB/s 2020-02-05T16:58:06.613 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2020-02-05T16:58:06.666 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-3 2020-02-05T16:58:06.666 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.666 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 133119 Links: 1 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-02-05 16:57:48.601951184 +0000 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-02-05 16:55:23.383004937 +0000 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-02-05 16:55:23.383004937 +0000 2020-02-05T16:58:06.667 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-02-05T16:58:06.707 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-02-05T16:58:06.707 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-02-05T16:58:06.707 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000305934 s, 1.7 MB/s 2020-02-05T16:58:06.717 INFO:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-4 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 6h/6d Inode: 138452 Links: 1 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2020-02-05T16:58:06.771 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2020-02-05 16:57:48.602951166 +0000 2020-02-05T16:58:06.772 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2020-02-05 16:55:23.918995551 +0000 2020-02-05T16:58:06.772 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2020-02-05 16:55:23.918995551 +0000 2020-02-05T16:58:06.772 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: - 2020-02-05T16:58:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2020-02-05T16:58:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2020-02-05T16:58:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000356888 s, 1.4 MB/s 2020-02-05T16:58:06.817 INFO:tasks.ceph:fs option selected, checking for scratch devs 2020-02-05T16:58:06.818 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2020-02-05T16:58:06.818 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2020-02-05T16:58:06.818 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:06.818 INFO:tasks.ceph:Generating config... 2020-02-05T16:58:06.820 INFO:tasks.ceph:[mds] mds debug frag = True 2020-02-05T16:58:06.820 INFO:tasks.ceph:[mds] mds bal split bits = 3 2020-02-05T16:58:06.821 INFO:tasks.ceph:[mds] mds bal split size = 100 2020-02-05T16:58:06.821 INFO:tasks.ceph:[mds] debug mds = 20 2020-02-05T16:58:06.821 INFO:tasks.ceph:[mds] mds bal merge size = 5 2020-02-05T16:58:06.822 INFO:tasks.ceph:[mds] debug ms = 1 2020-02-05T16:58:06.822 INFO:tasks.ceph:[mds] mds bal frag = True 2020-02-05T16:58:06.822 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2020-02-05T16:58:06.822 INFO:tasks.ceph:[mds] mds op complaint time = 180 2020-02-05T16:58:06.823 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2020-02-05T16:58:06.823 INFO:tasks.ceph:[mds] osd op complaint time = 180 2020-02-05T16:58:06.823 INFO:tasks.ceph:[mds] mds verify scatter = True 2020-02-05T16:58:06.823 INFO:tasks.ceph:[client] fuse_default_permissions = 0 2020-02-05T16:58:06.823 INFO:tasks.ceph:[client] debug ms = 1 2020-02-05T16:58:06.824 INFO:tasks.ceph:[client] debug client = 20 2020-02-05T16:58:06.824 INFO:tasks.ceph:[client] client mount timeout = 600 2020-02-05T16:58:06.824 INFO:tasks.ceph:[mon] debug mon = 20 2020-02-05T16:58:06.824 INFO:tasks.ceph:[mon] debug paxos = 20 2020-02-05T16:58:06.825 INFO:tasks.ceph:[mon] mon op complaint time = 120 2020-02-05T16:58:06.825 INFO:tasks.ceph:[mon] debug ms = 1 2020-02-05T16:58:06.825 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2020-02-05T16:58:06.825 INFO:tasks.ceph:[osd] debug ms = 1 2020-02-05T16:58:06.825 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2020-02-05T16:58:06.825 INFO:tasks.ceph:[osd] debug osd = 25 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] debug bluestore = 20 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] debug bluefs = 20 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] osd_max_omap_entries_per_request = 10000 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] debug journal = 20 2020-02-05T16:58:06.826 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] debug rocksdb = 10 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] bdev async discard = True 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] debug filestore = 20 2020-02-05T16:58:06.827 INFO:tasks.ceph:[osd] bdev enable discard = True 2020-02-05T16:58:06.828 INFO:tasks.ceph:[osd] osd op complaint time = 180 2020-02-05T16:58:06.828 INFO:tasks.ceph:Setting up mon.a... 2020-02-05T16:58:06.828 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2020-02-05T16:58:06.957 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/ceph.keyring 2020-02-05T16:58:06.969 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2020-02-05T16:58:07.095 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0644 /etc/ceph/ceph.keyring 2020-02-05T16:58:07.202 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.8'), ('mon.c', '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]'), ('mon.b', '172.21.15.164')] 2020-02-05T16:58:07.203 INFO:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.8 --addv c '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]' --add b 172.21.15.164 --print /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:07.297 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: generated fsid db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:epoch 0 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:fsid db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:last_changed 2020-02-05T16:58:07.296635+0000 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:created 2020-02-05T16:58:07.296635+0000 2020-02-05T16:58:07.298 INFO:teuthology.orchestra.run.smithi008.stdout:min_mon_release 0 (unknown) 2020-02-05T16:58:07.299 INFO:teuthology.orchestra.run.smithi008.stdout:0: [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] mon.a 2020-02-05T16:58:07.299 INFO:teuthology.orchestra.run.smithi008.stdout:1: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.b 2020-02-05T16:58:07.299 INFO:teuthology.orchestra.run.smithi008.stdout:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2020-02-05T16:58:07.299 INFO:teuthology.orchestra.run.smithi008.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2020-02-05T16:58:07.300 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7... 2020-02-05T16:58:07.301 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-02-05T16:58:07.307 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2020-02-05T16:58:07.480 INFO:teuthology.orchestra.run.smithi008.stdout:[global] 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: chdir = "" 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: auth supported = cephx 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: filestore xattr use omap = true 2020-02-05T16:58:07.481 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: mon clock drift allowed = 1.000 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: osd crush chooseleaf type = 0 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: auth debug = true 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.482 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on old message = true 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: ms die on bug = true 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: mon max pg per osd = 10000 # >= luminous 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: mon pg warn max object skew = 0 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: # disable pg_autoscaler by default for new pools 2020-02-05T16:58:07.483 INFO:teuthology.orchestra.run.smithi008.stdout: osd_pool_default_pg_autoscale_mode = off 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default size = 2 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow primary affinity = true 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd allow pg remap = true 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on legacy crush tunables = false 2020-02-05T16:58:07.484 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on crush straw calc version zero = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on no sortbitwise = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on osd down out interval zero = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: mon warn on too few osds = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: mon_warn_on_pool_no_redundancy = false 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.485 INFO:teuthology.orchestra.run.smithi008.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-02-05T16:58:07.486 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.486 INFO:teuthology.orchestra.run.smithi008.stdout: osd default data pool replay window = 5 2020-02-05T16:58:07.486 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.486 INFO:teuthology.orchestra.run.smithi008.stdout: mon allow pool delete = true 2020-02-05T16:58:07.486 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: mon cluster log file level = debug 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: debug asserts on shutdown = true 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: mon host = "172.21.15.8,[v2:172.21.15.164:3301,v1:172.21.15.164:6790],172.21.15.164" 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: fsid = db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout:[osd] 2020-02-05T16:58:07.487 INFO:teuthology.orchestra.run.smithi008.stdout: osd journal size = 100 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub load threshold = 5.0 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: osd scrub max interval = 600 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: osd recover clone overlap = true 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: osd recovery max chunk = 1048576 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.488 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug shutdown = true 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug op order = true 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug verify stray on activate = true 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: osd open classes on start = true 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug pg log writeout = true 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.489 INFO:teuthology.orchestra.run.smithi008.stdout: osd deep scrub update digest min age = 30 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: osd map max advance = 10 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: journal zero on create = true 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.490 INFO:teuthology.orchestra.run.smithi008.stdout: filestore ondisk finisher threads = 3 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: filestore apply finisher threads = 3 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: bdev debug aio = true 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: osd debug misdirected ops = true 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd full ratio = 0.9 2020-02-05T16:58:07.491 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore fsck on mount = True 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: debug osd = 25 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluestore = 20 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: debug bluefs = 20 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd backfillfull_ratio = 0.85 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: osd_max_omap_entries_per_request = 10000 2020-02-05T16:58:07.492 INFO:teuthology.orchestra.run.smithi008.stdout: debug journal = 20 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore allocator = bitmap 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: debug rocksdb = 10 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd nearfull ratio = 0.8 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: osd objectstore = bluestore 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: bdev async discard = True 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: osd failsafe full ratio = 0.95 2020-02-05T16:58:07.493 INFO:teuthology.orchestra.run.smithi008.stdout: bluestore block size = 96636764160 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout: debug filestore = 20 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout: bdev enable discard = True 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout: osd op complaint time = 180 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr] 2020-02-05T16:58:07.494 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: debug mgr = 20 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: mgr/telemetry/nag = false 2020-02-05T16:58:07.495 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.496 INFO:teuthology.orchestra.run.smithi008.stdout:[mon] 2020-02-05T16:58:07.496 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2020-02-05T16:58:07.496 INFO:teuthology.orchestra.run.smithi008.stdout: debug mon = 20 2020-02-05T16:58:07.496 INFO:teuthology.orchestra.run.smithi008.stdout: debug paxos = 20 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: debug auth = 20 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon data avail warn = 5 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon mgr mkfs grace = 240 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min pgs per osd = 4 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd reporter subtree level = osd 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon osd prime pg temp = true 2020-02-05T16:58:07.497 INFO:teuthology.orchestra.run.smithi008.stdout: mon reweight min bytes per osd = 10 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout: mon op complaint time = 120 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout:[client] 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout: rgw cache enabled = true 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable ops log = true 2020-02-05T16:58:07.498 INFO:teuthology.orchestra.run.smithi008.stdout: rgw enable usage log = true 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: fuse_default_permissions = 0 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: debug client = 20 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout: client mount timeout = 600 2020-02-05T16:58:07.499 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.a] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.c] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout:[mon.b] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout:[mds.a] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout:[mds.b] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout:[mds] 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout: mds debug frag = True 2020-02-05T16:58:07.500 INFO:teuthology.orchestra.run.smithi008.stdout: mds bal split bits = 3 2020-02-05T16:58:07.501 INFO:teuthology.orchestra.run.smithi008.stdout: mds bal split size = 100 2020-02-05T16:58:07.501 INFO:teuthology.orchestra.run.smithi008.stdout: debug mds = 20 2020-02-05T16:58:07.501 INFO:teuthology.orchestra.run.smithi008.stdout: mds bal merge size = 5 2020-02-05T16:58:07.501 INFO:teuthology.orchestra.run.smithi008.stdout: debug ms = 1 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: mds bal frag = True 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: mds bal fragment size max = 10000 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: mds op complaint time = 180 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: mds debug scatterstat = True 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: osd op complaint time = 180 2020-02-05T16:58:07.502 INFO:teuthology.orchestra.run.smithi008.stdout: mds verify scatter = True 2020-02-05T16:58:07.503 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2020-02-05T16:58:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2020-02-05T16:58:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2020-02-05T16:58:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2020-02-05T16:58:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2020-02-05T16:58:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2020-02-05T16:58:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2020-02-05T16:58:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2020-02-05T16:58:07.508 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.8,[v2:172.21.15.164:3301,v1:172.21.15.164:6790],172.21.15.164" 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2020-02-05T16:58:07.509 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2020-02-05T16:58:07.510 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.511 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2020-02-05T16:58:07.511 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2020-02-05T16:58:07.511 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2020-02-05T16:58:07.511 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.511 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.512 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2020-02-05T16:58:07.513 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 25 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 20 2020-02-05T16:58:07.514 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 20 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: osd_max_omap_entries_per_request = 10000 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: debug journal = 20 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore allocator = bitmap 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 10 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2020-02-05T16:58:07.515 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: bdev async discard = True 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: debug filestore = 20 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: bdev enable discard = True 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2020-02-05T16:58:07.516 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.517 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2020-02-05T16:58:07.517 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-02-05T16:58:07.517 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2020-02-05T16:58:07.517 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-02-05T16:58:07.517 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-02-05T16:58:07.518 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2020-02-05T16:58:07.519 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: mon op complaint time = 120 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2020-02-05T16:58:07.520 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2020-02-05T16:58:07.521 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2020-02-05T16:58:07.521 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2020-02-05T16:58:07.521 INFO:teuthology.orchestra.run.smithi164.stdout: fuse_default_permissions = 0 2020-02-05T16:58:07.521 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-02-05T16:58:07.521 INFO:teuthology.orchestra.run.smithi164.stdout: debug client = 20 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout: client mount timeout = 600 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2020-02-05T16:58:07.522 INFO:teuthology.orchestra.run.smithi164.stdout:[mds] 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug frag = True 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal split bits = 3 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal split size = 100 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: debug mds = 20 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal merge size = 5 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2020-02-05T16:58:07.523 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal frag = True 2020-02-05T16:58:07.524 INFO:teuthology.orchestra.run.smithi164.stdout: mds bal fragment size max = 10000 2020-02-05T16:58:07.524 INFO:teuthology.orchestra.run.smithi164.stdout: mds op complaint time = 180 2020-02-05T16:58:07.524 INFO:teuthology.orchestra.run.smithi164.stdout: mds debug scatterstat = True 2020-02-05T16:58:07.524 INFO:teuthology.orchestra.run.smithi164.stdout: osd op complaint time = 180 2020-02-05T16:58:07.524 INFO:teuthology.orchestra.run.smithi164.stdout: mds verify scatter = True 2020-02-05T16:58:07.535 INFO:tasks.ceph:Creating admin key on mon.a... 2020-02-05T16:58:07.536 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2020-02-05T16:58:07.680 INFO:tasks.ceph:Copying monmap to all nodes... 2020-02-05T16:58:07.742 DEBUG:teuthology.orchestra.remote:smithi008:/etc/ceph/ceph.keyring is 206B 2020-02-05T16:58:07.888 DEBUG:teuthology.orchestra.remote:smithi008:/home/ubuntu/cephtest/ceph.monmap is 379B 2020-02-05T16:58:07.958 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:58:07.959 INFO:teuthology.orchestra.run.smithi008:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-02-05T16:58:08.115 INFO:teuthology.orchestra.run.smithi008:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:08.174 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:58:08.174 INFO:teuthology.orchestra.run.smithi164:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2020-02-05T16:58:08.345 INFO:teuthology.orchestra.run.smithi164:> cat > /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:08.402 INFO:tasks.ceph:Setting up mon nodes... 2020-02-05T16:58:08.402 INFO:tasks.ceph:Setting up mgr nodes... 2020-02-05T16:58:08.402 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2020-02-05T16:58:08.581 INFO:teuthology.orchestra.run.smithi008.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2020-02-05T16:58:08.588 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2020-02-05T16:58:08.774 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2020-02-05T16:58:08.780 INFO:tasks.ceph:Setting up mds nodes... 2020-02-05T16:58:08.781 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2020-02-05T16:58:08.956 INFO:teuthology.orchestra.run.smithi008.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2020-02-05T16:58:08.962 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2020-02-05T16:58:09.061 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2020-02-05T16:58:09.226 INFO:teuthology.orchestra.run.smithi164.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2020-02-05T16:58:09.232 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2020-02-05T16:58:09.324 INFO:tasks.ceph_client:Setting up client nodes... 2020-02-05T16:58:09.324 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2020-02-05T16:58:09.458 INFO:teuthology.orchestra.run.smithi008.stdout:creating /etc/ceph/ceph.client.0.keyring 2020-02-05T16:58:09.507 INFO:tasks.ceph:Running mkfs on osd nodes... 2020-02-05T16:58:09.508 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi008.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}} 2020-02-05T16:58:09.508 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2020-02-05T16:58:09.612 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:09.612 INFO:tasks.ceph:{} 2020-02-05T16:58:09.612 INFO:tasks.ceph:osd.0 2020-02-05T16:58:09.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:58:09.613 INFO:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-02-05T16:58:10.205 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:10.206 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:10.207 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:10.207 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:10.212 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2020-02-05T16:58:10.212 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2020-02-05T16:58:10.422 INFO:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2020-02-05T16:58:10.529 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2020-02-05T16:58:10.632 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:10.633 INFO:tasks.ceph:{} 2020-02-05T16:58:10.633 INFO:tasks.ceph:osd.1 2020-02-05T16:58:10.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:58:10.633 INFO:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-02-05T16:58:11.226 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:11.227 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:11.228 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:11.228 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:11.231 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2020-02-05T16:58:11.231 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2020-02-05T16:58:11.391 INFO:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2020-02-05T16:58:11.490 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2020-02-05T16:58:11.590 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:11.590 INFO:tasks.ceph:{} 2020-02-05T16:58:11.590 INFO:tasks.ceph:osd.2 2020-02-05T16:58:11.591 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:58:11.591 INFO:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-02-05T16:58:12.171 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:12.171 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:12.171 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:12.171 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:12.172 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:12.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2020-02-05T16:58:12.177 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2020-02-05T16:58:12.392 INFO:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2020-02-05T16:58:12.490 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2020-02-05T16:58:12.591 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:12.592 INFO:tasks.ceph:{} 2020-02-05T16:58:12.592 INFO:tasks.ceph:osd.3 2020-02-05T16:58:12.592 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com 2020-02-05T16:58:12.592 INFO:teuthology.orchestra.run.smithi008:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-02-05T16:58:13.154 INFO:teuthology.orchestra.run.smithi008.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:13.154 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout: = reflink=1 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:13.155 INFO:teuthology.orchestra.run.smithi008.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:13.156 INFO:teuthology.orchestra.run.smithi008.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:13.160 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi008.front.sepia.ceph.com -o noatime 2020-02-05T16:58:13.160 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2020-02-05T16:58:13.330 INFO:teuthology.orchestra.run.smithi008:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2020-02-05T16:58:13.427 INFO:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:13.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.584+0000 7fcacbf62ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2020-02-05T16:58:13.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.586+0000 7fcacbf62ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2020-02-05T16:58:13.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.586+0000 7fcacbf62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-02-05T16:58:13.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.586+0000 7fcacbf62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-02-05T16:58:13.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.586+0000 7fcacbf62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2020-02-05T16:58:13.599 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:13.598+0000 7fcacbf62ec0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2020-02-05T16:58:15.973 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2020-02-05T16:58:16.076 INFO:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:16.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.206+0000 7f45ebe06ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2020-02-05T16:58:16.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.207+0000 7f45ebe06ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2020-02-05T16:58:16.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.207+0000 7f45ebe06ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-02-05T16:58:16.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.207+0000 7f45ebe06ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-02-05T16:58:16.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.207+0000 7f45ebe06ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2020-02-05T16:58:16.212 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:16.211+0000 7f45ebe06ec0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2020-02-05T16:58:18.586 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2020-02-05T16:58:18.687 INFO:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:18.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.817+0000 7fb6a52c9ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2020-02-05T16:58:18.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.817+0000 7fb6a52c9ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2020-02-05T16:58:18.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.818+0000 7fb6a52c9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-02-05T16:58:18.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.818+0000 7fb6a52c9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-02-05T16:58:18.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.818+0000 7fb6a52c9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2020-02-05T16:58:18.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:18.821+0000 7fb6a52c9ec0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2020-02-05T16:58:21.198 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2020-02-05T16:58:21.298 INFO:teuthology.orchestra.run.smithi008:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:21.431 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.430+0000 7f3294542ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2020-02-05T16:58:21.433 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.430+0000 7f3294542ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2020-02-05T16:58:21.433 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.431+0000 7f3294542ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-02-05T16:58:21.433 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.431+0000 7f3294542ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-02-05T16:58:21.433 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.431+0000 7f3294542ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2020-02-05T16:58:21.438 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:21.434+0000 7f3294542ec0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2020-02-05T16:58:23.813 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2020-02-05T16:58:23.910 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2020-02-05T16:58:24.014 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:24.014 INFO:tasks.ceph:{} 2020-02-05T16:58:24.014 INFO:tasks.ceph:osd.4 2020-02-05T16:58:24.014 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:58:24.014 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:24.581 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:24.582 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:24.582 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:24.582 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:24.582 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:24.587 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-02-05T16:58:24.587 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2020-02-05T16:58:24.770 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2020-02-05T16:58:24.870 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2020-02-05T16:58:24.964 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:24.964 INFO:tasks.ceph:{} 2020-02-05T16:58:24.965 INFO:tasks.ceph:osd.5 2020-02-05T16:58:24.965 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:58:24.965 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2020-02-05T16:58:25.537 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:25.537 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:25.537 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:25.537 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:25.538 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:25.544 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-02-05T16:58:25.544 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2020-02-05T16:58:25.706 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2020-02-05T16:58:25.804 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2020-02-05T16:58:25.898 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:25.898 INFO:tasks.ceph:{} 2020-02-05T16:58:25.898 INFO:tasks.ceph:osd.6 2020-02-05T16:58:25.898 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:58:25.898 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2020-02-05T16:58:26.473 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:26.473 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:26.473 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:26.473 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 2020-02-05T16:58:26.473 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:26.474 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:26.474 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:26.474 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:26.474 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:26.474 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:26.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-02-05T16:58:26.481 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2020-02-05T16:58:26.673 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2020-02-05T16:58:26.768 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2020-02-05T16:58:26.865 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2020-02-05T16:58:26.865 INFO:tasks.ceph:{} 2020-02-05T16:58:26.866 INFO:tasks.ceph:osd.7 2020-02-05T16:58:26.866 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2020-02-05T16:58:26.866 INFO:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2020-02-05T16:58:27.416 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2020-02-05T16:58:27.417 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2020-02-05T16:58:27.418 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=11443, version=2 2020-02-05T16:58:27.418 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2020-02-05T16:58:27.418 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2020-02-05T16:58:27.423 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2020-02-05T16:58:27.423 INFO:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2020-02-05T16:58:27.587 INFO:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2020-02-05T16:58:27.684 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:27.840 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.838+0000 7f6ccf6ecec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2020-02-05T16:58:27.841 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.838+0000 7f6ccf6ecec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2020-02-05T16:58:27.841 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.839+0000 7f6ccf6ecec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-02-05T16:58:27.841 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.839+0000 7f6ccf6ecec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-02-05T16:58:27.841 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.839+0000 7f6ccf6ecec0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2020-02-05T16:58:27.847 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:27.845+0000 7f6ccf6ecec0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2020-02-05T16:58:30.227 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2020-02-05T16:58:30.325 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:30.454 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.452+0000 7f05f3779ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2020-02-05T16:58:30.455 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.453+0000 7f05f3779ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2020-02-05T16:58:30.455 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.453+0000 7f05f3779ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-02-05T16:58:30.455 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.453+0000 7f05f3779ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-02-05T16:58:30.455 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.453+0000 7f05f3779ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2020-02-05T16:58:30.458 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:30.456+0000 7f05f3779ec0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2020-02-05T16:58:32.832 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2020-02-05T16:58:32.929 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:33.055 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.053+0000 7f0b50d00ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2020-02-05T16:58:33.056 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.054+0000 7f0b50d00ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2020-02-05T16:58:33.056 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.054+0000 7f0b50d00ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-02-05T16:58:33.056 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.054+0000 7f0b50d00ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-02-05T16:58:33.056 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.054+0000 7f0b50d00ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2020-02-05T16:58:33.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:33.057+0000 7f0b50d00ec0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2020-02-05T16:58:35.434 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2020-02-05T16:58:35.527 INFO:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:35.551 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:35.557 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T16:58:35.649 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.647+0000 7f39d2785ec0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2020-02-05T16:58:35.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.648+0000 7f39d2785ec0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2020-02-05T16:58:35.650 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.648+0000 7f39d2785ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-02-05T16:58:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.648+0000 7f39d2785ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-02-05T16:58:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.648+0000 7f39d2785ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2020-02-05T16:58:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:35.651+0000 7f39d2785ec0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2020-02-05T16:58:38.026 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2020-02-05T16:58:38.116 INFO:tasks.ceph:Reading keys from all nodes... 2020-02-05T16:58:38.116 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:38.176 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.TOaor740Nh 2020-02-05T16:58:38.176 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.TOaor740Nh 2020-02-05T16:58:38.277 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.TOaor740Nh 2020-02-05T16:58:38.440 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.TOaor740Nh is 56B 2020-02-05T16:58:38.502 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.TOaor740Nh 2020-02-05T16:58:38.556 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:38.612 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.psjNWfZNdD 2020-02-05T16:58:38.614 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.psjNWfZNdD 2020-02-05T16:58:38.719 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.psjNWfZNdD 2020-02-05T16:58:38.873 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.psjNWfZNdD is 56B 2020-02-05T16:58:38.933 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.psjNWfZNdD 2020-02-05T16:58:38.988 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:39.042 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.9ob2qFkvlL 2020-02-05T16:58:39.043 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.9ob2qFkvlL 2020-02-05T16:58:39.143 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.9ob2qFkvlL 2020-02-05T16:58:39.308 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.9ob2qFkvlL is 56B 2020-02-05T16:58:39.371 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.9ob2qFkvlL 2020-02-05T16:58:39.429 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:39.483 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.NA286uSYFh 2020-02-05T16:58:39.483 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.NA286uSYFh 2020-02-05T16:58:39.581 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.NA286uSYFh 2020-02-05T16:58:39.740 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.NA286uSYFh is 56B 2020-02-05T16:58:39.801 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.NA286uSYFh 2020-02-05T16:58:39.853 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:39.906 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.gpaNpb9Lzb 2020-02-05T16:58:39.906 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.gpaNpb9Lzb 2020-02-05T16:58:40.005 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.gpaNpb9Lzb 2020-02-05T16:58:40.166 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.gpaNpb9Lzb is 56B 2020-02-05T16:58:40.231 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.gpaNpb9Lzb 2020-02-05T16:58:40.285 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:40.342 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.Ln6rOT2czT 2020-02-05T16:58:40.342 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.Ln6rOT2czT 2020-02-05T16:58:40.442 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.Ln6rOT2czT 2020-02-05T16:58:40.594 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.Ln6rOT2czT is 56B 2020-02-05T16:58:40.659 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.Ln6rOT2czT 2020-02-05T16:58:40.714 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:40.769 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.9Wpuj5Kh4A 2020-02-05T16:58:40.770 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.9Wpuj5Kh4A 2020-02-05T16:58:40.863 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.9Wpuj5Kh4A 2020-02-05T16:58:41.008 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.9Wpuj5Kh4A is 56B 2020-02-05T16:58:41.066 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.9Wpuj5Kh4A 2020-02-05T16:58:41.117 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:41.168 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.E2mNCp9j3G 2020-02-05T16:58:41.168 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.E2mNCp9j3G 2020-02-05T16:58:41.264 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.E2mNCp9j3G 2020-02-05T16:58:41.416 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.E2mNCp9j3G is 56B 2020-02-05T16:58:41.474 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.E2mNCp9j3G 2020-02-05T16:58:41.528 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:41.577 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.DoCMnBhnbc 2020-02-05T16:58:41.578 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.DoCMnBhnbc 2020-02-05T16:58:41.669 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.DoCMnBhnbc 2020-02-05T16:58:41.816 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.DoCMnBhnbc is 56B 2020-02-05T16:58:41.875 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.DoCMnBhnbc 2020-02-05T16:58:41.926 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:41.978 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.Q0htz8V7I1 2020-02-05T16:58:41.979 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.Q0htz8V7I1 2020-02-05T16:58:42.072 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.Q0htz8V7I1 2020-02-05T16:58:42.217 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.Q0htz8V7I1 is 56B 2020-02-05T16:58:42.275 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.Q0htz8V7I1 2020-02-05T16:58:42.327 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:42.381 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.NVtUSeTwHS 2020-02-05T16:58:42.382 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.NVtUSeTwHS 2020-02-05T16:58:42.477 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.NVtUSeTwHS 2020-02-05T16:58:42.620 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.NVtUSeTwHS is 56B 2020-02-05T16:58:42.678 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.NVtUSeTwHS 2020-02-05T16:58:42.728 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:42.780 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.1RdnNJg3h3 2020-02-05T16:58:42.781 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.1RdnNJg3h3 2020-02-05T16:58:42.874 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.1RdnNJg3h3 2020-02-05T16:58:43.017 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.1RdnNJg3h3 is 56B 2020-02-05T16:58:43.074 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.1RdnNJg3h3 2020-02-05T16:58:43.186 DEBUG:teuthology.orchestra.remote:smithi008:/etc/ceph/ceph.client.0.keyring is 59B 2020-02-05T16:58:43.250 INFO:tasks.ceph:Adding keys to all mons... 2020-02-05T16:58:43.250 INFO:teuthology.orchestra.run.smithi008:> sudo tee -a /etc/ceph/ceph.keyring 2020-02-05T16:58:43.255 INFO:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2020-02-05T16:58:43.344 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr.y] 2020-02-05T16:58:43.344 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAg9DpexDecIhAAVPcSFPSPfglqJm06OjzpBw== 2020-02-05T16:58:43.344 INFO:teuthology.orchestra.run.smithi008.stdout:[mds.a] 2020-02-05T16:58:43.344 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAg9DpeE57yOBAAhTH4P7igKwULWdVmRLvC7w== 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.0] 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAl9DpeV67kIhAAFEapEBpOxMKS5UKDsDIkmg== 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.1] 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAo9DpeHZhaDBAA2sWiPt2TGQgJBGZmr6hNlw== 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.2] 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAq9DpeYq28MBAAFGoxfSuwSutYaCxDI/jBig== 2020-02-05T16:58:43.345 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.3] 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAt9DpeVi+tGRAAfrANtMIeXSWGjeE5NXX0CA== 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout:[mgr.x] 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAg9Dpeec0dLhAAmrJgfV/oGL6SfrlSUej0zQ== 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout:[mds.b] 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAh9DpeydNsDRAANJhh5u+coW3rE3RNAiJ+ug== 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.4] 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAz9Dpe484GMhAAd8X7SX62p+RQUlnct+dr4w== 2020-02-05T16:58:43.346 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.5] 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA29DpeIlsJGxAASWgKv22Kww1HlXxDDl9yag== 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.6] 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA59Dpe9zRCAxAAyjk43Kjq23+yo6TVyU6AVg== 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout:[osd.7] 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQA79DpeNFSpJhAAbT9XLZtTAxPbThTXPa9/PA== 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout:[client.0] 2020-02-05T16:58:43.347 INFO:teuthology.orchestra.run.smithi008.stdout: key = AQAh9DpePF1IGxAAc4ZryInkMB/w0r/AiW+ZsQ== 2020-02-05T16:58:43.348 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.y] 2020-02-05T16:58:43.348 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAg9DpexDecIhAAVPcSFPSPfglqJm06OjzpBw== 2020-02-05T16:58:43.349 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.a] 2020-02-05T16:58:43.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAg9DpeE57yOBAAhTH4P7igKwULWdVmRLvC7w== 2020-02-05T16:58:43.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2020-02-05T16:58:43.349 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAl9DpeV67kIhAAFEapEBpOxMKS5UKDsDIkmg== 2020-02-05T16:58:43.349 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAo9DpeHZhaDBAA2sWiPt2TGQgJBGZmr6hNlw== 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAq9DpeYq28MBAAFGoxfSuwSutYaCxDI/jBig== 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAt9DpeVi+tGRAAfrANtMIeXSWGjeE5NXX0CA== 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAg9Dpeec0dLhAAmrJgfV/oGL6SfrlSUej0zQ== 2020-02-05T16:58:43.350 INFO:teuthology.orchestra.run.smithi164.stdout:[mds.b] 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAh9DpeydNsDRAANJhh5u+coW3rE3RNAiJ+ug== 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAz9Dpe484GMhAAd8X7SX62p+RQUlnct+dr4w== 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQA29DpeIlsJGxAASWgKv22Kww1HlXxDDl9yag== 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.6] 2020-02-05T16:58:43.351 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQA59Dpe9zRCAxAAyjk43Kjq23+yo6TVyU6AVg== 2020-02-05T16:58:43.352 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.7] 2020-02-05T16:58:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQA79DpeNFSpJhAAbT9XLZtTAxPbThTXPa9/PA== 2020-02-05T16:58:43.352 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2020-02-05T16:58:43.352 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQAh9DpePF1IGxAAc4ZryInkMB/w0r/AiW+ZsQ== 2020-02-05T16:58:43.353 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-02-05T16:58:43.358 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-02-05T16:58:43.490 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.495 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.631 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.636 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.772 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.778 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.907 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:43.913 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.048 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.053 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.186 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-02-05T16:58:44.192 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2020-02-05T16:58:44.328 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.333 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.466 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.471 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.603 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.609 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.742 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.747 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.882 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:44.887 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2020-02-05T16:58:45.017 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-02-05T16:58:45.023 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2020-02-05T16:58:45.155 INFO:tasks.ceph:Running mkfs on mon nodes... 2020-02-05T16:58:45.155 INFO:teuthology.orchestra.run.smithi008:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2020-02-05T16:58:45.251 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-02-05T16:58:45.422 INFO:teuthology.orchestra.run.smithi008:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2020-02-05T16:58:45.521 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2020-02-05T16:58:45.618 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-02-05T16:58:45.772 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2020-02-05T16:58:45.865 INFO:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2020-02-05T16:58:45.958 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2020-02-05T16:58:46.123 INFO:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2020-02-05T16:58:46.225 INFO:teuthology.orchestra.run.smithi008:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:46.231 INFO:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2020-02-05T16:58:46.286 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2020-02-05T16:58:46.286 INFO:tasks.ceph.mon.a:Restarting daemon 2020-02-05T16:58:46.287 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2020-02-05T16:58:46.293 INFO:tasks.ceph.mon.a:Started 2020-02-05T16:58:46.293 INFO:tasks.ceph.mon.b:Restarting daemon 2020-02-05T16:58:46.293 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2020-02-05T16:58:46.299 INFO:tasks.ceph.mon.b:Started 2020-02-05T16:58:46.300 INFO:tasks.ceph.mon.c:Restarting daemon 2020-02-05T16:58:46.300 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2020-02-05T16:58:46.306 INFO:tasks.ceph.mon.c:Started 2020-02-05T16:58:46.306 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2020-02-05T16:58:46.306 INFO:tasks.ceph.mgr.y:Restarting daemon 2020-02-05T16:58:46.306 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2020-02-05T16:58:46.312 INFO:tasks.ceph.mgr.y:Started 2020-02-05T16:58:46.312 INFO:tasks.ceph.mgr.x:Restarting daemon 2020-02-05T16:58:46.313 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2020-02-05T16:58:46.316 INFO:tasks.ceph.mgr.x:Started 2020-02-05T16:58:46.316 INFO:tasks.ceph:Setting crush tunables to default 2020-02-05T16:58:46.317 INFO:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd crush tunables default 2020-02-05T16:58:51.582 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.495+0000 7fa108e96700 1 Processor -- start 2020-02-05T16:58:51.582 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.495+0000 7fa108e96700 1 -- start start 2020-02-05T16:58:51.582 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa108e96700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:51.582 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa108e96700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa108e96700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040aa980 con 0x7fa1040ace50 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa108e96700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa104148c70 con 0x7fa1040abd50 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa108e96700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa104148f10 con 0x7fa104145c20 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa103fff700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa1040ace50 0x7fa1041491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:57674/0 (socket says 172.21.15.8:57674) 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa103fff700 1 -- 172.21.15.8:0/1877331462 learned_addr learned my addr 172.21.15.8:0/1877331462 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.583 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.497+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.498+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-02-05T16:58:51.584 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.498+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.498+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.498+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.699+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.700+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.700+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.700+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:46.700+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2020-02-05T16:58:51.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.100+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.101+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.101+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.101+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.101+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.102+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2020-02-05T16:58:51.587 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.102+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.102+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.102+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.102+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.903+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.903+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-02-05T16:58:51.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2020-02-05T16:58:51.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:47.904+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2020-02-05T16:58:51.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:49.507+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.569+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710989883 0 0) 0x7fa0f40010d0 con 0x7fa1040ace50 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.569+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec003230 con 0x7fa1040ace50 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 304990926 0 0) 0x7fa0ec003230 con 0x7fa1040ace50 2020-02-05T16:58:51.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 msgr2=0x7fa104146050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104145c20 0x7fa104146050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 msgr2=0x7fa1040aa430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 --2- 172.21.15.8:0/1877331462 >> v2:172.21.15.164:3300/0 conn(0x7fa1040abd50 0x7fa1040aa430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1041454a0 con 0x7fa1040ace50 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.1 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fa0f4001980 con 0x7fa1040ace50 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 >> v1:172.21.15.164:6789/0 conn(0x7fa1040ace50 legacy=0x7fa1041491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:51.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0ec003910 0x7fa0ec003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec004f20 0x7fa0ec00b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104148c70 con 0x7fa0ec003910 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa104148f10 con 0x7fa0ec004f20 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.570+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec00bd40 con 0x7fa0ec0045a0 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.571+0000 7fa103fff700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0ec003910 0x7fa0ec003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.571+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.571+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec004f20 0x7fa0ec00b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec004f20 msgr2=0x7fa0ec00b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa0ec004f20 0x7fa0ec00b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0ec003910 msgr2=0x7fa0ec003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0ec003910 0x7fa0ec003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 -- 172.21.15.8:0/1877331462 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec00bf60 con 0x7fa0ec0045a0 2020-02-05T16:58:51.594 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.572+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.573+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.573+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).ready entity=mon.2 client_cookie=73d083fc2dd1d9e2 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f804be00 con 0x7fa0ec0045a0 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa1037fe700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f8054790 con 0x7fa0ec0045a0 2020-02-05T16:58:51.595 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.574+0000 7fa102ffd700 1 -- 172.21.15.8:0/1877331462 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f80526b0 con 0x7fa0ec0045a0 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.575+0000 7fa108e96700 1 -- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 msgr2=0x7fa0ec0049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.575+0000 7fa108e96700 1 --2- 172.21.15.8:0/1877331462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0ec0045a0 0x7fa0ec0049d0 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7fa0f8002f30 tx=0x7fa0f804fca0).stop 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.575+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1877331462 reap_dead start 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.576+0000 7fa108e96700 1 -- 172.21.15.8:0/1877331462 shutdown_connections 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.576+0000 7fa108e96700 1 -- 172.21.15.8:0/1877331462 >> 172.21.15.8:0/1877331462 conn(0x7fa104061510 msgr2=0x7fa1040ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.576+0000 7fa108e96700 1 -- 172.21.15.8:0/1877331462 shutdown_connections 2020-02-05T16:58:51.596 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.576+0000 7fa108e96700 1 -- 172.21.15.8:0/1877331462 wait complete. 2020-02-05T16:58:51.599 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.581+0000 7fa108e96700 1 Processor -- start 2020-02-05T16:58:51.599 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.581+0000 7fa108e96700 1 -- start start 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.582+0000 7fa108e96700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104063b10 0x7fa104063f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.582+0000 7fa108e96700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa104130310 0x7fa104130740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.582+0000 7fa108e96700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104064490 con 0x7fa104062b10 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.582+0000 7fa108e96700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa104063020 con 0x7fa104130310 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.582+0000 7fa108e96700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1040632c0 con 0x7fa104063b10 2020-02-05T16:58:51.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.583+0000 7fa103fff700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa104062b10 0x7fa104065430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:57718/0 (socket says 172.21.15.8:57718) 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.583+0000 7fa10a0f8700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa104130310 0x7fa104130740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.583+0000 7fa103fff700 1 -- 172.21.15.8:0/1359160155 learned_addr learned my addr 172.21.15.8:0/1359160155 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1359160155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104063b10 msgr2=0x7fa104063f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104063b10 0x7fa104063f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1359160155 >> v1:172.21.15.164:6789/0 conn(0x7fa104062b10 legacy=0x7fa104065430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 -- 172.21.15.8:0/1359160155 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104065e40 con 0x7fa104130310 2020-02-05T16:58:51.601 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> v2:172.21.15.8:3300/0 conn(0x7fa104130310 0x7fa104130740 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> v2:172.21.15.8:3300/0 conn(0x7fa104130310 0x7fa104130740 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.584+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0fc0128f0 con 0x7fa104130310 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.602 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0fc00e070 con 0x7fa104130310 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0fc00d460 con 0x7fa104130310 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104130c90 con 0x7fa104130310 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa104131180 con 0x7fa104130310 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.586+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 1) v1 ==== 349+0+0 (secure 0 0 0) 0x7fa0fc00da00 con 0x7fa104130310 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.587+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.587+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa0fc009980 con 0x7fa104130310 2020-02-05T16:58:51.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.588+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104056640 con 0x7fa104130310 2020-02-05T16:58:51.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.592+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.592+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7fa104065e40 con 0x7fa104130310 2020-02-05T16:58:51.705 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.703+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fa104065c30 con 0x7fa104130310 2020-02-05T16:58:51.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.708+0000 7fa10a0f8700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:51.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.708+0000 7fa100ff9700 1 -- 172.21.15.8:0/1359160155 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fa0fc009bc0 con 0x7fa104130310 2020-02-05T16:58:51.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 msgr2=0x7fa104130740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:51.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7fa1040aa1e0 tx=0x7fa0fc010f00).stop 2020-02-05T16:58:51.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 shutdown_connections 2020-02-05T16:58:51.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa104130310 0x7fa104130740 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:51.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 --2- 172.21.15.8:0/1359160155 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa104063b10 0x7fa104063f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:51.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 >> 172.21.15.8:0/1359160155 conn(0x7fa1040ac680 msgr2=0x7fa10405f110 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:51.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 shutdown_connections 2020-02-05T16:58:51.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:58:51.710+0000 7fa108e96700 1 -- 172.21.15.8:0/1359160155 wait complete. 2020-02-05T16:58:51.715 INFO:teuthology.orchestra.run.smithi008.stderr:adjusted tunables profile to default 2020-02-05T16:58:51.731 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2020-02-05T16:58:51.731 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:51.777 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.e1BwtXF6Xf 2020-02-05T16:58:51.778 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.e1BwtXF6Xf 2020-02-05T16:58:51.871 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.e1BwtXF6Xf 2020-02-05T16:58:52.014 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.e1BwtXF6Xf is 37B 2020-02-05T16:58:52.073 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.e1BwtXF6Xf 2020-02-05T16:58:52.124 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:52.174 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.Tvvh5oTQQh 2020-02-05T16:58:52.174 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.Tvvh5oTQQh 2020-02-05T16:58:52.273 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.Tvvh5oTQQh 2020-02-05T16:58:52.414 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.Tvvh5oTQQh is 37B 2020-02-05T16:58:52.471 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.Tvvh5oTQQh 2020-02-05T16:58:52.523 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:52.574 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.MSM2VnqSYE 2020-02-05T16:58:52.574 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.MSM2VnqSYE 2020-02-05T16:58:52.660 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.MSM2VnqSYE 2020-02-05T16:58:52.804 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.MSM2VnqSYE is 37B 2020-02-05T16:58:52.867 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.MSM2VnqSYE 2020-02-05T16:58:52.918 INFO:teuthology.orchestra.run.smithi008:> mktemp 2020-02-05T16:58:52.967 INFO:teuthology.orchestra.run.smithi008.stdout:/tmp/tmp.ngMz5ArHPS 2020-02-05T16:58:52.967 INFO:teuthology.orchestra.run.smithi008:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.ngMz5ArHPS 2020-02-05T16:58:53.064 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 0666 /tmp/tmp.ngMz5ArHPS 2020-02-05T16:58:53.212 DEBUG:teuthology.orchestra.remote:smithi008:/tmp/tmp.ngMz5ArHPS is 37B 2020-02-05T16:58:53.253 INFO:teuthology.orchestra.run.smithi008:> rm -fr /tmp/tmp.ngMz5ArHPS 2020-02-05T16:58:53.305 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:53.359 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.KXuEjTYGxP 2020-02-05T16:58:53.359 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.KXuEjTYGxP 2020-02-05T16:58:53.447 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.KXuEjTYGxP 2020-02-05T16:58:53.580 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.KXuEjTYGxP is 37B 2020-02-05T16:58:53.634 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.KXuEjTYGxP 2020-02-05T16:58:53.686 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:53.735 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.c0iD5J8Wur 2020-02-05T16:58:53.736 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.c0iD5J8Wur 2020-02-05T16:58:53.825 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.c0iD5J8Wur 2020-02-05T16:58:53.964 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.c0iD5J8Wur is 37B 2020-02-05T16:58:54.029 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.c0iD5J8Wur 2020-02-05T16:58:54.079 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:54.168 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.5yJ92iMHj0 2020-02-05T16:58:54.168 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.5yJ92iMHj0 2020-02-05T16:58:54.252 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.5yJ92iMHj0 2020-02-05T16:58:54.386 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.5yJ92iMHj0 is 37B 2020-02-05T16:58:54.442 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.5yJ92iMHj0 2020-02-05T16:58:54.492 INFO:teuthology.orchestra.run.smithi164:> mktemp 2020-02-05T16:58:54.541 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.i8MZTUenot 2020-02-05T16:58:54.542 INFO:teuthology.orchestra.run.smithi164:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.i8MZTUenot 2020-02-05T16:58:54.626 INFO:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.i8MZTUenot 2020-02-05T16:58:54.758 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.i8MZTUenot is 37B 2020-02-05T16:58:54.811 INFO:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.i8MZTUenot 2020-02-05T16:58:54.858 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 2bc6f3a9-039a-4231-9455-307d8891ced6 0 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.048+0000 7f3c576c7700 1 Processor -- start 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.048+0000 7f3c576c7700 1 -- start start 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c576c7700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c5008f240 0x7f3c500908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c576c7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c500a9e90 0x7f3c500a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c576c7700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c500aab90 con 0x7f3c5008fff0 2020-02-05T16:58:55.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c576c7700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c5008f670 con 0x7f3c5008f240 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c576c7700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c50090650 con 0x7f3c500a9e90 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c55463700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c5008f240 0x7f3c500908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c55463700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c5008f240 0x7f3c500908f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.164:53016/0 (socket says 172.21.15.164:53016) 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c55463700 1 -- 172.21.15.164:0/2434210877 learned_addr learned my addr 172.21.15.164:0/2434210877 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.049+0000 7f3c56465700 1 --2- 172.21.15.164:0/2434210877 >> v2:172.21.15.164:3300/0 conn(0x7f3c500a9e90 0x7f3c500a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 -- 172.21.15.164:0/2434210877 >> v2:172.21.15.164:3300/0 conn(0x7f3c500a9e90 msgr2=0x7f3c500a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:55.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> v2:172.21.15.164:3300/0 conn(0x7f3c500a9e90 0x7f3c500a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 -- 172.21.15.164:0/2434210877 >> v1:172.21.15.8:6789/0 conn(0x7f3c5008fff0 legacy=0x7f3c500a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 -- 172.21.15.164:0/2434210877 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c500a86e0 con 0x7f3c5008f240 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c54c62700 1 -- 172.21.15.164:0/2434210877 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031992819 0 0) 0x7f3c500aab90 con 0x7f3c5008fff0 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> v2:172.21.15.8:3300/0 conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> v2:172.21.15.8:3300/0 conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.050+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).ready entity=mon.0 client_cookie=c786fd4ec0a7c158 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c54c62700 1 -- 172.21.15.164:0/2434210877 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c40011760 con 0x7f3c5008f240 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c55463700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=THROTTLE_DONE pgs=10 cs=0 l=1 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c54c62700 1 -- 172.21.15.164:0/2434210877 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40008d30 con 0x7f3c5008f240 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c54c62700 1 -- 172.21.15.164:0/2434210877 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c40011d40 con 0x7f3c5008f240 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c576c7700 1 -- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 msgr2=0x7f3c500908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.051+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 secure :-1 s=READY pgs=10 cs=0 l=1 rx=0x7f3c500a9620 tx=0x7f3c4000ff30).stop 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.052+0000 7f3c576c7700 1 -- 172.21.15.164:0/2434210877 shutdown_connections 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.052+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2434210877 >> v2:172.21.15.164:3300/0 conn(0x7f3c500a9e90 0x7f3c500a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.052+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2434210877 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c5008f240 0x7f3c500908f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.052+0000 7f3c576c7700 1 -- 172.21.15.164:0/2434210877 >> 172.21.15.164:0/2434210877 conn(0x7f3c500600f0 msgr2=0x7f3c5005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.052+0000 7f3c576c7700 1 -- 172.21.15.164:0/2434210877 shutdown_connections 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.053+0000 7f3c576c7700 1 -- 172.21.15.164:0/2434210877 wait complete. 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.055+0000 7f3c576c7700 1 Processor -- start 2020-02-05T16:58:55.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.055+0000 7f3c576c7700 1 -- start start 2020-02-05T16:58:55.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c576c7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c50061010 0x7f3c50062260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c576c7700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c576c7700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5009dfe0 con 0x7f3c500627b0 2020-02-05T16:58:55.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c576c7700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c50062d30 con 0x7f3c50061010 2020-02-05T16:58:55.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c576c7700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c50062fd0 con 0x7f3c500a01b0 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c55463700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3c500627b0 0x7f3c5009d7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39904/0 (socket says 172.21.15.164:39904) 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c55463700 1 -- 172.21.15.164:0/2474779870 learned_addr learned my addr 172.21.15.164:0/2474779870 (peer_addr_for_me v1:172.21.15.164:0/0) 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.056+0000 7f3c55c64700 1 --2- 172.21.15.164:0/2474779870 >> v2:172.21.15.164:3300/0 conn(0x7f3c50061010 0x7f3c50062260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103527928 0 0) 0x7f3c5009dfe0 con 0x7f3c500627b0 2020-02-05T16:58:55.065 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38003230 con 0x7f3c500627b0 2020-02-05T16:58:55.067 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.1 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3817271364 0 0) 0x7f3c38003230 con 0x7f3c500627b0 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 -- 172.21.15.164:0/2474779870 >> v2:172.21.15.164:3300/0 conn(0x7f3c50061010 msgr2=0x7f3c50062260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> v2:172.21.15.164:3300/0 conn(0x7f3c50061010 0x7f3c50062260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 -- 172.21.15.164:0/2474779870 >> v1:172.21.15.164:6789/0 conn(0x7f3c500627b0 legacy=0x7f3c5009d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 -- 172.21.15.164:0/2474779870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c500a0b30 con 0x7f3c500a01b0 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.068 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.057+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c400085f0 con 0x7f3c500a01b0 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c40011960 con 0x7f3c500a01b0 2020-02-05T16:58:55.069 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5013d530 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c480056c0 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5013d900 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 349+0+0 (secure 0 0 0) 0x7f3c48005cf0 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.058+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.059+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f3c48012ca0 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.059+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c50056640 con 0x7f3c500a01b0 2020-02-05T16:58:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.062+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.062+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7f3c48007160 con 0x7f3c500a01b0 2020-02-05T16:58:55.163 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.161+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2bc6f3a9-039a-4231-9455-307d8891ced6", "id": 0} v 0) v1 -- 0x7f3c50068f50 con 0x7f3c500a01b0 2020-02-05T16:58:55.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.168+0000 7f3c56465700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.168+0000 7f3c467fc700 1 -- 172.21.15.164:0/2474779870 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2bc6f3a9-039a-4231-9455-307d8891ced6", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c48007380 con 0x7f3c500a01b0 2020-02-05T16:58:55.170 INFO:teuthology.orchestra.run.smithi164.stdout:0 2020-02-05T16:58:55.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.169+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 msgr2=0x7f3c500a05e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.169+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7f3c40008830 tx=0x7f3c40010380).stop 2020-02-05T16:58:55.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 shutdown_connections 2020-02-05T16:58:55.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2474779870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c500a01b0 0x7f3c500a05e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 --2- 172.21.15.164:0/2474779870 >> v2:172.21.15.164:3300/0 conn(0x7f3c50061010 0x7f3c50062260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 >> 172.21.15.164:0/2474779870 conn(0x7f3c500602c0 msgr2=0x7f3c5007cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 shutdown_connections 2020-02-05T16:58:55.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.170+0000 7f3c576c7700 1 -- 172.21.15.164:0/2474779870 wait complete. 2020-02-05T16:58:55.190 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 61d816aa-3e9c-45f3-97e1-ca9c5510305f 1 2020-02-05T16:58:55.385 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.372+0000 7f725e1d3700 1 Processor -- start 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.372+0000 7f725e1d3700 1 -- start start 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.372+0000 7f725e1d3700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725808fff0 0x7f725808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.372+0000 7f725e1d3700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72580a9e90 0x7f72580a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725e1d3700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580aab90 con 0x7f72580908f0 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725e1d3700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7258090d20 con 0x7f72580a9e90 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725e1d3700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7258090650 con 0x7f725808fff0 2020-02-05T16:58:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72580a9e90 0x7f72580a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f72580a9e90 0x7f72580a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55752/0 (socket says 172.21.15.164:55752) 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 -- 172.21.15.164:0/428549527 learned_addr learned my addr 172.21.15.164:0/428549527 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f7257fff700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725808fff0 0x7f725808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 -- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725808fff0 msgr2=0x7f725808f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725808fff0 0x7f725808f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 -- 172.21.15.164:0/428549527 >> v1:172.21.15.8:6789/0 conn(0x7f72580908f0 legacy=0x7f72580a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.373+0000 7f725cf71700 1 -- 172.21.15.164:0/428549527 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72580a86e0 con 0x7f72580a9e90 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> v2:172.21.15.164:3300/0 conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7250008c90 tx=0x7f72500102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> v2:172.21.15.164:3300/0 conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7250008c90 tx=0x7f72500102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f7250008c90 tx=0x7f72500102b0).ready entity=mon.1 client_cookie=b1ecc3e016af0272 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f7250008c90 tx=0x7f72500102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f7250008c90 tx=0x7f72500102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f7256ffd700 1 -- 172.21.15.164:0/428549527 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f725000a350 con 0x7f72580a9e90 2020-02-05T16:58:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725cf71700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f7250008c90 tx=0x7f72500102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f7256ffd700 1 -- 172.21.15.164:0/428549527 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7250011c30 con 0x7f72580a9e90 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f7256ffd700 1 -- 172.21.15.164:0/428549527 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f725000a930 con 0x7f72580a9e90 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725e1d3700 1 -- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 msgr2=0x7f72580a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.374+0000 7f725e1d3700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f7250008c90 tx=0x7f72500102b0).stop 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 -- 172.21.15.164:0/428549527 shutdown_connections 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72580a9e90 0x7f72580a8190 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 --2- 172.21.15.164:0/428549527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725808fff0 0x7f725808f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 -- 172.21.15.164:0/428549527 >> 172.21.15.164:0/428549527 conn(0x7f72580600f0 msgr2=0x7f725805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 -- 172.21.15.164:0/428549527 shutdown_connections 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.375+0000 7f725e1d3700 1 -- 172.21.15.164:0/428549527 wait complete. 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.377+0000 7f725e1d3700 1 Processor -- start 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.377+0000 7f725e1d3700 1 -- start start 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725e1d3700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725e1d3700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f7258065ee0 0x7f7258066310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.390 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725e1d3700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72580a3280 con 0x7f72580a46e0 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725e1d3700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72580a2d80 con 0x7f7258065ee0 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725e1d3700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72580a2fe0 con 0x7f725807e0f0 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f7257fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f7257fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47554/0 (socket says 172.21.15.164:47554) 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f7257fff700 1 -- 172.21.15.164:0/3489480965 learned_addr learned my addr 172.21.15.164:0/3489480965 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.378+0000 7f725cf71700 1 --2- 172.21.15.164:0/3489480965 >> v2:172.21.15.8:3300/0 conn(0x7f7258065ee0 0x7f7258066310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 -- 172.21.15.164:0/3489480965 >> v2:172.21.15.8:3300/0 conn(0x7f7258065ee0 msgr2=0x7f7258066310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.391 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> v2:172.21.15.8:3300/0 conn(0x7f7258065ee0 0x7f7258066310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 -- 172.21.15.164:0/3489480965 >> v1:172.21.15.164:6789/0 conn(0x7f72580a46e0 legacy=0x7f725809ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858514941 0 0) 0x7f72580a3280 con 0x7f72580a46e0 2020-02-05T16:58:55.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 -- 172.21.15.164:0/3489480965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f725809e6d0 con 0x7f725807e0f0 2020-02-05T16:58:55.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.409 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.409 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.409 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.409 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.409 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.410 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7248012c40 con 0x7f725807e0f0 2020-02-05T16:58:55.410 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7248012860 con 0x7f725807e0f0 2020-02-05T16:58:55.410 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7248009580 con 0x7f725807e0f0 2020-02-05T16:58:55.417 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f725809edb0 con 0x7f725807e0f0 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.379+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725813f760 con 0x7f725807e0f0 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.380+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.380+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) v1 ==== 349+0+0 (secure 0 0 0) 0x7f7248009780 con 0x7f725807e0f0 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.380+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.380+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7f7248009a90 con 0x7f725807e0f0 2020-02-05T16:58:55.418 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.381+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7258056640 con 0x7f725807e0f0 2020-02-05T16:58:55.419 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.384+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.419 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.384+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7f724800e2f0 con 0x7f725807e0f0 2020-02-05T16:58:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.459+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.460+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mgrmap(e 2) v1 ==== 22031+0+0 (secure 0 0 0) 0x7f724800e510 con 0x7f725807e0f0 2020-02-05T16:58:55.483 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.481+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "61d816aa-3e9c-45f3-97e1-ca9c5510305f", "id": 1} v 0) v1 -- 0x7f7258068f50 con 0x7f725807e0f0 2020-02-05T16:58:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.488+0000 7f7257fff700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.488+0000 7f7254ff9700 1 -- 172.21.15.164:0/3489480965 <== mon.2 v2:172.21.15.164:3301/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "61d816aa-3e9c-45f3-97e1-ca9c5510305f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f724800ed90 con 0x7f725807e0f0 2020-02-05T16:58:55.490 INFO:teuthology.orchestra.run.smithi164.stdout:1 2020-02-05T16:58:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 msgr2=0x7f72580a2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f7248002f30 tx=0x7f7248010db0).stop 2020-02-05T16:58:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 shutdown_connections 2020-02-05T16:58:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 --2- 172.21.15.164:0/3489480965 >> v2:172.21.15.8:3300/0 conn(0x7f7258065ee0 0x7f7258066310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 --2- 172.21.15.164:0/3489480965 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f725807e0f0 0x7f72580a2830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.490+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 >> 172.21.15.164:0/3489480965 conn(0x7f725807cfc0 msgr2=0x7f725805ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.491+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 shutdown_connections 2020-02-05T16:58:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.491+0000 7f725e1d3700 1 -- 172.21.15.164:0/3489480965 wait complete. 2020-02-05T16:58:55.508 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 3ab2a8b9-2960-436d-9acd-28c9edf793c2 2 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.695+0000 7feea3d37700 1 Processor -- start 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 -- start start 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fee9c0a9e90 0x7fee9c0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0aab90 con 0x7fee9c08fff0 2020-02-05T16:58:55.704 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c08f670 con 0x7fee9c0a9e90 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.696+0000 7feea3d37700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9c090650 con 0x7fee9c08f240 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.697+0000 7feea1ad3700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.697+0000 7feea1ad3700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47578/0 (socket says 172.21.15.164:47578) 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.697+0000 7feea22d4700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fee9c08fff0 0x7fee9c0a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39938/0 (socket says 172.21.15.164:39938) 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.697+0000 7feea1ad3700 1 -- 172.21.15.164:0/3622909123 learned_addr learned my addr 172.21.15.164:0/3622909123 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.697+0000 7feea2ad5700 1 --2- 172.21.15.164:0/3622909123 >> v2:172.21.15.8:3300/0 conn(0x7fee9c0a9e90 0x7fee9c0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea12d2700 1 -- 172.21.15.164:0/3622909123 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449707792 0 0) 0x7fee9c0aab90 con 0x7fee9c08fff0 2020-02-05T16:58:55.705 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea12d2700 1 -- 172.21.15.164:0/3622909123 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee88003230 con 0x7fee9c08fff0 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 -- 172.21.15.164:0/3622909123 >> v2:172.21.15.8:3300/0 conn(0x7fee9c0a9e90 msgr2=0x7fee9c0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> v2:172.21.15.8:3300/0 conn(0x7fee9c0a9e90 0x7fee9c0a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 -- 172.21.15.164:0/3622909123 >> v1:172.21.15.164:6789/0 conn(0x7fee9c08fff0 legacy=0x7fee9c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 -- 172.21.15.164:0/3622909123 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c0a86e0 con 0x7fee9c08f240 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee8c008910 tx=0x7fee8c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee8c008910 tx=0x7fee8c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.706 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c00ff30).ready entity=mon.2 client_cookie=345ed7452b9341f6 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea12d2700 1 -- 172.21.15.164:0/3622909123 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c0117a0 con 0x7fee9c08f240 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.698+0000 7feea1ad3700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.699+0000 7feea12d2700 1 -- 172.21.15.164:0/3622909123 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c008d30 con 0x7fee9c08f240 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.699+0000 7feea12d2700 1 -- 172.21.15.164:0/3622909123 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c011d80 con 0x7fee9c08f240 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.699+0000 7feea3d37700 1 -- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 msgr2=0x7fee9c0908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.707 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.699+0000 7feea3d37700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c00ff30).stop 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 -- 172.21.15.164:0/3622909123 shutdown_connections 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 --2- 172.21.15.164:0/3622909123 >> v2:172.21.15.8:3300/0 conn(0x7fee9c0a9e90 0x7fee9c0a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 --2- 172.21.15.164:0/3622909123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee9c08f240 0x7fee9c0908f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 -- 172.21.15.164:0/3622909123 >> 172.21.15.164:0/3622909123 conn(0x7fee9c0600f0 msgr2=0x7fee9c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 -- 172.21.15.164:0/3622909123 shutdown_connections 2020-02-05T16:58:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.700+0000 7feea3d37700 1 -- 172.21.15.164:0/3622909123 wait complete. 2020-02-05T16:58:55.712 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 Processor -- start 2020-02-05T16:58:55.712 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 -- start start 2020-02-05T16:58:55.712 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 0x7fee9c0a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee9c0a0cd0 0x7fee9c09edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee9c0a1d70 con 0x7fee9c09f300 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c0a1670 con 0x7fee9c07def0 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.703+0000 7feea3d37700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee9c1230e0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee9c0a0cd0 0x7fee9c09edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee9c0a0cd0 0x7fee9c09edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55786/0 (socket says 172.21.15.164:55786) 2020-02-05T16:58:55.713 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 -- 172.21.15.164:0/2459553780 learned_addr learned my addr 172.21.15.164:0/2459553780 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea22d4700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 0x7fee9c0a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 -- 172.21.15.164:0/2459553780 >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 msgr2=0x7fee9c0a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 0x7fee9c0a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 -- 172.21.15.164:0/2459553780 >> v1:172.21.15.8:6789/0 conn(0x7fee9c09f300 legacy=0x7fee9c13f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 -- 172.21.15.164:0/2459553780 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee9c07d0b0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea22d4700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 0x7fee9c0a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.704+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.164:3300/0 conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.714 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.164:3300/0 conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c007340 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8c0119a0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.715 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8c0056c0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee9c0a18f0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.705+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c07e320 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.706+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.706+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.706+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 2) v1 ==== 22031+0+0 (secure 0 0 0) 0x7fee8c005990 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.707+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7fee8c028e50 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.716 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.707+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee9c056640 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.717 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.710+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.717 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.710+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7fee8c00ddc0 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.809+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3ab2a8b9-2960-436d-9acd-28c9edf793c2", "id": 2} v 0) v1 -- 0x7fee9c0a4870 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.819+0000 7feea1ad3700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.820+0000 7fee92ffd700 1 -- 172.21.15.164:0/2459553780 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ab2a8b9-2960-436d-9acd-28c9edf793c2", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fee8c00b240 con 0x7fee9c0a0cd0 2020-02-05T16:58:55.822 INFO:teuthology.orchestra.run.smithi164.stdout:2 2020-02-05T16:58:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 msgr2=0x7fee9c09edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fee8c008910 tx=0x7fee8c0118e0).stop 2020-02-05T16:58:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 shutdown_connections 2020-02-05T16:58:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 --2- 172.21.15.164:0/2459553780 >> v2:172.21.15.8:3300/0 conn(0x7fee9c07def0 0x7fee9c0a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 --2- 172.21.15.164:0/2459553780 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee9c0a0cd0 0x7fee9c09edb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.821+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 >> 172.21.15.164:0/2459553780 conn(0x7fee9c090e00 msgr2=0x7fee9c07d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.822+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 shutdown_connections 2020-02-05T16:58:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:55.822+0000 7feea3d37700 1 -- 172.21.15.164:0/2459553780 wait complete. 2020-02-05T16:58:55.842 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new f01e2b49-ee11-419a-92d8-6f3b52f73229 3 2020-02-05T16:58:56.049 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.040+0000 7f07da9be700 1 Processor -- start 2020-02-05T16:58:56.049 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.040+0000 7f07da9be700 1 -- start start 2020-02-05T16:58:56.049 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07da9be700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f07d408fff0 0x7f07d408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07da9be700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07d40a9e90 0x7f07d40a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07da9be700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07d40aab90 con 0x7f07d40908f0 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07da9be700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d4090d20 con 0x7f07d408fff0 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07da9be700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d4090650 con 0x7f07d40a9e90 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07d975c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07d40a9e90 0x7f07d40a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07d975c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07d40a9e90 0x7f07d40a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55796/0 (socket says 172.21.15.164:55796) 2020-02-05T16:58:56.050 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07d8f5b700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f07d408fff0 0x7f07d408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.041+0000 7f07d975c700 1 -- 172.21.15.164:0/2238900689 learned_addr learned my addr 172.21.15.164:0/2238900689 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.042+0000 7f07d975c700 1 -- 172.21.15.164:0/2238900689 >> v2:172.21.15.8:3300/0 conn(0x7f07d408fff0 msgr2=0x7f07d408f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.042+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> v2:172.21.15.8:3300/0 conn(0x7f07d408fff0 0x7f07d408f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.042+0000 7f07d975c700 1 -- 172.21.15.164:0/2238900689 >> v1:172.21.15.8:6789/0 conn(0x7f07d40908f0 legacy=0x7f07d40a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.042+0000 7f07d975c700 1 -- 172.21.15.164:0/2238900689 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d40a86e0 con 0x7f07d40a9e90 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.042+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> v2:172.21.15.164:3300/0 conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f07c4002f30 tx=0x7f07c400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.051 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> v2:172.21.15.164:3300/0 conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f07c4002f30 tx=0x7f07c400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f07c4002f30 tx=0x7f07c400ff30).ready entity=mon.1 client_cookie=eaa8c3ebb10de769 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f07c4002f30 tx=0x7f07c400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f07c4002f30 tx=0x7f07c400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d37fe700 1 -- 172.21.15.164:0/2238900689 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07c4011760 con 0x7f07d40a9e90 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d975c700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f07c4002f30 tx=0x7f07c400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d37fe700 1 -- 172.21.15.164:0/2238900689 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c4008d30 con 0x7f07d40a9e90 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07d37fe700 1 -- 172.21.15.164:0/2238900689 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07c4011d40 con 0x7f07d40a9e90 2020-02-05T16:58:56.052 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07da9be700 1 -- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 msgr2=0x7f07d40a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.043+0000 7f07da9be700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f07c4002f30 tx=0x7f07c400ff30).stop 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.044+0000 7f07da9be700 1 -- 172.21.15.164:0/2238900689 shutdown_connections 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.045+0000 7f07da9be700 1 --2- 172.21.15.164:0/2238900689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d40a9e90 0x7f07d40a8190 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.045+0000 7f07da9be700 1 --2- 172.21.15.164:0/2238900689 >> v2:172.21.15.8:3300/0 conn(0x7f07d408fff0 0x7f07d408f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.045+0000 7f07da9be700 1 -- 172.21.15.164:0/2238900689 >> 172.21.15.164:0/2238900689 conn(0x7f07d40600f0 msgr2=0x7f07d405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.045+0000 7f07da9be700 1 -- 172.21.15.164:0/2238900689 shutdown_connections 2020-02-05T16:58:56.053 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.045+0000 7f07da9be700 1 -- 172.21.15.164:0/2238900689 wait complete. 2020-02-05T16:58:56.054 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.047+0000 7f07da9be700 1 Processor -- start 2020-02-05T16:58:56.054 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.047+0000 7f07da9be700 1 -- start start 2020-02-05T16:58:56.058 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07d4135960 0x7f07d4079e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.058 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 0x7f07d40a6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.058 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 0x7f07d41383f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.058 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d40a6bd0 con 0x7f07d40a65a0 2020-02-05T16:58:56.058 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d40a7540 con 0x7f07d4135960 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.048+0000 7f07da9be700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4135d90 con 0x7f07d407a360 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d3fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 0x7f07d40a6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d8f5b700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07d4135960 0x7f07d4079e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d975c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 0x7f07d41383f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d3fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 0x7f07d40a6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47600/0 (socket says 172.21.15.164:47600) 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d3fff700 1 -- 172.21.15.164:0/3979711922 learned_addr learned my addr 172.21.15.164:0/3979711922 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.049+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.164:3300/0 conn(0x7f07d4135960 0x7f07d4079e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55800/0 (socket says 172.21.15.164:55800) 2020-02-05T16:58:56.059 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 -- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 msgr2=0x7f07d40a6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 0x7f07d40a6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 -- 172.21.15.164:0/3979711922 >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 msgr2=0x7f07d41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 0x7f07d41383f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 -- 172.21.15.164:0/3979711922 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d40a2020 con 0x7f07d4135960 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d975c700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 0x7f07d41383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.164:3300/0 conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.060 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.164:3300/0 conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.050+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07cc0127f0 con 0x7f07d4135960 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc007150 con 0x7f07d4135960 2020-02-05T16:58:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07cc012dd0 con 0x7f07d4135960 2020-02-05T16:58:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d407aae0 con 0x7f07d4135960 2020-02-05T16:58:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.051+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d4138940 con 0x7f07d4135960 2020-02-05T16:58:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.052+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.052+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 2) v1 ==== 22031+0+0 (secure 0 0 0) 0x7f07cc0056c0 con 0x7f07d4135960 2020-02-05T16:58:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.052+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.052+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d4056640 con 0x7f07d4135960 2020-02-05T16:58:56.063 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.052+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7f07cc028bf0 con 0x7f07d4135960 2020-02-05T16:58:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.056+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.056+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7f07cc00d0c0 con 0x7f07d4135960 2020-02-05T16:58:56.156 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.154+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f01e2b49-ee11-419a-92d8-6f3b52f73229", "id": 3} v 0) v1 -- 0x7f07d4068f50 con 0x7f07d4135960 2020-02-05T16:58:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.166+0000 7f07d8f5b700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.166+0000 7f07d17fa700 1 -- 172.21.15.164:0/3979711922 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f01e2b49-ee11-419a-92d8-6f3b52f73229", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f07cc008dd0 con 0x7f07d4135960 2020-02-05T16:58:56.168 INFO:teuthology.orchestra.run.smithi164.stdout:3 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.167+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 msgr2=0x7f07d4079e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.167+0000 7f07da9be700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f07cc002f30 tx=0x7f07cc010d40).stop 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 shutdown_connections 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 --2- 172.21.15.164:0/3979711922 >> v2:172.21.15.8:3300/0 conn(0x7f07d40a65a0 0x7f07d41383f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07d4135960 0x7f07d4079e10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 --2- 172.21.15.164:0/3979711922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07d407a360 0x7f07d40a6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 >> 172.21.15.164:0/3979711922 conn(0x7f07d4060050 msgr2=0x7f07d40a28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 shutdown_connections 2020-02-05T16:58:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.168+0000 7f07da9be700 1 -- 172.21.15.164:0/3979711922 wait complete. 2020-02-05T16:58:56.189 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 9cf99cf4-de20-44f7-bbae-57d8c0e93b08 4 2020-02-05T16:58:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.384+0000 7fcd957f0700 1 Processor -- start 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.384+0000 7fcd957f0700 1 -- start start 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd957f0700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 0x7fcd9008f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd957f0700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd957f0700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900aa3e0 con 0x7fcd900a7990 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd957f0700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd90090d20 con 0x7fcd9008fff0 2020-02-05T16:58:56.393 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd957f0700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd90090650 con 0x7fcd900908f0 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd8effd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd8effd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47602/0 (socket says 172.21.15.164:47602) 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd8effd700 1 -- 172.21.15.164:0/1675252622 learned_addr learned my addr 172.21.15.164:0/1675252622 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.385+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1675252622 >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 0x7fcd9008f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 -- 172.21.15.164:0/1675252622 >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 msgr2=0x7fcd9008f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 0x7fcd9008f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.394 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 -- 172.21.15.164:0/1675252622 >> v1:172.21.15.8:6789/0 conn(0x7fcd900a7990 legacy=0x7fcd900a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 -- 172.21.15.164:0/1675252622 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd900aab90 con 0x7fcd900908f0 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1675252622 >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 0x7fcd9008f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcd78008910 tx=0x7fcd7800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcd78008910 tx=0x7fcd7800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7fcd78008910 tx=0x7fcd7800ff30).ready entity=mon.2 client_cookie=fd2ef0b934c457f6 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fcd78008910 tx=0x7fcd7800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.395 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fcd78008910 tx=0x7fcd7800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=THROTTLE_DONE pgs=13 cs=0 l=1 rx=0x7fcd78008910 tx=0x7fcd7800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8e7fc700 1 -- 172.21.15.164:0/1675252622 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd78011990 con 0x7fcd900908f0 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8e7fc700 1 -- 172.21.15.164:0/1675252622 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd780115b0 con 0x7fcd900908f0 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.387+0000 7fcd957f0700 1 -- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 msgr2=0x7fcd900a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.387+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 secure :-1 s=READY pgs=13 cs=0 l=1 rx=0x7fcd78008910 tx=0x7fcd7800ff30).stop 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.386+0000 7fcd8e7fc700 1 -- 172.21.15.164:0/1675252622 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd7800a160 con 0x7fcd900908f0 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 -- 172.21.15.164:0/1675252622 shutdown_connections 2020-02-05T16:58:56.396 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1675252622 >> v2:172.21.15.164:3300/0 conn(0x7fcd9008fff0 0x7fcd9008f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1675252622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd900908f0 0x7fcd900a9e90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 -- 172.21.15.164:0/1675252622 >> 172.21.15.164:0/1675252622 conn(0x7fcd900600f0 msgr2=0x7fcd9005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 -- 172.21.15.164:0/1675252622 shutdown_connections 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.388+0000 7fcd957f0700 1 -- 172.21.15.164:0/1675252622 wait complete. 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.390+0000 7fcd957f0700 1 Processor -- start 2020-02-05T16:58:56.397 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.390+0000 7fcd957f0700 1 -- start start 2020-02-05T16:58:56.400 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd957f0700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcd900a6f40 0x7fcd90079050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.401 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd957f0700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd90078600 0x7fcd9013f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.401 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd957f0700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd900a8750 con 0x7fcd900a3da0 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd957f0700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd900a5e00 con 0x7fcd900a6f40 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd957f0700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd900a6a00 con 0x7fcd90078600 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd8effd700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcd900a3da0 0x7fcd9009e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39968/0 (socket says 172.21.15.164:39968) 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd8effd700 1 -- 172.21.15.164:0/1440429557 learned_addr learned my addr 172.21.15.164:0/1440429557 (peer_addr_for_me v1:172.21.15.164:0/0) 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> v2:172.21.15.164:3300/0 conn(0x7fcd900a6f40 0x7fcd90079050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.391+0000 7fcd8ffff700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd90078600 0x7fcd9013f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.402 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427533189 0 0) 0x7fcd900a8750 con 0x7fcd900a3da0 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd74003230 con 0x7fcd900a3da0 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd8f7fe700 1 -- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd90078600 msgr2=0x7fcd9013f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd90078600 0x7fcd9013f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd8f7fe700 1 -- 172.21.15.164:0/1440429557 >> v1:172.21.15.164:6789/0 conn(0x7fcd900a3da0 legacy=0x7fcd9009e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd8f7fe700 1 -- 172.21.15.164:0/1440429557 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9009e8d0 con 0x7fcd900a6f40 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.392+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> v2:172.21.15.164:3300/0 conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> v2:172.21.15.164:3300/0 conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd80007160 con 0x7fcd900a6f40 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd80005ab0 con 0x7fcd900a6f40 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd900a5690 con 0x7fcd900a6f40 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd800124b0 con 0x7fcd900a6f40 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.393+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd900795a0 con 0x7fcd900a6f40 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.394+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.395+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 2) v1 ==== 22031+0+0 (secure 0 0 0) 0x7fcd800127d0 con 0x7fcd900a6f40 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.395+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.395+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7fcd80028d30 con 0x7fcd900a6f40 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.395+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd90056640 con 0x7fcd900a6f40 2020-02-05T16:58:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.398+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.406 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.398+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67593 (secure 0 0 0) 0x7fcd8000cc80 con 0x7fcd900a6f40 2020-02-05T16:58:56.470 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.468+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.472 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.470+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mgrmap(e 3) v1 ==== 44173+0+0 (secure 0 0 0) 0x7fcd80012ae0 con 0x7fcd900a6f40 2020-02-05T16:58:56.474 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.470+0000 7fcd87fff700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 0x7fcd7403be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.474 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.472+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 0x7fcd7403be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.475 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.473+0000 7fcd8effd700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 0x7fcd7403be80 crc :-1 s=READY pgs=5 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.504 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.502+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9cf99cf4-de20-44f7-bbae-57d8c0e93b08", "id": 4} v 0) v1 -- 0x7fcd900600f0 con 0x7fcd900a6f40 2020-02-05T16:58:56.515 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.513+0000 7fcd8f7fe700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.515 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.513+0000 7fcd87fff700 1 -- 172.21.15.164:0/1440429557 <== mon.1 v2:172.21.15.164:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cf99cf4-de20-44f7-bbae-57d8c0e93b08", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd80011e00 con 0x7fcd900a6f40 2020-02-05T16:58:56.516 INFO:teuthology.orchestra.run.smithi164.stdout:4 2020-02-05T16:58:56.517 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 msgr2=0x7fcd7403be80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.517 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 0x7fcd7403be80 crc :-1 s=READY pgs=5 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.517 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 msgr2=0x7fcd90079050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7fcd80002f30 tx=0x7fcd80010c00).stop 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 shutdown_connections 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcd90078600 0x7fcd9013f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcd900a6f40 0x7fcd90079050 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 --2- 172.21.15.164:0/1440429557 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fcd74039a20 0x7fcd7403be80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 >> 172.21.15.164:0/1440429557 conn(0x7fcd90065ea0 msgr2=0x7fcd9007d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.518 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 shutdown_connections 2020-02-05T16:58:56.519 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.515+0000 7fcd957f0700 1 -- 172.21.15.164:0/1440429557 wait complete. 2020-02-05T16:58:56.537 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new c1ecad70-c4c6-4b35-896c-25b1d9e5a546 5 2020-02-05T16:58:56.743 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.734+0000 7fd64dba2700 1 Processor -- start 2020-02-05T16:58:56.743 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 -- start start 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd64808fff0 0x7fd64808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480908f0 0x7fd6480a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6480a7990 0x7fd6480a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd648090d20 con 0x7fd64808fff0 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd648090650 con 0x7fd6480a7990 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.735+0000 7fd64dba2700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64808ece0 con 0x7fd6480908f0 2020-02-05T16:58:56.744 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd64c940700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6480a7990 0x7fd6480a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.745 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd6477fe700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480908f0 0x7fd6480a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.745 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd64c940700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6480a7990 0x7fd6480a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55828/0 (socket says 172.21.15.164:55828) 2020-02-05T16:58:56.745 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd64c940700 1 -- 172.21.15.164:0/999443311 learned_addr learned my addr 172.21.15.164:0/999443311 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:56.746 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd647fff700 1 --2- 172.21.15.164:0/999443311 >> v2:172.21.15.8:3300/0 conn(0x7fd64808fff0 0x7fd64808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.746 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd64c940700 1 -- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480908f0 msgr2=0x7fd6480a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:56.746 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.736+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480908f0 0x7fd6480a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.746 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 -- 172.21.15.164:0/999443311 >> v2:172.21.15.8:3300/0 conn(0x7fd64808fff0 msgr2=0x7fd64808f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:56.746 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> v2:172.21.15.8:3300/0 conn(0x7fd64808fff0 0x7fd64808f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 -- 172.21.15.164:0/999443311 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480aa3e0 con 0x7fd6480a7990 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> v2:172.21.15.164:3300/0 conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd648090810 tx=0x7fd64000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> v2:172.21.15.164:3300/0 conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd648090810 tx=0x7fd64000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.737+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fd648090810 tx=0x7fd64000ff30).ready entity=mon.1 client_cookie=ae8610f30d6fff73 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fd648090810 tx=0x7fd64000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fd648090810 tx=0x7fd64000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.747 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd64c940700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fd648090810 tx=0x7fd64000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd646ffd700 1 -- 172.21.15.164:0/999443311 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd640011760 con 0x7fd6480a7990 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd646ffd700 1 -- 172.21.15.164:0/999443311 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd640008d30 con 0x7fd6480a7990 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd646ffd700 1 -- 172.21.15.164:0/999443311 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd640011d40 con 0x7fd6480a7990 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd64dba2700 1 -- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 msgr2=0x7fd6480a7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.738+0000 7fd64dba2700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fd648090810 tx=0x7fd64000ff30).stop 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 -- 172.21.15.164:0/999443311 shutdown_connections 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a7990 0x7fd6480a7dc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.748 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 --2- 172.21.15.164:0/999443311 >> v2:172.21.15.8:3300/0 conn(0x7fd64808fff0 0x7fd64808f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.749 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 --2- 172.21.15.164:0/999443311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480908f0 0x7fd6480a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:56.749 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 -- 172.21.15.164:0/999443311 >> 172.21.15.164:0/999443311 conn(0x7fd6480600f0 msgr2=0x7fd64805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.749 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 -- 172.21.15.164:0/999443311 shutdown_connections 2020-02-05T16:58:56.749 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.739+0000 7fd64dba2700 1 -- 172.21.15.164:0/999443311 wait complete. 2020-02-05T16:58:56.753 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.742+0000 7fd64dba2700 1 Processor -- start 2020-02-05T16:58:56.753 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.742+0000 7fd64dba2700 1 -- start start 2020-02-05T16:58:56.753 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd64dba2700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6480a3950 0x7fd648078d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.753 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd64dba2700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 0x7fd64813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.753 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd64dba2700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd648076440 con 0x7fd6480a6960 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd64dba2700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd648078830 con 0x7fd648077f20 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd64dba2700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd648077c80 con 0x7fd6480a3950 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd6477fe700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd6480a6960 0x7fd64809dec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39990/0 (socket says 172.21.15.164:39990) 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd647fff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd6480a3950 0x7fd648078d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.743+0000 7fd6477fe700 1 -- 172.21.15.164:0/2776267191 learned_addr learned my addr 172.21.15.164:0/2776267191 (peer_addr_for_me v1:172.21.15.164:0/0) 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd64c940700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 0x7fd64813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.754 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495070080 0 0) 0x7fd648076440 con 0x7fd6480a6960 2020-02-05T16:58:56.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd628003230 con 0x7fd6480a6960 2020-02-05T16:58:56.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd647fff700 1 -- 172.21.15.164:0/2776267191 >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 msgr2=0x7fd64813f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 0x7fd64813f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd647fff700 1 -- 172.21.15.164:0/2776267191 >> v1:172.21.15.164:6789/0 conn(0x7fd6480a6960 legacy=0x7fd64809dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.755 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd647fff700 1 -- 172.21.15.164:0/2776267191 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd648078350 con 0x7fd6480a3950 2020-02-05T16:58:56.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.744+0000 7fd64c940700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 0x7fd64813f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:58:56.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.164:3300/0 conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.164:3300/0 conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.756 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.757 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.757 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd638012a70 con 0x7fd6480a3950 2020-02-05T16:58:56.758 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6480a1ee0 con 0x7fd6480a3950 2020-02-05T16:58:56.758 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd64813fc20 con 0x7fd6480a3950 2020-02-05T16:58:56.758 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.758 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd638012690 con 0x7fd6480a3950 2020-02-05T16:58:56.758 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.745+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd63800e640 con 0x7fd6480a3950 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.747+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd648056640 con 0x7fd6480a3950 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.747+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.748+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.748+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 3) v1 ==== 44173+0+0 (secure 0 0 0) 0x7fd638007170 con 0x7fd6480a3950 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.748+0000 7fd644ff9700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 0x7fd62803c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.748+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7fd638049d40 con 0x7fd6480a3950 2020-02-05T16:58:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.749+0000 7fd6477fe700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 0x7fd62803c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.749+0000 7fd6477fe700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 0x7fd62803c370 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.754+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.754+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fd63800d0b0 con 0x7fd6480a3950 2020-02-05T16:58:56.929 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.927+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c1ecad70-c4c6-4b35-896c-25b1d9e5a546", "id": 5} v 0) v1 -- 0x7fd648068f50 con 0x7fd6480a3950 2020-02-05T16:58:56.941 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.939+0000 7fd647fff700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:56.942 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.939+0000 7fd644ff9700 1 -- 172.21.15.164:0/2776267191 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c1ecad70-c4c6-4b35-896c-25b1d9e5a546", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd638031820 con 0x7fd6480a3950 2020-02-05T16:58:56.942 INFO:teuthology.orchestra.run.smithi164.stdout:5 2020-02-05T16:58:56.945 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 msgr2=0x7fd62803c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 0x7fd62803c370 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 msgr2=0x7fd648078d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7fd638002f30 tx=0x7fd638010d40).stop 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 shutdown_connections 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 --2- 172.21.15.164:0/2776267191 >> v2:172.21.15.8:3300/0 conn(0x7fd648077f20 0x7fd64813f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.946 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a3950 0x7fd648078d40 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.947 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 --2- 172.21.15.164:0/2776267191 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd628039e60 0x7fd62803c370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:56.947 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.943+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 >> 172.21.15.164:0/2776267191 conn(0x7fd6480698e0 msgr2=0x7fd64807e990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:56.947 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.944+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 shutdown_connections 2020-02-05T16:58:56.947 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:56.944+0000 7fd64dba2700 1 -- 172.21.15.164:0/2776267191 wait complete. 2020-02-05T16:58:56.965 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 96135535-1048-41e4-8969-723592815994 6 2020-02-05T16:58:57.170 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.162+0000 7f2fcd8f7700 1 Processor -- start 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.162+0000 7f2fcd8f7700 1 -- start start 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fc808fff0 0x7f2fc808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc80908f0 0x7f2fc80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 0x7f2fc80a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc8090d20 con 0x7f2fc80a7990 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc8090650 con 0x7f2fc808fff0 2020-02-05T16:58:57.171 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fcd8f7700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc808ece0 con 0x7f2fc80908f0 2020-02-05T16:58:57.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fc77fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fc808fff0 0x7f2fc808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fc77fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fc808fff0 0x7f2fc808f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55840/0 (socket says 172.21.15.164:55840) 2020-02-05T16:58:57.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1470170013 learned_addr learned my addr 172.21.15.164:0/1470170013 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:57.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fc6ffd700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc80908f0 0x7f2fc80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.172 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.163+0000 7f2fc7fff700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 0x7f2fc80a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc80908f0 msgr2=0x7f2fc80a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc80908f0 0x7f2fc80a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1470170013 >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 msgr2=0x7f2fc80a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 0x7f2fc80a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1470170013 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80aa3e0 con 0x7f2fc808fff0 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc7fff700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 0x7f2fc80a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:58:57.173 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.164:3300/0 conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.164:3300/0 conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).ready entity=mon.1 client_cookie=5152796335f619a1 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.164+0000 7f2fc67fc700 1 -- 172.21.15.164:0/1470170013 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb8011760 con 0x7f2fc808fff0 2020-02-05T16:58:57.174 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.165+0000 7f2fc67fc700 1 -- 172.21.15.164:0/1470170013 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8008d30 con 0x7f2fc808fff0 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.165+0000 7f2fc67fc700 1 -- 172.21.15.164:0/1470170013 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb8011d40 con 0x7f2fc808fff0 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.165+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 msgr2=0x7f2fc808f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.165+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f2fc8090810 tx=0x7f2fb800ff30).stop 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1470170013 shutdown_connections 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1470170013 >> v2:172.21.15.8:3300/0 conn(0x7f2fc80a7990 0x7f2fc80a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc808fff0 0x7f2fc808f240 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.175 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1470170013 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc80908f0 0x7f2fc80a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.176 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1470170013 >> 172.21.15.164:0/1470170013 conn(0x7f2fc80600f0 msgr2=0x7f2fc805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:57.176 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1470170013 shutdown_connections 2020-02-05T16:58:57.176 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.166+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1470170013 wait complete. 2020-02-05T16:58:57.189 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.169+0000 7f2fcd8f7700 1 Processor -- start 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.169+0000 7f2fcd8f7700 1 -- start start 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fc812c6a0 0x7f2fc8121930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 0x7f2fc81366b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f2fc807ddc0 0x7f2fc807e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc8122200 con 0x7f2fc807ddc0 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc8122460 con 0x7f2fc812c6a0 2020-02-05T16:58:57.190 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fcd8f7700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8122700 con 0x7f2fc8136280 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fc6ffd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 0x7f2fc81366b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fc77fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2fc812c6a0 0x7f2fc8121930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.170+0000 7f2fc6ffd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 0x7f2fc81366b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47644/0 (socket says 172.21.15.164:47644) 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc6ffd700 1 -- 172.21.15.164:0/1810609070 learned_addr learned my addr 172.21.15.164:0/1810609070 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc7fff700 1 --2- 172.21.15.164:0/1810609070 >> v2:172.21.15.8:3300/0 conn(0x7f2fc807ddc0 0x7f2fc807e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 msgr2=0x7f2fc81366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 0x7f2fc81366b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1810609070 >> v2:172.21.15.8:3300/0 conn(0x7f2fc807ddc0 msgr2=0x7f2fc807e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> v2:172.21.15.8:3300/0 conn(0x7f2fc807ddc0 0x7f2fc807e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 -- 172.21.15.164:0/1810609070 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc807e740 con 0x7f2fc812c6a0 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.171+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> v2:172.21.15.164:3300/0 conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> v2:172.21.15.164:3300/0 conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.193 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.193 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb800f780 con 0x7f2fc812c6a0 2020-02-05T16:58:57.193 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.193 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8011960 con 0x7f2fc812c6a0 2020-02-05T16:58:57.193 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8121e80 con 0x7f2fc812c6a0 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2fb80056c0 con 0x7f2fc812c6a0 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.172+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc80a0f60 con 0x7f2fc812c6a0 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.173+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.174+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc8056640 con 0x7f2fc812c6a0 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.174+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.174+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 3) v1 ==== 44173+0+0 (secure 0 0 0) 0x7f2fb80059d0 con 0x7f2fc812c6a0 2020-02-05T16:58:57.194 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.175+0000 7f2fbffff700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 0x7f2fac03bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.175+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f2fb80499f0 con 0x7f2fc812c6a0 2020-02-05T16:58:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.176+0000 7f2fc6ffd700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 0x7f2fac03bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.176+0000 7f2fc6ffd700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 0x7f2fac03bdd0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.179+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.179+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f2fb8031550 con 0x7f2fc812c6a0 2020-02-05T16:58:57.351 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.349+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "96135535-1048-41e4-8969-723592815994", "id": 6} v 0) v1 -- 0x7f2fc8068f50 con 0x7f2fc812c6a0 2020-02-05T16:58:57.380 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.362+0000 7f2fc77fe700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.380 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.362+0000 7f2fbffff700 1 -- 172.21.15.164:0/1810609070 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "96135535-1048-41e4-8969-723592815994", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2fb80317b0 con 0x7f2fc812c6a0 2020-02-05T16:58:57.381 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 msgr2=0x7f2fac03bdd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.381 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 0x7f2fac03bdd0 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.381 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 msgr2=0x7f2fc8121930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.381 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f2fc81228c0 tx=0x7f2fb80118a0).stop 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 shutdown_connections 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2fac0398c0 0x7f2fac03bdd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> v2:172.21.15.8:3300/0 conn(0x7f2fc807ddc0 0x7f2fc807e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2fc812c6a0 0x7f2fc8121930 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 --2- 172.21.15.164:0/1810609070 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2fc8136280 0x7f2fc81366b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 >> 172.21.15.164:0/1810609070 conn(0x7f2fc8066c30 msgr2=0x7f2fc812af80 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 shutdown_connections 2020-02-05T16:58:57.382 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.365+0000 7f2fcd8f7700 1 -- 172.21.15.164:0/1810609070 wait complete. 2020-02-05T16:58:57.383 INFO:teuthology.orchestra.run.smithi164.stdout:6 2020-02-05T16:58:57.393 INFO:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd new 049fe10e-349c-4d45-9361-4d25ae4c0487 7 2020-02-05T16:58:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.601+0000 7fbf28fba700 1 Processor -- start 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.601+0000 7fbf28fba700 1 -- start start 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.601+0000 7fbf28fba700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf2408f240 0x7fbf240908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf28fba700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a9e90 0x7fbf240a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf28fba700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240aab90 con 0x7fbf2408fff0 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf28fba700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2408f670 con 0x7fbf2408f240 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf28fba700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf24090650 con 0x7fbf240a9e90 2020-02-05T16:58:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf2359e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a9e90 0x7fbf240a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf2259c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf2408f240 0x7fbf240908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf22d9d700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fbf2408fff0 0x7fbf240a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.164:35994/0 (socket says 172.21.15.164:35994) 2020-02-05T16:58:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.602+0000 7fbf22d9d700 1 -- 172.21.15.164:0/2129094681 learned_addr learned my addr 172.21.15.164:0/2129094681 (peer_addr_for_me v1:172.21.15.164:0/0) 2020-02-05T16:58:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf21d9b700 1 -- 172.21.15.164:0/2129094681 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4083016710 0 0) 0x7fbf240aab90 con 0x7fbf2408fff0 2020-02-05T16:58:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf21d9b700 1 -- 172.21.15.164:0/2129094681 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003230 con 0x7fbf2408fff0 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf2259c700 1 -- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a9e90 msgr2=0x7fbf240a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a9e90 0x7fbf240a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf2259c700 1 -- 172.21.15.164:0/2129094681 >> v1:172.21.15.8:6789/0 conn(0x7fbf2408fff0 legacy=0x7fbf240a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.603+0000 7fbf2259c700 1 -- 172.21.15.164:0/2129094681 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240a86e0 con 0x7fbf2408f240 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> v2:172.21.15.164:3300/0 conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> v2:172.21.15.164:3300/0 conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).ready entity=mon.1 client_cookie=9b71daab31d16311 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf21d9b700 1 -- 172.21.15.164:0/2129094681 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf180116e0 con 0x7fbf2408f240 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf2259c700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf21d9b700 1 -- 172.21.15.164:0/2129094681 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18008c50 con 0x7fbf2408f240 2020-02-05T16:58:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf21d9b700 1 -- 172.21.15.164:0/2129094681 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf18011cc0 con 0x7fbf2408f240 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf28fba700 1 -- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 msgr2=0x7fbf240908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.604+0000 7fbf28fba700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7fbf2409ede0 tx=0x7fbf1800fe70).stop 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.605+0000 7fbf28fba700 1 -- 172.21.15.164:0/2129094681 shutdown_connections 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.605+0000 7fbf28fba700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a9e90 0x7fbf240a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.605+0000 7fbf28fba700 1 --2- 172.21.15.164:0/2129094681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbf2408f240 0x7fbf240908f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.605+0000 7fbf28fba700 1 -- 172.21.15.164:0/2129094681 >> 172.21.15.164:0/2129094681 conn(0x7fbf240600f0 msgr2=0x7fbf2405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.606+0000 7fbf28fba700 1 -- 172.21.15.164:0/2129094681 shutdown_connections 2020-02-05T16:58:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.606+0000 7fbf28fba700 1 -- 172.21.15.164:0/2129094681 wait complete. 2020-02-05T16:58:57.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.608+0000 7fbf28fba700 1 Processor -- start 2020-02-05T16:58:57.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.608+0000 7fbf28fba700 1 -- start start 2020-02-05T16:58:57.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 0x7fbf240a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fbf241288a0 0x7fbf240a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.618 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.619 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240a7740 con 0x7fbf241288a0 2020-02-05T16:58:57.619 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf240a3270 con 0x7fbf240a2780 2020-02-05T16:58:57.619 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf28fba700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf240a3510 con 0x7fbf240a7310 2020-02-05T16:58:57.619 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.609+0000 7fbf22d9d700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 0x7fbf240a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.619 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.630 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47662/0 (socket says 172.21.15.164:47662) 2020-02-05T16:58:57.630 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf22d9d700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 0x7fbf240a2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55860/0 (socket says 172.21.15.164:55860) 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 -- 172.21.15.164:0/1808743826 learned_addr learned my addr 172.21.15.164:0/1808743826 (peer_addr_for_me v2:172.21.15.164:0/0) 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2259c700 1 --2- 172.21.15.164:0/1808743826 >> v2:172.21.15.8:3300/0 conn(0x7fbf241288a0 0x7fbf240a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 -- 172.21.15.164:0/1808743826 >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 msgr2=0x7fbf240a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 0x7fbf240a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 -- 172.21.15.164:0/1808743826 >> v2:172.21.15.8:3300/0 conn(0x7fbf241288a0 msgr2=0x7fbf240a6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.631 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.610+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> v2:172.21.15.8:3300/0 conn(0x7fbf241288a0 0x7fbf240a6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.632 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 -- 172.21.15.164:0/1808743826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240a49f0 con 0x7fbf240a7310 2020-02-05T16:58:57.632 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.632 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf1c0129f0 con 0x7fbf240a7310 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.633 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf1c012610 con 0x7fbf240a7310 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2407eff0 con 0x7fbf240a7310 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf1c00e450 con 0x7fbf240a7310 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.611+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf24128cd0 con 0x7fbf240a7310 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.613+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.613+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 3) v1 ==== 44173+0+0 (secure 0 0 0) 0x7fbf1c007170 con 0x7fbf240a7310 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.613+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.634 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.613+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf24056640 con 0x7fbf240a7310 2020-02-05T16:58:57.635 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.614+0000 7fbf137fe700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 0x7fbf0803bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:57.635 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.614+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7fbf1c0499f0 con 0x7fbf240a7310 2020-02-05T16:58:57.635 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.614+0000 7fbf22d9d700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 0x7fbf0803bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:57.635 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.614+0000 7fbf22d9d700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 0x7fbf0803bee0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:57.636 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.618+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.636 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.618+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fbf1c00d0b0 con 0x7fbf240a7310 2020-02-05T16:58:57.788 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.786+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd new", "uuid": "049fe10e-349c-4d45-9361-4d25ae4c0487", "id": 7} v 0) v1 -- 0x7fbf24068f50 con 0x7fbf240a7310 2020-02-05T16:58:57.798 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.795+0000 7fbf2359e700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=THROTTLE_DONE pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:57.798 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.795+0000 7fbf137fe700 1 -- 172.21.15.164:0/1808743826 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "049fe10e-349c-4d45-9361-4d25ae4c0487", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fbf1c0314d0 con 0x7fbf240a7310 2020-02-05T16:58:57.798 INFO:teuthology.orchestra.run.smithi164.stdout:7 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 msgr2=0x7fbf0803bee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 0x7fbf0803bee0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 msgr2=0x7fbf240a5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 secure :-1 s=READY pgs=14 cs=0 l=1 rx=0x7fbf2409e980 tx=0x7fbf1c010ca0).stop 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 shutdown_connections 2020-02-05T16:58:57.801 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fbf080399d0 0x7fbf0803bee0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbf240a7310 0x7fbf240a5690 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> v2:172.21.15.164:3300/0 conn(0x7fbf240a2780 0x7fbf240a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 --2- 172.21.15.164:0/1808743826 >> v2:172.21.15.8:3300/0 conn(0x7fbf241288a0 0x7fbf240a6dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.798+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 >> 172.21.15.164:0/1808743826 conn(0x7fbf240aa960 msgr2=0x7fbf240650a0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.799+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 shutdown_connections 2020-02-05T16:58:57.802 INFO:teuthology.orchestra.run.smithi164.stderr:2020-02-05T16:58:57.799+0000 7fbf28fba700 1 -- 172.21.15.164:0/1808743826 wait complete. 2020-02-05T16:58:57.818 INFO:tasks.ceph.osd.0:Restarting daemon 2020-02-05T16:58:57.819 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2020-02-05T16:58:57.825 INFO:tasks.ceph.osd.0:Started 2020-02-05T16:58:57.825 INFO:tasks.ceph.osd.1:Restarting daemon 2020-02-05T16:58:57.825 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2020-02-05T16:58:57.831 INFO:tasks.ceph.osd.1:Started 2020-02-05T16:58:57.831 INFO:tasks.ceph.osd.2:Restarting daemon 2020-02-05T16:58:57.831 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2020-02-05T16:58:57.836 INFO:tasks.ceph.osd.2:Started 2020-02-05T16:58:57.837 INFO:tasks.ceph.osd.3:Restarting daemon 2020-02-05T16:58:57.837 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2020-02-05T16:58:57.844 INFO:tasks.ceph.osd.3:Started 2020-02-05T16:58:57.844 INFO:tasks.ceph.osd.4:Restarting daemon 2020-02-05T16:58:57.844 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2020-02-05T16:58:57.852 INFO:tasks.ceph.osd.4:Started 2020-02-05T16:58:57.852 INFO:tasks.ceph.osd.5:Restarting daemon 2020-02-05T16:58:57.853 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2020-02-05T16:58:57.858 INFO:tasks.ceph.osd.5:Started 2020-02-05T16:58:57.858 INFO:tasks.ceph.osd.6:Restarting daemon 2020-02-05T16:58:57.858 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2020-02-05T16:58:57.863 INFO:tasks.ceph.osd.6:Started 2020-02-05T16:58:57.863 INFO:tasks.ceph.osd.7:Restarting daemon 2020-02-05T16:58:57.863 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2020-02-05T16:58:57.868 INFO:tasks.ceph.osd.7:Started 2020-02-05T16:58:57.868 INFO:tasks.ceph:Waiting for OSDs to come up 2020-02-05T16:58:58.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T16:58:58.489+0000 7f7c7652fec0 -1 Falling back to public interface 2020-02-05T16:58:58.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T16:58:58.496+0000 7fd45c3baec0 -1 Falling back to public interface 2020-02-05T16:58:58.508 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T16:58:58.506+0000 7f5587fcaec0 -1 Falling back to public interface 2020-02-05T16:58:58.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T16:58:58.509+0000 7f63d2971ec0 -1 Falling back to public interface 2020-02-05T16:58:58.516 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T16:58:58.513+0000 7fcd8fdc0ec0 -1 Falling back to public interface 2020-02-05T16:58:58.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T16:58:58.517+0000 7f2f01846ec0 -1 Falling back to public interface 2020-02-05T16:58:58.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T16:58:58.520+0000 7f157d45cec0 -1 Falling back to public interface 2020-02-05T16:58:58.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T16:58:58.527+0000 7f72b4a8fec0 -1 Falling back to public interface 2020-02-05T16:58:58.675 INFO:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-02-05T16:58:58.888 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.878+0000 7f384cb82700 1 Processor -- start 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.879+0000 7f384cb82700 1 -- start start 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.880+0000 7f384cb82700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f384814f0e0 0x7f3848151520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.880+0000 7f384cb82700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38480a0ff0 con 0x7f38480a2670 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.880+0000 7f384cb82700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38480a2df0 con 0x7f38480a1cd0 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.880+0000 7f384cb82700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38480a2b50 con 0x7f384814f0e0 2020-02-05T16:58:58.889 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f3847fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f38480a2670 0x7f38480aa520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45272/0 (socket says 172.21.15.8:45272) 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f3847fff700 1 -- 172.21.15.8:0/2173007498 learned_addr learned my addr 172.21.15.8:0/2173007498 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> v2:172.21.15.164:3300/0 conn(0x7f384814f0e0 0x7f3848151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696141799 0 0) 0x7f38480a0ff0 con 0x7f38480a2670 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3830003230 con 0x7f38480a2670 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f384dde4700 1 -- 172.21.15.8:0/2173007498 >> v1:172.21.15.164:6789/0 conn(0x7f38480a1cd0 legacy=0x7f3848149350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:58.891 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f384dde4700 1 -- 172.21.15.8:0/2173007498 >> v1:172.21.15.8:6789/0 conn(0x7f38480a2670 legacy=0x7f38480aa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779365424 0 0) 0x7f38480a2df0 con 0x7f38480a1cd0 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.881+0000 7f384dde4700 1 -- 172.21.15.8:0/2173007498 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38480aad20 con 0x7f384814f0e0 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> v2:172.21.15.164:3300/0 conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3840008c90 tx=0x7f38400101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> v2:172.21.15.164:3300/0 conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3840008c90 tx=0x7f38400101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f38400101f0).ready entity=mon.1 client_cookie=bb3bc47db4d9278 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f38400101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.892 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f384000a160 con 0x7f384814f0e0 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f38400101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384dde4700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f38400101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3840011b70 con 0x7f384814f0e0 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f3846ffd700 1 -- 172.21.15.8:0/2173007498 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f384000a740 con 0x7f384814f0e0 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384cb82700 1 -- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 msgr2=0x7f3848151520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.882+0000 7f384cb82700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f38400101f0).stop 2020-02-05T16:58:58.893 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.883+0000 7f384cb82700 1 -- 172.21.15.8:0/2173007498 shutdown_connections 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.883+0000 7f384cb82700 1 --2- 172.21.15.8:0/2173007498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f384814f0e0 0x7f3848151520 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.883+0000 7f384cb82700 1 -- 172.21.15.8:0/2173007498 >> 172.21.15.8:0/2173007498 conn(0x7f38480a8b70 msgr2=0x7f38480a8fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.883+0000 7f384cb82700 1 -- 172.21.15.8:0/2173007498 shutdown_connections 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.883+0000 7f384cb82700 1 -- 172.21.15.8:0/2173007498 wait complete. 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.886+0000 7f384cb82700 1 Processor -- start 2020-02-05T16:58:58.894 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.886+0000 7f384cb82700 1 -- start start 2020-02-05T16:58:58.896 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f384cb82700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f38480a34a0 0x7f3848076200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f384cb82700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38480a42b0 con 0x7f3848121a50 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f384cb82700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3848138050 con 0x7f38480658a0 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f384cb82700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38480a3ad0 con 0x7f38480a34a0 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f38477fe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f3848121a50 0x7f3848137850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45280/0 (socket says 172.21.15.8:45280) 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f3847fff700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f38480658a0 0x7f3848148520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:57764/0 (socket says 172.21.15.8:57764) 2020-02-05T16:58:58.897 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.887+0000 7f38477fe700 1 -- 172.21.15.8:0/841163021 learned_addr learned my addr 172.21.15.8:0/841163021 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> v2:172.21.15.8:3300/0 conn(0x7f38480a34a0 0x7f3848076200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345104015 0 0) 0x7f38480a42b0 con 0x7f3848121a50 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3828003230 con 0x7f3848121a50 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481948906 0 0) 0x7f3848138050 con 0x7f38480658a0 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38480a42b0 con 0x7f38480658a0 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f384dde4700 1 -- 172.21.15.8:0/841163021 >> v1:172.21.15.164:6789/0 conn(0x7f38480658a0 legacy=0x7f3848148520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:58.898 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f384dde4700 1 -- 172.21.15.8:0/841163021 >> v1:172.21.15.8:6789/0 conn(0x7f3848121a50 legacy=0x7f3848137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f384dde4700 1 -- 172.21.15.8:0/841163021 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848078770 con 0x7f38480a34a0 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.888+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> v2:172.21.15.8:3300/0 conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> v2:172.21.15.8:3300/0 conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3840005a40 con 0x7f38480a34a0 2020-02-05T16:58:58.899 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.900 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384000a360 con 0x7f38480a34a0 2020-02-05T16:58:58.900 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38400112b0 con 0x7f38480a34a0 2020-02-05T16:58:58.901 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3848076750 con 0x7f38480a34a0 2020-02-05T16:58:58.901 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.889+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3848076c70 con 0x7f38480a34a0 2020-02-05T16:58:58.901 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.890+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.901 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.891+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38480a38d0 con 0x7f38480a34a0 2020-02-05T16:58:58.902 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.891+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f38400114b0 con 0x7f38480a34a0 2020-02-05T16:58:58.902 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.891+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.891+0000 7f3844ff9700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 0x7f382803c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.892+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f3840049c00 con 0x7f38480a34a0 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.892+0000 7f3847fff700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 0x7f382803c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.892+0000 7f3847fff700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 0x7f382803c690 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.899+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:58.903 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:58.899+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3840031750 con 0x7f38480a34a0 2020-02-05T16:58:59.065 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.062+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3848056640 con 0x7f38480a34a0 2020-02-05T16:58:59.066 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.064+0000 7f384dde4700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:58:59.066 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.064+0000 7f3844ff9700 1 -- 172.21.15.8:0/841163021 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7f3840031490 con 0x7f38480a34a0 2020-02-05T16:58:59.066 INFO:teuthology.misc.health.smithi008.stdout: 2020-02-05T16:58:59.067 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":10,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:58:57.789374+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:58:59.069 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.065+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 msgr2=0x7f382803c690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:59.069 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.065+0000 7f384cb82700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 0x7f382803c690 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:59.069 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.065+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 msgr2=0x7f3848076200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:58:59.069 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.065+0000 7f384cb82700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f3840008c90 tx=0x7f3840011e80).stop 2020-02-05T16:58:59.069 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 shutdown_connections 2020-02-05T16:58:59.070 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f382803a180 0x7f382803c690 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:59.070 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 --2- 172.21.15.8:0/841163021 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f38480a34a0 0x7f3848076200 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:58:59.070 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 >> 172.21.15.8:0/841163021 conn(0x7f38480aa320 msgr2=0x7f3848127360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:58:59.070 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 shutdown_connections 2020-02-05T16:58:59.070 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:58:59.066+0000 7f384cb82700 1 -- 172.21.15.8:0/841163021 wait complete. 2020-02-05T16:58:59.084 DEBUG:teuthology.misc:0 of 8 OSDs are up 2020-02-05T16:58:59.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T16:58:59.606+0000 7f7c7652fec0 -1 osd.0 0 log_to_monitors {default=true} 2020-02-05T16:58:59.624 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T16:58:59.622+0000 7f5587fcaec0 -1 osd.2 0 log_to_monitors {default=true} 2020-02-05T16:58:59.625 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T16:58:59.622+0000 7fd45c3baec0 -1 osd.1 0 log_to_monitors {default=true} 2020-02-05T16:58:59.631 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T16:58:59.628+0000 7f63d2971ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-02-05T16:58:59.632 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T16:58:59.630+0000 7fcd8fdc0ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-02-05T16:58:59.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T16:58:59.633+0000 7f2f01846ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-02-05T16:58:59.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T16:58:59.637+0000 7f157d45cec0 -1 osd.6 0 log_to_monitors {default=true} 2020-02-05T16:58:59.639 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T16:58:59.637+0000 7f72b4a8fec0 -1 osd.7 0 log_to_monitors {default=true} 2020-02-05T16:59:01.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T16:59:01.505+0000 7f7c5ecf9700 -1 osd.0 0 waiting for initial osdmap 2020-02-05T16:59:01.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T16:59:01.505+0000 7fcd7858a700 -1 osd.4 0 waiting for initial osdmap 2020-02-05T16:59:01.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T16:59:01.506+0000 7f5570794700 -1 osd.2 0 waiting for initial osdmap 2020-02-05T16:59:01.509 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T16:59:01.506+0000 7f63bb13b700 -1 osd.3 0 waiting for initial osdmap 2020-02-05T16:59:01.510 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T16:59:01.508+0000 7f1565c26700 -1 osd.6 0 waiting for initial osdmap 2020-02-05T16:59:01.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T16:59:01.508+0000 7fd444b84700 -1 osd.1 0 waiting for initial osdmap 2020-02-05T16:59:01.511 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T16:59:01.508+0000 7f729d259700 -1 osd.7 0 waiting for initial osdmap 2020-02-05T16:59:01.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T16:59:01.512+0000 7f2eea010700 -1 osd.5 0 waiting for initial osdmap 2020-02-05T16:59:05.735 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T16:59:05.741 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T16:59:05.889 INFO:teuthology.orchestra.run.smithi008:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:06.086 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.076+0000 7f3161442700 1 Processor -- start 2020-02-05T16:59:06.086 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.077+0000 7f3161442700 1 -- start start 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.077+0000 7f3161442700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f315c0a2a20 0x7f315c0a3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.077+0000 7f3161442700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f315c0a3eb0 0x7f315c14ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f3161442700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0a9620 con 0x7f315c0aafc0 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f3161442700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c146710 con 0x7f315c0a3eb0 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f3161442700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0a32a0 con 0x7f315c0a2a20 2020-02-05T16:59:06.087 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f315b7fe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f315c0aafc0 0x7f315c145f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45506/0 (socket says 172.21.15.8:45506) 2020-02-05T16:59:06.088 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f315b7fe700 1 -- 172.21.15.8:0/1949002517 learned_addr learned my addr 172.21.15.8:0/1949002517 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:06.088 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.8:3300/0 conn(0x7f315c0a3eb0 0x7f315c14ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.088 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.078+0000 7f315affd700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.164:3300/0 conn(0x7f315c0a2a20 0x7f315c0a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.088 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315a7fc700 1 -- 172.21.15.8:0/1949002517 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297920550 0 0) 0x7f315c0a9620 con 0x7f315c0aafc0 2020-02-05T16:59:06.088 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315a7fc700 1 -- 172.21.15.8:0/1949002517 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3144003230 con 0x7f315c0aafc0 2020-02-05T16:59:06.089 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 -- 172.21.15.8:0/1949002517 >> v2:172.21.15.164:3300/0 conn(0x7f315c0a2a20 msgr2=0x7f315c0a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:06.089 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.164:3300/0 conn(0x7f315c0a2a20 0x7f315c0a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:06.089 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 -- 172.21.15.8:0/1949002517 >> v1:172.21.15.8:6789/0 conn(0x7f315c0aafc0 legacy=0x7f315c145f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.089 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 -- 172.21.15.8:0/1949002517 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0a2e50 con 0x7f315c0a3eb0 2020-02-05T16:59:06.089 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.8:3300/0 conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f315c0a3460 tx=0x7f315400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.8:3300/0 conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f315c0a3460 tx=0x7f315400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.079+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f315c0a3460 tx=0x7f315400fe70).ready entity=mon.0 client_cookie=2855b0eb024e119a server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f315c0a3460 tx=0x7f315400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f315c0a3460 tx=0x7f315400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315a7fc700 1 -- 172.21.15.8:0/1949002517 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3154008d30 con 0x7f315c0a3eb0 2020-02-05T16:59:06.090 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315bfff700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f315c0a3460 tx=0x7f315400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315a7fc700 1 -- 172.21.15.8:0/1949002517 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3154008950 con 0x7f315c0a3eb0 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f3161442700 1 -- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 msgr2=0x7f315c14ff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f315a7fc700 1 -- 172.21.15.8:0/1949002517 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3154010b20 con 0x7f315c0a3eb0 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.080+0000 7f3161442700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f315c0a3460 tx=0x7f315400fe70).stop 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.081+0000 7f3161442700 1 -- 172.21.15.8:0/1949002517 shutdown_connections 2020-02-05T16:59:06.091 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.081+0000 7f3161442700 1 --2- 172.21.15.8:0/1949002517 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c0a3eb0 0x7f315c14ff10 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.092 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.081+0000 7f3161442700 1 --2- 172.21.15.8:0/1949002517 >> v2:172.21.15.164:3300/0 conn(0x7f315c0a2a20 0x7f315c0a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:06.092 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.081+0000 7f3161442700 1 -- 172.21.15.8:0/1949002517 >> 172.21.15.8:0/1949002517 conn(0x7f315c069aa0 msgr2=0x7f315c05fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:06.092 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.082+0000 7f3161442700 1 -- 172.21.15.8:0/1949002517 shutdown_connections 2020-02-05T16:59:06.092 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.082+0000 7f3161442700 1 -- 172.21.15.8:0/1949002517 wait complete. 2020-02-05T16:59:06.092 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.084+0000 7f3161442700 1 Processor -- start 2020-02-05T16:59:06.097 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.084+0000 7f3161442700 1 -- start start 2020-02-05T16:59:06.097 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f3161442700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f315c12e050 0x7f315c07b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:06.097 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f3161442700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0a5bf0 con 0x7f315c127b20 2020-02-05T16:59:06.097 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f3161442700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c07c3a0 con 0x7f315c07bf70 2020-02-05T16:59:06.098 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f3161442700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c128750 con 0x7f315c12e050 2020-02-05T16:59:06.098 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f315b7fe700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f315c12e050 0x7f315c07b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.098 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f315bfff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f315c127b20 0x7f315c127f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45516/0 (socket says 172.21.15.8:45516) 2020-02-05T16:59:06.098 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.085+0000 7f315bfff700 1 -- 172.21.15.8:0/607917822 learned_addr learned my addr 172.21.15.8:0/607917822 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:06.099 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.086+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919804369 0 0) 0x7f315c0a5bf0 con 0x7f315c127b20 2020-02-05T16:59:06.099 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.086+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f313c003230 con 0x7f315c127b20 2020-02-05T16:59:06.099 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.086+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628578982 0 0) 0x7f315c07c3a0 con 0x7f315c07bf70 2020-02-05T16:59:06.099 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.086+0000 7f315b7fe700 1 -- 172.21.15.8:0/607917822 >> v1:172.21.15.164:6789/0 conn(0x7f315c07bf70 legacy=0x7f315c0a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.100 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.086+0000 7f315b7fe700 1 -- 172.21.15.8:0/607917822 >> v1:172.21.15.8:6789/0 conn(0x7f315c127b20 legacy=0x7f315c127f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.100 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 -- 172.21.15.8:0/607917822 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c11d840 con 0x7f315c12e050 2020-02-05T16:59:06.100 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> v2:172.21.15.8:3300/0 conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.100 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> v2:172.21.15.8:3300/0 conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.101 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.101 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.101 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f314c012570 con 0x7f315c12e050 2020-02-05T16:59:06.101 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.102 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.102 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c005c60 con 0x7f315c12e050 2020-02-05T16:59:06.102 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f314c012b50 con 0x7f315c12e050 2020-02-05T16:59:06.102 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.087+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c11b820 con 0x7f315c12e050 2020-02-05T16:59:06.102 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.088+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c11bd70 con 0x7f315c12e050 2020-02-05T16:59:06.103 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.089+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.103 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.089+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c056640 con 0x7f315c12e050 2020-02-05T16:59:06.103 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.089+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f314c00a320 con 0x7f315c12e050 2020-02-05T16:59:06.103 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.089+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.104 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.090+0000 7f3143fff700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 0x7f313c03c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:06.104 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.090+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f314c049c30 con 0x7f315c12e050 2020-02-05T16:59:06.104 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.091+0000 7f315affd700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 0x7f313c03c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.104 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.091+0000 7f315affd700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 0x7f313c03c360 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.105 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.095+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.106 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.096+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f314c031720 con 0x7f315c12e050 2020-02-05T16:59:06.256 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.253+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f315c060190 con 0x7f315c12e050 2020-02-05T16:59:06.257 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.255+0000 7f315b7fe700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.257 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.255+0000 7f3143fff700 1 -- 172.21.15.8:0/607917822 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10391 (secure 0 0 0) 0x7f314c031980 con 0x7f315c12e050 2020-02-05T16:59:06.258 INFO:teuthology.misc.health.smithi008.stdout: 2020-02-05T16:59:06.258 INFO:teuthology.misc.health.smithi008.stdout:{"epoch":14,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:03.522883+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:06.259 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 msgr2=0x7f313c03c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.259 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 0x7f313c03c360 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.259 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 msgr2=0x7f315c07b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.259 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f314c001210 tx=0x7f314c010cc0).stop 2020-02-05T16:59:06.260 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 shutdown_connections 2020-02-05T16:59:06.260 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f315c12e050 0x7f315c07b2f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.260 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 --2- 172.21.15.8:0/607917822 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f313c039e50 0x7f313c03c360 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.261 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 >> 172.21.15.8:0/607917822 conn(0x7f315c05fd50 msgr2=0x7f315c142dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:06.261 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 shutdown_connections 2020-02-05T16:59:06.261 INFO:teuthology.misc.health.smithi008.stderr:2020-02-05T16:59:06.257+0000 7f3161442700 1 -- 172.21.15.8:0/607917822 wait complete. 2020-02-05T16:59:06.275 DEBUG:teuthology.misc:8 of 8 OSDs are up 2020-02-05T16:59:06.276 INFO:tasks.ceph:Creating RBD pool 2020-02-05T16:59:06.276 INFO:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd pool create rbd 8 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.484+0000 7f3b18223700 1 Processor -- start 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.484+0000 7f3b18223700 1 -- start start 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b18223700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b1008f240 0x7f3b100908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b18223700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3b100a9e90 0x7f3b100a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b18223700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b100aab90 con 0x7f3b1008fff0 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b18223700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1008f670 con 0x7f3b100a9e90 2020-02-05T16:59:06.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b18223700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b10090650 con 0x7f3b1008f240 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b16fc1700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3b100a9e90 0x7f3b100a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b16fc1700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3b100a9e90 0x7f3b100a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44142/0 (socket says 172.21.15.8:44142) 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b16fc1700 1 -- 172.21.15.8:0/2975585283 learned_addr learned my addr 172.21.15.8:0/2975585283 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.485+0000 7f3b15fbf700 1 --2- 172.21.15.8:0/2975585283 >> v2:172.21.15.164:3300/0 conn(0x7f3b1008f240 0x7f3b100908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 -- 172.21.15.8:0/2975585283 >> v2:172.21.15.164:3300/0 conn(0x7f3b1008f240 msgr2=0x7f3b100908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:06.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> v2:172.21.15.164:3300/0 conn(0x7f3b1008f240 0x7f3b100908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 -- 172.21.15.8:0/2975585283 >> v1:172.21.15.164:6789/0 conn(0x7f3b1008fff0 legacy=0x7f3b100a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 -- 172.21.15.8:0/2975585283 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b100a86e0 con 0x7f3b100a9e90 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> v2:172.21.15.8:3300/0 conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3b100a9620 tx=0x7f3b080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> v2:172.21.15.8:3300/0 conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3b100a9620 tx=0x7f3b080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.486+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f3b100a9620 tx=0x7f3b080102b0).ready entity=mon.0 client_cookie=617b9482eab96175 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3b100a9620 tx=0x7f3b080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b157be700 1 -- 172.21.15.8:0/2975585283 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b0800a160 con 0x7f3b100a9e90 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3b100a9620 tx=0x7f3b080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f3b100a9620 tx=0x7f3b080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b157be700 1 -- 172.21.15.8:0/2975585283 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b08011bf0 con 0x7f3b100a9e90 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b157be700 1 -- 172.21.15.8:0/2975585283 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b0800a740 con 0x7f3b100a9e90 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b18223700 1 -- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 msgr2=0x7f3b100a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.487+0000 7f3b18223700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f3b100a9620 tx=0x7f3b080102b0).stop 2020-02-05T16:59:06.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.488+0000 7f3b18223700 1 -- 172.21.15.8:0/2975585283 shutdown_connections 2020-02-05T16:59:06.498 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.488+0000 7f3b18223700 1 --2- 172.21.15.8:0/2975585283 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3b100a9e90 0x7f3b100a8190 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.498 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.488+0000 7f3b18223700 1 --2- 172.21.15.8:0/2975585283 >> v2:172.21.15.164:3300/0 conn(0x7f3b1008f240 0x7f3b100908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:06.498 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.488+0000 7f3b18223700 1 -- 172.21.15.8:0/2975585283 >> 172.21.15.8:0/2975585283 conn(0x7f3b100600f0 msgr2=0x7f3b1005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:06.498 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.489+0000 7f3b18223700 1 -- 172.21.15.8:0/2975585283 shutdown_connections 2020-02-05T16:59:06.498 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.489+0000 7f3b18223700 1 -- 172.21.15.8:0/2975585283 wait complete. 2020-02-05T16:59:06.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.493+0000 7f3b18223700 1 Processor -- start 2020-02-05T16:59:06.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.493+0000 7f3b18223700 1 -- start start 2020-02-05T16:59:06.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.495+0000 7f3b18223700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b100a0160 0x7f3b101338d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:06.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.495+0000 7f3b18223700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b1013b6a0 0x7f3b1013dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:06.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.495+0000 7f3b18223700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b10132d80 con 0x7f3b10060c90 2020-02-05T16:59:06.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.495+0000 7f3b18223700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b1009f6a0 con 0x7f3b1013b6a0 2020-02-05T16:59:06.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.495+0000 7f3b18223700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1009f900 con 0x7f3b100a0160 2020-02-05T16:59:06.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b15fbf700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f3b10060c90 0x7f3b100a0cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45528/0 (socket says 172.21.15.8:45528) 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b15fbf700 1 -- 172.21.15.8:0/1809648533 learned_addr learned my addr 172.21.15.8:0/1809648533 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b167c0700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b100a0160 0x7f3b101338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> v2:172.21.15.164:3300/0 conn(0x7f3b1013b6a0 0x7f3b1013dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894436282 0 0) 0x7f3b10132d80 con 0x7f3b10060c90 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.496+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af8003230 con 0x7f3b10060c90 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2742785244 0 0) 0x7f3af8003230 con 0x7f3b10060c90 2020-02-05T16:59:06.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b10132d80 con 0x7f3b10060c90 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f3b0c001490 con 0x7f3b10060c90 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 -- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b100a0160 msgr2=0x7f3b101338d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b100a0160 0x7f3b101338d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 -- 172.21.15.8:0/1809648533 >> v1:172.21.15.8:6789/0 conn(0x7f3b10060c90 legacy=0x7f3b100a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 -- 172.21.15.8:0/1809648533 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10077710 con 0x7f3b1013b6a0 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> v2:172.21.15.164:3300/0 conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.505 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.497+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> v2:172.21.15.164:3300/0 conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b08005c30 con 0x7f3b1013b6a0 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b0800a360 con 0x7f3b1013b6a0 2020-02-05T16:59:06.506 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b100a14b0 con 0x7f3b1013b6a0 2020-02-05T16:59:06.507 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b0800adf0 con 0x7f3b1013b6a0 2020-02-05T16:59:06.507 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.498+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b100610c0 con 0x7f3b1013b6a0 2020-02-05T16:59:06.507 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.500+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.507 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.500+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b10056640 con 0x7f3b1013b6a0 2020-02-05T16:59:06.508 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.500+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f3b08007150 con 0x7f3b1013b6a0 2020-02-05T16:59:06.508 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.500+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.508 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.501+0000 7f3b02ffd700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 0x7f3af803c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:06.509 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.501+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f3b08049e50 con 0x7f3b1013b6a0 2020-02-05T16:59:06.509 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.501+0000 7f3b167c0700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 0x7f3af803c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:06.509 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.502+0000 7f3b167c0700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 0x7f3af803c850 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:06.511 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.509+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:06.511 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.509+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3b080319a0 con 0x7f3b1013b6a0 2020-02-05T16:59:06.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:06.669+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f3b10068f50 con 0x7f3b1013b6a0 2020-02-05T16:59:07.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.563+0000 7f3b16fc1700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.563+0000 7f3b02ffd700 1 -- 172.21.15.8:0/1809648533 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f3b08031c00 con 0x7f3b1013b6a0 2020-02-05T16:59:07.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.566+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 msgr2=0x7f3af803c850 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.566+0000 7f3b18223700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 0x7f3af803c850 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.566+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 msgr2=0x7f3b1013dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.566+0000 7f3b18223700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f3b100a0930 tx=0x7f3b0c0078a0).stop 2020-02-05T16:59:07.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.567+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 shutdown_connections 2020-02-05T16:59:07.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.567+0000 7f3b18223700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3af803a340 0x7f3af803c850 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.568+0000 7f3b18223700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b1013b6a0 0x7f3b1013dae0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.568+0000 7f3b18223700 1 --2- 172.21.15.8:0/1809648533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b100a0160 0x7f3b101338d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.568+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 >> 172.21.15.8:0/1809648533 conn(0x7f3b100908f0 msgr2=0x7f3b1005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:07.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.568+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 shutdown_connections 2020-02-05T16:59:07.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.568+0000 7f3b18223700 1 -- 172.21.15.8:0/1809648533 wait complete. 2020-02-05T16:59:07.572 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'rbd' created 2020-02-05T16:59:07.590 INFO:teuthology.orchestra.run.smithi008:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2020-02-05T16:59:07.811 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.801+0000 7f80777db700 1 Processor -- start 2020-02-05T16:59:07.811 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.801+0000 7f80777db700 1 -- start start 2020-02-05T16:59:07.811 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.802+0000 7f80777db700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.802+0000 7f80777db700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 0x7f80700a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.802+0000 7f80777db700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80700aab90 con 0x7f80700908f0 2020-02-05T16:59:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.802+0000 7f80777db700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8070090d20 con 0x7f80700a9e90 2020-02-05T16:59:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.802+0000 7f80777db700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8070090650 con 0x7f807008fff0 2020-02-05T16:59:07.812 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8076579700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 0x7f80700a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8075d78700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8075d78700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44202/0 (socket says 172.21.15.8:44202) 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8075d78700 1 -- 172.21.15.8:0/2186782232 learned_addr learned my addr 172.21.15.8:0/2186782232 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8076579700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 0x7f80700a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44154/0 (socket says 172.21.15.8:44154) 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.803+0000 7f8075577700 1 --1- 172.21.15.8:0/2186782232 >> v1:172.21.15.8:6789/0 conn(0x7f80700908f0 0x7f80700a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45538/0 (socket says 172.21.15.8:45538) 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 -- 172.21.15.8:0/2186782232 >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 msgr2=0x7f80700a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:07.813 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 0x7f80700a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 -- 172.21.15.8:0/2186782232 >> v1:172.21.15.8:6789/0 conn(0x7f80700908f0 legacy=0x7f80700a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 -- 172.21.15.8:0/2186782232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80700a86e0 con 0x7f807008fff0 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f806c008910 tx=0x7f806c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f806c008910 tx=0x7f806c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.804+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f806c008910 tx=0x7f806c00ff30).ready entity=mon.2 client_cookie=d3d5c5191faa2e01 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f806c008910 tx=0x7f806c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.814 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f806c008910 tx=0x7f806c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.815 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8074d76700 1 -- 172.21.15.8:0/2186782232 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f806c0117a0 con 0x7f807008fff0 2020-02-05T16:59:07.815 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8075d78700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f806c008910 tx=0x7f806c00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.815 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8074d76700 1 -- 172.21.15.8:0/2186782232 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c008d30 con 0x7f807008fff0 2020-02-05T16:59:07.815 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f8074d76700 1 -- 172.21.15.8:0/2186782232 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f806c011d80 con 0x7f807008fff0 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f80777db700 1 -- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 msgr2=0x7f807008f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.805+0000 7f80777db700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f806c008910 tx=0x7f806c00ff30).stop 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 -- 172.21.15.8:0/2186782232 shutdown_connections 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 --2- 172.21.15.8:0/2186782232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807008fff0 0x7f807008f240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 --2- 172.21.15.8:0/2186782232 >> v2:172.21.15.8:3300/0 conn(0x7f80700a9e90 0x7f80700a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 -- 172.21.15.8:0/2186782232 >> 172.21.15.8:0/2186782232 conn(0x7f80700600f0 msgr2=0x7f807005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:07.816 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 -- 172.21.15.8:0/2186782232 shutdown_connections 2020-02-05T16:59:07.817 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.807+0000 7f80777db700 1 -- 172.21.15.8:0/2186782232 wait complete. 2020-02-05T16:59:07.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.810+0000 7f80777db700 1 Processor -- start 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.810+0000 7f80777db700 1 -- start start 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8070066870 0x7f807009ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f807007c990 0x7f807009e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807009e640 0x7f80701385a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80700a0490 con 0x7f807007c990 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80700a06f0 con 0x7f8070066870 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f80777db700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80700a0990 con 0x7f807009e640 2020-02-05T16:59:07.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f8075577700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f807007c990 0x7f807009e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075d78700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8070066870 0x7f807009ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.812+0000 7f8075577700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f807007c990 0x7f807009e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44158/0 (socket says 172.21.15.8:44158) 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 -- 172.21.15.8:0/960144176 learned_addr learned my addr 172.21.15.8:0/960144176 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8076579700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807009e640 0x7f80701385a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 -- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807009e640 msgr2=0x7f80701385a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807009e640 0x7f80701385a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 -- 172.21.15.8:0/960144176 >> v2:172.21.15.164:3300/0 conn(0x7f8070066870 msgr2=0x7f807009ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> v2:172.21.15.164:3300/0 conn(0x7f8070066870 0x7f807009ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 -- 172.21.15.8:0/960144176 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8070138af0 con 0x7f807007c990 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.813+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> v2:172.21.15.8:3300/0 conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> v2:172.21.15.8:3300/0 conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:07.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80600048d0 con 0x7f807007c990 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80600044f0 con 0x7f807007c990 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8070138dc0 con 0x7f807007c990 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.814+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8060012390 con 0x7f807007c990 2020-02-05T16:59:07.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.815+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80700650d0 con 0x7f807007c990 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.816+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.816+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8070056640 con 0x7f807007c990 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.816+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f80600126a0 con 0x7f807007c990 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.816+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.817+0000 7f80667fc700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 0x7f805803be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:07.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.817+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7f8060049600 con 0x7f807007c990 2020-02-05T16:59:07.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.818+0000 7f8075d78700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 0x7f805803be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:07.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.818+0000 7f8075d78700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 0x7f805803be70 crc :-1 s=READY pgs=38 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:07.826 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.824+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:07.827 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.824+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f8060031150 con 0x7f807007c990 2020-02-05T16:59:07.987 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:07.985+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8070068f50 con 0x7f807007c990 2020-02-05T16:59:08.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.567+0000 7f8075577700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.568+0000 7f80667fc700 1 -- 172.21.15.8:0/960144176 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f80600313b0 con 0x7f807007c990 2020-02-05T16:59:08.574 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.572+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 msgr2=0x7f805803be70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.574 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.572+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 0x7f805803be70 crc :-1 s=READY pgs=38 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.572+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 msgr2=0x7f807009e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.572+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f80700a08b0 tx=0x7f8060010a50).stop 2020-02-05T16:59:08.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.573+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 shutdown_connections 2020-02-05T16:59:08.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.573+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8058039960 0x7f805803be70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.573+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f807009e640 0x7f80701385a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.573+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> v2:172.21.15.164:3300/0 conn(0x7f8070066870 0x7f807009ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.574+0000 7f80777db700 1 --2- 172.21.15.8:0/960144176 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f807007c990 0x7f807009e0f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.574+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 >> 172.21.15.8:0/960144176 conn(0x7f807007d5a0 msgr2=0x7f80700ab410 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:08.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.574+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 shutdown_connections 2020-02-05T16:59:08.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.574+0000 7f80777db700 1 -- 172.21.15.8:0/960144176 wait complete. 2020-02-05T16:59:08.580 INFO:teuthology.orchestra.run.smithi008.stderr:enabled application 'rbd' on pool 'rbd' 2020-02-05T16:59:08.602 INFO:tasks.ceph:Setting up CephFS filesystem... 2020-02-05T16:59:08.608 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.839+0000 7f3fdbe67700 1 Processor -- start 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.839+0000 7f3fdbe67700 1 -- start start 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.840+0000 7f3fdbe67700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd408f240 0x7f3fd40908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.840+0000 7f3fdbe67700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 0x7f3fd40a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.840+0000 7f3fdbe67700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd40aab90 con 0x7f3fd408fff0 2020-02-05T16:59:08.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.840+0000 7f3fdbe67700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd408f670 con 0x7f3fd408f240 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.840+0000 7f3fdbe67700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd4090650 con 0x7f3fd40a9e90 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd408f240 0x7f3fd40908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd408f240 0x7f3fd40908f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44166/0 (socket says 172.21.15.8:44166) 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 -- 172.21.15.8:0/2613272159 learned_addr learned my addr 172.21.15.8:0/2613272159 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fdac05700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 0x7f3fd40a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 -- 172.21.15.8:0/2613272159 >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 msgr2=0x7f3fd40a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:08.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 0x7f3fd40a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 -- 172.21.15.8:0/2613272159 >> v1:172.21.15.8:6789/0 conn(0x7f3fd408fff0 legacy=0x7f3fd40a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 -- 172.21.15.8:0/2613272159 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd40a86e0 con 0x7f3fd408f240 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fdac05700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 0x7f3fd40a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.8:3300/0 conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.8:3300/0 conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.841+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).ready entity=mon.0 client_cookie=e48d5f93e8c9d7b5 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:08.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9402700 1 -- 172.21.15.8:0/2613272159 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc4011950 con 0x7f3fd408f240 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9402700 1 -- 172.21.15.8:0/2613272159 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4011570 con 0x7f3fd408f240 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fd9402700 1 -- 172.21.15.8:0/2613272159 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc400a160 con 0x7f3fd408f240 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fdbe67700 1 -- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 msgr2=0x7f3fd40908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.842+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f3fd40a9620 tx=0x7f3fc400ff30).stop 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 -- 172.21.15.8:0/2613272159 shutdown_connections 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/2613272159 >> v2:172.21.15.164:3300/0 conn(0x7f3fd40a9e90 0x7f3fd40a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/2613272159 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd408f240 0x7f3fd40908f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 -- 172.21.15.8:0/2613272159 >> 172.21.15.8:0/2613272159 conn(0x7f3fd40600f0 msgr2=0x7f3fd405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 -- 172.21.15.8:0/2613272159 shutdown_connections 2020-02-05T16:59:08.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.843+0000 7f3fdbe67700 1 -- 172.21.15.8:0/2613272159 wait complete. 2020-02-05T16:59:08.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.846+0000 7f3fdbe67700 1 Processor -- start 2020-02-05T16:59:08.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.847+0000 7f3fdbe67700 1 -- start start 2020-02-05T16:59:08.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.848+0000 7f3fdbe67700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd40a1e70 0x7f3fd409e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:08.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.848+0000 7f3fdbe67700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fd409e590 0x7f3fd413d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:08.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fdbe67700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fd40a73f0 con 0x7f3fd4079010 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fdbe67700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fd40a0e80 con 0x7f3fd40a1e70 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fdbe67700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fd40a1120 con 0x7f3fd409e590 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fd9c03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd40a1e70 0x7f3fd409e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fd9c03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3fd40a1e70 0x7f3fd409e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44174/0 (socket says 172.21.15.8:44174) 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fd9c03700 1 -- 172.21.15.8:0/752532538 learned_addr learned my addr 172.21.15.8:0/752532538 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.849+0000 7f3fdac05700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fd409e590 0x7f3fd413d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:08.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 -- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fd409e590 msgr2=0x7f3fd413d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fd409e590 0x7f3fd413d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:08.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 -- 172.21.15.8:0/752532538 >> v1:172.21.15.8:6789/0 conn(0x7f3fd4079010 legacy=0x7f3fd40a6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:08.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 -- 172.21.15.8:0/752532538 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd409ee90 con 0x7f3fd40a1e70 2020-02-05T16:59:08.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> v2:172.21.15.8:3300/0 conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> v2:172.21.15.8:3300/0 conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.850+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc4007340 con 0x7f3fd40a1e70 2020-02-05T16:59:08.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fc4011b50 con 0x7f3fd40a1e70 2020-02-05T16:59:08.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fc40056c0 con 0x7f3fd40a1e70 2020-02-05T16:59:08.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fd40a1660 con 0x7f3fd40a1e70 2020-02-05T16:59:08.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.851+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fd40765c0 con 0x7f3fd40a1e70 2020-02-05T16:59:08.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.852+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.854+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f3fc40059d0 con 0x7f3fd40a1e70 2020-02-05T16:59:08.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.854+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.855+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fd40a0c70 con 0x7f3fd40a1e70 2020-02-05T16:59:08.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.855+0000 7f3fcaffd700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 0x7f3fbc03be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:08.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.855+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f3fc4049d20 con 0x7f3fd40a1e70 2020-02-05T16:59:08.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.856+0000 7f3fda404700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 0x7f3fbc03be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:08.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.856+0000 7f3fda404700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 0x7f3fbc03be70 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:08.863 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.861+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:08.863 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:08.861+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3fc4031870 con 0x7f3fd40a1e70 2020-02-05T16:59:09.030 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.019+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3fd4056640 con 0x7f3fd40a1e70 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.021+0000 7f3fd9c03700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.021+0000 7f3fcaffd700 1 -- 172.21.15.8:0/752532538 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11778 (secure 0 0 0) 0x7f3fc4031ad0 con 0x7f3fd40a1e70 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 msgr2=0x7f3fbc03be70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 0x7f3fbc03be70 crc :-1 s=READY pgs=39 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 msgr2=0x7f3fd409e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.031 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f3fd40a1040 tx=0x7f3fc4011df0).stop 2020-02-05T16:59:09.032 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 shutdown_connections 2020-02-05T16:59:09.032 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3fbc039960 0x7f3fbc03be70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.032 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3fd409e590 0x7f3fd413d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.032 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 --2- 172.21.15.8:0/752532538 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3fd40a1e70 0x7f3fd409e040 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.033 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.023+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 >> 172.21.15.8:0/752532538 conn(0x7f3fd405ae20 msgr2=0x7f3fd407eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:09.033 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.024+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 shutdown_connections 2020-02-05T16:59:09.033 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.024+0000 7f3fdbe67700 1 -- 172.21.15.8:0/752532538 wait complete. 2020-02-05T16:59:09.033 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:09.034 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":16,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:08.554912+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:09.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:06.700677+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-02-05T16:59:09.046 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-02-05T16:59:09.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.266+0000 7f086a9a0700 1 Processor -- start 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.266+0000 7f086a9a0700 1 -- start start 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.267+0000 7f086a9a0700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f08640ace50 0x7f08640aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.267+0000 7f086a9a0700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.267+0000 7f086a9a0700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640aa980 con 0x7f08640abd50 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.267+0000 7f086a9a0700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0864148c70 con 0x7f08640ace50 2020-02-05T16:59:09.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.267+0000 7f086a9a0700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0864148f10 con 0x7f0864145c20 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.268+0000 7f086bc02700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.268+0000 7f086bc02700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44230/0 (socket says 172.21.15.8:44230) 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.268+0000 7f086bc02700 1 -- 172.21.15.8:0/1019685230 learned_addr learned my addr 172.21.15.8:0/1019685230 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.268+0000 7f086a19f700 1 --2- 172.21.15.8:0/1019685230 >> v2:172.21.15.164:3300/0 conn(0x7f08640ace50 0x7f08640aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 -- 172.21.15.8:0/1019685230 >> v2:172.21.15.164:3300/0 conn(0x7f08640ace50 msgr2=0x7f08640aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:09.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> v2:172.21.15.164:3300/0 conn(0x7f08640ace50 0x7f08640aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 -- 172.21.15.8:0/1019685230 >> v1:172.21.15.164:6789/0 conn(0x7f08640abd50 legacy=0x7f08641491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 -- 172.21.15.8:0/1019685230 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08641454a0 con 0x7f0864145c20 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f085c008c90 tx=0x7f085c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f085c008c90 tx=0x7f085c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.269+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f085c008c90 tx=0x7f085c0102b0).ready entity=mon.2 client_cookie=b95784465bff1b2f server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f085c008c90 tx=0x7f085c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f085c008c90 tx=0x7f085c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086919d700 1 -- 172.21.15.8:0/1019685230 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f085c00a350 con 0x7f0864145c20 2020-02-05T16:59:09.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086bc02700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f085c008c90 tx=0x7f085c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086919d700 1 -- 172.21.15.8:0/1019685230 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f085c011c30 con 0x7f0864145c20 2020-02-05T16:59:09.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086919d700 1 -- 172.21.15.8:0/1019685230 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f085c00a930 con 0x7f0864145c20 2020-02-05T16:59:09.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086a9a0700 1 -- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 msgr2=0x7f0864146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.270+0000 7f086a9a0700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f085c008c90 tx=0x7f085c0102b0).stop 2020-02-05T16:59:09.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 -- 172.21.15.8:0/1019685230 shutdown_connections 2020-02-05T16:59:09.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 --2- 172.21.15.8:0/1019685230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0864145c20 0x7f0864146050 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 --2- 172.21.15.8:0/1019685230 >> v2:172.21.15.164:3300/0 conn(0x7f08640ace50 0x7f08640aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:09.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 -- 172.21.15.8:0/1019685230 >> 172.21.15.8:0/1019685230 conn(0x7f0864061510 msgr2=0x7f08640ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:09.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 -- 172.21.15.8:0/1019685230 shutdown_connections 2020-02-05T16:59:09.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.271+0000 7f086a9a0700 1 -- 172.21.15.8:0/1019685230 wait complete. 2020-02-05T16:59:09.284 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.275+0000 7f086a9a0700 1 Processor -- start 2020-02-05T16:59:09.284 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.275+0000 7f086a9a0700 1 -- start start 2020-02-05T16:59:09.284 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.277+0000 7f086a9a0700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f08640a5c60 0x7f086407a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:09.284 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.277+0000 7f086a9a0700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 0x7f086411e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.277+0000 7f086a9a0700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08640a3270 con 0x7f086411eaf0 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.277+0000 7f086a9a0700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f086407ba10 con 0x7f08640a5c60 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.277+0000 7f086a9a0700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f086407c8a0 con 0x7f086407a6f0 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.278+0000 7f086a19f700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f08640a5c60 0x7f086407a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.278+0000 7f086a19f700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f08640a5c60 0x7f086407a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44186/0 (socket says 172.21.15.8:44186) 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.278+0000 7f086a19f700 1 -- 172.21.15.8:0/371927105 learned_addr learned my addr 172.21.15.8:0/371927105 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:09.285 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.278+0000 7f086999e700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 0x7f086411e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 -- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 msgr2=0x7f086411e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 0x7f086411e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 -- 172.21.15.8:0/371927105 >> v1:172.21.15.164:6789/0 conn(0x7f086411eaf0 legacy=0x7f08640247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 -- 172.21.15.8:0/371927105 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f086407c5b0 con 0x7f08640a5c60 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086999e700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 0x7f086411e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:09.286 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> v2:172.21.15.8:3300/0 conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> v2:172.21.15.8:3300/0 conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0860007160 con 0x7f08640a5c60 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.279+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.280+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0860005ab0 con 0x7f08640a5c60 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.280+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08640a9a90 con 0x7f08640a5c60 2020-02-05T16:59:09.287 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.280+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.280+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0860012610 con 0x7f08640a5c60 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.280+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f086411ef20 con 0x7f08640a5c60 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.281+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.281+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0864056640 con 0x7f08640a5c60 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.282+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f0860012970 con 0x7f08640a5c60 2020-02-05T16:59:09.288 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.282+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.290 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.282+0000 7f085affd700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 0x7f084c03bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:09.290 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.282+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f0860049930 con 0x7f08640a5c60 2020-02-05T16:59:09.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.283+0000 7f086999e700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 0x7f084c03bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.283+0000 7f086999e700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 0x7f084c03bf70 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.286+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.286+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f086000d0b0 con 0x7f08640a5c60 2020-02-05T16:59:09.445 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.443+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0864066210 con 0x7f08640a5c60 2020-02-05T16:59:09.446 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.444+0000 7f086a19f700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.446 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.444+0000 7f085affd700 1 -- 172.21.15.8:0/371927105 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0860031460 con 0x7f08640a5c60 2020-02-05T16:59:09.447 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:09.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.445+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 msgr2=0x7f084c03bf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.445+0000 7f086a9a0700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 0x7f084c03bf70 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.445+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 msgr2=0x7f086407a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.445+0000 7f086a9a0700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7f086407af50 tx=0x7f0860010e20).stop 2020-02-05T16:59:09.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 shutdown_connections 2020-02-05T16:59:09.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f084c039a60 0x7f084c03bf70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f08640a5c60 0x7f086407a1a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 --2- 172.21.15.8:0/371927105 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f086407a6f0 0x7f086411e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 >> 172.21.15.8:0/371927105 conn(0x7f086405e920 msgr2=0x7f086407d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:09.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 shutdown_connections 2020-02-05T16:59:09.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.446+0000 7f086a9a0700 1 -- 172.21.15.8:0/371927105 wait complete. 2020-02-05T16:59:09.472 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2020-02-05T16:59:09.706 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.694+0000 7ff4cff60700 1 Processor -- start 2020-02-05T16:59:09.706 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.694+0000 7ff4cff60700 1 -- start start 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.695+0000 7ff4cff60700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4c808fff0 0x7ff4c808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.695+0000 7ff4cff60700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.695+0000 7ff4cff60700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80aa3e0 con 0x7ff4c80a7990 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.695+0000 7ff4cff60700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4c8090d20 con 0x7ff4c808fff0 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.695+0000 7ff4cff60700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8090650 con 0x7ff4c80908f0 2020-02-05T16:59:09.707 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.696+0000 7ff4cecfe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7ff4c80a7990 0x7ff4c80a7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45580/0 (socket says 172.21.15.8:45580) 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.696+0000 7ff4cecfe700 1 -- 172.21.15.8:0/3067765246 learned_addr learned my addr 172.21.15.8:0/3067765246 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.696+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.696+0000 7ff4ce4fd700 1 --2- 172.21.15.8:0/3067765246 >> v2:172.21.15.164:3300/0 conn(0x7ff4c808fff0 0x7ff4c808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cd4fb700 1 -- 172.21.15.8:0/3067765246 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95946215 0 0) 0x7ff4c0001450 con 0x7ff4c80a7990 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cd4fb700 1 -- 172.21.15.8:0/3067765246 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b4003230 con 0x7ff4c80a7990 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/3067765246 >> v2:172.21.15.164:3300/0 conn(0x7ff4c808fff0 msgr2=0x7ff4c808f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> v2:172.21.15.164:3300/0 conn(0x7ff4c808fff0 0x7ff4c808f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:09.708 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/3067765246 >> v1:172.21.15.8:6789/0 conn(0x7ff4c80a7990 legacy=0x7ff4c80a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/3067765246 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c80aab90 con 0x7ff4c80908f0 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.697+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).ready entity=mon.2 client_cookie=3890e5147f1c72bf server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cd4fb700 1 -- 172.21.15.8:0/3067765246 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b80116e0 con 0x7ff4c80908f0 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cd4fb700 1 -- 172.21.15.8:0/3067765246 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b8008c50 con 0x7ff4c80908f0 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cd4fb700 1 -- 172.21.15.8:0/3067765246 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b8011cc0 con 0x7ff4c80908f0 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cff60700 1 -- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 msgr2=0x7ff4c80a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.698+0000 7ff4cff60700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b800fe70).stop 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.699+0000 7ff4cff60700 1 -- 172.21.15.8:0/3067765246 shutdown_connections 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.699+0000 7ff4cff60700 1 --2- 172.21.15.8:0/3067765246 >> v2:172.21.15.164:3300/0 conn(0x7ff4c808fff0 0x7ff4c808f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.699+0000 7ff4cff60700 1 --2- 172.21.15.8:0/3067765246 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c80908f0 0x7ff4c80a9e90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.699+0000 7ff4cff60700 1 -- 172.21.15.8:0/3067765246 >> 172.21.15.8:0/3067765246 conn(0x7ff4c80600f0 msgr2=0x7ff4c805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.700+0000 7ff4cff60700 1 -- 172.21.15.8:0/3067765246 shutdown_connections 2020-02-05T16:59:09.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.700+0000 7ff4cff60700 1 -- 172.21.15.8:0/3067765246 wait complete. 2020-02-05T16:59:09.712 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.704+0000 7ff4cff60700 1 Processor -- start 2020-02-05T16:59:09.712 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.704+0000 7ff4cff60700 1 -- start start 2020-02-05T16:59:09.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.706+0000 7ff4cff60700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.706+0000 7ff4cff60700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80a2050 con 0x7ff4c80a0e80 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.706+0000 7ff4cff60700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4c80a1410 con 0x7ff4c80a3950 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.706+0000 7ff4cff60700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c80a06a0 con 0x7ff4c8121670 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.707+0000 7ff4ce4fd700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff4c80a3950 0x7ff4c809edb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:58068/0 (socket says 172.21.15.8:58068) 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.707+0000 7ff4cdcfc700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.707+0000 7ff4ce4fd700 1 -- 172.21.15.8:0/601390133 learned_addr learned my addr 172.21.15.8:0/601390133 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.707+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708586148 0 0) 0x7ff4c80a1410 con 0x7ff4c80a3950 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.707+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b0003230 con 0x7ff4c80a3950 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403156603 0 0) 0x7ff4c80a2050 con 0x7ff4c80a0e80 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/601390133 >> v1:172.21.15.164:6789/0 conn(0x7ff4c80a3950 legacy=0x7ff4c809edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/601390133 >> v1:172.21.15.8:6789/0 conn(0x7ff4c80a0e80 legacy=0x7ff4c813f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 -- 172.21.15.8:0/601390133 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c807cab0 con 0x7ff4c8121670 2020-02-05T16:59:09.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.708+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b8011cc0 con 0x7ff4c8121670 2020-02-05T16:59:09.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4b80118e0 con 0x7ff4c8121670 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c8140f70 con 0x7ff4c8121670 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4b8007530 con 0x7ff4c8121670 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.709+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c8141180 con 0x7ff4c8121670 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.710+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.711+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7ff4b80056c0 con 0x7ff4c8121670 2020-02-05T16:59:09.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.711+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.711+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c8056640 con 0x7ff4c8121670 2020-02-05T16:59:09.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.711+0000 7ff4beffd700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 0x7ff4b003c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:09.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.712+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7ff4b8049790 con 0x7ff4c8121670 2020-02-05T16:59:09.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.712+0000 7ff4ce4fd700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 0x7ff4b003c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:09.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.713+0000 7ff4ce4fd700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 0x7ff4b003c2b0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:09.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.723+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.723+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7ff4b80312a0 con 0x7ff4c8121670 2020-02-05T16:59:09.887 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.885+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff4c8068f50 con 0x7ff4c8121670 2020-02-05T16:59:09.888 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.886+0000 7ff4cdcfc700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.886+0000 7ff4beffd700 1 -- 172.21.15.8:0/601390133 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+64 (secure 0 0 0) 0x7ff4b8031500 con 0x7ff4c8121670 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout:[ 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout: { 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout: "poolnum": 1, 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout: "poolname": "rbd" 2020-02-05T16:59:09.889 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T16:59:09.890 INFO:teuthology.orchestra.run.smithi008.stdout:] 2020-02-05T16:59:09.891 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 msgr2=0x7ff4b003c2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.891 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 0x7ff4b003c2b0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.891 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 msgr2=0x7ff4c807c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:09.891 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7ff4b8008910 tx=0x7ff4b8008c50).stop 2020-02-05T16:59:09.891 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 shutdown_connections 2020-02-05T16:59:09.892 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff4b0039da0 0x7ff4b003c2b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.892 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 --2- 172.21.15.8:0/601390133 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c8121670 0x7ff4c807c560 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:09.892 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 >> 172.21.15.8:0/601390133 conn(0x7ff4c80ab730 msgr2=0x7ff4c807ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:09.892 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.888+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 shutdown_connections 2020-02-05T16:59:09.892 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:09.889+0000 7ff4cff60700 1 -- 172.21.15.8:0/601390133 wait complete. 2020-02-05T16:59:09.909 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2020-02-05T16:59:09.909 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2020-02-05T16:59:10.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.139+0000 7fc58ecac700 1 Processor -- start 2020-02-05T16:59:10.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.140+0000 7fc58ecac700 1 -- start start 2020-02-05T16:59:10.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.140+0000 7fc58ecac700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc58808fff0 0x7fc58808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:10.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.140+0000 7fc58ecac700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58ecac700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880aa3e0 con 0x7fc5880a7990 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58ecac700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc588090d20 con 0x7fc58808fff0 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58ecac700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc588090650 con 0x7fc5880908f0 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58ca48700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58da4a700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fc5880a7990 0x7fc5880a7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45592/0 (socket says 172.21.15.8:45592) 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58da4a700 1 -- 172.21.15.8:0/91056207 learned_addr learned my addr 172.21.15.8:0/91056207 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.141+0000 7fc58d249700 1 --2- 172.21.15.8:0/91056207 >> v2:172.21.15.164:3300/0 conn(0x7fc58808fff0 0x7fc58808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.142+0000 7fc58ca48700 1 -- 172.21.15.8:0/91056207 >> v2:172.21.15.164:3300/0 conn(0x7fc58808fff0 msgr2=0x7fc58808f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.142+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> v2:172.21.15.164:3300/0 conn(0x7fc58808fff0 0x7fc58808f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.142+0000 7fc58ca48700 1 -- 172.21.15.8:0/91056207 >> v1:172.21.15.8:6789/0 conn(0x7fc5880a7990 legacy=0x7fc5880a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.142+0000 7fc58ca48700 1 -- 172.21.15.8:0/91056207 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880aab90 con 0x7fc5880908f0 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.142+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc578008910 tx=0x7fc57800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc578008910 tx=0x7fc57800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fc578008910 tx=0x7fc57800fe70).ready entity=mon.2 client_cookie=566a32e552bf8614 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fc578008910 tx=0x7fc57800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fc578008910 tx=0x7fc57800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ca48700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fc578008910 tx=0x7fc57800fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc57ffff700 1 -- 172.21.15.8:0/91056207 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5780116e0 con 0x7fc5880908f0 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc57ffff700 1 -- 172.21.15.8:0/91056207 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc578008c50 con 0x7fc5880908f0 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc57ffff700 1 -- 172.21.15.8:0/91056207 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc578011cc0 con 0x7fc5880908f0 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ecac700 1 -- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 msgr2=0x7fc5880a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.143+0000 7fc58ecac700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fc578008910 tx=0x7fc57800fe70).stop 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 -- 172.21.15.8:0/91056207 shutdown_connections 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 --2- 172.21.15.8:0/91056207 >> v2:172.21.15.164:3300/0 conn(0x7fc58808fff0 0x7fc58808f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 --2- 172.21.15.8:0/91056207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5880908f0 0x7fc5880a9e90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 -- 172.21.15.8:0/91056207 >> 172.21.15.8:0/91056207 conn(0x7fc5880600f0 msgr2=0x7fc58805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 -- 172.21.15.8:0/91056207 shutdown_connections 2020-02-05T16:59:10.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.145+0000 7fc58ecac700 1 -- 172.21.15.8:0/91056207 wait complete. 2020-02-05T16:59:10.156 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.149+0000 7fc58ecac700 1 Processor -- start 2020-02-05T16:59:10.156 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.149+0000 7fc58ecac700 1 -- start start 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.151+0000 7fc58ecac700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fc588069c60 0x7fc5881364c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.151+0000 7fc58ecac700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880a2830 con 0x7fc5880a17c0 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.151+0000 7fc58ecac700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880a1bf0 con 0x7fc588136a10 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.151+0000 7fc58ecac700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5880a8470 con 0x7fc588069c60 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.152+0000 7fc58d249700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fc588069c60 0x7fc5881364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.152+0000 7fc58d249700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fc588069c60 0x7fc5881364c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44212/0 (socket says 172.21.15.8:44212) 2020-02-05T16:59:10.157 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.152+0000 7fc58d249700 1 -- 172.21.15.8:0/1825502906 learned_addr learned my addr 172.21.15.8:0/1825502906 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 -- 172.21.15.8:0/1825502906 >> v1:172.21.15.164:6789/0 conn(0x7fc588136a10 legacy=0x7fc5880a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 -- 172.21.15.8:0/1825502906 >> v1:172.21.15.8:6789/0 conn(0x7fc5880a17c0 legacy=0x7fc58813dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668843939 0 0) 0x7fc5880a1bf0 con 0x7fc588136a10 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 -- 172.21.15.8:0/1825502906 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58807f620 con 0x7fc588069c60 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> v2:172.21.15.8:3300/0 conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> v2:172.21.15.8:3300/0 conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.153+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.158 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.159 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc58400ebc0 con 0x7fc588069c60 2020-02-05T16:59:10.159 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.161 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc588136e40 con 0x7fc588069c60 2020-02-05T16:59:10.161 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.161 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58400e7e0 con 0x7fc588069c60 2020-02-05T16:59:10.161 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc58400ee00 con 0x7fc588069c60 2020-02-05T16:59:10.161 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.154+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc588060ae0 con 0x7fc588069c60 2020-02-05T16:59:10.163 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.155+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.163 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.156+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fc584007180 con 0x7fc588069c60 2020-02-05T16:59:10.163 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.156+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.164 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.156+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc588061ac0 con 0x7fc588069c60 2020-02-05T16:59:10.164 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.157+0000 7fc57dffb700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 0x7fc56803bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:10.164 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.157+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fc5840498e0 con 0x7fc588069c60 2020-02-05T16:59:10.164 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.157+0000 7fc58ca48700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 0x7fc56803bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.164 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.158+0000 7fc58ca48700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 0x7fc56803bd20 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.167 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.165+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.168 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.165+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fc5840313b0 con 0x7fc588069c60 2020-02-05T16:59:10.325 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.322+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fc588056640 con 0x7fc588069c60 2020-02-05T16:59:10.592 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.590+0000 7fc58d249700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.593 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.590+0000 7fc57dffb700 1 -- 172.21.15.8:0/1825502906 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc584007490 con 0x7fc588069c60 2020-02-05T16:59:10.597 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 msgr2=0x7fc56803bd20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.597 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 0x7fc56803bd20 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.597 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 msgr2=0x7fc5881364c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fc584003070 tx=0x7fc584010e40).stop 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 shutdown_connections 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc588069c60 0x7fc5881364c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.594+0000 7fc58ecac700 1 --2- 172.21.15.8:0/1825502906 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc568039810 0x7fc56803bd20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.595+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 >> 172.21.15.8:0/1825502906 conn(0x7fc58808fe50 msgr2=0x7fc5880a3750 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.595+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 shutdown_connections 2020-02-05T16:59:10.598 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.595+0000 7fc58ecac700 1 -- 172.21.15.8:0/1825502906 wait complete. 2020-02-05T16:59:10.599 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'cephfs_metadata' created 2020-02-05T16:59:10.619 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2020-02-05T16:59:10.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.844+0000 7f0e16081700 1 Processor -- start 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.844+0000 7f0e16081700 1 -- start start 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e16081700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 0x7f0e100908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e16081700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e100a9e90 0x7f0e100a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e16081700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e100aab90 con 0x7f0e1008fff0 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e16081700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e1008f670 con 0x7f0e100a9e90 2020-02-05T16:59:10.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e16081700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e10090650 con 0x7f0e1008f240 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e14e1f700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e100a9e90 0x7f0e100a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e14e1f700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e100a9e90 0x7f0e100a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44224/0 (socket says 172.21.15.8:44224) 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e14e1f700 1 -- 172.21.15.8:0/3667018180 learned_addr learned my addr 172.21.15.8:0/3667018180 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.845+0000 7f0e0f7fe700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 0x7f0e100908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e14e1f700 1 -- 172.21.15.8:0/3667018180 >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 msgr2=0x7f0e100908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 0x7f0e100908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:10.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e14e1f700 1 -- 172.21.15.8:0/3667018180 >> v1:172.21.15.164:6789/0 conn(0x7f0e1008fff0 legacy=0x7f0e100a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e0effd700 1 -- 172.21.15.8:0/3667018180 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387248479 0 0) 0x7f0e100aab90 con 0x7f0e1008fff0 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e14e1f700 1 -- 172.21.15.8:0/3667018180 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e100a86e0 con 0x7f0e100a9e90 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e0f7fe700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 0x7f0e100908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.846+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.8:3300/0 conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0e100a9620 tx=0x7f0e080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.8:3300/0 conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0e100a9620 tx=0x7f0e080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f0e100a9620 tx=0x7f0e080102b0).ready entity=mon.0 client_cookie=b7aadf15078f04a8 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f0e100a9620 tx=0x7f0e080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e0effd700 1 -- 172.21.15.8:0/3667018180 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0800a160 con 0x7f0e100a9e90 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f0e100a9620 tx=0x7f0e080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e14e1f700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f0e100a9620 tx=0x7f0e080102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e0effd700 1 -- 172.21.15.8:0/3667018180 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08011bf0 con 0x7f0e100a9e90 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e0effd700 1 -- 172.21.15.8:0/3667018180 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e0800a740 con 0x7f0e100a9e90 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e16081700 1 -- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 msgr2=0x7f0e100a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.847+0000 7f0e16081700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f0e100a9620 tx=0x7f0e080102b0).stop 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 -- 172.21.15.8:0/3667018180 shutdown_connections 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 --2- 172.21.15.8:0/3667018180 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e100a9e90 0x7f0e100a8190 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 --2- 172.21.15.8:0/3667018180 >> v2:172.21.15.164:3300/0 conn(0x7f0e1008f240 0x7f0e100908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 -- 172.21.15.8:0/3667018180 >> 172.21.15.8:0/3667018180 conn(0x7f0e100600f0 msgr2=0x7f0e1005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 -- 172.21.15.8:0/3667018180 shutdown_connections 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.849+0000 7f0e16081700 1 -- 172.21.15.8:0/3667018180 wait complete. 2020-02-05T16:59:10.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.851+0000 7f0e16081700 1 Processor -- start 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 -- start start 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e10138b00 0x7f0e1009ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e10134b40 0x7f0e101278c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e10128240 con 0x7f0e10127e10 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e10138f30 con 0x7f0e10138b00 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.852+0000 7f0e16081700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10138080 con 0x7f0e10134b40 2020-02-05T16:59:10.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.853+0000 7f0e0ffff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e10138b00 0x7f0e1009ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.853+0000 7f0e0f7fe700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e10134b40 0x7f0e101278c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.853+0000 7f0e0ffff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0e10138b00 0x7f0e1009ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44228/0 (socket says 172.21.15.8:44228) 2020-02-05T16:59:10.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.853+0000 7f0e0ffff700 1 -- 172.21.15.8:0/2276041721 learned_addr learned my addr 172.21.15.8:0/2276041721 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:10.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.853+0000 7f0e14e1f700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f0e10127e10 0x7f0e10126790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45612/0 (socket says 172.21.15.8:45612) 2020-02-05T16:59:10.864 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 -- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e10134b40 msgr2=0x7f0e101278c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.864 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e10134b40 0x7f0e101278c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 -- 172.21.15.8:0/2276041721 >> v1:172.21.15.8:6789/0 conn(0x7f0e10127e10 legacy=0x7f0e10126790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812967545 0 0) 0x7f0e10128240 con 0x7f0e10127e10 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 -- 172.21.15.8:0/2276041721 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e10134f70 con 0x7f0e10138b00 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> v2:172.21.15.8:3300/0 conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> v2:172.21.15.8:3300/0 conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.865 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e00006bc0 con 0x7f0e10138b00 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.854+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e000067e0 con 0x7f0e10138b00 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.855+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e100a1ee0 con 0x7f0e10138b00 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.855+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0e00012290 con 0x7f0e10138b00 2020-02-05T16:59:10.866 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.855+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e10126f90 con 0x7f0e10138b00 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.855+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.856+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f0e00012490 con 0x7f0e10138b00 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.856+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.857+0000 7f0e0cff9700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 0x7f0df003bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.857+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f0e00049690 con 0x7f0e10138b00 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.857+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e10056640 con 0x7f0e10138b00 2020-02-05T16:59:10.867 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.857+0000 7f0e0f7fe700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 0x7f0df003bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:10.868 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.858+0000 7f0e0f7fe700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 0x7f0df003bdc0 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:10.868 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.862+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:10.868 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:10.862+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f0e00031180 con 0x7f0e10138b00 2020-02-05T16:59:11.022 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.020+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f0e10068f50 con 0x7f0e10138b00 2020-02-05T16:59:11.606 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.603+0000 7f0e0ffff700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.607 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.603+0000 7f0e0cff9700 1 -- 172.21.15.8:0/2276041721 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v19) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0e000313e0 con 0x7f0e10138b00 2020-02-05T16:59:11.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.614+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 msgr2=0x7f0df003bdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.614+0000 7f0e16081700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 0x7f0df003bdc0 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.614+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 msgr2=0x7f0e1009ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.614+0000 7f0e16081700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).stop 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 shutdown_connections 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0df00398b0 0x7f0df003bdc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0e10138b00 0x7f0e1009ee00 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0x7f0e00000b80 tx=0x7f0e000109e0).stop 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 --2- 172.21.15.8:0/2276041721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e10134b40 0x7f0e101278c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:11.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 >> 172.21.15.8:0/2276041721 conn(0x7f0e10060250 msgr2=0x7f0e100660b0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:11.620 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.615+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 shutdown_connections 2020-02-05T16:59:11.620 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.616+0000 7f0e16081700 1 -- 172.21.15.8:0/2276041721 wait complete. 2020-02-05T16:59:11.620 INFO:teuthology.orchestra.run.smithi008.stderr:pool 'cephfs_data' created 2020-02-05T16:59:11.636 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2020-02-05T16:59:11.871 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.863+0000 7f1f440fe700 1 Processor -- start 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.863+0000 7f1f440fe700 1 -- start start 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f440fe700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f440fe700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0aa3e0 con 0x7f1f3c08f240 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f440fe700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c0aab90 con 0x7f1f3c08fff0 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f440fe700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c08f670 con 0x7f1f3c0908f0 2020-02-05T16:59:11.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f42e9c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f41e9a700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f1f3c08f240 0x7f1f3c0a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45616/0 (socket says 172.21.15.8:45616) 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.864+0000 7f1f41e9a700 1 -- 172.21.15.8:0/3439822019 learned_addr learned my addr 172.21.15.8:0/3439822019 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.865+0000 7f1f42e9c700 1 -- 172.21.15.8:0/3439822019 >> v1:172.21.15.164:6789/0 conn(0x7f1f3c08fff0 legacy=0x7f1f3c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.865+0000 7f1f42e9c700 1 -- 172.21.15.8:0/3439822019 >> v1:172.21.15.8:6789/0 conn(0x7f1f3c08f240 legacy=0x7f1f3c0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.865+0000 7f1f42e9c700 1 -- 172.21.15.8:0/3439822019 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c05b340 con 0x7f1f3c0908f0 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.865+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> v2:172.21.15.8:3300/0 conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1f34008c90 tx=0x7f1f340101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> v2:172.21.15.8:3300/0 conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1f34008c90 tx=0x7f1f340101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f1f34008c90 tx=0x7f1f340101f0).ready entity=mon.0 client_cookie=9ec3527457a9e192 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f1f34008c90 tx=0x7f1f340101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f41699700 1 -- 172.21.15.8:0/3439822019 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3400a160 con 0x7f1f3c0908f0 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f1f34008c90 tx=0x7f1f340101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f41699700 1 -- 172.21.15.8:0/3439822019 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f34011b70 con 0x7f1f3c0908f0 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f42e9c700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7f1f34008c90 tx=0x7f1f340101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f440fe700 1 -- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 msgr2=0x7f1f3c0a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f41699700 1 -- 172.21.15.8:0/3439822019 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f3400a740 con 0x7f1f3c0908f0 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.866+0000 7f1f440fe700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7f1f34008c90 tx=0x7f1f340101f0).stop 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.867+0000 7f1f440fe700 1 -- 172.21.15.8:0/3439822019 shutdown_connections 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.867+0000 7f1f440fe700 1 --2- 172.21.15.8:0/3439822019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f1f3c0908f0 0x7f1f3c0a9e90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.867+0000 7f1f440fe700 1 -- 172.21.15.8:0/3439822019 >> 172.21.15.8:0/3439822019 conn(0x7f1f3c0600f0 msgr2=0x7f1f3c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.868+0000 7f1f440fe700 1 -- 172.21.15.8:0/3439822019 shutdown_connections 2020-02-05T16:59:11.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.868+0000 7f1f440fe700 1 -- 172.21.15.8:0/3439822019 wait complete. 2020-02-05T16:59:11.878 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.870+0000 7f1f440fe700 1 Processor -- start 2020-02-05T16:59:11.878 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.870+0000 7f1f440fe700 1 -- start start 2020-02-05T16:59:11.878 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f440fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1f3c0a3e70 0x7f1f3c0a7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:11.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f440fe700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:11.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f440fe700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3c09e7d0 con 0x7f1f3c0a5ac0 2020-02-05T16:59:11.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f440fe700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f3c09f050 con 0x7f1f3c0a3e70 2020-02-05T16:59:11.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f440fe700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3c0a0b00 con 0x7f1f3c0a5690 2020-02-05T16:59:11.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f42e9c700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f1f3c0a5ac0 0x7f1f3c0a6ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45626/0 (socket says 172.21.15.8:45626) 2020-02-05T16:59:11.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f42e9c700 1 -- 172.21.15.8:0/4161057566 learned_addr learned my addr 172.21.15.8:0/4161057566 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:11.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:11.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.871+0000 7f1f4269b700 1 --2- 172.21.15.8:0/4161057566 >> v2:172.21.15.164:3300/0 conn(0x7f1f3c0a3e70 0x7f1f3c0a7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766528881 0 0) 0x7f1f3c09e7d0 con 0x7f1f3c0a5ac0 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f24003230 con 0x7f1f3c0a5ac0 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f41e9a700 1 -- 172.21.15.8:0/4161057566 >> v2:172.21.15.164:3300/0 conn(0x7f1f3c0a3e70 msgr2=0x7f1f3c0a7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> v2:172.21.15.164:3300/0 conn(0x7f1f3c0a3e70 0x7f1f3c0a7fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f41e9a700 1 -- 172.21.15.8:0/4161057566 >> v1:172.21.15.8:6789/0 conn(0x7f1f3c0a5ac0 legacy=0x7f1f3c0a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.872+0000 7f1f41e9a700 1 -- 172.21.15.8:0/4161057566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3c0a0cf0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f38005ca0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f3c07c7b0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f380058c0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.873+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f380064a0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.874+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3c0a74e0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.875+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.875+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f1f38006780 con 0x7f1f3c0a5690 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.875+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.876+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f3c056640 con 0x7f1f3c0a5690 2020-02-05T16:59:11.883 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.876+0000 7f1f2effd700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 0x7f1f2403c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:11.884 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.876+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5557+0+0 (secure 0 0 0) 0x7f1f3c0aa3e0 con 0x7f1f3c0a5690 2020-02-05T16:59:11.884 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.877+0000 7f1f4269b700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 0x7f1f2403c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:11.884 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.877+0000 7f1f4269b700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 0x7f1f2403c440 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:11.889 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.886+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:11.889 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:11.886+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f1f3c0a0cf0 con 0x7f1f3c0a5690 2020-02-05T16:59:12.090 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.088+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1f3c0666e0 con 0x7f1f3c0a5690 2020-02-05T16:59:12.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T16:59:12.090+0000 7f9e870de700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2020-02-05T16:59:12.105 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.103+0000 7f1f41e9a700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.103+0000 7f1f2effd700 1 -- 172.21.15.8:0/4161057566 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 187+0+0 (secure 0 0 0) 0x7f1f380310f0 con 0x7f1f3c0a5690 2020-02-05T16:59:12.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.107+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 msgr2=0x7f1f2403c440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.107+0000 7f1f440fe700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 0x7f1f2403c440 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.107+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 msgr2=0x7f1f3c07c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.107+0000 7f1f440fe700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f1f38000fe0 tx=0x7f1f380109b0).stop 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 shutdown_connections 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1f24039f30 0x7f1f2403c440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 --2- 172.21.15.8:0/4161057566 >> v2:172.21.15.164:3300/0 conn(0x7f1f3c0a3e70 0x7f1f3c0a7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 --2- 172.21.15.8:0/4161057566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f3c0a5690 0x7f1f3c07c260 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 >> 172.21.15.8:0/4161057566 conn(0x7f1f3c065650 msgr2=0x7f1f3c08fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:12.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.108+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 shutdown_connections 2020-02-05T16:59:12.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.109+0000 7f1f440fe700 1 -- 172.21.15.8:0/4161057566 wait complete. 2020-02-05T16:59:12.115 INFO:teuthology.orchestra.run.smithi008.stderr:new fs with metadata pool 2 and data pool 3 2020-02-05T16:59:12.128 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:12.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.346+0000 7ff366680700 1 Processor -- start 2020-02-05T16:59:12.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.346+0000 7ff366680700 1 -- start start 2020-02-05T16:59:12.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.347+0000 7ff366680700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff36008fff0 0x7ff36008f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.347+0000 7ff366680700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600aab90 con 0x7ff3600a9e90 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.347+0000 7ff366680700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36005b340 con 0x7ff3600908f0 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.347+0000 7ff366680700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff360090d20 con 0x7ff36008fff0 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.348+0000 7ff36541e700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7ff3600a9e90 0x7ff3600a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45630/0 (socket says 172.21.15.8:45630) 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.348+0000 7ff36541e700 1 -- 172.21.15.8:0/1625502367 learned_addr learned my addr 172.21.15.8:0/1625502367 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.348+0000 7ff364c1d700 1 --2- 172.21.15.8:0/1625502367 >> v2:172.21.15.164:3300/0 conn(0x7ff36008fff0 0x7ff36008f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.348+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905450668 0 0) 0x7ff3600aab90 con 0x7ff3600a9e90 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.348+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff350003230 con 0x7ff3600a9e90 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 291168408 0 0) 0x7ff350003230 con 0x7ff3600a9e90 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 >> v2:172.21.15.164:3300/0 conn(0x7ff36008fff0 msgr2=0x7ff36008f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 --2- 172.21.15.8:0/1625502367 >> v2:172.21.15.164:3300/0 conn(0x7ff36008fff0 0x7ff36008f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 >> v1:172.21.15.164:6789/0 conn(0x7ff3600908f0 legacy=0x7ff3600a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600aa2c0 con 0x7ff3600a9e90 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7ff358001ad0 con 0x7ff3600a9e90 2020-02-05T16:59:12.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 >> v1:172.21.15.8:6789/0 conn(0x7ff3600a9e90 legacy=0x7ff3600a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3500038b0 0x7ff350003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3500045a0 0x7ff3500049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff360090d20 con 0x7ff3500045a0 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35004fe10 con 0x7ff350004f20 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff350050050 con 0x7ff3500038b0 2020-02-05T16:59:12.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.349+0000 7ff35ffff700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3500045a0 0x7ff3500049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff364c1d700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3500038b0 0x7ff350003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 -- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3500038b0 msgr2=0x7ff350003d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff3500038b0 0x7ff350003d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 -- 172.21.15.8:0/1625502367 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3500045a0 msgr2=0x7ff3500049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3500045a0 0x7ff3500049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:12.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff36541e700 1 -- 172.21.15.8:0/1625502367 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff350050350 con 0x7ff350004f20 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.350+0000 7ff35ffff700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3500045a0 0x7ff3500049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff3580017e0 tx=0x7ff35800c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 -- 172.21.15.8:0/1625502367 reap_dead start 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff3580017e0 tx=0x7ff35800c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7ff3580017e0 tx=0x7ff35800c560).ready entity=mon.1 client_cookie=dcb7633d24ce9b2b server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff3580017e0 tx=0x7ff35800c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff358005d40 con 0x7ff350004f20 2020-02-05T16:59:12.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff3580017e0 tx=0x7ff35800c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff36541e700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7ff3580017e0 tx=0x7ff35800c560).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.351+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358005960 con 0x7ff350004f20 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.352+0000 7ff35f7fe700 1 -- 172.21.15.8:0/1625502367 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff35800e2d0 con 0x7ff350004f20 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.352+0000 7ff366680700 1 -- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 msgr2=0x7ff35000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.352+0000 7ff366680700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7ff3580017e0 tx=0x7ff35800c560).stop 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.353+0000 7ff366680700 1 -- 172.21.15.8:0/1625502367 shutdown_connections 2020-02-05T16:59:12.364 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.353+0000 7ff366680700 1 --2- 172.21.15.8:0/1625502367 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff350004f20 0x7ff35000b7f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.365 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.353+0000 7ff366680700 1 -- 172.21.15.8:0/1625502367 >> 172.21.15.8:0/1625502367 conn(0x7ff3600600f0 msgr2=0x7ff36005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:12.365 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.353+0000 7ff366680700 1 -- 172.21.15.8:0/1625502367 shutdown_connections 2020-02-05T16:59:12.365 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.353+0000 7ff366680700 1 -- 172.21.15.8:0/1625502367 wait complete. 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.357+0000 7ff366680700 1 Processor -- start 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.357+0000 7ff366680700 1 -- start start 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.358+0000 7ff366680700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7ff3600a2b50 0x7ff36009d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.358+0000 7ff366680700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff360066c40 con 0x7ff36009dcc0 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.358+0000 7ff366680700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3600a3ca0 con 0x7ff3600a7160 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.358+0000 7ff366680700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3600a40f0 con 0x7ff3600a2b50 2020-02-05T16:59:12.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.359+0000 7ff36541e700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7ff36009dcc0 0x7ff36013fb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45644/0 (socket says 172.21.15.8:45644) 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.359+0000 7ff36541e700 1 -- 172.21.15.8:0/2555013610 learned_addr learned my addr 172.21.15.8:0/2555013610 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.359+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> v2:172.21.15.8:3300/0 conn(0x7ff3600a2b50 0x7ff36009d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.359+0000 7ff35ffff700 1 -- 172.21.15.8:0/2555013610 >> v1:172.21.15.164:6789/0 conn(0x7ff3600a7160 legacy=0x7ff360066440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.359+0000 7ff35ffff700 1 -- 172.21.15.8:0/2555013610 >> v1:172.21.15.8:6789/0 conn(0x7ff36009dcc0 legacy=0x7ff36013fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193203510 0 0) 0x7ff360066c40 con 0x7ff36009dcc0 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 -- 172.21.15.8:0/2555013610 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3600a7f90 con 0x7ff3600a2b50 2020-02-05T16:59:12.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> v2:172.21.15.8:3300/0 conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> v2:172.21.15.8:3300/0 conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34c00cd10 con 0x7ff3600a2b50 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00c930 con 0x7ff3600a2b50 2020-02-05T16:59:12.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34c012780 con 0x7ff3600a2b50 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.360+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36009e0f0 con 0x7ff3600a2b50 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.361+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff360141320 con 0x7ff3600a2b50 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.361+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.362+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7ff34c012980 con 0x7ff3600a2b50 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.362+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.362+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff360056640 con 0x7ff3600a2b50 2020-02-05T16:59:12.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.363+0000 7ff35d7fa700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 0x7ff34003bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:12.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.363+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5673+0+0 (secure 0 0 0) 0x7ff34c001a70 con 0x7ff3600a2b50 2020-02-05T16:59:12.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.363+0000 7ff364c1d700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 0x7ff34003bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.364+0000 7ff364c1d700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 0x7ff34003bdc0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.370+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.370+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7ff34c031330 con 0x7ff3600a2b50 2020-02-05T16:59:12.525 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.522+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff3600a9100 con 0x7ff3600a2b50 2020-02-05T16:59:12.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.524+0000 7ff35ffff700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.524+0000 7ff35d7fa700 1 -- 172.21.15.8:0/2555013610 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14637 (secure 0 0 0) 0x7ff34c031590 con 0x7ff3600a2b50 2020-02-05T16:59:12.527 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:12.527 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":20,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:12.091595+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:12.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 msgr2=0x7ff34003bdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 0x7ff34003bdc0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 msgr2=0x7ff36009d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7ff34c001000 tx=0x7ff34c010ce0).stop 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 shutdown_connections 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff3400398b0 0x7ff34003bdc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 --2- 172.21.15.8:0/2555013610 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff3600a2b50 0x7ff36009d770 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 >> 172.21.15.8:0/2555013610 conn(0x7ff360060050 msgr2=0x7ff3600aadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:12.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 shutdown_connections 2020-02-05T16:59:12.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.526+0000 7ff366680700 1 -- 172.21.15.8:0/2555013610 wait complete. 2020-02-05T16:59:12.549 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:12.789 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.779+0000 7f030c466700 1 Processor -- start 2020-02-05T16:59:12.789 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.779+0000 7f030c466700 1 -- start start 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.780+0000 7f030c466700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030408fff0 0x7f030408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.780+0000 7f030c466700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 0x7f03040a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.780+0000 7f030c466700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040aab90 con 0x7f03040908f0 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.780+0000 7f030c466700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0304090d20 con 0x7f030408fff0 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.780+0000 7f030c466700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0304090650 con 0x7f03040a9e90 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.781+0000 7f030aa03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030408fff0 0x7f030408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.790 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.781+0000 7f030b204700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 0x7f03040a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.781+0000 7f030aa03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030408fff0 0x7f030408f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44270/0 (socket says 172.21.15.8:44270) 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.781+0000 7f030aa03700 1 -- 172.21.15.8:0/219871111 learned_addr learned my addr 172.21.15.8:0/219871111 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 -- 172.21.15.8:0/219871111 >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 msgr2=0x7f03040a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 0x7f03040a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 -- 172.21.15.8:0/219871111 >> v1:172.21.15.8:6789/0 conn(0x7f03040908f0 legacy=0x7f03040a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 -- 172.21.15.8:0/219871111 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03040a86e0 con 0x7f030408fff0 2020-02-05T16:59:12.791 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030b204700 1 --2- 172.21.15.8:0/219871111 >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 0x7f03040a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> v2:172.21.15.8:3300/0 conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f03000032b0 tx=0x7f03000102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> v2:172.21.15.8:3300/0 conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f03000032b0 tx=0x7f03000102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.782+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f03000102b0).ready entity=mon.0 client_cookie=f9a2b82b6d4b5146 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f03000102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f0309a01700 1 -- 172.21.15.8:0/219871111 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f030000a160 con 0x7f030408fff0 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f03000102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.792 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f0309a01700 1 -- 172.21.15.8:0/219871111 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0300011bf0 con 0x7f030408fff0 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f030aa03700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=THROTTLE_DONE pgs=64 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f03000102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f0309a01700 1 -- 172.21.15.8:0/219871111 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f030000a740 con 0x7f030408fff0 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f030c466700 1 -- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 msgr2=0x7f030408f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.783+0000 7f030c466700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 secure :-1 s=READY pgs=64 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f03000102b0).stop 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.784+0000 7f030c466700 1 -- 172.21.15.8:0/219871111 shutdown_connections 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.784+0000 7f030c466700 1 --2- 172.21.15.8:0/219871111 >> v2:172.21.15.164:3300/0 conn(0x7f03040a9e90 0x7f03040a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.784+0000 7f030c466700 1 --2- 172.21.15.8:0/219871111 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030408fff0 0x7f030408f240 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.793 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.785+0000 7f030c466700 1 -- 172.21.15.8:0/219871111 >> 172.21.15.8:0/219871111 conn(0x7f03040600f0 msgr2=0x7f030405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:12.794 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.785+0000 7f030c466700 1 -- 172.21.15.8:0/219871111 shutdown_connections 2020-02-05T16:59:12.794 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.785+0000 7f030c466700 1 -- 172.21.15.8:0/219871111 wait complete. 2020-02-05T16:59:12.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.789+0000 7f030c466700 1 Processor -- start 2020-02-05T16:59:12.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.789+0000 7f030c466700 1 -- start start 2020-02-05T16:59:12.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030c466700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030409ee00 0x7f03040a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:12.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030c466700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03040a1330 con 0x7f03040a7f20 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030c466700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0304060c90 con 0x7f03040a7240 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030c466700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03040a0940 con 0x7f030409ee00 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030aa03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030409ee00 0x7f03040a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030aa03700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f030409ee00 0x7f03040a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44274/0 (socket says 172.21.15.8:44274) 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.791+0000 7f030aa03700 1 -- 172.21.15.8:0/3856693160 learned_addr learned my addr 172.21.15.8:0/3856693160 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 -- 172.21.15.8:0/3856693160 >> v1:172.21.15.164:6789/0 conn(0x7f03040a7240 legacy=0x7f0304136280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 -- 172.21.15.8:0/3856693160 >> v1:172.21.15.8:6789/0 conn(0x7f03040a7f20 legacy=0x7f0304136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 -- 172.21.15.8:0/3856693160 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0304060ec0 con 0x7f030409ee00 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057553102 0 0) 0x7f03040a1330 con 0x7f03040a7f20 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> v2:172.21.15.8:3300/0 conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> v2:172.21.15.8:3300/0 conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.792+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0300005c30 con 0x7f030409ee00 2020-02-05T16:59:12.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030000a360 con 0x7f030409ee00 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0300007530 con 0x7f030409ee00 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f030413e910 con 0x7f030409ee00 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.793+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03040a2640 con 0x7f030409ee00 2020-02-05T16:59:12.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.794+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.794+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f030000f360 con 0x7f030409ee00 2020-02-05T16:59:12.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.794+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.794+0000 7f02fb7fe700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 0x7f02ec0400d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:12.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.794+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f030004a2e0 con 0x7f030409ee00 2020-02-05T16:59:12.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.795+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0304056640 con 0x7f030409ee00 2020-02-05T16:59:12.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.795+0000 7f030a202700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 0x7f02ec0400d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:12.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.796+0000 7f030a202700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 0x7f02ec0400d0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:12.805 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.803+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.805 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.803+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f0300031e10 con 0x7f030409ee00 2020-02-05T16:59:12.960 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.957+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0304068f50 con 0x7f030409ee00 2020-02-05T16:59:12.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.959+0000 7f030aa03700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:12.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.959+0000 7f02fb7fe700 1 -- 172.21.15.8:0/3856693160 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+14637 (secure 0 0 0) 0x7f0300007810 con 0x7f030409ee00 2020-02-05T16:59:12.962 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:12.962 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":20,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:12.091595+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:12.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 msgr2=0x7f02ec0400d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 0x7f02ec0400d0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 msgr2=0x7f03040a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f03000032b0 tx=0x7f0300011f20).stop 2020-02-05T16:59:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 shutdown_connections 2020-02-05T16:59:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f030409ee00 0x7f03040a0de0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 --2- 172.21.15.8:0/3856693160 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f02ec03dbc0 0x7f02ec0400d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:12.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 >> 172.21.15.8:0/3856693160 conn(0x7f030407fb10 msgr2=0x7f030407cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:12.965 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 shutdown_connections 2020-02-05T16:59:12.965 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:12.961+0000 7f030c466700 1 -- 172.21.15.8:0/3856693160 wait complete. 2020-02-05T16:59:12.983 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2020-02-05T16:59:13.212 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 Processor -- start 2020-02-05T16:59:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 -- start start 2020-02-05T16:59:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 0x7ffb1c0908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c0aab90 con 0x7ffb1c08fff0 2020-02-05T16:59:13.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb1c08f670 con 0x7ffb1c08f240 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.202+0000 7ffb22921700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c090650 con 0x7ffb1c0a9e90 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.203+0000 7ffb1bfff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 0x7ffb1c0908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.203+0000 7ffb216bf700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.203+0000 7ffb216bf700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44336/0 (socket says 172.21.15.8:44336) 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.203+0000 7ffb216bf700 1 -- 172.21.15.8:0/2827994918 learned_addr learned my addr 172.21.15.8:0/2827994918 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.204+0000 7ffb216bf700 1 -- 172.21.15.8:0/2827994918 >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 msgr2=0x7ffb1c0908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.204+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 0x7ffb1c0908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:13.214 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.204+0000 7ffb216bf700 1 -- 172.21.15.8:0/2827994918 >> v1:172.21.15.164:6789/0 conn(0x7ffb1c08fff0 legacy=0x7ffb1c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.204+0000 7ffb216bf700 1 -- 172.21.15.8:0/2827994918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c0a86e0 con 0x7ffb1c0a9e90 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.204+0000 7ffb1bfff700 1 --2- 172.21.15.8:0/2827994918 >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 0x7ffb1c0908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb14008c90 tx=0x7ffb140102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb14008c90 tx=0x7ffb140102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7ffb14008c90 tx=0x7ffb140102b0).ready entity=mon.2 client_cookie=ef800ac9408bc0e7 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ffb14008c90 tx=0x7ffb140102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.215 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ffb14008c90 tx=0x7ffb140102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb216bf700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7ffb14008c90 tx=0x7ffb140102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.205+0000 7ffb1b7fe700 1 -- 172.21.15.8:0/2827994918 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb1400a350 con 0x7ffb1c0a9e90 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.206+0000 7ffb1b7fe700 1 -- 172.21.15.8:0/2827994918 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb14011c30 con 0x7ffb1c0a9e90 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.206+0000 7ffb1b7fe700 1 -- 172.21.15.8:0/2827994918 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb1400a930 con 0x7ffb1c0a9e90 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.206+0000 7ffb22921700 1 -- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 msgr2=0x7ffb1c0a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.206+0000 7ffb22921700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7ffb14008c90 tx=0x7ffb140102b0).stop 2020-02-05T16:59:13.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.207+0000 7ffb22921700 1 -- 172.21.15.8:0/2827994918 shutdown_connections 2020-02-05T16:59:13.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.207+0000 7ffb22921700 1 --2- 172.21.15.8:0/2827994918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0a9e90 0x7ffb1c0a8190 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:13.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.207+0000 7ffb22921700 1 --2- 172.21.15.8:0/2827994918 >> v2:172.21.15.164:3300/0 conn(0x7ffb1c08f240 0x7ffb1c0908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:13.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.207+0000 7ffb22921700 1 -- 172.21.15.8:0/2827994918 >> 172.21.15.8:0/2827994918 conn(0x7ffb1c0600f0 msgr2=0x7ffb1c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:13.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.208+0000 7ffb22921700 1 -- 172.21.15.8:0/2827994918 shutdown_connections 2020-02-05T16:59:13.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.208+0000 7ffb22921700 1 -- 172.21.15.8:0/2827994918 wait complete. 2020-02-05T16:59:13.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.211+0000 7ffb22921700 1 Processor -- start 2020-02-05T16:59:13.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.212+0000 7ffb22921700 1 -- start start 2020-02-05T16:59:13.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.213+0000 7ffb22921700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:13.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.213+0000 7ffb22921700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c07cf10 con 0x7ffb1c126280 2020-02-05T16:59:13.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.213+0000 7ffb22921700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb1c127340 con 0x7ffb1c126f10 2020-02-05T16:59:13.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.213+0000 7ffb22921700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1c060e10 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.214+0000 7ffb20ebe700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.214+0000 7ffb216bf700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7ffb1c126280 0x7ffb1c13f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45674/0 (socket says 172.21.15.8:45674) 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.214+0000 7ffb216bf700 1 -- 172.21.15.8:0/1342721350 learned_addr learned my addr 172.21.15.8:0/1342721350 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3730315965 0 0) 0x7ffb1c07cf10 con 0x7ffb1c126280 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb00003230 con 0x7ffb1c126280 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356827529 0 0) 0x7ffb1c127340 con 0x7ffb1c126f10 2020-02-05T16:59:13.221 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb20ebe700 1 -- 172.21.15.8:0/1342721350 >> v1:172.21.15.164:6789/0 conn(0x7ffb1c126f10 legacy=0x7ffb1c125a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb20ebe700 1 -- 172.21.15.8:0/1342721350 >> v1:172.21.15.8:6789/0 conn(0x7ffb1c126280 legacy=0x7ffb1c13f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb20ebe700 1 -- 172.21.15.8:0/1342721350 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb1c07f620 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.215+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb04006bc0 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb040067e0 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb1c127570 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.216+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb04012230 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.225 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.217+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb1c141140 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.226 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.218+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.220+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb1c056640 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.220+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7ffb04012540 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.220+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.220+0000 7ffb197fa700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 0x7ffb0003c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.220+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7ffb04049510 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.221+0000 7ffb1bfff700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 0x7ffb0003c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:13.227 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.222+0000 7ffb1bfff700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 0x7ffb0003c330 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:13.231 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.229+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:13.231 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.229+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7ffb04031240 con 0x7ffb1c0ab2c0 2020-02-05T16:59:13.440 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:13.438+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ffb1c068f50 con 0x7ffb1c0ab2c0 2020-02-05T16:59:14.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.113+0000 7ffb20ebe700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.113+0000 7ffb197fa700 1 -- 172.21.15.8:0/1342721350 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7ffb04031df0 con 0x7ffb1c0ab2c0 2020-02-05T16:59:14.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.116+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 msgr2=0x7ffb0003c330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.116+0000 7ffb22921700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 0x7ffb0003c330 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.116+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 msgr2=0x7ffb1c07c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.116+0000 7ffb22921700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7ffb04000e90 tx=0x7ffb04010980).stop 2020-02-05T16:59:14.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 shutdown_connections 2020-02-05T16:59:14.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb1c0ab2c0 0x7ffb1c07c9c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 --2- 172.21.15.8:0/1342721350 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ffb00039e20 0x7ffb0003c330 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.121 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 >> 172.21.15.8:0/1342721350 conn(0x7ffb1c08f240 msgr2=0x7ffb1c0908f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:14.121 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 shutdown_connections 2020-02-05T16:59:14.121 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.117+0000 7ffb22921700 1 -- 172.21.15.8:0/1342721350 wait complete. 2020-02-05T16:59:14.145 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-02-05T16:59:14.381 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.367+0000 7f4be08ff700 1 Processor -- start 2020-02-05T16:59:14.381 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.367+0000 7f4be08ff700 1 -- start start 2020-02-05T16:59:14.381 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.368+0000 7f4be08ff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:14.385 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.368+0000 7f4be08ff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:14.385 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.368+0000 7f4be08ff700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0aa3e0 con 0x7f4bdc0a7990 2020-02-05T16:59:14.386 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.368+0000 7f4be08ff700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc090d20 con 0x7f4bdc0908f0 2020-02-05T16:59:14.386 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.368+0000 7f4be08ff700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc090650 con 0x7f4bdc08fff0 2020-02-05T16:59:14.388 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.369+0000 7f4bdad9d700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.369+0000 7f4bda59c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.369+0000 7f4bda59c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:59852/0 (socket says 172.21.15.8:59852) 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.369+0000 7f4bda59c700 1 -- 172.21.15.8:0/4207291870 learned_addr learned my addr 172.21.15.8:0/4207291870 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bdad9d700 1 -- 172.21.15.8:0/4207291870 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 msgr2=0x7f4bdc0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.389 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bdad9d700 1 -- 172.21.15.8:0/4207291870 >> v1:172.21.15.164:6789/0 conn(0x7f4bdc0a7990 legacy=0x7f4bdc0a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.390 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bdad9d700 1 -- 172.21.15.8:0/4207291870 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0aab90 con 0x7f4bdc08fff0 2020-02-05T16:59:14.390 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4207291870 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:14.390 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.370+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.392 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.392 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).ready entity=mon.2 client_cookie=d4fd65f43c6f0124 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.393 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.394 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.394 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=THROTTLE_DONE pgs=26 cs=0 l=1 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bd9d9b700 1 -- 172.21.15.8:0/4207291870 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bcc0116e0 con 0x7f4bdc08fff0 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.371+0000 7f4bd9d9b700 1 -- 172.21.15.8:0/4207291870 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc008c50 con 0x7f4bdc08fff0 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.372+0000 7f4bd9d9b700 1 -- 172.21.15.8:0/4207291870 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bcc011cc0 con 0x7f4bdc08fff0 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.372+0000 7f4be08ff700 1 -- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 msgr2=0x7f4bdc08f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.372+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 secure :-1 s=READY pgs=26 cs=0 l=1 rx=0x7f4bcc008910 tx=0x7f4bcc00fe70).stop 2020-02-05T16:59:14.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.373+0000 7f4be08ff700 1 -- 172.21.15.8:0/4207291870 shutdown_connections 2020-02-05T16:59:14.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.373+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4207291870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc08fff0 0x7f4bdc08f240 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.373+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4207291870 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0908f0 0x7f4bdc0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.373+0000 7f4be08ff700 1 -- 172.21.15.8:0/4207291870 >> 172.21.15.8:0/4207291870 conn(0x7f4bdc0600f0 msgr2=0x7f4bdc05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:14.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.373+0000 7f4be08ff700 1 -- 172.21.15.8:0/4207291870 shutdown_connections 2020-02-05T16:59:14.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.374+0000 7f4be08ff700 1 -- 172.21.15.8:0/4207291870 wait complete. 2020-02-05T16:59:14.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.377+0000 7f4be08ff700 1 Processor -- start 2020-02-05T16:59:14.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.378+0000 7f4be08ff700 1 -- start start 2020-02-05T16:59:14.402 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0a1340 0x7f4bdc132c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.402 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4bdc136d30 0x7f4bdc065e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.402 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc0a77a0 0x7f4bdc0a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.402 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc136550 con 0x7f4bdc136d30 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc1367f0 con 0x7f4bdc0a1340 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4be08ff700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc136a90 con 0x7f4bdc0a77a0 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4bda59c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4bdc136d30 0x7f4bdc065e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4bda59c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4bdc136d30 0x7f4bdc065e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44304/0 (socket says 172.21.15.8:44304) 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4bda59c700 1 -- 172.21.15.8:0/4102141941 learned_addr learned my addr 172.21.15.8:0/4102141941 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4102141941 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0a1340 0x7f4bdc132c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.403 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.380+0000 7f4bdb59e700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc0a77a0 0x7f4bdc0a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 -- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc0a77a0 msgr2=0x7f4bdc0a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc0a77a0 0x7f4bdc0a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 -- 172.21.15.8:0/4102141941 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0a1340 msgr2=0x7f4bdc132c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0a1340 0x7f4bdc132c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 -- 172.21.15.8:0/4102141941 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc07e290 con 0x7f4bdc136d30 2020-02-05T16:59:14.404 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.381+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> v2:172.21.15.8:3300/0 conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> v2:172.21.15.8:3300/0 conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd0012ad0 con 0x7f4bdc136d30 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.405 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd000e160 con 0x7f4bdc136d30 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bd00058b0 con 0x7f4bdc136d30 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.382+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc066780 con 0x7f4bdc136d30 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.383+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc1331e0 con 0x7f4bdc136d30 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.383+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.384+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f4bd0005bc0 con 0x7f4bdc136d30 2020-02-05T16:59:14.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.384+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.384+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc056640 con 0x7f4bdc136d30 2020-02-05T16:59:14.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.384+0000 7f4bcb7fe700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 0x7f4bc003bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.385+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f4bd0049760 con 0x7f4bdc136d30 2020-02-05T16:59:14.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.385+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 0x7f4bc003bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.386+0000 7f4bdad9d700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 0x7f4bc003bea0 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.391+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.391+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f4bd00311f0 con 0x7f4bdc136d30 2020-02-05T16:59:14.599 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.597+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bdc068f50 con 0x7f4bdc136d30 2020-02-05T16:59:14.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.598+0000 7f4bda59c700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.600 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.598+0000 7f4bcb7fe700 1 -- 172.21.15.8:0/4102141941 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7f4bd002fde0 con 0x7f4bdc136d30 2020-02-05T16:59:14.601 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:14.601 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-02-05T16:59:12.091946+0000","modified":"2020-02-05T16:59:14.099803+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-02-05T16:59:14.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 msgr2=0x7f4bc003bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.603 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 0x7f4bc003bea0 crc :-1 s=READY pgs=48 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 msgr2=0x7f4bdc065e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f4bdc136710 tx=0x7f4bd0010fe0).stop 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 shutdown_connections 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f4bc0039990 0x7f4bc003bea0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bdc0a77a0 0x7f4bdc0a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> v2:172.21.15.164:3300/0 conn(0x7f4bdc0a1340 0x7f4bdc132c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.604 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 --2- 172.21.15.8:0/4102141941 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4bdc136d30 0x7f4bdc065e90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.605 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 >> 172.21.15.8:0/4102141941 conn(0x7f4bdc08f240 msgr2=0x7f4bdc128b60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:14.605 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.600+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 shutdown_connections 2020-02-05T16:59:14.605 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.601+0000 7f4be08ff700 1 -- 172.21.15.8:0/4102141941 wait complete. 2020-02-05T16:59:14.605 INFO:teuthology.orchestra.run.smithi008.stderr:dumped fsmap epoch 3 2020-02-05T16:59:14.622 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:14.846 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.838+0000 7fa0ec1d6700 1 Processor -- start 2020-02-05T16:59:14.846 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.838+0000 7fa0ec1d6700 1 -- start start 2020-02-05T16:59:14.846 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e408fff0 0x7fa0e408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa0e40908f0 0x7fa0e40a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e40a7990 0x7fa0e40a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e4090d20 con 0x7fa0e40a7990 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e4090650 con 0x7fa0e40908f0 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ec1d6700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e408ece0 con 0x7fa0e408fff0 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0eaf74700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e40a7990 0x7fa0e40a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.847 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0eaf74700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e40a7990 0x7fa0e40a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44314/0 (socket says 172.21.15.8:44314) 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0ea773700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e408fff0 0x7fa0e408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.839+0000 7fa0eaf74700 1 -- 172.21.15.8:0/3731175540 learned_addr learned my addr 172.21.15.8:0/3731175540 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0e9f72700 1 --2- 172.21.15.8:0/3731175540 >> v2:172.21.15.164:3300/0 conn(0x7fa0e40908f0 0x7fa0e40a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 -- 172.21.15.8:0/3731175540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e408fff0 msgr2=0x7fa0e408f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e408fff0 0x7fa0e408f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 -- 172.21.15.8:0/3731175540 >> v2:172.21.15.164:3300/0 conn(0x7fa0e40908f0 msgr2=0x7fa0e40a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:14.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> v2:172.21.15.164:3300/0 conn(0x7fa0e40908f0 0x7fa0e40a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 -- 172.21.15.8:0/3731175540 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e40aa3e0 con 0x7fa0e40a7990 2020-02-05T16:59:14.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.840+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> v2:172.21.15.8:3300/0 conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> v2:172.21.15.8:3300/0 conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).ready entity=mon.0 client_cookie=1522b5895eee2d94 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0e9771700 1 -- 172.21.15.8:0/3731175540 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0dc011760 con 0x7fa0e40a7990 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0e9771700 1 -- 172.21.15.8:0/3731175540 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc008d30 con 0x7fa0e40a7990 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0eaf74700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 msgr2=0x7fa0e40a7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0e9771700 1 -- 172.21.15.8:0/3731175540 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0dc011d40 con 0x7fa0e40a7990 2020-02-05T16:59:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.841+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7fa0e4067240 tx=0x7fa0dc00ff30).stop 2020-02-05T16:59:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/3731175540 shutdown_connections 2020-02-05T16:59:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e40a7990 0x7fa0e40a7dc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/3731175540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e408fff0 0x7fa0e408f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/3731175540 >> v2:172.21.15.164:3300/0 conn(0x7fa0e40908f0 0x7fa0e40a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:14.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/3731175540 >> 172.21.15.8:0/3731175540 conn(0x7fa0e40600f0 msgr2=0x7fa0e405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:14.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/3731175540 shutdown_connections 2020-02-05T16:59:14.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.843+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/3731175540 wait complete. 2020-02-05T16:59:14.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.845+0000 7fa0ec1d6700 1 Processor -- start 2020-02-05T16:59:14.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.845+0000 7fa0ec1d6700 1 -- start start 2020-02-05T16:59:14.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ec1d6700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e4125cf0 0x7fa0e4122be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ec1d6700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 0x7fa0e409de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ec1d6700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0e4123130 con 0x7fa0e4137ec0 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ec1d6700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0e4127580 con 0x7fa0e4125cf0 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ec1d6700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0e4135010 con 0x7fa0e41352b0 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ea773700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e4125cf0 0x7fa0e4122be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ea773700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fa0e4125cf0 0x7fa0e4122be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44316/0 (socket says 172.21.15.8:44316) 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ea773700 1 -- 172.21.15.8:0/1038353009 learned_addr learned my addr 172.21.15.8:0/1038353009 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:14.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0e9f72700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 0x7fa0e409de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.847+0000 7fa0ea773700 1 -- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 msgr2=0x7fa0e409de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 0x7fa0e409de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 -- 172.21.15.8:0/1038353009 >> v1:172.21.15.164:6789/0 conn(0x7fa0e4137ec0 legacy=0x7fa0e41382f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 -- 172.21.15.8:0/1038353009 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e409e380 con 0x7fa0e4125cf0 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0e9f72700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 0x7fa0e409de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> v2:172.21.15.8:3300/0 conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.859 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> v2:172.21.15.8:3300/0 conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.860 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0d4011d30 con 0x7fa0e4125cf0 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0d4011950 con 0x7fa0e4125cf0 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0e41356e0 con 0x7fa0e4125cf0 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0d4005aa0 con 0x7fa0e4125cf0 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.848+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0e4138cb0 con 0x7fa0e4125cf0 2020-02-05T16:59:14.861 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.849+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.850+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0e4056640 con 0x7fa0e4125cf0 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.850+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fa0d4007140 con 0x7fa0e4125cf0 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.850+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.851+0000 7fa0daffd700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 0x7fa0c403be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.851+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fa0d4049be0 con 0x7fa0e4125cf0 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.851+0000 7fa0e9f72700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 0x7fa0c403be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:14.862 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.852+0000 7fa0e9f72700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 0x7fa0c403be80 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:14.863 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.858+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:14.864 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:14.858+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fa0e4138cb0 con 0x7fa0e4125cf0 2020-02-05T16:59:15.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.015+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0e4123670 con 0x7fa0e4125cf0 2020-02-05T16:59:15.019 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.017+0000 7fa0ea773700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.020 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.017+0000 7fa0daffd700 1 -- 172.21.15.8:0/1038353009 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7fa0d4005d70 con 0x7fa0e4125cf0 2020-02-05T16:59:15.020 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:15.020 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:15.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 msgr2=0x7fa0c403be80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 0x7fa0c403be80 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 msgr2=0x7fa0e4122be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fa0e4127740 tx=0x7fa0d4010ef0).stop 2020-02-05T16:59:15.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 shutdown_connections 2020-02-05T16:59:15.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa0c4039970 0x7fa0c403be80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fa0e4125cf0 0x7fa0e4122be0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 --2- 172.21.15.8:0/1038353009 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0e41352b0 0x7fa0e409de30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.025 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.020+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 >> 172.21.15.8:0/1038353009 conn(0x7fa0e408f350 msgr2=0x7fa0e40a2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:15.025 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.021+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 shutdown_connections 2020-02-05T16:59:15.025 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.021+0000 7fa0ec1d6700 1 -- 172.21.15.8:0/1038353009 wait complete. 2020-02-05T16:59:15.046 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2020-02-05T16:59:15.046 INFO:tasks.ceph.mds.a:Restarting daemon 2020-02-05T16:59:15.046 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2020-02-05T16:59:15.052 INFO:tasks.ceph.mds.a:Started 2020-02-05T16:59:15.052 INFO:tasks.ceph.mds.b:Restarting daemon 2020-02-05T16:59:15.053 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2020-02-05T16:59:15.059 INFO:tasks.ceph.mds.b:Started 2020-02-05T16:59:15.059 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:15.061 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2020-02-05T16:59:15.216 INFO:tasks.ceph.mds.a.smithi008.stdout:starting mds.a at 2020-02-05T16:59:15.232 INFO:tasks.ceph.mds.b.smithi164.stdout:starting mds.b at 2020-02-05T16:59:15.271 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.263+0000 7f3a450a9700 1 Processor -- start 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.263+0000 7f3a450a9700 1 -- start start 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.264+0000 7f3a450a9700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a4008f240 0x7f3a400908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.264+0000 7f3a450a9700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a400aab90 con 0x7f3a4008fff0 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.264+0000 7f3a450a9700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4005b340 con 0x7f3a400a9e90 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.264+0000 7f3a450a9700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4008f670 con 0x7f3a4008f240 2020-02-05T16:59:15.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a4008f240 0x7f3a400908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a4008f240 0x7f3a400908f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44336/0 (socket says 172.21.15.8:44336) 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 -- 172.21.15.8:0/1015854360 learned_addr learned my addr 172.21.15.8:0/1015854360 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 -- 172.21.15.8:0/1015854360 >> v1:172.21.15.164:6789/0 conn(0x7f3a400a9e90 legacy=0x7f3a400a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 -- 172.21.15.8:0/1015854360 >> v1:172.21.15.8:6789/0 conn(0x7f3a4008fff0 legacy=0x7f3a400a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 -- 172.21.15.8:0/1015854360 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400aa2c0 con 0x7f3a4008f240 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> v2:172.21.15.8:3300/0 conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3a34008910 tx=0x7f3a3400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> v2:172.21.15.8:3300/0 conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3a34008910 tx=0x7f3a3400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.265+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7f3a34008910 tx=0x7f3a3400ff30).ready entity=mon.0 client_cookie=c67ab7e8ab987ec9 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f3a34008910 tx=0x7f3a3400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f3a34008910 tx=0x7f3a3400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3e59c700 1 -- 172.21.15.8:0/1015854360 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a340117a0 con 0x7f3a4008f240 2020-02-05T16:59:15.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f3a34008910 tx=0x7f3a3400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3e59c700 1 -- 172.21.15.8:0/1015854360 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a34008d30 con 0x7f3a4008f240 2020-02-05T16:59:15.280 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a450a9700 1 -- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 msgr2=0x7f3a400908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a3e59c700 1 -- 172.21.15.8:0/1015854360 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a34011d80 con 0x7f3a4008f240 2020-02-05T16:59:15.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.266+0000 7f3a450a9700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7f3a34008910 tx=0x7f3a3400ff30).stop 2020-02-05T16:59:15.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.267+0000 7f3a450a9700 1 -- 172.21.15.8:0/1015854360 shutdown_connections 2020-02-05T16:59:15.281 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.267+0000 7f3a450a9700 1 --2- 172.21.15.8:0/1015854360 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a4008f240 0x7f3a400908f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.282 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.267+0000 7f3a450a9700 1 -- 172.21.15.8:0/1015854360 >> 172.21.15.8:0/1015854360 conn(0x7f3a400600f0 msgr2=0x7f3a4005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:15.282 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.267+0000 7f3a450a9700 1 -- 172.21.15.8:0/1015854360 shutdown_connections 2020-02-05T16:59:15.282 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.267+0000 7f3a450a9700 1 -- 172.21.15.8:0/1015854360 wait complete. 2020-02-05T16:59:15.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.269+0000 7f3a450a9700 1 Processor -- start 2020-02-05T16:59:15.291 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.269+0000 7f3a450a9700 1 -- start start 2020-02-05T16:59:15.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a40090c70 0x7f3a400a2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 0x7f3a40066680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 0x7f3a400654d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400a3cf0 con 0x7f3a40090c70 2020-02-05T16:59:15.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400a3f90 con 0x7f3a400650a0 2020-02-05T16:59:15.293 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a450a9700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400a2b90 con 0x7f3a400a4230 2020-02-05T16:59:15.293 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.270+0000 7f3a3f59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a40090c70 0x7f3a400a2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.293 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3a40090c70 0x7f3a400a2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44342/0 (socket says 172.21.15.8:44342) 2020-02-05T16:59:15.293 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 -- 172.21.15.8:0/3851949584 learned_addr learned my addr 172.21.15.8:0/3851949584 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:15.293 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 0x7f3a40066680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.294 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a448a8700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 0x7f3a400654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.294 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 -- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 msgr2=0x7f3a40066680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.294 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 0x7f3a40066680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.294 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 -- 172.21.15.8:0/3851949584 >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 msgr2=0x7f3a400654d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.295 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 0x7f3a400654d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.295 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 -- 172.21.15.8:0/3851949584 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a40066bd0 con 0x7f3a40090c70 2020-02-05T16:59:15.296 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a448a8700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 0x7f3a400654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:15.296 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 0x7f3a40066680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:15.297 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.8:3300/0 conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.297 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.8:3300/0 conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.297 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.271+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.297 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.298 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.298 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a300124a0 con 0x7f3a40090c70 2020-02-05T16:59:15.298 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.298 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a300120c0 con 0x7f3a40090c70 2020-02-05T16:59:15.298 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4007b350 con 0x7f3a40090c70 2020-02-05T16:59:15.299 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a30012a80 con 0x7f3a40090c70 2020-02-05T16:59:15.299 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.272+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a40065c60 con 0x7f3a40090c70 2020-02-05T16:59:15.299 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.273+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.300 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.274+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40056640 con 0x7f3a40090c70 2020-02-05T16:59:15.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.274+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f3a30012c80 con 0x7f3a40090c70 2020-02-05T16:59:15.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.274+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.275+0000 7f3a27fff700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 0x7f3a2003be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.275+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f3a30049340 con 0x7f3a40090c70 2020-02-05T16:59:15.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.275+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 0x7f3a2003be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.302 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.276+0000 7f3a3ed9d700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 0x7f3a2003be70 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.302 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.281+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.302 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.282+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3a30030e90 con 0x7f3a40090c70 2020-02-05T16:59:15.441 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.439+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a400a3ad0 con 0x7f3a40090c70 2020-02-05T16:59:15.444 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.441+0000 7f3a3f59e700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.444 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.441+0000 7f3a27fff700 1 -- 172.21.15.8:0/3851949584 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7f3a300310f0 con 0x7f3a40090c70 2020-02-05T16:59:15.445 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:15.445 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:15.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.443+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 msgr2=0x7f3a2003be70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.443+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 0x7f3a2003be70 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.443+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 msgr2=0x7f3a400a2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.443+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f3a400a3eb0 tx=0x7f3a30010a10).stop 2020-02-05T16:59:15.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 shutdown_connections 2020-02-05T16:59:15.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3a20039960 0x7f3a2003be70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.450 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> v2:172.21.15.164:3300/0 conn(0x7f3a400650a0 0x7f3a400654d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3a40090c70 0x7f3a400a2640 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 --2- 172.21.15.8:0/3851949584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a400a4230 0x7f3a40066680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 >> 172.21.15.8:0/3851949584 conn(0x7f3a4008f240 msgr2=0x7f3a400ab640 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:15.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 shutdown_connections 2020-02-05T16:59:15.451 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.444+0000 7f3a450a9700 1 -- 172.21.15.8:0/3851949584 wait complete. 2020-02-05T16:59:15.468 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:06.700677+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:10.361501+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:11.053750+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-02-05T16:59:15.469 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-02-05T16:59:15.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.693+0000 7f3c5a528700 1 Processor -- start 2020-02-05T16:59:15.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.693+0000 7f3c5a528700 1 -- start start 2020-02-05T16:59:15.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5a528700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 0x7f3c540aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:15.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5a528700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54145c20 0x7f3c54146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5a528700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540aa980 con 0x7f3c540abd50 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5a528700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c54148c70 con 0x7f3c54145c20 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5a528700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c54148f10 con 0x7f3c540ace50 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.694+0000 7f3c5b78a700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54145c20 0x7f3c54146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c59d27700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 0x7f3c540aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c59d27700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 0x7f3c540aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:59906/0 (socket says 172.21.15.8:59906) 2020-02-05T16:59:15.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c59d27700 1 -- 172.21.15.8:0/357905439 learned_addr learned my addr 172.21.15.8:0/357905439 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:15.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c5b78a700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54145c20 0x7f3c54146050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44354/0 (socket says 172.21.15.8:44354) 2020-02-05T16:59:15.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c5b78a700 1 -- 172.21.15.8:0/357905439 >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 msgr2=0x7f3c540aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:15.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.695+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 0x7f3c540aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 -- 172.21.15.8:0/357905439 >> v1:172.21.15.164:6789/0 conn(0x7f3c540abd50 legacy=0x7f3c541491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 -- 172.21.15.8:0/357905439 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c541454a0 con 0x7f3c54145c20 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> v2:172.21.15.8:3300/0 conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> v2:172.21.15.8:3300/0 conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).ready entity=mon.0 client_cookie=accf15fa6c1e485c server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c58d25700 1 -- 172.21.15.8:0/357905439 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4c00a160 con 0x7f3c54145c20 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c5b78a700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.696+0000 7f3c58d25700 1 -- 172.21.15.8:0/357905439 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4c011b30 con 0x7f3c54145c20 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.697+0000 7f3c58d25700 1 -- 172.21.15.8:0/357905439 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4c00a740 con 0x7f3c54145c20 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.697+0000 7f3c5a528700 1 -- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 msgr2=0x7f3c54146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.697+0000 7f3c5a528700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f3c540ab530 tx=0x7f3c4c0101f0).stop 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 -- 172.21.15.8:0/357905439 shutdown_connections 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 --2- 172.21.15.8:0/357905439 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54145c20 0x7f3c54146050 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 --2- 172.21.15.8:0/357905439 >> v2:172.21.15.164:3300/0 conn(0x7f3c540ace50 0x7f3c540aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 -- 172.21.15.8:0/357905439 >> 172.21.15.8:0/357905439 conn(0x7f3c54061510 msgr2=0x7f3c540ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 -- 172.21.15.8:0/357905439 shutdown_connections 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.698+0000 7f3c5a528700 1 -- 172.21.15.8:0/357905439 wait complete. 2020-02-05T16:59:15.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.700+0000 7f3c5a528700 1 Processor -- start 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.700+0000 7f3c5a528700 1 -- start start 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.701+0000 7f3c5a528700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 0x7f3c54147b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.701+0000 7f3c5a528700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54146690 0x7f3c54146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.701+0000 7f3c5a528700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54147010 con 0x7f3c54132110 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.701+0000 7f3c5a528700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c540a4150 con 0x7f3c54146690 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.701+0000 7f3c5a528700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c54125c70 con 0x7f3c54144ac0 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59526700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54146690 0x7f3c54146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.742 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59526700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3c54146690 0x7f3c54146ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44356/0 (socket says 172.21.15.8:44356) 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59526700 1 -- 172.21.15.8:0/2136524158 learned_addr learned my addr 172.21.15.8:0/2136524158 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59d27700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 0x7f3c54147b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59526700 1 -- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 msgr2=0x7f3c54147b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.702+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 0x7f3c54147b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 -- 172.21.15.8:0/2136524158 >> v1:172.21.15.164:6789/0 conn(0x7f3c54132110 legacy=0x7f3c54132540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 -- 172.21.15.8:0/2136524158 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54148050 con 0x7f3c54146690 2020-02-05T16:59:15.743 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59d27700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 0x7f3c54147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> v2:172.21.15.8:3300/0 conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> v2:172.21.15.8:3300/0 conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.744 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.703+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4800cc00 con 0x7f3c54146690 2020-02-05T16:59:15.745 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.704+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c4800c820 con 0x7f3c54146690 2020-02-05T16:59:15.745 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.704+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c54132d40 con 0x7f3c54146690 2020-02-05T16:59:15.745 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.704+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c4800ce00 con 0x7f3c54146690 2020-02-05T16:59:15.745 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.704+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c54076200 con 0x7f3c54146690 2020-02-05T16:59:15.745 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.705+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.706+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c54056640 con 0x7f3c54146690 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.706+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f3c480064e0 con 0x7f3c54146690 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.706+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.707+0000 7f3c467fc700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 0x7f3c3c03bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.707+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f3c540aa980 con 0x7f3c54146690 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.708+0000 7f3c59d27700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 0x7f3c3c03bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:15.746 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.708+0000 7f3c59d27700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 0x7f3c3c03bf20 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:15.748 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.711+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.748 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.711+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3c48030e50 con 0x7f3c54146690 2020-02-05T16:59:15.872 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.870+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3c54066210 con 0x7f3c54146690 2020-02-05T16:59:15.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.871+0000 7f3c59526700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:15.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.871+0000 7f3c467fc700 1 -- 172.21.15.8:0/2136524158 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3c480310b0 con 0x7f3c54146690 2020-02-05T16:59:15.874 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 msgr2=0x7f3c3c03bf20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 0x7f3c3c03bf20 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 msgr2=0x7f3c54146ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f3c48001460 tx=0x7f3c48010ba0).stop 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 shutdown_connections 2020-02-05T16:59:15.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3c3c039a10 0x7f3c3c03bf20 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3c54144ac0 0x7f3c54147b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 --2- 172.21.15.8:0/2136524158 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3c54146690 0x7f3c54146ac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 >> 172.21.15.8:0/2136524158 conn(0x7f3c540abd50 msgr2=0x7f3c54025080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 shutdown_connections 2020-02-05T16:59:15.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:15.873+0000 7f3c5a528700 1 -- 172.21.15.8:0/2136524158 wait complete. 2020-02-05T16:59:15.896 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-02-05T16:59:16.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.123+0000 7fd377e99700 1 Processor -- start 2020-02-05T16:59:16.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.123+0000 7fd377e99700 1 -- start start 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.124+0000 7fd377e99700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd3700908f0 0x7fd3700a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.124+0000 7fd377e99700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3700aa3e0 con 0x7fd37008fff0 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.124+0000 7fd377e99700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3700aab90 con 0x7fd37008f240 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.124+0000 7fd377e99700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37008f670 con 0x7fd3700908f0 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.125+0000 7fd376436700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fd37008fff0 0x7fd3700a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45746/0 (socket says 172.21.15.8:45746) 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.125+0000 7fd376436700 1 -- 172.21.15.8:0/1491829286 learned_addr learned my addr 172.21.15.8:0/1491829286 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:16.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.125+0000 7fd376c37700 1 --2- 172.21.15.8:0/1491829286 >> v2:172.21.15.8:3300/0 conn(0x7fd3700908f0 0x7fd3700a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.125+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034089435 0 0) 0x7fd36c0010d0 con 0x7fd37008fff0 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.125+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd35c003230 con 0x7fd37008fff0 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1102573492 0 0) 0x7fd35c003230 con 0x7fd37008fff0 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 >> v2:172.21.15.8:3300/0 conn(0x7fd3700908f0 msgr2=0x7fd3700a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 --2- 172.21.15.8:0/1491829286 >> v2:172.21.15.8:3300/0 conn(0x7fd3700908f0 0x7fd3700a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 >> v1:172.21.15.164:6789/0 conn(0x7fd37008f240 legacy=0x7fd3700a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37005b340 con 0x7fd37008fff0 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fd36c001980 con 0x7fd37008fff0 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 >> v1:172.21.15.8:6789/0 conn(0x7fd37008fff0 legacy=0x7fd3700a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd35c0045a0 0x7fd35c0049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd35c004f20 0x7fd35c00b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c00bdd0 con 0x7fd35c004f20 2020-02-05T16:59:16.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c04ffd0 con 0x7fd35c0038b0 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.126+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35c050210 con 0x7fd35c0045a0 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.127+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.127+0000 7fd375c35700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd35c0045a0 0x7fd35c0049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.127+0000 7fd376c37700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd35c004f20 0x7fd35c00b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 -- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd35c0045a0 msgr2=0x7fd35c0049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd35c0045a0 0x7fd35c0049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 -- 172.21.15.8:0/1491829286 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd35c004f20 msgr2=0x7fd35c00b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd35c004f20 0x7fd35c00b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 -- 172.21.15.8:0/1491829286 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35c050510 con 0x7fd35c0038b0 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376c37700 1 -- 172.21.15.8:0/1491829286 reap_dead start 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).ready entity=mon.1 client_cookie=5e6218527acea6c2 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c0059e0 con 0x7fd35c0038b0 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd376436700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c008240 con 0x7fd35c0038b0 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd375434700 1 -- 172.21.15.8:0/1491829286 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c007160 con 0x7fd35c0038b0 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd377e99700 1 -- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 msgr2=0x7fd35c003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.128+0000 7fd377e99700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fd36c001690 tx=0x7fd36c00c1e0).stop 2020-02-05T16:59:16.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.129+0000 7fd377e99700 1 -- 172.21.15.8:0/1491829286 shutdown_connections 2020-02-05T16:59:16.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.129+0000 7fd377e99700 1 --2- 172.21.15.8:0/1491829286 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35c0038b0 0x7fd35c003d20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.129+0000 7fd377e99700 1 -- 172.21.15.8:0/1491829286 >> 172.21.15.8:0/1491829286 conn(0x7fd3700600f0 msgr2=0x7fd37005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:16.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.129+0000 7fd377e99700 1 -- 172.21.15.8:0/1491829286 shutdown_connections 2020-02-05T16:59:16.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.129+0000 7fd377e99700 1 -- 172.21.15.8:0/1491829286 wait complete. 2020-02-05T16:59:16.145 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.133+0000 7fd377e99700 1 Processor -- start 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.133+0000 7fd377e99700 1 -- start start 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.134+0000 7fd377e99700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd370063bf0 0x7fd3700652a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.134+0000 7fd377e99700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd37007b810 0x7fd37007bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.134+0000 7fd377e99700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd370062f20 con 0x7fd3700ab9d0 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.134+0000 7fd377e99700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd37007c190 con 0x7fd37007b810 2020-02-05T16:59:16.146 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.134+0000 7fd377e99700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd370062710 con 0x7fd370063bf0 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd376436700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fd3700ab9d0 0x7fd370061f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45758/0 (socket says 172.21.15.8:45758) 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd376436700 1 -- 172.21.15.8:0/2187811504 learned_addr learned my addr 172.21.15.8:0/2187811504 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd375c35700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd370063bf0 0x7fd3700652a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458303054 0 0) 0x7fd370062f20 con 0x7fd3700ab9d0 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd358003230 con 0x7fd3700ab9d0 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1662822564 0 0) 0x7fd358003230 con 0x7fd3700ab9d0 2020-02-05T16:59:16.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.135+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd370062f20 con 0x7fd3700ab9d0 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fd36c005dd0 con 0x7fd3700ab9d0 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd370063bf0 msgr2=0x7fd3700652a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd370063bf0 0x7fd3700652a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 >> v2:172.21.15.164:3300/0 conn(0x7fd37007b810 msgr2=0x7fd37007bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> v2:172.21.15.164:3300/0 conn(0x7fd37007b810 0x7fd37007bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 >> v1:172.21.15.8:6789/0 conn(0x7fd3700ab9d0 legacy=0x7fd370061f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3580087e0 0x7fd35800ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35800b1c0 0x7fd35800d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd370062710 con 0x7fd358003a20 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd358012120 con 0x7fd35800b1c0 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd358012360 con 0x7fd3580087e0 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.136+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.149 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376c37700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35800b1c0 0x7fd35800d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376436700 1 -- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3580087e0 msgr2=0x7fd35800ac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3580087e0 0x7fd35800ac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376436700 1 -- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35800b1c0 msgr2=0x7fd35800d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35800b1c0 0x7fd35800d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376436700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd37013e750 con 0x7fd358003a20 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.137+0000 7fd376c37700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd35800b1c0 0x7fd35800d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:16.150 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd37013d740 con 0x7fd358003a20 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd37013dc60 con 0x7fd358003a20 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c00ed30 con 0x7fd358003a20 2020-02-05T16:59:16.151 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36c001460 con 0x7fd358003a20 2020-02-05T16:59:16.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.138+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd36c007bb0 con 0x7fd358003a20 2020-02-05T16:59:16.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.139+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.152 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.141+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fd36c008610 con 0x7fd358003a20 2020-02-05T16:59:16.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.141+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.153 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.141+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c003360 con 0x7fd358003a20 2020-02-05T16:59:16.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.141+0000 7fd366ffd700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd35804b080 0x7fd35804d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.141+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd36c049eb0 con 0x7fd358003a20 2020-02-05T16:59:16.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.142+0000 7fd375c35700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd35804b080 0x7fd35804d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.154 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.143+0000 7fd375c35700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd35804b080 0x7fd35804d590 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.155 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.148+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.155 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.148+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fd36c0319e0 con 0x7fd358003a20 2020-02-05T16:59:16.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.300+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd34c002220 con 0x7fd358003a20 2020-02-05T16:59:16.305 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.301+0000 7fd376436700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.305 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.301+0000 7fd366ffd700 1 -- 172.21.15.8:0/2187811504 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd36c031c40 con 0x7fd358003a20 2020-02-05T16:59:16.305 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:16.306 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd35804b080 msgr2=0x7fd35804d590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.306 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd35804b080 0x7fd35804d590 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.306 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 msgr2=0x7fd358003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 --2- 172.21.15.8:0/2187811504 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd358003a20 0x7fd358003e50 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fd36c00b890 tx=0x7fd36c005b20).stop 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd376c37700 1 -- 172.21.15.8:0/2187811504 reap_dead start 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd376c37700 1 -- 172.21.15.8:0/2187811504 reap_dead start 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 shutdown_connections 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.303+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 >> 172.21.15.8:0/2187811504 conn(0x7fd370090bb0 msgr2=0x7fd3700aae50 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.304+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 shutdown_connections 2020-02-05T16:59:16.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.304+0000 7fd377e99700 1 -- 172.21.15.8:0/2187811504 wait complete. 2020-02-05T16:59:16.325 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-02-05T16:59:16.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.549+0000 7fe37123a700 1 Processor -- start 2020-02-05T16:59:16.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.549+0000 7fe37123a700 1 -- start start 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe37123a700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe37123a700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0aa3e0 con 0x7fe36c08f240 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe37123a700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe36c0aab90 con 0x7fe36c08fff0 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe37123a700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c08f670 con 0x7fe36c0908f0 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe362d9d700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fe36c08f240 0x7fe36c0a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45774/0 (socket says 172.21.15.8:45774) 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.550+0000 7fe362d9d700 1 -- 172.21.15.8:0/3366385572 learned_addr learned my addr 172.21.15.8:0/3366385572 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:16.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe36ad9d700 1 -- 172.21.15.8:0/3366385572 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770611321 0 0) 0x7fe3540010d0 con 0x7fe36c08f240 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe36ad9d700 1 -- 172.21.15.8:0/3366385572 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe358003230 con 0x7fe36c08f240 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe370a39700 1 -- 172.21.15.8:0/3366385572 >> v1:172.21.15.164:6789/0 conn(0x7fe36c08fff0 legacy=0x7fe36c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe370a39700 1 -- 172.21.15.8:0/3366385572 >> v1:172.21.15.8:6789/0 conn(0x7fe36c08f240 legacy=0x7fe36c0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe370a39700 1 -- 172.21.15.8:0/3366385572 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c05b340 con 0x7fe36c0908f0 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.551+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe3640032b0 tx=0x7fe3640101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe3640032b0 tx=0x7fe3640101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fe3640032b0 tx=0x7fe3640101f0).ready entity=mon.2 client_cookie=1a73952a4abe3a92 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe3640032b0 tx=0x7fe3640101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe36ad9d700 1 -- 172.21.15.8:0/3366385572 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe36400a160 con 0x7fe36c0908f0 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe3640032b0 tx=0x7fe3640101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe36ad9d700 1 -- 172.21.15.8:0/3366385572 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe364011bb0 con 0x7fe36c0908f0 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe370a39700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe3640032b0 tx=0x7fe3640101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe36ad9d700 1 -- 172.21.15.8:0/3366385572 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe36400a740 con 0x7fe36c0908f0 2020-02-05T16:59:16.560 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe37123a700 1 -- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 msgr2=0x7fe36c0a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.552+0000 7fe37123a700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fe3640032b0 tx=0x7fe3640101f0).stop 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.553+0000 7fe37123a700 1 -- 172.21.15.8:0/3366385572 shutdown_connections 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.553+0000 7fe37123a700 1 --2- 172.21.15.8:0/3366385572 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0908f0 0x7fe36c0a9e90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.553+0000 7fe37123a700 1 -- 172.21.15.8:0/3366385572 >> 172.21.15.8:0/3366385572 conn(0x7fe36c0600f0 msgr2=0x7fe36c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.554+0000 7fe37123a700 1 -- 172.21.15.8:0/3366385572 shutdown_connections 2020-02-05T16:59:16.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.554+0000 7fe37123a700 1 -- 172.21.15.8:0/3366385572 wait complete. 2020-02-05T16:59:16.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.557+0000 7fe37123a700 1 Processor -- start 2020-02-05T16:59:16.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.558+0000 7fe37123a700 1 -- start start 2020-02-05T16:59:16.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe36c09f800 0x7fe36c132c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0a66d0 0x7fe36c0a7330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 0x7fe36c09da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c132470 con 0x7fe36c09f800 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe36c132710 con 0x7fe36c09d640 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.559+0000 7fe37123a700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c1329b0 con 0x7fe36c0a66d0 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.560+0000 7fe36b59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe36c09f800 0x7fe36c132c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.560+0000 7fe370a39700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 0x7fe36c09da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.560+0000 7fe36b59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe36c09f800 0x7fe36c132c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44394/0 (socket says 172.21.15.8:44394) 2020-02-05T16:59:16.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.560+0000 7fe36b59e700 1 -- 172.21.15.8:0/3070238890 learned_addr learned my addr 172.21.15.8:0/3070238890 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:16.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.560+0000 7fe362d9d700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0a66d0 0x7fe36c0a7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 -- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0a66d0 msgr2=0x7fe36c0a7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0a66d0 0x7fe36c0a7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 -- 172.21.15.8:0/3070238890 >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 msgr2=0x7fe36c09da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 0x7fe36c09da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 -- 172.21.15.8:0/3070238890 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0a7880 con 0x7fe36c09f800 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe370a39700 1 --2- 172.21.15.8:0/3070238890 >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 0x7fe36c09da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> v2:172.21.15.8:3300/0 conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> v2:172.21.15.8:3300/0 conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.561+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe35c006bc0 con 0x7fe36c09f800 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c0067e0 con 0x7fe36c09f800 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe36c0a6b00 con 0x7fe36c09f800 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe35c014480 con 0x7fe36c09f800 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.562+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe36c09dfc0 con 0x7fe36c09f800 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.563+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.564+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe36c056640 con 0x7fe36c09f800 2020-02-05T16:59:16.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.564+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fe35c014680 con 0x7fe36c09f800 2020-02-05T16:59:16.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.564+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.565+0000 7fe368d99700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 0x7fe35003bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.565+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fe35c04bae0 con 0x7fe36c09f800 2020-02-05T16:59:16.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.565+0000 7fe362d9d700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 0x7fe35003bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.566+0000 7fe362d9d700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 0x7fe35003bea0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.569+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.569+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fe35c033630 con 0x7fe36c09f800 2020-02-05T16:59:16.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.726+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe36c068f50 con 0x7fe36c09f800 2020-02-05T16:59:16.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.727+0000 7fe36b59e700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.727+0000 7fe368d99700 1 -- 172.21.15.8:0/3070238890 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7fe35c033890 con 0x7fe36c09f800 2020-02-05T16:59:16.730 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:16.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 msgr2=0x7fe35003bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.732 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 0x7fe35003bea0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.732 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 msgr2=0x7fe36c132c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.732 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7fe36c132630 tx=0x7fe35c012c50).stop 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 shutdown_connections 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe350039990 0x7fe35003bea0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> v2:172.21.15.164:3300/0 conn(0x7fe36c09d640 0x7fe36c09da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe36c09f800 0x7fe36c132c50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 --2- 172.21.15.8:0/3070238890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe36c0a66d0 0x7fe36c0a7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.733 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.729+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 >> 172.21.15.8:0/3070238890 conn(0x7fe36c0908f0 msgr2=0x7fe36c08f930 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:16.734 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.730+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 shutdown_connections 2020-02-05T16:59:16.734 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.730+0000 7fe37123a700 1 -- 172.21.15.8:0/3070238890 wait complete. 2020-02-05T16:59:16.751 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2020-02-05T16:59:16.751 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2020-02-05T16:59:16.752 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2020-02-05T16:59:16.980 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.972+0000 7fd2af59e700 1 Processor -- start 2020-02-05T16:59:16.980 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 -- start start 2020-02-05T16:59:16.980 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 0x7fd2b00aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.980 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd2b00abd50 0x7fd2b01491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.980 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2b0145c20 0x7fd2b0146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b0148c70 con 0x7fd2b00abd50 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b0148f10 con 0x7fd2b00ace50 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.973+0000 7fd2af59e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00aa980 con 0x7fd2b0145c20 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.974+0000 7fd2ae59c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd2b00abd50 0x7fd2b01491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.974+0000 7fd2ae59c700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd2b00abd50 0x7fd2b01491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44402/0 (socket says 172.21.15.8:44402) 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.974+0000 7fd2ae59c700 1 -- 172.21.15.8:0/998664715 learned_addr learned my addr 172.21.15.8:0/998664715 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:16.981 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.974+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2b0145c20 0x7fd2b0146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.974+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 0x7fd2b00aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 -- 172.21.15.8:0/998664715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2b0145c20 msgr2=0x7fd2b0146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2b0145c20 0x7fd2b0146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 -- 172.21.15.8:0/998664715 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 msgr2=0x7fd2b00aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 0x7fd2b00aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 -- 172.21.15.8:0/998664715 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b01454a0 con 0x7fd2b00abd50 2020-02-05T16:59:16.982 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 0x7fd2b00aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.8:3300/0 conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.8:3300/0 conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).ready entity=mon.0 client_cookie=2795a4c08490cd0c server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2add9b700 1 -- 172.21.15.8:0/998664715 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a4011760 con 0x7fd2b00abd50 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.975+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.983 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.976+0000 7fd2add9b700 1 -- 172.21.15.8:0/998664715 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a4008d30 con 0x7fd2b00abd50 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.976+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.976+0000 7fd2add9b700 1 -- 172.21.15.8:0/998664715 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a4011d40 con 0x7fd2b00abd50 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.976+0000 7fd2af59e700 1 -- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 msgr2=0x7fd2b01491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.976+0000 7fd2af59e700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7fd2b0148e30 tx=0x7fd2a400ff30).stop 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 -- 172.21.15.8:0/998664715 shutdown_connections 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2b0145c20 0x7fd2b0146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 --2- 172.21.15.8:0/998664715 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00ace50 0x7fd2b00aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:16.984 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 --2- 172.21.15.8:0/998664715 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd2b00abd50 0x7fd2b01491b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.985 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 -- 172.21.15.8:0/998664715 >> 172.21.15.8:0/998664715 conn(0x7fd2b0061510 msgr2=0x7fd2b00ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:16.985 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 -- 172.21.15.8:0/998664715 shutdown_connections 2020-02-05T16:59:16.985 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.977+0000 7fd2af59e700 1 -- 172.21.15.8:0/998664715 wait complete. 2020-02-05T16:59:16.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.979+0000 7fd2af59e700 1 Processor -- start 2020-02-05T16:59:16.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.979+0000 7fd2af59e700 1 -- start start 2020-02-05T16:59:16.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2af59e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd2b00a2590 0x7fd2b01484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2af59e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b00a4240 con 0x7fd2b00a5a50 2020-02-05T16:59:16.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2af59e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b007d190 con 0x7fd2b00a4540 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2af59e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00a6120 con 0x7fd2b00a2590 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2ae59c700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fd2b00a5a50 0x7fd2b007c990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45794/0 (socket says 172.21.15.8:45794) 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.980+0000 7fd2ae59c700 1 -- 172.21.15.8:0/2070946101 learned_addr learned my addr 172.21.15.8:0/2070946101 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.981+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/2070946101 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00a2590 0x7fd2b01484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.981+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984790018 0 0) 0x7fd2b00a4240 con 0x7fd2b00a5a50 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.981+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd294003230 con 0x7fd2b00a5a50 2020-02-05T16:59:16.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.981+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 664967931 0 0) 0x7fd294003230 con 0x7fd2b00a5a50 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b00a4240 con 0x7fd2b00a5a50 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fd2a4008d30 con 0x7fd2b00a5a50 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00a2590 msgr2=0x7fd2b01484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 --2- 172.21.15.8:0/2070946101 >> v2:172.21.15.164:3300/0 conn(0x7fd2b00a2590 0x7fd2b01484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 >> v1:172.21.15.164:6789/0 conn(0x7fd2b00a4540 legacy=0x7fd2b00a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 >> v1:172.21.15.8:6789/0 conn(0x7fd2b00a5a50 legacy=0x7fd2b007c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd294003a20 0x7fd294003e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2940087e0 0x7fd29400ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00a6120 con 0x7fd294003a20 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2940046e0 con 0x7fd29400b260 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd294004920 con 0x7fd2940087e0 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd294003a20 0x7fd294003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.982+0000 7fd2ae59c700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2940087e0 0x7fd29400ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 -- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2940087e0 msgr2=0x7fd29400ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd2940087e0 0x7fd29400ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 -- 172.21.15.8:0/2070946101 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd294003a20 msgr2=0x7fd294003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd294003a20 0x7fd294003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b013c720 con 0x7fd29400b260 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b013a700 con 0x7fd29400b260 2020-02-05T16:59:16.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd294003a20 0x7fd294003e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 -- 172.21.15.8:0/2070946101 reap_dead start 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b00a2590 con 0x7fd29400b260 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.983+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a8012b00 con 0x7fd29400b260 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a8012720 con 0x7fd29400b260 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.984+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2a80058b0 con 0x7fd29400b260 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.985+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.985+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fd2a8005ab0 con 0x7fd29400b260 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.985+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.986+0000 7fd29f7fe700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 0x7fd29403fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:16.996 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.986+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd2a8049820 con 0x7fd29400b260 2020-02-05T16:59:16.996 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.986+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd288003360 con 0x7fd29400b260 2020-02-05T16:59:16.996 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.986+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 0x7fd29403fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:16.996 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.987+0000 7fd2aed9d700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 0x7fd29403fae0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:16.997 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.995+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:16.998 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:16.995+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fd2a800d0a0 con 0x7fd29400b260 2020-02-05T16:59:17.194 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.191+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fd2880037a0 con 0x7fd29400b260 2020-02-05T16:59:17.199 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.197+0000 7fd2b4c90700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.199 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.197+0000 7fd29f7fe700 1 -- 172.21.15.8:0/2070946101 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+91772 (secure 0 0 0) 0x7fd2a8031350 con 0x7fd29400b260 2020-02-05T16:59:17.201 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:17.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.199+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 msgr2=0x7fd29403fae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.199+0000 7fd2af59e700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 0x7fd29403fae0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.199+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 msgr2=0x7fd29400d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.199+0000 7fd2af59e700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fd2a8007920 tx=0x7fd2a8010c30).stop 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 shutdown_connections 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd29403d5d0 0x7fd29403fae0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 --2- 172.21.15.8:0/2070946101 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd29400b260 0x7fd29400d740 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 >> 172.21.15.8:0/2070946101 conn(0x7fd2b00ace50 msgr2=0x7fd2b0024fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 shutdown_connections 2020-02-05T16:59:17.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.200+0000 7fd2af59e700 1 -- 172.21.15.8:0/2070946101 wait complete. 2020-02-05T16:59:17.226 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":4,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":21877},{"type":"v1","addr":"172.21.15.164:6801","nonce":21877}]},"active_addr":"172.21.15.164:6801/21877","active_change":"2020-02-05T16:58:55.453890+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if services are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_services":{"name":"warn_on_stray_services","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if services are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"str","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"str","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["ansible","cephadm","deepsea","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if services are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_services":{"name":"warn_on_stray_services","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if services are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"str","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"str","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["ansible","cephadm","deepsea","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":2172863700}]}]}} 2020-02-05T16:59:17.244 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2020-02-05T16:59:17.244 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2020-02-05T16:59:17.245 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:17.482 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.468+0000 7f21561cb700 1 Processor -- start 2020-02-05T16:59:17.482 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.468+0000 7f21561cb700 1 -- start start 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f21561cb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f21500908f0 0x7f21500a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f21561cb700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500aa3e0 con 0x7f215008f240 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f21561cb700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21500aab90 con 0x7f215008fff0 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f21561cb700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f215008f670 con 0x7f21500908f0 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f213ffff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f215008f240 0x7f21500a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45804/0 (socket says 172.21.15.8:45804) 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f213ffff700 1 -- 172.21.15.8:0/3987951388 learned_addr learned my addr 172.21.15.8:0/3987951388 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:17.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.469+0000 7f2154f69700 1 --2- 172.21.15.8:0/3987951388 >> v2:172.21.15.8:3300/0 conn(0x7f21500908f0 0x7f21500a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095514119 0 0) 0x7f21440010d0 con 0x7f215008f240 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2138003230 con 0x7f215008f240 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 488971676 0 0) 0x7f2138003230 con 0x7f215008f240 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 >> v2:172.21.15.8:3300/0 conn(0x7f21500908f0 msgr2=0x7f21500a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 --2- 172.21.15.8:0/3987951388 >> v2:172.21.15.8:3300/0 conn(0x7f21500908f0 0x7f21500a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 >> v1:172.21.15.164:6789/0 conn(0x7f215008fff0 legacy=0x7f21500a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f215005b340 con 0x7f215008f240 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f2144001980 con 0x7f215008f240 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 >> v1:172.21.15.8:6789/0 conn(0x7f215008f240 legacy=0x7f21500a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.470+0000 7f214f7fe700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214f7fe700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21380045a0 0x7f21380049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214f7fe700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2138004f20 0x7f213800b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213800bdd0 con 0x7f21380038b0 2020-02-05T16:59:17.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f213804ffd0 con 0x7f2138004f20 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2138050210 con 0x7f21380045a0 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f213ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21380045a0 0x7f21380049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.471+0000 7f2154f69700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2138004f20 0x7f213800b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 -- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21380045a0 msgr2=0x7f21380049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21380045a0 0x7f21380049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 -- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2138004f20 msgr2=0x7f213800b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:17.486 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2138004f20 0x7f213800b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 -- 172.21.15.8:0/3987951388 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138050510 con 0x7f21380038b0 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f2154f69700 1 -- 172.21.15.8:0/3987951388 reap_dead start 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2140001000 tx=0x7f214000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2140001000 tx=0x7f214000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.472+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f2140001000 tx=0x7f214000fdd0).ready entity=mon.0 client_cookie=294a3ffeee36aa5d server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f2140001000 tx=0x7f214000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.487 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2140003540 con 0x7f21380038b0 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f2140001000 tx=0x7f214000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214ffff700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f2140001000 tx=0x7f214000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2140003160 con 0x7f21380038b0 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f214f7fe700 1 -- 172.21.15.8:0/3987951388 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2140003b20 con 0x7f21380038b0 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f21561cb700 1 -- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 msgr2=0x7f2138003d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.473+0000 7f21561cb700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f2140001000 tx=0x7f214000fdd0).stop 2020-02-05T16:59:17.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.474+0000 7f21561cb700 1 -- 172.21.15.8:0/3987951388 shutdown_connections 2020-02-05T16:59:17.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.474+0000 7f21561cb700 1 --2- 172.21.15.8:0/3987951388 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f21380038b0 0x7f2138003d20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.474+0000 7f21561cb700 1 -- 172.21.15.8:0/3987951388 >> 172.21.15.8:0/3987951388 conn(0x7f21500600f0 msgr2=0x7f215005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.474+0000 7f21561cb700 1 -- 172.21.15.8:0/3987951388 shutdown_connections 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.474+0000 7f21561cb700 1 -- 172.21.15.8:0/3987951388 wait complete. 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.477+0000 7f21561cb700 1 Processor -- start 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.478+0000 7f21561cb700 1 -- start start 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f21561cb700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21501350d0 0x7f2150135500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:17.496 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f21561cb700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215013d9f0 0x7f215013fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f21561cb700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f215007e100 con 0x7f2150123e80 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f21561cb700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21500a5690 con 0x7f215013d9f0 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f21561cb700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21501223d0 con 0x7f21501350d0 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f2154f69700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215013d9f0 0x7f215013fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.479+0000 7f2154f69700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215013d9f0 0x7f215013fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:59992/0 (socket says 172.21.15.8:59992) 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 -- 172.21.15.8:0/1872212202 learned_addr learned my addr 172.21.15.8:0/1872212202 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:17.497 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f213ffff700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21501350d0 0x7f2150135500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.500 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 -- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21501350d0 msgr2=0x7f2150135500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.500 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21501350d0 0x7f2150135500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.500 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 -- 172.21.15.8:0/1872212202 >> v1:172.21.15.164:6789/0 conn(0x7f2150123e80 legacy=0x7f215007d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 -- 172.21.15.8:0/1872212202 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21501228d0 con 0x7f215013d9f0 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548213800 0 0) 0x7f21500aab90 con 0x7f2150123e80 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.480+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> v2:172.21.15.164:3300/0 conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> v2:172.21.15.164:3300/0 conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2148008760 con 0x7f215013d9f0 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2148005a40 con 0x7f215013d9f0 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2148008d40 con 0x7f215013d9f0 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.481+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21501237c0 con 0x7f215013d9f0 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.482+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21501218d0 con 0x7f215013d9f0 2020-02-05T16:59:17.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.483+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.483+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2150056640 con 0x7f215013d9f0 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.483+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f2148011730 con 0x7f215013d9f0 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.484+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.484+0000 7f214d7fa700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 0x7f213403bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.484+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f2148049250 con 0x7f215013d9f0 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.485+0000 7f214ffff700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 0x7f213403bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.503 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.485+0000 7f214ffff700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 0x7f213403bd50 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.493+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.504 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.493+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f2148030d40 con 0x7f215013d9f0 2020-02-05T16:59:17.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.654+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2150068f50 con 0x7f215013d9f0 2020-02-05T16:59:17.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.656+0000 7f2154f69700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.659 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.656+0000 7f214d7fa700 1 -- 172.21.15.8:0/1872212202 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7f2148030fa0 con 0x7f215013d9f0 2020-02-05T16:59:17.659 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:17.659 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:17.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 msgr2=0x7f213403bd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 0x7f213403bd50 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 msgr2=0x7f215013fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f21500a9c90 tx=0x7f2148010ea0).stop 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 shutdown_connections 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215013d9f0 0x7f215013fe30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2134039840 0x7f213403bd50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 --2- 172.21.15.8:0/1872212202 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f21501350d0 0x7f2150135500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.658+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 >> 172.21.15.8:0/1872212202 conn(0x7f215008f240 msgr2=0x7f21500ab640 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:17.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.659+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 shutdown_connections 2020-02-05T16:59:17.663 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.659+0000 7f21561cb700 1 -- 172.21.15.8:0/1872212202 wait complete. 2020-02-05T16:59:17.682 INFO:tasks.ceph.ceph_manager.ceph:all up! 2020-02-05T16:59:17.682 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:17.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.904+0000 7f182388e700 1 Processor -- start 2020-02-05T16:59:17.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.904+0000 7f182388e700 1 -- start start 2020-02-05T16:59:17.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f181c08fff0 0x7f181c08f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 0x7f181c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0a7990 0x7f181c0a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c090d20 con 0x7f181c08fff0 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c090650 con 0x7f181c0908f0 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f182388e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c08ece0 con 0x7f181c0a7990 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f1821e2b700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f181c08fff0 0x7f181c08f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f1821e2b700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f181c08fff0 0x7f181c08f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44444/0 (socket says 172.21.15.8:44444) 2020-02-05T16:59:17.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.905+0000 7f1821e2b700 1 -- 172.21.15.8:0/917768522 learned_addr learned my addr 172.21.15.8:0/917768522 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f182162a700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 0x7f181c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f182262c700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0a7990 0x7f181c0a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 -- 172.21.15.8:0/917768522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0a7990 msgr2=0x7f181c0a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0a7990 0x7f181c0a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 -- 172.21.15.8:0/917768522 >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 msgr2=0x7f181c0a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:17.920 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 0x7f181c0a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 -- 172.21.15.8:0/917768522 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0aa3e0 con 0x7f181c08fff0 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f182162a700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 0x7f181c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.906+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.8:3300/0 conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f181c09ede0 tx=0x7f18180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.8:3300/0 conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f181c09ede0 tx=0x7f18180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f181c09ede0 tx=0x7f18180102b0).ready entity=mon.0 client_cookie=68b63f8f873681a2 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f181c09ede0 tx=0x7f18180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f181c09ede0 tx=0x7f18180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1821e2b700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f181c09ede0 tx=0x7f18180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1820e29700 1 -- 172.21.15.8:0/917768522 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181800a160 con 0x7f181c08fff0 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1820e29700 1 -- 172.21.15.8:0/917768522 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1818011bf0 con 0x7f181c08fff0 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f1820e29700 1 -- 172.21.15.8:0/917768522 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181800a740 con 0x7f181c08fff0 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f182388e700 1 -- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 msgr2=0x7f181c08f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.907+0000 7f182388e700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f181c09ede0 tx=0x7f18180102b0).stop 2020-02-05T16:59:17.922 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 -- 172.21.15.8:0/917768522 shutdown_connections 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f181c0a7990 0x7f181c0a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 --2- 172.21.15.8:0/917768522 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f181c08fff0 0x7f181c08f240 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 --2- 172.21.15.8:0/917768522 >> v2:172.21.15.164:3300/0 conn(0x7f181c0908f0 0x7f181c0a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 -- 172.21.15.8:0/917768522 >> 172.21.15.8:0/917768522 conn(0x7f181c0600f0 msgr2=0x7f181c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 -- 172.21.15.8:0/917768522 shutdown_connections 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.909+0000 7f182388e700 1 -- 172.21.15.8:0/917768522 wait complete. 2020-02-05T16:59:17.923 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.911+0000 7f182388e700 1 Processor -- start 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.912+0000 7f182388e700 1 -- start start 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.913+0000 7f182388e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f181c09da70 0x7f181c09dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f182388e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c13e4b0 con 0x7f181c063f00 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f182388e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c13f6f0 con 0x7f181c0617f0 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f182388e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c0644e0 con 0x7f181c09da70 2020-02-05T16:59:17.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f1821e2b700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f181c063f00 0x7f181c062560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45830/0 (socket says 172.21.15.8:45830) 2020-02-05T16:59:17.926 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f1821e2b700 1 -- 172.21.15.8:0/4269718978 learned_addr learned my addr 172.21.15.8:0/4269718978 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:17.926 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.914+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> v2:172.21.15.164:3300/0 conn(0x7f181c09da70 0x7f181c09dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3560514789 0 0) 0x7f181c13e4b0 con 0x7f181c063f00 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1804003230 con 0x7f181c063f00 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517631488 0 0) 0x7f181c13f6f0 con 0x7f181c0617f0 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c13e4b0 con 0x7f181c0617f0 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f182262c700 1 -- 172.21.15.8:0/4269718978 >> v1:172.21.15.164:6789/0 conn(0x7f181c0617f0 legacy=0x7f181c132b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.915+0000 7f182262c700 1 -- 172.21.15.8:0/4269718978 >> v1:172.21.15.8:6789/0 conn(0x7f181c063f00 legacy=0x7f181c062560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:17.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 -- 172.21.15.8:0/4269718978 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c140930 con 0x7f181c09da70 2020-02-05T16:59:17.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> v2:172.21.15.164:3300/0 conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> v2:172.21.15.164:3300/0 conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1814005aa0 con 0x7f181c09da70 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.930 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.916+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18140056c0 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.917+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c13f920 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.917+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1814012780 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.917+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c13fe10 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.918+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.918+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c056640 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.919+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f1814007180 con 0x7f181c09da70 2020-02-05T16:59:17.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.919+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.932 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.919+0000 7f18127fc700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 0x7f180403c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:17.932 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.920+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f18140494d0 con 0x7f181c09da70 2020-02-05T16:59:17.932 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.920+0000 7f1821e2b700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 0x7f180403c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:17.932 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.921+0000 7f1821e2b700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 0x7f180403c590 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:17.932 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.929+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:17.933 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:17.929+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f1814031020 con 0x7f181c09da70 2020-02-05T16:59:18.091 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.089+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f181c068f50 con 0x7f181c09da70 2020-02-05T16:59:18.093 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.091+0000 7f182262c700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=THROTTLE_DONE pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.094 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.091+0000 7f18127fc700 1 -- 172.21.15.8:0/4269718978 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7f1814007490 con 0x7f181c09da70 2020-02-05T16:59:18.094 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:18.094 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:18.095 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 msgr2=0x7f180403c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 0x7f180403c590 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 msgr2=0x7f181c09dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 secure :-1 s=READY pgs=81 cs=0 l=1 rx=0x7f181c0a2040 tx=0x7f1814010ab0).stop 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 shutdown_connections 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f181c09da70 0x7f181c09dea0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 --2- 172.21.15.8:0/4269718978 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f180403a080 0x7f180403c590 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.093+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 >> 172.21.15.8:0/4269718978 conn(0x7f181c05b0c0 msgr2=0x7f181c07cc30 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.094+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 shutdown_connections 2020-02-05T16:59:18.097 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.094+0000 7f182388e700 1 -- 172.21.15.8:0/4269718978 wait complete. 2020-02-05T16:59:18.119 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2020-02-05T16:59:18.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.343+0000 7feb15042700 1 Processor -- start 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.343+0000 7feb15042700 1 -- start start 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.344+0000 7feb15042700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7feb100aa430 0x7feb1013f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.344+0000 7feb15042700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100aa860 con 0x7feb100ace50 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.344+0000 7feb15042700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb101454a0 con 0x7feb100abd50 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.344+0000 7feb15042700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb101456d0 con 0x7feb100aa430 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.345+0000 7feb14841700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7feb100ace50 0x7feb101491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45838/0 (socket says 172.21.15.8:45838) 2020-02-05T16:59:18.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.345+0000 7feb14841700 1 -- 172.21.15.8:0/1692736731 learned_addr learned my addr 172.21.15.8:0/1692736731 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.345+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> v2:172.21.15.8:3300/0 conn(0x7feb100aa430 0x7feb1013f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.345+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317101058 0 0) 0x7feb100aa860 con 0x7feb100ace50 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.345+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf8003230 con 0x7feb100ace50 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984149550 0 0) 0x7feb101454a0 con 0x7feb100abd50 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb100aa860 con 0x7feb100abd50 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1126302528 0 0) 0x7feaf8003230 con 0x7feb100ace50 2020-02-05T16:59:18.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 >> v2:172.21.15.8:3300/0 conn(0x7feb100aa430 msgr2=0x7feb1013f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 --2- 172.21.15.8:0/1692736731 >> v2:172.21.15.8:3300/0 conn(0x7feb100aa430 0x7feb1013f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 >> v1:172.21.15.164:6789/0 conn(0x7feb100abd50 legacy=0x7feb10145c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1013f560 con 0x7feb100ace50 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7feb00001750 con 0x7feb100ace50 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> v2:172.21.15.8:3300/0 conn(0x7feb100aa430 0x7feb1013f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 >> v1:172.21.15.8:6789/0 conn(0x7feb100ace50 legacy=0x7feb101491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf8003a90 0x7feaf8003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feaf80045a0 0x7feaf8004a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb101456d0 con 0x7feaf80045a0 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8050040 con 0x7feaf8005120 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.346+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf8050280 con 0x7feaf8003a90 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.347+0000 7feb14841700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf8003a90 0x7feaf8003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.347+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.347+0000 7feb0ffff700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feaf80045a0 0x7feaf8004a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 -- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf8003a90 msgr2=0x7feaf8003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf8003a90 0x7feaf8003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 -- 172.21.15.8:0/1692736731 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feaf80045a0 msgr2=0x7feaf8004a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feaf80045a0 0x7feaf8004a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 -- 172.21.15.8:0/1692736731 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf8050520 con 0x7feaf8005120 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb14841700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feaf8003a90 0x7feaf8003ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:18.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feb080032b0 tx=0x7feb0800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 -- 172.21.15.8:0/1692736731 reap_dead start 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feb080032b0 tx=0x7feb0800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb0800c330).ready entity=mon.1 client_cookie=aaa0050b34d8d263 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb0800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb08005a40 con 0x7feaf8005120 2020-02-05T16:59:18.361 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb0800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0800cb30 con 0x7feaf8005120 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb162a4700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb0800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.348+0000 7feb0f7fe700 1 -- 172.21.15.8:0/1692736731 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb0800e350 con 0x7feaf8005120 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.349+0000 7feb15042700 1 -- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 msgr2=0x7feaf804f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.349+0000 7feb15042700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb0800c330).stop 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.350+0000 7feb15042700 1 -- 172.21.15.8:0/1692736731 shutdown_connections 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.350+0000 7feb15042700 1 --2- 172.21.15.8:0/1692736731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feaf8005120 0x7feaf804f880 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.362 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.350+0000 7feb15042700 1 -- 172.21.15.8:0/1692736731 >> 172.21.15.8:0/1692736731 conn(0x7feb10061510 msgr2=0x7feb100ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.350+0000 7feb15042700 1 -- 172.21.15.8:0/1692736731 shutdown_connections 2020-02-05T16:59:18.363 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.350+0000 7feb15042700 1 -- 172.21.15.8:0/1692736731 wait complete. 2020-02-05T16:59:18.366 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.355+0000 7feb15042700 1 Processor -- start 2020-02-05T16:59:18.366 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.355+0000 7feb15042700 1 -- start start 2020-02-05T16:59:18.366 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb15042700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb1007d6e0 0x7feb1007db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.366 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb15042700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7feb1007e060 0x7feb10132ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb15042700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1007e490 con 0x7feb100a4d60 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb15042700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb10061940 con 0x7feb1007e060 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb15042700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb100a37e0 con 0x7feb1007d6e0 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb162a4700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7feb1007e060 0x7feb10132ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb162a4700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7feb1007e060 0x7feb10132ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44472/0 (socket says 172.21.15.8:44472) 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb162a4700 1 -- 172.21.15.8:0/1041508865 learned_addr learned my addr 172.21.15.8:0/1041508865 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:18.367 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.357+0000 7feb14841700 1 --1- 172.21.15.8:0/1041508865 >> v1:172.21.15.164:6789/0 conn(0x7feb100a4d60 0x7feb100a3a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:58336/0 (socket says 172.21.15.8:58336) 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb0ffff700 1 --2- 172.21.15.8:0/1041508865 >> v2:172.21.15.164:3300/0 conn(0x7feb1007d6e0 0x7feb1007db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1732835581 0 0) 0x7feb1007e490 con 0x7feb100a4d60 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf0003230 con 0x7feb100a4d60 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb162a4700 1 -- 172.21.15.8:0/1041508865 >> v2:172.21.15.164:3300/0 conn(0x7feb1007d6e0 msgr2=0x7feb1007db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> v2:172.21.15.164:3300/0 conn(0x7feb1007d6e0 0x7feb1007db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb162a4700 1 -- 172.21.15.8:0/1041508865 >> v1:172.21.15.164:6789/0 conn(0x7feb100a4d60 legacy=0x7feb100a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.358+0000 7feb162a4700 1 -- 172.21.15.8:0/1041508865 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100a4240 con 0x7feb1007e060 2020-02-05T16:59:18.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> v2:172.21.15.8:3300/0 conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> v2:172.21.15.8:3300/0 conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb080073b0 con 0x7feb1007e060 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.369 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb08005c40 con 0x7feb1007e060 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb08008690 con 0x7feb1007e060 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.359+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb100a54e0 con 0x7feb1007e060 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.360+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb101402b0 con 0x7feb1007e060 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.361+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.361+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feb10056640 con 0x7feb1007e060 2020-02-05T16:59:18.370 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.362+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7feb0800b220 con 0x7feb1007e060 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.362+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb0d7fa700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7feaf0039f00 0x7feaf003c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb162a4700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7feb0000b7d0 con 0x7feb1007e060 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb0d7fa700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] conn(0x7feaf00405b0 0x7feaf0042a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 --> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feaf00430d0 con 0x7feaf00405b0 2020-02-05T16:59:18.371 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.363+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7feb080350a0 con 0x7feb1007e060 2020-02-05T16:59:18.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.364+0000 7feb14841700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7feaf0039f00 0x7feaf003c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.364+0000 7feb0ffff700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] conn(0x7feaf00405b0 0x7feaf0042a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.372 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.364+0000 7feb14841700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7feaf0039f00 0x7feaf003c410 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.373 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.365+0000 7feb0ffff700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] conn(0x7feaf00405b0 0x7feaf0042a20 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.373 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.367+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== osd.0 v2:172.21.15.8:6800/22695 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7feaf00430d0 con 0x7feaf00405b0 2020-02-05T16:59:18.394 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.392+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 --> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feb1007ff00 con 0x7feaf00405b0 2020-02-05T16:59:18.395 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.393+0000 7feb0d7fa700 1 -- 172.21.15.8:0/1041508865 <== osd.0 v2:172.21.15.8:6800/22695 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feb1007ff00 con 0x7feaf00405b0 2020-02-05T16:59:18.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.393+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] conn(0x7feaf00405b0 msgr2=0x7feaf0042a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.396 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.393+0000 7feb15042700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:6800/22695,v1:172.21.15.8:6801/22695] conn(0x7feaf00405b0 0x7feaf0042a20 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.394+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7feaf0039f00 msgr2=0x7feaf003c410 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.394+0000 7feb15042700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7feaf0039f00 0x7feaf003c410 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.394+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 msgr2=0x7feb10132ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.394+0000 7feb15042700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7feb080032b0 tx=0x7feb08010370).stop 2020-02-05T16:59:18.397 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.394+0000 7feb162a4700 1 -- 172.21.15.8:0/1041508865 reap_dead start 2020-02-05T16:59:18.398 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.395+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 shutdown_connections 2020-02-05T16:59:18.398 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.395+0000 7feb15042700 1 --2- 172.21.15.8:0/1041508865 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7feb1007e060 0x7feb10132ef0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.398 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.395+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 >> 172.21.15.8:0/1041508865 conn(0x7feb100abd50 msgr2=0x7feb1005ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.398 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.395+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 shutdown_connections 2020-02-05T16:59:18.399 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.395+0000 7feb15042700 1 -- 172.21.15.8:0/1041508865 wait complete. 2020-02-05T16:59:18.414 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:18.415 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2020-02-05T16:59:18.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.643+0000 7ff2f54a9700 1 Processor -- start 2020-02-05T16:59:18.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.643+0000 7ff2f54a9700 1 -- start start 2020-02-05T16:59:18.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f54a9700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7ff2f00ace50 0x7ff2f00aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f54a9700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2f0145c20 0x7ff2f0146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f54a9700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f00aa980 con 0x7ff2f00abd50 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f54a9700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f0148c70 con 0x7ff2f00ace50 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f54a9700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f0148f10 con 0x7ff2f0145c20 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2effff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7ff2f00abd50 0x7ff2f01491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45860/0 (socket says 172.21.15.8:45860) 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2effff700 1 -- 172.21.15.8:0/3919945454 learned_addr learned my addr 172.21.15.8:0/3919945454 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.644+0000 7ff2f4ca8700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.8:3300/0 conn(0x7ff2f00ace50 0x7ff2f00aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.654 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.164:3300/0 conn(0x7ff2f0145c20 0x7ff2f0146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847850085 0 0) 0x7ff2dc0010d0 con 0x7ff2f00abd50 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2d4003230 con 0x7ff2f00abd50 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2182288915 0 0) 0x7ff2d4003230 con 0x7ff2f00abd50 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 >> v2:172.21.15.164:3300/0 conn(0x7ff2f0145c20 msgr2=0x7ff2f0146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.164:3300/0 conn(0x7ff2f0145c20 0x7ff2f0146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 >> v2:172.21.15.8:3300/0 conn(0x7ff2f00ace50 msgr2=0x7ff2f00aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.655 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.8:3300/0 conn(0x7ff2f00ace50 0x7ff2f00aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.645+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f01454a0 con 0x7ff2f00abd50 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7ff2dc001980 con 0x7ff2f00abd50 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 >> v1:172.21.15.8:6789/0 conn(0x7ff2f00abd50 legacy=0x7ff2f01491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d4003960 0x7ff2d4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d40045a0 0x7ff2d40049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.656 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0148c70 con 0x7ff2d4004fc0 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0148f10 con 0x7ff2d40045a0 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2d404ff60 con 0x7ff2d4003960 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.164:3300/0 conn(0x7ff2f0145c20 0x7ff2f0146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2f4ca8700 1 --2- 172.21.15.8:0/3919945454 >> v2:172.21.15.8:3300/0 conn(0x7ff2f00ace50 0x7ff2f00aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.646+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2effff700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d40045a0 0x7ff2d40049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 -- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d4003960 msgr2=0x7ff2d4003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2d4003960 0x7ff2d4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 -- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d40045a0 msgr2=0x7ff2d40049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d40045a0 0x7ff2d40049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 -- 172.21.15.8:0/3919945454 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d4050260 con 0x7ff2d4004fc0 2020-02-05T16:59:18.660 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2effff700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2d40045a0 0x7ff2d40049d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 -- 172.21.15.8:0/3919945454 reap_dead start 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).ready entity=mon.0 client_cookie=36494a9a325d5b25 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.661 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2e8005a40 con 0x7ff2d4004fc0 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e800cb30 con 0x7ff2d4004fc0 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2ef7fe700 1 -- 172.21.15.8:0/3919945454 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2e800e350 con 0x7ff2d4004fc0 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 msgr2=0x7ff2d404f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.647+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7ff2e80032b0 tx=0x7ff2e800c330).stop 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.648+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3919945454 shutdown_connections 2020-02-05T16:59:18.662 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.648+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3919945454 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7ff2d4004fc0 0x7ff2d404f800 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.663 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.648+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3919945454 >> 172.21.15.8:0/3919945454 conn(0x7ff2f0061510 msgr2=0x7ff2f00ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.663 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.648+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3919945454 shutdown_connections 2020-02-05T16:59:18.663 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.648+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3919945454 wait complete. 2020-02-05T16:59:18.667 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.651+0000 7ff2f54a9700 1 Processor -- start 2020-02-05T16:59:18.667 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.651+0000 7ff2f54a9700 1 -- start start 2020-02-05T16:59:18.667 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.652+0000 7ff2f54a9700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2f00a7390 0x7ff2f0144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.667 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.652+0000 7ff2f54a9700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2f0143b20 0x7ff2f0137940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.652+0000 7ff2f54a9700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f0145220 con 0x7ff2f0120920 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.652+0000 7ff2f54a9700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f00a31e0 con 0x7ff2f00a7390 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.652+0000 7ff2f54a9700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0144790 con 0x7ff2f0143b20 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.653+0000 7ff2effff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2f00a7390 0x7ff2f0144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.653+0000 7ff2f670b700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2f0143b20 0x7ff2f0137940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.653+0000 7ff2effff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff2f00a7390 0x7ff2f0144cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60050/0 (socket says 172.21.15.8:60050) 2020-02-05T16:59:18.668 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.653+0000 7ff2effff700 1 -- 172.21.15.8:0/3727322409 learned_addr learned my addr 172.21.15.8:0/3727322409 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2effff700 1 -- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2f0143b20 msgr2=0x7ff2f0137940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2f0143b20 0x7ff2f0137940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2effff700 1 -- 172.21.15.8:0/3727322409 >> v1:172.21.15.164:6789/0 conn(0x7ff2f0120920 legacy=0x7ff2f0143320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2effff700 1 -- 172.21.15.8:0/3727322409 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00a77c0 con 0x7ff2f00a7390 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff2f0143b20 0x7ff2f0137940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.654+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> v2:172.21.15.164:3300/0 conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.669 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> v2:172.21.15.164:3300/0 conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2dc00cc00 con 0x7ff2f00a7390 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc00c820 con 0x7ff2f00a7390 2020-02-05T16:59:18.670 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f00a8d50 con 0x7ff2f00a7390 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2dc00ce00 con 0x7ff2f00a7390 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.655+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f0143f50 con 0x7ff2f00a7390 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.657+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff2f0056640 con 0x7ff2f00a7390 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.657+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.658+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.658+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7ff2dc004770 con 0x7ff2f00a7390 2020-02-05T16:59:18.671 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.659+0000 7ff2effff700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.659+0000 7ff2ed7fa700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff2cc039a60 0x7ff2cc03bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.659+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7ff2f00aa980 con 0x7ff2f00a7390 2020-02-05T16:59:18.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.660+0000 7ff2ed7fa700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] conn(0x7ff2cc040230 0x7ff2cc0426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.660+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 --> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff2cc042d50 con 0x7ff2cc040230 2020-02-05T16:59:18.672 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.660+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2dc04a220 con 0x7ff2f00a7390 2020-02-05T16:59:18.674 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.660+0000 7ff2f4ca8700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff2cc039a60 0x7ff2cc03bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.674 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.660+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] conn(0x7ff2cc040230 0x7ff2cc0426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.674 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.661+0000 7ff2f4ca8700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff2cc039a60 0x7ff2cc03bf70 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.674 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.661+0000 7ff2f670b700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] conn(0x7ff2cc040230 0x7ff2cc0426a0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.674 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.663+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== osd.1 v2:172.21.15.8:6808/22696 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7ff2cc042d50 con 0x7ff2cc040230 2020-02-05T16:59:18.692 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.689+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 --> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff2f011f9a0 con 0x7ff2cc040230 2020-02-05T16:59:18.692 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.690+0000 7ff2ed7fa700 1 -- 172.21.15.8:0/3727322409 <== osd.1 v2:172.21.15.8:6808/22696 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff2f011f9a0 con 0x7ff2cc040230 2020-02-05T16:59:18.700 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 >> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] conn(0x7ff2cc040230 msgr2=0x7ff2cc0426a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.700 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.8:6808/22696,v1:172.21.15.8:6809/22696] conn(0x7ff2cc040230 0x7ff2cc0426a0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff2cc039a60 msgr2=0x7ff2cc03bf70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7ff2cc039a60 0x7ff2cc03bf70 crc :-1 s=READY pgs=59 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 msgr2=0x7ff2f0144cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7ff2dc001c10 tx=0x7ff2dc010a40).stop 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f670b700 1 -- 172.21.15.8:0/3727322409 reap_dead start 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 shutdown_connections 2020-02-05T16:59:18.701 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 --2- 172.21.15.8:0/3727322409 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff2f00a7390 0x7ff2f0144cd0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.702 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.691+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 >> 172.21.15.8:0/3727322409 conn(0x7ff2f00ace50 msgr2=0x7ff2f00ac240 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.702 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.692+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 shutdown_connections 2020-02-05T16:59:18.702 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.692+0000 7ff2f54a9700 1 -- 172.21.15.8:0/3727322409 wait complete. 2020-02-05T16:59:18.713 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:18.713 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2020-02-05T16:59:18.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.942+0000 7f2962810700 1 Processor -- start 2020-02-05T16:59:18.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 -- start start 2020-02-05T16:59:18.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f295c08fff0 0x7f295c08f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:18.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c0aa3e0 con 0x7f295c0a7990 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295c090d20 con 0x7f295c08fff0 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.943+0000 7f2962810700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c090650 con 0x7f295c0908f0 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.944+0000 7f29615ae700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f295c0a7990 0x7f295c0a7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45886/0 (socket says 172.21.15.8:45886) 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.944+0000 7f29615ae700 1 -- 172.21.15.8:0/1074351144 learned_addr learned my addr 172.21.15.8:0/1074351144 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.944+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.944+0000 7f2960dad700 1 --2- 172.21.15.8:0/1074351144 >> v2:172.21.15.164:3300/0 conn(0x7f295c08fff0 0x7f295c08f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 -- 172.21.15.8:0/1074351144 >> v2:172.21.15.164:3300/0 conn(0x7f295c08fff0 msgr2=0x7f295c08f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> v2:172.21.15.164:3300/0 conn(0x7f295c08fff0 0x7f295c08f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 -- 172.21.15.8:0/1074351144 >> v1:172.21.15.8:6789/0 conn(0x7f295c0a7990 legacy=0x7f295c0a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 -- 172.21.15.8:0/1074351144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c0aab90 con 0x7f295c0908f0 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2950008910 tx=0x7f295000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2950008910 tx=0x7f295000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.945+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f2950008910 tx=0x7f295000fe70).ready entity=mon.2 client_cookie=8e6921f876c858f4 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f2950008910 tx=0x7f295000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295b7fe700 1 -- 172.21.15.8:0/1074351144 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29500116e0 con 0x7f295c0908f0 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f2950008910 tx=0x7f295000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295b7fe700 1 -- 172.21.15.8:0/1074351144 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2950008c50 con 0x7f295c0908f0 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295bfff700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7f2950008910 tx=0x7f295000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f295b7fe700 1 -- 172.21.15.8:0/1074351144 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2950011cc0 con 0x7f295c0908f0 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f2962810700 1 -- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 msgr2=0x7f295c0a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.946+0000 7f2962810700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7f2950008910 tx=0x7f295000fe70).stop 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.947+0000 7f2962810700 1 -- 172.21.15.8:0/1074351144 shutdown_connections 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.947+0000 7f2962810700 1 --2- 172.21.15.8:0/1074351144 >> v2:172.21.15.164:3300/0 conn(0x7f295c08fff0 0x7f295c08f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.947+0000 7f2962810700 1 --2- 172.21.15.8:0/1074351144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0908f0 0x7f295c0a9e90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.947+0000 7f2962810700 1 -- 172.21.15.8:0/1074351144 >> 172.21.15.8:0/1074351144 conn(0x7f295c0600f0 msgr2=0x7f295c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.948+0000 7f2962810700 1 -- 172.21.15.8:0/1074351144 shutdown_connections 2020-02-05T16:59:18.956 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.948+0000 7f2962810700 1 -- 172.21.15.8:0/1074351144 wait complete. 2020-02-05T16:59:18.959 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.951+0000 7f2962810700 1 Processor -- start 2020-02-05T16:59:18.959 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.952+0000 7f2962810700 1 -- start start 2020-02-05T16:59:18.959 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2962810700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0a85e0 0x7f295c07d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.960 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2962810700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f295c09f580 0x7f295c09f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2962810700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295c07d710 con 0x7f295c0a2640 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2962810700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295c065ae0 con 0x7f295c09f580 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2962810700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295c065d80 con 0x7f295c0a85e0 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f295bfff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f295c09f580 0x7f295c09f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f2960dad700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0a85e0 0x7f295c07d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f295bfff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f295c09f580 0x7f295c09f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60062/0 (socket says 172.21.15.8:60062) 2020-02-05T16:59:18.961 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.954+0000 7f295bfff700 1 -- 172.21.15.8:0/1624934990 learned_addr learned my addr 172.21.15.8:0/1624934990 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.955+0000 7f295bfff700 1 -- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0a85e0 msgr2=0x7f295c07d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.955+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f295c0a85e0 0x7f295c07d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.955+0000 7f295bfff700 1 -- 172.21.15.8:0/1624934990 >> v1:172.21.15.164:6789/0 conn(0x7f295c0a2640 legacy=0x7f295c0a2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.955+0000 7f295bfff700 1 -- 172.21.15.8:0/1624934990 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c138290 con 0x7f295c09f580 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.955+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> v2:172.21.15.164:3300/0 conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> v2:172.21.15.164:3300/0 conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.962 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f295000f760 con 0x7f295c09f580 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29500118e0 con 0x7f295c09f580 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c0652c0 con 0x7f295c09f580 2020-02-05T16:59:18.963 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2950007530 con 0x7f295c09f580 2020-02-05T16:59:18.964 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.956+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c0a3270 con 0x7f295c09f580 2020-02-05T16:59:18.965 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.957+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.965 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.958+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f295c056640 con 0x7f295c09f580 2020-02-05T16:59:18.965 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.958+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f29500056c0 con 0x7f295c09f580 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.958+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f295bfff700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=THROTTLE_DONE pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f29597fa700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f293c039940 0x7f293c03be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f2950049d20 con 0x7f295c09f580 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f29597fa700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] conn(0x7f293c040110 0x7f293c042580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:18.966 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 --> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f293c042c30 con 0x7f293c040110 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f295004a140 con 0x7f295c09f580 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.959+0000 7f2960dad700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f293c039940 0x7f293c03be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.960+0000 7f29615ae700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] conn(0x7f293c040110 0x7f293c042580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.960+0000 7f2960dad700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f293c039940 0x7f293c03be50 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.960+0000 7f29615ae700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] conn(0x7f293c040110 0x7f293c042580 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:18.967 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.961+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== osd.2 v2:172.21.15.8:6816/22697 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7f293c042c30 con 0x7f293c040110 2020-02-05T16:59:18.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.988+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 --> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f295c068f50 con 0x7f293c040110 2020-02-05T16:59:18.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.988+0000 7f29597fa700 1 -- 172.21.15.8:0/1624934990 <== osd.2 v2:172.21.15.8:6816/22697 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f295c068f50 con 0x7f293c040110 2020-02-05T16:59:18.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 >> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] conn(0x7f293c040110 msgr2=0x7f293c042580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.8:6816/22697,v1:172.21.15.8:6817/22697] conn(0x7f293c040110 0x7f293c042580 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f293c039940 msgr2=0x7f293c03be50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f293c039940 0x7f293c03be50 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 msgr2=0x7f295c09f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:18.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f2962810700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 secure :-1 s=READY pgs=88 cs=0 l=1 rx=0x7f295c07d940 tx=0x7f2950011820).stop 2020-02-05T16:59:18.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.989+0000 7f29615ae700 1 -- 172.21.15.8:0/1624934990 reap_dead start 2020-02-05T16:59:18.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.990+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 shutdown_connections 2020-02-05T16:59:18.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.990+0000 7f2962810700 1 --2- 172.21.15.8:0/1624934990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f295c09f580 0x7f295c09f9b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:18.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.990+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 >> 172.21.15.8:0/1624934990 conn(0x7f295c05b0c0 msgr2=0x7f295c07c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:18.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.990+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 shutdown_connections 2020-02-05T16:59:18.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:18.990+0000 7f2962810700 1 -- 172.21.15.8:0/1624934990 wait complete. 2020-02-05T16:59:19.010 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:19.011 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2020-02-05T16:59:19.242 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.231+0000 7f7765c06700 1 Processor -- start 2020-02-05T16:59:19.242 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.231+0000 7f7765c06700 1 -- start start 2020-02-05T16:59:19.242 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.232+0000 7f7765c06700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f776008f240 0x7f77600908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.242 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.232+0000 7f7765c06700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.242 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.232+0000 7f7765c06700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77600aab90 con 0x7f776008fff0 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.232+0000 7f7765c06700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776008f670 con 0x7f776008f240 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.232+0000 7f7765c06700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7760090650 con 0x7f77600a9e90 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f775ffff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f776008fff0 0x7f77600a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45898/0 (socket says 172.21.15.8:45898) 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f775ffff700 1 -- 172.21.15.8:0/3232046805 learned_addr learned my addr 172.21.15.8:0/3232046805 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f775f7fe700 1 --2- 172.21.15.8:0/3232046805 >> v2:172.21.15.164:3300/0 conn(0x7f776008f240 0x7f77600908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.243 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f775effd700 1 -- 172.21.15.8:0/3232046805 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329187856 0 0) 0x7f77600aab90 con 0x7f776008fff0 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.233+0000 7f775effd700 1 -- 172.21.15.8:0/3232046805 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7748003230 con 0x7f776008fff0 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 -- 172.21.15.8:0/3232046805 >> v2:172.21.15.164:3300/0 conn(0x7f776008f240 msgr2=0x7f77600908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> v2:172.21.15.164:3300/0 conn(0x7f776008f240 0x7f77600908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 -- 172.21.15.8:0/3232046805 >> v1:172.21.15.8:6789/0 conn(0x7f776008fff0 legacy=0x7f77600a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 -- 172.21.15.8:0/3232046805 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600a86e0 con 0x7f77600a9e90 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7758008c90 tx=0x7f77580101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.244 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7758008c90 tx=0x7f77580101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f7758008c90 tx=0x7f77580101f0).ready entity=mon.2 client_cookie=593e75ce42581b55 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7758008c90 tx=0x7f77580101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.234+0000 7f775effd700 1 -- 172.21.15.8:0/3232046805 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f775800a160 con 0x7f77600a9e90 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7758008c90 tx=0x7f77580101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f77649a4700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7758008c90 tx=0x7f77580101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f775effd700 1 -- 172.21.15.8:0/3232046805 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7758011b70 con 0x7f77600a9e90 2020-02-05T16:59:19.245 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f775effd700 1 -- 172.21.15.8:0/3232046805 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f775800a740 con 0x7f77600a9e90 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f7765c06700 1 -- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 msgr2=0x7f77600a8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.235+0000 7f7765c06700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f7758008c90 tx=0x7f77580101f0).stop 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 -- 172.21.15.8:0/3232046805 shutdown_connections 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 --2- 172.21.15.8:0/3232046805 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f77600a9e90 0x7f77600a8190 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 --2- 172.21.15.8:0/3232046805 >> v2:172.21.15.164:3300/0 conn(0x7f776008f240 0x7f77600908f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 -- 172.21.15.8:0/3232046805 >> 172.21.15.8:0/3232046805 conn(0x7f77600600f0 msgr2=0x7f776005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.246 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 -- 172.21.15.8:0/3232046805 shutdown_connections 2020-02-05T16:59:19.247 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.236+0000 7f7765c06700 1 -- 172.21.15.8:0/3232046805 wait complete. 2020-02-05T16:59:19.247 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.239+0000 7f7765c06700 1 Processor -- start 2020-02-05T16:59:19.247 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.239+0000 7f7765c06700 1 -- start start 2020-02-05T16:59:19.250 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f7765c06700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f77600a4000 0x7f776007c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.250 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f7765c06700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f776009de40 0x7f776009e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f7765c06700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776007c820 con 0x7f77600a0520 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f7765c06700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776007d2e0 con 0x7f77600a4000 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f7765c06700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776007ca50 con 0x7f776009de40 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f775ffff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f77600a4000 0x7f776007c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.240+0000 7f775ffff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f77600a4000 0x7f776007c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44522/0 (socket says 172.21.15.8:44522) 2020-02-05T16:59:19.251 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775ffff700 1 -- 172.21.15.8:0/3210851921 learned_addr learned my addr 172.21.15.8:0/3210851921 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775f7fe700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f776009de40 0x7f776009e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775ffff700 1 -- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f776009de40 msgr2=0x7f776009e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f776009de40 0x7f776009e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775ffff700 1 -- 172.21.15.8:0/3210851921 >> v1:172.21.15.8:6789/0 conn(0x7f77600a0520 legacy=0x7f776007a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 335198329 0 0) 0x7f776007c820 con 0x7f77600a0520 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.241+0000 7f775ffff700 1 -- 172.21.15.8:0/3210851921 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776009e7c0 con 0x7f77600a4000 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> v2:172.21.15.8:3300/0 conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.252 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> v2:172.21.15.8:3300/0 conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7750005a60 con 0x7f77600a4000 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7750005680 con 0x7f77600a4000 2020-02-05T16:59:19.253 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7750012620 con 0x7f77600a4000 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.242+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77600a0950 con 0x7f77600a4000 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.243+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776007ab40 con 0x7f77600a4000 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.244+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.244+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f7750012940 con 0x7f77600a4000 2020-02-05T16:59:19.254 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.244+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.244+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f77600a0f20 con 0x7f77600a4000 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.244+0000 7f775cff9700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f77400398b0 0x7f774003bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f7750049340 con 0x7f77600a4000 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775ffff700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775cff9700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] conn(0x7f7740040080 0x7f77400424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.255 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 --> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7740042ba0 con 0x7f7740040080 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f776007d2e0 con 0x7f77600a4000 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f775f7fe700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f77400398b0 0x7f774003bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.245+0000 7f77649a4700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] conn(0x7f7740040080 0x7f77400424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.246+0000 7f775f7fe700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f77400398b0 0x7f774003bdc0 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.246+0000 7f77649a4700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] conn(0x7f7740040080 0x7f77400424f0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.256 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.247+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== osd.3 v2:172.21.15.8:6819/22698 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7f7740042ba0 con 0x7f7740040080 2020-02-05T16:59:19.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.272+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 --> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7760056640 con 0x7f7740040080 2020-02-05T16:59:19.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.273+0000 7f775cff9700 1 -- 172.21.15.8:0/3210851921 <== osd.3 v2:172.21.15.8:6819/22698 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7760056640 con 0x7f7740040080 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] conn(0x7f7740040080 msgr2=0x7f77400424f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:6819/22698,v1:172.21.15.8:6821/22698] conn(0x7f7740040080 0x7f77400424f0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f77400398b0 msgr2=0x7f774003bdc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f77400398b0 0x7f774003bdc0 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 msgr2=0x7f776007c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f7750000e90 tx=0x7f77500109f0).stop 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f77649a4700 1 -- 172.21.15.8:0/3210851921 reap_dead start 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 shutdown_connections 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 --2- 172.21.15.8:0/3210851921 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f77600a4000 0x7f776007c2d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 >> 172.21.15.8:0/3210851921 conn(0x7f7760090e00 msgr2=0x7f776005ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 shutdown_connections 2020-02-05T16:59:19.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.274+0000 7f7765c06700 1 -- 172.21.15.8:0/3210851921 wait complete. 2020-02-05T16:59:19.295 INFO:teuthology.orchestra.run.smithi008.stdout:55834574854 2020-02-05T16:59:19.296 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2020-02-05T16:59:19.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.514+0000 7f88a10e3700 1 Processor -- start 2020-02-05T16:59:19.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.514+0000 7f88a10e3700 1 -- start start 2020-02-05T16:59:19.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.515+0000 7f88a10e3700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f889c08f240 0x7f889c0908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.526 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.515+0000 7f88a10e3700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f889c0a9e90 0x7f889c0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.515+0000 7f88a10e3700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c0aab90 con 0x7f889c08fff0 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.515+0000 7f88a10e3700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c08f670 con 0x7f889c08f240 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.515+0000 7f88a10e3700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c090650 con 0x7f889c0a9e90 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f889b59e700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f889c08fff0 0x7f889c0a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45916/0 (socket says 172.21.15.8:45916) 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f889b59e700 1 -- 172.21.15.8:0/2090655783 learned_addr learned my addr 172.21.15.8:0/2090655783 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> v2:172.21.15.8:3300/0 conn(0x7f889c08f240 0x7f889c0908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2090655783 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a9e90 0x7f889c0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.527 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424741403 0 0) 0x7f889c0aab90 con 0x7f889c08fff0 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.516+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8884003230 con 0x7f889c08fff0 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1515294343 0 0) 0x7f8884003230 con 0x7f889c08fff0 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a9e90 msgr2=0x7f889c0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 --2- 172.21.15.8:0/2090655783 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a9e90 0x7f889c0a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 >> v2:172.21.15.8:3300/0 conn(0x7f889c08f240 msgr2=0x7f889c0908f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.528 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 --2- 172.21.15.8:0/2090655783 >> v2:172.21.15.8:3300/0 conn(0x7f889c08f240 0x7f889c0908f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c0a86e0 con 0x7f889c08fff0 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f888c001ad0 con 0x7f889c08fff0 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 >> v1:172.21.15.8:6789/0 conn(0x7f889c08fff0 legacy=0x7f889c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8884003890 0x7f8884003d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8884004f20 0x7f888404f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.529 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c090650 con 0x7f88840045a0 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888404fe10 con 0x7f8884004f20 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8884050050 con 0x7f8884003890 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.517+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889b59e700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8884003890 0x7f8884003d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8884004f20 0x7f888404f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 -- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8884003890 msgr2=0x7f8884003d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.530 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8884003890 0x7f8884003d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 -- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8884004f20 msgr2=0x7f888404f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8884004f20 0x7f888404f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 -- 172.21.15.8:0/2090655783 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884050350 con 0x7f88840045a0 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f88a08e2700 1 -- 172.21.15.8:0/2090655783 reap_dead start 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.518+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8890002f30 tx=0x7f8890007bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8890002f30 tx=0x7f8890007bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.531 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f8890002f30 tx=0x7f8890007bd0).ready entity=mon.0 client_cookie=515387b540bda5 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8890002f30 tx=0x7f8890007bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8890008d30 con 0x7f88840045a0 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8890002f30 tx=0x7f8890007bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7f8890002f30 tx=0x7f8890007bd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8890008950 con 0x7f88840045a0 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f889a59c700 1 -- 172.21.15.8:0/2090655783 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8890010cf0 con 0x7f88840045a0 2020-02-05T16:59:19.532 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f88a10e3700 1 -- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 msgr2=0x7f88840049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.519+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7f8890002f30 tx=0x7f8890007bd0).stop 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.520+0000 7f88a10e3700 1 -- 172.21.15.8:0/2090655783 shutdown_connections 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.520+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2090655783 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f88840045a0 0x7f88840049d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.520+0000 7f88a10e3700 1 -- 172.21.15.8:0/2090655783 >> 172.21.15.8:0/2090655783 conn(0x7f889c0600f0 msgr2=0x7f889c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.521+0000 7f88a10e3700 1 -- 172.21.15.8:0/2090655783 shutdown_connections 2020-02-05T16:59:19.533 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.521+0000 7f88a10e3700 1 -- 172.21.15.8:0/2090655783 wait complete. 2020-02-05T16:59:19.537 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.525+0000 7f88a10e3700 1 Processor -- start 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.525+0000 7f88a10e3700 1 -- start start 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a10e3700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f889c0a0980 0x7f889c07cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a10e3700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f889c07d460 0x7f889c13f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a10e3700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f889c060cc0 con 0x7f889c0a6fc0 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a10e3700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0a0db0 con 0x7f889c07d460 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a10e3700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f889c0a1800 con 0x7f889c0a0980 2020-02-05T16:59:19.538 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a08e2700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f889c07d460 0x7f889c13f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a08e2700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f889c07d460 0x7f889c13f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44550/0 (socket says 172.21.15.8:44550) 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f88a08e2700 1 -- 172.21.15.8:0/2579054689 learned_addr learned my addr 172.21.15.8:0/2579054689 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.526+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2579054689 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a0980 0x7f889c07cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 -- 172.21.15.8:0/2579054689 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a0980 msgr2=0x7f889c07cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> v2:172.21.15.164:3300/0 conn(0x7f889c0a0980 0x7f889c07cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 -- 172.21.15.8:0/2579054689 >> v1:172.21.15.164:6789/0 conn(0x7f889c0a6fc0 legacy=0x7f889c09ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.539 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 -- 172.21.15.8:0/2579054689 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f889c066920 con 0x7f889c07d460 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> v2:172.21.15.8:3300/0 conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> v2:172.21.15.8:3300/0 conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.527+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.540 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8894005aa0 con 0x7f889c07d460 2020-02-05T16:59:19.541 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88940056c0 con 0x7f889c07d460 2020-02-05T16:59:19.541 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c09fd20 con 0x7f889c07d460 2020-02-05T16:59:19.542 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88940085d0 con 0x7f889c07d460 2020-02-05T16:59:19.542 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.528+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c0a1f00 con 0x7f889c07d460 2020-02-05T16:59:19.542 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.529+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.542 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.530+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f889c056640 con 0x7f889c07d460 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.530+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f8894007140 con 0x7f889c07d460 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.530+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f88a08e2700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f8883fff700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f887c039900 0x7f887c03be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f8894049050 con 0x7f889c07d460 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f8883fff700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] conn(0x7f887c0400d0 0x7f887c042540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.543 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 --> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f887c042bf0 con 0x7f887c0400d0 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.531+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8894049470 con 0x7f889c07d460 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.532+0000 7f889b59e700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f887c039900 0x7f887c03be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.532+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] conn(0x7f887c0400d0 0x7f887c042540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.532+0000 7f889b59e700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f887c039900 0x7f887c03be10 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.533+0000 7f889ad9d700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] conn(0x7f887c0400d0 0x7f887c042540 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.544 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.535+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== osd.4 v2:172.21.15.164:6802/22834 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7f887c042bf0 con 0x7f887c0400d0 2020-02-05T16:59:19.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.560+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 --> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f889c068f50 con 0x7f887c0400d0 2020-02-05T16:59:19.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.561+0000 7f8883fff700 1 -- 172.21.15.8:0/2579054689 <== osd.4 v2:172.21.15.164:6802/22834 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f889c068f50 con 0x7f887c0400d0 2020-02-05T16:59:19.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] conn(0x7f887c0400d0 msgr2=0x7f887c042540 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6802/22834,v1:172.21.15.164:6803/22834] conn(0x7f887c0400d0 0x7f887c042540 crc :-1 s=READY pgs=6 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f887c039900 msgr2=0x7f887c03be10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f887c039900 0x7f887c03be10 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 msgr2=0x7f889c13f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7f889c0a9c90 tx=0x7f8894010b20).stop 2020-02-05T16:59:19.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a08e2700 1 -- 172.21.15.8:0/2579054689 reap_dead start 2020-02-05T16:59:19.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 shutdown_connections 2020-02-05T16:59:19.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 --2- 172.21.15.8:0/2579054689 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f889c07d460 0x7f889c13f900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.562+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 >> 172.21.15.8:0/2579054689 conn(0x7f889c0aa160 msgr2=0x7f889c0a2360 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.563+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 shutdown_connections 2020-02-05T16:59:19.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.563+0000 7f88a10e3700 1 -- 172.21.15.8:0/2579054689 wait complete. 2020-02-05T16:59:19.582 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:19.583 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2020-02-05T16:59:19.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.809+0000 7f8b515c5700 1 Processor -- start 2020-02-05T16:59:19.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.809+0000 7f8b515c5700 1 -- start start 2020-02-05T16:59:19.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.809+0000 7f8b515c5700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b4c0ace50 0x7f8b4c0aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b515c5700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4c0aa980 con 0x7f8b4c0abd50 2020-02-05T16:59:19.819 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b515c5700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4c1454a0 con 0x7f8b4c145c20 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b515c5700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4c1456d0 con 0x7f8b4c0ace50 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b4bfff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f8b4c0abd50 0x7f8b4c1491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45938/0 (socket says 172.21.15.8:45938) 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b4bfff700 1 -- 172.21.15.8:0/591824856 learned_addr learned my addr 172.21.15.8:0/591824856 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.810+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c0ace50 0x7f8b4c0aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391275819 0 0) 0x7f8b400010d0 con 0x7f8b4c0abd50 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b34003230 con 0x7f8b4c0abd50 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680760924 0 0) 0x7f8b4c1454a0 con 0x7f8b4c145c20 2020-02-05T16:59:19.820 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b400010d0 con 0x7f8b4c145c20 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1276960623 0 0) 0x7f8b34003230 con 0x7f8b4c0abd50 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c0ace50 msgr2=0x7f8b4c0aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.811+0000 7f8b4b7fe700 1 --2- 172.21.15.8:0/591824856 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c0ace50 0x7f8b4c0aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 >> v1:172.21.15.164:6789/0 conn(0x7f8b4c145c20 legacy=0x7f8b4c13f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c146050 con 0x7f8b4c0abd50 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f8b40001980 con 0x7f8b4c0abd50 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 >> v1:172.21.15.8:6789/0 conn(0x7f8b4c0abd50 legacy=0x7f8b4c1491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.821 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b340045a0 0x7f8b34004a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b34005120 0x7f8b3400b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c1456d0 con 0x7f8b34003a90 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34050040 con 0x7f8b340045a0 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34050280 con 0x7f8b34005120 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.822 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.812+0000 7f8b52827700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b34005120 0x7f8b3400b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b4bfff700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b340045a0 0x7f8b34004a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 -- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b34005120 msgr2=0x7f8b3400b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8b34005120 0x7f8b3400b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 -- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b340045a0 msgr2=0x7f8b34004a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b340045a0 0x7f8b34004a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b52827700 1 -- 172.21.15.8:0/591824856 reap_dead start 2020-02-05T16:59:19.823 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 -- 172.21.15.8:0/591824856 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34050520 con 0x7f8b34003a90 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b4bfff700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8b340045a0 0x7f8b34004a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.813+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).ready entity=mon.0 client_cookie=ee4a1631cac9c704 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c008950 con 0x7f8b34003a90 2020-02-05T16:59:19.824 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c010ab0 con 0x7f8b34003a90 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b515c5700 1 -- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 msgr2=0x7f8b34003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b4b7fe700 1 -- 172.21.15.8:0/591824856 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c0056c0 con 0x7f8b34003a90 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.814+0000 7f8b515c5700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f8b3c002f30 tx=0x7f8b3c010160).stop 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.815+0000 7f8b515c5700 1 -- 172.21.15.8:0/591824856 shutdown_connections 2020-02-05T16:59:19.825 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.815+0000 7f8b515c5700 1 --2- 172.21.15.8:0/591824856 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b34003a90 0x7f8b34003ee0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.826 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.815+0000 7f8b515c5700 1 -- 172.21.15.8:0/591824856 >> 172.21.15.8:0/591824856 conn(0x7f8b4c061510 msgr2=0x7f8b4c0ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.826 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.815+0000 7f8b515c5700 1 -- 172.21.15.8:0/591824856 shutdown_connections 2020-02-05T16:59:19.826 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.815+0000 7f8b515c5700 1 -- 172.21.15.8:0/591824856 wait complete. 2020-02-05T16:59:19.830 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.818+0000 7f8b515c5700 1 Processor -- start 2020-02-05T16:59:19.830 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.818+0000 7f8b515c5700 1 -- start start 2020-02-05T16:59:19.830 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b515c5700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8b4c0a7060 0x7f8b4c146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.830 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b515c5700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8b4c146be0 0x7f8b4c1324a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.830 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b515c5700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4c1440c0 con 0x7f8b4c1329f0 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b515c5700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4c143b80 con 0x7f8b4c0a7060 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b515c5700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4c143e20 con 0x7f8b4c146be0 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b52827700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f8b4c1329f0 0x7f8b4c0751f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45950/0 (socket says 172.21.15.8:45950) 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b52827700 1 -- 172.21.15.8:0/1790151401 learned_addr learned my addr 172.21.15.8:0/1790151401 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> v2:172.21.15.8:3300/0 conn(0x7f8b4c0a7060 0x7f8b4c146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.819+0000 7f8b4bfff700 1 --2- 172.21.15.8:0/1790151401 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c146be0 0x7f8b4c1324a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612655031 0 0) 0x7f8b4c1440c0 con 0x7f8b4c1329f0 2020-02-05T16:59:19.831 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b2c003230 con 0x7f8b4c1329f0 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 -- 172.21.15.8:0/1790151401 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c146be0 msgr2=0x7f8b4c1324a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> v2:172.21.15.164:3300/0 conn(0x7f8b4c146be0 0x7f8b4c1324a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 -- 172.21.15.8:0/1790151401 >> v1:172.21.15.8:6789/0 conn(0x7f8b4c1329f0 legacy=0x7f8b4c0751f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 -- 172.21.15.8:0/1790151401 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c124fd0 con 0x7f8b4c0a7060 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> v2:172.21.15.8:3300/0 conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> v2:172.21.15.8:3300/0 conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.832 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c0058d0 con 0x7f8b4c0a7060 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.820+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.821+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c008b50 con 0x7f8b4c0a7060 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.821+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c00f1f0 con 0x7f8b4c0a7060 2020-02-05T16:59:19.833 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.821+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c148470 con 0x7f8b4c0a7060 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.821+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4c132e20 con 0x7f8b4c0a7060 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.822+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.822+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b4c143970 con 0x7f8b4c0a7060 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.822+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f8b3c00f500 con 0x7f8b4c0a7060 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.822+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b50dc4700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:19.834 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b497fa700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8b2c039f10 0x7f8b2c03c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f8b3c0496a0 con 0x7f8b4c0a7060 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b497fa700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] conn(0x7f8b2c0405c0 0x7f8b2c042a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 --> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b2c0430e0 con 0x7f8b2c0405c0 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.823+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b3c049ac0 con 0x7f8b4c0a7060 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.824+0000 7f8b4bfff700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8b2c039f10 0x7f8b2c03c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.824+0000 7f8b52827700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] conn(0x7f8b2c0405c0 0x7f8b2c042a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:19.835 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.824+0000 7f8b52827700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] conn(0x7f8b2c0405c0 0x7f8b2c042a30 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.836 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.824+0000 7f8b4bfff700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8b2c039f10 0x7f8b2c03c420 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:19.836 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.826+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== osd.5 v2:172.21.15.164:6807/22833 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7f8b2c0430e0 con 0x7f8b2c0405c0 2020-02-05T16:59:19.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.851+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 --> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b4c0a8b20 con 0x7f8b2c0405c0 2020-02-05T16:59:19.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.852+0000 7f8b497fa700 1 -- 172.21.15.8:0/1790151401 <== osd.5 v2:172.21.15.164:6807/22833 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8b4c0a8b20 con 0x7f8b2c0405c0 2020-02-05T16:59:19.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] conn(0x7f8b2c0405c0 msgr2=0x7f8b2c042a30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.856 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6807/22833,v1:172.21.15.164:6809/22833] conn(0x7f8b2c0405c0 0x7f8b2c042a30 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8b2c039f10 msgr2=0x7f8b2c03c420 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8b2c039f10 0x7f8b2c03c420 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 msgr2=0x7f8b4c146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.853+0000 7f8b515c5700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f8b4c143d40 tx=0x7f8b3c0073b0).stop 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b52827700 1 -- 172.21.15.8:0/1790151401 reap_dead start 2020-02-05T16:59:19.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 shutdown_connections 2020-02-05T16:59:19.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b515c5700 1 --2- 172.21.15.8:0/1790151401 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8b4c0a7060 0x7f8b4c146690 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:19.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 >> 172.21.15.8:0/1790151401 conn(0x7f8b4c07fc10 msgr2=0x7f8b4c05bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:19.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 shutdown_connections 2020-02-05T16:59:19.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:19.854+0000 7f8b515c5700 1 -- 172.21.15.8:0/1790151401 wait complete. 2020-02-05T16:59:19.874 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:19.875 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2020-02-05T16:59:20.105 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.095+0000 7f1152842700 1 Processor -- start 2020-02-05T16:59:20.105 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.095+0000 7f1152842700 1 -- start start 2020-02-05T16:59:20.105 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f114c0abd50 0x7f114c1491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f114c145c20 0x7f114c146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c148c70 con 0x7f114c0ace50 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c148f10 con 0x7f114c0abd50 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.096+0000 7f1152842700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0aa980 con 0x7f114c145c20 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.097+0000 7f1152041700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.106 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.097+0000 7f1151840700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f114c0abd50 0x7f114c1491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.097+0000 7f1152041700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44578/0 (socket says 172.21.15.8:44578) 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.097+0000 7f1152041700 1 -- 172.21.15.8:0/645407204 learned_addr learned my addr 172.21.15.8:0/645407204 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.097+0000 7f1153aa4700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f114c145c20 0x7f114c146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 -- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f114c145c20 msgr2=0x7f114c146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f114c145c20 0x7f114c146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 -- 172.21.15.8:0/645407204 >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 msgr2=0x7f114c0aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.107 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 -- 172.21.15.8:0/645407204 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c1454a0 con 0x7f114c0abd50 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1152041700 1 --2- 172.21.15.8:0/645407204 >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> v2:172.21.15.164:3300/0 conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1140002f30 tx=0x7f114000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> v2:172.21.15.164:3300/0 conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1140002f30 tx=0x7f114000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f1140002f30 tx=0x7f114000fe70).ready entity=mon.1 client_cookie=97ad3778fb775b1a server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.108 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.098+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f1140002f30 tx=0x7f114000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f115103f700 1 -- 172.21.15.8:0/645407204 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11400116a0 con 0x7f114c0abd50 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f1140002f30 tx=0x7f114000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f1151840700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7f1140002f30 tx=0x7f114000fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f115103f700 1 -- 172.21.15.8:0/645407204 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1140008c50 con 0x7f114c0abd50 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f115103f700 1 -- 172.21.15.8:0/645407204 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1140011c80 con 0x7f114c0abd50 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f1152842700 1 -- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 msgr2=0x7f114c1491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.109 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.099+0000 7f1152842700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7f1140002f30 tx=0x7f114000fe70).stop 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 -- 172.21.15.8:0/645407204 shutdown_connections 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f114c145c20 0x7f114c146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 --2- 172.21.15.8:0/645407204 >> v2:172.21.15.8:3300/0 conn(0x7f114c0ace50 0x7f114c0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 --2- 172.21.15.8:0/645407204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f114c0abd50 0x7f114c1491b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 -- 172.21.15.8:0/645407204 >> 172.21.15.8:0/645407204 conn(0x7f114c061510 msgr2=0x7f114c0ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 -- 172.21.15.8:0/645407204 shutdown_connections 2020-02-05T16:59:20.110 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.100+0000 7f1152842700 1 -- 172.21.15.8:0/645407204 wait complete. 2020-02-05T16:59:20.111 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.102+0000 7f1152842700 1 Processor -- start 2020-02-05T16:59:20.111 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.103+0000 7f1152842700 1 -- start start 2020-02-05T16:59:20.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.104+0000 7f1152842700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f114c144700 0x7f114c11f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.104+0000 7f1152842700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f114c144e90 0x7f114c0a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.104+0000 7f1152842700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0a97e0 con 0x7f114c11fbd0 2020-02-05T16:59:20.114 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.104+0000 7f1152842700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c0a8d40 con 0x7f114c144700 2020-02-05T16:59:20.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.104+0000 7f1152842700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c0a9a10 con 0x7f114c144e90 2020-02-05T16:59:20.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.105+0000 7f1153aa4700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f114c11fbd0 0x7f114c0a8fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45968/0 (socket says 172.21.15.8:45968) 2020-02-05T16:59:20.115 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.105+0000 7f1153aa4700 1 -- 172.21.15.8:0/3847825996 learned_addr learned my addr 172.21.15.8:0/3847825996 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:20.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.105+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> v2:172.21.15.8:3300/0 conn(0x7f114c144700 0x7f114c11f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.105+0000 7f1151840700 1 --2- 172.21.15.8:0/3847825996 >> v2:172.21.15.164:3300/0 conn(0x7f114c144e90 0x7f114c0a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f1152041700 1 -- 172.21.15.8:0/3847825996 >> v2:172.21.15.164:3300/0 conn(0x7f114c144e90 msgr2=0x7f114c0a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> v2:172.21.15.164:3300/0 conn(0x7f114c144e90 0x7f114c0a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f1152041700 1 -- 172.21.15.8:0/3847825996 >> v1:172.21.15.8:6789/0 conn(0x7f114c11fbd0 legacy=0x7f114c0a8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f1152041700 1 -- 172.21.15.8:0/3847825996 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c0a4160 con 0x7f114c144700 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218273861 0 0) 0x7f114c0a97e0 con 0x7f114c11fbd0 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.106+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> v2:172.21.15.8:3300/0 conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> v2:172.21.15.8:3300/0 conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.117 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1148007530 con 0x7f114c144700 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1148007150 con 0x7f114c144700 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1148012d50 con 0x7f114c144700 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c0a4430 con 0x7f114c144700 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.107+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c137410 con 0x7f114c144700 2020-02-05T16:59:20.118 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.108+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.108+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f114c056640 con 0x7f114c144700 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.109+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f1148010170 con 0x7f114c144700 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.109+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.109+0000 7f1152041700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.109+0000 7f113effd700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1134039950 0x7f113403be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.110+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f11480498e0 con 0x7f114c144700 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.110+0000 7f113effd700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] conn(0x7f1134040120 0x7f1134042590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.119 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.110+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 --> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1134042c40 con 0x7f1134040120 2020-02-05T16:59:20.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.110+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1148049d00 con 0x7f114c144700 2020-02-05T16:59:20.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.110+0000 7f1151840700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1134039950 0x7f113403be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.111+0000 7f1153aa4700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] conn(0x7f1134040120 0x7f1134042590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.111+0000 7f1151840700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1134039950 0x7f113403be60 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.120 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.111+0000 7f1153aa4700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] conn(0x7f1134040120 0x7f1134042590 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.121 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.113+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== osd.6 v2:172.21.15.164:6813/22836 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7f1134042c40 con 0x7f1134040120 2020-02-05T16:59:20.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.139+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 --> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f114c124500 con 0x7f1134040120 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.140+0000 7f113effd700 1 -- 172.21.15.8:0/3847825996 <== osd.6 v2:172.21.15.164:6813/22836 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f114c124500 con 0x7f1134040120 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] conn(0x7f1134040120 msgr2=0x7f1134042590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6813/22836,v1:172.21.15.164:6816/22836] conn(0x7f1134040120 0x7f1134042590 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1134039950 msgr2=0x7f113403be60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f1134039950 0x7f113403be60 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.147 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 msgr2=0x7f114c11f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1152842700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f114c0a8c60 tx=0x7f1148010ea0).stop 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.141+0000 7f1153aa4700 1 -- 172.21.15.8:0/3847825996 reap_dead start 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.142+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 shutdown_connections 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.142+0000 7f1152842700 1 --2- 172.21.15.8:0/3847825996 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f114c144700 0x7f114c11f680 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.142+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 >> 172.21.15.8:0/3847825996 conn(0x7f114c0aaf80 msgr2=0x7f114c123a90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.142+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 shutdown_connections 2020-02-05T16:59:20.148 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.142+0000 7f1152842700 1 -- 172.21.15.8:0/3847825996 wait complete. 2020-02-05T16:59:20.163 INFO:teuthology.orchestra.run.smithi008.stdout:55834574855 2020-02-05T16:59:20.164 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.394+0000 7fe5aabb7700 1 Processor -- start 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.394+0000 7fe5aabb7700 1 -- start start 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.395+0000 7fe5aabb7700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe5a40ace50 0x7fe5a40aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.395+0000 7fe5aabb7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe5a40abd50 0x7fe5a41491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.395+0000 7fe5aabb7700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40aa980 con 0x7fe5a4145c20 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.395+0000 7fe5aabb7700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a4148c70 con 0x7fe5a40ace50 2020-02-05T16:59:20.406 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.395+0000 7fe5aabb7700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a4148f10 con 0x7fe5a40abd50 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.396+0000 7fe5aa3b6700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe5a40ace50 0x7fe5a40aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.396+0000 7fe5aa3b6700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fe5a40ace50 0x7fe5a40aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44594/0 (socket says 172.21.15.8:44594) 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.396+0000 7fe5aa3b6700 1 -- 172.21.15.8:0/845627194 learned_addr learned my addr 172.21.15.8:0/845627194 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.396+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/845627194 >> v2:172.21.15.164:3300/0 conn(0x7fe5a40abd50 0x7fe5a41491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 -- 172.21.15.8:0/845627194 >> v2:172.21.15.164:3300/0 conn(0x7fe5a40abd50 msgr2=0x7fe5a41491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> v2:172.21.15.164:3300/0 conn(0x7fe5a40abd50 0x7fe5a41491b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.407 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 -- 172.21.15.8:0/845627194 >> v1:172.21.15.164:6789/0 conn(0x7fe5a4145c20 legacy=0x7fe5a413f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 -- 172.21.15.8:0/845627194 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a41454a0 con 0x7fe5a40ace50 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> v2:172.21.15.8:3300/0 conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> v2:172.21.15.8:3300/0 conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).ready entity=mon.0 client_cookie=b08ff8ab35e024ba server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.397+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5a93b4700 1 -- 172.21.15.8:0/845627194 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5a0011760 con 0x7fe5a40ace50 2020-02-05T16:59:20.408 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.409 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5a93b4700 1 -- 172.21.15.8:0/845627194 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0008d30 con 0x7fe5a40ace50 2020-02-05T16:59:20.409 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.409 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5aabb7700 1 -- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 msgr2=0x7fe5a40aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.409 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fe5a4148e30 tx=0x7fe5a000ff30).stop 2020-02-05T16:59:20.409 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.398+0000 7fe5a93b4700 1 -- 172.21.15.8:0/845627194 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5a0011d40 con 0x7fe5a40ace50 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 -- 172.21.15.8:0/845627194 shutdown_connections 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/845627194 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fe5a40ace50 0x7fe5a40aa430 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/845627194 >> v2:172.21.15.164:3300/0 conn(0x7fe5a40abd50 0x7fe5a41491b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 -- 172.21.15.8:0/845627194 >> 172.21.15.8:0/845627194 conn(0x7fe5a4061510 msgr2=0x7fe5a40ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 -- 172.21.15.8:0/845627194 shutdown_connections 2020-02-05T16:59:20.410 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.399+0000 7fe5aabb7700 1 -- 172.21.15.8:0/845627194 wait complete. 2020-02-05T16:59:20.411 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.403+0000 7fe5aabb7700 1 Processor -- start 2020-02-05T16:59:20.411 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.403+0000 7fe5aabb7700 1 -- start start 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.405+0000 7fe5aabb7700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.405+0000 7fe5aabb7700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40a3270 con 0x7fe5a40a89d0 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.405+0000 7fe5aabb7700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a40a9460 con 0x7fe5a40a9030 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.405+0000 7fe5aabb7700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a4146690 con 0x7fe5a40625a0 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.406+0000 7fe5aa3b6700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fe5a40a89d0 0x7fe5a4124340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45982/0 (socket says 172.21.15.8:45982) 2020-02-05T16:59:20.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.406+0000 7fe5aa3b6700 1 -- 172.21.15.8:0/2244489861 learned_addr learned my addr 172.21.15.8:0/2244489861 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.406+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.406+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972194936 0 0) 0x7fe5a0007a40 con 0x7fe5a40a89d0 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe58c003230 con 0x7fe5a40a89d0 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1287879671 0 0) 0x7fe5a40a9460 con 0x7fe5a40a9030 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5a0007a40 con 0x7fe5a40a9030 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe5a9bb5700 1 -- 172.21.15.8:0/2244489861 >> v1:172.21.15.164:6789/0 conn(0x7fe5a40a9030 legacy=0x7fe5a412e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe5a9bb5700 1 -- 172.21.15.8:0/2244489861 >> v1:172.21.15.8:6789/0 conn(0x7fe5a40a89d0 legacy=0x7fe5a4124340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4150592521 0 0) 0x7fe58c003230 con 0x7fe5a40a89d0 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.407+0000 7fe5a9bb5700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a40a9690 con 0x7fe5a40625a0 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5a00085f0 con 0x7fe5a40625a0 2020-02-05T16:59:20.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.408+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.409+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.409+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a0011960 con 0x7fe5a40625a0 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.409+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a40a9960 con 0x7fe5a40625a0 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.409+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe594007350 con 0x7fe5a40625a0 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.409+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a40247f0 con 0x7fe5a40625a0 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.410+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.411+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe5a4056640 con 0x7fe5a40625a0 2020-02-05T16:59:20.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.411+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fe594014530 con 0x7fe5a40625a0 2020-02-05T16:59:20.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.411+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.419 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.412+0000 7fe5a9bb5700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.419 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.412+0000 7fe59affd700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe58c03a080 0x7fe58c03c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.419 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.412+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fe59404b4f0 con 0x7fe5a40625a0 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.413+0000 7fe59affd700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] conn(0x7fe58c040730 0x7fe58c042ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.413+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe58c043250 con 0x7fe58c040730 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.413+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe59404b890 con 0x7fe5a40625a0 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.413+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe58c03a080 0x7fe58c03c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.413+0000 7fe5abe19700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] conn(0x7fe58c040730 0x7fe58c042ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.420 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.414+0000 7fe5aa3b6700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe58c03a080 0x7fe58c03c590 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.421 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.414+0000 7fe5abe19700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] conn(0x7fe58c040730 0x7fe58c042ba0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.421 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.417+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== osd.7 v2:172.21.15.164:6822/22835 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23187 (crc 0 0 0) 0x7fe58c043250 con 0x7fe58c040730 2020-02-05T16:59:20.445 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.442+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 --> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe5a4123f90 con 0x7fe58c040730 2020-02-05T16:59:20.446 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.444+0000 7fe59affd700 1 -- 172.21.15.8:0/2244489861 <== osd.7 v2:172.21.15.164:6822/22835 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe5a4123f90 con 0x7fe58c040730 2020-02-05T16:59:20.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.444+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] conn(0x7fe58c040730 msgr2=0x7fe58c042ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.444+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6822/22835,v1:172.21.15.164:6825/22835] conn(0x7fe58c040730 0x7fe58c042ba0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe58c03a080 msgr2=0x7fe58c03c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fe58c03a080 0x7fe58c03c590 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 msgr2=0x7fe5a4063ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 --2- 172.21.15.8:0/2244489861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe5a40625a0 0x7fe5a4063ff0 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7fe5a0008830 tx=0x7fe5a0008450).stop 2020-02-05T16:59:20.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5abe19700 1 -- 172.21.15.8:0/2244489861 reap_dead start 2020-02-05T16:59:20.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 shutdown_connections 2020-02-05T16:59:20.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 >> 172.21.15.8:0/2244489861 conn(0x7fe5a40ac550 msgr2=0x7fe5a405e920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 shutdown_connections 2020-02-05T16:59:20.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.445+0000 7fe5aabb7700 1 -- 172.21.15.8:0/2244489861 wait complete. 2020-02-05T16:59:20.470 INFO:teuthology.orchestra.run.smithi008.stdout:55834574854 2020-02-05T16:59:20.478 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2020-02-05T16:59:20.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.704+0000 7f6ce1cb6700 1 Processor -- start 2020-02-05T16:59:20.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.704+0000 7f6ce1cb6700 1 -- start start 2020-02-05T16:59:20.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6ce1cb6700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cdc08fff0 0x7f6cdc08f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6ce1cb6700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f6cdc0a9e90 0x7f6cdc0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6ce1cb6700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0aab90 con 0x7f6cdc0908f0 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6ce1cb6700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc090d20 con 0x7f6cdc0a9e90 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6ce1cb6700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc090650 con 0x7f6cdc08fff0 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6cdb7fe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f6cdc0908f0 0x7f6cdc0a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:45992/0 (socket says 172.21.15.8:45992) 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.705+0000 7f6cdb7fe700 1 -- 172.21.15.8:0/866737960 learned_addr learned my addr 172.21.15.8:0/866737960 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:20.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.706+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/866737960 >> v2:172.21.15.8:3300/0 conn(0x7f6cdc0a9e90 0x7f6cdc0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.706+0000 7f6cdbfff700 1 --2- 172.21.15.8:0/866737960 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc08fff0 0x7f6cdc08f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.706+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933045670 0 0) 0x7f6cdc0aab90 con 0x7f6cdc0908f0 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.706+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4003230 con 0x7f6cdc0908f0 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.706+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1245804143 0 0) 0x7f6cc4003230 con 0x7f6cdc0908f0 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc08fff0 msgr2=0x7f6cdc08f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 --2- 172.21.15.8:0/866737960 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc08fff0 0x7f6cdc08f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 >> v2:172.21.15.8:3300/0 conn(0x7f6cdc0a9e90 msgr2=0x7f6cdc0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 --2- 172.21.15.8:0/866737960 >> v2:172.21.15.8:3300/0 conn(0x7f6cdc0a9e90 0x7f6cdc0a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0a86e0 con 0x7f6cdc0908f0 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f6cd0001ad0 con 0x7f6cdc0908f0 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/866737960 >> v2:172.21.15.8:3300/0 conn(0x7f6cdc0a9e90 0x7f6cdc0a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 >> v1:172.21.15.8:6789/0 conn(0x7f6cdc0908f0 legacy=0x7f6cdc0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc40038e0 0x7f6cc4003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.707+0000 7f6cdaffd700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdaffd700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc4004f20 0x7f6cc404f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc090d20 con 0x7f6cc40045a0 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc090650 con 0x7f6cc40038e0 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc404fe90 con 0x7f6cc4004f20 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdbfff700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc40038e0 0x7f6cc4003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 -- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc4004f20 msgr2=0x7f6cc404f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cc4004f20 0x7f6cc404f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 -- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc40038e0 msgr2=0x7f6cc4003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cc40038e0 0x7f6cc4003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 -- 172.21.15.8:0/866737960 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc4050190 con 0x7f6cc40045a0 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6ce0a54700 1 -- 172.21.15.8:0/866737960 reap_dead start 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.708+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6cd0001380 tx=0x7f6cd0011030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6cd0001380 tx=0x7f6cd0011030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6cd0001380 tx=0x7f6cd0011030).ready entity=mon.0 client_cookie=27628c42d0433e6b server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6cd0001380 tx=0x7f6cd0011030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6cd0001380 tx=0x7f6cd0011030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6cd0001380 tx=0x7f6cd0011030).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd000e4a0 con 0x7f6cc40045a0 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000e0c0 con 0x7f6cc40045a0 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6cdaffd700 1 -- 172.21.15.8:0/866737960 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd000ea80 con 0x7f6cc40045a0 2020-02-05T16:59:20.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 msgr2=0x7f6cc40049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.709+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6cd0001380 tx=0x7f6cd0011030).stop 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.710+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/866737960 shutdown_connections 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.710+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/866737960 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cc40045a0 0x7f6cc40049d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.710+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/866737960 >> 172.21.15.8:0/866737960 conn(0x7f6cdc0600f0 msgr2=0x7f6cdc05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.710+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/866737960 shutdown_connections 2020-02-05T16:59:20.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.710+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/866737960 wait complete. 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.712+0000 7f6ce1cb6700 1 Processor -- start 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.712+0000 7f6ce1cb6700 1 -- start start 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.713+0000 7f6ce1cb6700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6cdc126240 0x7f6cdc140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.713+0000 7f6ce1cb6700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0a0d20 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.713+0000 7f6ce1cb6700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc07e760 con 0x7f6cdc125e10 2020-02-05T16:59:20.724 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.713+0000 7f6ce1cb6700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0a0540 con 0x7f6cdc126240 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.713+0000 7f6cdbfff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f6cdc09f1a0 0x7f6cdc07df60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46004/0 (socket says 172.21.15.8:46004) 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cdbfff700 1 -- 172.21.15.8:0/2079650235 learned_addr learned my addr 172.21.15.8:0/2079650235 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc126240 0x7f6cdc140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322306985 0 0) 0x7f6cdc0a0d20 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc003230 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3936174516 0 0) 0x7f6cbc003230 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.725 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cdc0a0d20 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f6ccc001490 con 0x7f6cdc09f1a0 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc126240 msgr2=0x7f6cdc140b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 --2- 172.21.15.8:0/2079650235 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc126240 0x7f6cdc140b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.714+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 >> v1:172.21.15.164:6789/0 conn(0x7f6cdc125e10 legacy=0x7f6cdc13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 >> v1:172.21.15.8:6789/0 conn(0x7f6cdc09f1a0 legacy=0x7f6cdc07df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.726 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cbc003b40 0x7f6cbc003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cbc004640 0x7f6cbc004b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0a0540 con 0x7f6cbc005170 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc00bf20 con 0x7f6cbc003b40 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cbc00c160 con 0x7f6cbc004640 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> v2:172.21.15.164:3300/0 conn(0x7f6cdc126240 0x7f6cdc140b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:20.727 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cdbfff700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cbc003b40 0x7f6cbc003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cbc004640 0x7f6cbc004b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.715+0000 7f6ce0a54700 1 -- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cbc004640 msgr2=0x7f6cbc004b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cbc004640 0x7f6cbc004b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 -- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cbc003b40 msgr2=0x7f6cbc003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6cbc003b40 0x7f6cbc003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc126670 con 0x7f6cbc005170 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6cdb7fe700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6cbc004640 0x7f6cbc004b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0a3f40 con 0x7f6cbc005170 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc126e20 con 0x7f6cbc005170 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 -- 172.21.15.8:0/2079650235 reap_dead start 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.729 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd4007520 con 0x7f6cbc005170 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd4007140 con 0x7f6cbc005170 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.716+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd4012c10 con 0x7f6cbc005170 2020-02-05T16:59:20.730 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.717+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.717+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4003360 con 0x7f6cbc005170 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.718+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f6cd40056c0 con 0x7f6cbc005170 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.718+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.718+0000 7f6cd8ff9700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 0x7f6cbc03fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.718+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f6cd40080d0 con 0x7f6cbc005170 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.719+0000 7f6cdbfff700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 0x7f6cbc03fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:20.731 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.719+0000 7f6cdbfff700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 0x7f6cbc03fa70 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:20.732 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.722+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.732 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.722+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f6cd4031590 con 0x7f6cbc005170 2020-02-05T16:59:20.878 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.875+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f6cb40021f0 con 0x7f6cbc005170 2020-02-05T16:59:20.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.876+0000 7f6ce0a54700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:20.879 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.876+0000 7f6cd8ff9700 1 -- 172.21.15.8:0/2079650235 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6cd4049370 con 0x7f6cbc005170 2020-02-05T16:59:20.879 INFO:teuthology.orchestra.run.smithi008.stdout:55834574854 2020-02-05T16:59:20.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 msgr2=0x7f6cbc03fa70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.880 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 0x7f6cbc03fa70 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 msgr2=0x7f6cbc00b880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:20.881 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f6cd00018a0 tx=0x7f6cd4010dc0).stop 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 shutdown_connections 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6cbc03d560 0x7f6cbc03fa70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 --2- 172.21.15.8:0/2079650235 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6cbc005170 0x7f6cbc00b880 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.878+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 >> 172.21.15.8:0/2079650235 conn(0x7f6cdc0aa7f0 msgr2=0x7f6cdc0a2720 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.879+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 shutdown_connections 2020-02-05T16:59:20.882 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:20.879+0000 7f6ce1cb6700 1 -- 172.21.15.8:0/2079650235 wait complete. 2020-02-05T16:59:20.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.0 2020-02-05T16:59:20.900 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2020-02-05T16:59:21.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.118+0000 7f105c7cb700 1 Processor -- start 2020-02-05T16:59:21.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.118+0000 7f105c7cb700 1 -- start start 2020-02-05T16:59:21.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105c7cb700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105c7cb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f10540a9e90 0x7f10540a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105c7cb700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540aab90 con 0x7f10540908f0 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105c7cb700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1054090d20 con 0x7f10540a9e90 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105c7cb700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1054090650 con 0x7f105408fff0 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105a567700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f10540908f0 0x7f10540a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46018/0 (socket says 172.21.15.8:46018) 2020-02-05T16:59:21.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105a567700 1 -- 172.21.15.8:0/1475398542 learned_addr learned my addr 172.21.15.8:0/1475398542 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105b569700 1 --2- 172.21.15.8:0/1475398542 >> v2:172.21.15.8:3300/0 conn(0x7f10540a9e90 0x7f10540a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.119+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750560459 0 0) 0x7f10540aab90 con 0x7f10540908f0 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040003230 con 0x7f10540908f0 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f105ad68700 1 -- 172.21.15.8:0/1475398542 >> v2:172.21.15.8:3300/0 conn(0x7f10540a9e90 msgr2=0x7f10540a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> v2:172.21.15.8:3300/0 conn(0x7f10540a9e90 0x7f10540a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f105ad68700 1 -- 172.21.15.8:0/1475398542 >> v1:172.21.15.8:6789/0 conn(0x7f10540908f0 legacy=0x7f10540a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1051069058 0 0) 0x7f1040003230 con 0x7f10540908f0 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f105ad68700 1 -- 172.21.15.8:0/1475398542 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540a86e0 con 0x7f105408fff0 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.120+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1050008910 tx=0x7f10440019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1050008910 tx=0x7f10440019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7f1050008910 tx=0x7f10440019d0).ready entity=mon.2 client_cookie=9156fc1c8629d65d server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f1050008910 tx=0x7f10440019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.135 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1044001750 con 0x7f105408fff0 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f1050008910 tx=0x7f10440019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105ad68700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=THROTTLE_DONE pgs=32 cs=0 l=1 rx=0x7f1050008910 tx=0x7f10440019d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044001560 con 0x7f105408fff0 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f1059d66700 1 -- 172.21.15.8:0/1475398542 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10500117d0 con 0x7f105408fff0 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105c7cb700 1 -- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 msgr2=0x7f105408f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.121+0000 7f105c7cb700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 secure :-1 s=READY pgs=32 cs=0 l=1 rx=0x7f1050008910 tx=0x7f10440019d0).stop 2020-02-05T16:59:21.136 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 -- 172.21.15.8:0/1475398542 shutdown_connections 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 --2- 172.21.15.8:0/1475398542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f105408fff0 0x7f105408f240 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 --2- 172.21.15.8:0/1475398542 >> v2:172.21.15.8:3300/0 conn(0x7f10540a9e90 0x7f10540a8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 -- 172.21.15.8:0/1475398542 >> 172.21.15.8:0/1475398542 conn(0x7f10540600f0 msgr2=0x7f105405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 -- 172.21.15.8:0/1475398542 shutdown_connections 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.123+0000 7f105c7cb700 1 -- 172.21.15.8:0/1475398542 wait complete. 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.125+0000 7f105c7cb700 1 Processor -- start 2020-02-05T16:59:21.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.126+0000 7f105c7cb700 1 -- start start 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105c7cb700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f10540a75c0 0x7f10540a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105c7cb700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10540a04a0 0x7f1054066520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105c7cb700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105409e8d0 con 0x7f1054066a70 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105c7cb700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10540a2b80 con 0x7f10540a75c0 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105c7cb700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105409ff60 con 0x7f10540a04a0 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.127+0000 7f105ad68700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f10540a75c0 0x7f10540a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f10540a75c0 0x7f10540a3030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60198/0 (socket says 172.21.15.8:60198) 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 -- 172.21.15.8:0/4114504909 learned_addr learned my addr 172.21.15.8:0/4114504909 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105a567700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10540a04a0 0x7f1054066520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 -- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10540a04a0 msgr2=0x7f1054066520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10540a04a0 0x7f1054066520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 -- 172.21.15.8:0/4114504909 >> v1:172.21.15.164:6789/0 conn(0x7f1054066a70 legacy=0x7f105409e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.128+0000 7f105ad68700 1 -- 172.21.15.8:0/4114504909 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540a3580 con 0x7f10540a75c0 2020-02-05T16:59:21.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> v2:172.21.15.164:3300/0 conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> v2:172.21.15.164:3300/0 conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f105000f760 con 0x7f10540a75c0 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105413f4a0 con 0x7f10540a75c0 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10500113f0 con 0x7f10540a75c0 2020-02-05T16:59:21.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.129+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10500056c0 con 0x7f10540a75c0 2020-02-05T16:59:21.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.130+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105413f6b0 con 0x7f10540a75c0 2020-02-05T16:59:21.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.131+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.131+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f10500059a0 con 0x7f10540a75c0 2020-02-05T16:59:21.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.131+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.132+0000 7f104b7fe700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 0x7f103c03bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.132+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f1050049c70 con 0x7f10540a75c0 2020-02-05T16:59:21.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.132+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1054056640 con 0x7f10540a75c0 2020-02-05T16:59:21.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.132+0000 7f105a567700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 0x7f103c03bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.133+0000 7f105a567700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 0x7f103c03bdf0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.140+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.144 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.140+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f10500317a0 con 0x7f10540a75c0 2020-02-05T16:59:21.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.301+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f105409fd40 con 0x7f10540a75c0 2020-02-05T16:59:21.304 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.302+0000 7f105ad68700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.305 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.302+0000 7f104b7fe700 1 -- 172.21.15.8:0/4114504909 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1050031a00 con 0x7f10540a75c0 2020-02-05T16:59:21.305 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.304+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 msgr2=0x7f103c03bdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.307 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.304+0000 7f105c7cb700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 0x7f103c03bdf0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.304+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 msgr2=0x7f10540a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.304+0000 7f105c7cb700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7f10540a0120 tx=0x7f1050011e10).stop 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 shutdown_connections 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f103c0398e0 0x7f103c03bdf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10540a75c0 0x7f10540a3030 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 --2- 172.21.15.8:0/4114504909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10540a04a0 0x7f1054066520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.308 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 >> 172.21.15.8:0/4114504909 conn(0x7f10540a9df0 msgr2=0x7f1054129990 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:21.309 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 shutdown_connections 2020-02-05T16:59:21.309 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.305+0000 7f105c7cb700 1 -- 172.21.15.8:0/4114504909 wait complete. 2020-02-05T16:59:21.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2020-02-05T16:59:21.324 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2020-02-05T16:59:21.555 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.544+0000 7f2cda90e700 1 Processor -- start 2020-02-05T16:59:21.555 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.544+0000 7f2cda90e700 1 -- start start 2020-02-05T16:59:21.555 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.545+0000 7f2cda90e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2cd40908f0 0x7f2cd40a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.555 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.545+0000 7f2cda90e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40aa3e0 con 0x7f2cd408fff0 2020-02-05T16:59:21.555 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.545+0000 7f2cda90e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40aab90 con 0x7f2cd408f240 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.545+0000 7f2cda90e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd408f670 con 0x7f2cd40908f0 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.546+0000 7f2cd8eab700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f2cd408fff0 0x7f2cd40a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46034/0 (socket says 172.21.15.8:46034) 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.546+0000 7f2cd8eab700 1 -- 172.21.15.8:0/3445535546 learned_addr learned my addr 172.21.15.8:0/3445535546 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.546+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40908f0 0x7f2cd40a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.547+0000 7f2cd37fe700 1 -- 172.21.15.8:0/3445535546 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743328025 0 0) 0x7f2cc4001450 con 0x7f2cd408fff0 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.547+0000 7f2cd96ac700 1 -- 172.21.15.8:0/3445535546 >> v1:172.21.15.164:6789/0 conn(0x7f2cd408f240 legacy=0x7f2cd40a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.556 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.547+0000 7f2cd96ac700 1 -- 172.21.15.8:0/3445535546 >> v1:172.21.15.8:6789/0 conn(0x7f2cd408fff0 legacy=0x7f2cd40a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.547+0000 7f2cd96ac700 1 -- 172.21.15.8:0/3445535546 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd405b340 con 0x7f2cd40908f0 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.547+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).ready entity=mon.1 client_cookie=149b78788396d31f server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.557 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd37fe700 1 -- 172.21.15.8:0/3445535546 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ccc008d30 con 0x7f2cd40908f0 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd96ac700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd37fe700 1 -- 172.21.15.8:0/3445535546 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ccc008950 con 0x7f2cd40908f0 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.548+0000 7f2cd37fe700 1 -- 172.21.15.8:0/3445535546 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ccc011c40 con 0x7f2cd40908f0 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.549+0000 7f2cda90e700 1 -- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 msgr2=0x7f2cd40a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.549+0000 7f2cda90e700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f2ccc008910 tx=0x7f2ccc00fdf0).stop 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.550+0000 7f2cda90e700 1 -- 172.21.15.8:0/3445535546 shutdown_connections 2020-02-05T16:59:21.558 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.550+0000 7f2cda90e700 1 --2- 172.21.15.8:0/3445535546 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40908f0 0x7f2cd40a9e90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.550+0000 7f2cda90e700 1 -- 172.21.15.8:0/3445535546 >> 172.21.15.8:0/3445535546 conn(0x7f2cd40600f0 msgr2=0x7f2cd405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:21.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.550+0000 7f2cda90e700 1 -- 172.21.15.8:0/3445535546 shutdown_connections 2020-02-05T16:59:21.559 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.551+0000 7f2cda90e700 1 -- 172.21.15.8:0/3445535546 wait complete. 2020-02-05T16:59:21.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.554+0000 7f2cda90e700 1 Processor -- start 2020-02-05T16:59:21.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.554+0000 7f2cda90e700 1 -- start start 2020-02-05T16:59:21.561 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.556+0000 7f2cda90e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2cd40aba40 0x7f2cd4066960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.556+0000 7f2cda90e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd40aae50 con 0x7f2cd409ddf0 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.556+0000 7f2cda90e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cd409e220 con 0x7f2cd40a7660 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.556+0000 7f2cda90e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cd407f620 con 0x7f2cd40aba40 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.557+0000 7f2cd3fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f2cd409ddf0 0x7f2cd40a6e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46038/0 (socket says 172.21.15.8:46038) 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.557+0000 7f2cd3fff700 1 -- 172.21.15.8:0/601477044 learned_addr learned my addr 172.21.15.8:0/601477044 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.557+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40aba40 0x7f2cd4066960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.562 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.557+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 244470921 0 0) 0x7f2cd40aae50 con 0x7f2cd409ddf0 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cb4003230 con 0x7f2cd409ddf0 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 -- 172.21.15.8:0/601477044 >> v1:172.21.15.164:6789/0 conn(0x7f2cd40a7660 legacy=0x7f2cd413fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 -- 172.21.15.8:0/601477044 >> v1:172.21.15.8:6789/0 conn(0x7f2cd409ddf0 legacy=0x7f2cd40a6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 -- 172.21.15.8:0/601477044 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd409e450 con 0x7f2cd40aba40 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> v2:172.21.15.164:3300/0 conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.563 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.558+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cc4005dc0 con 0x7f2cd40aba40 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cc40059e0 con 0x7f2cd40aba40 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd4141670 con 0x7f2cd40aba40 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cc4012ae0 con 0x7f2cd40aba40 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.559+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd4141b90 con 0x7f2cd40aba40 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.561+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.561+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd40664d0 con 0x7f2cd40aba40 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.561+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f2cc4011310 con 0x7f2cd40aba40 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.561+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.562+0000 7f2cd17fa700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 0x7f2cb403c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.562+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f2cd40aa3e0 con 0x7f2cd40aba40 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.563+0000 7f2cd3fff700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 0x7f2cb403c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.563+0000 7f2cd3fff700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 0x7f2cb403c360 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.585 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.572+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.572+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f2cc4012da0 con 0x7f2cd40aba40 2020-02-05T16:59:21.735 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.733+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2cd4056640 con 0x7f2cd40aba40 2020-02-05T16:59:21.736 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.734+0000 7f2cd8eab700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.736 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.734+0000 7f2cd17fa700 1 -- 172.21.15.8:0/601477044 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2cc40319e0 con 0x7f2cd40aba40 2020-02-05T16:59:21.736 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:21.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 msgr2=0x7f2cb403c360 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 0x7f2cb403c360 crc :-1 s=READY pgs=69 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 msgr2=0x7f2cd4066960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f2cc4000f00 tx=0x7f2cc4010cc0).stop 2020-02-05T16:59:21.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 shutdown_connections 2020-02-05T16:59:21.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f2cb4039e50 0x7f2cb403c360 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.740 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 --2- 172.21.15.8:0/601477044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cd40aba40 0x7f2cd4066960 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 >> 172.21.15.8:0/601477044 conn(0x7f2cd408ffa0 msgr2=0x7f2cd407ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:21.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 shutdown_connections 2020-02-05T16:59:21.741 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.737+0000 7f2cda90e700 1 -- 172.21.15.8:0/601477044 wait complete. 2020-02-05T16:59:21.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2020-02-05T16:59:21.762 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2020-02-05T16:59:21.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.979+0000 7fee16260700 1 Processor -- start 2020-02-05T16:59:21.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.979+0000 7fee16260700 1 -- start start 2020-02-05T16:59:21.989 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 0x7fee100a7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee10090d20 con 0x7fee100a7990 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee10090650 con 0x7fee100908f0 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee16260700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1008ece0 con 0x7fee1008fff0 2020-02-05T16:59:21.990 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee14ffe700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 0x7fee100a7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee0f7fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.980+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee14ffe700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 0x7fee100a7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44668/0 (socket says 172.21.15.8:44668) 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44716/0 (socket says 172.21.15.8:44716) 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 -- 172.21.15.8:0/2202780043 learned_addr learned my addr 172.21.15.8:0/2202780043 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0f7fe700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60220/0 (socket says 172.21.15.8:60220) 2020-02-05T16:59:21.991 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 -- 172.21.15.8:0/2202780043 >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 msgr2=0x7fee100a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 -- 172.21.15.8:0/2202780043 >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 msgr2=0x7fee100a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.981+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 0x7fee100a7dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.982+0000 7fee0ffff700 1 -- 172.21.15.8:0/2202780043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100aa3e0 con 0x7fee1008fff0 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.982+0000 7fee0f7fe700 1 --2- 172.21.15.8:0/2202780043 >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.982+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee00002f30 tx=0x7fee0000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.992 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.982+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee00002f30 tx=0x7fee0000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.982+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7fee00002f30 tx=0x7fee0000ff30).ready entity=mon.2 client_cookie=c004b327e90c555e server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fee00002f30 tx=0x7fee0000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fee00002f30 tx=0x7fee0000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0ffff700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7fee00002f30 tx=0x7fee0000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0effd700 1 -- 172.21.15.8:0/2202780043 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee00011950 con 0x7fee1008fff0 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0effd700 1 -- 172.21.15.8:0/2202780043 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00011570 con 0x7fee1008fff0 2020-02-05T16:59:21.993 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee0effd700 1 -- 172.21.15.8:0/2202780043 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000a160 con 0x7fee1008fff0 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 -- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 msgr2=0x7fee1008f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7fee00002f30 tx=0x7fee0000ff30).stop 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 -- 172.21.15.8:0/2202780043 shutdown_connections 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 --2- 172.21.15.8:0/2202780043 >> v2:172.21.15.8:3300/0 conn(0x7fee100a7990 0x7fee100a7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 --2- 172.21.15.8:0/2202780043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1008fff0 0x7fee1008f240 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 --2- 172.21.15.8:0/2202780043 >> v2:172.21.15.164:3300/0 conn(0x7fee100908f0 0x7fee100a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.983+0000 7fee16260700 1 -- 172.21.15.8:0/2202780043 >> 172.21.15.8:0/2202780043 conn(0x7fee100600f0 msgr2=0x7fee1005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:21.994 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.984+0000 7fee16260700 1 -- 172.21.15.8:0/2202780043 shutdown_connections 2020-02-05T16:59:21.995 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.984+0000 7fee16260700 1 -- 172.21.15.8:0/2202780043 wait complete. 2020-02-05T16:59:21.997 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.987+0000 7fee16260700 1 Processor -- start 2020-02-05T16:59:21.998 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.988+0000 7fee16260700 1 -- start start 2020-02-05T16:59:21.998 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee16260700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee100a14e0 0x7fee1009fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.999 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee16260700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1009db60 0x7fee1009df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:21.999 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee16260700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee100a1ae0 con 0x7fee10065350 2020-02-05T16:59:21.999 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee16260700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1009e4e0 con 0x7fee100a14e0 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee16260700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee100a06c0 con 0x7fee1009db60 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee14ffe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fee10065350 0x7fee10065780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46056/0 (socket says 172.21.15.8:46056) 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee14ffe700 1 -- 172.21.15.8:0/3445838309 learned_addr learned my addr 172.21.15.8:0/3445838309 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.990+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> v2:172.21.15.164:3300/0 conn(0x7fee100a14e0 0x7fee1009fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.991+0000 7fee0f7fe700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1009db60 0x7fee1009df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.991+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358633101 0 0) 0x7fee100a1ae0 con 0x7fee10065350 2020-02-05T16:59:22.000 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.991+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4003230 con 0x7fee10065350 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 -- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1009db60 msgr2=0x7fee1009df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1009db60 0x7fee1009df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 -- 172.21.15.8:0/3445838309 >> v1:172.21.15.8:6789/0 conn(0x7fee10065350 legacy=0x7fee10065780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 -- 172.21.15.8:0/3445838309 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee100a0300 con 0x7fee100a14e0 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> v2:172.21.15.164:3300/0 conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> v2:172.21.15.164:3300/0 conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.001 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.992+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee00007340 con 0x7fee100a14e0 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee00011b50 con 0x7fee100a14e0 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee1013fa00 con 0x7fee100a14e0 2020-02-05T16:59:22.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0000f740 con 0x7fee100a14e0 2020-02-05T16:59:22.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.993+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee1013fc10 con 0x7fee100a14e0 2020-02-05T16:59:22.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.994+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee10056640 con 0x7fee100a14e0 2020-02-05T16:59:22.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.994+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.995+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fee000056c0 con 0x7fee100a14e0 2020-02-05T16:59:22.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.995+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.996+0000 7fee0cff9700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 0x7fedf403c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.996+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fee00049910 con 0x7fee100a14e0 2020-02-05T16:59:22.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.997+0000 7fee0f7fe700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 0x7fedf403c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:21.997+0000 7fee0f7fe700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 0x7fedf403c430 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.005+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.006+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fee00031460 con 0x7fee100a14e0 2020-02-05T16:59:22.165 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.163+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fee10068f50 con 0x7fee100a14e0 2020-02-05T16:59:22.166 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.164+0000 7fee0ffff700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.166 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.164+0000 7fee0cff9700 1 -- 172.21.15.8:0/3445838309 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fee000316c0 con 0x7fee100a14e0 2020-02-05T16:59:22.167 INFO:teuthology.orchestra.run.smithi008.stdout:55834574854 2020-02-05T16:59:22.169 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 msgr2=0x7fedf403c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 0x7fedf403c430 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 msgr2=0x7fee1009fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fee100a0880 tx=0x7fee00011a50).stop 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 shutdown_connections 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fedf4039f20 0x7fedf403c430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.170 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee100a14e0 0x7fee1009fdb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.171 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 --2- 172.21.15.8:0/3445838309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee1009db60 0x7fee1009df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.171 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 >> 172.21.15.8:0/3445838309 conn(0x7fee1008fe50 msgr2=0x7fee1007dcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:22.171 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 shutdown_connections 2020-02-05T16:59:22.171 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.166+0000 7fee16260700 1 -- 172.21.15.8:0/3445838309 wait complete. 2020-02-05T16:59:22.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2020-02-05T16:59:22.187 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2020-02-05T16:59:22.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.405+0000 7f6e26974700 1 Processor -- start 2020-02-05T16:59:22.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.405+0000 7f6e26974700 1 -- start start 2020-02-05T16:59:22.414 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e26974700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6e2008f240 0x7f6e200908f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e26974700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e200aab90 con 0x7f6e2008fff0 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e26974700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e2005b340 con 0x7f6e200a9e90 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e26974700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2008f670 con 0x7f6e2008f240 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e24f11700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f6e2008fff0 0x7f6e200a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46060/0 (socket says 172.21.15.8:46060) 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e24f11700 1 -- 172.21.15.8:0/2483882682 learned_addr learned my addr 172.21.15.8:0/2483882682 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.406+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> v2:172.21.15.164:3300/0 conn(0x7f6e2008f240 0x7f6e200908f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.415 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1f7fe700 1 -- 172.21.15.8:0/2483882682 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785345580 0 0) 0x7f6e200aab90 con 0x7f6e2008fff0 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1f7fe700 1 -- 172.21.15.8:0/2483882682 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003230 con 0x7f6e2008fff0 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1ffff700 1 -- 172.21.15.8:0/2483882682 >> v1:172.21.15.164:6789/0 conn(0x7f6e200a9e90 legacy=0x7f6e200a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1ffff700 1 -- 172.21.15.8:0/2483882682 >> v1:172.21.15.8:6789/0 conn(0x7f6e2008fff0 legacy=0x7f6e200a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1ffff700 1 -- 172.21.15.8:0/2483882682 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e200aa2c0 con 0x7f6e2008f240 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.407+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> v2:172.21.15.164:3300/0 conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6e14008910 tx=0x7f6e1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> v2:172.21.15.164:3300/0 conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6e14008910 tx=0x7f6e1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.416 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f6e14008910 tx=0x7f6e1400fe70).ready entity=mon.1 client_cookie=e83822f85a22c405 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f6e14008910 tx=0x7f6e1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1f7fe700 1 -- 172.21.15.8:0/2483882682 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e140116e0 con 0x7f6e2008f240 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f6e14008910 tx=0x7f6e1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f6e14008910 tx=0x7f6e1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1f7fe700 1 -- 172.21.15.8:0/2483882682 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14008c50 con 0x7f6e2008f240 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e1f7fe700 1 -- 172.21.15.8:0/2483882682 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e14011cc0 con 0x7f6e2008f240 2020-02-05T16:59:22.417 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e26974700 1 -- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 msgr2=0x7f6e200908f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.408+0000 7f6e26974700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f6e14008910 tx=0x7f6e1400fe70).stop 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.410+0000 7f6e26974700 1 -- 172.21.15.8:0/2483882682 shutdown_connections 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.410+0000 7f6e26974700 1 --2- 172.21.15.8:0/2483882682 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2008f240 0x7f6e200908f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.410+0000 7f6e26974700 1 -- 172.21.15.8:0/2483882682 >> 172.21.15.8:0/2483882682 conn(0x7f6e200600f0 msgr2=0x7f6e2005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.410+0000 7f6e26974700 1 -- 172.21.15.8:0/2483882682 shutdown_connections 2020-02-05T16:59:22.418 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.410+0000 7f6e26974700 1 -- 172.21.15.8:0/2483882682 wait complete. 2020-02-05T16:59:22.424 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.412+0000 7f6e26974700 1 Processor -- start 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.412+0000 7f6e26974700 1 -- start start 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.413+0000 7f6e26974700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f6e20122530 0x7f6e2007c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.413+0000 7f6e26974700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6e2013d8b0 0x7f6e2013fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.413+0000 7f6e26974700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e20122fa0 con 0x7f6e200a00d0 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.413+0000 7f6e26974700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e20122a60 con 0x7f6e20122530 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.413+0000 7f6e26974700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e20122d00 con 0x7f6e2013d8b0 2020-02-05T16:59:22.425 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.414+0000 7f6e24f11700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f6e20122530 0x7f6e2007c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.414+0000 7f6e1ffff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f6e200a00d0 0x7f6e200a0500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46070/0 (socket says 172.21.15.8:46070) 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.414+0000 7f6e1ffff700 1 -- 172.21.15.8:0/1393284705 learned_addr learned my addr 172.21.15.8:0/1393284705 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.414+0000 7f6e25712700 1 --2- 172.21.15.8:0/1393284705 >> v2:172.21.15.164:3300/0 conn(0x7f6e2013d8b0 0x7f6e2013fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 -- 172.21.15.8:0/1393284705 >> v2:172.21.15.164:3300/0 conn(0x7f6e2013d8b0 msgr2=0x7f6e2013fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> v2:172.21.15.164:3300/0 conn(0x7f6e2013d8b0 0x7f6e2013fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253233890 0 0) 0x7f6e20122fa0 con 0x7f6e200a00d0 2020-02-05T16:59:22.426 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 -- 172.21.15.8:0/1393284705 >> v1:172.21.15.8:6789/0 conn(0x7f6e200a00d0 legacy=0x7f6e200a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 -- 172.21.15.8:0/1393284705 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2007cbc0 con 0x7f6e20122530 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> v2:172.21.15.8:3300/0 conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> v2:172.21.15.8:3300/0 conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e10005cc0 con 0x7f6e20122530 2020-02-05T16:59:22.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.415+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.416+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.416+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e100058e0 con 0x7f6e20122530 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.416+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e200a0fd0 con 0x7f6e20122530 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.416+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e100064a0 con 0x7f6e20122530 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.416+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e200a4380 con 0x7f6e20122530 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.417+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.418+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e20056640 con 0x7f6e20122530 2020-02-05T16:59:22.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.418+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f6e100067b0 con 0x7f6e20122530 2020-02-05T16:59:22.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.418+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.419+0000 7f6e1d7fa700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 0x7f6e0003bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.419+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f6e10049610 con 0x7f6e20122530 2020-02-05T16:59:22.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.419+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 0x7f6e0003bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.430 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.420+0000 7f6e1ffff700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 0x7f6e0003bd40 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.431 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.423+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.431 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.424+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f6e10031040 con 0x7f6e20122530 2020-02-05T16:59:22.586 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.584+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6e20068f50 con 0x7f6e20122530 2020-02-05T16:59:22.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.584+0000 7f6e24f11700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.588 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.585+0000 7f6e1d7fa700 1 -- 172.21.15.8:0/1393284705 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6e100312a0 con 0x7f6e20122530 2020-02-05T16:59:22.588 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:22.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.586+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 msgr2=0x7f6e0003bd40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.586+0000 7f6e26974700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 0x7f6e0003bd40 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.586+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 msgr2=0x7f6e2007c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.589 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.586+0000 7f6e26974700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f6e200a5c80 tx=0x7f6e10010970).stop 2020-02-05T16:59:22.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 shutdown_connections 2020-02-05T16:59:22.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f6e00039830 0x7f6e0003bd40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 --2- 172.21.15.8:0/1393284705 >> v2:172.21.15.164:3300/0 conn(0x7f6e2013d8b0 0x7f6e2013fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 --2- 172.21.15.8:0/1393284705 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f6e20122530 0x7f6e2007c670 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.590 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 >> 172.21.15.8:0/1393284705 conn(0x7f6e20066be0 msgr2=0x7f6e2007dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:22.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 shutdown_connections 2020-02-05T16:59:22.591 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.587+0000 7f6e26974700 1 -- 172.21.15.8:0/1393284705 wait complete. 2020-02-05T16:59:22.609 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2020-02-05T16:59:22.609 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.829+0000 7f89281a1700 1 Processor -- start 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.829+0000 7f89281a1700 1 -- start start 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f89281a1700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f892008fff0 0x7f892008f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f89281a1700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f89281a1700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200aa3e0 con 0x7f89200a7990 2020-02-05T16:59:22.839 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f89281a1700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8920090d20 con 0x7f892008fff0 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f89281a1700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8920090650 con 0x7f89200908f0 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f8925f3d700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f892673e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f892008fff0 0x7f892008f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f8925f3d700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44744/0 (socket says 172.21.15.8:44744) 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.830+0000 7f8925f3d700 1 -- 172.21.15.8:0/7511644 learned_addr learned my addr 172.21.15.8:0/7511644 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.831+0000 7f8925f3d700 1 -- 172.21.15.8:0/7511644 >> v2:172.21.15.164:3300/0 conn(0x7f892008fff0 msgr2=0x7f892008f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.831+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> v2:172.21.15.164:3300/0 conn(0x7f892008fff0 0x7f892008f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:22.840 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.831+0000 7f8925f3d700 1 -- 172.21.15.8:0/7511644 >> v1:172.21.15.164:6789/0 conn(0x7f89200a7990 legacy=0x7f89200a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.831+0000 7f8925f3d700 1 -- 172.21.15.8:0/7511644 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200aab90 con 0x7f89200908f0 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.831+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8910008910 tx=0x7f891000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8910008910 tx=0x7f891000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f8910008910 tx=0x7f891000ff30).ready entity=mon.2 client_cookie=aa0e9e3323308f5e server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f8910008910 tx=0x7f891000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.841 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f8910008910 tx=0x7f891000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.842 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f8925f3d700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f8910008910 tx=0x7f891000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.842 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f892573c700 1 -- 172.21.15.8:0/7511644 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89100117a0 con 0x7f89200908f0 2020-02-05T16:59:22.842 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f892573c700 1 -- 172.21.15.8:0/7511644 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8910008d30 con 0x7f89200908f0 2020-02-05T16:59:22.842 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f892573c700 1 -- 172.21.15.8:0/7511644 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8910011d80 con 0x7f89200908f0 2020-02-05T16:59:22.842 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f89281a1700 1 -- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 msgr2=0x7f89200a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.843 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.832+0000 7f89281a1700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f8910008910 tx=0x7f891000ff30).stop 2020-02-05T16:59:22.843 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.833+0000 7f89281a1700 1 -- 172.21.15.8:0/7511644 shutdown_connections 2020-02-05T16:59:22.843 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.834+0000 7f89281a1700 1 --2- 172.21.15.8:0/7511644 >> v2:172.21.15.164:3300/0 conn(0x7f892008fff0 0x7f892008f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:22.843 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.834+0000 7f89281a1700 1 --2- 172.21.15.8:0/7511644 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f89200908f0 0x7f89200a9e90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:22.843 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.834+0000 7f89281a1700 1 -- 172.21.15.8:0/7511644 >> 172.21.15.8:0/7511644 conn(0x7f89200600f0 msgr2=0x7f892005a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:22.844 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.834+0000 7f89281a1700 1 -- 172.21.15.8:0/7511644 shutdown_connections 2020-02-05T16:59:22.844 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.834+0000 7f89281a1700 1 -- 172.21.15.8:0/7511644 wait complete. 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.837+0000 7f89281a1700 1 Processor -- start 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.838+0000 7f89281a1700 1 -- start start 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f89281a1700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f89281a1700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8920065620 con 0x7f892007bae0 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f89281a1700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89200a6ba0 con 0x7f8920066350 2020-02-05T16:59:22.848 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f89281a1700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89200a2070 con 0x7f892007da90 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f8925f3d700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f892007bae0 0x7f8920065b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46080/0 (socket says 172.21.15.8:46080) 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f8925f3d700 1 -- 172.21.15.8:0/2769258460 learned_addr learned my addr 172.21.15.8:0/2769258460 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.840+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.841+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793716256 0 0) 0x7f8910008d30 con 0x7f892007bae0 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.841+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8908003230 con 0x7f892007bae0 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.841+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251545220 0 0) 0x7f89200a6ba0 con 0x7f8920066350 2020-02-05T16:59:22.849 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.841+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910008d30 con 0x7f8920066350 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.841+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1643461992 0 0) 0x7f8908003230 con 0x7f892007bae0 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89200a6ba0 con 0x7f892007bae0 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f89100085f0 con 0x7f892007bae0 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 -- 172.21.15.8:0/2769258460 >> v1:172.21.15.164:6789/0 conn(0x7f8920066350 legacy=0x7f892013f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 -- 172.21.15.8:0/2769258460 >> v1:172.21.15.8:6789/0 conn(0x7f892007bae0 legacy=0x7f8920065b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 -- 172.21.15.8:0/2769258460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892007bf10 con 0x7f892007da90 2020-02-05T16:59:22.850 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.842+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f891c0056c0 con 0x7f892007da90 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.851 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c007150 con 0x7f892007da90 2020-02-05T16:59:22.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892007c180 con 0x7f892007da90 2020-02-05T16:59:22.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f891c005ca0 con 0x7f892007da90 2020-02-05T16:59:22.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.843+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8920141000 con 0x7f892007da90 2020-02-05T16:59:22.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.844+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.852 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.845+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f891c0125b0 con 0x7f892007da90 2020-02-05T16:59:22.853 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.845+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.853 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.845+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89200a7680 con 0x7f892007da90 2020-02-05T16:59:22.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.846+0000 7f8916ffd700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 0x7f890803c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:22.854 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.846+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f891c049960 con 0x7f892007da90 2020-02-05T16:59:22.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.846+0000 7f8925f3d700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 0x7f890803c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:22.855 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.847+0000 7f8925f3d700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 0x7f890803c930 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:22.857 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.855+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:22.858 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:22.855+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f891c00d0c0 con 0x7f892007da90 2020-02-05T16:59:23.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.014+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8920056640 con 0x7f892007da90 2020-02-05T16:59:23.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.015+0000 7f892673e700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.015+0000 7f8916ffd700 1 -- 172.21.15.8:0/2769258460 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f891c007530 con 0x7f892007da90 2020-02-05T16:59:23.018 INFO:teuthology.orchestra.run.smithi008.stdout:55834574853 2020-02-05T16:59:23.021 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 msgr2=0x7f890803c930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.022 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 0x7f890803c930 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.022 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 msgr2=0x7f89200650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.022 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f8920066cd0 tx=0x7f8910008450).stop 2020-02-05T16:59:23.022 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 shutdown_connections 2020-02-05T16:59:23.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f890803a420 0x7f890803c930 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 --2- 172.21.15.8:0/2769258460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f892007da90 0x7f89200650d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 >> 172.21.15.8:0/2769258460 conn(0x7f892005b0c0 msgr2=0x7f892009cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:23.023 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 shutdown_connections 2020-02-05T16:59:23.024 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.019+0000 7f89281a1700 1 -- 172.21.15.8:0/2769258460 wait complete. 2020-02-05T16:59:23.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.5 2020-02-05T16:59:23.042 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2020-02-05T16:59:23.263 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.253+0000 7f90b3fff700 1 Processor -- start 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.253+0000 7f90b3fff700 1 -- start start 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.254+0000 7f90b3fff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90b40ace50 0x7f90b40aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.254+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.254+0000 7f90b3fff700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b40aa980 con 0x7f90b4145c20 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.254+0000 7f90b3fff700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90b4148c70 con 0x7f90b40ace50 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.254+0000 7f90b3fff700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b4148f10 con 0x7f90b40abd50 2020-02-05T16:59:23.264 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.255+0000 7f90b2ffd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.255+0000 7f90b2ffd700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44758/0 (socket says 172.21.15.8:44758) 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.255+0000 7f90b2ffd700 1 -- 172.21.15.8:0/2180960600 learned_addr learned my addr 172.21.15.8:0/2180960600 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.255+0000 7f90b37fe700 1 --2- 172.21.15.8:0/2180960600 >> v2:172.21.15.164:3300/0 conn(0x7f90b40ace50 0x7f90b40aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 -- 172.21.15.8:0/2180960600 >> v2:172.21.15.164:3300/0 conn(0x7f90b40ace50 msgr2=0x7f90b40aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> v2:172.21.15.164:3300/0 conn(0x7f90b40ace50 0x7f90b40aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:23.265 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 -- 172.21.15.8:0/2180960600 >> v1:172.21.15.164:6789/0 conn(0x7f90b4145c20 legacy=0x7f90b413f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.266 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 -- 172.21.15.8:0/2180960600 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b41454a0 con 0x7f90b40abd50 2020-02-05T16:59:23.266 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f90a8008910 tx=0x7f90a800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.266 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f90a8008910 tx=0x7f90a800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.266 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.256+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f90a8008910 tx=0x7f90a800ff30).ready entity=mon.2 client_cookie=7d17cf575800923 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.266 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f90a8008910 tx=0x7f90a800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.267 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f90a8008910 tx=0x7f90a800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.267 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b2ffd700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7f90a8008910 tx=0x7f90a800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.267 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b27fc700 1 -- 172.21.15.8:0/2180960600 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a8011760 con 0x7f90b40abd50 2020-02-05T16:59:23.267 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b27fc700 1 -- 172.21.15.8:0/2180960600 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a8008d30 con 0x7f90b40abd50 2020-02-05T16:59:23.267 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b27fc700 1 -- 172.21.15.8:0/2180960600 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a8011d40 con 0x7f90b40abd50 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b3fff700 1 -- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 msgr2=0x7f90b41491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.257+0000 7f90b3fff700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7f90a8008910 tx=0x7f90a800ff30).stop 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 -- 172.21.15.8:0/2180960600 shutdown_connections 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 --2- 172.21.15.8:0/2180960600 >> v2:172.21.15.164:3300/0 conn(0x7f90b40ace50 0x7f90b40aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 --2- 172.21.15.8:0/2180960600 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40abd50 0x7f90b41491b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 -- 172.21.15.8:0/2180960600 >> 172.21.15.8:0/2180960600 conn(0x7f90b4061510 msgr2=0x7f90b40ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:23.268 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 -- 172.21.15.8:0/2180960600 shutdown_connections 2020-02-05T16:59:23.269 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.259+0000 7f90b3fff700 1 -- 172.21.15.8:0/2180960600 wait complete. 2020-02-05T16:59:23.272 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.263+0000 7f90b3fff700 1 Processor -- start 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.263+0000 7f90b3fff700 1 -- start start 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b3fff700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b3fff700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b41238a0 con 0x7f90b411c200 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b3fff700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90b411ccc0 con 0x7f90b411da20 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b3fff700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90b411c780 con 0x7f90b40a36d0 2020-02-05T16:59:23.273 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b37fe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f90b411c200 0x7f90b411d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46094/0 (socket says 172.21.15.8:46094) 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.265+0000 7f90b37fe700 1 -- 172.21.15.8:0/3368190082 learned_addr learned my addr 172.21.15.8:0/3368190082 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.266+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.266+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359951465 0 0) 0x7f90b41238a0 con 0x7f90b411c200 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.266+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9094003230 con 0x7f90b411c200 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.266+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432961969 0 0) 0x7f90b411ccc0 con 0x7f90b411da20 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.266+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90b41238a0 con 0x7f90b411da20 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3491622396 0 0) 0x7f9094003230 con 0x7f90b411c200 2020-02-05T16:59:23.274 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 -- 172.21.15.8:0/3368190082 >> v1:172.21.15.164:6789/0 conn(0x7f90b411da20 legacy=0x7f90b40731e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 -- 172.21.15.8:0/3368190082 >> v1:172.21.15.8:6789/0 conn(0x7f90b411c200 legacy=0x7f90b411d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 -- 172.21.15.8:0/3368190082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b40a3b00 con 0x7f90b40a36d0 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.267+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.275 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90a4001810 con 0x7f90b40a36d0 2020-02-05T16:59:23.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90a4001620 con 0x7f90b40a36d0 2020-02-05T16:59:23.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90b4076a00 con 0x7f90b40a36d0 2020-02-05T16:59:23.276 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90ac006db0 con 0x7f90b40a36d0 2020-02-05T16:59:23.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.268+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90b4076fa0 con 0x7f90b40a36d0 2020-02-05T16:59:23.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.269+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.270+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f90ac00b290 con 0x7f90b40a36d0 2020-02-05T16:59:23.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.270+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.277 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.270+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90b4056640 con 0x7f90b40a36d0 2020-02-05T16:59:23.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.271+0000 7f909bfff700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 0x7f909403c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.278 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.271+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f90b40aa980 con 0x7f90b40a36d0 2020-02-05T16:59:23.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.272+0000 7f90b37fe700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 0x7f909403c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.279 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.272+0000 7f90b37fe700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 0x7f909403c5b0 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.290 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.281+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.290 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.281+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f90ac030d20 con 0x7f90b40a36d0 2020-02-05T16:59:23.443 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.441+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f90b405f800 con 0x7f90b40a36d0 2020-02-05T16:59:23.444 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.442+0000 7f90b949e700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.444 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.442+0000 7f909bfff700 1 -- 172.21.15.8:0/3368190082 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f90ac030f80 con 0x7f90b40a36d0 2020-02-05T16:59:23.445 INFO:teuthology.orchestra.run.smithi008.stdout:55834574855 2020-02-05T16:59:23.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 msgr2=0x7f909403c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.447 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 0x7f909403c5b0 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 msgr2=0x7f90b4123350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f90a4001a10 tx=0x7f90a4003370).stop 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 shutdown_connections 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90b40a36d0 0x7f90b4123350 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 --2- 172.21.15.8:0/3368190082 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f909403a0a0 0x7f909403c5b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.444+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 >> 172.21.15.8:0/3368190082 conn(0x7f90b40ace50 msgr2=0x7f90b4147e90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:23.448 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.445+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 shutdown_connections 2020-02-05T16:59:23.449 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.445+0000 7f90b3fff700 1 -- 172.21.15.8:0/3368190082 wait complete. 2020-02-05T16:59:23.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2020-02-05T16:59:23.467 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2020-02-05T16:59:23.706 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.686+0000 7f113a00a700 1 Processor -- start 2020-02-05T16:59:23.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.687+0000 7f113a00a700 1 -- start start 2020-02-05T16:59:23.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.688+0000 7f113a00a700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 0x7f11340aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:23.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.688+0000 7f113a00a700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:23.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.688+0000 7f113a00a700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11340aa980 con 0x7f11340ace50 2020-02-05T16:59:23.709 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.688+0000 7f113a00a700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1134148c70 con 0x7f11340abd50 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.688+0000 7f113a00a700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1134148f10 con 0x7f1134145c20 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.689+0000 7f1139008700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 0x7f11340aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.689+0000 7f113b26c700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.689+0000 7f1139008700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 0x7f11340aa430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60276/0 (socket says 172.21.15.8:60276) 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.689+0000 7f1139008700 1 -- 172.21.15.8:0/3903364904 learned_addr learned my addr 172.21.15.8:0/3903364904 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.689+0000 7f1139809700 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f11340ace50 0x7f11341491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.8:58590/0 (socket says 172.21.15.8:58590) 2020-02-05T16:59:23.710 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 -- 172.21.15.8:0/3903364904 >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 msgr2=0x7f11340aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 0x7f11340aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 -- 172.21.15.8:0/3903364904 >> v1:172.21.15.164:6789/0 conn(0x7f11340ace50 legacy=0x7f11341491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 -- 172.21.15.8:0/3903364904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11341454a0 con 0x7f1134145c20 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f112c008c90 tx=0x7f112c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f112c008c90 tx=0x7f112c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f112c008c90 tx=0x7f112c0101f0).ready entity=mon.2 client_cookie=8c73f808f174a937 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.711 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f112c008c90 tx=0x7f112c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.712 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f112c008c90 tx=0x7f112c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.712 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f113b26c700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f112c008c90 tx=0x7f112c0101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.712 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.690+0000 7f1138807700 1 -- 172.21.15.8:0/3903364904 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f112c00a160 con 0x7f1134145c20 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f1138807700 1 -- 172.21.15.8:0/3903364904 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f112c011b70 con 0x7f1134145c20 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f113a00a700 1 -- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 msgr2=0x7f1134146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f1138807700 1 -- 172.21.15.8:0/3903364904 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f112c00a740 con 0x7f1134145c20 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.691+0000 7f113a00a700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f112c008c90 tx=0x7f112c0101f0).stop 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.692+0000 7f113a00a700 1 -- 172.21.15.8:0/3903364904 shutdown_connections 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.692+0000 7f113a00a700 1 --2- 172.21.15.8:0/3903364904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1134145c20 0x7f1134146050 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.713 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.692+0000 7f113a00a700 1 --2- 172.21.15.8:0/3903364904 >> v2:172.21.15.164:3300/0 conn(0x7f11340abd50 0x7f11340aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.692+0000 7f113a00a700 1 -- 172.21.15.8:0/3903364904 >> 172.21.15.8:0/3903364904 conn(0x7f1134061510 msgr2=0x7f11340ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.693+0000 7f113a00a700 1 -- 172.21.15.8:0/3903364904 shutdown_connections 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.693+0000 7f113a00a700 1 -- 172.21.15.8:0/3903364904 wait complete. 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.695+0000 7f113a00a700 1 Processor -- start 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.695+0000 7f113a00a700 1 -- start start 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f1134148570 0x7f113407df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.714 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11340a7c70 0x7f11340a80a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113411d430 con 0x7f1134148570 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f113411d6d0 con 0x7f11340a7c70 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113a00a700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f113412a380 con 0x7f113412a620 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f1139809700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f1134148570 0x7f113407df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f1139008700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f1139008700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44776/0 (socket says 172.21.15.8:44776) 2020-02-05T16:59:23.715 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f113b26c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f11340a7c70 0x7f11340a80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.696+0000 7f1139008700 1 -- 172.21.15.8:0/1331539510 learned_addr learned my addr 172.21.15.8:0/1331539510 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 -- 172.21.15.8:0/1331539510 >> v2:172.21.15.164:3300/0 conn(0x7f11340a7c70 msgr2=0x7f11340a80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> v2:172.21.15.164:3300/0 conn(0x7f11340a7c70 0x7f11340a80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 -- 172.21.15.8:0/1331539510 >> v2:172.21.15.8:3300/0 conn(0x7f1134148570 msgr2=0x7f113407df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> v2:172.21.15.8:3300/0 conn(0x7f1134148570 0x7f113407df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 -- 172.21.15.8:0/1331539510 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113411cba0 con 0x7f113412a620 2020-02-05T16:59:23.716 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.697+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1124007160 con 0x7f113412a620 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.717 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1124005ab0 con 0x7f113412a620 2020-02-05T16:59:23.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113411ee90 con 0x7f113412a620 2020-02-05T16:59:23.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1124012670 con 0x7f113412a620 2020-02-05T16:59:23.718 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.698+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113411d000 con 0x7f113412a620 2020-02-05T16:59:23.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.699+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1134056640 con 0x7f113412a620 2020-02-05T16:59:23.719 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.700+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.701+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f1124012980 con 0x7f113412a620 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.701+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.701+0000 7f112a7fc700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 0x7f111c03be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.701+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f1124049b30 con 0x7f113412a620 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.702+0000 7f1139809700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 0x7f111c03be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:23.720 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.703+0000 7f1139809700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 0x7f111c03be70 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:23.721 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.711+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.721 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.711+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f1124031560 con 0x7f113412a620 2020-02-05T16:59:23.870 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.867+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f1134066210 con 0x7f113412a620 2020-02-05T16:59:23.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.868+0000 7f1139008700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:23.873 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.868+0000 7f112a7fc700 1 -- 172.21.15.8:0/1331539510 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f11240317c0 con 0x7f113412a620 2020-02-05T16:59:23.873 INFO:teuthology.orchestra.run.smithi008.stdout:55834574854 2020-02-05T16:59:23.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.871+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 msgr2=0x7f111c03be70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.874 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.871+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 0x7f111c03be70 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.871+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 msgr2=0x7f113412aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:23.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.871+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7f113411d5f0 tx=0x7f1124010e40).stop 2020-02-05T16:59:23.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 shutdown_connections 2020-02-05T16:59:23.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f111c039960 0x7f111c03be70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.875 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> v2:172.21.15.164:3300/0 conn(0x7f11340a7c70 0x7f11340a80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> v2:172.21.15.8:3300/0 conn(0x7f1134148570 0x7f113407df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 --2- 172.21.15.8:0/1331539510 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f113412a620 0x7f113412aa50 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:23.876 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 >> 172.21.15.8:0/1331539510 conn(0x7f11340aad70 msgr2=0x7f11340a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:23.877 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 shutdown_connections 2020-02-05T16:59:23.878 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:23.872+0000 7f113a00a700 1 -- 172.21.15.8:0/1331539510 wait complete. 2020-02-05T16:59:23.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2020-02-05T16:59:23.897 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2020-02-05T16:59:23.899 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-02-05T16:59:24.126 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.115+0000 7f5914976700 1 Processor -- start 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.115+0000 7f5914976700 1 -- start start 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5914976700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f59100ace50 0x7f59100aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5914976700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59100aa980 con 0x7f5910145c20 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5914976700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59101454a0 con 0x7f59100abd50 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5914976700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59101456d0 con 0x7f59100ace50 2020-02-05T16:59:24.127 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5915177700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f5910145c20 0x7f591013f010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46120/0 (socket says 172.21.15.8:46120) 2020-02-05T16:59:24.128 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f5915177700 1 -- 172.21.15.8:0/3096374161 learned_addr learned my addr 172.21.15.8:0/3096374161 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:24.128 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.116+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> v2:172.21.15.8:3300/0 conn(0x7f59100ace50 0x7f59100aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.128 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394275467 0 0) 0x7f5908001450 con 0x7f5910145c20 2020-02-05T16:59:24.128 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f8003230 con 0x7f5910145c20 2020-02-05T16:59:24.128 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805350744 0 0) 0x7f59101454a0 con 0x7f59100abd50 2020-02-05T16:59:24.129 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5908001450 con 0x7f59100abd50 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590f59e700 1 -- 172.21.15.8:0/3096374161 >> v1:172.21.15.164:6789/0 conn(0x7f59100abd50 legacy=0x7f59101491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.117+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 523993152 0 0) 0x7f58f8003230 con 0x7f5910145c20 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.118+0000 7f590f59e700 1 -- 172.21.15.8:0/3096374161 >> v1:172.21.15.8:6789/0 conn(0x7f5910145c20 legacy=0x7f591013f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.118+0000 7f590f59e700 1 -- 172.21.15.8:0/3096374161 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910146050 con 0x7f59100ace50 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.118+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> v2:172.21.15.8:3300/0 conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5900008910 tx=0x7f5908001f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.118+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> v2:172.21.15.8:3300/0 conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5900008910 tx=0x7f5908001f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.130 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.118+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f5900008910 tx=0x7f5908001f00).ready entity=mon.0 client_cookie=5504e1d092f85593 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f5900008910 tx=0x7f5908001f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5908001d00 con 0x7f59100ace50 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f5900008910 tx=0x7f5908001f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590f59e700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f5900008910 tx=0x7f5908001f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5908001b10 con 0x7f59100ace50 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f590e59c700 1 -- 172.21.15.8:0/3096374161 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59000117d0 con 0x7f59100ace50 2020-02-05T16:59:24.131 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f5914976700 1 -- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 msgr2=0x7f59100aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.119+0000 7f5914976700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f5900008910 tx=0x7f5908001f00).stop 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.120+0000 7f5914976700 1 -- 172.21.15.8:0/3096374161 shutdown_connections 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.120+0000 7f5914976700 1 --2- 172.21.15.8:0/3096374161 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100ace50 0x7f59100aa430 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.120+0000 7f5914976700 1 -- 172.21.15.8:0/3096374161 >> 172.21.15.8:0/3096374161 conn(0x7f5910061510 msgr2=0x7f59100ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.121+0000 7f5914976700 1 -- 172.21.15.8:0/3096374161 shutdown_connections 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.121+0000 7f5914976700 1 -- 172.21.15.8:0/3096374161 wait complete. 2020-02-05T16:59:24.132 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.123+0000 7f5914976700 1 Processor -- start 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.123+0000 7f5914976700 1 -- start start 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5914976700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59100a7150 0x7f59100a60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5914976700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f59100a85d0 0x7f591007e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5914976700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59100a8a00 con 0x7f591011e060 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5914976700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59100a6c10 con 0x7f59100a85d0 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5914976700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59100a6eb0 con 0x7f59100a7150 2020-02-05T16:59:24.133 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f590f59e700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f591011e060 0x7f591012cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46122/0 (socket says 172.21.15.8:46122) 2020-02-05T16:59:24.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f590f59e700 1 -- 172.21.15.8:0/102155900 learned_addr learned my addr 172.21.15.8:0/102155900 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:24.134 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.124+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> v2:172.21.15.8:3300/0 conn(0x7f59100a85d0 0x7f591007e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664199358 0 0) 0x7f59100a8a00 con 0x7f591011e060 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f590ed9d700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59100a7150 0x7f59100a60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58f0003230 con 0x7f591011e060 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3513162463 0 0) 0x7f58f0003230 con 0x7f591011e060 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 --> v1:172.21.15.8:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59100a8a00 con 0x7f591011e060 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.125+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f5900008690 con 0x7f591011e060 2020-02-05T16:59:24.137 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 -- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59100a7150 msgr2=0x7f59100a60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59100a7150 0x7f59100a60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 -- 172.21.15.8:0/102155900 >> v1:172.21.15.8:6789/0 conn(0x7f591011e060 legacy=0x7f591012cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 -- 172.21.15.8:0/102155900 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591007ef30 con 0x7f59100a85d0 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> v2:172.21.15.8:3300/0 conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> v2:172.21.15.8:3300/0 conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.138 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.139 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f590800ccd0 con 0x7f59100a85d0 2020-02-05T16:59:24.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.140 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590800c8f0 con 0x7f59100a85d0 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5910137710 con 0x7f59100a85d0 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.126+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5908004c50 con 0x7f59100a85d0 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.127+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5910137ce0 con 0x7f59100a85d0 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.128+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.128+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f5908012640 con 0x7f59100a85d0 2020-02-05T16:59:24.141 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.129+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.129+0000 7f58f7fff700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 0x7f58f003c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.129+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f59100aa980 con 0x7f59100a85d0 2020-02-05T16:59:24.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.129+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5910056640 con 0x7f59100a85d0 2020-02-05T16:59:24.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.130+0000 7f590f59e700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 0x7f58f003c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.142 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.130+0000 7f590f59e700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 0x7f58f003c890 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.136+0000 7f5915177700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.143 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.136+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f5908030df0 con 0x7f59100a85d0 2020-02-05T16:59:24.292 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.289+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 --> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5910066210 con 0x7f58f003a380 2020-02-05T16:59:24.301 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.297+0000 7f58f7fff700 1 -- 172.21.15.8:0/102155900 <== mgr.4104 v2:172.21.15.164:6800/21877 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66267 (crc 0 0 0) 0x7f5910066210 con 0x7f58f003a380 2020-02-05T16:59:24.301 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:24.302 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 msgr2=0x7f58f003c890 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.302 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 0x7f58f003c890 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 msgr2=0x7f591007e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f59100a6dd0 tx=0x7f5900010280).stop 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 shutdown_connections 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f58f003a380 0x7f58f003c890 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f59100a85d0 0x7f591007e9e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.299+0000 7f5914976700 1 --2- 172.21.15.8:0/102155900 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59100a7150 0x7f59100a60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.303 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.300+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 >> 172.21.15.8:0/102155900 conn(0x7f59100ac240 msgr2=0x7f591005ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:24.304 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.300+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 shutdown_connections 2020-02-05T16:59:24.304 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.300+0000 7f5914976700 1 -- 172.21.15.8:0/102155900 wait complete. 2020-02-05T16:59:24.304 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2020-02-05T16:59:24.325 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-02-05T16:59:23.470478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390432,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584288,"statfs":{"total":773094113280,"available":764502310912,"internally_reserved":8589934592,"allocated":1802240,"data_stored":607732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001451"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.238483+0000","last_change":"2020-02-05T16:59:12.238019+0000","last_active":"2020-02-05T16:59:12.238483+0000","last_peered":"2020-02-05T16:59:12.238483+0000","last_clean":"2020-02-05T16:59:12.238483+0000","last_became_active":"2020-02-05T16:59:12.237491+0000","last_became_peered":"2020-02-05T16:59:12.237491+0000","last_unstale":"2020-02-05T16:59:12.238483+0000","last_undegraded":"2020-02-05T16:59:12.238483+0000","last_fullsized":"2020-02-05T16:59:12.238483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.106187+0000","last_change":"2020-02-05T16:59:08.577368+0000","last_active":"2020-02-05T16:59:12.106187+0000","last_peered":"2020-02-05T16:59:12.106187+0000","last_clean":"2020-02-05T16:59:12.106187+0000","last_became_active":"2020-02-05T16:59:08.576789+0000","last_became_peered":"2020-02-05T16:59:08.576789+0000","last_unstale":"2020-02-05T16:59:12.106187+0000","last_undegraded":"2020-02-05T16:59:12.106187+0000","last_fullsized":"2020-02-05T16:59:12.106187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.148974+0000","last_change":"2020-02-05T16:59:11.613478+0000","last_active":"2020-02-05T16:59:16.148974+0000","last_peered":"2020-02-05T16:59:16.148974+0000","last_clean":"2020-02-05T16:59:16.148974+0000","last_became_active":"2020-02-05T16:59:11.613011+0000","last_became_peered":"2020-02-05T16:59:11.613011+0000","last_unstale":"2020-02-05T16:59:16.148974+0000","last_undegraded":"2020-02-05T16:59:16.148974+0000","last_fullsized":"2020-02-05T16:59:16.148974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.110491+0000","last_change":"2020-02-05T16:59:12.109354+0000","last_active":"2020-02-05T16:59:12.110491+0000","last_peered":"2020-02-05T16:59:12.110491+0000","last_clean":"2020-02-05T16:59:12.110491+0000","last_became_active":"2020-02-05T16:59:12.108756+0000","last_became_peered":"2020-02-05T16:59:12.108756+0000","last_unstale":"2020-02-05T16:59:12.110491+0000","last_undegraded":"2020-02-05T16:59:12.110491+0000","last_fullsized":"2020-02-05T16:59:12.110491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.103192+0000","last_change":"2020-02-05T16:59:08.577372+0000","last_active":"2020-02-05T16:59:12.103192+0000","last_peered":"2020-02-05T16:59:12.103192+0000","last_clean":"2020-02-05T16:59:12.103192+0000","last_became_active":"2020-02-05T16:59:08.576472+0000","last_became_peered":"2020-02-05T16:59:08.576472+0000","last_unstale":"2020-02-05T16:59:12.103192+0000","last_undegraded":"2020-02-05T16:59:12.103192+0000","last_fullsized":"2020-02-05T16:59:12.103192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":"17","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.144099+0000","last_change":"2020-02-05T16:59:11.613923+0000","last_active":"2020-02-05T16:59:16.144099+0000","last_peered":"2020-02-05T16:59:16.144099+0000","last_clean":"2020-02-05T16:59:16.144099+0000","last_became_active":"2020-02-05T16:59:11.613525+0000","last_became_peered":"2020-02-05T16:59:11.613525+0000","last_unstale":"2020-02-05T16:59:16.144099+0000","last_undegraded":"2020-02-05T16:59:16.144099+0000","last_fullsized":"2020-02-05T16:59:16.144099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.110444+0000","last_change":"2020-02-05T16:59:12.108489+0000","last_active":"2020-02-05T16:59:12.110444+0000","last_peered":"2020-02-05T16:59:12.110444+0000","last_clean":"2020-02-05T16:59:12.110444+0000","last_became_active":"2020-02-05T16:59:12.107889+0000","last_became_peered":"2020-02-05T16:59:12.107889+0000","last_unstale":"2020-02-05T16:59:12.110444+0000","last_undegraded":"2020-02-05T16:59:12.110444+0000","last_fullsized":"2020-02-05T16:59:12.110444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.106591+0000","last_change":"2020-02-05T16:59:08.576988+0000","last_active":"2020-02-05T16:59:12.106591+0000","last_peered":"2020-02-05T16:59:12.106591+0000","last_clean":"2020-02-05T16:59:12.106591+0000","last_became_active":"2020-02-05T16:59:08.576300+0000","last_became_peered":"2020-02-05T16:59:08.576300+0000","last_unstale":"2020-02-05T16:59:12.106591+0000","last_undegraded":"2020-02-05T16:59:12.106591+0000","last_fullsized":"2020-02-05T16:59:12.106591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.149185+0000","last_change":"2020-02-05T16:59:11.616540+0000","last_active":"2020-02-05T16:59:16.149185+0000","last_peered":"2020-02-05T16:59:16.149185+0000","last_clean":"2020-02-05T16:59:16.149185+0000","last_became_active":"2020-02-05T16:59:11.616100+0000","last_became_peered":"2020-02-05T16:59:11.616100+0000","last_unstale":"2020-02-05T16:59:16.149185+0000","last_undegraded":"2020-02-05T16:59:16.149185+0000","last_fullsized":"2020-02-05T16:59:16.149185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.112850+0000","last_change":"2020-02-05T16:59:12.112422+0000","last_active":"2020-02-05T16:59:12.112850+0000","last_peered":"2020-02-05T16:59:12.112850+0000","last_clean":"2020-02-05T16:59:12.112850+0000","last_became_active":"2020-02-05T16:59:12.112019+0000","last_became_peered":"2020-02-05T16:59:12.112019+0000","last_unstale":"2020-02-05T16:59:12.112850+0000","last_undegraded":"2020-02-05T16:59:12.112850+0000","last_fullsized":"2020-02-05T16:59:12.112850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234949+0000","last_change":"2020-02-05T16:59:08.724622+0000","last_active":"2020-02-05T16:59:12.234949+0000","last_peered":"2020-02-05T16:59:12.234949+0000","last_clean":"2020-02-05T16:59:12.234949+0000","last_became_active":"2020-02-05T16:59:08.723481+0000","last_became_peered":"2020-02-05T16:59:08.723481+0000","last_unstale":"2020-02-05T16:59:12.234949+0000","last_undegraded":"2020-02-05T16:59:12.234949+0000","last_fullsized":"2020-02-05T16:59:12.234949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.099503+0000","last_change":"2020-02-05T16:59:11.616582+0000","last_active":"2020-02-05T16:59:12.099503+0000","last_peered":"2020-02-05T16:59:12.099503+0000","last_clean":"2020-02-05T16:59:12.099503+0000","last_became_active":"2020-02-05T16:59:11.615401+0000","last_became_peered":"2020-02-05T16:59:11.615401+0000","last_unstale":"2020-02-05T16:59:12.099503+0000","last_undegraded":"2020-02-05T16:59:12.099503+0000","last_fullsized":"2020-02-05T16:59:12.099503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.109996+0000","last_change":"2020-02-05T16:59:12.108066+0000","last_active":"2020-02-05T16:59:12.109996+0000","last_peered":"2020-02-05T16:59:12.109996+0000","last_clean":"2020-02-05T16:59:12.109996+0000","last_became_active":"2020-02-05T16:59:12.107619+0000","last_became_peered":"2020-02-05T16:59:12.107619+0000","last_unstale":"2020-02-05T16:59:12.109996+0000","last_undegraded":"2020-02-05T16:59:12.109996+0000","last_fullsized":"2020-02-05T16:59:12.109996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234852+0000","last_change":"2020-02-05T16:59:08.724786+0000","last_active":"2020-02-05T16:59:12.234852+0000","last_peered":"2020-02-05T16:59:12.234852+0000","last_clean":"2020-02-05T16:59:12.234852+0000","last_became_active":"2020-02-05T16:59:08.723665+0000","last_became_peered":"2020-02-05T16:59:08.723665+0000","last_unstale":"2020-02-05T16:59:12.234852+0000","last_undegraded":"2020-02-05T16:59:12.234852+0000","last_fullsized":"2020-02-05T16:59:12.234852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.143035+0000","last_change":"2020-02-05T16:59:11.614904+0000","last_active":"2020-02-05T16:59:16.143035+0000","last_peered":"2020-02-05T16:59:16.143035+0000","last_clean":"2020-02-05T16:59:16.143035+0000","last_became_active":"2020-02-05T16:59:11.614579+0000","last_became_peered":"2020-02-05T16:59:11.614579+0000","last_unstale":"2020-02-05T16:59:16.143035+0000","last_undegraded":"2020-02-05T16:59:16.143035+0000","last_fullsized":"2020-02-05T16:59:16.143035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.109411+0000","last_change":"2020-02-05T16:59:12.109069+0000","last_active":"2020-02-05T16:59:12.109411+0000","last_peered":"2020-02-05T16:59:12.109411+0000","last_clean":"2020-02-05T16:59:12.109411+0000","last_became_active":"2020-02-05T16:59:12.108517+0000","last_became_peered":"2020-02-05T16:59:12.108517+0000","last_unstale":"2020-02-05T16:59:12.109411+0000","last_undegraded":"2020-02-05T16:59:12.109411+0000","last_fullsized":"2020-02-05T16:59:12.109411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.105907+0000","last_change":"2020-02-05T16:59:08.724708+0000","last_active":"2020-02-05T16:59:12.105907+0000","last_peered":"2020-02-05T16:59:12.105907+0000","last_clean":"2020-02-05T16:59:12.105907+0000","last_became_active":"2020-02-05T16:59:08.723327+0000","last_became_peered":"2020-02-05T16:59:08.723327+0000","last_unstale":"2020-02-05T16:59:12.105907+0000","last_undegraded":"2020-02-05T16:59:12.105907+0000","last_fullsized":"2020-02-05T16:59:12.105907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:14.651542+0000","last_change":"2020-02-05T16:59:11.613922+0000","last_active":"2020-02-05T16:59:14.651542+0000","last_peered":"2020-02-05T16:59:14.651542+0000","last_clean":"2020-02-05T16:59:14.651542+0000","last_became_active":"2020-02-05T16:59:11.613550+0000","last_became_peered":"2020-02-05T16:59:11.613550+0000","last_unstale":"2020-02-05T16:59:14.651542+0000","last_undegraded":"2020-02-05T16:59:14.651542+0000","last_fullsized":"2020-02-05T16:59:14.651542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.111843+0000","last_change":"2020-02-05T16:59:12.111451+0000","last_active":"2020-02-05T16:59:12.111843+0000","last_peered":"2020-02-05T16:59:12.111843+0000","last_clean":"2020-02-05T16:59:12.111843+0000","last_became_active":"2020-02-05T16:59:12.110786+0000","last_became_peered":"2020-02-05T16:59:12.110786+0000","last_unstale":"2020-02-05T16:59:12.111843+0000","last_undegraded":"2020-02-05T16:59:12.111843+0000","last_fullsized":"2020-02-05T16:59:12.111843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234783+0000","last_change":"2020-02-05T16:59:08.725966+0000","last_active":"2020-02-05T16:59:12.234783+0000","last_peered":"2020-02-05T16:59:12.234783+0000","last_clean":"2020-02-05T16:59:12.234783+0000","last_became_active":"2020-02-05T16:59:08.723246+0000","last_became_peered":"2020-02-05T16:59:08.723246+0000","last_unstale":"2020-02-05T16:59:12.234783+0000","last_undegraded":"2020-02-05T16:59:12.234783+0000","last_fullsized":"2020-02-05T16:59:12.234783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.148626+0000","last_change":"2020-02-05T16:59:11.612206+0000","last_active":"2020-02-05T16:59:16.148626+0000","last_peered":"2020-02-05T16:59:16.148626+0000","last_clean":"2020-02-05T16:59:16.148626+0000","last_became_active":"2020-02-05T16:59:11.611539+0000","last_became_peered":"2020-02-05T16:59:11.611539+0000","last_unstale":"2020-02-05T16:59:16.148626+0000","last_undegraded":"2020-02-05T16:59:16.148626+0000","last_fullsized":"2020-02-05T16:59:16.148626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.108963+0000","last_change":"2020-02-05T16:59:12.108437+0000","last_active":"2020-02-05T16:59:12.108963+0000","last_peered":"2020-02-05T16:59:12.108963+0000","last_clean":"2020-02-05T16:59:12.108963+0000","last_became_active":"2020-02-05T16:59:12.107863+0000","last_became_peered":"2020-02-05T16:59:12.107863+0000","last_unstale":"2020-02-05T16:59:12.108963+0000","last_undegraded":"2020-02-05T16:59:12.108963+0000","last_fullsized":"2020-02-05T16:59:12.108963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234841+0000","last_change":"2020-02-05T16:59:08.579318+0000","last_active":"2020-02-05T16:59:12.234841+0000","last_peered":"2020-02-05T16:59:12.234841+0000","last_clean":"2020-02-05T16:59:12.234841+0000","last_became_active":"2020-02-05T16:59:08.578888+0000","last_became_peered":"2020-02-05T16:59:08.578888+0000","last_unstale":"2020-02-05T16:59:12.234841+0000","last_undegraded":"2020-02-05T16:59:12.234841+0000","last_fullsized":"2020-02-05T16:59:12.234841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.150618+0000","last_change":"2020-02-05T16:59:11.613302+0000","last_active":"2020-02-05T16:59:16.150618+0000","last_peered":"2020-02-05T16:59:16.150618+0000","last_clean":"2020-02-05T16:59:16.150618+0000","last_became_active":"2020-02-05T16:59:11.612808+0000","last_became_peered":"2020-02-05T16:59:11.612808+0000","last_unstale":"2020-02-05T16:59:16.150618+0000","last_undegraded":"2020-02-05T16:59:16.150618+0000","last_fullsized":"2020-02-05T16:59:16.150618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":76121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":75961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":76921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048816,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323024,"statfs":{"total":96636764160,"available":95562776576,"internally_reserved":1073741824,"allocated":237568,"data_stored":77081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":75429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":75395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":75395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":75429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-02-05T16:59:24.329 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2020-02-05T16:59:24.564 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.555+0000 7fd98475d700 1 Processor -- start 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.555+0000 7fd98475d700 1 -- start start 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd98475d700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd97c08fff0 0x7fd97c08f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd98475d700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd97c0a9e90 0x7fd97c0a8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd98475d700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0aab90 con 0x7fd97c0908f0 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd98475d700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c090d20 con 0x7fd97c0a9e90 2020-02-05T16:59:24.565 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd98475d700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c090650 con 0x7fd97c08fff0 2020-02-05T16:59:24.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd9824f9700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fd97c0908f0 0x7fd97c0a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46132/0 (socket says 172.21.15.8:46132) 2020-02-05T16:59:24.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd9824f9700 1 -- 172.21.15.8:0/3683663041 learned_addr learned my addr 172.21.15.8:0/3683663041 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:24.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd982cfa700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd97c08fff0 0x7fd97c08f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.566 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> v2:172.21.15.164:3300/0 conn(0x7fd97c0a9e90 0x7fd97c0a8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.556+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432739087 0 0) 0x7fd97c0aab90 con 0x7fd97c0908f0 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968003230 con 0x7fd97c0908f0 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1958443487 0 0) 0x7fd968003230 con 0x7fd97c0908f0 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd97c08fff0 msgr2=0x7fd97c08f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd97c08fff0 0x7fd97c08f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 >> v2:172.21.15.164:3300/0 conn(0x7fd97c0a9e90 msgr2=0x7fd97c0a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:24.567 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 --2- 172.21.15.8:0/3683663041 >> v2:172.21.15.164:3300/0 conn(0x7fd97c0a9e90 0x7fd97c0a8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0a86e0 con 0x7fd97c0908f0 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fd96c001750 con 0x7fd97c0908f0 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.557+0000 7fd982cfa700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd97c08fff0 0x7fd97c08f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 >> v1:172.21.15.8:6789/0 conn(0x7fd97c0908f0 legacy=0x7fd97c0a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd968003860 0x7fd968003c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9680045a0 0x7fd9680049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.568 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c090d20 con 0x7fd968004f20 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd97c090650 con 0x7fd968003860 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9680053a0 con 0x7fd9680045a0 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd9824f9700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9680045a0 0x7fd9680049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.558+0000 7fd982cfa700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd968003860 0x7fd968003c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 -- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9680045a0 msgr2=0x7fd9680049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:24.569 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9680045a0 0x7fd9680049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 -- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd968003860 msgr2=0x7fd968003c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd968003860 0x7fd968003c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 -- 172.21.15.8:0/3683663041 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96804fe40 con 0x7fd968004f20 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd982cfa700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd968003860 0x7fd968003c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd974002f30 tx=0x7fd974010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd974002f30 tx=0x7fd974010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.570 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7fd974002f30 tx=0x7fd974010160).ready entity=mon.0 client_cookie=85a0d9bd6fed3de4 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.559+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fd974002f30 tx=0x7fd974010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fd974002f30 tx=0x7fd974010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd974008970 con 0x7fd968004f20 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd9834fb700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fd974002f30 tx=0x7fd974010160).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd974010ab0 con 0x7fd968004f20 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd98475d700 1 -- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 msgr2=0x7fd96804f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.571 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd98475d700 1 --2- 172.21.15.8:0/3683663041 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd968004f20 0x7fd96804f800 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7fd974002f30 tx=0x7fd974010160).stop 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd981cf8700 1 -- 172.21.15.8:0/3683663041 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9740110c0 con 0x7fd968004f20 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.560+0000 7fd9834fb700 1 -- 172.21.15.8:0/3683663041 reap_dead start 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.561+0000 7fd98475d700 1 -- 172.21.15.8:0/3683663041 shutdown_connections 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.561+0000 7fd98475d700 1 -- 172.21.15.8:0/3683663041 >> 172.21.15.8:0/3683663041 conn(0x7fd97c0600f0 msgr2=0x7fd97c05a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.561+0000 7fd98475d700 1 -- 172.21.15.8:0/3683663041 shutdown_connections 2020-02-05T16:59:24.572 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.561+0000 7fd98475d700 1 -- 172.21.15.8:0/3683663041 wait complete. 2020-02-05T16:59:24.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.563+0000 7fd98475d700 1 Processor -- start 2020-02-05T16:59:24.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.563+0000 7fd98475d700 1 -- start start 2020-02-05T16:59:24.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.564+0000 7fd98475d700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd97c0a3b10 0x7fd97c09dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.564+0000 7fd98475d700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd97c09e410 0x7fd97c0a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.575 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.564+0000 7fd98475d700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97c0a63f0 con 0x7fd97c0a6ea0 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.564+0000 7fd98475d700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c09f9b0 con 0x7fd97c0a3b10 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.564+0000 7fd98475d700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97c09fc50 con 0x7fd97c09e410 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.565+0000 7fd982cfa700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd97c0a3b10 0x7fd97c09dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.565+0000 7fd982cfa700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fd97c0a3b10 0x7fd97c09dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44762/0 (socket says 172.21.15.8:44762) 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.565+0000 7fd982cfa700 1 -- 172.21.15.8:0/4000881736 learned_addr learned my addr 172.21.15.8:0/4000881736 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:24.576 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.565+0000 7fd9824f9700 1 --2- 172.21.15.8:0/4000881736 >> v2:172.21.15.164:3300/0 conn(0x7fd97c09e410 0x7fd97c0a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913068148 0 0) 0x7fd97c0a63f0 con 0x7fd97c0a6ea0 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd964003230 con 0x7fd97c0a6ea0 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 -- 172.21.15.8:0/4000881736 >> v2:172.21.15.164:3300/0 conn(0x7fd97c09e410 msgr2=0x7fd97c0a6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> v2:172.21.15.164:3300/0 conn(0x7fd97c09e410 0x7fd97c0a6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 -- 172.21.15.8:0/4000881736 >> v1:172.21.15.8:6789/0 conn(0x7fd97c0a6ea0 legacy=0x7fd97c0650b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 -- 172.21.15.8:0/4000881736 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c09f6a0 con 0x7fd97c0a3b10 2020-02-05T16:59:24.577 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> v2:172.21.15.8:3300/0 conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> v2:172.21.15.8:3300/0 conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.566+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd978012850 con 0x7fd97c0a3b10 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.578 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978005a40 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97c0a72d0 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd978007150 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.567+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97c0a5f80 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.568+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.568+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fd978007470 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.568+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97c056640 con 0x7fd97c0a3b10 2020-02-05T16:59:24.579 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.568+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.569+0000 7fd9737fe700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 0x7fd96403c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.569+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fd978049ba0 con 0x7fd97c0a3b10 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.569+0000 7fd9824f9700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 0x7fd96403c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.570+0000 7fd9824f9700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 0x7fd96403c3e0 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.573+0000 7fd982cfa700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=THROTTLE_DONE pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:24.580 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.573+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fd978031670 con 0x7fd97c0a3b10 2020-02-05T16:59:24.728 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.725+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 --> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd97c0a00b0 con 0x7fd964039ed0 2020-02-05T16:59:24.734 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.732+0000 7fd9737fe700 1 -- 172.21.15.8:0/4000881736 <== mgr.4104 v2:172.21.15.164:6800/21877 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+66267 (crc 0 0 0) 0x7fd97c0a00b0 con 0x7fd964039ed0 2020-02-05T16:59:24.735 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:24.737 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.734+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 msgr2=0x7fd96403c3e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.737 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.734+0000 7fd98475d700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 0x7fd96403c3e0 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.737 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.734+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 msgr2=0x7fd97c09dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.734+0000 7fd98475d700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 secure :-1 s=READY pgs=131 cs=0 l=1 rx=0x7fd97c0a6620 tx=0x7fd978010ed0).stop 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 shutdown_connections 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fd964039ed0 0x7fd96403c3e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 --2- 172.21.15.8:0/4000881736 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fd97c0a3b10 0x7fd97c09dec0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 --2- 172.21.15.8:0/4000881736 >> v2:172.21.15.164:3300/0 conn(0x7fd97c09e410 0x7fd97c0a6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 >> 172.21.15.8:0/4000881736 conn(0x7fd97c0a46f0 msgr2=0x7fd97c0a7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:24.738 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 shutdown_connections 2020-02-05T16:59:24.739 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.735+0000 7fd98475d700 1 -- 172.21.15.8:0/4000881736 wait complete. 2020-02-05T16:59:24.739 INFO:teuthology.orchestra.run.smithi008.stderr:dumped all 2020-02-05T16:59:24.758 INFO:teuthology.orchestra.run.smithi008.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2020-02-05T16:59:23.470478+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":48,"acting":48,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":48,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390432,"kb_used_data":1760,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584288,"statfs":{"total":773094113280,"available":764502310912,"internally_reserved":8589934592,"allocated":1802240,"data_stored":607732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001451"},"pg_stats":[{"pgid":"3.7","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.238483+0000","last_change":"2020-02-05T16:59:12.238019+0000","last_active":"2020-02-05T16:59:12.238483+0000","last_peered":"2020-02-05T16:59:12.238483+0000","last_clean":"2020-02-05T16:59:12.238483+0000","last_became_active":"2020-02-05T16:59:12.237491+0000","last_became_peered":"2020-02-05T16:59:12.237491+0000","last_unstale":"2020-02-05T16:59:12.238483+0000","last_undegraded":"2020-02-05T16:59:12.238483+0000","last_fullsized":"2020-02-05T16:59:12.238483+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.106187+0000","last_change":"2020-02-05T16:59:08.577368+0000","last_active":"2020-02-05T16:59:12.106187+0000","last_peered":"2020-02-05T16:59:12.106187+0000","last_clean":"2020-02-05T16:59:12.106187+0000","last_became_active":"2020-02-05T16:59:08.576789+0000","last_became_peered":"2020-02-05T16:59:08.576789+0000","last_unstale":"2020-02-05T16:59:12.106187+0000","last_undegraded":"2020-02-05T16:59:12.106187+0000","last_fullsized":"2020-02-05T16:59:12.106187+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"21'3","reported_seq":"13","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.148974+0000","last_change":"2020-02-05T16:59:11.613478+0000","last_active":"2020-02-05T16:59:16.148974+0000","last_peered":"2020-02-05T16:59:16.148974+0000","last_clean":"2020-02-05T16:59:16.148974+0000","last_became_active":"2020-02-05T16:59:11.613011+0000","last_became_peered":"2020-02-05T16:59:11.613011+0000","last_unstale":"2020-02-05T16:59:16.148974+0000","last_undegraded":"2020-02-05T16:59:16.148974+0000","last_fullsized":"2020-02-05T16:59:16.148974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.110491+0000","last_change":"2020-02-05T16:59:12.109354+0000","last_active":"2020-02-05T16:59:12.110491+0000","last_peered":"2020-02-05T16:59:12.110491+0000","last_clean":"2020-02-05T16:59:12.110491+0000","last_became_active":"2020-02-05T16:59:12.108756+0000","last_became_peered":"2020-02-05T16:59:12.108756+0000","last_unstale":"2020-02-05T16:59:12.110491+0000","last_undegraded":"2020-02-05T16:59:12.110491+0000","last_fullsized":"2020-02-05T16:59:12.110491+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.103192+0000","last_change":"2020-02-05T16:59:08.577372+0000","last_active":"2020-02-05T16:59:12.103192+0000","last_peered":"2020-02-05T16:59:12.103192+0000","last_clean":"2020-02-05T16:59:12.103192+0000","last_became_active":"2020-02-05T16:59:08.576472+0000","last_became_peered":"2020-02-05T16:59:08.576472+0000","last_unstale":"2020-02-05T16:59:12.103192+0000","last_undegraded":"2020-02-05T16:59:12.103192+0000","last_fullsized":"2020-02-05T16:59:12.103192+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":"17","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.144099+0000","last_change":"2020-02-05T16:59:11.613923+0000","last_active":"2020-02-05T16:59:16.144099+0000","last_peered":"2020-02-05T16:59:16.144099+0000","last_clean":"2020-02-05T16:59:16.144099+0000","last_became_active":"2020-02-05T16:59:11.613525+0000","last_became_peered":"2020-02-05T16:59:11.613525+0000","last_unstale":"2020-02-05T16:59:16.144099+0000","last_undegraded":"2020-02-05T16:59:16.144099+0000","last_fullsized":"2020-02-05T16:59:16.144099+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.110444+0000","last_change":"2020-02-05T16:59:12.108489+0000","last_active":"2020-02-05T16:59:12.110444+0000","last_peered":"2020-02-05T16:59:12.110444+0000","last_clean":"2020-02-05T16:59:12.110444+0000","last_became_active":"2020-02-05T16:59:12.107889+0000","last_became_peered":"2020-02-05T16:59:12.107889+0000","last_unstale":"2020-02-05T16:59:12.110444+0000","last_undegraded":"2020-02-05T16:59:12.110444+0000","last_fullsized":"2020-02-05T16:59:12.110444+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.106591+0000","last_change":"2020-02-05T16:59:08.576988+0000","last_active":"2020-02-05T16:59:12.106591+0000","last_peered":"2020-02-05T16:59:12.106591+0000","last_clean":"2020-02-05T16:59:12.106591+0000","last_became_active":"2020-02-05T16:59:08.576300+0000","last_became_peered":"2020-02-05T16:59:08.576300+0000","last_unstale":"2020-02-05T16:59:12.106591+0000","last_undegraded":"2020-02-05T16:59:12.106591+0000","last_fullsized":"2020-02-05T16:59:12.106591+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.149185+0000","last_change":"2020-02-05T16:59:11.616540+0000","last_active":"2020-02-05T16:59:16.149185+0000","last_peered":"2020-02-05T16:59:16.149185+0000","last_clean":"2020-02-05T16:59:16.149185+0000","last_became_active":"2020-02-05T16:59:11.616100+0000","last_became_peered":"2020-02-05T16:59:11.616100+0000","last_unstale":"2020-02-05T16:59:16.149185+0000","last_undegraded":"2020-02-05T16:59:16.149185+0000","last_fullsized":"2020-02-05T16:59:16.149185+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.112850+0000","last_change":"2020-02-05T16:59:12.112422+0000","last_active":"2020-02-05T16:59:12.112850+0000","last_peered":"2020-02-05T16:59:12.112850+0000","last_clean":"2020-02-05T16:59:12.112850+0000","last_became_active":"2020-02-05T16:59:12.112019+0000","last_became_peered":"2020-02-05T16:59:12.112019+0000","last_unstale":"2020-02-05T16:59:12.112850+0000","last_undegraded":"2020-02-05T16:59:12.112850+0000","last_fullsized":"2020-02-05T16:59:12.112850+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234949+0000","last_change":"2020-02-05T16:59:08.724622+0000","last_active":"2020-02-05T16:59:12.234949+0000","last_peered":"2020-02-05T16:59:12.234949+0000","last_clean":"2020-02-05T16:59:12.234949+0000","last_became_active":"2020-02-05T16:59:08.723481+0000","last_became_peered":"2020-02-05T16:59:08.723481+0000","last_unstale":"2020-02-05T16:59:12.234949+0000","last_undegraded":"2020-02-05T16:59:12.234949+0000","last_fullsized":"2020-02-05T16:59:12.234949+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"10","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.099503+0000","last_change":"2020-02-05T16:59:11.616582+0000","last_active":"2020-02-05T16:59:12.099503+0000","last_peered":"2020-02-05T16:59:12.099503+0000","last_clean":"2020-02-05T16:59:12.099503+0000","last_became_active":"2020-02-05T16:59:11.615401+0000","last_became_peered":"2020-02-05T16:59:11.615401+0000","last_unstale":"2020-02-05T16:59:12.099503+0000","last_undegraded":"2020-02-05T16:59:12.099503+0000","last_fullsized":"2020-02-05T16:59:12.099503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.109996+0000","last_change":"2020-02-05T16:59:12.108066+0000","last_active":"2020-02-05T16:59:12.109996+0000","last_peered":"2020-02-05T16:59:12.109996+0000","last_clean":"2020-02-05T16:59:12.109996+0000","last_became_active":"2020-02-05T16:59:12.107619+0000","last_became_peered":"2020-02-05T16:59:12.107619+0000","last_unstale":"2020-02-05T16:59:12.109996+0000","last_undegraded":"2020-02-05T16:59:12.109996+0000","last_fullsized":"2020-02-05T16:59:12.109996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"13","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234852+0000","last_change":"2020-02-05T16:59:08.724786+0000","last_active":"2020-02-05T16:59:12.234852+0000","last_peered":"2020-02-05T16:59:12.234852+0000","last_clean":"2020-02-05T16:59:12.234852+0000","last_became_active":"2020-02-05T16:59:08.723665+0000","last_became_peered":"2020-02-05T16:59:08.723665+0000","last_unstale":"2020-02-05T16:59:12.234852+0000","last_undegraded":"2020-02-05T16:59:12.234852+0000","last_fullsized":"2020-02-05T16:59:12.234852+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'5","reported_seq":"15","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.143035+0000","last_change":"2020-02-05T16:59:11.614904+0000","last_active":"2020-02-05T16:59:16.143035+0000","last_peered":"2020-02-05T16:59:16.143035+0000","last_clean":"2020-02-05T16:59:16.143035+0000","last_became_active":"2020-02-05T16:59:11.614579+0000","last_became_peered":"2020-02-05T16:59:11.614579+0000","last_unstale":"2020-02-05T16:59:16.143035+0000","last_undegraded":"2020-02-05T16:59:16.143035+0000","last_fullsized":"2020-02-05T16:59:16.143035+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.109411+0000","last_change":"2020-02-05T16:59:12.109069+0000","last_active":"2020-02-05T16:59:12.109411+0000","last_peered":"2020-02-05T16:59:12.109411+0000","last_clean":"2020-02-05T16:59:12.109411+0000","last_became_active":"2020-02-05T16:59:12.108517+0000","last_became_peered":"2020-02-05T16:59:12.108517+0000","last_unstale":"2020-02-05T16:59:12.109411+0000","last_undegraded":"2020-02-05T16:59:12.109411+0000","last_fullsized":"2020-02-05T16:59:12.109411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.105907+0000","last_change":"2020-02-05T16:59:08.724708+0000","last_active":"2020-02-05T16:59:12.105907+0000","last_peered":"2020-02-05T16:59:12.105907+0000","last_clean":"2020-02-05T16:59:12.105907+0000","last_became_active":"2020-02-05T16:59:08.723327+0000","last_became_peered":"2020-02-05T16:59:08.723327+0000","last_unstale":"2020-02-05T16:59:12.105907+0000","last_undegraded":"2020-02-05T16:59:12.105907+0000","last_fullsized":"2020-02-05T16:59:12.105907+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"21'1","reported_seq":"11","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:14.651542+0000","last_change":"2020-02-05T16:59:11.613922+0000","last_active":"2020-02-05T16:59:14.651542+0000","last_peered":"2020-02-05T16:59:14.651542+0000","last_clean":"2020-02-05T16:59:14.651542+0000","last_became_active":"2020-02-05T16:59:11.613550+0000","last_became_peered":"2020-02-05T16:59:11.613550+0000","last_unstale":"2020-02-05T16:59:14.651542+0000","last_undegraded":"2020-02-05T16:59:14.651542+0000","last_fullsized":"2020-02-05T16:59:14.651542+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.111843+0000","last_change":"2020-02-05T16:59:12.111451+0000","last_active":"2020-02-05T16:59:12.111843+0000","last_peered":"2020-02-05T16:59:12.111843+0000","last_clean":"2020-02-05T16:59:12.111843+0000","last_became_active":"2020-02-05T16:59:12.110786+0000","last_became_peered":"2020-02-05T16:59:12.110786+0000","last_unstale":"2020-02-05T16:59:12.111843+0000","last_undegraded":"2020-02-05T16:59:12.111843+0000","last_fullsized":"2020-02-05T16:59:12.111843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234783+0000","last_change":"2020-02-05T16:59:08.725966+0000","last_active":"2020-02-05T16:59:12.234783+0000","last_peered":"2020-02-05T16:59:12.234783+0000","last_clean":"2020-02-05T16:59:12.234783+0000","last_became_active":"2020-02-05T16:59:08.723246+0000","last_became_peered":"2020-02-05T16:59:08.723246+0000","last_unstale":"2020-02-05T16:59:12.234783+0000","last_undegraded":"2020-02-05T16:59:12.234783+0000","last_fullsized":"2020-02-05T16:59:12.234783+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"21'6","reported_seq":"16","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.148626+0000","last_change":"2020-02-05T16:59:11.612206+0000","last_active":"2020-02-05T16:59:16.148626+0000","last_peered":"2020-02-05T16:59:16.148626+0000","last_clean":"2020-02-05T16:59:16.148626+0000","last_became_active":"2020-02-05T16:59:11.611539+0000","last_became_peered":"2020-02-05T16:59:11.611539+0000","last_unstale":"2020-02-05T16:59:16.148626+0000","last_undegraded":"2020-02-05T16:59:16.148626+0000","last_fullsized":"2020-02-05T16:59:16.148626+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"9","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.108963+0000","last_change":"2020-02-05T16:59:12.108437+0000","last_active":"2020-02-05T16:59:12.108963+0000","last_peered":"2020-02-05T16:59:12.108963+0000","last_clean":"2020-02-05T16:59:12.108963+0000","last_became_active":"2020-02-05T16:59:12.107863+0000","last_became_peered":"2020-02-05T16:59:12.107863+0000","last_unstale":"2020-02-05T16:59:12.108963+0000","last_undegraded":"2020-02-05T16:59:12.108963+0000","last_fullsized":"2020-02-05T16:59:12.108963+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:11.589285+0000","last_clean_scrub_stamp":"2020-02-05T16:59:11.589285+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"12","reported_epoch":"20","state":"active+clean","last_fresh":"2020-02-05T16:59:12.234841+0000","last_change":"2020-02-05T16:59:08.579318+0000","last_active":"2020-02-05T16:59:12.234841+0000","last_peered":"2020-02-05T16:59:12.234841+0000","last_clean":"2020-02-05T16:59:12.234841+0000","last_became_active":"2020-02-05T16:59:08.578888+0000","last_became_peered":"2020-02-05T16:59:08.578888+0000","last_unstale":"2020-02-05T16:59:12.234841+0000","last_undegraded":"2020-02-05T16:59:12.234841+0000","last_fullsized":"2020-02-05T16:59:12.234841+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:07.543337+0000","last_clean_scrub_stamp":"2020-02-05T16:59:07.543337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"21'8","reported_seq":"18","reported_epoch":"21","state":"active+clean","last_fresh":"2020-02-05T16:59:16.150618+0000","last_change":"2020-02-05T16:59:11.613302+0000","last_active":"2020-02-05T16:59:16.150618+0000","last_peered":"2020-02-05T16:59:16.150618+0000","last_clean":"2020-02-05T16:59:16.150618+0000","last_became_active":"2020-02-05T16:59:11.612808+0000","last_became_peered":"2020-02-05T16:59:11.612808+0000","last_unstale":"2020-02-05T16:59:16.150618+0000","last_undegraded":"2020-02-05T16:59:16.150618+0000","last_fullsized":"2020-02-05T16:59:16.150618+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2020-02-05T16:59:10.576161+0000","last_clean_scrub_stamp":"2020-02-05T16:59:10.576161+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":16,"acting":16,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048812,"kb_used_data":228,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323028,"statfs":{"total":96636764160,"available":95562780672,"internally_reserved":1073741824,"allocated":233472,"data_stored":76121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048804,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323036,"statfs":{"total":96636764160,"available":95562788864,"internally_reserved":1073741824,"allocated":225280,"data_stored":75961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048808,"kb_used_data":224,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323032,"statfs":{"total":96636764160,"available":95562784768,"internally_reserved":1073741824,"allocated":229376,"data_stored":76921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048816,"kb_used_data":232,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323024,"statfs":{"total":96636764160,"available":95562776576,"internally_reserved":1073741824,"allocated":237568,"data_stored":77081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":75429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":75395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048796,"kb_used_data":212,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323044,"statfs":{"total":96636764160,"available":95562797056,"internally_reserved":1073741824,"allocated":217088,"data_stored":75395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048800,"kb_used_data":216,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323040,"statfs":{"total":96636764160,"available":95562792960,"internally_reserved":1073741824,"allocated":221184,"data_stored":75429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}]}} 2020-02-05T16:59:24.760 INFO:tasks.ceph.ceph_manager.ceph:clean! 2020-02-05T16:59:24.761 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2020-02-05T16:59:24.761 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2020-02-05T16:59:24.761 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2020-02-05T16:59:25.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.982+0000 7f3d297b0700 1 Processor -- start 2020-02-05T16:59:25.002 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.982+0000 7f3d297b0700 1 -- start start 2020-02-05T16:59:25.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d297b0700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f3d240ace50 0x7f3d240aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d297b0700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d24145c20 0x7f3d24146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d297b0700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d240aa980 con 0x7f3d240abd50 2020-02-05T16:59:25.003 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d297b0700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d24148c70 con 0x7f3d240ace50 2020-02-05T16:59:25.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d297b0700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d24148f10 con 0x7f3d24145c20 2020-02-05T16:59:25.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d23fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f3d240abd50 0x7f3d241491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46150/0 (socket says 172.21.15.8:46150) 2020-02-05T16:59:25.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.983+0000 7f3d23fff700 1 -- 172.21.15.8:0/2442589172 learned_addr learned my addr 172.21.15.8:0/2442589172 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:25.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.984+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d24145c20 0x7f3d24146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.004 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.984+0000 7f3d28faf700 1 --2- 172.21.15.8:0/2442589172 >> v2:172.21.15.8:3300/0 conn(0x7f3d240ace50 0x7f3d240aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.005 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.984+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253741810 0 0) 0x7f3d100010d0 con 0x7f3d240abd50 2020-02-05T16:59:25.005 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.984+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d08003230 con 0x7f3d240abd50 2020-02-05T16:59:25.005 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.984+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 990143358 0 0) 0x7f3d08003230 con 0x7f3d240abd50 2020-02-05T16:59:25.005 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d24145c20 msgr2=0x7f3d24146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d24145c20 0x7f3d24146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 >> v2:172.21.15.8:3300/0 conn(0x7f3d240ace50 msgr2=0x7f3d240aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 --2- 172.21.15.8:0/2442589172 >> v2:172.21.15.8:3300/0 conn(0x7f3d240ace50 0x7f3d240aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d241454a0 con 0x7f3d240abd50 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7f3d10001980 con 0x7f3d240abd50 2020-02-05T16:59:25.006 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 >> v1:172.21.15.8:6789/0 conn(0x7f3d240abd50 legacy=0x7f3d241491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d08003910 0x7f3d08003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d080045a0 0x7f3d080049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d24148c70 con 0x7f3d08004f20 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d24148f10 con 0x7f3d080045a0 2020-02-05T16:59:25.007 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0804fee0 con 0x7f3d08003910 2020-02-05T16:59:25.008 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.985+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d24145c20 0x7f3d24146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:25.008 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.008 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d080045a0 0x7f3d080049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.008 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d28faf700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d08003910 0x7f3d08003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.008 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 -- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d08003910 msgr2=0x7f3d08003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d08003910 0x7f3d08003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 -- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d080045a0 msgr2=0x7f3d080049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d080045a0 0x7f3d080049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 -- 172.21.15.8:0/2442589172 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d08050180 con 0x7f3d08004f20 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.986+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.009 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.010 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=READY pgs=133 cs=0 l=1 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).ready entity=mon.0 client_cookie=112d71de3867d771 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.010 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.010 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d1c005c30 con 0x7f3d08004f20 2020-02-05T16:59:25.010 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=THROTTLE_DONE pgs=133 cs=0 l=1 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d1c010c50 con 0x7f3d08004f20 2020-02-05T16:59:25.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d237fe700 1 -- 172.21.15.8:0/2442589172 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d1c00cc10 con 0x7f3d08004f20 2020-02-05T16:59:25.011 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d297b0700 1 -- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 msgr2=0x7f3d0804f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.012 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d297b0700 1 --2- 172.21.15.8:0/2442589172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f3d08004f20 0x7f3d0804f800 secure :-1 s=READY pgs=133 cs=0 l=1 rx=0x7f3d1c000b20 tx=0x7f3d1c010250).stop 2020-02-05T16:59:25.012 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.987+0000 7f3d2aa12700 1 -- 172.21.15.8:0/2442589172 reap_dead start 2020-02-05T16:59:25.012 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.988+0000 7f3d297b0700 1 -- 172.21.15.8:0/2442589172 shutdown_connections 2020-02-05T16:59:25.012 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.988+0000 7f3d297b0700 1 -- 172.21.15.8:0/2442589172 >> 172.21.15.8:0/2442589172 conn(0x7f3d24061510 msgr2=0x7f3d240ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:25.012 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.989+0000 7f3d297b0700 1 -- 172.21.15.8:0/2442589172 shutdown_connections 2020-02-05T16:59:25.013 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.989+0000 7f3d297b0700 1 -- 172.21.15.8:0/2442589172 wait complete. 2020-02-05T16:59:25.013 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.993+0000 7f3d297b0700 1 Processor -- start 2020-02-05T16:59:25.013 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.993+0000 7f3d297b0700 1 -- start start 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.994+0000 7f3d297b0700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d2411eca0 0x7f3d2407e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.994+0000 7f3d297b0700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2411dab0 con 0x7f3d24024b70 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.994+0000 7f3d297b0700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d2411ca00 con 0x7f3d2407ed70 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.994+0000 7f3d297b0700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d2411e170 con 0x7f3d2411eca0 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.995+0000 7f3d28faf700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f3d24024b70 0x7f3d2411c200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46162/0 (socket says 172.21.15.8:46162) 2020-02-05T16:59:25.014 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.995+0000 7f3d28faf700 1 -- 172.21.15.8:0/2168791375 learned_addr learned my addr 172.21.15.8:0/2168791375 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:25.015 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.995+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> v2:172.21.15.164:3300/0 conn(0x7f3d2411eca0 0x7f3d2407e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.015 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.995+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041933074 0 0) 0x7f3d2411dab0 con 0x7f3d24024b70 2020-02-05T16:59:25.015 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.995+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00003230 con 0x7f3d24024b70 2020-02-05T16:59:25.015 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4124222214 0 0) 0x7f3d00003230 con 0x7f3d24024b70 2020-02-05T16:59:25.015 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 -- 172.21.15.8:0/2168791375 >> v1:172.21.15.164:6789/0 conn(0x7f3d2407ed70 legacy=0x7f3d24137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 -- 172.21.15.8:0/2168791375 >> v1:172.21.15.8:6789/0 conn(0x7f3d24024b70 legacy=0x7f3d2411c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 -- 172.21.15.8:0/2168791375 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d240a6a00 con 0x7f3d2411eca0 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> v2:172.21.15.164:3300/0 conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> v2:172.21.15.164:3300/0 conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.996+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.016 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d14001490 con 0x7f3d2411eca0 2020-02-05T16:59:25.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d140012a0 con 0x7f3d2411eca0 2020-02-05T16:59:25.017 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d240a3080 con 0x7f3d2411eca0 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d100088e0 con 0x7f3d2411eca0 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.997+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d2411d1c0 con 0x7f3d2411eca0 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.998+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.999+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f3d1000b250 con 0x7f3d2411eca0 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.999+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.018 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.999+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d24056640 con 0x7f3d2411eca0 2020-02-05T16:59:25.019 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.999+0000 7f3d217fa700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 0x7f3d0003c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.019 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:24.999+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f3d240aa980 con 0x7f3d2411eca0 2020-02-05T16:59:25.020 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.000+0000 7f3d28faf700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 0x7f3d0003c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.020 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.001+0000 7f3d28faf700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 0x7f3d0003c390 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.020 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.009+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.020 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.009+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f3d10030960 con 0x7f3d2411eca0 2020-02-05T16:59:25.222 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.216+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3d24066210 con 0x7f3d2411eca0 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.217+0000 7f3d23fff700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.217+0000 7f3d217fa700 1 -- 172.21.15.8:0/2168791375 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7f3d10030bc0 con 0x7f3d2411eca0 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 msgr2=0x7f3d0003c390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 0x7f3d0003c390 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 msgr2=0x7f3d2407e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.223 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f3d14001690 tx=0x7f3d08003820).stop 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 shutdown_connections 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f3d00039e80 0x7f3d0003c390 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 --2- 172.21.15.8:0/2168791375 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d2411eca0 0x7f3d2407e820 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 >> 172.21.15.8:0/2168791375 conn(0x7f3d240abd50 msgr2=0x7f3d2405e920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 shutdown_connections 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.220+0000 7f3d297b0700 1 -- 172.21.15.8:0/2168791375 wait complete. 2020-02-05T16:59:25.224 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:25.241 INFO:teuthology.orchestra.run.smithi008.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2020-02-05T16:59:25.242 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2020-02-05T16:59:25.243 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:25.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.465+0000 7fac5f59e700 1 Processor -- start 2020-02-05T16:59:25.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.465+0000 7fac5f59e700 1 -- start start 2020-02-05T16:59:25.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5f59e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac600abd50 0x7fac600aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5f59e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac60145c20 0x7fac60146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5f59e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac600aa980 con 0x7fac600ace50 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5f59e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac60148c70 con 0x7fac600abd50 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5f59e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac60148f10 con 0x7fac60145c20 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5ed9d700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fac600ace50 0x7fac601491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46170/0 (socket says 172.21.15.8:46170) 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5ed9d700 1 -- 172.21.15.8:0/3963378813 learned_addr learned my addr 172.21.15.8:0/3963378813 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:25.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> v2:172.21.15.164:3300/0 conn(0x7fac600abd50 0x7fac600aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.466+0000 7fac64994700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac60145c20 0x7fac60146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5dd9b700 1 -- 172.21.15.8:0/3963378813 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112041631 0 0) 0x7fac50001450 con 0x7fac600ace50 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5dd9b700 1 -- 172.21.15.8:0/3963378813 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48003230 con 0x7fac600ace50 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5e59c700 1 -- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac60145c20 msgr2=0x7fac60146050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac60145c20 0x7fac60146050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5e59c700 1 -- 172.21.15.8:0/3963378813 >> v1:172.21.15.8:6789/0 conn(0x7fac600ace50 legacy=0x7fac601491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5e59c700 1 -- 172.21.15.8:0/3963378813 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac601454a0 con 0x7fac600abd50 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.467+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> v2:172.21.15.164:3300/0 conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac60148e30 tx=0x7fac5400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> v2:172.21.15.164:3300/0 conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac60148e30 tx=0x7fac5400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fac60148e30 tx=0x7fac5400fe70).ready entity=mon.1 client_cookie=e31686b72b7650f8 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fac60148e30 tx=0x7fac5400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fac60148e30 tx=0x7fac5400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5e59c700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7fac60148e30 tx=0x7fac5400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5dd9b700 1 -- 172.21.15.8:0/3963378813 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac540116e0 con 0x7fac600abd50 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5dd9b700 1 -- 172.21.15.8:0/3963378813 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac54008c50 con 0x7fac600abd50 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5f59e700 1 -- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 msgr2=0x7fac600aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5f59e700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7fac60148e30 tx=0x7fac5400fe70).stop 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.468+0000 7fac5dd9b700 1 -- 172.21.15.8:0/3963378813 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac54011cc0 con 0x7fac600abd50 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 -- 172.21.15.8:0/3963378813 shutdown_connections 2020-02-05T16:59:25.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac60145c20 0x7fac60146050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 --2- 172.21.15.8:0/3963378813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac600abd50 0x7fac600aa430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 -- 172.21.15.8:0/3963378813 >> 172.21.15.8:0/3963378813 conn(0x7fac60061510 msgr2=0x7fac600ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:25.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 -- 172.21.15.8:0/3963378813 shutdown_connections 2020-02-05T16:59:25.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.470+0000 7fac5f59e700 1 -- 172.21.15.8:0/3963378813 wait complete. 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.472+0000 7fac5f59e700 1 Processor -- start 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.472+0000 7fac5f59e700 1 -- start start 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.473+0000 7fac5f59e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.473+0000 7fac5f59e700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac6012fb20 0x7fac60147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.473+0000 7fac5f59e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac601479c0 con 0x7fac600a7140 2020-02-05T16:59:25.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.473+0000 7fac5f59e700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac60146c80 con 0x7fac6012fb20 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.473+0000 7fac5f59e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac60139af0 con 0x7fac600a7570 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.474+0000 7fac5e59c700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.474+0000 7fac64994700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac6012fb20 0x7fac60147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.474+0000 7fac5e59c700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44846/0 (socket says 172.21.15.8:44846) 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.474+0000 7fac5e59c700 1 -- 172.21.15.8:0/61039814 learned_addr learned my addr 172.21.15.8:0/61039814 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 -- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 msgr2=0x7fac60146f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 -- 172.21.15.8:0/61039814 >> v1:172.21.15.164:6789/0 conn(0x7fac600a7140 legacy=0x7fac6012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 -- 172.21.15.8:0/61039814 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac60139ce0 con 0x7fac6012fb20 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac5e59c700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> v2:172.21.15.164:3300/0 conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> v2:172.21.15.164:3300/0 conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.475+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.485 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac58007160 con 0x7fac6012fb20 2020-02-05T16:59:25.488 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58005ab0 con 0x7fac6012fb20 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac601323d0 con 0x7fac6012fb20 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac580124f0 con 0x7fac6012fb20 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.476+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac601328f0 con 0x7fac6012fb20 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.477+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.478+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.489 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.478+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fac580126f0 con 0x7fac6012fb20 2020-02-05T16:59:25.490 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.478+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac60056640 con 0x7fac6012fb20 2020-02-05T16:59:25.490 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.479+0000 7fac4f7fe700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 0x7fac4403c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.490 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.479+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fac58049a20 con 0x7fac6012fb20 2020-02-05T16:59:25.490 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.480+0000 7fac5ed9d700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 0x7fac4403c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.490 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.480+0000 7fac5ed9d700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 0x7fac4403c020 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.489+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.489+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fac5800d0a0 con 0x7fac6012fb20 2020-02-05T16:59:25.650 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.648+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac60066210 con 0x7fac6012fb20 2020-02-05T16:59:25.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.651+0000 7fac64994700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.653 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.651+0000 7fac4f7fe700 1 -- 172.21.15.8:0/61039814 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7fac58032380 con 0x7fac6012fb20 2020-02-05T16:59:25.653 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:25.654 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 msgr2=0x7fac4403c020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 0x7fac4403c020 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 msgr2=0x7fac60147470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7fac58002f30 tx=0x7fac58010c40).stop 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 shutdown_connections 2020-02-05T16:59:25.657 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac6012fb20 0x7fac60147470 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fac44039b10 0x7fac4403c020 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 --2- 172.21.15.8:0/61039814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac600a7570 0x7fac60146f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 >> 172.21.15.8:0/61039814 conn(0x7fac60024fe0 msgr2=0x7fac60126080 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:25.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 shutdown_connections 2020-02-05T16:59:25.658 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.654+0000 7fac5f59e700 1 -- 172.21.15.8:0/61039814 wait complete. 2020-02-05T16:59:25.678 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:06.700677+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:10.361501+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:11.053750+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-02-05T16:59:25.679 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-02-05T16:59:25.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.907+0000 7f0f9fdbb700 1 Processor -- start 2020-02-05T16:59:25.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.907+0000 7f0f9fdbb700 1 -- start start 2020-02-05T16:59:25.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.908+0000 7f0f9fdbb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0f9808fff0 0x7f0f9808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.908+0000 7f0f9fdbb700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.908+0000 7f0f9fdbb700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f980aa3e0 con 0x7f0f980a7990 2020-02-05T16:59:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.908+0000 7f0f9fdbb700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f98090d20 con 0x7f0f9808fff0 2020-02-05T16:59:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.908+0000 7f0f9fdbb700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98090650 con 0x7f0f980908f0 2020-02-05T16:59:25.921 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.909+0000 7f0f9db57700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.909+0000 7f0f9db57700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.8:44852/0 (socket says 172.21.15.8:44852) 2020-02-05T16:59:25.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.909+0000 7f0f9e358700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0f9808fff0 0x7f0f9808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.909+0000 7f0f9db57700 1 -- 172.21.15.8:0/1740224054 learned_addr learned my addr 172.21.15.8:0/1740224054 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:25.924 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.909+0000 7f0f9eb59700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f0f980a7990 0x7f0f980a7dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46188/0 (socket says 172.21.15.8:46188) 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9d356700 1 -- 172.21.15.8:0/1740224054 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917586549 0 0) 0x7f0f90001450 con 0x7f0f980a7990 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9d356700 1 -- 172.21.15.8:0/1740224054 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80003230 con 0x7f0f980a7990 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 -- 172.21.15.8:0/1740224054 >> v2:172.21.15.8:3300/0 conn(0x7f0f9808fff0 msgr2=0x7f0f9808f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> v2:172.21.15.8:3300/0 conn(0x7f0f9808fff0 0x7f0f9808f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 -- 172.21.15.8:0/1740224054 >> v1:172.21.15.8:6789/0 conn(0x7f0f980a7990 legacy=0x7f0f980a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.925 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 -- 172.21.15.8:0/1740224054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980aab90 con 0x7f0f980908f0 2020-02-05T16:59:25.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f94008910 tx=0x7f0f9400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f94008910 tx=0x7f0f9400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.927 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.910+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f0f94008910 tx=0x7f0f9400ff30).ready entity=mon.2 client_cookie=d6f63dc4a4cd16ef server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f0f94008910 tx=0x7f0f9400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f0f94008910 tx=0x7f0f9400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9d356700 1 -- 172.21.15.8:0/1740224054 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f940117a0 con 0x7f0f980908f0 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9db57700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f0f94008910 tx=0x7f0f9400ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9d356700 1 -- 172.21.15.8:0/1740224054 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f94008d30 con 0x7f0f980908f0 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9d356700 1 -- 172.21.15.8:0/1740224054 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f94011d80 con 0x7f0f980908f0 2020-02-05T16:59:25.928 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 msgr2=0x7f0f980a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.929 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.911+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f0f94008910 tx=0x7f0f9400ff30).stop 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/1740224054 shutdown_connections 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/1740224054 >> v2:172.21.15.8:3300/0 conn(0x7f0f9808fff0 0x7f0f9808f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/1740224054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f980908f0 0x7f0f980a9e90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/1740224054 >> 172.21.15.8:0/1740224054 conn(0x7f0f980600f0 msgr2=0x7f0f9805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/1740224054 shutdown_connections 2020-02-05T16:59:25.931 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.913+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/1740224054 wait complete. 2020-02-05T16:59:25.944 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.917+0000 7f0f9fdbb700 1 Processor -- start 2020-02-05T16:59:25.944 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.917+0000 7f0f9fdbb700 1 -- start start 2020-02-05T16:59:25.944 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9807da90 0x7f0f980a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.944 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f9807bb10 0x7f0f9807bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0f980a7ed0 0x7f0f980a8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9807f620 con 0x7f0f980a7ed0 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f980a8850 con 0x7f0f9807bb10 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9fdbb700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f98138560 con 0x7f0f9807da90 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.919+0000 7f0f9eb59700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0f980a7ed0 0x7f0f980a8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.945 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9e358700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9807da90 0x7f0f980a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9db57700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f9807bb10 0x7f0f9807bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f0f980a7ed0 0x7f0f980a8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44812/0 (socket says 172.21.15.8:44812) 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 -- 172.21.15.8:0/983899019 learned_addr learned my addr 172.21.15.8:0/983899019 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 -- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9807da90 msgr2=0x7f0f980a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9807da90 0x7f0f980a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 -- 172.21.15.8:0/983899019 >> v2:172.21.15.164:3300/0 conn(0x7f0f9807bb10 msgr2=0x7f0f9807bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:25.946 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> v2:172.21.15.164:3300/0 conn(0x7f0f9807bb10 0x7f0f9807bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.920+0000 7f0f9eb59700 1 -- 172.21.15.8:0/983899019 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f980a7810 con 0x7f0f980a7ed0 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> v2:172.21.15.8:3300/0 conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> v2:172.21.15.8:3300/0 conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f90005c80 con 0x7f0f980a7ed0 2020-02-05T16:59:25.947 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.948 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f900058a0 con 0x7f0f980a7ed0 2020-02-05T16:59:25.948 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f90012a90 con 0x7f0f980a7ed0 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f980a6fc0 con 0x7f0f980a7ed0 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.921+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f980aaf40 con 0x7f0f980a7ed0 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.922+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.923+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f98056640 con 0x7f0f980a7ed0 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.923+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f0f900112c0 con 0x7f0f980a7ed0 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.924+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.949 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.924+0000 7f0f8effd700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 0x7f0f7803bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:25.951 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.924+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f0f980aa3e0 con 0x7f0f980a7ed0 2020-02-05T16:59:25.951 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.925+0000 7f0f9e358700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 0x7f0f7803bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:25.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.925+0000 7f0f9e358700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 0x7f0f7803bea0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:25.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.930+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:25.952 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:25.930+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f0f90031860 con 0x7f0f980a7ed0 2020-02-05T16:59:26.092 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.087+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f0f98068f50 con 0x7f0f980a7ed0 2020-02-05T16:59:26.092 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.088+0000 7f0f9eb59700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.092 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.088+0000 7f0f8effd700 1 -- 172.21.15.8:0/983899019 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f0f90012d60 con 0x7f0f980a7ed0 2020-02-05T16:59:26.092 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:26.094 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.091+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 msgr2=0x7f0f7803bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.094 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.091+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 0x7f0f7803bea0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.095 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.091+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 msgr2=0x7f0f980a8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.095 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.091+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f0f980a5e30 tx=0x7f0f90010cf0).stop 2020-02-05T16:59:26.095 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 shutdown_connections 2020-02-05T16:59:26.095 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f0f78039990 0x7f0f7803bea0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f0f980a7ed0 0x7f0f980a8300 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f9807da90 0x7f0f980a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 --2- 172.21.15.8:0/983899019 >> v2:172.21.15.164:3300/0 conn(0x7f0f9807bb10 0x7f0f9807bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 >> 172.21.15.8:0/983899019 conn(0x7f0f980a3540 msgr2=0x7f0f980a2640 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 shutdown_connections 2020-02-05T16:59:26.096 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.092+0000 7f0f9fdbb700 1 -- 172.21.15.8:0/983899019 wait complete. 2020-02-05T16:59:26.115 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-02-05T16:59:26.343 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.333+0000 7fa30a53a700 1 Processor -- start 2020-02-05T16:59:26.346 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.333+0000 7fa30a53a700 1 -- start start 2020-02-05T16:59:26.346 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.334+0000 7fa30a53a700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa30408fff0 0x7fa30408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:26.346 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.334+0000 7fa30a53a700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3040aab90 con 0x7fa3040a9e90 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.334+0000 7fa30a53a700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30405b340 con 0x7fa3040908f0 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.334+0000 7fa30a53a700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa304090d20 con 0x7fa30408fff0 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.335+0000 7fa3092d8700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fa3040a9e90 0x7fa3040a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46202/0 (socket says 172.21.15.8:46202) 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.335+0000 7fa308ad7700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa30408fff0 0x7fa30408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.335+0000 7fa3092d8700 1 -- 172.21.15.8:0/3362934912 learned_addr learned my addr 172.21.15.8:0/3362934912 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926750815 0 0) 0x7fa3040aab90 con 0x7fa3040a9e90 2020-02-05T16:59:26.347 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003230 con 0x7fa3040a9e90 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40855343 0 0) 0x7fa30405b340 con 0x7fa3040908f0 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 -- 172.21.15.8:0/3362934912 >> v1:172.21.15.164:6789/0 conn(0x7fa3040908f0 legacy=0x7fa3040a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 -- 172.21.15.8:0/3362934912 >> v1:172.21.15.8:6789/0 conn(0x7fa3040a9e90 legacy=0x7fa3040a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 -- 172.21.15.8:0/3362934912 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040aa2c0 con 0x7fa30408fff0 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> v2:172.21.15.164:3300/0 conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> v2:172.21.15.164:3300/0 conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.348 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.336+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).ready entity=mon.1 client_cookie=c42caf6292ad73b0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f40116e0 con 0x7fa30408fff0 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa308ad7700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f4008c50 con 0x7fa30408fff0 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa3037fe700 1 -- 172.21.15.8:0/3362934912 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f4011cc0 con 0x7fa30408fff0 2020-02-05T16:59:26.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa30a53a700 1 -- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 msgr2=0x7fa30408f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.337+0000 7fa30a53a700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f400fe70).stop 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.339+0000 7fa30a53a700 1 -- 172.21.15.8:0/3362934912 shutdown_connections 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.339+0000 7fa30a53a700 1 --2- 172.21.15.8:0/3362934912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa30408fff0 0x7fa30408f240 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.339+0000 7fa30a53a700 1 -- 172.21.15.8:0/3362934912 >> 172.21.15.8:0/3362934912 conn(0x7fa3040600f0 msgr2=0x7fa30405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.339+0000 7fa30a53a700 1 -- 172.21.15.8:0/3362934912 shutdown_connections 2020-02-05T16:59:26.350 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.339+0000 7fa30a53a700 1 -- 172.21.15.8:0/3362934912 wait complete. 2020-02-05T16:59:26.354 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.341+0000 7fa30a53a700 1 Processor -- start 2020-02-05T16:59:26.354 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.341+0000 7fa30a53a700 1 -- start start 2020-02-05T16:59:26.354 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa30a53a700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa3040a7a20 0x7fa30407c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.354 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa30a53a700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3040a0830 0x7fa30409f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.354 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa30a53a700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa30409fb70 con 0x7fa3040a0400 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa30a53a700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa304122620 con 0x7fa3040a7a20 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa30a53a700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3041228c0 con 0x7fa3040a0830 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa303fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fa3040a0400 0x7fa30409ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46206/0 (socket says 172.21.15.8:46206) 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.342+0000 7fa303fff700 1 -- 172.21.15.8:0/1442395714 learned_addr learned my addr 172.21.15.8:0/1442395714 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.343+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> v2:172.21.15.164:3300/0 conn(0x7fa3040a7a20 0x7fa30407c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.343+0000 7fa3092d8700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3040a0830 0x7fa30409f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.355 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.343+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3918390739 0 0) 0x7fa30409fb70 con 0x7fa3040a0400 2020-02-05T16:59:26.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 -- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3040a0830 msgr2=0x7fa30409f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3040a0830 0x7fa30409f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 -- 172.21.15.8:0/1442395714 >> v1:172.21.15.8:6789/0 conn(0x7fa3040a0400 legacy=0x7fa30409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 -- 172.21.15.8:0/1442395714 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3040a0c60 con 0x7fa3040a7a20 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> v2:172.21.15.164:3300/0 conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> v2:172.21.15.164:3300/0 conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.344+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f4007150 con 0x7fa3040a7a20 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f40118e0 con 0x7fa3040a7a20 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa304122e00 con 0x7fa3040a7a20 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f40058b0 con 0x7fa3040a7a20 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.345+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30407cab0 con 0x7fa3040a7a20 2020-02-05T16:59:26.358 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.346+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.347+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa304122410 con 0x7fa3040a7a20 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.347+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fa2f4005b80 con 0x7fa3040a7a20 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.347+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.348+0000 7fa3017fa700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 0x7fa2e403bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.348+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fa2f4049b20 con 0x7fa3040a7a20 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.348+0000 7fa303fff700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 0x7fa2e403bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.359 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.349+0000 7fa303fff700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 0x7fa2e403bdf0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.356+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.360 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.356+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fa3040a0c60 con 0x7fa3040a7a20 2020-02-05T16:59:26.517 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.514+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa304056640 con 0x7fa3040a7a20 2020-02-05T16:59:26.517 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.515+0000 7fa308ad7700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=THROTTLE_DONE pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.518 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.515+0000 7fa3017fa700 1 -- 172.21.15.8:0/1442395714 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7fa2f40324a0 con 0x7fa3040a7a20 2020-02-05T16:59:26.518 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:26.520 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 msgr2=0x7fa2e403bdf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.520 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 0x7fa2e403bdf0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.520 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 msgr2=0x7fa30407c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.520 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 secure :-1 s=READY pgs=117 cs=0 l=1 rx=0x7fa2f4008910 tx=0x7fa2f4011800).stop 2020-02-05T16:59:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 shutdown_connections 2020-02-05T16:59:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fa2e40398e0 0x7fa2e403bdf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa3040a0830 0x7fa30409f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 --2- 172.21.15.8:0/1442395714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa3040a7a20 0x7fa30407c560 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.521 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.517+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 >> 172.21.15.8:0/1442395714 conn(0x7fa30405b0c0 msgr2=0x7fa304065450 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:26.522 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.518+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 shutdown_connections 2020-02-05T16:59:26.522 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.518+0000 7fa30a53a700 1 -- 172.21.15.8:0/1442395714 wait complete. 2020-02-05T16:59:26.539 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-02-05T16:59:26.780 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.771+0000 7f833b59e700 1 Processor -- start 2020-02-05T16:59:26.780 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.771+0000 7f833b59e700 1 -- start start 2020-02-05T16:59:26.780 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833b59e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0abd50 0x7f833c0aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:26.780 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833b59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c145c20 0x7f833c146050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833b59e700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c0aa980 con 0x7f833c0ace50 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833b59e700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c148c70 con 0x7f833c145c20 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833b59e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c148f10 con 0x7f833c0abd50 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f8340cfb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c145c20 0x7f833c146050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c145c20 0x7f833c146050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44834/0 (socket says 172.21.15.8:44834) 2020-02-05T16:59:26.781 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 -- 172.21.15.8:0/1563610923 learned_addr learned my addr 172.21.15.8:0/1563610923 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:26.782 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.773+0000 7f833ad9d700 1 --1- 172.21.15.8:0/1563610923 >> v1:172.21.15.8:6789/0 conn(0x7f833c0ace50 0x7f833c1491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46214/0 (socket says 172.21.15.8:46214) 2020-02-05T16:59:26.782 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f833a59c700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0abd50 0x7f833c0aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.782 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 -- 172.21.15.8:0/1563610923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0abd50 msgr2=0x7f833c0aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:26.784 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0abd50 0x7f833c0aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:26.784 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 -- 172.21.15.8:0/1563610923 >> v1:172.21.15.8:6789/0 conn(0x7f833c0ace50 legacy=0x7f833c1491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.784 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 -- 172.21.15.8:0/1563610923 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c1454a0 con 0x7f833c145c20 2020-02-05T16:59:26.785 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> v2:172.21.15.8:3300/0 conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f833c061e20 tx=0x7f83340102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.785 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> v2:172.21.15.8:3300/0 conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f833c061e20 tx=0x7f83340102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.785 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.774+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f83340102b0).ready entity=mon.0 client_cookie=2d6d219ff9c14a25 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.785 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f83340102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8339d9b700 1 -- 172.21.15.8:0/1563610923 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f833400a350 con 0x7f833c145c20 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f83340102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8339d9b700 1 -- 172.21.15.8:0/1563610923 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8334011c30 con 0x7f833c145c20 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8340cfb700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f83340102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f8339d9b700 1 -- 172.21.15.8:0/1563610923 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f833400a930 con 0x7f833c145c20 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f833b59e700 1 -- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 msgr2=0x7f833c146050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.775+0000 7f833b59e700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f83340102b0).stop 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 -- 172.21.15.8:0/1563610923 shutdown_connections 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c145c20 0x7f833c146050 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 --2- 172.21.15.8:0/1563610923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0abd50 0x7f833c0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 -- 172.21.15.8:0/1563610923 >> 172.21.15.8:0/1563610923 conn(0x7f833c061510 msgr2=0x7f833c0ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 -- 172.21.15.8:0/1563610923 shutdown_connections 2020-02-05T16:59:26.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.776+0000 7f833b59e700 1 -- 172.21.15.8:0/1563610923 wait complete. 2020-02-05T16:59:26.794 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.779+0000 7f833b59e700 1 Processor -- start 2020-02-05T16:59:26.795 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.779+0000 7f833b59e700 1 -- start start 2020-02-05T16:59:26.795 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f833b59e700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 0x7f833c0a6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.795 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f833b59e700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c07cb10 0x7f833c07cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.795 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f833b59e700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833c07d490 con 0x7f833c0a4710 2020-02-05T16:59:26.795 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f833b59e700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833c0a3240 con 0x7f833c07cb10 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f833b59e700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833c0a4be0 con 0x7f833c0a2590 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f8340cfb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c07cb10 0x7f833c07cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.780+0000 7f8340cfb700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f833c07cb10 0x7f833c07cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44838/0 (socket says 172.21.15.8:44838) 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 -- 172.21.15.8:0/2091138915 learned_addr learned my addr 172.21.15.8:0/2091138915 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f833a59c700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 0x7f833c0a6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 -- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 msgr2=0x7f833c0a6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.796 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 0x7f833c0a6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 -- 172.21.15.8:0/2091138915 >> v1:172.21.15.164:6789/0 conn(0x7f833c0a4710 legacy=0x7f833c12ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 -- 172.21.15.8:0/2091138915 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833c0a53c0 con 0x7f833c07cb10 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f833a59c700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 0x7f833c0a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> v2:172.21.15.8:3300/0 conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> v2:172.21.15.8:3300/0 conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.781+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.797 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8334005c30 con 0x7f833c07cb10 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f833400a550 con 0x7f833c07cb10 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833c12f3b0 con 0x7f833c07cb10 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83340114a0 con 0x7f833c07cb10 2020-02-05T16:59:26.798 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.782+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833c1395c0 con 0x7f833c07cb10 2020-02-05T16:59:26.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.783+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.799 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.783+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f8334010740 con 0x7f833c07cb10 2020-02-05T16:59:26.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.783+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.784+0000 7f832b7fe700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 0x7f832003bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:26.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.784+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f8334049cf0 con 0x7f833c07cb10 2020-02-05T16:59:26.800 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.784+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833c056640 con 0x7f833c07cb10 2020-02-05T16:59:26.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.785+0000 7f833ad9d700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 0x7f832003bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:26.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.785+0000 7f833ad9d700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 0x7f832003bfa0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:26.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.791+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.801 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.791+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f8334031aa0 con 0x7f833c07cb10 2020-02-05T16:59:26.950 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.948+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f833c066210 con 0x7f833c07cb10 2020-02-05T16:59:26.951 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.949+0000 7f8340cfb700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:26.951 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.949+0000 7f832b7fe700 1 -- 172.21.15.8:0/2091138915 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8334010a50 con 0x7f833c07cb10 2020-02-05T16:59:26.952 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:26.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 msgr2=0x7f832003bfa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 0x7f832003bfa0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 msgr2=0x7f833c07cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:26.953 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f833c061e20 tx=0x7f8334007f90).stop 2020-02-05T16:59:26.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 shutdown_connections 2020-02-05T16:59:26.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f833c07cb10 0x7f833c07cf40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.954 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8320039a90 0x7f832003bfa0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 --2- 172.21.15.8:0/2091138915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f833c0a2590 0x7f833c0a6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:26.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 >> 172.21.15.8:0/2091138915 conn(0x7f833c05f4d0 msgr2=0x7f833c05e920 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:26.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 shutdown_connections 2020-02-05T16:59:26.955 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:26.951+0000 7f833b59e700 1 -- 172.21.15.8:0/2091138915 wait complete. 2020-02-05T16:59:26.971 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.189+0000 7f63c169c700 1 Processor -- start 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.189+0000 7f63c169c700 1 -- start start 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63c169c700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63bc0ace50 0x7f63bc0aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63c169c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63bc0abd50 0x7f63bc1491b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63c169c700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0aa980 con 0x7f63bc145c20 2020-02-05T16:59:27.200 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63c169c700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc148c70 con 0x7f63bc0abd50 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63c169c700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc148f10 con 0x7f63bc0ace50 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.190+0000 7f63bbfff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63bc0abd50 0x7f63bc1491b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63bc0abd50 0x7f63bc1491b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.8:60400/0 (socket says 172.21.15.8:60400) 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 -- 172.21.15.8:0/392980309 learned_addr learned my addr 172.21.15.8:0/392980309 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63bc0ace50 0x7f63bc0aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 -- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63bc0ace50 msgr2=0x7f63bc0aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:27.201 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63bc0ace50 0x7f63bc0aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 -- 172.21.15.8:0/392980309 >> v1:172.21.15.164:6789/0 conn(0x7f63bc145c20 legacy=0x7f63bc13f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 -- 172.21.15.8:0/392980309 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc1454a0 con 0x7f63bc0abd50 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.191+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> v2:172.21.15.164:3300/0 conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f63bc061e20 tx=0x7f63b000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> v2:172.21.15.164:3300/0 conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f63bc061e20 tx=0x7f63b000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f63bc061e20 tx=0x7f63b000ff30).ready entity=mon.1 client_cookie=82288c4ddf8687f4 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f63bc061e20 tx=0x7f63b000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f63bc061e20 tx=0x7f63b000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.202 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bbfff700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f63bc061e20 tx=0x7f63b000ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bb7fe700 1 -- 172.21.15.8:0/392980309 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b0011760 con 0x7f63bc0abd50 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.192+0000 7f63bb7fe700 1 -- 172.21.15.8:0/392980309 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b0008d30 con 0x7f63bc0abd50 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.193+0000 7f63bb7fe700 1 -- 172.21.15.8:0/392980309 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b0011d40 con 0x7f63bc0abd50 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.193+0000 7f63c169c700 1 -- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 msgr2=0x7f63bc1491b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.193+0000 7f63c169c700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f63bc061e20 tx=0x7f63b000ff30).stop 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.194+0000 7f63c169c700 1 -- 172.21.15.8:0/392980309 shutdown_connections 2020-02-05T16:59:27.203 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.194+0000 7f63c169c700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63bc0ace50 0x7f63bc0aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:27.204 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.194+0000 7f63c169c700 1 --2- 172.21.15.8:0/392980309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc0abd50 0x7f63bc1491b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:27.204 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.194+0000 7f63c169c700 1 -- 172.21.15.8:0/392980309 >> 172.21.15.8:0/392980309 conn(0x7f63bc061510 msgr2=0x7f63bc0ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:27.205 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.194+0000 7f63c169c700 1 -- 172.21.15.8:0/392980309 shutdown_connections 2020-02-05T16:59:27.205 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.195+0000 7f63c169c700 1 -- 172.21.15.8:0/392980309 wait complete. 2020-02-05T16:59:27.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.198+0000 7f63c169c700 1 Processor -- start 2020-02-05T16:59:27.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.199+0000 7f63c169c700 1 -- start start 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.200+0000 7f63c169c700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63bc07c020 0x7f63bc146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.200+0000 7f63c169c700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc143c40 con 0x7f63bc1471f0 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.201+0000 7f63c169c700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc11de50 con 0x7f63bc143810 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.201+0000 7f63c169c700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc146cb0 con 0x7f63bc07c020 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.201+0000 7f63c28fe700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f63bc1471f0 0x7f63bc024ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46236/0 (socket says 172.21.15.8:46236) 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.201+0000 7f63c28fe700 1 -- 172.21.15.8:0/3259399469 learned_addr learned my addr 172.21.15.8:0/3259399469 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:27.208 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.201+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> v2:172.21.15.164:3300/0 conn(0x7f63bc07c020 0x7f63bc146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745392404 0 0) 0x7f63bc143c40 con 0x7f63bc1471f0 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f639c003230 con 0x7f63bc1471f0 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320582283 0 0) 0x7f63bc11de50 con 0x7f63bc143810 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63c0e9b700 1 -- 172.21.15.8:0/3259399469 >> v1:172.21.15.164:6789/0 conn(0x7f63bc143810 legacy=0x7f63bc0247f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63c0e9b700 1 -- 172.21.15.8:0/3259399469 >> v1:172.21.15.8:6789/0 conn(0x7f63bc1471f0 legacy=0x7f63bc024ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.202+0000 7f63c0e9b700 1 -- 172.21.15.8:0/3259399469 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc147620 con 0x7f63bc07c020 2020-02-05T16:59:27.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.203+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> v2:172.21.15.164:3300/0 conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.203+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> v2:172.21.15.164:3300/0 conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.203+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.203+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.203+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63ac012b70 con 0x7f63bc07c020 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc148120 con 0x7f63bc07c020 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac012790 con 0x7f63bc07c020 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63ac0058b0 con 0x7f63bc07c020 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.204+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc138830 con 0x7f63bc07c020 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.205+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.205+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc056640 con 0x7f63bc07c020 2020-02-05T16:59:27.212 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.206+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f63ac005b90 con 0x7f63bc07c020 2020-02-05T16:59:27.212 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.206+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.206+0000 7f63b97fa700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 0x7f639c03c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:27.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.206+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f63ac049b70 con 0x7f63bc07c020 2020-02-05T16:59:27.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.207+0000 7f63bbfff700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 0x7f639c03c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:27.213 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.207+0000 7f63bbfff700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 0x7f639c03c370 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:27.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.216+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.216+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f63ac00d0a0 con 0x7f63bc07c020 2020-02-05T16:59:27.423 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.420+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63bc066210 con 0x7f63bc07c020 2020-02-05T16:59:27.423 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.422+0000 7f63c0e9b700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=THROTTLE_DONE pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:27.424 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.422+0000 7f63b97fa700 1 -- 172.21.15.8:0/3259399469 <== mon.1 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f63ac032450 con 0x7f63bc07c020 2020-02-05T16:59:27.424 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:27.424 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4332,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.8:6833/4097051851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":4097051851},{"type":"v1","addr":"172.21.15.8:6833","nonce":4097051851}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-02-05T16:59:12.091946+0000","modified":"2020-02-05T16:59:17.122886+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4336":{"gid":4336,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.164:6835/1530677404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1530677404},{"type":"v1","addr":"172.21.15.164:6835","nonce":1530677404}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-02-05T16:59:27.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.424+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 msgr2=0x7f639c03c370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.424+0000 7f63c169c700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 0x7f639c03c370 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:27.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.424+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 msgr2=0x7f63bc146690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:27.427 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.424+0000 7f63c169c700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 secure :-1 s=READY pgs=122 cs=0 l=1 rx=0x7f63ac002f30 tx=0x7f63ac010c60).stop 2020-02-05T16:59:27.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 shutdown_connections 2020-02-05T16:59:27.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63bc07c020 0x7f63bc146690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:27.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 --2- 172.21.15.8:0/3259399469 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f639c039e60 0x7f639c03c370 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:27.428 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 >> 172.21.15.8:0/3259399469 conn(0x7f63bc11c2f0 msgr2=0x7f63bc11e170 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:27.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 shutdown_connections 2020-02-05T16:59:27.429 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:27.425+0000 7f63c169c700 1 -- 172.21.15.8:0/3259399469 wait complete. 2020-02-05T16:59:27.430 INFO:teuthology.orchestra.run.smithi008.stderr:dumped fsmap epoch 7 2020-02-05T16:59:27.650 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4336}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4336': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4336, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 1530677404, u'type': u'v2', u'addr': u'172.21.15.164:6834'}, {u'nonce': 1530677404, u'type': u'v1', u'addr': u'172.21.15.164:6835'}]}, u'addr': u'172.21.15.164:6835/1530677404'}}, u'fs_name': u'cephfs', u'created': u'2020-02-05T16:59:12.091946+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2020-02-05T16:59:17.122886+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2020-02-05T16:59:27.651 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-02-05T16:59:27.651 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2020-02-05T16:59:27.884 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-02-05T16:59:27.884 INFO:teuthology.orchestra.run.smithi164.stdout: "cluster_fsid": "db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7", 2020-02-05T16:59:27.885 INFO:teuthology.orchestra.run.smithi164.stdout: "whoami": 0, 2020-02-05T16:59:27.885 INFO:teuthology.orchestra.run.smithi164.stdout: "id": 4336, 2020-02-05T16:59:27.885 INFO:teuthology.orchestra.run.smithi164.stdout: "want_state": "up:active", 2020-02-05T16:59:27.885 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "up:active", 2020-02-05T16:59:27.886 INFO:teuthology.orchestra.run.smithi164.stdout: "rank_uptime": 11.753940310999999, 2020-02-05T16:59:27.886 INFO:teuthology.orchestra.run.smithi164.stdout: "mdsmap_epoch": 6, 2020-02-05T16:59:27.886 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_epoch": 21, 2020-02-05T16:59:27.886 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_epoch_barrier": 21, 2020-02-05T16:59:27.887 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12.647418761999999 2020-02-05T16:59:27.887 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-02-05T16:59:27.900 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7", "whoami": 0, "id": 4336, "want_state": "up:active", "state": "up:active", "rank_uptime": 11.753940310999999, "mdsmap_epoch": 6, "osdmap_epoch": 21, "osdmap_epoch_barrier": 21, "uptime": 12.647418761999999 } 2020-02-05T16:59:27.900 INFO:teuthology.run_tasks:Running task ceph-fuse... 2020-02-05T16:59:27.934 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2020-02-05T16:59:27.935 INFO:tasks.ceph_fuse:config is None 2020-02-05T16:59:27.935 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2020-02-05T16:59:27.936 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2020-02-05T16:59:27.936 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2020-02-05T16:59:28.182 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.169+0000 7f8c6de82700 1 Processor -- start 2020-02-05T16:59:28.182 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.169+0000 7f8c6de82700 1 -- start start 2020-02-05T16:59:28.182 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c6de82700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6808fff0 0x7f8c6808f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:28.182 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c6de82700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 0x7f8c680a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c6de82700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c680aa3e0 con 0x7f8c680a7990 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c6de82700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c68090d20 con 0x7f8c6808fff0 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c6de82700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c68090650 con 0x7f8c680908f0 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c67fff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6808fff0 0x7f8c6808f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c67fff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6808fff0 0x7f8c6808f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44858/0 (socket says 172.21.15.8:44858) 2020-02-05T16:59:28.183 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.170+0000 7f8c67fff700 1 -- 172.21.15.8:0/1924077464 learned_addr learned my addr 172.21.15.8:0/1924077464 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c677fe700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 0x7f8c680a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c67fff700 1 -- 172.21.15.8:0/1924077464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 msgr2=0x7f8c680a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 0x7f8c680a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c67fff700 1 -- 172.21.15.8:0/1924077464 >> v1:172.21.15.164:6789/0 conn(0x7f8c680a7990 legacy=0x7f8c680a7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c67fff700 1 -- 172.21.15.8:0/1924077464 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c680aab90 con 0x7f8c6808fff0 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c677fe700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 0x7f8c680a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:28.184 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.171+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> v2:172.21.15.8:3300/0 conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8c58008910 tx=0x7f8c5800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.185 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> v2:172.21.15.8:3300/0 conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8c58008910 tx=0x7f8c5800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.185 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f8c58008910 tx=0x7f8c5800ff30).ready entity=mon.0 client_cookie=3131e55040c53088 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.185 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f8c58008910 tx=0x7f8c5800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.185 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f8c58008910 tx=0x7f8c5800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.185 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c66ffd700 1 -- 172.21.15.8:0/1924077464 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c58011990 con 0x7f8c6808fff0 2020-02-05T16:59:28.191 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f8c58008910 tx=0x7f8c5800ff30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.191 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c66ffd700 1 -- 172.21.15.8:0/1924077464 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c580115b0 con 0x7f8c6808fff0 2020-02-05T16:59:28.191 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c66ffd700 1 -- 172.21.15.8:0/1924077464 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c5800a160 con 0x7f8c6808fff0 2020-02-05T16:59:28.191 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c6de82700 1 -- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 msgr2=0x7f8c6808f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.191 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.172+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f8c58008910 tx=0x7f8c5800ff30).stop 2020-02-05T16:59:28.192 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 -- 172.21.15.8:0/1924077464 shutdown_connections 2020-02-05T16:59:28.192 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6808fff0 0x7f8c6808f240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.192 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1924077464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8c680908f0 0x7f8c680a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:28.192 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 -- 172.21.15.8:0/1924077464 >> 172.21.15.8:0/1924077464 conn(0x7f8c680600f0 msgr2=0x7f8c6805a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 -- 172.21.15.8:0/1924077464 shutdown_connections 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.174+0000 7f8c6de82700 1 -- 172.21.15.8:0/1924077464 wait complete. 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.176+0000 7f8c6de82700 1 Processor -- start 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.176+0000 7f8c6de82700 1 -- start start 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c6de82700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6807ded0 0x7f8c6809dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c6de82700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 0x7f8c6813faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.193 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c6de82700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c68066770 con 0x7f8c6809e130 2020-02-05T16:59:28.195 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c6de82700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680a6f20 con 0x7f8c6807ded0 2020-02-05T16:59:28.195 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c6de82700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c680a71c0 con 0x7f8c6813d6b0 2020-02-05T16:59:28.195 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c67fff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6807ded0 0x7f8c6809dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.195 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c67fff700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8c6807ded0 0x7f8c6809dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44864/0 (socket says 172.21.15.8:44864) 2020-02-05T16:59:28.195 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.177+0000 7f8c67fff700 1 -- 172.21.15.8:0/1296116246 learned_addr learned my addr 172.21.15.8:0/1296116246 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 -- 172.21.15.8:0/1296116246 >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 msgr2=0x7f8c6813faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c6cc20700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 0x7f8c6813faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 0x7f8c6813faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 -- 172.21.15.8:0/1296116246 >> v1:172.21.15.8:6789/0 conn(0x7f8c6809e130 legacy=0x7f8c68065a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 -- 172.21.15.8:0/1296116246 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c6807f620 con 0x7f8c6807ded0 2020-02-05T16:59:28.196 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c6cc20700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 0x7f8c6813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.8:3300/0 conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.178+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.8:3300/0 conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.197 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c5800f760 con 0x7f8c6807ded0 2020-02-05T16:59:28.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c58011b90 con 0x7f8c6807ded0 2020-02-05T16:59:28.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c58007530 con 0x7f8c6807ded0 2020-02-05T16:59:28.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c68066270 con 0x7f8c6807ded0 2020-02-05T16:59:28.207 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.179+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c680a6b00 con 0x7f8c6807ded0 2020-02-05T16:59:28.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.180+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.181+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c68056640 con 0x7f8c6807ded0 2020-02-05T16:59:28.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.181+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f8c580056c0 con 0x7f8c6807ded0 2020-02-05T16:59:28.209 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.181+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.182+0000 7f8c64ff9700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 0x7f8c4803be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.182+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f8c58049dd0 con 0x7f8c6807ded0 2020-02-05T16:59:28.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.182+0000 7f8c677fe700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 0x7f8c4803be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.183+0000 7f8c677fe700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 0x7f8c4803be40 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.210 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.188+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.211 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.188+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f8c58031920 con 0x7f8c6807ded0 2020-02-05T16:59:28.349 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.346+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8c68068f50 con 0x7f8c6807ded0 2020-02-05T16:59:28.351 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.349+0000 7f8c67fff700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.351 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.349+0000 7f8c64ff9700 1 -- 172.21.15.8:0/1296116246 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+14624 (secure 0 0 0) 0x7f8c58031b80 con 0x7f8c6807ded0 2020-02-05T16:59:28.352 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:28.352 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":21,"fsid":"db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7","created":"2020-02-05T16:58:51.539372+0000","modified":"2020-02-05T16:59:13.095481+0000","last_up_change":"2020-02-05T16:59:02.503851+0000","last_in_change":"2020-02-05T16:59:02.503851+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2020-02-05T16:59:06.700677+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2020-02-05T16:59:10.361501+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2020-02-05T16:59:11.053750+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2bc6f3a9-039a-4231-9455-307d8891ced6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6800","nonce":22695},{"type":"v1","addr":"172.21.15.8:6801","nonce":22695}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6802","nonce":22695},{"type":"v1","addr":"172.21.15.8:6803","nonce":22695}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6806","nonce":22695},{"type":"v1","addr":"172.21.15.8:6807","nonce":22695}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6804","nonce":22695},{"type":"v1","addr":"172.21.15.8:6805","nonce":22695}]},"public_addr":"172.21.15.8:6801/22695","cluster_addr":"172.21.15.8:6803/22695","heartbeat_back_addr":"172.21.15.8:6807/22695","heartbeat_front_addr":"172.21.15.8:6805/22695","state":["exists","up"]},{"osd":1,"uuid":"61d816aa-3e9c-45f3-97e1-ca9c5510305f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6808","nonce":22696},{"type":"v1","addr":"172.21.15.8:6809","nonce":22696}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6810","nonce":22696},{"type":"v1","addr":"172.21.15.8:6811","nonce":22696}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6814","nonce":22696},{"type":"v1","addr":"172.21.15.8:6815","nonce":22696}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6812","nonce":22696},{"type":"v1","addr":"172.21.15.8:6813","nonce":22696}]},"public_addr":"172.21.15.8:6809/22696","cluster_addr":"172.21.15.8:6811/22696","heartbeat_back_addr":"172.21.15.8:6815/22696","heartbeat_front_addr":"172.21.15.8:6813/22696","state":["exists","up"]},{"osd":2,"uuid":"3ab2a8b9-2960-436d-9acd-28c9edf793c2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6816","nonce":22697},{"type":"v1","addr":"172.21.15.8:6817","nonce":22697}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6818","nonce":22697},{"type":"v1","addr":"172.21.15.8:6820","nonce":22697}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6827","nonce":22697},{"type":"v1","addr":"172.21.15.8:6829","nonce":22697}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6822","nonce":22697},{"type":"v1","addr":"172.21.15.8:6825","nonce":22697}]},"public_addr":"172.21.15.8:6817/22697","cluster_addr":"172.21.15.8:6820/22697","heartbeat_back_addr":"172.21.15.8:6829/22697","heartbeat_front_addr":"172.21.15.8:6825/22697","state":["exists","up"]},{"osd":3,"uuid":"f01e2b49-ee11-419a-92d8-6f3b52f73229","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6819","nonce":22698},{"type":"v1","addr":"172.21.15.8:6821","nonce":22698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6823","nonce":22698},{"type":"v1","addr":"172.21.15.8:6824","nonce":22698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6830","nonce":22698},{"type":"v1","addr":"172.21.15.8:6831","nonce":22698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6826","nonce":22698},{"type":"v1","addr":"172.21.15.8:6828","nonce":22698}]},"public_addr":"172.21.15.8:6821/22698","cluster_addr":"172.21.15.8:6824/22698","heartbeat_back_addr":"172.21.15.8:6831/22698","heartbeat_front_addr":"172.21.15.8:6828/22698","state":["exists","up"]},{"osd":4,"uuid":"9cf99cf4-de20-44f7-bbae-57d8c0e93b08","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":22834},{"type":"v1","addr":"172.21.15.164:6803","nonce":22834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":22834},{"type":"v1","addr":"172.21.15.164:6805","nonce":22834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":22834},{"type":"v1","addr":"172.21.15.164:6812","nonce":22834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":22834},{"type":"v1","addr":"172.21.15.164:6808","nonce":22834}]},"public_addr":"172.21.15.164:6803/22834","cluster_addr":"172.21.15.164:6805/22834","heartbeat_back_addr":"172.21.15.164:6812/22834","heartbeat_front_addr":"172.21.15.164:6808/22834","state":["exists","up"]},{"osd":5,"uuid":"c1ecad70-c4c6-4b35-896c-25b1d9e5a546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6807","nonce":22833},{"type":"v1","addr":"172.21.15.164:6809","nonce":22833}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6811","nonce":22833},{"type":"v1","addr":"172.21.15.164:6814","nonce":22833}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6819","nonce":22833},{"type":"v1","addr":"172.21.15.164:6821","nonce":22833}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6815","nonce":22833},{"type":"v1","addr":"172.21.15.164:6817","nonce":22833}]},"public_addr":"172.21.15.164:6809/22833","cluster_addr":"172.21.15.164:6814/22833","heartbeat_back_addr":"172.21.15.164:6821/22833","heartbeat_front_addr":"172.21.15.164:6817/22833","state":["exists","up"]},{"osd":6,"uuid":"96135535-1048-41e4-8969-723592815994","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6813","nonce":22836},{"type":"v1","addr":"172.21.15.164:6816","nonce":22836}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":22836},{"type":"v1","addr":"172.21.15.164:6820","nonce":22836}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":22836},{"type":"v1","addr":"172.21.15.164:6827","nonce":22836}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6823","nonce":22836},{"type":"v1","addr":"172.21.15.164:6824","nonce":22836}]},"public_addr":"172.21.15.164:6816/22836","cluster_addr":"172.21.15.164:6820/22836","heartbeat_back_addr":"172.21.15.164:6827/22836","heartbeat_front_addr":"172.21.15.164:6824/22836","state":["exists","up"]},{"osd":7,"uuid":"049fe10e-349c-4d45-9361-4d25ae4c0487","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":22835},{"type":"v1","addr":"172.21.15.164:6825","nonce":22835}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":22835},{"type":"v1","addr":"172.21.15.164:6829","nonce":22835}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":22835},{"type":"v1","addr":"172.21.15.164:6833","nonce":22835}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":22835},{"type":"v1","addr":"172.21.15.164:6831","nonce":22835}]},"public_addr":"172.21.15.164:6825/22835","cluster_addr":"172.21.15.164:6829/22835","heartbeat_back_addr":"172.21.15.164:6833/22835","heartbeat_front_addr":"172.21.15.164:6831/22835","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.575221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.615694+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.623134+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.625216+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.608958+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.624172+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.612971+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2020-02-05T16:59:00.690484+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2020-02-05T16:59:28.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.350+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 msgr2=0x7f8c4803be40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.350+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 0x7f8c4803be40 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.356 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.350+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 msgr2=0x7f8c6809dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.350+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f8c680a70e0 tx=0x7f8c58011f80).stop 2020-02-05T16:59:28.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 shutdown_connections 2020-02-05T16:59:28.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8c48039930 0x7f8c4803be40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.357 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1296116246 >> v2:172.21.15.164:3300/0 conn(0x7f8c6813d6b0 0x7f8c6813faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 --2- 172.21.15.8:0/1296116246 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8c6807ded0 0x7f8c6809dbe0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 >> 172.21.15.8:0/1296116246 conn(0x7f8c680a7ab0 msgr2=0x7f8c680aaed0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:28.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 shutdown_connections 2020-02-05T16:59:28.368 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.351+0000 7f8c6de82700 1 -- 172.21.15.8:0/1296116246 wait complete. 2020-02-05T16:59:28.379 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:06.700677+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:10.361501+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'20', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2020-02-05T16:59:11.053750+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'21', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2020-02-05T16:59:28.380 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2020-02-05T16:59:28.614 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.604+0000 7f433bd67700 1 Processor -- start 2020-02-05T16:59:28.614 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.604+0000 7f433bd67700 1 -- start start 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433bd67700 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433bd67700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43340aab90 con 0x7f43340a9e90 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433bd67700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f433405b340 con 0x7f43340908f0 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433bd67700 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4334090d20 con 0x7f433408fff0 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433ab05700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f43340a9e90 0x7f43340a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46258/0 (socket says 172.21.15.8:46258) 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433ab05700 1 -- 172.21.15.8:0/2078779038 learned_addr learned my addr 172.21.15.8:0/2078779038 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:28.615 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.605+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.606+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849514987 0 0) 0x7f433405b340 con 0x7f43340908f0 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.606+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320003230 con 0x7f43340908f0 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.606+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569712609 0 0) 0x7f43340aab90 con 0x7f43340a9e90 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.606+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f433405b340 con 0x7f43340a9e90 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 -- 172.21.15.8:0/2078779038 >> v1:172.21.15.164:6789/0 conn(0x7f43340908f0 legacy=0x7f43340a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 -- 172.21.15.8:0/2078779038 >> v1:172.21.15.8:6789/0 conn(0x7f43340a9e90 legacy=0x7f43340a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.616 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 -- 172.21.15.8:0/2078779038 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43340aa2c0 con 0x7f433408fff0 2020-02-05T16:59:28.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f43300032b0 tx=0x7f43300101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f43300032b0 tx=0x7f43300101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f43300032b0 tx=0x7f43300101f0).ready entity=mon.2 client_cookie=3fcc8fe99ec71c62 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.607+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f43300032b0 tx=0x7f43300101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.617 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f43300032b0 tx=0x7f43300101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f433000a160 con 0x7f433408fff0 2020-02-05T16:59:28.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f433a304700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f43300032b0 tx=0x7f43300101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.618 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330011bb0 con 0x7f433408fff0 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f4339302700 1 -- 172.21.15.8:0/2078779038 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f433000a740 con 0x7f433408fff0 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f433bd67700 1 -- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 msgr2=0x7f433408f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.608+0000 7f433bd67700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f43300032b0 tx=0x7f43300101f0).stop 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.609+0000 7f433bd67700 1 -- 172.21.15.8:0/2078779038 shutdown_connections 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.609+0000 7f433bd67700 1 --2- 172.21.15.8:0/2078779038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f433408fff0 0x7f433408f240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.609+0000 7f433bd67700 1 -- 172.21.15.8:0/2078779038 >> 172.21.15.8:0/2078779038 conn(0x7f43340600f0 msgr2=0x7f433405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:28.619 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.609+0000 7f433bd67700 1 -- 172.21.15.8:0/2078779038 shutdown_connections 2020-02-05T16:59:28.620 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.609+0000 7f433bd67700 1 -- 172.21.15.8:0/2078779038 wait complete. 2020-02-05T16:59:28.637 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.613+0000 7f433bd67700 1 Processor -- start 2020-02-05T16:59:28.637 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.613+0000 7f433bd67700 1 -- start start 2020-02-05T16:59:28.637 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.615+0000 7f433bd67700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4334069c60 0x7f4334063e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.638 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.615+0000 7f433bd67700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4334062240 0x7f43340a16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.638 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.615+0000 7f433bd67700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43340a1c10 con 0x7f4334061e10 2020-02-05T16:59:28.639 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.615+0000 7f433bd67700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4334061160 con 0x7f4334069c60 2020-02-05T16:59:28.639 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.615+0000 7f433bd67700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4334061400 con 0x7f4334062240 2020-02-05T16:59:28.639 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4334069c60 0x7f4334063e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.640 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f4334069c60 0x7f4334063e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.8:3300/0 says I am v2:172.21.15.8:44878/0 (socket says 172.21.15.8:44878) 2020-02-05T16:59:28.640 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 -- 172.21.15.8:0/1621695848 learned_addr learned my addr 172.21.15.8:0/1621695848 (peer_addr_for_me v2:172.21.15.8:0/0) 2020-02-05T16:59:28.640 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433ab05700 1 --2- 172.21.15.8:0/1621695848 >> v2:172.21.15.164:3300/0 conn(0x7f4334062240 0x7f43340a16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.640 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 -- 172.21.15.8:0/1621695848 >> v2:172.21.15.164:3300/0 conn(0x7f4334062240 msgr2=0x7f43340a16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.640 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> v2:172.21.15.164:3300/0 conn(0x7f4334062240 0x7f43340a16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 -- 172.21.15.8:0/1621695848 >> v1:172.21.15.164:6789/0 conn(0x7f4334061e10 legacy=0x7f43340a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 -- 172.21.15.8:0/1621695848 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4334062670 con 0x7f4334069c60 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.616+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> v2:172.21.15.8:3300/0 conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> v2:172.21.15.8:3300/0 conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.642 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.642 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.642 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4330005a40 con 0x7f4334069c60 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433000a360 con 0x7f4334069c60 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4330007340 con 0x7f4334069c60 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4334060cc0 con 0x7f4334069c60 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.617+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f433413da80 con 0x7f4334069c60 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.618+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.643 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.619+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4334056640 con 0x7f4334069c60 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.619+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f4330007650 con 0x7f4334069c60 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.619+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.620+0000 7f4326ffd700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 0x7f431c03bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.620+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f433004a310 con 0x7f4334069c60 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.621+0000 7f4339b03700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 0x7f431c03bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.621+0000 7f4339b03700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 0x7f431c03bea0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.624+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.644 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.624+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f4330031e60 con 0x7f4334069c60 2020-02-05T16:59:28.782 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.780+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4334068f50 con 0x7f4334069c60 2020-02-05T16:59:28.784 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.780+0000 7f433a304700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:28.784 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.781+0000 7f4326ffd700 1 -- 172.21.15.8:0/1621695848 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v21) v1 ==== 92+0+10 (secure 0 0 0) 0x7f43300320c0 con 0x7f4334069c60 2020-02-05T16:59:28.784 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.782+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 msgr2=0x7f431c03bea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.782+0000 7f433bd67700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 0x7f431c03bea0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.782+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 msgr2=0x7f4334063e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.782+0000 7f433bd67700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f43340ab7a0 tx=0x7f4330011ee0).stop 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 shutdown_connections 2020-02-05T16:59:28.786 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 --2- 172.21.15.8:0/1621695848 >> v2:172.21.15.164:3300/0 conn(0x7f4334062240 0x7f43340a16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f4334069c60 0x7f4334063e90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 --2- 172.21.15.8:0/1621695848 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f431c039990 0x7f431c03bea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:28.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 >> 172.21.15.8:0/1621695848 conn(0x7f4334060050 msgr2=0x7f433407ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:28.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 shutdown_connections 2020-02-05T16:59:28.787 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:28.783+0000 7f433bd67700 1 -- 172.21.15.8:0/1621695848 wait complete. 2020-02-05T16:59:28.809 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2020-02-05T16:59:29.051 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.041+0000 7f8a29357700 1 Processor -- start 2020-02-05T16:59:29.051 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.041+0000 7f8a29357700 1 -- start start 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a29357700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8a240ace50 0x7f8a240aa430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a29357700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240aa980 con 0x7f8a240abd50 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a29357700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a241454a0 con 0x7f8a24145c20 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a29357700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a241456d0 con 0x7f8a240ace50 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a28b56700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8a240ace50 0x7f8a240aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.052 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.042+0000 7f8a23fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f8a240abd50 0x7f8a241491b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46268/0 (socket says 172.21.15.8:46268) 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.043+0000 7f8a23fff700 1 -- 172.21.15.8:0/917631155 learned_addr learned my addr 172.21.15.8:0/917631155 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.043+0000 7f8a28b56700 1 -- 172.21.15.8:0/917631155 >> v1:172.21.15.164:6789/0 conn(0x7f8a24145c20 legacy=0x7f8a2413f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.043+0000 7f8a28b56700 1 -- 172.21.15.8:0/917631155 >> v1:172.21.15.8:6789/0 conn(0x7f8a240abd50 legacy=0x7f8a241491b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.043+0000 7f8a28b56700 1 -- 172.21.15.8:0/917631155 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24146050 con 0x7f8a240ace50 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> v2:172.21.15.8:3300/0 conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8a14008910 tx=0x7f8a1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> v2:172.21.15.8:3300/0 conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8a14008910 tx=0x7f8a1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.053 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f8a14008910 tx=0x7f8a1400fe70).ready entity=mon.0 client_cookie=28b5bf254559dcc0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f8a14008910 tx=0x7f8a1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f8a14008910 tx=0x7f8a1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a28b56700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f8a14008910 tx=0x7f8a1400fe70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a237fe700 1 -- 172.21.15.8:0/917631155 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a140116e0 con 0x7f8a240ace50 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.044+0000 7f8a237fe700 1 -- 172.21.15.8:0/917631155 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a14008c50 con 0x7f8a240ace50 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a237fe700 1 -- 172.21.15.8:0/917631155 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a14011cc0 con 0x7f8a240ace50 2020-02-05T16:59:29.054 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a29357700 1 -- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 msgr2=0x7f8a240aa430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a29357700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f8a14008910 tx=0x7f8a1400fe70).stop 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a29357700 1 -- 172.21.15.8:0/917631155 shutdown_connections 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a29357700 1 --2- 172.21.15.8:0/917631155 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a240ace50 0x7f8a240aa430 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.045+0000 7f8a29357700 1 -- 172.21.15.8:0/917631155 >> 172.21.15.8:0/917631155 conn(0x7f8a24061510 msgr2=0x7f8a240ac2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.046+0000 7f8a29357700 1 -- 172.21.15.8:0/917631155 shutdown_connections 2020-02-05T16:59:29.055 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.046+0000 7f8a29357700 1 -- 172.21.15.8:0/917631155 wait complete. 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.049+0000 7f8a29357700 1 Processor -- start 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.050+0000 7f8a29357700 1 -- start start 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a29357700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8a2411a160 0x7f8a2407eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a29357700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a240a30d0 con 0x7f8a24119d30 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a29357700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2407f000 con 0x7f8a2405be10 2020-02-05T16:59:29.063 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a29357700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a24114750 con 0x7f8a2411a160 2020-02-05T16:59:29.064 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a2a5b9700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7f8a2411a160 0x7f8a2407eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.064 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a23fff700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7f8a24119d30 0x7f8a2407e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46276/0 (socket says 172.21.15.8:46276) 2020-02-05T16:59:29.064 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.052+0000 7f8a23fff700 1 -- 172.21.15.8:0/2139631478 learned_addr learned my addr 172.21.15.8:0/2139631478 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.064 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 -- 172.21.15.8:0/2139631478 >> v1:172.21.15.164:6789/0 conn(0x7f8a2405be10 legacy=0x7f8a24119150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.065 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 -- 172.21.15.8:0/2139631478 >> v1:172.21.15.8:6789/0 conn(0x7f8a24119d30 legacy=0x7f8a2407e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.065 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200519496 0 0) 0x7f8a240a30d0 con 0x7f8a24119d30 2020-02-05T16:59:29.066 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 -- 172.21.15.8:0/2139631478 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a24114370 con 0x7f8a2411a160 2020-02-05T16:59:29.066 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> v2:172.21.15.8:3300/0 conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.066 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> v2:172.21.15.8:3300/0 conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.053+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c012320 con 0x7f8a2411a160 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c006b60 con 0x7f8a2411a160 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.067 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8a1c012940 con 0x7f8a2411a160 2020-02-05T16:59:29.068 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24114c90 con 0x7f8a2411a160 2020-02-05T16:59:29.068 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.054+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2412fb50 con 0x7f8a2411a160 2020-02-05T16:59:29.068 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.055+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.069 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.056+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7f8a1c012ca0 con 0x7f8a2411a160 2020-02-05T16:59:29.069 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.056+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.069 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.056+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24056640 con 0x7f8a2411a160 2020-02-05T16:59:29.069 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.056+0000 7f8a217fa700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 0x7f8a0803bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.070 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.056+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7f8a1c049db0 con 0x7f8a2411a160 2020-02-05T16:59:29.070 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.057+0000 7f8a28b56700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 0x7f8a0803bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.070 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.058+0000 7f8a28b56700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 0x7f8a0803bd50 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.072 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.064+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.072 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.064+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7f8a1c0318e0 con 0x7f8a2411a160 2020-02-05T16:59:29.216 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.214+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8a24066210 con 0x7f8a2411a160 2020-02-05T16:59:29.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.214+0000 7f8a2a5b9700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.217 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.214+0000 7f8a217fa700 1 -- 172.21.15.8:0/2139631478 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8a1c031b40 con 0x7f8a2411a160 2020-02-05T16:59:29.217 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:29.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 msgr2=0x7f8a0803bd50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 0x7f8a0803bd50 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 msgr2=0x7f8a2407eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.219 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f8a1c001210 tx=0x7f8a1c010c60).stop 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 shutdown_connections 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7f8a08039840 0x7f8a0803bd50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 --2- 172.21.15.8:0/2139631478 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7f8a2411a160 0x7f8a2407eab0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.216+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 >> 172.21.15.8:0/2139631478 conn(0x7f8a240ace50 msgr2=0x7f8a241433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.217+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 shutdown_connections 2020-02-05T16:59:29.220 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.217+0000 7f8a29357700 1 -- 172.21.15.8:0/2139631478 wait complete. 2020-02-05T16:59:29.238 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2020-02-05T16:59:29.472 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.459+0000 7fae696df700 1 Processor -- start 2020-02-05T16:59:29.472 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.459+0000 7fae696df700 1 -- start start 2020-02-05T16:59:29.472 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae696df700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae6408fff0 0x7fae6408f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.472 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae696df700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640aab90 con 0x7fae640908f0 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae696df700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6405b340 con 0x7fae640a9e90 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae696df700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae64090d20 con 0x7fae6408fff0 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae62ffd700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fae640908f0 0x7fae640a7990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46282/0 (socket says 172.21.15.8:46282) 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.460+0000 7fae62ffd700 1 -- 172.21.15.8:0/836396362 learned_addr learned my addr 172.21.15.8:0/836396362 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.461+0000 7fae637fe700 1 --2- 172.21.15.8:0/836396362 >> v2:172.21.15.164:3300/0 conn(0x7fae6408fff0 0x7fae6408f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.473 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.461+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579960217 0 0) 0x7fae640aab90 con 0x7fae640908f0 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.461+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae50003230 con 0x7fae640908f0 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3135526780 0 0) 0x7fae50003230 con 0x7fae640908f0 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 >> v2:172.21.15.164:3300/0 conn(0x7fae6408fff0 msgr2=0x7fae6408f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 --2- 172.21.15.8:0/836396362 >> v2:172.21.15.164:3300/0 conn(0x7fae6408fff0 0x7fae6408f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 >> v1:172.21.15.164:6789/0 conn(0x7fae640a9e90 legacy=0x7fae640a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 --> v1:172.21.15.8:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640aa2c0 con 0x7fae640908f0 2020-02-05T16:59:29.474 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v1:172.21.15.8:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 599595003 0 0) 0x7fae54001ad0 con 0x7fae640908f0 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 >> v1:172.21.15.8:6789/0 conn(0x7fae640908f0 legacy=0x7fae640a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50003860 0x7fae50003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae500045a0 0x7fae500049d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae64090d20 con 0x7fae50004f20 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5004fde0 con 0x7fae50003860 2020-02-05T16:59:29.475 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae50050020 con 0x7fae500045a0 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.462+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae62ffd700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae500045a0 0x7fae500049d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae637fe700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50003860 0x7fae50003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae63fff700 1 -- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae500045a0 msgr2=0x7fae500049d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae500045a0 0x7fae500049d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae63fff700 1 -- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50003860 msgr2=0x7fae50003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2020-02-05T16:59:29.476 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.463+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50003860 0x7fae50003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2020-02-05T16:59:29.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 -- 172.21.15.8:0/836396362 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae50050320 con 0x7fae50004f20 2020-02-05T16:59:29.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fae5c001000 tx=0x7fae5c00fda0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 -- 172.21.15.8:0/836396362 reap_dead start 2020-02-05T16:59:29.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fae5c001000 tx=0x7fae5c00fda0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.477 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fae5c001000 tx=0x7fae5c00fda0).ready entity=mon.0 client_cookie=ae734c44ed5e1b14 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fae5c001000 tx=0x7fae5c00fda0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5c003540 con 0x7fae50004f20 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fae5c001000 tx=0x7fae5c00fda0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c003160 con 0x7fae50004f20 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae63fff700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fae5c001000 tx=0x7fae5c00fda0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.478 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae696df700 1 -- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 msgr2=0x7fae5000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.464+0000 7fae627fc700 1 -- 172.21.15.8:0/836396362 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5c003b20 con 0x7fae50004f20 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.465+0000 7fae696df700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fae5c001000 tx=0x7fae5c00fda0).stop 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.466+0000 7fae696df700 1 -- 172.21.15.8:0/836396362 shutdown_connections 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.466+0000 7fae696df700 1 --2- 172.21.15.8:0/836396362 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae50004f20 0x7fae5000b7f0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.466+0000 7fae696df700 1 -- 172.21.15.8:0/836396362 >> 172.21.15.8:0/836396362 conn(0x7fae640600f0 msgr2=0x7fae6405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.466+0000 7fae696df700 1 -- 172.21.15.8:0/836396362 shutdown_connections 2020-02-05T16:59:29.479 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.466+0000 7fae696df700 1 -- 172.21.15.8:0/836396362 wait complete. 2020-02-05T16:59:29.482 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.470+0000 7fae696df700 1 Processor -- start 2020-02-05T16:59:29.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.470+0000 7fae696df700 1 -- start start 2020-02-05T16:59:29.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae696df700 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 0x7fae6407d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae696df700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fae6407df20 0x7fae6413d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.483 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae696df700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae6409e7d0 con 0x7fae64124e50 2020-02-05T16:59:29.484 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae696df700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640a21d0 con 0x7fae6407df20 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae696df700 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae64066cd0 con 0x7fae64124a20 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae62ffd700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fae64124e50 0x7fae64132db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46294/0 (socket says 172.21.15.8:46294) 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.472+0000 7fae62ffd700 1 -- 172.21.15.8:0/3894970637 learned_addr learned my addr 172.21.15.8:0/3894970637 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.473+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.8:3300/0 conn(0x7fae6407df20 0x7fae6413d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.473+0000 7fae637fe700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 0x7fae6407d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.491 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.473+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1595251711 0 0) 0x7fae6409e7d0 con 0x7fae64124e50 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.473+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae48003230 con 0x7fae64124e50 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 -- 172.21.15.8:0/3894970637 >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 msgr2=0x7fae6407d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 0x7fae6407d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 -- 172.21.15.8:0/3894970637 >> v1:172.21.15.8:6789/0 conn(0x7fae64124e50 legacy=0x7fae64132db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 66410310 0 0) 0x7fae48003230 con 0x7fae64124e50 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 -- 172.21.15.8:0/3894970637 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae641335b0 con 0x7fae6407df20 2020-02-05T16:59:29.492 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae637fe700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 0x7fae6407d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2020-02-05T16:59:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.8:3300/0 conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.8:3300/0 conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.474+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.493 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5c010600 con 0x7fae6407df20 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5c003740 con 0x7fae6407df20 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae5c010be0 con 0x7fae6407df20 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae6413dc30 con 0x7fae6407df20 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.475+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae6413de40 con 0x7fae6407df20 2020-02-05T16:59:29.494 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.476+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.477+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.477+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fae5c00a180 con 0x7fae6407df20 2020-02-05T16:59:29.495 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.477+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64056640 con 0x7fae6407df20 2020-02-05T16:59:29.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.477+0000 7fae5bfff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 0x7fae4803c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.478+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fae5c049240 con 0x7fae6407df20 2020-02-05T16:59:29.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.478+0000 7fae637fe700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 0x7fae4803c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.501 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.479+0000 7fae637fe700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 0x7fae4803c390 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.481+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.502 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.482+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fae5c010de0 con 0x7fae6407df20 2020-02-05T16:59:29.641 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.638+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae64068f50 con 0x7fae6407df20 2020-02-05T16:59:29.642 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.639+0000 7fae63fff700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.642 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.640+0000 7fae5bfff700 1 -- 172.21.15.8:0/3894970637 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+10 (secure 0 0 0) 0x7fae5c030fb0 con 0x7fae6407df20 2020-02-05T16:59:29.642 INFO:teuthology.orchestra.run.smithi008.stdout:pg_num: 8 2020-02-05T16:59:29.645 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.641+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 msgr2=0x7fae4803c390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.646 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.641+0000 7fae696df700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 0x7fae4803c390 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.646 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.641+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 msgr2=0x7fae6413d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.646 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.641+0000 7fae696df700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fae640a2390 tx=0x7fae5c00f7b0).stop 2020-02-05T16:59:29.646 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 shutdown_connections 2020-02-05T16:59:29.646 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fae48039e80 0x7fae4803c390 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.647 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 --2- 172.21.15.8:0/3894970637 >> v2:172.21.15.164:3300/0 conn(0x7fae64124a20 0x7fae6407d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.647 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 --2- 172.21.15.8:0/3894970637 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fae6407df20 0x7fae6413d6e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.647 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 >> 172.21.15.8:0/3894970637 conn(0x7fae64069c50 msgr2=0x7fae640a4180 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:29.647 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 shutdown_connections 2020-02-05T16:59:29.647 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.642+0000 7fae696df700 1 -- 172.21.15.8:0/3894970637 wait complete. 2020-02-05T16:59:29.664 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2020-02-05T16:59:29.664 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2020-02-05T16:59:29.898 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.887+0000 7fc0db890700 1 Processor -- start 2020-02-05T16:59:29.898 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.887+0000 7fc0db890700 1 -- start start 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.888+0000 7fc0db890700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fc0d40908f0 0x7fc0d40a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.888+0000 7fc0db890700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d40aa3e0 con 0x7fc0d408f240 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.888+0000 7fc0db890700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d40aab90 con 0x7fc0d408fff0 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.888+0000 7fc0db890700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d408f670 con 0x7fc0d40908f0 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d962c700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fc0d408f240 0x7fc0d40a8190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46304/0 (socket says 172.21.15.8:46304) 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d962c700 1 -- 172.21.15.8:0/3084451172 learned_addr learned my addr 172.21.15.8:0/3084451172 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.899 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> v2:172.21.15.8:3300/0 conn(0x7fc0d40908f0 0x7fc0d40a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762363848 0 0) 0x7fc0d00010d0 con 0x7fc0d408f240 2020-02-05T16:59:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0c0003230 con 0x7fc0d408f240 2020-02-05T16:59:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514374568 0 0) 0x7fc0d40aab90 con 0x7fc0d408fff0 2020-02-05T16:59:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.889+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d00010d0 con 0x7fc0d408fff0 2020-02-05T16:59:29.900 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3927822446 0 0) 0x7fc0c0003230 con 0x7fc0d408f240 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 -- 172.21.15.8:0/3084451172 >> v1:172.21.15.164:6789/0 conn(0x7fc0d408fff0 legacy=0x7fc0d40a7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 -- 172.21.15.8:0/3084451172 >> v1:172.21.15.8:6789/0 conn(0x7fc0d408f240 legacy=0x7fc0d40a8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 -- 172.21.15.8:0/3084451172 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d405b340 con 0x7fc0d40908f0 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> v2:172.21.15.8:3300/0 conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> v2:172.21.15.8:3300/0 conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.890+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).ready entity=mon.0 client_cookie=e0561fc66b2b397b server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.901 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.902 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0d0001980 con 0x7fc0d40908f0 2020-02-05T16:59:29.902 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0001790 con 0x7fc0d40908f0 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0da62e700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0db890700 1 -- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 msgr2=0x7fc0d40a9e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0d8e2b700 1 -- 172.21.15.8:0/3084451172 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0cc00a350 con 0x7fc0d40908f0 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.891+0000 7fc0db890700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7fc0cc008c90 tx=0x7fc0d0001b80).stop 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.892+0000 7fc0db890700 1 -- 172.21.15.8:0/3084451172 shutdown_connections 2020-02-05T16:59:29.903 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.892+0000 7fc0db890700 1 --2- 172.21.15.8:0/3084451172 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d40908f0 0x7fc0d40a9e90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:29.904 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.892+0000 7fc0db890700 1 -- 172.21.15.8:0/3084451172 >> 172.21.15.8:0/3084451172 conn(0x7fc0d40600f0 msgr2=0x7fc0d405a630 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:29.904 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.892+0000 7fc0db890700 1 -- 172.21.15.8:0/3084451172 shutdown_connections 2020-02-05T16:59:29.904 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.893+0000 7fc0db890700 1 -- 172.21.15.8:0/3084451172 wait complete. 2020-02-05T16:59:29.904 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.895+0000 7fc0db890700 1 Processor -- start 2020-02-05T16:59:29.904 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.895+0000 7fc0db890700 1 -- start start 2020-02-05T16:59:29.907 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.896+0000 7fc0db890700 1 --2- >> v2:172.21.15.8:3300/0 conn(0x7fc0d407e380 0x7fc0d40a4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.914 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.896+0000 7fc0db890700 1 -- --> v1:172.21.15.8:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d40a2020 con 0x7fc0d40a1a10 2020-02-05T16:59:29.914 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.896+0000 7fc0db890700 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d407cec0 con 0x7fc0d41385e0 2020-02-05T16:59:29.914 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.896+0000 7fc0db890700 1 -- --> v2:172.21.15.8:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d407f620 con 0x7fc0d407e380 2020-02-05T16:59:29.914 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.897+0000 7fc0d962c700 1 --1- >> v1:172.21.15.8:6789/0 conn(0x7fc0d40a1a10 0x7fc0d407c6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.8:6789/0 says I am v1:172.21.15.8:46310/0 (socket says 172.21.15.8:46310) 2020-02-05T16:59:29.914 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.897+0000 7fc0d962c700 1 -- 172.21.15.8:0/697224804 learned_addr learned my addr 172.21.15.8:0/697224804 (peer_addr_for_me v1:172.21.15.8:0/0) 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.897+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> v2:172.21.15.8:3300/0 conn(0x7fc0d407e380 0x7fc0d40a4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.897+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v1:172.21.15.8:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173051110 0 0) 0x7fc0d0001460 con 0x7fc0d40a1a10 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 --> v1:172.21.15.8:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0bc003230 con 0x7fc0d40a1a10 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.1 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216664144 0 0) 0x7fc0d407cec0 con 0x7fc0d41385e0 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d0001460 con 0x7fc0d41385e0 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v1:172.21.15.8:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3012217201 0 0) 0x7fc0bc003230 con 0x7fc0d40a1a10 2020-02-05T16:59:29.915 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0d9e2d700 1 -- 172.21.15.8:0/697224804 >> v1:172.21.15.164:6789/0 conn(0x7fc0d41385e0 legacy=0x7fc0d413f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0d9e2d700 1 -- 172.21.15.8:0/697224804 >> v1:172.21.15.8:6789/0 conn(0x7fc0d40a1a10 legacy=0x7fc0d407c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.898+0000 7fc0d9e2d700 1 -- 172.21.15.8:0/697224804 --> v2:172.21.15.8:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4138a10 con 0x7fc0d407e380 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> v2:172.21.15.8:3300/0 conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> v2:172.21.15.8:3300/0 conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.916 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0d0004900 con 0x7fc0d407e380 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c8011e10 con 0x7fc0d407e380 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d41411e0 con 0x7fc0d407e380 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.899+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c8006bc0 con 0x7fc0d407e380 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.900+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d41413f0 con 0x7fc0d407e380 2020-02-05T16:59:29.917 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.900+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.901+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 4 ==== mgrmap(e 4) v1 ==== 44213+0+0 (secure 0 0 0) 0x7fc0c800a310 con 0x7fc0d407e380 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.901+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4056640 con 0x7fc0d407e380 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.901+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.902+0000 7fc0c67fc700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 0x7fc0bc03c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.902+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5673+0+0 (secure 0 0 0) 0x7fc0c8049060 con 0x7fc0d407e380 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.902+0000 7fc0d962c700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 0x7fc0bc03c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2020-02-05T16:59:29.918 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.903+0000 7fc0d962c700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 0x7fc0bc03c530 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2020-02-05T16:59:29.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.908+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:29.919 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:29.908+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+116403 (secure 0 0 0) 0x7fc0c8030b30 con 0x7fc0d407e380 2020-02-05T16:59:30.116 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.114+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 --> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0d409e670 con 0x7fc0d407e380 2020-02-05T16:59:30.124 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.115+0000 7fc0d9e2d700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.115+0000 7fc0c67fc700 1 -- 172.21.15.8:0/697224804 <== mon.0 v2:172.21.15.8:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7fc0c8030d90 con 0x7fc0d407e380 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 msgr2=0x7fc0bc03c530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 0x7fc0bc03c530 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 msgr2=0x7fc0d40a4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7fc0d0001380 tx=0x7fc0d0004b00).stop 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 shutdown_connections 2020-02-05T16:59:30.125 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.164:6800/21877,v1:172.21.15.164:6801/21877] conn(0x7fc0bc03a020 0x7fc0bc03c530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 --2- 172.21.15.8:0/697224804 >> [v2:172.21.15.8:3300/0,v1:172.21.15.8:6789/0] conn(0x7fc0d407e380 0x7fc0d40a4370 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 >> 172.21.15.8:0/697224804 conn(0x7fc0d4128840 msgr2=0x7fc0d408fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.117+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 shutdown_connections 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stderr:2020-02-05T16:59:30.118+0000 7fc0db890700 1 -- 172.21.15.8:0/697224804 wait complete. 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stderr:dumped fsmap epoch 7 2020-02-05T16:59:30.126 INFO:teuthology.orchestra.run.smithi008.stdout: 2020-02-05T16:59:30.127 INFO:teuthology.orchestra.run.smithi008.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4332,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.8:6833/4097051851","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.8:6832","nonce":4097051851},{"type":"v1","addr":"172.21.15.8:6833","nonce":4097051851}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2020-02-05T16:59:12.091946+0000","modified":"2020-02-05T16:59:17.122886+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4336},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4336":{"gid":4336,"name":"b","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.164:6835/1530677404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6834","nonce":1530677404},{"type":"v1","addr":"172.21.15.164:6835","nonce":1530677404}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2020-02-05T16:59:30.340 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4336}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4336': {u'export_targets': [], u'name': u'b', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4336, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 1530677404, u'type': u'v2', u'addr': u'172.21.15.164:6834'}, {u'nonce': 1530677404, u'type': u'v1', u'addr': u'172.21.15.164:6835'}]}, u'addr': u'172.21.15.164:6835/1530677404'}}, u'fs_name': u'cephfs', u'created': u'2020-02-05T16:59:12.091946+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2020-02-05T16:59:17.122886+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2020-02-05T16:59:30.340 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2020-02-05T16:59:30.340 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok status 2020-02-05T16:59:30.572 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-02-05T16:59:30.572 INFO:teuthology.orchestra.run.smithi164.stdout: "cluster_fsid": "db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7", 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "whoami": 0, 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "id": 4336, 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "want_state": "up:active", 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "up:active", 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "rank_uptime": 14.442440738, 2020-02-05T16:59:30.573 INFO:teuthology.orchestra.run.smithi164.stdout: "mdsmap_epoch": 6, 2020-02-05T16:59:30.574 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_epoch": 21, 2020-02-05T16:59:30.574 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_epoch_barrier": 21, 2020-02-05T16:59:30.574 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 15.335921285 2020-02-05T16:59:30.574 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-02-05T16:59:30.587 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "db6fa56f-aafd-4e35-9c18-a4a39bcbe6f7", "whoami": 0, "id": 4336, "want_state": "up:active", "state": "up:active", "rank_uptime": 14.442440738, "mdsmap_epoch": 6, "osdmap_epoch": 21, "osdmap_epoch_barrier": 21, "uptime": 15.335921285 } 2020-02-05T16:59:30.587 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2020-02-05T16:59:30.587 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2020-02-05T16:59:30.587 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi008.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2020-02-05T16:59:30.588 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:30.588 INFO:teuthology.orchestra.run.smithi008:> mkdir -p /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:30.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:30.659 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-02-05T16:59:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-02-05T16:59:30.780 INFO:teuthology.orchestra.run.smithi008.stderr:mount: /sys/fs/fuse/connections: mount point does not exist. 2020-02-05T16:59:30.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:30.781 INFO:teuthology.orchestra.run.smithi008:> ls /sys/fs/fuse/connections 2020-02-05T16:59:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-02-05T16:59:30.852 INFO:teuthology.orchestra.run.smithi008.stderr:ls: cannot access '/sys/fs/fuse/connections': No such file or directory 2020-02-05T16:59:30.853 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2020-02-05T16:59:30.853 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:30.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:30.859 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-02-05T16:59:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-02-05T16:59:30.981 INFO:teuthology.orchestra.run.smithi008.stderr:mount: /sys/fs/fuse/connections: mount point does not exist. 2020-02-05T16:59:30.981 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:30.981 INFO:teuthology.orchestra.run.smithi008:> ls /sys/fs/fuse/connections 2020-02-05T16:59:31.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi008.stderr:2020-02-05T16:59:31.017+0000 7f2eca5001c0 -1 init, newargv = 0x55ab29a5e750 newargc=7 2020-02-05T16:59:31.019 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi008.stderr:ceph-fuse[25605]: starting ceph client 2020-02-05T16:59:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 2 2020-02-05T16:59:31.053 INFO:teuthology.orchestra.run.smithi008.stderr:ls: cannot access '/sys/fs/fuse/connections': No such file or directory 2020-02-05T16:59:31.097 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi008.stderr:ceph-fuse[25605]: starting fuse 2020-02-05T16:59:32.155 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.155 INFO:teuthology.orchestra.run.smithi008:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2020-02-05T16:59:32.260 DEBUG:teuthology.orchestra.run:got remote process result: 32 2020-02-05T16:59:32.278 INFO:teuthology.orchestra.run.smithi008.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2020-02-05T16:59:32.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.279 INFO:teuthology.orchestra.run.smithi008:> ls /sys/fs/fuse/connections 2020-02-05T16:59:32.351 INFO:teuthology.orchestra.run.smithi008.stdout:40 2020-02-05T16:59:32.351 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [40] 2020-02-05T16:59:32.351 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.351 INFO:teuthology.orchestra.run.smithi008:> sudo python3 -c ' 2020-02-05T16:59:32.351 INFO:teuthology.orchestra.run.smithi008:> import glob 2020-02-05T16:59:32.351 INFO:teuthology.orchestra.run.smithi008:> import re 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> import os 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> import subprocess 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> def find_socket(client_name): 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> files = glob.glob(asok_path) 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> 2020-02-05T16:59:32.352 INFO:teuthology.orchestra.run.smithi008:> # Given a non-glob path, it better be there 2020-02-05T16:59:32.353 INFO:teuthology.orchestra.run.smithi008:> if "*" not in asok_path: 2020-02-05T16:59:32.353 INFO:teuthology.orchestra.run.smithi008:> assert(len(files) == 1) 2020-02-05T16:59:32.353 INFO:teuthology.orchestra.run.smithi008:> return files[0] 2020-02-05T16:59:32.353 INFO:teuthology.orchestra.run.smithi008:> 2020-02-05T16:59:32.353 INFO:teuthology.orchestra.run.smithi008:> for f in files: 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> if os.path.exists("/proc/{0}".format(pid)): 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> return f 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> print(find_socket("client.0")) 2020-02-05T16:59:32.354 INFO:teuthology.orchestra.run.smithi008:> ' 2020-02-05T16:59:32.526 INFO:teuthology.orchestra.run.smithi008.stdout:/var/run/ceph/ceph-client.0.25605.asok 2020-02-05T16:59:32.527 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.25605.asok 2020-02-05T16:59:32.527 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.527 INFO:teuthology.orchestra.run.smithi008:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.25605.asok status 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout: "metadata": { 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_sha1": "ecafd7283ef390be73a605986afe2a0f325a3255", 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout: "ceph_version": "ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev)", 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout: "entity_id": "0", 2020-02-05T16:59:32.754 INFO:teuthology.orchestra.run.smithi008.stdout: "hostname": "smithi008", 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "pid": "25605", 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "root": "/" 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "dentry_count": 0, 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "dentry_pinned_count": 0, 2020-02-05T16:59:32.755 INFO:teuthology.orchestra.run.smithi008.stdout: "id": 4617, 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "inst": { 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "name": { 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "client", 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "num": 4617 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": { 2020-02-05T16:59:32.756 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:0", 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3855507206 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": { 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: "type": "v1", 2020-02-05T16:59:32.757 INFO:teuthology.orchestra.run.smithi008.stdout: "addr": "172.21.15.8:0", 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "nonce": 3855507206 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "inst_str": "client.4617 172.21.15.8:0/3855507206", 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "addr_str": "172.21.15.8:0/3855507206", 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "inode_count": 1, 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_epoch": 6, 2020-02-05T16:59:32.758 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_epoch": 21, 2020-02-05T16:59:32.759 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_epoch_barrier": 0, 2020-02-05T16:59:32.759 INFO:teuthology.orchestra.run.smithi008.stdout: "blacklisted": false 2020-02-05T16:59:32.759 INFO:teuthology.orchestra.run.smithi008.stdout:} 2020-02-05T16:59:32.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.759 INFO:teuthology.orchestra.run.smithi008:> stat --file-system '--printf=%T 2020-02-05T16:59:32.760 INFO:teuthology.orchestra.run.smithi008:> ' -- /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:32.831 INFO:teuthology.orchestra.run.smithi008.stdout:fuseblk 2020-02-05T16:59:32.831 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:32.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T16:59:32.831 INFO:teuthology.orchestra.run.smithi008:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:32.953 INFO:teuthology.run_tasks:Running task check-counter... 2020-02-05T16:59:32.964 INFO:teuthology.run_tasks:Running task workunit... 2020-02-05T16:59:32.986 INFO:tasks.workunit:Pulling workunits from ref ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:59:32.986 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-02-05T16:59:32.986 INFO:tasks.workunit:timeout=3h 2020-02-05T16:59:32.995 INFO:tasks.workunit:cleanup=True 2020-02-05T16:59:32.995 INFO:teuthology.orchestra.run.smithi008:> stat -- /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:33.058 INFO:teuthology.orchestra.run.smithi008.stdout: File: /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:33.058 INFO:teuthology.orchestra.run.smithi008.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Device: 28h/40d Inode: 1 Links: 2 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Context: system_u:object_r:fusefs_t:s0 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Modify: 2020-02-05 16:59:16.131148422 +0000 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout:Change: 2020-02-05 16:59:32.924579868 +0000 2020-02-05T16:59:33.059 INFO:teuthology.orchestra.run.smithi008.stdout: Birth: - 2020-02-05T16:59:33.060 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2020-02-05T16:59:33.060 INFO:teuthology.orchestra.run.smithi008:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2020-02-05T16:59:33.177 INFO:teuthology.orchestra.run.smithi008:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ecafd7283ef390be73a605986afe2a0f325a3255 2020-02-05T16:59:33.235 INFO:tasks.workunit.client.0.smithi008.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-02-05T16:59:35.877 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T16:59:35.884 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:00:06.047 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:00:06.054 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:00:36.188 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:00:36.194 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:01:03.087 INFO:tasks.workunit.client.0.smithi008.stderr:Note: checking out 'ecafd7283ef390be73a605986afe2a0f325a3255'. 2020-02-05T17:01:03.087 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr:changes and commit them, and you can discard any commits you make in this 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr:state without impacting any branches by performing another checkout. 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr:If you want to create a new branch to retain commits you create, you may 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-02-05T17:01:03.088 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:03.089 INFO:tasks.workunit.client.0.smithi008.stderr: git checkout -b 2020-02-05T17:01:03.089 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:03.089 INFO:tasks.workunit.client.0.smithi008.stderr:HEAD is now at ecafd7283e client: enable async dir operations for testing 2020-02-05T17:01:03.108 INFO:teuthology.orchestra.run.smithi008:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-02-05T17:01:03.180 INFO:tasks.workunit.client.0.smithi008.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2020-02-05T17:01:03.190 INFO:tasks.workunit.client.0.smithi008.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-02-05T17:01:03.190 INFO:tasks.workunit.client.0.smithi008.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2020-02-05T17:01:03.414 INFO:tasks.workunit.client.0.smithi008.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2020-02-05T17:01:03.508 INFO:tasks.workunit.client.0.smithi008.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2020-02-05T17:01:03.590 INFO:tasks.workunit.client.0.smithi008.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2020-02-05T17:01:03.594 INFO:tasks.workunit.client.0.smithi008.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-02-05T17:01:03.595 INFO:tasks.workunit.client.0.smithi008.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2020-02-05T17:01:03.673 INFO:tasks.workunit.client.0.smithi008.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2020-02-05T17:01:03.745 DEBUG:teuthology.orchestra.remote:smithi008:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2020-02-05T17:01:03.812 INFO:tasks.workunit:Running workunits matching kernel_untar_build.sh on client.0... 2020-02-05T17:01:03.813 INFO:tasks.workunit:Running workunit kernel_untar_build.sh... 2020-02-05T17:01:03.813 INFO:teuthology.orchestra.run.smithi008:workunit test kernel_untar_build.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ecafd7283ef390be73a605986afe2a0f325a3255 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh 2020-02-05T17:01:03.976 INFO:tasks.workunit.client.0.smithi008.stderr:--2020-02-05 17:01:03-- http://download.ceph.com/qa/linux-5.4.tar.gz 2020-02-05T17:01:03.979 INFO:tasks.workunit.client.0.smithi008.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2020-02-05T17:01:04.050 INFO:tasks.workunit.client.0.smithi008.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2020-02-05T17:01:04.119 INFO:tasks.workunit.client.0.smithi008.stderr:HTTP request sent, awaiting response... 200 OK 2020-02-05T17:01:04.120 INFO:tasks.workunit.client.0.smithi008.stderr:Length: 172616875 (165M) [application/octet-stream] 2020-02-05T17:01:04.120 INFO:tasks.workunit.client.0.smithi008.stderr:Saving to: ‘linux.tar.gz’ 2020-02-05T17:01:04.128 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:04.262 INFO:tasks.workunit.client.0.smithi008.stderr: 0K .......... .......... .......... .......... .......... 0% 355K 7m55s 2020-02-05T17:01:04.335 INFO:tasks.workunit.client.0.smithi008.stderr: 50K .......... .......... .......... .......... .......... 0% 700K 5m58s 2020-02-05T17:01:04.343 INFO:tasks.workunit.client.0.smithi008.stderr: 100K .......... .......... .......... .......... .......... 0% 4.90M 4m10s 2020-02-05T17:01:04.369 INFO:tasks.workunit.client.0.smithi008.stderr: 150K .......... .......... .......... .......... .......... 0% 2.05M 3m27s 2020-02-05T17:01:04.375 INFO:tasks.workunit.client.0.smithi008.stderr: 200K .......... .......... .......... .......... .......... 0% 5.81M 2m51s 2020-02-05T17:01:04.381 INFO:tasks.workunit.client.0.smithi008.stderr: 250K .......... .......... .......... .......... .......... 0% 8.12M 2m26s 2020-02-05T17:01:04.386 INFO:tasks.workunit.client.0.smithi008.stderr: 300K .......... .......... .......... .......... .......... 0% 11.1M 2m7s 2020-02-05T17:01:04.391 INFO:tasks.workunit.client.0.smithi008.stderr: 350K .......... .......... .......... .......... .......... 0% 10.1M 1m53s 2020-02-05T17:01:04.397 INFO:tasks.workunit.client.0.smithi008.stderr: 400K .......... .......... .......... .......... .......... 0% 10.4M 1m43s 2020-02-05T17:01:04.399 INFO:tasks.workunit.client.0.smithi008.stderr: 450K .......... .......... .......... .......... .......... 0% 12.8M 94s 2020-02-05T17:01:04.403 INFO:tasks.workunit.client.0.smithi008.stderr: 500K .......... .......... .......... .......... .......... 0% 12.9M 86s 2020-02-05T17:01:04.408 INFO:tasks.workunit.client.0.smithi008.stderr: 550K .......... .......... .......... .......... .......... 0% 11.3M 80s 2020-02-05T17:01:04.412 INFO:tasks.workunit.client.0.smithi008.stderr: 600K .......... .......... .......... .......... .......... 0% 8.89M 75s 2020-02-05T17:01:04.417 INFO:tasks.workunit.client.0.smithi008.stderr: 650K .......... .......... .......... .......... .......... 0% 10.5M 71s 2020-02-05T17:01:04.423 INFO:tasks.workunit.client.0.smithi008.stderr: 700K .......... .......... .......... .......... .......... 0% 10.9M 67s 2020-02-05T17:01:04.428 INFO:tasks.workunit.client.0.smithi008.stderr: 750K .......... .......... .......... .......... .......... 0% 8.83M 64s 2020-02-05T17:01:04.434 INFO:tasks.workunit.client.0.smithi008.stderr: 800K .......... .......... .......... .......... .......... 0% 7.34M 62s 2020-02-05T17:01:04.439 INFO:tasks.workunit.client.0.smithi008.stderr: 850K .......... .......... .......... .......... .......... 0% 8.69M 59s 2020-02-05T17:01:04.444 INFO:tasks.workunit.client.0.smithi008.stderr: 900K .......... .......... .......... .......... .......... 0% 11.4M 57s 2020-02-05T17:01:04.448 INFO:tasks.workunit.client.0.smithi008.stderr: 950K .......... .......... .......... .......... .......... 0% 11.7M 55s 2020-02-05T17:01:04.453 INFO:tasks.workunit.client.0.smithi008.stderr: 1000K .......... .......... .......... .......... .......... 0% 10.1M 53s 2020-02-05T17:01:04.461 INFO:tasks.workunit.client.0.smithi008.stderr: 1050K .......... .......... .......... .......... .......... 0% 11.0M 51s 2020-02-05T17:01:04.462 INFO:tasks.workunit.client.0.smithi008.stderr: 1100K .......... .......... .......... .......... .......... 0% 10.8M 50s 2020-02-05T17:01:04.467 INFO:tasks.workunit.client.0.smithi008.stderr: 1150K .......... .......... .......... .......... .......... 0% 10.2M 48s 2020-02-05T17:01:04.473 INFO:tasks.workunit.client.0.smithi008.stderr: 1200K .......... .......... .......... .......... .......... 0% 7.93M 47s 2020-02-05T17:01:04.478 INFO:tasks.workunit.client.0.smithi008.stderr: 1250K .......... .......... .......... .......... .......... 0% 8.97M 46s 2020-02-05T17:01:04.484 INFO:tasks.workunit.client.0.smithi008.stderr: 1300K .......... .......... .......... .......... .......... 0% 9.15M 45s 2020-02-05T17:01:04.489 INFO:tasks.workunit.client.0.smithi008.stderr: 1350K .......... .......... .......... .......... .......... 0% 9.79M 44s 2020-02-05T17:01:04.495 INFO:tasks.workunit.client.0.smithi008.stderr: 1400K .......... .......... .......... .......... .......... 0% 7.80M 43s 2020-02-05T17:01:04.500 INFO:tasks.workunit.client.0.smithi008.stderr: 1450K .......... .......... .......... .......... .......... 0% 8.89M 42s 2020-02-05T17:01:04.505 INFO:tasks.workunit.client.0.smithi008.stderr: 1500K .......... .......... .......... .......... .......... 0% 9.39M 41s 2020-02-05T17:01:04.514 INFO:tasks.workunit.client.0.smithi008.stderr: 1550K .......... .......... .......... .......... .......... 0% 6.74M 41s 2020-02-05T17:01:04.518 INFO:tasks.workunit.client.0.smithi008.stderr: 1600K .......... .......... .......... .......... .......... 0% 9.87M 40s 2020-02-05T17:01:04.522 INFO:tasks.workunit.client.0.smithi008.stderr: 1650K .......... .......... .......... .......... .......... 1% 11.0M 39s 2020-02-05T17:01:04.526 INFO:tasks.workunit.client.0.smithi008.stderr: 1700K .......... .......... .......... .......... .......... 1% 11.0M 39s 2020-02-05T17:01:04.532 INFO:tasks.workunit.client.0.smithi008.stderr: 1750K .......... .......... .......... .......... .......... 1% 10.1M 38s 2020-02-05T17:01:04.536 INFO:tasks.workunit.client.0.smithi008.stderr: 1800K .......... .......... .......... .......... .......... 1% 9.40M 37s 2020-02-05T17:01:04.542 INFO:tasks.workunit.client.0.smithi008.stderr: 1850K .......... .......... .......... .......... .......... 1% 8.85M 37s 2020-02-05T17:01:04.548 INFO:tasks.workunit.client.0.smithi008.stderr: 1900K .......... .......... .......... .......... .......... 1% 9.11M 36s 2020-02-05T17:01:04.554 INFO:tasks.workunit.client.0.smithi008.stderr: 1950K .......... .......... .......... .......... .......... 1% 8.27M 36s 2020-02-05T17:01:04.561 INFO:tasks.workunit.client.0.smithi008.stderr: 2000K .......... .......... .......... .......... .......... 1% 6.52M 36s 2020-02-05T17:01:04.568 INFO:tasks.workunit.client.0.smithi008.stderr: 2050K .......... .......... .......... .......... .......... 1% 7.27M 35s 2020-02-05T17:01:04.573 INFO:tasks.workunit.client.0.smithi008.stderr: 2100K .......... .......... .......... .......... .......... 1% 9.61M 35s 2020-02-05T17:01:04.577 INFO:tasks.workunit.client.0.smithi008.stderr: 2150K .......... .......... .......... .......... .......... 1% 12.6M 34s 2020-02-05T17:01:04.580 INFO:tasks.workunit.client.0.smithi008.stderr: 2200K .......... .......... .......... .......... .......... 1% 12.8M 34s 2020-02-05T17:01:04.583 INFO:tasks.workunit.client.0.smithi008.stderr: 2250K .......... .......... .......... .......... .......... 1% 16.2M 33s 2020-02-05T17:01:04.587 INFO:tasks.workunit.client.0.smithi008.stderr: 2300K .......... .......... .......... .......... .......... 1% 12.0M 33s 2020-02-05T17:01:04.593 INFO:tasks.workunit.client.0.smithi008.stderr: 2350K .......... .......... .......... .......... .......... 1% 11.0M 33s 2020-02-05T17:01:04.597 INFO:tasks.workunit.client.0.smithi008.stderr: 2400K .......... .......... .......... .......... .......... 1% 9.09M 32s 2020-02-05T17:01:04.602 INFO:tasks.workunit.client.0.smithi008.stderr: 2450K .......... .......... .......... .......... .......... 1% 10.7M 32s 2020-02-05T17:01:04.620 INFO:tasks.workunit.client.0.smithi008.stderr: 2500K .......... .......... .......... .......... .......... 1% 2.86M 32s 2020-02-05T17:01:04.629 INFO:tasks.workunit.client.0.smithi008.stderr: 2550K .......... .......... .......... .......... .......... 1% 5.31M 32s 2020-02-05T17:01:04.653 INFO:tasks.workunit.client.0.smithi008.stderr: 2600K .......... .......... .......... .......... .......... 1% 2.13M 33s 2020-02-05T17:01:04.660 INFO:tasks.workunit.client.0.smithi008.stderr: 2650K .......... .......... .......... .......... .......... 1% 5.67M 33s 2020-02-05T17:01:04.667 INFO:tasks.workunit.client.0.smithi008.stderr: 2700K .......... .......... .......... .......... .......... 1% 6.58M 33s 2020-02-05T17:01:04.674 INFO:tasks.workunit.client.0.smithi008.stderr: 2750K .......... .......... .......... .......... .......... 1% 7.59M 33s 2020-02-05T17:01:04.690 INFO:tasks.workunit.client.0.smithi008.stderr: 2800K .......... .......... .......... .......... .......... 1% 3.10M 33s 2020-02-05T17:01:04.697 INFO:tasks.workunit.client.0.smithi008.stderr: 2850K .......... .......... .......... .......... .......... 1% 6.40M 33s 2020-02-05T17:01:04.713 INFO:tasks.workunit.client.0.smithi008.stderr: 2900K .......... .......... .......... .......... .......... 1% 3.17M 33s 2020-02-05T17:01:04.721 INFO:tasks.workunit.client.0.smithi008.stderr: 2950K .......... .......... .......... .......... .......... 1% 5.87M 33s 2020-02-05T17:01:04.728 INFO:tasks.workunit.client.0.smithi008.stderr: 3000K .......... .......... .......... .......... .......... 1% 6.27M 33s 2020-02-05T17:01:04.735 INFO:tasks.workunit.client.0.smithi008.stderr: 3050K .......... .......... .......... .......... .......... 1% 8.10M 33s 2020-02-05T17:01:04.743 INFO:tasks.workunit.client.0.smithi008.stderr: 3100K .......... .......... .......... .......... .......... 1% 7.11M 33s 2020-02-05T17:01:04.753 INFO:tasks.workunit.client.0.smithi008.stderr: 3150K .......... .......... .......... .......... .......... 1% 4.21M 33s 2020-02-05T17:01:04.766 INFO:tasks.workunit.client.0.smithi008.stderr: 3200K .......... .......... .......... .......... .......... 1% 3.89M 33s 2020-02-05T17:01:04.779 INFO:tasks.workunit.client.0.smithi008.stderr: 3250K .......... .......... .......... .......... .......... 1% 3.56M 33s 2020-02-05T17:01:04.785 INFO:tasks.workunit.client.0.smithi008.stderr: 3300K .......... .......... .......... .......... .......... 1% 8.91M 33s 2020-02-05T17:01:04.790 INFO:tasks.workunit.client.0.smithi008.stderr: 3350K .......... .......... .......... .......... .......... 2% 8.60M 33s 2020-02-05T17:01:04.794 INFO:tasks.workunit.client.0.smithi008.stderr: 3400K .......... .......... .......... .......... .......... 2% 11.5M 32s 2020-02-05T17:01:04.798 INFO:tasks.workunit.client.0.smithi008.stderr: 3450K .......... .......... .......... .......... .......... 2% 12.0M 32s 2020-02-05T17:01:04.814 INFO:tasks.workunit.client.0.smithi008.stderr: 3500K .......... .......... .......... .......... .......... 2% 3.08M 32s 2020-02-05T17:01:04.821 INFO:tasks.workunit.client.0.smithi008.stderr: 3550K .......... .......... .......... .......... .......... 2% 6.99M 32s 2020-02-05T17:01:04.847 INFO:tasks.workunit.client.0.smithi008.stderr: 3600K .......... .......... .......... .......... .......... 2% 2.47M 33s 2020-02-05T17:01:04.848 INFO:tasks.workunit.client.0.smithi008.stderr: 3650K .......... .......... .......... .......... .......... 2% 6.97M 32s 2020-02-05T17:01:04.855 INFO:tasks.workunit.client.0.smithi008.stderr: 3700K .......... .......... .......... .......... .......... 2% 7.95M 32s 2020-02-05T17:01:04.862 INFO:tasks.workunit.client.0.smithi008.stderr: 3750K .......... .......... .......... .......... .......... 2% 6.85M 32s 2020-02-05T17:01:04.868 INFO:tasks.workunit.client.0.smithi008.stderr: 3800K .......... .......... .......... .......... .......... 2% 7.51M 32s 2020-02-05T17:01:04.876 INFO:tasks.workunit.client.0.smithi008.stderr: 3850K .......... .......... .......... .......... .......... 2% 6.67M 32s 2020-02-05T17:01:04.882 INFO:tasks.workunit.client.0.smithi008.stderr: 3900K .......... .......... .......... .......... .......... 2% 7.55M 32s 2020-02-05T17:01:04.890 INFO:tasks.workunit.client.0.smithi008.stderr: 3950K .......... .......... .......... .......... .......... 2% 6.15M 32s 2020-02-05T17:01:04.904 INFO:tasks.workunit.client.0.smithi008.stderr: 4000K .......... .......... .......... .......... .......... 2% 3.53M 32s 2020-02-05T17:01:04.911 INFO:tasks.workunit.client.0.smithi008.stderr: 4050K .......... .......... .......... .......... .......... 2% 6.34M 32s 2020-02-05T17:01:04.917 INFO:tasks.workunit.client.0.smithi008.stderr: 4100K .......... .......... .......... .......... .......... 2% 8.75M 32s 2020-02-05T17:01:04.922 INFO:tasks.workunit.client.0.smithi008.stderr: 4150K .......... .......... .......... .......... .......... 2% 9.06M 31s 2020-02-05T17:01:04.926 INFO:tasks.workunit.client.0.smithi008.stderr: 4200K .......... .......... .......... .......... .......... 2% 11.6M 31s 2020-02-05T17:01:04.932 INFO:tasks.workunit.client.0.smithi008.stderr: 4250K .......... .......... .......... .......... .......... 2% 7.72M 31s 2020-02-05T17:01:04.937 INFO:tasks.workunit.client.0.smithi008.stderr: 4300K .......... .......... .......... .......... .......... 2% 11.2M 31s 2020-02-05T17:01:04.948 INFO:tasks.workunit.client.0.smithi008.stderr: 4350K .......... .......... .......... .......... .......... 2% 4.49M 31s 2020-02-05T17:01:04.959 INFO:tasks.workunit.client.0.smithi008.stderr: 4400K .......... .......... .......... .......... .......... 2% 4.48M 31s 2020-02-05T17:01:04.964 INFO:tasks.workunit.client.0.smithi008.stderr: 4450K .......... .......... .......... .......... .......... 2% 9.55M 31s 2020-02-05T17:01:04.969 INFO:tasks.workunit.client.0.smithi008.stderr: 4500K .......... .......... .......... .......... .......... 2% 10.2M 31s 2020-02-05T17:01:04.973 INFO:tasks.workunit.client.0.smithi008.stderr: 4550K .......... .......... .......... .......... .......... 2% 11.9M 30s 2020-02-05T17:01:04.977 INFO:tasks.workunit.client.0.smithi008.stderr: 4600K .......... .......... .......... .......... .......... 2% 12.2M 30s 2020-02-05T17:01:04.982 INFO:tasks.workunit.client.0.smithi008.stderr: 4650K .......... .......... .......... .......... .......... 2% 9.93M 30s 2020-02-05T17:01:04.991 INFO:tasks.workunit.client.0.smithi008.stderr: 4700K .......... .......... .......... .......... .......... 2% 5.32M 30s 2020-02-05T17:01:05.000 INFO:tasks.workunit.client.0.smithi008.stderr: 4750K .......... .......... .......... .......... .......... 2% 5.35M 30s 2020-02-05T17:01:05.010 INFO:tasks.workunit.client.0.smithi008.stderr: 4800K .......... .......... .......... .......... .......... 2% 4.74M 30s 2020-02-05T17:01:05.016 INFO:tasks.workunit.client.0.smithi008.stderr: 4850K .......... .......... .......... .......... .......... 2% 8.58M 30s 2020-02-05T17:01:05.023 INFO:tasks.workunit.client.0.smithi008.stderr: 4900K .......... .......... .......... .......... .......... 2% 7.64M 30s 2020-02-05T17:01:05.029 INFO:tasks.workunit.client.0.smithi008.stderr: 4950K .......... .......... .......... .......... .......... 2% 7.88M 30s 2020-02-05T17:01:05.035 INFO:tasks.workunit.client.0.smithi008.stderr: 5000K .......... .......... .......... .......... .......... 2% 7.94M 30s 2020-02-05T17:01:05.041 INFO:tasks.workunit.client.0.smithi008.stderr: 5050K .......... .......... .......... .......... .......... 3% 7.75M 30s 2020-02-05T17:01:05.046 INFO:tasks.workunit.client.0.smithi008.stderr: 5100K .......... .......... .......... .......... .......... 3% 9.37M 29s 2020-02-05T17:01:05.052 INFO:tasks.workunit.client.0.smithi008.stderr: 5150K .......... .......... .......... .......... .......... 3% 8.22M 29s 2020-02-05T17:01:05.060 INFO:tasks.workunit.client.0.smithi008.stderr: 5200K .......... .......... .......... .......... .......... 3% 6.32M 29s 2020-02-05T17:01:05.070 INFO:tasks.workunit.client.0.smithi008.stderr: 5250K .......... .......... .......... .......... .......... 3% 4.70M 29s 2020-02-05T17:01:05.076 INFO:tasks.workunit.client.0.smithi008.stderr: 5300K .......... .......... .......... .......... .......... 3% 9.06M 29s 2020-02-05T17:01:05.082 INFO:tasks.workunit.client.0.smithi008.stderr: 5350K .......... .......... .......... .......... .......... 3% 8.05M 29s 2020-02-05T17:01:05.086 INFO:tasks.workunit.client.0.smithi008.stderr: 5400K .......... .......... .......... .......... .......... 3% 10.2M 29s 2020-02-05T17:01:05.091 INFO:tasks.workunit.client.0.smithi008.stderr: 5450K .......... .......... .......... .......... .......... 3% 9.27M 29s 2020-02-05T17:01:05.095 INFO:tasks.workunit.client.0.smithi008.stderr: 5500K .......... .......... .......... .......... .......... 3% 14.1M 29s 2020-02-05T17:01:05.102 INFO:tasks.workunit.client.0.smithi008.stderr: 5550K .......... .......... .......... .......... .......... 3% 6.97M 29s 2020-02-05T17:01:05.108 INFO:tasks.workunit.client.0.smithi008.stderr: 5600K .......... .......... .......... .......... .......... 3% 9.76M 28s 2020-02-05T17:01:05.115 INFO:tasks.workunit.client.0.smithi008.stderr: 5650K .......... .......... .......... .......... .......... 3% 6.75M 28s 2020-02-05T17:01:05.127 INFO:tasks.workunit.client.0.smithi008.stderr: 5700K .......... .......... .......... .......... .......... 3% 4.32M 28s 2020-02-05T17:01:05.133 INFO:tasks.workunit.client.0.smithi008.stderr: 5750K .......... .......... .......... .......... .......... 3% 7.16M 28s 2020-02-05T17:01:05.139 INFO:tasks.workunit.client.0.smithi008.stderr: 5800K .......... .......... .......... .......... .......... 3% 7.70M 28s 2020-02-05T17:01:05.148 INFO:tasks.workunit.client.0.smithi008.stderr: 5850K .......... .......... .......... .......... .......... 3% 5.96M 28s 2020-02-05T17:01:05.154 INFO:tasks.workunit.client.0.smithi008.stderr: 5900K .......... .......... .......... .......... .......... 3% 7.43M 28s 2020-02-05T17:01:05.160 INFO:tasks.workunit.client.0.smithi008.stderr: 5950K .......... .......... .......... .......... .......... 3% 7.88M 28s 2020-02-05T17:01:05.167 INFO:tasks.workunit.client.0.smithi008.stderr: 6000K .......... .......... .......... .......... .......... 3% 7.27M 28s 2020-02-05T17:01:05.175 INFO:tasks.workunit.client.0.smithi008.stderr: 6050K .......... .......... .......... .......... .......... 3% 5.73M 28s 2020-02-05T17:01:05.182 INFO:tasks.workunit.client.0.smithi008.stderr: 6100K .......... .......... .......... .......... .......... 3% 7.21M 28s 2020-02-05T17:01:05.190 INFO:tasks.workunit.client.0.smithi008.stderr: 6150K .......... .......... .......... .......... .......... 3% 6.35M 28s 2020-02-05T17:01:05.198 INFO:tasks.workunit.client.0.smithi008.stderr: 6200K .......... .......... .......... .......... .......... 3% 5.98M 28s 2020-02-05T17:01:05.207 INFO:tasks.workunit.client.0.smithi008.stderr: 6250K .......... .......... .......... .......... .......... 3% 5.92M 28s 2020-02-05T17:01:05.212 INFO:tasks.workunit.client.0.smithi008.stderr: 6300K .......... .......... .......... .......... .......... 3% 8.14M 28s 2020-02-05T17:01:05.217 INFO:tasks.workunit.client.0.smithi008.stderr: 6350K .......... .......... .......... .......... .......... 3% 10.8M 28s 2020-02-05T17:01:05.222 INFO:tasks.workunit.client.0.smithi008.stderr: 6400K .......... .......... .......... .......... .......... 3% 9.33M 28s 2020-02-05T17:01:05.228 INFO:tasks.workunit.client.0.smithi008.stderr: 6450K .......... .......... .......... .......... .......... 3% 7.68M 28s 2020-02-05T17:01:05.234 INFO:tasks.workunit.client.0.smithi008.stderr: 6500K .......... .......... .......... .......... .......... 3% 9.06M 28s 2020-02-05T17:01:05.240 INFO:tasks.workunit.client.0.smithi008.stderr: 6550K .......... .......... .......... .......... .......... 3% 7.95M 27s 2020-02-05T17:01:05.249 INFO:tasks.workunit.client.0.smithi008.stderr: 6600K .......... .......... .......... .......... .......... 3% 7.17M 27s 2020-02-05T17:01:05.256 INFO:tasks.workunit.client.0.smithi008.stderr: 6650K .......... .......... .......... .......... .......... 3% 5.55M 27s 2020-02-05T17:01:05.263 INFO:tasks.workunit.client.0.smithi008.stderr: 6700K .......... .......... .......... .......... .......... 4% 6.66M 27s 2020-02-05T17:01:05.271 INFO:tasks.workunit.client.0.smithi008.stderr: 6750K .......... .......... .......... .......... .......... 4% 6.48M 27s 2020-02-05T17:01:05.279 INFO:tasks.workunit.client.0.smithi008.stderr: 6800K .......... .......... .......... .......... .......... 4% 6.06M 27s 2020-02-05T17:01:05.294 INFO:tasks.workunit.client.0.smithi008.stderr: 6850K .......... .......... .......... .......... .......... 4% 3.37M 27s 2020-02-05T17:01:05.299 INFO:tasks.workunit.client.0.smithi008.stderr: 6900K .......... .......... .......... .......... .......... 4% 8.07M 27s 2020-02-05T17:01:05.303 INFO:tasks.workunit.client.0.smithi008.stderr: 6950K .......... .......... .......... .......... .......... 4% 10.4M 27s 2020-02-05T17:01:05.308 INFO:tasks.workunit.client.0.smithi008.stderr: 7000K .......... .......... .......... .......... .......... 4% 12.0M 27s 2020-02-05T17:01:05.313 INFO:tasks.workunit.client.0.smithi008.stderr: 7050K .......... .......... .......... .......... .......... 4% 9.29M 27s 2020-02-05T17:01:05.319 INFO:tasks.workunit.client.0.smithi008.stderr: 7100K .......... .......... .......... .......... .......... 4% 8.20M 27s 2020-02-05T17:01:05.325 INFO:tasks.workunit.client.0.smithi008.stderr: 7150K .......... .......... .......... .......... .......... 4% 8.50M 27s 2020-02-05T17:01:05.330 INFO:tasks.workunit.client.0.smithi008.stderr: 7200K .......... .......... .......... .......... .......... 4% 9.31M 27s 2020-02-05T17:01:05.338 INFO:tasks.workunit.client.0.smithi008.stderr: 7250K .......... .......... .......... .......... .......... 4% 7.02M 27s 2020-02-05T17:01:05.351 INFO:tasks.workunit.client.0.smithi008.stderr: 7300K .......... .......... .......... .......... .......... 4% 9.91M 27s 2020-02-05T17:01:05.352 INFO:tasks.workunit.client.0.smithi008.stderr: 7350K .......... .......... .......... .......... .......... 4% 8.61M 27s 2020-02-05T17:01:05.358 INFO:tasks.workunit.client.0.smithi008.stderr: 7400K .......... .......... .......... .......... .......... 4% 7.63M 27s 2020-02-05T17:01:05.362 INFO:tasks.workunit.client.0.smithi008.stderr: 7450K .......... .......... .......... .......... .......... 4% 6.57M 27s 2020-02-05T17:01:05.368 INFO:tasks.workunit.client.0.smithi008.stderr: 7500K .......... .......... .......... .......... .......... 4% 7.37M 27s 2020-02-05T17:01:05.374 INFO:tasks.workunit.client.0.smithi008.stderr: 7550K .......... .......... .......... .......... .......... 4% 7.60M 27s 2020-02-05T17:01:05.380 INFO:tasks.workunit.client.0.smithi008.stderr: 7600K .......... .......... .......... .......... .......... 4% 8.83M 26s 2020-02-05T17:01:05.388 INFO:tasks.workunit.client.0.smithi008.stderr: 7650K .......... .......... .......... .......... .......... 4% 5.92M 26s 2020-02-05T17:01:05.396 INFO:tasks.workunit.client.0.smithi008.stderr: 7700K .......... .......... .......... .......... .......... 4% 6.82M 26s 2020-02-05T17:01:05.404 INFO:tasks.workunit.client.0.smithi008.stderr: 7750K .......... .......... .......... .......... .......... 4% 6.23M 26s 2020-02-05T17:01:05.412 INFO:tasks.workunit.client.0.smithi008.stderr: 7800K .......... .......... .......... .......... .......... 4% 6.07M 26s 2020-02-05T17:01:05.421 INFO:tasks.workunit.client.0.smithi008.stderr: 7850K .......... .......... .......... .......... .......... 4% 4.97M 26s 2020-02-05T17:01:05.429 INFO:tasks.workunit.client.0.smithi008.stderr: 7900K .......... .......... .......... .......... .......... 4% 6.50M 26s 2020-02-05T17:01:05.437 INFO:tasks.workunit.client.0.smithi008.stderr: 7950K .......... .......... .......... .......... .......... 4% 6.45M 26s 2020-02-05T17:01:05.445 INFO:tasks.workunit.client.0.smithi008.stderr: 8000K .......... .......... .......... .......... .......... 4% 5.59M 26s 2020-02-05T17:01:05.455 INFO:tasks.workunit.client.0.smithi008.stderr: 8050K .......... .......... .......... .......... .......... 4% 5.27M 26s 2020-02-05T17:01:05.463 INFO:tasks.workunit.client.0.smithi008.stderr: 8100K .......... .......... .......... .......... .......... 4% 5.75M 26s 2020-02-05T17:01:05.471 INFO:tasks.workunit.client.0.smithi008.stderr: 8150K .......... .......... .......... .......... .......... 4% 5.95M 26s 2020-02-05T17:01:05.479 INFO:tasks.workunit.client.0.smithi008.stderr: 8200K .......... .......... .......... .......... .......... 4% 5.94M 26s 2020-02-05T17:01:05.486 INFO:tasks.workunit.client.0.smithi008.stderr: 8250K .......... .......... .......... .......... .......... 4% 7.26M 26s 2020-02-05T17:01:05.490 INFO:tasks.workunit.client.0.smithi008.stderr: 8300K .......... .......... .......... .......... .......... 4% 11.4M 26s 2020-02-05T17:01:05.495 INFO:tasks.workunit.client.0.smithi008.stderr: 8350K .......... .......... .......... .......... .......... 4% 10.3M 26s 2020-02-05T17:01:05.498 INFO:tasks.workunit.client.0.smithi008.stderr: 8400K .......... .......... .......... .......... .......... 5% 12.6M 26s 2020-02-05T17:01:05.502 INFO:tasks.workunit.client.0.smithi008.stderr: 8450K .......... .......... .......... .......... .......... 5% 13.5M 26s 2020-02-05T17:01:05.505 INFO:tasks.workunit.client.0.smithi008.stderr: 8500K .......... .......... .......... .......... .......... 5% 16.0M 26s 2020-02-05T17:01:05.509 INFO:tasks.workunit.client.0.smithi008.stderr: 8550K .......... .......... .......... .......... .......... 5% 12.9M 26s 2020-02-05T17:01:05.513 INFO:tasks.workunit.client.0.smithi008.stderr: 8600K .......... .......... .......... .......... .......... 5% 12.2M 26s 2020-02-05T17:01:05.517 INFO:tasks.workunit.client.0.smithi008.stderr: 8650K .......... .......... .......... .......... .......... 5% 11.0M 26s 2020-02-05T17:01:05.523 INFO:tasks.workunit.client.0.smithi008.stderr: 8700K .......... .......... .......... .......... .......... 5% 8.36M 26s 2020-02-05T17:01:05.528 INFO:tasks.workunit.client.0.smithi008.stderr: 8750K .......... .......... .......... .......... .......... 5% 10.2M 26s 2020-02-05T17:01:05.533 INFO:tasks.workunit.client.0.smithi008.stderr: 8800K .......... .......... .......... .......... .......... 5% 10.7M 25s 2020-02-05T17:01:05.538 INFO:tasks.workunit.client.0.smithi008.stderr: 8850K .......... .......... .......... .......... .......... 5% 8.67M 25s 2020-02-05T17:01:05.543 INFO:tasks.workunit.client.0.smithi008.stderr: 8900K .......... .......... .......... .......... .......... 5% 9.44M 25s 2020-02-05T17:01:05.549 INFO:tasks.workunit.client.0.smithi008.stderr: 8950K .......... .......... .......... .......... .......... 5% 9.11M 25s 2020-02-05T17:01:05.556 INFO:tasks.workunit.client.0.smithi008.stderr: 9000K .......... .......... .......... .......... .......... 5% 7.09M 25s 2020-02-05T17:01:05.564 INFO:tasks.workunit.client.0.smithi008.stderr: 9050K .......... .......... .......... .......... .......... 5% 6.25M 25s 2020-02-05T17:01:05.569 INFO:tasks.workunit.client.0.smithi008.stderr: 9100K .......... .......... .......... .......... .......... 5% 8.32M 25s 2020-02-05T17:01:05.572 INFO:tasks.workunit.client.0.smithi008.stderr: 9150K .......... .......... .......... .......... .......... 5% 17.4M 25s 2020-02-05T17:01:05.576 INFO:tasks.workunit.client.0.smithi008.stderr: 9200K .......... .......... .......... .......... .......... 5% 12.7M 25s 2020-02-05T17:01:05.580 INFO:tasks.workunit.client.0.smithi008.stderr: 9250K .......... .......... .......... .......... .......... 5% 11.0M 25s 2020-02-05T17:01:05.585 INFO:tasks.workunit.client.0.smithi008.stderr: 9300K .......... .......... .......... .......... .......... 5% 10.9M 25s 2020-02-05T17:01:05.589 INFO:tasks.workunit.client.0.smithi008.stderr: 9350K .......... .......... .......... .......... .......... 5% 12.5M 25s 2020-02-05T17:01:05.593 INFO:tasks.workunit.client.0.smithi008.stderr: 9400K .......... .......... .......... .......... .......... 5% 11.2M 25s 2020-02-05T17:01:05.598 INFO:tasks.workunit.client.0.smithi008.stderr: 9450K .......... .......... .......... .......... .......... 5% 8.99M 25s 2020-02-05T17:01:05.604 INFO:tasks.workunit.client.0.smithi008.stderr: 9500K .......... .......... .......... .......... .......... 5% 9.34M 25s 2020-02-05T17:01:05.610 INFO:tasks.workunit.client.0.smithi008.stderr: 9550K .......... .......... .......... .......... .......... 5% 7.95M 25s 2020-02-05T17:01:05.616 INFO:tasks.workunit.client.0.smithi008.stderr: 9600K .......... .......... .......... .......... .......... 5% 9.04M 25s 2020-02-05T17:01:05.622 INFO:tasks.workunit.client.0.smithi008.stderr: 9650K .......... .......... .......... .......... .......... 5% 7.28M 25s 2020-02-05T17:01:05.628 INFO:tasks.workunit.client.0.smithi008.stderr: 9700K .......... .......... .......... .......... .......... 5% 8.05M 25s 2020-02-05T17:01:05.635 INFO:tasks.workunit.client.0.smithi008.stderr: 9750K .......... .......... .......... .......... .......... 5% 7.91M 25s 2020-02-05T17:01:05.642 INFO:tasks.workunit.client.0.smithi008.stderr: 9800K .......... .......... .......... .......... .......... 5% 7.01M 25s 2020-02-05T17:01:05.650 INFO:tasks.workunit.client.0.smithi008.stderr: 9850K .......... .......... .......... .......... .......... 5% 6.07M 25s 2020-02-05T17:01:05.659 INFO:tasks.workunit.client.0.smithi008.stderr: 9900K .......... .......... .......... .......... .......... 5% 5.20M 25s 2020-02-05T17:01:05.666 INFO:tasks.workunit.client.0.smithi008.stderr: 9950K .......... .......... .......... .......... .......... 5% 7.06M 24s 2020-02-05T17:01:05.672 INFO:tasks.workunit.client.0.smithi008.stderr: 10000K .......... .......... .......... .......... .......... 5% 7.70M 24s 2020-02-05T17:01:05.679 INFO:tasks.workunit.client.0.smithi008.stderr: 10050K .......... .......... .......... .......... .......... 5% 7.07M 24s 2020-02-05T17:01:05.684 INFO:tasks.workunit.client.0.smithi008.stderr: 10100K .......... .......... .......... .......... .......... 6% 9.76M 24s 2020-02-05T17:01:05.690 INFO:tasks.workunit.client.0.smithi008.stderr: 10150K .......... .......... .......... .......... .......... 6% 8.91M 24s 2020-02-05T17:01:05.695 INFO:tasks.workunit.client.0.smithi008.stderr: 10200K .......... .......... .......... .......... .......... 6% 8.93M 24s 2020-02-05T17:01:05.702 INFO:tasks.workunit.client.0.smithi008.stderr: 10250K .......... .......... .......... .......... .......... 6% 7.65M 24s 2020-02-05T17:01:05.707 INFO:tasks.workunit.client.0.smithi008.stderr: 10300K .......... .......... .......... .......... .......... 6% 8.27M 24s 2020-02-05T17:01:05.713 INFO:tasks.workunit.client.0.smithi008.stderr: 10350K .......... .......... .......... .......... .......... 6% 8.30M 24s 2020-02-05T17:01:05.719 INFO:tasks.workunit.client.0.smithi008.stderr: 10400K .......... .......... .......... .......... .......... 6% 8.55M 24s 2020-02-05T17:01:05.727 INFO:tasks.workunit.client.0.smithi008.stderr: 10450K .......... .......... .......... .......... .......... 6% 6.08M 24s 2020-02-05T17:01:05.734 INFO:tasks.workunit.client.0.smithi008.stderr: 10500K .......... .......... .......... .......... .......... 6% 6.95M 24s 2020-02-05T17:01:05.742 INFO:tasks.workunit.client.0.smithi008.stderr: 10550K .......... .......... .......... .......... .......... 6% 6.58M 24s 2020-02-05T17:01:05.749 INFO:tasks.workunit.client.0.smithi008.stderr: 10600K .......... .......... .......... .......... .......... 6% 6.70M 24s 2020-02-05T17:01:05.757 INFO:tasks.workunit.client.0.smithi008.stderr: 10650K .......... .......... .......... .......... .......... 6% 5.97M 24s 2020-02-05T17:01:05.764 INFO:tasks.workunit.client.0.smithi008.stderr: 10700K .......... .......... .......... .......... .......... 6% 6.70M 24s 2020-02-05T17:01:05.770 INFO:tasks.workunit.client.0.smithi008.stderr: 10750K .......... .......... .......... .......... .......... 6% 7.82M 24s 2020-02-05T17:01:05.777 INFO:tasks.workunit.client.0.smithi008.stderr: 10800K .......... .......... .......... .......... .......... 6% 7.65M 24s 2020-02-05T17:01:05.783 INFO:tasks.workunit.client.0.smithi008.stderr: 10850K .......... .......... .......... .......... .......... 6% 7.12M 24s 2020-02-05T17:01:05.790 INFO:tasks.workunit.client.0.smithi008.stderr: 10900K .......... .......... .......... .......... .......... 6% 8.16M 24s 2020-02-05T17:01:05.796 INFO:tasks.workunit.client.0.smithi008.stderr: 10950K .......... .......... .......... .......... .......... 6% 7.39M 24s 2020-02-05T17:01:05.803 INFO:tasks.workunit.client.0.smithi008.stderr: 11000K .......... .......... .......... .......... .......... 6% 7.21M 24s 2020-02-05T17:01:05.811 INFO:tasks.workunit.client.0.smithi008.stderr: 11050K .......... .......... .......... .......... .......... 6% 5.99M 24s 2020-02-05T17:01:05.819 INFO:tasks.workunit.client.0.smithi008.stderr: 11100K .......... .......... .......... .......... .......... 6% 6.41M 24s 2020-02-05T17:01:05.827 INFO:tasks.workunit.client.0.smithi008.stderr: 11150K .......... .......... .......... .......... .......... 6% 6.23M 24s 2020-02-05T17:01:05.835 INFO:tasks.workunit.client.0.smithi008.stderr: 11200K .......... .......... .......... .......... .......... 6% 6.03M 24s 2020-02-05T17:01:05.845 INFO:tasks.workunit.client.0.smithi008.stderr: 11250K .......... .......... .......... .......... .......... 6% 5.28M 24s 2020-02-05T17:01:05.852 INFO:tasks.workunit.client.0.smithi008.stderr: 11300K .......... .......... .......... .......... .......... 6% 6.05M 24s 2020-02-05T17:01:05.861 INFO:tasks.workunit.client.0.smithi008.stderr: 11350K .......... .......... .......... .......... .......... 6% 5.77M 24s 2020-02-05T17:01:05.870 INFO:tasks.workunit.client.0.smithi008.stderr: 11400K .......... .......... .......... .......... .......... 6% 5.36M 24s 2020-02-05T17:01:05.881 INFO:tasks.workunit.client.0.smithi008.stderr: 11450K .......... .......... .......... .......... .......... 6% 4.71M 24s 2020-02-05T17:01:05.891 INFO:tasks.workunit.client.0.smithi008.stderr: 11500K .......... .......... .......... .......... .......... 6% 4.77M 24s 2020-02-05T17:01:05.907 INFO:tasks.workunit.client.0.smithi008.stderr: 11550K .......... .......... .......... .......... .......... 6% 4.90M 24s 2020-02-05T17:01:05.911 INFO:tasks.workunit.client.0.smithi008.stderr: 11600K .......... .......... .......... .......... .......... 6% 4.90M 24s 2020-02-05T17:01:05.921 INFO:tasks.workunit.client.0.smithi008.stderr: 11650K .......... .......... .......... .......... .......... 6% 4.80M 24s 2020-02-05T17:01:05.929 INFO:tasks.workunit.client.0.smithi008.stderr: 11700K .......... .......... .......... .......... .......... 6% 5.82M 24s 2020-02-05T17:01:05.942 INFO:tasks.workunit.client.0.smithi008.stderr: 11750K .......... .......... .......... .......... .......... 7% 3.74M 24s 2020-02-05T17:01:05.948 INFO:tasks.workunit.client.0.smithi008.stderr: 11800K .......... .......... .......... .......... .......... 7% 8.23M 24s 2020-02-05T17:01:05.955 INFO:tasks.workunit.client.0.smithi008.stderr: 11850K .......... .......... .......... .......... .......... 7% 6.69M 24s 2020-02-05T17:01:05.961 INFO:tasks.workunit.client.0.smithi008.stderr: 11900K .......... .......... .......... .......... .......... 7% 8.71M 24s 2020-02-05T17:01:05.966 INFO:tasks.workunit.client.0.smithi008.stderr: 11950K .......... .......... .......... .......... .......... 7% 8.28M 24s 2020-02-05T17:01:05.972 INFO:tasks.workunit.client.0.smithi008.stderr: 12000K .......... .......... .......... .......... .......... 7% 8.44M 24s 2020-02-05T17:01:05.978 INFO:tasks.workunit.client.0.smithi008.stderr: 12050K .......... .......... .......... .......... .......... 7% 8.71M 24s 2020-02-05T17:01:05.983 INFO:tasks.workunit.client.0.smithi008.stderr: 12100K .......... .......... .......... .......... .......... 7% 9.84M 24s 2020-02-05T17:01:05.989 INFO:tasks.workunit.client.0.smithi008.stderr: 12150K .......... .......... .......... .......... .......... 7% 7.80M 24s 2020-02-05T17:01:05.996 INFO:tasks.workunit.client.0.smithi008.stderr: 12200K .......... .......... .......... .......... .......... 7% 7.25M 24s 2020-02-05T17:01:06.002 INFO:tasks.workunit.client.0.smithi008.stderr: 12250K .......... .......... .......... .......... .......... 7% 7.17M 24s 2020-02-05T17:01:06.008 INFO:tasks.workunit.client.0.smithi008.stderr: 12300K .......... .......... .......... .......... .......... 7% 9.43M 24s 2020-02-05T17:01:06.014 INFO:tasks.workunit.client.0.smithi008.stderr: 12350K .......... .......... .......... .......... .......... 7% 8.60M 24s 2020-02-05T17:01:06.019 INFO:tasks.workunit.client.0.smithi008.stderr: 12400K .......... .......... .......... .......... .......... 7% 8.02M 24s 2020-02-05T17:01:06.047 INFO:tasks.workunit.client.0.smithi008.stderr: 12450K .......... .......... .......... .......... .......... 7% 7.59M 24s 2020-02-05T17:01:06.047 INFO:tasks.workunit.client.0.smithi008.stderr: 12500K .......... .......... .......... .......... .......... 7% 11.1M 24s 2020-02-05T17:01:06.047 INFO:tasks.workunit.client.0.smithi008.stderr: 12550K .......... .......... .......... .......... .......... 7% 8.95M 24s 2020-02-05T17:01:06.047 INFO:tasks.workunit.client.0.smithi008.stderr: 12600K .......... .......... .......... .......... .......... 7% 9.25M 24s 2020-02-05T17:01:06.047 INFO:tasks.workunit.client.0.smithi008.stderr: 12650K .......... .......... .......... .......... .......... 7% 11.1M 24s 2020-02-05T17:01:06.048 INFO:tasks.workunit.client.0.smithi008.stderr: 12700K .......... .......... .......... .......... .......... 7% 15.6M 24s 2020-02-05T17:01:06.059 INFO:tasks.workunit.client.0.smithi008.stderr: 12750K .......... .......... .......... .......... .......... 7% 13.5M 24s 2020-02-05T17:01:06.059 INFO:tasks.workunit.client.0.smithi008.stderr: 12800K .......... .......... .......... .......... .......... 7% 11.7M 23s 2020-02-05T17:01:06.061 INFO:tasks.workunit.client.0.smithi008.stderr: 12850K .......... .......... .......... .......... .......... 7% 9.97M 23s 2020-02-05T17:01:06.066 INFO:tasks.workunit.client.0.smithi008.stderr: 12900K .......... .......... .......... .......... .......... 7% 10.7M 23s 2020-02-05T17:01:06.071 INFO:tasks.workunit.client.0.smithi008.stderr: 12950K .......... .......... .......... .......... .......... 7% 9.85M 23s 2020-02-05T17:01:06.076 INFO:tasks.workunit.client.0.smithi008.stderr: 13000K .......... .......... .......... .......... .......... 7% 9.37M 23s 2020-02-05T17:01:06.083 INFO:tasks.workunit.client.0.smithi008.stderr: 13050K .......... .......... .......... .......... .......... 7% 6.90M 23s 2020-02-05T17:01:06.089 INFO:tasks.workunit.client.0.smithi008.stderr: 13100K .......... .......... .......... .......... .......... 7% 8.01M 23s 2020-02-05T17:01:06.095 INFO:tasks.workunit.client.0.smithi008.stderr: 13150K .......... .......... .......... .......... .......... 7% 8.99M 23s 2020-02-05T17:01:06.099 INFO:tasks.workunit.client.0.smithi008.stderr: 13200K .......... .......... .......... .......... .......... 7% 11.3M 23s 2020-02-05T17:01:06.103 INFO:tasks.workunit.client.0.smithi008.stderr: 13250K .......... .......... .......... .......... .......... 7% 10.4M 23s 2020-02-05T17:01:06.107 INFO:tasks.workunit.client.0.smithi008.stderr: 13300K .......... .......... .......... .......... .......... 7% 12.2M 23s 2020-02-05T17:01:06.112 INFO:tasks.workunit.client.0.smithi008.stderr: 13350K .......... .......... .......... .......... .......... 7% 11.6M 23s 2020-02-05T17:01:06.116 INFO:tasks.workunit.client.0.smithi008.stderr: 13400K .......... .......... .......... .......... .......... 7% 11.7M 23s 2020-02-05T17:01:06.121 INFO:tasks.workunit.client.0.smithi008.stderr: 13450K .......... .......... .......... .......... .......... 8% 10.1M 23s 2020-02-05T17:01:06.125 INFO:tasks.workunit.client.0.smithi008.stderr: 13500K .......... .......... .......... .......... .......... 8% 11.5M 23s 2020-02-05T17:01:06.129 INFO:tasks.workunit.client.0.smithi008.stderr: 13550K .......... .......... .......... .......... .......... 8% 11.3M 23s 2020-02-05T17:01:06.133 INFO:tasks.workunit.client.0.smithi008.stderr: 13600K .......... .......... .......... .......... .......... 8% 11.7M 23s 2020-02-05T17:01:06.138 INFO:tasks.workunit.client.0.smithi008.stderr: 13650K .......... .......... .......... .......... .......... 8% 9.94M 23s 2020-02-05T17:01:06.144 INFO:tasks.workunit.client.0.smithi008.stderr: 13700K .......... .......... .......... .......... .......... 8% 8.94M 23s 2020-02-05T17:01:06.151 INFO:tasks.workunit.client.0.smithi008.stderr: 13750K .......... .......... .......... .......... .......... 8% 7.09M 23s 2020-02-05T17:01:06.158 INFO:tasks.workunit.client.0.smithi008.stderr: 13800K .......... .......... .......... .......... .......... 8% 7.16M 23s 2020-02-05T17:01:06.165 INFO:tasks.workunit.client.0.smithi008.stderr: 13850K .......... .......... .......... .......... .......... 8% 6.39M 23s 2020-02-05T17:01:06.172 INFO:tasks.workunit.client.0.smithi008.stderr: 13900K .......... .......... .......... .......... .......... 8% 7.31M 23s 2020-02-05T17:01:06.179 INFO:tasks.workunit.client.0.smithi008.stderr: 13950K .......... .......... .......... .......... .......... 8% 7.14M 23s 2020-02-05T17:01:06.190 INFO:tasks.workunit.client.0.smithi008.stderr: 14000K .......... .......... .......... .......... .......... 8% 4.59M 23s 2020-02-05T17:01:06.196 INFO:tasks.workunit.client.0.smithi008.stderr: 14050K .......... .......... .......... .......... .......... 8% 7.18M 23s 2020-02-05T17:01:06.201 INFO:tasks.workunit.client.0.smithi008.stderr: 14100K .......... .......... .......... .......... .......... 8% 9.83M 23s 2020-02-05T17:01:06.206 INFO:tasks.workunit.client.0.smithi008.stderr: 14150K .......... .......... .......... .......... .......... 8% 9.73M 23s 2020-02-05T17:01:06.212 INFO:tasks.workunit.client.0.smithi008.stderr: 14200K .......... .......... .......... .......... .......... 8% 9.03M 23s 2020-02-05T17:01:06.219 INFO:tasks.workunit.client.0.smithi008.stderr: 14250K .......... .......... .......... .......... .......... 8% 6.34M 23s 2020-02-05T17:01:06.226 INFO:tasks.workunit.client.0.smithi008.stderr: 14300K .......... .......... .......... .......... .......... 8% 7.09M 23s 2020-02-05T17:01:06.233 INFO:tasks.workunit.client.0.smithi008.stderr: 14350K .......... .......... .......... .......... .......... 8% 7.10M 23s 2020-02-05T17:01:06.240 INFO:tasks.workunit.client.0.smithi008.stderr: 14400K .......... .......... .......... .......... .......... 8% 7.07M 23s 2020-02-05T17:01:06.248 INFO:tasks.workunit.client.0.smithi008.stderr: 14450K .......... .......... .......... .......... .......... 8% 6.22M 23s 2020-02-05T17:01:06.254 INFO:tasks.workunit.client.0.smithi008.stderr: 14500K .......... .......... .......... .......... .......... 8% 7.89M 23s 2020-02-05T17:01:06.260 INFO:tasks.workunit.client.0.smithi008.stderr: 14550K .......... .......... .......... .......... .......... 8% 7.70M 23s 2020-02-05T17:01:06.269 INFO:tasks.workunit.client.0.smithi008.stderr: 14600K .......... .......... .......... .......... .......... 8% 7.03M 23s 2020-02-05T17:01:06.276 INFO:tasks.workunit.client.0.smithi008.stderr: 14650K .......... .......... .......... .......... .......... 8% 6.29M 23s 2020-02-05T17:01:06.281 INFO:tasks.workunit.client.0.smithi008.stderr: 14700K .......... .......... .......... .......... .......... 8% 8.62M 23s 2020-02-05T17:01:06.286 INFO:tasks.workunit.client.0.smithi008.stderr: 14750K .......... .......... .......... .......... .......... 8% 9.07M 22s 2020-02-05T17:01:06.292 INFO:tasks.workunit.client.0.smithi008.stderr: 14800K .......... .......... .......... .......... .......... 8% 8.65M 22s 2020-02-05T17:01:06.298 INFO:tasks.workunit.client.0.smithi008.stderr: 14850K .......... .......... .......... .......... .......... 8% 7.80M 22s 2020-02-05T17:01:06.303 INFO:tasks.workunit.client.0.smithi008.stderr: 14900K .......... .......... .......... .......... .......... 8% 8.68M 22s 2020-02-05T17:01:06.308 INFO:tasks.workunit.client.0.smithi008.stderr: 14950K .......... .......... .......... .......... .......... 8% 10.7M 22s 2020-02-05T17:01:06.313 INFO:tasks.workunit.client.0.smithi008.stderr: 15000K .......... .......... .......... .......... .......... 8% 10.6M 22s 2020-02-05T17:01:06.318 INFO:tasks.workunit.client.0.smithi008.stderr: 15050K .......... .......... .......... .......... .......... 8% 9.01M 22s 2020-02-05T17:01:06.323 INFO:tasks.workunit.client.0.smithi008.stderr: 15100K .......... .......... .......... .......... .......... 8% 10.6M 22s 2020-02-05T17:01:06.328 INFO:tasks.workunit.client.0.smithi008.stderr: 15150K .......... .......... .......... .......... .......... 9% 10.1M 22s 2020-02-05T17:01:06.334 INFO:tasks.workunit.client.0.smithi008.stderr: 15200K .......... .......... .......... .......... .......... 9% 8.34M 22s 2020-02-05T17:01:06.341 INFO:tasks.workunit.client.0.smithi008.stderr: 15250K .......... .......... .......... .......... .......... 9% 6.38M 22s 2020-02-05T17:01:06.348 INFO:tasks.workunit.client.0.smithi008.stderr: 15300K .......... .......... .......... .......... .......... 9% 7.11M 22s 2020-02-05T17:01:06.354 INFO:tasks.workunit.client.0.smithi008.stderr: 15350K .......... .......... .......... .......... .......... 9% 7.93M 22s 2020-02-05T17:01:06.360 INFO:tasks.workunit.client.0.smithi008.stderr: 15400K .......... .......... .......... .......... .......... 9% 7.89M 22s 2020-02-05T17:01:06.367 INFO:tasks.workunit.client.0.smithi008.stderr: 15450K .......... .......... .......... .......... .......... 9% 7.65M 22s 2020-02-05T17:01:06.371 INFO:tasks.workunit.client.0.smithi008.stderr: 15500K .......... .......... .......... .......... .......... 9% 11.2M 22s 2020-02-05T17:01:06.374 INFO:tasks.workunit.client.0.smithi008.stderr: 15550K .......... .......... .......... .......... .......... 9% 14.2M 22s 2020-02-05T17:01:06.378 INFO:tasks.workunit.client.0.smithi008.stderr: 15600K .......... .......... .......... .......... .......... 9% 14.5M 22s 2020-02-05T17:01:06.381 INFO:tasks.workunit.client.0.smithi008.stderr: 15650K .......... .......... .......... .......... .......... 9% 12.6M 22s 2020-02-05T17:01:06.385 INFO:tasks.workunit.client.0.smithi008.stderr: 15700K .......... .......... .......... .......... .......... 9% 13.9M 22s 2020-02-05T17:01:06.389 INFO:tasks.workunit.client.0.smithi008.stderr: 15750K .......... .......... .......... .......... .......... 9% 14.3M 22s 2020-02-05T17:01:06.392 INFO:tasks.workunit.client.0.smithi008.stderr: 15800K .......... .......... .......... .......... .......... 9% 14.3M 22s 2020-02-05T17:01:06.400 INFO:tasks.workunit.client.0.smithi008.stderr: 15850K .......... .......... .......... .......... .......... 9% 6.77M 22s 2020-02-05T17:01:06.402 INFO:tasks.workunit.client.0.smithi008.stderr: 15900K .......... .......... .......... .......... .......... 9% 18.9M 22s 2020-02-05T17:01:06.404 INFO:tasks.workunit.client.0.smithi008.stderr: 15950K .......... .......... .......... .......... .......... 9% 18.6M 22s 2020-02-05T17:01:06.407 INFO:tasks.workunit.client.0.smithi008.stderr: 16000K .......... .......... .......... .......... .......... 9% 18.9M 22s 2020-02-05T17:01:06.410 INFO:tasks.workunit.client.0.smithi008.stderr: 16050K .......... .......... .......... .......... .......... 9% 16.5M 22s 2020-02-05T17:01:06.412 INFO:tasks.workunit.client.0.smithi008.stderr: 16100K .......... .......... .......... .......... .......... 9% 19.3M 22s 2020-02-05T17:01:06.415 INFO:tasks.workunit.client.0.smithi008.stderr: 16150K .......... .......... .......... .......... .......... 9% 18.1M 22s 2020-02-05T17:01:06.417 INFO:tasks.workunit.client.0.smithi008.stderr: 16200K .......... .......... .......... .......... .......... 9% 19.3M 22s 2020-02-05T17:01:06.420 INFO:tasks.workunit.client.0.smithi008.stderr: 16250K .......... .......... .......... .......... .......... 9% 16.8M 21s 2020-02-05T17:01:06.423 INFO:tasks.workunit.client.0.smithi008.stderr: 16300K .......... .......... .......... .......... .......... 9% 19.2M 21s 2020-02-05T17:01:06.425 INFO:tasks.workunit.client.0.smithi008.stderr: 16350K .......... .......... .......... .......... .......... 9% 19.4M 21s 2020-02-05T17:01:06.429 INFO:tasks.workunit.client.0.smithi008.stderr: 16400K .......... .......... .......... .......... .......... 9% 13.3M 21s 2020-02-05T17:01:06.432 INFO:tasks.workunit.client.0.smithi008.stderr: 16450K .......... .......... .......... .......... .......... 9% 14.8M 21s 2020-02-05T17:01:06.436 INFO:tasks.workunit.client.0.smithi008.stderr: 16500K .......... .......... .......... .......... .......... 9% 14.1M 21s 2020-02-05T17:01:06.439 INFO:tasks.workunit.client.0.smithi008.stderr: 16550K .......... .......... .......... .......... .......... 9% 15.3M 21s 2020-02-05T17:01:06.444 INFO:tasks.workunit.client.0.smithi008.stderr: 16600K .......... .......... .......... .......... .......... 9% 13.0M 21s 2020-02-05T17:01:06.446 INFO:tasks.workunit.client.0.smithi008.stderr: 16650K .......... .......... .......... .......... .......... 9% 14.6M 21s 2020-02-05T17:01:06.450 INFO:tasks.workunit.client.0.smithi008.stderr: 16700K .......... .......... .......... .......... .......... 9% 15.0M 21s 2020-02-05T17:01:06.453 INFO:tasks.workunit.client.0.smithi008.stderr: 16750K .......... .......... .......... .......... .......... 9% 14.3M 21s 2020-02-05T17:01:06.457 INFO:tasks.workunit.client.0.smithi008.stderr: 16800K .......... .......... .......... .......... .......... 9% 12.6M 21s 2020-02-05T17:01:06.462 INFO:tasks.workunit.client.0.smithi008.stderr: 16850K .......... .......... .......... .......... .......... 10% 9.18M 21s 2020-02-05T17:01:06.466 INFO:tasks.workunit.client.0.smithi008.stderr: 16900K .......... .......... .......... .......... .......... 10% 12.6M 21s 2020-02-05T17:01:06.469 INFO:tasks.workunit.client.0.smithi008.stderr: 16950K .......... .......... .......... .......... .......... 10% 13.6M 21s 2020-02-05T17:01:06.473 INFO:tasks.workunit.client.0.smithi008.stderr: 17000K .......... .......... .......... .......... .......... 10% 12.4M 21s 2020-02-05T17:01:06.478 INFO:tasks.workunit.client.0.smithi008.stderr: 17050K .......... .......... .......... .......... .......... 10% 11.5M 21s 2020-02-05T17:01:06.481 INFO:tasks.workunit.client.0.smithi008.stderr: 17100K .......... .......... .......... .......... .......... 10% 14.9M 21s 2020-02-05T17:01:06.485 INFO:tasks.workunit.client.0.smithi008.stderr: 17150K .......... .......... .......... .......... .......... 10% 13.3M 21s 2020-02-05T17:01:06.489 INFO:tasks.workunit.client.0.smithi008.stderr: 17200K .......... .......... .......... .......... .......... 10% 10.8M 21s 2020-02-05T17:01:06.493 INFO:tasks.workunit.client.0.smithi008.stderr: 17250K .......... .......... .......... .......... .......... 10% 12.5M 21s 2020-02-05T17:01:06.497 INFO:tasks.workunit.client.0.smithi008.stderr: 17300K .......... .......... .......... .......... .......... 10% 14.2M 21s 2020-02-05T17:01:06.501 INFO:tasks.workunit.client.0.smithi008.stderr: 17350K .......... .......... .......... .......... .......... 10% 11.9M 21s 2020-02-05T17:01:06.505 INFO:tasks.workunit.client.0.smithi008.stderr: 17400K .......... .......... .......... .......... .......... 10% 11.1M 21s 2020-02-05T17:01:06.511 INFO:tasks.workunit.client.0.smithi008.stderr: 17450K .......... .......... .......... .......... .......... 10% 8.73M 21s 2020-02-05T17:01:06.516 INFO:tasks.workunit.client.0.smithi008.stderr: 17500K .......... .......... .......... .......... .......... 10% 10.1M 21s 2020-02-05T17:01:06.522 INFO:tasks.workunit.client.0.smithi008.stderr: 17550K .......... .......... .......... .......... .......... 10% 8.45M 21s 2020-02-05T17:01:06.527 INFO:tasks.workunit.client.0.smithi008.stderr: 17600K .......... .......... .......... .......... .......... 10% 8.44M 21s 2020-02-05T17:01:06.534 INFO:tasks.workunit.client.0.smithi008.stderr: 17650K .......... .......... .......... .......... .......... 10% 7.76M 21s 2020-02-05T17:01:06.539 INFO:tasks.workunit.client.0.smithi008.stderr: 17700K .......... .......... .......... .......... .......... 10% 9.61M 21s 2020-02-05T17:01:06.545 INFO:tasks.workunit.client.0.smithi008.stderr: 17750K .......... .......... .......... .......... .......... 10% 7.73M 21s 2020-02-05T17:01:06.552 INFO:tasks.workunit.client.0.smithi008.stderr: 17800K .......... .......... .......... .......... .......... 10% 7.06M 21s 2020-02-05T17:01:06.577 INFO:tasks.workunit.client.0.smithi008.stderr: 17850K .......... .......... .......... .......... .......... 10% 6.77M 21s 2020-02-05T17:01:06.577 INFO:tasks.workunit.client.0.smithi008.stderr: 17900K .......... .......... .......... .......... .......... 10% 8.06M 21s 2020-02-05T17:01:06.577 INFO:tasks.workunit.client.0.smithi008.stderr: 17950K .......... .......... .......... .......... .......... 10% 8.50M 20s 2020-02-05T17:01:06.578 INFO:tasks.workunit.client.0.smithi008.stderr: 18000K .......... .......... .......... .......... .......... 10% 7.84M 20s 2020-02-05T17:01:06.584 INFO:tasks.workunit.client.0.smithi008.stderr: 18050K .......... .......... .......... .......... .......... 10% 7.42M 20s 2020-02-05T17:01:06.597 INFO:tasks.workunit.client.0.smithi008.stderr: 18100K .......... .......... .......... .......... .......... 10% 6.71M 20s 2020-02-05T17:01:06.597 INFO:tasks.workunit.client.0.smithi008.stderr: 18150K .......... .......... .......... .......... .......... 10% 10.9M 20s 2020-02-05T17:01:06.600 INFO:tasks.workunit.client.0.smithi008.stderr: 18200K .......... .......... .......... .......... .......... 10% 11.3M 20s 2020-02-05T17:01:06.606 INFO:tasks.workunit.client.0.smithi008.stderr: 18250K .......... .......... .......... .......... .......... 10% 8.54M 20s 2020-02-05T17:01:06.613 INFO:tasks.workunit.client.0.smithi008.stderr: 18300K .......... .......... .......... .......... .......... 10% 8.11M 20s 2020-02-05T17:01:06.618 INFO:tasks.workunit.client.0.smithi008.stderr: 18350K .......... .......... .......... .......... .......... 10% 7.75M 20s 2020-02-05T17:01:06.625 INFO:tasks.workunit.client.0.smithi008.stderr: 18400K .......... .......... .......... .......... .......... 10% 7.15M 20s 2020-02-05T17:01:06.633 INFO:tasks.workunit.client.0.smithi008.stderr: 18450K .......... .......... .......... .......... .......... 10% 5.98M 20s 2020-02-05T17:01:06.640 INFO:tasks.workunit.client.0.smithi008.stderr: 18500K .......... .......... .......... .......... .......... 11% 6.93M 20s 2020-02-05T17:01:06.648 INFO:tasks.workunit.client.0.smithi008.stderr: 18550K .......... .......... .......... .......... .......... 11% 6.73M 20s 2020-02-05T17:01:06.655 INFO:tasks.workunit.client.0.smithi008.stderr: 18600K .......... .......... .......... .......... .......... 11% 6.85M 20s 2020-02-05T17:01:06.663 INFO:tasks.workunit.client.0.smithi008.stderr: 18650K .......... .......... .......... .......... .......... 11% 5.67M 20s 2020-02-05T17:01:06.671 INFO:tasks.workunit.client.0.smithi008.stderr: 18700K .......... .......... .......... .......... .......... 11% 6.20M 20s 2020-02-05T17:01:06.680 INFO:tasks.workunit.client.0.smithi008.stderr: 18750K .......... .......... .......... .......... .......... 11% 6.18M 20s 2020-02-05T17:01:06.686 INFO:tasks.workunit.client.0.smithi008.stderr: 18800K .......... .......... .......... .......... .......... 11% 6.59M 20s 2020-02-05T17:01:06.694 INFO:tasks.workunit.client.0.smithi008.stderr: 18850K .......... .......... .......... .......... .......... 11% 6.19M 20s 2020-02-05T17:01:06.701 INFO:tasks.workunit.client.0.smithi008.stderr: 18900K .......... .......... .......... .......... .......... 11% 6.79M 20s 2020-02-05T17:01:06.709 INFO:tasks.workunit.client.0.smithi008.stderr: 18950K .......... .......... .......... .......... .......... 11% 6.42M 20s 2020-02-05T17:01:06.717 INFO:tasks.workunit.client.0.smithi008.stderr: 19000K .......... .......... .......... .......... .......... 11% 6.30M 20s 2020-02-05T17:01:06.724 INFO:tasks.workunit.client.0.smithi008.stderr: 19050K .......... .......... .......... .......... .......... 11% 6.21M 20s 2020-02-05T17:01:06.730 INFO:tasks.workunit.client.0.smithi008.stderr: 19100K .......... .......... .......... .......... .......... 11% 8.74M 20s 2020-02-05T17:01:06.735 INFO:tasks.workunit.client.0.smithi008.stderr: 19150K .......... .......... .......... .......... .......... 11% 8.91M 20s 2020-02-05T17:01:06.742 INFO:tasks.workunit.client.0.smithi008.stderr: 19200K .......... .......... .......... .......... .......... 11% 8.28M 20s 2020-02-05T17:01:06.750 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:01:06.750 INFO:tasks.workunit.client.0.smithi008.stderr: 19250K .......... .......... .......... .......... .......... 11% 6.09M 20s 2020-02-05T17:01:06.755 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:01:06.757 INFO:tasks.workunit.client.0.smithi008.stderr: 19300K .......... .......... .......... .......... .......... 11% 6.46M 20s 2020-02-05T17:01:06.761 INFO:tasks.workunit.client.0.smithi008.stderr: 19350K .......... .......... .......... .......... .......... 11% 11.4M 20s 2020-02-05T17:01:06.767 INFO:tasks.workunit.client.0.smithi008.stderr: 19400K .......... .......... .......... .......... .......... 11% 9.09M 20s 2020-02-05T17:01:06.774 INFO:tasks.workunit.client.0.smithi008.stderr: 19450K .......... .......... .......... .......... .......... 11% 7.36M 20s 2020-02-05T17:01:06.779 INFO:tasks.workunit.client.0.smithi008.stderr: 19500K .......... .......... .......... .......... .......... 11% 8.33M 20s 2020-02-05T17:01:06.785 INFO:tasks.workunit.client.0.smithi008.stderr: 19550K .......... .......... .......... .......... .......... 11% 8.93M 20s 2020-02-05T17:01:06.790 INFO:tasks.workunit.client.0.smithi008.stderr: 19600K .......... .......... .......... .......... .......... 11% 8.51M 20s 2020-02-05T17:01:06.796 INFO:tasks.workunit.client.0.smithi008.stderr: 19650K .......... .......... .......... .......... .......... 11% 7.99M 20s 2020-02-05T17:01:06.802 INFO:tasks.workunit.client.0.smithi008.stderr: 19700K .......... .......... .......... .......... .......... 11% 8.89M 20s 2020-02-05T17:01:06.808 INFO:tasks.workunit.client.0.smithi008.stderr: 19750K .......... .......... .......... .......... .......... 11% 8.56M 20s 2020-02-05T17:01:06.815 INFO:tasks.workunit.client.0.smithi008.stderr: 19800K .......... .......... .......... .......... .......... 11% 6.69M 20s 2020-02-05T17:01:06.825 INFO:tasks.workunit.client.0.smithi008.stderr: 19850K .......... .......... .......... .......... .......... 11% 5.42M 20s 2020-02-05T17:01:06.832 INFO:tasks.workunit.client.0.smithi008.stderr: 19900K .......... .......... .......... .......... .......... 11% 6.41M 20s 2020-02-05T17:01:06.840 INFO:tasks.workunit.client.0.smithi008.stderr: 19950K .......... .......... .......... .......... .......... 11% 5.28M 20s 2020-02-05T17:01:06.844 INFO:tasks.workunit.client.0.smithi008.stderr: 20000K .......... .......... .......... .......... .......... 11% 14.0M 20s 2020-02-05T17:01:06.849 INFO:tasks.workunit.client.0.smithi008.stderr: 20050K .......... .......... .......... .......... .......... 11% 10.5M 20s 2020-02-05T17:01:06.853 INFO:tasks.workunit.client.0.smithi008.stderr: 20100K .......... .......... .......... .......... .......... 11% 11.9M 20s 2020-02-05T17:01:06.857 INFO:tasks.workunit.client.0.smithi008.stderr: 20150K .......... .......... .......... .......... .......... 11% 11.2M 20s 2020-02-05T17:01:06.862 INFO:tasks.workunit.client.0.smithi008.stderr: 20200K .......... .......... .......... .......... .......... 12% 11.6M 20s 2020-02-05T17:01:06.867 INFO:tasks.workunit.client.0.smithi008.stderr: 20250K .......... .......... .......... .......... .......... 12% 8.28M 20s 2020-02-05T17:01:06.873 INFO:tasks.workunit.client.0.smithi008.stderr: 20300K .......... .......... .......... .......... .......... 12% 9.14M 20s 2020-02-05T17:01:06.879 INFO:tasks.workunit.client.0.smithi008.stderr: 20350K .......... .......... .......... .......... .......... 12% 8.72M 20s 2020-02-05T17:01:06.885 INFO:tasks.workunit.client.0.smithi008.stderr: 20400K .......... .......... .......... .......... .......... 12% 7.42M 20s 2020-02-05T17:01:06.893 INFO:tasks.workunit.client.0.smithi008.stderr: 20450K .......... .......... .......... .......... .......... 12% 6.49M 20s 2020-02-05T17:01:06.897 INFO:tasks.workunit.client.0.smithi008.stderr: 20500K .......... .......... .......... .......... .......... 12% 10.3M 20s 2020-02-05T17:01:06.902 INFO:tasks.workunit.client.0.smithi008.stderr: 20550K .......... .......... .......... .......... .......... 12% 11.5M 20s 2020-02-05T17:01:06.907 INFO:tasks.workunit.client.0.smithi008.stderr: 20600K .......... .......... .......... .......... .......... 12% 9.26M 20s 2020-02-05T17:01:06.914 INFO:tasks.workunit.client.0.smithi008.stderr: 20650K .......... .......... .......... .......... .......... 12% 6.94M 20s 2020-02-05T17:01:06.919 INFO:tasks.workunit.client.0.smithi008.stderr: 20700K .......... .......... .......... .......... .......... 12% 9.26M 20s 2020-02-05T17:01:06.925 INFO:tasks.workunit.client.0.smithi008.stderr: 20750K .......... .......... .......... .......... .......... 12% 8.86M 20s 2020-02-05T17:01:06.929 INFO:tasks.workunit.client.0.smithi008.stderr: 20800K .......... .......... .......... .......... .......... 12% 10.6M 20s 2020-02-05T17:01:06.934 INFO:tasks.workunit.client.0.smithi008.stderr: 20850K .......... .......... .......... .......... .......... 12% 9.66M 20s 2020-02-05T17:01:06.937 INFO:tasks.workunit.client.0.smithi008.stderr: 20900K .......... .......... .......... .......... .......... 12% 14.4M 20s 2020-02-05T17:01:06.941 INFO:tasks.workunit.client.0.smithi008.stderr: 20950K .......... .......... .......... .......... .......... 12% 14.1M 20s 2020-02-05T17:01:06.945 INFO:tasks.workunit.client.0.smithi008.stderr: 21000K .......... .......... .......... .......... .......... 12% 12.7M 20s 2020-02-05T17:01:06.950 INFO:tasks.workunit.client.0.smithi008.stderr: 21050K .......... .......... .......... .......... .......... 12% 9.94M 20s 2020-02-05T17:01:06.955 INFO:tasks.workunit.client.0.smithi008.stderr: 21100K .......... .......... .......... .......... .......... 12% 10.2M 20s 2020-02-05T17:01:06.960 INFO:tasks.workunit.client.0.smithi008.stderr: 21150K .......... .......... .......... .......... .......... 12% 9.69M 20s 2020-02-05T17:01:06.965 INFO:tasks.workunit.client.0.smithi008.stderr: 21200K .......... .......... .......... .......... .......... 12% 8.98M 20s 2020-02-05T17:01:06.972 INFO:tasks.workunit.client.0.smithi008.stderr: 21250K .......... .......... .......... .......... .......... 12% 7.64M 20s 2020-02-05T17:01:06.977 INFO:tasks.workunit.client.0.smithi008.stderr: 21300K .......... .......... .......... .......... .......... 12% 8.54M 20s 2020-02-05T17:01:06.982 INFO:tasks.workunit.client.0.smithi008.stderr: 21350K .......... .......... .......... .......... .......... 12% 9.50M 20s 2020-02-05T17:01:06.987 INFO:tasks.workunit.client.0.smithi008.stderr: 21400K .......... .......... .......... .......... .......... 12% 9.94M 20s 2020-02-05T17:01:06.994 INFO:tasks.workunit.client.0.smithi008.stderr: 21450K .......... .......... .......... .......... .......... 12% 7.52M 20s 2020-02-05T17:01:06.999 INFO:tasks.workunit.client.0.smithi008.stderr: 21500K .......... .......... .......... .......... .......... 12% 9.29M 20s 2020-02-05T17:01:07.004 INFO:tasks.workunit.client.0.smithi008.stderr: 21550K .......... .......... .......... .......... .......... 12% 10.3M 20s 2020-02-05T17:01:07.008 INFO:tasks.workunit.client.0.smithi008.stderr: 21600K .......... .......... .......... .......... .......... 12% 11.1M 20s 2020-02-05T17:01:07.013 INFO:tasks.workunit.client.0.smithi008.stderr: 21650K .......... .......... .......... .......... .......... 12% 9.41M 20s 2020-02-05T17:01:07.018 INFO:tasks.workunit.client.0.smithi008.stderr: 21700K .......... .......... .......... .......... .......... 12% 10.4M 20s 2020-02-05T17:01:07.022 INFO:tasks.workunit.client.0.smithi008.stderr: 21750K .......... .......... .......... .......... .......... 12% 11.2M 20s 2020-02-05T17:01:07.027 INFO:tasks.workunit.client.0.smithi008.stderr: 21800K .......... .......... .......... .......... .......... 12% 11.1M 20s 2020-02-05T17:01:07.032 INFO:tasks.workunit.client.0.smithi008.stderr: 21850K .......... .......... .......... .......... .......... 12% 8.80M 20s 2020-02-05T17:01:07.038 INFO:tasks.workunit.client.0.smithi008.stderr: 21900K .......... .......... .......... .......... .......... 13% 8.96M 19s 2020-02-05T17:01:07.043 INFO:tasks.workunit.client.0.smithi008.stderr: 21950K .......... .......... .......... .......... .......... 13% 9.87M 19s 2020-02-05T17:01:07.048 INFO:tasks.workunit.client.0.smithi008.stderr: 22000K .......... .......... .......... .......... .......... 13% 10.0M 19s 2020-02-05T17:01:07.055 INFO:tasks.workunit.client.0.smithi008.stderr: 22050K .......... .......... .......... .......... .......... 13% 6.87M 19s 2020-02-05T17:01:07.061 INFO:tasks.workunit.client.0.smithi008.stderr: 22100K .......... .......... .......... .......... .......... 13% 7.55M 19s 2020-02-05T17:01:07.068 INFO:tasks.workunit.client.0.smithi008.stderr: 22150K .......... .......... .......... .......... .......... 13% 7.51M 19s 2020-02-05T17:01:07.076 INFO:tasks.workunit.client.0.smithi008.stderr: 22200K .......... .......... .......... .......... .......... 13% 5.94M 19s 2020-02-05T17:01:07.083 INFO:tasks.workunit.client.0.smithi008.stderr: 22250K .......... .......... .......... .......... .......... 13% 6.90M 19s 2020-02-05T17:01:07.089 INFO:tasks.workunit.client.0.smithi008.stderr: 22300K .......... .......... .......... .......... .......... 13% 8.55M 19s 2020-02-05T17:01:07.095 INFO:tasks.workunit.client.0.smithi008.stderr: 22350K .......... .......... .......... .......... .......... 13% 8.23M 19s 2020-02-05T17:01:07.101 INFO:tasks.workunit.client.0.smithi008.stderr: 22400K .......... .......... .......... .......... .......... 13% 7.67M 19s 2020-02-05T17:01:07.109 INFO:tasks.workunit.client.0.smithi008.stderr: 22450K .......... .......... .......... .......... .......... 13% 6.51M 19s 2020-02-05T17:01:07.115 INFO:tasks.workunit.client.0.smithi008.stderr: 22500K .......... .......... .......... .......... .......... 13% 7.69M 19s 2020-02-05T17:01:07.122 INFO:tasks.workunit.client.0.smithi008.stderr: 22550K .......... .......... .......... .......... .......... 13% 8.02M 19s 2020-02-05T17:01:07.128 INFO:tasks.workunit.client.0.smithi008.stderr: 22600K .......... .......... .......... .......... .......... 13% 7.49M 19s 2020-02-05T17:01:07.135 INFO:tasks.workunit.client.0.smithi008.stderr: 22650K .......... .......... .......... .......... .......... 13% 6.41M 19s 2020-02-05T17:01:07.143 INFO:tasks.workunit.client.0.smithi008.stderr: 22700K .......... .......... .......... .......... .......... 13% 6.89M 19s 2020-02-05T17:01:07.149 INFO:tasks.workunit.client.0.smithi008.stderr: 22750K .......... .......... .......... .......... .......... 13% 7.29M 19s 2020-02-05T17:01:07.157 INFO:tasks.workunit.client.0.smithi008.stderr: 22800K .......... .......... .......... .......... .......... 13% 6.47M 19s 2020-02-05T17:01:07.165 INFO:tasks.workunit.client.0.smithi008.stderr: 22850K .......... .......... .......... .......... .......... 13% 5.89M 19s 2020-02-05T17:01:07.173 INFO:tasks.workunit.client.0.smithi008.stderr: 22900K .......... .......... .......... .......... .......... 13% 6.60M 19s 2020-02-05T17:01:07.181 INFO:tasks.workunit.client.0.smithi008.stderr: 22950K .......... .......... .......... .......... .......... 13% 6.10M 19s 2020-02-05T17:01:07.189 INFO:tasks.workunit.client.0.smithi008.stderr: 23000K .......... .......... .......... .......... .......... 13% 5.89M 19s 2020-02-05T17:01:07.198 INFO:tasks.workunit.client.0.smithi008.stderr: 23050K .......... .......... .......... .......... .......... 13% 5.39M 19s 2020-02-05T17:01:07.204 INFO:tasks.workunit.client.0.smithi008.stderr: 23100K .......... .......... .......... .......... .......... 13% 7.29M 19s 2020-02-05T17:01:07.211 INFO:tasks.workunit.client.0.smithi008.stderr: 23150K .......... .......... .......... .......... .......... 13% 7.54M 19s 2020-02-05T17:01:07.218 INFO:tasks.workunit.client.0.smithi008.stderr: 23200K .......... .......... .......... .......... .......... 13% 7.31M 19s 2020-02-05T17:01:07.226 INFO:tasks.workunit.client.0.smithi008.stderr: 23250K .......... .......... .......... .......... .......... 13% 6.23M 19s 2020-02-05T17:01:07.233 INFO:tasks.workunit.client.0.smithi008.stderr: 23300K .......... .......... .......... .......... .......... 13% 6.36M 19s 2020-02-05T17:01:07.241 INFO:tasks.workunit.client.0.smithi008.stderr: 23350K .......... .......... .......... .......... .......... 13% 6.96M 19s 2020-02-05T17:01:07.248 INFO:tasks.workunit.client.0.smithi008.stderr: 23400K .......... .......... .......... .......... .......... 13% 6.68M 19s 2020-02-05T17:01:07.256 INFO:tasks.workunit.client.0.smithi008.stderr: 23450K .......... .......... .......... .......... .......... 13% 5.67M 19s 2020-02-05T17:01:07.264 INFO:tasks.workunit.client.0.smithi008.stderr: 23500K .......... .......... .......... .......... .......... 13% 6.58M 19s 2020-02-05T17:01:07.271 INFO:tasks.workunit.client.0.smithi008.stderr: 23550K .......... .......... .......... .......... .......... 14% 6.21M 19s 2020-02-05T17:01:07.278 INFO:tasks.workunit.client.0.smithi008.stderr: 23600K .......... .......... .......... .......... .......... 14% 7.37M 19s 2020-02-05T17:01:07.286 INFO:tasks.workunit.client.0.smithi008.stderr: 23650K .......... .......... .......... .......... .......... 14% 6.16M 19s 2020-02-05T17:01:07.293 INFO:tasks.workunit.client.0.smithi008.stderr: 23700K .......... .......... .......... .......... .......... 14% 6.91M 19s 2020-02-05T17:01:07.299 INFO:tasks.workunit.client.0.smithi008.stderr: 23750K .......... .......... .......... .......... .......... 14% 8.22M 19s 2020-02-05T17:01:07.305 INFO:tasks.workunit.client.0.smithi008.stderr: 23800K .......... .......... .......... .......... .......... 14% 7.77M 19s 2020-02-05T17:01:07.312 INFO:tasks.workunit.client.0.smithi008.stderr: 23850K .......... .......... .......... .......... .......... 14% 7.36M 19s 2020-02-05T17:01:07.319 INFO:tasks.workunit.client.0.smithi008.stderr: 23900K .......... .......... .......... .......... .......... 14% 7.40M 19s 2020-02-05T17:01:07.326 INFO:tasks.workunit.client.0.smithi008.stderr: 23950K .......... .......... .......... .......... .......... 14% 6.51M 19s 2020-02-05T17:01:07.335 INFO:tasks.workunit.client.0.smithi008.stderr: 24000K .......... .......... .......... .......... .......... 14% 5.98M 19s 2020-02-05T17:01:07.348 INFO:tasks.workunit.client.0.smithi008.stderr: 24050K .......... .......... .......... .......... .......... 14% 3.39M 19s 2020-02-05T17:01:07.354 INFO:tasks.workunit.client.0.smithi008.stderr: 24100K .......... .......... .......... .......... .......... 14% 8.75M 19s 2020-02-05T17:01:07.360 INFO:tasks.workunit.client.0.smithi008.stderr: 24150K .......... .......... .......... .......... .......... 14% 8.28M 19s 2020-02-05T17:01:07.365 INFO:tasks.workunit.client.0.smithi008.stderr: 24200K .......... .......... .......... .......... .......... 14% 9.90M 19s 2020-02-05T17:01:07.371 INFO:tasks.workunit.client.0.smithi008.stderr: 24250K .......... .......... .......... .......... .......... 14% 7.84M 19s 2020-02-05T17:01:07.375 INFO:tasks.workunit.client.0.smithi008.stderr: 24300K .......... .......... .......... .......... .......... 14% 11.1M 19s 2020-02-05T17:01:07.380 INFO:tasks.workunit.client.0.smithi008.stderr: 24350K .......... .......... .......... .......... .......... 14% 11.2M 19s 2020-02-05T17:01:07.385 INFO:tasks.workunit.client.0.smithi008.stderr: 24400K .......... .......... .......... .......... .......... 14% 8.67M 19s 2020-02-05T17:01:07.392 INFO:tasks.workunit.client.0.smithi008.stderr: 24450K .......... .......... .......... .......... .......... 14% 7.06M 19s 2020-02-05T17:01:07.397 INFO:tasks.workunit.client.0.smithi008.stderr: 24500K .......... .......... .......... .......... .......... 14% 10.0M 19s 2020-02-05T17:01:07.403 INFO:tasks.workunit.client.0.smithi008.stderr: 24550K .......... .......... .......... .......... .......... 14% 8.96M 19s 2020-02-05T17:01:07.408 INFO:tasks.workunit.client.0.smithi008.stderr: 24600K .......... .......... .......... .......... .......... 14% 9.74M 19s 2020-02-05T17:01:07.415 INFO:tasks.workunit.client.0.smithi008.stderr: 24650K .......... .......... .......... .......... .......... 14% 6.68M 19s 2020-02-05T17:01:07.420 INFO:tasks.workunit.client.0.smithi008.stderr: 24700K .......... .......... .......... .......... .......... 14% 10.0M 19s 2020-02-05T17:01:07.424 INFO:tasks.workunit.client.0.smithi008.stderr: 24750K .......... .......... .......... .......... .......... 14% 11.3M 19s 2020-02-05T17:01:07.428 INFO:tasks.workunit.client.0.smithi008.stderr: 24800K .......... .......... .......... .......... .......... 14% 13.3M 19s 2020-02-05T17:01:07.432 INFO:tasks.workunit.client.0.smithi008.stderr: 24850K .......... .......... .......... .......... .......... 14% 11.0M 19s 2020-02-05T17:01:07.437 INFO:tasks.workunit.client.0.smithi008.stderr: 24900K .......... .......... .......... .......... .......... 14% 10.2M 19s 2020-02-05T17:01:07.442 INFO:tasks.workunit.client.0.smithi008.stderr: 24950K .......... .......... .......... .......... .......... 14% 9.53M 19s 2020-02-05T17:01:07.447 INFO:tasks.workunit.client.0.smithi008.stderr: 25000K .......... .......... .......... .......... .......... 14% 10.6M 19s 2020-02-05T17:01:07.453 INFO:tasks.workunit.client.0.smithi008.stderr: 25050K .......... .......... .......... .......... .......... 14% 8.22M 19s 2020-02-05T17:01:07.458 INFO:tasks.workunit.client.0.smithi008.stderr: 25100K .......... .......... .......... .......... .......... 14% 9.96M 19s 2020-02-05T17:01:07.465 INFO:tasks.workunit.client.0.smithi008.stderr: 25150K .......... .......... .......... .......... .......... 14% 7.13M 19s 2020-02-05T17:01:07.471 INFO:tasks.workunit.client.0.smithi008.stderr: 25200K .......... .......... .......... .......... .......... 14% 7.77M 19s 2020-02-05T17:01:07.478 INFO:tasks.workunit.client.0.smithi008.stderr: 25250K .......... .......... .......... .......... .......... 15% 7.56M 19s 2020-02-05T17:01:07.483 INFO:tasks.workunit.client.0.smithi008.stderr: 25300K .......... .......... .......... .......... .......... 15% 9.03M 19s 2020-02-05T17:01:07.487 INFO:tasks.workunit.client.0.smithi008.stderr: 25350K .......... .......... .......... .......... .......... 15% 10.8M 19s 2020-02-05T17:01:07.496 INFO:tasks.workunit.client.0.smithi008.stderr: 25400K .......... .......... .......... .......... .......... 15% 7.69M 19s 2020-02-05T17:01:07.499 INFO:tasks.workunit.client.0.smithi008.stderr: 25450K .......... .......... .......... .......... .......... 15% 8.65M 19s 2020-02-05T17:01:07.505 INFO:tasks.workunit.client.0.smithi008.stderr: 25500K .......... .......... .......... .......... .......... 15% 8.95M 19s 2020-02-05T17:01:07.511 INFO:tasks.workunit.client.0.smithi008.stderr: 25550K .......... .......... .......... .......... .......... 15% 8.03M 19s 2020-02-05T17:01:07.516 INFO:tasks.workunit.client.0.smithi008.stderr: 25600K .......... .......... .......... .......... .......... 15% 9.21M 19s 2020-02-05T17:01:07.523 INFO:tasks.workunit.client.0.smithi008.stderr: 25650K .......... .......... .......... .......... .......... 15% 7.37M 19s 2020-02-05T17:01:07.529 INFO:tasks.workunit.client.0.smithi008.stderr: 25700K .......... .......... .......... .......... .......... 15% 8.41M 19s 2020-02-05T17:01:07.534 INFO:tasks.workunit.client.0.smithi008.stderr: 25750K .......... .......... .......... .......... .......... 15% 9.10M 19s 2020-02-05T17:01:07.539 INFO:tasks.workunit.client.0.smithi008.stderr: 25800K .......... .......... .......... .......... .......... 15% 8.78M 19s 2020-02-05T17:01:07.546 INFO:tasks.workunit.client.0.smithi008.stderr: 25850K .......... .......... .......... .......... .......... 15% 7.29M 19s 2020-02-05T17:01:07.553 INFO:tasks.workunit.client.0.smithi008.stderr: 25900K .......... .......... .......... .......... .......... 15% 8.02M 19s 2020-02-05T17:01:07.560 INFO:tasks.workunit.client.0.smithi008.stderr: 25950K .......... .......... .......... .......... .......... 15% 6.87M 19s 2020-02-05T17:01:07.567 INFO:tasks.workunit.client.0.smithi008.stderr: 26000K .......... .......... .......... .......... .......... 15% 6.73M 19s 2020-02-05T17:01:07.575 INFO:tasks.workunit.client.0.smithi008.stderr: 26050K .......... .......... .......... .......... .......... 15% 5.91M 19s 2020-02-05T17:01:07.582 INFO:tasks.workunit.client.0.smithi008.stderr: 26100K .......... .......... .......... .......... .......... 15% 7.07M 19s 2020-02-05T17:01:07.588 INFO:tasks.workunit.client.0.smithi008.stderr: 26150K .......... .......... .......... .......... .......... 15% 7.83M 19s 2020-02-05T17:01:07.594 INFO:tasks.workunit.client.0.smithi008.stderr: 26200K .......... .......... .......... .......... .......... 15% 8.45M 19s 2020-02-05T17:01:07.604 INFO:tasks.workunit.client.0.smithi008.stderr: 26250K .......... .......... .......... .......... .......... 15% 6.91M 19s 2020-02-05T17:01:07.610 INFO:tasks.workunit.client.0.smithi008.stderr: 26300K .......... .......... .......... .......... .......... 15% 5.25M 19s 2020-02-05T17:01:07.616 INFO:tasks.workunit.client.0.smithi008.stderr: 26350K .......... .......... .......... .......... .......... 15% 8.58M 19s 2020-02-05T17:01:07.621 INFO:tasks.workunit.client.0.smithi008.stderr: 26400K .......... .......... .......... .......... .......... 15% 9.23M 19s 2020-02-05T17:01:07.628 INFO:tasks.workunit.client.0.smithi008.stderr: 26450K .......... .......... .......... .......... .......... 15% 7.41M 19s 2020-02-05T17:01:07.633 INFO:tasks.workunit.client.0.smithi008.stderr: 26500K .......... .......... .......... .......... .......... 15% 8.72M 19s 2020-02-05T17:01:07.640 INFO:tasks.workunit.client.0.smithi008.stderr: 26550K .......... .......... .......... .......... .......... 15% 7.11M 19s 2020-02-05T17:01:07.646 INFO:tasks.workunit.client.0.smithi008.stderr: 26600K .......... .......... .......... .......... .......... 15% 8.91M 19s 2020-02-05T17:01:07.652 INFO:tasks.workunit.client.0.smithi008.stderr: 26650K .......... .......... .......... .......... .......... 15% 8.05M 19s 2020-02-05T17:01:07.656 INFO:tasks.workunit.client.0.smithi008.stderr: 26700K .......... .......... .......... .......... .......... 15% 10.6M 19s 2020-02-05T17:01:07.661 INFO:tasks.workunit.client.0.smithi008.stderr: 26750K .......... .......... .......... .......... .......... 15% 10.7M 19s 2020-02-05T17:01:07.665 INFO:tasks.workunit.client.0.smithi008.stderr: 26800K .......... .......... .......... .......... .......... 15% 11.5M 19s 2020-02-05T17:01:07.671 INFO:tasks.workunit.client.0.smithi008.stderr: 26850K .......... .......... .......... .......... .......... 15% 7.86M 19s 2020-02-05T17:01:07.676 INFO:tasks.workunit.client.0.smithi008.stderr: 26900K .......... .......... .......... .......... .......... 15% 10.7M 19s 2020-02-05T17:01:07.680 INFO:tasks.workunit.client.0.smithi008.stderr: 26950K .......... .......... .......... .......... .......... 16% 10.8M 19s 2020-02-05T17:01:07.685 INFO:tasks.workunit.client.0.smithi008.stderr: 27000K .......... .......... .......... .......... .......... 16% 9.92M 19s 2020-02-05T17:01:07.691 INFO:tasks.workunit.client.0.smithi008.stderr: 27050K .......... .......... .......... .......... .......... 16% 8.69M 19s 2020-02-05T17:01:07.695 INFO:tasks.workunit.client.0.smithi008.stderr: 27100K .......... .......... .......... .......... .......... 16% 10.6M 19s 2020-02-05T17:01:07.701 INFO:tasks.workunit.client.0.smithi008.stderr: 27150K .......... .......... .......... .......... .......... 16% 11.2M 19s 2020-02-05T17:01:07.704 INFO:tasks.workunit.client.0.smithi008.stderr: 27200K .......... .......... .......... .......... .......... 16% 11.0M 19s 2020-02-05T17:01:07.709 INFO:tasks.workunit.client.0.smithi008.stderr: 27250K .......... .......... .......... .......... .......... 16% 9.70M 19s 2020-02-05T17:01:07.716 INFO:tasks.workunit.client.0.smithi008.stderr: 27300K .......... .......... .......... .......... .......... 16% 8.37M 19s 2020-02-05T17:01:07.722 INFO:tasks.workunit.client.0.smithi008.stderr: 27350K .......... .......... .......... .......... .......... 16% 7.29M 19s 2020-02-05T17:01:07.730 INFO:tasks.workunit.client.0.smithi008.stderr: 27400K .......... .......... .......... .......... .......... 16% 6.68M 19s 2020-02-05T17:01:07.738 INFO:tasks.workunit.client.0.smithi008.stderr: 27450K .......... .......... .......... .......... .......... 16% 5.97M 19s 2020-02-05T17:01:07.745 INFO:tasks.workunit.client.0.smithi008.stderr: 27500K .......... .......... .......... .......... .......... 16% 6.58M 19s 2020-02-05T17:01:07.752 INFO:tasks.workunit.client.0.smithi008.stderr: 27550K .......... .......... .......... .......... .......... 16% 6.39M 19s 2020-02-05T17:01:07.759 INFO:tasks.workunit.client.0.smithi008.stderr: 27600K .......... .......... .......... .......... .......... 16% 7.11M 19s 2020-02-05T17:01:07.767 INFO:tasks.workunit.client.0.smithi008.stderr: 27650K .......... .......... .......... .......... .......... 16% 7.14M 19s 2020-02-05T17:01:07.773 INFO:tasks.workunit.client.0.smithi008.stderr: 27700K .......... .......... .......... .......... .......... 16% 7.63M 19s 2020-02-05T17:01:07.779 INFO:tasks.workunit.client.0.smithi008.stderr: 27750K .......... .......... .......... .......... .......... 16% 7.87M 19s 2020-02-05T17:01:07.786 INFO:tasks.workunit.client.0.smithi008.stderr: 27800K .......... .......... .......... .......... .......... 16% 7.27M 19s 2020-02-05T17:01:07.793 INFO:tasks.workunit.client.0.smithi008.stderr: 27850K .......... .......... .......... .......... .......... 16% 6.62M 19s 2020-02-05T17:01:07.800 INFO:tasks.workunit.client.0.smithi008.stderr: 27900K .......... .......... .......... .......... .......... 16% 6.96M 19s 2020-02-05T17:01:07.807 INFO:tasks.workunit.client.0.smithi008.stderr: 27950K .......... .......... .......... .......... .......... 16% 7.09M 19s 2020-02-05T17:01:07.815 INFO:tasks.workunit.client.0.smithi008.stderr: 28000K .......... .......... .......... .......... .......... 16% 6.72M 19s 2020-02-05T17:01:07.823 INFO:tasks.workunit.client.0.smithi008.stderr: 28050K .......... .......... .......... .......... .......... 16% 5.80M 19s 2020-02-05T17:01:07.830 INFO:tasks.workunit.client.0.smithi008.stderr: 28100K .......... .......... .......... .......... .......... 16% 6.67M 19s 2020-02-05T17:01:07.841 INFO:tasks.workunit.client.0.smithi008.stderr: 28150K .......... .......... .......... .......... .......... 16% 4.49M 19s 2020-02-05T17:01:07.846 INFO:tasks.workunit.client.0.smithi008.stderr: 28200K .......... .......... .......... .......... .......... 16% 9.55M 19s 2020-02-05T17:01:07.851 INFO:tasks.workunit.client.0.smithi008.stderr: 28250K .......... .......... .......... .......... .......... 16% 9.20M 18s 2020-02-05T17:01:07.855 INFO:tasks.workunit.client.0.smithi008.stderr: 28300K .......... .......... .......... .......... .......... 16% 11.5M 18s 2020-02-05T17:01:07.860 INFO:tasks.workunit.client.0.smithi008.stderr: 28350K .......... .......... .......... .......... .......... 16% 10.8M 18s 2020-02-05T17:01:07.865 INFO:tasks.workunit.client.0.smithi008.stderr: 28400K .......... .......... .......... .......... .......... 16% 10.0M 18s 2020-02-05T17:01:07.871 INFO:tasks.workunit.client.0.smithi008.stderr: 28450K .......... .......... .......... .......... .......... 16% 7.87M 18s 2020-02-05T17:01:07.876 INFO:tasks.workunit.client.0.smithi008.stderr: 28500K .......... .......... .......... .......... .......... 16% 9.66M 18s 2020-02-05T17:01:07.881 INFO:tasks.workunit.client.0.smithi008.stderr: 28550K .......... .......... .......... .......... .......... 16% 8.91M 18s 2020-02-05T17:01:07.887 INFO:tasks.workunit.client.0.smithi008.stderr: 28600K .......... .......... .......... .......... .......... 16% 8.81M 18s 2020-02-05T17:01:07.894 INFO:tasks.workunit.client.0.smithi008.stderr: 28650K .......... .......... .......... .......... .......... 17% 7.40M 18s 2020-02-05T17:01:07.898 INFO:tasks.workunit.client.0.smithi008.stderr: 28700K .......... .......... .......... .......... .......... 17% 10.7M 18s 2020-02-05T17:01:07.903 INFO:tasks.workunit.client.0.smithi008.stderr: 28750K .......... .......... .......... .......... .......... 17% 10.9M 18s 2020-02-05T17:01:07.909 INFO:tasks.workunit.client.0.smithi008.stderr: 28800K .......... .......... .......... .......... .......... 17% 10.5M 18s 2020-02-05T17:01:07.912 INFO:tasks.workunit.client.0.smithi008.stderr: 28850K .......... .......... .......... .......... .......... 17% 9.74M 18s 2020-02-05T17:01:07.917 INFO:tasks.workunit.client.0.smithi008.stderr: 28900K .......... .......... .......... .......... .......... 17% 10.9M 18s 2020-02-05T17:01:07.920 INFO:tasks.workunit.client.0.smithi008.stderr: 28950K .......... .......... .......... .......... .......... 17% 14.4M 18s 2020-02-05T17:01:07.924 INFO:tasks.workunit.client.0.smithi008.stderr: 29000K .......... .......... .......... .......... .......... 17% 13.7M 18s 2020-02-05T17:01:07.928 INFO:tasks.workunit.client.0.smithi008.stderr: 29050K .......... .......... .......... .......... .......... 17% 11.1M 18s 2020-02-05T17:01:07.931 INFO:tasks.workunit.client.0.smithi008.stderr: 29100K .......... .......... .......... .......... .......... 17% 15.2M 18s 2020-02-05T17:01:07.934 INFO:tasks.workunit.client.0.smithi008.stderr: 29150K .......... .......... .......... .......... .......... 17% 15.1M 18s 2020-02-05T17:01:07.938 INFO:tasks.workunit.client.0.smithi008.stderr: 29200K .......... .......... .......... .......... .......... 17% 14.4M 18s 2020-02-05T17:01:07.942 INFO:tasks.workunit.client.0.smithi008.stderr: 29250K .......... .......... .......... .......... .......... 17% 10.3M 18s 2020-02-05T17:01:07.947 INFO:tasks.workunit.client.0.smithi008.stderr: 29300K .......... .......... .......... .......... .......... 17% 11.9M 18s 2020-02-05T17:01:07.951 INFO:tasks.workunit.client.0.smithi008.stderr: 29350K .......... .......... .......... .......... .......... 17% 11.0M 18s 2020-02-05T17:01:07.956 INFO:tasks.workunit.client.0.smithi008.stderr: 29400K .......... .......... .......... .......... .......... 17% 10.2M 18s 2020-02-05T17:01:07.963 INFO:tasks.workunit.client.0.smithi008.stderr: 29450K .......... .......... .......... .......... .......... 17% 7.96M 18s 2020-02-05T17:01:07.968 INFO:tasks.workunit.client.0.smithi008.stderr: 29500K .......... .......... .......... .......... .......... 17% 7.92M 18s 2020-02-05T17:01:07.973 INFO:tasks.workunit.client.0.smithi008.stderr: 29550K .......... .......... .......... .......... .......... 17% 9.72M 18s 2020-02-05T17:01:07.979 INFO:tasks.workunit.client.0.smithi008.stderr: 29600K .......... .......... .......... .......... .......... 17% 9.15M 18s 2020-02-05T17:01:07.986 INFO:tasks.workunit.client.0.smithi008.stderr: 29650K .......... .......... .......... .......... .......... 17% 7.21M 18s 2020-02-05T17:01:07.990 INFO:tasks.workunit.client.0.smithi008.stderr: 29700K .......... .......... .......... .......... .......... 17% 9.57M 18s 2020-02-05T17:01:07.995 INFO:tasks.workunit.client.0.smithi008.stderr: 29750K .......... .......... .......... .......... .......... 17% 10.6M 18s 2020-02-05T17:01:08.000 INFO:tasks.workunit.client.0.smithi008.stderr: 29800K .......... .......... .......... .......... .......... 17% 10.8M 18s 2020-02-05T17:01:08.005 INFO:tasks.workunit.client.0.smithi008.stderr: 29850K .......... .......... .......... .......... .......... 17% 8.75M 18s 2020-02-05T17:01:08.010 INFO:tasks.workunit.client.0.smithi008.stderr: 29900K .......... .......... .......... .......... .......... 17% 9.22M 18s 2020-02-05T17:01:08.016 INFO:tasks.workunit.client.0.smithi008.stderr: 29950K .......... .......... .......... .......... .......... 17% 9.11M 18s 2020-02-05T17:01:08.022 INFO:tasks.workunit.client.0.smithi008.stderr: 30000K .......... .......... .......... .......... .......... 17% 8.57M 18s 2020-02-05T17:01:08.028 INFO:tasks.workunit.client.0.smithi008.stderr: 30050K .......... .......... .......... .......... .......... 17% 7.52M 18s 2020-02-05T17:01:08.034 INFO:tasks.workunit.client.0.smithi008.stderr: 30100K .......... .......... .......... .......... .......... 17% 8.17M 18s 2020-02-05T17:01:08.040 INFO:tasks.workunit.client.0.smithi008.stderr: 30150K .......... .......... .......... .......... .......... 17% 8.09M 18s 2020-02-05T17:01:08.046 INFO:tasks.workunit.client.0.smithi008.stderr: 30200K .......... .......... .......... .......... .......... 17% 8.08M 18s 2020-02-05T17:01:08.053 INFO:tasks.workunit.client.0.smithi008.stderr: 30250K .......... .......... .......... .......... .......... 17% 6.93M 18s 2020-02-05T17:01:08.058 INFO:tasks.workunit.client.0.smithi008.stderr: 30300K .......... .......... .......... .......... .......... 18% 10.7M 18s 2020-02-05T17:01:08.064 INFO:tasks.workunit.client.0.smithi008.stderr: 30350K .......... .......... .......... .......... .......... 18% 10.7M 18s 2020-02-05T17:01:08.069 INFO:tasks.workunit.client.0.smithi008.stderr: 30400K .......... .......... .......... .......... .......... 18% 7.56M 18s 2020-02-05T17:01:08.074 INFO:tasks.workunit.client.0.smithi008.stderr: 30450K .......... .......... .......... .......... .......... 18% 9.49M 18s 2020-02-05T17:01:08.079 INFO:tasks.workunit.client.0.smithi008.stderr: 30500K .......... .......... .......... .......... .......... 18% 9.58M 18s 2020-02-05T17:01:08.085 INFO:tasks.workunit.client.0.smithi008.stderr: 30550K .......... .......... .......... .......... .......... 18% 8.61M 18s 2020-02-05T17:01:08.091 INFO:tasks.workunit.client.0.smithi008.stderr: 30600K .......... .......... .......... .......... .......... 18% 8.18M 18s 2020-02-05T17:01:08.098 INFO:tasks.workunit.client.0.smithi008.stderr: 30650K .......... .......... .......... .......... .......... 18% 7.19M 18s 2020-02-05T17:01:08.103 INFO:tasks.workunit.client.0.smithi008.stderr: 30700K .......... .......... .......... .......... .......... 18% 8.36M 18s 2020-02-05T17:01:08.108 INFO:tasks.workunit.client.0.smithi008.stderr: 30750K .......... .......... .......... .......... .......... 18% 9.99M 18s 2020-02-05T17:01:08.113 INFO:tasks.workunit.client.0.smithi008.stderr: 30800K .......... .......... .......... .......... .......... 18% 10.2M 18s 2020-02-05T17:01:08.119 INFO:tasks.workunit.client.0.smithi008.stderr: 30850K .......... .......... .......... .......... .......... 18% 8.40M 18s 2020-02-05T17:01:08.126 INFO:tasks.workunit.client.0.smithi008.stderr: 30900K .......... .......... .......... .......... .......... 18% 7.45M 18s 2020-02-05T17:01:08.132 INFO:tasks.workunit.client.0.smithi008.stderr: 30950K .......... .......... .......... .......... .......... 18% 7.41M 18s 2020-02-05T17:01:08.139 INFO:tasks.workunit.client.0.smithi008.stderr: 31000K .......... .......... .......... .......... .......... 18% 7.16M 18s 2020-02-05T17:01:08.147 INFO:tasks.workunit.client.0.smithi008.stderr: 31050K .......... .......... .......... .......... .......... 18% 5.91M 18s 2020-02-05T17:01:08.155 INFO:tasks.workunit.client.0.smithi008.stderr: 31100K .......... .......... .......... .......... .......... 18% 6.81M 18s 2020-02-05T17:01:08.162 INFO:tasks.workunit.client.0.smithi008.stderr: 31150K .......... .......... .......... .......... .......... 18% 6.70M 18s 2020-02-05T17:01:08.168 INFO:tasks.workunit.client.0.smithi008.stderr: 31200K .......... .......... .......... .......... .......... 18% 8.06M 18s 2020-02-05T17:01:08.175 INFO:tasks.workunit.client.0.smithi008.stderr: 31250K .......... .......... .......... .......... .......... 18% 6.62M 18s 2020-02-05T17:01:08.183 INFO:tasks.workunit.client.0.smithi008.stderr: 31300K .......... .......... .......... .......... .......... 18% 6.15M 18s 2020-02-05T17:01:08.191 INFO:tasks.workunit.client.0.smithi008.stderr: 31350K .......... .......... .......... .......... .......... 18% 6.18M 18s 2020-02-05T17:01:08.198 INFO:tasks.workunit.client.0.smithi008.stderr: 31400K .......... .......... .......... .......... .......... 18% 7.42M 18s 2020-02-05T17:01:08.207 INFO:tasks.workunit.client.0.smithi008.stderr: 31450K .......... .......... .......... .......... .......... 18% 5.25M 18s 2020-02-05T17:01:08.215 INFO:tasks.workunit.client.0.smithi008.stderr: 31500K .......... .......... .......... .......... .......... 18% 6.18M 18s 2020-02-05T17:01:08.223 INFO:tasks.workunit.client.0.smithi008.stderr: 31550K .......... .......... .......... .......... .......... 18% 6.31M 18s 2020-02-05T17:01:08.231 INFO:tasks.workunit.client.0.smithi008.stderr: 31600K .......... .......... .......... .......... .......... 18% 6.09M 18s 2020-02-05T17:01:08.240 INFO:tasks.workunit.client.0.smithi008.stderr: 31650K .......... .......... .......... .......... .......... 18% 5.27M 18s 2020-02-05T17:01:08.248 INFO:tasks.workunit.client.0.smithi008.stderr: 31700K .......... .......... .......... .......... .......... 18% 5.98M 18s 2020-02-05T17:01:08.256 INFO:tasks.workunit.client.0.smithi008.stderr: 31750K .......... .......... .......... .......... .......... 18% 5.93M 18s 2020-02-05T17:01:08.265 INFO:tasks.workunit.client.0.smithi008.stderr: 31800K .......... .......... .......... .......... .......... 18% 5.74M 18s 2020-02-05T17:01:08.275 INFO:tasks.workunit.client.0.smithi008.stderr: 31850K .......... .......... .......... .......... .......... 18% 4.71M 18s 2020-02-05T17:01:08.284 INFO:tasks.workunit.client.0.smithi008.stderr: 31900K .......... .......... .......... .......... .......... 18% 5.82M 18s 2020-02-05T17:01:08.292 INFO:tasks.workunit.client.0.smithi008.stderr: 31950K .......... .......... .......... .......... .......... 18% 5.67M 18s 2020-02-05T17:01:08.300 INFO:tasks.workunit.client.0.smithi008.stderr: 32000K .......... .......... .......... .......... .......... 19% 6.82M 18s 2020-02-05T17:01:08.307 INFO:tasks.workunit.client.0.smithi008.stderr: 32050K .......... .......... .......... .......... .......... 19% 6.10M 18s 2020-02-05T17:01:08.312 INFO:tasks.workunit.client.0.smithi008.stderr: 32100K .......... .......... .......... .......... .......... 19% 9.10M 18s 2020-02-05T17:01:08.318 INFO:tasks.workunit.client.0.smithi008.stderr: 32150K .......... .......... .......... .......... .......... 19% 8.79M 18s 2020-02-05T17:01:08.324 INFO:tasks.workunit.client.0.smithi008.stderr: 32200K .......... .......... .......... .......... .......... 19% 8.80M 18s 2020-02-05T17:01:08.334 INFO:tasks.workunit.client.0.smithi008.stderr: 32250K .......... .......... .......... .......... .......... 19% 4.72M 18s 2020-02-05T17:01:08.338 INFO:tasks.workunit.client.0.smithi008.stderr: 32300K .......... .......... .......... .......... .......... 19% 11.4M 18s 2020-02-05T17:01:08.343 INFO:tasks.workunit.client.0.smithi008.stderr: 32350K .......... .......... .......... .......... .......... 19% 10.2M 18s 2020-02-05T17:01:08.347 INFO:tasks.workunit.client.0.smithi008.stderr: 32400K .......... .......... .......... .......... .......... 19% 10.8M 18s 2020-02-05T17:01:08.353 INFO:tasks.workunit.client.0.smithi008.stderr: 32450K .......... .......... .......... .......... .......... 19% 8.77M 18s 2020-02-05T17:01:08.358 INFO:tasks.workunit.client.0.smithi008.stderr: 32500K .......... .......... .......... .......... .......... 19% 10.5M 18s 2020-02-05T17:01:08.364 INFO:tasks.workunit.client.0.smithi008.stderr: 32550K .......... .......... .......... .......... .......... 19% 8.98M 18s 2020-02-05T17:01:08.369 INFO:tasks.workunit.client.0.smithi008.stderr: 32600K .......... .......... .......... .......... .......... 19% 8.18M 18s 2020-02-05T17:01:08.375 INFO:tasks.workunit.client.0.smithi008.stderr: 32650K .......... .......... .......... .......... .......... 19% 8.37M 18s 2020-02-05T17:01:08.379 INFO:tasks.workunit.client.0.smithi008.stderr: 32700K .......... .......... .......... .......... .......... 19% 11.3M 18s 2020-02-05T17:01:08.384 INFO:tasks.workunit.client.0.smithi008.stderr: 32750K .......... .......... .......... .......... .......... 19% 10.1M 18s 2020-02-05T17:01:08.390 INFO:tasks.workunit.client.0.smithi008.stderr: 32800K .......... .......... .......... .......... .......... 19% 9.09M 18s 2020-02-05T17:01:08.394 INFO:tasks.workunit.client.0.smithi008.stderr: 32850K .......... .......... .......... .......... .......... 19% 9.68M 18s 2020-02-05T17:01:08.396 INFO:tasks.workunit.client.0.smithi008.stderr: 32900K .......... .......... .......... .......... .......... 19% 19.1M 18s 2020-02-05T17:01:08.399 INFO:tasks.workunit.client.0.smithi008.stderr: 32950K .......... .......... .......... .......... .......... 19% 19.1M 18s 2020-02-05T17:01:08.401 INFO:tasks.workunit.client.0.smithi008.stderr: 33000K .......... .......... .......... .......... .......... 19% 19.9M 18s 2020-02-05T17:01:08.405 INFO:tasks.workunit.client.0.smithi008.stderr: 33050K .......... .......... .......... .......... .......... 19% 13.5M 18s 2020-02-05T17:01:08.409 INFO:tasks.workunit.client.0.smithi008.stderr: 33100K .......... .......... .......... .......... .......... 19% 13.5M 18s 2020-02-05T17:01:08.413 INFO:tasks.workunit.client.0.smithi008.stderr: 33150K .......... .......... .......... .......... .......... 19% 13.8M 18s 2020-02-05T17:01:08.418 INFO:tasks.workunit.client.0.smithi008.stderr: 33200K .......... .......... .......... .......... .......... 19% 16.7M 17s 2020-02-05T17:01:08.419 INFO:tasks.workunit.client.0.smithi008.stderr: 33250K .......... .......... .......... .......... .......... 19% 13.4M 17s 2020-02-05T17:01:08.422 INFO:tasks.workunit.client.0.smithi008.stderr: 33300K .......... .......... .......... .......... .......... 19% 15.5M 17s 2020-02-05T17:01:08.425 INFO:tasks.workunit.client.0.smithi008.stderr: 33350K .......... .......... .......... .......... .......... 19% 14.8M 17s 2020-02-05T17:01:08.429 INFO:tasks.workunit.client.0.smithi008.stderr: 33400K .......... .......... .......... .......... .......... 19% 15.4M 17s 2020-02-05T17:01:08.433 INFO:tasks.workunit.client.0.smithi008.stderr: 33450K .......... .......... .......... .......... .......... 19% 12.1M 17s 2020-02-05T17:01:08.436 INFO:tasks.workunit.client.0.smithi008.stderr: 33500K .......... .......... .......... .......... .......... 19% 13.9M 17s 2020-02-05T17:01:08.441 INFO:tasks.workunit.client.0.smithi008.stderr: 33550K .......... .......... .......... .......... .......... 19% 12.3M 17s 2020-02-05T17:01:08.444 INFO:tasks.workunit.client.0.smithi008.stderr: 33600K .......... .......... .......... .......... .......... 19% 11.6M 17s 2020-02-05T17:01:08.449 INFO:tasks.workunit.client.0.smithi008.stderr: 33650K .......... .......... .......... .......... .......... 19% 11.0M 17s 2020-02-05T17:01:08.452 INFO:tasks.workunit.client.0.smithi008.stderr: 33700K .......... .......... .......... .......... .......... 20% 13.7M 17s 2020-02-05T17:01:08.456 INFO:tasks.workunit.client.0.smithi008.stderr: 33750K .......... .......... .......... .......... .......... 20% 13.4M 17s 2020-02-05T17:01:08.460 INFO:tasks.workunit.client.0.smithi008.stderr: 33800K .......... .......... .......... .......... .......... 20% 13.2M 17s 2020-02-05T17:01:08.464 INFO:tasks.workunit.client.0.smithi008.stderr: 33850K .......... .......... .......... .......... .......... 20% 11.3M 17s 2020-02-05T17:01:08.469 INFO:tasks.workunit.client.0.smithi008.stderr: 33900K .......... .......... .......... .......... .......... 20% 13.0M 17s 2020-02-05T17:01:08.473 INFO:tasks.workunit.client.0.smithi008.stderr: 33950K .......... .......... .......... .......... .......... 20% 12.3M 17s 2020-02-05T17:01:08.476 INFO:tasks.workunit.client.0.smithi008.stderr: 34000K .......... .......... .......... .......... .......... 20% 12.2M 17s 2020-02-05T17:01:08.481 INFO:tasks.workunit.client.0.smithi008.stderr: 34050K .......... .......... .......... .......... .......... 20% 10.0M 17s 2020-02-05T17:01:08.485 INFO:tasks.workunit.client.0.smithi008.stderr: 34100K .......... .......... .......... .......... .......... 20% 11.9M 17s 2020-02-05T17:01:08.489 INFO:tasks.workunit.client.0.smithi008.stderr: 34150K .......... .......... .......... .......... .......... 20% 11.7M 17s 2020-02-05T17:01:08.493 INFO:tasks.workunit.client.0.smithi008.stderr: 34200K .......... .......... .......... .......... .......... 20% 11.3M 17s 2020-02-05T17:01:08.500 INFO:tasks.workunit.client.0.smithi008.stderr: 34250K .......... .......... .......... .......... .......... 20% 8.03M 17s 2020-02-05T17:01:08.506 INFO:tasks.workunit.client.0.smithi008.stderr: 34300K .......... .......... .......... .......... .......... 20% 8.17M 17s 2020-02-05T17:01:08.512 INFO:tasks.workunit.client.0.smithi008.stderr: 34350K .......... .......... .......... .......... .......... 20% 8.16M 17s 2020-02-05T17:01:08.517 INFO:tasks.workunit.client.0.smithi008.stderr: 34400K .......... .......... .......... .......... .......... 20% 9.21M 17s 2020-02-05T17:01:08.526 INFO:tasks.workunit.client.0.smithi008.stderr: 34450K .......... .......... .......... .......... .......... 20% 5.34M 17s 2020-02-05T17:01:08.530 INFO:tasks.workunit.client.0.smithi008.stderr: 34500K .......... .......... .......... .......... .......... 20% 11.4M 17s 2020-02-05T17:01:08.535 INFO:tasks.workunit.client.0.smithi008.stderr: 34550K .......... .......... .......... .......... .......... 20% 10.7M 17s 2020-02-05T17:01:08.539 INFO:tasks.workunit.client.0.smithi008.stderr: 34600K .......... .......... .......... .......... .......... 20% 11.6M 17s 2020-02-05T17:01:08.544 INFO:tasks.workunit.client.0.smithi008.stderr: 34650K .......... .......... .......... .......... .......... 20% 9.80M 17s 2020-02-05T17:01:08.548 INFO:tasks.workunit.client.0.smithi008.stderr: 34700K .......... .......... .......... .......... .......... 20% 11.2M 17s 2020-02-05T17:01:08.553 INFO:tasks.workunit.client.0.smithi008.stderr: 34750K .......... .......... .......... .......... .......... 20% 9.90M 17s 2020-02-05T17:01:08.563 INFO:tasks.workunit.client.0.smithi008.stderr: 34800K .......... .......... .......... .......... .......... 20% 9.77M 17s 2020-02-05T17:01:08.564 INFO:tasks.workunit.client.0.smithi008.stderr: 34850K .......... .......... .......... .......... .......... 20% 8.45M 17s 2020-02-05T17:01:08.569 INFO:tasks.workunit.client.0.smithi008.stderr: 34900K .......... .......... .......... .......... .......... 20% 9.98M 17s 2020-02-05T17:01:08.574 INFO:tasks.workunit.client.0.smithi008.stderr: 34950K .......... .......... .......... .......... .......... 20% 9.87M 17s 2020-02-05T17:01:08.580 INFO:tasks.workunit.client.0.smithi008.stderr: 35000K .......... .......... .......... .......... .......... 20% 9.11M 17s 2020-02-05T17:01:08.586 INFO:tasks.workunit.client.0.smithi008.stderr: 35050K .......... .......... .......... .......... .......... 20% 7.46M 17s 2020-02-05T17:01:08.592 INFO:tasks.workunit.client.0.smithi008.stderr: 35100K .......... .......... .......... .......... .......... 20% 8.06M 17s 2020-02-05T17:01:08.598 INFO:tasks.workunit.client.0.smithi008.stderr: 35150K .......... .......... .......... .......... .......... 20% 8.95M 17s 2020-02-05T17:01:08.604 INFO:tasks.workunit.client.0.smithi008.stderr: 35200K .......... .......... .......... .......... .......... 20% 8.25M 17s 2020-02-05T17:01:08.611 INFO:tasks.workunit.client.0.smithi008.stderr: 35250K .......... .......... .......... .......... .......... 20% 6.66M 17s 2020-02-05T17:01:08.618 INFO:tasks.workunit.client.0.smithi008.stderr: 35300K .......... .......... .......... .......... .......... 20% 6.69M 17s 2020-02-05T17:01:08.625 INFO:tasks.workunit.client.0.smithi008.stderr: 35350K .......... .......... .......... .......... .......... 21% 6.89M 17s 2020-02-05T17:01:08.633 INFO:tasks.workunit.client.0.smithi008.stderr: 35400K .......... .......... .......... .......... .......... 21% 6.36M 17s 2020-02-05T17:01:08.643 INFO:tasks.workunit.client.0.smithi008.stderr: 35450K .......... .......... .......... .......... .......... 21% 5.17M 17s 2020-02-05T17:01:08.650 INFO:tasks.workunit.client.0.smithi008.stderr: 35500K .......... .......... .......... .......... .......... 21% 6.75M 17s 2020-02-05T17:01:08.657 INFO:tasks.workunit.client.0.smithi008.stderr: 35550K .......... .......... .......... .......... .......... 21% 6.63M 17s 2020-02-05T17:01:08.663 INFO:tasks.workunit.client.0.smithi008.stderr: 35600K .......... .......... .......... .......... .......... 21% 7.69M 17s 2020-02-05T17:01:08.669 INFO:tasks.workunit.client.0.smithi008.stderr: 35650K .......... .......... .......... .......... .......... 21% 8.33M 17s 2020-02-05T17:01:08.674 INFO:tasks.workunit.client.0.smithi008.stderr: 35700K .......... .......... .......... .......... .......... 21% 10.3M 17s 2020-02-05T17:01:08.678 INFO:tasks.workunit.client.0.smithi008.stderr: 35750K .......... .......... .......... .......... .......... 21% 10.7M 17s 2020-02-05T17:01:08.683 INFO:tasks.workunit.client.0.smithi008.stderr: 35800K .......... .......... .......... .......... .......... 21% 10.6M 17s 2020-02-05T17:01:08.688 INFO:tasks.workunit.client.0.smithi008.stderr: 35850K .......... .......... .......... .......... .......... 21% 9.25M 17s 2020-02-05T17:01:08.693 INFO:tasks.workunit.client.0.smithi008.stderr: 35900K .......... .......... .......... .......... .......... 21% 10.4M 17s 2020-02-05T17:01:08.698 INFO:tasks.workunit.client.0.smithi008.stderr: 35950K .......... .......... .......... .......... .......... 21% 9.26M 17s 2020-02-05T17:01:08.705 INFO:tasks.workunit.client.0.smithi008.stderr: 36000K .......... .......... .......... .......... .......... 21% 7.00M 17s 2020-02-05T17:01:08.713 INFO:tasks.workunit.client.0.smithi008.stderr: 36050K .......... .......... .......... .......... .......... 21% 6.38M 17s 2020-02-05T17:01:08.720 INFO:tasks.workunit.client.0.smithi008.stderr: 36100K .......... .......... .......... .......... .......... 21% 7.28M 17s 2020-02-05T17:01:08.727 INFO:tasks.workunit.client.0.smithi008.stderr: 36150K .......... .......... .......... .......... .......... 21% 6.85M 17s 2020-02-05T17:01:08.734 INFO:tasks.workunit.client.0.smithi008.stderr: 36200K .......... .......... .......... .......... .......... 21% 7.04M 17s 2020-02-05T17:01:08.742 INFO:tasks.workunit.client.0.smithi008.stderr: 36250K .......... .......... .......... .......... .......... 21% 5.93M 17s 2020-02-05T17:01:08.749 INFO:tasks.workunit.client.0.smithi008.stderr: 36300K .......... .......... .......... .......... .......... 21% 6.84M 17s 2020-02-05T17:01:08.760 INFO:tasks.workunit.client.0.smithi008.stderr: 36350K .......... .......... .......... .......... .......... 21% 4.54M 17s 2020-02-05T17:01:08.765 INFO:tasks.workunit.client.0.smithi008.stderr: 36400K .......... .......... .......... .......... .......... 21% 9.85M 17s 2020-02-05T17:01:08.772 INFO:tasks.workunit.client.0.smithi008.stderr: 36450K .......... .......... .......... .......... .......... 21% 6.84M 17s 2020-02-05T17:01:08.778 INFO:tasks.workunit.client.0.smithi008.stderr: 36500K .......... .......... .......... .......... .......... 21% 8.50M 17s 2020-02-05T17:01:08.782 INFO:tasks.workunit.client.0.smithi008.stderr: 36550K .......... .......... .......... .......... .......... 21% 12.3M 17s 2020-02-05T17:01:08.786 INFO:tasks.workunit.client.0.smithi008.stderr: 36600K .......... .......... .......... .......... .......... 21% 10.8M 17s 2020-02-05T17:01:08.791 INFO:tasks.workunit.client.0.smithi008.stderr: 36650K .......... .......... .......... .......... .......... 21% 9.39M 17s 2020-02-05T17:01:08.797 INFO:tasks.workunit.client.0.smithi008.stderr: 36700K .......... .......... .......... .......... .......... 21% 9.32M 17s 2020-02-05T17:01:08.803 INFO:tasks.workunit.client.0.smithi008.stderr: 36750K .......... .......... .......... .......... .......... 21% 8.85M 17s 2020-02-05T17:01:08.808 INFO:tasks.workunit.client.0.smithi008.stderr: 36800K .......... .......... .......... .......... .......... 21% 8.34M 17s 2020-02-05T17:01:08.814 INFO:tasks.workunit.client.0.smithi008.stderr: 36850K .......... .......... .......... .......... .......... 21% 7.26M 17s 2020-02-05T17:01:08.821 INFO:tasks.workunit.client.0.smithi008.stderr: 36900K .......... .......... .......... .......... .......... 21% 7.34M 17s 2020-02-05T17:01:08.826 INFO:tasks.workunit.client.0.smithi008.stderr: 36950K .......... .......... .......... .......... .......... 21% 10.6M 17s 2020-02-05T17:01:08.829 INFO:tasks.workunit.client.0.smithi008.stderr: 37000K .......... .......... .......... .......... .......... 21% 16.9M 17s 2020-02-05T17:01:08.833 INFO:tasks.workunit.client.0.smithi008.stderr: 37050K .......... .......... .......... .......... .......... 22% 12.1M 17s 2020-02-05T17:01:08.836 INFO:tasks.workunit.client.0.smithi008.stderr: 37100K .......... .......... .......... .......... .......... 22% 14.3M 17s 2020-02-05T17:01:08.839 INFO:tasks.workunit.client.0.smithi008.stderr: 37150K .......... .......... .......... .......... .......... 22% 14.0M 17s 2020-02-05T17:01:08.843 INFO:tasks.workunit.client.0.smithi008.stderr: 37200K .......... .......... .......... .......... .......... 22% 14.2M 17s 2020-02-05T17:01:08.847 INFO:tasks.workunit.client.0.smithi008.stderr: 37250K .......... .......... .......... .......... .......... 22% 11.3M 17s 2020-02-05T17:01:08.851 INFO:tasks.workunit.client.0.smithi008.stderr: 37300K .......... .......... .......... .......... .......... 22% 11.5M 17s 2020-02-05T17:01:08.856 INFO:tasks.workunit.client.0.smithi008.stderr: 37350K .......... .......... .......... .......... .......... 22% 10.4M 17s 2020-02-05T17:01:08.861 INFO:tasks.workunit.client.0.smithi008.stderr: 37400K .......... .......... .......... .......... .......... 22% 9.74M 17s 2020-02-05T17:01:08.868 INFO:tasks.workunit.client.0.smithi008.stderr: 37450K .......... .......... .......... .......... .......... 22% 7.84M 17s 2020-02-05T17:01:08.873 INFO:tasks.workunit.client.0.smithi008.stderr: 37500K .......... .......... .......... .......... .......... 22% 9.22M 17s 2020-02-05T17:01:08.880 INFO:tasks.workunit.client.0.smithi008.stderr: 37550K .......... .......... .......... .......... .......... 22% 7.76M 17s 2020-02-05T17:01:08.886 INFO:tasks.workunit.client.0.smithi008.stderr: 37600K .......... .......... .......... .......... .......... 22% 7.85M 17s 2020-02-05T17:01:08.895 INFO:tasks.workunit.client.0.smithi008.stderr: 37650K .......... .......... .......... .......... .......... 22% 5.47M 17s 2020-02-05T17:01:08.900 INFO:tasks.workunit.client.0.smithi008.stderr: 37700K .......... .......... .......... .......... .......... 22% 8.31M 17s 2020-02-05T17:01:08.906 INFO:tasks.workunit.client.0.smithi008.stderr: 37750K .......... .......... .......... .......... .......... 22% 8.35M 17s 2020-02-05T17:01:08.913 INFO:tasks.workunit.client.0.smithi008.stderr: 37800K .......... .......... .......... .......... .......... 22% 7.39M 17s 2020-02-05T17:01:08.921 INFO:tasks.workunit.client.0.smithi008.stderr: 37850K .......... .......... .......... .......... .......... 22% 6.15M 17s 2020-02-05T17:01:08.928 INFO:tasks.workunit.client.0.smithi008.stderr: 37900K .......... .......... .......... .......... .......... 22% 7.14M 17s 2020-02-05T17:01:08.935 INFO:tasks.workunit.client.0.smithi008.stderr: 37950K .......... .......... .......... .......... .......... 22% 6.47M 17s 2020-02-05T17:01:08.943 INFO:tasks.workunit.client.0.smithi008.stderr: 38000K .......... .......... .......... .......... .......... 22% 6.42M 17s 2020-02-05T17:01:08.951 INFO:tasks.workunit.client.0.smithi008.stderr: 38050K .......... .......... .......... .......... .......... 22% 5.96M 17s 2020-02-05T17:01:08.959 INFO:tasks.workunit.client.0.smithi008.stderr: 38100K .......... .......... .......... .......... .......... 22% 6.77M 17s 2020-02-05T17:01:08.966 INFO:tasks.workunit.client.0.smithi008.stderr: 38150K .......... .......... .......... .......... .......... 22% 6.18M 17s 2020-02-05T17:01:08.975 INFO:tasks.workunit.client.0.smithi008.stderr: 38200K .......... .......... .......... .......... .......... 22% 6.87M 17s 2020-02-05T17:01:08.982 INFO:tasks.workunit.client.0.smithi008.stderr: 38250K .......... .......... .......... .......... .......... 22% 5.62M 17s 2020-02-05T17:01:08.991 INFO:tasks.workunit.client.0.smithi008.stderr: 38300K .......... .......... .......... .......... .......... 22% 5.88M 17s 2020-02-05T17:01:08.997 INFO:tasks.workunit.client.0.smithi008.stderr: 38350K .......... .......... .......... .......... .......... 22% 6.56M 17s 2020-02-05T17:01:09.002 INFO:tasks.workunit.client.0.smithi008.stderr: 38400K .......... .......... .......... .......... .......... 22% 10.1M 17s 2020-02-05T17:01:09.008 INFO:tasks.workunit.client.0.smithi008.stderr: 38450K .......... .......... .......... .......... .......... 22% 8.66M 17s 2020-02-05T17:01:09.012 INFO:tasks.workunit.client.0.smithi008.stderr: 38500K .......... .......... .......... .......... .......... 22% 11.7M 16s 2020-02-05T17:01:09.018 INFO:tasks.workunit.client.0.smithi008.stderr: 38550K .......... .......... .......... .......... .......... 22% 7.76M 16s 2020-02-05T17:01:09.022 INFO:tasks.workunit.client.0.smithi008.stderr: 38600K .......... .......... .......... .......... .......... 22% 13.1M 16s 2020-02-05T17:01:09.026 INFO:tasks.workunit.client.0.smithi008.stderr: 38650K .......... .......... .......... .......... .......... 22% 11.2M 16s 2020-02-05T17:01:09.030 INFO:tasks.workunit.client.0.smithi008.stderr: 38700K .......... .......... .......... .......... .......... 22% 13.4M 16s 2020-02-05T17:01:09.033 INFO:tasks.workunit.client.0.smithi008.stderr: 38750K .......... .......... .......... .......... .......... 23% 14.9M 16s 2020-02-05T17:01:09.036 INFO:tasks.workunit.client.0.smithi008.stderr: 38800K .......... .......... .......... .......... .......... 23% 14.2M 16s 2020-02-05T17:01:09.041 INFO:tasks.workunit.client.0.smithi008.stderr: 38850K .......... .......... .......... .......... .......... 23% 11.4M 16s 2020-02-05T17:01:09.045 INFO:tasks.workunit.client.0.smithi008.stderr: 38900K .......... .......... .......... .......... .......... 23% 12.7M 16s 2020-02-05T17:01:09.048 INFO:tasks.workunit.client.0.smithi008.stderr: 38950K .......... .......... .......... .......... .......... 23% 13.1M 16s 2020-02-05T17:01:09.052 INFO:tasks.workunit.client.0.smithi008.stderr: 39000K .......... .......... .......... .......... .......... 23% 13.1M 16s 2020-02-05T17:01:09.057 INFO:tasks.workunit.client.0.smithi008.stderr: 39050K .......... .......... .......... .......... .......... 23% 10.9M 16s 2020-02-05T17:01:09.060 INFO:tasks.workunit.client.0.smithi008.stderr: 39100K .......... .......... .......... .......... .......... 23% 12.8M 16s 2020-02-05T17:01:09.065 INFO:tasks.workunit.client.0.smithi008.stderr: 39150K .......... .......... .......... .......... .......... 23% 12.5M 16s 2020-02-05T17:01:09.069 INFO:tasks.workunit.client.0.smithi008.stderr: 39200K .......... .......... .......... .......... .......... 23% 12.3M 16s 2020-02-05T17:01:09.073 INFO:tasks.workunit.client.0.smithi008.stderr: 39250K .......... .......... .......... .......... .......... 23% 11.4M 16s 2020-02-05T17:01:09.078 INFO:tasks.workunit.client.0.smithi008.stderr: 39300K .......... .......... .......... .......... .......... 23% 9.67M 16s 2020-02-05T17:01:09.084 INFO:tasks.workunit.client.0.smithi008.stderr: 39350K .......... .......... .......... .......... .......... 23% 8.81M 16s 2020-02-05T17:01:09.089 INFO:tasks.workunit.client.0.smithi008.stderr: 39400K .......... .......... .......... .......... .......... 23% 8.82M 16s 2020-02-05T17:01:09.095 INFO:tasks.workunit.client.0.smithi008.stderr: 39450K .......... .......... .......... .......... .......... 23% 8.17M 16s 2020-02-05T17:01:09.101 INFO:tasks.workunit.client.0.smithi008.stderr: 39500K .......... .......... .......... .......... .......... 23% 8.95M 16s 2020-02-05T17:01:09.107 INFO:tasks.workunit.client.0.smithi008.stderr: 39550K .......... .......... .......... .......... .......... 23% 7.57M 16s 2020-02-05T17:01:09.114 INFO:tasks.workunit.client.0.smithi008.stderr: 39600K .......... .......... .......... .......... .......... 23% 7.47M 16s 2020-02-05T17:01:09.122 INFO:tasks.workunit.client.0.smithi008.stderr: 39650K .......... .......... .......... .......... .......... 23% 6.19M 16s 2020-02-05T17:01:09.129 INFO:tasks.workunit.client.0.smithi008.stderr: 39700K .......... .......... .......... .......... .......... 23% 6.61M 16s 2020-02-05T17:01:09.137 INFO:tasks.workunit.client.0.smithi008.stderr: 39750K .......... .......... .......... .......... .......... 23% 6.07M 16s 2020-02-05T17:01:09.145 INFO:tasks.workunit.client.0.smithi008.stderr: 39800K .......... .......... .......... .......... .......... 23% 6.14M 16s 2020-02-05T17:01:09.156 INFO:tasks.workunit.client.0.smithi008.stderr: 39850K .......... .......... .......... .......... .......... 23% 4.72M 16s 2020-02-05T17:01:09.163 INFO:tasks.workunit.client.0.smithi008.stderr: 39900K .......... .......... .......... .......... .......... 23% 6.31M 16s 2020-02-05T17:01:09.172 INFO:tasks.workunit.client.0.smithi008.stderr: 39950K .......... .......... .......... .......... .......... 23% 5.79M 16s 2020-02-05T17:01:09.180 INFO:tasks.workunit.client.0.smithi008.stderr: 40000K .......... .......... .......... .......... .......... 23% 6.01M 16s 2020-02-05T17:01:09.189 INFO:tasks.workunit.client.0.smithi008.stderr: 40050K .......... .......... .......... .......... .......... 23% 5.37M 16s 2020-02-05T17:01:09.197 INFO:tasks.workunit.client.0.smithi008.stderr: 40100K .......... .......... .......... .......... .......... 23% 5.99M 16s 2020-02-05T17:01:09.205 INFO:tasks.workunit.client.0.smithi008.stderr: 40150K .......... .......... .......... .......... .......... 23% 6.22M 16s 2020-02-05T17:01:09.213 INFO:tasks.workunit.client.0.smithi008.stderr: 40200K .......... .......... .......... .......... .......... 23% 5.95M 16s 2020-02-05T17:01:09.222 INFO:tasks.workunit.client.0.smithi008.stderr: 40250K .......... .......... .......... .......... .......... 23% 5.31M 16s 2020-02-05T17:01:09.231 INFO:tasks.workunit.client.0.smithi008.stderr: 40300K .......... .......... .......... .......... .......... 23% 5.70M 16s 2020-02-05T17:01:09.239 INFO:tasks.workunit.client.0.smithi008.stderr: 40350K .......... .......... .......... .......... .......... 23% 5.70M 16s 2020-02-05T17:01:09.249 INFO:tasks.workunit.client.0.smithi008.stderr: 40400K .......... .......... .......... .......... .......... 23% 5.33M 16s 2020-02-05T17:01:09.263 INFO:tasks.workunit.client.0.smithi008.stderr: 40450K .......... .......... .......... .......... .......... 24% 3.22M 16s 2020-02-05T17:01:09.269 INFO:tasks.workunit.client.0.smithi008.stderr: 40500K .......... .......... .......... .......... .......... 24% 8.72M 16s 2020-02-05T17:01:09.274 INFO:tasks.workunit.client.0.smithi008.stderr: 40550K .......... .......... .......... .......... .......... 24% 9.09M 16s 2020-02-05T17:01:09.281 INFO:tasks.workunit.client.0.smithi008.stderr: 40600K .......... .......... .......... .......... .......... 24% 7.32M 16s 2020-02-05T17:01:09.288 INFO:tasks.workunit.client.0.smithi008.stderr: 40650K .......... .......... .......... .......... .......... 24% 6.44M 16s 2020-02-05T17:01:09.295 INFO:tasks.workunit.client.0.smithi008.stderr: 40700K .......... .......... .......... .......... .......... 24% 7.86M 16s 2020-02-05T17:01:09.301 INFO:tasks.workunit.client.0.smithi008.stderr: 40750K .......... .......... .......... .......... .......... 24% 8.16M 16s 2020-02-05T17:01:09.306 INFO:tasks.workunit.client.0.smithi008.stderr: 40800K .......... .......... .......... .......... .......... 24% 9.24M 16s 2020-02-05T17:01:09.312 INFO:tasks.workunit.client.0.smithi008.stderr: 40850K .......... .......... .......... .......... .......... 24% 7.92M 16s 2020-02-05T17:01:09.318 INFO:tasks.workunit.client.0.smithi008.stderr: 40900K .......... .......... .......... .......... .......... 24% 9.00M 16s 2020-02-05T17:01:09.324 INFO:tasks.workunit.client.0.smithi008.stderr: 40950K .......... .......... .......... .......... .......... 24% 7.29M 16s 2020-02-05T17:01:09.329 INFO:tasks.workunit.client.0.smithi008.stderr: 41000K .......... .......... .......... .......... .......... 24% 9.69M 16s 2020-02-05T17:01:09.334 INFO:tasks.workunit.client.0.smithi008.stderr: 41050K .......... .......... .......... .......... .......... 24% 8.76M 16s 2020-02-05T17:01:09.338 INFO:tasks.workunit.client.0.smithi008.stderr: 41100K .......... .......... .......... .......... .......... 24% 16.7M 16s 2020-02-05T17:01:09.342 INFO:tasks.workunit.client.0.smithi008.stderr: 41150K .......... .......... .......... .......... .......... 24% 11.3M 16s 2020-02-05T17:01:09.346 INFO:tasks.workunit.client.0.smithi008.stderr: 41200K .......... .......... .......... .......... .......... 24% 12.6M 16s 2020-02-05T17:01:09.350 INFO:tasks.workunit.client.0.smithi008.stderr: 41250K .......... .......... .......... .......... .......... 24% 12.1M 16s 2020-02-05T17:01:09.354 INFO:tasks.workunit.client.0.smithi008.stderr: 41300K .......... .......... .......... .......... .......... 24% 12.5M 16s 2020-02-05T17:01:09.357 INFO:tasks.workunit.client.0.smithi008.stderr: 41350K .......... .......... .......... .......... .......... 24% 13.0M 16s 2020-02-05T17:01:09.361 INFO:tasks.workunit.client.0.smithi008.stderr: 41400K .......... .......... .......... .......... .......... 24% 13.1M 16s 2020-02-05T17:01:09.366 INFO:tasks.workunit.client.0.smithi008.stderr: 41450K .......... .......... .......... .......... .......... 24% 10.1M 16s 2020-02-05T17:01:09.371 INFO:tasks.workunit.client.0.smithi008.stderr: 41500K .......... .......... .......... .......... .......... 24% 11.2M 16s 2020-02-05T17:01:09.375 INFO:tasks.workunit.client.0.smithi008.stderr: 41550K .......... .......... .......... .......... .......... 24% 10.1M 16s 2020-02-05T17:01:09.381 INFO:tasks.workunit.client.0.smithi008.stderr: 41600K .......... .......... .......... .......... .......... 24% 9.01M 16s 2020-02-05T17:01:09.388 INFO:tasks.workunit.client.0.smithi008.stderr: 41650K .......... .......... .......... .......... .......... 24% 7.11M 16s 2020-02-05T17:01:09.394 INFO:tasks.workunit.client.0.smithi008.stderr: 41700K .......... .......... .......... .......... .......... 24% 7.81M 16s 2020-02-05T17:01:09.402 INFO:tasks.workunit.client.0.smithi008.stderr: 41750K .......... .......... .......... .......... .......... 24% 6.42M 16s 2020-02-05T17:01:09.408 INFO:tasks.workunit.client.0.smithi008.stderr: 41800K .......... .......... .......... .......... .......... 24% 8.47M 16s 2020-02-05T17:01:09.413 INFO:tasks.workunit.client.0.smithi008.stderr: 41850K .......... .......... .......... .......... .......... 24% 8.35M 16s 2020-02-05T17:01:09.418 INFO:tasks.workunit.client.0.smithi008.stderr: 41900K .......... .......... .......... .......... .......... 24% 10.2M 16s 2020-02-05T17:01:09.423 INFO:tasks.workunit.client.0.smithi008.stderr: 41950K .......... .......... .......... .......... .......... 24% 10.1M 16s 2020-02-05T17:01:09.427 INFO:tasks.workunit.client.0.smithi008.stderr: 42000K .......... .......... .......... .......... .......... 24% 12.1M 16s 2020-02-05T17:01:09.431 INFO:tasks.workunit.client.0.smithi008.stderr: 42050K .......... .......... .......... .......... .......... 24% 10.7M 16s 2020-02-05T17:01:09.436 INFO:tasks.workunit.client.0.smithi008.stderr: 42100K .......... .......... .......... .......... .......... 25% 11.1M 16s 2020-02-05T17:01:09.441 INFO:tasks.workunit.client.0.smithi008.stderr: 42150K .......... .......... .......... .......... .......... 25% 10.1M 16s 2020-02-05T17:01:09.446 INFO:tasks.workunit.client.0.smithi008.stderr: 42200K .......... .......... .......... .......... .......... 25% 10.2M 16s 2020-02-05T17:01:09.452 INFO:tasks.workunit.client.0.smithi008.stderr: 42250K .......... .......... .......... .......... .......... 25% 7.92M 16s 2020-02-05T17:01:09.457 INFO:tasks.workunit.client.0.smithi008.stderr: 42300K .......... .......... .......... .......... .......... 25% 9.99M 16s 2020-02-05T17:01:09.462 INFO:tasks.workunit.client.0.smithi008.stderr: 42350K .......... .......... .......... .......... .......... 25% 9.39M 16s 2020-02-05T17:01:09.467 INFO:tasks.workunit.client.0.smithi008.stderr: 42400K .......... .......... .......... .......... .......... 25% 9.35M 16s 2020-02-05T17:01:09.473 INFO:tasks.workunit.client.0.smithi008.stderr: 42450K .......... .......... .......... .......... .......... 25% 7.82M 16s 2020-02-05T17:01:09.479 INFO:tasks.workunit.client.0.smithi008.stderr: 42500K .......... .......... .......... .......... .......... 25% 8.78M 16s 2020-02-05T17:01:09.484 INFO:tasks.workunit.client.0.smithi008.stderr: 42550K .......... .......... .......... .......... .......... 25% 8.58M 16s 2020-02-05T17:01:09.490 INFO:tasks.workunit.client.0.smithi008.stderr: 42600K .......... .......... .......... .......... .......... 25% 8.66M 16s 2020-02-05T17:01:09.500 INFO:tasks.workunit.client.0.smithi008.stderr: 42650K .......... .......... .......... .......... .......... 25% 4.80M 16s 2020-02-05T17:01:09.505 INFO:tasks.workunit.client.0.smithi008.stderr: 42700K .......... .......... .......... .......... .......... 25% 9.73M 16s 2020-02-05T17:01:09.510 INFO:tasks.workunit.client.0.smithi008.stderr: 42750K .......... .......... .......... .......... .......... 25% 10.2M 16s 2020-02-05T17:01:09.515 INFO:tasks.workunit.client.0.smithi008.stderr: 42800K .......... .......... .......... .......... .......... 25% 9.87M 16s 2020-02-05T17:01:09.522 INFO:tasks.workunit.client.0.smithi008.stderr: 42850K .......... .......... .......... .......... .......... 25% 6.96M 16s 2020-02-05T17:01:09.528 INFO:tasks.workunit.client.0.smithi008.stderr: 42900K .......... .......... .......... .......... .......... 25% 7.74M 16s 2020-02-05T17:01:09.535 INFO:tasks.workunit.client.0.smithi008.stderr: 42950K .......... .......... .......... .......... .......... 25% 7.27M 16s 2020-02-05T17:01:09.542 INFO:tasks.workunit.client.0.smithi008.stderr: 43000K .......... .......... .......... .......... .......... 25% 7.15M 16s 2020-02-05T17:01:09.549 INFO:tasks.workunit.client.0.smithi008.stderr: 43050K .......... .......... .......... .......... .......... 25% 6.64M 16s 2020-02-05T17:01:09.554 INFO:tasks.workunit.client.0.smithi008.stderr: 43100K .......... .......... .......... .......... .......... 25% 9.83M 16s 2020-02-05T17:01:09.559 INFO:tasks.workunit.client.0.smithi008.stderr: 43150K .......... .......... .......... .......... .......... 25% 10.9M 16s 2020-02-05T17:01:09.564 INFO:tasks.workunit.client.0.smithi008.stderr: 43200K .......... .......... .......... .......... .......... 25% 8.53M 16s 2020-02-05T17:01:09.570 INFO:tasks.workunit.client.0.smithi008.stderr: 43250K .......... .......... .......... .......... .......... 25% 7.93M 16s 2020-02-05T17:01:09.578 INFO:tasks.workunit.client.0.smithi008.stderr: 43300K .......... .......... .......... .......... .......... 25% 9.82M 16s 2020-02-05T17:01:09.581 INFO:tasks.workunit.client.0.smithi008.stderr: 43350K .......... .......... .......... .......... .......... 25% 9.43M 16s 2020-02-05T17:01:09.586 INFO:tasks.workunit.client.0.smithi008.stderr: 43400K .......... .......... .......... .......... .......... 25% 8.62M 16s 2020-02-05T17:01:09.595 INFO:tasks.workunit.client.0.smithi008.stderr: 43450K .......... .......... .......... .......... .......... 25% 5.96M 16s 2020-02-05T17:01:09.603 INFO:tasks.workunit.client.0.smithi008.stderr: 43500K .......... .......... .......... .......... .......... 25% 6.27M 16s 2020-02-05T17:01:09.611 INFO:tasks.workunit.client.0.smithi008.stderr: 43550K .......... .......... .......... .......... .......... 25% 6.31M 16s 2020-02-05T17:01:09.617 INFO:tasks.workunit.client.0.smithi008.stderr: 43600K .......... .......... .......... .......... .......... 25% 7.02M 16s 2020-02-05T17:01:09.626 INFO:tasks.workunit.client.0.smithi008.stderr: 43650K .......... .......... .......... .......... .......... 25% 6.05M 16s 2020-02-05T17:01:09.631 INFO:tasks.workunit.client.0.smithi008.stderr: 43700K .......... .......... .......... .......... .......... 25% 7.96M 16s 2020-02-05T17:01:09.637 INFO:tasks.workunit.client.0.smithi008.stderr: 43750K .......... .......... .......... .......... .......... 25% 8.51M 16s 2020-02-05T17:01:09.642 INFO:tasks.workunit.client.0.smithi008.stderr: 43800K .......... .......... .......... .......... .......... 26% 8.70M 16s 2020-02-05T17:01:09.650 INFO:tasks.workunit.client.0.smithi008.stderr: 43850K .......... .......... .......... .......... .......... 26% 6.59M 16s 2020-02-05T17:01:09.657 INFO:tasks.workunit.client.0.smithi008.stderr: 43900K .......... .......... .......... .......... .......... 26% 7.25M 16s 2020-02-05T17:01:09.664 INFO:tasks.workunit.client.0.smithi008.stderr: 43950K .......... .......... .......... .......... .......... 26% 6.69M 16s 2020-02-05T17:01:09.672 INFO:tasks.workunit.client.0.smithi008.stderr: 44000K .......... .......... .......... .......... .......... 26% 6.10M 16s 2020-02-05T17:01:09.681 INFO:tasks.workunit.client.0.smithi008.stderr: 44050K .......... .......... .......... .......... .......... 26% 5.36M 16s 2020-02-05T17:01:09.687 INFO:tasks.workunit.client.0.smithi008.stderr: 44100K .......... .......... .......... .......... .......... 26% 8.14M 16s 2020-02-05T17:01:09.693 INFO:tasks.workunit.client.0.smithi008.stderr: 44150K .......... .......... .......... .......... .......... 26% 7.87M 16s 2020-02-05T17:01:09.700 INFO:tasks.workunit.client.0.smithi008.stderr: 44200K .......... .......... .......... .......... .......... 26% 7.55M 16s 2020-02-05T17:01:09.709 INFO:tasks.workunit.client.0.smithi008.stderr: 44250K .......... .......... .......... .......... .......... 26% 5.78M 16s 2020-02-05T17:01:09.717 INFO:tasks.workunit.client.0.smithi008.stderr: 44300K .......... .......... .......... .......... .......... 26% 6.12M 16s 2020-02-05T17:01:09.725 INFO:tasks.workunit.client.0.smithi008.stderr: 44350K .......... .......... .......... .......... .......... 26% 6.03M 16s 2020-02-05T17:01:09.733 INFO:tasks.workunit.client.0.smithi008.stderr: 44400K .......... .......... .......... .......... .......... 26% 6.09M 16s 2020-02-05T17:01:09.742 INFO:tasks.workunit.client.0.smithi008.stderr: 44450K .......... .......... .......... .......... .......... 26% 5.35M 16s 2020-02-05T17:01:09.757 INFO:tasks.workunit.client.0.smithi008.stderr: 44500K .......... .......... .......... .......... .......... 26% 6.29M 16s 2020-02-05T17:01:09.763 INFO:tasks.workunit.client.0.smithi008.stderr: 44550K .......... .......... .......... .......... .......... 26% 3.58M 16s 2020-02-05T17:01:09.768 INFO:tasks.workunit.client.0.smithi008.stderr: 44600K .......... .......... .......... .......... .......... 26% 8.98M 16s 2020-02-05T17:01:09.775 INFO:tasks.workunit.client.0.smithi008.stderr: 44650K .......... .......... .......... .......... .......... 26% 7.06M 16s 2020-02-05T17:01:09.781 INFO:tasks.workunit.client.0.smithi008.stderr: 44700K .......... .......... .......... .......... .......... 26% 8.49M 16s 2020-02-05T17:01:09.787 INFO:tasks.workunit.client.0.smithi008.stderr: 44750K .......... .......... .......... .......... .......... 26% 8.68M 16s 2020-02-05T17:01:09.793 INFO:tasks.workunit.client.0.smithi008.stderr: 44800K .......... .......... .......... .......... .......... 26% 7.80M 16s 2020-02-05T17:01:09.801 INFO:tasks.workunit.client.0.smithi008.stderr: 44850K .......... .......... .......... .......... .......... 26% 6.29M 16s 2020-02-05T17:01:09.809 INFO:tasks.workunit.client.0.smithi008.stderr: 44900K .......... .......... .......... .......... .......... 26% 6.51M 16s 2020-02-05T17:01:09.817 INFO:tasks.workunit.client.0.smithi008.stderr: 44950K .......... .......... .......... .......... .......... 26% 5.94M 16s 2020-02-05T17:01:09.824 INFO:tasks.workunit.client.0.smithi008.stderr: 45000K .......... .......... .......... .......... .......... 26% 6.14M 16s 2020-02-05T17:01:09.834 INFO:tasks.workunit.client.0.smithi008.stderr: 45050K .......... .......... .......... .......... .......... 26% 5.31M 16s 2020-02-05T17:01:09.840 INFO:tasks.workunit.client.0.smithi008.stderr: 45100K .......... .......... .......... .......... .......... 26% 7.45M 16s 2020-02-05T17:01:09.843 INFO:tasks.workunit.client.0.smithi008.stderr: 45150K .......... .......... .......... .......... .......... 26% 12.9M 16s 2020-02-05T17:01:09.846 INFO:tasks.workunit.client.0.smithi008.stderr: 45200K .......... .......... .......... .......... .......... 26% 16.5M 16s 2020-02-05T17:01:09.850 INFO:tasks.workunit.client.0.smithi008.stderr: 45250K .......... .......... .......... .......... .......... 26% 14.6M 16s 2020-02-05T17:01:09.852 INFO:tasks.workunit.client.0.smithi008.stderr: 45300K .......... .......... .......... .......... .......... 26% 17.3M 16s 2020-02-05T17:01:09.856 INFO:tasks.workunit.client.0.smithi008.stderr: 45350K .......... .......... .......... .......... .......... 26% 15.0M 16s 2020-02-05T17:01:09.859 INFO:tasks.workunit.client.0.smithi008.stderr: 45400K .......... .......... .......... .......... .......... 26% 14.9M 16s 2020-02-05T17:01:09.863 INFO:tasks.workunit.client.0.smithi008.stderr: 45450K .......... .......... .......... .......... .......... 26% 11.9M 16s 2020-02-05T17:01:09.867 INFO:tasks.workunit.client.0.smithi008.stderr: 45500K .......... .......... .......... .......... .......... 27% 12.4M 16s 2020-02-05T17:01:09.871 INFO:tasks.workunit.client.0.smithi008.stderr: 45550K .......... .......... .......... .......... .......... 27% 12.3M 16s 2020-02-05T17:01:09.876 INFO:tasks.workunit.client.0.smithi008.stderr: 45600K .......... .......... .......... .......... .......... 27% 10.7M 15s 2020-02-05T17:01:09.882 INFO:tasks.workunit.client.0.smithi008.stderr: 45650K .......... .......... .......... .......... .......... 27% 8.68M 15s 2020-02-05T17:01:09.887 INFO:tasks.workunit.client.0.smithi008.stderr: 45700K .......... .......... .......... .......... .......... 27% 9.51M 15s 2020-02-05T17:01:09.892 INFO:tasks.workunit.client.0.smithi008.stderr: 45750K .......... .......... .......... .......... .......... 27% 8.81M 15s 2020-02-05T17:01:09.898 INFO:tasks.workunit.client.0.smithi008.stderr: 45800K .......... .......... .......... .......... .......... 27% 8.43M 15s 2020-02-05T17:01:09.905 INFO:tasks.workunit.client.0.smithi008.stderr: 45850K .......... .......... .......... .......... .......... 27% 6.83M 15s 2020-02-05T17:01:09.910 INFO:tasks.workunit.client.0.smithi008.stderr: 45900K .......... .......... .......... .......... .......... 27% 9.86M 15s 2020-02-05T17:01:09.915 INFO:tasks.workunit.client.0.smithi008.stderr: 45950K .......... .......... .......... .......... .......... 27% 10.1M 15s 2020-02-05T17:01:09.920 INFO:tasks.workunit.client.0.smithi008.stderr: 46000K .......... .......... .......... .......... .......... 27% 10.5M 15s 2020-02-05T17:01:09.925 INFO:tasks.workunit.client.0.smithi008.stderr: 46050K .......... .......... .......... .......... .......... 27% 9.18M 15s 2020-02-05T17:01:09.929 INFO:tasks.workunit.client.0.smithi008.stderr: 46100K .......... .......... .......... .......... .......... 27% 11.1M 15s 2020-02-05T17:01:09.934 INFO:tasks.workunit.client.0.smithi008.stderr: 46150K .......... .......... .......... .......... .......... 27% 10.7M 15s 2020-02-05T17:01:09.939 INFO:tasks.workunit.client.0.smithi008.stderr: 46200K .......... .......... .......... .......... .......... 27% 10.4M 15s 2020-02-05T17:01:09.944 INFO:tasks.workunit.client.0.smithi008.stderr: 46250K .......... .......... .......... .......... .......... 27% 8.42M 15s 2020-02-05T17:01:09.949 INFO:tasks.workunit.client.0.smithi008.stderr: 46300K .......... .......... .......... .......... .......... 27% 10.4M 15s 2020-02-05T17:01:09.953 INFO:tasks.workunit.client.0.smithi008.stderr: 46350K .......... .......... .......... .......... .......... 27% 12.8M 15s 2020-02-05T17:01:09.956 INFO:tasks.workunit.client.0.smithi008.stderr: 46400K .......... .......... .......... .......... .......... 27% 13.5M 15s 2020-02-05T17:01:09.960 INFO:tasks.workunit.client.0.smithi008.stderr: 46450K .......... .......... .......... .......... .......... 27% 14.0M 15s 2020-02-05T17:01:09.963 INFO:tasks.workunit.client.0.smithi008.stderr: 46500K .......... .......... .......... .......... .......... 27% 16.0M 15s 2020-02-05T17:01:09.966 INFO:tasks.workunit.client.0.smithi008.stderr: 46550K .......... .......... .......... .......... .......... 27% 15.7M 15s 2020-02-05T17:01:09.969 INFO:tasks.workunit.client.0.smithi008.stderr: 46600K .......... .......... .......... .......... .......... 27% 15.1M 15s 2020-02-05T17:01:09.973 INFO:tasks.workunit.client.0.smithi008.stderr: 46650K .......... .......... .......... .......... .......... 27% 12.5M 15s 2020-02-05T17:01:09.976 INFO:tasks.workunit.client.0.smithi008.stderr: 46700K .......... .......... .......... .......... .......... 27% 15.7M 15s 2020-02-05T17:01:09.981 INFO:tasks.workunit.client.0.smithi008.stderr: 46750K .......... .......... .......... .......... .......... 27% 9.89M 15s 2020-02-05T17:01:09.983 INFO:tasks.workunit.client.0.smithi008.stderr: 46800K .......... .......... .......... .......... .......... 27% 18.6M 15s 2020-02-05T17:01:09.986 INFO:tasks.workunit.client.0.smithi008.stderr: 46850K .......... .......... .......... .......... .......... 27% 16.1M 15s 2020-02-05T17:01:09.989 INFO:tasks.workunit.client.0.smithi008.stderr: 46900K .......... .......... .......... .......... .......... 27% 20.0M 15s 2020-02-05T17:01:09.991 INFO:tasks.workunit.client.0.smithi008.stderr: 46950K .......... .......... .......... .......... .......... 27% 20.3M 15s 2020-02-05T17:01:09.994 INFO:tasks.workunit.client.0.smithi008.stderr: 47000K .......... .......... .......... .......... .......... 27% 20.5M 15s 2020-02-05T17:01:09.997 INFO:tasks.workunit.client.0.smithi008.stderr: 47050K .......... .......... .......... .......... .......... 27% 16.6M 15s 2020-02-05T17:01:09.999 INFO:tasks.workunit.client.0.smithi008.stderr: 47100K .......... .......... .......... .......... .......... 27% 19.9M 15s 2020-02-05T17:01:10.002 INFO:tasks.workunit.client.0.smithi008.stderr: 47150K .......... .......... .......... .......... .......... 28% 19.1M 15s 2020-02-05T17:01:10.004 INFO:tasks.workunit.client.0.smithi008.stderr: 47200K .......... .......... .......... .......... .......... 28% 19.5M 15s 2020-02-05T17:01:10.007 INFO:tasks.workunit.client.0.smithi008.stderr: 47250K .......... .......... .......... .......... .......... 28% 17.3M 15s 2020-02-05T17:01:10.009 INFO:tasks.workunit.client.0.smithi008.stderr: 47300K .......... .......... .......... .......... .......... 28% 19.9M 15s 2020-02-05T17:01:10.012 INFO:tasks.workunit.client.0.smithi008.stderr: 47350K .......... .......... .......... .......... .......... 28% 19.8M 15s 2020-02-05T17:01:10.014 INFO:tasks.workunit.client.0.smithi008.stderr: 47400K .......... .......... .......... .......... .......... 28% 19.9M 15s 2020-02-05T17:01:10.017 INFO:tasks.workunit.client.0.smithi008.stderr: 47450K .......... .......... .......... .......... .......... 28% 16.9M 15s 2020-02-05T17:01:10.020 INFO:tasks.workunit.client.0.smithi008.stderr: 47500K .......... .......... .......... .......... .......... 28% 19.2M 15s 2020-02-05T17:01:10.022 INFO:tasks.workunit.client.0.smithi008.stderr: 47550K .......... .......... .......... .......... .......... 28% 18.4M 15s 2020-02-05T17:01:10.025 INFO:tasks.workunit.client.0.smithi008.stderr: 47600K .......... .......... .......... .......... .......... 28% 19.5M 15s 2020-02-05T17:01:10.028 INFO:tasks.workunit.client.0.smithi008.stderr: 47650K .......... .......... .......... .......... .......... 28% 16.9M 15s 2020-02-05T17:01:10.030 INFO:tasks.workunit.client.0.smithi008.stderr: 47700K .......... .......... .......... .......... .......... 28% 19.1M 15s 2020-02-05T17:01:10.033 INFO:tasks.workunit.client.0.smithi008.stderr: 47750K .......... .......... .......... .......... .......... 28% 18.8M 15s 2020-02-05T17:01:10.036 INFO:tasks.workunit.client.0.smithi008.stderr: 47800K .......... .......... .......... .......... .......... 28% 18.9M 15s 2020-02-05T17:01:10.039 INFO:tasks.workunit.client.0.smithi008.stderr: 47850K .......... .......... .......... .......... .......... 28% 15.7M 15s 2020-02-05T17:01:10.041 INFO:tasks.workunit.client.0.smithi008.stderr: 47900K .......... .......... .......... .......... .......... 28% 18.3M 15s 2020-02-05T17:01:10.044 INFO:tasks.workunit.client.0.smithi008.stderr: 47950K .......... .......... .......... .......... .......... 28% 18.4M 15s 2020-02-05T17:01:10.047 INFO:tasks.workunit.client.0.smithi008.stderr: 48000K .......... .......... .......... .......... .......... 28% 18.2M 15s 2020-02-05T17:01:10.051 INFO:tasks.workunit.client.0.smithi008.stderr: 48050K .......... .......... .......... .......... .......... 28% 11.6M 15s 2020-02-05T17:01:10.056 INFO:tasks.workunit.client.0.smithi008.stderr: 48100K .......... .......... .......... .......... .......... 28% 11.1M 15s 2020-02-05T17:01:10.061 INFO:tasks.workunit.client.0.smithi008.stderr: 48150K .......... .......... .......... .......... .......... 28% 9.54M 15s 2020-02-05T17:01:10.065 INFO:tasks.workunit.client.0.smithi008.stderr: 48200K .......... .......... .......... .......... .......... 28% 10.8M 15s 2020-02-05T17:01:10.071 INFO:tasks.workunit.client.0.smithi008.stderr: 48250K .......... .......... .......... .......... .......... 28% 8.35M 15s 2020-02-05T17:01:10.076 INFO:tasks.workunit.client.0.smithi008.stderr: 48300K .......... .......... .......... .......... .......... 28% 9.64M 15s 2020-02-05T17:01:10.082 INFO:tasks.workunit.client.0.smithi008.stderr: 48350K .......... .......... .......... .......... .......... 28% 9.11M 15s 2020-02-05T17:01:10.087 INFO:tasks.workunit.client.0.smithi008.stderr: 48400K .......... .......... .......... .......... .......... 28% 8.81M 15s 2020-02-05T17:01:10.096 INFO:tasks.workunit.client.0.smithi008.stderr: 48450K .......... .......... .......... .......... .......... 28% 5.64M 15s 2020-02-05T17:01:10.104 INFO:tasks.workunit.client.0.smithi008.stderr: 48500K .......... .......... .......... .......... .......... 28% 5.99M 15s 2020-02-05T17:01:10.114 INFO:tasks.workunit.client.0.smithi008.stderr: 48550K .......... .......... .......... .......... .......... 28% 5.50M 15s 2020-02-05T17:01:10.130 INFO:tasks.workunit.client.0.smithi008.stderr: 48600K .......... .......... .......... .......... .......... 28% 5.44M 15s 2020-02-05T17:01:10.138 INFO:tasks.workunit.client.0.smithi008.stderr: 48650K .......... .......... .......... .......... .......... 28% 3.15M 15s 2020-02-05T17:01:10.144 INFO:tasks.workunit.client.0.smithi008.stderr: 48700K .......... .......... .......... .......... .......... 28% 7.72M 15s 2020-02-05T17:01:10.149 INFO:tasks.workunit.client.0.smithi008.stderr: 48750K .......... .......... .......... .......... .......... 28% 8.92M 15s 2020-02-05T17:01:10.155 INFO:tasks.workunit.client.0.smithi008.stderr: 48800K .......... .......... .......... .......... .......... 28% 8.47M 15s 2020-02-05T17:01:10.162 INFO:tasks.workunit.client.0.smithi008.stderr: 48850K .......... .......... .......... .......... .......... 29% 7.23M 15s 2020-02-05T17:01:10.167 INFO:tasks.workunit.client.0.smithi008.stderr: 48900K .......... .......... .......... .......... .......... 29% 8.86M 15s 2020-02-05T17:01:10.172 INFO:tasks.workunit.client.0.smithi008.stderr: 48950K .......... .......... .......... .......... .......... 29% 9.86M 15s 2020-02-05T17:01:10.177 INFO:tasks.workunit.client.0.smithi008.stderr: 49000K .......... .......... .......... .......... .......... 29% 9.99M 15s 2020-02-05T17:01:10.183 INFO:tasks.workunit.client.0.smithi008.stderr: 49050K .......... .......... .......... .......... .......... 29% 7.91M 15s 2020-02-05T17:01:10.190 INFO:tasks.workunit.client.0.smithi008.stderr: 49100K .......... .......... .......... .......... .......... 29% 7.76M 15s 2020-02-05T17:01:10.194 INFO:tasks.workunit.client.0.smithi008.stderr: 49150K .......... .......... .......... .......... .......... 29% 9.82M 15s 2020-02-05T17:01:10.200 INFO:tasks.workunit.client.0.smithi008.stderr: 49200K .......... .......... .......... .......... .......... 29% 9.10M 15s 2020-02-05T17:01:10.205 INFO:tasks.workunit.client.0.smithi008.stderr: 49250K .......... .......... .......... .......... .......... 29% 8.86M 15s 2020-02-05T17:01:10.210 INFO:tasks.workunit.client.0.smithi008.stderr: 49300K .......... .......... .......... .......... .......... 29% 9.39M 15s 2020-02-05T17:01:10.215 INFO:tasks.workunit.client.0.smithi008.stderr: 49350K .......... .......... .......... .......... .......... 29% 11.5M 15s 2020-02-05T17:01:10.218 INFO:tasks.workunit.client.0.smithi008.stderr: 49400K .......... .......... .......... .......... .......... 29% 14.3M 15s 2020-02-05T17:01:10.222 INFO:tasks.workunit.client.0.smithi008.stderr: 49450K .......... .......... .......... .......... .......... 29% 12.8M 15s 2020-02-05T17:01:10.225 INFO:tasks.workunit.client.0.smithi008.stderr: 49500K .......... .......... .......... .......... .......... 29% 14.4M 15s 2020-02-05T17:01:10.229 INFO:tasks.workunit.client.0.smithi008.stderr: 49550K .......... .......... .......... .......... .......... 29% 12.6M 15s 2020-02-05T17:01:10.232 INFO:tasks.workunit.client.0.smithi008.stderr: 49600K .......... .......... .......... .......... .......... 29% 16.1M 15s 2020-02-05T17:01:10.236 INFO:tasks.workunit.client.0.smithi008.stderr: 49650K .......... .......... .......... .......... .......... 29% 13.1M 15s 2020-02-05T17:01:10.239 INFO:tasks.workunit.client.0.smithi008.stderr: 49700K .......... .......... .......... .......... .......... 29% 13.9M 15s 2020-02-05T17:01:10.243 INFO:tasks.workunit.client.0.smithi008.stderr: 49750K .......... .......... .......... .......... .......... 29% 11.5M 15s 2020-02-05T17:01:10.250 INFO:tasks.workunit.client.0.smithi008.stderr: 49800K .......... .......... .......... .......... .......... 29% 8.58M 15s 2020-02-05T17:01:10.254 INFO:tasks.workunit.client.0.smithi008.stderr: 49850K .......... .......... .......... .......... .......... 29% 9.17M 15s 2020-02-05T17:01:10.258 INFO:tasks.workunit.client.0.smithi008.stderr: 49900K .......... .......... .......... .......... .......... 29% 14.6M 15s 2020-02-05T17:01:10.262 INFO:tasks.workunit.client.0.smithi008.stderr: 49950K .......... .......... .......... .......... .......... 29% 12.9M 15s 2020-02-05T17:01:10.264 INFO:tasks.workunit.client.0.smithi008.stderr: 50000K .......... .......... .......... .......... .......... 29% 18.3M 15s 2020-02-05T17:01:10.267 INFO:tasks.workunit.client.0.smithi008.stderr: 50050K .......... .......... .......... .......... .......... 29% 16.2M 15s 2020-02-05T17:01:10.270 INFO:tasks.workunit.client.0.smithi008.stderr: 50100K .......... .......... .......... .......... .......... 29% 19.7M 15s 2020-02-05T17:01:10.272 INFO:tasks.workunit.client.0.smithi008.stderr: 50150K .......... .......... .......... .......... .......... 29% 19.8M 15s 2020-02-05T17:01:10.274 INFO:tasks.workunit.client.0.smithi008.stderr: 50200K .......... .......... .......... .......... .......... 29% 19.5M 14s 2020-02-05T17:01:10.278 INFO:tasks.workunit.client.0.smithi008.stderr: 50250K .......... .......... .......... .......... .......... 29% 16.3M 14s 2020-02-05T17:01:10.280 INFO:tasks.workunit.client.0.smithi008.stderr: 50300K .......... .......... .......... .......... .......... 29% 18.7M 14s 2020-02-05T17:01:10.283 INFO:tasks.workunit.client.0.smithi008.stderr: 50350K .......... .......... .......... .......... .......... 29% 19.4M 14s 2020-02-05T17:01:10.285 INFO:tasks.workunit.client.0.smithi008.stderr: 50400K .......... .......... .......... .......... .......... 29% 19.0M 14s 2020-02-05T17:01:10.289 INFO:tasks.workunit.client.0.smithi008.stderr: 50450K .......... .......... .......... .......... .......... 29% 15.4M 14s 2020-02-05T17:01:10.292 INFO:tasks.workunit.client.0.smithi008.stderr: 50500K .......... .......... .......... .......... .......... 29% 13.1M 14s 2020-02-05T17:01:10.296 INFO:tasks.workunit.client.0.smithi008.stderr: 50550K .......... .......... .......... .......... .......... 30% 11.9M 14s 2020-02-05T17:01:10.301 INFO:tasks.workunit.client.0.smithi008.stderr: 50600K .......... .......... .......... .......... .......... 30% 10.9M 14s 2020-02-05T17:01:10.308 INFO:tasks.workunit.client.0.smithi008.stderr: 50650K .......... .......... .......... .......... .......... 30% 7.14M 14s 2020-02-05T17:01:10.314 INFO:tasks.workunit.client.0.smithi008.stderr: 50700K .......... .......... .......... .......... .......... 30% 7.82M 14s 2020-02-05T17:01:10.320 INFO:tasks.workunit.client.0.smithi008.stderr: 50750K .......... .......... .......... .......... .......... 30% 8.12M 14s 2020-02-05T17:01:10.327 INFO:tasks.workunit.client.0.smithi008.stderr: 50800K .......... .......... .......... .......... .......... 30% 7.70M 14s 2020-02-05T17:01:10.339 INFO:tasks.workunit.client.0.smithi008.stderr: 50850K .......... .......... .......... .......... .......... 30% 3.96M 14s 2020-02-05T17:01:10.345 INFO:tasks.workunit.client.0.smithi008.stderr: 50900K .......... .......... .......... .......... .......... 30% 8.16M 14s 2020-02-05T17:01:10.350 INFO:tasks.workunit.client.0.smithi008.stderr: 50950K .......... .......... .......... .......... .......... 30% 10.1M 14s 2020-02-05T17:01:10.354 INFO:tasks.workunit.client.0.smithi008.stderr: 51000K .......... .......... .......... .......... .......... 30% 12.9M 14s 2020-02-05T17:01:10.358 INFO:tasks.workunit.client.0.smithi008.stderr: 51050K .......... .......... .......... .......... .......... 30% 10.4M 14s 2020-02-05T17:01:10.362 INFO:tasks.workunit.client.0.smithi008.stderr: 51100K .......... .......... .......... .......... .......... 30% 12.2M 14s 2020-02-05T17:01:10.367 INFO:tasks.workunit.client.0.smithi008.stderr: 51150K .......... .......... .......... .......... .......... 30% 11.5M 14s 2020-02-05T17:01:10.372 INFO:tasks.workunit.client.0.smithi008.stderr: 51200K .......... .......... .......... .......... .......... 30% 8.85M 14s 2020-02-05T17:01:10.379 INFO:tasks.workunit.client.0.smithi008.stderr: 51250K .......... .......... .......... .......... .......... 30% 7.35M 14s 2020-02-05T17:01:10.384 INFO:tasks.workunit.client.0.smithi008.stderr: 51300K .......... .......... .......... .......... .......... 30% 8.83M 14s 2020-02-05T17:01:10.390 INFO:tasks.workunit.client.0.smithi008.stderr: 51350K .......... .......... .......... .......... .......... 30% 9.01M 14s 2020-02-05T17:01:10.396 INFO:tasks.workunit.client.0.smithi008.stderr: 51400K .......... .......... .......... .......... .......... 30% 8.02M 14s 2020-02-05T17:01:10.403 INFO:tasks.workunit.client.0.smithi008.stderr: 51450K .......... .......... .......... .......... .......... 30% 6.81M 14s 2020-02-05T17:01:10.409 INFO:tasks.workunit.client.0.smithi008.stderr: 51500K .......... .......... .......... .......... .......... 30% 8.28M 14s 2020-02-05T17:01:10.415 INFO:tasks.workunit.client.0.smithi008.stderr: 51550K .......... .......... .......... .......... .......... 30% 8.55M 14s 2020-02-05T17:01:10.420 INFO:tasks.workunit.client.0.smithi008.stderr: 51600K .......... .......... .......... .......... .......... 30% 8.85M 14s 2020-02-05T17:01:10.425 INFO:tasks.workunit.client.0.smithi008.stderr: 51650K .......... .......... .......... .......... .......... 30% 9.13M 14s 2020-02-05T17:01:10.431 INFO:tasks.workunit.client.0.smithi008.stderr: 51700K .......... .......... .......... .......... .......... 30% 9.30M 14s 2020-02-05T17:01:10.436 INFO:tasks.workunit.client.0.smithi008.stderr: 51750K .......... .......... .......... .......... .......... 30% 9.38M 14s 2020-02-05T17:01:10.443 INFO:tasks.workunit.client.0.smithi008.stderr: 51800K .......... .......... .......... .......... .......... 30% 9.76M 14s 2020-02-05T17:01:10.447 INFO:tasks.workunit.client.0.smithi008.stderr: 51850K .......... .......... .......... .......... .......... 30% 7.82M 14s 2020-02-05T17:01:10.453 INFO:tasks.workunit.client.0.smithi008.stderr: 51900K .......... .......... .......... .......... .......... 30% 8.89M 14s 2020-02-05T17:01:10.458 INFO:tasks.workunit.client.0.smithi008.stderr: 51950K .......... .......... .......... .......... .......... 30% 9.53M 14s 2020-02-05T17:01:10.463 INFO:tasks.workunit.client.0.smithi008.stderr: 52000K .......... .......... .......... .......... .......... 30% 9.81M 14s 2020-02-05T17:01:10.469 INFO:tasks.workunit.client.0.smithi008.stderr: 52050K .......... .......... .......... .......... .......... 30% 8.05M 14s 2020-02-05T17:01:10.474 INFO:tasks.workunit.client.0.smithi008.stderr: 52100K .......... .......... .......... .......... .......... 30% 9.23M 14s 2020-02-05T17:01:10.479 INFO:tasks.workunit.client.0.smithi008.stderr: 52150K .......... .......... .......... .......... .......... 30% 9.14M 14s 2020-02-05T17:01:10.485 INFO:tasks.workunit.client.0.smithi008.stderr: 52200K .......... .......... .......... .......... .......... 30% 8.55M 14s 2020-02-05T17:01:10.491 INFO:tasks.workunit.client.0.smithi008.stderr: 52250K .......... .......... .......... .......... .......... 31% 7.71M 14s 2020-02-05T17:01:10.497 INFO:tasks.workunit.client.0.smithi008.stderr: 52300K .......... .......... .......... .......... .......... 31% 8.92M 14s 2020-02-05T17:01:10.502 INFO:tasks.workunit.client.0.smithi008.stderr: 52350K .......... .......... .......... .......... .......... 31% 9.01M 14s 2020-02-05T17:01:10.508 INFO:tasks.workunit.client.0.smithi008.stderr: 52400K .......... .......... .......... .......... .......... 31% 9.06M 14s 2020-02-05T17:01:10.514 INFO:tasks.workunit.client.0.smithi008.stderr: 52450K .......... .......... .......... .......... .......... 31% 7.24M 14s 2020-02-05T17:01:10.520 INFO:tasks.workunit.client.0.smithi008.stderr: 52500K .......... .......... .......... .......... .......... 31% 8.66M 14s 2020-02-05T17:01:10.526 INFO:tasks.workunit.client.0.smithi008.stderr: 52550K .......... .......... .......... .......... .......... 31% 7.79M 14s 2020-02-05T17:01:10.532 INFO:tasks.workunit.client.0.smithi008.stderr: 52600K .......... .......... .......... .......... .......... 31% 8.54M 14s 2020-02-05T17:01:10.539 INFO:tasks.workunit.client.0.smithi008.stderr: 52650K .......... .......... .......... .......... .......... 31% 6.79M 14s 2020-02-05T17:01:10.550 INFO:tasks.workunit.client.0.smithi008.stderr: 52700K .......... .......... .......... .......... .......... 31% 4.29M 14s 2020-02-05T17:01:10.555 INFO:tasks.workunit.client.0.smithi008.stderr: 52750K .......... .......... .......... .......... .......... 31% 10.2M 14s 2020-02-05T17:01:10.561 INFO:tasks.workunit.client.0.smithi008.stderr: 52800K .......... .......... .......... .......... .......... 31% 7.88M 14s 2020-02-05T17:01:10.567 INFO:tasks.workunit.client.0.smithi008.stderr: 52850K .......... .......... .......... .......... .......... 31% 8.70M 14s 2020-02-05T17:01:10.573 INFO:tasks.workunit.client.0.smithi008.stderr: 52900K .......... .......... .......... .......... .......... 31% 10.6M 14s 2020-02-05T17:01:10.577 INFO:tasks.workunit.client.0.smithi008.stderr: 52950K .......... .......... .......... .......... .......... 31% 10.4M 14s 2020-02-05T17:01:10.581 INFO:tasks.workunit.client.0.smithi008.stderr: 53000K .......... .......... .......... .......... .......... 31% 10.6M 14s 2020-02-05T17:01:10.587 INFO:tasks.workunit.client.0.smithi008.stderr: 53050K .......... .......... .......... .......... .......... 31% 8.86M 14s 2020-02-05T17:01:10.592 INFO:tasks.workunit.client.0.smithi008.stderr: 53100K .......... .......... .......... .......... .......... 31% 10.2M 14s 2020-02-05T17:01:10.596 INFO:tasks.workunit.client.0.smithi008.stderr: 53150K .......... .......... .......... .......... .......... 31% 10.5M 14s 2020-02-05T17:01:10.601 INFO:tasks.workunit.client.0.smithi008.stderr: 53200K .......... .......... .......... .......... .......... 31% 10.5M 14s 2020-02-05T17:01:10.606 INFO:tasks.workunit.client.0.smithi008.stderr: 53250K .......... .......... .......... .......... .......... 31% 8.85M 14s 2020-02-05T17:01:10.610 INFO:tasks.workunit.client.0.smithi008.stderr: 53300K .......... .......... .......... .......... .......... 31% 11.4M 14s 2020-02-05T17:01:10.615 INFO:tasks.workunit.client.0.smithi008.stderr: 53350K .......... .......... .......... .......... .......... 31% 11.6M 14s 2020-02-05T17:01:10.619 INFO:tasks.workunit.client.0.smithi008.stderr: 53400K .......... .......... .......... .......... .......... 31% 10.7M 14s 2020-02-05T17:01:10.623 INFO:tasks.workunit.client.0.smithi008.stderr: 53450K .......... .......... .......... .......... .......... 31% 12.6M 14s 2020-02-05T17:01:10.627 INFO:tasks.workunit.client.0.smithi008.stderr: 53500K .......... .......... .......... .......... .......... 31% 12.8M 14s 2020-02-05T17:01:10.631 INFO:tasks.workunit.client.0.smithi008.stderr: 53550K .......... .......... .......... .......... .......... 31% 12.5M 14s 2020-02-05T17:01:10.637 INFO:tasks.workunit.client.0.smithi008.stderr: 53600K .......... .......... .......... .......... .......... 31% 12.4M 14s 2020-02-05T17:01:10.640 INFO:tasks.workunit.client.0.smithi008.stderr: 53650K .......... .......... .......... .......... .......... 31% 11.2M 14s 2020-02-05T17:01:10.643 INFO:tasks.workunit.client.0.smithi008.stderr: 53700K .......... .......... .......... .......... .......... 31% 11.6M 14s 2020-02-05T17:01:10.646 INFO:tasks.workunit.client.0.smithi008.stderr: 53750K .......... .......... .......... .......... .......... 31% 14.8M 14s 2020-02-05T17:01:10.651 INFO:tasks.workunit.client.0.smithi008.stderr: 53800K .......... .......... .......... .......... .......... 31% 15.0M 14s 2020-02-05T17:01:10.654 INFO:tasks.workunit.client.0.smithi008.stderr: 53850K .......... .......... .......... .......... .......... 31% 11.1M 14s 2020-02-05T17:01:10.658 INFO:tasks.workunit.client.0.smithi008.stderr: 53900K .......... .......... .......... .......... .......... 32% 12.3M 14s 2020-02-05T17:01:10.662 INFO:tasks.workunit.client.0.smithi008.stderr: 53950K .......... .......... .......... .......... .......... 32% 11.9M 14s 2020-02-05T17:01:10.668 INFO:tasks.workunit.client.0.smithi008.stderr: 54000K .......... .......... .......... .......... .......... 32% 9.08M 14s 2020-02-05T17:01:10.675 INFO:tasks.workunit.client.0.smithi008.stderr: 54050K .......... .......... .......... .......... .......... 32% 6.45M 14s 2020-02-05T17:01:10.681 INFO:tasks.workunit.client.0.smithi008.stderr: 54100K .......... .......... .......... .......... .......... 32% 9.26M 14s 2020-02-05T17:01:10.687 INFO:tasks.workunit.client.0.smithi008.stderr: 54150K .......... .......... .......... .......... .......... 32% 8.12M 14s 2020-02-05T17:01:10.693 INFO:tasks.workunit.client.0.smithi008.stderr: 54200K .......... .......... .......... .......... .......... 32% 8.02M 14s 2020-02-05T17:01:10.700 INFO:tasks.workunit.client.0.smithi008.stderr: 54250K .......... .......... .......... .......... .......... 32% 7.04M 14s 2020-02-05T17:01:10.706 INFO:tasks.workunit.client.0.smithi008.stderr: 54300K .......... .......... .......... .......... .......... 32% 7.74M 14s 2020-02-05T17:01:10.713 INFO:tasks.workunit.client.0.smithi008.stderr: 54350K .......... .......... .......... .......... .......... 32% 7.55M 14s 2020-02-05T17:01:10.719 INFO:tasks.workunit.client.0.smithi008.stderr: 54400K .......... .......... .......... .......... .......... 32% 7.36M 14s 2020-02-05T17:01:10.726 INFO:tasks.workunit.client.0.smithi008.stderr: 54450K .......... .......... .......... .......... .......... 32% 6.67M 14s 2020-02-05T17:01:10.732 INFO:tasks.workunit.client.0.smithi008.stderr: 54500K .......... .......... .......... .......... .......... 32% 9.39M 14s 2020-02-05T17:01:10.736 INFO:tasks.workunit.client.0.smithi008.stderr: 54550K .......... .......... .......... .......... .......... 32% 11.0M 14s 2020-02-05T17:01:10.740 INFO:tasks.workunit.client.0.smithi008.stderr: 54600K .......... .......... .......... .......... .......... 32% 11.5M 14s 2020-02-05T17:01:10.745 INFO:tasks.workunit.client.0.smithi008.stderr: 54650K .......... .......... .......... .......... .......... 32% 9.38M 14s 2020-02-05T17:01:10.750 INFO:tasks.workunit.client.0.smithi008.stderr: 54700K .......... .......... .......... .......... .......... 32% 11.1M 14s 2020-02-05T17:01:10.754 INFO:tasks.workunit.client.0.smithi008.stderr: 54750K .......... .......... .......... .......... .......... 32% 11.0M 14s 2020-02-05T17:01:10.759 INFO:tasks.workunit.client.0.smithi008.stderr: 54800K .......... .......... .......... .......... .......... 32% 10.7M 14s 2020-02-05T17:01:10.764 INFO:tasks.workunit.client.0.smithi008.stderr: 54850K .......... .......... .......... .......... .......... 32% 9.49M 14s 2020-02-05T17:01:10.771 INFO:tasks.workunit.client.0.smithi008.stderr: 54900K .......... .......... .......... .......... .......... 32% 7.78M 14s 2020-02-05T17:01:10.778 INFO:tasks.workunit.client.0.smithi008.stderr: 54950K .......... .......... .......... .......... .......... 32% 6.54M 14s 2020-02-05T17:01:10.781 INFO:tasks.workunit.client.0.smithi008.stderr: 55000K .......... .......... .......... .......... .......... 32% 12.9M 14s 2020-02-05T17:01:10.786 INFO:tasks.workunit.client.0.smithi008.stderr: 55050K .......... .......... .......... .......... .......... 32% 10.5M 14s 2020-02-05T17:01:10.792 INFO:tasks.workunit.client.0.smithi008.stderr: 55100K .......... .......... .......... .......... .......... 32% 9.11M 14s 2020-02-05T17:01:10.797 INFO:tasks.workunit.client.0.smithi008.stderr: 55150K .......... .......... .......... .......... .......... 32% 9.90M 14s 2020-02-05T17:01:10.802 INFO:tasks.workunit.client.0.smithi008.stderr: 55200K .......... .......... .......... .......... .......... 32% 8.71M 14s 2020-02-05T17:01:10.809 INFO:tasks.workunit.client.0.smithi008.stderr: 55250K .......... .......... .......... .......... .......... 32% 7.52M 14s 2020-02-05T17:01:10.815 INFO:tasks.workunit.client.0.smithi008.stderr: 55300K .......... .......... .......... .......... .......... 32% 8.14M 14s 2020-02-05T17:01:10.821 INFO:tasks.workunit.client.0.smithi008.stderr: 55350K .......... .......... .......... .......... .......... 32% 8.04M 14s 2020-02-05T17:01:10.827 INFO:tasks.workunit.client.0.smithi008.stderr: 55400K .......... .......... .......... .......... .......... 32% 7.57M 14s 2020-02-05T17:01:10.835 INFO:tasks.workunit.client.0.smithi008.stderr: 55450K .......... .......... .......... .......... .......... 32% 6.35M 14s 2020-02-05T17:01:10.842 INFO:tasks.workunit.client.0.smithi008.stderr: 55500K .......... .......... .......... .......... .......... 32% 7.67M 14s 2020-02-05T17:01:10.848 INFO:tasks.workunit.client.0.smithi008.stderr: 55550K .......... .......... .......... .......... .......... 32% 7.42M 14s 2020-02-05T17:01:10.855 INFO:tasks.workunit.client.0.smithi008.stderr: 55600K .......... .......... .......... .......... .......... 33% 7.33M 14s 2020-02-05T17:01:10.862 INFO:tasks.workunit.client.0.smithi008.stderr: 55650K .......... .......... .......... .......... .......... 33% 6.33M 14s 2020-02-05T17:01:10.869 INFO:tasks.workunit.client.0.smithi008.stderr: 55700K .......... .......... .......... .......... .......... 33% 7.25M 14s 2020-02-05T17:01:10.876 INFO:tasks.workunit.client.0.smithi008.stderr: 55750K .......... .......... .......... .......... .......... 33% 7.55M 14s 2020-02-05T17:01:10.882 INFO:tasks.workunit.client.0.smithi008.stderr: 55800K .......... .......... .......... .......... .......... 33% 7.27M 14s 2020-02-05T17:01:10.891 INFO:tasks.workunit.client.0.smithi008.stderr: 55850K .......... .......... .......... .......... .......... 33% 5.95M 14s 2020-02-05T17:01:10.899 INFO:tasks.workunit.client.0.smithi008.stderr: 55900K .......... .......... .......... .......... .......... 33% 6.14M 14s 2020-02-05T17:01:10.906 INFO:tasks.workunit.client.0.smithi008.stderr: 55950K .......... .......... .......... .......... .......... 33% 6.86M 14s 2020-02-05T17:01:10.914 INFO:tasks.workunit.client.0.smithi008.stderr: 56000K .......... .......... .......... .......... .......... 33% 6.18M 14s 2020-02-05T17:01:10.923 INFO:tasks.workunit.client.0.smithi008.stderr: 56050K .......... .......... .......... .......... .......... 33% 5.17M 14s 2020-02-05T17:01:10.932 INFO:tasks.workunit.client.0.smithi008.stderr: 56100K .......... .......... .......... .......... .......... 33% 5.89M 14s 2020-02-05T17:01:10.941 INFO:tasks.workunit.client.0.smithi008.stderr: 56150K .......... .......... .......... .......... .......... 33% 5.55M 14s 2020-02-05T17:01:10.949 INFO:tasks.workunit.client.0.smithi008.stderr: 56200K .......... .......... .......... .......... .......... 33% 5.78M 14s 2020-02-05T17:01:10.960 INFO:tasks.workunit.client.0.smithi008.stderr: 56250K .......... .......... .......... .......... .......... 33% 4.61M 14s 2020-02-05T17:01:10.967 INFO:tasks.workunit.client.0.smithi008.stderr: 56300K .......... .......... .......... .......... .......... 33% 6.44M 14s 2020-02-05T17:01:10.973 INFO:tasks.workunit.client.0.smithi008.stderr: 56350K .......... .......... .......... .......... .......... 33% 7.64M 14s 2020-02-05T17:01:10.978 INFO:tasks.workunit.client.0.smithi008.stderr: 56400K .......... .......... .......... .......... .......... 33% 9.18M 14s 2020-02-05T17:01:10.984 INFO:tasks.workunit.client.0.smithi008.stderr: 56450K .......... .......... .......... .......... .......... 33% 8.27M 14s 2020-02-05T17:01:10.989 INFO:tasks.workunit.client.0.smithi008.stderr: 56500K .......... .......... .......... .......... .......... 33% 10.1M 14s 2020-02-05T17:01:10.994 INFO:tasks.workunit.client.0.smithi008.stderr: 56550K .......... .......... .......... .......... .......... 33% 10.0M 14s 2020-02-05T17:01:10.999 INFO:tasks.workunit.client.0.smithi008.stderr: 56600K .......... .......... .......... .......... .......... 33% 10.2M 14s 2020-02-05T17:01:11.006 INFO:tasks.workunit.client.0.smithi008.stderr: 56650K .......... .......... .......... .......... .......... 33% 6.75M 14s 2020-02-05T17:01:11.012 INFO:tasks.workunit.client.0.smithi008.stderr: 56700K .......... .......... .......... .......... .......... 33% 7.55M 14s 2020-02-05T17:01:11.018 INFO:tasks.workunit.client.0.smithi008.stderr: 56750K .......... .......... .......... .......... .......... 33% 8.73M 14s 2020-02-05T17:01:11.028 INFO:tasks.workunit.client.0.smithi008.stderr: 56800K .......... .......... .......... .......... .......... 33% 4.66M 14s 2020-02-05T17:01:11.033 INFO:tasks.workunit.client.0.smithi008.stderr: 56850K .......... .......... .......... .......... .......... 33% 9.46M 14s 2020-02-05T17:01:11.038 INFO:tasks.workunit.client.0.smithi008.stderr: 56900K .......... .......... .......... .......... .......... 33% 12.1M 14s 2020-02-05T17:01:11.042 INFO:tasks.workunit.client.0.smithi008.stderr: 56950K .......... .......... .......... .......... .......... 33% 11.8M 14s 2020-02-05T17:01:11.046 INFO:tasks.workunit.client.0.smithi008.stderr: 57000K .......... .......... .......... .......... .......... 33% 12.5M 14s 2020-02-05T17:01:11.051 INFO:tasks.workunit.client.0.smithi008.stderr: 57050K .......... .......... .......... .......... .......... 33% 9.10M 14s 2020-02-05T17:01:11.057 INFO:tasks.workunit.client.0.smithi008.stderr: 57100K .......... .......... .......... .......... .......... 33% 9.00M 14s 2020-02-05T17:01:11.063 INFO:tasks.workunit.client.0.smithi008.stderr: 57150K .......... .......... .......... .......... .......... 33% 7.54M 14s 2020-02-05T17:01:11.068 INFO:tasks.workunit.client.0.smithi008.stderr: 57200K .......... .......... .......... .......... .......... 33% 9.13M 14s 2020-02-05T17:01:11.073 INFO:tasks.workunit.client.0.smithi008.stderr: 57250K .......... .......... .......... .......... .......... 33% 9.98M 14s 2020-02-05T17:01:11.077 INFO:tasks.workunit.client.0.smithi008.stderr: 57300K .......... .......... .......... .......... .......... 34% 11.8M 13s 2020-02-05T17:01:11.082 INFO:tasks.workunit.client.0.smithi008.stderr: 57350K .......... .......... .......... .......... .......... 34% 10.5M 13s 2020-02-05T17:01:11.086 INFO:tasks.workunit.client.0.smithi008.stderr: 57400K .......... .......... .......... .......... .......... 34% 10.6M 13s 2020-02-05T17:01:11.092 INFO:tasks.workunit.client.0.smithi008.stderr: 57450K .......... .......... .......... .......... .......... 34% 8.86M 13s 2020-02-05T17:01:11.096 INFO:tasks.workunit.client.0.smithi008.stderr: 57500K .......... .......... .......... .......... .......... 34% 11.1M 13s 2020-02-05T17:01:11.100 INFO:tasks.workunit.client.0.smithi008.stderr: 57550K .......... .......... .......... .......... .......... 34% 12.5M 13s 2020-02-05T17:01:11.104 INFO:tasks.workunit.client.0.smithi008.stderr: 57600K .......... .......... .......... .......... .......... 34% 12.2M 13s 2020-02-05T17:01:11.110 INFO:tasks.workunit.client.0.smithi008.stderr: 57650K .......... .......... .......... .......... .......... 34% 8.86M 13s 2020-02-05T17:01:11.117 INFO:tasks.workunit.client.0.smithi008.stderr: 57700K .......... .......... .......... .......... .......... 34% 7.44M 13s 2020-02-05T17:01:11.121 INFO:tasks.workunit.client.0.smithi008.stderr: 57750K .......... .......... .......... .......... .......... 34% 9.87M 13s 2020-02-05T17:01:11.126 INFO:tasks.workunit.client.0.smithi008.stderr: 57800K .......... .......... .......... .......... .......... 34% 10.3M 13s 2020-02-05T17:01:11.130 INFO:tasks.workunit.client.0.smithi008.stderr: 57850K .......... .......... .......... .......... .......... 34% 11.8M 13s 2020-02-05T17:01:11.134 INFO:tasks.workunit.client.0.smithi008.stderr: 57900K .......... .......... .......... .......... .......... 34% 13.6M 13s 2020-02-05T17:01:11.137 INFO:tasks.workunit.client.0.smithi008.stderr: 57950K .......... .......... .......... .......... .......... 34% 13.2M 13s 2020-02-05T17:01:11.141 INFO:tasks.workunit.client.0.smithi008.stderr: 58000K .......... .......... .......... .......... .......... 34% 12.0M 13s 2020-02-05T17:01:11.147 INFO:tasks.workunit.client.0.smithi008.stderr: 58050K .......... .......... .......... .......... .......... 34% 9.27M 13s 2020-02-05T17:01:11.152 INFO:tasks.workunit.client.0.smithi008.stderr: 58100K .......... .......... .......... .......... .......... 34% 9.96M 13s 2020-02-05T17:01:11.158 INFO:tasks.workunit.client.0.smithi008.stderr: 58150K .......... .......... .......... .......... .......... 34% 7.72M 13s 2020-02-05T17:01:11.164 INFO:tasks.workunit.client.0.smithi008.stderr: 58200K .......... .......... .......... .......... .......... 34% 8.75M 13s 2020-02-05T17:01:11.170 INFO:tasks.workunit.client.0.smithi008.stderr: 58250K .......... .......... .......... .......... .......... 34% 7.85M 13s 2020-02-05T17:01:11.174 INFO:tasks.workunit.client.0.smithi008.stderr: 58300K .......... .......... .......... .......... .......... 34% 10.6M 13s 2020-02-05T17:01:11.178 INFO:tasks.workunit.client.0.smithi008.stderr: 58350K .......... .......... .......... .......... .......... 34% 12.8M 13s 2020-02-05T17:01:11.182 INFO:tasks.workunit.client.0.smithi008.stderr: 58400K .......... .......... .......... .......... .......... 34% 12.2M 13s 2020-02-05T17:01:11.189 INFO:tasks.workunit.client.0.smithi008.stderr: 58450K .......... .......... .......... .......... .......... 34% 10.4M 13s 2020-02-05T17:01:11.191 INFO:tasks.workunit.client.0.smithi008.stderr: 58500K .......... .......... .......... .......... .......... 34% 12.8M 13s 2020-02-05T17:01:11.195 INFO:tasks.workunit.client.0.smithi008.stderr: 58550K .......... .......... .......... .......... .......... 34% 12.6M 13s 2020-02-05T17:01:11.198 INFO:tasks.workunit.client.0.smithi008.stderr: 58600K .......... .......... .......... .......... .......... 34% 12.9M 13s 2020-02-05T17:01:11.203 INFO:tasks.workunit.client.0.smithi008.stderr: 58650K .......... .......... .......... .......... .......... 34% 10.7M 13s 2020-02-05T17:01:11.207 INFO:tasks.workunit.client.0.smithi008.stderr: 58700K .......... .......... .......... .......... .......... 34% 12.8M 13s 2020-02-05T17:01:11.211 INFO:tasks.workunit.client.0.smithi008.stderr: 58750K .......... .......... .......... .......... .......... 34% 12.6M 13s 2020-02-05T17:01:11.214 INFO:tasks.workunit.client.0.smithi008.stderr: 58800K .......... .......... .......... .......... .......... 34% 12.5M 13s 2020-02-05T17:01:11.219 INFO:tasks.workunit.client.0.smithi008.stderr: 58850K .......... .......... .......... .......... .......... 34% 10.4M 13s 2020-02-05T17:01:11.223 INFO:tasks.workunit.client.0.smithi008.stderr: 58900K .......... .......... .......... .......... .......... 34% 11.7M 13s 2020-02-05T17:01:11.228 INFO:tasks.workunit.client.0.smithi008.stderr: 58950K .......... .......... .......... .......... .......... 35% 11.4M 13s 2020-02-05T17:01:11.232 INFO:tasks.workunit.client.0.smithi008.stderr: 59000K .......... .......... .......... .......... .......... 35% 11.7M 13s 2020-02-05T17:01:11.239 INFO:tasks.workunit.client.0.smithi008.stderr: 59050K .......... .......... .......... .......... .......... 35% 7.22M 13s 2020-02-05T17:01:11.242 INFO:tasks.workunit.client.0.smithi008.stderr: 59100K .......... .......... .......... .......... .......... 35% 13.8M 13s 2020-02-05T17:01:11.246 INFO:tasks.workunit.client.0.smithi008.stderr: 59150K .......... .......... .......... .......... .......... 35% 13.1M 13s 2020-02-05T17:01:11.251 INFO:tasks.workunit.client.0.smithi008.stderr: 59200K .......... .......... .......... .......... .......... 35% 11.5M 13s 2020-02-05T17:01:11.258 INFO:tasks.workunit.client.0.smithi008.stderr: 59250K .......... .......... .......... .......... .......... 35% 6.67M 13s 2020-02-05T17:01:11.264 INFO:tasks.workunit.client.0.smithi008.stderr: 59300K .......... .......... .......... .......... .......... 35% 7.73M 13s 2020-02-05T17:01:11.270 INFO:tasks.workunit.client.0.smithi008.stderr: 59350K .......... .......... .......... .......... .......... 35% 8.00M 13s 2020-02-05T17:01:11.276 INFO:tasks.workunit.client.0.smithi008.stderr: 59400K .......... .......... .......... .......... .......... 35% 8.84M 13s 2020-02-05T17:01:11.282 INFO:tasks.workunit.client.0.smithi008.stderr: 59450K .......... .......... .......... .......... .......... 35% 7.34M 13s 2020-02-05T17:01:11.287 INFO:tasks.workunit.client.0.smithi008.stderr: 59500K .......... .......... .......... .......... .......... 35% 9.81M 13s 2020-02-05T17:01:11.292 INFO:tasks.workunit.client.0.smithi008.stderr: 59550K .......... .......... .......... .......... .......... 35% 9.62M 13s 2020-02-05T17:01:11.298 INFO:tasks.workunit.client.0.smithi008.stderr: 59600K .......... .......... .......... .......... .......... 35% 8.75M 13s 2020-02-05T17:01:11.306 INFO:tasks.workunit.client.0.smithi008.stderr: 59650K .......... .......... .......... .......... .......... 35% 6.04M 13s 2020-02-05T17:01:11.314 INFO:tasks.workunit.client.0.smithi008.stderr: 59700K .......... .......... .......... .......... .......... 35% 6.62M 13s 2020-02-05T17:01:11.322 INFO:tasks.workunit.client.0.smithi008.stderr: 59750K .......... .......... .......... .......... .......... 35% 6.26M 13s 2020-02-05T17:01:11.330 INFO:tasks.workunit.client.0.smithi008.stderr: 59800K .......... .......... .......... .......... .......... 35% 5.72M 13s 2020-02-05T17:01:11.338 INFO:tasks.workunit.client.0.smithi008.stderr: 59850K .......... .......... .......... .......... .......... 35% 5.68M 13s 2020-02-05T17:01:11.345 INFO:tasks.workunit.client.0.smithi008.stderr: 59900K .......... .......... .......... .......... .......... 35% 7.45M 13s 2020-02-05T17:01:11.351 INFO:tasks.workunit.client.0.smithi008.stderr: 59950K .......... .......... .......... .......... .......... 35% 8.53M 13s 2020-02-05T17:01:11.356 INFO:tasks.workunit.client.0.smithi008.stderr: 60000K .......... .......... .......... .......... .......... 35% 9.28M 13s 2020-02-05T17:01:11.362 INFO:tasks.workunit.client.0.smithi008.stderr: 60050K .......... .......... .......... .......... .......... 35% 7.85M 13s 2020-02-05T17:01:11.367 INFO:tasks.workunit.client.0.smithi008.stderr: 60100K .......... .......... .......... .......... .......... 35% 9.60M 13s 2020-02-05T17:01:11.373 INFO:tasks.workunit.client.0.smithi008.stderr: 60150K .......... .......... .......... .......... .......... 35% 8.78M 13s 2020-02-05T17:01:11.379 INFO:tasks.workunit.client.0.smithi008.stderr: 60200K .......... .......... .......... .......... .......... 35% 8.25M 13s 2020-02-05T17:01:11.386 INFO:tasks.workunit.client.0.smithi008.stderr: 60250K .......... .......... .......... .......... .......... 35% 7.00M 13s 2020-02-05T17:01:11.393 INFO:tasks.workunit.client.0.smithi008.stderr: 60300K .......... .......... .......... .......... .......... 35% 7.93M 13s 2020-02-05T17:01:11.399 INFO:tasks.workunit.client.0.smithi008.stderr: 60350K .......... .......... .......... .......... .......... 35% 7.57M 13s 2020-02-05T17:01:11.405 INFO:tasks.workunit.client.0.smithi008.stderr: 60400K .......... .......... .......... .......... .......... 35% 7.39M 13s 2020-02-05T17:01:11.414 INFO:tasks.workunit.client.0.smithi008.stderr: 60450K .......... .......... .......... .......... .......... 35% 5.81M 13s 2020-02-05T17:01:11.421 INFO:tasks.workunit.client.0.smithi008.stderr: 60500K .......... .......... .......... .......... .......... 35% 6.86M 13s 2020-02-05T17:01:11.428 INFO:tasks.workunit.client.0.smithi008.stderr: 60550K .......... .......... .......... .......... .......... 35% 6.62M 13s 2020-02-05T17:01:11.436 INFO:tasks.workunit.client.0.smithi008.stderr: 60600K .......... .......... .......... .......... .......... 35% 6.50M 13s 2020-02-05T17:01:11.444 INFO:tasks.workunit.client.0.smithi008.stderr: 60650K .......... .......... .......... .......... .......... 36% 5.87M 13s 2020-02-05T17:01:11.451 INFO:tasks.workunit.client.0.smithi008.stderr: 60700K .......... .......... .......... .......... .......... 36% 7.13M 13s 2020-02-05T17:01:11.458 INFO:tasks.workunit.client.0.smithi008.stderr: 60750K .......... .......... .......... .......... .......... 36% 6.77M 13s 2020-02-05T17:01:11.465 INFO:tasks.workunit.client.0.smithi008.stderr: 60800K .......... .......... .......... .......... .......... 36% 6.80M 13s 2020-02-05T17:01:11.474 INFO:tasks.workunit.client.0.smithi008.stderr: 60850K .......... .......... .......... .......... .......... 36% 5.60M 13s 2020-02-05T17:01:11.489 INFO:tasks.workunit.client.0.smithi008.stderr: 60900K .......... .......... .......... .......... .......... 36% 3.33M 13s 2020-02-05T17:01:11.495 INFO:tasks.workunit.client.0.smithi008.stderr: 60950K .......... .......... .......... .......... .......... 36% 7.86M 13s 2020-02-05T17:01:11.501 INFO:tasks.workunit.client.0.smithi008.stderr: 61000K .......... .......... .......... .......... .......... 36% 7.86M 13s 2020-02-05T17:01:11.507 INFO:tasks.workunit.client.0.smithi008.stderr: 61050K .......... .......... .......... .......... .......... 36% 7.82M 13s 2020-02-05T17:01:11.512 INFO:tasks.workunit.client.0.smithi008.stderr: 61100K .......... .......... .......... .......... .......... 36% 8.61M 13s 2020-02-05T17:01:11.516 INFO:tasks.workunit.client.0.smithi008.stderr: 61150K .......... .......... .......... .......... .......... 36% 12.9M 13s 2020-02-05T17:01:11.522 INFO:tasks.workunit.client.0.smithi008.stderr: 61200K .......... .......... .......... .......... .......... 36% 9.63M 13s 2020-02-05T17:01:11.528 INFO:tasks.workunit.client.0.smithi008.stderr: 61250K .......... .......... .......... .......... .......... 36% 7.71M 13s 2020-02-05T17:01:11.532 INFO:tasks.workunit.client.0.smithi008.stderr: 61300K .......... .......... .......... .......... .......... 36% 10.3M 13s 2020-02-05T17:01:11.538 INFO:tasks.workunit.client.0.smithi008.stderr: 61350K .......... .......... .......... .......... .......... 36% 9.30M 13s 2020-02-05T17:01:11.544 INFO:tasks.workunit.client.0.smithi008.stderr: 61400K .......... .......... .......... .......... .......... 36% 8.34M 13s 2020-02-05T17:01:11.550 INFO:tasks.workunit.client.0.smithi008.stderr: 61450K .......... .......... .......... .......... .......... 36% 7.22M 13s 2020-02-05T17:01:11.555 INFO:tasks.workunit.client.0.smithi008.stderr: 61500K .......... .......... .......... .......... .......... 36% 8.92M 13s 2020-02-05T17:01:11.560 INFO:tasks.workunit.client.0.smithi008.stderr: 61550K .......... .......... .......... .......... .......... 36% 12.8M 13s 2020-02-05T17:01:11.565 INFO:tasks.workunit.client.0.smithi008.stderr: 61600K .......... .......... .......... .......... .......... 36% 8.86M 13s 2020-02-05T17:01:11.569 INFO:tasks.workunit.client.0.smithi008.stderr: 61650K .......... .......... .......... .......... .......... 36% 11.9M 13s 2020-02-05T17:01:11.572 INFO:tasks.workunit.client.0.smithi008.stderr: 61700K .......... .......... .......... .......... .......... 36% 16.2M 13s 2020-02-05T17:01:11.575 INFO:tasks.workunit.client.0.smithi008.stderr: 61750K .......... .......... .......... .......... .......... 36% 16.1M 13s 2020-02-05T17:01:11.579 INFO:tasks.workunit.client.0.smithi008.stderr: 61800K .......... .......... .......... .......... .......... 36% 12.2M 13s 2020-02-05T17:01:11.583 INFO:tasks.workunit.client.0.smithi008.stderr: 61850K .......... .......... .......... .......... .......... 36% 13.2M 13s 2020-02-05T17:01:11.586 INFO:tasks.workunit.client.0.smithi008.stderr: 61900K .......... .......... .......... .......... .......... 36% 13.0M 13s 2020-02-05T17:01:11.591 INFO:tasks.workunit.client.0.smithi008.stderr: 61950K .......... .......... .......... .......... .......... 36% 11.3M 13s 2020-02-05T17:01:11.597 INFO:tasks.workunit.client.0.smithi008.stderr: 62000K .......... .......... .......... .......... .......... 36% 11.1M 13s 2020-02-05T17:01:11.601 INFO:tasks.workunit.client.0.smithi008.stderr: 62050K .......... .......... .......... .......... .......... 36% 9.19M 13s 2020-02-05T17:01:11.605 INFO:tasks.workunit.client.0.smithi008.stderr: 62100K .......... .......... .......... .......... .......... 36% 10.8M 13s 2020-02-05T17:01:11.610 INFO:tasks.workunit.client.0.smithi008.stderr: 62150K .......... .......... .......... .......... .......... 36% 9.82M 13s 2020-02-05T17:01:11.616 INFO:tasks.workunit.client.0.smithi008.stderr: 62200K .......... .......... .......... .......... .......... 36% 8.40M 13s 2020-02-05T17:01:11.624 INFO:tasks.workunit.client.0.smithi008.stderr: 62250K .......... .......... .......... .......... .......... 36% 5.95M 13s 2020-02-05T17:01:11.629 INFO:tasks.workunit.client.0.smithi008.stderr: 62300K .......... .......... .......... .......... .......... 36% 9.87M 13s 2020-02-05T17:01:11.634 INFO:tasks.workunit.client.0.smithi008.stderr: 62350K .......... .......... .......... .......... .......... 37% 9.98M 13s 2020-02-05T17:01:11.639 INFO:tasks.workunit.client.0.smithi008.stderr: 62400K .......... .......... .......... .......... .......... 37% 10.6M 13s 2020-02-05T17:01:11.644 INFO:tasks.workunit.client.0.smithi008.stderr: 62450K .......... .......... .......... .......... .......... 37% 9.61M 13s 2020-02-05T17:01:11.648 INFO:tasks.workunit.client.0.smithi008.stderr: 62500K .......... .......... .......... .......... .......... 37% 11.4M 13s 2020-02-05T17:01:11.652 INFO:tasks.workunit.client.0.smithi008.stderr: 62550K .......... .......... .......... .......... .......... 37% 11.1M 13s 2020-02-05T17:01:11.657 INFO:tasks.workunit.client.0.smithi008.stderr: 62600K .......... .......... .......... .......... .......... 37% 10.8M 13s 2020-02-05T17:01:11.662 INFO:tasks.workunit.client.0.smithi008.stderr: 62650K .......... .......... .......... .......... .......... 37% 9.62M 13s 2020-02-05T17:01:11.666 INFO:tasks.workunit.client.0.smithi008.stderr: 62700K .......... .......... .......... .......... .......... 37% 10.9M 13s 2020-02-05T17:01:11.671 INFO:tasks.workunit.client.0.smithi008.stderr: 62750K .......... .......... .......... .......... .......... 37% 11.3M 13s 2020-02-05T17:01:11.675 INFO:tasks.workunit.client.0.smithi008.stderr: 62800K .......... .......... .......... .......... .......... 37% 10.8M 13s 2020-02-05T17:01:11.681 INFO:tasks.workunit.client.0.smithi008.stderr: 62850K .......... .......... .......... .......... .......... 37% 8.58M 13s 2020-02-05T17:01:11.686 INFO:tasks.workunit.client.0.smithi008.stderr: 62900K .......... .......... .......... .......... .......... 37% 9.23M 13s 2020-02-05T17:01:11.692 INFO:tasks.workunit.client.0.smithi008.stderr: 62950K .......... .......... .......... .......... .......... 37% 9.51M 13s 2020-02-05T17:01:11.698 INFO:tasks.workunit.client.0.smithi008.stderr: 63000K .......... .......... .......... .......... .......... 37% 8.21M 13s 2020-02-05T17:01:11.704 INFO:tasks.workunit.client.0.smithi008.stderr: 63050K .......... .......... .......... .......... .......... 37% 7.38M 13s 2020-02-05T17:01:11.710 INFO:tasks.workunit.client.0.smithi008.stderr: 63100K .......... .......... .......... .......... .......... 37% 8.28M 13s 2020-02-05T17:01:11.720 INFO:tasks.workunit.client.0.smithi008.stderr: 63150K .......... .......... .......... .......... .......... 37% 4.80M 13s 2020-02-05T17:01:11.726 INFO:tasks.workunit.client.0.smithi008.stderr: 63200K .......... .......... .......... .......... .......... 37% 8.73M 13s 2020-02-05T17:01:11.732 INFO:tasks.workunit.client.0.smithi008.stderr: 63250K .......... .......... .......... .......... .......... 37% 7.53M 13s 2020-02-05T17:01:11.738 INFO:tasks.workunit.client.0.smithi008.stderr: 63300K .......... .......... .......... .......... .......... 37% 9.01M 13s 2020-02-05T17:01:11.744 INFO:tasks.workunit.client.0.smithi008.stderr: 63350K .......... .......... .......... .......... .......... 37% 8.52M 13s 2020-02-05T17:01:11.749 INFO:tasks.workunit.client.0.smithi008.stderr: 63400K .......... .......... .......... .......... .......... 37% 8.90M 13s 2020-02-05T17:01:11.756 INFO:tasks.workunit.client.0.smithi008.stderr: 63450K .......... .......... .......... .......... .......... 37% 6.87M 13s 2020-02-05T17:01:11.763 INFO:tasks.workunit.client.0.smithi008.stderr: 63500K .......... .......... .......... .......... .......... 37% 7.30M 13s 2020-02-05T17:01:11.769 INFO:tasks.workunit.client.0.smithi008.stderr: 63550K .......... .......... .......... .......... .......... 37% 7.86M 13s 2020-02-05T17:01:11.776 INFO:tasks.workunit.client.0.smithi008.stderr: 63600K .......... .......... .......... .......... .......... 37% 7.27M 13s 2020-02-05T17:01:11.783 INFO:tasks.workunit.client.0.smithi008.stderr: 63650K .......... .......... .......... .......... .......... 37% 6.45M 13s 2020-02-05T17:01:11.790 INFO:tasks.workunit.client.0.smithi008.stderr: 63700K .......... .......... .......... .......... .......... 37% 7.22M 13s 2020-02-05T17:01:11.797 INFO:tasks.workunit.client.0.smithi008.stderr: 63750K .......... .......... .......... .......... .......... 37% 6.99M 13s 2020-02-05T17:01:11.804 INFO:tasks.workunit.client.0.smithi008.stderr: 63800K .......... .......... .......... .......... .......... 37% 7.17M 13s 2020-02-05T17:01:11.812 INFO:tasks.workunit.client.0.smithi008.stderr: 63850K .......... .......... .......... .......... .......... 37% 6.26M 13s 2020-02-05T17:01:11.818 INFO:tasks.workunit.client.0.smithi008.stderr: 63900K .......... .......... .......... .......... .......... 37% 7.42M 13s 2020-02-05T17:01:11.825 INFO:tasks.workunit.client.0.smithi008.stderr: 63950K .......... .......... .......... .......... .......... 37% 7.21M 13s 2020-02-05T17:01:11.833 INFO:tasks.workunit.client.0.smithi008.stderr: 64000K .......... .......... .......... .......... .......... 37% 7.09M 13s 2020-02-05T17:01:11.839 INFO:tasks.workunit.client.0.smithi008.stderr: 64050K .......... .......... .......... .......... .......... 38% 6.59M 13s 2020-02-05T17:01:11.847 INFO:tasks.workunit.client.0.smithi008.stderr: 64100K .......... .......... .......... .......... .......... 38% 7.40M 13s 2020-02-05T17:01:11.853 INFO:tasks.workunit.client.0.smithi008.stderr: 64150K .......... .......... .......... .......... .......... 38% 6.89M 13s 2020-02-05T17:01:11.859 INFO:tasks.workunit.client.0.smithi008.stderr: 64200K .......... .......... .......... .......... .......... 38% 7.97M 13s 2020-02-05T17:01:11.866 INFO:tasks.workunit.client.0.smithi008.stderr: 64250K .......... .......... .......... .......... .......... 38% 7.35M 13s 2020-02-05T17:01:11.871 INFO:tasks.workunit.client.0.smithi008.stderr: 64300K .......... .......... .......... .......... .......... 38% 9.46M 13s 2020-02-05T17:01:11.874 INFO:tasks.workunit.client.0.smithi008.stderr: 64350K .......... .......... .......... .......... .......... 38% 12.8M 13s 2020-02-05T17:01:11.878 INFO:tasks.workunit.client.0.smithi008.stderr: 64400K .......... .......... .......... .......... .......... 38% 12.7M 13s 2020-02-05T17:01:11.883 INFO:tasks.workunit.client.0.smithi008.stderr: 64450K .......... .......... .......... .......... .......... 38% 11.0M 13s 2020-02-05T17:01:11.887 INFO:tasks.workunit.client.0.smithi008.stderr: 64500K .......... .......... .......... .......... .......... 38% 12.0M 13s 2020-02-05T17:01:11.891 INFO:tasks.workunit.client.0.smithi008.stderr: 64550K .......... .......... .......... .......... .......... 38% 11.1M 13s 2020-02-05T17:01:11.896 INFO:tasks.workunit.client.0.smithi008.stderr: 64600K .......... .......... .......... .......... .......... 38% 10.6M 12s 2020-02-05T17:01:11.901 INFO:tasks.workunit.client.0.smithi008.stderr: 64650K .......... .......... .......... .......... .......... 38% 9.23M 12s 2020-02-05T17:01:11.906 INFO:tasks.workunit.client.0.smithi008.stderr: 64700K .......... .......... .......... .......... .......... 38% 10.2M 12s 2020-02-05T17:01:11.911 INFO:tasks.workunit.client.0.smithi008.stderr: 64750K .......... .......... .......... .......... .......... 38% 9.87M 12s 2020-02-05T17:01:11.916 INFO:tasks.workunit.client.0.smithi008.stderr: 64800K .......... .......... .......... .......... .......... 38% 10.0M 12s 2020-02-05T17:01:11.922 INFO:tasks.workunit.client.0.smithi008.stderr: 64850K .......... .......... .......... .......... .......... 38% 8.42M 12s 2020-02-05T17:01:11.927 INFO:tasks.workunit.client.0.smithi008.stderr: 64900K .......... .......... .......... .......... .......... 38% 9.31M 12s 2020-02-05T17:01:11.932 INFO:tasks.workunit.client.0.smithi008.stderr: 64950K .......... .......... .......... .......... .......... 38% 9.09M 12s 2020-02-05T17:01:11.942 INFO:tasks.workunit.client.0.smithi008.stderr: 65000K .......... .......... .......... .......... .......... 38% 4.83M 12s 2020-02-05T17:01:11.946 INFO:tasks.workunit.client.0.smithi008.stderr: 65050K .......... .......... .......... .......... .......... 38% 11.7M 12s 2020-02-05T17:01:11.951 INFO:tasks.workunit.client.0.smithi008.stderr: 65100K .......... .......... .......... .......... .......... 38% 11.4M 12s 2020-02-05T17:01:11.956 INFO:tasks.workunit.client.0.smithi008.stderr: 65150K .......... .......... .......... .......... .......... 38% 11.0M 12s 2020-02-05T17:01:11.961 INFO:tasks.workunit.client.0.smithi008.stderr: 65200K .......... .......... .......... .......... .......... 38% 8.33M 12s 2020-02-05T17:01:11.967 INFO:tasks.workunit.client.0.smithi008.stderr: 65250K .......... .......... .......... .......... .......... 38% 9.17M 12s 2020-02-05T17:01:11.972 INFO:tasks.workunit.client.0.smithi008.stderr: 65300K .......... .......... .......... .......... .......... 38% 9.46M 12s 2020-02-05T17:01:11.977 INFO:tasks.workunit.client.0.smithi008.stderr: 65350K .......... .......... .......... .......... .......... 38% 8.63M 12s 2020-02-05T17:01:11.983 INFO:tasks.workunit.client.0.smithi008.stderr: 65400K .......... .......... .......... .......... .......... 38% 8.70M 12s 2020-02-05T17:01:11.990 INFO:tasks.workunit.client.0.smithi008.stderr: 65450K .......... .......... .......... .......... .......... 38% 7.04M 12s 2020-02-05T17:01:11.996 INFO:tasks.workunit.client.0.smithi008.stderr: 65500K .......... .......... .......... .......... .......... 38% 7.91M 12s 2020-02-05T17:01:12.002 INFO:tasks.workunit.client.0.smithi008.stderr: 65550K .......... .......... .......... .......... .......... 38% 8.22M 12s 2020-02-05T17:01:12.008 INFO:tasks.workunit.client.0.smithi008.stderr: 65600K .......... .......... .......... .......... .......... 38% 10.2M 12s 2020-02-05T17:01:12.013 INFO:tasks.workunit.client.0.smithi008.stderr: 65650K .......... .......... .......... .......... .......... 38% 8.15M 12s 2020-02-05T17:01:12.016 INFO:tasks.workunit.client.0.smithi008.stderr: 65700K .......... .......... .......... .......... .......... 39% 14.8M 12s 2020-02-05T17:01:12.019 INFO:tasks.workunit.client.0.smithi008.stderr: 65750K .......... .......... .......... .......... .......... 39% 14.3M 12s 2020-02-05T17:01:12.022 INFO:tasks.workunit.client.0.smithi008.stderr: 65800K .......... .......... .......... .......... .......... 39% 15.1M 12s 2020-02-05T17:01:12.026 INFO:tasks.workunit.client.0.smithi008.stderr: 65850K .......... .......... .......... .......... .......... 39% 13.0M 12s 2020-02-05T17:01:12.030 INFO:tasks.workunit.client.0.smithi008.stderr: 65900K .......... .......... .......... .......... .......... 39% 13.2M 12s 2020-02-05T17:01:12.034 INFO:tasks.workunit.client.0.smithi008.stderr: 65950K .......... .......... .......... .......... .......... 39% 13.1M 12s 2020-02-05T17:01:12.037 INFO:tasks.workunit.client.0.smithi008.stderr: 66000K .......... .......... .......... .......... .......... 39% 13.0M 12s 2020-02-05T17:01:12.042 INFO:tasks.workunit.client.0.smithi008.stderr: 66050K .......... .......... .......... .......... .......... 39% 10.0M 12s 2020-02-05T17:01:12.047 INFO:tasks.workunit.client.0.smithi008.stderr: 66100K .......... .......... .......... .......... .......... 39% 11.2M 12s 2020-02-05T17:01:12.052 INFO:tasks.workunit.client.0.smithi008.stderr: 66150K .......... .......... .......... .......... .......... 39% 9.97M 12s 2020-02-05T17:01:12.057 INFO:tasks.workunit.client.0.smithi008.stderr: 66200K .......... .......... .......... .......... .......... 39% 9.84M 12s 2020-02-05T17:01:12.063 INFO:tasks.workunit.client.0.smithi008.stderr: 66250K .......... .......... .......... .......... .......... 39% 7.41M 12s 2020-02-05T17:01:12.070 INFO:tasks.workunit.client.0.smithi008.stderr: 66300K .......... .......... .......... .......... .......... 39% 7.57M 12s 2020-02-05T17:01:12.077 INFO:tasks.workunit.client.0.smithi008.stderr: 66350K .......... .......... .......... .......... .......... 39% 6.77M 12s 2020-02-05T17:01:12.084 INFO:tasks.workunit.client.0.smithi008.stderr: 66400K .......... .......... .......... .......... .......... 39% 7.73M 12s 2020-02-05T17:01:12.092 INFO:tasks.workunit.client.0.smithi008.stderr: 66450K .......... .......... .......... .......... .......... 39% 6.14M 12s 2020-02-05T17:01:12.099 INFO:tasks.workunit.client.0.smithi008.stderr: 66500K .......... .......... .......... .......... .......... 39% 6.99M 12s 2020-02-05T17:01:12.107 INFO:tasks.workunit.client.0.smithi008.stderr: 66550K .......... .......... .......... .......... .......... 39% 6.17M 12s 2020-02-05T17:01:12.114 INFO:tasks.workunit.client.0.smithi008.stderr: 66600K .......... .......... .......... .......... .......... 39% 6.87M 12s 2020-02-05T17:01:12.120 INFO:tasks.workunit.client.0.smithi008.stderr: 66650K .......... .......... .......... .......... .......... 39% 7.18M 12s 2020-02-05T17:01:12.127 INFO:tasks.workunit.client.0.smithi008.stderr: 66700K .......... .......... .......... .......... .......... 39% 7.69M 12s 2020-02-05T17:01:12.133 INFO:tasks.workunit.client.0.smithi008.stderr: 66750K .......... .......... .......... .......... .......... 39% 7.54M 12s 2020-02-05T17:01:12.140 INFO:tasks.workunit.client.0.smithi008.stderr: 66800K .......... .......... .......... .......... .......... 39% 7.16M 12s 2020-02-05T17:01:12.149 INFO:tasks.workunit.client.0.smithi008.stderr: 66850K .......... .......... .......... .......... .......... 39% 5.53M 12s 2020-02-05T17:01:12.156 INFO:tasks.workunit.client.0.smithi008.stderr: 66900K .......... .......... .......... .......... .......... 39% 6.77M 12s 2020-02-05T17:01:12.164 INFO:tasks.workunit.client.0.smithi008.stderr: 66950K .......... .......... .......... .......... .......... 39% 6.49M 12s 2020-02-05T17:01:12.171 INFO:tasks.workunit.client.0.smithi008.stderr: 67000K .......... .......... .......... .......... .......... 39% 6.95M 12s 2020-02-05T17:01:12.180 INFO:tasks.workunit.client.0.smithi008.stderr: 67050K .......... .......... .......... .......... .......... 39% 5.58M 12s 2020-02-05T17:01:12.187 INFO:tasks.workunit.client.0.smithi008.stderr: 67100K .......... .......... .......... .......... .......... 39% 6.72M 12s 2020-02-05T17:01:12.194 INFO:tasks.workunit.client.0.smithi008.stderr: 67150K .......... .......... .......... .......... .......... 39% 6.59M 12s 2020-02-05T17:01:12.202 INFO:tasks.workunit.client.0.smithi008.stderr: 67200K .......... .......... .......... .......... .......... 39% 6.20M 12s 2020-02-05T17:01:12.214 INFO:tasks.workunit.client.0.smithi008.stderr: 67250K .......... .......... .......... .......... .......... 39% 4.15M 12s 2020-02-05T17:01:12.221 INFO:tasks.workunit.client.0.smithi008.stderr: 67300K .......... .......... .......... .......... .......... 39% 7.33M 12s 2020-02-05T17:01:12.227 INFO:tasks.workunit.client.0.smithi008.stderr: 67350K .......... .......... .......... .......... .......... 39% 8.45M 12s 2020-02-05T17:01:12.232 INFO:tasks.workunit.client.0.smithi008.stderr: 67400K .......... .......... .......... .......... .......... 40% 7.83M 12s 2020-02-05T17:01:12.240 INFO:tasks.workunit.client.0.smithi008.stderr: 67450K .......... .......... .......... .......... .......... 40% 6.25M 12s 2020-02-05T17:01:12.247 INFO:tasks.workunit.client.0.smithi008.stderr: 67500K .......... .......... .......... .......... .......... 40% 7.05M 12s 2020-02-05T17:01:12.254 INFO:tasks.workunit.client.0.smithi008.stderr: 67550K .......... .......... .......... .......... .......... 40% 7.54M 12s 2020-02-05T17:01:12.261 INFO:tasks.workunit.client.0.smithi008.stderr: 67600K .......... .......... .......... .......... .......... 40% 6.65M 12s 2020-02-05T17:01:12.269 INFO:tasks.workunit.client.0.smithi008.stderr: 67650K .......... .......... .......... .......... .......... 40% 5.73M 12s 2020-02-05T17:01:12.277 INFO:tasks.workunit.client.0.smithi008.stderr: 67700K .......... .......... .......... .......... .......... 40% 6.58M 12s 2020-02-05T17:01:12.283 INFO:tasks.workunit.client.0.smithi008.stderr: 67750K .......... .......... .......... .......... .......... 40% 8.28M 12s 2020-02-05T17:01:12.288 INFO:tasks.workunit.client.0.smithi008.stderr: 67800K .......... .......... .......... .......... .......... 40% 9.15M 12s 2020-02-05T17:01:12.294 INFO:tasks.workunit.client.0.smithi008.stderr: 67850K .......... .......... .......... .......... .......... 40% 8.02M 12s 2020-02-05T17:01:12.300 INFO:tasks.workunit.client.0.smithi008.stderr: 67900K .......... .......... .......... .......... .......... 40% 8.40M 12s 2020-02-05T17:01:12.306 INFO:tasks.workunit.client.0.smithi008.stderr: 67950K .......... .......... .......... .......... .......... 40% 7.64M 12s 2020-02-05T17:01:12.314 INFO:tasks.workunit.client.0.smithi008.stderr: 68000K .......... .......... .......... .......... .......... 40% 6.84M 12s 2020-02-05T17:01:12.322 INFO:tasks.workunit.client.0.smithi008.stderr: 68050K .......... .......... .......... .......... .......... 40% 5.65M 12s 2020-02-05T17:01:12.330 INFO:tasks.workunit.client.0.smithi008.stderr: 68100K .......... .......... .......... .......... .......... 40% 6.31M 12s 2020-02-05T17:01:12.337 INFO:tasks.workunit.client.0.smithi008.stderr: 68150K .......... .......... .......... .......... .......... 40% 6.98M 12s 2020-02-05T17:01:12.343 INFO:tasks.workunit.client.0.smithi008.stderr: 68200K .......... .......... .......... .......... .......... 40% 7.32M 12s 2020-02-05T17:01:12.351 INFO:tasks.workunit.client.0.smithi008.stderr: 68250K .......... .......... .......... .......... .......... 40% 6.55M 12s 2020-02-05T17:01:12.357 INFO:tasks.workunit.client.0.smithi008.stderr: 68300K .......... .......... .......... .......... .......... 40% 7.64M 12s 2020-02-05T17:01:12.365 INFO:tasks.workunit.client.0.smithi008.stderr: 68350K .......... .......... .......... .......... .......... 40% 6.98M 12s 2020-02-05T17:01:12.372 INFO:tasks.workunit.client.0.smithi008.stderr: 68400K .......... .......... .......... .......... .......... 40% 6.47M 12s 2020-02-05T17:01:12.380 INFO:tasks.workunit.client.0.smithi008.stderr: 68450K .......... .......... .......... .......... .......... 40% 6.30M 12s 2020-02-05T17:01:12.387 INFO:tasks.workunit.client.0.smithi008.stderr: 68500K .......... .......... .......... .......... .......... 40% 6.75M 12s 2020-02-05T17:01:12.393 INFO:tasks.workunit.client.0.smithi008.stderr: 68550K .......... .......... .......... .......... .......... 40% 7.58M 12s 2020-02-05T17:01:12.400 INFO:tasks.workunit.client.0.smithi008.stderr: 68600K .......... .......... .......... .......... .......... 40% 7.09M 12s 2020-02-05T17:01:12.409 INFO:tasks.workunit.client.0.smithi008.stderr: 68650K .......... .......... .......... .......... .......... 40% 5.44M 12s 2020-02-05T17:01:12.418 INFO:tasks.workunit.client.0.smithi008.stderr: 68700K .......... .......... .......... .......... .......... 40% 5.90M 12s 2020-02-05T17:01:12.426 INFO:tasks.workunit.client.0.smithi008.stderr: 68750K .......... .......... .......... .......... .......... 40% 6.08M 12s 2020-02-05T17:01:12.434 INFO:tasks.workunit.client.0.smithi008.stderr: 68800K .......... .......... .......... .......... .......... 40% 6.01M 12s 2020-02-05T17:01:12.444 INFO:tasks.workunit.client.0.smithi008.stderr: 68850K .......... .......... .......... .......... .......... 40% 4.88M 12s 2020-02-05T17:01:12.454 INFO:tasks.workunit.client.0.smithi008.stderr: 68900K .......... .......... .......... .......... .......... 40% 5.18M 12s 2020-02-05T17:01:12.463 INFO:tasks.workunit.client.0.smithi008.stderr: 68950K .......... .......... .......... .......... .......... 40% 5.05M 12s 2020-02-05T17:01:12.473 INFO:tasks.workunit.client.0.smithi008.stderr: 69000K .......... .......... .......... .......... .......... 40% 4.80M 12s 2020-02-05T17:01:12.485 INFO:tasks.workunit.client.0.smithi008.stderr: 69050K .......... .......... .......... .......... .......... 40% 4.27M 12s 2020-02-05T17:01:12.498 INFO:tasks.workunit.client.0.smithi008.stderr: 69100K .......... .......... .......... .......... .......... 41% 3.52M 12s 2020-02-05T17:01:12.505 INFO:tasks.workunit.client.0.smithi008.stderr: 69150K .......... .......... .......... .......... .......... 41% 7.42M 12s 2020-02-05T17:01:12.512 INFO:tasks.workunit.client.0.smithi008.stderr: 69200K .......... .......... .......... .......... .......... 41% 7.07M 12s 2020-02-05T17:01:12.519 INFO:tasks.workunit.client.0.smithi008.stderr: 69250K .......... .......... .......... .......... .......... 41% 6.81M 12s 2020-02-05T17:01:12.526 INFO:tasks.workunit.client.0.smithi008.stderr: 69300K .......... .......... .......... .......... .......... 41% 7.24M 12s 2020-02-05T17:01:12.532 INFO:tasks.workunit.client.0.smithi008.stderr: 69350K .......... .......... .......... .......... .......... 41% 7.53M 12s 2020-02-05T17:01:12.538 INFO:tasks.workunit.client.0.smithi008.stderr: 69400K .......... .......... .......... .......... .......... 41% 8.26M 12s 2020-02-05T17:01:12.545 INFO:tasks.workunit.client.0.smithi008.stderr: 69450K .......... .......... .......... .......... .......... 41% 6.96M 12s 2020-02-05T17:01:12.551 INFO:tasks.workunit.client.0.smithi008.stderr: 69500K .......... .......... .......... .......... .......... 41% 7.61M 12s 2020-02-05T17:01:12.558 INFO:tasks.workunit.client.0.smithi008.stderr: 69550K .......... .......... .......... .......... .......... 41% 7.55M 12s 2020-02-05T17:01:12.564 INFO:tasks.workunit.client.0.smithi008.stderr: 69600K .......... .......... .......... .......... .......... 41% 8.27M 12s 2020-02-05T17:01:12.571 INFO:tasks.workunit.client.0.smithi008.stderr: 69650K .......... .......... .......... .......... .......... 41% 7.01M 12s 2020-02-05T17:01:12.575 INFO:tasks.workunit.client.0.smithi008.stderr: 69700K .......... .......... .......... .......... .......... 41% 10.2M 12s 2020-02-05T17:01:12.578 INFO:tasks.workunit.client.0.smithi008.stderr: 69750K .......... .......... .......... .......... .......... 41% 16.9M 12s 2020-02-05T17:01:12.580 INFO:tasks.workunit.client.0.smithi008.stderr: 69800K .......... .......... .......... .......... .......... 41% 17.8M 12s 2020-02-05T17:01:12.584 INFO:tasks.workunit.client.0.smithi008.stderr: 69850K .......... .......... .......... .......... .......... 41% 13.4M 12s 2020-02-05T17:01:12.588 INFO:tasks.workunit.client.0.smithi008.stderr: 69900K .......... .......... .......... .......... .......... 41% 14.3M 12s 2020-02-05T17:01:12.591 INFO:tasks.workunit.client.0.smithi008.stderr: 69950K .......... .......... .......... .......... .......... 41% 16.3M 12s 2020-02-05T17:01:12.594 INFO:tasks.workunit.client.0.smithi008.stderr: 70000K .......... .......... .......... .......... .......... 41% 15.9M 12s 2020-02-05T17:01:12.598 INFO:tasks.workunit.client.0.smithi008.stderr: 70050K .......... .......... .......... .......... .......... 41% 13.8M 12s 2020-02-05T17:01:12.601 INFO:tasks.workunit.client.0.smithi008.stderr: 70100K .......... .......... .......... .......... .......... 41% 14.2M 12s 2020-02-05T17:01:12.604 INFO:tasks.workunit.client.0.smithi008.stderr: 70150K .......... .......... .......... .......... .......... 41% 15.3M 12s 2020-02-05T17:01:12.608 INFO:tasks.workunit.client.0.smithi008.stderr: 70200K .......... .......... .......... .......... .......... 41% 13.0M 12s 2020-02-05T17:01:12.612 INFO:tasks.workunit.client.0.smithi008.stderr: 70250K .......... .......... .......... .......... .......... 41% 10.5M 12s 2020-02-05T17:01:12.617 INFO:tasks.workunit.client.0.smithi008.stderr: 70300K .......... .......... .......... .......... .......... 41% 11.8M 12s 2020-02-05T17:01:12.622 INFO:tasks.workunit.client.0.smithi008.stderr: 70350K .......... .......... .......... .......... .......... 41% 9.02M 12s 2020-02-05T17:01:12.628 INFO:tasks.workunit.client.0.smithi008.stderr: 70400K .......... .......... .......... .......... .......... 41% 8.52M 12s 2020-02-05T17:01:12.636 INFO:tasks.workunit.client.0.smithi008.stderr: 70450K .......... .......... .......... .......... .......... 41% 6.07M 12s 2020-02-05T17:01:12.641 INFO:tasks.workunit.client.0.smithi008.stderr: 70500K .......... .......... .......... .......... .......... 41% 10.3M 12s 2020-02-05T17:01:12.645 INFO:tasks.workunit.client.0.smithi008.stderr: 70550K .......... .......... .......... .......... .......... 41% 10.4M 12s 2020-02-05T17:01:12.652 INFO:tasks.workunit.client.0.smithi008.stderr: 70600K .......... .......... .......... .......... .......... 41% 7.86M 12s 2020-02-05T17:01:12.669 INFO:tasks.workunit.client.0.smithi008.stderr: 70650K .......... .......... .......... .......... .......... 41% 7.92M 12s 2020-02-05T17:01:12.669 INFO:tasks.workunit.client.0.smithi008.stderr: 70700K .......... .......... .......... .......... .......... 41% 10.2M 12s 2020-02-05T17:01:12.670 INFO:tasks.workunit.client.0.smithi008.stderr: 70750K .......... .......... .......... .......... .......... 42% 10.6M 12s 2020-02-05T17:01:12.672 INFO:tasks.workunit.client.0.smithi008.stderr: 70800K .......... .......... .......... .......... .......... 42% 10.5M 12s 2020-02-05T17:01:12.678 INFO:tasks.workunit.client.0.smithi008.stderr: 70850K .......... .......... .......... .......... .......... 42% 8.11M 12s 2020-02-05T17:01:12.684 INFO:tasks.workunit.client.0.smithi008.stderr: 70900K .......... .......... .......... .......... .......... 42% 8.51M 12s 2020-02-05T17:01:12.691 INFO:tasks.workunit.client.0.smithi008.stderr: 70950K .......... .......... .......... .......... .......... 42% 7.07M 12s 2020-02-05T17:01:12.698 INFO:tasks.workunit.client.0.smithi008.stderr: 71000K .......... .......... .......... .......... .......... 42% 7.05M 12s 2020-02-05T17:01:12.707 INFO:tasks.workunit.client.0.smithi008.stderr: 71050K .......... .......... .......... .......... .......... 42% 5.56M 12s 2020-02-05T17:01:12.714 INFO:tasks.workunit.client.0.smithi008.stderr: 71100K .......... .......... .......... .......... .......... 42% 6.37M 12s 2020-02-05T17:01:12.721 INFO:tasks.workunit.client.0.smithi008.stderr: 71150K .......... .......... .......... .......... .......... 42% 6.99M 12s 2020-02-05T17:01:12.729 INFO:tasks.workunit.client.0.smithi008.stderr: 71200K .......... .......... .......... .......... .......... 42% 6.67M 12s 2020-02-05T17:01:12.737 INFO:tasks.workunit.client.0.smithi008.stderr: 71250K .......... .......... .......... .......... .......... 42% 5.70M 12s 2020-02-05T17:01:12.745 INFO:tasks.workunit.client.0.smithi008.stderr: 71300K .......... .......... .......... .......... .......... 42% 6.56M 12s 2020-02-05T17:01:12.755 INFO:tasks.workunit.client.0.smithi008.stderr: 71350K .......... .......... .......... .......... .......... 42% 4.63M 12s 2020-02-05T17:01:12.761 INFO:tasks.workunit.client.0.smithi008.stderr: 71400K .......... .......... .......... .......... .......... 42% 7.96M 12s 2020-02-05T17:01:12.768 INFO:tasks.workunit.client.0.smithi008.stderr: 71450K .......... .......... .......... .......... .......... 42% 7.13M 12s 2020-02-05T17:01:12.773 INFO:tasks.workunit.client.0.smithi008.stderr: 71500K .......... .......... .......... .......... .......... 42% 9.99M 12s 2020-02-05T17:01:12.778 INFO:tasks.workunit.client.0.smithi008.stderr: 71550K .......... .......... .......... .......... .......... 42% 10.0M 12s 2020-02-05T17:01:12.783 INFO:tasks.workunit.client.0.smithi008.stderr: 71600K .......... .......... .......... .......... .......... 42% 9.81M 12s 2020-02-05T17:01:12.789 INFO:tasks.workunit.client.0.smithi008.stderr: 71650K .......... .......... .......... .......... .......... 42% 7.48M 12s 2020-02-05T17:01:12.795 INFO:tasks.workunit.client.0.smithi008.stderr: 71700K .......... .......... .......... .......... .......... 42% 7.98M 12s 2020-02-05T17:01:12.802 INFO:tasks.workunit.client.0.smithi008.stderr: 71750K .......... .......... .......... .......... .......... 42% 7.28M 12s 2020-02-05T17:01:12.809 INFO:tasks.workunit.client.0.smithi008.stderr: 71800K .......... .......... .......... .......... .......... 42% 6.89M 12s 2020-02-05T17:01:12.818 INFO:tasks.workunit.client.0.smithi008.stderr: 71850K .......... .......... .......... .......... .......... 42% 5.84M 12s 2020-02-05T17:01:12.825 INFO:tasks.workunit.client.0.smithi008.stderr: 71900K .......... .......... .......... .......... .......... 42% 6.38M 12s 2020-02-05T17:01:12.832 INFO:tasks.workunit.client.0.smithi008.stderr: 71950K .......... .......... .......... .......... .......... 42% 6.86M 12s 2020-02-05T17:01:12.839 INFO:tasks.workunit.client.0.smithi008.stderr: 72000K .......... .......... .......... .......... .......... 42% 7.20M 12s 2020-02-05T17:01:12.847 INFO:tasks.workunit.client.0.smithi008.stderr: 72050K .......... .......... .......... .......... .......... 42% 5.97M 12s 2020-02-05T17:01:12.854 INFO:tasks.workunit.client.0.smithi008.stderr: 72100K .......... .......... .......... .......... .......... 42% 6.88M 12s 2020-02-05T17:01:12.862 INFO:tasks.workunit.client.0.smithi008.stderr: 72150K .......... .......... .......... .......... .......... 42% 7.04M 12s 2020-02-05T17:01:12.869 INFO:tasks.workunit.client.0.smithi008.stderr: 72200K .......... .......... .......... .......... .......... 42% 6.87M 12s 2020-02-05T17:01:12.878 INFO:tasks.workunit.client.0.smithi008.stderr: 72250K .......... .......... .......... .......... .......... 42% 5.20M 12s 2020-02-05T17:01:12.885 INFO:tasks.workunit.client.0.smithi008.stderr: 72300K .......... .......... .......... .......... .......... 42% 6.59M 12s 2020-02-05T17:01:12.893 INFO:tasks.workunit.client.0.smithi008.stderr: 72350K .......... .......... .......... .......... .......... 42% 6.58M 12s 2020-02-05T17:01:12.901 INFO:tasks.workunit.client.0.smithi008.stderr: 72400K .......... .......... .......... .......... .......... 42% 6.17M 12s 2020-02-05T17:01:12.909 INFO:tasks.workunit.client.0.smithi008.stderr: 72450K .......... .......... .......... .......... .......... 43% 5.50M 12s 2020-02-05T17:01:12.915 INFO:tasks.workunit.client.0.smithi008.stderr: 72500K .......... .......... .......... .......... .......... 43% 8.15M 12s 2020-02-05T17:01:12.921 INFO:tasks.workunit.client.0.smithi008.stderr: 72550K .......... .......... .......... .......... .......... 43% 8.79M 12s 2020-02-05T17:01:12.926 INFO:tasks.workunit.client.0.smithi008.stderr: 72600K .......... .......... .......... .......... .......... 43% 9.42M 12s 2020-02-05T17:01:12.932 INFO:tasks.workunit.client.0.smithi008.stderr: 72650K .......... .......... .......... .......... .......... 43% 8.09M 12s 2020-02-05T17:01:12.938 INFO:tasks.workunit.client.0.smithi008.stderr: 72700K .......... .......... .......... .......... .......... 43% 8.64M 12s 2020-02-05T17:01:12.943 INFO:tasks.workunit.client.0.smithi008.stderr: 72750K .......... .......... .......... .......... .......... 43% 8.78M 12s 2020-02-05T17:01:12.949 INFO:tasks.workunit.client.0.smithi008.stderr: 72800K .......... .......... .......... .......... .......... 43% 8.53M 12s 2020-02-05T17:01:12.956 INFO:tasks.workunit.client.0.smithi008.stderr: 72850K .......... .......... .......... .......... .......... 43% 7.35M 12s 2020-02-05T17:01:12.962 INFO:tasks.workunit.client.0.smithi008.stderr: 72900K .......... .......... .......... .......... .......... 43% 8.56M 12s 2020-02-05T17:01:12.969 INFO:tasks.workunit.client.0.smithi008.stderr: 72950K .......... .......... .......... .......... .......... 43% 6.64M 12s 2020-02-05T17:01:12.976 INFO:tasks.workunit.client.0.smithi008.stderr: 73000K .......... .......... .......... .......... .......... 43% 6.63M 12s 2020-02-05T17:01:12.985 INFO:tasks.workunit.client.0.smithi008.stderr: 73050K .......... .......... .......... .......... .......... 43% 5.44M 12s 2020-02-05T17:01:12.993 INFO:tasks.workunit.client.0.smithi008.stderr: 73100K .......... .......... .......... .......... .......... 43% 6.40M 12s 2020-02-05T17:01:13.000 INFO:tasks.workunit.client.0.smithi008.stderr: 73150K .......... .......... .......... .......... .......... 43% 6.51M 12s 2020-02-05T17:01:13.011 INFO:tasks.workunit.client.0.smithi008.stderr: 73200K .......... .......... .......... .......... .......... 43% 4.29M 12s 2020-02-05T17:01:13.016 INFO:tasks.workunit.client.0.smithi008.stderr: 73250K .......... .......... .......... .......... .......... 43% 9.79M 12s 2020-02-05T17:01:13.020 INFO:tasks.workunit.client.0.smithi008.stderr: 73300K .......... .......... .......... .......... .......... 43% 13.4M 12s 2020-02-05T17:01:13.023 INFO:tasks.workunit.client.0.smithi008.stderr: 73350K .......... .......... .......... .......... .......... 43% 13.7M 12s 2020-02-05T17:01:13.027 INFO:tasks.workunit.client.0.smithi008.stderr: 73400K .......... .......... .......... .......... .......... 43% 14.5M 12s 2020-02-05T17:01:13.031 INFO:tasks.workunit.client.0.smithi008.stderr: 73450K .......... .......... .......... .......... .......... 43% 11.1M 12s 2020-02-05T17:01:13.035 INFO:tasks.workunit.client.0.smithi008.stderr: 73500K .......... .......... .......... .......... .......... 43% 11.6M 12s 2020-02-05T17:01:13.039 INFO:tasks.workunit.client.0.smithi008.stderr: 73550K .......... .......... .......... .......... .......... 43% 12.2M 12s 2020-02-05T17:01:13.044 INFO:tasks.workunit.client.0.smithi008.stderr: 73600K .......... .......... .......... .......... .......... 43% 11.4M 12s 2020-02-05T17:01:13.049 INFO:tasks.workunit.client.0.smithi008.stderr: 73650K .......... .......... .......... .......... .......... 43% 9.39M 11s 2020-02-05T17:01:13.054 INFO:tasks.workunit.client.0.smithi008.stderr: 73700K .......... .......... .......... .......... .......... 43% 9.86M 11s 2020-02-05T17:01:13.059 INFO:tasks.workunit.client.0.smithi008.stderr: 73750K .......... .......... .......... .......... .......... 43% 9.69M 11s 2020-02-05T17:01:13.065 INFO:tasks.workunit.client.0.smithi008.stderr: 73800K .......... .......... .......... .......... .......... 43% 8.03M 11s 2020-02-05T17:01:13.070 INFO:tasks.workunit.client.0.smithi008.stderr: 73850K .......... .......... .......... .......... .......... 43% 9.72M 11s 2020-02-05T17:01:13.074 INFO:tasks.workunit.client.0.smithi008.stderr: 73900K .......... .......... .......... .......... .......... 43% 13.0M 11s 2020-02-05T17:01:13.078 INFO:tasks.workunit.client.0.smithi008.stderr: 73950K .......... .......... .......... .......... .......... 43% 12.6M 11s 2020-02-05T17:01:13.083 INFO:tasks.workunit.client.0.smithi008.stderr: 74000K .......... .......... .......... .......... .......... 43% 9.97M 11s 2020-02-05T17:01:13.088 INFO:tasks.workunit.client.0.smithi008.stderr: 74050K .......... .......... .......... .......... .......... 43% 8.27M 11s 2020-02-05T17:01:13.093 INFO:tasks.workunit.client.0.smithi008.stderr: 74100K .......... .......... .......... .......... .......... 43% 10.6M 11s 2020-02-05T17:01:13.098 INFO:tasks.workunit.client.0.smithi008.stderr: 74150K .......... .......... .......... .......... .......... 44% 10.6M 11s 2020-02-05T17:01:13.102 INFO:tasks.workunit.client.0.smithi008.stderr: 74200K .......... .......... .......... .......... .......... 44% 10.9M 11s 2020-02-05T17:01:13.108 INFO:tasks.workunit.client.0.smithi008.stderr: 74250K .......... .......... .......... .......... .......... 44% 8.32M 11s 2020-02-05T17:01:13.112 INFO:tasks.workunit.client.0.smithi008.stderr: 74300K .......... .......... .......... .......... .......... 44% 10.8M 11s 2020-02-05T17:01:13.117 INFO:tasks.workunit.client.0.smithi008.stderr: 74350K .......... .......... .......... .......... .......... 44% 11.5M 11s 2020-02-05T17:01:13.121 INFO:tasks.workunit.client.0.smithi008.stderr: 74400K .......... .......... .......... .......... .......... 44% 11.1M 11s 2020-02-05T17:01:13.126 INFO:tasks.workunit.client.0.smithi008.stderr: 74450K .......... .......... .......... .......... .......... 44% 9.40M 11s 2020-02-05T17:01:13.131 INFO:tasks.workunit.client.0.smithi008.stderr: 74500K .......... .......... .......... .......... .......... 44% 10.7M 11s 2020-02-05T17:01:13.137 INFO:tasks.workunit.client.0.smithi008.stderr: 74550K .......... .......... .......... .......... .......... 44% 8.73M 11s 2020-02-05T17:01:13.142 INFO:tasks.workunit.client.0.smithi008.stderr: 74600K .......... .......... .......... .......... .......... 44% 8.90M 11s 2020-02-05T17:01:13.150 INFO:tasks.workunit.client.0.smithi008.stderr: 74650K .......... .......... .......... .......... .......... 44% 6.34M 11s 2020-02-05T17:01:13.156 INFO:tasks.workunit.client.0.smithi008.stderr: 74700K .......... .......... .......... .......... .......... 44% 8.53M 11s 2020-02-05T17:01:13.161 INFO:tasks.workunit.client.0.smithi008.stderr: 74750K .......... .......... .......... .......... .......... 44% 9.12M 11s 2020-02-05T17:01:13.167 INFO:tasks.workunit.client.0.smithi008.stderr: 74800K .......... .......... .......... .......... .......... 44% 8.17M 11s 2020-02-05T17:01:13.174 INFO:tasks.workunit.client.0.smithi008.stderr: 74850K .......... .......... .......... .......... .......... 44% 7.07M 11s 2020-02-05T17:01:13.179 INFO:tasks.workunit.client.0.smithi008.stderr: 74900K .......... .......... .......... .......... .......... 44% 10.0M 11s 2020-02-05T17:01:13.183 INFO:tasks.workunit.client.0.smithi008.stderr: 74950K .......... .......... .......... .......... .......... 44% 10.9M 11s 2020-02-05T17:01:13.187 INFO:tasks.workunit.client.0.smithi008.stderr: 75000K .......... .......... .......... .......... .......... 44% 11.3M 11s 2020-02-05T17:01:13.192 INFO:tasks.workunit.client.0.smithi008.stderr: 75050K .......... .......... .......... .......... .......... 44% 10.3M 11s 2020-02-05T17:01:13.196 INFO:tasks.workunit.client.0.smithi008.stderr: 75100K .......... .......... .......... .......... .......... 44% 11.9M 11s 2020-02-05T17:01:13.201 INFO:tasks.workunit.client.0.smithi008.stderr: 75150K .......... .......... .......... .......... .......... 44% 9.34M 11s 2020-02-05T17:01:13.207 INFO:tasks.workunit.client.0.smithi008.stderr: 75200K .......... .......... .......... .......... .......... 44% 9.66M 11s 2020-02-05T17:01:13.213 INFO:tasks.workunit.client.0.smithi008.stderr: 75250K .......... .......... .......... .......... .......... 44% 8.34M 11s 2020-02-05T17:01:13.218 INFO:tasks.workunit.client.0.smithi008.stderr: 75300K .......... .......... .......... .......... .......... 44% 8.65M 11s 2020-02-05T17:01:13.224 INFO:tasks.workunit.client.0.smithi008.stderr: 75350K .......... .......... .......... .......... .......... 44% 8.79M 11s 2020-02-05T17:01:13.233 INFO:tasks.workunit.client.0.smithi008.stderr: 75400K .......... .......... .......... .......... .......... 44% 7.88M 11s 2020-02-05T17:01:13.240 INFO:tasks.workunit.client.0.smithi008.stderr: 75450K .......... .......... .......... .......... .......... 44% 4.94M 11s 2020-02-05T17:01:13.246 INFO:tasks.workunit.client.0.smithi008.stderr: 75500K .......... .......... .......... .......... .......... 44% 7.77M 11s 2020-02-05T17:01:13.252 INFO:tasks.workunit.client.0.smithi008.stderr: 75550K .......... .......... .......... .......... .......... 44% 8.38M 11s 2020-02-05T17:01:13.259 INFO:tasks.workunit.client.0.smithi008.stderr: 75600K .......... .......... .......... .......... .......... 44% 8.09M 11s 2020-02-05T17:01:13.266 INFO:tasks.workunit.client.0.smithi008.stderr: 75650K .......... .......... .......... .......... .......... 44% 6.51M 11s 2020-02-05T17:01:13.273 INFO:tasks.workunit.client.0.smithi008.stderr: 75700K .......... .......... .......... .......... .......... 44% 6.91M 11s 2020-02-05T17:01:13.278 INFO:tasks.workunit.client.0.smithi008.stderr: 75750K .......... .......... .......... .......... .......... 44% 8.80M 11s 2020-02-05T17:01:13.282 INFO:tasks.workunit.client.0.smithi008.stderr: 75800K .......... .......... .......... .......... .......... 44% 10.8M 11s 2020-02-05T17:01:13.289 INFO:tasks.workunit.client.0.smithi008.stderr: 75850K .......... .......... .......... .......... .......... 45% 8.02M 11s 2020-02-05T17:01:13.294 INFO:tasks.workunit.client.0.smithi008.stderr: 75900K .......... .......... .......... .......... .......... 45% 8.73M 11s 2020-02-05T17:01:13.299 INFO:tasks.workunit.client.0.smithi008.stderr: 75950K .......... .......... .......... .......... .......... 45% 10.7M 11s 2020-02-05T17:01:13.305 INFO:tasks.workunit.client.0.smithi008.stderr: 76000K .......... .......... .......... .......... .......... 45% 8.56M 11s 2020-02-05T17:01:13.310 INFO:tasks.workunit.client.0.smithi008.stderr: 76050K .......... .......... .......... .......... .......... 45% 8.39M 11s 2020-02-05T17:01:13.315 INFO:tasks.workunit.client.0.smithi008.stderr: 76100K .......... .......... .......... .......... .......... 45% 10.1M 11s 2020-02-05T17:01:13.319 INFO:tasks.workunit.client.0.smithi008.stderr: 76150K .......... .......... .......... .......... .......... 45% 10.9M 11s 2020-02-05T17:01:13.324 INFO:tasks.workunit.client.0.smithi008.stderr: 76200K .......... .......... .......... .......... .......... 45% 12.0M 11s 2020-02-05T17:01:13.328 INFO:tasks.workunit.client.0.smithi008.stderr: 76250K .......... .......... .......... .......... .......... 45% 10.3M 11s 2020-02-05T17:01:13.333 INFO:tasks.workunit.client.0.smithi008.stderr: 76300K .......... .......... .......... .......... .......... 45% 10.7M 11s 2020-02-05T17:01:13.337 INFO:tasks.workunit.client.0.smithi008.stderr: 76350K .......... .......... .......... .......... .......... 45% 11.2M 11s 2020-02-05T17:01:13.342 INFO:tasks.workunit.client.0.smithi008.stderr: 76400K .......... .......... .......... .......... .......... 45% 10.2M 11s 2020-02-05T17:01:13.349 INFO:tasks.workunit.client.0.smithi008.stderr: 76450K .......... .......... .......... .......... .......... 45% 7.84M 11s 2020-02-05T17:01:13.354 INFO:tasks.workunit.client.0.smithi008.stderr: 76500K .......... .......... .......... .......... .......... 45% 8.58M 11s 2020-02-05T17:01:13.360 INFO:tasks.workunit.client.0.smithi008.stderr: 76550K .......... .......... .......... .......... .......... 45% 8.37M 11s 2020-02-05T17:01:13.367 INFO:tasks.workunit.client.0.smithi008.stderr: 76600K .......... .......... .......... .......... .......... 45% 7.27M 11s 2020-02-05T17:01:13.374 INFO:tasks.workunit.client.0.smithi008.stderr: 76650K .......... .......... .......... .......... .......... 45% 6.71M 11s 2020-02-05T17:01:13.381 INFO:tasks.workunit.client.0.smithi008.stderr: 76700K .......... .......... .......... .......... .......... 45% 7.29M 11s 2020-02-05T17:01:13.388 INFO:tasks.workunit.client.0.smithi008.stderr: 76750K .......... .......... .......... .......... .......... 45% 7.38M 11s 2020-02-05T17:01:13.396 INFO:tasks.workunit.client.0.smithi008.stderr: 76800K .......... .......... .......... .......... .......... 45% 6.32M 11s 2020-02-05T17:01:13.406 INFO:tasks.workunit.client.0.smithi008.stderr: 76850K .......... .......... .......... .......... .......... 45% 4.88M 11s 2020-02-05T17:01:13.415 INFO:tasks.workunit.client.0.smithi008.stderr: 76900K .......... .......... .......... .......... .......... 45% 5.42M 11s 2020-02-05T17:01:13.424 INFO:tasks.workunit.client.0.smithi008.stderr: 76950K .......... .......... .......... .......... .......... 45% 5.51M 11s 2020-02-05T17:01:13.432 INFO:tasks.workunit.client.0.smithi008.stderr: 77000K .......... .......... .......... .......... .......... 45% 5.39M 11s 2020-02-05T17:01:13.443 INFO:tasks.workunit.client.0.smithi008.stderr: 77050K .......... .......... .......... .......... .......... 45% 4.85M 11s 2020-02-05T17:01:13.450 INFO:tasks.workunit.client.0.smithi008.stderr: 77100K .......... .......... .......... .......... .......... 45% 6.01M 11s 2020-02-05T17:01:13.458 INFO:tasks.workunit.client.0.smithi008.stderr: 77150K .......... .......... .......... .......... .......... 45% 6.46M 11s 2020-02-05T17:01:13.465 INFO:tasks.workunit.client.0.smithi008.stderr: 77200K .......... .......... .......... .......... .......... 45% 7.33M 11s 2020-02-05T17:01:13.473 INFO:tasks.workunit.client.0.smithi008.stderr: 77250K .......... .......... .......... .......... .......... 45% 5.64M 11s 2020-02-05T17:01:13.486 INFO:tasks.workunit.client.0.smithi008.stderr: 77300K .......... .......... .......... .......... .......... 45% 3.88M 11s 2020-02-05T17:01:13.493 INFO:tasks.workunit.client.0.smithi008.stderr: 77350K .......... .......... .......... .......... .......... 45% 7.01M 11s 2020-02-05T17:01:13.499 INFO:tasks.workunit.client.0.smithi008.stderr: 77400K .......... .......... .......... .......... .......... 45% 7.58M 11s 2020-02-05T17:01:13.507 INFO:tasks.workunit.client.0.smithi008.stderr: 77450K .......... .......... .......... .......... .......... 45% 6.34M 11s 2020-02-05T17:01:13.513 INFO:tasks.workunit.client.0.smithi008.stderr: 77500K .......... .......... .......... .......... .......... 46% 7.99M 11s 2020-02-05T17:01:13.519 INFO:tasks.workunit.client.0.smithi008.stderr: 77550K .......... .......... .......... .......... .......... 46% 8.50M 11s 2020-02-05T17:01:13.524 INFO:tasks.workunit.client.0.smithi008.stderr: 77600K .......... .......... .......... .......... .......... 46% 8.41M 11s 2020-02-05T17:01:13.530 INFO:tasks.workunit.client.0.smithi008.stderr: 77650K .......... .......... .......... .......... .......... 46% 8.16M 11s 2020-02-05T17:01:13.535 INFO:tasks.workunit.client.0.smithi008.stderr: 77700K .......... .......... .......... .......... .......... 46% 9.73M 11s 2020-02-05T17:01:13.541 INFO:tasks.workunit.client.0.smithi008.stderr: 77750K .......... .......... .......... .......... .......... 46% 9.03M 11s 2020-02-05T17:01:13.547 INFO:tasks.workunit.client.0.smithi008.stderr: 77800K .......... .......... .......... .......... .......... 46% 8.01M 11s 2020-02-05T17:01:13.553 INFO:tasks.workunit.client.0.smithi008.stderr: 77850K .......... .......... .......... .......... .......... 46% 8.39M 11s 2020-02-05T17:01:13.558 INFO:tasks.workunit.client.0.smithi008.stderr: 77900K .......... .......... .......... .......... .......... 46% 9.24M 11s 2020-02-05T17:01:13.563 INFO:tasks.workunit.client.0.smithi008.stderr: 77950K .......... .......... .......... .......... .......... 46% 9.27M 11s 2020-02-05T17:01:13.568 INFO:tasks.workunit.client.0.smithi008.stderr: 78000K .......... .......... .......... .......... .......... 46% 10.5M 11s 2020-02-05T17:01:13.573 INFO:tasks.workunit.client.0.smithi008.stderr: 78050K .......... .......... .......... .......... .......... 46% 9.41M 11s 2020-02-05T17:01:13.577 INFO:tasks.workunit.client.0.smithi008.stderr: 78100K .......... .......... .......... .......... .......... 46% 12.2M 11s 2020-02-05T17:01:13.582 INFO:tasks.workunit.client.0.smithi008.stderr: 78150K .......... .......... .......... .......... .......... 46% 10.6M 11s 2020-02-05T17:01:13.587 INFO:tasks.workunit.client.0.smithi008.stderr: 78200K .......... .......... .......... .......... .......... 46% 9.47M 11s 2020-02-05T17:01:13.594 INFO:tasks.workunit.client.0.smithi008.stderr: 78250K .......... .......... .......... .......... .......... 46% 7.17M 11s 2020-02-05T17:01:13.600 INFO:tasks.workunit.client.0.smithi008.stderr: 78300K .......... .......... .......... .......... .......... 46% 7.81M 11s 2020-02-05T17:01:13.607 INFO:tasks.workunit.client.0.smithi008.stderr: 78350K .......... .......... .......... .......... .......... 46% 7.35M 11s 2020-02-05T17:01:13.614 INFO:tasks.workunit.client.0.smithi008.stderr: 78400K .......... .......... .......... .......... .......... 46% 7.13M 11s 2020-02-05T17:01:13.621 INFO:tasks.workunit.client.0.smithi008.stderr: 78450K .......... .......... .......... .......... .......... 46% 6.64M 11s 2020-02-05T17:01:13.626 INFO:tasks.workunit.client.0.smithi008.stderr: 78500K .......... .......... .......... .......... .......... 46% 8.61M 11s 2020-02-05T17:01:13.631 INFO:tasks.workunit.client.0.smithi008.stderr: 78550K .......... .......... .......... .......... .......... 46% 9.77M 11s 2020-02-05T17:01:13.636 INFO:tasks.workunit.client.0.smithi008.stderr: 78600K .......... .......... .......... .......... .......... 46% 9.92M 11s 2020-02-05T17:01:13.643 INFO:tasks.workunit.client.0.smithi008.stderr: 78650K .......... .......... .......... .......... .......... 46% 7.45M 11s 2020-02-05T17:01:13.648 INFO:tasks.workunit.client.0.smithi008.stderr: 78700K .......... .......... .......... .......... .......... 46% 10.0M 11s 2020-02-05T17:01:13.653 INFO:tasks.workunit.client.0.smithi008.stderr: 78750K .......... .......... .......... .......... .......... 46% 9.86M 11s 2020-02-05T17:01:13.657 INFO:tasks.workunit.client.0.smithi008.stderr: 78800K .......... .......... .......... .......... .......... 46% 10.2M 11s 2020-02-05T17:01:13.662 INFO:tasks.workunit.client.0.smithi008.stderr: 78850K .......... .......... .......... .......... .......... 46% 11.3M 11s 2020-02-05T17:01:13.665 INFO:tasks.workunit.client.0.smithi008.stderr: 78900K .......... .......... .......... .......... .......... 46% 13.4M 11s 2020-02-05T17:01:13.670 INFO:tasks.workunit.client.0.smithi008.stderr: 78950K .......... .......... .......... .......... .......... 46% 11.7M 11s 2020-02-05T17:01:13.675 INFO:tasks.workunit.client.0.smithi008.stderr: 79000K .......... .......... .......... .......... .......... 46% 10.4M 11s 2020-02-05T17:01:13.680 INFO:tasks.workunit.client.0.smithi008.stderr: 79050K .......... .......... .......... .......... .......... 46% 8.09M 11s 2020-02-05T17:01:13.685 INFO:tasks.workunit.client.0.smithi008.stderr: 79100K .......... .......... .......... .......... .......... 46% 10.3M 11s 2020-02-05T17:01:13.690 INFO:tasks.workunit.client.0.smithi008.stderr: 79150K .......... .......... .......... .......... .......... 46% 10.0M 11s 2020-02-05T17:01:13.696 INFO:tasks.workunit.client.0.smithi008.stderr: 79200K .......... .......... .......... .......... .......... 47% 8.98M 11s 2020-02-05T17:01:13.702 INFO:tasks.workunit.client.0.smithi008.stderr: 79250K .......... .......... .......... .......... .......... 47% 7.73M 11s 2020-02-05T17:01:13.707 INFO:tasks.workunit.client.0.smithi008.stderr: 79300K .......... .......... .......... .......... .......... 47% 8.56M 11s 2020-02-05T17:01:13.713 INFO:tasks.workunit.client.0.smithi008.stderr: 79350K .......... .......... .......... .......... .......... 47% 9.13M 11s 2020-02-05T17:01:13.718 INFO:tasks.workunit.client.0.smithi008.stderr: 79400K .......... .......... .......... .......... .......... 47% 8.80M 11s 2020-02-05T17:01:13.726 INFO:tasks.workunit.client.0.smithi008.stderr: 79450K .......... .......... .......... .......... .......... 47% 6.85M 11s 2020-02-05T17:01:13.736 INFO:tasks.workunit.client.0.smithi008.stderr: 79500K .......... .......... .......... .......... .......... 47% 7.15M 11s 2020-02-05T17:01:13.742 INFO:tasks.workunit.client.0.smithi008.stderr: 79550K .......... .......... .......... .......... .......... 47% 5.00M 11s 2020-02-05T17:01:13.748 INFO:tasks.workunit.client.0.smithi008.stderr: 79600K .......... .......... .......... .......... .......... 47% 8.54M 11s 2020-02-05T17:01:13.755 INFO:tasks.workunit.client.0.smithi008.stderr: 79650K .......... .......... .......... .......... .......... 47% 7.87M 11s 2020-02-05T17:01:13.760 INFO:tasks.workunit.client.0.smithi008.stderr: 79700K .......... .......... .......... .......... .......... 47% 8.76M 11s 2020-02-05T17:01:13.766 INFO:tasks.workunit.client.0.smithi008.stderr: 79750K .......... .......... .......... .......... .......... 47% 8.39M 11s 2020-02-05T17:01:13.772 INFO:tasks.workunit.client.0.smithi008.stderr: 79800K .......... .......... .......... .......... .......... 47% 7.41M 11s 2020-02-05T17:01:13.779 INFO:tasks.workunit.client.0.smithi008.stderr: 79850K .......... .......... .......... .......... .......... 47% 6.75M 11s 2020-02-05T17:01:13.786 INFO:tasks.workunit.client.0.smithi008.stderr: 79900K .......... .......... .......... .......... .......... 47% 7.46M 11s 2020-02-05T17:01:13.793 INFO:tasks.workunit.client.0.smithi008.stderr: 79950K .......... .......... .......... .......... .......... 47% 7.03M 11s 2020-02-05T17:01:13.799 INFO:tasks.workunit.client.0.smithi008.stderr: 80000K .......... .......... .......... .......... .......... 47% 7.59M 11s 2020-02-05T17:01:13.808 INFO:tasks.workunit.client.0.smithi008.stderr: 80050K .......... .......... .......... .......... .......... 47% 5.92M 11s 2020-02-05T17:01:13.826 INFO:tasks.workunit.client.0.smithi008.stderr: 80100K .......... .......... .......... .......... .......... 47% 7.05M 11s 2020-02-05T17:01:13.826 INFO:tasks.workunit.client.0.smithi008.stderr: 80150K .......... .......... .......... .......... .......... 47% 6.99M 11s 2020-02-05T17:01:13.828 INFO:tasks.workunit.client.0.smithi008.stderr: 80200K .......... .......... .......... .......... .......... 47% 7.15M 11s 2020-02-05T17:01:13.836 INFO:tasks.workunit.client.0.smithi008.stderr: 80250K .......... .......... .......... .......... .......... 47% 6.10M 11s 2020-02-05T17:01:13.844 INFO:tasks.workunit.client.0.smithi008.stderr: 80300K .......... .......... .......... .......... .......... 47% 6.46M 11s 2020-02-05T17:01:13.851 INFO:tasks.workunit.client.0.smithi008.stderr: 80350K .......... .......... .......... .......... .......... 47% 6.93M 11s 2020-02-05T17:01:13.859 INFO:tasks.workunit.client.0.smithi008.stderr: 80400K .......... .......... .......... .......... .......... 47% 6.38M 11s 2020-02-05T17:01:13.867 INFO:tasks.workunit.client.0.smithi008.stderr: 80450K .......... .......... .......... .......... .......... 47% 5.59M 11s 2020-02-05T17:01:13.874 INFO:tasks.workunit.client.0.smithi008.stderr: 80500K .......... .......... .......... .......... .......... 47% 7.06M 11s 2020-02-05T17:01:13.882 INFO:tasks.workunit.client.0.smithi008.stderr: 80550K .......... .......... .......... .......... .......... 47% 6.60M 11s 2020-02-05T17:01:13.889 INFO:tasks.workunit.client.0.smithi008.stderr: 80600K .......... .......... .......... .......... .......... 47% 6.62M 11s 2020-02-05T17:01:13.898 INFO:tasks.workunit.client.0.smithi008.stderr: 80650K .......... .......... .......... .......... .......... 47% 5.47M 11s 2020-02-05T17:01:13.907 INFO:tasks.workunit.client.0.smithi008.stderr: 80700K .......... .......... .......... .......... .......... 47% 6.14M 11s 2020-02-05T17:01:13.914 INFO:tasks.workunit.client.0.smithi008.stderr: 80750K .......... .......... .......... .......... .......... 47% 6.33M 11s 2020-02-05T17:01:13.920 INFO:tasks.workunit.client.0.smithi008.stderr: 80800K .......... .......... .......... .......... .......... 47% 7.35M 11s 2020-02-05T17:01:13.928 INFO:tasks.workunit.client.0.smithi008.stderr: 80850K .......... .......... .......... .......... .......... 47% 6.03M 11s 2020-02-05T17:01:13.937 INFO:tasks.workunit.client.0.smithi008.stderr: 80900K .......... .......... .......... .......... .......... 48% 6.57M 11s 2020-02-05T17:01:13.943 INFO:tasks.workunit.client.0.smithi008.stderr: 80950K .......... .......... .......... .......... .......... 48% 6.77M 11s 2020-02-05T17:01:13.951 INFO:tasks.workunit.client.0.smithi008.stderr: 81000K .......... .......... .......... .......... .......... 48% 5.96M 11s 2020-02-05T17:01:13.962 INFO:tasks.workunit.client.0.smithi008.stderr: 81050K .......... .......... .......... .......... .......... 48% 4.92M 11s 2020-02-05T17:01:13.970 INFO:tasks.workunit.client.0.smithi008.stderr: 81100K .......... .......... .......... .......... .......... 48% 5.84M 11s 2020-02-05T17:01:13.977 INFO:tasks.workunit.client.0.smithi008.stderr: 81150K .......... .......... .......... .......... .......... 48% 6.32M 11s 2020-02-05T17:01:13.986 INFO:tasks.workunit.client.0.smithi008.stderr: 81200K .......... .......... .......... .......... .......... 48% 5.69M 11s 2020-02-05T17:01:13.996 INFO:tasks.workunit.client.0.smithi008.stderr: 81250K .......... .......... .......... .......... .......... 48% 5.14M 11s 2020-02-05T17:01:14.004 INFO:tasks.workunit.client.0.smithi008.stderr: 81300K .......... .......... .......... .......... .......... 48% 5.95M 11s 2020-02-05T17:01:14.013 INFO:tasks.workunit.client.0.smithi008.stderr: 81350K .......... .......... .......... .......... .......... 48% 5.37M 11s 2020-02-05T17:01:14.027 INFO:tasks.workunit.client.0.smithi008.stderr: 81400K .......... .......... .......... .......... .......... 48% 3.43M 11s 2020-02-05T17:01:14.034 INFO:tasks.workunit.client.0.smithi008.stderr: 81450K .......... .......... .......... .......... .......... 48% 6.50M 11s 2020-02-05T17:01:14.041 INFO:tasks.workunit.client.0.smithi008.stderr: 81500K .......... .......... .......... .......... .......... 48% 8.93M 11s 2020-02-05T17:01:14.046 INFO:tasks.workunit.client.0.smithi008.stderr: 81550K .......... .......... .......... .......... .......... 48% 7.86M 11s 2020-02-05T17:01:14.053 INFO:tasks.workunit.client.0.smithi008.stderr: 81600K .......... .......... .......... .......... .......... 48% 7.40M 11s 2020-02-05T17:01:14.059 INFO:tasks.workunit.client.0.smithi008.stderr: 81650K .......... .......... .......... .......... .......... 48% 7.16M 11s 2020-02-05T17:01:14.063 INFO:tasks.workunit.client.0.smithi008.stderr: 81700K .......... .......... .......... .......... .......... 48% 12.4M 11s 2020-02-05T17:01:14.067 INFO:tasks.workunit.client.0.smithi008.stderr: 81750K .......... .......... .......... .......... .......... 48% 12.8M 11s 2020-02-05T17:01:14.071 INFO:tasks.workunit.client.0.smithi008.stderr: 81800K .......... .......... .......... .......... .......... 48% 12.8M 11s 2020-02-05T17:01:14.076 INFO:tasks.workunit.client.0.smithi008.stderr: 81850K .......... .......... .......... .......... .......... 48% 10.8M 11s 2020-02-05T17:01:14.081 INFO:tasks.workunit.client.0.smithi008.stderr: 81900K .......... .......... .......... .......... .......... 48% 8.72M 11s 2020-02-05T17:01:14.087 INFO:tasks.workunit.client.0.smithi008.stderr: 81950K .......... .......... .......... .......... .......... 48% 8.46M 11s 2020-02-05T17:01:14.093 INFO:tasks.workunit.client.0.smithi008.stderr: 82000K .......... .......... .......... .......... .......... 48% 8.21M 11s 2020-02-05T17:01:14.098 INFO:tasks.workunit.client.0.smithi008.stderr: 82050K .......... .......... .......... .......... .......... 48% 9.16M 11s 2020-02-05T17:01:14.101 INFO:tasks.workunit.client.0.smithi008.stderr: 82100K .......... .......... .......... .......... .......... 48% 13.2M 10s 2020-02-05T17:01:14.104 INFO:tasks.workunit.client.0.smithi008.stderr: 82150K .......... .......... .......... .......... .......... 48% 18.3M 10s 2020-02-05T17:01:14.106 INFO:tasks.workunit.client.0.smithi008.stderr: 82200K .......... .......... .......... .......... .......... 48% 18.9M 10s 2020-02-05T17:01:14.112 INFO:tasks.workunit.client.0.smithi008.stderr: 82250K .......... .......... .......... .......... .......... 48% 13.3M 10s 2020-02-05T17:01:14.114 INFO:tasks.workunit.client.0.smithi008.stderr: 82300K .......... .......... .......... .......... .......... 48% 12.8M 10s 2020-02-05T17:01:14.117 INFO:tasks.workunit.client.0.smithi008.stderr: 82350K .......... .......... .......... .......... .......... 48% 15.0M 10s 2020-02-05T17:01:14.121 INFO:tasks.workunit.client.0.smithi008.stderr: 82400K .......... .......... .......... .......... .......... 48% 13.4M 10s 2020-02-05T17:01:14.126 INFO:tasks.workunit.client.0.smithi008.stderr: 82450K .......... .......... .......... .......... .......... 48% 10.2M 10s 2020-02-05T17:01:14.130 INFO:tasks.workunit.client.0.smithi008.stderr: 82500K .......... .......... .......... .......... .......... 48% 11.2M 10s 2020-02-05T17:01:14.136 INFO:tasks.workunit.client.0.smithi008.stderr: 82550K .......... .......... .......... .......... .......... 49% 9.17M 10s 2020-02-05T17:01:14.142 INFO:tasks.workunit.client.0.smithi008.stderr: 82600K .......... .......... .......... .......... .......... 49% 8.43M 10s 2020-02-05T17:01:14.150 INFO:tasks.workunit.client.0.smithi008.stderr: 82650K .......... .......... .......... .......... .......... 49% 6.33M 10s 2020-02-05T17:01:14.158 INFO:tasks.workunit.client.0.smithi008.stderr: 82700K .......... .......... .......... .......... .......... 49% 6.94M 10s 2020-02-05T17:01:14.165 INFO:tasks.workunit.client.0.smithi008.stderr: 82750K .......... .......... .......... .......... .......... 49% 6.01M 10s 2020-02-05T17:01:14.172 INFO:tasks.workunit.client.0.smithi008.stderr: 82800K .......... .......... .......... .......... .......... 49% 6.52M 10s 2020-02-05T17:01:14.178 INFO:tasks.workunit.client.0.smithi008.stderr: 82850K .......... .......... .......... .......... .......... 49% 7.25M 10s 2020-02-05T17:01:14.185 INFO:tasks.workunit.client.0.smithi008.stderr: 82900K .......... .......... .......... .......... .......... 49% 15.6M 10s 2020-02-05T17:01:14.185 INFO:tasks.workunit.client.0.smithi008.stderr: 82950K .......... .......... .......... .......... .......... 49% 16.1M 10s 2020-02-05T17:01:14.192 INFO:tasks.workunit.client.0.smithi008.stderr: 83000K .......... .......... .......... .......... .......... 49% 13.7M 10s 2020-02-05T17:01:14.193 INFO:tasks.workunit.client.0.smithi008.stderr: 83050K .......... .......... .......... .......... .......... 49% 10.1M 10s 2020-02-05T17:01:14.199 INFO:tasks.workunit.client.0.smithi008.stderr: 83100K .......... .......... .......... .......... .......... 49% 12.5M 10s 2020-02-05T17:01:14.201 INFO:tasks.workunit.client.0.smithi008.stderr: 83150K .......... .......... .......... .......... .......... 49% 11.7M 10s 2020-02-05T17:01:14.205 INFO:tasks.workunit.client.0.smithi008.stderr: 83200K .......... .......... .......... .......... .......... 49% 12.6M 10s 2020-02-05T17:01:14.210 INFO:tasks.workunit.client.0.smithi008.stderr: 83250K .......... .......... .......... .......... .......... 49% 9.23M 10s 2020-02-05T17:01:14.217 INFO:tasks.workunit.client.0.smithi008.stderr: 83300K .......... .......... .......... .......... .......... 49% 9.46M 10s 2020-02-05T17:01:14.221 INFO:tasks.workunit.client.0.smithi008.stderr: 83350K .......... .......... .......... .......... .......... 49% 9.36M 10s 2020-02-05T17:01:14.226 INFO:tasks.workunit.client.0.smithi008.stderr: 83400K .......... .......... .......... .......... .......... 49% 9.39M 10s 2020-02-05T17:01:14.231 INFO:tasks.workunit.client.0.smithi008.stderr: 83450K .......... .......... .......... .......... .......... 49% 9.10M 10s 2020-02-05T17:01:14.236 INFO:tasks.workunit.client.0.smithi008.stderr: 83500K .......... .......... .......... .......... .......... 49% 9.91M 10s 2020-02-05T17:01:14.242 INFO:tasks.workunit.client.0.smithi008.stderr: 83550K .......... .......... .......... .......... .......... 49% 9.04M 10s 2020-02-05T17:01:14.251 INFO:tasks.workunit.client.0.smithi008.stderr: 83600K .......... .......... .......... .......... .......... 49% 8.42M 10s 2020-02-05T17:01:14.256 INFO:tasks.workunit.client.0.smithi008.stderr: 83650K .......... .......... .......... .......... .......... 49% 5.62M 10s 2020-02-05T17:01:14.261 INFO:tasks.workunit.client.0.smithi008.stderr: 83700K .......... .......... .......... .......... .......... 49% 10.6M 10s 2020-02-05T17:01:14.265 INFO:tasks.workunit.client.0.smithi008.stderr: 83750K .......... .......... .......... .......... .......... 49% 11.5M 10s 2020-02-05T17:01:14.269 INFO:tasks.workunit.client.0.smithi008.stderr: 83800K .......... .......... .......... .......... .......... 49% 11.1M 10s 2020-02-05T17:01:14.275 INFO:tasks.workunit.client.0.smithi008.stderr: 83850K .......... .......... .......... .......... .......... 49% 8.35M 10s 2020-02-05T17:01:14.280 INFO:tasks.workunit.client.0.smithi008.stderr: 83900K .......... .......... .......... .......... .......... 49% 10.8M 10s 2020-02-05T17:01:14.287 INFO:tasks.workunit.client.0.smithi008.stderr: 83950K .......... .......... .......... .......... .......... 49% 10.3M 10s 2020-02-05T17:01:14.290 INFO:tasks.workunit.client.0.smithi008.stderr: 84000K .......... .......... .......... .......... .......... 49% 9.59M 10s 2020-02-05T17:01:14.296 INFO:tasks.workunit.client.0.smithi008.stderr: 84050K .......... .......... .......... .......... .......... 49% 7.66M 10s 2020-02-05T17:01:14.304 INFO:tasks.workunit.client.0.smithi008.stderr: 84100K .......... .......... .......... .......... .......... 49% 7.98M 10s 2020-02-05T17:01:14.309 INFO:tasks.workunit.client.0.smithi008.stderr: 84150K .......... .......... .......... .......... .......... 49% 7.05M 10s 2020-02-05T17:01:14.316 INFO:tasks.workunit.client.0.smithi008.stderr: 84200K .......... .......... .......... .......... .......... 49% 8.04M 10s 2020-02-05T17:01:14.324 INFO:tasks.workunit.client.0.smithi008.stderr: 84250K .......... .......... .......... .......... .......... 50% 5.58M 10s 2020-02-05T17:01:14.332 INFO:tasks.workunit.client.0.smithi008.stderr: 84300K .......... .......... .......... .......... .......... 50% 6.66M 10s 2020-02-05T17:01:14.340 INFO:tasks.workunit.client.0.smithi008.stderr: 84350K .......... .......... .......... .......... .......... 50% 6.05M 10s 2020-02-05T17:01:14.346 INFO:tasks.workunit.client.0.smithi008.stderr: 84400K .......... .......... .......... .......... .......... 50% 7.87M 10s 2020-02-05T17:01:14.353 INFO:tasks.workunit.client.0.smithi008.stderr: 84450K .......... .......... .......... .......... .......... 50% 7.01M 10s 2020-02-05T17:01:14.359 INFO:tasks.workunit.client.0.smithi008.stderr: 84500K .......... .......... .......... .......... .......... 50% 8.12M 10s 2020-02-05T17:01:14.365 INFO:tasks.workunit.client.0.smithi008.stderr: 84550K .......... .......... .......... .......... .......... 50% 8.27M 10s 2020-02-05T17:01:14.372 INFO:tasks.workunit.client.0.smithi008.stderr: 84600K .......... .......... .......... .......... .......... 50% 7.20M 10s 2020-02-05T17:01:14.379 INFO:tasks.workunit.client.0.smithi008.stderr: 84650K .......... .......... .......... .......... .......... 50% 6.14M 10s 2020-02-05T17:01:14.386 INFO:tasks.workunit.client.0.smithi008.stderr: 84700K .......... .......... .......... .......... .......... 50% 7.66M 10s 2020-02-05T17:01:14.392 INFO:tasks.workunit.client.0.smithi008.stderr: 84750K .......... .......... .......... .......... .......... 50% 7.75M 10s 2020-02-05T17:01:14.399 INFO:tasks.workunit.client.0.smithi008.stderr: 84800K .......... .......... .......... .......... .......... 50% 7.32M 10s 2020-02-05T17:01:14.407 INFO:tasks.workunit.client.0.smithi008.stderr: 84850K .......... .......... .......... .......... .......... 50% 6.00M 10s 2020-02-05T17:01:14.414 INFO:tasks.workunit.client.0.smithi008.stderr: 84900K .......... .......... .......... .......... .......... 50% 7.25M 10s 2020-02-05T17:01:14.420 INFO:tasks.workunit.client.0.smithi008.stderr: 84950K .......... .......... .......... .......... .......... 50% 7.55M 10s 2020-02-05T17:01:14.427 INFO:tasks.workunit.client.0.smithi008.stderr: 85000K .......... .......... .......... .......... .......... 50% 6.97M 10s 2020-02-05T17:01:14.437 INFO:tasks.workunit.client.0.smithi008.stderr: 85050K .......... .......... .......... .......... .......... 50% 5.02M 10s 2020-02-05T17:01:14.445 INFO:tasks.workunit.client.0.smithi008.stderr: 85100K .......... .......... .......... .......... .......... 50% 6.39M 10s 2020-02-05T17:01:14.453 INFO:tasks.workunit.client.0.smithi008.stderr: 85150K .......... .......... .......... .......... .......... 50% 5.61M 10s 2020-02-05T17:01:14.462 INFO:tasks.workunit.client.0.smithi008.stderr: 85200K .......... .......... .......... .......... .......... 50% 5.54M 10s 2020-02-05T17:01:14.472 INFO:tasks.workunit.client.0.smithi008.stderr: 85250K .......... .......... .......... .......... .......... 50% 4.85M 10s 2020-02-05T17:01:14.481 INFO:tasks.workunit.client.0.smithi008.stderr: 85300K .......... .......... .......... .......... .......... 50% 6.03M 10s 2020-02-05T17:01:14.488 INFO:tasks.workunit.client.0.smithi008.stderr: 85350K .......... .......... .......... .......... .......... 50% 6.25M 10s 2020-02-05T17:01:14.494 INFO:tasks.workunit.client.0.smithi008.stderr: 85400K .......... .......... .......... .......... .......... 50% 7.95M 10s 2020-02-05T17:01:14.502 INFO:tasks.workunit.client.0.smithi008.stderr: 85450K .......... .......... .......... .......... .......... 50% 6.44M 10s 2020-02-05T17:01:14.511 INFO:tasks.workunit.client.0.smithi008.stderr: 85500K .......... .......... .......... .......... .......... 50% 5.22M 10s 2020-02-05T17:01:14.516 INFO:tasks.workunit.client.0.smithi008.stderr: 85550K .......... .......... .......... .......... .......... 50% 11.1M 10s 2020-02-05T17:01:14.520 INFO:tasks.workunit.client.0.smithi008.stderr: 85600K .......... .......... .......... .......... .......... 50% 10.4M 10s 2020-02-05T17:01:14.525 INFO:tasks.workunit.client.0.smithi008.stderr: 85650K .......... .......... .......... .......... .......... 50% 9.41M 10s 2020-02-05T17:01:14.530 INFO:tasks.workunit.client.0.smithi008.stderr: 85700K .......... .......... .......... .......... .......... 50% 11.0M 10s 2020-02-05T17:01:14.535 INFO:tasks.workunit.client.0.smithi008.stderr: 85750K .......... .......... .......... .......... .......... 50% 9.38M 10s 2020-02-05T17:01:14.541 INFO:tasks.workunit.client.0.smithi008.stderr: 85800K .......... .......... .......... .......... .......... 50% 7.89M 10s 2020-02-05T17:01:14.548 INFO:tasks.workunit.client.0.smithi008.stderr: 85850K .......... .......... .......... .......... .......... 50% 7.08M 10s 2020-02-05T17:01:14.555 INFO:tasks.workunit.client.0.smithi008.stderr: 85900K .......... .......... .......... .......... .......... 50% 7.16M 10s 2020-02-05T17:01:14.562 INFO:tasks.workunit.client.0.smithi008.stderr: 85950K .......... .......... .......... .......... .......... 51% 7.11M 10s 2020-02-05T17:01:14.569 INFO:tasks.workunit.client.0.smithi008.stderr: 86000K .......... .......... .......... .......... .......... 51% 7.18M 10s 2020-02-05T17:01:14.575 INFO:tasks.workunit.client.0.smithi008.stderr: 86050K .......... .......... .......... .......... .......... 51% 7.04M 10s 2020-02-05T17:01:14.579 INFO:tasks.workunit.client.0.smithi008.stderr: 86100K .......... .......... .......... .......... .......... 51% 11.6M 10s 2020-02-05T17:01:14.582 INFO:tasks.workunit.client.0.smithi008.stderr: 86150K .......... .......... .......... .......... .......... 51% 14.0M 10s 2020-02-05T17:01:14.585 INFO:tasks.workunit.client.0.smithi008.stderr: 86200K .......... .......... .......... .......... .......... 51% 16.6M 10s 2020-02-05T17:01:14.589 INFO:tasks.workunit.client.0.smithi008.stderr: 86250K .......... .......... .......... .......... .......... 51% 13.9M 10s 2020-02-05T17:01:14.592 INFO:tasks.workunit.client.0.smithi008.stderr: 86300K .......... .......... .......... .......... .......... 51% 16.6M 10s 2020-02-05T17:01:14.595 INFO:tasks.workunit.client.0.smithi008.stderr: 86350K .......... .......... .......... .......... .......... 51% 16.0M 10s 2020-02-05T17:01:14.598 INFO:tasks.workunit.client.0.smithi008.stderr: 86400K .......... .......... .......... .......... .......... 51% 14.4M 10s 2020-02-05T17:01:14.604 INFO:tasks.workunit.client.0.smithi008.stderr: 86450K .......... .......... .......... .......... .......... 51% 12.9M 10s 2020-02-05T17:01:14.606 INFO:tasks.workunit.client.0.smithi008.stderr: 86500K .......... .......... .......... .......... .......... 51% 12.7M 10s 2020-02-05T17:01:14.611 INFO:tasks.workunit.client.0.smithi008.stderr: 86550K .......... .......... .......... .......... .......... 51% 11.3M 10s 2020-02-05T17:01:14.616 INFO:tasks.workunit.client.0.smithi008.stderr: 86600K .......... .......... .......... .......... .......... 51% 9.71M 10s 2020-02-05T17:01:14.623 INFO:tasks.workunit.client.0.smithi008.stderr: 86650K .......... .......... .......... .......... .......... 51% 7.45M 10s 2020-02-05T17:01:14.629 INFO:tasks.workunit.client.0.smithi008.stderr: 86700K .......... .......... .......... .......... .......... 51% 8.07M 10s 2020-02-05T17:01:14.634 INFO:tasks.workunit.client.0.smithi008.stderr: 86750K .......... .......... .......... .......... .......... 51% 8.59M 10s 2020-02-05T17:01:14.641 INFO:tasks.workunit.client.0.smithi008.stderr: 86800K .......... .......... .......... .......... .......... 51% 8.86M 10s 2020-02-05T17:01:14.648 INFO:tasks.workunit.client.0.smithi008.stderr: 86850K .......... .......... .......... .......... .......... 51% 5.96M 10s 2020-02-05T17:01:14.655 INFO:tasks.workunit.client.0.smithi008.stderr: 86900K .......... .......... .......... .......... .......... 51% 7.38M 10s 2020-02-05T17:01:14.661 INFO:tasks.workunit.client.0.smithi008.stderr: 86950K .......... .......... .......... .......... .......... 51% 7.41M 10s 2020-02-05T17:01:14.667 INFO:tasks.workunit.client.0.smithi008.stderr: 87000K .......... .......... .......... .......... .......... 51% 8.09M 10s 2020-02-05T17:01:14.675 INFO:tasks.workunit.client.0.smithi008.stderr: 87050K .......... .......... .......... .......... .......... 51% 6.83M 10s 2020-02-05T17:01:14.680 INFO:tasks.workunit.client.0.smithi008.stderr: 87100K .......... .......... .......... .......... .......... 51% 8.20M 10s 2020-02-05T17:01:14.686 INFO:tasks.workunit.client.0.smithi008.stderr: 87150K .......... .......... .......... .......... .......... 51% 8.63M 10s 2020-02-05T17:01:14.692 INFO:tasks.workunit.client.0.smithi008.stderr: 87200K .......... .......... .......... .......... .......... 51% 8.43M 10s 2020-02-05T17:01:14.698 INFO:tasks.workunit.client.0.smithi008.stderr: 87250K .......... .......... .......... .......... .......... 51% 7.24M 10s 2020-02-05T17:01:14.704 INFO:tasks.workunit.client.0.smithi008.stderr: 87300K .......... .......... .......... .......... .......... 51% 8.38M 10s 2020-02-05T17:01:14.711 INFO:tasks.workunit.client.0.smithi008.stderr: 87350K .......... .......... .......... .......... .......... 51% 8.08M 10s 2020-02-05T17:01:14.717 INFO:tasks.workunit.client.0.smithi008.stderr: 87400K .......... .......... .......... .......... .......... 51% 7.35M 10s 2020-02-05T17:01:14.724 INFO:tasks.workunit.client.0.smithi008.stderr: 87450K .......... .......... .......... .......... .......... 51% 7.01M 10s 2020-02-05T17:01:14.730 INFO:tasks.workunit.client.0.smithi008.stderr: 87500K .......... .......... .......... .......... .......... 51% 8.41M 10s 2020-02-05T17:01:14.736 INFO:tasks.workunit.client.0.smithi008.stderr: 87550K .......... .......... .......... .......... .......... 51% 8.96M 10s 2020-02-05T17:01:14.740 INFO:tasks.workunit.client.0.smithi008.stderr: 87600K .......... .......... .......... .......... .......... 51% 8.90M 10s 2020-02-05T17:01:14.746 INFO:tasks.workunit.client.0.smithi008.stderr: 87650K .......... .......... .......... .......... .......... 52% 8.92M 10s 2020-02-05T17:01:14.753 INFO:tasks.workunit.client.0.smithi008.stderr: 87700K .......... .......... .......... .......... .......... 52% 7.19M 10s 2020-02-05T17:01:14.757 INFO:tasks.workunit.client.0.smithi008.stderr: 87750K .......... .......... .......... .......... .......... 52% 11.2M 10s 2020-02-05T17:01:14.761 INFO:tasks.workunit.client.0.smithi008.stderr: 87800K .......... .......... .......... .......... .......... 52% 11.3M 10s 2020-02-05T17:01:14.766 INFO:tasks.workunit.client.0.smithi008.stderr: 87850K .......... .......... .......... .......... .......... 52% 9.33M 10s 2020-02-05T17:01:14.770 INFO:tasks.workunit.client.0.smithi008.stderr: 87900K .......... .......... .......... .......... .......... 52% 12.2M 10s 2020-02-05T17:01:14.775 INFO:tasks.workunit.client.0.smithi008.stderr: 87950K .......... .......... .......... .......... .......... 52% 11.5M 10s 2020-02-05T17:01:14.779 INFO:tasks.workunit.client.0.smithi008.stderr: 88000K .......... .......... .......... .......... .......... 52% 11.4M 10s 2020-02-05T17:01:14.785 INFO:tasks.workunit.client.0.smithi008.stderr: 88050K .......... .......... .......... .......... .......... 52% 9.42M 10s 2020-02-05T17:01:14.788 INFO:tasks.workunit.client.0.smithi008.stderr: 88100K .......... .......... .......... .......... .......... 52% 11.6M 10s 2020-02-05T17:01:14.793 INFO:tasks.workunit.client.0.smithi008.stderr: 88150K .......... .......... .......... .......... .......... 52% 11.2M 10s 2020-02-05T17:01:14.797 INFO:tasks.workunit.client.0.smithi008.stderr: 88200K .......... .......... .......... .......... .......... 52% 10.9M 10s 2020-02-05T17:01:14.803 INFO:tasks.workunit.client.0.smithi008.stderr: 88250K .......... .......... .......... .......... .......... 52% 8.92M 10s 2020-02-05T17:01:14.809 INFO:tasks.workunit.client.0.smithi008.stderr: 88300K .......... .......... .......... .......... .......... 52% 8.34M 10s 2020-02-05T17:01:14.816 INFO:tasks.workunit.client.0.smithi008.stderr: 88350K .......... .......... .......... .......... .......... 52% 6.82M 10s 2020-02-05T17:01:14.823 INFO:tasks.workunit.client.0.smithi008.stderr: 88400K .......... .......... .......... .......... .......... 52% 6.87M 10s 2020-02-05T17:01:14.831 INFO:tasks.workunit.client.0.smithi008.stderr: 88450K .......... .......... .......... .......... .......... 52% 6.32M 10s 2020-02-05T17:01:14.838 INFO:tasks.workunit.client.0.smithi008.stderr: 88500K .......... .......... .......... .......... .......... 52% 7.25M 10s 2020-02-05T17:01:14.845 INFO:tasks.workunit.client.0.smithi008.stderr: 88550K .......... .......... .......... .......... .......... 52% 6.69M 10s 2020-02-05T17:01:14.853 INFO:tasks.workunit.client.0.smithi008.stderr: 88600K .......... .......... .......... .......... .......... 52% 6.54M 10s 2020-02-05T17:01:14.860 INFO:tasks.workunit.client.0.smithi008.stderr: 88650K .......... .......... .......... .......... .......... 52% 6.07M 10s 2020-02-05T17:01:14.867 INFO:tasks.workunit.client.0.smithi008.stderr: 88700K .......... .......... .......... .......... .......... 52% 7.68M 10s 2020-02-05T17:01:14.876 INFO:tasks.workunit.client.0.smithi008.stderr: 88750K .......... .......... .......... .......... .......... 52% 7.73M 10s 2020-02-05T17:01:14.878 INFO:tasks.workunit.client.0.smithi008.stderr: 88800K .......... .......... .......... .......... .......... 52% 9.56M 10s 2020-02-05T17:01:14.884 INFO:tasks.workunit.client.0.smithi008.stderr: 88850K .......... .......... .......... .......... .......... 52% 7.66M 10s 2020-02-05T17:01:14.890 INFO:tasks.workunit.client.0.smithi008.stderr: 88900K .......... .......... .......... .......... .......... 52% 9.27M 10s 2020-02-05T17:01:14.895 INFO:tasks.workunit.client.0.smithi008.stderr: 88950K .......... .......... .......... .......... .......... 52% 9.04M 10s 2020-02-05T17:01:14.903 INFO:tasks.workunit.client.0.smithi008.stderr: 89000K .......... .......... .......... .......... .......... 52% 8.84M 10s 2020-02-05T17:01:14.907 INFO:tasks.workunit.client.0.smithi008.stderr: 89050K .......... .......... .......... .......... .......... 52% 7.55M 10s 2020-02-05T17:01:14.913 INFO:tasks.workunit.client.0.smithi008.stderr: 89100K .......... .......... .......... .......... .......... 52% 8.82M 10s 2020-02-05T17:01:14.918 INFO:tasks.workunit.client.0.smithi008.stderr: 89150K .......... .......... .......... .......... .......... 52% 8.48M 10s 2020-02-05T17:01:14.924 INFO:tasks.workunit.client.0.smithi008.stderr: 89200K .......... .......... .......... .......... .......... 52% 9.01M 10s 2020-02-05T17:01:14.930 INFO:tasks.workunit.client.0.smithi008.stderr: 89250K .......... .......... .......... .......... .......... 52% 7.47M 10s 2020-02-05T17:01:14.937 INFO:tasks.workunit.client.0.smithi008.stderr: 89300K .......... .......... .......... .......... .......... 53% 7.97M 10s 2020-02-05T17:01:14.942 INFO:tasks.workunit.client.0.smithi008.stderr: 89350K .......... .......... .......... .......... .......... 53% 8.56M 10s 2020-02-05T17:01:14.948 INFO:tasks.workunit.client.0.smithi008.stderr: 89400K .......... .......... .......... .......... .......... 53% 8.78M 10s 2020-02-05T17:01:14.955 INFO:tasks.workunit.client.0.smithi008.stderr: 89450K .......... .......... .......... .......... .......... 53% 7.14M 10s 2020-02-05T17:01:14.961 INFO:tasks.workunit.client.0.smithi008.stderr: 89500K .......... .......... .......... .......... .......... 53% 7.59M 10s 2020-02-05T17:01:14.968 INFO:tasks.workunit.client.0.smithi008.stderr: 89550K .......... .......... .......... .......... .......... 53% 6.85M 10s 2020-02-05T17:01:14.981 INFO:tasks.workunit.client.0.smithi008.stderr: 89600K .......... .......... .......... .......... .......... 53% 3.77M 10s 2020-02-05T17:01:14.988 INFO:tasks.workunit.client.0.smithi008.stderr: 89650K .......... .......... .......... .......... .......... 53% 7.76M 10s 2020-02-05T17:01:14.993 INFO:tasks.workunit.client.0.smithi008.stderr: 89700K .......... .......... .......... .......... .......... 53% 9.06M 10s 2020-02-05T17:01:14.998 INFO:tasks.workunit.client.0.smithi008.stderr: 89750K .......... .......... .......... .......... .......... 53% 9.04M 10s 2020-02-05T17:01:15.004 INFO:tasks.workunit.client.0.smithi008.stderr: 89800K .......... .......... .......... .......... .......... 53% 8.95M 10s 2020-02-05T17:01:15.011 INFO:tasks.workunit.client.0.smithi008.stderr: 89850K .......... .......... .......... .......... .......... 53% 7.02M 10s 2020-02-05T17:01:15.018 INFO:tasks.workunit.client.0.smithi008.stderr: 89900K .......... .......... .......... .......... .......... 53% 7.90M 10s 2020-02-05T17:01:15.023 INFO:tasks.workunit.client.0.smithi008.stderr: 89950K .......... .......... .......... .......... .......... 53% 7.96M 10s 2020-02-05T17:01:15.029 INFO:tasks.workunit.client.0.smithi008.stderr: 90000K .......... .......... .......... .......... .......... 53% 8.16M 10s 2020-02-05T17:01:15.037 INFO:tasks.workunit.client.0.smithi008.stderr: 90050K .......... .......... .......... .......... .......... 53% 6.17M 10s 2020-02-05T17:01:15.043 INFO:tasks.workunit.client.0.smithi008.stderr: 90100K .......... .......... .......... .......... .......... 53% 7.63M 10s 2020-02-05T17:01:15.049 INFO:tasks.workunit.client.0.smithi008.stderr: 90150K .......... .......... .......... .......... .......... 53% 8.18M 9s 2020-02-05T17:01:15.055 INFO:tasks.workunit.client.0.smithi008.stderr: 90200K .......... .......... .......... .......... .......... 53% 8.42M 9s 2020-02-05T17:01:15.061 INFO:tasks.workunit.client.0.smithi008.stderr: 90250K .......... .......... .......... .......... .......... 53% 8.44M 9s 2020-02-05T17:01:15.065 INFO:tasks.workunit.client.0.smithi008.stderr: 90300K .......... .......... .......... .......... .......... 53% 11.5M 9s 2020-02-05T17:01:15.069 INFO:tasks.workunit.client.0.smithi008.stderr: 90350K .......... .......... .......... .......... .......... 53% 12.5M 9s 2020-02-05T17:01:15.073 INFO:tasks.workunit.client.0.smithi008.stderr: 90400K .......... .......... .......... .......... .......... 53% 11.1M 9s 2020-02-05T17:01:15.079 INFO:tasks.workunit.client.0.smithi008.stderr: 90450K .......... .......... .......... .......... .......... 53% 8.46M 9s 2020-02-05T17:01:15.083 INFO:tasks.workunit.client.0.smithi008.stderr: 90500K .......... .......... .......... .......... .......... 53% 11.5M 9s 2020-02-05T17:01:15.087 INFO:tasks.workunit.client.0.smithi008.stderr: 90550K .......... .......... .......... .......... .......... 53% 11.2M 9s 2020-02-05T17:01:15.092 INFO:tasks.workunit.client.0.smithi008.stderr: 90600K .......... .......... .......... .......... .......... 53% 11.5M 9s 2020-02-05T17:01:15.097 INFO:tasks.workunit.client.0.smithi008.stderr: 90650K .......... .......... .......... .......... .......... 53% 9.87M 9s 2020-02-05T17:01:15.101 INFO:tasks.workunit.client.0.smithi008.stderr: 90700K .......... .......... .......... .......... .......... 53% 11.4M 9s 2020-02-05T17:01:15.105 INFO:tasks.workunit.client.0.smithi008.stderr: 90750K .......... .......... .......... .......... .......... 53% 11.6M 9s 2020-02-05T17:01:15.110 INFO:tasks.workunit.client.0.smithi008.stderr: 90800K .......... .......... .......... .......... .......... 53% 11.2M 9s 2020-02-05T17:01:15.115 INFO:tasks.workunit.client.0.smithi008.stderr: 90850K .......... .......... .......... .......... .......... 53% 9.39M 9s 2020-02-05T17:01:15.119 INFO:tasks.workunit.client.0.smithi008.stderr: 90900K .......... .......... .......... .......... .......... 53% 10.5M 9s 2020-02-05T17:01:15.124 INFO:tasks.workunit.client.0.smithi008.stderr: 90950K .......... .......... .......... .......... .......... 53% 11.6M 9s 2020-02-05T17:01:15.128 INFO:tasks.workunit.client.0.smithi008.stderr: 91000K .......... .......... .......... .......... .......... 54% 11.7M 9s 2020-02-05T17:01:15.133 INFO:tasks.workunit.client.0.smithi008.stderr: 91050K .......... .......... .......... .......... .......... 54% 9.93M 9s 2020-02-05T17:01:15.137 INFO:tasks.workunit.client.0.smithi008.stderr: 91100K .......... .......... .......... .......... .......... 54% 11.3M 9s 2020-02-05T17:01:15.143 INFO:tasks.workunit.client.0.smithi008.stderr: 91150K .......... .......... .......... .......... .......... 54% 8.99M 9s 2020-02-05T17:01:15.149 INFO:tasks.workunit.client.0.smithi008.stderr: 91200K .......... .......... .......... .......... .......... 54% 7.88M 9s 2020-02-05T17:01:15.155 INFO:tasks.workunit.client.0.smithi008.stderr: 91250K .......... .......... .......... .......... .......... 54% 8.21M 9s 2020-02-05T17:01:15.160 INFO:tasks.workunit.client.0.smithi008.stderr: 91300K .......... .......... .......... .......... .......... 54% 9.86M 9s 2020-02-05T17:01:15.166 INFO:tasks.workunit.client.0.smithi008.stderr: 91350K .......... .......... .......... .......... .......... 54% 8.67M 9s 2020-02-05T17:01:15.171 INFO:tasks.workunit.client.0.smithi008.stderr: 91400K .......... .......... .......... .......... .......... 54% 8.65M 9s 2020-02-05T17:01:15.178 INFO:tasks.workunit.client.0.smithi008.stderr: 91450K .......... .......... .......... .......... .......... 54% 7.59M 9s 2020-02-05T17:01:15.183 INFO:tasks.workunit.client.0.smithi008.stderr: 91500K .......... .......... .......... .......... .......... 54% 8.95M 9s 2020-02-05T17:01:15.189 INFO:tasks.workunit.client.0.smithi008.stderr: 91550K .......... .......... .......... .......... .......... 54% 8.50M 9s 2020-02-05T17:01:15.195 INFO:tasks.workunit.client.0.smithi008.stderr: 91600K .......... .......... .......... .......... .......... 54% 8.13M 9s 2020-02-05T17:01:15.202 INFO:tasks.workunit.client.0.smithi008.stderr: 91650K .......... .......... .......... .......... .......... 54% 6.70M 9s 2020-02-05T17:01:15.208 INFO:tasks.workunit.client.0.smithi008.stderr: 91700K .......... .......... .......... .......... .......... 54% 7.64M 9s 2020-02-05T17:01:15.215 INFO:tasks.workunit.client.0.smithi008.stderr: 91750K .......... .......... .......... .......... .......... 54% 7.30M 9s 2020-02-05T17:01:15.226 INFO:tasks.workunit.client.0.smithi008.stderr: 91800K .......... .......... .......... .......... .......... 54% 4.62M 9s 2020-02-05T17:01:15.233 INFO:tasks.workunit.client.0.smithi008.stderr: 91850K .......... .......... .......... .......... .......... 54% 6.50M 9s 2020-02-05T17:01:15.239 INFO:tasks.workunit.client.0.smithi008.stderr: 91900K .......... .......... .......... .......... .......... 54% 7.67M 9s 2020-02-05T17:01:15.245 INFO:tasks.workunit.client.0.smithi008.stderr: 91950K .......... .......... .......... .......... .......... 54% 9.30M 9s 2020-02-05T17:01:15.249 INFO:tasks.workunit.client.0.smithi008.stderr: 92000K .......... .......... .......... .......... .......... 54% 11.6M 9s 2020-02-05T17:01:15.255 INFO:tasks.workunit.client.0.smithi008.stderr: 92050K .......... .......... .......... .......... .......... 54% 8.71M 9s 2020-02-05T17:01:15.260 INFO:tasks.workunit.client.0.smithi008.stderr: 92100K .......... .......... .......... .......... .......... 54% 9.11M 9s 2020-02-05T17:01:15.266 INFO:tasks.workunit.client.0.smithi008.stderr: 92150K .......... .......... .......... .......... .......... 54% 8.34M 9s 2020-02-05T17:01:15.272 INFO:tasks.workunit.client.0.smithi008.stderr: 92200K .......... .......... .......... .......... .......... 54% 7.78M 9s 2020-02-05T17:01:15.279 INFO:tasks.workunit.client.0.smithi008.stderr: 92250K .......... .......... .......... .......... .......... 54% 6.96M 9s 2020-02-05T17:01:15.285 INFO:tasks.workunit.client.0.smithi008.stderr: 92300K .......... .......... .......... .......... .......... 54% 8.99M 9s 2020-02-05T17:01:15.290 INFO:tasks.workunit.client.0.smithi008.stderr: 92350K .......... .......... .......... .......... .......... 54% 9.10M 9s 2020-02-05T17:01:15.296 INFO:tasks.workunit.client.0.smithi008.stderr: 92400K .......... .......... .......... .......... .......... 54% 8.91M 9s 2020-02-05T17:01:15.303 INFO:tasks.workunit.client.0.smithi008.stderr: 92450K .......... .......... .......... .......... .......... 54% 6.35M 9s 2020-02-05T17:01:15.310 INFO:tasks.workunit.client.0.smithi008.stderr: 92500K .......... .......... .......... .......... .......... 54% 7.66M 9s 2020-02-05T17:01:15.316 INFO:tasks.workunit.client.0.smithi008.stderr: 92550K .......... .......... .......... .......... .......... 54% 7.36M 9s 2020-02-05T17:01:15.323 INFO:tasks.workunit.client.0.smithi008.stderr: 92600K .......... .......... .......... .......... .......... 54% 6.77M 9s 2020-02-05T17:01:15.332 INFO:tasks.workunit.client.0.smithi008.stderr: 92650K .......... .......... .......... .......... .......... 54% 5.85M 9s 2020-02-05T17:01:15.340 INFO:tasks.workunit.client.0.smithi008.stderr: 92700K .......... .......... .......... .......... .......... 55% 6.51M 9s 2020-02-05T17:01:15.345 INFO:tasks.workunit.client.0.smithi008.stderr: 92750K .......... .......... .......... .......... .......... 55% 8.03M 9s 2020-02-05T17:01:15.352 INFO:tasks.workunit.client.0.smithi008.stderr: 92800K .......... .......... .......... .......... .......... 55% 8.87M 9s 2020-02-05T17:01:15.356 INFO:tasks.workunit.client.0.smithi008.stderr: 92850K .......... .......... .......... .......... .......... 55% 8.72M 9s 2020-02-05T17:01:15.361 INFO:tasks.workunit.client.0.smithi008.stderr: 92900K .......... .......... .......... .......... .......... 55% 10.4M 9s 2020-02-05T17:01:15.366 INFO:tasks.workunit.client.0.smithi008.stderr: 92950K .......... .......... .......... .......... .......... 55% 10.5M 9s 2020-02-05T17:01:15.371 INFO:tasks.workunit.client.0.smithi008.stderr: 93000K .......... .......... .......... .......... .......... 55% 8.98M 9s 2020-02-05T17:01:15.378 INFO:tasks.workunit.client.0.smithi008.stderr: 93050K .......... .......... .......... .......... .......... 55% 6.83M 9s 2020-02-05T17:01:15.385 INFO:tasks.workunit.client.0.smithi008.stderr: 93100K .......... .......... .......... .......... .......... 55% 7.43M 9s 2020-02-05T17:01:15.392 INFO:tasks.workunit.client.0.smithi008.stderr: 93150K .......... .......... .......... .......... .......... 55% 7.56M 9s 2020-02-05T17:01:15.398 INFO:tasks.workunit.client.0.smithi008.stderr: 93200K .......... .......... .......... .......... .......... 55% 7.13M 9s 2020-02-05T17:01:15.407 INFO:tasks.workunit.client.0.smithi008.stderr: 93250K .......... .......... .......... .......... .......... 55% 5.76M 9s 2020-02-05T17:01:15.414 INFO:tasks.workunit.client.0.smithi008.stderr: 93300K .......... .......... .......... .......... .......... 55% 6.70M 9s 2020-02-05T17:01:15.422 INFO:tasks.workunit.client.0.smithi008.stderr: 93350K .......... .......... .......... .......... .......... 55% 6.13M 9s 2020-02-05T17:01:15.430 INFO:tasks.workunit.client.0.smithi008.stderr: 93400K .......... .......... .......... .......... .......... 55% 6.00M 9s 2020-02-05T17:01:15.440 INFO:tasks.workunit.client.0.smithi008.stderr: 93450K .......... .......... .......... .......... .......... 55% 5.08M 9s 2020-02-05T17:01:15.449 INFO:tasks.workunit.client.0.smithi008.stderr: 93500K .......... .......... .......... .......... .......... 55% 5.61M 9s 2020-02-05T17:01:15.457 INFO:tasks.workunit.client.0.smithi008.stderr: 93550K .......... .......... .......... .......... .......... 55% 5.88M 9s 2020-02-05T17:01:15.466 INFO:tasks.workunit.client.0.smithi008.stderr: 93600K .......... .......... .......... .......... .......... 55% 5.81M 9s 2020-02-05T17:01:15.481 INFO:tasks.workunit.client.0.smithi008.stderr: 93650K .......... .......... .......... .......... .......... 55% 4.94M 9s 2020-02-05T17:01:15.488 INFO:tasks.workunit.client.0.smithi008.stderr: 93700K .......... .......... .......... .......... .......... 55% 4.06M 9s 2020-02-05T17:01:15.493 INFO:tasks.workunit.client.0.smithi008.stderr: 93750K .......... .......... .......... .......... .......... 55% 8.08M 9s 2020-02-05T17:01:15.499 INFO:tasks.workunit.client.0.smithi008.stderr: 93800K .......... .......... .......... .......... .......... 55% 7.97M 9s 2020-02-05T17:01:15.508 INFO:tasks.workunit.client.0.smithi008.stderr: 93850K .......... .......... .......... .......... .......... 55% 7.90M 9s 2020-02-05T17:01:15.511 INFO:tasks.workunit.client.0.smithi008.stderr: 93900K .......... .......... .......... .......... .......... 55% 8.39M 9s 2020-02-05T17:01:15.517 INFO:tasks.workunit.client.0.smithi008.stderr: 93950K .......... .......... .......... .......... .......... 55% 8.90M 9s 2020-02-05T17:01:15.523 INFO:tasks.workunit.client.0.smithi008.stderr: 94000K .......... .......... .......... .......... .......... 55% 8.46M 9s 2020-02-05T17:01:15.530 INFO:tasks.workunit.client.0.smithi008.stderr: 94050K .......... .......... .......... .......... .......... 55% 6.67M 9s 2020-02-05T17:01:15.537 INFO:tasks.workunit.client.0.smithi008.stderr: 94100K .......... .......... .......... .......... .......... 55% 7.42M 9s 2020-02-05T17:01:15.543 INFO:tasks.workunit.client.0.smithi008.stderr: 94150K .......... .......... .......... .......... .......... 55% 7.25M 9s 2020-02-05T17:01:15.551 INFO:tasks.workunit.client.0.smithi008.stderr: 94200K .......... .......... .......... .......... .......... 55% 6.15M 9s 2020-02-05T17:01:15.558 INFO:tasks.workunit.client.0.smithi008.stderr: 94250K .......... .......... .......... .......... .......... 55% 6.87M 9s 2020-02-05T17:01:15.563 INFO:tasks.workunit.client.0.smithi008.stderr: 94300K .......... .......... .......... .......... .......... 55% 9.78M 9s 2020-02-05T17:01:15.567 INFO:tasks.workunit.client.0.smithi008.stderr: 94350K .......... .......... .......... .......... .......... 56% 11.7M 9s 2020-02-05T17:01:15.570 INFO:tasks.workunit.client.0.smithi008.stderr: 94400K .......... .......... .......... .......... .......... 56% 14.9M 9s 2020-02-05T17:01:15.573 INFO:tasks.workunit.client.0.smithi008.stderr: 94450K .......... .......... .......... .......... .......... 56% 14.9M 9s 2020-02-05T17:01:15.578 INFO:tasks.workunit.client.0.smithi008.stderr: 94500K .......... .......... .......... .......... .......... 56% 14.8M 9s 2020-02-05T17:01:15.580 INFO:tasks.workunit.client.0.smithi008.stderr: 94550K .......... .......... .......... .......... .......... 56% 16.0M 9s 2020-02-05T17:01:15.583 INFO:tasks.workunit.client.0.smithi008.stderr: 94600K .......... .......... .......... .......... .......... 56% 15.9M 9s 2020-02-05T17:01:15.587 INFO:tasks.workunit.client.0.smithi008.stderr: 94650K .......... .......... .......... .......... .......... 56% 12.5M 9s 2020-02-05T17:01:15.590 INFO:tasks.workunit.client.0.smithi008.stderr: 94700K .......... .......... .......... .......... .......... 56% 13.4M 9s 2020-02-05T17:01:15.595 INFO:tasks.workunit.client.0.smithi008.stderr: 94750K .......... .......... .......... .......... .......... 56% 13.5M 9s 2020-02-05T17:01:15.598 INFO:tasks.workunit.client.0.smithi008.stderr: 94800K .......... .......... .......... .......... .......... 56% 12.0M 9s 2020-02-05T17:01:15.604 INFO:tasks.workunit.client.0.smithi008.stderr: 94850K .......... .......... .......... .......... .......... 56% 9.02M 9s 2020-02-05T17:01:15.610 INFO:tasks.workunit.client.0.smithi008.stderr: 94900K .......... .......... .......... .......... .......... 56% 9.67M 9s 2020-02-05T17:01:15.615 INFO:tasks.workunit.client.0.smithi008.stderr: 94950K .......... .......... .......... .......... .......... 56% 9.08M 9s 2020-02-05T17:01:15.620 INFO:tasks.workunit.client.0.smithi008.stderr: 95000K .......... .......... .......... .......... .......... 56% 8.01M 9s 2020-02-05T17:01:15.627 INFO:tasks.workunit.client.0.smithi008.stderr: 95050K .......... .......... .......... .......... .......... 56% 7.48M 9s 2020-02-05T17:01:15.632 INFO:tasks.workunit.client.0.smithi008.stderr: 95100K .......... .......... .......... .......... .......... 56% 9.03M 9s 2020-02-05T17:01:15.638 INFO:tasks.workunit.client.0.smithi008.stderr: 95150K .......... .......... .......... .......... .......... 56% 9.04M 9s 2020-02-05T17:01:15.643 INFO:tasks.workunit.client.0.smithi008.stderr: 95200K .......... .......... .......... .......... .......... 56% 8.95M 9s 2020-02-05T17:01:15.650 INFO:tasks.workunit.client.0.smithi008.stderr: 95250K .......... .......... .......... .......... .......... 56% 6.94M 9s 2020-02-05T17:01:15.657 INFO:tasks.workunit.client.0.smithi008.stderr: 95300K .......... .......... .......... .......... .......... 56% 7.50M 9s 2020-02-05T17:01:15.665 INFO:tasks.workunit.client.0.smithi008.stderr: 95350K .......... .......... .......... .......... .......... 56% 6.93M 9s 2020-02-05T17:01:15.671 INFO:tasks.workunit.client.0.smithi008.stderr: 95400K .......... .......... .......... .......... .......... 56% 6.94M 9s 2020-02-05T17:01:15.678 INFO:tasks.workunit.client.0.smithi008.stderr: 95450K .......... .......... .......... .......... .......... 56% 6.59M 9s 2020-02-05T17:01:15.685 INFO:tasks.workunit.client.0.smithi008.stderr: 95500K .......... .......... .......... .......... .......... 56% 8.15M 9s 2020-02-05T17:01:15.692 INFO:tasks.workunit.client.0.smithi008.stderr: 95550K .......... .......... .......... .......... .......... 56% 6.99M 9s 2020-02-05T17:01:15.699 INFO:tasks.workunit.client.0.smithi008.stderr: 95600K .......... .......... .......... .......... .......... 56% 6.77M 9s 2020-02-05T17:01:15.708 INFO:tasks.workunit.client.0.smithi008.stderr: 95650K .......... .......... .......... .......... .......... 56% 5.65M 9s 2020-02-05T17:01:15.718 INFO:tasks.workunit.client.0.smithi008.stderr: 95700K .......... .......... .......... .......... .......... 56% 5.86M 9s 2020-02-05T17:01:15.723 INFO:tasks.workunit.client.0.smithi008.stderr: 95750K .......... .......... .......... .......... .......... 56% 6.78M 9s 2020-02-05T17:01:15.729 INFO:tasks.workunit.client.0.smithi008.stderr: 95800K .......... .......... .......... .......... .......... 56% 8.24M 9s 2020-02-05T17:01:15.738 INFO:tasks.workunit.client.0.smithi008.stderr: 95850K .......... .......... .......... .......... .......... 56% 5.61M 9s 2020-02-05T17:01:15.748 INFO:tasks.workunit.client.0.smithi008.stderr: 95900K .......... .......... .......... .......... .......... 56% 4.82M 9s 2020-02-05T17:01:15.754 INFO:tasks.workunit.client.0.smithi008.stderr: 95950K .......... .......... .......... .......... .......... 56% 7.57M 9s 2020-02-05T17:01:15.760 INFO:tasks.workunit.client.0.smithi008.stderr: 96000K .......... .......... .......... .......... .......... 56% 8.32M 9s 2020-02-05T17:01:15.766 INFO:tasks.workunit.client.0.smithi008.stderr: 96050K .......... .......... .......... .......... .......... 57% 8.15M 9s 2020-02-05T17:01:15.771 INFO:tasks.workunit.client.0.smithi008.stderr: 96100K .......... .......... .......... .......... .......... 57% 9.17M 9s 2020-02-05T17:01:15.777 INFO:tasks.workunit.client.0.smithi008.stderr: 96150K .......... .......... .......... .......... .......... 57% 8.23M 9s 2020-02-05T17:01:15.782 INFO:tasks.workunit.client.0.smithi008.stderr: 96200K .......... .......... .......... .......... .......... 57% 8.87M 9s 2020-02-05T17:01:15.789 INFO:tasks.workunit.client.0.smithi008.stderr: 96250K .......... .......... .......... .......... .......... 57% 7.73M 9s 2020-02-05T17:01:15.795 INFO:tasks.workunit.client.0.smithi008.stderr: 96300K .......... .......... .......... .......... .......... 57% 8.32M 9s 2020-02-05T17:01:15.800 INFO:tasks.workunit.client.0.smithi008.stderr: 96350K .......... .......... .......... .......... .......... 57% 9.37M 9s 2020-02-05T17:01:15.806 INFO:tasks.workunit.client.0.smithi008.stderr: 96400K .......... .......... .......... .......... .......... 57% 8.21M 9s 2020-02-05T17:01:15.813 INFO:tasks.workunit.client.0.smithi008.stderr: 96450K .......... .......... .......... .......... .......... 57% 6.68M 9s 2020-02-05T17:01:15.821 INFO:tasks.workunit.client.0.smithi008.stderr: 96500K .......... .......... .......... .......... .......... 57% 6.69M 9s 2020-02-05T17:01:15.828 INFO:tasks.workunit.client.0.smithi008.stderr: 96550K .......... .......... .......... .......... .......... 57% 6.90M 9s 2020-02-05T17:01:15.834 INFO:tasks.workunit.client.0.smithi008.stderr: 96600K .......... .......... .......... .......... .......... 57% 7.43M 9s 2020-02-05T17:01:15.842 INFO:tasks.workunit.client.0.smithi008.stderr: 96650K .......... .......... .......... .......... .......... 57% 6.06M 9s 2020-02-05T17:01:15.850 INFO:tasks.workunit.client.0.smithi008.stderr: 96700K .......... .......... .......... .......... .......... 57% 6.62M 9s 2020-02-05T17:01:15.857 INFO:tasks.workunit.client.0.smithi008.stderr: 96750K .......... .......... .......... .......... .......... 57% 6.55M 9s 2020-02-05T17:01:15.865 INFO:tasks.workunit.client.0.smithi008.stderr: 96800K .......... .......... .......... .......... .......... 57% 6.19M 9s 2020-02-05T17:01:15.875 INFO:tasks.workunit.client.0.smithi008.stderr: 96850K .......... .......... .......... .......... .......... 57% 5.26M 9s 2020-02-05T17:01:15.881 INFO:tasks.workunit.client.0.smithi008.stderr: 96900K .......... .......... .......... .......... .......... 57% 6.82M 9s 2020-02-05T17:01:15.887 INFO:tasks.workunit.client.0.smithi008.stderr: 96950K .......... .......... .......... .......... .......... 57% 8.48M 9s 2020-02-05T17:01:15.892 INFO:tasks.workunit.client.0.smithi008.stderr: 97000K .......... .......... .......... .......... .......... 57% 9.24M 9s 2020-02-05T17:01:15.899 INFO:tasks.workunit.client.0.smithi008.stderr: 97050K .......... .......... .......... .......... .......... 57% 7.20M 9s 2020-02-05T17:01:15.905 INFO:tasks.workunit.client.0.smithi008.stderr: 97100K .......... .......... .......... .......... .......... 57% 8.10M 9s 2020-02-05T17:01:15.912 INFO:tasks.workunit.client.0.smithi008.stderr: 97150K .......... .......... .......... .......... .......... 57% 6.98M 9s 2020-02-05T17:01:15.920 INFO:tasks.workunit.client.0.smithi008.stderr: 97200K .......... .......... .......... .......... .......... 57% 6.58M 9s 2020-02-05T17:01:15.927 INFO:tasks.workunit.client.0.smithi008.stderr: 97250K .......... .......... .......... .......... .......... 57% 6.63M 9s 2020-02-05T17:01:15.935 INFO:tasks.workunit.client.0.smithi008.stderr: 97300K .......... .......... .......... .......... .......... 57% 6.85M 9s 2020-02-05T17:01:15.941 INFO:tasks.workunit.client.0.smithi008.stderr: 97350K .......... .......... .......... .......... .......... 57% 6.87M 9s 2020-02-05T17:01:15.948 INFO:tasks.workunit.client.0.smithi008.stderr: 97400K .......... .......... .......... .......... .......... 57% 7.78M 9s 2020-02-05T17:01:15.955 INFO:tasks.workunit.client.0.smithi008.stderr: 97450K .......... .......... .......... .......... .......... 57% 6.43M 9s 2020-02-05T17:01:15.962 INFO:tasks.workunit.client.0.smithi008.stderr: 97500K .......... .......... .......... .......... .......... 57% 7.35M 9s 2020-02-05T17:01:15.968 INFO:tasks.workunit.client.0.smithi008.stderr: 97550K .......... .......... .......... .......... .......... 57% 8.21M 9s 2020-02-05T17:01:15.975 INFO:tasks.workunit.client.0.smithi008.stderr: 97600K .......... .......... .......... .......... .......... 57% 7.15M 9s 2020-02-05T17:01:15.984 INFO:tasks.workunit.client.0.smithi008.stderr: 97650K .......... .......... .......... .......... .......... 57% 5.36M 9s 2020-02-05T17:01:15.993 INFO:tasks.workunit.client.0.smithi008.stderr: 97700K .......... .......... .......... .......... .......... 57% 5.76M 9s 2020-02-05T17:01:16.008 INFO:tasks.workunit.client.0.smithi008.stderr: 97750K .......... .......... .......... .......... .......... 58% 5.43M 9s 2020-02-05T17:01:16.014 INFO:tasks.workunit.client.0.smithi008.stderr: 97800K .......... .......... .......... .......... .......... 58% 3.76M 9s 2020-02-05T17:01:16.021 INFO:tasks.workunit.client.0.smithi008.stderr: 97850K .......... .......... .......... .......... .......... 58% 6.99M 9s 2020-02-05T17:01:16.027 INFO:tasks.workunit.client.0.smithi008.stderr: 97900K .......... .......... .......... .......... .......... 58% 7.61M 9s 2020-02-05T17:01:16.034 INFO:tasks.workunit.client.0.smithi008.stderr: 97950K .......... .......... .......... .......... .......... 58% 7.85M 9s 2020-02-05T17:01:16.041 INFO:tasks.workunit.client.0.smithi008.stderr: 98000K .......... .......... .......... .......... .......... 58% 7.16M 9s 2020-02-05T17:01:16.048 INFO:tasks.workunit.client.0.smithi008.stderr: 98050K .......... .......... .......... .......... .......... 58% 6.22M 9s 2020-02-05T17:01:16.055 INFO:tasks.workunit.client.0.smithi008.stderr: 98100K .......... .......... .......... .......... .......... 58% 7.32M 9s 2020-02-05T17:01:16.062 INFO:tasks.workunit.client.0.smithi008.stderr: 98150K .......... .......... .......... .......... .......... 58% 7.68M 9s 2020-02-05T17:01:16.068 INFO:tasks.workunit.client.0.smithi008.stderr: 98200K .......... .......... .......... .......... .......... 58% 7.28M 9s 2020-02-05T17:01:16.076 INFO:tasks.workunit.client.0.smithi008.stderr: 98250K .......... .......... .......... .......... .......... 58% 6.52M 9s 2020-02-05T17:01:16.082 INFO:tasks.workunit.client.0.smithi008.stderr: 98300K .......... .......... .......... .......... .......... 58% 7.34M 9s 2020-02-05T17:01:16.088 INFO:tasks.workunit.client.0.smithi008.stderr: 98350K .......... .......... .......... .......... .......... 58% 8.28M 9s 2020-02-05T17:01:16.095 INFO:tasks.workunit.client.0.smithi008.stderr: 98400K .......... .......... .......... .......... .......... 58% 6.76M 9s 2020-02-05T17:01:16.101 INFO:tasks.workunit.client.0.smithi008.stderr: 98450K .......... .......... .......... .......... .......... 58% 8.34M 9s 2020-02-05T17:01:16.106 INFO:tasks.workunit.client.0.smithi008.stderr: 98500K .......... .......... .......... .......... .......... 58% 10.4M 9s 2020-02-05T17:01:16.110 INFO:tasks.workunit.client.0.smithi008.stderr: 98550K .......... .......... .......... .......... .......... 58% 11.2M 9s 2020-02-05T17:01:16.114 INFO:tasks.workunit.client.0.smithi008.stderr: 98600K .......... .......... .......... .......... .......... 58% 12.5M 9s 2020-02-05T17:01:16.119 INFO:tasks.workunit.client.0.smithi008.stderr: 98650K .......... .......... .......... .......... .......... 58% 10.7M 8s 2020-02-05T17:01:16.123 INFO:tasks.workunit.client.0.smithi008.stderr: 98700K .......... .......... .......... .......... .......... 58% 9.84M 8s 2020-02-05T17:01:16.128 INFO:tasks.workunit.client.0.smithi008.stderr: 98750K .......... .......... .......... .......... .......... 58% 10.9M 8s 2020-02-05T17:01:16.133 INFO:tasks.workunit.client.0.smithi008.stderr: 98800K .......... .......... .......... .......... .......... 58% 9.58M 8s 2020-02-05T17:01:16.139 INFO:tasks.workunit.client.0.smithi008.stderr: 98850K .......... .......... .......... .......... .......... 58% 8.62M 8s 2020-02-05T17:01:16.144 INFO:tasks.workunit.client.0.smithi008.stderr: 98900K .......... .......... .......... .......... .......... 58% 9.64M 8s 2020-02-05T17:01:16.150 INFO:tasks.workunit.client.0.smithi008.stderr: 98950K .......... .......... .......... .......... .......... 58% 8.92M 8s 2020-02-05T17:01:16.153 INFO:tasks.workunit.client.0.smithi008.stderr: 99000K .......... .......... .......... .......... .......... 58% 13.4M 8s 2020-02-05T17:01:16.158 INFO:tasks.workunit.client.0.smithi008.stderr: 99050K .......... .......... .......... .......... .......... 58% 10.2M 8s 2020-02-05T17:01:16.163 INFO:tasks.workunit.client.0.smithi008.stderr: 99100K .......... .......... .......... .......... .......... 58% 9.87M 8s 2020-02-05T17:01:16.166 INFO:tasks.workunit.client.0.smithi008.stderr: 99150K .......... .......... .......... .......... .......... 58% 13.3M 8s 2020-02-05T17:01:16.170 INFO:tasks.workunit.client.0.smithi008.stderr: 99200K .......... .......... .......... .......... .......... 58% 14.8M 8s 2020-02-05T17:01:16.174 INFO:tasks.workunit.client.0.smithi008.stderr: 99250K .......... .......... .......... .......... .......... 58% 10.5M 8s 2020-02-05T17:01:16.178 INFO:tasks.workunit.client.0.smithi008.stderr: 99300K .......... .......... .......... .......... .......... 58% 13.0M 8s 2020-02-05T17:01:16.182 INFO:tasks.workunit.client.0.smithi008.stderr: 99350K .......... .......... .......... .......... .......... 58% 13.0M 8s 2020-02-05T17:01:16.185 INFO:tasks.workunit.client.0.smithi008.stderr: 99400K .......... .......... .......... .......... .......... 58% 13.1M 8s 2020-02-05T17:01:16.190 INFO:tasks.workunit.client.0.smithi008.stderr: 99450K .......... .......... .......... .......... .......... 59% 10.8M 8s 2020-02-05T17:01:16.194 INFO:tasks.workunit.client.0.smithi008.stderr: 99500K .......... .......... .......... .......... .......... 59% 12.0M 8s 2020-02-05T17:01:16.198 INFO:tasks.workunit.client.0.smithi008.stderr: 99550K .......... .......... .......... .......... .......... 59% 12.9M 8s 2020-02-05T17:01:16.203 INFO:tasks.workunit.client.0.smithi008.stderr: 99600K .......... .......... .......... .......... .......... 59% 8.97M 8s 2020-02-05T17:01:16.210 INFO:tasks.workunit.client.0.smithi008.stderr: 99650K .......... .......... .......... .......... .......... 59% 7.33M 8s 2020-02-05T17:01:16.216 INFO:tasks.workunit.client.0.smithi008.stderr: 99700K .......... .......... .......... .......... .......... 59% 8.20M 8s 2020-02-05T17:01:16.222 INFO:tasks.workunit.client.0.smithi008.stderr: 99750K .......... .......... .......... .......... .......... 59% 9.12M 8s 2020-02-05T17:01:16.228 INFO:tasks.workunit.client.0.smithi008.stderr: 99800K .......... .......... .......... .......... .......... 59% 7.77M 8s 2020-02-05T17:01:16.235 INFO:tasks.workunit.client.0.smithi008.stderr: 99850K .......... .......... .......... .......... .......... 59% 6.73M 8s 2020-02-05T17:01:16.242 INFO:tasks.workunit.client.0.smithi008.stderr: 99900K .......... .......... .......... .......... .......... 59% 7.12M 8s 2020-02-05T17:01:16.248 INFO:tasks.workunit.client.0.smithi008.stderr: 99950K .......... .......... .......... .......... .......... 59% 7.83M 8s 2020-02-05T17:01:16.257 INFO:tasks.workunit.client.0.smithi008.stderr:100000K .......... .......... .......... .......... .......... 59% 5.56M 8s 2020-02-05T17:01:16.263 INFO:tasks.workunit.client.0.smithi008.stderr:100050K .......... .......... .......... .......... .......... 59% 7.99M 8s 2020-02-05T17:01:16.269 INFO:tasks.workunit.client.0.smithi008.stderr:100100K .......... .......... .......... .......... .......... 59% 8.60M 8s 2020-02-05T17:01:16.281 INFO:tasks.workunit.client.0.smithi008.stderr:100150K .......... .......... .......... .......... .......... 59% 8.68M 8s 2020-02-05T17:01:16.281 INFO:tasks.workunit.client.0.smithi008.stderr:100200K .......... .......... .......... .......... .......... 59% 9.09M 8s 2020-02-05T17:01:16.285 INFO:tasks.workunit.client.0.smithi008.stderr:100250K .......... .......... .......... .......... .......... 59% 8.99M 8s 2020-02-05T17:01:16.291 INFO:tasks.workunit.client.0.smithi008.stderr:100300K .......... .......... .......... .......... .......... 59% 7.97M 8s 2020-02-05T17:01:16.297 INFO:tasks.workunit.client.0.smithi008.stderr:100350K .......... .......... .......... .......... .......... 59% 8.87M 8s 2020-02-05T17:01:16.303 INFO:tasks.workunit.client.0.smithi008.stderr:100400K .......... .......... .......... .......... .......... 59% 8.06M 8s 2020-02-05T17:01:16.311 INFO:tasks.workunit.client.0.smithi008.stderr:100450K .......... .......... .......... .......... .......... 59% 6.50M 8s 2020-02-05T17:01:16.318 INFO:tasks.workunit.client.0.smithi008.stderr:100500K .......... .......... .......... .......... .......... 59% 6.88M 8s 2020-02-05T17:01:16.324 INFO:tasks.workunit.client.0.smithi008.stderr:100550K .......... .......... .......... .......... .......... 59% 7.68M 8s 2020-02-05T17:01:16.330 INFO:tasks.workunit.client.0.smithi008.stderr:100600K .......... .......... .......... .......... .......... 59% 8.57M 8s 2020-02-05T17:01:16.337 INFO:tasks.workunit.client.0.smithi008.stderr:100650K .......... .......... .......... .......... .......... 59% 6.43M 8s 2020-02-05T17:01:16.343 INFO:tasks.workunit.client.0.smithi008.stderr:100700K .......... .......... .......... .......... .......... 59% 7.97M 8s 2020-02-05T17:01:16.350 INFO:tasks.workunit.client.0.smithi008.stderr:100750K .......... .......... .......... .......... .......... 59% 7.05M 8s 2020-02-05T17:01:16.357 INFO:tasks.workunit.client.0.smithi008.stderr:100800K .......... .......... .......... .......... .......... 59% 6.88M 8s 2020-02-05T17:01:16.365 INFO:tasks.workunit.client.0.smithi008.stderr:100850K .......... .......... .......... .......... .......... 59% 6.36M 8s 2020-02-05T17:01:16.370 INFO:tasks.workunit.client.0.smithi008.stderr:100900K .......... .......... .......... .......... .......... 59% 9.40M 8s 2020-02-05T17:01:16.375 INFO:tasks.workunit.client.0.smithi008.stderr:100950K .......... .......... .......... .......... .......... 59% 8.52M 8s 2020-02-05T17:01:16.381 INFO:tasks.workunit.client.0.smithi008.stderr:101000K .......... .......... .......... .......... .......... 59% 9.23M 8s 2020-02-05T17:01:16.388 INFO:tasks.workunit.client.0.smithi008.stderr:101050K .......... .......... .......... .......... .......... 59% 6.82M 8s 2020-02-05T17:01:16.394 INFO:tasks.workunit.client.0.smithi008.stderr:101100K .......... .......... .......... .......... .......... 60% 8.40M 8s 2020-02-05T17:01:16.400 INFO:tasks.workunit.client.0.smithi008.stderr:101150K .......... .......... .......... .......... .......... 60% 8.23M 8s 2020-02-05T17:01:16.405 INFO:tasks.workunit.client.0.smithi008.stderr:101200K .......... .......... .......... .......... .......... 60% 9.27M 8s 2020-02-05T17:01:16.411 INFO:tasks.workunit.client.0.smithi008.stderr:101250K .......... .......... .......... .......... .......... 60% 8.72M 8s 2020-02-05T17:01:16.415 INFO:tasks.workunit.client.0.smithi008.stderr:101300K .......... .......... .......... .......... .......... 60% 11.7M 8s 2020-02-05T17:01:16.419 INFO:tasks.workunit.client.0.smithi008.stderr:101350K .......... .......... .......... .......... .......... 60% 11.7M 8s 2020-02-05T17:01:16.423 INFO:tasks.workunit.client.0.smithi008.stderr:101400K .......... .......... .......... .......... .......... 60% 11.2M 8s 2020-02-05T17:01:16.427 INFO:tasks.workunit.client.0.smithi008.stderr:101450K .......... .......... .......... .......... .......... 60% 11.2M 8s 2020-02-05T17:01:16.431 INFO:tasks.workunit.client.0.smithi008.stderr:101500K .......... .......... .......... .......... .......... 60% 13.7M 8s 2020-02-05T17:01:16.436 INFO:tasks.workunit.client.0.smithi008.stderr:101550K .......... .......... .......... .......... .......... 60% 13.2M 8s 2020-02-05T17:01:16.440 INFO:tasks.workunit.client.0.smithi008.stderr:101600K .......... .......... .......... .......... .......... 60% 13.5M 8s 2020-02-05T17:01:16.446 INFO:tasks.workunit.client.0.smithi008.stderr:101650K .......... .......... .......... .......... .......... 60% 11.4M 8s 2020-02-05T17:01:16.448 INFO:tasks.workunit.client.0.smithi008.stderr:101700K .......... .......... .......... .......... .......... 60% 13.6M 8s 2020-02-05T17:01:16.450 INFO:tasks.workunit.client.0.smithi008.stderr:101750K .......... .......... .......... .......... .......... 60% 13.8M 8s 2020-02-05T17:01:16.454 INFO:tasks.workunit.client.0.smithi008.stderr:101800K .......... .......... .......... .......... .......... 60% 13.6M 8s 2020-02-05T17:01:16.461 INFO:tasks.workunit.client.0.smithi008.stderr:101850K .......... .......... .......... .......... .......... 60% 6.47M 8s 2020-02-05T17:01:16.464 INFO:tasks.workunit.client.0.smithi008.stderr:101900K .......... .......... .......... .......... .......... 60% 16.9M 8s 2020-02-05T17:01:16.467 INFO:tasks.workunit.client.0.smithi008.stderr:101950K .......... .......... .......... .......... .......... 60% 17.1M 8s 2020-02-05T17:01:16.470 INFO:tasks.workunit.client.0.smithi008.stderr:102000K .......... .......... .......... .......... .......... 60% 16.9M 8s 2020-02-05T17:01:16.473 INFO:tasks.workunit.client.0.smithi008.stderr:102050K .......... .......... .......... .......... .......... 60% 14.9M 8s 2020-02-05T17:01:16.476 INFO:tasks.workunit.client.0.smithi008.stderr:102100K .......... .......... .......... .......... .......... 60% 14.0M 8s 2020-02-05T17:01:16.481 INFO:tasks.workunit.client.0.smithi008.stderr:102150K .......... .......... .......... .......... .......... 60% 11.5M 8s 2020-02-05T17:01:16.486 INFO:tasks.workunit.client.0.smithi008.stderr:102200K .......... .......... .......... .......... .......... 60% 10.1M 8s 2020-02-05T17:01:16.491 INFO:tasks.workunit.client.0.smithi008.stderr:102250K .......... .......... .......... .......... .......... 60% 9.41M 8s 2020-02-05T17:01:16.495 INFO:tasks.workunit.client.0.smithi008.stderr:102300K .......... .......... .......... .......... .......... 60% 10.6M 8s 2020-02-05T17:01:16.501 INFO:tasks.workunit.client.0.smithi008.stderr:102350K .......... .......... .......... .......... .......... 60% 9.93M 8s 2020-02-05T17:01:16.505 INFO:tasks.workunit.client.0.smithi008.stderr:102400K .......... .......... .......... .......... .......... 60% 10.0M 8s 2020-02-05T17:01:16.510 INFO:tasks.workunit.client.0.smithi008.stderr:102450K .......... .......... .......... .......... .......... 60% 9.83M 8s 2020-02-05T17:01:16.514 INFO:tasks.workunit.client.0.smithi008.stderr:102500K .......... .......... .......... .......... .......... 60% 12.7M 8s 2020-02-05T17:01:16.517 INFO:tasks.workunit.client.0.smithi008.stderr:102550K .......... .......... .......... .......... .......... 60% 14.3M 8s 2020-02-05T17:01:16.522 INFO:tasks.workunit.client.0.smithi008.stderr:102600K .......... .......... .......... .......... .......... 60% 10.2M 8s 2020-02-05T17:01:16.527 INFO:tasks.workunit.client.0.smithi008.stderr:102650K .......... .......... .......... .......... .......... 60% 10.4M 8s 2020-02-05T17:01:16.531 INFO:tasks.workunit.client.0.smithi008.stderr:102700K .......... .......... .......... .......... .......... 60% 12.3M 8s 2020-02-05T17:01:16.535 INFO:tasks.workunit.client.0.smithi008.stderr:102750K .......... .......... .......... .......... .......... 60% 12.1M 8s 2020-02-05T17:01:16.537 INFO:tasks.workunit.client.0.smithi008.stderr:102800K .......... .......... .......... .......... .......... 61% 19.2M 8s 2020-02-05T17:01:16.541 INFO:tasks.workunit.client.0.smithi008.stderr:102850K .......... .......... .......... .......... .......... 61% 12.1M 8s 2020-02-05T17:01:16.545 INFO:tasks.workunit.client.0.smithi008.stderr:102900K .......... .......... .......... .......... .......... 61% 11.9M 8s 2020-02-05T17:01:16.550 INFO:tasks.workunit.client.0.smithi008.stderr:102950K .......... .......... .......... .......... .......... 61% 10.6M 8s 2020-02-05T17:01:16.555 INFO:tasks.workunit.client.0.smithi008.stderr:103000K .......... .......... .......... .......... .......... 61% 9.68M 8s 2020-02-05T17:01:16.562 INFO:tasks.workunit.client.0.smithi008.stderr:103050K .......... .......... .......... .......... .......... 61% 7.98M 8s 2020-02-05T17:01:16.567 INFO:tasks.workunit.client.0.smithi008.stderr:103100K .......... .......... .......... .......... .......... 61% 8.84M 8s 2020-02-05T17:01:16.573 INFO:tasks.workunit.client.0.smithi008.stderr:103150K .......... .......... .......... .......... .......... 61% 8.39M 8s 2020-02-05T17:01:16.582 INFO:tasks.workunit.client.0.smithi008.stderr:103200K .......... .......... .......... .......... .......... 61% 5.77M 8s 2020-02-05T17:01:16.588 INFO:tasks.workunit.client.0.smithi008.stderr:103250K .......... .......... .......... .......... .......... 61% 7.04M 8s 2020-02-05T17:01:16.592 INFO:tasks.workunit.client.0.smithi008.stderr:103300K .......... .......... .......... .......... .......... 61% 12.4M 8s 2020-02-05T17:01:16.596 INFO:tasks.workunit.client.0.smithi008.stderr:103350K .......... .......... .......... .......... .......... 61% 12.9M 8s 2020-02-05T17:01:16.600 INFO:tasks.workunit.client.0.smithi008.stderr:103400K .......... .......... .......... .......... .......... 61% 12.8M 8s 2020-02-05T17:01:16.604 INFO:tasks.workunit.client.0.smithi008.stderr:103450K .......... .......... .......... .......... .......... 61% 10.5M 8s 2020-02-05T17:01:16.608 INFO:tasks.workunit.client.0.smithi008.stderr:103500K .......... .......... .......... .......... .......... 61% 12.4M 8s 2020-02-05T17:01:16.612 INFO:tasks.workunit.client.0.smithi008.stderr:103550K .......... .......... .......... .......... .......... 61% 12.4M 8s 2020-02-05T17:01:16.616 INFO:tasks.workunit.client.0.smithi008.stderr:103600K .......... .......... .......... .......... .......... 61% 12.2M 8s 2020-02-05T17:01:16.623 INFO:tasks.workunit.client.0.smithi008.stderr:103650K .......... .......... .......... .......... .......... 61% 7.31M 8s 2020-02-05T17:01:16.629 INFO:tasks.workunit.client.0.smithi008.stderr:103700K .......... .......... .......... .......... .......... 61% 8.22M 8s 2020-02-05T17:01:16.635 INFO:tasks.workunit.client.0.smithi008.stderr:103750K .......... .......... .......... .......... .......... 61% 8.19M 8s 2020-02-05T17:01:16.641 INFO:tasks.workunit.client.0.smithi008.stderr:103800K .......... .......... .......... .......... .......... 61% 8.14M 8s 2020-02-05T17:01:16.648 INFO:tasks.workunit.client.0.smithi008.stderr:103850K .......... .......... .......... .......... .......... 61% 7.27M 8s 2020-02-05T17:01:16.654 INFO:tasks.workunit.client.0.smithi008.stderr:103900K .......... .......... .......... .......... .......... 61% 8.65M 8s 2020-02-05T17:01:16.659 INFO:tasks.workunit.client.0.smithi008.stderr:103950K .......... .......... .......... .......... .......... 61% 8.43M 8s 2020-02-05T17:01:16.665 INFO:tasks.workunit.client.0.smithi008.stderr:104000K .......... .......... .......... .......... .......... 61% 8.05M 8s 2020-02-05T17:01:16.672 INFO:tasks.workunit.client.0.smithi008.stderr:104050K .......... .......... .......... .......... .......... 61% 7.15M 8s 2020-02-05T17:01:16.681 INFO:tasks.workunit.client.0.smithi008.stderr:104100K .......... .......... .......... .......... .......... 61% 5.47M 8s 2020-02-05T17:01:16.686 INFO:tasks.workunit.client.0.smithi008.stderr:104150K .......... .......... .......... .......... .......... 61% 9.25M 8s 2020-02-05T17:01:16.691 INFO:tasks.workunit.client.0.smithi008.stderr:104200K .......... .......... .......... .......... .......... 61% 9.34M 8s 2020-02-05T17:01:16.696 INFO:tasks.workunit.client.0.smithi008.stderr:104250K .......... .......... .......... .......... .......... 61% 10.5M 8s 2020-02-05T17:01:16.702 INFO:tasks.workunit.client.0.smithi008.stderr:104300K .......... .......... .......... .......... .......... 61% 8.74M 8s 2020-02-05T17:01:16.711 INFO:tasks.workunit.client.0.smithi008.stderr:104350K .......... .......... .......... .......... .......... 61% 7.48M 8s 2020-02-05T17:01:16.714 INFO:tasks.workunit.client.0.smithi008.stderr:104400K .......... .......... .......... .......... .......... 61% 8.45M 8s 2020-02-05T17:01:16.721 INFO:tasks.workunit.client.0.smithi008.stderr:104450K .......... .......... .......... .......... .......... 61% 7.24M 8s 2020-02-05T17:01:16.727 INFO:tasks.workunit.client.0.smithi008.stderr:104500K .......... .......... .......... .......... .......... 62% 8.09M 8s 2020-02-05T17:01:16.733 INFO:tasks.workunit.client.0.smithi008.stderr:104550K .......... .......... .......... .......... .......... 62% 8.07M 8s 2020-02-05T17:01:16.739 INFO:tasks.workunit.client.0.smithi008.stderr:104600K .......... .......... .......... .......... .......... 62% 7.93M 8s 2020-02-05T17:01:16.746 INFO:tasks.workunit.client.0.smithi008.stderr:104650K .......... .......... .......... .......... .......... 62% 6.86M 8s 2020-02-05T17:01:16.753 INFO:tasks.workunit.client.0.smithi008.stderr:104700K .......... .......... .......... .......... .......... 62% 7.43M 8s 2020-02-05T17:01:16.759 INFO:tasks.workunit.client.0.smithi008.stderr:104750K .......... .......... .......... .......... .......... 62% 7.99M 8s 2020-02-05T17:01:16.764 INFO:tasks.workunit.client.0.smithi008.stderr:104800K .......... .......... .......... .......... .......... 62% 9.47M 8s 2020-02-05T17:01:16.770 INFO:tasks.workunit.client.0.smithi008.stderr:104850K .......... .......... .......... .......... .......... 62% 8.42M 8s 2020-02-05T17:01:16.774 INFO:tasks.workunit.client.0.smithi008.stderr:104900K .......... .......... .......... .......... .......... 62% 11.2M 8s 2020-02-05T17:01:16.780 INFO:tasks.workunit.client.0.smithi008.stderr:104950K .......... .......... .......... .......... .......... 62% 9.64M 8s 2020-02-05T17:01:16.785 INFO:tasks.workunit.client.0.smithi008.stderr:105000K .......... .......... .......... .......... .......... 62% 8.64M 8s 2020-02-05T17:01:16.793 INFO:tasks.workunit.client.0.smithi008.stderr:105050K .......... .......... .......... .......... .......... 62% 6.95M 8s 2020-02-05T17:01:16.799 INFO:tasks.workunit.client.0.smithi008.stderr:105100K .......... .......... .......... .......... .......... 62% 7.61M 8s 2020-02-05T17:01:16.807 INFO:tasks.workunit.client.0.smithi008.stderr:105150K .......... .......... .......... .......... .......... 62% 6.17M 8s 2020-02-05T17:01:16.815 INFO:tasks.workunit.client.0.smithi008.stderr:105200K .......... .......... .......... .......... .......... 62% 6.22M 8s 2020-02-05T17:01:16.824 INFO:tasks.workunit.client.0.smithi008.stderr:105250K .......... .......... .......... .......... .......... 62% 4.95M 8s 2020-02-05T17:01:16.833 INFO:tasks.workunit.client.0.smithi008.stderr:105300K .......... .......... .......... .......... .......... 62% 5.57M 8s 2020-02-05T17:01:16.842 INFO:tasks.workunit.client.0.smithi008.stderr:105350K .......... .......... .......... .......... .......... 62% 5.84M 8s 2020-02-05T17:01:16.850 INFO:tasks.workunit.client.0.smithi008.stderr:105400K .......... .......... .......... .......... .......... 62% 5.64M 8s 2020-02-05T17:01:16.860 INFO:tasks.workunit.client.0.smithi008.stderr:105450K .......... .......... .......... .......... .......... 62% 4.91M 8s 2020-02-05T17:01:16.869 INFO:tasks.workunit.client.0.smithi008.stderr:105500K .......... .......... .......... .......... .......... 62% 5.46M 8s 2020-02-05T17:01:16.878 INFO:tasks.workunit.client.0.smithi008.stderr:105550K .......... .......... .......... .......... .......... 62% 5.38M 8s 2020-02-05T17:01:16.888 INFO:tasks.workunit.client.0.smithi008.stderr:105600K .......... .......... .......... .......... .......... 62% 5.24M 8s 2020-02-05T17:01:16.900 INFO:tasks.workunit.client.0.smithi008.stderr:105650K .......... .......... .......... .......... .......... 62% 4.28M 8s 2020-02-05T17:01:16.909 INFO:tasks.workunit.client.0.smithi008.stderr:105700K .......... .......... .......... .......... .......... 62% 4.98M 8s 2020-02-05T17:01:16.919 INFO:tasks.workunit.client.0.smithi008.stderr:105750K .......... .......... .......... .......... .......... 62% 5.12M 8s 2020-02-05T17:01:16.928 INFO:tasks.workunit.client.0.smithi008.stderr:105800K .......... .......... .......... .......... .......... 62% 4.98M 8s 2020-02-05T17:01:16.940 INFO:tasks.workunit.client.0.smithi008.stderr:105850K .......... .......... .......... .......... .......... 62% 4.11M 8s 2020-02-05T17:01:16.950 INFO:tasks.workunit.client.0.smithi008.stderr:105900K .......... .......... .......... .......... .......... 62% 4.74M 8s 2020-02-05T17:01:16.965 INFO:tasks.workunit.client.0.smithi008.stderr:105950K .......... .......... .......... .......... .......... 62% 3.27M 8s 2020-02-05T17:01:16.971 INFO:tasks.workunit.client.0.smithi008.stderr:106000K .......... .......... .......... .......... .......... 62% 7.52M 8s 2020-02-05T17:01:16.979 INFO:tasks.workunit.client.0.smithi008.stderr:106050K .......... .......... .......... .......... .......... 62% 5.98M 8s 2020-02-05T17:01:16.984 INFO:tasks.workunit.client.0.smithi008.stderr:106100K .......... .......... .......... .......... .......... 62% 10.3M 8s 2020-02-05T17:01:16.989 INFO:tasks.workunit.client.0.smithi008.stderr:106150K .......... .......... .......... .......... .......... 63% 10.9M 8s 2020-02-05T17:01:16.994 INFO:tasks.workunit.client.0.smithi008.stderr:106200K .......... .......... .......... .......... .......... 63% 10.0M 8s 2020-02-05T17:01:16.999 INFO:tasks.workunit.client.0.smithi008.stderr:106250K .......... .......... .......... .......... .......... 63% 8.64M 8s 2020-02-05T17:01:17.004 INFO:tasks.workunit.client.0.smithi008.stderr:106300K .......... .......... .......... .......... .......... 63% 9.80M 8s 2020-02-05T17:01:17.010 INFO:tasks.workunit.client.0.smithi008.stderr:106350K .......... .......... .......... .......... .......... 63% 9.21M 8s 2020-02-05T17:01:17.014 INFO:tasks.workunit.client.0.smithi008.stderr:106400K .......... .......... .......... .......... .......... 63% 9.82M 8s 2020-02-05T17:01:17.020 INFO:tasks.workunit.client.0.smithi008.stderr:106450K .......... .......... .......... .......... .......... 63% 8.20M 8s 2020-02-05T17:01:17.026 INFO:tasks.workunit.client.0.smithi008.stderr:106500K .......... .......... .......... .......... .......... 63% 8.37M 8s 2020-02-05T17:01:17.034 INFO:tasks.workunit.client.0.smithi008.stderr:106550K .......... .......... .......... .......... .......... 63% 8.06M 8s 2020-02-05T17:01:17.035 INFO:tasks.workunit.client.0.smithi008.stderr:106600K .......... .......... .......... .......... .......... 63% 13.8M 7s 2020-02-05T17:01:17.040 INFO:tasks.workunit.client.0.smithi008.stderr:106650K .......... .......... .......... .......... .......... 63% 12.8M 7s 2020-02-05T17:01:17.043 INFO:tasks.workunit.client.0.smithi008.stderr:106700K .......... .......... .......... .......... .......... 63% 13.0M 7s 2020-02-05T17:01:17.047 INFO:tasks.workunit.client.0.smithi008.stderr:106750K .......... .......... .......... .......... .......... 63% 13.8M 7s 2020-02-05T17:01:17.051 INFO:tasks.workunit.client.0.smithi008.stderr:106800K .......... .......... .......... .......... .......... 63% 11.4M 7s 2020-02-05T17:01:17.056 INFO:tasks.workunit.client.0.smithi008.stderr:106850K .......... .......... .......... .......... .......... 63% 10.2M 7s 2020-02-05T17:01:17.060 INFO:tasks.workunit.client.0.smithi008.stderr:106900K .......... .......... .......... .......... .......... 63% 11.2M 7s 2020-02-05T17:01:17.065 INFO:tasks.workunit.client.0.smithi008.stderr:106950K .......... .......... .......... .......... .......... 63% 10.3M 7s 2020-02-05T17:01:17.069 INFO:tasks.workunit.client.0.smithi008.stderr:107000K .......... .......... .......... .......... .......... 63% 11.3M 7s 2020-02-05T17:01:17.074 INFO:tasks.workunit.client.0.smithi008.stderr:107050K .......... .......... .......... .......... .......... 63% 9.72M 7s 2020-02-05T17:01:17.079 INFO:tasks.workunit.client.0.smithi008.stderr:107100K .......... .......... .......... .......... .......... 63% 11.1M 7s 2020-02-05T17:01:17.083 INFO:tasks.workunit.client.0.smithi008.stderr:107150K .......... .......... .......... .......... .......... 63% 11.3M 7s 2020-02-05T17:01:17.088 INFO:tasks.workunit.client.0.smithi008.stderr:107200K .......... .......... .......... .......... .......... 63% 10.2M 7s 2020-02-05T17:01:17.093 INFO:tasks.workunit.client.0.smithi008.stderr:107250K .......... .......... .......... .......... .......... 63% 8.88M 7s 2020-02-05T17:01:17.099 INFO:tasks.workunit.client.0.smithi008.stderr:107300K .......... .......... .......... .......... .......... 63% 8.68M 7s 2020-02-05T17:01:17.103 INFO:tasks.workunit.client.0.smithi008.stderr:107350K .......... .......... .......... .......... .......... 63% 12.0M 7s 2020-02-05T17:01:17.108 INFO:tasks.workunit.client.0.smithi008.stderr:107400K .......... .......... .......... .......... .......... 63% 9.62M 7s 2020-02-05T17:01:17.118 INFO:tasks.workunit.client.0.smithi008.stderr:107450K .......... .......... .......... .......... .......... 63% 8.02M 7s 2020-02-05T17:01:17.119 INFO:tasks.workunit.client.0.smithi008.stderr:107500K .......... .......... .......... .......... .......... 63% 9.41M 7s 2020-02-05T17:01:17.124 INFO:tasks.workunit.client.0.smithi008.stderr:107550K .......... .......... .......... .......... .......... 63% 9.59M 7s 2020-02-05T17:01:17.129 INFO:tasks.workunit.client.0.smithi008.stderr:107600K .......... .......... .......... .......... .......... 63% 10.8M 7s 2020-02-05T17:01:17.135 INFO:tasks.workunit.client.0.smithi008.stderr:107650K .......... .......... .......... .......... .......... 63% 8.38M 7s 2020-02-05T17:01:17.140 INFO:tasks.workunit.client.0.smithi008.stderr:107700K .......... .......... .......... .......... .......... 63% 9.04M 7s 2020-02-05T17:01:17.146 INFO:tasks.workunit.client.0.smithi008.stderr:107750K .......... .......... .......... .......... .......... 63% 8.84M 7s 2020-02-05T17:01:17.152 INFO:tasks.workunit.client.0.smithi008.stderr:107800K .......... .......... .......... .......... .......... 63% 8.05M 7s 2020-02-05T17:01:17.159 INFO:tasks.workunit.client.0.smithi008.stderr:107850K .......... .......... .......... .......... .......... 64% 6.67M 7s 2020-02-05T17:01:17.165 INFO:tasks.workunit.client.0.smithi008.stderr:107900K .......... .......... .......... .......... .......... 64% 8.56M 7s 2020-02-05T17:01:17.171 INFO:tasks.workunit.client.0.smithi008.stderr:107950K .......... .......... .......... .......... .......... 64% 8.07M 7s 2020-02-05T17:01:17.177 INFO:tasks.workunit.client.0.smithi008.stderr:108000K .......... .......... .......... .......... .......... 64% 8.63M 7s 2020-02-05T17:01:17.184 INFO:tasks.workunit.client.0.smithi008.stderr:108050K .......... .......... .......... .......... .......... 64% 6.97M 7s 2020-02-05T17:01:17.192 INFO:tasks.workunit.client.0.smithi008.stderr:108100K .......... .......... .......... .......... .......... 64% 7.10M 7s 2020-02-05T17:01:17.197 INFO:tasks.workunit.client.0.smithi008.stderr:108150K .......... .......... .......... .......... .......... 64% 7.56M 7s 2020-02-05T17:01:17.206 INFO:tasks.workunit.client.0.smithi008.stderr:108200K .......... .......... .......... .......... .......... 64% 5.20M 7s 2020-02-05T17:01:17.213 INFO:tasks.workunit.client.0.smithi008.stderr:108250K .......... .......... .......... .......... .......... 64% 7.76M 7s 2020-02-05T17:01:17.218 INFO:tasks.workunit.client.0.smithi008.stderr:108300K .......... .......... .......... .......... .......... 64% 9.54M 7s 2020-02-05T17:01:17.223 INFO:tasks.workunit.client.0.smithi008.stderr:108350K .......... .......... .......... .......... .......... 64% 9.33M 7s 2020-02-05T17:01:17.228 INFO:tasks.workunit.client.0.smithi008.stderr:108400K .......... .......... .......... .......... .......... 64% 8.94M 7s 2020-02-05T17:01:17.235 INFO:tasks.workunit.client.0.smithi008.stderr:108450K .......... .......... .......... .......... .......... 64% 7.60M 7s 2020-02-05T17:01:17.241 INFO:tasks.workunit.client.0.smithi008.stderr:108500K .......... .......... .......... .......... .......... 64% 8.21M 7s 2020-02-05T17:01:17.247 INFO:tasks.workunit.client.0.smithi008.stderr:108550K .......... .......... .......... .......... .......... 64% 8.11M 7s 2020-02-05T17:01:17.254 INFO:tasks.workunit.client.0.smithi008.stderr:108600K .......... .......... .......... .......... .......... 64% 7.17M 7s 2020-02-05T17:01:17.261 INFO:tasks.workunit.client.0.smithi008.stderr:108650K .......... .......... .......... .......... .......... 64% 6.38M 7s 2020-02-05T17:01:17.268 INFO:tasks.workunit.client.0.smithi008.stderr:108700K .......... .......... .......... .......... .......... 64% 6.87M 7s 2020-02-05T17:01:17.275 INFO:tasks.workunit.client.0.smithi008.stderr:108750K .......... .......... .......... .......... .......... 64% 7.60M 7s 2020-02-05T17:01:17.281 INFO:tasks.workunit.client.0.smithi008.stderr:108800K .......... .......... .......... .......... .......... 64% 8.05M 7s 2020-02-05T17:01:17.288 INFO:tasks.workunit.client.0.smithi008.stderr:108850K .......... .......... .......... .......... .......... 64% 6.84M 7s 2020-02-05T17:01:17.294 INFO:tasks.workunit.client.0.smithi008.stderr:108900K .......... .......... .......... .......... .......... 64% 7.73M 7s 2020-02-05T17:01:17.301 INFO:tasks.workunit.client.0.smithi008.stderr:108950K .......... .......... .......... .......... .......... 64% 6.97M 7s 2020-02-05T17:01:17.311 INFO:tasks.workunit.client.0.smithi008.stderr:109000K .......... .......... .......... .......... .......... 64% 7.38M 7s 2020-02-05T17:01:17.315 INFO:tasks.workunit.client.0.smithi008.stderr:109050K .......... .......... .......... .......... .......... 64% 6.75M 7s 2020-02-05T17:01:17.322 INFO:tasks.workunit.client.0.smithi008.stderr:109100K .......... .......... .......... .......... .......... 64% 7.95M 7s 2020-02-05T17:01:17.328 INFO:tasks.workunit.client.0.smithi008.stderr:109150K .......... .......... .......... .......... .......... 64% 7.12M 7s 2020-02-05T17:01:17.335 INFO:tasks.workunit.client.0.smithi008.stderr:109200K .......... .......... .......... .......... .......... 64% 6.85M 7s 2020-02-05T17:01:17.344 INFO:tasks.workunit.client.0.smithi008.stderr:109250K .......... .......... .......... .......... .......... 64% 6.07M 7s 2020-02-05T17:01:17.351 INFO:tasks.workunit.client.0.smithi008.stderr:109300K .......... .......... .......... .......... .......... 64% 6.87M 7s 2020-02-05T17:01:17.359 INFO:tasks.workunit.client.0.smithi008.stderr:109350K .......... .......... .......... .......... .......... 64% 6.21M 7s 2020-02-05T17:01:17.367 INFO:tasks.workunit.client.0.smithi008.stderr:109400K .......... .......... .......... .......... .......... 64% 6.14M 7s 2020-02-05T17:01:17.375 INFO:tasks.workunit.client.0.smithi008.stderr:109450K .......... .......... .......... .......... .......... 64% 5.70M 7s 2020-02-05T17:01:17.381 INFO:tasks.workunit.client.0.smithi008.stderr:109500K .......... .......... .......... .......... .......... 64% 7.71M 7s 2020-02-05T17:01:17.388 INFO:tasks.workunit.client.0.smithi008.stderr:109550K .......... .......... .......... .......... .......... 65% 7.46M 7s 2020-02-05T17:01:17.395 INFO:tasks.workunit.client.0.smithi008.stderr:109600K .......... .......... .......... .......... .......... 65% 7.33M 7s 2020-02-05T17:01:17.402 INFO:tasks.workunit.client.0.smithi008.stderr:109650K .......... .......... .......... .......... .......... 65% 6.45M 7s 2020-02-05T17:01:17.409 INFO:tasks.workunit.client.0.smithi008.stderr:109700K .......... .......... .......... .......... .......... 65% 6.85M 7s 2020-02-05T17:01:17.416 INFO:tasks.workunit.client.0.smithi008.stderr:109750K .......... .......... .......... .......... .......... 65% 7.40M 7s 2020-02-05T17:01:17.423 INFO:tasks.workunit.client.0.smithi008.stderr:109800K .......... .......... .......... .......... .......... 65% 6.71M 7s 2020-02-05T17:01:17.430 INFO:tasks.workunit.client.0.smithi008.stderr:109850K .......... .......... .......... .......... .......... 65% 6.65M 7s 2020-02-05T17:01:17.438 INFO:tasks.workunit.client.0.smithi008.stderr:109900K .......... .......... .......... .......... .......... 65% 6.93M 7s 2020-02-05T17:01:17.446 INFO:tasks.workunit.client.0.smithi008.stderr:109950K .......... .......... .......... .......... .......... 65% 7.00M 7s 2020-02-05T17:01:17.452 INFO:tasks.workunit.client.0.smithi008.stderr:110000K .......... .......... .......... .......... .......... 65% 6.77M 7s 2020-02-05T17:01:17.464 INFO:tasks.workunit.client.0.smithi008.stderr:110050K .......... .......... .......... .......... .......... 65% 3.78M 7s 2020-02-05T17:01:17.470 INFO:tasks.workunit.client.0.smithi008.stderr:110100K .......... .......... .......... .......... .......... 65% 10.2M 7s 2020-02-05T17:01:17.473 INFO:tasks.workunit.client.0.smithi008.stderr:110150K .......... .......... .......... .......... .......... 65% 12.2M 7s 2020-02-05T17:01:17.478 INFO:tasks.workunit.client.0.smithi008.stderr:110200K .......... .......... .......... .......... .......... 65% 9.72M 7s 2020-02-05T17:01:17.484 INFO:tasks.workunit.client.0.smithi008.stderr:110250K .......... .......... .......... .......... .......... 65% 8.33M 7s 2020-02-05T17:01:17.488 INFO:tasks.workunit.client.0.smithi008.stderr:110300K .......... .......... .......... .......... .......... 65% 11.4M 7s 2020-02-05T17:01:17.493 INFO:tasks.workunit.client.0.smithi008.stderr:110350K .......... .......... .......... .......... .......... 65% 11.0M 7s 2020-02-05T17:01:17.498 INFO:tasks.workunit.client.0.smithi008.stderr:110400K .......... .......... .......... .......... .......... 65% 10.7M 7s 2020-02-05T17:01:17.502 INFO:tasks.workunit.client.0.smithi008.stderr:110450K .......... .......... .......... .......... .......... 65% 9.70M 7s 2020-02-05T17:01:17.506 INFO:tasks.workunit.client.0.smithi008.stderr:110500K .......... .......... .......... .......... .......... 65% 11.4M 7s 2020-02-05T17:01:17.511 INFO:tasks.workunit.client.0.smithi008.stderr:110550K .......... .......... .......... .......... .......... 65% 11.1M 7s 2020-02-05T17:01:17.516 INFO:tasks.workunit.client.0.smithi008.stderr:110600K .......... .......... .......... .......... .......... 65% 9.71M 7s 2020-02-05T17:01:17.522 INFO:tasks.workunit.client.0.smithi008.stderr:110650K .......... .......... .......... .......... .......... 65% 8.24M 7s 2020-02-05T17:01:17.527 INFO:tasks.workunit.client.0.smithi008.stderr:110700K .......... .......... .......... .......... .......... 65% 9.24M 7s 2020-02-05T17:01:17.530 INFO:tasks.workunit.client.0.smithi008.stderr:110750K .......... .......... .......... .......... .......... 65% 14.0M 7s 2020-02-05T17:01:17.533 INFO:tasks.workunit.client.0.smithi008.stderr:110800K .......... .......... .......... .......... .......... 65% 15.4M 7s 2020-02-05T17:01:17.537 INFO:tasks.workunit.client.0.smithi008.stderr:110850K .......... .......... .......... .......... .......... 65% 14.2M 7s 2020-02-05T17:01:17.540 INFO:tasks.workunit.client.0.smithi008.stderr:110900K .......... .......... .......... .......... .......... 65% 16.5M 7s 2020-02-05T17:01:17.543 INFO:tasks.workunit.client.0.smithi008.stderr:110950K .......... .......... .......... .......... .......... 65% 13.8M 7s 2020-02-05T17:01:17.546 INFO:tasks.workunit.client.0.smithi008.stderr:111000K .......... .......... .......... .......... .......... 65% 20.3M 7s 2020-02-05T17:01:17.550 INFO:tasks.workunit.client.0.smithi008.stderr:111050K .......... .......... .......... .......... .......... 65% 14.1M 7s 2020-02-05T17:01:17.552 INFO:tasks.workunit.client.0.smithi008.stderr:111100K .......... .......... .......... .......... .......... 65% 15.4M 7s 2020-02-05T17:01:17.556 INFO:tasks.workunit.client.0.smithi008.stderr:111150K .......... .......... .......... .......... .......... 65% 14.6M 7s 2020-02-05T17:01:17.560 INFO:tasks.workunit.client.0.smithi008.stderr:111200K .......... .......... .......... .......... .......... 65% 13.0M 7s 2020-02-05T17:01:17.564 INFO:tasks.workunit.client.0.smithi008.stderr:111250K .......... .......... .......... .......... .......... 66% 10.6M 7s 2020-02-05T17:01:17.569 INFO:tasks.workunit.client.0.smithi008.stderr:111300K .......... .......... .......... .......... .......... 66% 10.9M 7s 2020-02-05T17:01:17.574 INFO:tasks.workunit.client.0.smithi008.stderr:111350K .......... .......... .......... .......... .......... 66% 10.3M 7s 2020-02-05T17:01:17.579 INFO:tasks.workunit.client.0.smithi008.stderr:111400K .......... .......... .......... .......... .......... 66% 8.55M 7s 2020-02-05T17:01:17.585 INFO:tasks.workunit.client.0.smithi008.stderr:111450K .......... .......... .......... .......... .......... 66% 8.68M 7s 2020-02-05T17:01:17.590 INFO:tasks.workunit.client.0.smithi008.stderr:111500K .......... .......... .......... .......... .......... 66% 9.18M 7s 2020-02-05T17:01:17.596 INFO:tasks.workunit.client.0.smithi008.stderr:111550K .......... .......... .......... .......... .......... 66% 9.05M 7s 2020-02-05T17:01:17.602 INFO:tasks.workunit.client.0.smithi008.stderr:111600K .......... .......... .......... .......... .......... 66% 8.81M 7s 2020-02-05T17:01:17.608 INFO:tasks.workunit.client.0.smithi008.stderr:111650K .......... .......... .......... .......... .......... 66% 7.49M 7s 2020-02-05T17:01:17.614 INFO:tasks.workunit.client.0.smithi008.stderr:111700K .......... .......... .......... .......... .......... 66% 8.02M 7s 2020-02-05T17:01:17.620 INFO:tasks.workunit.client.0.smithi008.stderr:111750K .......... .......... .......... .......... .......... 66% 8.18M 7s 2020-02-05T17:01:17.626 INFO:tasks.workunit.client.0.smithi008.stderr:111800K .......... .......... .......... .......... .......... 66% 8.02M 7s 2020-02-05T17:01:17.633 INFO:tasks.workunit.client.0.smithi008.stderr:111850K .......... .......... .......... .......... .......... 66% 7.03M 7s 2020-02-05T17:01:17.650 INFO:tasks.workunit.client.0.smithi008.stderr:111900K .......... .......... .......... .......... .......... 66% 7.78M 7s 2020-02-05T17:01:17.650 INFO:tasks.workunit.client.0.smithi008.stderr:111950K .......... .......... .......... .......... .......... 66% 8.06M 7s 2020-02-05T17:01:17.653 INFO:tasks.workunit.client.0.smithi008.stderr:112000K .......... .......... .......... .......... .......... 66% 6.78M 7s 2020-02-05T17:01:17.662 INFO:tasks.workunit.client.0.smithi008.stderr:112050K .......... .......... .......... .......... .......... 66% 5.36M 7s 2020-02-05T17:01:17.670 INFO:tasks.workunit.client.0.smithi008.stderr:112100K .......... .......... .......... .......... .......... 66% 6.18M 7s 2020-02-05T17:01:17.678 INFO:tasks.workunit.client.0.smithi008.stderr:112150K .......... .......... .......... .......... .......... 66% 6.25M 7s 2020-02-05T17:01:17.685 INFO:tasks.workunit.client.0.smithi008.stderr:112200K .......... .......... .......... .......... .......... 66% 6.84M 7s 2020-02-05T17:01:17.694 INFO:tasks.workunit.client.0.smithi008.stderr:112250K .......... .......... .......... .......... .......... 66% 6.46M 7s 2020-02-05T17:01:17.701 INFO:tasks.workunit.client.0.smithi008.stderr:112300K .......... .......... .......... .......... .......... 66% 5.39M 7s 2020-02-05T17:01:17.706 INFO:tasks.workunit.client.0.smithi008.stderr:112350K .......... .......... .......... .......... .......... 66% 9.72M 7s 2020-02-05T17:01:17.711 INFO:tasks.workunit.client.0.smithi008.stderr:112400K .......... .......... .......... .......... .......... 66% 9.92M 7s 2020-02-05T17:01:17.717 INFO:tasks.workunit.client.0.smithi008.stderr:112450K .......... .......... .......... .......... .......... 66% 8.63M 7s 2020-02-05T17:01:17.722 INFO:tasks.workunit.client.0.smithi008.stderr:112500K .......... .......... .......... .......... .......... 66% 9.15M 7s 2020-02-05T17:01:17.728 INFO:tasks.workunit.client.0.smithi008.stderr:112550K .......... .......... .......... .......... .......... 66% 7.87M 7s 2020-02-05T17:01:17.734 INFO:tasks.workunit.client.0.smithi008.stderr:112600K .......... .......... .......... .......... .......... 66% 8.32M 7s 2020-02-05T17:01:17.742 INFO:tasks.workunit.client.0.smithi008.stderr:112650K .......... .......... .......... .......... .......... 66% 6.54M 7s 2020-02-05T17:01:17.748 INFO:tasks.workunit.client.0.smithi008.stderr:112700K .......... .......... .......... .......... .......... 66% 7.50M 7s 2020-02-05T17:01:17.763 INFO:tasks.workunit.client.0.smithi008.stderr:112750K .......... .......... .......... .......... .......... 66% 7.13M 7s 2020-02-05T17:01:17.768 INFO:tasks.workunit.client.0.smithi008.stderr:112800K .......... .......... .......... .......... .......... 66% 6.82M 7s 2020-02-05T17:01:17.771 INFO:tasks.workunit.client.0.smithi008.stderr:112850K .......... .......... .......... .......... .......... 66% 5.63M 7s 2020-02-05T17:01:17.779 INFO:tasks.workunit.client.0.smithi008.stderr:112900K .......... .......... .......... .......... .......... 67% 6.88M 7s 2020-02-05T17:01:17.786 INFO:tasks.workunit.client.0.smithi008.stderr:112950K .......... .......... .......... .......... .......... 67% 6.64M 7s 2020-02-05T17:01:17.793 INFO:tasks.workunit.client.0.smithi008.stderr:113000K .......... .......... .......... .......... .......... 67% 6.58M 7s 2020-02-05T17:01:17.802 INFO:tasks.workunit.client.0.smithi008.stderr:113050K .......... .......... .......... .......... .......... 67% 5.33M 7s 2020-02-05T17:01:17.811 INFO:tasks.workunit.client.0.smithi008.stderr:113100K .......... .......... .......... .......... .......... 67% 6.63M 7s 2020-02-05T17:01:17.817 INFO:tasks.workunit.client.0.smithi008.stderr:113150K .......... .......... .......... .......... .......... 67% 6.67M 7s 2020-02-05T17:01:17.824 INFO:tasks.workunit.client.0.smithi008.stderr:113200K .......... .......... .......... .......... .......... 67% 7.08M 7s 2020-02-05T17:01:17.832 INFO:tasks.workunit.client.0.smithi008.stderr:113250K .......... .......... .......... .......... .......... 67% 5.90M 7s 2020-02-05T17:01:17.839 INFO:tasks.workunit.client.0.smithi008.stderr:113300K .......... .......... .......... .......... .......... 67% 6.95M 7s 2020-02-05T17:01:17.845 INFO:tasks.workunit.client.0.smithi008.stderr:113350K .......... .......... .......... .......... .......... 67% 7.95M 7s 2020-02-05T17:01:17.856 INFO:tasks.workunit.client.0.smithi008.stderr:113400K .......... .......... .......... .......... .......... 67% 7.92M 7s 2020-02-05T17:01:17.858 INFO:tasks.workunit.client.0.smithi008.stderr:113450K .......... .......... .......... .......... .......... 67% 6.85M 7s 2020-02-05T17:01:17.864 INFO:tasks.workunit.client.0.smithi008.stderr:113500K .......... .......... .......... .......... .......... 67% 8.27M 7s 2020-02-05T17:01:17.870 INFO:tasks.workunit.client.0.smithi008.stderr:113550K .......... .......... .......... .......... .......... 67% 7.93M 7s 2020-02-05T17:01:17.877 INFO:tasks.workunit.client.0.smithi008.stderr:113600K .......... .......... .......... .......... .......... 67% 7.45M 7s 2020-02-05T17:01:17.885 INFO:tasks.workunit.client.0.smithi008.stderr:113650K .......... .......... .......... .......... .......... 67% 6.42M 7s 2020-02-05T17:01:17.891 INFO:tasks.workunit.client.0.smithi008.stderr:113700K .......... .......... .......... .......... .......... 67% 7.22M 7s 2020-02-05T17:01:17.901 INFO:tasks.workunit.client.0.smithi008.stderr:113750K .......... .......... .......... .......... .......... 67% 7.28M 7s 2020-02-05T17:01:17.906 INFO:tasks.workunit.client.0.smithi008.stderr:113800K .......... .......... .......... .......... .......... 67% 5.98M 7s 2020-02-05T17:01:17.916 INFO:tasks.workunit.client.0.smithi008.stderr:113850K .......... .......... .......... .......... .......... 67% 5.12M 7s 2020-02-05T17:01:17.924 INFO:tasks.workunit.client.0.smithi008.stderr:113900K .......... .......... .......... .......... .......... 67% 5.81M 7s 2020-02-05T17:01:17.931 INFO:tasks.workunit.client.0.smithi008.stderr:113950K .......... .......... .......... .......... .......... 67% 6.64M 7s 2020-02-05T17:01:17.940 INFO:tasks.workunit.client.0.smithi008.stderr:114000K .......... .......... .......... .......... .......... 67% 6.25M 7s 2020-02-05T17:01:17.950 INFO:tasks.workunit.client.0.smithi008.stderr:114050K .......... .......... .......... .......... .......... 67% 4.94M 7s 2020-02-05T17:01:17.958 INFO:tasks.workunit.client.0.smithi008.stderr:114100K .......... .......... .......... .......... .......... 67% 5.94M 7s 2020-02-05T17:01:17.973 INFO:tasks.workunit.client.0.smithi008.stderr:114150K .......... .......... .......... .......... .......... 67% 3.15M 7s 2020-02-05T17:01:17.978 INFO:tasks.workunit.client.0.smithi008.stderr:114200K .......... .......... .......... .......... .......... 67% 9.99M 7s 2020-02-05T17:01:17.983 INFO:tasks.workunit.client.0.smithi008.stderr:114250K .......... .......... .......... .......... .......... 67% 9.46M 7s 2020-02-05T17:01:17.987 INFO:tasks.workunit.client.0.smithi008.stderr:114300K .......... .......... .......... .......... .......... 67% 10.0M 7s 2020-02-05T17:01:17.992 INFO:tasks.workunit.client.0.smithi008.stderr:114350K .......... .......... .......... .......... .......... 67% 9.86M 7s 2020-02-05T17:01:17.998 INFO:tasks.workunit.client.0.smithi008.stderr:114400K .......... .......... .......... .......... .......... 67% 9.08M 7s 2020-02-05T17:01:18.005 INFO:tasks.workunit.client.0.smithi008.stderr:114450K .......... .......... .......... .......... .......... 67% 7.19M 7s 2020-02-05T17:01:18.009 INFO:tasks.workunit.client.0.smithi008.stderr:114500K .......... .......... .......... .......... .......... 67% 10.3M 7s 2020-02-05T17:01:18.014 INFO:tasks.workunit.client.0.smithi008.stderr:114550K .......... .......... .......... .......... .......... 67% 11.1M 7s 2020-02-05T17:01:18.019 INFO:tasks.workunit.client.0.smithi008.stderr:114600K .......... .......... .......... .......... .......... 68% 9.75M 7s 2020-02-05T17:01:18.025 INFO:tasks.workunit.client.0.smithi008.stderr:114650K .......... .......... .......... .......... .......... 68% 8.32M 7s 2020-02-05T17:01:18.030 INFO:tasks.workunit.client.0.smithi008.stderr:114700K .......... .......... .......... .......... .......... 68% 9.66M 7s 2020-02-05T17:01:18.035 INFO:tasks.workunit.client.0.smithi008.stderr:114750K .......... .......... .......... .......... .......... 68% 10.1M 7s 2020-02-05T17:01:18.041 INFO:tasks.workunit.client.0.smithi008.stderr:114800K .......... .......... .......... .......... .......... 68% 11.9M 7s 2020-02-05T17:01:18.043 INFO:tasks.workunit.client.0.smithi008.stderr:114850K .......... .......... .......... .......... .......... 68% 9.85M 7s 2020-02-05T17:01:18.047 INFO:tasks.workunit.client.0.smithi008.stderr:114900K .......... .......... .......... .......... .......... 68% 13.8M 6s 2020-02-05T17:01:18.049 INFO:tasks.workunit.client.0.smithi008.stderr:114950K .......... .......... .......... .......... .......... 68% 17.8M 6s 2020-02-05T17:01:18.052 INFO:tasks.workunit.client.0.smithi008.stderr:115000K .......... .......... .......... .......... .......... 68% 17.8M 6s 2020-02-05T17:01:18.057 INFO:tasks.workunit.client.0.smithi008.stderr:115050K .......... .......... .......... .......... .......... 68% 10.8M 6s 2020-02-05T17:01:18.060 INFO:tasks.workunit.client.0.smithi008.stderr:115100K .......... .......... .......... .......... .......... 68% 13.1M 6s 2020-02-05T17:01:18.066 INFO:tasks.workunit.client.0.smithi008.stderr:115150K .......... .......... .......... .......... .......... 68% 11.0M 6s 2020-02-05T17:01:18.070 INFO:tasks.workunit.client.0.smithi008.stderr:115200K .......... .......... .......... .......... .......... 68% 9.57M 6s 2020-02-05T17:01:18.076 INFO:tasks.workunit.client.0.smithi008.stderr:115250K .......... .......... .......... .......... .......... 68% 8.27M 6s 2020-02-05T17:01:18.080 INFO:tasks.workunit.client.0.smithi008.stderr:115300K .......... .......... .......... .......... .......... 68% 11.3M 6s 2020-02-05T17:01:18.085 INFO:tasks.workunit.client.0.smithi008.stderr:115350K .......... .......... .......... .......... .......... 68% 11.0M 6s 2020-02-05T17:01:18.090 INFO:tasks.workunit.client.0.smithi008.stderr:115400K .......... .......... .......... .......... .......... 68% 11.0M 6s 2020-02-05T17:01:18.094 INFO:tasks.workunit.client.0.smithi008.stderr:115450K .......... .......... .......... .......... .......... 68% 9.09M 6s 2020-02-05T17:01:18.101 INFO:tasks.workunit.client.0.smithi008.stderr:115500K .......... .......... .......... .......... .......... 68% 8.11M 6s 2020-02-05T17:01:18.105 INFO:tasks.workunit.client.0.smithi008.stderr:115550K .......... .......... .......... .......... .......... 68% 11.1M 6s 2020-02-05T17:01:18.109 INFO:tasks.workunit.client.0.smithi008.stderr:115600K .......... .......... .......... .......... .......... 68% 11.1M 6s 2020-02-05T17:01:18.115 INFO:tasks.workunit.client.0.smithi008.stderr:115650K .......... .......... .......... .......... .......... 68% 9.41M 6s 2020-02-05T17:01:18.119 INFO:tasks.workunit.client.0.smithi008.stderr:115700K .......... .......... .......... .......... .......... 68% 11.3M 6s 2020-02-05T17:01:18.123 INFO:tasks.workunit.client.0.smithi008.stderr:115750K .......... .......... .......... .......... .......... 68% 11.1M 6s 2020-02-05T17:01:18.128 INFO:tasks.workunit.client.0.smithi008.stderr:115800K .......... .......... .......... .......... .......... 68% 11.3M 6s 2020-02-05T17:01:18.133 INFO:tasks.workunit.client.0.smithi008.stderr:115850K .......... .......... .......... .......... .......... 68% 8.77M 6s 2020-02-05T17:01:18.138 INFO:tasks.workunit.client.0.smithi008.stderr:115900K .......... .......... .......... .......... .......... 68% 10.5M 6s 2020-02-05T17:01:18.142 INFO:tasks.workunit.client.0.smithi008.stderr:115950K .......... .......... .......... .......... .......... 68% 10.8M 6s 2020-02-05T17:01:18.147 INFO:tasks.workunit.client.0.smithi008.stderr:116000K .......... .......... .......... .......... .......... 68% 10.3M 6s 2020-02-05T17:01:18.153 INFO:tasks.workunit.client.0.smithi008.stderr:116050K .......... .......... .......... .......... .......... 68% 8.79M 6s 2020-02-05T17:01:18.158 INFO:tasks.workunit.client.0.smithi008.stderr:116100K .......... .......... .......... .......... .......... 68% 10.2M 6s 2020-02-05T17:01:18.163 INFO:tasks.workunit.client.0.smithi008.stderr:116150K .......... .......... .......... .......... .......... 68% 10.2M 6s 2020-02-05T17:01:18.168 INFO:tasks.workunit.client.0.smithi008.stderr:116200K .......... .......... .......... .......... .......... 68% 9.55M 6s 2020-02-05T17:01:18.173 INFO:tasks.workunit.client.0.smithi008.stderr:116250K .......... .......... .......... .......... .......... 68% 8.67M 6s 2020-02-05T17:01:18.178 INFO:tasks.workunit.client.0.smithi008.stderr:116300K .......... .......... .......... .......... .......... 69% 11.2M 6s 2020-02-05T17:01:18.182 INFO:tasks.workunit.client.0.smithi008.stderr:116350K .......... .......... .......... .......... .......... 69% 11.5M 6s 2020-02-05T17:01:18.187 INFO:tasks.workunit.client.0.smithi008.stderr:116400K .......... .......... .......... .......... .......... 69% 8.46M 6s 2020-02-05T17:01:18.192 INFO:tasks.workunit.client.0.smithi008.stderr:116450K .......... .......... .......... .......... .......... 69% 10.8M 6s 2020-02-05T17:01:18.196 INFO:tasks.workunit.client.0.smithi008.stderr:116500K .......... .......... .......... .......... .......... 69% 12.3M 6s 2020-02-05T17:01:18.200 INFO:tasks.workunit.client.0.smithi008.stderr:116550K .......... .......... .......... .......... .......... 69% 12.1M 6s 2020-02-05T17:01:18.204 INFO:tasks.workunit.client.0.smithi008.stderr:116600K .......... .......... .......... .......... .......... 69% 11.7M 6s 2020-02-05T17:01:18.209 INFO:tasks.workunit.client.0.smithi008.stderr:116650K .......... .......... .......... .......... .......... 69% 9.92M 6s 2020-02-05T17:01:18.214 INFO:tasks.workunit.client.0.smithi008.stderr:116700K .......... .......... .......... .......... .......... 69% 10.1M 6s 2020-02-05T17:01:18.223 INFO:tasks.workunit.client.0.smithi008.stderr:116750K .......... .......... .......... .......... .......... 69% 8.44M 6s 2020-02-05T17:01:18.227 INFO:tasks.workunit.client.0.smithi008.stderr:116800K .......... .......... .......... .......... .......... 69% 7.90M 6s 2020-02-05T17:01:18.233 INFO:tasks.workunit.client.0.smithi008.stderr:116850K .......... .......... .......... .......... .......... 69% 6.88M 6s 2020-02-05T17:01:18.240 INFO:tasks.workunit.client.0.smithi008.stderr:116900K .......... .......... .......... .......... .......... 69% 7.23M 6s 2020-02-05T17:01:18.247 INFO:tasks.workunit.client.0.smithi008.stderr:116950K .......... .......... .......... .......... .......... 69% 7.41M 6s 2020-02-05T17:01:18.255 INFO:tasks.workunit.client.0.smithi008.stderr:117000K .......... .......... .......... .......... .......... 69% 6.69M 6s 2020-02-05T17:01:18.266 INFO:tasks.workunit.client.0.smithi008.stderr:117050K .......... .......... .......... .......... .......... 69% 5.41M 6s 2020-02-05T17:01:18.271 INFO:tasks.workunit.client.0.smithi008.stderr:117100K .......... .......... .......... .......... .......... 69% 6.06M 6s 2020-02-05T17:01:18.278 INFO:tasks.workunit.client.0.smithi008.stderr:117150K .......... .......... .......... .......... .......... 69% 7.98M 6s 2020-02-05T17:01:18.286 INFO:tasks.workunit.client.0.smithi008.stderr:117200K .......... .......... .......... .......... .......... 69% 6.22M 6s 2020-02-05T17:01:18.294 INFO:tasks.workunit.client.0.smithi008.stderr:117250K .......... .......... .......... .......... .......... 69% 5.52M 6s 2020-02-05T17:01:18.300 INFO:tasks.workunit.client.0.smithi008.stderr:117300K .......... .......... .......... .......... .......... 69% 8.26M 6s 2020-02-05T17:01:18.305 INFO:tasks.workunit.client.0.smithi008.stderr:117350K .......... .......... .......... .......... .......... 69% 8.83M 6s 2020-02-05T17:01:18.311 INFO:tasks.workunit.client.0.smithi008.stderr:117400K .......... .......... .......... .......... .......... 69% 8.74M 6s 2020-02-05T17:01:18.317 INFO:tasks.workunit.client.0.smithi008.stderr:117450K .......... .......... .......... .......... .......... 69% 7.39M 6s 2020-02-05T17:01:18.323 INFO:tasks.workunit.client.0.smithi008.stderr:117500K .......... .......... .......... .......... .......... 69% 9.14M 6s 2020-02-05T17:01:18.329 INFO:tasks.workunit.client.0.smithi008.stderr:117550K .......... .......... .......... .......... .......... 69% 8.42M 6s 2020-02-05T17:01:18.335 INFO:tasks.workunit.client.0.smithi008.stderr:117600K .......... .......... .......... .......... .......... 69% 7.90M 6s 2020-02-05T17:01:18.342 INFO:tasks.workunit.client.0.smithi008.stderr:117650K .......... .......... .......... .......... .......... 69% 7.15M 6s 2020-02-05T17:01:18.346 INFO:tasks.workunit.client.0.smithi008.stderr:117700K .......... .......... .......... .......... .......... 69% 10.1M 6s 2020-02-05T17:01:18.354 INFO:tasks.workunit.client.0.smithi008.stderr:117750K .......... .......... .......... .......... .......... 69% 10.3M 6s 2020-02-05T17:01:18.355 INFO:tasks.workunit.client.0.smithi008.stderr:117800K .......... .......... .......... .......... .......... 69% 14.3M 6s 2020-02-05T17:01:18.358 INFO:tasks.workunit.client.0.smithi008.stderr:117850K .......... .......... .......... .......... .......... 69% 12.6M 6s 2020-02-05T17:01:18.361 INFO:tasks.workunit.client.0.smithi008.stderr:117900K .......... .......... .......... .......... .......... 69% 14.7M 6s 2020-02-05T17:01:18.365 INFO:tasks.workunit.client.0.smithi008.stderr:117950K .......... .......... .......... .......... .......... 70% 14.4M 6s 2020-02-05T17:01:18.368 INFO:tasks.workunit.client.0.smithi008.stderr:118000K .......... .......... .......... .......... .......... 70% 14.6M 6s 2020-02-05T17:01:18.372 INFO:tasks.workunit.client.0.smithi008.stderr:118050K .......... .......... .......... .......... .......... 70% 12.6M 6s 2020-02-05T17:01:18.375 INFO:tasks.workunit.client.0.smithi008.stderr:118100K .......... .......... .......... .......... .......... 70% 14.7M 6s 2020-02-05T17:01:18.379 INFO:tasks.workunit.client.0.smithi008.stderr:118150K .......... .......... .......... .......... .......... 70% 14.7M 6s 2020-02-05T17:01:18.382 INFO:tasks.workunit.client.0.smithi008.stderr:118200K .......... .......... .......... .......... .......... 70% 14.6M 6s 2020-02-05T17:01:18.389 INFO:tasks.workunit.client.0.smithi008.stderr:118250K .......... .......... .......... .......... .......... 70% 7.07M 6s 2020-02-05T17:01:18.391 INFO:tasks.workunit.client.0.smithi008.stderr:118300K .......... .......... .......... .......... .......... 70% 20.4M 6s 2020-02-05T17:01:18.393 INFO:tasks.workunit.client.0.smithi008.stderr:118350K .......... .......... .......... .......... .......... 70% 20.8M 6s 2020-02-05T17:01:18.396 INFO:tasks.workunit.client.0.smithi008.stderr:118400K .......... .......... .......... .......... .......... 70% 21.1M 6s 2020-02-05T17:01:18.398 INFO:tasks.workunit.client.0.smithi008.stderr:118450K .......... .......... .......... .......... .......... 70% 18.0M 6s 2020-02-05T17:01:18.401 INFO:tasks.workunit.client.0.smithi008.stderr:118500K .......... .......... .......... .......... .......... 70% 20.5M 6s 2020-02-05T17:01:18.403 INFO:tasks.workunit.client.0.smithi008.stderr:118550K .......... .......... .......... .......... .......... 70% 19.8M 6s 2020-02-05T17:01:18.406 INFO:tasks.workunit.client.0.smithi008.stderr:118600K .......... .......... .......... .......... .......... 70% 19.4M 6s 2020-02-05T17:01:18.409 INFO:tasks.workunit.client.0.smithi008.stderr:118650K .......... .......... .......... .......... .......... 70% 16.0M 6s 2020-02-05T17:01:18.415 INFO:tasks.workunit.client.0.smithi008.stderr:118700K .......... .......... .......... .......... .......... 70% 8.60M 6s 2020-02-05T17:01:18.421 INFO:tasks.workunit.client.0.smithi008.stderr:118750K .......... .......... .......... .......... .......... 70% 8.31M 6s 2020-02-05T17:01:18.427 INFO:tasks.workunit.client.0.smithi008.stderr:118800K .......... .......... .......... .......... .......... 70% 7.84M 6s 2020-02-05T17:01:18.433 INFO:tasks.workunit.client.0.smithi008.stderr:118850K .......... .......... .......... .......... .......... 70% 9.04M 6s 2020-02-05T17:01:18.436 INFO:tasks.workunit.client.0.smithi008.stderr:118900K .......... .......... .......... .......... .......... 70% 13.2M 6s 2020-02-05T17:01:18.438 INFO:tasks.workunit.client.0.smithi008.stderr:118950K .......... .......... .......... .......... .......... 70% 19.1M 6s 2020-02-05T17:01:18.441 INFO:tasks.workunit.client.0.smithi008.stderr:119000K .......... .......... .......... .......... .......... 70% 18.6M 6s 2020-02-05T17:01:18.444 INFO:tasks.workunit.client.0.smithi008.stderr:119050K .......... .......... .......... .......... .......... 70% 16.3M 6s 2020-02-05T17:01:18.448 INFO:tasks.workunit.client.0.smithi008.stderr:119100K .......... .......... .......... .......... .......... 70% 12.1M 6s 2020-02-05T17:01:18.452 INFO:tasks.workunit.client.0.smithi008.stderr:119150K .......... .......... .......... .......... .......... 70% 13.3M 6s 2020-02-05T17:01:18.455 INFO:tasks.workunit.client.0.smithi008.stderr:119200K .......... .......... .......... .......... .......... 70% 14.7M 6s 2020-02-05T17:01:18.460 INFO:tasks.workunit.client.0.smithi008.stderr:119250K .......... .......... .......... .......... .......... 70% 10.8M 6s 2020-02-05T17:01:18.464 INFO:tasks.workunit.client.0.smithi008.stderr:119300K .......... .......... .......... .......... .......... 70% 11.9M 6s 2020-02-05T17:01:18.468 INFO:tasks.workunit.client.0.smithi008.stderr:119350K .......... .......... .......... .......... .......... 70% 11.2M 6s 2020-02-05T17:01:18.473 INFO:tasks.workunit.client.0.smithi008.stderr:119400K .......... .......... .......... .......... .......... 70% 9.74M 6s 2020-02-05T17:01:18.478 INFO:tasks.workunit.client.0.smithi008.stderr:119450K .......... .......... .......... .......... .......... 70% 10.1M 6s 2020-02-05T17:01:18.483 INFO:tasks.workunit.client.0.smithi008.stderr:119500K .......... .......... .......... .......... .......... 70% 10.6M 6s 2020-02-05T17:01:18.487 INFO:tasks.workunit.client.0.smithi008.stderr:119550K .......... .......... .......... .......... .......... 70% 11.6M 6s 2020-02-05T17:01:18.493 INFO:tasks.workunit.client.0.smithi008.stderr:119600K .......... .......... .......... .......... .......... 70% 8.51M 6s 2020-02-05T17:01:18.498 INFO:tasks.workunit.client.0.smithi008.stderr:119650K .......... .......... .......... .......... .......... 71% 9.60M 6s 2020-02-05T17:01:18.502 INFO:tasks.workunit.client.0.smithi008.stderr:119700K .......... .......... .......... .......... .......... 71% 11.2M 6s 2020-02-05T17:01:18.507 INFO:tasks.workunit.client.0.smithi008.stderr:119750K .......... .......... .......... .......... .......... 71% 11.0M 6s 2020-02-05T17:01:18.511 INFO:tasks.workunit.client.0.smithi008.stderr:119800K .......... .......... .......... .......... .......... 71% 11.6M 6s 2020-02-05T17:01:18.516 INFO:tasks.workunit.client.0.smithi008.stderr:119850K .......... .......... .......... .......... .......... 71% 9.93M 6s 2020-02-05T17:01:18.520 INFO:tasks.workunit.client.0.smithi008.stderr:119900K .......... .......... .......... .......... .......... 71% 11.6M 6s 2020-02-05T17:01:18.526 INFO:tasks.workunit.client.0.smithi008.stderr:119950K .......... .......... .......... .......... .......... 71% 8.83M 6s 2020-02-05T17:01:18.530 INFO:tasks.workunit.client.0.smithi008.stderr:120000K .......... .......... .......... .......... .......... 71% 9.78M 6s 2020-02-05T17:01:18.536 INFO:tasks.workunit.client.0.smithi008.stderr:120050K .......... .......... .......... .......... .......... 71% 8.16M 6s 2020-02-05T17:01:18.542 INFO:tasks.workunit.client.0.smithi008.stderr:120100K .......... .......... .......... .......... .......... 71% 8.55M 6s 2020-02-05T17:01:18.549 INFO:tasks.workunit.client.0.smithi008.stderr:120150K .......... .......... .......... .......... .......... 71% 7.51M 6s 2020-02-05T17:01:18.556 INFO:tasks.workunit.client.0.smithi008.stderr:120200K .......... .......... .......... .......... .......... 71% 7.29M 6s 2020-02-05T17:01:18.564 INFO:tasks.workunit.client.0.smithi008.stderr:120250K .......... .......... .......... .......... .......... 71% 5.89M 6s 2020-02-05T17:01:18.572 INFO:tasks.workunit.client.0.smithi008.stderr:120300K .......... .......... .......... .......... .......... 71% 6.53M 6s 2020-02-05T17:01:18.578 INFO:tasks.workunit.client.0.smithi008.stderr:120350K .......... .......... .......... .......... .......... 71% 7.60M 6s 2020-02-05T17:01:18.584 INFO:tasks.workunit.client.0.smithi008.stderr:120400K .......... .......... .......... .......... .......... 71% 7.96M 6s 2020-02-05T17:01:18.592 INFO:tasks.workunit.client.0.smithi008.stderr:120450K .......... .......... .......... .......... .......... 71% 6.30M 6s 2020-02-05T17:01:18.602 INFO:tasks.workunit.client.0.smithi008.stderr:120500K .......... .......... .......... .......... .......... 71% 4.97M 6s 2020-02-05T17:01:18.608 INFO:tasks.workunit.client.0.smithi008.stderr:120550K .......... .......... .......... .......... .......... 71% 7.76M 6s 2020-02-05T17:01:18.615 INFO:tasks.workunit.client.0.smithi008.stderr:120600K .......... .......... .......... .......... .......... 71% 7.35M 6s 2020-02-05T17:01:18.622 INFO:tasks.workunit.client.0.smithi008.stderr:120650K .......... .......... .......... .......... .......... 71% 6.39M 6s 2020-02-05T17:01:18.629 INFO:tasks.workunit.client.0.smithi008.stderr:120700K .......... .......... .......... .......... .......... 71% 7.24M 6s 2020-02-05T17:01:18.636 INFO:tasks.workunit.client.0.smithi008.stderr:120750K .......... .......... .......... .......... .......... 71% 7.22M 6s 2020-02-05T17:01:18.643 INFO:tasks.workunit.client.0.smithi008.stderr:120800K .......... .......... .......... .......... .......... 71% 6.73M 6s 2020-02-05T17:01:18.652 INFO:tasks.workunit.client.0.smithi008.stderr:120850K .......... .......... .......... .......... .......... 71% 5.75M 6s 2020-02-05T17:01:18.659 INFO:tasks.workunit.client.0.smithi008.stderr:120900K .......... .......... .......... .......... .......... 71% 6.26M 6s 2020-02-05T17:01:18.668 INFO:tasks.workunit.client.0.smithi008.stderr:120950K .......... .......... .......... .......... .......... 71% 5.67M 6s 2020-02-05T17:01:18.674 INFO:tasks.workunit.client.0.smithi008.stderr:121000K .......... .......... .......... .......... .......... 71% 7.88M 6s 2020-02-05T17:01:18.681 INFO:tasks.workunit.client.0.smithi008.stderr:121050K .......... .......... .......... .......... .......... 71% 6.84M 6s 2020-02-05T17:01:18.687 INFO:tasks.workunit.client.0.smithi008.stderr:121100K .......... .......... .......... .......... .......... 71% 8.30M 6s 2020-02-05T17:01:18.693 INFO:tasks.workunit.client.0.smithi008.stderr:121150K .......... .......... .......... .......... .......... 71% 8.41M 6s 2020-02-05T17:01:18.699 INFO:tasks.workunit.client.0.smithi008.stderr:121200K .......... .......... .......... .......... .......... 71% 7.70M 6s 2020-02-05T17:01:18.706 INFO:tasks.workunit.client.0.smithi008.stderr:121250K .......... .......... .......... .......... .......... 71% 6.79M 6s 2020-02-05T17:01:18.713 INFO:tasks.workunit.client.0.smithi008.stderr:121300K .......... .......... .......... .......... .......... 71% 7.71M 6s 2020-02-05T17:01:18.719 INFO:tasks.workunit.client.0.smithi008.stderr:121350K .......... .......... .......... .......... .......... 72% 7.53M 6s 2020-02-05T17:01:18.726 INFO:tasks.workunit.client.0.smithi008.stderr:121400K .......... .......... .......... .......... .......... 72% 7.27M 6s 2020-02-05T17:01:18.734 INFO:tasks.workunit.client.0.smithi008.stderr:121450K .......... .......... .......... .......... .......... 72% 5.92M 6s 2020-02-05T17:01:18.740 INFO:tasks.workunit.client.0.smithi008.stderr:121500K .......... .......... .......... .......... .......... 72% 7.90M 6s 2020-02-05T17:01:18.746 INFO:tasks.workunit.client.0.smithi008.stderr:121550K .......... .......... .......... .......... .......... 72% 8.00M 6s 2020-02-05T17:01:18.753 INFO:tasks.workunit.client.0.smithi008.stderr:121600K .......... .......... .......... .......... .......... 72% 7.82M 6s 2020-02-05T17:01:18.762 INFO:tasks.workunit.client.0.smithi008.stderr:121650K .......... .......... .......... .......... .......... 72% 5.68M 6s 2020-02-05T17:01:18.769 INFO:tasks.workunit.client.0.smithi008.stderr:121700K .......... .......... .......... .......... .......... 72% 6.82M 6s 2020-02-05T17:01:18.776 INFO:tasks.workunit.client.0.smithi008.stderr:121750K .......... .......... .......... .......... .......... 72% 6.13M 6s 2020-02-05T17:01:18.783 INFO:tasks.workunit.client.0.smithi008.stderr:121800K .......... .......... .......... .......... .......... 72% 7.11M 6s 2020-02-05T17:01:18.793 INFO:tasks.workunit.client.0.smithi008.stderr:121850K .......... .......... .......... .......... .......... 72% 5.09M 6s 2020-02-05T17:01:18.802 INFO:tasks.workunit.client.0.smithi008.stderr:121900K .......... .......... .......... .......... .......... 72% 5.92M 6s 2020-02-05T17:01:18.808 INFO:tasks.workunit.client.0.smithi008.stderr:121950K .......... .......... .......... .......... .......... 72% 6.68M 6s 2020-02-05T17:01:18.813 INFO:tasks.workunit.client.0.smithi008.stderr:122000K .......... .......... .......... .......... .......... 72% 9.69M 6s 2020-02-05T17:01:18.819 INFO:tasks.workunit.client.0.smithi008.stderr:122050K .......... .......... .......... .......... .......... 72% 8.73M 6s 2020-02-05T17:01:18.825 INFO:tasks.workunit.client.0.smithi008.stderr:122100K .......... .......... .......... .......... .......... 72% 8.80M 6s 2020-02-05T17:01:18.830 INFO:tasks.workunit.client.0.smithi008.stderr:122150K .......... .......... .......... .......... .......... 72% 8.60M 6s 2020-02-05T17:01:18.837 INFO:tasks.workunit.client.0.smithi008.stderr:122200K .......... .......... .......... .......... .......... 72% 7.83M 6s 2020-02-05T17:01:18.844 INFO:tasks.workunit.client.0.smithi008.stderr:122250K .......... .......... .......... .......... .......... 72% 6.58M 6s 2020-02-05T17:01:18.850 INFO:tasks.workunit.client.0.smithi008.stderr:122300K .......... .......... .......... .......... .......... 72% 7.61M 6s 2020-02-05T17:01:18.863 INFO:tasks.workunit.client.0.smithi008.stderr:122350K .......... .......... .......... .......... .......... 72% 3.68M 6s 2020-02-05T17:01:18.868 INFO:tasks.workunit.client.0.smithi008.stderr:122400K .......... .......... .......... .......... .......... 72% 10.6M 6s 2020-02-05T17:01:18.874 INFO:tasks.workunit.client.0.smithi008.stderr:122450K .......... .......... .......... .......... .......... 72% 8.28M 6s 2020-02-05T17:01:18.879 INFO:tasks.workunit.client.0.smithi008.stderr:122500K .......... .......... .......... .......... .......... 72% 10.0M 6s 2020-02-05T17:01:18.883 INFO:tasks.workunit.client.0.smithi008.stderr:122550K .......... .......... .......... .......... .......... 72% 10.7M 6s 2020-02-05T17:01:18.887 INFO:tasks.workunit.client.0.smithi008.stderr:122600K .......... .......... .......... .......... .......... 72% 11.9M 6s 2020-02-05T17:01:18.892 INFO:tasks.workunit.client.0.smithi008.stderr:122650K .......... .......... .......... .......... .......... 72% 10.5M 6s 2020-02-05T17:01:18.897 INFO:tasks.workunit.client.0.smithi008.stderr:122700K .......... .......... .......... .......... .......... 72% 10.4M 6s 2020-02-05T17:01:18.901 INFO:tasks.workunit.client.0.smithi008.stderr:122750K .......... .......... .......... .......... .......... 72% 10.8M 6s 2020-02-05T17:01:18.905 INFO:tasks.workunit.client.0.smithi008.stderr:122800K .......... .......... .......... .......... .......... 72% 12.0M 6s 2020-02-05T17:01:18.911 INFO:tasks.workunit.client.0.smithi008.stderr:122850K .......... .......... .......... .......... .......... 72% 8.42M 5s 2020-02-05T17:01:18.917 INFO:tasks.workunit.client.0.smithi008.stderr:122900K .......... .......... .......... .......... .......... 72% 9.98M 5s 2020-02-05T17:01:18.923 INFO:tasks.workunit.client.0.smithi008.stderr:122950K .......... .......... .......... .......... .......... 72% 8.87M 5s 2020-02-05T17:01:18.926 INFO:tasks.workunit.client.0.smithi008.stderr:123000K .......... .......... .......... .......... .......... 72% 10.8M 5s 2020-02-05T17:01:18.931 INFO:tasks.workunit.client.0.smithi008.stderr:123050K .......... .......... .......... .......... .......... 73% 9.91M 5s 2020-02-05T17:01:18.934 INFO:tasks.workunit.client.0.smithi008.stderr:123100K .......... .......... .......... .......... .......... 73% 14.4M 5s 2020-02-05T17:01:18.938 INFO:tasks.workunit.client.0.smithi008.stderr:123150K .......... .......... .......... .......... .......... 73% 13.1M 5s 2020-02-05T17:01:18.942 INFO:tasks.workunit.client.0.smithi008.stderr:123200K .......... .......... .......... .......... .......... 73% 12.3M 5s 2020-02-05T17:01:18.946 INFO:tasks.workunit.client.0.smithi008.stderr:123250K .......... .......... .......... .......... .......... 73% 11.6M 5s 2020-02-05T17:01:18.950 INFO:tasks.workunit.client.0.smithi008.stderr:123300K .......... .......... .......... .......... .......... 73% 13.9M 5s 2020-02-05T17:01:18.954 INFO:tasks.workunit.client.0.smithi008.stderr:123350K .......... .......... .......... .......... .......... 73% 11.6M 5s 2020-02-05T17:01:18.959 INFO:tasks.workunit.client.0.smithi008.stderr:123400K .......... .......... .......... .......... .......... 73% 10.8M 5s 2020-02-05T17:01:18.965 INFO:tasks.workunit.client.0.smithi008.stderr:123450K .......... .......... .......... .......... .......... 73% 8.25M 5s 2020-02-05T17:01:18.976 INFO:tasks.workunit.client.0.smithi008.stderr:123500K .......... .......... .......... .......... .......... 73% 9.40M 5s 2020-02-05T17:01:18.976 INFO:tasks.workunit.client.0.smithi008.stderr:123550K .......... .......... .......... .......... .......... 73% 9.34M 5s 2020-02-05T17:01:18.980 INFO:tasks.workunit.client.0.smithi008.stderr:123600K .......... .......... .......... .......... .......... 73% 9.02M 5s 2020-02-05T17:01:18.986 INFO:tasks.workunit.client.0.smithi008.stderr:123650K .......... .......... .......... .......... .......... 73% 8.06M 5s 2020-02-05T17:01:18.993 INFO:tasks.workunit.client.0.smithi008.stderr:123700K .......... .......... .......... .......... .......... 73% 7.59M 5s 2020-02-05T17:01:18.997 INFO:tasks.workunit.client.0.smithi008.stderr:123750K .......... .......... .......... .......... .......... 73% 10.6M 5s 2020-02-05T17:01:19.002 INFO:tasks.workunit.client.0.smithi008.stderr:123800K .......... .......... .......... .......... .......... 73% 10.4M 5s 2020-02-05T17:01:19.008 INFO:tasks.workunit.client.0.smithi008.stderr:123850K .......... .......... .......... .......... .......... 73% 8.10M 5s 2020-02-05T17:01:19.015 INFO:tasks.workunit.client.0.smithi008.stderr:123900K .......... .......... .......... .......... .......... 73% 8.65M 5s 2020-02-05T17:01:19.020 INFO:tasks.workunit.client.0.smithi008.stderr:123950K .......... .......... .......... .......... .......... 73% 8.30M 5s 2020-02-05T17:01:19.026 INFO:tasks.workunit.client.0.smithi008.stderr:124000K .......... .......... .......... .......... .......... 73% 7.50M 5s 2020-02-05T17:01:19.034 INFO:tasks.workunit.client.0.smithi008.stderr:124050K .......... .......... .......... .......... .......... 73% 6.20M 5s 2020-02-05T17:01:19.041 INFO:tasks.workunit.client.0.smithi008.stderr:124100K .......... .......... .......... .......... .......... 73% 7.10M 5s 2020-02-05T17:01:19.048 INFO:tasks.workunit.client.0.smithi008.stderr:124150K .......... .......... .......... .......... .......... 73% 7.58M 5s 2020-02-05T17:01:19.054 INFO:tasks.workunit.client.0.smithi008.stderr:124200K .......... .......... .......... .......... .......... 73% 7.64M 5s 2020-02-05T17:01:19.060 INFO:tasks.workunit.client.0.smithi008.stderr:124250K .......... .......... .......... .......... .......... 73% 7.88M 5s 2020-02-05T17:01:19.065 INFO:tasks.workunit.client.0.smithi008.stderr:124300K .......... .......... .......... .......... .......... 73% 9.15M 5s 2020-02-05T17:01:19.071 INFO:tasks.workunit.client.0.smithi008.stderr:124350K .......... .......... .......... .......... .......... 73% 9.09M 5s 2020-02-05T17:01:19.077 INFO:tasks.workunit.client.0.smithi008.stderr:124400K .......... .......... .......... .......... .......... 73% 7.59M 5s 2020-02-05T17:01:19.084 INFO:tasks.workunit.client.0.smithi008.stderr:124450K .......... .......... .......... .......... .......... 73% 6.95M 5s 2020-02-05T17:01:19.089 INFO:tasks.workunit.client.0.smithi008.stderr:124500K .......... .......... .......... .......... .......... 73% 9.47M 5s 2020-02-05T17:01:19.094 INFO:tasks.workunit.client.0.smithi008.stderr:124550K .......... .......... .......... .......... .......... 73% 10.9M 5s 2020-02-05T17:01:19.100 INFO:tasks.workunit.client.0.smithi008.stderr:124600K .......... .......... .......... .......... .......... 73% 8.19M 5s 2020-02-05T17:01:19.104 INFO:tasks.workunit.client.0.smithi008.stderr:124650K .......... .......... .......... .......... .......... 73% 9.99M 5s 2020-02-05T17:01:19.111 INFO:tasks.workunit.client.0.smithi008.stderr:124700K .......... .......... .......... .......... .......... 74% 11.2M 5s 2020-02-05T17:01:19.113 INFO:tasks.workunit.client.0.smithi008.stderr:124750K .......... .......... .......... .......... .......... 74% 12.3M 5s 2020-02-05T17:01:19.116 INFO:tasks.workunit.client.0.smithi008.stderr:124800K .......... .......... .......... .......... .......... 74% 15.5M 5s 2020-02-05T17:01:19.119 INFO:tasks.workunit.client.0.smithi008.stderr:124850K .......... .......... .......... .......... .......... 74% 13.7M 5s 2020-02-05T17:01:19.122 INFO:tasks.workunit.client.0.smithi008.stderr:124900K .......... .......... .......... .......... .......... 74% 16.5M 5s 2020-02-05T17:01:19.125 INFO:tasks.workunit.client.0.smithi008.stderr:124950K .......... .......... .......... .......... .......... 74% 16.7M 5s 2020-02-05T17:01:19.128 INFO:tasks.workunit.client.0.smithi008.stderr:125000K .......... .......... .......... .......... .......... 74% 17.6M 5s 2020-02-05T17:01:19.131 INFO:tasks.workunit.client.0.smithi008.stderr:125050K .......... .......... .......... .......... .......... 74% 14.9M 5s 2020-02-05T17:01:19.134 INFO:tasks.workunit.client.0.smithi008.stderr:125100K .......... .......... .......... .......... .......... 74% 18.0M 5s 2020-02-05T17:01:19.137 INFO:tasks.workunit.client.0.smithi008.stderr:125150K .......... .......... .......... .......... .......... 74% 17.5M 5s 2020-02-05T17:01:19.139 INFO:tasks.workunit.client.0.smithi008.stderr:125200K .......... .......... .......... .......... .......... 74% 17.5M 5s 2020-02-05T17:01:19.143 INFO:tasks.workunit.client.0.smithi008.stderr:125250K .......... .......... .......... .......... .......... 74% 14.0M 5s 2020-02-05T17:01:19.146 INFO:tasks.workunit.client.0.smithi008.stderr:125300K .......... .......... .......... .......... .......... 74% 16.9M 5s 2020-02-05T17:01:19.149 INFO:tasks.workunit.client.0.smithi008.stderr:125350K .......... .......... .......... .......... .......... 74% 16.7M 5s 2020-02-05T17:01:19.153 INFO:tasks.workunit.client.0.smithi008.stderr:125400K .......... .......... .......... .......... .......... 74% 15.4M 5s 2020-02-05T17:01:19.159 INFO:tasks.workunit.client.0.smithi008.stderr:125450K .......... .......... .......... .......... .......... 74% 7.66M 5s 2020-02-05T17:01:19.164 INFO:tasks.workunit.client.0.smithi008.stderr:125500K .......... .......... .......... .......... .......... 74% 10.2M 5s 2020-02-05T17:01:19.169 INFO:tasks.workunit.client.0.smithi008.stderr:125550K .......... .......... .......... .......... .......... 74% 9.71M 5s 2020-02-05T17:01:19.174 INFO:tasks.workunit.client.0.smithi008.stderr:125600K .......... .......... .......... .......... .......... 74% 9.39M 5s 2020-02-05T17:01:19.181 INFO:tasks.workunit.client.0.smithi008.stderr:125650K .......... .......... .......... .......... .......... 74% 6.90M 5s 2020-02-05T17:01:19.188 INFO:tasks.workunit.client.0.smithi008.stderr:125700K .......... .......... .......... .......... .......... 74% 7.65M 5s 2020-02-05T17:01:19.195 INFO:tasks.workunit.client.0.smithi008.stderr:125750K .......... .......... .......... .......... .......... 74% 6.67M 5s 2020-02-05T17:01:19.203 INFO:tasks.workunit.client.0.smithi008.stderr:125800K .......... .......... .......... .......... .......... 74% 6.47M 5s 2020-02-05T17:01:19.211 INFO:tasks.workunit.client.0.smithi008.stderr:125850K .......... .......... .......... .......... .......... 74% 5.70M 5s 2020-02-05T17:01:19.217 INFO:tasks.workunit.client.0.smithi008.stderr:125900K .......... .......... .......... .......... .......... 74% 8.18M 5s 2020-02-05T17:01:19.223 INFO:tasks.workunit.client.0.smithi008.stderr:125950K .......... .......... .......... .......... .......... 74% 8.16M 5s 2020-02-05T17:01:19.229 INFO:tasks.workunit.client.0.smithi008.stderr:126000K .......... .......... .......... .......... .......... 74% 8.37M 5s 2020-02-05T17:01:19.236 INFO:tasks.workunit.client.0.smithi008.stderr:126050K .......... .......... .......... .......... .......... 74% 6.31M 5s 2020-02-05T17:01:19.243 INFO:tasks.workunit.client.0.smithi008.stderr:126100K .......... .......... .......... .......... .......... 74% 7.59M 5s 2020-02-05T17:01:19.250 INFO:tasks.workunit.client.0.smithi008.stderr:126150K .......... .......... .......... .......... .......... 74% 7.24M 5s 2020-02-05T17:01:19.256 INFO:tasks.workunit.client.0.smithi008.stderr:126200K .......... .......... .......... .......... .......... 74% 7.13M 5s 2020-02-05T17:01:19.262 INFO:tasks.workunit.client.0.smithi008.stderr:126250K .......... .......... .......... .......... .......... 74% 7.93M 5s 2020-02-05T17:01:19.267 INFO:tasks.workunit.client.0.smithi008.stderr:126300K .......... .......... .......... .......... .......... 74% 10.4M 5s 2020-02-05T17:01:19.271 INFO:tasks.workunit.client.0.smithi008.stderr:126350K .......... .......... .......... .......... .......... 74% 11.1M 5s 2020-02-05T17:01:19.277 INFO:tasks.workunit.client.0.smithi008.stderr:126400K .......... .......... .......... .......... .......... 75% 9.42M 5s 2020-02-05T17:01:19.290 INFO:tasks.workunit.client.0.smithi008.stderr:126450K .......... .......... .......... .......... .......... 75% 3.79M 5s 2020-02-05T17:01:19.296 INFO:tasks.workunit.client.0.smithi008.stderr:126500K .......... .......... .......... .......... .......... 75% 7.39M 5s 2020-02-05T17:01:19.303 INFO:tasks.workunit.client.0.smithi008.stderr:126550K .......... .......... .......... .......... .......... 75% 7.34M 5s 2020-02-05T17:01:19.309 INFO:tasks.workunit.client.0.smithi008.stderr:126600K .......... .......... .......... .......... .......... 75% 8.01M 5s 2020-02-05T17:01:19.316 INFO:tasks.workunit.client.0.smithi008.stderr:126650K .......... .......... .......... .......... .......... 75% 7.26M 5s 2020-02-05T17:01:19.322 INFO:tasks.workunit.client.0.smithi008.stderr:126700K .......... .......... .......... .......... .......... 75% 8.51M 5s 2020-02-05T17:01:19.328 INFO:tasks.workunit.client.0.smithi008.stderr:126750K .......... .......... .......... .......... .......... 75% 7.70M 5s 2020-02-05T17:01:19.332 INFO:tasks.workunit.client.0.smithi008.stderr:126800K .......... .......... .......... .......... .......... 75% 10.8M 5s 2020-02-05T17:01:19.337 INFO:tasks.workunit.client.0.smithi008.stderr:126850K .......... .......... .......... .......... .......... 75% 10.1M 5s 2020-02-05T17:01:19.340 INFO:tasks.workunit.client.0.smithi008.stderr:126900K .......... .......... .......... .......... .......... 75% 13.3M 5s 2020-02-05T17:01:19.344 INFO:tasks.workunit.client.0.smithi008.stderr:126950K .......... .......... .......... .......... .......... 75% 13.2M 5s 2020-02-05T17:01:19.349 INFO:tasks.workunit.client.0.smithi008.stderr:127000K .......... .......... .......... .......... .......... 75% 11.4M 5s 2020-02-05T17:01:19.354 INFO:tasks.workunit.client.0.smithi008.stderr:127050K .......... .......... .......... .......... .......... 75% 9.60M 5s 2020-02-05T17:01:19.356 INFO:tasks.workunit.client.0.smithi008.stderr:127100K .......... .......... .......... .......... .......... 75% 16.3M 5s 2020-02-05T17:01:19.360 INFO:tasks.workunit.client.0.smithi008.stderr:127150K .......... .......... .......... .......... .......... 75% 18.7M 5s 2020-02-05T17:01:19.364 INFO:tasks.workunit.client.0.smithi008.stderr:127200K .......... .......... .......... .......... .......... 75% 10.7M 5s 2020-02-05T17:01:19.369 INFO:tasks.workunit.client.0.smithi008.stderr:127250K .......... .......... .......... .......... .......... 75% 9.36M 5s 2020-02-05T17:01:19.373 INFO:tasks.workunit.client.0.smithi008.stderr:127300K .......... .......... .......... .......... .......... 75% 11.7M 5s 2020-02-05T17:01:19.378 INFO:tasks.workunit.client.0.smithi008.stderr:127350K .......... .......... .......... .......... .......... 75% 11.1M 5s 2020-02-05T17:01:19.382 INFO:tasks.workunit.client.0.smithi008.stderr:127400K .......... .......... .......... .......... .......... 75% 11.6M 5s 2020-02-05T17:01:19.387 INFO:tasks.workunit.client.0.smithi008.stderr:127450K .......... .......... .......... .......... .......... 75% 9.96M 5s 2020-02-05T17:01:19.391 INFO:tasks.workunit.client.0.smithi008.stderr:127500K .......... .......... .......... .......... .......... 75% 11.3M 5s 2020-02-05T17:01:19.395 INFO:tasks.workunit.client.0.smithi008.stderr:127550K .......... .......... .......... .......... .......... 75% 11.7M 5s 2020-02-05T17:01:19.401 INFO:tasks.workunit.client.0.smithi008.stderr:127600K .......... .......... .......... .......... .......... 75% 8.79M 5s 2020-02-05T17:01:19.408 INFO:tasks.workunit.client.0.smithi008.stderr:127650K .......... .......... .......... .......... .......... 75% 7.15M 5s 2020-02-05T17:01:19.414 INFO:tasks.workunit.client.0.smithi008.stderr:127700K .......... .......... .......... .......... .......... 75% 8.46M 5s 2020-02-05T17:01:19.419 INFO:tasks.workunit.client.0.smithi008.stderr:127750K .......... .......... .......... .......... .......... 75% 8.83M 5s 2020-02-05T17:01:19.426 INFO:tasks.workunit.client.0.smithi008.stderr:127800K .......... .......... .......... .......... .......... 75% 7.30M 5s 2020-02-05T17:01:19.432 INFO:tasks.workunit.client.0.smithi008.stderr:127850K .......... .......... .......... .......... .......... 75% 7.95M 5s 2020-02-05T17:01:19.437 INFO:tasks.workunit.client.0.smithi008.stderr:127900K .......... .......... .......... .......... .......... 75% 9.38M 5s 2020-02-05T17:01:19.442 INFO:tasks.workunit.client.0.smithi008.stderr:127950K .......... .......... .......... .......... .......... 75% 9.51M 5s 2020-02-05T17:01:19.448 INFO:tasks.workunit.client.0.smithi008.stderr:128000K .......... .......... .......... .......... .......... 75% 8.06M 5s 2020-02-05T17:01:19.456 INFO:tasks.workunit.client.0.smithi008.stderr:128050K .......... .......... .......... .......... .......... 75% 6.64M 5s 2020-02-05T17:01:19.462 INFO:tasks.workunit.client.0.smithi008.stderr:128100K .......... .......... .......... .......... .......... 76% 7.69M 5s 2020-02-05T17:01:19.468 INFO:tasks.workunit.client.0.smithi008.stderr:128150K .......... .......... .......... .......... .......... 76% 8.04M 5s 2020-02-05T17:01:19.475 INFO:tasks.workunit.client.0.smithi008.stderr:128200K .......... .......... .......... .......... .......... 76% 7.30M 5s 2020-02-05T17:01:19.483 INFO:tasks.workunit.client.0.smithi008.stderr:128250K .......... .......... .......... .......... .......... 76% 5.83M 5s 2020-02-05T17:01:19.490 INFO:tasks.workunit.client.0.smithi008.stderr:128300K .......... .......... .......... .......... .......... 76% 7.34M 5s 2020-02-05T17:01:19.497 INFO:tasks.workunit.client.0.smithi008.stderr:128350K .......... .......... .......... .......... .......... 76% 6.92M 5s 2020-02-05T17:01:19.504 INFO:tasks.workunit.client.0.smithi008.stderr:128400K .......... .......... .......... .......... .......... 76% 6.85M 5s 2020-02-05T17:01:19.513 INFO:tasks.workunit.client.0.smithi008.stderr:128450K .......... .......... .......... .......... .......... 76% 5.82M 5s 2020-02-05T17:01:19.518 INFO:tasks.workunit.client.0.smithi008.stderr:128500K .......... .......... .......... .......... .......... 76% 8.32M 5s 2020-02-05T17:01:19.524 INFO:tasks.workunit.client.0.smithi008.stderr:128550K .......... .......... .......... .......... .......... 76% 9.19M 5s 2020-02-05T17:01:19.529 INFO:tasks.workunit.client.0.smithi008.stderr:128600K .......... .......... .......... .......... .......... 76% 8.84M 5s 2020-02-05T17:01:19.536 INFO:tasks.workunit.client.0.smithi008.stderr:128650K .......... .......... .......... .......... .......... 76% 9.24M 5s 2020-02-05T17:01:19.540 INFO:tasks.workunit.client.0.smithi008.stderr:128700K .......... .......... .......... .......... .......... 76% 8.05M 5s 2020-02-05T17:01:19.544 INFO:tasks.workunit.client.0.smithi008.stderr:128750K .......... .......... .......... .......... .......... 76% 13.4M 5s 2020-02-05T17:01:19.547 INFO:tasks.workunit.client.0.smithi008.stderr:128800K .......... .......... .......... .......... .......... 76% 13.7M 5s 2020-02-05T17:01:19.554 INFO:tasks.workunit.client.0.smithi008.stderr:128850K .......... .......... .......... .......... .......... 76% 8.34M 5s 2020-02-05T17:01:19.558 INFO:tasks.workunit.client.0.smithi008.stderr:128900K .......... .......... .......... .......... .......... 76% 9.81M 5s 2020-02-05T17:01:19.562 INFO:tasks.workunit.client.0.smithi008.stderr:128950K .......... .......... .......... .......... .......... 76% 13.8M 5s 2020-02-05T17:01:19.567 INFO:tasks.workunit.client.0.smithi008.stderr:129000K .......... .......... .......... .......... .......... 76% 8.75M 5s 2020-02-05T17:01:19.574 INFO:tasks.workunit.client.0.smithi008.stderr:129050K .......... .......... .......... .......... .......... 76% 8.00M 5s 2020-02-05T17:01:19.579 INFO:tasks.workunit.client.0.smithi008.stderr:129100K .......... .......... .......... .......... .......... 76% 9.25M 5s 2020-02-05T17:01:19.583 INFO:tasks.workunit.client.0.smithi008.stderr:129150K .......... .......... .......... .......... .......... 76% 10.7M 5s 2020-02-05T17:01:19.588 INFO:tasks.workunit.client.0.smithi008.stderr:129200K .......... .......... .......... .......... .......... 76% 10.9M 5s 2020-02-05T17:01:19.594 INFO:tasks.workunit.client.0.smithi008.stderr:129250K .......... .......... .......... .......... .......... 76% 8.91M 5s 2020-02-05T17:01:19.598 INFO:tasks.workunit.client.0.smithi008.stderr:129300K .......... .......... .......... .......... .......... 76% 10.4M 5s 2020-02-05T17:01:19.604 INFO:tasks.workunit.client.0.smithi008.stderr:129350K .......... .......... .......... .......... .......... 76% 8.65M 5s 2020-02-05T17:01:19.610 INFO:tasks.workunit.client.0.smithi008.stderr:129400K .......... .......... .......... .......... .......... 76% 8.06M 5s 2020-02-05T17:01:19.618 INFO:tasks.workunit.client.0.smithi008.stderr:129450K .......... .......... .......... .......... .......... 76% 6.15M 5s 2020-02-05T17:01:19.624 INFO:tasks.workunit.client.0.smithi008.stderr:129500K .......... .......... .......... .......... .......... 76% 8.31M 5s 2020-02-05T17:01:19.629 INFO:tasks.workunit.client.0.smithi008.stderr:129550K .......... .......... .......... .......... .......... 76% 8.75M 5s 2020-02-05T17:01:19.635 INFO:tasks.workunit.client.0.smithi008.stderr:129600K .......... .......... .......... .......... .......... 76% 8.62M 5s 2020-02-05T17:01:19.641 INFO:tasks.workunit.client.0.smithi008.stderr:129650K .......... .......... .......... .......... .......... 76% 7.63M 5s 2020-02-05T17:01:19.646 INFO:tasks.workunit.client.0.smithi008.stderr:129700K .......... .......... .......... .......... .......... 76% 10.4M 5s 2020-02-05T17:01:19.650 INFO:tasks.workunit.client.0.smithi008.stderr:129750K .......... .......... .......... .......... .......... 77% 11.1M 5s 2020-02-05T17:01:19.654 INFO:tasks.workunit.client.0.smithi008.stderr:129800K .......... .......... .......... .......... .......... 77% 12.6M 5s 2020-02-05T17:01:19.659 INFO:tasks.workunit.client.0.smithi008.stderr:129850K .......... .......... .......... .......... .......... 77% 11.0M 5s 2020-02-05T17:01:19.662 INFO:tasks.workunit.client.0.smithi008.stderr:129900K .......... .......... .......... .......... .......... 77% 12.9M 5s 2020-02-05T17:01:19.665 INFO:tasks.workunit.client.0.smithi008.stderr:129950K .......... .......... .......... .......... .......... 77% 15.3M 5s 2020-02-05T17:01:19.669 INFO:tasks.workunit.client.0.smithi008.stderr:130000K .......... .......... .......... .......... .......... 77% 15.0M 5s 2020-02-05T17:01:19.672 INFO:tasks.workunit.client.0.smithi008.stderr:130050K .......... .......... .......... .......... .......... 77% 12.5M 5s 2020-02-05T17:01:19.676 INFO:tasks.workunit.client.0.smithi008.stderr:130100K .......... .......... .......... .......... .......... 77% 14.8M 5s 2020-02-05T17:01:19.679 INFO:tasks.workunit.client.0.smithi008.stderr:130150K .......... .......... .......... .......... .......... 77% 14.2M 5s 2020-02-05T17:01:19.683 INFO:tasks.workunit.client.0.smithi008.stderr:130200K .......... .......... .......... .......... .......... 77% 14.5M 5s 2020-02-05T17:01:19.688 INFO:tasks.workunit.client.0.smithi008.stderr:130250K .......... .......... .......... .......... .......... 77% 10.1M 5s 2020-02-05T17:01:19.695 INFO:tasks.workunit.client.0.smithi008.stderr:130300K .......... .......... .......... .......... .......... 77% 7.16M 5s 2020-02-05T17:01:19.702 INFO:tasks.workunit.client.0.smithi008.stderr:130350K .......... .......... .......... .......... .......... 77% 7.12M 5s 2020-02-05T17:01:19.708 INFO:tasks.workunit.client.0.smithi008.stderr:130400K .......... .......... .......... .......... .......... 77% 7.25M 5s 2020-02-05T17:01:19.716 INFO:tasks.workunit.client.0.smithi008.stderr:130450K .......... .......... .......... .......... .......... 77% 6.26M 5s 2020-02-05T17:01:19.724 INFO:tasks.workunit.client.0.smithi008.stderr:130500K .......... .......... .......... .......... .......... 77% 6.36M 5s 2020-02-05T17:01:19.738 INFO:tasks.workunit.client.0.smithi008.stderr:130550K .......... .......... .......... .......... .......... 77% 3.49M 5s 2020-02-05T17:01:19.745 INFO:tasks.workunit.client.0.smithi008.stderr:130600K .......... .......... .......... .......... .......... 77% 6.90M 5s 2020-02-05T17:01:19.752 INFO:tasks.workunit.client.0.smithi008.stderr:130650K .......... .......... .......... .......... .......... 77% 7.23M 5s 2020-02-05T17:01:19.757 INFO:tasks.workunit.client.0.smithi008.stderr:130700K .......... .......... .......... .......... .......... 77% 8.43M 5s 2020-02-05T17:01:19.761 INFO:tasks.workunit.client.0.smithi008.stderr:130750K .......... .......... .......... .......... .......... 77% 13.0M 5s 2020-02-05T17:01:19.765 INFO:tasks.workunit.client.0.smithi008.stderr:130800K .......... .......... .......... .......... .......... 77% 12.8M 5s 2020-02-05T17:01:19.770 INFO:tasks.workunit.client.0.smithi008.stderr:130850K .......... .......... .......... .......... .......... 77% 9.09M 5s 2020-02-05T17:01:19.777 INFO:tasks.workunit.client.0.smithi008.stderr:130900K .......... .......... .......... .......... .......... 77% 7.94M 4s 2020-02-05T17:01:19.783 INFO:tasks.workunit.client.0.smithi008.stderr:130950K .......... .......... .......... .......... .......... 77% 7.47M 4s 2020-02-05T17:01:19.789 INFO:tasks.workunit.client.0.smithi008.stderr:131000K .......... .......... .......... .......... .......... 77% 8.29M 4s 2020-02-05T17:01:19.795 INFO:tasks.workunit.client.0.smithi008.stderr:131050K .......... .......... .......... .......... .......... 77% 7.90M 4s 2020-02-05T17:01:19.799 INFO:tasks.workunit.client.0.smithi008.stderr:131100K .......... .......... .......... .......... .......... 77% 12.2M 4s 2020-02-05T17:01:19.803 INFO:tasks.workunit.client.0.smithi008.stderr:131150K .......... .......... .......... .......... .......... 77% 13.3M 4s 2020-02-05T17:01:19.806 INFO:tasks.workunit.client.0.smithi008.stderr:131200K .......... .......... .......... .......... .......... 77% 12.7M 4s 2020-02-05T17:01:19.812 INFO:tasks.workunit.client.0.smithi008.stderr:131250K .......... .......... .......... .......... .......... 77% 8.56M 4s 2020-02-05T17:01:19.818 INFO:tasks.workunit.client.0.smithi008.stderr:131300K .......... .......... .......... .......... .......... 77% 9.44M 4s 2020-02-05T17:01:19.823 INFO:tasks.workunit.client.0.smithi008.stderr:131350K .......... .......... .......... .......... .......... 77% 9.21M 4s 2020-02-05T17:01:19.827 INFO:tasks.workunit.client.0.smithi008.stderr:131400K .......... .......... .......... .......... .......... 77% 10.4M 4s 2020-02-05T17:01:19.832 INFO:tasks.workunit.client.0.smithi008.stderr:131450K .......... .......... .......... .......... .......... 78% 9.34M 4s 2020-02-05T17:01:19.837 INFO:tasks.workunit.client.0.smithi008.stderr:131500K .......... .......... .......... .......... .......... 78% 11.3M 4s 2020-02-05T17:01:19.841 INFO:tasks.workunit.client.0.smithi008.stderr:131550K .......... .......... .......... .......... .......... 78% 12.4M 4s 2020-02-05T17:01:19.845 INFO:tasks.workunit.client.0.smithi008.stderr:131600K .......... .......... .......... .......... .......... 78% 12.7M 4s 2020-02-05T17:01:19.850 INFO:tasks.workunit.client.0.smithi008.stderr:131650K .......... .......... .......... .......... .......... 78% 8.68M 4s 2020-02-05T17:01:19.855 INFO:tasks.workunit.client.0.smithi008.stderr:131700K .......... .......... .......... .......... .......... 78% 9.64M 4s 2020-02-05T17:01:19.861 INFO:tasks.workunit.client.0.smithi008.stderr:131750K .......... .......... .......... .......... .......... 78% 9.19M 4s 2020-02-05T17:01:19.866 INFO:tasks.workunit.client.0.smithi008.stderr:131800K .......... .......... .......... .......... .......... 78% 8.94M 4s 2020-02-05T17:01:19.874 INFO:tasks.workunit.client.0.smithi008.stderr:131850K .......... .......... .......... .......... .......... 78% 7.26M 4s 2020-02-05T17:01:19.880 INFO:tasks.workunit.client.0.smithi008.stderr:131900K .......... .......... .......... .......... .......... 78% 7.26M 4s 2020-02-05T17:01:19.885 INFO:tasks.workunit.client.0.smithi008.stderr:131950K .......... .......... .......... .......... .......... 78% 8.54M 4s 2020-02-05T17:01:19.891 INFO:tasks.workunit.client.0.smithi008.stderr:132000K .......... .......... .......... .......... .......... 78% 9.11M 4s 2020-02-05T17:01:19.898 INFO:tasks.workunit.client.0.smithi008.stderr:132050K .......... .......... .......... .......... .......... 78% 7.32M 4s 2020-02-05T17:01:19.903 INFO:tasks.workunit.client.0.smithi008.stderr:132100K .......... .......... .......... .......... .......... 78% 8.26M 4s 2020-02-05T17:01:19.909 INFO:tasks.workunit.client.0.smithi008.stderr:132150K .......... .......... .......... .......... .......... 78% 8.99M 4s 2020-02-05T17:01:19.913 INFO:tasks.workunit.client.0.smithi008.stderr:132200K .......... .......... .......... .......... .......... 78% 10.1M 4s 2020-02-05T17:01:19.918 INFO:tasks.workunit.client.0.smithi008.stderr:132250K .......... .......... .......... .......... .......... 78% 10.4M 4s 2020-02-05T17:01:19.922 INFO:tasks.workunit.client.0.smithi008.stderr:132300K .......... .......... .......... .......... .......... 78% 11.8M 4s 2020-02-05T17:01:19.926 INFO:tasks.workunit.client.0.smithi008.stderr:132350K .......... .......... .......... .......... .......... 78% 11.7M 4s 2020-02-05T17:01:19.930 INFO:tasks.workunit.client.0.smithi008.stderr:132400K .......... .......... .......... .......... .......... 78% 12.7M 4s 2020-02-05T17:01:19.936 INFO:tasks.workunit.client.0.smithi008.stderr:132450K .......... .......... .......... .......... .......... 78% 9.21M 4s 2020-02-05T17:01:19.942 INFO:tasks.workunit.client.0.smithi008.stderr:132500K .......... .......... .......... .......... .......... 78% 8.42M 4s 2020-02-05T17:01:19.948 INFO:tasks.workunit.client.0.smithi008.stderr:132550K .......... .......... .......... .......... .......... 78% 7.62M 4s 2020-02-05T17:01:19.955 INFO:tasks.workunit.client.0.smithi008.stderr:132600K .......... .......... .......... .......... .......... 78% 7.16M 4s 2020-02-05T17:01:19.963 INFO:tasks.workunit.client.0.smithi008.stderr:132650K .......... .......... .......... .......... .......... 78% 5.92M 4s 2020-02-05T17:01:19.971 INFO:tasks.workunit.client.0.smithi008.stderr:132700K .......... .......... .......... .......... .......... 78% 6.88M 4s 2020-02-05T17:01:19.982 INFO:tasks.workunit.client.0.smithi008.stderr:132750K .......... .......... .......... .......... .......... 78% 5.94M 4s 2020-02-05T17:01:19.989 INFO:tasks.workunit.client.0.smithi008.stderr:132800K .......... .......... .......... .......... .......... 78% 4.79M 4s 2020-02-05T17:01:19.996 INFO:tasks.workunit.client.0.smithi008.stderr:132850K .......... .......... .......... .......... .......... 78% 6.53M 4s 2020-02-05T17:01:20.002 INFO:tasks.workunit.client.0.smithi008.stderr:132900K .......... .......... .......... .......... .......... 78% 8.23M 4s 2020-02-05T17:01:20.008 INFO:tasks.workunit.client.0.smithi008.stderr:132950K .......... .......... .......... .......... .......... 78% 9.03M 4s 2020-02-05T17:01:20.012 INFO:tasks.workunit.client.0.smithi008.stderr:133000K .......... .......... .......... .......... .......... 78% 9.71M 4s 2020-02-05T17:01:20.017 INFO:tasks.workunit.client.0.smithi008.stderr:133050K .......... .......... .......... .......... .......... 78% 9.76M 4s 2020-02-05T17:01:20.022 INFO:tasks.workunit.client.0.smithi008.stderr:133100K .......... .......... .......... .......... .......... 78% 10.8M 4s 2020-02-05T17:01:20.026 INFO:tasks.workunit.client.0.smithi008.stderr:133150K .......... .......... .......... .......... .......... 79% 11.4M 4s 2020-02-05T17:01:20.030 INFO:tasks.workunit.client.0.smithi008.stderr:133200K .......... .......... .......... .......... .......... 79% 11.1M 4s 2020-02-05T17:01:20.036 INFO:tasks.workunit.client.0.smithi008.stderr:133250K .......... .......... .......... .......... .......... 79% 8.99M 4s 2020-02-05T17:01:20.041 INFO:tasks.workunit.client.0.smithi008.stderr:133300K .......... .......... .......... .......... .......... 79% 10.3M 4s 2020-02-05T17:01:20.046 INFO:tasks.workunit.client.0.smithi008.stderr:133350K .......... .......... .......... .......... .......... 79% 9.08M 4s 2020-02-05T17:01:20.052 INFO:tasks.workunit.client.0.smithi008.stderr:133400K .......... .......... .......... .......... .......... 79% 8.59M 4s 2020-02-05T17:01:20.060 INFO:tasks.workunit.client.0.smithi008.stderr:133450K .......... .......... .......... .......... .......... 79% 6.18M 4s 2020-02-05T17:01:20.068 INFO:tasks.workunit.client.0.smithi008.stderr:133500K .......... .......... .......... .......... .......... 79% 6.48M 4s 2020-02-05T17:01:20.074 INFO:tasks.workunit.client.0.smithi008.stderr:133550K .......... .......... .......... .......... .......... 79% 7.14M 4s 2020-02-05T17:01:20.082 INFO:tasks.workunit.client.0.smithi008.stderr:133600K .......... .......... .......... .......... .......... 79% 7.00M 4s 2020-02-05T17:01:20.089 INFO:tasks.workunit.client.0.smithi008.stderr:133650K .......... .......... .......... .......... .......... 79% 6.01M 4s 2020-02-05T17:01:20.096 INFO:tasks.workunit.client.0.smithi008.stderr:133700K .......... .......... .......... .......... .......... 79% 7.24M 4s 2020-02-05T17:01:20.103 INFO:tasks.workunit.client.0.smithi008.stderr:133750K .......... .......... .......... .......... .......... 79% 7.14M 4s 2020-02-05T17:01:20.110 INFO:tasks.workunit.client.0.smithi008.stderr:133800K .......... .......... .......... .......... .......... 79% 6.86M 4s 2020-02-05T17:01:20.119 INFO:tasks.workunit.client.0.smithi008.stderr:133850K .......... .......... .......... .......... .......... 79% 5.52M 4s 2020-02-05T17:01:20.125 INFO:tasks.workunit.client.0.smithi008.stderr:133900K .......... .......... .......... .......... .......... 79% 8.89M 4s 2020-02-05T17:01:20.130 INFO:tasks.workunit.client.0.smithi008.stderr:133950K .......... .......... .......... .......... .......... 79% 8.85M 4s 2020-02-05T17:01:20.136 INFO:tasks.workunit.client.0.smithi008.stderr:134000K .......... .......... .......... .......... .......... 79% 8.51M 4s 2020-02-05T17:01:20.143 INFO:tasks.workunit.client.0.smithi008.stderr:134050K .......... .......... .......... .......... .......... 79% 6.84M 4s 2020-02-05T17:01:20.148 INFO:tasks.workunit.client.0.smithi008.stderr:134100K .......... .......... .......... .......... .......... 79% 8.89M 4s 2020-02-05T17:01:20.155 INFO:tasks.workunit.client.0.smithi008.stderr:134150K .......... .......... .......... .......... .......... 79% 8.02M 4s 2020-02-05T17:01:20.161 INFO:tasks.workunit.client.0.smithi008.stderr:134200K .......... .......... .......... .......... .......... 79% 7.77M 4s 2020-02-05T17:01:20.167 INFO:tasks.workunit.client.0.smithi008.stderr:134250K .......... .......... .......... .......... .......... 79% 7.03M 4s 2020-02-05T17:01:20.172 INFO:tasks.workunit.client.0.smithi008.stderr:134300K .......... .......... .......... .......... .......... 79% 11.3M 4s 2020-02-05T17:01:20.176 INFO:tasks.workunit.client.0.smithi008.stderr:134350K .......... .......... .......... .......... .......... 79% 11.5M 4s 2020-02-05T17:01:20.180 INFO:tasks.workunit.client.0.smithi008.stderr:134400K .......... .......... .......... .......... .......... 79% 12.1M 4s 2020-02-05T17:01:20.184 INFO:tasks.workunit.client.0.smithi008.stderr:134450K .......... .......... .......... .......... .......... 79% 12.2M 4s 2020-02-05T17:01:20.188 INFO:tasks.workunit.client.0.smithi008.stderr:134500K .......... .......... .......... .......... .......... 79% 12.9M 4s 2020-02-05T17:01:20.193 INFO:tasks.workunit.client.0.smithi008.stderr:134550K .......... .......... .......... .......... .......... 79% 9.54M 4s 2020-02-05T17:01:20.198 INFO:tasks.workunit.client.0.smithi008.stderr:134600K .......... .......... .......... .......... .......... 79% 9.41M 4s 2020-02-05T17:01:20.210 INFO:tasks.workunit.client.0.smithi008.stderr:134650K .......... .......... .......... .......... .......... 79% 4.47M 4s 2020-02-05T17:01:20.215 INFO:tasks.workunit.client.0.smithi008.stderr:134700K .......... .......... .......... .......... .......... 79% 8.80M 4s 2020-02-05T17:01:20.221 INFO:tasks.workunit.client.0.smithi008.stderr:134750K .......... .......... .......... .......... .......... 79% 8.21M 4s 2020-02-05T17:01:20.227 INFO:tasks.workunit.client.0.smithi008.stderr:134800K .......... .......... .......... .......... .......... 79% 7.84M 4s 2020-02-05T17:01:20.232 INFO:tasks.workunit.client.0.smithi008.stderr:134850K .......... .......... .......... .......... .......... 80% 8.56M 4s 2020-02-05T17:01:20.237 INFO:tasks.workunit.client.0.smithi008.stderr:134900K .......... .......... .......... .......... .......... 80% 11.4M 4s 2020-02-05T17:01:20.241 INFO:tasks.workunit.client.0.smithi008.stderr:134950K .......... .......... .......... .......... .......... 80% 11.0M 4s 2020-02-05T17:01:20.246 INFO:tasks.workunit.client.0.smithi008.stderr:135000K .......... .......... .......... .......... .......... 80% 11.1M 4s 2020-02-05T17:01:20.250 INFO:tasks.workunit.client.0.smithi008.stderr:135050K .......... .......... .......... .......... .......... 80% 10.5M 4s 2020-02-05T17:01:20.254 INFO:tasks.workunit.client.0.smithi008.stderr:135100K .......... .......... .......... .......... .......... 80% 12.7M 4s 2020-02-05T17:01:20.258 INFO:tasks.workunit.client.0.smithi008.stderr:135150K .......... .......... .......... .......... .......... 80% 11.2M 4s 2020-02-05T17:01:20.262 INFO:tasks.workunit.client.0.smithi008.stderr:135200K .......... .......... .......... .......... .......... 80% 12.7M 4s 2020-02-05T17:01:20.266 INFO:tasks.workunit.client.0.smithi008.stderr:135250K .......... .......... .......... .......... .......... 80% 11.1M 4s 2020-02-05T17:01:20.271 INFO:tasks.workunit.client.0.smithi008.stderr:135300K .......... .......... .......... .......... .......... 80% 11.4M 4s 2020-02-05T17:01:20.275 INFO:tasks.workunit.client.0.smithi008.stderr:135350K .......... .......... .......... .......... .......... 80% 11.0M 4s 2020-02-05T17:01:20.279 INFO:tasks.workunit.client.0.smithi008.stderr:135400K .......... .......... .......... .......... .......... 80% 12.7M 4s 2020-02-05T17:01:20.284 INFO:tasks.workunit.client.0.smithi008.stderr:135450K .......... .......... .......... .......... .......... 80% 9.63M 4s 2020-02-05T17:01:20.289 INFO:tasks.workunit.client.0.smithi008.stderr:135500K .......... .......... .......... .......... .......... 80% 10.5M 4s 2020-02-05T17:01:20.294 INFO:tasks.workunit.client.0.smithi008.stderr:135550K .......... .......... .......... .......... .......... 80% 9.14M 4s 2020-02-05T17:01:20.299 INFO:tasks.workunit.client.0.smithi008.stderr:135600K .......... .......... .......... .......... .......... 80% 9.45M 4s 2020-02-05T17:01:20.305 INFO:tasks.workunit.client.0.smithi008.stderr:135650K .......... .......... .......... .......... .......... 80% 8.90M 4s 2020-02-05T17:01:20.309 INFO:tasks.workunit.client.0.smithi008.stderr:135700K .......... .......... .......... .......... .......... 80% 10.9M 4s 2020-02-05T17:01:20.314 INFO:tasks.workunit.client.0.smithi008.stderr:135750K .......... .......... .......... .......... .......... 80% 10.8M 4s 2020-02-05T17:01:20.318 INFO:tasks.workunit.client.0.smithi008.stderr:135800K .......... .......... .......... .......... .......... 80% 10.9M 4s 2020-02-05T17:01:20.323 INFO:tasks.workunit.client.0.smithi008.stderr:135850K .......... .......... .......... .......... .......... 80% 9.58M 4s 2020-02-05T17:01:20.328 INFO:tasks.workunit.client.0.smithi008.stderr:135900K .......... .......... .......... .......... .......... 80% 10.6M 4s 2020-02-05T17:01:20.334 INFO:tasks.workunit.client.0.smithi008.stderr:135950K .......... .......... .......... .......... .......... 80% 10.9M 4s 2020-02-05T17:01:20.338 INFO:tasks.workunit.client.0.smithi008.stderr:136000K .......... .......... .......... .......... .......... 80% 9.00M 4s 2020-02-05T17:01:20.343 INFO:tasks.workunit.client.0.smithi008.stderr:136050K .......... .......... .......... .......... .......... 80% 9.95M 4s 2020-02-05T17:01:20.347 INFO:tasks.workunit.client.0.smithi008.stderr:136100K .......... .......... .......... .......... .......... 80% 12.1M 4s 2020-02-05T17:01:20.351 INFO:tasks.workunit.client.0.smithi008.stderr:136150K .......... .......... .......... .......... .......... 80% 12.4M 4s 2020-02-05T17:01:20.355 INFO:tasks.workunit.client.0.smithi008.stderr:136200K .......... .......... .......... .......... .......... 80% 11.8M 4s 2020-02-05T17:01:20.360 INFO:tasks.workunit.client.0.smithi008.stderr:136250K .......... .......... .......... .......... .......... 80% 10.1M 4s 2020-02-05T17:01:20.364 INFO:tasks.workunit.client.0.smithi008.stderr:136300K .......... .......... .......... .......... .......... 80% 11.8M 4s 2020-02-05T17:01:20.368 INFO:tasks.workunit.client.0.smithi008.stderr:136350K .......... .......... .......... .......... .......... 80% 11.4M 4s 2020-02-05T17:01:20.373 INFO:tasks.workunit.client.0.smithi008.stderr:136400K .......... .......... .......... .......... .......... 80% 10.5M 4s 2020-02-05T17:01:20.378 INFO:tasks.workunit.client.0.smithi008.stderr:136450K .......... .......... .......... .......... .......... 80% 8.94M 4s 2020-02-05T17:01:20.383 INFO:tasks.workunit.client.0.smithi008.stderr:136500K .......... .......... .......... .......... .......... 81% 9.88M 4s 2020-02-05T17:01:20.388 INFO:tasks.workunit.client.0.smithi008.stderr:136550K .......... .......... .......... .......... .......... 81% 10.5M 4s 2020-02-05T17:01:20.393 INFO:tasks.workunit.client.0.smithi008.stderr:136600K .......... .......... .......... .......... .......... 81% 10.2M 4s 2020-02-05T17:01:20.399 INFO:tasks.workunit.client.0.smithi008.stderr:136650K .......... .......... .......... .......... .......... 81% 8.43M 4s 2020-02-05T17:01:20.404 INFO:tasks.workunit.client.0.smithi008.stderr:136700K .......... .......... .......... .......... .......... 81% 9.13M 4s 2020-02-05T17:01:20.410 INFO:tasks.workunit.client.0.smithi008.stderr:136750K .......... .......... .......... .......... .......... 81% 8.74M 4s 2020-02-05T17:01:20.415 INFO:tasks.workunit.client.0.smithi008.stderr:136800K .......... .......... .......... .......... .......... 81% 8.64M 4s 2020-02-05T17:01:20.425 INFO:tasks.workunit.client.0.smithi008.stderr:136850K .......... .......... .......... .......... .......... 81% 5.16M 4s 2020-02-05T17:01:20.431 INFO:tasks.workunit.client.0.smithi008.stderr:136900K .......... .......... .......... .......... .......... 81% 8.27M 4s 2020-02-05T17:01:20.437 INFO:tasks.workunit.client.0.smithi008.stderr:136950K .......... .......... .......... .......... .......... 81% 8.15M 4s 2020-02-05T17:01:20.443 INFO:tasks.workunit.client.0.smithi008.stderr:137000K .......... .......... .......... .......... .......... 81% 8.36M 4s 2020-02-05T17:01:20.450 INFO:tasks.workunit.client.0.smithi008.stderr:137050K .......... .......... .......... .......... .......... 81% 6.63M 4s 2020-02-05T17:01:20.457 INFO:tasks.workunit.client.0.smithi008.stderr:137100K .......... .......... .......... .......... .......... 81% 7.13M 4s 2020-02-05T17:01:20.464 INFO:tasks.workunit.client.0.smithi008.stderr:137150K .......... .......... .......... .......... .......... 81% 6.98M 4s 2020-02-05T17:01:20.471 INFO:tasks.workunit.client.0.smithi008.stderr:137200K .......... .......... .......... .......... .......... 81% 6.77M 4s 2020-02-05T17:01:20.479 INFO:tasks.workunit.client.0.smithi008.stderr:137250K .......... .......... .......... .......... .......... 81% 6.15M 4s 2020-02-05T17:01:20.486 INFO:tasks.workunit.client.0.smithi008.stderr:137300K .......... .......... .......... .......... .......... 81% 7.46M 4s 2020-02-05T17:01:20.493 INFO:tasks.workunit.client.0.smithi008.stderr:137350K .......... .......... .......... .......... .......... 81% 7.15M 4s 2020-02-05T17:01:20.499 INFO:tasks.workunit.client.0.smithi008.stderr:137400K .......... .......... .......... .......... .......... 81% 7.12M 4s 2020-02-05T17:01:20.507 INFO:tasks.workunit.client.0.smithi008.stderr:137450K .......... .......... .......... .......... .......... 81% 6.57M 4s 2020-02-05T17:01:20.513 INFO:tasks.workunit.client.0.smithi008.stderr:137500K .......... .......... .......... .......... .......... 81% 7.49M 4s 2020-02-05T17:01:20.520 INFO:tasks.workunit.client.0.smithi008.stderr:137550K .......... .......... .......... .......... .......... 81% 8.00M 4s 2020-02-05T17:01:20.526 INFO:tasks.workunit.client.0.smithi008.stderr:137600K .......... .......... .......... .......... .......... 81% 7.63M 4s 2020-02-05T17:01:20.534 INFO:tasks.workunit.client.0.smithi008.stderr:137650K .......... .......... .......... .......... .......... 81% 6.37M 4s 2020-02-05T17:01:20.540 INFO:tasks.workunit.client.0.smithi008.stderr:137700K .......... .......... .......... .......... .......... 81% 7.31M 4s 2020-02-05T17:01:20.547 INFO:tasks.workunit.client.0.smithi008.stderr:137750K .......... .......... .......... .......... .......... 81% 6.95M 4s 2020-02-05T17:01:20.555 INFO:tasks.workunit.client.0.smithi008.stderr:137800K .......... .......... .......... .......... .......... 81% 6.65M 4s 2020-02-05T17:01:20.563 INFO:tasks.workunit.client.0.smithi008.stderr:137850K .......... .......... .......... .......... .......... 81% 5.61M 4s 2020-02-05T17:01:20.571 INFO:tasks.workunit.client.0.smithi008.stderr:137900K .......... .......... .......... .......... .......... 81% 6.93M 4s 2020-02-05T17:01:20.577 INFO:tasks.workunit.client.0.smithi008.stderr:137950K .......... .......... .......... .......... .......... 81% 7.26M 4s 2020-02-05T17:01:20.583 INFO:tasks.workunit.client.0.smithi008.stderr:138000K .......... .......... .......... .......... .......... 81% 8.27M 4s 2020-02-05T17:01:20.588 INFO:tasks.workunit.client.0.smithi008.stderr:138050K .......... .......... .......... .......... .......... 81% 9.11M 4s 2020-02-05T17:01:20.595 INFO:tasks.workunit.client.0.smithi008.stderr:138100K .......... .......... .......... .......... .......... 81% 7.04M 4s 2020-02-05T17:01:20.603 INFO:tasks.workunit.client.0.smithi008.stderr:138150K .......... .......... .......... .......... .......... 81% 6.23M 4s 2020-02-05T17:01:20.611 INFO:tasks.workunit.client.0.smithi008.stderr:138200K .......... .......... .......... .......... .......... 82% 6.17M 4s 2020-02-05T17:01:20.621 INFO:tasks.workunit.client.0.smithi008.stderr:138250K .......... .......... .......... .......... .......... 82% 5.23M 4s 2020-02-05T17:01:20.629 INFO:tasks.workunit.client.0.smithi008.stderr:138300K .......... .......... .......... .......... .......... 82% 5.86M 4s 2020-02-05T17:01:20.638 INFO:tasks.workunit.client.0.smithi008.stderr:138350K .......... .......... .......... .......... .......... 82% 5.66M 4s 2020-02-05T17:01:20.647 INFO:tasks.workunit.client.0.smithi008.stderr:138400K .......... .......... .......... .......... .......... 82% 5.50M 4s 2020-02-05T17:01:20.657 INFO:tasks.workunit.client.0.smithi008.stderr:138450K .......... .......... .......... .......... .......... 82% 4.55M 4s 2020-02-05T17:01:20.667 INFO:tasks.workunit.client.0.smithi008.stderr:138500K .......... .......... .......... .......... .......... 82% 5.34M 4s 2020-02-05T17:01:20.676 INFO:tasks.workunit.client.0.smithi008.stderr:138550K .......... .......... .......... .......... .......... 82% 4.97M 4s 2020-02-05T17:01:20.686 INFO:tasks.workunit.client.0.smithi008.stderr:138600K .......... .......... .......... .......... .......... 82% 5.20M 4s 2020-02-05T17:01:20.697 INFO:tasks.workunit.client.0.smithi008.stderr:138650K .......... .......... .......... .......... .......... 82% 4.26M 4s 2020-02-05T17:01:20.707 INFO:tasks.workunit.client.0.smithi008.stderr:138700K .......... .......... .......... .......... .......... 82% 5.11M 4s 2020-02-05T17:01:20.718 INFO:tasks.workunit.client.0.smithi008.stderr:138750K .......... .......... .......... .......... .......... 82% 4.07M 4s 2020-02-05T17:01:20.723 INFO:tasks.workunit.client.0.smithi008.stderr:138800K .......... .......... .......... .......... .......... 82% 9.42M 4s 2020-02-05T17:01:20.729 INFO:tasks.workunit.client.0.smithi008.stderr:138850K .......... .......... .......... .......... .......... 82% 8.10M 4s 2020-02-05T17:01:20.733 INFO:tasks.workunit.client.0.smithi008.stderr:138900K .......... .......... .......... .......... .......... 82% 12.9M 4s 2020-02-05T17:01:20.737 INFO:tasks.workunit.client.0.smithi008.stderr:138950K .......... .......... .......... .......... .......... 82% 12.1M 4s 2020-02-05T17:01:20.741 INFO:tasks.workunit.client.0.smithi008.stderr:139000K .......... .......... .......... .......... .......... 82% 11.5M 4s 2020-02-05T17:01:20.747 INFO:tasks.workunit.client.0.smithi008.stderr:139050K .......... .......... .......... .......... .......... 82% 9.31M 4s 2020-02-05T17:01:20.751 INFO:tasks.workunit.client.0.smithi008.stderr:139100K .......... .......... .......... .......... .......... 82% 10.6M 4s 2020-02-05T17:01:20.756 INFO:tasks.workunit.client.0.smithi008.stderr:139150K .......... .......... .......... .......... .......... 82% 10.5M 4s 2020-02-05T17:01:20.760 INFO:tasks.workunit.client.0.smithi008.stderr:139200K .......... .......... .......... .......... .......... 82% 11.6M 4s 2020-02-05T17:01:20.765 INFO:tasks.workunit.client.0.smithi008.stderr:139250K .......... .......... .......... .......... .......... 82% 9.13M 3s 2020-02-05T17:01:20.769 INFO:tasks.workunit.client.0.smithi008.stderr:139300K .......... .......... .......... .......... .......... 82% 12.5M 3s 2020-02-05T17:01:20.777 INFO:tasks.workunit.client.0.smithi008.stderr:139350K .......... .......... .......... .......... .......... 82% 9.02M 3s 2020-02-05T17:01:20.781 INFO:tasks.workunit.client.0.smithi008.stderr:139400K .......... .......... .......... .......... .......... 82% 7.96M 3s 2020-02-05T17:01:20.787 INFO:tasks.workunit.client.0.smithi008.stderr:139450K .......... .......... .......... .......... .......... 82% 8.36M 3s 2020-02-05T17:01:20.792 INFO:tasks.workunit.client.0.smithi008.stderr:139500K .......... .......... .......... .......... .......... 82% 9.91M 3s 2020-02-05T17:01:20.796 INFO:tasks.workunit.client.0.smithi008.stderr:139550K .......... .......... .......... .......... .......... 82% 12.0M 3s 2020-02-05T17:01:20.799 INFO:tasks.workunit.client.0.smithi008.stderr:139600K .......... .......... .......... .......... .......... 82% 13.3M 3s 2020-02-05T17:01:20.804 INFO:tasks.workunit.client.0.smithi008.stderr:139650K .......... .......... .......... .......... .......... 82% 10.3M 3s 2020-02-05T17:01:20.807 INFO:tasks.workunit.client.0.smithi008.stderr:139700K .......... .......... .......... .......... .......... 82% 16.6M 3s 2020-02-05T17:01:20.810 INFO:tasks.workunit.client.0.smithi008.stderr:139750K .......... .......... .......... .......... .......... 82% 16.7M 3s 2020-02-05T17:01:20.829 INFO:tasks.workunit.client.0.smithi008.stderr:139800K .......... .......... .......... .......... .......... 82% 15.2M 3s 2020-02-05T17:01:20.829 INFO:tasks.workunit.client.0.smithi008.stderr:139850K .......... .......... .......... .......... .......... 82% 9.88M 3s 2020-02-05T17:01:20.829 INFO:tasks.workunit.client.0.smithi008.stderr:139900K .......... .......... .......... .......... .......... 83% 10.4M 3s 2020-02-05T17:01:20.829 INFO:tasks.workunit.client.0.smithi008.stderr:139950K .......... .......... .......... .......... .......... 83% 8.55M 3s 2020-02-05T17:01:20.833 INFO:tasks.workunit.client.0.smithi008.stderr:140000K .......... .......... .......... .......... .......... 83% 9.95M 3s 2020-02-05T17:01:20.839 INFO:tasks.workunit.client.0.smithi008.stderr:140050K .......... .......... .......... .......... .......... 83% 8.13M 3s 2020-02-05T17:01:20.844 INFO:tasks.workunit.client.0.smithi008.stderr:140100K .......... .......... .......... .......... .......... 83% 10.6M 3s 2020-02-05T17:01:20.848 INFO:tasks.workunit.client.0.smithi008.stderr:140150K .......... .......... .......... .......... .......... 83% 11.2M 3s 2020-02-05T17:01:20.853 INFO:tasks.workunit.client.0.smithi008.stderr:140200K .......... .......... .......... .......... .......... 83% 11.1M 3s 2020-02-05T17:01:20.858 INFO:tasks.workunit.client.0.smithi008.stderr:140250K .......... .......... .......... .......... .......... 83% 9.99M 3s 2020-02-05T17:01:20.862 INFO:tasks.workunit.client.0.smithi008.stderr:140300K .......... .......... .......... .......... .......... 83% 11.5M 3s 2020-02-05T17:01:20.867 INFO:tasks.workunit.client.0.smithi008.stderr:140350K .......... .......... .......... .......... .......... 83% 10.2M 3s 2020-02-05T17:01:20.873 INFO:tasks.workunit.client.0.smithi008.stderr:140400K .......... .......... .......... .......... .......... 83% 8.68M 3s 2020-02-05T17:01:20.880 INFO:tasks.workunit.client.0.smithi008.stderr:140450K .......... .......... .......... .......... .......... 83% 6.19M 3s 2020-02-05T17:01:20.887 INFO:tasks.workunit.client.0.smithi008.stderr:140500K .......... .......... .......... .......... .......... 83% 7.56M 3s 2020-02-05T17:01:20.893 INFO:tasks.workunit.client.0.smithi008.stderr:140550K .......... .......... .......... .......... .......... 83% 7.71M 3s 2020-02-05T17:01:20.899 INFO:tasks.workunit.client.0.smithi008.stderr:140600K .......... .......... .......... .......... .......... 83% 8.24M 3s 2020-02-05T17:01:20.906 INFO:tasks.workunit.client.0.smithi008.stderr:140650K .......... .......... .......... .......... .......... 83% 7.00M 3s 2020-02-05T17:01:20.912 INFO:tasks.workunit.client.0.smithi008.stderr:140700K .......... .......... .......... .......... .......... 83% 7.77M 3s 2020-02-05T17:01:20.919 INFO:tasks.workunit.client.0.smithi008.stderr:140750K .......... .......... .......... .......... .......... 83% 7.73M 3s 2020-02-05T17:01:20.925 INFO:tasks.workunit.client.0.smithi008.stderr:140800K .......... .......... .......... .......... .......... 83% 7.66M 3s 2020-02-05T17:01:20.933 INFO:tasks.workunit.client.0.smithi008.stderr:140850K .......... .......... .......... .......... .......... 83% 6.41M 3s 2020-02-05T17:01:20.940 INFO:tasks.workunit.client.0.smithi008.stderr:140900K .......... .......... .......... .......... .......... 83% 6.95M 3s 2020-02-05T17:01:20.950 INFO:tasks.workunit.client.0.smithi008.stderr:140950K .......... .......... .......... .......... .......... 83% 5.00M 3s 2020-02-05T17:01:20.955 INFO:tasks.workunit.client.0.smithi008.stderr:141000K .......... .......... .......... .......... .......... 83% 8.82M 3s 2020-02-05T17:01:20.963 INFO:tasks.workunit.client.0.smithi008.stderr:141050K .......... .......... .......... .......... .......... 83% 6.08M 3s 2020-02-05T17:01:20.970 INFO:tasks.workunit.client.0.smithi008.stderr:141100K .......... .......... .......... .......... .......... 83% 7.25M 3s 2020-02-05T17:01:20.976 INFO:tasks.workunit.client.0.smithi008.stderr:141150K .......... .......... .......... .......... .......... 83% 7.56M 3s 2020-02-05T17:01:20.982 INFO:tasks.workunit.client.0.smithi008.stderr:141200K .......... .......... .......... .......... .......... 83% 8.11M 3s 2020-02-05T17:01:20.989 INFO:tasks.workunit.client.0.smithi008.stderr:141250K .......... .......... .......... .......... .......... 83% 7.15M 3s 2020-02-05T17:01:20.995 INFO:tasks.workunit.client.0.smithi008.stderr:141300K .......... .......... .......... .......... .......... 83% 7.88M 3s 2020-02-05T17:01:21.002 INFO:tasks.workunit.client.0.smithi008.stderr:141350K .......... .......... .......... .......... .......... 83% 7.87M 3s 2020-02-05T17:01:21.009 INFO:tasks.workunit.client.0.smithi008.stderr:141400K .......... .......... .......... .......... .......... 83% 6.74M 3s 2020-02-05T17:01:21.016 INFO:tasks.workunit.client.0.smithi008.stderr:141450K .......... .......... .......... .......... .......... 83% 6.41M 3s 2020-02-05T17:01:21.025 INFO:tasks.workunit.client.0.smithi008.stderr:141500K .......... .......... .......... .......... .......... 83% 6.90M 3s 2020-02-05T17:01:21.031 INFO:tasks.workunit.client.0.smithi008.stderr:141550K .......... .......... .......... .......... .......... 84% 6.61M 3s 2020-02-05T17:01:21.038 INFO:tasks.workunit.client.0.smithi008.stderr:141600K .......... .......... .......... .......... .......... 84% 7.07M 3s 2020-02-05T17:01:21.045 INFO:tasks.workunit.client.0.smithi008.stderr:141650K .......... .......... .......... .......... .......... 84% 6.97M 3s 2020-02-05T17:01:21.050 INFO:tasks.workunit.client.0.smithi008.stderr:141700K .......... .......... .......... .......... .......... 84% 9.51M 3s 2020-02-05T17:01:21.055 INFO:tasks.workunit.client.0.smithi008.stderr:141750K .......... .......... .......... .......... .......... 84% 8.64M 3s 2020-02-05T17:01:21.061 INFO:tasks.workunit.client.0.smithi008.stderr:141800K .......... .......... .......... .......... .......... 84% 9.34M 3s 2020-02-05T17:01:21.067 INFO:tasks.workunit.client.0.smithi008.stderr:141850K .......... .......... .......... .......... .......... 84% 7.55M 3s 2020-02-05T17:01:21.073 INFO:tasks.workunit.client.0.smithi008.stderr:141900K .......... .......... .......... .......... .......... 84% 8.59M 3s 2020-02-05T17:01:21.079 INFO:tasks.workunit.client.0.smithi008.stderr:141950K .......... .......... .......... .......... .......... 84% 7.78M 3s 2020-02-05T17:01:21.086 INFO:tasks.workunit.client.0.smithi008.stderr:142000K .......... .......... .......... .......... .......... 84% 7.84M 3s 2020-02-05T17:01:21.094 INFO:tasks.workunit.client.0.smithi008.stderr:142050K .......... .......... .......... .......... .......... 84% 6.15M 3s 2020-02-05T17:01:21.101 INFO:tasks.workunit.client.0.smithi008.stderr:142100K .......... .......... .......... .......... .......... 84% 6.99M 3s 2020-02-05T17:01:21.108 INFO:tasks.workunit.client.0.smithi008.stderr:142150K .......... .......... .......... .......... .......... 84% 6.42M 3s 2020-02-05T17:01:21.115 INFO:tasks.workunit.client.0.smithi008.stderr:142200K .......... .......... .......... .......... .......... 84% 6.66M 3s 2020-02-05T17:01:21.125 INFO:tasks.workunit.client.0.smithi008.stderr:142250K .......... .......... .......... .......... .......... 84% 5.35M 3s 2020-02-05T17:01:21.133 INFO:tasks.workunit.client.0.smithi008.stderr:142300K .......... .......... .......... .......... .......... 84% 6.16M 3s 2020-02-05T17:01:21.141 INFO:tasks.workunit.client.0.smithi008.stderr:142350K .......... .......... .......... .......... .......... 84% 6.04M 3s 2020-02-05T17:01:21.149 INFO:tasks.workunit.client.0.smithi008.stderr:142400K .......... .......... .......... .......... .......... 84% 6.22M 3s 2020-02-05T17:01:21.158 INFO:tasks.workunit.client.0.smithi008.stderr:142450K .......... .......... .......... .......... .......... 84% 5.02M 3s 2020-02-05T17:01:21.166 INFO:tasks.workunit.client.0.smithi008.stderr:142500K .......... .......... .......... .......... .......... 84% 6.43M 3s 2020-02-05T17:01:21.173 INFO:tasks.workunit.client.0.smithi008.stderr:142550K .......... .......... .......... .......... .......... 84% 6.57M 3s 2020-02-05T17:01:21.180 INFO:tasks.workunit.client.0.smithi008.stderr:142600K .......... .......... .......... .......... .......... 84% 6.60M 3s 2020-02-05T17:01:21.188 INFO:tasks.workunit.client.0.smithi008.stderr:142650K .......... .......... .......... .......... .......... 84% 6.46M 3s 2020-02-05T17:01:21.194 INFO:tasks.workunit.client.0.smithi008.stderr:142700K .......... .......... .......... .......... .......... 84% 7.93M 3s 2020-02-05T17:01:21.199 INFO:tasks.workunit.client.0.smithi008.stderr:142750K .......... .......... .......... .......... .......... 84% 10.3M 3s 2020-02-05T17:01:21.203 INFO:tasks.workunit.client.0.smithi008.stderr:142800K .......... .......... .......... .......... .......... 84% 10.6M 3s 2020-02-05T17:01:21.212 INFO:tasks.workunit.client.0.smithi008.stderr:142850K .......... .......... .......... .......... .......... 84% 5.42M 3s 2020-02-05T17:01:21.215 INFO:tasks.workunit.client.0.smithi008.stderr:142900K .......... .......... .......... .......... .......... 84% 15.2M 3s 2020-02-05T17:01:21.219 INFO:tasks.workunit.client.0.smithi008.stderr:142950K .......... .......... .......... .......... .......... 84% 14.7M 3s 2020-02-05T17:01:21.222 INFO:tasks.workunit.client.0.smithi008.stderr:143000K .......... .......... .......... .......... .......... 84% 13.7M 3s 2020-02-05T17:01:21.226 INFO:tasks.workunit.client.0.smithi008.stderr:143050K .......... .......... .......... .......... .......... 84% 11.7M 3s 2020-02-05T17:01:21.230 INFO:tasks.workunit.client.0.smithi008.stderr:143100K .......... .......... .......... .......... .......... 84% 13.5M 3s 2020-02-05T17:01:21.235 INFO:tasks.workunit.client.0.smithi008.stderr:143150K .......... .......... .......... .......... .......... 84% 9.91M 3s 2020-02-05T17:01:21.240 INFO:tasks.workunit.client.0.smithi008.stderr:143200K .......... .......... .......... .......... .......... 84% 9.79M 3s 2020-02-05T17:01:21.246 INFO:tasks.workunit.client.0.smithi008.stderr:143250K .......... .......... .......... .......... .......... 85% 8.75M 3s 2020-02-05T17:01:21.251 INFO:tasks.workunit.client.0.smithi008.stderr:143300K .......... .......... .......... .......... .......... 85% 9.35M 3s 2020-02-05T17:01:21.256 INFO:tasks.workunit.client.0.smithi008.stderr:143350K .......... .......... .......... .......... .......... 85% 9.26M 3s 2020-02-05T17:01:21.260 INFO:tasks.workunit.client.0.smithi008.stderr:143400K .......... .......... .......... .......... .......... 85% 11.5M 3s 2020-02-05T17:01:21.265 INFO:tasks.workunit.client.0.smithi008.stderr:143450K .......... .......... .......... .......... .......... 85% 11.1M 3s 2020-02-05T17:01:21.270 INFO:tasks.workunit.client.0.smithi008.stderr:143500K .......... .......... .......... .......... .......... 85% 10.2M 3s 2020-02-05T17:01:21.275 INFO:tasks.workunit.client.0.smithi008.stderr:143550K .......... .......... .......... .......... .......... 85% 9.37M 3s 2020-02-05T17:01:21.281 INFO:tasks.workunit.client.0.smithi008.stderr:143600K .......... .......... .......... .......... .......... 85% 8.52M 3s 2020-02-05T17:01:21.285 INFO:tasks.workunit.client.0.smithi008.stderr:143650K .......... .......... .......... .......... .......... 85% 10.3M 3s 2020-02-05T17:01:21.289 INFO:tasks.workunit.client.0.smithi008.stderr:143700K .......... .......... .......... .......... .......... 85% 13.3M 3s 2020-02-05T17:01:21.292 INFO:tasks.workunit.client.0.smithi008.stderr:143750K .......... .......... .......... .......... .......... 85% 12.4M 3s 2020-02-05T17:01:21.295 INFO:tasks.workunit.client.0.smithi008.stderr:143800K .......... .......... .......... .......... .......... 85% 16.7M 3s 2020-02-05T17:01:21.299 INFO:tasks.workunit.client.0.smithi008.stderr:143850K .......... .......... .......... .......... .......... 85% 12.7M 3s 2020-02-05T17:01:21.304 INFO:tasks.workunit.client.0.smithi008.stderr:143900K .......... .......... .......... .......... .......... 85% 11.3M 3s 2020-02-05T17:01:21.308 INFO:tasks.workunit.client.0.smithi008.stderr:143950K .......... .......... .......... .......... .......... 85% 11.1M 3s 2020-02-05T17:01:21.312 INFO:tasks.workunit.client.0.smithi008.stderr:144000K .......... .......... .......... .......... .......... 85% 11.7M 3s 2020-02-05T17:01:21.318 INFO:tasks.workunit.client.0.smithi008.stderr:144050K .......... .......... .......... .......... .......... 85% 8.92M 3s 2020-02-05T17:01:21.323 INFO:tasks.workunit.client.0.smithi008.stderr:144100K .......... .......... .......... .......... .......... 85% 9.25M 3s 2020-02-05T17:01:21.330 INFO:tasks.workunit.client.0.smithi008.stderr:144150K .......... .......... .......... .......... .......... 85% 7.15M 3s 2020-02-05T17:01:21.335 INFO:tasks.workunit.client.0.smithi008.stderr:144200K .......... .......... .......... .......... .......... 85% 10.3M 3s 2020-02-05T17:01:21.340 INFO:tasks.workunit.client.0.smithi008.stderr:144250K .......... .......... .......... .......... .......... 85% 9.74M 3s 2020-02-05T17:01:21.344 INFO:tasks.workunit.client.0.smithi008.stderr:144300K .......... .......... .......... .......... .......... 85% 11.6M 3s 2020-02-05T17:01:21.348 INFO:tasks.workunit.client.0.smithi008.stderr:144350K .......... .......... .......... .......... .......... 85% 11.8M 3s 2020-02-05T17:01:21.352 INFO:tasks.workunit.client.0.smithi008.stderr:144400K .......... .......... .......... .......... .......... 85% 11.3M 3s 2020-02-05T17:01:21.359 INFO:tasks.workunit.client.0.smithi008.stderr:144450K .......... .......... .......... .......... .......... 85% 8.38M 3s 2020-02-05T17:01:21.365 INFO:tasks.workunit.client.0.smithi008.stderr:144500K .......... .......... .......... .......... .......... 85% 7.90M 3s 2020-02-05T17:01:21.370 INFO:tasks.workunit.client.0.smithi008.stderr:144550K .......... .......... .......... .......... .......... 85% 8.69M 3s 2020-02-05T17:01:21.377 INFO:tasks.workunit.client.0.smithi008.stderr:144600K .......... .......... .......... .......... .......... 85% 7.52M 3s 2020-02-05T17:01:21.384 INFO:tasks.workunit.client.0.smithi008.stderr:144650K .......... .......... .......... .......... .......... 85% 6.38M 3s 2020-02-05T17:01:21.390 INFO:tasks.workunit.client.0.smithi008.stderr:144700K .......... .......... .......... .......... .......... 85% 9.12M 3s 2020-02-05T17:01:21.395 INFO:tasks.workunit.client.0.smithi008.stderr:144750K .......... .......... .......... .......... .......... 85% 9.22M 3s 2020-02-05T17:01:21.400 INFO:tasks.workunit.client.0.smithi008.stderr:144800K .......... .......... .......... .......... .......... 85% 10.0M 3s 2020-02-05T17:01:21.406 INFO:tasks.workunit.client.0.smithi008.stderr:144850K .......... .......... .......... .......... .......... 85% 8.26M 3s 2020-02-05T17:01:21.411 INFO:tasks.workunit.client.0.smithi008.stderr:144900K .......... .......... .......... .......... .......... 85% 8.79M 3s 2020-02-05T17:01:21.417 INFO:tasks.workunit.client.0.smithi008.stderr:144950K .......... .......... .......... .......... .......... 86% 8.74M 3s 2020-02-05T17:01:21.423 INFO:tasks.workunit.client.0.smithi008.stderr:145000K .......... .......... .......... .......... .......... 86% 8.58M 3s 2020-02-05T17:01:21.433 INFO:tasks.workunit.client.0.smithi008.stderr:145050K .......... .......... .......... .......... .......... 86% 5.37M 3s 2020-02-05T17:01:21.438 INFO:tasks.workunit.client.0.smithi008.stderr:145100K .......... .......... .......... .......... .......... 86% 7.90M 3s 2020-02-05T17:01:21.444 INFO:tasks.workunit.client.0.smithi008.stderr:145150K .......... .......... .......... .......... .......... 86% 8.13M 3s 2020-02-05T17:01:21.451 INFO:tasks.workunit.client.0.smithi008.stderr:145200K .......... .......... .......... .......... .......... 86% 7.06M 3s 2020-02-05T17:01:21.458 INFO:tasks.workunit.client.0.smithi008.stderr:145250K .......... .......... .......... .......... .......... 86% 7.43M 3s 2020-02-05T17:01:21.464 INFO:tasks.workunit.client.0.smithi008.stderr:145300K .......... .......... .......... .......... .......... 86% 7.87M 3s 2020-02-05T17:01:21.469 INFO:tasks.workunit.client.0.smithi008.stderr:145350K .......... .......... .......... .......... .......... 86% 8.59M 3s 2020-02-05T17:01:21.476 INFO:tasks.workunit.client.0.smithi008.stderr:145400K .......... .......... .......... .......... .......... 86% 7.80M 3s 2020-02-05T17:01:21.484 INFO:tasks.workunit.client.0.smithi008.stderr:145450K .......... .......... .......... .......... .......... 86% 6.08M 3s 2020-02-05T17:01:21.490 INFO:tasks.workunit.client.0.smithi008.stderr:145500K .......... .......... .......... .......... .......... 86% 8.11M 3s 2020-02-05T17:01:21.496 INFO:tasks.workunit.client.0.smithi008.stderr:145550K .......... .......... .......... .......... .......... 86% 8.61M 3s 2020-02-05T17:01:21.502 INFO:tasks.workunit.client.0.smithi008.stderr:145600K .......... .......... .......... .......... .......... 86% 7.42M 3s 2020-02-05T17:01:21.510 INFO:tasks.workunit.client.0.smithi008.stderr:145650K .......... .......... .......... .......... .......... 86% 6.34M 3s 2020-02-05T17:01:21.517 INFO:tasks.workunit.client.0.smithi008.stderr:145700K .......... .......... .......... .......... .......... 86% 7.01M 3s 2020-02-05T17:01:21.524 INFO:tasks.workunit.client.0.smithi008.stderr:145750K .......... .......... .......... .......... .......... 86% 6.64M 3s 2020-02-05T17:01:21.531 INFO:tasks.workunit.client.0.smithi008.stderr:145800K .......... .......... .......... .......... .......... 86% 7.50M 3s 2020-02-05T17:01:21.537 INFO:tasks.workunit.client.0.smithi008.stderr:145850K .......... .......... .......... .......... .......... 86% 7.63M 3s 2020-02-05T17:01:21.543 INFO:tasks.workunit.client.0.smithi008.stderr:145900K .......... .......... .......... .......... .......... 86% 8.15M 3s 2020-02-05T17:01:21.550 INFO:tasks.workunit.client.0.smithi008.stderr:145950K .......... .......... .......... .......... .......... 86% 7.21M 3s 2020-02-05T17:01:21.557 INFO:tasks.workunit.client.0.smithi008.stderr:146000K .......... .......... .......... .......... .......... 86% 7.03M 3s 2020-02-05T17:01:21.566 INFO:tasks.workunit.client.0.smithi008.stderr:146050K .......... .......... .......... .......... .......... 86% 5.50M 3s 2020-02-05T17:01:21.573 INFO:tasks.workunit.client.0.smithi008.stderr:146100K .......... .......... .......... .......... .......... 86% 6.54M 3s 2020-02-05T17:01:21.579 INFO:tasks.workunit.client.0.smithi008.stderr:146150K .......... .......... .......... .......... .......... 86% 7.83M 3s 2020-02-05T17:01:21.585 INFO:tasks.workunit.client.0.smithi008.stderr:146200K .......... .......... .......... .......... .......... 86% 8.86M 3s 2020-02-05T17:01:21.592 INFO:tasks.workunit.client.0.smithi008.stderr:146250K .......... .......... .......... .......... .......... 86% 6.48M 3s 2020-02-05T17:01:21.598 INFO:tasks.workunit.client.0.smithi008.stderr:146300K .......... .......... .......... .......... .......... 86% 8.28M 3s 2020-02-05T17:01:21.604 INFO:tasks.workunit.client.0.smithi008.stderr:146350K .......... .......... .......... .......... .......... 86% 7.93M 3s 2020-02-05T17:01:21.611 INFO:tasks.workunit.client.0.smithi008.stderr:146400K .......... .......... .......... .......... .......... 86% 7.01M 3s 2020-02-05T17:01:21.619 INFO:tasks.workunit.client.0.smithi008.stderr:146450K .......... .......... .......... .......... .......... 86% 6.07M 3s 2020-02-05T17:01:21.626 INFO:tasks.workunit.client.0.smithi008.stderr:146500K .......... .......... .......... .......... .......... 86% 7.64M 3s 2020-02-05T17:01:21.631 INFO:tasks.workunit.client.0.smithi008.stderr:146550K .......... .......... .......... .......... .......... 86% 9.64M 3s 2020-02-05T17:01:21.636 INFO:tasks.workunit.client.0.smithi008.stderr:146600K .......... .......... .......... .......... .......... 86% 9.72M 3s 2020-02-05T17:01:21.643 INFO:tasks.workunit.client.0.smithi008.stderr:146650K .......... .......... .......... .......... .......... 87% 7.23M 3s 2020-02-05T17:01:21.649 INFO:tasks.workunit.client.0.smithi008.stderr:146700K .......... .......... .......... .......... .......... 87% 8.57M 3s 2020-02-05T17:01:21.656 INFO:tasks.workunit.client.0.smithi008.stderr:146750K .......... .......... .......... .......... .......... 87% 6.73M 3s 2020-02-05T17:01:21.662 INFO:tasks.workunit.client.0.smithi008.stderr:146800K .......... .......... .......... .......... .......... 87% 7.20M 3s 2020-02-05T17:01:21.669 INFO:tasks.workunit.client.0.smithi008.stderr:146850K .......... .......... .......... .......... .......... 87% 7.21M 3s 2020-02-05T17:01:21.680 INFO:tasks.workunit.client.0.smithi008.stderr:146900K .......... .......... .......... .......... .......... 87% 7.59M 3s 2020-02-05T17:01:21.684 INFO:tasks.workunit.client.0.smithi008.stderr:146950K .......... .......... .......... .......... .......... 87% 5.55M 3s 2020-02-05T17:01:21.687 INFO:tasks.workunit.client.0.smithi008.stderr:147000K .......... .......... .......... .......... .......... 87% 15.0M 3s 2020-02-05T17:01:21.690 INFO:tasks.workunit.client.0.smithi008.stderr:147050K .......... .......... .......... .......... .......... 87% 14.1M 3s 2020-02-05T17:01:21.694 INFO:tasks.workunit.client.0.smithi008.stderr:147100K .......... .......... .......... .......... .......... 87% 16.1M 3s 2020-02-05T17:01:21.697 INFO:tasks.workunit.client.0.smithi008.stderr:147150K .......... .......... .......... .......... .......... 87% 15.4M 3s 2020-02-05T17:01:21.700 INFO:tasks.workunit.client.0.smithi008.stderr:147200K .......... .......... .......... .......... .......... 87% 14.7M 3s 2020-02-05T17:01:21.704 INFO:tasks.workunit.client.0.smithi008.stderr:147250K .......... .......... .......... .......... .......... 87% 12.9M 3s 2020-02-05T17:01:21.707 INFO:tasks.workunit.client.0.smithi008.stderr:147300K .......... .......... .......... .......... .......... 87% 14.1M 3s 2020-02-05T17:01:21.711 INFO:tasks.workunit.client.0.smithi008.stderr:147350K .......... .......... .......... .......... .......... 87% 13.5M 3s 2020-02-05T17:01:21.715 INFO:tasks.workunit.client.0.smithi008.stderr:147400K .......... .......... .......... .......... .......... 87% 13.6M 3s 2020-02-05T17:01:21.719 INFO:tasks.workunit.client.0.smithi008.stderr:147450K .......... .......... .......... .......... .......... 87% 11.3M 3s 2020-02-05T17:01:21.722 INFO:tasks.workunit.client.0.smithi008.stderr:147500K .......... .......... .......... .......... .......... 87% 16.0M 3s 2020-02-05T17:01:21.725 INFO:tasks.workunit.client.0.smithi008.stderr:147550K .......... .......... .......... .......... .......... 87% 15.0M 3s 2020-02-05T17:01:21.729 INFO:tasks.workunit.client.0.smithi008.stderr:147600K .......... .......... .......... .......... .......... 87% 13.4M 2s 2020-02-05T17:01:21.733 INFO:tasks.workunit.client.0.smithi008.stderr:147650K .......... .......... .......... .......... .......... 87% 11.6M 2s 2020-02-05T17:01:21.737 INFO:tasks.workunit.client.0.smithi008.stderr:147700K .......... .......... .......... .......... .......... 87% 12.1M 2s 2020-02-05T17:01:21.741 INFO:tasks.workunit.client.0.smithi008.stderr:147750K .......... .......... .......... .......... .......... 87% 12.1M 2s 2020-02-05T17:01:21.745 INFO:tasks.workunit.client.0.smithi008.stderr:147800K .......... .......... .......... .......... .......... 87% 13.5M 2s 2020-02-05T17:01:21.748 INFO:tasks.workunit.client.0.smithi008.stderr:147850K .......... .......... .......... .......... .......... 87% 13.8M 2s 2020-02-05T17:01:21.751 INFO:tasks.workunit.client.0.smithi008.stderr:147900K .......... .......... .......... .......... .......... 87% 18.8M 2s 2020-02-05T17:01:21.753 INFO:tasks.workunit.client.0.smithi008.stderr:147950K .......... .......... .......... .......... .......... 87% 18.2M 2s 2020-02-05T17:01:21.756 INFO:tasks.workunit.client.0.smithi008.stderr:148000K .......... .......... .......... .......... .......... 87% 16.6M 2s 2020-02-05T17:01:21.760 INFO:tasks.workunit.client.0.smithi008.stderr:148050K .......... .......... .......... .......... .......... 87% 13.2M 2s 2020-02-05T17:01:21.764 INFO:tasks.workunit.client.0.smithi008.stderr:148100K .......... .......... .......... .......... .......... 87% 14.2M 2s 2020-02-05T17:01:21.767 INFO:tasks.workunit.client.0.smithi008.stderr:148150K .......... .......... .......... .......... .......... 87% 13.2M 2s 2020-02-05T17:01:21.771 INFO:tasks.workunit.client.0.smithi008.stderr:148200K .......... .......... .......... .......... .......... 87% 13.0M 2s 2020-02-05T17:01:21.776 INFO:tasks.workunit.client.0.smithi008.stderr:148250K .......... .......... .......... .......... .......... 87% 10.0M 2s 2020-02-05T17:01:21.779 INFO:tasks.workunit.client.0.smithi008.stderr:148300K .......... .......... .......... .......... .......... 88% 16.8M 2s 2020-02-05T17:01:21.782 INFO:tasks.workunit.client.0.smithi008.stderr:148350K .......... .......... .......... .......... .......... 88% 17.7M 2s 2020-02-05T17:01:21.786 INFO:tasks.workunit.client.0.smithi008.stderr:148400K .......... .......... .......... .......... .......... 88% 12.5M 2s 2020-02-05T17:01:21.791 INFO:tasks.workunit.client.0.smithi008.stderr:148450K .......... .......... .......... .......... .......... 88% 9.40M 2s 2020-02-05T17:01:21.796 INFO:tasks.workunit.client.0.smithi008.stderr:148500K .......... .......... .......... .......... .......... 88% 10.0M 2s 2020-02-05T17:01:21.800 INFO:tasks.workunit.client.0.smithi008.stderr:148550K .......... .......... .......... .......... .......... 88% 10.7M 2s 2020-02-05T17:01:21.805 INFO:tasks.workunit.client.0.smithi008.stderr:148600K .......... .......... .......... .......... .......... 88% 10.5M 2s 2020-02-05T17:01:21.810 INFO:tasks.workunit.client.0.smithi008.stderr:148650K .......... .......... .......... .......... .......... 88% 9.93M 2s 2020-02-05T17:01:21.814 INFO:tasks.workunit.client.0.smithi008.stderr:148700K .......... .......... .......... .......... .......... 88% 11.1M 2s 2020-02-05T17:01:21.819 INFO:tasks.workunit.client.0.smithi008.stderr:148750K .......... .......... .......... .......... .......... 88% 11.1M 2s 2020-02-05T17:01:21.823 INFO:tasks.workunit.client.0.smithi008.stderr:148800K .......... .......... .......... .......... .......... 88% 11.3M 2s 2020-02-05T17:01:21.828 INFO:tasks.workunit.client.0.smithi008.stderr:148850K .......... .......... .......... .......... .......... 88% 9.78M 2s 2020-02-05T17:01:21.837 INFO:tasks.workunit.client.0.smithi008.stderr:148900K .......... .......... .......... .......... .......... 88% 10.7M 2s 2020-02-05T17:01:21.838 INFO:tasks.workunit.client.0.smithi008.stderr:148950K .......... .......... .......... .......... .......... 88% 9.90M 2s 2020-02-05T17:01:21.845 INFO:tasks.workunit.client.0.smithi008.stderr:149000K .......... .......... .......... .......... .......... 88% 8.06M 2s 2020-02-05T17:01:21.851 INFO:tasks.workunit.client.0.smithi008.stderr:149050K .......... .......... .......... .......... .......... 88% 6.39M 2s 2020-02-05T17:01:21.859 INFO:tasks.workunit.client.0.smithi008.stderr:149100K .......... .......... .......... .......... .......... 88% 7.67M 2s 2020-02-05T17:01:21.868 INFO:tasks.workunit.client.0.smithi008.stderr:149150K .......... .......... .......... .......... .......... 88% 5.12M 2s 2020-02-05T17:01:21.873 INFO:tasks.workunit.client.0.smithi008.stderr:149200K .......... .......... .......... .......... .......... 88% 9.02M 2s 2020-02-05T17:01:21.880 INFO:tasks.workunit.client.0.smithi008.stderr:149250K .......... .......... .......... .......... .......... 88% 7.39M 2s 2020-02-05T17:01:21.885 INFO:tasks.workunit.client.0.smithi008.stderr:149300K .......... .......... .......... .......... .......... 88% 8.24M 2s 2020-02-05T17:01:21.892 INFO:tasks.workunit.client.0.smithi008.stderr:149350K .......... .......... .......... .......... .......... 88% 7.61M 2s 2020-02-05T17:01:21.898 INFO:tasks.workunit.client.0.smithi008.stderr:149400K .......... .......... .......... .......... .......... 88% 7.59M 2s 2020-02-05T17:01:21.905 INFO:tasks.workunit.client.0.smithi008.stderr:149450K .......... .......... .......... .......... .......... 88% 7.16M 2s 2020-02-05T17:01:21.911 INFO:tasks.workunit.client.0.smithi008.stderr:149500K .......... .......... .......... .......... .......... 88% 8.27M 2s 2020-02-05T17:01:21.916 INFO:tasks.workunit.client.0.smithi008.stderr:149550K .......... .......... .......... .......... .......... 88% 9.09M 2s 2020-02-05T17:01:21.922 INFO:tasks.workunit.client.0.smithi008.stderr:149600K .......... .......... .......... .......... .......... 88% 8.70M 2s 2020-02-05T17:01:21.929 INFO:tasks.workunit.client.0.smithi008.stderr:149650K .......... .......... .......... .......... .......... 88% 6.87M 2s 2020-02-05T17:01:21.935 INFO:tasks.workunit.client.0.smithi008.stderr:149700K .......... .......... .......... .......... .......... 88% 7.94M 2s 2020-02-05T17:01:21.941 INFO:tasks.workunit.client.0.smithi008.stderr:149750K .......... .......... .......... .......... .......... 88% 7.71M 2s 2020-02-05T17:01:21.948 INFO:tasks.workunit.client.0.smithi008.stderr:149800K .......... .......... .......... .......... .......... 88% 7.54M 2s 2020-02-05T17:01:21.955 INFO:tasks.workunit.client.0.smithi008.stderr:149850K .......... .......... .......... .......... .......... 88% 6.59M 2s 2020-02-05T17:01:21.960 INFO:tasks.workunit.client.0.smithi008.stderr:149900K .......... .......... .......... .......... .......... 88% 10.1M 2s 2020-02-05T17:01:21.984 INFO:tasks.workunit.client.0.smithi008.stderr:149950K .......... .......... .......... .......... .......... 88% 11.8M 2s 2020-02-05T17:01:21.984 INFO:tasks.workunit.client.0.smithi008.stderr:150000K .......... .......... .......... .......... .......... 89% 8.43M 2s 2020-02-05T17:01:21.984 INFO:tasks.workunit.client.0.smithi008.stderr:150050K .......... .......... .......... .......... .......... 89% 7.27M 2s 2020-02-05T17:01:21.984 INFO:tasks.workunit.client.0.smithi008.stderr:150100K .......... .......... .......... .......... .......... 89% 8.41M 2s 2020-02-05T17:01:21.999 INFO:tasks.workunit.client.0.smithi008.stderr:150150K .......... .......... .......... .......... .......... 89% 8.63M 2s 2020-02-05T17:01:21.999 INFO:tasks.workunit.client.0.smithi008.stderr:150200K .......... .......... .......... .......... .......... 89% 7.97M 2s 2020-02-05T17:01:22.002 INFO:tasks.workunit.client.0.smithi008.stderr:150250K .......... .......... .......... .......... .......... 89% 6.34M 2s 2020-02-05T17:01:22.009 INFO:tasks.workunit.client.0.smithi008.stderr:150300K .......... .......... .......... .......... .......... 89% 6.79M 2s 2020-02-05T17:01:22.016 INFO:tasks.workunit.client.0.smithi008.stderr:150350K .......... .......... .......... .......... .......... 89% 6.81M 2s 2020-02-05T17:01:22.024 INFO:tasks.workunit.client.0.smithi008.stderr:150400K .......... .......... .......... .......... .......... 89% 6.41M 2s 2020-02-05T17:01:22.034 INFO:tasks.workunit.client.0.smithi008.stderr:150450K .......... .......... .......... .......... .......... 89% 5.40M 2s 2020-02-05T17:01:22.043 INFO:tasks.workunit.client.0.smithi008.stderr:150500K .......... .......... .......... .......... .......... 89% 5.64M 2s 2020-02-05T17:01:22.050 INFO:tasks.workunit.client.0.smithi008.stderr:150550K .......... .......... .......... .......... .......... 89% 6.53M 2s 2020-02-05T17:01:22.056 INFO:tasks.workunit.client.0.smithi008.stderr:150600K .......... .......... .......... .......... .......... 89% 7.11M 2s 2020-02-05T17:01:22.065 INFO:tasks.workunit.client.0.smithi008.stderr:150650K .......... .......... .......... .......... .......... 89% 5.81M 2s 2020-02-05T17:01:22.072 INFO:tasks.workunit.client.0.smithi008.stderr:150700K .......... .......... .......... .......... .......... 89% 6.71M 2s 2020-02-05T17:01:22.080 INFO:tasks.workunit.client.0.smithi008.stderr:150750K .......... .......... .......... .......... .......... 89% 6.24M 2s 2020-02-05T17:01:22.089 INFO:tasks.workunit.client.0.smithi008.stderr:150800K .......... .......... .......... .......... .......... 89% 5.69M 2s 2020-02-05T17:01:22.099 INFO:tasks.workunit.client.0.smithi008.stderr:150850K .......... .......... .......... .......... .......... 89% 4.76M 2s 2020-02-05T17:01:22.108 INFO:tasks.workunit.client.0.smithi008.stderr:150900K .......... .......... .......... .......... .......... 89% 5.50M 2s 2020-02-05T17:01:22.118 INFO:tasks.workunit.client.0.smithi008.stderr:150950K .......... .......... .......... .......... .......... 89% 5.30M 2s 2020-02-05T17:01:22.135 INFO:tasks.workunit.client.0.smithi008.stderr:151000K .......... .......... .......... .......... .......... 89% 5.30M 2s 2020-02-05T17:01:22.139 INFO:tasks.workunit.client.0.smithi008.stderr:151050K .......... .......... .......... .......... .......... 89% 3.72M 2s 2020-02-05T17:01:22.143 INFO:tasks.workunit.client.0.smithi008.stderr:151100K .......... .......... .......... .......... .......... 89% 10.7M 2s 2020-02-05T17:01:22.149 INFO:tasks.workunit.client.0.smithi008.stderr:151150K .......... .......... .......... .......... .......... 89% 9.99M 2s 2020-02-05T17:01:22.153 INFO:tasks.workunit.client.0.smithi008.stderr:151200K .......... .......... .......... .......... .......... 89% 10.4M 2s 2020-02-05T17:01:22.159 INFO:tasks.workunit.client.0.smithi008.stderr:151250K .......... .......... .......... .......... .......... 89% 8.07M 2s 2020-02-05T17:01:22.164 INFO:tasks.workunit.client.0.smithi008.stderr:151300K .......... .......... .......... .......... .......... 89% 9.19M 2s 2020-02-05T17:01:22.170 INFO:tasks.workunit.client.0.smithi008.stderr:151350K .......... .......... .......... .......... .......... 89% 9.24M 2s 2020-02-05T17:01:22.176 INFO:tasks.workunit.client.0.smithi008.stderr:151400K .......... .......... .......... .......... .......... 89% 8.26M 2s 2020-02-05T17:01:22.183 INFO:tasks.workunit.client.0.smithi008.stderr:151450K .......... .......... .......... .......... .......... 89% 6.78M 2s 2020-02-05T17:01:22.188 INFO:tasks.workunit.client.0.smithi008.stderr:151500K .......... .......... .......... .......... .......... 89% 8.71M 2s 2020-02-05T17:01:22.195 INFO:tasks.workunit.client.0.smithi008.stderr:151550K .......... .......... .......... .......... .......... 89% 7.84M 2s 2020-02-05T17:01:22.201 INFO:tasks.workunit.client.0.smithi008.stderr:151600K .......... .......... .......... .......... .......... 89% 7.69M 2s 2020-02-05T17:01:22.207 INFO:tasks.workunit.client.0.smithi008.stderr:151650K .......... .......... .......... .......... .......... 89% 7.46M 2s 2020-02-05T17:01:22.212 INFO:tasks.workunit.client.0.smithi008.stderr:151700K .......... .......... .......... .......... .......... 90% 10.9M 2s 2020-02-05T17:01:22.215 INFO:tasks.workunit.client.0.smithi008.stderr:151750K .......... .......... .......... .......... .......... 90% 15.0M 2s 2020-02-05T17:01:22.218 INFO:tasks.workunit.client.0.smithi008.stderr:151800K .......... .......... .......... .......... .......... 90% 16.6M 2s 2020-02-05T17:01:22.222 INFO:tasks.workunit.client.0.smithi008.stderr:151850K .......... .......... .......... .......... .......... 90% 11.1M 2s 2020-02-05T17:01:22.226 INFO:tasks.workunit.client.0.smithi008.stderr:151900K .......... .......... .......... .......... .......... 90% 12.9M 2s 2020-02-05T17:01:22.230 INFO:tasks.workunit.client.0.smithi008.stderr:151950K .......... .......... .......... .......... .......... 90% 12.6M 2s 2020-02-05T17:01:22.234 INFO:tasks.workunit.client.0.smithi008.stderr:152000K .......... .......... .......... .......... .......... 90% 11.9M 2s 2020-02-05T17:01:22.240 INFO:tasks.workunit.client.0.smithi008.stderr:152050K .......... .......... .......... .......... .......... 90% 8.91M 2s 2020-02-05T17:01:22.245 INFO:tasks.workunit.client.0.smithi008.stderr:152100K .......... .......... .......... .......... .......... 90% 8.50M 2s 2020-02-05T17:01:22.252 INFO:tasks.workunit.client.0.smithi008.stderr:152150K .......... .......... .......... .......... .......... 90% 8.48M 2s 2020-02-05T17:01:22.257 INFO:tasks.workunit.client.0.smithi008.stderr:152200K .......... .......... .......... .......... .......... 90% 7.90M 2s 2020-02-05T17:01:22.265 INFO:tasks.workunit.client.0.smithi008.stderr:152250K .......... .......... .......... .......... .......... 90% 6.30M 2s 2020-02-05T17:01:22.274 INFO:tasks.workunit.client.0.smithi008.stderr:152300K .......... .......... .......... .......... .......... 90% 6.77M 2s 2020-02-05T17:01:22.281 INFO:tasks.workunit.client.0.smithi008.stderr:152350K .......... .......... .......... .......... .......... 90% 5.61M 2s 2020-02-05T17:01:22.287 INFO:tasks.workunit.client.0.smithi008.stderr:152400K .......... .......... .......... .......... .......... 90% 8.03M 2s 2020-02-05T17:01:22.294 INFO:tasks.workunit.client.0.smithi008.stderr:152450K .......... .......... .......... .......... .......... 90% 6.96M 2s 2020-02-05T17:01:22.300 INFO:tasks.workunit.client.0.smithi008.stderr:152500K .......... .......... .......... .......... .......... 90% 8.66M 2s 2020-02-05T17:01:22.306 INFO:tasks.workunit.client.0.smithi008.stderr:152550K .......... .......... .......... .......... .......... 90% 7.99M 2s 2020-02-05T17:01:22.312 INFO:tasks.workunit.client.0.smithi008.stderr:152600K .......... .......... .......... .......... .......... 90% 7.75M 2s 2020-02-05T17:01:22.321 INFO:tasks.workunit.client.0.smithi008.stderr:152650K .......... .......... .......... .......... .......... 90% 5.68M 2s 2020-02-05T17:01:22.328 INFO:tasks.workunit.client.0.smithi008.stderr:152700K .......... .......... .......... .......... .......... 90% 6.84M 2s 2020-02-05T17:01:22.336 INFO:tasks.workunit.client.0.smithi008.stderr:152750K .......... .......... .......... .......... .......... 90% 6.20M 2s 2020-02-05T17:01:22.345 INFO:tasks.workunit.client.0.smithi008.stderr:152800K .......... .......... .......... .......... .......... 90% 6.15M 2s 2020-02-05T17:01:22.353 INFO:tasks.workunit.client.0.smithi008.stderr:152850K .......... .......... .......... .......... .......... 90% 5.82M 2s 2020-02-05T17:01:22.360 INFO:tasks.workunit.client.0.smithi008.stderr:152900K .......... .......... .......... .......... .......... 90% 6.75M 2s 2020-02-05T17:01:22.367 INFO:tasks.workunit.client.0.smithi008.stderr:152950K .......... .......... .......... .......... .......... 90% 7.27M 2s 2020-02-05T17:01:22.372 INFO:tasks.workunit.client.0.smithi008.stderr:153000K .......... .......... .......... .......... .......... 90% 8.41M 2s 2020-02-05T17:01:22.379 INFO:tasks.workunit.client.0.smithi008.stderr:153050K .......... .......... .......... .......... .......... 90% 6.95M 2s 2020-02-05T17:01:22.385 INFO:tasks.workunit.client.0.smithi008.stderr:153100K .......... .......... .......... .......... .......... 90% 8.09M 2s 2020-02-05T17:01:22.390 INFO:tasks.workunit.client.0.smithi008.stderr:153150K .......... .......... .......... .......... .......... 90% 9.31M 2s 2020-02-05T17:01:22.397 INFO:tasks.workunit.client.0.smithi008.stderr:153200K .......... .......... .......... .......... .......... 90% 8.57M 2s 2020-02-05T17:01:22.405 INFO:tasks.workunit.client.0.smithi008.stderr:153250K .......... .......... .......... .......... .......... 90% 5.34M 2s 2020-02-05T17:01:22.411 INFO:tasks.workunit.client.0.smithi008.stderr:153300K .......... .......... .......... .......... .......... 90% 8.78M 2s 2020-02-05T17:01:22.416 INFO:tasks.workunit.client.0.smithi008.stderr:153350K .......... .......... .......... .......... .......... 91% 8.37M 2s 2020-02-05T17:01:22.423 INFO:tasks.workunit.client.0.smithi008.stderr:153400K .......... .......... .......... .......... .......... 91% 7.91M 2s 2020-02-05T17:01:22.430 INFO:tasks.workunit.client.0.smithi008.stderr:153450K .......... .......... .......... .......... .......... 91% 7.25M 2s 2020-02-05T17:01:22.436 INFO:tasks.workunit.client.0.smithi008.stderr:153500K .......... .......... .......... .......... .......... 91% 7.92M 2s 2020-02-05T17:01:22.442 INFO:tasks.workunit.client.0.smithi008.stderr:153550K .......... .......... .......... .......... .......... 91% 7.95M 2s 2020-02-05T17:01:22.446 INFO:tasks.workunit.client.0.smithi008.stderr:153600K .......... .......... .......... .......... .......... 91% 10.0M 2s 2020-02-05T17:01:22.452 INFO:tasks.workunit.client.0.smithi008.stderr:153650K .......... .......... .......... .......... .......... 91% 8.28M 2s 2020-02-05T17:01:22.456 INFO:tasks.workunit.client.0.smithi008.stderr:153700K .......... .......... .......... .......... .......... 91% 12.0M 2s 2020-02-05T17:01:22.461 INFO:tasks.workunit.client.0.smithi008.stderr:153750K .......... .......... .......... .......... .......... 91% 10.8M 2s 2020-02-05T17:01:22.465 INFO:tasks.workunit.client.0.smithi008.stderr:153800K .......... .......... .......... .......... .......... 91% 11.6M 2s 2020-02-05T17:01:22.471 INFO:tasks.workunit.client.0.smithi008.stderr:153850K .......... .......... .......... .......... .......... 91% 8.13M 2s 2020-02-05T17:01:22.478 INFO:tasks.workunit.client.0.smithi008.stderr:153900K .......... .......... .......... .......... .......... 91% 8.08M 2s 2020-02-05T17:01:22.483 INFO:tasks.workunit.client.0.smithi008.stderr:153950K .......... .......... .......... .......... .......... 91% 8.01M 2s 2020-02-05T17:01:22.491 INFO:tasks.workunit.client.0.smithi008.stderr:154000K .......... .......... .......... .......... .......... 91% 7.89M 2s 2020-02-05T17:01:22.498 INFO:tasks.workunit.client.0.smithi008.stderr:154050K .......... .......... .......... .......... .......... 91% 6.02M 2s 2020-02-05T17:01:22.505 INFO:tasks.workunit.client.0.smithi008.stderr:154100K .......... .......... .......... .......... .......... 91% 7.36M 2s 2020-02-05T17:01:22.512 INFO:tasks.workunit.client.0.smithi008.stderr:154150K .......... .......... .......... .......... .......... 91% 6.62M 2s 2020-02-05T17:01:22.519 INFO:tasks.workunit.client.0.smithi008.stderr:154200K .......... .......... .......... .......... .......... 91% 6.85M 2s 2020-02-05T17:01:22.527 INFO:tasks.workunit.client.0.smithi008.stderr:154250K .......... .......... .......... .......... .......... 91% 5.98M 2s 2020-02-05T17:01:22.535 INFO:tasks.workunit.client.0.smithi008.stderr:154300K .......... .......... .......... .......... .......... 91% 6.71M 2s 2020-02-05T17:01:22.543 INFO:tasks.workunit.client.0.smithi008.stderr:154350K .......... .......... .......... .......... .......... 91% 6.44M 2s 2020-02-05T17:01:22.550 INFO:tasks.workunit.client.0.smithi008.stderr:154400K .......... .......... .......... .......... .......... 91% 6.49M 2s 2020-02-05T17:01:22.559 INFO:tasks.workunit.client.0.smithi008.stderr:154450K .......... .......... .......... .......... .......... 91% 5.26M 2s 2020-02-05T17:01:22.567 INFO:tasks.workunit.client.0.smithi008.stderr:154500K .......... .......... .......... .......... .......... 91% 6.44M 2s 2020-02-05T17:01:22.574 INFO:tasks.workunit.client.0.smithi008.stderr:154550K .......... .......... .......... .......... .......... 91% 6.43M 2s 2020-02-05T17:01:22.580 INFO:tasks.workunit.client.0.smithi008.stderr:154600K .......... .......... .......... .......... .......... 91% 8.22M 2s 2020-02-05T17:01:22.586 INFO:tasks.workunit.client.0.smithi008.stderr:154650K .......... .......... .......... .......... .......... 91% 7.40M 2s 2020-02-05T17:01:22.591 INFO:tasks.workunit.client.0.smithi008.stderr:154700K .......... .......... .......... .......... .......... 91% 9.89M 2s 2020-02-05T17:01:22.597 INFO:tasks.workunit.client.0.smithi008.stderr:154750K .......... .......... .......... .......... .......... 91% 9.35M 2s 2020-02-05T17:01:22.602 INFO:tasks.workunit.client.0.smithi008.stderr:154800K .......... .......... .......... .......... .......... 91% 9.39M 2s 2020-02-05T17:01:22.608 INFO:tasks.workunit.client.0.smithi008.stderr:154850K .......... .......... .......... .......... .......... 91% 8.07M 2s 2020-02-05T17:01:22.614 INFO:tasks.workunit.client.0.smithi008.stderr:154900K .......... .......... .......... .......... .......... 91% 9.07M 2s 2020-02-05T17:01:22.619 INFO:tasks.workunit.client.0.smithi008.stderr:154950K .......... .......... .......... .......... .......... 91% 8.29M 2s 2020-02-05T17:01:22.625 INFO:tasks.workunit.client.0.smithi008.stderr:155000K .......... .......... .......... .......... .......... 91% 8.72M 2s 2020-02-05T17:01:22.631 INFO:tasks.workunit.client.0.smithi008.stderr:155050K .......... .......... .......... .......... .......... 92% 7.69M 2s 2020-02-05T17:01:22.642 INFO:tasks.workunit.client.0.smithi008.stderr:155100K .......... .......... .......... .......... .......... 92% 4.29M 2s 2020-02-05T17:01:22.648 INFO:tasks.workunit.client.0.smithi008.stderr:155150K .......... .......... .......... .......... .......... 92% 9.21M 2s 2020-02-05T17:01:22.652 INFO:tasks.workunit.client.0.smithi008.stderr:155200K .......... .......... .......... .......... .......... 92% 10.3M 2s 2020-02-05T17:01:22.656 INFO:tasks.workunit.client.0.smithi008.stderr:155250K .......... .......... .......... .......... .......... 92% 13.5M 2s 2020-02-05T17:01:22.659 INFO:tasks.workunit.client.0.smithi008.stderr:155300K .......... .......... .......... .......... .......... 92% 15.1M 2s 2020-02-05T17:01:22.664 INFO:tasks.workunit.client.0.smithi008.stderr:155350K .......... .......... .......... .......... .......... 92% 10.6M 2s 2020-02-05T17:01:22.669 INFO:tasks.workunit.client.0.smithi008.stderr:155400K .......... .......... .......... .......... .......... 92% 9.93M 2s 2020-02-05T17:01:22.677 INFO:tasks.workunit.client.0.smithi008.stderr:155450K .......... .......... .......... .......... .......... 92% 6.66M 2s 2020-02-05T17:01:22.682 INFO:tasks.workunit.client.0.smithi008.stderr:155500K .......... .......... .......... .......... .......... 92% 8.35M 2s 2020-02-05T17:01:22.688 INFO:tasks.workunit.client.0.smithi008.stderr:155550K .......... .......... .......... .......... .......... 92% 8.65M 2s 2020-02-05T17:01:22.694 INFO:tasks.workunit.client.0.smithi008.stderr:155600K .......... .......... .......... .......... .......... 92% 8.34M 2s 2020-02-05T17:01:22.699 INFO:tasks.workunit.client.0.smithi008.stderr:155650K .......... .......... .......... .......... .......... 92% 8.65M 2s 2020-02-05T17:01:22.703 INFO:tasks.workunit.client.0.smithi008.stderr:155700K .......... .......... .......... .......... .......... 92% 13.6M 2s 2020-02-05T17:01:22.708 INFO:tasks.workunit.client.0.smithi008.stderr:155750K .......... .......... .......... .......... .......... 92% 12.9M 2s 2020-02-05T17:01:22.712 INFO:tasks.workunit.client.0.smithi008.stderr:155800K .......... .......... .......... .......... .......... 92% 9.00M 2s 2020-02-05T17:01:22.716 INFO:tasks.workunit.client.0.smithi008.stderr:155850K .......... .......... .......... .......... .......... 92% 11.1M 2s 2020-02-05T17:01:22.719 INFO:tasks.workunit.client.0.smithi008.stderr:155900K .......... .......... .......... .......... .......... 92% 14.9M 2s 2020-02-05T17:01:22.722 INFO:tasks.workunit.client.0.smithi008.stderr:155950K .......... .......... .......... .......... .......... 92% 14.5M 1s 2020-02-05T17:01:22.726 INFO:tasks.workunit.client.0.smithi008.stderr:156000K .......... .......... .......... .......... .......... 92% 14.4M 1s 2020-02-05T17:01:22.730 INFO:tasks.workunit.client.0.smithi008.stderr:156050K .......... .......... .......... .......... .......... 92% 11.5M 1s 2020-02-05T17:01:22.734 INFO:tasks.workunit.client.0.smithi008.stderr:156100K .......... .......... .......... .......... .......... 92% 13.8M 1s 2020-02-05T17:01:22.737 INFO:tasks.workunit.client.0.smithi008.stderr:156150K .......... .......... .......... .......... .......... 92% 16.1M 1s 2020-02-05T17:01:22.740 INFO:tasks.workunit.client.0.smithi008.stderr:156200K .......... .......... .......... .......... .......... 92% 15.8M 1s 2020-02-05T17:01:22.744 INFO:tasks.workunit.client.0.smithi008.stderr:156250K .......... .......... .......... .......... .......... 92% 11.4M 1s 2020-02-05T17:01:22.749 INFO:tasks.workunit.client.0.smithi008.stderr:156300K .......... .......... .......... .......... .......... 92% 12.3M 1s 2020-02-05T17:01:22.752 INFO:tasks.workunit.client.0.smithi008.stderr:156350K .......... .......... .......... .......... .......... 92% 11.9M 1s 2020-02-05T17:01:22.757 INFO:tasks.workunit.client.0.smithi008.stderr:156400K .......... .......... .......... .......... .......... 92% 10.5M 1s 2020-02-05T17:01:22.764 INFO:tasks.workunit.client.0.smithi008.stderr:156450K .......... .......... .......... .......... .......... 92% 7.30M 1s 2020-02-05T17:01:22.770 INFO:tasks.workunit.client.0.smithi008.stderr:156500K .......... .......... .......... .......... .......... 92% 8.27M 1s 2020-02-05T17:01:22.776 INFO:tasks.workunit.client.0.smithi008.stderr:156550K .......... .......... .......... .......... .......... 92% 7.95M 1s 2020-02-05T17:01:22.782 INFO:tasks.workunit.client.0.smithi008.stderr:156600K .......... .......... .......... .......... .......... 92% 7.77M 1s 2020-02-05T17:01:22.790 INFO:tasks.workunit.client.0.smithi008.stderr:156650K .......... .......... .......... .......... .......... 92% 6.68M 1s 2020-02-05T17:01:22.797 INFO:tasks.workunit.client.0.smithi008.stderr:156700K .......... .......... .......... .......... .......... 92% 7.04M 1s 2020-02-05T17:01:22.804 INFO:tasks.workunit.client.0.smithi008.stderr:156750K .......... .......... .......... .......... .......... 93% 6.76M 1s 2020-02-05T17:01:22.812 INFO:tasks.workunit.client.0.smithi008.stderr:156800K .......... .......... .......... .......... .......... 93% 6.58M 1s 2020-02-05T17:01:22.821 INFO:tasks.workunit.client.0.smithi008.stderr:156850K .......... .......... .......... .......... .......... 93% 5.38M 1s 2020-02-05T17:01:22.828 INFO:tasks.workunit.client.0.smithi008.stderr:156900K .......... .......... .......... .......... .......... 93% 6.59M 1s 2020-02-05T17:01:22.834 INFO:tasks.workunit.client.0.smithi008.stderr:156950K .......... .......... .......... .......... .......... 93% 8.28M 1s 2020-02-05T17:01:22.840 INFO:tasks.workunit.client.0.smithi008.stderr:157000K .......... .......... .......... .......... .......... 93% 8.02M 1s 2020-02-05T17:01:22.849 INFO:tasks.workunit.client.0.smithi008.stderr:157050K .......... .......... .......... .......... .......... 93% 5.80M 1s 2020-02-05T17:01:22.857 INFO:tasks.workunit.client.0.smithi008.stderr:157100K .......... .......... .......... .......... .......... 93% 6.09M 1s 2020-02-05T17:01:22.864 INFO:tasks.workunit.client.0.smithi008.stderr:157150K .......... .......... .......... .......... .......... 93% 6.00M 1s 2020-02-05T17:01:22.873 INFO:tasks.workunit.client.0.smithi008.stderr:157200K .......... .......... .......... .......... .......... 93% 5.73M 1s 2020-02-05T17:01:22.879 INFO:tasks.workunit.client.0.smithi008.stderr:157250K .......... .......... .......... .......... .......... 93% 7.72M 1s 2020-02-05T17:01:22.888 INFO:tasks.workunit.client.0.smithi008.stderr:157300K .......... .......... .......... .......... .......... 93% 9.10M 1s 2020-02-05T17:01:22.894 INFO:tasks.workunit.client.0.smithi008.stderr:157350K .......... .......... .......... .......... .......... 93% 5.41M 1s 2020-02-05T17:01:22.903 INFO:tasks.workunit.client.0.smithi008.stderr:157400K .......... .......... .......... .......... .......... 93% 8.21M 1s 2020-02-05T17:01:22.907 INFO:tasks.workunit.client.0.smithi008.stderr:157450K .......... .......... .......... .......... .......... 93% 6.71M 1s 2020-02-05T17:01:22.917 INFO:tasks.workunit.client.0.smithi008.stderr:157500K .......... .......... .......... .......... .......... 93% 7.97M 1s 2020-02-05T17:01:22.920 INFO:tasks.workunit.client.0.smithi008.stderr:157550K .......... .......... .......... .......... .......... 93% 7.16M 1s 2020-02-05T17:01:22.927 INFO:tasks.workunit.client.0.smithi008.stderr:157600K .......... .......... .......... .......... .......... 93% 6.79M 1s 2020-02-05T17:01:22.935 INFO:tasks.workunit.client.0.smithi008.stderr:157650K .......... .......... .......... .......... .......... 93% 6.23M 1s 2020-02-05T17:01:22.942 INFO:tasks.workunit.client.0.smithi008.stderr:157700K .......... .......... .......... .......... .......... 93% 6.95M 1s 2020-02-05T17:01:22.949 INFO:tasks.workunit.client.0.smithi008.stderr:157750K .......... .......... .......... .......... .......... 93% 6.67M 1s 2020-02-05T17:01:22.956 INFO:tasks.workunit.client.0.smithi008.stderr:157800K .......... .......... .......... .......... .......... 93% 6.95M 1s 2020-02-05T17:01:22.965 INFO:tasks.workunit.client.0.smithi008.stderr:157850K .......... .......... .......... .......... .......... 93% 5.77M 1s 2020-02-05T17:01:22.972 INFO:tasks.workunit.client.0.smithi008.stderr:157900K .......... .......... .......... .......... .......... 93% 6.49M 1s 2020-02-05T17:01:22.979 INFO:tasks.workunit.client.0.smithi008.stderr:157950K .......... .......... .......... .......... .......... 93% 6.82M 1s 2020-02-05T17:01:22.985 INFO:tasks.workunit.client.0.smithi008.stderr:158000K .......... .......... .......... .......... .......... 93% 8.06M 1s 2020-02-05T17:01:22.995 INFO:tasks.workunit.client.0.smithi008.stderr:158050K .......... .......... .......... .......... .......... 93% 6.54M 1s 2020-02-05T17:01:22.999 INFO:tasks.workunit.client.0.smithi008.stderr:158100K .......... .......... .......... .......... .......... 93% 7.91M 1s 2020-02-05T17:01:23.006 INFO:tasks.workunit.client.0.smithi008.stderr:158150K .......... .......... .......... .......... .......... 93% 7.02M 1s 2020-02-05T17:01:23.014 INFO:tasks.workunit.client.0.smithi008.stderr:158200K .......... .......... .......... .......... .......... 93% 6.57M 1s 2020-02-05T17:01:23.022 INFO:tasks.workunit.client.0.smithi008.stderr:158250K .......... .......... .......... .......... .......... 93% 5.55M 1s 2020-02-05T17:01:23.029 INFO:tasks.workunit.client.0.smithi008.stderr:158300K .......... .......... .......... .......... .......... 93% 7.10M 1s 2020-02-05T17:01:23.035 INFO:tasks.workunit.client.0.smithi008.stderr:158350K .......... .......... .......... .......... .......... 93% 8.19M 1s 2020-02-05T17:01:23.040 INFO:tasks.workunit.client.0.smithi008.stderr:158400K .......... .......... .......... .......... .......... 93% 9.02M 1s 2020-02-05T17:01:23.049 INFO:tasks.workunit.client.0.smithi008.stderr:158450K .......... .......... .......... .......... .......... 94% 5.87M 1s 2020-02-05T17:01:23.056 INFO:tasks.workunit.client.0.smithi008.stderr:158500K .......... .......... .......... .......... .......... 94% 6.94M 1s 2020-02-05T17:01:23.063 INFO:tasks.workunit.client.0.smithi008.stderr:158550K .......... .......... .......... .......... .......... 94% 7.07M 1s 2020-02-05T17:01:23.069 INFO:tasks.workunit.client.0.smithi008.stderr:158600K .......... .......... .......... .......... .......... 94% 7.53M 1s 2020-02-05T17:01:23.077 INFO:tasks.workunit.client.0.smithi008.stderr:158650K .......... .......... .......... .......... .......... 94% 6.54M 1s 2020-02-05T17:01:23.084 INFO:tasks.workunit.client.0.smithi008.stderr:158700K .......... .......... .......... .......... .......... 94% 6.42M 1s 2020-02-05T17:01:23.091 INFO:tasks.workunit.client.0.smithi008.stderr:158750K .......... .......... .......... .......... .......... 94% 7.13M 1s 2020-02-05T17:01:23.099 INFO:tasks.workunit.client.0.smithi008.stderr:158800K .......... .......... .......... .......... .......... 94% 6.51M 1s 2020-02-05T17:01:23.108 INFO:tasks.workunit.client.0.smithi008.stderr:158850K .......... .......... .......... .......... .......... 94% 5.54M 1s 2020-02-05T17:01:23.115 INFO:tasks.workunit.client.0.smithi008.stderr:158900K .......... .......... .......... .......... .......... 94% 6.15M 1s 2020-02-05T17:01:23.124 INFO:tasks.workunit.client.0.smithi008.stderr:158950K .......... .......... .......... .......... .......... 94% 5.89M 1s 2020-02-05T17:01:23.132 INFO:tasks.workunit.client.0.smithi008.stderr:159000K .......... .......... .......... .......... .......... 94% 5.74M 1s 2020-02-05T17:01:23.142 INFO:tasks.workunit.client.0.smithi008.stderr:159050K .......... .......... .......... .......... .......... 94% 5.17M 1s 2020-02-05T17:01:23.151 INFO:tasks.workunit.client.0.smithi008.stderr:159100K .......... .......... .......... .......... .......... 94% 5.63M 1s 2020-02-05T17:01:23.159 INFO:tasks.workunit.client.0.smithi008.stderr:159150K .......... .......... .......... .......... .......... 94% 5.65M 1s 2020-02-05T17:01:23.173 INFO:tasks.workunit.client.0.smithi008.stderr:159200K .......... .......... .......... .......... .......... 94% 3.39M 1s 2020-02-05T17:01:23.181 INFO:tasks.workunit.client.0.smithi008.stderr:159250K .......... .......... .......... .......... .......... 94% 6.53M 1s 2020-02-05T17:01:23.187 INFO:tasks.workunit.client.0.smithi008.stderr:159300K .......... .......... .......... .......... .......... 94% 7.26M 1s 2020-02-05T17:01:23.193 INFO:tasks.workunit.client.0.smithi008.stderr:159350K .......... .......... .......... .......... .......... 94% 8.67M 1s 2020-02-05T17:01:23.199 INFO:tasks.workunit.client.0.smithi008.stderr:159400K .......... .......... .......... .......... .......... 94% 7.92M 1s 2020-02-05T17:01:23.207 INFO:tasks.workunit.client.0.smithi008.stderr:159450K .......... .......... .......... .......... .......... 94% 6.26M 1s 2020-02-05T17:01:23.213 INFO:tasks.workunit.client.0.smithi008.stderr:159500K .......... .......... .......... .......... .......... 94% 7.70M 1s 2020-02-05T17:01:23.220 INFO:tasks.workunit.client.0.smithi008.stderr:159550K .......... .......... .......... .......... .......... 94% 7.13M 1s 2020-02-05T17:01:23.227 INFO:tasks.workunit.client.0.smithi008.stderr:159600K .......... .......... .......... .......... .......... 94% 7.11M 1s 2020-02-05T17:01:23.235 INFO:tasks.workunit.client.0.smithi008.stderr:159650K .......... .......... .......... .......... .......... 94% 6.03M 1s 2020-02-05T17:01:23.242 INFO:tasks.workunit.client.0.smithi008.stderr:159700K .......... .......... .......... .......... .......... 94% 6.88M 1s 2020-02-05T17:01:23.250 INFO:tasks.workunit.client.0.smithi008.stderr:159750K .......... .......... .......... .......... .......... 94% 6.71M 1s 2020-02-05T17:01:23.256 INFO:tasks.workunit.client.0.smithi008.stderr:159800K .......... .......... .......... .......... .......... 94% 7.30M 1s 2020-02-05T17:01:23.263 INFO:tasks.workunit.client.0.smithi008.stderr:159850K .......... .......... .......... .......... .......... 94% 7.39M 1s 2020-02-05T17:01:23.267 INFO:tasks.workunit.client.0.smithi008.stderr:159900K .......... .......... .......... .......... .......... 94% 10.5M 1s 2020-02-05T17:01:23.271 INFO:tasks.workunit.client.0.smithi008.stderr:159950K .......... .......... .......... .......... .......... 94% 13.8M 1s 2020-02-05T17:01:23.274 INFO:tasks.workunit.client.0.smithi008.stderr:160000K .......... .......... .......... .......... .......... 94% 14.4M 1s 2020-02-05T17:01:23.278 INFO:tasks.workunit.client.0.smithi008.stderr:160050K .......... .......... .......... .......... .......... 94% 11.3M 1s 2020-02-05T17:01:23.282 INFO:tasks.workunit.client.0.smithi008.stderr:160100K .......... .......... .......... .......... .......... 95% 12.0M 1s 2020-02-05T17:01:23.286 INFO:tasks.workunit.client.0.smithi008.stderr:160150K .......... .......... .......... .......... .......... 95% 12.8M 1s 2020-02-05T17:01:23.289 INFO:tasks.workunit.client.0.smithi008.stderr:160200K .......... .......... .......... .......... .......... 95% 16.3M 1s 2020-02-05T17:01:23.293 INFO:tasks.workunit.client.0.smithi008.stderr:160250K .......... .......... .......... .......... .......... 95% 13.5M 1s 2020-02-05T17:01:23.295 INFO:tasks.workunit.client.0.smithi008.stderr:160300K .......... .......... .......... .......... .......... 95% 17.9M 1s 2020-02-05T17:01:23.298 INFO:tasks.workunit.client.0.smithi008.stderr:160350K .......... .......... .......... .......... .......... 95% 17.3M 1s 2020-02-05T17:01:23.301 INFO:tasks.workunit.client.0.smithi008.stderr:160400K .......... .......... .......... .......... .......... 95% 17.4M 1s 2020-02-05T17:01:23.304 INFO:tasks.workunit.client.0.smithi008.stderr:160450K .......... .......... .......... .......... .......... 95% 15.5M 1s 2020-02-05T17:01:23.307 INFO:tasks.workunit.client.0.smithi008.stderr:160500K .......... .......... .......... .......... .......... 95% 18.4M 1s 2020-02-05T17:01:23.310 INFO:tasks.workunit.client.0.smithi008.stderr:160550K .......... .......... .......... .......... .......... 95% 13.9M 1s 2020-02-05T17:01:23.313 INFO:tasks.workunit.client.0.smithi008.stderr:160600K .......... .......... .......... .......... .......... 95% 19.2M 1s 2020-02-05T17:01:23.316 INFO:tasks.workunit.client.0.smithi008.stderr:160650K .......... .......... .......... .......... .......... 95% 16.9M 1s 2020-02-05T17:01:23.318 INFO:tasks.workunit.client.0.smithi008.stderr:160700K .......... .......... .......... .......... .......... 95% 20.6M 1s 2020-02-05T17:01:23.321 INFO:tasks.workunit.client.0.smithi008.stderr:160750K .......... .......... .......... .......... .......... 95% 19.4M 1s 2020-02-05T17:01:23.323 INFO:tasks.workunit.client.0.smithi008.stderr:160800K .......... .......... .......... .......... .......... 95% 18.6M 1s 2020-02-05T17:01:23.326 INFO:tasks.workunit.client.0.smithi008.stderr:160850K .......... .......... .......... .......... .......... 95% 16.6M 1s 2020-02-05T17:01:23.329 INFO:tasks.workunit.client.0.smithi008.stderr:160900K .......... .......... .......... .......... .......... 95% 18.6M 1s 2020-02-05T17:01:23.332 INFO:tasks.workunit.client.0.smithi008.stderr:160950K .......... .......... .......... .......... .......... 95% 18.6M 1s 2020-02-05T17:01:23.334 INFO:tasks.workunit.client.0.smithi008.stderr:161000K .......... .......... .......... .......... .......... 95% 20.1M 1s 2020-02-05T17:01:23.337 INFO:tasks.workunit.client.0.smithi008.stderr:161050K .......... .......... .......... .......... .......... 95% 17.5M 1s 2020-02-05T17:01:23.339 INFO:tasks.workunit.client.0.smithi008.stderr:161100K .......... .......... .......... .......... .......... 95% 18.1M 1s 2020-02-05T17:01:23.343 INFO:tasks.workunit.client.0.smithi008.stderr:161150K .......... .......... .......... .......... .......... 95% 13.2M 1s 2020-02-05T17:01:23.347 INFO:tasks.workunit.client.0.smithi008.stderr:161200K .......... .......... .......... .......... .......... 95% 12.9M 1s 2020-02-05T17:01:23.351 INFO:tasks.workunit.client.0.smithi008.stderr:161250K .......... .......... .......... .......... .......... 95% 13.0M 1s 2020-02-05T17:01:23.354 INFO:tasks.workunit.client.0.smithi008.stderr:161300K .......... .......... .......... .......... .......... 95% 14.4M 1s 2020-02-05T17:01:23.358 INFO:tasks.workunit.client.0.smithi008.stderr:161350K .......... .......... .......... .......... .......... 95% 13.2M 1s 2020-02-05T17:01:23.361 INFO:tasks.workunit.client.0.smithi008.stderr:161400K .......... .......... .......... .......... .......... 95% 13.6M 1s 2020-02-05T17:01:23.368 INFO:tasks.workunit.client.0.smithi008.stderr:161450K .......... .......... .......... .......... .......... 95% 7.70M 1s 2020-02-05T17:01:23.371 INFO:tasks.workunit.client.0.smithi008.stderr:161500K .......... .......... .......... .......... .......... 95% 15.4M 1s 2020-02-05T17:01:23.374 INFO:tasks.workunit.client.0.smithi008.stderr:161550K .......... .......... .......... .......... .......... 95% 15.2M 1s 2020-02-05T17:01:23.377 INFO:tasks.workunit.client.0.smithi008.stderr:161600K .......... .......... .......... .......... .......... 95% 15.0M 1s 2020-02-05T17:01:23.381 INFO:tasks.workunit.client.0.smithi008.stderr:161650K .......... .......... .......... .......... .......... 95% 12.2M 1s 2020-02-05T17:01:23.386 INFO:tasks.workunit.client.0.smithi008.stderr:161700K .......... .......... .......... .......... .......... 95% 12.6M 1s 2020-02-05T17:01:23.391 INFO:tasks.workunit.client.0.smithi008.stderr:161750K .......... .......... .......... .......... .......... 95% 9.20M 1s 2020-02-05T17:01:23.397 INFO:tasks.workunit.client.0.smithi008.stderr:161800K .......... .......... .......... .......... .......... 96% 8.45M 1s 2020-02-05T17:01:23.404 INFO:tasks.workunit.client.0.smithi008.stderr:161850K .......... .......... .......... .......... .......... 96% 7.00M 1s 2020-02-05T17:01:23.410 INFO:tasks.workunit.client.0.smithi008.stderr:161900K .......... .......... .......... .......... .......... 96% 8.12M 1s 2020-02-05T17:01:23.416 INFO:tasks.workunit.client.0.smithi008.stderr:161950K .......... .......... .......... .......... .......... 96% 7.80M 1s 2020-02-05T17:01:23.422 INFO:tasks.workunit.client.0.smithi008.stderr:162000K .......... .......... .......... .......... .......... 96% 7.93M 1s 2020-02-05T17:01:23.430 INFO:tasks.workunit.client.0.smithi008.stderr:162050K .......... .......... .......... .......... .......... 96% 6.19M 1s 2020-02-05T17:01:23.437 INFO:tasks.workunit.client.0.smithi008.stderr:162100K .......... .......... .......... .......... .......... 96% 7.43M 1s 2020-02-05T17:01:23.443 INFO:tasks.workunit.client.0.smithi008.stderr:162150K .......... .......... .......... .......... .......... 96% 7.65M 1s 2020-02-05T17:01:23.450 INFO:tasks.workunit.client.0.smithi008.stderr:162200K .......... .......... .......... .......... .......... 96% 6.85M 1s 2020-02-05T17:01:23.458 INFO:tasks.workunit.client.0.smithi008.stderr:162250K .......... .......... .......... .......... .......... 96% 6.10M 1s 2020-02-05T17:01:23.465 INFO:tasks.workunit.client.0.smithi008.stderr:162300K .......... .......... .......... .......... .......... 96% 7.68M 1s 2020-02-05T17:01:23.471 INFO:tasks.workunit.client.0.smithi008.stderr:162350K .......... .......... .......... .......... .......... 96% 7.75M 1s 2020-02-05T17:01:23.477 INFO:tasks.workunit.client.0.smithi008.stderr:162400K .......... .......... .......... .......... .......... 96% 8.36M 1s 2020-02-05T17:01:23.483 INFO:tasks.workunit.client.0.smithi008.stderr:162450K .......... .......... .......... .......... .......... 96% 7.79M 1s 2020-02-05T17:01:23.490 INFO:tasks.workunit.client.0.smithi008.stderr:162500K .......... .......... .......... .......... .......... 96% 7.94M 1s 2020-02-05T17:01:23.495 INFO:tasks.workunit.client.0.smithi008.stderr:162550K .......... .......... .......... .......... .......... 96% 8.29M 1s 2020-02-05T17:01:23.501 INFO:tasks.workunit.client.0.smithi008.stderr:162600K .......... .......... .......... .......... .......... 96% 8.24M 1s 2020-02-05T17:01:23.508 INFO:tasks.workunit.client.0.smithi008.stderr:162650K .......... .......... .......... .......... .......... 96% 6.77M 1s 2020-02-05T17:01:23.514 INFO:tasks.workunit.client.0.smithi008.stderr:162700K .......... .......... .......... .......... .......... 96% 8.15M 1s 2020-02-05T17:01:23.520 INFO:tasks.workunit.client.0.smithi008.stderr:162750K .......... .......... .......... .......... .......... 96% 8.83M 1s 2020-02-05T17:01:23.526 INFO:tasks.workunit.client.0.smithi008.stderr:162800K .......... .......... .......... .......... .......... 96% 7.19M 1s 2020-02-05T17:01:23.534 INFO:tasks.workunit.client.0.smithi008.stderr:162850K .......... .......... .......... .......... .......... 96% 6.05M 1s 2020-02-05T17:01:23.539 INFO:tasks.workunit.client.0.smithi008.stderr:162900K .......... .......... .......... .......... .......... 96% 10.4M 1s 2020-02-05T17:01:23.544 INFO:tasks.workunit.client.0.smithi008.stderr:162950K .......... .......... .......... .......... .......... 96% 10.7M 1s 2020-02-05T17:01:23.548 INFO:tasks.workunit.client.0.smithi008.stderr:163000K .......... .......... .......... .......... .......... 96% 11.1M 1s 2020-02-05T17:01:23.553 INFO:tasks.workunit.client.0.smithi008.stderr:163050K .......... .......... .......... .......... .......... 96% 9.42M 1s 2020-02-05T17:01:23.558 INFO:tasks.workunit.client.0.smithi008.stderr:163100K .......... .......... .......... .......... .......... 96% 10.7M 1s 2020-02-05T17:01:23.562 INFO:tasks.workunit.client.0.smithi008.stderr:163150K .......... .......... .......... .......... .......... 96% 10.2M 1s 2020-02-05T17:01:23.567 INFO:tasks.workunit.client.0.smithi008.stderr:163200K .......... .......... .......... .......... .......... 96% 10.3M 1s 2020-02-05T17:01:23.573 INFO:tasks.workunit.client.0.smithi008.stderr:163250K .......... .......... .......... .......... .......... 96% 8.29M 1s 2020-02-05T17:01:23.587 INFO:tasks.workunit.client.0.smithi008.stderr:163300K .......... .......... .......... .......... .......... 96% 3.66M 1s 2020-02-05T17:01:23.592 INFO:tasks.workunit.client.0.smithi008.stderr:163350K .......... .......... .......... .......... .......... 96% 9.11M 1s 2020-02-05T17:01:23.597 INFO:tasks.workunit.client.0.smithi008.stderr:163400K .......... .......... .......... .......... .......... 96% 9.04M 1s 2020-02-05T17:01:23.604 INFO:tasks.workunit.client.0.smithi008.stderr:163450K .......... .......... .......... .......... .......... 96% 7.06M 1s 2020-02-05T17:01:23.610 INFO:tasks.workunit.client.0.smithi008.stderr:163500K .......... .......... .......... .......... .......... 97% 8.41M 1s 2020-02-05T17:01:23.616 INFO:tasks.workunit.client.0.smithi008.stderr:163550K .......... .......... .......... .......... .......... 97% 8.43M 1s 2020-02-05T17:01:23.623 INFO:tasks.workunit.client.0.smithi008.stderr:163600K .......... .......... .......... .......... .......... 97% 7.42M 1s 2020-02-05T17:01:23.630 INFO:tasks.workunit.client.0.smithi008.stderr:163650K .......... .......... .......... .......... .......... 97% 6.29M 1s 2020-02-05T17:01:23.637 INFO:tasks.workunit.client.0.smithi008.stderr:163700K .......... .......... .......... .......... .......... 97% 7.24M 1s 2020-02-05T17:01:23.645 INFO:tasks.workunit.client.0.smithi008.stderr:163750K .......... .......... .......... .......... .......... 97% 6.59M 1s 2020-02-05T17:01:23.653 INFO:tasks.workunit.client.0.smithi008.stderr:163800K .......... .......... .......... .......... .......... 97% 6.34M 1s 2020-02-05T17:01:23.660 INFO:tasks.workunit.client.0.smithi008.stderr:163850K .......... .......... .......... .......... .......... 97% 6.27M 1s 2020-02-05T17:01:23.666 INFO:tasks.workunit.client.0.smithi008.stderr:163900K .......... .......... .......... .......... .......... 97% 8.53M 1s 2020-02-05T17:01:23.670 INFO:tasks.workunit.client.0.smithi008.stderr:163950K .......... .......... .......... .......... .......... 97% 10.1M 1s 2020-02-05T17:01:23.674 INFO:tasks.workunit.client.0.smithi008.stderr:164000K .......... .......... .......... .......... .......... 97% 13.8M 1s 2020-02-05T17:01:23.678 INFO:tasks.workunit.client.0.smithi008.stderr:164050K .......... .......... .......... .......... .......... 97% 11.1M 1s 2020-02-05T17:01:23.683 INFO:tasks.workunit.client.0.smithi008.stderr:164100K .......... .......... .......... .......... .......... 97% 11.8M 1s 2020-02-05T17:01:23.688 INFO:tasks.workunit.client.0.smithi008.stderr:164150K .......... .......... .......... .......... .......... 97% 9.42M 1s 2020-02-05T17:01:23.693 INFO:tasks.workunit.client.0.smithi008.stderr:164200K .......... .......... .......... .......... .......... 97% 9.03M 1s 2020-02-05T17:01:23.699 INFO:tasks.workunit.client.0.smithi008.stderr:164250K .......... .......... .......... .......... .......... 97% 7.84M 1s 2020-02-05T17:01:23.705 INFO:tasks.workunit.client.0.smithi008.stderr:164300K .......... .......... .......... .......... .......... 97% 8.75M 1s 2020-02-05T17:01:23.711 INFO:tasks.workunit.client.0.smithi008.stderr:164350K .......... .......... .......... .......... .......... 97% 8.24M 0s 2020-02-05T17:01:23.717 INFO:tasks.workunit.client.0.smithi008.stderr:164400K .......... .......... .......... .......... .......... 97% 8.20M 0s 2020-02-05T17:01:23.724 INFO:tasks.workunit.client.0.smithi008.stderr:164450K .......... .......... .......... .......... .......... 97% 6.50M 0s 2020-02-05T17:01:23.731 INFO:tasks.workunit.client.0.smithi008.stderr:164500K .......... .......... .......... .......... .......... 97% 7.71M 0s 2020-02-05T17:01:23.737 INFO:tasks.workunit.client.0.smithi008.stderr:164550K .......... .......... .......... .......... .......... 97% 7.39M 0s 2020-02-05T17:01:23.743 INFO:tasks.workunit.client.0.smithi008.stderr:164600K .......... .......... .......... .......... .......... 97% 8.56M 0s 2020-02-05T17:01:23.751 INFO:tasks.workunit.client.0.smithi008.stderr:164650K .......... .......... .......... .......... .......... 97% 6.37M 0s 2020-02-05T17:01:23.757 INFO:tasks.workunit.client.0.smithi008.stderr:164700K .......... .......... .......... .......... .......... 97% 7.61M 0s 2020-02-05T17:01:23.763 INFO:tasks.workunit.client.0.smithi008.stderr:164750K .......... .......... .......... .......... .......... 97% 8.28M 0s 2020-02-05T17:01:23.768 INFO:tasks.workunit.client.0.smithi008.stderr:164800K .......... .......... .......... .......... .......... 97% 9.05M 0s 2020-02-05T17:01:23.774 INFO:tasks.workunit.client.0.smithi008.stderr:164850K .......... .......... .......... .......... .......... 97% 8.13M 0s 2020-02-05T17:01:23.780 INFO:tasks.workunit.client.0.smithi008.stderr:164900K .......... .......... .......... .......... .......... 97% 9.12M 0s 2020-02-05T17:01:23.785 INFO:tasks.workunit.client.0.smithi008.stderr:164950K .......... .......... .......... .......... .......... 97% 8.72M 0s 2020-02-05T17:01:23.790 INFO:tasks.workunit.client.0.smithi008.stderr:165000K .......... .......... .......... .......... .......... 97% 9.58M 0s 2020-02-05T17:01:23.796 INFO:tasks.workunit.client.0.smithi008.stderr:165050K .......... .......... .......... .......... .......... 97% 8.37M 0s 2020-02-05T17:01:23.801 INFO:tasks.workunit.client.0.smithi008.stderr:165100K .......... .......... .......... .......... .......... 97% 10.6M 0s 2020-02-05T17:01:23.806 INFO:tasks.workunit.client.0.smithi008.stderr:165150K .......... .......... .......... .......... .......... 98% 9.93M 0s 2020-02-05T17:01:23.811 INFO:tasks.workunit.client.0.smithi008.stderr:165200K .......... .......... .......... .......... .......... 98% 9.69M 0s 2020-02-05T17:01:23.816 INFO:tasks.workunit.client.0.smithi008.stderr:165250K .......... .......... .......... .......... .......... 98% 8.97M 0s 2020-02-05T17:01:23.821 INFO:tasks.workunit.client.0.smithi008.stderr:165300K .......... .......... .......... .......... .......... 98% 10.0M 0s 2020-02-05T17:01:23.826 INFO:tasks.workunit.client.0.smithi008.stderr:165350K .......... .......... .......... .......... .......... 98% 9.09M 0s 2020-02-05T17:01:23.831 INFO:tasks.workunit.client.0.smithi008.stderr:165400K .......... .......... .......... .......... .......... 98% 9.64M 0s 2020-02-05T17:01:23.838 INFO:tasks.workunit.client.0.smithi008.stderr:165450K .......... .......... .......... .......... .......... 98% 7.63M 0s 2020-02-05T17:01:23.844 INFO:tasks.workunit.client.0.smithi008.stderr:165500K .......... .......... .......... .......... .......... 98% 8.59M 0s 2020-02-05T17:01:23.851 INFO:tasks.workunit.client.0.smithi008.stderr:165550K .......... .......... .......... .......... .......... 98% 6.22M 0s 2020-02-05T17:01:23.856 INFO:tasks.workunit.client.0.smithi008.stderr:165600K .......... .......... .......... .......... .......... 98% 9.66M 0s 2020-02-05T17:01:23.863 INFO:tasks.workunit.client.0.smithi008.stderr:165650K .......... .......... .......... .......... .......... 98% 7.94M 0s 2020-02-05T17:01:23.868 INFO:tasks.workunit.client.0.smithi008.stderr:165700K .......... .......... .......... .......... .......... 98% 9.35M 0s 2020-02-05T17:01:23.874 INFO:tasks.workunit.client.0.smithi008.stderr:165750K .......... .......... .......... .......... .......... 98% 8.58M 0s 2020-02-05T17:01:23.879 INFO:tasks.workunit.client.0.smithi008.stderr:165800K .......... .......... .......... .......... .......... 98% 8.23M 0s 2020-02-05T17:01:23.886 INFO:tasks.workunit.client.0.smithi008.stderr:165850K .......... .......... .......... .......... .......... 98% 7.24M 0s 2020-02-05T17:01:23.892 INFO:tasks.workunit.client.0.smithi008.stderr:165900K .......... .......... .......... .......... .......... 98% 8.92M 0s 2020-02-05T17:01:23.897 INFO:tasks.workunit.client.0.smithi008.stderr:165950K .......... .......... .......... .......... .......... 98% 8.64M 0s 2020-02-05T17:01:23.904 INFO:tasks.workunit.client.0.smithi008.stderr:166000K .......... .......... .......... .......... .......... 98% 7.64M 0s 2020-02-05T17:01:23.911 INFO:tasks.workunit.client.0.smithi008.stderr:166050K .......... .......... .......... .......... .......... 98% 6.63M 0s 2020-02-05T17:01:23.918 INFO:tasks.workunit.client.0.smithi008.stderr:166100K .......... .......... .......... .......... .......... 98% 7.22M 0s 2020-02-05T17:01:23.925 INFO:tasks.workunit.client.0.smithi008.stderr:166150K .......... .......... .......... .......... .......... 98% 7.25M 0s 2020-02-05T17:01:23.932 INFO:tasks.workunit.client.0.smithi008.stderr:166200K .......... .......... .......... .......... .......... 98% 6.65M 0s 2020-02-05T17:01:23.941 INFO:tasks.workunit.client.0.smithi008.stderr:166250K .......... .......... .......... .......... .......... 98% 5.77M 0s 2020-02-05T17:01:23.948 INFO:tasks.workunit.client.0.smithi008.stderr:166300K .......... .......... .......... .......... .......... 98% 6.42M 0s 2020-02-05T17:01:23.956 INFO:tasks.workunit.client.0.smithi008.stderr:166350K .......... .......... .......... .......... .......... 98% 6.78M 0s 2020-02-05T17:01:23.963 INFO:tasks.workunit.client.0.smithi008.stderr:166400K .......... .......... .......... .......... .......... 98% 6.77M 0s 2020-02-05T17:01:23.972 INFO:tasks.workunit.client.0.smithi008.stderr:166450K .......... .......... .......... .......... .......... 98% 5.30M 0s 2020-02-05T17:01:23.978 INFO:tasks.workunit.client.0.smithi008.stderr:166500K .......... .......... .......... .......... .......... 98% 7.70M 0s 2020-02-05T17:01:23.983 INFO:tasks.workunit.client.0.smithi008.stderr:166550K .......... .......... .......... .......... .......... 98% 8.78M 0s 2020-02-05T17:01:23.988 INFO:tasks.workunit.client.0.smithi008.stderr:166600K .......... .......... .......... .......... .......... 98% 10.0M 0s 2020-02-05T17:01:23.996 INFO:tasks.workunit.client.0.smithi008.stderr:166650K .......... .......... .......... .......... .......... 98% 8.46M 0s 2020-02-05T17:01:24.007 INFO:tasks.workunit.client.0.smithi008.stderr:166700K .......... .......... .......... .......... .......... 98% 9.59M 0s 2020-02-05T17:01:24.007 INFO:tasks.workunit.client.0.smithi008.stderr:166750K .......... .......... .......... .......... .......... 98% 9.57M 0s 2020-02-05T17:01:24.009 INFO:tasks.workunit.client.0.smithi008.stderr:166800K .......... .......... .......... .......... .......... 98% 10.0M 0s 2020-02-05T17:01:24.015 INFO:tasks.workunit.client.0.smithi008.stderr:166850K .......... .......... .......... .......... .......... 99% 8.71M 0s 2020-02-05T17:01:24.020 INFO:tasks.workunit.client.0.smithi008.stderr:166900K .......... .......... .......... .......... .......... 99% 9.48M 0s 2020-02-05T17:01:24.025 INFO:tasks.workunit.client.0.smithi008.stderr:166950K .......... .......... .......... .......... .......... 99% 9.18M 0s 2020-02-05T17:01:24.031 INFO:tasks.workunit.client.0.smithi008.stderr:167000K .......... .......... .......... .......... .......... 99% 9.15M 0s 2020-02-05T17:01:24.037 INFO:tasks.workunit.client.0.smithi008.stderr:167050K .......... .......... .......... .......... .......... 99% 7.90M 0s 2020-02-05T17:01:24.042 INFO:tasks.workunit.client.0.smithi008.stderr:167100K .......... .......... .......... .......... .......... 99% 9.50M 0s 2020-02-05T17:01:24.047 INFO:tasks.workunit.client.0.smithi008.stderr:167150K .......... .......... .......... .......... .......... 99% 9.98M 0s 2020-02-05T17:01:24.052 INFO:tasks.workunit.client.0.smithi008.stderr:167200K .......... .......... .......... .......... .......... 99% 9.93M 0s 2020-02-05T17:01:24.057 INFO:tasks.workunit.client.0.smithi008.stderr:167250K .......... .......... .......... .......... .......... 99% 8.86M 0s 2020-02-05T17:01:24.062 INFO:tasks.workunit.client.0.smithi008.stderr:167300K .......... .......... .......... .......... .......... 99% 9.98M 0s 2020-02-05T17:01:24.067 INFO:tasks.workunit.client.0.smithi008.stderr:167350K .......... .......... .......... .......... .......... 99% 9.74M 0s 2020-02-05T17:01:24.075 INFO:tasks.workunit.client.0.smithi008.stderr:167400K .......... .......... .......... .......... .......... 99% 5.75M 0s 2020-02-05T17:01:24.079 INFO:tasks.workunit.client.0.smithi008.stderr:167450K .......... .......... .......... .......... .......... 99% 12.7M 0s 2020-02-05T17:01:24.082 INFO:tasks.workunit.client.0.smithi008.stderr:167500K .......... .......... .......... .......... .......... 99% 17.2M 0s 2020-02-05T17:01:24.085 INFO:tasks.workunit.client.0.smithi008.stderr:167550K .......... .......... .......... .......... .......... 99% 17.5M 0s 2020-02-05T17:01:24.088 INFO:tasks.workunit.client.0.smithi008.stderr:167600K .......... .......... .......... .......... .......... 99% 16.6M 0s 2020-02-05T17:01:24.091 INFO:tasks.workunit.client.0.smithi008.stderr:167650K .......... .......... .......... .......... .......... 99% 15.1M 0s 2020-02-05T17:01:24.095 INFO:tasks.workunit.client.0.smithi008.stderr:167700K .......... .......... .......... .......... .......... 99% 11.8M 0s 2020-02-05T17:01:24.100 INFO:tasks.workunit.client.0.smithi008.stderr:167750K .......... .......... .......... .......... .......... 99% 11.3M 0s 2020-02-05T17:01:24.106 INFO:tasks.workunit.client.0.smithi008.stderr:167800K .......... .......... .......... .......... .......... 99% 9.02M 0s 2020-02-05T17:01:24.111 INFO:tasks.workunit.client.0.smithi008.stderr:167850K .......... .......... .......... .......... .......... 99% 8.75M 0s 2020-02-05T17:01:24.115 INFO:tasks.workunit.client.0.smithi008.stderr:167900K .......... .......... .......... .......... .......... 99% 10.7M 0s 2020-02-05T17:01:24.120 INFO:tasks.workunit.client.0.smithi008.stderr:167950K .......... .......... .......... .......... .......... 99% 11.1M 0s 2020-02-05T17:01:24.124 INFO:tasks.workunit.client.0.smithi008.stderr:168000K .......... .......... .......... .......... .......... 99% 11.9M 0s 2020-02-05T17:01:24.129 INFO:tasks.workunit.client.0.smithi008.stderr:168050K .......... .......... .......... .......... .......... 99% 9.48M 0s 2020-02-05T17:01:24.132 INFO:tasks.workunit.client.0.smithi008.stderr:168100K .......... .......... .......... .......... .......... 99% 15.4M 0s 2020-02-05T17:01:24.135 INFO:tasks.workunit.client.0.smithi008.stderr:168150K .......... .......... .......... .......... .......... 99% 15.6M 0s 2020-02-05T17:01:24.138 INFO:tasks.workunit.client.0.smithi008.stderr:168200K .......... .......... .......... .......... .......... 99% 14.9M 0s 2020-02-05T17:01:24.142 INFO:tasks.workunit.client.0.smithi008.stderr:168250K .......... .......... .......... .......... .......... 99% 13.6M 0s 2020-02-05T17:01:24.145 INFO:tasks.workunit.client.0.smithi008.stderr:168300K .......... .......... .......... .......... .......... 99% 17.0M 0s 2020-02-05T17:01:24.147 INFO:tasks.workunit.client.0.smithi008.stderr:168350K .......... .......... .......... .......... .......... 99% 18.3M 0s 2020-02-05T17:01:24.150 INFO:tasks.workunit.client.0.smithi008.stderr:168400K .......... .......... .......... .......... .......... 99% 16.9M 0s 2020-02-05T17:01:24.154 INFO:tasks.workunit.client.0.smithi008.stderr:168450K .......... .......... .......... .......... .......... 99% 14.6M 0s 2020-02-05T17:01:24.157 INFO:tasks.workunit.client.0.smithi008.stderr:168500K .......... .......... .......... .......... .......... 99% 16.0M 0s 2020-02-05T17:01:24.158 INFO:tasks.workunit.client.0.smithi008.stderr:168550K .......... .......... . 100% 21.0M=20s 2020-02-05T17:01:24.158 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:24.158 INFO:tasks.workunit.client.0.smithi008.stderr:2020-02-05 17:01:24 (8.22 MB/s) - ‘linux.tar.gz’ saved [172616875/172616875] 2020-02-05T17:01:24.159 INFO:tasks.workunit.client.0.smithi008.stderr: 2020-02-05T17:01:36.890 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:01:36.896 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:02:07.024 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:02:07.030 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:02:37.146 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:02:37.151 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:03:07.413 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:03:07.418 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:03:37.576 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:03:37.580 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:04:07.716 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:04:07.721 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:04:37.854 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:04:37.859 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:05:07.999 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:05:08.012 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:05:38.134 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:05:38.141 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:06:08.283 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:06:08.288 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:06:38.420 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:06:38.426 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:07:08.562 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:07:08.568 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:07:38.690 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:07:38.695 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:08:08.822 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:08:08.828 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:08:40.281 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:08:40.287 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:09:10.445 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:09:10.450 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:09:40.587 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:09:40.593 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:10:10.714 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:10:10.723 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:10:40.846 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:10:40.857 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:11:11.003 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:11:11.009 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:11:41.141 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:11:41.152 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:12:11.272 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:12:11.277 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:12:41.408 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:12:41.413 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:13:09.638 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/basic/fixdep 2020-02-05T17:13:10.099 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/conf.o 2020-02-05T17:13:10.390 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/confdata.o 2020-02-05T17:13:10.891 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/expr.o 2020-02-05T17:13:11.247 INFO:tasks.workunit.client.0.smithi008.stdout: LEX scripts/kconfig/lexer.lex.c 2020-02-05T17:13:11.314 INFO:tasks.workunit.client.0.smithi008.stdout: YACC scripts/kconfig/parser.tab.[ch] 2020-02-05T17:13:11.420 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/lexer.lex.o 2020-02-05T17:13:11.539 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:13:11.544 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:13:11.977 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/parser.tab.o 2020-02-05T17:13:12.629 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/preprocess.o 2020-02-05T17:13:12.881 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kconfig/symbol.o 2020-02-05T17:13:13.331 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTLD scripts/kconfig/conf 2020-02-05T17:13:13.674 INFO:tasks.workunit.client.0.smithi008.stdout:*** Default configuration is based on 'x86_64_defconfig' 2020-02-05T17:13:41.712 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:13:41.717 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:13:47.305 INFO:tasks.workunit.client.0.smithi008.stdout:# 2020-02-05T17:13:47.306 INFO:tasks.workunit.client.0.smithi008.stdout:# configuration written to .config 2020-02-05T17:13:47.306 INFO:tasks.workunit.client.0.smithi008.stdout:# 2020-02-05T17:13:54.892 INFO:tasks.workunit.client.0.smithi008.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_32.h 2020-02-05T17:13:54.901 INFO:tasks.workunit.client.0.smithi008.stdout: SYSHDR arch/x86/include/generated/asm/unistd_32_ia32.h 2020-02-05T17:13:54.912 INFO:tasks.workunit.client.0.smithi008.stdout: SYSHDR arch/x86/include/generated/asm/unistd_64_x32.h 2020-02-05T17:13:54.915 INFO:tasks.workunit.client.0.smithi008.stdout: SYSTBL arch/x86/include/generated/asm/syscalls_64.h 2020-02-05T17:13:54.950 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/uapi/linux/version.h 2020-02-05T17:13:55.006 INFO:tasks.workunit.client.0.smithi008.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_32.h 2020-02-05T17:13:55.030 INFO:tasks.workunit.client.0.smithi008.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_64.h 2020-02-05T17:13:55.093 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/bpf_perf_event.h 2020-02-05T17:13:55.106 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC arch/x86/tools/relocs_32.o 2020-02-05T17:13:55.118 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/errno.h 2020-02-05T17:13:55.142 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/fcntl.h 2020-02-05T17:13:55.163 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctl.h 2020-02-05T17:13:55.189 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/ioctls.h 2020-02-05T17:13:55.215 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/ipcbuf.h 2020-02-05T17:13:55.231 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/config/kernel.release 2020-02-05T17:13:55.243 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/param.h 2020-02-05T17:13:55.265 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/poll.h 2020-02-05T17:13:55.269 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC arch/x86/tools/relocs_64.o 2020-02-05T17:13:55.295 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/resource.h 2020-02-05T17:13:55.323 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/socket.h 2020-02-05T17:13:55.354 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/sockios.h 2020-02-05T17:13:55.380 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/termbits.h 2020-02-05T17:13:55.385 INFO:tasks.workunit.client.0.smithi008.stdout: SYSHDR arch/x86/include/generated/uapi/asm/unistd_x32.h 2020-02-05T17:13:55.417 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/termios.h 2020-02-05T17:13:55.419 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/uapi/asm/types.h 2020-02-05T17:13:55.475 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC arch/x86/tools/relocs_common.o 2020-02-05T17:13:55.735 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/dma-contiguous.h 2020-02-05T17:13:55.740 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/early_ioremap.h 2020-02-05T17:13:55.753 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/export.h 2020-02-05T17:13:55.778 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/mcs_spinlock.h 2020-02-05T17:13:55.780 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/mm-arch-hooks.h 2020-02-05T17:13:55.784 INFO:tasks.workunit.client.0.smithi008.stdout: WRAP arch/x86/include/generated/asm/mmiowb.h 2020-02-05T17:13:55.815 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/utsrelease.h 2020-02-05T17:13:55.869 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTLD arch/x86/tools/relocs 2020-02-05T17:13:56.006 INFO:tasks.workunit.client.0.smithi008.stdout: DESCEND objtool 2020-02-05T17:13:56.032 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/kallsyms 2020-02-05T17:13:56.035 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/pnmtologo 2020-02-05T17:13:56.142 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/selinux/genheaders/genheaders 2020-02-05T17:13:56.160 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/conmakehash 2020-02-05T17:13:56.184 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/selinux/mdp/mdp 2020-02-05T17:13:56.604 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/sortextable 2020-02-05T17:13:56.614 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/asn1_compiler 2020-02-05T17:13:56.751 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/extract-cert 2020-02-05T17:13:57.123 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/fixdep.o 2020-02-05T17:13:57.347 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTLD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/fixdep-in.o 2020-02-05T17:13:57.437 INFO:tasks.workunit.client.0.smithi008.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/fixdep 2020-02-05T17:13:57.890 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/builtin-check.o 2020-02-05T17:13:58.008 INFO:tasks.workunit.client.0.smithi008.stdout: MKDIR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/arch/x86/lib/ 2020-02-05T17:13:58.038 INFO:tasks.workunit.client.0.smithi008.stdout: GEN /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/arch/x86/lib/inat-tables.c 2020-02-05T17:13:58.153 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/arch/x86/decode.o 2020-02-05T17:13:58.332 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/exec-cmd.o 2020-02-05T17:13:58.739 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/builtin-orc.o 2020-02-05T17:13:59.154 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/help.o 2020-02-05T17:13:59.154 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/check.o 2020-02-05T17:13:59.452 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/orc_gen.o 2020-02-05T17:13:59.868 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/orc_dump.o 2020-02-05T17:14:00.649 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/pager.o 2020-02-05T17:14:00.888 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/elf.o 2020-02-05T17:14:01.190 INFO:tasks.workunit.client.0.smithi008.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/arch/x86/objtool-in.o 2020-02-05T17:14:01.278 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/parse-options.o 2020-02-05T17:14:01.299 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/special.o 2020-02-05T17:14:01.580 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/objtool.o 2020-02-05T17:14:02.085 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/libstring.o 2020-02-05T17:14:02.306 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/run-command.o 2020-02-05T17:14:02.492 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/libctype.o 2020-02-05T17:14:03.186 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/sigchain.o 2020-02-05T17:14:10.077 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/subcmd-config.o 2020-02-05T17:14:10.204 INFO:tasks.workunit.client.0.smithi008.stdout: CC /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/str_error_r.o 2020-02-05T17:14:10.287 INFO:tasks.workunit.client.0.smithi008.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/libsubcmd-in.o 2020-02-05T17:14:10.608 INFO:tasks.workunit.client.0.smithi008.stdout: AR /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/libsubcmd.a 2020-02-05T17:14:11.850 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:14:11.856 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:14:12.245 INFO:tasks.workunit.client.0.smithi008.stdout: LD /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/objtool-in.o 2020-02-05T17:14:13.171 INFO:tasks.workunit.client.0.smithi008.stdout:/home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool 2020-02-05T17:14:13.179 INFO:tasks.workunit.client.0.smithi008.stdout: LINK /home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-5.4/tools/objtool/objtool 2020-02-05T17:14:13.226 INFO:tasks.workunit.client.0.smithi008.stdout: CC scripts/mod/empty.o 2020-02-05T17:14:13.482 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/mod/mk_elfconfig 2020-02-05T17:14:13.515 INFO:tasks.workunit.client.0.smithi008.stdout: CC scripts/mod/devicetable-offsets.s 2020-02-05T17:14:13.864 INFO:tasks.workunit.client.0.smithi008.stdout: MKELF scripts/mod/elfconfig.h 2020-02-05T17:14:13.934 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/mod/modpost.o 2020-02-05T17:14:14.274 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/mod/sumversion.o 2020-02-05T17:14:14.742 INFO:tasks.workunit.client.0.smithi008.stdout: UPD scripts/mod/devicetable-offsets.h 2020-02-05T17:14:14.795 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC scripts/mod/file2alias.o 2020-02-05T17:14:15.534 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTLD scripts/mod/modpost 2020-02-05T17:14:16.660 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/bounds.s 2020-02-05T17:14:16.771 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/timeconst.h 2020-02-05T17:14:17.174 INFO:tasks.workunit.client.0.smithi008.stdout: CALL scripts/atomic/check-atomics.sh 2020-02-05T17:14:18.022 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/bounds.h 2020-02-05T17:14:18.068 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/asm-offsets.s 2020-02-05T17:14:25.646 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/asm-offsets.h 2020-02-05T17:14:25.678 INFO:tasks.workunit.client.0.smithi008.stdout: CALL scripts/checksyscalls.sh 2020-02-05T17:14:26.114 INFO:tasks.workunit.client.0.smithi008.stdout: CHK include/generated/compile.h 2020-02-05T17:14:26.131 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC usr/gen_init_cpio 2020-02-05T17:14:26.169 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/main.o 2020-02-05T17:14:26.171 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/do_mounts.o 2020-02-05T17:14:26.182 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/do_mounts_initrd.o 2020-02-05T17:14:26.196 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/do_mounts_md.o 2020-02-05T17:14:26.227 INFO:tasks.workunit.client.0.smithi008.stdout: UPD include/generated/compile.h 2020-02-05T17:14:26.580 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/crypto/built-in.a 2020-02-05T17:14:26.593 INFO:tasks.workunit.client.0.smithi008.stdout: GEN usr/initramfs_data.cpio 2020-02-05T17:14:26.729 INFO:tasks.workunit.client.0.smithi008.stdout: AS usr/initramfs_data.o 2020-02-05T17:14:26.861 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vsyscall/vsyscall_64.o 2020-02-05T17:14:26.977 INFO:tasks.workunit.client.0.smithi008.stdout: AR usr/built-in.a 2020-02-05T17:14:26.989 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/bpf/core.o 2020-02-05T17:14:27.131 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vma.o 2020-02-05T17:14:34.903 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vdso32-setup.o 2020-02-05T17:14:35.364 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/vsyscall/vsyscall_emu_64.o 2020-02-05T17:14:35.812 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/entry/vsyscall/built-in.a 2020-02-05T17:14:36.126 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/initramfs.o 2020-02-05T17:14:36.254 INFO:tasks.workunit.client.0.smithi008.stdout: LDS arch/x86/entry/vdso/vdso.lds 2020-02-05T17:14:36.431 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/cgroup.o 2020-02-05T17:14:36.834 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/dma/mapping.o 2020-02-05T17:14:38.345 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/vdso/vdso-note.o 2020-02-05T17:14:38.361 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/bpf/built-in.a 2020-02-05T17:14:38.503 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/events/core.o 2020-02-05T17:14:38.539 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/rstat.o 2020-02-05T17:14:38.714 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/irqdesc.o 2020-02-05T17:14:38.739 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vclock_gettime.o 2020-02-05T17:14:43.192 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:14:43.197 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:14:44.013 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/dma/direct.o 2020-02-05T17:14:44.071 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vgetcpu.o 2020-02-05T17:14:44.332 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/calibrate.o 2020-02-05T17:14:44.534 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/handle.o 2020-02-05T17:14:45.268 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/manage.o 2020-02-05T17:14:47.861 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/namespace.o 2020-02-05T17:14:49.786 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/init_task.o 2020-02-05T17:14:50.175 INFO:tasks.workunit.client.0.smithi008.stdout: HOSTCC arch/x86/entry/vdso/vdso2c 2020-02-05T17:14:50.294 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/events/ring_buffer.o 2020-02-05T17:14:50.624 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/spurious.o 2020-02-05T17:14:51.036 INFO:tasks.workunit.client.0.smithi008.stdout: LDS arch/x86/entry/vdso/vdso32/vdso32.lds 2020-02-05T17:14:51.124 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/dma/dummy.o 2020-02-05T17:14:51.212 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vdso32/vclock_gettime.o 2020-02-05T17:14:51.382 INFO:tasks.workunit.client.0.smithi008.stdout: CC certs/system_keyring.o 2020-02-05T17:14:51.691 INFO:tasks.workunit.client.0.smithi008.stdout: CC init/version.o 2020-02-05T17:14:54.846 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/cgroup-v1.o 2020-02-05T17:14:57.152 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/resend.o 2020-02-05T17:14:57.262 INFO:tasks.workunit.client.0.smithi008.stdout: EXTRACT_CERTS 2020-02-05T17:14:57.500 INFO:tasks.workunit.client.0.smithi008.stdout: AS certs/system_certificates.o 2020-02-05T17:14:57.534 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/vdso/vdso32/note.o 2020-02-05T17:14:57.914 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/vdso/vdso32/system_call.o 2020-02-05T17:14:57.977 INFO:tasks.workunit.client.0.smithi008.stdout: AR certs/built-in.a 2020-02-05T17:14:58.060 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/filemap.o 2020-02-05T17:14:58.196 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/chip.o 2020-02-05T17:14:58.332 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/vdso/vdso32/sigreturn.o 2020-02-05T17:14:58.697 INFO:tasks.workunit.client.0.smithi008.stdout: AR init/built-in.a 2020-02-05T17:14:58.835 INFO:tasks.workunit.client.0.smithi008.stdout: VDSO arch/x86/entry/vdso/vdso64.so.dbg 2020-02-05T17:14:59.047 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/events/callchain.o 2020-02-05T17:14:59.187 INFO:tasks.workunit.client.0.smithi008.stdout: VDSO arch/x86/entry/vdso/vdso32.so.dbg 2020-02-05T17:14:59.221 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/dma/swiotlb.o 2020-02-05T17:15:00.087 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/init.o 2020-02-05T17:15:05.131 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/inode.o 2020-02-05T17:15:05.982 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/dma/built-in.a 2020-02-05T17:15:09.992 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/root.o 2020-02-05T17:15:13.354 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:15:13.359 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:15:15.112 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/symlink.o 2020-02-05T17:15:19.808 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/waitq.o 2020-02-05T17:15:24.610 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/expire.o 2020-02-05T17:15:29.432 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/autofs/dev-ioctl.o 2020-02-05T17:15:36.072 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/autofs/built-in.a 2020-02-05T17:15:36.456 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/debugfs/inode.o 2020-02-05T17:15:42.242 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/debugfs/file.o 2020-02-05T17:15:43.524 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:15:43.530 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:15:47.830 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/debugfs/built-in.a 2020-02-05T17:15:48.092 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/devpts/inode.o 2020-02-05T17:15:53.775 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/devpts/built-in.a 2020-02-05T17:15:54.037 INFO:tasks.workunit.client.0.smithi008.stdout: CC [M] fs/efivarfs/inode.o 2020-02-05T17:15:59.129 INFO:tasks.workunit.client.0.smithi008.stdout: CC [M] fs/efivarfs/file.o 2020-02-05T17:16:04.061 INFO:tasks.workunit.client.0.smithi008.stdout: CC [M] fs/efivarfs/super.o 2020-02-05T17:16:09.539 INFO:tasks.workunit.client.0.smithi008.stdout: LD [M] fs/efivarfs/efivarfs.o 2020-02-05T17:16:09.742 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/exportfs/expfs.o 2020-02-05T17:16:13.667 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:16:13.672 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:16:14.645 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/exportfs/built-in.a 2020-02-05T17:16:15.115 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/balloc.o 2020-02-05T17:16:21.944 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/bitmap.o 2020-02-05T17:16:27.802 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/block_validity.o 2020-02-05T17:16:33.977 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/dir.o 2020-02-05T17:16:39.930 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/ext4_jbd2.o 2020-02-05T17:16:43.809 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:16:43.815 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:16:46.341 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/extents.o 2020-02-05T17:16:53.796 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/extents_status.o 2020-02-05T17:17:00.605 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/file.o 2020-02-05T17:17:06.545 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/fsmap.o 2020-02-05T17:17:13.297 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/fsync.o 2020-02-05T17:17:13.939 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:17:13.945 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:17:19.629 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/hash.o 2020-02-05T17:17:25.491 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/ialloc.o 2020-02-05T17:17:32.780 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/indirect.o 2020-02-05T17:17:39.486 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/inline.o 2020-02-05T17:17:44.095 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:17:44.102 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:17:45.778 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/inode.o 2020-02-05T17:17:53.694 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/ioctl.o 2020-02-05T17:18:01.332 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/mballoc.o 2020-02-05T17:18:09.146 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/migrate.o 2020-02-05T17:18:14.241 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:18:14.246 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:18:14.994 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/mmp.o 2020-02-05T17:18:20.973 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/move_extent.o 2020-02-05T17:18:27.060 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/namei.o 2020-02-05T17:18:34.243 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/page-io.o 2020-02-05T17:18:40.725 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/readpage.o 2020-02-05T17:18:44.405 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:18:44.410 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:18:46.936 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/resize.o 2020-02-05T17:18:53.270 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/super.o 2020-02-05T17:19:03.669 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/symlink.o 2020-02-05T17:19:09.612 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/sysfs.o 2020-02-05T17:19:14.543 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:19:14.549 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:19:15.554 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/xattr.o 2020-02-05T17:19:22.063 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/xattr_trusted.o 2020-02-05T17:19:27.949 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/xattr_user.o 2020-02-05T17:19:33.900 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/acl.o 2020-02-05T17:19:40.041 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ext4/xattr_security.o 2020-02-05T17:19:44.681 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:19:44.687 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:19:46.040 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/ext4/built-in.a 2020-02-05T17:19:47.331 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/cache.o 2020-02-05T17:19:52.522 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/dir.o 2020-02-05T17:19:58.157 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/fatent.o 2020-02-05T17:20:03.770 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/file.o 2020-02-05T17:20:10.238 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/inode.o 2020-02-05T17:20:14.811 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:20:14.816 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:20:16.436 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/misc.o 2020-02-05T17:20:21.447 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/nfs.o 2020-02-05T17:20:26.362 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/namei_vfat.o 2020-02-05T17:20:31.995 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fat/namei_msdos.o 2020-02-05T17:20:37.660 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/fat/built-in.a 2020-02-05T17:20:38.220 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/hugetlbfs/inode.o 2020-02-05T17:20:44.656 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/hugetlbfs/built-in.a 2020-02-05T17:20:44.926 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/apply.o 2020-02-05T17:20:44.941 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:20:44.947 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:20:50.254 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/buffered-io.o 2020-02-05T17:20:56.768 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/direct-io.o 2020-02-05T17:21:02.641 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/fiemap.o 2020-02-05T17:21:08.506 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/seek.o 2020-02-05T17:21:14.027 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/iomap/swapfile.o 2020-02-05T17:21:15.079 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:21:15.085 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:21:19.948 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/iomap/built-in.a 2020-02-05T17:21:20.431 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/namei.o 2020-02-05T17:21:25.499 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/inode.o 2020-02-05T17:21:31.494 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/dir.o 2020-02-05T17:21:36.415 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/util.o 2020-02-05T17:21:41.269 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/rock.o 2020-02-05T17:21:45.260 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:21:45.265 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:21:46.229 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/export.o 2020-02-05T17:21:51.171 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/joliet.o 2020-02-05T17:21:56.126 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/isofs/compress.o 2020-02-05T17:22:01.549 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/isofs/built-in.a 2020-02-05T17:22:02.055 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/transaction.o 2020-02-05T17:22:08.474 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/commit.o 2020-02-05T17:22:14.562 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/recovery.o 2020-02-05T17:22:15.390 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:22:15.396 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:22:20.002 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/checkpoint.o 2020-02-05T17:22:25.537 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/revoke.o 2020-02-05T17:22:30.715 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/jbd2/journal.o 2020-02-05T17:22:38.211 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/jbd2/built-in.a 2020-02-05T17:22:38.604 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/kernfs/mount.o 2020-02-05T17:22:43.792 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/kernfs/inode.o 2020-02-05T17:22:45.520 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:22:45.525 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:22:49.379 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/kernfs/dir.o 2020-02-05T17:22:54.566 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/kernfs/file.o 2020-02-05T17:23:00.074 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/kernfs/symlink.o 2020-02-05T17:23:04.996 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/kernfs/built-in.a 2020-02-05T17:23:05.531 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/clntlock.o 2020-02-05T17:23:14.220 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/clntproc.o 2020-02-05T17:23:15.666 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:23:15.672 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:23:22.853 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/clntxdr.o 2020-02-05T17:23:31.194 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/host.o 2020-02-05T17:23:39.706 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svc.o 2020-02-05T17:23:45.844 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:23:45.850 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:23:48.308 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svclock.o 2020-02-05T17:23:56.787 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svcshare.o 2020-02-05T17:24:05.267 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svcproc.o 2020-02-05T17:24:13.546 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svcsubs.o 2020-02-05T17:24:15.978 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:24:15.984 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:24:21.712 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/mon.o 2020-02-05T17:24:30.187 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/xdr.o 2020-02-05T17:24:38.564 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/clnt4xdr.o 2020-02-05T17:24:46.132 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:24:46.138 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:24:46.926 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/xdr4.o 2020-02-05T17:24:55.491 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/svc4proc.o 2020-02-05T17:25:03.805 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/lockd/procfs.o 2020-02-05T17:25:10.039 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/lockd/built-in.a 2020-02-05T17:25:11.066 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/client.o 2020-02-05T17:25:16.335 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:25:16.340 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:25:20.611 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/dir.o 2020-02-05T17:25:29.954 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/file.o 2020-02-05T17:25:38.874 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/getroot.o 2020-02-05T17:25:47.067 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:25:47.071 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:25:47.200 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/inode.o 2020-02-05T17:25:56.587 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/super.o 2020-02-05T17:26:05.927 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/io.o 2020-02-05T17:26:14.381 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/direct.o 2020-02-05T17:26:17.207 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:26:17.213 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:26:22.964 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/pagelist.o 2020-02-05T17:26:31.546 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/read.o 2020-02-05T17:26:40.373 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/symlink.o 2020-02-05T17:26:47.344 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:26:47.349 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:26:48.585 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/unlink.o 2020-02-05T17:26:57.330 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/write.o 2020-02-05T17:27:06.845 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/namespace.o 2020-02-05T17:27:15.389 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/mount_clnt.o 2020-02-05T17:27:17.490 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:27:17.495 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:27:23.985 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfstrace.o 2020-02-05T17:27:33.561 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/export.o 2020-02-05T17:27:42.111 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/sysfs.o 2020-02-05T17:27:47.629 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:27:47.634 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:27:50.488 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfsroot.o 2020-02-05T17:27:58.861 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/sysctl.o 2020-02-05T17:28:07.161 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs2super.o 2020-02-05T17:28:15.607 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/proc.o 2020-02-05T17:28:17.767 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:28:17.772 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:28:24.200 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs2xdr.o 2020-02-05T17:28:32.897 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs3super.o 2020-02-05T17:28:41.381 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs3client.o 2020-02-05T17:28:47.915 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:28:47.922 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:28:49.837 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs3proc.o 2020-02-05T17:28:59.157 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs3xdr.o 2020-02-05T17:29:07.971 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs3acl.o 2020-02-05T17:29:16.523 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4proc.o 2020-02-05T17:29:18.056 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:29:18.062 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:29:26.376 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4xdr.o 2020-02-05T17:29:36.283 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4state.o 2020-02-05T17:29:45.180 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4renewd.o 2020-02-05T17:29:48.190 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:29:48.195 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:29:53.563 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4super.o 2020-02-05T17:30:01.989 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4file.o 2020-02-05T17:30:10.723 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/delegation.o 2020-02-05T17:30:18.370 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:30:18.376 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:30:19.542 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4idmap.o 2020-02-05T17:30:28.326 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/callback.o 2020-02-05T17:30:36.922 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/callback_xdr.o 2020-02-05T17:30:45.487 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/callback_proc.o 2020-02-05T17:30:48.508 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:30:48.514 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:30:54.249 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4namespace.o 2020-02-05T17:31:02.779 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4getroot.o 2020-02-05T17:31:11.277 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4client.o 2020-02-05T17:31:18.639 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:31:18.644 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:31:19.951 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4session.o 2020-02-05T17:31:28.545 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/dns_resolve.o 2020-02-05T17:31:36.752 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4trace.o 2020-02-05T17:31:46.462 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs/nfs4sysctl.o 2020-02-05T17:31:48.786 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:31:48.792 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:31:54.989 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/nfs/built-in.a 2020-02-05T17:31:57.008 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs_common/nfsacl.o 2020-02-05T17:32:02.229 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nfs_common/grace.o 2020-02-05T17:32:08.571 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/nfs_common/built-in.a 2020-02-05T17:32:08.889 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nls/nls_base.o 2020-02-05T17:32:12.490 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nls/nls_cp437.o 2020-02-05T17:32:15.936 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nls/nls_ascii.o 2020-02-05T17:32:18.924 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:32:18.930 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:32:19.414 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nls/nls_iso8859-1.o 2020-02-05T17:32:22.856 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nls/nls_utf8.o 2020-02-05T17:32:26.363 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/nls/built-in.a 2020-02-05T17:32:26.900 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/dnotify/dnotify.o 2020-02-05T17:32:32.221 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/notify/dnotify/built-in.a 2020-02-05T17:32:32.424 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/notify/fanotify/built-in.a 2020-02-05T17:32:32.627 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/inotify/inotify_fsnotify.o 2020-02-05T17:32:37.596 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/inotify/inotify_user.o 2020-02-05T17:32:44.329 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/notify/inotify/built-in.a 2020-02-05T17:32:44.550 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/fsnotify.o 2020-02-05T17:32:49.062 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:32:49.069 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:32:50.043 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/notification.o 2020-02-05T17:32:55.601 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/group.o 2020-02-05T17:33:01.650 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/mark.o 2020-02-05T17:33:07.259 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/notify/fdinfo.o 2020-02-05T17:33:12.618 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/notify/built-in.a 2020-02-05T17:33:13.530 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/task_mmu.o 2020-02-05T17:33:19.201 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:33:19.207 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:33:20.002 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/inode.o 2020-02-05T17:33:25.437 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/root.o 2020-02-05T17:33:30.673 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/base.o 2020-02-05T17:33:37.747 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/generic.o 2020-02-05T17:33:43.106 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/array.o 2020-02-05T17:33:49.355 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:33:49.360 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:33:49.763 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/fd.o 2020-02-05T17:33:54.967 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/proc_tty.o 2020-02-05T17:34:00.214 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/cmdline.o 2020-02-05T17:34:04.776 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/consoles.o 2020-02-05T17:34:09.935 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/cpuinfo.o 2020-02-05T17:34:14.617 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/devices.o 2020-02-05T17:34:18.947 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/interrupts.o 2020-02-05T17:34:19.511 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:34:19.517 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:34:23.458 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/loadavg.o 2020-02-05T17:34:28.361 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/meminfo.o 2020-02-05T17:34:34.547 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/stat.o 2020-02-05T17:34:39.540 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/uptime.o 2020-02-05T17:34:44.087 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/util.o 2020-02-05T17:34:48.583 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/version.o 2020-02-05T17:34:49.649 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:34:49.656 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:34:53.042 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/softirqs.o 2020-02-05T17:34:57.684 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/namespaces.o 2020-02-05T17:35:03.685 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/self.o 2020-02-05T17:35:08.617 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/thread_self.o 2020-02-05T17:35:13.854 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/proc_sysctl.o 2020-02-05T17:35:19.407 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/proc_net.o 2020-02-05T17:35:19.796 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:35:19.802 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:35:25.637 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/kcore.o 2020-02-05T17:35:31.258 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/vmcore.o 2020-02-05T17:35:37.132 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/kmsg.o 2020-02-05T17:35:41.656 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc/page.o 2020-02-05T17:35:47.686 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/proc/built-in.a 2020-02-05T17:35:49.081 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/dquot.o 2020-02-05T17:35:49.960 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:35:49.981 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:35:55.093 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/quota_v2.o 2020-02-05T17:35:59.727 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/quota_tree.o 2020-02-05T17:36:04.568 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/quota.o 2020-02-05T17:36:11.228 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/kqid.o 2020-02-05T17:36:15.559 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/compat.o 2020-02-05T17:36:20.118 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:36:20.123 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:36:21.876 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/quota/netlink.o 2020-02-05T17:36:28.035 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/quota/built-in.a 2020-02-05T17:36:28.537 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ramfs/inode.o 2020-02-05T17:36:34.165 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ramfs/file-mmu.o 2020-02-05T17:36:38.972 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/ramfs/built-in.a 2020-02-05T17:36:39.274 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sysfs/file.o 2020-02-05T17:36:44.221 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sysfs/dir.o 2020-02-05T17:36:48.867 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sysfs/symlink.o 2020-02-05T17:36:50.262 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:36:50.267 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:36:53.830 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sysfs/mount.o 2020-02-05T17:36:59.849 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sysfs/group.o 2020-02-05T17:37:03.650 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/sysfs/built-in.a 2020-02-05T17:37:04.044 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/tracefs/inode.o 2020-02-05T17:37:09.596 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/tracefs/built-in.a 2020-02-05T17:37:09.742 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/open.o 2020-02-05T17:37:17.012 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/read_write.o 2020-02-05T17:37:20.408 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:37:20.414 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:37:24.114 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/file_table.o 2020-02-05T17:37:30.687 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/super.o 2020-02-05T17:37:36.837 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/char_dev.o 2020-02-05T17:37:42.896 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/stat.o 2020-02-05T17:37:49.470 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/exec.o 2020-02-05T17:37:50.547 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:37:50.551 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:37:57.229 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/pipe.o 2020-02-05T17:38:03.873 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/namei.o 2020-02-05T17:38:12.365 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fcntl.o 2020-02-05T17:38:18.889 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/ioctl.o 2020-02-05T17:38:20.684 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:38:20.690 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:38:25.510 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/readdir.o 2020-02-05T17:38:32.095 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/select.o 2020-02-05T17:38:41.052 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/dcache.o 2020-02-05T17:38:47.041 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/inode.o 2020-02-05T17:38:51.322 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:38:51.327 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:38:54.052 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/attr.o 2020-02-05T17:38:59.811 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/bad_inode.o 2020-02-05T17:39:04.259 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/file.o 2020-02-05T17:39:10.482 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/filesystems.o 2020-02-05T17:39:16.582 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/namespace.o 2020-02-05T17:39:21.453 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:39:21.459 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:39:24.333 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/seq_file.o 2020-02-05T17:39:29.586 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/xattr.o 2020-02-05T17:39:36.106 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/libfs.o 2020-02-05T17:39:42.192 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs-writeback.o 2020-02-05T17:39:49.475 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/pnode.o 2020-02-05T17:39:51.574 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:39:51.579 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:39:54.203 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/splice.o 2020-02-05T17:40:01.291 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/sync.o 2020-02-05T17:40:07.778 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/utimes.o 2020-02-05T17:40:14.122 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/d_path.o 2020-02-05T17:40:20.301 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/stack.o 2020-02-05T17:40:21.731 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:40:21.737 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:40:24.546 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs_struct.o 2020-02-05T17:40:29.233 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/statfs.o 2020-02-05T17:40:35.574 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs_pin.o 2020-02-05T17:40:40.290 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/nsfs.o 2020-02-05T17:40:45.393 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs_types.o 2020-02-05T17:40:49.833 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs_context.o 2020-02-05T17:40:51.877 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:40:51.882 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:40:56.245 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fs_parser.o 2020-02-05T17:41:01.447 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fsopen.o 2020-02-05T17:41:07.612 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/buffer.o 2020-02-05T17:41:15.264 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/block_dev.o 2020-02-05T17:41:22.023 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:41:22.028 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:41:22.078 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/direct-io.o 2020-02-05T17:41:27.899 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/mpage.o 2020-02-05T17:41:33.791 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/proc_namespace.o 2020-02-05T17:41:38.883 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/eventpoll.o 2020-02-05T17:41:47.857 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/anon_inodes.o 2020-02-05T17:41:52.154 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:41:52.160 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:41:53.018 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/signalfd.o 2020-02-05T17:41:59.494 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/timerfd.o 2020-02-05T17:42:05.915 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/eventfd.o 2020-02-05T17:42:12.032 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/aio.o 2020-02-05T17:42:19.439 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/io_uring.o 2020-02-05T17:42:22.292 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:42:22.298 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:42:28.642 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/locks.o 2020-02-05T17:42:35.589 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/compat.o 2020-02-05T17:42:41.901 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/compat_ioctl.o 2020-02-05T17:42:51.394 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/binfmt_misc.o 2020-02-05T17:42:52.421 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:42:52.427 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:42:57.804 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/binfmt_script.o 2020-02-05T17:43:02.449 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/binfmt_elf.o 2020-02-05T17:43:08.670 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/compat_binfmt_elf.o 2020-02-05T17:43:14.857 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/mbcache.o 2020-02-05T17:43:19.728 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/posix_acl.o 2020-02-05T17:43:22.602 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:43:22.607 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:43:24.439 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/coredump.o 2020-02-05T17:43:32.071 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/drop_caches.o 2020-02-05T17:43:37.624 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/fhandle.o 2020-02-05T17:43:44.224 INFO:tasks.workunit.client.0.smithi008.stdout: CC fs/dcookies.o 2020-02-05T17:43:50.975 INFO:tasks.workunit.client.0.smithi008.stdout: AR fs/built-in.a 2020-02-05T17:43:52.759 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:43:52.767 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:43:58.375 INFO:tasks.workunit.client.0.smithi008.stdout: OBJCOPY arch/x86/entry/vdso/vdso64.so 2020-02-05T17:43:58.398 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/events/hw_breakpoint.o 2020-02-05T17:43:58.419 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/livepatch/built-in.a 2020-02-05T17:43:58.432 INFO:tasks.workunit.client.0.smithi008.stdout: OBJCOPY arch/x86/entry/vdso/vdso32.so 2020-02-05T17:43:58.488 INFO:tasks.workunit.client.0.smithi008.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-64.c 2020-02-05T17:43:58.524 INFO:tasks.workunit.client.0.smithi008.stdout: VDSO2C arch/x86/entry/vdso/vdso-image-32.c 2020-02-05T17:43:58.594 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vdso-image-64.o 2020-02-05T17:43:58.669 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/compat.o 2020-02-05T17:44:00.780 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/vdso/vdso-image-32.o 2020-02-05T17:44:02.902 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/entry/vdso/built-in.a 2020-02-05T17:44:03.937 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/util.o 2020-02-05T17:44:09.720 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/msgutil.o 2020-02-05T17:44:14.995 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/msg.o 2020-02-05T17:44:21.673 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/sem.o 2020-02-05T17:44:22.894 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:44:22.900 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:44:28.582 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/shm.o 2020-02-05T17:44:35.899 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/syscall.o 2020-02-05T17:44:42.151 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/ipc_sysctl.o 2020-02-05T17:44:47.259 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/mqueue.o 2020-02-05T17:44:53.083 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:44:53.089 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:44:55.975 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/namespace.o 2020-02-05T17:45:00.843 INFO:tasks.workunit.client.0.smithi008.stdout: CC ipc/mq_sysctl.o 2020-02-05T17:45:04.829 INFO:tasks.workunit.client.0.smithi008.stdout: AR ipc/built-in.a 2020-02-05T17:45:05.189 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/entry_64.o 2020-02-05T17:45:05.211 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/freezer.o 2020-02-05T17:45:05.359 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/events/uprobes.o 2020-02-05T17:45:05.364 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/dummychip.o 2020-02-05T17:45:05.622 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/mempool.o 2020-02-05T17:45:05.702 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/integrity/iint.o 2020-02-05T17:45:06.201 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/gc.o 2020-02-05T17:45:10.658 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/devres.o 2020-02-05T17:45:11.620 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/legacy_freezer.o 2020-02-05T17:45:12.371 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/key.o 2020-02-05T17:45:12.457 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/oom_kill.o 2020-02-05T17:45:14.936 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/autoprobe.o 2020-02-05T17:45:16.087 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/cgroup/cpuset.o 2020-02-05T17:45:17.082 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/keyring.o 2020-02-05T17:45:18.807 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/irqdomain.o 2020-02-05T17:45:18.862 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/fadvise.o 2020-02-05T17:45:22.221 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/cgroup/built-in.a 2020-02-05T17:45:22.856 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/keyctl.o 2020-02-05T17:45:23.221 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:45:23.225 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:45:23.924 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/proc.o 2020-02-05T17:45:24.429 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/maccess.o 2020-02-05T17:45:27.968 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/migration.o 2020-02-05T17:45:28.295 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/permission.o 2020-02-05T17:45:28.299 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/page-writeback.o 2020-02-05T17:45:31.406 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/cpuhotplug.o 2020-02-05T17:45:32.439 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/process_keys.o 2020-02-05T17:45:34.204 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/readahead.o 2020-02-05T17:45:34.837 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/pm.o 2020-02-05T17:45:37.811 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/request_key.o 2020-02-05T17:45:39.530 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/swap.o 2020-02-05T17:45:39.686 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/msi.o 2020-02-05T17:45:44.490 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/affinity.o 2020-02-05T17:45:44.756 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/request_key_auth.o 2020-02-05T17:45:46.444 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/truncate.o 2020-02-05T17:45:47.943 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/irq/matrix.o 2020-02-05T17:45:48.772 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/user_defined.o 2020-02-05T17:45:51.371 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/vmscan.o 2020-02-05T17:45:52.804 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/compat.o 2020-02-05T17:45:53.257 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/irq/built-in.a 2020-02-05T17:45:53.368 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:45:53.372 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:45:58.067 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/proc.o 2020-02-05T17:45:58.200 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/shmem.o 2020-02-05T17:46:02.389 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/sysctl.o 2020-02-05T17:46:05.032 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/util.o 2020-02-05T17:46:06.464 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/keys/keyctl_pkey.o 2020-02-05T17:46:10.694 INFO:tasks.workunit.client.0.smithi008.stdout: CC mm/mmzone.o 2020-02-05T17:46:10.788 INFO:tasks.workunit.client.0.smithi008.stdout: AR security/keys/built-in.a 2020-02-05T17:46:11.500 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/mutex.o 2020-02-05T17:46:11.580 INFO:tasks.workunit.client.0.smithi008.stdout: GEN security/selinux/flask.h security/selinux/av_permissions.h 2020-02-05T17:46:11.683 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/selinux/avc.o 2020-02-05T17:46:12.146 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/asymmetric_type.o 2020-02-05T17:46:15.194 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/semaphore.o 2020-02-05T17:46:18.644 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/restrict.o 2020-02-05T17:46:19.243 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/selinux/hooks.o 2020-02-05T17:46:19.548 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/rwsem.o 2020-02-05T17:46:21.602 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/signature.o 2020-02-05T17:46:22.962 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/percpu-rwsem.o 2020-02-05T17:46:23.506 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:46:23.510 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:46:25.279 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/public_key.o 2020-02-05T17:46:25.904 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/spinlock.o 2020-02-05T17:46:28.496 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/osq_lock.o 2020-02-05T17:46:28.873 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/selinux/selinuxfs.o 2020-02-05T17:46:29.410 INFO:tasks.workunit.client.0.smithi008.stdout: ASN.1 crypto/asymmetric_keys/x509.asn1.[ch] 2020-02-05T17:46:29.442 INFO:tasks.workunit.client.0.smithi008.stdout: ASN.1 crypto/asymmetric_keys/x509_akid.asn1.[ch] 2020-02-05T17:46:29.493 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/x509_public_key.o 2020-02-05T17:46:31.456 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/qspinlock.o 2020-02-05T17:46:33.508 INFO:tasks.workunit.client.0.smithi008.stdout: ASN.1 crypto/asymmetric_keys/pkcs7.asn1.[ch] 2020-02-05T17:46:33.563 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/pkcs7_trust.o 2020-02-05T17:46:33.963 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/rtmutex.o 2020-02-05T17:46:36.256 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/pkcs7_verify.o 2020-02-05T17:46:36.361 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/selinux/netlink.o 2020-02-05T17:46:37.375 INFO:tasks.workunit.client.0.smithi008.stdout: CC kernel/locking/qrwlock.o 2020-02-05T17:46:39.688 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/x509.asn1.o 2020-02-05T17:46:39.860 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/locking/built-in.a 2020-02-05T17:46:40.095 INFO:tasks.workunit.client.0.smithi008.stdout: CC crypto/asymmetric_keys/x509_akid.asn1.o 2020-02-05T17:46:40.262 INFO:tasks.workunit.client.0.smithi008.stdout: CC security/integrity/integrity_audit.o 2020-02-05T17:46:40.383 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/thunk_64.o 2020-02-05T17:46:40.558 INFO:tasks.workunit.client.0.smithi008.stdout: AR kernel/events/built-in.a 2020-02-05T17:46:41.430 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/syscall_64.o 2020-02-05T17:46:41.436 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/partitions/check.o 2020-02-05T17:46:43.083 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsargs.o 2020-02-05T17:46:44.724 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/common.o 2020-02-05T17:46:46.449 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/partitions/msdos.o 2020-02-05T17:46:46.617 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dscontrol.o 2020-02-05T17:46:52.579 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsdebug.o 2020-02-05T17:46:52.830 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/partitions/efi.o 2020-02-05T17:46:53.034 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/entry/entry_64_compat.o 2020-02-05T17:46:53.604 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/entry/syscall_32.o 2020-02-05T17:46:53.645 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:46:53.650 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:46:55.817 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsfield.o 2020-02-05T17:46:56.589 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/entry/built-in.a 2020-02-05T17:46:57.471 INFO:tasks.workunit.client.0.smithi008.stdout: AR block/partitions/built-in.a 2020-02-05T17:46:57.500 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/amd/core.o 2020-02-05T17:46:57.663 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/bio.o 2020-02-05T17:46:59.238 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsinit.o 2020-02-05T17:47:02.435 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/amd/uncore.o 2020-02-05T17:47:02.500 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsmethod.o 2020-02-05T17:47:03.366 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/elevator.o 2020-02-05T17:47:05.900 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsmthdat.o 2020-02-05T17:47:07.308 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/amd/ibs.o 2020-02-05T17:47:08.680 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-core.o 2020-02-05T17:47:09.219 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsobject.o 2020-02-05T17:47:12.468 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/amd/iommu.o 2020-02-05T17:47:12.549 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsopcode.o 2020-02-05T17:47:15.207 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-sysfs.o 2020-02-05T17:47:15.846 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dspkginit.o 2020-02-05T17:47:17.302 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/events/amd/built-in.a 2020-02-05T17:47:17.845 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/core.o 2020-02-05T17:47:19.185 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dsutils.o 2020-02-05T17:47:20.427 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-flush.o 2020-02-05T17:47:22.505 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dswexec.o 2020-02-05T17:47:23.334 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/bts.o 2020-02-05T17:47:23.792 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:47:23.798 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:47:26.879 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dswload.o 2020-02-05T17:47:27.078 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-settings.o 2020-02-05T17:47:28.854 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/ds.o 2020-02-05T17:47:30.239 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dswload2.o 2020-02-05T17:47:31.841 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-ioc.o 2020-02-05T17:47:33.615 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dswscope.o 2020-02-05T17:47:34.119 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/knc.o 2020-02-05T17:47:36.284 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-map.o 2020-02-05T17:47:36.868 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/dswstate.o 2020-02-05T17:47:38.882 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/lbr.o 2020-02-05T17:47:40.219 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evevent.o 2020-02-05T17:47:40.835 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-exec.o 2020-02-05T17:47:43.526 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evgpe.o 2020-02-05T17:47:43.840 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/p4.o 2020-02-05T17:47:45.271 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-merge.o 2020-02-05T17:47:46.950 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evgpeblk.o 2020-02-05T17:47:48.755 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/p6.o 2020-02-05T17:47:50.285 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evgpeinit.o 2020-02-05T17:47:50.379 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-softirq.o 2020-02-05T17:47:53.560 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/pt.o 2020-02-05T17:47:53.602 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evgpeutil.o 2020-02-05T17:47:54.988 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-timeout.o 2020-02-05T17:47:55.902 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:47:55.907 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:47:56.965 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evglock.o 2020-02-05T17:48:01.039 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evhandler.o 2020-02-05T17:48:01.267 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-lib.o 2020-02-05T17:48:01.410 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/rapl.o 2020-02-05T17:48:04.397 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evmisc.o 2020-02-05T17:48:05.749 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq.o 2020-02-05T17:48:06.424 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/uncore.o 2020-02-05T17:48:07.734 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evregion.o 2020-02-05T17:48:10.977 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evrgnini.o 2020-02-05T17:48:11.684 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/uncore_nhmex.o 2020-02-05T17:48:12.113 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-tag.o 2020-02-05T17:48:14.364 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evsci.o 2020-02-05T17:48:16.691 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-stat.o 2020-02-05T17:48:16.903 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/uncore_snb.o 2020-02-05T17:48:17.645 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evxface.o 2020-02-05T17:48:21.026 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-sysfs.o 2020-02-05T17:48:21.086 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evxfevnt.o 2020-02-05T17:48:21.929 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/uncore_snbep.o 2020-02-05T17:48:24.463 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evxfgpe.o 2020-02-05T17:48:25.928 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-cpumap.o 2020-02-05T17:48:26.060 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:48:26.064 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:48:27.536 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/intel/cstate.o 2020-02-05T17:48:27.910 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/evxfregn.o 2020-02-05T17:48:30.361 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-sched.o 2020-02-05T17:48:31.213 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exconcat.o 2020-02-05T17:48:33.047 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/events/intel/built-in.a 2020-02-05T17:48:34.011 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/core.o 2020-02-05T17:48:35.625 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exconfig.o 2020-02-05T17:48:35.987 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/ioctl.o 2020-02-05T17:48:38.952 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exconvrt.o 2020-02-05T17:48:39.979 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/probe.o 2020-02-05T17:48:40.960 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/genhd.o 2020-02-05T17:48:42.338 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/excreate.o 2020-02-05T17:48:42.645 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/events/msr.o 2020-02-05T17:48:45.687 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exdebug.o 2020-02-05T17:48:46.212 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/partition-generic.o 2020-02-05T17:48:47.468 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/events/built-in.a 2020-02-05T17:48:48.799 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/ia32/sys_ia32.o 2020-02-05T17:48:49.023 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exdump.o 2020-02-05T17:48:51.142 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/ioprio.o 2020-02-05T17:48:52.290 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exfield.o 2020-02-05T17:48:54.133 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/ia32/ia32_signal.o 2020-02-05T17:48:55.656 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exfldio.o 2020-02-05T17:48:56.210 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:48:56.214 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:48:56.388 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/badblocks.o 2020-02-05T17:48:59.451 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/ia32/audit.o 2020-02-05T17:48:59.710 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exmisc.o 2020-02-05T17:49:00.394 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/ia32/built-in.a 2020-02-05T17:49:01.272 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-rq-qos.o 2020-02-05T17:49:01.274 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/acpi/boot.o 2020-02-05T17:49:03.230 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exmutex.o 2020-02-05T17:49:05.641 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/bounce.o 2020-02-05T17:49:06.597 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exnames.o 2020-02-05T17:49:06.953 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/acpi/sleep.o 2020-02-05T17:49:09.910 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exoparg1.o 2020-02-05T17:49:10.966 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/scsi_ioctl.o 2020-02-05T17:49:11.637 INFO:tasks.workunit.client.0.smithi008.stdout: AS arch/x86/kernel/acpi/wakeup_64.o 2020-02-05T17:49:12.053 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/acpi/cppc_msr.o 2020-02-05T17:49:13.327 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exoparg2.o 2020-02-05T17:49:15.651 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/bsg.o 2020-02-05T17:49:16.323 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/acpi/cstate.o 2020-02-05T17:49:16.698 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exoparg3.o 2020-02-05T17:49:20.079 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exoparg6.o 2020-02-05T17:49:20.370 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/mq-deadline.o 2020-02-05T17:49:20.714 INFO:tasks.workunit.client.0.smithi008.stdout: AR arch/x86/kernel/acpi/built-in.a 2020-02-05T17:49:21.306 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/apic.o 2020-02-05T17:49:23.380 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exprep.o 2020-02-05T17:49:24.968 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/kyber-iosched.o 2020-02-05T17:49:26.433 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:49:26.438 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:49:26.715 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exregion.o 2020-02-05T17:49:27.701 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/apic_common.o 2020-02-05T17:49:30.008 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exresnte.o 2020-02-05T17:49:30.502 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/apic_noop.o 2020-02-05T17:49:30.521 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/compat_ioctl.o 2020-02-05T17:49:36.020 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/ipi.o 2020-02-05T17:49:36.325 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exresolv.o 2020-02-05T17:49:36.768 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-pci.o 2020-02-05T17:49:38.665 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/vector.o 2020-02-05T17:49:39.623 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exresop.o 2020-02-05T17:49:41.420 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-mq-debugfs.o 2020-02-05T17:49:42.934 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exserial.o 2020-02-05T17:49:43.683 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/hw_nmi.o 2020-02-05T17:49:45.939 INFO:tasks.workunit.client.0.smithi008.stdout: CC block/blk-pm.o 2020-02-05T17:49:46.241 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exstore.o 2020-02-05T17:49:48.014 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/kernel/apic/io_apic.o 2020-02-05T17:49:49.609 INFO:tasks.workunit.client.0.smithi008.stdout: CC drivers/acpi/acpica/exstoren.o 2020-02-05T17:49:50.357 INFO:tasks.workunit.client.0.smithi008.stdout: AR block/built-in.a 2020-02-05T17:49:51.608 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/init.o 2020-02-05T17:49:56.618 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:49:56.624 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:49:58.128 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/init_64.o 2020-02-05T17:50:05.109 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/fault.o 2020-02-05T17:50:12.728 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/ioremap.o 2020-02-05T17:50:19.251 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/extable.o 2020-02-05T17:50:25.064 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/pageattr.o 2020-02-05T17:50:26.749 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:50:26.756 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:50:31.049 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/mmap.o 2020-02-05T17:50:36.382 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/pat.o 2020-02-05T17:50:41.931 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/pgtable.o 2020-02-05T17:50:48.288 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/physaddr.o 2020-02-05T17:50:52.985 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/setup_nx.o 2020-02-05T17:50:56.880 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/tlb.o 2020-02-05T17:50:56.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:50:56.895 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:51:02.489 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/cpu_entry_area.o 2020-02-05T17:51:07.640 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/pat_rbtree.o 2020-02-05T17:51:12.700 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/hugetlbpage.o 2020-02-05T17:51:19.320 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/numa.o 2020-02-05T17:51:25.423 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/numa_64.o 2020-02-05T17:51:26.560 INFO:tasks.ceph.mon.c.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f46fb2bc700 time 2020-02-05T17:51:26.559417+0000 2020-02-05T17:51:26.561 INFO:tasks.ceph.mon.c.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.561 INFO:tasks.ceph.mon.b.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f874768b700 time 2020-02-05T17:51:26.559474+0000 2020-02-05T17:51:26.561 INFO:tasks.ceph.mon.b.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f8754d18324] 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.564 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr:*** Caught signal (Aborted) ** 2020-02-05T17:51:26.565 INFO:tasks.ceph.mon.b.smithi164.stderr: in thread 7f874768b700 thread_name:ms_dispatch 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr:2020-02-05T17:51:26.561+0000 7f874768b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f874768b700 time 2020-02-05T17:51:26.559474+0000 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f8754d18324] 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.566 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.567 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.568 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.568 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.568 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.568 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f4708949324] 2020-02-05T17:51:26.568 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.569 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr:2020-02-05T17:51:26.561+0000 7f46fb2bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f46fb2bc700 time 2020-02-05T17:51:26.559417+0000 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.570 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f4708949324] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.571 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr:*** Caught signal (Aborted) ** 2020-02-05T17:51:26.572 INFO:tasks.ceph.mon.c.smithi164.stderr: in thread 7f46fb2bc700 thread_name:ms_dispatch 2020-02-05T17:51:26.573 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.573 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (()+0x12dc0) [0x7f8752a62dc0] 2020-02-05T17:51:26.573 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f875173d8df] 2020-02-05T17:51:26.573 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (abort()+0x127) [0x7f8751727cf5] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f8754d18375] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.574 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: 13: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: 14: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr:2020-02-05T17:51:26.565+0000 7f874768b700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: in thread 7f874768b700 thread_name:ms_dispatch 2020-02-05T17:51:26.575 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (()+0x12dc0) [0x7f8752a62dc0] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f875173d8df] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (abort()+0x127) [0x7f8751727cf5] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f8754d18375] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.576 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 13: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.577 INFO:tasks.ceph.mon.b.smithi164.stderr: 14: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.b.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (()+0x12dc0) [0x7f4706693dc0] 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f470536e8df] 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (abort()+0x127) [0x7f4705358cf5] 2020-02-05T17:51:26.578 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f4708949375] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.579 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: 13: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: 14: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr:2020-02-05T17:51:26.565+0000 7f46fb2bc700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: in thread 7f46fb2bc700 thread_name:ms_dispatch 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.580 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (()+0x12dc0) [0x7f4706693dc0] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f470536e8df] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (abort()+0x127) [0x7f4705358cf5] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f4708949375] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.581 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 13: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: 14: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.582 INFO:tasks.ceph.mon.c.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.583 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.590 INFO:tasks.ceph.mon.b.smithi164.stderr: -1> 2020-02-05T17:51:26.561+0000 7f874768b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f874768b700 time 2020-02-05T17:51:26.559474+0000 2020-02-05T17:51:26.590 INFO:tasks.ceph.mon.b.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f8754d18324] 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.591 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.592 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: 0> 2020-02-05T17:51:26.565+0000 7f874768b700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: in thread 7f874768b700 thread_name:ms_dispatch 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (()+0x12dc0) [0x7f8752a62dc0] 2020-02-05T17:51:26.593 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f875173d8df] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (abort()+0x127) [0x7f8751727cf5] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f8754d18375] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.594 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 13: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: 14: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.595 INFO:tasks.ceph.mon.b.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.596 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.596 INFO:tasks.ceph.mon.c.smithi164.stderr: -1> 2020-02-05T17:51:26.561+0000 7f46fb2bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f46fb2bc700 time 2020-02-05T17:51:26.559417+0000 2020-02-05T17:51:26.596 INFO:tasks.ceph.mon.c.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.596 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f4708949324] 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.597 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.598 INFO:tasks.ceph.mon.c.smithi164.stderr: 0> 2020-02-05T17:51:26.565+0000 7f46fb2bc700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: in thread 7f46fb2bc700 thread_name:ms_dispatch 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (()+0x12dc0) [0x7f4706693dc0] 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f470536e8df] 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (abort()+0x127) [0x7f4705358cf5] 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f4708949375] 2020-02-05T17:51:26.599 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.600 INFO:tasks.ceph.mon.c.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.601 INFO:tasks.ceph.mon.c.smithi164.stderr: 13: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.601 INFO:tasks.ceph.mon.c.smithi164.stderr: 14: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.601 INFO:tasks.ceph.mon.c.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.601 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.603 INFO:tasks.ceph.mon.b.smithi164.stderr: -1> 2020-02-05T17:51:26.561+0000 7f874768b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f874768b700 time 2020-02-05T17:51:26.559474+0000 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f8754d18324] 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.604 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.605 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 0> 2020-02-05T17:51:26.565+0000 7f874768b700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: in thread 7f874768b700 thread_name:ms_dispatch 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 1: (()+0x12dc0) [0x7f8752a62dc0] 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f875173d8df] 2020-02-05T17:51:26.606 INFO:tasks.ceph.mon.b.smithi164.stderr: 3: (abort()+0x127) [0x7f8751727cf5] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f8754d18375] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 5: (()+0x27b53e) [0x7f8754d1853e] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x55ab2adca379] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x55ab2adcfb37] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x55ab2acdb3f9] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x55ab2acdc303] 2020-02-05T17:51:26.607 INFO:tasks.ceph.mon.b.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x55ab2ad0b48c] 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f8754f31b8a] 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8754fd5f01] 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: 13: (()+0x82de) [0x7f8752a582de] 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: 14: (clone()+0x43) [0x7f8751802133] 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.608 INFO:tasks.ceph.mon.b.smithi164.stderr: 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: -1> 2020-02-05T17:51:26.561+0000 7f46fb2bc700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: In function 'ceph::time_detail::timespan ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f46fb2bc700 time 2020-02-05T17:51:26.559417+0000 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/huge/release/15.0.0-9851-gecafd72/rpm/el8/BUILD/ceph-15.0.0-9851-gecafd72/src/common/ceph_time.h: 485: FAILED ceph_assert(z >= signedspan::zero()) 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x156) [0x7f4708949324] 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.609 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.610 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 0> 2020-02-05T17:51:26.565+0000 7f46fb2bc700 -1 *** Caught signal (Aborted) ** 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: in thread 7f46fb2bc700 thread_name:ms_dispatch 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: ceph version 15.0.0-9851-gecafd72 (ecafd7283ef390be73a605986afe2a0f325a3255) octopus (dev) 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 1: (()+0x12dc0) [0x7f4706693dc0] 2020-02-05T17:51:26.611 INFO:tasks.ceph.mon.c.smithi164.stderr: 2: (gsignal()+0x10f) [0x7f470536e8df] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 3: (abort()+0x127) [0x7f4705358cf5] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a7) [0x7f4708949375] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 5: (()+0x27b53e) [0x7f470894953e] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 6: (Paxos::handle_begin(boost::intrusive_ptr)+0x9a9) [0x5568d52d2379] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 7: (Paxos::dispatch(boost::intrusive_ptr)+0x3d7) [0x5568d52d7b37] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 8: (Monitor::dispatch_op(boost::intrusive_ptr)+0x1089) [0x5568d51e33f9] 2020-02-05T17:51:26.612 INFO:tasks.ceph.mon.c.smithi164.stderr: 9: (Monitor::_ms_dispatch(Message*)+0xa33) [0x5568d51e4303] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 10: (Dispatcher::ms_dispatch2(boost::intrusive_ptr const&)+0x5c) [0x5568d521348c] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 11: (DispatchQueue::entry()+0x126a) [0x7f4708b62b8a] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 12: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f4708c06f01] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 13: (()+0x82de) [0x7f47066892de] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 14: (clone()+0x43) [0x7f4705433133] 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2020-02-05T17:51:26.613 INFO:tasks.ceph.mon.c.smithi164.stderr: 2020-02-05T17:51:27.025 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:51:27.031 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:51:27.313 INFO:tasks.ceph.mon.c.smithi164.stderr:daemon-helper: command crashed with signal 6 2020-02-05T17:51:27.333 INFO:tasks.ceph.mon.b.smithi164.stderr:daemon-helper: command crashed with signal 6 2020-02-05T17:51:30.149 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/amdtopology.o 2020-02-05T17:51:30.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2020-02-05T17:51:30.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2020-02-05T17:51:36.042 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/srat.o 2020-02-05T17:51:36.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~6s 2020-02-05T17:51:36.724 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~6s 2020-02-05T17:51:41.580 INFO:tasks.workunit.client.0.smithi008.stdout: CC arch/x86/mm/pkeys.o 2020-02-05T17:51:43.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~13s 2020-02-05T17:51:43.035 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~13s 2020-02-05T17:51:49.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~19s 2020-02-05T17:51:49.347 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~19s 2020-02-05T17:51:55.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~25s 2020-02-05T17:51:55.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~25s 2020-02-05T17:51:57.923 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:51:57.930 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:52:01.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~32s 2020-02-05T17:52:01.974 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~32s 2020-02-05T17:52:08.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~38s 2020-02-05T17:52:08.289 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~38s 2020-02-05T17:52:14.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~44s 2020-02-05T17:52:14.602 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~44s 2020-02-05T17:52:20.914 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~51s 2020-02-05T17:52:20.915 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~51s 2020-02-05T17:52:27.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~57s 2020-02-05T17:52:27.225 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~57s 2020-02-05T17:52:28.101 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:52:28.108 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:52:33.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~63s 2020-02-05T17:52:33.538 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~63s 2020-02-05T17:52:39.849 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~69s 2020-02-05T17:52:39.850 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~69s 2020-02-05T17:52:46.161 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~76s 2020-02-05T17:52:46.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~76s 2020-02-05T17:52:52.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~82s 2020-02-05T17:52:52.475 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~82s 2020-02-05T17:52:58.242 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:52:58.249 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:52:58.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~88s 2020-02-05T17:52:58.786 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~88s 2020-02-05T17:53:05.099 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~95s 2020-02-05T17:53:05.100 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~95s 2020-02-05T17:53:11.410 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~101s 2020-02-05T17:53:11.411 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~101s 2020-02-05T17:53:17.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~107s 2020-02-05T17:53:17.721 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~107s 2020-02-05T17:53:24.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~114s 2020-02-05T17:53:24.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~114s 2020-02-05T17:53:28.371 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:53:28.377 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:53:30.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~120s 2020-02-05T17:53:30.346 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~120s 2020-02-05T17:53:32.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:32.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:36.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~126s 2020-02-05T17:53:36.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~126s 2020-02-05T17:53:37.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:37.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:42.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:42.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:42.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~133s 2020-02-05T17:53:42.970 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~133s 2020-02-05T17:53:47.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:47.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:49.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~139s 2020-02-05T17:53:49.282 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~139s 2020-02-05T17:53:52.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:52.474+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:55.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~145s 2020-02-05T17:53:55.594 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~145s 2020-02-05T17:53:57.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:53:57.475+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:53:58.543 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:53:58.549 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:54:01.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~151s 2020-02-05T17:54:01.906 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~151s 2020-02-05T17:54:02.478 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:02.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:07.479 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:07.477+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:08.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~158s 2020-02-05T17:54:08.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~158s 2020-02-05T17:54:12.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:12.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:14.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~164s 2020-02-05T17:54:14.537 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~164s 2020-02-05T17:54:17.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:17.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:20.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~170s 2020-02-05T17:54:20.855 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~170s 2020-02-05T17:54:22.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:22.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:27.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~177s 2020-02-05T17:54:27.172 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~177s 2020-02-05T17:54:27.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:27.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:28.716 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:54:28.722 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:54:32.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:32.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~183s 2020-02-05T17:54:33.483 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~183s 2020-02-05T17:54:37.488 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:37.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:39.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~189s 2020-02-05T17:54:39.794 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~189s 2020-02-05T17:54:42.488 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:42.485+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:46.109 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~196s 2020-02-05T17:54:46.110 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~196s 2020-02-05T17:54:47.490 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:47.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 281 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:52.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~202s 2020-02-05T17:54:52.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~202s 2020-02-05T17:54:52.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:52.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:57.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:54:57.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:54:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~208s 2020-02-05T17:54:58.739 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~208s 2020-02-05T17:54:58.862 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:54:58.868 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:55:02.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:02.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:05.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~215s 2020-02-05T17:55:05.051 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~215s 2020-02-05T17:55:07.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:07.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:11.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~221s 2020-02-05T17:55:11.375 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~221s 2020-02-05T17:55:12.496 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:12.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:17.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:17.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:17.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~227s 2020-02-05T17:55:17.688 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~227s 2020-02-05T17:55:22.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:22.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:24.004 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~234s 2020-02-05T17:55:24.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~234s 2020-02-05T17:55:27.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:27.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 437 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:28.999 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:55:29.005 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:55:30.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~240s 2020-02-05T17:55:30.320 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~240s 2020-02-05T17:55:32.505 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:32.500+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:36.631 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~246s 2020-02-05T17:55:36.632 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~246s 2020-02-05T17:55:37.503 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:37.501+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 476 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:42.505 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:42.503+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:42.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~253s 2020-02-05T17:55:42.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~253s 2020-02-05T17:55:47.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:47.503+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:49.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~259s 2020-02-05T17:55:49.255 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~259s 2020-02-05T17:55:52.508 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:52.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:55.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~265s 2020-02-05T17:55:55.568 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~265s 2020-02-05T17:55:57.508 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:55:57.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:55:59.139 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:55:59.151 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:56:01.881 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~271s 2020-02-05T17:56:01.882 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~271s 2020-02-05T17:56:02.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:02.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:07.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:07.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 593 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:08.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~278s 2020-02-05T17:56:08.201 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~278s 2020-02-05T17:56:12.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:12.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:14.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~284s 2020-02-05T17:56:14.511 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~284s 2020-02-05T17:56:17.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:17.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:20.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~290s 2020-02-05T17:56:20.825 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~290s 2020-02-05T17:56:22.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:22.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:27.139 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~297s 2020-02-05T17:56:27.140 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~297s 2020-02-05T17:56:27.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:27.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 671 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:29.385 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:56:29.391 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:56:32.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:32.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 689 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:33.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~303s 2020-02-05T17:56:33.452 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~303s 2020-02-05T17:56:33.453 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2020-02-05T17:56:33.453 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T17:56:33.453 INFO:teuthology.orchestra.run.smithi008:> stat --file-system '--printf=%T 2020-02-05T17:56:33.453 INFO:teuthology.orchestra.run.smithi008:> ' -- /home/ubuntu/cephtest/mnt.0 2020-02-05T17:56:37.519 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:37.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:42.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:42.518+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:47.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:47.520+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:52.523 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:52.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 767 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:57.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:56:57.523+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:56:59.529 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:56:59.535 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:57:02.527 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:02.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:07.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:07.526+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:12.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:12.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:17.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:17.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 866 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:22.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:22.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 884 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:27.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:27.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:29.671 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:57:29.677 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:57:32.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:32.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:37.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:37.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:42.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:42.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:47.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:47.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:52.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:52.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:57.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:57:57.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:57:59.808 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:57:59.814 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:58:02.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:02.540+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:07.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:07.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:12.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:12.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:17.548 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:17.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:22.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:22.546+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:27.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:27.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1138 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:29.955 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:58:29.962 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:58:32.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:32.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:37.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:37.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1179 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:42.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:42.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:47.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:47.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:52.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:52.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:58:57.563 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:58:57.560+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1261 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:00.123 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:59:00.130 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:59:02.566 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:02.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1281 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:07.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:07.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:12.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:12.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:17.570 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:17.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1342 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:22.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:22.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:27.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:27.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:30.263 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:59:30.269 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T17:59:32.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:32.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:37.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:37.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1424 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:42.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:42.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1445 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:47.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:47.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:52.579 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:52.578+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1486 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T17:59:57.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T17:59:57.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:00.395 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:00:00.401 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:00:02.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:02.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1527 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:07.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:07.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:12.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:12.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1568 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:17.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:17.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:22.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:22.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:27.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:27.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:30.542 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:00:30.549 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:00:32.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:32.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:37.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:37.590+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1670 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:42.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:42.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:47.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:47.594+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:52.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:52.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1731 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:00:57.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:00:57.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:00.674 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:01:00.680 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:01:02.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:02.598+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:07.603 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:07.600+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:12.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:12.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:17.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:17.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:22.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:22.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:27.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:27.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:30.826 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:01:30.834 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:01:32.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:32.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:37.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:37.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:42.614 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:42.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:47.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:47.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1957 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:52.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:52.614+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1976 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:01:57.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:01:57.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 1998 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:00.972 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:02:00.978 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:02:02.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:02.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:07.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:07.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:12.623 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:12.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2058 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:17.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:17.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:22.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:22.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:27.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:27.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:31.119 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:02:31.125 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:02:32.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:32.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:37.631 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:37.629+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:42.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:42.630+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2181 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:47.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:47.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:52.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:52.633+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:02:57.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:02:57.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:01.258 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:03:01.264 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:03:02.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:02.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:07.641 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:07.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:12.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:12.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:17.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:17.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2326 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:22.650 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:22.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:27.647 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:27.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:31.387 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:03:31.394 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:03:32.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:32.646+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:37.650 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:37.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:42.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:42.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:47.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:47.650+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:52.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:52.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:03:57.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:03:57.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:01.561 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:04:01.568 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:04:02.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:02.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2509 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:07.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:07.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:12.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:12.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:17.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:17.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:22.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:22.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2591 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:27.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:27.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:31.714 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:04:31.721 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:04:32.671 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:32.669+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:37.673 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:37.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:42.677 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:42.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2673 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:47.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:47.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2693 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:52.678 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:52.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2714 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:04:57.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:04:57.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:01.894 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:05:01.903 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:05:02.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:02.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:07.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:07.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:12.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:12.682+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:17.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:17.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:22.689 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:22.686+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:27.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:27.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:32.032 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:05:32.038 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:05:32.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:32.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:37.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:37.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:42.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:42.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2919 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:47.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:47.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:52.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:52.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2960 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:05:57.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:05:57.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 2980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:02.221 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:06:02.228 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:06:02.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:02.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:07.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:07.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:12.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:12.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3042 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:17.707 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:17.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:22.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:22.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:27.712 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:27.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:32.343 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:06:32.349 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:06:32.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:32.710+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:37.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:37.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:42.717 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:42.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3164 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:47.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:47.715+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:52.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:52.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3205 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:06:57.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:06:57.721+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3226 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:02.488 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:07:02.495 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:07:02.725 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:02.722+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3246 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:07.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:07.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:12.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:12.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3286 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:17.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:17.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:22.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:22.729+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:27.733 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:27.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:32.618 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:07:32.624 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:07:32.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:32.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:37.737 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:37.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:42.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:42.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:47.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:47.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:52.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:52.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:07:57.744 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:07:57.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:02.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:02.743+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3491 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:02.799 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:08:02.805 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:08:07.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:07.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:12.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:12.746+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:17.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:17.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:22.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:22.750+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:27.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:27.753+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:32.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:32.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3614 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:32.929 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:08:32.936 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:08:37.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:37.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3636 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:42.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:42.759+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:47.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:47.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3677 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:52.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:52.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:08:57.768 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:08:57.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:02.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:02.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3737 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:03.084 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:09:03.090 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:09:07.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:07.768+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:12.773 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:12.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:17.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:17.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:22.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:22.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:27.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:27.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:32.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:32.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:33.240 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:09:33.246 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:09:37.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:37.779+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:42.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:42.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3901 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:47.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:47.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3922 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:52.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:52.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:09:57.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:09:57.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:02.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:02.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 3983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:03.386 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:10:03.392 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:10:07.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:07.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:12.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:12.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:17.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:17.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:22.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:22.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4065 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:27.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:27.801+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:32.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:32.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4106 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:33.516 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:10:33.523 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:10:37.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:37.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4126 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:42.810 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:42.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:47.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:47.809+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4167 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:52.815 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:52.811+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:10:57.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:10:57.813+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4208 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:02.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:02.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:03.656 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:11:03.662 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:11:07.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:07.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4249 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:12.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:12.819+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4270 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:17.824 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:17.821+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4290 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:22.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:22.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4311 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:27.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:27.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4331 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:32.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:32.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4352 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:33.461 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 272, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2020-02-05T18:11:33.462 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 272, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2020-02-05T18:11:33.809 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:11:33.816 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:11:37.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:37.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4372 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:42.835 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:42.833+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4393 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:47.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:47.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:52.841 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:52.836+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4433 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:11:57.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:11:57.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:02.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:02.839+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:03.971 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:12:03.977 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:12:07.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:07.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:12.847 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:12.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:17.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:17.845+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4536 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:22.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:22.847+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4556 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:27.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:27.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:32.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:32.851+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:34.130 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:12:34.137 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:12:37.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:37.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:42.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:42.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:47.860 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:47.857+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:52.863 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:52.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4679 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:12:57.865 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:12:57.863+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4700 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:02.868 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:02.865+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:04.281 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:13:04.288 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:13:07.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:07.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4741 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:12.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:12.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:17.874 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:17.872+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4782 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:22.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:22.874+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:27.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:27.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:32.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:32.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4843 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:34.412 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:13:34.419 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:13:37.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:37.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:42.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:42.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:47.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:47.884+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:52.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:52.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:13:57.893 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:13:57.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:02.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:02.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4965 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:04.558 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:14:04.565 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:14:07.896 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:07.892+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 4987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:12.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:12.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:17.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:17.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:22.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:22.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5047 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:27.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:27.902+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:32.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:32.904+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5089 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:34.722 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:14:34.729 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:14:37.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:37.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:42.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:42.909+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:47.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:47.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:52.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:52.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5171 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:14:57.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:14:57.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5191 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:02.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:02.917+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:04.877 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:15:04.884 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:15:07.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:07.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5232 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:12.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:12.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:17.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:17.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:22.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:22.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:27.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:27.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:32.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:32.930+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:35.020 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:15:35.026 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:15:37.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:37.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:42.937 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:42.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:47.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:47.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:52.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:52.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:15:57.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:15:57.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:02.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:02.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:05.221 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:16:05.227 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:16:07.949 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:07.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5477 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:12.951 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:12.948+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:17.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:17.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5518 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:22.955 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:22.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:27.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:27.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5559 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:32.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:32.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:35.417 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:16:35.424 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:16:37.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:37.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:42.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:42.962+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:47.968 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:47.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5641 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:52.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:52.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:16:57.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:16:57.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:02.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:02.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:05.622 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:17:05.628 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:17:07.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:07.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:12.979 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:12.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:17.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:17.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5764 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:22.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:22.981+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:27.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:27.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5805 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:32.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:32.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:35.793 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:17:35.799 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:17:37.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:37.988+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:42.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:42.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:47.996 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:47.992+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:52.996 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:52.994+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:17:57.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:17:57.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:03.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:02.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:06.007 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:18:06.013 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:18:08.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:07.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:13.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:13.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 5990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:18.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:18.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:23.011 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:23.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:28.012 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:28.008+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:33.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:33.011+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6072 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:36.142 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:18:36.152 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:18:38.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:38.013+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:43.019 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:43.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:48.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:48.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:53.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:53.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:18:58.027 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:18:58.025+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6174 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:03.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:03.027+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:06.340 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:19:06.346 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:19:08.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:08.029+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:13.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:13.031+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:18.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:18.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:23.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:23.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6276 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:28.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:28.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:33.043 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:33.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:36.538 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:19:36.544 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:19:38.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:38.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:43.050 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:43.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6358 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:48.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:48.047+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:53.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:53.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:19:58.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:19:58.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:03.059 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:03.054+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:06.692 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:20:06.699 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:20:08.061 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:08.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:13.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:13.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:18.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:18.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:23.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:23.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:28.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:28.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:33.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:33.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6564 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:36.850 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:20:36.856 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:20:38.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:38.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:43.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:43.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6604 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:48.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:48.077+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6624 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:53.082 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:53.079+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6646 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:20:58.085 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:20:58.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:03.087 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:03.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:06.999 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:21:07.006 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:21:08.089 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:08.086+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:13.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:13.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:18.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:18.090+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:23.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:23.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6768 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:28.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:28.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6787 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:33.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:33.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6810 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:37.145 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:21:37.151 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:21:38.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:38.099+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:43.107 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:43.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:48.108 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:48.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6869 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:53.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:53.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:21:58.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:21:58.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:03.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:03.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:07.284 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:22:07.290 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:22:08.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:08.118+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6951 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:13.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:13.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:18.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:18.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 6992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:23.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:23.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7014 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:28.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:28.126+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7033 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:33.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:33.129+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7056 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:37.424 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:22:37.429 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:22:38.135 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:38.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:43.137 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:43.134+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7096 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:48.141 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:48.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:53.142 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:53.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7137 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:22:58.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:22:58.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7156 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:03.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:03.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:07.656 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:23:07.662 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:23:08.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:08.145+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:13.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:13.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:18.155 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:18.152+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:23.158 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:23.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:28.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:28.156+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:33.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:33.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:37.803 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:23:37.809 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:23:38.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:38.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:43.167 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:43.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:48.170 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:48.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7361 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:53.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:53.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:23:58.176 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:23:58.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:03.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:03.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:07.989 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:24:07.996 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:24:08.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:08.176+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:13.182 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:13.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:18.184 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:18.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:23.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:23.184+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:28.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:28.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:33.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:33.189+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:38.195 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:38.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:38.217 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:24:38.221 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:24:43.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:43.195+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:48.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:48.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7607 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:53.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:53.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7628 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:24:58.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:24:58.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:03.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:03.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7667 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:08.211 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:08.209+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7688 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:08.349 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:25:08.355 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:25:13.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:13.212+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:18.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:18.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:23.223 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:23.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:28.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:28.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7769 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:33.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:33.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:38.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:38.225+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7810 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:38.545 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:25:38.551 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:25:43.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:43.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:48.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:48.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:53.236 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:53.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:25:58.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:25:58.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7891 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:03.243 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:03.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:08.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:08.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:08.734 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:26:08.740 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:26:13.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:13.244+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:18.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:18.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:23.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:23.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 7994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:28.255 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:28.252+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8014 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:33.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:33.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:38.261 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:38.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:38.946 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:26:38.953 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:26:43.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:43.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:48.265 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:48.263+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:53.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:53.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:26:58.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:26:58.267+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8136 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:03.274 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:03.269+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:08.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:08.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:09.134 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:27:09.140 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:27:13.278 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:13.275+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:18.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:18.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:23.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:23.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:28.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:28.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:33.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:33.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:38.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:38.289+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:39.314 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:27:39.320 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:27:43.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:43.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:48.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:48.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:53.300 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:53.297+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:27:58.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:27:58.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:03.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:03.302+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:08.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:08.305+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8423 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:09.451 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:28:09.457 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:28:13.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:13.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:18.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:18.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:23.317 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:23.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8486 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:28.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:28.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:33.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:33.317+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8527 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:38.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:38.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:39.666 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:28:39.672 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:28:43.326 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:43.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:48.330 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:48.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:53.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:53.328+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:28:58.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:28:58.331+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8628 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:03.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:03.333+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8649 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:08.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:08.336+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:09.879 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:29:09.885 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:29:13.343 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:13.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:18.345 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:18.342+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:23.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:23.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8731 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:28.351 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:28.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:33.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:33.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:38.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:38.353+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:40.123 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:29:40.130 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:29:43.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:43.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:48.362 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:48.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:53.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:53.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:29:58.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:29:58.364+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:03.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:03.367+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:08.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:08.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:10.315 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:30:10.323 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:30:13.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:13.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:18.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:18.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:23.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:23.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8976 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:28.384 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:28.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 8997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:33.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:33.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:38.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:38.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:40.522 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:30:40.528 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:30:43.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:43.390+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9058 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:48.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:48.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:53.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:53.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:30:58.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:30:58.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:03.407 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:03.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:08.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:08.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:10.735 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:31:10.741 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:31:13.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:13.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:18.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:18.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:23.416 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:23.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:28.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:28.415+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:33.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:33.418+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:38.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:38.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:40.979 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:31:40.985 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:31:43.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:43.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:48.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:48.426+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:53.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:53.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:31:58.435 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:31:58.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9365 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:03.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:03.435+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:08.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:08.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:11.292 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:32:11.298 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:32:13.445 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:13.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:18.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:18.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:23.450 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:23.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:28.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:28.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9487 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:33.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:33.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:38.460 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:38.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:41.530 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:32:41.536 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:32:43.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:43.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:48.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:48.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9569 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:53.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:53.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:32:58.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:32:58.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:03.475 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:03.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:08.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:08.474+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:11.687 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:33:11.694 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:33:13.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:13.477+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9674 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:18.483 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:18.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:23.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:23.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9714 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:28.490 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:28.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:33.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:33.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:38.496 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:38.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:41.905 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:33:41.912 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:33:43.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:43.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:48.502 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:48.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:53.504 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:53.502+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9838 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:33:58.508 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:33:58.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9856 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:03.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:03.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:08.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:08.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9897 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:12.085 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:34:12.092 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:34:13.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:13.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9919 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:18.520 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:18.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:23.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:23.518+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:28.525 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:28.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 9979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:33.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:33.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:38.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:38.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:42.306 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:34:42.313 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:34:43.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:43.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:48.536 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:48.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:53.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:53.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:34:58.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:34:58.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:03.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:03.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10122 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:08.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:08.547+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:12.512 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:35:12.518 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:35:13.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:13.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:18.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:18.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:23.561 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:23.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:28.562 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:28.560+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:33.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:33.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10245 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:38.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:38.566+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10265 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:42.659 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:35:42.665 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:35:43.572 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:43.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:48.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:48.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:53.579 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:53.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:35:58.580 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:35:58.577+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10347 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:03.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:03.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:08.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:08.583+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:12.794 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:36:12.800 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:36:13.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:13.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:18.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:18.590+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:23.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:23.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:28.601 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:28.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10470 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:33.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:33.600+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10491 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:38.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:38.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:42.941 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:36:42.947 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:36:43.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:43.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:48.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:48.609+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:53.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:53.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:36:58.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:36:58.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:03.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:03.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10614 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:08.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:08.621+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:13.072 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:37:13.078 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:37:13.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:13.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:18.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:18.626+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10674 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:23.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:23.629+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:28.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:28.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:33.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:33.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10737 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:38.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:38.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:43.269 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:37:43.276 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:37:43.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:43.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:48.648 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:48.645+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10797 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:53.652 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:53.648+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:37:58.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:37:58.651+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:03.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:03.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:08.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:08.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:13.449 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:38:13.456 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:38:13.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:13.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10901 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:18.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:18.665+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10919 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:23.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:23.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:28.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:28.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10960 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:33.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:33.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 10983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:38.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:38.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:43.679 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:38:43.688 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:38:43.688 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:43.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:48.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:48.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11042 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:53.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:53.687+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11065 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:38:58.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:38:58.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11083 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:03.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:03.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11106 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:08.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:08.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11124 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:13.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:13.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:13.830 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:39:13.837 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:39:18.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:18.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11165 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:23.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:23.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:28.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:28.710+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11206 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:33.716 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:33.714+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:38.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:38.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11247 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:43.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:43.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11270 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:44.007 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:39:44.013 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:39:48.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:48.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11288 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:53.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:53.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:39:58.733 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:39:58.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11329 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:03.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:03.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:08.740 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:08.737+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11370 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:13.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:13.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11392 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:14.212 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:40:14.219 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:40:18.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:18.743+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11411 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:23.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:23.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11433 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:28.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:28.750+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11452 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:33.755 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:33.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:38.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:38.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11493 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:43.763 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:43.759+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:44.399 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:40:44.408 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:40:48.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:48.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:53.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:53.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:40:58.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:40:58.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11575 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:03.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:03.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:08.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:08.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:13.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:13.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:14.544 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:41:14.550 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:41:18.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:18.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11657 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:23.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:23.789+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:28.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:28.792+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:33.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:33.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:38.802 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:38.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:43.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:43.801+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:44.752 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:41:44.759 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:41:48.808 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:48.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:53.811 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:53.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:41:58.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:41:58.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:03.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:03.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:08.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:08.822+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:13.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:13.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:14.974 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:42:14.980 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:42:18.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:18.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:23.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:23.833+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:28.839 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:28.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:33.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:33.840+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11965 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:38.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:38.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 11985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:43.848 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:43.846+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:45.111 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:42:45.117 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:42:48.852 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:48.850+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:53.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:53.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12047 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:42:58.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:42:58.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12066 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:03.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:03.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:08.866 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:08.862+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:13.868 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:13.865+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:15.307 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:43:15.313 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:43:18.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:18.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12148 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:23.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:23.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:28.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:28.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12189 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:33.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:33.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:38.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:38.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12230 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:43.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:43.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:45.505 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:43:45.511 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:43:48.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:48.889+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12270 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:53.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:53.892+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:43:58.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:43:58.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12311 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:03.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:03.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:08.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:08.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12352 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:13.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:13.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:15.751 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:44:15.757 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:44:18.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:18.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12393 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:23.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:23.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12416 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:28.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:28.917+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:33.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:33.920+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:38.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:38.922+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:43.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:43.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:45.896 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:44:45.903 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:44:48.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:48.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:53.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:53.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12539 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:44:58.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:44:58.935+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:03.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:03.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:08.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:08.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:13.950 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:13.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:16.044 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:45:16.051 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:45:18.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:18.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12639 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:23.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:23.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12662 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:28.960 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:28.957+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12680 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:33.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:33.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12702 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:38.966 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:38.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:43.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:43.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12743 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:46.196 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:45:46.203 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:45:48.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:48.972+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:53.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:53.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:45:58.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:45:58.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:03.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:03.981+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12825 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:08.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:08.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:13.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:13.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12866 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:16.336 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:46:16.348 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:46:18.996 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:18.993+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:24.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:23.996+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12907 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:29.002 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:28.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12927 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:34.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:34.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12948 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:39.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:39.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12967 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:44.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:44.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 12988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:46.485 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:46:46.491 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:46:49.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:49.014+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:54.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:54.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13029 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:46:59.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:46:59.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13049 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:04.027 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:04.024+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:09.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:09.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:14.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:14.031+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13111 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:16.623 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:47:16.629 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:47:19.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:19.035+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13131 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:24.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:24.039+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:29.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:29.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:34.049 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:34.045+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:39.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:39.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:44.055 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:44.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:46.762 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:47:46.769 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:47:49.059 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:49.055+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:54.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:54.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:47:59.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:47:59.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13295 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:04.069 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:04.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:09.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:09.069+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:14.076 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:14.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:16.895 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:48:16.902 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:48:19.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:19.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:24.083 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:24.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:29.087 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:29.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:34.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:34.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13439 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:39.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:39.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:44.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:44.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13480 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:47.031 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:48:47.038 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:48:49.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:49.100+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:54.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:54.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:48:59.109 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:48:59.107+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:04.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:04.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:09.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:09.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:14.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:14.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:17.184 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:49:17.190 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:49:19.124 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:19.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:24.128 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:24.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13644 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:29.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:29.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:34.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:34.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:39.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:39.134+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:44.142 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:44.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:47.338 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:49:47.344 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:49:49.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:49.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:54.149 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:54.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13767 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:49:59.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:49:59.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:04.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:04.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13808 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:09.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:09.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:14.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:14.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13849 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:17.470 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:50:17.476 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:50:19.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:19.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:24.170 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:24.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13890 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:29.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:29.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:34.179 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:34.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13931 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:39.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:39.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:44.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:44.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13972 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:47.627 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:50:47.633 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:50:49.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:49.187+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 13991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:54.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:54.190+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14013 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:50:59.197 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:50:59.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14031 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:04.201 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:04.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:09.205 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:09.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14073 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:14.208 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:14.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:17.758 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:51:17.764 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:51:19.211 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:19.208+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14113 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:24.216 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:24.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:29.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:29.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:34.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:34.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:39.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:39.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:44.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:44.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:47.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:51:47.897 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:51:49.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:49.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:54.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:54.235+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:51:59.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:51:59.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:04.245 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:04.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:09.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:09.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:14.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:14.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:18.030 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:52:18.036 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:52:19.256 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:19.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:24.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:24.257+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:29.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:29.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14401 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:34.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:34.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:39.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:39.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:44.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:44.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14462 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:48.168 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:52:48.192 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:52:49.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:49.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:54.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:54.281+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:52:59.289 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:52:59.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14523 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:04.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:04.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:09.298 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:09.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14565 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:14.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:14.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14585 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:18.334 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:53:18.340 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:53:19.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:19.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14605 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:24.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:24.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:29.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:29.310+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14647 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:34.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:34.315+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14667 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:39.323 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:39.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14687 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:44.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:44.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:48.474 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:53:48.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:53:49.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:49.327+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:54.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:54.330+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:53:59.338 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:53:59.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14769 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:04.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:04.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:09.345 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:09.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:14.349 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:14.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:18.639 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:54:18.646 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:54:19.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:19.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:24.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:24.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:29.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:29.357+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:34.363 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:34.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14913 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:39.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:39.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:44.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:44.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:48.770 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:54:48.777 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:54:49.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:49.372+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:54.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:54.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 14995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:54:59.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:54:59.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15014 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:04.385 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:04.382+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:09.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:09.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15056 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:14.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:14.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:18.909 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:55:18.915 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:55:19.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:19.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15096 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:24.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:24.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:29.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:29.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15137 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:34.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:34.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:39.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:39.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:44.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:44.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:49.055 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:55:49.062 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:55:49.421 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:49.419+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:54.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:54.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15241 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:55:59.427 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:55:59.425+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:04.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:04.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:09.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:09.433+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:14.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:14.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:19.206 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:56:19.212 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:56:19.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:19.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:24.447 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:24.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:29.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:29.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:34.455 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:34.452+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:39.460 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:39.455+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15423 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:44.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:44.459+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:49.353 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:56:49.360 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:56:49.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:49.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:54.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:54.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15487 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:56:59.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:56:59.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:04.478 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:04.474+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:09.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:09.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:14.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:14.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15568 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:19.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:19.487+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:19.502 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:57:19.506 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:57:24.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:24.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:29.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:29.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:34.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:34.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:39.504 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:39.502+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:44.508 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:44.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:49.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:49.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:49.626 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:57:49.632 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:57:54.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:54.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:57:59.520 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:57:59.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:04.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:04.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:09.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:09.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:14.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:14.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15814 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:19.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:19.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:19.763 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:58:19.770 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:58:24.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:24.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:29.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:29.540+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:34.548 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:34.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:39.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:39.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:44.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:44.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:49.561 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:49.558+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15957 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:49.903 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:58:49.910 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:58:54.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:54.562+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15977 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:58:59.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:58:59.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 15997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:04.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:04.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:09.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:09.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:14.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:14.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:19.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:19.583+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:20.060 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:59:20.067 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:59:24.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:24.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:29.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:29.590+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:34.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:34.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:39.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:39.599+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:44.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:44.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:49.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:49.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:50.194 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:59:50.200 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T18:59:54.614 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:54.611+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T18:59:59.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T18:59:59.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:04.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:04.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:09.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:09.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:14.630 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:14.628+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:19.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:19.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:20.350 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:00:20.356 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:00:24.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:24.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:29.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:29.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:34.647 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:34.645+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:39.652 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:39.650+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:44.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:44.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:49.661 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:49.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:50.513 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:00:50.519 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:00:54.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:54.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:00:59.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:00:59.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:04.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:04.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:09.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:09.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:14.684 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:14.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16551 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:19.688 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:19.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:20.653 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:01:20.659 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:01:24.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:24.689+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16608 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:29.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:29.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16643 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:34.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:34.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:39.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:39.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16716 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:44.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:44.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:49.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:49.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:50.791 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:01:50.797 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:01:54.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:54.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:01:59.723 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:01:59.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:04.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:04.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16900 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:09.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:09.729+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:14.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:14.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 16973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:19.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:19.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17008 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:20.969 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:02:20.978 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:02:24.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:24.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17046 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:29.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:29.744+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:34.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:34.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:39.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:39.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:44.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:44.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17192 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:49.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:49.763+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17227 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:51.107 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:02:51.113 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:02:54.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:54.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17265 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:02:59.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:02:59.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:04.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:04.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:09.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:09.782+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17371 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:14.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:14.787+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17408 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:19.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:19.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17442 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:21.254 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:03:21.261 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:03:24.798 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:24.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:29.803 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:29.800+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:34.808 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:34.806+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:39.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:39.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:44.818 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:44.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:49.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:49.819+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:51.401 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:03:51.409 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:03:54.830 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:54.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:03:59.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:03:59.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:04.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:04.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:09.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:09.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17797 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:14.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:14.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:19.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:19.847+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:21.565 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:04:21.571 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:04:24.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:24.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17904 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:29.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:29.857+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:34.865 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:34.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 17975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:39.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:39.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:44.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:44.872+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18046 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:49.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:49.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:51.711 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:04:51.724 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:04:54.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:54.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:04:59.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:04:59.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:04.893 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:04.891+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18191 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:09.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:09.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18230 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:14.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:14.899+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:19.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:19.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:21.901 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:05:21.907 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:05:24.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:24.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:29.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:29.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:34.920 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:34.917+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:39.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:39.922+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:44.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:44.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:49.936 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:49.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:52.048 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:05:52.054 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:05:54.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:54.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:05:59.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:05:59.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:04.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:04.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:09.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:09.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18702 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:14.957 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:14.954+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:19.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:19.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:22.196 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:06:22.202 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:06:24.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:24.965+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:29.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:29.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:34.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:34.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:39.980 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:39.978+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:44.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:44.982+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 18975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:49.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:49.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:06:52.350 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:06:52.356 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:06:54.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:54.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:00.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:06:59.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19097 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:05.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:05.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:10.011 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:10.007+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:15.016 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:15.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:20.020 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:20.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19254 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:22.487 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:07:22.494 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:07:25.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:25.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:30.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:30.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19333 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:35.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:35.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19370 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:40.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:40.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19412 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:45.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:45.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:50.050 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:50.047+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19491 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:07:52.626 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:07:52.632 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:07:55.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:07:55.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:00.060 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:00.058+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:05.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:05.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19608 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:10.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:10.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19649 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:15.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:15.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:20.081 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:20.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:22.747 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:08:22.754 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:08:25.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:25.082+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:30.091 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:30.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:35.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:35.093+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:40.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:40.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:45.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:45.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:50.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:50.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 19965 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:08:52.887 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:08:52.894 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:08:55.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:08:55.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20002 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:00.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:00.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:05.130 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:05.128+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:10.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:10.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:15.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:15.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:20.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:20.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:23.050 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:09:23.056 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:09:25.151 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:25.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:30.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:30.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:35.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:35.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:40.167 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:40.165+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:45.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:45.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:50.178 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:50.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:09:53.217 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:09:53.224 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:09:55.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:09:55.180+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:00.189 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:00.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:05.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:05.190+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:10.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:10.196+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:15.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:15.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:20.210 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:20.207+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:23.348 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:10:23.354 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:10:25.216 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:25.212+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:30.220 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:30.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:35.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:35.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:40.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:40.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:45.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:45.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20869 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:50.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:50.239+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:10:53.497 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:10:53.503 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:10:55.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:10:55.245+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20947 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:00.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:00.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 20991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:05.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:05.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:10.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:10.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:15.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:15.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:20.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:20.271+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:23.627 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:11:23.634 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:11:25.280 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:25.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:30.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:30.281+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:35.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:35.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:40.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:40.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:45.301 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:45.297+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:50.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:50.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:11:53.766 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:11:53.772 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:11:55.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:11:55.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:00.316 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:00.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:05.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:05.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:10.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:10.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:15.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:15.330+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21579 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:20.338 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:20.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:23.914 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:12:23.921 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:12:25.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:25.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:30.348 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:30.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21701 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:35.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:35.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21737 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:40.360 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:40.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:45.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:45.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:50.368 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:50.366+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:12:54.052 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:12:54.058 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:12:55.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:12:55.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:00.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:00.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:05.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:05.383+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 21975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:10.392 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:10.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:15.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:15.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:20.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:20.399+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22096 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:24.194 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:13:24.202 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:13:25.407 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:25.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:30.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:30.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22174 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:35.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:35.415+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:40.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:40.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:45.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:45.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:50.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:50.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:13:54.350 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:13:54.357 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:13:55.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:13:55.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:00.445 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:00.443+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22411 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:05.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:05.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:10.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:10.453+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:15.461 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:15.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22527 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:20.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:20.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22569 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:24.507 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:14:24.513 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:14:25.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:25.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22607 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:30.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:30.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:35.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:35.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:40.488 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:40.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22727 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:45.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:45.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22765 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:50.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:50.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:14:54.654 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:14:54.660 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:14:55.503 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:14:55.500+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22843 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:00.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:00.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:05.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:05.512+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:10.520 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:10.518+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 22964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:15.526 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:15.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:20.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:20.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:24.784 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:15:24.790 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:15:25.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:25.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:30.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:30.541+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:35.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:35.547+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:40.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:40.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:45.561 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:45.558+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:50.566 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:50.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:15:55.034 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:15:55.040 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:15:55.571 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:15:55.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:00.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:00.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:05.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:05.578+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:10.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:10.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:15.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:15.591+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23485 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:20.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:20.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:25.182 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:16:25.188 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:16:25.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:25.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:30.612 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:30.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:35.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:35.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23646 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:40.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:40.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23690 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:45.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:45.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:50.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:50.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:16:55.324 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:16:55.331 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:16:55.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:16:55.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23808 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:00.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:00.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23852 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:05.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:05.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23890 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:10.654 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:10.651+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23933 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:15.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:15.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 23970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:20.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:20.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24013 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:25.478 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:17:25.485 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:17:25.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:25.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:30.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:30.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:35.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:35.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:40.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:40.683+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:45.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:45.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24206 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:50.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:50.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:17:55.618 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:17:55.624 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:17:55.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:17:55.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:00.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:00.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:05.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:05.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:10.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:10.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:15.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:15.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:20.732 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:20.730+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:25.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:25.737+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:25.755 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:18:25.760 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:18:30.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:30.744+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24537 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:35.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:35.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24575 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:40.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:40.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:45.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:45.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:50.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:50.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:55.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:18:55.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24724 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:18:55.893 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:18:55.899 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:19:00.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:00.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:05.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:05.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:10.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:10.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:15.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:15.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:20.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:20.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24907 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:25.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:25.809+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:26.073 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:19:26.079 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:19:30.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:30.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 24981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:35.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:35.821+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:40.830 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:40.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25055 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:45.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:45.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25094 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:50.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:50.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:55.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:19:55.847+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:19:56.264 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:19:56.271 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:20:00.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:00.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25202 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:05.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:05.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:10.867 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:10.864+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:15.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:15.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:20.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:20.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:25.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:25.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:26.435 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:20:26.442 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:20:30.891 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:30.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25430 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:35.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:35.894+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:40.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:40.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:45.910 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:45.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25548 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:50.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:50.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:55.921 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:20:55.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25624 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:20:56.657 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:20:56.664 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:21:00.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:00.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:05.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:05.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25700 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:10.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:10.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:15.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:15.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:20.951 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:20.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25809 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:25.957 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:25.955+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:26.819 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:21:26.826 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:21:30.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:30.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25884 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:35.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:35.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:40.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:40.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:45.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:45.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 25999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:50.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:50.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26034 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:55.995 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:21:55.992+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:21:57.034 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:21:57.042 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:22:01.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:00.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:06.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:06.004+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:11.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:11.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:16.019 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:16.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:21.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:21.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:26.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:26.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:27.265 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:22:27.272 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:22:31.038 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:31.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:36.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:36.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:41.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:41.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:46.058 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:46.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:51.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:51.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:56.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:22:56.067+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26523 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:22:57.712 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:22:57.718 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:23:01.076 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:01.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26559 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:06.082 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:06.079+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:11.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:11.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:16.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:16.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26673 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:21.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:21.098+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:26.107 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:26.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:27.941 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:23:27.947 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:23:31.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:31.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:36.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:36.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:41.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:41.122+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:46.132 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:46.129+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:51.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:51.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26934 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:56.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:23:56.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 26973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:23:58.079 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:23:58.085 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:24:01.150 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:01.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:06.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:06.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27048 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:11.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:11.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:16.170 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:16.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:21.175 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:21.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27160 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:26.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:26.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:28.250 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:24:28.256 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:24:31.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:31.185+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:36.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:36.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:41.201 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:41.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:46.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:46.203+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:51.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:51.209+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27384 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:56.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:24:56.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27423 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:24:58.472 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:24:58.478 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:25:01.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:01.221+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:06.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:06.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:11.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:11.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:16.243 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:16.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:21.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:21.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:26.255 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:26.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:28.695 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:25:28.701 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:25:31.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:31.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27683 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:36.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:36.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:41.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:41.275+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:46.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:46.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:51.291 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:51.289+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:56.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:25:56.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27873 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:25:58.824 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:25:58.831 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:26:01.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:01.301+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:06.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:06.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27948 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:11.317 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:11.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 27982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:16.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:16.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:21.330 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:21.327+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:26.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:26.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:29.037 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:26:29.044 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:26:31.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:31.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:36.350 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:36.346+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:41.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:41.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:46.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:46.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:51.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:51.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:56.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:26:56.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28315 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:26:59.226 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:26:59.232 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:27:01.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:01.383+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:06.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:06.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:11.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:11.396+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:16.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:16.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:21.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:21.409+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:26.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:26.415+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:29.439 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:27:29.445 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:27:31.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:31.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:36.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:36.428+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28607 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:41.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:41.434+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28641 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:46.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:46.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28680 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:51.450 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:51.447+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28713 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:56.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:27:56.453+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28753 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:27:59.819 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:27:59.826 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:28:01.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:01.459+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28787 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:06.469 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:06.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28825 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:11.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:11.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:16.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:16.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:21.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:21.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28933 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:26.496 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:26.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 28971 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:30.051 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:28:30.057 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:28:31.503 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:31.500+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29005 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:36.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:36.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:41.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:41.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:46.523 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:46.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:51.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:51.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:28:56.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:28:56.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29190 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:00.196 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:29:00.202 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:29:01.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:01.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:06.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:06.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:11.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:11.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:16.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:16.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:21.570 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:21.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29371 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:26.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:26.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29408 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:30.367 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:29:30.374 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:29:31.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:31.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:36.591 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:36.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29481 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:41.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:41.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:46.604 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:46.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:51.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:51.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:29:56.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:29:56.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:00.574 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:30:00.580 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:30:01.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:01.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:06.631 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:06.629+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:11.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:11.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:16.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:16.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:21.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:21.651+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:26.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:26.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29843 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:30.753 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:30:30.760 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:30:31.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:31.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:36.672 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:36.669+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:41.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:41.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29951 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:46.685 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:46.682+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 29987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:51.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:51.689+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:30:56.699 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:30:56.696+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:00.976 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:31:00.982 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:31:01.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:01.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:06.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:06.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30131 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:11.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:11.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:16.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:16.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:21.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:21.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30239 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:26.742 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:26.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:31.182 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:31:31.188 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:31:31.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:31.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:36.756 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:36.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:41.764 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:41.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30394 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:46.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:46.769+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:51.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:51.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:31:56.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:31:56.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:01.403 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:32:01.409 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:32:01.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:01.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:06.798 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:06.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:11.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:11.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30630 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:16.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:16.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:21.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:21.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:26.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:26.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:31.566 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:32:31.573 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:32:31.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:31.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:36.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:36.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:41.847 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:41.844+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:46.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:46.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:51.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:51.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:32:56.868 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:32:56.866+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 30991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:01.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:01.873+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:01.879 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:33:01.883 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:33:06.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:06.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:11.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:11.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:16.896 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:16.894+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:21.903 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:21.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:26.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:26.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:31.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:31.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:32.002 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:33:32.008 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:33:36.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:36.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:41.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:41.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:46.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:46.938+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:51.950 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:51.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:33:56.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:33:56.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:01.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:01.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:02.144 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:34:02.149 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:34:06.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:06.967+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:11.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:11.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:16.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:16.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:21.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:21.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:27.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:26.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:32.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:32.005+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:32.346 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:34:32.352 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:34:37.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:37.011+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31783 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:42.019 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:42.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31818 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:47.027 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:47.025+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:52.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:52.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:34:57.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:34:57.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:02.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:02.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 31978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:02.558 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:35:02.565 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:35:07.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:07.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:12.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:12.067+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32058 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:17.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:17.074+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:22.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:22.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32138 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:27.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:27.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:32.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:32.100+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:32.813 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:35:32.820 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:35:37.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:37.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:42.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:42.118+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:47.130 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:47.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32342 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:52.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:52.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:35:57.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:35:57.144+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:02.154 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:02.152+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:02.951 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:36:02.959 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:36:07.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:07.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:12.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:12.169+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:17.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:17.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:22.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:22.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:27.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:27.195+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:32.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:32.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:33.165 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:36:33.172 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:36:37.216 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:37.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32742 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:42.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:42.221+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:47.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:47.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:52.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:52.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32858 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:36:57.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:36:57.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:02.259 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:02.256+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:03.353 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:37:03.360 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:37:07.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:07.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 32981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:12.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:12.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:17.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:17.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:22.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:22.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:27.303 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:27.301+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:32.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:32.310+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:33.523 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:37:33.530 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:37:37.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:37.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:42.330 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:42.328+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:47.338 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:47.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:52.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:52.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:37:57.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:37:57.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:02.363 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:02.360+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:03.704 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:38:03.711 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:38:07.373 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:07.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:12.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:12.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:17.390 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:17.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:22.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:22.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:27.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:27.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:32.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:32.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:33.837 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:38:33.843 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:38:37.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:37.423+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:42.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:42.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33741 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:47.442 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:47.439+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:52.450 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:52.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:38:57.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:38:57.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:02.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:02.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:03.972 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:39:03.979 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:39:07.475 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:07.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:12.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:12.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 33987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:17.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:17.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:22.503 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:22.500+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:27.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:27.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34113 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:32.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:32.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:34.245 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:39:34.252 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:39:37.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:37.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:42.539 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:42.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34231 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:47.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:47.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34272 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:52.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:52.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34309 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:39:57.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:39:57.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:02.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:02.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:04.387 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:40:04.394 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:40:07.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:07.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:12.591 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:12.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:17.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:17.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34507 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:22.608 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:22.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34543 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:27.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:27.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:32.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:32.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:34.690 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:40:34.697 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:40:37.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:37.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:42.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:42.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:47.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:47.650+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:52.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:52.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:40:57.671 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:40:57.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:02.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:02.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:04.830 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:41:04.836 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:41:07.689 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:07.686+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:12.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:12.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34933 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:17.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:17.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 34975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:22.716 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:22.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35012 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:27.725 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:27.722+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:32.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:32.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:34.977 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:41:34.984 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:41:37.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:37.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:42.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:42.750+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:47.761 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:47.759+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35210 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:52.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:52.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35249 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:41:57.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:41:57.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35290 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:02.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:02.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35328 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:05.108 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:42:05.115 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:42:07.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:07.794+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:12.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:12.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:17.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:17.813+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:22.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:22.822+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:27.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:27.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35524 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:32.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:32.840+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:35.247 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:42:35.254 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:42:37.852 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:37.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:42.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:42.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:47.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:47.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:52.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:52.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:42:57.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:42:57.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:02.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:02.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35797 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:05.438 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:43:05.444 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:43:07.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:07.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:12.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:12.915+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:17.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:17.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:22.934 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:22.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35953 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:27.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:27.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 35992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:32.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:32.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36031 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:35.593 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:43:35.599 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:43:37.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:37.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:42.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:42.967+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:47.979 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:47.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36148 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:52.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:52.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:43:57.996 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:43:57.993+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36226 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:03.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:03.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36265 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:05.733 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:44:05.740 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:44:08.019 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:08.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:13.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:13.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:18.033 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:18.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:23.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:23.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:28.054 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:28.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:33.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:33.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:35.880 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:44:35.887 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:44:38.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:38.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:43.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:43.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:48.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:48.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:53.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:53.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:44:58.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:44:58.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:03.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:03.114+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:06.029 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:45:06.035 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:45:08.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:08.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:13.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:13.132+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:18.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:18.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:23.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:23.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36888 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:28.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:28.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:33.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:33.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 36966 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:36.191 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:45:36.197 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:45:38.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:38.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:43.191 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:43.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:48.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:48.199+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37083 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:53.211 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:53.208+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:45:58.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:45:58.218+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37160 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:03.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:03.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:06.342 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:46:06.349 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:46:08.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:08.237+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:13.248 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:13.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:18.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:18.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:23.266 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:23.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:28.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:28.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:33.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:33.282+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:36.522 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:46:36.530 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:46:38.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:38.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:43.303 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:43.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:48.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:48.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:53.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:53.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:46:58.329 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:46:58.326+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:03.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:03.336+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:06.678 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:47:06.684 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:47:08.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:08.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:13.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:13.353+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:18.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:18.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:23.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:23.372+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:28.384 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:28.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37853 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:33.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:33.392+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37890 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:36.915 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:47:36.921 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:47:38.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:38.401+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37930 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:43.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:43.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 37966 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:48.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:48.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:53.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:53.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:47:58.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:47:58.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:03.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:03.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:07.137 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:48:07.144 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:48:08.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:08.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:13.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:13.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:18.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:18.477+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:23.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:23.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:28.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:28.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:33.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:33.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38350 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:37.300 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:48:37.307 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:48:38.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:38.515+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:43.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:43.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:48.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:48.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:53.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:53.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:48:58.556 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:48:58.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:03.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:03.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:07.497 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:49:07.503 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:49:08.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:08.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:13.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:13.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:18.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:18.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:23.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:23.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:28.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:28.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:33.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:33.626+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:37.677 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:49:37.683 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:49:38.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:38.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38858 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:43.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:43.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:48.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:48.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:53.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:53.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 38979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:49:58.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:49:58.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:03.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:03.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39058 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:07.907 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:50:07.914 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:50:08.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:08.689+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:13.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:13.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:18.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:18.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:23.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:23.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:28.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:28.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:33.740 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:33.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:38.114 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:50:38.120 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:50:38.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:38.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:43.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:43.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:48.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:48.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:53.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:53.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:50:58.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:50:58.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:03.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:03.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:08.335 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:51:08.341 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:51:08.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:08.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:13.818 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:13.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39619 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:18.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:18.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:23.839 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:23.836+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:28.848 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:28.845+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:33.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:33.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:38.540 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:51:38.546 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:51:38.868 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:38.865+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39820 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:43.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:43.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:48.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:48.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39900 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:53.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:53.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:51:58.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:51:58.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 39981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:03.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:03.915+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:08.754 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:52:08.760 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:52:08.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:08.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:13.937 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:13.935+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:18.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:18.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:23.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:23.954+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:28.966 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:28.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:33.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:33.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:38.966 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:52:38.972 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:52:38.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:38.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:43.995 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:43.992+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:49.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:49.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:54.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:54.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:52:59.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:52:59.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:04.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:04.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:09.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:09.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40548 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:09.106 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:53:09.112 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:53:14.055 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:14.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:19.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:19.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40628 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:24.076 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:24.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:29.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:29.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:34.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:34.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:39.105 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:39.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:39.286 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:53:39.292 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:53:44.117 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:44.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:49.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:49.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40870 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:54.139 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:54.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:53:59.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:53:59.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40950 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:04.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:04.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 40990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:09.170 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:09.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:09.491 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:54:09.498 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:54:14.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:14.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:19.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:19.187+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41110 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:24.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:24.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:29.210 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:29.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41190 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:34.220 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:34.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:39.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:39.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41271 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:39.731 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:54:39.737 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:54:44.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:44.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:49.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:49.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41353 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:54.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:54.256+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:54:59.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:54:59.267+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41433 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:04.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:04.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:09.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:09.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:09.859 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:55:09.865 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:55:14.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:14.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:19.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:19.304+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:24.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:24.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:29.326 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:29.324+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41677 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:34.336 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:34.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:39.348 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:39.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41757 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:40.082 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:55:40.089 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:55:44.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:44.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:49.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:49.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:54.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:54.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:55:59.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:55:59.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41913 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:04.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:04.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:09.407 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:09.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 41992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:10.228 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:56:10.234 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:56:14.416 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:14.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:19.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:19.423+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:24.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:24.433+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:29.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:29.443+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:34.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:34.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:39.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:39.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:40.460 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:56:40.467 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:56:44.478 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:44.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42261 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:49.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:49.485+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42303 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:54.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:54.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:56:59.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:56:59.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:04.519 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:04.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:09.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:09.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:10.640 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:57:10.646 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:57:14.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:14.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:19.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:19.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42539 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:24.563 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:24.560+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:29.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:29.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42619 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:34.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:34.583+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:39.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:39.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42697 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:40.777 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:57:40.788 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:57:44.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:44.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:49.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:49.614+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:54.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:54.626+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:57:59.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:57:59.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:04.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:04.648+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:09.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:09.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:10.999 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:58:11.005 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:58:14.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:14.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 42971 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:19.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:19.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43013 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:24.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:24.691+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43050 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:29.703 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:29.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:34.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:34.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:39.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:39.723+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43171 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:41.187 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:58:41.194 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:58:44.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:44.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43208 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:49.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:49.743+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:52.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:58:52.212+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:52.213954+0000) 2020-02-05T19:58:53.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:58:53.338+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:53.339358+0000) 2020-02-05T19:58:54.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:58:54.751+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.752602+0000) 2020-02-05T19:58:54.757 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:58:54.751+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.753319+0000) 2020-02-05T19:58:54.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:58:54.752+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.753959+0000) 2020-02-05T19:58:54.758 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:58:54.752+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.754351+0000) 2020-02-05T19:58:54.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:58:54.753+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.754751+0000) 2020-02-05T19:58:54.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:58:54.753+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.755153+0000) 2020-02-05T19:58:54.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:54.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43287 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:54.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:58:54.749+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.750517+0000) 2020-02-05T19:58:54.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:58:54.749+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.751015+0000) 2020-02-05T19:58:54.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:58:54.750+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.751947+0000) 2020-02-05T19:58:54.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:58:54.751+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.752560+0000) 2020-02-05T19:58:54.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:54.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.753271+0000) 2020-02-05T19:58:54.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:54.752+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.753703+0000) 2020-02-05T19:58:54.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:58:54.753+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.754315+0000) 2020-02-05T19:58:54.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:54.759+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:54.760596+0000) 2020-02-05T19:58:55.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:55.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:55.756779+0000) 2020-02-05T19:58:55.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:55.758+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:55.759712+0000) 2020-02-05T19:58:56.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:58:56.614+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:56.615571+0000) 2020-02-05T19:58:56.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:58:56.623+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:56.624130+0000) 2020-02-05T19:58:56.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:58:56.623+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:56.624352+0000) 2020-02-05T19:58:58.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:58:58.594+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:58.595761+0000) 2020-02-05T19:58:58.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:58:58.966+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:58.967822+0000) 2020-02-05T19:58:59.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:59.253+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.254359+0000) 2020-02-05T19:58:59.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:58:59.759+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.761104+0000) 2020-02-05T19:58:59.764 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:58:59.761+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.763012+0000) 2020-02-05T19:58:59.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:58:59.762+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.763949+0000) 2020-02-05T19:58:59.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:58:59.763+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.764598+0000) 2020-02-05T19:58:59.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:58:59.763+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.764903+0000) 2020-02-05T19:58:59.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:58:59.764+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.765374+0000) 2020-02-05T19:58:59.767 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:58:59.763+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43329 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:58:59.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:59.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.765808+0000) 2020-02-05T19:58:59.769 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:58:59.765+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.766777+0000) 2020-02-05T19:58:59.769 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:58:59.766+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.767797+0000) 2020-02-05T19:58:59.769 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:58:59.765+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.766842+0000) 2020-02-05T19:58:59.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:59.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.767583+0000) 2020-02-05T19:58:59.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:58:59.768+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.769228+0000) 2020-02-05T19:58:59.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:58:59.767+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.769026+0000) 2020-02-05T19:58:59.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:58:59.769+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.770284+0000) 2020-02-05T19:58:59.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:58:59.863+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:58:59.864345+0000) 2020-02-05T19:59:00.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:00.103+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:00.104540+0000) 2020-02-05T19:59:00.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:00.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:00.104627+0000) 2020-02-05T19:59:00.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:00.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:00.104877+0000) 2020-02-05T19:59:02.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:02.213+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:02.214195+0000) 2020-02-05T19:59:03.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:03.338+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:03.339671+0000) 2020-02-05T19:59:04.771 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:04.769+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.770430+0000) 2020-02-05T19:59:04.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:04.770+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.771497+0000) 2020-02-05T19:59:04.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:04.772+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.773786+0000) 2020-02-05T19:59:04.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:04.773+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.774223+0000) 2020-02-05T19:59:04.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:04.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.775250+0000) 2020-02-05T19:59:04.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:04.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:04.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:04.773+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.775148+0000) 2020-02-05T19:59:04.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:04.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.775878+0000) 2020-02-05T19:59:04.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:04.775+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.776565+0000) 2020-02-05T19:59:04.778 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:04.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.775603+0000) 2020-02-05T19:59:04.778 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:04.774+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.775933+0000) 2020-02-05T19:59:04.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:04.775+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.776753+0000) 2020-02-05T19:59:04.779 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:04.775+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.777059+0000) 2020-02-05T19:59:04.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:04.776+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.777318+0000) 2020-02-05T19:59:04.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:04.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:04.779846+0000) 2020-02-05T19:59:05.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:05.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:05.757011+0000) 2020-02-05T19:59:05.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:05.766+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:05.767809+0000) 2020-02-05T19:59:06.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:06.614+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:06.615824+0000) 2020-02-05T19:59:06.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:06.623+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:06.624356+0000) 2020-02-05T19:59:06.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:06.623+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:06.624598+0000) 2020-02-05T19:59:08.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:08.595+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:08.596022+0000) 2020-02-05T19:59:08.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:08.966+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:08.968050+0000) 2020-02-05T19:59:09.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:09.253+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.254593+0000) 2020-02-05T19:59:09.782 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:09.780+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.781116+0000) 2020-02-05T19:59:09.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:09.780+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.781463+0000) 2020-02-05T19:59:09.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:09.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.783348+0000) 2020-02-05T19:59:09.785 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:09.782+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.783864+0000) 2020-02-05T19:59:09.785 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:09.783+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.784563+0000) 2020-02-05T19:59:09.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:09.783+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.784852+0000) 2020-02-05T19:59:09.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:09.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:09.787 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:09.783+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.785355+0000) 2020-02-05T19:59:09.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:09.783+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.785108+0000) 2020-02-05T19:59:09.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:09.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.785888+0000) 2020-02-05T19:59:09.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:09.785+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.786314+0000) 2020-02-05T19:59:09.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:09.786+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.787420+0000) 2020-02-05T19:59:09.789 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:09.786+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.788006+0000) 2020-02-05T19:59:09.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:09.786+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.787874+0000) 2020-02-05T19:59:09.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:09.794+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.795629+0000) 2020-02-05T19:59:09.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:09.863+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:09.864569+0000) 2020-02-05T19:59:10.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:10.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:10.104818+0000) 2020-02-05T19:59:10.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:10.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:10.104969+0000) 2020-02-05T19:59:10.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:10.104+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:10.105082+0000) 2020-02-05T19:59:11.400 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:59:11.406 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:59:12.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:12.213+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:12.214468+0000) 2020-02-05T19:59:13.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:13.339+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:13.339924+0000) 2020-02-05T19:59:14.798 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:14.790+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.792050+0000) 2020-02-05T19:59:14.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:14.791+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.793027+0000) 2020-02-05T19:59:14.798 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:14.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43445 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:14.799 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:14.794+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.795666+0000) 2020-02-05T19:59:14.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:14.794+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.795934+0000) 2020-02-05T19:59:14.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:14.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.790746+0000) 2020-02-05T19:59:14.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:14.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.794357+0000) 2020-02-05T19:59:14.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:14.794+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.795114+0000) 2020-02-05T19:59:14.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:14.794+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.795490+0000) 2020-02-05T19:59:14.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:14.794+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.795851+0000) 2020-02-05T19:59:14.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:14.795+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.796246+0000) 2020-02-05T19:59:14.801 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:14.795+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.796624+0000) 2020-02-05T19:59:14.801 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:14.795+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.796638+0000) 2020-02-05T19:59:14.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:14.799+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.800451+0000) 2020-02-05T19:59:14.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:14.799+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:14.800721+0000) 2020-02-05T19:59:15.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:15.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:15.757217+0000) 2020-02-05T19:59:15.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:15.775+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:15.775872+0000) 2020-02-05T19:59:16.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:16.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:16.616061+0000) 2020-02-05T19:59:16.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:16.624+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:16.624664+0000) 2020-02-05T19:59:16.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:16.624+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:16.624849+0000) 2020-02-05T19:59:18.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:18.595+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:18.596317+0000) 2020-02-05T19:59:18.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:18.967+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:18.968298+0000) 2020-02-05T19:59:19.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:19.254+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.254818+0000) 2020-02-05T19:59:19.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:19.799+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.800030+0000) 2020-02-05T19:59:19.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:19.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.801625+0000) 2020-02-05T19:59:19.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:19.801+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.802180+0000) 2020-02-05T19:59:19.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:19.801+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.802583+0000) 2020-02-05T19:59:19.804 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:19.802+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.803337+0000) 2020-02-05T19:59:19.804 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:19.801+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.803163+0000) 2020-02-05T19:59:19.805 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:19.802+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.804250+0000) 2020-02-05T19:59:19.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:19.803+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.804635+0000) 2020-02-05T19:59:19.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:19.804+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.805205+0000) 2020-02-05T19:59:19.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:19.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43487 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:19.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:19.804+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.805588+0000) 2020-02-05T19:59:19.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:19.804+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.806049+0000) 2020-02-05T19:59:19.808 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:19.805+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.807134+0000) 2020-02-05T19:59:19.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:19.808+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.809352+0000) 2020-02-05T19:59:19.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:19.809+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.811035+0000) 2020-02-05T19:59:19.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:19.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:19.864766+0000) 2020-02-05T19:59:20.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:20.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:20.105048+0000) 2020-02-05T19:59:20.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:20.104+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:20.105306+0000) 2020-02-05T19:59:20.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:20.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:20.105213+0000) 2020-02-05T19:59:22.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:22.214+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:22.214741+0000) 2020-02-05T19:59:23.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:23.339+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:23.340178+0000) 2020-02-05T19:59:24.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:24.810+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.811505+0000) 2020-02-05T19:59:24.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:24.811+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.812283+0000) 2020-02-05T19:59:24.814 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:24.812+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.812864+0000) 2020-02-05T19:59:24.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:24.813+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.814060+0000) 2020-02-05T19:59:24.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:24.813+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.814586+0000) 2020-02-05T19:59:24.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:24.815+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.815616+0000) 2020-02-05T19:59:24.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:24.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43524 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:24.817 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:24.815+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.816182+0000) 2020-02-05T19:59:24.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:24.815+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.816625+0000) 2020-02-05T19:59:24.818 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:24.816+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.817215+0000) 2020-02-05T19:59:24.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:24.817+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.817510+0000) 2020-02-05T19:59:24.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:24.816+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.818022+0000) 2020-02-05T19:59:24.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:24.818+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.818719+0000) 2020-02-05T19:59:24.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:24.822+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.823314+0000) 2020-02-05T19:59:24.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:24.822+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:24.823590+0000) 2020-02-05T19:59:25.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:25.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:25.757435+0000) 2020-02-05T19:59:25.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:25.783+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:25.783906+0000) 2020-02-05T19:59:26.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:26.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:26.616331+0000) 2020-02-05T19:59:26.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:26.624+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:26.624858+0000) 2020-02-05T19:59:26.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:26.624+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:26.625060+0000) 2020-02-05T19:59:28.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:28.596+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:28.596563+0000) 2020-02-05T19:59:28.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:28.968+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:28.968531+0000) 2020-02-05T19:59:29.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:29.254+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.255049+0000) 2020-02-05T19:59:29.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:29.820+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.821556+0000) 2020-02-05T19:59:29.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:29.821+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.823055+0000) 2020-02-05T19:59:29.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:29.822+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.823530+0000) 2020-02-05T19:59:29.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:29.823+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.824023+0000) 2020-02-05T19:59:29.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:29.823+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.824380+0000) 2020-02-05T19:59:29.826 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:29.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.824878+0000) 2020-02-05T19:59:29.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:29.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:29.827 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:29.825+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.825844+0000) 2020-02-05T19:59:29.827 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:29.825+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.826243+0000) 2020-02-05T19:59:29.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:29.825+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.826985+0000) 2020-02-05T19:59:29.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:29.826+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.827465+0000) 2020-02-05T19:59:29.829 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:29.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.827818+0000) 2020-02-05T19:59:29.829 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:29.827+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.828530+0000) 2020-02-05T19:59:29.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:29.826+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.827799+0000) 2020-02-05T19:59:29.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:29.831+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.832660+0000) 2020-02-05T19:59:29.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:29.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:29.864978+0000) 2020-02-05T19:59:30.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:30.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:30.105280+0000) 2020-02-05T19:59:30.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:30.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:30.105512+0000) 2020-02-05T19:59:30.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:30.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:30.105508+0000) 2020-02-05T19:59:32.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:32.214+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:32.214998+0000) 2020-02-05T19:59:33.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:33.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:33.340415+0000) 2020-02-05T19:59:34.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:34.830+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.830675+0000) 2020-02-05T19:59:34.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:34.830+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.831298+0000) 2020-02-05T19:59:34.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:34.831+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.832417+0000) 2020-02-05T19:59:34.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:34.831+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.832949+0000) 2020-02-05T19:59:34.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:34.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.833420+0000) 2020-02-05T19:59:34.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:34.833+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.834022+0000) 2020-02-05T19:59:34.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:34.833+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.834426+0000) 2020-02-05T19:59:34.836 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:34.834+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.834899+0000) 2020-02-05T19:59:34.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:34.834+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.835345+0000) 2020-02-05T19:59:34.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:34.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:34.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:34.835+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.836645+0000) 2020-02-05T19:59:34.838 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:34.836+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.836949+0000) 2020-02-05T19:59:34.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:34.836+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.837213+0000) 2020-02-05T19:59:34.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:34.839+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.840926+0000) 2020-02-05T19:59:34.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:34.840+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:34.841238+0000) 2020-02-05T19:59:35.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:35.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:35.757645+0000) 2020-02-05T19:59:35.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:35.795+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:35.795609+0000) 2020-02-05T19:59:36.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:36.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:36.616561+0000) 2020-02-05T19:59:36.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:36.624+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:36.625119+0000) 2020-02-05T19:59:36.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:36.624+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:36.625301+0000) 2020-02-05T19:59:38.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:38.596+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:38.596833+0000) 2020-02-05T19:59:38.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:38.968+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:38.968754+0000) 2020-02-05T19:59:39.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:39.255+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.255312+0000) 2020-02-05T19:59:39.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:39.840+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.841430+0000) 2020-02-05T19:59:39.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:39.841+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.842040+0000) 2020-02-05T19:59:39.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:39.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.842655+0000) 2020-02-05T19:59:39.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:39.842+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.843117+0000) 2020-02-05T19:59:39.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:39.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.844429+0000) 2020-02-05T19:59:39.846 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:39.844+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.845255+0000) 2020-02-05T19:59:39.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:39.844+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:39.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:39.844+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.845533+0000) 2020-02-05T19:59:39.848 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:39.845+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.846292+0000) 2020-02-05T19:59:39.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:39.846+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.846908+0000) 2020-02-05T19:59:39.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:39.847+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.847460+0000) 2020-02-05T19:59:39.850 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:39.847+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.847704+0000) 2020-02-05T19:59:39.850 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:39.847+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.848372+0000) 2020-02-05T19:59:39.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:39.845+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.846586+0000) 2020-02-05T19:59:39.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:39.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.851722+0000) 2020-02-05T19:59:39.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:39.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:39.865151+0000) 2020-02-05T19:59:40.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:40.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:40.105495+0000) 2020-02-05T19:59:40.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:40.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:40.105715+0000) 2020-02-05T19:59:40.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:40.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:40.105719+0000) 2020-02-05T19:59:41.588 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:59:41.594 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T19:59:42.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:42.214+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:42.215228+0000) 2020-02-05T19:59:43.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:43.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:43.340699+0000) 2020-02-05T19:59:44.853 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:44.851+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.851907+0000) 2020-02-05T19:59:44.853 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:44.852+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.852266+0000) 2020-02-05T19:59:44.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:44.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.852684+0000) 2020-02-05T19:59:44.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:44.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.852918+0000) 2020-02-05T19:59:44.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:44.852+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.853011+0000) 2020-02-05T19:59:44.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:44.853+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.853571+0000) 2020-02-05T19:59:44.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:44.853+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.853605+0000) 2020-02-05T19:59:44.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:44.853+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.854156+0000) 2020-02-05T19:59:44.856 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:44.854+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:44.856 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:44.855+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.855414+0000) 2020-02-05T19:59:44.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:44.855+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.855575+0000) 2020-02-05T19:59:44.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:44.855+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.856063+0000) 2020-02-05T19:59:44.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:44.855+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.856083+0000) 2020-02-05T19:59:44.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:44.855+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.856374+0000) 2020-02-05T19:59:44.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:44.863+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:44.863601+0000) 2020-02-05T19:59:45.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:45.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:45.757873+0000) 2020-02-05T19:59:45.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:45.803+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:45.803837+0000) 2020-02-05T19:59:46.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:46.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:46.616791+0000) 2020-02-05T19:59:46.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:46.625+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:46.625363+0000) 2020-02-05T19:59:46.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:46.625+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:46.625545+0000) 2020-02-05T19:59:48.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:48.596+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:48.597100+0000) 2020-02-05T19:59:48.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:48.967+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:48.968989+0000) 2020-02-05T19:59:49.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:49.254+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.255545+0000) 2020-02-05T19:59:49.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:49.860+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.861458+0000) 2020-02-05T19:59:49.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:49.862+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.863010+0000) 2020-02-05T19:59:49.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:49.863+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.863863+0000) 2020-02-05T19:59:49.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:49.863+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.864515+0000) 2020-02-05T19:59:49.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:49.864+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.865042+0000) 2020-02-05T19:59:49.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:49.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.865297+0000) 2020-02-05T19:59:49.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:49.864+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.865279+0000) 2020-02-05T19:59:49.867 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:49.865+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.865660+0000) 2020-02-05T19:59:49.867 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:49.865+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43725 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:49.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:49.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.866229+0000) 2020-02-05T19:59:49.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:49.866+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.867108+0000) 2020-02-05T19:59:49.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:49.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.867423+0000) 2020-02-05T19:59:49.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:49.866+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.867500+0000) 2020-02-05T19:59:49.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:49.866+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.867690+0000) 2020-02-05T19:59:49.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:49.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.867727+0000) 2020-02-05T19:59:49.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:49.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:49.868497+0000) 2020-02-05T19:59:50.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:50.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:50.105723+0000) 2020-02-05T19:59:50.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:50.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:50.105902+0000) 2020-02-05T19:59:50.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:50.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:50.105919+0000) 2020-02-05T19:59:52.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:52.215+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:52.215481+0000) 2020-02-05T19:59:53.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:53.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:53.340957+0000) 2020-02-05T19:59:54.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:54.870+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.871756+0000) 2020-02-05T19:59:54.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:54.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.872443+0000) 2020-02-05T19:59:54.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:54.872+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.872775+0000) 2020-02-05T19:59:54.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:54.872+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.873086+0000) 2020-02-05T19:59:54.875 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:54.872+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.873815+0000) 2020-02-05T19:59:54.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:54.873+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.874118+0000) 2020-02-05T19:59:54.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:54.873+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.874380+0000) 2020-02-05T19:59:54.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:54.873+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.874604+0000) 2020-02-05T19:59:54.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:54.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.875302+0000) 2020-02-05T19:59:54.876 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:54.875+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.875574+0000) 2020-02-05T19:59:54.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:54.875+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.875991+0000) 2020-02-05T19:59:54.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:54.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:54.878 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:54.875+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.876192+0000) 2020-02-05T19:59:54.878 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:54.876+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.876642+0000) 2020-02-05T19:59:54.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:54.881+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:54.882737+0000) 2020-02-05T19:59:55.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:55.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:55.758122+0000) 2020-02-05T19:59:55.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:55.814+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:55.815388+0000) 2020-02-05T19:59:56.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:56.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:56.617025+0000) 2020-02-05T19:59:56.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:56.624+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:56.625592+0000) 2020-02-05T19:59:56.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:56.624+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:56.625783+0000) 2020-02-05T19:59:58.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:58.597+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:58.597343+0000) 2020-02-05T19:59:58.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:58.968+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:58.969210+0000) 2020-02-05T19:59:59.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:59.254+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.255784+0000) 2020-02-05T19:59:59.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:59.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.865540+0000) 2020-02-05T19:59:59.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T19:59:59.880+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.881457+0000) 2020-02-05T19:59:59.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T19:59:59.881+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.882689+0000) 2020-02-05T19:59:59.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:59.883+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.884468+0000) 2020-02-05T19:59:59.886 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T19:59:59.884+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.885196+0000) 2020-02-05T19:59:59.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T19:59:59.884+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.884719+0000) 2020-02-05T19:59:59.887 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T19:59:59.885+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.885790+0000) 2020-02-05T19:59:59.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T19:59:59.886+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.886517+0000) 2020-02-05T19:59:59.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T19:59:59.886+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.887411+0000) 2020-02-05T19:59:59.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:59.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.887846+0000) 2020-02-05T19:59:59.890 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T19:59:59.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T19:59:59.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T19:59:59.887+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.888198+0000) 2020-02-05T19:59:59.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T19:59:59.887+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.888226+0000) 2020-02-05T19:59:59.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T19:59:59.887+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.888462+0000) 2020-02-05T19:59:59.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T19:59:59.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.889010+0000) 2020-02-05T19:59:59.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T19:59:59.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T18:59:59.890490+0000) 2020-02-05T20:00:00.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:00.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:00.105937+0000) 2020-02-05T20:00:00.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:00.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:00.106103+0000) 2020-02-05T20:00:00.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:00.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:00.106159+0000) 2020-02-05T20:00:02.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:02.214+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:02.215697+0000) 2020-02-05T20:00:03.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:03.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:03.341215+0000) 2020-02-05T20:00:04.894 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:04.891+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.892847+0000) 2020-02-05T20:00:04.898 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:04.895+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.896604+0000) 2020-02-05T20:00:04.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:04.896+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.897532+0000) 2020-02-05T20:00:04.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:04.896+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.897725+0000) 2020-02-05T20:00:04.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:04.897+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.898778+0000) 2020-02-05T20:00:04.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:04.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:04.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:04.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.899319+0000) 2020-02-05T20:00:04.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:04.898+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.899381+0000) 2020-02-05T20:00:04.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:04.898+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.899964+0000) 2020-02-05T20:00:04.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:04.898+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.899672+0000) 2020-02-05T20:00:04.902 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:04.898+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.900026+0000) 2020-02-05T20:00:04.902 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:04.898+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.900296+0000) 2020-02-05T20:00:04.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:04.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.900281+0000) 2020-02-05T20:00:04.903 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:04.899+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.901232+0000) 2020-02-05T20:00:04.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:04.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:04.907838+0000) 2020-02-05T20:00:05.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:05.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:05.758333+0000) 2020-02-05T20:00:05.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:05.825+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:05.826791+0000) 2020-02-05T20:00:06.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:06.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:06.617325+0000) 2020-02-05T20:00:06.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:06.624+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:06.625838+0000) 2020-02-05T20:00:06.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:06.624+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:06.625986+0000) 2020-02-05T20:00:08.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:08.596+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:08.597637+0000) 2020-02-05T20:00:08.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:08.968+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:08.969463+0000) 2020-02-05T20:00:09.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:09.254+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.256032+0000) 2020-02-05T20:00:09.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:09.865+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.865788+0000) 2020-02-05T20:00:09.904 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:09.902+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.903334+0000) 2020-02-05T20:00:09.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:09.904+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.905620+0000) 2020-02-05T20:00:09.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:09.905+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.906704+0000) 2020-02-05T20:00:09.908 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:09.905+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.906715+0000) 2020-02-05T20:00:09.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:09.906+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.907527+0000) 2020-02-05T20:00:09.909 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:09.907+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.908335+0000) 2020-02-05T20:00:09.909 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:09.906+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.908154+0000) 2020-02-05T20:00:09.910 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:09.907+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.908619+0000) 2020-02-05T20:00:09.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:09.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.908895+0000) 2020-02-05T20:00:09.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:09.908+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.909377+0000) 2020-02-05T20:00:09.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:09.908+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.909719+0000) 2020-02-05T20:00:09.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:09.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:09.912 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:09.908+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.910162+0000) 2020-02-05T20:00:09.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:09.908+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.910251+0000) 2020-02-05T20:00:09.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:09.912+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:09.913343+0000) 2020-02-05T20:00:10.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:10.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:10.106142+0000) 2020-02-05T20:00:10.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:10.104+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:10.106288+0000) 2020-02-05T20:00:10.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:10.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:10.106398+0000) 2020-02-05T20:00:11.718 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:00:11.725 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:00:12.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:12.215+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:12.215998+0000) 2020-02-05T20:00:13.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:13.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:13.341489+0000) 2020-02-05T20:00:14.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:14.915+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.916668+0000) 2020-02-05T20:00:14.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:14.915+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.916962+0000) 2020-02-05T20:00:14.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:14.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.917588+0000) 2020-02-05T20:00:14.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:14.916+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.918124+0000) 2020-02-05T20:00:14.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:14.916+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.917807+0000) 2020-02-05T20:00:14.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:14.917+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.918845+0000) 2020-02-05T20:00:14.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:14.918+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.919296+0000) 2020-02-05T20:00:14.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:14.918+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.919596+0000) 2020-02-05T20:00:14.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:14.918+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.919978+0000) 2020-02-05T20:00:14.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:14.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43918 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:14.922 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:14.918+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.920085+0000) 2020-02-05T20:00:14.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:14.918+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.920162+0000) 2020-02-05T20:00:14.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:14.919+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.920774+0000) 2020-02-05T20:00:14.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:14.919+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.920481+0000) 2020-02-05T20:00:14.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:14.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:14.926154+0000) 2020-02-05T20:00:15.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:15.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:15.758566+0000) 2020-02-05T20:00:15.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:15.834+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:15.834894+0000) 2020-02-05T20:00:16.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:16.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:16.617564+0000) 2020-02-05T20:00:16.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:16.625+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:16.626133+0000) 2020-02-05T20:00:16.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:16.625+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:16.626235+0000) 2020-02-05T20:00:18.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:18.597+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:18.597930+0000) 2020-02-05T20:00:18.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:18.969+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:18.969724+0000) 2020-02-05T20:00:19.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:19.255+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.256303+0000) 2020-02-05T20:00:19.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:19.864+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.865978+0000) 2020-02-05T20:00:19.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:19.924+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.925488+0000) 2020-02-05T20:00:19.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:19.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.926319+0000) 2020-02-05T20:00:19.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:19.925+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.926423+0000) 2020-02-05T20:00:19.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:19.927+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.928944+0000) 2020-02-05T20:00:19.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:19.928+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.929472+0000) 2020-02-05T20:00:19.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:19.928+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.929503+0000) 2020-02-05T20:00:19.931 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:19.928+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.930005+0000) 2020-02-05T20:00:19.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:19.929+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.930226+0000) 2020-02-05T20:00:19.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:19.928+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.930211+0000) 2020-02-05T20:00:19.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:19.929+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.930668+0000) 2020-02-05T20:00:19.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:19.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:19.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:19.929+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.930834+0000) 2020-02-05T20:00:19.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:19.930+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.931652+0000) 2020-02-05T20:00:19.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:19.933+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.934485+0000) 2020-02-05T20:00:19.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:19.933+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:19.934913+0000) 2020-02-05T20:00:20.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:20.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:20.106372+0000) 2020-02-05T20:00:20.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:20.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:20.106526+0000) 2020-02-05T20:00:20.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:20.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:20.106655+0000) 2020-02-05T20:00:22.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:22.214+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:22.216236+0000) 2020-02-05T20:00:23.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:23.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:23.341733+0000) 2020-02-05T20:00:24.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:24.936+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.937644+0000) 2020-02-05T20:00:24.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:24.937+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.938693+0000) 2020-02-05T20:00:24.940 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:24.937+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.938620+0000) 2020-02-05T20:00:24.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:24.938+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.939428+0000) 2020-02-05T20:00:24.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:24.938+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.939922+0000) 2020-02-05T20:00:24.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:24.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.940148+0000) 2020-02-05T20:00:24.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:24.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 43997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:24.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:24.938+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.940382+0000) 2020-02-05T20:00:24.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:24.940+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.941216+0000) 2020-02-05T20:00:24.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:24.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.942265+0000) 2020-02-05T20:00:24.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:24.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.941462+0000) 2020-02-05T20:00:24.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:24.941+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.942478+0000) 2020-02-05T20:00:24.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:24.941+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.942930+0000) 2020-02-05T20:00:24.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:24.945+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.946877+0000) 2020-02-05T20:00:24.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:24.946+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:24.947400+0000) 2020-02-05T20:00:25.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:25.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:25.758749+0000) 2020-02-05T20:00:25.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:25.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:25.843201+0000) 2020-02-05T20:00:26.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:26.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:26.617934+0000) 2020-02-05T20:00:26.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:26.626+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:26.626380+0000) 2020-02-05T20:00:26.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:26.626+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:26.626494+0000) 2020-02-05T20:00:28.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:28.597+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:28.598188+0000) 2020-02-05T20:00:28.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:28.969+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:28.969960+0000) 2020-02-05T20:00:29.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:29.256+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.256541+0000) 2020-02-05T20:00:29.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:29.865+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.866262+0000) 2020-02-05T20:00:29.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:29.945+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.946581+0000) 2020-02-05T20:00:29.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:29.946+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.947732+0000) 2020-02-05T20:00:29.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:29.947+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.948291+0000) 2020-02-05T20:00:29.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:29.947+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.948711+0000) 2020-02-05T20:00:29.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:29.947+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.949103+0000) 2020-02-05T20:00:29.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:29.948+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.950093+0000) 2020-02-05T20:00:29.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:29.949+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.950502+0000) 2020-02-05T20:00:29.952 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:29.949+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.951046+0000) 2020-02-05T20:00:29.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:29.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:29.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:29.950+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.951702+0000) 2020-02-05T20:00:29.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:29.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.952710+0000) 2020-02-05T20:00:29.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:29.951+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.952588+0000) 2020-02-05T20:00:29.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:29.951+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.952977+0000) 2020-02-05T20:00:29.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:29.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.955557+0000) 2020-02-05T20:00:29.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:29.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:29.958160+0000) 2020-02-05T20:00:30.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:30.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:30.106638+0000) 2020-02-05T20:00:30.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:30.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:30.106748+0000) 2020-02-05T20:00:30.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:30.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:30.107030+0000) 2020-02-05T20:00:32.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:32.215+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:32.216476+0000) 2020-02-05T20:00:33.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:33.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:33.342050+0000) 2020-02-05T20:00:34.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:34.955+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.956877+0000) 2020-02-05T20:00:34.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:34.957+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.959101+0000) 2020-02-05T20:00:34.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:34.958+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.959972+0000) 2020-02-05T20:00:34.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:34.958+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.959970+0000) 2020-02-05T20:00:34.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:34.958+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.960218+0000) 2020-02-05T20:00:34.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:34.959+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.960809+0000) 2020-02-05T20:00:34.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:34.960+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.961438+0000) 2020-02-05T20:00:34.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:34.959+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.961009+0000) 2020-02-05T20:00:34.963 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:34.961+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.962269+0000) 2020-02-05T20:00:34.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:34.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.962836+0000) 2020-02-05T20:00:34.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:34.962+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.963163+0000) 2020-02-05T20:00:34.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:34.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:34.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:34.961+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.963367+0000) 2020-02-05T20:00:34.965 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:34.962+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.963667+0000) 2020-02-05T20:00:34.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:34.969+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:34.970706+0000) 2020-02-05T20:00:35.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:35.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:35.758955+0000) 2020-02-05T20:00:35.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:35.850+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:35.851123+0000) 2020-02-05T20:00:36.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:36.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:36.618320+0000) 2020-02-05T20:00:36.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:36.626+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:36.626629+0000) 2020-02-05T20:00:36.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:36.626+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:36.626756+0000) 2020-02-05T20:00:38.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:38.597+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:38.598460+0000) 2020-02-05T20:00:38.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:38.970+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:38.970177+0000) 2020-02-05T20:00:39.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:39.256+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.256770+0000) 2020-02-05T20:00:39.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:39.866+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.866437+0000) 2020-02-05T20:00:39.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:39.967+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.967848+0000) 2020-02-05T20:00:39.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:39.968+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.968661+0000) 2020-02-05T20:00:39.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:39.968+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.968997+0000) 2020-02-05T20:00:39.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:39.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.970427+0000) 2020-02-05T20:00:39.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:39.969+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.970807+0000) 2020-02-05T20:00:39.972 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:39.969+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.971280+0000) 2020-02-05T20:00:39.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:39.970+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.971784+0000) 2020-02-05T20:00:39.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:39.972+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.972565+0000) 2020-02-05T20:00:39.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:39.973+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.973729+0000) 2020-02-05T20:00:39.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:39.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.973522+0000) 2020-02-05T20:00:39.975 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:39.972+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.973818+0000) 2020-02-05T20:00:39.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:39.972+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.973964+0000) 2020-02-05T20:00:39.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:39.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:39.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:39.974+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.974457+0000) 2020-02-05T20:00:39.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:39.979+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:39.979726+0000) 2020-02-05T20:00:40.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:40.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:40.106863+0000) 2020-02-05T20:00:40.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:40.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:40.107002+0000) 2020-02-05T20:00:40.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:40.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:40.107247+0000) 2020-02-05T20:00:41.940 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:00:41.946 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:00:42.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:42.216+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:42.216728+0000) 2020-02-05T20:00:43.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:43.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:43.342294+0000) 2020-02-05T20:00:44.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:44.977+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.978598+0000) 2020-02-05T20:00:44.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:44.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.980525+0000) 2020-02-05T20:00:44.983 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:44.980+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.981679+0000) 2020-02-05T20:00:44.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:44.980+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.981955+0000) 2020-02-05T20:00:44.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:44.981+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.982576+0000) 2020-02-05T20:00:44.984 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:44.981+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.982950+0000) 2020-02-05T20:00:44.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:44.982+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.983571+0000) 2020-02-05T20:00:44.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:44.982+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.983796+0000) 2020-02-05T20:00:44.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:44.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.984508+0000) 2020-02-05T20:00:44.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:44.983+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.984813+0000) 2020-02-05T20:00:44.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:44.984+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.985750+0000) 2020-02-05T20:00:44.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:44.984+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.985822+0000) 2020-02-05T20:00:44.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:44.984+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.986358+0000) 2020-02-05T20:00:44.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:44.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:44.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:44.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:44.990966+0000) 2020-02-05T20:00:45.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:45.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:45.759163+0000) 2020-02-05T20:00:45.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:45.861+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:45.862596+0000) 2020-02-05T20:00:46.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:46.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:46.618563+0000) 2020-02-05T20:00:46.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:46.625+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:46.626898+0000) 2020-02-05T20:00:46.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:46.625+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:46.626989+0000) 2020-02-05T20:00:48.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:48.598+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:48.598754+0000) 2020-02-05T20:00:48.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:48.969+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:48.970469+0000) 2020-02-05T20:00:49.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:49.256+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.257019+0000) 2020-02-05T20:00:49.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:49.865+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.866704+0000) 2020-02-05T20:00:49.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:49.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.992176+0000) 2020-02-05T20:00:49.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:49.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.992637+0000) 2020-02-05T20:00:49.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:49.992+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.993124+0000) 2020-02-05T20:00:49.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:49.993+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.994784+0000) 2020-02-05T20:00:49.997 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:49.994+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.995942+0000) 2020-02-05T20:00:49.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:49.993+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.995157+0000) 2020-02-05T20:00:49.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:49.994+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.995552+0000) 2020-02-05T20:00:49.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:49.995+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.996242+0000) 2020-02-05T20:00:49.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:49.995+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.996904+0000) 2020-02-05T20:00:49.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:49.995+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.997065+0000) 2020-02-05T20:00:49.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:49.996+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.997733+0000) 2020-02-05T20:00:49.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:49.996+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:49.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:49.997+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.998105+0000) 2020-02-05T20:00:50.000 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:49.997+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:49.999097+0000) 2020-02-05T20:00:50.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:50.004+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:50.005705+0000) 2020-02-05T20:00:50.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:50.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:50.107091+0000) 2020-02-05T20:00:50.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:50.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:50.107214+0000) 2020-02-05T20:00:50.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:50.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:50.107480+0000) 2020-02-05T20:00:52.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:52.216+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:52.216995+0000) 2020-02-05T20:00:53.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:53.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:53.342553+0000) 2020-02-05T20:00:55.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:55.003+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.005049+0000) 2020-02-05T20:00:55.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:00:55.005+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.005810+0000) 2020-02-05T20:00:55.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:55.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.005637+0000) 2020-02-05T20:00:55.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:00:55.005+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.006066+0000) 2020-02-05T20:00:55.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:00:55.006+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.006586+0000) 2020-02-05T20:00:55.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:55.007+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.007930+0000) 2020-02-05T20:00:55.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:55.006+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.007951+0000) 2020-02-05T20:00:55.009 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:55.007+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.008284+0000) 2020-02-05T20:00:55.010 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:00:55.007+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.007497+0000) 2020-02-05T20:00:55.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:55.007+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.008700+0000) 2020-02-05T20:00:55.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:55.008+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.009423+0000) 2020-02-05T20:00:55.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:55.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.009620+0000) 2020-02-05T20:00:55.011 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:00:55.009+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:00:55.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:00:55.009+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.010224+0000) 2020-02-05T20:00:55.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:55.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.016658+0000) 2020-02-05T20:00:55.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:55.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.759374+0000) 2020-02-05T20:00:55.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:55.870+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:55.871143+0000) 2020-02-05T20:00:56.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:00:56.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:56.618800+0000) 2020-02-05T20:00:56.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:00:56.626+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:56.627137+0000) 2020-02-05T20:00:56.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:00:56.626+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:56.627287+0000) 2020-02-05T20:00:58.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:00:58.598+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:58.599005+0000) 2020-02-05T20:00:58.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:00:58.969+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:58.970700+0000) 2020-02-05T20:00:59.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:00:59.256+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:59.257316+0000) 2020-02-05T20:00:59.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:00:59.866+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:00:59.866914+0000) 2020-02-05T20:01:00.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:00.015+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.016256+0000) 2020-02-05T20:01:00.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:00.015+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.015960+0000) 2020-02-05T20:01:00.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:00.016+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.017240+0000) 2020-02-05T20:01:00.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:00.017+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.018216+0000) 2020-02-05T20:01:00.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:00.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.018665+0000) 2020-02-05T20:01:00.020 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:00.017+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.018844+0000) 2020-02-05T20:01:00.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:00.018+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.018853+0000) 2020-02-05T20:01:00.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:00.019+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.019572+0000) 2020-02-05T20:01:00.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:00.018+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.019276+0000) 2020-02-05T20:01:00.021 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:00.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.020232+0000) 2020-02-05T20:01:00.022 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:00.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:00.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:00.019+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.020519+0000) 2020-02-05T20:01:00.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:00.020+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.021478+0000) 2020-02-05T20:01:00.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:00.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.023470+0000) 2020-02-05T20:01:00.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:00.026+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.027674+0000) 2020-02-05T20:01:00.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:00.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.107338+0000) 2020-02-05T20:01:00.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:00.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.107493+0000) 2020-02-05T20:01:00.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:00.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:00.107685+0000) 2020-02-05T20:01:02.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:02.216+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:02.217249+0000) 2020-02-05T20:01:03.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:03.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:03.342810+0000) 2020-02-05T20:01:03.925 INFO:tasks.workunit.client.0.smithi008.stderr:make: *** [Makefile:1652: kernel] Terminated 2020-02-05T20:01:03.925 DEBUG:teuthology.orchestra.run:got remote process result: 124 2020-02-05T20:01:05.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:05.026+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.027206+0000) 2020-02-05T20:01:05.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:05.026+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.027429+0000) 2020-02-05T20:01:05.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:05.027+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.027744+0000) 2020-02-05T20:01:05.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:05.027+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.028386+0000) 2020-02-05T20:01:05.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:05.027+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.028799+0000) 2020-02-05T20:01:05.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:05.028+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.029216+0000) 2020-02-05T20:01:05.030 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:05.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.029500+0000) 2020-02-05T20:01:05.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:05.030+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.030647+0000) 2020-02-05T20:01:05.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:05.029+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.030907+0000) 2020-02-05T20:01:05.032 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:05.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44312 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:05.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:05.031+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.031817+0000) 2020-02-05T20:01:05.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:05.031+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.031979+0000) 2020-02-05T20:01:05.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:05.031+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.032277+0000) 2020-02-05T20:01:05.034 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:05.031+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.032036+0000) 2020-02-05T20:01:05.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:05.037+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.038119+0000) 2020-02-05T20:01:05.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:05.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.759586+0000) 2020-02-05T20:01:05.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:05.878+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:05.879320+0000) 2020-02-05T20:01:06.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:06.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:06.619051+0000) 2020-02-05T20:01:06.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:06.626+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:06.627429+0000) 2020-02-05T20:01:06.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:06.626+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:06.627530+0000) 2020-02-05T20:01:08.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:08.598+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:08.599266+0000) 2020-02-05T20:01:08.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:08.970+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:08.970935+0000) 2020-02-05T20:01:09.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:09.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:09.257587+0000) 2020-02-05T20:01:09.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:09.866+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:09.867158+0000) 2020-02-05T20:01:10.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:10.036+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.037069+0000) 2020-02-05T20:01:10.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:10.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.038177+0000) 2020-02-05T20:01:10.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:10.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.038616+0000) 2020-02-05T20:01:10.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:10.038+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.039122+0000) 2020-02-05T20:01:10.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:10.038+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.039699+0000) 2020-02-05T20:01:10.041 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:10.038+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.039339+0000) 2020-02-05T20:01:10.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:10.039+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.040260+0000) 2020-02-05T20:01:10.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:10.039+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.040513+0000) 2020-02-05T20:01:10.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:10.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44353 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:10.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:10.041+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.041809+0000) 2020-02-05T20:01:10.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:10.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.043131+0000) 2020-02-05T20:01:10.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:10.042+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.043384+0000) 2020-02-05T20:01:10.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:10.043+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.043589+0000) 2020-02-05T20:01:10.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:10.045+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.045530+0000) 2020-02-05T20:01:10.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:10.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.047717+0000) 2020-02-05T20:01:10.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:10.107+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.107550+0000) 2020-02-05T20:01:10.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:10.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.107705+0000) 2020-02-05T20:01:10.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:10.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:10.107938+0000) 2020-02-05T20:01:12.162 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:01:12.169 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:01:12.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:12.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:12.217532+0000) 2020-02-05T20:01:13.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:13.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:13.343049+0000) 2020-02-05T20:01:15.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:15.045+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.046941+0000) 2020-02-05T20:01:15.049 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:15.047+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.048527+0000) 2020-02-05T20:01:15.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:15.049+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.049654+0000) 2020-02-05T20:01:15.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:15.049+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.050169+0000) 2020-02-05T20:01:15.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:15.049+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.050406+0000) 2020-02-05T20:01:15.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:15.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.051125+0000) 2020-02-05T20:01:15.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:15.050+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.051403+0000) 2020-02-05T20:01:15.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:15.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.051918+0000) 2020-02-05T20:01:15.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:15.050+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.052025+0000) 2020-02-05T20:01:15.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:15.051+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.052256+0000) 2020-02-05T20:01:15.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:15.051+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.051869+0000) 2020-02-05T20:01:15.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:15.052+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.052679+0000) 2020-02-05T20:01:15.054 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:15.052+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.052837+0000) 2020-02-05T20:01:15.055 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:15.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:15.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:15.057+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.058721+0000) 2020-02-05T20:01:15.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:15.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.759794+0000) 2020-02-05T20:01:15.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:15.886+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:15.887370+0000) 2020-02-05T20:01:16.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:16.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:16.619315+0000) 2020-02-05T20:01:16.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:16.627+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:16.627665+0000) 2020-02-05T20:01:16.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:16.627+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:16.627823+0000) 2020-02-05T20:01:18.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:18.599+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:18.599562+0000) 2020-02-05T20:01:18.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:18.970+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:18.971201+0000) 2020-02-05T20:01:19.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:19.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:19.257836+0000) 2020-02-05T20:01:19.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:19.867+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:19.867466+0000) 2020-02-05T20:01:20.060 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:20.057+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.058942+0000) 2020-02-05T20:01:20.061 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:20.059+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.060135+0000) 2020-02-05T20:01:20.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:20.060+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.060963+0000) 2020-02-05T20:01:20.063 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:20.060+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.061961+0000) 2020-02-05T20:01:20.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:20.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.062291+0000) 2020-02-05T20:01:20.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:20.062+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.063363+0000) 2020-02-05T20:01:20.065 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:20.063+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.064221+0000) 2020-02-05T20:01:20.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:20.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44430 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:20.066 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:20.064+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.064983+0000) 2020-02-05T20:01:20.066 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:20.064+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.064774+0000) 2020-02-05T20:01:20.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:20.064+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.065483+0000) 2020-02-05T20:01:20.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:20.064+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.065949+0000) 2020-02-05T20:01:20.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:20.065+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.065593+0000) 2020-02-05T20:01:20.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:20.065+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.065936+0000) 2020-02-05T20:01:20.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:20.067+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.068683+0000) 2020-02-05T20:01:20.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:20.107+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.107749+0000) 2020-02-05T20:01:20.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:20.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.107894+0000) 2020-02-05T20:01:20.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:20.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:20.108122+0000) 2020-02-05T20:01:22.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:22.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:22.217821+0000) 2020-02-05T20:01:23.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:23.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:23.343279+0000) 2020-02-05T20:01:25.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:25.069+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.070505+0000) 2020-02-05T20:01:25.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:25.070+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.071627+0000) 2020-02-05T20:01:25.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:25.070+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.071320+0000) 2020-02-05T20:01:25.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:25.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.072337+0000) 2020-02-05T20:01:25.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:25.071+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.072604+0000) 2020-02-05T20:01:25.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:25.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.073400+0000) 2020-02-05T20:01:25.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:25.072+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.073917+0000) 2020-02-05T20:01:25.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:25.073+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.074279+0000) 2020-02-05T20:01:25.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:25.074+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.074730+0000) 2020-02-05T20:01:25.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:25.074+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.075322+0000) 2020-02-05T20:01:25.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:25.075+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.075780+0000) 2020-02-05T20:01:25.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:25.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:25.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:25.075+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.076090+0000) 2020-02-05T20:01:25.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:25.076+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.076417+0000) 2020-02-05T20:01:25.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:25.082+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.083312+0000) 2020-02-05T20:01:25.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:25.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.760002+0000) 2020-02-05T20:01:25.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:25.894+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:25.894496+0000) 2020-02-05T20:01:26.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:26.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:26.619504+0000) 2020-02-05T20:01:26.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:26.626+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:26.627924+0000) 2020-02-05T20:01:26.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:26.627+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:26.628132+0000) 2020-02-05T20:01:28.601 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:28.599+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:28.599820+0000) 2020-02-05T20:01:28.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:28.971+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:28.971471+0000) 2020-02-05T20:01:29.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:29.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:29.258071+0000) 2020-02-05T20:01:29.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:29.867+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:29.867705+0000) 2020-02-05T20:01:30.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:30.079+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.080684+0000) 2020-02-05T20:01:30.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:30.080+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.081787+0000) 2020-02-05T20:01:30.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:30.082+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.082683+0000) 2020-02-05T20:01:30.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:30.082+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.083397+0000) 2020-02-05T20:01:30.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:30.082+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.083442+0000) 2020-02-05T20:01:30.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:30.084+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.084665+0000) 2020-02-05T20:01:30.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:30.084+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.084619+0000) 2020-02-05T20:01:30.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:30.084+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.085045+0000) 2020-02-05T20:01:30.087 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:30.084+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.085553+0000) 2020-02-05T20:01:30.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:30.084+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.085284+0000) 2020-02-05T20:01:30.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:30.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.086406+0000) 2020-02-05T20:01:30.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:30.086+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.086881+0000) 2020-02-05T20:01:30.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:30.086+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.087048+0000) 2020-02-05T20:01:30.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:30.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44507 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:30.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:30.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.091720+0000) 2020-02-05T20:01:30.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:30.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.107992+0000) 2020-02-05T20:01:30.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:30.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.108067+0000) 2020-02-05T20:01:30.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:30.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:30.108300+0000) 2020-02-05T20:01:32.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:32.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:32.218104+0000) 2020-02-05T20:01:33.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:33.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:33.343537+0000) 2020-02-05T20:01:35.095 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:35.093+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.093919+0000) 2020-02-05T20:01:35.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:35.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.094869+0000) 2020-02-05T20:01:35.096 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:35.095+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.095278+0000) 2020-02-05T20:01:35.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:35.094+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.095841+0000) 2020-02-05T20:01:35.097 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:35.096+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.096145+0000) 2020-02-05T20:01:35.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:35.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.096478+0000) 2020-02-05T20:01:35.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:35.095+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.097171+0000) 2020-02-05T20:01:35.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:35.096+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.096753+0000) 2020-02-05T20:01:35.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:35.095+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.097330+0000) 2020-02-05T20:01:35.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:35.096+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.097848+0000) 2020-02-05T20:01:35.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:35.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:35.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:35.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.099026+0000) 2020-02-05T20:01:35.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:35.097+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.098609+0000) 2020-02-05T20:01:35.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:35.097+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.099300+0000) 2020-02-05T20:01:35.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:35.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.104994+0000) 2020-02-05T20:01:35.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:35.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.760216+0000) 2020-02-05T20:01:35.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:35.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:35.902955+0000) 2020-02-05T20:01:36.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:36.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:36.619769+0000) 2020-02-05T20:01:36.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:36.628+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:36.628153+0000) 2020-02-05T20:01:36.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:36.628+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:36.628354+0000) 2020-02-05T20:01:38.601 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:38.599+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:38.600083+0000) 2020-02-05T20:01:38.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:38.970+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:38.971716+0000) 2020-02-05T20:01:39.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:39.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:39.258331+0000) 2020-02-05T20:01:39.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:39.866+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:39.867929+0000) 2020-02-05T20:01:40.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:40.102+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.103294+0000) 2020-02-05T20:01:40.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:40.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.104503+0000) 2020-02-05T20:01:40.106 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:40.104+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.105236+0000) 2020-02-05T20:01:40.107 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:40.104+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.106019+0000) 2020-02-05T20:01:40.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:40.105+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.106498+0000) 2020-02-05T20:01:40.108 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:40.104+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.106352+0000) 2020-02-05T20:01:40.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:40.105+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.106833+0000) 2020-02-05T20:01:40.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:40.105+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.107133+0000) 2020-02-05T20:01:40.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:40.106+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.107516+0000) 2020-02-05T20:01:40.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:40.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.107958+0000) 2020-02-05T20:01:40.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:40.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.108168+0000) 2020-02-05T20:01:40.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:40.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:40.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:40.107+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.108468+0000) 2020-02-05T20:01:40.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:40.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.108488+0000) 2020-02-05T20:01:40.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:40.109+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.111252+0000) 2020-02-05T20:01:40.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:40.110+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.111993+0000) 2020-02-05T20:01:40.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:40.115+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:40.116071+0000) 2020-02-05T20:01:42.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:42.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:42.218348+0000) 2020-02-05T20:01:42.308 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:01:42.314 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:01:43.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:43.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:43.343784+0000) 2020-02-05T20:01:45.115 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:45.113+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.114248+0000) 2020-02-05T20:01:45.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:45.113+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.114629+0000) 2020-02-05T20:01:45.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:45.114+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.115816+0000) 2020-02-05T20:01:45.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:45.115+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.116659+0000) 2020-02-05T20:01:45.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:45.116+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.117224+0000) 2020-02-05T20:01:45.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:45.115+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.117409+0000) 2020-02-05T20:01:45.119 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:45.116+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:45.120 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:45.117+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.118958+0000) 2020-02-05T20:01:45.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:45.116+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.118377+0000) 2020-02-05T20:01:45.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:45.117+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.119183+0000) 2020-02-05T20:01:45.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:45.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.119374+0000) 2020-02-05T20:01:45.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:45.118+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.119968+0000) 2020-02-05T20:01:45.122 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:45.118+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.120365+0000) 2020-02-05T20:01:45.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:45.119+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.120514+0000) 2020-02-05T20:01:45.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:45.119+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.120997+0000) 2020-02-05T20:01:45.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:45.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.760465+0000) 2020-02-05T20:01:45.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:45.909+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:45.910901+0000) 2020-02-05T20:01:46.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:46.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:46.620007+0000) 2020-02-05T20:01:46.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:46.627+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:46.628383+0000) 2020-02-05T20:01:46.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:46.627+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:46.628605+0000) 2020-02-05T20:01:46.860 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:46.858+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:46.859238+0000) 2020-02-05T20:01:48.601 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:48.598+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:48.600333+0000) 2020-02-05T20:01:48.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:48.971+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:48.971952+0000) 2020-02-05T20:01:49.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:49.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:49.258562+0000) 2020-02-05T20:01:49.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:49.867+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:49.868162+0000) 2020-02-05T20:01:50.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:50.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.108384+0000) 2020-02-05T20:01:50.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:50.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.108652+0000) 2020-02-05T20:01:50.124 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:50.122+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.123516+0000) 2020-02-05T20:01:50.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:50.123+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.124062+0000) 2020-02-05T20:01:50.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:50.123+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.124656+0000) 2020-02-05T20:01:50.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:50.124+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.125575+0000) 2020-02-05T20:01:50.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:50.125+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.126488+0000) 2020-02-05T20:01:50.128 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:50.125+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.127011+0000) 2020-02-05T20:01:50.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:50.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.127520+0000) 2020-02-05T20:01:50.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:50.126+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.127792+0000) 2020-02-05T20:01:50.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:50.126+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:50.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:50.127+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.129393+0000) 2020-02-05T20:01:50.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:50.128+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.129796+0000) 2020-02-05T20:01:50.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:50.128+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.129998+0000) 2020-02-05T20:01:50.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:50.129+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.130501+0000) 2020-02-05T20:01:50.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:50.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.131840+0000) 2020-02-05T20:01:50.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:50.132+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:50.133201+0000) 2020-02-05T20:01:52.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:52.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:52.218629+0000) 2020-02-05T20:01:53.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:53.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:53.344018+0000) 2020-02-05T20:01:55.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:55.133+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.134693+0000) 2020-02-05T20:01:55.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:55.133+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.135043+0000) 2020-02-05T20:01:55.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:01:55.133+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.135307+0000) 2020-02-05T20:01:55.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:01:55.134+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.135715+0000) 2020-02-05T20:01:55.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:55.135+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.136658+0000) 2020-02-05T20:01:55.138 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:55.135+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.137034+0000) 2020-02-05T20:01:55.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:55.135+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.137057+0000) 2020-02-05T20:01:55.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:01:55.135+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.137227+0000) 2020-02-05T20:01:55.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:55.136+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.137667+0000) 2020-02-05T20:01:55.139 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:55.136+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.137626+0000) 2020-02-05T20:01:55.140 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:01:55.136+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.138069+0000) 2020-02-05T20:01:55.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:55.137+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.138496+0000) 2020-02-05T20:01:55.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:01:55.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:01:55.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:55.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.139176+0000) 2020-02-05T20:01:55.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:55.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.144669+0000) 2020-02-05T20:01:55.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:55.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.760682+0000) 2020-02-05T20:01:55.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:55.918+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:55.919293+0000) 2020-02-05T20:01:56.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:01:56.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:56.620257+0000) 2020-02-05T20:01:56.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:01:56.627+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:56.628618+0000) 2020-02-05T20:01:56.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:01:56.627+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:56.628815+0000) 2020-02-05T20:01:56.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:01:56.858+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:56.859535+0000) 2020-02-05T20:01:58.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:01:58.599+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:58.600644+0000) 2020-02-05T20:01:58.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:01:58.971+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:58.972203+0000) 2020-02-05T20:01:59.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:01:59.258+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:59.258781+0000) 2020-02-05T20:01:59.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:01:59.867+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:01:59.868395+0000) 2020-02-05T20:02:00.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:00.107+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.108662+0000) 2020-02-05T20:02:00.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:00.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.108907+0000) 2020-02-05T20:02:00.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:00.142+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.143623+0000) 2020-02-05T20:02:00.146 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:00.144+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.145069+0000) 2020-02-05T20:02:00.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:00.143+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.145173+0000) 2020-02-05T20:02:00.146 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:00.144+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.145864+0000) 2020-02-05T20:02:00.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:00.145+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.146280+0000) 2020-02-05T20:02:00.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:00.145+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.146773+0000) 2020-02-05T20:02:00.148 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:00.146+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.147371+0000) 2020-02-05T20:02:00.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:00.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.147644+0000) 2020-02-05T20:02:00.149 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:00.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:00.149 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:00.146+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.147816+0000) 2020-02-05T20:02:00.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:00.147+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.149003+0000) 2020-02-05T20:02:00.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:00.147+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.148974+0000) 2020-02-05T20:02:00.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:00.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.148938+0000) 2020-02-05T20:02:00.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:00.148+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.149711+0000) 2020-02-05T20:02:00.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:00.153+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:00.154179+0000) 2020-02-05T20:02:02.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:02.217+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:02.218859+0000) 2020-02-05T20:02:03.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:03.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:03.344270+0000) 2020-02-05T20:02:03.926 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2020-02-05T20:02:05.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:05.151+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.152922+0000) 2020-02-05T20:02:05.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:05.152+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.153318+0000) 2020-02-05T20:02:05.155 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:05.153+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.154585+0000) 2020-02-05T20:02:05.157 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:05.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.154159+0000) 2020-02-05T20:02:05.157 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:05.155+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.156211+0000) 2020-02-05T20:02:05.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:05.155+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.156998+0000) 2020-02-05T20:02:05.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:05.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.156777+0000) 2020-02-05T20:02:05.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:05.157+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.158399+0000) 2020-02-05T20:02:05.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:05.157+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.159034+0000) 2020-02-05T20:02:05.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:05.157+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.158919+0000) 2020-02-05T20:02:05.161 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:05.158+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.159676+0000) 2020-02-05T20:02:05.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:05.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:05.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:05.158+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.160251+0000) 2020-02-05T20:02:05.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:05.162+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.163689+0000) 2020-02-05T20:02:05.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:05.162+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.163971+0000) 2020-02-05T20:02:05.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:05.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.760906+0000) 2020-02-05T20:02:05.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:05.926+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:05.927560+0000) 2020-02-05T20:02:06.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:06.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:06.620508+0000) 2020-02-05T20:02:06.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:06.628+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:06.628848+0000) 2020-02-05T20:02:06.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:06.628+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:06.629109+0000) 2020-02-05T20:02:06.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:06.859+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:06.859783+0000) 2020-02-05T20:02:08.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:08.600+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:08.600883+0000) 2020-02-05T20:02:08.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:08.971+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:08.972462+0000) 2020-02-05T20:02:09.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:09.257+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:09.259028+0000) 2020-02-05T20:02:09.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:09.868+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:09.868637+0000) 2020-02-05T20:02:10.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:10.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.108964+0000) 2020-02-05T20:02:10.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:10.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.109128+0000) 2020-02-05T20:02:10.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:10.164+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.165197+0000) 2020-02-05T20:02:10.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:10.164+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.166150+0000) 2020-02-05T20:02:10.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:10.165+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.166553+0000) 2020-02-05T20:02:10.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:10.166+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.167652+0000) 2020-02-05T20:02:10.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:10.167+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.168363+0000) 2020-02-05T20:02:10.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:10.168+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.169429+0000) 2020-02-05T20:02:10.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:10.167+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.168995+0000) 2020-02-05T20:02:10.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:10.168+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.169884+0000) 2020-02-05T20:02:10.171 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:10.169+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.170444+0000) 2020-02-05T20:02:10.171 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:10.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:10.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:10.169+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.170977+0000) 2020-02-05T20:02:10.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:10.169+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.171149+0000) 2020-02-05T20:02:10.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:10.169+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.171279+0000) 2020-02-05T20:02:10.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:10.172+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.173295+0000) 2020-02-05T20:02:10.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:10.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:10.178259+0000) 2020-02-05T20:02:12.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:12.218+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:12.219100+0000) 2020-02-05T20:02:12.454 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:02:12.460 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:02:13.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:13.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:13.344505+0000) 2020-02-05T20:02:15.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:15.174+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.176043+0000) 2020-02-05T20:02:15.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:15.176+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.177588+0000) 2020-02-05T20:02:15.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:15.177+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.178133+0000) 2020-02-05T20:02:15.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:15.177+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.178266+0000) 2020-02-05T20:02:15.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:15.177+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.178582+0000) 2020-02-05T20:02:15.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:15.177+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.179002+0000) 2020-02-05T20:02:15.180 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:15.177+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.179281+0000) 2020-02-05T20:02:15.181 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:15.179+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.180126+0000) 2020-02-05T20:02:15.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:15.178+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.179796+0000) 2020-02-05T20:02:15.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:15.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.181196+0000) 2020-02-05T20:02:15.182 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:15.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:15.182 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:15.180+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.181469+0000) 2020-02-05T20:02:15.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:15.180+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.181877+0000) 2020-02-05T20:02:15.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:15.185+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.186999+0000) 2020-02-05T20:02:15.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:15.186+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.187352+0000) 2020-02-05T20:02:15.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:15.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.761131+0000) 2020-02-05T20:02:15.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:15.935+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:15.936937+0000) 2020-02-05T20:02:16.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:16.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:16.620736+0000) 2020-02-05T20:02:16.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:16.628+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:16.629114+0000) 2020-02-05T20:02:16.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:16.628+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:16.629330+0000) 2020-02-05T20:02:16.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:16.859+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:16.860030+0000) 2020-02-05T20:02:18.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:18.600+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:18.601130+0000) 2020-02-05T20:02:18.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:18.972+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:18.972772+0000) 2020-02-05T20:02:19.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:19.258+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:19.259279+0000) 2020-02-05T20:02:19.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:19.868+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:19.868867+0000) 2020-02-05T20:02:20.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:20.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.109194+0000) 2020-02-05T20:02:20.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:20.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.109397+0000) 2020-02-05T20:02:20.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:20.184+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.185839+0000) 2020-02-05T20:02:20.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:20.185+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.186810+0000) 2020-02-05T20:02:20.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:20.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.187786+0000) 2020-02-05T20:02:20.189 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:20.186+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.188028+0000) 2020-02-05T20:02:20.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:20.187+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.188730+0000) 2020-02-05T20:02:20.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:20.188+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.189681+0000) 2020-02-05T20:02:20.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:20.189+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.190232+0000) 2020-02-05T20:02:20.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:20.189+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.190984+0000) 2020-02-05T20:02:20.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:20.190+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.191517+0000) 2020-02-05T20:02:20.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:20.190+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.191712+0000) 2020-02-05T20:02:20.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:20.190+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.191938+0000) 2020-02-05T20:02:20.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:20.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:20.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:20.193+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.194357+0000) 2020-02-05T20:02:20.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:20.194+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.195679+0000) 2020-02-05T20:02:20.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:20.194+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:20.195947+0000) 2020-02-05T20:02:22.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:22.218+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:22.219395+0000) 2020-02-05T20:02:23.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:23.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:23.344745+0000) 2020-02-05T20:02:25.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:25.198+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.198663+0000) 2020-02-05T20:02:25.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:25.198+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.199117+0000) 2020-02-05T20:02:25.203 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:25.199+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.199962+0000) 2020-02-05T20:02:25.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:25.200+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.201402+0000) 2020-02-05T20:02:25.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:25.201+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.201979+0000) 2020-02-05T20:02:25.205 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:25.201+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.202755+0000) 2020-02-05T20:02:25.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:25.202+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.203160+0000) 2020-02-05T20:02:25.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:25.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.203522+0000) 2020-02-05T20:02:25.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:25.203+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.203417+0000) 2020-02-05T20:02:25.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:25.203+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.204198+0000) 2020-02-05T20:02:25.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:25.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44929 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:25.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:25.204+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.205063+0000) 2020-02-05T20:02:25.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:25.204+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.205024+0000) 2020-02-05T20:02:25.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:25.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.205177+0000) 2020-02-05T20:02:25.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:25.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.209165+0000) 2020-02-05T20:02:25.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:25.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.761321+0000) 2020-02-05T20:02:25.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:25.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:25.948477+0000) 2020-02-05T20:02:26.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:26.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:26.620976+0000) 2020-02-05T20:02:26.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:26.629+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:26.629339+0000) 2020-02-05T20:02:26.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:26.629+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:26.629586+0000) 2020-02-05T20:02:26.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:26.859+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:26.860281+0000) 2020-02-05T20:02:28.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:28.600+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:28.601391+0000) 2020-02-05T20:02:28.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:28.972+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:28.973045+0000) 2020-02-05T20:02:29.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:29.259+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:29.259512+0000) 2020-02-05T20:02:29.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:29.869+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:29.869148+0000) 2020-02-05T20:02:30.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:30.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.109467+0000) 2020-02-05T20:02:30.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:30.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.109683+0000) 2020-02-05T20:02:30.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:30.210+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.210157+0000) 2020-02-05T20:02:30.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:30.210+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.210540+0000) 2020-02-05T20:02:30.212 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:30.211+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.211122+0000) 2020-02-05T20:02:30.213 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:30.211+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.211623+0000) 2020-02-05T20:02:30.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:30.212+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.212474+0000) 2020-02-05T20:02:30.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:30.212+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.212801+0000) 2020-02-05T20:02:30.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:30.212+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.213174+0000) 2020-02-05T20:02:30.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:30.212+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.212909+0000) 2020-02-05T20:02:30.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:30.213+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.213799+0000) 2020-02-05T20:02:30.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:30.213+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.213612+0000) 2020-02-05T20:02:30.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:30.214+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 44973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:30.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:30.213+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.214242+0000) 2020-02-05T20:02:30.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:30.214+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.214746+0000) 2020-02-05T20:02:30.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:30.215+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.215485+0000) 2020-02-05T20:02:30.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:30.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:30.222129+0000) 2020-02-05T20:02:32.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:32.219+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:32.219666+0000) 2020-02-05T20:02:33.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:33.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:33.345015+0000) 2020-02-05T20:02:35.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:35.219+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.220318+0000) 2020-02-05T20:02:35.222 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:35.219+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.221045+0000) 2020-02-05T20:02:35.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:35.220+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.221503+0000) 2020-02-05T20:02:35.223 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:35.220+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.221781+0000) 2020-02-05T20:02:35.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:35.221+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.222449+0000) 2020-02-05T20:02:35.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:35.223+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.223676+0000) 2020-02-05T20:02:35.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:35.223+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.224389+0000) 2020-02-05T20:02:35.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:35.224+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.224539+0000) 2020-02-05T20:02:35.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:35.224+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.225162+0000) 2020-02-05T20:02:35.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:35.225+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:35.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:35.225+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.226235+0000) 2020-02-05T20:02:35.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:35.225+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.226239+0000) 2020-02-05T20:02:35.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:35.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.227130+0000) 2020-02-05T20:02:35.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:35.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.228731+0000) 2020-02-05T20:02:35.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:35.229+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.230868+0000) 2020-02-05T20:02:35.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:35.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.761530+0000) 2020-02-05T20:02:35.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:35.959+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:35.960391+0000) 2020-02-05T20:02:36.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:36.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:36.621175+0000) 2020-02-05T20:02:36.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:36.628+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:36.629575+0000) 2020-02-05T20:02:36.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:36.628+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:36.629781+0000) 2020-02-05T20:02:36.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:36.860+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:36.860547+0000) 2020-02-05T20:02:38.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:38.601+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:38.601670+0000) 2020-02-05T20:02:38.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:38.972+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:38.973313+0000) 2020-02-05T20:02:39.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:39.258+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:39.259747+0000) 2020-02-05T20:02:39.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:39.868+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:39.869411+0000) 2020-02-05T20:02:40.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:40.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.109728+0000) 2020-02-05T20:02:40.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:40.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.109967+0000) 2020-02-05T20:02:40.232 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:40.230+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.231528+0000) 2020-02-05T20:02:40.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:40.232+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.232915+0000) 2020-02-05T20:02:40.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:40.232+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.233402+0000) 2020-02-05T20:02:40.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:40.232+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.233749+0000) 2020-02-05T20:02:40.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:40.232+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.234055+0000) 2020-02-05T20:02:40.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:40.234+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.234700+0000) 2020-02-05T20:02:40.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:40.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.235169+0000) 2020-02-05T20:02:40.237 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:40.235+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.236469+0000) 2020-02-05T20:02:40.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:40.236+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.236689+0000) 2020-02-05T20:02:40.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:40.235+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.236954+0000) 2020-02-05T20:02:40.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:40.237+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.237761+0000) 2020-02-05T20:02:40.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:40.236+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.237704+0000) 2020-02-05T20:02:40.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:40.237+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:40.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:40.237+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.238377+0000) 2020-02-05T20:02:40.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:40.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:40.244496+0000) 2020-02-05T20:02:42.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:42.219+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:42.219922+0000) 2020-02-05T20:02:42.588 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:02:42.595 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:02:43.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:43.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:43.345245+0000) 2020-02-05T20:02:45.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:45.243+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.244506+0000) 2020-02-05T20:02:45.246 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:45.244+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.245119+0000) 2020-02-05T20:02:45.246 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:45.244+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.245744+0000) 2020-02-05T20:02:45.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:45.245+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.246098+0000) 2020-02-05T20:02:45.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:45.246+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.246754+0000) 2020-02-05T20:02:45.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:45.246+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.247304+0000) 2020-02-05T20:02:45.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:45.247+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.247459+0000) 2020-02-05T20:02:45.249 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:45.246+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.247866+0000) 2020-02-05T20:02:45.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:45.247+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.248108+0000) 2020-02-05T20:02:45.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:45.247+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.248294+0000) 2020-02-05T20:02:45.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:45.248+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.248690+0000) 2020-02-05T20:02:45.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:45.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:45.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:45.248+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.248742+0000) 2020-02-05T20:02:45.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:45.248+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.249847+0000) 2020-02-05T20:02:45.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:45.254+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.255206+0000) 2020-02-05T20:02:45.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:45.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.761736+0000) 2020-02-05T20:02:45.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:45.969+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:45.970411+0000) 2020-02-05T20:02:46.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:46.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:46.621406+0000) 2020-02-05T20:02:46.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:46.629+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:46.629808+0000) 2020-02-05T20:02:46.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:46.629+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:46.629983+0000) 2020-02-05T20:02:46.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:46.860+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:46.860829+0000) 2020-02-05T20:02:48.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:48.601+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:48.601920+0000) 2020-02-05T20:02:48.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:48.972+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:48.973543+0000) 2020-02-05T20:02:49.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:49.258+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:49.259996+0000) 2020-02-05T20:02:49.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:49.868+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:49.869634+0000) 2020-02-05T20:02:50.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:50.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.109952+0000) 2020-02-05T20:02:50.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:50.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.110235+0000) 2020-02-05T20:02:50.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:50.254+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.255373+0000) 2020-02-05T20:02:50.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:50.255+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.256193+0000) 2020-02-05T20:02:50.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:50.255+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.256740+0000) 2020-02-05T20:02:50.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:50.256+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.256537+0000) 2020-02-05T20:02:50.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:50.256+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.256851+0000) 2020-02-05T20:02:50.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:50.256+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.257142+0000) 2020-02-05T20:02:50.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:50.256+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.257369+0000) 2020-02-05T20:02:50.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:50.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.257759+0000) 2020-02-05T20:02:50.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:50.256+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.257779+0000) 2020-02-05T20:02:50.260 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:50.257+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.258344+0000) 2020-02-05T20:02:50.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:50.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.258474+0000) 2020-02-05T20:02:50.261 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:50.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:50.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:50.258+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.259175+0000) 2020-02-05T20:02:50.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:50.257+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.258811+0000) 2020-02-05T20:02:50.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:50.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:50.265826+0000) 2020-02-05T20:02:52.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:52.220+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:52.220175+0000) 2020-02-05T20:02:53.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:53.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:53.345483+0000) 2020-02-05T20:02:55.265 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:55.263+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.264593+0000) 2020-02-05T20:02:55.266 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:55.264+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.265222+0000) 2020-02-05T20:02:55.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:02:55.265+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.265600+0000) 2020-02-05T20:02:55.267 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:55.265+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.266111+0000) 2020-02-05T20:02:55.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:02:55.266+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.266554+0000) 2020-02-05T20:02:55.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:55.265+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.266580+0000) 2020-02-05T20:02:55.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:55.266+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.267139+0000) 2020-02-05T20:02:55.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:55.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.267533+0000) 2020-02-05T20:02:55.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:55.267+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.267590+0000) 2020-02-05T20:02:55.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:55.267+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.268289+0000) 2020-02-05T20:02:55.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:55.268+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.268738+0000) 2020-02-05T20:02:55.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:02:55.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:02:55.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:02:55.269+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.269498+0000) 2020-02-05T20:02:55.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:02:55.269+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.269938+0000) 2020-02-05T20:02:55.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:55.273+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.274648+0000) 2020-02-05T20:02:55.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:55.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.761947+0000) 2020-02-05T20:02:55.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:55.977+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:55.978145+0000) 2020-02-05T20:02:56.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:02:56.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:56.621726+0000) 2020-02-05T20:02:56.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:02:56.629+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:56.630035+0000) 2020-02-05T20:02:56.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:02:56.629+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:56.630223+0000) 2020-02-05T20:02:56.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:02:56.860+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:56.861123+0000) 2020-02-05T20:02:58.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:02:58.602+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:58.602164+0000) 2020-02-05T20:02:58.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:02:58.973+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:58.973784+0000) 2020-02-05T20:02:59.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:02:59.259+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:59.260276+0000) 2020-02-05T20:02:59.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:02:59.868+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:02:59.869896+0000) 2020-02-05T20:03:00.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:00.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.110181+0000) 2020-02-05T20:03:00.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:00.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.110483+0000) 2020-02-05T20:03:00.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:00.273+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.274796+0000) 2020-02-05T20:03:00.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:00.274+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.275577+0000) 2020-02-05T20:03:00.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:00.275+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.276583+0000) 2020-02-05T20:03:00.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:00.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.277026+0000) 2020-02-05T20:03:00.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:00.276+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.277376+0000) 2020-02-05T20:03:00.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:00.276+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.277845+0000) 2020-02-05T20:03:00.279 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:00.277+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.278465+0000) 2020-02-05T20:03:00.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:00.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.278876+0000) 2020-02-05T20:03:00.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:00.277+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.279135+0000) 2020-02-05T20:03:00.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:00.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.279848+0000) 2020-02-05T20:03:00.281 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:00.279+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.280458+0000) 2020-02-05T20:03:00.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:00.279+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.280996+0000) 2020-02-05T20:03:00.282 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:00.279+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45207 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:00.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:00.280+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.281849+0000) 2020-02-05T20:03:00.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:00.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:00.284159+0000) 2020-02-05T20:03:02.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:02.219+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:02.220502+0000) 2020-02-05T20:03:03.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:03.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:03.345730+0000) 2020-02-05T20:03:03.927 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: , None)> 2020-02-05T20:03:03.927 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.0... 2020-02-05T20:03:03.928 INFO:teuthology.orchestra.run.smithi008:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2020-02-05T20:03:04.502 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa/tasks/workunit.py", line 138, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa/tasks/workunit.py", line 288, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 87, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 37, in resurrect_traceback reraise(*exc_info) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 24, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-async-dirops/qa/tasks/workunit.py", line 411, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 433, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test kernel_untar_build.sh) on smithi008 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ecafd7283ef390be73a605986afe2a0f325a3255 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/kernel_untar_build.sh' 2020-02-05T20:03:04.503 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2020-02-05T20:03:04.522 DEBUG:tasks.check_counter:Getting stats from a 2020-02-05T20:03:04.523 INFO:teuthology.orchestra.run.smithi008:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok perf dump 2020-02-05T20:03:04.900 INFO:teuthology.orchestra.run.smithi008.stdout:{ 2020-02-05T20:03:04.900 INFO:teuthology.orchestra.run.smithi008.stdout: "AsyncMessenger::Worker-0": { 2020-02-05T20:03:04.900 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_messages": 2374, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages": 3884, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_bytes": 23808206, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_bytes": 924607, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_created_connections": 1579, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_active_connections": 18446744073709551614, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_total_time": 2.520803887, 2020-02-05T20:03:04.901 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_send_time": 0.240019979, 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_recv_time": 1.959138094, 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_fast_dispatch_time": 0.036863732, 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 2834, 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 5.901416154, 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.002082362 2020-02-05T20:03:04.902 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 0, 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 0.000000000, 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.000000000 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.903 INFO:teuthology.orchestra.run.smithi008.stdout: "AsyncMessenger::Worker-1": { 2020-02-05T20:03:04.904 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_messages": 2345, 2020-02-05T20:03:04.904 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages": 6204, 2020-02-05T20:03:04.904 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_bytes": 34948214, 2020-02-05T20:03:04.904 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_bytes": 2813017, 2020-02-05T20:03:04.904 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_created_connections": 2349, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_active_connections": 0, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_total_time": 3.650501664, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_send_time": 0.327872120, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_recv_time": 2.835219513, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:04.905 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 4642, 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 8.835469839, 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.001903375 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 0, 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 0.000000000, 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.000000000 2020-02-05T20:03:04.906 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "AsyncMessenger::Worker-2": { 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_messages": 815, 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages": 2191, 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_recv_bytes": 12113175, 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_bytes": 2044992, 2020-02-05T20:03:04.907 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_created_connections": 817, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_active_connections": 0, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_total_time": 1.209805472, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_send_time": 0.139891088, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_recv_time": 0.976406258, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 1649, 2020-02-05T20:03:04.908 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 2.757100271, 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.001671983 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 0, 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 0.000000000, 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.000000000 2020-02-05T20:03:04.909 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: "cct": { 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: "total_workers": 0, 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: "unhealthy_workers": 0 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: "mempool": { 2020-02-05T20:03:04.910 INFO:teuthology.orchestra.run.smithi008.stdout: "bloom_filter_bytes": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bloom_filter_items": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_alloc_bytes": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_alloc_items": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_data_bytes": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_data_items": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_onode_bytes": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_onode_items": 0, 2020-02-05T20:03:04.911 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_other_bytes": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_cache_other_items": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_bytes": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_fsck_items": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_txc_bytes": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_txc_items": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_writing_deferred_bytes": 0, 2020-02-05T20:03:04.912 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_writing_deferred_items": 0, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_writing_bytes": 0, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "bluestore_writing_items": 0, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_bytes": 0, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "bluefs_items": 0, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "buffer_anon_bytes": 75845, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "buffer_anon_items": 45, 2020-02-05T20:03:04.913 INFO:teuthology.orchestra.run.smithi008.stdout: "buffer_meta_bytes": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "buffer_meta_items": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_bytes": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_items": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mapbl_bytes": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_mapbl_items": 0, 2020-02-05T20:03:04.914 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pglog_bytes": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "osd_pglog_items": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "osdmap_bytes": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "osdmap_items": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "osdmap_mapping_bytes": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "osdmap_mapping_items": 0, 2020-02-05T20:03:04.915 INFO:teuthology.orchestra.run.smithi008.stdout: "pgmap_bytes": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "pgmap_items": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_co_bytes": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "mds_co_items": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "unittest_1_bytes": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "unittest_1_items": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "unittest_2_bytes": 0, 2020-02-05T20:03:04.916 INFO:teuthology.orchestra.run.smithi008.stdout: "unittest_2_items": 0 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: }, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "throttle-msgr_dispatch_throttler-mds": { 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "val": 0, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "max": 104857600, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "get_started": 0, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "get": 5530, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "get_sum": 70459276, 2020-02-05T20:03:04.917 INFO:teuthology.orchestra.run.smithi008.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "get_or_fail_success": 5530, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "take": 0, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "take_sum": 0, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "put": 5530, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "put_sum": 70459276, 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "wait": { 2020-02-05T20:03:04.918 INFO:teuthology.orchestra.run.smithi008.stdout: "avgcount": 0, 2020-02-05T20:03:04.919 INFO:teuthology.orchestra.run.smithi008.stdout: "sum": 0.000000000, 2020-02-05T20:03:04.919 INFO:teuthology.orchestra.run.smithi008.stdout: "avgtime": 0.000000000 2020-02-05T20:03:04.919 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T20:03:04.919 INFO:teuthology.orchestra.run.smithi008.stdout: } 2020-02-05T20:03:04.919 INFO:teuthology.orchestra.run.smithi008.stdout:} 2020-02-05T20:03:04.920 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2020-02-05T20:03:04.920 DEBUG:tasks.check_counter:Getting stats from b 2020-02-05T20:03:04.921 INFO:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok perf dump 2020-02-05T20:03:05.153 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2020-02-05T20:03:05.153 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-0": { 2020-02-05T20:03:05.153 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 17914, 2020-02-05T20:03:05.153 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 20084, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 39342341, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 103920786, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 2338, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 1, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 5.707677117, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 1.193942001, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 3.387695198, 2020-02-05T20:03:05.154 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.660634418, 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 18542, 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 16.531006012, 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000891543 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:05.155 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-1": { 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 244329, 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 397324, 2020-02-05T20:03:05.156 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 105104612, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 203291139, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 2351, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 50.638310643, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 20.258192642, 2020-02-05T20:03:05.157 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 35.809009712, 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 0.346395884, 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 395765, 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 39.720946027, 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000100364 2020-02-05T20:03:05.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 601346, 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.235103431, 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000390 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "AsyncMessenger::Worker-2": { 2020-02-05T20:03:05.159 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_messages": 22509, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages": 22621, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_recv_bytes": 8316429, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_bytes": 217640368, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_created_connections": 133, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_active_connections": 3, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_total_time": 4.419473716, 2020-02-05T20:03:05.160 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_send_time": 2.121847862, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_recv_time": 0.909808083, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_running_fast_dispatch_time": 1.109120507, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_send_messages_queue_lat": { 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 22565, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 12.231806694, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000542069 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.161 INFO:teuthology.orchestra.run.smithi164.stdout: "msgr_handle_ack_lat": { 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "cct": { 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "total_workers": 1, 2020-02-05T20:03:05.162 INFO:teuthology.orchestra.run.smithi164.stdout: "unhealthy_workers": 0 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-MDSRank": { 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 27418, 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 61.854178400, 2020-02-05T20:03:05.163 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.002255969 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "finisher-PurgeQueue": { 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "queue_len": 0, 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "complete_latency": { 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 3668, 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.466040452, 2020-02-05T20:03:05.164 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000127055 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": { 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "request": 152608, 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "reply": 152608, 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "reply_latency": { 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 152608, 2020-02-05T20:03:05.165 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 291.292351264, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.001908761 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "forward": 0, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_fetch": 0, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_commit": 3244, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_split": 113, 2020-02-05T20:03:05.166 INFO:teuthology.orchestra.run.smithi164.stdout: "dir_merge": 0, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes": 74371, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_top": 10108, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_bottom": 64263, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pin_tail": 0, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_pinned": 59930, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_expired": 3119, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "inodes_with_caps": 59700, 2020-02-05T20:03:05.167 INFO:teuthology.orchestra.run.smithi164.stdout: "caps": 59700, 2020-02-05T20:03:05.168 INFO:teuthology.orchestra.run.smithi164.stdout: "subtrees": 2, 2020-02-05T20:03:05.168 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse": 158122, 2020-02-05T20:03:05.168 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_hit": 141433, 2020-02-05T20:03:05.168 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_forward": 0, 2020-02-05T20:03:05.168 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_discover": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_dir_fetch": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_remote_ino": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "traverse_lock": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "load_cent": 2381, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "q": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "exported": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "exported_inodes": 0, 2020-02-05T20:03:05.169 INFO:teuthology.orchestra.run.smithi164.stdout: "imported": 0, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "imported_inodes": 0, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_dir_fetch": 0, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_backtrace_fetch": 0, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "openino_peer_discover": 0, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rfiles": 69033, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rbytes": 1100720303, 2020-02-05T20:03:05.170 INFO:teuthology.orchestra.run.smithi164.stdout: "root_rsnaps": 0 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_cache": { 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays": 0, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_delayed": 0, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "num_strays_enqueuing": 0, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_created": 785, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_enqueued": 785, 2020-02-05T20:03:05.171 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_reintegrated": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "strays_migrated": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_processing": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_enqueued": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "num_recovering_prioritized": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_started": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "recovery_completed": 0, 2020-02-05T20:03:05.172 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_enqueue_scrub": 0, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_exportdir": 0, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_flush": 0, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragmentdir": 113, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_fragstats": 0, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ireq_inodestats": 0 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.173 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_log": { 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "evadd": 156134, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "evex": 61876, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "evtrm": 61876, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "ev": 94258, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "evexg": 0, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "evexd": 1024, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "segadd": 213, 2020-02-05T20:03:05.174 INFO:teuthology.orchestra.run.smithi164.stdout: "segex": 85, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "segtrm": 85, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "seg": 129, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "segexg": 0, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "segexd": 1, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "expos": 123625787, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "wrpos": 311632297, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "rdpos": 0, 2020-02-05T20:03:05.175 INFO:teuthology.orchestra.run.smithi164.stdout: "jlat": { 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 7763, 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 289.023903190, 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.037230954 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: "replayed": 0 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.176 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_mem": { 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "ino": 74374, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "ino+": 75158, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "ino-": 784, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "dir": 6112, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "dir+": 6225, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "dir-": 113, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "dn": 75103, 2020-02-05T20:03:05.177 INFO:teuthology.orchestra.run.smithi164.stdout: "dn+": 79006, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "dn-": 3903, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "cap": 59699, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "cap+": 75146, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "cap-": 15447, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "rss": 786064, 2020-02-05T20:03:05.178 INFO:teuthology.orchestra.run.smithi164.stdout: "heap": 315556 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_server": { 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_client_request": 159764, 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "dispatch_server_request": 0, 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_request": 152608, 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_client_session": 159, 2020-02-05T20:03:05.179 INFO:teuthology.orchestra.run.smithi164.stdout: "handle_slave_request": 0, 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "req_create_latency": { 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 69801, 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 196.415081827, 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.002813929 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getattr_latency": { 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 58419, 2020-02-05T20:03:05.180 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 42.066225083, 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000720077 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: "req_getfilelock_latency": { 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "req_link_latency": { 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookup_latency": { 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 15896, 2020-02-05T20:03:05.182 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 17.815036298, 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.001120724 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookuphash_latency": { 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupino_latency": { 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupname_latency": { 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.184 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupparent_latency": { 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lookupsnap_latency": { 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "req_lssnap_latency": { 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.186 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mkdir_latency": { 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 5309, 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 17.500524673, 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.003296388 2020-02-05T20:03:05.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mknod_latency": { 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.188 INFO:teuthology.orchestra.run.smithi164.stdout: "req_mksnap_latency": { 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "req_open_latency": { 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 841, 2020-02-05T20:03:05.189 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.756230406, 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000899203 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "req_readdir_latency": { 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 948, 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 6.049325824, 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.006381145 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.190 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rename_latency": { 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 78, 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.319272149, 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.004093232 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "req_renamesnap_latency": { 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.191 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmdir_latency": { 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.192 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmsnap_latency": { 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "req_rmxattr_latency": { 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.193 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setattr_latency": { 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 542, 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 7.128364643, 2020-02-05T20:03:05.194 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.013151964 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setdirlayout_latency": { 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setfilelock_latency": { 2020-02-05T20:03:05.195 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setlayout_latency": { 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.196 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: "req_setxattr_latency": { 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.197 INFO:teuthology.orchestra.run.smithi164.stdout: "req_symlink_latency": { 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 35, 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.075865482, 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.002167585 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "req_unlink_latency": { 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 739, 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 3.166424879, 2020-02-05T20:03:05.198 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.004284742 2020-02-05T20:03:05.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.199 INFO:teuthology.orchestra.run.smithi164.stdout: "cap_revoke_eviction": 0 2020-02-05T20:03:05.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.199 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_sessions": { 2020-02-05T20:03:05.199 INFO:teuthology.orchestra.run.smithi164.stdout: "session_count": 1, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "session_add": 1, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "session_remove": 0, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_open": 1, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "sessions_stale": 0, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "total_load": 390, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "average_load": 390, 2020-02-05T20:03:05.200 INFO:teuthology.orchestra.run.smithi164.stdout: "avg_session_uptime": 3129 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "mempool": { 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_bytes": 82080, 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "bloom_filter_items": 82080, 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_bytes": 0, 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_alloc_items": 0, 2020-02-05T20:03:05.201 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_bytes": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_data_items": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_bytes": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_onode_items": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_bytes": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_cache_other_items": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_bytes": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_fsck_items": 0, 2020-02-05T20:03:05.202 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_bytes": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_txc_items": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_bytes": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_deferred_items": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_bytes": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluestore_writing_items": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_bytes": 0, 2020-02-05T20:03:05.203 INFO:teuthology.orchestra.run.smithi164.stdout: "bluefs_items": 0, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_bytes": 70205856, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_anon_items": 6075, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_bytes": 139832, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "buffer_meta_items": 1589, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_bytes": 0, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_items": 0, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_bytes": 0, 2020-02-05T20:03:05.204 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_mapbl_items": 0, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_bytes": 0, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_pglog_items": 0, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_bytes": 5048, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_items": 75, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_bytes": 0, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap_mapping_items": 0, 2020-02-05T20:03:05.205 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_bytes": 0, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "pgmap_items": 0, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_bytes": 221140049, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "mds_co_items": 1744545, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_bytes": 0, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_1_items": 0, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_bytes": 0, 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: "unittest_2_items": 0 2020-02-05T20:03:05.206 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "objecter": { 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_active": 0, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_laggy": 0, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send": 45751, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_send_bytes": 354642909, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_resend": 31, 2020-02-05T20:03:05.207 INFO:teuthology.orchestra.run.smithi164.stdout: "op_reply": 45751, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "op": 45751, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "op_r": 602, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "op_w": 45149, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "op_rmw": 0, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "op_pg": 0, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_stat": 416, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_create": 31169, 2020-02-05T20:03:05.208 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_read": 602, 2020-02-05T20:03:05.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_write": 8365, 2020-02-05T20:03:05.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writefull": 751, 2020-02-05T20:03:05.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_writesame": 0, 2020-02-05T20:03:05.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_append": 0, 2020-02-05T20:03:05.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_zero": 0, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_truncate": 97, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_delete": 914, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_mapext": 0, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_sparse_read": 0, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_clonerange": 0, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_getxattr": 0, 2020-02-05T20:03:05.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_setxattr": 62338, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_cmpxattr": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_rmxattr": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_resetxattrs": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_call": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_watch": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_notify": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_src_cmpxattr": 0, 2020-02-05T20:03:05.211 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_pgls_filter": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "osdop_other": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_active": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_send": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_resend": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "linger_ping": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_active": 0, 2020-02-05T20:03:05.212 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_send": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "poolop_resend": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_active": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_send": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "poolstat_resend": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_active": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_send": 0, 2020-02-05T20:03:05.213 INFO:teuthology.orchestra.run.smithi164.stdout: "statfs_resend": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "command_active": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "command_send": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "command_resend": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "map_epoch": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "map_full": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "map_inc": 0, 2020-02-05T20:03:05.214 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_sessions": 8, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_open": 72, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_session_close": 64, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "osd_laggy": 0, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_wr": 7665, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_rd": 0, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "omap_del": 649 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.215 INFO:teuthology.orchestra.run.smithi164.stdout: "purge_queue": { 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops": 0, 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_ops_high_water": 4, 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing": 0, 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executing_high_water": 2, 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_executed": 785, 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: "pq_item_in_journal": 0 2020-02-05T20:03:05.216 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-msgr_dispatch_throttler-mds": { 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 284748, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 131690931, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:05.217 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 284748, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 284748, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 131690931, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.218 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_bytes": { 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 104857600, 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2020-02-05T20:03:05.219 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2020-02-05T20:03:05.220 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2020-02-05T20:03:05.220 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:05.220 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 45751, 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 354722194, 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 44740, 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 354722194, 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2020-02-05T20:03:05.223 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-objecter_ops": { 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2020-02-05T20:03:05.224 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 1024, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 0, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 0, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 0, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 45751, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 45751, 2020-02-05T20:03:05.225 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 45751, 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 45751, 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.226 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle": { 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 0, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 785, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 79285, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:05.227 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 785, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 602, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 79285, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.228 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: "throttle-write_buf_throttle-0x55b2d6475ae0": { 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 599, 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: "max": 3758096384, 2020-02-05T20:03:05.229 INFO:teuthology.orchestra.run.smithi164.stdout: "get_started": 0, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "get": 156134, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "get_sum": 307437993, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_fail": 0, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "get_or_fail_success": 156134, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "take": 0, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "take_sum": 0, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "put": 7763, 2020-02-05T20:03:05.230 INFO:teuthology.orchestra.run.smithi164.stdout: "put_sum": 307437394, 2020-02-05T20:03:05.231 INFO:teuthology.orchestra.run.smithi164.stdout: "wait": { 2020-02-05T20:03:05.231 INFO:teuthology.orchestra.run.smithi164.stdout: "avgcount": 0, 2020-02-05T20:03:05.231 INFO:teuthology.orchestra.run.smithi164.stdout: "sum": 0.000000000, 2020-02-05T20:03:05.232 INFO:teuthology.orchestra.run.smithi164.stdout: "avgtime": 0.000000000 2020-02-05T20:03:05.232 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.232 INFO:teuthology.orchestra.run.smithi164.stdout: } 2020-02-05T20:03:05.232 INFO:teuthology.orchestra.run.smithi164.stdout:} 2020-02-05T20:03:05.234 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=113 2020-02-05T20:03:05.234 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2020-02-05T20:03:05.278 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2020-02-05T20:03:05.278 INFO:teuthology.orchestra.run:Running command with timeout 900 2020-02-05T20:03:05.279 INFO:teuthology.orchestra.run.smithi008:> stat --file-system '--printf=%T 2020-02-05T20:03:05.279 INFO:teuthology.orchestra.run.smithi008:> ' -- /home/ubuntu/cephtest/mnt.0 2020-02-05T20:03:05.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:05.285+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.287042+0000) 2020-02-05T20:03:05.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:05.287+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.288738+0000) 2020-02-05T20:03:05.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:05.288+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.289557+0000) 2020-02-05T20:03:05.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:05.288+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.289835+0000) 2020-02-05T20:03:05.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:05.289+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.290318+0000) 2020-02-05T20:03:05.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:05.289+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.290728+0000) 2020-02-05T20:03:05.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:05.289+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.290991+0000) 2020-02-05T20:03:05.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:05.289+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.290876+0000) 2020-02-05T20:03:05.293 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:05.290+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.291208+0000) 2020-02-05T20:03:05.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:05.290+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.291895+0000) 2020-02-05T20:03:05.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:05.290+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.291822+0000) 2020-02-05T20:03:05.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:05.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:05.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:05.291+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.292172+0000) 2020-02-05T20:03:05.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:05.291+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.292750+0000) 2020-02-05T20:03:05.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:05.295+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.296629+0000) 2020-02-05T20:03:05.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:05.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.762178+0000) 2020-02-05T20:03:05.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:05.985+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:05.986368+0000) 2020-02-05T20:03:06.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:06.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:06.621966+0000) 2020-02-05T20:03:06.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:06.629+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:06.630293+0000) 2020-02-05T20:03:06.632 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:06.629+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:06.630449+0000) 2020-02-05T20:03:06.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:06.860+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:06.861373+0000) 2020-02-05T20:03:08.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:08.601+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:08.602476+0000) 2020-02-05T20:03:08.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:08.973+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:08.974020+0000) 2020-02-05T20:03:09.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:09.260+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:09.260518+0000) 2020-02-05T20:03:09.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:09.869+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:09.870130+0000) 2020-02-05T20:03:10.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:10.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.110456+0000) 2020-02-05T20:03:10.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:10.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.110693+0000) 2020-02-05T20:03:10.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:10.297+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.299051+0000) 2020-02-05T20:03:10.301 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:10.299+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.300311+0000) 2020-02-05T20:03:10.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:10.299+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.300655+0000) 2020-02-05T20:03:10.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:10.299+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.301212+0000) 2020-02-05T20:03:10.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:10.301+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.302165+0000) 2020-02-05T20:03:10.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:10.301+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.302720+0000) 2020-02-05T20:03:10.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:10.301+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.303062+0000) 2020-02-05T20:03:10.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:10.302+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.303445+0000) 2020-02-05T20:03:10.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:10.301+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.303019+0000) 2020-02-05T20:03:10.305 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:10.302+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:10.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:10.303+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.304150+0000) 2020-02-05T20:03:10.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:10.303+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.304443+0000) 2020-02-05T20:03:10.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:10.303+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.305085+0000) 2020-02-05T20:03:10.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:10.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.307432+0000) 2020-02-05T20:03:10.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:10.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:10.307741+0000) 2020-02-05T20:03:12.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:12.219+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:12.220764+0000) 2020-02-05T20:03:12.728 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:03:12.734 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:03:13.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:13.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:13.345992+0000) 2020-02-05T20:03:15.311 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:15.309+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.310161+0000) 2020-02-05T20:03:15.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:15.309+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.310401+0000) 2020-02-05T20:03:15.313 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:15.310+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.311609+0000) 2020-02-05T20:03:15.313 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:15.311+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.312333+0000) 2020-02-05T20:03:15.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:15.311+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.312903+0000) 2020-02-05T20:03:15.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:15.311+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.313160+0000) 2020-02-05T20:03:15.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:15.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:15.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:15.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.314273+0000) 2020-02-05T20:03:15.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:15.313+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.314760+0000) 2020-02-05T20:03:15.316 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:15.313+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.314337+0000) 2020-02-05T20:03:15.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:15.315+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.316296+0000) 2020-02-05T20:03:15.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:15.315+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.316715+0000) 2020-02-05T20:03:15.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:15.315+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.316879+0000) 2020-02-05T20:03:15.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:15.316+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.317645+0000) 2020-02-05T20:03:15.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:15.322+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.323617+0000) 2020-02-05T20:03:15.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:15.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.762387+0000) 2020-02-05T20:03:15.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:15.996+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:15.996712+0000) 2020-02-05T20:03:16.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:16.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:16.622164+0000) 2020-02-05T20:03:16.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:16.630+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:16.630559+0000) 2020-02-05T20:03:16.632 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:16.630+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:16.630709+0000) 2020-02-05T20:03:16.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:16.860+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:16.861657+0000) 2020-02-05T20:03:18.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:18.601+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:18.602718+0000) 2020-02-05T20:03:18.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:18.974+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:18.974281+0000) 2020-02-05T20:03:19.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:19.260+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:19.260754+0000) 2020-02-05T20:03:19.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:19.870+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:19.870374+0000) 2020-02-05T20:03:20.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:20.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.110700+0000) 2020-02-05T20:03:20.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:20.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.110934+0000) 2020-02-05T20:03:20.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:20.318+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.319382+0000) 2020-02-05T20:03:20.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:20.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.321202+0000) 2020-02-05T20:03:20.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:20.320+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.321735+0000) 2020-02-05T20:03:20.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:20.320+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.321861+0000) 2020-02-05T20:03:20.324 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:20.322+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.323321+0000) 2020-02-05T20:03:20.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:20.322+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.323906+0000) 2020-02-05T20:03:20.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:20.322+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.323891+0000) 2020-02-05T20:03:20.326 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:20.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:20.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:20.323+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.324845+0000) 2020-02-05T20:03:20.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:20.323+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.324913+0000) 2020-02-05T20:03:20.328 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:20.323+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.324504+0000) 2020-02-05T20:03:20.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:20.323+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.325392+0000) 2020-02-05T20:03:20.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:20.324+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.325596+0000) 2020-02-05T20:03:20.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:20.327+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.328622+0000) 2020-02-05T20:03:20.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:20.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:20.329819+0000) 2020-02-05T20:03:22.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:22.220+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:22.221039+0000) 2020-02-05T20:03:23.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:23.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:23.346248+0000) 2020-02-05T20:03:25.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:25.329+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.329262+0000) 2020-02-05T20:03:25.331 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:25.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.329726+0000) 2020-02-05T20:03:25.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:25.329+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.331168+0000) 2020-02-05T20:03:25.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:25.331+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.331461+0000) 2020-02-05T20:03:25.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:25.332+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.332360+0000) 2020-02-05T20:03:25.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:25.332+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.332837+0000) 2020-02-05T20:03:25.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:25.332+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.333513+0000) 2020-02-05T20:03:25.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:25.332+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.333561+0000) 2020-02-05T20:03:25.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:25.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.334155+0000) 2020-02-05T20:03:25.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:25.334+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.334647+0000) 2020-02-05T20:03:25.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:25.333+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.334880+0000) 2020-02-05T20:03:25.336 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:25.333+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:25.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:25.333+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.335301+0000) 2020-02-05T20:03:25.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:25.333+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.335401+0000) 2020-02-05T20:03:25.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:25.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.340231+0000) 2020-02-05T20:03:25.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:25.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:25.762635+0000) 2020-02-05T20:03:26.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:26.003+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:26.004843+0000) 2020-02-05T20:03:26.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:26.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:26.622422+0000) 2020-02-05T20:03:26.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:26.629+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:26.630873+0000) 2020-02-05T20:03:26.632 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:26.629+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:26.630933+0000) 2020-02-05T20:03:26.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:26.861+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:26.861910+0000) 2020-02-05T20:03:28.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:28.602+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:28.602961+0000) 2020-02-05T20:03:28.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:28.973+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:28.974530+0000) 2020-02-05T20:03:29.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:29.259+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:29.260967+0000) 2020-02-05T20:03:29.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:29.869+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:29.870626+0000) 2020-02-05T20:03:30.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:30.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.110979+0000) 2020-02-05T20:03:30.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:30.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.111157+0000) 2020-02-05T20:03:30.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:30.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.340798+0000) 2020-02-05T20:03:30.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:30.340+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.341133+0000) 2020-02-05T20:03:30.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:30.341+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.342346+0000) 2020-02-05T20:03:30.344 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:30.341+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.343172+0000) 2020-02-05T20:03:30.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:30.342+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.344190+0000) 2020-02-05T20:03:30.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:30.342+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.344025+0000) 2020-02-05T20:03:30.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:30.343+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.344632+0000) 2020-02-05T20:03:30.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:30.343+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.345027+0000) 2020-02-05T20:03:30.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:30.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:30.347 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:30.344+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.345929+0000) 2020-02-05T20:03:30.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:30.345+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.346441+0000) 2020-02-05T20:03:30.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:30.345+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.346608+0000) 2020-02-05T20:03:30.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:30.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.347182+0000) 2020-02-05T20:03:30.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:30.345+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.346597+0000) 2020-02-05T20:03:30.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:30.352+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:30.353638+0000) 2020-02-05T20:03:32.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:32.220+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:32.221350+0000) 2020-02-05T20:03:33.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:33.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:33.346504+0000) 2020-02-05T20:03:35.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:35.349+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.350274+0000) 2020-02-05T20:03:35.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:35.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.351413+0000) 2020-02-05T20:03:35.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:35.350+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.351699+0000) 2020-02-05T20:03:35.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:35.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.352671+0000) 2020-02-05T20:03:35.354 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:35.351+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.352967+0000) 2020-02-05T20:03:35.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:35.352+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.353477+0000) 2020-02-05T20:03:35.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:35.352+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.354026+0000) 2020-02-05T20:03:35.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:35.352+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.354391+0000) 2020-02-05T20:03:35.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:35.353+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.354704+0000) 2020-02-05T20:03:35.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:35.353+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.355093+0000) 2020-02-05T20:03:35.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:35.354+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.355468+0000) 2020-02-05T20:03:35.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:35.355+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.356667+0000) 2020-02-05T20:03:35.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:35.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:35.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:35.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.360946+0000) 2020-02-05T20:03:35.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:35.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.362239+0000) 2020-02-05T20:03:35.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:35.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:35.762851+0000) 2020-02-05T20:03:36.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:36.011+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:36.012984+0000) 2020-02-05T20:03:36.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:36.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:36.622651+0000) 2020-02-05T20:03:36.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:36.630+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:36.631131+0000) 2020-02-05T20:03:36.632 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:36.630+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:36.631194+0000) 2020-02-05T20:03:36.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:36.861+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:36.862151+0000) 2020-02-05T20:03:38.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:38.602+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:38.603229+0000) 2020-02-05T20:03:38.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:38.973+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:38.974759+0000) 2020-02-05T20:03:39.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:39.260+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:39.261193+0000) 2020-02-05T20:03:39.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:39.870+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:39.870854+0000) 2020-02-05T20:03:40.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:40.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.111267+0000) 2020-02-05T20:03:40.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:40.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.111478+0000) 2020-02-05T20:03:40.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:40.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.363346+0000) 2020-02-05T20:03:40.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:40.362+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.363684+0000) 2020-02-05T20:03:40.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:40.362+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.363711+0000) 2020-02-05T20:03:40.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:40.362+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.363722+0000) 2020-02-05T20:03:40.366 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:40.362+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.364147+0000) 2020-02-05T20:03:40.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:40.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.364480+0000) 2020-02-05T20:03:40.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:40.363+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.364717+0000) 2020-02-05T20:03:40.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:40.364+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.365462+0000) 2020-02-05T20:03:40.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:40.363+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.365272+0000) 2020-02-05T20:03:40.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:40.364+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.365683+0000) 2020-02-05T20:03:40.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:40.364+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.366024+0000) 2020-02-05T20:03:40.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:40.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.366688+0000) 2020-02-05T20:03:40.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:40.365+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.366841+0000) 2020-02-05T20:03:40.369 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:40.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:40.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:40.373+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:40.374167+0000) 2020-02-05T20:03:42.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:42.221+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:42.221617+0000) 2020-02-05T20:03:42.869 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:03:42.875 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:03:43.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:43.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:43.346735+0000) 2020-02-05T20:03:45.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:45.372+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.373134+0000) 2020-02-05T20:03:45.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:45.371+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.373055+0000) 2020-02-05T20:03:45.375 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:45.372+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.373619+0000) 2020-02-05T20:03:45.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:45.372+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.374134+0000) 2020-02-05T20:03:45.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:45.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.374924+0000) 2020-02-05T20:03:45.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:45.374+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.375070+0000) 2020-02-05T20:03:45.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:45.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.375515+0000) 2020-02-05T20:03:45.377 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:45.374+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.375511+0000) 2020-02-05T20:03:45.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:45.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.375579+0000) 2020-02-05T20:03:45.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:45.374+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.375947+0000) 2020-02-05T20:03:45.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:45.375+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.376613+0000) 2020-02-05T20:03:45.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:45.375+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.376959+0000) 2020-02-05T20:03:45.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:45.376+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.378091+0000) 2020-02-05T20:03:45.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:45.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:45.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:45.383+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.384966+0000) 2020-02-05T20:03:45.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:45.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:45.763066+0000) 2020-02-05T20:03:46.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:46.023+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:46.024554+0000) 2020-02-05T20:03:46.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:46.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:46.622888+0000) 2020-02-05T20:03:46.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:46.630+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:46.631367+0000) 2020-02-05T20:03:46.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:46.630+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:46.631455+0000) 2020-02-05T20:03:46.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:46.861+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:46.862412+0000) 2020-02-05T20:03:48.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:48.602+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:48.603485+0000) 2020-02-05T20:03:48.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:48.974+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:48.974991+0000) 2020-02-05T20:03:49.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:49.260+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:49.261410+0000) 2020-02-05T20:03:49.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:49.870+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:49.871137+0000) 2020-02-05T20:03:50.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:50.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.111545+0000) 2020-02-05T20:03:50.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:50.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.111707+0000) 2020-02-05T20:03:50.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:50.383+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.384371+0000) 2020-02-05T20:03:50.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:50.384+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.384923+0000) 2020-02-05T20:03:50.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:50.384+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.385533+0000) 2020-02-05T20:03:50.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:50.385+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.386655+0000) 2020-02-05T20:03:50.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:50.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.387123+0000) 2020-02-05T20:03:50.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:50.387+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.388043+0000) 2020-02-05T20:03:50.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:50.388+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.388674+0000) 2020-02-05T20:03:50.390 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:50.387+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.388848+0000) 2020-02-05T20:03:50.390 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:50.388+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.389429+0000) 2020-02-05T20:03:50.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:50.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:50.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:50.388+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.389906+0000) 2020-02-05T20:03:50.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:50.389+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.389751+0000) 2020-02-05T20:03:50.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:50.389+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.389936+0000) 2020-02-05T20:03:50.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:50.390+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.390698+0000) 2020-02-05T20:03:50.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:50.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:50.396618+0000) 2020-02-05T20:03:52.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:52.221+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:52.221876+0000) 2020-02-05T20:03:53.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:53.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:53.346995+0000) 2020-02-05T20:03:55.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:55.394+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.395691+0000) 2020-02-05T20:03:55.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:55.395+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.396315+0000) 2020-02-05T20:03:55.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:55.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.397045+0000) 2020-02-05T20:03:55.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:03:55.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.397359+0000) 2020-02-05T20:03:55.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:55.396+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.397869+0000) 2020-02-05T20:03:55.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:03:55.398+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.399120+0000) 2020-02-05T20:03:55.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:03:55.398+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.398503+0000) 2020-02-05T20:03:55.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:55.398+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.399888+0000) 2020-02-05T20:03:55.401 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:03:55.399+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:03:55.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:55.399+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.400506+0000) 2020-02-05T20:03:55.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:55.400+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.400522+0000) 2020-02-05T20:03:55.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:55.400+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.400828+0000) 2020-02-05T20:03:55.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:03:55.400+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.400980+0000) 2020-02-05T20:03:55.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:55.400+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.401532+0000) 2020-02-05T20:03:55.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:55.406+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.407927+0000) 2020-02-05T20:03:55.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:55.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:55.763340+0000) 2020-02-05T20:03:56.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:56.031+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:56.033014+0000) 2020-02-05T20:03:56.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:03:56.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:56.623159+0000) 2020-02-05T20:03:56.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:03:56.631+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:56.631600+0000) 2020-02-05T20:03:56.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:03:56.631+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:56.631726+0000) 2020-02-05T20:03:56.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:03:56.862+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:56.862746+0000) 2020-02-05T20:03:58.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:03:58.603+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:58.603708+0000) 2020-02-05T20:03:58.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:03:58.974+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:58.975279+0000) 2020-02-05T20:03:59.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:03:59.261+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:59.261686+0000) 2020-02-05T20:03:59.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:03:59.870+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:03:59.871372+0000) 2020-02-05T20:04:00.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:00.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.111812+0000) 2020-02-05T20:04:00.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:00.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.111938+0000) 2020-02-05T20:04:00.407 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:00.404+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.405579+0000) 2020-02-05T20:04:00.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:00.406+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.406660+0000) 2020-02-05T20:04:00.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:00.406+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.407726+0000) 2020-02-05T20:04:00.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:00.408+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.408839+0000) 2020-02-05T20:04:00.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:00.408+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.409149+0000) 2020-02-05T20:04:00.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:00.409+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.409885+0000) 2020-02-05T20:04:00.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:00.409+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.410195+0000) 2020-02-05T20:04:00.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:00.410+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.410449+0000) 2020-02-05T20:04:00.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:00.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:00.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:00.410+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.410850+0000) 2020-02-05T20:04:00.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:00.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.411227+0000) 2020-02-05T20:04:00.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:00.410+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.411668+0000) 2020-02-05T20:04:00.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:00.411+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.411912+0000) 2020-02-05T20:04:00.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:00.413+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.414813+0000) 2020-02-05T20:04:00.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:00.414+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:00.415133+0000) 2020-02-05T20:04:02.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:02.221+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:02.222116+0000) 2020-02-05T20:04:03.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:03.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:03.347243+0000) 2020-02-05T20:04:05.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:05.415+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.416479+0000) 2020-02-05T20:04:05.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:05.417+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.418136+0000) 2020-02-05T20:04:05.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:05.419+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.420263+0000) 2020-02-05T20:04:05.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:05.420+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.420524+0000) 2020-02-05T20:04:05.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:05.420+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.420716+0000) 2020-02-05T20:04:05.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:05.420+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.421305+0000) 2020-02-05T20:04:05.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:05.421+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.421945+0000) 2020-02-05T20:04:05.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:05.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45716 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:05.430 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:05.417+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.418552+0000) 2020-02-05T20:04:05.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:05.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.419840+0000) 2020-02-05T20:04:05.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:05.419+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.420290+0000) 2020-02-05T20:04:05.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:05.419+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.420795+0000) 2020-02-05T20:04:05.431 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:05.420+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.421304+0000) 2020-02-05T20:04:05.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:05.427+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.428642+0000) 2020-02-05T20:04:05.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:05.427+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.428975+0000) 2020-02-05T20:04:05.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:05.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:05.763493+0000) 2020-02-05T20:04:06.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:06.041+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:06.041357+0000) 2020-02-05T20:04:06.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:06.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:06.623407+0000) 2020-02-05T20:04:06.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:06.631+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:06.631833+0000) 2020-02-05T20:04:06.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:06.631+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:06.631985+0000) 2020-02-05T20:04:06.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:06.862+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:06.863006+0000) 2020-02-05T20:04:08.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:08.603+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:08.603951+0000) 2020-02-05T20:04:08.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:08.975+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:08.975529+0000) 2020-02-05T20:04:09.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:09.260+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:09.261957+0000) 2020-02-05T20:04:09.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:09.871+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:09.871604+0000) 2020-02-05T20:04:10.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:10.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.112048+0000) 2020-02-05T20:04:10.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:10.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.112205+0000) 2020-02-05T20:04:10.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:10.427+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.427643+0000) 2020-02-05T20:04:10.429 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:10.427+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.428157+0000) 2020-02-05T20:04:10.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:10.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.429102+0000) 2020-02-05T20:04:10.430 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:10.428+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.429681+0000) 2020-02-05T20:04:10.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:10.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.430337+0000) 2020-02-05T20:04:10.432 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:10.430+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.430830+0000) 2020-02-05T20:04:10.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:10.430+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.431421+0000) 2020-02-05T20:04:10.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:10.430+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.432041+0000) 2020-02-05T20:04:10.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:10.431+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.431502+0000) 2020-02-05T20:04:10.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:10.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:10.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:10.432+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.432712+0000) 2020-02-05T20:04:10.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:10.431+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.432291+0000) 2020-02-05T20:04:10.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:10.433+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.434052+0000) 2020-02-05T20:04:10.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:10.433+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.434382+0000) 2020-02-05T20:04:10.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:10.439+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:10.440735+0000) 2020-02-05T20:04:12.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:12.222+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:12.222357+0000) 2020-02-05T20:04:13.023 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:04:13.034 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:04:13.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:13.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:13.347542+0000) 2020-02-05T20:04:15.439 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:15.438+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.438433+0000) 2020-02-05T20:04:15.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:15.439+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.439413+0000) 2020-02-05T20:04:15.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:15.439+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.439633+0000) 2020-02-05T20:04:15.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:15.439+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.439963+0000) 2020-02-05T20:04:15.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:15.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.441508+0000) 2020-02-05T20:04:15.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:15.441+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.441850+0000) 2020-02-05T20:04:15.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:15.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:15.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:15.441+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.442390+0000) 2020-02-05T20:04:15.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:15.442+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.442690+0000) 2020-02-05T20:04:15.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:15.443+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.443117+0000) 2020-02-05T20:04:15.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:15.443+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.443965+0000) 2020-02-05T20:04:15.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:15.444+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.444154+0000) 2020-02-05T20:04:15.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:15.444+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.444794+0000) 2020-02-05T20:04:15.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:15.444+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.444672+0000) 2020-02-05T20:04:15.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:15.448+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.449649+0000) 2020-02-05T20:04:15.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:15.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:15.763851+0000) 2020-02-05T20:04:16.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:16.053+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:16.053659+0000) 2020-02-05T20:04:16.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:16.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:16.623637+0000) 2020-02-05T20:04:16.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:16.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:16.632066+0000) 2020-02-05T20:04:16.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:16.632+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:16.632287+0000) 2020-02-05T20:04:16.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:16.862+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:16.863253+0000) 2020-02-05T20:04:18.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:18.603+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:18.604235+0000) 2020-02-05T20:04:18.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:18.975+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:18.975800+0000) 2020-02-05T20:04:19.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:19.262+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:19.262199+0000) 2020-02-05T20:04:19.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:19.870+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:19.871845+0000) 2020-02-05T20:04:20.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:20.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.112280+0000) 2020-02-05T20:04:20.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:20.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.112488+0000) 2020-02-05T20:04:20.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:20.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.449164+0000) 2020-02-05T20:04:20.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:20.448+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.449382+0000) 2020-02-05T20:04:20.451 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:20.449+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.450149+0000) 2020-02-05T20:04:20.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:20.448+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.449734+0000) 2020-02-05T20:04:20.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:20.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.450576+0000) 2020-02-05T20:04:20.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:20.450+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.451292+0000) 2020-02-05T20:04:20.454 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:20.452+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:20.455 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:20.452+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.453239+0000) 2020-02-05T20:04:20.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:20.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.452582+0000) 2020-02-05T20:04:20.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:20.451+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.452949+0000) 2020-02-05T20:04:20.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:20.452+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.453342+0000) 2020-02-05T20:04:20.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:20.453+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.453598+0000) 2020-02-05T20:04:20.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:20.453+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.453826+0000) 2020-02-05T20:04:20.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:20.453+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.454073+0000) 2020-02-05T20:04:20.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:20.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:20.457444+0000) 2020-02-05T20:04:22.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:22.222+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:22.222644+0000) 2020-02-05T20:04:23.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:23.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:23.347786+0000) 2020-02-05T20:04:25.460 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:25.456+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.458045+0000) 2020-02-05T20:04:25.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:25.457+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.458726+0000) 2020-02-05T20:04:25.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:25.459+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.460342+0000) 2020-02-05T20:04:25.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:25.459+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.460893+0000) 2020-02-05T20:04:25.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:25.460+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.461545+0000) 2020-02-05T20:04:25.463 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:25.461+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.461789+0000) 2020-02-05T20:04:25.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:25.462+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.462656+0000) 2020-02-05T20:04:25.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:25.462+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.463310+0000) 2020-02-05T20:04:25.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:25.462+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.463047+0000) 2020-02-05T20:04:25.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:25.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:25.466 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:25.463+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.464032+0000) 2020-02-05T20:04:25.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:25.463+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.464928+0000) 2020-02-05T20:04:25.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:25.464+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.464847+0000) 2020-02-05T20:04:25.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:25.465+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.466817+0000) 2020-02-05T20:04:25.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:25.467+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.468536+0000) 2020-02-05T20:04:25.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:25.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:25.764136+0000) 2020-02-05T20:04:26.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:26.060+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:26.061654+0000) 2020-02-05T20:04:26.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:26.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:26.623876+0000) 2020-02-05T20:04:26.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:26.631+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:26.632331+0000) 2020-02-05T20:04:26.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:26.631+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:26.632526+0000) 2020-02-05T20:04:26.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:26.863+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:26.863511+0000) 2020-02-05T20:04:28.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:28.603+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:28.604481+0000) 2020-02-05T20:04:28.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:28.975+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:28.976108+0000) 2020-02-05T20:04:29.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:29.261+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:29.262410+0000) 2020-02-05T20:04:29.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:29.871+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:29.872126+0000) 2020-02-05T20:04:30.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:30.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.112510+0000) 2020-02-05T20:04:30.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:30.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.112667+0000) 2020-02-05T20:04:30.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:30.469+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.470321+0000) 2020-02-05T20:04:30.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:30.469+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.470691+0000) 2020-02-05T20:04:30.472 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:30.470+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.471302+0000) 2020-02-05T20:04:30.473 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:30.470+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.471419+0000) 2020-02-05T20:04:30.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:30.470+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.471542+0000) 2020-02-05T20:04:30.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:30.470+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.471782+0000) 2020-02-05T20:04:30.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:30.471+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.472226+0000) 2020-02-05T20:04:30.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:30.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.472595+0000) 2020-02-05T20:04:30.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:30.471+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.472540+0000) 2020-02-05T20:04:30.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:30.471+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.472723+0000) 2020-02-05T20:04:30.475 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:30.471+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.473292+0000) 2020-02-05T20:04:30.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:30.472+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.473947+0000) 2020-02-05T20:04:30.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:30.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:30.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:30.473+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.475203+0000) 2020-02-05T20:04:30.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:30.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:30.476986+0000) 2020-02-05T20:04:32.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:32.221+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:32.222907+0000) 2020-02-05T20:04:33.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:33.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:33.348042+0000) 2020-02-05T20:04:35.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:35.480+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.481645+0000) 2020-02-05T20:04:35.485 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:35.482+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.483067+0000) 2020-02-05T20:04:35.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:35.482+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.483766+0000) 2020-02-05T20:04:35.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:35.479+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.481282+0000) 2020-02-05T20:04:35.486 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:35.481+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.482647+0000) 2020-02-05T20:04:35.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:35.484+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.485538+0000) 2020-02-05T20:04:35.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:35.483+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.484510+0000) 2020-02-05T20:04:35.487 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:35.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.485213+0000) 2020-02-05T20:04:35.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:35.485+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.486220+0000) 2020-02-05T20:04:35.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:35.485+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.486377+0000) 2020-02-05T20:04:35.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:35.485+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.486589+0000) 2020-02-05T20:04:35.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:35.484+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.485820+0000) 2020-02-05T20:04:35.489 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:35.484+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.486174+0000) 2020-02-05T20:04:35.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:35.485+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 45958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:35.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:35.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.491449+0000) 2020-02-05T20:04:35.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:35.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:35.764408+0000) 2020-02-05T20:04:36.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:36.068+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:36.069741+0000) 2020-02-05T20:04:36.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:36.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:36.624138+0000) 2020-02-05T20:04:36.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:36.631+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:36.632557+0000) 2020-02-05T20:04:36.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:36.631+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:36.632765+0000) 2020-02-05T20:04:36.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:36.862+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:36.863855+0000) 2020-02-05T20:04:38.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:38.603+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:38.604671+0000) 2020-02-05T20:04:38.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:38.975+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:38.976306+0000) 2020-02-05T20:04:39.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:39.261+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:39.262666+0000) 2020-02-05T20:04:39.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:39.871+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:39.872386+0000) 2020-02-05T20:04:40.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:40.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.112753+0000) 2020-02-05T20:04:40.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:40.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.112890+0000) 2020-02-05T20:04:40.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:40.492+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.493332+0000) 2020-02-05T20:04:40.495 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:40.492+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.493845+0000) 2020-02-05T20:04:40.495 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:40.493+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.494466+0000) 2020-02-05T20:04:40.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:40.493+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.495059+0000) 2020-02-05T20:04:40.497 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:40.494+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.495905+0000) 2020-02-05T20:04:40.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:40.494+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.495765+0000) 2020-02-05T20:04:40.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:40.495+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.496402+0000) 2020-02-05T20:04:40.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:40.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.496969+0000) 2020-02-05T20:04:40.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:40.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.497299+0000) 2020-02-05T20:04:40.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:40.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46004 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:40.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:40.496+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.497652+0000) 2020-02-05T20:04:40.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:40.496+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.497880+0000) 2020-02-05T20:04:40.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:40.497+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.498485+0000) 2020-02-05T20:04:40.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:40.498+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.499319+0000) 2020-02-05T20:04:40.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:40.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:40.505642+0000) 2020-02-05T20:04:42.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:42.222+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:42.223163+0000) 2020-02-05T20:04:43.178 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:04:43.184 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:04:43.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:43.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:43.348336+0000) 2020-02-05T20:04:45.505 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:45.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.504008+0000) 2020-02-05T20:04:45.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:45.502+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.503792+0000) 2020-02-05T20:04:45.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:45.504+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.505171+0000) 2020-02-05T20:04:45.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:45.505+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.506619+0000) 2020-02-05T20:04:45.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:45.505+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.506989+0000) 2020-02-05T20:04:45.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:45.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.507581+0000) 2020-02-05T20:04:45.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:45.512+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.513308+0000) 2020-02-05T20:04:45.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:45.506+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.508025+0000) 2020-02-05T20:04:45.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:45.507+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.508612+0000) 2020-02-05T20:04:45.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:45.504+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.506257+0000) 2020-02-05T20:04:45.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:45.506+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.507507+0000) 2020-02-05T20:04:45.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:45.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:45.522 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:45.507+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.509194+0000) 2020-02-05T20:04:45.522 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:45.507+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.508829+0000) 2020-02-05T20:04:45.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:45.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.509347+0000) 2020-02-05T20:04:45.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:45.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:45.764620+0000) 2020-02-05T20:04:46.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:46.076+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:46.077837+0000) 2020-02-05T20:04:46.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:46.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:46.624363+0000) 2020-02-05T20:04:46.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:46.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:46.632783+0000) 2020-02-05T20:04:46.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:46.632+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:46.632997+0000) 2020-02-05T20:04:46.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:46.863+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:46.864106+0000) 2020-02-05T20:04:48.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:48.604+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:48.604945+0000) 2020-02-05T20:04:48.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:48.976+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:48.976589+0000) 2020-02-05T20:04:49.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:49.261+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:49.262858+0000) 2020-02-05T20:04:49.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:49.872+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:49.872609+0000) 2020-02-05T20:04:50.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:50.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.113010+0000) 2020-02-05T20:04:50.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:50.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.113124+0000) 2020-02-05T20:04:50.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:50.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.515544+0000) 2020-02-05T20:04:50.517 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:50.515+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.516505+0000) 2020-02-05T20:04:50.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:50.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.516942+0000) 2020-02-05T20:04:50.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:50.518+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.519066+0000) 2020-02-05T20:04:50.529 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:50.518+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.519148+0000) 2020-02-05T20:04:50.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:50.518+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.519519+0000) 2020-02-05T20:04:50.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:50.518+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.520036+0000) 2020-02-05T20:04:50.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:50.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.521104+0000) 2020-02-05T20:04:50.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:50.516+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.517487+0000) 2020-02-05T20:04:50.530 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:50.516+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.518324+0000) 2020-02-05T20:04:50.530 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:50.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.518640+0000) 2020-02-05T20:04:50.531 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:50.518+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.519765+0000) 2020-02-05T20:04:50.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:50.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:50.531 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:50.519+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.520887+0000) 2020-02-05T20:04:50.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:50.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:50.527593+0000) 2020-02-05T20:04:52.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:52.222+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:52.223479+0000) 2020-02-05T20:04:53.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:53.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:53.348593+0000) 2020-02-05T20:04:55.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:55.525+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.526602+0000) 2020-02-05T20:04:55.528 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:55.526+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.527150+0000) 2020-02-05T20:04:55.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:04:55.526+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.527660+0000) 2020-02-05T20:04:55.529 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:55.527+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.528353+0000) 2020-02-05T20:04:55.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:55.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.528765+0000) 2020-02-05T20:04:55.530 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:55.527+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.528696+0000) 2020-02-05T20:04:55.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:55.528+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.529566+0000) 2020-02-05T20:04:55.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:04:55.528+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.529964+0000) 2020-02-05T20:04:55.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:55.528+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.530008+0000) 2020-02-05T20:04:55.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:04:55.529+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.531212+0000) 2020-02-05T20:04:55.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:04:55.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:04:55.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:04:55.530+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.532029+0000) 2020-02-05T20:04:55.535 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:55.531+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.532842+0000) 2020-02-05T20:04:55.535 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:55.531+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.532588+0000) 2020-02-05T20:04:55.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:55.538+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.539424+0000) 2020-02-05T20:04:55.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:55.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:55.764855+0000) 2020-02-05T20:04:56.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:56.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:56.086050+0000) 2020-02-05T20:04:56.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:04:56.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:56.624613+0000) 2020-02-05T20:04:56.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:04:56.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:56.633007+0000) 2020-02-05T20:04:56.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:04:56.632+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:56.633262+0000) 2020-02-05T20:04:56.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:04:56.863+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:56.864344+0000) 2020-02-05T20:04:58.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:04:58.604+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:58.605209+0000) 2020-02-05T20:04:58.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:04:58.976+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:58.976828+0000) 2020-02-05T20:04:59.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:04:59.262+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:59.263126+0000) 2020-02-05T20:04:59.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:04:59.872+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:04:59.872815+0000) 2020-02-05T20:05:00.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:00.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.113258+0000) 2020-02-05T20:05:00.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:00.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.113352+0000) 2020-02-05T20:05:00.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:00.536+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.537758+0000) 2020-02-05T20:05:00.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:00.537+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.538591+0000) 2020-02-05T20:05:00.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:00.538+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.539584+0000) 2020-02-05T20:05:00.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:00.538+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.540385+0000) 2020-02-05T20:05:00.542 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:00.539+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.540679+0000) 2020-02-05T20:05:00.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:00.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:00.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:00.540+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.541269+0000) 2020-02-05T20:05:00.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:00.540+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.541827+0000) 2020-02-05T20:05:00.552 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:00.542+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.543338+0000) 2020-02-05T20:05:00.552 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:00.542+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.543600+0000) 2020-02-05T20:05:00.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:00.540+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.541902+0000) 2020-02-05T20:05:00.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:00.540+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.542319+0000) 2020-02-05T20:05:00.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:00.541+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.542592+0000) 2020-02-05T20:05:00.553 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:00.541+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.543110+0000) 2020-02-05T20:05:00.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:00.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:00.545268+0000) 2020-02-05T20:05:02.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:02.223+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:02.223766+0000) 2020-02-05T20:05:03.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:03.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:03.348856+0000) 2020-02-05T20:05:05.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:05.544+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.545748+0000) 2020-02-05T20:05:05.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:05.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.546617+0000) 2020-02-05T20:05:05.548 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:05.546+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.547199+0000) 2020-02-05T20:05:05.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:05.546+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.548273+0000) 2020-02-05T20:05:05.550 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:05.547+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.549182+0000) 2020-02-05T20:05:05.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:05.548+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.549585+0000) 2020-02-05T20:05:05.557 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:05.548+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.550299+0000) 2020-02-05T20:05:05.557 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:05.550+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.551674+0000) 2020-02-05T20:05:05.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:05.550+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.552365+0000) 2020-02-05T20:05:05.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:05.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:05.558 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:05.550+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.552307+0000) 2020-02-05T20:05:05.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:05.549+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.550429+0000) 2020-02-05T20:05:05.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:05.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.552637+0000) 2020-02-05T20:05:05.559 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:05.549+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.550780+0000) 2020-02-05T20:05:05.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:05.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.561209+0000) 2020-02-05T20:05:05.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:05.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:05.765065+0000) 2020-02-05T20:05:06.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:06.096+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:06.097206+0000) 2020-02-05T20:05:06.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:06.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:06.624851+0000) 2020-02-05T20:05:06.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:06.633+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:06.633269+0000) 2020-02-05T20:05:06.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:06.633+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:06.633490+0000) 2020-02-05T20:05:06.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:06.864+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:06.864641+0000) 2020-02-05T20:05:08.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:08.604+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:08.605499+0000) 2020-02-05T20:05:08.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:08.976+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:08.977061+0000) 2020-02-05T20:05:09.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:09.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:09.263366+0000) 2020-02-05T20:05:09.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:09.873+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:09.873068+0000) 2020-02-05T20:05:10.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:10.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.113507+0000) 2020-02-05T20:05:10.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:10.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.113622+0000) 2020-02-05T20:05:10.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:10.558+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.559824+0000) 2020-02-05T20:05:10.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:10.559+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.560785+0000) 2020-02-05T20:05:10.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:10.559+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.561295+0000) 2020-02-05T20:05:10.566 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:10.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46249 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:10.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:10.559+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.559306+0000) 2020-02-05T20:05:10.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:10.561+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.561889+0000) 2020-02-05T20:05:10.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:10.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.562529+0000) 2020-02-05T20:05:10.567 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:10.562+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.562774+0000) 2020-02-05T20:05:10.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:10.563+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.563389+0000) 2020-02-05T20:05:10.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:10.562+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.563931+0000) 2020-02-05T20:05:10.569 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:10.562+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.564415+0000) 2020-02-05T20:05:10.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:10.562+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.563825+0000) 2020-02-05T20:05:10.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:10.564+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.564793+0000) 2020-02-05T20:05:10.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:10.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.566636+0000) 2020-02-05T20:05:10.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:10.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:10.566880+0000) 2020-02-05T20:05:12.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:12.223+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:12.224000+0000) 2020-02-05T20:05:13.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:13.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:13.349107+0000) 2020-02-05T20:05:13.384 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:05:13.391 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:05:15.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:15.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.569161+0000) 2020-02-05T20:05:15.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:15.568+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.569651+0000) 2020-02-05T20:05:15.571 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:15.569+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.570495+0000) 2020-02-05T20:05:15.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:15.569+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.570854+0000) 2020-02-05T20:05:15.572 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:15.569+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.570982+0000) 2020-02-05T20:05:15.573 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:15.570+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.571463+0000) 2020-02-05T20:05:15.573 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:15.569+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.571200+0000) 2020-02-05T20:05:15.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:15.571+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.572506+0000) 2020-02-05T20:05:15.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:15.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.572646+0000) 2020-02-05T20:05:15.574 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:15.571+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.573391+0000) 2020-02-05T20:05:15.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:15.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:15.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:15.572+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.573966+0000) 2020-02-05T20:05:15.576 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:15.573+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.574540+0000) 2020-02-05T20:05:15.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:15.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.578329+0000) 2020-02-05T20:05:15.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:15.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.578700+0000) 2020-02-05T20:05:15.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:15.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:15.765341+0000) 2020-02-05T20:05:16.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:16.104+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:16.105872+0000) 2020-02-05T20:05:16.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:16.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:16.625065+0000) 2020-02-05T20:05:16.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:16.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:16.633500+0000) 2020-02-05T20:05:16.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:16.632+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:16.633720+0000) 2020-02-05T20:05:16.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:16.864+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:16.864866+0000) 2020-02-05T20:05:18.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:18.605+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:18.605762+0000) 2020-02-05T20:05:18.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:18.976+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:18.977342+0000) 2020-02-05T20:05:19.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:19.262+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:19.263594+0000) 2020-02-05T20:05:19.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:19.872+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:19.873356+0000) 2020-02-05T20:05:20.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:20.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.113751+0000) 2020-02-05T20:05:20.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:20.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.113922+0000) 2020-02-05T20:05:20.579 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:20.576+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.577736+0000) 2020-02-05T20:05:20.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:20.577+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.578441+0000) 2020-02-05T20:05:20.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:20.580+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.581147+0000) 2020-02-05T20:05:20.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:20.581+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.581627+0000) 2020-02-05T20:05:20.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:20.581+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.582074+0000) 2020-02-05T20:05:20.583 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:20.581+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.582576+0000) 2020-02-05T20:05:20.584 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:20.581+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.582814+0000) 2020-02-05T20:05:20.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:20.582+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.582956+0000) 2020-02-05T20:05:20.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:20.584+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.584615+0000) 2020-02-05T20:05:20.586 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:20.584+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.584663+0000) 2020-02-05T20:05:20.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:20.582+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.583980+0000) 2020-02-05T20:05:20.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:20.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:20.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:20.584+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.584907+0000) 2020-02-05T20:05:20.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:20.586+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.587818+0000) 2020-02-05T20:05:20.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:20.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:20.588124+0000) 2020-02-05T20:05:22.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:22.223+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:22.224253+0000) 2020-02-05T20:05:23.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:23.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:23.349346+0000) 2020-02-05T20:05:25.591 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:25.589+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.590156+0000) 2020-02-05T20:05:25.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:25.590+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.591101+0000) 2020-02-05T20:05:25.593 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:25.591+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.592103+0000) 2020-02-05T20:05:25.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:25.591+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.592457+0000) 2020-02-05T20:05:25.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:25.592+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.593329+0000) 2020-02-05T20:05:25.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:25.592+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.593750+0000) 2020-02-05T20:05:25.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:25.593+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.594147+0000) 2020-02-05T20:05:25.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:25.593+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.594490+0000) 2020-02-05T20:05:25.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:25.594+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.595331+0000) 2020-02-05T20:05:25.597 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:25.594+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.596001+0000) 2020-02-05T20:05:25.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:25.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:25.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:25.595+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.595989+0000) 2020-02-05T20:05:25.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:25.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.596366+0000) 2020-02-05T20:05:25.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:25.596+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.596542+0000) 2020-02-05T20:05:25.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:25.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.602197+0000) 2020-02-05T20:05:25.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:25.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:25.765531+0000) 2020-02-05T20:05:26.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:26.112+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:26.113852+0000) 2020-02-05T20:05:26.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:26.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:26.625332+0000) 2020-02-05T20:05:26.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:26.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:26.633720+0000) 2020-02-05T20:05:26.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:26.632+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:26.633910+0000) 2020-02-05T20:05:26.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:26.864+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:26.865116+0000) 2020-02-05T20:05:28.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:28.605+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:28.606039+0000) 2020-02-05T20:05:28.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:28.976+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:28.977585+0000) 2020-02-05T20:05:29.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:29.262+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:29.263811+0000) 2020-02-05T20:05:29.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:29.872+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:29.873600+0000) 2020-02-05T20:05:30.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:30.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.113999+0000) 2020-02-05T20:05:30.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:30.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.114188+0000) 2020-02-05T20:05:30.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:30.601+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.601955+0000) 2020-02-05T20:05:30.604 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:30.601+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.602373+0000) 2020-02-05T20:05:30.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:30.603+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.604741+0000) 2020-02-05T20:05:30.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:30.606+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.607212+0000) 2020-02-05T20:05:30.611 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:30.603+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.605041+0000) 2020-02-05T20:05:30.611 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:30.604+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.605480+0000) 2020-02-05T20:05:30.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:30.604+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.605713+0000) 2020-02-05T20:05:30.612 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:30.604+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.605998+0000) 2020-02-05T20:05:30.612 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:30.602+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.603168+0000) 2020-02-05T20:05:30.612 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:30.603+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.604192+0000) 2020-02-05T20:05:30.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:30.604+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.604956+0000) 2020-02-05T20:05:30.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:30.605+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.605992+0000) 2020-02-05T20:05:30.613 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:30.606+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.606858+0000) 2020-02-05T20:05:30.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:30.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:30.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:30.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:30.612018+0000) 2020-02-05T20:05:32.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:32.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:32.224520+0000) 2020-02-05T20:05:33.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:33.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:33.349609+0000) 2020-02-05T20:05:35.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:35.611+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.612120+0000) 2020-02-05T20:05:35.614 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:35.612+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.613182+0000) 2020-02-05T20:05:35.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:35.613+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.614119+0000) 2020-02-05T20:05:35.615 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:35.613+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.614391+0000) 2020-02-05T20:05:35.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:35.614+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.615517+0000) 2020-02-05T20:05:35.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:35.615+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.616478+0000) 2020-02-05T20:05:35.619 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:35.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.616143+0000) 2020-02-05T20:05:35.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:35.616+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.616720+0000) 2020-02-05T20:05:35.619 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:35.616+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.617072+0000) 2020-02-05T20:05:35.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:35.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.617700+0000) 2020-02-05T20:05:35.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:35.617+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.618607+0000) 2020-02-05T20:05:35.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:35.617+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.618228+0000) 2020-02-05T20:05:35.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:35.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46451 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:35.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:35.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.622120+0000) 2020-02-05T20:05:35.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:35.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.624568+0000) 2020-02-05T20:05:35.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:35.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:35.765755+0000) 2020-02-05T20:05:36.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:36.117+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:36.118500+0000) 2020-02-05T20:05:36.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:36.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:36.625580+0000) 2020-02-05T20:05:36.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:36.632+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:36.633935+0000) 2020-02-05T20:05:36.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:36.633+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:36.634136+0000) 2020-02-05T20:05:36.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:36.865+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:36.865366+0000) 2020-02-05T20:05:38.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:38.606+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:38.606300+0000) 2020-02-05T20:05:38.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:38.977+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:38.977841+0000) 2020-02-05T20:05:39.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:39.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:39.264014+0000) 2020-02-05T20:05:39.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:39.873+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:39.873833+0000) 2020-02-05T20:05:40.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:40.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.114226+0000) 2020-02-05T20:05:40.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:40.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.114416+0000) 2020-02-05T20:05:40.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:40.623+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.624188+0000) 2020-02-05T20:05:40.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:40.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.624476+0000) 2020-02-05T20:05:40.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:40.625+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.625470+0000) 2020-02-05T20:05:40.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:40.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.626835+0000) 2020-02-05T20:05:40.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:40.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.626047+0000) 2020-02-05T20:05:40.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:40.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.626759+0000) 2020-02-05T20:05:40.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:40.625+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.627006+0000) 2020-02-05T20:05:40.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:40.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.627346+0000) 2020-02-05T20:05:40.630 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:40.626+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.627359+0000) 2020-02-05T20:05:40.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:40.627+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.628299+0000) 2020-02-05T20:05:40.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:40.628+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.628424+0000) 2020-02-05T20:05:40.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:40.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.628881+0000) 2020-02-05T20:05:40.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:40.628+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.629301+0000) 2020-02-05T20:05:40.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:40.629+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:40.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:40.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:40.635133+0000) 2020-02-05T20:05:42.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:42.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:42.224770+0000) 2020-02-05T20:05:43.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:43.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:43.349845+0000) 2020-02-05T20:05:43.597 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:05:43.603 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:05:45.636 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:45.634+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.635320+0000) 2020-02-05T20:05:45.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:45.634+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.635939+0000) 2020-02-05T20:05:45.638 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:45.636+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.636535+0000) 2020-02-05T20:05:45.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:45.635+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.636988+0000) 2020-02-05T20:05:45.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:45.637+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.637641+0000) 2020-02-05T20:05:45.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:45.637+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.638190+0000) 2020-02-05T20:05:45.640 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:45.638+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.639256+0000) 2020-02-05T20:05:45.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:45.638+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.639382+0000) 2020-02-05T20:05:45.642 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:45.639+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.639698+0000) 2020-02-05T20:05:45.642 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:45.639+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.639813+0000) 2020-02-05T20:05:45.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:45.640+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.640587+0000) 2020-02-05T20:05:45.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:45.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.640521+0000) 2020-02-05T20:05:45.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:45.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.640752+0000) 2020-02-05T20:05:45.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:45.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46535 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:45.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:45.643+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.644787+0000) 2020-02-05T20:05:45.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:45.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:45.765985+0000) 2020-02-05T20:05:46.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:46.126+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:46.126573+0000) 2020-02-05T20:05:46.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:46.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:46.625835+0000) 2020-02-05T20:05:46.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:46.633+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:46.634178+0000) 2020-02-05T20:05:46.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:46.633+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:46.634364+0000) 2020-02-05T20:05:46.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:46.865+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:46.865646+0000) 2020-02-05T20:05:48.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:48.606+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:48.606557+0000) 2020-02-05T20:05:48.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:48.977+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:48.978073+0000) 2020-02-05T20:05:49.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:49.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:49.264269+0000) 2020-02-05T20:05:49.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:49.873+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:49.874124+0000) 2020-02-05T20:05:50.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:50.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.114487+0000) 2020-02-05T20:05:50.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:50.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.114691+0000) 2020-02-05T20:05:50.649 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:50.647+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.647721+0000) 2020-02-05T20:05:50.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:50.647+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.648603+0000) 2020-02-05T20:05:50.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:50.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.649159+0000) 2020-02-05T20:05:50.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:50.648+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.649677+0000) 2020-02-05T20:05:50.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:50.649+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.650134+0000) 2020-02-05T20:05:50.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:50.649+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.650557+0000) 2020-02-05T20:05:50.652 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:50.650+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.651260+0000) 2020-02-05T20:05:50.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:50.651+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.651928+0000) 2020-02-05T20:05:50.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:50.651+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.652106+0000) 2020-02-05T20:05:50.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:50.651+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.652400+0000) 2020-02-05T20:05:50.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:50.652+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.653625+0000) 2020-02-05T20:05:50.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:50.652+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.653257+0000) 2020-02-05T20:05:50.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:50.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.654189+0000) 2020-02-05T20:05:50.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:50.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:50.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:50.660+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:50.661611+0000) 2020-02-05T20:05:52.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:52.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:52.224999+0000) 2020-02-05T20:05:53.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:53.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:53.350107+0000) 2020-02-05T20:05:55.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:55.657+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.659055+0000) 2020-02-05T20:05:55.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:55.658+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.659312+0000) 2020-02-05T20:05:55.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:55.659+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.660931+0000) 2020-02-05T20:05:55.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:55.660+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.662015+0000) 2020-02-05T20:05:55.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:05:55.661+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.662256+0000) 2020-02-05T20:05:55.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:05:55.662+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.662729+0000) 2020-02-05T20:05:55.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:55.661+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.662897+0000) 2020-02-05T20:05:55.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:55.662+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.663507+0000) 2020-02-05T20:05:55.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:55.663+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.664062+0000) 2020-02-05T20:05:55.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:55.664+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.665146+0000) 2020-02-05T20:05:55.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:55.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.664482+0000) 2020-02-05T20:05:55.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:05:55.664+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.665275+0000) 2020-02-05T20:05:55.669 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:05:55.665+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.665775+0000) 2020-02-05T20:05:55.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:05:55.665+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46617 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:05:55.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:55.668+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.669530+0000) 2020-02-05T20:05:55.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:55.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:55.766220+0000) 2020-02-05T20:05:56.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:56.134+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:56.134655+0000) 2020-02-05T20:05:56.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:05:56.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:56.626044+0000) 2020-02-05T20:05:56.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:05:56.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:56.634383+0000) 2020-02-05T20:05:56.636 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:05:56.634+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:56.634600+0000) 2020-02-05T20:05:56.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:05:56.864+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:56.865902+0000) 2020-02-05T20:05:58.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:05:58.605+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:58.606799+0000) 2020-02-05T20:05:58.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:05:58.977+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:58.978340+0000) 2020-02-05T20:05:59.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:05:59.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:59.264501+0000) 2020-02-05T20:05:59.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:05:59.874+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:05:59.874342+0000) 2020-02-05T20:06:00.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:00.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.114684+0000) 2020-02-05T20:06:00.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:00.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.114887+0000) 2020-02-05T20:06:00.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:00.670+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.672021+0000) 2020-02-05T20:06:00.674 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:00.671+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.672753+0000) 2020-02-05T20:06:00.674 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:00.672+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.673411+0000) 2020-02-05T20:06:00.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:00.673+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.675177+0000) 2020-02-05T20:06:00.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:00.674+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.675517+0000) 2020-02-05T20:06:00.677 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:00.675+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.676099+0000) 2020-02-05T20:06:00.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:00.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.676270+0000) 2020-02-05T20:06:00.678 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:00.675+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.676584+0000) 2020-02-05T20:06:00.678 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:00.675+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.676705+0000) 2020-02-05T20:06:00.679 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:00.675+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.677223+0000) 2020-02-05T20:06:00.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:00.676+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.677566+0000) 2020-02-05T20:06:00.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:00.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:00.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:00.676+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.677269+0000) 2020-02-05T20:06:00.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:00.679+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.680857+0000) 2020-02-05T20:06:00.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:00.680+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:00.681094+0000) 2020-02-05T20:06:02.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:02.223+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:02.225242+0000) 2020-02-05T20:06:03.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:03.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:03.350356+0000) 2020-02-05T20:06:05.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:05.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.682500+0000) 2020-02-05T20:06:05.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:05.683+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.684643+0000) 2020-02-05T20:06:05.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:05.683+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.685168+0000) 2020-02-05T20:06:05.686 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:05.685+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.685519+0000) 2020-02-05T20:06:05.687 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:05.684+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.686137+0000) 2020-02-05T20:06:05.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:05.686+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.686337+0000) 2020-02-05T20:06:05.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:05.686+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.686704+0000) 2020-02-05T20:06:05.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:05.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.686754+0000) 2020-02-05T20:06:05.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:05.686+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.687005+0000) 2020-02-05T20:06:05.689 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:05.686+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.687728+0000) 2020-02-05T20:06:05.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:05.686+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.687879+0000) 2020-02-05T20:06:05.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:05.685+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.687331+0000) 2020-02-05T20:06:05.690 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:05.687+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.687605+0000) 2020-02-05T20:06:05.690 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:05.686+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46701 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:05.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:05.696+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.696740+0000) 2020-02-05T20:06:05.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:05.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:05.766457+0000) 2020-02-05T20:06:06.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:06.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:06.142629+0000) 2020-02-05T20:06:06.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:06.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:06.626277+0000) 2020-02-05T20:06:06.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:06.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:06.634593+0000) 2020-02-05T20:06:06.636 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:06.634+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:06.634847+0000) 2020-02-05T20:06:06.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:06.865+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:06.866173+0000) 2020-02-05T20:06:08.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:08.606+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:08.607060+0000) 2020-02-05T20:06:08.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:08.977+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:08.978573+0000) 2020-02-05T20:06:09.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:09.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:09.264781+0000) 2020-02-05T20:06:09.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:09.873+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:09.874570+0000) 2020-02-05T20:06:10.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:10.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.114939+0000) 2020-02-05T20:06:10.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:10.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.115096+0000) 2020-02-05T20:06:10.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:10.692+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.694167+0000) 2020-02-05T20:06:10.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:10.694+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.695385+0000) 2020-02-05T20:06:10.697 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:10.693+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.695248+0000) 2020-02-05T20:06:10.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:10.695+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.696211+0000) 2020-02-05T20:06:10.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:10.695+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.696633+0000) 2020-02-05T20:06:10.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:10.696+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.697365+0000) 2020-02-05T20:06:10.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:10.696+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.697720+0000) 2020-02-05T20:06:10.700 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:10.696+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.698228+0000) 2020-02-05T20:06:10.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:10.698+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.699125+0000) 2020-02-05T20:06:10.701 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:10.697+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.699353+0000) 2020-02-05T20:06:10.701 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:10.698+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.699738+0000) 2020-02-05T20:06:10.701 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:10.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:10.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:10.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.700245+0000) 2020-02-05T20:06:10.702 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:10.699+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.700462+0000) 2020-02-05T20:06:10.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:10.704+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:10.705467+0000) 2020-02-05T20:06:12.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:12.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:12.225506+0000) 2020-02-05T20:06:13.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:13.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:13.350710+0000) 2020-02-05T20:06:13.729 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:06:13.735 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:06:15.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:15.704+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.705621+0000) 2020-02-05T20:06:15.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:15.705+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.706311+0000) 2020-02-05T20:06:15.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:15.705+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.706683+0000) 2020-02-05T20:06:15.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:15.705+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.707231+0000) 2020-02-05T20:06:15.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:15.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.707949+0000) 2020-02-05T20:06:15.710 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:15.707+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.708784+0000) 2020-02-05T20:06:15.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:15.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.709615+0000) 2020-02-05T20:06:15.711 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:15.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46783 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:15.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:15.708+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.709960+0000) 2020-02-05T20:06:15.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:15.709+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.710636+0000) 2020-02-05T20:06:15.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:15.710+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.711806+0000) 2020-02-05T20:06:15.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:15.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.711329+0000) 2020-02-05T20:06:15.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:15.710+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.712151+0000) 2020-02-05T20:06:15.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:15.711+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.713201+0000) 2020-02-05T20:06:15.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:15.717+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.718531+0000) 2020-02-05T20:06:15.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:15.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:15.766676+0000) 2020-02-05T20:06:16.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:16.149+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:16.150807+0000) 2020-02-05T20:06:16.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:16.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:16.626500+0000) 2020-02-05T20:06:16.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:16.633+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:16.634806+0000) 2020-02-05T20:06:16.636 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:16.633+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:16.635015+0000) 2020-02-05T20:06:16.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:16.865+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:16.866424+0000) 2020-02-05T20:06:18.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:18.605+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:18.607305+0000) 2020-02-05T20:06:18.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:18.977+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:18.978842+0000) 2020-02-05T20:06:19.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:19.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:19.265015+0000) 2020-02-05T20:06:19.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:19.873+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:19.874805+0000) 2020-02-05T20:06:20.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:20.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.115197+0000) 2020-02-05T20:06:20.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:20.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.115307+0000) 2020-02-05T20:06:20.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:20.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.714484+0000) 2020-02-05T20:06:20.716 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:20.713+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.715255+0000) 2020-02-05T20:06:20.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:20.714+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.715624+0000) 2020-02-05T20:06:20.717 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:20.715+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.716178+0000) 2020-02-05T20:06:20.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:20.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.718184+0000) 2020-02-05T20:06:20.719 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:20.717+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.718681+0000) 2020-02-05T20:06:20.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:20.717+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.718869+0000) 2020-02-05T20:06:20.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:20.719+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.720320+0000) 2020-02-05T20:06:20.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:20.719+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.720639+0000) 2020-02-05T20:06:20.722 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:20.719+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.721021+0000) 2020-02-05T20:06:20.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:20.720+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.721147+0000) 2020-02-05T20:06:20.723 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:20.719+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.721371+0000) 2020-02-05T20:06:20.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:20.720+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.721668+0000) 2020-02-05T20:06:20.723 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:20.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46829 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:20.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:20.724+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:20.726037+0000) 2020-02-05T20:06:22.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:22.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:22.225763+0000) 2020-02-05T20:06:23.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:23.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:23.350994+0000) 2020-02-05T20:06:25.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:25.725+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.726991+0000) 2020-02-05T20:06:25.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:25.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.727744+0000) 2020-02-05T20:06:25.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:25.727+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.728454+0000) 2020-02-05T20:06:25.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:25.727+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.728944+0000) 2020-02-05T20:06:25.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:25.728+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.730163+0000) 2020-02-05T20:06:25.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:25.730+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.731865+0000) 2020-02-05T20:06:25.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:25.730+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.731990+0000) 2020-02-05T20:06:25.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:25.731+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.732273+0000) 2020-02-05T20:06:25.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:25.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:25.734 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:25.731+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.732723+0000) 2020-02-05T20:06:25.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:25.732+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.733215+0000) 2020-02-05T20:06:25.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:25.732+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.733634+0000) 2020-02-05T20:06:25.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:25.732+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.733653+0000) 2020-02-05T20:06:25.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:25.732+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.734160+0000) 2020-02-05T20:06:25.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:25.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.741124+0000) 2020-02-05T20:06:25.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:25.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:25.766872+0000) 2020-02-05T20:06:26.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:26.158+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:26.159134+0000) 2020-02-05T20:06:26.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:26.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:26.626713+0000) 2020-02-05T20:06:26.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:26.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:26.635018+0000) 2020-02-05T20:06:26.636 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:26.634+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:26.635250+0000) 2020-02-05T20:06:26.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:26.866+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:26.866724+0000) 2020-02-05T20:06:28.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:28.606+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:28.607537+0000) 2020-02-05T20:06:28.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:28.978+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:28.979070+0000) 2020-02-05T20:06:29.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:29.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:29.265311+0000) 2020-02-05T20:06:29.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:29.874+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:29.875036+0000) 2020-02-05T20:06:30.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:30.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.115492+0000) 2020-02-05T20:06:30.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:30.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.115519+0000) 2020-02-05T20:06:30.739 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:30.736+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.737960+0000) 2020-02-05T20:06:30.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:30.737+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.738658+0000) 2020-02-05T20:06:30.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:30.737+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.739351+0000) 2020-02-05T20:06:30.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:30.739+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.740143+0000) 2020-02-05T20:06:30.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:30.740+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.741095+0000) 2020-02-05T20:06:30.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:30.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.741445+0000) 2020-02-05T20:06:30.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:30.740+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.741677+0000) 2020-02-05T20:06:30.743 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:30.740+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.742174+0000) 2020-02-05T20:06:30.744 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:30.740+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.742312+0000) 2020-02-05T20:06:30.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:30.742+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.743622+0000) 2020-02-05T20:06:30.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:30.742+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.743981+0000) 2020-02-05T20:06:30.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:30.742+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.744272+0000) 2020-02-05T20:06:30.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:30.743+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:30.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:30.743+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.744328+0000) 2020-02-05T20:06:30.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:30.748+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:30.749703+0000) 2020-02-05T20:06:32.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:32.225+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:32.226024+0000) 2020-02-05T20:06:33.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:33.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:33.351231+0000) 2020-02-05T20:06:35.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:35.749+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.750146+0000) 2020-02-05T20:06:35.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:35.749+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.750700+0000) 2020-02-05T20:06:35.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:35.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.751777+0000) 2020-02-05T20:06:35.753 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:35.750+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.751953+0000) 2020-02-05T20:06:35.753 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:35.751+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.752175+0000) 2020-02-05T20:06:35.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:35.752+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.753173+0000) 2020-02-05T20:06:35.754 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:35.752+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.753509+0000) 2020-02-05T20:06:35.755 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:35.752+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.753589+0000) 2020-02-05T20:06:35.756 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:35.753+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.754506+0000) 2020-02-05T20:06:35.756 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:35.753+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.754788+0000) 2020-02-05T20:06:35.756 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:35.753+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.754905+0000) 2020-02-05T20:06:35.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:35.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:35.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:35.754+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.755596+0000) 2020-02-05T20:06:35.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:35.754+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.756005+0000) 2020-02-05T20:06:35.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:35.761+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.762876+0000) 2020-02-05T20:06:35.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:35.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:35.767017+0000) 2020-02-05T20:06:36.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:36.166+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:36.167352+0000) 2020-02-05T20:06:36.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:36.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:36.626948+0000) 2020-02-05T20:06:36.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:36.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:36.635279+0000) 2020-02-05T20:06:36.636 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:36.634+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:36.635468+0000) 2020-02-05T20:06:36.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:36.866+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:36.866961+0000) 2020-02-05T20:06:38.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:38.607+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:38.607796+0000) 2020-02-05T20:06:38.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:38.978+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:38.979354+0000) 2020-02-05T20:06:39.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:39.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:39.265543+0000) 2020-02-05T20:06:39.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:39.874+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:39.875318+0000) 2020-02-05T20:06:40.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:40.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.115798+0000) 2020-02-05T20:06:40.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:40.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.115798+0000) 2020-02-05T20:06:40.762 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:40.759+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.760837+0000) 2020-02-05T20:06:40.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:40.760+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.761822+0000) 2020-02-05T20:06:40.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:40.761+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.762474+0000) 2020-02-05T20:06:40.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:40.761+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.763044+0000) 2020-02-05T20:06:40.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:40.763+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.764490+0000) 2020-02-05T20:06:40.766 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:40.763+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.764614+0000) 2020-02-05T20:06:40.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:40.763+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.764771+0000) 2020-02-05T20:06:40.770 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:40.765+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.766237+0000) 2020-02-05T20:06:40.771 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:40.764+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.765708+0000) 2020-02-05T20:06:40.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:40.764+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.765877+0000) 2020-02-05T20:06:40.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:40.765+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.767115+0000) 2020-02-05T20:06:40.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:40.765+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.767264+0000) 2020-02-05T20:06:40.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:40.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 46995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:40.772 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:40.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.767474+0000) 2020-02-05T20:06:40.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:40.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:40.771604+0000) 2020-02-05T20:06:42.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:42.224+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:42.226245+0000) 2020-02-05T20:06:43.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:43.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:43.351491+0000) 2020-02-05T20:06:43.925 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:06:43.933 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:06:45.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:45.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.767216+0000) 2020-02-05T20:06:45.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:45.772+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.773652+0000) 2020-02-05T20:06:45.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:45.773+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.774326+0000) 2020-02-05T20:06:45.776 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:45.773+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.774913+0000) 2020-02-05T20:06:45.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:45.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.775436+0000) 2020-02-05T20:06:45.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:45.774+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.775637+0000) 2020-02-05T20:06:45.777 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:45.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.776141+0000) 2020-02-05T20:06:45.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:45.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47033 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:45.779 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:45.776+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.777767+0000) 2020-02-05T20:06:45.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:45.776+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.777826+0000) 2020-02-05T20:06:45.780 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:45.777+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.778635+0000) 2020-02-05T20:06:45.780 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:45.777+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.778830+0000) 2020-02-05T20:06:45.781 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:45.777+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.779125+0000) 2020-02-05T20:06:45.781 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:45.777+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.778783+0000) 2020-02-05T20:06:45.781 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:45.778+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.779463+0000) 2020-02-05T20:06:45.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:45.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:45.783691+0000) 2020-02-05T20:06:46.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:46.175+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:46.175469+0000) 2020-02-05T20:06:46.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:46.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:46.627184+0000) 2020-02-05T20:06:46.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:46.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:46.635448+0000) 2020-02-05T20:06:46.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:46.634+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:46.635688+0000) 2020-02-05T20:06:46.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:46.866+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:46.867215+0000) 2020-02-05T20:06:48.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:48.607+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:48.608048+0000) 2020-02-05T20:06:48.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:48.979+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:48.979586+0000) 2020-02-05T20:06:49.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:49.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:49.265775+0000) 2020-02-05T20:06:49.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:49.875+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:49.875550+0000) 2020-02-05T20:06:50.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:50.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.116066+0000) 2020-02-05T20:06:50.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:50.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.116074+0000) 2020-02-05T20:06:50.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:50.783+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.784069+0000) 2020-02-05T20:06:50.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:50.783+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.784620+0000) 2020-02-05T20:06:50.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:50.784+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.785443+0000) 2020-02-05T20:06:50.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:50.785+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.786451+0000) 2020-02-05T20:06:50.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:50.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.786104+0000) 2020-02-05T20:06:50.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:50.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.786808+0000) 2020-02-05T20:06:50.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:50.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.787055+0000) 2020-02-05T20:06:50.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:50.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.787437+0000) 2020-02-05T20:06:50.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:50.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47078 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:50.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:50.788+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.789161+0000) 2020-02-05T20:06:50.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:50.788+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.789228+0000) 2020-02-05T20:06:50.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:50.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.789043+0000) 2020-02-05T20:06:50.791 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:50.788+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.789408+0000) 2020-02-05T20:06:50.792 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:50.789+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.790271+0000) 2020-02-05T20:06:50.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:50.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:50.792918+0000) 2020-02-05T20:06:52.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:52.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:52.226501+0000) 2020-02-05T20:06:53.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:53.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:53.351746+0000) 2020-02-05T20:06:55.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:55.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.767480+0000) 2020-02-05T20:06:55.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:55.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.795345+0000) 2020-02-05T20:06:55.797 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:55.795+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.796434+0000) 2020-02-05T20:06:55.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:06:55.796+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.796900+0000) 2020-02-05T20:06:55.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:55.796+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.797480+0000) 2020-02-05T20:06:55.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:55.796+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.797632+0000) 2020-02-05T20:06:55.799 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:06:55.797+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.798055+0000) 2020-02-05T20:06:55.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:06:55.798+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.798490+0000) 2020-02-05T20:06:55.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:55.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.798692+0000) 2020-02-05T20:06:55.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:55.798+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.799254+0000) 2020-02-05T20:06:55.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:06:55.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:06:55.801 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:06:55.799+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.799738+0000) 2020-02-05T20:06:55.801 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:55.799+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.800237+0000) 2020-02-05T20:06:55.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:55.799+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.800725+0000) 2020-02-05T20:06:55.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:55.800+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.800913+0000) 2020-02-05T20:06:55.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:55.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:55.808103+0000) 2020-02-05T20:06:56.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:56.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:56.183211+0000) 2020-02-05T20:06:56.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:06:56.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:56.627419+0000) 2020-02-05T20:06:56.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:06:56.635+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:56.635662+0000) 2020-02-05T20:06:56.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:06:56.635+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:56.635966+0000) 2020-02-05T20:06:56.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:06:56.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:56.867490+0000) 2020-02-05T20:06:58.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:06:58.607+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:58.608309+0000) 2020-02-05T20:06:58.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:06:58.979+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:58.979836+0000) 2020-02-05T20:06:59.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:06:59.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:59.266005+0000) 2020-02-05T20:06:59.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:06:59.875+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:06:59.875796+0000) 2020-02-05T20:07:00.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:00.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.116362+0000) 2020-02-05T20:07:00.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:00.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.116362+0000) 2020-02-05T20:07:00.806 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:00.804+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.805044+0000) 2020-02-05T20:07:00.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:00.806+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.806911+0000) 2020-02-05T20:07:00.808 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:00.806+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.807302+0000) 2020-02-05T20:07:00.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:00.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.807756+0000) 2020-02-05T20:07:00.809 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:00.807+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.808173+0000) 2020-02-05T20:07:00.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:00.808+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.808470+0000) 2020-02-05T20:07:00.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:00.809+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.809265+0000) 2020-02-05T20:07:00.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:00.808+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.809183+0000) 2020-02-05T20:07:00.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:00.809+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.809747+0000) 2020-02-05T20:07:00.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:00.809+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.810044+0000) 2020-02-05T20:07:00.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:00.810+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.810827+0000) 2020-02-05T20:07:00.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:00.810+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.810838+0000) 2020-02-05T20:07:00.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:00.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:00.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:00.812+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.812827+0000) 2020-02-05T20:07:00.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:00.814+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:00.814220+0000) 2020-02-05T20:07:02.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:02.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:02.226771+0000) 2020-02-05T20:07:03.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:03.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:03.351997+0000) 2020-02-05T20:07:05.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:05.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.767756+0000) 2020-02-05T20:07:05.818 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:05.815+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.816999+0000) 2020-02-05T20:07:05.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:05.818+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.818793+0000) 2020-02-05T20:07:05.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:05.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.819726+0000) 2020-02-05T20:07:05.821 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:05.818+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.819414+0000) 2020-02-05T20:07:05.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:05.819+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.820200+0000) 2020-02-05T20:07:05.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:05.819+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.820445+0000) 2020-02-05T20:07:05.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:05.819+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.820814+0000) 2020-02-05T20:07:05.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:05.820+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.821495+0000) 2020-02-05T20:07:05.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:05.821+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.821767+0000) 2020-02-05T20:07:05.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:05.821+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.822069+0000) 2020-02-05T20:07:05.824 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:05.822+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:05.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:05.822+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.822829+0000) 2020-02-05T20:07:05.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:05.822+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.823350+0000) 2020-02-05T20:07:05.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:05.822+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.823335+0000) 2020-02-05T20:07:05.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:05.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:05.827795+0000) 2020-02-05T20:07:06.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:06.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:06.190190+0000) 2020-02-05T20:07:06.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:06.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:06.627628+0000) 2020-02-05T20:07:06.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:06.634+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:06.635867+0000) 2020-02-05T20:07:06.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:06.635+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:06.636236+0000) 2020-02-05T20:07:06.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:06.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:06.867724+0000) 2020-02-05T20:07:08.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:08.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:08.608632+0000) 2020-02-05T20:07:08.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:08.979+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:08.980130+0000) 2020-02-05T20:07:09.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:09.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:09.266234+0000) 2020-02-05T20:07:09.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:09.875+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:09.876028+0000) 2020-02-05T20:07:10.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:10.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.116686+0000) 2020-02-05T20:07:10.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:10.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.116688+0000) 2020-02-05T20:07:10.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:10.825+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.827055+0000) 2020-02-05T20:07:10.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:10.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.829028+0000) 2020-02-05T20:07:10.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:10.828+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.829505+0000) 2020-02-05T20:07:10.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:10.829+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.830085+0000) 2020-02-05T20:07:10.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:10.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.830349+0000) 2020-02-05T20:07:10.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:10.830+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.831552+0000) 2020-02-05T20:07:10.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:10.831+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.831777+0000) 2020-02-05T20:07:10.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:10.830+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.832001+0000) 2020-02-05T20:07:10.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:10.831+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.832761+0000) 2020-02-05T20:07:10.836 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:10.833+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.833842+0000) 2020-02-05T20:07:10.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:10.833+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:10.837 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:10.833+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.834093+0000) 2020-02-05T20:07:10.837 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:10.833+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.834271+0000) 2020-02-05T20:07:10.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:10.832+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.833378+0000) 2020-02-05T20:07:10.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:10.838+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:10.839725+0000) 2020-02-05T20:07:12.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:12.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:12.227017+0000) 2020-02-05T20:07:13.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:13.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:13.352250+0000) 2020-02-05T20:07:14.105 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:07:14.111 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:07:15.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:15.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.767990+0000) 2020-02-05T20:07:15.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:15.840+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.841031+0000) 2020-02-05T20:07:15.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:15.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.841496+0000) 2020-02-05T20:07:15.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:15.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.843141+0000) 2020-02-05T20:07:15.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:15.841+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.842285+0000) 2020-02-05T20:07:15.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:15.843+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.843859+0000) 2020-02-05T20:07:15.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:15.844+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.844871+0000) 2020-02-05T20:07:15.847 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:15.843+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.844381+0000) 2020-02-05T20:07:15.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:15.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.844597+0000) 2020-02-05T20:07:15.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:15.844+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.845482+0000) 2020-02-05T20:07:15.848 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:15.845+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.845601+0000) 2020-02-05T20:07:15.848 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:15.845+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47281 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:15.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:15.845+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.846044+0000) 2020-02-05T20:07:15.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:15.845+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.846232+0000) 2020-02-05T20:07:15.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:15.845+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.846054+0000) 2020-02-05T20:07:15.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:15.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:15.848012+0000) 2020-02-05T20:07:16.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:16.197+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:16.198388+0000) 2020-02-05T20:07:16.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:16.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:16.627878+0000) 2020-02-05T20:07:16.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:16.635+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:16.636103+0000) 2020-02-05T20:07:16.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:16.635+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:16.636399+0000) 2020-02-05T20:07:16.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:16.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:16.867980+0000) 2020-02-05T20:07:18.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:18.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:18.608892+0000) 2020-02-05T20:07:18.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:18.979+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:18.980362+0000) 2020-02-05T20:07:19.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:19.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:19.266509+0000) 2020-02-05T20:07:19.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:19.875+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:19.876296+0000) 2020-02-05T20:07:20.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:20.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.116979+0000) 2020-02-05T20:07:20.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:20.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.116979+0000) 2020-02-05T20:07:20.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:20.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.851964+0000) 2020-02-05T20:07:20.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:20.851+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.852452+0000) 2020-02-05T20:07:20.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:20.852+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.853307+0000) 2020-02-05T20:07:20.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:20.853+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.853631+0000) 2020-02-05T20:07:20.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:20.853+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.854579+0000) 2020-02-05T20:07:20.856 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:20.854+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.854611+0000) 2020-02-05T20:07:20.857 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:20.854+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.855218+0000) 2020-02-05T20:07:20.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:20.854+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.855975+0000) 2020-02-05T20:07:20.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:20.855+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.856524+0000) 2020-02-05T20:07:20.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:20.855+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.856176+0000) 2020-02-05T20:07:20.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:20.855+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.857018+0000) 2020-02-05T20:07:20.859 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:20.857+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.857632+0000) 2020-02-05T20:07:20.859 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:20.857+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.857828+0000) 2020-02-05T20:07:20.860 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:20.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:20.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:20.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:20.860329+0000) 2020-02-05T20:07:22.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:22.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:22.227233+0000) 2020-02-05T20:07:23.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:23.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:23.352501+0000) 2020-02-05T20:07:25.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:25.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.768291+0000) 2020-02-05T20:07:25.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:25.863+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.865136+0000) 2020-02-05T20:07:25.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:25.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.865910+0000) 2020-02-05T20:07:25.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:25.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.866517+0000) 2020-02-05T20:07:25.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:25.865+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.866853+0000) 2020-02-05T20:07:25.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:25.866+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.867524+0000) 2020-02-05T20:07:25.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:25.866+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.867836+0000) 2020-02-05T20:07:25.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:25.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.868372+0000) 2020-02-05T20:07:25.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:25.867+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.868594+0000) 2020-02-05T20:07:25.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:25.868+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.869105+0000) 2020-02-05T20:07:25.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:25.868+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.869403+0000) 2020-02-05T20:07:25.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:25.868+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.869806+0000) 2020-02-05T20:07:25.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:25.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47365 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:25.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:25.869+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.870848+0000) 2020-02-05T20:07:25.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:25.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.870299+0000) 2020-02-05T20:07:25.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:25.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:25.878655+0000) 2020-02-05T20:07:26.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:26.205+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:26.206620+0000) 2020-02-05T20:07:26.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:26.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:26.628113+0000) 2020-02-05T20:07:26.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:26.635+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:26.636312+0000) 2020-02-05T20:07:26.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:26.635+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:26.636602+0000) 2020-02-05T20:07:26.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:26.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:26.868240+0000) 2020-02-05T20:07:28.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:28.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:28.609150+0000) 2020-02-05T20:07:28.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:28.980+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:28.980629+0000) 2020-02-05T20:07:29.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:29.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:29.266744+0000) 2020-02-05T20:07:29.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:29.875+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:29.876526+0000) 2020-02-05T20:07:30.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:30.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.117270+0000) 2020-02-05T20:07:30.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:30.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.117271+0000) 2020-02-05T20:07:30.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:30.874+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.875731+0000) 2020-02-05T20:07:30.878 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:30.875+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.877281+0000) 2020-02-05T20:07:30.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:30.876+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.877699+0000) 2020-02-05T20:07:30.879 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:30.876+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.877796+0000) 2020-02-05T20:07:30.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:30.877+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.878188+0000) 2020-02-05T20:07:30.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:30.877+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.878732+0000) 2020-02-05T20:07:30.880 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:30.876+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.878391+0000) 2020-02-05T20:07:30.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:30.878+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.879455+0000) 2020-02-05T20:07:30.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:30.878+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.880042+0000) 2020-02-05T20:07:30.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:30.878+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.880013+0000) 2020-02-05T20:07:30.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:30.879+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.880628+0000) 2020-02-05T20:07:30.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:30.879+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.880776+0000) 2020-02-05T20:07:30.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:30.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47410 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:30.883 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:30.880+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.881536+0000) 2020-02-05T20:07:30.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:30.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:30.885496+0000) 2020-02-05T20:07:32.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:32.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:32.227479+0000) 2020-02-05T20:07:33.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:33.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:33.352799+0000) 2020-02-05T20:07:35.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:35.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.768536+0000) 2020-02-05T20:07:35.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:35.885+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.886979+0000) 2020-02-05T20:07:35.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:35.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.888009+0000) 2020-02-05T20:07:35.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:35.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.888456+0000) 2020-02-05T20:07:35.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:35.887+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.888749+0000) 2020-02-05T20:07:35.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:35.888+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.889410+0000) 2020-02-05T20:07:35.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:35.888+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.889763+0000) 2020-02-05T20:07:35.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:35.888+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.890053+0000) 2020-02-05T20:07:35.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:35.889+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.890967+0000) 2020-02-05T20:07:35.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:35.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.891415+0000) 2020-02-05T20:07:35.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:35.890+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.891842+0000) 2020-02-05T20:07:35.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:35.890+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.891616+0000) 2020-02-05T20:07:35.894 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:35.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:35.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:35.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.892198+0000) 2020-02-05T20:07:35.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:35.891+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.892836+0000) 2020-02-05T20:07:35.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:35.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:35.895446+0000) 2020-02-05T20:07:36.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:36.214+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:36.214950+0000) 2020-02-05T20:07:36.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:36.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:36.628319+0000) 2020-02-05T20:07:36.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:36.636+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:36.636517+0000) 2020-02-05T20:07:36.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:36.636+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:36.636882+0000) 2020-02-05T20:07:36.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:36.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:36.868479+0000) 2020-02-05T20:07:38.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:38.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:38.609395+0000) 2020-02-05T20:07:38.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:38.979+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:38.980896+0000) 2020-02-05T20:07:39.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:39.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:39.266999+0000) 2020-02-05T20:07:39.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:39.876+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:39.876804+0000) 2020-02-05T20:07:40.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:40.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.117568+0000) 2020-02-05T20:07:40.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:40.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.117568+0000) 2020-02-05T20:07:40.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:40.895+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.897012+0000) 2020-02-05T20:07:40.900 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:40.897+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.899128+0000) 2020-02-05T20:07:40.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:40.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.899381+0000) 2020-02-05T20:07:40.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:40.898+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.899866+0000) 2020-02-05T20:07:40.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:40.899+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.900953+0000) 2020-02-05T20:07:40.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:40.900+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.901876+0000) 2020-02-05T20:07:40.904 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:40.901+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.902707+0000) 2020-02-05T20:07:40.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:40.901+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.903193+0000) 2020-02-05T20:07:40.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:40.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47493 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:40.905 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:40.901+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.903322+0000) 2020-02-05T20:07:40.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:40.902+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.903833+0000) 2020-02-05T20:07:40.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:40.902+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.904168+0000) 2020-02-05T20:07:40.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:40.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.906814+0000) 2020-02-05T20:07:40.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:40.906+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.907281+0000) 2020-02-05T20:07:40.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:40.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:40.908555+0000) 2020-02-05T20:07:42.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:42.226+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:42.227779+0000) 2020-02-05T20:07:43.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:43.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:43.353087+0000) 2020-02-05T20:07:44.318 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:07:44.327 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:07:45.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:45.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.768759+0000) 2020-02-05T20:07:45.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:45.909+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.910706+0000) 2020-02-05T20:07:45.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:45.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.911285+0000) 2020-02-05T20:07:45.913 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:45.910+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.911859+0000) 2020-02-05T20:07:45.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:45.910+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.911913+0000) 2020-02-05T20:07:45.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:45.911+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.912508+0000) 2020-02-05T20:07:45.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:45.911+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.912815+0000) 2020-02-05T20:07:45.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:45.911+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.912992+0000) 2020-02-05T20:07:45.915 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:45.912+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.913374+0000) 2020-02-05T20:07:45.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:45.912+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.914161+0000) 2020-02-05T20:07:45.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:45.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.914718+0000) 2020-02-05T20:07:45.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:45.913+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.915006+0000) 2020-02-05T20:07:45.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:45.914+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.915255+0000) 2020-02-05T20:07:45.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:45.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.915018+0000) 2020-02-05T20:07:45.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:45.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:45.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:45.917+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:45.919025+0000) 2020-02-05T20:07:46.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:46.222+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:46.223193+0000) 2020-02-05T20:07:46.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:46.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:46.628483+0000) 2020-02-05T20:07:46.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:46.635+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:46.636775+0000) 2020-02-05T20:07:46.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:46.636+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:46.637067+0000) 2020-02-05T20:07:46.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:46.867+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:46.868648+0000) 2020-02-05T20:07:48.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:48.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:48.609700+0000) 2020-02-05T20:07:48.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:48.980+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:48.981133+0000) 2020-02-05T20:07:49.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:49.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:49.267262+0000) 2020-02-05T20:07:49.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:49.876+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:49.877055+0000) 2020-02-05T20:07:50.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:50.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.117873+0000) 2020-02-05T20:07:50.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:50.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.117875+0000) 2020-02-05T20:07:50.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:50.920+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.921448+0000) 2020-02-05T20:07:50.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:50.921+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.922257+0000) 2020-02-05T20:07:50.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:50.921+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.922798+0000) 2020-02-05T20:07:50.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:50.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.923265+0000) 2020-02-05T20:07:50.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:50.922+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.923832+0000) 2020-02-05T20:07:50.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:50.923+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.924705+0000) 2020-02-05T20:07:50.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:50.923+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.924990+0000) 2020-02-05T20:07:50.927 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:50.924+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.925665+0000) 2020-02-05T20:07:50.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:50.924+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.926048+0000) 2020-02-05T20:07:50.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:50.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:50.928 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:50.926+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.927287+0000) 2020-02-05T20:07:50.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:50.926+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.927314+0000) 2020-02-05T20:07:50.929 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:50.926+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.927710+0000) 2020-02-05T20:07:50.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:50.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.931523+0000) 2020-02-05T20:07:50.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:50.931+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:50.932935+0000) 2020-02-05T20:07:52.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:52.227+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:52.228037+0000) 2020-02-05T20:07:53.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:53.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:53.353350+0000) 2020-02-05T20:07:55.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:55.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.769007+0000) 2020-02-05T20:07:55.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:55.931+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.931672+0000) 2020-02-05T20:07:55.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:55.932+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.932402+0000) 2020-02-05T20:07:55.934 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:55.931+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.933238+0000) 2020-02-05T20:07:55.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:55.932+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.933552+0000) 2020-02-05T20:07:55.936 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:55.935+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.935338+0000) 2020-02-05T20:07:55.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:07:55.933+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.935210+0000) 2020-02-05T20:07:55.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:55.936+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.936397+0000) 2020-02-05T20:07:55.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:55.936+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.936543+0000) 2020-02-05T20:07:55.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:55.936+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.936969+0000) 2020-02-05T20:07:55.938 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:07:55.936+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.937586+0000) 2020-02-05T20:07:55.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:55.937+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.937863+0000) 2020-02-05T20:07:55.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:07:55.936+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.938033+0000) 2020-02-05T20:07:55.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:07:55.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:07:55.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:07:55.936+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.938239+0000) 2020-02-05T20:07:55.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:55.942+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:55.942325+0000) 2020-02-05T20:07:56.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:56.231+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:56.231279+0000) 2020-02-05T20:07:56.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:07:56.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:56.628710+0000) 2020-02-05T20:07:56.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:07:56.636+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:56.636997+0000) 2020-02-05T20:07:56.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:07:56.637+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:56.637282+0000) 2020-02-05T20:07:56.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:07:56.868+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:56.868868+0000) 2020-02-05T20:07:58.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:07:58.608+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:58.609923+0000) 2020-02-05T20:07:58.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:07:58.980+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:58.981364+0000) 2020-02-05T20:07:59.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:07:59.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:59.267500+0000) 2020-02-05T20:07:59.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:07:59.876+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:07:59.877361+0000) 2020-02-05T20:08:00.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:00.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.118149+0000) 2020-02-05T20:08:00.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:00.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.118151+0000) 2020-02-05T20:08:00.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:00.941+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.942919+0000) 2020-02-05T20:08:00.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:00.943+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.944843+0000) 2020-02-05T20:08:00.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:00.944+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.945443+0000) 2020-02-05T20:08:00.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:00.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.946057+0000) 2020-02-05T20:08:00.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:00.945+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.946482+0000) 2020-02-05T20:08:00.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:00.945+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.946988+0000) 2020-02-05T20:08:00.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:00.946+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.947410+0000) 2020-02-05T20:08:00.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:00.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.948242+0000) 2020-02-05T20:08:00.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:00.947+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.949242+0000) 2020-02-05T20:08:00.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:00.948+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.949498+0000) 2020-02-05T20:08:00.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:00.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.950053+0000) 2020-02-05T20:08:00.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:00.948+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.950085+0000) 2020-02-05T20:08:00.952 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:00.948+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.950181+0000) 2020-02-05T20:08:00.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:00.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:00.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:00.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:00.955346+0000) 2020-02-05T20:08:02.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:02.227+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:02.228311+0000) 2020-02-05T20:08:03.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:03.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:03.353612+0000) 2020-02-05T20:08:05.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:05.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.769275+0000) 2020-02-05T20:08:05.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:05.954+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.955135+0000) 2020-02-05T20:08:05.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:05.955+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.957027+0000) 2020-02-05T20:08:05.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:05.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.957601+0000) 2020-02-05T20:08:05.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:05.956+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.957817+0000) 2020-02-05T20:08:05.960 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:05.957+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.958671+0000) 2020-02-05T20:08:05.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:05.957+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.958792+0000) 2020-02-05T20:08:05.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:05.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.958797+0000) 2020-02-05T20:08:05.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:05.958+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.959112+0000) 2020-02-05T20:08:05.961 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:05.959+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.960354+0000) 2020-02-05T20:08:05.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:05.958+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.960317+0000) 2020-02-05T20:08:05.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:05.959+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.960840+0000) 2020-02-05T20:08:05.962 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:05.959+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.960862+0000) 2020-02-05T20:08:05.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:05.959+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.961050+0000) 2020-02-05T20:08:05.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:05.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47690 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:05.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:05.963+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:05.964603+0000) 2020-02-05T20:08:06.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:06.237+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:06.238866+0000) 2020-02-05T20:08:06.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:06.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:06.628975+0000) 2020-02-05T20:08:06.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:06.636+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:06.637267+0000) 2020-02-05T20:08:06.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:06.636+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:06.637491+0000) 2020-02-05T20:08:06.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:06.868+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:06.869109+0000) 2020-02-05T20:08:08.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:08.609+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:08.610191+0000) 2020-02-05T20:08:08.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:08.980+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:08.981596+0000) 2020-02-05T20:08:09.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:09.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:09.267744+0000) 2020-02-05T20:08:09.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:09.876+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:09.877588+0000) 2020-02-05T20:08:10.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:10.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.118474+0000) 2020-02-05T20:08:10.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:10.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.118469+0000) 2020-02-05T20:08:10.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:10.966+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.967262+0000) 2020-02-05T20:08:10.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:10.966+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.967721+0000) 2020-02-05T20:08:10.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:10.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.968118+0000) 2020-02-05T20:08:10.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:10.968+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.969316+0000) 2020-02-05T20:08:10.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:10.970+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.971254+0000) 2020-02-05T20:08:10.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:10.970+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.971647+0000) 2020-02-05T20:08:10.973 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:10.970+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.972064+0000) 2020-02-05T20:08:10.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:10.973+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.974069+0000) 2020-02-05T20:08:10.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:10.972+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.974331+0000) 2020-02-05T20:08:10.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:10.973+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.974583+0000) 2020-02-05T20:08:10.976 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:10.974+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.975323+0000) 2020-02-05T20:08:10.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:10.973+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.974855+0000) 2020-02-05T20:08:10.977 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:10.973+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.975188+0000) 2020-02-05T20:08:10.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:10.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:10.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:10.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:10.979582+0000) 2020-02-05T20:08:12.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:12.228+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:12.228644+0000) 2020-02-05T20:08:13.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:13.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:13.353841+0000) 2020-02-05T20:08:14.539 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:08:14.546 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:08:15.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:15.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.769530+0000) 2020-02-05T20:08:15.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:15.980+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.981470+0000) 2020-02-05T20:08:15.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:15.981+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.982627+0000) 2020-02-05T20:08:15.984 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:15.983+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.983491+0000) 2020-02-05T20:08:15.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:15.982+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.984008+0000) 2020-02-05T20:08:15.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:15.983+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.984393+0000) 2020-02-05T20:08:15.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:15.984+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.984910+0000) 2020-02-05T20:08:15.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:15.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.984969+0000) 2020-02-05T20:08:15.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:15.984+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.985351+0000) 2020-02-05T20:08:15.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:15.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47770 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:15.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:15.985+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.985981+0000) 2020-02-05T20:08:15.988 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:15.985+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.986266+0000) 2020-02-05T20:08:15.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:15.986+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.986533+0000) 2020-02-05T20:08:15.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:15.986+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.987393+0000) 2020-02-05T20:08:15.989 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:15.987+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.987665+0000) 2020-02-05T20:08:15.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:15.992+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:15.993968+0000) 2020-02-05T20:08:16.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:16.246+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:16.247022+0000) 2020-02-05T20:08:16.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:16.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:16.629188+0000) 2020-02-05T20:08:16.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:16.636+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:16.637516+0000) 2020-02-05T20:08:16.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:16.636+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:16.637712+0000) 2020-02-05T20:08:16.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:16.868+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:16.869346+0000) 2020-02-05T20:08:18.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:18.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:18.610476+0000) 2020-02-05T20:08:18.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:18.981+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:18.981831+0000) 2020-02-05T20:08:19.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:19.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:19.267974+0000) 2020-02-05T20:08:19.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:19.877+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:19.877852+0000) 2020-02-05T20:08:20.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:20.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.118780+0000) 2020-02-05T20:08:20.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:20.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.118775+0000) 2020-02-05T20:08:20.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:20.990+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.991736+0000) 2020-02-05T20:08:20.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:20.992+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.993526+0000) 2020-02-05T20:08:20.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:20.993+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.994278+0000) 2020-02-05T20:08:20.995 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:20.993+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.994155+0000) 2020-02-05T20:08:20.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:20.993+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.994855+0000) 2020-02-05T20:08:20.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:20.994+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.995070+0000) 2020-02-05T20:08:20.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:20.994+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.995006+0000) 2020-02-05T20:08:20.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:20.994+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.994953+0000) 2020-02-05T20:08:20.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:20.994+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.995406+0000) 2020-02-05T20:08:20.998 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:20.994+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.995668+0000) 2020-02-05T20:08:20.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:20.995+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.996533+0000) 2020-02-05T20:08:20.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:20.995+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.995775+0000) 2020-02-05T20:08:20.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:20.996+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:20.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:20.996+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:20.996976+0000) 2020-02-05T20:08:21.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:20.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:21.000559+0000) 2020-02-05T20:08:22.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:22.228+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:22.228899+0000) 2020-02-05T20:08:23.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:23.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:23.354065+0000) 2020-02-05T20:08:25.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:25.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:25.769767+0000) 2020-02-05T20:08:26.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:26.001+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.002714+0000) 2020-02-05T20:08:26.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:26.002+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.003242+0000) 2020-02-05T20:08:26.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:26.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.004119+0000) 2020-02-05T20:08:26.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:26.003+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.004261+0000) 2020-02-05T20:08:26.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:26.004+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.004576+0000) 2020-02-05T20:08:26.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:26.003+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.004478+0000) 2020-02-05T20:08:26.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:26.004+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.005211+0000) 2020-02-05T20:08:26.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:26.003+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.004988+0000) 2020-02-05T20:08:26.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:26.005+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.005749+0000) 2020-02-05T20:08:26.008 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:26.005+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.006113+0000) 2020-02-05T20:08:26.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:26.005+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.006223+0000) 2020-02-05T20:08:26.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:26.005+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.006283+0000) 2020-02-05T20:08:26.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:26.006+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.006859+0000) 2020-02-05T20:08:26.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:26.007+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:26.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:26.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.011543+0000) 2020-02-05T20:08:26.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:26.254+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.255155+0000) 2020-02-05T20:08:26.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:26.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.629403+0000) 2020-02-05T20:08:26.639 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:26.637+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.637761+0000) 2020-02-05T20:08:26.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:26.637+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.637922+0000) 2020-02-05T20:08:26.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:26.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:26.869563+0000) 2020-02-05T20:08:28.612 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:28.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:28.610712+0000) 2020-02-05T20:08:28.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:28.981+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:28.982110+0000) 2020-02-05T20:08:29.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:29.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:29.268204+0000) 2020-02-05T20:08:29.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:29.877+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:29.878124+0000) 2020-02-05T20:08:30.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:30.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:30.119082+0000) 2020-02-05T20:08:30.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:30.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:30.119080+0000) 2020-02-05T20:08:31.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:31.011+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.013049+0000) 2020-02-05T20:08:31.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:31.011+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.012722+0000) 2020-02-05T20:08:31.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:31.013+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.014339+0000) 2020-02-05T20:08:31.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:31.013+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.014296+0000) 2020-02-05T20:08:31.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:31.014+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.014677+0000) 2020-02-05T20:08:31.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:31.014+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.015319+0000) 2020-02-05T20:08:31.017 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:31.014+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.015883+0000) 2020-02-05T20:08:31.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:31.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.017018+0000) 2020-02-05T20:08:31.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:31.016+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.016469+0000) 2020-02-05T20:08:31.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:31.016+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.017001+0000) 2020-02-05T20:08:31.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:31.017+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.017531+0000) 2020-02-05T20:08:31.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:31.017+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.017767+0000) 2020-02-05T20:08:31.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:31.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:31.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:31.017+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.018224+0000) 2020-02-05T20:08:31.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:31.021+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:31.022690+0000) 2020-02-05T20:08:32.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:32.228+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:32.229147+0000) 2020-02-05T20:08:33.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:33.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:33.354314+0000) 2020-02-05T20:08:35.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:35.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:35.770028+0000) 2020-02-05T20:08:36.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:36.023+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.024662+0000) 2020-02-05T20:08:36.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:36.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.025144+0000) 2020-02-05T20:08:36.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:36.024+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.025683+0000) 2020-02-05T20:08:36.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:36.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.025841+0000) 2020-02-05T20:08:36.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:36.025+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.026246+0000) 2020-02-05T20:08:36.028 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:36.026+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.026633+0000) 2020-02-05T20:08:36.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:36.026+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.027328+0000) 2020-02-05T20:08:36.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:36.026+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.028018+0000) 2020-02-05T20:08:36.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:36.027+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.027898+0000) 2020-02-05T20:08:36.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:36.027+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.028542+0000) 2020-02-05T20:08:36.031 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:36.029+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.029642+0000) 2020-02-05T20:08:36.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:36.029+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.030254+0000) 2020-02-05T20:08:36.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:36.030+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.030589+0000) 2020-02-05T20:08:36.032 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:36.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47930 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:36.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:36.034+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.035750+0000) 2020-02-05T20:08:36.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:36.262+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.263141+0000) 2020-02-05T20:08:36.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:36.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.629622+0000) 2020-02-05T20:08:36.639 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:36.637+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.637969+0000) 2020-02-05T20:08:36.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:36.637+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.638138+0000) 2020-02-05T20:08:36.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:36.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:36.869774+0000) 2020-02-05T20:08:38.612 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:38.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:38.610961+0000) 2020-02-05T20:08:38.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:38.981+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:38.982334+0000) 2020-02-05T20:08:39.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:39.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:39.268445+0000) 2020-02-05T20:08:39.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:39.878+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:39.878353+0000) 2020-02-05T20:08:40.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:40.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:40.119379+0000) 2020-02-05T20:08:40.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:40.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:40.119379+0000) 2020-02-05T20:08:41.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:41.034+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.036036+0000) 2020-02-05T20:08:41.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:41.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.036663+0000) 2020-02-05T20:08:41.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:41.036+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.037231+0000) 2020-02-05T20:08:41.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:41.036+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.037472+0000) 2020-02-05T20:08:41.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:41.038+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.039039+0000) 2020-02-05T20:08:41.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:41.038+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.039445+0000) 2020-02-05T20:08:41.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:41.038+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.039912+0000) 2020-02-05T20:08:41.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:41.039+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.040330+0000) 2020-02-05T20:08:41.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:41.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 47973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:41.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:41.041+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.041481+0000) 2020-02-05T20:08:41.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:41.041+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.042165+0000) 2020-02-05T20:08:41.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:41.041+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.042627+0000) 2020-02-05T20:08:41.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:41.042+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.042890+0000) 2020-02-05T20:08:41.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:41.042+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.043201+0000) 2020-02-05T20:08:41.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:41.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:41.047378+0000) 2020-02-05T20:08:42.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:42.229+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:42.229383+0000) 2020-02-05T20:08:43.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:43.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:43.354569+0000) 2020-02-05T20:08:44.744 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:08:44.750 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:08:45.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:45.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:45.770336+0000) 2020-02-05T20:08:46.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:46.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.044556+0000) 2020-02-05T20:08:46.047 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:46.045+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.046411+0000) 2020-02-05T20:08:46.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:46.045+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.046911+0000) 2020-02-05T20:08:46.049 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:46.047+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.048419+0000) 2020-02-05T20:08:46.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:46.048+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.049210+0000) 2020-02-05T20:08:46.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:46.048+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.049951+0000) 2020-02-05T20:08:46.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:46.049+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.050405+0000) 2020-02-05T20:08:46.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:46.050+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.050643+0000) 2020-02-05T20:08:46.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:46.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.051776+0000) 2020-02-05T20:08:46.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:46.051+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:46.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:46.051+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.051420+0000) 2020-02-05T20:08:46.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:46.051+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.051480+0000) 2020-02-05T20:08:46.054 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:46.051+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.052283+0000) 2020-02-05T20:08:46.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:46.051+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.052273+0000) 2020-02-05T20:08:46.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:46.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.054774+0000) 2020-02-05T20:08:46.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:46.270+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.270955+0000) 2020-02-05T20:08:46.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:46.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.629836+0000) 2020-02-05T20:08:46.639 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:46.638+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.638184+0000) 2020-02-05T20:08:46.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:46.638+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.638432+0000) 2020-02-05T20:08:46.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:46.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:46.869983+0000) 2020-02-05T20:08:48.612 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:48.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:48.611211+0000) 2020-02-05T20:08:48.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:48.982+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:48.982567+0000) 2020-02-05T20:08:49.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:49.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:49.268692+0000) 2020-02-05T20:08:49.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:49.878+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:49.878582+0000) 2020-02-05T20:08:50.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:50.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:50.119699+0000) 2020-02-05T20:08:50.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:50.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:50.119699+0000) 2020-02-05T20:08:51.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:51.058+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.058556+0000) 2020-02-05T20:08:51.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:51.058+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.058933+0000) 2020-02-05T20:08:51.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:51.059+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.059600+0000) 2020-02-05T20:08:51.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:51.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.060014+0000) 2020-02-05T20:08:51.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:51.060+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.060219+0000) 2020-02-05T20:08:51.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:51.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.060477+0000) 2020-02-05T20:08:51.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:51.060+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.060800+0000) 2020-02-05T20:08:51.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:51.061+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.061560+0000) 2020-02-05T20:08:51.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:51.060+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.061315+0000) 2020-02-05T20:08:51.064 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:51.061+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.061628+0000) 2020-02-05T20:08:51.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:51.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.061852+0000) 2020-02-05T20:08:51.065 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:51.061+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.062360+0000) 2020-02-05T20:08:51.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:51.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:51.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:51.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.062878+0000) 2020-02-05T20:08:51.065 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:51.063+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:51.063930+0000) 2020-02-05T20:08:52.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:52.229+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:52.229717+0000) 2020-02-05T20:08:53.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:53.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:53.354805+0000) 2020-02-05T20:08:55.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:55.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:55.770571+0000) 2020-02-05T20:08:56.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:56.067+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.068632+0000) 2020-02-05T20:08:56.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:56.068+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.069709+0000) 2020-02-05T20:08:56.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:56.068+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.070278+0000) 2020-02-05T20:08:56.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:56.070+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.071300+0000) 2020-02-05T20:08:56.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:08:56.070+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.071654+0000) 2020-02-05T20:08:56.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:56.070+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.071777+0000) 2020-02-05T20:08:56.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:08:56.071+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.072516+0000) 2020-02-05T20:08:56.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:56.071+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.072671+0000) 2020-02-05T20:08:56.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:56.072+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.073631+0000) 2020-02-05T20:08:56.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:56.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.073834+0000) 2020-02-05T20:08:56.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:08:56.072+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.073994+0000) 2020-02-05T20:08:56.076 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:08:56.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:08:56.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:08:56.072+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.074206+0000) 2020-02-05T20:08:56.076 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:56.073+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.074538+0000) 2020-02-05T20:08:56.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:56.077+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.078790+0000) 2020-02-05T20:08:56.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:56.277+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.278969+0000) 2020-02-05T20:08:56.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:08:56.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.630137+0000) 2020-02-05T20:08:56.639 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:08:56.637+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.638396+0000) 2020-02-05T20:08:56.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:08:56.637+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.638773+0000) 2020-02-05T20:08:56.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:08:56.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:56.870204+0000) 2020-02-05T20:08:58.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:08:58.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:58.611495+0000) 2020-02-05T20:08:58.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:08:58.981+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:58.982801+0000) 2020-02-05T20:08:59.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:08:59.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:59.268928+0000) 2020-02-05T20:08:59.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:08:59.877+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:08:59.878851+0000) 2020-02-05T20:09:00.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:00.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:00.119983+0000) 2020-02-05T20:09:00.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:00.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:00.119983+0000) 2020-02-05T20:09:01.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:01.077+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.078964+0000) 2020-02-05T20:09:01.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:01.079+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.080443+0000) 2020-02-05T20:09:01.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:01.079+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.080960+0000) 2020-02-05T20:09:01.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:01.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.081704+0000) 2020-02-05T20:09:01.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:01.080+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.082214+0000) 2020-02-05T20:09:01.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:01.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.082887+0000) 2020-02-05T20:09:01.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:01.081+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.082862+0000) 2020-02-05T20:09:01.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:01.082+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.083339+0000) 2020-02-05T20:09:01.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:01.082+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.083757+0000) 2020-02-05T20:09:01.086 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:01.083+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.084689+0000) 2020-02-05T20:09:01.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:01.083+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.085347+0000) 2020-02-05T20:09:01.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:01.082+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.084371+0000) 2020-02-05T20:09:01.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:01.084+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.085596+0000) 2020-02-05T20:09:01.087 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:01.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:01.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:01.089+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:01.090998+0000) 2020-02-05T20:09:02.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:02.228+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:02.229914+0000) 2020-02-05T20:09:03.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:03.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:03.355083+0000) 2020-02-05T20:09:05.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:05.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:05.770813+0000) 2020-02-05T20:09:06.094 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:06.091+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.092690+0000) 2020-02-05T20:09:06.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:06.092+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.093510+0000) 2020-02-05T20:09:06.095 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:06.092+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.094134+0000) 2020-02-05T20:09:06.095 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:06.093+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.094470+0000) 2020-02-05T20:09:06.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:06.093+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.095308+0000) 2020-02-05T20:09:06.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:06.094+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.095679+0000) 2020-02-05T20:09:06.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:06.094+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.096108+0000) 2020-02-05T20:09:06.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:06.095+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.096316+0000) 2020-02-05T20:09:06.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:06.095+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.096751+0000) 2020-02-05T20:09:06.098 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:06.096+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.097115+0000) 2020-02-05T20:09:06.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:06.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48172 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:06.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:06.096+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.097808+0000) 2020-02-05T20:09:06.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:06.096+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.097759+0000) 2020-02-05T20:09:06.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:06.097+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.098837+0000) 2020-02-05T20:09:06.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:06.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.104682+0000) 2020-02-05T20:09:06.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:06.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.287157+0000) 2020-02-05T20:09:06.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:06.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.630352+0000) 2020-02-05T20:09:06.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:06.637+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.638623+0000) 2020-02-05T20:09:06.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:06.637+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.639002+0000) 2020-02-05T20:09:06.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:06.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:06.870468+0000) 2020-02-05T20:09:08.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:08.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:08.611703+0000) 2020-02-05T20:09:08.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:08.982+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:08.983107+0000) 2020-02-05T20:09:09.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:09.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:09.269194+0000) 2020-02-05T20:09:09.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:09.878+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:09.879118+0000) 2020-02-05T20:09:10.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:10.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:10.120276+0000) 2020-02-05T20:09:10.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:10.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:10.120276+0000) 2020-02-05T20:09:11.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:11.103+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.104564+0000) 2020-02-05T20:09:11.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:11.103+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.105084+0000) 2020-02-05T20:09:11.107 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:11.104+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.105853+0000) 2020-02-05T20:09:11.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:11.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.106058+0000) 2020-02-05T20:09:11.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:11.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.106399+0000) 2020-02-05T20:09:11.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:11.105+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.106610+0000) 2020-02-05T20:09:11.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:11.105+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.106713+0000) 2020-02-05T20:09:11.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:11.106+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.107186+0000) 2020-02-05T20:09:11.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:11.105+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.107066+0000) 2020-02-05T20:09:11.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:11.106+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.108186+0000) 2020-02-05T20:09:11.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:11.107+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.109007+0000) 2020-02-05T20:09:11.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:11.107+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.109291+0000) 2020-02-05T20:09:11.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:11.107+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:11.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:11.108+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.109675+0000) 2020-02-05T20:09:11.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:11.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:11.113859+0000) 2020-02-05T20:09:12.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:12.228+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:12.230162+0000) 2020-02-05T20:09:13.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:13.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:13.355328+0000) 2020-02-05T20:09:14.932 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:09:14.939 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:09:15.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:15.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:15.771053+0000) 2020-02-05T20:09:16.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:16.113+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.114863+0000) 2020-02-05T20:09:16.116 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:16.114+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.115215+0000) 2020-02-05T20:09:16.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:16.114+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.115499+0000) 2020-02-05T20:09:16.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:16.114+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.116076+0000) 2020-02-05T20:09:16.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:16.114+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.116406+0000) 2020-02-05T20:09:16.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:16.115+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.116705+0000) 2020-02-05T20:09:16.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:16.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.117212+0000) 2020-02-05T20:09:16.119 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:16.116+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.117840+0000) 2020-02-05T20:09:16.119 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:16.116+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.118103+0000) 2020-02-05T20:09:16.120 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:16.117+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.118643+0000) 2020-02-05T20:09:16.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:16.116+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.118291+0000) 2020-02-05T20:09:16.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:16.117+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.119260+0000) 2020-02-05T20:09:16.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:16.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.119612+0000) 2020-02-05T20:09:16.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:16.119+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:16.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:16.124+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.125357+0000) 2020-02-05T20:09:16.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:16.298+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.299068+0000) 2020-02-05T20:09:16.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:16.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.630603+0000) 2020-02-05T20:09:16.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:16.637+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.638849+0000) 2020-02-05T20:09:16.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:16.638+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.639237+0000) 2020-02-05T20:09:16.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:16.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:16.870681+0000) 2020-02-05T20:09:18.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:18.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:18.611948+0000) 2020-02-05T20:09:18.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:18.982+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:18.983351+0000) 2020-02-05T20:09:19.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:19.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:19.269430+0000) 2020-02-05T20:09:19.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:19.878+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:19.879353+0000) 2020-02-05T20:09:20.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:20.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:20.120581+0000) 2020-02-05T20:09:20.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:20.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:20.120580+0000) 2020-02-05T20:09:21.127 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:21.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.125576+0000) 2020-02-05T20:09:21.127 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:21.125+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.126327+0000) 2020-02-05T20:09:21.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:21.127+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.128177+0000) 2020-02-05T20:09:21.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:21.127+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.128864+0000) 2020-02-05T20:09:21.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:21.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.129469+0000) 2020-02-05T20:09:21.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:21.128+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.130100+0000) 2020-02-05T20:09:21.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:21.129+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.130935+0000) 2020-02-05T20:09:21.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:21.129+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.130758+0000) 2020-02-05T20:09:21.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:21.130+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.131540+0000) 2020-02-05T20:09:21.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:21.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.132045+0000) 2020-02-05T20:09:21.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:21.130+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.131664+0000) 2020-02-05T20:09:21.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:21.130+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.132134+0000) 2020-02-05T20:09:21.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:21.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:21.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:21.131+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.133184+0000) 2020-02-05T20:09:21.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:21.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:21.140633+0000) 2020-02-05T20:09:22.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:22.229+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:22.230401+0000) 2020-02-05T20:09:23.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:23.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:23.355582+0000) 2020-02-05T20:09:25.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:25.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:25.771317+0000) 2020-02-05T20:09:26.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:26.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.137474+0000) 2020-02-05T20:09:26.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:26.137+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.138138+0000) 2020-02-05T20:09:26.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:26.139+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.140404+0000) 2020-02-05T20:09:26.142 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:26.140+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.141167+0000) 2020-02-05T20:09:26.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:26.139+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.141319+0000) 2020-02-05T20:09:26.143 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:26.140+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.141912+0000) 2020-02-05T20:09:26.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:26.141+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.142166+0000) 2020-02-05T20:09:26.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:26.141+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.142524+0000) 2020-02-05T20:09:26.144 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:26.141+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.142760+0000) 2020-02-05T20:09:26.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:26.142+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.143658+0000) 2020-02-05T20:09:26.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:26.142+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.143536+0000) 2020-02-05T20:09:26.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:26.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:26.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:26.142+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.143936+0000) 2020-02-05T20:09:26.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:26.142+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.144261+0000) 2020-02-05T20:09:26.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:26.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.151890+0000) 2020-02-05T20:09:26.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:26.306+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.306982+0000) 2020-02-05T20:09:26.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:26.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.630804+0000) 2020-02-05T20:09:26.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:26.638+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.639063+0000) 2020-02-05T20:09:26.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:26.638+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.639462+0000) 2020-02-05T20:09:26.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:26.869+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:26.870916+0000) 2020-02-05T20:09:28.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:28.610+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:28.612149+0000) 2020-02-05T20:09:28.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:28.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:28.983617+0000) 2020-02-05T20:09:29.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:29.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:29.269667+0000) 2020-02-05T20:09:29.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:29.879+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:29.879585+0000) 2020-02-05T20:09:30.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:30.120+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:30.120860+0000) 2020-02-05T20:09:30.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:30.120+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:30.120860+0000) 2020-02-05T20:09:31.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:31.148+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.149291+0000) 2020-02-05T20:09:31.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:31.148+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.149865+0000) 2020-02-05T20:09:31.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:31.150+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.151495+0000) 2020-02-05T20:09:31.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:31.150+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.151918+0000) 2020-02-05T20:09:31.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:31.151+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.152556+0000) 2020-02-05T20:09:31.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:31.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.153315+0000) 2020-02-05T20:09:31.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:31.152+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.153617+0000) 2020-02-05T20:09:31.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:31.152+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.153775+0000) 2020-02-05T20:09:31.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:31.153+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.154425+0000) 2020-02-05T20:09:31.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:31.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.154051+0000) 2020-02-05T20:09:31.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:31.152+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.154178+0000) 2020-02-05T20:09:31.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:31.153+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.154953+0000) 2020-02-05T20:09:31.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:31.153+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:31.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:31.154+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.155534+0000) 2020-02-05T20:09:31.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:31.161+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:31.162576+0000) 2020-02-05T20:09:32.232 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:32.230+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:32.230699+0000) 2020-02-05T20:09:33.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:33.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:33.355827+0000) 2020-02-05T20:09:35.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:35.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:35.771560+0000) 2020-02-05T20:09:36.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:36.159+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.160578+0000) 2020-02-05T20:09:36.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:36.159+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.160749+0000) 2020-02-05T20:09:36.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:36.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.161679+0000) 2020-02-05T20:09:36.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:36.161+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.162379+0000) 2020-02-05T20:09:36.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:36.161+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.163055+0000) 2020-02-05T20:09:36.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:36.162+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.163541+0000) 2020-02-05T20:09:36.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:36.162+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.164011+0000) 2020-02-05T20:09:36.165 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:36.163+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.164430+0000) 2020-02-05T20:09:36.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:36.163+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.164686+0000) 2020-02-05T20:09:36.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:36.164+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.166117+0000) 2020-02-05T20:09:36.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:36.164+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.166027+0000) 2020-02-05T20:09:36.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:36.165+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48414 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:36.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:36.165+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.166767+0000) 2020-02-05T20:09:36.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:36.165+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.167304+0000) 2020-02-05T20:09:36.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:36.173+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.174151+0000) 2020-02-05T20:09:36.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:36.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.315514+0000) 2020-02-05T20:09:36.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:36.630+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.631032+0000) 2020-02-05T20:09:36.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:36.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.639323+0000) 2020-02-05T20:09:36.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:36.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.639714+0000) 2020-02-05T20:09:36.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:36.870+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:36.871138+0000) 2020-02-05T20:09:38.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:38.611+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:38.612389+0000) 2020-02-05T20:09:38.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:38.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:38.983867+0000) 2020-02-05T20:09:39.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:39.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:39.269894+0000) 2020-02-05T20:09:39.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:39.879+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:39.879837+0000) 2020-02-05T20:09:40.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:40.120+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:40.121194+0000) 2020-02-05T20:09:40.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:40.120+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:40.121168+0000) 2020-02-05T20:09:41.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:41.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.171999+0000) 2020-02-05T20:09:41.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:41.171+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.172378+0000) 2020-02-05T20:09:41.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:41.172+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.173919+0000) 2020-02-05T20:09:41.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:41.173+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.174393+0000) 2020-02-05T20:09:41.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:41.173+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.174853+0000) 2020-02-05T20:09:41.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:41.173+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.175063+0000) 2020-02-05T20:09:41.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:41.174+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.175550+0000) 2020-02-05T20:09:41.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:41.174+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.176031+0000) 2020-02-05T20:09:41.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:41.174+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.175740+0000) 2020-02-05T20:09:41.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:41.175+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.176636+0000) 2020-02-05T20:09:41.178 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:41.176+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.177610+0000) 2020-02-05T20:09:41.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:41.175+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.177393+0000) 2020-02-05T20:09:41.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:41.176+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.178356+0000) 2020-02-05T20:09:41.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:41.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:41.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:41.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:41.180689+0000) 2020-02-05T20:09:42.232 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:42.230+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:42.230950+0000) 2020-02-05T20:09:43.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:43.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:43.356079+0000) 2020-02-05T20:09:45.121 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:09:45.127 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:09:45.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:45.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:45.771840+0000) 2020-02-05T20:09:46.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:46.184+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.184906+0000) 2020-02-05T20:09:46.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:46.186+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.186221+0000) 2020-02-05T20:09:46.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:46.186+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.186931+0000) 2020-02-05T20:09:46.188 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:46.186+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.187311+0000) 2020-02-05T20:09:46.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:46.187+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.187751+0000) 2020-02-05T20:09:46.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:46.188+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.188133+0000) 2020-02-05T20:09:46.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:46.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.188544+0000) 2020-02-05T20:09:46.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:46.188+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.188987+0000) 2020-02-05T20:09:46.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:46.188+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.189041+0000) 2020-02-05T20:09:46.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:46.188+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.189286+0000) 2020-02-05T20:09:46.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:46.189+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.189879+0000) 2020-02-05T20:09:46.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:46.190+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.190143+0000) 2020-02-05T20:09:46.192 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:46.189+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48493 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:46.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:46.189+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.190276+0000) 2020-02-05T20:09:46.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:46.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.197633+0000) 2020-02-05T20:09:46.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:46.323+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.323497+0000) 2020-02-05T20:09:46.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:46.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.631270+0000) 2020-02-05T20:09:46.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:46.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.639533+0000) 2020-02-05T20:09:46.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:46.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.639896+0000) 2020-02-05T20:09:46.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:46.870+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:46.871391+0000) 2020-02-05T20:09:48.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:48.612+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:48.612712+0000) 2020-02-05T20:09:48.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:48.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:48.984127+0000) 2020-02-05T20:09:49.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:49.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:49.270140+0000) 2020-02-05T20:09:49.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:49.879+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:49.880132+0000) 2020-02-05T20:09:50.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:50.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:50.121503+0000) 2020-02-05T20:09:50.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:50.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:50.121507+0000) 2020-02-05T20:09:51.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:51.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.196379+0000) 2020-02-05T20:09:51.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:51.196+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.197326+0000) 2020-02-05T20:09:51.199 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:51.196+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.197405+0000) 2020-02-05T20:09:51.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:51.196+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.197964+0000) 2020-02-05T20:09:51.200 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:51.198+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.199145+0000) 2020-02-05T20:09:51.200 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:51.198+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.199356+0000) 2020-02-05T20:09:51.201 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:51.199+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.199861+0000) 2020-02-05T20:09:51.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:51.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.200199+0000) 2020-02-05T20:09:51.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:51.200+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.201284+0000) 2020-02-05T20:09:51.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:51.200+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.201114+0000) 2020-02-05T20:09:51.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:51.201+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.201878+0000) 2020-02-05T20:09:51.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:51.201+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.201927+0000) 2020-02-05T20:09:51.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:51.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48536 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:51.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:51.204+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.206020+0000) 2020-02-05T20:09:51.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:51.206+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:51.207409+0000) 2020-02-05T20:09:52.232 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:52.230+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:52.231202+0000) 2020-02-05T20:09:53.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:53.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:53.356334+0000) 2020-02-05T20:09:55.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:55.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:55.772128+0000) 2020-02-05T20:09:56.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:56.206+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.207579+0000) 2020-02-05T20:09:56.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:56.207+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.208145+0000) 2020-02-05T20:09:56.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:56.207+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.208830+0000) 2020-02-05T20:09:56.210 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:56.208+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.209178+0000) 2020-02-05T20:09:56.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:09:56.208+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.209254+0000) 2020-02-05T20:09:56.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:56.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.209841+0000) 2020-02-05T20:09:56.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:56.209+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.210420+0000) 2020-02-05T20:09:56.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:56.210+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.210661+0000) 2020-02-05T20:09:56.213 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:09:56.211+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.211550+0000) 2020-02-05T20:09:56.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:56.211+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.211929+0000) 2020-02-05T20:09:56.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:09:56.212+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.212784+0000) 2020-02-05T20:09:56.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:09:56.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:09:56.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:09:56.213+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.213748+0000) 2020-02-05T20:09:56.216 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:56.212+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.213320+0000) 2020-02-05T20:09:56.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:56.218+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.219340+0000) 2020-02-05T20:09:56.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:56.330+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.331699+0000) 2020-02-05T20:09:56.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:09:56.630+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.631505+0000) 2020-02-05T20:09:56.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:09:56.638+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.639765+0000) 2020-02-05T20:09:56.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:09:56.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.640137+0000) 2020-02-05T20:09:56.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:09:56.871+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:56.871666+0000) 2020-02-05T20:09:58.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:09:58.612+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:58.612957+0000) 2020-02-05T20:09:58.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:09:58.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:58.984354+0000) 2020-02-05T20:09:59.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:09:59.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:59.270376+0000) 2020-02-05T20:09:59.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:09:59.879+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:09:59.880392+0000) 2020-02-05T20:10:00.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:00.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:00.121806+0000) 2020-02-05T20:10:00.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:00.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:00.121806+0000) 2020-02-05T20:10:01.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:01.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.219970+0000) 2020-02-05T20:10:01.222 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:01.219+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.220909+0000) 2020-02-05T20:10:01.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:01.221+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.222481+0000) 2020-02-05T20:10:01.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:01.222+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.223094+0000) 2020-02-05T20:10:01.225 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:01.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:01.225 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:01.222+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.223715+0000) 2020-02-05T20:10:01.229 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:01.223+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.224019+0000) 2020-02-05T20:10:01.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:01.223+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.224038+0000) 2020-02-05T20:10:01.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:01.223+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.224600+0000) 2020-02-05T20:10:01.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:01.224+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.225137+0000) 2020-02-05T20:10:01.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:01.224+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.225814+0000) 2020-02-05T20:10:01.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:01.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.228551+0000) 2020-02-05T20:10:01.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:01.224+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.224731+0000) 2020-02-05T20:10:01.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:01.224+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.225225+0000) 2020-02-05T20:10:01.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:01.225+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:01.225520+0000) 2020-02-05T20:10:02.233 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:02.231+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:02.231523+0000) 2020-02-05T20:10:03.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:03.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:03.356591+0000) 2020-02-05T20:10:05.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:05.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:05.772397+0000) 2020-02-05T20:10:06.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:06.228+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.229571+0000) 2020-02-05T20:10:06.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:06.229+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.230308+0000) 2020-02-05T20:10:06.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:06.229+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.230789+0000) 2020-02-05T20:10:06.232 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:06.230+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.231323+0000) 2020-02-05T20:10:06.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:06.231+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.232016+0000) 2020-02-05T20:10:06.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:06.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.232207+0000) 2020-02-05T20:10:06.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:06.232+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.232651+0000) 2020-02-05T20:10:06.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:06.233+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.233672+0000) 2020-02-05T20:10:06.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:06.233+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.234443+0000) 2020-02-05T20:10:06.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:06.234+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.234751+0000) 2020-02-05T20:10:06.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:06.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.234716+0000) 2020-02-05T20:10:06.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:06.233+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.234625+0000) 2020-02-05T20:10:06.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:06.235+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:06.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:06.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.238748+0000) 2020-02-05T20:10:06.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:06.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.239034+0000) 2020-02-05T20:10:06.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:06.340+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.341261+0000) 2020-02-05T20:10:06.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:06.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.631742+0000) 2020-02-05T20:10:06.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:06.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.639979+0000) 2020-02-05T20:10:06.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:06.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.640370+0000) 2020-02-05T20:10:06.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:06.871+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:06.871896+0000) 2020-02-05T20:10:08.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:08.613+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:08.613229+0000) 2020-02-05T20:10:08.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:08.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:08.984631+0000) 2020-02-05T20:10:09.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:09.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:09.270614+0000) 2020-02-05T20:10:09.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:09.879+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:09.880623+0000) 2020-02-05T20:10:10.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:10.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:10.122090+0000) 2020-02-05T20:10:10.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:10.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:10.122089+0000) 2020-02-05T20:10:11.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:11.241+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.241867+0000) 2020-02-05T20:10:11.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:11.242+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.243270+0000) 2020-02-05T20:10:11.245 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:11.242+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.243623+0000) 2020-02-05T20:10:11.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:11.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.244000+0000) 2020-02-05T20:10:11.245 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:11.243+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.244528+0000) 2020-02-05T20:10:11.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:11.244+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.244669+0000) 2020-02-05T20:10:11.246 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:11.244+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.245368+0000) 2020-02-05T20:10:11.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:11.244+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.245368+0000) 2020-02-05T20:10:11.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:11.245+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.245647+0000) 2020-02-05T20:10:11.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:11.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.246559+0000) 2020-02-05T20:10:11.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:11.246+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.246579+0000) 2020-02-05T20:10:11.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:11.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:11.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:11.246+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.247314+0000) 2020-02-05T20:10:11.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:11.249+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.250205+0000) 2020-02-05T20:10:11.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:11.251+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:11.252479+0000) 2020-02-05T20:10:12.233 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:12.231+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:12.231786+0000) 2020-02-05T20:10:13.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:13.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:13.356826+0000) 2020-02-05T20:10:15.335 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:10:15.342 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:10:15.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:15.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:15.772636+0000) 2020-02-05T20:10:16.253 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:16.250+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.251907+0000) 2020-02-05T20:10:16.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:16.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.253318+0000) 2020-02-05T20:10:16.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:16.253+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.254225+0000) 2020-02-05T20:10:16.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:16.253+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.254838+0000) 2020-02-05T20:10:16.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:16.255+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.255719+0000) 2020-02-05T20:10:16.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:16.254+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.255361+0000) 2020-02-05T20:10:16.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:16.256+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.256591+0000) 2020-02-05T20:10:16.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:16.256+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.256875+0000) 2020-02-05T20:10:16.259 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:16.257+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:16.261 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:16.257+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.258341+0000) 2020-02-05T20:10:16.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:16.257+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.258833+0000) 2020-02-05T20:10:16.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:16.258+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.259357+0000) 2020-02-05T20:10:16.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:16.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.259610+0000) 2020-02-05T20:10:16.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:16.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.265473+0000) 2020-02-05T20:10:16.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:16.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.265813+0000) 2020-02-05T20:10:16.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:16.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.353214+0000) 2020-02-05T20:10:16.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:16.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.631994+0000) 2020-02-05T20:10:16.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:16.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.640190+0000) 2020-02-05T20:10:16.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:16.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.640577+0000) 2020-02-05T20:10:16.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:16.872+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:16.872134+0000) 2020-02-05T20:10:18.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:18.613+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:18.613485+0000) 2020-02-05T20:10:18.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:18.983+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:18.984918+0000) 2020-02-05T20:10:19.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:19.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:19.270841+0000) 2020-02-05T20:10:19.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:19.880+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:19.880847+0000) 2020-02-05T20:10:20.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:20.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:20.122340+0000) 2020-02-05T20:10:20.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:20.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:20.122337+0000) 2020-02-05T20:10:21.266 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:21.263+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.264295+0000) 2020-02-05T20:10:21.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:21.264+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.264734+0000) 2020-02-05T20:10:21.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:21.263+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.263821+0000) 2020-02-05T20:10:21.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:21.264+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.265907+0000) 2020-02-05T20:10:21.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:21.265+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.266370+0000) 2020-02-05T20:10:21.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:21.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.266726+0000) 2020-02-05T20:10:21.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:21.267+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.267843+0000) 2020-02-05T20:10:21.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:21.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.268630+0000) 2020-02-05T20:10:21.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:21.268+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.268547+0000) 2020-02-05T20:10:21.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:21.267+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.268995+0000) 2020-02-05T20:10:21.271 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:21.268+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.269870+0000) 2020-02-05T20:10:21.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:21.269+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:21.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:21.269+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.270119+0000) 2020-02-05T20:10:21.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:21.269+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.270485+0000) 2020-02-05T20:10:21.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:21.273+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:21.274568+0000) 2020-02-05T20:10:22.233 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:22.231+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:22.231953+0000) 2020-02-05T20:10:23.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:23.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:23.357066+0000) 2020-02-05T20:10:25.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:25.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:25.772860+0000) 2020-02-05T20:10:26.276 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:26.273+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.274960+0000) 2020-02-05T20:10:26.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:26.274+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.275726+0000) 2020-02-05T20:10:26.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:26.274+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.276356+0000) 2020-02-05T20:10:26.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:26.276+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.277381+0000) 2020-02-05T20:10:26.279 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:26.276+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.277794+0000) 2020-02-05T20:10:26.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:26.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.278217+0000) 2020-02-05T20:10:26.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:26.277+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.278414+0000) 2020-02-05T20:10:26.280 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:26.277+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.278730+0000) 2020-02-05T20:10:26.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:26.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.279283+0000) 2020-02-05T20:10:26.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:26.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.279762+0000) 2020-02-05T20:10:26.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:26.279+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.281224+0000) 2020-02-05T20:10:26.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:26.279+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.281079+0000) 2020-02-05T20:10:26.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:26.280+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.281752+0000) 2020-02-05T20:10:26.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:26.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:26.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:26.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.285129+0000) 2020-02-05T20:10:26.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:26.359+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.360304+0000) 2020-02-05T20:10:26.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:26.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.632222+0000) 2020-02-05T20:10:26.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:26.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.640362+0000) 2020-02-05T20:10:26.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:26.639+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.640777+0000) 2020-02-05T20:10:26.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:26.871+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:26.872372+0000) 2020-02-05T20:10:28.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:28.612+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:28.613746+0000) 2020-02-05T20:10:28.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:28.984+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:28.985176+0000) 2020-02-05T20:10:29.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:29.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:29.271123+0000) 2020-02-05T20:10:29.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:29.880+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:29.881071+0000) 2020-02-05T20:10:30.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:30.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:30.122680+0000) 2020-02-05T20:10:30.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:30.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:30.122680+0000) 2020-02-05T20:10:31.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:31.285+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.286549+0000) 2020-02-05T20:10:31.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:31.287+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.288786+0000) 2020-02-05T20:10:31.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:31.288+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.289867+0000) 2020-02-05T20:10:31.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:31.289+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.290226+0000) 2020-02-05T20:10:31.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:31.289+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.290864+0000) 2020-02-05T20:10:31.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:31.290+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.291970+0000) 2020-02-05T20:10:31.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:31.290+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.292038+0000) 2020-02-05T20:10:31.294 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:31.291+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.292203+0000) 2020-02-05T20:10:31.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:31.290+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.292365+0000) 2020-02-05T20:10:31.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:31.291+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.292880+0000) 2020-02-05T20:10:31.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:31.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.293420+0000) 2020-02-05T20:10:31.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:31.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48856 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:31.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:31.292+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.294068+0000) 2020-02-05T20:10:31.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:31.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.293546+0000) 2020-02-05T20:10:31.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:31.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:31.295917+0000) 2020-02-05T20:10:32.233 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:32.230+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:32.232170+0000) 2020-02-05T20:10:33.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:33.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:33.357353+0000) 2020-02-05T20:10:35.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:35.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:35.773144+0000) 2020-02-05T20:10:36.299 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:36.297+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.298376+0000) 2020-02-05T20:10:36.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:36.298+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.299189+0000) 2020-02-05T20:10:36.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:36.298+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.299532+0000) 2020-02-05T20:10:36.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:36.299+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.301047+0000) 2020-02-05T20:10:36.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:36.299+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.301402+0000) 2020-02-05T20:10:36.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:36.302+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.303491+0000) 2020-02-05T20:10:36.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:36.301+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.302827+0000) 2020-02-05T20:10:36.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:36.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.303792+0000) 2020-02-05T20:10:36.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:36.303+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.304874+0000) 2020-02-05T20:10:36.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:36.303+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.305298+0000) 2020-02-05T20:10:36.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:36.304+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.305617+0000) 2020-02-05T20:10:36.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:36.304+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:36.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:36.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.305892+0000) 2020-02-05T20:10:36.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:36.308+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.309985+0000) 2020-02-05T20:10:36.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:36.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.311557+0000) 2020-02-05T20:10:36.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:36.368+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.368818+0000) 2020-02-05T20:10:36.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:36.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.632437+0000) 2020-02-05T20:10:36.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:36.640+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.640571+0000) 2020-02-05T20:10:36.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:36.640+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.640944+0000) 2020-02-05T20:10:36.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:36.871+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:36.872641+0000) 2020-02-05T20:10:38.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:38.613+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:38.614004+0000) 2020-02-05T20:10:38.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:38.985+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:38.985412+0000) 2020-02-05T20:10:39.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:39.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:39.271362+0000) 2020-02-05T20:10:39.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:39.881+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:39.881297+0000) 2020-02-05T20:10:40.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:40.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:40.122977+0000) 2020-02-05T20:10:40.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:40.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:40.122977+0000) 2020-02-05T20:10:41.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:41.310+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.310525+0000) 2020-02-05T20:10:41.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:41.311+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.312859+0000) 2020-02-05T20:10:41.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:41.313+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.313539+0000) 2020-02-05T20:10:41.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:41.312+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.313841+0000) 2020-02-05T20:10:41.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:41.312+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.314386+0000) 2020-02-05T20:10:41.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:41.314+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.314913+0000) 2020-02-05T20:10:41.316 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:41.315+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.315569+0000) 2020-02-05T20:10:41.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:41.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.315634+0000) 2020-02-05T20:10:41.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:41.314+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.316040+0000) 2020-02-05T20:10:41.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:41.316+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.316533+0000) 2020-02-05T20:10:41.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:41.315+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.316828+0000) 2020-02-05T20:10:41.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:41.315+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.317126+0000) 2020-02-05T20:10:41.319 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:41.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:41.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:41.318+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.318936+0000) 2020-02-05T20:10:41.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:41.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:41.320418+0000) 2020-02-05T20:10:42.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:42.231+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:42.232412+0000) 2020-02-05T20:10:43.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:43.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:43.357613+0000) 2020-02-05T20:10:45.514 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:10:45.520 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:10:45.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:45.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:45.773399+0000) 2020-02-05T20:10:46.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:46.323+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.324060+0000) 2020-02-05T20:10:46.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:46.323+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.324745+0000) 2020-02-05T20:10:46.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:46.321+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.323246+0000) 2020-02-05T20:10:46.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:46.323+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.324488+0000) 2020-02-05T20:10:46.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:46.324+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.325504+0000) 2020-02-05T20:10:46.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:46.325+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.326235+0000) 2020-02-05T20:10:46.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:46.326+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.327384+0000) 2020-02-05T20:10:46.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:46.325+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.326626+0000) 2020-02-05T20:10:46.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:46.326+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.327797+0000) 2020-02-05T20:10:46.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:46.327+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.328313+0000) 2020-02-05T20:10:46.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:46.326+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 48975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:46.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:46.327+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.328903+0000) 2020-02-05T20:10:46.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:46.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.332742+0000) 2020-02-05T20:10:46.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:46.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.333014+0000) 2020-02-05T20:10:46.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:46.332+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.333287+0000) 2020-02-05T20:10:46.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:46.374+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.375137+0000) 2020-02-05T20:10:46.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:46.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.632602+0000) 2020-02-05T20:10:46.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:46.639+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.640795+0000) 2020-02-05T20:10:46.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:46.640+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.641141+0000) 2020-02-05T20:10:46.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:46.872+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:46.872904+0000) 2020-02-05T20:10:48.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:48.613+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:48.614252+0000) 2020-02-05T20:10:48.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:48.984+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:48.985672+0000) 2020-02-05T20:10:49.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:49.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:49.271548+0000) 2020-02-05T20:10:49.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:49.880+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:49.881544+0000) 2020-02-05T20:10:50.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:50.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:50.123280+0000) 2020-02-05T20:10:50.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:50.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:50.123277+0000) 2020-02-05T20:10:51.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:51.332+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.333442+0000) 2020-02-05T20:10:51.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:51.333+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.334523+0000) 2020-02-05T20:10:51.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:51.334+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.335130+0000) 2020-02-05T20:10:51.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:51.334+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.335635+0000) 2020-02-05T20:10:51.338 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:51.335+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.336883+0000) 2020-02-05T20:10:51.339 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:51.336+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.337823+0000) 2020-02-05T20:10:51.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:51.337+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.338193+0000) 2020-02-05T20:10:51.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:51.336+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.338285+0000) 2020-02-05T20:10:51.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:51.337+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.338826+0000) 2020-02-05T20:10:51.341 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:51.338+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.339254+0000) 2020-02-05T20:10:51.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:51.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.339598+0000) 2020-02-05T20:10:51.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:51.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49016 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:51.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:51.338+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.339813+0000) 2020-02-05T20:10:51.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:51.338+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.340044+0000) 2020-02-05T20:10:51.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:51.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:51.343406+0000) 2020-02-05T20:10:52.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:52.231+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:52.232671+0000) 2020-02-05T20:10:53.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:53.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:53.357822+0000) 2020-02-05T20:10:55.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:55.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:55.773640+0000) 2020-02-05T20:10:56.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:56.344+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.345238+0000) 2020-02-05T20:10:56.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:56.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.347148+0000) 2020-02-05T20:10:56.349 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:56.346+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.347769+0000) 2020-02-05T20:10:56.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:56.347+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.348594+0000) 2020-02-05T20:10:56.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:56.348+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.349301+0000) 2020-02-05T20:10:56.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:56.348+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.349683+0000) 2020-02-05T20:10:56.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:56.348+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.349985+0000) 2020-02-05T20:10:56.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:56.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.350721+0000) 2020-02-05T20:10:56.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:10:56.349+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.350544+0000) 2020-02-05T20:10:56.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:10:56.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49055 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:10:56.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:10:56.349+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.351126+0000) 2020-02-05T20:10:56.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:56.350+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.351568+0000) 2020-02-05T20:10:56.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:10:56.349+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.351379+0000) 2020-02-05T20:10:56.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:10:56.350+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.351931+0000) 2020-02-05T20:10:56.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:56.353+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.354823+0000) 2020-02-05T20:10:56.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:56.379+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.380629+0000) 2020-02-05T20:10:56.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:10:56.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.632811+0000) 2020-02-05T20:10:56.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:10:56.640+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.641046+0000) 2020-02-05T20:10:56.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:10:56.640+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.641307+0000) 2020-02-05T20:10:56.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:10:56.872+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:56.873144+0000) 2020-02-05T20:10:58.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:10:58.613+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:58.614568+0000) 2020-02-05T20:10:58.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:10:58.984+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:58.985919+0000) 2020-02-05T20:10:59.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:10:59.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:59.271774+0000) 2020-02-05T20:10:59.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:10:59.880+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:10:59.881771+0000) 2020-02-05T20:11:00.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:00.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:00.123558+0000) 2020-02-05T20:11:00.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:00.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:00.123558+0000) 2020-02-05T20:11:01.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:01.355+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.356491+0000) 2020-02-05T20:11:01.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:01.356+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.357649+0000) 2020-02-05T20:11:01.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:01.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.358530+0000) 2020-02-05T20:11:01.360 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:01.358+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.359243+0000) 2020-02-05T20:11:01.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:01.358+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.359839+0000) 2020-02-05T20:11:01.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:01.358+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.359965+0000) 2020-02-05T20:11:01.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:01.358+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.359888+0000) 2020-02-05T20:11:01.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:01.360+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.361155+0000) 2020-02-05T20:11:01.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:01.359+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.360783+0000) 2020-02-05T20:11:01.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:01.360+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.361758+0000) 2020-02-05T20:11:01.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:01.361+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.362378+0000) 2020-02-05T20:11:01.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:01.360+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49097 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:01.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:01.360+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.361756+0000) 2020-02-05T20:11:01.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:01.361+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.362739+0000) 2020-02-05T20:11:01.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:01.367+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:01.368797+0000) 2020-02-05T20:11:02.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:02.232+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:02.232885+0000) 2020-02-05T20:11:03.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:03.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:03.358102+0000) 2020-02-05T20:11:05.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:05.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:05.773873+0000) 2020-02-05T20:11:06.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:06.366+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.367666+0000) 2020-02-05T20:11:06.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:06.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.368751+0000) 2020-02-05T20:11:06.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:06.367+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.369155+0000) 2020-02-05T20:11:06.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:06.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.369524+0000) 2020-02-05T20:11:06.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:06.368+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.369805+0000) 2020-02-05T20:11:06.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:06.369+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.370339+0000) 2020-02-05T20:11:06.372 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:06.369+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.370919+0000) 2020-02-05T20:11:06.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:06.371+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.372332+0000) 2020-02-05T20:11:06.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:06.370+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.372132+0000) 2020-02-05T20:11:06.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:06.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.373244+0000) 2020-02-05T20:11:06.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:06.371+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.373166+0000) 2020-02-05T20:11:06.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:06.371+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.373202+0000) 2020-02-05T20:11:06.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:06.372+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:06.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:06.372+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.373803+0000) 2020-02-05T20:11:06.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:06.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.378320+0000) 2020-02-05T20:11:06.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:06.387+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.387646+0000) 2020-02-05T20:11:06.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:06.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.633027+0000) 2020-02-05T20:11:06.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:06.640+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.641284+0000) 2020-02-05T20:11:06.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:06.640+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.641522+0000) 2020-02-05T20:11:06.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:06.871+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:06.873366+0000) 2020-02-05T20:11:08.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:08.614+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:08.614895+0000) 2020-02-05T20:11:08.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:08.985+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:08.986172+0000) 2020-02-05T20:11:09.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:09.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:09.272057+0000) 2020-02-05T20:11:09.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:09.881+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:09.881998+0000) 2020-02-05T20:11:10.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:10.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:10.123870+0000) 2020-02-05T20:11:10.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:10.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:10.123871+0000) 2020-02-05T20:11:11.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:11.378+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.380134+0000) 2020-02-05T20:11:11.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:11.379+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.380627+0000) 2020-02-05T20:11:11.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:11.380+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.381542+0000) 2020-02-05T20:11:11.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:11.379+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.381343+0000) 2020-02-05T20:11:11.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:11.380+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.381825+0000) 2020-02-05T20:11:11.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:11.381+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.382164+0000) 2020-02-05T20:11:11.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:11.380+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.382312+0000) 2020-02-05T20:11:11.385 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:11.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.383017+0000) 2020-02-05T20:11:11.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:11.382+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.383628+0000) 2020-02-05T20:11:11.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:11.381+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.383340+0000) 2020-02-05T20:11:11.386 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:11.383+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.384124+0000) 2020-02-05T20:11:11.386 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:11.383+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.384401+0000) 2020-02-05T20:11:11.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:11.383+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.384772+0000) 2020-02-05T20:11:11.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:11.383+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:11.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:11.389+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:11.390824+0000) 2020-02-05T20:11:12.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:12.232+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:12.233105+0000) 2020-02-05T20:11:13.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:13.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:13.358343+0000) 2020-02-05T20:11:15.695 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:11:15.701 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:11:15.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:15.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:15.774124+0000) 2020-02-05T20:11:16.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:16.389+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.389786+0000) 2020-02-05T20:11:16.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:16.389+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.390447+0000) 2020-02-05T20:11:16.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:16.391+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.392258+0000) 2020-02-05T20:11:16.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:16.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.391555+0000) 2020-02-05T20:11:16.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:16.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.393499+0000) 2020-02-05T20:11:16.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:16.392+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.393927+0000) 2020-02-05T20:11:16.396 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:16.393+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.394133+0000) 2020-02-05T20:11:16.396 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:16.393+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.394980+0000) 2020-02-05T20:11:16.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:16.395+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.395950+0000) 2020-02-05T20:11:16.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:16.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:16.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:16.395+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.396380+0000) 2020-02-05T20:11:16.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:16.396+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.397108+0000) 2020-02-05T20:11:16.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:16.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.395710+0000) 2020-02-05T20:11:16.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:16.396+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.397293+0000) 2020-02-05T20:11:16.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:16.404+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.405301+0000) 2020-02-05T20:11:16.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:16.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.633285+0000) 2020-02-05T20:11:16.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:16.641+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.641502+0000) 2020-02-05T20:11:16.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:16.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.641769+0000) 2020-02-05T20:11:16.875 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:16.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:16.873643+0000) 2020-02-05T20:11:18.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:18.614+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:18.615157+0000) 2020-02-05T20:11:18.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:18.986+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:18.986416+0000) 2020-02-05T20:11:19.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:19.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:19.272325+0000) 2020-02-05T20:11:19.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:19.881+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:19.882253+0000) 2020-02-05T20:11:20.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:20.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:20.124152+0000) 2020-02-05T20:11:20.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:20.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:20.124147+0000) 2020-02-05T20:11:21.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:21.401+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.401794+0000) 2020-02-05T20:11:21.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:21.402+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.402752+0000) 2020-02-05T20:11:21.405 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:21.402+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.403487+0000) 2020-02-05T20:11:21.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:21.402+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.403798+0000) 2020-02-05T20:11:21.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:21.403+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.404265+0000) 2020-02-05T20:11:21.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:21.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.404785+0000) 2020-02-05T20:11:21.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:21.405+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.406477+0000) 2020-02-05T20:11:21.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:21.403+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.404920+0000) 2020-02-05T20:11:21.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:21.406+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.407300+0000) 2020-02-05T20:11:21.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:21.404+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.405275+0000) 2020-02-05T20:11:21.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:21.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.405822+0000) 2020-02-05T20:11:21.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:21.405+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.405821+0000) 2020-02-05T20:11:21.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:21.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:21.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:21.407+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.407460+0000) 2020-02-05T20:11:21.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:21.411+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:21.413068+0000) 2020-02-05T20:11:22.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:22.232+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:22.233333+0000) 2020-02-05T20:11:23.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:23.144+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:23.145027+0000) 2020-02-05T20:11:23.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:23.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:23.358693+0000) 2020-02-05T20:11:25.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:25.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:25.774396+0000) 2020-02-05T20:11:26.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:26.411+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.412742+0000) 2020-02-05T20:11:26.414 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:26.412+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.413121+0000) 2020-02-05T20:11:26.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:26.412+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.413795+0000) 2020-02-05T20:11:26.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:26.413+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.413649+0000) 2020-02-05T20:11:26.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:26.414+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.414827+0000) 2020-02-05T20:11:26.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:26.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.415646+0000) 2020-02-05T20:11:26.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:26.415+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.416155+0000) 2020-02-05T20:11:26.418 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:26.416+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.416934+0000) 2020-02-05T20:11:26.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:26.416+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.417260+0000) 2020-02-05T20:11:26.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:26.416+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.417875+0000) 2020-02-05T20:11:26.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:26.417+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.418021+0000) 2020-02-05T20:11:26.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:26.417+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.418313+0000) 2020-02-05T20:11:26.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:26.418+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49295 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:26.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:26.421+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.423014+0000) 2020-02-05T20:11:26.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:26.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.423301+0000) 2020-02-05T20:11:26.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:26.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.633495+0000) 2020-02-05T20:11:26.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:26.641+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.641702+0000) 2020-02-05T20:11:26.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:26.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.641980+0000) 2020-02-05T20:11:26.875 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:26.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:26.873874+0000) 2020-02-05T20:11:28.617 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:28.615+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:28.615465+0000) 2020-02-05T20:11:28.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:28.985+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:28.986666+0000) 2020-02-05T20:11:29.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:29.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:29.272533+0000) 2020-02-05T20:11:29.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:29.882+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:29.882480+0000) 2020-02-05T20:11:30.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:30.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:30.124482+0000) 2020-02-05T20:11:30.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:30.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:30.124485+0000) 2020-02-05T20:11:31.426 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:31.423+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.424521+0000) 2020-02-05T20:11:31.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:31.424+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.425406+0000) 2020-02-05T20:11:31.427 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:31.424+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.425939+0000) 2020-02-05T20:11:31.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:31.425+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.426344+0000) 2020-02-05T20:11:31.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:31.425+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.426666+0000) 2020-02-05T20:11:31.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:31.426+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.427475+0000) 2020-02-05T20:11:31.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:31.427+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.427817+0000) 2020-02-05T20:11:31.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:31.428+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.428552+0000) 2020-02-05T20:11:31.430 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:31.428+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.428960+0000) 2020-02-05T20:11:31.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:31.428+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.429179+0000) 2020-02-05T20:11:31.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:31.428+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.428615+0000) 2020-02-05T20:11:31.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:31.429+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.429979+0000) 2020-02-05T20:11:31.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:31.430+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.430957+0000) 2020-02-05T20:11:31.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:31.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:31.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:31.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:31.437781+0000) 2020-02-05T20:11:32.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:32.233+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:32.233563+0000) 2020-02-05T20:11:33.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:33.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:33.153673+0000) 2020-02-05T20:11:33.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:33.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:33.358946+0000) 2020-02-05T20:11:35.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:35.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:35.774621+0000) 2020-02-05T20:11:36.439 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:36.436+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.437952+0000) 2020-02-05T20:11:36.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:36.438+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.438640+0000) 2020-02-05T20:11:36.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:36.437+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.438487+0000) 2020-02-05T20:11:36.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:36.439+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.439413+0000) 2020-02-05T20:11:36.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:36.439+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.440041+0000) 2020-02-05T20:11:36.442 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:36.438+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.439774+0000) 2020-02-05T20:11:36.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:36.439+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.440479+0000) 2020-02-05T20:11:36.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:36.439+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.440601+0000) 2020-02-05T20:11:36.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:36.440+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.440966+0000) 2020-02-05T20:11:36.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:36.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:36.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:36.441+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.441647+0000) 2020-02-05T20:11:36.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:36.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.441478+0000) 2020-02-05T20:11:36.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:36.441+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.442399+0000) 2020-02-05T20:11:36.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:36.442+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.442916+0000) 2020-02-05T20:11:36.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:36.447+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.448167+0000) 2020-02-05T20:11:36.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:36.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.633734+0000) 2020-02-05T20:11:36.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:36.640+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.641930+0000) 2020-02-05T20:11:36.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:36.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.642220+0000) 2020-02-05T20:11:36.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:36.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:36.874116+0000) 2020-02-05T20:11:38.617 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:38.615+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:38.615768+0000) 2020-02-05T20:11:38.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:38.985+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:38.986957+0000) 2020-02-05T20:11:39.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:39.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:39.272765+0000) 2020-02-05T20:11:39.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:39.881+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:39.882709+0000) 2020-02-05T20:11:40.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:40.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:40.124840+0000) 2020-02-05T20:11:40.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:40.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:40.124840+0000) 2020-02-05T20:11:41.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:41.445+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.446314+0000) 2020-02-05T20:11:41.449 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:41.447+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.448264+0000) 2020-02-05T20:11:41.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:41.448+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.449237+0000) 2020-02-05T20:11:41.451 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:41.448+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.449234+0000) 2020-02-05T20:11:41.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:41.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.449546+0000) 2020-02-05T20:11:41.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:41.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.450154+0000) 2020-02-05T20:11:41.452 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:41.449+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.450927+0000) 2020-02-05T20:11:41.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:41.449+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.450292+0000) 2020-02-05T20:11:41.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:41.450+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.451268+0000) 2020-02-05T20:11:41.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:41.451+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.452274+0000) 2020-02-05T20:11:41.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:41.452+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.453038+0000) 2020-02-05T20:11:41.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:41.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.453445+0000) 2020-02-05T20:11:41.455 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:41.453+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:41.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:41.453+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.453809+0000) 2020-02-05T20:11:41.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:41.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:41.456623+0000) 2020-02-05T20:11:42.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:42.233+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:42.233729+0000) 2020-02-05T20:11:43.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:43.161+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:43.162181+0000) 2020-02-05T20:11:43.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:43.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:43.359205+0000) 2020-02-05T20:11:45.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:45.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:45.774880+0000) 2020-02-05T20:11:45.831 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:11:45.838 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:11:46.459 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:46.456+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.458024+0000) 2020-02-05T20:11:46.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:46.460+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.461123+0000) 2020-02-05T20:11:46.462 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:46.460+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.461324+0000) 2020-02-05T20:11:46.463 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:46.460+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.460885+0000) 2020-02-05T20:11:46.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:46.461+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.461589+0000) 2020-02-05T20:11:46.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:46.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.462019+0000) 2020-02-05T20:11:46.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:46.461+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.462268+0000) 2020-02-05T20:11:46.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:46.461+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.462954+0000) 2020-02-05T20:11:46.465 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:46.462+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.463377+0000) 2020-02-05T20:11:46.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:46.462+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.463841+0000) 2020-02-05T20:11:46.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:46.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49455 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:46.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:46.463+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.464340+0000) 2020-02-05T20:11:46.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:46.464+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.464854+0000) 2020-02-05T20:11:46.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:46.465+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.466150+0000) 2020-02-05T20:11:46.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:46.467+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.469039+0000) 2020-02-05T20:11:46.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:46.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.633950+0000) 2020-02-05T20:11:46.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:46.641+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.642141+0000) 2020-02-05T20:11:46.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:46.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.642454+0000) 2020-02-05T20:11:46.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:46.874+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:46.874348+0000) 2020-02-05T20:11:48.617 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:48.615+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:48.616035+0000) 2020-02-05T20:11:48.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:48.986+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:48.987290+0000) 2020-02-05T20:11:49.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:49.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:49.272999+0000) 2020-02-05T20:11:49.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:49.882+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:49.882941+0000) 2020-02-05T20:11:50.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:50.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:50.125106+0000) 2020-02-05T20:11:50.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:50.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:50.125105+0000) 2020-02-05T20:11:51.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:51.468+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.469396+0000) 2020-02-05T20:11:51.472 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:51.469+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.470623+0000) 2020-02-05T20:11:51.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:51.468+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.470234+0000) 2020-02-05T20:11:51.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:51.470+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.471109+0000) 2020-02-05T20:11:51.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:51.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.471515+0000) 2020-02-05T20:11:51.473 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:51.470+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.471971+0000) 2020-02-05T20:11:51.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:51.471+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.472913+0000) 2020-02-05T20:11:51.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:51.471+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.473229+0000) 2020-02-05T20:11:51.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:51.472+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.473970+0000) 2020-02-05T20:11:51.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:51.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.474621+0000) 2020-02-05T20:11:51.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:51.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.474647+0000) 2020-02-05T20:11:51.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:51.473+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.474840+0000) 2020-02-05T20:11:51.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:51.473+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.475164+0000) 2020-02-05T20:11:51.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:51.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49496 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:51.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:51.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:51.479997+0000) 2020-02-05T20:11:52.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:52.232+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:52.233959+0000) 2020-02-05T20:11:53.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:53.173+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:53.174138+0000) 2020-02-05T20:11:53.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:53.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:53.359467+0000) 2020-02-05T20:11:55.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:55.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:55.775136+0000) 2020-02-05T20:11:56.482 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:56.479+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.480289+0000) 2020-02-05T20:11:56.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:56.480+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.481154+0000) 2020-02-05T20:11:56.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:56.480+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.482291+0000) 2020-02-05T20:11:56.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:56.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.483858+0000) 2020-02-05T20:11:56.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:11:56.483+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.484779+0000) 2020-02-05T20:11:56.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:11:56.483+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.485249+0000) 2020-02-05T20:11:56.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:11:56.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49535 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:11:56.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:56.484+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.486136+0000) 2020-02-05T20:11:56.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:11:56.485+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.486423+0000) 2020-02-05T20:11:56.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:56.485+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.486904+0000) 2020-02-05T20:11:56.489 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:56.486+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.487501+0000) 2020-02-05T20:11:56.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:56.486+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.487589+0000) 2020-02-05T20:11:56.489 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:11:56.486+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.487756+0000) 2020-02-05T20:11:56.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:56.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.488043+0000) 2020-02-05T20:11:56.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:56.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.494045+0000) 2020-02-05T20:11:56.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:11:56.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.634204+0000) 2020-02-05T20:11:56.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:11:56.641+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.642361+0000) 2020-02-05T20:11:56.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:11:56.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.642614+0000) 2020-02-05T20:11:56.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:11:56.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:56.874586+0000) 2020-02-05T20:11:58.617 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:11:58.615+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:58.616305+0000) 2020-02-05T20:11:58.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:11:58.986+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:58.987488+0000) 2020-02-05T20:11:59.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:11:59.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:59.273273+0000) 2020-02-05T20:11:59.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:11:59.882+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:11:59.883167+0000) 2020-02-05T20:12:00.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:00.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:00.125412+0000) 2020-02-05T20:12:00.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:00.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:00.125414+0000) 2020-02-05T20:12:01.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:01.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.491282+0000) 2020-02-05T20:12:01.493 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:01.490+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.491809+0000) 2020-02-05T20:12:01.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:01.491+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.492269+0000) 2020-02-05T20:12:01.494 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:01.491+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.492795+0000) 2020-02-05T20:12:01.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:01.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.494885+0000) 2020-02-05T20:12:01.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:01.493+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.494520+0000) 2020-02-05T20:12:01.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:01.493+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.495320+0000) 2020-02-05T20:12:01.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:01.494+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.495842+0000) 2020-02-05T20:12:01.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:01.495+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.496575+0000) 2020-02-05T20:12:01.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:01.494+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.496292+0000) 2020-02-05T20:12:01.498 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:01.495+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.497341+0000) 2020-02-05T20:12:01.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:01.496+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.497938+0000) 2020-02-05T20:12:01.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:01.496+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.497954+0000) 2020-02-05T20:12:01.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:01.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:01.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:01.501+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:01.502672+0000) 2020-02-05T20:12:02.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:02.233+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:02.234231+0000) 2020-02-05T20:12:03.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:03.181+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:03.182160+0000) 2020-02-05T20:12:03.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:03.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:03.359722+0000) 2020-02-05T20:12:05.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:05.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:05.775405+0000) 2020-02-05T20:12:06.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:06.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.505510+0000) 2020-02-05T20:12:06.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:06.504+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.505956+0000) 2020-02-05T20:12:06.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:06.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.506338+0000) 2020-02-05T20:12:06.508 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:06.505+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.506581+0000) 2020-02-05T20:12:06.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:06.506+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.507321+0000) 2020-02-05T20:12:06.509 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:06.507+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.508298+0000) 2020-02-05T20:12:06.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:06.507+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.508750+0000) 2020-02-05T20:12:06.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:06.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.509258+0000) 2020-02-05T20:12:06.510 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:06.508+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.509648+0000) 2020-02-05T20:12:06.511 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:06.508+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.510116+0000) 2020-02-05T20:12:06.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:06.508+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.510182+0000) 2020-02-05T20:12:06.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:06.509+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.510573+0000) 2020-02-05T20:12:06.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:06.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:06.513 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:06.509+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.510790+0000) 2020-02-05T20:12:06.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:06.516+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.517618+0000) 2020-02-05T20:12:06.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:06.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.634366+0000) 2020-02-05T20:12:06.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:06.641+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.642573+0000) 2020-02-05T20:12:06.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:06.641+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.642817+0000) 2020-02-05T20:12:06.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:06.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:06.874796+0000) 2020-02-05T20:12:08.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:08.615+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:08.616563+0000) 2020-02-05T20:12:08.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:08.987+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:08.987764+0000) 2020-02-05T20:12:09.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:09.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:09.273517+0000) 2020-02-05T20:12:09.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:09.882+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:09.883398+0000) 2020-02-05T20:12:10.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:10.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:10.125770+0000) 2020-02-05T20:12:10.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:10.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:10.125770+0000) 2020-02-05T20:12:11.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:11.515+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.516782+0000) 2020-02-05T20:12:11.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:11.517+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.518136+0000) 2020-02-05T20:12:11.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:11.517+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.518438+0000) 2020-02-05T20:12:11.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:11.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.518652+0000) 2020-02-05T20:12:11.520 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:11.518+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.519065+0000) 2020-02-05T20:12:11.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:11.518+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.519555+0000) 2020-02-05T20:12:11.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:11.518+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.519791+0000) 2020-02-05T20:12:11.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:11.518+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.519951+0000) 2020-02-05T20:12:11.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:11.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.520285+0000) 2020-02-05T20:12:11.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:11.520+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.521644+0000) 2020-02-05T20:12:11.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:11.519+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.521118+0000) 2020-02-05T20:12:11.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:11.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.522324+0000) 2020-02-05T20:12:11.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:11.521+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.522792+0000) 2020-02-05T20:12:11.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:11.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:11.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:11.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:11.526044+0000) 2020-02-05T20:12:12.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:12.233+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:12.234473+0000) 2020-02-05T20:12:13.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:13.191+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:13.192684+0000) 2020-02-05T20:12:13.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:13.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:13.359942+0000) 2020-02-05T20:12:15.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:15.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:15.775643+0000) 2020-02-05T20:12:15.987 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:12:15.993 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:12:16.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:16.526+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.527798+0000) 2020-02-05T20:12:16.529 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:16.527+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.528318+0000) 2020-02-05T20:12:16.530 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:16.528+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.529409+0000) 2020-02-05T20:12:16.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:16.529+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.530695+0000) 2020-02-05T20:12:16.532 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:16.530+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.531366+0000) 2020-02-05T20:12:16.533 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:16.530+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.531717+0000) 2020-02-05T20:12:16.533 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:16.530+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.531959+0000) 2020-02-05T20:12:16.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:16.531+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.532426+0000) 2020-02-05T20:12:16.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:16.532+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.533800+0000) 2020-02-05T20:12:16.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:16.532+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.533738+0000) 2020-02-05T20:12:16.536 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:16.533+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.534633+0000) 2020-02-05T20:12:16.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:16.533+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.534591+0000) 2020-02-05T20:12:16.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:16.533+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.534662+0000) 2020-02-05T20:12:16.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:16.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:16.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:16.537+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.538581+0000) 2020-02-05T20:12:16.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:16.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.634581+0000) 2020-02-05T20:12:16.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:16.642+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.642793+0000) 2020-02-05T20:12:16.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:16.642+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.643040+0000) 2020-02-05T20:12:16.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:16.874+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:16.875013+0000) 2020-02-05T20:12:18.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:18.616+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:18.616828+0000) 2020-02-05T20:12:18.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:18.986+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:18.988012+0000) 2020-02-05T20:12:19.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:19.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:19.273767+0000) 2020-02-05T20:12:19.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:19.882+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:19.883650+0000) 2020-02-05T20:12:20.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:20.125+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:20.126053+0000) 2020-02-05T20:12:20.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:20.125+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:20.126051+0000) 2020-02-05T20:12:21.542 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:21.539+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.541108+0000) 2020-02-05T20:12:21.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:21.540+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.541639+0000) 2020-02-05T20:12:21.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:21.541+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.542418+0000) 2020-02-05T20:12:21.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:21.542+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.543308+0000) 2020-02-05T20:12:21.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:21.542+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.543531+0000) 2020-02-05T20:12:21.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:21.542+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.543521+0000) 2020-02-05T20:12:21.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:21.543+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.544477+0000) 2020-02-05T20:12:21.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:21.542+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.543878+0000) 2020-02-05T20:12:21.546 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:21.543+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.544509+0000) 2020-02-05T20:12:21.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:21.543+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.544704+0000) 2020-02-05T20:12:21.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:21.543+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.545166+0000) 2020-02-05T20:12:21.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:21.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49735 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:21.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:21.544+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.546105+0000) 2020-02-05T20:12:21.548 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:21.544+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.546096+0000) 2020-02-05T20:12:21.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:21.549+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:21.550437+0000) 2020-02-05T20:12:22.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:22.233+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:22.234644+0000) 2020-02-05T20:12:23.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:23.203+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:23.203671+0000) 2020-02-05T20:12:23.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:23.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:23.360230+0000) 2020-02-05T20:12:25.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:25.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:25.775907+0000) 2020-02-05T20:12:26.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:26.550+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.551899+0000) 2020-02-05T20:12:26.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:26.550+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.552264+0000) 2020-02-05T20:12:26.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:26.552+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.552596+0000) 2020-02-05T20:12:26.555 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:26.552+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.553745+0000) 2020-02-05T20:12:26.555 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:26.554+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.554488+0000) 2020-02-05T20:12:26.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:26.555+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.555066+0000) 2020-02-05T20:12:26.556 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:26.555+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.555418+0000) 2020-02-05T20:12:26.557 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:26.554+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.555791+0000) 2020-02-05T20:12:26.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:26.556+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.556338+0000) 2020-02-05T20:12:26.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:26.556+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.556680+0000) 2020-02-05T20:12:26.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:26.555+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.556787+0000) 2020-02-05T20:12:26.558 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:26.556+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.557223+0000) 2020-02-05T20:12:26.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:26.556+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.557818+0000) 2020-02-05T20:12:26.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:26.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:26.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:26.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.563790+0000) 2020-02-05T20:12:26.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:26.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.634797+0000) 2020-02-05T20:12:26.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:26.642+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.643012+0000) 2020-02-05T20:12:26.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:26.643+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.643275+0000) 2020-02-05T20:12:26.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:26.873+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:26.875231+0000) 2020-02-05T20:12:28.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:28.616+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:28.617087+0000) 2020-02-05T20:12:28.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:28.988+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:28.988225+0000) 2020-02-05T20:12:29.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:29.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:29.273992+0000) 2020-02-05T20:12:29.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:29.883+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:29.883944+0000) 2020-02-05T20:12:30.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:30.125+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:30.126354+0000) 2020-02-05T20:12:30.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:30.125+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:30.126355+0000) 2020-02-05T20:12:31.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:31.562+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.564046+0000) 2020-02-05T20:12:31.567 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:31.564+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.566268+0000) 2020-02-05T20:12:31.568 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:31.566+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.567305+0000) 2020-02-05T20:12:31.569 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:31.565+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.567054+0000) 2020-02-05T20:12:31.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:31.566+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.567769+0000) 2020-02-05T20:12:31.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:31.568+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.569368+0000) 2020-02-05T20:12:31.571 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:31.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.569825+0000) 2020-02-05T20:12:31.572 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:31.569+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:31.572 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:31.569+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.570855+0000) 2020-02-05T20:12:31.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:31.569+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.570712+0000) 2020-02-05T20:12:31.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:31.570+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.571925+0000) 2020-02-05T20:12:31.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:31.569+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.571333+0000) 2020-02-05T20:12:31.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:31.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.572515+0000) 2020-02-05T20:12:31.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:31.571+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.572532+0000) 2020-02-05T20:12:31.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:31.578+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:31.580038+0000) 2020-02-05T20:12:32.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:32.234+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:32.234877+0000) 2020-02-05T20:12:33.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:33.214+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:33.215477+0000) 2020-02-05T20:12:33.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:33.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:33.360474+0000) 2020-02-05T20:12:35.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:35.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:35.776173+0000) 2020-02-05T20:12:36.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:36.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.577602+0000) 2020-02-05T20:12:36.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:36.577+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.578139+0000) 2020-02-05T20:12:36.579 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:36.577+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.578552+0000) 2020-02-05T20:12:36.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:36.577+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.578938+0000) 2020-02-05T20:12:36.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:36.578+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.579453+0000) 2020-02-05T20:12:36.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:36.578+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.579704+0000) 2020-02-05T20:12:36.581 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:36.577+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.579358+0000) 2020-02-05T20:12:36.581 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:36.578+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.580178+0000) 2020-02-05T20:12:36.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:36.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.580529+0000) 2020-02-05T20:12:36.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:36.580+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.581435+0000) 2020-02-05T20:12:36.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:36.579+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.580963+0000) 2020-02-05T20:12:36.583 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:36.580+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.582407+0000) 2020-02-05T20:12:36.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:36.580+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.581889+0000) 2020-02-05T20:12:36.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:36.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:36.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:36.587+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.588199+0000) 2020-02-05T20:12:36.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:36.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.635015+0000) 2020-02-05T20:12:36.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:36.642+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.643254+0000) 2020-02-05T20:12:36.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:36.642+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.643478+0000) 2020-02-05T20:12:36.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:36.874+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:36.875483+0000) 2020-02-05T20:12:38.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:38.616+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:38.617278+0000) 2020-02-05T20:12:38.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:38.987+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:38.988468+0000) 2020-02-05T20:12:39.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:39.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:39.274264+0000) 2020-02-05T20:12:39.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:39.883+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:39.884180+0000) 2020-02-05T20:12:40.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:40.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:40.126696+0000) 2020-02-05T20:12:40.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:40.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:40.126698+0000) 2020-02-05T20:12:41.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:41.587+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.588530+0000) 2020-02-05T20:12:41.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:41.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.589559+0000) 2020-02-05T20:12:41.591 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:41.588+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.589554+0000) 2020-02-05T20:12:41.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:41.590+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.590625+0000) 2020-02-05T20:12:41.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:41.590+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.591062+0000) 2020-02-05T20:12:41.593 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:41.591+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.591712+0000) 2020-02-05T20:12:41.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:41.590+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.591956+0000) 2020-02-05T20:12:41.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:41.591+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.592654+0000) 2020-02-05T20:12:41.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:41.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.592714+0000) 2020-02-05T20:12:41.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:41.591+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.593146+0000) 2020-02-05T20:12:41.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:41.591+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.592978+0000) 2020-02-05T20:12:41.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:41.592+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.593657+0000) 2020-02-05T20:12:41.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:41.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.593437+0000) 2020-02-05T20:12:41.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:41.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:41.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:41.596+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:41.597657+0000) 2020-02-05T20:12:42.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:42.234+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:42.235084+0000) 2020-02-05T20:12:43.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:43.226+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:43.227390+0000) 2020-02-05T20:12:43.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:43.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:43.360707+0000) 2020-02-05T20:12:45.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:45.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:45.776413+0000) 2020-02-05T20:12:46.185 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:12:46.191 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:12:46.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:46.599+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.600227+0000) 2020-02-05T20:12:46.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:46.600+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.600921+0000) 2020-02-05T20:12:46.602 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:46.600+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.601389+0000) 2020-02-05T20:12:46.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:46.600+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.602037+0000) 2020-02-05T20:12:46.603 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:46.602+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.602520+0000) 2020-02-05T20:12:46.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:46.601+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.602899+0000) 2020-02-05T20:12:46.604 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:46.602+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.603273+0000) 2020-02-05T20:12:46.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:46.602+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.602784+0000) 2020-02-05T20:12:46.605 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:46.603+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.603624+0000) 2020-02-05T20:12:46.605 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:46.603+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.604256+0000) 2020-02-05T20:12:46.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:46.604+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.605186+0000) 2020-02-05T20:12:46.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:46.604+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.605531+0000) 2020-02-05T20:12:46.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:46.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49930 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:46.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:46.605+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.606383+0000) 2020-02-05T20:12:46.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:46.608+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.609143+0000) 2020-02-05T20:12:46.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:46.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.635265+0000) 2020-02-05T20:12:46.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:46.642+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.643487+0000) 2020-02-05T20:12:46.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:46.642+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.643670+0000) 2020-02-05T20:12:46.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:46.875+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:46.875717+0000) 2020-02-05T20:12:48.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:48.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:48.617535+0000) 2020-02-05T20:12:48.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:48.987+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:48.988745+0000) 2020-02-05T20:12:49.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:49.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:49.274518+0000) 2020-02-05T20:12:49.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:49.883+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:49.884407+0000) 2020-02-05T20:12:50.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:50.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:50.126956+0000) 2020-02-05T20:12:50.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:50.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:50.126946+0000) 2020-02-05T20:12:51.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:51.611+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.612283+0000) 2020-02-05T20:12:51.614 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:51.612+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.612786+0000) 2020-02-05T20:12:51.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:51.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.613799+0000) 2020-02-05T20:12:51.615 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:51.613+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.614242+0000) 2020-02-05T20:12:51.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:51.613+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.614822+0000) 2020-02-05T20:12:51.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:51.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.614737+0000) 2020-02-05T20:12:51.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:51.614+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.615214+0000) 2020-02-05T20:12:51.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:51.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.615906+0000) 2020-02-05T20:12:51.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:51.615+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.616368+0000) 2020-02-05T20:12:51.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:51.615+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.616924+0000) 2020-02-05T20:12:51.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:51.616+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.616633+0000) 2020-02-05T20:12:51.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:51.617+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.617712+0000) 2020-02-05T20:12:51.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:51.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 49970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:51.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:51.618+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.618559+0000) 2020-02-05T20:12:51.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:51.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:51.627449+0000) 2020-02-05T20:12:52.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:52.234+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:52.235301+0000) 2020-02-05T20:12:53.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:53.235+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:53.235820+0000) 2020-02-05T20:12:53.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:53.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:53.361002+0000) 2020-02-05T20:12:55.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:55.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:55.776649+0000) 2020-02-05T20:12:56.626 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:12:56.624+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.624679+0000) 2020-02-05T20:12:56.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:56.624+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.626006+0000) 2020-02-05T20:12:56.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:56.625+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.626503+0000) 2020-02-05T20:12:56.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:56.626+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.626789+0000) 2020-02-05T20:12:56.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:56.626+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.627780+0000) 2020-02-05T20:12:56.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:56.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.627890+0000) 2020-02-05T20:12:56.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:12:56.626+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.627371+0000) 2020-02-05T20:12:56.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:12:56.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:12:56.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:56.627+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.628306+0000) 2020-02-05T20:12:56.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:12:56.628+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.628957+0000) 2020-02-05T20:12:56.632 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:12:56.629+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.629862+0000) 2020-02-05T20:12:56.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:56.627+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.628839+0000) 2020-02-05T20:12:56.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:56.628+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.629190+0000) 2020-02-05T20:12:56.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:56.628+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.629674+0000) 2020-02-05T20:12:56.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:12:56.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.635410+0000) 2020-02-05T20:12:56.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:56.635+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.636195+0000) 2020-02-05T20:12:56.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:12:56.642+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.643653+0000) 2020-02-05T20:12:56.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:12:56.642+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.643875+0000) 2020-02-05T20:12:56.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:12:56.875+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:56.875922+0000) 2020-02-05T20:12:58.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:12:58.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:58.617794+0000) 2020-02-05T20:12:58.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:12:58.987+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:58.988994+0000) 2020-02-05T20:12:59.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:12:59.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:59.274721+0000) 2020-02-05T20:12:59.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:12:59.884+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:12:59.884661+0000) 2020-02-05T20:13:00.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:00.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:00.127248+0000) 2020-02-05T20:13:00.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:00.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:00.127248+0000) 2020-02-05T20:13:01.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:01.631+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.632612+0000) 2020-02-05T20:13:01.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:01.633+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.634007+0000) 2020-02-05T20:13:01.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:01.634+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.635118+0000) 2020-02-05T20:13:01.636 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:01.634+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.635395+0000) 2020-02-05T20:13:01.637 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:01.635+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.636328+0000) 2020-02-05T20:13:01.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:01.635+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.636894+0000) 2020-02-05T20:13:01.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:01.635+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.635793+0000) 2020-02-05T20:13:01.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:01.636+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.637356+0000) 2020-02-05T20:13:01.640 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:01.638+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50050 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:01.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:01.637+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.638743+0000) 2020-02-05T20:13:01.641 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:01.638+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.639133+0000) 2020-02-05T20:13:01.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:01.638+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.639373+0000) 2020-02-05T20:13:01.641 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:01.638+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.639109+0000) 2020-02-05T20:13:01.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:01.638+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.639876+0000) 2020-02-05T20:13:01.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:01.643+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:01.644206+0000) 2020-02-05T20:13:02.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:02.235+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:02.235533+0000) 2020-02-05T20:13:03.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:03.247+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:03.247634+0000) 2020-02-05T20:13:03.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:03.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:03.361249+0000) 2020-02-05T20:13:05.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:05.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:05.776882+0000) 2020-02-05T20:13:06.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:06.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.636021+0000) 2020-02-05T20:13:06.646 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:06.644+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.644657+0000) 2020-02-05T20:13:06.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:06.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.645026+0000) 2020-02-05T20:13:06.647 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:06.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.646176+0000) 2020-02-05T20:13:06.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:06.647+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.647573+0000) 2020-02-05T20:13:06.649 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:06.647+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.648137+0000) 2020-02-05T20:13:06.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:06.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.648689+0000) 2020-02-05T20:13:06.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:06.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.649973+0000) 2020-02-05T20:13:06.651 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:06.649+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.650300+0000) 2020-02-05T20:13:06.652 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:06.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:06.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:06.650+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.650618+0000) 2020-02-05T20:13:06.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:06.649+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.650652+0000) 2020-02-05T20:13:06.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:06.649+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.650985+0000) 2020-02-05T20:13:06.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:06.650+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.651341+0000) 2020-02-05T20:13:06.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:06.650+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.651689+0000) 2020-02-05T20:13:06.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:06.654+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.655497+0000) 2020-02-05T20:13:06.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:06.875+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:06.876194+0000) 2020-02-05T20:13:08.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:08.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:08.618079+0000) 2020-02-05T20:13:08.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:08.988+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:08.989254+0000) 2020-02-05T20:13:09.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:09.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:09.274981+0000) 2020-02-05T20:13:09.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:09.884+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:09.884887+0000) 2020-02-05T20:13:10.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:10.127+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:10.127544+0000) 2020-02-05T20:13:10.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:10.127+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:10.127543+0000) 2020-02-05T20:13:11.658 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:11.655+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.656245+0000) 2020-02-05T20:13:11.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:11.656+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.658000+0000) 2020-02-05T20:13:11.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:11.657+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.658872+0000) 2020-02-05T20:13:11.660 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:11.657+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.659031+0000) 2020-02-05T20:13:11.661 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:11.659+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.659519+0000) 2020-02-05T20:13:11.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:11.658+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.659184+0000) 2020-02-05T20:13:11.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:11.658+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.659561+0000) 2020-02-05T20:13:11.662 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:11.659+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.660098+0000) 2020-02-05T20:13:11.662 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:11.658+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.660018+0000) 2020-02-05T20:13:11.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:11.659+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.660174+0000) 2020-02-05T20:13:11.663 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:11.660+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.661090+0000) 2020-02-05T20:13:11.663 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:11.661+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.661469+0000) 2020-02-05T20:13:11.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:11.661+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.661885+0000) 2020-02-05T20:13:11.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:11.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:11.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:11.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:11.666487+0000) 2020-02-05T20:13:12.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:12.235+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:12.235761+0000) 2020-02-05T20:13:13.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:13.255+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:13.255642+0000) 2020-02-05T20:13:13.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:13.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:13.361512+0000) 2020-02-05T20:13:13.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:13.394+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:13.394312+0000) 2020-02-05T20:13:13.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:13.394+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:13.394729+0000) 2020-02-05T20:13:15.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:15.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:15.777146+0000) 2020-02-05T20:13:16.406 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:13:16.412 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:13:16.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:16.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.636222+0000) 2020-02-05T20:13:16.669 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:16.666+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.667777+0000) 2020-02-05T20:13:16.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:16.668+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.669289+0000) 2020-02-05T20:13:16.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:16.669+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.669967+0000) 2020-02-05T20:13:16.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:16.669+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.670513+0000) 2020-02-05T20:13:16.672 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:16.669+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.670218+0000) 2020-02-05T20:13:16.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:16.669+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.670845+0000) 2020-02-05T20:13:16.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:16.670+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.671502+0000) 2020-02-05T20:13:16.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:16.670+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.670992+0000) 2020-02-05T20:13:16.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:16.670+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.671939+0000) 2020-02-05T20:13:16.674 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:16.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.672231+0000) 2020-02-05T20:13:16.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:16.671+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.672368+0000) 2020-02-05T20:13:16.675 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:16.673+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.673910+0000) 2020-02-05T20:13:16.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:16.673+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.674166+0000) 2020-02-05T20:13:16.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:16.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:16.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:16.677+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.678829+0000) 2020-02-05T20:13:16.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:16.876+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:16.876458+0000) 2020-02-05T20:13:18.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:18.618+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:18.618316+0000) 2020-02-05T20:13:18.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:18.988+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:18.989482+0000) 2020-02-05T20:13:19.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:19.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:19.275212+0000) 2020-02-05T20:13:19.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:19.884+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:19.885124+0000) 2020-02-05T20:13:20.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:20.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:20.127829+0000) 2020-02-05T20:13:20.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:20.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:20.127824+0000) 2020-02-05T20:13:21.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:21.679+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.681290+0000) 2020-02-05T20:13:21.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:21.682+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.682310+0000) 2020-02-05T20:13:21.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:21.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.682544+0000) 2020-02-05T20:13:21.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:21.681+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.683177+0000) 2020-02-05T20:13:21.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:21.683+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.683499+0000) 2020-02-05T20:13:21.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:21.683+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.683671+0000) 2020-02-05T20:13:21.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:21.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.684059+0000) 2020-02-05T20:13:21.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:21.684+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.684363+0000) 2020-02-05T20:13:21.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:21.683+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50207 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:21.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:21.683+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.684516+0000) 2020-02-05T20:13:21.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:21.685+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.685792+0000) 2020-02-05T20:13:21.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:21.683+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.685033+0000) 2020-02-05T20:13:21.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:21.684+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.685470+0000) 2020-02-05T20:13:21.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:21.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.686370+0000) 2020-02-05T20:13:21.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:21.689+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:21.689404+0000) 2020-02-05T20:13:22.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:22.234+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:22.235997+0000) 2020-02-05T20:13:23.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:23.267+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:23.267425+0000) 2020-02-05T20:13:23.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:23.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:23.361766+0000) 2020-02-05T20:13:23.396 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:23.394+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:23.394597+0000) 2020-02-05T20:13:23.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:23.394+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:23.394935+0000) 2020-02-05T20:13:25.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:25.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:25.777382+0000) 2020-02-05T20:13:26.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:26.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.636419+0000) 2020-02-05T20:13:26.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:26.689+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.690188+0000) 2020-02-05T20:13:26.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:26.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.691132+0000) 2020-02-05T20:13:26.693 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:26.690+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.691819+0000) 2020-02-05T20:13:26.693 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:26.690+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.692062+0000) 2020-02-05T20:13:26.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:26.691+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.692474+0000) 2020-02-05T20:13:26.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:26.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.692920+0000) 2020-02-05T20:13:26.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:26.691+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.693127+0000) 2020-02-05T20:13:26.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:26.691+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.693160+0000) 2020-02-05T20:13:26.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:26.692+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.694041+0000) 2020-02-05T20:13:26.695 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:26.692+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.693731+0000) 2020-02-05T20:13:26.697 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:26.693+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.694428+0000) 2020-02-05T20:13:26.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:26.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.694688+0000) 2020-02-05T20:13:26.697 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:26.693+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.694524+0000) 2020-02-05T20:13:26.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:26.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50246 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:26.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:26.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.698550+0000) 2020-02-05T20:13:26.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:26.875+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:26.876712+0000) 2020-02-05T20:13:28.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:28.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:28.618573+0000) 2020-02-05T20:13:28.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:28.988+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:28.989732+0000) 2020-02-05T20:13:29.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:29.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:29.275469+0000) 2020-02-05T20:13:29.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:29.884+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:29.885307+0000) 2020-02-05T20:13:30.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:30.127+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:30.128127+0000) 2020-02-05T20:13:30.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:30.127+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:30.128127+0000) 2020-02-05T20:13:31.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:31.699+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.700571+0000) 2020-02-05T20:13:31.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:31.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.700778+0000) 2020-02-05T20:13:31.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:31.700+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.702196+0000) 2020-02-05T20:13:31.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:31.702+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.703897+0000) 2020-02-05T20:13:31.705 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:31.702+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.703869+0000) 2020-02-05T20:13:31.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:31.703+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.704294+0000) 2020-02-05T20:13:31.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:31.703+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.704638+0000) 2020-02-05T20:13:31.707 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:31.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50286 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:31.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:31.703+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.705360+0000) 2020-02-05T20:13:31.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:31.704+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.705854+0000) 2020-02-05T20:13:31.707 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:31.705+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.706264+0000) 2020-02-05T20:13:31.708 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:31.705+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.706521+0000) 2020-02-05T20:13:31.708 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:31.705+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.706934+0000) 2020-02-05T20:13:31.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:31.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.710677+0000) 2020-02-05T20:13:31.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:31.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:31.710984+0000) 2020-02-05T20:13:32.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:32.234+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:32.236273+0000) 2020-02-05T20:13:33.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:33.276+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:33.277785+0000) 2020-02-05T20:13:33.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:33.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:33.362035+0000) 2020-02-05T20:13:33.396 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:33.393+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:33.394877+0000) 2020-02-05T20:13:33.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:33.394+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:33.395170+0000) 2020-02-05T20:13:35.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:35.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:35.777628+0000) 2020-02-05T20:13:36.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:36.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.636667+0000) 2020-02-05T20:13:36.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:36.710+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.711578+0000) 2020-02-05T20:13:36.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:36.711+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.712321+0000) 2020-02-05T20:13:36.714 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:36.711+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.712863+0000) 2020-02-05T20:13:36.714 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:36.710+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.712177+0000) 2020-02-05T20:13:36.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:36.712+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.713456+0000) 2020-02-05T20:13:36.715 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:36.712+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.713928+0000) 2020-02-05T20:13:36.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:36.713+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.714392+0000) 2020-02-05T20:13:36.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:36.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.714818+0000) 2020-02-05T20:13:36.716 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:36.714+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.715280+0000) 2020-02-05T20:13:36.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:36.714+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.715590+0000) 2020-02-05T20:13:36.717 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:36.714+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.716011+0000) 2020-02-05T20:13:36.718 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:36.715+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.716851+0000) 2020-02-05T20:13:36.718 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:36.715+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.716991+0000) 2020-02-05T20:13:36.719 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:36.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50326 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:36.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:36.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.723116+0000) 2020-02-05T20:13:36.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:36.876+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:36.876965+0000) 2020-02-05T20:13:38.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:38.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:38.618818+0000) 2020-02-05T20:13:38.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:38.988+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:38.989990+0000) 2020-02-05T20:13:39.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:39.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:39.275654+0000) 2020-02-05T20:13:39.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:39.884+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:39.885541+0000) 2020-02-05T20:13:40.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:40.127+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:40.128470+0000) 2020-02-05T20:13:40.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:40.127+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:40.128470+0000) 2020-02-05T20:13:41.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:41.723+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.724593+0000) 2020-02-05T20:13:41.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:41.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.724967+0000) 2020-02-05T20:13:41.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:41.724+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.725723+0000) 2020-02-05T20:13:41.727 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:41.725+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.726270+0000) 2020-02-05T20:13:41.728 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:41.725+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.726484+0000) 2020-02-05T20:13:41.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:41.725+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.726863+0000) 2020-02-05T20:13:41.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:41.725+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.727002+0000) 2020-02-05T20:13:41.729 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:41.726+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.728057+0000) 2020-02-05T20:13:41.729 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:41.726+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.727801+0000) 2020-02-05T20:13:41.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:41.727+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.728389+0000) 2020-02-05T20:13:41.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:41.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:41.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:41.728+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.729185+0000) 2020-02-05T20:13:41.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:41.728+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.729560+0000) 2020-02-05T20:13:41.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:41.728+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.729736+0000) 2020-02-05T20:13:41.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:41.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:41.734999+0000) 2020-02-05T20:13:42.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:42.235+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:42.236512+0000) 2020-02-05T20:13:43.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:43.285+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:43.286949+0000) 2020-02-05T20:13:43.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:43.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:43.362323+0000) 2020-02-05T20:13:43.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:43.394+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:43.395144+0000) 2020-02-05T20:13:43.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:43.394+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:43.395394+0000) 2020-02-05T20:13:45.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:45.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:45.777862+0000) 2020-02-05T20:13:46.611 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:13:46.618 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:13:46.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:46.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.636861+0000) 2020-02-05T20:13:46.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:46.732+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.733794+0000) 2020-02-05T20:13:46.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:46.733+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.734737+0000) 2020-02-05T20:13:46.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:46.734+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.735684+0000) 2020-02-05T20:13:46.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:46.735+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.736411+0000) 2020-02-05T20:13:46.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:46.734+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.736242+0000) 2020-02-05T20:13:46.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:46.736+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.737283+0000) 2020-02-05T20:13:46.739 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:46.736+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.738053+0000) 2020-02-05T20:13:46.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:46.737+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.738309+0000) 2020-02-05T20:13:46.741 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:46.737+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.738616+0000) 2020-02-05T20:13:46.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:46.738+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.739580+0000) 2020-02-05T20:13:46.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:46.738+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.739280+0000) 2020-02-05T20:13:46.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:46.738+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.739994+0000) 2020-02-05T20:13:46.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:46.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:46.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:46.739+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.740446+0000) 2020-02-05T20:13:46.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:46.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.741599+0000) 2020-02-05T20:13:46.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:46.876+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:46.877202+0000) 2020-02-05T20:13:48.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:48.618+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:48.619070+0000) 2020-02-05T20:13:48.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:48.989+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:48.990260+0000) 2020-02-05T20:13:49.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:49.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:49.275899+0000) 2020-02-05T20:13:49.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:49.885+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:49.885772+0000) 2020-02-05T20:13:50.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:50.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:50.128774+0000) 2020-02-05T20:13:50.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:50.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:50.128774+0000) 2020-02-05T20:13:51.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:51.744+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.746200+0000) 2020-02-05T20:13:51.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:51.745+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.746883+0000) 2020-02-05T20:13:51.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:51.747+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.748954+0000) 2020-02-05T20:13:51.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:51.748+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.749501+0000) 2020-02-05T20:13:51.751 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:51.748+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.749750+0000) 2020-02-05T20:13:51.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:51.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.749911+0000) 2020-02-05T20:13:51.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:51.749+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.750263+0000) 2020-02-05T20:13:51.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:51.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.750724+0000) 2020-02-05T20:13:51.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:51.749+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.750578+0000) 2020-02-05T20:13:51.753 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:51.750+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.751151+0000) 2020-02-05T20:13:51.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:51.750+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.751589+0000) 2020-02-05T20:13:51.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:51.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.751968+0000) 2020-02-05T20:13:51.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:51.750+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.751939+0000) 2020-02-05T20:13:51.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:51.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50445 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:51.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:51.755+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:51.756573+0000) 2020-02-05T20:13:52.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:52.236+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:52.236731+0000) 2020-02-05T20:13:53.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:53.294+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:53.295320+0000) 2020-02-05T20:13:53.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:53.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:53.362635+0000) 2020-02-05T20:13:53.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:53.394+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:53.395327+0000) 2020-02-05T20:13:53.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:53.394+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:53.395604+0000) 2020-02-05T20:13:55.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:55.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:55.778135+0000) 2020-02-05T20:13:56.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:56.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.637126+0000) 2020-02-05T20:13:56.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:13:56.757+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.759274+0000) 2020-02-05T20:13:56.761 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:56.758+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.759962+0000) 2020-02-05T20:13:56.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:13:56.759+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.760698+0000) 2020-02-05T20:13:56.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:56.759+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.760942+0000) 2020-02-05T20:13:56.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:13:56.760+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.761462+0000) 2020-02-05T20:13:56.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:56.760+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.761456+0000) 2020-02-05T20:13:56.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:13:56.760+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.761867+0000) 2020-02-05T20:13:56.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:13:56.758+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.760315+0000) 2020-02-05T20:13:56.765 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:56.762+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.763534+0000) 2020-02-05T20:13:56.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:56.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.762348+0000) 2020-02-05T20:13:56.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:56.762+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.763531+0000) 2020-02-05T20:13:56.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:13:56.761+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.762921+0000) 2020-02-05T20:13:56.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:13:56.761+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.763016+0000) 2020-02-05T20:13:56.767 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:13:56.763+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50486 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:13:56.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:56.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.770063+0000) 2020-02-05T20:13:56.879 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:13:56.876+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:56.877468+0000) 2020-02-05T20:13:58.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:13:58.617+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:58.619310+0000) 2020-02-05T20:13:58.992 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:13:58.989+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:58.990506+0000) 2020-02-05T20:13:59.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:13:59.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:59.276147+0000) 2020-02-05T20:13:59.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:13:59.885+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:13:59.886008+0000) 2020-02-05T20:14:00.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:00.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:00.129078+0000) 2020-02-05T20:14:00.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:00.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:00.129079+0000) 2020-02-05T20:14:01.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:01.769+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.771069+0000) 2020-02-05T20:14:01.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:01.771+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.772129+0000) 2020-02-05T20:14:01.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:01.771+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.772470+0000) 2020-02-05T20:14:01.774 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:01.771+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.772644+0000) 2020-02-05T20:14:01.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:01.772+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.773279+0000) 2020-02-05T20:14:01.775 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:01.772+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.773729+0000) 2020-02-05T20:14:01.775 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:01.772+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.774020+0000) 2020-02-05T20:14:01.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:01.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.774357+0000) 2020-02-05T20:14:01.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:01.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.774835+0000) 2020-02-05T20:14:01.776 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:01.774+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.775231+0000) 2020-02-05T20:14:01.776 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:01.774+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.775261+0000) 2020-02-05T20:14:01.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:01.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50526 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:01.777 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:01.774+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.776282+0000) 2020-02-05T20:14:01.778 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:01.775+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.777241+0000) 2020-02-05T20:14:01.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:01.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:01.780847+0000) 2020-02-05T20:14:02.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:02.236+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:02.236993+0000) 2020-02-05T20:14:03.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:03.306+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:03.307143+0000) 2020-02-05T20:14:03.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:03.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:03.362875+0000) 2020-02-05T20:14:03.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:03.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:03.395568+0000) 2020-02-05T20:14:03.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:03.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:03.395806+0000) 2020-02-05T20:14:05.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:05.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:05.778367+0000) 2020-02-05T20:14:06.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:06.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.637387+0000) 2020-02-05T20:14:06.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:06.780+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.781979+0000) 2020-02-05T20:14:06.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:06.781+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.782805+0000) 2020-02-05T20:14:06.784 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:06.782+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.783532+0000) 2020-02-05T20:14:06.785 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:06.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.783273+0000) 2020-02-05T20:14:06.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:06.782+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.784204+0000) 2020-02-05T20:14:06.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:06.784+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.785704+0000) 2020-02-05T20:14:06.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:06.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.787304+0000) 2020-02-05T20:14:06.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:06.785+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.786168+0000) 2020-02-05T20:14:06.792 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:06.785+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.786435+0000) 2020-02-05T20:14:06.792 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:06.783+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.785192+0000) 2020-02-05T20:14:06.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:06.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:06.793 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:06.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.786934+0000) 2020-02-05T20:14:06.793 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:06.785+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.787046+0000) 2020-02-05T20:14:06.793 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:06.786+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.787634+0000) 2020-02-05T20:14:06.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:06.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.792464+0000) 2020-02-05T20:14:06.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:06.877+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:06.877716+0000) 2020-02-05T20:14:08.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:08.619+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:08.619557+0000) 2020-02-05T20:14:08.992 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:08.989+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:08.990787+0000) 2020-02-05T20:14:09.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:09.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:09.276344+0000) 2020-02-05T20:14:09.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:09.886+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:09.886269+0000) 2020-02-05T20:14:10.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:10.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:10.129361+0000) 2020-02-05T20:14:10.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:10.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:10.129361+0000) 2020-02-05T20:14:11.793 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:11.790+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.791506+0000) 2020-02-05T20:14:11.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:11.791+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.792404+0000) 2020-02-05T20:14:11.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:11.791+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.793025+0000) 2020-02-05T20:14:11.794 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:11.792+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.792804+0000) 2020-02-05T20:14:11.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:11.794+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.795482+0000) 2020-02-05T20:14:11.797 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:11.795+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.795761+0000) 2020-02-05T20:14:11.797 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:11.795+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.796047+0000) 2020-02-05T20:14:11.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:11.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.796866+0000) 2020-02-05T20:14:11.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:11.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50605 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:11.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:11.796+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.797075+0000) 2020-02-05T20:14:11.799 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:11.796+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.797336+0000) 2020-02-05T20:14:11.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:11.797+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.798409+0000) 2020-02-05T20:14:11.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:11.797+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.798612+0000) 2020-02-05T20:14:11.800 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:11.797+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.798100+0000) 2020-02-05T20:14:11.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:11.803+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:11.804676+0000) 2020-02-05T20:14:12.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:12.236+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:12.237234+0000) 2020-02-05T20:14:13.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:13.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:13.315576+0000) 2020-02-05T20:14:13.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:13.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:13.363117+0000) 2020-02-05T20:14:13.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:13.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:13.395800+0000) 2020-02-05T20:14:13.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:13.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:13.396010+0000) 2020-02-05T20:14:15.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:15.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:15.778607+0000) 2020-02-05T20:14:16.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:16.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.637628+0000) 2020-02-05T20:14:16.763 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:14:16.770 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:14:16.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:16.803+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.803516+0000) 2020-02-05T20:14:16.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:16.803+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.803829+0000) 2020-02-05T20:14:16.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:16.803+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.803490+0000) 2020-02-05T20:14:16.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:16.804+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.804767+0000) 2020-02-05T20:14:16.806 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:16.804+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.805076+0000) 2020-02-05T20:14:16.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:16.805+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.805633+0000) 2020-02-05T20:14:16.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:16.806+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.806376+0000) 2020-02-05T20:14:16.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:16.806+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.807024+0000) 2020-02-05T20:14:16.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:16.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50646 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:16.809 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:16.807+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.807595+0000) 2020-02-05T20:14:16.809 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:16.807+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.808162+0000) 2020-02-05T20:14:16.810 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:16.808+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.808716+0000) 2020-02-05T20:14:16.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:16.809+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.809110+0000) 2020-02-05T20:14:16.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:16.808+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.809078+0000) 2020-02-05T20:14:16.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:16.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.812814+0000) 2020-02-05T20:14:16.879 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:16.877+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:16.877977+0000) 2020-02-05T20:14:18.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:18.619+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:18.619805+0000) 2020-02-05T20:14:18.992 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:18.991+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:18.991061+0000) 2020-02-05T20:14:19.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:19.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:19.276592+0000) 2020-02-05T20:14:19.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:19.886+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:19.886505+0000) 2020-02-05T20:14:20.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:20.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:20.129673+0000) 2020-02-05T20:14:20.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:20.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:20.129673+0000) 2020-02-05T20:14:21.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:21.811+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.812803+0000) 2020-02-05T20:14:21.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:21.812+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.813720+0000) 2020-02-05T20:14:21.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:21.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.814285+0000) 2020-02-05T20:14:21.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:21.814+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.815475+0000) 2020-02-05T20:14:21.819 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:21.816+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.817107+0000) 2020-02-05T20:14:21.819 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:21.816+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.817241+0000) 2020-02-05T20:14:21.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:21.817+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.818296+0000) 2020-02-05T20:14:21.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:21.817+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.818539+0000) 2020-02-05T20:14:21.821 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:21.817+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.817958+0000) 2020-02-05T20:14:21.821 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:21.818+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.818463+0000) 2020-02-05T20:14:21.822 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:21.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:21.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:21.818+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.819289+0000) 2020-02-05T20:14:21.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:21.818+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.819276+0000) 2020-02-05T20:14:21.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:21.819+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.819905+0000) 2020-02-05T20:14:21.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:21.823+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:21.824562+0000) 2020-02-05T20:14:22.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:22.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:22.237490+0000) 2020-02-05T20:14:23.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:23.321+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:23.322654+0000) 2020-02-05T20:14:23.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:23.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:23.363350+0000) 2020-02-05T20:14:23.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:23.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:23.396028+0000) 2020-02-05T20:14:23.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:23.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:23.396292+0000) 2020-02-05T20:14:25.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:25.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:25.778857+0000) 2020-02-05T20:14:26.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:26.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.637856+0000) 2020-02-05T20:14:26.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:26.824+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.825272+0000) 2020-02-05T20:14:26.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:26.825+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.826284+0000) 2020-02-05T20:14:26.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:26.825+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.826841+0000) 2020-02-05T20:14:26.832 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:26.826+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.827868+0000) 2020-02-05T20:14:26.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:26.827+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.828492+0000) 2020-02-05T20:14:26.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:26.828+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.829353+0000) 2020-02-05T20:14:26.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:26.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.824511+0000) 2020-02-05T20:14:26.833 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:26.828+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.828994+0000) 2020-02-05T20:14:26.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:26.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.829573+0000) 2020-02-05T20:14:26.833 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:26.829+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.829941+0000) 2020-02-05T20:14:26.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:26.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.830443+0000) 2020-02-05T20:14:26.834 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:26.830+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.830884+0000) 2020-02-05T20:14:26.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:26.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:26.834 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:26.830+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.831279+0000) 2020-02-05T20:14:26.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:26.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.836412+0000) 2020-02-05T20:14:26.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:26.877+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:26.878193+0000) 2020-02-05T20:14:28.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:28.619+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:28.620082+0000) 2020-02-05T20:14:28.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:28.990+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:28.991369+0000) 2020-02-05T20:14:29.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:29.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:29.276833+0000) 2020-02-05T20:14:29.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:29.885+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:29.886741+0000) 2020-02-05T20:14:30.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:30.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:30.129950+0000) 2020-02-05T20:14:30.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:30.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:30.129949+0000) 2020-02-05T20:14:31.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:31.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.838235+0000) 2020-02-05T20:14:31.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:31.838+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.839582+0000) 2020-02-05T20:14:31.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:31.838+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.839960+0000) 2020-02-05T20:14:31.842 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:31.839+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.840141+0000) 2020-02-05T20:14:31.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:31.839+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.840618+0000) 2020-02-05T20:14:31.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:31.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.841057+0000) 2020-02-05T20:14:31.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:31.840+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.841191+0000) 2020-02-05T20:14:31.844 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:31.841+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.842298+0000) 2020-02-05T20:14:31.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:31.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50765 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:31.844 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:31.842+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.842613+0000) 2020-02-05T20:14:31.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:31.840+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.841844+0000) 2020-02-05T20:14:31.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:31.842+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.843367+0000) 2020-02-05T20:14:31.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:31.843+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.843545+0000) 2020-02-05T20:14:31.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:31.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.845156+0000) 2020-02-05T20:14:31.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:31.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:31.845611+0000) 2020-02-05T20:14:32.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:32.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:32.237725+0000) 2020-02-05T20:14:33.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:33.328+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:33.330036+0000) 2020-02-05T20:14:33.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:33.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:33.363646+0000) 2020-02-05T20:14:33.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:33.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:33.396282+0000) 2020-02-05T20:14:33.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:33.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:33.396480+0000) 2020-02-05T20:14:35.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:35.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:35.779118+0000) 2020-02-05T20:14:36.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:36.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.638116+0000) 2020-02-05T20:14:36.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:36.846+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.848025+0000) 2020-02-05T20:14:36.850 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:36.847+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.848959+0000) 2020-02-05T20:14:36.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:36.848+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.848602+0000) 2020-02-05T20:14:36.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:36.848+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.849426+0000) 2020-02-05T20:14:36.851 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:36.849+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.850304+0000) 2020-02-05T20:14:36.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:36.849+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.850871+0000) 2020-02-05T20:14:36.853 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:36.851+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.851661+0000) 2020-02-05T20:14:36.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:36.850+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.851862+0000) 2020-02-05T20:14:36.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:36.851+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.852584+0000) 2020-02-05T20:14:36.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:36.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:36.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:36.853+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.853697+0000) 2020-02-05T20:14:36.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:36.853+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.854060+0000) 2020-02-05T20:14:36.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:36.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.855321+0000) 2020-02-05T20:14:36.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:36.858+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.859196+0000) 2020-02-05T20:14:36.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:36.858+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.859541+0000) 2020-02-05T20:14:36.879 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:36.877+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:36.878387+0000) 2020-02-05T20:14:38.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:38.620+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:38.620345+0000) 2020-02-05T20:14:38.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:38.990+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:38.991552+0000) 2020-02-05T20:14:39.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:39.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:39.277059+0000) 2020-02-05T20:14:39.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:39.886+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:39.887017+0000) 2020-02-05T20:14:40.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:40.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:40.130236+0000) 2020-02-05T20:14:40.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:40.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:40.130234+0000) 2020-02-05T20:14:41.861 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:41.857+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.858901+0000) 2020-02-05T20:14:41.861 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:41.859+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.860321+0000) 2020-02-05T20:14:41.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:41.859+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.860851+0000) 2020-02-05T20:14:41.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:41.860+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.860835+0000) 2020-02-05T20:14:41.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:41.861+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.862057+0000) 2020-02-05T20:14:41.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:41.861+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.862251+0000) 2020-02-05T20:14:41.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:41.861+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.862488+0000) 2020-02-05T20:14:41.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:41.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.862896+0000) 2020-02-05T20:14:41.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:41.861+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.862983+0000) 2020-02-05T20:14:41.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:41.863+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.863611+0000) 2020-02-05T20:14:41.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:41.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.863993+0000) 2020-02-05T20:14:41.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:41.863+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.864026+0000) 2020-02-05T20:14:41.866 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:41.863+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:41.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:41.864+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.864805+0000) 2020-02-05T20:14:41.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:41.869+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:41.870696+0000) 2020-02-05T20:14:42.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:42.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:42.237951+0000) 2020-02-05T20:14:43.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:43.336+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:43.337491+0000) 2020-02-05T20:14:43.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:43.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:43.363996+0000) 2020-02-05T20:14:43.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:43.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:43.396513+0000) 2020-02-05T20:14:43.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:43.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:43.396725+0000) 2020-02-05T20:14:45.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:45.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:45.779346+0000) 2020-02-05T20:14:46.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:46.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.638343+0000) 2020-02-05T20:14:46.871 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:46.868+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.869919+0000) 2020-02-05T20:14:46.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:46.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.872476+0000) 2020-02-05T20:14:46.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:46.871+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.871834+0000) 2020-02-05T20:14:46.875 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:46.872+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.872827+0000) 2020-02-05T20:14:46.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:46.872+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.873164+0000) 2020-02-05T20:14:46.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:46.873+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.874304+0000) 2020-02-05T20:14:46.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:46.873+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.874729+0000) 2020-02-05T20:14:46.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:46.873+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.874158+0000) 2020-02-05T20:14:46.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:46.875+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.875709+0000) 2020-02-05T20:14:46.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:46.875+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.875683+0000) 2020-02-05T20:14:46.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:46.875+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.875961+0000) 2020-02-05T20:14:46.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:46.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:46.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:46.878+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.879321+0000) 2020-02-05T20:14:46.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:46.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.875294+0000) 2020-02-05T20:14:46.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:46.880+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.881885+0000) 2020-02-05T20:14:46.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:46.875+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:46.876534+0000) 2020-02-05T20:14:46.911 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:14:46.916 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:14:48.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:48.619+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:48.620568+0000) 2020-02-05T20:14:48.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:48.991+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:48.991814+0000) 2020-02-05T20:14:49.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:49.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:49.277333+0000) 2020-02-05T20:14:49.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:49.886+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:49.887277+0000) 2020-02-05T20:14:50.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:50.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:50.130524+0000) 2020-02-05T20:14:50.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:50.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:50.130524+0000) 2020-02-05T20:14:51.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:51.880+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.881338+0000) 2020-02-05T20:14:51.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:51.881+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.882166+0000) 2020-02-05T20:14:51.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:51.881+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.882460+0000) 2020-02-05T20:14:51.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:51.882+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.883068+0000) 2020-02-05T20:14:51.885 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:51.882+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.883634+0000) 2020-02-05T20:14:51.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:51.883+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.884392+0000) 2020-02-05T20:14:51.886 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:51.882+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.884229+0000) 2020-02-05T20:14:51.886 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:51.883+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.884756+0000) 2020-02-05T20:14:51.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:51.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.885018+0000) 2020-02-05T20:14:51.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:51.884+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.885664+0000) 2020-02-05T20:14:51.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:51.885+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.886678+0000) 2020-02-05T20:14:51.888 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:51.884+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.886367+0000) 2020-02-05T20:14:51.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:51.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:51.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:51.890+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.891554+0000) 2020-02-05T20:14:51.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:51.890+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:51.891803+0000) 2020-02-05T20:14:52.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:52.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:52.238183+0000) 2020-02-05T20:14:53.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:53.344+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:53.344498+0000) 2020-02-05T20:14:53.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:53.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:53.364234+0000) 2020-02-05T20:14:53.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:53.395+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:53.396795+0000) 2020-02-05T20:14:53.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:53.395+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:53.396945+0000) 2020-02-05T20:14:55.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:55.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:55.779581+0000) 2020-02-05T20:14:56.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:56.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.638595+0000) 2020-02-05T20:14:56.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:56.878+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.879520+0000) 2020-02-05T20:14:56.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:56.892+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.893706+0000) 2020-02-05T20:14:56.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:14:56.893+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.894517+0000) 2020-02-05T20:14:56.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:56.894+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.896296+0000) 2020-02-05T20:14:56.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:14:56.895+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.896572+0000) 2020-02-05T20:14:56.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:56.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.897153+0000) 2020-02-05T20:14:56.898 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:14:56.896+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.897500+0000) 2020-02-05T20:14:56.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:56.896+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.897869+0000) 2020-02-05T20:14:56.899 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:14:56.896+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.897986+0000) 2020-02-05T20:14:56.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:14:56.897+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.898695+0000) 2020-02-05T20:14:56.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:14:56.897+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.898548+0000) 2020-02-05T20:14:56.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:14:56.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 50966 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:14:56.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:14:56.897+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.899102+0000) 2020-02-05T20:14:56.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:14:56.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.899490+0000) 2020-02-05T20:14:56.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:56.900+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.901387+0000) 2020-02-05T20:14:56.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:56.901+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:56.902924+0000) 2020-02-05T20:14:58.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:14:58.619+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:58.620837+0000) 2020-02-05T20:14:58.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:14:58.991+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:58.992118+0000) 2020-02-05T20:14:59.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:14:59.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:59.277635+0000) 2020-02-05T20:14:59.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:14:59.887+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:14:59.887508+0000) 2020-02-05T20:15:00.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:00.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:00.130825+0000) 2020-02-05T20:15:00.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:00.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:00.130826+0000) 2020-02-05T20:15:01.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:01.905+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.906504+0000) 2020-02-05T20:15:01.909 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:01.904+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.906122+0000) 2020-02-05T20:15:01.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:01.905+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.907331+0000) 2020-02-05T20:15:01.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:01.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.908714+0000) 2020-02-05T20:15:01.910 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:01.907+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.908814+0000) 2020-02-05T20:15:01.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:01.908+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.909838+0000) 2020-02-05T20:15:01.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:01.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:01.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:01.908+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.909996+0000) 2020-02-05T20:15:01.913 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:01.908+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.910402+0000) 2020-02-05T20:15:01.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:01.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.910663+0000) 2020-02-05T20:15:01.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:01.908+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.909227+0000) 2020-02-05T20:15:01.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:01.909+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.910930+0000) 2020-02-05T20:15:01.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:01.909+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.910751+0000) 2020-02-05T20:15:01.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:01.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.916484+0000) 2020-02-05T20:15:01.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:01.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:01.916755+0000) 2020-02-05T20:15:02.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:02.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:02.238474+0000) 2020-02-05T20:15:03.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:03.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:03.352702+0000) 2020-02-05T20:15:03.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:03.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:03.364498+0000) 2020-02-05T20:15:03.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:03.396+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:03.397154+0000) 2020-02-05T20:15:03.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:03.396+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:03.397024+0000) 2020-02-05T20:15:05.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:05.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:05.779790+0000) 2020-02-05T20:15:06.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:06.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.638823+0000) 2020-02-05T20:15:06.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:06.878+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.879774+0000) 2020-02-05T20:15:06.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:06.916+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.918311+0000) 2020-02-05T20:15:06.922 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:06.917+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.918761+0000) 2020-02-05T20:15:06.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:06.918+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.919734+0000) 2020-02-05T20:15:06.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:06.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.915367+0000) 2020-02-05T20:15:06.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:06.918+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.919789+0000) 2020-02-05T20:15:06.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:06.915+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.916105+0000) 2020-02-05T20:15:06.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:06.915+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.916878+0000) 2020-02-05T20:15:06.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:06.916+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.917349+0000) 2020-02-05T20:15:06.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:06.918+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.919482+0000) 2020-02-05T20:15:06.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:06.919+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.920163+0000) 2020-02-05T20:15:06.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:06.919+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.920511+0000) 2020-02-05T20:15:06.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:06.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51047 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:06.925 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:06.920+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.921471+0000) 2020-02-05T20:15:06.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:06.920+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.921802+0000) 2020-02-05T20:15:06.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:06.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:06.924737+0000) 2020-02-05T20:15:08.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:08.620+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:08.621075+0000) 2020-02-05T20:15:08.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:08.992+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:08.992349+0000) 2020-02-05T20:15:09.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:09.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:09.277868+0000) 2020-02-05T20:15:09.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:09.887+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:09.887758+0000) 2020-02-05T20:15:10.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:10.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:10.131126+0000) 2020-02-05T20:15:10.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:10.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:10.131126+0000) 2020-02-05T20:15:11.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:11.926+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.926450+0000) 2020-02-05T20:15:11.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:11.925+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.927367+0000) 2020-02-05T20:15:11.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:11.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.928296+0000) 2020-02-05T20:15:11.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:11.926+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.928388+0000) 2020-02-05T20:15:11.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:11.927+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.928847+0000) 2020-02-05T20:15:11.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:11.928+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.928659+0000) 2020-02-05T20:15:11.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:11.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.929644+0000) 2020-02-05T20:15:11.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:11.929+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.929563+0000) 2020-02-05T20:15:11.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:11.930+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.930485+0000) 2020-02-05T20:15:11.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:11.929+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.930826+0000) 2020-02-05T20:15:11.941 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:11.931+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.931922+0000) 2020-02-05T20:15:11.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:11.930+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51086 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:11.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:11.930+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.932382+0000) 2020-02-05T20:15:11.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:11.931+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.932422+0000) 2020-02-05T20:15:11.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:11.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:11.937331+0000) 2020-02-05T20:15:12.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:12.237+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:12.238715+0000) 2020-02-05T20:15:13.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:13.360+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:13.361041+0000) 2020-02-05T20:15:13.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:13.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:13.364789+0000) 2020-02-05T20:15:13.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:13.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:13.397241+0000) 2020-02-05T20:15:13.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:13.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:13.397364+0000) 2020-02-05T20:15:15.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:15.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:15.780029+0000) 2020-02-05T20:15:16.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:16.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.639054+0000) 2020-02-05T20:15:16.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:16.879+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.880023+0000) 2020-02-05T20:15:16.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:16.936+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.937375+0000) 2020-02-05T20:15:16.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:16.937+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.938526+0000) 2020-02-05T20:15:16.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:16.938+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.939528+0000) 2020-02-05T20:15:16.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:16.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.940579+0000) 2020-02-05T20:15:16.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:16.939+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.940634+0000) 2020-02-05T20:15:16.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:16.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.940887+0000) 2020-02-05T20:15:16.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:16.940+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.941230+0000) 2020-02-05T20:15:16.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:16.940+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.941569+0000) 2020-02-05T20:15:16.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:16.940+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.941663+0000) 2020-02-05T20:15:16.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:16.941+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.942468+0000) 2020-02-05T20:15:16.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:16.940+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.942247+0000) 2020-02-05T20:15:16.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:16.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51130 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:16.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:16.942+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.943687+0000) 2020-02-05T20:15:16.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:16.942+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.943147+0000) 2020-02-05T20:15:16.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:16.947+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:16.948645+0000) 2020-02-05T20:15:17.118 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:15:17.125 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:15:18.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:18.620+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:18.621351+0000) 2020-02-05T20:15:18.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:18.991+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:18.992590+0000) 2020-02-05T20:15:19.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:19.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:19.278135+0000) 2020-02-05T20:15:19.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:19.887+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:19.887988+0000) 2020-02-05T20:15:20.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:20.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:20.131476+0000) 2020-02-05T20:15:20.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:20.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:20.131476+0000) 2020-02-05T20:15:21.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:21.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.948968+0000) 2020-02-05T20:15:21.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:21.948+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.950084+0000) 2020-02-05T20:15:21.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:21.949+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.950546+0000) 2020-02-05T20:15:21.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:21.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.951764+0000) 2020-02-05T20:15:21.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:21.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.952636+0000) 2020-02-05T20:15:21.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:21.951+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.952618+0000) 2020-02-05T20:15:21.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:21.951+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.953042+0000) 2020-02-05T20:15:21.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:21.952+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.954137+0000) 2020-02-05T20:15:21.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:21.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.953749+0000) 2020-02-05T20:15:21.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:21.953+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.954291+0000) 2020-02-05T20:15:21.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:21.953+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.954748+0000) 2020-02-05T20:15:21.957 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:21.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:21.958 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:21.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.955100+0000) 2020-02-05T20:15:21.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:21.954+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.955434+0000) 2020-02-05T20:15:21.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:21.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:21.961728+0000) 2020-02-05T20:15:22.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:22.238+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:22.238949+0000) 2020-02-05T20:15:23.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:23.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:23.365038+0000) 2020-02-05T20:15:23.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:23.368+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:23.369363+0000) 2020-02-05T20:15:23.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:23.396+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:23.397501+0000) 2020-02-05T20:15:23.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:23.396+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:23.397599+0000) 2020-02-05T20:15:25.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:25.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:25.780292+0000) 2020-02-05T20:15:26.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:26.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.639332+0000) 2020-02-05T20:15:26.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:26.879+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.880255+0000) 2020-02-05T20:15:26.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:26.959+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.960110+0000) 2020-02-05T20:15:26.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:26.959+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.960691+0000) 2020-02-05T20:15:26.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:26.959+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.961161+0000) 2020-02-05T20:15:26.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:26.960+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.961489+0000) 2020-02-05T20:15:26.964 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:26.961+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.962494+0000) 2020-02-05T20:15:26.964 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:26.960+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.962263+0000) 2020-02-05T20:15:26.965 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:26.961+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.962700+0000) 2020-02-05T20:15:26.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:26.962+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.963370+0000) 2020-02-05T20:15:26.967 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:26.962+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.963472+0000) 2020-02-05T20:15:26.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:26.962+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.963533+0000) 2020-02-05T20:15:26.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:26.963+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.964240+0000) 2020-02-05T20:15:26.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:26.962+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.963871+0000) 2020-02-05T20:15:26.968 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:26.963+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.964650+0000) 2020-02-05T20:15:26.968 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:26.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:26.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:26.967+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:26.969007+0000) 2020-02-05T20:15:28.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:28.621+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:28.621666+0000) 2020-02-05T20:15:28.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:28.992+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:28.992822+0000) 2020-02-05T20:15:29.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:29.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:29.278368+0000) 2020-02-05T20:15:29.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:29.887+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:29.888261+0000) 2020-02-05T20:15:30.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:30.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:30.131794+0000) 2020-02-05T20:15:30.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:30.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:30.131796+0000) 2020-02-05T20:15:31.972 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:31.969+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.970542+0000) 2020-02-05T20:15:31.973 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:31.970+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.971849+0000) 2020-02-05T20:15:31.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:31.971+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.972322+0000) 2020-02-05T20:15:31.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:31.971+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.972552+0000) 2020-02-05T20:15:31.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:31.971+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.973016+0000) 2020-02-05T20:15:31.974 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:31.972+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.973579+0000) 2020-02-05T20:15:31.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:31.972+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.973734+0000) 2020-02-05T20:15:31.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:31.972+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.974178+0000) 2020-02-05T20:15:31.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:31.973+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.974488+0000) 2020-02-05T20:15:31.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:31.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.975236+0000) 2020-02-05T20:15:31.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:31.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:31.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:31.975+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.976849+0000) 2020-02-05T20:15:31.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:31.975+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.976785+0000) 2020-02-05T20:15:31.978 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:31.975+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.977290+0000) 2020-02-05T20:15:31.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:31.981+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:31.982104+0000) 2020-02-05T20:15:32.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:32.238+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:32.239187+0000) 2020-02-05T20:15:33.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:33.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:33.365266+0000) 2020-02-05T20:15:33.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:33.376+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:33.377384+0000) 2020-02-05T20:15:33.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:33.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:33.397808+0000) 2020-02-05T20:15:33.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:33.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:33.397750+0000) 2020-02-05T20:15:35.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:35.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:35.780523+0000) 2020-02-05T20:15:36.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:36.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.639574+0000) 2020-02-05T20:15:36.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:36.880+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.880556+0000) 2020-02-05T20:15:36.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:36.980+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.981836+0000) 2020-02-05T20:15:36.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:36.981+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.982263+0000) 2020-02-05T20:15:36.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:36.981+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.982577+0000) 2020-02-05T20:15:36.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:36.982+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.983630+0000) 2020-02-05T20:15:36.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:36.983+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.984169+0000) 2020-02-05T20:15:36.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:36.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.984637+0000) 2020-02-05T20:15:36.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:36.982+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.984386+0000) 2020-02-05T20:15:36.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:36.984+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.985308+0000) 2020-02-05T20:15:36.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:36.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.985525+0000) 2020-02-05T20:15:36.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:36.984+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.986161+0000) 2020-02-05T20:15:36.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:36.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.986153+0000) 2020-02-05T20:15:36.988 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:36.985+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.986979+0000) 2020-02-05T20:15:36.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:36.986+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.987781+0000) 2020-02-05T20:15:36.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:36.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:36.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:36.992+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:36.993121+0000) 2020-02-05T20:15:38.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:38.621+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:38.621926+0000) 2020-02-05T20:15:38.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:38.992+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:38.993059+0000) 2020-02-05T20:15:39.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:39.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:39.278600+0000) 2020-02-05T20:15:39.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:39.887+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:39.888489+0000) 2020-02-05T20:15:40.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:40.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:40.132096+0000) 2020-02-05T20:15:40.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:40.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:40.132098+0000) 2020-02-05T20:15:41.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:41.992+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.993716+0000) 2020-02-05T20:15:41.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:41.995+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.996322+0000) 2020-02-05T20:15:41.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:41.994+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.994905+0000) 2020-02-05T20:15:41.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:41.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.995757+0000) 2020-02-05T20:15:41.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:41.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.996879+0000) 2020-02-05T20:15:41.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:41.996+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.997689+0000) 2020-02-05T20:15:42.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:41.997+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.998139+0000) 2020-02-05T20:15:42.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:41.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.998444+0000) 2020-02-05T20:15:42.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:41.997+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.998715+0000) 2020-02-05T20:15:42.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:41.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:42.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:41.998+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.999280+0000) 2020-02-05T20:15:42.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:41.999+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:41.999667+0000) 2020-02-05T20:15:42.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:41.999+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:42.000113+0000) 2020-02-05T20:15:42.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:41.999+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:42.000370+0000) 2020-02-05T20:15:42.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:42.005+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:42.006427+0000) 2020-02-05T20:15:42.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:42.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:42.239415+0000) 2020-02-05T20:15:43.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:43.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:43.365504+0000) 2020-02-05T20:15:43.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:43.384+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:43.385189+0000) 2020-02-05T20:15:43.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:43.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:43.398034+0000) 2020-02-05T20:15:43.400 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:43.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:43.398034+0000) 2020-02-05T20:15:45.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:45.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:45.780762+0000) 2020-02-05T20:15:46.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:46.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:46.639838+0000) 2020-02-05T20:15:46.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:46.880+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:46.880855+0000) 2020-02-05T20:15:47.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:47.003+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.004314+0000) 2020-02-05T20:15:47.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:47.005+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.006992+0000) 2020-02-05T20:15:47.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:47.006+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.007188+0000) 2020-02-05T20:15:47.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:47.006+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.007418+0000) 2020-02-05T20:15:47.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:47.006+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.007847+0000) 2020-02-05T20:15:47.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:47.007+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.008344+0000) 2020-02-05T20:15:47.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:47.007+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.008534+0000) 2020-02-05T20:15:47.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:47.008+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.009581+0000) 2020-02-05T20:15:47.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:47.008+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.009132+0000) 2020-02-05T20:15:47.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:47.009+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.010192+0000) 2020-02-05T20:15:47.012 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:47.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:47.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:47.010+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.010900+0000) 2020-02-05T20:15:47.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:47.010+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.011129+0000) 2020-02-05T20:15:47.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:47.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.012610+0000) 2020-02-05T20:15:47.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:47.014+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:47.015233+0000) 2020-02-05T20:15:47.339 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:15:47.346 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:15:48.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:48.621+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:48.622197+0000) 2020-02-05T20:15:48.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:48.992+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:48.993389+0000) 2020-02-05T20:15:49.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:49.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:49.278847+0000) 2020-02-05T20:15:49.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:49.888+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:49.888726+0000) 2020-02-05T20:15:50.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:50.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:50.132399+0000) 2020-02-05T20:15:50.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:50.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:50.132399+0000) 2020-02-05T20:15:52.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:52.015+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.015712+0000) 2020-02-05T20:15:52.018 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:52.015+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.016689+0000) 2020-02-05T20:15:52.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:52.016+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.017930+0000) 2020-02-05T20:15:52.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:52.018+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.019342+0000) 2020-02-05T20:15:52.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:52.019+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.020305+0000) 2020-02-05T20:15:52.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:52.021+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.021488+0000) 2020-02-05T20:15:52.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:52.021+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.021680+0000) 2020-02-05T20:15:52.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:52.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:52.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:52.022+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.022866+0000) 2020-02-05T20:15:52.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:52.017+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.018682+0000) 2020-02-05T20:15:52.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:52.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.021394+0000) 2020-02-05T20:15:52.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:52.021+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.022337+0000) 2020-02-05T20:15:52.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:52.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.022740+0000) 2020-02-05T20:15:52.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:52.025+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.026737+0000) 2020-02-05T20:15:52.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:52.026+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.028065+0000) 2020-02-05T20:15:52.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:52.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:52.239708+0000) 2020-02-05T20:15:53.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:53.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:53.365772+0000) 2020-02-05T20:15:53.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:53.393+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:53.393558+0000) 2020-02-05T20:15:53.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:53.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:53.398305+0000) 2020-02-05T20:15:53.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:53.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:53.398310+0000) 2020-02-05T20:15:55.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:55.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:55.780988+0000) 2020-02-05T20:15:56.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:56.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:56.640111+0000) 2020-02-05T20:15:56.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:56.880+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:56.881131+0000) 2020-02-05T20:15:57.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:15:57.027+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.028764+0000) 2020-02-05T20:15:57.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:15:57.028+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.029340+0000) 2020-02-05T20:15:57.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:15:57.029+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.030071+0000) 2020-02-05T20:15:57.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:15:57.029+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.030884+0000) 2020-02-05T20:15:57.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:15:57.031+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.031790+0000) 2020-02-05T20:15:57.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:57.031+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.032382+0000) 2020-02-05T20:15:57.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:57.031+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.032782+0000) 2020-02-05T20:15:57.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:15:57.033+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.033778+0000) 2020-02-05T20:15:57.035 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:57.033+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.034236+0000) 2020-02-05T20:15:57.036 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:15:57.033+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.034335+0000) 2020-02-05T20:15:57.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:57.033+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.034735+0000) 2020-02-05T20:15:57.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:15:57.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:15:57.037 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:15:57.034+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.034914+0000) 2020-02-05T20:15:57.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:57.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.037919+0000) 2020-02-05T20:15:57.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:57.037+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:57.038809+0000) 2020-02-05T20:15:58.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:15:58.622+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:58.622480+0000) 2020-02-05T20:15:58.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:15:58.993+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:58.993636+0000) 2020-02-05T20:15:59.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:15:59.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:59.279111+0000) 2020-02-05T20:15:59.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:15:59.888+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:15:59.888968+0000) 2020-02-05T20:16:00.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:00.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:00.132742+0000) 2020-02-05T20:16:00.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:00.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:00.132742+0000) 2020-02-05T20:16:02.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:02.038+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.039455+0000) 2020-02-05T20:16:02.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:02.039+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.040902+0000) 2020-02-05T20:16:02.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:02.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.041693+0000) 2020-02-05T20:16:02.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:02.041+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.041571+0000) 2020-02-05T20:16:02.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:02.042+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.042613+0000) 2020-02-05T20:16:02.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:02.042+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.043271+0000) 2020-02-05T20:16:02.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:02.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.043989+0000) 2020-02-05T20:16:02.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:02.043+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.043894+0000) 2020-02-05T20:16:02.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:02.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.044499+0000) 2020-02-05T20:16:02.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:02.043+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.044624+0000) 2020-02-05T20:16:02.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:02.044+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.045397+0000) 2020-02-05T20:16:02.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:02.045+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.046199+0000) 2020-02-05T20:16:02.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:02.046+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.046714+0000) 2020-02-05T20:16:02.048 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:02.046+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:02.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:02.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.051118+0000) 2020-02-05T20:16:02.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:02.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:02.239953+0000) 2020-02-05T20:16:03.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:03.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:03.366046+0000) 2020-02-05T20:16:03.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:03.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:03.398608+0000) 2020-02-05T20:16:03.400 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:03.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:03.398608+0000) 2020-02-05T20:16:03.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:03.401+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:03.401547+0000) 2020-02-05T20:16:05.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:05.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:05.781286+0000) 2020-02-05T20:16:06.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:06.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:06.640356+0000) 2020-02-05T20:16:06.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:06.881+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:06.881365+0000) 2020-02-05T20:16:07.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:07.052+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.052543+0000) 2020-02-05T20:16:07.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:07.053+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.053425+0000) 2020-02-05T20:16:07.055 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:07.053+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.054265+0000) 2020-02-05T20:16:07.056 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:07.054+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.054935+0000) 2020-02-05T20:16:07.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:07.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.055438+0000) 2020-02-05T20:16:07.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:07.055+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.056024+0000) 2020-02-05T20:16:07.058 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:07.057+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.057243+0000) 2020-02-05T20:16:07.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:07.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.057838+0000) 2020-02-05T20:16:07.059 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:07.057+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.058374+0000) 2020-02-05T20:16:07.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:07.058+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.058716+0000) 2020-02-05T20:16:07.061 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:07.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:07.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:07.059+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.059833+0000) 2020-02-05T20:16:07.061 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:07.059+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.060025+0000) 2020-02-05T20:16:07.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:07.060+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.060110+0000) 2020-02-05T20:16:07.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:07.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:07.061255+0000) 2020-02-05T20:16:08.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:08.622+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:08.622737+0000) 2020-02-05T20:16:08.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:08.993+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:08.993874+0000) 2020-02-05T20:16:09.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:09.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:09.279376+0000) 2020-02-05T20:16:09.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:09.888+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:09.889203+0000) 2020-02-05T20:16:10.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:10.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:10.133045+0000) 2020-02-05T20:16:10.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:10.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:10.133045+0000) 2020-02-05T20:16:12.067 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:12.065+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.066195+0000) 2020-02-05T20:16:12.068 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:12.066+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.067382+0000) 2020-02-05T20:16:12.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:12.067+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.068706+0000) 2020-02-05T20:16:12.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:12.067+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.068701+0000) 2020-02-05T20:16:12.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:12.068+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.069488+0000) 2020-02-05T20:16:12.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:12.069+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.069659+0000) 2020-02-05T20:16:12.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:12.068+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.070006+0000) 2020-02-05T20:16:12.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:12.069+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.070340+0000) 2020-02-05T20:16:12.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:12.069+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.070394+0000) 2020-02-05T20:16:12.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:12.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51585 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:12.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:12.069+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.070980+0000) 2020-02-05T20:16:12.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:12.070+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.070925+0000) 2020-02-05T20:16:12.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:12.070+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.071023+0000) 2020-02-05T20:16:12.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:12.070+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.071304+0000) 2020-02-05T20:16:12.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:12.073+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.074485+0000) 2020-02-05T20:16:12.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:12.240+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:12.240191+0000) 2020-02-05T20:16:13.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:13.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:13.366297+0000) 2020-02-05T20:16:13.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:13.397+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:13.398880+0000) 2020-02-05T20:16:13.400 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:13.397+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:13.398881+0000) 2020-02-05T20:16:13.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:13.407+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:13.408405+0000) 2020-02-05T20:16:15.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:15.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:15.781529+0000) 2020-02-05T20:16:16.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:16.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:16.640584+0000) 2020-02-05T20:16:16.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:16.881+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:16.881645+0000) 2020-02-05T20:16:17.077 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:17.074+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.075892+0000) 2020-02-05T20:16:17.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:17.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.076543+0000) 2020-02-05T20:16:17.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:17.076+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.077036+0000) 2020-02-05T20:16:17.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:17.077+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.077525+0000) 2020-02-05T20:16:17.079 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:17.076+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.077958+0000) 2020-02-05T20:16:17.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:17.077+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.078610+0000) 2020-02-05T20:16:17.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:17.079+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.080210+0000) 2020-02-05T20:16:17.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:17.078+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.079536+0000) 2020-02-05T20:16:17.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:17.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.079529+0000) 2020-02-05T20:16:17.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:17.078+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.079709+0000) 2020-02-05T20:16:17.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:17.080+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.080877+0000) 2020-02-05T20:16:17.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:17.080+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.081321+0000) 2020-02-05T20:16:17.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:17.081+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.081507+0000) 2020-02-05T20:16:17.084 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:17.082+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:17.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:17.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:17.085718+0000) 2020-02-05T20:16:17.577 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:16:17.584 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:16:18.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:18.621+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:18.623005+0000) 2020-02-05T20:16:18.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:18.993+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:18.994146+0000) 2020-02-05T20:16:19.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:19.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:19.279610+0000) 2020-02-05T20:16:19.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:19.888+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:19.889441+0000) 2020-02-05T20:16:20.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:20.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:20.133332+0000) 2020-02-05T20:16:20.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:20.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:20.133332+0000) 2020-02-05T20:16:22.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:22.087+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.088588+0000) 2020-02-05T20:16:22.090 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:22.087+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.089050+0000) 2020-02-05T20:16:22.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:22.088+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.089815+0000) 2020-02-05T20:16:22.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:22.089+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.090618+0000) 2020-02-05T20:16:22.092 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:22.089+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.091002+0000) 2020-02-05T20:16:22.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:22.090+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.091724+0000) 2020-02-05T20:16:22.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:22.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.092261+0000) 2020-02-05T20:16:22.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:22.091+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.092619+0000) 2020-02-05T20:16:22.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:22.092+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.093463+0000) 2020-02-05T20:16:22.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:22.092+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.093834+0000) 2020-02-05T20:16:22.095 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:22.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:22.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:22.092+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.094303+0000) 2020-02-05T20:16:22.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:22.092+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.094290+0000) 2020-02-05T20:16:22.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:22.093+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.095126+0000) 2020-02-05T20:16:22.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:22.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.097909+0000) 2020-02-05T20:16:22.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:22.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:22.240458+0000) 2020-02-05T20:16:23.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:23.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:23.366547+0000) 2020-02-05T20:16:23.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:23.398+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:23.399150+0000) 2020-02-05T20:16:23.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:23.398+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:23.399150+0000) 2020-02-05T20:16:23.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:23.415+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:23.416318+0000) 2020-02-05T20:16:25.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:25.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:25.781762+0000) 2020-02-05T20:16:26.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:26.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:26.640843+0000) 2020-02-05T20:16:26.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:26.880+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:26.881954+0000) 2020-02-05T20:16:27.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:27.098+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.099333+0000) 2020-02-05T20:16:27.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:27.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.100197+0000) 2020-02-05T20:16:27.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:27.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.101851+0000) 2020-02-05T20:16:27.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:27.100+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.102381+0000) 2020-02-05T20:16:27.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:27.101+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.102548+0000) 2020-02-05T20:16:27.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:27.101+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.102301+0000) 2020-02-05T20:16:27.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:27.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.102364+0000) 2020-02-05T20:16:27.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:27.102+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.103539+0000) 2020-02-05T20:16:27.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:27.102+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.103660+0000) 2020-02-05T20:16:27.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:27.102+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.104048+0000) 2020-02-05T20:16:27.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:27.103+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.104712+0000) 2020-02-05T20:16:27.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:27.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.103800+0000) 2020-02-05T20:16:27.108 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:27.103+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.104344+0000) 2020-02-05T20:16:27.109 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:27.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:27.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:27.107+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:27.108903+0000) 2020-02-05T20:16:28.625 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:28.621+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:28.623269+0000) 2020-02-05T20:16:28.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:28.993+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:28.994412+0000) 2020-02-05T20:16:29.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:29.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:29.279861+0000) 2020-02-05T20:16:29.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:29.889+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:29.889689+0000) 2020-02-05T20:16:30.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:30.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:30.133656+0000) 2020-02-05T20:16:30.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:30.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:30.133656+0000) 2020-02-05T20:16:32.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:32.111+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.112663+0000) 2020-02-05T20:16:32.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:32.112+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.113601+0000) 2020-02-05T20:16:32.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:32.112+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.114001+0000) 2020-02-05T20:16:32.116 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:32.113+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.114461+0000) 2020-02-05T20:16:32.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:32.113+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.114850+0000) 2020-02-05T20:16:32.116 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:32.114+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.115245+0000) 2020-02-05T20:16:32.117 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:32.114+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.115426+0000) 2020-02-05T20:16:32.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:32.114+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.115709+0000) 2020-02-05T20:16:32.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:32.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.115904+0000) 2020-02-05T20:16:32.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:32.115+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.116647+0000) 2020-02-05T20:16:32.118 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:32.115+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.117253+0000) 2020-02-05T20:16:32.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:32.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.116669+0000) 2020-02-05T20:16:32.119 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:32.115+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.117037+0000) 2020-02-05T20:16:32.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:32.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:32.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:32.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.122467+0000) 2020-02-05T20:16:32.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:32.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:32.240719+0000) 2020-02-05T20:16:33.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:33.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:33.366777+0000) 2020-02-05T20:16:33.401 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:33.398+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:33.399412+0000) 2020-02-05T20:16:33.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:33.398+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:33.399412+0000) 2020-02-05T20:16:33.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:33.424+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:33.424745+0000) 2020-02-05T20:16:35.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:35.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:35.781996+0000) 2020-02-05T20:16:36.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:36.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:36.641072+0000) 2020-02-05T20:16:36.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:36.880+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:36.882186+0000) 2020-02-05T20:16:37.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:37.123+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.124810+0000) 2020-02-05T20:16:37.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:37.124+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.125574+0000) 2020-02-05T20:16:37.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:37.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.126051+0000) 2020-02-05T20:16:37.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:37.125+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.126775+0000) 2020-02-05T20:16:37.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:37.126+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.127491+0000) 2020-02-05T20:16:37.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:37.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.128031+0000) 2020-02-05T20:16:37.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:37.127+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.128847+0000) 2020-02-05T20:16:37.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:37.127+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.129301+0000) 2020-02-05T20:16:37.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:37.127+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.128952+0000) 2020-02-05T20:16:37.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:37.128+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.129770+0000) 2020-02-05T20:16:37.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:37.129+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.130288+0000) 2020-02-05T20:16:37.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:37.129+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.130764+0000) 2020-02-05T20:16:37.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:37.128+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.130350+0000) 2020-02-05T20:16:37.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:37.128+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:37.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:37.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:37.134130+0000) 2020-02-05T20:16:38.625 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:38.622+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:38.623599+0000) 2020-02-05T20:16:38.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:38.994+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:38.994631+0000) 2020-02-05T20:16:39.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:39.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:39.280124+0000) 2020-02-05T20:16:39.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:39.889+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:39.889923+0000) 2020-02-05T20:16:40.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:40.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:40.133961+0000) 2020-02-05T20:16:40.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:40.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:40.133961+0000) 2020-02-05T20:16:42.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:42.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.136545+0000) 2020-02-05T20:16:42.139 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:42.135+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.136774+0000) 2020-02-05T20:16:42.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:42.136+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.137234+0000) 2020-02-05T20:16:42.140 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:42.136+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.137528+0000) 2020-02-05T20:16:42.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:42.136+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.138265+0000) 2020-02-05T20:16:42.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:42.137+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.139067+0000) 2020-02-05T20:16:42.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:42.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.139874+0000) 2020-02-05T20:16:42.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:42.138+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.140086+0000) 2020-02-05T20:16:42.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:42.139+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.140659+0000) 2020-02-05T20:16:42.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:42.140+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.141233+0000) 2020-02-05T20:16:42.146 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:42.140+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.141769+0000) 2020-02-05T20:16:42.146 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:42.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.142054+0000) 2020-02-05T20:16:42.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:42.140+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.142116+0000) 2020-02-05T20:16:42.147 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:42.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:42.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:42.144+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.145294+0000) 2020-02-05T20:16:42.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:42.240+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:42.240939+0000) 2020-02-05T20:16:43.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:43.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:43.367036+0000) 2020-02-05T20:16:43.401 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:43.399+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:43.399683+0000) 2020-02-05T20:16:43.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:43.399+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:43.399683+0000) 2020-02-05T20:16:43.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:43.432+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:43.433388+0000) 2020-02-05T20:16:45.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:45.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:45.782254+0000) 2020-02-05T20:16:46.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:46.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:46.641296+0000) 2020-02-05T20:16:46.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:46.881+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:46.882490+0000) 2020-02-05T20:16:47.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:47.147+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.148553+0000) 2020-02-05T20:16:47.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:47.147+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.148943+0000) 2020-02-05T20:16:47.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:47.149+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.150155+0000) 2020-02-05T20:16:47.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:47.149+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.150451+0000) 2020-02-05T20:16:47.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:47.149+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.150866+0000) 2020-02-05T20:16:47.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:47.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.151507+0000) 2020-02-05T20:16:47.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:47.150+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.151804+0000) 2020-02-05T20:16:47.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:47.151+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.152346+0000) 2020-02-05T20:16:47.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:47.153+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.154149+0000) 2020-02-05T20:16:47.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:47.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.159108+0000) 2020-02-05T20:16:47.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:47.151+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.153077+0000) 2020-02-05T20:16:47.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:47.151+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.152766+0000) 2020-02-05T20:16:47.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:47.151+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.153409+0000) 2020-02-05T20:16:47.164 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:47.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:47.154270+0000) 2020-02-05T20:16:47.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:47.153+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51876 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:47.706 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:16:47.713 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:16:48.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:48.623+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:48.623898+0000) 2020-02-05T20:16:48.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:48.994+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:48.994857+0000) 2020-02-05T20:16:49.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:49.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:49.280369+0000) 2020-02-05T20:16:49.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:49.889+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:49.890157+0000) 2020-02-05T20:16:50.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:50.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:50.134249+0000) 2020-02-05T20:16:50.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:50.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:50.134245+0000) 2020-02-05T20:16:52.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:52.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.161249+0000) 2020-02-05T20:16:52.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:52.160+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.161502+0000) 2020-02-05T20:16:52.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:52.161+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.162167+0000) 2020-02-05T20:16:52.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:52.161+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.162512+0000) 2020-02-05T20:16:52.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:52.161+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.162999+0000) 2020-02-05T20:16:52.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:52.162+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.163344+0000) 2020-02-05T20:16:52.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:52.162+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.163671+0000) 2020-02-05T20:16:52.167 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:52.162+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.164115+0000) 2020-02-05T20:16:52.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:52.164+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.165472+0000) 2020-02-05T20:16:52.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:52.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51917 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:52.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:52.165+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.166532+0000) 2020-02-05T20:16:52.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:52.164+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.166225+0000) 2020-02-05T20:16:52.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:52.165+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.166652+0000) 2020-02-05T20:16:52.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:52.165+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.166944+0000) 2020-02-05T20:16:52.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:52.170+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.171457+0000) 2020-02-05T20:16:52.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:52.240+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:52.241176+0000) 2020-02-05T20:16:53.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:53.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:53.367268+0000) 2020-02-05T20:16:53.401 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:53.399+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:53.399948+0000) 2020-02-05T20:16:53.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:53.399+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:53.399947+0000) 2020-02-05T20:16:53.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:53.441+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:53.441321+0000) 2020-02-05T20:16:55.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:55.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:55.782462+0000) 2020-02-05T20:16:56.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:56.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:56.641530+0000) 2020-02-05T20:16:56.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:56.882+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:56.882736+0000) 2020-02-05T20:16:57.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:16:57.170+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.171435+0000) 2020-02-05T20:16:57.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:16:57.170+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.171947+0000) 2020-02-05T20:16:57.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:57.172+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.173254+0000) 2020-02-05T20:16:57.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:16:57.172+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.173658+0000) 2020-02-05T20:16:57.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:57.173+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.174150+0000) 2020-02-05T20:16:57.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:16:57.173+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.174805+0000) 2020-02-05T20:16:57.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:57.174+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.175365+0000) 2020-02-05T20:16:57.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:57.174+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.176049+0000) 2020-02-05T20:16:57.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:57.175+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.176657+0000) 2020-02-05T20:16:57.180 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:16:57.175+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.177313+0000) 2020-02-05T20:16:57.180 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:16:57.176+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.177814+0000) 2020-02-05T20:16:57.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:16:57.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.177885+0000) 2020-02-05T20:16:57.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:16:57.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 51960 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:16:57.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:16:57.177+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.178937+0000) 2020-02-05T20:16:57.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:57.181+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:57.182124+0000) 2020-02-05T20:16:58.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:16:58.623+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:58.624164+0000) 2020-02-05T20:16:58.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:16:58.994+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:58.995122+0000) 2020-02-05T20:16:59.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:16:59.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:59.280619+0000) 2020-02-05T20:16:59.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:16:59.890+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:16:59.890391+0000) 2020-02-05T20:17:00.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:00.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:00.134540+0000) 2020-02-05T20:17:00.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:00.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:00.134542+0000) 2020-02-05T20:17:02.191 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:02.188+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.188069+0000) 2020-02-05T20:17:02.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:02.188+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.188536+0000) 2020-02-05T20:17:02.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:02.184+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.187503+0000) 2020-02-05T20:17:02.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:02.190+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.190301+0000) 2020-02-05T20:17:02.192 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:02.184+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.191573+0000) 2020-02-05T20:17:02.193 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:02.192+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.192555+0000) 2020-02-05T20:17:02.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:02.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.193004+0000) 2020-02-05T20:17:02.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:02.193+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.194011+0000) 2020-02-05T20:17:02.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:02.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.194494+0000) 2020-02-05T20:17:02.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:02.192+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.193882+0000) 2020-02-05T20:17:02.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:02.192+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.194098+0000) 2020-02-05T20:17:02.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:02.192+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.194272+0000) 2020-02-05T20:17:02.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:02.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:02.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:02.193+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.194540+0000) 2020-02-05T20:17:02.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:02.200+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.200787+0000) 2020-02-05T20:17:02.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:02.239+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:02.241378+0000) 2020-02-05T20:17:03.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:03.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:03.367540+0000) 2020-02-05T20:17:03.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:03.400+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:03.400186+0000) 2020-02-05T20:17:03.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:03.400+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:03.400186+0000) 2020-02-05T20:17:03.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:03.450+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:03.450071+0000) 2020-02-05T20:17:05.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:05.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:05.782712+0000) 2020-02-05T20:17:06.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:06.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:06.641762+0000) 2020-02-05T20:17:06.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:06.882+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:06.882986+0000) 2020-02-05T20:17:07.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:07.198+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.199436+0000) 2020-02-05T20:17:07.201 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:07.199+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.200434+0000) 2020-02-05T20:17:07.202 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:07.198+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.200252+0000) 2020-02-05T20:17:07.202 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:07.200+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.201541+0000) 2020-02-05T20:17:07.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:07.200+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.201921+0000) 2020-02-05T20:17:07.203 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:07.201+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.202541+0000) 2020-02-05T20:17:07.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:07.201+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.203060+0000) 2020-02-05T20:17:07.205 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:07.202+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.203484+0000) 2020-02-05T20:17:07.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:07.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.204011+0000) 2020-02-05T20:17:07.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:07.202+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.204207+0000) 2020-02-05T20:17:07.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:07.203+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.205031+0000) 2020-02-05T20:17:07.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:07.204+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.205692+0000) 2020-02-05T20:17:07.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:07.204+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.206330+0000) 2020-02-05T20:17:07.208 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:07.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:07.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:07.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:07.210630+0000) 2020-02-05T20:17:08.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:08.623+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:08.624398+0000) 2020-02-05T20:17:08.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:08.994+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:08.995353+0000) 2020-02-05T20:17:09.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:09.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:09.280900+0000) 2020-02-05T20:17:09.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:09.889+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:09.890625+0000) 2020-02-05T20:17:10.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:10.134+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:10.134775+0000) 2020-02-05T20:17:10.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:10.134+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:10.134775+0000) 2020-02-05T20:17:12.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:12.211+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.212597+0000) 2020-02-05T20:17:12.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:12.212+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.213338+0000) 2020-02-05T20:17:12.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:12.213+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.213895+0000) 2020-02-05T20:17:12.216 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:12.213+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.214440+0000) 2020-02-05T20:17:12.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:12.215+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.216221+0000) 2020-02-05T20:17:12.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:12.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.216966+0000) 2020-02-05T20:17:12.219 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:12.215+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.216544+0000) 2020-02-05T20:17:12.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:12.216+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.217344+0000) 2020-02-05T20:17:12.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:12.215+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.215552+0000) 2020-02-05T20:17:12.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:12.216+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.217114+0000) 2020-02-05T20:17:12.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:12.218+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.219241+0000) 2020-02-05T20:17:12.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:12.218+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.219138+0000) 2020-02-05T20:17:12.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:12.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:12.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:12.219+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.220373+0000) 2020-02-05T20:17:12.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:12.225+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.226464+0000) 2020-02-05T20:17:12.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:12.241+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:12.241666+0000) 2020-02-05T20:17:13.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:13.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:13.367798+0000) 2020-02-05T20:17:13.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:13.399+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:13.400460+0000) 2020-02-05T20:17:13.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:13.399+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:13.400463+0000) 2020-02-05T20:17:13.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:13.457+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:13.458423+0000) 2020-02-05T20:17:15.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:15.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:15.782905+0000) 2020-02-05T20:17:16.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:16.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:16.641998+0000) 2020-02-05T20:17:16.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:16.882+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:16.883291+0000) 2020-02-05T20:17:17.227 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:17.224+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.225690+0000) 2020-02-05T20:17:17.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:17.225+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.226374+0000) 2020-02-05T20:17:17.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:17.225+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.226745+0000) 2020-02-05T20:17:17.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:17.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.227466+0000) 2020-02-05T20:17:17.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:17.227+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.227757+0000) 2020-02-05T20:17:17.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:17.227+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.228144+0000) 2020-02-05T20:17:17.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:17.228+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.228587+0000) 2020-02-05T20:17:17.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:17.228+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.228779+0000) 2020-02-05T20:17:17.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:17.228+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.229608+0000) 2020-02-05T20:17:17.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:17.228+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.229075+0000) 2020-02-05T20:17:17.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:17.229+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.230214+0000) 2020-02-05T20:17:17.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:17.230+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.230676+0000) 2020-02-05T20:17:17.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:17.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:17.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:17.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.235837+0000) 2020-02-05T20:17:17.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:17.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:17.236127+0000) 2020-02-05T20:17:17.962 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:17:17.968 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:17:18.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:18.624+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:18.624665+0000) 2020-02-05T20:17:18.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:18.994+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:18.995621+0000) 2020-02-05T20:17:19.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:19.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:19.281144+0000) 2020-02-05T20:17:19.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:19.890+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:19.890878+0000) 2020-02-05T20:17:20.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:20.134+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:20.135074+0000) 2020-02-05T20:17:20.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:20.134+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:20.135074+0000) 2020-02-05T20:17:22.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:22.236+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.237367+0000) 2020-02-05T20:17:22.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:22.237+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.237633+0000) 2020-02-05T20:17:22.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:22.237+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.238275+0000) 2020-02-05T20:17:22.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:22.238+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.238772+0000) 2020-02-05T20:17:22.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:22.238+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.239231+0000) 2020-02-05T20:17:22.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:22.238+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.239158+0000) 2020-02-05T20:17:22.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:22.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.239594+0000) 2020-02-05T20:17:22.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:22.238+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.239967+0000) 2020-02-05T20:17:22.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:22.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.239902+0000) 2020-02-05T20:17:22.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:22.241+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.241808+0000) 2020-02-05T20:17:22.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:22.241+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.241966+0000) 2020-02-05T20:17:22.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:22.241+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.241903+0000) 2020-02-05T20:17:22.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:22.240+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.241246+0000) 2020-02-05T20:17:22.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:22.243+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:22.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:22.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.247624+0000) 2020-02-05T20:17:22.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:22.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:22.247909+0000) 2020-02-05T20:17:23.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:23.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:23.368054+0000) 2020-02-05T20:17:23.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:23.400+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:23.400738+0000) 2020-02-05T20:17:23.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:23.400+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:23.400738+0000) 2020-02-05T20:17:23.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:23.465+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:23.466504+0000) 2020-02-05T20:17:25.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:25.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:25.783134+0000) 2020-02-05T20:17:26.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:26.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:26.642311+0000) 2020-02-05T20:17:26.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:26.883+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:26.883538+0000) 2020-02-05T20:17:27.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:27.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.249337+0000) 2020-02-05T20:17:27.251 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:27.248+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.249766+0000) 2020-02-05T20:17:27.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:27.250+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.251089+0000) 2020-02-05T20:17:27.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:27.250+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.251486+0000) 2020-02-05T20:17:27.253 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:27.251+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.252194+0000) 2020-02-05T20:17:27.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:27.252+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.252955+0000) 2020-02-05T20:17:27.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:27.252+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.253374+0000) 2020-02-05T20:17:27.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:27.253+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.253700+0000) 2020-02-05T20:17:27.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:27.253+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.254381+0000) 2020-02-05T20:17:27.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:27.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.254537+0000) 2020-02-05T20:17:27.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:27.254+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.255020+0000) 2020-02-05T20:17:27.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:27.254+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.255026+0000) 2020-02-05T20:17:27.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:27.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:27.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:27.254+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.255345+0000) 2020-02-05T20:17:27.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:27.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:27.257874+0000) 2020-02-05T20:17:28.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:28.624+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:28.624924+0000) 2020-02-05T20:17:28.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:28.995+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:28.995864+0000) 2020-02-05T20:17:29.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:29.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:29.281399+0000) 2020-02-05T20:17:29.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:29.890+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:29.891119+0000) 2020-02-05T20:17:30.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:30.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:30.135378+0000) 2020-02-05T20:17:30.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:30.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:30.135378+0000) 2020-02-05T20:17:32.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:32.241+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.242027+0000) 2020-02-05T20:17:32.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:32.260+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.261382+0000) 2020-02-05T20:17:32.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:32.260+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.262019+0000) 2020-02-05T20:17:32.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:32.262+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.263480+0000) 2020-02-05T20:17:32.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:32.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.264427+0000) 2020-02-05T20:17:32.267 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:32.264+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.265145+0000) 2020-02-05T20:17:32.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:32.264+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.265536+0000) 2020-02-05T20:17:32.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:32.264+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.264817+0000) 2020-02-05T20:17:32.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:32.265+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.266222+0000) 2020-02-05T20:17:32.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:32.266+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.267470+0000) 2020-02-05T20:17:32.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:32.266+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.267042+0000) 2020-02-05T20:17:32.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:32.267+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.267868+0000) 2020-02-05T20:17:32.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:32.267+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.268098+0000) 2020-02-05T20:17:32.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:32.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:32.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:32.270+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.271139+0000) 2020-02-05T20:17:32.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:32.270+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:32.271477+0000) 2020-02-05T20:17:33.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:33.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:33.368265+0000) 2020-02-05T20:17:33.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:33.400+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:33.401014+0000) 2020-02-05T20:17:33.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:33.400+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:33.401014+0000) 2020-02-05T20:17:33.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:33.473+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:33.474734+0000) 2020-02-05T20:17:35.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:35.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:35.783380+0000) 2020-02-05T20:17:36.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:36.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:36.642543+0000) 2020-02-05T20:17:36.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:36.883+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:36.883731+0000) 2020-02-05T20:17:37.276 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:37.273+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.274740+0000) 2020-02-05T20:17:37.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:37.274+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.275662+0000) 2020-02-05T20:17:37.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:37.275+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.276355+0000) 2020-02-05T20:17:37.278 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:37.276+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.277145+0000) 2020-02-05T20:17:37.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:37.276+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.277533+0000) 2020-02-05T20:17:37.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:37.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.278468+0000) 2020-02-05T20:17:37.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:37.278+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.278828+0000) 2020-02-05T20:17:37.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:37.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.279433+0000) 2020-02-05T20:17:37.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:37.279+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.279621+0000) 2020-02-05T20:17:37.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:37.279+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:37.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:37.279+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.280327+0000) 2020-02-05T20:17:37.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:37.280+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.280858+0000) 2020-02-05T20:17:37.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:37.280+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.281380+0000) 2020-02-05T20:17:37.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:37.279+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.280689+0000) 2020-02-05T20:17:37.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:37.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:37.284034+0000) 2020-02-05T20:17:38.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:38.625+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:38.625219+0000) 2020-02-05T20:17:38.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:38.995+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:38.996136+0000) 2020-02-05T20:17:39.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:39.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:39.281634+0000) 2020-02-05T20:17:39.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:39.890+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:39.891354+0000) 2020-02-05T20:17:40.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:40.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:40.135723+0000) 2020-02-05T20:17:40.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:40.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:40.135723+0000) 2020-02-05T20:17:42.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:42.242+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.242284+0000) 2020-02-05T20:17:42.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:42.283+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.284304+0000) 2020-02-05T20:17:42.286 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:42.283+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.284810+0000) 2020-02-05T20:17:42.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:42.284+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.285405+0000) 2020-02-05T20:17:42.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:42.286+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.286736+0000) 2020-02-05T20:17:42.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:42.286+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.287329+0000) 2020-02-05T20:17:42.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:42.287+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.287832+0000) 2020-02-05T20:17:42.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:42.287+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.288299+0000) 2020-02-05T20:17:42.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:42.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.289560+0000) 2020-02-05T20:17:42.291 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:42.288+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.289810+0000) 2020-02-05T20:17:42.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:42.289+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.290509+0000) 2020-02-05T20:17:42.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:42.289+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.290154+0000) 2020-02-05T20:17:42.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:42.290+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.291285+0000) 2020-02-05T20:17:42.293 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:42.291+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.291773+0000) 2020-02-05T20:17:42.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:42.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:42.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:42.293+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:42.294419+0000) 2020-02-05T20:17:43.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:43.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:43.368504+0000) 2020-02-05T20:17:43.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:43.401+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:43.401341+0000) 2020-02-05T20:17:43.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:43.401+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:43.401341+0000) 2020-02-05T20:17:43.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:43.482+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:43.482642+0000) 2020-02-05T20:17:45.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:45.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:45.783645+0000) 2020-02-05T20:17:46.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:46.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:46.642813+0000) 2020-02-05T20:17:46.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:46.882+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:46.883979+0000) 2020-02-05T20:17:47.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:47.297+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.298152+0000) 2020-02-05T20:17:47.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:47.297+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.298687+0000) 2020-02-05T20:17:47.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:47.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.299365+0000) 2020-02-05T20:17:47.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:47.298+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.299383+0000) 2020-02-05T20:17:47.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:47.298+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.299788+0000) 2020-02-05T20:17:47.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:47.298+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.300310+0000) 2020-02-05T20:17:47.302 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:47.299+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.300224+0000) 2020-02-05T20:17:47.302 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:47.299+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.300730+0000) 2020-02-05T20:17:47.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:47.300+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.301732+0000) 2020-02-05T20:17:47.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:47.300+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.302187+0000) 2020-02-05T20:17:47.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:47.300+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.302234+0000) 2020-02-05T20:17:47.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:47.301+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.302709+0000) 2020-02-05T20:17:47.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:47.301+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.303170+0000) 2020-02-05T20:17:47.305 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:47.302+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:47.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:47.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:47.309093+0000) 2020-02-05T20:17:48.124 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:17:48.131 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:17:48.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:48.624+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:48.625511+0000) 2020-02-05T20:17:48.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:48.996+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:48.996366+0000) 2020-02-05T20:17:49.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:49.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:49.281871+0000) 2020-02-05T20:17:49.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:49.891+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:49.891582+0000) 2020-02-05T20:17:50.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:50.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:50.136037+0000) 2020-02-05T20:17:50.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:50.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:50.136037+0000) 2020-02-05T20:17:52.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:52.241+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.242555+0000) 2020-02-05T20:17:52.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:52.308+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.310333+0000) 2020-02-05T20:17:52.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:52.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.311409+0000) 2020-02-05T20:17:52.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:52.310+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.311826+0000) 2020-02-05T20:17:52.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:52.312+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.312219+0000) 2020-02-05T20:17:52.314 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:52.312+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.312516+0000) 2020-02-05T20:17:52.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:52.312+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.312896+0000) 2020-02-05T20:17:52.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:52.314+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.315063+0000) 2020-02-05T20:17:52.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:52.312+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.313948+0000) 2020-02-05T20:17:52.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:52.313+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.314767+0000) 2020-02-05T20:17:52.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:52.314+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.316028+0000) 2020-02-05T20:17:52.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:52.314+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.315679+0000) 2020-02-05T20:17:52.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:52.314+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.316256+0000) 2020-02-05T20:17:52.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:52.314+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.316352+0000) 2020-02-05T20:17:52.319 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:52.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:52.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:52.318+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:52.319275+0000) 2020-02-05T20:17:53.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:53.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:53.368745+0000) 2020-02-05T20:17:53.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:53.401+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:53.401598+0000) 2020-02-05T20:17:53.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:53.401+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:53.401599+0000) 2020-02-05T20:17:53.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:53.490+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:53.490941+0000) 2020-02-05T20:17:55.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:55.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:55.783883+0000) 2020-02-05T20:17:56.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:56.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:56.643064+0000) 2020-02-05T20:17:56.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:56.883+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:56.884238+0000) 2020-02-05T20:17:57.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:57.323+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.323520+0000) 2020-02-05T20:17:57.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:57.321+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.322981+0000) 2020-02-05T20:17:57.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:57.324+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.324593+0000) 2020-02-05T20:17:57.326 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:17:57.325+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.325394+0000) 2020-02-05T20:17:57.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:17:57.324+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.325715+0000) 2020-02-05T20:17:57.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:17:57.325+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.325808+0000) 2020-02-05T20:17:57.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:57.326+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.326519+0000) 2020-02-05T20:17:57.328 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:17:57.326+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.326982+0000) 2020-02-05T20:17:57.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:17:57.325+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.326677+0000) 2020-02-05T20:17:57.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:17:57.325+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.326746+0000) 2020-02-05T20:17:57.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:17:57.325+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.327271+0000) 2020-02-05T20:17:57.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:17:57.326+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.327878+0000) 2020-02-05T20:17:57.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:17:57.328+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:17:57.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:57.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.331447+0000) 2020-02-05T20:17:57.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:57.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:57.331648+0000) 2020-02-05T20:17:58.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:17:58.624+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:58.625821+0000) 2020-02-05T20:17:58.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:17:58.995+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:58.996609+0000) 2020-02-05T20:17:59.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:17:59.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:59.282131+0000) 2020-02-05T20:17:59.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:17:59.890+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:17:59.891852+0000) 2020-02-05T20:18:00.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:00.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:00.136323+0000) 2020-02-05T20:18:00.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:00.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:00.136322+0000) 2020-02-05T20:18:02.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:02.241+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.242814+0000) 2020-02-05T20:18:02.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:02.334+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.335386+0000) 2020-02-05T20:18:02.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:02.335+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.336480+0000) 2020-02-05T20:18:02.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:02.335+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.337233+0000) 2020-02-05T20:18:02.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:02.336+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.337416+0000) 2020-02-05T20:18:02.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:02.336+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.338033+0000) 2020-02-05T20:18:02.339 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:02.337+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.338144+0000) 2020-02-05T20:18:02.340 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:02.337+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.338491+0000) 2020-02-05T20:18:02.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:02.337+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.339176+0000) 2020-02-05T20:18:02.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:02.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.339124+0000) 2020-02-05T20:18:02.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:02.338+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.340199+0000) 2020-02-05T20:18:02.342 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:02.338+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.340044+0000) 2020-02-05T20:18:02.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:02.339+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.341279+0000) 2020-02-05T20:18:02.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:02.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:02.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:02.343+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.345044+0000) 2020-02-05T20:18:02.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:02.344+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:02.345552+0000) 2020-02-05T20:18:03.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:03.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:03.368987+0000) 2020-02-05T20:18:03.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:03.400+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:03.401864+0000) 2020-02-05T20:18:03.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:03.400+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:03.401864+0000) 2020-02-05T20:18:03.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:03.498+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:03.499016+0000) 2020-02-05T20:18:05.288 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 272, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() timeout 2020-02-05T20:18:05.290 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 766, in gevent._greenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 303, in copy_file_to copy_to_log(src, logger, capture=stream) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 272, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 108, in next line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() timeout 2020-02-05T20:18:05.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:05.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:05.784143+0000) 2020-02-05T20:18:06.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:06.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:06.643301+0000) 2020-02-05T20:18:06.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:06.883+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:06.884564+0000) 2020-02-05T20:18:07.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:07.346+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.347284+0000) 2020-02-05T20:18:07.350 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:07.347+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.348413+0000) 2020-02-05T20:18:07.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:07.346+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.347805+0000) 2020-02-05T20:18:07.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:07.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.350014+0000) 2020-02-05T20:18:07.352 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:07.349+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.350407+0000) 2020-02-05T20:18:07.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:07.349+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.350562+0000) 2020-02-05T20:18:07.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:07.349+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.350925+0000) 2020-02-05T20:18:07.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:07.350+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.351742+0000) 2020-02-05T20:18:07.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:07.350+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.351897+0000) 2020-02-05T20:18:07.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:07.350+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.351992+0000) 2020-02-05T20:18:07.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:07.350+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.352121+0000) 2020-02-05T20:18:07.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:07.350+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.352355+0000) 2020-02-05T20:18:07.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:07.351+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.353011+0000) 2020-02-05T20:18:07.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:07.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:07.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:07.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:07.357901+0000) 2020-02-05T20:18:08.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:08.625+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:08.626091+0000) 2020-02-05T20:18:08.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:08.995+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:08.996847+0000) 2020-02-05T20:18:09.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:09.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:09.282350+0000) 2020-02-05T20:18:09.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:09.891+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:09.892116+0000) 2020-02-05T20:18:10.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:10.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:10.136604+0000) 2020-02-05T20:18:10.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:10.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:10.136604+0000) 2020-02-05T20:18:12.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:12.242+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.243057+0000) 2020-02-05T20:18:12.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:12.359+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.360967+0000) 2020-02-05T20:18:12.363 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:12.360+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.361530+0000) 2020-02-05T20:18:12.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:12.360+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.361727+0000) 2020-02-05T20:18:12.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:12.361+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.362599+0000) 2020-02-05T20:18:12.364 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:12.362+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.363204+0000) 2020-02-05T20:18:12.364 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:12.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.363377+0000) 2020-02-05T20:18:12.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:12.362+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.363619+0000) 2020-02-05T20:18:12.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:12.362+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.364040+0000) 2020-02-05T20:18:12.366 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:12.363+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.364460+0000) 2020-02-05T20:18:12.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:12.363+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.364743+0000) 2020-02-05T20:18:12.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:12.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:12.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:12.363+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.365070+0000) 2020-02-05T20:18:12.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:12.364+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.365952+0000) 2020-02-05T20:18:12.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:12.365+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.366583+0000) 2020-02-05T20:18:12.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:12.370+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:12.371212+0000) 2020-02-05T20:18:13.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:13.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:13.369231+0000) 2020-02-05T20:18:13.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:13.401+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:13.402140+0000) 2020-02-05T20:18:13.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:13.401+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:13.402139+0000) 2020-02-05T20:18:13.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:13.510+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:13.511186+0000) 2020-02-05T20:18:15.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:15.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:15.784378+0000) 2020-02-05T20:18:16.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:16.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:16.643533+0000) 2020-02-05T20:18:16.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:16.883+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:16.884840+0000) 2020-02-05T20:18:17.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:17.370+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.371416+0000) 2020-02-05T20:18:17.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:17.371+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.372113+0000) 2020-02-05T20:18:17.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:17.371+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.372356+0000) 2020-02-05T20:18:17.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:17.371+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.373020+0000) 2020-02-05T20:18:17.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:17.372+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.373159+0000) 2020-02-05T20:18:17.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:17.372+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.373479+0000) 2020-02-05T20:18:17.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:17.373+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.374267+0000) 2020-02-05T20:18:17.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:17.373+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.374743+0000) 2020-02-05T20:18:17.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:17.373+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.375021+0000) 2020-02-05T20:18:17.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:17.373+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.375340+0000) 2020-02-05T20:18:17.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:17.374+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.375621+0000) 2020-02-05T20:18:17.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:17.374+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.376390+0000) 2020-02-05T20:18:17.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:17.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:17.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:17.375+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.376928+0000) 2020-02-05T20:18:17.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:17.378+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:17.379153+0000) 2020-02-05T20:18:18.312 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:18:18.318 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:18:18.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:18.625+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:18.626410+0000) 2020-02-05T20:18:18.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:18.996+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:18.997120+0000) 2020-02-05T20:18:19.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:19.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:19.282563+0000) 2020-02-05T20:18:19.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:19.891+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:19.892359+0000) 2020-02-05T20:18:20.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:20.136+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:20.136889+0000) 2020-02-05T20:18:20.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:20.136+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:20.136889+0000) 2020-02-05T20:18:22.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:22.242+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.243313+0000) 2020-02-05T20:18:22.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:22.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.383015+0000) 2020-02-05T20:18:22.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:22.383+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.384928+0000) 2020-02-05T20:18:22.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:22.383+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.384221+0000) 2020-02-05T20:18:22.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:22.383+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.385196+0000) 2020-02-05T20:18:22.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:22.384+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.385782+0000) 2020-02-05T20:18:22.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:22.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.385568+0000) 2020-02-05T20:18:22.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:22.384+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.386116+0000) 2020-02-05T20:18:22.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:22.385+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.386588+0000) 2020-02-05T20:18:22.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:22.385+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.386948+0000) 2020-02-05T20:18:22.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:22.385+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.387323+0000) 2020-02-05T20:18:22.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:22.386+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.387700+0000) 2020-02-05T20:18:22.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:22.387+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.388933+0000) 2020-02-05T20:18:22.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:22.387+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.389043+0000) 2020-02-05T20:18:22.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:22.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52673 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:22.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:22.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:22.394390+0000) 2020-02-05T20:18:23.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:23.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:23.369486+0000) 2020-02-05T20:18:23.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:23.401+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:23.402407+0000) 2020-02-05T20:18:23.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:23.401+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:23.402407+0000) 2020-02-05T20:18:23.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:23.518+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:23.519462+0000) 2020-02-05T20:18:25.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:25.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:25.784609+0000) 2020-02-05T20:18:26.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:26.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:26.643763+0000) 2020-02-05T20:18:26.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:26.884+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:26.885078+0000) 2020-02-05T20:18:27.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:27.394+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.395712+0000) 2020-02-05T20:18:27.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:27.394+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.395500+0000) 2020-02-05T20:18:27.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:27.394+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.396284+0000) 2020-02-05T20:18:27.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:27.395+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.397232+0000) 2020-02-05T20:18:27.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:27.396+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.397573+0000) 2020-02-05T20:18:27.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:27.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.398289+0000) 2020-02-05T20:18:27.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:27.397+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.398982+0000) 2020-02-05T20:18:27.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:27.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.399541+0000) 2020-02-05T20:18:27.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:27.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.399576+0000) 2020-02-05T20:18:27.402 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:27.398+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.400002+0000) 2020-02-05T20:18:27.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:27.399+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.400524+0000) 2020-02-05T20:18:27.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:27.399+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.400958+0000) 2020-02-05T20:18:27.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:27.399+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.401229+0000) 2020-02-05T20:18:27.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:27.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:27.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:27.405+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:27.406669+0000) 2020-02-05T20:18:28.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:28.626+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:28.626748+0000) 2020-02-05T20:18:28.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:28.996+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:28.997361+0000) 2020-02-05T20:18:29.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:29.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:29.282818+0000) 2020-02-05T20:18:29.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:29.892+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:29.892594+0000) 2020-02-05T20:18:30.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:30.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:30.137182+0000) 2020-02-05T20:18:30.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:30.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:30.137183+0000) 2020-02-05T20:18:32.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:32.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.243562+0000) 2020-02-05T20:18:32.409 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:32.406+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.407483+0000) 2020-02-05T20:18:32.410 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:32.407+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.408465+0000) 2020-02-05T20:18:32.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:32.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.409405+0000) 2020-02-05T20:18:32.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:32.408+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.409703+0000) 2020-02-05T20:18:32.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:32.409+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.410988+0000) 2020-02-05T20:18:32.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:32.410+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.411522+0000) 2020-02-05T20:18:32.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:32.411+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.412666+0000) 2020-02-05T20:18:32.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:32.411+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.412742+0000) 2020-02-05T20:18:32.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:32.412+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.413560+0000) 2020-02-05T20:18:32.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:32.413+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.414480+0000) 2020-02-05T20:18:32.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:32.413+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.414648+0000) 2020-02-05T20:18:32.416 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:32.413+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.414728+0000) 2020-02-05T20:18:32.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:32.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52757 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:32.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:32.413+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.414824+0000) 2020-02-05T20:18:32.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:32.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:32.416995+0000) 2020-02-05T20:18:33.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:33.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:33.369699+0000) 2020-02-05T20:18:33.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:33.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:33.402686+0000) 2020-02-05T20:18:33.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:33.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:33.402686+0000) 2020-02-05T20:18:33.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:33.527+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:33.527972+0000) 2020-02-05T20:18:35.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:35.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:35.784842+0000) 2020-02-05T20:18:36.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:36.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:36.643996+0000) 2020-02-05T20:18:36.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:36.884+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:36.885316+0000) 2020-02-05T20:18:37.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:37.419+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.420219+0000) 2020-02-05T20:18:37.422 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:37.419+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.420752+0000) 2020-02-05T20:18:37.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:37.420+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.421153+0000) 2020-02-05T20:18:37.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:37.421+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.421582+0000) 2020-02-05T20:18:37.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:37.421+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.422050+0000) 2020-02-05T20:18:37.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:37.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.423016+0000) 2020-02-05T20:18:37.425 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:37.422+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.423407+0000) 2020-02-05T20:18:37.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:37.424+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.424466+0000) 2020-02-05T20:18:37.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:37.423+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.424993+0000) 2020-02-05T20:18:37.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:37.423+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.425057+0000) 2020-02-05T20:18:37.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:37.425+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.426130+0000) 2020-02-05T20:18:37.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:37.425+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.425819+0000) 2020-02-05T20:18:37.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:37.425+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.426181+0000) 2020-02-05T20:18:37.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:37.426+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52800 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:37.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:37.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:37.429415+0000) 2020-02-05T20:18:38.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:38.626+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:38.627031+0000) 2020-02-05T20:18:38.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:38.997+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:38.997597+0000) 2020-02-05T20:18:39.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:39.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:39.283104+0000) 2020-02-05T20:18:39.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:39.892+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:39.892826+0000) 2020-02-05T20:18:40.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:40.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:40.137496+0000) 2020-02-05T20:18:40.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:40.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:40.137494+0000) 2020-02-05T20:18:42.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:42.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.243821+0000) 2020-02-05T20:18:42.439 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:42.435+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.436157+0000) 2020-02-05T20:18:42.439 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:42.436+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.436897+0000) 2020-02-05T20:18:42.439 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:42.430+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.431925+0000) 2020-02-05T20:18:42.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:42.432+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.433946+0000) 2020-02-05T20:18:42.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:42.433+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.434278+0000) 2020-02-05T20:18:42.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:42.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.434789+0000) 2020-02-05T20:18:42.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:42.435+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.436387+0000) 2020-02-05T20:18:42.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:42.433+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.435017+0000) 2020-02-05T20:18:42.440 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:42.434+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.435443+0000) 2020-02-05T20:18:42.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:42.437+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.437683+0000) 2020-02-05T20:18:42.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:42.436+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.437375+0000) 2020-02-05T20:18:42.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:42.437+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.438368+0000) 2020-02-05T20:18:42.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:42.438+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.438821+0000) 2020-02-05T20:18:42.442 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:42.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:42.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:42.443+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:42.444219+0000) 2020-02-05T20:18:43.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:43.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:43.369927+0000) 2020-02-05T20:18:43.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:43.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:43.402926+0000) 2020-02-05T20:18:43.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:43.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:43.402926+0000) 2020-02-05T20:18:43.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:43.535+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:43.535960+0000) 2020-02-05T20:18:45.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:45.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:45.785107+0000) 2020-02-05T20:18:46.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:46.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:46.644266+0000) 2020-02-05T20:18:46.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:46.885+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:46.885564+0000) 2020-02-05T20:18:47.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:47.444+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.445048+0000) 2020-02-05T20:18:47.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:47.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.446418+0000) 2020-02-05T20:18:47.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:47.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.448523+0000) 2020-02-05T20:18:47.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:47.449+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.450247+0000) 2020-02-05T20:18:47.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:47.449+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.449070+0000) 2020-02-05T20:18:47.455 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:47.449+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.449269+0000) 2020-02-05T20:18:47.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:47.449+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.450148+0000) 2020-02-05T20:18:47.456 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:47.449+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.450777+0000) 2020-02-05T20:18:47.456 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:47.446+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.447086+0000) 2020-02-05T20:18:47.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:47.446+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.447367+0000) 2020-02-05T20:18:47.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:47.449+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.450264+0000) 2020-02-05T20:18:47.457 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:47.450+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.450996+0000) 2020-02-05T20:18:47.457 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:47.450+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.451284+0000) 2020-02-05T20:18:47.457 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:47.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:47.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:47.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:47.456953+0000) 2020-02-05T20:18:48.501 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:18:48.507 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:18:48.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:48.626+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:48.627332+0000) 2020-02-05T20:18:48.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:48.997+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:48.997872+0000) 2020-02-05T20:18:49.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:49.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:49.283369+0000) 2020-02-05T20:18:49.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:49.893+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:49.893124+0000) 2020-02-05T20:18:50.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:50.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:50.137802+0000) 2020-02-05T20:18:50.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:50.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:50.137802+0000) 2020-02-05T20:18:52.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:52.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.244058+0000) 2020-02-05T20:18:52.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:52.455+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.456916+0000) 2020-02-05T20:18:52.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:52.457+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.457420+0000) 2020-02-05T20:18:52.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:52.457+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.458150+0000) 2020-02-05T20:18:52.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:52.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.458865+0000) 2020-02-05T20:18:52.466 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:52.459+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.459657+0000) 2020-02-05T20:18:52.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:52.461+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.462105+0000) 2020-02-05T20:18:52.466 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:52.461+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.462109+0000) 2020-02-05T20:18:52.466 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:52.462+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.463075+0000) 2020-02-05T20:18:52.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:52.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:52.467 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:52.458+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.459606+0000) 2020-02-05T20:18:52.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:52.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.461414+0000) 2020-02-05T20:18:52.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:52.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.462837+0000) 2020-02-05T20:18:52.467 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:52.461+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.462426+0000) 2020-02-05T20:18:52.467 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:52.462+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.463604+0000) 2020-02-05T20:18:52.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:52.468+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:52.469883+0000) 2020-02-05T20:18:53.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:53.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:53.370221+0000) 2020-02-05T20:18:53.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:53.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:53.403168+0000) 2020-02-05T20:18:53.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:53.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:53.403169+0000) 2020-02-05T20:18:53.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:53.543+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:53.544368+0000) 2020-02-05T20:18:55.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:55.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:55.785375+0000) 2020-02-05T20:18:56.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:56.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:56.644504+0000) 2020-02-05T20:18:56.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:56.885+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:56.885813+0000) 2020-02-05T20:18:57.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:57.468+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.469555+0000) 2020-02-05T20:18:57.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:18:57.468+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.469157+0000) 2020-02-05T20:18:57.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:57.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.471214+0000) 2020-02-05T20:18:57.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:57.470+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.471723+0000) 2020-02-05T20:18:57.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:57.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.472339+0000) 2020-02-05T20:18:57.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:18:57.472+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.472628+0000) 2020-02-05T20:18:57.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:18:57.472+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.473260+0000) 2020-02-05T20:18:57.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:18:57.473+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.474185+0000) 2020-02-05T20:18:57.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:18:57.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 52968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:18:57.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:18:57.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.474635+0000) 2020-02-05T20:18:57.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:18:57.474+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.475102+0000) 2020-02-05T20:18:57.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:18:57.474+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.475517+0000) 2020-02-05T20:18:57.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:18:57.475+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.475719+0000) 2020-02-05T20:18:57.480 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:57.478+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.478680+0000) 2020-02-05T20:18:57.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:57.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:57.479858+0000) 2020-02-05T20:18:58.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:18:58.627+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:58.627657+0000) 2020-02-05T20:18:58.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:18:58.997+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:58.998131+0000) 2020-02-05T20:18:59.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:18:59.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:59.283604+0000) 2020-02-05T20:18:59.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:18:59.892+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:18:59.893377+0000) 2020-02-05T20:19:00.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:00.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:00.138106+0000) 2020-02-05T20:19:00.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:00.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:00.138108+0000) 2020-02-05T20:19:02.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:02.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.244252+0000) 2020-02-05T20:19:02.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:02.479+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.480381+0000) 2020-02-05T20:19:02.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:02.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.481225+0000) 2020-02-05T20:19:02.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:02.480+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.481465+0000) 2020-02-05T20:19:02.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:02.481+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.482761+0000) 2020-02-05T20:19:02.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:02.482+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.483471+0000) 2020-02-05T20:19:02.485 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:02.483+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.484122+0000) 2020-02-05T20:19:02.486 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:02.483+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.484158+0000) 2020-02-05T20:19:02.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:02.483+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.484369+0000) 2020-02-05T20:19:02.489 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:02.484+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.485912+0000) 2020-02-05T20:19:02.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:02.485+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.486487+0000) 2020-02-05T20:19:02.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:02.485+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.486638+0000) 2020-02-05T20:19:02.490 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:02.484+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.485194+0000) 2020-02-05T20:19:02.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:02.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:02.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:02.486+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.487238+0000) 2020-02-05T20:19:02.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:02.486+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:02.487930+0000) 2020-02-05T20:19:03.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:03.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:03.370493+0000) 2020-02-05T20:19:03.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:03.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:03.403403+0000) 2020-02-05T20:19:03.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:03.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:03.403403+0000) 2020-02-05T20:19:03.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:03.550+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:03.551316+0000) 2020-02-05T20:19:05.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:05.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:05.785604+0000) 2020-02-05T20:19:06.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:06.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:06.644740+0000) 2020-02-05T20:19:06.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:06.885+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:06.886137+0000) 2020-02-05T20:19:07.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:07.491+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.492430+0000) 2020-02-05T20:19:07.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:07.491+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.492994+0000) 2020-02-05T20:19:07.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:07.493+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.493566+0000) 2020-02-05T20:19:07.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:07.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.494119+0000) 2020-02-05T20:19:07.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:07.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.494698+0000) 2020-02-05T20:19:07.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:07.494+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.495148+0000) 2020-02-05T20:19:07.497 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:07.494+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.495267+0000) 2020-02-05T20:19:07.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:07.494+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.495684+0000) 2020-02-05T20:19:07.497 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:07.495+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.496321+0000) 2020-02-05T20:19:07.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:07.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.496715+0000) 2020-02-05T20:19:07.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:07.496+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.496659+0000) 2020-02-05T20:19:07.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:07.495+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.495865+0000) 2020-02-05T20:19:07.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:07.496+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.497354+0000) 2020-02-05T20:19:07.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:07.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:07.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:07.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:07.502724+0000) 2020-02-05T20:19:08.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:08.627+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:08.627890+0000) 2020-02-05T20:19:09.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:08.997+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:08.998458+0000) 2020-02-05T20:19:09.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:09.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:09.283841+0000) 2020-02-05T20:19:09.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:09.892+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:09.893609+0000) 2020-02-05T20:19:10.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:10.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:10.138409+0000) 2020-02-05T20:19:10.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:10.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:10.138409+0000) 2020-02-05T20:19:12.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:12.244+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.244481+0000) 2020-02-05T20:19:12.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:12.502+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.503700+0000) 2020-02-05T20:19:12.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:12.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.504741+0000) 2020-02-05T20:19:12.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:12.504+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.505120+0000) 2020-02-05T20:19:12.507 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:12.504+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.505390+0000) 2020-02-05T20:19:12.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:12.505+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.505858+0000) 2020-02-05T20:19:12.507 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:12.505+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.506134+0000) 2020-02-05T20:19:12.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:12.506+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.506927+0000) 2020-02-05T20:19:12.508 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:12.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.507482+0000) 2020-02-05T20:19:12.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:12.507+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.507876+0000) 2020-02-05T20:19:12.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:12.507+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.508659+0000) 2020-02-05T20:19:12.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:12.508+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.508852+0000) 2020-02-05T20:19:12.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:12.508+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.509342+0000) 2020-02-05T20:19:12.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:12.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.509595+0000) 2020-02-05T20:19:12.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:12.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:12.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:12.513+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:12.514864+0000) 2020-02-05T20:19:13.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:13.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:13.370704+0000) 2020-02-05T20:19:13.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:13.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:13.403653+0000) 2020-02-05T20:19:13.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:13.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:13.403655+0000) 2020-02-05T20:19:13.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:13.558+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:13.559899+0000) 2020-02-05T20:19:15.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:15.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:15.785844+0000) 2020-02-05T20:19:16.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:16.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:16.644968+0000) 2020-02-05T20:19:16.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:16.885+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:16.886377+0000) 2020-02-05T20:19:17.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:17.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.515239+0000) 2020-02-05T20:19:17.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:17.514+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.515826+0000) 2020-02-05T20:19:17.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:17.516+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.518102+0000) 2020-02-05T20:19:17.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:17.517+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.518544+0000) 2020-02-05T20:19:17.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:17.519+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.520143+0000) 2020-02-05T20:19:17.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:17.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.520559+0000) 2020-02-05T20:19:17.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:17.519+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.520952+0000) 2020-02-05T20:19:17.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:17.520+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.521243+0000) 2020-02-05T20:19:17.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:17.520+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.521693+0000) 2020-02-05T20:19:17.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:17.520+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.522174+0000) 2020-02-05T20:19:17.524 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:17.521+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.522174+0000) 2020-02-05T20:19:17.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:17.520+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.522284+0000) 2020-02-05T20:19:17.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:17.521+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.522673+0000) 2020-02-05T20:19:17.525 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:17.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53136 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:17.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:17.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:17.525323+0000) 2020-02-05T20:19:18.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:18.627+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:18.628158+0000) 2020-02-05T20:19:18.673 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:19:18.679 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:19:19.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:18.998+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:18.998703+0000) 2020-02-05T20:19:19.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:19.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:19.284064+0000) 2020-02-05T20:19:19.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:19.893+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:19.893873+0000) 2020-02-05T20:19:20.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:20.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:20.138770+0000) 2020-02-05T20:19:20.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:20.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:20.138769+0000) 2020-02-05T20:19:22.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:22.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.244742+0000) 2020-02-05T20:19:22.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:22.527+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.529167+0000) 2020-02-05T20:19:22.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:22.527+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.529195+0000) 2020-02-05T20:19:22.532 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:22.530+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.531148+0000) 2020-02-05T20:19:22.533 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:22.530+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.531714+0000) 2020-02-05T20:19:22.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:22.530+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.531629+0000) 2020-02-05T20:19:22.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:22.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.532328+0000) 2020-02-05T20:19:22.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:22.531+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.532459+0000) 2020-02-05T20:19:22.535 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:22.532+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.533236+0000) 2020-02-05T20:19:22.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:22.532+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.533650+0000) 2020-02-05T20:19:22.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:22.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.534305+0000) 2020-02-05T20:19:22.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:22.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:22.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:22.532+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.534380+0000) 2020-02-05T20:19:22.544 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:22.533+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.534592+0000) 2020-02-05T20:19:22.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:22.533+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.535323+0000) 2020-02-05T20:19:22.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:22.537+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:22.539019+0000) 2020-02-05T20:19:23.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:23.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:23.370932+0000) 2020-02-05T20:19:23.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:23.402+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:23.403908+0000) 2020-02-05T20:19:23.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:23.402+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:23.403908+0000) 2020-02-05T20:19:23.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:23.567+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:23.568277+0000) 2020-02-05T20:19:25.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:25.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:25.786104+0000) 2020-02-05T20:19:26.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:26.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:26.645281+0000) 2020-02-05T20:19:26.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:26.885+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:26.886663+0000) 2020-02-05T20:19:27.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:27.540+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.542101+0000) 2020-02-05T20:19:27.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:27.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.542303+0000) 2020-02-05T20:19:27.545 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:27.541+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.542926+0000) 2020-02-05T20:19:27.545 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:27.541+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.542507+0000) 2020-02-05T20:19:27.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:27.542+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.543505+0000) 2020-02-05T20:19:27.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:27.541+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.543267+0000) 2020-02-05T20:19:27.548 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:27.543+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.544301+0000) 2020-02-05T20:19:27.548 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:27.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.544390+0000) 2020-02-05T20:19:27.549 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:27.543+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.544451+0000) 2020-02-05T20:19:27.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:27.543+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.545123+0000) 2020-02-05T20:19:27.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:27.543+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:27.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:27.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.546099+0000) 2020-02-05T20:19:27.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:27.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.546788+0000) 2020-02-05T20:19:27.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:27.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.551837+0000) 2020-02-05T20:19:27.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:27.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:27.553686+0000) 2020-02-05T20:19:28.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:28.627+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:28.628459+0000) 2020-02-05T20:19:29.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:28.998+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:28.998993+0000) 2020-02-05T20:19:29.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:29.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:29.284339+0000) 2020-02-05T20:19:29.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:29.893+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:29.894133+0000) 2020-02-05T20:19:30.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:30.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:30.139052+0000) 2020-02-05T20:19:30.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:30.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:30.139052+0000) 2020-02-05T20:19:32.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:32.244+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.244999+0000) 2020-02-05T20:19:32.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:32.549+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.550941+0000) 2020-02-05T20:19:32.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:32.550+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.551553+0000) 2020-02-05T20:19:32.553 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:32.550+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.552138+0000) 2020-02-05T20:19:32.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:32.551+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.552564+0000) 2020-02-05T20:19:32.555 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:32.552+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.553734+0000) 2020-02-05T20:19:32.555 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:32.553+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.554290+0000) 2020-02-05T20:19:32.556 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:32.554+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.555289+0000) 2020-02-05T20:19:32.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:32.554+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.555694+0000) 2020-02-05T20:19:32.557 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:32.554+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.555948+0000) 2020-02-05T20:19:32.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:32.555+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.556531+0000) 2020-02-05T20:19:32.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:32.555+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.556727+0000) 2020-02-05T20:19:32.559 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:32.555+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.557154+0000) 2020-02-05T20:19:32.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:32.556+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.557306+0000) 2020-02-05T20:19:32.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:32.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:32.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:32.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:32.560722+0000) 2020-02-05T20:19:33.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:33.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:33.371214+0000) 2020-02-05T20:19:33.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:33.403+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:33.404151+0000) 2020-02-05T20:19:33.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:33.403+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:33.404151+0000) 2020-02-05T20:19:33.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:33.576+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:33.576412+0000) 2020-02-05T20:19:35.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:35.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:35.786375+0000) 2020-02-05T20:19:36.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:36.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:36.645538+0000) 2020-02-05T20:19:36.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:36.886+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:36.886922+0000) 2020-02-05T20:19:37.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:37.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.563992+0000) 2020-02-05T20:19:37.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:37.563+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.564339+0000) 2020-02-05T20:19:37.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:37.564+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.565511+0000) 2020-02-05T20:19:37.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:37.564+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.565905+0000) 2020-02-05T20:19:37.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:37.565+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.566945+0000) 2020-02-05T20:19:37.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:37.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.566574+0000) 2020-02-05T20:19:37.569 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:37.566+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.567312+0000) 2020-02-05T20:19:37.570 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:37.566+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.567823+0000) 2020-02-05T20:19:37.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:37.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.568399+0000) 2020-02-05T20:19:37.570 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:37.567+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.568794+0000) 2020-02-05T20:19:37.571 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:37.567+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.569103+0000) 2020-02-05T20:19:37.571 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:37.567+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.569400+0000) 2020-02-05T20:19:37.571 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:37.568+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.569486+0000) 2020-02-05T20:19:37.572 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:37.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:37.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:37.574+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:37.575121+0000) 2020-02-05T20:19:38.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:38.627+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:38.628715+0000) 2020-02-05T20:19:39.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:38.999+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:38.999297+0000) 2020-02-05T20:19:39.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:39.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:39.284565+0000) 2020-02-05T20:19:39.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:39.894+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:39.894367+0000) 2020-02-05T20:19:40.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:40.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:40.139359+0000) 2020-02-05T20:19:40.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:40.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:40.139357+0000) 2020-02-05T20:19:42.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:42.243+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.245397+0000) 2020-02-05T20:19:42.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:42.575+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.575734+0000) 2020-02-05T20:19:42.578 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:42.574+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.576233+0000) 2020-02-05T20:19:42.580 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:42.576+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.578039+0000) 2020-02-05T20:19:42.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:42.578+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.578782+0000) 2020-02-05T20:19:42.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:42.577+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.579394+0000) 2020-02-05T20:19:42.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:42.579+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.579894+0000) 2020-02-05T20:19:42.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:42.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.580377+0000) 2020-02-05T20:19:42.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:42.580+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.580699+0000) 2020-02-05T20:19:42.584 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:42.578+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.580345+0000) 2020-02-05T20:19:42.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:42.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:42.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:42.580+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.581833+0000) 2020-02-05T20:19:42.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:42.580+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.582359+0000) 2020-02-05T20:19:42.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:42.581+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.581596+0000) 2020-02-05T20:19:42.585 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:42.581+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.581452+0000) 2020-02-05T20:19:42.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:42.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:42.587990+0000) 2020-02-05T20:19:43.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:43.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:43.371473+0000) 2020-02-05T20:19:43.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:43.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:43.404382+0000) 2020-02-05T20:19:43.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:43.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:43.404382+0000) 2020-02-05T20:19:43.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:43.584+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:43.584720+0000) 2020-02-05T20:19:45.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:45.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:45.786605+0000) 2020-02-05T20:19:46.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:46.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:46.645800+0000) 2020-02-05T20:19:46.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:46.886+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:46.887157+0000) 2020-02-05T20:19:47.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:47.586+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.587924+0000) 2020-02-05T20:19:47.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:47.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.588602+0000) 2020-02-05T20:19:47.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:47.587+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.589040+0000) 2020-02-05T20:19:47.591 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:47.588+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.590197+0000) 2020-02-05T20:19:47.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:47.589+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.590507+0000) 2020-02-05T20:19:47.592 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:47.589+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.590972+0000) 2020-02-05T20:19:47.592 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:47.590+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.591247+0000) 2020-02-05T20:19:47.595 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:47.590+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.592306+0000) 2020-02-05T20:19:47.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:47.591+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.592487+0000) 2020-02-05T20:19:47.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:47.591+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.592968+0000) 2020-02-05T20:19:47.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:47.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:47.596 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:47.592+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.593740+0000) 2020-02-05T20:19:47.596 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:47.591+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.592253+0000) 2020-02-05T20:19:47.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:47.591+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.593051+0000) 2020-02-05T20:19:47.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:47.597+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:47.598180+0000) 2020-02-05T20:19:48.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:48.628+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:48.629037+0000) 2020-02-05T20:19:48.835 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:19:48.842 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:19:49.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:48.998+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:48.999549+0000) 2020-02-05T20:19:49.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:49.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:49.284807+0000) 2020-02-05T20:19:49.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:49.893+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:49.894595+0000) 2020-02-05T20:19:50.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:50.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:50.139693+0000) 2020-02-05T20:19:50.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:50.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:50.139693+0000) 2020-02-05T20:19:52.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:52.245+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.245691+0000) 2020-02-05T20:19:52.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:52.598+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.599327+0000) 2020-02-05T20:19:52.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:52.598+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.599824+0000) 2020-02-05T20:19:52.602 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:52.598+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.600011+0000) 2020-02-05T20:19:52.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:52.599+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.600693+0000) 2020-02-05T20:19:52.603 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:52.600+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.601126+0000) 2020-02-05T20:19:52.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:52.600+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.601748+0000) 2020-02-05T20:19:52.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:52.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.602310+0000) 2020-02-05T20:19:52.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:52.601+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.602805+0000) 2020-02-05T20:19:52.605 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:52.602+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.603224+0000) 2020-02-05T20:19:52.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:52.602+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.603590+0000) 2020-02-05T20:19:52.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:52.603+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.605099+0000) 2020-02-05T20:19:52.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:52.604+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.605827+0000) 2020-02-05T20:19:52.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:52.604+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.605902+0000) 2020-02-05T20:19:52.608 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:52.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53433 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:52.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:52.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:52.608329+0000) 2020-02-05T20:19:53.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:53.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:53.371711+0000) 2020-02-05T20:19:53.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:53.403+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:53.404627+0000) 2020-02-05T20:19:53.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:53.403+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:53.404625+0000) 2020-02-05T20:19:53.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:53.591+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:53.592868+0000) 2020-02-05T20:19:55.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:55.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:55.786830+0000) 2020-02-05T20:19:56.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:56.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:56.646064+0000) 2020-02-05T20:19:56.889 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:56.886+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:56.887402+0000) 2020-02-05T20:19:57.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:57.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.612557+0000) 2020-02-05T20:19:57.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:57.611+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.613021+0000) 2020-02-05T20:19:57.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:19:57.612+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.613776+0000) 2020-02-05T20:19:57.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:19:57.612+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.614395+0000) 2020-02-05T20:19:57.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:19:57.613+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.614831+0000) 2020-02-05T20:19:57.617 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:19:57.614+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.615542+0000) 2020-02-05T20:19:57.617 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:19:57.613+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.615405+0000) 2020-02-05T20:19:57.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:57.615+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.616600+0000) 2020-02-05T20:19:57.618 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:19:57.615+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.616874+0000) 2020-02-05T20:19:57.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:19:57.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:19:57.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:57.616+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.617374+0000) 2020-02-05T20:19:57.619 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:19:57.616+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.617845+0000) 2020-02-05T20:19:57.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:57.616+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.618041+0000) 2020-02-05T20:19:57.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:19:57.617+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.618460+0000) 2020-02-05T20:19:57.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:57.621+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:57.622457+0000) 2020-02-05T20:19:58.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:19:58.628+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:58.629301+0000) 2020-02-05T20:19:59.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:19:58.998+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:58.999841+0000) 2020-02-05T20:19:59.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:19:59.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:59.285106+0000) 2020-02-05T20:19:59.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:19:59.894+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:19:59.894823+0000) 2020-02-05T20:20:00.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:00.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:00.139999+0000) 2020-02-05T20:20:00.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:00.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:00.139999+0000) 2020-02-05T20:20:02.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:02.245+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.245924+0000) 2020-02-05T20:20:02.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:02.621+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.622190+0000) 2020-02-05T20:20:02.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:02.621+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.622907+0000) 2020-02-05T20:20:02.625 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:02.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.624136+0000) 2020-02-05T20:20:02.626 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:02.623+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.624633+0000) 2020-02-05T20:20:02.626 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:02.623+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.625046+0000) 2020-02-05T20:20:02.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:02.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.626494+0000) 2020-02-05T20:20:02.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:02.625+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.626896+0000) 2020-02-05T20:20:02.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:02.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.627472+0000) 2020-02-05T20:20:02.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:02.626+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.627482+0000) 2020-02-05T20:20:02.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:02.626+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.628332+0000) 2020-02-05T20:20:02.630 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:02.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53518 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:02.630 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:02.627+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.629020+0000) 2020-02-05T20:20:02.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:02.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.629046+0000) 2020-02-05T20:20:02.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:02.628+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.629419+0000) 2020-02-05T20:20:02.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:02.632+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:02.633235+0000) 2020-02-05T20:20:03.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:03.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:03.371955+0000) 2020-02-05T20:20:03.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:03.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:03.404881+0000) 2020-02-05T20:20:03.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:03.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:03.404881+0000) 2020-02-05T20:20:03.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:03.600+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:03.601193+0000) 2020-02-05T20:20:05.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:05.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:05.787068+0000) 2020-02-05T20:20:06.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:06.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:06.646366+0000) 2020-02-05T20:20:06.889 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:06.887+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:06.887672+0000) 2020-02-05T20:20:07.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:07.634+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.634788+0000) 2020-02-05T20:20:07.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:07.633+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.635007+0000) 2020-02-05T20:20:07.637 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:07.635+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.636061+0000) 2020-02-05T20:20:07.638 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:07.636+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.636613+0000) 2020-02-05T20:20:07.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:07.636+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.637693+0000) 2020-02-05T20:20:07.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:07.637+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.638498+0000) 2020-02-05T20:20:07.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:07.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.638931+0000) 2020-02-05T20:20:07.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:07.638+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.639109+0000) 2020-02-05T20:20:07.642 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:07.639+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.639913+0000) 2020-02-05T20:20:07.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:07.638+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.639479+0000) 2020-02-05T20:20:07.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:07.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.640266+0000) 2020-02-05T20:20:07.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:07.640+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.640637+0000) 2020-02-05T20:20:07.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:07.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.641389+0000) 2020-02-05T20:20:07.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:07.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53559 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:07.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:07.646+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:07.647709+0000) 2020-02-05T20:20:08.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:08.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:08.629554+0000) 2020-02-05T20:20:09.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:08.999+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:09.000130+0000) 2020-02-05T20:20:09.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:09.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:09.285363+0000) 2020-02-05T20:20:09.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:09.893+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:09.895002+0000) 2020-02-05T20:20:10.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:10.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:10.140211+0000) 2020-02-05T20:20:10.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:10.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:10.140320+0000) 2020-02-05T20:20:12.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:12.245+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.246177+0000) 2020-02-05T20:20:12.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:12.645+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.646689+0000) 2020-02-05T20:20:12.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:12.646+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.647765+0000) 2020-02-05T20:20:12.650 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:12.648+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.649002+0000) 2020-02-05T20:20:12.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:12.648+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.649307+0000) 2020-02-05T20:20:12.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:12.648+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.649937+0000) 2020-02-05T20:20:12.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:12.649+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.650122+0000) 2020-02-05T20:20:12.652 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:12.650+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.650587+0000) 2020-02-05T20:20:12.653 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:12.651+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.652171+0000) 2020-02-05T20:20:12.654 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:12.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53602 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:12.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:12.652+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.653038+0000) 2020-02-05T20:20:12.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:12.652+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.653240+0000) 2020-02-05T20:20:12.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:12.653+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.653452+0000) 2020-02-05T20:20:12.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:12.653+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.653680+0000) 2020-02-05T20:20:12.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:12.655+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.656863+0000) 2020-02-05T20:20:12.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:12.656+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:12.657215+0000) 2020-02-05T20:20:13.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:13.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:13.372166+0000) 2020-02-05T20:20:13.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:13.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:13.405154+0000) 2020-02-05T20:20:13.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:13.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:13.405154+0000) 2020-02-05T20:20:13.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:13.611+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:13.612841+0000) 2020-02-05T20:20:15.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:15.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:15.787339+0000) 2020-02-05T20:20:16.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:16.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:16.646602+0000) 2020-02-05T20:20:16.889 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:16.887+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:16.887934+0000) 2020-02-05T20:20:17.659 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:17.656+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.657635+0000) 2020-02-05T20:20:17.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:17.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.659290+0000) 2020-02-05T20:20:17.661 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:17.659+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.660260+0000) 2020-02-05T20:20:17.662 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:17.659+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.660264+0000) 2020-02-05T20:20:17.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:17.660+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.660763+0000) 2020-02-05T20:20:17.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:17.660+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.661793+0000) 2020-02-05T20:20:17.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:17.661+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.662375+0000) 2020-02-05T20:20:17.664 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:17.662+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.662741+0000) 2020-02-05T20:20:17.665 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:17.663+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.663901+0000) 2020-02-05T20:20:17.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:17.663+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.664332+0000) 2020-02-05T20:20:17.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:17.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:17.666 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:17.663+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.664783+0000) 2020-02-05T20:20:17.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:17.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.664776+0000) 2020-02-05T20:20:17.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:17.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.666835+0000) 2020-02-05T20:20:17.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:17.670+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:17.671206+0000) 2020-02-05T20:20:18.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:18.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:18.629848+0000) 2020-02-05T20:20:19.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:18.999+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:19.000390+0000) 2020-02-05T20:20:19.024 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:20:19.031 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:20:19.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:19.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:19.285605+0000) 2020-02-05T20:20:19.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:19.894+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:19.895273+0000) 2020-02-05T20:20:20.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:20.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:20.140624+0000) 2020-02-05T20:20:20.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:20.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:20.140551+0000) 2020-02-05T20:20:22.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:22.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.246455+0000) 2020-02-05T20:20:22.671 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:22.668+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.669452+0000) 2020-02-05T20:20:22.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:22.669+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.670668+0000) 2020-02-05T20:20:22.672 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:22.670+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.671346+0000) 2020-02-05T20:20:22.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:22.671+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.671800+0000) 2020-02-05T20:20:22.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:22.671+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.672157+0000) 2020-02-05T20:20:22.674 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:22.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.672459+0000) 2020-02-05T20:20:22.674 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:22.672+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.673006+0000) 2020-02-05T20:20:22.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:22.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.673702+0000) 2020-02-05T20:20:22.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:22.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.674431+0000) 2020-02-05T20:20:22.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:22.674+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.675111+0000) 2020-02-05T20:20:22.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:22.674+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.675381+0000) 2020-02-05T20:20:22.677 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:22.675+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.675817+0000) 2020-02-05T20:20:22.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:22.676+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.676760+0000) 2020-02-05T20:20:22.678 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:22.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53687 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:22.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:22.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:22.683337+0000) 2020-02-05T20:20:23.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:23.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:23.372486+0000) 2020-02-05T20:20:23.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:23.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:23.405340+0000) 2020-02-05T20:20:23.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:23.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:23.405405+0000) 2020-02-05T20:20:23.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:23.620+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:23.621446+0000) 2020-02-05T20:20:25.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:25.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:25.787575+0000) 2020-02-05T20:20:26.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:26.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:26.646938+0000) 2020-02-05T20:20:26.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:26.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:26.888267+0000) 2020-02-05T20:20:27.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:27.680+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.681611+0000) 2020-02-05T20:20:27.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:27.681+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.682262+0000) 2020-02-05T20:20:27.684 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:27.682+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.683341+0000) 2020-02-05T20:20:27.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:27.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.684566+0000) 2020-02-05T20:20:27.687 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:27.685+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.685413+0000) 2020-02-05T20:20:27.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:27.684+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.685675+0000) 2020-02-05T20:20:27.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:27.685+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.686241+0000) 2020-02-05T20:20:27.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:27.685+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.686263+0000) 2020-02-05T20:20:27.688 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:27.686+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.686887+0000) 2020-02-05T20:20:27.689 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:27.686+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.687502+0000) 2020-02-05T20:20:27.689 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:27.687+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:27.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:27.687+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.688588+0000) 2020-02-05T20:20:27.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:27.689+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.689471+0000) 2020-02-05T20:20:27.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:27.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.689130+0000) 2020-02-05T20:20:27.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:27.694+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:27.695925+0000) 2020-02-05T20:20:28.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:28.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:28.630088+0000) 2020-02-05T20:20:29.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:29.000+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:29.000627+0000) 2020-02-05T20:20:29.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:29.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:29.285869+0000) 2020-02-05T20:20:29.897 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:29.894+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:29.895524+0000) 2020-02-05T20:20:30.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:30.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:30.140864+0000) 2020-02-05T20:20:30.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:30.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:30.140858+0000) 2020-02-05T20:20:32.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:32.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.246656+0000) 2020-02-05T20:20:32.695 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:32.692+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.693909+0000) 2020-02-05T20:20:32.696 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:32.694+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.695297+0000) 2020-02-05T20:20:32.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:32.694+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.695778+0000) 2020-02-05T20:20:32.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:32.695+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.696185+0000) 2020-02-05T20:20:32.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:32.696+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.696847+0000) 2020-02-05T20:20:32.699 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:32.697+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.697669+0000) 2020-02-05T20:20:32.700 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:32.698+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.698818+0000) 2020-02-05T20:20:32.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:32.698+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.699159+0000) 2020-02-05T20:20:32.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:32.698+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.699304+0000) 2020-02-05T20:20:32.701 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:32.698+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.699785+0000) 2020-02-05T20:20:32.702 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:32.699+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.699729+0000) 2020-02-05T20:20:32.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:32.700+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.700514+0000) 2020-02-05T20:20:32.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:32.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:32.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:32.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.704637+0000) 2020-02-05T20:20:32.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:32.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:32.705042+0000) 2020-02-05T20:20:33.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:33.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:33.372744+0000) 2020-02-05T20:20:33.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:33.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:33.405626+0000) 2020-02-05T20:20:33.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:33.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:33.405626+0000) 2020-02-05T20:20:33.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:33.629+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:33.629995+0000) 2020-02-05T20:20:35.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:35.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:35.787834+0000) 2020-02-05T20:20:36.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:36.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:36.647197+0000) 2020-02-05T20:20:36.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:36.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:36.888514+0000) 2020-02-05T20:20:37.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:37.705+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.705970+0000) 2020-02-05T20:20:37.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:37.707+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.707525+0000) 2020-02-05T20:20:37.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:37.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.708159+0000) 2020-02-05T20:20:37.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:37.708+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.708641+0000) 2020-02-05T20:20:37.710 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:37.708+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.708927+0000) 2020-02-05T20:20:37.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:37.709+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.709525+0000) 2020-02-05T20:20:37.712 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:37.710+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.710655+0000) 2020-02-05T20:20:37.712 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:37.710+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.711304+0000) 2020-02-05T20:20:37.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:37.711+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.711895+0000) 2020-02-05T20:20:37.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:37.712+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.712938+0000) 2020-02-05T20:20:37.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:37.712+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.713262+0000) 2020-02-05T20:20:37.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:37.713+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.713446+0000) 2020-02-05T20:20:37.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:37.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:37.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:37.718+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.718183+0000) 2020-02-05T20:20:37.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:37.718+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:37.718485+0000) 2020-02-05T20:20:38.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:38.630+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:38.630334+0000) 2020-02-05T20:20:39.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:39.000+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:39.000901+0000) 2020-02-05T20:20:39.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:39.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:39.286134+0000) 2020-02-05T20:20:39.897 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:39.895+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:39.895759+0000) 2020-02-05T20:20:40.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:40.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:40.141106+0000) 2020-02-05T20:20:40.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:40.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:40.141104+0000) 2020-02-05T20:20:42.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:42.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.246939+0000) 2020-02-05T20:20:42.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:42.717+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.718810+0000) 2020-02-05T20:20:42.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:42.718+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.719306+0000) 2020-02-05T20:20:42.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:42.720+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.721302+0000) 2020-02-05T20:20:42.723 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:42.720+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.720740+0000) 2020-02-05T20:20:42.724 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:42.721+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.721550+0000) 2020-02-05T20:20:42.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:42.721+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.722267+0000) 2020-02-05T20:20:42.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:42.722+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.722476+0000) 2020-02-05T20:20:42.725 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:42.722+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.723317+0000) 2020-02-05T20:20:42.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:42.722+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.723375+0000) 2020-02-05T20:20:42.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:42.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.723796+0000) 2020-02-05T20:20:42.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:42.723+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.724465+0000) 2020-02-05T20:20:42.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:42.724+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.725168+0000) 2020-02-05T20:20:42.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:42.725+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.726199+0000) 2020-02-05T20:20:42.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:42.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53853 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:42.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:42.729+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:42.730790+0000) 2020-02-05T20:20:43.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:43.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:43.372977+0000) 2020-02-05T20:20:43.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:43.404+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:43.405861+0000) 2020-02-05T20:20:43.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:43.404+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:43.405862+0000) 2020-02-05T20:20:43.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:43.637+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:43.638532+0000) 2020-02-05T20:20:45.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:45.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:45.788114+0000) 2020-02-05T20:20:46.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:46.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:46.647437+0000) 2020-02-05T20:20:46.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:46.887+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:46.888763+0000) 2020-02-05T20:20:47.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:47.730+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.731807+0000) 2020-02-05T20:20:47.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:47.731+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.732438+0000) 2020-02-05T20:20:47.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:47.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.733061+0000) 2020-02-05T20:20:47.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:47.732+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.733651+0000) 2020-02-05T20:20:47.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:47.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.734488+0000) 2020-02-05T20:20:47.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:47.733+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.734939+0000) 2020-02-05T20:20:47.737 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:47.734+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.735772+0000) 2020-02-05T20:20:47.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:47.735+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.736528+0000) 2020-02-05T20:20:47.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:47.735+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.736866+0000) 2020-02-05T20:20:47.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:47.735+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.737259+0000) 2020-02-05T20:20:47.739 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:47.736+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.737532+0000) 2020-02-05T20:20:47.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:47.736+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.738240+0000) 2020-02-05T20:20:47.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:47.737+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.738459+0000) 2020-02-05T20:20:47.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:47.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:47.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:47.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:47.745538+0000) 2020-02-05T20:20:48.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:48.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:48.630534+0000) 2020-02-05T20:20:49.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:49.000+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:49.001139+0000) 2020-02-05T20:20:49.238 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:20:49.245 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:20:49.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:49.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:49.286310+0000) 2020-02-05T20:20:49.897 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:49.895+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:49.895988+0000) 2020-02-05T20:20:50.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:50.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:50.141347+0000) 2020-02-05T20:20:50.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:50.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:50.141378+0000) 2020-02-05T20:20:52.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:52.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.247186+0000) 2020-02-05T20:20:52.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:52.746+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.747860+0000) 2020-02-05T20:20:52.751 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:52.747+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.749062+0000) 2020-02-05T20:20:52.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:52.745+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.746499+0000) 2020-02-05T20:20:52.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:52.745+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.746912+0000) 2020-02-05T20:20:52.751 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:52.746+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.747554+0000) 2020-02-05T20:20:52.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:52.747+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.748432+0000) 2020-02-05T20:20:52.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:52.748+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.749578+0000) 2020-02-05T20:20:52.753 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:52.748+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.749794+0000) 2020-02-05T20:20:52.753 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:52.748+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.749969+0000) 2020-02-05T20:20:52.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:52.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:52.753 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:52.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.750446+0000) 2020-02-05T20:20:52.754 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:52.749+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.751414+0000) 2020-02-05T20:20:52.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:52.750+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.751525+0000) 2020-02-05T20:20:52.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:52.752+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.754031+0000) 2020-02-05T20:20:52.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:52.753+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:52.754241+0000) 2020-02-05T20:20:53.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:53.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:53.373272+0000) 2020-02-05T20:20:53.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:53.405+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:53.406139+0000) 2020-02-05T20:20:53.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:53.405+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:53.406140+0000) 2020-02-05T20:20:53.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:53.646+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:53.647032+0000) 2020-02-05T20:20:55.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:55.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:55.788403+0000) 2020-02-05T20:20:56.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:56.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:56.647663+0000) 2020-02-05T20:20:56.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:56.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:56.889010+0000) 2020-02-05T20:20:57.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:20:57.755+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.756945+0000) 2020-02-05T20:20:57.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:57.756+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.757556+0000) 2020-02-05T20:20:57.760 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:57.757+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.758525+0000) 2020-02-05T20:20:57.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:20:57.757+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.758942+0000) 2020-02-05T20:20:57.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:20:57.758+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.759474+0000) 2020-02-05T20:20:57.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:20:57.758+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.759882+0000) 2020-02-05T20:20:57.761 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:20:57.759+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.760455+0000) 2020-02-05T20:20:57.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:20:57.759+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.760839+0000) 2020-02-05T20:20:57.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:20:57.759+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.761159+0000) 2020-02-05T20:20:57.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:57.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.762423+0000) 2020-02-05T20:20:57.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:57.761+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.762876+0000) 2020-02-05T20:20:57.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:57.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.763031+0000) 2020-02-05T20:20:57.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:20:57.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 53975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:20:57.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:20:57.762+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.763507+0000) 2020-02-05T20:20:57.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:57.765+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:57.766557+0000) 2020-02-05T20:20:58.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:20:58.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:58.630802+0000) 2020-02-05T20:20:59.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:20:59.000+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:59.001383+0000) 2020-02-05T20:20:59.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:20:59.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:59.286476+0000) 2020-02-05T20:20:59.897 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:20:59.895+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:20:59.896249+0000) 2020-02-05T20:21:00.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:00.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:00.141673+0000) 2020-02-05T20:21:00.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:00.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:00.141670+0000) 2020-02-05T20:21:02.249 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:02.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.247465+0000) 2020-02-05T20:21:02.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:02.767+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.768662+0000) 2020-02-05T20:21:02.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:02.769+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.770427+0000) 2020-02-05T20:21:02.773 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:02.770+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.771459+0000) 2020-02-05T20:21:02.774 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:02.770+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.771480+0000) 2020-02-05T20:21:02.774 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:02.770+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.772285+0000) 2020-02-05T20:21:02.774 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:02.771+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.772654+0000) 2020-02-05T20:21:02.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:02.770+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.771915+0000) 2020-02-05T20:21:02.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:02.771+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.773018+0000) 2020-02-05T20:21:02.775 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:02.772+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.773482+0000) 2020-02-05T20:21:02.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:02.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.774153+0000) 2020-02-05T20:21:02.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:02.773+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.774494+0000) 2020-02-05T20:21:02.776 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:02.773+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.775122+0000) 2020-02-05T20:21:02.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:02.774+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.775552+0000) 2020-02-05T20:21:02.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:02.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:02.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:02.778+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:02.779684+0000) 2020-02-05T20:21:03.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:03.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:03.373512+0000) 2020-02-05T20:21:03.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:03.405+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:03.406379+0000) 2020-02-05T20:21:03.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:03.405+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:03.406379+0000) 2020-02-05T20:21:03.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:03.654+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:03.655295+0000) 2020-02-05T20:21:05.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:05.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:05.788644+0000) 2020-02-05T20:21:06.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:06.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:06.647894+0000) 2020-02-05T20:21:06.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:06.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:06.889267+0000) 2020-02-05T20:21:07.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:07.778+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.781252+0000) 2020-02-05T20:21:07.784 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:07.781+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.782722+0000) 2020-02-05T20:21:07.784 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:07.781+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.782989+0000) 2020-02-05T20:21:07.785 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:07.778+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.783889+0000) 2020-02-05T20:21:07.785 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:07.778+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.783727+0000) 2020-02-05T20:21:07.786 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:07.783+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.785080+0000) 2020-02-05T20:21:07.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:07.784+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.785762+0000) 2020-02-05T20:21:07.787 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:07.785+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.786116+0000) 2020-02-05T20:21:07.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:07.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.786620+0000) 2020-02-05T20:21:07.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:07.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54056 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:07.789 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:07.783+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.786613+0000) 2020-02-05T20:21:07.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:07.786+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.787548+0000) 2020-02-05T20:21:07.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:07.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.786926+0000) 2020-02-05T20:21:07.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:07.792+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.793843+0000) 2020-02-05T20:21:07.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:07.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:07.794354+0000) 2020-02-05T20:21:08.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:08.630+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:08.631066+0000) 2020-02-05T20:21:09.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:09.001+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:09.001654+0000) 2020-02-05T20:21:09.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:09.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:09.286737+0000) 2020-02-05T20:21:09.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:09.895+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:09.896485+0000) 2020-02-05T20:21:10.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:10.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:10.141912+0000) 2020-02-05T20:21:10.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:10.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:10.141909+0000) 2020-02-05T20:21:12.249 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:12.246+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.247732+0000) 2020-02-05T20:21:12.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:12.790+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.791680+0000) 2020-02-05T20:21:12.795 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:12.791+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.793319+0000) 2020-02-05T20:21:12.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:12.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.794257+0000) 2020-02-05T20:21:12.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:12.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.794619+0000) 2020-02-05T20:21:12.796 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:12.793+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.794875+0000) 2020-02-05T20:21:12.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:12.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.795281+0000) 2020-02-05T20:21:12.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:12.794+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.795533+0000) 2020-02-05T20:21:12.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:12.795+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.797222+0000) 2020-02-05T20:21:12.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:12.796+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.797569+0000) 2020-02-05T20:21:12.800 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:12.796+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.798023+0000) 2020-02-05T20:21:12.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:12.796+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.798378+0000) 2020-02-05T20:21:12.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:12.797+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:12.801 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:12.797+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.799096+0000) 2020-02-05T20:21:12.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:12.801+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.802282+0000) 2020-02-05T20:21:12.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:12.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:12.803111+0000) 2020-02-05T20:21:13.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:13.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:13.373746+0000) 2020-02-05T20:21:13.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:13.406+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:13.406628+0000) 2020-02-05T20:21:13.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:13.406+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:13.406626+0000) 2020-02-05T20:21:13.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:13.664+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:13.664801+0000) 2020-02-05T20:21:15.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:15.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:15.788831+0000) 2020-02-05T20:21:16.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:16.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:16.648149+0000) 2020-02-05T20:21:16.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:16.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:16.889634+0000) 2020-02-05T20:21:17.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:17.803+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.804810+0000) 2020-02-05T20:21:17.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:17.803+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.805401+0000) 2020-02-05T20:21:17.807 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:17.804+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.806081+0000) 2020-02-05T20:21:17.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:17.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.806152+0000) 2020-02-05T20:21:17.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:17.806+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.807069+0000) 2020-02-05T20:21:17.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:17.806+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.808208+0000) 2020-02-05T20:21:17.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:17.807+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.809048+0000) 2020-02-05T20:21:17.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:17.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.809455+0000) 2020-02-05T20:21:17.811 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:17.808+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:17.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:17.808+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.809975+0000) 2020-02-05T20:21:17.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:17.809+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.810514+0000) 2020-02-05T20:21:17.812 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:17.809+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.810775+0000) 2020-02-05T20:21:17.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:17.809+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.810816+0000) 2020-02-05T20:21:17.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:17.811+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.812211+0000) 2020-02-05T20:21:17.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:17.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:17.814602+0000) 2020-02-05T20:21:18.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:18.629+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:18.631359+0000) 2020-02-05T20:21:19.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:19.001+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:19.001896+0000) 2020-02-05T20:21:19.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:19.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:19.286993+0000) 2020-02-05T20:21:19.460 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:21:19.466 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:21:19.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:19.895+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:19.896738+0000) 2020-02-05T20:21:20.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:20.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:20.142183+0000) 2020-02-05T20:21:20.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:20.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:20.142187+0000) 2020-02-05T20:21:22.249 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:22.247+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.247974+0000) 2020-02-05T20:21:22.818 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:22.814+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.816132+0000) 2020-02-05T20:21:22.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:22.815+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.816818+0000) 2020-02-05T20:21:22.818 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:22.816+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.817427+0000) 2020-02-05T20:21:22.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:22.817+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.818919+0000) 2020-02-05T20:21:22.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:22.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.819436+0000) 2020-02-05T20:21:22.821 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:22.818+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.819728+0000) 2020-02-05T20:21:22.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:22.818+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.819967+0000) 2020-02-05T20:21:22.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:22.819+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.820493+0000) 2020-02-05T20:21:22.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:22.819+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.820977+0000) 2020-02-05T20:21:22.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:22.820+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.821673+0000) 2020-02-05T20:21:22.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:22.820+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54181 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:22.823 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:22.820+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.821996+0000) 2020-02-05T20:21:22.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:22.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.822313+0000) 2020-02-05T20:21:22.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:22.821+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.822616+0000) 2020-02-05T20:21:22.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:22.827+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:22.828944+0000) 2020-02-05T20:21:23.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:23.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:23.373958+0000) 2020-02-05T20:21:23.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:23.405+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:23.406880+0000) 2020-02-05T20:21:23.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:23.405+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:23.406880+0000) 2020-02-05T20:21:23.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:23.672+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:23.673204+0000) 2020-02-05T20:21:25.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:25.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:25.789124+0000) 2020-02-05T20:21:26.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:26.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:26.648378+0000) 2020-02-05T20:21:26.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:26.888+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:26.889928+0000) 2020-02-05T20:21:27.829 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:27.826+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.827981+0000) 2020-02-05T20:21:27.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:27.827+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.828659+0000) 2020-02-05T20:21:27.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:27.828+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.829191+0000) 2020-02-05T20:21:27.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:27.829+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.830804+0000) 2020-02-05T20:21:27.840 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:27.832+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.833501+0000) 2020-02-05T20:21:27.840 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:27.832+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.833683+0000) 2020-02-05T20:21:27.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:27.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:27.840 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:27.833+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.834554+0000) 2020-02-05T20:21:27.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:27.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.830200+0000) 2020-02-05T20:21:27.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:27.831+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.832302+0000) 2020-02-05T20:21:27.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:27.829+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.830715+0000) 2020-02-05T20:21:27.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:27.830+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.831312+0000) 2020-02-05T20:21:27.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:27.830+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.831612+0000) 2020-02-05T20:21:27.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:27.831+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.832867+0000) 2020-02-05T20:21:27.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:27.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:27.840177+0000) 2020-02-05T20:21:28.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:28.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:28.631643+0000) 2020-02-05T20:21:29.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:29.001+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:29.002165+0000) 2020-02-05T20:21:29.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:29.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:29.287229+0000) 2020-02-05T20:21:29.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:29.896+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:29.896980+0000) 2020-02-05T20:21:30.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:30.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:30.142454+0000) 2020-02-05T20:21:30.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:30.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:30.142464+0000) 2020-02-05T20:21:32.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:32.247+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.248226+0000) 2020-02-05T20:21:32.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:32.838+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.839902+0000) 2020-02-05T20:21:32.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:32.840+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.840437+0000) 2020-02-05T20:21:32.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:32.841+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.841131+0000) 2020-02-05T20:21:32.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:32.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.841894+0000) 2020-02-05T20:21:32.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:32.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.842215+0000) 2020-02-05T20:21:32.844 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:32.841+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.842710+0000) 2020-02-05T20:21:32.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:32.842+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.842988+0000) 2020-02-05T20:21:32.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:32.843+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.843403+0000) 2020-02-05T20:21:32.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:32.842+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.843973+0000) 2020-02-05T20:21:32.846 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:32.842+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.843970+0000) 2020-02-05T20:21:32.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:32.844+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.844345+0000) 2020-02-05T20:21:32.847 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:32.843+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.845221+0000) 2020-02-05T20:21:32.847 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:32.843+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.845403+0000) 2020-02-05T20:21:32.847 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:32.844+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:32.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:32.850+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:32.850155+0000) 2020-02-05T20:21:33.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:33.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:33.374178+0000) 2020-02-05T20:21:33.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:33.407+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:33.407132+0000) 2020-02-05T20:21:33.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:33.407+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:33.407132+0000) 2020-02-05T20:21:33.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:33.685+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:33.685158+0000) 2020-02-05T20:21:35.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:35.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:35.789351+0000) 2020-02-05T20:21:36.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:36.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:36.648610+0000) 2020-02-05T20:21:36.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:36.889+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:36.890178+0000) 2020-02-05T20:21:37.854 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:37.852+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.852780+0000) 2020-02-05T20:21:37.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:37.852+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.853186+0000) 2020-02-05T20:21:37.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:37.853+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.853548+0000) 2020-02-05T20:21:37.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:37.852+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.854019+0000) 2020-02-05T20:21:37.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:37.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.854536+0000) 2020-02-05T20:21:37.856 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:37.853+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.854756+0000) 2020-02-05T20:21:37.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:37.854+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.855232+0000) 2020-02-05T20:21:37.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:37.855+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.855676+0000) 2020-02-05T20:21:37.858 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:37.856+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.856524+0000) 2020-02-05T20:21:37.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:37.856+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.857339+0000) 2020-02-05T20:21:37.859 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:37.856+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.856958+0000) 2020-02-05T20:21:37.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:37.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.857749+0000) 2020-02-05T20:21:37.859 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:37.857+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.858264+0000) 2020-02-05T20:21:37.860 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:37.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54303 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:37.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:37.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:37.864924+0000) 2020-02-05T20:21:38.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:38.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:38.631862+0000) 2020-02-05T20:21:39.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:39.001+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:39.002407+0000) 2020-02-05T20:21:39.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:39.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:39.287459+0000) 2020-02-05T20:21:39.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:39.896+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:39.897222+0000) 2020-02-05T20:21:40.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:40.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:40.142738+0000) 2020-02-05T20:21:40.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:40.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:40.142740+0000) 2020-02-05T20:21:42.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:42.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.248494+0000) 2020-02-05T20:21:42.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:42.864+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.865125+0000) 2020-02-05T20:21:42.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:42.864+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.865728+0000) 2020-02-05T20:21:42.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:42.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.866402+0000) 2020-02-05T20:21:42.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:42.866+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.867121+0000) 2020-02-05T20:21:42.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:42.866+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.867544+0000) 2020-02-05T20:21:42.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:42.867+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.868451+0000) 2020-02-05T20:21:42.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:42.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.867960+0000) 2020-02-05T20:21:42.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:42.867+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.868775+0000) 2020-02-05T20:21:42.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:42.869+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.869872+0000) 2020-02-05T20:21:42.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:42.869+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.870092+0000) 2020-02-05T20:21:42.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:42.869+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.870395+0000) 2020-02-05T20:21:42.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:42.870+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.870495+0000) 2020-02-05T20:21:42.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:42.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:42.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:42.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.875869+0000) 2020-02-05T20:21:42.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:42.875+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:42.876423+0000) 2020-02-05T20:21:43.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:43.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:43.374404+0000) 2020-02-05T20:21:43.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:43.406+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:43.407336+0000) 2020-02-05T20:21:43.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:43.406+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:43.407336+0000) 2020-02-05T20:21:43.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:43.697+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:43.698197+0000) 2020-02-05T20:21:45.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:45.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:45.789577+0000) 2020-02-05T20:21:46.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:46.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:46.648848+0000) 2020-02-05T20:21:46.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:46.890+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:46.890465+0000) 2020-02-05T20:21:47.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:47.875+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.876070+0000) 2020-02-05T20:21:47.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:47.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.877002+0000) 2020-02-05T20:21:47.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:47.876+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.877776+0000) 2020-02-05T20:21:47.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:47.877+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.878193+0000) 2020-02-05T20:21:47.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:47.877+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.878615+0000) 2020-02-05T20:21:47.880 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:47.878+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.879336+0000) 2020-02-05T20:21:47.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:47.879+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.880828+0000) 2020-02-05T20:21:47.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:47.880+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.880656+0000) 2020-02-05T20:21:47.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:47.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.880839+0000) 2020-02-05T20:21:47.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:47.881+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.881682+0000) 2020-02-05T20:21:47.883 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:47.881+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.882382+0000) 2020-02-05T20:21:47.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:47.882+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.882585+0000) 2020-02-05T20:21:47.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:47.882+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.883198+0000) 2020-02-05T20:21:47.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:47.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:47.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:47.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:47.886983+0000) 2020-02-05T20:21:48.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:48.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:48.632113+0000) 2020-02-05T20:21:49.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:49.001+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:49.002670+0000) 2020-02-05T20:21:49.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:49.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:49.287719+0000) 2020-02-05T20:21:49.693 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:21:49.700 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:21:49.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:49.896+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:49.897416+0000) 2020-02-05T20:21:50.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:50.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:50.143015+0000) 2020-02-05T20:21:50.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:50.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:50.143019+0000) 2020-02-05T20:21:52.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:52.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.248732+0000) 2020-02-05T20:21:52.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:52.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.890278+0000) 2020-02-05T20:21:52.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:52.890+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.890711+0000) 2020-02-05T20:21:52.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:52.889+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.890887+0000) 2020-02-05T20:21:52.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:52.890+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.891404+0000) 2020-02-05T20:21:52.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:52.891+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.891693+0000) 2020-02-05T20:21:52.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:52.891+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.892183+0000) 2020-02-05T20:21:52.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:52.891+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.892662+0000) 2020-02-05T20:21:52.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:52.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.893293+0000) 2020-02-05T20:21:52.895 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:52.893+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.894229+0000) 2020-02-05T20:21:52.896 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:52.894+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.894505+0000) 2020-02-05T20:21:52.896 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:52.894+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.894986+0000) 2020-02-05T20:21:52.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:52.894+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.895042+0000) 2020-02-05T20:21:52.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:52.895+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.895884+0000) 2020-02-05T20:21:52.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:52.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:52.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:52.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:52.900146+0000) 2020-02-05T20:21:53.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:53.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:53.374671+0000) 2020-02-05T20:21:53.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:53.406+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:53.407565+0000) 2020-02-05T20:21:53.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:53.406+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:53.407588+0000) 2020-02-05T20:21:53.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:53.704+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:53.705800+0000) 2020-02-05T20:21:55.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:55.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:55.789828+0000) 2020-02-05T20:21:56.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:56.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:56.649122+0000) 2020-02-05T20:21:56.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:56.890+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:56.890672+0000) 2020-02-05T20:21:57.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:21:57.900+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.901983+0000) 2020-02-05T20:21:57.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:57.901+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.902994+0000) 2020-02-05T20:21:57.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:57.903+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.904514+0000) 2020-02-05T20:21:57.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:21:57.903+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.904171+0000) 2020-02-05T20:21:57.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:21:57.904+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.905010+0000) 2020-02-05T20:21:57.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:57.904+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.905660+0000) 2020-02-05T20:21:57.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:21:57.904+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.906011+0000) 2020-02-05T20:21:57.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:57.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.906673+0000) 2020-02-05T20:21:57.908 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:57.906+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.907197+0000) 2020-02-05T20:21:57.909 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:21:57.907+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.907529+0000) 2020-02-05T20:21:57.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:21:57.907+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.907985+0000) 2020-02-05T20:21:57.909 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:21:57.907+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.908146+0000) 2020-02-05T20:21:57.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:21:57.907+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.908504+0000) 2020-02-05T20:21:57.910 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:21:57.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:21:57.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:57.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:57.916513+0000) 2020-02-05T20:21:58.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:21:58.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:58.632352+0000) 2020-02-05T20:21:59.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:21:59.002+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:59.002938+0000) 2020-02-05T20:21:59.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:21:59.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:59.287948+0000) 2020-02-05T20:21:59.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:21:59.896+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:21:59.897654+0000) 2020-02-05T20:22:00.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:00.143+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:00.143321+0000) 2020-02-05T20:22:00.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:00.143+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:00.143317+0000) 2020-02-05T20:22:02.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:02.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.249071+0000) 2020-02-05T20:22:02.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:02.912+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.913148+0000) 2020-02-05T20:22:02.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:02.914+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.915364+0000) 2020-02-05T20:22:02.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:02.915+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.916357+0000) 2020-02-05T20:22:02.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:02.914+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.916052+0000) 2020-02-05T20:22:02.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:02.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.917102+0000) 2020-02-05T20:22:02.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:02.916+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.917182+0000) 2020-02-05T20:22:02.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:02.916+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.917601+0000) 2020-02-05T20:22:02.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:02.916+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.917922+0000) 2020-02-05T20:22:02.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:02.918+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.918651+0000) 2020-02-05T20:22:02.920 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:02.917+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.918303+0000) 2020-02-05T20:22:02.921 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:02.917+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.918911+0000) 2020-02-05T20:22:02.921 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:02.918+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.919315+0000) 2020-02-05T20:22:02.921 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:02.919+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.920135+0000) 2020-02-05T20:22:02.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:02.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:02.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:02.924+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:02.925912+0000) 2020-02-05T20:22:03.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:03.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:03.374889+0000) 2020-02-05T20:22:03.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:03.406+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:03.407821+0000) 2020-02-05T20:22:03.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:03.406+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:03.407824+0000) 2020-02-05T20:22:03.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:03.711+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:03.712728+0000) 2020-02-05T20:22:05.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:05.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:05.790053+0000) 2020-02-05T20:22:06.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:06.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:06.649345+0000) 2020-02-05T20:22:06.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:06.890+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:06.890973+0000) 2020-02-05T20:22:07.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:07.924+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.925339+0000) 2020-02-05T20:22:07.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:07.925+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.926186+0000) 2020-02-05T20:22:07.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:07.926+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.926649+0000) 2020-02-05T20:22:07.929 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:07.926+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.927662+0000) 2020-02-05T20:22:07.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:07.927+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.928807+0000) 2020-02-05T20:22:07.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:07.928+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.929461+0000) 2020-02-05T20:22:07.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:07.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.929968+0000) 2020-02-05T20:22:07.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:07.930+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.930555+0000) 2020-02-05T20:22:07.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:07.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.930942+0000) 2020-02-05T20:22:07.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:07.930+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.931623+0000) 2020-02-05T20:22:07.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:07.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:07.933 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:07.931+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.931812+0000) 2020-02-05T20:22:07.934 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:07.931+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.932258+0000) 2020-02-05T20:22:07.934 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:07.932+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.932664+0000) 2020-02-05T20:22:07.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:07.933+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:07.934465+0000) 2020-02-05T20:22:08.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:08.632+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:08.632599+0000) 2020-02-05T20:22:09.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:09.002+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:09.003188+0000) 2020-02-05T20:22:09.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:09.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:09.288178+0000) 2020-02-05T20:22:09.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:09.897+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:09.897918+0000) 2020-02-05T20:22:10.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:10.143+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:10.143594+0000) 2020-02-05T20:22:10.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:10.143+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:10.143613+0000) 2020-02-05T20:22:12.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:12.247+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.249299+0000) 2020-02-05T20:22:12.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:12.936+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.937628+0000) 2020-02-05T20:22:12.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:12.937+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.938811+0000) 2020-02-05T20:22:12.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:12.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.939809+0000) 2020-02-05T20:22:12.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:12.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.940292+0000) 2020-02-05T20:22:12.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:12.939+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.940867+0000) 2020-02-05T20:22:12.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:12.939+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.940823+0000) 2020-02-05T20:22:12.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:12.938+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.940184+0000) 2020-02-05T20:22:12.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:12.940+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.941429+0000) 2020-02-05T20:22:12.943 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:12.940+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.941513+0000) 2020-02-05T20:22:12.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:12.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.942277+0000) 2020-02-05T20:22:12.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:12.941+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.942509+0000) 2020-02-05T20:22:12.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:12.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.942399+0000) 2020-02-05T20:22:12.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:12.941+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.942674+0000) 2020-02-05T20:22:12.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:12.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:12.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:12.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:12.949491+0000) 2020-02-05T20:22:13.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:13.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:13.375124+0000) 2020-02-05T20:22:13.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:13.406+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:13.407992+0000) 2020-02-05T20:22:13.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:13.406+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:13.408048+0000) 2020-02-05T20:22:13.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:13.722+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:13.722455+0000) 2020-02-05T20:22:15.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:15.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:15.790320+0000) 2020-02-05T20:22:16.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:16.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:16.649604+0000) 2020-02-05T20:22:16.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:16.889+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:16.891178+0000) 2020-02-05T20:22:17.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:17.947+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.949407+0000) 2020-02-05T20:22:17.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:17.948+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.949699+0000) 2020-02-05T20:22:17.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:17.949+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.950169+0000) 2020-02-05T20:22:17.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:17.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.950414+0000) 2020-02-05T20:22:17.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:17.949+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.950653+0000) 2020-02-05T20:22:17.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:17.949+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.950979+0000) 2020-02-05T20:22:17.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:17.950+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.951435+0000) 2020-02-05T20:22:17.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:17.949+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.951004+0000) 2020-02-05T20:22:17.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:17.950+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.951878+0000) 2020-02-05T20:22:17.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:17.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.952946+0000) 2020-02-05T20:22:17.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:17.950+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.951928+0000) 2020-02-05T20:22:17.955 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:17.951+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.952619+0000) 2020-02-05T20:22:17.956 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:17.952+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.953563+0000) 2020-02-05T20:22:17.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:17.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:17.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:17.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:17.957868+0000) 2020-02-05T20:22:18.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:18.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:18.632814+0000) 2020-02-05T20:22:19.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:19.003+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:19.003439+0000) 2020-02-05T20:22:19.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:19.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:19.288405+0000) 2020-02-05T20:22:19.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:19.897+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:19.898163+0000) 2020-02-05T20:22:19.921 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:22:19.927 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:22:20.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:20.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:20.143806+0000) 2020-02-05T20:22:20.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:20.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:20.143893+0000) 2020-02-05T20:22:22.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:22.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.249566+0000) 2020-02-05T20:22:22.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:22.957+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.958959+0000) 2020-02-05T20:22:22.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:22.958+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.959499+0000) 2020-02-05T20:22:22.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:22.959+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.960547+0000) 2020-02-05T20:22:22.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:22.959+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.960565+0000) 2020-02-05T20:22:22.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:22.959+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.960706+0000) 2020-02-05T20:22:22.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:22.959+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.960997+0000) 2020-02-05T20:22:22.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:22.960+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.961510+0000) 2020-02-05T20:22:22.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:22.960+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.962138+0000) 2020-02-05T20:22:22.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:22.960+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.962049+0000) 2020-02-05T20:22:22.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:22.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.962801+0000) 2020-02-05T20:22:22.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:22.963+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.964362+0000) 2020-02-05T20:22:22.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:22.963+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.964516+0000) 2020-02-05T20:22:22.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:22.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:22.967 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:22.963+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.965066+0000) 2020-02-05T20:22:22.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:22.970+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:22.971632+0000) 2020-02-05T20:22:23.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:23.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:23.375341+0000) 2020-02-05T20:22:23.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:23.408+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:23.408215+0000) 2020-02-05T20:22:23.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:23.408+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:23.408302+0000) 2020-02-05T20:22:23.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:23.730+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:23.730985+0000) 2020-02-05T20:22:25.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:25.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:25.790582+0000) 2020-02-05T20:22:26.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:26.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:26.649839+0000) 2020-02-05T20:22:26.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:26.890+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:26.891487+0000) 2020-02-05T20:22:27.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:27.969+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.969493+0000) 2020-02-05T20:22:27.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:27.968+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.970375+0000) 2020-02-05T20:22:27.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:27.972+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.972454+0000) 2020-02-05T20:22:27.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:27.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.974248+0000) 2020-02-05T20:22:27.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:27.973+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.974702+0000) 2020-02-05T20:22:27.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:27.975+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.975645+0000) 2020-02-05T20:22:27.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:27.975+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.976040+0000) 2020-02-05T20:22:27.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:27.975+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.976578+0000) 2020-02-05T20:22:27.980 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:27.974+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.976399+0000) 2020-02-05T20:22:27.980 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:27.975+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.977094+0000) 2020-02-05T20:22:27.980 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:27.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:27.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:27.976+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.976883+0000) 2020-02-05T20:22:27.981 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:27.976+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.976907+0000) 2020-02-05T20:22:27.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:27.979+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.979203+0000) 2020-02-05T20:22:27.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:27.981+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:27.981671+0000) 2020-02-05T20:22:28.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:28.632+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:28.633058+0000) 2020-02-05T20:22:29.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:29.003+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:29.003670+0000) 2020-02-05T20:22:29.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:29.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:29.288657+0000) 2020-02-05T20:22:29.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:29.897+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:29.898382+0000) 2020-02-05T20:22:30.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:30.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:30.144097+0000) 2020-02-05T20:22:30.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:30.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:30.144137+0000) 2020-02-05T20:22:32.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:32.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.249790+0000) 2020-02-05T20:22:32.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:32.981+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.982888+0000) 2020-02-05T20:22:32.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:32.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.983625+0000) 2020-02-05T20:22:32.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:32.982+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.983555+0000) 2020-02-05T20:22:32.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:32.984+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.985344+0000) 2020-02-05T20:22:32.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:32.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.985903+0000) 2020-02-05T20:22:32.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:32.985+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.986494+0000) 2020-02-05T20:22:32.988 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:32.985+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.987028+0000) 2020-02-05T20:22:32.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:32.986+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.987487+0000) 2020-02-05T20:22:32.989 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:32.986+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.987507+0000) 2020-02-05T20:22:32.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:32.987+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.988240+0000) 2020-02-05T20:22:32.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:32.986+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.987992+0000) 2020-02-05T20:22:32.990 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:32.986+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.988219+0000) 2020-02-05T20:22:32.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:32.987+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.988611+0000) 2020-02-05T20:22:32.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:32.988+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:32.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:32.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:32.992404+0000) 2020-02-05T20:22:33.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:33.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:33.375556+0000) 2020-02-05T20:22:33.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:33.407+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:33.408416+0000) 2020-02-05T20:22:33.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:33.407+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:33.408502+0000) 2020-02-05T20:22:33.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:33.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:33.738895+0000) 2020-02-05T20:22:35.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:35.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:35.790832+0000) 2020-02-05T20:22:36.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:36.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:36.650158+0000) 2020-02-05T20:22:36.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:36.890+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:36.891740+0000) 2020-02-05T20:22:37.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:37.994+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.995656+0000) 2020-02-05T20:22:37.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:37.995+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.996839+0000) 2020-02-05T20:22:37.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:37.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.997083+0000) 2020-02-05T20:22:37.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:37.996+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.997797+0000) 2020-02-05T20:22:37.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:37.997+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.998412+0000) 2020-02-05T20:22:38.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:37.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.998824+0000) 2020-02-05T20:22:38.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:37.998+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.999109+0000) 2020-02-05T20:22:38.000 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:37.998+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.999284+0000) 2020-02-05T20:22:38.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:37.997+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.999403+0000) 2020-02-05T20:22:38.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:37.998+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.999568+0000) 2020-02-05T20:22:38.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:37.998+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:37.999797+0000) 2020-02-05T20:22:38.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:37.999+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:38.001003+0000) 2020-02-05T20:22:38.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:38.000+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54786 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:38.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:38.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:38.005968+0000) 2020-02-05T20:22:38.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:38.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:38.006251+0000) 2020-02-05T20:22:38.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:38.631+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:38.633294+0000) 2020-02-05T20:22:39.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:39.002+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:39.003907+0000) 2020-02-05T20:22:39.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:39.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:39.288882+0000) 2020-02-05T20:22:39.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:39.897+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:39.898660+0000) 2020-02-05T20:22:40.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:40.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:40.144367+0000) 2020-02-05T20:22:40.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:40.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:40.144374+0000) 2020-02-05T20:22:42.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:42.248+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:42.250041+0000) 2020-02-05T20:22:43.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:43.007+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.008835+0000) 2020-02-05T20:22:43.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:43.007+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.009051+0000) 2020-02-05T20:22:43.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:43.008+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.009543+0000) 2020-02-05T20:22:43.011 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:43.009+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.010435+0000) 2020-02-05T20:22:43.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:43.008+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.010069+0000) 2020-02-05T20:22:43.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:43.009+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.010696+0000) 2020-02-05T20:22:43.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:43.010+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.011120+0000) 2020-02-05T20:22:43.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:43.010+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.011512+0000) 2020-02-05T20:22:43.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:43.010+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.011530+0000) 2020-02-05T20:22:43.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:43.010+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.011965+0000) 2020-02-05T20:22:43.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:43.010+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.011794+0000) 2020-02-05T20:22:43.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:43.011+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.012422+0000) 2020-02-05T20:22:43.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:43.011+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.012980+0000) 2020-02-05T20:22:43.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:43.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:43.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:43.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.016683+0000) 2020-02-05T20:22:43.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:43.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.375779+0000) 2020-02-05T20:22:43.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:43.407+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.408621+0000) 2020-02-05T20:22:43.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:43.407+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.408728+0000) 2020-02-05T20:22:43.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:43.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:43.747115+0000) 2020-02-05T20:22:45.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:45.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:45.791055+0000) 2020-02-05T20:22:46.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:46.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:46.650388+0000) 2020-02-05T20:22:46.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:46.891+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:46.892037+0000) 2020-02-05T20:22:48.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:48.018+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.020384+0000) 2020-02-05T20:22:48.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:48.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.021328+0000) 2020-02-05T20:22:48.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:48.021+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.022144+0000) 2020-02-05T20:22:48.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:48.021+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.022589+0000) 2020-02-05T20:22:48.024 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:48.021+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.022878+0000) 2020-02-05T20:22:48.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:48.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.023430+0000) 2020-02-05T20:22:48.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:48.022+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.023694+0000) 2020-02-05T20:22:48.025 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:48.022+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.024133+0000) 2020-02-05T20:22:48.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:48.023+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.025236+0000) 2020-02-05T20:22:48.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:48.023+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.025360+0000) 2020-02-05T20:22:48.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:48.024+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.025711+0000) 2020-02-05T20:22:48.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:48.024+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.026211+0000) 2020-02-05T20:22:48.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:48.024+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.026305+0000) 2020-02-05T20:22:48.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:48.025+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54866 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:48.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:48.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.029312+0000) 2020-02-05T20:22:48.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:48.632+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:48.633514+0000) 2020-02-05T20:22:49.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:49.003+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:49.004141+0000) 2020-02-05T20:22:49.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:49.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:49.289141+0000) 2020-02-05T20:22:49.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:49.898+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:49.898895+0000) 2020-02-05T20:22:50.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:50.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:50.144664+0000) 2020-02-05T20:22:50.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:50.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:50.144664+0000) 2020-02-05T20:22:50.151 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:22:50.157 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:22:52.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:52.249+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:52.250238+0000) 2020-02-05T20:22:53.036 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:53.033+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.034414+0000) 2020-02-05T20:22:53.036 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:53.032+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.033948+0000) 2020-02-05T20:22:53.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:53.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.035486+0000) 2020-02-05T20:22:53.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:53.034+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.035609+0000) 2020-02-05T20:22:53.037 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:53.035+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.036335+0000) 2020-02-05T20:22:53.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:53.035+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.036406+0000) 2020-02-05T20:22:53.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:53.035+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.036970+0000) 2020-02-05T20:22:53.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:53.036+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.037504+0000) 2020-02-05T20:22:53.039 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:53.036+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.037481+0000) 2020-02-05T20:22:53.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:53.036+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.038150+0000) 2020-02-05T20:22:53.040 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:53.037+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.038571+0000) 2020-02-05T20:22:53.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:53.038+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.039261+0000) 2020-02-05T20:22:53.041 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:53.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:53.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:53.038+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.039487+0000) 2020-02-05T20:22:53.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:53.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.045604+0000) 2020-02-05T20:22:53.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:53.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.376053+0000) 2020-02-05T20:22:53.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:53.408+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.408834+0000) 2020-02-05T20:22:53.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:53.408+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.409035+0000) 2020-02-05T20:22:53.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:53.754+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:53.755119+0000) 2020-02-05T20:22:55.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:55.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:55.791281+0000) 2020-02-05T20:22:56.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:56.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:56.650616+0000) 2020-02-05T20:22:56.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:56.891+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:56.892293+0000) 2020-02-05T20:22:58.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:58.043+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.044756+0000) 2020-02-05T20:22:58.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:58.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.045608+0000) 2020-02-05T20:22:58.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:22:58.045+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.046165+0000) 2020-02-05T20:22:58.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:58.045+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.046528+0000) 2020-02-05T20:22:58.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:22:58.046+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.047617+0000) 2020-02-05T20:22:58.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:22:58.046+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.048024+0000) 2020-02-05T20:22:58.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:22:58.047+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.048643+0000) 2020-02-05T20:22:58.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:58.047+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.048587+0000) 2020-02-05T20:22:58.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:22:58.047+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.048979+0000) 2020-02-05T20:22:58.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:58.048+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.049306+0000) 2020-02-05T20:22:58.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:22:58.049+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.050249+0000) 2020-02-05T20:22:58.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:22:58.048+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.049664+0000) 2020-02-05T20:22:58.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:22:58.048+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.050397+0000) 2020-02-05T20:22:58.054 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:22:58.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:22:58.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:58.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.056294+0000) 2020-02-05T20:22:58.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:22:58.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:58.633731+0000) 2020-02-05T20:22:59.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:22:59.003+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:59.004353+0000) 2020-02-05T20:22:59.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:22:59.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:59.289362+0000) 2020-02-05T20:22:59.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:22:59.898+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:22:59.899128+0000) 2020-02-05T20:23:00.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:00.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:00.144935+0000) 2020-02-05T20:23:00.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:00.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:00.144940+0000) 2020-02-05T20:23:02.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:02.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:02.250495+0000) 2020-02-05T20:23:03.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:03.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.057736+0000) 2020-02-05T20:23:03.061 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:03.058+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.059409+0000) 2020-02-05T20:23:03.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:03.058+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.059144+0000) 2020-02-05T20:23:03.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:03.060+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.060993+0000) 2020-02-05T20:23:03.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:03.060+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.061447+0000) 2020-02-05T20:23:03.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:03.060+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.061939+0000) 2020-02-05T20:23:03.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:03.061+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.062244+0000) 2020-02-05T20:23:03.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:03.062+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.062637+0000) 2020-02-05T20:23:03.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:03.062+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.063233+0000) 2020-02-05T20:23:03.065 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:03.062+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.063509+0000) 2020-02-05T20:23:03.065 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:03.063+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.063725+0000) 2020-02-05T20:23:03.066 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:03.063+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.063975+0000) 2020-02-05T20:23:03.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:03.062+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.063931+0000) 2020-02-05T20:23:03.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:03.064+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 54985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:03.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:03.065+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.066733+0000) 2020-02-05T20:23:03.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:03.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.376282+0000) 2020-02-05T20:23:03.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:03.408+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.409044+0000) 2020-02-05T20:23:03.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:03.408+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.409305+0000) 2020-02-05T20:23:03.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:03.762+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:03.763223+0000) 2020-02-05T20:23:05.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:05.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:05.791512+0000) 2020-02-05T20:23:06.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:06.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:06.650852+0000) 2020-02-05T20:23:06.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:06.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:06.892591+0000) 2020-02-05T20:23:08.072 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:08.069+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.070780+0000) 2020-02-05T20:23:08.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:08.070+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.071845+0000) 2020-02-05T20:23:08.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:08.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.072750+0000) 2020-02-05T20:23:08.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:08.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.073635+0000) 2020-02-05T20:23:08.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:08.073+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.073757+0000) 2020-02-05T20:23:08.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:08.074+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.074658+0000) 2020-02-05T20:23:08.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:08.074+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.074739+0000) 2020-02-05T20:23:08.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:08.074+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.075146+0000) 2020-02-05T20:23:08.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:08.074+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.075366+0000) 2020-02-05T20:23:08.077 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:08.074+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.075509+0000) 2020-02-05T20:23:08.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:08.074+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.075675+0000) 2020-02-05T20:23:08.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:08.075+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.076326+0000) 2020-02-05T20:23:08.078 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:08.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:08.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:08.079+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.080742+0000) 2020-02-05T20:23:08.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:08.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.081263+0000) 2020-02-05T20:23:08.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:08.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:08.633972+0000) 2020-02-05T20:23:09.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:09.004+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:09.004579+0000) 2020-02-05T20:23:09.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:09.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:09.289576+0000) 2020-02-05T20:23:09.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:09.899+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:09.899366+0000) 2020-02-05T20:23:10.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:10.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:10.145218+0000) 2020-02-05T20:23:10.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:10.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:10.145214+0000) 2020-02-05T20:23:12.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:12.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:12.250755+0000) 2020-02-05T20:23:13.083 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:13.080+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.081772+0000) 2020-02-05T20:23:13.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:13.082+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.083048+0000) 2020-02-05T20:23:13.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:13.083+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.084018+0000) 2020-02-05T20:23:13.086 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:13.081+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.082298+0000) 2020-02-05T20:23:13.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:13.082+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.083958+0000) 2020-02-05T20:23:13.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:13.083+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.084488+0000) 2020-02-05T20:23:13.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:13.084+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.085235+0000) 2020-02-05T20:23:13.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:13.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.084946+0000) 2020-02-05T20:23:13.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:13.085+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.086019+0000) 2020-02-05T20:23:13.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:13.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.086342+0000) 2020-02-05T20:23:13.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:13.086+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.086811+0000) 2020-02-05T20:23:13.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:13.086+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.087153+0000) 2020-02-05T20:23:13.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:13.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55065 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:13.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:13.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.091393+0000) 2020-02-05T20:23:13.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:13.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.091974+0000) 2020-02-05T20:23:13.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:13.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.376545+0000) 2020-02-05T20:23:13.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:13.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.409377+0000) 2020-02-05T20:23:13.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:13.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.409504+0000) 2020-02-05T20:23:13.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:13.771+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:13.771547+0000) 2020-02-05T20:23:15.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:15.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:15.791740+0000) 2020-02-05T20:23:16.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:16.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:16.651103+0000) 2020-02-05T20:23:16.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:16.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:16.892828+0000) 2020-02-05T20:23:18.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:18.092+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.093732+0000) 2020-02-05T20:23:18.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:18.095+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.095692+0000) 2020-02-05T20:23:18.097 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:18.094+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.095915+0000) 2020-02-05T20:23:18.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:18.095+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.096335+0000) 2020-02-05T20:23:18.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:18.096+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.097265+0000) 2020-02-05T20:23:18.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:18.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.097569+0000) 2020-02-05T20:23:18.100 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:18.097+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.097931+0000) 2020-02-05T20:23:18.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:18.097+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.098291+0000) 2020-02-05T20:23:18.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:18.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.099760+0000) 2020-02-05T20:23:18.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:18.099+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.099781+0000) 2020-02-05T20:23:18.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:18.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.100291+0000) 2020-02-05T20:23:18.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:18.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.100785+0000) 2020-02-05T20:23:18.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:18.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:18.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:18.103+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.104930+0000) 2020-02-05T20:23:18.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:18.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.106598+0000) 2020-02-05T20:23:18.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:18.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:18.634235+0000) 2020-02-05T20:23:19.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:19.004+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:19.004805+0000) 2020-02-05T20:23:19.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:19.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:19.289829+0000) 2020-02-05T20:23:19.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:19.899+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:19.899586+0000) 2020-02-05T20:23:20.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:20.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:20.145489+0000) 2020-02-05T20:23:20.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:20.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:20.145497+0000) 2020-02-05T20:23:20.374 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:23:20.380 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:23:22.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:22.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:22.250993+0000) 2020-02-05T20:23:23.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:23.106+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.106877+0000) 2020-02-05T20:23:23.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:23.108+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.108190+0000) 2020-02-05T20:23:23.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:23.109+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.109354+0000) 2020-02-05T20:23:23.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:23.109+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.109575+0000) 2020-02-05T20:23:23.113 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:23.110+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.110155+0000) 2020-02-05T20:23:23.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:23.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.111195+0000) 2020-02-05T20:23:23.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:23.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.111780+0000) 2020-02-05T20:23:23.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:23.111+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.111787+0000) 2020-02-05T20:23:23.114 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:23.109+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.110042+0000) 2020-02-05T20:23:23.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:23.113+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.113109+0000) 2020-02-05T20:23:23.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:23.112+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.112488+0000) 2020-02-05T20:23:23.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:23.113+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.113519+0000) 2020-02-05T20:23:23.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:23.113+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.113574+0000) 2020-02-05T20:23:23.117 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:23.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55145 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:23.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:23.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.116387+0000) 2020-02-05T20:23:23.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:23.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.376791+0000) 2020-02-05T20:23:23.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:23.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.409613+0000) 2020-02-05T20:23:23.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:23.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.409733+0000) 2020-02-05T20:23:23.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:23.780+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:23.780145+0000) 2020-02-05T20:23:25.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:25.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:25.791990+0000) 2020-02-05T20:23:26.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:26.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:26.651358+0000) 2020-02-05T20:23:26.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:26.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:26.893129+0000) 2020-02-05T20:23:28.126 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:28.121+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.122195+0000) 2020-02-05T20:23:28.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:28.121+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.122544+0000) 2020-02-05T20:23:28.127 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:28.121+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.122997+0000) 2020-02-05T20:23:28.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:28.123+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.124605+0000) 2020-02-05T20:23:28.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:28.120+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.120659+0000) 2020-02-05T20:23:28.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:28.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.121404+0000) 2020-02-05T20:23:28.127 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:28.123+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.123725+0000) 2020-02-05T20:23:28.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:28.123+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.125028+0000) 2020-02-05T20:23:28.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:28.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.126716+0000) 2020-02-05T20:23:28.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:28.125+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.125624+0000) 2020-02-05T20:23:28.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:28.125+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.125619+0000) 2020-02-05T20:23:28.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:28.125+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.125975+0000) 2020-02-05T20:23:28.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:28.126+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:28.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:28.129+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.130440+0000) 2020-02-05T20:23:28.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:28.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.131134+0000) 2020-02-05T20:23:28.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:28.634+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:28.634473+0000) 2020-02-05T20:23:29.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:29.004+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:29.005119+0000) 2020-02-05T20:23:29.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:29.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:29.290060+0000) 2020-02-05T20:23:29.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:29.898+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:29.899813+0000) 2020-02-05T20:23:30.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:30.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:30.145729+0000) 2020-02-05T20:23:30.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:30.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:30.145738+0000) 2020-02-05T20:23:32.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:32.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:32.251250+0000) 2020-02-05T20:23:33.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:33.132+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.132417+0000) 2020-02-05T20:23:33.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:33.132+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.133270+0000) 2020-02-05T20:23:33.147 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:33.134+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.134730+0000) 2020-02-05T20:23:33.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:33.135+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.136224+0000) 2020-02-05T20:23:33.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:33.137+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.137934+0000) 2020-02-05T20:23:33.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:33.137+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.138115+0000) 2020-02-05T20:23:33.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:33.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:33.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:33.139+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.139474+0000) 2020-02-05T20:23:33.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:33.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.133939+0000) 2020-02-05T20:23:33.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:33.135+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.136753+0000) 2020-02-05T20:23:33.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:33.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.142603+0000) 2020-02-05T20:23:33.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:33.134+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.135849+0000) 2020-02-05T20:23:33.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:33.136+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.137233+0000) 2020-02-05T20:23:33.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:33.137+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.139030+0000) 2020-02-05T20:23:33.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:33.138+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.139058+0000) 2020-02-05T20:23:33.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:33.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.377027+0000) 2020-02-05T20:23:33.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:33.408+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.409820+0000) 2020-02-05T20:23:33.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:33.408+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.409901+0000) 2020-02-05T20:23:33.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:33.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:33.788345+0000) 2020-02-05T20:23:35.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:35.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:35.792258+0000) 2020-02-05T20:23:36.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:36.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:36.651592+0000) 2020-02-05T20:23:36.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:36.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:36.893366+0000) 2020-02-05T20:23:38.149 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:38.145+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.146662+0000) 2020-02-05T20:23:38.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:38.145+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.147023+0000) 2020-02-05T20:23:38.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:38.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.147550+0000) 2020-02-05T20:23:38.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:38.146+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.147793+0000) 2020-02-05T20:23:38.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:38.147+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.148135+0000) 2020-02-05T20:23:38.150 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:38.145+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.146398+0000) 2020-02-05T20:23:38.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:38.148+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.149188+0000) 2020-02-05T20:23:38.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:38.148+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.148557+0000) 2020-02-05T20:23:38.152 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:38.148+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.148995+0000) 2020-02-05T20:23:38.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:38.149+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.150311+0000) 2020-02-05T20:23:38.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:38.149+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.150303+0000) 2020-02-05T20:23:38.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:38.150+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.150962+0000) 2020-02-05T20:23:38.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:38.150+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.150610+0000) 2020-02-05T20:23:38.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:38.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55265 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:38.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:38.154+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.156062+0000) 2020-02-05T20:23:38.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:38.634+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:38.634633+0000) 2020-02-05T20:23:39.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:39.004+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:39.005353+0000) 2020-02-05T20:23:39.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:39.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:39.290423+0000) 2020-02-05T20:23:39.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:39.899+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:39.900045+0000) 2020-02-05T20:23:40.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:40.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:40.146002+0000) 2020-02-05T20:23:40.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:40.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:40.145991+0000) 2020-02-05T20:23:42.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:42.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:42.251507+0000) 2020-02-05T20:23:43.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:43.155+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.156538+0000) 2020-02-05T20:23:43.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:43.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.158637+0000) 2020-02-05T20:23:43.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:43.158+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.159378+0000) 2020-02-05T20:23:43.161 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:43.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.159093+0000) 2020-02-05T20:23:43.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:43.158+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.160195+0000) 2020-02-05T20:23:43.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:43.159+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.160611+0000) 2020-02-05T20:23:43.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:43.159+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.160692+0000) 2020-02-05T20:23:43.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:43.160+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.161268+0000) 2020-02-05T20:23:43.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:43.160+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.161295+0000) 2020-02-05T20:23:43.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:43.160+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.162004+0000) 2020-02-05T20:23:43.164 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:43.160+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.162079+0000) 2020-02-05T20:23:43.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:43.160+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.161714+0000) 2020-02-05T20:23:43.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:43.161+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.162509+0000) 2020-02-05T20:23:43.165 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:43.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:43.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:43.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.166942+0000) 2020-02-05T20:23:43.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:43.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.377239+0000) 2020-02-05T20:23:43.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:43.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.410022+0000) 2020-02-05T20:23:43.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:43.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.410161+0000) 2020-02-05T20:23:43.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:43.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:43.795149+0000) 2020-02-05T20:23:45.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:45.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:45.792487+0000) 2020-02-05T20:23:46.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:46.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:46.651903+0000) 2020-02-05T20:23:46.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:46.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:46.893714+0000) 2020-02-05T20:23:48.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:48.167+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.169329+0000) 2020-02-05T20:23:48.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:48.169+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.170202+0000) 2020-02-05T20:23:48.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:48.169+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.170902+0000) 2020-02-05T20:23:48.173 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:48.169+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.171305+0000) 2020-02-05T20:23:48.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:48.170+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.171982+0000) 2020-02-05T20:23:48.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:48.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.172328+0000) 2020-02-05T20:23:48.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:48.171+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.172874+0000) 2020-02-05T20:23:48.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:48.172+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.173332+0000) 2020-02-05T20:23:48.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:48.172+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.173592+0000) 2020-02-05T20:23:48.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:48.173+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.174459+0000) 2020-02-05T20:23:48.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:48.173+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.174677+0000) 2020-02-05T20:23:48.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:48.173+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.175328+0000) 2020-02-05T20:23:48.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:48.175+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.176131+0000) 2020-02-05T20:23:48.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:48.174+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:48.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:48.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.179957+0000) 2020-02-05T20:23:48.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:48.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:48.634808+0000) 2020-02-05T20:23:49.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:49.004+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:49.005564+0000) 2020-02-05T20:23:49.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:49.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:49.290648+0000) 2020-02-05T20:23:49.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:49.899+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:49.900317+0000) 2020-02-05T20:23:50.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:50.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:50.146257+0000) 2020-02-05T20:23:50.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:50.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:50.146258+0000) 2020-02-05T20:23:50.562 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:23:50.568 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:23:52.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:52.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:52.251764+0000) 2020-02-05T20:23:53.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:53.181+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.182171+0000) 2020-02-05T20:23:53.188 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:53.183+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.184471+0000) 2020-02-05T20:23:53.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:53.184+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.185643+0000) 2020-02-05T20:23:53.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:53.184+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.185992+0000) 2020-02-05T20:23:53.189 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:53.180+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.181822+0000) 2020-02-05T20:23:53.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:53.182+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.183878+0000) 2020-02-05T20:23:53.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:53.183+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.185157+0000) 2020-02-05T20:23:53.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:53.184+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.186242+0000) 2020-02-05T20:23:53.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:53.185+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.187046+0000) 2020-02-05T20:23:53.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:53.186+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.187320+0000) 2020-02-05T20:23:53.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:53.186+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.187616+0000) 2020-02-05T20:23:53.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:53.186+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.187696+0000) 2020-02-05T20:23:53.191 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:53.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55385 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:53.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:53.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.190926+0000) 2020-02-05T20:23:53.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:53.190+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.191365+0000) 2020-02-05T20:23:53.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:53.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.377471+0000) 2020-02-05T20:23:53.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:53.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.410289+0000) 2020-02-05T20:23:53.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:53.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.410426+0000) 2020-02-05T20:23:53.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:53.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:53.803164+0000) 2020-02-05T20:23:55.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:55.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:55.792749+0000) 2020-02-05T20:23:56.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:56.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:56.652138+0000) 2020-02-05T20:23:56.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:56.893+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:56.893945+0000) 2020-02-05T20:23:58.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:23:58.191+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.193108+0000) 2020-02-05T20:23:58.195 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:23:58.193+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.194315+0000) 2020-02-05T20:23:58.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:58.194+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.195535+0000) 2020-02-05T20:23:58.197 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:58.194+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.196049+0000) 2020-02-05T20:23:58.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:58.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.196618+0000) 2020-02-05T20:23:58.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:58.196+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.197285+0000) 2020-02-05T20:23:58.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:23:58.197+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.198659+0000) 2020-02-05T20:23:58.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:58.198+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.199135+0000) 2020-02-05T20:23:58.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:23:58.198+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.199869+0000) 2020-02-05T20:23:58.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:23:58.198+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.199720+0000) 2020-02-05T20:23:58.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:23:58.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:23:58.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:23:58.198+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.200214+0000) 2020-02-05T20:23:58.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:23:58.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.200112+0000) 2020-02-05T20:23:58.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:23:58.199+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.200485+0000) 2020-02-05T20:23:58.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:58.205+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.206223+0000) 2020-02-05T20:23:58.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:23:58.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:58.635014+0000) 2020-02-05T20:23:59.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:23:59.005+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:59.005783+0000) 2020-02-05T20:23:59.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:23:59.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:59.290857+0000) 2020-02-05T20:23:59.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:23:59.900+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:23:59.900547+0000) 2020-02-05T20:24:00.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:00.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:00.146524+0000) 2020-02-05T20:24:00.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:00.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:00.146520+0000) 2020-02-05T20:24:02.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:02.251+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:02.252019+0000) 2020-02-05T20:24:03.206 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:03.203+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.204359+0000) 2020-02-05T20:24:03.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:03.203+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.205211+0000) 2020-02-05T20:24:03.207 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:03.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.205956+0000) 2020-02-05T20:24:03.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:03.206+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.207405+0000) 2020-02-05T20:24:03.209 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:03.206+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.207758+0000) 2020-02-05T20:24:03.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:03.207+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.208266+0000) 2020-02-05T20:24:03.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:03.207+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.208833+0000) 2020-02-05T20:24:03.211 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:03.208+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.209752+0000) 2020-02-05T20:24:03.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:03.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.210568+0000) 2020-02-05T20:24:03.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:03.209+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.210956+0000) 2020-02-05T20:24:03.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:03.209+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.211130+0000) 2020-02-05T20:24:03.213 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:03.210+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.211594+0000) 2020-02-05T20:24:03.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:03.210+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.211789+0000) 2020-02-05T20:24:03.214 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:03.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:03.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:03.214+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.215136+0000) 2020-02-05T20:24:03.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:03.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.377690+0000) 2020-02-05T20:24:03.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:03.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.410591+0000) 2020-02-05T20:24:03.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:03.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.410532+0000) 2020-02-05T20:24:03.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:03.811+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:03.811594+0000) 2020-02-05T20:24:05.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:05.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:05.792976+0000) 2020-02-05T20:24:06.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:06.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:06.652403+0000) 2020-02-05T20:24:06.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:06.892+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:06.894193+0000) 2020-02-05T20:24:08.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:08.217+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.218209+0000) 2020-02-05T20:24:08.220 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:08.218+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.219192+0000) 2020-02-05T20:24:08.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:08.219+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.220410+0000) 2020-02-05T20:24:08.222 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:08.220+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.221064+0000) 2020-02-05T20:24:08.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:08.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.221824+0000) 2020-02-05T20:24:08.223 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:08.221+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.222217+0000) 2020-02-05T20:24:08.224 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:08.221+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.222906+0000) 2020-02-05T20:24:08.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:08.222+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.223524+0000) 2020-02-05T20:24:08.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:08.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.223800+0000) 2020-02-05T20:24:08.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:08.222+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.224001+0000) 2020-02-05T20:24:08.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:08.222+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.223518+0000) 2020-02-05T20:24:08.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:08.222+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.224254+0000) 2020-02-05T20:24:08.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:08.222+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.224257+0000) 2020-02-05T20:24:08.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:08.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:08.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:08.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.228786+0000) 2020-02-05T20:24:08.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:08.633+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:08.635238+0000) 2020-02-05T20:24:09.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:09.005+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:09.006004+0000) 2020-02-05T20:24:09.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:09.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:09.291135+0000) 2020-02-05T20:24:09.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:09.900+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:09.900775+0000) 2020-02-05T20:24:10.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:10.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:10.146800+0000) 2020-02-05T20:24:10.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:10.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:10.146803+0000) 2020-02-05T20:24:12.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:12.250+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:12.252260+0000) 2020-02-05T20:24:13.232 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:13.230+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.231245+0000) 2020-02-05T20:24:13.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:13.231+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.232230+0000) 2020-02-05T20:24:13.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:13.231+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.232916+0000) 2020-02-05T20:24:13.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:13.231+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.233277+0000) 2020-02-05T20:24:13.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:13.232+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.233637+0000) 2020-02-05T20:24:13.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:13.232+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.234265+0000) 2020-02-05T20:24:13.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:13.234+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.235123+0000) 2020-02-05T20:24:13.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:13.233+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.234900+0000) 2020-02-05T20:24:13.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:13.234+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.236091+0000) 2020-02-05T20:24:13.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:13.235+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.236754+0000) 2020-02-05T20:24:13.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:13.235+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.236966+0000) 2020-02-05T20:24:13.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:13.235+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:13.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:13.236+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.237425+0000) 2020-02-05T20:24:13.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:13.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.237721+0000) 2020-02-05T20:24:13.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:13.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.242501+0000) 2020-02-05T20:24:13.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:13.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.377906+0000) 2020-02-05T20:24:13.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:13.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.410770+0000) 2020-02-05T20:24:13.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:13.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.410742+0000) 2020-02-05T20:24:13.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:13.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:13.819887+0000) 2020-02-05T20:24:15.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:15.793+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:15.793203+0000) 2020-02-05T20:24:16.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:16.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:16.652626+0000) 2020-02-05T20:24:16.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:16.893+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:16.894488+0000) 2020-02-05T20:24:18.244 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:18.242+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.242393+0000) 2020-02-05T20:24:18.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:18.242+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.243924+0000) 2020-02-05T20:24:18.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:18.243+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.245358+0000) 2020-02-05T20:24:18.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:18.245+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.245668+0000) 2020-02-05T20:24:18.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:18.246+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.246403+0000) 2020-02-05T20:24:18.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:18.246+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.246772+0000) 2020-02-05T20:24:18.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:18.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.247331+0000) 2020-02-05T20:24:18.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:18.245+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.247406+0000) 2020-02-05T20:24:18.249 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:18.247+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.247716+0000) 2020-02-05T20:24:18.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:18.246+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.248136+0000) 2020-02-05T20:24:18.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:18.248+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.248600+0000) 2020-02-05T20:24:18.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:18.247+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.249394+0000) 2020-02-05T20:24:18.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:18.248+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.249496+0000) 2020-02-05T20:24:18.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:18.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:18.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:18.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.253909+0000) 2020-02-05T20:24:18.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:18.634+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:18.635495+0000) 2020-02-05T20:24:19.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:19.005+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:19.006217+0000) 2020-02-05T20:24:19.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:19.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:19.291351+0000) 2020-02-05T20:24:19.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:19.900+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:19.901053+0000) 2020-02-05T20:24:20.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:20.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:20.147060+0000) 2020-02-05T20:24:20.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:20.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:20.147065+0000) 2020-02-05T20:24:20.775 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:24:20.781 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:24:22.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:22.251+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:22.252508+0000) 2020-02-05T20:24:23.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:23.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.255228+0000) 2020-02-05T20:24:23.257 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:23.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.255925+0000) 2020-02-05T20:24:23.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:23.254+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.256301+0000) 2020-02-05T20:24:23.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:23.255+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.257396+0000) 2020-02-05T20:24:23.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:23.256+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.257744+0000) 2020-02-05T20:24:23.260 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:23.257+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.258498+0000) 2020-02-05T20:24:23.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:23.257+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.258930+0000) 2020-02-05T20:24:23.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:23.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.259595+0000) 2020-02-05T20:24:23.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:23.258+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.259606+0000) 2020-02-05T20:24:23.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:23.259+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.260176+0000) 2020-02-05T20:24:23.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:23.260+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.261407+0000) 2020-02-05T20:24:23.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:23.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.261556+0000) 2020-02-05T20:24:23.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:23.260+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.261961+0000) 2020-02-05T20:24:23.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:23.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:23.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:23.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.266764+0000) 2020-02-05T20:24:23.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:23.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.378146+0000) 2020-02-05T20:24:23.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:23.409+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.411024+0000) 2020-02-05T20:24:23.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:23.409+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.411019+0000) 2020-02-05T20:24:23.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:23.826+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:23.827937+0000) 2020-02-05T20:24:25.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:25.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:25.793441+0000) 2020-02-05T20:24:26.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:26.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:26.652851+0000) 2020-02-05T20:24:26.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:26.894+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:26.894727+0000) 2020-02-05T20:24:28.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:28.266+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.267393+0000) 2020-02-05T20:24:28.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:28.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.268651+0000) 2020-02-05T20:24:28.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:28.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.269144+0000) 2020-02-05T20:24:28.271 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:28.268+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.269606+0000) 2020-02-05T20:24:28.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:28.268+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.270116+0000) 2020-02-05T20:24:28.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:28.269+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.270562+0000) 2020-02-05T20:24:28.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:28.269+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.270932+0000) 2020-02-05T20:24:28.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:28.269+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.271247+0000) 2020-02-05T20:24:28.273 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:28.270+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.271842+0000) 2020-02-05T20:24:28.274 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:28.271+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.273031+0000) 2020-02-05T20:24:28.274 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:28.271+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.273040+0000) 2020-02-05T20:24:28.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:28.272+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.273664+0000) 2020-02-05T20:24:28.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:28.272+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.273882+0000) 2020-02-05T20:24:28.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:28.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:28.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:28.276+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.277565+0000) 2020-02-05T20:24:28.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:28.635+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:28.635762+0000) 2020-02-05T20:24:29.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:29.005+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:29.006435+0000) 2020-02-05T20:24:29.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:29.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:29.291580+0000) 2020-02-05T20:24:29.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:29.900+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:29.901306+0000) 2020-02-05T20:24:30.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:30.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:30.147297+0000) 2020-02-05T20:24:30.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:30.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:30.147306+0000) 2020-02-05T20:24:32.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:32.252+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:32.252763+0000) 2020-02-05T20:24:33.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:33.279+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.280279+0000) 2020-02-05T20:24:33.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:33.279+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.280898+0000) 2020-02-05T20:24:33.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:33.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.281901+0000) 2020-02-05T20:24:33.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:33.281+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.282128+0000) 2020-02-05T20:24:33.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:33.281+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.282530+0000) 2020-02-05T20:24:33.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:33.282+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.283244+0000) 2020-02-05T20:24:33.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:33.283+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.283726+0000) 2020-02-05T20:24:33.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:33.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.284182+0000) 2020-02-05T20:24:33.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:33.284+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.284932+0000) 2020-02-05T20:24:33.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:33.283+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.284986+0000) 2020-02-05T20:24:33.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:33.285+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.286206+0000) 2020-02-05T20:24:33.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:33.286+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.286913+0000) 2020-02-05T20:24:33.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:33.286+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.287456+0000) 2020-02-05T20:24:33.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:33.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:33.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:33.290+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.291699+0000) 2020-02-05T20:24:33.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:33.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.378381+0000) 2020-02-05T20:24:33.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:33.410+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.411257+0000) 2020-02-05T20:24:33.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:33.410+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.411313+0000) 2020-02-05T20:24:33.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:33.835+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:33.836492+0000) 2020-02-05T20:24:35.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:35.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:35.793669+0000) 2020-02-05T20:24:36.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:36.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:36.653104+0000) 2020-02-05T20:24:36.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:36.894+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:36.894980+0000) 2020-02-05T20:24:38.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:38.294+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.295111+0000) 2020-02-05T20:24:38.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:38.294+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.295383+0000) 2020-02-05T20:24:38.297 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:38.294+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.295720+0000) 2020-02-05T20:24:38.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:38.295+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.295933+0000) 2020-02-05T20:24:38.298 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:38.295+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.296249+0000) 2020-02-05T20:24:38.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:38.295+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.296642+0000) 2020-02-05T20:24:38.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:38.297+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.297561+0000) 2020-02-05T20:24:38.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:38.297+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.297357+0000) 2020-02-05T20:24:38.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:38.297+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.297705+0000) 2020-02-05T20:24:38.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:38.296+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.297362+0000) 2020-02-05T20:24:38.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:38.298+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.298886+0000) 2020-02-05T20:24:38.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:38.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.298679+0000) 2020-02-05T20:24:38.301 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:38.298+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.299230+0000) 2020-02-05T20:24:38.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:38.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:38.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:38.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.301345+0000) 2020-02-05T20:24:38.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:38.635+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:38.635985+0000) 2020-02-05T20:24:39.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:39.005+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:39.006675+0000) 2020-02-05T20:24:39.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:39.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:39.291816+0000) 2020-02-05T20:24:39.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:39.900+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:39.901547+0000) 2020-02-05T20:24:40.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:40.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:40.147526+0000) 2020-02-05T20:24:40.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:40.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:40.147625+0000) 2020-02-05T20:24:42.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:42.252+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:42.253055+0000) 2020-02-05T20:24:43.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:43.305+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.306498+0000) 2020-02-05T20:24:43.308 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:43.305+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.307055+0000) 2020-02-05T20:24:43.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:43.306+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.307654+0000) 2020-02-05T20:24:43.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:43.308+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.309065+0000) 2020-02-05T20:24:43.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:43.309+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.309749+0000) 2020-02-05T20:24:43.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:43.309+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.310266+0000) 2020-02-05T20:24:43.312 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:43.309+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.310495+0000) 2020-02-05T20:24:43.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:43.310+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.311305+0000) 2020-02-05T20:24:43.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:43.311+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.312296+0000) 2020-02-05T20:24:43.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:43.311+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.312231+0000) 2020-02-05T20:24:43.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:43.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.312696+0000) 2020-02-05T20:24:43.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:43.311+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.312836+0000) 2020-02-05T20:24:43.315 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:43.312+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.313068+0000) 2020-02-05T20:24:43.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:43.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:43.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:43.318+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.319304+0000) 2020-02-05T20:24:43.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:43.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.378612+0000) 2020-02-05T20:24:43.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:43.410+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.411411+0000) 2020-02-05T20:24:43.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:43.410+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.411574+0000) 2020-02-05T20:24:43.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:43.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:43.843522+0000) 2020-02-05T20:24:45.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:45.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:45.793921+0000) 2020-02-05T20:24:46.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:46.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:46.653341+0000) 2020-02-05T20:24:46.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:46.894+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:46.895235+0000) 2020-02-05T20:24:48.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:48.318+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.319688+0000) 2020-02-05T20:24:48.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:48.320+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.321277+0000) 2020-02-05T20:24:48.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:48.320+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.321665+0000) 2020-02-05T20:24:48.330 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:48.321+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.322199+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:48.321+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.322810+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:48.322+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.323461+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:48.323+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.324774+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:48.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.327122+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:48.320+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.321022+0000) 2020-02-05T20:24:48.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:48.322+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.322705+0000) 2020-02-05T20:24:48.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:48.323+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.323822+0000) 2020-02-05T20:24:48.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:48.324+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.324693+0000) 2020-02-05T20:24:48.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:48.324+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.324889+0000) 2020-02-05T20:24:48.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:48.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55824 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:48.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:48.325+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.325926+0000) 2020-02-05T20:24:48.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:48.635+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:48.636216+0000) 2020-02-05T20:24:49.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:49.006+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:49.006897+0000) 2020-02-05T20:24:49.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:49.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:49.292026+0000) 2020-02-05T20:24:49.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:49.901+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:49.901810+0000) 2020-02-05T20:24:50.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:50.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:50.147778+0000) 2020-02-05T20:24:50.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:50.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:50.147879+0000) 2020-02-05T20:24:50.964 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:24:50.970 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:24:52.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:52.253+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:52.253313+0000) 2020-02-05T20:24:53.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:53.331+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.332376+0000) 2020-02-05T20:24:53.334 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:53.331+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.332935+0000) 2020-02-05T20:24:53.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:53.333+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.333465+0000) 2020-02-05T20:24:53.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:53.333+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.334418+0000) 2020-02-05T20:24:53.336 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:53.333+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.334581+0000) 2020-02-05T20:24:53.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:53.333+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.334189+0000) 2020-02-05T20:24:53.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:53.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.334459+0000) 2020-02-05T20:24:53.337 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:53.334+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.335240+0000) 2020-02-05T20:24:53.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:53.335+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.335994+0000) 2020-02-05T20:24:53.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:53.335+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.335876+0000) 2020-02-05T20:24:53.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:53.336+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.336419+0000) 2020-02-05T20:24:53.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:53.334+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.335881+0000) 2020-02-05T20:24:53.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:53.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.337036+0000) 2020-02-05T20:24:53.340 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:53.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:53.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:53.340+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.341293+0000) 2020-02-05T20:24:53.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:53.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.378850+0000) 2020-02-05T20:24:53.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:53.411+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.411626+0000) 2020-02-05T20:24:53.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:53.411+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.411778+0000) 2020-02-05T20:24:53.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:53.854+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:53.855286+0000) 2020-02-05T20:24:55.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:55.793+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:55.794174+0000) 2020-02-05T20:24:56.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:56.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:56.653559+0000) 2020-02-05T20:24:56.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:56.895+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:56.895533+0000) 2020-02-05T20:24:58.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:58.343+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.344809+0000) 2020-02-05T20:24:58.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:24:58.344+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.345289+0000) 2020-02-05T20:24:58.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:24:58.345+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.345668+0000) 2020-02-05T20:24:58.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:24:58.345+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.346345+0000) 2020-02-05T20:24:58.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:24:58.345+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.346636+0000) 2020-02-05T20:24:58.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:58.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.347439+0000) 2020-02-05T20:24:58.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:58.346+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.347655+0000) 2020-02-05T20:24:58.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:24:58.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.348503+0000) 2020-02-05T20:24:58.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:24:58.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.348252+0000) 2020-02-05T20:24:58.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:58.348+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.349100+0000) 2020-02-05T20:24:58.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:58.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.349618+0000) 2020-02-05T20:24:58.351 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:24:58.348+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.349877+0000) 2020-02-05T20:24:58.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:24:58.349+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.350045+0000) 2020-02-05T20:24:58.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:24:58.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:24:58.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:58.355+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.356308+0000) 2020-02-05T20:24:58.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:24:58.636+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:58.636478+0000) 2020-02-05T20:24:59.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:24:59.006+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:59.007143+0000) 2020-02-05T20:24:59.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:24:59.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:59.292270+0000) 2020-02-05T20:24:59.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:24:59.901+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:24:59.902038+0000) 2020-02-05T20:25:00.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:00.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:00.148069+0000) 2020-02-05T20:25:00.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:00.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:00.148108+0000) 2020-02-05T20:25:02.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:02.253+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:02.253562+0000) 2020-02-05T20:25:03.356 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:03.354+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.354736+0000) 2020-02-05T20:25:03.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:03.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.356399+0000) 2020-02-05T20:25:03.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:03.355+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.355657+0000) 2020-02-05T20:25:03.360 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:03.358+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.358836+0000) 2020-02-05T20:25:03.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:03.358+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.359333+0000) 2020-02-05T20:25:03.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:03.358+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.359996+0000) 2020-02-05T20:25:03.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:03.360+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.360836+0000) 2020-02-05T20:25:03.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:03.360+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.361273+0000) 2020-02-05T20:25:03.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:03.360+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.361646+0000) 2020-02-05T20:25:03.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:03.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.361738+0000) 2020-02-05T20:25:03.363 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:03.361+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.362386+0000) 2020-02-05T20:25:03.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:03.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.362297+0000) 2020-02-05T20:25:03.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:03.361+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.362172+0000) 2020-02-05T20:25:03.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:03.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:03.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:03.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.367762+0000) 2020-02-05T20:25:03.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:03.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.379088+0000) 2020-02-05T20:25:03.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:03.410+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.411871+0000) 2020-02-05T20:25:03.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:03.410+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.412005+0000) 2020-02-05T20:25:03.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:03.862+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:03.863565+0000) 2020-02-05T20:25:05.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:05.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:05.794400+0000) 2020-02-05T20:25:06.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:06.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:06.653806+0000) 2020-02-05T20:25:06.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:06.895+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:06.895818+0000) 2020-02-05T20:25:08.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:08.368+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.369509+0000) 2020-02-05T20:25:08.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:08.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.370684+0000) 2020-02-05T20:25:08.372 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:08.370+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.370887+0000) 2020-02-05T20:25:08.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:08.371+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.371405+0000) 2020-02-05T20:25:08.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:08.370+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.371765+0000) 2020-02-05T20:25:08.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:08.371+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.372195+0000) 2020-02-05T20:25:08.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:08.371+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.372527+0000) 2020-02-05T20:25:08.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:08.371+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.372709+0000) 2020-02-05T20:25:08.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:08.372+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.373198+0000) 2020-02-05T20:25:08.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:08.373+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.373889+0000) 2020-02-05T20:25:08.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:08.374+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.374791+0000) 2020-02-05T20:25:08.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:08.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.374449+0000) 2020-02-05T20:25:08.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:08.375+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.375686+0000) 2020-02-05T20:25:08.377 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:08.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 55982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:08.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:08.378+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.379302+0000) 2020-02-05T20:25:08.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:08.636+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:08.636723+0000) 2020-02-05T20:25:09.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:09.007+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:09.007374+0000) 2020-02-05T20:25:09.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:09.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:09.292490+0000) 2020-02-05T20:25:09.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:09.902+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:09.902306+0000) 2020-02-05T20:25:10.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:10.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:10.148335+0000) 2020-02-05T20:25:10.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:10.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:10.148353+0000) 2020-02-05T20:25:12.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:12.252+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:12.253826+0000) 2020-02-05T20:25:13.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:13.379+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.380861+0000) 2020-02-05T20:25:13.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:13.381+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.382147+0000) 2020-02-05T20:25:13.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:13.382+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.383209+0000) 2020-02-05T20:25:13.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:13.382+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.383877+0000) 2020-02-05T20:25:13.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:13.382+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.384320+0000) 2020-02-05T20:25:13.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:13.382+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.384176+0000) 2020-02-05T20:25:13.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:13.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.385181+0000) 2020-02-05T20:25:13.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:13.384+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.385289+0000) 2020-02-05T20:25:13.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:13.384+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.386327+0000) 2020-02-05T20:25:13.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:13.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.386956+0000) 2020-02-05T20:25:13.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:13.385+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.386876+0000) 2020-02-05T20:25:13.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:13.386+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.387633+0000) 2020-02-05T20:25:13.389 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:13.386+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.387904+0000) 2020-02-05T20:25:13.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:13.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.388437+0000) 2020-02-05T20:25:13.390 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:13.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:13.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:13.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.391571+0000) 2020-02-05T20:25:13.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:13.411+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.412121+0000) 2020-02-05T20:25:13.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:13.411+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.412254+0000) 2020-02-05T20:25:13.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:13.870+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:13.871867+0000) 2020-02-05T20:25:15.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:15.793+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:15.794657+0000) 2020-02-05T20:25:16.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:16.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:16.654050+0000) 2020-02-05T20:25:16.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:16.895+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:16.896074+0000) 2020-02-05T20:25:18.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:18.394+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.395462+0000) 2020-02-05T20:25:18.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:18.394+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.396237+0000) 2020-02-05T20:25:18.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:18.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.396525+0000) 2020-02-05T20:25:18.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:18.395+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.396711+0000) 2020-02-05T20:25:18.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:18.396+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.397639+0000) 2020-02-05T20:25:18.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:18.398+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.399151+0000) 2020-02-05T20:25:18.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:18.397+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.399066+0000) 2020-02-05T20:25:18.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:18.397+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.399352+0000) 2020-02-05T20:25:18.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:18.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.399994+0000) 2020-02-05T20:25:18.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:18.399+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.400400+0000) 2020-02-05T20:25:18.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:18.399+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.400685+0000) 2020-02-05T20:25:18.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:18.399+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.401164+0000) 2020-02-05T20:25:18.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:18.400+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.401287+0000) 2020-02-05T20:25:18.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:18.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:18.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:18.402+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.403985+0000) 2020-02-05T20:25:18.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:18.636+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:18.636989+0000) 2020-02-05T20:25:19.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:19.006+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:19.007587+0000) 2020-02-05T20:25:19.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:19.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:19.292696+0000) 2020-02-05T20:25:19.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:19.901+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:19.902545+0000) 2020-02-05T20:25:20.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:20.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:20.148585+0000) 2020-02-05T20:25:20.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:20.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:20.148639+0000) 2020-02-05T20:25:21.177 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:25:21.184 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:25:22.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:22.253+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:22.254085+0000) 2020-02-05T20:25:23.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:23.379+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.381111+0000) 2020-02-05T20:25:23.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:23.407+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.408242+0000) 2020-02-05T20:25:23.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:23.407+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.408720+0000) 2020-02-05T20:25:23.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:23.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.409310+0000) 2020-02-05T20:25:23.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:23.409+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.410552+0000) 2020-02-05T20:25:23.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:23.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.411849+0000) 2020-02-05T20:25:23.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:23.410+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.412164+0000) 2020-02-05T20:25:23.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:23.411+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.412469+0000) 2020-02-05T20:25:23.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:23.411+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.413215+0000) 2020-02-05T20:25:23.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:23.412+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.413437+0000) 2020-02-05T20:25:23.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:23.412+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.413573+0000) 2020-02-05T20:25:23.415 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:23.412+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:23.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:23.412+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.413990+0000) 2020-02-05T20:25:23.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:23.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.413262+0000) 2020-02-05T20:25:23.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:23.412+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.414032+0000) 2020-02-05T20:25:23.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:23.415+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.416142+0000) 2020-02-05T20:25:23.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:23.417+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.418715+0000) 2020-02-05T20:25:23.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:23.879+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:23.879925+0000) 2020-02-05T20:25:25.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:25.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:25.794887+0000) 2020-02-05T20:25:26.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:26.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:26.654270+0000) 2020-02-05T20:25:26.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:26.895+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:26.896325+0000) 2020-02-05T20:25:28.426 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:28.420+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.421676+0000) 2020-02-05T20:25:28.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:28.420+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.422229+0000) 2020-02-05T20:25:28.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:28.421+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.423009+0000) 2020-02-05T20:25:28.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:28.423+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.424475+0000) 2020-02-05T20:25:28.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:28.417+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.418920+0000) 2020-02-05T20:25:28.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:28.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.420768+0000) 2020-02-05T20:25:28.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:28.418+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.419479+0000) 2020-02-05T20:25:28.429 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:28.421+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.423008+0000) 2020-02-05T20:25:28.430 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:28.422+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.423600+0000) 2020-02-05T20:25:28.430 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:28.422+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.424013+0000) 2020-02-05T20:25:28.431 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:28.423+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.424951+0000) 2020-02-05T20:25:28.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:28.423+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.425265+0000) 2020-02-05T20:25:28.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:28.424+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.425559+0000) 2020-02-05T20:25:28.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:28.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:28.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:28.430+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.431497+0000) 2020-02-05T20:25:28.641 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:28.636+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:28.637213+0000) 2020-02-05T20:25:29.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:29.007+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:29.007807+0000) 2020-02-05T20:25:29.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:29.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:29.292918+0000) 2020-02-05T20:25:29.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:29.901+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:29.902769+0000) 2020-02-05T20:25:30.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:30.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:30.148839+0000) 2020-02-05T20:25:30.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:30.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:30.148903+0000) 2020-02-05T20:25:30.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:30.162+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:30.162986+0000) 2020-02-05T20:25:32.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:32.252+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:32.254322+0000) 2020-02-05T20:25:33.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:33.379+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.381353+0000) 2020-02-05T20:25:33.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:33.411+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.412689+0000) 2020-02-05T20:25:33.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:33.430+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.431277+0000) 2020-02-05T20:25:33.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:33.431+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.432560+0000) 2020-02-05T20:25:33.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:33.432+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.433675+0000) 2020-02-05T20:25:33.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:33.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.434143+0000) 2020-02-05T20:25:33.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:33.433+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.434866+0000) 2020-02-05T20:25:33.436 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:33.434+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.435134+0000) 2020-02-05T20:25:33.437 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:33.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.435761+0000) 2020-02-05T20:25:33.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:33.435+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.436694+0000) 2020-02-05T20:25:33.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:33.435+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.437374+0000) 2020-02-05T20:25:33.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:33.435+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.437375+0000) 2020-02-05T20:25:33.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:33.436+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.437790+0000) 2020-02-05T20:25:33.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:33.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56181 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:33.443 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:33.435+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.436424+0000) 2020-02-05T20:25:33.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:33.435+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.437062+0000) 2020-02-05T20:25:33.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:33.443+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.444573+0000) 2020-02-05T20:25:33.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:33.887+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:33.888138+0000) 2020-02-05T20:25:35.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:35.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:35.795149+0000) 2020-02-05T20:25:36.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:36.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:36.654519+0000) 2020-02-05T20:25:36.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:36.895+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:36.896580+0000) 2020-02-05T20:25:38.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:38.443+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.444946+0000) 2020-02-05T20:25:38.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:38.443+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.445398+0000) 2020-02-05T20:25:38.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:38.444+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.445957+0000) 2020-02-05T20:25:38.448 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:38.445+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.446527+0000) 2020-02-05T20:25:38.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:38.445+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.446702+0000) 2020-02-05T20:25:38.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:38.446+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.447505+0000) 2020-02-05T20:25:38.449 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:38.446+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.447829+0000) 2020-02-05T20:25:38.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:38.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.448141+0000) 2020-02-05T20:25:38.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:38.447+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.448805+0000) 2020-02-05T20:25:38.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:38.448+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.449459+0000) 2020-02-05T20:25:38.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:38.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:38.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:38.447+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.449179+0000) 2020-02-05T20:25:38.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:38.449+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.450834+0000) 2020-02-05T20:25:38.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:38.453+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.454718+0000) 2020-02-05T20:25:38.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:38.453+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.455065+0000) 2020-02-05T20:25:38.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:38.636+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:38.637495+0000) 2020-02-05T20:25:39.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:39.006+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:39.008043+0000) 2020-02-05T20:25:39.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:39.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:39.293152+0000) 2020-02-05T20:25:39.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:39.902+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:39.903029+0000) 2020-02-05T20:25:40.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:40.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:40.149094+0000) 2020-02-05T20:25:40.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:40.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:40.149200+0000) 2020-02-05T20:25:40.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:40.162+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:40.163207+0000) 2020-02-05T20:25:42.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:42.253+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:42.254688+0000) 2020-02-05T20:25:43.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:43.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.381652+0000) 2020-02-05T20:25:43.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:43.412+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.412921+0000) 2020-02-05T20:25:43.456 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:43.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.453970+0000) 2020-02-05T20:25:43.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:43.454+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.455104+0000) 2020-02-05T20:25:43.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:43.455+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.456709+0000) 2020-02-05T20:25:43.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:43.456+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.457443+0000) 2020-02-05T20:25:43.459 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:43.455+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.457380+0000) 2020-02-05T20:25:43.459 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:43.456+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.458328+0000) 2020-02-05T20:25:43.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:43.457+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.458641+0000) 2020-02-05T20:25:43.462 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:43.458+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.459269+0000) 2020-02-05T20:25:43.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:43.458+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.459820+0000) 2020-02-05T20:25:43.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:43.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.460619+0000) 2020-02-05T20:25:43.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:43.459+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.461054+0000) 2020-02-05T20:25:43.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:43.458+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.459464+0000) 2020-02-05T20:25:43.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:43.460+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.461554+0000) 2020-02-05T20:25:43.464 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:43.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:43.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:43.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.466241+0000) 2020-02-05T20:25:43.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:43.895+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:43.896325+0000) 2020-02-05T20:25:45.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:45.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:45.795447+0000) 2020-02-05T20:25:46.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:46.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:46.654749+0000) 2020-02-05T20:25:46.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:46.896+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:46.896828+0000) 2020-02-05T20:25:48.469 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:48.466+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.467949+0000) 2020-02-05T20:25:48.470 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:48.467+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.468830+0000) 2020-02-05T20:25:48.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:48.468+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.469497+0000) 2020-02-05T20:25:48.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:48.468+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.469868+0000) 2020-02-05T20:25:48.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:48.468+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.470264+0000) 2020-02-05T20:25:48.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:48.469+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.470391+0000) 2020-02-05T20:25:48.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:48.469+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.470642+0000) 2020-02-05T20:25:48.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:48.469+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.470830+0000) 2020-02-05T20:25:48.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:48.470+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.471426+0000) 2020-02-05T20:25:48.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:48.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.471662+0000) 2020-02-05T20:25:48.474 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:48.470+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.472041+0000) 2020-02-05T20:25:48.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:48.469+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.471356+0000) 2020-02-05T20:25:48.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:48.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.471674+0000) 2020-02-05T20:25:48.475 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:48.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:48.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:48.475+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.476258+0000) 2020-02-05T20:25:48.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:48.637+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:48.637789+0000) 2020-02-05T20:25:49.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:49.007+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:49.008300+0000) 2020-02-05T20:25:49.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:49.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:49.293410+0000) 2020-02-05T20:25:49.905 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:49.902+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:49.903309+0000) 2020-02-05T20:25:50.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:50.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:50.149338+0000) 2020-02-05T20:25:50.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:50.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:50.149509+0000) 2020-02-05T20:25:50.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:50.163+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:50.163427+0000) 2020-02-05T20:25:51.327 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:25:51.334 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:25:52.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:52.254+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:52.254947+0000) 2020-02-05T20:25:53.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:53.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.381858+0000) 2020-02-05T20:25:53.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:53.412+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.413158+0000) 2020-02-05T20:25:53.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:53.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.479135+0000) 2020-02-05T20:25:53.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:53.479+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.480454+0000) 2020-02-05T20:25:53.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:53.479+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.480924+0000) 2020-02-05T20:25:53.483 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:53.480+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.481686+0000) 2020-02-05T20:25:53.484 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:53.482+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.483177+0000) 2020-02-05T20:25:53.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:53.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.483628+0000) 2020-02-05T20:25:53.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:53.482+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.483653+0000) 2020-02-05T20:25:53.486 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:53.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.484596+0000) 2020-02-05T20:25:53.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:53.483+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.484518+0000) 2020-02-05T20:25:53.487 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:53.483+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.484785+0000) 2020-02-05T20:25:53.487 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:53.483+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.485155+0000) 2020-02-05T20:25:53.487 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:53.483+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.485336+0000) 2020-02-05T20:25:53.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:53.484+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.485656+0000) 2020-02-05T20:25:53.488 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:53.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:53.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:53.487+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.488668+0000) 2020-02-05T20:25:53.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:53.907+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:53.908146+0000) 2020-02-05T20:25:55.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:55.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:55.795710+0000) 2020-02-05T20:25:56.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:56.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:56.655027+0000) 2020-02-05T20:25:56.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:56.896+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:56.897128+0000) 2020-02-05T20:25:58.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:58.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.491371+0000) 2020-02-05T20:25:58.494 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:25:58.491+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.492971+0000) 2020-02-05T20:25:58.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:58.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.493829+0000) 2020-02-05T20:25:58.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:25:58.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.494683+0000) 2020-02-05T20:25:58.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:25:58.494+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.495265+0000) 2020-02-05T20:25:58.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:25:58.494+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.495714+0000) 2020-02-05T20:25:58.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:25:58.494+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.496226+0000) 2020-02-05T20:25:58.498 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:25:58.495+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.496657+0000) 2020-02-05T20:25:58.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:58.496+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.497150+0000) 2020-02-05T20:25:58.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:25:58.496+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.497454+0000) 2020-02-05T20:25:58.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:58.496+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.497867+0000) 2020-02-05T20:25:58.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:25:58.496+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.497657+0000) 2020-02-05T20:25:58.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:25:58.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:25:58.500 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:58.497+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.498662+0000) 2020-02-05T20:25:58.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:58.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.499714+0000) 2020-02-05T20:25:58.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:25:58.637+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:58.638044+0000) 2020-02-05T20:25:59.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:25:59.008+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:59.008553+0000) 2020-02-05T20:25:59.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:25:59.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:59.293641+0000) 2020-02-05T20:25:59.905 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:25:59.903+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:25:59.903543+0000) 2020-02-05T20:26:00.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:00.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:00.149568+0000) 2020-02-05T20:26:00.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:00.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:00.149755+0000) 2020-02-05T20:26:00.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:00.162+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:00.163676+0000) 2020-02-05T20:26:02.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:02.254+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:02.255203+0000) 2020-02-05T20:26:03.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:03.381+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.382103+0000) 2020-02-05T20:26:03.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:03.413+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.413455+0000) 2020-02-05T20:26:03.506 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:03.503+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.504379+0000) 2020-02-05T20:26:03.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:03.505+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.505710+0000) 2020-02-05T20:26:03.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:03.504+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.505363+0000) 2020-02-05T20:26:03.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:03.506+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.506330+0000) 2020-02-05T20:26:03.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:03.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.507599+0000) 2020-02-05T20:26:03.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:03.506+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.507052+0000) 2020-02-05T20:26:03.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:03.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.508647+0000) 2020-02-05T20:26:03.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:03.509+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.509465+0000) 2020-02-05T20:26:03.511 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:03.509+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.509654+0000) 2020-02-05T20:26:03.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:03.509+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.509596+0000) 2020-02-05T20:26:03.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:03.509+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.509705+0000) 2020-02-05T20:26:03.512 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:03.509+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.510165+0000) 2020-02-05T20:26:03.512 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:03.510+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.510347+0000) 2020-02-05T20:26:03.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:03.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:03.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:03.515+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.515465+0000) 2020-02-05T20:26:03.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:03.916+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:03.916612+0000) 2020-02-05T20:26:05.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:05.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:05.795941+0000) 2020-02-05T20:26:06.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:06.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:06.655285+0000) 2020-02-05T20:26:06.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:06.896+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:06.897402+0000) 2020-02-05T20:26:08.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:08.516+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.516549+0000) 2020-02-05T20:26:08.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:08.517+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.518248+0000) 2020-02-05T20:26:08.520 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:08.517+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.518467+0000) 2020-02-05T20:26:08.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:08.518+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.519832+0000) 2020-02-05T20:26:08.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:08.520+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.520682+0000) 2020-02-05T20:26:08.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:08.519+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.520772+0000) 2020-02-05T20:26:08.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:08.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.521455+0000) 2020-02-05T20:26:08.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:08.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.522328+0000) 2020-02-05T20:26:08.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:08.521+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.522864+0000) 2020-02-05T20:26:08.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:08.522+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.522558+0000) 2020-02-05T20:26:08.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:08.522+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56455 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:08.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:08.522+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.523426+0000) 2020-02-05T20:26:08.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:08.523+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.523459+0000) 2020-02-05T20:26:08.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:08.523+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.523668+0000) 2020-02-05T20:26:08.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:08.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.529864+0000) 2020-02-05T20:26:08.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:08.637+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:08.638277+0000) 2020-02-05T20:26:09.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:09.007+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:09.008759+0000) 2020-02-05T20:26:09.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:09.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:09.293852+0000) 2020-02-05T20:26:09.905 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:09.902+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:09.903776+0000) 2020-02-05T20:26:10.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:10.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:10.149820+0000) 2020-02-05T20:26:10.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:10.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:10.150016+0000) 2020-02-05T20:26:10.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:10.162+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:10.163904+0000) 2020-02-05T20:26:12.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:12.255+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:12.255497+0000) 2020-02-05T20:26:13.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:13.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.382352+0000) 2020-02-05T20:26:13.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:13.412+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.413750+0000) 2020-02-05T20:26:13.530 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:13.527+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.527693+0000) 2020-02-05T20:26:13.531 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:13.527+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.529049+0000) 2020-02-05T20:26:13.531 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:13.528+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.529550+0000) 2020-02-05T20:26:13.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:13.528+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.529167+0000) 2020-02-05T20:26:13.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:13.529+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.530398+0000) 2020-02-05T20:26:13.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:13.530+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.531318+0000) 2020-02-05T20:26:13.533 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:13.530+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.531883+0000) 2020-02-05T20:26:13.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:13.531+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.532319+0000) 2020-02-05T20:26:13.534 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:13.531+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.532654+0000) 2020-02-05T20:26:13.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:13.532+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.532737+0000) 2020-02-05T20:26:13.535 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:13.532+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.533310+0000) 2020-02-05T20:26:13.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:13.532+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.533353+0000) 2020-02-05T20:26:13.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:13.533+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.533931+0000) 2020-02-05T20:26:13.536 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:13.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:13.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:13.538+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.539508+0000) 2020-02-05T20:26:13.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:13.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:13.924916+0000) 2020-02-05T20:26:15.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:15.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:15.796197+0000) 2020-02-05T20:26:16.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:16.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:16.655517+0000) 2020-02-05T20:26:16.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:16.897+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:16.897620+0000) 2020-02-05T20:26:18.541 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:18.538+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.540024+0000) 2020-02-05T20:26:18.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:18.540+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.541213+0000) 2020-02-05T20:26:18.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:18.540+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.541717+0000) 2020-02-05T20:26:18.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:18.540+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.541690+0000) 2020-02-05T20:26:18.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:18.541+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.542072+0000) 2020-02-05T20:26:18.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:18.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.542552+0000) 2020-02-05T20:26:18.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:18.542+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.542904+0000) 2020-02-05T20:26:18.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:18.542+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.543366+0000) 2020-02-05T20:26:18.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:18.544+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.544510+0000) 2020-02-05T20:26:18.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:18.543+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.544806+0000) 2020-02-05T20:26:18.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:18.544+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.545265+0000) 2020-02-05T20:26:18.547 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:18.545+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.545691+0000) 2020-02-05T20:26:18.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:18.546+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.546482+0000) 2020-02-05T20:26:18.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:18.547+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:18.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:18.548+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.549737+0000) 2020-02-05T20:26:18.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:18.638+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:18.638515+0000) 2020-02-05T20:26:19.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:19.008+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:19.008979+0000) 2020-02-05T20:26:19.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:19.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:19.294063+0000) 2020-02-05T20:26:19.905 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:19.903+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:19.904019+0000) 2020-02-05T20:26:20.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:20.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:20.150092+0000) 2020-02-05T20:26:20.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:20.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:20.150250+0000) 2020-02-05T20:26:20.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:20.163+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:20.164177+0000) 2020-02-05T20:26:21.470 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:26:21.476 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:26:22.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:22.255+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:22.255801+0000) 2020-02-05T20:26:23.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:23.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.382627+0000) 2020-02-05T20:26:23.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:23.413+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.413984+0000) 2020-02-05T20:26:23.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:23.553+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.553789+0000) 2020-02-05T20:26:23.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:23.553+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.554745+0000) 2020-02-05T20:26:23.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:23.555+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.556035+0000) 2020-02-05T20:26:23.558 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:23.555+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.556789+0000) 2020-02-05T20:26:23.558 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:23.556+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.557408+0000) 2020-02-05T20:26:23.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:23.557+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.558177+0000) 2020-02-05T20:26:23.560 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:23.557+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.558204+0000) 2020-02-05T20:26:23.561 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:23.557+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.558655+0000) 2020-02-05T20:26:23.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:23.557+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.559024+0000) 2020-02-05T20:26:23.561 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:23.558+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.558936+0000) 2020-02-05T20:26:23.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:23.558+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.559022+0000) 2020-02-05T20:26:23.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:23.558+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.559216+0000) 2020-02-05T20:26:23.562 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:23.559+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56575 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:23.562 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:23.558+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.559860+0000) 2020-02-05T20:26:23.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:23.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.564681+0000) 2020-02-05T20:26:23.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:23.932+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:23.932885+0000) 2020-02-05T20:26:25.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:25.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:25.796429+0000) 2020-02-05T20:26:26.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:26.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:26.655747+0000) 2020-02-05T20:26:26.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:26.897+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:26.897864+0000) 2020-02-05T20:26:28.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:28.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.564988+0000) 2020-02-05T20:26:28.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:28.564+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.565482+0000) 2020-02-05T20:26:28.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:28.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.566041+0000) 2020-02-05T20:26:28.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:28.566+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.567455+0000) 2020-02-05T20:26:28.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:28.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.567954+0000) 2020-02-05T20:26:28.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:28.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.568389+0000) 2020-02-05T20:26:28.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:28.568+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.569210+0000) 2020-02-05T20:26:28.572 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:28.569+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.569544+0000) 2020-02-05T20:26:28.572 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:28.569+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.569795+0000) 2020-02-05T20:26:28.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:28.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.569135+0000) 2020-02-05T20:26:28.573 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:28.568+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.569236+0000) 2020-02-05T20:26:28.573 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:28.569+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.570146+0000) 2020-02-05T20:26:28.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:28.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:28.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:28.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.573486+0000) 2020-02-05T20:26:28.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:28.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.573810+0000) 2020-02-05T20:26:28.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:28.638+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:28.638768+0000) 2020-02-05T20:26:29.013 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:29.008+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:29.009196+0000) 2020-02-05T20:26:29.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:29.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:29.294331+0000) 2020-02-05T20:26:29.905 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:29.903+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:29.904275+0000) 2020-02-05T20:26:30.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:30.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:30.150318+0000) 2020-02-05T20:26:30.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:30.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:30.150532+0000) 2020-02-05T20:26:30.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:30.163+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:30.164360+0000) 2020-02-05T20:26:32.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:32.255+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:32.256054+0000) 2020-02-05T20:26:33.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:33.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.382852+0000) 2020-02-05T20:26:33.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:33.413+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.414238+0000) 2020-02-05T20:26:33.579 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:33.576+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.577117+0000) 2020-02-05T20:26:33.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:33.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.577891+0000) 2020-02-05T20:26:33.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:33.577+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.578725+0000) 2020-02-05T20:26:33.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:33.578+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.579961+0000) 2020-02-05T20:26:33.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:33.579+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.580357+0000) 2020-02-05T20:26:33.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:33.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.580694+0000) 2020-02-05T20:26:33.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:33.579+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.580990+0000) 2020-02-05T20:26:33.583 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:33.581+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.581482+0000) 2020-02-05T20:26:33.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:33.581+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.581915+0000) 2020-02-05T20:26:33.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:33.581+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.582386+0000) 2020-02-05T20:26:33.584 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:33.582+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.582737+0000) 2020-02-05T20:26:33.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:33.582+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.583124+0000) 2020-02-05T20:26:33.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:33.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.582987+0000) 2020-02-05T20:26:33.585 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:33.582+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.583290+0000) 2020-02-05T20:26:33.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:33.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:33.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:33.939+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:33.940919+0000) 2020-02-05T20:26:35.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:35.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:35.796679+0000) 2020-02-05T20:26:36.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:36.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:36.656035+0000) 2020-02-05T20:26:36.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:36.898+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:36.898114+0000) 2020-02-05T20:26:38.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:38.588+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.590190+0000) 2020-02-05T20:26:38.593 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:38.590+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.592269+0000) 2020-02-05T20:26:38.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:38.591+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.592501+0000) 2020-02-05T20:26:38.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:38.591+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.592812+0000) 2020-02-05T20:26:38.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:38.592+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.593173+0000) 2020-02-05T20:26:38.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:38.592+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.593727+0000) 2020-02-05T20:26:38.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:38.592+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.593917+0000) 2020-02-05T20:26:38.596 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:38.593+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.594282+0000) 2020-02-05T20:26:38.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:38.593+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.595193+0000) 2020-02-05T20:26:38.597 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:38.594+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.595832+0000) 2020-02-05T20:26:38.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:38.594+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.595960+0000) 2020-02-05T20:26:38.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:38.594+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.595346+0000) 2020-02-05T20:26:38.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:38.594+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:38.599 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:38.595+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.596380+0000) 2020-02-05T20:26:38.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:38.597+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.598864+0000) 2020-02-05T20:26:38.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:38.637+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:38.639003+0000) 2020-02-05T20:26:39.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:39.008+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:39.009365+0000) 2020-02-05T20:26:39.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:39.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:39.294562+0000) 2020-02-05T20:26:39.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:39.904+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:39.904507+0000) 2020-02-05T20:26:40.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:40.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:40.150587+0000) 2020-02-05T20:26:40.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:40.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:40.150787+0000) 2020-02-05T20:26:40.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:40.163+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:40.164590+0000) 2020-02-05T20:26:42.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:42.255+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:42.256397+0000) 2020-02-05T20:26:43.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:43.381+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.383077+0000) 2020-02-05T20:26:43.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:43.414+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.414473+0000) 2020-02-05T20:26:43.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:43.601+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.602156+0000) 2020-02-05T20:26:43.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:43.602+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.603804+0000) 2020-02-05T20:26:43.605 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:43.603+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.604061+0000) 2020-02-05T20:26:43.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:43.603+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.604468+0000) 2020-02-05T20:26:43.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:43.603+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.604857+0000) 2020-02-05T20:26:43.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:43.604+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.605568+0000) 2020-02-05T20:26:43.607 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:43.604+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.605996+0000) 2020-02-05T20:26:43.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:43.605+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.606676+0000) 2020-02-05T20:26:43.608 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:43.605+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.606824+0000) 2020-02-05T20:26:43.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:43.606+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.607708+0000) 2020-02-05T20:26:43.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:43.606+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.608018+0000) 2020-02-05T20:26:43.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:43.606+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.608371+0000) 2020-02-05T20:26:43.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:43.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:43.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:43.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.609787+0000) 2020-02-05T20:26:43.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:43.609+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.610714+0000) 2020-02-05T20:26:43.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:43.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:43.948972+0000) 2020-02-05T20:26:45.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:45.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:45.796888+0000) 2020-02-05T20:26:46.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:46.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:46.656304+0000) 2020-02-05T20:26:46.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:46.897+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:46.898421+0000) 2020-02-05T20:26:48.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:48.614+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.615529+0000) 2020-02-05T20:26:48.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:48.614+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.615778+0000) 2020-02-05T20:26:48.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:48.615+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.616582+0000) 2020-02-05T20:26:48.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:48.615+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.617191+0000) 2020-02-05T20:26:48.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:48.616+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.617476+0000) 2020-02-05T20:26:48.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:48.616+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.618013+0000) 2020-02-05T20:26:48.620 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:48.617+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.618413+0000) 2020-02-05T20:26:48.620 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:48.617+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.618779+0000) 2020-02-05T20:26:48.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:48.617+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.619190+0000) 2020-02-05T20:26:48.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:48.618+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.619489+0000) 2020-02-05T20:26:48.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:48.618+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.619847+0000) 2020-02-05T20:26:48.622 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:48.618+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.620049+0000) 2020-02-05T20:26:48.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:48.619+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.620611+0000) 2020-02-05T20:26:48.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:48.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56770 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:48.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:48.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.626873+0000) 2020-02-05T20:26:48.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:48.637+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:48.639289+0000) 2020-02-05T20:26:49.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:49.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:49.009589+0000) 2020-02-05T20:26:49.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:49.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:49.294766+0000) 2020-02-05T20:26:49.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:49.903+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:49.904743+0000) 2020-02-05T20:26:50.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:50.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:50.150819+0000) 2020-02-05T20:26:50.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:50.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:50.151060+0000) 2020-02-05T20:26:50.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:50.164+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:50.164823+0000) 2020-02-05T20:26:51.659 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:26:51.665 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:26:52.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:52.255+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:52.256660+0000) 2020-02-05T20:26:53.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:53.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.383338+0000) 2020-02-05T20:26:53.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:53.413+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.414638+0000) 2020-02-05T20:26:53.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:53.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.626954+0000) 2020-02-05T20:26:53.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:53.627+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.628428+0000) 2020-02-05T20:26:53.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:53.627+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.628957+0000) 2020-02-05T20:26:53.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:53.628+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.629512+0000) 2020-02-05T20:26:53.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:53.628+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.629744+0000) 2020-02-05T20:26:53.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:53.629+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.630338+0000) 2020-02-05T20:26:53.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:53.629+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.630475+0000) 2020-02-05T20:26:53.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:53.629+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.630757+0000) 2020-02-05T20:26:53.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:53.629+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.631063+0000) 2020-02-05T20:26:53.633 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:53.630+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.631497+0000) 2020-02-05T20:26:53.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:53.630+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.631736+0000) 2020-02-05T20:26:53.634 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:53.630+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.631959+0000) 2020-02-05T20:26:53.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:53.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:53.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:53.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.638237+0000) 2020-02-05T20:26:53.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:53.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.638641+0000) 2020-02-05T20:26:53.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:53.957+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:53.957262+0000) 2020-02-05T20:26:55.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:55.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:55.797126+0000) 2020-02-05T20:26:56.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:56.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:56.656539+0000) 2020-02-05T20:26:56.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:56.897+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:56.898708+0000) 2020-02-05T20:26:58.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:26:58.637+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.638624+0000) 2020-02-05T20:26:58.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:26:58.638+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.639868+0000) 2020-02-05T20:26:58.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:26:58.638+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.638248+0000) 2020-02-05T20:26:58.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:26:58.640+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.640156+0000) 2020-02-05T20:26:58.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:58.640+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.640775+0000) 2020-02-05T20:26:58.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:26:58.641+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.641114+0000) 2020-02-05T20:26:58.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:58.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.641557+0000) 2020-02-05T20:26:58.645 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:26:58.640+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.642137+0000) 2020-02-05T20:26:58.645 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:26:58.641+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.642986+0000) 2020-02-05T20:26:58.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:58.643+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.643222+0000) 2020-02-05T20:26:58.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:26:58.642+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.643666+0000) 2020-02-05T20:26:58.650 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:26:58.642+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.644269+0000) 2020-02-05T20:26:58.650 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:26:58.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56848 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:26:58.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:58.646+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.646303+0000) 2020-02-05T20:26:58.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:58.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:58.647591+0000) 2020-02-05T20:26:59.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:26:59.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:59.009806+0000) 2020-02-05T20:26:59.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:26:59.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:59.294969+0000) 2020-02-05T20:26:59.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:26:59.904+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:26:59.904975+0000) 2020-02-05T20:27:00.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:00.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:00.151159+0000) 2020-02-05T20:27:00.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:00.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:00.151274+0000) 2020-02-05T20:27:00.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:00.164+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:00.165048+0000) 2020-02-05T20:27:02.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:02.256+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:02.256920+0000) 2020-02-05T20:27:03.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:03.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.383629+0000) 2020-02-05T20:27:03.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:03.413+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.414863+0000) 2020-02-05T20:27:03.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:03.648+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.649416+0000) 2020-02-05T20:27:03.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:03.649+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.650844+0000) 2020-02-05T20:27:03.652 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:03.650+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.651504+0000) 2020-02-05T20:27:03.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:03.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.651919+0000) 2020-02-05T20:27:03.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:03.651+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.652337+0000) 2020-02-05T20:27:03.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:03.651+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.652766+0000) 2020-02-05T20:27:03.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:03.652+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.653465+0000) 2020-02-05T20:27:03.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:03.652+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.654205+0000) 2020-02-05T20:27:03.656 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:03.653+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.654646+0000) 2020-02-05T20:27:03.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:03.653+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.654639+0000) 2020-02-05T20:27:03.657 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:03.653+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.654926+0000) 2020-02-05T20:27:03.657 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:03.653+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.655027+0000) 2020-02-05T20:27:03.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:03.655+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56888 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:03.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:03.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.659897+0000) 2020-02-05T20:27:03.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:03.661+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.662112+0000) 2020-02-05T20:27:03.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:03.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:03.965282+0000) 2020-02-05T20:27:05.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:05.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:05.390340+0000) 2020-02-05T20:27:05.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:05.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:05.797304+0000) 2020-02-05T20:27:06.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:06.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:06.656757+0000) 2020-02-05T20:27:06.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:06.898+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:06.898952+0000) 2020-02-05T20:27:08.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:08.661+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.662882+0000) 2020-02-05T20:27:08.665 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:08.662+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.663948+0000) 2020-02-05T20:27:08.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:08.662+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.664390+0000) 2020-02-05T20:27:08.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:08.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.664991+0000) 2020-02-05T20:27:08.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:08.664+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.666184+0000) 2020-02-05T20:27:08.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:08.665+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.666477+0000) 2020-02-05T20:27:08.668 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:08.665+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.666882+0000) 2020-02-05T20:27:08.668 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:08.665+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.667345+0000) 2020-02-05T20:27:08.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:08.666+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.667681+0000) 2020-02-05T20:27:08.669 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:08.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.668220+0000) 2020-02-05T20:27:08.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:08.666+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.668325+0000) 2020-02-05T20:27:08.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:08.667+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.668872+0000) 2020-02-05T20:27:08.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:08.667+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.669024+0000) 2020-02-05T20:27:08.671 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:08.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56926 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:08.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:08.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:08.676757+0000) 2020-02-05T20:27:09.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:09.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:09.010022+0000) 2020-02-05T20:27:09.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:09.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:09.295173+0000) 2020-02-05T20:27:09.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:09.904+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:09.905209+0000) 2020-02-05T20:27:10.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:10.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:10.151465+0000) 2020-02-05T20:27:10.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:10.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:10.151529+0000) 2020-02-05T20:27:10.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:10.164+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:10.165346+0000) 2020-02-05T20:27:12.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:12.256+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:12.257213+0000) 2020-02-05T20:27:13.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:13.383+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.383878+0000) 2020-02-05T20:27:13.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:13.414+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.415118+0000) 2020-02-05T20:27:13.676 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:13.674+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.675070+0000) 2020-02-05T20:27:13.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:13.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.676394+0000) 2020-02-05T20:27:13.678 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:13.675+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.676950+0000) 2020-02-05T20:27:13.679 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:13.676+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.677946+0000) 2020-02-05T20:27:13.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:13.677+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.679019+0000) 2020-02-05T20:27:13.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:13.678+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.679439+0000) 2020-02-05T20:27:13.681 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:13.678+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.679659+0000) 2020-02-05T20:27:13.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:13.678+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.680095+0000) 2020-02-05T20:27:13.682 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:13.678+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.679852+0000) 2020-02-05T20:27:13.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:13.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.680977+0000) 2020-02-05T20:27:13.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:13.679+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.681076+0000) 2020-02-05T20:27:13.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:13.680+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.681596+0000) 2020-02-05T20:27:13.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:13.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 56967 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:13.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:13.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.685035+0000) 2020-02-05T20:27:13.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:13.684+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.686016+0000) 2020-02-05T20:27:13.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:13.972+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:13.973358+0000) 2020-02-05T20:27:15.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:15.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:15.390616+0000) 2020-02-05T20:27:15.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:15.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:15.797556+0000) 2020-02-05T20:27:16.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:16.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:16.657011+0000) 2020-02-05T20:27:16.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:16.898+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:16.899216+0000) 2020-02-05T20:27:18.696 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:18.689+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.690583+0000) 2020-02-05T20:27:18.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:18.689+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.690630+0000) 2020-02-05T20:27:18.696 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:18.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.692274+0000) 2020-02-05T20:27:18.697 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:18.690+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.691923+0000) 2020-02-05T20:27:18.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:18.687+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.688924+0000) 2020-02-05T20:27:18.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:18.690+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.691286+0000) 2020-02-05T20:27:18.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:18.691+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.692529+0000) 2020-02-05T20:27:18.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:18.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.694834+0000) 2020-02-05T20:27:18.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:18.691+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.692894+0000) 2020-02-05T20:27:18.698 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:18.692+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.693361+0000) 2020-02-05T20:27:18.698 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:18.692+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.693890+0000) 2020-02-05T20:27:18.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:18.692+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.694120+0000) 2020-02-05T20:27:18.700 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:18.692+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.694236+0000) 2020-02-05T20:27:18.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:18.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57005 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:18.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:18.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:18.698547+0000) 2020-02-05T20:27:19.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:19.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:19.010276+0000) 2020-02-05T20:27:19.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:19.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:19.295328+0000) 2020-02-05T20:27:19.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:19.904+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:19.905442+0000) 2020-02-05T20:27:20.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:20.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:20.151754+0000) 2020-02-05T20:27:20.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:20.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:20.151751+0000) 2020-02-05T20:27:20.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:20.164+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:20.165592+0000) 2020-02-05T20:27:21.873 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:27:21.881 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:27:22.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:22.256+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:22.257483+0000) 2020-02-05T20:27:23.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:23.383+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.384142+0000) 2020-02-05T20:27:23.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:23.414+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.415384+0000) 2020-02-05T20:27:23.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:23.700+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.701547+0000) 2020-02-05T20:27:23.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:23.700+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.702315+0000) 2020-02-05T20:27:23.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:23.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.702871+0000) 2020-02-05T20:27:23.705 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:23.702+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.703509+0000) 2020-02-05T20:27:23.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:23.702+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.703947+0000) 2020-02-05T20:27:23.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:23.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.704451+0000) 2020-02-05T20:27:23.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:23.703+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.704596+0000) 2020-02-05T20:27:23.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:23.703+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.704996+0000) 2020-02-05T20:27:23.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:23.705+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.706737+0000) 2020-02-05T20:27:23.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:23.704+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.706183+0000) 2020-02-05T20:27:23.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:23.705+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.706481+0000) 2020-02-05T20:27:23.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:23.705+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.707260+0000) 2020-02-05T20:27:23.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:23.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.707253+0000) 2020-02-05T20:27:23.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:23.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:23.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:23.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.711647+0000) 2020-02-05T20:27:23.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:23.981+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:23.981854+0000) 2020-02-05T20:27:25.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:25.390+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:25.390880+0000) 2020-02-05T20:27:25.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:25.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:25.797789+0000) 2020-02-05T20:27:26.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:26.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:26.657280+0000) 2020-02-05T20:27:26.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:26.898+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:26.899504+0000) 2020-02-05T20:27:28.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:28.713+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.714365+0000) 2020-02-05T20:27:28.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:28.713+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.714793+0000) 2020-02-05T20:27:28.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:28.714+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.715378+0000) 2020-02-05T20:27:28.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:28.712+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.713930+0000) 2020-02-05T20:27:28.719 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:28.716+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.716897+0000) 2020-02-05T20:27:28.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:28.715+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.716920+0000) 2020-02-05T20:27:28.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:28.717+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.717725+0000) 2020-02-05T20:27:28.721 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:28.718+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.718731+0000) 2020-02-05T20:27:28.721 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:28.717+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.718106+0000) 2020-02-05T20:27:28.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:28.717+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.718642+0000) 2020-02-05T20:27:28.731 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:28.718+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.719235+0000) 2020-02-05T20:27:28.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:28.718+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.719986+0000) 2020-02-05T20:27:28.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:28.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:28.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:28.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.721675+0000) 2020-02-05T20:27:28.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:28.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:28.721991+0000) 2020-02-05T20:27:29.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:29.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:29.010538+0000) 2020-02-05T20:27:29.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:29.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:29.295537+0000) 2020-02-05T20:27:29.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:29.905+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:29.905736+0000) 2020-02-05T20:27:30.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:30.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:30.151972+0000) 2020-02-05T20:27:30.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:30.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:30.152063+0000) 2020-02-05T20:27:30.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:30.165+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:30.165822+0000) 2020-02-05T20:27:32.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:32.257+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:32.257708+0000) 2020-02-05T20:27:33.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:33.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.384476+0000) 2020-02-05T20:27:33.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:33.415+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.415627+0000) 2020-02-05T20:27:33.726 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:33.724+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.725415+0000) 2020-02-05T20:27:33.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:33.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.726354+0000) 2020-02-05T20:27:33.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:33.726+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.727233+0000) 2020-02-05T20:27:33.730 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:33.728+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.728693+0000) 2020-02-05T20:27:33.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:33.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.729134+0000) 2020-02-05T20:27:33.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:33.728+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.729409+0000) 2020-02-05T20:27:33.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:33.730+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.730509+0000) 2020-02-05T20:27:33.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:33.729+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.730670+0000) 2020-02-05T20:27:33.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:33.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.731006+0000) 2020-02-05T20:27:33.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:33.731+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.731653+0000) 2020-02-05T20:27:33.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:33.732+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.733141+0000) 2020-02-05T20:27:33.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:33.732+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.733130+0000) 2020-02-05T20:27:33.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:33.732+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.733214+0000) 2020-02-05T20:27:33.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:33.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:33.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:33.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.738021+0000) 2020-02-05T20:27:33.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:33.989+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:33.990143+0000) 2020-02-05T20:27:35.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:35.390+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:35.391133+0000) 2020-02-05T20:27:35.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:35.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:35.798019+0000) 2020-02-05T20:27:36.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:36.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:36.657488+0000) 2020-02-05T20:27:36.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:36.899+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:36.899742+0000) 2020-02-05T20:27:38.742 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:38.740+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.740916+0000) 2020-02-05T20:27:38.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:38.740+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.741209+0000) 2020-02-05T20:27:38.743 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:38.740+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.741742+0000) 2020-02-05T20:27:38.743 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:38.741+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.742369+0000) 2020-02-05T20:27:38.744 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:38.741+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.743018+0000) 2020-02-05T20:27:38.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:38.742+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.743367+0000) 2020-02-05T20:27:38.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:38.743+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.743423+0000) 2020-02-05T20:27:38.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:38.742+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.743943+0000) 2020-02-05T20:27:38.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:38.743+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.744410+0000) 2020-02-05T20:27:38.746 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:38.743+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.743968+0000) 2020-02-05T20:27:38.747 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:38.743+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.744321+0000) 2020-02-05T20:27:38.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:38.745+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.745470+0000) 2020-02-05T20:27:38.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:38.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.745758+0000) 2020-02-05T20:27:38.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:38.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57165 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:38.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:38.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:38.747050+0000) 2020-02-05T20:27:39.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:39.009+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:39.010813+0000) 2020-02-05T20:27:39.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:39.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:39.295749+0000) 2020-02-05T20:27:39.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:39.905+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:39.905964+0000) 2020-02-05T20:27:40.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:40.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:40.152237+0000) 2020-02-05T20:27:40.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:40.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:40.152299+0000) 2020-02-05T20:27:40.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:40.165+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:40.166053+0000) 2020-02-05T20:27:42.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:42.257+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:42.257993+0000) 2020-02-05T20:27:43.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:43.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.384758+0000) 2020-02-05T20:27:43.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:43.415+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.415942+0000) 2020-02-05T20:27:43.754 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:43.750+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.751212+0000) 2020-02-05T20:27:43.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:43.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.753527+0000) 2020-02-05T20:27:43.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:43.752+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.754036+0000) 2020-02-05T20:27:43.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:43.753+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.754896+0000) 2020-02-05T20:27:43.757 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:43.752+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.753141+0000) 2020-02-05T20:27:43.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:43.753+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.754392+0000) 2020-02-05T20:27:43.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:43.754+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.755698+0000) 2020-02-05T20:27:43.761 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:43.756+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.756666+0000) 2020-02-05T20:27:43.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:43.756+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.757169+0000) 2020-02-05T20:27:43.761 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:43.756+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.757052+0000) 2020-02-05T20:27:43.761 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:43.757+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.757947+0000) 2020-02-05T20:27:43.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:43.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57207 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:43.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:43.756+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.757823+0000) 2020-02-05T20:27:43.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:43.757+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.758512+0000) 2020-02-05T20:27:43.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:43.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.761575+0000) 2020-02-05T20:27:43.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:43.997+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:43.997320+0000) 2020-02-05T20:27:45.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:45.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:45.391394+0000) 2020-02-05T20:27:45.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:45.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:45.798271+0000) 2020-02-05T20:27:46.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:46.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:46.657719+0000) 2020-02-05T20:27:46.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:46.899+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:46.899996+0000) 2020-02-05T20:27:48.765 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:48.762+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.762825+0000) 2020-02-05T20:27:48.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:48.763+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.764095+0000) 2020-02-05T20:27:48.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:48.763+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.764383+0000) 2020-02-05T20:27:48.767 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:48.764+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.765252+0000) 2020-02-05T20:27:48.767 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:48.764+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.765958+0000) 2020-02-05T20:27:48.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:48.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.766053+0000) 2020-02-05T20:27:48.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:48.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.766533+0000) 2020-02-05T20:27:48.769 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:48.767+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.767644+0000) 2020-02-05T20:27:48.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:48.767+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.768345+0000) 2020-02-05T20:27:48.770 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:48.767+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.768572+0000) 2020-02-05T20:27:48.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:48.768+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.769285+0000) 2020-02-05T20:27:48.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:48.769+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.769810+0000) 2020-02-05T20:27:48.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:48.769+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.770209+0000) 2020-02-05T20:27:48.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:48.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57246 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:48.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:48.771+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:48.772184+0000) 2020-02-05T20:27:49.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:49.010+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:49.011031+0000) 2020-02-05T20:27:49.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:49.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:49.295962+0000) 2020-02-05T20:27:49.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:49.906+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:49.906192+0000) 2020-02-05T20:27:50.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:50.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:50.152546+0000) 2020-02-05T20:27:50.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:50.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:50.152484+0000) 2020-02-05T20:27:50.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:50.166+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:50.166301+0000) 2020-02-05T20:27:52.094 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:27:52.100 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:27:52.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:52.258+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:52.258257+0000) 2020-02-05T20:27:53.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:53.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.385009+0000) 2020-02-05T20:27:53.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:53.416+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.416178+0000) 2020-02-05T20:27:53.777 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:53.776+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.776266+0000) 2020-02-05T20:27:53.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:53.776+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.776948+0000) 2020-02-05T20:27:53.780 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:53.778+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.778465+0000) 2020-02-05T20:27:53.780 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:53.778+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.778738+0000) 2020-02-05T20:27:53.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:53.778+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.778956+0000) 2020-02-05T20:27:53.781 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:53.778+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.779351+0000) 2020-02-05T20:27:53.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:53.779+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.779637+0000) 2020-02-05T20:27:53.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:53.779+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.780013+0000) 2020-02-05T20:27:53.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:53.780+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.780471+0000) 2020-02-05T20:27:53.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:53.780+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.780865+0000) 2020-02-05T20:27:53.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:53.781+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.781265+0000) 2020-02-05T20:27:53.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:53.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.782401+0000) 2020-02-05T20:27:53.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:53.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57289 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:53.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:53.789+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.789495+0000) 2020-02-05T20:27:53.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:53.789+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:53.789828+0000) 2020-02-05T20:27:54.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:54.005+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:54.006049+0000) 2020-02-05T20:27:55.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:55.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:55.391670+0000) 2020-02-05T20:27:55.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:55.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:55.798500+0000) 2020-02-05T20:27:56.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:56.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:56.657979+0000) 2020-02-05T20:27:56.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:56.900+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:56.900307+0000) 2020-02-05T20:27:58.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:58.787+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.788334+0000) 2020-02-05T20:27:58.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:27:58.789+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.789484+0000) 2020-02-05T20:27:58.792 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:27:58.789+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.790524+0000) 2020-02-05T20:27:58.793 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:27:58.789+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.790321+0000) 2020-02-05T20:27:58.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:27:58.789+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.790952+0000) 2020-02-05T20:27:58.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:58.790+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.791507+0000) 2020-02-05T20:27:58.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:27:58.791+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.792210+0000) 2020-02-05T20:27:58.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:58.791+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.792660+0000) 2020-02-05T20:27:58.795 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:27:58.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.792423+0000) 2020-02-05T20:27:58.795 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:27:58.792+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.792994+0000) 2020-02-05T20:27:58.795 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:27:58.793+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.793392+0000) 2020-02-05T20:27:58.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:58.793+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.794295+0000) 2020-02-05T20:27:58.796 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:27:58.794+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.794863+0000) 2020-02-05T20:27:58.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:27:58.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:27:58.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:58.798+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:58.800008+0000) 2020-02-05T20:27:59.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:27:59.010+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:59.011279+0000) 2020-02-05T20:27:59.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:27:59.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:59.296206+0000) 2020-02-05T20:27:59.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:27:59.905+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:27:59.906427+0000) 2020-02-05T20:28:00.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:00.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:00.152743+0000) 2020-02-05T20:28:00.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:00.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:00.152714+0000) 2020-02-05T20:28:00.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:00.165+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:00.166534+0000) 2020-02-05T20:28:02.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:02.258+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:02.258489+0000) 2020-02-05T20:28:03.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:03.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.385260+0000) 2020-02-05T20:28:03.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:03.415+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.416415+0000) 2020-02-05T20:28:03.804 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:03.801+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.801955+0000) 2020-02-05T20:28:03.804 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:03.802+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.802478+0000) 2020-02-05T20:28:03.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:03.802+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.803502+0000) 2020-02-05T20:28:03.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:03.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.803812+0000) 2020-02-05T20:28:03.806 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:03.802+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.803176+0000) 2020-02-05T20:28:03.807 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:03.803+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.804139+0000) 2020-02-05T20:28:03.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:03.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.804508+0000) 2020-02-05T20:28:03.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:03.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.805040+0000) 2020-02-05T20:28:03.808 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:03.804+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.804675+0000) 2020-02-05T20:28:03.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:03.804+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.805811+0000) 2020-02-05T20:28:03.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:03.805+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.806117+0000) 2020-02-05T20:28:03.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:03.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.806559+0000) 2020-02-05T20:28:03.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:03.806+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.806479+0000) 2020-02-05T20:28:03.810 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:03.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:03.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:03.812+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:03.813172+0000) 2020-02-05T20:28:04.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:04.013+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:04.014433+0000) 2020-02-05T20:28:05.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:05.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:05.391932+0000) 2020-02-05T20:28:05.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:05.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:05.798754+0000) 2020-02-05T20:28:06.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:06.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:06.658282+0000) 2020-02-05T20:28:06.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:06.900+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:06.900562+0000) 2020-02-05T20:28:08.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:08.811+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.812705+0000) 2020-02-05T20:28:08.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:08.813+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.814539+0000) 2020-02-05T20:28:08.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:08.813+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.815162+0000) 2020-02-05T20:28:08.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:08.814+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.815629+0000) 2020-02-05T20:28:08.817 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:08.814+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.816057+0000) 2020-02-05T20:28:08.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:08.815+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.816511+0000) 2020-02-05T20:28:08.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:08.816+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.817294+0000) 2020-02-05T20:28:08.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:08.816+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.817921+0000) 2020-02-05T20:28:08.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:08.816+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.818034+0000) 2020-02-05T20:28:08.821 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:08.817+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.818851+0000) 2020-02-05T20:28:08.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:08.817+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.819096+0000) 2020-02-05T20:28:08.821 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:08.818+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.819448+0000) 2020-02-05T20:28:08.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:08.817+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.818395+0000) 2020-02-05T20:28:08.822 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:08.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57408 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:08.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:08.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:08.825524+0000) 2020-02-05T20:28:09.013 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:09.010+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:09.011499+0000) 2020-02-05T20:28:09.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:09.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:09.296363+0000) 2020-02-05T20:28:09.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:09.905+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:09.906668+0000) 2020-02-05T20:28:10.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:10.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:10.152983+0000) 2020-02-05T20:28:10.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:10.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:10.152975+0000) 2020-02-05T20:28:10.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:10.165+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:10.166766+0000) 2020-02-05T20:28:12.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:12.257+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:12.258723+0000) 2020-02-05T20:28:13.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:13.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.385509+0000) 2020-02-05T20:28:13.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:13.415+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.416652+0000) 2020-02-05T20:28:13.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:13.824+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.825569+0000) 2020-02-05T20:28:13.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:13.825+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.826238+0000) 2020-02-05T20:28:13.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:13.825+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.826753+0000) 2020-02-05T20:28:13.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:13.826+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.827150+0000) 2020-02-05T20:28:13.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:13.826+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.827883+0000) 2020-02-05T20:28:13.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:13.827+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.828170+0000) 2020-02-05T20:28:13.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:13.827+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.828493+0000) 2020-02-05T20:28:13.830 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:13.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.829342+0000) 2020-02-05T20:28:13.831 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:13.828+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.830216+0000) 2020-02-05T20:28:13.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:13.829+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.830650+0000) 2020-02-05T20:28:13.832 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:13.829+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.831104+0000) 2020-02-05T20:28:13.833 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:13.829+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.831158+0000) 2020-02-05T20:28:13.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:13.830+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.831475+0000) 2020-02-05T20:28:13.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:13.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57451 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:13.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:13.833+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:13.834850+0000) 2020-02-05T20:28:14.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:14.020+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:14.021563+0000) 2020-02-05T20:28:15.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:15.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:15.392171+0000) 2020-02-05T20:28:15.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:15.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:15.798994+0000) 2020-02-05T20:28:16.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:16.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:16.658519+0000) 2020-02-05T20:28:16.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:16.899+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:16.900798+0000) 2020-02-05T20:28:18.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:18.836+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.837722+0000) 2020-02-05T20:28:18.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:18.837+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.838841+0000) 2020-02-05T20:28:18.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:18.838+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.839222+0000) 2020-02-05T20:28:18.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:18.837+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.839325+0000) 2020-02-05T20:28:18.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:18.838+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.839935+0000) 2020-02-05T20:28:18.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:18.838+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.840032+0000) 2020-02-05T20:28:18.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:18.839+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.840366+0000) 2020-02-05T20:28:18.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:18.839+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.840512+0000) 2020-02-05T20:28:18.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:18.839+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.840578+0000) 2020-02-05T20:28:18.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:18.839+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.841096+0000) 2020-02-05T20:28:18.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:18.840+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.841913+0000) 2020-02-05T20:28:18.844 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:18.840+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.842097+0000) 2020-02-05T20:28:18.844 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:18.841+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.843010+0000) 2020-02-05T20:28:18.847 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:18.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:18.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:18.847+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:18.848240+0000) 2020-02-05T20:28:19.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:19.011+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:19.011763+0000) 2020-02-05T20:28:19.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:19.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:19.296581+0000) 2020-02-05T20:28:19.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:19.906+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:19.906941+0000) 2020-02-05T20:28:20.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:20.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:20.153249+0000) 2020-02-05T20:28:20.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:20.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:20.153250+0000) 2020-02-05T20:28:20.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:20.166+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:20.166998+0000) 2020-02-05T20:28:22.227 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:28:22.234 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:28:22.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:22.257+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:22.258933+0000) 2020-02-05T20:28:23.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:23.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.385740+0000) 2020-02-05T20:28:23.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:23.416+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.416880+0000) 2020-02-05T20:28:23.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:23.849+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.850884+0000) 2020-02-05T20:28:23.853 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:23.849+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.851193+0000) 2020-02-05T20:28:23.853 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:23.850+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.851616+0000) 2020-02-05T20:28:23.854 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:23.850+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.852054+0000) 2020-02-05T20:28:23.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:23.851+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.852523+0000) 2020-02-05T20:28:23.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:23.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.853207+0000) 2020-02-05T20:28:23.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:23.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.853776+0000) 2020-02-05T20:28:23.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:23.852+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.854228+0000) 2020-02-05T20:28:23.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:23.853+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.854624+0000) 2020-02-05T20:28:23.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:23.854+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.855456+0000) 2020-02-05T20:28:23.858 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:23.854+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.855970+0000) 2020-02-05T20:28:23.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:23.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.856198+0000) 2020-02-05T20:28:23.858 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:23.854+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.856308+0000) 2020-02-05T20:28:23.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:23.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:23.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:23.857+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:23.858838+0000) 2020-02-05T20:28:24.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:24.033+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:24.033693+0000) 2020-02-05T20:28:25.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:25.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:25.392462+0000) 2020-02-05T20:28:25.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:25.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:25.799280+0000) 2020-02-05T20:28:26.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:26.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:26.658785+0000) 2020-02-05T20:28:26.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:26.900+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:26.901052+0000) 2020-02-05T20:28:28.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:28.861+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.862890+0000) 2020-02-05T20:28:28.865 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:28.861+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.862849+0000) 2020-02-05T20:28:28.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:28.862+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.863584+0000) 2020-02-05T20:28:28.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:28.862+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.863843+0000) 2020-02-05T20:28:28.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:28.863+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.864582+0000) 2020-02-05T20:28:28.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:28.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.864465+0000) 2020-02-05T20:28:28.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:28.864+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.865600+0000) 2020-02-05T20:28:28.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:28.863+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.864381+0000) 2020-02-05T20:28:28.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:28.863+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.864995+0000) 2020-02-05T20:28:28.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:28.865+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.866518+0000) 2020-02-05T20:28:28.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:28.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.866898+0000) 2020-02-05T20:28:28.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:28.866+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.867698+0000) 2020-02-05T20:28:28.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:28.866+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.867840+0000) 2020-02-05T20:28:28.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:28.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:28.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:28.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:28.871449+0000) 2020-02-05T20:28:29.013 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:29.011+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:29.012008+0000) 2020-02-05T20:28:29.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:29.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:29.296814+0000) 2020-02-05T20:28:29.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:29.906+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:29.907213+0000) 2020-02-05T20:28:30.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:30.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:30.153511+0000) 2020-02-05T20:28:30.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:30.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:30.153511+0000) 2020-02-05T20:28:30.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:30.166+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:30.167262+0000) 2020-02-05T20:28:32.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:32.257+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:32.259164+0000) 2020-02-05T20:28:33.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:33.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.385977+0000) 2020-02-05T20:28:33.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:33.416+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.417141+0000) 2020-02-05T20:28:33.876 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:33.873+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.874059+0000) 2020-02-05T20:28:33.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:33.873+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.874554+0000) 2020-02-05T20:28:33.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:33.874+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.875385+0000) 2020-02-05T20:28:33.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:33.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.875916+0000) 2020-02-05T20:28:33.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:33.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.876154+0000) 2020-02-05T20:28:33.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:33.875+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.876628+0000) 2020-02-05T20:28:33.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:33.874+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.876351+0000) 2020-02-05T20:28:33.880 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:33.875+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.877298+0000) 2020-02-05T20:28:33.880 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:33.876+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.877606+0000) 2020-02-05T20:28:33.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:33.876+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.877486+0000) 2020-02-05T20:28:33.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:33.878+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.879116+0000) 2020-02-05T20:28:33.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:33.877+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.878946+0000) 2020-02-05T20:28:33.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:33.877+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.878686+0000) 2020-02-05T20:28:33.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:33.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:33.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:33.882+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:33.883951+0000) 2020-02-05T20:28:34.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:34.042+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:34.042856+0000) 2020-02-05T20:28:35.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:35.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:35.392726+0000) 2020-02-05T20:28:35.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:35.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:35.799513+0000) 2020-02-05T20:28:36.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:36.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:36.659022+0000) 2020-02-05T20:28:36.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:36.899+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:36.901341+0000) 2020-02-05T20:28:38.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:38.885+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.886558+0000) 2020-02-05T20:28:38.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:38.885+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.887014+0000) 2020-02-05T20:28:38.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:38.885+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.886649+0000) 2020-02-05T20:28:38.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:38.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.887488+0000) 2020-02-05T20:28:38.890 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:38.886+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.888262+0000) 2020-02-05T20:28:38.891 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:38.887+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.888600+0000) 2020-02-05T20:28:38.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:38.887+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.888785+0000) 2020-02-05T20:28:38.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:38.888+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.889157+0000) 2020-02-05T20:28:38.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:38.888+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.889332+0000) 2020-02-05T20:28:38.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:38.888+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.889701+0000) 2020-02-05T20:28:38.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:38.888+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.889928+0000) 2020-02-05T20:28:38.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:38.889+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.890613+0000) 2020-02-05T20:28:38.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:38.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.890491+0000) 2020-02-05T20:28:38.894 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:38.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:38.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:38.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:38.895924+0000) 2020-02-05T20:28:39.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:39.012+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:39.012361+0000) 2020-02-05T20:28:39.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:39.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:39.297023+0000) 2020-02-05T20:28:39.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:39.907+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:39.907445+0000) 2020-02-05T20:28:40.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:40.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:40.153746+0000) 2020-02-05T20:28:40.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:40.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:40.153749+0000) 2020-02-05T20:28:40.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:40.167+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:40.167490+0000) 2020-02-05T20:28:42.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:42.258+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:42.259488+0000) 2020-02-05T20:28:43.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:43.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.386217+0000) 2020-02-05T20:28:43.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:43.417+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.417393+0000) 2020-02-05T20:28:43.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:43.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.897876+0000) 2020-02-05T20:28:43.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:43.897+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.898363+0000) 2020-02-05T20:28:43.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:43.897+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.899074+0000) 2020-02-05T20:28:43.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:43.898+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.899689+0000) 2020-02-05T20:28:43.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:43.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.899992+0000) 2020-02-05T20:28:43.902 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:43.900+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.901267+0000) 2020-02-05T20:28:43.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:43.900+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.901536+0000) 2020-02-05T20:28:43.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:43.900+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.901890+0000) 2020-02-05T20:28:43.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:43.901+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.902511+0000) 2020-02-05T20:28:43.907 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:43.900+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.902245+0000) 2020-02-05T20:28:43.907 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:43.901+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.902863+0000) 2020-02-05T20:28:43.907 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:43.901+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.903196+0000) 2020-02-05T20:28:43.907 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:43.902+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.904202+0000) 2020-02-05T20:28:43.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:43.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:43.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:43.909+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:43.910591+0000) 2020-02-05T20:28:44.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:44.050+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:44.050538+0000) 2020-02-05T20:28:45.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:45.392+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:45.392993+0000) 2020-02-05T20:28:45.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:45.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:45.799766+0000) 2020-02-05T20:28:46.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:46.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:46.659289+0000) 2020-02-05T20:28:46.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:46.901+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:46.901640+0000) 2020-02-05T20:28:48.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:48.909+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.910899+0000) 2020-02-05T20:28:48.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:48.911+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.911613+0000) 2020-02-05T20:28:48.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:48.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.912054+0000) 2020-02-05T20:28:48.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:48.912+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.912226+0000) 2020-02-05T20:28:48.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:48.912+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.912329+0000) 2020-02-05T20:28:48.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:48.911+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.913022+0000) 2020-02-05T20:28:48.915 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:48.912+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.913545+0000) 2020-02-05T20:28:48.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:48.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.913661+0000) 2020-02-05T20:28:48.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:48.913+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.913655+0000) 2020-02-05T20:28:48.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:48.912+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.914377+0000) 2020-02-05T20:28:48.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:48.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.915500+0000) 2020-02-05T20:28:48.917 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:48.913+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.915099+0000) 2020-02-05T20:28:48.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:48.914+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.916368+0000) 2020-02-05T20:28:48.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:48.917+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:48.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:48.922+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:48.922732+0000) 2020-02-05T20:28:49.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:49.012+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:49.012540+0000) 2020-02-05T20:28:49.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:49.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:49.297271+0000) 2020-02-05T20:28:49.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:49.907+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:49.907706+0000) 2020-02-05T20:28:50.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:50.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:50.153997+0000) 2020-02-05T20:28:50.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:50.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:50.153997+0000) 2020-02-05T20:28:50.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:50.167+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:50.167725+0000) 2020-02-05T20:28:52.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:52.258+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:52.259724+0000) 2020-02-05T20:28:52.455 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:28:52.461 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:28:53.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:53.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.386470+0000) 2020-02-05T20:28:53.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:53.416+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.417628+0000) 2020-02-05T20:28:53.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:53.921+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.922912+0000) 2020-02-05T20:28:53.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:53.922+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.923778+0000) 2020-02-05T20:28:53.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:53.923+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.924565+0000) 2020-02-05T20:28:53.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:53.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.925153+0000) 2020-02-05T20:28:53.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:53.924+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.925451+0000) 2020-02-05T20:28:53.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:53.924+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.925723+0000) 2020-02-05T20:28:53.928 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:53.924+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.926057+0000) 2020-02-05T20:28:53.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:53.925+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.926422+0000) 2020-02-05T20:28:53.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:53.926+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.928329+0000) 2020-02-05T20:28:53.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:53.927+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.928535+0000) 2020-02-05T20:28:53.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:53.928+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.929425+0000) 2020-02-05T20:28:53.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:53.928+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.929673+0000) 2020-02-05T20:28:53.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:53.929+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.930793+0000) 2020-02-05T20:28:53.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:53.930+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:53.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:53.935+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:53.937012+0000) 2020-02-05T20:28:54.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:54.058+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:54.059307+0000) 2020-02-05T20:28:55.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:55.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:55.393296+0000) 2020-02-05T20:28:55.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:55.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:55.800026+0000) 2020-02-05T20:28:56.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:56.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:56.659578+0000) 2020-02-05T20:28:56.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:56.900+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:56.901869+0000) 2020-02-05T20:28:58.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:28:58.935+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.936591+0000) 2020-02-05T20:28:58.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:58.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.937488+0000) 2020-02-05T20:28:58.940 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:28:58.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.938662+0000) 2020-02-05T20:28:58.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:28:58.938+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.939315+0000) 2020-02-05T20:28:58.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:58.938+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.939986+0000) 2020-02-05T20:28:58.941 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:28:58.939+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.940325+0000) 2020-02-05T20:28:58.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:28:58.940+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.940832+0000) 2020-02-05T20:28:58.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:58.940+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.941151+0000) 2020-02-05T20:28:58.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:28:58.941+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.941404+0000) 2020-02-05T20:28:58.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:28:58.941+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.942151+0000) 2020-02-05T20:28:58.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:28:58.942+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.942676+0000) 2020-02-05T20:28:58.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:58.942+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.943614+0000) 2020-02-05T20:28:58.945 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:28:58.943+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.943910+0000) 2020-02-05T20:28:58.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:28:58.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:28:58.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:58.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:58.949262+0000) 2020-02-05T20:28:59.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:28:59.011+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:59.012764+0000) 2020-02-05T20:28:59.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:28:59.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:59.297512+0000) 2020-02-05T20:28:59.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:28:59.907+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:28:59.907934+0000) 2020-02-05T20:29:00.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:00.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:00.154259+0000) 2020-02-05T20:29:00.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:00.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:00.154235+0000) 2020-02-05T20:29:00.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:00.167+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:00.168026+0000) 2020-02-05T20:29:02.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:02.259+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:02.259983+0000) 2020-02-05T20:29:03.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:03.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.386733+0000) 2020-02-05T20:29:03.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:03.417+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.417871+0000) 2020-02-05T20:29:03.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:03.947+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.948608+0000) 2020-02-05T20:29:03.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:03.948+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.949995+0000) 2020-02-05T20:29:03.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:03.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.950403+0000) 2020-02-05T20:29:03.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:03.950+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.951330+0000) 2020-02-05T20:29:03.953 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:03.950+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.951689+0000) 2020-02-05T20:29:03.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:03.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.953600+0000) 2020-02-05T20:29:03.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:03.953+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.953497+0000) 2020-02-05T20:29:03.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:03.953+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.954360+0000) 2020-02-05T20:29:03.956 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:03.954+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.954813+0000) 2020-02-05T20:29:03.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:03.954+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.954741+0000) 2020-02-05T20:29:03.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:03.954+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.955190+0000) 2020-02-05T20:29:03.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:03.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.956109+0000) 2020-02-05T20:29:03.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:03.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.956528+0000) 2020-02-05T20:29:03.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:03.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:03.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:03.959+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:03.961023+0000) 2020-02-05T20:29:04.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:04.066+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:04.067422+0000) 2020-02-05T20:29:05.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:05.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:05.393541+0000) 2020-02-05T20:29:05.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:05.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:05.800331+0000) 2020-02-05T20:29:06.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:06.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:06.659837+0000) 2020-02-05T20:29:06.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:06.901+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:06.902102+0000) 2020-02-05T20:29:08.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:08.961+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.962334+0000) 2020-02-05T20:29:08.965 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:08.962+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.963410+0000) 2020-02-05T20:29:08.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:08.963+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.964102+0000) 2020-02-05T20:29:08.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:08.964+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.965213+0000) 2020-02-05T20:29:08.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:08.964+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.964953+0000) 2020-02-05T20:29:08.967 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:08.964+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.965593+0000) 2020-02-05T20:29:08.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:08.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.966042+0000) 2020-02-05T20:29:08.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:08.965+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.966451+0000) 2020-02-05T20:29:08.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:08.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.967214+0000) 2020-02-05T20:29:08.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:08.967+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.967740+0000) 2020-02-05T20:29:08.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:08.967+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.968054+0000) 2020-02-05T20:29:08.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:08.968+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.968554+0000) 2020-02-05T20:29:08.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:08.968+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.968563+0000) 2020-02-05T20:29:08.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:08.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:08.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:08.972+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:08.973297+0000) 2020-02-05T20:29:09.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:09.011+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:09.012974+0000) 2020-02-05T20:29:09.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:09.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:09.297722+0000) 2020-02-05T20:29:09.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:09.907+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:09.908141+0000) 2020-02-05T20:29:10.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:10.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:10.154502+0000) 2020-02-05T20:29:10.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:10.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:10.154503+0000) 2020-02-05T20:29:10.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:10.167+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:10.168279+0000) 2020-02-05T20:29:12.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:12.259+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:12.260239+0000) 2020-02-05T20:29:13.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:13.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.386985+0000) 2020-02-05T20:29:13.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:13.417+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.418172+0000) 2020-02-05T20:29:13.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:13.975+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.976427+0000) 2020-02-05T20:29:13.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:13.976+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.977130+0000) 2020-02-05T20:29:13.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:13.977+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.978174+0000) 2020-02-05T20:29:13.980 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:13.977+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.978362+0000) 2020-02-05T20:29:13.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:13.977+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.978786+0000) 2020-02-05T20:29:13.981 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:13.978+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.979278+0000) 2020-02-05T20:29:13.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:13.978+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.979452+0000) 2020-02-05T20:29:13.982 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:13.979+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.980009+0000) 2020-02-05T20:29:13.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:13.980+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.980556+0000) 2020-02-05T20:29:13.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:13.980+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.980790+0000) 2020-02-05T20:29:13.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:13.980+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.981132+0000) 2020-02-05T20:29:13.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:13.981+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.982018+0000) 2020-02-05T20:29:13.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:13.980+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.981935+0000) 2020-02-05T20:29:13.984 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:13.982+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:13.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:13.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:13.987939+0000) 2020-02-05T20:29:14.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:14.074+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:14.075204+0000) 2020-02-05T20:29:15.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:15.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:15.393791+0000) 2020-02-05T20:29:15.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:15.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:15.800552+0000) 2020-02-05T20:29:16.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:16.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:16.660070+0000) 2020-02-05T20:29:16.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:16.901+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:16.902348+0000) 2020-02-05T20:29:18.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:18.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.988817+0000) 2020-02-05T20:29:18.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:18.988+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.989247+0000) 2020-02-05T20:29:18.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:18.988+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.989445+0000) 2020-02-05T20:29:18.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:18.988+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.989615+0000) 2020-02-05T20:29:18.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:18.989+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.990318+0000) 2020-02-05T20:29:18.992 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:18.989+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.990350+0000) 2020-02-05T20:29:18.993 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:18.990+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.991123+0000) 2020-02-05T20:29:18.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:18.990+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.991136+0000) 2020-02-05T20:29:18.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:18.990+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.991558+0000) 2020-02-05T20:29:18.994 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:18.991+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.992124+0000) 2020-02-05T20:29:18.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:18.991+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.992668+0000) 2020-02-05T20:29:18.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:18.992+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.993185+0000) 2020-02-05T20:29:18.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:18.993+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.993930+0000) 2020-02-05T20:29:18.997 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:18.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 57977 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:18.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:18.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:18.998279+0000) 2020-02-05T20:29:19.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:19.012+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:19.013195+0000) 2020-02-05T20:29:19.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:19.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:19.297930+0000) 2020-02-05T20:29:19.910 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:19.907+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:19.908352+0000) 2020-02-05T20:29:20.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:20.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:20.154759+0000) 2020-02-05T20:29:20.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:20.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:20.154782+0000) 2020-02-05T20:29:20.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:20.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:20.168548+0000) 2020-02-05T20:29:22.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:22.260+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:22.260492+0000) 2020-02-05T20:29:22.651 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:29:22.657 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:29:23.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:23.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:23.387241+0000) 2020-02-05T20:29:23.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:23.417+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:23.418415+0000) 2020-02-05T20:29:24.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:23.999+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.001002+0000) 2020-02-05T20:29:24.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:24.001+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.001788+0000) 2020-02-05T20:29:24.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:24.001+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.002688+0000) 2020-02-05T20:29:24.004 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:24.002+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.003307+0000) 2020-02-05T20:29:24.005 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:24.002+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.003715+0000) 2020-02-05T20:29:24.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:24.003+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.004958+0000) 2020-02-05T20:29:24.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:24.005+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.006435+0000) 2020-02-05T20:29:24.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:24.005+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.006199+0000) 2020-02-05T20:29:24.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:24.006+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.006470+0000) 2020-02-05T20:29:24.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:24.005+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.006319+0000) 2020-02-05T20:29:24.009 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:24.006+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.007065+0000) 2020-02-05T20:29:24.009 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:24.006+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.007274+0000) 2020-02-05T20:29:24.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:24.007+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:24.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:24.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.010311+0000) 2020-02-05T20:29:24.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:24.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.011701+0000) 2020-02-05T20:29:24.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:24.080+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:24.081412+0000) 2020-02-05T20:29:25.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:25.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:25.394042+0000) 2020-02-05T20:29:25.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:25.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:25.800818+0000) 2020-02-05T20:29:26.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:26.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:26.660353+0000) 2020-02-05T20:29:26.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:26.902+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:26.902614+0000) 2020-02-05T20:29:29.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:29.014+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.015594+0000) 2020-02-05T20:29:29.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:29.015+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.016916+0000) 2020-02-05T20:29:29.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:29.016+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.017095+0000) 2020-02-05T20:29:29.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:29.016+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.017785+0000) 2020-02-05T20:29:29.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:29.017+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.017968+0000) 2020-02-05T20:29:29.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:29.017+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.018503+0000) 2020-02-05T20:29:29.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:29.018+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.018761+0000) 2020-02-05T20:29:29.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:29.018+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.019337+0000) 2020-02-05T20:29:29.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:29.018+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.019394+0000) 2020-02-05T20:29:29.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:29.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:29.024 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:29.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.019746+0000) 2020-02-05T20:29:29.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:29.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.019868+0000) 2020-02-05T20:29:29.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:29.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.020221+0000) 2020-02-05T20:29:29.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:29.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.023003+0000) 2020-02-05T20:29:29.027 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:29.019+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.020674+0000) 2020-02-05T20:29:29.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:29.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.298140+0000) 2020-02-05T20:29:29.910 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:29.908+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:29.908596+0000) 2020-02-05T20:29:30.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:30.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:30.154995+0000) 2020-02-05T20:29:30.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:30.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:30.155095+0000) 2020-02-05T20:29:30.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:30.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:30.168787+0000) 2020-02-05T20:29:32.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:32.260+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:32.260678+0000) 2020-02-05T20:29:33.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:33.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:33.387474+0000) 2020-02-05T20:29:33.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:33.417+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:33.418622+0000) 2020-02-05T20:29:34.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:34.024+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.025165+0000) 2020-02-05T20:29:34.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:34.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.025787+0000) 2020-02-05T20:29:34.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:34.025+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.026260+0000) 2020-02-05T20:29:34.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:34.028+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.028505+0000) 2020-02-05T20:29:34.035 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:34.028+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.028967+0000) 2020-02-05T20:29:34.035 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:34.028+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.029250+0000) 2020-02-05T20:29:34.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:34.029+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.029926+0000) 2020-02-05T20:29:34.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:34.030+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.030659+0000) 2020-02-05T20:29:34.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:34.030+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.030786+0000) 2020-02-05T20:29:34.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:34.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:34.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:34.025+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.026945+0000) 2020-02-05T20:29:34.036 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:34.026+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.027757+0000) 2020-02-05T20:29:34.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:34.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.031650+0000) 2020-02-05T20:29:34.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:34.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.034619+0000) 2020-02-05T20:29:34.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:34.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.034945+0000) 2020-02-05T20:29:34.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:34.087+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:34.089021+0000) 2020-02-05T20:29:35.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:35.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:35.394342+0000) 2020-02-05T20:29:35.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:35.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:35.763960+0000) 2020-02-05T20:29:35.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:35.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:35.801054+0000) 2020-02-05T20:29:36.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:36.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:36.660582+0000) 2020-02-05T20:29:36.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:36.901+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:36.902837+0000) 2020-02-05T20:29:39.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:39.035+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.037021+0000) 2020-02-05T20:29:39.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:39.037+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.038436+0000) 2020-02-05T20:29:39.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:39.038+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.039426+0000) 2020-02-05T20:29:39.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:39.038+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.040056+0000) 2020-02-05T20:29:39.041 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:39.038+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.040109+0000) 2020-02-05T20:29:39.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:39.039+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.040526+0000) 2020-02-05T20:29:39.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:39.040+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.041129+0000) 2020-02-05T20:29:39.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:39.039+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.041225+0000) 2020-02-05T20:29:39.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:39.040+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.041880+0000) 2020-02-05T20:29:39.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:39.041+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.042750+0000) 2020-02-05T20:29:39.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:39.041+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.042863+0000) 2020-02-05T20:29:39.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:39.041+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.043171+0000) 2020-02-05T20:29:39.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:39.042+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.044238+0000) 2020-02-05T20:29:39.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:39.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:39.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:39.046+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.047861+0000) 2020-02-05T20:29:39.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:39.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.298349+0000) 2020-02-05T20:29:39.910 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:39.908+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:39.908822+0000) 2020-02-05T20:29:40.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:40.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:40.155255+0000) 2020-02-05T20:29:40.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:40.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:40.155344+0000) 2020-02-05T20:29:40.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:40.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:40.169011+0000) 2020-02-05T20:29:42.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:42.259+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:42.260932+0000) 2020-02-05T20:29:43.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:43.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:43.387742+0000) 2020-02-05T20:29:43.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:43.418+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:43.418877+0000) 2020-02-05T20:29:44.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:44.050+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.051844+0000) 2020-02-05T20:29:44.056 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:44.050+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.050902+0000) 2020-02-05T20:29:44.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:44.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.052597+0000) 2020-02-05T20:29:44.057 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:44.054+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.054395+0000) 2020-02-05T20:29:44.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:44.054+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.054530+0000) 2020-02-05T20:29:44.058 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:44.054+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.055016+0000) 2020-02-05T20:29:44.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:44.056+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.056260+0000) 2020-02-05T20:29:44.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:44.055+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.056682+0000) 2020-02-05T20:29:44.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:44.055+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.057055+0000) 2020-02-05T20:29:44.060 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:44.055+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:44.060 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:44.056+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.058098+0000) 2020-02-05T20:29:44.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:44.056+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.058266+0000) 2020-02-05T20:29:44.061 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:44.056+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.057571+0000) 2020-02-05T20:29:44.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:44.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.057910+0000) 2020-02-05T20:29:44.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:44.058+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.058248+0000) 2020-02-05T20:29:44.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:44.096+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:44.096459+0000) 2020-02-05T20:29:45.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:45.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:45.394563+0000) 2020-02-05T20:29:45.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:45.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:45.764137+0000) 2020-02-05T20:29:45.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:45.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:45.801315+0000) 2020-02-05T20:29:46.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:46.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:46.660916+0000) 2020-02-05T20:29:46.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:46.902+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:46.903080+0000) 2020-02-05T20:29:49.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:49.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.062225+0000) 2020-02-05T20:29:49.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:49.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.063424+0000) 2020-02-05T20:29:49.065 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:49.063+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.064142+0000) 2020-02-05T20:29:49.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:49.063+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.064721+0000) 2020-02-05T20:29:49.067 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:49.064+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.065601+0000) 2020-02-05T20:29:49.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:49.064+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.066151+0000) 2020-02-05T20:29:49.068 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:49.065+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.067046+0000) 2020-02-05T20:29:49.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:49.065+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.066724+0000) 2020-02-05T20:29:49.070 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:49.067+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.068618+0000) 2020-02-05T20:29:49.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:49.067+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.068913+0000) 2020-02-05T20:29:49.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:49.068+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.069361+0000) 2020-02-05T20:29:49.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:49.068+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.069493+0000) 2020-02-05T20:29:49.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:49.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:49.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:49.069+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.070271+0000) 2020-02-05T20:29:49.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:49.070+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.071812+0000) 2020-02-05T20:29:49.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:49.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.298580+0000) 2020-02-05T20:29:49.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:49.908+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:49.909023+0000) 2020-02-05T20:29:50.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:50.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:50.155501+0000) 2020-02-05T20:29:50.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:50.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:50.155654+0000) 2020-02-05T20:29:50.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:50.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:50.169272+0000) 2020-02-05T20:29:52.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:52.259+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:52.261168+0000) 2020-02-05T20:29:52.857 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:29:52.863 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:29:53.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:53.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:53.388004+0000) 2020-02-05T20:29:53.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:53.418+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:53.419122+0000) 2020-02-05T20:29:54.077 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:54.074+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.075399+0000) 2020-02-05T20:29:54.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:54.075+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.076361+0000) 2020-02-05T20:29:54.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:54.075+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.076882+0000) 2020-02-05T20:29:54.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:54.076+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.077504+0000) 2020-02-05T20:29:54.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:54.076+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.077755+0000) 2020-02-05T20:29:54.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:54.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.078027+0000) 2020-02-05T20:29:54.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:54.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.078355+0000) 2020-02-05T20:29:54.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:54.077+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.078749+0000) 2020-02-05T20:29:54.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:54.078+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.079938+0000) 2020-02-05T20:29:54.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:54.079+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.081292+0000) 2020-02-05T20:29:54.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:54.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.081470+0000) 2020-02-05T20:29:54.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:54.080+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.081884+0000) 2020-02-05T20:29:54.084 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:54.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:54.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:54.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.084158+0000) 2020-02-05T20:29:54.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:54.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.084400+0000) 2020-02-05T20:29:54.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:54.102+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:54.103651+0000) 2020-02-05T20:29:55.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:55.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:55.394823+0000) 2020-02-05T20:29:55.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:55.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:55.764403+0000) 2020-02-05T20:29:55.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:55.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:55.801538+0000) 2020-02-05T20:29:56.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:56.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:56.661154+0000) 2020-02-05T20:29:56.905 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:56.902+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:56.903346+0000) 2020-02-05T20:29:59.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:29:59.088+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.090151+0000) 2020-02-05T20:29:59.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:29:59.089+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.090872+0000) 2020-02-05T20:29:59.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:29:59.089+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.091198+0000) 2020-02-05T20:29:59.097 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:29:59.090+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.092176+0000) 2020-02-05T20:29:59.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:29:59.091+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.093268+0000) 2020-02-05T20:29:59.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:59.087+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.088102+0000) 2020-02-05T20:29:59.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:29:59.090+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.091780+0000) 2020-02-05T20:29:59.098 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:29:59.091+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.092942+0000) 2020-02-05T20:29:59.098 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:29:59.092+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.093531+0000) 2020-02-05T20:29:59.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:29:59.093+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.094484+0000) 2020-02-05T20:29:59.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:59.093+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.094451+0000) 2020-02-05T20:29:59.099 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:29:59.093+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.095198+0000) 2020-02-05T20:29:59.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:29:59.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:29:59.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:59.093+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.095048+0000) 2020-02-05T20:29:59.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:59.099+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.100643+0000) 2020-02-05T20:29:59.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:29:59.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.298802+0000) 2020-02-05T20:29:59.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:29:59.908+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:29:59.909313+0000) 2020-02-05T20:30:00.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:00.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:00.155733+0000) 2020-02-05T20:30:00.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:00.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:00.155911+0000) 2020-02-05T20:30:00.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:00.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:00.169491+0000) 2020-02-05T20:30:02.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:02.260+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:02.261409+0000) 2020-02-05T20:30:03.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:03.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:03.388301+0000) 2020-02-05T20:30:03.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:03.418+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:03.419365+0000) 2020-02-05T20:30:04.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:04.100+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.101587+0000) 2020-02-05T20:30:04.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:04.100+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.102053+0000) 2020-02-05T20:30:04.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:04.101+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.102872+0000) 2020-02-05T20:30:04.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:04.102+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.103801+0000) 2020-02-05T20:30:04.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:04.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.104505+0000) 2020-02-05T20:30:04.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:04.104+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.105068+0000) 2020-02-05T20:30:04.107 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:04.104+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.105465+0000) 2020-02-05T20:30:04.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:04.105+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.106462+0000) 2020-02-05T20:30:04.108 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:04.105+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.106590+0000) 2020-02-05T20:30:04.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:04.106+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.107583+0000) 2020-02-05T20:30:04.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:04.105+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.107029+0000) 2020-02-05T20:30:04.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:04.106+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.107770+0000) 2020-02-05T20:30:04.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:04.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:04.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:04.106+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.107995+0000) 2020-02-05T20:30:04.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:04.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.113384+0000) 2020-02-05T20:30:04.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:04.112+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:04.113426+0000) 2020-02-05T20:30:05.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:05.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:05.395067+0000) 2020-02-05T20:30:05.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:05.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:05.764641+0000) 2020-02-05T20:30:05.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:05.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:05.801796+0000) 2020-02-05T20:30:06.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:06.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:06.661388+0000) 2020-02-05T20:30:06.905 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:06.902+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:06.903626+0000) 2020-02-05T20:30:09.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:09.112+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.113383+0000) 2020-02-05T20:30:09.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:09.113+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.114891+0000) 2020-02-05T20:30:09.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:09.114+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.115609+0000) 2020-02-05T20:30:09.117 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:09.115+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.116065+0000) 2020-02-05T20:30:09.118 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:09.115+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.116953+0000) 2020-02-05T20:30:09.119 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:09.116+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.117528+0000) 2020-02-05T20:30:09.119 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:09.116+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.117956+0000) 2020-02-05T20:30:09.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:09.116+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.117890+0000) 2020-02-05T20:30:09.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:09.116+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.118060+0000) 2020-02-05T20:30:09.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:09.116+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.118264+0000) 2020-02-05T20:30:09.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:09.117+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.118702+0000) 2020-02-05T20:30:09.121 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:09.117+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.119133+0000) 2020-02-05T20:30:09.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:09.117+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.119039+0000) 2020-02-05T20:30:09.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:09.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:09.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:09.123+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.124547+0000) 2020-02-05T20:30:09.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:09.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.299011+0000) 2020-02-05T20:30:09.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:09.908+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:09.909522+0000) 2020-02-05T20:30:10.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:10.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:10.155986+0000) 2020-02-05T20:30:10.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:10.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:10.156120+0000) 2020-02-05T20:30:10.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:10.169+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:10.169700+0000) 2020-02-05T20:30:12.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:12.261+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:12.261697+0000) 2020-02-05T20:30:13.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:13.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:13.388557+0000) 2020-02-05T20:30:13.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:13.418+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:13.419614+0000) 2020-02-05T20:30:14.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:14.113+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.113654+0000) 2020-02-05T20:30:14.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:14.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.125679+0000) 2020-02-05T20:30:14.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:14.124+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.126185+0000) 2020-02-05T20:30:14.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:14.125+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.126608+0000) 2020-02-05T20:30:14.128 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:14.125+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.127006+0000) 2020-02-05T20:30:14.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:14.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.127511+0000) 2020-02-05T20:30:14.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:14.126+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.127731+0000) 2020-02-05T20:30:14.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:14.126+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.127618+0000) 2020-02-05T20:30:14.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:14.127+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.128286+0000) 2020-02-05T20:30:14.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:14.127+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.128713+0000) 2020-02-05T20:30:14.131 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:14.127+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.128990+0000) 2020-02-05T20:30:14.131 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:14.128+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.129528+0000) 2020-02-05T20:30:14.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:14.126+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.128198+0000) 2020-02-05T20:30:14.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:14.128+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.129678+0000) 2020-02-05T20:30:14.132 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:14.129+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:14.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:14.135+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:14.136390+0000) 2020-02-05T20:30:15.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:15.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:15.395348+0000) 2020-02-05T20:30:15.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:15.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:15.764936+0000) 2020-02-05T20:30:15.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:15.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:15.802023+0000) 2020-02-05T20:30:16.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:16.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:16.661632+0000) 2020-02-05T20:30:16.905 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:16.903+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:16.903858+0000) 2020-02-05T20:30:19.138 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:19.135+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.136687+0000) 2020-02-05T20:30:19.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:19.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.137493+0000) 2020-02-05T20:30:19.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:19.134+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.135906+0000) 2020-02-05T20:30:19.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:19.136+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.137849+0000) 2020-02-05T20:30:19.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:19.137+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.138622+0000) 2020-02-05T20:30:19.140 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:19.137+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.138419+0000) 2020-02-05T20:30:19.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:19.137+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.139133+0000) 2020-02-05T20:30:19.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:19.138+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.140131+0000) 2020-02-05T20:30:19.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:19.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.141141+0000) 2020-02-05T20:30:19.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:19.140+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.141871+0000) 2020-02-05T20:30:19.143 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:19.141+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.142256+0000) 2020-02-05T20:30:19.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:19.141+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.142589+0000) 2020-02-05T20:30:19.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:19.142+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.143893+0000) 2020-02-05T20:30:19.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:19.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:19.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:19.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.146373+0000) 2020-02-05T20:30:19.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:19.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.299248+0000) 2020-02-05T20:30:19.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:19.909+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:19.909725+0000) 2020-02-05T20:30:20.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:20.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:20.156249+0000) 2020-02-05T20:30:20.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:20.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:20.156330+0000) 2020-02-05T20:30:20.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:20.168+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:20.169936+0000) 2020-02-05T20:30:22.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:22.261+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:22.261952+0000) 2020-02-05T20:30:23.028 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:30:23.034 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:30:23.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:23.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:23.388803+0000) 2020-02-05T20:30:23.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:23.418+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:23.419862+0000) 2020-02-05T20:30:24.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:24.123+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.124147+0000) 2020-02-05T20:30:24.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:24.148+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.149464+0000) 2020-02-05T20:30:24.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:24.148+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.149928+0000) 2020-02-05T20:30:24.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:24.149+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.150898+0000) 2020-02-05T20:30:24.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:24.150+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.151111+0000) 2020-02-05T20:30:24.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:24.150+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.151425+0000) 2020-02-05T20:30:24.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:24.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.152005+0000) 2020-02-05T20:30:24.154 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:24.151+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.152419+0000) 2020-02-05T20:30:24.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:24.151+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.152885+0000) 2020-02-05T20:30:24.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:24.153+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.154485+0000) 2020-02-05T20:30:24.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:24.153+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.154695+0000) 2020-02-05T20:30:24.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:24.154+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.155237+0000) 2020-02-05T20:30:24.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:24.153+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.155227+0000) 2020-02-05T20:30:24.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:24.154+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.155593+0000) 2020-02-05T20:30:24.158 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:24.155+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:24.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:24.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:24.159666+0000) 2020-02-05T20:30:25.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:25.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:25.395649+0000) 2020-02-05T20:30:25.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:25.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:25.765208+0000) 2020-02-05T20:30:25.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:25.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:25.802312+0000) 2020-02-05T20:30:26.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:26.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:26.661878+0000) 2020-02-05T20:30:26.905 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:26.903+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:26.904119+0000) 2020-02-05T20:30:29.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:29.162+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.163730+0000) 2020-02-05T20:30:29.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:29.163+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.164064+0000) 2020-02-05T20:30:29.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:29.163+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.164409+0000) 2020-02-05T20:30:29.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:29.164+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.165089+0000) 2020-02-05T20:30:29.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:29.165+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.165906+0000) 2020-02-05T20:30:29.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:29.165+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.166258+0000) 2020-02-05T20:30:29.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:29.165+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.166695+0000) 2020-02-05T20:30:29.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:29.166+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.167251+0000) 2020-02-05T20:30:29.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:29.166+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.167599+0000) 2020-02-05T20:30:29.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:29.166+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.167872+0000) 2020-02-05T20:30:29.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:29.168+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.168783+0000) 2020-02-05T20:30:29.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:29.168+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.169037+0000) 2020-02-05T20:30:29.170 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:29.168+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.169220+0000) 2020-02-05T20:30:29.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:29.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58551 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:29.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:29.173+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.174904+0000) 2020-02-05T20:30:29.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:29.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.299443+0000) 2020-02-05T20:30:29.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:29.909+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:29.909961+0000) 2020-02-05T20:30:30.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:30.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:30.156533+0000) 2020-02-05T20:30:30.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:30.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:30.156586+0000) 2020-02-05T20:30:30.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:30.169+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:30.170196+0000) 2020-02-05T20:30:32.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:32.261+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:32.262206+0000) 2020-02-05T20:30:33.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:33.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:33.389060+0000) 2020-02-05T20:30:33.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:33.419+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:33.420138+0000) 2020-02-05T20:30:34.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:34.132+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.132445+0000) 2020-02-05T20:30:34.178 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:34.175+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.176700+0000) 2020-02-05T20:30:34.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:34.176+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.176998+0000) 2020-02-05T20:30:34.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:34.176+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.177385+0000) 2020-02-05T20:30:34.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:34.176+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.178043+0000) 2020-02-05T20:30:34.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:34.177+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.178676+0000) 2020-02-05T20:30:34.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:34.178+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.179274+0000) 2020-02-05T20:30:34.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:34.178+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.179165+0000) 2020-02-05T20:30:34.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:34.178+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.179977+0000) 2020-02-05T20:30:34.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:34.179+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.180071+0000) 2020-02-05T20:30:34.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:34.180+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.180567+0000) 2020-02-05T20:30:34.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:34.180+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.181274+0000) 2020-02-05T20:30:34.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:34.180+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.181362+0000) 2020-02-05T20:30:34.183 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:34.181+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.182077+0000) 2020-02-05T20:30:34.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:34.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58593 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:34.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:34.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:34.184630+0000) 2020-02-05T20:30:35.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:35.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:35.395902+0000) 2020-02-05T20:30:35.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:35.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:35.765444+0000) 2020-02-05T20:30:35.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:35.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:35.802547+0000) 2020-02-05T20:30:36.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:36.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:36.662142+0000) 2020-02-05T20:30:36.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:36.903+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:36.904359+0000) 2020-02-05T20:30:39.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:39.189+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.189527+0000) 2020-02-05T20:30:39.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:39.190+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.190571+0000) 2020-02-05T20:30:39.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:39.189+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.190178+0000) 2020-02-05T20:30:39.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:39.191+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.191760+0000) 2020-02-05T20:30:39.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:39.192+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.192511+0000) 2020-02-05T20:30:39.195 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:39.193+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.193375+0000) 2020-02-05T20:30:39.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:39.192+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.192681+0000) 2020-02-05T20:30:39.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:39.192+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.193185+0000) 2020-02-05T20:30:39.196 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:39.193+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.193490+0000) 2020-02-05T20:30:39.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:39.194+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.194137+0000) 2020-02-05T20:30:39.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:39.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.194577+0000) 2020-02-05T20:30:39.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:39.193+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.194383+0000) 2020-02-05T20:30:39.197 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:39.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:39.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:39.200+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.200440+0000) 2020-02-05T20:30:39.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:39.200+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.200707+0000) 2020-02-05T20:30:39.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:39.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.299654+0000) 2020-02-05T20:30:39.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:39.910+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:39.910172+0000) 2020-02-05T20:30:40.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:40.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:40.156856+0000) 2020-02-05T20:30:40.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:40.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:40.156856+0000) 2020-02-05T20:30:40.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:40.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:40.170427+0000) 2020-02-05T20:30:42.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:42.262+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:42.262499+0000) 2020-02-05T20:30:43.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:43.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:43.389290+0000) 2020-02-05T20:30:43.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:43.419+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:43.420418+0000) 2020-02-05T20:30:44.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:44.139+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.140687+0000) 2020-02-05T20:30:44.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:44.199+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.200905+0000) 2020-02-05T20:30:44.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:44.200+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.200656+0000) 2020-02-05T20:30:44.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:44.200+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.201999+0000) 2020-02-05T20:30:44.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:44.201+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.202315+0000) 2020-02-05T20:30:44.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:44.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.202861+0000) 2020-02-05T20:30:44.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:44.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.205135+0000) 2020-02-05T20:30:44.207 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:44.204+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.205476+0000) 2020-02-05T20:30:44.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:44.204+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.204538+0000) 2020-02-05T20:30:44.207 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:44.204+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.205845+0000) 2020-02-05T20:30:44.208 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:44.205+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.206303+0000) 2020-02-05T20:30:44.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:44.207+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.207508+0000) 2020-02-05T20:30:44.209 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:44.207+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:44.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:44.207+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.208093+0000) 2020-02-05T20:30:44.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:44.207+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.208163+0000) 2020-02-05T20:30:44.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:44.212+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:44.213930+0000) 2020-02-05T20:30:45.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:45.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:45.396166+0000) 2020-02-05T20:30:45.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:45.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:45.765668+0000) 2020-02-05T20:30:45.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:45.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:45.802775+0000) 2020-02-05T20:30:46.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:46.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:46.662398+0000) 2020-02-05T20:30:46.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:46.904+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:46.904622+0000) 2020-02-05T20:30:49.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:49.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.213197+0000) 2020-02-05T20:30:49.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:49.213+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.214191+0000) 2020-02-05T20:30:49.216 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:49.213+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.214456+0000) 2020-02-05T20:30:49.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:49.213+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.214898+0000) 2020-02-05T20:30:49.217 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:49.214+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.215962+0000) 2020-02-05T20:30:49.218 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:49.215+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.216174+0000) 2020-02-05T20:30:49.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:49.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.216722+0000) 2020-02-05T20:30:49.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:49.216+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.217326+0000) 2020-02-05T20:30:49.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:49.216+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.217903+0000) 2020-02-05T20:30:49.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:49.218+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.218975+0000) 2020-02-05T20:30:49.221 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:49.217+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.218781+0000) 2020-02-05T20:30:49.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:49.218+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.219379+0000) 2020-02-05T20:30:49.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:49.219+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.219461+0000) 2020-02-05T20:30:49.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:49.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:49.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:49.221+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.222443+0000) 2020-02-05T20:30:49.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:49.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.299871+0000) 2020-02-05T20:30:49.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:49.909+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:49.910382+0000) 2020-02-05T20:30:50.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:50.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:50.157108+0000) 2020-02-05T20:30:50.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:50.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:50.157112+0000) 2020-02-05T20:30:50.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:50.169+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:50.170641+0000) 2020-02-05T20:30:52.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:52.262+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:52.262789+0000) 2020-02-05T20:30:53.207 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:30:53.214 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:30:53.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:53.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:53.389550+0000) 2020-02-05T20:30:53.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:53.419+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:53.420669+0000) 2020-02-05T20:30:54.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:54.148+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.148937+0000) 2020-02-05T20:30:54.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:54.225+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.226065+0000) 2020-02-05T20:30:54.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:54.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.227398+0000) 2020-02-05T20:30:54.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:54.226+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.227605+0000) 2020-02-05T20:30:54.230 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:54.226+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.227946+0000) 2020-02-05T20:30:54.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:54.227+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.228736+0000) 2020-02-05T20:30:54.231 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:54.228+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.229888+0000) 2020-02-05T20:30:54.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:54.229+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.230314+0000) 2020-02-05T20:30:54.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:54.230+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.230816+0000) 2020-02-05T20:30:54.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:54.231+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.231707+0000) 2020-02-05T20:30:54.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:54.230+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.231933+0000) 2020-02-05T20:30:54.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:54.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.232189+0000) 2020-02-05T20:30:54.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:54.231+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.232357+0000) 2020-02-05T20:30:54.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:54.232+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.232612+0000) 2020-02-05T20:30:54.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:54.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:54.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:54.232+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:54.233588+0000) 2020-02-05T20:30:55.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:55.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:55.396455+0000) 2020-02-05T20:30:55.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:55.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:55.765901+0000) 2020-02-05T20:30:55.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:55.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:55.803025+0000) 2020-02-05T20:30:56.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:56.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:56.662648+0000) 2020-02-05T20:30:56.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:56.904+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:56.904921+0000) 2020-02-05T20:30:59.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:30:59.237+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.238545+0000) 2020-02-05T20:30:59.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:59.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.239959+0000) 2020-02-05T20:30:59.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:30:59.240+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.241129+0000) 2020-02-05T20:30:59.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:59.240+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.241465+0000) 2020-02-05T20:30:59.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:30:59.240+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.242026+0000) 2020-02-05T20:30:59.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:30:59.241+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.242357+0000) 2020-02-05T20:30:59.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:30:59.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.242896+0000) 2020-02-05T20:30:59.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:30:59.242+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.243080+0000) 2020-02-05T20:30:59.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:30:59.243+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.243781+0000) 2020-02-05T20:30:59.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:30:59.243+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.244350+0000) 2020-02-05T20:30:59.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:30:59.243+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.244314+0000) 2020-02-05T20:30:59.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:30:59.244+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.245036+0000) 2020-02-05T20:30:59.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:59.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.245400+0000) 2020-02-05T20:30:59.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:30:59.245+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:30:59.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:59.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.250438+0000) 2020-02-05T20:30:59.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:30:59.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.300106+0000) 2020-02-05T20:30:59.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:30:59.909+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:30:59.910597+0000) 2020-02-05T20:31:00.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:00.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:00.157335+0000) 2020-02-05T20:31:00.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:00.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:00.157398+0000) 2020-02-05T20:31:00.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:00.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:00.170852+0000) 2020-02-05T20:31:02.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:02.262+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:02.263074+0000) 2020-02-05T20:31:03.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:03.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:03.389821+0000) 2020-02-05T20:31:03.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:03.420+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:03.420950+0000) 2020-02-05T20:31:04.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:04.159+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.160738+0000) 2020-02-05T20:31:04.253 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:04.250+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.251390+0000) 2020-02-05T20:31:04.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:04.252+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.253018+0000) 2020-02-05T20:31:04.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:04.253+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.254145+0000) 2020-02-05T20:31:04.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:04.253+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.253573+0000) 2020-02-05T20:31:04.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:04.253+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.254075+0000) 2020-02-05T20:31:04.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:04.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.254945+0000) 2020-02-05T20:31:04.257 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:04.254+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.255947+0000) 2020-02-05T20:31:04.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:04.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.255695+0000) 2020-02-05T20:31:04.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:04.255+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.256437+0000) 2020-02-05T20:31:04.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:04.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.257152+0000) 2020-02-05T20:31:04.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:04.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.257472+0000) 2020-02-05T20:31:04.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:04.256+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.257278+0000) 2020-02-05T20:31:04.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:04.257+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.257782+0000) 2020-02-05T20:31:04.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:04.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:04.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:04.261+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:04.262489+0000) 2020-02-05T20:31:05.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:05.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:05.396719+0000) 2020-02-05T20:31:05.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:05.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:05.766133+0000) 2020-02-05T20:31:05.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:05.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:05.803283+0000) 2020-02-05T20:31:06.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:06.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:06.662878+0000) 2020-02-05T20:31:06.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:06.904+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:06.905176+0000) 2020-02-05T20:31:09.266 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:09.262+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.263411+0000) 2020-02-05T20:31:09.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:09.264+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.265206+0000) 2020-02-05T20:31:09.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:09.264+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.265771+0000) 2020-02-05T20:31:09.267 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:09.265+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.266245+0000) 2020-02-05T20:31:09.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:09.266+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.267279+0000) 2020-02-05T20:31:09.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:09.265+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.266817+0000) 2020-02-05T20:31:09.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:09.266+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.267815+0000) 2020-02-05T20:31:09.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:09.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.268638+0000) 2020-02-05T20:31:09.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:09.268+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.269666+0000) 2020-02-05T20:31:09.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:09.267+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.269345+0000) 2020-02-05T20:31:09.273 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:09.269+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.270447+0000) 2020-02-05T20:31:09.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:09.269+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.270587+0000) 2020-02-05T20:31:09.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:09.269+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.270955+0000) 2020-02-05T20:31:09.274 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:09.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58877 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:09.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:09.272+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.273533+0000) 2020-02-05T20:31:09.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:09.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.300355+0000) 2020-02-05T20:31:09.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:09.910+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:09.910807+0000) 2020-02-05T20:31:10.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:10.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:10.157545+0000) 2020-02-05T20:31:10.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:10.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:10.157669+0000) 2020-02-05T20:31:10.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:10.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:10.171063+0000) 2020-02-05T20:31:12.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:12.262+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:12.263315+0000) 2020-02-05T20:31:13.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:13.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:13.390007+0000) 2020-02-05T20:31:13.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:13.420+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:13.421181+0000) 2020-02-05T20:31:14.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:14.168+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.169259+0000) 2020-02-05T20:31:14.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:14.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.278482+0000) 2020-02-05T20:31:14.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:14.277+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.278733+0000) 2020-02-05T20:31:14.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:14.278+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.279270+0000) 2020-02-05T20:31:14.281 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:14.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.279909+0000) 2020-02-05T20:31:14.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:14.279+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.280507+0000) 2020-02-05T20:31:14.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:14.279+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.281049+0000) 2020-02-05T20:31:14.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:14.280+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.281568+0000) 2020-02-05T20:31:14.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:14.280+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.281828+0000) 2020-02-05T20:31:14.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:14.281+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.282378+0000) 2020-02-05T20:31:14.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:14.280+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.282054+0000) 2020-02-05T20:31:14.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:14.282+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.283457+0000) 2020-02-05T20:31:14.286 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:14.282+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.283667+0000) 2020-02-05T20:31:14.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:14.282+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.283826+0000) 2020-02-05T20:31:14.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:14.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58918 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:14.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:14.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:14.287709+0000) 2020-02-05T20:31:15.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:15.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:15.396971+0000) 2020-02-05T20:31:15.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:15.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:15.766358+0000) 2020-02-05T20:31:15.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:15.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:15.803548+0000) 2020-02-05T20:31:16.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:16.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:16.663144+0000) 2020-02-05T20:31:16.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:16.904+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:16.905379+0000) 2020-02-05T20:31:19.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:19.289+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.290284+0000) 2020-02-05T20:31:19.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:19.290+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.291072+0000) 2020-02-05T20:31:19.293 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:19.290+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.291816+0000) 2020-02-05T20:31:19.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:19.290+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.292048+0000) 2020-02-05T20:31:19.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:19.291+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.292666+0000) 2020-02-05T20:31:19.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:19.292+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.293135+0000) 2020-02-05T20:31:19.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:19.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.293509+0000) 2020-02-05T20:31:19.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:19.292+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.293975+0000) 2020-02-05T20:31:19.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:19.293+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.294712+0000) 2020-02-05T20:31:19.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:19.293+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.295041+0000) 2020-02-05T20:31:19.297 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:19.294+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.295450+0000) 2020-02-05T20:31:19.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:19.294+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.295750+0000) 2020-02-05T20:31:19.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:19.294+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.295988+0000) 2020-02-05T20:31:19.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:19.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:19.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:19.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.300551+0000) 2020-02-05T20:31:19.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:19.301+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.302772+0000) 2020-02-05T20:31:19.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:19.910+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:19.911007+0000) 2020-02-05T20:31:20.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:20.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:20.157781+0000) 2020-02-05T20:31:20.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:20.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:20.157986+0000) 2020-02-05T20:31:20.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:20.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:20.171299+0000) 2020-02-05T20:31:22.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:22.262+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:22.263568+0000) 2020-02-05T20:31:23.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:23.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:23.390298+0000) 2020-02-05T20:31:23.411 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:31:23.418 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:31:23.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:23.420+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:23.421406+0000) 2020-02-05T20:31:24.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:24.177+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.177569+0000) 2020-02-05T20:31:24.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:24.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.304242+0000) 2020-02-05T20:31:24.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:24.302+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.303918+0000) 2020-02-05T20:31:24.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:24.303+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.304778+0000) 2020-02-05T20:31:24.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:24.304+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.305270+0000) 2020-02-05T20:31:24.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:24.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.305891+0000) 2020-02-05T20:31:24.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:24.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.306491+0000) 2020-02-05T20:31:24.308 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:24.306+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.307121+0000) 2020-02-05T20:31:24.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:24.306+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.307713+0000) 2020-02-05T20:31:24.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:24.306+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.307524+0000) 2020-02-05T20:31:24.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:24.306+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.307918+0000) 2020-02-05T20:31:24.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:24.307+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.309058+0000) 2020-02-05T20:31:24.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:24.307+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.308755+0000) 2020-02-05T20:31:24.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:24.308+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.310172+0000) 2020-02-05T20:31:24.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:24.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 58999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:24.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:24.313+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:24.314541+0000) 2020-02-05T20:31:25.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:25.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:25.397212+0000) 2020-02-05T20:31:25.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:25.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:25.766575+0000) 2020-02-05T20:31:25.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:25.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:25.803801+0000) 2020-02-05T20:31:26.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:26.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:26.663420+0000) 2020-02-05T20:31:26.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:26.904+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:26.905689+0000) 2020-02-05T20:31:29.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:29.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.300761+0000) 2020-02-05T20:31:29.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:29.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.316695+0000) 2020-02-05T20:31:29.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:29.316+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.317685+0000) 2020-02-05T20:31:29.319 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:29.317+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.318381+0000) 2020-02-05T20:31:29.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:29.317+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.319133+0000) 2020-02-05T20:31:29.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:29.318+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.320043+0000) 2020-02-05T20:31:29.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:29.319+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.320519+0000) 2020-02-05T20:31:29.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:29.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.320475+0000) 2020-02-05T20:31:29.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:29.320+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.321283+0000) 2020-02-05T20:31:29.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:29.319+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.321100+0000) 2020-02-05T20:31:29.324 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:29.321+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.322063+0000) 2020-02-05T20:31:29.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:29.321+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.322699+0000) 2020-02-05T20:31:29.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:29.320+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.322391+0000) 2020-02-05T20:31:29.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:29.320+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.322380+0000) 2020-02-05T20:31:29.325 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:29.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:29.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:29.326+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.327729+0000) 2020-02-05T20:31:29.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:29.911+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:29.911270+0000) 2020-02-05T20:31:30.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:30.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:30.158009+0000) 2020-02-05T20:31:30.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:30.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:30.158176+0000) 2020-02-05T20:31:30.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:30.171+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:30.171509+0000) 2020-02-05T20:31:32.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:32.263+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:32.263797+0000) 2020-02-05T20:31:33.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:33.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:33.390547+0000) 2020-02-05T20:31:33.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:33.421+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:33.421675+0000) 2020-02-05T20:31:34.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:34.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.189386+0000) 2020-02-05T20:31:34.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:34.329+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.329192+0000) 2020-02-05T20:31:34.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:34.330+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.330387+0000) 2020-02-05T20:31:34.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:34.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.330841+0000) 2020-02-05T20:31:34.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:34.331+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.331175+0000) 2020-02-05T20:31:34.333 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:34.331+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.331524+0000) 2020-02-05T20:31:34.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:34.331+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.331804+0000) 2020-02-05T20:31:34.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:34.330+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.332028+0000) 2020-02-05T20:31:34.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:34.331+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.332488+0000) 2020-02-05T20:31:34.335 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:34.333+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.333212+0000) 2020-02-05T20:31:34.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:34.332+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.334231+0000) 2020-02-05T20:31:34.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:34.333+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.334971+0000) 2020-02-05T20:31:34.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:34.333+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.335390+0000) 2020-02-05T20:31:34.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:34.334+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.335565+0000) 2020-02-05T20:31:34.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:34.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:34.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:34.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:34.339171+0000) 2020-02-05T20:31:35.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:35.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:35.397477+0000) 2020-02-05T20:31:35.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:35.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:35.766811+0000) 2020-02-05T20:31:35.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:35.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:35.804048+0000) 2020-02-05T20:31:36.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:36.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:36.663664+0000) 2020-02-05T20:31:36.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:36.905+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:36.905988+0000) 2020-02-05T20:31:39.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:39.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.300985+0000) 2020-02-05T20:31:39.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:39.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.341623+0000) 2020-02-05T20:31:39.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:39.341+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.343382+0000) 2020-02-05T20:31:39.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:39.344+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.345462+0000) 2020-02-05T20:31:39.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:39.344+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.345792+0000) 2020-02-05T20:31:39.351 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:39.344+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.346003+0000) 2020-02-05T20:31:39.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:39.345+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.346970+0000) 2020-02-05T20:31:39.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:39.345+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.346767+0000) 2020-02-05T20:31:39.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:39.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:39.352 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:39.341+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.342567+0000) 2020-02-05T20:31:39.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:39.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.344037+0000) 2020-02-05T20:31:39.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:39.343+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.344226+0000) 2020-02-05T20:31:39.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:39.343+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.344489+0000) 2020-02-05T20:31:39.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:39.343+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.344901+0000) 2020-02-05T20:31:39.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:39.344+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.345167+0000) 2020-02-05T20:31:39.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:39.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.352463+0000) 2020-02-05T20:31:39.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:39.910+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:39.911516+0000) 2020-02-05T20:31:40.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:40.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:40.158227+0000) 2020-02-05T20:31:40.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:40.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:40.158380+0000) 2020-02-05T20:31:40.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:40.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:40.171739+0000) 2020-02-05T20:31:42.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:42.263+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:42.264032+0000) 2020-02-05T20:31:43.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:43.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:43.390794+0000) 2020-02-05T20:31:43.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:43.421+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:43.421903+0000) 2020-02-05T20:31:44.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:44.196+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.197360+0000) 2020-02-05T20:31:44.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:44.353+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.354405+0000) 2020-02-05T20:31:44.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:44.354+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.355854+0000) 2020-02-05T20:31:44.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:44.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.356372+0000) 2020-02-05T20:31:44.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:44.355+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.356989+0000) 2020-02-05T20:31:44.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:44.356+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.357275+0000) 2020-02-05T20:31:44.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:44.356+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.357592+0000) 2020-02-05T20:31:44.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:44.356+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.357786+0000) 2020-02-05T20:31:44.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:44.357+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.358590+0000) 2020-02-05T20:31:44.360 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:44.357+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.358734+0000) 2020-02-05T20:31:44.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:44.357+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.358202+0000) 2020-02-05T20:31:44.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:44.357+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.358380+0000) 2020-02-05T20:31:44.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:44.358+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.360025+0000) 2020-02-05T20:31:44.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:44.358+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.359521+0000) 2020-02-05T20:31:44.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:44.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:44.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:44.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:44.364275+0000) 2020-02-05T20:31:45.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:45.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:45.397744+0000) 2020-02-05T20:31:45.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:45.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:45.767070+0000) 2020-02-05T20:31:45.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:45.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:45.804343+0000) 2020-02-05T20:31:46.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:46.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:46.663923+0000) 2020-02-05T20:31:46.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:46.905+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:46.906239+0000) 2020-02-05T20:31:49.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:49.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.301221+0000) 2020-02-05T20:31:49.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:49.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.366386+0000) 2020-02-05T20:31:49.368 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:49.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.366819+0000) 2020-02-05T20:31:49.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:49.366+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.367318+0000) 2020-02-05T20:31:49.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:49.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.367996+0000) 2020-02-05T20:31:49.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:49.367+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.368213+0000) 2020-02-05T20:31:49.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:49.368+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.369532+0000) 2020-02-05T20:31:49.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:49.369+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.370696+0000) 2020-02-05T20:31:49.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:49.369+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.370797+0000) 2020-02-05T20:31:49.372 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:49.370+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.371325+0000) 2020-02-05T20:31:49.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:49.370+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.371803+0000) 2020-02-05T20:31:49.373 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:49.371+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.372140+0000) 2020-02-05T20:31:49.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:49.371+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.372796+0000) 2020-02-05T20:31:49.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:49.371+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.372865+0000) 2020-02-05T20:31:49.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:49.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:49.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:49.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.377238+0000) 2020-02-05T20:31:49.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:49.910+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:49.911716+0000) 2020-02-05T20:31:50.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:50.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:50.158487+0000) 2020-02-05T20:31:50.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:50.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:50.158635+0000) 2020-02-05T20:31:50.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:50.170+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:50.171949+0000) 2020-02-05T20:31:52.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:52.263+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:52.264279+0000) 2020-02-05T20:31:53.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:53.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:53.391040+0000) 2020-02-05T20:31:53.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:53.421+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:53.422133+0000) 2020-02-05T20:31:53.559 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:31:53.565 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:31:54.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:54.204+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.205907+0000) 2020-02-05T20:31:54.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:54.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.378486+0000) 2020-02-05T20:31:54.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:54.378+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.379487+0000) 2020-02-05T20:31:54.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:54.379+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.380703+0000) 2020-02-05T20:31:54.382 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:54.378+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.380094+0000) 2020-02-05T20:31:54.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:54.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.381812+0000) 2020-02-05T20:31:54.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:54.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.382429+0000) 2020-02-05T20:31:54.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:54.381+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.382663+0000) 2020-02-05T20:31:54.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:54.382+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.383334+0000) 2020-02-05T20:31:54.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:54.381+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.383154+0000) 2020-02-05T20:31:54.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:54.382+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.384271+0000) 2020-02-05T20:31:54.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:54.382+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.384142+0000) 2020-02-05T20:31:54.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:54.383+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.384871+0000) 2020-02-05T20:31:54.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:54.383+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.384916+0000) 2020-02-05T20:31:54.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:54.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:54.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:54.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:54.387602+0000) 2020-02-05T20:31:55.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:55.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:55.398018+0000) 2020-02-05T20:31:55.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:55.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:55.767347+0000) 2020-02-05T20:31:55.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:55.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:55.804606+0000) 2020-02-05T20:31:56.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:56.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:56.664161+0000) 2020-02-05T20:31:56.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:56.906+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:56.906476+0000) 2020-02-05T20:31:59.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:59.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.301454+0000) 2020-02-05T20:31:59.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:31:59.390+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.390886+0000) 2020-02-05T20:31:59.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:31:59.390+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.391781+0000) 2020-02-05T20:31:59.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:31:59.392+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.393317+0000) 2020-02-05T20:31:59.395 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:31:59.392+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.393882+0000) 2020-02-05T20:31:59.395 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:31:59.393+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.394399+0000) 2020-02-05T20:31:59.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:31:59.394+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.395322+0000) 2020-02-05T20:31:59.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:31:59.396+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.396487+0000) 2020-02-05T20:31:59.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:31:59.395+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.396124+0000) 2020-02-05T20:31:59.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:31:59.396+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.396947+0000) 2020-02-05T20:31:59.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:59.393+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.394943+0000) 2020-02-05T20:31:59.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:59.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.395305+0000) 2020-02-05T20:31:59.400 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:31:59.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:31:59.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:31:59.397+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.398161+0000) 2020-02-05T20:31:59.401 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:59.397+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.398599+0000) 2020-02-05T20:31:59.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:31:59.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.402209+0000) 2020-02-05T20:31:59.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:31:59.911+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:31:59.911921+0000) 2020-02-05T20:32:00.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:00.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:00.158721+0000) 2020-02-05T20:32:00.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:00.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:00.158865+0000) 2020-02-05T20:32:00.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:00.171+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:00.172192+0000) 2020-02-05T20:32:02.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:02.264+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:02.264475+0000) 2020-02-05T20:32:03.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:03.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:03.391300+0000) 2020-02-05T20:32:03.424 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:03.421+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:03.422367+0000) 2020-02-05T20:32:04.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:04.213+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.214210+0000) 2020-02-05T20:32:04.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:04.402+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.403963+0000) 2020-02-05T20:32:04.406 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:04.403+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.404929+0000) 2020-02-05T20:32:04.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:04.404+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.405646+0000) 2020-02-05T20:32:04.407 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:04.405+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.406205+0000) 2020-02-05T20:32:04.408 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:04.405+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.406721+0000) 2020-02-05T20:32:04.409 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:04.407+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.407518+0000) 2020-02-05T20:32:04.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:04.407+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.408161+0000) 2020-02-05T20:32:04.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:04.408+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.409532+0000) 2020-02-05T20:32:04.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:04.409+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.410504+0000) 2020-02-05T20:32:04.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:04.408+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.408662+0000) 2020-02-05T20:32:04.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:04.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.410133+0000) 2020-02-05T20:32:04.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:04.409+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.410309+0000) 2020-02-05T20:32:04.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:04.410+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.411040+0000) 2020-02-05T20:32:04.415 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:04.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:04.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:04.415+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:04.416129+0000) 2020-02-05T20:32:05.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:05.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:05.398302+0000) 2020-02-05T20:32:05.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:05.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:05.767574+0000) 2020-02-05T20:32:05.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:05.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:05.804861+0000) 2020-02-05T20:32:06.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:06.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:06.664351+0000) 2020-02-05T20:32:06.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:06.906+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:06.906707+0000) 2020-02-05T20:32:09.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:09.301+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.301688+0000) 2020-02-05T20:32:09.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:09.415+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.416917+0000) 2020-02-05T20:32:09.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:09.417+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.418319+0000) 2020-02-05T20:32:09.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:09.417+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.418821+0000) 2020-02-05T20:32:09.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:09.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.419129+0000) 2020-02-05T20:32:09.421 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:09.418+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.419678+0000) 2020-02-05T20:32:09.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:09.420+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.420842+0000) 2020-02-05T20:32:09.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:09.421+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.421802+0000) 2020-02-05T20:32:09.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:09.421+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.422131+0000) 2020-02-05T20:32:09.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:09.422+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.423299+0000) 2020-02-05T20:32:09.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:09.423+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:09.436 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:09.419+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.420815+0000) 2020-02-05T20:32:09.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:09.420+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.421333+0000) 2020-02-05T20:32:09.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:09.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.425891+0000) 2020-02-05T20:32:09.437 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:09.421+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.422263+0000) 2020-02-05T20:32:09.437 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:09.421+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.422831+0000) 2020-02-05T20:32:09.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:09.911+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:09.912154+0000) 2020-02-05T20:32:10.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:10.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:10.158964+0000) 2020-02-05T20:32:10.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:10.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:10.159102+0000) 2020-02-05T20:32:10.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:10.172+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:10.172436+0000) 2020-02-05T20:32:12.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:12.264+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:12.264726+0000) 2020-02-05T20:32:13.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:13.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:13.391533+0000) 2020-02-05T20:32:13.424 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:13.422+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:13.422601+0000) 2020-02-05T20:32:14.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:14.221+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.222428+0000) 2020-02-05T20:32:14.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:14.429+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.430204+0000) 2020-02-05T20:32:14.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:14.430+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.431374+0000) 2020-02-05T20:32:14.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:14.431+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.432135+0000) 2020-02-05T20:32:14.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:14.431+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.431973+0000) 2020-02-05T20:32:14.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:14.431+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.432614+0000) 2020-02-05T20:32:14.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:14.432+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.433567+0000) 2020-02-05T20:32:14.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:14.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.433797+0000) 2020-02-05T20:32:14.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:14.432+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.433099+0000) 2020-02-05T20:32:14.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:14.433+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.434302+0000) 2020-02-05T20:32:14.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:14.434+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.434958+0000) 2020-02-05T20:32:14.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:14.434+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.435328+0000) 2020-02-05T20:32:14.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:14.435+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.435781+0000) 2020-02-05T20:32:14.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:14.435+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.436188+0000) 2020-02-05T20:32:14.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:14.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:14.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:14.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:14.441526+0000) 2020-02-05T20:32:15.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:15.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:15.398582+0000) 2020-02-05T20:32:15.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:15.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:15.767810+0000) 2020-02-05T20:32:15.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:15.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:15.805129+0000) 2020-02-05T20:32:16.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:16.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:16.664586+0000) 2020-02-05T20:32:16.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:16.906+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:16.906954+0000) 2020-02-05T20:32:19.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:19.301+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.301923+0000) 2020-02-05T20:32:19.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:19.442+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.443412+0000) 2020-02-05T20:32:19.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:19.442+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.443804+0000) 2020-02-05T20:32:19.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:19.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.444358+0000) 2020-02-05T20:32:19.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:19.443+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.444524+0000) 2020-02-05T20:32:19.446 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:19.444+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.445084+0000) 2020-02-05T20:32:19.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:19.444+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.445884+0000) 2020-02-05T20:32:19.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:19.446+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.447068+0000) 2020-02-05T20:32:19.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:19.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.447555+0000) 2020-02-05T20:32:19.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:19.446+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.446541+0000) 2020-02-05T20:32:19.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:19.447+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.447466+0000) 2020-02-05T20:32:19.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:19.448+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.449163+0000) 2020-02-05T20:32:19.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:19.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.449340+0000) 2020-02-05T20:32:19.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:19.449+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.449579+0000) 2020-02-05T20:32:19.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:19.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:19.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:19.452+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.453803+0000) 2020-02-05T20:32:19.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:19.912+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:19.912391+0000) 2020-02-05T20:32:20.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:20.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:20.159194+0000) 2020-02-05T20:32:20.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:20.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:20.159337+0000) 2020-02-05T20:32:20.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:20.172+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:20.172649+0000) 2020-02-05T20:32:22.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:22.264+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:22.264975+0000) 2020-02-05T20:32:23.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:23.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:23.391787+0000) 2020-02-05T20:32:23.424 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:23.422+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:23.422839+0000) 2020-02-05T20:32:23.696 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:32:23.703 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:32:24.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:24.229+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.229756+0000) 2020-02-05T20:32:24.458 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:24.455+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.456301+0000) 2020-02-05T20:32:24.458 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:24.457+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.457108+0000) 2020-02-05T20:32:24.459 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:24.457+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.457659+0000) 2020-02-05T20:32:24.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:24.457+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.458207+0000) 2020-02-05T20:32:24.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:24.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.458586+0000) 2020-02-05T20:32:24.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:24.458+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.459229+0000) 2020-02-05T20:32:24.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:24.458+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.459379+0000) 2020-02-05T20:32:24.461 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:24.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.459702+0000) 2020-02-05T20:32:24.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:24.459+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.460066+0000) 2020-02-05T20:32:24.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:24.460+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.461153+0000) 2020-02-05T20:32:24.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:24.461+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.461673+0000) 2020-02-05T20:32:24.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:24.461+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.461885+0000) 2020-02-05T20:32:24.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:24.461+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.462129+0000) 2020-02-05T20:32:24.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:24.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:24.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:24.465+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:24.466451+0000) 2020-02-05T20:32:25.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:25.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:25.398806+0000) 2020-02-05T20:32:25.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:25.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:25.768059+0000) 2020-02-05T20:32:25.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:25.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:25.805351+0000) 2020-02-05T20:32:26.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:26.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:26.664816+0000) 2020-02-05T20:32:26.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:26.907+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:26.907204+0000) 2020-02-05T20:32:29.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:29.302+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.302159+0000) 2020-02-05T20:32:29.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:29.468+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.469589+0000) 2020-02-05T20:32:29.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:29.469+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.470308+0000) 2020-02-05T20:32:29.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:29.470+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.471051+0000) 2020-02-05T20:32:29.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:29.471+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.471956+0000) 2020-02-05T20:32:29.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:29.471+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.472170+0000) 2020-02-05T20:32:29.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:29.471+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.472706+0000) 2020-02-05T20:32:29.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:29.472+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.473739+0000) 2020-02-05T20:32:29.478 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:29.473+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.474199+0000) 2020-02-05T20:32:29.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:29.474+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.475032+0000) 2020-02-05T20:32:29.479 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:29.474+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.475228+0000) 2020-02-05T20:32:29.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:29.475+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.475888+0000) 2020-02-05T20:32:29.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:29.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:29.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:29.473+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.474512+0000) 2020-02-05T20:32:29.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:29.474+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.475349+0000) 2020-02-05T20:32:29.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:29.479+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.480867+0000) 2020-02-05T20:32:29.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:29.911+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:29.912604+0000) 2020-02-05T20:32:30.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:30.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:30.159500+0000) 2020-02-05T20:32:30.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:30.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:30.159549+0000) 2020-02-05T20:32:30.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:30.171+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:30.172879+0000) 2020-02-05T20:32:32.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:32.265+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:32.265249+0000) 2020-02-05T20:32:33.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:33.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:33.392039+0000) 2020-02-05T20:32:33.424 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:33.422+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:33.423106+0000) 2020-02-05T20:32:34.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:34.238+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.239021+0000) 2020-02-05T20:32:34.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:34.481+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.482661+0000) 2020-02-05T20:32:34.484 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:34.480+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.482071+0000) 2020-02-05T20:32:34.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:34.482+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.483476+0000) 2020-02-05T20:32:34.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:34.482+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.483946+0000) 2020-02-05T20:32:34.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:34.483+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.484808+0000) 2020-02-05T20:32:34.486 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:34.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.484869+0000) 2020-02-05T20:32:34.487 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:34.484+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.485242+0000) 2020-02-05T20:32:34.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:34.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.485659+0000) 2020-02-05T20:32:34.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:34.485+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.486539+0000) 2020-02-05T20:32:34.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:34.485+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.486769+0000) 2020-02-05T20:32:34.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:34.485+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.487060+0000) 2020-02-05T20:32:34.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:34.486+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.487999+0000) 2020-02-05T20:32:34.489 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:34.486+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.488142+0000) 2020-02-05T20:32:34.490 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:34.487+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:34.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:34.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:34.494315+0000) 2020-02-05T20:32:35.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:35.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:35.399051+0000) 2020-02-05T20:32:35.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:35.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:35.768373+0000) 2020-02-05T20:32:35.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:35.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:35.805585+0000) 2020-02-05T20:32:36.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:36.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:36.665047+0000) 2020-02-05T20:32:36.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:36.906+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:36.907462+0000) 2020-02-05T20:32:39.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:39.301+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.302344+0000) 2020-02-05T20:32:39.495 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:39.492+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.493500+0000) 2020-02-05T20:32:39.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:39.493+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.494111+0000) 2020-02-05T20:32:39.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:39.493+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.494529+0000) 2020-02-05T20:32:39.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:39.494+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.495149+0000) 2020-02-05T20:32:39.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:39.494+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.496276+0000) 2020-02-05T20:32:39.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:39.495+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.496822+0000) 2020-02-05T20:32:39.498 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:39.496+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.497576+0000) 2020-02-05T20:32:39.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:39.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.498067+0000) 2020-02-05T20:32:39.500 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:39.497+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.498633+0000) 2020-02-05T20:32:39.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:39.498+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.499567+0000) 2020-02-05T20:32:39.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:39.498+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.500000+0000) 2020-02-05T20:32:39.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:39.499+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.500668+0000) 2020-02-05T20:32:39.502 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:39.499+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.500965+0000) 2020-02-05T20:32:39.502 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:39.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:39.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:39.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.506069+0000) 2020-02-05T20:32:39.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:39.911+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:39.912812+0000) 2020-02-05T20:32:40.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:40.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:40.159774+0000) 2020-02-05T20:32:40.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:40.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:40.159774+0000) 2020-02-05T20:32:40.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:40.172+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:40.173140+0000) 2020-02-05T20:32:42.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:42.264+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:42.265500+0000) 2020-02-05T20:32:43.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:43.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:43.392276+0000) 2020-02-05T20:32:43.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:43.422+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:43.423352+0000) 2020-02-05T20:32:44.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:44.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.245754+0000) 2020-02-05T20:32:44.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:44.504+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.505954+0000) 2020-02-05T20:32:44.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:44.506+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.508026+0000) 2020-02-05T20:32:44.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:44.507+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.508254+0000) 2020-02-05T20:32:44.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:44.507+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.508738+0000) 2020-02-05T20:32:44.510 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:44.507+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.509053+0000) 2020-02-05T20:32:44.511 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:44.507+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.509301+0000) 2020-02-05T20:32:44.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:44.508+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.509856+0000) 2020-02-05T20:32:44.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:44.509+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.510135+0000) 2020-02-05T20:32:44.512 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:44.508+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.510361+0000) 2020-02-05T20:32:44.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:44.509+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.511171+0000) 2020-02-05T20:32:44.513 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:44.509+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.510611+0000) 2020-02-05T20:32:44.513 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:44.509+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.510675+0000) 2020-02-05T20:32:44.513 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:44.510+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.511812+0000) 2020-02-05T20:32:44.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:44.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:44.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:44.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:44.515862+0000) 2020-02-05T20:32:45.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:45.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:45.399287+0000) 2020-02-05T20:32:45.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:45.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:45.768607+0000) 2020-02-05T20:32:45.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:45.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:45.805818+0000) 2020-02-05T20:32:46.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:46.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:46.665326+0000) 2020-02-05T20:32:46.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:46.906+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:46.907726+0000) 2020-02-05T20:32:49.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:49.301+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.302587+0000) 2020-02-05T20:32:49.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:49.517+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.518482+0000) 2020-02-05T20:32:49.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:49.518+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.519113+0000) 2020-02-05T20:32:49.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:49.518+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.519492+0000) 2020-02-05T20:32:49.522 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:49.519+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.520402+0000) 2020-02-05T20:32:49.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:49.519+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.521008+0000) 2020-02-05T20:32:49.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:49.520+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.521541+0000) 2020-02-05T20:32:49.523 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:49.520+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.522035+0000) 2020-02-05T20:32:49.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:49.521+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.522884+0000) 2020-02-05T20:32:49.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:49.521+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.522485+0000) 2020-02-05T20:32:49.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:49.522+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.524169+0000) 2020-02-05T20:32:49.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:49.523+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.524555+0000) 2020-02-05T20:32:49.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:49.523+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.524691+0000) 2020-02-05T20:32:49.527 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:49.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:49.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:49.527+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.528785+0000) 2020-02-05T20:32:49.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:49.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.529139+0000) 2020-02-05T20:32:49.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:49.912+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:49.913036+0000) 2020-02-05T20:32:50.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:50.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:50.160018+0000) 2020-02-05T20:32:50.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:50.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:50.160016+0000) 2020-02-05T20:32:50.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:50.172+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:50.173346+0000) 2020-02-05T20:32:52.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:52.264+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:52.265752+0000) 2020-02-05T20:32:53.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:53.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:53.392523+0000) 2020-02-05T20:32:53.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:53.423+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:53.423645+0000) 2020-02-05T20:32:53.876 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:32:53.882 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:32:54.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:54.254+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.254625+0000) 2020-02-05T20:32:54.533 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:54.530+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.531172+0000) 2020-02-05T20:32:54.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:54.530+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.531650+0000) 2020-02-05T20:32:54.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:54.531+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.532164+0000) 2020-02-05T20:32:54.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:54.531+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.533055+0000) 2020-02-05T20:32:54.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:54.533+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.535006+0000) 2020-02-05T20:32:54.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:54.534+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.535334+0000) 2020-02-05T20:32:54.537 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:54.534+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.535752+0000) 2020-02-05T20:32:54.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:54.534+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.536206+0000) 2020-02-05T20:32:54.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:54.535+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.536818+0000) 2020-02-05T20:32:54.538 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:54.535+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.537032+0000) 2020-02-05T20:32:54.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:54.536+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.537602+0000) 2020-02-05T20:32:54.540 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:54.537+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.538261+0000) 2020-02-05T20:32:54.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:54.537+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59731 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:54.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:54.540+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.541452+0000) 2020-02-05T20:32:54.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:54.542+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:54.544033+0000) 2020-02-05T20:32:55.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:55.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:55.399523+0000) 2020-02-05T20:32:55.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:55.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:55.768877+0000) 2020-02-05T20:32:55.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:55.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:55.806047+0000) 2020-02-05T20:32:56.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:56.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:56.665551+0000) 2020-02-05T20:32:56.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:56.907+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:56.907985+0000) 2020-02-05T20:32:59.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:59.302+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.302820+0000) 2020-02-05T20:32:59.550 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:32:59.544+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.545965+0000) 2020-02-05T20:32:59.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:32:59.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.546384+0000) 2020-02-05T20:32:59.550 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:32:59.546+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.547462+0000) 2020-02-05T20:32:59.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:32:59.543+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.545057+0000) 2020-02-05T20:32:59.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:59.545+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.546752+0000) 2020-02-05T20:32:59.551 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:59.546+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.547759+0000) 2020-02-05T20:32:59.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:59.548+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.549352+0000) 2020-02-05T20:32:59.551 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:32:59.548+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.549767+0000) 2020-02-05T20:32:59.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:32:59.548+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.550028+0000) 2020-02-05T20:32:59.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:32:59.548+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.550217+0000) 2020-02-05T20:32:59.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:32:59.549+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.551153+0000) 2020-02-05T20:32:59.553 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:32:59.549+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.551030+0000) 2020-02-05T20:32:59.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:32:59.550+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.551704+0000) 2020-02-05T20:32:59.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:32:59.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:32:59.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:32:59.557+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.558604+0000) 2020-02-05T20:32:59.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:32:59.912+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:32:59.913277+0000) 2020-02-05T20:33:00.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:00.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:00.160257+0000) 2020-02-05T20:33:00.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:00.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:00.160259+0000) 2020-02-05T20:33:00.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:00.173+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:00.173656+0000) 2020-02-05T20:33:02.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:02.265+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:02.266005+0000) 2020-02-05T20:33:03.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:03.392+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:03.392771+0000) 2020-02-05T20:33:03.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:03.423+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:03.423876+0000) 2020-02-05T20:33:04.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:04.261+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.261980+0000) 2020-02-05T20:33:04.559 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:04.556+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.557648+0000) 2020-02-05T20:33:04.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:04.557+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.558998+0000) 2020-02-05T20:33:04.561 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:04.558+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.559719+0000) 2020-02-05T20:33:04.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:04.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.560242+0000) 2020-02-05T20:33:04.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:04.559+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.561080+0000) 2020-02-05T20:33:04.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:04.559+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.561362+0000) 2020-02-05T20:33:04.565 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:04.560+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.561564+0000) 2020-02-05T20:33:04.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:04.562+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.563271+0000) 2020-02-05T20:33:04.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:04.560+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.562205+0000) 2020-02-05T20:33:04.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:04.561+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.562893+0000) 2020-02-05T20:33:04.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:04.563+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.564806+0000) 2020-02-05T20:33:04.567 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:04.562+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.564059+0000) 2020-02-05T20:33:04.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:04.562+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.564298+0000) 2020-02-05T20:33:04.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:04.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:04.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:04.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:04.572419+0000) 2020-02-05T20:33:05.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:05.399+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:05.399758+0000) 2020-02-05T20:33:05.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:05.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:05.769167+0000) 2020-02-05T20:33:05.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:05.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:05.806314+0000) 2020-02-05T20:33:06.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:06.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:06.665770+0000) 2020-02-05T20:33:06.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:06.907+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:06.908232+0000) 2020-02-05T20:33:09.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:09.302+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.303073+0000) 2020-02-05T20:33:09.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:09.571+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.572718+0000) 2020-02-05T20:33:09.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:09.572+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.573299+0000) 2020-02-05T20:33:09.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:09.572+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.573389+0000) 2020-02-05T20:33:09.575 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:09.572+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.573861+0000) 2020-02-05T20:33:09.576 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:09.572+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.574389+0000) 2020-02-05T20:33:09.576 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:09.574+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.575065+0000) 2020-02-05T20:33:09.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:09.573+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.574995+0000) 2020-02-05T20:33:09.577 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:09.574+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.575690+0000) 2020-02-05T20:33:09.577 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:09.574+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.576321+0000) 2020-02-05T20:33:09.578 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:09.575+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.576699+0000) 2020-02-05T20:33:09.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:09.575+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.576944+0000) 2020-02-05T20:33:09.579 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:09.575+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.576872+0000) 2020-02-05T20:33:09.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:09.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.577531+0000) 2020-02-05T20:33:09.579 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:09.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59852 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:09.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:09.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.581578+0000) 2020-02-05T20:33:09.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:09.912+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:09.913483+0000) 2020-02-05T20:33:10.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:10.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:10.160480+0000) 2020-02-05T20:33:10.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:10.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:10.160516+0000) 2020-02-05T20:33:10.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:10.173+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:10.173859+0000) 2020-02-05T20:33:12.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:12.265+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:12.266273+0000) 2020-02-05T20:33:13.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:13.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:13.393035+0000) 2020-02-05T20:33:13.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:13.423+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:13.424124+0000) 2020-02-05T20:33:14.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:14.270+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.270365+0000) 2020-02-05T20:33:14.586 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:14.583+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.584186+0000) 2020-02-05T20:33:14.586 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:14.583+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.584570+0000) 2020-02-05T20:33:14.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:14.583+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.585093+0000) 2020-02-05T20:33:14.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:14.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.585538+0000) 2020-02-05T20:33:14.588 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:14.584+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.586311+0000) 2020-02-05T20:33:14.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:14.585+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.586512+0000) 2020-02-05T20:33:14.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:14.585+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.586729+0000) 2020-02-05T20:33:14.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:14.586+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.587517+0000) 2020-02-05T20:33:14.589 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:14.586+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.587841+0000) 2020-02-05T20:33:14.589 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:14.586+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.588301+0000) 2020-02-05T20:33:14.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:14.587+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.589069+0000) 2020-02-05T20:33:14.591 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:14.588+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.589146+0000) 2020-02-05T20:33:14.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:14.588+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.589567+0000) 2020-02-05T20:33:14.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:14.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59891 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:14.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:14.592+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:14.593135+0000) 2020-02-05T20:33:15.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:15.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:15.399976+0000) 2020-02-05T20:33:15.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:15.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:15.769381+0000) 2020-02-05T20:33:15.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:15.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:15.806551+0000) 2020-02-05T20:33:16.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:16.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:16.666040+0000) 2020-02-05T20:33:16.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:16.907+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:16.908473+0000) 2020-02-05T20:33:19.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:19.303+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.303398+0000) 2020-02-05T20:33:19.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:19.595+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.595580+0000) 2020-02-05T20:33:19.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:19.595+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.597356+0000) 2020-02-05T20:33:19.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:19.598+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.598196+0000) 2020-02-05T20:33:19.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:19.596+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.598391+0000) 2020-02-05T20:33:19.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:19.598+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.598798+0000) 2020-02-05T20:33:19.600 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:19.598+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.599057+0000) 2020-02-05T20:33:19.601 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:19.598+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.599843+0000) 2020-02-05T20:33:19.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:19.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.600020+0000) 2020-02-05T20:33:19.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:19.599+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.600484+0000) 2020-02-05T20:33:19.602 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:19.599+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.600656+0000) 2020-02-05T20:33:19.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:19.599+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.601200+0000) 2020-02-05T20:33:19.603 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:19.600+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.602380+0000) 2020-02-05T20:33:19.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:19.601+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.602712+0000) 2020-02-05T20:33:19.604 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:19.601+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59931 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:19.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:19.605+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.606648+0000) 2020-02-05T20:33:19.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:19.913+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:19.913690+0000) 2020-02-05T20:33:20.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:20.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:20.160730+0000) 2020-02-05T20:33:20.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:20.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:20.160733+0000) 2020-02-05T20:33:20.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:20.174+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:20.174069+0000) 2020-02-05T20:33:22.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:22.266+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:22.266518+0000) 2020-02-05T20:33:23.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:23.392+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:23.393271+0000) 2020-02-05T20:33:23.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:23.424+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:23.424373+0000) 2020-02-05T20:33:24.067 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:33:24.075 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:33:24.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:24.277+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.278923+0000) 2020-02-05T20:33:24.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:24.607+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.608992+0000) 2020-02-05T20:33:24.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:24.608+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.609746+0000) 2020-02-05T20:33:24.611 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:24.609+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.610336+0000) 2020-02-05T20:33:24.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:24.610+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.611581+0000) 2020-02-05T20:33:24.613 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:24.610+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.611942+0000) 2020-02-05T20:33:24.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:24.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.612596+0000) 2020-02-05T20:33:24.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:24.612+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.612795+0000) 2020-02-05T20:33:24.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:24.612+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.613232+0000) 2020-02-05T20:33:24.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:24.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.613464+0000) 2020-02-05T20:33:24.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:24.613+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.613927+0000) 2020-02-05T20:33:24.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:24.613+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.614177+0000) 2020-02-05T20:33:24.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:24.614+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.614802+0000) 2020-02-05T20:33:24.616 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:24.614+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.615022+0000) 2020-02-05T20:33:24.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:24.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 59971 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:24.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:24.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:24.621043+0000) 2020-02-05T20:33:25.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:25.399+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:25.400225+0000) 2020-02-05T20:33:25.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:25.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:25.769635+0000) 2020-02-05T20:33:25.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:25.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:25.806756+0000) 2020-02-05T20:33:26.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:26.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:26.666318+0000) 2020-02-05T20:33:26.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:26.908+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:26.908722+0000) 2020-02-05T20:33:29.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:29.302+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.303626+0000) 2020-02-05T20:33:29.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:29.620+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.621874+0000) 2020-02-05T20:33:29.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:29.621+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.622937+0000) 2020-02-05T20:33:29.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:29.622+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.622902+0000) 2020-02-05T20:33:29.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:29.622+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.623346+0000) 2020-02-05T20:33:29.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:29.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.623686+0000) 2020-02-05T20:33:29.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:29.623+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.624231+0000) 2020-02-05T20:33:29.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:29.625+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.625541+0000) 2020-02-05T20:33:29.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:29.625+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.625714+0000) 2020-02-05T20:33:29.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:29.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.626039+0000) 2020-02-05T20:33:29.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:29.626+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.626415+0000) 2020-02-05T20:33:29.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:29.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.626641+0000) 2020-02-05T20:33:29.629 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:29.626+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.627013+0000) 2020-02-05T20:33:29.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:29.626+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.627703+0000) 2020-02-05T20:33:29.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:29.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60013 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:29.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:29.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.629942+0000) 2020-02-05T20:33:29.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:29.912+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:29.913915+0000) 2020-02-05T20:33:30.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:30.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:30.160945+0000) 2020-02-05T20:33:30.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:30.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:30.160944+0000) 2020-02-05T20:33:30.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:30.173+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:30.174351+0000) 2020-02-05T20:33:32.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:32.266+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:32.266786+0000) 2020-02-05T20:33:33.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:33.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:33.393569+0000) 2020-02-05T20:33:33.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:33.423+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:33.424613+0000) 2020-02-05T20:33:34.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:34.285+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.286534+0000) 2020-02-05T20:33:34.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:34.633+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.634342+0000) 2020-02-05T20:33:34.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:34.635+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.635728+0000) 2020-02-05T20:33:34.637 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:34.635+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.636238+0000) 2020-02-05T20:33:34.638 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:34.635+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.636923+0000) 2020-02-05T20:33:34.639 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:34.637+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.638034+0000) 2020-02-05T20:33:34.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:34.637+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.638480+0000) 2020-02-05T20:33:34.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:34.637+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.637508+0000) 2020-02-05T20:33:34.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:34.638+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.638616+0000) 2020-02-05T20:33:34.641 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:34.638+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.638934+0000) 2020-02-05T20:33:34.641 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:34.639+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.639715+0000) 2020-02-05T20:33:34.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:34.639+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.640049+0000) 2020-02-05T20:33:34.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:34.639+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.640301+0000) 2020-02-05T20:33:34.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:34.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.640602+0000) 2020-02-05T20:33:34.643 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:34.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:34.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:34.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:34.647406+0000) 2020-02-05T20:33:35.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:35.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:35.400505+0000) 2020-02-05T20:33:35.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:35.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:35.769895+0000) 2020-02-05T20:33:35.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:35.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:35.807007+0000) 2020-02-05T20:33:36.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:36.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:36.666541+0000) 2020-02-05T20:33:36.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:36.908+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:36.908950+0000) 2020-02-05T20:33:39.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:39.303+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.303886+0000) 2020-02-05T20:33:39.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:39.645+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.646688+0000) 2020-02-05T20:33:39.649 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:39.646+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.647012+0000) 2020-02-05T20:33:39.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:39.648+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.648507+0000) 2020-02-05T20:33:39.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:39.648+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.649829+0000) 2020-02-05T20:33:39.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:39.649+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.649629+0000) 2020-02-05T20:33:39.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:39.649+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.650265+0000) 2020-02-05T20:33:39.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:39.650+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.650934+0000) 2020-02-05T20:33:39.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:39.650+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.651510+0000) 2020-02-05T20:33:39.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:39.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.651278+0000) 2020-02-05T20:33:39.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:39.651+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.651969+0000) 2020-02-05T20:33:39.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:39.651+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.652244+0000) 2020-02-05T20:33:39.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:39.651+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.652183+0000) 2020-02-05T20:33:39.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:39.651+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.652625+0000) 2020-02-05T20:33:39.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:39.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:39.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:39.652+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.653965+0000) 2020-02-05T20:33:39.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:39.913+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:39.914125+0000) 2020-02-05T20:33:40.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:40.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:40.161175+0000) 2020-02-05T20:33:40.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:40.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:40.161176+0000) 2020-02-05T20:33:40.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:40.173+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:40.174578+0000) 2020-02-05T20:33:42.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:42.266+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:42.267022+0000) 2020-02-05T20:33:43.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:43.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:43.393793+0000) 2020-02-05T20:33:43.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:43.423+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:43.424858+0000) 2020-02-05T20:33:44.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:44.295+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.296016+0000) 2020-02-05T20:33:44.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:44.657+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.657978+0000) 2020-02-05T20:33:44.661 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:44.659+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.659538+0000) 2020-02-05T20:33:44.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:44.660+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.661304+0000) 2020-02-05T20:33:44.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:44.660+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.661537+0000) 2020-02-05T20:33:44.663 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:44.661+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.661621+0000) 2020-02-05T20:33:44.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:44.660+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.661922+0000) 2020-02-05T20:33:44.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:44.661+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.662579+0000) 2020-02-05T20:33:44.665 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:44.662+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.663034+0000) 2020-02-05T20:33:44.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:44.663+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.663424+0000) 2020-02-05T20:33:44.665 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:44.662+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.663186+0000) 2020-02-05T20:33:44.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:44.663+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.664141+0000) 2020-02-05T20:33:44.666 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:44.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.664150+0000) 2020-02-05T20:33:44.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:44.664+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.665324+0000) 2020-02-05T20:33:44.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:44.665+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:44.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:44.667+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:44.668504+0000) 2020-02-05T20:33:45.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:45.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:45.400716+0000) 2020-02-05T20:33:45.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:45.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:45.770165+0000) 2020-02-05T20:33:45.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:45.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:45.807266+0000) 2020-02-05T20:33:46.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:46.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:46.666735+0000) 2020-02-05T20:33:46.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:46.908+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:46.909196+0000) 2020-02-05T20:33:49.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:49.303+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.304121+0000) 2020-02-05T20:33:49.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:49.670+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.670779+0000) 2020-02-05T20:33:49.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:49.670+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.671117+0000) 2020-02-05T20:33:49.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:49.670+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.671720+0000) 2020-02-05T20:33:49.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:49.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.672406+0000) 2020-02-05T20:33:49.675 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:49.672+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.673647+0000) 2020-02-05T20:33:49.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:49.673+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.673980+0000) 2020-02-05T20:33:49.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:49.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.674101+0000) 2020-02-05T20:33:49.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:49.674+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.674563+0000) 2020-02-05T20:33:49.676 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:49.673+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.675023+0000) 2020-02-05T20:33:49.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:49.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.675448+0000) 2020-02-05T20:33:49.677 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:49.674+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.675344+0000) 2020-02-05T20:33:49.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:49.675+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.675763+0000) 2020-02-05T20:33:49.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:49.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60174 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:49.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:49.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.680086+0000) 2020-02-05T20:33:49.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:49.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.680354+0000) 2020-02-05T20:33:49.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:49.913+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:49.914338+0000) 2020-02-05T20:33:50.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:50.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:50.161406+0000) 2020-02-05T20:33:50.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:50.161+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:50.161415+0000) 2020-02-05T20:33:50.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:50.174+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:50.174802+0000) 2020-02-05T20:33:52.269 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:52.267+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:52.267287+0000) 2020-02-05T20:33:53.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:53.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:53.393983+0000) 2020-02-05T20:33:53.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:53.424+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:53.425146+0000) 2020-02-05T20:33:54.263 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:33:54.269 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:33:54.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:54.302+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.303407+0000) 2020-02-05T20:33:54.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:54.682+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.683429+0000) 2020-02-05T20:33:54.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:54.683+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.684611+0000) 2020-02-05T20:33:54.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:54.684+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.685845+0000) 2020-02-05T20:33:54.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:54.685+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.686244+0000) 2020-02-05T20:33:54.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:54.686+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.686668+0000) 2020-02-05T20:33:54.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:54.686+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.687503+0000) 2020-02-05T20:33:54.689 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:54.687+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.687729+0000) 2020-02-05T20:33:54.690 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:54.687+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.688005+0000) 2020-02-05T20:33:54.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:54.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.687781+0000) 2020-02-05T20:33:54.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:54.688+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.688472+0000) 2020-02-05T20:33:54.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:54.688+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.688846+0000) 2020-02-05T20:33:54.691 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:54.689+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.689519+0000) 2020-02-05T20:33:54.691 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:54.687+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.688824+0000) 2020-02-05T20:33:54.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:54.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:54.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:54.691+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:54.692892+0000) 2020-02-05T20:33:55.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:55.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:55.400959+0000) 2020-02-05T20:33:55.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:55.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:55.770397+0000) 2020-02-05T20:33:55.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:55.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:55.807504+0000) 2020-02-05T20:33:56.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:56.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:56.666979+0000) 2020-02-05T20:33:56.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:56.909+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:56.909461+0000) 2020-02-05T20:33:59.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:59.303+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.304373+0000) 2020-02-05T20:33:59.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:59.695+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.696408+0000) 2020-02-05T20:33:59.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:33:59.695+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.696724+0000) 2020-02-05T20:33:59.699 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:33:59.696+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.697175+0000) 2020-02-05T20:33:59.699 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:33:59.697+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.697815+0000) 2020-02-05T20:33:59.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:33:59.696+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.697577+0000) 2020-02-05T20:33:59.700 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:33:59.697+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.698359+0000) 2020-02-05T20:33:59.701 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:33:59.697+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.698361+0000) 2020-02-05T20:33:59.701 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:59.697+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.698796+0000) 2020-02-05T20:33:59.702 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:33:59.699+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.699577+0000) 2020-02-05T20:33:59.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:33:59.698+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.699861+0000) 2020-02-05T20:33:59.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:33:59.699+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.700263+0000) 2020-02-05T20:33:59.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:59.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.700547+0000) 2020-02-05T20:33:59.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:33:59.700+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.700942+0000) 2020-02-05T20:33:59.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:33:59.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60254 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:33:59.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:33:59.704+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.705843+0000) 2020-02-05T20:33:59.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:33:59.913+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:33:59.914577+0000) 2020-02-05T20:34:00.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:00.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:00.161659+0000) 2020-02-05T20:34:00.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:00.161+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:00.161700+0000) 2020-02-05T20:34:00.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:00.174+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:00.175035+0000) 2020-02-05T20:34:02.269 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:02.266+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:02.267550+0000) 2020-02-05T20:34:03.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:03.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:03.394245+0000) 2020-02-05T20:34:03.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:03.425+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:03.425377+0000) 2020-02-05T20:34:04.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:04.310+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.311198+0000) 2020-02-05T20:34:04.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:04.707+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.708728+0000) 2020-02-05T20:34:04.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:04.706+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.707961+0000) 2020-02-05T20:34:04.712 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:04.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.710604+0000) 2020-02-05T20:34:04.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:04.710+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.711538+0000) 2020-02-05T20:34:04.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:04.711+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.712823+0000) 2020-02-05T20:34:04.715 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:04.712+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.713127+0000) 2020-02-05T20:34:04.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:04.712+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.713478+0000) 2020-02-05T20:34:04.715 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:04.712+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.713680+0000) 2020-02-05T20:34:04.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:04.712+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.714244+0000) 2020-02-05T20:34:04.717 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:04.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.714226+0000) 2020-02-05T20:34:04.717 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:04.713+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.714933+0000) 2020-02-05T20:34:04.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:04.712+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.713970+0000) 2020-02-05T20:34:04.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:04.714+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.715203+0000) 2020-02-05T20:34:04.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:04.714+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:04.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:04.720+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:04.721290+0000) 2020-02-05T20:34:05.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:05.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:05.401201+0000) 2020-02-05T20:34:05.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:05.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:05.770646+0000) 2020-02-05T20:34:05.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:05.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:05.807756+0000) 2020-02-05T20:34:06.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:06.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:06.667203+0000) 2020-02-05T20:34:06.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:06.908+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:06.909710+0000) 2020-02-05T20:34:09.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:09.304+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.304626+0000) 2020-02-05T20:34:09.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:09.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.723698+0000) 2020-02-05T20:34:09.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:09.723+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.724387+0000) 2020-02-05T20:34:09.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:09.724+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.725452+0000) 2020-02-05T20:34:09.732 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:09.724+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.725606+0000) 2020-02-05T20:34:09.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:09.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.726605+0000) 2020-02-05T20:34:09.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:09.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.727042+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:09.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.727433+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:09.721+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.722729+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:09.722+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.723762+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:09.724+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.726285+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:09.726+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.727456+0000) 2020-02-05T20:34:09.733 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:09.726+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.727941+0000) 2020-02-05T20:34:09.734 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:09.726+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.728344+0000) 2020-02-05T20:34:09.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:09.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:09.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:09.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.733173+0000) 2020-02-05T20:34:09.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:09.914+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:09.914820+0000) 2020-02-05T20:34:10.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:10.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:10.161904+0000) 2020-02-05T20:34:10.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:10.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:10.161919+0000) 2020-02-05T20:34:10.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:10.175+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:10.175269+0000) 2020-02-05T20:34:12.269 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:12.266+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:12.267808+0000) 2020-02-05T20:34:13.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:13.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:13.394498+0000) 2020-02-05T20:34:13.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:13.425+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:13.425615+0000) 2020-02-05T20:34:14.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:14.319+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.319695+0000) 2020-02-05T20:34:14.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:14.734+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.734788+0000) 2020-02-05T20:34:14.736 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:14.735+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.735586+0000) 2020-02-05T20:34:14.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:14.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.736501+0000) 2020-02-05T20:34:14.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:14.736+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.736883+0000) 2020-02-05T20:34:14.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:14.737+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.737251+0000) 2020-02-05T20:34:14.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:14.738+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.738241+0000) 2020-02-05T20:34:14.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:14.736+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.737880+0000) 2020-02-05T20:34:14.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:14.737+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.738893+0000) 2020-02-05T20:34:14.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:14.739+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.739674+0000) 2020-02-05T20:34:14.741 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:14.737+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.739299+0000) 2020-02-05T20:34:14.742 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:14.738+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.739893+0000) 2020-02-05T20:34:14.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:14.739+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.740883+0000) 2020-02-05T20:34:14.742 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:14.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.741407+0000) 2020-02-05T20:34:14.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:14.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:14.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:14.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:14.746008+0000) 2020-02-05T20:34:15.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:15.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:15.401457+0000) 2020-02-05T20:34:15.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:15.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:15.770885+0000) 2020-02-05T20:34:15.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:15.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:15.807989+0000) 2020-02-05T20:34:16.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:16.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:16.667437+0000) 2020-02-05T20:34:16.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:16.909+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:16.909961+0000) 2020-02-05T20:34:19.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:19.303+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.304857+0000) 2020-02-05T20:34:19.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:19.746+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.748293+0000) 2020-02-05T20:34:19.750 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:19.748+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.749197+0000) 2020-02-05T20:34:19.751 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:19.747+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.749374+0000) 2020-02-05T20:34:19.752 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:19.749+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.750725+0000) 2020-02-05T20:34:19.752 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:19.750+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.751161+0000) 2020-02-05T20:34:19.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:19.750+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.751313+0000) 2020-02-05T20:34:19.753 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:19.750+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.751836+0000) 2020-02-05T20:34:19.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:19.750+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.752010+0000) 2020-02-05T20:34:19.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:19.751+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.752363+0000) 2020-02-05T20:34:19.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:19.751+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.752991+0000) 2020-02-05T20:34:19.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:19.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.753508+0000) 2020-02-05T20:34:19.755 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:19.752+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.753909+0000) 2020-02-05T20:34:19.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:19.751+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.753022+0000) 2020-02-05T20:34:19.756 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:19.753+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60414 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:19.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:19.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.761989+0000) 2020-02-05T20:34:19.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:19.913+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:19.915020+0000) 2020-02-05T20:34:20.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:20.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:20.162142+0000) 2020-02-05T20:34:20.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:20.161+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:20.162142+0000) 2020-02-05T20:34:20.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:20.174+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:20.175478+0000) 2020-02-05T20:34:22.269 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:22.267+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:22.268043+0000) 2020-02-05T20:34:23.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:23.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:23.394746+0000) 2020-02-05T20:34:23.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:23.424+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:23.425842+0000) 2020-02-05T20:34:24.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:24.327+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.328285+0000) 2020-02-05T20:34:24.451 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:34:24.458 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:34:24.761 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:24.758+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.759646+0000) 2020-02-05T20:34:24.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:24.759+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.760718+0000) 2020-02-05T20:34:24.763 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:24.760+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.762102+0000) 2020-02-05T20:34:24.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:24.761+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.762268+0000) 2020-02-05T20:34:24.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:24.761+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.762836+0000) 2020-02-05T20:34:24.765 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:24.762+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.763836+0000) 2020-02-05T20:34:24.765 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:24.762+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.763568+0000) 2020-02-05T20:34:24.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:24.763+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.765005+0000) 2020-02-05T20:34:24.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:24.764+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.765139+0000) 2020-02-05T20:34:24.767 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:24.764+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.765659+0000) 2020-02-05T20:34:24.768 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:24.765+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.766427+0000) 2020-02-05T20:34:24.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:24.764+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.766237+0000) 2020-02-05T20:34:24.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:24.765+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.766455+0000) 2020-02-05T20:34:24.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:24.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:24.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:24.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:24.770386+0000) 2020-02-05T20:34:25.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:25.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:25.401695+0000) 2020-02-05T20:34:25.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:25.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:25.771119+0000) 2020-02-05T20:34:25.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:25.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:25.808267+0000) 2020-02-05T20:34:26.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:26.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:26.667700+0000) 2020-02-05T20:34:26.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:26.909+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:26.910192+0000) 2020-02-05T20:34:29.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:29.304+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.305123+0000) 2020-02-05T20:34:29.775 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:29.772+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.773469+0000) 2020-02-05T20:34:29.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:29.773+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.774528+0000) 2020-02-05T20:34:29.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:29.774+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.775987+0000) 2020-02-05T20:34:29.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:29.775+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.776516+0000) 2020-02-05T20:34:29.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:29.775+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.777037+0000) 2020-02-05T20:34:29.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:29.776+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.777768+0000) 2020-02-05T20:34:29.779 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:29.776+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.778008+0000) 2020-02-05T20:34:29.780 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:29.776+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.778273+0000) 2020-02-05T20:34:29.781 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:29.778+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.779529+0000) 2020-02-05T20:34:29.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:29.778+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.779937+0000) 2020-02-05T20:34:29.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:29.778+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.780054+0000) 2020-02-05T20:34:29.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:29.778+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.780265+0000) 2020-02-05T20:34:29.782 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:29.779+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:29.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:29.780+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.781888+0000) 2020-02-05T20:34:29.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:29.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.783115+0000) 2020-02-05T20:34:29.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:29.914+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:29.915249+0000) 2020-02-05T20:34:30.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:30.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:30.162377+0000) 2020-02-05T20:34:30.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:30.161+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:30.162382+0000) 2020-02-05T20:34:30.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:30.174+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:30.175730+0000) 2020-02-05T20:34:32.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:32.267+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:32.268323+0000) 2020-02-05T20:34:33.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:33.394+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:33.394991+0000) 2020-02-05T20:34:33.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:33.425+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:33.426039+0000) 2020-02-05T20:34:34.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:34.335+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.336531+0000) 2020-02-05T20:34:34.787 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:34.784+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.785598+0000) 2020-02-05T20:34:34.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:34.786+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.787406+0000) 2020-02-05T20:34:34.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:34.787+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.788394+0000) 2020-02-05T20:34:34.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:34.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.788964+0000) 2020-02-05T20:34:34.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:34.787+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.789395+0000) 2020-02-05T20:34:34.791 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:34.788+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.789841+0000) 2020-02-05T20:34:34.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:34.790+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.791150+0000) 2020-02-05T20:34:34.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:34.790+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.791605+0000) 2020-02-05T20:34:34.793 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:34.790+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.792057+0000) 2020-02-05T20:34:34.794 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:34.791+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.792474+0000) 2020-02-05T20:34:34.795 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:34.792+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.793204+0000) 2020-02-05T20:34:34.795 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:34.791+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.793273+0000) 2020-02-05T20:34:34.795 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:34.791+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.793375+0000) 2020-02-05T20:34:34.795 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:34.792+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:34.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:34.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:34.799471+0000) 2020-02-05T20:34:35.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:35.401+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:35.401942+0000) 2020-02-05T20:34:35.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:35.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:35.771383+0000) 2020-02-05T20:34:35.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:35.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:35.808506+0000) 2020-02-05T20:34:36.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:36.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:36.667962+0000) 2020-02-05T20:34:36.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:36.909+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:36.910459+0000) 2020-02-05T20:34:39.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:39.304+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.305361+0000) 2020-02-05T20:34:39.804 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:39.802+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.803136+0000) 2020-02-05T20:34:39.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:39.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.803569+0000) 2020-02-05T20:34:39.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:39.802+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.803764+0000) 2020-02-05T20:34:39.806 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:39.802+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.804257+0000) 2020-02-05T20:34:39.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:39.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.804806+0000) 2020-02-05T20:34:39.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:39.803+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.804707+0000) 2020-02-05T20:34:39.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:39.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.805633+0000) 2020-02-05T20:34:39.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:39.804+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.805827+0000) 2020-02-05T20:34:39.808 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:39.804+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.805597+0000) 2020-02-05T20:34:39.808 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:39.804+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.806067+0000) 2020-02-05T20:34:39.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:39.804+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.806202+0000) 2020-02-05T20:34:39.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:39.805+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.806588+0000) 2020-02-05T20:34:39.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:39.806+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:39.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:39.806+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.807839+0000) 2020-02-05T20:34:39.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:39.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.809039+0000) 2020-02-05T20:34:39.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:39.914+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:39.915456+0000) 2020-02-05T20:34:40.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:40.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:40.162643+0000) 2020-02-05T20:34:40.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:40.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:40.162651+0000) 2020-02-05T20:34:40.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:40.175+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:40.175974+0000) 2020-02-05T20:34:42.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:42.267+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:42.268577+0000) 2020-02-05T20:34:43.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:43.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:43.395227+0000) 2020-02-05T20:34:43.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:43.425+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:43.426357+0000) 2020-02-05T20:34:44.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:44.343+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.344876+0000) 2020-02-05T20:34:44.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:44.812+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.813685+0000) 2020-02-05T20:34:44.819 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:44.814+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.816400+0000) 2020-02-05T20:34:44.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:44.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.813877+0000) 2020-02-05T20:34:44.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:44.813+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.814229+0000) 2020-02-05T20:34:44.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:44.815+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.816111+0000) 2020-02-05T20:34:44.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:44.815+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.816674+0000) 2020-02-05T20:34:44.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:44.815+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.817006+0000) 2020-02-05T20:34:44.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:44.815+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.817350+0000) 2020-02-05T20:34:44.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:44.816+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.817980+0000) 2020-02-05T20:34:44.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:44.816+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.817774+0000) 2020-02-05T20:34:44.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:44.817+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.818977+0000) 2020-02-05T20:34:44.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:44.817+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.818454+0000) 2020-02-05T20:34:44.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:44.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.819518+0000) 2020-02-05T20:34:44.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:44.819+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60614 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:44.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:44.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:44.825475+0000) 2020-02-05T20:34:45.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:45.401+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:45.402164+0000) 2020-02-05T20:34:45.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:45.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:45.771591+0000) 2020-02-05T20:34:45.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:45.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:45.808690+0000) 2020-02-05T20:34:46.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:46.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:46.668191+0000) 2020-02-05T20:34:46.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:46.910+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:46.910725+0000) 2020-02-05T20:34:49.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:49.305+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.305594+0000) 2020-02-05T20:34:49.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:49.824+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.825987+0000) 2020-02-05T20:34:49.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:49.827+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.828095+0000) 2020-02-05T20:34:49.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:49.827+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.828564+0000) 2020-02-05T20:34:49.830 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:49.827+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.829240+0000) 2020-02-05T20:34:49.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:49.828+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.829656+0000) 2020-02-05T20:34:49.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:49.828+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.829698+0000) 2020-02-05T20:34:49.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:49.829+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.830168+0000) 2020-02-05T20:34:49.832 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:49.828+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.830373+0000) 2020-02-05T20:34:49.833 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:49.830+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.831458+0000) 2020-02-05T20:34:49.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:49.830+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.831829+0000) 2020-02-05T20:34:49.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:49.830+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.832414+0000) 2020-02-05T20:34:49.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:49.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:49.834 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:49.831+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.832846+0000) 2020-02-05T20:34:49.835 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:49.832+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.833540+0000) 2020-02-05T20:34:49.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:49.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.837518+0000) 2020-02-05T20:34:49.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:49.914+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:49.915617+0000) 2020-02-05T20:34:50.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:50.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:50.162880+0000) 2020-02-05T20:34:50.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:50.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:50.162870+0000) 2020-02-05T20:34:50.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:50.175+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:50.176176+0000) 2020-02-05T20:34:52.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:52.268+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:52.268830+0000) 2020-02-05T20:34:53.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:53.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:53.395497+0000) 2020-02-05T20:34:53.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:53.425+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:53.426552+0000) 2020-02-05T20:34:54.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:54.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.353141+0000) 2020-02-05T20:34:54.673 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:34:54.679 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:34:54.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:54.836+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.837985+0000) 2020-02-05T20:34:54.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:54.838+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.838757+0000) 2020-02-05T20:34:54.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:54.838+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.839478+0000) 2020-02-05T20:34:54.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:54.839+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.840167+0000) 2020-02-05T20:34:54.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:54.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.840371+0000) 2020-02-05T20:34:54.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:54.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.840493+0000) 2020-02-05T20:34:54.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:54.840+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.840783+0000) 2020-02-05T20:34:54.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:54.840+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.841424+0000) 2020-02-05T20:34:54.843 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:54.841+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.841859+0000) 2020-02-05T20:34:54.844 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:54.842+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.842450+0000) 2020-02-05T20:34:54.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:54.841+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.842286+0000) 2020-02-05T20:34:54.844 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:54.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.842727+0000) 2020-02-05T20:34:54.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:54.842+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.842885+0000) 2020-02-05T20:34:54.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:54.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:54.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:54.848+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:54.849246+0000) 2020-02-05T20:34:55.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:55.401+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:55.402400+0000) 2020-02-05T20:34:55.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:55.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:55.771795+0000) 2020-02-05T20:34:55.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:55.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:55.808894+0000) 2020-02-05T20:34:56.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:56.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:56.668414+0000) 2020-02-05T20:34:56.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:56.910+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:56.910960+0000) 2020-02-05T20:34:59.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:59.305+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.305830+0000) 2020-02-05T20:34:59.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:34:59.849+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.850221+0000) 2020-02-05T20:34:59.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:34:59.850+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.851212+0000) 2020-02-05T20:34:59.853 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:34:59.850+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.851467+0000) 2020-02-05T20:34:59.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:59.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.851917+0000) 2020-02-05T20:34:59.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:34:59.850+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.851973+0000) 2020-02-05T20:34:59.854 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:59.851+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.852330+0000) 2020-02-05T20:34:59.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:34:59.852+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.852890+0000) 2020-02-05T20:34:59.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:34:59.852+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.852814+0000) 2020-02-05T20:34:59.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:34:59.853+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.853908+0000) 2020-02-05T20:34:59.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:59.854+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.854851+0000) 2020-02-05T20:34:59.857 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:34:59.855+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.855642+0000) 2020-02-05T20:34:59.857 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:34:59.855+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.856079+0000) 2020-02-05T20:34:59.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:34:59.855+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.856451+0000) 2020-02-05T20:34:59.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:34:59.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:34:59.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:34:59.860+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.861757+0000) 2020-02-05T20:34:59.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:34:59.915+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:34:59.915806+0000) 2020-02-05T20:35:00.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:00.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:00.163110+0000) 2020-02-05T20:35:00.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:00.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:00.163107+0000) 2020-02-05T20:35:00.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:00.176+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:00.176422+0000) 2020-02-05T20:35:02.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:02.268+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:02.269071+0000) 2020-02-05T20:35:03.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:03.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:03.395775+0000) 2020-02-05T20:35:03.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:03.426+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:03.426808+0000) 2020-02-05T20:35:04.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:04.361+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.361247+0000) 2020-02-05T20:35:04.864 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:04.861+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.862880+0000) 2020-02-05T20:35:04.865 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:04.862+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.863719+0000) 2020-02-05T20:35:04.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:04.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.864670+0000) 2020-02-05T20:35:04.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:04.863+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.864851+0000) 2020-02-05T20:35:04.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:04.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.865212+0000) 2020-02-05T20:35:04.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:04.864+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.865628+0000) 2020-02-05T20:35:04.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:04.865+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.866357+0000) 2020-02-05T20:35:04.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:04.865+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.866276+0000) 2020-02-05T20:35:04.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:04.866+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.866810+0000) 2020-02-05T20:35:04.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:04.866+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.867396+0000) 2020-02-05T20:35:04.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:04.867+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.867796+0000) 2020-02-05T20:35:04.870 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:04.868+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.868846+0000) 2020-02-05T20:35:04.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:04.868+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.868733+0000) 2020-02-05T20:35:04.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:04.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:04.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:04.875+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:04.876127+0000) 2020-02-05T20:35:05.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:05.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:05.402656+0000) 2020-02-05T20:35:05.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:05.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:05.772039+0000) 2020-02-05T20:35:05.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:05.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:05.809133+0000) 2020-02-05T20:35:06.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:06.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:06.668645+0000) 2020-02-05T20:35:06.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:06.910+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:06.911210+0000) 2020-02-05T20:35:09.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:09.306+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.306071+0000) 2020-02-05T20:35:09.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:09.874+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.874132+0000) 2020-02-05T20:35:09.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:09.875+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.875905+0000) 2020-02-05T20:35:09.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:09.876+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.876676+0000) 2020-02-05T20:35:09.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:09.877+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.877764+0000) 2020-02-05T20:35:09.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:09.877+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.877764+0000) 2020-02-05T20:35:09.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:09.878+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.879243+0000) 2020-02-05T20:35:09.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:09.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.879349+0000) 2020-02-05T20:35:09.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:09.879+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.879765+0000) 2020-02-05T20:35:09.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:09.879+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.880012+0000) 2020-02-05T20:35:09.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:09.880+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.880497+0000) 2020-02-05T20:35:09.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:09.880+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.880749+0000) 2020-02-05T20:35:09.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:09.881+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.881688+0000) 2020-02-05T20:35:09.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:09.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.883856+0000) 2020-02-05T20:35:09.885 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:09.881+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.882290+0000) 2020-02-05T20:35:09.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:09.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60814 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:09.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:09.915+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:09.916020+0000) 2020-02-05T20:35:10.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:10.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:10.163341+0000) 2020-02-05T20:35:10.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:10.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:10.163343+0000) 2020-02-05T20:35:10.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:10.176+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:10.176628+0000) 2020-02-05T20:35:12.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:12.269+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:12.269316+0000) 2020-02-05T20:35:13.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:13.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:13.396021+0000) 2020-02-05T20:35:13.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:13.426+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:13.427131+0000) 2020-02-05T20:35:14.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:14.368+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.369604+0000) 2020-02-05T20:35:14.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:14.887+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.888310+0000) 2020-02-05T20:35:14.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:14.888+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.889143+0000) 2020-02-05T20:35:14.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:14.889+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.889502+0000) 2020-02-05T20:35:14.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:14.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.889734+0000) 2020-02-05T20:35:14.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:14.889+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.890581+0000) 2020-02-05T20:35:14.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:14.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.890866+0000) 2020-02-05T20:35:14.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:14.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.891438+0000) 2020-02-05T20:35:14.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:14.891+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.891507+0000) 2020-02-05T20:35:14.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:14.891+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.892075+0000) 2020-02-05T20:35:14.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:14.892+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.893241+0000) 2020-02-05T20:35:14.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:14.892+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.893794+0000) 2020-02-05T20:35:14.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:14.893+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.894009+0000) 2020-02-05T20:35:14.896 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:14.893+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.894323+0000) 2020-02-05T20:35:14.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:14.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:14.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:14.897+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:14.898798+0000) 2020-02-05T20:35:15.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:15.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:15.402872+0000) 2020-02-05T20:35:15.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:15.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:15.772229+0000) 2020-02-05T20:35:15.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:15.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:15.809342+0000) 2020-02-05T20:35:16.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:16.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:16.668916+0000) 2020-02-05T20:35:16.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:16.911+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:16.911491+0000) 2020-02-05T20:35:19.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:19.305+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.306343+0000) 2020-02-05T20:35:19.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:19.903+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.904208+0000) 2020-02-05T20:35:19.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:19.904+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.904759+0000) 2020-02-05T20:35:19.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:19.904+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.905483+0000) 2020-02-05T20:35:19.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:19.904+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.905974+0000) 2020-02-05T20:35:19.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:19.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.906568+0000) 2020-02-05T20:35:19.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:19.905+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.906832+0000) 2020-02-05T20:35:19.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:19.906+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.907115+0000) 2020-02-05T20:35:19.909 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:19.907+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.907847+0000) 2020-02-05T20:35:19.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:19.908+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.908548+0000) 2020-02-05T20:35:19.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:19.908+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.908709+0000) 2020-02-05T20:35:19.910 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:19.908+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.909142+0000) 2020-02-05T20:35:19.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:19.908+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.909363+0000) 2020-02-05T20:35:19.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:19.909+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.909527+0000) 2020-02-05T20:35:19.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:19.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:19.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:19.915+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.916119+0000) 2020-02-05T20:35:19.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:19.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:19.916888+0000) 2020-02-05T20:35:20.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:20.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:20.163521+0000) 2020-02-05T20:35:20.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:20.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:20.163640+0000) 2020-02-05T20:35:20.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:20.175+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:20.176865+0000) 2020-02-05T20:35:22.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:22.269+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:22.269557+0000) 2020-02-05T20:35:23.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:23.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:23.396273+0000) 2020-02-05T20:35:23.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:23.426+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:23.427402+0000) 2020-02-05T20:35:24.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:24.376+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.377808+0000) 2020-02-05T20:35:24.852 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:35:24.858 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:35:24.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:24.915+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.916392+0000) 2020-02-05T20:35:24.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:24.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.917027+0000) 2020-02-05T20:35:24.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:24.917+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.917485+0000) 2020-02-05T20:35:24.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:24.917+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.918336+0000) 2020-02-05T20:35:24.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:24.917+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.918676+0000) 2020-02-05T20:35:24.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:24.917+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.918950+0000) 2020-02-05T20:35:24.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:24.918+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.919184+0000) 2020-02-05T20:35:24.921 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:24.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.919200+0000) 2020-02-05T20:35:24.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:24.919+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.920318+0000) 2020-02-05T20:35:24.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:24.919+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.919916+0000) 2020-02-05T20:35:24.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:24.920+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.920594+0000) 2020-02-05T20:35:24.923 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:24.919+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.920344+0000) 2020-02-05T20:35:24.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:24.920+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.920925+0000) 2020-02-05T20:35:24.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:24.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:24.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:24.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:24.926577+0000) 2020-02-05T20:35:25.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:25.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:25.403150+0000) 2020-02-05T20:35:25.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:25.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:25.772433+0000) 2020-02-05T20:35:25.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:25.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:25.809550+0000) 2020-02-05T20:35:26.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:26.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:26.669168+0000) 2020-02-05T20:35:26.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:26.911+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:26.911801+0000) 2020-02-05T20:35:29.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:29.305+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.306602+0000) 2020-02-05T20:35:29.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:29.915+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.916269+0000) 2020-02-05T20:35:29.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:29.925+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.926423+0000) 2020-02-05T20:35:29.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:29.927+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.927918+0000) 2020-02-05T20:35:29.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:29.927+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.928533+0000) 2020-02-05T20:35:29.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:29.927+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.928659+0000) 2020-02-05T20:35:29.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:29.928+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.928954+0000) 2020-02-05T20:35:29.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:29.928+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.929961+0000) 2020-02-05T20:35:29.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:29.929+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.929949+0000) 2020-02-05T20:35:29.932 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:29.929+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.930005+0000) 2020-02-05T20:35:29.932 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:29.929+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.930453+0000) 2020-02-05T20:35:29.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:29.930+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.930817+0000) 2020-02-05T20:35:29.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:29.931+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.931843+0000) 2020-02-05T20:35:29.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:29.930+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.931167+0000) 2020-02-05T20:35:29.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:29.930+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.931597+0000) 2020-02-05T20:35:29.936 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:29.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 60974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:29.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:29.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:29.938245+0000) 2020-02-05T20:35:30.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:30.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:30.163764+0000) 2020-02-05T20:35:30.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:30.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:30.163866+0000) 2020-02-05T20:35:30.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:30.176+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:30.177173+0000) 2020-02-05T20:35:32.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:32.268+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:32.269849+0000) 2020-02-05T20:35:33.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:33.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:33.396507+0000) 2020-02-05T20:35:33.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:33.426+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:33.427632+0000) 2020-02-05T20:35:34.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:34.385+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.386054+0000) 2020-02-05T20:35:34.942 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:34.940+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.941122+0000) 2020-02-05T20:35:34.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:34.941+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.942263+0000) 2020-02-05T20:35:34.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:34.941+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.942661+0000) 2020-02-05T20:35:34.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:34.942+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.943159+0000) 2020-02-05T20:35:34.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:34.942+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.944129+0000) 2020-02-05T20:35:34.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:34.943+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.944448+0000) 2020-02-05T20:35:34.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:34.943+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.944840+0000) 2020-02-05T20:35:34.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:34.944+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.945869+0000) 2020-02-05T20:35:34.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:34.944+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.945782+0000) 2020-02-05T20:35:34.948 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:34.945+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.946488+0000) 2020-02-05T20:35:34.948 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:34.944+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.946248+0000) 2020-02-05T20:35:34.948 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:34.945+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.946913+0000) 2020-02-05T20:35:34.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:34.946+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.947193+0000) 2020-02-05T20:35:34.950 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:34.947+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61015 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:34.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:34.952+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:34.953498+0000) 2020-02-05T20:35:35.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:35.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:35.403373+0000) 2020-02-05T20:35:35.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:35.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:35.772660+0000) 2020-02-05T20:35:35.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:35.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:35.809758+0000) 2020-02-05T20:35:36.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:36.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:36.669341+0000) 2020-02-05T20:35:36.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:36.911+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:36.912053+0000) 2020-02-05T20:35:39.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:39.306+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.306836+0000) 2020-02-05T20:35:39.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:39.915+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.916523+0000) 2020-02-05T20:35:39.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:39.952+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.953725+0000) 2020-02-05T20:35:39.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:39.953+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.954704+0000) 2020-02-05T20:35:39.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:39.954+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.955634+0000) 2020-02-05T20:35:39.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:39.954+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.956159+0000) 2020-02-05T20:35:39.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:39.956+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.957253+0000) 2020-02-05T20:35:39.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:39.956+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.957606+0000) 2020-02-05T20:35:39.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:39.956+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.957979+0000) 2020-02-05T20:35:39.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:39.957+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.958290+0000) 2020-02-05T20:35:39.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:39.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.958939+0000) 2020-02-05T20:35:39.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:39.958+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.959775+0000) 2020-02-05T20:35:39.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:39.958+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.960062+0000) 2020-02-05T20:35:39.962 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:39.959+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.960454+0000) 2020-02-05T20:35:39.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:39.958+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.960331+0000) 2020-02-05T20:35:39.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:39.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:39.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:39.964+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:39.965585+0000) 2020-02-05T20:35:40.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:40.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:40.163972+0000) 2020-02-05T20:35:40.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:40.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:40.164155+0000) 2020-02-05T20:35:40.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:40.176+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:40.177423+0000) 2020-02-05T20:35:42.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:42.269+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:42.270107+0000) 2020-02-05T20:35:43.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:43.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:43.396754+0000) 2020-02-05T20:35:43.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:43.426+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:43.427885+0000) 2020-02-05T20:35:44.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:44.396+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.397329+0000) 2020-02-05T20:35:44.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:44.965+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.966829+0000) 2020-02-05T20:35:44.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:44.966+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.967403+0000) 2020-02-05T20:35:44.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:44.967+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.969263+0000) 2020-02-05T20:35:44.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:44.967+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.968400+0000) 2020-02-05T20:35:44.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:44.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.969398+0000) 2020-02-05T20:35:44.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:44.969+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.970562+0000) 2020-02-05T20:35:44.973 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:44.969+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.970120+0000) 2020-02-05T20:35:44.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:44.969+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.970902+0000) 2020-02-05T20:35:44.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:44.969+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.971250+0000) 2020-02-05T20:35:44.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:44.971+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.972542+0000) 2020-02-05T20:35:44.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:44.971+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.973124+0000) 2020-02-05T20:35:44.975 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:44.972+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.973740+0000) 2020-02-05T20:35:44.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:44.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.973887+0000) 2020-02-05T20:35:44.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:44.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:44.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:44.977+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:44.978139+0000) 2020-02-05T20:35:45.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:45.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:45.403620+0000) 2020-02-05T20:35:45.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:45.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:45.772892+0000) 2020-02-05T20:35:45.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:45.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:45.809967+0000) 2020-02-05T20:35:46.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:46.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:46.669579+0000) 2020-02-05T20:35:46.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:46.911+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:46.912314+0000) 2020-02-05T20:35:49.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:49.306+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.307121+0000) 2020-02-05T20:35:49.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:49.916+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.916773+0000) 2020-02-05T20:35:49.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:49.978+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.980041+0000) 2020-02-05T20:35:49.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:49.979+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.980569+0000) 2020-02-05T20:35:49.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:49.980+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.981247+0000) 2020-02-05T20:35:49.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:49.981+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.982710+0000) 2020-02-05T20:35:49.985 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:49.980+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.982329+0000) 2020-02-05T20:35:49.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:49.982+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.983432+0000) 2020-02-05T20:35:49.986 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:49.982+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.983496+0000) 2020-02-05T20:35:49.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:49.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.984245+0000) 2020-02-05T20:35:49.988 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:49.983+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.984989+0000) 2020-02-05T20:35:49.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:49.983+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.984760+0000) 2020-02-05T20:35:49.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:49.984+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.985722+0000) 2020-02-05T20:35:49.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:49.985+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.986480+0000) 2020-02-05T20:35:49.988 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:49.984+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.986086+0000) 2020-02-05T20:35:49.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:49.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:49.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:49.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:49.992275+0000) 2020-02-05T20:35:50.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:50.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:50.164255+0000) 2020-02-05T20:35:50.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:50.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:50.164378+0000) 2020-02-05T20:35:50.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:50.177+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:50.177638+0000) 2020-02-05T20:35:52.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:52.269+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:52.270367+0000) 2020-02-05T20:35:53.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:53.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:53.397005+0000) 2020-02-05T20:35:53.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:53.427+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:53.428115+0000) 2020-02-05T20:35:54.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:54.405+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.405786+0000) 2020-02-05T20:35:54.981 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:35:54.987 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:35:54.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:54.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.994699+0000) 2020-02-05T20:35:54.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:54.994+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.995670+0000) 2020-02-05T20:35:54.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:54.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.995152+0000) 2020-02-05T20:35:54.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:54.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.996002+0000) 2020-02-05T20:35:54.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:35:54.995+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.996131+0000) 2020-02-05T20:35:54.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:35:54.995+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.996462+0000) 2020-02-05T20:35:54.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:54.995+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.996692+0000) 2020-02-05T20:35:54.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:35:54.994+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.995662+0000) 2020-02-05T20:35:55.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:54.995+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.996986+0000) 2020-02-05T20:35:55.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:35:54.996+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.997832+0000) 2020-02-05T20:35:55.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:54.996+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.998082+0000) 2020-02-05T20:35:55.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:35:54.997+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.999082+0000) 2020-02-05T20:35:55.001 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:35:54.995+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:54.997351+0000) 2020-02-05T20:35:55.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:35:54.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61175 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:35:55.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:55.000+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:55.001611+0000) 2020-02-05T20:35:55.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:35:55.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:55.403921+0000) 2020-02-05T20:35:55.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:35:55.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:55.773132+0000) 2020-02-05T20:35:55.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:55.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:55.810166+0000) 2020-02-05T20:35:56.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:35:56.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:56.669849+0000) 2020-02-05T20:35:56.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:35:56.911+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:56.912572+0000) 2020-02-05T20:35:59.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:35:59.307+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:59.307350+0000) 2020-02-05T20:35:59.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:35:59.916+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:35:59.917012+0000) 2020-02-05T20:36:00.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:00.005+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.006143+0000) 2020-02-05T20:36:00.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:00.005+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.006567+0000) 2020-02-05T20:36:00.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:00.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.007131+0000) 2020-02-05T20:36:00.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:00.006+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.007611+0000) 2020-02-05T20:36:00.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:00.006+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.007947+0000) 2020-02-05T20:36:00.010 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:00.007+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.008304+0000) 2020-02-05T20:36:00.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:00.007+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.008742+0000) 2020-02-05T20:36:00.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:00.007+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.009364+0000) 2020-02-05T20:36:00.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:00.009+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.010245+0000) 2020-02-05T20:36:00.012 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:00.009+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.010798+0000) 2020-02-05T20:36:00.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:00.009+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.011177+0000) 2020-02-05T20:36:00.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:00.009+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.011321+0000) 2020-02-05T20:36:00.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:00.010+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.011501+0000) 2020-02-05T20:36:00.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:00.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:00.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:00.015+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.016882+0000) 2020-02-05T20:36:00.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:00.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.164538+0000) 2020-02-05T20:36:00.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:00.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.164649+0000) 2020-02-05T20:36:00.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:00.176+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:00.177882+0000) 2020-02-05T20:36:02.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:02.269+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:02.270663+0000) 2020-02-05T20:36:03.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:03.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:03.397241+0000) 2020-02-05T20:36:03.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:03.428+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:03.428401+0000) 2020-02-05T20:36:04.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:04.414+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:04.414380+0000) 2020-02-05T20:36:05.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:05.016+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.018109+0000) 2020-02-05T20:36:05.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:05.018+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.019581+0000) 2020-02-05T20:36:05.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:05.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.020149+0000) 2020-02-05T20:36:05.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:05.018+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.020228+0000) 2020-02-05T20:36:05.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:05.020+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.020839+0000) 2020-02-05T20:36:05.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:05.020+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.021677+0000) 2020-02-05T20:36:05.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:05.021+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.022040+0000) 2020-02-05T20:36:05.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:05.021+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.022478+0000) 2020-02-05T20:36:05.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:05.022+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.022626+0000) 2020-02-05T20:36:05.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:05.023+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.023645+0000) 2020-02-05T20:36:05.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:05.022+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.023714+0000) 2020-02-05T20:36:05.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:05.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.024114+0000) 2020-02-05T20:36:05.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:05.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:05.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:05.027+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.027779+0000) 2020-02-05T20:36:05.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:05.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.028263+0000) 2020-02-05T20:36:05.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:05.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.404160+0000) 2020-02-05T20:36:05.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:05.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.773367+0000) 2020-02-05T20:36:05.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:05.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:05.810379+0000) 2020-02-05T20:36:06.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:06.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:06.670113+0000) 2020-02-05T20:36:06.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:06.912+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:06.912828+0000) 2020-02-05T20:36:09.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:09.306+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:09.307578+0000) 2020-02-05T20:36:09.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:09.916+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:09.917271+0000) 2020-02-05T20:36:10.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:10.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.030954+0000) 2020-02-05T20:36:10.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:10.031+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.032875+0000) 2020-02-05T20:36:10.034 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:10.032+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.033496+0000) 2020-02-05T20:36:10.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:10.033+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.034410+0000) 2020-02-05T20:36:10.036 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:10.033+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.034879+0000) 2020-02-05T20:36:10.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:10.033+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.035370+0000) 2020-02-05T20:36:10.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:10.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.035613+0000) 2020-02-05T20:36:10.037 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:10.034+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.035675+0000) 2020-02-05T20:36:10.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:10.034+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.035940+0000) 2020-02-05T20:36:10.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:10.035+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.036545+0000) 2020-02-05T20:36:10.038 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:10.035+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.037112+0000) 2020-02-05T20:36:10.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:10.035+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.037248+0000) 2020-02-05T20:36:10.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:10.036+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.037583+0000) 2020-02-05T20:36:10.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:10.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:10.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:10.040+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.041192+0000) 2020-02-05T20:36:10.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:10.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.164758+0000) 2020-02-05T20:36:10.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:10.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.164870+0000) 2020-02-05T20:36:10.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:10.177+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:10.178117+0000) 2020-02-05T20:36:12.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:12.270+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:12.270910+0000) 2020-02-05T20:36:13.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:13.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:13.397484+0000) 2020-02-05T20:36:13.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:13.427+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:13.428634+0000) 2020-02-05T20:36:14.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:14.425+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:14.426193+0000) 2020-02-05T20:36:15.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:15.043+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.045181+0000) 2020-02-05T20:36:15.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:15.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.046104+0000) 2020-02-05T20:36:15.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:15.045+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.046782+0000) 2020-02-05T20:36:15.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:15.045+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.046902+0000) 2020-02-05T20:36:15.049 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:15.046+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.047671+0000) 2020-02-05T20:36:15.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:15.047+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.048533+0000) 2020-02-05T20:36:15.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:15.046+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.048387+0000) 2020-02-05T20:36:15.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:15.047+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.048665+0000) 2020-02-05T20:36:15.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:15.047+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.049017+0000) 2020-02-05T20:36:15.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:15.048+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.049803+0000) 2020-02-05T20:36:15.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:15.049+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.050223+0000) 2020-02-05T20:36:15.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:15.049+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.050548+0000) 2020-02-05T20:36:15.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:15.048+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.050027+0000) 2020-02-05T20:36:15.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:15.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:15.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:15.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.057672+0000) 2020-02-05T20:36:15.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:15.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.404392+0000) 2020-02-05T20:36:15.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:15.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.773582+0000) 2020-02-05T20:36:15.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:15.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:15.810623+0000) 2020-02-05T20:36:16.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:16.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:16.670379+0000) 2020-02-05T20:36:16.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:16.912+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:16.913090+0000) 2020-02-05T20:36:19.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:19.306+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:19.307829+0000) 2020-02-05T20:36:19.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:19.916+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:19.917494+0000) 2020-02-05T20:36:20.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:20.057+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.058451+0000) 2020-02-05T20:36:20.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:20.058+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.059365+0000) 2020-02-05T20:36:20.061 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:20.058+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.060027+0000) 2020-02-05T20:36:20.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:20.058+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.060091+0000) 2020-02-05T20:36:20.062 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:20.059+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.060493+0000) 2020-02-05T20:36:20.062 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:20.059+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.060827+0000) 2020-02-05T20:36:20.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:20.060+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.061838+0000) 2020-02-05T20:36:20.063 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:20.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.061689+0000) 2020-02-05T20:36:20.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:20.061+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.062974+0000) 2020-02-05T20:36:20.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:20.061+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.063097+0000) 2020-02-05T20:36:20.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:20.062+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.063395+0000) 2020-02-05T20:36:20.066 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:20.063+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.064247+0000) 2020-02-05T20:36:20.066 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:20.062+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.064060+0000) 2020-02-05T20:36:20.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:20.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:20.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:20.068+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.069673+0000) 2020-02-05T20:36:20.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:20.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.164977+0000) 2020-02-05T20:36:20.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:20.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.165150+0000) 2020-02-05T20:36:20.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:20.177+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:20.178317+0000) 2020-02-05T20:36:22.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:22.270+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:22.271184+0000) 2020-02-05T20:36:23.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:23.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:23.397703+0000) 2020-02-05T20:36:23.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:23.427+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:23.428920+0000) 2020-02-05T20:36:24.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:24.433+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:24.434434+0000) 2020-02-05T20:36:25.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:25.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.069879+0000) 2020-02-05T20:36:25.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:25.071+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.072151+0000) 2020-02-05T20:36:25.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:25.071+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.072156+0000) 2020-02-05T20:36:25.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:25.072+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.072611+0000) 2020-02-05T20:36:25.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:25.072+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.073115+0000) 2020-02-05T20:36:25.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:25.073+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.074380+0000) 2020-02-05T20:36:25.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:25.073+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.074349+0000) 2020-02-05T20:36:25.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:25.073+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.074829+0000) 2020-02-05T20:36:25.077 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:25.074+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.075945+0000) 2020-02-05T20:36:25.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:25.075+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.076300+0000) 2020-02-05T20:36:25.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:25.076+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.076792+0000) 2020-02-05T20:36:25.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:25.076+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.077159+0000) 2020-02-05T20:36:25.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:25.075+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.076833+0000) 2020-02-05T20:36:25.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:25.077+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:25.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:25.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.078881+0000) 2020-02-05T20:36:25.180 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:36:25.186 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:36:25.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:25.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.404674+0000) 2020-02-05T20:36:25.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:25.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.773790+0000) 2020-02-05T20:36:25.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:25.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:25.810829+0000) 2020-02-05T20:36:26.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:26.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:26.670689+0000) 2020-02-05T20:36:26.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:26.912+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:26.913352+0000) 2020-02-05T20:36:29.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:29.307+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:29.308115+0000) 2020-02-05T20:36:29.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:29.917+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:29.917723+0000) 2020-02-05T20:36:30.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:30.084+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.085906+0000) 2020-02-05T20:36:30.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:30.086+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.087056+0000) 2020-02-05T20:36:30.089 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:30.086+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.087442+0000) 2020-02-05T20:36:30.089 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:30.086+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.087859+0000) 2020-02-05T20:36:30.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:30.087+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.088352+0000) 2020-02-05T20:36:30.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:30.088+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.088784+0000) 2020-02-05T20:36:30.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:30.088+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.089242+0000) 2020-02-05T20:36:30.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:30.088+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.089309+0000) 2020-02-05T20:36:30.091 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:30.087+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.088913+0000) 2020-02-05T20:36:30.092 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:30.089+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.089626+0000) 2020-02-05T20:36:30.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:30.089+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.089871+0000) 2020-02-05T20:36:30.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:30.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:30.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:30.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.092302+0000) 2020-02-05T20:36:30.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:30.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.092698+0000) 2020-02-05T20:36:30.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:30.092+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.093826+0000) 2020-02-05T20:36:30.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:30.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.165201+0000) 2020-02-05T20:36:30.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:30.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.165390+0000) 2020-02-05T20:36:30.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:30.177+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:30.178520+0000) 2020-02-05T20:36:32.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:32.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:32.271476+0000) 2020-02-05T20:36:33.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:33.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:33.397975+0000) 2020-02-05T20:36:33.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:33.428+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:33.429144+0000) 2020-02-05T20:36:34.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:34.441+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:34.442461+0000) 2020-02-05T20:36:35.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:35.096+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.097415+0000) 2020-02-05T20:36:35.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:35.098+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.098841+0000) 2020-02-05T20:36:35.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:35.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.099770+0000) 2020-02-05T20:36:35.101 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:35.099+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.100319+0000) 2020-02-05T20:36:35.102 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:35.099+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.100200+0000) 2020-02-05T20:36:35.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:35.100+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.100946+0000) 2020-02-05T20:36:35.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:35.100+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.101409+0000) 2020-02-05T20:36:35.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:35.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.101680+0000) 2020-02-05T20:36:35.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:35.101+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.101994+0000) 2020-02-05T20:36:35.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:35.101+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.102414+0000) 2020-02-05T20:36:35.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:35.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.102498+0000) 2020-02-05T20:36:35.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:35.102+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.102736+0000) 2020-02-05T20:36:35.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:35.102+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.103362+0000) 2020-02-05T20:36:35.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:35.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:35.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:35.107+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.108484+0000) 2020-02-05T20:36:35.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:35.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.404897+0000) 2020-02-05T20:36:35.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:35.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.774032+0000) 2020-02-05T20:36:35.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:35.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:35.811064+0000) 2020-02-05T20:36:36.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:36.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:36.670920+0000) 2020-02-05T20:36:36.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:36.913+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:36.913696+0000) 2020-02-05T20:36:39.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:39.307+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:39.308385+0000) 2020-02-05T20:36:39.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:39.917+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:39.917955+0000) 2020-02-05T20:36:40.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:40.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.110889+0000) 2020-02-05T20:36:40.113 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:40.110+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.111465+0000) 2020-02-05T20:36:40.113 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:40.111+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.112195+0000) 2020-02-05T20:36:40.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:40.111+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.112308+0000) 2020-02-05T20:36:40.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:40.111+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.112375+0000) 2020-02-05T20:36:40.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:40.112+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.112757+0000) 2020-02-05T20:36:40.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:40.113+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.113484+0000) 2020-02-05T20:36:40.115 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:40.112+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.113949+0000) 2020-02-05T20:36:40.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:40.113+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.114762+0000) 2020-02-05T20:36:40.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:40.114+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.114462+0000) 2020-02-05T20:36:40.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:40.115+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.115579+0000) 2020-02-05T20:36:40.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:40.115+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.115860+0000) 2020-02-05T20:36:40.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:40.116+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:40.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:40.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.120569+0000) 2020-02-05T20:36:40.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:40.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.120892+0000) 2020-02-05T20:36:40.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:40.165+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.165446+0000) 2020-02-05T20:36:40.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:40.165+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.165645+0000) 2020-02-05T20:36:40.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:40.178+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:40.178773+0000) 2020-02-05T20:36:42.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:42.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:42.271741+0000) 2020-02-05T20:36:43.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:43.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:43.398202+0000) 2020-02-05T20:36:43.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:43.428+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:43.429401+0000) 2020-02-05T20:36:44.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:44.449+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:44.451017+0000) 2020-02-05T20:36:45.125 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:45.122+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.123462+0000) 2020-02-05T20:36:45.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:45.122+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.123865+0000) 2020-02-05T20:36:45.126 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:45.123+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.124445+0000) 2020-02-05T20:36:45.126 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:45.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.125125+0000) 2020-02-05T20:36:45.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:45.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.125580+0000) 2020-02-05T20:36:45.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:45.126+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.126552+0000) 2020-02-05T20:36:45.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:45.126+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.126850+0000) 2020-02-05T20:36:45.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:45.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.127148+0000) 2020-02-05T20:36:45.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:45.126+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.127500+0000) 2020-02-05T20:36:45.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:45.127+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.128206+0000) 2020-02-05T20:36:45.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:45.127+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.128354+0000) 2020-02-05T20:36:45.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:45.128+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.129209+0000) 2020-02-05T20:36:45.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:45.129+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.129514+0000) 2020-02-05T20:36:45.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:45.129+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61574 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:45.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:45.131+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.132931+0000) 2020-02-05T20:36:45.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:45.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.405136+0000) 2020-02-05T20:36:45.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:45.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.774282+0000) 2020-02-05T20:36:45.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:45.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:45.811367+0000) 2020-02-05T20:36:46.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:46.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:46.671161+0000) 2020-02-05T20:36:46.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:46.913+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:46.913954+0000) 2020-02-05T20:36:49.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:49.308+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:49.308612+0000) 2020-02-05T20:36:49.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:49.917+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:49.918210+0000) 2020-02-05T20:36:50.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:50.134+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.135357+0000) 2020-02-05T20:36:50.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:50.134+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.135813+0000) 2020-02-05T20:36:50.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:50.136+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.137313+0000) 2020-02-05T20:36:50.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:50.137+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.137751+0000) 2020-02-05T20:36:50.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:50.137+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.138265+0000) 2020-02-05T20:36:50.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:50.137+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.138903+0000) 2020-02-05T20:36:50.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:50.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.139336+0000) 2020-02-05T20:36:50.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:50.138+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.139817+0000) 2020-02-05T20:36:50.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:50.139+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.140259+0000) 2020-02-05T20:36:50.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:50.139+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.140507+0000) 2020-02-05T20:36:50.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:50.141+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.141773+0000) 2020-02-05T20:36:50.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:50.141+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.142317+0000) 2020-02-05T20:36:50.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:50.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:50.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:50.139+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.140998+0000) 2020-02-05T20:36:50.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:50.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.147438+0000) 2020-02-05T20:36:50.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:50.165+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.165611+0000) 2020-02-05T20:36:50.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:50.165+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.165800+0000) 2020-02-05T20:36:50.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:50.178+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:50.178971+0000) 2020-02-05T20:36:52.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:52.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:52.271979+0000) 2020-02-05T20:36:53.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:53.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:53.398480+0000) 2020-02-05T20:36:53.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:53.429+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:53.429646+0000) 2020-02-05T20:36:54.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:54.461+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:54.461648+0000) 2020-02-05T20:36:55.148 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:36:55.146+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.147061+0000) 2020-02-05T20:36:55.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:55.147+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.148625+0000) 2020-02-05T20:36:55.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:55.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.149059+0000) 2020-02-05T20:36:55.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:55.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.149534+0000) 2020-02-05T20:36:55.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:36:55.149+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.150132+0000) 2020-02-05T20:36:55.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:55.150+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.150761+0000) 2020-02-05T20:36:55.153 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:36:55.151+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.151509+0000) 2020-02-05T20:36:55.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:36:55.150+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.152037+0000) 2020-02-05T20:36:55.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:55.151+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.152591+0000) 2020-02-05T20:36:55.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:55.152+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.153152+0000) 2020-02-05T20:36:55.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:55.153+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.153740+0000) 2020-02-05T20:36:55.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:36:55.154+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.154904+0000) 2020-02-05T20:36:55.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:36:55.154+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.154839+0000) 2020-02-05T20:36:55.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:36:55.155+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:36:55.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:55.160+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.161775+0000) 2020-02-05T20:36:55.360 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:36:55.366 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:36:55.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:36:55.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.405363+0000) 2020-02-05T20:36:55.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:36:55.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.774497+0000) 2020-02-05T20:36:55.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:55.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:55.811578+0000) 2020-02-05T20:36:56.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:36:56.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:56.671427+0000) 2020-02-05T20:36:56.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:36:56.914+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:56.914205+0000) 2020-02-05T20:36:59.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:36:59.308+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:59.308849+0000) 2020-02-05T20:36:59.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:36:59.918+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:36:59.918462+0000) 2020-02-05T20:37:00.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:00.161+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.161729+0000) 2020-02-05T20:37:00.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:00.162+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.162882+0000) 2020-02-05T20:37:00.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:00.163+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.163355+0000) 2020-02-05T20:37:00.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:00.162+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.163702+0000) 2020-02-05T20:37:00.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:00.162+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.164405+0000) 2020-02-05T20:37:00.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:00.164+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.164345+0000) 2020-02-05T20:37:00.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:00.164+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.164665+0000) 2020-02-05T20:37:00.167 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:00.165+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.165259+0000) 2020-02-05T20:37:00.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:00.165+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.165302+0000) 2020-02-05T20:37:00.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:00.164+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.165795+0000) 2020-02-05T20:37:00.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:00.164+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.166201+0000) 2020-02-05T20:37:00.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:00.165+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.167062+0000) 2020-02-05T20:37:00.169 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:00.166+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.167423+0000) 2020-02-05T20:37:00.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:00.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61689 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:00.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:00.167+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.169092+0000) 2020-02-05T20:37:00.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:00.172+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.172422+0000) 2020-02-05T20:37:00.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:00.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:00.179148+0000) 2020-02-05T20:37:02.274 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:02.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:02.272236+0000) 2020-02-05T20:37:03.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:03.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:03.398725+0000) 2020-02-05T20:37:03.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:03.428+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:03.429870+0000) 2020-02-05T20:37:04.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:04.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:04.473496+0000) 2020-02-05T20:37:05.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:05.172+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.173362+0000) 2020-02-05T20:37:05.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:05.174+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.175328+0000) 2020-02-05T20:37:05.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:05.173+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.175129+0000) 2020-02-05T20:37:05.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:05.174+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.175915+0000) 2020-02-05T20:37:05.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:05.175+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.176340+0000) 2020-02-05T20:37:05.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:05.175+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.176466+0000) 2020-02-05T20:37:05.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:05.176+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.177585+0000) 2020-02-05T20:37:05.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:05.176+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.177480+0000) 2020-02-05T20:37:05.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:05.177+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.178624+0000) 2020-02-05T20:37:05.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:05.177+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.179016+0000) 2020-02-05T20:37:05.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:05.178+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.179545+0000) 2020-02-05T20:37:05.181 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:05.179+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.180460+0000) 2020-02-05T20:37:05.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:05.180+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:05.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:05.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.184432+0000) 2020-02-05T20:37:05.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:05.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.184946+0000) 2020-02-05T20:37:05.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:05.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.405643+0000) 2020-02-05T20:37:05.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:05.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.774697+0000) 2020-02-05T20:37:05.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:05.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:05.811792+0000) 2020-02-05T20:37:06.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:06.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:06.671676+0000) 2020-02-05T20:37:06.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:06.913+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:06.914489+0000) 2020-02-05T20:37:06.917 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:06.914+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:06.916331+0000) 2020-02-05T20:37:09.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:09.308+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:09.309127+0000) 2020-02-05T20:37:09.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:09.917+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:09.918705+0000) 2020-02-05T20:37:10.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:10.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.167350+0000) 2020-02-05T20:37:10.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:10.178+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.179373+0000) 2020-02-05T20:37:10.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:10.187+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.188789+0000) 2020-02-05T20:37:10.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:10.187+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.188779+0000) 2020-02-05T20:37:10.191 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:10.188+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.189659+0000) 2020-02-05T20:37:10.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:10.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.189799+0000) 2020-02-05T20:37:10.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:10.188+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.189992+0000) 2020-02-05T20:37:10.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:10.189+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.190422+0000) 2020-02-05T20:37:10.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:10.189+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.190668+0000) 2020-02-05T20:37:10.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:10.189+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.190984+0000) 2020-02-05T20:37:10.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:10.190+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.191236+0000) 2020-02-05T20:37:10.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:10.190+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.191576+0000) 2020-02-05T20:37:10.206 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:10.190+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.191483+0000) 2020-02-05T20:37:10.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:10.192+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.193574+0000) 2020-02-05T20:37:10.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:10.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61767 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:10.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:10.191+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.192235+0000) 2020-02-05T20:37:10.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:10.199+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:10.200301+0000) 2020-02-05T20:37:12.274 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:12.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:12.272501+0000) 2020-02-05T20:37:13.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:13.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:13.398956+0000) 2020-02-05T20:37:13.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:13.429+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:13.430145+0000) 2020-02-05T20:37:14.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:14.481+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:14.482759+0000) 2020-02-05T20:37:15.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:15.200+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.201971+0000) 2020-02-05T20:37:15.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:15.201+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.202350+0000) 2020-02-05T20:37:15.205 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:15.202+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.204102+0000) 2020-02-05T20:37:15.206 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:15.203+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.204427+0000) 2020-02-05T20:37:15.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:15.204+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.205559+0000) 2020-02-05T20:37:15.207 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:15.204+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.205717+0000) 2020-02-05T20:37:15.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:15.205+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.206633+0000) 2020-02-05T20:37:15.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:15.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.206867+0000) 2020-02-05T20:37:15.209 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:15.206+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.207489+0000) 2020-02-05T20:37:15.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:15.205+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.207340+0000) 2020-02-05T20:37:15.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:15.206+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.207471+0000) 2020-02-05T20:37:15.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:15.206+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.207960+0000) 2020-02-05T20:37:15.211 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:15.208+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:15.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:15.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.210438+0000) 2020-02-05T20:37:15.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:15.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.210792+0000) 2020-02-05T20:37:15.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:15.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.405886+0000) 2020-02-05T20:37:15.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:15.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.774929+0000) 2020-02-05T20:37:15.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:15.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:15.812007+0000) 2020-02-05T20:37:16.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:16.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:16.671925+0000) 2020-02-05T20:37:16.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:16.913+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:16.914744+0000) 2020-02-05T20:37:16.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:16.915+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:16.916629+0000) 2020-02-05T20:37:19.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:19.308+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:19.309365+0000) 2020-02-05T20:37:19.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:19.918+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:19.918941+0000) 2020-02-05T20:37:20.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:20.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.167652+0000) 2020-02-05T20:37:20.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:20.178+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.179608+0000) 2020-02-05T20:37:20.216 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:20.213+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.214587+0000) 2020-02-05T20:37:20.216 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:20.213+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.215038+0000) 2020-02-05T20:37:20.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:20.215+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.216062+0000) 2020-02-05T20:37:20.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:20.215+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.216995+0000) 2020-02-05T20:37:20.219 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:20.216+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.217714+0000) 2020-02-05T20:37:20.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:20.216+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.217644+0000) 2020-02-05T20:37:20.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:20.216+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.218226+0000) 2020-02-05T20:37:20.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:20.217+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.218892+0000) 2020-02-05T20:37:20.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:20.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.219433+0000) 2020-02-05T20:37:20.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:20.218+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.219600+0000) 2020-02-05T20:37:20.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:20.218+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.220095+0000) 2020-02-05T20:37:20.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:20.218+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.220292+0000) 2020-02-05T20:37:20.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:20.218+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.220279+0000) 2020-02-05T20:37:20.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:20.221+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:20.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:20.224+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:20.225903+0000) 2020-02-05T20:37:22.274 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:22.271+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:22.272759+0000) 2020-02-05T20:37:23.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:23.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:23.399211+0000) 2020-02-05T20:37:23.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:23.429+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:23.430416+0000) 2020-02-05T20:37:24.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:24.494+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:24.494920+0000) 2020-02-05T20:37:25.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:25.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.228060+0000) 2020-02-05T20:37:25.230 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:25.227+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.228893+0000) 2020-02-05T20:37:25.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:25.228+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.229710+0000) 2020-02-05T20:37:25.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:25.229+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.230261+0000) 2020-02-05T20:37:25.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:25.229+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.231255+0000) 2020-02-05T20:37:25.233 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:25.230+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.231628+0000) 2020-02-05T20:37:25.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:25.230+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.231987+0000) 2020-02-05T20:37:25.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:25.231+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.232459+0000) 2020-02-05T20:37:25.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:25.231+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.232770+0000) 2020-02-05T20:37:25.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:25.232+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.233676+0000) 2020-02-05T20:37:25.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:25.231+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.233293+0000) 2020-02-05T20:37:25.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:25.232+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.233927+0000) 2020-02-05T20:37:25.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:25.232+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.234243+0000) 2020-02-05T20:37:25.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:25.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:25.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:25.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.240163+0000) 2020-02-05T20:37:25.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:25.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.406091+0000) 2020-02-05T20:37:25.531 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:37:25.537 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:37:25.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:25.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.775153+0000) 2020-02-05T20:37:25.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:25.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:25.812157+0000) 2020-02-05T20:37:26.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:26.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:26.672188+0000) 2020-02-05T20:37:26.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:26.914+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:26.914999+0000) 2020-02-05T20:37:26.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:26.916+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:26.916879+0000) 2020-02-05T20:37:29.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:29.308+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:29.309643+0000) 2020-02-05T20:37:29.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:29.918+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:29.919211+0000) 2020-02-05T20:37:30.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:30.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.167903+0000) 2020-02-05T20:37:30.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:30.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.179833+0000) 2020-02-05T20:37:30.244 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:30.241+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.242442+0000) 2020-02-05T20:37:30.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:30.241+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.242808+0000) 2020-02-05T20:37:30.245 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:30.241+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.242975+0000) 2020-02-05T20:37:30.246 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:30.242+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.243488+0000) 2020-02-05T20:37:30.246 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:30.242+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.243591+0000) 2020-02-05T20:37:30.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:30.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.244037+0000) 2020-02-05T20:37:30.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:30.242+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.244376+0000) 2020-02-05T20:37:30.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:30.243+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.245157+0000) 2020-02-05T20:37:30.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:30.245+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.246553+0000) 2020-02-05T20:37:30.249 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:30.244+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.246298+0000) 2020-02-05T20:37:30.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:30.245+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.246842+0000) 2020-02-05T20:37:30.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:30.246+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.247550+0000) 2020-02-05T20:37:30.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:30.246+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.248019+0000) 2020-02-05T20:37:30.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:30.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:30.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:30.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:30.251979+0000) 2020-02-05T20:37:32.274 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:32.272+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:32.273022+0000) 2020-02-05T20:37:33.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:33.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:33.399504+0000) 2020-02-05T20:37:33.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:33.430+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:33.430648+0000) 2020-02-05T20:37:34.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:34.502+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:34.502703+0000) 2020-02-05T20:37:35.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:35.254+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.255257+0000) 2020-02-05T20:37:35.257 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:35.254+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.255994+0000) 2020-02-05T20:37:35.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:35.255+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.257026+0000) 2020-02-05T20:37:35.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:35.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.257476+0000) 2020-02-05T20:37:35.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:35.256+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.257831+0000) 2020-02-05T20:37:35.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:35.256+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.257999+0000) 2020-02-05T20:37:35.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:35.257+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.258529+0000) 2020-02-05T20:37:35.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:35.257+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.258410+0000) 2020-02-05T20:37:35.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:35.258+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.259175+0000) 2020-02-05T20:37:35.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:35.257+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.258712+0000) 2020-02-05T20:37:35.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:35.257+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.259328+0000) 2020-02-05T20:37:35.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:35.258+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.259565+0000) 2020-02-05T20:37:35.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:35.258+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.260255+0000) 2020-02-05T20:37:35.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:35.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 61964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:35.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:35.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.266810+0000) 2020-02-05T20:37:35.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:35.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.406364+0000) 2020-02-05T20:37:35.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:35.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.775409+0000) 2020-02-05T20:37:35.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:35.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:35.812414+0000) 2020-02-05T20:37:36.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:36.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:36.672425+0000) 2020-02-05T20:37:36.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:36.914+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:36.915226+0000) 2020-02-05T20:37:36.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:36.916+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:36.917171+0000) 2020-02-05T20:37:39.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:39.309+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:39.309880+0000) 2020-02-05T20:37:39.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:39.919+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:39.919481+0000) 2020-02-05T20:37:40.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:40.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.168138+0000) 2020-02-05T20:37:40.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:40.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.180066+0000) 2020-02-05T20:37:40.273 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:40.266+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.267073+0000) 2020-02-05T20:37:40.273 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:40.267+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.268697+0000) 2020-02-05T20:37:40.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:40.268+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.269064+0000) 2020-02-05T20:37:40.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:40.269+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.270189+0000) 2020-02-05T20:37:40.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:40.270+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.271255+0000) 2020-02-05T20:37:40.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:40.266+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.268243+0000) 2020-02-05T20:37:40.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:40.267+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.269319+0000) 2020-02-05T20:37:40.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:40.269+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.270614+0000) 2020-02-05T20:37:40.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:40.270+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.271488+0000) 2020-02-05T20:37:40.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:40.270+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.272094+0000) 2020-02-05T20:37:40.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:40.271+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.272925+0000) 2020-02-05T20:37:40.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:40.271+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.272625+0000) 2020-02-05T20:37:40.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:40.272+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.273328+0000) 2020-02-05T20:37:40.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:40.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:40.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:40.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:40.279098+0000) 2020-02-05T20:37:42.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:42.272+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:42.273260+0000) 2020-02-05T20:37:43.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:43.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:43.399744+0000) 2020-02-05T20:37:43.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:43.430+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:43.430876+0000) 2020-02-05T20:37:44.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:44.510+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:44.510847+0000) 2020-02-05T20:37:45.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:45.280+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.282250+0000) 2020-02-05T20:37:45.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:45.281+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.282682+0000) 2020-02-05T20:37:45.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:45.282+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.283946+0000) 2020-02-05T20:37:45.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:45.282+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.284056+0000) 2020-02-05T20:37:45.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:45.283+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.284294+0000) 2020-02-05T20:37:45.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:45.283+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.284677+0000) 2020-02-05T20:37:45.287 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:45.284+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.285253+0000) 2020-02-05T20:37:45.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:45.284+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.285528+0000) 2020-02-05T20:37:45.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:45.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.285685+0000) 2020-02-05T20:37:45.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:45.284+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.286052+0000) 2020-02-05T20:37:45.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:45.284+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.286338+0000) 2020-02-05T20:37:45.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:45.285+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.286823+0000) 2020-02-05T20:37:45.289 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:45.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:45.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:45.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.290220+0000) 2020-02-05T20:37:45.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:45.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.290534+0000) 2020-02-05T20:37:45.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:45.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.406634+0000) 2020-02-05T20:37:45.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:45.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.775591+0000) 2020-02-05T20:37:45.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:45.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:45.812625+0000) 2020-02-05T20:37:46.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:46.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:46.672653+0000) 2020-02-05T20:37:46.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:46.914+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:46.915461+0000) 2020-02-05T20:37:46.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:46.916+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:46.917455+0000) 2020-02-05T20:37:49.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:49.309+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:49.310139+0000) 2020-02-05T20:37:49.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:49.918+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:49.919728+0000) 2020-02-05T20:37:50.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:50.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.168384+0000) 2020-02-05T20:37:50.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:50.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.180303+0000) 2020-02-05T20:37:50.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:50.293+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.294288+0000) 2020-02-05T20:37:50.297 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:50.294+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.295705+0000) 2020-02-05T20:37:50.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:50.295+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.296110+0000) 2020-02-05T20:37:50.298 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:50.295+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.296676+0000) 2020-02-05T20:37:50.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:50.296+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.296534+0000) 2020-02-05T20:37:50.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:50.296+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.297118+0000) 2020-02-05T20:37:50.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:50.297+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.297744+0000) 2020-02-05T20:37:50.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:50.296+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.297805+0000) 2020-02-05T20:37:50.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:50.297+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.298304+0000) 2020-02-05T20:37:50.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:50.297+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.298431+0000) 2020-02-05T20:37:50.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:50.298+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.298922+0000) 2020-02-05T20:37:50.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:50.298+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.299073+0000) 2020-02-05T20:37:50.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:50.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.299025+0000) 2020-02-05T20:37:50.301 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:50.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:50.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:50.303+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:50.304484+0000) 2020-02-05T20:37:52.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:52.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:52.273518+0000) 2020-02-05T20:37:53.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:53.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:53.400004+0000) 2020-02-05T20:37:53.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:53.431+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:53.431115+0000) 2020-02-05T20:37:54.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:54.519+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:54.519127+0000) 2020-02-05T20:37:55.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:55.305+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.305360+0000) 2020-02-05T20:37:55.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:37:55.305+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.305904+0000) 2020-02-05T20:37:55.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:55.308+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.308674+0000) 2020-02-05T20:37:55.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:55.309+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.309243+0000) 2020-02-05T20:37:55.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:55.310+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.310294+0000) 2020-02-05T20:37:55.312 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:55.310+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.310585+0000) 2020-02-05T20:37:55.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:55.310+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.311041+0000) 2020-02-05T20:37:55.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:37:55.311+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.311637+0000) 2020-02-05T20:37:55.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:37:55.311+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.311415+0000) 2020-02-05T20:37:55.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:55.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.312034+0000) 2020-02-05T20:37:55.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:55.311+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.312213+0000) 2020-02-05T20:37:55.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:37:55.311+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.312387+0000) 2020-02-05T20:37:55.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:37:55.312+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.312653+0000) 2020-02-05T20:37:55.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:37:55.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62122 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:37:55.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:55.318+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.318667+0000) 2020-02-05T20:37:55.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:37:55.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.406854+0000) 2020-02-05T20:37:55.719 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:37:55.726 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:37:55.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:37:55.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.775780+0000) 2020-02-05T20:37:55.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:55.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:55.812832+0000) 2020-02-05T20:37:56.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:37:56.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:56.672889+0000) 2020-02-05T20:37:56.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:37:56.915+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:56.915686+0000) 2020-02-05T20:37:56.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:37:56.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:56.917690+0000) 2020-02-05T20:37:59.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:37:59.309+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:59.310376+0000) 2020-02-05T20:37:59.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:37:59.918+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:37:59.919973+0000) 2020-02-05T20:38:00.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:00.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.168698+0000) 2020-02-05T20:38:00.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:00.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.180536+0000) 2020-02-05T20:38:00.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:00.319+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.321057+0000) 2020-02-05T20:38:00.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:00.320+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.321301+0000) 2020-02-05T20:38:00.323 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:00.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.321592+0000) 2020-02-05T20:38:00.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:00.320+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.321994+0000) 2020-02-05T20:38:00.324 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:00.321+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.322204+0000) 2020-02-05T20:38:00.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:00.322+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.322897+0000) 2020-02-05T20:38:00.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:00.321+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.322603+0000) 2020-02-05T20:38:00.325 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:00.322+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.323197+0000) 2020-02-05T20:38:00.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:00.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.323843+0000) 2020-02-05T20:38:00.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:00.323+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.324214+0000) 2020-02-05T20:38:00.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:00.324+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.324968+0000) 2020-02-05T20:38:00.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:00.324+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.325109+0000) 2020-02-05T20:38:00.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:00.324+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.325180+0000) 2020-02-05T20:38:00.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:00.326+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:00.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:00.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:00.329276+0000) 2020-02-05T20:38:02.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:02.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:02.273772+0000) 2020-02-05T20:38:03.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:03.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:03.400209+0000) 2020-02-05T20:38:03.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:03.430+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:03.431298+0000) 2020-02-05T20:38:04.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:04.524+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:04.525875+0000) 2020-02-05T20:38:05.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:05.331+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.332130+0000) 2020-02-05T20:38:05.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:05.332+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.333340+0000) 2020-02-05T20:38:05.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:05.332+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.333641+0000) 2020-02-05T20:38:05.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:05.332+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.333841+0000) 2020-02-05T20:38:05.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:05.333+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.334261+0000) 2020-02-05T20:38:05.336 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:05.333+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.334298+0000) 2020-02-05T20:38:05.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:05.334+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.334423+0000) 2020-02-05T20:38:05.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:05.334+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.334643+0000) 2020-02-05T20:38:05.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:05.334+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.335206+0000) 2020-02-05T20:38:05.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:05.335+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.335507+0000) 2020-02-05T20:38:05.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:05.335+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.335922+0000) 2020-02-05T20:38:05.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:05.334+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.335525+0000) 2020-02-05T20:38:05.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:05.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.336214+0000) 2020-02-05T20:38:05.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:05.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:05.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:05.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.340862+0000) 2020-02-05T20:38:05.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:05.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.407078+0000) 2020-02-05T20:38:05.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:05.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.776004+0000) 2020-02-05T20:38:05.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:05.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:05.813051+0000) 2020-02-05T20:38:06.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:06.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:06.673160+0000) 2020-02-05T20:38:06.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:06.915+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:06.915929+0000) 2020-02-05T20:38:06.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:06.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:06.917958+0000) 2020-02-05T20:38:09.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:09.309+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:09.310637+0000) 2020-02-05T20:38:09.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:09.919+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:09.920194+0000) 2020-02-05T20:38:10.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:10.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.168977+0000) 2020-02-05T20:38:10.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:10.179+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.180767+0000) 2020-02-05T20:38:10.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:10.344+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.345411+0000) 2020-02-05T20:38:10.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:10.346+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.346795+0000) 2020-02-05T20:38:10.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:10.346+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.347605+0000) 2020-02-05T20:38:10.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:10.347+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.347492+0000) 2020-02-05T20:38:10.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:10.346+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.347953+0000) 2020-02-05T20:38:10.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:10.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.348422+0000) 2020-02-05T20:38:10.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:10.348+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.348854+0000) 2020-02-05T20:38:10.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:10.349+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.349785+0000) 2020-02-05T20:38:10.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:10.349+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.350490+0000) 2020-02-05T20:38:10.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:10.349+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.350710+0000) 2020-02-05T20:38:10.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:10.350+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.351194+0000) 2020-02-05T20:38:10.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:10.350+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.351400+0000) 2020-02-05T20:38:10.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:10.350+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.351137+0000) 2020-02-05T20:38:10.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:10.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:10.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:10.354+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:10.355184+0000) 2020-02-05T20:38:12.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:12.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:12.274026+0000) 2020-02-05T20:38:13.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:13.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:13.400400+0000) 2020-02-05T20:38:13.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:13.430+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:13.431489+0000) 2020-02-05T20:38:14.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:14.533+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:14.534207+0000) 2020-02-05T20:38:15.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:15.356+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.357623+0000) 2020-02-05T20:38:15.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:15.358+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.359348+0000) 2020-02-05T20:38:15.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:15.358+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.359714+0000) 2020-02-05T20:38:15.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:15.360+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.360554+0000) 2020-02-05T20:38:15.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:15.360+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.360556+0000) 2020-02-05T20:38:15.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:15.359+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.360915+0000) 2020-02-05T20:38:15.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:15.360+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.361380+0000) 2020-02-05T20:38:15.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:15.361+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.362246+0000) 2020-02-05T20:38:15.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:15.362+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.362847+0000) 2020-02-05T20:38:15.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:15.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.363750+0000) 2020-02-05T20:38:15.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:15.363+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.364366+0000) 2020-02-05T20:38:15.366 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:15.363+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.364427+0000) 2020-02-05T20:38:15.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:15.364+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.365101+0000) 2020-02-05T20:38:15.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:15.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:15.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:15.369+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.370701+0000) 2020-02-05T20:38:15.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:15.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.407235+0000) 2020-02-05T20:38:15.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:15.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.776214+0000) 2020-02-05T20:38:15.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:15.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:15.813297+0000) 2020-02-05T20:38:16.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:16.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:16.673471+0000) 2020-02-05T20:38:16.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:16.915+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:16.916185+0000) 2020-02-05T20:38:16.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:16.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:16.918175+0000) 2020-02-05T20:38:19.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:19.309+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:19.310887+0000) 2020-02-05T20:38:19.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:19.919+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:19.920414+0000) 2020-02-05T20:38:20.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:20.169+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.169228+0000) 2020-02-05T20:38:20.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:20.180+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.180990+0000) 2020-02-05T20:38:20.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:20.371+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.371673+0000) 2020-02-05T20:38:20.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:20.372+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.373208+0000) 2020-02-05T20:38:20.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:20.372+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.373939+0000) 2020-02-05T20:38:20.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:20.373+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.373762+0000) 2020-02-05T20:38:20.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:20.375+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.375571+0000) 2020-02-05T20:38:20.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:20.375+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.375479+0000) 2020-02-05T20:38:20.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:20.375+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.376094+0000) 2020-02-05T20:38:20.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:20.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.376866+0000) 2020-02-05T20:38:20.379 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:20.376+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.377873+0000) 2020-02-05T20:38:20.379 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:20.377+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.378363+0000) 2020-02-05T20:38:20.380 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:20.377+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.378826+0000) 2020-02-05T20:38:20.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:20.377+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.379021+0000) 2020-02-05T20:38:20.381 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:20.379+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:20.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:20.382+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.383836+0000) 2020-02-05T20:38:20.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:20.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:20.384203+0000) 2020-02-05T20:38:22.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:22.274+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:22.274266+0000) 2020-02-05T20:38:23.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:23.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:23.400704+0000) 2020-02-05T20:38:23.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:23.431+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:23.431723+0000) 2020-02-05T20:38:24.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:24.541+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:24.542518+0000) 2020-02-05T20:38:25.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:25.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.386244+0000) 2020-02-05T20:38:25.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:25.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.386953+0000) 2020-02-05T20:38:25.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:25.386+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.386837+0000) 2020-02-05T20:38:25.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:25.386+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.387435+0000) 2020-02-05T20:38:25.389 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:25.387+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.388350+0000) 2020-02-05T20:38:25.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:25.389+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.389576+0000) 2020-02-05T20:38:25.391 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:25.388+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.389911+0000) 2020-02-05T20:38:25.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:25.390+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.390542+0000) 2020-02-05T20:38:25.393 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:25.391+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.391475+0000) 2020-02-05T20:38:25.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:25.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.391673+0000) 2020-02-05T20:38:25.393 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:25.391+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.391771+0000) 2020-02-05T20:38:25.394 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:25.390+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.391402+0000) 2020-02-05T20:38:25.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:25.392+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:25.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:25.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.396895+0000) 2020-02-05T20:38:25.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:25.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.397353+0000) 2020-02-05T20:38:25.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:25.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.407478+0000) 2020-02-05T20:38:25.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:25.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.776471+0000) 2020-02-05T20:38:25.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:25.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:25.813526+0000) 2020-02-05T20:38:25.917 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:38:25.924 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:38:26.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:26.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:26.673726+0000) 2020-02-05T20:38:26.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:26.916+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:26.916481+0000) 2020-02-05T20:38:26.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:26.918+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:26.918417+0000) 2020-02-05T20:38:29.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:29.310+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:29.311123+0000) 2020-02-05T20:38:29.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:29.920+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:29.920646+0000) 2020-02-05T20:38:30.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:30.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.169482+0000) 2020-02-05T20:38:30.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:30.180+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.181250+0000) 2020-02-05T20:38:30.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:30.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.399821+0000) 2020-02-05T20:38:30.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:30.399+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.400317+0000) 2020-02-05T20:38:30.402 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:30.400+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.401072+0000) 2020-02-05T20:38:30.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:30.400+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.401781+0000) 2020-02-05T20:38:30.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:30.400+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.401929+0000) 2020-02-05T20:38:30.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:30.401+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.402644+0000) 2020-02-05T20:38:30.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:30.402+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.403401+0000) 2020-02-05T20:38:30.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:30.402+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.403696+0000) 2020-02-05T20:38:30.408 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:30.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.404859+0000) 2020-02-05T20:38:30.408 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:30.403+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.404880+0000) 2020-02-05T20:38:30.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:30.403+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.405374+0000) 2020-02-05T20:38:30.409 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:30.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.405492+0000) 2020-02-05T20:38:30.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:30.404+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.405741+0000) 2020-02-05T20:38:30.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:30.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:30.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:30.408+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:30.409744+0000) 2020-02-05T20:38:32.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:32.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:32.274546+0000) 2020-02-05T20:38:33.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:33.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:33.400939+0000) 2020-02-05T20:38:33.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:33.431+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:33.431992+0000) 2020-02-05T20:38:34.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:34.554+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:34.554408+0000) 2020-02-05T20:38:35.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:35.409+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.411018+0000) 2020-02-05T20:38:35.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:35.413+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.414594+0000) 2020-02-05T20:38:35.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:35.414+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.415440+0000) 2020-02-05T20:38:35.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:35.414+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.415483+0000) 2020-02-05T20:38:35.418 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:35.414+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.415843+0000) 2020-02-05T20:38:35.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:35.415+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.416622+0000) 2020-02-05T20:38:35.418 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:35.416+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.417374+0000) 2020-02-05T20:38:35.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:35.416+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.417719+0000) 2020-02-05T20:38:35.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:35.416+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.417836+0000) 2020-02-05T20:38:35.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:35.417+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.418480+0000) 2020-02-05T20:38:35.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:35.417+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.418531+0000) 2020-02-05T20:38:35.421 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:35.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:35.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:35.417+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.419049+0000) 2020-02-05T20:38:35.421 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:35.418+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.419402+0000) 2020-02-05T20:38:35.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:35.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.425243+0000) 2020-02-05T20:38:35.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:35.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.776726+0000) 2020-02-05T20:38:35.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:35.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:35.813774+0000) 2020-02-05T20:38:36.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:36.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:36.673980+0000) 2020-02-05T20:38:36.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:36.915+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:36.916743+0000) 2020-02-05T20:38:36.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:36.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:36.918628+0000) 2020-02-05T20:38:39.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:39.311+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:39.311348+0000) 2020-02-05T20:38:39.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:39.920+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:39.920873+0000) 2020-02-05T20:38:40.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:40.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.169777+0000) 2020-02-05T20:38:40.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:40.181+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.181499+0000) 2020-02-05T20:38:40.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:40.422+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.423937+0000) 2020-02-05T20:38:40.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:40.425+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.426879+0000) 2020-02-05T20:38:40.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:40.426+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.427238+0000) 2020-02-05T20:38:40.429 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:40.426+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.427642+0000) 2020-02-05T20:38:40.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:40.427+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.428123+0000) 2020-02-05T20:38:40.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:40.426+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.428390+0000) 2020-02-05T20:38:40.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:40.427+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.428463+0000) 2020-02-05T20:38:40.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:40.428+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.429316+0000) 2020-02-05T20:38:40.431 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:40.428+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.429521+0000) 2020-02-05T20:38:40.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:40.428+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.430080+0000) 2020-02-05T20:38:40.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:40.429+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.430660+0000) 2020-02-05T20:38:40.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:40.429+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.430919+0000) 2020-02-05T20:38:40.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:40.429+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.431021+0000) 2020-02-05T20:38:40.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:40.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62477 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:40.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:40.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:40.435930+0000) 2020-02-05T20:38:42.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:42.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:42.158940+0000) 2020-02-05T20:38:42.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:42.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:42.274848+0000) 2020-02-05T20:38:43.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:43.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:43.401176+0000) 2020-02-05T20:38:43.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:43.432+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:43.432274+0000) 2020-02-05T20:38:44.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:44.562+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:44.562466+0000) 2020-02-05T20:38:45.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:45.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.438252+0000) 2020-02-05T20:38:45.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:45.437+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.438754+0000) 2020-02-05T20:38:45.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:45.438+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.439337+0000) 2020-02-05T20:38:45.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:45.438+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.439852+0000) 2020-02-05T20:38:45.442 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:45.439+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.440258+0000) 2020-02-05T20:38:45.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:45.439+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.440484+0000) 2020-02-05T20:38:45.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:45.440+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.441433+0000) 2020-02-05T20:38:45.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:45.439+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.441323+0000) 2020-02-05T20:38:45.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:45.440+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.442252+0000) 2020-02-05T20:38:45.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:45.440+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.442000+0000) 2020-02-05T20:38:45.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:45.441+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.443141+0000) 2020-02-05T20:38:45.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:45.442+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.443469+0000) 2020-02-05T20:38:45.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:45.442+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.443921+0000) 2020-02-05T20:38:45.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:45.443+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:45.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:45.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.447462+0000) 2020-02-05T20:38:45.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:45.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.777021+0000) 2020-02-05T20:38:45.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:45.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:45.814019+0000) 2020-02-05T20:38:46.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:46.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:46.674211+0000) 2020-02-05T20:38:46.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:46.915+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:46.916978+0000) 2020-02-05T20:38:46.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:46.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:46.918860+0000) 2020-02-05T20:38:49.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:49.311+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:49.311573+0000) 2020-02-05T20:38:49.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:49.921+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:49.921120+0000) 2020-02-05T20:38:50.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:50.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.169958+0000) 2020-02-05T20:38:50.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:50.181+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.181722+0000) 2020-02-05T20:38:50.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:50.451+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.451574+0000) 2020-02-05T20:38:50.453 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:50.451+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.451952+0000) 2020-02-05T20:38:50.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:50.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.452571+0000) 2020-02-05T20:38:50.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:50.452+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.453037+0000) 2020-02-05T20:38:50.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:50.453+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.453722+0000) 2020-02-05T20:38:50.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:50.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.453894+0000) 2020-02-05T20:38:50.455 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:50.452+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.453817+0000) 2020-02-05T20:38:50.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:50.453+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.454852+0000) 2020-02-05T20:38:50.456 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:50.455+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.455203+0000) 2020-02-05T20:38:50.457 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:50.454+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.455678+0000) 2020-02-05T20:38:50.457 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:50.454+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.455945+0000) 2020-02-05T20:38:50.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:50.455+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.456673+0000) 2020-02-05T20:38:50.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:50.455+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62556 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:50.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:50.456+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.457500+0000) 2020-02-05T20:38:50.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:50.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:50.460858+0000) 2020-02-05T20:38:52.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:52.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:52.159205+0000) 2020-02-05T20:38:52.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:52.273+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:52.275038+0000) 2020-02-05T20:38:53.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:53.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:53.401414+0000) 2020-02-05T20:38:53.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:53.431+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:53.432504+0000) 2020-02-05T20:38:54.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:54.569+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:54.570890+0000) 2020-02-05T20:38:55.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:55.462+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.463233+0000) 2020-02-05T20:38:55.465 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:55.463+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.464473+0000) 2020-02-05T20:38:55.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:55.464+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.465119+0000) 2020-02-05T20:38:55.467 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:38:55.464+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.465704+0000) 2020-02-05T20:38:55.467 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:38:55.464+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.465873+0000) 2020-02-05T20:38:55.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:38:55.464+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.465999+0000) 2020-02-05T20:38:55.468 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:55.465+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.466365+0000) 2020-02-05T20:38:55.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:38:55.465+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.466956+0000) 2020-02-05T20:38:55.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:55.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.466838+0000) 2020-02-05T20:38:55.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:55.466+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.467330+0000) 2020-02-05T20:38:55.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:55.466+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.468219+0000) 2020-02-05T20:38:55.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:38:55.466+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.467997+0000) 2020-02-05T20:38:55.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:38:55.467+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.469130+0000) 2020-02-05T20:38:55.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:38:55.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:38:55.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:55.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.472904+0000) 2020-02-05T20:38:55.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:38:55.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.777287+0000) 2020-02-05T20:38:55.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:55.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:55.814253+0000) 2020-02-05T20:38:56.088 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:38:56.095 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:38:56.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:38:56.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:56.674472+0000) 2020-02-05T20:38:56.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:38:56.916+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:56.917235+0000) 2020-02-05T20:38:56.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:38:56.918+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:56.919094+0000) 2020-02-05T20:38:59.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:38:59.310+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:59.311809+0000) 2020-02-05T20:38:59.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:38:59.920+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:38:59.921370+0000) 2020-02-05T20:39:00.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:00.169+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.170202+0000) 2020-02-05T20:39:00.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:00.181+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.181947+0000) 2020-02-05T20:39:00.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:00.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.476595+0000) 2020-02-05T20:39:00.478 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:00.476+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.477170+0000) 2020-02-05T20:39:00.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:00.477+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.478107+0000) 2020-02-05T20:39:00.479 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:00.477+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.478496+0000) 2020-02-05T20:39:00.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:00.478+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.479263+0000) 2020-02-05T20:39:00.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:00.478+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.479667+0000) 2020-02-05T20:39:00.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:00.478+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.479975+0000) 2020-02-05T20:39:00.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:00.478+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.480339+0000) 2020-02-05T20:39:00.482 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:00.479+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.480809+0000) 2020-02-05T20:39:00.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:00.480+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.481621+0000) 2020-02-05T20:39:00.483 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:00.480+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.481796+0000) 2020-02-05T20:39:00.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:00.480+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.482290+0000) 2020-02-05T20:39:00.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:00.480+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.482300+0000) 2020-02-05T20:39:00.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:00.483+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:00.484386+0000) 2020-02-05T20:39:00.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:00.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62635 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:02.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:02.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:02.159476+0000) 2020-02-05T20:39:02.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:02.274+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:02.275276+0000) 2020-02-05T20:39:03.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:03.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:03.401714+0000) 2020-02-05T20:39:03.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:03.431+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:03.432739+0000) 2020-02-05T20:39:04.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:04.578+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:04.579136+0000) 2020-02-05T20:39:05.493 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:05.491+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.492146+0000) 2020-02-05T20:39:05.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:05.491+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.492817+0000) 2020-02-05T20:39:05.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:05.492+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.493189+0000) 2020-02-05T20:39:05.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:05.492+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.493801+0000) 2020-02-05T20:39:05.495 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:05.492+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.493672+0000) 2020-02-05T20:39:05.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:05.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.494486+0000) 2020-02-05T20:39:05.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:05.493+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.494988+0000) 2020-02-05T20:39:05.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:05.494+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.495764+0000) 2020-02-05T20:39:05.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:05.494+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.495943+0000) 2020-02-05T20:39:05.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:05.494+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.495746+0000) 2020-02-05T20:39:05.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:05.495+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.496567+0000) 2020-02-05T20:39:05.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:05.496+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.497069+0000) 2020-02-05T20:39:05.499 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:05.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.496869+0000) 2020-02-05T20:39:05.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:05.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:05.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:05.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.499243+0000) 2020-02-05T20:39:05.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:05.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.777513+0000) 2020-02-05T20:39:05.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:05.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:05.814483+0000) 2020-02-05T20:39:06.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:06.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:06.674730+0000) 2020-02-05T20:39:06.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:06.916+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:06.917474+0000) 2020-02-05T20:39:06.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:06.917+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:06.919337+0000) 2020-02-05T20:39:09.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:09.311+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:09.312045+0000) 2020-02-05T20:39:09.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:09.920+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:09.921613+0000) 2020-02-05T20:39:10.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:10.169+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.170481+0000) 2020-02-05T20:39:10.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:10.181+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.182169+0000) 2020-02-05T20:39:10.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:10.502+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.503397+0000) 2020-02-05T20:39:10.505 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:10.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.503799+0000) 2020-02-05T20:39:10.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:10.504+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.505583+0000) 2020-02-05T20:39:10.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:10.503+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.505330+0000) 2020-02-05T20:39:10.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:10.505+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.506787+0000) 2020-02-05T20:39:10.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:10.507+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.508165+0000) 2020-02-05T20:39:10.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:10.506+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.507723+0000) 2020-02-05T20:39:10.510 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:10.507+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.508818+0000) 2020-02-05T20:39:10.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:10.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.508795+0000) 2020-02-05T20:39:10.511 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:10.508+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.509281+0000) 2020-02-05T20:39:10.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:10.508+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.509737+0000) 2020-02-05T20:39:10.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:10.508+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.510121+0000) 2020-02-05T20:39:10.513 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:10.509+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.510726+0000) 2020-02-05T20:39:10.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:10.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62717 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:10.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:10.511+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:10.513041+0000) 2020-02-05T20:39:12.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:12.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:12.159738+0000) 2020-02-05T20:39:12.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:12.274+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:12.275519+0000) 2020-02-05T20:39:13.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:13.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:13.401964+0000) 2020-02-05T20:39:13.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:13.432+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:13.433007+0000) 2020-02-05T20:39:14.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:14.586+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:14.587629+0000) 2020-02-05T20:39:15.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:15.515+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.516463+0000) 2020-02-05T20:39:15.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:15.516+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.517935+0000) 2020-02-05T20:39:15.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:15.516+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.518304+0000) 2020-02-05T20:39:15.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:15.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.518519+0000) 2020-02-05T20:39:15.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:15.517+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.519036+0000) 2020-02-05T20:39:15.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:15.518+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.519378+0000) 2020-02-05T20:39:15.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:15.518+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.519504+0000) 2020-02-05T20:39:15.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:15.518+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.519893+0000) 2020-02-05T20:39:15.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:15.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.521291+0000) 2020-02-05T20:39:15.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:15.520+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.521310+0000) 2020-02-05T20:39:15.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:15.520+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.521462+0000) 2020-02-05T20:39:15.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:15.520+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.521678+0000) 2020-02-05T20:39:15.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:15.520+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.522136+0000) 2020-02-05T20:39:15.525 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:15.522+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:15.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:15.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.527284+0000) 2020-02-05T20:39:15.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:15.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.777726+0000) 2020-02-05T20:39:15.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:15.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:15.814689+0000) 2020-02-05T20:39:16.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:16.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:16.674959+0000) 2020-02-05T20:39:16.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:16.917+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:16.917738+0000) 2020-02-05T20:39:16.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:16.919+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:16.919622+0000) 2020-02-05T20:39:19.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:19.311+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:19.312328+0000) 2020-02-05T20:39:19.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:19.921+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:19.921879+0000) 2020-02-05T20:39:20.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:20.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.170730+0000) 2020-02-05T20:39:20.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:20.181+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.182409+0000) 2020-02-05T20:39:20.532 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:20.530+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.530604+0000) 2020-02-05T20:39:20.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:20.529+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.530998+0000) 2020-02-05T20:39:20.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:20.530+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.531570+0000) 2020-02-05T20:39:20.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:20.531+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.532376+0000) 2020-02-05T20:39:20.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:20.531+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.533059+0000) 2020-02-05T20:39:20.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:20.532+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.533065+0000) 2020-02-05T20:39:20.535 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:20.532+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.533404+0000) 2020-02-05T20:39:20.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:20.532+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.533994+0000) 2020-02-05T20:39:20.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:20.533+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.534206+0000) 2020-02-05T20:39:20.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:20.533+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.535017+0000) 2020-02-05T20:39:20.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:20.534+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.535133+0000) 2020-02-05T20:39:20.537 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:20.534+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.534777+0000) 2020-02-05T20:39:20.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:20.534+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.535275+0000) 2020-02-05T20:39:20.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:20.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:20.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:20.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:20.541127+0000) 2020-02-05T20:39:22.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:22.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:22.160004+0000) 2020-02-05T20:39:22.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:22.275+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:22.275764+0000) 2020-02-05T20:39:23.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:23.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:23.402199+0000) 2020-02-05T20:39:23.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:23.432+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:23.433305+0000) 2020-02-05T20:39:24.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:24.595+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:24.596148+0000) 2020-02-05T20:39:25.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:25.541+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.541988+0000) 2020-02-05T20:39:25.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:25.543+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.543963+0000) 2020-02-05T20:39:25.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:25.543+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.544903+0000) 2020-02-05T20:39:25.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:25.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.545416+0000) 2020-02-05T20:39:25.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:25.544+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.545676+0000) 2020-02-05T20:39:25.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:25.548+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.548586+0000) 2020-02-05T20:39:25.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:25.548+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.548738+0000) 2020-02-05T20:39:25.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:25.548+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.548867+0000) 2020-02-05T20:39:25.554 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:25.548+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.549293+0000) 2020-02-05T20:39:25.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:25.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:25.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:25.545+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.546248+0000) 2020-02-05T20:39:25.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:25.545+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.546712+0000) 2020-02-05T20:39:25.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:25.546+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.547624+0000) 2020-02-05T20:39:25.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:25.546+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.548009+0000) 2020-02-05T20:39:25.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:25.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.552280+0000) 2020-02-05T20:39:25.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:25.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.777939+0000) 2020-02-05T20:39:25.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:25.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:25.814907+0000) 2020-02-05T20:39:26.260 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:39:26.267 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:39:26.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:26.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:26.675202+0000) 2020-02-05T20:39:26.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:26.917+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:26.918036+0000) 2020-02-05T20:39:26.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:26.919+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:26.919864+0000) 2020-02-05T20:39:29.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:29.312+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:29.312590+0000) 2020-02-05T20:39:29.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:29.921+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:29.922139+0000) 2020-02-05T20:39:30.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:30.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.170962+0000) 2020-02-05T20:39:30.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:30.182+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.182637+0000) 2020-02-05T20:39:30.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:30.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.555302+0000) 2020-02-05T20:39:30.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:30.554+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.555936+0000) 2020-02-05T20:39:30.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:30.556+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.556550+0000) 2020-02-05T20:39:30.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:30.556+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.557229+0000) 2020-02-05T20:39:30.559 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:30.556+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.558030+0000) 2020-02-05T20:39:30.560 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:30.557+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.558519+0000) 2020-02-05T20:39:30.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:30.558+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.559230+0000) 2020-02-05T20:39:30.561 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:30.558+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.559407+0000) 2020-02-05T20:39:30.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:30.558+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.559536+0000) 2020-02-05T20:39:30.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:30.560+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.561046+0000) 2020-02-05T20:39:30.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:30.560+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.561053+0000) 2020-02-05T20:39:30.563 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:30.561+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.561615+0000) 2020-02-05T20:39:30.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:30.561+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.561803+0000) 2020-02-05T20:39:30.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:30.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:30.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:30.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:30.567391+0000) 2020-02-05T20:39:32.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:32.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:32.160259+0000) 2020-02-05T20:39:32.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:32.275+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:32.276023+0000) 2020-02-05T20:39:33.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:33.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:33.402466+0000) 2020-02-05T20:39:33.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:33.433+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:33.433540+0000) 2020-02-05T20:39:34.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:34.604+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:34.605056+0000) 2020-02-05T20:39:35.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:35.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.568942+0000) 2020-02-05T20:39:35.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:35.568+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.569845+0000) 2020-02-05T20:39:35.572 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:35.569+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.570368+0000) 2020-02-05T20:39:35.572 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:35.569+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.570839+0000) 2020-02-05T20:39:35.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:35.570+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.571405+0000) 2020-02-05T20:39:35.573 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:35.571+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.571808+0000) 2020-02-05T20:39:35.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:35.571+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.572215+0000) 2020-02-05T20:39:35.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:35.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.573128+0000) 2020-02-05T20:39:35.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:35.572+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.572592+0000) 2020-02-05T20:39:35.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:35.573+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.573891+0000) 2020-02-05T20:39:35.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:35.574+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.574961+0000) 2020-02-05T20:39:35.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:35.573+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.574890+0000) 2020-02-05T20:39:35.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:35.575+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.575951+0000) 2020-02-05T20:39:35.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:35.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:35.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:35.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.580564+0000) 2020-02-05T20:39:35.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:35.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.778153+0000) 2020-02-05T20:39:35.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:35.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:35.815140+0000) 2020-02-05T20:39:36.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:36.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:36.675484+0000) 2020-02-05T20:39:36.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:36.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:36.918288+0000) 2020-02-05T20:39:36.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:36.919+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:36.920120+0000) 2020-02-05T20:39:39.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:39.312+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:39.312845+0000) 2020-02-05T20:39:39.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:39.922+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:39.922367+0000) 2020-02-05T20:39:40.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:40.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.171190+0000) 2020-02-05T20:39:40.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:40.182+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.182866+0000) 2020-02-05T20:39:40.585 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:40.583+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.584177+0000) 2020-02-05T20:39:40.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:40.583+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.584451+0000) 2020-02-05T20:39:40.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:40.583+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.583515+0000) 2020-02-05T20:39:40.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:40.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.585350+0000) 2020-02-05T20:39:40.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:40.584+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.585127+0000) 2020-02-05T20:39:40.587 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:40.585+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.586250+0000) 2020-02-05T20:39:40.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:40.585+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.586470+0000) 2020-02-05T20:39:40.588 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:40.585+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.586577+0000) 2020-02-05T20:39:40.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:40.586+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.587398+0000) 2020-02-05T20:39:40.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:40.586+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.587390+0000) 2020-02-05T20:39:40.589 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:40.586+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.587926+0000) 2020-02-05T20:39:40.590 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:40.587+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.587957+0000) 2020-02-05T20:39:40.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:40.587+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.588328+0000) 2020-02-05T20:39:40.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:40.590+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 62963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:40.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:40.593+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:40.594394+0000) 2020-02-05T20:39:42.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:42.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:42.160523+0000) 2020-02-05T20:39:42.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:42.276+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:42.276277+0000) 2020-02-05T20:39:43.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:43.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:43.402732+0000) 2020-02-05T20:39:43.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:43.433+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:43.433772+0000) 2020-02-05T20:39:44.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:44.616+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:44.616558+0000) 2020-02-05T20:39:45.598 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:45.596+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.596631+0000) 2020-02-05T20:39:45.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:45.597+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.597944+0000) 2020-02-05T20:39:45.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:45.598+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.598320+0000) 2020-02-05T20:39:45.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:45.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.598866+0000) 2020-02-05T20:39:45.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:45.600+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.600487+0000) 2020-02-05T20:39:45.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:45.600+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.600823+0000) 2020-02-05T20:39:45.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:45.601+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.601789+0000) 2020-02-05T20:39:45.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:45.601+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.602330+0000) 2020-02-05T20:39:45.604 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:45.602+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.602920+0000) 2020-02-05T20:39:45.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:45.602+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.603109+0000) 2020-02-05T20:39:45.605 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:45.602+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.602907+0000) 2020-02-05T20:39:45.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:45.603+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.603632+0000) 2020-02-05T20:39:45.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:45.603+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.603664+0000) 2020-02-05T20:39:45.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:45.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:45.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:45.608+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.608506+0000) 2020-02-05T20:39:45.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:45.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.778327+0000) 2020-02-05T20:39:45.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:45.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:45.815372+0000) 2020-02-05T20:39:46.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:46.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:46.675720+0000) 2020-02-05T20:39:46.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:46.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:46.918520+0000) 2020-02-05T20:39:46.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:46.920+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:46.920368+0000) 2020-02-05T20:39:49.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:49.312+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:49.313107+0000) 2020-02-05T20:39:49.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:49.921+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:49.922606+0000) 2020-02-05T20:39:50.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:50.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.171465+0000) 2020-02-05T20:39:50.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:50.182+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.183132+0000) 2020-02-05T20:39:50.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:50.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.610129+0000) 2020-02-05T20:39:50.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:50.610+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.610735+0000) 2020-02-05T20:39:50.613 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:50.610+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.611453+0000) 2020-02-05T20:39:50.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:50.610+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.611999+0000) 2020-02-05T20:39:50.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:50.612+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.612901+0000) 2020-02-05T20:39:50.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:50.613+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.613896+0000) 2020-02-05T20:39:50.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:50.612+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.613886+0000) 2020-02-05T20:39:50.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:50.613+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.614196+0000) 2020-02-05T20:39:50.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:50.614+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.615105+0000) 2020-02-05T20:39:50.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:50.614+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.615591+0000) 2020-02-05T20:39:50.618 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:50.614+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.615897+0000) 2020-02-05T20:39:50.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:50.614+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.615125+0000) 2020-02-05T20:39:50.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:50.615+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.615687+0000) 2020-02-05T20:39:50.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:50.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:50.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:50.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:50.622835+0000) 2020-02-05T20:39:52.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:52.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:52.160756+0000) 2020-02-05T20:39:52.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:52.276+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:52.276537+0000) 2020-02-05T20:39:53.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:53.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:53.402970+0000) 2020-02-05T20:39:53.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:53.433+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:53.434036+0000) 2020-02-05T20:39:54.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:54.622+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:54.623659+0000) 2020-02-05T20:39:55.626 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:55.624+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.625340+0000) 2020-02-05T20:39:55.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:39:55.624+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.625845+0000) 2020-02-05T20:39:55.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:55.626+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.627383+0000) 2020-02-05T20:39:55.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:39:55.626+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.627265+0000) 2020-02-05T20:39:55.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:39:55.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.626903+0000) 2020-02-05T20:39:55.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:55.625+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.626387+0000) 2020-02-05T20:39:55.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:55.627+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.628041+0000) 2020-02-05T20:39:55.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:55.627+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.628459+0000) 2020-02-05T20:39:55.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:39:55.627+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.628741+0000) 2020-02-05T20:39:55.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:39:55.628+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.629307+0000) 2020-02-05T20:39:55.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:39:55.628+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.629253+0000) 2020-02-05T20:39:55.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:55.629+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.629639+0000) 2020-02-05T20:39:55.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:39:55.630+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:39:55.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:55.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.632188+0000) 2020-02-05T20:39:55.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:55.633+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.634410+0000) 2020-02-05T20:39:55.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:39:55.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.778543+0000) 2020-02-05T20:39:55.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:55.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:55.815574+0000) 2020-02-05T20:39:56.413 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:39:56.420 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:39:56.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:39:56.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:56.675937+0000) 2020-02-05T20:39:56.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:39:56.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:56.918778+0000) 2020-02-05T20:39:56.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:39:56.920+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:56.920655+0000) 2020-02-05T20:39:59.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:39:59.312+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:59.313376+0000) 2020-02-05T20:39:59.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:39:59.921+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:39:59.922859+0000) 2020-02-05T20:40:00.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:00.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.171730+0000) 2020-02-05T20:40:00.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:00.182+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.183367+0000) 2020-02-05T20:40:00.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:00.635+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.636607+0000) 2020-02-05T20:40:00.638 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:00.636+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.637656+0000) 2020-02-05T20:40:00.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:00.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.638117+0000) 2020-02-05T20:40:00.639 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:00.637+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.638472+0000) 2020-02-05T20:40:00.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:00.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.638750+0000) 2020-02-05T20:40:00.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:00.638+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.639135+0000) 2020-02-05T20:40:00.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:00.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.639466+0000) 2020-02-05T20:40:00.641 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:00.638+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.640221+0000) 2020-02-05T20:40:00.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:00.639+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.640470+0000) 2020-02-05T20:40:00.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:00.640+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.641703+0000) 2020-02-05T20:40:00.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:00.639+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.641275+0000) 2020-02-05T20:40:00.644 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:00.640+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.642270+0000) 2020-02-05T20:40:00.644 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:00.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.642319+0000) 2020-02-05T20:40:00.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:00.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:00.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:00.645+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:00.646185+0000) 2020-02-05T20:40:02.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:02.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:02.161019+0000) 2020-02-05T20:40:02.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:02.275+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:02.276798+0000) 2020-02-05T20:40:03.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:03.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:03.403212+0000) 2020-02-05T20:40:03.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:03.433+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:03.434318+0000) 2020-02-05T20:40:04.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:04.631+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:04.631934+0000) 2020-02-05T20:40:05.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:05.648+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.649409+0000) 2020-02-05T20:40:05.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:05.648+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.649735+0000) 2020-02-05T20:40:05.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:05.649+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.650905+0000) 2020-02-05T20:40:05.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:05.649+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.651370+0000) 2020-02-05T20:40:05.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:05.651+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.652610+0000) 2020-02-05T20:40:05.655 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:05.652+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.653401+0000) 2020-02-05T20:40:05.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:05.653+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.654181+0000) 2020-02-05T20:40:05.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:05.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.654868+0000) 2020-02-05T20:40:05.657 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:05.653+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.655123+0000) 2020-02-05T20:40:05.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:05.654+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.655573+0000) 2020-02-05T20:40:05.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:05.654+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.655204+0000) 2020-02-05T20:40:05.658 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:05.654+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.655615+0000) 2020-02-05T20:40:05.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:05.654+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.656026+0000) 2020-02-05T20:40:05.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:05.655+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63167 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:05.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:05.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.659331+0000) 2020-02-05T20:40:05.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:05.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.778786+0000) 2020-02-05T20:40:05.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:05.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:05.815813+0000) 2020-02-05T20:40:06.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:06.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:06.676180+0000) 2020-02-05T20:40:06.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:06.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:06.919037+0000) 2020-02-05T20:40:06.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:06.919+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:06.920898+0000) 2020-02-05T20:40:09.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:09.312+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:09.313631+0000) 2020-02-05T20:40:09.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:09.922+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:09.923113+0000) 2020-02-05T20:40:10.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:10.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.172002+0000) 2020-02-05T20:40:10.185 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:10.182+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.183593+0000) 2020-02-05T20:40:10.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:10.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.661413+0000) 2020-02-05T20:40:10.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:10.661+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.662812+0000) 2020-02-05T20:40:10.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:10.662+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.663351+0000) 2020-02-05T20:40:10.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:10.662+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.664053+0000) 2020-02-05T20:40:10.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:10.663+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.664572+0000) 2020-02-05T20:40:10.666 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:10.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.664756+0000) 2020-02-05T20:40:10.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:10.664+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.665354+0000) 2020-02-05T20:40:10.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:10.663+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.665170+0000) 2020-02-05T20:40:10.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:10.664+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.665982+0000) 2020-02-05T20:40:10.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:10.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.666520+0000) 2020-02-05T20:40:10.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:10.666+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.667681+0000) 2020-02-05T20:40:10.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:10.666+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.667990+0000) 2020-02-05T20:40:10.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:10.667+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.668752+0000) 2020-02-05T20:40:10.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:10.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63209 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:10.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:10.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:10.670946+0000) 2020-02-05T20:40:12.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:12.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:12.161334+0000) 2020-02-05T20:40:12.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:12.276+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:12.277057+0000) 2020-02-05T20:40:13.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:13.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:13.403490+0000) 2020-02-05T20:40:13.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:13.434+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:13.434561+0000) 2020-02-05T20:40:14.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:14.639+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:14.640423+0000) 2020-02-05T20:40:15.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:15.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.673541+0000) 2020-02-05T20:40:15.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:15.673+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.674469+0000) 2020-02-05T20:40:15.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:15.673+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.674913+0000) 2020-02-05T20:40:15.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:15.674+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.675608+0000) 2020-02-05T20:40:15.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:15.674+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.675825+0000) 2020-02-05T20:40:15.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:15.674+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.676164+0000) 2020-02-05T20:40:15.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:15.675+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.676869+0000) 2020-02-05T20:40:15.680 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:15.676+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.677492+0000) 2020-02-05T20:40:15.680 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:15.676+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.677928+0000) 2020-02-05T20:40:15.680 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:15.677+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.678405+0000) 2020-02-05T20:40:15.681 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:15.676+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.678279+0000) 2020-02-05T20:40:15.681 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:15.678+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.680291+0000) 2020-02-05T20:40:15.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:15.679+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.681007+0000) 2020-02-05T20:40:15.684 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:15.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63248 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:15.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:15.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.683726+0000) 2020-02-05T20:40:15.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:15.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.778996+0000) 2020-02-05T20:40:15.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:15.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:15.816040+0000) 2020-02-05T20:40:16.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:16.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:16.676371+0000) 2020-02-05T20:40:16.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:16.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:16.919336+0000) 2020-02-05T20:40:16.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:16.920+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:16.921143+0000) 2020-02-05T20:40:19.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:19.313+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:19.313853+0000) 2020-02-05T20:40:19.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:19.922+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:19.923310+0000) 2020-02-05T20:40:20.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:20.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.172264+0000) 2020-02-05T20:40:20.185 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:20.183+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.183838+0000) 2020-02-05T20:40:20.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:20.686+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.687696+0000) 2020-02-05T20:40:20.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:20.687+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.688815+0000) 2020-02-05T20:40:20.691 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:20.688+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.689492+0000) 2020-02-05T20:40:20.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:20.689+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.690884+0000) 2020-02-05T20:40:20.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:20.689+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.690651+0000) 2020-02-05T20:40:20.693 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:20.690+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.691407+0000) 2020-02-05T20:40:20.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:20.690+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.691965+0000) 2020-02-05T20:40:20.693 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:20.690+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.692247+0000) 2020-02-05T20:40:20.694 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:20.691+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.692657+0000) 2020-02-05T20:40:20.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:20.692+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.693375+0000) 2020-02-05T20:40:20.695 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:20.692+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.693797+0000) 2020-02-05T20:40:20.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:20.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.694470+0000) 2020-02-05T20:40:20.696 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:20.693+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.694805+0000) 2020-02-05T20:40:20.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:20.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:20.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:20.699+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:20.700900+0000) 2020-02-05T20:40:22.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:22.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:22.161651+0000) 2020-02-05T20:40:22.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:22.276+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:22.277317+0000) 2020-02-05T20:40:23.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:23.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:23.403750+0000) 2020-02-05T20:40:23.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:23.434+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:23.434800+0000) 2020-02-05T20:40:24.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:24.647+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:24.648919+0000) 2020-02-05T20:40:25.704 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:25.701+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.702512+0000) 2020-02-05T20:40:25.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:25.700+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.701991+0000) 2020-02-05T20:40:25.705 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:25.701+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.703039+0000) 2020-02-05T20:40:25.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:25.702+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.703721+0000) 2020-02-05T20:40:25.705 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:25.702+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.704327+0000) 2020-02-05T20:40:25.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:25.703+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.704447+0000) 2020-02-05T20:40:25.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:25.703+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.704935+0000) 2020-02-05T20:40:25.707 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:25.704+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.705266+0000) 2020-02-05T20:40:25.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:25.704+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.705735+0000) 2020-02-05T20:40:25.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:25.704+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.705706+0000) 2020-02-05T20:40:25.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:25.705+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.706698+0000) 2020-02-05T20:40:25.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:25.705+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.706493+0000) 2020-02-05T20:40:25.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:25.707+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63331 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:25.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:25.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.709925+0000) 2020-02-05T20:40:25.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:25.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.710218+0000) 2020-02-05T20:40:25.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:25.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.779209+0000) 2020-02-05T20:40:25.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:25.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:25.816344+0000) 2020-02-05T20:40:26.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:26.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:26.676581+0000) 2020-02-05T20:40:26.694 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:40:26.701 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:40:26.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:26.918+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:26.919573+0000) 2020-02-05T20:40:26.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:26.920+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:26.921403+0000) 2020-02-05T20:40:29.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:29.313+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:29.314146+0000) 2020-02-05T20:40:29.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:29.923+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:29.923610+0000) 2020-02-05T20:40:30.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:30.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.172534+0000) 2020-02-05T20:40:30.185 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:30.183+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.184133+0000) 2020-02-05T20:40:30.716 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:30.714+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.715139+0000) 2020-02-05T20:40:30.718 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:30.714+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.716312+0000) 2020-02-05T20:40:30.719 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:30.716+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.717563+0000) 2020-02-05T20:40:30.719 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:30.716+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.718233+0000) 2020-02-05T20:40:30.720 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:30.717+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.719027+0000) 2020-02-05T20:40:30.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:30.718+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.719261+0000) 2020-02-05T20:40:30.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:30.718+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.719574+0000) 2020-02-05T20:40:30.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:30.719+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.720365+0000) 2020-02-05T20:40:30.722 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:30.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.720568+0000) 2020-02-05T20:40:30.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:30.719+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.720829+0000) 2020-02-05T20:40:30.723 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:30.719+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.720606+0000) 2020-02-05T20:40:30.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:30.719+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.720323+0000) 2020-02-05T20:40:30.723 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:30.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:30.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:30.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.725414+0000) 2020-02-05T20:40:30.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:30.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:30.725688+0000) 2020-02-05T20:40:32.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:32.161+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:32.161958+0000) 2020-02-05T20:40:32.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:32.276+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:32.277581+0000) 2020-02-05T20:40:33.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:33.403+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:33.404006+0000) 2020-02-05T20:40:33.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:33.434+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:33.435062+0000) 2020-02-05T20:40:34.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:34.656+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:34.657117+0000) 2020-02-05T20:40:35.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:35.726+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.727843+0000) 2020-02-05T20:40:35.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:35.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.729246+0000) 2020-02-05T20:40:35.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:35.729+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.730383+0000) 2020-02-05T20:40:35.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:35.728+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.730304+0000) 2020-02-05T20:40:35.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:35.728+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.729779+0000) 2020-02-05T20:40:35.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:35.730+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.731204+0000) 2020-02-05T20:40:35.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:35.729+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.730790+0000) 2020-02-05T20:40:35.734 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:35.730+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.732336+0000) 2020-02-05T20:40:35.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:35.731+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.732814+0000) 2020-02-05T20:40:35.734 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:35.731+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.733324+0000) 2020-02-05T20:40:35.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:35.732+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.733729+0000) 2020-02-05T20:40:35.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:35.732+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.734269+0000) 2020-02-05T20:40:35.736 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:35.733+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.734921+0000) 2020-02-05T20:40:35.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:35.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63412 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:35.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:35.740+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.741319+0000) 2020-02-05T20:40:35.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:35.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.779417+0000) 2020-02-05T20:40:35.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:35.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:35.816556+0000) 2020-02-05T20:40:36.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:36.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:36.676792+0000) 2020-02-05T20:40:36.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:36.919+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:36.919834+0000) 2020-02-05T20:40:36.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:36.921+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:36.921710+0000) 2020-02-05T20:40:39.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:39.314+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:39.314382+0000) 2020-02-05T20:40:39.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:39.923+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:39.923846+0000) 2020-02-05T20:40:40.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:40.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.172788+0000) 2020-02-05T20:40:40.185 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:40.184+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.184362+0000) 2020-02-05T20:40:40.743 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:40.741+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.742053+0000) 2020-02-05T20:40:40.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:40.743+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.743585+0000) 2020-02-05T20:40:40.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:40.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.744352+0000) 2020-02-05T20:40:40.747 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:40.743+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.744823+0000) 2020-02-05T20:40:40.747 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:40.744+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.745548+0000) 2020-02-05T20:40:40.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:40.744+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.746090+0000) 2020-02-05T20:40:40.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:40.747+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.747250+0000) 2020-02-05T20:40:40.749 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:40.746+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.747899+0000) 2020-02-05T20:40:40.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:40.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.748307+0000) 2020-02-05T20:40:40.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:40.749+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.749202+0000) 2020-02-05T20:40:40.751 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:40.747+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.749252+0000) 2020-02-05T20:40:40.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:40.748+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.749523+0000) 2020-02-05T20:40:40.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:40.749+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.749806+0000) 2020-02-05T20:40:40.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:40.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63455 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:40.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:40.752+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:40.752606+0000) 2020-02-05T20:40:42.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:42.161+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:42.162198+0000) 2020-02-05T20:40:42.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:42.277+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:42.277844+0000) 2020-02-05T20:40:43.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:43.403+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:43.404244+0000) 2020-02-05T20:40:43.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:43.434+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:43.435350+0000) 2020-02-05T20:40:44.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:44.665+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:44.666144+0000) 2020-02-05T20:40:45.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:45.754+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.755401+0000) 2020-02-05T20:40:45.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:45.755+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.756072+0000) 2020-02-05T20:40:45.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:45.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.756708+0000) 2020-02-05T20:40:45.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:45.757+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.758452+0000) 2020-02-05T20:40:45.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:45.758+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.759254+0000) 2020-02-05T20:40:45.761 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:45.758+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.759742+0000) 2020-02-05T20:40:45.761 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:45.758+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.760138+0000) 2020-02-05T20:40:45.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:45.759+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.760756+0000) 2020-02-05T20:40:45.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:45.760+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.761567+0000) 2020-02-05T20:40:45.763 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:45.761+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.762707+0000) 2020-02-05T20:40:45.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:45.761+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.763004+0000) 2020-02-05T20:40:45.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:45.762+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.763357+0000) 2020-02-05T20:40:45.765 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:45.761+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.763331+0000) 2020-02-05T20:40:45.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:45.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63496 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:45.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:45.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.765577+0000) 2020-02-05T20:40:45.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:45.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.779607+0000) 2020-02-05T20:40:45.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:45.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:45.816779+0000) 2020-02-05T20:40:46.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:46.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:46.677023+0000) 2020-02-05T20:40:46.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:46.919+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:46.920091+0000) 2020-02-05T20:40:46.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:46.921+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:46.921956+0000) 2020-02-05T20:40:49.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:49.313+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:49.314560+0000) 2020-02-05T20:40:49.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:49.923+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:49.924125+0000) 2020-02-05T20:40:50.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:50.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.173003+0000) 2020-02-05T20:40:50.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:50.183+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.184636+0000) 2020-02-05T20:40:50.771 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:50.769+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.769815+0000) 2020-02-05T20:40:50.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:50.770+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.771223+0000) 2020-02-05T20:40:50.773 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:50.771+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.771565+0000) 2020-02-05T20:40:50.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:50.771+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.772595+0000) 2020-02-05T20:40:50.775 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:50.772+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.773646+0000) 2020-02-05T20:40:50.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:50.772+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.773341+0000) 2020-02-05T20:40:50.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:50.773+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.774300+0000) 2020-02-05T20:40:50.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:50.773+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.774718+0000) 2020-02-05T20:40:50.776 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:50.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.774672+0000) 2020-02-05T20:40:50.777 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:50.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.774844+0000) 2020-02-05T20:40:50.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:50.774+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.774815+0000) 2020-02-05T20:40:50.777 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:50.774+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.775597+0000) 2020-02-05T20:40:50.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:50.775+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.775840+0000) 2020-02-05T20:40:50.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:50.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:50.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:50.777+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:50.778577+0000) 2020-02-05T20:40:52.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:52.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:52.162495+0000) 2020-02-05T20:40:52.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:52.277+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:52.278091+0000) 2020-02-05T20:40:53.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:53.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:53.404485+0000) 2020-02-05T20:40:53.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:53.434+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:53.435585+0000) 2020-02-05T20:40:54.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:54.676+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:54.676858+0000) 2020-02-05T20:40:55.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:55.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.779780+0000) 2020-02-05T20:40:55.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:55.783+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.784138+0000) 2020-02-05T20:40:55.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:55.783+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.784708+0000) 2020-02-05T20:40:55.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:40:55.784+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.785304+0000) 2020-02-05T20:40:55.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:55.785+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.785706+0000) 2020-02-05T20:40:55.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:40:55.786+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.786906+0000) 2020-02-05T20:40:55.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:40:55.786+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.787116+0000) 2020-02-05T20:40:55.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:40:55.787+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.787737+0000) 2020-02-05T20:40:55.789 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:40:55.787+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.788338+0000) 2020-02-05T20:40:55.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:55.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.788344+0000) 2020-02-05T20:40:55.791 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:55.787+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.788621+0000) 2020-02-05T20:40:55.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:40:55.787+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.788880+0000) 2020-02-05T20:40:55.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:40:55.788+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.788868+0000) 2020-02-05T20:40:55.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:55.787+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.787892+0000) 2020-02-05T20:40:55.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:40:55.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:40:55.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:55.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.794809+0000) 2020-02-05T20:40:55.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:55.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:55.817015+0000) 2020-02-05T20:40:56.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:40:56.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:56.677266+0000) 2020-02-05T20:40:56.852 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:40:56.858 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:40:56.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:40:56.919+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:56.920279+0000) 2020-02-05T20:40:56.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:40:56.921+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:56.922157+0000) 2020-02-05T20:40:59.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:40:59.314+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:59.314817+0000) 2020-02-05T20:40:59.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:40:59.923+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:40:59.924312+0000) 2020-02-05T20:41:00.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:00.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.173241+0000) 2020-02-05T20:41:00.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:00.184+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.184859+0000) 2020-02-05T20:41:00.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:00.794+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.795867+0000) 2020-02-05T20:41:00.798 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:00.795+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.796712+0000) 2020-02-05T20:41:00.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:00.796+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.797509+0000) 2020-02-05T20:41:00.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:00.796+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.797926+0000) 2020-02-05T20:41:00.800 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:00.798+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.798697+0000) 2020-02-05T20:41:00.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:00.797+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.798410+0000) 2020-02-05T20:41:00.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:00.798+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.798917+0000) 2020-02-05T20:41:00.801 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:00.798+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.799201+0000) 2020-02-05T20:41:00.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:00.798+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.799585+0000) 2020-02-05T20:41:00.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:00.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.800073+0000) 2020-02-05T20:41:00.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:00.799+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.800137+0000) 2020-02-05T20:41:00.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:00.800+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.800953+0000) 2020-02-05T20:41:00.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:00.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:00.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:00.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.806759+0000) 2020-02-05T20:41:00.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:00.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:00.808511+0000) 2020-02-05T20:41:02.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:02.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:02.162717+0000) 2020-02-05T20:41:02.280 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:02.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:02.278346+0000) 2020-02-05T20:41:03.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:03.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:03.404725+0000) 2020-02-05T20:41:03.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:03.435+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:03.435827+0000) 2020-02-05T20:41:04.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:04.684+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:04.685249+0000) 2020-02-05T20:41:05.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:05.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.780051+0000) 2020-02-05T20:41:05.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:05.808+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.809628+0000) 2020-02-05T20:41:05.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:05.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.810027+0000) 2020-02-05T20:41:05.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:05.809+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.810476+0000) 2020-02-05T20:41:05.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:05.810+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.811318+0000) 2020-02-05T20:41:05.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:05.810+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.811919+0000) 2020-02-05T20:41:05.813 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:05.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.811913+0000) 2020-02-05T20:41:05.814 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:05.812+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.813407+0000) 2020-02-05T20:41:05.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:05.812+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.813682+0000) 2020-02-05T20:41:05.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:05.813+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.814143+0000) 2020-02-05T20:41:05.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:05.814+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.814416+0000) 2020-02-05T20:41:05.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:05.813+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.814728+0000) 2020-02-05T20:41:05.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:05.814+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.814995+0000) 2020-02-05T20:41:05.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:05.815+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.816075+0000) 2020-02-05T20:41:05.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:05.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63662 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:05.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:05.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.817131+0000) 2020-02-05T20:41:05.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:05.818+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:05.819425+0000) 2020-02-05T20:41:06.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:06.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:06.677470+0000) 2020-02-05T20:41:06.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:06.920+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:06.920521+0000) 2020-02-05T20:41:06.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:06.922+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:06.922394+0000) 2020-02-05T20:41:09.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:09.314+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:09.315070+0000) 2020-02-05T20:41:09.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:09.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:09.924546+0000) 2020-02-05T20:41:10.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:10.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.173529+0000) 2020-02-05T20:41:10.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:10.184+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.185116+0000) 2020-02-05T20:41:10.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:10.821+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.822403+0000) 2020-02-05T20:41:10.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:10.822+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.823438+0000) 2020-02-05T20:41:10.826 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:10.824+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.824535+0000) 2020-02-05T20:41:10.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:10.825+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.825708+0000) 2020-02-05T20:41:10.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:10.824+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.826028+0000) 2020-02-05T20:41:10.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:10.825+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.826457+0000) 2020-02-05T20:41:10.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:10.825+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.826940+0000) 2020-02-05T20:41:10.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:10.825+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.826963+0000) 2020-02-05T20:41:10.829 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:10.827+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.827641+0000) 2020-02-05T20:41:10.829 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:10.827+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.827757+0000) 2020-02-05T20:41:10.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:10.827+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.828711+0000) 2020-02-05T20:41:10.830 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:10.828+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.828720+0000) 2020-02-05T20:41:10.830 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:10.828+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.828981+0000) 2020-02-05T20:41:10.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:10.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:10.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:10.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:10.830984+0000) 2020-02-05T20:41:12.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:12.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:12.162978+0000) 2020-02-05T20:41:12.280 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:12.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:12.278619+0000) 2020-02-05T20:41:13.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:13.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:13.404981+0000) 2020-02-05T20:41:13.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:13.435+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:13.436063+0000) 2020-02-05T20:41:14.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:14.693+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:14.693568+0000) 2020-02-05T20:41:15.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:15.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.780318+0000) 2020-02-05T20:41:15.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:15.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.817303+0000) 2020-02-05T20:41:15.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:15.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.835748+0000) 2020-02-05T20:41:15.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:15.836+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.837285+0000) 2020-02-05T20:41:15.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:15.837+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.837658+0000) 2020-02-05T20:41:15.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:15.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.838242+0000) 2020-02-05T20:41:15.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:15.837+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.838305+0000) 2020-02-05T20:41:15.840 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:15.838+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.838726+0000) 2020-02-05T20:41:15.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:15.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.838785+0000) 2020-02-05T20:41:15.841 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:15.838+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.839012+0000) 2020-02-05T20:41:15.841 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:15.838+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.838972+0000) 2020-02-05T20:41:15.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:15.838+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.839641+0000) 2020-02-05T20:41:15.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:15.840+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.840508+0000) 2020-02-05T20:41:15.842 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:15.840+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.840759+0000) 2020-02-05T20:41:15.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:15.839+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.840172+0000) 2020-02-05T20:41:15.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:15.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:15.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:15.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:15.847487+0000) 2020-02-05T20:41:16.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:16.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:16.677670+0000) 2020-02-05T20:41:16.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:16.920+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:16.920774+0000) 2020-02-05T20:41:16.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:16.922+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:16.922610+0000) 2020-02-05T20:41:19.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:19.314+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:19.315317+0000) 2020-02-05T20:41:19.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:19.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:19.924780+0000) 2020-02-05T20:41:20.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:20.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.173749+0000) 2020-02-05T20:41:20.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:20.185+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.185347+0000) 2020-02-05T20:41:20.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:20.849+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.850186+0000) 2020-02-05T20:41:20.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:20.850+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.850647+0000) 2020-02-05T20:41:20.853 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:20.851+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.851658+0000) 2020-02-05T20:41:20.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:20.851+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.851847+0000) 2020-02-05T20:41:20.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:20.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.852107+0000) 2020-02-05T20:41:20.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:20.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.852503+0000) 2020-02-05T20:41:20.854 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:20.852+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.853341+0000) 2020-02-05T20:41:20.855 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:20.853+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.853732+0000) 2020-02-05T20:41:20.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:20.853+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.854064+0000) 2020-02-05T20:41:20.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:20.853+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.854464+0000) 2020-02-05T20:41:20.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:20.853+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.854748+0000) 2020-02-05T20:41:20.856 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:20.853+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.854793+0000) 2020-02-05T20:41:20.857 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:20.855+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.855545+0000) 2020-02-05T20:41:20.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:20.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63786 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:20.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:20.860+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:20.861436+0000) 2020-02-05T20:41:22.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:22.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:22.163240+0000) 2020-02-05T20:41:22.280 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:22.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:22.278872+0000) 2020-02-05T20:41:23.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:23.405+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:23.405227+0000) 2020-02-05T20:41:23.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:23.436+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:23.436374+0000) 2020-02-05T20:41:24.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:24.705+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:24.705540+0000) 2020-02-05T20:41:25.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:25.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.780548+0000) 2020-02-05T20:41:25.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:25.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.817541+0000) 2020-02-05T20:41:25.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:25.861+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.862484+0000) 2020-02-05T20:41:25.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:25.861+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.862951+0000) 2020-02-05T20:41:25.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:25.863+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.864740+0000) 2020-02-05T20:41:25.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:25.863+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.865064+0000) 2020-02-05T20:41:25.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:25.864+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.865542+0000) 2020-02-05T20:41:25.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:25.863+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.865380+0000) 2020-02-05T20:41:25.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:25.864+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.866132+0000) 2020-02-05T20:41:25.871 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:25.865+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.867344+0000) 2020-02-05T20:41:25.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:25.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:25.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:25.862+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.863315+0000) 2020-02-05T20:41:25.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:25.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.864427+0000) 2020-02-05T20:41:25.872 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:25.863+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.864540+0000) 2020-02-05T20:41:25.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:25.863+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.865012+0000) 2020-02-05T20:41:25.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:25.866+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.867143+0000) 2020-02-05T20:41:25.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:25.869+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:25.870700+0000) 2020-02-05T20:41:26.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:26.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:26.677882+0000) 2020-02-05T20:41:26.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:26.919+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:26.921070+0000) 2020-02-05T20:41:26.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:26.921+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:26.922800+0000) 2020-02-05T20:41:26.991 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:41:26.997 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:41:29.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:29.315+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:29.315585+0000) 2020-02-05T20:41:29.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:29.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:29.925021+0000) 2020-02-05T20:41:30.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:30.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.173978+0000) 2020-02-05T20:41:30.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:30.185+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.185595+0000) 2020-02-05T20:41:30.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:30.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.875434+0000) 2020-02-05T20:41:30.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:30.875+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.875886+0000) 2020-02-05T20:41:30.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:30.875+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.876305+0000) 2020-02-05T20:41:30.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:30.875+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.876668+0000) 2020-02-05T20:41:30.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:30.876+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.877277+0000) 2020-02-05T20:41:30.880 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:30.876+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.877678+0000) 2020-02-05T20:41:30.881 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:30.877+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.878618+0000) 2020-02-05T20:41:30.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:30.878+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.878465+0000) 2020-02-05T20:41:30.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:30.878+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.879147+0000) 2020-02-05T20:41:30.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:30.878+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.879703+0000) 2020-02-05T20:41:30.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:30.879+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.880579+0000) 2020-02-05T20:41:30.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:30.879+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.881146+0000) 2020-02-05T20:41:30.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:30.880+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.881464+0000) 2020-02-05T20:41:30.884 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:30.880+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63869 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:30.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:30.886+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:30.887894+0000) 2020-02-05T20:41:32.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:32.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:32.163498+0000) 2020-02-05T20:41:32.280 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:32.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:32.279128+0000) 2020-02-05T20:41:33.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:33.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:33.405484+0000) 2020-02-05T20:41:33.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:33.436+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:33.436658+0000) 2020-02-05T20:41:34.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:34.714+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:34.714181+0000) 2020-02-05T20:41:35.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:35.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.780774+0000) 2020-02-05T20:41:35.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:35.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.817773+0000) 2020-02-05T20:41:35.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:35.886+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.887922+0000) 2020-02-05T20:41:35.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:35.887+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.888561+0000) 2020-02-05T20:41:35.891 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:35.888+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.889148+0000) 2020-02-05T20:41:35.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:35.888+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.889416+0000) 2020-02-05T20:41:35.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:35.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.890459+0000) 2020-02-05T20:41:35.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:35.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.890774+0000) 2020-02-05T20:41:35.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:35.889+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.890874+0000) 2020-02-05T20:41:35.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:35.890+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.891506+0000) 2020-02-05T20:41:35.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:35.890+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.892027+0000) 2020-02-05T20:41:35.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:35.890+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.892295+0000) 2020-02-05T20:41:35.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:35.891+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.893158+0000) 2020-02-05T20:41:35.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:35.891+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.893234+0000) 2020-02-05T20:41:35.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:35.892+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.893894+0000) 2020-02-05T20:41:35.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:35.893+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:35.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:35.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:35.899418+0000) 2020-02-05T20:41:36.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:36.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:36.678118+0000) 2020-02-05T20:41:36.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:36.920+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:36.921322+0000) 2020-02-05T20:41:36.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:36.921+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:36.923030+0000) 2020-02-05T20:41:39.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:39.314+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:39.315822+0000) 2020-02-05T20:41:39.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:39.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:39.925298+0000) 2020-02-05T20:41:40.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:40.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.174281+0000) 2020-02-05T20:41:40.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:40.184+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.185822+0000) 2020-02-05T20:41:40.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:40.898+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.899470+0000) 2020-02-05T20:41:40.901 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:40.899+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.900419+0000) 2020-02-05T20:41:40.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:40.900+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.901781+0000) 2020-02-05T20:41:40.903 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:40.900+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.901565+0000) 2020-02-05T20:41:40.904 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:40.901+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.902452+0000) 2020-02-05T20:41:40.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:40.901+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.902816+0000) 2020-02-05T20:41:40.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:40.901+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.902701+0000) 2020-02-05T20:41:40.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:40.902+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.903346+0000) 2020-02-05T20:41:40.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:40.902+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.904055+0000) 2020-02-05T20:41:40.905 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:40.902+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.904312+0000) 2020-02-05T20:41:40.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:40.903+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.904826+0000) 2020-02-05T20:41:40.906 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:40.904+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.905596+0000) 2020-02-05T20:41:40.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:40.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63952 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:40.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:40.908+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.909251+0000) 2020-02-05T20:41:40.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:40.908+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:40.909510+0000) 2020-02-05T20:41:42.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:42.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:42.163769+0000) 2020-02-05T20:41:42.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:42.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:42.279468+0000) 2020-02-05T20:41:43.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:43.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:43.405719+0000) 2020-02-05T20:41:43.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:43.435+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:43.436890+0000) 2020-02-05T20:41:44.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:44.720+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:44.721358+0000) 2020-02-05T20:41:45.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:45.780+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.781008+0000) 2020-02-05T20:41:45.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:45.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.818010+0000) 2020-02-05T20:41:45.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:45.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.913991+0000) 2020-02-05T20:41:45.915 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:45.913+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.914599+0000) 2020-02-05T20:41:45.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:45.915+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.916235+0000) 2020-02-05T20:41:45.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:45.915+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.916833+0000) 2020-02-05T20:41:45.918 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:45.915+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.917274+0000) 2020-02-05T20:41:45.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:45.916+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.918034+0000) 2020-02-05T20:41:45.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:45.916+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.918139+0000) 2020-02-05T20:41:45.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:45.917+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.918899+0000) 2020-02-05T20:41:45.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:45.917+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.919219+0000) 2020-02-05T20:41:45.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:45.918+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.919687+0000) 2020-02-05T20:41:45.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:45.918+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.919530+0000) 2020-02-05T20:41:45.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:45.919+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.920218+0000) 2020-02-05T20:41:45.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:45.919+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.920290+0000) 2020-02-05T20:41:45.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:45.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 63994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:45.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:45.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:45.923038+0000) 2020-02-05T20:41:46.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:46.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:46.678349+0000) 2020-02-05T20:41:46.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:46.920+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:46.921648+0000) 2020-02-05T20:41:46.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:46.922+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:46.923276+0000) 2020-02-05T20:41:49.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:49.315+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:49.316111+0000) 2020-02-05T20:41:49.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:49.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:49.925518+0000) 2020-02-05T20:41:50.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:50.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.174536+0000) 2020-02-05T20:41:50.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:50.185+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.186125+0000) 2020-02-05T20:41:50.929 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:50.925+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.927268+0000) 2020-02-05T20:41:50.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:50.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.928368+0000) 2020-02-05T20:41:50.930 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:50.927+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.928740+0000) 2020-02-05T20:41:50.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:50.928+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.929391+0000) 2020-02-05T20:41:50.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:50.928+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.929502+0000) 2020-02-05T20:41:50.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:50.929+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.930481+0000) 2020-02-05T20:41:50.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:50.928+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.930393+0000) 2020-02-05T20:41:50.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:50.929+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.930888+0000) 2020-02-05T20:41:50.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:50.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.931805+0000) 2020-02-05T20:41:50.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:50.930+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.931753+0000) 2020-02-05T20:41:50.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:50.931+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.932709+0000) 2020-02-05T20:41:50.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:50.931+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.932680+0000) 2020-02-05T20:41:50.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:50.931+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.933093+0000) 2020-02-05T20:41:50.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:50.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64035 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:50.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:50.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:50.938327+0000) 2020-02-05T20:41:52.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:52.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:52.164023+0000) 2020-02-05T20:41:52.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:52.278+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:52.279733+0000) 2020-02-05T20:41:53.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:53.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:53.405966+0000) 2020-02-05T20:41:53.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:53.436+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:53.437147+0000) 2020-02-05T20:41:54.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:54.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:54.729738+0000) 2020-02-05T20:41:55.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:55.780+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.781281+0000) 2020-02-05T20:41:55.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:55.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.818277+0000) 2020-02-05T20:41:55.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:41:55.938+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.939983+0000) 2020-02-05T20:41:55.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:55.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.940641+0000) 2020-02-05T20:41:55.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:41:55.940+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.941234+0000) 2020-02-05T20:41:55.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:55.940+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.941591+0000) 2020-02-05T20:41:55.943 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:41:55.940+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.942131+0000) 2020-02-05T20:41:55.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:55.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.942711+0000) 2020-02-05T20:41:55.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:41:55.942+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.943335+0000) 2020-02-05T20:41:55.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:55.942+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.943501+0000) 2020-02-05T20:41:55.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:41:55.942+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.943546+0000) 2020-02-05T20:41:55.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:55.942+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.943589+0000) 2020-02-05T20:41:55.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:55.942+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.943789+0000) 2020-02-05T20:41:55.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:41:55.942+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.944325+0000) 2020-02-05T20:41:55.946 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:41:55.943+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.944470+0000) 2020-02-05T20:41:55.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:41:55.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:41:55.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:55.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:55.949997+0000) 2020-02-05T20:41:56.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:41:56.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:56.678555+0000) 2020-02-05T20:41:56.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:41:56.920+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:56.921889+0000) 2020-02-05T20:41:56.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:41:56.922+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:56.923542+0000) 2020-02-05T20:41:57.167 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:41:57.173 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:41:59.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:41:59.315+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:59.316343+0000) 2020-02-05T20:41:59.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:41:59.924+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:41:59.925755+0000) 2020-02-05T20:42:00.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:00.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.174759+0000) 2020-02-05T20:42:00.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:00.185+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.186303+0000) 2020-02-05T20:42:00.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:00.952+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.953053+0000) 2020-02-05T20:42:00.955 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:00.952+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.953870+0000) 2020-02-05T20:42:00.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:00.953+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.954339+0000) 2020-02-05T20:42:00.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:00.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.955307+0000) 2020-02-05T20:42:00.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:00.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.955861+0000) 2020-02-05T20:42:00.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:00.955+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.956478+0000) 2020-02-05T20:42:00.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:00.955+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.956700+0000) 2020-02-05T20:42:00.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:00.956+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.957550+0000) 2020-02-05T20:42:00.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:00.956+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.957909+0000) 2020-02-05T20:42:00.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:00.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.958071+0000) 2020-02-05T20:42:00.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:00.957+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.958487+0000) 2020-02-05T20:42:00.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:00.957+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.958761+0000) 2020-02-05T20:42:00.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:00.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.959270+0000) 2020-02-05T20:42:00.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:00.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:00.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:00.961+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:00.962411+0000) 2020-02-05T20:42:02.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:02.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:02.164283+0000) 2020-02-05T20:42:02.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:02.279+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:02.279969+0000) 2020-02-05T20:42:03.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:03.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:03.406211+0000) 2020-02-05T20:42:03.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:03.436+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:03.437385+0000) 2020-02-05T20:42:04.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:04.736+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:04.737003+0000) 2020-02-05T20:42:05.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:05.780+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.781504+0000) 2020-02-05T20:42:05.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:05.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.818504+0000) 2020-02-05T20:42:05.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:05.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.966462+0000) 2020-02-05T20:42:05.968 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:05.966+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.967284+0000) 2020-02-05T20:42:05.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:05.967+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.968515+0000) 2020-02-05T20:42:05.970 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:05.967+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.968930+0000) 2020-02-05T20:42:05.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:05.967+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.969282+0000) 2020-02-05T20:42:05.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:05.968+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.969646+0000) 2020-02-05T20:42:05.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:05.969+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.970286+0000) 2020-02-05T20:42:05.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:05.969+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.970584+0000) 2020-02-05T20:42:05.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:05.970+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.971242+0000) 2020-02-05T20:42:05.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:05.970+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.971656+0000) 2020-02-05T20:42:05.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:05.971+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.972532+0000) 2020-02-05T20:42:05.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:05.971+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.972532+0000) 2020-02-05T20:42:05.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:05.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.972995+0000) 2020-02-05T20:42:05.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:05.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64160 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:05.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:05.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:05.979688+0000) 2020-02-05T20:42:06.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:06.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:06.678779+0000) 2020-02-05T20:42:06.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:06.921+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:06.922110+0000) 2020-02-05T20:42:06.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:06.922+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:06.923765+0000) 2020-02-05T20:42:09.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:09.316+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:09.316579+0000) 2020-02-05T20:42:09.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:09.925+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:09.926008+0000) 2020-02-05T20:42:10.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:10.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.174999+0000) 2020-02-05T20:42:10.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:10.186+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.186561+0000) 2020-02-05T20:42:10.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:10.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.979337+0000) 2020-02-05T20:42:10.983 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:10.980+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.981686+0000) 2020-02-05T20:42:10.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:10.980+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.982047+0000) 2020-02-05T20:42:10.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:10.981+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.982381+0000) 2020-02-05T20:42:10.984 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:10.981+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.983024+0000) 2020-02-05T20:42:10.985 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:10.982+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.983656+0000) 2020-02-05T20:42:10.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:10.983+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.984222+0000) 2020-02-05T20:42:10.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:10.983+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.984560+0000) 2020-02-05T20:42:10.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:10.983+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.984905+0000) 2020-02-05T20:42:10.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:10.984+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.985361+0000) 2020-02-05T20:42:10.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:10.984+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.985517+0000) 2020-02-05T20:42:10.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:10.983+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.985039+0000) 2020-02-05T20:42:10.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:10.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.985896+0000) 2020-02-05T20:42:10.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:10.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64202 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:10.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:10.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:10.992091+0000) 2020-02-05T20:42:12.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:12.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:12.164522+0000) 2020-02-05T20:42:12.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:12.279+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:12.280273+0000) 2020-02-05T20:42:13.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:13.405+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:13.406477+0000) 2020-02-05T20:42:13.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:13.437+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:13.437646+0000) 2020-02-05T20:42:14.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:14.744+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:14.745173+0000) 2020-02-05T20:42:15.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:15.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.781762+0000) 2020-02-05T20:42:15.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:15.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.818733+0000) 2020-02-05T20:42:15.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:15.993+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.994443+0000) 2020-02-05T20:42:15.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:15.992+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.994326+0000) 2020-02-05T20:42:15.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:15.994+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.995736+0000) 2020-02-05T20:42:15.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:15.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.996302+0000) 2020-02-05T20:42:15.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:15.996+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.997124+0000) 2020-02-05T20:42:15.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:15.996+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.997632+0000) 2020-02-05T20:42:16.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:15.997+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.998236+0000) 2020-02-05T20:42:16.000 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:15.997+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.998858+0000) 2020-02-05T20:42:16.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:15.998+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:15.999500+0000) 2020-02-05T20:42:16.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:15.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.000140+0000) 2020-02-05T20:42:16.001 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:15.999+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.000415+0000) 2020-02-05T20:42:16.002 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:15.999+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.000420+0000) 2020-02-05T20:42:16.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:15.999+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.001035+0000) 2020-02-05T20:42:16.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:16.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:16.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:16.005+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.006519+0000) 2020-02-05T20:42:16.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:16.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.678981+0000) 2020-02-05T20:42:16.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:16.921+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.922344+0000) 2020-02-05T20:42:16.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:16.923+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:16.923966+0000) 2020-02-05T20:42:19.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:19.316+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:19.316813+0000) 2020-02-05T20:42:19.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:19.925+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:19.926227+0000) 2020-02-05T20:42:20.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:20.174+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:20.175249+0000) 2020-02-05T20:42:20.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:20.186+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:20.186796+0000) 2020-02-05T20:42:21.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:21.007+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.008220+0000) 2020-02-05T20:42:21.010 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:21.008+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.009403+0000) 2020-02-05T20:42:21.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:21.008+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.010007+0000) 2020-02-05T20:42:21.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:21.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.010381+0000) 2020-02-05T20:42:21.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:21.010+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.010880+0000) 2020-02-05T20:42:21.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:21.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.011189+0000) 2020-02-05T20:42:21.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:21.010+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.011715+0000) 2020-02-05T20:42:21.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:21.010+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.011986+0000) 2020-02-05T20:42:21.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:21.011+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.012093+0000) 2020-02-05T20:42:21.014 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:21.012+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.012480+0000) 2020-02-05T20:42:21.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:21.012+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.012740+0000) 2020-02-05T20:42:21.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:21.012+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.013109+0000) 2020-02-05T20:42:21.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:21.012+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.012829+0000) 2020-02-05T20:42:21.016 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:21.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:21.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:21.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:21.017760+0000) 2020-02-05T20:42:22.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:22.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:22.164781+0000) 2020-02-05T20:42:22.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:22.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:22.280507+0000) 2020-02-05T20:42:23.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:23.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:23.406738+0000) 2020-02-05T20:42:23.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:23.436+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:23.437910+0000) 2020-02-05T20:42:24.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:24.756+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:24.756719+0000) 2020-02-05T20:42:25.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:25.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:25.781998+0000) 2020-02-05T20:42:25.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:25.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:25.818966+0000) 2020-02-05T20:42:26.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:26.020+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.020426+0000) 2020-02-05T20:42:26.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:26.021+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.021310+0000) 2020-02-05T20:42:26.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:26.021+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.021747+0000) 2020-02-05T20:42:26.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:26.022+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.022063+0000) 2020-02-05T20:42:26.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:26.022+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.022911+0000) 2020-02-05T20:42:26.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:26.023+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.023978+0000) 2020-02-05T20:42:26.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:26.024+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.024690+0000) 2020-02-05T20:42:26.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:26.025+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.025677+0000) 2020-02-05T20:42:26.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:26.026+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.026578+0000) 2020-02-05T20:42:26.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:26.025+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.026140+0000) 2020-02-05T20:42:26.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:26.026+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.026883+0000) 2020-02-05T20:42:26.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:26.027+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.027633+0000) 2020-02-05T20:42:26.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:26.027+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.027687+0000) 2020-02-05T20:42:26.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:26.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:26.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:26.033+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.033892+0000) 2020-02-05T20:42:26.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:26.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.679203+0000) 2020-02-05T20:42:26.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:26.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.922581+0000) 2020-02-05T20:42:26.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:26.923+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:26.924192+0000) 2020-02-05T20:42:27.312 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:42:27.319 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:42:29.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:29.316+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:29.317043+0000) 2020-02-05T20:42:29.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:29.926+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:29.926458+0000) 2020-02-05T20:42:30.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:30.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:30.175541+0000) 2020-02-05T20:42:30.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:30.186+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:30.187056+0000) 2020-02-05T20:42:31.036 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:31.033+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.034435+0000) 2020-02-05T20:42:31.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:31.034+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.036033+0000) 2020-02-05T20:42:31.038 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:31.035+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.036612+0000) 2020-02-05T20:42:31.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:31.036+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.037082+0000) 2020-02-05T20:42:31.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:31.036+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.037955+0000) 2020-02-05T20:42:31.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:31.038+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.038851+0000) 2020-02-05T20:42:31.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:31.038+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.039095+0000) 2020-02-05T20:42:31.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:31.038+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.039707+0000) 2020-02-05T20:42:31.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:31.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.039827+0000) 2020-02-05T20:42:31.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:31.040+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.040636+0000) 2020-02-05T20:42:31.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:31.040+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.040943+0000) 2020-02-05T20:42:31.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:31.040+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.041262+0000) 2020-02-05T20:42:31.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:31.039+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.041019+0000) 2020-02-05T20:42:31.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:31.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:31.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:31.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:31.045263+0000) 2020-02-05T20:42:32.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:32.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:32.165073+0000) 2020-02-05T20:42:32.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:32.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:32.280761+0000) 2020-02-05T20:42:33.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:33.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:33.406993+0000) 2020-02-05T20:42:33.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:33.437+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:33.438165+0000) 2020-02-05T20:42:34.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:34.764+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:34.765482+0000) 2020-02-05T20:42:35.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:35.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:35.782270+0000) 2020-02-05T20:42:35.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:35.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:35.819205+0000) 2020-02-05T20:42:36.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:36.047+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.048722+0000) 2020-02-05T20:42:36.051 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:36.048+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.049610+0000) 2020-02-05T20:42:36.051 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:36.049+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.050468+0000) 2020-02-05T20:42:36.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:36.049+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.050790+0000) 2020-02-05T20:42:36.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:36.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.051342+0000) 2020-02-05T20:42:36.053 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:36.051+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.051792+0000) 2020-02-05T20:42:36.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:36.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.052205+0000) 2020-02-05T20:42:36.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:36.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.052490+0000) 2020-02-05T20:42:36.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:36.052+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.053349+0000) 2020-02-05T20:42:36.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:36.052+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.053882+0000) 2020-02-05T20:42:36.055 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:36.053+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.053928+0000) 2020-02-05T20:42:36.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:36.053+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.054232+0000) 2020-02-05T20:42:36.056 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:36.054+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.055224+0000) 2020-02-05T20:42:36.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:36.055+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.056794+0000) 2020-02-05T20:42:36.058 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:36.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:36.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:36.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.679498+0000) 2020-02-05T20:42:36.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:36.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.922815+0000) 2020-02-05T20:42:36.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:36.924+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:36.924424+0000) 2020-02-05T20:42:39.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:39.316+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:39.317330+0000) 2020-02-05T20:42:39.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:39.925+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:39.926691+0000) 2020-02-05T20:42:40.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:40.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:40.175784+0000) 2020-02-05T20:42:40.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:40.186+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:40.187329+0000) 2020-02-05T20:42:41.064 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:41.061+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.062878+0000) 2020-02-05T20:42:41.064 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:41.062+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.063492+0000) 2020-02-05T20:42:41.066 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:41.064+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.065106+0000) 2020-02-05T20:42:41.067 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:41.064+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.065305+0000) 2020-02-05T20:42:41.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:41.065+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.065914+0000) 2020-02-05T20:42:41.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:41.065+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.067011+0000) 2020-02-05T20:42:41.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:41.066+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.066728+0000) 2020-02-05T20:42:41.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:41.066+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.067136+0000) 2020-02-05T20:42:41.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:41.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.067663+0000) 2020-02-05T20:42:41.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:41.067+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.068097+0000) 2020-02-05T20:42:41.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:41.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.068563+0000) 2020-02-05T20:42:41.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:41.067+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.068848+0000) 2020-02-05T20:42:41.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:41.068+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.069146+0000) 2020-02-05T20:42:41.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:41.069+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:41.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:41.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:41.076355+0000) 2020-02-05T20:42:42.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:42.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:42.165335+0000) 2020-02-05T20:42:42.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:42.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:42.280986+0000) 2020-02-05T20:42:43.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:43.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:43.407247+0000) 2020-02-05T20:42:43.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:43.437+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:43.438347+0000) 2020-02-05T20:42:44.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:44.773+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:44.773815+0000) 2020-02-05T20:42:45.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:45.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:45.782507+0000) 2020-02-05T20:42:45.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:45.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:45.819446+0000) 2020-02-05T20:42:46.078 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:46.076+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.077185+0000) 2020-02-05T20:42:46.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:46.077+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.077966+0000) 2020-02-05T20:42:46.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:46.077+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.078376+0000) 2020-02-05T20:42:46.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:46.078+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.079536+0000) 2020-02-05T20:42:46.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:46.078+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.079377+0000) 2020-02-05T20:42:46.081 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:46.079+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.080231+0000) 2020-02-05T20:42:46.082 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:46.079+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.080541+0000) 2020-02-05T20:42:46.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:46.080+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.081126+0000) 2020-02-05T20:42:46.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:46.081+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.081507+0000) 2020-02-05T20:42:46.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:46.080+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.081383+0000) 2020-02-05T20:42:46.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:46.081+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.082358+0000) 2020-02-05T20:42:46.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:46.081+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.082357+0000) 2020-02-05T20:42:46.085 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:46.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:46.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:46.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.085101+0000) 2020-02-05T20:42:46.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:46.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.085437+0000) 2020-02-05T20:42:46.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:46.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.679726+0000) 2020-02-05T20:42:46.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:46.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.923026+0000) 2020-02-05T20:42:46.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:46.924+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:46.924703+0000) 2020-02-05T20:42:49.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:49.316+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:49.317566+0000) 2020-02-05T20:42:49.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:49.926+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:49.926922+0000) 2020-02-05T20:42:50.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:50.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:50.176042+0000) 2020-02-05T20:42:50.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:50.186+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:50.187548+0000) 2020-02-05T20:42:51.090 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:51.087+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.088331+0000) 2020-02-05T20:42:51.091 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:51.088+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.089329+0000) 2020-02-05T20:42:51.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:51.090+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.090902+0000) 2020-02-05T20:42:51.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:51.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.091573+0000) 2020-02-05T20:42:51.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:51.090+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.091735+0000) 2020-02-05T20:42:51.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:51.091+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.092335+0000) 2020-02-05T20:42:51.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:51.091+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.092882+0000) 2020-02-05T20:42:51.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:51.092+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.093404+0000) 2020-02-05T20:42:51.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:51.093+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.094211+0000) 2020-02-05T20:42:51.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:51.094+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.094463+0000) 2020-02-05T20:42:51.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:51.094+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.094821+0000) 2020-02-05T20:42:51.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:51.094+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.095168+0000) 2020-02-05T20:42:51.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:51.094+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.095229+0000) 2020-02-05T20:42:51.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:51.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:51.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:51.098+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:51.100034+0000) 2020-02-05T20:42:52.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:52.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:52.165592+0000) 2020-02-05T20:42:52.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:52.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:52.281196+0000) 2020-02-05T20:42:53.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:53.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:53.407524+0000) 2020-02-05T20:42:53.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:53.437+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:53.438594+0000) 2020-02-05T20:42:54.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:54.780+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:54.780942+0000) 2020-02-05T20:42:55.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:55.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:55.782744+0000) 2020-02-05T20:42:55.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:55.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:55.819708+0000) 2020-02-05T20:42:56.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:56.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.103395+0000) 2020-02-05T20:42:56.105 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:42:56.101+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.103057+0000) 2020-02-05T20:42:56.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:56.103+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.104330+0000) 2020-02-05T20:42:56.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:42:56.103+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.105013+0000) 2020-02-05T20:42:56.107 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:42:56.104+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.105206+0000) 2020-02-05T20:42:56.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:42:56.103+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.105292+0000) 2020-02-05T20:42:56.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:42:56.104+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.105821+0000) 2020-02-05T20:42:56.108 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:42:56.105+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.106179+0000) 2020-02-05T20:42:56.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:56.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.106585+0000) 2020-02-05T20:42:56.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:56.105+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.106845+0000) 2020-02-05T20:42:56.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:42:56.106+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.107998+0000) 2020-02-05T20:42:56.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:56.107+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.108888+0000) 2020-02-05T20:42:56.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:56.107+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.108997+0000) 2020-02-05T20:42:56.112 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:42:56.109+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:42:56.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:56.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.112961+0000) 2020-02-05T20:42:56.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:42:56.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.679947+0000) 2020-02-05T20:42:56.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:42:56.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.923253+0000) 2020-02-05T20:42:56.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:42:56.923+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:56.924901+0000) 2020-02-05T20:42:57.501 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:42:57.507 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:42:59.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:42:59.317+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:59.317797+0000) 2020-02-05T20:42:59.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:42:59.926+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:42:59.927187+0000) 2020-02-05T20:43:00.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:00.174+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:00.176285+0000) 2020-02-05T20:43:00.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:00.187+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:00.187776+0000) 2020-02-05T20:43:01.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:01.115+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.116737+0000) 2020-02-05T20:43:01.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:01.115+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.117145+0000) 2020-02-05T20:43:01.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:01.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.117551+0000) 2020-02-05T20:43:01.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:01.117+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.118839+0000) 2020-02-05T20:43:01.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:01.118+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.119556+0000) 2020-02-05T20:43:01.121 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:01.117+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.119268+0000) 2020-02-05T20:43:01.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:01.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.120316+0000) 2020-02-05T20:43:01.122 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:01.119+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.120696+0000) 2020-02-05T20:43:01.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:01.119+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.121188+0000) 2020-02-05T20:43:01.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:01.119+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.121393+0000) 2020-02-05T20:43:01.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:01.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.121654+0000) 2020-02-05T20:43:01.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:01.120+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.121722+0000) 2020-02-05T20:43:01.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:01.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.121735+0000) 2020-02-05T20:43:01.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:01.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:01.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:01.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:01.128023+0000) 2020-02-05T20:43:02.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:02.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:02.165829+0000) 2020-02-05T20:43:02.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:02.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:02.281479+0000) 2020-02-05T20:43:03.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:03.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:03.407774+0000) 2020-02-05T20:43:03.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:03.438+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:03.438822+0000) 2020-02-05T20:43:04.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:04.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:04.787904+0000) 2020-02-05T20:43:05.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:05.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:05.783021+0000) 2020-02-05T20:43:05.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:05.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:05.819956+0000) 2020-02-05T20:43:06.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:06.129+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.130754+0000) 2020-02-05T20:43:06.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:06.131+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.132436+0000) 2020-02-05T20:43:06.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:06.132+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.133280+0000) 2020-02-05T20:43:06.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:06.131+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.132893+0000) 2020-02-05T20:43:06.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:06.132+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.133576+0000) 2020-02-05T20:43:06.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:06.133+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.134509+0000) 2020-02-05T20:43:06.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:06.133+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.134798+0000) 2020-02-05T20:43:06.136 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:06.133+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.134779+0000) 2020-02-05T20:43:06.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:06.133+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.135311+0000) 2020-02-05T20:43:06.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:06.135+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.136382+0000) 2020-02-05T20:43:06.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:06.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.136432+0000) 2020-02-05T20:43:06.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:06.135+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.136638+0000) 2020-02-05T20:43:06.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:06.135+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.136819+0000) 2020-02-05T20:43:06.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:06.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:06.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:06.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.140710+0000) 2020-02-05T20:43:06.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:06.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.680149+0000) 2020-02-05T20:43:06.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:06.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.923486+0000) 2020-02-05T20:43:06.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:06.923+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:06.925114+0000) 2020-02-05T20:43:09.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:09.317+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:09.318051+0000) 2020-02-05T20:43:09.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:09.927+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:09.927418+0000) 2020-02-05T20:43:10.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:10.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:10.176529+0000) 2020-02-05T20:43:10.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:10.187+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:10.188021+0000) 2020-02-05T20:43:11.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:11.143+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.144139+0000) 2020-02-05T20:43:11.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:11.144+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.145523+0000) 2020-02-05T20:43:11.147 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:11.145+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.146119+0000) 2020-02-05T20:43:11.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:11.145+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.146898+0000) 2020-02-05T20:43:11.148 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:11.146+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.147502+0000) 2020-02-05T20:43:11.149 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:11.146+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.148343+0000) 2020-02-05T20:43:11.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:11.147+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.148942+0000) 2020-02-05T20:43:11.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:11.147+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.148840+0000) 2020-02-05T20:43:11.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:11.148+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.149413+0000) 2020-02-05T20:43:11.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:11.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.149510+0000) 2020-02-05T20:43:11.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:11.148+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.149591+0000) 2020-02-05T20:43:11.152 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:11.149+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.150431+0000) 2020-02-05T20:43:11.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:11.149+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.150835+0000) 2020-02-05T20:43:11.154 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:11.152+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:11.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:11.153+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:11.154995+0000) 2020-02-05T20:43:12.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:12.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:12.166083+0000) 2020-02-05T20:43:12.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:12.280+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:12.281712+0000) 2020-02-05T20:43:13.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:13.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:13.408027+0000) 2020-02-05T20:43:13.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:13.438+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:13.439058+0000) 2020-02-05T20:43:14.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:14.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:14.795392+0000) 2020-02-05T20:43:15.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:15.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:15.783240+0000) 2020-02-05T20:43:15.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:15.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:15.820216+0000) 2020-02-05T20:43:16.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:16.157+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.158260+0000) 2020-02-05T20:43:16.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:16.158+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.159414+0000) 2020-02-05T20:43:16.162 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:16.159+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.160512+0000) 2020-02-05T20:43:16.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:16.161+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.162168+0000) 2020-02-05T20:43:16.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:16.161+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.162847+0000) 2020-02-05T20:43:16.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:16.161+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.163295+0000) 2020-02-05T20:43:16.165 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:16.162+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.163643+0000) 2020-02-05T20:43:16.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:16.161+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.163089+0000) 2020-02-05T20:43:16.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:16.162+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.164165+0000) 2020-02-05T20:43:16.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:16.163+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.165008+0000) 2020-02-05T20:43:16.166 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:16.163+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.164986+0000) 2020-02-05T20:43:16.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:16.164+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.165662+0000) 2020-02-05T20:43:16.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:16.164+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.166011+0000) 2020-02-05T20:43:16.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:16.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:16.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:16.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.170913+0000) 2020-02-05T20:43:16.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:16.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.680408+0000) 2020-02-05T20:43:16.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:16.922+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.923698+0000) 2020-02-05T20:43:16.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:16.924+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:16.925325+0000) 2020-02-05T20:43:19.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:19.318+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:19.318324+0000) 2020-02-05T20:43:19.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:19.927+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:19.927647+0000) 2020-02-05T20:43:20.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:20.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:20.176756+0000) 2020-02-05T20:43:20.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:20.188+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:20.188272+0000) 2020-02-05T20:43:21.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:21.173+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.173917+0000) 2020-02-05T20:43:21.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:21.175+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.175323+0000) 2020-02-05T20:43:21.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:21.175+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.175793+0000) 2020-02-05T20:43:21.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:21.175+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.176512+0000) 2020-02-05T20:43:21.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:21.175+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.176754+0000) 2020-02-05T20:43:21.178 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:21.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.177143+0000) 2020-02-05T20:43:21.179 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:21.176+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.177805+0000) 2020-02-05T20:43:21.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:21.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.179165+0000) 2020-02-05T20:43:21.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:21.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.179585+0000) 2020-02-05T20:43:21.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:21.178+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.179982+0000) 2020-02-05T20:43:21.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:21.178+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.179769+0000) 2020-02-05T20:43:21.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:21.180+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.180434+0000) 2020-02-05T20:43:21.182 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:21.179+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.180931+0000) 2020-02-05T20:43:21.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:21.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:21.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:21.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:21.188581+0000) 2020-02-05T20:43:22.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:22.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:22.166340+0000) 2020-02-05T20:43:22.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:22.281+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:22.281947+0000) 2020-02-05T20:43:23.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:23.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:23.408284+0000) 2020-02-05T20:43:23.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:23.439+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:23.439358+0000) 2020-02-05T20:43:24.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:24.803+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:24.803930+0000) 2020-02-05T20:43:25.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:25.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:25.783468+0000) 2020-02-05T20:43:25.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:25.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:25.820465+0000) 2020-02-05T20:43:26.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:26.186+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.187529+0000) 2020-02-05T20:43:26.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:26.187+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.188658+0000) 2020-02-05T20:43:26.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:26.188+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.189510+0000) 2020-02-05T20:43:26.191 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:26.187+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.189319+0000) 2020-02-05T20:43:26.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:26.188+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.190209+0000) 2020-02-05T20:43:26.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:26.189+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.190744+0000) 2020-02-05T20:43:26.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:26.190+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.191384+0000) 2020-02-05T20:43:26.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:26.190+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.191598+0000) 2020-02-05T20:43:26.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:26.191+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.192555+0000) 2020-02-05T20:43:26.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:26.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.192833+0000) 2020-02-05T20:43:26.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:26.191+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.193300+0000) 2020-02-05T20:43:26.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:26.192+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.194217+0000) 2020-02-05T20:43:26.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:26.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.194369+0000) 2020-02-05T20:43:26.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:26.195+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:26.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:26.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.198708+0000) 2020-02-05T20:43:26.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:26.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.680655+0000) 2020-02-05T20:43:26.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:26.923+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.923928+0000) 2020-02-05T20:43:26.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:26.924+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:26.925502+0000) 2020-02-05T20:43:27.698 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:43:27.704 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:43:29.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:29.317+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:29.318565+0000) 2020-02-05T20:43:29.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:29.926+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:29.927901+0000) 2020-02-05T20:43:30.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:30.176+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:30.177013+0000) 2020-02-05T20:43:30.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:30.187+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:30.188522+0000) 2020-02-05T20:43:31.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:31.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.203510+0000) 2020-02-05T20:43:31.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:31.203+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.205101+0000) 2020-02-05T20:43:31.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:31.204+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.205749+0000) 2020-02-05T20:43:31.207 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:31.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.205905+0000) 2020-02-05T20:43:31.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:31.205+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.206402+0000) 2020-02-05T20:43:31.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:31.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.206780+0000) 2020-02-05T20:43:31.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:31.206+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.207326+0000) 2020-02-05T20:43:31.209 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:31.206+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.207530+0000) 2020-02-05T20:43:31.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:31.206+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.207778+0000) 2020-02-05T20:43:31.210 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:31.206+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.208269+0000) 2020-02-05T20:43:31.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:31.206+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.208292+0000) 2020-02-05T20:43:31.210 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:31.208+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.209271+0000) 2020-02-05T20:43:31.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:31.207+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.209092+0000) 2020-02-05T20:43:31.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:31.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:31.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:31.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:31.216961+0000) 2020-02-05T20:43:32.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:32.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:32.166573+0000) 2020-02-05T20:43:32.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:32.281+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:32.282209+0000) 2020-02-05T20:43:33.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:33.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:33.408550+0000) 2020-02-05T20:43:33.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:33.438+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:33.439626+0000) 2020-02-05T20:43:34.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:34.811+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:34.811952+0000) 2020-02-05T20:43:35.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:35.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:35.783701+0000) 2020-02-05T20:43:35.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:35.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:35.820694+0000) 2020-02-05T20:43:36.218 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:36.215+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.216847+0000) 2020-02-05T20:43:36.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:36.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.217463+0000) 2020-02-05T20:43:36.220 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:36.217+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.218966+0000) 2020-02-05T20:43:36.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:36.218+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.219765+0000) 2020-02-05T20:43:36.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:36.219+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.220415+0000) 2020-02-05T20:43:36.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:36.219+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.221199+0000) 2020-02-05T20:43:36.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:36.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.221761+0000) 2020-02-05T20:43:36.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:36.220+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.222129+0000) 2020-02-05T20:43:36.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:36.221+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.222590+0000) 2020-02-05T20:43:36.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:36.221+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.222855+0000) 2020-02-05T20:43:36.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:36.222+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.223296+0000) 2020-02-05T20:43:36.225 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:36.222+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.223641+0000) 2020-02-05T20:43:36.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:36.222+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.223971+0000) 2020-02-05T20:43:36.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:36.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:36.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:36.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.228952+0000) 2020-02-05T20:43:36.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:36.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.680866+0000) 2020-02-05T20:43:36.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:36.923+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.924160+0000) 2020-02-05T20:43:36.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:36.924+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:36.925663+0000) 2020-02-05T20:43:39.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:39.317+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:39.318795+0000) 2020-02-05T20:43:39.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:39.927+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:39.928132+0000) 2020-02-05T20:43:40.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:40.176+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:40.177267+0000) 2020-02-05T20:43:40.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:40.187+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:40.188749+0000) 2020-02-05T20:43:41.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:41.230+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.232225+0000) 2020-02-05T20:43:41.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:41.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.232653+0000) 2020-02-05T20:43:41.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:41.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.233033+0000) 2020-02-05T20:43:41.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:41.232+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.233446+0000) 2020-02-05T20:43:41.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:41.232+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.234084+0000) 2020-02-05T20:43:41.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:41.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.233796+0000) 2020-02-05T20:43:41.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:41.233+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.234285+0000) 2020-02-05T20:43:41.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:41.233+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.234572+0000) 2020-02-05T20:43:41.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:41.233+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.235123+0000) 2020-02-05T20:43:41.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:41.234+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.235160+0000) 2020-02-05T20:43:41.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:41.234+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.235691+0000) 2020-02-05T20:43:41.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:41.235+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.236420+0000) 2020-02-05T20:43:41.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:41.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.237736+0000) 2020-02-05T20:43:41.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:41.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:41.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:41.241+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:41.242204+0000) 2020-02-05T20:43:42.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:42.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:42.166754+0000) 2020-02-05T20:43:42.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:42.281+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:42.282497+0000) 2020-02-05T20:43:43.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:43.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:43.408783+0000) 2020-02-05T20:43:43.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:43.439+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:43.439857+0000) 2020-02-05T20:43:44.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:44.818+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:44.819367+0000) 2020-02-05T20:43:45.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:45.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:45.783933+0000) 2020-02-05T20:43:45.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:45.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:45.820928+0000) 2020-02-05T20:43:46.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:46.245+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.245915+0000) 2020-02-05T20:43:46.248 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:46.246+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.247303+0000) 2020-02-05T20:43:46.249 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:46.247+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.248207+0000) 2020-02-05T20:43:46.250 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:46.247+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.248891+0000) 2020-02-05T20:43:46.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:46.248+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.248981+0000) 2020-02-05T20:43:46.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:46.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.249240+0000) 2020-02-05T20:43:46.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:46.248+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.249405+0000) 2020-02-05T20:43:46.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:46.249+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.250073+0000) 2020-02-05T20:43:46.252 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:46.249+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.250557+0000) 2020-02-05T20:43:46.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:46.250+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.250749+0000) 2020-02-05T20:43:46.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:46.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.251235+0000) 2020-02-05T20:43:46.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:46.251+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.251623+0000) 2020-02-05T20:43:46.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:46.251+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.252347+0000) 2020-02-05T20:43:46.254 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:46.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 64990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:46.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:46.257+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.258611+0000) 2020-02-05T20:43:46.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:46.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.681071+0000) 2020-02-05T20:43:46.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:46.923+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.924396+0000) 2020-02-05T20:43:46.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:46.925+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:46.925929+0000) 2020-02-05T20:43:49.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:49.318+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:49.319032+0000) 2020-02-05T20:43:49.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:49.927+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:49.928345+0000) 2020-02-05T20:43:50.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:50.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:50.177535+0000) 2020-02-05T20:43:50.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:50.188+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:50.188972+0000) 2020-02-05T20:43:51.260 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:51.257+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.258344+0000) 2020-02-05T20:43:51.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:51.257+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.258795+0000) 2020-02-05T20:43:51.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:51.259+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.260209+0000) 2020-02-05T20:43:51.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:51.260+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.260687+0000) 2020-02-05T20:43:51.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:51.261+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.261580+0000) 2020-02-05T20:43:51.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:51.260+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.261939+0000) 2020-02-05T20:43:51.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:51.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.262522+0000) 2020-02-05T20:43:51.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:51.261+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.262879+0000) 2020-02-05T20:43:51.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:51.262+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.263107+0000) 2020-02-05T20:43:51.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:51.262+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.263775+0000) 2020-02-05T20:43:51.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:51.263+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.264207+0000) 2020-02-05T20:43:51.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:51.263+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.264994+0000) 2020-02-05T20:43:51.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:51.264+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.265226+0000) 2020-02-05T20:43:51.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:51.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:51.266172+0000) 2020-02-05T20:43:51.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:51.267+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:52.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:52.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:52.166990+0000) 2020-02-05T20:43:52.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:52.282+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:52.282719+0000) 2020-02-05T20:43:53.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:53.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:53.409023+0000) 2020-02-05T20:43:53.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:53.439+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:53.440117+0000) 2020-02-05T20:43:54.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:54.826+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:54.827804+0000) 2020-02-05T20:43:55.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:55.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:55.784165+0000) 2020-02-05T20:43:55.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:55.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:55.821163+0000) 2020-02-05T20:43:56.274 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:43:56.272+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.273370+0000) 2020-02-05T20:43:56.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:56.273+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.274450+0000) 2020-02-05T20:43:56.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:56.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.274697+0000) 2020-02-05T20:43:56.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:43:56.274+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.275371+0000) 2020-02-05T20:43:56.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:56.274+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.275842+0000) 2020-02-05T20:43:56.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:56.275+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.276277+0000) 2020-02-05T20:43:56.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:43:56.275+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.276545+0000) 2020-02-05T20:43:56.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:56.276+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.276789+0000) 2020-02-05T20:43:56.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:56.277+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.277567+0000) 2020-02-05T20:43:56.279 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:43:56.277+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.278160+0000) 2020-02-05T20:43:56.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:43:56.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.278731+0000) 2020-02-05T20:43:56.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:43:56.278+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.279205+0000) 2020-02-05T20:43:56.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:43:56.278+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.279193+0000) 2020-02-05T20:43:56.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:56.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.283387+0000) 2020-02-05T20:43:56.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:43:56.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:43:56.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:43:56.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.681333+0000) 2020-02-05T20:43:56.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:43:56.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.924657+0000) 2020-02-05T20:43:56.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:43:56.925+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:56.926119+0000) 2020-02-05T20:43:57.911 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:43:57.918 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:43:59.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:43:59.318+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:59.319310+0000) 2020-02-05T20:43:59.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:43:59.928+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:43:59.928570+0000) 2020-02-05T20:44:00.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:00.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:00.177790+0000) 2020-02-05T20:44:00.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:00.188+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:00.189194+0000) 2020-02-05T20:44:01.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:01.288+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.288804+0000) 2020-02-05T20:44:01.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:01.289+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.290370+0000) 2020-02-05T20:44:01.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:01.290+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.290905+0000) 2020-02-05T20:44:01.292 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:01.290+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.291549+0000) 2020-02-05T20:44:01.293 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:01.291+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.292223+0000) 2020-02-05T20:44:01.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:01.291+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.292653+0000) 2020-02-05T20:44:01.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:01.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.293259+0000) 2020-02-05T20:44:01.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:01.293+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.294184+0000) 2020-02-05T20:44:01.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:01.294+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.294882+0000) 2020-02-05T20:44:01.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:01.294+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.295131+0000) 2020-02-05T20:44:01.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:01.293+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.295014+0000) 2020-02-05T20:44:01.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:01.295+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.295459+0000) 2020-02-05T20:44:01.300 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:01.298+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:01.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:01.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.298807+0000) 2020-02-05T20:44:01.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:01.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:01.299058+0000) 2020-02-05T20:44:02.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:02.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:02.167228+0000) 2020-02-05T20:44:02.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:02.282+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:02.282978+0000) 2020-02-05T20:44:03.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:03.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:03.409272+0000) 2020-02-05T20:44:03.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:03.439+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:03.440302+0000) 2020-02-05T20:44:04.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:04.835+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:04.835982+0000) 2020-02-05T20:44:05.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:05.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:05.784402+0000) 2020-02-05T20:44:05.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:05.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:05.821393+0000) 2020-02-05T20:44:06.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:06.304+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.305036+0000) 2020-02-05T20:44:06.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:06.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.305616+0000) 2020-02-05T20:44:06.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:06.304+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.305904+0000) 2020-02-05T20:44:06.308 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:06.305+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.306300+0000) 2020-02-05T20:44:06.308 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:06.306+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.307283+0000) 2020-02-05T20:44:06.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:06.306+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.307339+0000) 2020-02-05T20:44:06.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:06.307+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.307715+0000) 2020-02-05T20:44:06.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:06.307+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.307610+0000) 2020-02-05T20:44:06.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:06.307+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.308173+0000) 2020-02-05T20:44:06.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:06.308+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.308772+0000) 2020-02-05T20:44:06.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:06.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.309332+0000) 2020-02-05T20:44:06.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:06.308+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.309525+0000) 2020-02-05T20:44:06.313 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:06.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65156 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:06.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:06.313+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.314644+0000) 2020-02-05T20:44:06.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:06.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.316566+0000) 2020-02-05T20:44:06.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:06.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.681581+0000) 2020-02-05T20:44:06.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:06.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.924827+0000) 2020-02-05T20:44:06.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:06.926+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:06.926333+0000) 2020-02-05T20:44:09.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:09.318+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:09.319528+0000) 2020-02-05T20:44:09.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:09.927+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:09.928823+0000) 2020-02-05T20:44:10.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:10.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:10.178039+0000) 2020-02-05T20:44:10.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:10.189+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:10.189428+0000) 2020-02-05T20:44:11.319 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:11.316+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.317913+0000) 2020-02-05T20:44:11.320 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:11.317+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.318504+0000) 2020-02-05T20:44:11.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:11.318+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.319240+0000) 2020-02-05T20:44:11.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:11.319+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.319549+0000) 2020-02-05T20:44:11.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:11.318+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.319754+0000) 2020-02-05T20:44:11.321 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:11.319+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.320190+0000) 2020-02-05T20:44:11.322 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:11.319+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.320081+0000) 2020-02-05T20:44:11.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:11.320+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.320725+0000) 2020-02-05T20:44:11.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:11.319+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.320437+0000) 2020-02-05T20:44:11.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:11.320+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.321046+0000) 2020-02-05T20:44:11.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:11.321+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.321663+0000) 2020-02-05T20:44:11.325 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:11.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:11.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:11.325+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.326090+0000) 2020-02-05T20:44:11.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:11.326+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.327446+0000) 2020-02-05T20:44:11.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:11.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:11.329117+0000) 2020-02-05T20:44:12.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:12.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:12.167510+0000) 2020-02-05T20:44:12.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:12.283+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:12.283212+0000) 2020-02-05T20:44:13.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:13.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:13.409537+0000) 2020-02-05T20:44:13.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:13.440+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:13.440542+0000) 2020-02-05T20:44:14.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:14.844+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:14.844493+0000) 2020-02-05T20:44:15.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:15.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:15.784664+0000) 2020-02-05T20:44:15.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:15.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:15.821621+0000) 2020-02-05T20:44:16.331 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:16.329+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.329631+0000) 2020-02-05T20:44:16.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:16.330+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.330669+0000) 2020-02-05T20:44:16.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:16.331+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.331858+0000) 2020-02-05T20:44:16.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:16.330+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.330633+0000) 2020-02-05T20:44:16.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:16.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.331371+0000) 2020-02-05T20:44:16.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:16.331+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.331596+0000) 2020-02-05T20:44:16.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:16.332+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.332362+0000) 2020-02-05T20:44:16.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:16.332+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.332559+0000) 2020-02-05T20:44:16.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:16.332+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.332958+0000) 2020-02-05T20:44:16.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:16.334+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.335141+0000) 2020-02-05T20:44:16.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:16.335+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.335493+0000) 2020-02-05T20:44:16.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:16.336+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.336493+0000) 2020-02-05T20:44:16.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:16.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:16.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:16.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.338309+0000) 2020-02-05T20:44:16.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:16.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.338561+0000) 2020-02-05T20:44:16.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:16.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.681824+0000) 2020-02-05T20:44:16.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:16.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.925071+0000) 2020-02-05T20:44:16.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:16.926+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:16.926550+0000) 2020-02-05T20:44:19.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:19.319+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:19.319779+0000) 2020-02-05T20:44:19.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:19.928+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:19.929056+0000) 2020-02-05T20:44:20.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:20.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:20.178267+0000) 2020-02-05T20:44:20.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:20.188+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:20.189653+0000) 2020-02-05T20:44:21.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:21.342+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.343477+0000) 2020-02-05T20:44:21.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:21.343+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.345008+0000) 2020-02-05T20:44:21.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:21.344+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.344875+0000) 2020-02-05T20:44:21.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:21.344+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.345603+0000) 2020-02-05T20:44:21.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:21.345+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.346713+0000) 2020-02-05T20:44:21.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:21.347+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.347937+0000) 2020-02-05T20:44:21.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:21.347+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.348232+0000) 2020-02-05T20:44:21.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:21.347+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.348750+0000) 2020-02-05T20:44:21.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:21.347+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.348896+0000) 2020-02-05T20:44:21.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:21.349+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.349757+0000) 2020-02-05T20:44:21.351 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:21.349+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.350091+0000) 2020-02-05T20:44:21.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:21.350+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.350565+0000) 2020-02-05T20:44:21.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:21.350+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.350678+0000) 2020-02-05T20:44:21.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:21.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65276 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:21.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:21.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:21.357176+0000) 2020-02-05T20:44:22.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:22.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:22.167777+0000) 2020-02-05T20:44:22.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:22.282+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:22.283406+0000) 2020-02-05T20:44:23.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:23.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:23.409777+0000) 2020-02-05T20:44:23.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:23.439+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:23.440772+0000) 2020-02-05T20:44:24.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:24.851+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:24.852758+0000) 2020-02-05T20:44:25.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:25.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:25.784930+0000) 2020-02-05T20:44:25.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:25.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:25.821850+0000) 2020-02-05T20:44:26.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:26.358+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.359756+0000) 2020-02-05T20:44:26.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:26.359+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.360431+0000) 2020-02-05T20:44:26.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:26.359+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.360673+0000) 2020-02-05T20:44:26.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:26.359+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.360972+0000) 2020-02-05T20:44:26.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:26.358+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.360083+0000) 2020-02-05T20:44:26.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:26.359+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.361135+0000) 2020-02-05T20:44:26.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:26.360+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.361722+0000) 2020-02-05T20:44:26.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:26.361+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.362354+0000) 2020-02-05T20:44:26.365 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:26.361+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.362112+0000) 2020-02-05T20:44:26.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:26.361+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.362920+0000) 2020-02-05T20:44:26.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:26.360+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.362245+0000) 2020-02-05T20:44:26.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:26.362+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.363677+0000) 2020-02-05T20:44:26.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:26.363+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.364480+0000) 2020-02-05T20:44:26.368 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:26.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:26.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:26.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.366571+0000) 2020-02-05T20:44:26.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:26.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.682066+0000) 2020-02-05T20:44:26.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:26.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.925297+0000) 2020-02-05T20:44:26.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:26.925+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:26.926755+0000) 2020-02-05T20:44:28.071 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:44:28.078 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:44:29.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:29.318+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:29.320022+0000) 2020-02-05T20:44:29.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:29.928+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:29.929334+0000) 2020-02-05T20:44:30.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:30.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:30.178525+0000) 2020-02-05T20:44:30.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:30.189+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:30.189880+0000) 2020-02-05T20:44:31.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:31.370+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.372023+0000) 2020-02-05T20:44:31.374 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:31.371+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.372641+0000) 2020-02-05T20:44:31.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:31.372+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.373203+0000) 2020-02-05T20:44:31.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:31.372+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.373997+0000) 2020-02-05T20:44:31.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:31.373+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.374426+0000) 2020-02-05T20:44:31.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:31.372+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.373753+0000) 2020-02-05T20:44:31.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:31.373+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.374827+0000) 2020-02-05T20:44:31.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:31.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.376379+0000) 2020-02-05T20:44:31.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:31.375+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.376616+0000) 2020-02-05T20:44:31.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:31.374+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.375824+0000) 2020-02-05T20:44:31.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:31.374+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.375911+0000) 2020-02-05T20:44:31.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:31.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.375730+0000) 2020-02-05T20:44:31.379 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:31.375+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.376910+0000) 2020-02-05T20:44:31.381 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:31.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65358 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:31.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:31.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:31.381881+0000) 2020-02-05T20:44:32.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:32.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:32.168009+0000) 2020-02-05T20:44:32.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:32.282+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:32.283707+0000) 2020-02-05T20:44:33.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:33.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:33.410030+0000) 2020-02-05T20:44:33.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:33.440+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:33.441006+0000) 2020-02-05T20:44:34.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:34.860+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:34.861181+0000) 2020-02-05T20:44:35.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:35.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:35.785163+0000) 2020-02-05T20:44:35.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:35.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:35.822120+0000) 2020-02-05T20:44:36.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:36.384+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.385574+0000) 2020-02-05T20:44:36.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:36.385+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.386558+0000) 2020-02-05T20:44:36.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:36.387+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.388272+0000) 2020-02-05T20:44:36.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:36.387+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.388785+0000) 2020-02-05T20:44:36.391 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:36.388+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.389374+0000) 2020-02-05T20:44:36.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:36.387+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.389122+0000) 2020-02-05T20:44:36.392 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:36.389+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.390132+0000) 2020-02-05T20:44:36.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:36.389+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.390630+0000) 2020-02-05T20:44:36.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:36.389+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.391130+0000) 2020-02-05T20:44:36.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:36.390+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.391793+0000) 2020-02-05T20:44:36.394 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:36.390+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.391342+0000) 2020-02-05T20:44:36.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:36.390+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.392369+0000) 2020-02-05T20:44:36.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:36.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:36.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:36.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.396011+0000) 2020-02-05T20:44:36.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:36.395+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.396554+0000) 2020-02-05T20:44:36.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:36.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.682333+0000) 2020-02-05T20:44:36.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:36.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.925525+0000) 2020-02-05T20:44:36.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:36.926+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:36.926990+0000) 2020-02-05T20:44:39.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:39.319+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:39.320295+0000) 2020-02-05T20:44:39.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:39.928+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:39.929571+0000) 2020-02-05T20:44:40.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:40.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:40.178774+0000) 2020-02-05T20:44:40.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:40.189+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:40.190135+0000) 2020-02-05T20:44:41.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:41.398+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.399987+0000) 2020-02-05T20:44:41.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:41.401+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.402688+0000) 2020-02-05T20:44:41.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:41.401+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.402715+0000) 2020-02-05T20:44:41.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:41.402+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.403509+0000) 2020-02-05T20:44:41.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:41.402+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.404157+0000) 2020-02-05T20:44:41.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:41.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.404496+0000) 2020-02-05T20:44:41.406 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:41.404+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.405304+0000) 2020-02-05T20:44:41.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:41.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.405689+0000) 2020-02-05T20:44:41.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:41.405+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.406338+0000) 2020-02-05T20:44:41.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:41.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.406620+0000) 2020-02-05T20:44:41.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:41.406+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.407161+0000) 2020-02-05T20:44:41.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:41.405+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.407089+0000) 2020-02-05T20:44:41.410 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:41.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:41.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:41.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.410383+0000) 2020-02-05T20:44:41.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:41.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:41.410663+0000) 2020-02-05T20:44:42.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:42.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:42.168238+0000) 2020-02-05T20:44:42.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:42.283+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:42.283950+0000) 2020-02-05T20:44:43.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:43.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:43.410279+0000) 2020-02-05T20:44:43.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:43.440+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:43.441272+0000) 2020-02-05T20:44:44.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:44.869+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:44.869832+0000) 2020-02-05T20:44:45.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:45.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:45.785424+0000) 2020-02-05T20:44:45.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:45.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:45.822334+0000) 2020-02-05T20:44:46.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:46.414+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.415806+0000) 2020-02-05T20:44:46.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:46.414+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.415475+0000) 2020-02-05T20:44:46.418 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:46.415+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.416460+0000) 2020-02-05T20:44:46.419 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:46.416+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.417851+0000) 2020-02-05T20:44:46.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:46.416+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.417983+0000) 2020-02-05T20:44:46.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:46.416+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.418366+0000) 2020-02-05T20:44:46.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:46.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.418664+0000) 2020-02-05T20:44:46.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:46.418+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.419632+0000) 2020-02-05T20:44:46.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:46.418+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.420085+0000) 2020-02-05T20:44:46.422 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:46.418+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.420044+0000) 2020-02-05T20:44:46.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:46.419+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.420668+0000) 2020-02-05T20:44:46.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:46.419+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.421119+0000) 2020-02-05T20:44:46.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:46.419+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.421299+0000) 2020-02-05T20:44:46.425 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:46.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:46.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:46.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.425966+0000) 2020-02-05T20:44:46.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:46.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.682548+0000) 2020-02-05T20:44:46.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:46.924+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.925688+0000) 2020-02-05T20:44:46.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:46.926+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:46.927193+0000) 2020-02-05T20:44:49.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:49.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:49.320537+0000) 2020-02-05T20:44:49.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:49.929+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:49.929835+0000) 2020-02-05T20:44:50.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:50.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:50.179056+0000) 2020-02-05T20:44:50.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:50.189+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:50.190400+0000) 2020-02-05T20:44:51.431 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:51.428+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.429402+0000) 2020-02-05T20:44:51.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:51.429+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.430354+0000) 2020-02-05T20:44:51.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:51.429+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.430874+0000) 2020-02-05T20:44:51.433 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:51.431+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.432281+0000) 2020-02-05T20:44:51.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:51.430+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.432290+0000) 2020-02-05T20:44:51.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:51.432+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.433507+0000) 2020-02-05T20:44:51.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:51.432+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.433907+0000) 2020-02-05T20:44:51.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:51.433+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.434372+0000) 2020-02-05T20:44:51.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:51.433+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.434496+0000) 2020-02-05T20:44:51.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:51.433+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.434803+0000) 2020-02-05T20:44:51.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:51.433+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.434958+0000) 2020-02-05T20:44:51.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:51.433+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.435173+0000) 2020-02-05T20:44:51.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:51.434+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.435511+0000) 2020-02-05T20:44:51.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:51.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:51.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:51.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:51.439899+0000) 2020-02-05T20:44:52.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:52.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:52.168501+0000) 2020-02-05T20:44:52.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:52.282+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:52.284148+0000) 2020-02-05T20:44:53.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:53.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:53.410533+0000) 2020-02-05T20:44:53.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:53.441+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:53.441511+0000) 2020-02-05T20:44:54.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:54.881+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:54.881522+0000) 2020-02-05T20:44:55.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:55.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:55.785655+0000) 2020-02-05T20:44:55.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:55.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:55.822523+0000) 2020-02-05T20:44:56.445 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:56.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.443439+0000) 2020-02-05T20:44:56.446 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:44:56.443+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.444649+0000) 2020-02-05T20:44:56.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:56.444+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.445541+0000) 2020-02-05T20:44:56.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:44:56.444+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.445987+0000) 2020-02-05T20:44:56.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:44:56.445+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.446229+0000) 2020-02-05T20:44:56.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:56.445+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.446672+0000) 2020-02-05T20:44:56.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:56.446+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.447993+0000) 2020-02-05T20:44:56.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:44:56.445+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.447319+0000) 2020-02-05T20:44:56.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:44:56.446+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.447859+0000) 2020-02-05T20:44:56.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:56.448+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.449417+0000) 2020-02-05T20:44:56.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:44:56.448+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.449848+0000) 2020-02-05T20:44:56.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:56.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.450260+0000) 2020-02-05T20:44:56.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:44:56.449+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.450484+0000) 2020-02-05T20:44:56.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:44:56.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:44:56.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:56.453+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.454878+0000) 2020-02-05T20:44:56.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:44:56.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.682782+0000) 2020-02-05T20:44:56.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:44:56.925+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.925901+0000) 2020-02-05T20:44:56.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:44:56.926+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:56.927424+0000) 2020-02-05T20:44:58.248 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:44:58.254 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:44:59.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:44:59.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:59.320797+0000) 2020-02-05T20:44:59.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:44:59.929+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:44:59.930068+0000) 2020-02-05T20:45:00.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:00.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:00.179306+0000) 2020-02-05T20:45:00.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:00.190+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:00.190571+0000) 2020-02-05T20:45:01.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:01.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.458149+0000) 2020-02-05T20:45:01.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:01.458+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.459331+0000) 2020-02-05T20:45:01.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:01.458+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.459716+0000) 2020-02-05T20:45:01.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:01.459+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.460236+0000) 2020-02-05T20:45:01.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:01.459+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.460639+0000) 2020-02-05T20:45:01.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:01.459+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.461113+0000) 2020-02-05T20:45:01.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:01.460+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.461866+0000) 2020-02-05T20:45:01.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:01.461+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.462488+0000) 2020-02-05T20:45:01.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:01.461+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.462955+0000) 2020-02-05T20:45:01.465 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:01.462+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.464097+0000) 2020-02-05T20:45:01.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:01.462+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.464015+0000) 2020-02-05T20:45:01.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:01.463+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.464650+0000) 2020-02-05T20:45:01.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:01.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.467146+0000) 2020-02-05T20:45:01.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:01.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:01.467382+0000) 2020-02-05T20:45:01.469 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:01.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65605 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:02.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:02.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:02.168672+0000) 2020-02-05T20:45:02.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:02.283+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:02.284372+0000) 2020-02-05T20:45:03.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:03.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:03.410794+0000) 2020-02-05T20:45:03.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:03.441+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:03.441745+0000) 2020-02-05T20:45:04.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:04.892+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:04.893420+0000) 2020-02-05T20:45:05.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:05.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:05.785898+0000) 2020-02-05T20:45:05.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:05.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:05.822750+0000) 2020-02-05T20:45:06.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:06.471+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.473008+0000) 2020-02-05T20:45:06.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:06.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.473687+0000) 2020-02-05T20:45:06.475 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:06.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.474414+0000) 2020-02-05T20:45:06.478 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:06.475+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.476970+0000) 2020-02-05T20:45:06.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:06.476+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.477517+0000) 2020-02-05T20:45:06.479 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:06.476+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.477591+0000) 2020-02-05T20:45:06.480 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:06.476+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.477965+0000) 2020-02-05T20:45:06.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:06.477+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.478467+0000) 2020-02-05T20:45:06.480 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:06.477+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.478756+0000) 2020-02-05T20:45:06.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:06.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.479204+0000) 2020-02-05T20:45:06.481 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:06.478+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.479500+0000) 2020-02-05T20:45:06.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:06.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.479866+0000) 2020-02-05T20:45:06.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:06.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.481983+0000) 2020-02-05T20:45:06.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:06.482+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.483103+0000) 2020-02-05T20:45:06.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:06.481+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:06.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:06.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.683005+0000) 2020-02-05T20:45:06.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:06.925+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.926128+0000) 2020-02-05T20:45:06.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:06.927+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:06.927729+0000) 2020-02-05T20:45:09.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:09.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:09.321026+0000) 2020-02-05T20:45:09.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:09.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:09.930382+0000) 2020-02-05T20:45:10.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:10.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:10.179568+0000) 2020-02-05T20:45:10.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:10.190+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:10.190797+0000) 2020-02-05T20:45:11.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:11.488+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.489001+0000) 2020-02-05T20:45:11.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:11.489+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.489328+0000) 2020-02-05T20:45:11.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:11.489+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.489713+0000) 2020-02-05T20:45:11.492 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:11.490+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.490986+0000) 2020-02-05T20:45:11.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:11.490+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.491505+0000) 2020-02-05T20:45:11.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:11.491+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.491957+0000) 2020-02-05T20:45:11.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:11.492+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.492387+0000) 2020-02-05T20:45:11.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:11.491+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.492840+0000) 2020-02-05T20:45:11.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:11.491+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.493283+0000) 2020-02-05T20:45:11.495 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:11.493+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.493871+0000) 2020-02-05T20:45:11.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:11.492+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.493574+0000) 2020-02-05T20:45:11.496 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:11.493+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.494683+0000) 2020-02-05T20:45:11.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:11.493+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.495173+0000) 2020-02-05T20:45:11.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:11.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:11.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:11.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:11.499580+0000) 2020-02-05T20:45:12.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:12.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:12.168915+0000) 2020-02-05T20:45:12.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:12.283+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:12.284624+0000) 2020-02-05T20:45:13.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:13.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:13.411031+0000) 2020-02-05T20:45:13.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:13.441+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:13.441984+0000) 2020-02-05T20:45:14.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:14.904+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:14.905414+0000) 2020-02-05T20:45:15.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:15.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:15.786145+0000) 2020-02-05T20:45:15.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:15.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:15.822984+0000) 2020-02-05T20:45:16.505 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:16.502+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.503274+0000) 2020-02-05T20:45:16.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:16.502+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.503590+0000) 2020-02-05T20:45:16.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:16.502+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.504008+0000) 2020-02-05T20:45:16.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:16.503+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.504696+0000) 2020-02-05T20:45:16.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:16.505+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.506426+0000) 2020-02-05T20:45:16.508 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:16.506+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.506475+0000) 2020-02-05T20:45:16.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:16.506+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.507027+0000) 2020-02-05T20:45:16.508 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:16.507+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.507609+0000) 2020-02-05T20:45:16.509 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:16.506+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.507940+0000) 2020-02-05T20:45:16.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:16.507+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.507615+0000) 2020-02-05T20:45:16.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:16.507+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.508239+0000) 2020-02-05T20:45:16.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:16.507+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.508528+0000) 2020-02-05T20:45:16.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:16.508+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.509185+0000) 2020-02-05T20:45:16.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:16.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65730 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:16.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:16.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.512620+0000) 2020-02-05T20:45:16.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:16.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.683210+0000) 2020-02-05T20:45:16.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:16.925+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.926362+0000) 2020-02-05T20:45:16.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:16.927+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:16.927972+0000) 2020-02-05T20:45:19.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:19.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:19.321267+0000) 2020-02-05T20:45:19.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:19.929+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:19.930637+0000) 2020-02-05T20:45:20.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:20.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:20.179819+0000) 2020-02-05T20:45:20.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:20.190+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:20.191021+0000) 2020-02-05T20:45:21.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:21.516+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.517165+0000) 2020-02-05T20:45:21.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:21.516+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.517886+0000) 2020-02-05T20:45:21.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:21.517+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.518395+0000) 2020-02-05T20:45:21.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:21.518+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.519145+0000) 2020-02-05T20:45:21.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:21.518+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.519211+0000) 2020-02-05T20:45:21.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:21.519+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.519900+0000) 2020-02-05T20:45:21.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:21.519+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.520073+0000) 2020-02-05T20:45:21.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:21.520+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.521550+0000) 2020-02-05T20:45:21.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:21.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.521739+0000) 2020-02-05T20:45:21.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:21.522+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.522575+0000) 2020-02-05T20:45:21.524 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:21.521+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.523025+0000) 2020-02-05T20:45:21.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:21.522+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.523192+0000) 2020-02-05T20:45:21.525 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:21.523+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.523610+0000) 2020-02-05T20:45:21.527 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:21.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65769 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:21.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:21.525+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:21.526745+0000) 2020-02-05T20:45:22.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:22.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:22.169182+0000) 2020-02-05T20:45:22.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:22.284+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:22.284812+0000) 2020-02-05T20:45:23.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:23.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:23.411324+0000) 2020-02-05T20:45:23.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:23.441+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:23.442214+0000) 2020-02-05T20:45:24.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:24.911+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:24.912758+0000) 2020-02-05T20:45:25.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:25.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:25.786343+0000) 2020-02-05T20:45:25.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:25.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:25.823218+0000) 2020-02-05T20:45:26.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:26.530+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.531469+0000) 2020-02-05T20:45:26.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:26.532+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.532503+0000) 2020-02-05T20:45:26.534 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:26.532+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.533084+0000) 2020-02-05T20:45:26.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:26.533+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.533851+0000) 2020-02-05T20:45:26.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:26.533+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.534275+0000) 2020-02-05T20:45:26.536 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:26.533+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.534465+0000) 2020-02-05T20:45:26.536 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:26.533+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.534671+0000) 2020-02-05T20:45:26.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:26.534+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.535331+0000) 2020-02-05T20:45:26.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:26.534+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.535157+0000) 2020-02-05T20:45:26.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:26.534+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.535348+0000) 2020-02-05T20:45:26.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:26.534+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.535680+0000) 2020-02-05T20:45:26.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:26.535+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.535733+0000) 2020-02-05T20:45:26.538 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:26.535+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.536389+0000) 2020-02-05T20:45:26.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:26.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:26.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:26.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.543008+0000) 2020-02-05T20:45:26.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:26.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.683368+0000) 2020-02-05T20:45:26.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:26.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.926634+0000) 2020-02-05T20:45:26.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:26.927+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:26.928238+0000) 2020-02-05T20:45:28.388 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:45:28.395 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:45:29.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:29.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:29.321532+0000) 2020-02-05T20:45:29.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:29.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:29.930870+0000) 2020-02-05T20:45:30.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:30.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:30.180075+0000) 2020-02-05T20:45:30.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:30.190+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:30.191321+0000) 2020-02-05T20:45:31.547 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:31.545+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.545643+0000) 2020-02-05T20:45:31.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:31.545+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.546049+0000) 2020-02-05T20:45:31.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:31.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.546755+0000) 2020-02-05T20:45:31.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:31.546+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.547159+0000) 2020-02-05T20:45:31.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:31.546+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.547580+0000) 2020-02-05T20:45:31.549 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:31.547+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.548137+0000) 2020-02-05T20:45:31.550 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:31.547+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.548480+0000) 2020-02-05T20:45:31.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:31.548+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.549266+0000) 2020-02-05T20:45:31.551 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:31.549+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.550235+0000) 2020-02-05T20:45:31.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:31.549+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.550257+0000) 2020-02-05T20:45:31.552 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:31.549+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.550281+0000) 2020-02-05T20:45:31.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:31.550+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.550657+0000) 2020-02-05T20:45:31.553 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:31.550+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.551389+0000) 2020-02-05T20:45:31.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:31.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65849 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:31.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:31.557+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:31.558140+0000) 2020-02-05T20:45:32.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:32.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:32.169418+0000) 2020-02-05T20:45:32.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:32.284+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:32.285046+0000) 2020-02-05T20:45:33.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:33.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:33.411685+0000) 2020-02-05T20:45:33.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:33.441+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:33.442482+0000) 2020-02-05T20:45:34.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:34.920+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:34.921141+0000) 2020-02-05T20:45:35.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:35.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:35.786582+0000) 2020-02-05T20:45:35.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:35.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:35.823448+0000) 2020-02-05T20:45:36.561 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:36.559+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.560099+0000) 2020-02-05T20:45:36.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:36.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.560420+0000) 2020-02-05T20:45:36.562 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:36.559+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.560723+0000) 2020-02-05T20:45:36.562 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:36.560+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.561334+0000) 2020-02-05T20:45:36.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:36.561+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.562164+0000) 2020-02-05T20:45:36.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:36.562+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.562735+0000) 2020-02-05T20:45:36.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:36.561+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.563011+0000) 2020-02-05T20:45:36.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:36.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.563561+0000) 2020-02-05T20:45:36.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:36.563+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.564245+0000) 2020-02-05T20:45:36.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:36.563+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.564000+0000) 2020-02-05T20:45:36.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:36.564+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.564778+0000) 2020-02-05T20:45:36.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:36.564+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.565152+0000) 2020-02-05T20:45:36.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:36.564+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.565471+0000) 2020-02-05T20:45:36.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:36.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:36.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:36.569+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.571060+0000) 2020-02-05T20:45:36.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:36.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.683595+0000) 2020-02-05T20:45:36.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:36.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.926864+0000) 2020-02-05T20:45:36.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:36.928+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:36.928481+0000) 2020-02-05T20:45:39.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:39.320+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:39.321760+0000) 2020-02-05T20:45:39.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:39.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:39.931132+0000) 2020-02-05T20:45:40.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:40.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:40.180333+0000) 2020-02-05T20:45:40.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:40.190+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:40.191536+0000) 2020-02-05T20:45:41.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:41.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.573655+0000) 2020-02-05T20:45:41.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:41.574+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.574788+0000) 2020-02-05T20:45:41.576 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:41.574+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.575156+0000) 2020-02-05T20:45:41.577 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:41.575+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.575974+0000) 2020-02-05T20:45:41.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:41.574+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.575886+0000) 2020-02-05T20:45:41.578 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:41.575+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.576621+0000) 2020-02-05T20:45:41.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:41.576+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.577165+0000) 2020-02-05T20:45:41.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:41.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.577045+0000) 2020-02-05T20:45:41.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:41.577+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.577634+0000) 2020-02-05T20:45:41.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:41.577+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.577748+0000) 2020-02-05T20:45:41.580 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:41.577+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.578571+0000) 2020-02-05T20:45:41.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:41.577+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.578657+0000) 2020-02-05T20:45:41.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:41.578+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.578914+0000) 2020-02-05T20:45:41.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:41.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65931 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:41.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:41.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:41.584850+0000) 2020-02-05T20:45:42.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:42.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:42.169694+0000) 2020-02-05T20:45:42.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:42.285+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:42.285287+0000) 2020-02-05T20:45:43.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:43.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:43.411946+0000) 2020-02-05T20:45:43.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:43.442+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:43.442723+0000) 2020-02-05T20:45:44.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:44.929+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:44.929512+0000) 2020-02-05T20:45:45.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:45.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:45.786828+0000) 2020-02-05T20:45:45.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:45.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:45.823681+0000) 2020-02-05T20:45:46.589 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:46.587+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.588414+0000) 2020-02-05T20:45:46.590 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:46.587+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.587911+0000) 2020-02-05T20:45:46.590 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:46.588+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.589176+0000) 2020-02-05T20:45:46.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:46.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.589655+0000) 2020-02-05T20:45:46.591 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:46.589+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.590314+0000) 2020-02-05T20:45:46.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:46.590+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.590894+0000) 2020-02-05T20:45:46.592 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:46.590+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.591339+0000) 2020-02-05T20:45:46.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:46.591+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.592184+0000) 2020-02-05T20:45:46.594 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:46.592+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.592708+0000) 2020-02-05T20:45:46.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:46.592+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.593847+0000) 2020-02-05T20:45:46.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:46.592+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.593931+0000) 2020-02-05T20:45:46.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:46.593+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.594136+0000) 2020-02-05T20:45:46.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:46.593+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.594346+0000) 2020-02-05T20:45:46.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:46.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 65972 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:46.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:46.597+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.598775+0000) 2020-02-05T20:45:46.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:46.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.683808+0000) 2020-02-05T20:45:46.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:46.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.927093+0000) 2020-02-05T20:45:46.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:46.928+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:46.928763+0000) 2020-02-05T20:45:49.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:49.321+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:49.322015+0000) 2020-02-05T20:45:49.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:49.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:49.931367+0000) 2020-02-05T20:45:50.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:50.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:50.180590+0000) 2020-02-05T20:45:50.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:50.191+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:50.191783+0000) 2020-02-05T20:45:51.604 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:51.602+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.603247+0000) 2020-02-05T20:45:51.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:51.602+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.602722+0000) 2020-02-05T20:45:51.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:51.603+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.604072+0000) 2020-02-05T20:45:51.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:51.603+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.604930+0000) 2020-02-05T20:45:51.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:51.604+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.605467+0000) 2020-02-05T20:45:51.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:51.604+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.605718+0000) 2020-02-05T20:45:51.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:51.604+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.605968+0000) 2020-02-05T20:45:51.608 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:51.605+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.606234+0000) 2020-02-05T20:45:51.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:51.605+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.606340+0000) 2020-02-05T20:45:51.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:51.606+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.607007+0000) 2020-02-05T20:45:51.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:51.606+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.607739+0000) 2020-02-05T20:45:51.609 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:51.607+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.608411+0000) 2020-02-05T20:45:51.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:51.608+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.609187+0000) 2020-02-05T20:45:51.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:51.611+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66011 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:51.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:51.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:51.612394+0000) 2020-02-05T20:45:52.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:52.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:52.169939+0000) 2020-02-05T20:45:52.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:52.284+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:52.285486+0000) 2020-02-05T20:45:53.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:53.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:53.412141+0000) 2020-02-05T20:45:53.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:53.442+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:53.442953+0000) 2020-02-05T20:45:54.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:54.937+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:54.937902+0000) 2020-02-05T20:45:55.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:55.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:55.787119+0000) 2020-02-05T20:45:55.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:55.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:55.823876+0000) 2020-02-05T20:45:56.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:45:56.617+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.618893+0000) 2020-02-05T20:45:56.621 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:45:56.618+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.619437+0000) 2020-02-05T20:45:56.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:45:56.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.619881+0000) 2020-02-05T20:45:56.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:45:56.618+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.620307+0000) 2020-02-05T20:45:56.622 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:56.619+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.620855+0000) 2020-02-05T20:45:56.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:45:56.619+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.621136+0000) 2020-02-05T20:45:56.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:56.620+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.621992+0000) 2020-02-05T20:45:56.623 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:56.620+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.621929+0000) 2020-02-05T20:45:56.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:56.621+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.622959+0000) 2020-02-05T20:45:56.624 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:45:56.622+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.623430+0000) 2020-02-05T20:45:56.625 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:56.622+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.623548+0000) 2020-02-05T20:45:56.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:45:56.622+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.623899+0000) 2020-02-05T20:45:56.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:56.623+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.624324+0000) 2020-02-05T20:45:56.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:45:56.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:45:56.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:56.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.632393+0000) 2020-02-05T20:45:56.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:45:56.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.684068+0000) 2020-02-05T20:45:56.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:45:56.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.927308+0000) 2020-02-05T20:45:56.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:45:56.927+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:56.928983+0000) 2020-02-05T20:45:58.526 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:45:58.532 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:45:59.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:45:59.321+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:59.322245+0000) 2020-02-05T20:45:59.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:45:59.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:45:59.931604+0000) 2020-02-05T20:46:00.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:00.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:00.180887+0000) 2020-02-05T20:46:00.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:00.191+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:00.192046+0000) 2020-02-05T20:46:01.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:01.631+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.633017+0000) 2020-02-05T20:46:01.635 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:01.631+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.633211+0000) 2020-02-05T20:46:01.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:01.632+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.633854+0000) 2020-02-05T20:46:01.636 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:01.632+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.634246+0000) 2020-02-05T20:46:01.636 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:01.633+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.634931+0000) 2020-02-05T20:46:01.638 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:01.634+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.635519+0000) 2020-02-05T20:46:01.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:01.635+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.636890+0000) 2020-02-05T20:46:01.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:01.635+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.637401+0000) 2020-02-05T20:46:01.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:01.636+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.637632+0000) 2020-02-05T20:46:01.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:01.636+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.638057+0000) 2020-02-05T20:46:01.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:01.636+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.638125+0000) 2020-02-05T20:46:01.640 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:01.637+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.638843+0000) 2020-02-05T20:46:01.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:01.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.639468+0000) 2020-02-05T20:46:01.641 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:01.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:01.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:01.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:01.642211+0000) 2020-02-05T20:46:02.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:02.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:02.170120+0000) 2020-02-05T20:46:02.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:02.284+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:02.285755+0000) 2020-02-05T20:46:03.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:03.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:03.412393+0000) 2020-02-05T20:46:03.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:03.443+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:03.443184+0000) 2020-02-05T20:46:04.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:04.946+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:04.946357+0000) 2020-02-05T20:46:05.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:05.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:05.787351+0000) 2020-02-05T20:46:05.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:05.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:05.824137+0000) 2020-02-05T20:46:06.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:06.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.647025+0000) 2020-02-05T20:46:06.649 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:06.647+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.647903+0000) 2020-02-05T20:46:06.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:06.648+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.648342+0000) 2020-02-05T20:46:06.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:06.648+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.648708+0000) 2020-02-05T20:46:06.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:06.648+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.649672+0000) 2020-02-05T20:46:06.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:06.648+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.650157+0000) 2020-02-05T20:46:06.652 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:06.650+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.650330+0000) 2020-02-05T20:46:06.652 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:06.649+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.650630+0000) 2020-02-05T20:46:06.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:06.651+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.651130+0000) 2020-02-05T20:46:06.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:06.649+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.650879+0000) 2020-02-05T20:46:06.653 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:06.649+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.651352+0000) 2020-02-05T20:46:06.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:06.650+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.651984+0000) 2020-02-05T20:46:06.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:06.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.652038+0000) 2020-02-05T20:46:06.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:06.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:06.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:06.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.655918+0000) 2020-02-05T20:46:06.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:06.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.684328+0000) 2020-02-05T20:46:06.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:06.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.927531+0000) 2020-02-05T20:46:06.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:06.927+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:06.929209+0000) 2020-02-05T20:46:09.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:09.321+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:09.322480+0000) 2020-02-05T20:46:09.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:09.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:09.931770+0000) 2020-02-05T20:46:10.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:10.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:10.181136+0000) 2020-02-05T20:46:10.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:10.191+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:10.192324+0000) 2020-02-05T20:46:11.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:11.661+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.662151+0000) 2020-02-05T20:46:11.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:11.661+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.663023+0000) 2020-02-05T20:46:11.665 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:11.661+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.662852+0000) 2020-02-05T20:46:11.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:11.662+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.663524+0000) 2020-02-05T20:46:11.665 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:11.662+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.663961+0000) 2020-02-05T20:46:11.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:11.662+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.664361+0000) 2020-02-05T20:46:11.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:11.663+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.664556+0000) 2020-02-05T20:46:11.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:11.663+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.664926+0000) 2020-02-05T20:46:11.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:11.664+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.665334+0000) 2020-02-05T20:46:11.667 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:11.663+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.665381+0000) 2020-02-05T20:46:11.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:11.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.665787+0000) 2020-02-05T20:46:11.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:11.665+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.666659+0000) 2020-02-05T20:46:11.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:11.666+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66172 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:11.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:11.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.670806+0000) 2020-02-05T20:46:11.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:11.670+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:11.671246+0000) 2020-02-05T20:46:12.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:12.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:12.170362+0000) 2020-02-05T20:46:12.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:12.285+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:12.285968+0000) 2020-02-05T20:46:13.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:13.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:13.412687+0000) 2020-02-05T20:46:13.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:13.442+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:13.443421+0000) 2020-02-05T20:46:14.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:14.953+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:14.954529+0000) 2020-02-05T20:46:15.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:15.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:15.787595+0000) 2020-02-05T20:46:15.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:15.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:15.824332+0000) 2020-02-05T20:46:16.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:16.672+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.673105+0000) 2020-02-05T20:46:16.675 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:16.672+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.673641+0000) 2020-02-05T20:46:16.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:16.674+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.675736+0000) 2020-02-05T20:46:16.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:16.675+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.676553+0000) 2020-02-05T20:46:16.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:16.675+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.676965+0000) 2020-02-05T20:46:16.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:16.676+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.677712+0000) 2020-02-05T20:46:16.679 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:16.677+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.678633+0000) 2020-02-05T20:46:16.680 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:16.677+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.679073+0000) 2020-02-05T20:46:16.680 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:16.677+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.678804+0000) 2020-02-05T20:46:16.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:16.679+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.680070+0000) 2020-02-05T20:46:16.681 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:16.678+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.679567+0000) 2020-02-05T20:46:16.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:16.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:16.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:16.681+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.682538+0000) 2020-02-05T20:46:16.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:16.681+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.682877+0000) 2020-02-05T20:46:16.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:16.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.684271+0000) 2020-02-05T20:46:16.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:16.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.684509+0000) 2020-02-05T20:46:16.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:16.926+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.927730+0000) 2020-02-05T20:46:16.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:16.928+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:16.929489+0000) 2020-02-05T20:46:19.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:19.321+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:19.322719+0000) 2020-02-05T20:46:19.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:19.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:19.932003+0000) 2020-02-05T20:46:20.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:20.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:20.181395+0000) 2020-02-05T20:46:20.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:20.191+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:20.192589+0000) 2020-02-05T20:46:21.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:21.685+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.686777+0000) 2020-02-05T20:46:21.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:21.687+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.688992+0000) 2020-02-05T20:46:21.692 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:21.689+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.690678+0000) 2020-02-05T20:46:21.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:21.689+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.691048+0000) 2020-02-05T20:46:21.693 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:21.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.691520+0000) 2020-02-05T20:46:21.693 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:21.690+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.691939+0000) 2020-02-05T20:46:21.693 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:21.691+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.692539+0000) 2020-02-05T20:46:21.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:21.691+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.693062+0000) 2020-02-05T20:46:21.694 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:21.691+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.693247+0000) 2020-02-05T20:46:21.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:21.692+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.693341+0000) 2020-02-05T20:46:21.695 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:21.692+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.694236+0000) 2020-02-05T20:46:21.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:21.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.696181+0000) 2020-02-05T20:46:21.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:21.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.696441+0000) 2020-02-05T20:46:21.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:21.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:21.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:21.696+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:21.697250+0000) 2020-02-05T20:46:22.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:22.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:22.170625+0000) 2020-02-05T20:46:22.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:22.285+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:22.286202+0000) 2020-02-05T20:46:23.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:23.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:23.412934+0000) 2020-02-05T20:46:23.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:23.442+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:23.443674+0000) 2020-02-05T20:46:24.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:24.961+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:24.961863+0000) 2020-02-05T20:46:25.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:25.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:25.787849+0000) 2020-02-05T20:46:25.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:25.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:25.824568+0000) 2020-02-05T20:46:26.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:26.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.684839+0000) 2020-02-05T20:46:26.703 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:26.700+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.701935+0000) 2020-02-05T20:46:26.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:26.702+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.703639+0000) 2020-02-05T20:46:26.706 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:26.702+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.704405+0000) 2020-02-05T20:46:26.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:26.703+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.704783+0000) 2020-02-05T20:46:26.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:26.704+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.705861+0000) 2020-02-05T20:46:26.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:26.705+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.706405+0000) 2020-02-05T20:46:26.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:26.704+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.705763+0000) 2020-02-05T20:46:26.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:26.706+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.707506+0000) 2020-02-05T20:46:26.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:26.705+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.706941+0000) 2020-02-05T20:46:26.709 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:26.706+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.708191+0000) 2020-02-05T20:46:26.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:26.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.708922+0000) 2020-02-05T20:46:26.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:26.707+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.709281+0000) 2020-02-05T20:46:26.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:26.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.709326+0000) 2020-02-05T20:46:26.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:26.710+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:26.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:26.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.714305+0000) 2020-02-05T20:46:26.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:26.927+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.927956+0000) 2020-02-05T20:46:26.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:26.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:26.929720+0000) 2020-02-05T20:46:28.673 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:46:28.679 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:46:29.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:29.322+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:29.322948+0000) 2020-02-05T20:46:29.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:29.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:29.932257+0000) 2020-02-05T20:46:30.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:30.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:30.181688+0000) 2020-02-05T20:46:30.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:30.192+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:30.192821+0000) 2020-02-05T20:46:31.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:31.717+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.718319+0000) 2020-02-05T20:46:31.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:31.717+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.718848+0000) 2020-02-05T20:46:31.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:31.718+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.719914+0000) 2020-02-05T20:46:31.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:31.719+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.720903+0000) 2020-02-05T20:46:31.722 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:31.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.720696+0000) 2020-02-05T20:46:31.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:31.720+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.721693+0000) 2020-02-05T20:46:31.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:31.720+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.721911+0000) 2020-02-05T20:46:31.723 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:31.720+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.722304+0000) 2020-02-05T20:46:31.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:31.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.723160+0000) 2020-02-05T20:46:31.724 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:31.721+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.723192+0000) 2020-02-05T20:46:31.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:31.723+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.724109+0000) 2020-02-05T20:46:31.725 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:31.723+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.724130+0000) 2020-02-05T20:46:31.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:31.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.724660+0000) 2020-02-05T20:46:31.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:31.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:31.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:31.728+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:31.729430+0000) 2020-02-05T20:46:32.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:32.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:32.170846+0000) 2020-02-05T20:46:32.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:32.285+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:32.286462+0000) 2020-02-05T20:46:33.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:33.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:33.413232+0000) 2020-02-05T20:46:33.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:33.443+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:33.443916+0000) 2020-02-05T20:46:34.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:34.969+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:34.970431+0000) 2020-02-05T20:46:35.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:35.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:35.788137+0000) 2020-02-05T20:46:35.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:35.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:35.824820+0000) 2020-02-05T20:46:36.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:36.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.685134+0000) 2020-02-05T20:46:36.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:36.731+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.732812+0000) 2020-02-05T20:46:36.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:36.732+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.733454+0000) 2020-02-05T20:46:36.735 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:36.732+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.733836+0000) 2020-02-05T20:46:36.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:36.732+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.734245+0000) 2020-02-05T20:46:36.736 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:36.733+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.734768+0000) 2020-02-05T20:46:36.736 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:36.734+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.735674+0000) 2020-02-05T20:46:36.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:36.735+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.736116+0000) 2020-02-05T20:46:36.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:36.735+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.736825+0000) 2020-02-05T20:46:36.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:36.735+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.737082+0000) 2020-02-05T20:46:36.741 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:36.736+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.738184+0000) 2020-02-05T20:46:36.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:36.737+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.738880+0000) 2020-02-05T20:46:36.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:36.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.737865+0000) 2020-02-05T20:46:36.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:36.737+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.738442+0000) 2020-02-05T20:46:36.742 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:36.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:36.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:36.741+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.742659+0000) 2020-02-05T20:46:36.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:36.927+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.928196+0000) 2020-02-05T20:46:36.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:36.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:36.929987+0000) 2020-02-05T20:46:39.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:39.322+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:39.323189+0000) 2020-02-05T20:46:39.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:39.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:39.932489+0000) 2020-02-05T20:46:40.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:40.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:40.181963+0000) 2020-02-05T20:46:40.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:40.192+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:40.193116+0000) 2020-02-05T20:46:41.748 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:41.745+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.746920+0000) 2020-02-05T20:46:41.749 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:41.746+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.747894+0000) 2020-02-05T20:46:41.750 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:41.747+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.748702+0000) 2020-02-05T20:46:41.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:41.748+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.749227+0000) 2020-02-05T20:46:41.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:41.748+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.749492+0000) 2020-02-05T20:46:41.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:41.748+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.750055+0000) 2020-02-05T20:46:41.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:41.749+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.750430+0000) 2020-02-05T20:46:41.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:41.749+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.750779+0000) 2020-02-05T20:46:41.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:41.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.750981+0000) 2020-02-05T20:46:41.753 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:41.750+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.751994+0000) 2020-02-05T20:46:41.754 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:41.750+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.752173+0000) 2020-02-05T20:46:41.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:41.750+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.752189+0000) 2020-02-05T20:46:41.754 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:41.751+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.753044+0000) 2020-02-05T20:46:41.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:41.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:41.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:41.755+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:41.756396+0000) 2020-02-05T20:46:42.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:42.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:42.171051+0000) 2020-02-05T20:46:42.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:42.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:42.286693+0000) 2020-02-05T20:46:43.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:43.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:43.413472+0000) 2020-02-05T20:46:43.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:43.443+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:43.444151+0000) 2020-02-05T20:46:44.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:44.981+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:44.981519+0000) 2020-02-05T20:46:45.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:45.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:45.788400+0000) 2020-02-05T20:46:45.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:45.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:45.825071+0000) 2020-02-05T20:46:46.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:46.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.685409+0000) 2020-02-05T20:46:46.762 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:46.760+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.761216+0000) 2020-02-05T20:46:46.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:46.762+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.763305+0000) 2020-02-05T20:46:46.765 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:46.762+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.762767+0000) 2020-02-05T20:46:46.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:46.762+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.763055+0000) 2020-02-05T20:46:46.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:46.763+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.763992+0000) 2020-02-05T20:46:46.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:46.763+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.764964+0000) 2020-02-05T20:46:46.766 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:46.765+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.765533+0000) 2020-02-05T20:46:46.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:46.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.766046+0000) 2020-02-05T20:46:46.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:46.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.766582+0000) 2020-02-05T20:46:46.768 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:46.766+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.766903+0000) 2020-02-05T20:46:46.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:46.766+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.767747+0000) 2020-02-05T20:46:46.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:46.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.767718+0000) 2020-02-05T20:46:46.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:46.767+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.768111+0000) 2020-02-05T20:46:46.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:46.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66453 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:46.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:46.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.772435+0000) 2020-02-05T20:46:46.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:46.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.928468+0000) 2020-02-05T20:46:46.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:46.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:46.930200+0000) 2020-02-05T20:46:49.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:49.323+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:49.323426+0000) 2020-02-05T20:46:49.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:49.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:49.932804+0000) 2020-02-05T20:46:50.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:50.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:50.182211+0000) 2020-02-05T20:46:50.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:50.193+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:50.193374+0000) 2020-02-05T20:46:51.784 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:51.778+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.778818+0000) 2020-02-05T20:46:51.785 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:51.779+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.779892+0000) 2020-02-05T20:46:51.785 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:51.780+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.780647+0000) 2020-02-05T20:46:51.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:51.780+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.781144+0000) 2020-02-05T20:46:51.786 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:51.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.782208+0000) 2020-02-05T20:46:51.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:51.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.775799+0000) 2020-02-05T20:46:51.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:51.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.782551+0000) 2020-02-05T20:46:51.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:51.775+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.776809+0000) 2020-02-05T20:46:51.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:51.777+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.778480+0000) 2020-02-05T20:46:51.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:51.778+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.779491+0000) 2020-02-05T20:46:51.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:51.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.781514+0000) 2020-02-05T20:46:51.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:51.781+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.782701+0000) 2020-02-05T20:46:51.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:51.782+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.782950+0000) 2020-02-05T20:46:51.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:51.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66492 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:51.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:51.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:51.788684+0000) 2020-02-05T20:46:52.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:52.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:52.171288+0000) 2020-02-05T20:46:52.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:52.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:52.286863+0000) 2020-02-05T20:46:53.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:53.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:53.413713+0000) 2020-02-05T20:46:53.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:53.444+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:53.444388+0000) 2020-02-05T20:46:54.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:54.990+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:54.991066+0000) 2020-02-05T20:46:55.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:55.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:55.788655+0000) 2020-02-05T20:46:55.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:55.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:55.825368+0000) 2020-02-05T20:46:56.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:56.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.685637+0000) 2020-02-05T20:46:56.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:56.789+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.789791+0000) 2020-02-05T20:46:56.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:56.790+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.791580+0000) 2020-02-05T20:46:56.793 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:46:56.791+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.792031+0000) 2020-02-05T20:46:56.794 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:56.791+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.792398+0000) 2020-02-05T20:46:56.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:46:56.791+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.792758+0000) 2020-02-05T20:46:56.796 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:46:56.792+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.793256+0000) 2020-02-05T20:46:56.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:56.793+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.793753+0000) 2020-02-05T20:46:56.796 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:46:56.792+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.794015+0000) 2020-02-05T20:46:56.797 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:46:56.793+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.794308+0000) 2020-02-05T20:46:56.798 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:46:56.795+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.795791+0000) 2020-02-05T20:46:56.798 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:46:56.795+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.795876+0000) 2020-02-05T20:46:56.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:46:56.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.795347+0000) 2020-02-05T20:46:56.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:46:56.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:46:56.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:56.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.801941+0000) 2020-02-05T20:46:56.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:56.801+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.802499+0000) 2020-02-05T20:46:56.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:46:56.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.928723+0000) 2020-02-05T20:46:56.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:46:56.930+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:56.930418+0000) 2020-02-05T20:46:58.814 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:46:58.821 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:46:59.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:46:59.323+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:59.323679+0000) 2020-02-05T20:46:59.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:46:59.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:46:59.933038+0000) 2020-02-05T20:47:00.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:00.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:00.182475+0000) 2020-02-05T20:47:00.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:00.193+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:00.193623+0000) 2020-02-05T20:47:01.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:01.804+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.804196+0000) 2020-02-05T20:47:01.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:01.805+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.805903+0000) 2020-02-05T20:47:01.807 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:01.805+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.806119+0000) 2020-02-05T20:47:01.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:01.806+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.806744+0000) 2020-02-05T20:47:01.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:01.807+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.807315+0000) 2020-02-05T20:47:01.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:01.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.807759+0000) 2020-02-05T20:47:01.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:01.808+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.808300+0000) 2020-02-05T20:47:01.810 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:01.807+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.808406+0000) 2020-02-05T20:47:01.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:01.808+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.808703+0000) 2020-02-05T20:47:01.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:01.808+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.809018+0000) 2020-02-05T20:47:01.811 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:01.809+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.809645+0000) 2020-02-05T20:47:01.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:01.809+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.810169+0000) 2020-02-05T20:47:01.814 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:01.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:01.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:01.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.813670+0000) 2020-02-05T20:47:01.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:01.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:01.813929+0000) 2020-02-05T20:47:02.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:02.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:02.171508+0000) 2020-02-05T20:47:02.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:02.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:02.287089+0000) 2020-02-05T20:47:03.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:03.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:03.414010+0000) 2020-02-05T20:47:03.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:03.443+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:03.444622+0000) 2020-02-05T20:47:05.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:04.998+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:04.999642+0000) 2020-02-05T20:47:05.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:05.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:05.788887+0000) 2020-02-05T20:47:05.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:05.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:05.825602+0000) 2020-02-05T20:47:06.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:06.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.685912+0000) 2020-02-05T20:47:06.821 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:06.818+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.818545+0000) 2020-02-05T20:47:06.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:06.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.819233+0000) 2020-02-05T20:47:06.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:06.819+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.820502+0000) 2020-02-05T20:47:06.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:06.819+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.820891+0000) 2020-02-05T20:47:06.822 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:06.820+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.821262+0000) 2020-02-05T20:47:06.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:06.820+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.821435+0000) 2020-02-05T20:47:06.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:06.821+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.822101+0000) 2020-02-05T20:47:06.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:06.821+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.822658+0000) 2020-02-05T20:47:06.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:06.822+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.823625+0000) 2020-02-05T20:47:06.825 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:06.823+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.823539+0000) 2020-02-05T20:47:06.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:06.823+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.824018+0000) 2020-02-05T20:47:06.826 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:06.824+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.824682+0000) 2020-02-05T20:47:06.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:06.823+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.825005+0000) 2020-02-05T20:47:06.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:06.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:06.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:06.827+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.828819+0000) 2020-02-05T20:47:06.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:06.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.929001+0000) 2020-02-05T20:47:06.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:06.930+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:06.930615+0000) 2020-02-05T20:47:09.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:09.322+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:09.323967+0000) 2020-02-05T20:47:09.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:09.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:09.933301+0000) 2020-02-05T20:47:10.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:10.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:10.182659+0000) 2020-02-05T20:47:10.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:10.192+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:10.193845+0000) 2020-02-05T20:47:11.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:11.833+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.834107+0000) 2020-02-05T20:47:11.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:11.834+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.835163+0000) 2020-02-05T20:47:11.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:11.835+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.836292+0000) 2020-02-05T20:47:11.840 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:11.836+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.837022+0000) 2020-02-05T20:47:11.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:11.831+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.832243+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:11.832+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.833268+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:11.833+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.834726+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:11.836+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.837709+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:11.834+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.835372+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:11.835+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.836508+0000) 2020-02-05T20:47:11.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:11.837+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.838365+0000) 2020-02-05T20:47:11.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:11.838+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.838583+0000) 2020-02-05T20:47:11.842 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:11.838+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.838589+0000) 2020-02-05T20:47:11.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:11.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:11.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:11.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:11.843772+0000) 2020-02-05T20:47:12.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:12.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:12.171796+0000) 2020-02-05T20:47:12.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:12.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:12.287302+0000) 2020-02-05T20:47:13.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:13.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:13.414239+0000) 2020-02-05T20:47:13.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:13.443+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:13.444857+0000) 2020-02-05T20:47:15.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:15.010+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:15.011494+0000) 2020-02-05T20:47:15.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:15.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:15.789135+0000) 2020-02-05T20:47:15.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:15.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:15.825836+0000) 2020-02-05T20:47:16.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:16.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.686148+0000) 2020-02-05T20:47:16.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:16.845+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.846991+0000) 2020-02-05T20:47:16.851 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:16.849+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.849506+0000) 2020-02-05T20:47:16.851 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:16.849+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.849658+0000) 2020-02-05T20:47:16.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:16.849+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.850913+0000) 2020-02-05T20:47:16.853 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:16.850+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.851164+0000) 2020-02-05T20:47:16.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:16.851+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.852153+0000) 2020-02-05T20:47:16.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:16.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.853424+0000) 2020-02-05T20:47:16.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:16.852+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.853458+0000) 2020-02-05T20:47:16.855 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:16.852+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.853562+0000) 2020-02-05T20:47:16.856 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:16.853+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.853491+0000) 2020-02-05T20:47:16.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:16.853+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.853962+0000) 2020-02-05T20:47:16.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:16.853+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.854580+0000) 2020-02-05T20:47:16.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:16.855+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.856131+0000) 2020-02-05T20:47:16.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:16.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66693 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:16.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:16.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.857999+0000) 2020-02-05T20:47:16.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:16.929+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.929231+0000) 2020-02-05T20:47:16.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:16.930+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:16.930784+0000) 2020-02-05T20:47:19.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:19.323+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:19.324168+0000) 2020-02-05T20:47:19.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:19.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:19.933558+0000) 2020-02-05T20:47:20.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:20.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:20.182896+0000) 2020-02-05T20:47:20.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:20.193+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:20.194069+0000) 2020-02-05T20:47:21.864 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:21.861+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.862925+0000) 2020-02-05T20:47:21.865 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:21.862+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.863686+0000) 2020-02-05T20:47:21.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:21.863+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.864571+0000) 2020-02-05T20:47:21.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:21.863+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.864902+0000) 2020-02-05T20:47:21.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:21.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.865318+0000) 2020-02-05T20:47:21.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:21.864+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.866005+0000) 2020-02-05T20:47:21.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:21.864+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.866258+0000) 2020-02-05T20:47:21.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:21.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.867037+0000) 2020-02-05T20:47:21.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:21.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.867609+0000) 2020-02-05T20:47:21.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:21.866+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.867937+0000) 2020-02-05T20:47:21.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:21.867+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.868455+0000) 2020-02-05T20:47:21.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:21.867+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.868612+0000) 2020-02-05T20:47:21.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:21.868+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.869292+0000) 2020-02-05T20:47:21.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:21.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:21.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:21.874+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:21.875649+0000) 2020-02-05T20:47:22.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:22.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:22.172034+0000) 2020-02-05T20:47:22.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:22.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:22.287502+0000) 2020-02-05T20:47:23.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:23.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:23.414475+0000) 2020-02-05T20:47:23.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:23.444+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:23.445125+0000) 2020-02-05T20:47:25.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:25.018+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:25.019456+0000) 2020-02-05T20:47:25.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:25.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:25.789322+0000) 2020-02-05T20:47:25.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:25.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:25.826069+0000) 2020-02-05T20:47:26.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:26.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.686379+0000) 2020-02-05T20:47:26.878 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:26.875+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.877055+0000) 2020-02-05T20:47:26.879 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:26.876+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.877338+0000) 2020-02-05T20:47:26.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:26.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.878491+0000) 2020-02-05T20:47:26.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:26.879+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.880394+0000) 2020-02-05T20:47:26.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:26.878+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.880310+0000) 2020-02-05T20:47:26.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:26.880+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.881657+0000) 2020-02-05T20:47:26.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:26.880+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.881937+0000) 2020-02-05T20:47:26.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:26.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.882122+0000) 2020-02-05T20:47:26.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:26.881+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.882638+0000) 2020-02-05T20:47:26.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:26.881+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.882938+0000) 2020-02-05T20:47:26.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:26.881+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.882904+0000) 2020-02-05T20:47:26.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:26.882+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.883700+0000) 2020-02-05T20:47:26.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:26.884+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.885085+0000) 2020-02-05T20:47:26.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:26.884+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:26.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:26.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.886212+0000) 2020-02-05T20:47:26.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:26.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.929472+0000) 2020-02-05T20:47:26.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:26.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:26.930999+0000) 2020-02-05T20:47:28.978 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:47:28.985 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:47:29.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:29.323+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:29.324413+0000) 2020-02-05T20:47:29.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:29.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:29.933792+0000) 2020-02-05T20:47:30.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:30.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:30.183152+0000) 2020-02-05T20:47:30.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:30.193+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:30.194370+0000) 2020-02-05T20:47:31.892 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:31.889+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.890890+0000) 2020-02-05T20:47:31.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:31.890+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.891733+0000) 2020-02-05T20:47:31.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:31.891+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.892923+0000) 2020-02-05T20:47:31.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:31.892+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.893607+0000) 2020-02-05T20:47:31.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:31.892+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.894052+0000) 2020-02-05T20:47:31.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:31.893+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.894854+0000) 2020-02-05T20:47:31.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:31.893+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.895114+0000) 2020-02-05T20:47:31.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:31.894+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.895429+0000) 2020-02-05T20:47:31.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:31.895+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.896489+0000) 2020-02-05T20:47:31.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:31.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.896849+0000) 2020-02-05T20:47:31.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:31.895+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.896951+0000) 2020-02-05T20:47:31.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:31.895+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.897373+0000) 2020-02-05T20:47:31.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:31.897+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.898368+0000) 2020-02-05T20:47:31.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:31.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:31.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:31.901+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:31.902858+0000) 2020-02-05T20:47:32.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:32.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:32.172334+0000) 2020-02-05T20:47:32.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:32.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:32.287709+0000) 2020-02-05T20:47:33.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:33.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:33.414707+0000) 2020-02-05T20:47:33.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:33.444+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:33.445404+0000) 2020-02-05T20:47:35.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:35.030+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:35.030832+0000) 2020-02-05T20:47:35.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:35.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:35.789608+0000) 2020-02-05T20:47:35.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:35.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:35.826369+0000) 2020-02-05T20:47:36.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:36.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.686623+0000) 2020-02-05T20:47:36.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:36.904+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.905910+0000) 2020-02-05T20:47:36.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:36.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.906581+0000) 2020-02-05T20:47:36.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:36.906+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.907164+0000) 2020-02-05T20:47:36.908 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:36.906+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.907407+0000) 2020-02-05T20:47:36.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:36.906+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.907806+0000) 2020-02-05T20:47:36.910 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:36.907+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.908848+0000) 2020-02-05T20:47:36.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:36.908+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.909147+0000) 2020-02-05T20:47:36.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:36.908+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.909638+0000) 2020-02-05T20:47:36.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:36.908+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.910020+0000) 2020-02-05T20:47:36.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:36.909+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.910437+0000) 2020-02-05T20:47:36.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:36.909+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.910891+0000) 2020-02-05T20:47:36.912 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:36.910+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.911683+0000) 2020-02-05T20:47:36.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:36.911+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.912543+0000) 2020-02-05T20:47:36.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:36.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66853 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:36.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:36.916+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.917232+0000) 2020-02-05T20:47:36.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:36.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.929708+0000) 2020-02-05T20:47:36.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:36.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:36.931189+0000) 2020-02-05T20:47:39.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:39.324+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:39.324675+0000) 2020-02-05T20:47:39.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:39.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:39.934021+0000) 2020-02-05T20:47:40.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:40.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:40.183403+0000) 2020-02-05T20:47:40.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:40.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:40.194609+0000) 2020-02-05T20:47:41.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:41.919+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.920666+0000) 2020-02-05T20:47:41.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:41.918+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.920091+0000) 2020-02-05T20:47:41.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:41.920+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.921603+0000) 2020-02-05T20:47:41.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:41.921+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.922166+0000) 2020-02-05T20:47:41.924 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:41.922+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.923155+0000) 2020-02-05T20:47:41.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:41.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.923641+0000) 2020-02-05T20:47:41.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:41.923+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.924505+0000) 2020-02-05T20:47:41.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:41.924+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.925626+0000) 2020-02-05T20:47:41.927 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:41.924+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.925552+0000) 2020-02-05T20:47:41.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:41.924+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.926389+0000) 2020-02-05T20:47:41.928 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:41.925+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.926786+0000) 2020-02-05T20:47:41.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:41.926+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.927290+0000) 2020-02-05T20:47:41.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:41.925+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.927364+0000) 2020-02-05T20:47:41.931 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:41.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:41.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:41.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:41.931105+0000) 2020-02-05T20:47:42.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:42.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:42.172576+0000) 2020-02-05T20:47:42.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:42.287+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:42.287934+0000) 2020-02-05T20:47:43.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:43.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:43.414966+0000) 2020-02-05T20:47:43.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:43.445+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:43.445643+0000) 2020-02-05T20:47:45.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:45.038+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:45.039142+0000) 2020-02-05T20:47:45.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:45.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:45.789850+0000) 2020-02-05T20:47:45.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:45.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:45.826600+0000) 2020-02-05T20:47:46.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:46.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.686795+0000) 2020-02-05T20:47:46.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:46.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.929920+0000) 2020-02-05T20:47:46.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:46.929+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.931375+0000) 2020-02-05T20:47:46.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:46.935+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.936907+0000) 2020-02-05T20:47:46.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:46.936+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.937637+0000) 2020-02-05T20:47:46.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:46.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.938162+0000) 2020-02-05T20:47:46.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:46.937+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.938599+0000) 2020-02-05T20:47:46.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:46.937+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.938750+0000) 2020-02-05T20:47:46.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:46.938+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.939422+0000) 2020-02-05T20:47:46.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:46.937+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.939232+0000) 2020-02-05T20:47:46.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:46.938+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.939844+0000) 2020-02-05T20:47:46.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:46.940+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.941240+0000) 2020-02-05T20:47:46.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:46.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.941623+0000) 2020-02-05T20:47:46.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:46.940+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.941745+0000) 2020-02-05T20:47:46.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:46.940+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.941879+0000) 2020-02-05T20:47:46.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:46.940+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.942121+0000) 2020-02-05T20:47:46.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:46.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:46.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:46.944+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:46.945577+0000) 2020-02-05T20:47:49.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:49.324+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:49.324908+0000) 2020-02-05T20:47:49.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:49.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:49.934286+0000) 2020-02-05T20:47:50.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:50.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:50.183697+0000) 2020-02-05T20:47:50.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:50.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:50.194871+0000) 2020-02-05T20:47:51.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:51.948+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.949920+0000) 2020-02-05T20:47:51.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:51.949+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.950362+0000) 2020-02-05T20:47:51.953 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:51.950+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.952061+0000) 2020-02-05T20:47:51.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:51.952+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.953160+0000) 2020-02-05T20:47:51.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:51.952+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.953703+0000) 2020-02-05T20:47:51.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:51.953+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.954195+0000) 2020-02-05T20:47:51.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:51.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.955495+0000) 2020-02-05T20:47:51.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:51.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.955750+0000) 2020-02-05T20:47:51.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:51.955+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.956283+0000) 2020-02-05T20:47:51.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:51.955+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.956380+0000) 2020-02-05T20:47:51.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:51.953+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.954837+0000) 2020-02-05T20:47:51.961 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:51.955+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.956671+0000) 2020-02-05T20:47:51.961 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:51.955+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.956823+0000) 2020-02-05T20:47:51.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:51.958+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 66972 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:51.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:51.961+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:51.963032+0000) 2020-02-05T20:47:52.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:52.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:52.172879+0000) 2020-02-05T20:47:52.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:52.286+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:52.288141+0000) 2020-02-05T20:47:53.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:53.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:53.415223+0000) 2020-02-05T20:47:53.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:53.445+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:53.445918+0000) 2020-02-05T20:47:55.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:55.047+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:55.047617+0000) 2020-02-05T20:47:55.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:55.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:55.790121+0000) 2020-02-05T20:47:55.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:55.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:55.826889+0000) 2020-02-05T20:47:56.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:56.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.687040+0000) 2020-02-05T20:47:56.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:56.928+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.930210+0000) 2020-02-05T20:47:56.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:56.930+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.931631+0000) 2020-02-05T20:47:56.967 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:47:56.965+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.965530+0000) 2020-02-05T20:47:56.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:47:56.965+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.967069+0000) 2020-02-05T20:47:56.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:56.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.967516+0000) 2020-02-05T20:47:56.969 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:47:56.968+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.968056+0000) 2020-02-05T20:47:56.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:47:56.968+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.968328+0000) 2020-02-05T20:47:56.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:47:56.967+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.969057+0000) 2020-02-05T20:47:56.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:47:56.969+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.969140+0000) 2020-02-05T20:47:56.971 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:47:56.968+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.970121+0000) 2020-02-05T20:47:56.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:56.970+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.970468+0000) 2020-02-05T20:47:56.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:47:56.969+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.970828+0000) 2020-02-05T20:47:56.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:47:56.969+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.971348+0000) 2020-02-05T20:47:56.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:47:56.969+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.971131+0000) 2020-02-05T20:47:56.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:56.973+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.973448+0000) 2020-02-05T20:47:56.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:47:56.972+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67012 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:47:56.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:56.973+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:56.973709+0000) 2020-02-05T20:47:59.173 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:47:59.180 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:47:59.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:47:59.324+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:59.325146+0000) 2020-02-05T20:47:59.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:47:59.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:47:59.934561+0000) 2020-02-05T20:48:00.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:00.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:00.183933+0000) 2020-02-05T20:48:00.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:00.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:00.195126+0000) 2020-02-05T20:48:01.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:01.977+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.978849+0000) 2020-02-05T20:48:02.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:01.978+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.979835+0000) 2020-02-05T20:48:02.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:01.979+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.981188+0000) 2020-02-05T20:48:02.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:02.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:02.173128+0000) 2020-02-05T20:48:02.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:01.979+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.981175+0000) 2020-02-05T20:48:02.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:02.287+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:02.288376+0000) 2020-02-05T20:48:02.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:01.981+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.982521+0000) 2020-02-05T20:48:02.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:01.982+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.983762+0000) 2020-02-05T20:48:02.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:01.982+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.984138+0000) 2020-02-05T20:48:02.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:01.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:02.309 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:01.979+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.980371+0000) 2020-02-05T20:48:02.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:01.980+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.981827+0000) 2020-02-05T20:48:02.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:01.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.982187+0000) 2020-02-05T20:48:02.310 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:01.981+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.982629+0000) 2020-02-05T20:48:02.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:01.982+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.983343+0000) 2020-02-05T20:48:02.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:01.982+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.983924+0000) 2020-02-05T20:48:02.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:01.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:01.988551+0000) 2020-02-05T20:48:03.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:03.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:03.415484+0000) 2020-02-05T20:48:03.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:03.445+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:03.446185+0000) 2020-02-05T20:48:05.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:05.054+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:05.055723+0000) 2020-02-05T20:48:05.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:05.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:05.790397+0000) 2020-02-05T20:48:05.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:05.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:05.827122+0000) 2020-02-05T20:48:06.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:06.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.687305+0000) 2020-02-05T20:48:06.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:06.929+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.930471+0000) 2020-02-05T20:48:06.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:06.931+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.931861+0000) 2020-02-05T20:48:06.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:06.992+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.993371+0000) 2020-02-05T20:48:06.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:06.992+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.993657+0000) 2020-02-05T20:48:06.996 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:06.993+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.994952+0000) 2020-02-05T20:48:06.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:06.993+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.995336+0000) 2020-02-05T20:48:06.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:06.994+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.995739+0000) 2020-02-05T20:48:06.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:06.995+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.996410+0000) 2020-02-05T20:48:06.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:06.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.996829+0000) 2020-02-05T20:48:06.998 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:06.996+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.997318+0000) 2020-02-05T20:48:06.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:06.995+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.997239+0000) 2020-02-05T20:48:06.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:06.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.998066+0000) 2020-02-05T20:48:07.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:06.997+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.998797+0000) 2020-02-05T20:48:07.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:06.997+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.998948+0000) 2020-02-05T20:48:07.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:06.997+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:06.999393+0000) 2020-02-05T20:48:07.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:07.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:07.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:07.004+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:07.005805+0000) 2020-02-05T20:48:09.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:09.324+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:09.325456+0000) 2020-02-05T20:48:09.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:09.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:09.934807+0000) 2020-02-05T20:48:10.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:10.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:10.184191+0000) 2020-02-05T20:48:10.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:10.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:10.195308+0000) 2020-02-05T20:48:12.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:12.008+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.009368+0000) 2020-02-05T20:48:12.011 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:12.008+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.009076+0000) 2020-02-05T20:48:12.011 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:12.009+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.010405+0000) 2020-02-05T20:48:12.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:12.009+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.010736+0000) 2020-02-05T20:48:12.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:12.011+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.011708+0000) 2020-02-05T20:48:12.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:12.011+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.012425+0000) 2020-02-05T20:48:12.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:12.011+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.012594+0000) 2020-02-05T20:48:12.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:12.012+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.012517+0000) 2020-02-05T20:48:12.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:12.012+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.012686+0000) 2020-02-05T20:48:12.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:12.011+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.012946+0000) 2020-02-05T20:48:12.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:12.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.013291+0000) 2020-02-05T20:48:12.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:12.013+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.013623+0000) 2020-02-05T20:48:12.016 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:12.014+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.014564+0000) 2020-02-05T20:48:12.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:12.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:12.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:12.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.020168+0000) 2020-02-05T20:48:12.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:12.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.173368+0000) 2020-02-05T20:48:12.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:12.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:12.288657+0000) 2020-02-05T20:48:13.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:13.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:13.415743+0000) 2020-02-05T20:48:13.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:13.445+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:13.446408+0000) 2020-02-05T20:48:15.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:15.066+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:15.067360+0000) 2020-02-05T20:48:15.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:15.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:15.790596+0000) 2020-02-05T20:48:15.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:15.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:15.827334+0000) 2020-02-05T20:48:16.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:16.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:16.687542+0000) 2020-02-05T20:48:16.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:16.930+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:16.930718+0000) 2020-02-05T20:48:16.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:16.931+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:16.932079+0000) 2020-02-05T20:48:17.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:17.021+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.021510+0000) 2020-02-05T20:48:17.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:17.021+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.022382+0000) 2020-02-05T20:48:17.025 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:17.022+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.023207+0000) 2020-02-05T20:48:17.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:17.022+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.023592+0000) 2020-02-05T20:48:17.025 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:17.022+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.023751+0000) 2020-02-05T20:48:17.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:17.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.024333+0000) 2020-02-05T20:48:17.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:17.023+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.024567+0000) 2020-02-05T20:48:17.026 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:17.024+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.024841+0000) 2020-02-05T20:48:17.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:17.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.025099+0000) 2020-02-05T20:48:17.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:17.023+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.024868+0000) 2020-02-05T20:48:17.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:17.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.025396+0000) 2020-02-05T20:48:17.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:17.025+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.026382+0000) 2020-02-05T20:48:17.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:17.026+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.026581+0000) 2020-02-05T20:48:17.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:17.029+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:17.030114+0000) 2020-02-05T20:48:17.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:17.029+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67172 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:19.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:19.325+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:19.325695+0000) 2020-02-05T20:48:19.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:19.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:19.935035+0000) 2020-02-05T20:48:20.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:20.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:20.184417+0000) 2020-02-05T20:48:20.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:20.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:20.195534+0000) 2020-02-05T20:48:22.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:22.035+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.036943+0000) 2020-02-05T20:48:22.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:22.035+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.037036+0000) 2020-02-05T20:48:22.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:22.037+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.038085+0000) 2020-02-05T20:48:22.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:22.036+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.037628+0000) 2020-02-05T20:48:22.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:22.036+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.038044+0000) 2020-02-05T20:48:22.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:22.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.038533+0000) 2020-02-05T20:48:22.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:22.038+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.038704+0000) 2020-02-05T20:48:22.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:22.038+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.039248+0000) 2020-02-05T20:48:22.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:22.039+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.039509+0000) 2020-02-05T20:48:22.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:22.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.040151+0000) 2020-02-05T20:48:22.044 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:22.038+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.039914+0000) 2020-02-05T20:48:22.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:22.039+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.040672+0000) 2020-02-05T20:48:22.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:22.040+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.041206+0000) 2020-02-05T20:48:22.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:22.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:22.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:22.043+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.044432+0000) 2020-02-05T20:48:22.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:22.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.173628+0000) 2020-02-05T20:48:22.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:22.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:22.288891+0000) 2020-02-05T20:48:23.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:23.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:23.415990+0000) 2020-02-05T20:48:23.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:23.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:23.446690+0000) 2020-02-05T20:48:25.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:25.074+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:25.075397+0000) 2020-02-05T20:48:25.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:25.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:25.790828+0000) 2020-02-05T20:48:25.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:25.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:25.827562+0000) 2020-02-05T20:48:26.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:26.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:26.687811+0000) 2020-02-05T20:48:26.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:26.930+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:26.931018+0000) 2020-02-05T20:48:26.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:26.931+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:26.932327+0000) 2020-02-05T20:48:27.050 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:27.048+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.049489+0000) 2020-02-05T20:48:27.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:27.049+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.050278+0000) 2020-02-05T20:48:27.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:27.050+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.051168+0000) 2020-02-05T20:48:27.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:27.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.051655+0000) 2020-02-05T20:48:27.053 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:27.051+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.051950+0000) 2020-02-05T20:48:27.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:27.051+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.052338+0000) 2020-02-05T20:48:27.054 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:27.051+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.052706+0000) 2020-02-05T20:48:27.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:27.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.053575+0000) 2020-02-05T20:48:27.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:27.053+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.054055+0000) 2020-02-05T20:48:27.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:27.054+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.054783+0000) 2020-02-05T20:48:27.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:27.054+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.055090+0000) 2020-02-05T20:48:27.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:27.054+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.055295+0000) 2020-02-05T20:48:27.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:27.054+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.055369+0000) 2020-02-05T20:48:27.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:27.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:27.058692+0000) 2020-02-05T20:48:27.060 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:27.058+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:29.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:29.325+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:29.325925+0000) 2020-02-05T20:48:29.328 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:48:29.335 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:48:29.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:29.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:29.935301+0000) 2020-02-05T20:48:30.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:30.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:30.184710+0000) 2020-02-05T20:48:30.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:30.194+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:30.195753+0000) 2020-02-05T20:48:32.067 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:32.064+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.065919+0000) 2020-02-05T20:48:32.068 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:32.065+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.066383+0000) 2020-02-05T20:48:32.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:32.065+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.066684+0000) 2020-02-05T20:48:32.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:32.066+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.067288+0000) 2020-02-05T20:48:32.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:32.067+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.067613+0000) 2020-02-05T20:48:32.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:32.068+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.068568+0000) 2020-02-05T20:48:32.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:32.068+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.069505+0000) 2020-02-05T20:48:32.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:32.068+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.069829+0000) 2020-02-05T20:48:32.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:32.068+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.068943+0000) 2020-02-05T20:48:32.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:32.069+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.069830+0000) 2020-02-05T20:48:32.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:32.069+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.070717+0000) 2020-02-05T20:48:32.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:32.069+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.070259+0000) 2020-02-05T20:48:32.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:32.070+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.070850+0000) 2020-02-05T20:48:32.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:32.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:32.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:32.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.075042+0000) 2020-02-05T20:48:32.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:32.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.173799+0000) 2020-02-05T20:48:32.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:32.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:32.289062+0000) 2020-02-05T20:48:33.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:33.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:33.416244+0000) 2020-02-05T20:48:33.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:33.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:33.446943+0000) 2020-02-05T20:48:35.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:35.083+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:35.083905+0000) 2020-02-05T20:48:35.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:35.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:35.791122+0000) 2020-02-05T20:48:35.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:35.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:35.827799+0000) 2020-02-05T20:48:36.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:36.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:36.688046+0000) 2020-02-05T20:48:36.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:36.931+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:36.931269+0000) 2020-02-05T20:48:36.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:36.932+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:36.932540+0000) 2020-02-05T20:48:37.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:37.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.079971+0000) 2020-02-05T20:48:37.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:37.080+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.080930+0000) 2020-02-05T20:48:37.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:37.080+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.080657+0000) 2020-02-05T20:48:37.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:37.080+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.081425+0000) 2020-02-05T20:48:37.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:37.081+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.082232+0000) 2020-02-05T20:48:37.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:37.083+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.083512+0000) 2020-02-05T20:48:37.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:37.083+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.083796+0000) 2020-02-05T20:48:37.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:37.083+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.084269+0000) 2020-02-05T20:48:37.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:37.083+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.084886+0000) 2020-02-05T20:48:37.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:37.084+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.085153+0000) 2020-02-05T20:48:37.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:37.085+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.085658+0000) 2020-02-05T20:48:37.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:37.085+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.085950+0000) 2020-02-05T20:48:37.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:37.085+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.086789+0000) 2020-02-05T20:48:37.089 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:37.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:37.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:37.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:37.091706+0000) 2020-02-05T20:48:39.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:39.325+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:39.326149+0000) 2020-02-05T20:48:39.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:39.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:39.935544+0000) 2020-02-05T20:48:40.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:40.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:40.184972+0000) 2020-02-05T20:48:40.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:40.195+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:40.195997+0000) 2020-02-05T20:48:42.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:42.092+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.093556+0000) 2020-02-05T20:48:42.095 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:42.093+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.094616+0000) 2020-02-05T20:48:42.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:42.094+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.095045+0000) 2020-02-05T20:48:42.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:42.096+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.096580+0000) 2020-02-05T20:48:42.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:42.096+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.097086+0000) 2020-02-05T20:48:42.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:42.097+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.097802+0000) 2020-02-05T20:48:42.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:42.098+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.098735+0000) 2020-02-05T20:48:42.102 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:42.099+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.099535+0000) 2020-02-05T20:48:42.114 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:42.097+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.098485+0000) 2020-02-05T20:48:42.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:42.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.098924+0000) 2020-02-05T20:48:42.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:42.100+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.101714+0000) 2020-02-05T20:48:42.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:42.103+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.104591+0000) 2020-02-05T20:48:42.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:42.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.099505+0000) 2020-02-05T20:48:42.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:42.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.099989+0000) 2020-02-05T20:48:42.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:42.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:42.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:42.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.174037+0000) 2020-02-05T20:48:42.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:42.289+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:42.289271+0000) 2020-02-05T20:48:43.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:43.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:43.416484+0000) 2020-02-05T20:48:43.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:43.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:43.447202+0000) 2020-02-05T20:48:45.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:45.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:45.092144+0000) 2020-02-05T20:48:45.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:45.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:45.791389+0000) 2020-02-05T20:48:45.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:45.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:45.828065+0000) 2020-02-05T20:48:46.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:46.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:46.688309+0000) 2020-02-05T20:48:46.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:46.931+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:46.931521+0000) 2020-02-05T20:48:46.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:46.932+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:46.932797+0000) 2020-02-05T20:48:47.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:47.106+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.107721+0000) 2020-02-05T20:48:47.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:47.107+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.108580+0000) 2020-02-05T20:48:47.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:47.109+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.109770+0000) 2020-02-05T20:48:47.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:47.109+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.110519+0000) 2020-02-05T20:48:47.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:47.109+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.110256+0000) 2020-02-05T20:48:47.114 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:47.110+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.111196+0000) 2020-02-05T20:48:47.115 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:47.111+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.112258+0000) 2020-02-05T20:48:47.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:47.113+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.113451+0000) 2020-02-05T20:48:47.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:47.111+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.111863+0000) 2020-02-05T20:48:47.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:47.112+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.112928+0000) 2020-02-05T20:48:47.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:47.112+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.112985+0000) 2020-02-05T20:48:47.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:47.113+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.114207+0000) 2020-02-05T20:48:47.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:47.113+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.114276+0000) 2020-02-05T20:48:47.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:47.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:47.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:47.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:47.119698+0000) 2020-02-05T20:48:49.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:49.326+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:49.326404+0000) 2020-02-05T20:48:49.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:49.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:49.935779+0000) 2020-02-05T20:48:50.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:50.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:50.185210+0000) 2020-02-05T20:48:50.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:50.196+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:50.196256+0000) 2020-02-05T20:48:52.125 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:52.123+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.123923+0000) 2020-02-05T20:48:52.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:52.123+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.124782+0000) 2020-02-05T20:48:52.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:52.123+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.125395+0000) 2020-02-05T20:48:52.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:52.125+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.125821+0000) 2020-02-05T20:48:52.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:52.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.126349+0000) 2020-02-05T20:48:52.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:52.126+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.126742+0000) 2020-02-05T20:48:52.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:52.127+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.127583+0000) 2020-02-05T20:48:52.129 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:52.125+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.127263+0000) 2020-02-05T20:48:52.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:52.126+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.128049+0000) 2020-02-05T20:48:52.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:52.128+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.128582+0000) 2020-02-05T20:48:52.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:52.128+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.128655+0000) 2020-02-05T20:48:52.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:52.126+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.128341+0000) 2020-02-05T20:48:52.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:52.127+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.128963+0000) 2020-02-05T20:48:52.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:52.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:52.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:52.134+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.134950+0000) 2020-02-05T20:48:52.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:52.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.174247+0000) 2020-02-05T20:48:52.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:52.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:52.289524+0000) 2020-02-05T20:48:53.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:53.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:53.416711+0000) 2020-02-05T20:48:53.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:53.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:53.447526+0000) 2020-02-05T20:48:55.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:55.099+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:55.100416+0000) 2020-02-05T20:48:55.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:55.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:55.791620+0000) 2020-02-05T20:48:55.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:55.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:55.828367+0000) 2020-02-05T20:48:56.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:56.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:56.688531+0000) 2020-02-05T20:48:56.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:56.930+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:56.931779+0000) 2020-02-05T20:48:56.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:56.932+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:56.933084+0000) 2020-02-05T20:48:57.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:48:57.136+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.138037+0000) 2020-02-05T20:48:57.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:48:57.137+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.138994+0000) 2020-02-05T20:48:57.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:57.139+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.140421+0000) 2020-02-05T20:48:57.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:48:57.139+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.140785+0000) 2020-02-05T20:48:57.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:48:57.140+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.141370+0000) 2020-02-05T20:48:57.143 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:48:57.140+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.141803+0000) 2020-02-05T20:48:57.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:57.141+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.142646+0000) 2020-02-05T20:48:57.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:48:57.141+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.143173+0000) 2020-02-05T20:48:57.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:57.143+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.144400+0000) 2020-02-05T20:48:57.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:48:57.143+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.144493+0000) 2020-02-05T20:48:57.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:48:57.142+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.144344+0000) 2020-02-05T20:48:57.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:48:57.143+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.144597+0000) 2020-02-05T20:48:57.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:48:57.143+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.144723+0000) 2020-02-05T20:48:57.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:48:57.145+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:48:57.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:57.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:57.149438+0000) 2020-02-05T20:48:59.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:48:59.325+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:59.326655+0000) 2020-02-05T20:48:59.491 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:48:59.498 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:48:59.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:48:59.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:48:59.936000+0000) 2020-02-05T20:49:00.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:00.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:00.185483+0000) 2020-02-05T20:49:00.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:00.195+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:00.196493+0000) 2020-02-05T20:49:02.154 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:02.151+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.152844+0000) 2020-02-05T20:49:02.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:02.153+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.154068+0000) 2020-02-05T20:49:02.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:02.154+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.155104+0000) 2020-02-05T20:49:02.157 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:02.154+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.155562+0000) 2020-02-05T20:49:02.157 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:02.154+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.156320+0000) 2020-02-05T20:49:02.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:02.155+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.156664+0000) 2020-02-05T20:49:02.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:02.155+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.157059+0000) 2020-02-05T20:49:02.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:02.156+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.158084+0000) 2020-02-05T20:49:02.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:02.157+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.158527+0000) 2020-02-05T20:49:02.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:02.157+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.158647+0000) 2020-02-05T20:49:02.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:02.157+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.158809+0000) 2020-02-05T20:49:02.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:02.158+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.159514+0000) 2020-02-05T20:49:02.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:02.158+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.159758+0000) 2020-02-05T20:49:02.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:02.160+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67537 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:02.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:02.161+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.162988+0000) 2020-02-05T20:49:02.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:02.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.174472+0000) 2020-02-05T20:49:02.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:02.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:02.289819+0000) 2020-02-05T20:49:03.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:03.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:03.416969+0000) 2020-02-05T20:49:03.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:03.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:03.447760+0000) 2020-02-05T20:49:05.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:05.107+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:05.109028+0000) 2020-02-05T20:49:05.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:05.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:05.791846+0000) 2020-02-05T20:49:05.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:05.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:05.828597+0000) 2020-02-05T20:49:06.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:06.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:06.688776+0000) 2020-02-05T20:49:06.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:06.931+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:06.932037+0000) 2020-02-05T20:49:06.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:06.931+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:06.933285+0000) 2020-02-05T20:49:07.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:07.167+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.168739+0000) 2020-02-05T20:49:07.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:07.168+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.169469+0000) 2020-02-05T20:49:07.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:07.168+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.169896+0000) 2020-02-05T20:49:07.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:07.168+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.170195+0000) 2020-02-05T20:49:07.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:07.169+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.170437+0000) 2020-02-05T20:49:07.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:07.169+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.171042+0000) 2020-02-05T20:49:07.172 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:07.170+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.171510+0000) 2020-02-05T20:49:07.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:07.170+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.171921+0000) 2020-02-05T20:49:07.174 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:07.171+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.172902+0000) 2020-02-05T20:49:07.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:07.172+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.174038+0000) 2020-02-05T20:49:07.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:07.173+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.174227+0000) 2020-02-05T20:49:07.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:07.172+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.174338+0000) 2020-02-05T20:49:07.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:07.173+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.174441+0000) 2020-02-05T20:49:07.178 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:07.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:07.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:07.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:07.179445+0000) 2020-02-05T20:49:09.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:09.326+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:09.326892+0000) 2020-02-05T20:49:09.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:09.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:09.936263+0000) 2020-02-05T20:49:10.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:10.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:10.185825+0000) 2020-02-05T20:49:10.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:10.195+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:10.196727+0000) 2020-02-05T20:49:12.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:12.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.174728+0000) 2020-02-05T20:49:12.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:12.181+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.183039+0000) 2020-02-05T20:49:12.185 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:12.182+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.183757+0000) 2020-02-05T20:49:12.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:12.182+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.183868+0000) 2020-02-05T20:49:12.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:12.183+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.184724+0000) 2020-02-05T20:49:12.186 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:12.183+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.184891+0000) 2020-02-05T20:49:12.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:12.184+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.185491+0000) 2020-02-05T20:49:12.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:12.183+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.185370+0000) 2020-02-05T20:49:12.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:12.184+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.186345+0000) 2020-02-05T20:49:12.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:12.185+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.186757+0000) 2020-02-05T20:49:12.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:12.186+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.187175+0000) 2020-02-05T20:49:12.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:12.186+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.187367+0000) 2020-02-05T20:49:12.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:12.187+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.188974+0000) 2020-02-05T20:49:12.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:12.187+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.189026+0000) 2020-02-05T20:49:12.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:12.190+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:12.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:12.192+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.194041+0000) 2020-02-05T20:49:12.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:12.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:12.290016+0000) 2020-02-05T20:49:13.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:13.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:13.417206+0000) 2020-02-05T20:49:13.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:13.446+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:13.448035+0000) 2020-02-05T20:49:15.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:15.117+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:15.118139+0000) 2020-02-05T20:49:15.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:15.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:15.792112+0000) 2020-02-05T20:49:15.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:15.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:15.828848+0000) 2020-02-05T20:49:16.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:16.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:16.689010+0000) 2020-02-05T20:49:16.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:16.931+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:16.932292+0000) 2020-02-05T20:49:16.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:16.932+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:16.933533+0000) 2020-02-05T20:49:17.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:17.195+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.197193+0000) 2020-02-05T20:49:17.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:17.197+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.198254+0000) 2020-02-05T20:49:17.200 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:17.198+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.199150+0000) 2020-02-05T20:49:17.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:17.198+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.199597+0000) 2020-02-05T20:49:17.201 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:17.198+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.200046+0000) 2020-02-05T20:49:17.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:17.199+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.200517+0000) 2020-02-05T20:49:17.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:17.199+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.200907+0000) 2020-02-05T20:49:17.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:17.200+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.201317+0000) 2020-02-05T20:49:17.203 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:17.201+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.202483+0000) 2020-02-05T20:49:17.204 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:17.200+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.202310+0000) 2020-02-05T20:49:17.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:17.201+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.202980+0000) 2020-02-05T20:49:17.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:17.201+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.203299+0000) 2020-02-05T20:49:17.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:17.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:17.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:17.207+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.208605+0000) 2020-02-05T20:49:17.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:17.208+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:17.209756+0000) 2020-02-05T20:49:19.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:19.326+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:19.327185+0000) 2020-02-05T20:49:19.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:19.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:19.936492+0000) 2020-02-05T20:49:20.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:20.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:20.186076+0000) 2020-02-05T20:49:20.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:20.196+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:20.196944+0000) 2020-02-05T20:49:22.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:22.174+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.174981+0000) 2020-02-05T20:49:22.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:22.209+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.210736+0000) 2020-02-05T20:49:22.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:22.210+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.212302+0000) 2020-02-05T20:49:22.216 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:22.211+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.212927+0000) 2020-02-05T20:49:22.216 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:22.212+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.213488+0000) 2020-02-05T20:49:22.216 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:22.213+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.214719+0000) 2020-02-05T20:49:22.217 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:22.214+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.215694+0000) 2020-02-05T20:49:22.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:22.214+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.215198+0000) 2020-02-05T20:49:22.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:22.214+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.215486+0000) 2020-02-05T20:49:22.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:22.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.216403+0000) 2020-02-05T20:49:22.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:22.215+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.217382+0000) 2020-02-05T20:49:22.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:22.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.217027+0000) 2020-02-05T20:49:22.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:22.216+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.217913+0000) 2020-02-05T20:49:22.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:22.216+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.218044+0000) 2020-02-05T20:49:22.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:22.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:22.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:22.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.223392+0000) 2020-02-05T20:49:22.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:22.288+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:22.290281+0000) 2020-02-05T20:49:23.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:23.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:23.417497+0000) 2020-02-05T20:49:23.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:23.447+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:23.448321+0000) 2020-02-05T20:49:25.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:25.125+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:25.125690+0000) 2020-02-05T20:49:25.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:25.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:25.792351+0000) 2020-02-05T20:49:25.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:25.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:25.829062+0000) 2020-02-05T20:49:26.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:26.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:26.689287+0000) 2020-02-05T20:49:26.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:26.931+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:26.932546+0000) 2020-02-05T20:49:26.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:26.932+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:26.933740+0000) 2020-02-05T20:49:27.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:27.225+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.226494+0000) 2020-02-05T20:49:27.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:27.226+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.227646+0000) 2020-02-05T20:49:27.230 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:27.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.227967+0000) 2020-02-05T20:49:27.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:27.227+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.228302+0000) 2020-02-05T20:49:27.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:27.227+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.228885+0000) 2020-02-05T20:49:27.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:27.228+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.229439+0000) 2020-02-05T20:49:27.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:27.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.230036+0000) 2020-02-05T20:49:27.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:27.228+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.229991+0000) 2020-02-05T20:49:27.232 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:27.229+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.230956+0000) 2020-02-05T20:49:27.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:27.230+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.231601+0000) 2020-02-05T20:49:27.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:27.229+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.230918+0000) 2020-02-05T20:49:27.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:27.230+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.231721+0000) 2020-02-05T20:49:27.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:27.230+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.232188+0000) 2020-02-05T20:49:27.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:27.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:27.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:27.236+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:27.237214+0000) 2020-02-05T20:49:29.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:29.327+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:29.327409+0000) 2020-02-05T20:49:29.655 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:49:29.661 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:49:29.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:29.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:29.936707+0000) 2020-02-05T20:49:30.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:30.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:30.186329+0000) 2020-02-05T20:49:30.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:30.196+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:30.197176+0000) 2020-02-05T20:49:32.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:32.174+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.175245+0000) 2020-02-05T20:49:32.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:32.239+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.240680+0000) 2020-02-05T20:49:32.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:32.240+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.241912+0000) 2020-02-05T20:49:32.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:32.240+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.242334+0000) 2020-02-05T20:49:32.244 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:32.241+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.243043+0000) 2020-02-05T20:49:32.244 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:32.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.243197+0000) 2020-02-05T20:49:32.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:32.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.243601+0000) 2020-02-05T20:49:32.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:32.242+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.244309+0000) 2020-02-05T20:49:32.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:32.243+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.244778+0000) 2020-02-05T20:49:32.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:32.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.245005+0000) 2020-02-05T20:49:32.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:32.244+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.245465+0000) 2020-02-05T20:49:32.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:32.244+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.245569+0000) 2020-02-05T20:49:32.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:32.244+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.245948+0000) 2020-02-05T20:49:32.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:32.245+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.246433+0000) 2020-02-05T20:49:32.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:32.248+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.249427+0000) 2020-02-05T20:49:32.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:32.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:32.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:32.289+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:32.290518+0000) 2020-02-05T20:49:33.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:33.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:33.417774+0000) 2020-02-05T20:49:33.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:33.448+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:33.448557+0000) 2020-02-05T20:49:35.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:35.133+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:35.134023+0000) 2020-02-05T20:49:35.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:35.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:35.792575+0000) 2020-02-05T20:49:35.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:35.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:35.829323+0000) 2020-02-05T20:49:36.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:36.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:36.689508+0000) 2020-02-05T20:49:36.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:36.932+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:36.932772+0000) 2020-02-05T20:49:36.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:36.933+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:36.934021+0000) 2020-02-05T20:49:37.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:37.252+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.253505+0000) 2020-02-05T20:49:37.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:37.255+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.256610+0000) 2020-02-05T20:49:37.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:37.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.257192+0000) 2020-02-05T20:49:37.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:37.257+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.258541+0000) 2020-02-05T20:49:37.260 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:37.257+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.258731+0000) 2020-02-05T20:49:37.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:37.258+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.259347+0000) 2020-02-05T20:49:37.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:37.257+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.258973+0000) 2020-02-05T20:49:37.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:37.258+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.260375+0000) 2020-02-05T20:49:37.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:37.259+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.261143+0000) 2020-02-05T20:49:37.265 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:37.260+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.261487+0000) 2020-02-05T20:49:37.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:37.260+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.261526+0000) 2020-02-05T20:49:37.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:37.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.261045+0000) 2020-02-05T20:49:37.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:37.261+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.262840+0000) 2020-02-05T20:49:37.266 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:37.260+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:37.261301+0000) 2020-02-05T20:49:37.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:37.263+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:39.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:39.327+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:39.327658+0000) 2020-02-05T20:49:39.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:39.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:39.936947+0000) 2020-02-05T20:49:40.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:40.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:40.186666+0000) 2020-02-05T20:49:40.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:40.197+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:40.197406+0000) 2020-02-05T20:49:42.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:42.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.175498+0000) 2020-02-05T20:49:42.272 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:42.269+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.270661+0000) 2020-02-05T20:49:42.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:42.272+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.272930+0000) 2020-02-05T20:49:42.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:42.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.273436+0000) 2020-02-05T20:49:42.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:42.272+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.273317+0000) 2020-02-05T20:49:42.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:42.273+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.273424+0000) 2020-02-05T20:49:42.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:42.273+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.274417+0000) 2020-02-05T20:49:42.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:42.273+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.274484+0000) 2020-02-05T20:49:42.277 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:42.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.274489+0000) 2020-02-05T20:49:42.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:42.273+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.274962+0000) 2020-02-05T20:49:42.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:42.273+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.274321+0000) 2020-02-05T20:49:42.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:42.274+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.275123+0000) 2020-02-05T20:49:42.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:42.274+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.275346+0000) 2020-02-05T20:49:42.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:42.275+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.276371+0000) 2020-02-05T20:49:42.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:42.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:42.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:42.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.281393+0000) 2020-02-05T20:49:42.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:42.290+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:42.290755+0000) 2020-02-05T20:49:43.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:43.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:43.418011+0000) 2020-02-05T20:49:43.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:43.448+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:43.448789+0000) 2020-02-05T20:49:45.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:45.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:45.142149+0000) 2020-02-05T20:49:45.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:45.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:45.792813+0000) 2020-02-05T20:49:45.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:45.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:45.829563+0000) 2020-02-05T20:49:46.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:46.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:46.689740+0000) 2020-02-05T20:49:46.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:46.932+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:46.933003+0000) 2020-02-05T20:49:46.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:46.933+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:46.934268+0000) 2020-02-05T20:49:47.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:47.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.283694+0000) 2020-02-05T20:49:47.285 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:47.283+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.284411+0000) 2020-02-05T20:49:47.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:47.283+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.284937+0000) 2020-02-05T20:49:47.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:47.284+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.285362+0000) 2020-02-05T20:49:47.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:47.284+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.285718+0000) 2020-02-05T20:49:47.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:47.285+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.286071+0000) 2020-02-05T20:49:47.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:47.285+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.286730+0000) 2020-02-05T20:49:47.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:47.286+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.286954+0000) 2020-02-05T20:49:47.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:47.287+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.287776+0000) 2020-02-05T20:49:47.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:47.287+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.288318+0000) 2020-02-05T20:49:47.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:47.288+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.288687+0000) 2020-02-05T20:49:47.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:47.289+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.289701+0000) 2020-02-05T20:49:47.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:47.288+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.289724+0000) 2020-02-05T20:49:47.292 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:47.290+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:47.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:47.294+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:47.295342+0000) 2020-02-05T20:49:49.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:49.326+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:49.327920+0000) 2020-02-05T20:49:49.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:49.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:49.937184+0000) 2020-02-05T20:49:50.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:50.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:50.186952+0000) 2020-02-05T20:49:50.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:50.196+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:50.197666+0000) 2020-02-05T20:49:52.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:52.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.175728+0000) 2020-02-05T20:49:52.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:52.290+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.290947+0000) 2020-02-05T20:49:52.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:52.297+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.298268+0000) 2020-02-05T20:49:52.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:52.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.299586+0000) 2020-02-05T20:49:52.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:52.298+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.300057+0000) 2020-02-05T20:49:52.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:52.298+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.300021+0000) 2020-02-05T20:49:52.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:52.299+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.300354+0000) 2020-02-05T20:49:52.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:52.300+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.300620+0000) 2020-02-05T20:49:52.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:52.300+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.301124+0000) 2020-02-05T20:49:52.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:52.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.301604+0000) 2020-02-05T20:49:52.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:52.300+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.301572+0000) 2020-02-05T20:49:52.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:52.301+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.301851+0000) 2020-02-05T20:49:52.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:52.301+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.302009+0000) 2020-02-05T20:49:52.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:52.301+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.302145+0000) 2020-02-05T20:49:52.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:52.301+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.302794+0000) 2020-02-05T20:49:52.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:52.302+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:52.303255+0000) 2020-02-05T20:49:52.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:52.304+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:53.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:53.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:53.418315+0000) 2020-02-05T20:49:53.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:53.447+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:53.449021+0000) 2020-02-05T20:49:55.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:55.149+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:55.150591+0000) 2020-02-05T20:49:55.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:55.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:55.793047+0000) 2020-02-05T20:49:55.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:55.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:55.829826+0000) 2020-02-05T20:49:56.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:56.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:56.689975+0000) 2020-02-05T20:49:56.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:56.932+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:56.933263+0000) 2020-02-05T20:49:56.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:56.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:56.934511+0000) 2020-02-05T20:49:57.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:49:57.311+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.311778+0000) 2020-02-05T20:49:57.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:49:57.313+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.313486+0000) 2020-02-05T20:49:57.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:49:57.313+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.313505+0000) 2020-02-05T20:49:57.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:49:57.313+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.314340+0000) 2020-02-05T20:49:57.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:49:57.314+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.314822+0000) 2020-02-05T20:49:57.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:49:57.315+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.315750+0000) 2020-02-05T20:49:57.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:57.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.311819+0000) 2020-02-05T20:49:57.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:57.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.312399+0000) 2020-02-05T20:49:57.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:49:57.312+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.313175+0000) 2020-02-05T20:49:57.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:49:57.313+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.314072+0000) 2020-02-05T20:49:57.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:49:57.313+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.314408+0000) 2020-02-05T20:49:57.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:57.314+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.316012+0000) 2020-02-05T20:49:57.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:49:57.315+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.316530+0000) 2020-02-05T20:49:57.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:49:57.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 67985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:49:57.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:57.318+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:57.319660+0000) 2020-02-05T20:49:59.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:49:59.327+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:59.328182+0000) 2020-02-05T20:49:59.810 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:49:59.817 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:49:59.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:49:59.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:49:59.937357+0000) 2020-02-05T20:50:00.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:00.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:00.187186+0000) 2020-02-05T20:50:00.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:00.196+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:00.197961+0000) 2020-02-05T20:50:02.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:02.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.175981+0000) 2020-02-05T20:50:02.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:02.290+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.291218+0000) 2020-02-05T20:50:02.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:02.325+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.326767+0000) 2020-02-05T20:50:02.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:02.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.327826+0000) 2020-02-05T20:50:02.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:02.327+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.327733+0000) 2020-02-05T20:50:02.330 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:02.327+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.328099+0000) 2020-02-05T20:50:02.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:02.327+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.328395+0000) 2020-02-05T20:50:02.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:02.328+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.329430+0000) 2020-02-05T20:50:02.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:02.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.329910+0000) 2020-02-05T20:50:02.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:02.329+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.330141+0000) 2020-02-05T20:50:02.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:02.330+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.330716+0000) 2020-02-05T20:50:02.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:02.330+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.331201+0000) 2020-02-05T20:50:02.332 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:02.330+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.331421+0000) 2020-02-05T20:50:02.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:02.331+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.331657+0000) 2020-02-05T20:50:02.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:02.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.332201+0000) 2020-02-05T20:50:02.335 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:02.333+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:02.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:02.336+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:02.337704+0000) 2020-02-05T20:50:03.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:03.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:03.418568+0000) 2020-02-05T20:50:03.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:03.448+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:03.449244+0000) 2020-02-05T20:50:05.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:05.158+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:05.158865+0000) 2020-02-05T20:50:05.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:05.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:05.793352+0000) 2020-02-05T20:50:05.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:05.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:05.830064+0000) 2020-02-05T20:50:06.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:06.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:06.690205+0000) 2020-02-05T20:50:06.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:06.933+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:06.933498+0000) 2020-02-05T20:50:06.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:06.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:06.934767+0000) 2020-02-05T20:50:07.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:07.340+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.341213+0000) 2020-02-05T20:50:07.343 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:07.340+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.341815+0000) 2020-02-05T20:50:07.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:07.342+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.342833+0000) 2020-02-05T20:50:07.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:07.341+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.343021+0000) 2020-02-05T20:50:07.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:07.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.343468+0000) 2020-02-05T20:50:07.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:07.342+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.343428+0000) 2020-02-05T20:50:07.346 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:07.343+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.344134+0000) 2020-02-05T20:50:07.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:07.344+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.344655+0000) 2020-02-05T20:50:07.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:07.342+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.343978+0000) 2020-02-05T20:50:07.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:07.343+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.344213+0000) 2020-02-05T20:50:07.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:07.344+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.345251+0000) 2020-02-05T20:50:07.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:07.345+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.346252+0000) 2020-02-05T20:50:07.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:07.344+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.346044+0000) 2020-02-05T20:50:07.349 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:07.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:07.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:07.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:07.350920+0000) 2020-02-05T20:50:09.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:09.327+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:09.328441+0000) 2020-02-05T20:50:09.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:09.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:09.937641+0000) 2020-02-05T20:50:10.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:10.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:10.187486+0000) 2020-02-05T20:50:10.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:10.197+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:10.198189+0000) 2020-02-05T20:50:12.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:12.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.176234+0000) 2020-02-05T20:50:12.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:12.291+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.291484+0000) 2020-02-05T20:50:12.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:12.352+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.353258+0000) 2020-02-05T20:50:12.355 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:12.353+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.354278+0000) 2020-02-05T20:50:12.356 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:12.354+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.355111+0000) 2020-02-05T20:50:12.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:12.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.356229+0000) 2020-02-05T20:50:12.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:12.356+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.356643+0000) 2020-02-05T20:50:12.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:12.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.357568+0000) 2020-02-05T20:50:12.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:12.357+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.357968+0000) 2020-02-05T20:50:12.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:12.357+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.358536+0000) 2020-02-05T20:50:12.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:12.358+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.359260+0000) 2020-02-05T20:50:12.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:12.358+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.359579+0000) 2020-02-05T20:50:12.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:12.359+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.360007+0000) 2020-02-05T20:50:12.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:12.360+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.360506+0000) 2020-02-05T20:50:12.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:12.360+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.360736+0000) 2020-02-05T20:50:12.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:12.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:12.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:12.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:12.367000+0000) 2020-02-05T20:50:13.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:13.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:13.418855+0000) 2020-02-05T20:50:13.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:13.448+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:13.449487+0000) 2020-02-05T20:50:15.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:15.165+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:15.166282+0000) 2020-02-05T20:50:15.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:15.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:15.793585+0000) 2020-02-05T20:50:15.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:15.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:15.830358+0000) 2020-02-05T20:50:16.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:16.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:16.690397+0000) 2020-02-05T20:50:16.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:16.933+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:16.933720+0000) 2020-02-05T20:50:16.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:16.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:16.934977+0000) 2020-02-05T20:50:17.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:17.368+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.368617+0000) 2020-02-05T20:50:17.370 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:17.368+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.369585+0000) 2020-02-05T20:50:17.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:17.368+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.369978+0000) 2020-02-05T20:50:17.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:17.369+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.370504+0000) 2020-02-05T20:50:17.372 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:17.369+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.371038+0000) 2020-02-05T20:50:17.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:17.371+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.372039+0000) 2020-02-05T20:50:17.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:17.371+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.372675+0000) 2020-02-05T20:50:17.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:17.372+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.372861+0000) 2020-02-05T20:50:17.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:17.372+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.373241+0000) 2020-02-05T20:50:17.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:17.373+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.374129+0000) 2020-02-05T20:50:17.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:17.374+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.374711+0000) 2020-02-05T20:50:17.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:17.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.373754+0000) 2020-02-05T20:50:17.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:17.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.374822+0000) 2020-02-05T20:50:17.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:17.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:17.376954+0000) 2020-02-05T20:50:17.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:17.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:19.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:19.328+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:19.328671+0000) 2020-02-05T20:50:19.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:19.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:19.937901+0000) 2020-02-05T20:50:20.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:20.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:20.187679+0000) 2020-02-05T20:50:20.751 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:20.197+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:20.198435+0000) 2020-02-05T20:50:22.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:22.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.176499+0000) 2020-02-05T20:50:22.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:22.290+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.291674+0000) 2020-02-05T20:50:22.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:22.383+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.384534+0000) 2020-02-05T20:50:22.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:22.384+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.385459+0000) 2020-02-05T20:50:22.387 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:22.384+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.385950+0000) 2020-02-05T20:50:22.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:22.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.386385+0000) 2020-02-05T20:50:22.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:22.385+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.386586+0000) 2020-02-05T20:50:22.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:22.386+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.387163+0000) 2020-02-05T20:50:22.389 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:22.386+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.387284+0000) 2020-02-05T20:50:22.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:22.386+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.387792+0000) 2020-02-05T20:50:22.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:22.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.388522+0000) 2020-02-05T20:50:22.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:22.387+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.389108+0000) 2020-02-05T20:50:22.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:22.387+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.388890+0000) 2020-02-05T20:50:22.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:22.388+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.389545+0000) 2020-02-05T20:50:22.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:22.388+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.390226+0000) 2020-02-05T20:50:22.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:22.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68190 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:22.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:22.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:22.393233+0000) 2020-02-05T20:50:23.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:23.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:23.419087+0000) 2020-02-05T20:50:23.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:23.448+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:23.449663+0000) 2020-02-05T20:50:25.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:25.174+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:25.174578+0000) 2020-02-05T20:50:25.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:25.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:25.793811+0000) 2020-02-05T20:50:25.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:25.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:25.830587+0000) 2020-02-05T20:50:26.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:26.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:26.690628+0000) 2020-02-05T20:50:26.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:26.933+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:26.933968+0000) 2020-02-05T20:50:26.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:26.933+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:26.935199+0000) 2020-02-05T20:50:27.399 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:27.397+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.398313+0000) 2020-02-05T20:50:27.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:27.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.399917+0000) 2020-02-05T20:50:27.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:27.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.400271+0000) 2020-02-05T20:50:27.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:27.399+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.400946+0000) 2020-02-05T20:50:27.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:27.400+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.401112+0000) 2020-02-05T20:50:27.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:27.400+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.401360+0000) 2020-02-05T20:50:27.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:27.400+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.401771+0000) 2020-02-05T20:50:27.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:27.400+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.401694+0000) 2020-02-05T20:50:27.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:27.401+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.402901+0000) 2020-02-05T20:50:27.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:27.400+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.402123+0000) 2020-02-05T20:50:27.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:27.401+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.402836+0000) 2020-02-05T20:50:27.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:27.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.405036+0000) 2020-02-05T20:50:27.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:27.403+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.405302+0000) 2020-02-05T20:50:27.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:27.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:27.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:27.407+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:27.408095+0000) 2020-02-05T20:50:29.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:29.328+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:29.328895+0000) 2020-02-05T20:50:29.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:29.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:29.938131+0000) 2020-02-05T20:50:29.956 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:50:29.962 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:50:30.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:30.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:30.187940+0000) 2020-02-05T20:50:30.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:30.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:30.198703+0000) 2020-02-05T20:50:32.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:32.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.176734+0000) 2020-02-05T20:50:32.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:32.291+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.291908+0000) 2020-02-05T20:50:32.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:32.411+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.412447+0000) 2020-02-05T20:50:32.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:32.412+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.413134+0000) 2020-02-05T20:50:32.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:32.413+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.414530+0000) 2020-02-05T20:50:32.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:32.413+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.415099+0000) 2020-02-05T20:50:32.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:32.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.416359+0000) 2020-02-05T20:50:32.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:32.414+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.415927+0000) 2020-02-05T20:50:32.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:32.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.416870+0000) 2020-02-05T20:50:32.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:32.416+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.417594+0000) 2020-02-05T20:50:32.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:32.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.417765+0000) 2020-02-05T20:50:32.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:32.417+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.418657+0000) 2020-02-05T20:50:32.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:32.417+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.419048+0000) 2020-02-05T20:50:32.421 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:32.418+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.420048+0000) 2020-02-05T20:50:32.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:32.418+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.420020+0000) 2020-02-05T20:50:32.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:32.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:32.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:32.424+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:32.425461+0000) 2020-02-05T20:50:33.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:33.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:33.419328+0000) 2020-02-05T20:50:33.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:33.449+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:33.449879+0000) 2020-02-05T20:50:35.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:35.182+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:35.182762+0000) 2020-02-05T20:50:35.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:35.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:35.794050+0000) 2020-02-05T20:50:35.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:35.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:35.830817+0000) 2020-02-05T20:50:36.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:36.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:36.690862+0000) 2020-02-05T20:50:36.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:36.933+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:36.934219+0000) 2020-02-05T20:50:36.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:36.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:36.935424+0000) 2020-02-05T20:50:37.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:37.426+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.427602+0000) 2020-02-05T20:50:37.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:37.426+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.427718+0000) 2020-02-05T20:50:37.430 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:37.427+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.428384+0000) 2020-02-05T20:50:37.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:37.428+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.429718+0000) 2020-02-05T20:50:37.431 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:37.429+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.430275+0000) 2020-02-05T20:50:37.431 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:37.429+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.430599+0000) 2020-02-05T20:50:37.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:37.429+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.430921+0000) 2020-02-05T20:50:37.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:37.430+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.431643+0000) 2020-02-05T20:50:37.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:37.429+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.431284+0000) 2020-02-05T20:50:37.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:37.431+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.432165+0000) 2020-02-05T20:50:37.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:37.431+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.432562+0000) 2020-02-05T20:50:37.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:37.432+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.433129+0000) 2020-02-05T20:50:37.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:37.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.434727+0000) 2020-02-05T20:50:37.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:37.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:37.435932+0000) 2020-02-05T20:50:37.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:37.435+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:39.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:39.329+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:39.329133+0000) 2020-02-05T20:50:39.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:39.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:39.938363+0000) 2020-02-05T20:50:40.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:40.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:40.188184+0000) 2020-02-05T20:50:40.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:40.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:40.198970+0000) 2020-02-05T20:50:42.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:42.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.176993+0000) 2020-02-05T20:50:42.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:42.291+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.292141+0000) 2020-02-05T20:50:42.443 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:42.440+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.442045+0000) 2020-02-05T20:50:42.443 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:42.441+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.442468+0000) 2020-02-05T20:50:42.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:42.441+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.442977+0000) 2020-02-05T20:50:42.444 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:42.442+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.443477+0000) 2020-02-05T20:50:42.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:42.442+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.443893+0000) 2020-02-05T20:50:42.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:42.443+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.444513+0000) 2020-02-05T20:50:42.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:42.442+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.444308+0000) 2020-02-05T20:50:42.446 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:42.443+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.445069+0000) 2020-02-05T20:50:42.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:42.444+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.445522+0000) 2020-02-05T20:50:42.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:42.444+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.445967+0000) 2020-02-05T20:50:42.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:42.444+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.446158+0000) 2020-02-05T20:50:42.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:42.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.447195+0000) 2020-02-05T20:50:42.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:42.445+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.447239+0000) 2020-02-05T20:50:42.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:42.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:42.451582+0000) 2020-02-05T20:50:42.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:42.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:43.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:43.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:43.419575+0000) 2020-02-05T20:50:43.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:43.449+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:43.450140+0000) 2020-02-05T20:50:45.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:45.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:45.190621+0000) 2020-02-05T20:50:45.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:45.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:45.794313+0000) 2020-02-05T20:50:45.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:45.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:45.831073+0000) 2020-02-05T20:50:46.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:46.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:46.691128+0000) 2020-02-05T20:50:46.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:46.933+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:46.934470+0000) 2020-02-05T20:50:46.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:46.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:46.935637+0000) 2020-02-05T20:50:47.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:47.456+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.457943+0000) 2020-02-05T20:50:47.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:47.457+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.458497+0000) 2020-02-05T20:50:47.460 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:47.458+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.459121+0000) 2020-02-05T20:50:47.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:47.458+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.459539+0000) 2020-02-05T20:50:47.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:47.459+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.460665+0000) 2020-02-05T20:50:47.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:47.459+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.460780+0000) 2020-02-05T20:50:47.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:47.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.461312+0000) 2020-02-05T20:50:47.463 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:47.460+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.461930+0000) 2020-02-05T20:50:47.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:47.461+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.462573+0000) 2020-02-05T20:50:47.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:47.461+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.463015+0000) 2020-02-05T20:50:47.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:47.462+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.463573+0000) 2020-02-05T20:50:47.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:47.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.463533+0000) 2020-02-05T20:50:47.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:47.462+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.463888+0000) 2020-02-05T20:50:47.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:47.465+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:47.466432+0000) 2020-02-05T20:50:47.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:47.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:49.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:49.328+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:49.329332+0000) 2020-02-05T20:50:49.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:49.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:49.938613+0000) 2020-02-05T20:50:50.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:50.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:50.188416+0000) 2020-02-05T20:50:50.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:50.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:50.199187+0000) 2020-02-05T20:50:52.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:52.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.177235+0000) 2020-02-05T20:50:52.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:52.290+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.292367+0000) 2020-02-05T20:50:52.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:52.471+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.472128+0000) 2020-02-05T20:50:52.474 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:52.471+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.472699+0000) 2020-02-05T20:50:52.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:52.472+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.473427+0000) 2020-02-05T20:50:52.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:52.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.474212+0000) 2020-02-05T20:50:52.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:52.472+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.474016+0000) 2020-02-05T20:50:52.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:52.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.474627+0000) 2020-02-05T20:50:52.476 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:52.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.475225+0000) 2020-02-05T20:50:52.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:52.474+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.475728+0000) 2020-02-05T20:50:52.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:52.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.477166+0000) 2020-02-05T20:50:52.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:52.475+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.477100+0000) 2020-02-05T20:50:52.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:52.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.477836+0000) 2020-02-05T20:50:52.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:52.476+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.478122+0000) 2020-02-05T20:50:52.480 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:52.476+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.478363+0000) 2020-02-05T20:50:52.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:52.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:52.480053+0000) 2020-02-05T20:50:52.483 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:52.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68439 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:53.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:53.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:53.419807+0000) 2020-02-05T20:50:53.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:53.449+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:53.450414+0000) 2020-02-05T20:50:55.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:55.198+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:55.199573+0000) 2020-02-05T20:50:55.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:55.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:55.794543+0000) 2020-02-05T20:50:55.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:55.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:55.831381+0000) 2020-02-05T20:50:56.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:56.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:56.691359+0000) 2020-02-05T20:50:56.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:56.934+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:56.934727+0000) 2020-02-05T20:50:56.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:56.935+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:56.935884+0000) 2020-02-05T20:50:57.489 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:57.486+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.487759+0000) 2020-02-05T20:50:57.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:50:57.486+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.487482+0000) 2020-02-05T20:50:57.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:57.487+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.488617+0000) 2020-02-05T20:50:57.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:50:57.487+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.489023+0000) 2020-02-05T20:50:57.491 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:50:57.487+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.489278+0000) 2020-02-05T20:50:57.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:57.488+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.489559+0000) 2020-02-05T20:50:57.491 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:50:57.488+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.489821+0000) 2020-02-05T20:50:57.492 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:50:57.489+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.490623+0000) 2020-02-05T20:50:57.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:50:57.489+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.490900+0000) 2020-02-05T20:50:57.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:50:57.490+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.491826+0000) 2020-02-05T20:50:57.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:50:57.491+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.492354+0000) 2020-02-05T20:50:57.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:50:57.491+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.493125+0000) 2020-02-05T20:50:57.494 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:50:57.492+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.493620+0000) 2020-02-05T20:50:57.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:50:57.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:50:57.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:57.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:57.498532+0000) 2020-02-05T20:50:59.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:50:59.328+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:59.329572+0000) 2020-02-05T20:50:59.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:50:59.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:50:59.938874+0000) 2020-02-05T20:51:00.110 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:51:00.117 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:51:00.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:00.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:00.188730+0000) 2020-02-05T20:51:00.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:00.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:00.199412+0000) 2020-02-05T20:51:02.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:02.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.177491+0000) 2020-02-05T20:51:02.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:02.292+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.292638+0000) 2020-02-05T20:51:02.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:02.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.501675+0000) 2020-02-05T20:51:02.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:02.501+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.502294+0000) 2020-02-05T20:51:02.504 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:02.501+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.502909+0000) 2020-02-05T20:51:02.505 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:02.502+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.504007+0000) 2020-02-05T20:51:02.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:02.503+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.504335+0000) 2020-02-05T20:51:02.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:02.503+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.504657+0000) 2020-02-05T20:51:02.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:02.504+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.505634+0000) 2020-02-05T20:51:02.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:02.504+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.505643+0000) 2020-02-05T20:51:02.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:02.504+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.506018+0000) 2020-02-05T20:51:02.508 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:02.505+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.507369+0000) 2020-02-05T20:51:02.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:02.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.507380+0000) 2020-02-05T20:51:02.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:02.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.507735+0000) 2020-02-05T20:51:02.509 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:02.506+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.507925+0000) 2020-02-05T20:51:02.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:02.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:02.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:02.513+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:02.514173+0000) 2020-02-05T20:51:03.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:03.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:03.420019+0000) 2020-02-05T20:51:03.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:03.449+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:03.450623+0000) 2020-02-05T20:51:05.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:05.206+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:05.208044+0000) 2020-02-05T20:51:05.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:05.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:05.794761+0000) 2020-02-05T20:51:05.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:05.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:05.831617+0000) 2020-02-05T20:51:06.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:06.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:06.691587+0000) 2020-02-05T20:51:06.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:06.934+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:06.934957+0000) 2020-02-05T20:51:06.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:06.934+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:06.936115+0000) 2020-02-05T20:51:07.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:07.516+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.517605+0000) 2020-02-05T20:51:07.519 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:07.516+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.517739+0000) 2020-02-05T20:51:07.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:07.517+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.518468+0000) 2020-02-05T20:51:07.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:07.517+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.518730+0000) 2020-02-05T20:51:07.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:07.518+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.519106+0000) 2020-02-05T20:51:07.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:07.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.519181+0000) 2020-02-05T20:51:07.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:07.517+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.519249+0000) 2020-02-05T20:51:07.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:07.518+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.519530+0000) 2020-02-05T20:51:07.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:07.518+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.520041+0000) 2020-02-05T20:51:07.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:07.519+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.520293+0000) 2020-02-05T20:51:07.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:07.519+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.520732+0000) 2020-02-05T20:51:07.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:07.519+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.520687+0000) 2020-02-05T20:51:07.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:07.519+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.521018+0000) 2020-02-05T20:51:07.527 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:07.524+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:07.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:07.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:07.526675+0000) 2020-02-05T20:51:09.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:09.329+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:09.329811+0000) 2020-02-05T20:51:09.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:09.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:09.939117+0000) 2020-02-05T20:51:10.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:10.188+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:10.189006+0000) 2020-02-05T20:51:10.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:10.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:10.199677+0000) 2020-02-05T20:51:12.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:12.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.177689+0000) 2020-02-05T20:51:12.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:12.292+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.292894+0000) 2020-02-05T20:51:12.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:12.530+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.531302+0000) 2020-02-05T20:51:12.533 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:12.531+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.532373+0000) 2020-02-05T20:51:12.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:12.532+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.532649+0000) 2020-02-05T20:51:12.534 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:12.532+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.533181+0000) 2020-02-05T20:51:12.534 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:12.532+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.533611+0000) 2020-02-05T20:51:12.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:12.532+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.533837+0000) 2020-02-05T20:51:12.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:12.533+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.533869+0000) 2020-02-05T20:51:12.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:12.533+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.534367+0000) 2020-02-05T20:51:12.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:12.534+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.534688+0000) 2020-02-05T20:51:12.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:12.534+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.534620+0000) 2020-02-05T20:51:12.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:12.534+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.535754+0000) 2020-02-05T20:51:12.537 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:12.534+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.535328+0000) 2020-02-05T20:51:12.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:12.534+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.535269+0000) 2020-02-05T20:51:12.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:12.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68606 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:12.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:12.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:12.542404+0000) 2020-02-05T20:51:13.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:13.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:13.420233+0000) 2020-02-05T20:51:13.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:13.450+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:13.450832+0000) 2020-02-05T20:51:15.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:15.215+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:15.216196+0000) 2020-02-05T20:51:15.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:15.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:15.794988+0000) 2020-02-05T20:51:15.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:15.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:15.831866+0000) 2020-02-05T20:51:16.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:16.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:16.691819+0000) 2020-02-05T20:51:16.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:16.934+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:16.935217+0000) 2020-02-05T20:51:16.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:16.935+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:16.936338+0000) 2020-02-05T20:51:17.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:17.545+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.545562+0000) 2020-02-05T20:51:17.547 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:17.545+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.546404+0000) 2020-02-05T20:51:17.548 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:17.545+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.546928+0000) 2020-02-05T20:51:17.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:17.546+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.547393+0000) 2020-02-05T20:51:17.549 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:17.546+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.547749+0000) 2020-02-05T20:51:17.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:17.547+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.548306+0000) 2020-02-05T20:51:17.551 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:17.549+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.550231+0000) 2020-02-05T20:51:17.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:17.549+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.550363+0000) 2020-02-05T20:51:17.552 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:17.550+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.550569+0000) 2020-02-05T20:51:17.553 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:17.550+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.551409+0000) 2020-02-05T20:51:17.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:17.551+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.551547+0000) 2020-02-05T20:51:17.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:17.551+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.551721+0000) 2020-02-05T20:51:17.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:17.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:17.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:17.553+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.554574+0000) 2020-02-05T20:51:17.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:17.553+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:17.554827+0000) 2020-02-05T20:51:19.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:19.329+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:19.330053+0000) 2020-02-05T20:51:19.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:19.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:19.939367+0000) 2020-02-05T20:51:20.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:20.188+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:20.189257+0000) 2020-02-05T20:51:20.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:20.198+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:20.199920+0000) 2020-02-05T20:51:22.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:22.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.177950+0000) 2020-02-05T20:51:22.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:22.292+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.293145+0000) 2020-02-05T20:51:22.560 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:22.558+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.559391+0000) 2020-02-05T20:51:22.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:22.560+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.560750+0000) 2020-02-05T20:51:22.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:22.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.561587+0000) 2020-02-05T20:51:22.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:22.560+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.561998+0000) 2020-02-05T20:51:22.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:22.561+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.562415+0000) 2020-02-05T20:51:22.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:22.561+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.562793+0000) 2020-02-05T20:51:22.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:22.563+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.563459+0000) 2020-02-05T20:51:22.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:22.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.564777+0000) 2020-02-05T20:51:22.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:22.563+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.565031+0000) 2020-02-05T20:51:22.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:22.565+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.565539+0000) 2020-02-05T20:51:22.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:22.565+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.565830+0000) 2020-02-05T20:51:22.567 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:22.565+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.565870+0000) 2020-02-05T20:51:22.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:22.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.566226+0000) 2020-02-05T20:51:22.571 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:22.569+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68690 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:22.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:22.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:22.574311+0000) 2020-02-05T20:51:23.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:23.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:23.420421+0000) 2020-02-05T20:51:23.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:23.449+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:23.451012+0000) 2020-02-05T20:51:25.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:25.224+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:25.224480+0000) 2020-02-05T20:51:25.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:25.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:25.795271+0000) 2020-02-05T20:51:25.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:25.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:25.832152+0000) 2020-02-05T20:51:26.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:26.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:26.692105+0000) 2020-02-05T20:51:26.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:26.935+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:26.935477+0000) 2020-02-05T20:51:26.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:26.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:26.936538+0000) 2020-02-05T20:51:27.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:27.575+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.576614+0000) 2020-02-05T20:51:27.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:27.576+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.577064+0000) 2020-02-05T20:51:27.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:27.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.577737+0000) 2020-02-05T20:51:27.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:27.576+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.577918+0000) 2020-02-05T20:51:27.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:27.577+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.578314+0000) 2020-02-05T20:51:27.584 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:27.578+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.578947+0000) 2020-02-05T20:51:27.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:27.579+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.580051+0000) 2020-02-05T20:51:27.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:27.580+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.581146+0000) 2020-02-05T20:51:27.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:27.581+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.581638+0000) 2020-02-05T20:51:27.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:27.581+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.581937+0000) 2020-02-05T20:51:27.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:27.578+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.579251+0000) 2020-02-05T20:51:27.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:27.578+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.579365+0000) 2020-02-05T20:51:27.594 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:27.579+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.580993+0000) 2020-02-05T20:51:27.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:27.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:27.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:27.586+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:27.587393+0000) 2020-02-05T20:51:29.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:29.330+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:29.330346+0000) 2020-02-05T20:51:29.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:29.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:29.939609+0000) 2020-02-05T20:51:30.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:30.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:30.189513+0000) 2020-02-05T20:51:30.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:30.199+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:30.200164+0000) 2020-02-05T20:51:30.249 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:51:30.255 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:51:32.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:32.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.178230+0000) 2020-02-05T20:51:32.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:32.292+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.293387+0000) 2020-02-05T20:51:32.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:32.590+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.590705+0000) 2020-02-05T20:51:32.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:32.591+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.591751+0000) 2020-02-05T20:51:32.594 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:32.593+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.593133+0000) 2020-02-05T20:51:32.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:32.593+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.593899+0000) 2020-02-05T20:51:32.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:32.593+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.593944+0000) 2020-02-05T20:51:32.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:32.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.595110+0000) 2020-02-05T20:51:32.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:32.594+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.595202+0000) 2020-02-05T20:51:32.597 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:32.595+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.595647+0000) 2020-02-05T20:51:32.597 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:32.595+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.595997+0000) 2020-02-05T20:51:32.597 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:32.596+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.596604+0000) 2020-02-05T20:51:32.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:32.596+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.596774+0000) 2020-02-05T20:51:32.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:32.596+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.596864+0000) 2020-02-05T20:51:32.601 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:32.599+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:32.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:32.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.600130+0000) 2020-02-05T20:51:32.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:32.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:32.600431+0000) 2020-02-05T20:51:33.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:33.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:33.420700+0000) 2020-02-05T20:51:33.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:33.451+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:33.451241+0000) 2020-02-05T20:51:35.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:35.233+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:35.233170+0000) 2020-02-05T20:51:35.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:35.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:35.795497+0000) 2020-02-05T20:51:35.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:35.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:35.832383+0000) 2020-02-05T20:51:36.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:36.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:36.692367+0000) 2020-02-05T20:51:36.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:36.935+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:36.935709+0000) 2020-02-05T20:51:36.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:36.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:36.936765+0000) 2020-02-05T20:51:37.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:37.604+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.605426+0000) 2020-02-05T20:51:37.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:37.605+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.606103+0000) 2020-02-05T20:51:37.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:37.605+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.607052+0000) 2020-02-05T20:51:37.609 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:37.607+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.608302+0000) 2020-02-05T20:51:37.610 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:37.608+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.608706+0000) 2020-02-05T20:51:37.610 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:37.608+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.609492+0000) 2020-02-05T20:51:37.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:37.608+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.609170+0000) 2020-02-05T20:51:37.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:37.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.609966+0000) 2020-02-05T20:51:37.612 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:37.610+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.610697+0000) 2020-02-05T20:51:37.612 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:37.610+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.611291+0000) 2020-02-05T20:51:37.613 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:37.610+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.611486+0000) 2020-02-05T20:51:37.613 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:37.611+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.611930+0000) 2020-02-05T20:51:37.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:37.611+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.612377+0000) 2020-02-05T20:51:37.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:37.614+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:37.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:37.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:37.618827+0000) 2020-02-05T20:51:39.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:39.329+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:39.330569+0000) 2020-02-05T20:51:39.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:39.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:39.939864+0000) 2020-02-05T20:51:40.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:40.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:40.189783+0000) 2020-02-05T20:51:40.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:40.199+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:40.200387+0000) 2020-02-05T20:51:42.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:42.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.178513+0000) 2020-02-05T20:51:42.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:42.293+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.293680+0000) 2020-02-05T20:51:42.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:42.620+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.620643+0000) 2020-02-05T20:51:42.622 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:42.619+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.621016+0000) 2020-02-05T20:51:42.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:42.620+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.621440+0000) 2020-02-05T20:51:42.623 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:42.621+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.621921+0000) 2020-02-05T20:51:42.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:42.621+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.622634+0000) 2020-02-05T20:51:42.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:42.621+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.622062+0000) 2020-02-05T20:51:42.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:42.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.623444+0000) 2020-02-05T20:51:42.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:42.622+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.623898+0000) 2020-02-05T20:51:42.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:42.623+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.624070+0000) 2020-02-05T20:51:42.625 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:42.623+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.624761+0000) 2020-02-05T20:51:42.626 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:42.624+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.624914+0000) 2020-02-05T20:51:42.626 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:42.624+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.625349+0000) 2020-02-05T20:51:42.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:42.625+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.626406+0000) 2020-02-05T20:51:42.630 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:42.628+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68858 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:42.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:42.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:42.629760+0000) 2020-02-05T20:51:43.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:43.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:43.420916+0000) 2020-02-05T20:51:43.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:43.450+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:43.451442+0000) 2020-02-05T20:51:45.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:45.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:45.245137+0000) 2020-02-05T20:51:45.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:45.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:45.795741+0000) 2020-02-05T20:51:45.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:45.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:45.832616+0000) 2020-02-05T20:51:46.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:46.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:46.692600+0000) 2020-02-05T20:51:46.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:46.935+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:46.935986+0000) 2020-02-05T20:51:46.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:46.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:46.937002+0000) 2020-02-05T20:51:47.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:47.633+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.635256+0000) 2020-02-05T20:51:47.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:47.635+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.636394+0000) 2020-02-05T20:51:47.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:47.635+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.636759+0000) 2020-02-05T20:51:47.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:47.636+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.637479+0000) 2020-02-05T20:51:47.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:47.636+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.637663+0000) 2020-02-05T20:51:47.639 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:47.636+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.637870+0000) 2020-02-05T20:51:47.640 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:47.636+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.637768+0000) 2020-02-05T20:51:47.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:47.636+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.637852+0000) 2020-02-05T20:51:47.641 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:47.636+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.638388+0000) 2020-02-05T20:51:47.641 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:47.637+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.639101+0000) 2020-02-05T20:51:47.641 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:47.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.638585+0000) 2020-02-05T20:51:47.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:47.637+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.638703+0000) 2020-02-05T20:51:47.642 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:47.638+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.639156+0000) 2020-02-05T20:51:47.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:47.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:47.642874+0000) 2020-02-05T20:51:47.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:47.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:49.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:49.330+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:49.330839+0000) 2020-02-05T20:51:49.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:49.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:49.940130+0000) 2020-02-05T20:51:50.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:50.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:50.190045+0000) 2020-02-05T20:51:50.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:50.199+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:50.200644+0000) 2020-02-05T20:51:52.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:52.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.178785+0000) 2020-02-05T20:51:52.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:52.292+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.293935+0000) 2020-02-05T20:51:52.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:52.648+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.649478+0000) 2020-02-05T20:51:52.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:52.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.649671+0000) 2020-02-05T20:51:52.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:52.649+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.650648+0000) 2020-02-05T20:51:52.652 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:52.649+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.651312+0000) 2020-02-05T20:51:52.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:52.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.651815+0000) 2020-02-05T20:51:52.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:52.651+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.652328+0000) 2020-02-05T20:51:52.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:52.651+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.652631+0000) 2020-02-05T20:51:52.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:52.651+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.652700+0000) 2020-02-05T20:51:52.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:52.651+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.653099+0000) 2020-02-05T20:51:52.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:52.652+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.653534+0000) 2020-02-05T20:51:52.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:52.652+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.653378+0000) 2020-02-05T20:51:52.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:52.652+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.653977+0000) 2020-02-05T20:51:52.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:52.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.654498+0000) 2020-02-05T20:51:52.657 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:52.655+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:52.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:52.658+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:52.659781+0000) 2020-02-05T20:51:53.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:53.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:53.421177+0000) 2020-02-05T20:51:53.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:53.450+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:53.451670+0000) 2020-02-05T20:51:55.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:55.252+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:55.252971+0000) 2020-02-05T20:51:55.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:55.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:55.796022+0000) 2020-02-05T20:51:55.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:55.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:55.832872+0000) 2020-02-05T20:51:56.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:56.692+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:56.692844+0000) 2020-02-05T20:51:56.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:56.935+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:56.936221+0000) 2020-02-05T20:51:56.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:56.935+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:56.937234+0000) 2020-02-05T20:51:57.663 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:51:57.661+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.662382+0000) 2020-02-05T20:51:57.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:51:57.662+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.663537+0000) 2020-02-05T20:51:57.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:51:57.662+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.664084+0000) 2020-02-05T20:51:57.666 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:57.663+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.664291+0000) 2020-02-05T20:51:57.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:51:57.664+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.665482+0000) 2020-02-05T20:51:57.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:57.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.665873+0000) 2020-02-05T20:51:57.668 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:51:57.665+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.666376+0000) 2020-02-05T20:51:57.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:57.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.667569+0000) 2020-02-05T20:51:57.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:51:57.665+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.667067+0000) 2020-02-05T20:51:57.669 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:51:57.666+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.668046+0000) 2020-02-05T20:51:57.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:51:57.666+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.668152+0000) 2020-02-05T20:51:57.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:51:57.667+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.668770+0000) 2020-02-05T20:51:57.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:51:57.667+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.669066+0000) 2020-02-05T20:51:57.672 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:51:57.670+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 68987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:51:57.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:57.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:57.672360+0000) 2020-02-05T20:51:59.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:51:59.330+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:59.331172+0000) 2020-02-05T20:51:59.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:51:59.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:51:59.940361+0000) 2020-02-05T20:52:00.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:00.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:00.190288+0000) 2020-02-05T20:52:00.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:00.199+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:00.200932+0000) 2020-02-05T20:52:00.438 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:52:00.444 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:52:02.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:02.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.179028+0000) 2020-02-05T20:52:02.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:02.293+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.294195+0000) 2020-02-05T20:52:02.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:02.676+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.678017+0000) 2020-02-05T20:52:02.680 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:02.677+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.678683+0000) 2020-02-05T20:52:02.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:02.678+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.679580+0000) 2020-02-05T20:52:02.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:02.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.679825+0000) 2020-02-05T20:52:02.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:02.679+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.680365+0000) 2020-02-05T20:52:02.682 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:02.679+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.680516+0000) 2020-02-05T20:52:02.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:02.679+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.680546+0000) 2020-02-05T20:52:02.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:02.678+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.680398+0000) 2020-02-05T20:52:02.683 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:02.680+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.681686+0000) 2020-02-05T20:52:02.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:02.680+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.681241+0000) 2020-02-05T20:52:02.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:02.681+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.682747+0000) 2020-02-05T20:52:02.684 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:02.681+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.682890+0000) 2020-02-05T20:52:02.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:02.681+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.683056+0000) 2020-02-05T20:52:02.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:02.685+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:02.686221+0000) 2020-02-05T20:52:02.688 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:02.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:03.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:03.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:03.421464+0000) 2020-02-05T20:52:03.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:03.450+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:03.451866+0000) 2020-02-05T20:52:05.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:05.260+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:05.260792+0000) 2020-02-05T20:52:05.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:05.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:05.796285+0000) 2020-02-05T20:52:05.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:05.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:05.833128+0000) 2020-02-05T20:52:06.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:06.692+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:06.693132+0000) 2020-02-05T20:52:06.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:06.935+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:06.936471+0000) 2020-02-05T20:52:06.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:06.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:06.937486+0000) 2020-02-05T20:52:07.694 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:07.691+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.693322+0000) 2020-02-05T20:52:07.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:07.693+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.694201+0000) 2020-02-05T20:52:07.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:07.692+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.694089+0000) 2020-02-05T20:52:07.696 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:07.694+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.695366+0000) 2020-02-05T20:52:07.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:07.694+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.695864+0000) 2020-02-05T20:52:07.697 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:07.695+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.696268+0000) 2020-02-05T20:52:07.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:07.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.696627+0000) 2020-02-05T20:52:07.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:07.695+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.697003+0000) 2020-02-05T20:52:07.699 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:07.696+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.697683+0000) 2020-02-05T20:52:07.699 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:07.695+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.697266+0000) 2020-02-05T20:52:07.699 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:07.696+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.698086+0000) 2020-02-05T20:52:07.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:07.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.698675+0000) 2020-02-05T20:52:07.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:07.697+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.698698+0000) 2020-02-05T20:52:07.703 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:07.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69071 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:07.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:07.702+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:07.703241+0000) 2020-02-05T20:52:09.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:09.330+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:09.331429+0000) 2020-02-05T20:52:09.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:09.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:09.940588+0000) 2020-02-05T20:52:10.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:10.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:10.190537+0000) 2020-02-05T20:52:10.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:10.200+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:10.201155+0000) 2020-02-05T20:52:12.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:12.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.179273+0000) 2020-02-05T20:52:12.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:12.293+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.294472+0000) 2020-02-05T20:52:12.709 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:12.706+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.707530+0000) 2020-02-05T20:52:12.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:12.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.708525+0000) 2020-02-05T20:52:12.710 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:12.707+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.708846+0000) 2020-02-05T20:52:12.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:12.708+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.709106+0000) 2020-02-05T20:52:12.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:12.708+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.709511+0000) 2020-02-05T20:52:12.711 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:12.708+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.709723+0000) 2020-02-05T20:52:12.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:12.709+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.710126+0000) 2020-02-05T20:52:12.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:12.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.710579+0000) 2020-02-05T20:52:12.712 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:12.709+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.711039+0000) 2020-02-05T20:52:12.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:12.709+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.711086+0000) 2020-02-05T20:52:12.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:12.711+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.712385+0000) 2020-02-05T20:52:12.714 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:12.711+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.712522+0000) 2020-02-05T20:52:12.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:12.712+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.713731+0000) 2020-02-05T20:52:12.717 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:12.714+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:12.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:12.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:12.718154+0000) 2020-02-05T20:52:13.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:13.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:13.421704+0000) 2020-02-05T20:52:13.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:13.451+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:13.452116+0000) 2020-02-05T20:52:15.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:15.268+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:15.269057+0000) 2020-02-05T20:52:15.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:15.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:15.796515+0000) 2020-02-05T20:52:15.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:15.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:15.833320+0000) 2020-02-05T20:52:16.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:16.693+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:16.693361+0000) 2020-02-05T20:52:16.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:16.936+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:16.936740+0000) 2020-02-05T20:52:16.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:16.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:16.937672+0000) 2020-02-05T20:52:17.722 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:17.719+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.721209+0000) 2020-02-05T20:52:17.723 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:17.720+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.722021+0000) 2020-02-05T20:52:17.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:17.721+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.722707+0000) 2020-02-05T20:52:17.724 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:17.722+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.723310+0000) 2020-02-05T20:52:17.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:17.722+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.723743+0000) 2020-02-05T20:52:17.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:17.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.724711+0000) 2020-02-05T20:52:17.726 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:17.723+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.724768+0000) 2020-02-05T20:52:17.726 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:17.724+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.725249+0000) 2020-02-05T20:52:17.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:17.724+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.725633+0000) 2020-02-05T20:52:17.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:17.724+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.725541+0000) 2020-02-05T20:52:17.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:17.725+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.726277+0000) 2020-02-05T20:52:17.728 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:17.724+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.726195+0000) 2020-02-05T20:52:17.728 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:17.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.726415+0000) 2020-02-05T20:52:17.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:17.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:17.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:17.730+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:17.731518+0000) 2020-02-05T20:52:19.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:19.331+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:19.331690+0000) 2020-02-05T20:52:19.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:19.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:19.940818+0000) 2020-02-05T20:52:20.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:20.190+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:20.190771+0000) 2020-02-05T20:52:20.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:20.200+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:20.201367+0000) 2020-02-05T20:52:22.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:22.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.179515+0000) 2020-02-05T20:52:22.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:22.293+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.294662+0000) 2020-02-05T20:52:22.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:22.733+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.735057+0000) 2020-02-05T20:52:22.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:22.734+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.735371+0000) 2020-02-05T20:52:22.737 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:22.735+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.736402+0000) 2020-02-05T20:52:22.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:22.735+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.736887+0000) 2020-02-05T20:52:22.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:22.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.737358+0000) 2020-02-05T20:52:22.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:22.736+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.737501+0000) 2020-02-05T20:52:22.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:22.736+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.737943+0000) 2020-02-05T20:52:22.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:22.737+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.738874+0000) 2020-02-05T20:52:22.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:22.738+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.739714+0000) 2020-02-05T20:52:22.741 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:22.738+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.740007+0000) 2020-02-05T20:52:22.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:22.739+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.740680+0000) 2020-02-05T20:52:22.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:22.740+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.741835+0000) 2020-02-05T20:52:22.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:22.740+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.742083+0000) 2020-02-05T20:52:22.745 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:22.742+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69196 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:22.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:22.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:22.744599+0000) 2020-02-05T20:52:23.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:23.421+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:23.421944+0000) 2020-02-05T20:52:23.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:23.452+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:23.452398+0000) 2020-02-05T20:52:25.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:25.280+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:25.280458+0000) 2020-02-05T20:52:25.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:25.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:25.796744+0000) 2020-02-05T20:52:25.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:25.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:25.833549+0000) 2020-02-05T20:52:26.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:26.693+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:26.693594+0000) 2020-02-05T20:52:26.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:26.936+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:26.936999+0000) 2020-02-05T20:52:26.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:26.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:26.937877+0000) 2020-02-05T20:52:27.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:27.750+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.750671+0000) 2020-02-05T20:52:27.752 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:27.748+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.750208+0000) 2020-02-05T20:52:27.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:27.751+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.751437+0000) 2020-02-05T20:52:27.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:27.751+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.751885+0000) 2020-02-05T20:52:27.754 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:27.752+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.752910+0000) 2020-02-05T20:52:27.755 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:27.753+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.753417+0000) 2020-02-05T20:52:27.757 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:27.752+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.753418+0000) 2020-02-05T20:52:27.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:27.752+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.754349+0000) 2020-02-05T20:52:27.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:27.753+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.754748+0000) 2020-02-05T20:52:27.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:27.753+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.755098+0000) 2020-02-05T20:52:27.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:27.753+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.755328+0000) 2020-02-05T20:52:27.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:27.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.754163+0000) 2020-02-05T20:52:27.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:27.754+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.755039+0000) 2020-02-05T20:52:27.760 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:27.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69241 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:27.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:27.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:27.758108+0000) 2020-02-05T20:52:29.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:29.331+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:29.331925+0000) 2020-02-05T20:52:29.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:29.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:29.941063+0000) 2020-02-05T20:52:30.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:30.190+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:30.191008+0000) 2020-02-05T20:52:30.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:30.201+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:30.201555+0000) 2020-02-05T20:52:30.639 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:52:30.645 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:52:32.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:32.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.179766+0000) 2020-02-05T20:52:32.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:32.294+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.294917+0000) 2020-02-05T20:52:32.766 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:32.763+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.764663+0000) 2020-02-05T20:52:32.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:32.764+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.765228+0000) 2020-02-05T20:52:32.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:32.764+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.765530+0000) 2020-02-05T20:52:32.768 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:32.765+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.766818+0000) 2020-02-05T20:52:32.768 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:32.765+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.766575+0000) 2020-02-05T20:52:32.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:32.765+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.766521+0000) 2020-02-05T20:52:32.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:32.766+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.767384+0000) 2020-02-05T20:52:32.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:32.766+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.767715+0000) 2020-02-05T20:52:32.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:32.767+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.768343+0000) 2020-02-05T20:52:32.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:32.767+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.768638+0000) 2020-02-05T20:52:32.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:32.768+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.769587+0000) 2020-02-05T20:52:32.771 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:32.767+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.769115+0000) 2020-02-05T20:52:32.771 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:32.768+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.769671+0000) 2020-02-05T20:52:32.774 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:32.771+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69281 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:32.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:32.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:32.775149+0000) 2020-02-05T20:52:33.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:33.421+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:33.422181+0000) 2020-02-05T20:52:33.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:33.451+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:33.452626+0000) 2020-02-05T20:52:35.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:35.287+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:35.288578+0000) 2020-02-05T20:52:35.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:35.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:35.796984+0000) 2020-02-05T20:52:35.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:35.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:35.833777+0000) 2020-02-05T20:52:36.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:36.692+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:36.693838+0000) 2020-02-05T20:52:36.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:36.936+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:36.937258+0000) 2020-02-05T20:52:36.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:36.936+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:36.938067+0000) 2020-02-05T20:52:37.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:37.777+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.778386+0000) 2020-02-05T20:52:37.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:37.779+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.780186+0000) 2020-02-05T20:52:37.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:37.780+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.781392+0000) 2020-02-05T20:52:37.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:37.780+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.781792+0000) 2020-02-05T20:52:37.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:37.782+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.783024+0000) 2020-02-05T20:52:37.784 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:37.782+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.782846+0000) 2020-02-05T20:52:37.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:37.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.782264+0000) 2020-02-05T20:52:37.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:37.782+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.783813+0000) 2020-02-05T20:52:37.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:37.783+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.784324+0000) 2020-02-05T20:52:37.786 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:37.783+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.783802+0000) 2020-02-05T20:52:37.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:37.784+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.785308+0000) 2020-02-05T20:52:37.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:37.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.786046+0000) 2020-02-05T20:52:37.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:37.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.786108+0000) 2020-02-05T20:52:37.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:37.787+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:37.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:37.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:37.792247+0000) 2020-02-05T20:52:39.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:39.331+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:39.332188+0000) 2020-02-05T20:52:39.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:39.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:39.941331+0000) 2020-02-05T20:52:40.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:40.190+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:40.191322+0000) 2020-02-05T20:52:40.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:40.200+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:40.201827+0000) 2020-02-05T20:52:42.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:42.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.180029+0000) 2020-02-05T20:52:42.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:42.294+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.295152+0000) 2020-02-05T20:52:42.795 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:42.793+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.794216+0000) 2020-02-05T20:52:42.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:42.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.794663+0000) 2020-02-05T20:52:42.796 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:42.794+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.795117+0000) 2020-02-05T20:52:42.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:42.794+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.795876+0000) 2020-02-05T20:52:42.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:42.795+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.796647+0000) 2020-02-05T20:52:42.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:42.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.796807+0000) 2020-02-05T20:52:42.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:42.796+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.797309+0000) 2020-02-05T20:52:42.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:42.797+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.797546+0000) 2020-02-05T20:52:42.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:42.797+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.798325+0000) 2020-02-05T20:52:42.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:42.799+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.799640+0000) 2020-02-05T20:52:42.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:42.799+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.800308+0000) 2020-02-05T20:52:42.802 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:42.799+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.800403+0000) 2020-02-05T20:52:42.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:42.800+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.800622+0000) 2020-02-05T20:52:42.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:42.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:42.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:42.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:42.806073+0000) 2020-02-05T20:52:43.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:43.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:43.422410+0000) 2020-02-05T20:52:43.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:43.451+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:43.452832+0000) 2020-02-05T20:52:45.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:45.295+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:45.296994+0000) 2020-02-05T20:52:45.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:45.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:45.797217+0000) 2020-02-05T20:52:45.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:45.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:45.833996+0000) 2020-02-05T20:52:46.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:46.693+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:46.694112+0000) 2020-02-05T20:52:46.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:46.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:46.937490+0000) 2020-02-05T20:52:46.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:46.937+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:46.938262+0000) 2020-02-05T20:52:47.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:47.807+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.808248+0000) 2020-02-05T20:52:47.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:47.808+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.809804+0000) 2020-02-05T20:52:47.812 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:47.810+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.810737+0000) 2020-02-05T20:52:47.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:47.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.811734+0000) 2020-02-05T20:52:47.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:47.811+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.812253+0000) 2020-02-05T20:52:47.813 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:47.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.812390+0000) 2020-02-05T20:52:47.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:47.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.812780+0000) 2020-02-05T20:52:47.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:47.812+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.813772+0000) 2020-02-05T20:52:47.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:47.812+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.813355+0000) 2020-02-05T20:52:47.816 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:47.813+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.814254+0000) 2020-02-05T20:52:47.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:47.814+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.815389+0000) 2020-02-05T20:52:47.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:47.815+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.815451+0000) 2020-02-05T20:52:47.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:47.815+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.815743+0000) 2020-02-05T20:52:47.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:47.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69410 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:47.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:47.818+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:47.819929+0000) 2020-02-05T20:52:49.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:49.331+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:49.332436+0000) 2020-02-05T20:52:49.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:49.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:49.941591+0000) 2020-02-05T20:52:50.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:50.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:50.191707+0000) 2020-02-05T20:52:50.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:50.201+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:50.202015+0000) 2020-02-05T20:52:52.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:52.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.180259+0000) 2020-02-05T20:52:52.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:52.295+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.295405+0000) 2020-02-05T20:52:52.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:52.823+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.824899+0000) 2020-02-05T20:52:52.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:52.824+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.825510+0000) 2020-02-05T20:52:52.827 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:52.825+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.825967+0000) 2020-02-05T20:52:52.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:52.825+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.826329+0000) 2020-02-05T20:52:52.829 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:52.825+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.826532+0000) 2020-02-05T20:52:52.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:52.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.827138+0000) 2020-02-05T20:52:52.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:52.826+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.827538+0000) 2020-02-05T20:52:52.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:52.826+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.828056+0000) 2020-02-05T20:52:52.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:52.826+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.826800+0000) 2020-02-05T20:52:52.831 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:52.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.828973+0000) 2020-02-05T20:52:52.831 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:52.829+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.829768+0000) 2020-02-05T20:52:52.831 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:52.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.829730+0000) 2020-02-05T20:52:52.832 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:52.829+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.829720+0000) 2020-02-05T20:52:52.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:52.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:52.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:52.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:52.836180+0000) 2020-02-05T20:52:53.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:53.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:53.422693+0000) 2020-02-05T20:52:53.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:53.452+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:53.453064+0000) 2020-02-05T20:52:55.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:55.304+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:55.305254+0000) 2020-02-05T20:52:55.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:55.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:55.797448+0000) 2020-02-05T20:52:55.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:55.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:55.834267+0000) 2020-02-05T20:52:56.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:56.693+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:56.694367+0000) 2020-02-05T20:52:56.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:56.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:56.937738+0000) 2020-02-05T20:52:56.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:56.938+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:56.938475+0000) 2020-02-05T20:52:57.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:52:57.837+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.838124+0000) 2020-02-05T20:52:57.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:52:57.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.838671+0000) 2020-02-05T20:52:57.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:57.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.840736+0000) 2020-02-05T20:52:57.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:52:57.840+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.841230+0000) 2020-02-05T20:52:57.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:52:57.841+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.841637+0000) 2020-02-05T20:52:57.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:52:57.841+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.842103+0000) 2020-02-05T20:52:57.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:52:57.842+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.842860+0000) 2020-02-05T20:52:57.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:57.842+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.843714+0000) 2020-02-05T20:52:57.845 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:52:57.843+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.844146+0000) 2020-02-05T20:52:57.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:52:57.843+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.844118+0000) 2020-02-05T20:52:57.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:52:57.844+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.844492+0000) 2020-02-05T20:52:57.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:52:57.843+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.844323+0000) 2020-02-05T20:52:57.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:57.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.847537+0000) 2020-02-05T20:52:57.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:57.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:57.847737+0000) 2020-02-05T20:52:57.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:52:57.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:52:59.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:52:59.332+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:59.332701+0000) 2020-02-05T20:52:59.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:52:59.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:52:59.941815+0000) 2020-02-05T20:53:00.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:00.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:00.191944+0000) 2020-02-05T20:53:00.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:00.201+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:00.202303+0000) 2020-02-05T20:53:00.783 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:53:00.791 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:53:02.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:02.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.180515+0000) 2020-02-05T20:53:02.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:02.295+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.295680+0000) 2020-02-05T20:53:02.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:02.852+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.853359+0000) 2020-02-05T20:53:02.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:02.852+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.854012+0000) 2020-02-05T20:53:02.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:02.853+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.854541+0000) 2020-02-05T20:53:02.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:02.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.855353+0000) 2020-02-05T20:53:02.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:02.854+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.855885+0000) 2020-02-05T20:53:02.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:02.855+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.856477+0000) 2020-02-05T20:53:02.858 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:02.855+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.856155+0000) 2020-02-05T20:53:02.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:02.856+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.857064+0000) 2020-02-05T20:53:02.859 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:02.857+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.858055+0000) 2020-02-05T20:53:02.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:02.858+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.859180+0000) 2020-02-05T20:53:02.861 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:02.858+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.859173+0000) 2020-02-05T20:53:02.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:02.859+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.859490+0000) 2020-02-05T20:53:02.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:02.859+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.859955+0000) 2020-02-05T20:53:02.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:02.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:02.863032+0000) 2020-02-05T20:53:02.865 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:02.863+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:03.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:03.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:03.422901+0000) 2020-02-05T20:53:03.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:03.452+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:03.453391+0000) 2020-02-05T20:53:05.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:05.316+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:05.317833+0000) 2020-02-05T20:53:05.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:05.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:05.797721+0000) 2020-02-05T20:53:05.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:05.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:05.834557+0000) 2020-02-05T20:53:06.696 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:06.694+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:06.694599+0000) 2020-02-05T20:53:06.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:06.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:06.937987+0000) 2020-02-05T20:53:06.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:06.938+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:06.938652+0000) 2020-02-05T20:53:07.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:07.869+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.870542+0000) 2020-02-05T20:53:07.873 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:07.871+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.871801+0000) 2020-02-05T20:53:07.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:07.872+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.872729+0000) 2020-02-05T20:53:07.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:07.872+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.873248+0000) 2020-02-05T20:53:07.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:07.872+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.873816+0000) 2020-02-05T20:53:07.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:07.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.874223+0000) 2020-02-05T20:53:07.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:07.873+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.874368+0000) 2020-02-05T20:53:07.876 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:07.874+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.875086+0000) 2020-02-05T20:53:07.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:07.874+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.875591+0000) 2020-02-05T20:53:07.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:07.874+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.875643+0000) 2020-02-05T20:53:07.877 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:07.875+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.875916+0000) 2020-02-05T20:53:07.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:07.875+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.876240+0000) 2020-02-05T20:53:07.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:07.876+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.877090+0000) 2020-02-05T20:53:07.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:07.877+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:07.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:07.881+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:07.882249+0000) 2020-02-05T20:53:09.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:09.332+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:09.332939+0000) 2020-02-05T20:53:09.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:09.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:09.942069+0000) 2020-02-05T20:53:10.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:10.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:10.192197+0000) 2020-02-05T20:53:10.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:10.202+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:10.202611+0000) 2020-02-05T20:53:12.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:12.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.180716+0000) 2020-02-05T20:53:12.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:12.295+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.295943+0000) 2020-02-05T20:53:12.884 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:12.882+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.883269+0000) 2020-02-05T20:53:12.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:12.882+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.883965+0000) 2020-02-05T20:53:12.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:12.883+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.884617+0000) 2020-02-05T20:53:12.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:12.885+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.886308+0000) 2020-02-05T20:53:12.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:12.885+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.886009+0000) 2020-02-05T20:53:12.888 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:12.884+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.885014+0000) 2020-02-05T20:53:12.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:12.885+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.886894+0000) 2020-02-05T20:53:12.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:12.886+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.887385+0000) 2020-02-05T20:53:12.889 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:12.886+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.887249+0000) 2020-02-05T20:53:12.889 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:12.887+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.888084+0000) 2020-02-05T20:53:12.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:12.887+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.888361+0000) 2020-02-05T20:53:12.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:12.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.888499+0000) 2020-02-05T20:53:12.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:12.888+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.888718+0000) 2020-02-05T20:53:12.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:12.891+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:12.893040+0000) 2020-02-05T20:53:12.894 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:12.892+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:13.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:13.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:13.423111+0000) 2020-02-05T20:53:13.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:13.453+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:13.453638+0000) 2020-02-05T20:53:15.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:15.325+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:15.325833+0000) 2020-02-05T20:53:15.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:15.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:15.797934+0000) 2020-02-05T20:53:15.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:15.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:15.834785+0000) 2020-02-05T20:53:16.696 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:16.693+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:16.694852+0000) 2020-02-05T20:53:16.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:16.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:16.938244+0000) 2020-02-05T20:53:16.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:16.937+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:16.938844+0000) 2020-02-05T20:53:17.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:17.897+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.899207+0000) 2020-02-05T20:53:17.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:17.898+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.899967+0000) 2020-02-05T20:53:17.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:17.899+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.900382+0000) 2020-02-05T20:53:17.902 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:17.900+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.901320+0000) 2020-02-05T20:53:17.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:17.900+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.901736+0000) 2020-02-05T20:53:17.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:17.900+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.901766+0000) 2020-02-05T20:53:17.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:17.900+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.902198+0000) 2020-02-05T20:53:17.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:17.900+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.902119+0000) 2020-02-05T20:53:17.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:17.901+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.902228+0000) 2020-02-05T20:53:17.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:17.901+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.902662+0000) 2020-02-05T20:53:17.905 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:17.901+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.903016+0000) 2020-02-05T20:53:17.906 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:17.902+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.903804+0000) 2020-02-05T20:53:17.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:17.902+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.903460+0000) 2020-02-05T20:53:17.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:17.904+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:17.905112+0000) 2020-02-05T20:53:17.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:17.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69660 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:19.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:19.332+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:19.333174+0000) 2020-02-05T20:53:19.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:19.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:19.942352+0000) 2020-02-05T20:53:20.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:20.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:20.192477+0000) 2020-02-05T20:53:20.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:20.202+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:20.202838+0000) 2020-02-05T20:53:22.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:22.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.180979+0000) 2020-02-05T20:53:22.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:22.295+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.296179+0000) 2020-02-05T20:53:22.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:22.914+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.914063+0000) 2020-02-05T20:53:22.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:22.914+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.914393+0000) 2020-02-05T20:53:22.916 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:22.914+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.914734+0000) 2020-02-05T20:53:22.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:22.913+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.915371+0000) 2020-02-05T20:53:22.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:22.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.915840+0000) 2020-02-05T20:53:22.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:22.916+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.916990+0000) 2020-02-05T20:53:22.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:22.915+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.917284+0000) 2020-02-05T20:53:22.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:22.916+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.917589+0000) 2020-02-05T20:53:22.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:22.917+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.917747+0000) 2020-02-05T20:53:22.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:22.917+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.917853+0000) 2020-02-05T20:53:22.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:22.916+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.917872+0000) 2020-02-05T20:53:22.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:22.916+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.918031+0000) 2020-02-05T20:53:22.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:22.916+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.918275+0000) 2020-02-05T20:53:22.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:22.921+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:22.921222+0000) 2020-02-05T20:53:22.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:22.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:23.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:23.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:23.423342+0000) 2020-02-05T20:53:23.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:23.453+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:23.453862+0000) 2020-02-05T20:53:25.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:25.337+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:25.337379+0000) 2020-02-05T20:53:25.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:25.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:25.798204+0000) 2020-02-05T20:53:25.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:25.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:25.834962+0000) 2020-02-05T20:53:26.696 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:26.694+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:26.695119+0000) 2020-02-05T20:53:26.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:26.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:26.938493+0000) 2020-02-05T20:53:26.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:26.937+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:26.939060+0000) 2020-02-05T20:53:27.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:27.925+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.926814+0000) 2020-02-05T20:53:27.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:27.926+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.927204+0000) 2020-02-05T20:53:27.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:27.926+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.927822+0000) 2020-02-05T20:53:27.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:27.927+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.928452+0000) 2020-02-05T20:53:27.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:27.927+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.929092+0000) 2020-02-05T20:53:27.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:27.928+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.929532+0000) 2020-02-05T20:53:27.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:27.928+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.929483+0000) 2020-02-05T20:53:27.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:27.930+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.931346+0000) 2020-02-05T20:53:27.933 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:27.929+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.931326+0000) 2020-02-05T20:53:27.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:27.930+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.931773+0000) 2020-02-05T20:53:27.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:27.930+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.932370+0000) 2020-02-05T20:53:27.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:27.931+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.932692+0000) 2020-02-05T20:53:27.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:27.931+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.932994+0000) 2020-02-05T20:53:27.938 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:27.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69742 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:27.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:27.940+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:27.941173+0000) 2020-02-05T20:53:29.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:29.332+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:29.333458+0000) 2020-02-05T20:53:29.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:29.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:29.942636+0000) 2020-02-05T20:53:30.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:30.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:30.192731+0000) 2020-02-05T20:53:30.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:30.202+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:30.203147+0000) 2020-02-05T20:53:30.945 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:53:30.951 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:53:32.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:32.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.181242+0000) 2020-02-05T20:53:32.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:32.295+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.296486+0000) 2020-02-05T20:53:32.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:32.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.940129+0000) 2020-02-05T20:53:32.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:32.939+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.940952+0000) 2020-02-05T20:53:32.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:32.940+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.941623+0000) 2020-02-05T20:53:32.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:32.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.942672+0000) 2020-02-05T20:53:32.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:32.941+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.943065+0000) 2020-02-05T20:53:32.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:32.942+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.944002+0000) 2020-02-05T20:53:32.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:32.944+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.945364+0000) 2020-02-05T20:53:32.947 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:32.944+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.945754+0000) 2020-02-05T20:53:32.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:32.944+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.946022+0000) 2020-02-05T20:53:32.948 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:32.945+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.947040+0000) 2020-02-05T20:53:32.948 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:32.945+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.947190+0000) 2020-02-05T20:53:32.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:32.945+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.947219+0000) 2020-02-05T20:53:32.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:32.946+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.947955+0000) 2020-02-05T20:53:32.951 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:32.948+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69782 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:32.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:32.951+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:32.952308+0000) 2020-02-05T20:53:33.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:33.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:33.423551+0000) 2020-02-05T20:53:33.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:33.453+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:33.454116+0000) 2020-02-05T20:53:35.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:35.345+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:35.346775+0000) 2020-02-05T20:53:35.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:35.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:35.798440+0000) 2020-02-05T20:53:35.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:35.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:35.835190+0000) 2020-02-05T20:53:36.696 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:36.694+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:36.695358+0000) 2020-02-05T20:53:36.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:36.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:36.938736+0000) 2020-02-05T20:53:36.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:36.937+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:36.939268+0000) 2020-02-05T20:53:37.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:37.955+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.956313+0000) 2020-02-05T20:53:37.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:37.955+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.956774+0000) 2020-02-05T20:53:37.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:37.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.957105+0000) 2020-02-05T20:53:37.959 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:37.956+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.957463+0000) 2020-02-05T20:53:37.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:37.956+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.957728+0000) 2020-02-05T20:53:37.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:37.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.958394+0000) 2020-02-05T20:53:37.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:37.957+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.958799+0000) 2020-02-05T20:53:37.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:37.957+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.959268+0000) 2020-02-05T20:53:37.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:37.958+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.959955+0000) 2020-02-05T20:53:37.961 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:37.959+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.960397+0000) 2020-02-05T20:53:37.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:37.958+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.960296+0000) 2020-02-05T20:53:37.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:37.959+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.961000+0000) 2020-02-05T20:53:37.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:37.960+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.961752+0000) 2020-02-05T20:53:37.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:37.964+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:37.965360+0000) 2020-02-05T20:53:37.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:37.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:39.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:39.332+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:39.333738+0000) 2020-02-05T20:53:39.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:39.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:39.942867+0000) 2020-02-05T20:53:40.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:40.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:40.192962+0000) 2020-02-05T20:53:40.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:40.202+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:40.203371+0000) 2020-02-05T20:53:42.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:42.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.181513+0000) 2020-02-05T20:53:42.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:42.296+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.296772+0000) 2020-02-05T20:53:42.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:42.969+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.970580+0000) 2020-02-05T20:53:42.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:42.970+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.971207+0000) 2020-02-05T20:53:42.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:42.970+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.971548+0000) 2020-02-05T20:53:42.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:42.970+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.971813+0000) 2020-02-05T20:53:42.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:42.971+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.972704+0000) 2020-02-05T20:53:42.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:42.972+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.973150+0000) 2020-02-05T20:53:42.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:42.972+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.973235+0000) 2020-02-05T20:53:42.975 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:42.971+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.973393+0000) 2020-02-05T20:53:42.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:42.972+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.973903+0000) 2020-02-05T20:53:42.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:42.972+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.974096+0000) 2020-02-05T20:53:42.976 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:42.972+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.973965+0000) 2020-02-05T20:53:42.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:42.973+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.974470+0000) 2020-02-05T20:53:42.977 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:42.974+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.975635+0000) 2020-02-05T20:53:42.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:42.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:42.979505+0000) 2020-02-05T20:53:42.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:42.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:43.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:43.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:43.423783+0000) 2020-02-05T20:53:43.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:43.453+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:43.454318+0000) 2020-02-05T20:53:45.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:45.354+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:45.354920+0000) 2020-02-05T20:53:45.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:45.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:45.798665+0000) 2020-02-05T20:53:45.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:45.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:45.835371+0000) 2020-02-05T20:53:46.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:46.694+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:46.695634+0000) 2020-02-05T20:53:46.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:46.938+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:46.939081+0000) 2020-02-05T20:53:46.941 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:46.938+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:46.939548+0000) 2020-02-05T20:53:47.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:47.983+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.984347+0000) 2020-02-05T20:53:47.988 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:47.985+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.986683+0000) 2020-02-05T20:53:47.988 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:47.986+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.987327+0000) 2020-02-05T20:53:47.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:47.986+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.987477+0000) 2020-02-05T20:53:47.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:47.986+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.988377+0000) 2020-02-05T20:53:47.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:47.987+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.988601+0000) 2020-02-05T20:53:47.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:47.988+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.989213+0000) 2020-02-05T20:53:47.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:47.988+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.989132+0000) 2020-02-05T20:53:47.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:47.987+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.989235+0000) 2020-02-05T20:53:47.992 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:47.988+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.989884+0000) 2020-02-05T20:53:47.992 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:47.988+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.990057+0000) 2020-02-05T20:53:47.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:47.988+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.989873+0000) 2020-02-05T20:53:47.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:47.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.990317+0000) 2020-02-05T20:53:47.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:47.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:47.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:47.994+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:47.995762+0000) 2020-02-05T20:53:49.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:49.333+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:49.333974+0000) 2020-02-05T20:53:49.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:49.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:49.943125+0000) 2020-02-05T20:53:50.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:50.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:50.193196+0000) 2020-02-05T20:53:50.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:50.202+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:50.203602+0000) 2020-02-05T20:53:52.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:52.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:52.181767+0000) 2020-02-05T20:53:52.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:52.296+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:52.297002+0000) 2020-02-05T20:53:52.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:52.997+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:52.998519+0000) 2020-02-05T20:53:53.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:52.997+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:52.998825+0000) 2020-02-05T20:53:53.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:52.998+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.000275+0000) 2020-02-05T20:53:53.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:52.999+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.000614+0000) 2020-02-05T20:53:53.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:53.000+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.001237+0000) 2020-02-05T20:53:53.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:53.001+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.002460+0000) 2020-02-05T20:53:53.004 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:53.000+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.002223+0000) 2020-02-05T20:53:53.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:53.001+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.002864+0000) 2020-02-05T20:53:53.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:53.002+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.003551+0000) 2020-02-05T20:53:53.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:53.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.004440+0000) 2020-02-05T20:53:53.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:53.003+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.004610+0000) 2020-02-05T20:53:53.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:53.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.004713+0000) 2020-02-05T20:53:53.006 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:53.003+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.004914+0000) 2020-02-05T20:53:53.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:53.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69948 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:53.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:53.007+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.008410+0000) 2020-02-05T20:53:53.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:53.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.424019+0000) 2020-02-05T20:53:53.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:53.453+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:53.454519+0000) 2020-02-05T20:53:55.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:55.361+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:55.361911+0000) 2020-02-05T20:53:55.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:55.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:55.798888+0000) 2020-02-05T20:53:55.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:55.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:55.835624+0000) 2020-02-05T20:53:56.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:56.695+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:56.695891+0000) 2020-02-05T20:53:56.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:56.937+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:56.939325+0000) 2020-02-05T20:53:56.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:56.938+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:56.939865+0000) 2020-02-05T20:53:58.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:58.012+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.013531+0000) 2020-02-05T20:53:58.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:53:58.013+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.014231+0000) 2020-02-05T20:53:58.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:53:58.013+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.014853+0000) 2020-02-05T20:53:58.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:53:58.014+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.015678+0000) 2020-02-05T20:53:58.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:53:58.014+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.016161+0000) 2020-02-05T20:53:58.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:53:58.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.016780+0000) 2020-02-05T20:53:58.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:53:58.015+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.017105+0000) 2020-02-05T20:53:58.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:58.016+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.017619+0000) 2020-02-05T20:53:58.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:53:58.016+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.017965+0000) 2020-02-05T20:53:58.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:58.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.018221+0000) 2020-02-05T20:53:58.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:53:58.017+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.018912+0000) 2020-02-05T20:53:58.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:53:58.018+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.019567+0000) 2020-02-05T20:53:58.021 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:53:58.018+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.019910+0000) 2020-02-05T20:53:58.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:53:58.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 69992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:53:58.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:58.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:58.024644+0000) 2020-02-05T20:53:59.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:53:59.333+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:59.334210+0000) 2020-02-05T20:53:59.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:53:59.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:53:59.943363+0000) 2020-02-05T20:54:00.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:00.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:00.193485+0000) 2020-02-05T20:54:00.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:00.203+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:00.203829+0000) 2020-02-05T20:54:01.099 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:54:01.106 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:54:02.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:02.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:02.182027+0000) 2020-02-05T20:54:02.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:02.296+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:02.297259+0000) 2020-02-05T20:54:03.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:03.027+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.028634+0000) 2020-02-05T20:54:03.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:03.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.029853+0000) 2020-02-05T20:54:03.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:03.028+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.030025+0000) 2020-02-05T20:54:03.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:03.029+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.030388+0000) 2020-02-05T20:54:03.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:03.029+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.031280+0000) 2020-02-05T20:54:03.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:03.029+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.031341+0000) 2020-02-05T20:54:03.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:03.031+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.032375+0000) 2020-02-05T20:54:03.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:03.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.032567+0000) 2020-02-05T20:54:03.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:03.031+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.032820+0000) 2020-02-05T20:54:03.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:03.031+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.032805+0000) 2020-02-05T20:54:03.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:03.032+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.033545+0000) 2020-02-05T20:54:03.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:03.032+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.033759+0000) 2020-02-05T20:54:03.035 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:03.032+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.033891+0000) 2020-02-05T20:54:03.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:03.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70031 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:03.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:03.036+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.037247+0000) 2020-02-05T20:54:03.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:03.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.424252+0000) 2020-02-05T20:54:03.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:03.454+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:03.454730+0000) 2020-02-05T20:54:05.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:05.369+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:05.370202+0000) 2020-02-05T20:54:05.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:05.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:05.799123+0000) 2020-02-05T20:54:05.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:05.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:05.835874+0000) 2020-02-05T20:54:06.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:06.695+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:06.696200+0000) 2020-02-05T20:54:06.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:06.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:06.939573+0000) 2020-02-05T20:54:06.941 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:06.939+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:06.940085+0000) 2020-02-05T20:54:08.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:08.041+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.042244+0000) 2020-02-05T20:54:08.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:08.042+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.043049+0000) 2020-02-05T20:54:08.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:08.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.043761+0000) 2020-02-05T20:54:08.045 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:08.043+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.044256+0000) 2020-02-05T20:54:08.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:08.044+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.045392+0000) 2020-02-05T20:54:08.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:08.044+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.045784+0000) 2020-02-05T20:54:08.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:08.045+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.046278+0000) 2020-02-05T20:54:08.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:08.046+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.047140+0000) 2020-02-05T20:54:08.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:08.047+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.047769+0000) 2020-02-05T20:54:08.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:08.047+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.048007+0000) 2020-02-05T20:54:08.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:08.047+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.048031+0000) 2020-02-05T20:54:08.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:08.047+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.047916+0000) 2020-02-05T20:54:08.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:08.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:08.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:08.051+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.052298+0000) 2020-02-05T20:54:08.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:08.051+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:08.052598+0000) 2020-02-05T20:54:09.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:09.334+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:09.334466+0000) 2020-02-05T20:54:09.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:09.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:09.943562+0000) 2020-02-05T20:54:10.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:10.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:10.193670+0000) 2020-02-05T20:54:10.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:10.203+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:10.204060+0000) 2020-02-05T20:54:12.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:12.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:12.182268+0000) 2020-02-05T20:54:12.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:12.297+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:12.297528+0000) 2020-02-05T20:54:13.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:13.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.056362+0000) 2020-02-05T20:54:13.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:13.058+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.058528+0000) 2020-02-05T20:54:13.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:13.058+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.059435+0000) 2020-02-05T20:54:13.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:13.059+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.059932+0000) 2020-02-05T20:54:13.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:13.059+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.060473+0000) 2020-02-05T20:54:13.062 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:13.060+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.061066+0000) 2020-02-05T20:54:13.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:13.060+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.061431+0000) 2020-02-05T20:54:13.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:13.061+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.062067+0000) 2020-02-05T20:54:13.063 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:13.061+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.062388+0000) 2020-02-05T20:54:13.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:13.062+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.062690+0000) 2020-02-05T20:54:13.064 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:13.062+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.063002+0000) 2020-02-05T20:54:13.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:13.062+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.062954+0000) 2020-02-05T20:54:13.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:13.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.063440+0000) 2020-02-05T20:54:13.067 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:13.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:13.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:13.066+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.067587+0000) 2020-02-05T20:54:13.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:13.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.424526+0000) 2020-02-05T20:54:13.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:13.454+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:13.454959+0000) 2020-02-05T20:54:15.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:15.378+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:15.378697+0000) 2020-02-05T20:54:15.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:15.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:15.799348+0000) 2020-02-05T20:54:15.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:15.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:15.836170+0000) 2020-02-05T20:54:16.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:16.696+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:16.696434+0000) 2020-02-05T20:54:16.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:16.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:16.939830+0000) 2020-02-05T20:54:16.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:16.939+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:16.940343+0000) 2020-02-05T20:54:18.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:18.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.071626+0000) 2020-02-05T20:54:18.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:18.072+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.072537+0000) 2020-02-05T20:54:18.074 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:18.073+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.073183+0000) 2020-02-05T20:54:18.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:18.073+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.073623+0000) 2020-02-05T20:54:18.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:18.073+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.074053+0000) 2020-02-05T20:54:18.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:18.074+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.074499+0000) 2020-02-05T20:54:18.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:18.074+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.075149+0000) 2020-02-05T20:54:18.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:18.075+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.075666+0000) 2020-02-05T20:54:18.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:18.075+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.076140+0000) 2020-02-05T20:54:18.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:18.076+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.076508+0000) 2020-02-05T20:54:18.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:18.076+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.077246+0000) 2020-02-05T20:54:18.079 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:18.077+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.077124+0000) 2020-02-05T20:54:18.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:18.077+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.078040+0000) 2020-02-05T20:54:18.082 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:18.080+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:18.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:18.083+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:18.083575+0000) 2020-02-05T20:54:19.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:19.334+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:19.334698+0000) 2020-02-05T20:54:19.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:19.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:19.943825+0000) 2020-02-05T20:54:20.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:20.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:20.193907+0000) 2020-02-05T20:54:20.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:20.204+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:20.204290+0000) 2020-02-05T20:54:22.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:22.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:22.182526+0000) 2020-02-05T20:54:22.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:22.297+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:22.297827+0000) 2020-02-05T20:54:23.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:23.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.086260+0000) 2020-02-05T20:54:23.088 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:23.085+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.086898+0000) 2020-02-05T20:54:23.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:23.087+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.087804+0000) 2020-02-05T20:54:23.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:23.087+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.088360+0000) 2020-02-05T20:54:23.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:23.087+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.088847+0000) 2020-02-05T20:54:23.091 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:23.088+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.089336+0000) 2020-02-05T20:54:23.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:23.089+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.090097+0000) 2020-02-05T20:54:23.092 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:23.089+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.090308+0000) 2020-02-05T20:54:23.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:23.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.089982+0000) 2020-02-05T20:54:23.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:23.090+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.091639+0000) 2020-02-05T20:54:23.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:23.091+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.092112+0000) 2020-02-05T20:54:23.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:23.092+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.092641+0000) 2020-02-05T20:54:23.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:23.091+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.092744+0000) 2020-02-05T20:54:23.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:23.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:23.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:23.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.097611+0000) 2020-02-05T20:54:23.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:23.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.424763+0000) 2020-02-05T20:54:23.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:23.454+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:23.455175+0000) 2020-02-05T20:54:25.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:25.385+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:25.386706+0000) 2020-02-05T20:54:25.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:25.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:25.799590+0000) 2020-02-05T20:54:25.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:25.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:25.836434+0000) 2020-02-05T20:54:26.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:26.695+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:26.696672+0000) 2020-02-05T20:54:26.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:26.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:26.940073+0000) 2020-02-05T20:54:26.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:26.940+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:26.940626+0000) 2020-02-05T20:54:28.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:28.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.101955+0000) 2020-02-05T20:54:28.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:28.101+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.102414+0000) 2020-02-05T20:54:28.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:28.101+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.102748+0000) 2020-02-05T20:54:28.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:28.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.103224+0000) 2020-02-05T20:54:28.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:28.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.103555+0000) 2020-02-05T20:54:28.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:28.103+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.104610+0000) 2020-02-05T20:54:28.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:28.104+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.104928+0000) 2020-02-05T20:54:28.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:28.104+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.105601+0000) 2020-02-05T20:54:28.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:28.104+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.105251+0000) 2020-02-05T20:54:28.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:28.105+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.105948+0000) 2020-02-05T20:54:28.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:28.106+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.106593+0000) 2020-02-05T20:54:28.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:28.105+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.106294+0000) 2020-02-05T20:54:28.108 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:28.106+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.107442+0000) 2020-02-05T20:54:28.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:28.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:28.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:28.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:28.112331+0000) 2020-02-05T20:54:29.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:29.334+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:29.334942+0000) 2020-02-05T20:54:29.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:29.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:29.944055+0000) 2020-02-05T20:54:30.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:30.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:30.194142+0000) 2020-02-05T20:54:30.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:30.203+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:30.204548+0000) 2020-02-05T20:54:31.262 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:54:31.269 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:54:32.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:32.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:32.182779+0000) 2020-02-05T20:54:32.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:32.297+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:32.298108+0000) 2020-02-05T20:54:33.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:33.116+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.117473+0000) 2020-02-05T20:54:33.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:33.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.119485+0000) 2020-02-05T20:54:33.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:33.119+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.120143+0000) 2020-02-05T20:54:33.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:33.119+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.120709+0000) 2020-02-05T20:54:33.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:33.120+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.121071+0000) 2020-02-05T20:54:33.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:33.121+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.121819+0000) 2020-02-05T20:54:33.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:33.121+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.121748+0000) 2020-02-05T20:54:33.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:33.121+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.122363+0000) 2020-02-05T20:54:33.124 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:33.122+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.122644+0000) 2020-02-05T20:54:33.124 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:33.122+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.123418+0000) 2020-02-05T20:54:33.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:33.122+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.123780+0000) 2020-02-05T20:54:33.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:33.123+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.124434+0000) 2020-02-05T20:54:33.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:33.124+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.124675+0000) 2020-02-05T20:54:33.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:33.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:33.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:33.129+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.130573+0000) 2020-02-05T20:54:33.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:33.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.425014+0000) 2020-02-05T20:54:33.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:33.454+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:33.455386+0000) 2020-02-05T20:54:35.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:35.393+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:35.394610+0000) 2020-02-05T20:54:35.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:35.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:35.799807+0000) 2020-02-05T20:54:35.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:35.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:35.836663+0000) 2020-02-05T20:54:36.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:36.696+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:36.696965+0000) 2020-02-05T20:54:36.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:36.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:36.940255+0000) 2020-02-05T20:54:36.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:36.940+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:36.940812+0000) 2020-02-05T20:54:38.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:38.131+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.132832+0000) 2020-02-05T20:54:38.134 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:38.132+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.133650+0000) 2020-02-05T20:54:38.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:38.134+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.134567+0000) 2020-02-05T20:54:38.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:38.134+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.135271+0000) 2020-02-05T20:54:38.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:38.134+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.135821+0000) 2020-02-05T20:54:38.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:38.135+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.136231+0000) 2020-02-05T20:54:38.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:38.136+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.136695+0000) 2020-02-05T20:54:38.138 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:38.136+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.137423+0000) 2020-02-05T20:54:38.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:38.137+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.137885+0000) 2020-02-05T20:54:38.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:38.138+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.138634+0000) 2020-02-05T20:54:38.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:38.138+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.139177+0000) 2020-02-05T20:54:38.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:38.138+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.139465+0000) 2020-02-05T20:54:38.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:38.139+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.139939+0000) 2020-02-05T20:54:38.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:38.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:38.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:38.142+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:38.143355+0000) 2020-02-05T20:54:39.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:39.334+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:39.335177+0000) 2020-02-05T20:54:39.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:39.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:39.944323+0000) 2020-02-05T20:54:40.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:40.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:40.194389+0000) 2020-02-05T20:54:40.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:40.203+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:40.204756+0000) 2020-02-05T20:54:42.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:42.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:42.183051+0000) 2020-02-05T20:54:42.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:42.298+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:42.298358+0000) 2020-02-05T20:54:43.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:43.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.148445+0000) 2020-02-05T20:54:43.150 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:43.147+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.148897+0000) 2020-02-05T20:54:43.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:43.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.149417+0000) 2020-02-05T20:54:43.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:43.149+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.149823+0000) 2020-02-05T20:54:43.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:43.150+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.151067+0000) 2020-02-05T20:54:43.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:43.151+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.151828+0000) 2020-02-05T20:54:43.153 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:43.151+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.151953+0000) 2020-02-05T20:54:43.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:43.152+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.152542+0000) 2020-02-05T20:54:43.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:43.152+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.153504+0000) 2020-02-05T20:54:43.155 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:43.152+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.153529+0000) 2020-02-05T20:54:43.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:43.154+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.154934+0000) 2020-02-05T20:54:43.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:43.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.155071+0000) 2020-02-05T20:54:43.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:43.154+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.155112+0000) 2020-02-05T20:54:43.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:43.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:43.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:43.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.159925+0000) 2020-02-05T20:54:43.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:43.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.425251+0000) 2020-02-05T20:54:43.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:43.454+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:43.455603+0000) 2020-02-05T20:54:45.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:45.401+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:45.402528+0000) 2020-02-05T20:54:45.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:45.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:45.800033+0000) 2020-02-05T20:54:45.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:45.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:45.836895+0000) 2020-02-05T20:54:46.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:46.696+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:46.697198+0000) 2020-02-05T20:54:46.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:46.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:46.940518+0000) 2020-02-05T20:54:46.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:46.939+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:46.941058+0000) 2020-02-05T20:54:48.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:48.163+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.164417+0000) 2020-02-05T20:54:48.167 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:48.164+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.165163+0000) 2020-02-05T20:54:48.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:48.164+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.165453+0000) 2020-02-05T20:54:48.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:48.164+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.165891+0000) 2020-02-05T20:54:48.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:48.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.166420+0000) 2020-02-05T20:54:48.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:48.164+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.166028+0000) 2020-02-05T20:54:48.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:48.166+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.167468+0000) 2020-02-05T20:54:48.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:48.166+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.167767+0000) 2020-02-05T20:54:48.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:48.166+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.168004+0000) 2020-02-05T20:54:48.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:48.166+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.168052+0000) 2020-02-05T20:54:48.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:48.167+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.168863+0000) 2020-02-05T20:54:48.171 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:48.168+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.169576+0000) 2020-02-05T20:54:48.171 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:48.168+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.169870+0000) 2020-02-05T20:54:48.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:48.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:48.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:48.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:48.172331+0000) 2020-02-05T20:54:49.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:49.334+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:49.335365+0000) 2020-02-05T20:54:49.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:49.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:49.944600+0000) 2020-02-05T20:54:50.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:50.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:50.194656+0000) 2020-02-05T20:54:50.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:50.203+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:50.205005+0000) 2020-02-05T20:54:52.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:52.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:52.183341+0000) 2020-02-05T20:54:52.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:52.297+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:52.298641+0000) 2020-02-05T20:54:53.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:53.177+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.178513+0000) 2020-02-05T20:54:53.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:53.178+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.179715+0000) 2020-02-05T20:54:53.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:53.179+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.180710+0000) 2020-02-05T20:54:53.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:53.180+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.181393+0000) 2020-02-05T20:54:53.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:53.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.181833+0000) 2020-02-05T20:54:53.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:53.180+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.182170+0000) 2020-02-05T20:54:53.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:53.181+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.183088+0000) 2020-02-05T20:54:53.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:53.182+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.183764+0000) 2020-02-05T20:54:53.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:53.182+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.183686+0000) 2020-02-05T20:54:53.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:53.183+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.184756+0000) 2020-02-05T20:54:53.186 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:53.183+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.184873+0000) 2020-02-05T20:54:53.186 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:53.183+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.185206+0000) 2020-02-05T20:54:53.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:53.185+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:53.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:53.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.189117+0000) 2020-02-05T20:54:53.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:53.189+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.190358+0000) 2020-02-05T20:54:53.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:53.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.425487+0000) 2020-02-05T20:54:53.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:53.455+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:53.455762+0000) 2020-02-05T20:54:55.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:55.409+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:55.410742+0000) 2020-02-05T20:54:55.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:55.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:55.800326+0000) 2020-02-05T20:54:55.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:55.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:55.837177+0000) 2020-02-05T20:54:56.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:56.697+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:56.697429+0000) 2020-02-05T20:54:56.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:56.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:56.940788+0000) 2020-02-05T20:54:56.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:56.939+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:56.941253+0000) 2020-02-05T20:54:58.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:54:58.191+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.193366+0000) 2020-02-05T20:54:58.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:54:58.191+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.192928+0000) 2020-02-05T20:54:58.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:54:58.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.194313+0000) 2020-02-05T20:54:58.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:54:58.194+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.195547+0000) 2020-02-05T20:54:58.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:58.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.196198+0000) 2020-02-05T20:54:58.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:58.195+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.196674+0000) 2020-02-05T20:54:58.198 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:54:58.196+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.197589+0000) 2020-02-05T20:54:58.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:54:58.195+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.197277+0000) 2020-02-05T20:54:58.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:58.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.198222+0000) 2020-02-05T20:54:58.200 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:54:58.197+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.198463+0000) 2020-02-05T20:54:58.200 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:54:58.197+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.198738+0000) 2020-02-05T20:54:58.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:54:58.196+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.198399+0000) 2020-02-05T20:54:58.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:54:58.197+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.198760+0000) 2020-02-05T20:54:58.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:54:58.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:54:58.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:58.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:58.204553+0000) 2020-02-05T20:54:59.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:54:59.335+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:59.335604+0000) 2020-02-05T20:54:59.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:54:59.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:54:59.944833+0000) 2020-02-05T20:55:00.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:00.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:00.194910+0000) 2020-02-05T20:55:00.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:00.204+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:00.205314+0000) 2020-02-05T20:55:01.436 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:55:01.442 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:55:02.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:02.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:02.183601+0000) 2020-02-05T20:55:02.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:02.298+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:02.298882+0000) 2020-02-05T20:55:03.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:03.207+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.208411+0000) 2020-02-05T20:55:03.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:03.208+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.209550+0000) 2020-02-05T20:55:03.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:03.209+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.210156+0000) 2020-02-05T20:55:03.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:03.209+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.210830+0000) 2020-02-05T20:55:03.212 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:03.210+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.211587+0000) 2020-02-05T20:55:03.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:03.211+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.212559+0000) 2020-02-05T20:55:03.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:03.211+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.212453+0000) 2020-02-05T20:55:03.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:03.210+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.211865+0000) 2020-02-05T20:55:03.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:03.211+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.212972+0000) 2020-02-05T20:55:03.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:03.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.213756+0000) 2020-02-05T20:55:03.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:03.212+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.213780+0000) 2020-02-05T20:55:03.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:03.212+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.213494+0000) 2020-02-05T20:55:03.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:03.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.213931+0000) 2020-02-05T20:55:03.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:03.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70529 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:03.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:03.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.217670+0000) 2020-02-05T20:55:03.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:03.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.425741+0000) 2020-02-05T20:55:03.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:03.455+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:03.455968+0000) 2020-02-05T20:55:05.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:05.418+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:05.418695+0000) 2020-02-05T20:55:05.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:05.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:05.800554+0000) 2020-02-05T20:55:05.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:05.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:05.837377+0000) 2020-02-05T20:55:06.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:06.697+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:06.697661+0000) 2020-02-05T20:55:06.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:06.939+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:06.941025+0000) 2020-02-05T20:55:06.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:06.940+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:06.941477+0000) 2020-02-05T20:55:08.223 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:08.220+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.221953+0000) 2020-02-05T20:55:08.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:08.222+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.223103+0000) 2020-02-05T20:55:08.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:08.222+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.223741+0000) 2020-02-05T20:55:08.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:08.222+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.224282+0000) 2020-02-05T20:55:08.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:08.222+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.224396+0000) 2020-02-05T20:55:08.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:08.223+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.224926+0000) 2020-02-05T20:55:08.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:08.223+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.225042+0000) 2020-02-05T20:55:08.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:08.224+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.225517+0000) 2020-02-05T20:55:08.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:08.224+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.225876+0000) 2020-02-05T20:55:08.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:08.225+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.226416+0000) 2020-02-05T20:55:08.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:08.224+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.226006+0000) 2020-02-05T20:55:08.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:08.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.227323+0000) 2020-02-05T20:55:08.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:08.225+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.227357+0000) 2020-02-05T20:55:08.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:08.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:08.232262+0000) 2020-02-05T20:55:08.238 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:08.229+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:09.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:09.335+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:09.335852+0000) 2020-02-05T20:55:09.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:09.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:09.945066+0000) 2020-02-05T20:55:10.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:10.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:10.195159+0000) 2020-02-05T20:55:10.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:10.205+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:10.205588+0000) 2020-02-05T20:55:12.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:12.183+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:12.183868+0000) 2020-02-05T20:55:12.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:12.297+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:12.299123+0000) 2020-02-05T20:55:13.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:13.235+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.235922+0000) 2020-02-05T20:55:13.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:13.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.237706+0000) 2020-02-05T20:55:13.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:13.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.238170+0000) 2020-02-05T20:55:13.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:13.238+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.238617+0000) 2020-02-05T20:55:13.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:13.237+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.239236+0000) 2020-02-05T20:55:13.241 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:13.239+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.239625+0000) 2020-02-05T20:55:13.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:13.238+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.239953+0000) 2020-02-05T20:55:13.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:13.240+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.240427+0000) 2020-02-05T20:55:13.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:13.239+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.240470+0000) 2020-02-05T20:55:13.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:13.240+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.240956+0000) 2020-02-05T20:55:13.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:13.242+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.242267+0000) 2020-02-05T20:55:13.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:13.240+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.242278+0000) 2020-02-05T20:55:13.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:13.241+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.243032+0000) 2020-02-05T20:55:13.245 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:13.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:13.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:13.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.246900+0000) 2020-02-05T20:55:13.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:13.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.425967+0000) 2020-02-05T20:55:13.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:13.456+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:13.456200+0000) 2020-02-05T20:55:15.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:15.426+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:15.427169+0000) 2020-02-05T20:55:15.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:15.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:15.800792+0000) 2020-02-05T20:55:15.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:15.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:15.837613+0000) 2020-02-05T20:55:16.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:16.696+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:16.697896+0000) 2020-02-05T20:55:16.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:16.940+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:16.941276+0000) 2020-02-05T20:55:16.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:16.940+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:16.941688+0000) 2020-02-05T20:55:18.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:18.248+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.249767+0000) 2020-02-05T20:55:18.252 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:18.250+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.251187+0000) 2020-02-05T20:55:18.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:18.250+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.251784+0000) 2020-02-05T20:55:18.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:18.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.252406+0000) 2020-02-05T20:55:18.254 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:18.251+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.252872+0000) 2020-02-05T20:55:18.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:18.251+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.253173+0000) 2020-02-05T20:55:18.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:18.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.253818+0000) 2020-02-05T20:55:18.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:18.253+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.254220+0000) 2020-02-05T20:55:18.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:18.253+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.254629+0000) 2020-02-05T20:55:18.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:18.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.255096+0000) 2020-02-05T20:55:18.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:18.253+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.255259+0000) 2020-02-05T20:55:18.257 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:18.255+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.256471+0000) 2020-02-05T20:55:18.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:18.255+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.256521+0000) 2020-02-05T20:55:18.261 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:18.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:18.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:18.260+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:18.261072+0000) 2020-02-05T20:55:19.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:19.335+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:19.336150+0000) 2020-02-05T20:55:19.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:19.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:19.945315+0000) 2020-02-05T20:55:20.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:20.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:20.195405+0000) 2020-02-05T20:55:20.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:20.204+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:20.205815+0000) 2020-02-05T20:55:22.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:22.183+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:22.184123+0000) 2020-02-05T20:55:22.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:22.298+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:22.299367+0000) 2020-02-05T20:55:23.267 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:23.265+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.266375+0000) 2020-02-05T20:55:23.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:23.264+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.266273+0000) 2020-02-05T20:55:23.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:23.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.267193+0000) 2020-02-05T20:55:23.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:23.266+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.267595+0000) 2020-02-05T20:55:23.269 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:23.267+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.268594+0000) 2020-02-05T20:55:23.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:23.267+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.268943+0000) 2020-02-05T20:55:23.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:23.268+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.269627+0000) 2020-02-05T20:55:23.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:23.268+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.269900+0000) 2020-02-05T20:55:23.272 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:23.269+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.270817+0000) 2020-02-05T20:55:23.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:23.269+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.270629+0000) 2020-02-05T20:55:23.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:23.269+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.271012+0000) 2020-02-05T20:55:23.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:23.270+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.271798+0000) 2020-02-05T20:55:23.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:23.269+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.271396+0000) 2020-02-05T20:55:23.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:23.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:23.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:23.276+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.277538+0000) 2020-02-05T20:55:23.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:23.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.426245+0000) 2020-02-05T20:55:23.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:23.455+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:23.456412+0000) 2020-02-05T20:55:25.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:25.434+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:25.435605+0000) 2020-02-05T20:55:25.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:25.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:25.801019+0000) 2020-02-05T20:55:25.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:25.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:25.837845+0000) 2020-02-05T20:55:26.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:26.697+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:26.698128+0000) 2020-02-05T20:55:26.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:26.940+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:26.941535+0000) 2020-02-05T20:55:26.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:26.940+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:26.941932+0000) 2020-02-05T20:55:28.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:28.280+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.281208+0000) 2020-02-05T20:55:28.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:28.280+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.281644+0000) 2020-02-05T20:55:28.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:28.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.282024+0000) 2020-02-05T20:55:28.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:28.281+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.282588+0000) 2020-02-05T20:55:28.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:28.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.282895+0000) 2020-02-05T20:55:28.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:28.282+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.283419+0000) 2020-02-05T20:55:28.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:28.282+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.283989+0000) 2020-02-05T20:55:28.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:28.283+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.284380+0000) 2020-02-05T20:55:28.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:28.283+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.284823+0000) 2020-02-05T20:55:28.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:28.283+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.285038+0000) 2020-02-05T20:55:28.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:28.284+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.286168+0000) 2020-02-05T20:55:28.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:28.284+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.286323+0000) 2020-02-05T20:55:28.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:28.284+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.286058+0000) 2020-02-05T20:55:28.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:28.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:28.288161+0000) 2020-02-05T20:55:28.291 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:28.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:29.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:29.335+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:29.336383+0000) 2020-02-05T20:55:29.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:29.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:29.945551+0000) 2020-02-05T20:55:30.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:30.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:30.195699+0000) 2020-02-05T20:55:30.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:30.205+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:30.206041+0000) 2020-02-05T20:55:31.573 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:55:31.579 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:55:32.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:32.183+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:32.184410+0000) 2020-02-05T20:55:32.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:32.299+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:32.299644+0000) 2020-02-05T20:55:33.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:33.294+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.295716+0000) 2020-02-05T20:55:33.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:33.296+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.297066+0000) 2020-02-05T20:55:33.298 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:33.296+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.297543+0000) 2020-02-05T20:55:33.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:33.296+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.297943+0000) 2020-02-05T20:55:33.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:33.297+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.298307+0000) 2020-02-05T20:55:33.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:33.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.298418+0000) 2020-02-05T20:55:33.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:33.297+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.298933+0000) 2020-02-05T20:55:33.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:33.298+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.299535+0000) 2020-02-05T20:55:33.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:33.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.300679+0000) 2020-02-05T20:55:33.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:33.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.300702+0000) 2020-02-05T20:55:33.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:33.299+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.300896+0000) 2020-02-05T20:55:33.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:33.299+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.301121+0000) 2020-02-05T20:55:33.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:33.300+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.301852+0000) 2020-02-05T20:55:33.305 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:33.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:33.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:33.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.308245+0000) 2020-02-05T20:55:33.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:33.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.426519+0000) 2020-02-05T20:55:33.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:33.455+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:33.456635+0000) 2020-02-05T20:55:35.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:35.446+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:35.447301+0000) 2020-02-05T20:55:35.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:35.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:35.801297+0000) 2020-02-05T20:55:35.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:35.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:35.838115+0000) 2020-02-05T20:55:36.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:36.697+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:36.698361+0000) 2020-02-05T20:55:36.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:36.941+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:36.941786+0000) 2020-02-05T20:55:36.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:36.941+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:36.942216+0000) 2020-02-05T20:55:38.312 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:38.309+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.310916+0000) 2020-02-05T20:55:38.313 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:38.310+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.310609+0000) 2020-02-05T20:55:38.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:38.310+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.311785+0000) 2020-02-05T20:55:38.314 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:38.311+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.312363+0000) 2020-02-05T20:55:38.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:38.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.312589+0000) 2020-02-05T20:55:38.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:38.312+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.313409+0000) 2020-02-05T20:55:38.315 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:38.312+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.313798+0000) 2020-02-05T20:55:38.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:38.313+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.314042+0000) 2020-02-05T20:55:38.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:38.313+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.314464+0000) 2020-02-05T20:55:38.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:38.314+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.314722+0000) 2020-02-05T20:55:38.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:38.315+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.315744+0000) 2020-02-05T20:55:38.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:38.316+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.316763+0000) 2020-02-05T20:55:38.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:38.316+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.317036+0000) 2020-02-05T20:55:38.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:38.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:38.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:38.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:38.321448+0000) 2020-02-05T20:55:39.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:39.336+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:39.336652+0000) 2020-02-05T20:55:39.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:39.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:39.945772+0000) 2020-02-05T20:55:40.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:40.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:40.195943+0000) 2020-02-05T20:55:40.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:40.205+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:40.206335+0000) 2020-02-05T20:55:42.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:42.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:42.184706+0000) 2020-02-05T20:55:42.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:42.299+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:42.299875+0000) 2020-02-05T20:55:43.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:43.323+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.324581+0000) 2020-02-05T20:55:43.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:43.325+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.326474+0000) 2020-02-05T20:55:43.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:43.326+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.327498+0000) 2020-02-05T20:55:43.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:43.326+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.327779+0000) 2020-02-05T20:55:43.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:43.327+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.328284+0000) 2020-02-05T20:55:43.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:43.328+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.328633+0000) 2020-02-05T20:55:43.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:43.329+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.329738+0000) 2020-02-05T20:55:43.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:43.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.329972+0000) 2020-02-05T20:55:43.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:43.329+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.330100+0000) 2020-02-05T20:55:43.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:43.329+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.329662+0000) 2020-02-05T20:55:43.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:43.330+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.331311+0000) 2020-02-05T20:55:43.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:43.330+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.331282+0000) 2020-02-05T20:55:43.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:43.331+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.332355+0000) 2020-02-05T20:55:43.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:43.333+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.334595+0000) 2020-02-05T20:55:43.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:43.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:43.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:43.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.426727+0000) 2020-02-05T20:55:43.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:43.456+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:43.456854+0000) 2020-02-05T20:55:45.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:45.455+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:45.455800+0000) 2020-02-05T20:55:45.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:45.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:45.801531+0000) 2020-02-05T20:55:45.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:45.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:45.838382+0000) 2020-02-05T20:55:46.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:46.697+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:46.698602+0000) 2020-02-05T20:55:46.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:46.941+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:46.942072+0000) 2020-02-05T20:55:46.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:46.942+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:46.942485+0000) 2020-02-05T20:55:48.344 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:48.342+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.342521+0000) 2020-02-05T20:55:48.344 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:48.342+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.343119+0000) 2020-02-05T20:55:48.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:48.343+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.343924+0000) 2020-02-05T20:55:48.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:48.344+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.344863+0000) 2020-02-05T20:55:48.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:48.344+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.345449+0000) 2020-02-05T20:55:48.347 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:48.344+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.345743+0000) 2020-02-05T20:55:48.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:48.344+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.345959+0000) 2020-02-05T20:55:48.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:48.345+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.346561+0000) 2020-02-05T20:55:48.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:48.345+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.345971+0000) 2020-02-05T20:55:48.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:48.345+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.346509+0000) 2020-02-05T20:55:48.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:48.346+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.346566+0000) 2020-02-05T20:55:48.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:48.347+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.347656+0000) 2020-02-05T20:55:48.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:48.347+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.348288+0000) 2020-02-05T20:55:48.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:48.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70904 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:48.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:48.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:48.351418+0000) 2020-02-05T20:55:49.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:49.336+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:49.336879+0000) 2020-02-05T20:55:49.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:49.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:49.946015+0000) 2020-02-05T20:55:50.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:50.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:50.196202+0000) 2020-02-05T20:55:50.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:50.206+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:50.206561+0000) 2020-02-05T20:55:52.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:52.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:52.184967+0000) 2020-02-05T20:55:52.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:52.299+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:52.300109+0000) 2020-02-05T20:55:53.357 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:53.354+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.356033+0000) 2020-02-05T20:55:53.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:53.355+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.356519+0000) 2020-02-05T20:55:53.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:53.356+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.357746+0000) 2020-02-05T20:55:53.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:53.357+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.358999+0000) 2020-02-05T20:55:53.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:53.358+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.359708+0000) 2020-02-05T20:55:53.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:53.359+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.359831+0000) 2020-02-05T20:55:53.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:53.359+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.359756+0000) 2020-02-05T20:55:53.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:53.359+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.360466+0000) 2020-02-05T20:55:53.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:53.359+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.360089+0000) 2020-02-05T20:55:53.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:53.360+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.360988+0000) 2020-02-05T20:55:53.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:53.360+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.361565+0000) 2020-02-05T20:55:53.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:53.360+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.361393+0000) 2020-02-05T20:55:53.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:53.361+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.361915+0000) 2020-02-05T20:55:53.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:53.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:53.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:53.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.367215+0000) 2020-02-05T20:55:53.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:53.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.426959+0000) 2020-02-05T20:55:53.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:53.456+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:53.457146+0000) 2020-02-05T20:55:55.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:55.463+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:55.463621+0000) 2020-02-05T20:55:55.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:55.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:55.801758+0000) 2020-02-05T20:55:55.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:55.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:55.838638+0000) 2020-02-05T20:55:56.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:56.698+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:56.698829+0000) 2020-02-05T20:55:56.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:56.941+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:56.942252+0000) 2020-02-05T20:55:56.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:56.942+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:56.942676+0000) 2020-02-05T20:55:58.372 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:55:58.369+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.370659+0000) 2020-02-05T20:55:58.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:55:58.371+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.372267+0000) 2020-02-05T20:55:58.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:55:58.371+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.372647+0000) 2020-02-05T20:55:58.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:55:58.371+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.372942+0000) 2020-02-05T20:55:58.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:58.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.373504+0000) 2020-02-05T20:55:58.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:58.372+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.373672+0000) 2020-02-05T20:55:58.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:55:58.373+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.373811+0000) 2020-02-05T20:55:58.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:55:58.373+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.373811+0000) 2020-02-05T20:55:58.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:55:58.373+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.374312+0000) 2020-02-05T20:55:58.376 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:55:58.374+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.374874+0000) 2020-02-05T20:55:58.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:55:58.374+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.374754+0000) 2020-02-05T20:55:58.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:55:58.375+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.375812+0000) 2020-02-05T20:55:58.377 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:58.375+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.376357+0000) 2020-02-05T20:55:58.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:55:58.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 70987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:55:58.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:58.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:58.381060+0000) 2020-02-05T20:55:59.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:55:59.336+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:59.337195+0000) 2020-02-05T20:55:59.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:55:59.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:55:59.946288+0000) 2020-02-05T20:56:00.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:00.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:00.196478+0000) 2020-02-05T20:56:00.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:00.205+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:00.206787+0000) 2020-02-05T20:56:01.750 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:56:01.759 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:56:02.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:02.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:02.185247+0000) 2020-02-05T20:56:02.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:02.300+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:02.300376+0000) 2020-02-05T20:56:03.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:03.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.385852+0000) 2020-02-05T20:56:03.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:03.385+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.386590+0000) 2020-02-05T20:56:03.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:03.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.387269+0000) 2020-02-05T20:56:03.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:03.386+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.387525+0000) 2020-02-05T20:56:03.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:03.387+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.388227+0000) 2020-02-05T20:56:03.390 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:03.388+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.388869+0000) 2020-02-05T20:56:03.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:03.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.389423+0000) 2020-02-05T20:56:03.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:03.388+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.388886+0000) 2020-02-05T20:56:03.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:03.388+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.389208+0000) 2020-02-05T20:56:03.391 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:03.388+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.389805+0000) 2020-02-05T20:56:03.392 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:03.388+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.389989+0000) 2020-02-05T20:56:03.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:03.389+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.390178+0000) 2020-02-05T20:56:03.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:03.390+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.390056+0000) 2020-02-05T20:56:03.395 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:03.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:03.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:03.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.396450+0000) 2020-02-05T20:56:03.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:03.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.427225+0000) 2020-02-05T20:56:03.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:03.457+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:03.457360+0000) 2020-02-05T20:56:05.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:05.471+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:05.471563+0000) 2020-02-05T20:56:05.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:05.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:05.802047+0000) 2020-02-05T20:56:05.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:05.838+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:05.838861+0000) 2020-02-05T20:56:06.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:06.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:06.699068+0000) 2020-02-05T20:56:06.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:06.942+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:06.942482+0000) 2020-02-05T20:56:06.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:06.942+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:06.942868+0000) 2020-02-05T20:56:08.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:08.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.401127+0000) 2020-02-05T20:56:08.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:08.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.401492+0000) 2020-02-05T20:56:08.403 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:08.401+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.401942+0000) 2020-02-05T20:56:08.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:08.402+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.402489+0000) 2020-02-05T20:56:08.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:08.402+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.402670+0000) 2020-02-05T20:56:08.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:08.403+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.403441+0000) 2020-02-05T20:56:08.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:08.403+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.403834+0000) 2020-02-05T20:56:08.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:08.404+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.404694+0000) 2020-02-05T20:56:08.406 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:08.405+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.405356+0000) 2020-02-05T20:56:08.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:08.405+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.405567+0000) 2020-02-05T20:56:08.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:08.405+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.405600+0000) 2020-02-05T20:56:08.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:08.404+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.404665+0000) 2020-02-05T20:56:08.408 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:08.406+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.406466+0000) 2020-02-05T20:56:08.410 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:08.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:08.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:08.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:08.410112+0000) 2020-02-05T20:56:09.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:09.337+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:09.337436+0000) 2020-02-05T20:56:09.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:09.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:09.946528+0000) 2020-02-05T20:56:10.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:10.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:10.196723+0000) 2020-02-05T20:56:10.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:10.205+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:10.207038+0000) 2020-02-05T20:56:12.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:12.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:12.185516+0000) 2020-02-05T20:56:12.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:12.300+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:12.300678+0000) 2020-02-05T20:56:13.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:13.413+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.414496+0000) 2020-02-05T20:56:13.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:13.416+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.417114+0000) 2020-02-05T20:56:13.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:13.417+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.418087+0000) 2020-02-05T20:56:13.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:13.418+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.419071+0000) 2020-02-05T20:56:13.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:13.419+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.419996+0000) 2020-02-05T20:56:13.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:13.419+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.419994+0000) 2020-02-05T20:56:13.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:13.420+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.421261+0000) 2020-02-05T20:56:13.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:13.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.415275+0000) 2020-02-05T20:56:13.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:13.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.417507+0000) 2020-02-05T20:56:13.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:13.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.423141+0000) 2020-02-05T20:56:13.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:13.415+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.416472+0000) 2020-02-05T20:56:13.427 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:13.417+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.418639+0000) 2020-02-05T20:56:13.427 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:13.418+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.419116+0000) 2020-02-05T20:56:13.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:13.418+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.419707+0000) 2020-02-05T20:56:13.428 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:13.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71111 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:13.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:13.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.427381+0000) 2020-02-05T20:56:13.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:13.456+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:13.457582+0000) 2020-02-05T20:56:15.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:15.478+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:15.479644+0000) 2020-02-05T20:56:15.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:15.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:15.802283+0000) 2020-02-05T20:56:15.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:15.838+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:15.839137+0000) 2020-02-05T20:56:16.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:16.698+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:16.699295+0000) 2020-02-05T20:56:16.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:16.941+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:16.942728+0000) 2020-02-05T20:56:16.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:16.941+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:16.943116+0000) 2020-02-05T20:56:18.432 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:18.430+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.431287+0000) 2020-02-05T20:56:18.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:18.430+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.431791+0000) 2020-02-05T20:56:18.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:18.431+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.432409+0000) 2020-02-05T20:56:18.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:18.431+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.433279+0000) 2020-02-05T20:56:18.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:18.433+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.434304+0000) 2020-02-05T20:56:18.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:18.434+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.435322+0000) 2020-02-05T20:56:18.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:18.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.435384+0000) 2020-02-05T20:56:18.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:18.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.438958+0000) 2020-02-05T20:56:18.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:18.434+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.435699+0000) 2020-02-05T20:56:18.443 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:18.435+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.436455+0000) 2020-02-05T20:56:18.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:18.432+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.433913+0000) 2020-02-05T20:56:18.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:18.433+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.434836+0000) 2020-02-05T20:56:18.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:18.434+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.436336+0000) 2020-02-05T20:56:18.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:18.435+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:18.436751+0000) 2020-02-05T20:56:18.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:18.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71153 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:19.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:19.336+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:19.337645+0000) 2020-02-05T20:56:19.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:19.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:19.946759+0000) 2020-02-05T20:56:20.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:20.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:20.196915+0000) 2020-02-05T20:56:20.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:20.206+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:20.207321+0000) 2020-02-05T20:56:22.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:22.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:22.185811+0000) 2020-02-05T20:56:22.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:22.299+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:22.300938+0000) 2020-02-05T20:56:23.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:23.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.427590+0000) 2020-02-05T20:56:23.445 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:23.443+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.444459+0000) 2020-02-05T20:56:23.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:23.444+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.445570+0000) 2020-02-05T20:56:23.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:23.445+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.446855+0000) 2020-02-05T20:56:23.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:23.444+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.446228+0000) 2020-02-05T20:56:23.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:23.446+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.447572+0000) 2020-02-05T20:56:23.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:23.446+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.448047+0000) 2020-02-05T20:56:23.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:23.447+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.448540+0000) 2020-02-05T20:56:23.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:23.447+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.448961+0000) 2020-02-05T20:56:23.450 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:23.448+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.449511+0000) 2020-02-05T20:56:23.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:23.448+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.449703+0000) 2020-02-05T20:56:23.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:23.448+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.450260+0000) 2020-02-05T20:56:23.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:23.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.451157+0000) 2020-02-05T20:56:23.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:23.448+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.450279+0000) 2020-02-05T20:56:23.454 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:23.452+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:23.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:23.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.457642+0000) 2020-02-05T20:56:23.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:23.456+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:23.457743+0000) 2020-02-05T20:56:25.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:25.488+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:25.489543+0000) 2020-02-05T20:56:25.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:25.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:25.802517+0000) 2020-02-05T20:56:25.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:25.838+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:25.839376+0000) 2020-02-05T20:56:26.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:26.698+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:26.699575+0000) 2020-02-05T20:56:26.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:26.941+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:26.942999+0000) 2020-02-05T20:56:26.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:26.941+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:26.943317+0000) 2020-02-05T20:56:28.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:28.458+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.459063+0000) 2020-02-05T20:56:28.461 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:28.458+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.459645+0000) 2020-02-05T20:56:28.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:28.459+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.460730+0000) 2020-02-05T20:56:28.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:28.459+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.461252+0000) 2020-02-05T20:56:28.463 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:28.461+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.462224+0000) 2020-02-05T20:56:28.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:28.461+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.462487+0000) 2020-02-05T20:56:28.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:28.461+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.462663+0000) 2020-02-05T20:56:28.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:28.462+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.463570+0000) 2020-02-05T20:56:28.465 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:28.462+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.463887+0000) 2020-02-05T20:56:28.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:28.463+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.464683+0000) 2020-02-05T20:56:28.466 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:28.463+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.464568+0000) 2020-02-05T20:56:28.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:28.463+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.465161+0000) 2020-02-05T20:56:28.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:28.467+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.468371+0000) 2020-02-05T20:56:28.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:28.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:28.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:28.468+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:28.469558+0000) 2020-02-05T20:56:29.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:29.337+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:29.337866+0000) 2020-02-05T20:56:29.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:29.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:29.947019+0000) 2020-02-05T20:56:30.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:30.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:30.197138+0000) 2020-02-05T20:56:30.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:30.206+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:30.207543+0000) 2020-02-05T20:56:31.916 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:56:31.923 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:56:32.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:32.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:32.186094+0000) 2020-02-05T20:56:32.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:32.299+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:32.301149+0000) 2020-02-05T20:56:33.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:33.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.427845+0000) 2020-02-05T20:56:33.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:33.457+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.457945+0000) 2020-02-05T20:56:33.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:33.474+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.475245+0000) 2020-02-05T20:56:33.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:33.475+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.476257+0000) 2020-02-05T20:56:33.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:33.474+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.475754+0000) 2020-02-05T20:56:33.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:33.475+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.476874+0000) 2020-02-05T20:56:33.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:33.475+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.477257+0000) 2020-02-05T20:56:33.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:33.476+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.478147+0000) 2020-02-05T20:56:33.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:33.477+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.478831+0000) 2020-02-05T20:56:33.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:33.476+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.478220+0000) 2020-02-05T20:56:33.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:33.477+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.478689+0000) 2020-02-05T20:56:33.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:33.477+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.479350+0000) 2020-02-05T20:56:33.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:33.478+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.479659+0000) 2020-02-05T20:56:33.482 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:33.479+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.480423+0000) 2020-02-05T20:56:33.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:33.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.480790+0000) 2020-02-05T20:56:33.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:33.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:33.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:33.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:33.485727+0000) 2020-02-05T20:56:35.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:35.497+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:35.497649+0000) 2020-02-05T20:56:35.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:35.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:35.802731+0000) 2020-02-05T20:56:35.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:35.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:35.839608+0000) 2020-02-05T20:56:36.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:36.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:36.699801+0000) 2020-02-05T20:56:36.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:36.942+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:36.943333+0000) 2020-02-05T20:56:36.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:36.942+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:36.943599+0000) 2020-02-05T20:56:38.491 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:38.488+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.489972+0000) 2020-02-05T20:56:38.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:38.489+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.490604+0000) 2020-02-05T20:56:38.492 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:38.490+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.491519+0000) 2020-02-05T20:56:38.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:38.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.492043+0000) 2020-02-05T20:56:38.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:38.491+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.492529+0000) 2020-02-05T20:56:38.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:38.491+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.493239+0000) 2020-02-05T20:56:38.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:38.491+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.493201+0000) 2020-02-05T20:56:38.495 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:38.492+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.493697+0000) 2020-02-05T20:56:38.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:38.493+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.494564+0000) 2020-02-05T20:56:38.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:38.493+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.494713+0000) 2020-02-05T20:56:38.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:38.493+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.494709+0000) 2020-02-05T20:56:38.496 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:38.493+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.495184+0000) 2020-02-05T20:56:38.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:38.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:38.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:38.499+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.500241+0000) 2020-02-05T20:56:38.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:38.499+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:38.500616+0000) 2020-02-05T20:56:39.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:39.337+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:39.338071+0000) 2020-02-05T20:56:39.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:39.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:39.947301+0000) 2020-02-05T20:56:40.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:40.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:40.197396+0000) 2020-02-05T20:56:40.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:40.207+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:40.207768+0000) 2020-02-05T20:56:42.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:42.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:42.186394+0000) 2020-02-05T20:56:42.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:42.300+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:42.301394+0000) 2020-02-05T20:56:43.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:43.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.428083+0000) 2020-02-05T20:56:43.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:43.457+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.458174+0000) 2020-02-05T20:56:43.504 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:43.502+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.503351+0000) 2020-02-05T20:56:43.505 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:43.502+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.503946+0000) 2020-02-05T20:56:43.506 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:43.503+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.504609+0000) 2020-02-05T20:56:43.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:43.504+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.505734+0000) 2020-02-05T20:56:43.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:43.505+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.506403+0000) 2020-02-05T20:56:43.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:43.505+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.506982+0000) 2020-02-05T20:56:43.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:43.506+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.507403+0000) 2020-02-05T20:56:43.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:43.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.508032+0000) 2020-02-05T20:56:43.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:43.507+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.508579+0000) 2020-02-05T20:56:43.510 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:43.508+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.509698+0000) 2020-02-05T20:56:43.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:43.508+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.509840+0000) 2020-02-05T20:56:43.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:43.508+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.510048+0000) 2020-02-05T20:56:43.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:43.509+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.510255+0000) 2020-02-05T20:56:43.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:43.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:43.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:43.513+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:43.514669+0000) 2020-02-05T20:56:45.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:45.505+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:45.505671+0000) 2020-02-05T20:56:45.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:45.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:45.802958+0000) 2020-02-05T20:56:45.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:45.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:45.839859+0000) 2020-02-05T20:56:46.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:46.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:46.700030+0000) 2020-02-05T20:56:46.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:46.942+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:46.943591+0000) 2020-02-05T20:56:46.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:46.942+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:46.943856+0000) 2020-02-05T20:56:48.520 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:48.517+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.519053+0000) 2020-02-05T20:56:48.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:48.518+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.519205+0000) 2020-02-05T20:56:48.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:48.518+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.519874+0000) 2020-02-05T20:56:48.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:48.519+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.520731+0000) 2020-02-05T20:56:48.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:48.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.521018+0000) 2020-02-05T20:56:48.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:48.521+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.522731+0000) 2020-02-05T20:56:48.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:48.522+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.523134+0000) 2020-02-05T20:56:48.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:48.523+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.524284+0000) 2020-02-05T20:56:48.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:48.523+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.524685+0000) 2020-02-05T20:56:48.526 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:48.524+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.525191+0000) 2020-02-05T20:56:48.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:48.523+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.525092+0000) 2020-02-05T20:56:48.527 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:48.524+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.525529+0000) 2020-02-05T20:56:48.528 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:48.524+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.525939+0000) 2020-02-05T20:56:48.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:48.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:48.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:48.530+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:48.531493+0000) 2020-02-05T20:56:49.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:49.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:49.338391+0000) 2020-02-05T20:56:49.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:49.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:49.947540+0000) 2020-02-05T20:56:50.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:50.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:50.197660+0000) 2020-02-05T20:56:50.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:50.207+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:50.207994+0000) 2020-02-05T20:56:52.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:52.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:52.186723+0000) 2020-02-05T20:56:52.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:52.301+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:52.301661+0000) 2020-02-05T20:56:53.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:53.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.428332+0000) 2020-02-05T20:56:53.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:53.457+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.458363+0000) 2020-02-05T20:56:53.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:53.532+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.534040+0000) 2020-02-05T20:56:53.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:53.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.534785+0000) 2020-02-05T20:56:53.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:53.533+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.535296+0000) 2020-02-05T20:56:53.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:53.534+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.535958+0000) 2020-02-05T20:56:53.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:53.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.536361+0000) 2020-02-05T20:56:53.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:53.536+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.537148+0000) 2020-02-05T20:56:53.538 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:53.536+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.537602+0000) 2020-02-05T20:56:53.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:53.537+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.538731+0000) 2020-02-05T20:56:53.540 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:53.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.539021+0000) 2020-02-05T20:56:53.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:53.537+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.539124+0000) 2020-02-05T20:56:53.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:53.538+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.540149+0000) 2020-02-05T20:56:53.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:53.539+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.540610+0000) 2020-02-05T20:56:53.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:53.539+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.540684+0000) 2020-02-05T20:56:53.545 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:53.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71444 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:53.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:53.543+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:53.544510+0000) 2020-02-05T20:56:55.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:55.513+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:55.513860+0000) 2020-02-05T20:56:55.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:55.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:55.803195+0000) 2020-02-05T20:56:55.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:55.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:55.840139+0000) 2020-02-05T20:56:56.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:56.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:56.700298+0000) 2020-02-05T20:56:56.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:56.943+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:56.943839+0000) 2020-02-05T20:56:56.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:56.943+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:56.944113+0000) 2020-02-05T20:56:58.550 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:56:58.548+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.549538+0000) 2020-02-05T20:56:58.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:56:58.548+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.550041+0000) 2020-02-05T20:56:58.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:56:58.548+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.550291+0000) 2020-02-05T20:56:58.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:58.549+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.550827+0000) 2020-02-05T20:56:58.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:58.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.551789+0000) 2020-02-05T20:56:58.553 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:56:58.551+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.552311+0000) 2020-02-05T20:56:58.554 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:56:58.551+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.552956+0000) 2020-02-05T20:56:58.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:56:58.552+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.553867+0000) 2020-02-05T20:56:58.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:56:58.553+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.554576+0000) 2020-02-05T20:56:58.556 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:56:58.553+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.554656+0000) 2020-02-05T20:56:58.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:56:58.553+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.555052+0000) 2020-02-05T20:56:58.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:58.555+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.556271+0000) 2020-02-05T20:56:58.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:56:58.553+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.555372+0000) 2020-02-05T20:56:58.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:56:58.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71485 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:56:58.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:58.559+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:58.560598+0000) 2020-02-05T20:56:59.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:56:59.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:59.338607+0000) 2020-02-05T20:56:59.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:56:59.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:56:59.947791+0000) 2020-02-05T20:57:00.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:00.197+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:00.197914+0000) 2020-02-05T20:57:00.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:00.208+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:00.208253+0000) 2020-02-05T20:57:02.062 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:57:02.068 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:57:02.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:02.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:02.187010+0000) 2020-02-05T20:57:02.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:02.301+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:02.301918+0000) 2020-02-05T20:57:03.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:03.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.428590+0000) 2020-02-05T20:57:03.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:03.458+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.458593+0000) 2020-02-05T20:57:03.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:03.565+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.565181+0000) 2020-02-05T20:57:03.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:03.564+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.565784+0000) 2020-02-05T20:57:03.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:03.564+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.565926+0000) 2020-02-05T20:57:03.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:03.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.566921+0000) 2020-02-05T20:57:03.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:03.565+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.566941+0000) 2020-02-05T20:57:03.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:03.565+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.567232+0000) 2020-02-05T20:57:03.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:03.566+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.567841+0000) 2020-02-05T20:57:03.570 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:03.568+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.568873+0000) 2020-02-05T20:57:03.570 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:03.569+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.569364+0000) 2020-02-05T20:57:03.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:03.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.569435+0000) 2020-02-05T20:57:03.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:03.569+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.569572+0000) 2020-02-05T20:57:03.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:03.568+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.569901+0000) 2020-02-05T20:57:03.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:03.570+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.570119+0000) 2020-02-05T20:57:03.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:03.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71526 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:03.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:03.576+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:03.576943+0000) 2020-02-05T20:57:05.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:05.521+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:05.522412+0000) 2020-02-05T20:57:05.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:05.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:05.803431+0000) 2020-02-05T20:57:05.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:05.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:05.840368+0000) 2020-02-05T20:57:06.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:06.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:06.700537+0000) 2020-02-05T20:57:06.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:06.943+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:06.944077+0000) 2020-02-05T20:57:06.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:06.943+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:06.944359+0000) 2020-02-05T20:57:08.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:08.577+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.578746+0000) 2020-02-05T20:57:08.580 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:08.578+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.579566+0000) 2020-02-05T20:57:08.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:08.580+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.581069+0000) 2020-02-05T20:57:08.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:08.580+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.581959+0000) 2020-02-05T20:57:08.583 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:08.582+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.582489+0000) 2020-02-05T20:57:08.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:08.582+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.583089+0000) 2020-02-05T20:57:08.585 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:08.582+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.583395+0000) 2020-02-05T20:57:08.585 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:08.583+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.583556+0000) 2020-02-05T20:57:08.585 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:08.583+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.584274+0000) 2020-02-05T20:57:08.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:08.583+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.584247+0000) 2020-02-05T20:57:08.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:08.584+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.584891+0000) 2020-02-05T20:57:08.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:08.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.584756+0000) 2020-02-05T20:57:08.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:08.584+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.585187+0000) 2020-02-05T20:57:08.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:08.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:08.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:08.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:08.590488+0000) 2020-02-05T20:57:09.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:09.337+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:09.338818+0000) 2020-02-05T20:57:09.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:09.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:09.948042+0000) 2020-02-05T20:57:10.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:10.197+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:10.198162+0000) 2020-02-05T20:57:10.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:10.207+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:10.208494+0000) 2020-02-05T20:57:12.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:12.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:12.187304+0000) 2020-02-05T20:57:12.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:12.301+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:12.302136+0000) 2020-02-05T20:57:13.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:13.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.428783+0000) 2020-02-05T20:57:13.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:13.457+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.458829+0000) 2020-02-05T20:57:13.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:13.592+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.593350+0000) 2020-02-05T20:57:13.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:13.593+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.594349+0000) 2020-02-05T20:57:13.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:13.593+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.594815+0000) 2020-02-05T20:57:13.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:13.594+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.595342+0000) 2020-02-05T20:57:13.597 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:13.595+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.596251+0000) 2020-02-05T20:57:13.598 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:13.596+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.596678+0000) 2020-02-05T20:57:13.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:13.596+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.596880+0000) 2020-02-05T20:57:13.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:13.597+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.597663+0000) 2020-02-05T20:57:13.599 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:13.597+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.598622+0000) 2020-02-05T20:57:13.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:13.598+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.599245+0000) 2020-02-05T20:57:13.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:13.599+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.599457+0000) 2020-02-05T20:57:13.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:13.598+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.599754+0000) 2020-02-05T20:57:13.601 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:13.599+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.600192+0000) 2020-02-05T20:57:13.604 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:13.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:13.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:13.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:13.608007+0000) 2020-02-05T20:57:15.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:15.533+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:15.534148+0000) 2020-02-05T20:57:15.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:15.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:15.803682+0000) 2020-02-05T20:57:15.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:15.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:15.840620+0000) 2020-02-05T20:57:16.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:16.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:16.700789+0000) 2020-02-05T20:57:16.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:16.943+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:16.944326+0000) 2020-02-05T20:57:16.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:16.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:16.944633+0000) 2020-02-05T20:57:18.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:18.606+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.607213+0000) 2020-02-05T20:57:18.609 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:18.607+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.608262+0000) 2020-02-05T20:57:18.610 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:18.607+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.609054+0000) 2020-02-05T20:57:18.611 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:18.608+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.610043+0000) 2020-02-05T20:57:18.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:18.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.610780+0000) 2020-02-05T20:57:18.613 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:18.611+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.611629+0000) 2020-02-05T20:57:18.613 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:18.610+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.611388+0000) 2020-02-05T20:57:18.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:18.611+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.612945+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:18.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.619104+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:18.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.619587+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:18.612+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.612557+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:18.612+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.612698+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:18.613+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.614117+0000) 2020-02-05T20:57:18.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:18.613+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:18.614234+0000) 2020-02-05T20:57:18.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:18.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:19.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:19.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:19.339033+0000) 2020-02-05T20:57:19.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:19.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:19.948319+0000) 2020-02-05T20:57:20.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:20.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:20.198414+0000) 2020-02-05T20:57:20.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:20.207+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:20.208719+0000) 2020-02-05T20:57:22.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:22.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:22.187565+0000) 2020-02-05T20:57:22.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:22.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:22.302411+0000) 2020-02-05T20:57:23.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:23.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.429042+0000) 2020-02-05T20:57:23.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:23.458+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.459150+0000) 2020-02-05T20:57:23.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:23.624+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.625340+0000) 2020-02-05T20:57:23.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:23.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.625791+0000) 2020-02-05T20:57:23.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:23.625+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.626176+0000) 2020-02-05T20:57:23.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:23.627+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.627599+0000) 2020-02-05T20:57:23.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:23.627+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.627661+0000) 2020-02-05T20:57:23.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:23.627+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.628678+0000) 2020-02-05T20:57:23.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:23.629+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.629520+0000) 2020-02-05T20:57:23.633 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:23.628+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.629137+0000) 2020-02-05T20:57:23.633 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:23.629+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.630223+0000) 2020-02-05T20:57:23.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:23.630+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.630708+0000) 2020-02-05T20:57:23.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:23.629+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.630109+0000) 2020-02-05T20:57:23.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:23.629+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.630232+0000) 2020-02-05T20:57:23.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:23.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:23.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:23.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.632709+0000) 2020-02-05T20:57:23.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:23.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:23.632944+0000) 2020-02-05T20:57:25.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:25.544+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:25.545475+0000) 2020-02-05T20:57:25.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:25.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:25.803954+0000) 2020-02-05T20:57:25.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:25.840+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:25.840854+0000) 2020-02-05T20:57:26.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:26.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:26.701021+0000) 2020-02-05T20:57:26.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:26.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:26.944571+0000) 2020-02-05T20:57:26.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:26.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:26.944815+0000) 2020-02-05T20:57:28.640 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:28.637+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.638827+0000) 2020-02-05T20:57:28.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:28.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.639500+0000) 2020-02-05T20:57:28.641 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:28.639+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.640209+0000) 2020-02-05T20:57:28.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:28.639+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.640619+0000) 2020-02-05T20:57:28.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:28.640+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.641095+0000) 2020-02-05T20:57:28.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:28.641+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.641686+0000) 2020-02-05T20:57:28.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:28.640+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.642046+0000) 2020-02-05T20:57:28.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:28.641+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.642344+0000) 2020-02-05T20:57:28.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:28.642+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.642938+0000) 2020-02-05T20:57:28.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:28.642+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.643454+0000) 2020-02-05T20:57:28.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:28.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.644460+0000) 2020-02-05T20:57:28.646 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:28.644+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.644612+0000) 2020-02-05T20:57:28.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:28.644+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.644848+0000) 2020-02-05T20:57:28.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:28.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:28.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:28.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:28.649800+0000) 2020-02-05T20:57:29.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:29.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:29.339329+0000) 2020-02-05T20:57:29.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:29.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:29.948581+0000) 2020-02-05T20:57:30.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:30.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:30.198692+0000) 2020-02-05T20:57:30.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:30.208+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:30.208926+0000) 2020-02-05T20:57:32.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:32.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:32.187806+0000) 2020-02-05T20:57:32.201 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:57:32.207 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:57:32.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:32.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:32.302671+0000) 2020-02-05T20:57:33.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:33.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.429211+0000) 2020-02-05T20:57:33.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:33.458+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.459392+0000) 2020-02-05T20:57:33.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:33.653+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.654433+0000) 2020-02-05T20:57:33.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:33.654+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.655116+0000) 2020-02-05T20:57:33.658 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:33.655+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.656150+0000) 2020-02-05T20:57:33.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:33.656+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.656788+0000) 2020-02-05T20:57:33.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:33.657+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.658305+0000) 2020-02-05T20:57:33.659 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:33.657+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.658130+0000) 2020-02-05T20:57:33.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:33.657+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.658872+0000) 2020-02-05T20:57:33.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:33.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.659254+0000) 2020-02-05T20:57:33.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:33.658+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.659550+0000) 2020-02-05T20:57:33.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:33.659+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.659678+0000) 2020-02-05T20:57:33.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:33.659+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.660487+0000) 2020-02-05T20:57:33.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:33.660+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.660584+0000) 2020-02-05T20:57:33.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:33.660+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.660731+0000) 2020-02-05T20:57:33.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:33.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:33.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:33.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:33.664986+0000) 2020-02-05T20:57:35.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:35.553+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:35.554043+0000) 2020-02-05T20:57:35.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:35.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:35.804179+0000) 2020-02-05T20:57:35.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:35.840+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:35.841114+0000) 2020-02-05T20:57:36.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:36.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:36.701282+0000) 2020-02-05T20:57:36.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:36.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:36.944861+0000) 2020-02-05T20:57:36.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:36.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:36.945060+0000) 2020-02-05T20:57:38.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:38.668+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.669330+0000) 2020-02-05T20:57:38.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:38.668+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.670032+0000) 2020-02-05T20:57:38.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:38.669+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.670751+0000) 2020-02-05T20:57:38.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:38.670+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.671278+0000) 2020-02-05T20:57:38.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:38.670+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.671367+0000) 2020-02-05T20:57:38.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:38.670+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.671855+0000) 2020-02-05T20:57:38.674 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:38.671+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.672182+0000) 2020-02-05T20:57:38.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:38.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.672709+0000) 2020-02-05T20:57:38.674 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:38.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.673584+0000) 2020-02-05T20:57:38.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:38.673+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.673460+0000) 2020-02-05T20:57:38.675 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:38.673+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.674150+0000) 2020-02-05T20:57:38.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:38.673+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.674142+0000) 2020-02-05T20:57:38.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:38.674+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.674600+0000) 2020-02-05T20:57:38.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:38.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:38.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:38.679+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:38.680122+0000) 2020-02-05T20:57:39.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:39.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:39.339514+0000) 2020-02-05T20:57:39.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:39.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:39.948805+0000) 2020-02-05T20:57:40.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:40.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:40.198923+0000) 2020-02-05T20:57:40.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:40.208+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:40.209156+0000) 2020-02-05T20:57:42.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:42.188+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:42.188153+0000) 2020-02-05T20:57:42.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:42.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:42.302936+0000) 2020-02-05T20:57:43.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:43.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.429474+0000) 2020-02-05T20:57:43.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:43.459+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.459634+0000) 2020-02-05T20:57:43.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:43.682+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.683681+0000) 2020-02-05T20:57:43.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:43.684+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.685071+0000) 2020-02-05T20:57:43.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:43.683+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.685352+0000) 2020-02-05T20:57:43.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:43.684+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.686083+0000) 2020-02-05T20:57:43.688 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:43.685+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.686757+0000) 2020-02-05T20:57:43.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:43.686+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.687137+0000) 2020-02-05T20:57:43.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:43.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.687503+0000) 2020-02-05T20:57:43.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:43.686+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.687619+0000) 2020-02-05T20:57:43.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:43.686+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.688262+0000) 2020-02-05T20:57:43.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:43.687+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.688773+0000) 2020-02-05T20:57:43.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:43.688+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.689195+0000) 2020-02-05T20:57:43.690 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:43.688+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.689513+0000) 2020-02-05T20:57:43.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:43.688+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.690056+0000) 2020-02-05T20:57:43.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:43.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71858 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:43.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:43.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:43.696267+0000) 2020-02-05T20:57:45.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:45.560+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:45.561136+0000) 2020-02-05T20:57:45.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:45.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:45.804404+0000) 2020-02-05T20:57:45.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:45.841+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:45.841384+0000) 2020-02-05T20:57:46.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:46.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:46.701511+0000) 2020-02-05T20:57:46.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:46.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:46.945158+0000) 2020-02-05T20:57:46.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:46.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:46.945343+0000) 2020-02-05T20:57:48.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:48.696+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.697870+0000) 2020-02-05T20:57:48.699 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:48.697+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.698473+0000) 2020-02-05T20:57:48.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:48.697+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.699013+0000) 2020-02-05T20:57:48.701 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:48.698+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.699535+0000) 2020-02-05T20:57:48.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:48.699+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.701196+0000) 2020-02-05T20:57:48.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:48.700+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.701775+0000) 2020-02-05T20:57:48.703 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:48.699+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.701347+0000) 2020-02-05T20:57:48.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:48.701+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.702360+0000) 2020-02-05T20:57:48.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:48.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.702756+0000) 2020-02-05T20:57:48.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:48.702+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.703133+0000) 2020-02-05T20:57:48.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:48.701+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.703005+0000) 2020-02-05T20:57:48.705 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:48.701+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.703297+0000) 2020-02-05T20:57:48.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:48.703+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.704520+0000) 2020-02-05T20:57:48.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:48.707+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71899 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:48.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:48.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:48.711294+0000) 2020-02-05T20:57:49.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:49.339+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:49.339724+0000) 2020-02-05T20:57:49.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:49.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:49.949037+0000) 2020-02-05T20:57:50.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:50.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:50.199175+0000) 2020-02-05T20:57:50.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:50.209+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:50.209388+0000) 2020-02-05T20:57:52.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:52.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:52.188396+0000) 2020-02-05T20:57:52.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:52.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:52.303174+0000) 2020-02-05T20:57:53.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:53.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.429726+0000) 2020-02-05T20:57:53.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:53.459+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.459871+0000) 2020-02-05T20:57:53.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:53.714+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.715805+0000) 2020-02-05T20:57:53.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:53.715+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.716101+0000) 2020-02-05T20:57:53.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:53.715+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.716476+0000) 2020-02-05T20:57:53.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:53.715+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.717049+0000) 2020-02-05T20:57:53.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:53.715+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.716763+0000) 2020-02-05T20:57:53.719 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:53.716+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.717518+0000) 2020-02-05T20:57:53.719 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:53.716+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.717425+0000) 2020-02-05T20:57:53.720 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:53.716+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.717867+0000) 2020-02-05T20:57:53.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:53.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.718419+0000) 2020-02-05T20:57:53.721 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:53.717+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.719267+0000) 2020-02-05T20:57:53.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:53.718+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.719537+0000) 2020-02-05T20:57:53.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:53.718+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.719856+0000) 2020-02-05T20:57:53.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:53.719+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.720528+0000) 2020-02-05T20:57:53.725 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:53.722+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71941 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:53.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:53.724+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:53.725736+0000) 2020-02-05T20:57:55.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:55.568+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:55.569052+0000) 2020-02-05T20:57:55.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:55.804+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:55.804582+0000) 2020-02-05T20:57:55.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:55.841+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:55.841609+0000) 2020-02-05T20:57:56.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:56.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:56.701747+0000) 2020-02-05T20:57:56.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:56.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:56.945417+0000) 2020-02-05T20:57:56.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:56.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:56.945654+0000) 2020-02-05T20:57:58.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:57:58.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.730868+0000) 2020-02-05T20:57:58.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:57:58.731+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.731462+0000) 2020-02-05T20:57:58.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:57:58.731+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.731731+0000) 2020-02-05T20:57:58.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:57:58.731+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.732734+0000) 2020-02-05T20:57:58.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:58.733+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.733250+0000) 2020-02-05T20:57:58.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:57:58.732+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.733563+0000) 2020-02-05T20:57:58.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:57:58.732+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.734010+0000) 2020-02-05T20:57:58.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:58.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.734669+0000) 2020-02-05T20:57:58.736 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:57:58.735+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.735146+0000) 2020-02-05T20:57:58.746 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:57:58.734+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.736225+0000) 2020-02-05T20:57:58.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:57:58.734+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.736271+0000) 2020-02-05T20:57:58.746 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:58.736+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.736063+0000) 2020-02-05T20:57:58.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:57:58.736+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.736118+0000) 2020-02-05T20:57:58.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:57:58.737+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 71981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:57:58.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:58.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:58.745169+0000) 2020-02-05T20:57:59.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:57:59.338+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:59.339948+0000) 2020-02-05T20:57:59.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:57:59.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:57:59.949309+0000) 2020-02-05T20:58:00.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:00.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:00.199411+0000) 2020-02-05T20:58:00.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:00.208+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:00.209615+0000) 2020-02-05T20:58:02.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:02.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:02.188698+0000) 2020-02-05T20:58:02.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:02.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:02.303398+0000) 2020-02-05T20:58:02.339 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:58:02.345 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:58:03.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:03.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.429908+0000) 2020-02-05T20:58:03.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:03.459+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.460146+0000) 2020-02-05T20:58:03.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:03.743+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.745037+0000) 2020-02-05T20:58:03.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:03.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.746067+0000) 2020-02-05T20:58:03.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:03.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.746653+0000) 2020-02-05T20:58:03.748 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:03.745+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.746993+0000) 2020-02-05T20:58:03.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:03.746+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.747882+0000) 2020-02-05T20:58:03.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:03.747+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.748518+0000) 2020-02-05T20:58:03.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:03.748+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.749180+0000) 2020-02-05T20:58:03.756 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:03.751+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.752235+0000) 2020-02-05T20:58:03.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:03.748+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.749674+0000) 2020-02-05T20:58:03.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:03.749+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.750591+0000) 2020-02-05T20:58:03.758 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:03.749+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.750703+0000) 2020-02-05T20:58:03.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:03.750+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.752175+0000) 2020-02-05T20:58:03.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:03.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:03.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:03.751+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.753267+0000) 2020-02-05T20:58:03.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:03.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:03.760514+0000) 2020-02-05T20:58:05.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:05.576+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:05.577266+0000) 2020-02-05T20:58:05.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:05.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:05.804814+0000) 2020-02-05T20:58:05.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:05.840+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:05.841838+0000) 2020-02-05T20:58:06.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:06.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:06.701976+0000) 2020-02-05T20:58:06.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:06.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:06.945751+0000) 2020-02-05T20:58:06.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:06.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:06.945900+0000) 2020-02-05T20:58:08.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:08.756+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.757781+0000) 2020-02-05T20:58:08.760 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:08.758+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.759483+0000) 2020-02-05T20:58:08.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:08.759+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.760360+0000) 2020-02-05T20:58:08.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:08.759+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.760204+0000) 2020-02-05T20:58:08.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:08.759+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.760701+0000) 2020-02-05T20:58:08.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:08.759+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.760926+0000) 2020-02-05T20:58:08.763 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:08.759+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.761260+0000) 2020-02-05T20:58:08.763 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:08.760+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.761480+0000) 2020-02-05T20:58:08.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:08.759+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.760918+0000) 2020-02-05T20:58:08.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:08.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.762188+0000) 2020-02-05T20:58:08.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:08.761+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.762674+0000) 2020-02-05T20:58:08.765 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:08.761+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.762903+0000) 2020-02-05T20:58:08.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:08.761+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.763316+0000) 2020-02-05T20:58:08.768 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:08.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:08.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:08.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:08.768023+0000) 2020-02-05T20:58:09.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:09.339+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:09.340162+0000) 2020-02-05T20:58:09.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:09.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:09.949546+0000) 2020-02-05T20:58:10.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:10.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:10.199680+0000) 2020-02-05T20:58:10.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:10.208+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:10.209847+0000) 2020-02-05T20:58:12.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:12.188+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:12.188955+0000) 2020-02-05T20:58:12.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:12.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:12.303665+0000) 2020-02-05T20:58:13.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:13.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.430146+0000) 2020-02-05T20:58:13.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:13.459+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.460370+0000) 2020-02-05T20:58:13.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:13.770+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.771883+0000) 2020-02-05T20:58:13.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:13.771+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.772484+0000) 2020-02-05T20:58:13.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:13.771+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.772893+0000) 2020-02-05T20:58:13.774 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:13.772+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.773357+0000) 2020-02-05T20:58:13.776 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:13.773+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.774647+0000) 2020-02-05T20:58:13.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:13.774+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.775736+0000) 2020-02-05T20:58:13.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:13.774+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.775845+0000) 2020-02-05T20:58:13.777 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:13.774+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.775981+0000) 2020-02-05T20:58:13.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:13.774+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.776233+0000) 2020-02-05T20:58:13.778 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:13.775+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.776940+0000) 2020-02-05T20:58:13.778 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:13.775+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.777062+0000) 2020-02-05T20:58:13.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:13.776+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.777949+0000) 2020-02-05T20:58:13.779 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:13.776+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.778159+0000) 2020-02-05T20:58:13.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:13.780+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:13.781183+0000) 2020-02-05T20:58:13.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:13.780+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:15.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:15.586+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:15.587612+0000) 2020-02-05T20:58:15.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:15.804+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:15.805059+0000) 2020-02-05T20:58:15.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:15.841+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:15.842062+0000) 2020-02-05T20:58:16.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:16.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:16.702209+0000) 2020-02-05T20:58:16.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:16.944+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:16.945997+0000) 2020-02-05T20:58:16.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:16.944+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:16.946160+0000) 2020-02-05T20:58:18.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:18.786+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.788020+0000) 2020-02-05T20:58:18.790 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:18.787+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.788497+0000) 2020-02-05T20:58:18.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:18.787+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.789212+0000) 2020-02-05T20:58:18.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:18.789+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.790357+0000) 2020-02-05T20:58:18.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:18.790+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.791668+0000) 2020-02-05T20:58:18.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:18.790+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.791981+0000) 2020-02-05T20:58:18.794 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:18.790+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.792315+0000) 2020-02-05T20:58:18.795 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:18.791+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.792758+0000) 2020-02-05T20:58:18.795 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:18.791+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.793095+0000) 2020-02-05T20:58:18.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:18.791+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.792765+0000) 2020-02-05T20:58:18.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:18.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.793991+0000) 2020-02-05T20:58:18.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:18.792+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.793871+0000) 2020-02-05T20:58:18.797 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:18.793+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.794148+0000) 2020-02-05T20:58:18.798 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:18.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72145 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:18.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:18.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:18.798232+0000) 2020-02-05T20:58:19.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:19.339+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:19.340366+0000) 2020-02-05T20:58:19.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:19.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:19.949779+0000) 2020-02-05T20:58:20.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:20.199+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:20.199938+0000) 2020-02-05T20:58:20.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:20.209+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:20.210126+0000) 2020-02-05T20:58:22.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:22.188+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:22.189216+0000) 2020-02-05T20:58:22.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:22.302+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:22.303914+0000) 2020-02-05T20:58:23.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:23.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.430403+0000) 2020-02-05T20:58:23.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:23.459+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.460572+0000) 2020-02-05T20:58:23.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:23.800+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.801836+0000) 2020-02-05T20:58:23.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:23.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.803223+0000) 2020-02-05T20:58:23.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:23.802+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.803844+0000) 2020-02-05T20:58:23.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:23.803+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.804566+0000) 2020-02-05T20:58:23.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:23.803+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.805067+0000) 2020-02-05T20:58:23.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:23.804+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.805591+0000) 2020-02-05T20:58:23.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:23.805+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.806280+0000) 2020-02-05T20:58:23.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:23.805+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.806659+0000) 2020-02-05T20:58:23.808 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:23.805+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.807253+0000) 2020-02-05T20:58:23.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:23.806+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.807623+0000) 2020-02-05T20:58:23.809 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:23.806+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.808158+0000) 2020-02-05T20:58:23.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:23.806+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.808315+0000) 2020-02-05T20:58:23.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:23.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.809012+0000) 2020-02-05T20:58:23.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:23.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:23.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:23.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:23.814915+0000) 2020-02-05T20:58:25.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:25.596+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:25.596989+0000) 2020-02-05T20:58:25.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:25.804+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:25.805370+0000) 2020-02-05T20:58:25.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:25.841+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:25.842362+0000) 2020-02-05T20:58:26.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:26.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:26.702434+0000) 2020-02-05T20:58:26.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:26.945+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:26.946232+0000) 2020-02-05T20:58:26.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:26.945+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:26.946473+0000) 2020-02-05T20:58:28.819 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:28.816+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.817976+0000) 2020-02-05T20:58:28.820 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:28.817+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.818307+0000) 2020-02-05T20:58:28.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:28.819+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.820267+0000) 2020-02-05T20:58:28.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:28.819+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.820917+0000) 2020-02-05T20:58:28.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:28.820+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.821512+0000) 2020-02-05T20:58:28.823 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:28.819+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.821408+0000) 2020-02-05T20:58:28.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:28.820+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.821877+0000) 2020-02-05T20:58:28.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:28.821+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.822639+0000) 2020-02-05T20:58:28.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:28.821+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.823184+0000) 2020-02-05T20:58:28.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:28.822+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.823789+0000) 2020-02-05T20:58:28.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:28.823+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.824315+0000) 2020-02-05T20:58:28.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:28.823+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.824550+0000) 2020-02-05T20:58:28.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:28.823+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.824818+0000) 2020-02-05T20:58:28.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:28.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:28.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:28.831+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:28.832967+0000) 2020-02-05T20:58:29.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:29.339+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:29.340546+0000) 2020-02-05T20:58:29.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:29.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:29.950027+0000) 2020-02-05T20:58:30.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:30.199+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:30.200176+0000) 2020-02-05T20:58:30.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:30.209+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:30.210315+0000) 2020-02-05T20:58:32.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:32.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:32.189551+0000) 2020-02-05T20:58:32.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:32.303+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:32.304158+0000) 2020-02-05T20:58:32.485 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:58:32.491 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:58:33.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:33.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.430625+0000) 2020-02-05T20:58:33.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:33.460+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.460807+0000) 2020-02-05T20:58:33.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:33.829+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.830978+0000) 2020-02-05T20:58:33.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:33.831+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.832773+0000) 2020-02-05T20:58:33.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:33.832+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.833241+0000) 2020-02-05T20:58:33.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:33.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.833908+0000) 2020-02-05T20:58:33.837 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:33.834+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.834731+0000) 2020-02-05T20:58:33.837 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:33.835+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.835543+0000) 2020-02-05T20:58:33.838 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:33.834+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.835337+0000) 2020-02-05T20:58:33.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:33.835+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.836841+0000) 2020-02-05T20:58:33.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:33.835+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.836334+0000) 2020-02-05T20:58:33.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:33.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.837192+0000) 2020-02-05T20:58:33.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:33.836+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.836987+0000) 2020-02-05T20:58:33.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:33.836+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.837774+0000) 2020-02-05T20:58:33.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:33.837+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.838039+0000) 2020-02-05T20:58:33.842 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:33.840+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72272 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:33.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:33.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:33.841736+0000) 2020-02-05T20:58:35.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:35.604+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:35.605203+0000) 2020-02-05T20:58:35.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:35.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:35.805608+0000) 2020-02-05T20:58:35.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:35.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:35.842593+0000) 2020-02-05T20:58:36.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:36.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:36.702661+0000) 2020-02-05T20:58:36.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:36.946+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:36.946530+0000) 2020-02-05T20:58:36.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:36.946+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:36.946726+0000) 2020-02-05T20:58:38.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:38.845+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.846387+0000) 2020-02-05T20:58:38.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:38.846+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.847698+0000) 2020-02-05T20:58:38.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:38.847+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.848893+0000) 2020-02-05T20:58:38.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:38.848+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.849108+0000) 2020-02-05T20:58:38.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:38.848+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.849786+0000) 2020-02-05T20:58:38.851 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:38.849+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.849644+0000) 2020-02-05T20:58:38.852 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:38.849+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.850256+0000) 2020-02-05T20:58:38.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:38.849+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.850206+0000) 2020-02-05T20:58:38.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:38.850+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.851328+0000) 2020-02-05T20:58:38.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:38.850+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.851559+0000) 2020-02-05T20:58:38.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:38.850+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.850877+0000) 2020-02-05T20:58:38.853 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:38.851+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.851648+0000) 2020-02-05T20:58:38.854 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:38.851+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.851607+0000) 2020-02-05T20:58:38.856 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:38.854+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72312 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:38.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:38.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:38.859718+0000) 2020-02-05T20:58:39.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:39.340+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:39.340778+0000) 2020-02-05T20:58:39.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:39.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:39.950286+0000) 2020-02-05T20:58:40.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:40.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:40.200468+0000) 2020-02-05T20:58:40.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:40.210+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:40.210546+0000) 2020-02-05T20:58:42.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:42.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:42.189784+0000) 2020-02-05T20:58:42.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:42.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:42.304408+0000) 2020-02-05T20:58:43.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:43.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.430883+0000) 2020-02-05T20:58:43.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:43.460+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.461174+0000) 2020-02-05T20:58:43.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:43.861+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.862333+0000) 2020-02-05T20:58:43.864 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:43.862+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.862695+0000) 2020-02-05T20:58:43.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:43.862+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.863238+0000) 2020-02-05T20:58:43.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:43.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.864153+0000) 2020-02-05T20:58:43.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:43.863+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.864660+0000) 2020-02-05T20:58:43.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:43.864+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.864762+0000) 2020-02-05T20:58:43.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:43.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.865118+0000) 2020-02-05T20:58:43.867 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:43.865+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.865695+0000) 2020-02-05T20:58:43.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:43.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.866178+0000) 2020-02-05T20:58:43.867 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:43.865+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.866132+0000) 2020-02-05T20:58:43.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:43.865+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.866391+0000) 2020-02-05T20:58:43.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:43.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.866442+0000) 2020-02-05T20:58:43.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:43.866+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.867183+0000) 2020-02-05T20:58:43.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:43.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:43.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:43.868+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:43.869645+0000) 2020-02-05T20:58:45.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:45.612+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:45.613346+0000) 2020-02-05T20:58:45.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:45.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:45.805844+0000) 2020-02-05T20:58:45.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:45.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:45.842822+0000) 2020-02-05T20:58:46.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:46.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:46.702908+0000) 2020-02-05T20:58:46.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:46.946+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:46.946806+0000) 2020-02-05T20:58:46.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:46.946+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:46.946976+0000) 2020-02-05T20:58:48.876 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:48.874+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.875290+0000) 2020-02-05T20:58:48.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:48.875+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.875814+0000) 2020-02-05T20:58:48.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:48.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.876891+0000) 2020-02-05T20:58:48.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:48.876+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.877481+0000) 2020-02-05T20:58:48.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:48.877+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.878275+0000) 2020-02-05T20:58:48.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:48.877+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.878609+0000) 2020-02-05T20:58:48.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:48.880+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.880591+0000) 2020-02-05T20:58:48.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:48.880+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.880754+0000) 2020-02-05T20:58:48.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:48.880+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.881154+0000) 2020-02-05T20:58:48.884 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:48.881+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.881751+0000) 2020-02-05T20:58:48.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:48.882+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.882540+0000) 2020-02-05T20:58:48.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:48.878+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.879815+0000) 2020-02-05T20:58:48.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:48.881+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.881976+0000) 2020-02-05T20:58:48.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:48.884+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:48.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:48.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:48.886153+0000) 2020-02-05T20:58:49.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:49.340+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:49.340995+0000) 2020-02-05T20:58:49.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:49.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:49.950520+0000) 2020-02-05T20:58:50.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:50.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:50.200760+0000) 2020-02-05T20:58:50.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:50.210+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:50.210773+0000) 2020-02-05T20:58:52.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:52.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:52.190047+0000) 2020-02-05T20:58:52.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:52.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:52.304763+0000) 2020-02-05T20:58:53.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:53.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.431135+0000) 2020-02-05T20:58:53.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:53.461+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.461365+0000) 2020-02-05T20:58:53.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:53.889+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.890377+0000) 2020-02-05T20:58:53.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:53.890+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.890949+0000) 2020-02-05T20:58:53.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:53.890+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.891448+0000) 2020-02-05T20:58:53.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:53.891+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.891932+0000) 2020-02-05T20:58:53.894 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:53.891+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.892740+0000) 2020-02-05T20:58:53.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:53.892+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.892840+0000) 2020-02-05T20:58:53.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:53.892+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.893225+0000) 2020-02-05T20:58:53.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:53.892+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.893574+0000) 2020-02-05T20:58:53.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:53.893+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.894149+0000) 2020-02-05T20:58:53.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:53.893+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.894067+0000) 2020-02-05T20:58:53.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:53.893+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.894227+0000) 2020-02-05T20:58:53.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:53.893+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.894263+0000) 2020-02-05T20:58:53.896 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:53.893+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.894391+0000) 2020-02-05T20:58:53.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:53.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:53.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:53.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:53.899584+0000) 2020-02-05T20:58:55.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:55.624+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:55.625292+0000) 2020-02-05T20:58:55.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:55.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:55.806071+0000) 2020-02-05T20:58:55.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:55.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:55.843052+0000) 2020-02-05T20:58:56.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:56.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:56.703156+0000) 2020-02-05T20:58:56.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:56.946+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:56.947109+0000) 2020-02-05T20:58:56.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:56.946+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:56.947271+0000) 2020-02-05T20:58:58.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:58:58.901+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.902694+0000) 2020-02-05T20:58:58.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:58:58.902+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.903139+0000) 2020-02-05T20:58:58.904 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:58:58.902+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.903701+0000) 2020-02-05T20:58:58.907 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:58:58.905+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.905562+0000) 2020-02-05T20:58:58.908 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:58:58.905+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.906225+0000) 2020-02-05T20:58:58.908 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:58:58.905+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.906315+0000) 2020-02-05T20:58:58.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:58.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.906857+0000) 2020-02-05T20:58:58.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:58:58.906+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.907329+0000) 2020-02-05T20:58:58.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:58.906+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.907847+0000) 2020-02-05T20:58:58.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:58.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.908496+0000) 2020-02-05T20:58:58.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:58:58.908+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.908837+0000) 2020-02-05T20:58:58.910 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:58:58.908+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.908979+0000) 2020-02-05T20:58:58.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:58:58.909+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.909537+0000) 2020-02-05T20:58:58.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:58:58.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72480 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:58:58.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:58.914+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:58.915545+0000) 2020-02-05T20:58:59.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:58:59.340+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:59.341309+0000) 2020-02-05T20:58:59.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:58:59.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:58:59.950800+0000) 2020-02-05T20:59:00.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:00.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:00.200949+0000) 2020-02-05T20:59:00.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:00.210+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:00.210993+0000) 2020-02-05T20:59:02.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:02.190+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:02.190307+0000) 2020-02-05T20:59:02.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:02.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:02.305015+0000) 2020-02-05T20:59:02.624 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:59:02.630 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:59:03.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:03.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.431470+0000) 2020-02-05T20:59:03.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:03.460+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.461650+0000) 2020-02-05T20:59:03.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:03.917+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.918842+0000) 2020-02-05T20:59:03.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:03.918+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.919726+0000) 2020-02-05T20:59:03.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:03.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.920206+0000) 2020-02-05T20:59:03.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:03.920+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.921236+0000) 2020-02-05T20:59:03.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:03.920+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.921695+0000) 2020-02-05T20:59:03.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:03.920+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.922049+0000) 2020-02-05T20:59:03.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:03.921+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.922377+0000) 2020-02-05T20:59:03.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:03.922+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.923816+0000) 2020-02-05T20:59:03.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:03.922+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.923284+0000) 2020-02-05T20:59:03.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:03.923+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.923424+0000) 2020-02-05T20:59:03.926 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:03.923+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.924388+0000) 2020-02-05T20:59:03.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:03.923+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.925039+0000) 2020-02-05T20:59:03.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:03.923+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.923927+0000) 2020-02-05T20:59:03.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:03.927+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72523 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:03.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:03.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:03.930486+0000) 2020-02-05T20:59:05.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:05.632+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:05.633435+0000) 2020-02-05T20:59:05.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:05.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:05.806341+0000) 2020-02-05T20:59:05.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:05.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:05.843341+0000) 2020-02-05T20:59:06.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:06.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:06.703425+0000) 2020-02-05T20:59:06.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:06.947+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:06.947349+0000) 2020-02-05T20:59:06.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:06.947+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:06.947580+0000) 2020-02-05T20:59:08.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:08.932+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.933130+0000) 2020-02-05T20:59:08.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:08.934+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.934595+0000) 2020-02-05T20:59:08.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:08.935+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.936118+0000) 2020-02-05T20:59:08.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:08.935+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.936199+0000) 2020-02-05T20:59:08.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:08.936+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.936827+0000) 2020-02-05T20:59:08.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:08.936+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.937286+0000) 2020-02-05T20:59:08.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:08.937+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.937952+0000) 2020-02-05T20:59:08.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:08.937+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.938281+0000) 2020-02-05T20:59:08.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:08.937+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.938449+0000) 2020-02-05T20:59:08.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:08.937+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.938374+0000) 2020-02-05T20:59:08.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:08.938+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.938448+0000) 2020-02-05T20:59:08.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:08.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.938959+0000) 2020-02-05T20:59:08.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:08.941+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.942110+0000) 2020-02-05T20:59:08.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:08.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72564 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:08.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:08.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:08.943098+0000) 2020-02-05T20:59:09.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:09.340+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:09.341518+0000) 2020-02-05T20:59:09.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:09.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:09.951033+0000) 2020-02-05T20:59:10.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:10.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:10.201189+0000) 2020-02-05T20:59:10.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:10.210+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:10.211257+0000) 2020-02-05T20:59:12.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:12.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:12.190547+0000) 2020-02-05T20:59:12.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:12.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:12.305262+0000) 2020-02-05T20:59:13.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:13.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.431709+0000) 2020-02-05T20:59:13.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:13.461+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.461883+0000) 2020-02-05T20:59:13.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:13.947+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.949265+0000) 2020-02-05T20:59:13.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:13.948+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.949943+0000) 2020-02-05T20:59:13.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:13.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.950469+0000) 2020-02-05T20:59:13.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:13.949+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.950948+0000) 2020-02-05T20:59:13.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:13.950+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.951385+0000) 2020-02-05T20:59:13.953 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:13.950+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.951643+0000) 2020-02-05T20:59:13.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:13.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.952301+0000) 2020-02-05T20:59:13.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:13.951+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.952833+0000) 2020-02-05T20:59:13.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:13.952+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.953230+0000) 2020-02-05T20:59:13.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:13.952+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.953763+0000) 2020-02-05T20:59:13.955 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:13.952+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.954313+0000) 2020-02-05T20:59:13.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:13.953+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.954630+0000) 2020-02-05T20:59:13.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:13.953+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.954894+0000) 2020-02-05T20:59:13.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:13.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72608 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:13.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:13.959+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:13.960101+0000) 2020-02-05T20:59:15.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:15.641+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:15.642123+0000) 2020-02-05T20:59:15.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:15.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:15.806568+0000) 2020-02-05T20:59:15.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:15.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:15.843575+0000) 2020-02-05T20:59:16.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:16.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:16.703655+0000) 2020-02-05T20:59:16.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:16.946+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:16.947642+0000) 2020-02-05T20:59:16.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:16.946+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:16.947830+0000) 2020-02-05T20:59:18.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:18.962+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.963621+0000) 2020-02-05T20:59:18.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:18.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.964360+0000) 2020-02-05T20:59:18.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:18.963+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.964882+0000) 2020-02-05T20:59:18.966 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:18.964+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.965589+0000) 2020-02-05T20:59:18.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:18.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.965959+0000) 2020-02-05T20:59:18.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:18.965+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.966333+0000) 2020-02-05T20:59:18.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:18.965+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.966786+0000) 2020-02-05T20:59:18.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:18.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.966842+0000) 2020-02-05T20:59:18.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:18.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.967751+0000) 2020-02-05T20:59:18.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:18.966+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.967672+0000) 2020-02-05T20:59:18.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:18.966+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.968359+0000) 2020-02-05T20:59:18.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:18.968+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.969261+0000) 2020-02-05T20:59:18.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:18.967+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.968923+0000) 2020-02-05T20:59:18.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:18.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:18.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:18.972+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:18.973622+0000) 2020-02-05T20:59:19.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:19.341+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:19.341710+0000) 2020-02-05T20:59:19.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:19.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:19.951298+0000) 2020-02-05T20:59:20.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:20.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:20.201465+0000) 2020-02-05T20:59:20.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:20.210+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:20.211518+0000) 2020-02-05T20:59:22.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:22.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:22.190814+0000) 2020-02-05T20:59:22.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:22.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:22.305509+0000) 2020-02-05T20:59:23.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:23.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.431973+0000) 2020-02-05T20:59:23.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:23.461+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.462124+0000) 2020-02-05T20:59:23.980 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:23.978+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.979251+0000) 2020-02-05T20:59:23.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:23.978+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.979841+0000) 2020-02-05T20:59:23.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:23.979+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.980558+0000) 2020-02-05T20:59:23.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:23.979+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.980978+0000) 2020-02-05T20:59:23.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:23.979+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.981248+0000) 2020-02-05T20:59:23.983 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:23.980+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.982289+0000) 2020-02-05T20:59:23.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:23.981+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.982651+0000) 2020-02-05T20:59:23.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:23.982+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.983905+0000) 2020-02-05T20:59:23.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:23.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.984145+0000) 2020-02-05T20:59:23.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:23.983+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.984399+0000) 2020-02-05T20:59:23.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:23.983+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.984852+0000) 2020-02-05T20:59:23.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:23.983+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.985170+0000) 2020-02-05T20:59:23.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:23.984+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.985376+0000) 2020-02-05T20:59:23.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:23.986+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:23.987114+0000) 2020-02-05T20:59:23.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:23.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:25.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:25.650+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:25.650679+0000) 2020-02-05T20:59:25.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:25.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:25.806806+0000) 2020-02-05T20:59:25.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:25.843+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:25.843826+0000) 2020-02-05T20:59:26.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:26.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:26.703899+0000) 2020-02-05T20:59:26.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:26.947+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:26.947897+0000) 2020-02-05T20:59:26.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:26.947+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:26.948079+0000) 2020-02-05T20:59:28.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:28.993+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.994530+0000) 2020-02-05T20:59:28.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:28.994+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.995534+0000) 2020-02-05T20:59:28.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:28.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.996480+0000) 2020-02-05T20:59:28.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:28.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.996776+0000) 2020-02-05T20:59:28.998 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:28.996+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.997303+0000) 2020-02-05T20:59:28.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:28.996+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.997551+0000) 2020-02-05T20:59:28.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:28.997+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.998117+0000) 2020-02-05T20:59:28.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:28.997+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.998464+0000) 2020-02-05T20:59:29.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:28.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:28.999408+0000) 2020-02-05T20:59:29.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:28.998+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.000177+0000) 2020-02-05T20:59:29.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:28.999+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.000483+0000) 2020-02-05T20:59:29.002 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:28.998+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.000357+0000) 2020-02-05T20:59:29.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:28.999+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.000437+0000) 2020-02-05T20:59:29.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:29.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:29.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:29.003+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.004880+0000) 2020-02-05T20:59:29.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:29.341+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.341917+0000) 2020-02-05T20:59:29.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:29.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:29.951550+0000) 2020-02-05T20:59:30.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:30.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:30.201737+0000) 2020-02-05T20:59:30.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:30.211+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:30.211773+0000) 2020-02-05T20:59:32.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:32.190+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:32.191052+0000) 2020-02-05T20:59:32.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:32.304+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:32.305742+0000) 2020-02-05T20:59:32.770 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:59:32.777 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T20:59:33.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:33.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:33.432270+0000) 2020-02-05T20:59:33.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:33.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:33.462356+0000) 2020-02-05T20:59:34.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:34.008+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.010155+0000) 2020-02-05T20:59:34.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:34.010+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.011382+0000) 2020-02-05T20:59:34.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:34.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.011903+0000) 2020-02-05T20:59:34.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:34.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.012399+0000) 2020-02-05T20:59:34.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:34.011+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.012644+0000) 2020-02-05T20:59:34.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:34.011+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.012999+0000) 2020-02-05T20:59:34.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:34.012+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.013761+0000) 2020-02-05T20:59:34.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:34.013+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.014296+0000) 2020-02-05T20:59:34.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:34.013+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.014646+0000) 2020-02-05T20:59:34.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:34.013+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.014932+0000) 2020-02-05T20:59:34.017 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:34.014+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.015399+0000) 2020-02-05T20:59:34.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:34.014+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.015548+0000) 2020-02-05T20:59:34.017 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:34.014+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.015612+0000) 2020-02-05T20:59:34.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:34.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:34.019400+0000) 2020-02-05T20:59:34.020 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:34.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:35.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:35.658+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:35.658382+0000) 2020-02-05T20:59:35.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:35.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:35.807017+0000) 2020-02-05T20:59:35.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:35.842+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:35.844058+0000) 2020-02-05T20:59:36.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:36.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:36.704134+0000) 2020-02-05T20:59:36.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:36.946+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:36.948138+0000) 2020-02-05T20:59:36.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:36.946+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:36.948316+0000) 2020-02-05T20:59:39.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:39.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.025257+0000) 2020-02-05T20:59:39.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:39.025+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.026414+0000) 2020-02-05T20:59:39.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:39.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.026847+0000) 2020-02-05T20:59:39.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:39.025+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.027261+0000) 2020-02-05T20:59:39.030 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:39.026+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.027521+0000) 2020-02-05T20:59:39.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:39.026+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.027904+0000) 2020-02-05T20:59:39.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:39.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.028513+0000) 2020-02-05T20:59:39.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:39.027+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.028843+0000) 2020-02-05T20:59:39.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:39.028+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.029896+0000) 2020-02-05T20:59:39.032 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:39.028+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.030298+0000) 2020-02-05T20:59:39.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:39.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.030687+0000) 2020-02-05T20:59:39.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:39.029+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.030155+0000) 2020-02-05T20:59:39.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:39.029+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.031070+0000) 2020-02-05T20:59:39.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:39.033+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.034320+0000) 2020-02-05T20:59:39.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:39.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:39.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:39.341+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.342128+0000) 2020-02-05T20:59:39.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:39.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:39.951763+0000) 2020-02-05T20:59:40.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:40.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:40.201998+0000) 2020-02-05T20:59:40.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:40.211+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:40.211998+0000) 2020-02-05T20:59:42.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:42.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:42.191300+0000) 2020-02-05T20:59:42.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:42.305+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:42.305991+0000) 2020-02-05T20:59:43.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:43.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:43.432516+0000) 2020-02-05T20:59:43.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:43.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:43.462590+0000) 2020-02-05T20:59:44.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:44.039+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.039354+0000) 2020-02-05T20:59:44.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:44.041+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.041204+0000) 2020-02-05T20:59:44.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:44.041+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.041695+0000) 2020-02-05T20:59:44.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:44.040+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.042045+0000) 2020-02-05T20:59:44.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:44.041+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.042880+0000) 2020-02-05T20:59:44.044 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:44.041+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.043293+0000) 2020-02-05T20:59:44.045 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:44.044+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.044417+0000) 2020-02-05T20:59:44.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:44.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.045127+0000) 2020-02-05T20:59:44.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:44.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.045358+0000) 2020-02-05T20:59:44.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:44.045+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.045791+0000) 2020-02-05T20:59:44.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:44.044+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.046190+0000) 2020-02-05T20:59:44.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:44.044+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.046369+0000) 2020-02-05T20:59:44.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:44.045+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.046927+0000) 2020-02-05T20:59:44.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:44.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:44.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:44.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:44.051755+0000) 2020-02-05T20:59:45.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:45.667+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:45.667804+0000) 2020-02-05T20:59:45.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:45.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:45.807326+0000) 2020-02-05T20:59:45.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:45.844+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:45.844319+0000) 2020-02-05T20:59:46.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:46.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:46.704364+0000) 2020-02-05T20:59:46.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:46.947+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:46.948392+0000) 2020-02-05T20:59:46.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:46.947+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:46.948611+0000) 2020-02-05T20:59:49.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:49.055+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.056597+0000) 2020-02-05T20:59:49.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:49.056+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.057397+0000) 2020-02-05T20:59:49.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:49.056+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.057892+0000) 2020-02-05T20:59:49.059 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:49.057+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.058456+0000) 2020-02-05T20:59:49.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:49.058+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.059253+0000) 2020-02-05T20:59:49.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:49.058+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.059682+0000) 2020-02-05T20:59:49.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:49.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.060221+0000) 2020-02-05T20:59:49.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:49.059+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.060777+0000) 2020-02-05T20:59:49.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:49.059+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.060624+0000) 2020-02-05T20:59:49.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:49.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.061529+0000) 2020-02-05T20:59:49.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:49.061+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.062652+0000) 2020-02-05T20:59:49.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:49.061+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.062573+0000) 2020-02-05T20:59:49.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:49.061+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.063099+0000) 2020-02-05T20:59:49.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:49.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72900 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:49.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:49.067+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.068768+0000) 2020-02-05T20:59:49.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:49.341+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.342357+0000) 2020-02-05T20:59:49.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:49.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:49.951970+0000) 2020-02-05T20:59:50.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:50.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:50.202250+0000) 2020-02-05T20:59:50.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:50.211+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:50.212253+0000) 2020-02-05T20:59:52.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:52.190+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:52.191566+0000) 2020-02-05T20:59:52.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:52.305+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:52.306244+0000) 2020-02-05T20:59:53.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:53.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:53.432771+0000) 2020-02-05T20:59:53.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:53.461+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:53.462844+0000) 2020-02-05T20:59:54.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:54.069+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.070875+0000) 2020-02-05T20:59:54.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:54.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.071454+0000) 2020-02-05T20:59:54.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:54.070+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.071869+0000) 2020-02-05T20:59:54.074 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:54.071+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.072335+0000) 2020-02-05T20:59:54.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:54.071+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.072678+0000) 2020-02-05T20:59:54.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:54.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.073186+0000) 2020-02-05T20:59:54.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:54.072+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.074124+0000) 2020-02-05T20:59:54.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:54.073+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.074522+0000) 2020-02-05T20:59:54.076 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:54.073+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.074695+0000) 2020-02-05T20:59:54.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:54.073+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.074923+0000) 2020-02-05T20:59:54.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:54.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.075224+0000) 2020-02-05T20:59:54.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:54.075+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.076535+0000) 2020-02-05T20:59:54.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:54.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.077814+0000) 2020-02-05T20:59:54.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:54.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:54.079253+0000) 2020-02-05T20:59:54.081 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:54.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:55.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:55.674+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:55.675833+0000) 2020-02-05T20:59:55.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:55.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:55.807617+0000) 2020-02-05T20:59:55.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:55.843+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:55.844546+0000) 2020-02-05T20:59:56.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:56.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:56.704592+0000) 2020-02-05T20:59:56.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:56.948+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:56.948690+0000) 2020-02-05T20:59:56.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:56.948+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:56.948862+0000) 2020-02-05T20:59:59.087 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T20:59:59.084+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.085615+0000) 2020-02-05T20:59:59.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T20:59:59.083+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.085126+0000) 2020-02-05T20:59:59.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T20:59:59.085+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.086712+0000) 2020-02-05T20:59:59.088 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T20:59:59.086+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.087461+0000) 2020-02-05T20:59:59.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:59.086+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.088008+0000) 2020-02-05T20:59:59.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:59.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.088595+0000) 2020-02-05T20:59:59.090 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T20:59:59.088+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.089119+0000) 2020-02-05T20:59:59.090 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T20:59:59.087+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.088884+0000) 2020-02-05T20:59:59.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:59.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.089926+0000) 2020-02-05T20:59:59.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T20:59:59.089+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.090775+0000) 2020-02-05T20:59:59.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T20:59:59.089+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.091076+0000) 2020-02-05T20:59:59.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T20:59:59.089+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.091377+0000) 2020-02-05T20:59:59.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T20:59:59.089+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.091414+0000) 2020-02-05T20:59:59.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T20:59:59.093+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 72984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T20:59:59.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:59.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.095709+0000) 2020-02-05T20:59:59.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T20:59:59.341+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.342586+0000) 2020-02-05T20:59:59.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T20:59:59.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T19:59:59.952178+0000) 2020-02-05T21:00:00.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:00.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:00.202511+0000) 2020-02-05T21:00:00.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:00.211+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:00.212479+0000) 2020-02-05T21:00:02.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:02.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:02.191828+0000) 2020-02-05T21:00:02.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:02.306+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:02.306503+0000) 2020-02-05T21:00:02.933 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:00:02.939 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:00:03.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:03.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:03.432991+0000) 2020-02-05T21:00:03.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:03.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:03.463117+0000) 2020-02-05T21:00:04.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:04.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.102058+0000) 2020-02-05T21:00:04.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:04.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.101779+0000) 2020-02-05T21:00:04.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:04.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.102797+0000) 2020-02-05T21:00:04.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:04.101+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.102820+0000) 2020-02-05T21:00:04.105 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:04.102+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.103319+0000) 2020-02-05T21:00:04.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:04.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.103978+0000) 2020-02-05T21:00:04.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:04.104+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.104643+0000) 2020-02-05T21:00:04.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:04.104+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.105068+0000) 2020-02-05T21:00:04.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:04.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.104833+0000) 2020-02-05T21:00:04.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:04.105+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.105500+0000) 2020-02-05T21:00:04.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:04.105+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.105789+0000) 2020-02-05T21:00:04.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:04.106+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.106746+0000) 2020-02-05T21:00:04.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:04.105+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.106278+0000) 2020-02-05T21:00:04.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:04.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:04.110118+0000) 2020-02-05T21:00:04.112 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:04.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73026 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:05.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:05.683+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:05.683754+0000) 2020-02-05T21:00:05.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:05.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:05.807886+0000) 2020-02-05T21:00:05.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:05.843+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:05.844797+0000) 2020-02-05T21:00:06.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:06.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:06.704809+0000) 2020-02-05T21:00:06.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:06.948+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:06.948932+0000) 2020-02-05T21:00:06.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:06.948+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:06.949138+0000) 2020-02-05T21:00:09.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:09.117+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.117488+0000) 2020-02-05T21:00:09.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:09.117+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.119059+0000) 2020-02-05T21:00:09.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:09.118+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.118833+0000) 2020-02-05T21:00:09.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:09.118+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.119693+0000) 2020-02-05T21:00:09.121 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:09.118+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.119955+0000) 2020-02-05T21:00:09.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:09.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.120460+0000) 2020-02-05T21:00:09.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:09.119+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.120689+0000) 2020-02-05T21:00:09.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:09.120+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.121464+0000) 2020-02-05T21:00:09.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:09.121+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.122064+0000) 2020-02-05T21:00:09.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:09.122+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.122440+0000) 2020-02-05T21:00:09.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:09.122+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.123143+0000) 2020-02-05T21:00:09.124 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:09.122+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.122979+0000) 2020-02-05T21:00:09.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:09.122+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.123150+0000) 2020-02-05T21:00:09.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:09.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:09.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:09.128+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.129737+0000) 2020-02-05T21:00:09.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:09.342+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.342813+0000) 2020-02-05T21:00:09.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:09.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:09.952380+0000) 2020-02-05T21:00:10.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:10.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:10.202762+0000) 2020-02-05T21:00:10.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:10.212+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:10.212715+0000) 2020-02-05T21:00:12.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:12.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:12.192068+0000) 2020-02-05T21:00:12.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:12.306+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:12.306749+0000) 2020-02-05T21:00:13.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:13.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:13.433248+0000) 2020-02-05T21:00:13.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:13.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:13.463385+0000) 2020-02-05T21:00:14.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:14.130+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.131286+0000) 2020-02-05T21:00:14.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:14.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.131879+0000) 2020-02-05T21:00:14.134 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:14.131+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.132782+0000) 2020-02-05T21:00:14.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:14.132+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.133404+0000) 2020-02-05T21:00:14.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:14.133+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.134273+0000) 2020-02-05T21:00:14.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:14.133+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.134716+0000) 2020-02-05T21:00:14.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:14.134+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.135116+0000) 2020-02-05T21:00:14.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:14.134+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.134808+0000) 2020-02-05T21:00:14.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:14.134+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.135581+0000) 2020-02-05T21:00:14.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:14.134+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.135276+0000) 2020-02-05T21:00:14.138 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:14.135+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.136028+0000) 2020-02-05T21:00:14.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:14.135+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.136115+0000) 2020-02-05T21:00:14.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:14.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.137049+0000) 2020-02-05T21:00:14.142 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:14.140+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73111 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:14.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:14.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:14.143027+0000) 2020-02-05T21:00:15.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:15.690+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:15.692001+0000) 2020-02-05T21:00:15.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:15.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:15.808111+0000) 2020-02-05T21:00:15.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:15.844+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:15.845023+0000) 2020-02-05T21:00:16.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:16.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:16.705063+0000) 2020-02-05T21:00:16.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:16.948+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:16.949239+0000) 2020-02-05T21:00:16.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:16.948+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:16.949407+0000) 2020-02-05T21:00:19.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:19.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.147340+0000) 2020-02-05T21:00:19.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:19.148+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.148453+0000) 2020-02-05T21:00:19.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:19.148+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.149231+0000) 2020-02-05T21:00:19.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:19.148+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.149750+0000) 2020-02-05T21:00:19.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:19.149+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.149481+0000) 2020-02-05T21:00:19.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:19.149+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.150289+0000) 2020-02-05T21:00:19.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:19.149+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.150636+0000) 2020-02-05T21:00:19.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:19.149+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.150935+0000) 2020-02-05T21:00:19.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:19.151+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.151529+0000) 2020-02-05T21:00:19.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:19.151+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.152181+0000) 2020-02-05T21:00:19.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:19.152+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.153295+0000) 2020-02-05T21:00:19.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:19.153+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.153923+0000) 2020-02-05T21:00:19.155 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:19.153+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.154176+0000) 2020-02-05T21:00:19.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:19.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:19.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:19.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.158857+0000) 2020-02-05T21:00:19.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:19.342+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.343019+0000) 2020-02-05T21:00:19.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:19.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:19.952590+0000) 2020-02-05T21:00:20.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:20.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:20.203025+0000) 2020-02-05T21:00:20.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:20.212+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:20.212948+0000) 2020-02-05T21:00:22.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:22.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:22.192329+0000) 2020-02-05T21:00:22.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:22.306+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:22.306998+0000) 2020-02-05T21:00:23.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:23.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:23.433492+0000) 2020-02-05T21:00:23.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:23.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:23.463621+0000) 2020-02-05T21:00:24.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:24.162+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.163496+0000) 2020-02-05T21:00:24.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:24.163+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.164787+0000) 2020-02-05T21:00:24.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:24.164+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.164764+0000) 2020-02-05T21:00:24.167 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:24.164+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.165393+0000) 2020-02-05T21:00:24.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:24.164+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.166009+0000) 2020-02-05T21:00:24.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:24.165+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.166443+0000) 2020-02-05T21:00:24.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:24.165+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.166310+0000) 2020-02-05T21:00:24.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:24.166+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.166952+0000) 2020-02-05T21:00:24.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:24.167+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.167706+0000) 2020-02-05T21:00:24.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:24.166+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.167931+0000) 2020-02-05T21:00:24.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:24.168+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.168497+0000) 2020-02-05T21:00:24.170 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:24.168+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.168699+0000) 2020-02-05T21:00:24.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:24.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.168527+0000) 2020-02-05T21:00:24.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:24.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:24.170997+0000) 2020-02-05T21:00:24.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:24.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:25.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:25.699+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:25.700220+0000) 2020-02-05T21:00:25.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:25.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:25.808372+0000) 2020-02-05T21:00:25.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:25.844+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:25.845300+0000) 2020-02-05T21:00:26.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:26.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:26.705370+0000) 2020-02-05T21:00:26.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:26.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:26.949496+0000) 2020-02-05T21:00:26.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:26.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:26.949693+0000) 2020-02-05T21:00:29.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:29.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.178224+0000) 2020-02-05T21:00:29.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:29.177+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.178675+0000) 2020-02-05T21:00:29.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:29.178+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.179240+0000) 2020-02-05T21:00:29.181 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:29.179+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.179686+0000) 2020-02-05T21:00:29.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:29.179+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.180692+0000) 2020-02-05T21:00:29.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:29.180+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.181051+0000) 2020-02-05T21:00:29.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:29.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.181519+0000) 2020-02-05T21:00:29.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:29.180+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.181172+0000) 2020-02-05T21:00:29.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:29.181+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.181734+0000) 2020-02-05T21:00:29.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:29.181+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.182007+0000) 2020-02-05T21:00:29.184 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:29.182+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.182416+0000) 2020-02-05T21:00:29.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:29.181+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.182279+0000) 2020-02-05T21:00:29.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:29.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.183651+0000) 2020-02-05T21:00:29.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:29.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:29.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:29.187+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.188530+0000) 2020-02-05T21:00:29.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:29.342+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.343249+0000) 2020-02-05T21:00:29.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:29.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:29.952823+0000) 2020-02-05T21:00:30.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:30.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:30.203318+0000) 2020-02-05T21:00:30.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:30.212+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:30.213259+0000) 2020-02-05T21:00:32.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:32.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:32.192588+0000) 2020-02-05T21:00:32.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:32.307+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:32.307242+0000) 2020-02-05T21:00:33.140 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:00:33.146 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:00:33.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:33.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:33.433673+0000) 2020-02-05T21:00:33.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:33.462+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:33.463909+0000) 2020-02-05T21:00:34.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:34.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.193932+0000) 2020-02-05T21:00:34.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:34.193+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.194277+0000) 2020-02-05T21:00:34.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:34.193+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.194615+0000) 2020-02-05T21:00:34.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:34.193+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.194907+0000) 2020-02-05T21:00:34.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:34.195+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.195874+0000) 2020-02-05T21:00:34.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:34.196+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.196654+0000) 2020-02-05T21:00:34.199 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:34.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.196720+0000) 2020-02-05T21:00:34.199 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:34.196+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.197330+0000) 2020-02-05T21:00:34.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:34.197+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.197443+0000) 2020-02-05T21:00:34.200 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:34.197+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.197591+0000) 2020-02-05T21:00:34.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:34.197+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.197877+0000) 2020-02-05T21:00:34.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:34.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:34.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:34.200+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.201840+0000) 2020-02-05T21:00:34.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:34.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.202128+0000) 2020-02-05T21:00:34.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:34.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:34.202412+0000) 2020-02-05T21:00:35.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:35.708+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:35.708314+0000) 2020-02-05T21:00:35.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:35.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:35.808604+0000) 2020-02-05T21:00:35.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:35.845+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:35.845534+0000) 2020-02-05T21:00:36.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:36.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:36.705608+0000) 2020-02-05T21:00:36.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:36.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:36.949729+0000) 2020-02-05T21:00:36.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:36.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:36.949970+0000) 2020-02-05T21:00:39.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:39.206+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.206373+0000) 2020-02-05T21:00:39.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:39.207+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.207431+0000) 2020-02-05T21:00:39.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:39.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.208393+0000) 2020-02-05T21:00:39.210 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:39.209+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.209335+0000) 2020-02-05T21:00:39.211 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:39.209+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.209826+0000) 2020-02-05T21:00:39.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:39.209+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.210046+0000) 2020-02-05T21:00:39.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:39.210+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.211211+0000) 2020-02-05T21:00:39.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:39.210+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.211176+0000) 2020-02-05T21:00:39.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:39.210+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.211206+0000) 2020-02-05T21:00:39.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:39.211+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.211704+0000) 2020-02-05T21:00:39.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:39.212+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.212227+0000) 2020-02-05T21:00:39.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:39.212+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.212606+0000) 2020-02-05T21:00:39.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:39.211+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.212337+0000) 2020-02-05T21:00:39.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:39.214+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.214871+0000) 2020-02-05T21:00:39.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:39.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:39.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:39.343+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.343461+0000) 2020-02-05T21:00:39.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:39.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:39.953019+0000) 2020-02-05T21:00:40.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:40.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:40.203579+0000) 2020-02-05T21:00:40.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:40.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:40.213462+0000) 2020-02-05T21:00:42.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:42.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:42.192822+0000) 2020-02-05T21:00:42.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:42.306+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:42.307478+0000) 2020-02-05T21:00:43.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:43.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:43.433924+0000) 2020-02-05T21:00:43.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:43.463+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:43.464141+0000) 2020-02-05T21:00:44.222 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:44.219+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.220969+0000) 2020-02-05T21:00:44.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:44.222+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.223159+0000) 2020-02-05T21:00:44.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:44.222+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.223567+0000) 2020-02-05T21:00:44.225 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:44.223+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.224218+0000) 2020-02-05T21:00:44.225 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:44.223+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.224831+0000) 2020-02-05T21:00:44.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:44.223+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.224587+0000) 2020-02-05T21:00:44.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:44.223+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.225378+0000) 2020-02-05T21:00:44.227 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:44.224+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.226017+0000) 2020-02-05T21:00:44.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:44.225+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.226550+0000) 2020-02-05T21:00:44.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:44.225+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.226680+0000) 2020-02-05T21:00:44.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:44.226+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.227951+0000) 2020-02-05T21:00:44.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:44.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.228106+0000) 2020-02-05T21:00:44.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:44.226+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.228374+0000) 2020-02-05T21:00:44.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:44.229+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:44.230973+0000) 2020-02-05T21:00:44.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:44.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:45.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:45.715+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:45.716416+0000) 2020-02-05T21:00:45.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:45.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:45.808817+0000) 2020-02-05T21:00:45.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:45.844+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:45.845764+0000) 2020-02-05T21:00:46.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:46.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:46.705841+0000) 2020-02-05T21:00:46.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:46.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:46.949985+0000) 2020-02-05T21:00:46.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:46.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:46.950253+0000) 2020-02-05T21:00:49.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:49.239+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.240983+0000) 2020-02-05T21:00:49.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:49.239+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.240642+0000) 2020-02-05T21:00:49.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:49.242+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.243613+0000) 2020-02-05T21:00:49.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:49.242+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.243638+0000) 2020-02-05T21:00:49.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:49.242+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.244124+0000) 2020-02-05T21:00:49.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:49.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.238648+0000) 2020-02-05T21:00:49.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:49.238+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.239343+0000) 2020-02-05T21:00:49.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:49.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.239777+0000) 2020-02-05T21:00:49.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:49.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.242536+0000) 2020-02-05T21:00:49.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:49.239+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.240222+0000) 2020-02-05T21:00:49.248 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:49.240+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.241396+0000) 2020-02-05T21:00:49.248 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:49.242+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.243808+0000) 2020-02-05T21:00:49.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:49.243+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.244891+0000) 2020-02-05T21:00:49.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:49.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:49.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:49.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.249447+0000) 2020-02-05T21:00:49.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:49.342+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.343673+0000) 2020-02-05T21:00:49.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:49.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:49.953274+0000) 2020-02-05T21:00:50.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:50.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:50.203793+0000) 2020-02-05T21:00:50.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:50.212+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:50.213718+0000) 2020-02-05T21:00:52.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:52.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:52.193068+0000) 2020-02-05T21:00:52.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:52.306+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:52.307786+0000) 2020-02-05T21:00:53.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:53.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:53.434176+0000) 2020-02-05T21:00:53.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:53.463+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:53.464386+0000) 2020-02-05T21:00:54.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:54.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.253821+0000) 2020-02-05T21:00:54.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:54.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.253912+0000) 2020-02-05T21:00:54.256 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:54.254+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.255121+0000) 2020-02-05T21:00:54.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:54.254+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.255727+0000) 2020-02-05T21:00:54.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:54.254+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.255668+0000) 2020-02-05T21:00:54.257 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:54.255+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.256621+0000) 2020-02-05T21:00:54.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:54.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.257188+0000) 2020-02-05T21:00:54.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:54.256+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.257794+0000) 2020-02-05T21:00:54.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:54.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.258259+0000) 2020-02-05T21:00:54.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:54.257+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.258671+0000) 2020-02-05T21:00:54.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:54.257+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.258986+0000) 2020-02-05T21:00:54.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:54.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.259800+0000) 2020-02-05T21:00:54.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:54.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.260353+0000) 2020-02-05T21:00:54.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:54.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:54.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:54.263+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:54.264900+0000) 2020-02-05T21:00:55.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:55.725+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:55.725883+0000) 2020-02-05T21:00:55.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:55.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:55.809041+0000) 2020-02-05T21:00:55.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:55.845+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:55.845992+0000) 2020-02-05T21:00:56.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:56.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:56.706130+0000) 2020-02-05T21:00:56.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:56.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:56.950221+0000) 2020-02-05T21:00:56.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:56.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:56.950486+0000) 2020-02-05T21:00:59.270 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:59.267+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.268717+0000) 2020-02-05T21:00:59.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:00:59.268+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.269517+0000) 2020-02-05T21:00:59.271 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:00:59.268+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.269733+0000) 2020-02-05T21:00:59.271 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:00:59.269+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.270618+0000) 2020-02-05T21:00:59.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:59.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.271219+0000) 2020-02-05T21:00:59.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:00:59.270+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.271265+0000) 2020-02-05T21:00:59.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:00:59.270+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.271831+0000) 2020-02-05T21:00:59.274 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:00:59.271+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.272281+0000) 2020-02-05T21:00:59.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:00:59.270+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.272311+0000) 2020-02-05T21:00:59.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:59.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.273127+0000) 2020-02-05T21:00:59.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:00:59.271+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.272662+0000) 2020-02-05T21:00:59.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:00:59.273+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.274835+0000) 2020-02-05T21:00:59.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:00:59.273+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.275217+0000) 2020-02-05T21:00:59.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:00:59.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:00:59.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:59.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.279179+0000) 2020-02-05T21:00:59.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:00:59.343+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.343903+0000) 2020-02-05T21:00:59.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:00:59.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:00:59.953485+0000) 2020-02-05T21:01:00.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:00.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:00.204022+0000) 2020-02-05T21:01:00.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:00.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:00.213925+0000) 2020-02-05T21:01:02.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:02.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:02.193332+0000) 2020-02-05T21:01:02.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:02.307+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:02.308046+0000) 2020-02-05T21:01:03.270 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:01:03.276 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:01:03.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:03.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:03.434416+0000) 2020-02-05T21:01:03.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:03.463+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:03.464620+0000) 2020-02-05T21:01:04.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:04.281+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.282438+0000) 2020-02-05T21:01:04.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:04.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.284243+0000) 2020-02-05T21:01:04.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:04.283+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.284961+0000) 2020-02-05T21:01:04.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:04.284+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.285373+0000) 2020-02-05T21:01:04.287 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:04.285+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.286327+0000) 2020-02-05T21:01:04.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:04.284+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.285882+0000) 2020-02-05T21:01:04.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:04.285+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.286826+0000) 2020-02-05T21:01:04.289 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:04.286+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.287981+0000) 2020-02-05T21:01:04.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:04.287+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.288653+0000) 2020-02-05T21:01:04.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:04.288+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.289217+0000) 2020-02-05T21:01:04.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:04.287+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.289004+0000) 2020-02-05T21:01:04.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:04.288+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.289777+0000) 2020-02-05T21:01:04.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:04.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.290358+0000) 2020-02-05T21:01:04.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:04.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:04.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:04.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:04.293572+0000) 2020-02-05T21:01:05.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:05.733+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:05.733840+0000) 2020-02-05T21:01:05.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:05.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:05.809292+0000) 2020-02-05T21:01:05.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:05.845+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:05.846266+0000) 2020-02-05T21:01:06.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:06.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:06.706325+0000) 2020-02-05T21:01:06.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:06.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:06.950486+0000) 2020-02-05T21:01:06.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:06.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:06.950713+0000) 2020-02-05T21:01:09.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:09.297+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.298923+0000) 2020-02-05T21:01:09.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:09.298+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.299527+0000) 2020-02-05T21:01:09.301 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:09.299+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.300325+0000) 2020-02-05T21:01:09.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:09.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.300844+0000) 2020-02-05T21:01:09.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:09.299+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.301025+0000) 2020-02-05T21:01:09.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:09.300+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.301776+0000) 2020-02-05T21:01:09.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:09.301+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.302372+0000) 2020-02-05T21:01:09.304 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:09.301+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.302757+0000) 2020-02-05T21:01:09.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:09.301+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.303305+0000) 2020-02-05T21:01:09.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:09.302+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.303564+0000) 2020-02-05T21:01:09.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:09.302+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.304076+0000) 2020-02-05T21:01:09.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:09.303+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.304107+0000) 2020-02-05T21:01:09.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:09.303+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.304773+0000) 2020-02-05T21:01:09.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:09.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:09.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:09.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.309660+0000) 2020-02-05T21:01:09.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:09.343+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.344225+0000) 2020-02-05T21:01:09.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:09.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:09.953673+0000) 2020-02-05T21:01:10.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:10.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:10.204228+0000) 2020-02-05T21:01:10.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:10.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:10.214157+0000) 2020-02-05T21:01:12.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:12.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:12.193642+0000) 2020-02-05T21:01:12.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:12.307+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:12.308303+0000) 2020-02-05T21:01:13.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:13.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:13.434730+0000) 2020-02-05T21:01:13.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:13.464+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:13.464870+0000) 2020-02-05T21:01:14.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:14.313+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.314401+0000) 2020-02-05T21:01:14.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:14.313+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.315163+0000) 2020-02-05T21:01:14.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:14.314+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.316368+0000) 2020-02-05T21:01:14.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:14.315+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.316955+0000) 2020-02-05T21:01:14.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:14.316+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.317383+0000) 2020-02-05T21:01:14.319 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:14.317+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.318337+0000) 2020-02-05T21:01:14.319 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:14.317+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.318588+0000) 2020-02-05T21:01:14.320 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:14.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.319366+0000) 2020-02-05T21:01:14.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:14.318+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.319933+0000) 2020-02-05T21:01:14.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:14.318+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.319583+0000) 2020-02-05T21:01:14.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:14.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.320641+0000) 2020-02-05T21:01:14.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:14.319+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.320642+0000) 2020-02-05T21:01:14.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:14.321+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.322809+0000) 2020-02-05T21:01:14.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:14.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:14.323730+0000) 2020-02-05T21:01:14.325 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:14.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:15.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:15.740+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:15.741867+0000) 2020-02-05T21:01:15.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:15.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:15.809612+0000) 2020-02-05T21:01:15.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:15.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:15.846519+0000) 2020-02-05T21:01:16.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:16.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:16.706551+0000) 2020-02-05T21:01:16.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:16.949+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:16.950757+0000) 2020-02-05T21:01:16.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:16.949+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:16.950967+0000) 2020-02-05T21:01:19.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:19.329+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.330790+0000) 2020-02-05T21:01:19.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:19.330+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.331554+0000) 2020-02-05T21:01:19.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:19.330+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.332018+0000) 2020-02-05T21:01:19.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:19.330+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.332397+0000) 2020-02-05T21:01:19.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:19.332+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.333179+0000) 2020-02-05T21:01:19.334 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:19.332+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.333189+0000) 2020-02-05T21:01:19.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:19.332+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.333594+0000) 2020-02-05T21:01:19.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:19.332+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.333618+0000) 2020-02-05T21:01:19.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:19.334+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.335509+0000) 2020-02-05T21:01:19.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:19.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.334941+0000) 2020-02-05T21:01:19.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:19.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.336109+0000) 2020-02-05T21:01:19.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:19.335+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.336067+0000) 2020-02-05T21:01:19.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:19.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.338375+0000) 2020-02-05T21:01:19.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:19.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.338601+0000) 2020-02-05T21:01:19.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:19.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73656 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:19.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:19.343+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.344370+0000) 2020-02-05T21:01:19.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:19.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:19.953888+0000) 2020-02-05T21:01:20.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:20.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:20.204488+0000) 2020-02-05T21:01:20.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:20.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:20.214370+0000) 2020-02-05T21:01:22.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:22.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:22.193912+0000) 2020-02-05T21:01:22.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:22.307+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:22.308549+0000) 2020-02-05T21:01:23.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:23.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:23.434978+0000) 2020-02-05T21:01:23.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:23.464+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:23.465128+0000) 2020-02-05T21:01:24.346 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:24.344+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.345369+0000) 2020-02-05T21:01:24.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:24.344+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.345811+0000) 2020-02-05T21:01:24.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:24.345+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.346377+0000) 2020-02-05T21:01:24.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:24.345+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.346902+0000) 2020-02-05T21:01:24.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:24.345+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.347207+0000) 2020-02-05T21:01:24.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:24.346+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.348391+0000) 2020-02-05T21:01:24.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:24.348+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.349879+0000) 2020-02-05T21:01:24.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:24.348+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.350382+0000) 2020-02-05T21:01:24.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:24.348+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.350372+0000) 2020-02-05T21:01:24.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:24.349+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.350712+0000) 2020-02-05T21:01:24.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:24.349+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.350890+0000) 2020-02-05T21:01:24.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:24.353+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.354491+0000) 2020-02-05T21:01:24.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:24.353+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.354749+0000) 2020-02-05T21:01:24.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:24.353+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:24.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:24.354+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:24.355603+0000) 2020-02-05T21:01:25.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:25.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:25.748036+0000) 2020-02-05T21:01:25.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:25.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:25.809903+0000) 2020-02-05T21:01:25.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:25.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:25.846748+0000) 2020-02-05T21:01:26.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:26.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:26.706782+0000) 2020-02-05T21:01:26.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:26.950+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:26.951032+0000) 2020-02-05T21:01:26.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:26.950+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:26.951218+0000) 2020-02-05T21:01:29.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:29.345+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.345334+0000) 2020-02-05T21:01:29.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:29.361+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.362937+0000) 2020-02-05T21:01:29.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:29.362+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.363849+0000) 2020-02-05T21:01:29.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:29.362+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.363994+0000) 2020-02-05T21:01:29.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:29.363+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.364713+0000) 2020-02-05T21:01:29.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:29.364+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.365306+0000) 2020-02-05T21:01:29.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:29.364+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.365737+0000) 2020-02-05T21:01:29.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:29.365+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.366494+0000) 2020-02-05T21:01:29.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:29.365+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.366685+0000) 2020-02-05T21:01:29.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:29.365+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.367163+0000) 2020-02-05T21:01:29.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:29.365+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.367299+0000) 2020-02-05T21:01:29.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:29.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:29.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:29.365+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.366299+0000) 2020-02-05T21:01:29.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:29.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.366700+0000) 2020-02-05T21:01:29.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:29.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.366911+0000) 2020-02-05T21:01:29.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:29.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.373406+0000) 2020-02-05T21:01:29.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:29.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:29.954127+0000) 2020-02-05T21:01:30.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:30.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:30.204724+0000) 2020-02-05T21:01:30.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:30.214+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:30.214579+0000) 2020-02-05T21:01:32.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:32.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:32.194159+0000) 2020-02-05T21:01:32.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:32.308+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:32.308787+0000) 2020-02-05T21:01:33.415 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:01:33.421 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:01:33.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:33.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:33.435280+0000) 2020-02-05T21:01:33.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:33.465+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:33.465305+0000) 2020-02-05T21:01:34.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:34.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.376973+0000) 2020-02-05T21:01:34.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:34.377+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.377493+0000) 2020-02-05T21:01:34.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:34.378+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.378418+0000) 2020-02-05T21:01:34.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:34.378+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.378588+0000) 2020-02-05T21:01:34.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:34.379+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.379367+0000) 2020-02-05T21:01:34.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:34.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.379376+0000) 2020-02-05T21:01:34.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:34.379+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.379939+0000) 2020-02-05T21:01:34.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:34.379+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.379705+0000) 2020-02-05T21:01:34.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:34.380+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.380941+0000) 2020-02-05T21:01:34.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:34.381+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.382026+0000) 2020-02-05T21:01:34.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:34.381+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.382227+0000) 2020-02-05T21:01:34.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:34.382+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.382632+0000) 2020-02-05T21:01:34.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:34.383+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.383367+0000) 2020-02-05T21:01:34.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:34.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:34.385294+0000) 2020-02-05T21:01:34.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:34.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:35.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:35.756+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:35.756255+0000) 2020-02-05T21:01:35.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:35.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:35.810161+0000) 2020-02-05T21:01:35.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:35.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:35.846973+0000) 2020-02-05T21:01:36.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:36.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:36.707016+0000) 2020-02-05T21:01:36.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:36.950+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:36.951284+0000) 2020-02-05T21:01:36.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:36.951+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:36.951476+0000) 2020-02-05T21:01:39.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:39.344+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.345586+0000) 2020-02-05T21:01:39.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:39.390+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.392015+0000) 2020-02-05T21:01:39.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:39.391+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.392484+0000) 2020-02-05T21:01:39.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:39.393+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.394723+0000) 2020-02-05T21:01:39.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:39.394+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.395072+0000) 2020-02-05T21:01:39.397 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:39.395+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.396233+0000) 2020-02-05T21:01:39.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:39.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.396420+0000) 2020-02-05T21:01:39.398 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:39.395+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.396620+0000) 2020-02-05T21:01:39.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:39.396+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.397373+0000) 2020-02-05T21:01:39.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:39.396+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.397848+0000) 2020-02-05T21:01:39.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:39.397+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.397489+0000) 2020-02-05T21:01:39.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:39.397+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.397908+0000) 2020-02-05T21:01:39.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:39.397+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.398311+0000) 2020-02-05T21:01:39.400 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:39.397+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.398367+0000) 2020-02-05T21:01:39.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:39.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:39.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:39.400+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.401259+0000) 2020-02-05T21:01:39.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:39.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:39.954343+0000) 2020-02-05T21:01:40.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:40.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:40.205003+0000) 2020-02-05T21:01:40.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:40.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:40.214788+0000) 2020-02-05T21:01:42.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:42.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:42.194416+0000) 2020-02-05T21:01:42.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:42.308+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:42.309025+0000) 2020-02-05T21:01:43.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:43.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:43.435540+0000) 2020-02-05T21:01:43.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:43.464+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:43.465550+0000) 2020-02-05T21:01:44.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:44.406+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.406944+0000) 2020-02-05T21:01:44.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:44.406+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.407720+0000) 2020-02-05T21:01:44.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:44.408+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.409233+0000) 2020-02-05T21:01:44.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:44.408+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.409787+0000) 2020-02-05T21:01:44.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:44.409+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.410328+0000) 2020-02-05T21:01:44.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:44.408+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.408538+0000) 2020-02-05T21:01:44.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:44.409+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.410703+0000) 2020-02-05T21:01:44.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:44.409+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.410968+0000) 2020-02-05T21:01:44.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:44.410+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.411177+0000) 2020-02-05T21:01:44.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:44.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.412261+0000) 2020-02-05T21:01:44.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:44.411+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.412570+0000) 2020-02-05T21:01:44.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:44.412+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.412970+0000) 2020-02-05T21:01:44.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:44.412+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.413088+0000) 2020-02-05T21:01:44.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:44.415+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:44.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:44.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:44.419417+0000) 2020-02-05T21:01:45.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:45.763+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:45.764802+0000) 2020-02-05T21:01:45.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:45.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:45.810398+0000) 2020-02-05T21:01:45.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:45.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:45.847200+0000) 2020-02-05T21:01:46.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:46.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:46.707287+0000) 2020-02-05T21:01:46.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:46.951+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:46.951677+0000) 2020-02-05T21:01:46.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:46.951+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:46.951789+0000) 2020-02-05T21:01:49.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:49.345+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.345866+0000) 2020-02-05T21:01:49.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:49.422+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.423276+0000) 2020-02-05T21:01:49.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:49.422+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.423524+0000) 2020-02-05T21:01:49.425 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:49.423+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.424225+0000) 2020-02-05T21:01:49.426 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:49.423+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.424866+0000) 2020-02-05T21:01:49.426 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:49.424+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.425400+0000) 2020-02-05T21:01:49.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:49.425+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.425586+0000) 2020-02-05T21:01:49.427 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:49.424+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.425997+0000) 2020-02-05T21:01:49.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:49.427+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.428146+0000) 2020-02-05T21:01:49.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:49.428+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.428584+0000) 2020-02-05T21:01:49.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:49.428+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.428799+0000) 2020-02-05T21:01:49.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:49.425+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.426897+0000) 2020-02-05T21:01:49.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:49.426+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.427512+0000) 2020-02-05T21:01:49.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:49.426+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.427856+0000) 2020-02-05T21:01:49.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:49.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:49.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:49.432+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.433104+0000) 2020-02-05T21:01:49.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:49.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:49.954550+0000) 2020-02-05T21:01:50.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:50.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:50.205231+0000) 2020-02-05T21:01:50.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:50.213+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:50.215000+0000) 2020-02-05T21:01:52.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:52.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:52.194728+0000) 2020-02-05T21:01:52.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:52.309+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:52.309351+0000) 2020-02-05T21:01:53.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:53.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:53.435734+0000) 2020-02-05T21:01:53.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:53.465+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:53.465788+0000) 2020-02-05T21:01:54.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:54.438+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.439239+0000) 2020-02-05T21:01:54.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:54.439+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.440276+0000) 2020-02-05T21:01:54.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:54.440+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.441269+0000) 2020-02-05T21:01:54.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:54.441+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.442260+0000) 2020-02-05T21:01:54.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:54.441+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.442300+0000) 2020-02-05T21:01:54.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:54.442+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.442634+0000) 2020-02-05T21:01:54.447 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:54.437+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.438508+0000) 2020-02-05T21:01:54.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:54.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.439197+0000) 2020-02-05T21:01:54.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:54.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.439711+0000) 2020-02-05T21:01:54.448 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:54.439+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.440973+0000) 2020-02-05T21:01:54.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:54.440+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.441319+0000) 2020-02-05T21:01:54.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:54.440+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.441787+0000) 2020-02-05T21:01:54.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:54.440+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.442045+0000) 2020-02-05T21:01:54.449 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:54.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73947 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:54.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:54.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:54.446168+0000) 2020-02-05T21:01:55.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:55.772+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:55.772941+0000) 2020-02-05T21:01:55.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:55.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:55.810625+0000) 2020-02-05T21:01:55.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:55.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:55.847431+0000) 2020-02-05T21:01:56.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:56.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:56.707513+0000) 2020-02-05T21:01:56.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:56.951+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:56.951932+0000) 2020-02-05T21:01:56.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:56.951+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:56.952047+0000) 2020-02-05T21:01:59.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:59.345+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.346129+0000) 2020-02-05T21:01:59.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:01:59.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.453160+0000) 2020-02-05T21:01:59.460 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:01:59.453+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.454319+0000) 2020-02-05T21:01:59.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:01:59.456+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.456779+0000) 2020-02-05T21:01:59.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:01:59.457+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.457456+0000) 2020-02-05T21:01:59.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:01:59.457+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.457641+0000) 2020-02-05T21:01:59.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:59.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.451826+0000) 2020-02-05T21:01:59.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:59.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.456520+0000) 2020-02-05T21:01:59.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:01:59.452+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.453443+0000) 2020-02-05T21:01:59.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:01:59.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.454554+0000) 2020-02-05T21:01:59.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:01:59.456+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.457116+0000) 2020-02-05T21:01:59.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:01:59.457+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.458327+0000) 2020-02-05T21:01:59.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:59.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.458506+0000) 2020-02-05T21:01:59.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:01:59.458+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.458473+0000) 2020-02-05T21:01:59.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:01:59.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 73988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:01:59.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:01:59.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.461464+0000) 2020-02-05T21:01:59.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:01:59.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:01:59.954760+0000) 2020-02-05T21:02:00.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:00.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:00.205476+0000) 2020-02-05T21:02:00.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:00.214+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:00.215211+0000) 2020-02-05T21:02:02.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:02.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:02.194980+0000) 2020-02-05T21:02:02.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:02.309+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:02.309617+0000) 2020-02-05T21:02:03.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:03.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:03.435969+0000) 2020-02-05T21:02:03.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:03.465+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:03.466024+0000) 2020-02-05T21:02:03.554 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:02:03.560 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:02:04.469 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:04.466+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.467886+0000) 2020-02-05T21:02:04.470 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:04.467+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.468902+0000) 2020-02-05T21:02:04.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:04.468+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.469357+0000) 2020-02-05T21:02:04.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:04.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.470333+0000) 2020-02-05T21:02:04.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:04.470+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.470564+0000) 2020-02-05T21:02:04.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:04.470+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.471757+0000) 2020-02-05T21:02:04.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:04.471+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.471555+0000) 2020-02-05T21:02:04.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:04.471+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.472088+0000) 2020-02-05T21:02:04.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:04.472+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.472886+0000) 2020-02-05T21:02:04.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:04.473+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.473729+0000) 2020-02-05T21:02:04.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:04.472+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.473797+0000) 2020-02-05T21:02:04.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:04.473+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.473898+0000) 2020-02-05T21:02:04.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:04.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.474992+0000) 2020-02-05T21:02:04.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:04.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74029 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:04.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:04.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:04.483632+0000) 2020-02-05T21:02:05.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:05.780+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:05.781314+0000) 2020-02-05T21:02:05.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:05.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:05.810862+0000) 2020-02-05T21:02:05.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:05.847+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:05.847668+0000) 2020-02-05T21:02:06.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:06.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:06.707753+0000) 2020-02-05T21:02:06.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:06.952+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:06.952177+0000) 2020-02-05T21:02:06.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:06.952+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:06.952283+0000) 2020-02-05T21:02:09.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:09.345+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.346320+0000) 2020-02-05T21:02:09.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:09.480+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.481686+0000) 2020-02-05T21:02:09.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:09.481+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.482263+0000) 2020-02-05T21:02:09.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:09.481+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.482613+0000) 2020-02-05T21:02:09.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:09.482+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.483064+0000) 2020-02-05T21:02:09.485 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:09.482+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.483426+0000) 2020-02-05T21:02:09.486 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:09.482+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.484146+0000) 2020-02-05T21:02:09.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:09.483+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.484503+0000) 2020-02-05T21:02:09.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:09.482+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.483826+0000) 2020-02-05T21:02:09.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:09.483+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.484485+0000) 2020-02-05T21:02:09.490 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:09.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.485213+0000) 2020-02-05T21:02:09.491 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:09.483+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.485004+0000) 2020-02-05T21:02:09.492 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:09.483+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.485223+0000) 2020-02-05T21:02:09.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:09.484+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.485602+0000) 2020-02-05T21:02:09.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:09.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.490869+0000) 2020-02-05T21:02:09.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:09.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74072 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:09.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:09.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:09.954972+0000) 2020-02-05T21:02:10.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:10.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:10.205695+0000) 2020-02-05T21:02:10.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:10.214+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:10.215407+0000) 2020-02-05T21:02:12.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:12.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:12.195239+0000) 2020-02-05T21:02:12.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:12.308+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:12.309859+0000) 2020-02-05T21:02:13.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:13.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:13.436223+0000) 2020-02-05T21:02:13.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:13.465+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:13.466291+0000) 2020-02-05T21:02:14.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:14.494+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.495286+0000) 2020-02-05T21:02:14.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:14.496+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.497346+0000) 2020-02-05T21:02:14.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:14.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.498499+0000) 2020-02-05T21:02:14.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:14.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.499652+0000) 2020-02-05T21:02:14.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:14.501+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.502999+0000) 2020-02-05T21:02:14.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:14.500+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.501423+0000) 2020-02-05T21:02:14.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:14.501+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.502561+0000) 2020-02-05T21:02:14.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:14.495+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.496941+0000) 2020-02-05T21:02:14.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:14.497+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.498975+0000) 2020-02-05T21:02:14.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:14.498+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.500038+0000) 2020-02-05T21:02:14.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:14.499+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.500361+0000) 2020-02-05T21:02:14.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:14.499+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.500766+0000) 2020-02-05T21:02:14.512 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:14.500+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.501616+0000) 2020-02-05T21:02:14.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:14.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:14.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:14.507+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:14.508639+0000) 2020-02-05T21:02:15.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:15.792+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:15.793494+0000) 2020-02-05T21:02:15.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:15.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:15.811115+0000) 2020-02-05T21:02:15.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:15.846+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:15.847918+0000) 2020-02-05T21:02:16.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:16.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:16.707984+0000) 2020-02-05T21:02:16.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:16.951+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:16.952467+0000) 2020-02-05T21:02:16.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:16.951+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:16.952563+0000) 2020-02-05T21:02:19.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:19.346+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.346555+0000) 2020-02-05T21:02:19.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:19.511+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.512354+0000) 2020-02-05T21:02:19.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:19.511+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.513206+0000) 2020-02-05T21:02:19.518 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:19.512+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.514179+0000) 2020-02-05T21:02:19.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:19.513+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.514914+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:19.513+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.514267+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:19.513+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.514606+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:19.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.515808+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:19.515+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.516263+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:19.515+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.516997+0000) 2020-02-05T21:02:19.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:19.516+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.517518+0000) 2020-02-05T21:02:19.520 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:19.516+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.518152+0000) 2020-02-05T21:02:19.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:19.516+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.518265+0000) 2020-02-05T21:02:19.520 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:19.517+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.518542+0000) 2020-02-05T21:02:19.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:19.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:19.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:19.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.524933+0000) 2020-02-05T21:02:19.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:19.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:19.955176+0000) 2020-02-05T21:02:20.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:20.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:20.205914+0000) 2020-02-05T21:02:20.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:20.215+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:20.215627+0000) 2020-02-05T21:02:22.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:22.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:22.195484+0000) 2020-02-05T21:02:22.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:22.309+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:22.310112+0000) 2020-02-05T21:02:23.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:23.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:23.436474+0000) 2020-02-05T21:02:23.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:23.466+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:23.466532+0000) 2020-02-05T21:02:24.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:24.524+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.525920+0000) 2020-02-05T21:02:24.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:24.525+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.526928+0000) 2020-02-05T21:02:24.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:24.525+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.527206+0000) 2020-02-05T21:02:24.530 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:24.527+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.528603+0000) 2020-02-05T21:02:24.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:24.528+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.529473+0000) 2020-02-05T21:02:24.531 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:24.528+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.529958+0000) 2020-02-05T21:02:24.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:24.529+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.530347+0000) 2020-02-05T21:02:24.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:24.529+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.530842+0000) 2020-02-05T21:02:24.532 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:24.530+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.531583+0000) 2020-02-05T21:02:24.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:24.530+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.531794+0000) 2020-02-05T21:02:24.533 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:24.530+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.532067+0000) 2020-02-05T21:02:24.534 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:24.531+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.532399+0000) 2020-02-05T21:02:24.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:24.531+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.532853+0000) 2020-02-05T21:02:24.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:24.534+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:24.535209+0000) 2020-02-05T21:02:24.536 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:24.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:25.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:25.801+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:25.801462+0000) 2020-02-05T21:02:25.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:25.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:25.811336+0000) 2020-02-05T21:02:25.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:25.847+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:25.848152+0000) 2020-02-05T21:02:26.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:26.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:26.708223+0000) 2020-02-05T21:02:26.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:26.951+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:26.952704+0000) 2020-02-05T21:02:26.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:26.951+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:26.952818+0000) 2020-02-05T21:02:29.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:29.346+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.346785+0000) 2020-02-05T21:02:29.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:29.539+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.540914+0000) 2020-02-05T21:02:29.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:29.541+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.541911+0000) 2020-02-05T21:02:29.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:29.540+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.541878+0000) 2020-02-05T21:02:29.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:29.542+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.543029+0000) 2020-02-05T21:02:29.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:29.541+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.542559+0000) 2020-02-05T21:02:29.544 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:29.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.543526+0000) 2020-02-05T21:02:29.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:29.544+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.544718+0000) 2020-02-05T21:02:29.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:29.545+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.545685+0000) 2020-02-05T21:02:29.547 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:29.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.545635+0000) 2020-02-05T21:02:29.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:29.546+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.546228+0000) 2020-02-05T21:02:29.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:29.545+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.546841+0000) 2020-02-05T21:02:29.549 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:29.545+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.547143+0000) 2020-02-05T21:02:29.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:29.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.550863+0000) 2020-02-05T21:02:29.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:29.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.551106+0000) 2020-02-05T21:02:29.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:29.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:29.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:29.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:29.955408+0000) 2020-02-05T21:02:30.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:30.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:30.206146+0000) 2020-02-05T21:02:30.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:30.215+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:30.215870+0000) 2020-02-05T21:02:32.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:32.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:32.195740+0000) 2020-02-05T21:02:32.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:32.309+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:32.310410+0000) 2020-02-05T21:02:33.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:33.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:33.436720+0000) 2020-02-05T21:02:33.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:33.465+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:33.466767+0000) 2020-02-05T21:02:33.692 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:02:33.698 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:02:34.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:34.555+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.556920+0000) 2020-02-05T21:02:34.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:34.555+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.557309+0000) 2020-02-05T21:02:34.559 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:34.556+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.557641+0000) 2020-02-05T21:02:34.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:34.557+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.558393+0000) 2020-02-05T21:02:34.560 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:34.557+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.558651+0000) 2020-02-05T21:02:34.560 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:34.557+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.558844+0000) 2020-02-05T21:02:34.561 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:34.558+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.559890+0000) 2020-02-05T21:02:34.562 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:34.559+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.560315+0000) 2020-02-05T21:02:34.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:34.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.560961+0000) 2020-02-05T21:02:34.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:34.559+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.560960+0000) 2020-02-05T21:02:34.563 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:34.560+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.561543+0000) 2020-02-05T21:02:34.563 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:34.560+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.561875+0000) 2020-02-05T21:02:34.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:34.560+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.562187+0000) 2020-02-05T21:02:34.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:34.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:34.566274+0000) 2020-02-05T21:02:34.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:34.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:35.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:35.808+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:35.809424+0000) 2020-02-05T21:02:35.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:35.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:35.811553+0000) 2020-02-05T21:02:35.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:35.847+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:35.848467+0000) 2020-02-05T21:02:36.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:36.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:36.708412+0000) 2020-02-05T21:02:36.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:36.952+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:36.952937+0000) 2020-02-05T21:02:36.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:36.952+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:36.953091+0000) 2020-02-05T21:02:39.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:39.346+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.347017+0000) 2020-02-05T21:02:39.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:39.572+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.573414+0000) 2020-02-05T21:02:39.575 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:39.572+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.574123+0000) 2020-02-05T21:02:39.576 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:39.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.573961+0000) 2020-02-05T21:02:39.576 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:39.573+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.574684+0000) 2020-02-05T21:02:39.576 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:39.573+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.574990+0000) 2020-02-05T21:02:39.577 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:39.574+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.575644+0000) 2020-02-05T21:02:39.577 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:39.574+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.575855+0000) 2020-02-05T21:02:39.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:39.575+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.576370+0000) 2020-02-05T21:02:39.578 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:39.575+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.576484+0000) 2020-02-05T21:02:39.578 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:39.575+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.577297+0000) 2020-02-05T21:02:39.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:39.576+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.577468+0000) 2020-02-05T21:02:39.579 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:39.576+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.578094+0000) 2020-02-05T21:02:39.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:39.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.579383+0000) 2020-02-05T21:02:39.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:39.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:39.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:39.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.582782+0000) 2020-02-05T21:02:39.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:39.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:39.955628+0000) 2020-02-05T21:02:40.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:40.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:40.206384+0000) 2020-02-05T21:02:40.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:40.215+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:40.216102+0000) 2020-02-05T21:02:42.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:42.195+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:42.196011+0000) 2020-02-05T21:02:42.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:42.310+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:42.310706+0000) 2020-02-05T21:02:43.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:43.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:43.436952+0000) 2020-02-05T21:02:43.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:43.466+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:43.467000+0000) 2020-02-05T21:02:44.591 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:44.588+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.589456+0000) 2020-02-05T21:02:44.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:44.589+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.590219+0000) 2020-02-05T21:02:44.591 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:44.588+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.590316+0000) 2020-02-05T21:02:44.592 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:44.589+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.591034+0000) 2020-02-05T21:02:44.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:44.590+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.591529+0000) 2020-02-05T21:02:44.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:44.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.591455+0000) 2020-02-05T21:02:44.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:44.590+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.592299+0000) 2020-02-05T21:02:44.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:44.591+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.592062+0000) 2020-02-05T21:02:44.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:44.591+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.592843+0000) 2020-02-05T21:02:44.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:44.591+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.592944+0000) 2020-02-05T21:02:44.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:44.592+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.593597+0000) 2020-02-05T21:02:44.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:44.593+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.594207+0000) 2020-02-05T21:02:44.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:44.593+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.594635+0000) 2020-02-05T21:02:44.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:44.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:44.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:44.598+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:44.599481+0000) 2020-02-05T21:02:45.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:45.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:45.811813+0000) 2020-02-05T21:02:45.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:45.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:45.819863+0000) 2020-02-05T21:02:45.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:45.847+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:45.848708+0000) 2020-02-05T21:02:46.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:46.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:46.708647+0000) 2020-02-05T21:02:46.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:46.952+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:46.953169+0000) 2020-02-05T21:02:46.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:46.952+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:46.953356+0000) 2020-02-05T21:02:49.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:49.346+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.347319+0000) 2020-02-05T21:02:49.612 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:49.604+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.605195+0000) 2020-02-05T21:02:49.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:49.605+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.606610+0000) 2020-02-05T21:02:49.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:49.606+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.607438+0000) 2020-02-05T21:02:49.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:49.606+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.607872+0000) 2020-02-05T21:02:49.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:49.608+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.609409+0000) 2020-02-05T21:02:49.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:49.607+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.608359+0000) 2020-02-05T21:02:49.614 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:49.608+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.609830+0000) 2020-02-05T21:02:49.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:49.603+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.604513+0000) 2020-02-05T21:02:49.615 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:49.604+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.605491+0000) 2020-02-05T21:02:49.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:49.605+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.606950+0000) 2020-02-05T21:02:49.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:49.607+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.608652+0000) 2020-02-05T21:02:49.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:49.607+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.609341+0000) 2020-02-05T21:02:49.616 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:49.608+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.609770+0000) 2020-02-05T21:02:49.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:49.611+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74403 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:49.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:49.615+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.616304+0000) 2020-02-05T21:02:49.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:49.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:49.955910+0000) 2020-02-05T21:02:50.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:50.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:50.206636+0000) 2020-02-05T21:02:50.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:50.215+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:50.216351+0000) 2020-02-05T21:02:52.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:52.195+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:52.196249+0000) 2020-02-05T21:02:52.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:52.310+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:52.310976+0000) 2020-02-05T21:02:53.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:53.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:53.437191+0000) 2020-02-05T21:02:53.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:53.466+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:53.467311+0000) 2020-02-05T21:02:54.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:54.618+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.619450+0000) 2020-02-05T21:02:54.621 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:54.618+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.619719+0000) 2020-02-05T21:02:54.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:54.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.619983+0000) 2020-02-05T21:02:54.621 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:54.619+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.620378+0000) 2020-02-05T21:02:54.622 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:54.619+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.620828+0000) 2020-02-05T21:02:54.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:54.620+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.621465+0000) 2020-02-05T21:02:54.623 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:54.620+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.621556+0000) 2020-02-05T21:02:54.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:54.621+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.622575+0000) 2020-02-05T21:02:54.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:54.621+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.622802+0000) 2020-02-05T21:02:54.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:54.621+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.622660+0000) 2020-02-05T21:02:54.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:54.622+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.623597+0000) 2020-02-05T21:02:54.625 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:54.622+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.623596+0000) 2020-02-05T21:02:54.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:54.623+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.624309+0000) 2020-02-05T21:02:54.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:54.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:54.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:54.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:54.627703+0000) 2020-02-05T21:02:55.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:55.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:55.812044+0000) 2020-02-05T21:02:55.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:55.827+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:55.828208+0000) 2020-02-05T21:02:55.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:55.847+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:55.848876+0000) 2020-02-05T21:02:56.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:56.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:56.708882+0000) 2020-02-05T21:02:56.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:56.952+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:56.953466+0000) 2020-02-05T21:02:56.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:56.952+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:56.953642+0000) 2020-02-05T21:02:59.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:59.346+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.347552+0000) 2020-02-05T21:02:59.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:59.630+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.631744+0000) 2020-02-05T21:02:59.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:02:59.631+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.632382+0000) 2020-02-05T21:02:59.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:02:59.631+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.632930+0000) 2020-02-05T21:02:59.635 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:02:59.632+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.633290+0000) 2020-02-05T21:02:59.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:02:59.632+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.633196+0000) 2020-02-05T21:02:59.637 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:02:59.634+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.634893+0000) 2020-02-05T21:02:59.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:02:59.635+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.635844+0000) 2020-02-05T21:02:59.639 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:02:59.635+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.636269+0000) 2020-02-05T21:02:59.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:59.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.634397+0000) 2020-02-05T21:02:59.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:59.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.634905+0000) 2020-02-05T21:02:59.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:02:59.635+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.636709+0000) 2020-02-05T21:02:59.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:02:59.636+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.637106+0000) 2020-02-05T21:02:59.640 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:02:59.636+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.636977+0000) 2020-02-05T21:02:59.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:02:59.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:02:59.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:02:59.641+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.642394+0000) 2020-02-05T21:02:59.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:02:59.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:02:59.956158+0000) 2020-02-05T21:03:00.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:00.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:00.206872+0000) 2020-02-05T21:03:00.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:00.215+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:00.216562+0000) 2020-02-05T21:03:02.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:02.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:02.196542+0000) 2020-02-05T21:03:02.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:02.310+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:02.311275+0000) 2020-02-05T21:03:03.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:03.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:03.437469+0000) 2020-02-05T21:03:03.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:03.466+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:03.467484+0000) 2020-02-05T21:03:03.830 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:03:03.837 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:03:04.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:04.646+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.647555+0000) 2020-02-05T21:03:04.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:04.646+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.647823+0000) 2020-02-05T21:03:04.649 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:04.647+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.648626+0000) 2020-02-05T21:03:04.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:04.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.650037+0000) 2020-02-05T21:03:04.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:04.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.650225+0000) 2020-02-05T21:03:04.652 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:04.649+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.650616+0000) 2020-02-05T21:03:04.652 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:04.650+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.651207+0000) 2020-02-05T21:03:04.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:04.650+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.651325+0000) 2020-02-05T21:03:04.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:04.651+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.651449+0000) 2020-02-05T21:03:04.653 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:04.651+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.652005+0000) 2020-02-05T21:03:04.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:04.651+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.652525+0000) 2020-02-05T21:03:04.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:04.652+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.652964+0000) 2020-02-05T21:03:04.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:04.653+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.653419+0000) 2020-02-05T21:03:04.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:04.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:04.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:04.658+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:04.659880+0000) 2020-02-05T21:03:05.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:05.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:05.812315+0000) 2020-02-05T21:03:05.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:05.836+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:05.836529+0000) 2020-02-05T21:03:05.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:05.848+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:05.849110+0000) 2020-02-05T21:03:06.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:06.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:06.709137+0000) 2020-02-05T21:03:06.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:06.953+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:06.953676+0000) 2020-02-05T21:03:06.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:06.953+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:06.953860+0000) 2020-02-05T21:03:09.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:09.347+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.347776+0000) 2020-02-05T21:03:09.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:09.662+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.663908+0000) 2020-02-05T21:03:09.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:09.663+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.664513+0000) 2020-02-05T21:03:09.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:09.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.664986+0000) 2020-02-05T21:03:09.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:09.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.666166+0000) 2020-02-05T21:03:09.667 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:09.665+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.665782+0000) 2020-02-05T21:03:09.668 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:09.665+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.666141+0000) 2020-02-05T21:03:09.668 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:09.666+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.666751+0000) 2020-02-05T21:03:09.668 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:09.667+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.667634+0000) 2020-02-05T21:03:09.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:09.667+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.667683+0000) 2020-02-05T21:03:09.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:09.667+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.668648+0000) 2020-02-05T21:03:09.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:09.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.668655+0000) 2020-02-05T21:03:09.673 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:09.667+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.668868+0000) 2020-02-05T21:03:09.673 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:09.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74565 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:09.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:09.671+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.672111+0000) 2020-02-05T21:03:09.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:09.671+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.672388+0000) 2020-02-05T21:03:09.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:09.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:09.956359+0000) 2020-02-05T21:03:10.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:10.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:10.207123+0000) 2020-02-05T21:03:10.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:10.216+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:10.216772+0000) 2020-02-05T21:03:12.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:12.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:12.196787+0000) 2020-02-05T21:03:12.313 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:12.311+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:12.311523+0000) 2020-02-05T21:03:13.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:13.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:13.437701+0000) 2020-02-05T21:03:13.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:13.467+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:13.467722+0000) 2020-02-05T21:03:14.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:14.676+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.677648+0000) 2020-02-05T21:03:14.680 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:14.677+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.678258+0000) 2020-02-05T21:03:14.680 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:14.678+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.678864+0000) 2020-02-05T21:03:14.681 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:14.679+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.680219+0000) 2020-02-05T21:03:14.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:14.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.680710+0000) 2020-02-05T21:03:14.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:14.679+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.680999+0000) 2020-02-05T21:03:14.683 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:14.681+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.681710+0000) 2020-02-05T21:03:14.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:14.681+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.682137+0000) 2020-02-05T21:03:14.683 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:14.682+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.682496+0000) 2020-02-05T21:03:14.684 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:14.682+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.682848+0000) 2020-02-05T21:03:14.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:14.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.683833+0000) 2020-02-05T21:03:14.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:14.682+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.683951+0000) 2020-02-05T21:03:14.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:14.683+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.683936+0000) 2020-02-05T21:03:14.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:14.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:14.685935+0000) 2020-02-05T21:03:14.689 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:14.687+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:15.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:15.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:15.812540+0000) 2020-02-05T21:03:15.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:15.843+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:15.844833+0000) 2020-02-05T21:03:15.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:15.848+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:15.849375+0000) 2020-02-05T21:03:16.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:16.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:16.709363+0000) 2020-02-05T21:03:16.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:16.953+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:16.953930+0000) 2020-02-05T21:03:16.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:16.953+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:16.954071+0000) 2020-02-05T21:03:19.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:19.347+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.348008+0000) 2020-02-05T21:03:19.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:19.692+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.693435+0000) 2020-02-05T21:03:19.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:19.693+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.693961+0000) 2020-02-05T21:03:19.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:19.693+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.694325+0000) 2020-02-05T21:03:19.696 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:19.694+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.694729+0000) 2020-02-05T21:03:19.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:19.693+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.695015+0000) 2020-02-05T21:03:19.696 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:19.694+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.695868+0000) 2020-02-05T21:03:19.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:19.695+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.696222+0000) 2020-02-05T21:03:19.698 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:19.696+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.697132+0000) 2020-02-05T21:03:19.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:19.696+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.697523+0000) 2020-02-05T21:03:19.699 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:19.697+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.697713+0000) 2020-02-05T21:03:19.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:19.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.698520+0000) 2020-02-05T21:03:19.700 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:19.698+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.699036+0000) 2020-02-05T21:03:19.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:19.699+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.699462+0000) 2020-02-05T21:03:19.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:19.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:19.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:19.702+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.703499+0000) 2020-02-05T21:03:19.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:19.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:19.956594+0000) 2020-02-05T21:03:20.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:20.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:20.207353+0000) 2020-02-05T21:03:20.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:20.216+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:20.216980+0000) 2020-02-05T21:03:22.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:22.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:22.197035+0000) 2020-02-05T21:03:22.313 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:22.311+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:22.311772+0000) 2020-02-05T21:03:23.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:23.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:23.437957+0000) 2020-02-05T21:03:23.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:23.467+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:23.467966+0000) 2020-02-05T21:03:24.711 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:24.709+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.709667+0000) 2020-02-05T21:03:24.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:24.711+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.711403+0000) 2020-02-05T21:03:24.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:24.710+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.711175+0000) 2020-02-05T21:03:24.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:24.712+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.712419+0000) 2020-02-05T21:03:24.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:24.712+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.712816+0000) 2020-02-05T21:03:24.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:24.713+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.713164+0000) 2020-02-05T21:03:24.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:24.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.713388+0000) 2020-02-05T21:03:24.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:24.714+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.714452+0000) 2020-02-05T21:03:24.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:24.719+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.719727+0000) 2020-02-05T21:03:24.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:24.713+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.713747+0000) 2020-02-05T21:03:24.727 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:24.714+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.714810+0000) 2020-02-05T21:03:24.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:24.714+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.714855+0000) 2020-02-05T21:03:24.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:24.714+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.714839+0000) 2020-02-05T21:03:24.728 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:24.715+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:24.715472+0000) 2020-02-05T21:03:24.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:24.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74684 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:25.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:25.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:25.812800+0000) 2020-02-05T21:03:25.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:25.849+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:25.849611+0000) 2020-02-05T21:03:25.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:25.852+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:25.852583+0000) 2020-02-05T21:03:26.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:26.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:26.709612+0000) 2020-02-05T21:03:26.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:26.954+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:26.954189+0000) 2020-02-05T21:03:26.956 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:26.954+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:26.954320+0000) 2020-02-05T21:03:29.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:29.347+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.348292+0000) 2020-02-05T21:03:29.725 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:29.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.724342+0000) 2020-02-05T21:03:29.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:29.724+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.725394+0000) 2020-02-05T21:03:29.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:29.724+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.725023+0000) 2020-02-05T21:03:29.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:29.724+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.725803+0000) 2020-02-05T21:03:29.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:29.725+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.726128+0000) 2020-02-05T21:03:29.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:29.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.726571+0000) 2020-02-05T21:03:29.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:29.726+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.727544+0000) 2020-02-05T21:03:29.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:29.726+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.727933+0000) 2020-02-05T21:03:29.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:29.729+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.729987+0000) 2020-02-05T21:03:29.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:29.729+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.730602+0000) 2020-02-05T21:03:29.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:29.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.730450+0000) 2020-02-05T21:03:29.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:29.730+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.730779+0000) 2020-02-05T21:03:29.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:29.730+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.731124+0000) 2020-02-05T21:03:29.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:29.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.733624+0000) 2020-02-05T21:03:29.737 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:29.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:29.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:29.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:29.956831+0000) 2020-02-05T21:03:30.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:30.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:30.207586+0000) 2020-02-05T21:03:30.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:30.216+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:30.217178+0000) 2020-02-05T21:03:32.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:32.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:32.197283+0000) 2020-02-05T21:03:32.313 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:32.311+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:32.312028+0000) 2020-02-05T21:03:33.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:33.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:33.438209+0000) 2020-02-05T21:03:33.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:33.467+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:33.468207+0000) 2020-02-05T21:03:33.961 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:03:33.967 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:03:34.744 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:34.742+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.742748+0000) 2020-02-05T21:03:34.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:34.742+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.743498+0000) 2020-02-05T21:03:34.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:34.742+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.743855+0000) 2020-02-05T21:03:34.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:34.743+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.744120+0000) 2020-02-05T21:03:34.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:34.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.745172+0000) 2020-02-05T21:03:34.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:34.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.745742+0000) 2020-02-05T21:03:34.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:34.745+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.746065+0000) 2020-02-05T21:03:34.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:34.745+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.746598+0000) 2020-02-05T21:03:34.748 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:34.745+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.746897+0000) 2020-02-05T21:03:34.751 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:34.746+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.747314+0000) 2020-02-05T21:03:34.752 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:34.746+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.747342+0000) 2020-02-05T21:03:34.752 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:34.747+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.747515+0000) 2020-02-05T21:03:34.752 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:34.747+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.747765+0000) 2020-02-05T21:03:34.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:34.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:34.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:34.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:34.752613+0000) 2020-02-05T21:03:35.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:35.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:35.813156+0000) 2020-02-05T21:03:35.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:35.848+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:35.849849+0000) 2020-02-05T21:03:35.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:35.859+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:35.860652+0000) 2020-02-05T21:03:36.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:36.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:36.709879+0000) 2020-02-05T21:03:36.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:36.954+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:36.954483+0000) 2020-02-05T21:03:36.956 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:36.954+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:36.954543+0000) 2020-02-05T21:03:39.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:39.347+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.348552+0000) 2020-02-05T21:03:39.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:39.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.756878+0000) 2020-02-05T21:03:39.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:39.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.758315+0000) 2020-02-05T21:03:39.760 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:39.757+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.758689+0000) 2020-02-05T21:03:39.761 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:39.757+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.759258+0000) 2020-02-05T21:03:39.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:39.759+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.760476+0000) 2020-02-05T21:03:39.762 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:39.759+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.760530+0000) 2020-02-05T21:03:39.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:39.759+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.760954+0000) 2020-02-05T21:03:39.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:39.760+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.761559+0000) 2020-02-05T21:03:39.763 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:39.761+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.762574+0000) 2020-02-05T21:03:39.764 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:39.761+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.763056+0000) 2020-02-05T21:03:39.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:39.761+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.763206+0000) 2020-02-05T21:03:39.765 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:39.762+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.763685+0000) 2020-02-05T21:03:39.765 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:39.762+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.763792+0000) 2020-02-05T21:03:39.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:39.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.764887+0000) 2020-02-05T21:03:39.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:39.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74809 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:39.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:39.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:39.957044+0000) 2020-02-05T21:03:40.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:40.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:40.207822+0000) 2020-02-05T21:03:40.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:40.216+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:40.217402+0000) 2020-02-05T21:03:42.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:42.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:42.197523+0000) 2020-02-05T21:03:42.313 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:42.311+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:42.312278+0000) 2020-02-05T21:03:43.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:43.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:43.438464+0000) 2020-02-05T21:03:43.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:43.467+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:43.468429+0000) 2020-02-05T21:03:44.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:44.771+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.772968+0000) 2020-02-05T21:03:44.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:44.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.774858+0000) 2020-02-05T21:03:44.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:44.774+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.775354+0000) 2020-02-05T21:03:44.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:44.774+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.775791+0000) 2020-02-05T21:03:44.777 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:44.775+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.776322+0000) 2020-02-05T21:03:44.778 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:44.776+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.777312+0000) 2020-02-05T21:03:44.779 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:44.776+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.777734+0000) 2020-02-05T21:03:44.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:44.777+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.778483+0000) 2020-02-05T21:03:44.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:44.777+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.778691+0000) 2020-02-05T21:03:44.780 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:44.777+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.779036+0000) 2020-02-05T21:03:44.780 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:44.777+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.779152+0000) 2020-02-05T21:03:44.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:44.778+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.779467+0000) 2020-02-05T21:03:44.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:44.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.782413+0000) 2020-02-05T21:03:44.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:44.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:44.782651+0000) 2020-02-05T21:03:44.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:44.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74848 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:45.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:45.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:45.813417+0000) 2020-02-05T21:03:45.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:45.849+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:45.850149+0000) 2020-02-05T21:03:45.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:45.867+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:45.868199+0000) 2020-02-05T21:03:46.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:46.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:46.710130+0000) 2020-02-05T21:03:46.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:46.953+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:46.954767+0000) 2020-02-05T21:03:46.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:46.953+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:46.954770+0000) 2020-02-05T21:03:49.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:49.348+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.348807+0000) 2020-02-05T21:03:49.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:49.788+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.789495+0000) 2020-02-05T21:03:49.791 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:49.789+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.790260+0000) 2020-02-05T21:03:49.791 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:49.789+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.790565+0000) 2020-02-05T21:03:49.792 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:49.790+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.791306+0000) 2020-02-05T21:03:49.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:49.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.791745+0000) 2020-02-05T21:03:49.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:49.791+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.792520+0000) 2020-02-05T21:03:49.794 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:49.791+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.792608+0000) 2020-02-05T21:03:49.795 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:49.791+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.792924+0000) 2020-02-05T21:03:49.795 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:49.791+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.792629+0000) 2020-02-05T21:03:49.795 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:49.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.793549+0000) 2020-02-05T21:03:49.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:49.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.794490+0000) 2020-02-05T21:03:49.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:49.792+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.794350+0000) 2020-02-05T21:03:49.796 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:49.793+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.795391+0000) 2020-02-05T21:03:49.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:49.796+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.797332+0000) 2020-02-05T21:03:49.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:49.797+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:49.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:49.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:49.957303+0000) 2020-02-05T21:03:50.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:50.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:50.208038+0000) 2020-02-05T21:03:50.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:50.216+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:50.217633+0000) 2020-02-05T21:03:52.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:52.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:52.197779+0000) 2020-02-05T21:03:52.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:52.311+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:52.312531+0000) 2020-02-05T21:03:53.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:53.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:53.438718+0000) 2020-02-05T21:03:53.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:53.467+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:53.468663+0000) 2020-02-05T21:03:54.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:54.803+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.804182+0000) 2020-02-05T21:03:54.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:54.803+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.804882+0000) 2020-02-05T21:03:54.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:54.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.805806+0000) 2020-02-05T21:03:54.807 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:54.804+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.806398+0000) 2020-02-05T21:03:54.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:54.805+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.806964+0000) 2020-02-05T21:03:54.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:54.806+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.807649+0000) 2020-02-05T21:03:54.809 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:54.806+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.807465+0000) 2020-02-05T21:03:54.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:54.807+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.808921+0000) 2020-02-05T21:03:54.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:54.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.809972+0000) 2020-02-05T21:03:54.811 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:54.808+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.810163+0000) 2020-02-05T21:03:54.812 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:54.809+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.810599+0000) 2020-02-05T21:03:54.812 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:54.809+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.811186+0000) 2020-02-05T21:03:54.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:54.810+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.811517+0000) 2020-02-05T21:03:54.815 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:54.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74930 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:54.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:54.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:54.816299+0000) 2020-02-05T21:03:55.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:55.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:55.813652+0000) 2020-02-05T21:03:55.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:55.849+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:55.850407+0000) 2020-02-05T21:03:55.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:55.875+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:55.876287+0000) 2020-02-05T21:03:56.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:56.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:56.710365+0000) 2020-02-05T21:03:56.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:56.953+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:56.955016+0000) 2020-02-05T21:03:56.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:56.953+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:56.955047+0000) 2020-02-05T21:03:59.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:59.348+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.349039+0000) 2020-02-05T21:03:59.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:03:59.819+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.820812+0000) 2020-02-05T21:03:59.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:03:59.819+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.820820+0000) 2020-02-05T21:03:59.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:59.820+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.821979+0000) 2020-02-05T21:03:59.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:03:59.821+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.822589+0000) 2020-02-05T21:03:59.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:03:59.821+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.822752+0000) 2020-02-05T21:03:59.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:03:59.822+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.823325+0000) 2020-02-05T21:03:59.826 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:03:59.822+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.824106+0000) 2020-02-05T21:03:59.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:03:59.823+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.824480+0000) 2020-02-05T21:03:59.828 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:03:59.823+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.824735+0000) 2020-02-05T21:03:59.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:03:59.824+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.825899+0000) 2020-02-05T21:03:59.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:03:59.825+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.826469+0000) 2020-02-05T21:03:59.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:59.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.825339+0000) 2020-02-05T21:03:59.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:59.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.825714+0000) 2020-02-05T21:03:59.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:03:59.827+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.828789+0000) 2020-02-05T21:03:59.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:03:59.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 74974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:03:59.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:03:59.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:03:59.957462+0000) 2020-02-05T21:04:00.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:00.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:00.208273+0000) 2020-02-05T21:04:00.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:00.217+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:00.217844+0000) 2020-02-05T21:04:02.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:02.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:02.198004+0000) 2020-02-05T21:04:02.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:02.312+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:02.312825+0000) 2020-02-05T21:04:03.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:03.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:03.439011+0000) 2020-02-05T21:04:03.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:03.468+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:03.468892+0000) 2020-02-05T21:04:04.102 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:04:04.108 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:04:04.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:04.834+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.836036+0000) 2020-02-05T21:04:04.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:04.835+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.836694+0000) 2020-02-05T21:04:04.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:04.836+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.837469+0000) 2020-02-05T21:04:04.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:04.835+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.837188+0000) 2020-02-05T21:04:04.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:04.836+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.837851+0000) 2020-02-05T21:04:04.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:04.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.838318+0000) 2020-02-05T21:04:04.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:04.839+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.840313+0000) 2020-02-05T21:04:04.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:04.838+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.839943+0000) 2020-02-05T21:04:04.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:04.839+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.840862+0000) 2020-02-05T21:04:04.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:04.839+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.841158+0000) 2020-02-05T21:04:04.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:04.840+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.841897+0000) 2020-02-05T21:04:04.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:04.840+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.842027+0000) 2020-02-05T21:04:04.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:04.840+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.842385+0000) 2020-02-05T21:04:04.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:04.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75012 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:04.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:04.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:04.847584+0000) 2020-02-05T21:04:05.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:05.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:05.813861+0000) 2020-02-05T21:04:05.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:05.849+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:05.850594+0000) 2020-02-05T21:04:05.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:05.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:05.883207+0000) 2020-02-05T21:04:06.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:06.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:06.710659+0000) 2020-02-05T21:04:06.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:06.953+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:06.955295+0000) 2020-02-05T21:04:06.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:06.953+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:06.955296+0000) 2020-02-05T21:04:09.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:09.349+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.349336+0000) 2020-02-05T21:04:09.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:09.849+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.850157+0000) 2020-02-05T21:04:09.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:09.851+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.852673+0000) 2020-02-05T21:04:09.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:09.850+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.852235+0000) 2020-02-05T21:04:09.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:09.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.853491+0000) 2020-02-05T21:04:09.855 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:09.852+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.853757+0000) 2020-02-05T21:04:09.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:09.853+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.854334+0000) 2020-02-05T21:04:09.856 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:09.853+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.854591+0000) 2020-02-05T21:04:09.856 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:09.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.854910+0000) 2020-02-05T21:04:09.856 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:09.853+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.855272+0000) 2020-02-05T21:04:09.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:09.854+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.855959+0000) 2020-02-05T21:04:09.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:09.855+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.856787+0000) 2020-02-05T21:04:09.858 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:09.855+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.856836+0000) 2020-02-05T21:04:09.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:09.855+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.857111+0000) 2020-02-05T21:04:09.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:09.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:09.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:09.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.862325+0000) 2020-02-05T21:04:09.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:09.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:09.957670+0000) 2020-02-05T21:04:10.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:10.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:10.208511+0000) 2020-02-05T21:04:10.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:10.217+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:10.218050+0000) 2020-02-05T21:04:12.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:12.197+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:12.198290+0000) 2020-02-05T21:04:12.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:12.312+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:12.313055+0000) 2020-02-05T21:04:13.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:13.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:13.439242+0000) 2020-02-05T21:04:13.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:13.468+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:13.469135+0000) 2020-02-05T21:04:14.867 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:14.864+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.866267+0000) 2020-02-05T21:04:14.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:14.865+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.866867+0000) 2020-02-05T21:04:14.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:14.866+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.867384+0000) 2020-02-05T21:04:14.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:14.866+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.867889+0000) 2020-02-05T21:04:14.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:14.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.868429+0000) 2020-02-05T21:04:14.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:14.867+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.868572+0000) 2020-02-05T21:04:14.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:14.867+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.868916+0000) 2020-02-05T21:04:14.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:14.868+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.869578+0000) 2020-02-05T21:04:14.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:14.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.870007+0000) 2020-02-05T21:04:14.873 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:14.869+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.870862+0000) 2020-02-05T21:04:14.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:14.870+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.871980+0000) 2020-02-05T21:04:14.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:14.871+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.872376+0000) 2020-02-05T21:04:14.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:14.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.872369+0000) 2020-02-05T21:04:14.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:14.874+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75094 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:14.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:14.875+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:14.876130+0000) 2020-02-05T21:04:15.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:15.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:15.814072+0000) 2020-02-05T21:04:15.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:15.850+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:15.850880+0000) 2020-02-05T21:04:15.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:15.894+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:15.895119+0000) 2020-02-05T21:04:16.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:16.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:16.710879+0000) 2020-02-05T21:04:16.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:16.954+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:16.955556+0000) 2020-02-05T21:04:16.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:16.954+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:16.955554+0000) 2020-02-05T21:04:19.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:19.349+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.349564+0000) 2020-02-05T21:04:19.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:19.883+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.883651+0000) 2020-02-05T21:04:19.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:19.881+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.883309+0000) 2020-02-05T21:04:19.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:19.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.884451+0000) 2020-02-05T21:04:19.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:19.884+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.884921+0000) 2020-02-05T21:04:19.886 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:19.883+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.884672+0000) 2020-02-05T21:04:19.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:19.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.885470+0000) 2020-02-05T21:04:19.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:19.885+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.885551+0000) 2020-02-05T21:04:19.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:19.884+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.885749+0000) 2020-02-05T21:04:19.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:19.884+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.886304+0000) 2020-02-05T21:04:19.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:19.884+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.886327+0000) 2020-02-05T21:04:19.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:19.887+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.887053+0000) 2020-02-05T21:04:19.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:19.887+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.887553+0000) 2020-02-05T21:04:19.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:19.886+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.887642+0000) 2020-02-05T21:04:19.893 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:19.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75138 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:19.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:19.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.892991+0000) 2020-02-05T21:04:19.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:19.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:19.957846+0000) 2020-02-05T21:04:20.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:20.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:20.208755+0000) 2020-02-05T21:04:20.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:20.218+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:20.218324+0000) 2020-02-05T21:04:22.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:22.197+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:22.198544+0000) 2020-02-05T21:04:22.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:22.312+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:22.313260+0000) 2020-02-05T21:04:23.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:23.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:23.439539+0000) 2020-02-05T21:04:23.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:23.468+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:23.469377+0000) 2020-02-05T21:04:24.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:24.897+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.898224+0000) 2020-02-05T21:04:24.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:24.897+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.898865+0000) 2020-02-05T21:04:24.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:24.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.899374+0000) 2020-02-05T21:04:24.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:24.898+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.900042+0000) 2020-02-05T21:04:24.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:24.900+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.901178+0000) 2020-02-05T21:04:24.902 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:24.899+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.900632+0000) 2020-02-05T21:04:24.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:24.900+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.901646+0000) 2020-02-05T21:04:24.903 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:24.900+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.901758+0000) 2020-02-05T21:04:24.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:24.901+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.902324+0000) 2020-02-05T21:04:24.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:24.901+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.902710+0000) 2020-02-05T21:04:24.904 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:24.901+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.902816+0000) 2020-02-05T21:04:24.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:24.901+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.903282+0000) 2020-02-05T21:04:24.905 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:24.901+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.903387+0000) 2020-02-05T21:04:24.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:24.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:24.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:24.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:24.911422+0000) 2020-02-05T21:04:25.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:25.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:25.814265+0000) 2020-02-05T21:04:25.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:25.850+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:25.851122+0000) 2020-02-05T21:04:25.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:25.905+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:25.906484+0000) 2020-02-05T21:04:26.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:26.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:26.711168+0000) 2020-02-05T21:04:26.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:26.955+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:26.955787+0000) 2020-02-05T21:04:26.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:26.955+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:26.955792+0000) 2020-02-05T21:04:29.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:29.348+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.349800+0000) 2020-02-05T21:04:29.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:29.911+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.912973+0000) 2020-02-05T21:04:29.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:29.913+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.914174+0000) 2020-02-05T21:04:29.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:29.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.914816+0000) 2020-02-05T21:04:29.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:29.914+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.915308+0000) 2020-02-05T21:04:29.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:29.914+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.915658+0000) 2020-02-05T21:04:29.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:29.915+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.916028+0000) 2020-02-05T21:04:29.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:29.915+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.916430+0000) 2020-02-05T21:04:29.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:29.916+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.916661+0000) 2020-02-05T21:04:29.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:29.916+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.917497+0000) 2020-02-05T21:04:29.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:29.917+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.917667+0000) 2020-02-05T21:04:29.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:29.918+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.918417+0000) 2020-02-05T21:04:29.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:29.918+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.918725+0000) 2020-02-05T21:04:29.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:29.918+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.919348+0000) 2020-02-05T21:04:29.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:29.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.923485+0000) 2020-02-05T21:04:29.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:29.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75221 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:29.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:29.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:29.958129+0000) 2020-02-05T21:04:30.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:30.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:30.208981+0000) 2020-02-05T21:04:30.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:30.217+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:30.218538+0000) 2020-02-05T21:04:32.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:32.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:32.198782+0000) 2020-02-05T21:04:32.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:32.313+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:32.313644+0000) 2020-02-05T21:04:33.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:33.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:33.439792+0000) 2020-02-05T21:04:33.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:33.468+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:33.469607+0000) 2020-02-05T21:04:34.241 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:04:34.247 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:04:34.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:34.929+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.930120+0000) 2020-02-05T21:04:34.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:34.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.931266+0000) 2020-02-05T21:04:34.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:34.930+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.931664+0000) 2020-02-05T21:04:34.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:34.930+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.931792+0000) 2020-02-05T21:04:34.933 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:34.931+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.932061+0000) 2020-02-05T21:04:34.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:34.931+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.932427+0000) 2020-02-05T21:04:34.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:34.931+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.932586+0000) 2020-02-05T21:04:34.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:34.932+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.933245+0000) 2020-02-05T21:04:34.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:34.932+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.933994+0000) 2020-02-05T21:04:34.935 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:34.933+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.934342+0000) 2020-02-05T21:04:34.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:34.934+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.934872+0000) 2020-02-05T21:04:34.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:34.934+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.935080+0000) 2020-02-05T21:04:34.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:34.935+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.935778+0000) 2020-02-05T21:04:34.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:34.938+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:34.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:34.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:34.940470+0000) 2020-02-05T21:04:35.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:35.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:35.814433+0000) 2020-02-05T21:04:35.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:35.850+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:35.851368+0000) 2020-02-05T21:04:35.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:35.912+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:35.913956+0000) 2020-02-05T21:04:36.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:36.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:36.711359+0000) 2020-02-05T21:04:36.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:36.955+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:36.956032+0000) 2020-02-05T21:04:36.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:36.955+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:36.956033+0000) 2020-02-05T21:04:39.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:39.349+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.350041+0000) 2020-02-05T21:04:39.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:39.943+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.944690+0000) 2020-02-05T21:04:39.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:39.944+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.945482+0000) 2020-02-05T21:04:39.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:39.945+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.946180+0000) 2020-02-05T21:04:39.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:39.945+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.946777+0000) 2020-02-05T21:04:39.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:39.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.947573+0000) 2020-02-05T21:04:39.949 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:39.947+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.947663+0000) 2020-02-05T21:04:39.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:39.948+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.948832+0000) 2020-02-05T21:04:39.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:39.947+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.949054+0000) 2020-02-05T21:04:39.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:39.948+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.949244+0000) 2020-02-05T21:04:39.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:39.949+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.950124+0000) 2020-02-05T21:04:39.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:39.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.950819+0000) 2020-02-05T21:04:39.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:39.950+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.951015+0000) 2020-02-05T21:04:39.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:39.951+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.952026+0000) 2020-02-05T21:04:39.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:39.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.954246+0000) 2020-02-05T21:04:39.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:39.954+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:39.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:39.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:39.958291+0000) 2020-02-05T21:04:40.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:40.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:40.209211+0000) 2020-02-05T21:04:40.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:40.218+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:40.218740+0000) 2020-02-05T21:04:42.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:42.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:42.199043+0000) 2020-02-05T21:04:42.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:42.313+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:42.313942+0000) 2020-02-05T21:04:43.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:43.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:43.440030+0000) 2020-02-05T21:04:43.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:43.468+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:43.469840+0000) 2020-02-05T21:04:44.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:44.959+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.960968+0000) 2020-02-05T21:04:44.963 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:44.960+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.961350+0000) 2020-02-05T21:04:44.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:44.961+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.962484+0000) 2020-02-05T21:04:44.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:44.962+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.962654+0000) 2020-02-05T21:04:44.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:44.962+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.963391+0000) 2020-02-05T21:04:44.965 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:44.962+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.963899+0000) 2020-02-05T21:04:44.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:44.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.964679+0000) 2020-02-05T21:04:44.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:44.964+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.965378+0000) 2020-02-05T21:04:44.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:44.965+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.965728+0000) 2020-02-05T21:04:44.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:44.966+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.966603+0000) 2020-02-05T21:04:44.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:44.966+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.966733+0000) 2020-02-05T21:04:44.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:44.964+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.965436+0000) 2020-02-05T21:04:44.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:44.965+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.966294+0000) 2020-02-05T21:04:44.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:44.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:44.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:44.972+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:44.973569+0000) 2020-02-05T21:04:45.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:45.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:45.814657+0000) 2020-02-05T21:04:45.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:45.850+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:45.851568+0000) 2020-02-05T21:04:45.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:45.921+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:45.921848+0000) 2020-02-05T21:04:46.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:46.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:46.711593+0000) 2020-02-05T21:04:46.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:46.956+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:46.956293+0000) 2020-02-05T21:04:46.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:46.956+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:46.956288+0000) 2020-02-05T21:04:49.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:49.349+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.350278+0000) 2020-02-05T21:04:49.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:49.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.958779+0000) 2020-02-05T21:04:49.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:49.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.975005+0000) 2020-02-05T21:04:49.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:49.975+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.975567+0000) 2020-02-05T21:04:49.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:49.975+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.976519+0000) 2020-02-05T21:04:49.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:49.976+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.977181+0000) 2020-02-05T21:04:49.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:49.977+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.977681+0000) 2020-02-05T21:04:49.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:49.977+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.978768+0000) 2020-02-05T21:04:49.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:49.978+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.978615+0000) 2020-02-05T21:04:49.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:49.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.979452+0000) 2020-02-05T21:04:49.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:49.978+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.979787+0000) 2020-02-05T21:04:49.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:49.979+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.980068+0000) 2020-02-05T21:04:49.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:49.980+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.980414+0000) 2020-02-05T21:04:49.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:49.981+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.981881+0000) 2020-02-05T21:04:49.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:49.981+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.982218+0000) 2020-02-05T21:04:49.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:49.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75384 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:49.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:49.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:49.987896+0000) 2020-02-05T21:04:50.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:50.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:50.209472+0000) 2020-02-05T21:04:50.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:50.218+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:50.218956+0000) 2020-02-05T21:04:52.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:52.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:52.199293+0000) 2020-02-05T21:04:52.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:52.313+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:52.314197+0000) 2020-02-05T21:04:53.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:53.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:53.440281+0000) 2020-02-05T21:04:53.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:53.469+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:53.470129+0000) 2020-02-05T21:04:54.992 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:04:54.990+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.991321+0000) 2020-02-05T21:04:54.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:54.990+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.991814+0000) 2020-02-05T21:04:54.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:54.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.992650+0000) 2020-02-05T21:04:54.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:54.992+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.993198+0000) 2020-02-05T21:04:54.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:04:54.992+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.993372+0000) 2020-02-05T21:04:54.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:54.992+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.993662+0000) 2020-02-05T21:04:54.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:04:54.993+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.994346+0000) 2020-02-05T21:04:54.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:04:54.994+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.994804+0000) 2020-02-05T21:04:54.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:04:54.994+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.995743+0000) 2020-02-05T21:04:54.998 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:54.995+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.996948+0000) 2020-02-05T21:04:54.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:04:54.995+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.995663+0000) 2020-02-05T21:04:54.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:54.995+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.996378+0000) 2020-02-05T21:04:54.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:54.997+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:54.997861+0000) 2020-02-05T21:04:55.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:04:54.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:04:55.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:55.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:55.001497+0000) 2020-02-05T21:04:55.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:04:55.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:55.814860+0000) 2020-02-05T21:04:55.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:55.851+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:55.851773+0000) 2020-02-05T21:04:55.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:55.928+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:55.929041+0000) 2020-02-05T21:04:56.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:04:56.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:56.711781+0000) 2020-02-05T21:04:56.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:04:56.956+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:56.956546+0000) 2020-02-05T21:04:56.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:04:56.956+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:56.956550+0000) 2020-02-05T21:04:59.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:04:59.350+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:59.350511+0000) 2020-02-05T21:04:59.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:04:59.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:04:59.959029+0000) 2020-02-05T21:05:00.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:00.005+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.006981+0000) 2020-02-05T21:05:00.009 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:00.007+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.007642+0000) 2020-02-05T21:05:00.009 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:00.007+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.008059+0000) 2020-02-05T21:05:00.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:00.007+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.008737+0000) 2020-02-05T21:05:00.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:00.008+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.009689+0000) 2020-02-05T21:05:00.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:00.008+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.009903+0000) 2020-02-05T21:05:00.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:00.009+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.010326+0000) 2020-02-05T21:05:00.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:00.010+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.010638+0000) 2020-02-05T21:05:00.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:00.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.010780+0000) 2020-02-05T21:05:00.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:00.010+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.011453+0000) 2020-02-05T21:05:00.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:00.010+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.011870+0000) 2020-02-05T21:05:00.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:00.011+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.011740+0000) 2020-02-05T21:05:00.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:00.011+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.011955+0000) 2020-02-05T21:05:00.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:00.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:00.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:00.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.016848+0000) 2020-02-05T21:05:00.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:00.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.209695+0000) 2020-02-05T21:05:00.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:00.218+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:00.219164+0000) 2020-02-05T21:05:02.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:02.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:02.199599+0000) 2020-02-05T21:05:02.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:02.314+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:02.314466+0000) 2020-02-05T21:05:03.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:03.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:03.440521+0000) 2020-02-05T21:05:03.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:03.469+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:03.470361+0000) 2020-02-05T21:05:04.402 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:05:04.408 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:05:05.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:05.020+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.021837+0000) 2020-02-05T21:05:05.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:05.021+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.022572+0000) 2020-02-05T21:05:05.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:05.022+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.023137+0000) 2020-02-05T21:05:05.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:05.022+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.023776+0000) 2020-02-05T21:05:05.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:05.023+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.024108+0000) 2020-02-05T21:05:05.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:05.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.024420+0000) 2020-02-05T21:05:05.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:05.023+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.024911+0000) 2020-02-05T21:05:05.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:05.024+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.025289+0000) 2020-02-05T21:05:05.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:05.025+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.025650+0000) 2020-02-05T21:05:05.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:05.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.026527+0000) 2020-02-05T21:05:05.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:05.025+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.026389+0000) 2020-02-05T21:05:05.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:05.026+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.026806+0000) 2020-02-05T21:05:05.029 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:05.026+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.027108+0000) 2020-02-05T21:05:05.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:05.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.032203+0000) 2020-02-05T21:05:05.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:05.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:05.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:05.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.815104+0000) 2020-02-05T21:05:05.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:05.850+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.852011+0000) 2020-02-05T21:05:05.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:05.937+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:05.937292+0000) 2020-02-05T21:05:06.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:06.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:06.712042+0000) 2020-02-05T21:05:06.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:06.955+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:06.956807+0000) 2020-02-05T21:05:06.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:06.955+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:06.956807+0000) 2020-02-05T21:05:09.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:09.350+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:09.350783+0000) 2020-02-05T21:05:09.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:09.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:09.959339+0000) 2020-02-05T21:05:10.038 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:10.035+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.036500+0000) 2020-02-05T21:05:10.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:10.035+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.036839+0000) 2020-02-05T21:05:10.039 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:10.036+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.037989+0000) 2020-02-05T21:05:10.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:10.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.038709+0000) 2020-02-05T21:05:10.040 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:10.037+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.039192+0000) 2020-02-05T21:05:10.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:10.037+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.039331+0000) 2020-02-05T21:05:10.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:10.038+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.039945+0000) 2020-02-05T21:05:10.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:10.039+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.040432+0000) 2020-02-05T21:05:10.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:10.040+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.041907+0000) 2020-02-05T21:05:10.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:10.040+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.041149+0000) 2020-02-05T21:05:10.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:10.040+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.041611+0000) 2020-02-05T21:05:10.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:10.041+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.042677+0000) 2020-02-05T21:05:10.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:10.042+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.043635+0000) 2020-02-05T21:05:10.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:10.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75548 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:10.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:10.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.048954+0000) 2020-02-05T21:05:10.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:10.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.209926+0000) 2020-02-05T21:05:10.222 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:10.219+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:10.219395+0000) 2020-02-05T21:05:12.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:12.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:12.199842+0000) 2020-02-05T21:05:12.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:12.313+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:12.314717+0000) 2020-02-05T21:05:13.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:13.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:13.440760+0000) 2020-02-05T21:05:13.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:13.470+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:13.470596+0000) 2020-02-05T21:05:15.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:15.049+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.051147+0000) 2020-02-05T21:05:15.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:15.051+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.051770+0000) 2020-02-05T21:05:15.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:15.052+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.052227+0000) 2020-02-05T21:05:15.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:15.051+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.052912+0000) 2020-02-05T21:05:15.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:15.053+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.054496+0000) 2020-02-05T21:05:15.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:15.055+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.056819+0000) 2020-02-05T21:05:15.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:15.056+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.057867+0000) 2020-02-05T21:05:15.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:15.056+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.058051+0000) 2020-02-05T21:05:15.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:15.055+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.055369+0000) 2020-02-05T21:05:15.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:15.055+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.055722+0000) 2020-02-05T21:05:15.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:15.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.056621+0000) 2020-02-05T21:05:15.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:15.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.059851+0000) 2020-02-05T21:05:15.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:15.060+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.061061+0000) 2020-02-05T21:05:15.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:15.058+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.058189+0000) 2020-02-05T21:05:15.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:15.061+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:15.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:15.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.815315+0000) 2020-02-05T21:05:15.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:15.851+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.852269+0000) 2020-02-05T21:05:15.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:15.944+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:15.945586+0000) 2020-02-05T21:05:16.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:16.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:16.712260+0000) 2020-02-05T21:05:16.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:16.955+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:16.957051+0000) 2020-02-05T21:05:16.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:16.955+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:16.957086+0000) 2020-02-05T21:05:19.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:19.350+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:19.351043+0000) 2020-02-05T21:05:19.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:19.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:19.959635+0000) 2020-02-05T21:05:20.070 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:20.067+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.068707+0000) 2020-02-05T21:05:20.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:20.067+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.069329+0000) 2020-02-05T21:05:20.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:20.069+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.070131+0000) 2020-02-05T21:05:20.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:20.070+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.071367+0000) 2020-02-05T21:05:20.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:20.070+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.071826+0000) 2020-02-05T21:05:20.074 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:20.071+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.072435+0000) 2020-02-05T21:05:20.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:20.071+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.072418+0000) 2020-02-05T21:05:20.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:20.071+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.073163+0000) 2020-02-05T21:05:20.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:20.072+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.073868+0000) 2020-02-05T21:05:20.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:20.072+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.074169+0000) 2020-02-05T21:05:20.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:20.072+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.073846+0000) 2020-02-05T21:05:20.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:20.073+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.074838+0000) 2020-02-05T21:05:20.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:20.073+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.074581+0000) 2020-02-05T21:05:20.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:20.076+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.077563+0000) 2020-02-05T21:05:20.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:20.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:20.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:20.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.210164+0000) 2020-02-05T21:05:20.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:20.218+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:20.219612+0000) 2020-02-05T21:05:22.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:22.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:22.200109+0000) 2020-02-05T21:05:22.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:22.314+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:22.314963+0000) 2020-02-05T21:05:23.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:23.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:23.441012+0000) 2020-02-05T21:05:23.472 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:23.469+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:23.470826+0000) 2020-02-05T21:05:25.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:25.082+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.083650+0000) 2020-02-05T21:05:25.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:25.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.083860+0000) 2020-02-05T21:05:25.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:25.083+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.084465+0000) 2020-02-05T21:05:25.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:25.083+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.084810+0000) 2020-02-05T21:05:25.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:25.084+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.085283+0000) 2020-02-05T21:05:25.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:25.085+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.086263+0000) 2020-02-05T21:05:25.087 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:25.085+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.086393+0000) 2020-02-05T21:05:25.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:25.085+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.086927+0000) 2020-02-05T21:05:25.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:25.086+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.087589+0000) 2020-02-05T21:05:25.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:25.085+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.087171+0000) 2020-02-05T21:05:25.089 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:25.086+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.087684+0000) 2020-02-05T21:05:25.090 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:25.086+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.087607+0000) 2020-02-05T21:05:25.090 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:25.087+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.088589+0000) 2020-02-05T21:05:25.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:25.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:25.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:25.092+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.093661+0000) 2020-02-05T21:05:25.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:25.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.815544+0000) 2020-02-05T21:05:25.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:25.851+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.852501+0000) 2020-02-05T21:05:25.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:25.953+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:25.953952+0000) 2020-02-05T21:05:26.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:26.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:26.712484+0000) 2020-02-05T21:05:26.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:26.956+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:26.957320+0000) 2020-02-05T21:05:26.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:26.956+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:26.957391+0000) 2020-02-05T21:05:29.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:29.350+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:29.351329+0000) 2020-02-05T21:05:29.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:29.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:29.959902+0000) 2020-02-05T21:05:30.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:30.096+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.098240+0000) 2020-02-05T21:05:30.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:30.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.098838+0000) 2020-02-05T21:05:30.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:30.098+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.099638+0000) 2020-02-05T21:05:30.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:30.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.099125+0000) 2020-02-05T21:05:30.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:30.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.100607+0000) 2020-02-05T21:05:30.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:30.099+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.101135+0000) 2020-02-05T21:05:30.103 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:30.101+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.102604+0000) 2020-02-05T21:05:30.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:30.102+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.103299+0000) 2020-02-05T21:05:30.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:30.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.103626+0000) 2020-02-05T21:05:30.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:30.102+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.104201+0000) 2020-02-05T21:05:30.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:30.103+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.104689+0000) 2020-02-05T21:05:30.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:30.103+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.105175+0000) 2020-02-05T21:05:30.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:30.103+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.105251+0000) 2020-02-05T21:05:30.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:30.107+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75712 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:30.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:30.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.110329+0000) 2020-02-05T21:05:30.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:30.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.210407+0000) 2020-02-05T21:05:30.221 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:30.219+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:30.219836+0000) 2020-02-05T21:05:32.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:32.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:32.200354+0000) 2020-02-05T21:05:32.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:32.314+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:32.315193+0000) 2020-02-05T21:05:33.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:33.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:33.441263+0000) 2020-02-05T21:05:33.472 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:33.470+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:33.471060+0000) 2020-02-05T21:05:34.539 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:05:34.545 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:05:35.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:35.113+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.114532+0000) 2020-02-05T21:05:35.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:35.114+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.116076+0000) 2020-02-05T21:05:35.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:35.115+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.116382+0000) 2020-02-05T21:05:35.118 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:35.115+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.116738+0000) 2020-02-05T21:05:35.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:35.115+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.117365+0000) 2020-02-05T21:05:35.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:35.116+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.117322+0000) 2020-02-05T21:05:35.119 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:35.116+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.117605+0000) 2020-02-05T21:05:35.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:35.116+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.117987+0000) 2020-02-05T21:05:35.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:35.116+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.117630+0000) 2020-02-05T21:05:35.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:35.116+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.118163+0000) 2020-02-05T21:05:35.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:35.117+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.118938+0000) 2020-02-05T21:05:35.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:35.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.118664+0000) 2020-02-05T21:05:35.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:35.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.119690+0000) 2020-02-05T21:05:35.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:35.122+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:35.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:35.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.127152+0000) 2020-02-05T21:05:35.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:35.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.815757+0000) 2020-02-05T21:05:35.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:35.852+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.852711+0000) 2020-02-05T21:05:35.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:35.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:35.964840+0000) 2020-02-05T21:05:36.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:36.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:36.712719+0000) 2020-02-05T21:05:36.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:36.956+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:36.957632+0000) 2020-02-05T21:05:36.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:36.956+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:36.957678+0000) 2020-02-05T21:05:39.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:39.350+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:39.351561+0000) 2020-02-05T21:05:39.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:39.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:39.960135+0000) 2020-02-05T21:05:40.130 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:40.128+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.129223+0000) 2020-02-05T21:05:40.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:40.128+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.129986+0000) 2020-02-05T21:05:40.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:40.130+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.131853+0000) 2020-02-05T21:05:40.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:40.130+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.131513+0000) 2020-02-05T21:05:40.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:40.131+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.132562+0000) 2020-02-05T21:05:40.134 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:40.132+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.133329+0000) 2020-02-05T21:05:40.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:40.132+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.133830+0000) 2020-02-05T21:05:40.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:40.132+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.133985+0000) 2020-02-05T21:05:40.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:40.133+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.134741+0000) 2020-02-05T21:05:40.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:40.134+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.135191+0000) 2020-02-05T21:05:40.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:40.134+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.135824+0000) 2020-02-05T21:05:40.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:40.134+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.135967+0000) 2020-02-05T21:05:40.138 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:40.135+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.136975+0000) 2020-02-05T21:05:40.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:40.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:40.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:40.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.141289+0000) 2020-02-05T21:05:40.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:40.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.210707+0000) 2020-02-05T21:05:40.221 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:40.219+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:40.220053+0000) 2020-02-05T21:05:42.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:42.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:42.200625+0000) 2020-02-05T21:05:42.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:42.314+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:42.315469+0000) 2020-02-05T21:05:43.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:43.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:43.441519+0000) 2020-02-05T21:05:43.472 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:43.470+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:43.471348+0000) 2020-02-05T21:05:45.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:45.144+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.145378+0000) 2020-02-05T21:05:45.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:45.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.146592+0000) 2020-02-05T21:05:45.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:45.145+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.147025+0000) 2020-02-05T21:05:45.149 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:45.147+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.148510+0000) 2020-02-05T21:05:45.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:45.147+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.148675+0000) 2020-02-05T21:05:45.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:45.146+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.148052+0000) 2020-02-05T21:05:45.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:45.147+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.149169+0000) 2020-02-05T21:05:45.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:45.148+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.149745+0000) 2020-02-05T21:05:45.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:45.149+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.150909+0000) 2020-02-05T21:05:45.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:45.150+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.151545+0000) 2020-02-05T21:05:45.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:45.151+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.152149+0000) 2020-02-05T21:05:45.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:45.150+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.151666+0000) 2020-02-05T21:05:45.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:45.150+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.152194+0000) 2020-02-05T21:05:45.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:45.154+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.155472+0000) 2020-02-05T21:05:45.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:45.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75832 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:45.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:45.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.815929+0000) 2020-02-05T21:05:45.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:45.852+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.852936+0000) 2020-02-05T21:05:45.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:45.971+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:45.972073+0000) 2020-02-05T21:05:46.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:46.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:46.712944+0000) 2020-02-05T21:05:46.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:46.957+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:46.957889+0000) 2020-02-05T21:05:46.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:46.957+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:46.957968+0000) 2020-02-05T21:05:49.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:49.351+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:49.351798+0000) 2020-02-05T21:05:49.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:49.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:49.960365+0000) 2020-02-05T21:05:50.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:50.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.160615+0000) 2020-02-05T21:05:50.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:50.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.161654+0000) 2020-02-05T21:05:50.163 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:50.161+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.162172+0000) 2020-02-05T21:05:50.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:50.161+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.162541+0000) 2020-02-05T21:05:50.164 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:50.161+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.163143+0000) 2020-02-05T21:05:50.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:50.162+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.163943+0000) 2020-02-05T21:05:50.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:50.162+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.163472+0000) 2020-02-05T21:05:50.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:50.162+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.164088+0000) 2020-02-05T21:05:50.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:50.163+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.164972+0000) 2020-02-05T21:05:50.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:50.163+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.165322+0000) 2020-02-05T21:05:50.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:50.164+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.165499+0000) 2020-02-05T21:05:50.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:50.164+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.165651+0000) 2020-02-05T21:05:50.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:50.165+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.166530+0000) 2020-02-05T21:05:50.171 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:50.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75876 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:50.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:50.170+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.171855+0000) 2020-02-05T21:05:50.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:50.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.211000+0000) 2020-02-05T21:05:50.221 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:50.219+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:50.220330+0000) 2020-02-05T21:05:52.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:52.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:52.200862+0000) 2020-02-05T21:05:52.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:52.315+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:52.315771+0000) 2020-02-05T21:05:53.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:53.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:53.441783+0000) 2020-02-05T21:05:53.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:53.471+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:53.471580+0000) 2020-02-05T21:05:55.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:05:55.175+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.176310+0000) 2020-02-05T21:05:55.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:55.175+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.176766+0000) 2020-02-05T21:05:55.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:55.176+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.177200+0000) 2020-02-05T21:05:55.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:55.176+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.177948+0000) 2020-02-05T21:05:55.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:05:55.176+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.177726+0000) 2020-02-05T21:05:55.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:55.177+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.179101+0000) 2020-02-05T21:05:55.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:55.177+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.179285+0000) 2020-02-05T21:05:55.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:55.178+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.179392+0000) 2020-02-05T21:05:55.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:55.179+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.180220+0000) 2020-02-05T21:05:55.182 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:05:55.178+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.180057+0000) 2020-02-05T21:05:55.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:05:55.179+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.180946+0000) 2020-02-05T21:05:55.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:05:55.179+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.180742+0000) 2020-02-05T21:05:55.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:05:55.179+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.181237+0000) 2020-02-05T21:05:55.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:55.184+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.185671+0000) 2020-02-05T21:05:55.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:05:55.184+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:05:55.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:05:55.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.816135+0000) 2020-02-05T21:05:55.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:55.852+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.853166+0000) 2020-02-05T21:05:55.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:55.980+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:55.980362+0000) 2020-02-05T21:05:56.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:05:56.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:56.713271+0000) 2020-02-05T21:05:56.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:05:56.957+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:56.958129+0000) 2020-02-05T21:05:56.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:05:56.957+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:56.958197+0000) 2020-02-05T21:05:59.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:05:59.351+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:59.352035+0000) 2020-02-05T21:05:59.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:05:59.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:05:59.960600+0000) 2020-02-05T21:06:00.191 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:00.189+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.190516+0000) 2020-02-05T21:06:00.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:00.191+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.192126+0000) 2020-02-05T21:06:00.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:00.192+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.193821+0000) 2020-02-05T21:06:00.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:00.193+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.194269+0000) 2020-02-05T21:06:00.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:00.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.194538+0000) 2020-02-05T21:06:00.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:00.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.195116+0000) 2020-02-05T21:06:00.196 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:00.195+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.195633+0000) 2020-02-05T21:06:00.197 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:00.195+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.196245+0000) 2020-02-05T21:06:00.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:00.195+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.196456+0000) 2020-02-05T21:06:00.198 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:00.196+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.196553+0000) 2020-02-05T21:06:00.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:00.196+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.196813+0000) 2020-02-05T21:06:00.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:00.196+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.197293+0000) 2020-02-05T21:06:00.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:00.197+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.197708+0000) 2020-02-05T21:06:00.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:00.199+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.200070+0000) 2020-02-05T21:06:00.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:00.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:00.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:00.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.211180+0000) 2020-02-05T21:06:00.221 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:00.220+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:00.220534+0000) 2020-02-05T21:06:02.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:02.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:02.201104+0000) 2020-02-05T21:06:02.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:02.315+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:02.316027+0000) 2020-02-05T21:06:03.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:03.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:03.442046+0000) 2020-02-05T21:06:03.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:03.471+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:03.471816+0000) 2020-02-05T21:06:04.686 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:06:04.693 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:06:05.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:05.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.208612+0000) 2020-02-05T21:06:05.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:05.208+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.208730+0000) 2020-02-05T21:06:05.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:05.209+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.209333+0000) 2020-02-05T21:06:05.211 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:05.209+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.209891+0000) 2020-02-05T21:06:05.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:05.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.210471+0000) 2020-02-05T21:06:05.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:05.210+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.210842+0000) 2020-02-05T21:06:05.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:05.210+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.211230+0000) 2020-02-05T21:06:05.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:05.210+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.211992+0000) 2020-02-05T21:06:05.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:05.211+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.212098+0000) 2020-02-05T21:06:05.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:05.211+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.212754+0000) 2020-02-05T21:06:05.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:05.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.212686+0000) 2020-02-05T21:06:05.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:05.212+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.213052+0000) 2020-02-05T21:06:05.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:05.212+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.213300+0000) 2020-02-05T21:06:05.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:05.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 75996 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:05.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:05.216+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.217924+0000) 2020-02-05T21:06:05.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:05.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.816346+0000) 2020-02-05T21:06:05.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:05.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.853372+0000) 2020-02-05T21:06:05.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:05.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:05.988767+0000) 2020-02-05T21:06:06.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:06.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:06.713538+0000) 2020-02-05T21:06:06.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:06.957+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:06.958379+0000) 2020-02-05T21:06:06.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:06.958+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:06.958420+0000) 2020-02-05T21:06:09.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:09.352+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:09.352321+0000) 2020-02-05T21:06:09.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:09.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:09.960836+0000) 2020-02-05T21:06:10.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:10.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.211389+0000) 2020-02-05T21:06:10.222 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:10.220+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.221722+0000) 2020-02-05T21:06:10.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:10.222+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.223159+0000) 2020-02-05T21:06:10.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:10.222+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.222986+0000) 2020-02-05T21:06:10.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:10.222+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.223249+0000) 2020-02-05T21:06:10.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:10.224+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.224671+0000) 2020-02-05T21:06:10.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:10.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.224502+0000) 2020-02-05T21:06:10.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:10.224+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.225277+0000) 2020-02-05T21:06:10.227 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:10.224+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.225671+0000) 2020-02-05T21:06:10.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:10.224+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.225934+0000) 2020-02-05T21:06:10.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:10.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.226573+0000) 2020-02-05T21:06:10.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:10.226+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.226576+0000) 2020-02-05T21:06:10.229 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:10.225+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.226188+0000) 2020-02-05T21:06:10.229 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:10.225+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.226462+0000) 2020-02-05T21:06:10.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:10.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:10.231214+0000) 2020-02-05T21:06:10.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:10.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:12.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:12.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:12.201365+0000) 2020-02-05T21:06:12.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:12.315+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:12.316273+0000) 2020-02-05T21:06:13.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:13.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:13.442347+0000) 2020-02-05T21:06:13.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:13.471+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:13.472046+0000) 2020-02-05T21:06:15.238 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:15.235+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.236732+0000) 2020-02-05T21:06:15.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:15.236+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.237399+0000) 2020-02-05T21:06:15.238 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:15.236+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.237583+0000) 2020-02-05T21:06:15.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:15.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.243357+0000) 2020-02-05T21:06:15.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:15.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.243375+0000) 2020-02-05T21:06:15.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:15.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.243908+0000) 2020-02-05T21:06:15.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:15.243+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.244397+0000) 2020-02-05T21:06:15.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:15.243+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.244046+0000) 2020-02-05T21:06:15.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:15.243+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.244383+0000) 2020-02-05T21:06:15.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:15.244+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.244716+0000) 2020-02-05T21:06:15.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:15.244+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.245225+0000) 2020-02-05T21:06:15.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:15.244+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.245549+0000) 2020-02-05T21:06:15.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:15.245+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.245617+0000) 2020-02-05T21:06:15.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:15.248+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.249767+0000) 2020-02-05T21:06:15.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:15.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:15.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:15.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.816584+0000) 2020-02-05T21:06:15.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:15.852+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.853574+0000) 2020-02-05T21:06:15.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:15.995+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:15.996808+0000) 2020-02-05T21:06:16.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:16.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:16.713786+0000) 2020-02-05T21:06:16.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:16.958+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:16.958709+0000) 2020-02-05T21:06:16.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:16.958+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:16.958708+0000) 2020-02-05T21:06:16.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:16.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:16.971280+0000) 2020-02-05T21:06:19.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:19.351+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:19.352561+0000) 2020-02-05T21:06:19.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:19.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:19.961143+0000) 2020-02-05T21:06:20.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:20.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.211682+0000) 2020-02-05T21:06:20.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:20.255+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.256928+0000) 2020-02-05T21:06:20.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:20.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.257499+0000) 2020-02-05T21:06:20.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:20.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.257944+0000) 2020-02-05T21:06:20.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:20.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.258068+0000) 2020-02-05T21:06:20.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:20.258+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.258876+0000) 2020-02-05T21:06:20.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:20.258+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.259332+0000) 2020-02-05T21:06:20.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:20.259+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.260364+0000) 2020-02-05T21:06:20.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:20.259+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.260129+0000) 2020-02-05T21:06:20.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:20.260+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.260982+0000) 2020-02-05T21:06:20.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:20.260+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.261360+0000) 2020-02-05T21:06:20.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:20.260+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.261484+0000) 2020-02-05T21:06:20.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:20.260+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.260869+0000) 2020-02-05T21:06:20.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:20.261+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.262199+0000) 2020-02-05T21:06:20.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:20.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:20.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:20.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:20.267589+0000) 2020-02-05T21:06:22.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:22.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:22.201652+0000) 2020-02-05T21:06:22.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:22.316+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:22.316527+0000) 2020-02-05T21:06:23.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:23.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:23.442615+0000) 2020-02-05T21:06:23.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:23.471+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:23.472356+0000) 2020-02-05T21:06:25.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:25.272+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.273196+0000) 2020-02-05T21:06:25.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:25.273+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.274773+0000) 2020-02-05T21:06:25.276 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:25.274+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.275060+0000) 2020-02-05T21:06:25.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:25.274+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.275347+0000) 2020-02-05T21:06:25.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:25.275+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.275546+0000) 2020-02-05T21:06:25.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:25.275+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.276346+0000) 2020-02-05T21:06:25.278 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:25.275+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.276873+0000) 2020-02-05T21:06:25.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:25.276+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.277194+0000) 2020-02-05T21:06:25.279 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:25.276+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.277151+0000) 2020-02-05T21:06:25.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:25.275+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.276004+0000) 2020-02-05T21:06:25.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:25.276+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.277290+0000) 2020-02-05T21:06:25.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:25.277+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.277481+0000) 2020-02-05T21:06:25.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:25.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.279498+0000) 2020-02-05T21:06:25.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:25.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.279767+0000) 2020-02-05T21:06:25.282 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:25.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:25.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:25.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.816746+0000) 2020-02-05T21:06:25.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:25.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:25.853788+0000) 2020-02-05T21:06:26.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:26.004+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:26.004951+0000) 2020-02-05T21:06:26.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:26.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:26.714022+0000) 2020-02-05T21:06:26.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:26.958+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:26.958992+0000) 2020-02-05T21:06:26.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:26.958+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:26.958995+0000) 2020-02-05T21:06:26.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:26.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:26.971509+0000) 2020-02-05T21:06:29.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:29.351+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:29.352818+0000) 2020-02-05T21:06:29.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:29.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:29.961372+0000) 2020-02-05T21:06:30.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:30.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.211911+0000) 2020-02-05T21:06:30.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:30.287+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.288220+0000) 2020-02-05T21:06:30.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:30.288+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.289252+0000) 2020-02-05T21:06:30.290 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:30.288+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.289496+0000) 2020-02-05T21:06:30.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:30.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.289988+0000) 2020-02-05T21:06:30.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:30.289+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.290287+0000) 2020-02-05T21:06:30.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:30.289+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.290622+0000) 2020-02-05T21:06:30.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:30.290+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.290611+0000) 2020-02-05T21:06:30.293 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:30.291+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.292556+0000) 2020-02-05T21:06:30.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:30.292+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.292626+0000) 2020-02-05T21:06:30.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:30.291+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.292280+0000) 2020-02-05T21:06:30.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:30.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.292957+0000) 2020-02-05T21:06:30.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:30.292+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.293689+0000) 2020-02-05T21:06:30.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:30.292+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.293315+0000) 2020-02-05T21:06:30.298 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:30.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:30.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:30.295+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:30.296776+0000) 2020-02-05T21:06:32.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:32.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:32.201890+0000) 2020-02-05T21:06:32.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:32.316+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:32.316754+0000) 2020-02-05T21:06:33.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:33.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:33.442948+0000) 2020-02-05T21:06:33.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:33.471+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:33.472590+0000) 2020-02-05T21:06:34.859 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:06:34.865 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:06:35.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:35.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.303116+0000) 2020-02-05T21:06:35.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:35.302+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.303450+0000) 2020-02-05T21:06:35.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:35.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.303892+0000) 2020-02-05T21:06:35.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:35.302+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.303671+0000) 2020-02-05T21:06:35.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:35.303+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.304236+0000) 2020-02-05T21:06:35.306 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:35.303+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.304507+0000) 2020-02-05T21:06:35.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:35.304+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.305489+0000) 2020-02-05T21:06:35.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:35.303+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.305142+0000) 2020-02-05T21:06:35.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:35.304+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.305795+0000) 2020-02-05T21:06:35.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:35.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.306252+0000) 2020-02-05T21:06:35.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:35.305+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.306696+0000) 2020-02-05T21:06:35.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:35.305+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.306765+0000) 2020-02-05T21:06:35.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:35.308+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.310062+0000) 2020-02-05T21:06:35.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:35.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.310434+0000) 2020-02-05T21:06:35.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:35.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:35.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:35.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.817020+0000) 2020-02-05T21:06:35.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:35.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:35.853988+0000) 2020-02-05T21:06:36.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:36.012+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:36.013012+0000) 2020-02-05T21:06:36.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:36.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:36.714347+0000) 2020-02-05T21:06:36.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:36.957+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:36.959239+0000) 2020-02-05T21:06:36.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:36.957+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:36.959243+0000) 2020-02-05T21:06:36.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:36.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:36.971739+0000) 2020-02-05T21:06:39.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:39.352+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:39.353070+0000) 2020-02-05T21:06:39.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:39.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:39.961619+0000) 2020-02-05T21:06:40.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:40.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.212177+0000) 2020-02-05T21:06:40.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:40.315+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.316862+0000) 2020-02-05T21:06:40.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:40.317+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.318737+0000) 2020-02-05T21:06:40.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:40.318+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.319286+0000) 2020-02-05T21:06:40.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:40.317+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.319307+0000) 2020-02-05T21:06:40.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:40.318+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.319821+0000) 2020-02-05T21:06:40.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:40.319+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.320319+0000) 2020-02-05T21:06:40.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:40.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.320920+0000) 2020-02-05T21:06:40.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:40.320+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.321661+0000) 2020-02-05T21:06:40.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:40.320+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.321789+0000) 2020-02-05T21:06:40.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:40.320+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.322157+0000) 2020-02-05T21:06:40.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:40.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.323247+0000) 2020-02-05T21:06:40.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:40.322+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.323350+0000) 2020-02-05T21:06:40.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:40.322+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.323467+0000) 2020-02-05T21:06:40.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:40.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:40.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:40.327+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:40.328480+0000) 2020-02-05T21:06:42.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:42.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:42.202128+0000) 2020-02-05T21:06:42.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:42.315+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:42.316989+0000) 2020-02-05T21:06:43.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:43.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:43.443226+0000) 2020-02-05T21:06:43.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:43.472+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:43.472807+0000) 2020-02-05T21:06:45.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:45.331+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.332941+0000) 2020-02-05T21:06:45.335 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:45.332+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.333329+0000) 2020-02-05T21:06:45.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:45.332+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.333693+0000) 2020-02-05T21:06:45.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:45.332+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.334112+0000) 2020-02-05T21:06:45.337 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:45.333+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.334645+0000) 2020-02-05T21:06:45.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:45.333+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.334831+0000) 2020-02-05T21:06:45.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:45.334+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.335449+0000) 2020-02-05T21:06:45.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:45.334+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.335622+0000) 2020-02-05T21:06:45.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:45.333+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.334971+0000) 2020-02-05T21:06:45.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:45.334+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.335992+0000) 2020-02-05T21:06:45.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:45.335+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.336618+0000) 2020-02-05T21:06:45.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:45.335+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.336787+0000) 2020-02-05T21:06:45.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:45.336+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.337609+0000) 2020-02-05T21:06:45.343 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:45.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76315 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:45.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:45.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.343210+0000) 2020-02-05T21:06:45.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:45.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.817292+0000) 2020-02-05T21:06:45.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:45.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:45.854195+0000) 2020-02-05T21:06:46.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:46.019+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:46.020778+0000) 2020-02-05T21:06:46.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:46.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:46.714591+0000) 2020-02-05T21:06:46.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:46.958+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:46.959548+0000) 2020-02-05T21:06:46.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:46.958+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:46.959545+0000) 2020-02-05T21:06:46.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:46.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:46.971991+0000) 2020-02-05T21:06:49.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:49.353+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:49.353342+0000) 2020-02-05T21:06:49.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:49.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:49.961844+0000) 2020-02-05T21:06:50.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:50.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.212455+0000) 2020-02-05T21:06:50.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:50.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.348590+0000) 2020-02-05T21:06:50.350 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:50.347+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.348803+0000) 2020-02-05T21:06:50.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:50.348+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.349811+0000) 2020-02-05T21:06:50.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:50.349+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.350511+0000) 2020-02-05T21:06:50.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:50.349+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.350966+0000) 2020-02-05T21:06:50.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:50.350+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.351461+0000) 2020-02-05T21:06:50.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:50.350+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.352183+0000) 2020-02-05T21:06:50.354 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:50.351+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.352753+0000) 2020-02-05T21:06:50.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:50.352+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.353444+0000) 2020-02-05T21:06:50.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:50.352+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.353697+0000) 2020-02-05T21:06:50.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:50.351+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.353342+0000) 2020-02-05T21:06:50.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:50.352+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.353726+0000) 2020-02-05T21:06:50.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:50.352+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.353967+0000) 2020-02-05T21:06:50.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:50.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:50.356696+0000) 2020-02-05T21:06:50.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:50.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:52.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:52.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:52.202366+0000) 2020-02-05T21:06:52.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:52.316+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:52.317256+0000) 2020-02-05T21:06:53.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:53.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:53.443480+0000) 2020-02-05T21:06:53.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:53.472+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:53.473034+0000) 2020-02-05T21:06:55.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:55.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.363765+0000) 2020-02-05T21:06:55.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:55.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.364447+0000) 2020-02-05T21:06:55.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:55.364+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.365670+0000) 2020-02-05T21:06:55.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:55.364+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.365489+0000) 2020-02-05T21:06:55.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:55.364+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.366211+0000) 2020-02-05T21:06:55.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:55.365+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.366593+0000) 2020-02-05T21:06:55.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:06:55.366+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.367538+0000) 2020-02-05T21:06:55.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:06:55.366+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.367886+0000) 2020-02-05T21:06:55.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:55.367+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.368233+0000) 2020-02-05T21:06:55.370 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:06:55.367+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.368533+0000) 2020-02-05T21:06:55.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:55.367+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.368650+0000) 2020-02-05T21:06:55.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:06:55.367+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.368933+0000) 2020-02-05T21:06:55.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:06:55.367+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.369091+0000) 2020-02-05T21:06:55.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:06:55.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:06:55.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:55.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.375715+0000) 2020-02-05T21:06:55.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:06:55.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.817499+0000) 2020-02-05T21:06:55.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:55.854+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:55.854405+0000) 2020-02-05T21:06:56.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:56.028+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:56.028847+0000) 2020-02-05T21:06:56.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:06:56.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:56.714843+0000) 2020-02-05T21:06:56.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:06:56.958+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:56.959815+0000) 2020-02-05T21:06:56.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:06:56.958+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:56.959808+0000) 2020-02-05T21:06:56.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:06:56.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:56.972202+0000) 2020-02-05T21:06:59.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:06:59.353+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:59.353577+0000) 2020-02-05T21:06:59.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:06:59.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:06:59.962121+0000) 2020-02-05T21:07:00.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:00.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.212721+0000) 2020-02-05T21:07:00.379 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:00.378+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.378201+0000) 2020-02-05T21:07:00.381 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:00.378+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.379688+0000) 2020-02-05T21:07:00.381 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:00.380+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.380497+0000) 2020-02-05T21:07:00.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:00.379+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.381013+0000) 2020-02-05T21:07:00.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:00.381+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.381578+0000) 2020-02-05T21:07:00.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:00.382+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.382362+0000) 2020-02-05T21:07:00.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:00.382+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.382730+0000) 2020-02-05T21:07:00.384 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:00.381+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.382857+0000) 2020-02-05T21:07:00.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:00.383+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.383427+0000) 2020-02-05T21:07:00.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:00.382+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.383523+0000) 2020-02-05T21:07:00.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:00.382+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.383831+0000) 2020-02-05T21:07:00.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:00.383+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.384849+0000) 2020-02-05T21:07:00.386 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:00.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.385259+0000) 2020-02-05T21:07:00.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:00.387+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:00.387818+0000) 2020-02-05T21:07:00.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:00.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:02.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:02.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:02.202658+0000) 2020-02-05T21:07:02.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:02.316+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:02.317485+0000) 2020-02-05T21:07:03.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:03.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:03.443711+0000) 2020-02-05T21:07:03.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:03.473+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:03.473300+0000) 2020-02-05T21:07:05.056 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:07:05.062 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:07:05.395 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:05.392+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.393999+0000) 2020-02-05T21:07:05.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:05.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.395329+0000) 2020-02-05T21:07:05.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:05.394+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.396011+0000) 2020-02-05T21:07:05.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:05.395+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.396497+0000) 2020-02-05T21:07:05.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:05.395+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.396825+0000) 2020-02-05T21:07:05.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:05.395+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.397003+0000) 2020-02-05T21:07:05.399 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:05.396+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.397628+0000) 2020-02-05T21:07:05.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:05.397+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.398135+0000) 2020-02-05T21:07:05.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:05.397+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.398793+0000) 2020-02-05T21:07:05.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:05.397+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.399136+0000) 2020-02-05T21:07:05.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:05.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.399529+0000) 2020-02-05T21:07:05.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:05.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.399743+0000) 2020-02-05T21:07:05.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:05.398+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.400142+0000) 2020-02-05T21:07:05.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:05.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:05.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:05.404+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.405450+0000) 2020-02-05T21:07:05.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:05.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.817718+0000) 2020-02-05T21:07:05.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:05.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:05.854610+0000) 2020-02-05T21:07:06.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:06.035+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:06.036677+0000) 2020-02-05T21:07:06.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:06.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:06.715115+0000) 2020-02-05T21:07:06.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:06.959+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:06.960068+0000) 2020-02-05T21:07:06.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:06.959+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:06.960070+0000) 2020-02-05T21:07:06.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:06.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:06.972455+0000) 2020-02-05T21:07:09.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:09.352+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:09.353815+0000) 2020-02-05T21:07:09.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:09.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:09.962365+0000) 2020-02-05T21:07:10.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:10.212+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.212969+0000) 2020-02-05T21:07:10.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:10.408+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.409071+0000) 2020-02-05T21:07:10.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:10.408+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.409526+0000) 2020-02-05T21:07:10.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:10.409+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.410177+0000) 2020-02-05T21:07:10.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:10.410+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.411410+0000) 2020-02-05T21:07:10.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:10.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.411934+0000) 2020-02-05T21:07:10.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:10.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.412371+0000) 2020-02-05T21:07:10.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:10.411+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.413344+0000) 2020-02-05T21:07:10.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:10.412+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.413736+0000) 2020-02-05T21:07:10.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:10.413+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.414146+0000) 2020-02-05T21:07:10.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:10.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.414596+0000) 2020-02-05T21:07:10.416 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:10.413+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.415047+0000) 2020-02-05T21:07:10.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:10.413+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.415215+0000) 2020-02-05T21:07:10.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:10.414+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.415990+0000) 2020-02-05T21:07:10.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:10.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:10.418055+0000) 2020-02-05T21:07:10.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:10.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:12.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:12.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:12.202915+0000) 2020-02-05T21:07:12.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:12.316+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:12.317682+0000) 2020-02-05T21:07:13.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:13.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:13.443953+0000) 2020-02-05T21:07:13.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:13.472+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:13.473552+0000) 2020-02-05T21:07:15.426 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:15.423+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.424400+0000) 2020-02-05T21:07:15.426 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:15.423+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.424731+0000) 2020-02-05T21:07:15.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:15.425+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.426421+0000) 2020-02-05T21:07:15.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:15.426+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.427261+0000) 2020-02-05T21:07:15.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:15.426+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.427387+0000) 2020-02-05T21:07:15.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:15.427+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.428216+0000) 2020-02-05T21:07:15.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:15.426+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.428186+0000) 2020-02-05T21:07:15.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:15.427+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.428412+0000) 2020-02-05T21:07:15.430 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:15.428+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.429401+0000) 2020-02-05T21:07:15.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:15.428+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.429682+0000) 2020-02-05T21:07:15.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:15.428+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.430057+0000) 2020-02-05T21:07:15.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:15.428+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.429674+0000) 2020-02-05T21:07:15.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:15.428+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.430302+0000) 2020-02-05T21:07:15.435 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:15.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:15.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:15.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.436882+0000) 2020-02-05T21:07:15.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:15.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.817923+0000) 2020-02-05T21:07:15.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:15.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:15.854820+0000) 2020-02-05T21:07:16.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:16.043+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:16.044907+0000) 2020-02-05T21:07:16.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:16.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:16.715402+0000) 2020-02-05T21:07:16.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:16.959+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:16.960318+0000) 2020-02-05T21:07:16.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:16.959+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:16.960350+0000) 2020-02-05T21:07:16.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:16.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:16.972685+0000) 2020-02-05T21:07:19.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:19.353+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:19.354072+0000) 2020-02-05T21:07:19.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:19.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:19.962591+0000) 2020-02-05T21:07:20.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:20.212+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.213282+0000) 2020-02-05T21:07:20.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:20.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.440467+0000) 2020-02-05T21:07:20.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:20.439+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.440924+0000) 2020-02-05T21:07:20.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:20.440+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.441589+0000) 2020-02-05T21:07:20.443 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:20.441+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.442277+0000) 2020-02-05T21:07:20.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:20.441+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.442718+0000) 2020-02-05T21:07:20.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:20.442+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.443403+0000) 2020-02-05T21:07:20.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:20.442+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.443965+0000) 2020-02-05T21:07:20.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:20.442+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.444084+0000) 2020-02-05T21:07:20.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:20.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.444737+0000) 2020-02-05T21:07:20.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:20.443+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.445098+0000) 2020-02-05T21:07:20.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:20.443+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.445244+0000) 2020-02-05T21:07:20.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:20.444+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.445879+0000) 2020-02-05T21:07:20.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:20.444+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.445948+0000) 2020-02-05T21:07:20.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:20.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:20.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:20.450+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:20.451564+0000) 2020-02-05T21:07:22.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:22.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:22.203172+0000) 2020-02-05T21:07:22.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:22.317+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:22.317951+0000) 2020-02-05T21:07:23.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:23.443+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:23.444187+0000) 2020-02-05T21:07:23.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:23.473+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:23.473780+0000) 2020-02-05T21:07:25.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:25.453+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.454583+0000) 2020-02-05T21:07:25.457 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:25.455+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.456301+0000) 2020-02-05T21:07:25.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:25.456+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.456944+0000) 2020-02-05T21:07:25.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:25.456+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.457285+0000) 2020-02-05T21:07:25.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:25.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.457850+0000) 2020-02-05T21:07:25.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:25.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.458596+0000) 2020-02-05T21:07:25.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:25.458+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.459018+0000) 2020-02-05T21:07:25.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:25.458+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.459531+0000) 2020-02-05T21:07:25.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:25.458+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.459790+0000) 2020-02-05T21:07:25.464 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:25.459+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.460702+0000) 2020-02-05T21:07:25.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:25.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.461708+0000) 2020-02-05T21:07:25.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:25.460+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.461844+0000) 2020-02-05T21:07:25.465 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:25.461+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.462600+0000) 2020-02-05T21:07:25.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:25.462+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.463958+0000) 2020-02-05T21:07:25.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:25.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76635 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:25.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:25.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.818139+0000) 2020-02-05T21:07:25.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:25.853+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:25.855053+0000) 2020-02-05T21:07:26.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:26.052+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:26.052907+0000) 2020-02-05T21:07:26.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:26.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:26.715653+0000) 2020-02-05T21:07:26.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:26.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:26.960569+0000) 2020-02-05T21:07:26.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:26.960+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:26.960659+0000) 2020-02-05T21:07:26.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:26.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:26.972956+0000) 2020-02-05T21:07:29.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:29.353+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:29.354369+0000) 2020-02-05T21:07:29.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:29.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:29.962852+0000) 2020-02-05T21:07:30.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:30.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.213485+0000) 2020-02-05T21:07:30.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:30.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.471134+0000) 2020-02-05T21:07:30.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:30.470+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.471703+0000) 2020-02-05T21:07:30.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:30.472+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.472775+0000) 2020-02-05T21:07:30.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:30.472+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.473139+0000) 2020-02-05T21:07:30.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:30.472+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.473698+0000) 2020-02-05T21:07:30.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:30.473+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.473914+0000) 2020-02-05T21:07:30.476 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:30.473+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.474303+0000) 2020-02-05T21:07:30.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:30.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.474620+0000) 2020-02-05T21:07:30.477 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:30.474+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.474783+0000) 2020-02-05T21:07:30.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:30.474+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.474967+0000) 2020-02-05T21:07:30.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:30.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.475906+0000) 2020-02-05T21:07:30.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:30.474+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.475821+0000) 2020-02-05T21:07:30.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:30.478+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.479281+0000) 2020-02-05T21:07:30.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:30.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:30.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:30.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:30.480480+0000) 2020-02-05T21:07:32.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:32.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:32.203491+0000) 2020-02-05T21:07:32.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:32.317+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:32.318190+0000) 2020-02-05T21:07:33.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:33.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:33.444481+0000) 2020-02-05T21:07:33.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:33.473+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:33.474018+0000) 2020-02-05T21:07:35.236 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:07:35.242 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:07:35.486 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:35.484+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.485456+0000) 2020-02-05T21:07:35.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:35.485+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.486533+0000) 2020-02-05T21:07:35.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:35.486+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.487412+0000) 2020-02-05T21:07:35.489 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:35.486+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.487954+0000) 2020-02-05T21:07:35.490 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:35.487+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.488880+0000) 2020-02-05T21:07:35.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:35.488+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.489259+0000) 2020-02-05T21:07:35.491 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:35.489+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.489541+0000) 2020-02-05T21:07:35.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:35.489+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.490174+0000) 2020-02-05T21:07:35.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:35.490+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.490529+0000) 2020-02-05T21:07:35.492 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:35.490+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.491231+0000) 2020-02-05T21:07:35.493 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:35.490+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.491377+0000) 2020-02-05T21:07:35.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:35.491+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.491788+0000) 2020-02-05T21:07:35.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:35.491+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.492331+0000) 2020-02-05T21:07:35.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:35.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.493821+0000) 2020-02-05T21:07:35.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:35.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:35.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:35.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.818353+0000) 2020-02-05T21:07:35.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:35.854+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:35.855307+0000) 2020-02-05T21:07:36.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:36.059+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:36.060435+0000) 2020-02-05T21:07:36.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:36.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:36.715902+0000) 2020-02-05T21:07:36.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:36.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:36.960828+0000) 2020-02-05T21:07:36.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:36.960+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:36.960922+0000) 2020-02-05T21:07:36.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:36.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:36.973187+0000) 2020-02-05T21:07:39.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:39.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:39.354603+0000) 2020-02-05T21:07:39.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:39.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:39.963129+0000) 2020-02-05T21:07:40.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:40.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.213714+0000) 2020-02-05T21:07:40.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:40.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.500999+0000) 2020-02-05T21:07:40.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:40.500+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.500968+0000) 2020-02-05T21:07:40.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:40.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.501837+0000) 2020-02-05T21:07:40.503 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:40.501+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.502377+0000) 2020-02-05T21:07:40.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:40.502+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.502762+0000) 2020-02-05T21:07:40.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:40.502+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.503587+0000) 2020-02-05T21:07:40.505 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:40.502+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.504054+0000) 2020-02-05T21:07:40.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:40.504+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.504570+0000) 2020-02-05T21:07:40.506 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:40.504+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.505330+0000) 2020-02-05T21:07:40.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:40.505+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.506086+0000) 2020-02-05T21:07:40.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:40.504+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.506040+0000) 2020-02-05T21:07:40.508 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:40.505+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.505773+0000) 2020-02-05T21:07:40.508 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:40.505+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.506255+0000) 2020-02-05T21:07:40.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:40.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:40.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:40.509+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:40.510951+0000) 2020-02-05T21:07:42.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:42.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:42.203734+0000) 2020-02-05T21:07:42.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:42.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:42.318457+0000) 2020-02-05T21:07:43.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:43.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:43.444710+0000) 2020-02-05T21:07:43.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:43.474+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:43.474342+0000) 2020-02-05T21:07:45.516 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:45.513+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.514918+0000) 2020-02-05T21:07:45.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:45.514+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.515636+0000) 2020-02-05T21:07:45.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:45.515+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.516181+0000) 2020-02-05T21:07:45.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:45.515+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.516918+0000) 2020-02-05T21:07:45.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:45.516+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.517344+0000) 2020-02-05T21:07:45.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:45.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.518195+0000) 2020-02-05T21:07:45.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:45.517+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.518647+0000) 2020-02-05T21:07:45.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:45.518+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.519333+0000) 2020-02-05T21:07:45.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:45.519+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.519968+0000) 2020-02-05T21:07:45.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:45.519+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.520128+0000) 2020-02-05T21:07:45.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:45.520+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.520592+0000) 2020-02-05T21:07:45.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:45.520+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.521051+0000) 2020-02-05T21:07:45.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:45.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.521668+0000) 2020-02-05T21:07:45.526 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:45.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:45.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:45.525+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.526926+0000) 2020-02-05T21:07:45.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:45.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.818577+0000) 2020-02-05T21:07:45.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:45.854+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:45.855514+0000) 2020-02-05T21:07:46.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:46.067+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:46.068244+0000) 2020-02-05T21:07:46.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:46.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:46.716129+0000) 2020-02-05T21:07:46.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:46.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:46.961089+0000) 2020-02-05T21:07:46.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:46.960+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:46.961205+0000) 2020-02-05T21:07:46.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:46.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:46.973438+0000) 2020-02-05T21:07:49.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:49.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:49.354862+0000) 2020-02-05T21:07:49.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:49.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:49.963362+0000) 2020-02-05T21:07:50.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:50.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.213993+0000) 2020-02-05T21:07:50.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:50.530+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.531582+0000) 2020-02-05T21:07:50.533 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:50.531+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.532288+0000) 2020-02-05T21:07:50.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:50.532+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.532544+0000) 2020-02-05T21:07:50.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:50.532+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.533333+0000) 2020-02-05T21:07:50.536 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:50.534+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.534667+0000) 2020-02-05T21:07:50.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:50.534+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.534999+0000) 2020-02-05T21:07:50.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:50.534+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.535730+0000) 2020-02-05T21:07:50.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:50.535+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.536145+0000) 2020-02-05T21:07:50.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:50.535+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.536208+0000) 2020-02-05T21:07:50.538 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:50.536+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.537388+0000) 2020-02-05T21:07:50.539 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:50.537+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.537569+0000) 2020-02-05T21:07:50.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:50.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.537740+0000) 2020-02-05T21:07:50.539 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:50.536+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.537183+0000) 2020-02-05T21:07:50.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:50.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:50.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:50.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:50.542225+0000) 2020-02-05T21:07:52.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:52.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:52.203989+0000) 2020-02-05T21:07:52.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:52.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:52.318697+0000) 2020-02-05T21:07:53.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:53.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:53.444971+0000) 2020-02-05T21:07:53.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:53.474+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:53.474614+0000) 2020-02-05T21:07:55.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:55.546+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.546343+0000) 2020-02-05T21:07:55.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:07:55.546+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.547355+0000) 2020-02-05T21:07:55.549 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:07:55.548+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.548331+0000) 2020-02-05T21:07:55.549 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:07:55.548+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.548468+0000) 2020-02-05T21:07:55.550 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:07:55.548+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.549046+0000) 2020-02-05T21:07:55.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:55.548+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.549332+0000) 2020-02-05T21:07:55.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:55.550+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.550232+0000) 2020-02-05T21:07:55.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:55.550+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.550444+0000) 2020-02-05T21:07:55.552 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:55.550+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.550856+0000) 2020-02-05T21:07:55.552 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:55.551+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.551469+0000) 2020-02-05T21:07:55.553 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:07:55.550+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.551104+0000) 2020-02-05T21:07:55.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:55.551+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.551965+0000) 2020-02-05T21:07:55.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:55.551+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.552158+0000) 2020-02-05T21:07:55.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:55.556+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.556170+0000) 2020-02-05T21:07:55.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:07:55.555+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:07:55.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:07:55.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.818815+0000) 2020-02-05T21:07:55.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:55.855+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:55.855725+0000) 2020-02-05T21:07:56.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:56.076+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:56.076322+0000) 2020-02-05T21:07:56.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:07:56.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:56.716360+0000) 2020-02-05T21:07:56.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:07:56.961+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:56.961345+0000) 2020-02-05T21:07:56.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:07:56.961+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:56.961498+0000) 2020-02-05T21:07:56.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:07:56.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:56.973668+0000) 2020-02-05T21:07:59.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:07:59.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:59.355126+0000) 2020-02-05T21:07:59.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:07:59.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:07:59.963592+0000) 2020-02-05T21:08:00.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:00.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.214227+0000) 2020-02-05T21:08:00.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:00.561+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.562109+0000) 2020-02-05T21:08:00.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:00.561+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.562586+0000) 2020-02-05T21:08:00.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:00.562+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.563171+0000) 2020-02-05T21:08:00.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:00.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.564018+0000) 2020-02-05T21:08:00.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:00.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.564448+0000) 2020-02-05T21:08:00.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:00.564+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.564899+0000) 2020-02-05T21:08:00.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:00.565+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.565534+0000) 2020-02-05T21:08:00.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:00.565+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.566601+0000) 2020-02-05T21:08:00.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:00.566+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.566575+0000) 2020-02-05T21:08:00.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:00.566+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.567111+0000) 2020-02-05T21:08:00.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:00.567+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.567590+0000) 2020-02-05T21:08:00.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:00.567+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.567710+0000) 2020-02-05T21:08:00.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:00.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:00.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:00.571+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.572410+0000) 2020-02-05T21:08:00.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:00.571+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:00.572704+0000) 2020-02-05T21:08:02.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:02.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:02.204244+0000) 2020-02-05T21:08:02.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:02.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:02.318927+0000) 2020-02-05T21:08:03.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:03.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:03.445293+0000) 2020-02-05T21:08:03.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:03.473+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:03.474893+0000) 2020-02-05T21:08:05.578 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:05.576+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.577346+0000) 2020-02-05T21:08:05.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:05.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.577849+0000) 2020-02-05T21:08:05.579 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:05.577+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.578427+0000) 2020-02-05T21:08:05.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:05.578+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.579235+0000) 2020-02-05T21:08:05.580 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:05.578+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.579527+0000) 2020-02-05T21:08:05.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:05.579+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.580159+0000) 2020-02-05T21:08:05.581 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:05.579+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.580506+0000) 2020-02-05T21:08:05.582 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:05.579+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.580868+0000) 2020-02-05T21:08:05.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:05.580+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.581736+0000) 2020-02-05T21:08:05.583 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:08:05.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:05.581+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.582611+0000) 2020-02-05T21:08:05.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:05.581+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.583138+0000) 2020-02-05T21:08:05.585 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:05.582+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.583514+0000) 2020-02-05T21:08:05.587 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:08:05.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:05.585+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.586880+0000) 2020-02-05T21:08:05.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:05.586+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.587198+0000) 2020-02-05T21:08:05.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:05.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:05.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:05.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.819043+0000) 2020-02-05T21:08:05.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:05.855+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:05.855940+0000) 2020-02-05T21:08:06.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:06.083+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:06.084585+0000) 2020-02-05T21:08:06.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:06.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:06.716600+0000) 2020-02-05T21:08:06.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:06.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:06.961614+0000) 2020-02-05T21:08:06.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:06.960+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:06.961768+0000) 2020-02-05T21:08:06.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:06.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:06.973933+0000) 2020-02-05T21:08:09.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:09.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:09.355357+0000) 2020-02-05T21:08:09.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:09.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:09.963829+0000) 2020-02-05T21:08:10.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:10.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.214487+0000) 2020-02-05T21:08:10.595 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:10.592+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.593877+0000) 2020-02-05T21:08:10.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:10.593+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.594263+0000) 2020-02-05T21:08:10.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:10.593+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.594791+0000) 2020-02-05T21:08:10.596 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:10.593+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.595334+0000) 2020-02-05T21:08:10.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:10.594+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.595768+0000) 2020-02-05T21:08:10.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:10.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.596250+0000) 2020-02-05T21:08:10.597 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:10.595+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.596480+0000) 2020-02-05T21:08:10.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:10.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.597223+0000) 2020-02-05T21:08:10.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:10.597+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.598627+0000) 2020-02-05T21:08:10.600 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:10.597+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.598752+0000) 2020-02-05T21:08:10.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:10.598+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.599343+0000) 2020-02-05T21:08:10.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:10.598+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.599521+0000) 2020-02-05T21:08:10.601 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:10.598+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.599753+0000) 2020-02-05T21:08:10.604 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:10.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 76997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:10.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:10.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:10.604240+0000) 2020-02-05T21:08:12.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:12.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:12.204503+0000) 2020-02-05T21:08:12.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:12.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:12.319223+0000) 2020-02-05T21:08:13.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:13.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:13.445529+0000) 2020-02-05T21:08:13.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:13.474+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:13.475126+0000) 2020-02-05T21:08:15.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:15.608+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.610285+0000) 2020-02-05T21:08:15.612 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:15.610+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.611315+0000) 2020-02-05T21:08:15.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:15.610+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.611509+0000) 2020-02-05T21:08:15.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:15.610+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.612041+0000) 2020-02-05T21:08:15.613 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:15.611+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.612461+0000) 2020-02-05T21:08:15.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:15.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.612651+0000) 2020-02-05T21:08:15.614 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:15.611+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.612614+0000) 2020-02-05T21:08:15.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:15.612+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.613111+0000) 2020-02-05T21:08:15.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:15.612+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.613701+0000) 2020-02-05T21:08:15.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:15.612+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.613917+0000) 2020-02-05T21:08:15.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:15.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.614462+0000) 2020-02-05T21:08:15.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:15.613+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.614998+0000) 2020-02-05T21:08:15.617 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:15.614+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.615693+0000) 2020-02-05T21:08:15.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:15.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:15.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:15.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.620714+0000) 2020-02-05T21:08:15.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:15.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.819301+0000) 2020-02-05T21:08:15.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:15.855+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:15.856162+0000) 2020-02-05T21:08:16.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:16.094+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:16.095449+0000) 2020-02-05T21:08:16.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:16.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:16.716871+0000) 2020-02-05T21:08:16.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:16.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:16.961863+0000) 2020-02-05T21:08:16.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:16.960+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:16.962053+0000) 2020-02-05T21:08:16.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:16.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:16.974145+0000) 2020-02-05T21:08:19.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:19.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:19.355584+0000) 2020-02-05T21:08:19.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:19.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:19.964058+0000) 2020-02-05T21:08:20.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:20.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.214721+0000) 2020-02-05T21:08:20.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:20.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.623759+0000) 2020-02-05T21:08:20.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:20.620+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.622372+0000) 2020-02-05T21:08:20.626 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:20.623+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.624611+0000) 2020-02-05T21:08:20.626 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:20.623+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.624510+0000) 2020-02-05T21:08:20.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:20.624+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.625383+0000) 2020-02-05T21:08:20.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:20.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.626590+0000) 2020-02-05T21:08:20.628 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:20.625+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.627040+0000) 2020-02-05T21:08:20.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:20.626+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.627475+0000) 2020-02-05T21:08:20.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:20.626+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.628267+0000) 2020-02-05T21:08:20.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:20.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.628811+0000) 2020-02-05T21:08:20.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:20.627+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.629345+0000) 2020-02-05T21:08:20.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:20.627+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.629296+0000) 2020-02-05T21:08:20.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:20.628+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.630022+0000) 2020-02-05T21:08:20.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:20.632+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:20.633833+0000) 2020-02-05T21:08:20.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:20.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:22.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:22.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:22.204756+0000) 2020-02-05T21:08:22.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:22.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:22.319493+0000) 2020-02-05T21:08:23.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:23.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:23.445780+0000) 2020-02-05T21:08:23.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:23.474+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:23.475343+0000) 2020-02-05T21:08:25.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:25.637+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.638687+0000) 2020-02-05T21:08:25.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:25.639+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.640193+0000) 2020-02-05T21:08:25.642 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:25.639+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.640637+0000) 2020-02-05T21:08:25.642 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:25.640+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.641591+0000) 2020-02-05T21:08:25.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:25.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.643190+0000) 2020-02-05T21:08:25.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:25.642+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.643731+0000) 2020-02-05T21:08:25.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:25.642+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.643575+0000) 2020-02-05T21:08:25.646 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:25.642+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.644011+0000) 2020-02-05T21:08:25.646 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:25.642+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.644032+0000) 2020-02-05T21:08:25.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:25.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.643960+0000) 2020-02-05T21:08:25.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:25.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.644185+0000) 2020-02-05T21:08:25.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:25.643+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.645018+0000) 2020-02-05T21:08:25.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:25.643+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.644871+0000) 2020-02-05T21:08:25.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:25.648+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.649975+0000) 2020-02-05T21:08:25.651 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:25.648+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:25.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:25.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.819509+0000) 2020-02-05T21:08:25.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:25.855+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:25.856386+0000) 2020-02-05T21:08:26.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:26.103+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:26.103700+0000) 2020-02-05T21:08:26.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:26.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:26.717149+0000) 2020-02-05T21:08:26.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:26.961+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:26.962106+0000) 2020-02-05T21:08:26.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:26.961+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:26.962310+0000) 2020-02-05T21:08:26.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:26.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:26.974376+0000) 2020-02-05T21:08:29.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:29.354+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:29.355864+0000) 2020-02-05T21:08:29.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:29.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:29.964325+0000) 2020-02-05T21:08:30.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:30.214+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.214977+0000) 2020-02-05T21:08:30.657 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:30.654+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.655782+0000) 2020-02-05T21:08:30.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:30.655+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.656484+0000) 2020-02-05T21:08:30.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:30.655+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.656799+0000) 2020-02-05T21:08:30.658 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:30.656+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.657134+0000) 2020-02-05T21:08:30.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:30.656+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.657547+0000) 2020-02-05T21:08:30.659 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:30.656+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.657710+0000) 2020-02-05T21:08:30.660 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:30.657+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.658659+0000) 2020-02-05T21:08:30.660 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:30.657+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.658795+0000) 2020-02-05T21:08:30.660 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:30.657+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.659353+0000) 2020-02-05T21:08:30.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:30.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.659769+0000) 2020-02-05T21:08:30.661 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:30.658+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.659775+0000) 2020-02-05T21:08:30.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:30.659+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.660411+0000) 2020-02-05T21:08:30.662 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:30.658+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.660362+0000) 2020-02-05T21:08:30.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:30.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:30.664833+0000) 2020-02-05T21:08:30.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:30.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:32.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:32.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:32.205015+0000) 2020-02-05T21:08:32.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:32.318+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:32.319688+0000) 2020-02-05T21:08:33.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:33.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:33.446080+0000) 2020-02-05T21:08:33.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:33.475+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:33.475574+0000) 2020-02-05T21:08:35.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:35.670+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.671282+0000) 2020-02-05T21:08:35.674 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:35.671+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.672658+0000) 2020-02-05T21:08:35.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:35.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.673197+0000) 2020-02-05T21:08:35.675 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:35.672+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.673371+0000) 2020-02-05T21:08:35.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:35.672+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.673612+0000) 2020-02-05T21:08:35.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:35.673+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.674183+0000) 2020-02-05T21:08:35.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:35.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.675037+0000) 2020-02-05T21:08:35.677 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:35.674+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.675926+0000) 2020-02-05T21:08:35.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:35.674+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.675742+0000) 2020-02-05T21:08:35.677 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:35.675+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.676292+0000) 2020-02-05T21:08:35.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:35.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.676831+0000) 2020-02-05T21:08:35.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:35.675+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.677318+0000) 2020-02-05T21:08:35.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:35.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:35.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:35.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.681824+0000) 2020-02-05T21:08:35.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:35.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.682022+0000) 2020-02-05T21:08:35.705 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:08:35.709 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:08:35.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:35.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.819685+0000) 2020-02-05T21:08:35.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:35.855+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:35.856621+0000) 2020-02-05T21:08:36.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:36.111+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:36.111789+0000) 2020-02-05T21:08:36.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:36.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:36.717380+0000) 2020-02-05T21:08:36.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:36.960+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:36.962352+0000) 2020-02-05T21:08:36.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:36.961+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:36.962547+0000) 2020-02-05T21:08:36.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:36.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:36.974646+0000) 2020-02-05T21:08:39.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:39.355+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:39.356127+0000) 2020-02-05T21:08:39.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:39.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:39.964554+0000) 2020-02-05T21:08:40.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:40.214+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.215225+0000) 2020-02-05T21:08:40.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:40.686+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.687892+0000) 2020-02-05T21:08:40.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:40.687+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.688644+0000) 2020-02-05T21:08:40.690 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:40.687+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.688537+0000) 2020-02-05T21:08:40.690 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:40.687+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.689247+0000) 2020-02-05T21:08:40.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:40.688+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.689833+0000) 2020-02-05T21:08:40.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:40.688+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.689986+0000) 2020-02-05T21:08:40.692 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:40.689+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.690480+0000) 2020-02-05T21:08:40.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:40.690+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.691479+0000) 2020-02-05T21:08:40.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:40.690+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.691862+0000) 2020-02-05T21:08:40.693 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:40.691+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.692337+0000) 2020-02-05T21:08:40.694 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:40.691+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.693065+0000) 2020-02-05T21:08:40.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:40.691+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.693219+0000) 2020-02-05T21:08:40.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:40.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.693202+0000) 2020-02-05T21:08:40.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:40.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:40.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:40.699+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:40.700970+0000) 2020-02-05T21:08:42.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:42.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:42.205254+0000) 2020-02-05T21:08:42.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:42.319+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:42.319931+0000) 2020-02-05T21:08:43.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:43.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:43.446391+0000) 2020-02-05T21:08:43.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:43.475+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:43.475806+0000) 2020-02-05T21:08:45.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:45.701+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.702800+0000) 2020-02-05T21:08:45.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:45.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.704301+0000) 2020-02-05T21:08:45.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:45.703+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.704459+0000) 2020-02-05T21:08:45.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:45.703+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.704874+0000) 2020-02-05T21:08:45.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:45.704+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.705435+0000) 2020-02-05T21:08:45.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:45.704+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.705664+0000) 2020-02-05T21:08:45.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:45.705+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.706569+0000) 2020-02-05T21:08:45.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:45.706+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.707438+0000) 2020-02-05T21:08:45.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:45.705+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.707172+0000) 2020-02-05T21:08:45.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:45.706+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.707520+0000) 2020-02-05T21:08:45.709 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:45.707+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.708563+0000) 2020-02-05T21:08:45.710 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:45.706+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.708210+0000) 2020-02-05T21:08:45.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:45.707+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.709020+0000) 2020-02-05T21:08:45.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:45.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:45.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:45.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.716330+0000) 2020-02-05T21:08:45.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:45.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.819904+0000) 2020-02-05T21:08:45.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:45.856+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:45.856834+0000) 2020-02-05T21:08:46.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:46.121+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:46.121981+0000) 2020-02-05T21:08:46.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:46.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:46.717648+0000) 2020-02-05T21:08:46.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:46.962+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:46.962630+0000) 2020-02-05T21:08:46.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:46.962+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:46.962789+0000) 2020-02-05T21:08:46.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:46.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:46.974883+0000) 2020-02-05T21:08:49.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:49.356+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:49.356365+0000) 2020-02-05T21:08:49.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:49.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:49.964780+0000) 2020-02-05T21:08:50.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:50.214+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.215480+0000) 2020-02-05T21:08:50.719 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:50.715+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.717367+0000) 2020-02-05T21:08:50.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:50.718+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.718484+0000) 2020-02-05T21:08:50.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:50.719+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.719150+0000) 2020-02-05T21:08:50.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:50.720+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.720110+0000) 2020-02-05T21:08:50.722 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:50.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.721361+0000) 2020-02-05T21:08:50.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:50.719+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.721397+0000) 2020-02-05T21:08:50.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:50.720+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.721978+0000) 2020-02-05T21:08:50.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:50.722+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.722900+0000) 2020-02-05T21:08:50.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:50.723+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.723335+0000) 2020-02-05T21:08:50.725 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:50.722+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.723996+0000) 2020-02-05T21:08:50.725 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:50.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.724015+0000) 2020-02-05T21:08:50.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:50.722+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.724175+0000) 2020-02-05T21:08:50.726 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:50.724+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.724686+0000) 2020-02-05T21:08:50.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:50.726+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:50.726925+0000) 2020-02-05T21:08:50.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:50.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:52.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:52.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:52.205516+0000) 2020-02-05T21:08:52.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:52.319+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:52.320237+0000) 2020-02-05T21:08:53.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:53.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:53.446651+0000) 2020-02-05T21:08:53.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:53.474+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:53.476038+0000) 2020-02-05T21:08:55.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:08:55.733+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.734219+0000) 2020-02-05T21:08:55.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:55.734+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.734496+0000) 2020-02-05T21:08:55.736 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:55.733+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.734952+0000) 2020-02-05T21:08:55.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:55.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.735443+0000) 2020-02-05T21:08:55.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:08:55.735+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.735951+0000) 2020-02-05T21:08:55.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:08:55.735+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.736256+0000) 2020-02-05T21:08:55.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:55.735+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.736650+0000) 2020-02-05T21:08:55.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:55.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.737379+0000) 2020-02-05T21:08:55.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:08:55.737+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.738336+0000) 2020-02-05T21:08:55.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:08:55.738+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.738554+0000) 2020-02-05T21:08:55.740 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:55.738+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.739355+0000) 2020-02-05T21:08:55.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:55.738+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.739520+0000) 2020-02-05T21:08:55.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:55.739+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.739660+0000) 2020-02-05T21:08:55.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:55.742+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.743113+0000) 2020-02-05T21:08:55.744 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:08:55.743+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:08:55.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:08:55.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.820137+0000) 2020-02-05T21:08:55.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:55.856+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:55.857042+0000) 2020-02-05T21:08:56.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:56.128+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:56.129981+0000) 2020-02-05T21:08:56.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:08:56.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:56.717876+0000) 2020-02-05T21:08:56.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:08:56.962+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:56.962884+0000) 2020-02-05T21:08:56.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:08:56.962+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:56.963037+0000) 2020-02-05T21:08:56.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:08:56.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:56.975114+0000) 2020-02-05T21:08:59.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:08:59.355+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:59.356599+0000) 2020-02-05T21:08:59.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:08:59.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:08:59.965009+0000) 2020-02-05T21:09:00.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:00.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.215719+0000) 2020-02-05T21:09:00.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:00.748+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.749134+0000) 2020-02-05T21:09:00.751 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:00.749+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.749996+0000) 2020-02-05T21:09:00.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:00.749+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.750561+0000) 2020-02-05T21:09:00.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:00.750+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.751104+0000) 2020-02-05T21:09:00.753 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:00.750+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.751102+0000) 2020-02-05T21:09:00.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:00.750+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.751181+0000) 2020-02-05T21:09:00.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:00.750+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.751629+0000) 2020-02-05T21:09:00.753 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:00.751+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.752280+0000) 2020-02-05T21:09:00.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:00.751+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.752774+0000) 2020-02-05T21:09:00.754 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:00.752+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.753209+0000) 2020-02-05T21:09:00.755 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:00.752+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.752910+0000) 2020-02-05T21:09:00.755 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:00.752+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.753390+0000) 2020-02-05T21:09:00.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:00.753+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.753846+0000) 2020-02-05T21:09:00.760 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:00.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:00.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:00.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:00.760914+0000) 2020-02-05T21:09:02.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:02.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:02.205770+0000) 2020-02-05T21:09:02.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:02.320+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:02.320536+0000) 2020-02-05T21:09:03.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:03.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:03.446949+0000) 2020-02-05T21:09:03.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:03.475+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:03.476311+0000) 2020-02-05T21:09:05.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:05.764+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.765159+0000) 2020-02-05T21:09:05.767 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:05.765+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.766500+0000) 2020-02-05T21:09:05.768 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:05.765+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.766022+0000) 2020-02-05T21:09:05.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:05.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.766862+0000) 2020-02-05T21:09:05.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:05.766+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.767387+0000) 2020-02-05T21:09:05.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:05.766+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.767751+0000) 2020-02-05T21:09:05.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:05.767+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.768360+0000) 2020-02-05T21:09:05.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:05.767+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.768983+0000) 2020-02-05T21:09:05.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:05.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.769634+0000) 2020-02-05T21:09:05.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:05.768+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.769829+0000) 2020-02-05T21:09:05.772 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:05.769+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.770160+0000) 2020-02-05T21:09:05.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:05.770+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.770613+0000) 2020-02-05T21:09:05.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:05.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.770970+0000) 2020-02-05T21:09:05.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:05.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:05.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:05.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.775498+0000) 2020-02-05T21:09:05.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:05.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.820358+0000) 2020-02-05T21:09:05.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:05.856+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:05.857297+0000) 2020-02-05T21:09:05.902 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:09:05.908 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:09:06.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:06.137+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:06.138280+0000) 2020-02-05T21:09:06.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:06.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:06.718168+0000) 2020-02-05T21:09:06.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:06.962+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:06.963143+0000) 2020-02-05T21:09:06.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:06.962+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:06.963277+0000) 2020-02-05T21:09:06.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:06.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:06.975370+0000) 2020-02-05T21:09:09.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:09.356+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:09.356835+0000) 2020-02-05T21:09:09.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:09.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:09.965276+0000) 2020-02-05T21:09:10.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:10.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.215980+0000) 2020-02-05T21:09:10.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:10.778+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.779476+0000) 2020-02-05T21:09:10.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:10.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.780041+0000) 2020-02-05T21:09:10.782 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:10.780+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.781141+0000) 2020-02-05T21:09:10.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:10.780+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.781359+0000) 2020-02-05T21:09:10.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:10.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.781972+0000) 2020-02-05T21:09:10.783 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:10.782+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.782519+0000) 2020-02-05T21:09:10.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:10.782+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.783201+0000) 2020-02-05T21:09:10.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:10.782+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.783538+0000) 2020-02-05T21:09:10.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:10.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.783894+0000) 2020-02-05T21:09:10.785 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:10.783+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.784298+0000) 2020-02-05T21:09:10.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:10.783+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.784189+0000) 2020-02-05T21:09:10.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:10.784+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.784823+0000) 2020-02-05T21:09:10.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:10.784+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.785124+0000) 2020-02-05T21:09:10.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:10.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:10.788863+0000) 2020-02-05T21:09:10.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:10.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:12.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:12.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:12.206012+0000) 2020-02-05T21:09:12.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:12.320+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:12.320794+0000) 2020-02-05T21:09:13.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:13.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:13.447184+0000) 2020-02-05T21:09:13.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:13.475+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:13.476545+0000) 2020-02-05T21:09:15.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:15.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.794733+0000) 2020-02-05T21:09:15.797 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:15.795+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.796071+0000) 2020-02-05T21:09:15.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:15.795+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.796560+0000) 2020-02-05T21:09:15.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:15.796+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.797406+0000) 2020-02-05T21:09:15.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:15.796+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.797064+0000) 2020-02-05T21:09:15.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:15.796+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.797987+0000) 2020-02-05T21:09:15.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:15.797+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.798218+0000) 2020-02-05T21:09:15.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:15.797+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.798557+0000) 2020-02-05T21:09:15.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:15.798+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.799149+0000) 2020-02-05T21:09:15.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:15.799+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.800098+0000) 2020-02-05T21:09:15.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:15.799+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.800290+0000) 2020-02-05T21:09:15.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:15.799+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.800317+0000) 2020-02-05T21:09:15.802 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:15.799+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.800035+0000) 2020-02-05T21:09:15.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:15.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:15.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:15.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.805900+0000) 2020-02-05T21:09:15.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:15.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.820627+0000) 2020-02-05T21:09:15.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:15.856+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:15.857528+0000) 2020-02-05T21:09:16.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:16.146+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:16.146723+0000) 2020-02-05T21:09:16.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:16.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:16.718388+0000) 2020-02-05T21:09:16.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:16.963+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:16.963391+0000) 2020-02-05T21:09:16.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:16.963+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:16.963530+0000) 2020-02-05T21:09:16.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:16.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:16.975613+0000) 2020-02-05T21:09:19.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:19.356+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:19.357068+0000) 2020-02-05T21:09:19.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:19.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:19.965508+0000) 2020-02-05T21:09:20.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:20.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.216295+0000) 2020-02-05T21:09:20.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:20.809+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.810703+0000) 2020-02-05T21:09:20.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:20.810+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.811391+0000) 2020-02-05T21:09:20.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:20.810+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.811668+0000) 2020-02-05T21:09:20.813 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:20.811+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.812139+0000) 2020-02-05T21:09:20.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:20.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.812532+0000) 2020-02-05T21:09:20.814 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:20.812+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.813304+0000) 2020-02-05T21:09:20.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:20.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.813548+0000) 2020-02-05T21:09:20.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:20.812+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.814041+0000) 2020-02-05T21:09:20.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:20.814+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.814657+0000) 2020-02-05T21:09:20.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:20.814+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.815226+0000) 2020-02-05T21:09:20.817 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:20.814+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.815375+0000) 2020-02-05T21:09:20.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:20.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.816273+0000) 2020-02-05T21:09:20.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:20.816+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.816724+0000) 2020-02-05T21:09:20.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:20.819+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:20.820283+0000) 2020-02-05T21:09:20.822 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:20.820+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:22.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:22.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:22.206272+0000) 2020-02-05T21:09:22.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:22.320+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:22.321026+0000) 2020-02-05T21:09:23.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:23.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:23.447464+0000) 2020-02-05T21:09:23.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:23.475+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:23.476796+0000) 2020-02-05T21:09:25.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:25.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.820842+0000) 2020-02-05T21:09:25.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:25.825+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.826260+0000) 2020-02-05T21:09:25.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:25.826+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.827543+0000) 2020-02-05T21:09:25.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:25.826+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.827845+0000) 2020-02-05T21:09:25.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:25.827+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.828387+0000) 2020-02-05T21:09:25.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:25.827+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.828665+0000) 2020-02-05T21:09:25.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:25.828+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.829538+0000) 2020-02-05T21:09:25.831 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:25.829+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.830319+0000) 2020-02-05T21:09:25.832 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:25.830+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.831102+0000) 2020-02-05T21:09:25.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:25.830+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.830842+0000) 2020-02-05T21:09:25.833 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:25.832+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.832473+0000) 2020-02-05T21:09:25.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:25.832+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.832605+0000) 2020-02-05T21:09:25.834 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:25.832+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.833303+0000) 2020-02-05T21:09:25.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:25.833+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.833593+0000) 2020-02-05T21:09:25.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:25.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.836213+0000) 2020-02-05T21:09:25.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:25.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:25.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:25.857+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:25.857741+0000) 2020-02-05T21:09:26.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:26.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:26.154881+0000) 2020-02-05T21:09:26.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:26.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:26.718634+0000) 2020-02-05T21:09:26.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:26.963+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:26.963694+0000) 2020-02-05T21:09:26.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:26.963+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:26.963798+0000) 2020-02-05T21:09:26.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:26.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:26.975847+0000) 2020-02-05T21:09:29.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:29.356+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:29.357328+0000) 2020-02-05T21:09:29.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:29.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:29.965723+0000) 2020-02-05T21:09:30.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:30.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.216546+0000) 2020-02-05T21:09:30.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:30.840+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.841229+0000) 2020-02-05T21:09:30.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:30.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.841987+0000) 2020-02-05T21:09:30.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:30.843+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.843722+0000) 2020-02-05T21:09:30.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:30.844+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.844559+0000) 2020-02-05T21:09:30.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:30.844+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.845205+0000) 2020-02-05T21:09:30.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:30.844+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.845583+0000) 2020-02-05T21:09:30.847 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:30.845+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.846542+0000) 2020-02-05T21:09:30.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:30.846+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.847411+0000) 2020-02-05T21:09:30.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:30.847+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.847689+0000) 2020-02-05T21:09:30.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:30.847+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.847711+0000) 2020-02-05T21:09:30.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:30.847+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.847964+0000) 2020-02-05T21:09:30.850 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:30.847+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.848299+0000) 2020-02-05T21:09:30.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:30.849+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.850275+0000) 2020-02-05T21:09:30.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:30.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:30.851304+0000) 2020-02-05T21:09:30.853 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:30.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:32.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:32.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:32.206514+0000) 2020-02-05T21:09:32.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:32.321+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:32.321270+0000) 2020-02-05T21:09:33.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:33.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:33.447723+0000) 2020-02-05T21:09:33.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:33.476+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:33.477129+0000) 2020-02-05T21:09:35.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:35.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.821062+0000) 2020-02-05T21:09:35.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:35.856+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.857734+0000) 2020-02-05T21:09:35.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:35.857+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.859042+0000) 2020-02-05T21:09:35.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:35.858+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.859456+0000) 2020-02-05T21:09:35.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:35.860+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.861768+0000) 2020-02-05T21:09:35.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:35.860+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.862153+0000) 2020-02-05T21:09:35.865 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:35.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.864098+0000) 2020-02-05T21:09:35.865 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:35.862+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.864267+0000) 2020-02-05T21:09:35.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:35.863+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.864345+0000) 2020-02-05T21:09:35.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:35.863+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.864680+0000) 2020-02-05T21:09:35.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:35.863+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.864830+0000) 2020-02-05T21:09:35.867 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:35.863+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.865391+0000) 2020-02-05T21:09:35.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:35.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.865330+0000) 2020-02-05T21:09:35.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:35.865+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.866450+0000) 2020-02-05T21:09:35.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:35.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:35.867645+0000) 2020-02-05T21:09:35.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:35.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:36.082 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:09:36.088 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:09:36.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:36.161+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:36.162152+0000) 2020-02-05T21:09:36.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:36.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:36.718859+0000) 2020-02-05T21:09:36.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:36.962+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:36.963951+0000) 2020-02-05T21:09:36.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:36.962+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:36.964061+0000) 2020-02-05T21:09:36.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:36.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:36.976127+0000) 2020-02-05T21:09:39.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:39.357+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:39.357558+0000) 2020-02-05T21:09:39.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:39.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:39.965955+0000) 2020-02-05T21:09:40.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:40.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.216781+0000) 2020-02-05T21:09:40.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:40.873+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.874909+0000) 2020-02-05T21:09:40.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:40.874+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.875604+0000) 2020-02-05T21:09:40.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:40.875+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.876605+0000) 2020-02-05T21:09:40.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:40.875+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.877035+0000) 2020-02-05T21:09:40.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:40.876+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.877904+0000) 2020-02-05T21:09:40.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:40.877+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.878594+0000) 2020-02-05T21:09:40.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:40.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.878917+0000) 2020-02-05T21:09:40.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:40.878+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.879466+0000) 2020-02-05T21:09:40.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:40.878+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.879880+0000) 2020-02-05T21:09:40.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:40.877+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.879376+0000) 2020-02-05T21:09:40.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:40.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.880496+0000) 2020-02-05T21:09:40.883 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:40.879+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.880521+0000) 2020-02-05T21:09:40.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:40.879+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.880746+0000) 2020-02-05T21:09:40.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:40.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:40.884346+0000) 2020-02-05T21:09:40.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:40.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77735 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:42.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:42.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:42.206785+0000) 2020-02-05T21:09:42.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:42.320+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:42.321519+0000) 2020-02-05T21:09:42.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:42.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:42.608406+0000) 2020-02-05T21:09:43.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:43.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:43.448070+0000) 2020-02-05T21:09:43.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:43.477+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:43.477382+0000) 2020-02-05T21:09:45.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:45.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.821353+0000) 2020-02-05T21:09:45.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:45.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.889916+0000) 2020-02-05T21:09:45.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:45.891+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.891233+0000) 2020-02-05T21:09:45.892 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:45.891+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.891761+0000) 2020-02-05T21:09:45.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:45.890+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.891414+0000) 2020-02-05T21:09:45.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:45.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.892470+0000) 2020-02-05T21:09:45.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:45.892+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.892759+0000) 2020-02-05T21:09:45.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:45.894+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.894360+0000) 2020-02-05T21:09:45.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:45.893+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.894739+0000) 2020-02-05T21:09:45.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:45.894+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.895823+0000) 2020-02-05T21:09:45.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:45.894+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.895900+0000) 2020-02-05T21:09:45.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:45.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.896668+0000) 2020-02-05T21:09:45.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:45.894+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.896106+0000) 2020-02-05T21:09:45.898 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:45.895+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.896866+0000) 2020-02-05T21:09:45.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:45.899+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:45.899958+0000) 2020-02-05T21:09:45.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:45.899+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:46.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:46.171+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:46.171933+0000) 2020-02-05T21:09:46.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:46.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:46.719215+0000) 2020-02-05T21:09:46.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:46.963+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:46.964206+0000) 2020-02-05T21:09:46.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:46.963+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:46.964296+0000) 2020-02-05T21:09:46.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:46.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:46.976405+0000) 2020-02-05T21:09:49.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:49.356+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:49.357796+0000) 2020-02-05T21:09:49.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:49.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:49.966181+0000) 2020-02-05T21:09:50.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:50.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.217063+0000) 2020-02-05T21:09:50.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:50.905+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.906924+0000) 2020-02-05T21:09:50.909 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:50.905+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.907340+0000) 2020-02-05T21:09:50.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:50.906+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.907598+0000) 2020-02-05T21:09:50.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:50.906+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.907822+0000) 2020-02-05T21:09:50.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:50.907+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.908532+0000) 2020-02-05T21:09:50.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:50.907+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.908900+0000) 2020-02-05T21:09:50.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:50.908+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.909208+0000) 2020-02-05T21:09:50.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:50.908+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.909786+0000) 2020-02-05T21:09:50.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:50.908+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.909771+0000) 2020-02-05T21:09:50.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:50.909+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.910416+0000) 2020-02-05T21:09:50.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:50.909+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.910699+0000) 2020-02-05T21:09:50.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:50.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.910845+0000) 2020-02-05T21:09:50.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:50.909+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.911218+0000) 2020-02-05T21:09:50.915 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:50.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:50.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:50.914+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:50.915911+0000) 2020-02-05T21:09:52.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:52.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:52.207046+0000) 2020-02-05T21:09:52.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:52.320+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:52.321755+0000) 2020-02-05T21:09:52.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:52.607+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:52.608659+0000) 2020-02-05T21:09:53.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:53.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:53.448303+0000) 2020-02-05T21:09:53.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:53.476+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:53.477614+0000) 2020-02-05T21:09:55.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:55.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.821584+0000) 2020-02-05T21:09:55.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:55.918+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.919624+0000) 2020-02-05T21:09:55.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:09:55.919+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.920863+0000) 2020-02-05T21:09:55.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:09:55.920+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.921635+0000) 2020-02-05T21:09:55.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:55.921+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.922109+0000) 2020-02-05T21:09:55.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:09:55.921+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.922604+0000) 2020-02-05T21:09:55.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:09:55.921+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.923272+0000) 2020-02-05T21:09:55.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:55.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.923711+0000) 2020-02-05T21:09:55.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:55.923+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.924616+0000) 2020-02-05T21:09:55.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:55.923+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.925351+0000) 2020-02-05T21:09:55.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:09:55.924+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.926102+0000) 2020-02-05T21:09:55.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:55.925+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.926485+0000) 2020-02-05T21:09:55.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:55.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.926952+0000) 2020-02-05T21:09:55.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:09:55.925+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.927184+0000) 2020-02-05T21:09:55.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:55.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:55.929399+0000) 2020-02-05T21:09:55.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:09:55.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:09:56.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:56.182+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:56.183457+0000) 2020-02-05T21:09:56.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:09:56.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:56.719440+0000) 2020-02-05T21:09:56.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:09:56.963+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:56.964483+0000) 2020-02-05T21:09:56.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:09:56.963+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:56.964613+0000) 2020-02-05T21:09:56.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:09:56.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:56.976648+0000) 2020-02-05T21:09:59.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:09:59.357+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:59.358026+0000) 2020-02-05T21:09:59.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:09:59.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:09:59.966424+0000) 2020-02-05T21:10:00.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:00.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.217298+0000) 2020-02-05T21:10:00.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:00.934+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.935832+0000) 2020-02-05T21:10:00.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:00.935+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.936671+0000) 2020-02-05T21:10:00.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:00.936+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.937761+0000) 2020-02-05T21:10:00.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:00.937+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.938321+0000) 2020-02-05T21:10:00.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:00.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.939551+0000) 2020-02-05T21:10:00.941 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:00.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.939377+0000) 2020-02-05T21:10:00.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:00.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.939958+0000) 2020-02-05T21:10:00.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:00.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.940483+0000) 2020-02-05T21:10:00.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:00.939+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.941009+0000) 2020-02-05T21:10:00.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:00.939+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.940853+0000) 2020-02-05T21:10:00.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:00.940+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.941660+0000) 2020-02-05T21:10:00.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:00.940+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.942105+0000) 2020-02-05T21:10:00.943 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:00.941+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.942603+0000) 2020-02-05T21:10:00.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:00.945+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:00.946736+0000) 2020-02-05T21:10:00.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:00.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77899 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:02.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:02.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:02.207291+0000) 2020-02-05T21:10:02.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:02.321+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:02.322017+0000) 2020-02-05T21:10:02.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:02.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:02.608917+0000) 2020-02-05T21:10:03.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:03.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:03.448539+0000) 2020-02-05T21:10:03.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:03.476+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:03.477869+0000) 2020-02-05T21:10:05.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:05.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.821818+0000) 2020-02-05T21:10:05.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:05.952+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.953459+0000) 2020-02-05T21:10:05.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:05.953+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.955175+0000) 2020-02-05T21:10:05.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:05.954+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.955585+0000) 2020-02-05T21:10:05.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:05.955+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.956740+0000) 2020-02-05T21:10:05.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:05.954+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.956240+0000) 2020-02-05T21:10:05.959 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:05.956+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.957314+0000) 2020-02-05T21:10:05.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:05.956+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.957625+0000) 2020-02-05T21:10:05.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:05.955+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.957162+0000) 2020-02-05T21:10:05.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:05.956+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.957956+0000) 2020-02-05T21:10:05.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:05.957+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.958224+0000) 2020-02-05T21:10:05.960 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:05.957+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.958733+0000) 2020-02-05T21:10:05.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:05.957+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.958527+0000) 2020-02-05T21:10:05.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:05.959+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.960459+0000) 2020-02-05T21:10:05.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:05.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:05.961427+0000) 2020-02-05T21:10:05.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:05.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:06.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:06.190+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:06.191271+0000) 2020-02-05T21:10:06.259 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:10:06.266 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:10:06.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:06.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:06.719665+0000) 2020-02-05T21:10:06.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:06.963+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:06.964694+0000) 2020-02-05T21:10:06.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:06.963+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:06.964833+0000) 2020-02-05T21:10:06.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:06.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:06.976896+0000) 2020-02-05T21:10:09.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:09.357+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:09.358308+0000) 2020-02-05T21:10:09.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:09.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:09.966656+0000) 2020-02-05T21:10:10.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:10.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.217549+0000) 2020-02-05T21:10:10.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:10.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.968286+0000) 2020-02-05T21:10:10.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:10.967+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.969258+0000) 2020-02-05T21:10:10.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:10.968+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.969953+0000) 2020-02-05T21:10:10.971 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:10.969+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.970264+0000) 2020-02-05T21:10:10.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:10.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.970608+0000) 2020-02-05T21:10:10.972 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:10.970+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.971353+0000) 2020-02-05T21:10:10.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:10.970+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.972018+0000) 2020-02-05T21:10:10.973 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:10.970+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.972073+0000) 2020-02-05T21:10:10.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:10.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.972779+0000) 2020-02-05T21:10:10.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:10.971+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.972835+0000) 2020-02-05T21:10:10.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:10.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.973908+0000) 2020-02-05T21:10:10.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:10.973+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.974354+0000) 2020-02-05T21:10:10.975 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:10.972+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.974259+0000) 2020-02-05T21:10:10.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:10.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 77981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:10.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:10.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:10.980548+0000) 2020-02-05T21:10:12.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:12.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:12.207554+0000) 2020-02-05T21:10:12.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:12.321+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:12.322315+0000) 2020-02-05T21:10:12.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:12.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:12.609148+0000) 2020-02-05T21:10:13.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:13.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:13.448789+0000) 2020-02-05T21:10:13.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:13.477+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:13.478130+0000) 2020-02-05T21:10:15.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:15.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.822043+0000) 2020-02-05T21:10:15.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:15.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.982475+0000) 2020-02-05T21:10:15.984 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:15.982+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.983298+0000) 2020-02-05T21:10:15.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:15.982+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.983523+0000) 2020-02-05T21:10:15.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:15.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.984054+0000) 2020-02-05T21:10:15.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:15.984+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.985510+0000) 2020-02-05T21:10:15.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:15.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.986072+0000) 2020-02-05T21:10:15.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:15.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.986634+0000) 2020-02-05T21:10:15.988 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:15.985+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.986500+0000) 2020-02-05T21:10:15.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:15.985+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.987215+0000) 2020-02-05T21:10:15.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:15.985+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.986931+0000) 2020-02-05T21:10:15.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:15.986+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.988063+0000) 2020-02-05T21:10:15.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:15.987+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.989115+0000) 2020-02-05T21:10:15.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:15.987+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.989375+0000) 2020-02-05T21:10:15.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:15.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:15.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:15.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:15.994282+0000) 2020-02-05T21:10:16.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:16.198+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:16.199418+0000) 2020-02-05T21:10:16.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:16.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:16.719884+0000) 2020-02-05T21:10:16.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:16.964+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:16.964949+0000) 2020-02-05T21:10:16.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:16.964+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:16.965114+0000) 2020-02-05T21:10:16.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:16.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:16.977127+0000) 2020-02-05T21:10:19.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:19.358+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:19.358592+0000) 2020-02-05T21:10:19.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:19.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:19.966906+0000) 2020-02-05T21:10:20.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:20.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:20.217787+0000) 2020-02-05T21:10:21.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:20.998+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:20.999388+0000) 2020-02-05T21:10:21.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:20.999+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.000752+0000) 2020-02-05T21:10:21.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:21.000+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.001247+0000) 2020-02-05T21:10:21.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:20.999+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.001276+0000) 2020-02-05T21:10:21.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:21.000+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.001727+0000) 2020-02-05T21:10:21.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:21.001+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.002202+0000) 2020-02-05T21:10:21.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:21.001+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.002642+0000) 2020-02-05T21:10:21.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:21.001+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.002943+0000) 2020-02-05T21:10:21.005 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:21.002+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.003695+0000) 2020-02-05T21:10:21.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:21.002+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.004334+0000) 2020-02-05T21:10:21.005 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:21.002+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.004367+0000) 2020-02-05T21:10:21.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:21.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.005485+0000) 2020-02-05T21:10:21.006 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:21.004+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.005495+0000) 2020-02-05T21:10:21.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:21.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:21.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:21.007+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:21.008962+0000) 2020-02-05T21:10:22.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:22.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:22.207792+0000) 2020-02-05T21:10:22.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:22.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:22.322548+0000) 2020-02-05T21:10:22.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:22.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:22.609378+0000) 2020-02-05T21:10:23.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:23.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:23.449071+0000) 2020-02-05T21:10:23.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:23.477+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:23.478359+0000) 2020-02-05T21:10:25.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:25.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:25.822297+0000) 2020-02-05T21:10:26.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:26.013+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.014443+0000) 2020-02-05T21:10:26.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:26.013+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.014919+0000) 2020-02-05T21:10:26.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:26.014+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.015539+0000) 2020-02-05T21:10:26.017 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:26.014+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.015837+0000) 2020-02-05T21:10:26.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:26.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.016320+0000) 2020-02-05T21:10:26.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:26.016+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.016739+0000) 2020-02-05T21:10:26.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:26.016+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.017230+0000) 2020-02-05T21:10:26.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:26.017+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.018146+0000) 2020-02-05T21:10:26.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:26.017+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.018798+0000) 2020-02-05T21:10:26.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:26.018+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.019249+0000) 2020-02-05T21:10:26.021 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:26.018+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.019214+0000) 2020-02-05T21:10:26.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:26.018+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.019724+0000) 2020-02-05T21:10:26.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:26.019+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.020176+0000) 2020-02-05T21:10:26.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:26.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.024565+0000) 2020-02-05T21:10:26.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:26.024+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:26.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:26.206+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.206800+0000) 2020-02-05T21:10:26.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:26.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.720151+0000) 2020-02-05T21:10:26.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:26.964+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.965175+0000) 2020-02-05T21:10:26.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:26.964+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.965314+0000) 2020-02-05T21:10:26.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:26.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:26.977324+0000) 2020-02-05T21:10:29.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:29.357+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:29.358838+0000) 2020-02-05T21:10:29.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:29.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:29.967135+0000) 2020-02-05T21:10:30.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:30.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:30.218086+0000) 2020-02-05T21:10:31.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:31.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.029877+0000) 2020-02-05T21:10:31.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:31.029+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.030521+0000) 2020-02-05T21:10:31.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:31.029+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.031052+0000) 2020-02-05T21:10:31.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:31.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.031644+0000) 2020-02-05T21:10:31.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:31.031+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.032939+0000) 2020-02-05T21:10:31.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:31.033+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.033515+0000) 2020-02-05T21:10:31.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:31.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.033746+0000) 2020-02-05T21:10:31.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:31.032+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.034035+0000) 2020-02-05T21:10:31.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:31.034+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.034603+0000) 2020-02-05T21:10:31.036 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:31.034+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.035016+0000) 2020-02-05T21:10:31.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:31.035+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.035936+0000) 2020-02-05T21:10:31.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:31.036+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.036770+0000) 2020-02-05T21:10:31.038 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:31.036+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.036543+0000) 2020-02-05T21:10:31.041 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:31.039+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78145 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:31.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:31.041+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:31.042995+0000) 2020-02-05T21:10:32.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:32.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:32.208037+0000) 2020-02-05T21:10:32.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:32.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:32.322801+0000) 2020-02-05T21:10:32.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:32.609+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:32.609621+0000) 2020-02-05T21:10:33.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:33.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:33.449325+0000) 2020-02-05T21:10:33.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:33.478+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:33.478595+0000) 2020-02-05T21:10:35.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:35.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:35.822528+0000) 2020-02-05T21:10:36.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:36.044+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.045705+0000) 2020-02-05T21:10:36.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:36.045+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.046074+0000) 2020-02-05T21:10:36.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:36.045+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.046625+0000) 2020-02-05T21:10:36.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:36.046+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.047882+0000) 2020-02-05T21:10:36.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:36.047+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.048270+0000) 2020-02-05T21:10:36.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:36.047+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.048337+0000) 2020-02-05T21:10:36.051 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:36.047+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.048945+0000) 2020-02-05T21:10:36.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:36.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.049899+0000) 2020-02-05T21:10:36.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:36.049+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.050216+0000) 2020-02-05T21:10:36.052 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:36.050+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.050727+0000) 2020-02-05T21:10:36.052 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:36.050+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.051287+0000) 2020-02-05T21:10:36.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:36.050+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.051253+0000) 2020-02-05T21:10:36.053 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:36.051+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.051983+0000) 2020-02-05T21:10:36.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:36.052+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.053419+0000) 2020-02-05T21:10:36.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:36.055+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:36.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:36.214+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.214929+0000) 2020-02-05T21:10:36.435 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:10:36.441 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:10:36.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:36.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.720397+0000) 2020-02-05T21:10:36.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:36.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.965405+0000) 2020-02-05T21:10:36.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:36.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.965540+0000) 2020-02-05T21:10:36.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:36.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:36.977536+0000) 2020-02-05T21:10:39.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:39.359+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:39.359071+0000) 2020-02-05T21:10:39.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:39.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:39.967403+0000) 2020-02-05T21:10:40.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:40.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:40.218319+0000) 2020-02-05T21:10:41.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:41.061+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.062005+0000) 2020-02-05T21:10:41.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:41.063+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.063069+0000) 2020-02-05T21:10:41.064 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:41.063+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.063362+0000) 2020-02-05T21:10:41.065 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:41.063+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.064035+0000) 2020-02-05T21:10:41.065 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:41.064+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.064200+0000) 2020-02-05T21:10:41.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:41.064+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.064821+0000) 2020-02-05T21:10:41.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:41.064+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.064885+0000) 2020-02-05T21:10:41.066 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:41.065+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.065646+0000) 2020-02-05T21:10:41.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:41.065+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.065695+0000) 2020-02-05T21:10:41.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:41.065+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.065923+0000) 2020-02-05T21:10:41.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:41.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.070511+0000) 2020-02-05T21:10:41.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:41.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.070779+0000) 2020-02-05T21:10:41.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:41.065+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.066366+0000) 2020-02-05T21:10:41.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:41.066+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:41.067026+0000) 2020-02-05T21:10:41.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:41.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78227 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:42.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:42.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:42.208279+0000) 2020-02-05T21:10:42.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:42.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:42.323061+0000) 2020-02-05T21:10:42.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:42.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:42.609852+0000) 2020-02-05T21:10:43.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:43.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:43.449553+0000) 2020-02-05T21:10:43.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:43.477+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:43.478834+0000) 2020-02-05T21:10:45.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:45.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:45.822766+0000) 2020-02-05T21:10:46.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:46.075+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.076279+0000) 2020-02-05T21:10:46.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:46.075+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.076592+0000) 2020-02-05T21:10:46.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:46.076+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.077728+0000) 2020-02-05T21:10:46.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:46.077+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.077560+0000) 2020-02-05T21:10:46.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:46.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.079423+0000) 2020-02-05T21:10:46.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:46.079+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.080018+0000) 2020-02-05T21:10:46.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:46.079+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.080781+0000) 2020-02-05T21:10:46.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:46.080+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.081243+0000) 2020-02-05T21:10:46.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:46.081+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.081499+0000) 2020-02-05T21:10:46.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:46.081+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.081787+0000) 2020-02-05T21:10:46.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:46.081+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.082463+0000) 2020-02-05T21:10:46.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:46.081+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.082153+0000) 2020-02-05T21:10:46.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:46.081+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.082063+0000) 2020-02-05T21:10:46.087 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:46.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:46.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:46.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.089985+0000) 2020-02-05T21:10:46.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:46.221+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.222923+0000) 2020-02-05T21:10:46.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:46.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.720611+0000) 2020-02-05T21:10:46.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:46.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.965682+0000) 2020-02-05T21:10:46.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:46.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.965791+0000) 2020-02-05T21:10:46.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:46.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:46.977739+0000) 2020-02-05T21:10:49.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:49.358+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:49.359400+0000) 2020-02-05T21:10:49.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:49.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:49.967656+0000) 2020-02-05T21:10:50.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:50.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:50.218566+0000) 2020-02-05T21:10:51.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:51.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.092131+0000) 2020-02-05T21:10:51.094 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:51.092+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.092699+0000) 2020-02-05T21:10:51.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:51.092+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.093403+0000) 2020-02-05T21:10:51.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:51.093+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.094210+0000) 2020-02-05T21:10:51.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:51.093+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.094244+0000) 2020-02-05T21:10:51.096 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:51.093+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.094943+0000) 2020-02-05T21:10:51.096 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:51.094+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.095747+0000) 2020-02-05T21:10:51.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:51.095+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.096361+0000) 2020-02-05T21:10:51.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:51.095+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.096782+0000) 2020-02-05T21:10:51.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:51.096+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.096829+0000) 2020-02-05T21:10:51.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:51.097+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.097622+0000) 2020-02-05T21:10:51.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:51.096+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.098020+0000) 2020-02-05T21:10:51.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:51.097+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.097983+0000) 2020-02-05T21:10:51.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:51.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78309 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:51.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:51.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:51.102680+0000) 2020-02-05T21:10:52.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:52.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:52.208531+0000) 2020-02-05T21:10:52.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:52.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:52.323284+0000) 2020-02-05T21:10:52.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:52.609+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:52.610141+0000) 2020-02-05T21:10:53.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:53.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:53.449850+0000) 2020-02-05T21:10:53.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:53.478+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:53.479055+0000) 2020-02-05T21:10:55.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:55.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:55.822988+0000) 2020-02-05T21:10:56.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:56.106+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.107231+0000) 2020-02-05T21:10:56.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:10:56.107+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.108089+0000) 2020-02-05T21:10:56.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:56.107+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.108645+0000) 2020-02-05T21:10:56.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:10:56.108+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.109407+0000) 2020-02-05T21:10:56.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:56.109+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.110514+0000) 2020-02-05T21:10:56.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:10:56.110+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.111317+0000) 2020-02-05T21:10:56.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:56.110+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.111723+0000) 2020-02-05T21:10:56.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:56.108+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.108897+0000) 2020-02-05T21:10:56.115 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:10:56.109+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.109699+0000) 2020-02-05T21:10:56.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:10:56.112+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.113076+0000) 2020-02-05T21:10:56.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:10:56.112+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.113363+0000) 2020-02-05T21:10:56.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:56.113+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.113833+0000) 2020-02-05T21:10:56.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:56.116+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.117401+0000) 2020-02-05T21:10:56.119 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:10:56.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78347 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:10:56.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:56.117+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.118743+0000) 2020-02-05T21:10:56.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:56.230+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.231215+0000) 2020-02-05T21:10:56.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:10:56.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.720813+0000) 2020-02-05T21:10:56.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:10:56.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.965888+0000) 2020-02-05T21:10:56.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:10:56.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.966058+0000) 2020-02-05T21:10:56.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:10:56.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:56.977940+0000) 2020-02-05T21:10:59.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:10:59.358+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:59.359670+0000) 2020-02-05T21:10:59.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:10:59.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:10:59.967908+0000) 2020-02-05T21:11:00.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:00.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:00.218811+0000) 2020-02-05T21:11:01.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:01.122+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.123873+0000) 2020-02-05T21:11:01.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:01.124+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.125144+0000) 2020-02-05T21:11:01.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:01.125+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.125781+0000) 2020-02-05T21:11:01.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:01.125+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.126207+0000) 2020-02-05T21:11:01.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:01.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.126574+0000) 2020-02-05T21:11:01.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:01.125+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.127067+0000) 2020-02-05T21:11:01.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:01.126+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.127281+0000) 2020-02-05T21:11:01.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:01.128+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.128573+0000) 2020-02-05T21:11:01.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:01.127+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.128879+0000) 2020-02-05T21:11:01.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:01.128+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.129208+0000) 2020-02-05T21:11:01.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:01.129+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.129791+0000) 2020-02-05T21:11:01.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:01.129+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.129743+0000) 2020-02-05T21:11:01.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:01.129+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.130084+0000) 2020-02-05T21:11:01.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:01.132+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:01.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:01.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:01.134396+0000) 2020-02-05T21:11:02.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:02.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:02.208789+0000) 2020-02-05T21:11:02.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:02.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:02.323578+0000) 2020-02-05T21:11:02.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:02.609+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:02.610383+0000) 2020-02-05T21:11:03.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:03.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:03.450069+0000) 2020-02-05T21:11:03.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:03.478+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:03.479360+0000) 2020-02-05T21:11:05.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:05.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:05.823219+0000) 2020-02-05T21:11:06.139 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:06.137+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.138407+0000) 2020-02-05T21:11:06.140 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:06.138+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.139119+0000) 2020-02-05T21:11:06.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:06.138+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.139599+0000) 2020-02-05T21:11:06.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:06.139+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.140985+0000) 2020-02-05T21:11:06.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:06.139+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.141298+0000) 2020-02-05T21:11:06.143 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:06.141+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.142423+0000) 2020-02-05T21:11:06.144 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:06.142+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.143564+0000) 2020-02-05T21:11:06.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:06.142+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.143837+0000) 2020-02-05T21:11:06.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:06.142+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.144098+0000) 2020-02-05T21:11:06.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:06.143+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.144840+0000) 2020-02-05T21:11:06.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:06.143+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.145198+0000) 2020-02-05T21:11:06.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:06.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.145558+0000) 2020-02-05T21:11:06.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:06.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.146819+0000) 2020-02-05T21:11:06.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:06.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.147012+0000) 2020-02-05T21:11:06.150 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:06.147+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:06.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:06.235+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.236618+0000) 2020-02-05T21:11:06.640 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:11:06.646 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:11:06.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:06.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.720991+0000) 2020-02-05T21:11:06.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:06.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.966120+0000) 2020-02-05T21:11:06.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:06.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.966271+0000) 2020-02-05T21:11:06.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:06.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:06.978145+0000) 2020-02-05T21:11:09.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:09.359+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:09.359933+0000) 2020-02-05T21:11:09.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:09.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:09.968211+0000) 2020-02-05T21:11:10.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:10.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:10.219044+0000) 2020-02-05T21:11:11.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:11.153+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.154715+0000) 2020-02-05T21:11:11.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:11.153+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.155380+0000) 2020-02-05T21:11:11.157 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:11.155+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.156307+0000) 2020-02-05T21:11:11.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:11.155+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.156794+0000) 2020-02-05T21:11:11.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:11.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.157412+0000) 2020-02-05T21:11:11.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:11.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.158273+0000) 2020-02-05T21:11:11.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:11.157+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.158606+0000) 2020-02-05T21:11:11.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:11.158+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.159424+0000) 2020-02-05T21:11:11.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:11.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.159952+0000) 2020-02-05T21:11:11.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:11.158+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.160209+0000) 2020-02-05T21:11:11.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:11.160+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.161195+0000) 2020-02-05T21:11:11.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:11.160+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.161693+0000) 2020-02-05T21:11:11.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:11.159+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.161192+0000) 2020-02-05T21:11:11.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:11.163+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:11.164965+0000) 2020-02-05T21:11:11.166 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:11.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:12.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:12.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:12.209054+0000) 2020-02-05T21:11:12.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:12.322+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:12.323836+0000) 2020-02-05T21:11:12.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:12.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:12.610637+0000) 2020-02-05T21:11:13.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:13.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:13.450253+0000) 2020-02-05T21:11:13.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:13.479+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:13.479615+0000) 2020-02-05T21:11:15.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:15.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:15.823465+0000) 2020-02-05T21:11:16.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:16.170+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.171689+0000) 2020-02-05T21:11:16.173 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:16.169+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.171198+0000) 2020-02-05T21:11:16.173 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:16.170+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.172387+0000) 2020-02-05T21:11:16.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:16.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.172911+0000) 2020-02-05T21:11:16.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:16.172+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.173365+0000) 2020-02-05T21:11:16.175 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:16.172+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.173879+0000) 2020-02-05T21:11:16.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:16.173+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.174427+0000) 2020-02-05T21:11:16.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:16.173+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.174654+0000) 2020-02-05T21:11:16.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:16.173+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.175370+0000) 2020-02-05T21:11:16.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:16.174+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.175948+0000) 2020-02-05T21:11:16.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:16.175+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.177130+0000) 2020-02-05T21:11:16.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:16.176+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.177417+0000) 2020-02-05T21:11:16.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:16.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.177475+0000) 2020-02-05T21:11:16.182 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:16.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:16.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:16.181+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.182722+0000) 2020-02-05T21:11:16.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:16.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.244571+0000) 2020-02-05T21:11:16.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:16.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.721180+0000) 2020-02-05T21:11:16.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:16.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.966352+0000) 2020-02-05T21:11:16.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:16.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.966506+0000) 2020-02-05T21:11:16.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:16.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:16.978354+0000) 2020-02-05T21:11:19.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:19.359+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:19.360169+0000) 2020-02-05T21:11:19.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:19.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:19.968440+0000) 2020-02-05T21:11:20.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:20.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:20.219272+0000) 2020-02-05T21:11:21.188 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:21.185+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.187026+0000) 2020-02-05T21:11:21.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:21.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.187387+0000) 2020-02-05T21:11:21.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:21.186+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.187667+0000) 2020-02-05T21:11:21.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:21.186+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.187912+0000) 2020-02-05T21:11:21.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:21.187+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.188549+0000) 2020-02-05T21:11:21.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:21.187+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.189051+0000) 2020-02-05T21:11:21.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:21.188+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.190277+0000) 2020-02-05T21:11:21.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:21.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.190871+0000) 2020-02-05T21:11:21.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:21.190+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.191278+0000) 2020-02-05T21:11:21.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:21.190+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.191544+0000) 2020-02-05T21:11:21.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:21.191+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.192139+0000) 2020-02-05T21:11:21.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:21.191+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.192729+0000) 2020-02-05T21:11:21.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:21.191+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.192908+0000) 2020-02-05T21:11:21.197 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:21.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:21.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:21.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:21.198483+0000) 2020-02-05T21:11:22.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:22.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:22.209313+0000) 2020-02-05T21:11:22.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:22.323+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:22.324187+0000) 2020-02-05T21:11:22.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:22.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:22.610872+0000) 2020-02-05T21:11:23.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:23.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:23.450484+0000) 2020-02-05T21:11:23.481 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:23.479+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:23.479848+0000) 2020-02-05T21:11:25.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:25.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:25.823773+0000) 2020-02-05T21:11:26.202 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:26.200+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.201692+0000) 2020-02-05T21:11:26.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:26.201+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.202883+0000) 2020-02-05T21:11:26.204 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:26.202+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.203418+0000) 2020-02-05T21:11:26.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:26.202+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.203840+0000) 2020-02-05T21:11:26.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:26.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.204511+0000) 2020-02-05T21:11:26.206 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:26.204+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.205421+0000) 2020-02-05T21:11:26.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:26.204+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.206030+0000) 2020-02-05T21:11:26.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:26.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.206573+0000) 2020-02-05T21:11:26.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:26.205+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.206953+0000) 2020-02-05T21:11:26.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:26.206+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.207492+0000) 2020-02-05T21:11:26.209 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:26.205+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.207131+0000) 2020-02-05T21:11:26.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:26.206+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.207802+0000) 2020-02-05T21:11:26.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:26.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.208169+0000) 2020-02-05T21:11:26.213 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:26.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78593 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:26.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:26.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.213513+0000) 2020-02-05T21:11:26.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:26.255+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.256441+0000) 2020-02-05T21:11:26.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:26.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.721390+0000) 2020-02-05T21:11:26.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:26.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.966558+0000) 2020-02-05T21:11:26.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:26.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.966718+0000) 2020-02-05T21:11:26.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:26.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:26.978562+0000) 2020-02-05T21:11:29.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:29.360+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:29.360399+0000) 2020-02-05T21:11:29.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:29.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:29.968661+0000) 2020-02-05T21:11:30.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:30.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:30.219520+0000) 2020-02-05T21:11:31.218 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:31.215+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.216835+0000) 2020-02-05T21:11:31.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:31.217+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.218576+0000) 2020-02-05T21:11:31.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:31.217+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.219208+0000) 2020-02-05T21:11:31.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:31.219+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.220218+0000) 2020-02-05T21:11:31.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:31.219+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.220616+0000) 2020-02-05T21:11:31.222 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:31.219+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.220922+0000) 2020-02-05T21:11:31.222 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:31.220+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.221591+0000) 2020-02-05T21:11:31.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:31.220+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.221674+0000) 2020-02-05T21:11:31.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:31.220+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.221900+0000) 2020-02-05T21:11:31.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:31.221+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.223004+0000) 2020-02-05T21:11:31.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:31.221+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.222650+0000) 2020-02-05T21:11:31.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:31.221+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.223360+0000) 2020-02-05T21:11:31.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:31.222+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.223521+0000) 2020-02-05T21:11:31.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:31.226+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:31.227558+0000) 2020-02-05T21:11:31.229 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:31.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:32.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:32.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:32.209582+0000) 2020-02-05T21:11:32.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:32.323+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:32.324424+0000) 2020-02-05T21:11:32.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:32.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:32.611137+0000) 2020-02-05T21:11:33.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:33.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:33.450709+0000) 2020-02-05T21:11:33.481 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:33.480+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:33.480111+0000) 2020-02-05T21:11:35.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:35.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:35.824024+0000) 2020-02-05T21:11:36.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:36.232+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.233899+0000) 2020-02-05T21:11:36.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:36.235+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.235668+0000) 2020-02-05T21:11:36.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:36.236+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.236213+0000) 2020-02-05T21:11:36.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:36.235+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.236585+0000) 2020-02-05T21:11:36.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:36.236+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.236800+0000) 2020-02-05T21:11:36.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:36.235+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.237247+0000) 2020-02-05T21:11:36.239 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:36.237+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.237566+0000) 2020-02-05T21:11:36.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:36.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.237891+0000) 2020-02-05T21:11:36.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:36.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.239268+0000) 2020-02-05T21:11:36.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:36.239+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.239418+0000) 2020-02-05T21:11:36.241 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:36.239+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.239536+0000) 2020-02-05T21:11:36.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:36.238+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.239635+0000) 2020-02-05T21:11:36.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:36.238+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.240013+0000) 2020-02-05T21:11:36.244 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:36.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:36.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:36.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.245687+0000) 2020-02-05T21:11:36.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:36.264+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.264769+0000) 2020-02-05T21:11:36.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:36.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.721599+0000) 2020-02-05T21:11:36.829 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:11:36.835 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:11:36.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:36.965+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.966726+0000) 2020-02-05T21:11:36.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:36.965+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.966909+0000) 2020-02-05T21:11:36.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:36.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:36.978807+0000) 2020-02-05T21:11:39.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:39.359+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:39.360637+0000) 2020-02-05T21:11:39.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:39.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:39.968918+0000) 2020-02-05T21:11:40.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:40.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:40.219755+0000) 2020-02-05T21:11:41.250 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:41.248+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.249402+0000) 2020-02-05T21:11:41.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:41.248+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.250049+0000) 2020-02-05T21:11:41.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:41.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.250820+0000) 2020-02-05T21:11:41.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:41.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.253497+0000) 2020-02-05T21:11:41.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:41.250+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.251696+0000) 2020-02-05T21:11:41.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:41.251+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.252197+0000) 2020-02-05T21:11:41.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:41.251+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.252940+0000) 2020-02-05T21:11:41.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:41.251+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.252579+0000) 2020-02-05T21:11:41.257 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:41.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.253805+0000) 2020-02-05T21:11:41.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:41.253+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.254495+0000) 2020-02-05T21:11:41.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:41.252+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.254281+0000) 2020-02-05T21:11:41.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:41.253+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.254935+0000) 2020-02-05T21:11:41.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:41.254+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.255674+0000) 2020-02-05T21:11:41.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:41.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:41.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:41.260+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:41.261295+0000) 2020-02-05T21:11:42.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:42.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:42.209816+0000) 2020-02-05T21:11:42.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:42.323+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:42.324748+0000) 2020-02-05T21:11:42.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:42.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:42.611371+0000) 2020-02-05T21:11:43.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:43.450+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:43.450962+0000) 2020-02-05T21:11:43.481 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:43.479+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:43.480340+0000) 2020-02-05T21:11:45.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:45.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:45.824285+0000) 2020-02-05T21:11:46.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:46.263+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.264970+0000) 2020-02-05T21:11:46.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:46.265+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.266665+0000) 2020-02-05T21:11:46.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:46.265+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.267019+0000) 2020-02-05T21:11:46.268 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:46.266+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.267423+0000) 2020-02-05T21:11:46.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:46.266+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.267726+0000) 2020-02-05T21:11:46.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:46.266+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.267914+0000) 2020-02-05T21:11:46.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:46.267+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.268507+0000) 2020-02-05T21:11:46.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:46.268+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.269424+0000) 2020-02-05T21:11:46.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:46.268+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.269924+0000) 2020-02-05T21:11:46.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:46.267+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.269062+0000) 2020-02-05T21:11:46.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:46.269+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.270960+0000) 2020-02-05T21:11:46.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:46.270+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.271411+0000) 2020-02-05T21:11:46.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:46.269+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.271288+0000) 2020-02-05T21:11:46.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:46.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:46.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:46.274+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.275914+0000) 2020-02-05T21:11:46.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:46.274+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.275970+0000) 2020-02-05T21:11:46.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:46.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.721814+0000) 2020-02-05T21:11:46.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:46.966+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.966967+0000) 2020-02-05T21:11:46.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:46.966+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.967113+0000) 2020-02-05T21:11:46.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:46.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:46.979019+0000) 2020-02-05T21:11:49.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:49.360+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:49.360882+0000) 2020-02-05T21:11:49.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:49.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:49.969152+0000) 2020-02-05T21:11:50.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:50.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:50.220015+0000) 2020-02-05T21:11:51.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:51.278+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.279561+0000) 2020-02-05T21:11:51.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:51.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.280349+0000) 2020-02-05T21:11:51.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:51.279+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.281247+0000) 2020-02-05T21:11:51.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:51.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.284072+0000) 2020-02-05T21:11:51.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:51.283+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.283970+0000) 2020-02-05T21:11:51.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:51.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.284790+0000) 2020-02-05T21:11:51.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:51.284+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.285215+0000) 2020-02-05T21:11:51.287 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:51.284+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.285464+0000) 2020-02-05T21:11:51.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:51.284+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.286083+0000) 2020-02-05T21:11:51.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:51.284+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.286338+0000) 2020-02-05T21:11:51.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:51.285+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.286759+0000) 2020-02-05T21:11:51.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:51.285+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.287202+0000) 2020-02-05T21:11:51.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:51.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.289744+0000) 2020-02-05T21:11:51.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:51.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:51.290001+0000) 2020-02-05T21:11:51.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:51.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:52.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:52.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:52.210031+0000) 2020-02-05T21:11:52.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:52.324+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:52.325060+0000) 2020-02-05T21:11:52.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:52.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:52.611604+0000) 2020-02-05T21:11:53.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:53.450+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:53.451256+0000) 2020-02-05T21:11:53.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:53.479+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:53.480581+0000) 2020-02-05T21:11:55.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:55.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:55.824466+0000) 2020-02-05T21:11:56.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:56.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.276201+0000) 2020-02-05T21:11:56.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:11:56.297+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.297742+0000) 2020-02-05T21:11:56.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:11:56.297+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.298421+0000) 2020-02-05T21:11:56.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:56.297+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.298852+0000) 2020-02-05T21:11:56.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:56.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.299485+0000) 2020-02-05T21:11:56.301 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:11:56.299+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.300278+0000) 2020-02-05T21:11:56.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:56.299+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.300882+0000) 2020-02-05T21:11:56.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:11:56.301+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.301466+0000) 2020-02-05T21:11:56.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:56.301+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.302358+0000) 2020-02-05T21:11:56.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:56.302+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.302690+0000) 2020-02-05T21:11:56.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:11:56.302+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.302675+0000) 2020-02-05T21:11:56.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:11:56.302+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.303249+0000) 2020-02-05T21:11:56.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:56.303+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.304404+0000) 2020-02-05T21:11:56.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:56.303+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.304972+0000) 2020-02-05T21:11:56.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:11:56.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:11:56.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:56.307+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.308145+0000) 2020-02-05T21:11:56.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:11:56.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.722025+0000) 2020-02-05T21:11:56.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:11:56.966+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.967183+0000) 2020-02-05T21:11:56.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:11:56.966+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.967308+0000) 2020-02-05T21:11:56.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:11:56.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:56.979257+0000) 2020-02-05T21:11:59.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:11:59.360+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:59.361141+0000) 2020-02-05T21:11:59.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:11:59.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:11:59.969389+0000) 2020-02-05T21:12:00.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:00.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:00.220250+0000) 2020-02-05T21:12:01.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:01.312+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.313715+0000) 2020-02-05T21:12:01.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:01.313+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.314502+0000) 2020-02-05T21:12:01.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:01.315+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.315810+0000) 2020-02-05T21:12:01.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:01.315+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.315785+0000) 2020-02-05T21:12:01.318 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:01.315+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.316361+0000) 2020-02-05T21:12:01.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:01.315+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.316836+0000) 2020-02-05T21:12:01.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:01.316+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.317459+0000) 2020-02-05T21:12:01.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:01.317+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.317891+0000) 2020-02-05T21:12:01.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:01.316+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.317986+0000) 2020-02-05T21:12:01.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:01.317+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.318339+0000) 2020-02-05T21:12:01.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:01.317+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.318277+0000) 2020-02-05T21:12:01.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:01.317+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.318901+0000) 2020-02-05T21:12:01.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:01.319+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.319492+0000) 2020-02-05T21:12:01.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:01.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:01.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:01.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:01.324702+0000) 2020-02-05T21:12:02.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:02.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:02.210252+0000) 2020-02-05T21:12:02.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:02.324+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:02.325308+0000) 2020-02-05T21:12:02.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:02.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:02.611848+0000) 2020-02-05T21:12:03.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:03.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:03.451495+0000) 2020-02-05T21:12:03.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:03.479+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:03.480811+0000) 2020-02-05T21:12:05.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:05.824+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:05.824701+0000) 2020-02-05T21:12:06.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:06.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.284676+0000) 2020-02-05T21:12:06.330 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:06.328+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.329355+0000) 2020-02-05T21:12:06.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:06.329+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.329610+0000) 2020-02-05T21:12:06.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:06.329+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.330307+0000) 2020-02-05T21:12:06.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:06.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.330890+0000) 2020-02-05T21:12:06.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:06.330+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.331262+0000) 2020-02-05T21:12:06.333 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:06.330+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.331598+0000) 2020-02-05T21:12:06.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:06.330+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.331574+0000) 2020-02-05T21:12:06.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:06.331+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.332225+0000) 2020-02-05T21:12:06.337 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:06.333+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.334620+0000) 2020-02-05T21:12:06.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:06.332+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.332997+0000) 2020-02-05T21:12:06.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:06.332+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.333171+0000) 2020-02-05T21:12:06.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:06.334+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.334605+0000) 2020-02-05T21:12:06.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:06.334+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.335074+0000) 2020-02-05T21:12:06.340 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:06.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:06.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:06.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.339910+0000) 2020-02-05T21:12:06.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:06.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.722268+0000) 2020-02-05T21:12:06.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:06.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.967404+0000) 2020-02-05T21:12:06.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:06.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.967552+0000) 2020-02-05T21:12:06.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:06.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:06.979470+0000) 2020-02-05T21:12:07.031 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:12:07.037 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:12:09.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:09.360+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:09.361379+0000) 2020-02-05T21:12:09.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:09.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:09.969625+0000) 2020-02-05T21:12:10.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:10.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:10.220503+0000) 2020-02-05T21:12:11.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:11.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.344460+0000) 2020-02-05T21:12:11.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:11.344+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.345883+0000) 2020-02-05T21:12:11.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:11.345+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.346258+0000) 2020-02-05T21:12:11.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:11.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.346678+0000) 2020-02-05T21:12:11.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:11.346+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.347004+0000) 2020-02-05T21:12:11.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:11.346+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.347559+0000) 2020-02-05T21:12:11.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:11.347+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.348013+0000) 2020-02-05T21:12:11.350 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:11.347+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.348764+0000) 2020-02-05T21:12:11.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:11.348+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.348793+0000) 2020-02-05T21:12:11.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:11.348+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.349243+0000) 2020-02-05T21:12:11.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:11.349+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.349612+0000) 2020-02-05T21:12:11.352 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:11.350+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.350719+0000) 2020-02-05T21:12:11.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:11.350+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.350884+0000) 2020-02-05T21:12:11.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:11.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 78964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:11.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:11.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:11.358027+0000) 2020-02-05T21:12:12.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:12.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:12.210484+0000) 2020-02-05T21:12:12.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:12.325+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:12.325515+0000) 2020-02-05T21:12:12.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:12.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:12.612113+0000) 2020-02-05T21:12:13.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:13.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:13.451768+0000) 2020-02-05T21:12:13.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:13.480+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:13.481115+0000) 2020-02-05T21:12:15.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:15.824+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:15.824978+0000) 2020-02-05T21:12:16.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:16.292+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.292420+0000) 2020-02-05T21:12:16.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:16.359+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.360379+0000) 2020-02-05T21:12:16.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:16.360+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.361130+0000) 2020-02-05T21:12:16.363 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:16.360+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.361859+0000) 2020-02-05T21:12:16.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:16.361+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.362170+0000) 2020-02-05T21:12:16.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:16.362+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.362833+0000) 2020-02-05T21:12:16.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:16.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.363560+0000) 2020-02-05T21:12:16.365 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:16.364+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.364520+0000) 2020-02-05T21:12:16.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:16.364+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.364740+0000) 2020-02-05T21:12:16.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:16.365+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.365894+0000) 2020-02-05T21:12:16.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:16.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.366406+0000) 2020-02-05T21:12:16.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:16.365+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.365900+0000) 2020-02-05T21:12:16.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:16.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.366927+0000) 2020-02-05T21:12:16.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:16.366+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.367025+0000) 2020-02-05T21:12:16.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:16.369+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.370859+0000) 2020-02-05T21:12:16.373 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:16.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:16.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:16.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.722479+0000) 2020-02-05T21:12:16.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:16.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.967678+0000) 2020-02-05T21:12:16.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:16.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.967824+0000) 2020-02-05T21:12:16.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:16.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:16.979724+0000) 2020-02-05T21:12:19.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:19.361+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:19.361615+0000) 2020-02-05T21:12:19.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:19.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:19.969862+0000) 2020-02-05T21:12:20.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:20.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:20.220803+0000) 2020-02-05T21:12:21.379 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:21.376+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.377717+0000) 2020-02-05T21:12:21.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:21.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.378676+0000) 2020-02-05T21:12:21.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:21.378+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.379322+0000) 2020-02-05T21:12:21.380 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:21.378+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.379688+0000) 2020-02-05T21:12:21.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:21.379+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.380218+0000) 2020-02-05T21:12:21.382 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:21.380+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.380890+0000) 2020-02-05T21:12:21.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:21.380+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.381286+0000) 2020-02-05T21:12:21.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:21.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.382229+0000) 2020-02-05T21:12:21.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:21.381+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.382360+0000) 2020-02-05T21:12:21.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:21.382+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.383091+0000) 2020-02-05T21:12:21.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:21.383+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.383467+0000) 2020-02-05T21:12:21.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:21.383+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.384288+0000) 2020-02-05T21:12:21.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:21.384+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.384937+0000) 2020-02-05T21:12:21.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:21.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:21.387745+0000) 2020-02-05T21:12:21.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:21.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:22.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:22.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:22.210723+0000) 2020-02-05T21:12:22.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:22.325+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:22.325749+0000) 2020-02-05T21:12:22.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:22.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:22.612346+0000) 2020-02-05T21:12:23.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:23.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:23.451999+0000) 2020-02-05T21:12:23.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:23.481+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:23.481380+0000) 2020-02-05T21:12:25.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:25.825+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:25.825212+0000) 2020-02-05T21:12:26.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:26.300+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.300649+0000) 2020-02-05T21:12:26.394 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:26.391+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.393010+0000) 2020-02-05T21:12:26.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:26.392+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.393587+0000) 2020-02-05T21:12:26.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:26.394+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.395310+0000) 2020-02-05T21:12:26.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:26.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.395391+0000) 2020-02-05T21:12:26.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:26.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.395715+0000) 2020-02-05T21:12:26.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:26.395+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.396181+0000) 2020-02-05T21:12:26.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:26.397+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.397482+0000) 2020-02-05T21:12:26.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:26.397+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.397576+0000) 2020-02-05T21:12:26.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:26.397+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.398447+0000) 2020-02-05T21:12:26.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:26.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.399232+0000) 2020-02-05T21:12:26.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:26.399+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.399780+0000) 2020-02-05T21:12:26.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:26.399+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.399611+0000) 2020-02-05T21:12:26.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:26.400+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.400523+0000) 2020-02-05T21:12:26.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:26.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.402890+0000) 2020-02-05T21:12:26.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:26.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79086 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:26.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:26.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.722713+0000) 2020-02-05T21:12:26.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:26.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.967908+0000) 2020-02-05T21:12:26.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:26.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.968023+0000) 2020-02-05T21:12:26.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:26.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:26.979960+0000) 2020-02-05T21:12:29.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:29.361+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:29.361846+0000) 2020-02-05T21:12:29.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:29.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:29.970123+0000) 2020-02-05T21:12:30.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:30.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:30.221095+0000) 2020-02-05T21:12:31.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:31.407+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.409179+0000) 2020-02-05T21:12:31.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:31.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.409575+0000) 2020-02-05T21:12:31.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:31.408+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.409764+0000) 2020-02-05T21:12:31.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:31.408+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.410252+0000) 2020-02-05T21:12:31.412 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:31.409+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.410924+0000) 2020-02-05T21:12:31.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:31.409+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.410987+0000) 2020-02-05T21:12:31.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:31.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.411647+0000) 2020-02-05T21:12:31.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:31.410+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.411949+0000) 2020-02-05T21:12:31.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:31.410+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.411814+0000) 2020-02-05T21:12:31.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:31.411+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.412676+0000) 2020-02-05T21:12:31.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:31.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.412443+0000) 2020-02-05T21:12:31.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:31.411+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.412917+0000) 2020-02-05T21:12:31.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:31.412+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.413674+0000) 2020-02-05T21:12:31.419 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:31.416+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79124 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:31.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:31.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:31.419734+0000) 2020-02-05T21:12:32.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:32.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:32.210988+0000) 2020-02-05T21:12:32.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:32.324+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:32.325984+0000) 2020-02-05T21:12:32.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:32.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:32.612593+0000) 2020-02-05T21:12:33.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:33.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:33.452254+0000) 2020-02-05T21:12:33.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:33.480+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:33.481636+0000) 2020-02-05T21:12:35.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:35.824+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:35.825436+0000) 2020-02-05T21:12:36.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:36.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.308675+0000) 2020-02-05T21:12:36.425 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:36.422+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.423847+0000) 2020-02-05T21:12:36.425 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:36.423+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.424367+0000) 2020-02-05T21:12:36.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:36.424+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.425627+0000) 2020-02-05T21:12:36.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:36.424+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.426030+0000) 2020-02-05T21:12:36.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:36.424+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.426183+0000) 2020-02-05T21:12:36.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:36.425+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.426648+0000) 2020-02-05T21:12:36.428 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:36.426+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.427639+0000) 2020-02-05T21:12:36.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:36.425+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.427206+0000) 2020-02-05T21:12:36.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:36.425+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.427310+0000) 2020-02-05T21:12:36.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:36.426+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.427954+0000) 2020-02-05T21:12:36.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:36.426+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.428058+0000) 2020-02-05T21:12:36.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:36.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:36.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:36.432+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.433286+0000) 2020-02-05T21:12:36.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:36.433+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.434546+0000) 2020-02-05T21:12:36.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:36.433+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.434868+0000) 2020-02-05T21:12:36.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:36.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.722934+0000) 2020-02-05T21:12:36.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:36.966+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.968105+0000) 2020-02-05T21:12:36.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:36.966+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.968192+0000) 2020-02-05T21:12:36.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:36.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:36.980192+0000) 2020-02-05T21:12:37.214 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:12:37.220 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:12:39.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:39.361+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:39.362104+0000) 2020-02-05T21:12:39.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:39.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:39.970365+0000) 2020-02-05T21:12:40.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:40.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:40.221343+0000) 2020-02-05T21:12:41.439 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:41.436+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.437736+0000) 2020-02-05T21:12:41.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:41.438+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.439244+0000) 2020-02-05T21:12:41.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:41.439+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.440670+0000) 2020-02-05T21:12:41.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:41.439+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.440948+0000) 2020-02-05T21:12:41.443 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:41.440+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.441988+0000) 2020-02-05T21:12:41.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:41.441+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.442759+0000) 2020-02-05T21:12:41.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:41.442+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.443280+0000) 2020-02-05T21:12:41.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:41.441+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.443359+0000) 2020-02-05T21:12:41.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:41.441+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.443206+0000) 2020-02-05T21:12:41.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:41.442+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.443926+0000) 2020-02-05T21:12:41.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:41.442+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.444345+0000) 2020-02-05T21:12:41.449 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:41.446+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:41.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:41.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.448169+0000) 2020-02-05T21:12:41.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:41.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.448455+0000) 2020-02-05T21:12:41.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:41.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:41.449380+0000) 2020-02-05T21:12:42.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:42.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:42.211204+0000) 2020-02-05T21:12:42.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:42.325+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:42.326220+0000) 2020-02-05T21:12:42.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:42.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:42.612853+0000) 2020-02-05T21:12:43.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:43.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:43.452678+0000) 2020-02-05T21:12:43.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:43.481+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:43.481877+0000) 2020-02-05T21:12:45.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:45.824+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:45.825674+0000) 2020-02-05T21:12:46.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:46.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.316785+0000) 2020-02-05T21:12:46.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:46.452+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.453923+0000) 2020-02-05T21:12:46.456 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:46.454+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.455257+0000) 2020-02-05T21:12:46.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:46.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.456362+0000) 2020-02-05T21:12:46.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:46.455+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.456806+0000) 2020-02-05T21:12:46.458 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:46.456+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.457312+0000) 2020-02-05T21:12:46.459 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:46.456+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.457650+0000) 2020-02-05T21:12:46.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:46.457+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.458939+0000) 2020-02-05T21:12:46.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:46.457+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.459047+0000) 2020-02-05T21:12:46.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:46.457+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.459388+0000) 2020-02-05T21:12:46.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:46.458+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.459814+0000) 2020-02-05T21:12:46.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:46.459+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.460489+0000) 2020-02-05T21:12:46.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:46.458+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.459756+0000) 2020-02-05T21:12:46.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:46.458+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.460005+0000) 2020-02-05T21:12:46.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:46.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79246 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:46.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:46.464+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.465816+0000) 2020-02-05T21:12:46.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:46.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.723140+0000) 2020-02-05T21:12:46.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:46.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.968356+0000) 2020-02-05T21:12:46.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:46.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.968373+0000) 2020-02-05T21:12:46.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:46.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:46.980374+0000) 2020-02-05T21:12:49.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:49.361+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:49.362363+0000) 2020-02-05T21:12:49.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:49.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:49.970558+0000) 2020-02-05T21:12:50.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:50.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:50.221623+0000) 2020-02-05T21:12:51.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:51.469+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.470213+0000) 2020-02-05T21:12:51.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:51.469+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.470608+0000) 2020-02-05T21:12:51.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:51.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.471366+0000) 2020-02-05T21:12:51.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:51.470+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.471964+0000) 2020-02-05T21:12:51.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:51.470+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.471777+0000) 2020-02-05T21:12:51.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:51.472+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.473219+0000) 2020-02-05T21:12:51.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:51.471+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.472807+0000) 2020-02-05T21:12:51.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:51.473+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.474247+0000) 2020-02-05T21:12:51.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:51.473+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.474614+0000) 2020-02-05T21:12:51.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:51.473+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.474915+0000) 2020-02-05T21:12:51.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:51.474+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.475570+0000) 2020-02-05T21:12:51.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:51.474+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.476193+0000) 2020-02-05T21:12:51.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:51.474+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.476373+0000) 2020-02-05T21:12:51.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:51.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:51.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:51.479+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:51.480502+0000) 2020-02-05T21:12:52.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:52.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:52.211470+0000) 2020-02-05T21:12:52.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:52.325+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:52.326488+0000) 2020-02-05T21:12:52.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:52.612+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:52.613135+0000) 2020-02-05T21:12:53.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:53.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:53.452919+0000) 2020-02-05T21:12:53.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:53.481+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:53.482151+0000) 2020-02-05T21:12:55.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:55.824+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:55.825919+0000) 2020-02-05T21:12:56.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:56.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.325260+0000) 2020-02-05T21:12:56.486 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:12:56.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.484758+0000) 2020-02-05T21:12:56.487 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:56.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.485947+0000) 2020-02-05T21:12:56.488 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:56.485+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.487001+0000) 2020-02-05T21:12:56.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:12:56.486+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.487553+0000) 2020-02-05T21:12:56.489 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:56.486+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.487961+0000) 2020-02-05T21:12:56.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:56.487+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.488442+0000) 2020-02-05T21:12:56.490 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:12:56.488+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.489284+0000) 2020-02-05T21:12:56.492 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:56.489+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.490159+0000) 2020-02-05T21:12:56.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:12:56.488+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.489622+0000) 2020-02-05T21:12:56.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:56.489+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.490654+0000) 2020-02-05T21:12:56.493 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:12:56.489+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.491233+0000) 2020-02-05T21:12:56.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:56.490+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.491674+0000) 2020-02-05T21:12:56.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:12:56.490+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.492004+0000) 2020-02-05T21:12:56.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:12:56.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79326 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:12:56.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:56.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.496059+0000) 2020-02-05T21:12:56.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:12:56.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.723360+0000) 2020-02-05T21:12:56.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:12:56.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.968607+0000) 2020-02-05T21:12:56.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:12:56.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.968607+0000) 2020-02-05T21:12:56.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:12:56.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:56.980599+0000) 2020-02-05T21:12:59.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:12:59.362+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:59.362599+0000) 2020-02-05T21:12:59.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:12:59.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:12:59.970788+0000) 2020-02-05T21:13:00.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:00.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:00.221873+0000) 2020-02-05T21:13:01.502 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:01.499+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.500902+0000) 2020-02-05T21:13:01.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:01.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.502019+0000) 2020-02-05T21:13:01.503 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:01.501+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.502523+0000) 2020-02-05T21:13:01.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:01.502+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.503396+0000) 2020-02-05T21:13:01.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:01.502+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.504315+0000) 2020-02-05T21:13:01.506 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:01.503+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.504666+0000) 2020-02-05T21:13:01.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:01.504+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.505553+0000) 2020-02-05T21:13:01.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:01.504+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.505900+0000) 2020-02-05T21:13:01.508 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:01.504+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.506087+0000) 2020-02-05T21:13:01.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:01.505+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.507153+0000) 2020-02-05T21:13:01.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:01.506+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.507752+0000) 2020-02-05T21:13:01.509 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:01.506+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.507566+0000) 2020-02-05T21:13:01.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:01.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.508539+0000) 2020-02-05T21:13:01.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:01.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:01.511242+0000) 2020-02-05T21:13:01.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:01.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:02.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:02.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:02.211722+0000) 2020-02-05T21:13:02.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:02.326+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:02.326736+0000) 2020-02-05T21:13:02.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:02.612+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:02.613390+0000) 2020-02-05T21:13:03.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:03.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:03.453157+0000) 2020-02-05T21:13:03.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:03.481+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:03.482382+0000) 2020-02-05T21:13:05.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:05.825+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:05.826183+0000) 2020-02-05T21:13:06.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:06.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.333738+0000) 2020-02-05T21:13:06.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:06.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.518207+0000) 2020-02-05T21:13:06.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:06.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.518770+0000) 2020-02-05T21:13:06.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:06.517+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.519257+0000) 2020-02-05T21:13:06.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:06.518+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.519969+0000) 2020-02-05T21:13:06.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:06.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.521177+0000) 2020-02-05T21:13:06.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:06.520+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.521621+0000) 2020-02-05T21:13:06.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:06.520+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.521997+0000) 2020-02-05T21:13:06.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:06.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.522712+0000) 2020-02-05T21:13:06.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:06.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.523085+0000) 2020-02-05T21:13:06.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:06.522+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.523587+0000) 2020-02-05T21:13:06.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:06.522+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.523571+0000) 2020-02-05T21:13:06.525 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:06.523+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.524145+0000) 2020-02-05T21:13:06.526 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:06.523+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.524473+0000) 2020-02-05T21:13:06.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:06.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.527197+0000) 2020-02-05T21:13:06.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:06.526+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:06.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:06.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.723566+0000) 2020-02-05T21:13:06.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:06.967+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.968810+0000) 2020-02-05T21:13:06.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:06.967+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.968812+0000) 2020-02-05T21:13:06.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:06.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:06.980806+0000) 2020-02-05T21:13:07.343 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:13:07.350 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:13:09.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:09.362+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:09.362840+0000) 2020-02-05T21:13:09.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:09.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:09.971042+0000) 2020-02-05T21:13:10.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:10.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:10.222125+0000) 2020-02-05T21:13:11.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:11.532+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.534036+0000) 2020-02-05T21:13:11.536 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:11.534+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.535295+0000) 2020-02-05T21:13:11.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:11.534+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.535730+0000) 2020-02-05T21:13:11.537 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:11.535+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.536650+0000) 2020-02-05T21:13:11.538 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:11.535+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.537176+0000) 2020-02-05T21:13:11.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:11.535+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.536997+0000) 2020-02-05T21:13:11.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:11.537+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.538248+0000) 2020-02-05T21:13:11.539 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:11.536+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.538384+0000) 2020-02-05T21:13:11.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:11.537+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.538907+0000) 2020-02-05T21:13:11.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:11.537+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.538853+0000) 2020-02-05T21:13:11.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:11.538+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.539451+0000) 2020-02-05T21:13:11.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:11.539+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.540230+0000) 2020-02-05T21:13:11.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:11.539+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.540597+0000) 2020-02-05T21:13:11.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:11.541+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79444 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:11.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:11.545+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:11.546525+0000) 2020-02-05T21:13:12.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:12.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:12.211958+0000) 2020-02-05T21:13:12.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:12.326+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:12.326959+0000) 2020-02-05T21:13:12.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:12.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:12.613625+0000) 2020-02-05T21:13:13.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:13.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:13.453392+0000) 2020-02-05T21:13:13.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:13.481+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:13.482674+0000) 2020-02-05T21:13:15.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:15.826+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:15.826440+0000) 2020-02-05T21:13:16.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:16.340+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.341146+0000) 2020-02-05T21:13:16.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:16.547+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.549038+0000) 2020-02-05T21:13:16.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:16.548+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.549627+0000) 2020-02-05T21:13:16.551 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:16.549+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.550374+0000) 2020-02-05T21:13:16.551 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:16.548+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.549874+0000) 2020-02-05T21:13:16.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:16.551+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.552480+0000) 2020-02-05T21:13:16.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:16.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.553057+0000) 2020-02-05T21:13:16.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:16.551+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.552906+0000) 2020-02-05T21:13:16.555 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:16.552+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.553592+0000) 2020-02-05T21:13:16.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:16.552+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.554004+0000) 2020-02-05T21:13:16.556 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:16.552+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.553955+0000) 2020-02-05T21:13:16.556 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:16.553+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.554821+0000) 2020-02-05T21:13:16.556 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:16.553+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.554476+0000) 2020-02-05T21:13:16.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:16.553+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.554671+0000) 2020-02-05T21:13:16.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:16.557+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.558585+0000) 2020-02-05T21:13:16.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:16.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79486 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:16.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:16.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.723774+0000) 2020-02-05T21:13:16.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:16.968+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.969044+0000) 2020-02-05T21:13:16.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:16.968+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.969044+0000) 2020-02-05T21:13:16.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:16.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:16.981028+0000) 2020-02-05T21:13:19.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:19.362+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:19.363073+0000) 2020-02-05T21:13:19.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:19.971+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:19.971332+0000) 2020-02-05T21:13:20.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:20.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:20.222372+0000) 2020-02-05T21:13:21.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:21.565+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.565246+0000) 2020-02-05T21:13:21.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:21.565+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.565742+0000) 2020-02-05T21:13:21.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:21.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.566127+0000) 2020-02-05T21:13:21.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:21.566+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.566543+0000) 2020-02-05T21:13:21.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:21.567+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.567506+0000) 2020-02-05T21:13:21.569 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:21.567+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.567631+0000) 2020-02-05T21:13:21.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:21.567+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.568215+0000) 2020-02-05T21:13:21.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:21.568+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.568926+0000) 2020-02-05T21:13:21.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:21.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.569269+0000) 2020-02-05T21:13:21.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:21.569+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.569270+0000) 2020-02-05T21:13:21.571 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:21.569+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.569343+0000) 2020-02-05T21:13:21.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:21.569+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.569768+0000) 2020-02-05T21:13:21.572 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:21.568+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.569785+0000) 2020-02-05T21:13:21.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:21.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:21.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:21.573+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:21.574109+0000) 2020-02-05T21:13:22.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:22.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:22.212181+0000) 2020-02-05T21:13:22.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:22.326+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:22.327207+0000) 2020-02-05T21:13:22.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:22.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:22.613882+0000) 2020-02-05T21:13:23.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:23.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:23.453626+0000) 2020-02-05T21:13:23.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:23.482+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:23.482921+0000) 2020-02-05T21:13:25.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:25.825+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:25.826668+0000) 2020-02-05T21:13:26.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:26.347+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.348876+0000) 2020-02-05T21:13:26.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:26.580+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.581404+0000) 2020-02-05T21:13:26.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:26.580+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.581889+0000) 2020-02-05T21:13:26.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:26.581+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.582398+0000) 2020-02-05T21:13:26.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:26.582+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.582643+0000) 2020-02-05T21:13:26.585 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:26.582+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.583946+0000) 2020-02-05T21:13:26.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:26.582+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.583386+0000) 2020-02-05T21:13:26.585 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:26.583+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.583486+0000) 2020-02-05T21:13:26.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:26.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.584509+0000) 2020-02-05T21:13:26.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:26.584+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.585213+0000) 2020-02-05T21:13:26.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:26.585+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.585455+0000) 2020-02-05T21:13:26.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:26.585+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.585484+0000) 2020-02-05T21:13:26.587 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:26.584+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.585316+0000) 2020-02-05T21:13:26.587 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:26.584+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.585872+0000) 2020-02-05T21:13:26.591 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:26.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:26.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:26.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.590511+0000) 2020-02-05T21:13:26.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:26.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.724003+0000) 2020-02-05T21:13:26.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:26.968+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.969324+0000) 2020-02-05T21:13:26.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:26.968+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.969325+0000) 2020-02-05T21:13:26.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:26.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:26.981347+0000) 2020-02-05T21:13:29.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:29.362+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:29.363341+0000) 2020-02-05T21:13:29.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:29.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:29.971560+0000) 2020-02-05T21:13:30.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:30.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:30.222735+0000) 2020-02-05T21:13:31.597 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:31.594+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.595822+0000) 2020-02-05T21:13:31.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:31.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.596342+0000) 2020-02-05T21:13:31.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:31.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.596741+0000) 2020-02-05T21:13:31.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:31.596+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.597386+0000) 2020-02-05T21:13:31.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:31.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.598234+0000) 2020-02-05T21:13:31.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:31.598+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.598644+0000) 2020-02-05T21:13:31.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:31.599+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.599787+0000) 2020-02-05T21:13:31.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:31.600+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.600648+0000) 2020-02-05T21:13:31.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:31.600+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.600882+0000) 2020-02-05T21:13:31.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:31.600+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.601265+0000) 2020-02-05T21:13:31.603 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:31.601+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.601508+0000) 2020-02-05T21:13:31.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:31.601+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.602221+0000) 2020-02-05T21:13:31.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:31.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.604592+0000) 2020-02-05T21:13:31.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:31.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:31.604815+0000) 2020-02-05T21:13:31.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:31.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79604 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:32.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:32.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:32.212412+0000) 2020-02-05T21:13:32.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:32.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:32.327478+0000) 2020-02-05T21:13:32.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:32.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:32.614128+0000) 2020-02-05T21:13:33.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:33.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:33.453895+0000) 2020-02-05T21:13:33.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:33.482+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:33.483154+0000) 2020-02-05T21:13:35.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:35.826+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:35.826909+0000) 2020-02-05T21:13:36.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:36.356+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.357337+0000) 2020-02-05T21:13:36.612 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:36.609+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.610588+0000) 2020-02-05T21:13:36.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:36.611+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.612228+0000) 2020-02-05T21:13:36.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:36.611+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.612449+0000) 2020-02-05T21:13:36.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:36.611+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.612878+0000) 2020-02-05T21:13:36.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:36.613+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.613779+0000) 2020-02-05T21:13:36.615 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:36.613+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.614630+0000) 2020-02-05T21:13:36.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:36.614+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.615333+0000) 2020-02-05T21:13:36.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:36.614+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.615489+0000) 2020-02-05T21:13:36.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:36.615+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.615771+0000) 2020-02-05T21:13:36.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:36.615+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.616151+0000) 2020-02-05T21:13:36.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:36.616+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.616609+0000) 2020-02-05T21:13:36.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:36.616+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.616895+0000) 2020-02-05T21:13:36.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:36.617+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.617764+0000) 2020-02-05T21:13:36.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:36.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.619430+0000) 2020-02-05T21:13:36.623 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:36.621+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:36.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:36.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.724214+0000) 2020-02-05T21:13:36.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:36.969+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.969540+0000) 2020-02-05T21:13:36.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:36.969+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.969537+0000) 2020-02-05T21:13:36.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:36.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:36.981553+0000) 2020-02-05T21:13:37.473 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:13:37.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:13:39.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:39.362+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:39.363579+0000) 2020-02-05T21:13:39.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:39.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:39.971795+0000) 2020-02-05T21:13:40.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:40.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:40.222984+0000) 2020-02-05T21:13:41.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:41.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.627371+0000) 2020-02-05T21:13:41.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:41.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.628989+0000) 2020-02-05T21:13:41.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:41.628+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.629586+0000) 2020-02-05T21:13:41.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:41.629+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.630507+0000) 2020-02-05T21:13:41.632 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:41.630+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.630912+0000) 2020-02-05T21:13:41.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:41.630+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.631349+0000) 2020-02-05T21:13:41.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:41.630+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.631739+0000) 2020-02-05T21:13:41.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:41.631+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.632238+0000) 2020-02-05T21:13:41.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:41.631+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.632343+0000) 2020-02-05T21:13:41.634 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:41.631+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.632055+0000) 2020-02-05T21:13:41.634 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:41.632+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.632445+0000) 2020-02-05T21:13:41.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:41.632+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.633281+0000) 2020-02-05T21:13:41.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:41.633+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.633899+0000) 2020-02-05T21:13:41.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:41.635+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:41.636689+0000) 2020-02-05T21:13:41.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:41.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79684 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:42.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:42.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:42.212599+0000) 2020-02-05T21:13:42.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:42.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:42.327714+0000) 2020-02-05T21:13:42.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:42.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:42.614385+0000) 2020-02-05T21:13:43.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:43.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:43.454136+0000) 2020-02-05T21:13:43.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:43.482+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:43.483387+0000) 2020-02-05T21:13:45.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:45.826+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:45.827138+0000) 2020-02-05T21:13:46.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:46.365+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.366418+0000) 2020-02-05T21:13:46.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:46.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.642440+0000) 2020-02-05T21:13:46.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:46.643+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.643909+0000) 2020-02-05T21:13:46.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:46.644+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.645060+0000) 2020-02-05T21:13:46.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:46.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.645515+0000) 2020-02-05T21:13:46.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:46.645+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.646433+0000) 2020-02-05T21:13:46.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:46.645+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.646367+0000) 2020-02-05T21:13:46.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:46.645+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.646704+0000) 2020-02-05T21:13:46.648 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:46.646+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.647250+0000) 2020-02-05T21:13:46.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:46.646+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.647836+0000) 2020-02-05T21:13:46.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:46.647+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.648464+0000) 2020-02-05T21:13:46.650 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:46.648+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.648523+0000) 2020-02-05T21:13:46.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:46.648+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.649067+0000) 2020-02-05T21:13:46.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:46.648+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.649389+0000) 2020-02-05T21:13:46.654 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:46.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:46.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:46.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.654497+0000) 2020-02-05T21:13:46.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:46.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.724426+0000) 2020-02-05T21:13:46.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:46.969+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.969809+0000) 2020-02-05T21:13:46.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:46.969+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.969809+0000) 2020-02-05T21:13:46.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:46.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:46.981800+0000) 2020-02-05T21:13:49.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:49.363+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:49.363825+0000) 2020-02-05T21:13:49.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:49.971+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:49.972031+0000) 2020-02-05T21:13:50.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:50.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:50.223231+0000) 2020-02-05T21:13:51.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:51.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.658903+0000) 2020-02-05T21:13:51.661 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:51.658+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.659951+0000) 2020-02-05T21:13:51.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:51.660+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.661113+0000) 2020-02-05T21:13:51.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:51.660+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.661559+0000) 2020-02-05T21:13:51.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:51.660+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.661969+0000) 2020-02-05T21:13:51.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:51.661+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.662710+0000) 2020-02-05T21:13:51.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:51.662+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.662612+0000) 2020-02-05T21:13:51.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:51.662+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.662984+0000) 2020-02-05T21:13:51.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:51.663+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.663839+0000) 2020-02-05T21:13:51.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:51.664+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.665082+0000) 2020-02-05T21:13:51.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:51.664+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.665530+0000) 2020-02-05T21:13:51.667 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:51.665+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.665470+0000) 2020-02-05T21:13:51.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:51.665+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.665844+0000) 2020-02-05T21:13:51.671 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:51.669+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:51.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:51.670+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:51.671306+0000) 2020-02-05T21:13:52.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:52.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:52.212834+0000) 2020-02-05T21:13:52.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:52.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:52.327946+0000) 2020-02-05T21:13:52.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:52.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:52.614596+0000) 2020-02-05T21:13:53.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:53.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:53.454377+0000) 2020-02-05T21:13:53.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:53.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:53.483604+0000) 2020-02-05T21:13:55.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:55.826+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:55.827359+0000) 2020-02-05T21:13:56.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:56.374+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.374748+0000) 2020-02-05T21:13:56.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:56.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.675453+0000) 2020-02-05T21:13:56.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:13:56.675+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.676005+0000) 2020-02-05T21:13:56.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:56.676+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.676544+0000) 2020-02-05T21:13:56.678 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:13:56.676+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.677777+0000) 2020-02-05T21:13:56.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:56.677+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.678140+0000) 2020-02-05T21:13:56.680 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:13:56.677+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.678441+0000) 2020-02-05T21:13:56.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:56.677+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.678778+0000) 2020-02-05T21:13:56.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:56.677+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.679009+0000) 2020-02-05T21:13:56.681 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:56.678+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.679316+0000) 2020-02-05T21:13:56.682 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:13:56.678+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.678686+0000) 2020-02-05T21:13:56.682 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:13:56.679+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.679568+0000) 2020-02-05T21:13:56.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:13:56.679+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.680181+0000) 2020-02-05T21:13:56.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:56.679+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.680146+0000) 2020-02-05T21:13:56.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:56.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.683597+0000) 2020-02-05T21:13:56.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:13:56.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:13:56.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:13:56.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.724705+0000) 2020-02-05T21:13:56.972 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:13:56.969+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.970079+0000) 2020-02-05T21:13:56.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:13:56.969+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.970079+0000) 2020-02-05T21:13:56.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:13:56.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:56.982128+0000) 2020-02-05T21:13:59.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:13:59.363+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:59.364070+0000) 2020-02-05T21:13:59.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:13:59.971+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:13:59.972274+0000) 2020-02-05T21:14:00.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:00.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:00.223500+0000) 2020-02-05T21:14:01.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:01.690+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.691564+0000) 2020-02-05T21:14:01.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:01.690+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.692074+0000) 2020-02-05T21:14:01.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:01.691+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.693052+0000) 2020-02-05T21:14:01.695 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:01.691+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.693269+0000) 2020-02-05T21:14:01.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:01.693+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.694600+0000) 2020-02-05T21:14:01.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:01.694+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695426+0000) 2020-02-05T21:14:01.697 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:01.693+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.694612+0000) 2020-02-05T21:14:01.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:01.694+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695198+0000) 2020-02-05T21:14:01.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:01.694+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695463+0000) 2020-02-05T21:14:01.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:01.694+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695554+0000) 2020-02-05T21:14:01.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:01.694+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695850+0000) 2020-02-05T21:14:01.699 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:01.694+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.695955+0000) 2020-02-05T21:14:01.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:01.695+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.696883+0000) 2020-02-05T21:14:01.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:01.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:01.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:01.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:01.702500+0000) 2020-02-05T21:14:02.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:02.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:02.213051+0000) 2020-02-05T21:14:02.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:02.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:02.328169+0000) 2020-02-05T21:14:02.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:02.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:02.614814+0000) 2020-02-05T21:14:03.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:03.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:03.454668+0000) 2020-02-05T21:14:03.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:03.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:03.483834+0000) 2020-02-05T21:14:05.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:05.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:05.827601+0000) 2020-02-05T21:14:06.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:06.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.383135+0000) 2020-02-05T21:14:06.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:06.705+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.707064+0000) 2020-02-05T21:14:06.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:06.705+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.707107+0000) 2020-02-05T21:14:06.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:06.707+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.709188+0000) 2020-02-05T21:14:06.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:06.709+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.710167+0000) 2020-02-05T21:14:06.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:06.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.710312+0000) 2020-02-05T21:14:06.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:06.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.710913+0000) 2020-02-05T21:14:06.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:06.709+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.710682+0000) 2020-02-05T21:14:06.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:06.709+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.710861+0000) 2020-02-05T21:14:06.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:06.709+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.711234+0000) 2020-02-05T21:14:06.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:06.710+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.711801+0000) 2020-02-05T21:14:06.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:06.711+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.712532+0000) 2020-02-05T21:14:06.714 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:06.710+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.712231+0000) 2020-02-05T21:14:06.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:06.711+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.712805+0000) 2020-02-05T21:14:06.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:06.715+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:06.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:06.716+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.717197+0000) 2020-02-05T21:14:06.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:06.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.724879+0000) 2020-02-05T21:14:06.972 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:06.968+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.970304+0000) 2020-02-05T21:14:06.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:06.968+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.970308+0000) 2020-02-05T21:14:06.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:06.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:06.982347+0000) 2020-02-05T21:14:07.615 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:14:07.621 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:14:09.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:09.364+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:09.364386+0000) 2020-02-05T21:14:09.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:09.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:09.972531+0000) 2020-02-05T21:14:10.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:10.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:10.223755+0000) 2020-02-05T21:14:11.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:11.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.721801+0000) 2020-02-05T21:14:11.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:11.722+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.723039+0000) 2020-02-05T21:14:11.725 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:11.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.724531+0000) 2020-02-05T21:14:11.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:11.722+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.724314+0000) 2020-02-05T21:14:11.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:11.723+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.724989+0000) 2020-02-05T21:14:11.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:11.724+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.725517+0000) 2020-02-05T21:14:11.728 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:11.725+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.726848+0000) 2020-02-05T21:14:11.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:11.726+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.727689+0000) 2020-02-05T21:14:11.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:11.727+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.728125+0000) 2020-02-05T21:14:11.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:11.727+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.728551+0000) 2020-02-05T21:14:11.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:11.727+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.728748+0000) 2020-02-05T21:14:11.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:11.727+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.728987+0000) 2020-02-05T21:14:11.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:11.727+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.729187+0000) 2020-02-05T21:14:11.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:11.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:11.732924+0000) 2020-02-05T21:14:11.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:11.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:12.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:12.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:12.213297+0000) 2020-02-05T21:14:12.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:12.326+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:12.328392+0000) 2020-02-05T21:14:12.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:12.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:12.615050+0000) 2020-02-05T21:14:13.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:13.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:13.454907+0000) 2020-02-05T21:14:13.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:13.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:13.484116+0000) 2020-02-05T21:14:15.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:15.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:15.827838+0000) 2020-02-05T21:14:16.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:16.391+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.391604+0000) 2020-02-05T21:14:16.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:16.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.725108+0000) 2020-02-05T21:14:16.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:16.737+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.738726+0000) 2020-02-05T21:14:16.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:16.739+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.740555+0000) 2020-02-05T21:14:16.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:16.741+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.741116+0000) 2020-02-05T21:14:16.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:16.741+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.741763+0000) 2020-02-05T21:14:16.743 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:16.742+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.742319+0000) 2020-02-05T21:14:16.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:16.742+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.742627+0000) 2020-02-05T21:14:16.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:16.743+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.743497+0000) 2020-02-05T21:14:16.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:16.743+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.743903+0000) 2020-02-05T21:14:16.745 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:16.744+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.744253+0000) 2020-02-05T21:14:16.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:16.742+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.744350+0000) 2020-02-05T21:14:16.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:16.743+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.744680+0000) 2020-02-05T21:14:16.746 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:16.743+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.744765+0000) 2020-02-05T21:14:16.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:16.743+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.745233+0000) 2020-02-05T21:14:16.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:16.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 79962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:16.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:16.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.750480+0000) 2020-02-05T21:14:16.972 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:16.969+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.970569+0000) 2020-02-05T21:14:16.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:16.969+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.970572+0000) 2020-02-05T21:14:16.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:16.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:16.982556+0000) 2020-02-05T21:14:19.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:19.364+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:19.364642+0000) 2020-02-05T21:14:19.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:19.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:19.972799+0000) 2020-02-05T21:14:20.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:20.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:20.223993+0000) 2020-02-05T21:14:21.754 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:21.752+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.753362+0000) 2020-02-05T21:14:21.756 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:21.754+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.755312+0000) 2020-02-05T21:14:21.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:21.753+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.754769+0000) 2020-02-05T21:14:21.757 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:21.754+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.755580+0000) 2020-02-05T21:14:21.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:21.754+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.755972+0000) 2020-02-05T21:14:21.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:21.755+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.756686+0000) 2020-02-05T21:14:21.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:21.755+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.757056+0000) 2020-02-05T21:14:21.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:21.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.756622+0000) 2020-02-05T21:14:21.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:21.756+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.757530+0000) 2020-02-05T21:14:21.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:21.756+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.758010+0000) 2020-02-05T21:14:21.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:21.757+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.758371+0000) 2020-02-05T21:14:21.760 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:21.756+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.757415+0000) 2020-02-05T21:14:21.760 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:21.757+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.758831+0000) 2020-02-05T21:14:21.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:21.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:21.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:21.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:21.764176+0000) 2020-02-05T21:14:22.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:22.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:22.213543+0000) 2020-02-05T21:14:22.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:22.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:22.328652+0000) 2020-02-05T21:14:22.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:22.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:22.615301+0000) 2020-02-05T21:14:23.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:23.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:23.455171+0000) 2020-02-05T21:14:23.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:23.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:23.484374+0000) 2020-02-05T21:14:25.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:25.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:25.828157+0000) 2020-02-05T21:14:26.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:26.399+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.400058+0000) 2020-02-05T21:14:26.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:26.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.725370+0000) 2020-02-05T21:14:26.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:26.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.769558+0000) 2020-02-05T21:14:26.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:26.768+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.770139+0000) 2020-02-05T21:14:26.772 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:26.769+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.770719+0000) 2020-02-05T21:14:26.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:26.770+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.771502+0000) 2020-02-05T21:14:26.773 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:26.770+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.771701+0000) 2020-02-05T21:14:26.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:26.771+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.772418+0000) 2020-02-05T21:14:26.774 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:26.771+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.772699+0000) 2020-02-05T21:14:26.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:26.773+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.774186+0000) 2020-02-05T21:14:26.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:26.773+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.774568+0000) 2020-02-05T21:14:26.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:26.773+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.774846+0000) 2020-02-05T21:14:26.776 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:26.774+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.775571+0000) 2020-02-05T21:14:26.777 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:26.774+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.775542+0000) 2020-02-05T21:14:26.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:26.774+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.775785+0000) 2020-02-05T21:14:26.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:26.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.780166+0000) 2020-02-05T21:14:26.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:26.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:26.972 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:26.970+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.970835+0000) 2020-02-05T21:14:26.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:26.970+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.970837+0000) 2020-02-05T21:14:26.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:26.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:26.982763+0000) 2020-02-05T21:14:29.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:29.363+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:29.364879+0000) 2020-02-05T21:14:29.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:29.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:29.973042+0000) 2020-02-05T21:14:30.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:30.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:30.224222+0000) 2020-02-05T21:14:31.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:31.785+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.786859+0000) 2020-02-05T21:14:31.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:31.786+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.787213+0000) 2020-02-05T21:14:31.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:31.786+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.787582+0000) 2020-02-05T21:14:31.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:31.786+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.787848+0000) 2020-02-05T21:14:31.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:31.786+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.788185+0000) 2020-02-05T21:14:31.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:31.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.788660+0000) 2020-02-05T21:14:31.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:31.788+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.789160+0000) 2020-02-05T21:14:31.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:31.788+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.789727+0000) 2020-02-05T21:14:31.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:31.788+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.789916+0000) 2020-02-05T21:14:31.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:31.788+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.790261+0000) 2020-02-05T21:14:31.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:31.789+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.791127+0000) 2020-02-05T21:14:31.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:31.790+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.791349+0000) 2020-02-05T21:14:31.793 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:31.790+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.792288+0000) 2020-02-05T21:14:31.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:31.794+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:31.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:31.795+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:31.796718+0000) 2020-02-05T21:14:32.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:32.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:32.213757+0000) 2020-02-05T21:14:32.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:32.327+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:32.328955+0000) 2020-02-05T21:14:32.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:32.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:32.615526+0000) 2020-02-05T21:14:33.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:33.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:33.455497+0000) 2020-02-05T21:14:33.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:33.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:33.484611+0000) 2020-02-05T21:14:35.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:35.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:35.828405+0000) 2020-02-05T21:14:36.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:36.407+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.407960+0000) 2020-02-05T21:14:36.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:36.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.725600+0000) 2020-02-05T21:14:36.804 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:36.801+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.803005+0000) 2020-02-05T21:14:36.805 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:36.801+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.803390+0000) 2020-02-05T21:14:36.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:36.802+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.803720+0000) 2020-02-05T21:14:36.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:36.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.804486+0000) 2020-02-05T21:14:36.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:36.803+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.805045+0000) 2020-02-05T21:14:36.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:36.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.805262+0000) 2020-02-05T21:14:36.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:36.805+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.806409+0000) 2020-02-05T21:14:36.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:36.804+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.806317+0000) 2020-02-05T21:14:36.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:36.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.806815+0000) 2020-02-05T21:14:36.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:36.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.807220+0000) 2020-02-05T21:14:36.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:36.806+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.807392+0000) 2020-02-05T21:14:36.809 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:36.806+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.807669+0000) 2020-02-05T21:14:36.809 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:36.806+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.808241+0000) 2020-02-05T21:14:36.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:36.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:36.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:36.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.815002+0000) 2020-02-05T21:14:36.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:36.970+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.971103+0000) 2020-02-05T21:14:36.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:36.970+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.971103+0000) 2020-02-05T21:14:36.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:36.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:36.982970+0000) 2020-02-05T21:14:37.755 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:14:37.761 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:14:39.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:39.364+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:39.365157+0000) 2020-02-05T21:14:39.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:39.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:39.973324+0000) 2020-02-05T21:14:40.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:40.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:40.224476+0000) 2020-02-05T21:14:41.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:41.816+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.817991+0000) 2020-02-05T21:14:41.820 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:41.816+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.818090+0000) 2020-02-05T21:14:41.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:41.817+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.818646+0000) 2020-02-05T21:14:41.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:41.817+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.818828+0000) 2020-02-05T21:14:41.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:41.818+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.819679+0000) 2020-02-05T21:14:41.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:41.819+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.820530+0000) 2020-02-05T21:14:41.822 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:41.819+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.820641+0000) 2020-02-05T21:14:41.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:41.819+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.821111+0000) 2020-02-05T21:14:41.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:41.820+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.821552+0000) 2020-02-05T21:14:41.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:41.821+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.822883+0000) 2020-02-05T21:14:41.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:41.821+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.823155+0000) 2020-02-05T21:14:41.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:41.822+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.823619+0000) 2020-02-05T21:14:41.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:41.823+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.824185+0000) 2020-02-05T21:14:41.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:41.826+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:41.827754+0000) 2020-02-05T21:14:41.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:41.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:42.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:42.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:42.213976+0000) 2020-02-05T21:14:42.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:42.328+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:42.329174+0000) 2020-02-05T21:14:42.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:42.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:42.615766+0000) 2020-02-05T21:14:43.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:43.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:43.455830+0000) 2020-02-05T21:14:43.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:43.483+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:43.484859+0000) 2020-02-05T21:14:45.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:45.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:45.828670+0000) 2020-02-05T21:14:46.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:46.414+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.415134+0000) 2020-02-05T21:14:46.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:46.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.725830+0000) 2020-02-05T21:14:46.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:46.833+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.834592+0000) 2020-02-05T21:14:46.837 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:46.834+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.835704+0000) 2020-02-05T21:14:46.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:46.835+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.836227+0000) 2020-02-05T21:14:46.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:46.835+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.836594+0000) 2020-02-05T21:14:46.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:46.836+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.837165+0000) 2020-02-05T21:14:46.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:46.836+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.837521+0000) 2020-02-05T21:14:46.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:46.836+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.837798+0000) 2020-02-05T21:14:46.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:46.837+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.838450+0000) 2020-02-05T21:14:46.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:46.837+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.838695+0000) 2020-02-05T21:14:46.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:46.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.839027+0000) 2020-02-05T21:14:46.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:46.837+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.839048+0000) 2020-02-05T21:14:46.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:46.838+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.840060+0000) 2020-02-05T21:14:46.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:46.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.840608+0000) 2020-02-05T21:14:46.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:46.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.844185+0000) 2020-02-05T21:14:46.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:46.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:46.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:46.970+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.971373+0000) 2020-02-05T21:14:46.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:46.970+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.971369+0000) 2020-02-05T21:14:46.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:46.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:46.983190+0000) 2020-02-05T21:14:49.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:49.364+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:49.365395+0000) 2020-02-05T21:14:49.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:49.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:49.973568+0000) 2020-02-05T21:14:50.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:50.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:50.224737+0000) 2020-02-05T21:14:51.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:51.849+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.850199+0000) 2020-02-05T21:14:51.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:51.849+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.850747+0000) 2020-02-05T21:14:51.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:51.850+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.851589+0000) 2020-02-05T21:14:51.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:51.850+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.851297+0000) 2020-02-05T21:14:51.853 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:51.851+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.852141+0000) 2020-02-05T21:14:51.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:51.851+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.852974+0000) 2020-02-05T21:14:51.854 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:51.852+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.852629+0000) 2020-02-05T21:14:51.856 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:51.854+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.854483+0000) 2020-02-05T21:14:51.856 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:51.854+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.854714+0000) 2020-02-05T21:14:51.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:51.853+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.854670+0000) 2020-02-05T21:14:51.857 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:51.853+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.854913+0000) 2020-02-05T21:14:51.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:51.854+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.854775+0000) 2020-02-05T21:14:51.857 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:51.855+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.855970+0000) 2020-02-05T21:14:51.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:51.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:51.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:51.860+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:51.861169+0000) 2020-02-05T21:14:52.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:52.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:52.214192+0000) 2020-02-05T21:14:52.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:52.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:52.329413+0000) 2020-02-05T21:14:52.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:52.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:52.615980+0000) 2020-02-05T21:14:53.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:53.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:53.456082+0000) 2020-02-05T21:14:53.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:53.484+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:53.485157+0000) 2020-02-05T21:14:55.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:55.827+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:55.828968+0000) 2020-02-05T21:14:56.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:56.423+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.423800+0000) 2020-02-05T21:14:56.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:56.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.726066+0000) 2020-02-05T21:14:56.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:14:56.864+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.865249+0000) 2020-02-05T21:14:56.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:14:56.864+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.865820+0000) 2020-02-05T21:14:56.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:56.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.866274+0000) 2020-02-05T21:14:56.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:14:56.866+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.866545+0000) 2020-02-05T21:14:56.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:14:56.865+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.866557+0000) 2020-02-05T21:14:56.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:56.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.867045+0000) 2020-02-05T21:14:56.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:14:56.866+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.867635+0000) 2020-02-05T21:14:56.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:14:56.868+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.868746+0000) 2020-02-05T21:14:56.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:56.868+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.868868+0000) 2020-02-05T21:14:56.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:56.868+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.869319+0000) 2020-02-05T21:14:56.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:56.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.868189+0000) 2020-02-05T21:14:56.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:56.867+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.868563+0000) 2020-02-05T21:14:56.873 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:14:56.869+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.870241+0000) 2020-02-05T21:14:56.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:56.874+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.875146+0000) 2020-02-05T21:14:56.876 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:14:56.874+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:14:56.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:14:56.971+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.971624+0000) 2020-02-05T21:14:56.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:14:56.971+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.971624+0000) 2020-02-05T21:14:56.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:14:56.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:56.983435+0000) 2020-02-05T21:14:59.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:14:59.365+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:59.365631+0000) 2020-02-05T21:14:59.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:14:59.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:14:59.973795+0000) 2020-02-05T21:15:00.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:00.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:00.225024+0000) 2020-02-05T21:15:01.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:01.879+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.880159+0000) 2020-02-05T21:15:01.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:01.879+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.881043+0000) 2020-02-05T21:15:01.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:01.881+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.882770+0000) 2020-02-05T21:15:01.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:01.882+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.883345+0000) 2020-02-05T21:15:01.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:01.883+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.883750+0000) 2020-02-05T21:15:01.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:01.883+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.884953+0000) 2020-02-05T21:15:01.886 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:01.884+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.885097+0000) 2020-02-05T21:15:01.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:01.884+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.885431+0000) 2020-02-05T21:15:01.887 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:01.885+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.885810+0000) 2020-02-05T21:15:01.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:01.885+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.886650+0000) 2020-02-05T21:15:01.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:01.886+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.886517+0000) 2020-02-05T21:15:01.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:01.886+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.886845+0000) 2020-02-05T21:15:01.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:01.886+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.887165+0000) 2020-02-05T21:15:01.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:01.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:01.890755+0000) 2020-02-05T21:15:01.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:01.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:02.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:02.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:02.214487+0000) 2020-02-05T21:15:02.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:02.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:02.329729+0000) 2020-02-05T21:15:02.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:02.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:02.616208+0000) 2020-02-05T21:15:03.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:03.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:03.456316+0000) 2020-02-05T21:15:03.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:03.485+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:03.485434+0000) 2020-02-05T21:15:05.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:05.828+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:05.829205+0000) 2020-02-05T21:15:06.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:06.431+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.431769+0000) 2020-02-05T21:15:06.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:06.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.726347+0000) 2020-02-05T21:15:06.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:06.897+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.898273+0000) 2020-02-05T21:15:06.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:06.898+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.899137+0000) 2020-02-05T21:15:06.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:06.898+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.899676+0000) 2020-02-05T21:15:06.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:06.899+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.900314+0000) 2020-02-05T21:15:06.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:06.899+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.900833+0000) 2020-02-05T21:15:06.902 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:06.900+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.900909+0000) 2020-02-05T21:15:06.902 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:06.900+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.901401+0000) 2020-02-05T21:15:06.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:06.901+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.902367+0000) 2020-02-05T21:15:06.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:06.901+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.902600+0000) 2020-02-05T21:15:06.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:06.901+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.902897+0000) 2020-02-05T21:15:06.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:06.902+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.902908+0000) 2020-02-05T21:15:06.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:06.902+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.903136+0000) 2020-02-05T21:15:06.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:06.902+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.903471+0000) 2020-02-05T21:15:06.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:06.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.906751+0000) 2020-02-05T21:15:06.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:06.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:06.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:06.971+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.971867+0000) 2020-02-05T21:15:06.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:06.971+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.971868+0000) 2020-02-05T21:15:06.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:06.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:06.983638+0000) 2020-02-05T21:15:07.928 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:15:07.934 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:15:09.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:09.365+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:09.365865+0000) 2020-02-05T21:15:09.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:09.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:09.974022+0000) 2020-02-05T21:15:10.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:10.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:10.225277+0000) 2020-02-05T21:15:11.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:11.913+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.913868+0000) 2020-02-05T21:15:11.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:11.914+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.914500+0000) 2020-02-05T21:15:11.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:11.914+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.914861+0000) 2020-02-05T21:15:11.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:11.916+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.916403+0000) 2020-02-05T21:15:11.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:11.916+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.916652+0000) 2020-02-05T21:15:11.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:11.918+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.918985+0000) 2020-02-05T21:15:11.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:11.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.919950+0000) 2020-02-05T21:15:11.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:11.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.919556+0000) 2020-02-05T21:15:11.923 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:11.916+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.917030+0000) 2020-02-05T21:15:11.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:11.917+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.918004+0000) 2020-02-05T21:15:11.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:11.917+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.918232+0000) 2020-02-05T21:15:11.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:11.918+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.919108+0000) 2020-02-05T21:15:11.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:11.919+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.919505+0000) 2020-02-05T21:15:11.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:11.921+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:11.921865+0000) 2020-02-05T21:15:11.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:11.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80394 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:12.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:12.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:12.214725+0000) 2020-02-05T21:15:12.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:12.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:12.329965+0000) 2020-02-05T21:15:12.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:12.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:12.616436+0000) 2020-02-05T21:15:13.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:13.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:13.456547+0000) 2020-02-05T21:15:13.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:13.485+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:13.485669+0000) 2020-02-05T21:15:15.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:15.828+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:15.829386+0000) 2020-02-05T21:15:16.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:16.438+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.439877+0000) 2020-02-05T21:15:16.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:16.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.726600+0000) 2020-02-05T21:15:16.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:16.930+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.931203+0000) 2020-02-05T21:15:16.934 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:16.932+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.932657+0000) 2020-02-05T21:15:16.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:16.932+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.933044+0000) 2020-02-05T21:15:16.935 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:16.932+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.933367+0000) 2020-02-05T21:15:16.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:16.932+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.933977+0000) 2020-02-05T21:15:16.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:16.933+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.933729+0000) 2020-02-05T21:15:16.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:16.933+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.934579+0000) 2020-02-05T21:15:16.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:16.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.935297+0000) 2020-02-05T21:15:16.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:16.934+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.934977+0000) 2020-02-05T21:15:16.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:16.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.935787+0000) 2020-02-05T21:15:16.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:16.935+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.936142+0000) 2020-02-05T21:15:16.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:16.936+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.936767+0000) 2020-02-05T21:15:16.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:16.935+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.936538+0000) 2020-02-05T21:15:16.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:16.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:16.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:16.940+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.941207+0000) 2020-02-05T21:15:16.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:16.971+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.972053+0000) 2020-02-05T21:15:16.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:16.971+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.972060+0000) 2020-02-05T21:15:16.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:16.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:16.983880+0000) 2020-02-05T21:15:19.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:19.365+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:19.366134+0000) 2020-02-05T21:15:19.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:19.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:19.974252+0000) 2020-02-05T21:15:20.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:20.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:20.225550+0000) 2020-02-05T21:15:21.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:21.945+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.946623+0000) 2020-02-05T21:15:21.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:21.947+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.948042+0000) 2020-02-05T21:15:21.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:21.948+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.948799+0000) 2020-02-05T21:15:21.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:21.949+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.950071+0000) 2020-02-05T21:15:21.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:21.949+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.950549+0000) 2020-02-05T21:15:21.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:21.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.950627+0000) 2020-02-05T21:15:21.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:21.949+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.950355+0000) 2020-02-05T21:15:21.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:21.950+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.951201+0000) 2020-02-05T21:15:21.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:21.950+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.951790+0000) 2020-02-05T21:15:21.957 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:21.950+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.951610+0000) 2020-02-05T21:15:21.958 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:21.950+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.951334+0000) 2020-02-05T21:15:21.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:21.951+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.952177+0000) 2020-02-05T21:15:21.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:21.951+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.952093+0000) 2020-02-05T21:15:21.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:21.956+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:21.957167+0000) 2020-02-05T21:15:21.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:21.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:22.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:22.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:22.214904+0000) 2020-02-05T21:15:22.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:22.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:22.330172+0000) 2020-02-05T21:15:22.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:22.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:22.616688+0000) 2020-02-05T21:15:23.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:23.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:23.456801+0000) 2020-02-05T21:15:23.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:23.484+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:23.485923+0000) 2020-02-05T21:15:25.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:25.828+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:25.829641+0000) 2020-02-05T21:15:26.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:26.446+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.447699+0000) 2020-02-05T21:15:26.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:26.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.726836+0000) 2020-02-05T21:15:26.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:26.962+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.963474+0000) 2020-02-05T21:15:26.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:26.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.963869+0000) 2020-02-05T21:15:26.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:26.963+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.964475+0000) 2020-02-05T21:15:26.966 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:26.964+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.965461+0000) 2020-02-05T21:15:26.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:26.964+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.965793+0000) 2020-02-05T21:15:26.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:26.965+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.965999+0000) 2020-02-05T21:15:26.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:26.966+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.967392+0000) 2020-02-05T21:15:26.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:26.966+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.967777+0000) 2020-02-05T21:15:26.969 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:26.967+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.968161+0000) 2020-02-05T21:15:26.970 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:26.968+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.968650+0000) 2020-02-05T21:15:26.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:26.968+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.968723+0000) 2020-02-05T21:15:26.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:26.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.969773+0000) 2020-02-05T21:15:26.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:26.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.969993+0000) 2020-02-05T21:15:26.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:26.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.970181+0000) 2020-02-05T21:15:26.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:26.971+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.972192+0000) 2020-02-05T21:15:26.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:26.971+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.972220+0000) 2020-02-05T21:15:26.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:26.972+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:26.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:26.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:26.984106+0000) 2020-02-05T21:15:29.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:29.365+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:29.366394+0000) 2020-02-05T21:15:29.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:29.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:29.974507+0000) 2020-02-05T21:15:30.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:30.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:30.225799+0000) 2020-02-05T21:15:31.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:31.978+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.979437+0000) 2020-02-05T21:15:31.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:31.978+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.979845+0000) 2020-02-05T21:15:31.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:31.978+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.980225+0000) 2020-02-05T21:15:31.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:31.979+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.980742+0000) 2020-02-05T21:15:31.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:31.980+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.981245+0000) 2020-02-05T21:15:31.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:31.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.982746+0000) 2020-02-05T21:15:31.984 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:31.980+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.982214+0000) 2020-02-05T21:15:31.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:31.982+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.983751+0000) 2020-02-05T21:15:31.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:31.981+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.983309+0000) 2020-02-05T21:15:31.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:31.983+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.984152+0000) 2020-02-05T21:15:31.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:31.982+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.984090+0000) 2020-02-05T21:15:31.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:31.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.984839+0000) 2020-02-05T21:15:31.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:31.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.985487+0000) 2020-02-05T21:15:31.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:31.987+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:31.988207+0000) 2020-02-05T21:15:31.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:31.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:32.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:32.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:32.215154+0000) 2020-02-05T21:15:32.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:32.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:32.330479+0000) 2020-02-05T21:15:32.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:32.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:32.616919+0000) 2020-02-05T21:15:33.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:33.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:33.457063+0000) 2020-02-05T21:15:33.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:33.485+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:33.486180+0000) 2020-02-05T21:15:35.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:35.829+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:35.829927+0000) 2020-02-05T21:15:36.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:36.455+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.456805+0000) 2020-02-05T21:15:36.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:36.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.727054+0000) 2020-02-05T21:15:36.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:36.970+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.972361+0000) 2020-02-05T21:15:36.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:36.970+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.972352+0000) 2020-02-05T21:15:36.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:36.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.984315+0000) 2020-02-05T21:15:36.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:36.992+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.993990+0000) 2020-02-05T21:15:36.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:36.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.994516+0000) 2020-02-05T21:15:36.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:36.995+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.996116+0000) 2020-02-05T21:15:36.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:36.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.996451+0000) 2020-02-05T21:15:36.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:36.995+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.996672+0000) 2020-02-05T21:15:36.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:36.995+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.996989+0000) 2020-02-05T21:15:36.999 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:36.996+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.997946+0000) 2020-02-05T21:15:37.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:36.997+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.998796+0000) 2020-02-05T21:15:37.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:36.998+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:36.999810+0000) 2020-02-05T21:15:37.001 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:36.998+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:37.000146+0000) 2020-02-05T21:15:37.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:36.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:37.000280+0000) 2020-02-05T21:15:37.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:36.999+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:37.000595+0000) 2020-02-05T21:15:37.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:37.000+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:37.001324+0000) 2020-02-05T21:15:37.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:37.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:37.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:37.004+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:37.005942+0000) 2020-02-05T21:15:38.067 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:15:38.074 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:15:39.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:39.366+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:39.366649+0000) 2020-02-05T21:15:39.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:39.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:39.974724+0000) 2020-02-05T21:15:40.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:40.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:40.226040+0000) 2020-02-05T21:15:42.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:42.009+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.010313+0000) 2020-02-05T21:15:42.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:42.009+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.010847+0000) 2020-02-05T21:15:42.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:42.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.011700+0000) 2020-02-05T21:15:42.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:42.011+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.012391+0000) 2020-02-05T21:15:42.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:42.011+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.012734+0000) 2020-02-05T21:15:42.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:42.012+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.013305+0000) 2020-02-05T21:15:42.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:42.012+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.013695+0000) 2020-02-05T21:15:42.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:42.012+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.014152+0000) 2020-02-05T21:15:42.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:42.012+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.014189+0000) 2020-02-05T21:15:42.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:42.013+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.014935+0000) 2020-02-05T21:15:42.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:42.013+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.015398+0000) 2020-02-05T21:15:42.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:42.014+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.015602+0000) 2020-02-05T21:15:42.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:42.015+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.016417+0000) 2020-02-05T21:15:42.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:42.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.019945+0000) 2020-02-05T21:15:42.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:42.018+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:42.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:42.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.215471+0000) 2020-02-05T21:15:42.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:42.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.330714+0000) 2020-02-05T21:15:42.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:42.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:42.617130+0000) 2020-02-05T21:15:43.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:43.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:43.457300+0000) 2020-02-05T21:15:43.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:43.485+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:43.486407+0000) 2020-02-05T21:15:45.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:45.829+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:45.830151+0000) 2020-02-05T21:15:46.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:46.464+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:46.465114+0000) 2020-02-05T21:15:46.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:46.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:46.727315+0000) 2020-02-05T21:15:46.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:46.971+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:46.972647+0000) 2020-02-05T21:15:46.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:46.971+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:46.972675+0000) 2020-02-05T21:15:46.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:46.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:46.984622+0000) 2020-02-05T21:15:47.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:47.025+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.026381+0000) 2020-02-05T21:15:47.028 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:47.025+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.026813+0000) 2020-02-05T21:15:47.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:47.026+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.027677+0000) 2020-02-05T21:15:47.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:47.026+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.027178+0000) 2020-02-05T21:15:47.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:47.026+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.027441+0000) 2020-02-05T21:15:47.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:47.027+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.028562+0000) 2020-02-05T21:15:47.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:47.026+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.028335+0000) 2020-02-05T21:15:47.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:47.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.028993+0000) 2020-02-05T21:15:47.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:47.028+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.029691+0000) 2020-02-05T21:15:47.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:47.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.030213+0000) 2020-02-05T21:15:47.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:47.028+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.030192+0000) 2020-02-05T21:15:47.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:47.029+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.030768+0000) 2020-02-05T21:15:47.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:47.029+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.030952+0000) 2020-02-05T21:15:47.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:47.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:47.035684+0000) 2020-02-05T21:15:47.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:47.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:49.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:49.365+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:49.366901+0000) 2020-02-05T21:15:49.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:49.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:49.974960+0000) 2020-02-05T21:15:50.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:50.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:50.226287+0000) 2020-02-05T21:15:52.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:52.039+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.040515+0000) 2020-02-05T21:15:52.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:52.040+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.041352+0000) 2020-02-05T21:15:52.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:52.040+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.041746+0000) 2020-02-05T21:15:52.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:52.041+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.042605+0000) 2020-02-05T21:15:52.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:52.041+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.042967+0000) 2020-02-05T21:15:52.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:52.042+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.043748+0000) 2020-02-05T21:15:52.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:52.043+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.044830+0000) 2020-02-05T21:15:52.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:52.044+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.045165+0000) 2020-02-05T21:15:52.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:52.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.045834+0000) 2020-02-05T21:15:52.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:52.044+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.045775+0000) 2020-02-05T21:15:52.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:52.044+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.045709+0000) 2020-02-05T21:15:52.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:52.045+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.046740+0000) 2020-02-05T21:15:52.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:52.046+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.047394+0000) 2020-02-05T21:15:52.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:52.049+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.050307+0000) 2020-02-05T21:15:52.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:52.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:52.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:52.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.215646+0000) 2020-02-05T21:15:52.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:52.330+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.330952+0000) 2020-02-05T21:15:52.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:52.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:52.617335+0000) 2020-02-05T21:15:53.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:53.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:53.457551+0000) 2020-02-05T21:15:53.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:53.485+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:53.486655+0000) 2020-02-05T21:15:55.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:55.830+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:55.830373+0000) 2020-02-05T21:15:56.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:56.473+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:56.473413+0000) 2020-02-05T21:15:56.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:56.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:56.727567+0000) 2020-02-05T21:15:56.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:56.972+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:56.972930+0000) 2020-02-05T21:15:56.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:56.972+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:56.972894+0000) 2020-02-05T21:15:56.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:56.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:56.984899+0000) 2020-02-05T21:15:57.058 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:15:57.055+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.056792+0000) 2020-02-05T21:15:57.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:15:57.056+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.057313+0000) 2020-02-05T21:15:57.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:57.057+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.058532+0000) 2020-02-05T21:15:57.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:15:57.058+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.059335+0000) 2020-02-05T21:15:57.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:15:57.057+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.058982+0000) 2020-02-05T21:15:57.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:57.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.060356+0000) 2020-02-05T21:15:57.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:15:57.058+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.059877+0000) 2020-02-05T21:15:57.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:15:57.059+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.060937+0000) 2020-02-05T21:15:57.064 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:15:57.060+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.062344+0000) 2020-02-05T21:15:57.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:15:57.060+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.062067+0000) 2020-02-05T21:15:57.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:15:57.061+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.062941+0000) 2020-02-05T21:15:57.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:15:57.061+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.062851+0000) 2020-02-05T21:15:57.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:57.062+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.063601+0000) 2020-02-05T21:15:57.069 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:15:57.066+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:15:57.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:57.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:57.069375+0000) 2020-02-05T21:15:59.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:15:59.366+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:59.367131+0000) 2020-02-05T21:15:59.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:15:59.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:15:59.975192+0000) 2020-02-05T21:16:00.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:00.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:00.226527+0000) 2020-02-05T21:16:02.074 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:02.071+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.072801+0000) 2020-02-05T21:16:02.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:02.072+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.073814+0000) 2020-02-05T21:16:02.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:02.074+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.075676+0000) 2020-02-05T21:16:02.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:02.075+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.076420+0000) 2020-02-05T21:16:02.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:02.075+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.076921+0000) 2020-02-05T21:16:02.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:02.076+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.077471+0000) 2020-02-05T21:16:02.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:02.076+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.077699+0000) 2020-02-05T21:16:02.080 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:02.076+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.078015+0000) 2020-02-05T21:16:02.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:02.077+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.078501+0000) 2020-02-05T21:16:02.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:02.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.079446+0000) 2020-02-05T21:16:02.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:02.077+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.079052+0000) 2020-02-05T21:16:02.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:02.078+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.079998+0000) 2020-02-05T21:16:02.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:02.079+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.080364+0000) 2020-02-05T21:16:02.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:02.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:02.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:02.084+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.085124+0000) 2020-02-05T21:16:02.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:02.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.215861+0000) 2020-02-05T21:16:02.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:02.329+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.331234+0000) 2020-02-05T21:16:02.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:02.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:02.617601+0000) 2020-02-05T21:16:03.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:03.457+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:03.457850+0000) 2020-02-05T21:16:03.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:03.486+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:03.486894+0000) 2020-02-05T21:16:05.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:05.830+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:05.830608+0000) 2020-02-05T21:16:06.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:06.481+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:06.481758+0000) 2020-02-05T21:16:06.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:06.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:06.727799+0000) 2020-02-05T21:16:06.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:06.972+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:06.973164+0000) 2020-02-05T21:16:06.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:06.972+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:06.973162+0000) 2020-02-05T21:16:06.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:06.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:06.985166+0000) 2020-02-05T21:16:07.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:07.089+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.089466+0000) 2020-02-05T21:16:07.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:07.090+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.090721+0000) 2020-02-05T21:16:07.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:07.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.092448+0000) 2020-02-05T21:16:07.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:07.092+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.092689+0000) 2020-02-05T21:16:07.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:07.091+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.093161+0000) 2020-02-05T21:16:07.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:07.092+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.093843+0000) 2020-02-05T21:16:07.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:07.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.094617+0000) 2020-02-05T21:16:07.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:07.092+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.094385+0000) 2020-02-05T21:16:07.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:07.093+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.095075+0000) 2020-02-05T21:16:07.097 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:07.095+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.095902+0000) 2020-02-05T21:16:07.097 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:07.096+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.096282+0000) 2020-02-05T21:16:07.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:07.095+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.096506+0000) 2020-02-05T21:16:07.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:07.095+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.096578+0000) 2020-02-05T21:16:07.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:07.099+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:07.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:07.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:07.101547+0000) 2020-02-05T21:16:08.223 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:16:08.229 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:16:09.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:09.367+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:09.367319+0000) 2020-02-05T21:16:09.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:09.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:09.975422+0000) 2020-02-05T21:16:10.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:10.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:10.226792+0000) 2020-02-05T21:16:12.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:12.105+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.107039+0000) 2020-02-05T21:16:12.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:12.107+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.108444+0000) 2020-02-05T21:16:12.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:12.107+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.108842+0000) 2020-02-05T21:16:12.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:12.107+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.108818+0000) 2020-02-05T21:16:12.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:12.108+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.109981+0000) 2020-02-05T21:16:12.111 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:12.109+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.110112+0000) 2020-02-05T21:16:12.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:12.108+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.110199+0000) 2020-02-05T21:16:12.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:12.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.110875+0000) 2020-02-05T21:16:12.112 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:12.109+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.111045+0000) 2020-02-05T21:16:12.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:12.110+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.112178+0000) 2020-02-05T21:16:12.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:12.111+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.112596+0000) 2020-02-05T21:16:12.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:12.111+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.112980+0000) 2020-02-05T21:16:12.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:12.111+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.112491+0000) 2020-02-05T21:16:12.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:12.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:12.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:12.117+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.118347+0000) 2020-02-05T21:16:12.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:12.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.216090+0000) 2020-02-05T21:16:12.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:12.330+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.331490+0000) 2020-02-05T21:16:12.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:12.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:12.617820+0000) 2020-02-05T21:16:13.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:13.457+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:13.458090+0000) 2020-02-05T21:16:13.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:13.486+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:13.487122+0000) 2020-02-05T21:16:15.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:15.829+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:15.830837+0000) 2020-02-05T21:16:16.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:16.489+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:16.490000+0000) 2020-02-05T21:16:16.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:16.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:16.728029+0000) 2020-02-05T21:16:16.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:16.972+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:16.973410+0000) 2020-02-05T21:16:16.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:16.972+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:16.973416+0000) 2020-02-05T21:16:16.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:16.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:16.985400+0000) 2020-02-05T21:16:17.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:17.121+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.122208+0000) 2020-02-05T21:16:17.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:17.121+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.122810+0000) 2020-02-05T21:16:17.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:17.123+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.124186+0000) 2020-02-05T21:16:17.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:17.123+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.124754+0000) 2020-02-05T21:16:17.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:17.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.125345+0000) 2020-02-05T21:16:17.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:17.124+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.125996+0000) 2020-02-05T21:16:17.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:17.125+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.126906+0000) 2020-02-05T21:16:17.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:17.124+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.126381+0000) 2020-02-05T21:16:17.129 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:17.126+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.127194+0000) 2020-02-05T21:16:17.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:17.126+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.127530+0000) 2020-02-05T21:16:17.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:17.126+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.127933+0000) 2020-02-05T21:16:17.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:17.126+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.128163+0000) 2020-02-05T21:16:17.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:17.127+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.128831+0000) 2020-02-05T21:16:17.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:17.129+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:17.130956+0000) 2020-02-05T21:16:17.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:17.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80907 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:19.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:19.366+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:19.367606+0000) 2020-02-05T21:16:19.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:19.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:19.975685+0000) 2020-02-05T21:16:20.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:20.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:20.227047+0000) 2020-02-05T21:16:22.140 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:22.138+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.139297+0000) 2020-02-05T21:16:22.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:22.138+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.139860+0000) 2020-02-05T21:16:22.142 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:22.139+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.141015+0000) 2020-02-05T21:16:22.142 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:22.140+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.141556+0000) 2020-02-05T21:16:22.143 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:22.140+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.141856+0000) 2020-02-05T21:16:22.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:22.141+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.142248+0000) 2020-02-05T21:16:22.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:22.141+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.142292+0000) 2020-02-05T21:16:22.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:22.142+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.142732+0000) 2020-02-05T21:16:22.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:22.143+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.143803+0000) 2020-02-05T21:16:22.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:22.143+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.144204+0000) 2020-02-05T21:16:22.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:22.143+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.145048+0000) 2020-02-05T21:16:22.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:22.143+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.143996+0000) 2020-02-05T21:16:22.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:22.144+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.144514+0000) 2020-02-05T21:16:22.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:22.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.146332+0000) 2020-02-05T21:16:22.151 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:22.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:22.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:22.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.216347+0000) 2020-02-05T21:16:22.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:22.331+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.331710+0000) 2020-02-05T21:16:22.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:22.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:22.618041+0000) 2020-02-05T21:16:23.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:23.457+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:23.458320+0000) 2020-02-05T21:16:23.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:23.486+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:23.487356+0000) 2020-02-05T21:16:25.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:25.830+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:25.831047+0000) 2020-02-05T21:16:26.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:26.497+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:26.498507+0000) 2020-02-05T21:16:26.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:26.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:26.728317+0000) 2020-02-05T21:16:26.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:26.973+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:26.973715+0000) 2020-02-05T21:16:26.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:26.973+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:26.973696+0000) 2020-02-05T21:16:26.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:26.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:26.985631+0000) 2020-02-05T21:16:27.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:27.156+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.157388+0000) 2020-02-05T21:16:27.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:27.156+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.157823+0000) 2020-02-05T21:16:27.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:27.157+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.158653+0000) 2020-02-05T21:16:27.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:27.157+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.158238+0000) 2020-02-05T21:16:27.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:27.158+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.159192+0000) 2020-02-05T21:16:27.160 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:27.158+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.159342+0000) 2020-02-05T21:16:27.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:27.159+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.159734+0000) 2020-02-05T21:16:27.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:27.160+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.160771+0000) 2020-02-05T21:16:27.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:27.160+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.161139+0000) 2020-02-05T21:16:27.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:27.160+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.161200+0000) 2020-02-05T21:16:27.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:27.161+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.161470+0000) 2020-02-05T21:16:27.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:27.161+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.161666+0000) 2020-02-05T21:16:27.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:27.163+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.164688+0000) 2020-02-05T21:16:27.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:27.163+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:27.165002+0000) 2020-02-05T21:16:27.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:27.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 80986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:29.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:29.367+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:29.367863+0000) 2020-02-05T21:16:29.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:29.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:29.975941+0000) 2020-02-05T21:16:30.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:30.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:30.227305+0000) 2020-02-05T21:16:32.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:32.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.172710+0000) 2020-02-05T21:16:32.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:32.172+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.173218+0000) 2020-02-05T21:16:32.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:32.172+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.173678+0000) 2020-02-05T21:16:32.175 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:32.173+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.174373+0000) 2020-02-05T21:16:32.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:32.174+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.174895+0000) 2020-02-05T21:16:32.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:32.175+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.176471+0000) 2020-02-05T21:16:32.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:32.175+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.176320+0000) 2020-02-05T21:16:32.178 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:32.176+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.177034+0000) 2020-02-05T21:16:32.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:32.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.176966+0000) 2020-02-05T21:16:32.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:32.177+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.177692+0000) 2020-02-05T21:16:32.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:32.177+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.178253+0000) 2020-02-05T21:16:32.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:32.178+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.178543+0000) 2020-02-05T21:16:32.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:32.178+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.179125+0000) 2020-02-05T21:16:32.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:32.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.181188+0000) 2020-02-05T21:16:32.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:32.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:32.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:32.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.216657+0000) 2020-02-05T21:16:32.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:32.331+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.331921+0000) 2020-02-05T21:16:32.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:32.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:32.618296+0000) 2020-02-05T21:16:33.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:33.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:33.458585+0000) 2020-02-05T21:16:33.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:33.486+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:33.487591+0000) 2020-02-05T21:16:35.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:35.830+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:35.831341+0000) 2020-02-05T21:16:36.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:36.506+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:36.506706+0000) 2020-02-05T21:16:36.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:36.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:36.728555+0000) 2020-02-05T21:16:36.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:36.973+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:36.973916+0000) 2020-02-05T21:16:36.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:36.973+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:36.973908+0000) 2020-02-05T21:16:36.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:36.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:36.985865+0000) 2020-02-05T21:16:37.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:37.187+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.188354+0000) 2020-02-05T21:16:37.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:37.188+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.189203+0000) 2020-02-05T21:16:37.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:37.188+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.189871+0000) 2020-02-05T21:16:37.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:37.189+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.190207+0000) 2020-02-05T21:16:37.193 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:37.189+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.190345+0000) 2020-02-05T21:16:37.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:37.190+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.190518+0000) 2020-02-05T21:16:37.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:37.190+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.191499+0000) 2020-02-05T21:16:37.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:37.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.192349+0000) 2020-02-05T21:16:37.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:37.192+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.192666+0000) 2020-02-05T21:16:37.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:37.192+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.193037+0000) 2020-02-05T21:16:37.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:37.192+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.193138+0000) 2020-02-05T21:16:37.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:37.191+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.192990+0000) 2020-02-05T21:16:37.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:37.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.196606+0000) 2020-02-05T21:16:37.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:37.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:37.197653+0000) 2020-02-05T21:16:37.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:37.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81065 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:38.420 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:16:38.426 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:16:39.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:39.367+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:39.368134+0000) 2020-02-05T21:16:39.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:39.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:39.976172+0000) 2020-02-05T21:16:40.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:40.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:40.227570+0000) 2020-02-05T21:16:42.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:42.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.204363+0000) 2020-02-05T21:16:42.206 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:42.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.205589+0000) 2020-02-05T21:16:42.207 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:42.204+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.205839+0000) 2020-02-05T21:16:42.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:42.205+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.205931+0000) 2020-02-05T21:16:42.207 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:42.205+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.206487+0000) 2020-02-05T21:16:42.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:42.206+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.207206+0000) 2020-02-05T21:16:42.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:42.207+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.207554+0000) 2020-02-05T21:16:42.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:42.207+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.208152+0000) 2020-02-05T21:16:42.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:42.208+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.208628+0000) 2020-02-05T21:16:42.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:42.208+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.209029+0000) 2020-02-05T21:16:42.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:42.207+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.209037+0000) 2020-02-05T21:16:42.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:42.209+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.209746+0000) 2020-02-05T21:16:42.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:42.209+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.209543+0000) 2020-02-05T21:16:42.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:42.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.211498+0000) 2020-02-05T21:16:42.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:42.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:42.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:42.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.216811+0000) 2020-02-05T21:16:42.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:42.331+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.332154+0000) 2020-02-05T21:16:42.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:42.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:42.618507+0000) 2020-02-05T21:16:43.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:43.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:43.458846+0000) 2020-02-05T21:16:43.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:43.487+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:43.487823+0000) 2020-02-05T21:16:45.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:45.830+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:45.831570+0000) 2020-02-05T21:16:46.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:46.513+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:46.514632+0000) 2020-02-05T21:16:46.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:46.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:46.728787+0000) 2020-02-05T21:16:46.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:46.973+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:46.974147+0000) 2020-02-05T21:16:46.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:46.973+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:46.974144+0000) 2020-02-05T21:16:46.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:46.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:46.986116+0000) 2020-02-05T21:16:47.220 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:47.218+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.219465+0000) 2020-02-05T21:16:47.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:47.219+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.220486+0000) 2020-02-05T21:16:47.222 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:47.219+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.220697+0000) 2020-02-05T21:16:47.222 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:47.220+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.221395+0000) 2020-02-05T21:16:47.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:47.221+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.222010+0000) 2020-02-05T21:16:47.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:47.222+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.222421+0000) 2020-02-05T21:16:47.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:47.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.222919+0000) 2020-02-05T21:16:47.224 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:47.222+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.223411+0000) 2020-02-05T21:16:47.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:47.223+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.223727+0000) 2020-02-05T21:16:47.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:47.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.223671+0000) 2020-02-05T21:16:47.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:47.223+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.224069+0000) 2020-02-05T21:16:47.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:47.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.224773+0000) 2020-02-05T21:16:47.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:47.224+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.225081+0000) 2020-02-05T21:16:47.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:47.229+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:47.230273+0000) 2020-02-05T21:16:47.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:47.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:49.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:49.368+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:49.368395+0000) 2020-02-05T21:16:49.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:49.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:49.976359+0000) 2020-02-05T21:16:50.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:50.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:50.227831+0000) 2020-02-05T21:16:52.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:52.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.217002+0000) 2020-02-05T21:16:52.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:52.237+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.238232+0000) 2020-02-05T21:16:52.240 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:52.238+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.239377+0000) 2020-02-05T21:16:52.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:52.239+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.239756+0000) 2020-02-05T21:16:52.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:52.239+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.240246+0000) 2020-02-05T21:16:52.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:52.239+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.240621+0000) 2020-02-05T21:16:52.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:52.240+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.241415+0000) 2020-02-05T21:16:52.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:52.240+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.241566+0000) 2020-02-05T21:16:52.243 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:52.241+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.242140+0000) 2020-02-05T21:16:52.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:52.242+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.242686+0000) 2020-02-05T21:16:52.244 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:52.241+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.242542+0000) 2020-02-05T21:16:52.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:52.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.243193+0000) 2020-02-05T21:16:52.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:52.243+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.243636+0000) 2020-02-05T21:16:52.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:52.242+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.243779+0000) 2020-02-05T21:16:52.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:52.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:52.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:52.247+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.248764+0000) 2020-02-05T21:16:52.334 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:52.332+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.332377+0000) 2020-02-05T21:16:52.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:52.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:52.618717+0000) 2020-02-05T21:16:53.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:53.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:53.459140+0000) 2020-02-05T21:16:53.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:53.486+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:53.488032+0000) 2020-02-05T21:16:55.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:55.831+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:55.831794+0000) 2020-02-05T21:16:56.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:56.521+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:56.522790+0000) 2020-02-05T21:16:56.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:56.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:56.729020+0000) 2020-02-05T21:16:56.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:56.973+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:56.974392+0000) 2020-02-05T21:16:56.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:56.974+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:56.974415+0000) 2020-02-05T21:16:56.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:56.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:56.986345+0000) 2020-02-05T21:16:57.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:16:57.252+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.252843+0000) 2020-02-05T21:16:57.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:57.253+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.254882+0000) 2020-02-05T21:16:57.257 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:16:57.254+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.255700+0000) 2020-02-05T21:16:57.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:16:57.254+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.254521+0000) 2020-02-05T21:16:57.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:16:57.255+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.255524+0000) 2020-02-05T21:16:57.258 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:16:57.255+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.256455+0000) 2020-02-05T21:16:57.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:16:57.256+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.257201+0000) 2020-02-05T21:16:57.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:57.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.257739+0000) 2020-02-05T21:16:57.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:16:57.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.258231+0000) 2020-02-05T21:16:57.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:16:57.258+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.258568+0000) 2020-02-05T21:16:57.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:57.258+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.259564+0000) 2020-02-05T21:16:57.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:16:57.258+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.259866+0000) 2020-02-05T21:16:57.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:16:57.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.260523+0000) 2020-02-05T21:16:57.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:16:57.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:16:57.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:57.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:57.266117+0000) 2020-02-05T21:16:59.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:16:59.368+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:59.368636+0000) 2020-02-05T21:16:59.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:16:59.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:16:59.976588+0000) 2020-02-05T21:17:00.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:00.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:00.228083+0000) 2020-02-05T21:17:02.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:02.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.217258+0000) 2020-02-05T21:17:02.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:02.269+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.269327+0000) 2020-02-05T21:17:02.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:02.268+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.269577+0000) 2020-02-05T21:17:02.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:02.270+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.270369+0000) 2020-02-05T21:17:02.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:02.270+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.270770+0000) 2020-02-05T21:17:02.272 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:02.270+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.270709+0000) 2020-02-05T21:17:02.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:02.269+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.270881+0000) 2020-02-05T21:17:02.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:02.269+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.271368+0000) 2020-02-05T21:17:02.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:02.270+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.271825+0000) 2020-02-05T21:17:02.274 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:02.271+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.271963+0000) 2020-02-05T21:17:02.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:02.270+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.272068+0000) 2020-02-05T21:17:02.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:02.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.272865+0000) 2020-02-05T21:17:02.274 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:02.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.273239+0000) 2020-02-05T21:17:02.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:02.272+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.273803+0000) 2020-02-05T21:17:02.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:02.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.277065+0000) 2020-02-05T21:17:02.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:02.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:02.334 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:02.331+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.332658+0000) 2020-02-05T21:17:02.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:02.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:02.618953+0000) 2020-02-05T21:17:03.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:03.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:03.459376+0000) 2020-02-05T21:17:03.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:03.488+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:03.488280+0000) 2020-02-05T21:17:05.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:05.831+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:05.832053+0000) 2020-02-05T21:17:06.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:06.530+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:06.531156+0000) 2020-02-05T21:17:06.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:06.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:06.729293+0000) 2020-02-05T21:17:06.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:06.973+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:06.974688+0000) 2020-02-05T21:17:06.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:06.973+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:06.974688+0000) 2020-02-05T21:17:06.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:06.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:06.986595+0000) 2020-02-05T21:17:07.285 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:07.282+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.284240+0000) 2020-02-05T21:17:07.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:07.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.285506+0000) 2020-02-05T21:17:07.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:07.284+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.285688+0000) 2020-02-05T21:17:07.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:07.284+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.285805+0000) 2020-02-05T21:17:07.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:07.285+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286165+0000) 2020-02-05T21:17:07.288 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:07.285+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286335+0000) 2020-02-05T21:17:07.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:07.285+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286934+0000) 2020-02-05T21:17:07.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:07.285+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286861+0000) 2020-02-05T21:17:07.289 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:07.285+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286492+0000) 2020-02-05T21:17:07.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:07.285+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.286744+0000) 2020-02-05T21:17:07.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:07.286+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.287605+0000) 2020-02-05T21:17:07.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:07.287+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.288652+0000) 2020-02-05T21:17:07.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:07.287+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.288789+0000) 2020-02-05T21:17:07.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:07.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:07.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:07.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:07.293198+0000) 2020-02-05T21:17:08.591 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:17:08.598 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:17:09.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:09.367+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:09.368878+0000) 2020-02-05T21:17:09.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:09.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:09.976817+0000) 2020-02-05T21:17:10.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:10.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:10.228316+0000) 2020-02-05T21:17:12.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:12.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.217517+0000) 2020-02-05T21:17:12.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:12.297+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.298584+0000) 2020-02-05T21:17:12.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:12.298+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.299476+0000) 2020-02-05T21:17:12.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:12.298+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.299644+0000) 2020-02-05T21:17:12.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:12.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.300544+0000) 2020-02-05T21:17:12.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:12.298+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.300108+0000) 2020-02-05T21:17:12.302 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:12.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.301221+0000) 2020-02-05T21:17:12.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:12.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.302150+0000) 2020-02-05T21:17:12.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:12.300+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.302128+0000) 2020-02-05T21:17:12.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:12.301+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.302563+0000) 2020-02-05T21:17:12.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:12.301+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.302769+0000) 2020-02-05T21:17:12.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:12.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.303791+0000) 2020-02-05T21:17:12.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:12.302+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.303761+0000) 2020-02-05T21:17:12.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:12.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:12.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:12.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.308425+0000) 2020-02-05T21:17:12.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:12.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.311189+0000) 2020-02-05T21:17:12.334 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:12.332+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.332878+0000) 2020-02-05T21:17:12.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:12.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:12.619187+0000) 2020-02-05T21:17:13.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:13.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:13.459673+0000) 2020-02-05T21:17:13.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:13.487+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:13.488517+0000) 2020-02-05T21:17:15.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:15.831+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:15.832408+0000) 2020-02-05T21:17:16.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:16.537+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:16.538606+0000) 2020-02-05T21:17:16.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:16.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:16.729486+0000) 2020-02-05T21:17:16.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:16.974+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:16.974930+0000) 2020-02-05T21:17:16.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:16.974+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:16.974926+0000) 2020-02-05T21:17:16.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:16.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:16.986827+0000) 2020-02-05T21:17:17.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:17.314+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.315730+0000) 2020-02-05T21:17:17.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:17.314+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.316158+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:17.316+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.317482+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:17.316+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.318257+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:17.317+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.318947+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:17.317+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.319396+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:17.312+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.313703+0000) 2020-02-05T21:17:17.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:17.313+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.314993+0000) 2020-02-05T21:17:17.325 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:17.317+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.318242+0000) 2020-02-05T21:17:17.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:17.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.318802+0000) 2020-02-05T21:17:17.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:17.321+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.322392+0000) 2020-02-05T21:17:17.325 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:17.318+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.319498+0000) 2020-02-05T21:17:17.325 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:17.318+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.319755+0000) 2020-02-05T21:17:17.326 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:17.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:17.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:17.322+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:17.323493+0000) 2020-02-05T21:17:19.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:19.368+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:19.369142+0000) 2020-02-05T21:17:19.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:19.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:19.977116+0000) 2020-02-05T21:17:20.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:20.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:20.228564+0000) 2020-02-05T21:17:22.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:22.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.217775+0000) 2020-02-05T21:17:22.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:22.329+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.331237+0000) 2020-02-05T21:17:22.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:22.328+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.329548+0000) 2020-02-05T21:17:22.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:22.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.330423+0000) 2020-02-05T21:17:22.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:22.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.332385+0000) 2020-02-05T21:17:22.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:22.331+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.332653+0000) 2020-02-05T21:17:22.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:22.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.333043+0000) 2020-02-05T21:17:22.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:22.332+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.333944+0000) 2020-02-05T21:17:22.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:22.331+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.333348+0000) 2020-02-05T21:17:22.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:22.333+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.334568+0000) 2020-02-05T21:17:22.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:22.333+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.334931+0000) 2020-02-05T21:17:22.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:22.334+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.336159+0000) 2020-02-05T21:17:22.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:22.334+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.336300+0000) 2020-02-05T21:17:22.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:22.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.335362+0000) 2020-02-05T21:17:22.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:22.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:22.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:22.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.341128+0000) 2020-02-05T21:17:22.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:22.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:22.619409+0000) 2020-02-05T21:17:23.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:23.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:23.459913+0000) 2020-02-05T21:17:23.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:23.488+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:23.488777+0000) 2020-02-05T21:17:25.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:25.831+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:25.832633+0000) 2020-02-05T21:17:26.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:26.545+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:26.546932+0000) 2020-02-05T21:17:26.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:26.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:26.729719+0000) 2020-02-05T21:17:26.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:26.974+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:26.975188+0000) 2020-02-05T21:17:26.977 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:26.974+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:26.975189+0000) 2020-02-05T21:17:26.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:26.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:26.987051+0000) 2020-02-05T21:17:27.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:27.343+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.344417+0000) 2020-02-05T21:17:27.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:27.343+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.344872+0000) 2020-02-05T21:17:27.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:27.345+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.346399+0000) 2020-02-05T21:17:27.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:27.345+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.347050+0000) 2020-02-05T21:17:27.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:27.346+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.347603+0000) 2020-02-05T21:17:27.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:27.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.348023+0000) 2020-02-05T21:17:27.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:27.347+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.348454+0000) 2020-02-05T21:17:27.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:27.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.349050+0000) 2020-02-05T21:17:27.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:27.347+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.349089+0000) 2020-02-05T21:17:27.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:27.348+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.349596+0000) 2020-02-05T21:17:27.352 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:27.349+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.350940+0000) 2020-02-05T21:17:27.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:27.349+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.350995+0000) 2020-02-05T21:17:27.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:27.350+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.352292+0000) 2020-02-05T21:17:27.357 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:27.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:27.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:27.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:27.357281+0000) 2020-02-05T21:17:29.085 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:29.082+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:29.083704+0000) 2020-02-05T21:17:29.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:29.368+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:29.369367+0000) 2020-02-05T21:17:29.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:29.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:29.977373+0000) 2020-02-05T21:17:30.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:30.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:30.228812+0000) 2020-02-05T21:17:32.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:32.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.218029+0000) 2020-02-05T21:17:32.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:32.360+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.361293+0000) 2020-02-05T21:17:32.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:32.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.362885+0000) 2020-02-05T21:17:32.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:32.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.363301+0000) 2020-02-05T21:17:32.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:32.362+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.363652+0000) 2020-02-05T21:17:32.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:32.362+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.364192+0000) 2020-02-05T21:17:32.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:32.363+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.364526+0000) 2020-02-05T21:17:32.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:32.363+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.364824+0000) 2020-02-05T21:17:32.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:32.364+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.365423+0000) 2020-02-05T21:17:32.372 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:32.365+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.367056+0000) 2020-02-05T21:17:32.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:32.366+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.367743+0000) 2020-02-05T21:17:32.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:32.366+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.368108+0000) 2020-02-05T21:17:32.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:32.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.366404+0000) 2020-02-05T21:17:32.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:32.365+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.366959+0000) 2020-02-05T21:17:32.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:32.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:32.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:32.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.372889+0000) 2020-02-05T21:17:32.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:32.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:32.619617+0000) 2020-02-05T21:17:33.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:33.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:33.460152+0000) 2020-02-05T21:17:33.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:33.487+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:33.489029+0000) 2020-02-05T21:17:35.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:35.831+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:35.832890+0000) 2020-02-05T21:17:36.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:36.554+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:36.555180+0000) 2020-02-05T21:17:36.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:36.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:36.729965+0000) 2020-02-05T21:17:36.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:36.974+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:36.975487+0000) 2020-02-05T21:17:36.977 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:36.974+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:36.975477+0000) 2020-02-05T21:17:36.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:36.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:36.987285+0000) 2020-02-05T21:17:37.381 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:37.378+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.379767+0000) 2020-02-05T21:17:37.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:37.379+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.380617+0000) 2020-02-05T21:17:37.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:37.379+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.381031+0000) 2020-02-05T21:17:37.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:37.380+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.381977+0000) 2020-02-05T21:17:37.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:37.381+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.382405+0000) 2020-02-05T21:17:37.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:37.381+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.382619+0000) 2020-02-05T21:17:37.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:37.382+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.383190+0000) 2020-02-05T21:17:37.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:37.382+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.383412+0000) 2020-02-05T21:17:37.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:37.382+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.383861+0000) 2020-02-05T21:17:37.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:37.382+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.383964+0000) 2020-02-05T21:17:37.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:37.382+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.384271+0000) 2020-02-05T21:17:37.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:37.383+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.384695+0000) 2020-02-05T21:17:37.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:37.383+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.385102+0000) 2020-02-05T21:17:37.390 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:37.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:37.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:37.389+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:37.390193+0000) 2020-02-05T21:17:38.747 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:17:38.753 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:17:39.085 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:39.083+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:39.083964+0000) 2020-02-05T21:17:39.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:39.369+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:39.369643+0000) 2020-02-05T21:17:39.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:39.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:39.977647+0000) 2020-02-05T21:17:40.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:40.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:40.229055+0000) 2020-02-05T21:17:42.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:42.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.218289+0000) 2020-02-05T21:17:42.397 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:42.394+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.395768+0000) 2020-02-05T21:17:42.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:42.394+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.396097+0000) 2020-02-05T21:17:42.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:42.396+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.397213+0000) 2020-02-05T21:17:42.400 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:42.396+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.397973+0000) 2020-02-05T21:17:42.400 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:42.397+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.398449+0000) 2020-02-05T21:17:42.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:42.395+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.397348+0000) 2020-02-05T21:17:42.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:42.397+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.398987+0000) 2020-02-05T21:17:42.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:42.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.399800+0000) 2020-02-05T21:17:42.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:42.399+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.400764+0000) 2020-02-05T21:17:42.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:42.399+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.400475+0000) 2020-02-05T21:17:42.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:42.397+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.399126+0000) 2020-02-05T21:17:42.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:42.398+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.399593+0000) 2020-02-05T21:17:42.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:42.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.399576+0000) 2020-02-05T21:17:42.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:42.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81579 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:42.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:42.406+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.407544+0000) 2020-02-05T21:17:42.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:42.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:42.619840+0000) 2020-02-05T21:17:43.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:43.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:43.460403+0000) 2020-02-05T21:17:43.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:43.488+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:43.489299+0000) 2020-02-05T21:17:45.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:45.832+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:45.833129+0000) 2020-02-05T21:17:46.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:46.562+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:46.563248+0000) 2020-02-05T21:17:46.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:46.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:46.730195+0000) 2020-02-05T21:17:46.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:46.975+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:46.975731+0000) 2020-02-05T21:17:46.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:46.975+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:46.975737+0000) 2020-02-05T21:17:46.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:46.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:46.987515+0000) 2020-02-05T21:17:47.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:47.409+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.410781+0000) 2020-02-05T21:17:47.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:47.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.411775+0000) 2020-02-05T21:17:47.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:47.411+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.412371+0000) 2020-02-05T21:17:47.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:47.412+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.413173+0000) 2020-02-05T21:17:47.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:47.412+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.414179+0000) 2020-02-05T21:17:47.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:47.412+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.414326+0000) 2020-02-05T21:17:47.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:47.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.414388+0000) 2020-02-05T21:17:47.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:47.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.415226+0000) 2020-02-05T21:17:47.417 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:47.413+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.414852+0000) 2020-02-05T21:17:47.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:47.414+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.415497+0000) 2020-02-05T21:17:47.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:47.413+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.414972+0000) 2020-02-05T21:17:47.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:47.413+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.415120+0000) 2020-02-05T21:17:47.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:47.414+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.415647+0000) 2020-02-05T21:17:47.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:47.418+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:47.419880+0000) 2020-02-05T21:17:47.421 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:47.419+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81624 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:49.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:49.083+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:49.084226+0000) 2020-02-05T21:17:49.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:49.368+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:49.369878+0000) 2020-02-05T21:17:49.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:49.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:49.977881+0000) 2020-02-05T21:17:50.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:50.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:50.229307+0000) 2020-02-05T21:17:52.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:52.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.218551+0000) 2020-02-05T21:17:52.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:52.425+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.426599+0000) 2020-02-05T21:17:52.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:52.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.427415+0000) 2020-02-05T21:17:52.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:52.427+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.427618+0000) 2020-02-05T21:17:52.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:52.427+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.428075+0000) 2020-02-05T21:17:52.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:52.428+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.428460+0000) 2020-02-05T21:17:52.430 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:52.427+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.429017+0000) 2020-02-05T21:17:52.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:52.429+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.430105+0000) 2020-02-05T21:17:52.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:52.430+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.430492+0000) 2020-02-05T21:17:52.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:52.430+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.430985+0000) 2020-02-05T21:17:52.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:52.430+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.431097+0000) 2020-02-05T21:17:52.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:52.431+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.431495+0000) 2020-02-05T21:17:52.433 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:52.430+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.431546+0000) 2020-02-05T21:17:52.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:52.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.435412+0000) 2020-02-05T21:17:52.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:52.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.435677+0000) 2020-02-05T21:17:52.437 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:52.435+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:52.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:52.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:52.620063+0000) 2020-02-05T21:17:53.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:53.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:53.460692+0000) 2020-02-05T21:17:53.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:53.489+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:53.489536+0000) 2020-02-05T21:17:55.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:55.833+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:55.833377+0000) 2020-02-05T21:17:56.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:56.572+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:56.572149+0000) 2020-02-05T21:17:56.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:56.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:56.730423+0000) 2020-02-05T21:17:56.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:56.975+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:56.975985+0000) 2020-02-05T21:17:56.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:56.975+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:56.975992+0000) 2020-02-05T21:17:56.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:56.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:56.987752+0000) 2020-02-05T21:17:57.443 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:17:57.441+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.441541+0000) 2020-02-05T21:17:57.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:17:57.441+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.442243+0000) 2020-02-05T21:17:57.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:57.442+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.442999+0000) 2020-02-05T21:17:57.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:57.444+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.444841+0000) 2020-02-05T21:17:57.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:17:57.444+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.444995+0000) 2020-02-05T21:17:57.447 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:17:57.445+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.445355+0000) 2020-02-05T21:17:57.447 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:57.445+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.445691+0000) 2020-02-05T21:17:57.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:17:57.445+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.446042+0000) 2020-02-05T21:17:57.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:57.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.446546+0000) 2020-02-05T21:17:57.448 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:17:57.446+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.447096+0000) 2020-02-05T21:17:57.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:17:57.447+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.447358+0000) 2020-02-05T21:17:57.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:17:57.447+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.447944+0000) 2020-02-05T21:17:57.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:17:57.447+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.448157+0000) 2020-02-05T21:17:57.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:57.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:57.451825+0000) 2020-02-05T21:17:57.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:17:57.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:17:59.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:17:59.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:59.084503+0000) 2020-02-05T21:17:59.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:17:59.369+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:59.370133+0000) 2020-02-05T21:17:59.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:17:59.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:17:59.978122+0000) 2020-02-05T21:18:00.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:00.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:00.229566+0000) 2020-02-05T21:18:02.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:02.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.218815+0000) 2020-02-05T21:18:02.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:02.457+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.458897+0000) 2020-02-05T21:18:02.460 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:02.458+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.459336+0000) 2020-02-05T21:18:02.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:02.458+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.459814+0000) 2020-02-05T21:18:02.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:02.459+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.460638+0000) 2020-02-05T21:18:02.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:02.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.460142+0000) 2020-02-05T21:18:02.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:02.460+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.461255+0000) 2020-02-05T21:18:02.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:02.461+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.461511+0000) 2020-02-05T21:18:02.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:02.461+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.461577+0000) 2020-02-05T21:18:02.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:02.461+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.463056+0000) 2020-02-05T21:18:02.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:02.462+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.462872+0000) 2020-02-05T21:18:02.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:02.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.463998+0000) 2020-02-05T21:18:02.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:02.463+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.463702+0000) 2020-02-05T21:18:02.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:02.463+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.464105+0000) 2020-02-05T21:18:02.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:02.466+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.467230+0000) 2020-02-05T21:18:02.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:02.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81743 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:02.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:02.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:02.620339+0000) 2020-02-05T21:18:03.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:03.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:03.460916+0000) 2020-02-05T21:18:03.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:03.488+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:03.489807+0000) 2020-02-05T21:18:05.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:05.832+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:05.833711+0000) 2020-02-05T21:18:06.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:06.577+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:06.578766+0000) 2020-02-05T21:18:06.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:06.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:06.730642+0000) 2020-02-05T21:18:06.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:06.975+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:06.976233+0000) 2020-02-05T21:18:06.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:06.975+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:06.976213+0000) 2020-02-05T21:18:06.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:06.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:06.988004+0000) 2020-02-05T21:18:07.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:07.473+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.474330+0000) 2020-02-05T21:18:07.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:07.474+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.475189+0000) 2020-02-05T21:18:07.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:07.474+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.475613+0000) 2020-02-05T21:18:07.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:07.475+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.476293+0000) 2020-02-05T21:18:07.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:07.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.476475+0000) 2020-02-05T21:18:07.481 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:07.476+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.476759+0000) 2020-02-05T21:18:07.481 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:07.478+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.478690+0000) 2020-02-05T21:18:07.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:07.479+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.479732+0000) 2020-02-05T21:18:07.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:07.479+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.480026+0000) 2020-02-05T21:18:07.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:07.476+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.477290+0000) 2020-02-05T21:18:07.482 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:07.476+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.477630+0000) 2020-02-05T21:18:07.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:07.477+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.479057+0000) 2020-02-05T21:18:07.483 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:07.479+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.480346+0000) 2020-02-05T21:18:07.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:07.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:07.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:07.484+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:07.485660+0000) 2020-02-05T21:18:08.884 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:18:08.890 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:18:09.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:09.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:09.084823+0000) 2020-02-05T21:18:09.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:09.369+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:09.370372+0000) 2020-02-05T21:18:09.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:09.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:09.978350+0000) 2020-02-05T21:18:10.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:10.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:10.229818+0000) 2020-02-05T21:18:12.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:12.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.219063+0000) 2020-02-05T21:18:12.491 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:12.489+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.490332+0000) 2020-02-05T21:18:12.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:12.490+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.490937+0000) 2020-02-05T21:18:12.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:12.490+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.491389+0000) 2020-02-05T21:18:12.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:12.491+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.492276+0000) 2020-02-05T21:18:12.494 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:12.491+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.492826+0000) 2020-02-05T21:18:12.494 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:12.492+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.493324+0000) 2020-02-05T21:18:12.494 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:12.492+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.493577+0000) 2020-02-05T21:18:12.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:12.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.494019+0000) 2020-02-05T21:18:12.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:12.492+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.494002+0000) 2020-02-05T21:18:12.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:12.494+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.494420+0000) 2020-02-05T21:18:12.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:12.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.495712+0000) 2020-02-05T21:18:12.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:12.495+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.495647+0000) 2020-02-05T21:18:12.498 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:12.495+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.496382+0000) 2020-02-05T21:18:12.501 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:12.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81825 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:12.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:12.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.501666+0000) 2020-02-05T21:18:12.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:12.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:12.620553+0000) 2020-02-05T21:18:13.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:13.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:13.461126+0000) 2020-02-05T21:18:13.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:13.489+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:13.490035+0000) 2020-02-05T21:18:15.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:15.833+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:15.833948+0000) 2020-02-05T21:18:16.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:16.585+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:16.586734+0000) 2020-02-05T21:18:16.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:16.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:16.730956+0000) 2020-02-05T21:18:16.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:16.976+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:16.976449+0000) 2020-02-05T21:18:16.978 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:16.976+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:16.976506+0000) 2020-02-05T21:18:16.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:16.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:16.988312+0000) 2020-02-05T21:18:17.508 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:17.505+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.506699+0000) 2020-02-05T21:18:17.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:17.506+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.507298+0000) 2020-02-05T21:18:17.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:17.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.507677+0000) 2020-02-05T21:18:17.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:17.507+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.508386+0000) 2020-02-05T21:18:17.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:17.507+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.508036+0000) 2020-02-05T21:18:17.510 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:17.507+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.508715+0000) 2020-02-05T21:18:17.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:17.508+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.509815+0000) 2020-02-05T21:18:17.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:17.510+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.511071+0000) 2020-02-05T21:18:17.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:17.511+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.511654+0000) 2020-02-05T21:18:17.513 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:17.510+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.511318+0000) 2020-02-05T21:18:17.514 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:17.509+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.510374+0000) 2020-02-05T21:18:17.514 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:17.511+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.511875+0000) 2020-02-05T21:18:17.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:17.515+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81870 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:17.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:17.515+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.516282+0000) 2020-02-05T21:18:17.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:17.515+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:17.516615+0000) 2020-02-05T21:18:19.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:19.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:19.085105+0000) 2020-02-05T21:18:19.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:19.369+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:19.370618+0000) 2020-02-05T21:18:19.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:19.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:19.978581+0000) 2020-02-05T21:18:20.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:20.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:20.230060+0000) 2020-02-05T21:18:22.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:22.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.219314+0000) 2020-02-05T21:18:22.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:22.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.520768+0000) 2020-02-05T21:18:22.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:22.520+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.521283+0000) 2020-02-05T21:18:22.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:22.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.521860+0000) 2020-02-05T21:18:22.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:22.523+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.524387+0000) 2020-02-05T21:18:22.530 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:22.524+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.524960+0000) 2020-02-05T21:18:22.531 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:22.525+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.525665+0000) 2020-02-05T21:18:22.531 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:22.525+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.526190+0000) 2020-02-05T21:18:22.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:22.526+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.526499+0000) 2020-02-05T21:18:22.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:22.526+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.527196+0000) 2020-02-05T21:18:22.531 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:22.521+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.523051+0000) 2020-02-05T21:18:22.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:22.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.523591+0000) 2020-02-05T21:18:22.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:22.526+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.527424+0000) 2020-02-05T21:18:22.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:22.525+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.526281+0000) 2020-02-05T21:18:22.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:22.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:22.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:22.530+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.531842+0000) 2020-02-05T21:18:22.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:22.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:22.620750+0000) 2020-02-05T21:18:23.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:23.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:23.461363+0000) 2020-02-05T21:18:23.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:23.489+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:23.490274+0000) 2020-02-05T21:18:25.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:25.833+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:25.834186+0000) 2020-02-05T21:18:26.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:26.594+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:26.595468+0000) 2020-02-05T21:18:26.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:26.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:26.731187+0000) 2020-02-05T21:18:26.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:26.975+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:26.976707+0000) 2020-02-05T21:18:26.978 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:26.975+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:26.976788+0000) 2020-02-05T21:18:26.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:26.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:26.988531+0000) 2020-02-05T21:18:27.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:27.534+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.535627+0000) 2020-02-05T21:18:27.538 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:27.536+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.537339+0000) 2020-02-05T21:18:27.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:27.537+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.538130+0000) 2020-02-05T21:18:27.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:27.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.538589+0000) 2020-02-05T21:18:27.540 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:27.536+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.538306+0000) 2020-02-05T21:18:27.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:27.537+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.539327+0000) 2020-02-05T21:18:27.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:27.539+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.540324+0000) 2020-02-05T21:18:27.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:27.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.540724+0000) 2020-02-05T21:18:27.543 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:27.540+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.542006+0000) 2020-02-05T21:18:27.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:27.541+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.542603+0000) 2020-02-05T21:18:27.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:27.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.543023+0000) 2020-02-05T21:18:27.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:27.541+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.543313+0000) 2020-02-05T21:18:27.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:27.541+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.543305+0000) 2020-02-05T21:18:27.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:27.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:27.547139+0000) 2020-02-05T21:18:27.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:27.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81952 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:29.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:29.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:29.085372+0000) 2020-02-05T21:18:29.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:29.369+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:29.370868+0000) 2020-02-05T21:18:29.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:29.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:29.978813+0000) 2020-02-05T21:18:30.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:30.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:30.230310+0000) 2020-02-05T21:18:32.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:32.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.219576+0000) 2020-02-05T21:18:32.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:32.551+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.553045+0000) 2020-02-05T21:18:32.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:32.551+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.552409+0000) 2020-02-05T21:18:32.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:32.553+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.554234+0000) 2020-02-05T21:18:32.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:32.553+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.555022+0000) 2020-02-05T21:18:32.556 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:32.554+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.555240+0000) 2020-02-05T21:18:32.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:32.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.555558+0000) 2020-02-05T21:18:32.557 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:32.554+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.555992+0000) 2020-02-05T21:18:32.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:32.554+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.555974+0000) 2020-02-05T21:18:32.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:32.554+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.556180+0000) 2020-02-05T21:18:32.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:32.554+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.556312+0000) 2020-02-05T21:18:32.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:32.555+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.556505+0000) 2020-02-05T21:18:32.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:32.555+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.556892+0000) 2020-02-05T21:18:32.559 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:32.556+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.557484+0000) 2020-02-05T21:18:32.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:32.561+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.562591+0000) 2020-02-05T21:18:32.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:32.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 81990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:32.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:32.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:32.620973+0000) 2020-02-05T21:18:33.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:33.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:33.461673+0000) 2020-02-05T21:18:33.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:33.490+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:33.490479+0000) 2020-02-05T21:18:35.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:35.834+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:35.834450+0000) 2020-02-05T21:18:36.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:36.607+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:36.607397+0000) 2020-02-05T21:18:36.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:36.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:36.731461+0000) 2020-02-05T21:18:36.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:36.976+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:36.976950+0000) 2020-02-05T21:18:36.978 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:36.976+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:36.977054+0000) 2020-02-05T21:18:36.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:36.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:36.988752+0000) 2020-02-05T21:18:37.570 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:37.568+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.569409+0000) 2020-02-05T21:18:37.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:37.567+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.568954+0000) 2020-02-05T21:18:37.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:37.568+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.569621+0000) 2020-02-05T21:18:37.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:37.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.569950+0000) 2020-02-05T21:18:37.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:37.569+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.570781+0000) 2020-02-05T21:18:37.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:37.571+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.572066+0000) 2020-02-05T21:18:37.574 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:37.570+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.571779+0000) 2020-02-05T21:18:37.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:37.571+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.572772+0000) 2020-02-05T21:18:37.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:37.571+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.572479+0000) 2020-02-05T21:18:37.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:37.572+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.573877+0000) 2020-02-05T21:18:37.575 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:37.572+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.573446+0000) 2020-02-05T21:18:37.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:37.573+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.574685+0000) 2020-02-05T21:18:37.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:37.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.574559+0000) 2020-02-05T21:18:37.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:37.575+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:37.576967+0000) 2020-02-05T21:18:37.580 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:37.577+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82035 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:39.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:39.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:39.085670+0000) 2020-02-05T21:18:39.089 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:18:39.095 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:18:39.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:39.370+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:39.371150+0000) 2020-02-05T21:18:39.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:39.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:39.979111+0000) 2020-02-05T21:18:40.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:40.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:40.230566+0000) 2020-02-05T21:18:42.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:42.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.219815+0000) 2020-02-05T21:18:42.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:42.584+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.585828+0000) 2020-02-05T21:18:42.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:42.585+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.586978+0000) 2020-02-05T21:18:42.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:42.586+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.587805+0000) 2020-02-05T21:18:42.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:42.587+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.588262+0000) 2020-02-05T21:18:42.590 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:42.587+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.588586+0000) 2020-02-05T21:18:42.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:42.588+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.589166+0000) 2020-02-05T21:18:42.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:42.588+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.589600+0000) 2020-02-05T21:18:42.591 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:42.588+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.589732+0000) 2020-02-05T21:18:42.591 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:42.588+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.589984+0000) 2020-02-05T21:18:42.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:42.588+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.590036+0000) 2020-02-05T21:18:42.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:42.588+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.590306+0000) 2020-02-05T21:18:42.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:42.588+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.590412+0000) 2020-02-05T21:18:42.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:42.589+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.590598+0000) 2020-02-05T21:18:42.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:42.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.595499+0000) 2020-02-05T21:18:42.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:42.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82072 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:42.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:42.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:42.621189+0000) 2020-02-05T21:18:43.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:43.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:43.461907+0000) 2020-02-05T21:18:43.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:43.489+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:43.490703+0000) 2020-02-05T21:18:45.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:45.834+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:45.834686+0000) 2020-02-05T21:18:46.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:46.615+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:46.615686+0000) 2020-02-05T21:18:46.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:46.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:46.731695+0000) 2020-02-05T21:18:46.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:46.976+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:46.977210+0000) 2020-02-05T21:18:46.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:46.976+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:46.977333+0000) 2020-02-05T21:18:46.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:46.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:46.988975+0000) 2020-02-05T21:18:47.601 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:47.598+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.599749+0000) 2020-02-05T21:18:47.602 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:47.599+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.600925+0000) 2020-02-05T21:18:47.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:47.599+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.601202+0000) 2020-02-05T21:18:47.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:47.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.601634+0000) 2020-02-05T21:18:47.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:47.600+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.602183+0000) 2020-02-05T21:18:47.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:47.601+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.602725+0000) 2020-02-05T21:18:47.604 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:47.602+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.603115+0000) 2020-02-05T21:18:47.605 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:47.601+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.603172+0000) 2020-02-05T21:18:47.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:47.602+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.603747+0000) 2020-02-05T21:18:47.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:47.602+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.603528+0000) 2020-02-05T21:18:47.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:47.602+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.604041+0000) 2020-02-05T21:18:47.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:47.603+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.604497+0000) 2020-02-05T21:18:47.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:47.604+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.605156+0000) 2020-02-05T21:18:47.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:47.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:47.609392+0000) 2020-02-05T21:18:47.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:47.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82116 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:49.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:49.085+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:49.085917+0000) 2020-02-05T21:18:49.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:49.371+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:49.371342+0000) 2020-02-05T21:18:49.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:49.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:49.979367+0000) 2020-02-05T21:18:50.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:50.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:50.230819+0000) 2020-02-05T21:18:52.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:52.219+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.220079+0000) 2020-02-05T21:18:52.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:52.616+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.616649+0000) 2020-02-05T21:18:52.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:52.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.616868+0000) 2020-02-05T21:18:52.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:52.615+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.617398+0000) 2020-02-05T21:18:52.619 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:52.616+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.617802+0000) 2020-02-05T21:18:52.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:52.618+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.618206+0000) 2020-02-05T21:18:52.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:52.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.618910+0000) 2020-02-05T21:18:52.621 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:52.620+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.620623+0000) 2020-02-05T21:18:52.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:52.618+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.620288+0000) 2020-02-05T21:18:52.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:52.621+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.621158+0000) 2020-02-05T21:18:52.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:52.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.621391+0000) 2020-02-05T21:18:52.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:52.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.621495+0000) 2020-02-05T21:18:52.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:52.619+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.621192+0000) 2020-02-05T21:18:52.624 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:52.619+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.621270+0000) 2020-02-05T21:18:52.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:52.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.622755+0000) 2020-02-05T21:18:52.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:52.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:52.627001+0000) 2020-02-05T21:18:52.630 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:52.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:53.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:53.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:53.462140+0000) 2020-02-05T21:18:53.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:53.489+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:53.490910+0000) 2020-02-05T21:18:55.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:55.833+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:55.834905+0000) 2020-02-05T21:18:56.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:56.621+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:56.622553+0000) 2020-02-05T21:18:56.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:56.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:56.731930+0000) 2020-02-05T21:18:56.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:56.976+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:56.977489+0000) 2020-02-05T21:18:56.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:56.976+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:56.977690+0000) 2020-02-05T21:18:56.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:56.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:56.989282+0000) 2020-02-05T21:18:57.642 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:57.631+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.633370+0000) 2020-02-05T21:18:57.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:18:57.634+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.635690+0000) 2020-02-05T21:18:57.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:18:57.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.637695+0000) 2020-02-05T21:18:57.643 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:18:57.636+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.638252+0000) 2020-02-05T21:18:57.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:18:57.637+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.638714+0000) 2020-02-05T21:18:57.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:18:57.637+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.638955+0000) 2020-02-05T21:18:57.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:57.632+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.633833+0000) 2020-02-05T21:18:57.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:57.637+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.638338+0000) 2020-02-05T21:18:57.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:57.634+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.635438+0000) 2020-02-05T21:18:57.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:18:57.635+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.636206+0000) 2020-02-05T21:18:57.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:18:57.635+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.636626+0000) 2020-02-05T21:18:57.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:18:57.636+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.637280+0000) 2020-02-05T21:18:57.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:18:57.637+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.638690+0000) 2020-02-05T21:18:57.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:18:57.641+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:18:57.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:57.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:57.645006+0000) 2020-02-05T21:18:59.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:18:59.084+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:59.086108+0000) 2020-02-05T21:18:59.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:18:59.370+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:59.371580+0000) 2020-02-05T21:18:59.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:18:59.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:18:59.979635+0000) 2020-02-05T21:19:00.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:00.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:00.231070+0000) 2020-02-05T21:19:02.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:02.219+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.220343+0000) 2020-02-05T21:19:02.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:02.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.621810+0000) 2020-02-05T21:19:02.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:02.647+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.648533+0000) 2020-02-05T21:19:02.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:02.647+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.649005+0000) 2020-02-05T21:19:02.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:02.648+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.649427+0000) 2020-02-05T21:19:02.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:02.648+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.649842+0000) 2020-02-05T21:19:02.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:02.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.651183+0000) 2020-02-05T21:19:02.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:02.650+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.651830+0000) 2020-02-05T21:19:02.653 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:02.650+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.652005+0000) 2020-02-05T21:19:02.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:02.651+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.653198+0000) 2020-02-05T21:19:02.655 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:02.651+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.653125+0000) 2020-02-05T21:19:02.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:02.652+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.653692+0000) 2020-02-05T21:19:02.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:02.653+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.654370+0000) 2020-02-05T21:19:02.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:02.653+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.654846+0000) 2020-02-05T21:19:02.656 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:02.653+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.654953+0000) 2020-02-05T21:19:02.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:02.656+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:02.657145+0000) 2020-02-05T21:19:02.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:02.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:03.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:03.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:03.462370+0000) 2020-02-05T21:19:03.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:03.490+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:03.491147+0000) 2020-02-05T21:19:05.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:05.834+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:05.835135+0000) 2020-02-05T21:19:06.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:06.629+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:06.630201+0000) 2020-02-05T21:19:06.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:06.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:06.732190+0000) 2020-02-05T21:19:06.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:06.977+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:06.977737+0000) 2020-02-05T21:19:06.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:06.977+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:06.977925+0000) 2020-02-05T21:19:06.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:06.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:06.989509+0000) 2020-02-05T21:19:07.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:07.664+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.665245+0000) 2020-02-05T21:19:07.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:07.664+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.665889+0000) 2020-02-05T21:19:07.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:07.666+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.667411+0000) 2020-02-05T21:19:07.669 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:07.666+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.668025+0000) 2020-02-05T21:19:07.669 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:07.665+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.666695+0000) 2020-02-05T21:19:07.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:07.667+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.668425+0000) 2020-02-05T21:19:07.670 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:07.667+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.668964+0000) 2020-02-05T21:19:07.671 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:07.668+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.669664+0000) 2020-02-05T21:19:07.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:07.667+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.669284+0000) 2020-02-05T21:19:07.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:07.668+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.670290+0000) 2020-02-05T21:19:07.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:07.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.670923+0000) 2020-02-05T21:19:07.672 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:07.669+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.671113+0000) 2020-02-05T21:19:07.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:07.670+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.671927+0000) 2020-02-05T21:19:07.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:07.672+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:07.673581+0000) 2020-02-05T21:19:07.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:07.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:09.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:09.085+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:09.086370+0000) 2020-02-05T21:19:09.261 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:19:09.267 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:19:09.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:09.370+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:09.371761+0000) 2020-02-05T21:19:09.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:09.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:09.979864+0000) 2020-02-05T21:19:10.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:10.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:10.231329+0000) 2020-02-05T21:19:12.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:12.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.220600+0000) 2020-02-05T21:19:12.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:12.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.622047+0000) 2020-02-05T21:19:12.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:12.679+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.680995+0000) 2020-02-05T21:19:12.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:12.680+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.681925+0000) 2020-02-05T21:19:12.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:12.681+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.682496+0000) 2020-02-05T21:19:12.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:12.682+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.683282+0000) 2020-02-05T21:19:12.684 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:12.682+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.683415+0000) 2020-02-05T21:19:12.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:12.682+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.683896+0000) 2020-02-05T21:19:12.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:12.683+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.684327+0000) 2020-02-05T21:19:12.686 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:12.683+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.684665+0000) 2020-02-05T21:19:12.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:12.683+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.684969+0000) 2020-02-05T21:19:12.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:12.684+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.685692+0000) 2020-02-05T21:19:12.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:12.685+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.686968+0000) 2020-02-05T21:19:12.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:12.685+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.687411+0000) 2020-02-05T21:19:12.689 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:12.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.687002+0000) 2020-02-05T21:19:12.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:12.689+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82321 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:12.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:12.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:12.691892+0000) 2020-02-05T21:19:13.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:13.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:13.462637+0000) 2020-02-05T21:19:13.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:13.490+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:13.491362+0000) 2020-02-05T21:19:15.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:15.834+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:15.835404+0000) 2020-02-05T21:19:16.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:16.637+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:16.638605+0000) 2020-02-05T21:19:16.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:16.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:16.732420+0000) 2020-02-05T21:19:16.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:16.977+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:16.977987+0000) 2020-02-05T21:19:16.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:16.977+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:16.978200+0000) 2020-02-05T21:19:16.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:16.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:16.989736+0000) 2020-02-05T21:19:17.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:17.694+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.695811+0000) 2020-02-05T21:19:17.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:17.696+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.697418+0000) 2020-02-05T21:19:17.699 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:17.697+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.698122+0000) 2020-02-05T21:19:17.700 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:17.698+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.698650+0000) 2020-02-05T21:19:17.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:17.698+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.699227+0000) 2020-02-05T21:19:17.701 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:17.699+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.700095+0000) 2020-02-05T21:19:17.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:17.700+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.700993+0000) 2020-02-05T21:19:17.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:17.700+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.701523+0000) 2020-02-05T21:19:17.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:17.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.702400+0000) 2020-02-05T21:19:17.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:17.702+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.702546+0000) 2020-02-05T21:19:17.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:17.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.702875+0000) 2020-02-05T21:19:17.704 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:17.702+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.703172+0000) 2020-02-05T21:19:17.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:17.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.704129+0000) 2020-02-05T21:19:17.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:17.704+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:17.705992+0000) 2020-02-05T21:19:17.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:17.707+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:19.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:19.086+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:19.086640+0000) 2020-02-05T21:19:19.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:19.371+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:19.371975+0000) 2020-02-05T21:19:19.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:19.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:19.980136+0000) 2020-02-05T21:19:20.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:20.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:20.231565+0000) 2020-02-05T21:19:22.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:22.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.220848+0000) 2020-02-05T21:19:22.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:22.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.622340+0000) 2020-02-05T21:19:22.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:22.711+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.712666+0000) 2020-02-05T21:19:22.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:22.712+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.713184+0000) 2020-02-05T21:19:22.715 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:22.713+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.714304+0000) 2020-02-05T21:19:22.716 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:22.714+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.714810+0000) 2020-02-05T21:19:22.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:22.715+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.716834+0000) 2020-02-05T21:19:22.718 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:22.715+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.716310+0000) 2020-02-05T21:19:22.719 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:22.716+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.717876+0000) 2020-02-05T21:19:22.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:22.716+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.717811+0000) 2020-02-05T21:19:22.720 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:22.718+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.718538+0000) 2020-02-05T21:19:22.720 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:22.718+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.718886+0000) 2020-02-05T21:19:22.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:22.718+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.719251+0000) 2020-02-05T21:19:22.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:22.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.719745+0000) 2020-02-05T21:19:22.721 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:22.719+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.719591+0000) 2020-02-05T21:19:22.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:22.719+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:22.720675+0000) 2020-02-05T21:19:22.725 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:22.723+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:23.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:23.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:23.462869+0000) 2020-02-05T21:19:23.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:23.491+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:23.491609+0000) 2020-02-05T21:19:25.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:25.835+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:25.835672+0000) 2020-02-05T21:19:26.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:26.646+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:26.646828+0000) 2020-02-05T21:19:26.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:26.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:26.732675+0000) 2020-02-05T21:19:26.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:26.977+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:26.978242+0000) 2020-02-05T21:19:26.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:26.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:26.978424+0000) 2020-02-05T21:19:26.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:26.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:26.989962+0000) 2020-02-05T21:19:27.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:27.728+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.729949+0000) 2020-02-05T21:19:27.732 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:27.730+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.731112+0000) 2020-02-05T21:19:27.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:27.730+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.731615+0000) 2020-02-05T21:19:27.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:27.731+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.731943+0000) 2020-02-05T21:19:27.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:27.732+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.732587+0000) 2020-02-05T21:19:27.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:27.731+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.732782+0000) 2020-02-05T21:19:27.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:27.732+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.733494+0000) 2020-02-05T21:19:27.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:27.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.733876+0000) 2020-02-05T21:19:27.736 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:27.734+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.734845+0000) 2020-02-05T21:19:27.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:27.734+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.735067+0000) 2020-02-05T21:19:27.737 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:27.735+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.736250+0000) 2020-02-05T21:19:27.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:27.735+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.736143+0000) 2020-02-05T21:19:27.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:27.736+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.736597+0000) 2020-02-05T21:19:27.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:27.738+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:27.739468+0000) 2020-02-05T21:19:27.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:27.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:29.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:29.086+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:29.086901+0000) 2020-02-05T21:19:29.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:29.371+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:29.372204+0000) 2020-02-05T21:19:29.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:29.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:29.980374+0000) 2020-02-05T21:19:30.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:30.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:30.231826+0000) 2020-02-05T21:19:32.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:32.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.221139+0000) 2020-02-05T21:19:32.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:32.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.622568+0000) 2020-02-05T21:19:32.749 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:32.746+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.747366+0000) 2020-02-05T21:19:32.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:32.747+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.748227+0000) 2020-02-05T21:19:32.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:32.748+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.749164+0000) 2020-02-05T21:19:32.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:32.748+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.749782+0000) 2020-02-05T21:19:32.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:32.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.750400+0000) 2020-02-05T21:19:32.752 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:32.750+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.751307+0000) 2020-02-05T21:19:32.753 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:32.751+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.751936+0000) 2020-02-05T21:19:32.753 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:32.751+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.752250+0000) 2020-02-05T21:19:32.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:32.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.752651+0000) 2020-02-05T21:19:32.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:32.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.756165+0000) 2020-02-05T21:19:32.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:32.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.756403+0000) 2020-02-05T21:19:32.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:32.752+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.753292+0000) 2020-02-05T21:19:32.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:32.753+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.753651+0000) 2020-02-05T21:19:32.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:32.753+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:32.754108+0000) 2020-02-05T21:19:32.760 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:32.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:33.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:33.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:33.463098+0000) 2020-02-05T21:19:33.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:33.491+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:33.491811+0000) 2020-02-05T21:19:35.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:35.834+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:35.835915+0000) 2020-02-05T21:19:36.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:36.654+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:36.655184+0000) 2020-02-05T21:19:36.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:36.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:36.732906+0000) 2020-02-05T21:19:36.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:36.978+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:36.978497+0000) 2020-02-05T21:19:36.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:36.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:36.978692+0000) 2020-02-05T21:19:36.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:36.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:36.990193+0000) 2020-02-05T21:19:37.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:37.762+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.764054+0000) 2020-02-05T21:19:37.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:37.763+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.764481+0000) 2020-02-05T21:19:37.767 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:37.764+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.765379+0000) 2020-02-05T21:19:37.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:37.766+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.767503+0000) 2020-02-05T21:19:37.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:37.766+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.767769+0000) 2020-02-05T21:19:37.769 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:37.767+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.768191+0000) 2020-02-05T21:19:37.770 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:37.768+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.768807+0000) 2020-02-05T21:19:37.770 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:37.767+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.769002+0000) 2020-02-05T21:19:37.771 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:37.769+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.769542+0000) 2020-02-05T21:19:37.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:37.769+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.770299+0000) 2020-02-05T21:19:37.771 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:37.770+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.770559+0000) 2020-02-05T21:19:37.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:37.771+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.771553+0000) 2020-02-05T21:19:37.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:37.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.774590+0000) 2020-02-05T21:19:37.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:37.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:37.774915+0000) 2020-02-05T21:19:37.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:37.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:39.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:39.086+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:39.087167+0000) 2020-02-05T21:19:39.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:39.372+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:39.372432+0000) 2020-02-05T21:19:39.440 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:19:39.447 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:19:39.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:39.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:39.980598+0000) 2020-02-05T21:19:40.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:40.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:40.232041+0000) 2020-02-05T21:19:42.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:42.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.221403+0000) 2020-02-05T21:19:42.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:42.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.622783+0000) 2020-02-05T21:19:42.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:42.779+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.780570+0000) 2020-02-05T21:19:42.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:42.781+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.782494+0000) 2020-02-05T21:19:42.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:42.782+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.783142+0000) 2020-02-05T21:19:42.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:42.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.783622+0000) 2020-02-05T21:19:42.786 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:42.784+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.784520+0000) 2020-02-05T21:19:42.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:42.784+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.785006+0000) 2020-02-05T21:19:42.787 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:42.783+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.784544+0000) 2020-02-05T21:19:42.787 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:42.783+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.784946+0000) 2020-02-05T21:19:42.787 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:42.784+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.785323+0000) 2020-02-05T21:19:42.788 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:42.785+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.786043+0000) 2020-02-05T21:19:42.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:42.786+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.787016+0000) 2020-02-05T21:19:42.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:42.786+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.787054+0000) 2020-02-05T21:19:42.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:42.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.787367+0000) 2020-02-05T21:19:42.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:42.789+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:42.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:42.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:42.791933+0000) 2020-02-05T21:19:43.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:43.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:43.463355+0000) 2020-02-05T21:19:43.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:43.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:43.492023+0000) 2020-02-05T21:19:45.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:45.836+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:45.836147+0000) 2020-02-05T21:19:46.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:46.663+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:46.663537+0000) 2020-02-05T21:19:46.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:46.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:46.733140+0000) 2020-02-05T21:19:46.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:46.978+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:46.978736+0000) 2020-02-05T21:19:46.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:46.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:46.978924+0000) 2020-02-05T21:19:46.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:46.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:46.990445+0000) 2020-02-05T21:19:47.798 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:47.795+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.796773+0000) 2020-02-05T21:19:47.798 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:47.796+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.796912+0000) 2020-02-05T21:19:47.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:47.796+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.797716+0000) 2020-02-05T21:19:47.799 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:47.798+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.798415+0000) 2020-02-05T21:19:47.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:47.798+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.799010+0000) 2020-02-05T21:19:47.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:47.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.799733+0000) 2020-02-05T21:19:47.801 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:47.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.799675+0000) 2020-02-05T21:19:47.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:47.799+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.800621+0000) 2020-02-05T21:19:47.802 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:47.799+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.800179+0000) 2020-02-05T21:19:47.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:47.800+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.801155+0000) 2020-02-05T21:19:47.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:47.800+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.801599+0000) 2020-02-05T21:19:47.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:47.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.801793+0000) 2020-02-05T21:19:47.803 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:47.801+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.802081+0000) 2020-02-05T21:19:47.808 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:47.806+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:47.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:47.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:47.808802+0000) 2020-02-05T21:19:49.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:49.087+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:49.087466+0000) 2020-02-05T21:19:49.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:49.371+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:49.372671+0000) 2020-02-05T21:19:49.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:49.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:49.980849+0000) 2020-02-05T21:19:50.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:50.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:50.232300+0000) 2020-02-05T21:19:52.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:52.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.221687+0000) 2020-02-05T21:19:52.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:52.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.623026+0000) 2020-02-05T21:19:52.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:52.810+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.811416+0000) 2020-02-05T21:19:52.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:52.811+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.812897+0000) 2020-02-05T21:19:52.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:52.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.813785+0000) 2020-02-05T21:19:52.815 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:52.813+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.814167+0000) 2020-02-05T21:19:52.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:52.814+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.814953+0000) 2020-02-05T21:19:52.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:52.815+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.816125+0000) 2020-02-05T21:19:52.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:52.815+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.816592+0000) 2020-02-05T21:19:52.819 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:52.817+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.817684+0000) 2020-02-05T21:19:52.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:52.817+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.817786+0000) 2020-02-05T21:19:52.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:52.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.818828+0000) 2020-02-05T21:19:52.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:52.817+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.818235+0000) 2020-02-05T21:19:52.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:52.818+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.819089+0000) 2020-02-05T21:19:52.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:52.819+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.820245+0000) 2020-02-05T21:19:52.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:52.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:52.821841+0000) 2020-02-05T21:19:52.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:52.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82656 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:53.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:53.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:53.463589+0000) 2020-02-05T21:19:53.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:53.491+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:53.492352+0000) 2020-02-05T21:19:55.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:55.835+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:55.836362+0000) 2020-02-05T21:19:56.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:56.670+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:56.671735+0000) 2020-02-05T21:19:56.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:56.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:56.733371+0000) 2020-02-05T21:19:56.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:56.977+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:56.978977+0000) 2020-02-05T21:19:56.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:56.977+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:56.979175+0000) 2020-02-05T21:19:56.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:56.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:56.990673+0000) 2020-02-05T21:19:57.831 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:57.828+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.829565+0000) 2020-02-05T21:19:57.831 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:19:57.827+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.829153+0000) 2020-02-05T21:19:57.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:57.828+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.829755+0000) 2020-02-05T21:19:57.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:57.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.830680+0000) 2020-02-05T21:19:57.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:57.830+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.831187+0000) 2020-02-05T21:19:57.833 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:19:57.830+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.831548+0000) 2020-02-05T21:19:57.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:19:57.830+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.832070+0000) 2020-02-05T21:19:57.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:19:57.831+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.832588+0000) 2020-02-05T21:19:57.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:19:57.831+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.832829+0000) 2020-02-05T21:19:57.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:19:57.831+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.833038+0000) 2020-02-05T21:19:57.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:19:57.832+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.833370+0000) 2020-02-05T21:19:57.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:19:57.832+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.833740+0000) 2020-02-05T21:19:57.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:19:57.833+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.834606+0000) 2020-02-05T21:19:57.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:57.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:57.838984+0000) 2020-02-05T21:19:57.841 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:19:57.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82700 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:19:59.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:19:59.086+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:59.087715+0000) 2020-02-05T21:19:59.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:19:59.372+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:59.372925+0000) 2020-02-05T21:19:59.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:19:59.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:19:59.981141+0000) 2020-02-05T21:20:00.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:00.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:00.232562+0000) 2020-02-05T21:20:02.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:02.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.221960+0000) 2020-02-05T21:20:02.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:02.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.623333+0000) 2020-02-05T21:20:02.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:02.845+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.846099+0000) 2020-02-05T21:20:02.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:02.844+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.846362+0000) 2020-02-05T21:20:02.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:02.846+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.847227+0000) 2020-02-05T21:20:02.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:02.845+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.847268+0000) 2020-02-05T21:20:02.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:02.847+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.848175+0000) 2020-02-05T21:20:02.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:02.847+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.848541+0000) 2020-02-05T21:20:02.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:02.847+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.848935+0000) 2020-02-05T21:20:02.850 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:02.848+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.849386+0000) 2020-02-05T21:20:02.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:02.848+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.850267+0000) 2020-02-05T21:20:02.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:02.848+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.849930+0000) 2020-02-05T21:20:02.852 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:02.849+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.850922+0000) 2020-02-05T21:20:02.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:02.849+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.850812+0000) 2020-02-05T21:20:02.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:02.850+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.851065+0000) 2020-02-05T21:20:02.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:02.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:02.854238+0000) 2020-02-05T21:20:02.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:02.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:03.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:03.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:03.463804+0000) 2020-02-05T21:20:03.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:03.491+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:03.492607+0000) 2020-02-05T21:20:05.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:05.835+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:05.836722+0000) 2020-02-05T21:20:06.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:06.679+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:06.680155+0000) 2020-02-05T21:20:06.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:06.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:06.733607+0000) 2020-02-05T21:20:06.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:06.978+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:06.979224+0000) 2020-02-05T21:20:06.981 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:06.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:06.979388+0000) 2020-02-05T21:20:06.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:06.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:06.990960+0000) 2020-02-05T21:20:07.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:07.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.862936+0000) 2020-02-05T21:20:07.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:07.862+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.863427+0000) 2020-02-05T21:20:07.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:07.862+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.864272+0000) 2020-02-05T21:20:07.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:07.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.865427+0000) 2020-02-05T21:20:07.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:07.864+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.866001+0000) 2020-02-05T21:20:07.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:07.865+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.866501+0000) 2020-02-05T21:20:07.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:07.866+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.867539+0000) 2020-02-05T21:20:07.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:07.866+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.867986+0000) 2020-02-05T21:20:07.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:07.866+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.867641+0000) 2020-02-05T21:20:07.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:07.867+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.868583+0000) 2020-02-05T21:20:07.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:07.868+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.869466+0000) 2020-02-05T21:20:07.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:07.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.870323+0000) 2020-02-05T21:20:07.872 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:07.869+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.870285+0000) 2020-02-05T21:20:07.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:07.871+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:07.872946+0000) 2020-02-05T21:20:07.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:07.873+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82782 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:09.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:09.087+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:09.087978+0000) 2020-02-05T21:20:09.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:09.372+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:09.373164+0000) 2020-02-05T21:20:09.612 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:20:09.618 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:20:09.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:09.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:09.981371+0000) 2020-02-05T21:20:10.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:10.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:10.232745+0000) 2020-02-05T21:20:12.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:12.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.222227+0000) 2020-02-05T21:20:12.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:12.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.623590+0000) 2020-02-05T21:20:12.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:12.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.880489+0000) 2020-02-05T21:20:12.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:12.880+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.881610+0000) 2020-02-05T21:20:12.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:12.881+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.882311+0000) 2020-02-05T21:20:12.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:12.882+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.883294+0000) 2020-02-05T21:20:12.885 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:12.881+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.882677+0000) 2020-02-05T21:20:12.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:12.881+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.882980+0000) 2020-02-05T21:20:12.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:12.882+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.883616+0000) 2020-02-05T21:20:12.886 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:12.882+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.884048+0000) 2020-02-05T21:20:12.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:12.884+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.885237+0000) 2020-02-05T21:20:12.887 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:12.884+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.885788+0000) 2020-02-05T21:20:12.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:12.885+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.886507+0000) 2020-02-05T21:20:12.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:12.884+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.886053+0000) 2020-02-05T21:20:12.888 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:12.885+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.886421+0000) 2020-02-05T21:20:12.890 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:12.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:12.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:12.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:12.891462+0000) 2020-02-05T21:20:13.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:13.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:13.464029+0000) 2020-02-05T21:20:13.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:13.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:13.492797+0000) 2020-02-05T21:20:15.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:15.835+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:15.836907+0000) 2020-02-05T21:20:16.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:16.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:16.688133+0000) 2020-02-05T21:20:16.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:16.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:16.733863+0000) 2020-02-05T21:20:16.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:16.978+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:16.979484+0000) 2020-02-05T21:20:16.981 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:16.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:16.979712+0000) 2020-02-05T21:20:16.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:16.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:16.991179+0000) 2020-02-05T21:20:17.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:17.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.895920+0000) 2020-02-05T21:20:17.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:17.893+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.894530+0000) 2020-02-05T21:20:17.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:17.895+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.896849+0000) 2020-02-05T21:20:17.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:17.896+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.897409+0000) 2020-02-05T21:20:17.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:17.896+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.897736+0000) 2020-02-05T21:20:17.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:17.897+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.898236+0000) 2020-02-05T21:20:17.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:17.897+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.898389+0000) 2020-02-05T21:20:17.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:17.898+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.900035+0000) 2020-02-05T21:20:17.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:17.897+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.899407+0000) 2020-02-05T21:20:17.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:17.898+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.899725+0000) 2020-02-05T21:20:17.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:17.899+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.900524+0000) 2020-02-05T21:20:17.902 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:17.899+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.900835+0000) 2020-02-05T21:20:17.903 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:17.899+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.901342+0000) 2020-02-05T21:20:17.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:17.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:17.906240+0000) 2020-02-05T21:20:17.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:17.904+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:19.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:19.087+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:19.088241+0000) 2020-02-05T21:20:19.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:19.372+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:19.373402+0000) 2020-02-05T21:20:19.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:19.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:19.981709+0000) 2020-02-05T21:20:20.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:20.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:20.233042+0000) 2020-02-05T21:20:22.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:22.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.222520+0000) 2020-02-05T21:20:22.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:22.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.623825+0000) 2020-02-05T21:20:22.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:22.910+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.911474+0000) 2020-02-05T21:20:22.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:22.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.912339+0000) 2020-02-05T21:20:22.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:22.912+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.913359+0000) 2020-02-05T21:20:22.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:22.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.913579+0000) 2020-02-05T21:20:22.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:22.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.913795+0000) 2020-02-05T21:20:22.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:22.913+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.914884+0000) 2020-02-05T21:20:22.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:22.913+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.914629+0000) 2020-02-05T21:20:22.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:22.914+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.915941+0000) 2020-02-05T21:20:22.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:22.914+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.915723+0000) 2020-02-05T21:20:22.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:22.915+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.916419+0000) 2020-02-05T21:20:22.919 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:22.915+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.916799+0000) 2020-02-05T21:20:22.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:22.916+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.917560+0000) 2020-02-05T21:20:22.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:22.916+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.918138+0000) 2020-02-05T21:20:22.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:22.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:22.920881+0000) 2020-02-05T21:20:22.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:22.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:23.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:23.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:23.464246+0000) 2020-02-05T21:20:23.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:23.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:23.492997+0000) 2020-02-05T21:20:25.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:25.836+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:25.837135+0000) 2020-02-05T21:20:26.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:26.696+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:26.696610+0000) 2020-02-05T21:20:26.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:26.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:26.734032+0000) 2020-02-05T21:20:26.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:26.979+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:26.979732+0000) 2020-02-05T21:20:26.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:26.979+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:26.979921+0000) 2020-02-05T21:20:26.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:26.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:26.991426+0000) 2020-02-05T21:20:27.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:27.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.928692+0000) 2020-02-05T21:20:27.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:27.928+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.929577+0000) 2020-02-05T21:20:27.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:27.929+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.930136+0000) 2020-02-05T21:20:27.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:27.929+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.930501+0000) 2020-02-05T21:20:27.934 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:27.930+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.932114+0000) 2020-02-05T21:20:27.935 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:27.931+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.932892+0000) 2020-02-05T21:20:27.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:27.931+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.933351+0000) 2020-02-05T21:20:27.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:27.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.931417+0000) 2020-02-05T21:20:27.936 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:27.932+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.934285+0000) 2020-02-05T21:20:27.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:27.933+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.934741+0000) 2020-02-05T21:20:27.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:27.933+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.934804+0000) 2020-02-05T21:20:27.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:27.933+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.934398+0000) 2020-02-05T21:20:27.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:27.933+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.934355+0000) 2020-02-05T21:20:27.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:27.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:27.938572+0000) 2020-02-05T21:20:27.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:27.937+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:29.090 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:29.087+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:29.088501+0000) 2020-02-05T21:20:29.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:29.372+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:29.373635+0000) 2020-02-05T21:20:29.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:29.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:29.981953+0000) 2020-02-05T21:20:30.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:30.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:30.233335+0000) 2020-02-05T21:20:32.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:32.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.222758+0000) 2020-02-05T21:20:32.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:32.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.624050+0000) 2020-02-05T21:20:32.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:32.943+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.944142+0000) 2020-02-05T21:20:32.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:32.943+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.944963+0000) 2020-02-05T21:20:32.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:32.944+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.945243+0000) 2020-02-05T21:20:32.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:32.944+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.946275+0000) 2020-02-05T21:20:32.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:32.946+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.947142+0000) 2020-02-05T21:20:32.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:32.946+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.947519+0000) 2020-02-05T21:20:32.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:32.947+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.948260+0000) 2020-02-05T21:20:32.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:32.947+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.948396+0000) 2020-02-05T21:20:32.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:32.947+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.948740+0000) 2020-02-05T21:20:32.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:32.947+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.949374+0000) 2020-02-05T21:20:32.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:32.948+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.950284+0000) 2020-02-05T21:20:32.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:32.947+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.948744+0000) 2020-02-05T21:20:32.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:32.949+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.951156+0000) 2020-02-05T21:20:32.955 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:32.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 82987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:32.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:32.955+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:32.956643+0000) 2020-02-05T21:20:33.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:33.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:33.464482+0000) 2020-02-05T21:20:33.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:33.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:33.493136+0000) 2020-02-05T21:20:35.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:35.837+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:35.837366+0000) 2020-02-05T21:20:36.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:36.704+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:36.705123+0000) 2020-02-05T21:20:36.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:36.734+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:36.734249+0000) 2020-02-05T21:20:36.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:36.978+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:36.979966+0000) 2020-02-05T21:20:36.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:36.978+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:36.980131+0000) 2020-02-05T21:20:36.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:36.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:36.991692+0000) 2020-02-05T21:20:37.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:37.959+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.959463+0000) 2020-02-05T21:20:37.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:37.960+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.960696+0000) 2020-02-05T21:20:37.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:37.961+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.961323+0000) 2020-02-05T21:20:37.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:37.962+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.962353+0000) 2020-02-05T21:20:37.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:37.960+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.961936+0000) 2020-02-05T21:20:37.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:37.962+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.962918+0000) 2020-02-05T21:20:37.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:37.963+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.963371+0000) 2020-02-05T21:20:37.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:37.962+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.963739+0000) 2020-02-05T21:20:37.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:37.963+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.964425+0000) 2020-02-05T21:20:37.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:37.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.965237+0000) 2020-02-05T21:20:37.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:37.963+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.964454+0000) 2020-02-05T21:20:37.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:37.964+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.966095+0000) 2020-02-05T21:20:37.968 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:37.965+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.967040+0000) 2020-02-05T21:20:37.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:37.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:37.970491+0000) 2020-02-05T21:20:37.972 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:37.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83029 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:39.090 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:39.088+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:39.088763+0000) 2020-02-05T21:20:39.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:39.373+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:39.373869+0000) 2020-02-05T21:20:39.784 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:20:39.790 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:20:39.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:39.982+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:39.982198+0000) 2020-02-05T21:20:40.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:40.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:40.233561+0000) 2020-02-05T21:20:42.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:42.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.222999+0000) 2020-02-05T21:20:42.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:42.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.624347+0000) 2020-02-05T21:20:42.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:42.976+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.977173+0000) 2020-02-05T21:20:42.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:42.976+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.977693+0000) 2020-02-05T21:20:42.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:42.977+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.978313+0000) 2020-02-05T21:20:42.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:42.977+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.978795+0000) 2020-02-05T21:20:42.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:42.977+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.978530+0000) 2020-02-05T21:20:42.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:42.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.979693+0000) 2020-02-05T21:20:42.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:42.979+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.980694+0000) 2020-02-05T21:20:42.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:42.979+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.980770+0000) 2020-02-05T21:20:42.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:42.979+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.981232+0000) 2020-02-05T21:20:42.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:42.980+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.981959+0000) 2020-02-05T21:20:42.986 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:42.982+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.983498+0000) 2020-02-05T21:20:42.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:42.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.984687+0000) 2020-02-05T21:20:42.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:42.980+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.982356+0000) 2020-02-05T21:20:42.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:42.981+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:42.983403+0000) 2020-02-05T21:20:42.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:42.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:43.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:43.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:43.464762+0000) 2020-02-05T21:20:43.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:43.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:43.493339+0000) 2020-02-05T21:20:45.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:45.836+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:45.837620+0000) 2020-02-05T21:20:46.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:46.712+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:46.713569+0000) 2020-02-05T21:20:46.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:46.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:46.734478+0000) 2020-02-05T21:20:46.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:46.979+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:46.980203+0000) 2020-02-05T21:20:46.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:46.979+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:46.980340+0000) 2020-02-05T21:20:46.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:46.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:46.991926+0000) 2020-02-05T21:20:47.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:47.992+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.993106+0000) 2020-02-05T21:20:47.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:47.993+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.994068+0000) 2020-02-05T21:20:47.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:47.995+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.996595+0000) 2020-02-05T21:20:47.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:47.996+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.997229+0000) 2020-02-05T21:20:47.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:47.996+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.997436+0000) 2020-02-05T21:20:47.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:47.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.997580+0000) 2020-02-05T21:20:48.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:47.997+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.998032+0000) 2020-02-05T21:20:48.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:47.998+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.998685+0000) 2020-02-05T21:20:48.000 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:47.997+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.997844+0000) 2020-02-05T21:20:48.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:47.998+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.998722+0000) 2020-02-05T21:20:48.001 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:47.998+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.999446+0000) 2020-02-05T21:20:48.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:47.998+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.999702+0000) 2020-02-05T21:20:48.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:47.999+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:47.999744+0000) 2020-02-05T21:20:48.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:48.001+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:48.002579+0000) 2020-02-05T21:20:48.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:48.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83110 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:49.090 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:49.088+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:49.089011+0000) 2020-02-05T21:20:49.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:49.373+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:49.374128+0000) 2020-02-05T21:20:49.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:49.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:49.982456+0000) 2020-02-05T21:20:50.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:50.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:50.233807+0000) 2020-02-05T21:20:52.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:52.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:52.223195+0000) 2020-02-05T21:20:52.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:52.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:52.624622+0000) 2020-02-05T21:20:53.012 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:53.009+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.010232+0000) 2020-02-05T21:20:53.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:53.010+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.011196+0000) 2020-02-05T21:20:53.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:53.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.011974+0000) 2020-02-05T21:20:53.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:53.011+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.012008+0000) 2020-02-05T21:20:53.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:53.011+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.012457+0000) 2020-02-05T21:20:53.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:53.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.012856+0000) 2020-02-05T21:20:53.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:53.012+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.013018+0000) 2020-02-05T21:20:53.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:53.012+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.013192+0000) 2020-02-05T21:20:53.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:53.012+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.013519+0000) 2020-02-05T21:20:53.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:53.013+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.013609+0000) 2020-02-05T21:20:53.016 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:53.013+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.013896+0000) 2020-02-05T21:20:53.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:53.014+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.014842+0000) 2020-02-05T21:20:53.016 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:53.013+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.014344+0000) 2020-02-05T21:20:53.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:53.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.019222+0000) 2020-02-05T21:20:53.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:53.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:53.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:53.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.464973+0000) 2020-02-05T21:20:53.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:53.492+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:53.493567+0000) 2020-02-05T21:20:55.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:55.837+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:55.837877+0000) 2020-02-05T21:20:56.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:56.721+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:56.721993+0000) 2020-02-05T21:20:56.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:56.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:56.734729+0000) 2020-02-05T21:20:56.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:56.980+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:56.980468+0000) 2020-02-05T21:20:56.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:56.980+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:56.980648+0000) 2020-02-05T21:20:56.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:56.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:56.992164+0000) 2020-02-05T21:20:58.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:58.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.026634+0000) 2020-02-05T21:20:58.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:58.027+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.027407+0000) 2020-02-05T21:20:58.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:20:58.027+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.027751+0000) 2020-02-05T21:20:58.029 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:20:58.027+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.027283+0000) 2020-02-05T21:20:58.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:58.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.028808+0000) 2020-02-05T21:20:58.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:20:58.028+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.029255+0000) 2020-02-05T21:20:58.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:20:58.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.029850+0000) 2020-02-05T21:20:58.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:20:58.029+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.030388+0000) 2020-02-05T21:20:58.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:20:58.030+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.030772+0000) 2020-02-05T21:20:58.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:20:58.030+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.031419+0000) 2020-02-05T21:20:58.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:58.030+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.031677+0000) 2020-02-05T21:20:58.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:20:58.031+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.032055+0000) 2020-02-05T21:20:58.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:20:58.031+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.032062+0000) 2020-02-05T21:20:58.038 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:20:58.036+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:20:58.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:58.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:58.037822+0000) 2020-02-05T21:20:59.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:20:59.088+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:59.089278+0000) 2020-02-05T21:20:59.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:20:59.373+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:59.374320+0000) 2020-02-05T21:20:59.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:20:59.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:20:59.982687+0000) 2020-02-05T21:21:00.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:00.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:00.234044+0000) 2020-02-05T21:21:02.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:02.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:02.223484+0000) 2020-02-05T21:21:02.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:02.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:02.624851+0000) 2020-02-05T21:21:03.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:03.041+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.042387+0000) 2020-02-05T21:21:03.045 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:03.042+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.043353+0000) 2020-02-05T21:21:03.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:03.043+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.044062+0000) 2020-02-05T21:21:03.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:03.043+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.044722+0000) 2020-02-05T21:21:03.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:03.044+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.045281+0000) 2020-02-05T21:21:03.046 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:03.044+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.045505+0000) 2020-02-05T21:21:03.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:03.045+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.045659+0000) 2020-02-05T21:21:03.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:03.045+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.046176+0000) 2020-02-05T21:21:03.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:03.045+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.046493+0000) 2020-02-05T21:21:03.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:03.045+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.046529+0000) 2020-02-05T21:21:03.048 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:03.046+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.046610+0000) 2020-02-05T21:21:03.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:03.046+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.046826+0000) 2020-02-05T21:21:03.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:03.046+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.047240+0000) 2020-02-05T21:21:03.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:03.051+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.052286+0000) 2020-02-05T21:21:03.054 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:03.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:03.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:03.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.465204+0000) 2020-02-05T21:21:03.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:03.493+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:03.493811+0000) 2020-02-05T21:21:05.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:05.837+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:05.838135+0000) 2020-02-05T21:21:06.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:06.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:06.730122+0000) 2020-02-05T21:21:06.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:06.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:06.734950+0000) 2020-02-05T21:21:06.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:06.980+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:06.980821+0000) 2020-02-05T21:21:06.982 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:06.980+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:06.980892+0000) 2020-02-05T21:21:06.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:06.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:06.992433+0000) 2020-02-05T21:21:08.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:08.058+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.059528+0000) 2020-02-05T21:21:08.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:08.059+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.059878+0000) 2020-02-05T21:21:08.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:08.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.060345+0000) 2020-02-05T21:21:08.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:08.060+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.060697+0000) 2020-02-05T21:21:08.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:08.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.061345+0000) 2020-02-05T21:21:08.063 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:08.060+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.061745+0000) 2020-02-05T21:21:08.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:08.061+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.062242+0000) 2020-02-05T21:21:08.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:08.062+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.062584+0000) 2020-02-05T21:21:08.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:08.061+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.062920+0000) 2020-02-05T21:21:08.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:08.063+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.063418+0000) 2020-02-05T21:21:08.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:08.062+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.063692+0000) 2020-02-05T21:21:08.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:08.063+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.064128+0000) 2020-02-05T21:21:08.066 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:08.064+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.065191+0000) 2020-02-05T21:21:08.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:08.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:08.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:08.070+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:08.071842+0000) 2020-02-05T21:21:09.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:09.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:09.089518+0000) 2020-02-05T21:21:09.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:09.373+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:09.374534+0000) 2020-02-05T21:21:09.964 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:21:09.970 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:21:09.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:09.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:09.982875+0000) 2020-02-05T21:21:10.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:10.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:10.234290+0000) 2020-02-05T21:21:12.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:12.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:12.223745+0000) 2020-02-05T21:21:12.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:12.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:12.625124+0000) 2020-02-05T21:21:13.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:13.074+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.075313+0000) 2020-02-05T21:21:13.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:13.074+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.075808+0000) 2020-02-05T21:21:13.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:13.075+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.075740+0000) 2020-02-05T21:21:13.078 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:13.075+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.076422+0000) 2020-02-05T21:21:13.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:13.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.077129+0000) 2020-02-05T21:21:13.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:13.076+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.077341+0000) 2020-02-05T21:21:13.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:13.077+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.078104+0000) 2020-02-05T21:21:13.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:13.078+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.079457+0000) 2020-02-05T21:21:13.081 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:13.078+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.079553+0000) 2020-02-05T21:21:13.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:13.079+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.080120+0000) 2020-02-05T21:21:13.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:13.079+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.080301+0000) 2020-02-05T21:21:13.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:13.080+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.080649+0000) 2020-02-05T21:21:13.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:13.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.080564+0000) 2020-02-05T21:21:13.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:13.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83317 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:13.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:13.085+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.086285+0000) 2020-02-05T21:21:13.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:13.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.465480+0000) 2020-02-05T21:21:13.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:13.493+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:13.494021+0000) 2020-02-05T21:21:15.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:15.837+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:15.838364+0000) 2020-02-05T21:21:16.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:16.734+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:16.735190+0000) 2020-02-05T21:21:16.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:16.739+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:16.740210+0000) 2020-02-05T21:21:16.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:16.980+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:16.981070+0000) 2020-02-05T21:21:16.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:16.980+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:16.981154+0000) 2020-02-05T21:21:16.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:16.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:16.992661+0000) 2020-02-05T21:21:18.093 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:18.088+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.089638+0000) 2020-02-05T21:21:18.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:18.091+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.091684+0000) 2020-02-05T21:21:18.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:18.089+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.090489+0000) 2020-02-05T21:21:18.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:18.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.092036+0000) 2020-02-05T21:21:18.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:18.092+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.092884+0000) 2020-02-05T21:21:18.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:18.093+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.093842+0000) 2020-02-05T21:21:18.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:18.093+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.094529+0000) 2020-02-05T21:21:18.097 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:18.093+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.094619+0000) 2020-02-05T21:21:18.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:18.094+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.095505+0000) 2020-02-05T21:21:18.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:18.095+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.095896+0000) 2020-02-05T21:21:18.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:18.096+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.096674+0000) 2020-02-05T21:21:18.099 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:18.096+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.097070+0000) 2020-02-05T21:21:18.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:18.095+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.096881+0000) 2020-02-05T21:21:18.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:18.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:18.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:18.102+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:18.103500+0000) 2020-02-05T21:21:19.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:19.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:19.089751+0000) 2020-02-05T21:21:19.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:19.374+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:19.374792+0000) 2020-02-05T21:21:19.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:19.982+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:19.983137+0000) 2020-02-05T21:21:20.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:20.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:20.234543+0000) 2020-02-05T21:21:22.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:22.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:22.224005+0000) 2020-02-05T21:21:22.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:22.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:22.625413+0000) 2020-02-05T21:21:23.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:23.107+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.107718+0000) 2020-02-05T21:21:23.110 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:23.107+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.108222+0000) 2020-02-05T21:21:23.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:23.107+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.108688+0000) 2020-02-05T21:21:23.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:23.108+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.108936+0000) 2020-02-05T21:21:23.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:23.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.110315+0000) 2020-02-05T21:21:23.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:23.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.110611+0000) 2020-02-05T21:21:23.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:23.110+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.111044+0000) 2020-02-05T21:21:23.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:23.110+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.111492+0000) 2020-02-05T21:21:23.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:23.111+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.111879+0000) 2020-02-05T21:21:23.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:23.111+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.112232+0000) 2020-02-05T21:21:23.114 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:23.111+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.112894+0000) 2020-02-05T21:21:23.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:23.112+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.113391+0000) 2020-02-05T21:21:23.115 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:23.113+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.113586+0000) 2020-02-05T21:21:23.119 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:23.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:23.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:23.118+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.119309+0000) 2020-02-05T21:21:23.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:23.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.465698+0000) 2020-02-05T21:21:23.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:23.493+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:23.494256+0000) 2020-02-05T21:21:25.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:25.838+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:25.838611+0000) 2020-02-05T21:21:26.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:26.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:26.735464+0000) 2020-02-05T21:21:26.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:26.751+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:26.752109+0000) 2020-02-05T21:21:26.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:26.980+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:26.981328+0000) 2020-02-05T21:21:26.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:26.980+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:26.981408+0000) 2020-02-05T21:21:26.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:26.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:26.992898+0000) 2020-02-05T21:21:28.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:28.123+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.124066+0000) 2020-02-05T21:21:28.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:28.123+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.124589+0000) 2020-02-05T21:21:28.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:28.123+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.125083+0000) 2020-02-05T21:21:28.127 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:28.124+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.125872+0000) 2020-02-05T21:21:28.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:28.124+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.126233+0000) 2020-02-05T21:21:28.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:28.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.126813+0000) 2020-02-05T21:21:28.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:28.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.127588+0000) 2020-02-05T21:21:28.129 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:28.127+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.128142+0000) 2020-02-05T21:21:28.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:28.127+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.128923+0000) 2020-02-05T21:21:28.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:28.127+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.128731+0000) 2020-02-05T21:21:28.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:28.127+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.129188+0000) 2020-02-05T21:21:28.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:28.129+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.130504+0000) 2020-02-05T21:21:28.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:28.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.133601+0000) 2020-02-05T21:21:28.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:28.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:28.133853+0000) 2020-02-05T21:21:28.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:28.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:29.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:29.088+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:29.089982+0000) 2020-02-05T21:21:29.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:29.373+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:29.375041+0000) 2020-02-05T21:21:29.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:29.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:29.983368+0000) 2020-02-05T21:21:30.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:30.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:30.234792+0000) 2020-02-05T21:21:32.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:32.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:32.224252+0000) 2020-02-05T21:21:32.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:32.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:32.625642+0000) 2020-02-05T21:21:33.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:33.138+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.140074+0000) 2020-02-05T21:21:33.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:33.140+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.140658+0000) 2020-02-05T21:21:33.143 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:33.141+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.141346+0000) 2020-02-05T21:21:33.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:33.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.141848+0000) 2020-02-05T21:21:33.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:33.141+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.142495+0000) 2020-02-05T21:21:33.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:33.141+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.142667+0000) 2020-02-05T21:21:33.145 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:33.143+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.143750+0000) 2020-02-05T21:21:33.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:33.142+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.143518+0000) 2020-02-05T21:21:33.145 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:33.142+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.143719+0000) 2020-02-05T21:21:33.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:33.145+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.145336+0000) 2020-02-05T21:21:33.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:33.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.145230+0000) 2020-02-05T21:21:33.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:33.143+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.145132+0000) 2020-02-05T21:21:33.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:33.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.146055+0000) 2020-02-05T21:21:33.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:33.148+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.148118+0000) 2020-02-05T21:21:33.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:33.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:33.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:33.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.465901+0000) 2020-02-05T21:21:33.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:33.494+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:33.494482+0000) 2020-02-05T21:21:35.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:35.838+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:35.838859+0000) 2020-02-05T21:21:36.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:36.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:36.735695+0000) 2020-02-05T21:21:36.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:36.759+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:36.759754+0000) 2020-02-05T21:21:36.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:36.980+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:36.981575+0000) 2020-02-05T21:21:36.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:36.980+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:36.981728+0000) 2020-02-05T21:21:36.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:36.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:36.993214+0000) 2020-02-05T21:21:38.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:38.156+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.157652+0000) 2020-02-05T21:21:38.159 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:38.156+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.157866+0000) 2020-02-05T21:21:38.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:38.156+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.158151+0000) 2020-02-05T21:21:38.160 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:38.157+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.158931+0000) 2020-02-05T21:21:38.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:38.158+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.159683+0000) 2020-02-05T21:21:38.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:38.158+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.160026+0000) 2020-02-05T21:21:38.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:38.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.160550+0000) 2020-02-05T21:21:38.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:38.160+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.161072+0000) 2020-02-05T21:21:38.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:38.160+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.161403+0000) 2020-02-05T21:21:38.163 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:38.161+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.162308+0000) 2020-02-05T21:21:38.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:38.160+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.161981+0000) 2020-02-05T21:21:38.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:38.161+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.162509+0000) 2020-02-05T21:21:38.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:38.161+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.162832+0000) 2020-02-05T21:21:38.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:38.166+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:38.167090+0000) 2020-02-05T21:21:38.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:38.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:39.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:39.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:39.090211+0000) 2020-02-05T21:21:39.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:39.374+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:39.375329+0000) 2020-02-05T21:21:39.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:39.982+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:39.983601+0000) 2020-02-05T21:21:40.151 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:21:40.158 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:21:40.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:40.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:40.235003+0000) 2020-02-05T21:21:42.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:42.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:42.224526+0000) 2020-02-05T21:21:42.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:42.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:42.625874+0000) 2020-02-05T21:21:43.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:43.173+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.174541+0000) 2020-02-05T21:21:43.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:43.174+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.175270+0000) 2020-02-05T21:21:43.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:43.174+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.176079+0000) 2020-02-05T21:21:43.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:43.175+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.176699+0000) 2020-02-05T21:21:43.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:43.176+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.177409+0000) 2020-02-05T21:21:43.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:43.176+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.177574+0000) 2020-02-05T21:21:43.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:43.176+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.178013+0000) 2020-02-05T21:21:43.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:43.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.178597+0000) 2020-02-05T21:21:43.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:43.178+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.179593+0000) 2020-02-05T21:21:43.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:43.178+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.179621+0000) 2020-02-05T21:21:43.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:43.178+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.179864+0000) 2020-02-05T21:21:43.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:43.179+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.180415+0000) 2020-02-05T21:21:43.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:43.179+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.181023+0000) 2020-02-05T21:21:43.186 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:43.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:43.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:43.184+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.185562+0000) 2020-02-05T21:21:43.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:43.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.466130+0000) 2020-02-05T21:21:43.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:43.493+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:43.494729+0000) 2020-02-05T21:21:45.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:45.838+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:45.839118+0000) 2020-02-05T21:21:46.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:46.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:46.735927+0000) 2020-02-05T21:21:46.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:46.767+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:46.768054+0000) 2020-02-05T21:21:46.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:46.981+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:46.981825+0000) 2020-02-05T21:21:46.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:46.981+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:46.981954+0000) 2020-02-05T21:21:46.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:46.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:46.993445+0000) 2020-02-05T21:21:48.192 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:48.189+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.190957+0000) 2020-02-05T21:21:48.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:48.190+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.191684+0000) 2020-02-05T21:21:48.194 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:48.191+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.192606+0000) 2020-02-05T21:21:48.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:48.192+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.193062+0000) 2020-02-05T21:21:48.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:48.191+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.193078+0000) 2020-02-05T21:21:48.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:48.192+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.193949+0000) 2020-02-05T21:21:48.196 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:48.193+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.194703+0000) 2020-02-05T21:21:48.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:48.193+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.194589+0000) 2020-02-05T21:21:48.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:48.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.195156+0000) 2020-02-05T21:21:48.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:48.194+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.195749+0000) 2020-02-05T21:21:48.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:48.195+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.196347+0000) 2020-02-05T21:21:48.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:48.195+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.196500+0000) 2020-02-05T21:21:48.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:48.195+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.196837+0000) 2020-02-05T21:21:48.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:48.199+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:48.200506+0000) 2020-02-05T21:21:48.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:48.199+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83607 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:49.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:49.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:49.090414+0000) 2020-02-05T21:21:49.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:49.374+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:49.375565+0000) 2020-02-05T21:21:49.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:49.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:49.983837+0000) 2020-02-05T21:21:50.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:50.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:50.235243+0000) 2020-02-05T21:21:52.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:52.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:52.224764+0000) 2020-02-05T21:21:52.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:52.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:52.626145+0000) 2020-02-05T21:21:53.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:53.205+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.206789+0000) 2020-02-05T21:21:53.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:53.206+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.208173+0000) 2020-02-05T21:21:53.210 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:53.207+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.208857+0000) 2020-02-05T21:21:53.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:53.207+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.209184+0000) 2020-02-05T21:21:53.211 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:53.208+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.209315+0000) 2020-02-05T21:21:53.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:53.208+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.209764+0000) 2020-02-05T21:21:53.212 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:53.209+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.210422+0000) 2020-02-05T21:21:53.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:53.208+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.210269+0000) 2020-02-05T21:21:53.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:53.209+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.210666+0000) 2020-02-05T21:21:53.213 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:53.210+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.211216+0000) 2020-02-05T21:21:53.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:53.209+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.211290+0000) 2020-02-05T21:21:53.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:53.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.212263+0000) 2020-02-05T21:21:53.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:53.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.212862+0000) 2020-02-05T21:21:53.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:53.214+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.215689+0000) 2020-02-05T21:21:53.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:53.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83649 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:53.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:53.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.466494+0000) 2020-02-05T21:21:53.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:53.494+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:53.494943+0000) 2020-02-05T21:21:55.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:55.838+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:55.839355+0000) 2020-02-05T21:21:56.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:56.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:56.736168+0000) 2020-02-05T21:21:56.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:56.775+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:56.776367+0000) 2020-02-05T21:21:56.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:56.981+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:56.982063+0000) 2020-02-05T21:21:56.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:56.981+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:56.982178+0000) 2020-02-05T21:21:56.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:56.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:56.993687+0000) 2020-02-05T21:21:58.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:58.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.223648+0000) 2020-02-05T21:21:58.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:21:58.223+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.224159+0000) 2020-02-05T21:21:58.226 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:58.224+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.225249+0000) 2020-02-05T21:21:58.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:21:58.224+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.225809+0000) 2020-02-05T21:21:58.227 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:21:58.225+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.226539+0000) 2020-02-05T21:21:58.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:21:58.224+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.226343+0000) 2020-02-05T21:21:58.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:21:58.226+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.227495+0000) 2020-02-05T21:21:58.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:58.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.227837+0000) 2020-02-05T21:21:58.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:21:58.226+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.228143+0000) 2020-02-05T21:21:58.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:21:58.227+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.228826+0000) 2020-02-05T21:21:58.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:58.227+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.228657+0000) 2020-02-05T21:21:58.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:21:58.227+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.229213+0000) 2020-02-05T21:21:58.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:21:58.227+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.229329+0000) 2020-02-05T21:21:58.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:21:58.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:21:58.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:58.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:58.235895+0000) 2020-02-05T21:21:59.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:21:59.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:59.090661+0000) 2020-02-05T21:21:59.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:21:59.374+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:59.375816+0000) 2020-02-05T21:21:59.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:21:59.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:21:59.984064+0000) 2020-02-05T21:22:00.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:00.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:00.235473+0000) 2020-02-05T21:22:02.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:02.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:02.225013+0000) 2020-02-05T21:22:02.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:02.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:02.626405+0000) 2020-02-05T21:22:03.239 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:03.237+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.238356+0000) 2020-02-05T21:22:03.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:03.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.240558+0000) 2020-02-05T21:22:03.242 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:03.240+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.241163+0000) 2020-02-05T21:22:03.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:03.240+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.241936+0000) 2020-02-05T21:22:03.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:03.240+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.242130+0000) 2020-02-05T21:22:03.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:03.242+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.243225+0000) 2020-02-05T21:22:03.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:03.241+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.242973+0000) 2020-02-05T21:22:03.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:03.242+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.243768+0000) 2020-02-05T21:22:03.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:03.242+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.243720+0000) 2020-02-05T21:22:03.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:03.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.243965+0000) 2020-02-05T21:22:03.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:03.244+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.245502+0000) 2020-02-05T21:22:03.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:03.244+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.245458+0000) 2020-02-05T21:22:03.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:03.243+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.245228+0000) 2020-02-05T21:22:03.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:03.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:03.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:03.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.252114+0000) 2020-02-05T21:22:03.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:03.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.466736+0000) 2020-02-05T21:22:03.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:03.494+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:03.495153+0000) 2020-02-05T21:22:05.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:05.839+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:05.839587+0000) 2020-02-05T21:22:06.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:06.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:06.736400+0000) 2020-02-05T21:22:06.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:06.783+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:06.783821+0000) 2020-02-05T21:22:06.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:06.981+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:06.982317+0000) 2020-02-05T21:22:06.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:06.981+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:06.982482+0000) 2020-02-05T21:22:06.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:06.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:06.993923+0000) 2020-02-05T21:22:08.256 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:08.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.255397+0000) 2020-02-05T21:22:08.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:08.254+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.256050+0000) 2020-02-05T21:22:08.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:08.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.257378+0000) 2020-02-05T21:22:08.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:08.257+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.258140+0000) 2020-02-05T21:22:08.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:08.257+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.258268+0000) 2020-02-05T21:22:08.260 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:08.257+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.258882+0000) 2020-02-05T21:22:08.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:08.257+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.259132+0000) 2020-02-05T21:22:08.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:08.258+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.259567+0000) 2020-02-05T21:22:08.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:08.258+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.260244+0000) 2020-02-05T21:22:08.262 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:08.259+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.260788+0000) 2020-02-05T21:22:08.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:08.259+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.260759+0000) 2020-02-05T21:22:08.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:08.259+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.261107+0000) 2020-02-05T21:22:08.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:08.260+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.261931+0000) 2020-02-05T21:22:08.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:08.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:08.262036+0000) 2020-02-05T21:22:08.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:08.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:09.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:09.089+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:09.090903+0000) 2020-02-05T21:22:09.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:09.375+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:09.376072+0000) 2020-02-05T21:22:09.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:09.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:09.984372+0000) 2020-02-05T21:22:10.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:10.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:10.235701+0000) 2020-02-05T21:22:10.381 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:22:10.388 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:22:12.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:12.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:12.225264+0000) 2020-02-05T21:22:12.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:12.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:12.626658+0000) 2020-02-05T21:22:13.273 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:13.271+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.272133+0000) 2020-02-05T21:22:13.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:13.272+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.273477+0000) 2020-02-05T21:22:13.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:13.272+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.273752+0000) 2020-02-05T21:22:13.276 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:13.273+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.274528+0000) 2020-02-05T21:22:13.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:13.273+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.275069+0000) 2020-02-05T21:22:13.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:13.274+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.275399+0000) 2020-02-05T21:22:13.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:13.274+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.276040+0000) 2020-02-05T21:22:13.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:13.274+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.276205+0000) 2020-02-05T21:22:13.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:13.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.276815+0000) 2020-02-05T21:22:13.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:13.276+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.277140+0000) 2020-02-05T21:22:13.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:13.276+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.277641+0000) 2020-02-05T21:22:13.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:13.276+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.277946+0000) 2020-02-05T21:22:13.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:13.277+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.278740+0000) 2020-02-05T21:22:13.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:13.281+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:13.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:13.281+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.282934+0000) 2020-02-05T21:22:13.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:13.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.466972+0000) 2020-02-05T21:22:13.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:13.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:13.495387+0000) 2020-02-05T21:22:15.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:15.839+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:15.839811+0000) 2020-02-05T21:22:16.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:16.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:16.736628+0000) 2020-02-05T21:22:16.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:16.789+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:16.791059+0000) 2020-02-05T21:22:16.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:16.982+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:16.982568+0000) 2020-02-05T21:22:16.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:16.982+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:16.982743+0000) 2020-02-05T21:22:16.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:16.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:16.994146+0000) 2020-02-05T21:22:18.289 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:18.287+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.288283+0000) 2020-02-05T21:22:18.290 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:18.287+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.288522+0000) 2020-02-05T21:22:18.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:18.287+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.289033+0000) 2020-02-05T21:22:18.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:18.289+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.289604+0000) 2020-02-05T21:22:18.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:18.289+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.290159+0000) 2020-02-05T21:22:18.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:18.290+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.291272+0000) 2020-02-05T21:22:18.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:18.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.292511+0000) 2020-02-05T21:22:18.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:18.291+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.293046+0000) 2020-02-05T21:22:18.295 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:18.292+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.293734+0000) 2020-02-05T21:22:18.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:18.293+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.294330+0000) 2020-02-05T21:22:18.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:18.293+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.294154+0000) 2020-02-05T21:22:18.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:18.294+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.295169+0000) 2020-02-05T21:22:18.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:18.294+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.295192+0000) 2020-02-05T21:22:18.300 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:18.298+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83856 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:18.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:18.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:18.299779+0000) 2020-02-05T21:22:19.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:19.090+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:19.091135+0000) 2020-02-05T21:22:19.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:19.375+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:19.376290+0000) 2020-02-05T21:22:19.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:19.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:19.984649+0000) 2020-02-05T21:22:20.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:20.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:20.235968+0000) 2020-02-05T21:22:22.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:22.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:22.225511+0000) 2020-02-05T21:22:22.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:22.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:22.626893+0000) 2020-02-05T21:22:23.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:23.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.305847+0000) 2020-02-05T21:22:23.308 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:23.305+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.306937+0000) 2020-02-05T21:22:23.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:23.306+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.307314+0000) 2020-02-05T21:22:23.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:23.307+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.307603+0000) 2020-02-05T21:22:23.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:23.307+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.308541+0000) 2020-02-05T21:22:23.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:23.308+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.308576+0000) 2020-02-05T21:22:23.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:23.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.309653+0000) 2020-02-05T21:22:23.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:23.309+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.310310+0000) 2020-02-05T21:22:23.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:23.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.310946+0000) 2020-02-05T21:22:23.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:23.310+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.310758+0000) 2020-02-05T21:22:23.312 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:23.310+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.311435+0000) 2020-02-05T21:22:23.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:23.311+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.311682+0000) 2020-02-05T21:22:23.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:23.311+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.312118+0000) 2020-02-05T21:22:23.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:23.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.314815+0000) 2020-02-05T21:22:23.317 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:23.315+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:23.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:23.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.467181+0000) 2020-02-05T21:22:23.497 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:23.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:23.495606+0000) 2020-02-05T21:22:25.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:25.839+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:25.840044+0000) 2020-02-05T21:22:26.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:26.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:26.736859+0000) 2020-02-05T21:22:26.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:26.799+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:26.799518+0000) 2020-02-05T21:22:26.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:26.982+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:26.982822+0000) 2020-02-05T21:22:26.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:26.982+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:26.982980+0000) 2020-02-05T21:22:26.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:26.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:26.994404+0000) 2020-02-05T21:22:28.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:28.322+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.322189+0000) 2020-02-05T21:22:28.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:28.322+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.323010+0000) 2020-02-05T21:22:28.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:28.322+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.323294+0000) 2020-02-05T21:22:28.325 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:28.324+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.324223+0000) 2020-02-05T21:22:28.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:28.324+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.324752+0000) 2020-02-05T21:22:28.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:28.324+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.324980+0000) 2020-02-05T21:22:28.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:28.324+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.325289+0000) 2020-02-05T21:22:28.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:28.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.326115+0000) 2020-02-05T21:22:28.327 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:28.326+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.326240+0000) 2020-02-05T21:22:28.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:28.326+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.327211+0000) 2020-02-05T21:22:28.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:28.327+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.327595+0000) 2020-02-05T21:22:28.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:28.327+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.327736+0000) 2020-02-05T21:22:28.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:28.327+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.327711+0000) 2020-02-05T21:22:28.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:28.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:28.333142+0000) 2020-02-05T21:22:28.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:28.331+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:29.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:29.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:29.091406+0000) 2020-02-05T21:22:29.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:29.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:29.376511+0000) 2020-02-05T21:22:29.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:29.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:29.984881+0000) 2020-02-05T21:22:30.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:30.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:30.236230+0000) 2020-02-05T21:22:32.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:32.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:32.225760+0000) 2020-02-05T21:22:32.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:32.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:32.627133+0000) 2020-02-05T21:22:33.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:33.336+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.336965+0000) 2020-02-05T21:22:33.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:33.337+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.338125+0000) 2020-02-05T21:22:33.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:33.337+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.338993+0000) 2020-02-05T21:22:33.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:33.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.340510+0000) 2020-02-05T21:22:33.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:33.340+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.341370+0000) 2020-02-05T21:22:33.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:33.340+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.341132+0000) 2020-02-05T21:22:33.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:33.340+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.342055+0000) 2020-02-05T21:22:33.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:33.341+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.342899+0000) 2020-02-05T21:22:33.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:33.342+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.343321+0000) 2020-02-05T21:22:33.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:33.342+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.342530+0000) 2020-02-05T21:22:33.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:33.343+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.343557+0000) 2020-02-05T21:22:33.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:33.343+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.344012+0000) 2020-02-05T21:22:33.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:33.343+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.344328+0000) 2020-02-05T21:22:33.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:33.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.349957+0000) 2020-02-05T21:22:33.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:33.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 83981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:33.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:33.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.467384+0000) 2020-02-05T21:22:33.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:33.494+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:33.495852+0000) 2020-02-05T21:22:35.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:35.839+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:35.840342+0000) 2020-02-05T21:22:36.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:36.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:36.737144+0000) 2020-02-05T21:22:36.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:36.807+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:36.807993+0000) 2020-02-05T21:22:36.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:36.982+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:36.983105+0000) 2020-02-05T21:22:36.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:36.982+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:36.983263+0000) 2020-02-05T21:22:36.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:36.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:36.994634+0000) 2020-02-05T21:22:38.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:38.353+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.354219+0000) 2020-02-05T21:22:38.356 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:38.354+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.354804+0000) 2020-02-05T21:22:38.356 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:38.354+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.355368+0000) 2020-02-05T21:22:38.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:38.355+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.356256+0000) 2020-02-05T21:22:38.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:38.355+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.356632+0000) 2020-02-05T21:22:38.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:38.355+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.356907+0000) 2020-02-05T21:22:38.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:38.356+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.357032+0000) 2020-02-05T21:22:38.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:38.357+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.358037+0000) 2020-02-05T21:22:38.360 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:38.357+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.358119+0000) 2020-02-05T21:22:38.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:38.357+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.357848+0000) 2020-02-05T21:22:38.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:38.358+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.358765+0000) 2020-02-05T21:22:38.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:38.358+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.359234+0000) 2020-02-05T21:22:38.361 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:38.358+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.359131+0000) 2020-02-05T21:22:38.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:38.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:38.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:38.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:38.364887+0000) 2020-02-05T21:22:39.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:39.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:39.091730+0000) 2020-02-05T21:22:39.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:39.375+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:39.376763+0000) 2020-02-05T21:22:39.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:39.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:39.985115+0000) 2020-02-05T21:22:40.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:40.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:40.236490+0000) 2020-02-05T21:22:40.516 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:22:40.522 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:22:42.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:42.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:42.226017+0000) 2020-02-05T21:22:42.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:42.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:42.627349+0000) 2020-02-05T21:22:43.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:43.368+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.369257+0000) 2020-02-05T21:22:43.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:43.368+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.369694+0000) 2020-02-05T21:22:43.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:43.369+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.370210+0000) 2020-02-05T21:22:43.372 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:43.370+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.371111+0000) 2020-02-05T21:22:43.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:43.370+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.371889+0000) 2020-02-05T21:22:43.373 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:43.371+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.372250+0000) 2020-02-05T21:22:43.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:43.371+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.372905+0000) 2020-02-05T21:22:43.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:43.372+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.372877+0000) 2020-02-05T21:22:43.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:43.372+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.373316+0000) 2020-02-05T21:22:43.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:43.374+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.374531+0000) 2020-02-05T21:22:43.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:43.373+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.374830+0000) 2020-02-05T21:22:43.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:43.374+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.374937+0000) 2020-02-05T21:22:43.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:43.374+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.375327+0000) 2020-02-05T21:22:43.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:43.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84064 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:43.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:43.378+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.379789+0000) 2020-02-05T21:22:43.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:43.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.467637+0000) 2020-02-05T21:22:43.497 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:43.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:43.496063+0000) 2020-02-05T21:22:45.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:45.839+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:45.840574+0000) 2020-02-05T21:22:46.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:46.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:46.737360+0000) 2020-02-05T21:22:46.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:46.817+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:46.817833+0000) 2020-02-05T21:22:46.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:46.983+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:46.983341+0000) 2020-02-05T21:22:46.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:46.983+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:46.983550+0000) 2020-02-05T21:22:46.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:46.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:46.994866+0000) 2020-02-05T21:22:48.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:48.383+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.384372+0000) 2020-02-05T21:22:48.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:48.384+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.385737+0000) 2020-02-05T21:22:48.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:48.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.387023+0000) 2020-02-05T21:22:48.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:48.386+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.387282+0000) 2020-02-05T21:22:48.389 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:48.386+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.387952+0000) 2020-02-05T21:22:48.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:48.387+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388216+0000) 2020-02-05T21:22:48.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:48.387+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388174+0000) 2020-02-05T21:22:48.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:48.388+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388469+0000) 2020-02-05T21:22:48.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:48.388+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388626+0000) 2020-02-05T21:22:48.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:48.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388925+0000) 2020-02-05T21:22:48.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:48.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.388835+0000) 2020-02-05T21:22:48.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:48.389+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.389530+0000) 2020-02-05T21:22:48.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:48.389+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.389719+0000) 2020-02-05T21:22:48.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:48.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:48.394976+0000) 2020-02-05T21:22:48.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:48.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:49.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:49.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:49.091966+0000) 2020-02-05T21:22:49.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:49.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:49.376995+0000) 2020-02-05T21:22:49.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:49.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:49.985354+0000) 2020-02-05T21:22:50.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:50.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:50.236719+0000) 2020-02-05T21:22:52.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:52.226+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:52.226289+0000) 2020-02-05T21:22:52.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:52.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:52.627620+0000) 2020-02-05T21:22:53.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:53.399+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.400908+0000) 2020-02-05T21:22:53.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:53.401+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.401930+0000) 2020-02-05T21:22:53.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:53.401+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.402401+0000) 2020-02-05T21:22:53.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:53.402+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.402927+0000) 2020-02-05T21:22:53.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:53.402+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.403350+0000) 2020-02-05T21:22:53.410 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:53.403+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.405051+0000) 2020-02-05T21:22:53.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:53.404+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.405847+0000) 2020-02-05T21:22:53.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:53.405+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.406366+0000) 2020-02-05T21:22:53.411 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:53.405+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.406632+0000) 2020-02-05T21:22:53.411 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:53.403+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.403845+0000) 2020-02-05T21:22:53.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:53.404+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.404524+0000) 2020-02-05T21:22:53.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:53.406+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.406689+0000) 2020-02-05T21:22:53.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:53.406+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.407289+0000) 2020-02-05T21:22:53.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:53.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.411443+0000) 2020-02-05T21:22:53.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:53.411+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:53.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:53.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.467872+0000) 2020-02-05T21:22:53.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:53.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:53.496313+0000) 2020-02-05T21:22:55.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:55.840+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:55.840814+0000) 2020-02-05T21:22:56.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:56.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:56.737630+0000) 2020-02-05T21:22:56.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:56.825+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:56.826153+0000) 2020-02-05T21:22:56.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:56.982+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:56.983663+0000) 2020-02-05T21:22:56.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:56.982+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:56.983819+0000) 2020-02-05T21:22:56.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:56.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:56.995118+0000) 2020-02-05T21:22:58.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:22:58.415+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.417377+0000) 2020-02-05T21:22:58.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:58.417+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.418447+0000) 2020-02-05T21:22:58.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:22:58.417+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.418584+0000) 2020-02-05T21:22:58.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:58.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.418977+0000) 2020-02-05T21:22:58.421 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:22:58.418+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.419191+0000) 2020-02-05T21:22:58.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:22:58.418+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.419680+0000) 2020-02-05T21:22:58.422 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:58.419+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.420199+0000) 2020-02-05T21:22:58.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:22:58.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.420389+0000) 2020-02-05T21:22:58.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:22:58.419+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.420965+0000) 2020-02-05T21:22:58.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:22:58.419+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.420495+0000) 2020-02-05T21:22:58.424 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:22:58.420+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.421512+0000) 2020-02-05T21:22:58.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:58.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.422246+0000) 2020-02-05T21:22:58.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:22:58.419+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.420597+0000) 2020-02-05T21:22:58.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:22:58.426+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:22:58.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:58.427+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:58.428216+0000) 2020-02-05T21:22:59.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:22:59.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:59.092217+0000) 2020-02-05T21:22:59.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:22:59.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:59.377204+0000) 2020-02-05T21:22:59.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:22:59.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:22:59.985583+0000) 2020-02-05T21:23:00.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:00.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:00.236982+0000) 2020-02-05T21:23:02.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:02.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:02.226552+0000) 2020-02-05T21:23:02.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:02.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:02.627881+0000) 2020-02-05T21:23:03.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:03.433+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.434222+0000) 2020-02-05T21:23:03.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:03.433+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.434486+0000) 2020-02-05T21:23:03.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:03.433+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.435112+0000) 2020-02-05T21:23:03.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:03.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.435607+0000) 2020-02-05T21:23:03.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:03.434+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.435894+0000) 2020-02-05T21:23:03.437 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:03.435+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.436327+0000) 2020-02-05T21:23:03.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:03.435+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.436981+0000) 2020-02-05T21:23:03.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:03.436+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.437814+0000) 2020-02-05T21:23:03.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:03.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.438447+0000) 2020-02-05T21:23:03.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:03.438+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.439201+0000) 2020-02-05T21:23:03.441 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:03.437+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.439256+0000) 2020-02-05T21:23:03.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:03.437+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.439274+0000) 2020-02-05T21:23:03.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:03.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.439367+0000) 2020-02-05T21:23:03.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:03.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.442661+0000) 2020-02-05T21:23:03.445 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:03.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84230 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:03.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:03.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.468092+0000) 2020-02-05T21:23:03.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:03.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:03.496478+0000) 2020-02-05T21:23:05.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:05.840+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:05.841073+0000) 2020-02-05T21:23:06.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:06.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:06.737852+0000) 2020-02-05T21:23:06.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:06.834+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:06.834517+0000) 2020-02-05T21:23:06.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:06.983+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:06.983914+0000) 2020-02-05T21:23:06.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:06.983+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:06.984104+0000) 2020-02-05T21:23:06.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:06.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:06.995344+0000) 2020-02-05T21:23:08.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:08.449+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.450389+0000) 2020-02-05T21:23:08.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:08.451+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.452318+0000) 2020-02-05T21:23:08.454 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:08.450+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.452029+0000) 2020-02-05T21:23:08.454 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:08.451+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.452871+0000) 2020-02-05T21:23:08.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:08.452+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.453294+0000) 2020-02-05T21:23:08.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:08.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.453713+0000) 2020-02-05T21:23:08.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:08.452+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.454235+0000) 2020-02-05T21:23:08.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:08.452+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.454364+0000) 2020-02-05T21:23:08.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:08.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.455504+0000) 2020-02-05T21:23:08.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:08.453+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.455241+0000) 2020-02-05T21:23:08.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:08.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.456149+0000) 2020-02-05T21:23:08.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:08.454+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.455976+0000) 2020-02-05T21:23:08.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:08.454+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.456295+0000) 2020-02-05T21:23:08.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:08.459+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84271 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:08.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:08.459+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:08.460850+0000) 2020-02-05T21:23:09.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:09.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:09.092473+0000) 2020-02-05T21:23:09.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:09.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:09.377383+0000) 2020-02-05T21:23:09.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:09.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:09.985889+0000) 2020-02-05T21:23:10.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:10.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:10.237225+0000) 2020-02-05T21:23:10.648 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:23:10.655 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:23:12.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:12.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:12.226813+0000) 2020-02-05T21:23:12.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:12.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:12.628140+0000) 2020-02-05T21:23:13.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:13.467+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.468127+0000) 2020-02-05T21:23:13.470 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:13.467+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.468518+0000) 2020-02-05T21:23:13.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:13.467+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.468898+0000) 2020-02-05T21:23:13.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:13.468+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.469300+0000) 2020-02-05T21:23:13.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:13.468+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.470350+0000) 2020-02-05T21:23:13.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:13.469+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.470676+0000) 2020-02-05T21:23:13.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:13.470+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.471767+0000) 2020-02-05T21:23:13.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:13.470+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.472049+0000) 2020-02-05T21:23:13.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:13.471+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.472477+0000) 2020-02-05T21:23:13.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:13.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.473000+0000) 2020-02-05T21:23:13.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:13.471+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.472794+0000) 2020-02-05T21:23:13.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:13.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.473504+0000) 2020-02-05T21:23:13.478 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:13.475+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:13.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:13.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.477668+0000) 2020-02-05T21:23:13.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:13.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.478046+0000) 2020-02-05T21:23:13.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:13.495+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:13.496720+0000) 2020-02-05T21:23:15.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:15.841+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:15.841328+0000) 2020-02-05T21:23:16.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:16.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:16.738106+0000) 2020-02-05T21:23:16.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:16.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:16.842944+0000) 2020-02-05T21:23:16.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:16.983+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:16.984152+0000) 2020-02-05T21:23:16.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:16.983+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:16.984335+0000) 2020-02-05T21:23:16.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:16.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:16.995571+0000) 2020-02-05T21:23:18.482 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:18.480+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.481477+0000) 2020-02-05T21:23:18.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:18.482+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.483462+0000) 2020-02-05T21:23:18.485 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:18.483+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.484715+0000) 2020-02-05T21:23:18.486 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:18.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.485611+0000) 2020-02-05T21:23:18.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:18.484+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.486052+0000) 2020-02-05T21:23:18.487 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:18.484+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.486229+0000) 2020-02-05T21:23:18.488 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:18.485+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.486768+0000) 2020-02-05T21:23:18.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:18.486+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.487270+0000) 2020-02-05T21:23:18.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:18.486+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.487959+0000) 2020-02-05T21:23:18.489 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:18.486+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.487848+0000) 2020-02-05T21:23:18.490 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:18.487+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.488558+0000) 2020-02-05T21:23:18.490 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:18.487+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.489208+0000) 2020-02-05T21:23:18.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:18.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.490781+0000) 2020-02-05T21:23:18.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:18.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:18.491902+0000) 2020-02-05T21:23:18.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:18.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:19.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:19.091+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:19.092715+0000) 2020-02-05T21:23:19.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:19.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:19.377645+0000) 2020-02-05T21:23:19.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:19.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:19.986117+0000) 2020-02-05T21:23:20.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:20.218+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:20.218968+0000) 2020-02-05T21:23:20.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:20.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:20.237480+0000) 2020-02-05T21:23:22.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:22.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:22.227058+0000) 2020-02-05T21:23:22.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:22.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:22.628378+0000) 2020-02-05T21:23:23.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:23.498+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.498855+0000) 2020-02-05T21:23:23.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:23.500+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.501046+0000) 2020-02-05T21:23:23.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:23.500+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.502139+0000) 2020-02-05T21:23:23.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:23.503+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.503458+0000) 2020-02-05T21:23:23.505 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:23.501+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.502753+0000) 2020-02-05T21:23:23.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:23.501+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.503117+0000) 2020-02-05T21:23:23.505 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:23.501+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.503182+0000) 2020-02-05T21:23:23.506 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:23.502+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.503934+0000) 2020-02-05T21:23:23.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:23.504+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.504780+0000) 2020-02-05T21:23:23.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:23.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.505027+0000) 2020-02-05T21:23:23.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:23.505+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.505565+0000) 2020-02-05T21:23:23.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:23.502+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.504272+0000) 2020-02-05T21:23:23.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:23.505+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.505845+0000) 2020-02-05T21:23:23.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:23.508+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:23.508100+0000) 2020-02-05T21:23:23.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:23.509+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:25.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:25.841+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:25.841557+0000) 2020-02-05T21:23:26.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:26.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:26.738308+0000) 2020-02-05T21:23:26.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:26.850+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:26.851293+0000) 2020-02-05T21:23:26.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:26.983+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:26.984387+0000) 2020-02-05T21:23:26.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:26.983+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:26.984629+0000) 2020-02-05T21:23:26.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:26.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:26.995819+0000) 2020-02-05T21:23:28.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:28.515+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.516267+0000) 2020-02-05T21:23:28.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:28.517+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.518311+0000) 2020-02-05T21:23:28.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:28.518+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.519408+0000) 2020-02-05T21:23:28.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:28.518+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.519920+0000) 2020-02-05T21:23:28.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:28.518+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.520302+0000) 2020-02-05T21:23:28.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:28.518+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.519979+0000) 2020-02-05T21:23:28.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:28.519+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.520502+0000) 2020-02-05T21:23:28.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:28.520+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.521308+0000) 2020-02-05T21:23:28.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:28.519+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.521028+0000) 2020-02-05T21:23:28.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:28.520+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.521751+0000) 2020-02-05T21:23:28.525 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:28.520+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.522225+0000) 2020-02-05T21:23:28.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:28.521+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.522632+0000) 2020-02-05T21:23:28.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:28.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.522474+0000) 2020-02-05T21:23:28.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:28.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:28.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:28.527+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:28.528879+0000) 2020-02-05T21:23:29.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:29.092+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:29.092953+0000) 2020-02-05T21:23:29.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:29.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:29.377846+0000) 2020-02-05T21:23:29.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:29.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:29.986371+0000) 2020-02-05T21:23:30.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:30.218+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:30.219219+0000) 2020-02-05T21:23:30.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:30.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:30.237729+0000) 2020-02-05T21:23:30.248 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:30.246+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:30.247632+0000) 2020-02-05T21:23:32.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:32.226+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:32.227325+0000) 2020-02-05T21:23:32.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:32.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:32.628603+0000) 2020-02-05T21:23:33.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:33.532+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.533385+0000) 2020-02-05T21:23:33.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:33.532+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.533847+0000) 2020-02-05T21:23:33.536 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:33.533+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.534764+0000) 2020-02-05T21:23:33.537 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:33.534+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.535873+0000) 2020-02-05T21:23:33.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:33.535+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.537103+0000) 2020-02-05T21:23:33.538 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:33.535+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.536956+0000) 2020-02-05T21:23:33.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:33.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.538264+0000) 2020-02-05T21:23:33.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:33.536+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.538080+0000) 2020-02-05T21:23:33.540 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:33.537+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.538594+0000) 2020-02-05T21:23:33.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:33.537+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.538710+0000) 2020-02-05T21:23:33.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:33.537+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.539053+0000) 2020-02-05T21:23:33.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:33.537+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.538645+0000) 2020-02-05T21:23:33.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:33.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.540951+0000) 2020-02-05T21:23:33.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:33.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:33.542131+0000) 2020-02-05T21:23:33.545 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:33.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84477 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:35.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:35.840+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:35.841780+0000) 2020-02-05T21:23:36.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:36.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:36.738551+0000) 2020-02-05T21:23:36.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:36.861+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:36.862553+0000) 2020-02-05T21:23:36.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:36.984+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:36.984664+0000) 2020-02-05T21:23:36.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:36.984+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:36.984875+0000) 2020-02-05T21:23:36.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:36.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:36.996121+0000) 2020-02-05T21:23:38.551 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:38.549+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.550196+0000) 2020-02-05T21:23:38.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:38.550+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.551424+0000) 2020-02-05T21:23:38.553 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:38.550+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.551895+0000) 2020-02-05T21:23:38.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:38.551+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.552734+0000) 2020-02-05T21:23:38.554 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:38.551+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.553046+0000) 2020-02-05T21:23:38.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:38.553+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.554072+0000) 2020-02-05T21:23:38.555 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:38.553+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.554217+0000) 2020-02-05T21:23:38.556 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:38.552+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.553835+0000) 2020-02-05T21:23:38.556 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:38.553+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.554421+0000) 2020-02-05T21:23:38.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:38.554+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.555457+0000) 2020-02-05T21:23:38.557 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:38.554+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.556021+0000) 2020-02-05T21:23:38.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:38.555+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.556485+0000) 2020-02-05T21:23:38.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:38.555+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.556752+0000) 2020-02-05T21:23:38.562 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:38.559+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84518 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:38.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:38.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:38.562007+0000) 2020-02-05T21:23:39.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:39.092+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:39.093192+0000) 2020-02-05T21:23:39.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:39.376+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:39.378045+0000) 2020-02-05T21:23:39.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:39.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:39.986608+0000) 2020-02-05T21:23:40.221 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:40.218+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:40.219474+0000) 2020-02-05T21:23:40.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:40.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:40.237978+0000) 2020-02-05T21:23:40.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:40.246+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:40.247862+0000) 2020-02-05T21:23:40.787 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:23:40.793 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:23:42.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:42.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:42.227713+0000) 2020-02-05T21:23:42.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:42.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:42.628833+0000) 2020-02-05T21:23:43.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:43.566+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.567760+0000) 2020-02-05T21:23:43.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:43.568+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.568655+0000) 2020-02-05T21:23:43.571 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:43.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.569246+0000) 2020-02-05T21:23:43.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:43.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.569696+0000) 2020-02-05T21:23:43.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:43.568+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.570019+0000) 2020-02-05T21:23:43.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:43.569+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.570348+0000) 2020-02-05T21:23:43.572 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:43.569+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.570620+0000) 2020-02-05T21:23:43.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:43.570+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.571781+0000) 2020-02-05T21:23:43.573 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:43.570+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.571574+0000) 2020-02-05T21:23:43.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:43.571+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.572701+0000) 2020-02-05T21:23:43.574 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:43.571+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.572701+0000) 2020-02-05T21:23:43.574 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:43.571+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.572945+0000) 2020-02-05T21:23:43.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:43.572+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.573468+0000) 2020-02-05T21:23:43.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:43.577+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84559 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:43.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:43.579+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:43.580693+0000) 2020-02-05T21:23:45.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:45.841+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:45.841995+0000) 2020-02-05T21:23:46.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:46.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:46.738803+0000) 2020-02-05T21:23:46.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:46.870+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:46.870811+0000) 2020-02-05T21:23:46.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:46.984+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:46.984903+0000) 2020-02-05T21:23:46.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:46.984+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:46.985142+0000) 2020-02-05T21:23:46.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:46.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:46.996347+0000) 2020-02-05T21:23:48.586 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:48.583+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.584552+0000) 2020-02-05T21:23:48.586 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:48.583+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.584945+0000) 2020-02-05T21:23:48.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:48.585+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.585508+0000) 2020-02-05T21:23:48.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:48.585+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.586296+0000) 2020-02-05T21:23:48.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:48.586+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.587241+0000) 2020-02-05T21:23:48.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:48.586+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.587027+0000) 2020-02-05T21:23:48.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:48.586+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.587697+0000) 2020-02-05T21:23:48.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:48.587+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.588356+0000) 2020-02-05T21:23:48.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:48.588+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.588602+0000) 2020-02-05T21:23:48.590 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:48.587+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.588996+0000) 2020-02-05T21:23:48.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:48.589+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.589491+0000) 2020-02-05T21:23:48.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:48.589+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.590375+0000) 2020-02-05T21:23:48.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:48.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.590341+0000) 2020-02-05T21:23:48.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:48.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:48.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:48.597+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:48.598730+0000) 2020-02-05T21:23:49.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:49.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:49.093406+0000) 2020-02-05T21:23:49.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:49.377+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:49.378287+0000) 2020-02-05T21:23:49.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:49.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:49.986845+0000) 2020-02-05T21:23:50.221 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:50.219+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:50.219732+0000) 2020-02-05T21:23:50.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:50.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:50.238236+0000) 2020-02-05T21:23:50.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:50.247+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:50.248117+0000) 2020-02-05T21:23:52.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:52.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:52.227966+0000) 2020-02-05T21:23:52.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:52.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:52.629108+0000) 2020-02-05T21:23:53.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:53.598+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.599590+0000) 2020-02-05T21:23:53.601 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:53.599+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.600269+0000) 2020-02-05T21:23:53.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:53.600+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.601009+0000) 2020-02-05T21:23:53.603 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:53.601+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.601678+0000) 2020-02-05T21:23:53.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:53.601+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.602326+0000) 2020-02-05T21:23:53.604 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:53.602+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.602423+0000) 2020-02-05T21:23:53.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:53.601+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.603011+0000) 2020-02-05T21:23:53.605 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:53.602+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.603597+0000) 2020-02-05T21:23:53.605 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:53.603+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.604197+0000) 2020-02-05T21:23:53.607 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:53.604+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.605202+0000) 2020-02-05T21:23:53.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:53.605+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.606066+0000) 2020-02-05T21:23:53.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:53.605+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.606318+0000) 2020-02-05T21:23:53.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:53.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.608387+0000) 2020-02-05T21:23:53.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:53.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:53.608694+0000) 2020-02-05T21:23:53.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:53.611+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84642 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:55.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:55.841+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:55.842284+0000) 2020-02-05T21:23:56.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:56.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:56.739032+0000) 2020-02-05T21:23:56.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:56.879+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:56.880484+0000) 2020-02-05T21:23:56.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:56.984+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:56.985174+0000) 2020-02-05T21:23:56.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:56.984+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:56.985385+0000) 2020-02-05T21:23:56.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:56.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:56.996574+0000) 2020-02-05T21:23:58.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:23:58.615+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.616909+0000) 2020-02-05T21:23:58.620 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:58.617+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.618822+0000) 2020-02-05T21:23:58.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:23:58.619+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.619483+0000) 2020-02-05T21:23:58.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:23:58.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.619850+0000) 2020-02-05T21:23:58.621 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:23:58.619+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.620284+0000) 2020-02-05T21:23:58.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:23:58.620+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.621271+0000) 2020-02-05T21:23:58.623 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:23:58.621+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.621989+0000) 2020-02-05T21:23:58.624 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:23:58.621+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.622981+0000) 2020-02-05T21:23:58.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:23:58.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.623285+0000) 2020-02-05T21:23:58.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:23:58.622+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.622744+0000) 2020-02-05T21:23:58.625 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:58.622+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.623407+0000) 2020-02-05T21:23:58.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:58.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.627398+0000) 2020-02-05T21:23:58.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:58.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.627872+0000) 2020-02-05T21:23:58.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:23:58.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:23:58.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:58.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:58.629828+0000) 2020-02-05T21:23:59.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:23:59.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:59.093715+0000) 2020-02-05T21:23:59.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:23:59.378+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:59.378518+0000) 2020-02-05T21:23:59.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:23:59.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:23:59.987118+0000) 2020-02-05T21:24:00.221 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:00.219+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:00.219981+0000) 2020-02-05T21:24:00.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:00.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:00.238532+0000) 2020-02-05T21:24:00.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:00.247+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:00.248357+0000) 2020-02-05T21:24:02.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:02.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:02.228226+0000) 2020-02-05T21:24:02.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:02.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:02.629379+0000) 2020-02-05T21:24:03.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:03.633+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.634596+0000) 2020-02-05T21:24:03.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:03.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.635349+0000) 2020-02-05T21:24:03.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:03.635+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.636374+0000) 2020-02-05T21:24:03.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:03.636+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.636826+0000) 2020-02-05T21:24:03.638 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:03.636+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.637298+0000) 2020-02-05T21:24:03.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:03.636+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.637989+0000) 2020-02-05T21:24:03.639 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:03.637+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.637639+0000) 2020-02-05T21:24:03.640 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:03.638+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.639151+0000) 2020-02-05T21:24:03.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:03.638+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.639498+0000) 2020-02-05T21:24:03.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:03.638+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.640017+0000) 2020-02-05T21:24:03.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:03.639+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.640130+0000) 2020-02-05T21:24:03.642 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:03.639+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.640284+0000) 2020-02-05T21:24:03.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:03.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.640846+0000) 2020-02-05T21:24:03.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:03.640+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:03.641872+0000) 2020-02-05T21:24:03.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:03.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:05.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:05.842+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:05.842534+0000) 2020-02-05T21:24:06.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:06.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:06.739290+0000) 2020-02-05T21:24:06.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:06.888+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:06.888734+0000) 2020-02-05T21:24:06.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:06.985+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:06.985454+0000) 2020-02-05T21:24:06.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:06.985+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:06.985661+0000) 2020-02-05T21:24:06.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:06.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:06.996800+0000) 2020-02-05T21:24:08.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:08.649+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.650345+0000) 2020-02-05T21:24:08.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:08.651+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.652437+0000) 2020-02-05T21:24:08.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:08.652+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.652846+0000) 2020-02-05T21:24:08.654 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:08.653+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.653418+0000) 2020-02-05T21:24:08.655 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:08.652+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.653816+0000) 2020-02-05T21:24:08.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:08.653+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.654344+0000) 2020-02-05T21:24:08.657 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:08.654+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.654633+0000) 2020-02-05T21:24:08.657 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:08.654+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.655253+0000) 2020-02-05T21:24:08.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:08.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.654804+0000) 2020-02-05T21:24:08.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:08.654+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.655800+0000) 2020-02-05T21:24:08.659 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:08.656+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.656466+0000) 2020-02-05T21:24:08.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:08.657+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.657493+0000) 2020-02-05T21:24:08.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:08.656+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.657208+0000) 2020-02-05T21:24:08.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:08.660+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:08.662052+0000) 2020-02-05T21:24:08.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:08.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84764 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:09.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:09.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:09.093960+0000) 2020-02-05T21:24:09.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:09.377+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:09.378747+0000) 2020-02-05T21:24:09.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:09.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:09.987346+0000) 2020-02-05T21:24:10.221 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:10.220+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:10.220245+0000) 2020-02-05T21:24:10.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:10.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:10.238762+0000) 2020-02-05T21:24:10.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:10.248+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:10.248589+0000) 2020-02-05T21:24:10.918 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:24:10.924 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:24:12.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:12.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:12.228493+0000) 2020-02-05T21:24:12.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:12.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:12.629632+0000) 2020-02-05T21:24:13.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:13.668+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.669206+0000) 2020-02-05T21:24:13.671 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:13.668+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.669452+0000) 2020-02-05T21:24:13.672 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:13.669+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.670897+0000) 2020-02-05T21:24:13.672 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:13.670+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.671156+0000) 2020-02-05T21:24:13.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:13.670+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.671311+0000) 2020-02-05T21:24:13.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:13.671+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.672400+0000) 2020-02-05T21:24:13.674 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:13.671+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.671951+0000) 2020-02-05T21:24:13.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:13.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.673136+0000) 2020-02-05T21:24:13.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:13.673+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.674409+0000) 2020-02-05T21:24:13.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:13.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.674528+0000) 2020-02-05T21:24:13.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:13.674+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.674741+0000) 2020-02-05T21:24:13.677 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:13.674+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.675400+0000) 2020-02-05T21:24:13.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:13.674+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.675633+0000) 2020-02-05T21:24:13.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:13.677+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:13.678892+0000) 2020-02-05T21:24:13.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:13.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:15.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:15.842+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:15.842766+0000) 2020-02-05T21:24:16.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:16.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:16.739526+0000) 2020-02-05T21:24:16.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:16.896+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:16.896733+0000) 2020-02-05T21:24:16.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:16.985+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:16.985699+0000) 2020-02-05T21:24:16.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:16.985+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:16.985904+0000) 2020-02-05T21:24:16.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:16.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:16.996980+0000) 2020-02-05T21:24:18.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:18.683+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.683794+0000) 2020-02-05T21:24:18.686 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:18.685+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.685496+0000) 2020-02-05T21:24:18.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:18.686+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.686574+0000) 2020-02-05T21:24:18.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:18.686+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.687047+0000) 2020-02-05T21:24:18.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:18.687+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.687520+0000) 2020-02-05T21:24:18.689 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:18.687+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.687803+0000) 2020-02-05T21:24:18.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:18.688+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.688405+0000) 2020-02-05T21:24:18.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:18.689+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.689123+0000) 2020-02-05T21:24:18.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:18.688+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.688778+0000) 2020-02-05T21:24:18.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:18.689+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.689472+0000) 2020-02-05T21:24:18.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:18.690+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.690482+0000) 2020-02-05T21:24:18.692 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:18.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.691132+0000) 2020-02-05T21:24:18.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:18.691+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.691642+0000) 2020-02-05T21:24:18.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:18.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:18.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:18.696+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:18.696412+0000) 2020-02-05T21:24:19.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:19.094+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:19.094189+0000) 2020-02-05T21:24:19.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:19.378+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:19.378972+0000) 2020-02-05T21:24:19.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:19.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:19.987585+0000) 2020-02-05T21:24:20.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:20.220+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:20.220504+0000) 2020-02-05T21:24:20.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:20.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:20.239006+0000) 2020-02-05T21:24:20.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:20.248+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:20.248821+0000) 2020-02-05T21:24:22.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:22.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:22.228745+0000) 2020-02-05T21:24:22.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:22.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:22.629887+0000) 2020-02-05T21:24:23.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:23.701+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.702589+0000) 2020-02-05T21:24:23.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:23.701+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.702441+0000) 2020-02-05T21:24:23.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:23.701+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.702735+0000) 2020-02-05T21:24:23.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:23.701+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.703239+0000) 2020-02-05T21:24:23.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:23.702+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.703721+0000) 2020-02-05T21:24:23.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:23.702+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.703992+0000) 2020-02-05T21:24:23.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:23.703+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.704580+0000) 2020-02-05T21:24:23.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:23.703+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.704767+0000) 2020-02-05T21:24:23.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:23.703+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.704982+0000) 2020-02-05T21:24:23.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:23.704+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.705697+0000) 2020-02-05T21:24:23.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:23.705+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.706636+0000) 2020-02-05T21:24:23.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:23.705+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.706811+0000) 2020-02-05T21:24:23.709 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:23.705+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.707049+0000) 2020-02-05T21:24:23.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:23.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:23.710279+0000) 2020-02-05T21:24:23.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:23.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:25.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:25.842+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:25.842999+0000) 2020-02-05T21:24:26.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:26.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:26.739752+0000) 2020-02-05T21:24:26.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:26.904+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:26.905201+0000) 2020-02-05T21:24:26.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:26.984+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:26.985956+0000) 2020-02-05T21:24:26.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:26.984+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:26.986127+0000) 2020-02-05T21:24:26.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:26.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:26.997212+0000) 2020-02-05T21:24:28.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:28.716+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.718270+0000) 2020-02-05T21:24:28.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:28.719+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.720663+0000) 2020-02-05T21:24:28.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:28.719+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.721077+0000) 2020-02-05T21:24:28.723 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:28.720+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.721917+0000) 2020-02-05T21:24:28.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:28.721+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.722465+0000) 2020-02-05T21:24:28.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:28.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.723665+0000) 2020-02-05T21:24:28.725 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:28.721+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.722817+0000) 2020-02-05T21:24:28.725 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:28.722+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.723581+0000) 2020-02-05T21:24:28.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:28.723+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.724224+0000) 2020-02-05T21:24:28.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:28.723+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.724620+0000) 2020-02-05T21:24:28.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:28.723+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.725047+0000) 2020-02-05T21:24:28.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:28.724+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.725185+0000) 2020-02-05T21:24:28.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:28.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.725578+0000) 2020-02-05T21:24:28.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:28.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:28.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:28.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:28.732846+0000) 2020-02-05T21:24:29.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:29.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:29.094423+0000) 2020-02-05T21:24:29.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:29.378+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:29.379181+0000) 2020-02-05T21:24:29.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:29.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:29.987839+0000) 2020-02-05T21:24:30.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:30.219+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:30.220754+0000) 2020-02-05T21:24:30.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:30.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:30.239241+0000) 2020-02-05T21:24:30.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:30.248+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:30.249117+0000) 2020-02-05T21:24:32.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:32.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:32.228977+0000) 2020-02-05T21:24:32.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:32.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:32.630121+0000) 2020-02-05T21:24:33.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:33.733+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.734780+0000) 2020-02-05T21:24:33.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:33.735+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.736620+0000) 2020-02-05T21:24:33.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:33.736+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.737641+0000) 2020-02-05T21:24:33.739 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:33.737+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.738243+0000) 2020-02-05T21:24:33.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:33.737+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.738415+0000) 2020-02-05T21:24:33.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:33.738+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.739211+0000) 2020-02-05T21:24:33.740 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:33.738+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.739655+0000) 2020-02-05T21:24:33.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:33.739+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.740219+0000) 2020-02-05T21:24:33.742 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:33.739+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.740404+0000) 2020-02-05T21:24:33.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:33.739+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.740996+0000) 2020-02-05T21:24:33.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:33.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.741445+0000) 2020-02-05T21:24:33.743 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:33.739+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.741275+0000) 2020-02-05T21:24:33.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:33.740+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.741821+0000) 2020-02-05T21:24:33.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:33.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 84970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:33.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:33.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:33.746894+0000) 2020-02-05T21:24:35.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:35.842+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:35.843272+0000) 2020-02-05T21:24:36.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:36.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:36.739985+0000) 2020-02-05T21:24:36.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:36.912+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:36.913380+0000) 2020-02-05T21:24:36.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:36.985+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:36.986208+0000) 2020-02-05T21:24:36.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:36.985+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:36.986372+0000) 2020-02-05T21:24:36.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:36.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:36.997441+0000) 2020-02-05T21:24:38.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:38.751+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.752715+0000) 2020-02-05T21:24:38.754 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:38.751+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.753163+0000) 2020-02-05T21:24:38.755 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:38.752+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.753611+0000) 2020-02-05T21:24:38.756 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:38.753+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.754812+0000) 2020-02-05T21:24:38.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:38.753+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.754923+0000) 2020-02-05T21:24:38.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:38.753+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.755255+0000) 2020-02-05T21:24:38.757 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:38.754+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.755762+0000) 2020-02-05T21:24:38.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:38.755+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.756773+0000) 2020-02-05T21:24:38.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:38.756+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.757370+0000) 2020-02-05T21:24:38.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:38.756+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.757561+0000) 2020-02-05T21:24:38.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:38.756+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.757830+0000) 2020-02-05T21:24:38.760 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:38.756+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.757860+0000) 2020-02-05T21:24:38.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:38.757+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.758539+0000) 2020-02-05T21:24:38.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:38.762+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:38.763410+0000) 2020-02-05T21:24:38.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:38.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:39.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:39.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:39.094694+0000) 2020-02-05T21:24:39.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:39.378+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:39.379375+0000) 2020-02-05T21:24:39.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:39.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:39.988108+0000) 2020-02-05T21:24:40.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:40.220+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:40.221005+0000) 2020-02-05T21:24:40.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:40.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:40.239479+0000) 2020-02-05T21:24:40.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:40.248+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:40.249303+0000) 2020-02-05T21:24:41.057 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:24:41.064 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:24:42.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:42.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:42.229234+0000) 2020-02-05T21:24:42.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:42.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:42.630374+0000) 2020-02-05T21:24:43.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:43.769+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.770194+0000) 2020-02-05T21:24:43.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:43.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.771446+0000) 2020-02-05T21:24:43.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:43.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.775810+0000) 2020-02-05T21:24:43.780 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:43.771+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.772246+0000) 2020-02-05T21:24:43.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:43.771+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.772821+0000) 2020-02-05T21:24:43.781 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:43.772+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.773435+0000) 2020-02-05T21:24:43.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:43.774+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.775662+0000) 2020-02-05T21:24:43.783 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:43.768+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.769873+0000) 2020-02-05T21:24:43.783 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:43.769+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.770917+0000) 2020-02-05T21:24:43.783 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:43.772+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.773953+0000) 2020-02-05T21:24:43.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:43.772+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.774206+0000) 2020-02-05T21:24:43.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:43.773+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.774873+0000) 2020-02-05T21:24:43.785 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:43.774+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.775605+0000) 2020-02-05T21:24:43.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:43.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:43.779471+0000) 2020-02-05T21:24:43.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:43.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:45.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:45.842+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:45.843541+0000) 2020-02-05T21:24:46.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:46.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:46.740183+0000) 2020-02-05T21:24:46.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:46.920+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:46.921668+0000) 2020-02-05T21:24:46.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:46.985+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:46.986490+0000) 2020-02-05T21:24:46.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:46.985+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:46.986678+0000) 2020-02-05T21:24:46.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:46.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:46.997665+0000) 2020-02-05T21:24:48.787 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:48.784+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.785630+0000) 2020-02-05T21:24:48.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:48.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.786586+0000) 2020-02-05T21:24:48.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:48.786+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.787518+0000) 2020-02-05T21:24:48.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:48.786+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.787760+0000) 2020-02-05T21:24:48.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:48.786+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.787919+0000) 2020-02-05T21:24:48.790 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:48.786+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.788310+0000) 2020-02-05T21:24:48.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:48.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.788911+0000) 2020-02-05T21:24:48.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:48.788+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.789155+0000) 2020-02-05T21:24:48.791 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:48.787+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.789375+0000) 2020-02-05T21:24:48.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:48.788+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.790078+0000) 2020-02-05T21:24:48.792 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:48.788+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.789556+0000) 2020-02-05T21:24:48.792 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:48.788+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.790364+0000) 2020-02-05T21:24:48.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:48.789+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.791110+0000) 2020-02-05T21:24:48.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:48.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:48.794300+0000) 2020-02-05T21:24:48.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:48.794+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:49.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:49.094+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:49.094911+0000) 2020-02-05T21:24:49.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:49.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:49.379610+0000) 2020-02-05T21:24:49.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:49.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:49.988318+0000) 2020-02-05T21:24:50.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:50.220+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:50.221241+0000) 2020-02-05T21:24:50.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:50.238+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:50.239712+0000) 2020-02-05T21:24:50.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:50.249+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:50.249569+0000) 2020-02-05T21:24:52.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:52.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:52.229490+0000) 2020-02-05T21:24:52.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:52.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:52.630603+0000) 2020-02-05T21:24:53.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:53.800+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.801436+0000) 2020-02-05T21:24:53.803 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:53.800+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.801824+0000) 2020-02-05T21:24:53.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:53.801+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.802858+0000) 2020-02-05T21:24:53.805 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:53.802+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.803710+0000) 2020-02-05T21:24:53.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:53.802+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.804043+0000) 2020-02-05T21:24:53.806 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:53.803+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.804579+0000) 2020-02-05T21:24:53.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:53.803+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.805161+0000) 2020-02-05T21:24:53.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:53.803+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.805360+0000) 2020-02-05T21:24:53.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:53.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.805832+0000) 2020-02-05T21:24:53.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:53.805+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.806356+0000) 2020-02-05T21:24:53.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:53.805+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.806777+0000) 2020-02-05T21:24:53.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:53.806+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.807384+0000) 2020-02-05T21:24:53.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:53.806+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.807748+0000) 2020-02-05T21:24:53.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:53.811+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:53.812533+0000) 2020-02-05T21:24:53.814 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:53.811+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:55.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:55.843+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:55.843777+0000) 2020-02-05T21:24:56.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:56.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:56.740415+0000) 2020-02-05T21:24:56.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:56.932+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:56.933301+0000) 2020-02-05T21:24:56.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:56.986+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:56.986722+0000) 2020-02-05T21:24:56.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:56.986+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:56.986904+0000) 2020-02-05T21:24:56.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:56.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:56.997894+0000) 2020-02-05T21:24:58.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:24:58.816+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.817553+0000) 2020-02-05T21:24:58.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:58.817+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.818518+0000) 2020-02-05T21:24:58.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:24:58.818+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.819227+0000) 2020-02-05T21:24:58.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:24:58.818+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.819611+0000) 2020-02-05T21:24:58.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:24:58.820+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.821232+0000) 2020-02-05T21:24:58.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:24:58.819+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.820561+0000) 2020-02-05T21:24:58.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:58.819+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.820695+0000) 2020-02-05T21:24:58.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:24:58.819+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.821398+0000) 2020-02-05T21:24:58.825 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:24:58.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.821819+0000) 2020-02-05T21:24:58.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:58.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.822888+0000) 2020-02-05T21:24:58.827 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:58.822+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.823481+0000) 2020-02-05T21:24:58.827 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:24:58.821+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.822943+0000) 2020-02-05T21:24:58.828 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:24:58.821+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.823036+0000) 2020-02-05T21:24:58.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:24:58.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:24:58.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:58.827+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:58.828496+0000) 2020-02-05T21:24:59.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:24:59.093+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:59.095121+0000) 2020-02-05T21:24:59.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:24:59.378+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:59.379863+0000) 2020-02-05T21:24:59.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:24:59.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:24:59.988557+0000) 2020-02-05T21:25:00.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:00.220+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:00.221540+0000) 2020-02-05T21:25:00.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:00.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:00.239959+0000) 2020-02-05T21:25:00.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:00.249+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:00.249798+0000) 2020-02-05T21:25:02.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:02.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:02.229747+0000) 2020-02-05T21:25:02.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:02.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:02.630824+0000) 2020-02-05T21:25:03.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:03.833+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.834295+0000) 2020-02-05T21:25:03.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:03.833+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.834803+0000) 2020-02-05T21:25:03.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:03.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.835233+0000) 2020-02-05T21:25:03.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:03.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.835708+0000) 2020-02-05T21:25:03.837 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:03.834+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.835891+0000) 2020-02-05T21:25:03.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:03.836+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.837699+0000) 2020-02-05T21:25:03.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:03.835+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.837319+0000) 2020-02-05T21:25:03.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:03.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.838595+0000) 2020-02-05T21:25:03.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:03.838+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.839224+0000) 2020-02-05T21:25:03.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:03.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.839176+0000) 2020-02-05T21:25:03.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:03.838+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.839998+0000) 2020-02-05T21:25:03.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:03.838+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.840153+0000) 2020-02-05T21:25:03.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:03.839+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.840752+0000) 2020-02-05T21:25:03.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:03.841+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:03.842669+0000) 2020-02-05T21:25:03.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:03.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:05.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:05.843+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:05.844010+0000) 2020-02-05T21:25:06.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:06.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:06.740679+0000) 2020-02-05T21:25:06.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:06.941+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:06.941527+0000) 2020-02-05T21:25:06.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:06.986+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:06.986961+0000) 2020-02-05T21:25:06.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:06.986+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:06.987161+0000) 2020-02-05T21:25:07.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:06.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:06.998136+0000) 2020-02-05T21:25:08.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:08.849+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.850352+0000) 2020-02-05T21:25:08.853 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:08.850+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.851702+0000) 2020-02-05T21:25:08.853 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:08.850+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.852006+0000) 2020-02-05T21:25:08.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:08.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.852586+0000) 2020-02-05T21:25:08.854 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:08.851+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.853057+0000) 2020-02-05T21:25:08.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:08.852+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.853390+0000) 2020-02-05T21:25:08.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:08.852+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.853774+0000) 2020-02-05T21:25:08.856 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:08.852+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.854180+0000) 2020-02-05T21:25:08.856 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:08.854+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.854642+0000) 2020-02-05T21:25:08.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:08.854+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.854920+0000) 2020-02-05T21:25:08.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:08.854+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.855264+0000) 2020-02-05T21:25:08.857 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:08.854+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.856025+0000) 2020-02-05T21:25:08.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:08.854+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.856410+0000) 2020-02-05T21:25:08.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:08.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:08.860777+0000) 2020-02-05T21:25:08.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:08.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:09.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:09.094+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:09.095384+0000) 2020-02-05T21:25:09.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:09.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:09.380066+0000) 2020-02-05T21:25:09.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:09.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:09.988785+0000) 2020-02-05T21:25:10.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:10.221+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:10.221790+0000) 2020-02-05T21:25:10.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:10.249+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:10.250061+0000) 2020-02-05T21:25:10.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:10.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:10.240198+0000) 2020-02-05T21:25:11.227 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:25:11.233 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:25:12.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:12.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:12.229968+0000) 2020-02-05T21:25:12.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:12.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:12.631109+0000) 2020-02-05T21:25:13.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:13.867+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.867113+0000) 2020-02-05T21:25:13.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:13.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.867959+0000) 2020-02-05T21:25:13.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:13.867+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.868296+0000) 2020-02-05T21:25:13.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:13.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.868793+0000) 2020-02-05T21:25:13.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:13.869+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.869439+0000) 2020-02-05T21:25:13.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:13.869+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.869769+0000) 2020-02-05T21:25:13.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:13.869+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.870070+0000) 2020-02-05T21:25:13.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:13.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.870554+0000) 2020-02-05T21:25:13.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:13.871+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.871587+0000) 2020-02-05T21:25:13.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:13.871+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.872005+0000) 2020-02-05T21:25:13.875 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:13.872+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.872587+0000) 2020-02-05T21:25:13.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:13.873+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.873511+0000) 2020-02-05T21:25:13.875 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:13.873+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.873708+0000) 2020-02-05T21:25:13.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:13.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:13.877561+0000) 2020-02-05T21:25:13.880 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:13.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:15.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:15.843+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:15.844315+0000) 2020-02-05T21:25:16.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:16.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:16.740907+0000) 2020-02-05T21:25:16.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:16.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:16.949909+0000) 2020-02-05T21:25:16.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:16.986+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:16.987256+0000) 2020-02-05T21:25:16.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:16.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:16.987462+0000) 2020-02-05T21:25:16.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:16.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:16.998370+0000) 2020-02-05T21:25:18.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:18.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.884816+0000) 2020-02-05T21:25:18.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:18.884+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.885915+0000) 2020-02-05T21:25:18.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:18.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.886382+0000) 2020-02-05T21:25:18.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:18.886+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.886790+0000) 2020-02-05T21:25:18.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:18.886+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.887321+0000) 2020-02-05T21:25:18.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:18.887+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.888267+0000) 2020-02-05T21:25:18.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:18.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.888452+0000) 2020-02-05T21:25:18.890 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:18.888+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.888527+0000) 2020-02-05T21:25:18.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:18.888+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.888969+0000) 2020-02-05T21:25:18.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:18.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.889322+0000) 2020-02-05T21:25:18.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:18.889+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.889584+0000) 2020-02-05T21:25:18.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:18.889+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.890139+0000) 2020-02-05T21:25:18.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:18.890+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.890932+0000) 2020-02-05T21:25:18.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:18.893+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:18.894173+0000) 2020-02-05T21:25:18.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:18.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:19.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:19.095+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:19.095700+0000) 2020-02-05T21:25:19.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:19.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:19.380347+0000) 2020-02-05T21:25:19.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:19.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:19.989019+0000) 2020-02-05T21:25:20.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:20.221+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:20.222085+0000) 2020-02-05T21:25:20.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:20.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:20.240460+0000) 2020-02-05T21:25:20.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:20.249+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:20.250341+0000) 2020-02-05T21:25:22.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:22.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:22.230208+0000) 2020-02-05T21:25:22.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:22.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:22.631356+0000) 2020-02-05T21:25:23.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:23.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.901548+0000) 2020-02-05T21:25:23.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:23.900+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.902010+0000) 2020-02-05T21:25:23.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:23.901+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.902630+0000) 2020-02-05T21:25:23.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:23.902+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.903469+0000) 2020-02-05T21:25:23.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:23.902+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.903337+0000) 2020-02-05T21:25:23.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:23.903+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.904004+0000) 2020-02-05T21:25:23.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:23.903+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.904904+0000) 2020-02-05T21:25:23.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:23.904+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.905170+0000) 2020-02-05T21:25:23.907 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:23.905+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.905581+0000) 2020-02-05T21:25:23.908 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:23.905+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.906258+0000) 2020-02-05T21:25:23.908 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:23.905+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.906160+0000) 2020-02-05T21:25:23.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:23.906+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.907015+0000) 2020-02-05T21:25:23.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:23.906+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.907273+0000) 2020-02-05T21:25:23.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:23.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:23.912058+0000) 2020-02-05T21:25:23.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:23.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:25.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:25.843+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:25.844548+0000) 2020-02-05T21:25:26.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:26.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:26.741173+0000) 2020-02-05T21:25:26.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:26.957+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:26.958128+0000) 2020-02-05T21:25:26.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:26.987+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:26.987505+0000) 2020-02-05T21:25:26.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:26.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:26.987676+0000) 2020-02-05T21:25:27.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:26.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:26.998614+0000) 2020-02-05T21:25:28.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:28.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.917780+0000) 2020-02-05T21:25:28.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:28.917+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.918461+0000) 2020-02-05T21:25:28.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:28.919+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.919846+0000) 2020-02-05T21:25:28.921 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:28.919+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.920152+0000) 2020-02-05T21:25:28.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:28.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.920881+0000) 2020-02-05T21:25:28.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:28.920+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.920969+0000) 2020-02-05T21:25:28.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:28.920+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.921412+0000) 2020-02-05T21:25:28.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:28.921+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.921940+0000) 2020-02-05T21:25:28.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:28.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.922943+0000) 2020-02-05T21:25:28.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:28.923+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.924354+0000) 2020-02-05T21:25:28.926 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:28.923+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.924730+0000) 2020-02-05T21:25:28.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:28.923+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.924047+0000) 2020-02-05T21:25:28.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:28.924+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.924726+0000) 2020-02-05T21:25:28.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:28.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:28.929417+0000) 2020-02-05T21:25:28.931 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:28.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:29.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:29.095+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:29.095974+0000) 2020-02-05T21:25:29.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:29.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:29.380543+0000) 2020-02-05T21:25:29.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:29.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:29.989288+0000) 2020-02-05T21:25:30.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:30.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:30.222339+0000) 2020-02-05T21:25:30.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:30.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:30.240710+0000) 2020-02-05T21:25:30.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:30.249+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:30.250566+0000) 2020-02-05T21:25:32.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:32.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:32.230486+0000) 2020-02-05T21:25:32.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:32.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:32.631612+0000) 2020-02-05T21:25:33.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:33.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.936016+0000) 2020-02-05T21:25:33.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:33.935+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.936169+0000) 2020-02-05T21:25:33.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:33.935+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.936668+0000) 2020-02-05T21:25:33.939 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:33.936+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.937231+0000) 2020-02-05T21:25:33.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:33.936+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.937280+0000) 2020-02-05T21:25:33.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:33.936+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.937672+0000) 2020-02-05T21:25:33.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:33.937+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.938178+0000) 2020-02-05T21:25:33.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:33.937+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.938301+0000) 2020-02-05T21:25:33.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:33.938+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.939396+0000) 2020-02-05T21:25:33.941 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:33.939+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.939484+0000) 2020-02-05T21:25:33.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:33.939+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.939901+0000) 2020-02-05T21:25:33.942 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:33.939+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.940602+0000) 2020-02-05T21:25:33.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:33.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.941024+0000) 2020-02-05T21:25:33.946 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:33.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:33.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:33.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:33.949350+0000) 2020-02-05T21:25:35.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:35.844+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:35.844780+0000) 2020-02-05T21:25:36.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:36.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:36.741410+0000) 2020-02-05T21:25:36.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:36.965+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:36.966323+0000) 2020-02-05T21:25:36.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:36.987+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:36.987743+0000) 2020-02-05T21:25:36.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:36.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:36.987916+0000) 2020-02-05T21:25:37.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:36.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:36.998854+0000) 2020-02-05T21:25:38.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:38.950+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.951709+0000) 2020-02-05T21:25:38.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:38.951+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.951530+0000) 2020-02-05T21:25:38.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:38.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.953398+0000) 2020-02-05T21:25:38.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:38.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.953992+0000) 2020-02-05T21:25:38.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:38.953+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.953495+0000) 2020-02-05T21:25:38.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:38.953+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.954394+0000) 2020-02-05T21:25:38.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:38.954+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.954761+0000) 2020-02-05T21:25:38.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:38.954+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.955228+0000) 2020-02-05T21:25:38.957 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:38.955+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.956298+0000) 2020-02-05T21:25:38.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:38.956+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.956523+0000) 2020-02-05T21:25:38.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:38.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.957392+0000) 2020-02-05T21:25:38.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:38.957+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.957539+0000) 2020-02-05T21:25:38.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:38.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.957786+0000) 2020-02-05T21:25:38.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:38.962+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:38.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:38.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:38.963565+0000) 2020-02-05T21:25:39.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:39.096+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:39.096216+0000) 2020-02-05T21:25:39.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:39.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:39.380777+0000) 2020-02-05T21:25:39.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:39.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:39.989558+0000) 2020-02-05T21:25:40.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:40.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:40.222633+0000) 2020-02-05T21:25:40.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:40.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:40.240963+0000) 2020-02-05T21:25:40.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:40.250+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:40.250788+0000) 2020-02-05T21:25:41.382 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:25:41.389 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:25:42.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:42.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:42.230745+0000) 2020-02-05T21:25:42.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:42.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:42.631849+0000) 2020-02-05T21:25:43.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:43.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.968841+0000) 2020-02-05T21:25:43.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:43.969+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.970247+0000) 2020-02-05T21:25:43.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:43.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.971736+0000) 2020-02-05T21:25:43.973 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:43.971+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.971733+0000) 2020-02-05T21:25:43.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:43.971+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.971636+0000) 2020-02-05T21:25:43.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:43.971+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.972277+0000) 2020-02-05T21:25:43.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:43.972+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.972715+0000) 2020-02-05T21:25:43.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:43.972+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.973257+0000) 2020-02-05T21:25:43.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:43.972+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.972922+0000) 2020-02-05T21:25:43.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:43.972+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.973651+0000) 2020-02-05T21:25:43.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:43.973+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.974506+0000) 2020-02-05T21:25:43.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:43.973+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.974757+0000) 2020-02-05T21:25:43.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:43.974+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.974845+0000) 2020-02-05T21:25:43.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:43.979+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:43.980104+0000) 2020-02-05T21:25:43.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:43.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:45.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:45.844+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:45.845011+0000) 2020-02-05T21:25:46.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:46.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:46.741644+0000) 2020-02-05T21:25:46.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:46.974+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:46.974901+0000) 2020-02-05T21:25:46.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:46.987+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:46.987972+0000) 2020-02-05T21:25:46.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:46.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:46.988165+0000) 2020-02-05T21:25:47.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:46.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:46.999059+0000) 2020-02-05T21:25:48.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:48.985+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.986398+0000) 2020-02-05T21:25:48.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:48.986+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.987027+0000) 2020-02-05T21:25:48.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:48.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.987754+0000) 2020-02-05T21:25:48.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:48.987+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.988650+0000) 2020-02-05T21:25:48.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:48.988+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.989403+0000) 2020-02-05T21:25:48.991 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:48.989+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.990124+0000) 2020-02-05T21:25:48.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:48.988+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.989834+0000) 2020-02-05T21:25:48.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:48.989+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.990603+0000) 2020-02-05T21:25:48.993 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:48.990+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.991385+0000) 2020-02-05T21:25:48.993 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:48.990+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.991505+0000) 2020-02-05T21:25:48.995 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:48.991+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.991567+0000) 2020-02-05T21:25:48.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:48.991+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.991757+0000) 2020-02-05T21:25:48.996 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:48.992+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.992473+0000) 2020-02-05T21:25:48.996 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:48.994+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:48.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:48.995+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:48.996818+0000) 2020-02-05T21:25:49.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:49.096+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:49.096492+0000) 2020-02-05T21:25:49.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:49.379+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:49.381028+0000) 2020-02-05T21:25:49.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:49.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:49.989785+0000) 2020-02-05T21:25:50.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:50.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:50.222871+0000) 2020-02-05T21:25:50.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:50.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:50.241211+0000) 2020-02-05T21:25:50.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:50.250+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:50.251017+0000) 2020-02-05T21:25:52.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:52.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:52.230992+0000) 2020-02-05T21:25:52.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:52.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:52.632153+0000) 2020-02-05T21:25:54.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:53.999+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.000614+0000) 2020-02-05T21:25:54.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:54.000+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.001302+0000) 2020-02-05T21:25:54.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:54.000+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.001862+0000) 2020-02-05T21:25:54.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:54.002+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.003682+0000) 2020-02-05T21:25:54.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:54.001+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.003129+0000) 2020-02-05T21:25:54.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:54.002+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.004375+0000) 2020-02-05T21:25:54.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:54.003+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.005025+0000) 2020-02-05T21:25:54.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:54.004+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.005515+0000) 2020-02-05T21:25:54.007 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:54.004+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.005750+0000) 2020-02-05T21:25:54.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:54.005+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.006744+0000) 2020-02-05T21:25:54.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:54.005+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.007206+0000) 2020-02-05T21:25:54.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:54.005+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.007191+0000) 2020-02-05T21:25:54.009 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:54.006+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.008262+0000) 2020-02-05T21:25:54.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:54.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:54.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:54.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:54.014475+0000) 2020-02-05T21:25:55.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:55.844+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:55.845268+0000) 2020-02-05T21:25:56.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:56.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:56.741910+0000) 2020-02-05T21:25:56.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:56.986+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:56.986517+0000) 2020-02-05T21:25:56.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:56.986+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:56.988199+0000) 2020-02-05T21:25:56.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:56.986+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:56.988352+0000) 2020-02-05T21:25:57.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:56.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:56.999282+0000) 2020-02-05T21:25:59.018 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:25:59.015+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.017307+0000) 2020-02-05T21:25:59.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:59.017+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.018517+0000) 2020-02-05T21:25:59.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:25:59.017+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.018404+0000) 2020-02-05T21:25:59.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:25:59.017+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.018980+0000) 2020-02-05T21:25:59.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:25:59.018+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.019448+0000) 2020-02-05T21:25:59.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:59.019+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.020323+0000) 2020-02-05T21:25:59.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:25:59.019+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.020728+0000) 2020-02-05T21:25:59.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:59.019+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.021142+0000) 2020-02-05T21:25:59.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:25:59.021+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.022595+0000) 2020-02-05T21:25:59.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:25:59.021+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.023019+0000) 2020-02-05T21:25:59.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:25:59.021+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.023073+0000) 2020-02-05T21:25:59.025 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:25:59.022+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.023906+0000) 2020-02-05T21:25:59.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:59.025+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.026907+0000) 2020-02-05T21:25:59.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:59.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.027159+0000) 2020-02-05T21:25:59.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:25:59.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85667 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:25:59.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:25:59.095+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.096726+0000) 2020-02-05T21:25:59.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:25:59.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.381277+0000) 2020-02-05T21:25:59.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:25:59.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:25:59.990019+0000) 2020-02-05T21:26:00.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:00.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:00.223106+0000) 2020-02-05T21:26:00.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:00.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:00.241481+0000) 2020-02-05T21:26:00.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:00.251+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:00.251280+0000) 2020-02-05T21:26:02.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:02.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:02.231249+0000) 2020-02-05T21:26:02.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:02.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:02.632395+0000) 2020-02-05T21:26:04.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:04.035+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.036848+0000) 2020-02-05T21:26:04.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:04.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.037467+0000) 2020-02-05T21:26:04.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:04.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.037701+0000) 2020-02-05T21:26:04.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:04.036+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.037819+0000) 2020-02-05T21:26:04.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:04.037+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.038551+0000) 2020-02-05T21:26:04.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:04.037+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.038933+0000) 2020-02-05T21:26:04.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:04.038+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.039281+0000) 2020-02-05T21:26:04.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:04.038+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.039862+0000) 2020-02-05T21:26:04.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:04.039+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.040666+0000) 2020-02-05T21:26:04.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:04.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.041019+0000) 2020-02-05T21:26:04.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:04.040+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.041622+0000) 2020-02-05T21:26:04.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:04.040+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.042002+0000) 2020-02-05T21:26:04.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:04.041+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.042419+0000) 2020-02-05T21:26:04.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:04.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85707 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:04.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:04.045+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:04.046689+0000) 2020-02-05T21:26:05.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:05.845+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:05.845534+0000) 2020-02-05T21:26:06.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:06.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:06.742143+0000) 2020-02-05T21:26:06.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:06.987+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:06.988471+0000) 2020-02-05T21:26:06.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:06.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:06.988616+0000) 2020-02-05T21:26:06.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:06.994+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:06.994142+0000) 2020-02-05T21:26:07.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:06.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:06.999507+0000) 2020-02-05T21:26:09.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:09.052+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.052628+0000) 2020-02-05T21:26:09.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:09.053+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.053116+0000) 2020-02-05T21:26:09.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:09.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.053796+0000) 2020-02-05T21:26:09.055 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:09.053+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.054004+0000) 2020-02-05T21:26:09.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:09.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.054309+0000) 2020-02-05T21:26:09.056 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:09.055+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.055128+0000) 2020-02-05T21:26:09.056 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:09.055+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.055466+0000) 2020-02-05T21:26:09.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:09.054+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.055613+0000) 2020-02-05T21:26:09.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:09.056+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.056141+0000) 2020-02-05T21:26:09.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:09.055+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.056521+0000) 2020-02-05T21:26:09.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:09.055+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.057008+0000) 2020-02-05T21:26:09.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:09.055+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.057399+0000) 2020-02-05T21:26:09.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:09.055+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.056927+0000) 2020-02-05T21:26:09.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:09.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:09.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:09.062+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.062570+0000) 2020-02-05T21:26:09.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:09.096+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.096972+0000) 2020-02-05T21:26:09.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:09.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.381523+0000) 2020-02-05T21:26:09.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:09.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:09.990269+0000) 2020-02-05T21:26:10.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:10.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:10.223346+0000) 2020-02-05T21:26:10.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:10.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:10.241668+0000) 2020-02-05T21:26:10.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:10.250+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:10.251543+0000) 2020-02-05T21:26:11.520 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:26:11.527 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:26:12.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:12.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:12.231507+0000) 2020-02-05T21:26:12.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:12.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:12.632645+0000) 2020-02-05T21:26:14.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:14.065+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.066713+0000) 2020-02-05T21:26:14.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:14.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.067363+0000) 2020-02-05T21:26:14.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:14.066+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.067905+0000) 2020-02-05T21:26:14.070 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:14.067+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.068412+0000) 2020-02-05T21:26:14.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:14.067+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.068734+0000) 2020-02-05T21:26:14.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:14.068+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.069657+0000) 2020-02-05T21:26:14.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:14.068+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.069999+0000) 2020-02-05T21:26:14.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:14.069+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.070374+0000) 2020-02-05T21:26:14.072 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:14.069+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.070914+0000) 2020-02-05T21:26:14.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:14.071+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.072162+0000) 2020-02-05T21:26:14.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:14.069+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.071058+0000) 2020-02-05T21:26:14.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:14.071+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.072612+0000) 2020-02-05T21:26:14.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:14.072+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.073747+0000) 2020-02-05T21:26:14.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:14.074+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:14.075596+0000) 2020-02-05T21:26:14.079 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:14.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:15.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:15.844+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:15.845767+0000) 2020-02-05T21:26:16.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:16.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:16.742376+0000) 2020-02-05T21:26:16.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:16.987+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:16.988756+0000) 2020-02-05T21:26:16.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:16.987+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:16.988885+0000) 2020-02-05T21:26:17.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:16.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:16.999744+0000) 2020-02-05T21:26:17.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:17.001+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:17.002010+0000) 2020-02-05T21:26:19.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:19.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.084687+0000) 2020-02-05T21:26:19.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:19.083+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.084959+0000) 2020-02-05T21:26:19.087 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:19.085+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.086241+0000) 2020-02-05T21:26:19.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:19.085+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.086448+0000) 2020-02-05T21:26:19.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:19.085+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.086883+0000) 2020-02-05T21:26:19.088 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:19.086+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.087393+0000) 2020-02-05T21:26:19.089 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:19.086+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.087698+0000) 2020-02-05T21:26:19.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:19.086+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.087957+0000) 2020-02-05T21:26:19.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:19.087+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.088490+0000) 2020-02-05T21:26:19.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:19.088+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.089579+0000) 2020-02-05T21:26:19.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:19.088+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.089651+0000) 2020-02-05T21:26:19.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:19.089+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.090685+0000) 2020-02-05T21:26:19.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:19.089+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.090858+0000) 2020-02-05T21:26:19.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:19.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:19.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:19.095+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.097114+0000) 2020-02-05T21:26:19.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:19.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.097270+0000) 2020-02-05T21:26:19.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:19.380+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.381767+0000) 2020-02-05T21:26:19.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:19.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:19.990465+0000) 2020-02-05T21:26:20.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:20.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:20.223581+0000) 2020-02-05T21:26:20.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:20.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:20.241914+0000) 2020-02-05T21:26:20.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:20.250+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:20.251773+0000) 2020-02-05T21:26:22.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:22.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:22.231711+0000) 2020-02-05T21:26:22.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:22.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:22.632911+0000) 2020-02-05T21:26:24.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:24.097+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.098505+0000) 2020-02-05T21:26:24.100 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:24.096+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.098206+0000) 2020-02-05T21:26:24.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:24.098+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.099261+0000) 2020-02-05T21:26:24.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:24.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.099932+0000) 2020-02-05T21:26:24.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:24.099+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.100282+0000) 2020-02-05T21:26:24.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:24.098+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.100141+0000) 2020-02-05T21:26:24.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:24.099+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.101066+0000) 2020-02-05T21:26:24.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:24.099+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.100808+0000) 2020-02-05T21:26:24.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:24.100+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.101956+0000) 2020-02-05T21:26:24.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:24.100+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.101757+0000) 2020-02-05T21:26:24.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:24.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.102427+0000) 2020-02-05T21:26:24.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:24.101+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.103116+0000) 2020-02-05T21:26:24.105 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:24.102+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.104249+0000) 2020-02-05T21:26:24.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:24.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:24.107331+0000) 2020-02-05T21:26:24.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:24.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85873 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:25.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:25.845+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:25.845995+0000) 2020-02-05T21:26:26.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:26.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:26.742609+0000) 2020-02-05T21:26:26.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:26.988+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:26.989012+0000) 2020-02-05T21:26:26.991 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:26.988+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:26.989125+0000) 2020-02-05T21:26:27.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:26.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:26.999963+0000) 2020-02-05T21:26:27.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:27.009+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:27.010163+0000) 2020-02-05T21:26:29.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:29.096+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.097336+0000) 2020-02-05T21:26:29.116 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:29.114+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.115119+0000) 2020-02-05T21:26:29.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:29.113+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.114854+0000) 2020-02-05T21:26:29.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:29.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.115419+0000) 2020-02-05T21:26:29.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:29.114+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.115700+0000) 2020-02-05T21:26:29.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:29.114+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.116049+0000) 2020-02-05T21:26:29.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:29.115+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.116199+0000) 2020-02-05T21:26:29.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:29.115+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.116309+0000) 2020-02-05T21:26:29.118 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:29.116+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.117152+0000) 2020-02-05T21:26:29.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:29.115+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.117087+0000) 2020-02-05T21:26:29.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:29.116+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.118081+0000) 2020-02-05T21:26:29.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:29.116+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.117474+0000) 2020-02-05T21:26:29.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:29.116+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.117819+0000) 2020-02-05T21:26:29.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:29.117+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.119126+0000) 2020-02-05T21:26:29.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:29.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.123356+0000) 2020-02-05T21:26:29.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:29.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:29.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:29.380+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.381978+0000) 2020-02-05T21:26:29.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:29.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:29.990678+0000) 2020-02-05T21:26:30.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:30.222+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:30.223834+0000) 2020-02-05T21:26:30.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:30.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:30.242162+0000) 2020-02-05T21:26:30.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:30.251+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:30.252006+0000) 2020-02-05T21:26:32.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:32.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:32.231970+0000) 2020-02-05T21:26:32.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:32.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:32.633148+0000) 2020-02-05T21:26:34.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:34.129+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.130438+0000) 2020-02-05T21:26:34.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:34.129+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.130977+0000) 2020-02-05T21:26:34.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:34.130+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.131577+0000) 2020-02-05T21:26:34.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:34.131+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.132197+0000) 2020-02-05T21:26:34.133 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:34.131+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.132322+0000) 2020-02-05T21:26:34.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:34.131+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.132563+0000) 2020-02-05T21:26:34.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:34.131+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.132840+0000) 2020-02-05T21:26:34.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:34.132+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.133447+0000) 2020-02-05T21:26:34.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:34.132+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.133637+0000) 2020-02-05T21:26:34.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:34.132+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.133611+0000) 2020-02-05T21:26:34.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:34.132+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.134217+0000) 2020-02-05T21:26:34.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:34.132+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.134329+0000) 2020-02-05T21:26:34.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:34.133+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.135352+0000) 2020-02-05T21:26:34.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:34.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:34.141388+0000) 2020-02-05T21:26:34.143 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:34.140+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85957 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:35.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:35.845+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:35.846262+0000) 2020-02-05T21:26:36.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:36.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:36.742839+0000) 2020-02-05T21:26:36.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:36.988+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:36.989247+0000) 2020-02-05T21:26:36.991 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:36.988+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:36.989365+0000) 2020-02-05T21:26:37.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:36.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:37.000196+0000) 2020-02-05T21:26:37.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:37.020+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:37.020624+0000) 2020-02-05T21:26:39.099 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:39.097+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.097597+0000) 2020-02-05T21:26:39.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:39.147+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.148168+0000) 2020-02-05T21:26:39.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:39.147+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.148517+0000) 2020-02-05T21:26:39.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:39.147+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.148907+0000) 2020-02-05T21:26:39.150 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:39.148+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.149295+0000) 2020-02-05T21:26:39.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:39.147+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.149201+0000) 2020-02-05T21:26:39.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:39.148+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.149696+0000) 2020-02-05T21:26:39.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:39.148+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.149564+0000) 2020-02-05T21:26:39.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:39.148+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.149974+0000) 2020-02-05T21:26:39.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:39.149+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.150333+0000) 2020-02-05T21:26:39.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:39.148+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.150338+0000) 2020-02-05T21:26:39.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:39.149+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.150937+0000) 2020-02-05T21:26:39.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:39.149+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.151054+0000) 2020-02-05T21:26:39.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:39.150+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.151760+0000) 2020-02-05T21:26:39.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:39.156+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 85999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:39.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:39.154+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.155939+0000) 2020-02-05T21:26:39.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:39.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.382192+0000) 2020-02-05T21:26:39.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:39.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:39.990892+0000) 2020-02-05T21:26:40.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:40.223+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:40.224069+0000) 2020-02-05T21:26:40.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:40.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:40.242408+0000) 2020-02-05T21:26:40.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:40.251+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:40.252272+0000) 2020-02-05T21:26:41.666 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:26:41.672 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:26:42.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:42.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:42.232229+0000) 2020-02-05T21:26:42.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:42.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:42.633402+0000) 2020-02-05T21:26:44.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:44.162+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.163837+0000) 2020-02-05T21:26:44.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:44.163+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.164782+0000) 2020-02-05T21:26:44.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:44.164+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.165142+0000) 2020-02-05T21:26:44.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:44.164+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.165415+0000) 2020-02-05T21:26:44.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:44.164+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.165580+0000) 2020-02-05T21:26:44.167 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:44.165+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.165602+0000) 2020-02-05T21:26:44.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:44.165+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.165811+0000) 2020-02-05T21:26:44.168 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:44.165+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.166426+0000) 2020-02-05T21:26:44.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:44.166+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.166797+0000) 2020-02-05T21:26:44.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:44.166+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.167816+0000) 2020-02-05T21:26:44.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:44.166+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.167266+0000) 2020-02-05T21:26:44.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:44.168+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.168868+0000) 2020-02-05T21:26:44.170 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:44.168+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.168992+0000) 2020-02-05T21:26:44.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:44.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:44.172968+0000) 2020-02-05T21:26:44.176 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:44.174+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:45.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:45.845+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:45.846552+0000) 2020-02-05T21:26:46.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:46.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:46.743120+0000) 2020-02-05T21:26:46.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:46.989+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:46.989490+0000) 2020-02-05T21:26:46.991 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:46.989+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:46.989628+0000) 2020-02-05T21:26:47.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:47.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:47.000423+0000) 2020-02-05T21:26:47.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:47.028+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:47.028910+0000) 2020-02-05T21:26:49.099 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:49.097+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.097876+0000) 2020-02-05T21:26:49.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:49.180+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.181314+0000) 2020-02-05T21:26:49.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:49.180+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.181667+0000) 2020-02-05T21:26:49.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:49.182+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.182528+0000) 2020-02-05T21:26:49.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:49.182+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.182655+0000) 2020-02-05T21:26:49.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:49.184+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.184500+0000) 2020-02-05T21:26:49.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:49.182+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.183628+0000) 2020-02-05T21:26:49.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:49.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.184028+0000) 2020-02-05T21:26:49.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:49.182+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.183217+0000) 2020-02-05T21:26:49.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:49.183+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.184245+0000) 2020-02-05T21:26:49.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:49.183+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.184744+0000) 2020-02-05T21:26:49.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:49.184+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.184858+0000) 2020-02-05T21:26:49.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:49.185+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.185960+0000) 2020-02-05T21:26:49.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:49.184+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.185705+0000) 2020-02-05T21:26:49.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:49.189+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.190506+0000) 2020-02-05T21:26:49.192 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:49.189+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:49.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:49.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.382385+0000) 2020-02-05T21:26:49.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:49.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:49.991153+0000) 2020-02-05T21:26:50.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:50.223+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:50.224241+0000) 2020-02-05T21:26:50.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:50.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:50.242690+0000) 2020-02-05T21:26:50.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:50.252+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:50.252496+0000) 2020-02-05T21:26:52.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:52.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:52.232508+0000) 2020-02-05T21:26:52.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:52.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:52.633664+0000) 2020-02-05T21:26:54.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:54.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.195639+0000) 2020-02-05T21:26:54.198 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:54.196+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.197289+0000) 2020-02-05T21:26:54.199 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:54.197+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.197564+0000) 2020-02-05T21:26:54.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:54.197+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.198274+0000) 2020-02-05T21:26:54.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:54.198+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.198664+0000) 2020-02-05T21:26:54.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:54.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.198872+0000) 2020-02-05T21:26:54.200 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:54.198+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.199126+0000) 2020-02-05T21:26:54.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:54.198+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.199157+0000) 2020-02-05T21:26:54.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:54.198+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.199867+0000) 2020-02-05T21:26:54.202 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:54.199+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.199918+0000) 2020-02-05T21:26:54.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:54.200+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.200526+0000) 2020-02-05T21:26:54.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:54.199+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.200289+0000) 2020-02-05T21:26:54.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:54.200+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.201517+0000) 2020-02-05T21:26:54.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:54.204+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:54.205390+0000) 2020-02-05T21:26:54.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:54.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:55.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:55.846+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:55.846835+0000) 2020-02-05T21:26:56.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:56.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:56.743388+0000) 2020-02-05T21:26:56.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:56.989+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:56.989724+0000) 2020-02-05T21:26:56.991 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:56.989+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:56.989929+0000) 2020-02-05T21:26:57.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:56.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:57.000662+0000) 2020-02-05T21:26:57.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:57.040+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:57.040599+0000) 2020-02-05T21:26:59.099 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:59.097+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.098137+0000) 2020-02-05T21:26:59.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:59.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.212195+0000) 2020-02-05T21:26:59.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:26:59.211+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.212506+0000) 2020-02-05T21:26:59.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:26:59.213+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.213468+0000) 2020-02-05T21:26:59.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:26:59.214+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.214293+0000) 2020-02-05T21:26:59.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:26:59.214+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.214664+0000) 2020-02-05T21:26:59.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:26:59.214+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.215031+0000) 2020-02-05T21:26:59.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:26:59.215+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.215992+0000) 2020-02-05T21:26:59.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:26:59.215+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.216364+0000) 2020-02-05T21:26:59.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:26:59.216+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.216580+0000) 2020-02-05T21:26:59.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:26:59.216+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.216957+0000) 2020-02-05T21:26:59.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:26:59.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86165 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:26:59.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:59.214+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.215447+0000) 2020-02-05T21:26:59.228 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:26:59.215+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.216683+0000) 2020-02-05T21:26:59.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:59.217+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.218845+0000) 2020-02-05T21:26:59.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:59.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.219848+0000) 2020-02-05T21:26:59.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:26:59.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.382596+0000) 2020-02-05T21:26:59.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:26:59.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:26:59.991360+0000) 2020-02-05T21:27:00.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:00.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:00.224507+0000) 2020-02-05T21:27:00.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:00.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:00.242943+0000) 2020-02-05T21:27:00.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:00.252+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:00.252727+0000) 2020-02-05T21:27:02.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:02.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:02.232766+0000) 2020-02-05T21:27:02.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:02.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:02.633851+0000) 2020-02-05T21:27:04.230 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:04.227+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.229048+0000) 2020-02-05T21:27:04.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:04.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.229722+0000) 2020-02-05T21:27:04.232 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:04.229+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.230681+0000) 2020-02-05T21:27:04.232 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:04.229+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.230776+0000) 2020-02-05T21:27:04.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:04.230+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.230567+0000) 2020-02-05T21:27:04.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:04.230+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.230635+0000) 2020-02-05T21:27:04.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:04.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.231324+0000) 2020-02-05T21:27:04.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:04.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.232583+0000) 2020-02-05T21:27:04.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:04.230+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.232038+0000) 2020-02-05T21:27:04.234 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:04.231+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.232600+0000) 2020-02-05T21:27:04.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:04.231+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.231771+0000) 2020-02-05T21:27:04.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:04.232+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.232577+0000) 2020-02-05T21:27:04.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:04.232+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.232840+0000) 2020-02-05T21:27:04.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:04.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:04.237418+0000) 2020-02-05T21:27:04.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:04.237+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86205 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:05.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:05.846+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:05.847135+0000) 2020-02-05T21:27:06.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:06.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:06.743625+0000) 2020-02-05T21:27:06.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:06.989+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:06.989987+0000) 2020-02-05T21:27:06.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:06.989+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:06.990210+0000) 2020-02-05T21:27:07.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:06.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:07.000901+0000) 2020-02-05T21:27:07.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:07.047+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:07.048914+0000) 2020-02-05T21:27:09.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:09.098+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.098410+0000) 2020-02-05T21:27:09.246 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:09.244+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.245216+0000) 2020-02-05T21:27:09.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:09.245+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.246932+0000) 2020-02-05T21:27:09.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:09.246+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.247046+0000) 2020-02-05T21:27:09.249 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:09.246+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.247868+0000) 2020-02-05T21:27:09.249 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:09.247+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.248319+0000) 2020-02-05T21:27:09.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:09.248+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.248521+0000) 2020-02-05T21:27:09.250 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:09.247+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.248870+0000) 2020-02-05T21:27:09.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:09.248+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.249040+0000) 2020-02-05T21:27:09.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:09.248+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.249368+0000) 2020-02-05T21:27:09.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:09.249+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.249940+0000) 2020-02-05T21:27:09.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:09.250+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.250482+0000) 2020-02-05T21:27:09.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:09.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.249977+0000) 2020-02-05T21:27:09.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:09.250+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.250554+0000) 2020-02-05T21:27:09.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:09.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.252059+0000) 2020-02-05T21:27:09.255 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:09.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86248 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:09.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:09.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.382815+0000) 2020-02-05T21:27:09.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:09.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:09.991591+0000) 2020-02-05T21:27:10.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:10.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:10.224733+0000) 2020-02-05T21:27:10.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:10.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:10.243187+0000) 2020-02-05T21:27:10.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:10.252+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:10.252954+0000) 2020-02-05T21:27:11.797 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:27:11.804 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:27:12.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:12.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:12.232975+0000) 2020-02-05T21:27:12.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:12.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:12.634132+0000) 2020-02-05T21:27:14.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:14.258+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.259522+0000) 2020-02-05T21:27:14.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:14.259+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.260508+0000) 2020-02-05T21:27:14.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:14.259+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.260826+0000) 2020-02-05T21:27:14.262 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:14.260+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.261449+0000) 2020-02-05T21:27:14.263 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:14.261+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.262051+0000) 2020-02-05T21:27:14.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:14.262+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.262649+0000) 2020-02-05T21:27:14.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:14.262+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.262553+0000) 2020-02-05T21:27:14.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:14.262+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.263685+0000) 2020-02-05T21:27:14.265 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:14.262+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.263354+0000) 2020-02-05T21:27:14.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:14.264+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.265025+0000) 2020-02-05T21:27:14.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:14.264+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.264617+0000) 2020-02-05T21:27:14.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:14.264+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.265350+0000) 2020-02-05T21:27:14.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:14.268+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.269789+0000) 2020-02-05T21:27:14.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:14.269+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:14.270990+0000) 2020-02-05T21:27:14.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:14.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86289 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:15.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:15.846+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:15.847407+0000) 2020-02-05T21:27:16.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:16.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:16.743855+0000) 2020-02-05T21:27:16.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:16.990+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:16.990263+0000) 2020-02-05T21:27:16.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:16.990+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:16.990474+0000) 2020-02-05T21:27:17.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:17.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:17.001154+0000) 2020-02-05T21:27:17.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:17.057+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:17.058221+0000) 2020-02-05T21:27:19.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:19.098+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.098755+0000) 2020-02-05T21:27:19.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:19.276+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.277379+0000) 2020-02-05T21:27:19.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:19.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.278592+0000) 2020-02-05T21:27:19.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:19.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.279495+0000) 2020-02-05T21:27:19.281 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:19.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.279799+0000) 2020-02-05T21:27:19.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:19.279+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.280253+0000) 2020-02-05T21:27:19.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:19.280+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.280746+0000) 2020-02-05T21:27:19.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:19.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.280863+0000) 2020-02-05T21:27:19.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:19.281+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.281499+0000) 2020-02-05T21:27:19.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:19.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.282131+0000) 2020-02-05T21:27:19.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:19.281+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.282252+0000) 2020-02-05T21:27:19.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:19.282+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.282540+0000) 2020-02-05T21:27:19.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:19.282+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.282944+0000) 2020-02-05T21:27:19.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:19.282+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.283251+0000) 2020-02-05T21:27:19.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:19.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.287005+0000) 2020-02-05T21:27:19.289 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:19.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86331 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:19.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:19.381+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.383015+0000) 2020-02-05T21:27:19.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:19.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:19.991806+0000) 2020-02-05T21:27:20.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:20.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:20.224964+0000) 2020-02-05T21:27:20.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:20.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:20.243477+0000) 2020-02-05T21:27:20.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:20.252+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:20.253182+0000) 2020-02-05T21:27:22.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:22.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:22.233289+0000) 2020-02-05T21:27:22.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:22.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:22.634402+0000) 2020-02-05T21:27:24.294 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:24.292+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.293607+0000) 2020-02-05T21:27:24.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:24.292+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.294152+0000) 2020-02-05T21:27:24.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:24.293+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.295046+0000) 2020-02-05T21:27:24.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:24.294+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.295898+0000) 2020-02-05T21:27:24.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:24.295+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.296649+0000) 2020-02-05T21:27:24.298 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:24.296+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.297176+0000) 2020-02-05T21:27:24.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:24.296+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.297866+0000) 2020-02-05T21:27:24.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:24.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.298463+0000) 2020-02-05T21:27:24.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:24.297+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.298807+0000) 2020-02-05T21:27:24.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:24.297+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.299372+0000) 2020-02-05T21:27:24.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:24.298+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.300010+0000) 2020-02-05T21:27:24.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:24.299+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.300367+0000) 2020-02-05T21:27:24.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:24.298+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.300409+0000) 2020-02-05T21:27:24.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:24.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86371 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:24.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:24.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:24.305386+0000) 2020-02-05T21:27:25.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:25.846+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:25.847630+0000) 2020-02-05T21:27:26.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:26.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:26.744133+0000) 2020-02-05T21:27:26.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:26.989+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:26.990509+0000) 2020-02-05T21:27:26.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:26.989+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:26.990736+0000) 2020-02-05T21:27:27.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:27.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:27.001378+0000) 2020-02-05T21:27:27.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:27.066+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:27.066651+0000) 2020-02-05T21:27:29.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:29.097+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.099012+0000) 2020-02-05T21:27:29.311 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:29.309+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.310275+0000) 2020-02-05T21:27:29.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:29.309+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.310870+0000) 2020-02-05T21:27:29.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:29.310+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.311288+0000) 2020-02-05T21:27:29.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:29.310+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.311938+0000) 2020-02-05T21:27:29.315 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:29.310+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.311731+0000) 2020-02-05T21:27:29.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:29.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.312409+0000) 2020-02-05T21:27:29.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:29.310+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.312333+0000) 2020-02-05T21:27:29.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:29.311+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.312537+0000) 2020-02-05T21:27:29.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:29.311+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.313084+0000) 2020-02-05T21:27:29.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:29.312+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.314133+0000) 2020-02-05T21:27:29.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:29.313+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.314598+0000) 2020-02-05T21:27:29.318 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:29.312+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.313930+0000) 2020-02-05T21:27:29.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:29.314+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.315402+0000) 2020-02-05T21:27:29.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:29.318+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.319781+0000) 2020-02-05T21:27:29.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:29.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86414 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:29.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:29.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.383223+0000) 2020-02-05T21:27:29.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:29.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:29.992017+0000) 2020-02-05T21:27:30.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:30.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:30.225220+0000) 2020-02-05T21:27:30.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:30.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:30.243688+0000) 2020-02-05T21:27:30.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:30.252+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:30.253344+0000) 2020-02-05T21:27:32.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:32.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:32.233590+0000) 2020-02-05T21:27:32.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:32.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:32.634640+0000) 2020-02-05T21:27:34.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:34.326+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.327656+0000) 2020-02-05T21:27:34.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:34.327+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.328682+0000) 2020-02-05T21:27:34.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:34.327+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.328932+0000) 2020-02-05T21:27:34.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:34.328+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.329625+0000) 2020-02-05T21:27:34.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:34.329+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.330741+0000) 2020-02-05T21:27:34.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:34.329+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.331173+0000) 2020-02-05T21:27:34.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:34.329+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.331177+0000) 2020-02-05T21:27:34.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:34.331+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.332505+0000) 2020-02-05T21:27:34.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:34.331+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.332756+0000) 2020-02-05T21:27:34.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:34.331+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.333031+0000) 2020-02-05T21:27:34.336 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:34.332+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.333395+0000) 2020-02-05T21:27:34.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:34.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.333396+0000) 2020-02-05T21:27:34.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:34.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.335850+0000) 2020-02-05T21:27:34.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:34.335+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:34.336932+0000) 2020-02-05T21:27:34.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:34.336+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:35.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:35.847+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:35.847860+0000) 2020-02-05T21:27:36.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:36.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:36.744367+0000) 2020-02-05T21:27:36.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:36.989+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:36.990761+0000) 2020-02-05T21:27:36.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:36.989+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:36.991015+0000) 2020-02-05T21:27:37.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:37.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:37.001644+0000) 2020-02-05T21:27:37.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:37.074+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:37.075132+0000) 2020-02-05T21:27:39.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:39.098+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.099275+0000) 2020-02-05T21:27:39.345 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:39.342+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.343539+0000) 2020-02-05T21:27:39.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:39.343+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.344364+0000) 2020-02-05T21:27:39.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:39.344+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.345321+0000) 2020-02-05T21:27:39.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:39.344+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.345812+0000) 2020-02-05T21:27:39.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:39.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.346527+0000) 2020-02-05T21:27:39.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:39.346+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.347434+0000) 2020-02-05T21:27:39.349 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:39.345+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.347181+0000) 2020-02-05T21:27:39.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:39.347+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.348390+0000) 2020-02-05T21:27:39.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:39.346+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.347973+0000) 2020-02-05T21:27:39.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:39.347+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.348854+0000) 2020-02-05T21:27:39.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:39.347+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.349233+0000) 2020-02-05T21:27:39.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:39.348+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.349696+0000) 2020-02-05T21:27:39.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:39.349+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.350447+0000) 2020-02-05T21:27:39.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:39.350+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.352035+0000) 2020-02-05T21:27:39.357 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:39.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:39.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:39.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.383394+0000) 2020-02-05T21:27:39.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:39.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:39.992257+0000) 2020-02-05T21:27:40.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:40.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:40.225480+0000) 2020-02-05T21:27:40.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:40.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:40.243922+0000) 2020-02-05T21:27:40.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:40.253+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:40.253549+0000) 2020-02-05T21:27:41.951 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:27:41.958 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:27:42.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:42.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:42.233864+0000) 2020-02-05T21:27:42.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:42.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:42.634877+0000) 2020-02-05T21:27:44.363 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:44.360+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.361407+0000) 2020-02-05T21:27:44.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:44.362+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.363173+0000) 2020-02-05T21:27:44.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:44.362+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.363790+0000) 2020-02-05T21:27:44.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:44.363+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.364460+0000) 2020-02-05T21:27:44.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:44.364+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.365720+0000) 2020-02-05T21:27:44.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:44.365+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.366655+0000) 2020-02-05T21:27:44.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:44.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.367949+0000) 2020-02-05T21:27:44.372 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:44.363+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.365013+0000) 2020-02-05T21:27:44.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:44.364+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.366289+0000) 2020-02-05T21:27:44.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:44.365+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.367133+0000) 2020-02-05T21:27:44.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:44.365+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.367166+0000) 2020-02-05T21:27:44.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:44.366+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.367460+0000) 2020-02-05T21:27:44.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:44.366+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.367704+0000) 2020-02-05T21:27:44.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:44.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86537 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:44.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:44.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:44.374148+0000) 2020-02-05T21:27:45.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:45.847+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:45.848129+0000) 2020-02-05T21:27:46.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:46.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:46.744619+0000) 2020-02-05T21:27:46.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:46.990+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:46.991013+0000) 2020-02-05T21:27:46.993 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:46.990+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:46.991230+0000) 2020-02-05T21:27:47.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:47.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:47.001869+0000) 2020-02-05T21:27:47.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:47.083+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:47.083782+0000) 2020-02-05T21:27:49.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:49.098+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.099507+0000) 2020-02-05T21:27:49.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:49.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.378576+0000) 2020-02-05T21:27:49.380 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:49.378+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.379449+0000) 2020-02-05T21:27:49.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:49.378+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.380262+0000) 2020-02-05T21:27:49.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:49.380+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.381853+0000) 2020-02-05T21:27:49.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:49.381+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382119+0000) 2020-02-05T21:27:49.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:49.381+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382500+0000) 2020-02-05T21:27:49.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:49.381+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382467+0000) 2020-02-05T21:27:49.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:49.381+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382582+0000) 2020-02-05T21:27:49.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:49.381+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382878+0000) 2020-02-05T21:27:49.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:49.381+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.382847+0000) 2020-02-05T21:27:49.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:49.382+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.383486+0000) 2020-02-05T21:27:49.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:49.382+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.383655+0000) 2020-02-05T21:27:49.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:49.382+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.384302+0000) 2020-02-05T21:27:49.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:49.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.383634+0000) 2020-02-05T21:27:49.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:49.386+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.387995+0000) 2020-02-05T21:27:49.390 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:49.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:49.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:49.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:49.992467+0000) 2020-02-05T21:27:50.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:50.224+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:50.225700+0000) 2020-02-05T21:27:50.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:50.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:50.244148+0000) 2020-02-05T21:27:50.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:50.253+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:50.253765+0000) 2020-02-05T21:27:52.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:52.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:52.234141+0000) 2020-02-05T21:27:52.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:52.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:52.635145+0000) 2020-02-05T21:27:54.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:54.394+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.394672+0000) 2020-02-05T21:27:54.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:54.395+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.395381+0000) 2020-02-05T21:27:54.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:54.396+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.396363+0000) 2020-02-05T21:27:54.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:54.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.397581+0000) 2020-02-05T21:27:54.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:54.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.398283+0000) 2020-02-05T21:27:54.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:54.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.398034+0000) 2020-02-05T21:27:54.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:54.397+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.398759+0000) 2020-02-05T21:27:54.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:54.398+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.399771+0000) 2020-02-05T21:27:54.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:54.397+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.399376+0000) 2020-02-05T21:27:54.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:54.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.400166+0000) 2020-02-05T21:27:54.402 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:54.401+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.401135+0000) 2020-02-05T21:27:54.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:54.401+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.401179+0000) 2020-02-05T21:27:54.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:54.400+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.401472+0000) 2020-02-05T21:27:54.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:54.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86620 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:54.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:54.406+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:54.406555+0000) 2020-02-05T21:27:55.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:55.848+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:55.848321+0000) 2020-02-05T21:27:56.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:56.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:56.744858+0000) 2020-02-05T21:27:56.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:56.990+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:56.991270+0000) 2020-02-05T21:27:56.993 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:56.990+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:56.991498+0000) 2020-02-05T21:27:57.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:57.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:57.002167+0000) 2020-02-05T21:27:57.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:57.092+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:57.092204+0000) 2020-02-05T21:27:59.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:59.099+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.099777+0000) 2020-02-05T21:27:59.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:59.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.383828+0000) 2020-02-05T21:27:59.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:27:59.409+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.410415+0000) 2020-02-05T21:27:59.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:27:59.409+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.410881+0000) 2020-02-05T21:27:59.412 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:27:59.410+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.411146+0000) 2020-02-05T21:27:59.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:27:59.412+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.413678+0000) 2020-02-05T21:27:59.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:27:59.413+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.414568+0000) 2020-02-05T21:27:59.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:27:59.413+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.415122+0000) 2020-02-05T21:27:59.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:27:59.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.414987+0000) 2020-02-05T21:27:59.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:27:59.414+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.415834+0000) 2020-02-05T21:27:59.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:27:59.414+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.415992+0000) 2020-02-05T21:27:59.418 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:27:59.415+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.416638+0000) 2020-02-05T21:27:59.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:59.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.416741+0000) 2020-02-05T21:27:59.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:59.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.417232+0000) 2020-02-05T21:27:59.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:27:59.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.417638+0000) 2020-02-05T21:27:59.420 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:59.416+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.417094+0000) 2020-02-05T21:27:59.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:27:59.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:27:59.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:27:59.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:27:59.992675+0000) 2020-02-05T21:28:00.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:00.225+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:00.225934+0000) 2020-02-05T21:28:00.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:00.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:00.244359+0000) 2020-02-05T21:28:00.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:00.253+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:00.254038+0000) 2020-02-05T21:28:02.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:02.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:02.234414+0000) 2020-02-05T21:28:02.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:02.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:02.635409+0000) 2020-02-05T21:28:04.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:04.427+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.428284+0000) 2020-02-05T21:28:04.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:04.427+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.428965+0000) 2020-02-05T21:28:04.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:04.428+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.429507+0000) 2020-02-05T21:28:04.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:04.428+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.429782+0000) 2020-02-05T21:28:04.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:04.429+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.430493+0000) 2020-02-05T21:28:04.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:04.430+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.431195+0000) 2020-02-05T21:28:04.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:04.430+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.431518+0000) 2020-02-05T21:28:04.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:04.430+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.431672+0000) 2020-02-05T21:28:04.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:04.431+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.432691+0000) 2020-02-05T21:28:04.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:04.432+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.433195+0000) 2020-02-05T21:28:04.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:04.431+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.433100+0000) 2020-02-05T21:28:04.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:04.431+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.433320+0000) 2020-02-05T21:28:04.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:04.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.435165+0000) 2020-02-05T21:28:04.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:04.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:04.436747+0000) 2020-02-05T21:28:04.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:04.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86702 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:05.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:05.847+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:05.848550+0000) 2020-02-05T21:28:06.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:06.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:06.745124+0000) 2020-02-05T21:28:06.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:06.990+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:06.991530+0000) 2020-02-05T21:28:06.993 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:06.990+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:06.991729+0000) 2020-02-05T21:28:07.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:07.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:07.002441+0000) 2020-02-05T21:28:07.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:07.098+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:07.099683+0000) 2020-02-05T21:28:09.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:09.099+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.100033+0000) 2020-02-05T21:28:09.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:09.382+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.384040+0000) 2020-02-05T21:28:09.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:09.444+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.445192+0000) 2020-02-05T21:28:09.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:09.444+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.445753+0000) 2020-02-05T21:28:09.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:09.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.447463+0000) 2020-02-05T21:28:09.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:09.445+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.446881+0000) 2020-02-05T21:28:09.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:09.447+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.448898+0000) 2020-02-05T21:28:09.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:09.447+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.448741+0000) 2020-02-05T21:28:09.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:09.448+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.449650+0000) 2020-02-05T21:28:09.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:09.449+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.450536+0000) 2020-02-05T21:28:09.452 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:09.448+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.450258+0000) 2020-02-05T21:28:09.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:09.449+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.450784+0000) 2020-02-05T21:28:09.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:09.450+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.451668+0000) 2020-02-05T21:28:09.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:09.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.451141+0000) 2020-02-05T21:28:09.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:09.450+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.451568+0000) 2020-02-05T21:28:09.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:09.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.456203+0000) 2020-02-05T21:28:09.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:09.455+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:09.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:09.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:09.992829+0000) 2020-02-05T21:28:10.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:10.225+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:10.226206+0000) 2020-02-05T21:28:10.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:10.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:10.244584+0000) 2020-02-05T21:28:10.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:10.253+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:10.254265+0000) 2020-02-05T21:28:12.101 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:28:12.107 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:28:12.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:12.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:12.234706+0000) 2020-02-05T21:28:12.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:12.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:12.635640+0000) 2020-02-05T21:28:14.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:14.461+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.462466+0000) 2020-02-05T21:28:14.465 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:14.463+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.464430+0000) 2020-02-05T21:28:14.466 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:14.463+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.465040+0000) 2020-02-05T21:28:14.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:14.465+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.465538+0000) 2020-02-05T21:28:14.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:14.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.465932+0000) 2020-02-05T21:28:14.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:14.466+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.467118+0000) 2020-02-05T21:28:14.468 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:14.467+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.467551+0000) 2020-02-05T21:28:14.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:14.467+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.468054+0000) 2020-02-05T21:28:14.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:14.467+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.468727+0000) 2020-02-05T21:28:14.470 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:14.467+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.468079+0000) 2020-02-05T21:28:14.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:14.468+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.468507+0000) 2020-02-05T21:28:14.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:14.467+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.469058+0000) 2020-02-05T21:28:14.471 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:14.467+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.469051+0000) 2020-02-05T21:28:14.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:14.472+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:14.473121+0000) 2020-02-05T21:28:14.474 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:14.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:15.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:15.848+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:15.848767+0000) 2020-02-05T21:28:16.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:16.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:16.745358+0000) 2020-02-05T21:28:16.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:16.991+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:16.991781+0000) 2020-02-05T21:28:16.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:16.991+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:16.991971+0000) 2020-02-05T21:28:17.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:17.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:17.002652+0000) 2020-02-05T21:28:17.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:17.107+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:17.108202+0000) 2020-02-05T21:28:19.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:19.099+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.100282+0000) 2020-02-05T21:28:19.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:19.383+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.384355+0000) 2020-02-05T21:28:19.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:19.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.479200+0000) 2020-02-05T21:28:19.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:19.479+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.480248+0000) 2020-02-05T21:28:19.483 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:19.481+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.481864+0000) 2020-02-05T21:28:19.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:19.481+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.482244+0000) 2020-02-05T21:28:19.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:19.481+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.482523+0000) 2020-02-05T21:28:19.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:19.482+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.483279+0000) 2020-02-05T21:28:19.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:19.482+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.482990+0000) 2020-02-05T21:28:19.486 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:19.482+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.483928+0000) 2020-02-05T21:28:19.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:19.483+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.484611+0000) 2020-02-05T21:28:19.487 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:19.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.483730+0000) 2020-02-05T21:28:19.487 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:19.483+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.484297+0000) 2020-02-05T21:28:19.488 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:19.485+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.485508+0000) 2020-02-05T21:28:19.490 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:19.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:19.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:19.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.490470+0000) 2020-02-05T21:28:19.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:19.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.490828+0000) 2020-02-05T21:28:19.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:19.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:19.993039+0000) 2020-02-05T21:28:20.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:20.226+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:20.226470+0000) 2020-02-05T21:28:20.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:20.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:20.244791+0000) 2020-02-05T21:28:20.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:20.253+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:20.254449+0000) 2020-02-05T21:28:22.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:22.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:22.235031+0000) 2020-02-05T21:28:22.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:22.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:22.635875+0000) 2020-02-05T21:28:24.495 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:24.492+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.493657+0000) 2020-02-05T21:28:24.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:24.496+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.497267+0000) 2020-02-05T21:28:24.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:24.496+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.497856+0000) 2020-02-05T21:28:24.499 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:24.497+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.498115+0000) 2020-02-05T21:28:24.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:24.497+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.498472+0000) 2020-02-05T21:28:24.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:24.498+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.498668+0000) 2020-02-05T21:28:24.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:24.498+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.499216+0000) 2020-02-05T21:28:24.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:24.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.499777+0000) 2020-02-05T21:28:24.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:24.498+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.499984+0000) 2020-02-05T21:28:24.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:24.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.500194+0000) 2020-02-05T21:28:24.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:24.500+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.500607+0000) 2020-02-05T21:28:24.504 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:24.500+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.500780+0000) 2020-02-05T21:28:24.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:24.500+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.501036+0000) 2020-02-05T21:28:24.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:24.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86866 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:24.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:24.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:24.506731+0000) 2020-02-05T21:28:25.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:25.848+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:25.848970+0000) 2020-02-05T21:28:26.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:26.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:26.745611+0000) 2020-02-05T21:28:26.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:26.991+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:26.992009+0000) 2020-02-05T21:28:26.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:26.991+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:26.992205+0000) 2020-02-05T21:28:27.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:27.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:27.002882+0000) 2020-02-05T21:28:27.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:27.117+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:27.117610+0000) 2020-02-05T21:28:29.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:29.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.100525+0000) 2020-02-05T21:28:29.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:29.383+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.384605+0000) 2020-02-05T21:28:29.514 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:29.511+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.512336+0000) 2020-02-05T21:28:29.514 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:29.512+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.513015+0000) 2020-02-05T21:28:29.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:29.512+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.513527+0000) 2020-02-05T21:28:29.516 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:29.513+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.514302+0000) 2020-02-05T21:28:29.516 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:29.513+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.514339+0000) 2020-02-05T21:28:29.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:29.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.515129+0000) 2020-02-05T21:28:29.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:29.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.515685+0000) 2020-02-05T21:28:29.518 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:29.515+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.515673+0000) 2020-02-05T21:28:29.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:29.515+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.515493+0000) 2020-02-05T21:28:29.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:29.514+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.514831+0000) 2020-02-05T21:28:29.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:29.516+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.517183+0000) 2020-02-05T21:28:29.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:29.515+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.516241+0000) 2020-02-05T21:28:29.520 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:29.517+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.517698+0000) 2020-02-05T21:28:29.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:29.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.521992+0000) 2020-02-05T21:28:29.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:29.522+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:29.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:29.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:29.993238+0000) 2020-02-05T21:28:30.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:30.226+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:30.226686+0000) 2020-02-05T21:28:30.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:30.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:30.244968+0000) 2020-02-05T21:28:30.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:30.254+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:30.254674+0000) 2020-02-05T21:28:32.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:32.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:32.235282+0000) 2020-02-05T21:28:32.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:32.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:32.636162+0000) 2020-02-05T21:28:34.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:34.526+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.527991+0000) 2020-02-05T21:28:34.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:34.528+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.529179+0000) 2020-02-05T21:28:34.533 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:34.530+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.530888+0000) 2020-02-05T21:28:34.533 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:34.530+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.531142+0000) 2020-02-05T21:28:34.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:34.531+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.531994+0000) 2020-02-05T21:28:34.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:34.531+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.532369+0000) 2020-02-05T21:28:34.534 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:34.531+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.532955+0000) 2020-02-05T21:28:34.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:34.532+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.533222+0000) 2020-02-05T21:28:34.535 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:34.533+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.534016+0000) 2020-02-05T21:28:34.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:34.532+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.533840+0000) 2020-02-05T21:28:34.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:34.534+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.534493+0000) 2020-02-05T21:28:34.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:34.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.534435+0000) 2020-02-05T21:28:34.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:34.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:34.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:34.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.541237+0000) 2020-02-05T21:28:34.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:34.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:34.541567+0000) 2020-02-05T21:28:35.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:35.848+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:35.849196+0000) 2020-02-05T21:28:36.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:36.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:36.745843+0000) 2020-02-05T21:28:36.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:36.992+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:36.992264+0000) 2020-02-05T21:28:36.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:36.992+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:36.992487+0000) 2020-02-05T21:28:37.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:37.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:37.003171+0000) 2020-02-05T21:28:37.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:37.126+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:37.126498+0000) 2020-02-05T21:28:39.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:39.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.100810+0000) 2020-02-05T21:28:39.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:39.384+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.384837+0000) 2020-02-05T21:28:39.546 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:39.543+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.544701+0000) 2020-02-05T21:28:39.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:39.544+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.545727+0000) 2020-02-05T21:28:39.547 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:39.544+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.545945+0000) 2020-02-05T21:28:39.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:39.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.547057+0000) 2020-02-05T21:28:39.549 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:39.547+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.547481+0000) 2020-02-05T21:28:39.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:39.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.548999+0000) 2020-02-05T21:28:39.551 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:39.548+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.548807+0000) 2020-02-05T21:28:39.551 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:39.548+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.549007+0000) 2020-02-05T21:28:39.551 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:39.547+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.548249+0000) 2020-02-05T21:28:39.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:39.549+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.549701+0000) 2020-02-05T21:28:39.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:39.549+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.550493+0000) 2020-02-05T21:28:39.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:39.549+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.550975+0000) 2020-02-05T21:28:39.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:39.550+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.550969+0000) 2020-02-05T21:28:39.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:39.551+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.552608+0000) 2020-02-05T21:28:39.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:39.555+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 86993 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:39.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:39.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:39.993446+0000) 2020-02-05T21:28:40.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:40.226+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:40.226896+0000) 2020-02-05T21:28:40.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:40.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:40.245179+0000) 2020-02-05T21:28:40.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:40.254+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:40.254881+0000) 2020-02-05T21:28:42.231 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:28:42.238 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:28:42.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:42.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:42.235540+0000) 2020-02-05T21:28:42.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:42.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:42.636402+0000) 2020-02-05T21:28:44.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:44.561+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.562540+0000) 2020-02-05T21:28:44.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:44.562+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.563643+0000) 2020-02-05T21:28:44.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:44.563+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.564054+0000) 2020-02-05T21:28:44.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:44.563+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.564565+0000) 2020-02-05T21:28:44.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:44.563+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.564904+0000) 2020-02-05T21:28:44.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:44.564+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.565505+0000) 2020-02-05T21:28:44.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:44.565+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.566073+0000) 2020-02-05T21:28:44.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:44.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.567183+0000) 2020-02-05T21:28:44.568 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:44.566+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.567382+0000) 2020-02-05T21:28:44.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:44.566+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.567330+0000) 2020-02-05T21:28:44.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:44.566+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.567222+0000) 2020-02-05T21:28:44.569 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:44.567+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.567771+0000) 2020-02-05T21:28:44.570 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:44.567+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.568257+0000) 2020-02-05T21:28:44.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:44.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:44.572105+0000) 2020-02-05T21:28:44.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:44.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:45.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:45.849+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:45.849433+0000) 2020-02-05T21:28:46.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:46.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:46.746123+0000) 2020-02-05T21:28:46.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:46.992+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:46.992548+0000) 2020-02-05T21:28:46.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:46.992+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:46.992771+0000) 2020-02-05T21:28:47.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:47.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:47.003401+0000) 2020-02-05T21:28:47.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:47.134+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:47.134991+0000) 2020-02-05T21:28:49.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:49.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.101050+0000) 2020-02-05T21:28:49.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:49.385+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.385064+0000) 2020-02-05T21:28:49.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:49.579+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.579745+0000) 2020-02-05T21:28:49.582 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:49.580+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.580419+0000) 2020-02-05T21:28:49.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:49.581+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.581062+0000) 2020-02-05T21:28:49.582 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:49.581+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.581586+0000) 2020-02-05T21:28:49.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:49.581+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.582046+0000) 2020-02-05T21:28:49.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:49.582+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.582305+0000) 2020-02-05T21:28:49.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:49.582+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.582949+0000) 2020-02-05T21:28:49.584 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:49.582+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.582710+0000) 2020-02-05T21:28:49.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:49.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.583457+0000) 2020-02-05T21:28:49.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:49.583+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.583637+0000) 2020-02-05T21:28:49.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:49.583+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.584264+0000) 2020-02-05T21:28:49.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:49.584+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.584941+0000) 2020-02-05T21:28:49.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:49.584+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.584796+0000) 2020-02-05T21:28:49.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:49.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:49.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:49.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.589465+0000) 2020-02-05T21:28:49.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:49.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:49.993667+0000) 2020-02-05T21:28:50.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:50.227+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:50.227112+0000) 2020-02-05T21:28:50.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:50.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:50.245388+0000) 2020-02-05T21:28:50.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:50.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:50.255125+0000) 2020-02-05T21:28:52.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:52.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:52.235803+0000) 2020-02-05T21:28:52.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:52.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:52.636637+0000) 2020-02-05T21:28:54.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:54.592+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.593929+0000) 2020-02-05T21:28:54.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:54.594+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.595192+0000) 2020-02-05T21:28:54.597 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:54.594+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.595745+0000) 2020-02-05T21:28:54.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:54.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.596674+0000) 2020-02-05T21:28:54.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:54.595+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.596842+0000) 2020-02-05T21:28:54.599 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:54.596+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.597276+0000) 2020-02-05T21:28:54.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:54.596+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.598007+0000) 2020-02-05T21:28:54.600 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:54.597+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.598505+0000) 2020-02-05T21:28:54.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:54.597+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.599183+0000) 2020-02-05T21:28:54.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:54.598+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.599480+0000) 2020-02-05T21:28:54.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:54.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.603976+0000) 2020-02-05T21:28:54.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:54.603+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.604245+0000) 2020-02-05T21:28:54.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:54.599+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.600486+0000) 2020-02-05T21:28:54.609 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:54.599+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:54.600918+0000) 2020-02-05T21:28:54.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:54.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:55.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:55.848+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:55.849661+0000) 2020-02-05T21:28:56.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:56.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:56.746354+0000) 2020-02-05T21:28:56.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:56.991+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:56.992781+0000) 2020-02-05T21:28:56.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:56.991+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:56.993041+0000) 2020-02-05T21:28:57.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:57.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:57.003627+0000) 2020-02-05T21:28:57.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:57.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:57.143746+0000) 2020-02-05T21:28:59.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:59.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.101302+0000) 2020-02-05T21:28:59.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:59.384+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.385305+0000) 2020-02-05T21:28:59.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:28:59.609+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.610557+0000) 2020-02-05T21:28:59.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:28:59.609+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.610937+0000) 2020-02-05T21:28:59.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:28:59.610+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.612167+0000) 2020-02-05T21:28:59.614 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:28:59.611+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.612608+0000) 2020-02-05T21:28:59.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:28:59.612+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.613599+0000) 2020-02-05T21:28:59.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:28:59.612+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.614149+0000) 2020-02-05T21:28:59.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:59.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.614466+0000) 2020-02-05T21:28:59.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:59.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.614988+0000) 2020-02-05T21:28:59.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:28:59.613+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.615151+0000) 2020-02-05T21:28:59.617 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:28:59.614+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.615759+0000) 2020-02-05T21:28:59.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:28:59.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.616057+0000) 2020-02-05T21:28:59.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:59.615+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.616578+0000) 2020-02-05T21:28:59.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:28:59.614+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.615938+0000) 2020-02-05T21:28:59.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:28:59.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:28:59.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:28:59.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.622105+0000) 2020-02-05T21:28:59.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:28:59.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:28:59.993948+0000) 2020-02-05T21:29:00.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:00.226+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:00.227324+0000) 2020-02-05T21:29:00.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:00.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:00.245648+0000) 2020-02-05T21:29:00.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:00.254+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:00.255332+0000) 2020-02-05T21:29:02.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:02.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:02.236052+0000) 2020-02-05T21:29:02.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:02.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:02.636902+0000) 2020-02-05T21:29:04.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:04.626+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.627310+0000) 2020-02-05T21:29:04.629 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:04.624+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.626095+0000) 2020-02-05T21:29:04.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:04.627+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.628409+0000) 2020-02-05T21:29:04.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:04.627+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.629021+0000) 2020-02-05T21:29:04.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:04.628+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.629742+0000) 2020-02-05T21:29:04.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:04.629+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.630687+0000) 2020-02-05T21:29:04.632 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:04.629+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.630704+0000) 2020-02-05T21:29:04.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:04.630+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.631213+0000) 2020-02-05T21:29:04.633 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:04.630+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.631928+0000) 2020-02-05T21:29:04.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:04.630+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.632204+0000) 2020-02-05T21:29:04.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:04.631+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.632806+0000) 2020-02-05T21:29:04.634 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:04.631+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.632975+0000) 2020-02-05T21:29:04.635 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:04.631+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.633020+0000) 2020-02-05T21:29:04.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:04.636+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:04.637304+0000) 2020-02-05T21:29:04.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:04.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:05.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:05.849+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:05.849904+0000) 2020-02-05T21:29:06.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:06.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:06.746589+0000) 2020-02-05T21:29:06.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:06.992+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:06.993042+0000) 2020-02-05T21:29:06.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:06.992+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:06.993236+0000) 2020-02-05T21:29:07.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:07.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:07.003807+0000) 2020-02-05T21:29:07.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:07.151+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:07.152498+0000) 2020-02-05T21:29:09.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:09.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.101637+0000) 2020-02-05T21:29:09.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:09.384+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.385582+0000) 2020-02-05T21:29:09.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:09.641+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.642751+0000) 2020-02-05T21:29:09.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:09.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.643433+0000) 2020-02-05T21:29:09.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:09.641+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.643359+0000) 2020-02-05T21:29:09.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:09.643+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.644490+0000) 2020-02-05T21:29:09.646 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:09.643+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.644895+0000) 2020-02-05T21:29:09.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:09.644+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.645227+0000) 2020-02-05T21:29:09.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:09.644+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.645611+0000) 2020-02-05T21:29:09.647 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:09.643+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.645171+0000) 2020-02-05T21:29:09.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:09.645+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.646304+0000) 2020-02-05T21:29:09.648 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:09.644+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.646364+0000) 2020-02-05T21:29:09.648 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:09.645+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.647009+0000) 2020-02-05T21:29:09.648 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:09.645+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.647182+0000) 2020-02-05T21:29:09.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:09.646+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.647772+0000) 2020-02-05T21:29:09.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:09.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.651683+0000) 2020-02-05T21:29:09.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:09.651+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:09.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:09.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:09.994193+0000) 2020-02-05T21:29:10.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:10.226+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:10.227554+0000) 2020-02-05T21:29:10.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:10.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:10.245859+0000) 2020-02-05T21:29:10.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:10.254+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:10.255506+0000) 2020-02-05T21:29:12.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:12.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:12.236299+0000) 2020-02-05T21:29:12.374 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:29:12.380 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:29:12.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:12.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:12.637194+0000) 2020-02-05T21:29:14.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:14.658+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.659442+0000) 2020-02-05T21:29:14.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:14.658+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.659772+0000) 2020-02-05T21:29:14.661 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:14.658+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.659855+0000) 2020-02-05T21:29:14.662 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:14.659+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.660551+0000) 2020-02-05T21:29:14.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:14.660+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.661321+0000) 2020-02-05T21:29:14.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:14.660+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.661652+0000) 2020-02-05T21:29:14.663 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:14.660+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.661898+0000) 2020-02-05T21:29:14.664 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:14.661+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.662477+0000) 2020-02-05T21:29:14.664 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:14.661+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.662850+0000) 2020-02-05T21:29:14.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:14.662+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.663250+0000) 2020-02-05T21:29:14.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:14.662+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.663511+0000) 2020-02-05T21:29:14.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:14.662+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.664371+0000) 2020-02-05T21:29:14.666 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:14.663+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.664496+0000) 2020-02-05T21:29:14.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:14.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:14.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:14.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:14.670900+0000) 2020-02-05T21:29:15.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:15.849+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:15.850134+0000) 2020-02-05T21:29:16.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:16.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:16.746820+0000) 2020-02-05T21:29:16.995 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:16.992+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:16.993274+0000) 2020-02-05T21:29:16.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:16.992+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:16.993503+0000) 2020-02-05T21:29:17.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:17.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:17.004036+0000) 2020-02-05T21:29:17.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:17.160+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:17.161102+0000) 2020-02-05T21:29:19.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:19.100+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.101887+0000) 2020-02-05T21:29:19.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:19.384+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.385832+0000) 2020-02-05T21:29:19.676 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:19.673+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.675021+0000) 2020-02-05T21:29:19.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:19.674+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.675671+0000) 2020-02-05T21:29:19.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:19.674+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.676372+0000) 2020-02-05T21:29:19.678 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:19.675+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.676997+0000) 2020-02-05T21:29:19.678 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:19.676+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.677482+0000) 2020-02-05T21:29:19.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:19.676+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.677935+0000) 2020-02-05T21:29:19.679 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:19.676+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.678190+0000) 2020-02-05T21:29:19.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:19.677+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.679019+0000) 2020-02-05T21:29:19.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:19.678+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.679376+0000) 2020-02-05T21:29:19.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:19.678+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.679783+0000) 2020-02-05T21:29:19.682 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:19.679+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.680808+0000) 2020-02-05T21:29:19.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:19.679+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.680975+0000) 2020-02-05T21:29:19.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:19.680+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.681639+0000) 2020-02-05T21:29:19.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:19.684+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.685490+0000) 2020-02-05T21:29:19.688 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:19.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:19.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:19.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:19.994405+0000) 2020-02-05T21:29:20.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:20.227+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:20.227771+0000) 2020-02-05T21:29:20.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:20.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:20.246085+0000) 2020-02-05T21:29:20.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:20.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:20.255713+0000) 2020-02-05T21:29:22.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:22.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:22.236536+0000) 2020-02-05T21:29:22.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:22.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:22.637468+0000) 2020-02-05T21:29:24.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:24.690+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.692017+0000) 2020-02-05T21:29:24.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:24.692+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.693261+0000) 2020-02-05T21:29:24.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:24.693+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.694452+0000) 2020-02-05T21:29:24.696 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:24.694+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.695585+0000) 2020-02-05T21:29:24.697 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:24.693+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.695156+0000) 2020-02-05T21:29:24.697 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:24.694+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.695847+0000) 2020-02-05T21:29:24.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:24.694+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.695998+0000) 2020-02-05T21:29:24.698 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:24.695+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.697022+0000) 2020-02-05T21:29:24.698 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:24.695+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.696672+0000) 2020-02-05T21:29:24.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:24.696+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.697670+0000) 2020-02-05T21:29:24.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:24.696+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.697844+0000) 2020-02-05T21:29:24.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:24.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.698576+0000) 2020-02-05T21:29:24.700 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:24.697+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.698929+0000) 2020-02-05T21:29:24.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:24.699+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:24.700859+0000) 2020-02-05T21:29:24.705 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:24.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:25.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:25.849+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:25.850358+0000) 2020-02-05T21:29:26.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:26.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:26.747057+0000) 2020-02-05T21:29:26.995 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:26.992+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:26.993514+0000) 2020-02-05T21:29:26.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:26.992+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:26.993724+0000) 2020-02-05T21:29:27.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:27.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:27.004319+0000) 2020-02-05T21:29:27.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:27.169+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:27.169711+0000) 2020-02-05T21:29:29.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:29.101+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.102127+0000) 2020-02-05T21:29:29.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:29.385+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.386020+0000) 2020-02-05T21:29:29.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:29.707+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.708801+0000) 2020-02-05T21:29:29.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:29.708+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.709579+0000) 2020-02-05T21:29:29.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:29.710+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.711281+0000) 2020-02-05T21:29:29.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:29.710+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.711442+0000) 2020-02-05T21:29:29.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:29.710+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.712271+0000) 2020-02-05T21:29:29.714 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:29.712+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.713345+0000) 2020-02-05T21:29:29.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:29.711+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.713199+0000) 2020-02-05T21:29:29.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:29.712+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.713992+0000) 2020-02-05T21:29:29.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:29.713+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.714995+0000) 2020-02-05T21:29:29.716 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:29.713+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.714821+0000) 2020-02-05T21:29:29.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:29.714+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.715413+0000) 2020-02-05T21:29:29.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:29.714+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.715936+0000) 2020-02-05T21:29:29.717 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:29.713+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.715308+0000) 2020-02-05T21:29:29.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:29.719+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.720211+0000) 2020-02-05T21:29:29.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:29.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:29.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:29.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:29.994674+0000) 2020-02-05T21:29:30.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:30.227+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:30.228005+0000) 2020-02-05T21:29:30.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:30.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:30.246354+0000) 2020-02-05T21:29:30.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:30.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:30.255916+0000) 2020-02-05T21:29:32.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:32.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:32.236781+0000) 2020-02-05T21:29:32.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:32.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:32.637701+0000) 2020-02-05T21:29:34.729 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:34.725+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.726789+0000) 2020-02-05T21:29:34.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:34.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.727419+0000) 2020-02-05T21:29:34.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:34.726+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.728067+0000) 2020-02-05T21:29:34.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:34.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.728632+0000) 2020-02-05T21:29:34.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:34.727+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.729037+0000) 2020-02-05T21:29:34.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:34.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.729650+0000) 2020-02-05T21:29:34.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:34.729+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.730907+0000) 2020-02-05T21:29:34.733 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:34.730+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.731472+0000) 2020-02-05T21:29:34.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:34.730+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.731870+0000) 2020-02-05T21:29:34.734 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:34.731+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.732303+0000) 2020-02-05T21:29:34.735 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:34.731+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.732510+0000) 2020-02-05T21:29:34.735 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:34.731+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.732691+0000) 2020-02-05T21:29:34.736 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:34.731+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.733229+0000) 2020-02-05T21:29:34.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:34.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87439 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:34.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:34.738+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:34.739288+0000) 2020-02-05T21:29:35.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:35.849+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:35.850576+0000) 2020-02-05T21:29:36.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:36.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:36.747366+0000) 2020-02-05T21:29:36.995 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:36.993+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:36.993748+0000) 2020-02-05T21:29:36.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:36.993+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:36.993985+0000) 2020-02-05T21:29:37.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:37.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:37.004533+0000) 2020-02-05T21:29:37.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:37.177+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:37.178051+0000) 2020-02-05T21:29:39.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:39.101+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.102369+0000) 2020-02-05T21:29:39.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:39.386+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.386300+0000) 2020-02-05T21:29:39.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:39.742+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.743374+0000) 2020-02-05T21:29:39.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:39.742+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.743635+0000) 2020-02-05T21:29:39.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:39.743+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.744833+0000) 2020-02-05T21:29:39.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:39.744+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.745331+0000) 2020-02-05T21:29:39.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:39.744+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.745598+0000) 2020-02-05T21:29:39.747 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:39.744+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.745836+0000) 2020-02-05T21:29:39.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:39.744+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.746065+0000) 2020-02-05T21:29:39.748 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:39.745+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.746920+0000) 2020-02-05T21:29:39.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:39.745+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.747033+0000) 2020-02-05T21:29:39.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:39.746+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.747761+0000) 2020-02-05T21:29:39.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:39.746+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.747556+0000) 2020-02-05T21:29:39.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:39.747+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.748384+0000) 2020-02-05T21:29:39.750 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:39.746+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.748183+0000) 2020-02-05T21:29:39.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:39.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.753728+0000) 2020-02-05T21:29:39.755 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:39.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:39.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:39.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:39.994886+0000) 2020-02-05T21:29:40.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:40.227+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:40.228218+0000) 2020-02-05T21:29:40.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:40.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:40.246581+0000) 2020-02-05T21:29:40.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:40.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:40.256115+0000) 2020-02-05T21:29:42.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:42.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:42.237039+0000) 2020-02-05T21:29:42.511 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:29:42.518 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:29:42.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:42.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:42.637913+0000) 2020-02-05T21:29:44.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:44.761+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.761452+0000) 2020-02-05T21:29:44.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:44.761+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.761965+0000) 2020-02-05T21:29:44.765 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:44.762+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.762748+0000) 2020-02-05T21:29:44.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:44.762+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.763042+0000) 2020-02-05T21:29:44.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:44.763+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.763439+0000) 2020-02-05T21:29:44.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:44.763+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.763758+0000) 2020-02-05T21:29:44.768 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:44.763+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.763798+0000) 2020-02-05T21:29:44.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:44.764+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.764775+0000) 2020-02-05T21:29:44.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:44.764+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.765052+0000) 2020-02-05T21:29:44.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:44.765+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.765646+0000) 2020-02-05T21:29:44.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:44.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.764960+0000) 2020-02-05T21:29:44.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:44.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.765694+0000) 2020-02-05T21:29:44.769 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:44.765+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.765396+0000) 2020-02-05T21:29:44.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:44.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:44.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:44.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:44.773387+0000) 2020-02-05T21:29:45.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:45.850+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:45.850815+0000) 2020-02-05T21:29:46.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:46.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:46.747607+0000) 2020-02-05T21:29:46.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:46.993+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:46.994149+0000) 2020-02-05T21:29:46.996 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:46.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:46.994450+0000) 2020-02-05T21:29:47.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:47.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:47.004766+0000) 2020-02-05T21:29:47.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:47.186+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:47.186778+0000) 2020-02-05T21:29:49.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:49.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.102669+0000) 2020-02-05T21:29:49.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:49.385+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.386551+0000) 2020-02-05T21:29:49.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:49.776+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.778028+0000) 2020-02-05T21:29:49.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:49.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.779217+0000) 2020-02-05T21:29:49.780 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:49.778+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.779371+0000) 2020-02-05T21:29:49.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:49.778+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.779782+0000) 2020-02-05T21:29:49.781 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:49.779+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.780130+0000) 2020-02-05T21:29:49.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:49.779+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.780439+0000) 2020-02-05T21:29:49.782 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:49.780+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.780706+0000) 2020-02-05T21:29:49.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:49.779+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.780789+0000) 2020-02-05T21:29:49.783 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:49.781+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.781442+0000) 2020-02-05T21:29:49.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:49.781+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.782171+0000) 2020-02-05T21:29:49.784 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:49.781+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.782182+0000) 2020-02-05T21:29:49.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:49.782+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.782677+0000) 2020-02-05T21:29:49.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:49.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.782762+0000) 2020-02-05T21:29:49.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:49.785+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.786215+0000) 2020-02-05T21:29:49.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:49.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87564 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:49.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:49.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:49.995072+0000) 2020-02-05T21:29:50.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:50.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:50.228473+0000) 2020-02-05T21:29:50.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:50.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:50.246824+0000) 2020-02-05T21:29:50.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:50.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:50.256356+0000) 2020-02-05T21:29:52.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:52.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:52.237283+0000) 2020-02-05T21:29:52.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:52.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:52.638204+0000) 2020-02-05T21:29:54.795 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:54.792+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.793664+0000) 2020-02-05T21:29:54.796 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:54.793+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.794282+0000) 2020-02-05T21:29:54.796 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:54.793+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.794881+0000) 2020-02-05T21:29:54.797 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:54.794+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.795353+0000) 2020-02-05T21:29:54.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:54.794+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.795876+0000) 2020-02-05T21:29:54.799 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:54.795+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.796181+0000) 2020-02-05T21:29:54.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:54.795+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.796698+0000) 2020-02-05T21:29:54.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:54.795+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.797001+0000) 2020-02-05T21:29:54.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:54.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.797559+0000) 2020-02-05T21:29:54.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:54.797+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.797916+0000) 2020-02-05T21:29:54.800 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:54.797+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.798191+0000) 2020-02-05T21:29:54.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:54.797+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.798239+0000) 2020-02-05T21:29:54.801 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:54.799+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.799499+0000) 2020-02-05T21:29:54.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:54.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87599 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:54.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:54.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:54.806455+0000) 2020-02-05T21:29:55.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:55.850+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:55.851047+0000) 2020-02-05T21:29:56.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:56.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:56.747854+0000) 2020-02-05T21:29:56.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:56.994+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:56.994471+0000) 2020-02-05T21:29:56.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:56.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:56.994678+0000) 2020-02-05T21:29:57.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:57.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:57.005047+0000) 2020-02-05T21:29:57.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:57.197+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:57.198510+0000) 2020-02-05T21:29:59.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:59.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.102937+0000) 2020-02-05T21:29:59.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:59.385+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.386756+0000) 2020-02-05T21:29:59.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:59.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.808624+0000) 2020-02-05T21:29:59.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:29:59.809+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.809641+0000) 2020-02-05T21:29:59.811 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:59.809+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.810357+0000) 2020-02-05T21:29:59.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:29:59.809+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.810739+0000) 2020-02-05T21:29:59.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:29:59.810+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.811187+0000) 2020-02-05T21:29:59.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:29:59.810+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.811435+0000) 2020-02-05T21:29:59.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:29:59.810+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.811646+0000) 2020-02-05T21:29:59.814 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:29:59.810+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.811183+0000) 2020-02-05T21:29:59.814 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:29:59.811+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.811847+0000) 2020-02-05T21:29:59.814 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:29:59.812+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.812690+0000) 2020-02-05T21:29:59.815 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:29:59.813+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.813797+0000) 2020-02-05T21:29:59.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:59.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.814510+0000) 2020-02-05T21:29:59.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:29:59.813+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.814279+0000) 2020-02-05T21:29:59.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:29:59.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.818417+0000) 2020-02-05T21:29:59.821 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:29:59.819+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87644 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:29:59.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:29:59.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:29:59.995336+0000) 2020-02-05T21:30:00.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:00.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:00.228690+0000) 2020-02-05T21:30:00.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:00.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:00.247057+0000) 2020-02-05T21:30:00.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:00.255+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:00.256557+0000) 2020-02-05T21:30:02.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:02.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:02.237544+0000) 2020-02-05T21:30:02.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:02.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:02.638442+0000) 2020-02-05T21:30:04.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:04.824+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.825387+0000) 2020-02-05T21:30:04.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:04.825+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.826572+0000) 2020-02-05T21:30:04.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:04.826+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.826908+0000) 2020-02-05T21:30:04.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:04.827+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.827508+0000) 2020-02-05T21:30:04.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:04.826+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.828051+0000) 2020-02-05T21:30:04.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:04.827+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.828946+0000) 2020-02-05T21:30:04.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:04.828+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.829593+0000) 2020-02-05T21:30:04.831 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:04.829+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.829733+0000) 2020-02-05T21:30:04.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:04.829+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.830537+0000) 2020-02-05T21:30:04.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:04.830+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.831331+0000) 2020-02-05T21:30:04.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:04.830+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.830663+0000) 2020-02-05T21:30:04.836 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:04.830+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.830717+0000) 2020-02-05T21:30:04.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:04.830+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.831168+0000) 2020-02-05T21:30:04.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:04.834+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:04.835726+0000) 2020-02-05T21:30:04.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:04.836+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87680 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:05.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:05.850+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:05.851405+0000) 2020-02-05T21:30:06.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:06.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:06.748033+0000) 2020-02-05T21:30:06.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:06.994+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:06.994798+0000) 2020-02-05T21:30:06.996 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:06.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:06.994911+0000) 2020-02-05T21:30:07.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:07.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:07.005367+0000) 2020-02-05T21:30:07.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:07.205+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:07.206467+0000) 2020-02-05T21:30:09.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:09.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.103125+0000) 2020-02-05T21:30:09.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:09.386+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.386993+0000) 2020-02-05T21:30:09.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:09.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.842649+0000) 2020-02-05T21:30:09.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:09.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.843665+0000) 2020-02-05T21:30:09.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:09.843+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.844383+0000) 2020-02-05T21:30:09.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:09.843+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.844826+0000) 2020-02-05T21:30:09.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:09.844+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.845846+0000) 2020-02-05T21:30:09.847 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:09.844+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.845302+0000) 2020-02-05T21:30:09.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:09.845+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.845648+0000) 2020-02-05T21:30:09.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:09.845+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.846296+0000) 2020-02-05T21:30:09.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:09.846+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.847712+0000) 2020-02-05T21:30:09.849 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:09.847+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.847983+0000) 2020-02-05T21:30:09.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:09.847+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.848151+0000) 2020-02-05T21:30:09.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:09.847+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.848328+0000) 2020-02-05T21:30:09.850 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:09.847+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.848306+0000) 2020-02-05T21:30:09.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:09.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.852199+0000) 2020-02-05T21:30:09.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:09.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:09.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:09.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:09.995580+0000) 2020-02-05T21:30:10.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:10.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:10.228957+0000) 2020-02-05T21:30:10.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:10.247+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:10.247331+0000) 2020-02-05T21:30:10.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:10.256+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:10.256798+0000) 2020-02-05T21:30:12.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:12.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:12.237799+0000) 2020-02-05T21:30:12.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:12.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:12.638679+0000) 2020-02-05T21:30:12.666 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:30:12.672 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:30:14.860 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:14.857+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.858514+0000) 2020-02-05T21:30:14.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:14.859+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.859667+0000) 2020-02-05T21:30:14.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:14.859+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.860310+0000) 2020-02-05T21:30:14.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:14.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.860695+0000) 2020-02-05T21:30:14.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:14.860+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.861483+0000) 2020-02-05T21:30:14.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:14.860+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.862023+0000) 2020-02-05T21:30:14.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:14.861+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.862444+0000) 2020-02-05T21:30:14.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:14.862+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.862780+0000) 2020-02-05T21:30:14.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:14.862+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.863041+0000) 2020-02-05T21:30:14.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:14.862+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.863946+0000) 2020-02-05T21:30:14.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:14.864+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.864726+0000) 2020-02-05T21:30:14.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:14.864+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.864995+0000) 2020-02-05T21:30:14.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:14.864+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.865405+0000) 2020-02-05T21:30:14.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:14.869+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:14.870687+0000) 2020-02-05T21:30:14.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:14.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:15.853 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:15.850+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:15.851652+0000) 2020-02-05T21:30:16.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:16.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:16.748321+0000) 2020-02-05T21:30:16.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:16.994+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:16.995048+0000) 2020-02-05T21:30:16.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:16.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:16.995180+0000) 2020-02-05T21:30:17.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:17.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:17.005611+0000) 2020-02-05T21:30:17.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:17.215+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:17.216621+0000) 2020-02-05T21:30:19.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:19.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.103354+0000) 2020-02-05T21:30:19.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:19.386+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.387332+0000) 2020-02-05T21:30:19.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:19.874+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.875296+0000) 2020-02-05T21:30:19.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:19.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.876215+0000) 2020-02-05T21:30:19.878 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:19.875+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.877017+0000) 2020-02-05T21:30:19.879 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:19.876+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.877363+0000) 2020-02-05T21:30:19.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:19.876+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.878023+0000) 2020-02-05T21:30:19.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:19.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.878410+0000) 2020-02-05T21:30:19.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:19.877+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.878785+0000) 2020-02-05T21:30:19.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:19.878+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.879286+0000) 2020-02-05T21:30:19.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:19.878+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.879581+0000) 2020-02-05T21:30:19.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:19.878+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.880336+0000) 2020-02-05T21:30:19.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:19.879+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.880866+0000) 2020-02-05T21:30:19.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:19.879+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.881201+0000) 2020-02-05T21:30:19.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:19.881+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.882423+0000) 2020-02-05T21:30:19.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:19.884+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87805 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:19.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:19.887+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.888231+0000) 2020-02-05T21:30:19.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:19.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:19.995795+0000) 2020-02-05T21:30:20.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:20.227+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:20.229215+0000) 2020-02-05T21:30:20.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:20.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:20.247565+0000) 2020-02-05T21:30:20.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:20.256+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:20.256996+0000) 2020-02-05T21:30:22.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:22.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:22.238047+0000) 2020-02-05T21:30:22.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:22.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:22.638904+0000) 2020-02-05T21:30:24.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:24.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.890915+0000) 2020-02-05T21:30:24.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:24.891+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.892421+0000) 2020-02-05T21:30:24.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:24.891+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.892798+0000) 2020-02-05T21:30:24.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:24.891+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.893204+0000) 2020-02-05T21:30:24.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:24.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.893838+0000) 2020-02-05T21:30:24.895 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:24.892+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.893737+0000) 2020-02-05T21:30:24.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:24.893+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.894250+0000) 2020-02-05T21:30:24.896 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:24.893+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.894890+0000) 2020-02-05T21:30:24.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:24.893+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.894486+0000) 2020-02-05T21:30:24.897 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:24.894+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.895280+0000) 2020-02-05T21:30:24.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:24.895+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.896066+0000) 2020-02-05T21:30:24.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:24.894+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.896258+0000) 2020-02-05T21:30:24.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:24.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.896635+0000) 2020-02-05T21:30:24.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:24.899+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:24.900104+0000) 2020-02-05T21:30:24.905 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:24.902+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:25.853 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:25.851+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:25.851862+0000) 2020-02-05T21:30:26.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:26.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:26.748564+0000) 2020-02-05T21:30:26.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:26.994+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:26.995341+0000) 2020-02-05T21:30:26.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:26.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:26.995483+0000) 2020-02-05T21:30:27.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:27.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:27.005839+0000) 2020-02-05T21:30:27.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:27.228+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:27.228604+0000) 2020-02-05T21:30:29.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:29.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.103641+0000) 2020-02-05T21:30:29.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:29.387+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.387586+0000) 2020-02-05T21:30:29.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:29.909+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.910157+0000) 2020-02-05T21:30:29.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:29.909+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.910657+0000) 2020-02-05T21:30:29.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:29.909+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.910973+0000) 2020-02-05T21:30:29.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:29.909+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.911282+0000) 2020-02-05T21:30:29.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:29.910+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.911398+0000) 2020-02-05T21:30:29.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:29.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.911825+0000) 2020-02-05T21:30:29.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:29.910+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.912373+0000) 2020-02-05T21:30:29.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:29.911+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.912368+0000) 2020-02-05T21:30:29.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:29.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.912957+0000) 2020-02-05T21:30:29.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:29.911+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.912802+0000) 2020-02-05T21:30:29.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:29.911+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.913329+0000) 2020-02-05T21:30:29.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:29.912+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.914146+0000) 2020-02-05T21:30:29.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:29.912+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.914245+0000) 2020-02-05T21:30:29.921 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:29.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:29.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:29.920+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.921729+0000) 2020-02-05T21:30:29.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:29.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:29.996047+0000) 2020-02-05T21:30:30.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:30.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:30.229465+0000) 2020-02-05T21:30:30.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:30.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:30.247796+0000) 2020-02-05T21:30:30.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:30.256+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:30.257200+0000) 2020-02-05T21:30:32.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:32.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:32.238298+0000) 2020-02-05T21:30:32.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:32.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:32.639150+0000) 2020-02-05T21:30:34.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:34.924+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.925348+0000) 2020-02-05T21:30:34.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:34.924+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.925938+0000) 2020-02-05T21:30:34.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:34.925+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.926519+0000) 2020-02-05T21:30:34.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:34.926+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.927207+0000) 2020-02-05T21:30:34.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:34.926+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.927679+0000) 2020-02-05T21:30:34.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:34.926+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.928007+0000) 2020-02-05T21:30:34.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:34.926+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.928340+0000) 2020-02-05T21:30:34.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:34.928+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.929169+0000) 2020-02-05T21:30:34.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:34.928+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.929561+0000) 2020-02-05T21:30:34.931 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:34.927+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.929000+0000) 2020-02-05T21:30:34.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:34.929+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.930328+0000) 2020-02-05T21:30:34.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:34.929+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.931184+0000) 2020-02-05T21:30:34.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:34.929+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.931045+0000) 2020-02-05T21:30:34.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:34.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:34.936559+0000) 2020-02-05T21:30:34.938 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:34.935+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:35.853 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:35.851+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:35.852107+0000) 2020-02-05T21:30:36.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:36.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:36.748746+0000) 2020-02-05T21:30:36.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:36.994+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:36.995604+0000) 2020-02-05T21:30:36.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:36.994+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:36.995852+0000) 2020-02-05T21:30:37.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:37.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:37.006098+0000) 2020-02-05T21:30:37.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:37.236+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:37.237067+0000) 2020-02-05T21:30:39.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:39.102+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.103870+0000) 2020-02-05T21:30:39.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:39.387+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.387822+0000) 2020-02-05T21:30:39.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:39.942+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.942739+0000) 2020-02-05T21:30:39.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:39.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.943300+0000) 2020-02-05T21:30:39.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:39.943+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.943740+0000) 2020-02-05T21:30:39.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:39.945+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.945157+0000) 2020-02-05T21:30:39.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:39.945+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.945724+0000) 2020-02-05T21:30:39.948 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:39.944+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.945530+0000) 2020-02-05T21:30:39.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:39.946+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.946588+0000) 2020-02-05T21:30:39.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:39.945+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.946965+0000) 2020-02-05T21:30:39.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:39.946+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.947510+0000) 2020-02-05T21:30:39.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:39.946+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.947634+0000) 2020-02-05T21:30:39.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:39.948+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.948794+0000) 2020-02-05T21:30:39.950 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:39.947+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.948702+0000) 2020-02-05T21:30:39.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:39.948+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.949629+0000) 2020-02-05T21:30:39.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:39.953+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.953487+0000) 2020-02-05T21:30:39.955 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:39.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 87967 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:39.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:39.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:39.996308+0000) 2020-02-05T21:30:40.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:40.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:40.229693+0000) 2020-02-05T21:30:40.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:40.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:40.248025+0000) 2020-02-05T21:30:40.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:40.257+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:40.257397+0000) 2020-02-05T21:30:42.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:42.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:42.238561+0000) 2020-02-05T21:30:42.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:42.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:42.639383+0000) 2020-02-05T21:30:42.813 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:30:42.824 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:30:44.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:44.959+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.960310+0000) 2020-02-05T21:30:44.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:44.960+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.961498+0000) 2020-02-05T21:30:44.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:44.961+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.962876+0000) 2020-02-05T21:30:44.964 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:44.962+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.963305+0000) 2020-02-05T21:30:44.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:44.962+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.963971+0000) 2020-02-05T21:30:44.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:44.963+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.964362+0000) 2020-02-05T21:30:44.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:44.963+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.964674+0000) 2020-02-05T21:30:44.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:44.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.964965+0000) 2020-02-05T21:30:44.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:44.964+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.966033+0000) 2020-02-05T21:30:44.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:44.964+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.966172+0000) 2020-02-05T21:30:44.968 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:44.965+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.966947+0000) 2020-02-05T21:30:44.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:44.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.967168+0000) 2020-02-05T21:30:44.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:44.967+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.968992+0000) 2020-02-05T21:30:44.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:44.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:44.969266+0000) 2020-02-05T21:30:44.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:44.970+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88004 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:45.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:45.851+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:45.852336+0000) 2020-02-05T21:30:46.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:46.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:46.749000+0000) 2020-02-05T21:30:46.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:46.995+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:46.995844+0000) 2020-02-05T21:30:46.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:46.995+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:46.996237+0000) 2020-02-05T21:30:47.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:47.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:47.006329+0000) 2020-02-05T21:30:47.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:47.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:47.245547+0000) 2020-02-05T21:30:49.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:49.103+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.104133+0000) 2020-02-05T21:30:49.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:49.387+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.388056+0000) 2020-02-05T21:30:49.977 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:49.975+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.976364+0000) 2020-02-05T21:30:49.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:49.976+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.977680+0000) 2020-02-05T21:30:49.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:49.976+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.978049+0000) 2020-02-05T21:30:49.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:49.978+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.980062+0000) 2020-02-05T21:30:49.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:49.979+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.980454+0000) 2020-02-05T21:30:49.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:49.980+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.981561+0000) 2020-02-05T21:30:49.984 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:49.981+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.982696+0000) 2020-02-05T21:30:49.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:49.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.983270+0000) 2020-02-05T21:30:49.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:49.981+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.982961+0000) 2020-02-05T21:30:49.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:49.982+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.983690+0000) 2020-02-05T21:30:49.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:49.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.984369+0000) 2020-02-05T21:30:49.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:49.982+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.983888+0000) 2020-02-05T21:30:49.986 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:49.982+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.984220+0000) 2020-02-05T21:30:49.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:49.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88049 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:49.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:49.987+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.988865+0000) 2020-02-05T21:30:49.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:49.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:49.996485+0000) 2020-02-05T21:30:50.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:50.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:50.229922+0000) 2020-02-05T21:30:50.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:50.247+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:50.248258+0000) 2020-02-05T21:30:50.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:50.256+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:50.257603+0000) 2020-02-05T21:30:52.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:52.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:52.238900+0000) 2020-02-05T21:30:52.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:52.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:52.639615+0000) 2020-02-05T21:30:54.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:54.995+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.996173+0000) 2020-02-05T21:30:54.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:54.995+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.996747+0000) 2020-02-05T21:30:54.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:54.995+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.996869+0000) 2020-02-05T21:30:54.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:54.996+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.997446+0000) 2020-02-05T21:30:54.999 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:54.996+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.997755+0000) 2020-02-05T21:30:55.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:30:54.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.998193+0000) 2020-02-05T21:30:55.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:54.997+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:54.998491+0000) 2020-02-05T21:30:55.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:30:54.999+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.000448+0000) 2020-02-05T21:30:55.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:54.999+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.000397+0000) 2020-02-05T21:30:55.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:30:54.999+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.001102+0000) 2020-02-05T21:30:55.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:54.999+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.000837+0000) 2020-02-05T21:30:55.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:30:54.999+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.001029+0000) 2020-02-05T21:30:55.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:55.000+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.001212+0000) 2020-02-05T21:30:55.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:30:55.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:30:55.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:55.006+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.007268+0000) 2020-02-05T21:30:55.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:30:55.851+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:55.852546+0000) 2020-02-05T21:30:56.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:30:56.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:56.749274+0000) 2020-02-05T21:30:56.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:30:56.995+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:56.996079+0000) 2020-02-05T21:30:56.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:30:56.995+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:56.996493+0000) 2020-02-05T21:30:57.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:30:57.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:57.006555+0000) 2020-02-05T21:30:57.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:57.253+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:57.253994+0000) 2020-02-05T21:30:59.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:30:59.103+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:59.104384+0000) 2020-02-05T21:30:59.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:30:59.387+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:59.388347+0000) 2020-02-05T21:30:59.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:30:59.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:30:59.996705+0000) 2020-02-05T21:31:00.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:00.009+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.010690+0000) 2020-02-05T21:31:00.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:00.010+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.012041+0000) 2020-02-05T21:31:00.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:00.011+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.012432+0000) 2020-02-05T21:31:00.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:00.011+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.012749+0000) 2020-02-05T21:31:00.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:00.011+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.013175+0000) 2020-02-05T21:31:00.015 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:00.013+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.014409+0000) 2020-02-05T21:31:00.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:00.013+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.014777+0000) 2020-02-05T21:31:00.016 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:00.013+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.015262+0000) 2020-02-05T21:31:00.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:00.014+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.015591+0000) 2020-02-05T21:31:00.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:00.014+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.016285+0000) 2020-02-05T21:31:00.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:00.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.016115+0000) 2020-02-05T21:31:00.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:00.015+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.016407+0000) 2020-02-05T21:31:00.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:00.015+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.016543+0000) 2020-02-05T21:31:00.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:00.020+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.021285+0000) 2020-02-05T21:31:00.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:00.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:00.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:00.228+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.230122+0000) 2020-02-05T21:31:00.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:00.247+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.248494+0000) 2020-02-05T21:31:00.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:00.257+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:00.257812+0000) 2020-02-05T21:31:02.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:02.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:02.239160+0000) 2020-02-05T21:31:02.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:02.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:02.639847+0000) 2020-02-05T21:31:05.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:05.025+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.027289+0000) 2020-02-05T21:31:05.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:05.026+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.027863+0000) 2020-02-05T21:31:05.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:05.027+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.028609+0000) 2020-02-05T21:31:05.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:05.028+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.029260+0000) 2020-02-05T21:31:05.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:05.027+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.029410+0000) 2020-02-05T21:31:05.031 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:05.028+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.029516+0000) 2020-02-05T21:31:05.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:05.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.030500+0000) 2020-02-05T21:31:05.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:05.029+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.030999+0000) 2020-02-05T21:31:05.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:05.030+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.031326+0000) 2020-02-05T21:31:05.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:05.029+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.030870+0000) 2020-02-05T21:31:05.033 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:05.030+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.031652+0000) 2020-02-05T21:31:05.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:05.030+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.031931+0000) 2020-02-05T21:31:05.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:05.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.032599+0000) 2020-02-05T21:31:05.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:05.036+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:05.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:05.037+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.038841+0000) 2020-02-05T21:31:05.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:05.852+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:05.852760+0000) 2020-02-05T21:31:06.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:06.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:06.749468+0000) 2020-02-05T21:31:06.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:06.995+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:06.996317+0000) 2020-02-05T21:31:06.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:06.995+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:06.996720+0000) 2020-02-05T21:31:07.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:07.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:07.006803+0000) 2020-02-05T21:31:07.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:07.261+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:07.262421+0000) 2020-02-05T21:31:09.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:09.104+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:09.104678+0000) 2020-02-05T21:31:09.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:09.388+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:09.388577+0000) 2020-02-05T21:31:09.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:09.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:09.996955+0000) 2020-02-05T21:31:10.044 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:10.042+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.043362+0000) 2020-02-05T21:31:10.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:10.042+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.043647+0000) 2020-02-05T21:31:10.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:10.043+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.044477+0000) 2020-02-05T21:31:10.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:10.044+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.045802+0000) 2020-02-05T21:31:10.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:10.044+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.046225+0000) 2020-02-05T21:31:10.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:10.045+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.046722+0000) 2020-02-05T21:31:10.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:10.045+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.047072+0000) 2020-02-05T21:31:10.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:10.046+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.047492+0000) 2020-02-05T21:31:10.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:10.046+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.047916+0000) 2020-02-05T21:31:10.049 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:10.046+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.048100+0000) 2020-02-05T21:31:10.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:10.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.048776+0000) 2020-02-05T21:31:10.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:10.048+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.049119+0000) 2020-02-05T21:31:10.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:10.048+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.049531+0000) 2020-02-05T21:31:10.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:10.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.054465+0000) 2020-02-05T21:31:10.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:10.054+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:10.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:10.229+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.230344+0000) 2020-02-05T21:31:10.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:10.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.248705+0000) 2020-02-05T21:31:10.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:10.257+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:10.258021+0000) 2020-02-05T21:31:12.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:12.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:12.239423+0000) 2020-02-05T21:31:12.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:12.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:12.640024+0000) 2020-02-05T21:31:12.960 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:31:12.967 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:31:15.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:15.060+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.061231+0000) 2020-02-05T21:31:15.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:15.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.061721+0000) 2020-02-05T21:31:15.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:15.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.063377+0000) 2020-02-05T21:31:15.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:15.063+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.064192+0000) 2020-02-05T21:31:15.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:15.063+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.065031+0000) 2020-02-05T21:31:15.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:15.065+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.065604+0000) 2020-02-05T21:31:15.067 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:15.065+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.065885+0000) 2020-02-05T21:31:15.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:15.065+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.066752+0000) 2020-02-05T21:31:15.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:15.066+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.067324+0000) 2020-02-05T21:31:15.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:15.067+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.067645+0000) 2020-02-05T21:31:15.070 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:15.066+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.067509+0000) 2020-02-05T21:31:15.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:15.066+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.067857+0000) 2020-02-05T21:31:15.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:15.067+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.068357+0000) 2020-02-05T21:31:15.073 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:15.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88247 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:15.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:15.072+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.073151+0000) 2020-02-05T21:31:15.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:15.852+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:15.853004+0000) 2020-02-05T21:31:16.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:16.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:16.749704+0000) 2020-02-05T21:31:16.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:16.996+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:16.996553+0000) 2020-02-05T21:31:16.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:16.996+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:16.996908+0000) 2020-02-05T21:31:17.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:17.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:17.007033+0000) 2020-02-05T21:31:17.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:17.273+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:17.274271+0000) 2020-02-05T21:31:19.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:19.104+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:19.104923+0000) 2020-02-05T21:31:19.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:19.388+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:19.388806+0000) 2020-02-05T21:31:19.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:19.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:19.997187+0000) 2020-02-05T21:31:20.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:20.077+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.078551+0000) 2020-02-05T21:31:20.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:20.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.079545+0000) 2020-02-05T21:31:20.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:20.077+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.077674+0000) 2020-02-05T21:31:20.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:20.080+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.080900+0000) 2020-02-05T21:31:20.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:20.081+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.081889+0000) 2020-02-05T21:31:20.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:20.081+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.082240+0000) 2020-02-05T21:31:20.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:20.083+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.083955+0000) 2020-02-05T21:31:20.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:20.083+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.083832+0000) 2020-02-05T21:31:20.087 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:20.082+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.083515+0000) 2020-02-05T21:31:20.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:20.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.084491+0000) 2020-02-05T21:31:20.088 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:20.083+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.084777+0000) 2020-02-05T21:31:20.088 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:20.084+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.085201+0000) 2020-02-05T21:31:20.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:20.086+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.087398+0000) 2020-02-05T21:31:20.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:20.087+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.088370+0000) 2020-02-05T21:31:20.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:20.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:20.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:20.230+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.230575+0000) 2020-02-05T21:31:20.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:20.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.248930+0000) 2020-02-05T21:31:20.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:20.257+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:20.258281+0000) 2020-02-05T21:31:22.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:22.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:22.239667+0000) 2020-02-05T21:31:22.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:22.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:22.640336+0000) 2020-02-05T21:31:25.095 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:25.093+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.094396+0000) 2020-02-05T21:31:25.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:25.094+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.095183+0000) 2020-02-05T21:31:25.097 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:25.094+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.095780+0000) 2020-02-05T21:31:25.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:25.096+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.096528+0000) 2020-02-05T21:31:25.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:25.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.097287+0000) 2020-02-05T21:31:25.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:25.096+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.097380+0000) 2020-02-05T21:31:25.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:25.096+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.098039+0000) 2020-02-05T21:31:25.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:25.097+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.098709+0000) 2020-02-05T21:31:25.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:25.097+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.099024+0000) 2020-02-05T21:31:25.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:25.098+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.099009+0000) 2020-02-05T21:31:25.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:25.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.100332+0000) 2020-02-05T21:31:25.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:25.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.100003+0000) 2020-02-05T21:31:25.102 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:25.099+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.100389+0000) 2020-02-05T21:31:25.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:25.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88328 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:25.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:25.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.105544+0000) 2020-02-05T21:31:25.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:25.852+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:25.853173+0000) 2020-02-05T21:31:26.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:26.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:26.749950+0000) 2020-02-05T21:31:26.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:26.996+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:26.996806+0000) 2020-02-05T21:31:26.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:26.996+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:26.997155+0000) 2020-02-05T21:31:27.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:27.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:27.007288+0000) 2020-02-05T21:31:27.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:27.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:27.284199+0000) 2020-02-05T21:31:29.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:29.104+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:29.105186+0000) 2020-02-05T21:31:29.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:29.387+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:29.389032+0000) 2020-02-05T21:31:29.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:29.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:29.997417+0000) 2020-02-05T21:31:30.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:30.110+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.111194+0000) 2020-02-05T21:31:30.113 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:30.110+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.111670+0000) 2020-02-05T21:31:30.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:30.111+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.112762+0000) 2020-02-05T21:31:30.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:30.112+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.113103+0000) 2020-02-05T21:31:30.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:30.112+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.113368+0000) 2020-02-05T21:31:30.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:30.112+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.113910+0000) 2020-02-05T21:31:30.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:30.113+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.114133+0000) 2020-02-05T21:31:30.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:30.113+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.114429+0000) 2020-02-05T21:31:30.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:30.114+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.115468+0000) 2020-02-05T21:31:30.117 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:30.115+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.115743+0000) 2020-02-05T21:31:30.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:30.115+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.116066+0000) 2020-02-05T21:31:30.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:30.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.116525+0000) 2020-02-05T21:31:30.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:30.116+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.116795+0000) 2020-02-05T21:31:30.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:30.120+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.121642+0000) 2020-02-05T21:31:30.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:30.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:30.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:30.230+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.230795+0000) 2020-02-05T21:31:30.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:30.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.249174+0000) 2020-02-05T21:31:30.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:30.258+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:30.258494+0000) 2020-02-05T21:31:32.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:32.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:32.239934+0000) 2020-02-05T21:31:32.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:32.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:32.640570+0000) 2020-02-05T21:31:35.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:35.128+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.128202+0000) 2020-02-05T21:31:35.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:35.128+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.128784+0000) 2020-02-05T21:31:35.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:35.129+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.129818+0000) 2020-02-05T21:31:35.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:35.130+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.130464+0000) 2020-02-05T21:31:35.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:35.131+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.131502+0000) 2020-02-05T21:31:35.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:35.131+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.131717+0000) 2020-02-05T21:31:35.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:35.132+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.132392+0000) 2020-02-05T21:31:35.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:35.133+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.133432+0000) 2020-02-05T21:31:35.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:35.133+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.133428+0000) 2020-02-05T21:31:35.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:35.132+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.132785+0000) 2020-02-05T21:31:35.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:35.133+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.133477+0000) 2020-02-05T21:31:35.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:35.133+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.133573+0000) 2020-02-05T21:31:35.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:35.133+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.134003+0000) 2020-02-05T21:31:35.139 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:35.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:35.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:35.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.139101+0000) 2020-02-05T21:31:35.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:35.853+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:35.853375+0000) 2020-02-05T21:31:36.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:36.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:36.750174+0000) 2020-02-05T21:31:36.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:36.996+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:36.997054+0000) 2020-02-05T21:31:36.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:36.996+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:36.997385+0000) 2020-02-05T21:31:37.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:37.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:37.007514+0000) 2020-02-05T21:31:37.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:37.291+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:37.292255+0000) 2020-02-05T21:31:39.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:39.105+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:39.105500+0000) 2020-02-05T21:31:39.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:39.388+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:39.389272+0000) 2020-02-05T21:31:39.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:39.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:39.997652+0000) 2020-02-05T21:31:40.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:40.141+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.142512+0000) 2020-02-05T21:31:40.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:40.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.143595+0000) 2020-02-05T21:31:40.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:40.146+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.146417+0000) 2020-02-05T21:31:40.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:40.146+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.147396+0000) 2020-02-05T21:31:40.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:40.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.144495+0000) 2020-02-05T21:31:40.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:40.144+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.145253+0000) 2020-02-05T21:31:40.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:40.144+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.145884+0000) 2020-02-05T21:31:40.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:40.145+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.146285+0000) 2020-02-05T21:31:40.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:40.146+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.147212+0000) 2020-02-05T21:31:40.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:40.147+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.148144+0000) 2020-02-05T21:31:40.155 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:40.148+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.148790+0000) 2020-02-05T21:31:40.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:40.148+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.149190+0000) 2020-02-05T21:31:40.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:40.149+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.149609+0000) 2020-02-05T21:31:40.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:40.153+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88453 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:40.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:40.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.152279+0000) 2020-02-05T21:31:40.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:40.230+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.231051+0000) 2020-02-05T21:31:40.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:40.249+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.249405+0000) 2020-02-05T21:31:40.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:40.257+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:40.258722+0000) 2020-02-05T21:31:42.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:42.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:42.240203+0000) 2020-02-05T21:31:42.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:42.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:42.640805+0000) 2020-02-05T21:31:43.098 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:31:43.104 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:31:45.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:45.160+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.161375+0000) 2020-02-05T21:31:45.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:45.162+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.162567+0000) 2020-02-05T21:31:45.169 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:45.163+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.164291+0000) 2020-02-05T21:31:45.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:45.164+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.164825+0000) 2020-02-05T21:31:45.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:45.165+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.165593+0000) 2020-02-05T21:31:45.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:45.165+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.165845+0000) 2020-02-05T21:31:45.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:45.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.160071+0000) 2020-02-05T21:31:45.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:45.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.161132+0000) 2020-02-05T21:31:45.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:45.160+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.161334+0000) 2020-02-05T21:31:45.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:45.160+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.161982+0000) 2020-02-05T21:31:45.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:45.161+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.163028+0000) 2020-02-05T21:31:45.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:45.164+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.165334+0000) 2020-02-05T21:31:45.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:45.168+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.169217+0000) 2020-02-05T21:31:45.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:45.168+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.169449+0000) 2020-02-05T21:31:45.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:45.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:45.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:45.852+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:45.853580+0000) 2020-02-05T21:31:46.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:46.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:46.750433+0000) 2020-02-05T21:31:46.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:46.997+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:46.997312+0000) 2020-02-05T21:31:46.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:46.997+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:46.997695+0000) 2020-02-05T21:31:47.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:47.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:47.007747+0000) 2020-02-05T21:31:47.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:47.299+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:47.300529+0000) 2020-02-05T21:31:49.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:49.105+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:49.105776+0000) 2020-02-05T21:31:49.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:49.388+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:49.389531+0000) 2020-02-05T21:31:49.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:49.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:49.997871+0000) 2020-02-05T21:31:50.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:50.174+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.175289+0000) 2020-02-05T21:31:50.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:50.175+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.176179+0000) 2020-02-05T21:31:50.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:50.176+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.178011+0000) 2020-02-05T21:31:50.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:50.177+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.178391+0000) 2020-02-05T21:31:50.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:50.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.178704+0000) 2020-02-05T21:31:50.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:50.178+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.179501+0000) 2020-02-05T21:31:50.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:50.179+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.180200+0000) 2020-02-05T21:31:50.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:50.179+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.180415+0000) 2020-02-05T21:31:50.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:50.179+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.181372+0000) 2020-02-05T21:31:50.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:50.180+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.181811+0000) 2020-02-05T21:31:50.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:50.180+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.181805+0000) 2020-02-05T21:31:50.183 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:50.180+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.182224+0000) 2020-02-05T21:31:50.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:50.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.184860+0000) 2020-02-05T21:31:50.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:50.184+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.185115+0000) 2020-02-05T21:31:50.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:50.185+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88535 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:50.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:50.230+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.231213+0000) 2020-02-05T21:31:50.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:50.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.249662+0000) 2020-02-05T21:31:50.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:50.258+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:50.258930+0000) 2020-02-05T21:31:52.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:52.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:52.240399+0000) 2020-02-05T21:31:52.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:52.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:52.641037+0000) 2020-02-05T21:31:55.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:55.191+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.192535+0000) 2020-02-05T21:31:55.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:55.192+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.193164+0000) 2020-02-05T21:31:55.196 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:55.194+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.195146+0000) 2020-02-05T21:31:55.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:55.194+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.195408+0000) 2020-02-05T21:31:55.197 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:31:55.194+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.195780+0000) 2020-02-05T21:31:55.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:55.195+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.196816+0000) 2020-02-05T21:31:55.205 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:31:55.197+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.198344+0000) 2020-02-05T21:31:55.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:55.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.198651+0000) 2020-02-05T21:31:55.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:55.200+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.202011+0000) 2020-02-05T21:31:55.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:55.194+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.195898+0000) 2020-02-05T21:31:55.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:31:55.195+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.196903+0000) 2020-02-05T21:31:55.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:55.196+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.197545+0000) 2020-02-05T21:31:55.206 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:55.197+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.198755+0000) 2020-02-05T21:31:55.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:31:55.197+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.198913+0000) 2020-02-05T21:31:55.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:31:55.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:31:55.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:31:55.853+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:55.853789+0000) 2020-02-05T21:31:56.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:31:56.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:56.750705+0000) 2020-02-05T21:31:56.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:31:56.996+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:56.997557+0000) 2020-02-05T21:31:56.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:31:56.996+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:56.997948+0000) 2020-02-05T21:31:57.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:31:57.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:57.007962+0000) 2020-02-05T21:31:57.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:57.308+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:57.309016+0000) 2020-02-05T21:31:59.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:31:59.105+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:59.106021+0000) 2020-02-05T21:31:59.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:31:59.389+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:59.389761+0000) 2020-02-05T21:32:00.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:31:59.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:31:59.998122+0000) 2020-02-05T21:32:00.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:00.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.211185+0000) 2020-02-05T21:32:00.213 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:00.211+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.212182+0000) 2020-02-05T21:32:00.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:00.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.212828+0000) 2020-02-05T21:32:00.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:00.211+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.212905+0000) 2020-02-05T21:32:00.215 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:00.212+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.213496+0000) 2020-02-05T21:32:00.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:00.213+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.214515+0000) 2020-02-05T21:32:00.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:00.212+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.214091+0000) 2020-02-05T21:32:00.216 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:00.213+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.214780+0000) 2020-02-05T21:32:00.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:00.214+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.215393+0000) 2020-02-05T21:32:00.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:00.214+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.215641+0000) 2020-02-05T21:32:00.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:00.214+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.216091+0000) 2020-02-05T21:32:00.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:00.214+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.215956+0000) 2020-02-05T21:32:00.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:00.215+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.216530+0000) 2020-02-05T21:32:00.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:00.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:00.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:00.221+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.222653+0000) 2020-02-05T21:32:00.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:00.229+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.231341+0000) 2020-02-05T21:32:00.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:00.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.249808+0000) 2020-02-05T21:32:00.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:00.258+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:00.259147+0000) 2020-02-05T21:32:02.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:02.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:02.240711+0000) 2020-02-05T21:32:02.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:02.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:02.641328+0000) 2020-02-05T21:32:05.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:05.225+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.226391+0000) 2020-02-05T21:32:05.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:05.225+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.226940+0000) 2020-02-05T21:32:05.228 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:05.226+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.227580+0000) 2020-02-05T21:32:05.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:05.227+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.228580+0000) 2020-02-05T21:32:05.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:05.228+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.229438+0000) 2020-02-05T21:32:05.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:05.228+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.229848+0000) 2020-02-05T21:32:05.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:05.229+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.230470+0000) 2020-02-05T21:32:05.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:05.229+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.230587+0000) 2020-02-05T21:32:05.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:05.228+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.230098+0000) 2020-02-05T21:32:05.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:05.229+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.230714+0000) 2020-02-05T21:32:05.233 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:05.229+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.231288+0000) 2020-02-05T21:32:05.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:05.230+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.231967+0000) 2020-02-05T21:32:05.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:05.231+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.232924+0000) 2020-02-05T21:32:05.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:05.236+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.237633+0000) 2020-02-05T21:32:05.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:05.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:05.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:05.853+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:05.854015+0000) 2020-02-05T21:32:06.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:06.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:06.750954+0000) 2020-02-05T21:32:06.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:06.996+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:06.997818+0000) 2020-02-05T21:32:06.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:06.996+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:06.998175+0000) 2020-02-05T21:32:07.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:07.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:07.008198+0000) 2020-02-05T21:32:07.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:07.316+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:07.317281+0000) 2020-02-05T21:32:09.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:09.105+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:09.106325+0000) 2020-02-05T21:32:09.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:09.388+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:09.389976+0000) 2020-02-05T21:32:10.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:09.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:09.998306+0000) 2020-02-05T21:32:10.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:10.230+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.231559+0000) 2020-02-05T21:32:10.245 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:10.242+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.244022+0000) 2020-02-05T21:32:10.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:10.243+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.244462+0000) 2020-02-05T21:32:10.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:10.243+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.244804+0000) 2020-02-05T21:32:10.247 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:10.244+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.245140+0000) 2020-02-05T21:32:10.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:10.244+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.245523+0000) 2020-02-05T21:32:10.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:10.244+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.245546+0000) 2020-02-05T21:32:10.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:10.244+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.246028+0000) 2020-02-05T21:32:10.249 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:10.245+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.246614+0000) 2020-02-05T21:32:10.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:10.246+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.247172+0000) 2020-02-05T21:32:10.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:10.244+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.246234+0000) 2020-02-05T21:32:10.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:10.244+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.246020+0000) 2020-02-05T21:32:10.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:10.245+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.247402+0000) 2020-02-05T21:32:10.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:10.246+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.247641+0000) 2020-02-05T21:32:10.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:10.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.249941+0000) 2020-02-05T21:32:10.254 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:10.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88697 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:10.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:10.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.254157+0000) 2020-02-05T21:32:10.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:10.258+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:10.259311+0000) 2020-02-05T21:32:12.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:12.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:12.241000+0000) 2020-02-05T21:32:12.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:12.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:12.641563+0000) 2020-02-05T21:32:13.239 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:32:13.246 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:32:15.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:15.257+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.258409+0000) 2020-02-05T21:32:15.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:15.257+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.259035+0000) 2020-02-05T21:32:15.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:15.258+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.259912+0000) 2020-02-05T21:32:15.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:15.259+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.260111+0000) 2020-02-05T21:32:15.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:15.259+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.260841+0000) 2020-02-05T21:32:15.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:15.260+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.261445+0000) 2020-02-05T21:32:15.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:15.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.261638+0000) 2020-02-05T21:32:15.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:15.260+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.261873+0000) 2020-02-05T21:32:15.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:15.261+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.262436+0000) 2020-02-05T21:32:15.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:15.261+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.262672+0000) 2020-02-05T21:32:15.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:15.262+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.263397+0000) 2020-02-05T21:32:15.265 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:15.262+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.263679+0000) 2020-02-05T21:32:15.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:15.262+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.264245+0000) 2020-02-05T21:32:15.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:15.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.267036+0000) 2020-02-05T21:32:15.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:15.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:15.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:15.853+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:15.854202+0000) 2020-02-05T21:32:16.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:16.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:16.751186+0000) 2020-02-05T21:32:17.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:16.997+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:16.998075+0000) 2020-02-05T21:32:17.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:16.997+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:16.998401+0000) 2020-02-05T21:32:17.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:17.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:17.008427+0000) 2020-02-05T21:32:17.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:17.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:17.324922+0000) 2020-02-05T21:32:19.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:19.105+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:19.106588+0000) 2020-02-05T21:32:19.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:19.389+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:19.390250+0000) 2020-02-05T21:32:20.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:19.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:19.998540+0000) 2020-02-05T21:32:20.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:20.231+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.231806+0000) 2020-02-05T21:32:20.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:20.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.250142+0000) 2020-02-05T21:32:20.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:20.259+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.259502+0000) 2020-02-05T21:32:20.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:20.273+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.274519+0000) 2020-02-05T21:32:20.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:20.274+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.275672+0000) 2020-02-05T21:32:20.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:20.274+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.275924+0000) 2020-02-05T21:32:20.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:20.275+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.276511+0000) 2020-02-05T21:32:20.278 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:20.276+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.277262+0000) 2020-02-05T21:32:20.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:20.275+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.276714+0000) 2020-02-05T21:32:20.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:20.276+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.277765+0000) 2020-02-05T21:32:20.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:20.277+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.278205+0000) 2020-02-05T21:32:20.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:20.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.278853+0000) 2020-02-05T21:32:20.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:20.278+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.279608+0000) 2020-02-05T21:32:20.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:20.277+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.279310+0000) 2020-02-05T21:32:20.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:20.278+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.280283+0000) 2020-02-05T21:32:20.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:20.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.281101+0000) 2020-02-05T21:32:20.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:20.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:20.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:20.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:20.285931+0000) 2020-02-05T21:32:22.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:22.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:22.241245+0000) 2020-02-05T21:32:22.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:22.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:22.641799+0000) 2020-02-05T21:32:25.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:25.289+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.290289+0000) 2020-02-05T21:32:25.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:25.288+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.290059+0000) 2020-02-05T21:32:25.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:25.289+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.291132+0000) 2020-02-05T21:32:25.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:25.294+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.295541+0000) 2020-02-05T21:32:25.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:25.294+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.296059+0000) 2020-02-05T21:32:25.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:25.295+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.296690+0000) 2020-02-05T21:32:25.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:25.295+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.296844+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:25.290+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.291927+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:25.291+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.292747+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:25.292+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.293377+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:25.293+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.294323+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:25.294+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.295314+0000) 2020-02-05T21:32:25.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:25.294+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.295610+0000) 2020-02-05T21:32:25.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:25.298+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.299686+0000) 2020-02-05T21:32:25.303 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:25.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88814 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:25.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:25.854+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:25.854388+0000) 2020-02-05T21:32:26.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:26.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:26.751451+0000) 2020-02-05T21:32:27.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:26.997+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:26.998324+0000) 2020-02-05T21:32:27.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:26.997+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:26.998650+0000) 2020-02-05T21:32:27.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:27.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:27.008703+0000) 2020-02-05T21:32:27.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:27.334+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:27.334761+0000) 2020-02-05T21:32:29.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:29.106+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:29.106826+0000) 2020-02-05T21:32:29.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:29.390+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:29.390486+0000) 2020-02-05T21:32:30.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:29.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:29.998730+0000) 2020-02-05T21:32:30.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:30.231+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.232063+0000) 2020-02-05T21:32:30.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:30.249+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.250376+0000) 2020-02-05T21:32:30.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:30.259+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.259725+0000) 2020-02-05T21:32:30.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:30.306+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.307530+0000) 2020-02-05T21:32:30.310 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:30.308+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.308810+0000) 2020-02-05T21:32:30.310 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:30.309+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.309389+0000) 2020-02-05T21:32:30.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:30.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.310015+0000) 2020-02-05T21:32:30.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:30.309+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.310687+0000) 2020-02-05T21:32:30.312 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:30.311+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.311108+0000) 2020-02-05T21:32:30.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:30.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.311600+0000) 2020-02-05T21:32:30.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:30.310+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.311931+0000) 2020-02-05T21:32:30.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:30.310+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.312394+0000) 2020-02-05T21:32:30.314 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:30.313+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.313522+0000) 2020-02-05T21:32:30.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:30.312+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.314197+0000) 2020-02-05T21:32:30.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:30.312+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.314406+0000) 2020-02-05T21:32:30.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:30.314+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.314312+0000) 2020-02-05T21:32:30.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:30.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:30.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:30.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:30.319629+0000) 2020-02-05T21:32:32.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:32.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:32.241498+0000) 2020-02-05T21:32:32.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:32.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:32.642033+0000) 2020-02-05T21:32:35.325 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:35.322+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.323732+0000) 2020-02-05T21:32:35.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:35.323+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.324894+0000) 2020-02-05T21:32:35.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:35.324+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.325862+0000) 2020-02-05T21:32:35.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:35.325+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.326125+0000) 2020-02-05T21:32:35.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:35.325+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.326511+0000) 2020-02-05T21:32:35.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:35.325+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.326823+0000) 2020-02-05T21:32:35.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:35.326+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.327283+0000) 2020-02-05T21:32:35.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:35.326+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.327794+0000) 2020-02-05T21:32:35.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:35.326+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.327916+0000) 2020-02-05T21:32:35.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:35.327+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.328472+0000) 2020-02-05T21:32:35.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:35.328+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.329307+0000) 2020-02-05T21:32:35.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:35.327+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.329016+0000) 2020-02-05T21:32:35.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:35.328+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.329724+0000) 2020-02-05T21:32:35.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:35.332+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.333405+0000) 2020-02-05T21:32:35.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:35.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:35.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:35.853+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:35.854629+0000) 2020-02-05T21:32:36.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:36.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:36.751732+0000) 2020-02-05T21:32:37.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:36.998+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:36.998561+0000) 2020-02-05T21:32:37.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:36.998+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:36.998854+0000) 2020-02-05T21:32:37.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:37.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:37.008950+0000) 2020-02-05T21:32:37.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:37.342+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:37.343306+0000) 2020-02-05T21:32:39.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:39.106+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:39.107069+0000) 2020-02-05T21:32:39.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:39.389+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:39.390744+0000) 2020-02-05T21:32:40.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:39.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:39.998970+0000) 2020-02-05T21:32:40.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:40.231+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.232299+0000) 2020-02-05T21:32:40.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:40.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.250689+0000) 2020-02-05T21:32:40.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:40.259+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.259959+0000) 2020-02-05T21:32:40.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:40.340+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.341208+0000) 2020-02-05T21:32:40.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:40.340+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.341649+0000) 2020-02-05T21:32:40.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:40.341+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.341916+0000) 2020-02-05T21:32:40.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:40.342+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.343114+0000) 2020-02-05T21:32:40.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:40.342+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.343748+0000) 2020-02-05T21:32:40.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:40.343+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.344199+0000) 2020-02-05T21:32:40.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:40.343+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.343952+0000) 2020-02-05T21:32:40.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:40.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.344930+0000) 2020-02-05T21:32:40.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:40.344+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.344511+0000) 2020-02-05T21:32:40.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:40.344+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.345260+0000) 2020-02-05T21:32:40.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:40.345+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.345422+0000) 2020-02-05T21:32:40.347 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:40.345+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.345900+0000) 2020-02-05T21:32:40.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:40.345+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.346400+0000) 2020-02-05T21:32:40.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:40.349+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:40.350418+0000) 2020-02-05T21:32:40.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:40.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:42.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:42.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:42.241754+0000) 2020-02-05T21:32:42.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:42.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:42.642316+0000) 2020-02-05T21:32:43.374 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:32:43.385 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:32:45.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:45.357+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.358775+0000) 2020-02-05T21:32:45.363 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:45.360+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.360621+0000) 2020-02-05T21:32:45.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:45.360+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.361040+0000) 2020-02-05T21:32:45.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:45.358+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.359697+0000) 2020-02-05T21:32:45.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:45.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.361592+0000) 2020-02-05T21:32:45.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:45.361+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.362197+0000) 2020-02-05T21:32:45.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:45.362+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.362363+0000) 2020-02-05T21:32:45.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:45.362+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.363063+0000) 2020-02-05T21:32:45.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:45.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.362827+0000) 2020-02-05T21:32:45.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:45.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.363233+0000) 2020-02-05T21:32:45.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:45.362+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.363611+0000) 2020-02-05T21:32:45.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:45.363+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.363678+0000) 2020-02-05T21:32:45.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:45.363+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.364511+0000) 2020-02-05T21:32:45.370 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:45.367+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 88974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:45.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:45.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.368574+0000) 2020-02-05T21:32:45.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:45.854+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:45.854835+0000) 2020-02-05T21:32:46.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:46.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:46.751977+0000) 2020-02-05T21:32:47.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:46.998+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:46.998825+0000) 2020-02-05T21:32:47.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:46.998+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:46.999082+0000) 2020-02-05T21:32:47.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:47.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:47.009183+0000) 2020-02-05T21:32:47.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:47.350+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:47.351702+0000) 2020-02-05T21:32:49.111 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:49.106+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:49.107325+0000) 2020-02-05T21:32:49.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:49.389+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:49.390993+0000) 2020-02-05T21:32:50.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:49.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:49.999196+0000) 2020-02-05T21:32:50.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:50.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.232555+0000) 2020-02-05T21:32:50.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:50.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.250916+0000) 2020-02-05T21:32:50.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:50.259+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.260265+0000) 2020-02-05T21:32:50.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:50.373+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.374438+0000) 2020-02-05T21:32:50.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:50.375+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.375712+0000) 2020-02-05T21:32:50.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:50.376+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.377439+0000) 2020-02-05T21:32:50.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:50.376+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.377162+0000) 2020-02-05T21:32:50.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:50.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.378227+0000) 2020-02-05T21:32:50.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:50.377+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.378153+0000) 2020-02-05T21:32:50.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:50.377+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.378883+0000) 2020-02-05T21:32:50.380 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:50.377+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.378986+0000) 2020-02-05T21:32:50.381 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:50.378+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.379467+0000) 2020-02-05T21:32:50.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:50.379+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.380109+0000) 2020-02-05T21:32:50.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:50.379+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.380495+0000) 2020-02-05T21:32:50.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:50.379+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.380415+0000) 2020-02-05T21:32:50.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:50.380+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.380781+0000) 2020-02-05T21:32:50.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:50.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89016 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:50.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:50.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:50.386981+0000) 2020-02-05T21:32:52.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:52.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:52.242018+0000) 2020-02-05T21:32:52.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:52.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:52.642549+0000) 2020-02-05T21:32:55.393 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:32:55.390+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.391437+0000) 2020-02-05T21:32:55.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:55.391+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.392550+0000) 2020-02-05T21:32:55.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:55.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.393413+0000) 2020-02-05T21:32:55.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:55.392+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.393814+0000) 2020-02-05T21:32:55.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:55.393+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.394288+0000) 2020-02-05T21:32:55.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:55.393+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.394509+0000) 2020-02-05T21:32:55.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:55.394+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.395305+0000) 2020-02-05T21:32:55.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:55.394+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.394743+0000) 2020-02-05T21:32:55.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:55.395+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.396254+0000) 2020-02-05T21:32:55.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:55.396+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.396901+0000) 2020-02-05T21:32:55.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:32:55.396+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.396671+0000) 2020-02-05T21:32:55.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:32:55.397+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.397620+0000) 2020-02-05T21:32:55.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:32:55.397+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.397858+0000) 2020-02-05T21:32:55.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:55.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.402927+0000) 2020-02-05T21:32:55.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:32:55.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:32:55.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:32:55.854+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:55.855035+0000) 2020-02-05T21:32:56.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:32:56.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:56.752204+0000) 2020-02-05T21:32:57.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:32:56.998+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:56.999082+0000) 2020-02-05T21:32:57.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:32:56.998+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:56.999314+0000) 2020-02-05T21:32:57.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:32:57.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:57.009411+0000) 2020-02-05T21:32:57.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:57.360+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:57.361505+0000) 2020-02-05T21:32:59.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:32:59.107+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:59.107565+0000) 2020-02-05T21:32:59.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:32:59.390+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:59.391283+0000) 2020-02-05T21:33:00.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:32:59.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:32:59.999426+0000) 2020-02-05T21:33:00.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:00.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.232789+0000) 2020-02-05T21:33:00.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:00.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.251131+0000) 2020-02-05T21:33:00.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:00.259+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.260499+0000) 2020-02-05T21:33:00.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:00.410+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.410851+0000) 2020-02-05T21:33:00.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:00.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.411467+0000) 2020-02-05T21:33:00.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:00.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.412481+0000) 2020-02-05T21:33:00.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:00.412+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.413390+0000) 2020-02-05T21:33:00.415 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:00.412+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.413465+0000) 2020-02-05T21:33:00.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:00.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.414216+0000) 2020-02-05T21:33:00.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:00.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.414678+0000) 2020-02-05T21:33:00.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:00.413+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.414811+0000) 2020-02-05T21:33:00.417 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:00.414+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.415284+0000) 2020-02-05T21:33:00.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:00.414+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.415172+0000) 2020-02-05T21:33:00.418 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:00.415+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.416136+0000) 2020-02-05T21:33:00.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:00.415+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.415968+0000) 2020-02-05T21:33:00.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:00.415+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.416233+0000) 2020-02-05T21:33:00.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:00.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:00.420561+0000) 2020-02-05T21:33:00.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:00.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89097 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:02.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:02.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:02.242279+0000) 2020-02-05T21:33:02.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:02.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:02.642785+0000) 2020-02-05T21:33:05.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:05.427+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.428479+0000) 2020-02-05T21:33:05.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:05.428+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.428424+0000) 2020-02-05T21:33:05.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:05.428+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.429240+0000) 2020-02-05T21:33:05.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:05.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.429945+0000) 2020-02-05T21:33:05.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:05.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.430721+0000) 2020-02-05T21:33:05.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:05.430+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.431223+0000) 2020-02-05T21:33:05.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:05.431+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.432253+0000) 2020-02-05T21:33:05.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:05.431+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.432612+0000) 2020-02-05T21:33:05.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:05.432+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.432719+0000) 2020-02-05T21:33:05.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:05.432+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.432841+0000) 2020-02-05T21:33:05.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:05.432+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.433374+0000) 2020-02-05T21:33:05.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:05.433+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.433573+0000) 2020-02-05T21:33:05.437 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:05.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.433312+0000) 2020-02-05T21:33:05.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:05.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.437230+0000) 2020-02-05T21:33:05.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:05.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:05.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:05.854+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:05.855369+0000) 2020-02-05T21:33:06.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:06.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:06.752484+0000) 2020-02-05T21:33:07.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:06.999+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:06.999334+0000) 2020-02-05T21:33:07.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:06.999+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:06.999588+0000) 2020-02-05T21:33:07.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:07.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:07.009634+0000) 2020-02-05T21:33:07.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:07.372+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:07.373280+0000) 2020-02-05T21:33:09.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:09.107+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:09.107916+0000) 2020-02-05T21:33:09.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:09.390+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:09.391521+0000) 2020-02-05T21:33:10.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:09.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:09.999647+0000) 2020-02-05T21:33:10.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:10.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.232985+0000) 2020-02-05T21:33:10.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:10.251+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.251377+0000) 2020-02-05T21:33:10.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:10.260+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.260728+0000) 2020-02-05T21:33:10.446 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:10.443+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.444681+0000) 2020-02-05T21:33:10.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:10.444+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.445673+0000) 2020-02-05T21:33:10.447 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:10.444+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.445967+0000) 2020-02-05T21:33:10.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:10.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.446235+0000) 2020-02-05T21:33:10.448 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:10.445+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.446809+0000) 2020-02-05T21:33:10.449 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:10.446+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.446966+0000) 2020-02-05T21:33:10.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:10.446+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.447437+0000) 2020-02-05T21:33:10.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:10.446+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.447672+0000) 2020-02-05T21:33:10.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:10.447+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.447700+0000) 2020-02-05T21:33:10.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:10.447+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.448024+0000) 2020-02-05T21:33:10.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:10.448+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.449137+0000) 2020-02-05T21:33:10.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:10.448+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.449274+0000) 2020-02-05T21:33:10.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:10.448+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.449178+0000) 2020-02-05T21:33:10.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:10.451+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:10.452415+0000) 2020-02-05T21:33:10.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:10.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:12.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:12.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:12.242516+0000) 2020-02-05T21:33:12.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:12.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:12.643016+0000) 2020-02-05T21:33:13.511 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:33:13.519 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:33:15.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:15.459+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.460158+0000) 2020-02-05T21:33:15.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:15.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.461401+0000) 2020-02-05T21:33:15.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:15.460+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.461784+0000) 2020-02-05T21:33:15.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:15.461+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.462253+0000) 2020-02-05T21:33:15.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:15.462+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.463345+0000) 2020-02-05T21:33:15.465 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:15.463+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.463838+0000) 2020-02-05T21:33:15.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:15.463+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.464398+0000) 2020-02-05T21:33:15.466 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:15.464+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.465234+0000) 2020-02-05T21:33:15.466 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:15.464+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.465606+0000) 2020-02-05T21:33:15.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:15.465+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.465864+0000) 2020-02-05T21:33:15.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:15.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.466252+0000) 2020-02-05T21:33:15.468 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:15.465+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.466079+0000) 2020-02-05T21:33:15.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:15.466+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.466815+0000) 2020-02-05T21:33:15.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:15.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89216 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:15.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:15.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.472217+0000) 2020-02-05T21:33:15.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:15.855+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:15.855618+0000) 2020-02-05T21:33:16.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:16.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:16.752725+0000) 2020-02-05T21:33:17.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:16.999+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:16.999587+0000) 2020-02-05T21:33:17.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:16.999+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:16.999860+0000) 2020-02-05T21:33:17.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:17.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:17.009878+0000) 2020-02-05T21:33:17.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:17.381+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:17.381666+0000) 2020-02-05T21:33:19.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:19.107+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:19.108163+0000) 2020-02-05T21:33:19.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:19.391+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:19.391754+0000) 2020-02-05T21:33:20.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:19.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:19.999914+0000) 2020-02-05T21:33:20.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:20.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.233224+0000) 2020-02-05T21:33:20.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:20.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.251611+0000) 2020-02-05T21:33:20.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:20.260+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.260963+0000) 2020-02-05T21:33:20.479 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:20.476+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.477845+0000) 2020-02-05T21:33:20.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:20.477+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.478660+0000) 2020-02-05T21:33:20.480 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:20.477+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.479018+0000) 2020-02-05T21:33:20.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:20.478+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.479442+0000) 2020-02-05T21:33:20.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:20.478+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.479891+0000) 2020-02-05T21:33:20.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:20.478+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.480319+0000) 2020-02-05T21:33:20.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:20.480+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.481418+0000) 2020-02-05T21:33:20.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:20.481+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.481800+0000) 2020-02-05T21:33:20.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:20.481+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.482890+0000) 2020-02-05T21:33:20.484 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:20.482+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.483471+0000) 2020-02-05T21:33:20.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:20.481+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.483219+0000) 2020-02-05T21:33:20.485 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:20.482+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.484124+0000) 2020-02-05T21:33:20.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:20.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:20.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:20.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.488425+0000) 2020-02-05T21:33:20.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:20.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:20.488720+0000) 2020-02-05T21:33:22.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:22.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:22.242775+0000) 2020-02-05T21:33:22.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:22.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:22.643292+0000) 2020-02-05T21:33:25.496 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:25.492+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.494387+0000) 2020-02-05T21:33:25.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:25.494+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.494783+0000) 2020-02-05T21:33:25.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:25.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.495481+0000) 2020-02-05T21:33:25.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:25.494+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.496064+0000) 2020-02-05T21:33:25.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:25.496+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.496718+0000) 2020-02-05T21:33:25.498 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:25.495+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.496875+0000) 2020-02-05T21:33:25.499 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:25.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.497277+0000) 2020-02-05T21:33:25.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:25.495+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.497383+0000) 2020-02-05T21:33:25.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:25.497+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.497849+0000) 2020-02-05T21:33:25.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:25.498+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.498697+0000) 2020-02-05T21:33:25.500 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:25.497+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.498977+0000) 2020-02-05T21:33:25.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:25.498+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.500034+0000) 2020-02-05T21:33:25.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:25.501+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.501581+0000) 2020-02-05T21:33:25.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:25.502+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.502521+0000) 2020-02-05T21:33:25.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:25.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:25.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:25.854+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:25.855868+0000) 2020-02-05T21:33:26.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:26.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:26.752985+0000) 2020-02-05T21:33:27.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:26.998+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:26.999835+0000) 2020-02-05T21:33:27.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:26.998+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:27.000078+0000) 2020-02-05T21:33:27.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:27.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:27.010133+0000) 2020-02-05T21:33:27.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:27.388+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:27.389881+0000) 2020-02-05T21:33:29.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:29.106+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:29.108375+0000) 2020-02-05T21:33:29.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:29.391+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:29.392000+0000) 2020-02-05T21:33:30.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:29.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.000171+0000) 2020-02-05T21:33:30.235 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:30.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.233488+0000) 2020-02-05T21:33:30.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:30.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.251858+0000) 2020-02-05T21:33:30.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:30.260+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.261195+0000) 2020-02-05T21:33:30.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:30.508+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.510164+0000) 2020-02-05T21:33:30.513 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:30.511+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.512577+0000) 2020-02-05T21:33:30.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:30.512+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.513072+0000) 2020-02-05T21:33:30.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:30.512+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.513350+0000) 2020-02-05T21:33:30.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:30.512+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.513743+0000) 2020-02-05T21:33:30.515 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:30.513+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.514735+0000) 2020-02-05T21:33:30.516 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:30.513+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.514929+0000) 2020-02-05T21:33:30.516 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:30.514+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.515298+0000) 2020-02-05T21:33:30.517 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:30.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.516599+0000) 2020-02-05T21:33:30.518 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:30.515+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.516773+0000) 2020-02-05T21:33:30.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:30.516+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.517735+0000) 2020-02-05T21:33:30.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:30.517+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.518680+0000) 2020-02-05T21:33:30.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:30.520+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.521545+0000) 2020-02-05T21:33:30.523 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:30.520+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:30.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:30.521+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:30.523014+0000) 2020-02-05T21:33:32.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:32.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:32.243028+0000) 2020-02-05T21:33:32.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:32.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:32.643543+0000) 2020-02-05T21:33:35.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:35.526+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.527356+0000) 2020-02-05T21:33:35.529 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:35.526+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.527940+0000) 2020-02-05T21:33:35.530 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:35.528+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.529360+0000) 2020-02-05T21:33:35.531 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:35.529+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.530205+0000) 2020-02-05T21:33:35.532 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:35.529+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.531104+0000) 2020-02-05T21:33:35.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:35.529+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.530524+0000) 2020-02-05T21:33:35.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:35.530+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.531749+0000) 2020-02-05T21:33:35.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:35.530+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.532024+0000) 2020-02-05T21:33:35.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:35.530+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.532166+0000) 2020-02-05T21:33:35.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:35.531+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.532902+0000) 2020-02-05T21:33:35.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:35.531+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.533031+0000) 2020-02-05T21:33:35.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:35.531+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.533212+0000) 2020-02-05T21:33:35.535 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:35.532+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.533612+0000) 2020-02-05T21:33:35.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:35.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:35.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:35.536+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.537774+0000) 2020-02-05T21:33:35.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:35.855+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:35.856138+0000) 2020-02-05T21:33:36.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:36.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:36.753214+0000) 2020-02-05T21:33:37.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:36.999+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:37.000093+0000) 2020-02-05T21:33:37.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:36.999+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:37.000339+0000) 2020-02-05T21:33:37.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:37.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:37.010357+0000) 2020-02-05T21:33:37.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:37.397+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:37.398331+0000) 2020-02-05T21:33:39.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:39.107+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:39.108689+0000) 2020-02-05T21:33:39.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:39.391+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:39.392273+0000) 2020-02-05T21:33:40.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:39.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.000402+0000) 2020-02-05T21:33:40.235 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:40.232+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.233721+0000) 2020-02-05T21:33:40.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:40.251+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.252086+0000) 2020-02-05T21:33:40.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:40.260+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.261463+0000) 2020-02-05T21:33:40.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:40.542+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.543867+0000) 2020-02-05T21:33:40.546 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:40.542+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.544197+0000) 2020-02-05T21:33:40.546 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:40.543+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.545017+0000) 2020-02-05T21:33:40.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:40.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.545690+0000) 2020-02-05T21:33:40.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:40.544+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.545846+0000) 2020-02-05T21:33:40.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:40.545+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.546639+0000) 2020-02-05T21:33:40.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:40.545+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.547086+0000) 2020-02-05T21:33:40.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:40.546+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.547464+0000) 2020-02-05T21:33:40.551 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:40.546+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.548369+0000) 2020-02-05T21:33:40.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:40.547+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.548473+0000) 2020-02-05T21:33:40.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:40.547+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.549218+0000) 2020-02-05T21:33:40.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:40.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.549777+0000) 2020-02-05T21:33:40.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:40.548+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.549891+0000) 2020-02-05T21:33:40.556 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:40.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:40.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:40.555+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:40.556256+0000) 2020-02-05T21:33:42.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:42.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:42.243291+0000) 2020-02-05T21:33:42.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:42.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:42.643779+0000) 2020-02-05T21:33:43.667 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:33:43.673 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:33:45.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:45.560+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.561058+0000) 2020-02-05T21:33:45.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:45.560+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.561825+0000) 2020-02-05T21:33:45.565 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:45.562+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.563156+0000) 2020-02-05T21:33:45.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:45.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.563642+0000) 2020-02-05T21:33:45.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:45.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.562689+0000) 2020-02-05T21:33:45.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:45.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.564244+0000) 2020-02-05T21:33:45.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:45.563+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.564866+0000) 2020-02-05T21:33:45.567 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:45.565+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.566386+0000) 2020-02-05T21:33:45.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:45.564+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.565882+0000) 2020-02-05T21:33:45.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:45.565+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.566687+0000) 2020-02-05T21:33:45.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:45.565+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.566600+0000) 2020-02-05T21:33:45.569 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:45.566+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.567653+0000) 2020-02-05T21:33:45.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:45.566+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.567687+0000) 2020-02-05T21:33:45.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:45.568+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.569980+0000) 2020-02-05T21:33:45.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:45.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:45.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:45.855+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:45.856350+0000) 2020-02-05T21:33:46.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:46.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:46.753448+0000) 2020-02-05T21:33:47.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:46.999+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:47.000323+0000) 2020-02-05T21:33:47.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:46.999+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:47.000650+0000) 2020-02-05T21:33:47.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:47.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:47.010630+0000) 2020-02-05T21:33:47.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:47.405+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:47.406871+0000) 2020-02-05T21:33:49.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:49.108+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:49.108946+0000) 2020-02-05T21:33:49.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:49.391+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:49.392496+0000) 2020-02-05T21:33:50.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:49.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.000621+0000) 2020-02-05T21:33:50.235 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:50.233+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.233981+0000) 2020-02-05T21:33:50.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:50.251+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.252337+0000) 2020-02-05T21:33:50.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:50.261+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.261679+0000) 2020-02-05T21:33:50.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:50.577+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.578471+0000) 2020-02-05T21:33:50.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:50.579+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.580956+0000) 2020-02-05T21:33:50.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:50.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.581874+0000) 2020-02-05T21:33:50.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:50.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.584499+0000) 2020-02-05T21:33:50.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:50.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.588275+0000) 2020-02-05T21:33:50.610 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:50.581+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.582753+0000) 2020-02-05T21:33:50.610 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:50.583+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.584156+0000) 2020-02-05T21:33:50.610 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:50.583+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.585052+0000) 2020-02-05T21:33:50.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:50.578+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.580249+0000) 2020-02-05T21:33:50.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:50.579+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.581373+0000) 2020-02-05T21:33:50.611 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:50.580+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.582373+0000) 2020-02-05T21:33:50.611 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:50.582+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.583578+0000) 2020-02-05T21:33:50.611 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:50.582+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.583778+0000) 2020-02-05T21:33:50.611 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:50.583+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:50.584679+0000) 2020-02-05T21:33:50.612 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:50.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:52.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:52.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:52.243530+0000) 2020-02-05T21:33:52.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:52.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:52.644035+0000) 2020-02-05T21:33:55.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:33:55.595+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.597213+0000) 2020-02-05T21:33:55.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:33:55.594+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.596337+0000) 2020-02-05T21:33:55.600 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:55.596+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.597509+0000) 2020-02-05T21:33:55.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:55.597+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.598257+0000) 2020-02-05T21:33:55.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:33:55.597+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.598394+0000) 2020-02-05T21:33:55.600 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:55.597+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.598722+0000) 2020-02-05T21:33:55.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:55.598+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.599214+0000) 2020-02-05T21:33:55.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:55.598+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.599419+0000) 2020-02-05T21:33:55.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:33:55.599+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.600567+0000) 2020-02-05T21:33:55.603 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:55.600+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.601565+0000) 2020-02-05T21:33:55.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:55.600+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.601989+0000) 2020-02-05T21:33:55.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:55.601+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.602521+0000) 2020-02-05T21:33:55.604 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:33:55.601+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.602724+0000) 2020-02-05T21:33:55.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:55.605+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.606406+0000) 2020-02-05T21:33:55.608 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:33:55.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:33:55.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:33:55.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:55.856592+0000) 2020-02-05T21:33:56.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:33:56.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:56.753675+0000) 2020-02-05T21:33:57.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:33:56.999+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:57.000585+0000) 2020-02-05T21:33:57.002 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:33:56.999+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:57.000893+0000) 2020-02-05T21:33:57.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:33:57.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:57.010893+0000) 2020-02-05T21:33:57.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:57.416+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:57.416837+0000) 2020-02-05T21:33:59.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:33:59.108+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:59.109183+0000) 2020-02-05T21:33:59.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:33:59.392+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:33:59.392759+0000) 2020-02-05T21:34:00.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:00.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.000857+0000) 2020-02-05T21:34:00.235 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:00.233+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.234261+0000) 2020-02-05T21:34:00.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:00.251+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.252589+0000) 2020-02-05T21:34:00.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:00.261+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.261901+0000) 2020-02-05T21:34:00.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:00.611+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.613041+0000) 2020-02-05T21:34:00.615 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:00.612+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.614060+0000) 2020-02-05T21:34:00.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:00.612+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.613792+0000) 2020-02-05T21:34:00.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:00.613+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.614815+0000) 2020-02-05T21:34:00.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:00.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.615276+0000) 2020-02-05T21:34:00.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:00.615+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.617025+0000) 2020-02-05T21:34:00.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:00.615+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.616902+0000) 2020-02-05T21:34:00.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:00.616+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.617821+0000) 2020-02-05T21:34:00.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:00.616+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.618099+0000) 2020-02-05T21:34:00.621 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:00.616+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.618237+0000) 2020-02-05T21:34:00.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:00.617+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.618496+0000) 2020-02-05T21:34:00.622 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:00.617+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.618918+0000) 2020-02-05T21:34:00.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:00.618+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.619538+0000) 2020-02-05T21:34:00.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:00.619+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:00.620990+0000) 2020-02-05T21:34:00.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:00.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:02.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:02.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:02.243794+0000) 2020-02-05T21:34:02.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:02.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:02.644319+0000) 2020-02-05T21:34:05.632 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:05.629+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.631030+0000) 2020-02-05T21:34:05.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:05.630+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.631584+0000) 2020-02-05T21:34:05.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:05.631+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.632275+0000) 2020-02-05T21:34:05.635 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:05.632+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.633521+0000) 2020-02-05T21:34:05.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:05.631+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.632938+0000) 2020-02-05T21:34:05.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:05.632+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.633968+0000) 2020-02-05T21:34:05.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:05.633+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.634297+0000) 2020-02-05T21:34:05.636 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:05.633+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.634747+0000) 2020-02-05T21:34:05.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:05.632+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.633942+0000) 2020-02-05T21:34:05.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:05.633+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.635108+0000) 2020-02-05T21:34:05.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:05.634+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.635662+0000) 2020-02-05T21:34:05.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:05.635+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.636465+0000) 2020-02-05T21:34:05.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:05.634+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.636183+0000) 2020-02-05T21:34:05.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:05.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:05.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:05.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.641933+0000) 2020-02-05T21:34:05.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:05.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:05.856798+0000) 2020-02-05T21:34:06.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:06.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:06.753913+0000) 2020-02-05T21:34:07.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:07.000+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:07.000881+0000) 2020-02-05T21:34:07.002 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:07.000+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:07.001134+0000) 2020-02-05T21:34:07.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:07.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:07.011121+0000) 2020-02-05T21:34:07.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:07.427+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:07.428586+0000) 2020-02-05T21:34:09.111 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:09.108+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:09.109469+0000) 2020-02-05T21:34:09.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:09.391+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:09.392991+0000) 2020-02-05T21:34:10.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:10.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.001115+0000) 2020-02-05T21:34:10.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:10.234+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.234548+0000) 2020-02-05T21:34:10.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:10.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.252838+0000) 2020-02-05T21:34:10.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:10.261+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.262127+0000) 2020-02-05T21:34:10.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:10.646+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.647490+0000) 2020-02-05T21:34:10.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:10.647+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.648107+0000) 2020-02-05T21:34:10.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:10.648+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.649402+0000) 2020-02-05T21:34:10.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:10.649+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.649838+0000) 2020-02-05T21:34:10.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:10.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.650500+0000) 2020-02-05T21:34:10.653 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:10.649+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.650430+0000) 2020-02-05T21:34:10.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:10.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.651122+0000) 2020-02-05T21:34:10.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:10.649+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.650014+0000) 2020-02-05T21:34:10.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:10.650+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.651239+0000) 2020-02-05T21:34:10.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:10.652+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.653390+0000) 2020-02-05T21:34:10.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:10.651+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.651874+0000) 2020-02-05T21:34:10.656 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:10.651+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.652089+0000) 2020-02-05T21:34:10.656 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:10.652+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.652717+0000) 2020-02-05T21:34:10.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:10.656+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:10.657110+0000) 2020-02-05T21:34:10.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:10.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:12.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:12.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:12.244047+0000) 2020-02-05T21:34:12.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:12.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:12.644552+0000) 2020-02-05T21:34:13.801 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:34:13.807 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:34:15.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:15.662+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.663100+0000) 2020-02-05T21:34:15.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:15.663+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.665061+0000) 2020-02-05T21:34:15.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:15.664+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.665596+0000) 2020-02-05T21:34:15.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:15.665+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.665822+0000) 2020-02-05T21:34:15.668 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:15.666+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.666615+0000) 2020-02-05T21:34:15.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:15.666+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.667272+0000) 2020-02-05T21:34:15.669 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:15.666+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.667706+0000) 2020-02-05T21:34:15.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:15.666+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.667982+0000) 2020-02-05T21:34:15.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:15.668+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.668688+0000) 2020-02-05T21:34:15.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:15.667+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.668844+0000) 2020-02-05T21:34:15.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:15.668+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.669809+0000) 2020-02-05T21:34:15.671 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:15.669+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.669978+0000) 2020-02-05T21:34:15.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:15.669+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.670005+0000) 2020-02-05T21:34:15.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:15.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.673902+0000) 2020-02-05T21:34:15.677 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:15.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:15.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:15.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:15.856992+0000) 2020-02-05T21:34:16.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:16.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:16.754144+0000) 2020-02-05T21:34:17.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:17.000+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:17.001136+0000) 2020-02-05T21:34:17.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:17.000+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:17.001363+0000) 2020-02-05T21:34:17.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:17.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:17.011337+0000) 2020-02-05T21:34:17.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:17.440+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:17.440862+0000) 2020-02-05T21:34:19.111 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:19.109+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:19.109728+0000) 2020-02-05T21:34:19.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:19.393+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:19.393179+0000) 2020-02-05T21:34:20.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:20.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.001350+0000) 2020-02-05T21:34:20.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:20.234+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.234782+0000) 2020-02-05T21:34:20.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:20.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.253130+0000) 2020-02-05T21:34:20.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:20.262+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.262317+0000) 2020-02-05T21:34:20.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:20.681+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.682254+0000) 2020-02-05T21:34:20.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:20.681+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.681997+0000) 2020-02-05T21:34:20.684 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:20.682+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.683360+0000) 2020-02-05T21:34:20.685 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:20.682+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.682620+0000) 2020-02-05T21:34:20.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:20.683+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.684244+0000) 2020-02-05T21:34:20.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:20.685+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.686139+0000) 2020-02-05T21:34:20.699 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:20.686+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.687196+0000) 2020-02-05T21:34:20.699 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:20.687+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.687865+0000) 2020-02-05T21:34:20.699 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:20.687+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.688181+0000) 2020-02-05T21:34:20.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:20.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:20.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:20.684+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.685623+0000) 2020-02-05T21:34:20.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:20.685+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.686128+0000) 2020-02-05T21:34:20.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:20.686+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.687368+0000) 2020-02-05T21:34:20.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:20.687+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.688421+0000) 2020-02-05T21:34:20.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:20.691+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:20.692501+0000) 2020-02-05T21:34:22.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:22.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:22.244308+0000) 2020-02-05T21:34:22.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:22.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:22.644778+0000) 2020-02-05T21:34:25.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:25.697+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.698346+0000) 2020-02-05T21:34:25.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:25.697+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.698792+0000) 2020-02-05T21:34:25.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:25.698+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.699416+0000) 2020-02-05T21:34:25.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:25.700+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.701222+0000) 2020-02-05T21:34:25.703 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:25.701+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.701550+0000) 2020-02-05T21:34:25.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:25.701+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.701879+0000) 2020-02-05T21:34:25.704 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:25.702+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.702966+0000) 2020-02-05T21:34:25.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:25.702+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.703286+0000) 2020-02-05T21:34:25.705 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:25.703+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.704178+0000) 2020-02-05T21:34:25.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:25.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.704631+0000) 2020-02-05T21:34:25.706 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:25.704+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.704779+0000) 2020-02-05T21:34:25.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:25.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.705208+0000) 2020-02-05T21:34:25.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:25.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.708464+0000) 2020-02-05T21:34:25.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:25.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.708668+0000) 2020-02-05T21:34:25.711 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:25.709+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:25.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:25.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:25.857291+0000) 2020-02-05T21:34:26.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:26.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:26.754373+0000) 2020-02-05T21:34:27.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:27.001+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:27.001387+0000) 2020-02-05T21:34:27.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:27.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:27.001632+0000) 2020-02-05T21:34:27.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:27.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:27.011564+0000) 2020-02-05T21:34:27.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:27.448+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:27.449206+0000) 2020-02-05T21:34:29.111 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:29.109+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:29.109989+0000) 2020-02-05T21:34:29.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:29.392+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:29.393438+0000) 2020-02-05T21:34:30.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:30.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.001587+0000) 2020-02-05T21:34:30.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:30.234+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.235015+0000) 2020-02-05T21:34:30.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:30.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.253363+0000) 2020-02-05T21:34:30.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:30.261+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.262548+0000) 2020-02-05T21:34:30.716 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:30.713+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.714865+0000) 2020-02-05T21:34:30.725 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:30.717+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.717530+0000) 2020-02-05T21:34:30.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:30.717+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.717900+0000) 2020-02-05T21:34:30.726 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:30.719+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.720039+0000) 2020-02-05T21:34:30.726 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:30.719+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.720213+0000) 2020-02-05T21:34:30.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:30.720+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.720747+0000) 2020-02-05T21:34:30.726 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:30.720+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.721242+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:30.715+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.716992+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:30.716+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.717220+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:30.717+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.718489+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:30.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.718739+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:30.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.721313+0000) 2020-02-05T21:34:30.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:30.722+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.724033+0000) 2020-02-05T21:34:30.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:30.718+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:30.719063+0000) 2020-02-05T21:34:30.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:30.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:32.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:32.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:32.244566+0000) 2020-02-05T21:34:32.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:32.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:32.645029+0000) 2020-02-05T21:34:35.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:35.732+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.733427+0000) 2020-02-05T21:34:35.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:35.733+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.734175+0000) 2020-02-05T21:34:35.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:35.733+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.734862+0000) 2020-02-05T21:34:35.737 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:35.734+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.735683+0000) 2020-02-05T21:34:35.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:35.735+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.735418+0000) 2020-02-05T21:34:35.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:35.735+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.735958+0000) 2020-02-05T21:34:35.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:35.735+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.736773+0000) 2020-02-05T21:34:35.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:35.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.737276+0000) 2020-02-05T21:34:35.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:35.737+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.737543+0000) 2020-02-05T21:34:35.739 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:35.738+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.738474+0000) 2020-02-05T21:34:35.740 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:35.737+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.738952+0000) 2020-02-05T21:34:35.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:35.738+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.738939+0000) 2020-02-05T21:34:35.741 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:35.738+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.739093+0000) 2020-02-05T21:34:35.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:35.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:35.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:35.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.743071+0000) 2020-02-05T21:34:35.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:35.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:35.857533+0000) 2020-02-05T21:34:36.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:36.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:36.754599+0000) 2020-02-05T21:34:37.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:37.001+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:37.001667+0000) 2020-02-05T21:34:37.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:37.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:37.001864+0000) 2020-02-05T21:34:37.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:37.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:37.011791+0000) 2020-02-05T21:34:37.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:37.455+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:37.456852+0000) 2020-02-05T21:34:39.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:39.110+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:39.110325+0000) 2020-02-05T21:34:39.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:39.392+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:39.393634+0000) 2020-02-05T21:34:40.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:40.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.001801+0000) 2020-02-05T21:34:40.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:40.235+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.235257+0000) 2020-02-05T21:34:40.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:40.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.253638+0000) 2020-02-05T21:34:40.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:40.262+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.262787+0000) 2020-02-05T21:34:40.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:40.746+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.747175+0000) 2020-02-05T21:34:40.749 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:40.747+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.747816+0000) 2020-02-05T21:34:40.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:40.748+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.749204+0000) 2020-02-05T21:34:40.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:40.748+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.748984+0000) 2020-02-05T21:34:40.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:40.749+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.750109+0000) 2020-02-05T21:34:40.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:40.749+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.750403+0000) 2020-02-05T21:34:40.752 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:40.750+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.751554+0000) 2020-02-05T21:34:40.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:40.751+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.752448+0000) 2020-02-05T21:34:40.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:40.752+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.752717+0000) 2020-02-05T21:34:40.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:40.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.753197+0000) 2020-02-05T21:34:40.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:40.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.753317+0000) 2020-02-05T21:34:40.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:40.753+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.753494+0000) 2020-02-05T21:34:40.755 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:40.752+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.753221+0000) 2020-02-05T21:34:40.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:40.757+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:40.758211+0000) 2020-02-05T21:34:40.760 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:40.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89917 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:42.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:42.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:42.244818+0000) 2020-02-05T21:34:42.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:42.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:42.645337+0000) 2020-02-05T21:34:43.949 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:34:43.955 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:34:45.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:45.763+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.764343+0000) 2020-02-05T21:34:45.767 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:45.764+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.765933+0000) 2020-02-05T21:34:45.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:45.765+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.766349+0000) 2020-02-05T21:34:45.768 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:45.765+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.766761+0000) 2020-02-05T21:34:45.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:45.766+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.766546+0000) 2020-02-05T21:34:45.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:45.766+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.767221+0000) 2020-02-05T21:34:45.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:45.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.767610+0000) 2020-02-05T21:34:45.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:45.766+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.767735+0000) 2020-02-05T21:34:45.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:45.767+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.768254+0000) 2020-02-05T21:34:45.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:45.768+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.768806+0000) 2020-02-05T21:34:45.771 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:45.768+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.769293+0000) 2020-02-05T21:34:45.772 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:45.768+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.769017+0000) 2020-02-05T21:34:45.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:45.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 89955 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:45.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:45.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.774173+0000) 2020-02-05T21:34:45.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:45.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.774384+0000) 2020-02-05T21:34:45.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:45.857+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:45.857782+0000) 2020-02-05T21:34:46.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:46.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:46.754894+0000) 2020-02-05T21:34:47.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:47.001+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:47.001890+0000) 2020-02-05T21:34:47.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:47.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:47.002072+0000) 2020-02-05T21:34:47.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:47.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:47.012028+0000) 2020-02-05T21:34:47.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:47.464+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:47.465187+0000) 2020-02-05T21:34:49.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:49.109+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:49.110641+0000) 2020-02-05T21:34:49.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:49.393+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:49.393822+0000) 2020-02-05T21:34:50.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:50.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.002029+0000) 2020-02-05T21:34:50.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:50.234+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.235559+0000) 2020-02-05T21:34:50.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:50.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.253872+0000) 2020-02-05T21:34:50.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:50.262+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.263015+0000) 2020-02-05T21:34:50.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:50.780+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.781724+0000) 2020-02-05T21:34:50.783 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:50.780+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.782146+0000) 2020-02-05T21:34:50.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:50.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.783116+0000) 2020-02-05T21:34:50.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:50.782+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.783953+0000) 2020-02-05T21:34:50.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:50.782+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.784252+0000) 2020-02-05T21:34:50.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:50.783+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.784657+0000) 2020-02-05T21:34:50.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:50.784+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.785362+0000) 2020-02-05T21:34:50.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:50.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.785771+0000) 2020-02-05T21:34:50.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:50.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.786314+0000) 2020-02-05T21:34:50.788 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:50.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.786973+0000) 2020-02-05T21:34:50.788 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:50.785+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.787083+0000) 2020-02-05T21:34:50.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:50.786+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.787126+0000) 2020-02-05T21:34:50.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:50.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.787703+0000) 2020-02-05T21:34:50.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:50.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:50.790540+0000) 2020-02-05T21:34:50.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:50.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90000 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:52.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:52.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:52.245072+0000) 2020-02-05T21:34:52.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:52.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:52.645571+0000) 2020-02-05T21:34:55.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:55.797+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.798261+0000) 2020-02-05T21:34:55.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:55.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.799230+0000) 2020-02-05T21:34:55.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:34:55.798+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.800216+0000) 2020-02-05T21:34:55.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:55.799+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.801298+0000) 2020-02-05T21:34:55.803 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:55.799+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.801249+0000) 2020-02-05T21:34:55.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:55.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.801718+0000) 2020-02-05T21:34:55.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:55.801+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.802756+0000) 2020-02-05T21:34:55.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:34:55.802+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.803588+0000) 2020-02-05T21:34:55.805 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:34:55.802+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.803547+0000) 2020-02-05T21:34:55.805 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:34:55.802+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.804348+0000) 2020-02-05T21:34:55.806 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:55.802+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.803913+0000) 2020-02-05T21:34:55.807 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:34:55.803+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.804607+0000) 2020-02-05T21:34:55.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:55.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.805136+0000) 2020-02-05T21:34:55.811 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:34:55.808+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:34:55.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:55.810+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.811430+0000) 2020-02-05T21:34:55.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:34:55.856+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:55.858016+0000) 2020-02-05T21:34:56.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:34:56.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:56.755136+0000) 2020-02-05T21:34:57.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:34:57.001+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:57.002137+0000) 2020-02-05T21:34:57.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:34:57.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:57.002313+0000) 2020-02-05T21:34:57.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:34:57.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:57.012311+0000) 2020-02-05T21:34:57.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:57.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:57.473182+0000) 2020-02-05T21:34:59.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:34:59.109+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:59.110902+0000) 2020-02-05T21:34:59.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:34:59.393+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:34:59.394061+0000) 2020-02-05T21:35:00.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:00.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.002261+0000) 2020-02-05T21:35:00.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:00.234+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.235795+0000) 2020-02-05T21:35:00.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:00.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.254105+0000) 2020-02-05T21:35:00.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:00.262+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.263261+0000) 2020-02-05T21:35:00.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:00.814+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.815041+0000) 2020-02-05T21:35:00.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:00.814+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.816020+0000) 2020-02-05T21:35:00.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:00.816+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.817423+0000) 2020-02-05T21:35:00.820 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:00.816+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.818260+0000) 2020-02-05T21:35:00.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:00.818+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.819530+0000) 2020-02-05T21:35:00.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:00.819+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.820251+0000) 2020-02-05T21:35:00.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:00.819+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.820652+0000) 2020-02-05T21:35:00.822 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:00.819+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.820803+0000) 2020-02-05T21:35:00.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:00.820+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.821152+0000) 2020-02-05T21:35:00.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:00.819+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.821113+0000) 2020-02-05T21:35:00.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:00.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.821795+0000) 2020-02-05T21:35:00.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:00.820+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.821801+0000) 2020-02-05T21:35:00.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:00.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.825318+0000) 2020-02-05T21:35:00.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:00.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:00.825529+0000) 2020-02-05T21:35:00.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:00.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:02.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:02.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:02.245319+0000) 2020-02-05T21:35:02.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:02.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:02.645805+0000) 2020-02-05T21:35:05.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:05.832+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.833482+0000) 2020-02-05T21:35:05.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:05.833+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.834592+0000) 2020-02-05T21:35:05.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:05.834+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.835284+0000) 2020-02-05T21:35:05.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:05.834+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.835786+0000) 2020-02-05T21:35:05.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:05.835+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.836150+0000) 2020-02-05T21:35:05.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:05.835+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.836534+0000) 2020-02-05T21:35:05.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:05.835+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.837111+0000) 2020-02-05T21:35:05.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:05.836+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.837600+0000) 2020-02-05T21:35:05.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:05.836+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.837969+0000) 2020-02-05T21:35:05.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:05.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.838502+0000) 2020-02-05T21:35:05.840 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:05.837+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.838843+0000) 2020-02-05T21:35:05.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:05.837+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.838705+0000) 2020-02-05T21:35:05.841 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:05.837+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.838714+0000) 2020-02-05T21:35:05.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:05.840+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.841171+0000) 2020-02-05T21:35:05.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:05.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:05.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:05.858+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:05.858316+0000) 2020-02-05T21:35:06.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:06.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:06.755292+0000) 2020-02-05T21:35:07.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:07.000+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:07.002408+0000) 2020-02-05T21:35:07.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:07.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:07.002621+0000) 2020-02-05T21:35:07.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:07.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:07.012538+0000) 2020-02-05T21:35:07.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:07.481+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:07.481587+0000) 2020-02-05T21:35:09.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:09.110+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:09.111141+0000) 2020-02-05T21:35:09.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:09.394+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:09.394361+0000) 2020-02-05T21:35:10.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:10.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.002534+0000) 2020-02-05T21:35:10.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:10.235+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.236048+0000) 2020-02-05T21:35:10.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:10.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.254345+0000) 2020-02-05T21:35:10.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:10.263+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.263497+0000) 2020-02-05T21:35:10.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:10.851+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.851911+0000) 2020-02-05T21:35:10.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:10.850+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.852372+0000) 2020-02-05T21:35:10.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:10.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.852660+0000) 2020-02-05T21:35:10.854 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:10.851+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.853167+0000) 2020-02-05T21:35:10.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:10.852+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.853673+0000) 2020-02-05T21:35:10.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:10.854+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.854215+0000) 2020-02-05T21:35:10.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:10.854+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.854563+0000) 2020-02-05T21:35:10.856 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:10.854+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.854767+0000) 2020-02-05T21:35:10.857 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:10.853+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.855399+0000) 2020-02-05T21:35:10.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:10.855+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.855236+0000) 2020-02-05T21:35:10.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:10.854+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.856019+0000) 2020-02-05T21:35:10.858 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:10.854+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.856152+0000) 2020-02-05T21:35:10.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:10.856+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.856395+0000) 2020-02-05T21:35:10.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:10.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:10.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:10.862+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:10.862422+0000) 2020-02-05T21:35:12.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:12.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:12.245571+0000) 2020-02-05T21:35:12.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:12.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:12.646137+0000) 2020-02-05T21:35:14.098 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:35:14.105 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:35:15.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:15.857+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.858547+0000) 2020-02-05T21:35:15.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:15.866+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.867821+0000) 2020-02-05T21:35:15.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:15.868+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.869199+0000) 2020-02-05T21:35:15.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:15.868+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.869686+0000) 2020-02-05T21:35:15.871 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:15.868+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.869974+0000) 2020-02-05T21:35:15.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:15.868+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.870249+0000) 2020-02-05T21:35:15.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:15.870+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.871213+0000) 2020-02-05T21:35:15.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:15.868+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.870184+0000) 2020-02-05T21:35:15.873 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:15.869+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.870866+0000) 2020-02-05T21:35:15.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:15.869+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.871405+0000) 2020-02-05T21:35:15.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:15.870+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.871630+0000) 2020-02-05T21:35:15.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:15.871+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.873050+0000) 2020-02-05T21:35:15.875 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:15.871+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.873104+0000) 2020-02-05T21:35:15.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:15.872+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.873585+0000) 2020-02-05T21:35:15.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:15.875+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:15.876925+0000) 2020-02-05T21:35:15.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:15.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90206 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:16.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:16.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:16.755498+0000) 2020-02-05T21:35:17.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:17.001+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:17.002651+0000) 2020-02-05T21:35:17.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:17.001+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:17.002887+0000) 2020-02-05T21:35:17.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:17.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:17.012769+0000) 2020-02-05T21:35:17.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:17.488+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:17.489119+0000) 2020-02-05T21:35:19.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:19.110+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:19.111379+0000) 2020-02-05T21:35:19.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:19.393+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:19.394595+0000) 2020-02-05T21:35:20.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:20.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.002769+0000) 2020-02-05T21:35:20.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:20.235+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.236301+0000) 2020-02-05T21:35:20.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:20.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.254633+0000) 2020-02-05T21:35:20.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:20.262+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.263740+0000) 2020-02-05T21:35:20.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:20.881+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.882826+0000) 2020-02-05T21:35:20.885 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:20.882+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.883867+0000) 2020-02-05T21:35:20.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:20.883+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.884471+0000) 2020-02-05T21:35:20.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:20.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.885198+0000) 2020-02-05T21:35:20.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:20.885+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.886237+0000) 2020-02-05T21:35:20.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:20.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.886684+0000) 2020-02-05T21:35:20.889 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:20.885+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.887408+0000) 2020-02-05T21:35:20.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:20.886+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.887902+0000) 2020-02-05T21:35:20.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:20.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.888470+0000) 2020-02-05T21:35:20.890 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:20.887+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.888962+0000) 2020-02-05T21:35:20.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:20.887+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.889185+0000) 2020-02-05T21:35:20.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:20.888+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.889504+0000) 2020-02-05T21:35:20.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:20.888+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.889736+0000) 2020-02-05T21:35:20.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:20.891+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:20.892831+0000) 2020-02-05T21:35:20.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:20.892+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:22.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:22.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:22.245826+0000) 2020-02-05T21:35:22.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:22.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:22.646448+0000) 2020-02-05T21:35:25.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:25.857+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.858783+0000) 2020-02-05T21:35:25.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:25.898+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.899681+0000) 2020-02-05T21:35:25.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:25.899+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.900195+0000) 2020-02-05T21:35:25.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:25.899+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.900623+0000) 2020-02-05T21:35:25.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:25.899+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.900949+0000) 2020-02-05T21:35:25.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:25.900+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.901847+0000) 2020-02-05T21:35:25.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:25.901+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.902418+0000) 2020-02-05T21:35:25.904 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:25.901+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.903063+0000) 2020-02-05T21:35:25.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:25.903+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.904156+0000) 2020-02-05T21:35:25.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:25.903+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.905274+0000) 2020-02-05T21:35:25.907 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:25.903+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.905331+0000) 2020-02-05T21:35:25.907 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:25.904+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.905746+0000) 2020-02-05T21:35:25.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:25.904+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.905997+0000) 2020-02-05T21:35:25.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:25.906+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.908030+0000) 2020-02-05T21:35:25.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:25.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:25.908252+0000) 2020-02-05T21:35:25.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:25.909+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:26.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:26.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:26.755713+0000) 2020-02-05T21:35:27.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:27.002+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:27.002915+0000) 2020-02-05T21:35:27.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:27.002+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:27.003140+0000) 2020-02-05T21:35:27.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:27.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:27.012997+0000) 2020-02-05T21:35:27.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:27.499+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:27.500576+0000) 2020-02-05T21:35:29.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:29.111+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:29.111661+0000) 2020-02-05T21:35:29.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:29.393+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:29.394853+0000) 2020-02-05T21:35:30.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:30.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.002997+0000) 2020-02-05T21:35:30.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:30.235+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.236549+0000) 2020-02-05T21:35:30.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:30.254+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.254886+0000) 2020-02-05T21:35:30.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:30.263+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.263962+0000) 2020-02-05T21:35:30.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:30.914+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.915817+0000) 2020-02-05T21:35:30.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:30.915+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.916230+0000) 2020-02-05T21:35:30.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:30.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.916635+0000) 2020-02-05T21:35:30.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:30.916+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.917772+0000) 2020-02-05T21:35:30.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:30.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.918372+0000) 2020-02-05T21:35:30.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:30.917+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.918770+0000) 2020-02-05T21:35:30.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:30.917+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.918537+0000) 2020-02-05T21:35:30.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:30.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.919486+0000) 2020-02-05T21:35:30.921 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:30.918+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.920053+0000) 2020-02-05T21:35:30.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:30.919+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.920897+0000) 2020-02-05T21:35:30.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:30.920+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.921339+0000) 2020-02-05T21:35:30.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:30.920+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.921791+0000) 2020-02-05T21:35:30.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:30.924+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.925411+0000) 2020-02-05T21:35:30.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:30.924+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:30.925633+0000) 2020-02-05T21:35:30.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:30.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:32.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:32.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:32.246096+0000) 2020-02-05T21:35:32.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:32.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:32.646712+0000) 2020-02-05T21:35:35.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:35.858+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.859030+0000) 2020-02-05T21:35:35.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:35.934+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.935420+0000) 2020-02-05T21:35:35.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:35.936+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.938130+0000) 2020-02-05T21:35:35.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:35.935+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.937008+0000) 2020-02-05T21:35:35.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:35.935+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.937286+0000) 2020-02-05T21:35:35.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:35.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.938729+0000) 2020-02-05T21:35:35.943 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:35.937+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.939095+0000) 2020-02-05T21:35:35.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:35.932+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.934047+0000) 2020-02-05T21:35:35.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:35.933+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.934972+0000) 2020-02-05T21:35:35.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:35.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.938497+0000) 2020-02-05T21:35:35.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:35.934+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.935273+0000) 2020-02-05T21:35:35.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:35.934+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.936056+0000) 2020-02-05T21:35:35.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:35.935+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.936611+0000) 2020-02-05T21:35:35.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:35.936+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.937650+0000) 2020-02-05T21:35:35.946 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:35.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:35.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:35.946+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:35.947630+0000) 2020-02-05T21:35:36.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:36.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:36.755956+0000) 2020-02-05T21:35:37.005 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:37.002+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:37.003165+0000) 2020-02-05T21:35:37.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:37.002+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:37.003360+0000) 2020-02-05T21:35:37.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:37.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:37.013256+0000) 2020-02-05T21:35:37.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:37.507+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:37.507698+0000) 2020-02-05T21:35:39.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:39.111+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:39.111904+0000) 2020-02-05T21:35:39.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:39.394+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:39.395115+0000) 2020-02-05T21:35:40.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:40.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.003281+0000) 2020-02-05T21:35:40.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:40.236+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.236845+0000) 2020-02-05T21:35:40.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:40.254+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.255138+0000) 2020-02-05T21:35:40.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:40.263+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.264272+0000) 2020-02-05T21:35:40.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:40.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.950492+0000) 2020-02-05T21:35:40.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:40.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.952928+0000) 2020-02-05T21:35:40.959 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:40.949+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.950681+0000) 2020-02-05T21:35:40.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:40.950+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.951334+0000) 2020-02-05T21:35:40.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:40.950+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.951814+0000) 2020-02-05T21:35:40.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:40.951+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.952908+0000) 2020-02-05T21:35:40.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:40.952+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.953936+0000) 2020-02-05T21:35:40.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:40.948+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.949272+0000) 2020-02-05T21:35:40.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:40.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.952130+0000) 2020-02-05T21:35:40.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:40.952+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.952887+0000) 2020-02-05T21:35:40.961 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:40.954+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.954607+0000) 2020-02-05T21:35:40.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:40.954+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.954938+0000) 2020-02-05T21:35:40.961 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:40.955+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.955894+0000) 2020-02-05T21:35:40.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:40.959+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:40.960305+0000) 2020-02-05T21:35:40.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:40.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:42.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:42.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:42.246362+0000) 2020-02-05T21:35:42.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:42.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:42.646967+0000) 2020-02-05T21:35:44.237 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:35:44.243 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:35:45.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:45.858+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.859351+0000) 2020-02-05T21:35:45.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:45.966+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.967731+0000) 2020-02-05T21:35:45.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:45.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.968202+0000) 2020-02-05T21:35:45.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:45.968+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.969158+0000) 2020-02-05T21:35:45.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:45.968+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.969656+0000) 2020-02-05T21:35:45.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:45.969+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.970206+0000) 2020-02-05T21:35:45.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:45.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.970251+0000) 2020-02-05T21:35:45.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:45.970+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.970632+0000) 2020-02-05T21:35:45.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:45.970+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.971394+0000) 2020-02-05T21:35:45.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:45.971+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.972004+0000) 2020-02-05T21:35:45.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:45.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.972516+0000) 2020-02-05T21:35:45.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:45.971+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.972874+0000) 2020-02-05T21:35:45.974 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:45.972+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.972937+0000) 2020-02-05T21:35:45.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:45.972+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.973236+0000) 2020-02-05T21:35:45.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:45.975+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:45.976779+0000) 2020-02-05T21:35:45.980 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:45.978+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:46.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:46.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:46.756164+0000) 2020-02-05T21:35:47.005 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:47.002+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:47.003395+0000) 2020-02-05T21:35:47.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:47.003+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:47.003634+0000) 2020-02-05T21:35:47.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:47.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:47.013502+0000) 2020-02-05T21:35:47.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:47.515+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:47.516171+0000) 2020-02-05T21:35:49.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:49.111+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:49.112200+0000) 2020-02-05T21:35:49.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:49.394+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:49.395350+0000) 2020-02-05T21:35:50.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:50.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.003497+0000) 2020-02-05T21:35:50.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:50.236+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.237125+0000) 2020-02-05T21:35:50.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:50.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.255385+0000) 2020-02-05T21:35:50.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:50.264+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.264541+0000) 2020-02-05T21:35:50.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:50.982+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.983587+0000) 2020-02-05T21:35:50.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:50.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.984733+0000) 2020-02-05T21:35:50.988 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:50.986+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.986778+0000) 2020-02-05T21:35:50.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:50.987+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.987517+0000) 2020-02-05T21:35:50.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:50.986+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.987941+0000) 2020-02-05T21:35:50.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:50.987+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.988166+0000) 2020-02-05T21:35:50.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:50.987+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.988398+0000) 2020-02-05T21:35:50.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:50.988+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.989016+0000) 2020-02-05T21:35:50.990 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:50.988+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.989585+0000) 2020-02-05T21:35:50.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:50.989+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.990403+0000) 2020-02-05T21:35:50.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:50.990+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.990646+0000) 2020-02-05T21:35:50.992 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:50.990+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.991305+0000) 2020-02-05T21:35:50.993 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:50.991+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.991653+0000) 2020-02-05T21:35:50.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:50.994+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:50.995626+0000) 2020-02-05T21:35:50.997 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:50.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:52.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:52.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:52.246702+0000) 2020-02-05T21:35:52.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:52.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:52.647230+0000) 2020-02-05T21:35:55.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:55.859+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:55.859616+0000) 2020-02-05T21:35:56.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:56.001+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.002392+0000) 2020-02-05T21:35:56.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:35:56.002+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.003327+0000) 2020-02-05T21:35:56.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:56.002+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.003796+0000) 2020-02-05T21:35:56.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:35:56.003+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.004491+0000) 2020-02-05T21:35:56.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:35:56.003+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.004316+0000) 2020-02-05T21:35:56.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:56.004+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.004854+0000) 2020-02-05T21:35:56.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:56.005+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.005894+0000) 2020-02-05T21:35:56.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:35:56.005+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.006378+0000) 2020-02-05T21:35:56.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:56.005+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.006843+0000) 2020-02-05T21:35:56.008 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:35:56.006+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.007069+0000) 2020-02-05T21:35:56.009 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:35:56.006+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.007196+0000) 2020-02-05T21:35:56.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:56.007+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.008264+0000) 2020-02-05T21:35:56.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:56.007+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.008219+0000) 2020-02-05T21:35:56.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:56.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.011437+0000) 2020-02-05T21:35:56.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:35:56.011+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:35:56.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:35:56.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:56.756382+0000) 2020-02-05T21:35:57.005 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:35:57.003+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:57.003691+0000) 2020-02-05T21:35:57.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:35:57.003+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:57.003872+0000) 2020-02-05T21:35:57.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:35:57.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:57.013724+0000) 2020-02-05T21:35:57.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:57.524+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:57.524401+0000) 2020-02-05T21:35:59.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:35:59.112+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:59.112489+0000) 2020-02-05T21:35:59.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:35:59.395+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:35:59.395580+0000) 2020-02-05T21:36:00.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:00.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:00.003725+0000) 2020-02-05T21:36:00.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:00.237+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:00.237377+0000) 2020-02-05T21:36:00.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:00.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:00.255646+0000) 2020-02-05T21:36:00.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:00.264+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:00.264774+0000) 2020-02-05T21:36:01.019 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:01.016+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.017565+0000) 2020-02-05T21:36:01.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:01.017+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.018330+0000) 2020-02-05T21:36:01.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:01.018+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.019166+0000) 2020-02-05T21:36:01.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:01.018+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.019525+0000) 2020-02-05T21:36:01.021 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:01.018+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.019835+0000) 2020-02-05T21:36:01.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:01.019+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.020185+0000) 2020-02-05T21:36:01.023 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:01.021+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.021644+0000) 2020-02-05T21:36:01.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:01.021+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.021564+0000) 2020-02-05T21:36:01.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:01.020+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.021278+0000) 2020-02-05T21:36:01.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:01.021+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.022274+0000) 2020-02-05T21:36:01.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:01.022+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.022726+0000) 2020-02-05T21:36:01.025 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:01.022+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.023055+0000) 2020-02-05T21:36:01.025 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:01.022+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.024000+0000) 2020-02-05T21:36:01.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:01.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:01.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:01.028+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:01.029465+0000) 2020-02-05T21:36:02.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:02.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:02.246970+0000) 2020-02-05T21:36:02.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:02.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:02.647471+0000) 2020-02-05T21:36:05.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:05.859+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:05.859897+0000) 2020-02-05T21:36:06.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:06.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.034339+0000) 2020-02-05T21:36:06.038 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:06.036+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.036719+0000) 2020-02-05T21:36:06.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:06.037+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.037305+0000) 2020-02-05T21:36:06.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:06.037+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.037758+0000) 2020-02-05T21:36:06.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:06.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.038219+0000) 2020-02-05T21:36:06.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:06.038+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.038293+0000) 2020-02-05T21:36:06.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:06.038+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.038808+0000) 2020-02-05T21:36:06.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:06.037+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.037586+0000) 2020-02-05T21:36:06.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:06.038+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.039045+0000) 2020-02-05T21:36:06.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:06.039+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.039543+0000) 2020-02-05T21:36:06.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:06.039+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.040222+0000) 2020-02-05T21:36:06.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:06.040+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.040581+0000) 2020-02-05T21:36:06.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:06.040+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.041209+0000) 2020-02-05T21:36:06.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:06.041+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.041913+0000) 2020-02-05T21:36:06.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:06.045+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:06.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:06.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:06.756592+0000) 2020-02-05T21:36:07.005 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:07.003+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:07.003910+0000) 2020-02-05T21:36:07.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:07.003+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:07.004075+0000) 2020-02-05T21:36:07.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:07.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:07.013970+0000) 2020-02-05T21:36:07.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:07.532+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:07.532268+0000) 2020-02-05T21:36:09.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:09.112+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:09.112740+0000) 2020-02-05T21:36:09.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:09.395+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:09.395805+0000) 2020-02-05T21:36:10.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:10.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:10.003960+0000) 2020-02-05T21:36:10.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:10.237+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:10.237670+0000) 2020-02-05T21:36:10.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:10.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:10.255884+0000) 2020-02-05T21:36:10.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:10.263+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:10.264999+0000) 2020-02-05T21:36:11.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:11.052+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.052693+0000) 2020-02-05T21:36:11.055 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:11.052+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.053081+0000) 2020-02-05T21:36:11.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:11.052+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.053315+0000) 2020-02-05T21:36:11.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:11.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.053858+0000) 2020-02-05T21:36:11.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:11.053+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.054417+0000) 2020-02-05T21:36:11.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:11.053+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.054213+0000) 2020-02-05T21:36:11.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:11.053+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.054776+0000) 2020-02-05T21:36:11.057 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:11.054+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.055169+0000) 2020-02-05T21:36:11.057 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:11.054+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.055839+0000) 2020-02-05T21:36:11.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:11.055+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.055564+0000) 2020-02-05T21:36:11.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:11.055+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.056293+0000) 2020-02-05T21:36:11.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:11.057+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.057594+0000) 2020-02-05T21:36:11.059 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:11.057+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.058049+0000) 2020-02-05T21:36:11.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:11.062+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:11.063548+0000) 2020-02-05T21:36:11.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:11.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:12.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:12.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:12.247236+0000) 2020-02-05T21:36:12.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:12.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:12.647652+0000) 2020-02-05T21:36:14.384 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:36:14.390 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:36:15.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:15.859+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:15.860177+0000) 2020-02-05T21:36:16.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:16.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.068915+0000) 2020-02-05T21:36:16.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:16.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.071446+0000) 2020-02-05T21:36:16.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:16.070+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.071785+0000) 2020-02-05T21:36:16.077 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:16.072+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.073275+0000) 2020-02-05T21:36:16.077 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:16.073+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.075023+0000) 2020-02-05T21:36:16.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:16.071+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.072091+0000) 2020-02-05T21:36:16.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:16.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.072742+0000) 2020-02-05T21:36:16.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:16.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.073541+0000) 2020-02-05T21:36:16.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:16.073+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.073871+0000) 2020-02-05T21:36:16.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:16.074+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.074482+0000) 2020-02-05T21:36:16.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:16.074+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.074965+0000) 2020-02-05T21:36:16.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:16.074+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.075485+0000) 2020-02-05T21:36:16.079 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:16.074+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.075508+0000) 2020-02-05T21:36:16.081 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:16.079+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:16.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:16.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.081799+0000) 2020-02-05T21:36:16.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:16.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:16.756778+0000) 2020-02-05T21:36:17.006 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:17.004+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:17.004139+0000) 2020-02-05T21:36:17.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:17.004+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:17.004254+0000) 2020-02-05T21:36:17.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:17.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:17.014182+0000) 2020-02-05T21:36:17.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:17.539+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:17.540170+0000) 2020-02-05T21:36:19.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:19.111+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:19.112999+0000) 2020-02-05T21:36:19.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:19.395+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:19.396012+0000) 2020-02-05T21:36:20.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:20.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:20.004188+0000) 2020-02-05T21:36:20.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:20.236+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:20.237917+0000) 2020-02-05T21:36:20.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:20.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:20.256118+0000) 2020-02-05T21:36:20.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:20.264+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:20.265261+0000) 2020-02-05T21:36:21.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:21.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.086014+0000) 2020-02-05T21:36:21.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:21.084+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.086195+0000) 2020-02-05T21:36:21.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:21.085+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.086589+0000) 2020-02-05T21:36:21.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:21.086+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.087249+0000) 2020-02-05T21:36:21.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:21.086+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.087903+0000) 2020-02-05T21:36:21.089 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:21.087+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.088362+0000) 2020-02-05T21:36:21.090 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:21.087+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.088993+0000) 2020-02-05T21:36:21.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:21.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.089588+0000) 2020-02-05T21:36:21.091 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:21.088+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.089614+0000) 2020-02-05T21:36:21.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:21.088+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.089984+0000) 2020-02-05T21:36:21.092 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:21.089+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.091076+0000) 2020-02-05T21:36:21.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:21.090+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.091563+0000) 2020-02-05T21:36:21.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:21.089+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.091407+0000) 2020-02-05T21:36:21.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:21.095+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:21.096223+0000) 2020-02-05T21:36:21.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:21.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:22.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:22.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:22.247509+0000) 2020-02-05T21:36:22.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:22.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:22.647887+0000) 2020-02-05T21:36:25.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:25.859+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:25.860409+0000) 2020-02-05T21:36:26.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:26.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.102814+0000) 2020-02-05T21:36:26.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:26.102+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.103208+0000) 2020-02-05T21:36:26.105 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:26.103+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.104358+0000) 2020-02-05T21:36:26.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:26.103+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.104717+0000) 2020-02-05T21:36:26.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:26.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.105452+0000) 2020-02-05T21:36:26.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:26.104+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.105550+0000) 2020-02-05T21:36:26.107 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:26.105+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.106419+0000) 2020-02-05T21:36:26.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:26.104+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.105988+0000) 2020-02-05T21:36:26.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:26.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.107388+0000) 2020-02-05T21:36:26.109 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:26.106+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.108337+0000) 2020-02-05T21:36:26.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:26.107+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.108557+0000) 2020-02-05T21:36:26.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:26.107+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.109105+0000) 2020-02-05T21:36:26.110 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:26.108+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.109367+0000) 2020-02-05T21:36:26.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:26.111+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.112133+0000) 2020-02-05T21:36:26.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:26.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90794 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:26.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:26.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:26.756967+0000) 2020-02-05T21:36:27.006 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:27.003+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:27.004375+0000) 2020-02-05T21:36:27.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:27.003+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:27.004513+0000) 2020-02-05T21:36:27.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:27.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:27.014383+0000) 2020-02-05T21:36:27.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:27.547+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:27.548186+0000) 2020-02-05T21:36:29.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:29.112+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:29.113253+0000) 2020-02-05T21:36:29.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:29.395+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:29.396353+0000) 2020-02-05T21:36:30.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:30.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:30.004417+0000) 2020-02-05T21:36:30.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:30.237+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:30.238172+0000) 2020-02-05T21:36:30.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:30.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:30.256348+0000) 2020-02-05T21:36:30.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:30.264+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:30.265440+0000) 2020-02-05T21:36:31.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:31.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.121544+0000) 2020-02-05T21:36:31.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:31.122+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.123207+0000) 2020-02-05T21:36:31.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:31.121+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.122903+0000) 2020-02-05T21:36:31.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:31.122+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.123771+0000) 2020-02-05T21:36:31.125 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:31.123+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.124265+0000) 2020-02-05T21:36:31.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:31.123+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.125028+0000) 2020-02-05T21:36:31.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:31.123+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.125219+0000) 2020-02-05T21:36:31.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:31.124+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.125548+0000) 2020-02-05T21:36:31.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:31.125+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.126585+0000) 2020-02-05T21:36:31.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:31.125+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.126942+0000) 2020-02-05T21:36:31.129 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:31.125+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.126725+0000) 2020-02-05T21:36:31.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:31.125+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.127097+0000) 2020-02-05T21:36:31.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:31.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.127647+0000) 2020-02-05T21:36:31.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:31.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:31.132026+0000) 2020-02-05T21:36:31.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:31.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:32.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:32.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:32.247792+0000) 2020-02-05T21:36:32.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:32.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:32.648143+0000) 2020-02-05T21:36:35.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:35.860+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:35.860728+0000) 2020-02-05T21:36:36.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:36.137+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.138916+0000) 2020-02-05T21:36:36.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:36.138+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.139790+0000) 2020-02-05T21:36:36.141 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:36.139+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.140357+0000) 2020-02-05T21:36:36.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:36.140+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.141513+0000) 2020-02-05T21:36:36.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:36.140+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.141930+0000) 2020-02-05T21:36:36.143 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:36.141+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.142616+0000) 2020-02-05T21:36:36.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:36.141+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.142689+0000) 2020-02-05T21:36:36.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:36.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.143496+0000) 2020-02-05T21:36:36.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:36.142+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.143836+0000) 2020-02-05T21:36:36.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:36.142+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.143829+0000) 2020-02-05T21:36:36.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:36.142+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.144259+0000) 2020-02-05T21:36:36.146 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:36.143+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.144179+0000) 2020-02-05T21:36:36.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:36.143+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.144945+0000) 2020-02-05T21:36:36.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:36.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90877 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:36.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:36.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.153278+0000) 2020-02-05T21:36:36.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:36.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:36.757187+0000) 2020-02-05T21:36:37.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:37.003+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:37.004693+0000) 2020-02-05T21:36:37.006 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:37.003+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:37.004666+0000) 2020-02-05T21:36:37.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:37.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:37.014593+0000) 2020-02-05T21:36:37.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:37.555+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:37.556407+0000) 2020-02-05T21:36:39.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:39.112+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:39.113536+0000) 2020-02-05T21:36:39.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:39.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:39.396629+0000) 2020-02-05T21:36:40.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:40.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:40.004665+0000) 2020-02-05T21:36:40.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:40.237+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:40.238462+0000) 2020-02-05T21:36:40.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:40.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:40.256609+0000) 2020-02-05T21:36:40.267 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:40.265+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:40.265675+0000) 2020-02-05T21:36:41.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:41.154+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.156104+0000) 2020-02-05T21:36:41.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:41.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.157371+0000) 2020-02-05T21:36:41.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:41.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.157719+0000) 2020-02-05T21:36:41.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:41.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.158799+0000) 2020-02-05T21:36:41.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:41.158+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.159658+0000) 2020-02-05T21:36:41.161 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:41.158+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.159779+0000) 2020-02-05T21:36:41.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:41.159+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.160693+0000) 2020-02-05T21:36:41.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:41.159+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.160517+0000) 2020-02-05T21:36:41.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:41.160+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.161491+0000) 2020-02-05T21:36:41.164 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:41.159+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.161387+0000) 2020-02-05T21:36:41.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:41.160+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.162396+0000) 2020-02-05T21:36:41.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:41.160+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.162008+0000) 2020-02-05T21:36:41.165 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:41.161+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.163178+0000) 2020-02-05T21:36:41.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:41.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:41.166133+0000) 2020-02-05T21:36:41.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:41.165+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:42.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:42.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:42.248048+0000) 2020-02-05T21:36:42.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:42.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:42.648377+0000) 2020-02-05T21:36:44.554 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:36:44.560 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:36:45.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:45.860+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:45.860992+0000) 2020-02-05T21:36:46.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:46.170+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.172392+0000) 2020-02-05T21:36:46.175 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:46.172+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.173709+0000) 2020-02-05T21:36:46.175 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:46.173+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.174288+0000) 2020-02-05T21:36:46.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:46.174+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.175219+0000) 2020-02-05T21:36:46.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:46.173+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.175333+0000) 2020-02-05T21:36:46.177 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:46.174+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.176104+0000) 2020-02-05T21:36:46.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:46.175+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.176574+0000) 2020-02-05T21:36:46.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:46.176+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.177423+0000) 2020-02-05T21:36:46.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:46.176+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.177824+0000) 2020-02-05T21:36:46.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:46.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.177996+0000) 2020-02-05T21:36:46.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:46.177+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.178566+0000) 2020-02-05T21:36:46.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:46.177+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.178627+0000) 2020-02-05T21:36:46.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:46.177+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.178856+0000) 2020-02-05T21:36:46.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:46.181+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.182989+0000) 2020-02-05T21:36:46.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:46.182+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 90962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:46.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:46.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:46.757493+0000) 2020-02-05T21:36:47.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:47.004+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:47.004920+0000) 2020-02-05T21:36:47.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:47.004+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:47.004920+0000) 2020-02-05T21:36:47.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:47.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:47.014832+0000) 2020-02-05T21:36:47.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:47.564+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:47.564678+0000) 2020-02-05T21:36:49.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:49.113+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:49.113812+0000) 2020-02-05T21:36:49.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:49.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:49.396888+0000) 2020-02-05T21:36:50.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:50.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:50.004904+0000) 2020-02-05T21:36:50.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:50.238+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:50.238752+0000) 2020-02-05T21:36:50.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:50.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:50.256844+0000) 2020-02-05T21:36:50.267 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:50.265+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:50.265913+0000) 2020-02-05T21:36:51.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:51.188+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.189205+0000) 2020-02-05T21:36:51.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:51.190+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.191447+0000) 2020-02-05T21:36:51.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:51.191+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.192129+0000) 2020-02-05T21:36:51.194 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:51.191+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.192544+0000) 2020-02-05T21:36:51.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:51.191+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.192868+0000) 2020-02-05T21:36:51.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:51.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.193466+0000) 2020-02-05T21:36:51.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:51.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.193622+0000) 2020-02-05T21:36:51.196 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:51.192+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.193935+0000) 2020-02-05T21:36:51.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:51.193+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.194535+0000) 2020-02-05T21:36:51.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:51.194+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.195369+0000) 2020-02-05T21:36:51.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:51.194+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.195574+0000) 2020-02-05T21:36:51.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:51.194+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.195464+0000) 2020-02-05T21:36:51.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:51.194+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.195983+0000) 2020-02-05T21:36:51.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:51.199+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:51.200246+0000) 2020-02-05T21:36:51.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:51.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:52.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:52.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:52.248327+0000) 2020-02-05T21:36:52.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:52.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:52.648631+0000) 2020-02-05T21:36:55.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:55.861+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:55.861215+0000) 2020-02-05T21:36:56.208 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:56.205+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.206581+0000) 2020-02-05T21:36:56.209 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:36:56.206+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.207468+0000) 2020-02-05T21:36:56.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:56.206+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.207896+0000) 2020-02-05T21:36:56.210 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:56.207+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.208933+0000) 2020-02-05T21:36:56.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:36:56.209+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.210446+0000) 2020-02-05T21:36:56.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:36:56.209+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.210836+0000) 2020-02-05T21:36:56.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:56.209+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.211170+0000) 2020-02-05T21:36:56.213 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:56.210+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.212012+0000) 2020-02-05T21:36:56.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:56.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.213195+0000) 2020-02-05T21:36:56.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:36:56.212+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.213472+0000) 2020-02-05T21:36:56.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:36:56.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.213615+0000) 2020-02-05T21:36:56.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:56.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.213546+0000) 2020-02-05T21:36:56.216 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:36:56.212+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.214054+0000) 2020-02-05T21:36:56.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:36:56.216+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:36:56.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:56.217+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.218306+0000) 2020-02-05T21:36:56.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:36:56.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:56.757693+0000) 2020-02-05T21:36:57.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:36:57.004+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:57.005183+0000) 2020-02-05T21:36:57.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:36:57.004+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:57.005183+0000) 2020-02-05T21:36:57.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:36:57.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:57.015065+0000) 2020-02-05T21:36:57.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:57.572+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:57.572326+0000) 2020-02-05T21:36:59.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:36:59.113+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:59.114075+0000) 2020-02-05T21:36:59.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:36:59.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:36:59.397129+0000) 2020-02-05T21:37:00.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:00.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:00.005170+0000) 2020-02-05T21:37:00.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:00.238+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:00.239087+0000) 2020-02-05T21:37:00.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:00.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:00.257095+0000) 2020-02-05T21:37:00.267 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:00.266+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:00.266186+0000) 2020-02-05T21:37:01.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:01.223+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.223371+0000) 2020-02-05T21:37:01.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:01.224+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.224917+0000) 2020-02-05T21:37:01.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:01.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.225543+0000) 2020-02-05T21:37:01.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:01.225+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.225759+0000) 2020-02-05T21:37:01.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:01.224+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.226066+0000) 2020-02-05T21:37:01.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:01.225+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.226726+0000) 2020-02-05T21:37:01.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:01.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.226451+0000) 2020-02-05T21:37:01.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:01.225+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.227403+0000) 2020-02-05T21:37:01.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:01.227+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.227343+0000) 2020-02-05T21:37:01.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:01.227+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.228606+0000) 2020-02-05T21:37:01.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:01.227+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.228588+0000) 2020-02-05T21:37:01.230 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:01.228+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.228427+0000) 2020-02-05T21:37:01.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:01.228+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.228737+0000) 2020-02-05T21:37:01.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:01.233+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:01.233984+0000) 2020-02-05T21:37:01.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:01.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:02.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:02.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:02.248692+0000) 2020-02-05T21:37:02.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:02.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:02.648824+0000) 2020-02-05T21:37:05.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:05.860+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:05.861432+0000) 2020-02-05T21:37:06.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:06.238+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.239921+0000) 2020-02-05T21:37:06.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:06.237+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.239405+0000) 2020-02-05T21:37:06.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:06.238+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.240403+0000) 2020-02-05T21:37:06.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:06.240+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.241572+0000) 2020-02-05T21:37:06.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:06.241+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.242477+0000) 2020-02-05T21:37:06.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:06.240+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.242333+0000) 2020-02-05T21:37:06.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:06.242+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.243895+0000) 2020-02-05T21:37:06.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:06.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.244068+0000) 2020-02-05T21:37:06.246 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:06.243+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.244770+0000) 2020-02-05T21:37:06.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:06.243+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.245009+0000) 2020-02-05T21:37:06.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:06.244+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.245647+0000) 2020-02-05T21:37:06.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:06.244+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.245582+0000) 2020-02-05T21:37:06.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:06.244+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.246320+0000) 2020-02-05T21:37:06.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:06.247+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.248575+0000) 2020-02-05T21:37:06.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:06.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:06.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:06.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:06.757906+0000) 2020-02-05T21:37:07.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:07.004+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:07.005505+0000) 2020-02-05T21:37:07.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:07.004+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:07.005501+0000) 2020-02-05T21:37:07.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:07.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:07.015342+0000) 2020-02-05T21:37:07.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:07.578+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:07.579159+0000) 2020-02-05T21:37:09.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:09.113+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:09.114327+0000) 2020-02-05T21:37:09.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:09.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:09.397384+0000) 2020-02-05T21:37:10.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:10.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:10.005416+0000) 2020-02-05T21:37:10.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:10.238+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:10.239334+0000) 2020-02-05T21:37:10.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:10.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:10.257348+0000) 2020-02-05T21:37:10.267 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:10.265+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:10.266420+0000) 2020-02-05T21:37:11.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:11.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.258013+0000) 2020-02-05T21:37:11.260 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:11.257+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.258391+0000) 2020-02-05T21:37:11.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:11.258+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.259070+0000) 2020-02-05T21:37:11.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:11.258+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.259594+0000) 2020-02-05T21:37:11.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:11.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.260865+0000) 2020-02-05T21:37:11.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:11.259+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.259750+0000) 2020-02-05T21:37:11.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:11.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.261067+0000) 2020-02-05T21:37:11.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:11.260+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.262009+0000) 2020-02-05T21:37:11.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:11.261+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.261822+0000) 2020-02-05T21:37:11.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:11.262+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.262543+0000) 2020-02-05T21:37:11.265 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:11.262+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.264048+0000) 2020-02-05T21:37:11.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:11.263+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.264089+0000) 2020-02-05T21:37:11.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:11.263+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.264346+0000) 2020-02-05T21:37:11.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:11.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:11.267332+0000) 2020-02-05T21:37:11.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:11.269+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:12.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:12.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:12.248947+0000) 2020-02-05T21:37:12.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:12.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:12.649061+0000) 2020-02-05T21:37:14.707 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:37:14.714 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:37:15.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:15.860+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:15.861723+0000) 2020-02-05T21:37:16.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:16.275+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.275713+0000) 2020-02-05T21:37:16.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:16.277+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.277602+0000) 2020-02-05T21:37:16.280 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:16.276+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.277311+0000) 2020-02-05T21:37:16.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:16.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.278323+0000) 2020-02-05T21:37:16.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:16.277+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.278464+0000) 2020-02-05T21:37:16.280 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:16.277+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.279027+0000) 2020-02-05T21:37:16.281 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:16.278+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.279763+0000) 2020-02-05T21:37:16.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:16.278+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.279246+0000) 2020-02-05T21:37:16.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:16.279+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.280159+0000) 2020-02-05T21:37:16.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:16.280+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.280787+0000) 2020-02-05T21:37:16.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:16.280+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.280848+0000) 2020-02-05T21:37:16.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:16.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.281404+0000) 2020-02-05T21:37:16.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:16.281+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.281502+0000) 2020-02-05T21:37:16.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:16.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.286864+0000) 2020-02-05T21:37:16.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:16.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:16.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:16.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:16.758119+0000) 2020-02-05T21:37:17.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:17.005+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:17.005752+0000) 2020-02-05T21:37:17.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:17.005+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:17.005752+0000) 2020-02-05T21:37:17.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:17.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:17.015552+0000) 2020-02-05T21:37:17.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:17.586+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:17.587675+0000) 2020-02-05T21:37:19.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:19.114+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:19.114583+0000) 2020-02-05T21:37:19.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:19.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:19.397647+0000) 2020-02-05T21:37:20.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:20.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:20.005680+0000) 2020-02-05T21:37:20.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:20.239+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:20.239681+0000) 2020-02-05T21:37:20.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:20.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:20.257610+0000) 2020-02-05T21:37:20.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:20.265+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:20.266657+0000) 2020-02-05T21:37:21.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:21.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.291822+0000) 2020-02-05T21:37:21.294 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:21.291+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.292588+0000) 2020-02-05T21:37:21.295 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:21.292+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.293854+0000) 2020-02-05T21:37:21.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:21.292+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.293250+0000) 2020-02-05T21:37:21.296 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:21.293+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.294419+0000) 2020-02-05T21:37:21.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:21.296+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.296910+0000) 2020-02-05T21:37:21.298 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:21.296+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.297164+0000) 2020-02-05T21:37:21.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:21.297+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.297691+0000) 2020-02-05T21:37:21.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:21.296+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.297857+0000) 2020-02-05T21:37:21.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:21.298+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.298604+0000) 2020-02-05T21:37:21.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:21.298+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.298594+0000) 2020-02-05T21:37:21.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:21.298+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.298953+0000) 2020-02-05T21:37:21.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:21.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.298525+0000) 2020-02-05T21:37:21.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:21.302+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:21.303304+0000) 2020-02-05T21:37:21.305 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:21.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:22.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:22.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:22.249213+0000) 2020-02-05T21:37:22.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:22.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:22.649368+0000) 2020-02-05T21:37:25.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:25.861+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:25.861950+0000) 2020-02-05T21:37:26.310 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:26.307+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.308665+0000) 2020-02-05T21:37:26.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:26.311+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.312138+0000) 2020-02-05T21:37:26.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:26.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.312562+0000) 2020-02-05T21:37:26.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:26.314+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.315756+0000) 2020-02-05T21:37:26.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:26.313+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.314187+0000) 2020-02-05T21:37:26.322 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:26.315+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.316246+0000) 2020-02-05T21:37:26.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:26.315+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.316455+0000) 2020-02-05T21:37:26.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:26.311+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.311509+0000) 2020-02-05T21:37:26.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:26.312+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.312766+0000) 2020-02-05T21:37:26.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:26.312+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.313215+0000) 2020-02-05T21:37:26.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:26.313+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.313781+0000) 2020-02-05T21:37:26.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:26.315+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.315768+0000) 2020-02-05T21:37:26.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:26.316+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.316601+0000) 2020-02-05T21:37:26.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:26.321+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.322682+0000) 2020-02-05T21:37:26.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:26.321+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:26.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:26.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:26.758335+0000) 2020-02-05T21:37:27.007 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:27.005+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:27.006003+0000) 2020-02-05T21:37:27.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:27.005+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:27.006003+0000) 2020-02-05T21:37:27.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:27.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:27.015757+0000) 2020-02-05T21:37:27.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:27.597+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:27.597901+0000) 2020-02-05T21:37:29.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:29.114+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:29.114837+0000) 2020-02-05T21:37:29.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:29.396+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:29.397895+0000) 2020-02-05T21:37:30.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:30.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:30.005903+0000) 2020-02-05T21:37:30.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:30.239+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:30.239921+0000) 2020-02-05T21:37:30.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:30.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:30.257862+0000) 2020-02-05T21:37:30.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:30.265+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:30.266904+0000) 2020-02-05T21:37:31.328 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:31.325+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.326882+0000) 2020-02-05T21:37:31.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:31.327+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.328360+0000) 2020-02-05T21:37:31.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:31.328+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.328881+0000) 2020-02-05T21:37:31.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:31.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.330313+0000) 2020-02-05T21:37:31.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:31.330+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.331357+0000) 2020-02-05T21:37:31.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:31.331+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.332274+0000) 2020-02-05T21:37:31.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:31.332+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.332485+0000) 2020-02-05T21:37:31.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:31.332+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.333176+0000) 2020-02-05T21:37:31.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:31.332+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.333217+0000) 2020-02-05T21:37:31.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:31.333+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.333535+0000) 2020-02-05T21:37:31.335 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:31.332+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.333904+0000) 2020-02-05T21:37:31.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:31.333+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.334153+0000) 2020-02-05T21:37:31.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:31.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.334467+0000) 2020-02-05T21:37:31.340 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:31.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:31.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:31.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:31.340483+0000) 2020-02-05T21:37:32.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:32.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:32.249463+0000) 2020-02-05T21:37:32.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:32.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:32.649621+0000) 2020-02-05T21:37:35.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:35.861+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:35.862176+0000) 2020-02-05T21:37:36.345 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:36.343+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.344227+0000) 2020-02-05T21:37:36.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:36.345+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.346159+0000) 2020-02-05T21:37:36.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:36.345+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.346271+0000) 2020-02-05T21:37:36.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:36.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.346902+0000) 2020-02-05T21:37:36.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:36.346+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.347367+0000) 2020-02-05T21:37:36.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:36.346+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.347899+0000) 2020-02-05T21:37:36.349 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:36.347+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.348403+0000) 2020-02-05T21:37:36.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:36.348+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.349131+0000) 2020-02-05T21:37:36.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:36.349+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.350908+0000) 2020-02-05T21:37:36.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:36.349+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.349776+0000) 2020-02-05T21:37:36.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:36.349+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.350007+0000) 2020-02-05T21:37:36.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:36.350+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.351145+0000) 2020-02-05T21:37:36.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:36.350+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.351399+0000) 2020-02-05T21:37:36.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:36.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:36.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:36.353+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.354799+0000) 2020-02-05T21:37:36.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:36.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:36.758584+0000) 2020-02-05T21:37:37.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:37.005+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:37.006205+0000) 2020-02-05T21:37:37.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:37.005+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:37.006220+0000) 2020-02-05T21:37:37.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:37.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:37.015971+0000) 2020-02-05T21:37:37.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:37.607+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:37.607745+0000) 2020-02-05T21:37:39.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:39.115+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:39.115183+0000) 2020-02-05T21:37:39.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:39.397+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:39.398137+0000) 2020-02-05T21:37:40.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:40.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:40.006144+0000) 2020-02-05T21:37:40.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:40.240+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:40.240153+0000) 2020-02-05T21:37:40.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:40.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:40.258112+0000) 2020-02-05T21:37:40.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:40.266+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:40.267145+0000) 2020-02-05T21:37:41.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:41.361+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.362388+0000) 2020-02-05T21:37:41.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:41.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.362991+0000) 2020-02-05T21:37:41.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:41.362+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.363336+0000) 2020-02-05T21:37:41.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:41.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.363819+0000) 2020-02-05T21:37:41.365 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:41.363+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.364350+0000) 2020-02-05T21:37:41.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:41.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.364470+0000) 2020-02-05T21:37:41.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:41.364+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.364936+0000) 2020-02-05T21:37:41.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:41.363+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.364842+0000) 2020-02-05T21:37:41.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:41.365+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.365684+0000) 2020-02-05T21:37:41.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:41.365+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.366225+0000) 2020-02-05T21:37:41.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:41.366+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.366680+0000) 2020-02-05T21:37:41.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:41.366+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.367349+0000) 2020-02-05T21:37:41.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:41.367+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.367541+0000) 2020-02-05T21:37:41.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:41.369+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:41.370637+0000) 2020-02-05T21:37:41.374 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:41.372+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:42.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:42.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:42.249698+0000) 2020-02-05T21:37:42.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:42.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:42.649855+0000) 2020-02-05T21:37:44.838 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:37:44.844 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:37:45.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:45.862+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:45.862408+0000) 2020-02-05T21:37:46.381 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:46.378+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.379661+0000) 2020-02-05T21:37:46.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:46.378+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.379963+0000) 2020-02-05T21:37:46.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:46.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.381039+0000) 2020-02-05T21:37:46.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:46.380+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.381589+0000) 2020-02-05T21:37:46.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:46.380+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.381171+0000) 2020-02-05T21:37:46.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:46.381+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.382682+0000) 2020-02-05T21:37:46.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:46.381+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.382939+0000) 2020-02-05T21:37:46.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:46.383+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.383623+0000) 2020-02-05T21:37:46.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:46.383+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.384013+0000) 2020-02-05T21:37:46.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:46.383+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.384133+0000) 2020-02-05T21:37:46.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:46.383+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.384353+0000) 2020-02-05T21:37:46.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:46.384+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.384675+0000) 2020-02-05T21:37:46.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:46.384+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.384746+0000) 2020-02-05T21:37:46.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:46.386+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.387192+0000) 2020-02-05T21:37:46.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:46.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:46.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:46.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:46.758798+0000) 2020-02-05T21:37:47.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:47.005+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:47.006529+0000) 2020-02-05T21:37:47.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:47.005+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:47.006525+0000) 2020-02-05T21:37:47.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:47.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:47.016213+0000) 2020-02-05T21:37:47.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:47.615+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:47.615710+0000) 2020-02-05T21:37:49.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:49.114+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:49.115465+0000) 2020-02-05T21:37:49.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:49.398+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:49.398393+0000) 2020-02-05T21:37:50.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:50.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:50.006380+0000) 2020-02-05T21:37:50.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:50.239+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:50.240402+0000) 2020-02-05T21:37:50.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:50.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:50.258354+0000) 2020-02-05T21:37:50.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:50.266+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:50.267323+0000) 2020-02-05T21:37:51.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:51.395+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.396318+0000) 2020-02-05T21:37:51.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:51.396+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.397422+0000) 2020-02-05T21:37:51.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:51.396+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.397769+0000) 2020-02-05T21:37:51.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:51.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.398990+0000) 2020-02-05T21:37:51.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:51.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.399405+0000) 2020-02-05T21:37:51.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:51.398+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.399933+0000) 2020-02-05T21:37:51.401 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:51.399+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.400265+0000) 2020-02-05T21:37:51.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:51.398+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.399734+0000) 2020-02-05T21:37:51.402 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:51.400+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.401406+0000) 2020-02-05T21:37:51.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:51.400+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.401733+0000) 2020-02-05T21:37:51.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:51.400+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.401935+0000) 2020-02-05T21:37:51.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:51.401+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.402901+0000) 2020-02-05T21:37:51.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:51.401+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.402784+0000) 2020-02-05T21:37:51.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:51.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91509 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:51.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:51.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:51.408449+0000) 2020-02-05T21:37:52.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:52.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:52.249946+0000) 2020-02-05T21:37:52.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:52.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:52.650119+0000) 2020-02-05T21:37:55.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:55.862+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:55.862640+0000) 2020-02-05T21:37:56.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:56.412+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.412830+0000) 2020-02-05T21:37:56.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:56.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.414183+0000) 2020-02-05T21:37:56.415 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:37:56.414+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.414575+0000) 2020-02-05T21:37:56.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:56.413+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.415087+0000) 2020-02-05T21:37:56.417 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:56.414+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.415945+0000) 2020-02-05T21:37:56.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:37:56.416+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.416819+0000) 2020-02-05T21:37:56.419 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:56.417+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.417841+0000) 2020-02-05T21:37:56.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:37:56.416+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.417590+0000) 2020-02-05T21:37:56.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:37:56.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.418638+0000) 2020-02-05T21:37:56.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:37:56.417+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.418867+0000) 2020-02-05T21:37:56.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:56.419+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.419329+0000) 2020-02-05T21:37:56.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:37:56.417+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.419172+0000) 2020-02-05T21:37:56.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:56.418+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.419422+0000) 2020-02-05T21:37:56.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:56.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.422945+0000) 2020-02-05T21:37:56.425 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:37:56.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91548 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:37:56.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:37:56.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:56.759009+0000) 2020-02-05T21:37:57.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:37:57.005+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:57.006791+0000) 2020-02-05T21:37:57.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:37:57.005+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:57.006796+0000) 2020-02-05T21:37:57.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:37:57.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:57.016415+0000) 2020-02-05T21:37:57.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:57.625+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:57.626003+0000) 2020-02-05T21:37:59.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:37:59.114+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:59.115713+0000) 2020-02-05T21:37:59.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:37:59.397+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:37:59.398623+0000) 2020-02-05T21:38:00.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:00.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:00.006635+0000) 2020-02-05T21:38:00.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:00.239+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:00.240670+0000) 2020-02-05T21:38:00.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:00.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:00.258668+0000) 2020-02-05T21:38:00.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:00.266+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:00.267590+0000) 2020-02-05T21:38:01.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:01.429+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.430601+0000) 2020-02-05T21:38:01.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:01.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.431022+0000) 2020-02-05T21:38:01.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:01.430+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.431539+0000) 2020-02-05T21:38:01.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:01.431+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.432212+0000) 2020-02-05T21:38:01.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:01.431+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.432585+0000) 2020-02-05T21:38:01.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:01.431+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.432918+0000) 2020-02-05T21:38:01.435 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:01.432+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.433238+0000) 2020-02-05T21:38:01.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:01.432+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.433573+0000) 2020-02-05T21:38:01.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:01.432+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.433695+0000) 2020-02-05T21:38:01.436 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:01.433+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.435019+0000) 2020-02-05T21:38:01.437 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:01.433+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.434804+0000) 2020-02-05T21:38:01.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:01.433+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.434834+0000) 2020-02-05T21:38:01.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:01.433+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.435254+0000) 2020-02-05T21:38:01.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:01.439+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:01.440486+0000) 2020-02-05T21:38:01.442 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:01.439+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91591 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:02.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:02.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:02.250215+0000) 2020-02-05T21:38:02.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:02.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:02.650360+0000) 2020-02-05T21:38:05.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:05.861+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:05.862893+0000) 2020-02-05T21:38:06.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:06.445+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.446222+0000) 2020-02-05T21:38:06.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:06.445+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.447251+0000) 2020-02-05T21:38:06.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:06.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.448164+0000) 2020-02-05T21:38:06.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:06.447+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.448754+0000) 2020-02-05T21:38:06.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:06.447+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.449035+0000) 2020-02-05T21:38:06.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:06.447+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.449391+0000) 2020-02-05T21:38:06.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:06.449+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.450464+0000) 2020-02-05T21:38:06.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:06.449+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.451020+0000) 2020-02-05T21:38:06.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:06.450+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.451717+0000) 2020-02-05T21:38:06.453 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:06.451+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.452413+0000) 2020-02-05T21:38:06.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:06.450+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.452226+0000) 2020-02-05T21:38:06.454 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:06.450+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.452404+0000) 2020-02-05T21:38:06.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:06.451+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.452617+0000) 2020-02-05T21:38:06.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:06.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.456726+0000) 2020-02-05T21:38:06.464 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:06.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:06.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:06.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:06.759215+0000) 2020-02-05T21:38:07.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:07.006+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:07.007059+0000) 2020-02-05T21:38:07.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:07.006+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:07.007063+0000) 2020-02-05T21:38:07.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:07.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:07.016667+0000) 2020-02-05T21:38:07.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:07.633+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:07.634152+0000) 2020-02-05T21:38:09.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:09.115+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:09.115989+0000) 2020-02-05T21:38:09.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:09.397+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:09.398857+0000) 2020-02-05T21:38:10.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:10.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:10.006926+0000) 2020-02-05T21:38:10.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:10.239+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:10.240944+0000) 2020-02-05T21:38:10.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:10.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:10.258925+0000) 2020-02-05T21:38:10.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:10.266+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:10.267836+0000) 2020-02-05T21:38:11.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:11.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.464010+0000) 2020-02-05T21:38:11.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:11.463+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.464444+0000) 2020-02-05T21:38:11.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:11.463+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.465349+0000) 2020-02-05T21:38:11.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:11.465+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.466380+0000) 2020-02-05T21:38:11.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:11.465+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.466774+0000) 2020-02-05T21:38:11.468 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:11.465+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.466941+0000) 2020-02-05T21:38:11.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:11.465+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.467137+0000) 2020-02-05T21:38:11.469 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:11.466+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.467561+0000) 2020-02-05T21:38:11.469 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:11.466+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.467835+0000) 2020-02-05T21:38:11.469 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:11.466+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.468131+0000) 2020-02-05T21:38:11.470 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:11.467+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.468815+0000) 2020-02-05T21:38:11.470 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:11.468+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.469331+0000) 2020-02-05T21:38:11.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:11.468+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.469852+0000) 2020-02-05T21:38:11.475 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:11.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:11.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:11.476+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:11.477286+0000) 2020-02-05T21:38:12.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:12.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:12.250495+0000) 2020-02-05T21:38:12.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:12.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:12.650587+0000) 2020-02-05T21:38:14.979 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:38:14.985 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:38:15.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:15.862+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:15.863124+0000) 2020-02-05T21:38:16.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:16.479+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.480395+0000) 2020-02-05T21:38:16.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:16.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.481481+0000) 2020-02-05T21:38:16.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:16.481+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.482349+0000) 2020-02-05T21:38:16.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:16.481+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.483063+0000) 2020-02-05T21:38:16.484 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:16.482+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.483442+0000) 2020-02-05T21:38:16.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:16.482+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.483564+0000) 2020-02-05T21:38:16.486 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:16.483+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.484715+0000) 2020-02-05T21:38:16.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:16.483+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.484923+0000) 2020-02-05T21:38:16.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:16.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.485514+0000) 2020-02-05T21:38:16.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:16.484+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.485732+0000) 2020-02-05T21:38:16.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:16.484+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.486256+0000) 2020-02-05T21:38:16.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:16.485+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.486784+0000) 2020-02-05T21:38:16.488 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:16.485+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.486698+0000) 2020-02-05T21:38:16.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:16.489+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.490612+0000) 2020-02-05T21:38:16.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:16.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:16.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:16.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:16.759427+0000) 2020-02-05T21:38:17.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:17.006+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:17.007313+0000) 2020-02-05T21:38:17.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:17.006+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:17.007315+0000) 2020-02-05T21:38:17.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:17.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:17.016876+0000) 2020-02-05T21:38:17.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:17.644+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:17.645677+0000) 2020-02-05T21:38:19.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:19.115+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:19.116253+0000) 2020-02-05T21:38:19.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:19.398+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:19.399120+0000) 2020-02-05T21:38:20.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:20.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:20.007155+0000) 2020-02-05T21:38:20.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:20.240+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:20.241177+0000) 2020-02-05T21:38:20.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:20.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:20.259171+0000) 2020-02-05T21:38:20.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:20.267+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:20.268121+0000) 2020-02-05T21:38:21.499 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:21.496+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.497535+0000) 2020-02-05T21:38:21.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:21.497+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.498985+0000) 2020-02-05T21:38:21.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:21.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.499433+0000) 2020-02-05T21:38:21.501 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:21.498+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.499789+0000) 2020-02-05T21:38:21.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:21.498+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.500074+0000) 2020-02-05T21:38:21.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:21.499+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.500481+0000) 2020-02-05T21:38:21.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:21.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.501123+0000) 2020-02-05T21:38:21.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:21.500+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.501715+0000) 2020-02-05T21:38:21.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:21.500+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.502086+0000) 2020-02-05T21:38:21.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:21.501+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.503095+0000) 2020-02-05T21:38:21.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:21.502+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.503618+0000) 2020-02-05T21:38:21.505 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:21.503+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.504101+0000) 2020-02-05T21:38:21.505 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:21.502+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.504269+0000) 2020-02-05T21:38:21.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:21.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:21.509498+0000) 2020-02-05T21:38:21.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:21.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91753 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:22.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:22.250+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:22.250728+0000) 2020-02-05T21:38:22.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:22.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:22.650838+0000) 2020-02-05T21:38:25.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:25.862+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:25.863357+0000) 2020-02-05T21:38:26.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:26.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.515621+0000) 2020-02-05T21:38:26.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:26.514+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.516093+0000) 2020-02-05T21:38:26.518 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:26.515+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.516545+0000) 2020-02-05T21:38:26.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:26.516+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.517245+0000) 2020-02-05T21:38:26.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:26.516+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.517716+0000) 2020-02-05T21:38:26.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:26.517+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.518404+0000) 2020-02-05T21:38:26.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:26.517+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.519063+0000) 2020-02-05T21:38:26.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:26.518+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.519504+0000) 2020-02-05T21:38:26.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:26.518+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.520105+0000) 2020-02-05T21:38:26.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:26.519+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.520282+0000) 2020-02-05T21:38:26.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:26.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.520682+0000) 2020-02-05T21:38:26.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:26.519+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.520777+0000) 2020-02-05T21:38:26.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:26.520+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.521218+0000) 2020-02-05T21:38:26.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:26.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.526965+0000) 2020-02-05T21:38:26.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:26.526+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:26.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:26.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:26.759636+0000) 2020-02-05T21:38:27.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:27.006+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:27.007525+0000) 2020-02-05T21:38:27.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:27.006+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:27.007520+0000) 2020-02-05T21:38:27.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:27.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:27.017108+0000) 2020-02-05T21:38:27.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:27.651+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:27.652635+0000) 2020-02-05T21:38:29.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:29.115+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:29.116500+0000) 2020-02-05T21:38:29.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:29.398+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:29.399353+0000) 2020-02-05T21:38:30.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:30.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:30.007389+0000) 2020-02-05T21:38:30.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:30.240+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:30.241464+0000) 2020-02-05T21:38:30.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:30.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:30.259486+0000) 2020-02-05T21:38:30.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:30.267+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:30.268380+0000) 2020-02-05T21:38:31.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:31.533+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.534199+0000) 2020-02-05T21:38:31.536 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:31.532+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.534066+0000) 2020-02-05T21:38:31.537 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:31.534+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.535482+0000) 2020-02-05T21:38:31.537 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:31.534+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.535841+0000) 2020-02-05T21:38:31.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:31.535+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.536408+0000) 2020-02-05T21:38:31.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:31.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.537709+0000) 2020-02-05T21:38:31.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:31.536+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.537888+0000) 2020-02-05T21:38:31.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:31.537+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.538496+0000) 2020-02-05T21:38:31.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:31.537+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.539053+0000) 2020-02-05T21:38:31.541 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:31.537+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.539270+0000) 2020-02-05T21:38:31.541 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:31.538+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.539688+0000) 2020-02-05T21:38:31.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:31.539+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.540121+0000) 2020-02-05T21:38:31.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:31.539+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.540280+0000) 2020-02-05T21:38:31.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:31.543+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:31.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:31.546+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:31.547332+0000) 2020-02-05T21:38:32.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:32.250+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:32.250963+0000) 2020-02-05T21:38:32.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:32.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:32.651135+0000) 2020-02-05T21:38:35.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:35.862+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:35.863656+0000) 2020-02-05T21:38:36.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:36.549+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.550927+0000) 2020-02-05T21:38:36.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:36.550+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.551336+0000) 2020-02-05T21:38:36.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:36.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.552973+0000) 2020-02-05T21:38:36.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:36.552+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.553348+0000) 2020-02-05T21:38:36.555 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:36.552+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.553832+0000) 2020-02-05T21:38:36.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:36.553+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.555042+0000) 2020-02-05T21:38:36.556 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:36.553+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.554472+0000) 2020-02-05T21:38:36.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:36.554+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.555616+0000) 2020-02-05T21:38:36.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:36.554+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.555927+0000) 2020-02-05T21:38:36.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:36.555+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.556869+0000) 2020-02-05T21:38:36.558 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:36.555+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.557053+0000) 2020-02-05T21:38:36.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:36.555+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.557266+0000) 2020-02-05T21:38:36.559 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:36.556+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.557880+0000) 2020-02-05T21:38:36.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:36.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:36.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:36.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.563619+0000) 2020-02-05T21:38:36.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:36.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:36.759853+0000) 2020-02-05T21:38:37.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:37.007+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:37.007775+0000) 2020-02-05T21:38:37.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:37.007+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:37.007779+0000) 2020-02-05T21:38:37.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:37.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:37.017311+0000) 2020-02-05T21:38:37.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:37.660+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:37.660722+0000) 2020-02-05T21:38:39.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:39.116+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:39.116765+0000) 2020-02-05T21:38:39.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:39.399+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:39.399588+0000) 2020-02-05T21:38:40.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:40.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:40.007622+0000) 2020-02-05T21:38:40.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:40.241+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:40.241713+0000) 2020-02-05T21:38:40.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:40.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:40.259739+0000) 2020-02-05T21:38:40.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:40.268+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:40.268626+0000) 2020-02-05T21:38:41.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:41.568+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.569977+0000) 2020-02-05T21:38:41.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:41.569+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.570375+0000) 2020-02-05T21:38:41.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:41.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.570696+0000) 2020-02-05T21:38:41.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:41.570+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.571103+0000) 2020-02-05T21:38:41.573 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:41.570+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.571821+0000) 2020-02-05T21:38:41.574 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:41.571+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.572330+0000) 2020-02-05T21:38:41.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:41.572+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.572535+0000) 2020-02-05T21:38:41.575 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:41.573+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.573748+0000) 2020-02-05T21:38:41.575 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:41.573+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.574220+0000) 2020-02-05T21:38:41.575 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:41.573+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.574476+0000) 2020-02-05T21:38:41.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:41.574+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.574554+0000) 2020-02-05T21:38:41.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:41.574+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.574732+0000) 2020-02-05T21:38:41.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:41.574+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.575082+0000) 2020-02-05T21:38:41.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:41.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:41.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:41.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:41.580162+0000) 2020-02-05T21:38:42.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:42.250+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:42.251192+0000) 2020-02-05T21:38:42.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:42.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:42.651364+0000) 2020-02-05T21:38:45.117 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:38:45.123 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:38:45.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:45.863+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:45.863887+0000) 2020-02-05T21:38:46.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:46.585+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.586152+0000) 2020-02-05T21:38:46.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:46.586+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.587110+0000) 2020-02-05T21:38:46.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:46.586+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.587408+0000) 2020-02-05T21:38:46.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:46.586+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.587866+0000) 2020-02-05T21:38:46.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:46.588+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.588990+0000) 2020-02-05T21:38:46.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:46.589+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.589508+0000) 2020-02-05T21:38:46.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:46.589+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.589937+0000) 2020-02-05T21:38:46.592 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:46.587+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.588655+0000) 2020-02-05T21:38:46.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:46.588+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.589740+0000) 2020-02-05T21:38:46.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:46.590+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.591192+0000) 2020-02-05T21:38:46.594 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:46.591+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.591640+0000) 2020-02-05T21:38:46.594 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:46.591+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.591951+0000) 2020-02-05T21:38:46.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:46.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.591606+0000) 2020-02-05T21:38:46.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:46.595+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.596964+0000) 2020-02-05T21:38:46.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:46.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91953 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:46.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:46.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:46.760115+0000) 2020-02-05T21:38:47.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:47.007+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:47.008046+0000) 2020-02-05T21:38:47.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:47.007+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:47.008048+0000) 2020-02-05T21:38:47.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:47.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:47.017491+0000) 2020-02-05T21:38:47.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:47.668+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:47.669127+0000) 2020-02-05T21:38:49.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:49.116+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:49.117030+0000) 2020-02-05T21:38:49.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:49.399+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:49.399848+0000) 2020-02-05T21:38:50.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:50.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:50.007860+0000) 2020-02-05T21:38:50.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:50.241+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:50.241950+0000) 2020-02-05T21:38:50.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:50.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:50.259977+0000) 2020-02-05T21:38:50.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:50.268+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:50.268900+0000) 2020-02-05T21:38:51.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:51.603+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.603153+0000) 2020-02-05T21:38:51.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:51.603+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.603704+0000) 2020-02-05T21:38:51.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:51.603+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.603854+0000) 2020-02-05T21:38:51.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:51.604+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.604314+0000) 2020-02-05T21:38:51.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:51.604+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.604769+0000) 2020-02-05T21:38:51.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:51.604+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.604765+0000) 2020-02-05T21:38:51.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:51.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.605612+0000) 2020-02-05T21:38:51.608 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:51.605+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.605754+0000) 2020-02-05T21:38:51.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:51.605+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.606189+0000) 2020-02-05T21:38:51.610 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:51.606+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.607186+0000) 2020-02-05T21:38:51.610 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:51.606+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.606974+0000) 2020-02-05T21:38:51.610 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:51.606+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.607315+0000) 2020-02-05T21:38:51.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:51.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.606508+0000) 2020-02-05T21:38:51.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:51.613+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 91997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:51.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:51.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:51.610461+0000) 2020-02-05T21:38:52.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:52.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:52.251418+0000) 2020-02-05T21:38:52.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:52.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:52.651601+0000) 2020-02-05T21:38:55.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:55.863+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:55.864119+0000) 2020-02-05T21:38:56.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:38:56.620+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.620930+0000) 2020-02-05T21:38:56.624 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:56.620+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.620576+0000) 2020-02-05T21:38:56.624 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:56.617+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.618488+0000) 2020-02-05T21:38:56.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:56.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.619188+0000) 2020-02-05T21:38:56.625 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:38:56.618+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.619667+0000) 2020-02-05T21:38:56.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:38:56.620+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.621632+0000) 2020-02-05T21:38:56.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:56.621+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.622289+0000) 2020-02-05T21:38:56.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:38:56.623+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.624290+0000) 2020-02-05T21:38:56.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:38:56.624+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.624804+0000) 2020-02-05T21:38:56.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:56.624+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.624958+0000) 2020-02-05T21:38:56.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:38:56.625+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.625807+0000) 2020-02-05T21:38:56.627 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:56.625+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.626000+0000) 2020-02-05T21:38:56.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:56.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.628186+0000) 2020-02-05T21:38:56.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:56.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.628464+0000) 2020-02-05T21:38:56.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:38:56.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92034 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:38:56.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:38:56.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:56.760343+0000) 2020-02-05T21:38:57.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:38:57.008+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:57.008313+0000) 2020-02-05T21:38:57.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:38:57.008+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:57.008315+0000) 2020-02-05T21:38:57.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:38:57.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:57.017694+0000) 2020-02-05T21:38:57.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:57.675+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:57.676762+0000) 2020-02-05T21:38:59.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:38:59.117+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:59.117331+0000) 2020-02-05T21:38:59.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:38:59.399+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:38:59.400137+0000) 2020-02-05T21:39:00.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:00.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:00.008125+0000) 2020-02-05T21:39:00.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:00.241+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:00.242185+0000) 2020-02-05T21:39:00.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:00.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:00.260209+0000) 2020-02-05T21:39:00.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:00.268+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:00.269122+0000) 2020-02-05T21:39:01.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:01.636+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.637614+0000) 2020-02-05T21:39:01.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:01.637+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.638274+0000) 2020-02-05T21:39:01.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:01.637+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.638770+0000) 2020-02-05T21:39:01.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:01.640+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.641029+0000) 2020-02-05T21:39:01.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:01.641+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.641829+0000) 2020-02-05T21:39:01.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:01.642+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.642691+0000) 2020-02-05T21:39:01.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:01.642+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.643051+0000) 2020-02-05T21:39:01.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:01.642+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.643027+0000) 2020-02-05T21:39:01.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:01.642+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.643541+0000) 2020-02-05T21:39:01.646 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:01.642+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.643850+0000) 2020-02-05T21:39:01.646 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:01.643+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.644303+0000) 2020-02-05T21:39:01.646 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:01.643+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.643880+0000) 2020-02-05T21:39:01.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:01.644+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.645902+0000) 2020-02-05T21:39:01.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:01.646+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:01.647204+0000) 2020-02-05T21:39:01.651 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:01.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:02.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:02.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:02.251685+0000) 2020-02-05T21:39:02.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:02.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:02.651874+0000) 2020-02-05T21:39:05.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:05.863+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:05.864355+0000) 2020-02-05T21:39:06.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:06.653+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.654802+0000) 2020-02-05T21:39:06.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:06.655+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.656682+0000) 2020-02-05T21:39:06.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:06.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.658442+0000) 2020-02-05T21:39:06.660 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:06.658+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.659348+0000) 2020-02-05T21:39:06.661 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:06.658+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.659716+0000) 2020-02-05T21:39:06.661 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:06.659+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.660068+0000) 2020-02-05T21:39:06.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:06.659+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.660206+0000) 2020-02-05T21:39:06.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:06.660+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.660684+0000) 2020-02-05T21:39:06.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:06.660+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.661377+0000) 2020-02-05T21:39:06.663 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:06.661+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.661508+0000) 2020-02-05T21:39:06.663 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:06.661+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.662151+0000) 2020-02-05T21:39:06.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:06.661+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.661842+0000) 2020-02-05T21:39:06.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:06.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.665572+0000) 2020-02-05T21:39:06.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:06.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.666533+0000) 2020-02-05T21:39:06.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:06.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:06.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:06.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:06.760560+0000) 2020-02-05T21:39:07.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:07.008+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:07.008582+0000) 2020-02-05T21:39:07.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:07.008+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:07.008582+0000) 2020-02-05T21:39:07.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:07.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:07.017904+0000) 2020-02-05T21:39:07.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:07.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:07.688654+0000) 2020-02-05T21:39:09.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:09.117+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:09.117587+0000) 2020-02-05T21:39:09.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:09.399+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:09.400374+0000) 2020-02-05T21:39:10.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:10.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:10.008362+0000) 2020-02-05T21:39:10.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:10.242+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:10.242459+0000) 2020-02-05T21:39:10.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:10.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:10.260512+0000) 2020-02-05T21:39:10.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:10.268+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:10.269348+0000) 2020-02-05T21:39:11.675 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:11.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.673919+0000) 2020-02-05T21:39:11.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:11.673+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.674554+0000) 2020-02-05T21:39:11.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:11.675+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.676201+0000) 2020-02-05T21:39:11.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:11.676+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.676959+0000) 2020-02-05T21:39:11.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:11.676+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.677978+0000) 2020-02-05T21:39:11.679 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:11.677+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.678133+0000) 2020-02-05T21:39:11.680 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:11.677+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.678754+0000) 2020-02-05T21:39:11.680 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:11.678+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.679003+0000) 2020-02-05T21:39:11.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:11.678+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.679700+0000) 2020-02-05T21:39:11.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:11.679+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.679557+0000) 2020-02-05T21:39:11.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:11.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.680117+0000) 2020-02-05T21:39:11.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:11.679+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.680152+0000) 2020-02-05T21:39:11.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:11.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.683686+0000) 2020-02-05T21:39:11.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:11.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:11.685157+0000) 2020-02-05T21:39:11.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:11.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:12.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:12.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:12.251932+0000) 2020-02-05T21:39:12.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:12.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:12.652156+0000) 2020-02-05T21:39:15.249 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:39:15.255 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:39:15.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:15.863+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:15.864582+0000) 2020-02-05T21:39:16.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:16.692+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.693185+0000) 2020-02-05T21:39:16.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:16.692+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.693690+0000) 2020-02-05T21:39:16.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:16.693+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.694441+0000) 2020-02-05T21:39:16.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:16.693+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.694867+0000) 2020-02-05T21:39:16.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:16.693+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.695143+0000) 2020-02-05T21:39:16.697 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:16.694+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.695905+0000) 2020-02-05T21:39:16.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:16.695+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.696263+0000) 2020-02-05T21:39:16.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:16.695+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.696642+0000) 2020-02-05T21:39:16.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:16.696+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.697290+0000) 2020-02-05T21:39:16.699 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:16.696+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.697214+0000) 2020-02-05T21:39:16.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:16.696+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.697696+0000) 2020-02-05T21:39:16.700 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:16.696+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.698031+0000) 2020-02-05T21:39:16.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:16.696+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.698149+0000) 2020-02-05T21:39:16.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:16.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.702453+0000) 2020-02-05T21:39:16.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:16.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:16.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:16.760+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:16.760769+0000) 2020-02-05T21:39:17.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:17.007+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:17.008853+0000) 2020-02-05T21:39:17.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:17.007+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:17.008852+0000) 2020-02-05T21:39:17.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:17.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:17.018135+0000) 2020-02-05T21:39:17.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:17.696+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:17.696825+0000) 2020-02-05T21:39:19.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:19.116+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:19.117830+0000) 2020-02-05T21:39:19.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:19.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:19.400608+0000) 2020-02-05T21:39:20.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:20.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:20.008635+0000) 2020-02-05T21:39:20.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:20.241+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:20.242729+0000) 2020-02-05T21:39:20.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:20.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:20.260734+0000) 2020-02-05T21:39:20.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:20.268+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:20.269590+0000) 2020-02-05T21:39:21.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:21.707+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.708334+0000) 2020-02-05T21:39:21.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:21.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.710339+0000) 2020-02-05T21:39:21.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:21.709+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.710452+0000) 2020-02-05T21:39:21.712 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:21.709+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.710687+0000) 2020-02-05T21:39:21.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:21.710+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.711263+0000) 2020-02-05T21:39:21.713 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:21.710+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.711954+0000) 2020-02-05T21:39:21.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:21.710+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.712323+0000) 2020-02-05T21:39:21.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:21.711+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.712896+0000) 2020-02-05T21:39:21.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:21.711+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.713342+0000) 2020-02-05T21:39:21.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:21.712+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.713220+0000) 2020-02-05T21:39:21.715 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:21.712+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.713594+0000) 2020-02-05T21:39:21.716 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:21.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.713930+0000) 2020-02-05T21:39:21.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:21.712+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.714134+0000) 2020-02-05T21:39:21.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:21.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:21.718029+0000) 2020-02-05T21:39:21.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:21.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:22.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:22.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:22.252159+0000) 2020-02-05T21:39:22.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:22.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:22.652365+0000) 2020-02-05T21:39:25.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:25.864+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:25.864817+0000) 2020-02-05T21:39:26.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:26.726+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.727783+0000) 2020-02-05T21:39:26.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:26.726+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.727927+0000) 2020-02-05T21:39:26.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:26.727+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.728842+0000) 2020-02-05T21:39:26.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:26.728+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.729521+0000) 2020-02-05T21:39:26.731 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:26.728+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.730071+0000) 2020-02-05T21:39:26.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:26.729+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.730485+0000) 2020-02-05T21:39:26.732 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:26.730+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.731530+0000) 2020-02-05T21:39:26.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:26.729+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.731156+0000) 2020-02-05T21:39:26.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:26.730+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.731875+0000) 2020-02-05T21:39:26.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:26.731+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.732560+0000) 2020-02-05T21:39:26.734 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:26.731+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.732587+0000) 2020-02-05T21:39:26.734 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:26.731+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.732779+0000) 2020-02-05T21:39:26.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:26.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.732906+0000) 2020-02-05T21:39:26.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:26.734+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.735513+0000) 2020-02-05T21:39:26.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:26.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:26.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:26.760+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:26.760978+0000) 2020-02-05T21:39:27.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:27.008+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:27.009100+0000) 2020-02-05T21:39:27.011 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:27.008+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:27.009101+0000) 2020-02-05T21:39:27.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:27.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:27.018347+0000) 2020-02-05T21:39:27.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:27.705+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:27.705583+0000) 2020-02-05T21:39:29.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:29.117+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:29.118081+0000) 2020-02-05T21:39:29.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:29.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:29.400866+0000) 2020-02-05T21:39:30.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:30.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:30.008888+0000) 2020-02-05T21:39:30.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:30.242+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:30.242959+0000) 2020-02-05T21:39:30.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:30.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:30.260982+0000) 2020-02-05T21:39:30.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:30.269+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:30.269823+0000) 2020-02-05T21:39:31.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:31.741+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.743025+0000) 2020-02-05T21:39:31.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:31.744+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.745656+0000) 2020-02-05T21:39:31.758 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:31.745+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.746851+0000) 2020-02-05T21:39:31.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:31.746+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.747559+0000) 2020-02-05T21:39:31.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:31.746+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.747739+0000) 2020-02-05T21:39:31.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:31.746+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.748024+0000) 2020-02-05T21:39:31.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:31.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:31.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:31.741+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.742933+0000) 2020-02-05T21:39:31.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:31.742+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.743776+0000) 2020-02-05T21:39:31.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:31.743+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.744365+0000) 2020-02-05T21:39:31.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:31.743+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.744892+0000) 2020-02-05T21:39:31.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:31.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.746685+0000) 2020-02-05T21:39:31.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:31.751+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.752268+0000) 2020-02-05T21:39:31.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:31.745+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.746064+0000) 2020-02-05T21:39:31.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:31.745+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:31.746817+0000) 2020-02-05T21:39:32.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:32.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:32.252398+0000) 2020-02-05T21:39:32.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:32.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:32.652568+0000) 2020-02-05T21:39:35.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:35.864+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:35.865040+0000) 2020-02-05T21:39:36.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:36.759+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.760608+0000) 2020-02-05T21:39:36.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:36.759+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.760915+0000) 2020-02-05T21:39:36.766 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:36.761+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.762559+0000) 2020-02-05T21:39:36.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:36.762+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.763971+0000) 2020-02-05T21:39:36.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:36.761+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.762230+0000) 2020-02-05T21:39:36.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:36.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.762440+0000) 2020-02-05T21:39:36.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:36.762+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.763717+0000) 2020-02-05T21:39:36.767 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:36.762+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.763239+0000) 2020-02-05T21:39:36.767 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:36.762+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.763409+0000) 2020-02-05T21:39:36.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:36.763+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.764930+0000) 2020-02-05T21:39:36.768 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:36.763+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.765317+0000) 2020-02-05T21:39:36.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:36.763+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.765301+0000) 2020-02-05T21:39:36.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:36.764+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.765699+0000) 2020-02-05T21:39:36.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:36.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:36.770307+0000) 2020-02-05T21:39:36.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:36.768+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:37.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:37.008+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:37.009370+0000) 2020-02-05T21:39:37.011 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:37.008+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:37.009371+0000) 2020-02-05T21:39:37.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:37.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:37.018576+0000) 2020-02-05T21:39:37.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:37.717+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:37.717529+0000) 2020-02-05T21:39:39.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:39.116+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:39.118326+0000) 2020-02-05T21:39:39.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:39.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:39.401133+0000) 2020-02-05T21:39:40.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:40.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:40.009161+0000) 2020-02-05T21:39:40.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:40.242+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:40.243212+0000) 2020-02-05T21:39:40.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:40.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:40.261231+0000) 2020-02-05T21:39:40.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:40.269+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:40.270056+0000) 2020-02-05T21:39:41.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:41.775+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.776591+0000) 2020-02-05T21:39:41.778 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:41.775+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.777228+0000) 2020-02-05T21:39:41.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:41.776+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.777642+0000) 2020-02-05T21:39:41.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:41.776+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.778114+0000) 2020-02-05T21:39:41.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:41.777+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.778701+0000) 2020-02-05T21:39:41.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:41.778+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.779149+0000) 2020-02-05T21:39:41.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:41.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.780221+0000) 2020-02-05T21:39:41.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:41.779+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.780494+0000) 2020-02-05T21:39:41.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:41.779+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.780546+0000) 2020-02-05T21:39:41.782 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:41.779+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.780797+0000) 2020-02-05T21:39:41.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:41.780+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.781501+0000) 2020-02-05T21:39:41.783 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:41.780+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.781596+0000) 2020-02-05T21:39:41.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:41.780+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.782055+0000) 2020-02-05T21:39:41.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:41.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:41.786370+0000) 2020-02-05T21:39:41.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:41.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:42.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:42.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:42.252648+0000) 2020-02-05T21:39:42.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:42.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:42.652797+0000) 2020-02-05T21:39:43.513 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:43.511+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:43.511611+0000) 2020-02-05T21:39:45.386 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:39:45.393 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:39:45.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:45.865+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:45.865307+0000) 2020-02-05T21:39:46.793 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:46.791+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.792009+0000) 2020-02-05T21:39:46.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:46.792+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.792487+0000) 2020-02-05T21:39:46.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:46.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.793094+0000) 2020-02-05T21:39:46.795 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:46.792+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.793694+0000) 2020-02-05T21:39:46.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:46.793+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.794053+0000) 2020-02-05T21:39:46.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:46.794+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.794630+0000) 2020-02-05T21:39:46.797 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:46.793+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.795099+0000) 2020-02-05T21:39:46.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:46.795+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.795396+0000) 2020-02-05T21:39:46.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:46.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.795820+0000) 2020-02-05T21:39:46.798 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:46.795+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.796529+0000) 2020-02-05T21:39:46.798 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:46.795+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.796574+0000) 2020-02-05T21:39:46.799 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:46.796+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.797998+0000) 2020-02-05T21:39:46.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:46.796+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.798292+0000) 2020-02-05T21:39:46.803 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:46.801+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92435 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:46.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:46.802+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:46.802928+0000) 2020-02-05T21:39:47.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:47.008+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:47.009666+0000) 2020-02-05T21:39:47.011 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:47.008+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:47.009666+0000) 2020-02-05T21:39:47.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:47.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:47.018784+0000) 2020-02-05T21:39:47.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:47.724+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:47.725481+0000) 2020-02-05T21:39:49.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:49.117+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:49.118572+0000) 2020-02-05T21:39:49.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:49.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:49.401397+0000) 2020-02-05T21:39:50.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:50.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:50.009407+0000) 2020-02-05T21:39:50.245 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:50.242+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:50.243493+0000) 2020-02-05T21:39:50.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:50.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:50.261486+0000) 2020-02-05T21:39:50.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:50.269+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:50.270322+0000) 2020-02-05T21:39:51.823 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:51.807+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.809256+0000) 2020-02-05T21:39:51.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:51.809+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.811325+0000) 2020-02-05T21:39:51.824 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:51.811+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813164+0000) 2020-02-05T21:39:51.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:51.812+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813662+0000) 2020-02-05T21:39:51.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:51.812+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813780+0000) 2020-02-05T21:39:51.824 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:51.812+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.814222+0000) 2020-02-05T21:39:51.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:51.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:51.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:51.807+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.808119+0000) 2020-02-05T21:39:51.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:51.809+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.810659+0000) 2020-02-05T21:39:51.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:51.809+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.810996+0000) 2020-02-05T21:39:51.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:51.812+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813702+0000) 2020-02-05T21:39:51.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:51.818+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.819651+0000) 2020-02-05T21:39:51.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:51.811+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.812195+0000) 2020-02-05T21:39:51.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:51.812+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813439+0000) 2020-02-05T21:39:51.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:51.812+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:51.813781+0000) 2020-02-05T21:39:52.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:52.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:52.252869+0000) 2020-02-05T21:39:52.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:52.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:52.653026+0000) 2020-02-05T21:39:53.513 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:53.510+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:53.511843+0000) 2020-02-05T21:39:55.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:55.864+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:55.865559+0000) 2020-02-05T21:39:56.826 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:39:56.823+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.824868+0000) 2020-02-05T21:39:56.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:56.824+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.826030+0000) 2020-02-05T21:39:56.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:39:56.824+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.826284+0000) 2020-02-05T21:39:56.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:39:56.826+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.827606+0000) 2020-02-05T21:39:56.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:39:56.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.827699+0000) 2020-02-05T21:39:56.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:56.827+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.828147+0000) 2020-02-05T21:39:56.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:56.827+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.828565+0000) 2020-02-05T21:39:56.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:39:56.828+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.829603+0000) 2020-02-05T21:39:56.831 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:56.827+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.829187+0000) 2020-02-05T21:39:56.831 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:39:56.828+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.829848+0000) 2020-02-05T21:39:56.832 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:56.828+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.830331+0000) 2020-02-05T21:39:56.832 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:56.829+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.831006+0000) 2020-02-05T21:39:56.833 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:39:56.829+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.831105+0000) 2020-02-05T21:39:56.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:56.833+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:56.834569+0000) 2020-02-05T21:39:56.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:39:56.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:39:57.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:39:57.009+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:57.009908+0000) 2020-02-05T21:39:57.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:39:57.009+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:57.009908+0000) 2020-02-05T21:39:57.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:39:57.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:57.018996+0000) 2020-02-05T21:39:57.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:57.732+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:57.733639+0000) 2020-02-05T21:39:59.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:39:59.118+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:59.118822+0000) 2020-02-05T21:39:59.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:39:59.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:39:59.401649+0000) 2020-02-05T21:40:00.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:00.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:00.009633+0000) 2020-02-05T21:40:00.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:00.243+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:00.243747+0000) 2020-02-05T21:40:00.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:00.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:00.261714+0000) 2020-02-05T21:40:00.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:00.269+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:00.270548+0000) 2020-02-05T21:40:01.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:01.841+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.842658+0000) 2020-02-05T21:40:01.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:01.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.843127+0000) 2020-02-05T21:40:01.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:01.842+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.843692+0000) 2020-02-05T21:40:01.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:01.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.844328+0000) 2020-02-05T21:40:01.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:01.843+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.844488+0000) 2020-02-05T21:40:01.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:01.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.844933+0000) 2020-02-05T21:40:01.847 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:01.844+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.845661+0000) 2020-02-05T21:40:01.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:01.844+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.845774+0000) 2020-02-05T21:40:01.848 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:01.844+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.845849+0000) 2020-02-05T21:40:01.848 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:01.845+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.846652+0000) 2020-02-05T21:40:01.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:01.845+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.846918+0000) 2020-02-05T21:40:01.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:01.846+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.847449+0000) 2020-02-05T21:40:01.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:01.847+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.849024+0000) 2020-02-05T21:40:01.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:01.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92560 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:01.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:01.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:01.854331+0000) 2020-02-05T21:40:02.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:02.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:02.253098+0000) 2020-02-05T21:40:02.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:02.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:02.653290+0000) 2020-02-05T21:40:03.513 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:03.511+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:03.512135+0000) 2020-02-05T21:40:05.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:05.865+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:05.865828+0000) 2020-02-05T21:40:06.861 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:06.858+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.859792+0000) 2020-02-05T21:40:06.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:06.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.860002+0000) 2020-02-05T21:40:06.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:06.861+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.861639+0000) 2020-02-05T21:40:06.863 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:06.861+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.861928+0000) 2020-02-05T21:40:06.863 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:06.861+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.862479+0000) 2020-02-05T21:40:06.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:06.861+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.862744+0000) 2020-02-05T21:40:06.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:06.862+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.863212+0000) 2020-02-05T21:40:06.865 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:06.862+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.863477+0000) 2020-02-05T21:40:06.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:06.862+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.863897+0000) 2020-02-05T21:40:06.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:06.863+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.863858+0000) 2020-02-05T21:40:06.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:06.863+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.864149+0000) 2020-02-05T21:40:06.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:06.863+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.864008+0000) 2020-02-05T21:40:06.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:06.863+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.865031+0000) 2020-02-05T21:40:06.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:06.868+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:06.869639+0000) 2020-02-05T21:40:06.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:06.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:07.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:07.009+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:07.010169+0000) 2020-02-05T21:40:07.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:07.009+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:07.010169+0000) 2020-02-05T21:40:07.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:07.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:07.019209+0000) 2020-02-05T21:40:07.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:07.739+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:07.740276+0000) 2020-02-05T21:40:09.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:09.118+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:09.119078+0000) 2020-02-05T21:40:09.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:09.400+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:09.401898+0000) 2020-02-05T21:40:10.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:10.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:10.009902+0000) 2020-02-05T21:40:10.245 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:10.243+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:10.243980+0000) 2020-02-05T21:40:10.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:10.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:10.261962+0000) 2020-02-05T21:40:10.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:10.269+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:10.270771+0000) 2020-02-05T21:40:11.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:11.875+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.876668+0000) 2020-02-05T21:40:11.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:11.876+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.877388+0000) 2020-02-05T21:40:11.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:11.877+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.877850+0000) 2020-02-05T21:40:11.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:11.877+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.878855+0000) 2020-02-05T21:40:11.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:11.878+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.879414+0000) 2020-02-05T21:40:11.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:11.878+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.879776+0000) 2020-02-05T21:40:11.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:11.879+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.879950+0000) 2020-02-05T21:40:11.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:11.879+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.880392+0000) 2020-02-05T21:40:11.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:11.880+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.880741+0000) 2020-02-05T21:40:11.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:11.879+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.880735+0000) 2020-02-05T21:40:11.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:11.881+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.881882+0000) 2020-02-05T21:40:11.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:11.881+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.881536+0000) 2020-02-05T21:40:11.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:11.881+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.882523+0000) 2020-02-05T21:40:11.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:11.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92639 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:11.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:11.887+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:11.888492+0000) 2020-02-05T21:40:12.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:12.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:12.253313+0000) 2020-02-05T21:40:12.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:12.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:12.653528+0000) 2020-02-05T21:40:13.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:13.511+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:13.512368+0000) 2020-02-05T21:40:15.554 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:40:15.560 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:40:15.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:15.865+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:15.866124+0000) 2020-02-05T21:40:16.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:16.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.893318+0000) 2020-02-05T21:40:16.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:16.893+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.893981+0000) 2020-02-05T21:40:16.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:16.894+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.894905+0000) 2020-02-05T21:40:16.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:16.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.896067+0000) 2020-02-05T21:40:16.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:16.895+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.896581+0000) 2020-02-05T21:40:16.898 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:16.896+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.897246+0000) 2020-02-05T21:40:16.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:16.896+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.897317+0000) 2020-02-05T21:40:16.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:16.896+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.897950+0000) 2020-02-05T21:40:16.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:16.897+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.898284+0000) 2020-02-05T21:40:16.900 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:16.898+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.898446+0000) 2020-02-05T21:40:16.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:16.897+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.898372+0000) 2020-02-05T21:40:16.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:16.898+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.898930+0000) 2020-02-05T21:40:16.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:16.898+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.899879+0000) 2020-02-05T21:40:16.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:16.904+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:16.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:16.904+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:16.905572+0000) 2020-02-05T21:40:17.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:17.010+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:17.010474+0000) 2020-02-05T21:40:17.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:17.010+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:17.010486+0000) 2020-02-05T21:40:17.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:17.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:17.019428+0000) 2020-02-05T21:40:17.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:17.747+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:17.748132+0000) 2020-02-05T21:40:19.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:19.118+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:19.119325+0000) 2020-02-05T21:40:19.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:19.401+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:19.402154+0000) 2020-02-05T21:40:20.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:20.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:20.010164+0000) 2020-02-05T21:40:20.245 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:20.243+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:20.244218+0000) 2020-02-05T21:40:20.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:20.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:20.262246+0000) 2020-02-05T21:40:20.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:20.270+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:20.271048+0000) 2020-02-05T21:40:21.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:21.910+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.911911+0000) 2020-02-05T21:40:21.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:21.911+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.911598+0000) 2020-02-05T21:40:21.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:21.911+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.912643+0000) 2020-02-05T21:40:21.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:21.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.913417+0000) 2020-02-05T21:40:21.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:21.912+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.913484+0000) 2020-02-05T21:40:21.915 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:21.913+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.914003+0000) 2020-02-05T21:40:21.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:21.913+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.914035+0000) 2020-02-05T21:40:21.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:21.914+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.914632+0000) 2020-02-05T21:40:21.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:21.914+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.915639+0000) 2020-02-05T21:40:21.917 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:21.915+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.915611+0000) 2020-02-05T21:40:21.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:21.915+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.916255+0000) 2020-02-05T21:40:21.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:21.915+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.916912+0000) 2020-02-05T21:40:21.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:21.920+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.921145+0000) 2020-02-05T21:40:21.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:21.920+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:21.921375+0000) 2020-02-05T21:40:21.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:21.922+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:22.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:22.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:22.253545+0000) 2020-02-05T21:40:22.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:22.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:22.653753+0000) 2020-02-05T21:40:23.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:23.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:23.512609+0000) 2020-02-05T21:40:25.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:25.866+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:25.866398+0000) 2020-02-05T21:40:26.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:26.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.928472+0000) 2020-02-05T21:40:26.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:26.928+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.929535+0000) 2020-02-05T21:40:26.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:26.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.930444+0000) 2020-02-05T21:40:26.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:26.930+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.930422+0000) 2020-02-05T21:40:26.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:26.930+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.930963+0000) 2020-02-05T21:40:26.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:26.930+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.931434+0000) 2020-02-05T21:40:26.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:26.930+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.931697+0000) 2020-02-05T21:40:26.933 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:26.931+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.931724+0000) 2020-02-05T21:40:26.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:26.931+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.932261+0000) 2020-02-05T21:40:26.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:26.931+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.932298+0000) 2020-02-05T21:40:26.934 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:26.932+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.932747+0000) 2020-02-05T21:40:26.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:26.932+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.933063+0000) 2020-02-05T21:40:26.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:26.936+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.937235+0000) 2020-02-05T21:40:26.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:26.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:26.938721+0000) 2020-02-05T21:40:26.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:26.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:27.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:27.010+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:27.010672+0000) 2020-02-05T21:40:27.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:27.010+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:27.010672+0000) 2020-02-05T21:40:27.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:27.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:27.019642+0000) 2020-02-05T21:40:27.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:27.755+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:27.756833+0000) 2020-02-05T21:40:29.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:29.119+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:29.119598+0000) 2020-02-05T21:40:29.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:29.402+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:29.402412+0000) 2020-02-05T21:40:30.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:30.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:30.010401+0000) 2020-02-05T21:40:30.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:30.244+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:30.244478+0000) 2020-02-05T21:40:30.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:30.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:30.262533+0000) 2020-02-05T21:40:30.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:30.270+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:30.271277+0000) 2020-02-05T21:40:31.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:31.945+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.946381+0000) 2020-02-05T21:40:31.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:31.945+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.946724+0000) 2020-02-05T21:40:31.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:31.946+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.946468+0000) 2020-02-05T21:40:31.950 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:31.947+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.947641+0000) 2020-02-05T21:40:31.951 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:31.947+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.948012+0000) 2020-02-05T21:40:31.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:31.948+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.948584+0000) 2020-02-05T21:40:31.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:31.946+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.947523+0000) 2020-02-05T21:40:31.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:31.948+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.949068+0000) 2020-02-05T21:40:31.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:31.949+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.950333+0000) 2020-02-05T21:40:31.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:31.948+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.949488+0000) 2020-02-05T21:40:31.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:31.948+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.949907+0000) 2020-02-05T21:40:31.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:31.950+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.951371+0000) 2020-02-05T21:40:31.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:31.950+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.951384+0000) 2020-02-05T21:40:31.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:31.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:31.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:31.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:31.957637+0000) 2020-02-05T21:40:32.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:32.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:32.253785+0000) 2020-02-05T21:40:32.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:32.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:32.653962+0000) 2020-02-05T21:40:33.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:33.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:33.512869+0000) 2020-02-05T21:40:35.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:35.866+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:35.866621+0000) 2020-02-05T21:40:36.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:36.963+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.964535+0000) 2020-02-05T21:40:36.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:36.963+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.964835+0000) 2020-02-05T21:40:36.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:36.964+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.964791+0000) 2020-02-05T21:40:36.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:36.964+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.965276+0000) 2020-02-05T21:40:36.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:36.964+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.965435+0000) 2020-02-05T21:40:36.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:36.965+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.966073+0000) 2020-02-05T21:40:36.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:36.965+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.965930+0000) 2020-02-05T21:40:36.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:36.965+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.966117+0000) 2020-02-05T21:40:36.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:36.965+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.967048+0000) 2020-02-05T21:40:36.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:36.966+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.967195+0000) 2020-02-05T21:40:36.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:36.966+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.967067+0000) 2020-02-05T21:40:36.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:36.967+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.968194+0000) 2020-02-05T21:40:36.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:36.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.969012+0000) 2020-02-05T21:40:36.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:36.972+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:36.973472+0000) 2020-02-05T21:40:36.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:36.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:37.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:37.010+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:37.010896+0000) 2020-02-05T21:40:37.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:37.010+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:37.010899+0000) 2020-02-05T21:40:37.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:37.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:37.019867+0000) 2020-02-05T21:40:37.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:37.768+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:37.768284+0000) 2020-02-05T21:40:39.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:39.119+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:39.119935+0000) 2020-02-05T21:40:39.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:39.402+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:39.402640+0000) 2020-02-05T21:40:40.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:40.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:40.010633+0000) 2020-02-05T21:40:40.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:40.244+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:40.244715+0000) 2020-02-05T21:40:40.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:40.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:40.262778+0000) 2020-02-05T21:40:40.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:40.271+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:40.271510+0000) 2020-02-05T21:40:41.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:41.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.981059+0000) 2020-02-05T21:40:41.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:41.980+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.981022+0000) 2020-02-05T21:40:41.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:41.981+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.982069+0000) 2020-02-05T21:40:41.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:41.981+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.982235+0000) 2020-02-05T21:40:41.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:41.982+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.982490+0000) 2020-02-05T21:40:41.984 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:41.982+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.982978+0000) 2020-02-05T21:40:41.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:41.983+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.983388+0000) 2020-02-05T21:40:41.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:41.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.983788+0000) 2020-02-05T21:40:41.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:41.983+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.983893+0000) 2020-02-05T21:40:41.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:41.983+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.984351+0000) 2020-02-05T21:40:41.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:41.984+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.984599+0000) 2020-02-05T21:40:41.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:41.984+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.985255+0000) 2020-02-05T21:40:41.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:41.984+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.985178+0000) 2020-02-05T21:40:41.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:41.992+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:41.992242+0000) 2020-02-05T21:40:41.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:41.992+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92880 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:42.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:42.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:42.254001+0000) 2020-02-05T21:40:42.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:42.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:42.654179+0000) 2020-02-05T21:40:43.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:43.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:43.513131+0000) 2020-02-05T21:40:45.685 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:40:45.691 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:40:45.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:45.865+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:45.866856+0000) 2020-02-05T21:40:47.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:46.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:46.999310+0000) 2020-02-05T21:40:47.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:46.998+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:46.999571+0000) 2020-02-05T21:40:47.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:46.998+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:46.999990+0000) 2020-02-05T21:40:47.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:46.999+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.000539+0000) 2020-02-05T21:40:47.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:46.999+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.000959+0000) 2020-02-05T21:40:47.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:47.000+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.001518+0000) 2020-02-05T21:40:47.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:47.000+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.001494+0000) 2020-02-05T21:40:47.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:47.000+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.002004+0000) 2020-02-05T21:40:47.004 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:47.000+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.002358+0000) 2020-02-05T21:40:47.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:47.001+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.002943+0000) 2020-02-05T21:40:47.005 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:47.002+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.003753+0000) 2020-02-05T21:40:47.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:47.004+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.005631+0000) 2020-02-05T21:40:47.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:47.004+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.006166+0000) 2020-02-05T21:40:47.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:47.007+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.008727+0000) 2020-02-05T21:40:47.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:47.009+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.011098+0000) 2020-02-05T21:40:47.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:47.009+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.011086+0000) 2020-02-05T21:40:47.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:47.011+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:47.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:47.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.020032+0000) 2020-02-05T21:40:47.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:47.779+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:47.780127+0000) 2020-02-05T21:40:49.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:49.119+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:49.120198+0000) 2020-02-05T21:40:49.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:49.401+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:49.402877+0000) 2020-02-05T21:40:50.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:50.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:50.010866+0000) 2020-02-05T21:40:50.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:50.243+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:50.244973+0000) 2020-02-05T21:40:50.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:50.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:50.263035+0000) 2020-02-05T21:40:50.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:50.270+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:50.271739+0000) 2020-02-05T21:40:52.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:52.016+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.017748+0000) 2020-02-05T21:40:52.020 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:52.016+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.018051+0000) 2020-02-05T21:40:52.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:52.018+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.019376+0000) 2020-02-05T21:40:52.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:52.018+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.019864+0000) 2020-02-05T21:40:52.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:52.019+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.020351+0000) 2020-02-05T21:40:52.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:52.019+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.020944+0000) 2020-02-05T21:40:52.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:52.020+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.021461+0000) 2020-02-05T21:40:52.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:52.020+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.021868+0000) 2020-02-05T21:40:52.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:52.021+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.022592+0000) 2020-02-05T21:40:52.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:52.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.022934+0000) 2020-02-05T21:40:52.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:52.022+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.023791+0000) 2020-02-05T21:40:52.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:52.022+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.023902+0000) 2020-02-05T21:40:52.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:52.023+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.025055+0000) 2020-02-05T21:40:52.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:52.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:52.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:52.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.032419+0000) 2020-02-05T21:40:52.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:52.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.254205+0000) 2020-02-05T21:40:52.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:52.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:52.654387+0000) 2020-02-05T21:40:53.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:53.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:53.513375+0000) 2020-02-05T21:40:55.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:55.866+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:55.867123+0000) 2020-02-05T21:40:57.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:57.010+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.011322+0000) 2020-02-05T21:40:57.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:57.010+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.011316+0000) 2020-02-05T21:40:57.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:57.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.020229+0000) 2020-02-05T21:40:57.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:57.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.035218+0000) 2020-02-05T21:40:57.037 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:40:57.034+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.035657+0000) 2020-02-05T21:40:57.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:40:57.035+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.036667+0000) 2020-02-05T21:40:57.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:40:57.036+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.037407+0000) 2020-02-05T21:40:57.039 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:40:57.036+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.037867+0000) 2020-02-05T21:40:57.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:40:57.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.038441+0000) 2020-02-05T21:40:57.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:40:57.038+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.039650+0000) 2020-02-05T21:40:57.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:40:57.038+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.039497+0000) 2020-02-05T21:40:57.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:40:57.038+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.040089+0000) 2020-02-05T21:40:57.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:57.039+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.040072+0000) 2020-02-05T21:40:57.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:40:57.039+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.040467+0000) 2020-02-05T21:40:57.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:40:57.039+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.041371+0000) 2020-02-05T21:40:57.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:57.040+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.041750+0000) 2020-02-05T21:40:57.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:57.045+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.046110+0000) 2020-02-05T21:40:57.049 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:40:57.046+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 92995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:40:57.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:57.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:57.788341+0000) 2020-02-05T21:40:59.122 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:40:59.119+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:59.120490+0000) 2020-02-05T21:40:59.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:40:59.402+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:40:59.403121+0000) 2020-02-05T21:41:00.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:00.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:00.011126+0000) 2020-02-05T21:41:00.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:00.244+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:00.245242+0000) 2020-02-05T21:41:00.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:00.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:00.263273+0000) 2020-02-05T21:41:00.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:00.270+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:00.272031+0000) 2020-02-05T21:41:02.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:02.053+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.054392+0000) 2020-02-05T21:41:02.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:02.054+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.055385+0000) 2020-02-05T21:41:02.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:02.054+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.055525+0000) 2020-02-05T21:41:02.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:02.054+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.055692+0000) 2020-02-05T21:41:02.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:02.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.056208+0000) 2020-02-05T21:41:02.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:02.055+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.056402+0000) 2020-02-05T21:41:02.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:02.055+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.056980+0000) 2020-02-05T21:41:02.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:02.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.057286+0000) 2020-02-05T21:41:02.059 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:02.056+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.057363+0000) 2020-02-05T21:41:02.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:02.055+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.057317+0000) 2020-02-05T21:41:02.060 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:02.056+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.057505+0000) 2020-02-05T21:41:02.060 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:02.056+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.057745+0000) 2020-02-05T21:41:02.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:02.058+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.059234+0000) 2020-02-05T21:41:02.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:02.061+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.062182+0000) 2020-02-05T21:41:02.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:02.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:02.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:02.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.254470+0000) 2020-02-05T21:41:02.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:02.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:02.654614+0000) 2020-02-05T21:41:03.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:03.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:03.513606+0000) 2020-02-05T21:41:05.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:05.866+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:05.867367+0000) 2020-02-05T21:41:07.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:07.010+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.011505+0000) 2020-02-05T21:41:07.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:07.010+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.011593+0000) 2020-02-05T21:41:07.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:07.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.020417+0000) 2020-02-05T21:41:07.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:07.070+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.071208+0000) 2020-02-05T21:41:07.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:07.071+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.072377+0000) 2020-02-05T21:41:07.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:07.072+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.073429+0000) 2020-02-05T21:41:07.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:07.072+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.073196+0000) 2020-02-05T21:41:07.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:07.072+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.073598+0000) 2020-02-05T21:41:07.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:07.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.073713+0000) 2020-02-05T21:41:07.076 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:07.073+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.074719+0000) 2020-02-05T21:41:07.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:07.072+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.074325+0000) 2020-02-05T21:41:07.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:07.073+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.074794+0000) 2020-02-05T21:41:07.077 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:07.074+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.076041+0000) 2020-02-05T21:41:07.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:07.073+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.075324+0000) 2020-02-05T21:41:07.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:07.074+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.076068+0000) 2020-02-05T21:41:07.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:07.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.076555+0000) 2020-02-05T21:41:07.083 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:07.080+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93075 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:07.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:07.081+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.083018+0000) 2020-02-05T21:41:07.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:07.795+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:07.796364+0000) 2020-02-05T21:41:09.122 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:09.120+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:09.120803+0000) 2020-02-05T21:41:09.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:09.402+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:09.403360+0000) 2020-02-05T21:41:10.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:10.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:10.011381+0000) 2020-02-05T21:41:10.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:10.244+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:10.245495+0000) 2020-02-05T21:41:10.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:10.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:10.263467+0000) 2020-02-05T21:41:10.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:10.271+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:10.272315+0000) 2020-02-05T21:41:12.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:12.086+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.087476+0000) 2020-02-05T21:41:12.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:12.085+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.086977+0000) 2020-02-05T21:41:12.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:12.087+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.089297+0000) 2020-02-05T21:41:12.091 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:12.089+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.090162+0000) 2020-02-05T21:41:12.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:12.090+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.091126+0000) 2020-02-05T21:41:12.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:12.090+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.091357+0000) 2020-02-05T21:41:12.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:12.090+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.091667+0000) 2020-02-05T21:41:12.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:12.090+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.092050+0000) 2020-02-05T21:41:12.094 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:12.091+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.092835+0000) 2020-02-05T21:41:12.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:12.092+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.093103+0000) 2020-02-05T21:41:12.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:12.092+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.093999+0000) 2020-02-05T21:41:12.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:12.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.093916+0000) 2020-02-05T21:41:12.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:12.093+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.094555+0000) 2020-02-05T21:41:12.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:12.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.097587+0000) 2020-02-05T21:41:12.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:12.098+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:12.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:12.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.254754+0000) 2020-02-05T21:41:12.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:12.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:12.654851+0000) 2020-02-05T21:41:13.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:13.512+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:13.513867+0000) 2020-02-05T21:41:15.825 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:41:15.831 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:41:15.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:15.866+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:15.867574+0000) 2020-02-05T21:41:17.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:17.011+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.011776+0000) 2020-02-05T21:41:17.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:17.011+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.011867+0000) 2020-02-05T21:41:17.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:17.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.020653+0000) 2020-02-05T21:41:17.107 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:17.105+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.106360+0000) 2020-02-05T21:41:17.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:17.105+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.106875+0000) 2020-02-05T21:41:17.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:17.105+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.107267+0000) 2020-02-05T21:41:17.109 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:17.106+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.107944+0000) 2020-02-05T21:41:17.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:17.107+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.108933+0000) 2020-02-05T21:41:17.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:17.107+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.108941+0000) 2020-02-05T21:41:17.111 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:17.108+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.109354+0000) 2020-02-05T21:41:17.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:17.108+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.109728+0000) 2020-02-05T21:41:17.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:17.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.110199+0000) 2020-02-05T21:41:17.112 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:17.108+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.110202+0000) 2020-02-05T21:41:17.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:17.110+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.111433+0000) 2020-02-05T21:41:17.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:17.110+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.111547+0000) 2020-02-05T21:41:17.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:17.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.113121+0000) 2020-02-05T21:41:17.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:17.113+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.114340+0000) 2020-02-05T21:41:17.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:17.114+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:17.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:17.807+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:17.808291+0000) 2020-02-05T21:41:19.122 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:19.120+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:19.121067+0000) 2020-02-05T21:41:19.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:19.403+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:19.403595+0000) 2020-02-05T21:41:20.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:20.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:20.011609+0000) 2020-02-05T21:41:20.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:20.244+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:20.245680+0000) 2020-02-05T21:41:20.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:20.263+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:20.263751+0000) 2020-02-05T21:41:20.274 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:20.272+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:20.272588+0000) 2020-02-05T21:41:22.122 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:22.119+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.120604+0000) 2020-02-05T21:41:22.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:22.120+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.121794+0000) 2020-02-05T21:41:22.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:22.121+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.122359+0000) 2020-02-05T21:41:22.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:22.121+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.122787+0000) 2020-02-05T21:41:22.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:22.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.123166+0000) 2020-02-05T21:41:22.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:22.123+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.124634+0000) 2020-02-05T21:41:22.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:22.123+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.125244+0000) 2020-02-05T21:41:22.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:22.123+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.124930+0000) 2020-02-05T21:41:22.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:22.125+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.126418+0000) 2020-02-05T21:41:22.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:22.125+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.126734+0000) 2020-02-05T21:41:22.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:22.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.127349+0000) 2020-02-05T21:41:22.129 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:22.126+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.127475+0000) 2020-02-05T21:41:22.129 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:22.126+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.127613+0000) 2020-02-05T21:41:22.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:22.129+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.130939+0000) 2020-02-05T21:41:22.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:22.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:22.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:22.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.254985+0000) 2020-02-05T21:41:22.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:22.654+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:22.655108+0000) 2020-02-05T21:41:23.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:23.513+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:23.514142+0000) 2020-02-05T21:41:25.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:25.867+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:25.867792+0000) 2020-02-05T21:41:27.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:27.011+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.012127+0000) 2020-02-05T21:41:27.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:27.011+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.012118+0000) 2020-02-05T21:41:27.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:27.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.020892+0000) 2020-02-05T21:41:27.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:27.137+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.139234+0000) 2020-02-05T21:41:27.142 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:27.135+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.136763+0000) 2020-02-05T21:41:27.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:27.137+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.138050+0000) 2020-02-05T21:41:27.142 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:27.138+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.139332+0000) 2020-02-05T21:41:27.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:27.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.139271+0000) 2020-02-05T21:41:27.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:27.138+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.140336+0000) 2020-02-05T21:41:27.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:27.139+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.140569+0000) 2020-02-05T21:41:27.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:27.139+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.141237+0000) 2020-02-05T21:41:27.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:27.140+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.141528+0000) 2020-02-05T21:41:27.145 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:27.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.142270+0000) 2020-02-05T21:41:27.145 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:27.141+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.142715+0000) 2020-02-05T21:41:27.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:27.141+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.142965+0000) 2020-02-05T21:41:27.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:27.142+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.143504+0000) 2020-02-05T21:41:27.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:27.147+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.148875+0000) 2020-02-05T21:41:27.151 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:27.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93232 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:27.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:27.815+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:27.816570+0000) 2020-02-05T21:41:29.122 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:29.120+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:29.121313+0000) 2020-02-05T21:41:29.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:29.403+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:29.403850+0000) 2020-02-05T21:41:30.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:30.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:30.011801+0000) 2020-02-05T21:41:30.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:30.245+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:30.245929+0000) 2020-02-05T21:41:30.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:30.263+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:30.264003+0000) 2020-02-05T21:41:30.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:30.272+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:30.272811+0000) 2020-02-05T21:41:32.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:32.156+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.158135+0000) 2020-02-05T21:41:32.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:32.157+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.158536+0000) 2020-02-05T21:41:32.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:32.157+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.159170+0000) 2020-02-05T21:41:32.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:32.158+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.160065+0000) 2020-02-05T21:41:32.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:32.159+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.161096+0000) 2020-02-05T21:41:32.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:32.160+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.162417+0000) 2020-02-05T21:41:32.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:32.155+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.156800+0000) 2020-02-05T21:41:32.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:32.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.157577+0000) 2020-02-05T21:41:32.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:32.161+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.161169+0000) 2020-02-05T21:41:32.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:32.161+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.162217+0000) 2020-02-05T21:41:32.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:32.161+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.162199+0000) 2020-02-05T21:41:32.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:32.161+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.162516+0000) 2020-02-05T21:41:32.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:32.161+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.162918+0000) 2020-02-05T21:41:32.178 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:32.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93274 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:32.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:32.166+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.167891+0000) 2020-02-05T21:41:32.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:32.254+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.255225+0000) 2020-02-05T21:41:32.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:32.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:32.655357+0000) 2020-02-05T21:41:33.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:33.514+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:33.514410+0000) 2020-02-05T21:41:35.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:35.867+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:35.868018+0000) 2020-02-05T21:41:37.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:37.011+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.012384+0000) 2020-02-05T21:41:37.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:37.011+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.012380+0000) 2020-02-05T21:41:37.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:37.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.021131+0000) 2020-02-05T21:41:37.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:37.173+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.174163+0000) 2020-02-05T21:41:37.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:37.173+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.174222+0000) 2020-02-05T21:41:37.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:37.173+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.174679+0000) 2020-02-05T21:41:37.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:37.173+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.174927+0000) 2020-02-05T21:41:37.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:37.174+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.175256+0000) 2020-02-05T21:41:37.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:37.175+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.175693+0000) 2020-02-05T21:41:37.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:37.174+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.175274+0000) 2020-02-05T21:41:37.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:37.175+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.176415+0000) 2020-02-05T21:41:37.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:37.176+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.176923+0000) 2020-02-05T21:41:37.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:37.176+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.177539+0000) 2020-02-05T21:41:37.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:37.176+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.177761+0000) 2020-02-05T21:41:37.179 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:37.177+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.177545+0000) 2020-02-05T21:41:37.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:37.178+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.179059+0000) 2020-02-05T21:41:37.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:37.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:37.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:37.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.184282+0000) 2020-02-05T21:41:37.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:37.826+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:37.827540+0000) 2020-02-05T21:41:39.123 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:39.121+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:39.121579+0000) 2020-02-05T21:41:39.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:39.403+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:39.404135+0000) 2020-02-05T21:41:40.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:40.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:40.012018+0000) 2020-02-05T21:41:40.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:40.245+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:40.246173+0000) 2020-02-05T21:41:40.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:40.263+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:40.264347+0000) 2020-02-05T21:41:40.274 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:40.272+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:40.273031+0000) 2020-02-05T21:41:42.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:42.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.189913+0000) 2020-02-05T21:41:42.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:42.189+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.190107+0000) 2020-02-05T21:41:42.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:42.190+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.191453+0000) 2020-02-05T21:41:42.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:42.190+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.191179+0000) 2020-02-05T21:41:42.193 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:42.191+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.192041+0000) 2020-02-05T21:41:42.194 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:42.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.192474+0000) 2020-02-05T21:41:42.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:42.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.193744+0000) 2020-02-05T21:41:42.195 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:42.192+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.194006+0000) 2020-02-05T21:41:42.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:42.193+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.194703+0000) 2020-02-05T21:41:42.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:42.194+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.194588+0000) 2020-02-05T21:41:42.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:42.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.195424+0000) 2020-02-05T21:41:42.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:42.195+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.196192+0000) 2020-02-05T21:41:42.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:42.196+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.196692+0000) 2020-02-05T21:41:42.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:42.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.204016+0000) 2020-02-05T21:41:42.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:42.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93353 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:42.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:42.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.255503+0000) 2020-02-05T21:41:42.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:42.654+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:42.655585+0000) 2020-02-05T21:41:43.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:43.513+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:43.514646+0000) 2020-02-05T21:41:45.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:45.867+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:45.868296+0000) 2020-02-05T21:41:45.967 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:41:45.984 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:41:47.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:47.012+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.012701+0000) 2020-02-05T21:41:47.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:47.012+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.012681+0000) 2020-02-05T21:41:47.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:47.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.021370+0000) 2020-02-05T21:41:47.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:47.208+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.209266+0000) 2020-02-05T21:41:47.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:47.209+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.210195+0000) 2020-02-05T21:41:47.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:47.210+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.210685+0000) 2020-02-05T21:41:47.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:47.210+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.211304+0000) 2020-02-05T21:41:47.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:47.212+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.213002+0000) 2020-02-05T21:41:47.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:47.212+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.213205+0000) 2020-02-05T21:41:47.216 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:47.206+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.207354+0000) 2020-02-05T21:41:47.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:47.206+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.207991+0000) 2020-02-05T21:41:47.217 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:47.207+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.208776+0000) 2020-02-05T21:41:47.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:47.208+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.209531+0000) 2020-02-05T21:41:47.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:47.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.209964+0000) 2020-02-05T21:41:47.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:47.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.212368+0000) 2020-02-05T21:41:47.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:47.210+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.211804+0000) 2020-02-05T21:41:47.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:47.216+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.217885+0000) 2020-02-05T21:41:47.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:47.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:47.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:47.834+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:47.835486+0000) 2020-02-05T21:41:49.123 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:49.121+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:49.121816+0000) 2020-02-05T21:41:49.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:49.403+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:49.404342+0000) 2020-02-05T21:41:50.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:50.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:50.012294+0000) 2020-02-05T21:41:50.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:50.245+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:50.246365+0000) 2020-02-05T21:41:50.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:50.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:50.264682+0000) 2020-02-05T21:41:50.274 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:50.272+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:50.273288+0000) 2020-02-05T21:41:52.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:52.223+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.224751+0000) 2020-02-05T21:41:52.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:52.224+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.225092+0000) 2020-02-05T21:41:52.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:52.225+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.226518+0000) 2020-02-05T21:41:52.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:52.225+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.227022+0000) 2020-02-05T21:41:52.229 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:52.226+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.227316+0000) 2020-02-05T21:41:52.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:52.227+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.227665+0000) 2020-02-05T21:41:52.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:52.228+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.228712+0000) 2020-02-05T21:41:52.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:52.226+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.227973+0000) 2020-02-05T21:41:52.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:52.228+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.229352+0000) 2020-02-05T21:41:52.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:52.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.229441+0000) 2020-02-05T21:41:52.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:52.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.229883+0000) 2020-02-05T21:41:52.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:52.229+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.230062+0000) 2020-02-05T21:41:52.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:52.229+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.230254+0000) 2020-02-05T21:41:52.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:52.235+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:52.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:52.235+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.236407+0000) 2020-02-05T21:41:52.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:52.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.255740+0000) 2020-02-05T21:41:52.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:52.654+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:52.655823+0000) 2020-02-05T21:41:53.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:53.514+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:53.514893+0000) 2020-02-05T21:41:55.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:55.867+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:55.868564+0000) 2020-02-05T21:41:57.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:57.012+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.012948+0000) 2020-02-05T21:41:57.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:57.012+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.012953+0000) 2020-02-05T21:41:57.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:57.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.021624+0000) 2020-02-05T21:41:57.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:41:57.238+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.240029+0000) 2020-02-05T21:41:57.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:41:57.239+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.240997+0000) 2020-02-05T21:41:57.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:41:57.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.242723+0000) 2020-02-05T21:41:57.244 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:41:57.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.243284+0000) 2020-02-05T21:41:57.245 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:41:57.242+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.243708+0000) 2020-02-05T21:41:57.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:57.244+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.244521+0000) 2020-02-05T21:41:57.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:41:57.244+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.244680+0000) 2020-02-05T21:41:57.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:57.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.244451+0000) 2020-02-05T21:41:57.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:41:57.244+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.245175+0000) 2020-02-05T21:41:57.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:41:57.246+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.246571+0000) 2020-02-05T21:41:57.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:41:57.246+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.246956+0000) 2020-02-05T21:41:57.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:41:57.246+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.247333+0000) 2020-02-05T21:41:57.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:57.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.248182+0000) 2020-02-05T21:41:57.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:57.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.254853+0000) 2020-02-05T21:41:57.256 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:41:57.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:41:57.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:57.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:57.843780+0000) 2020-02-05T21:41:59.123 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:41:59.121+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:59.122063+0000) 2020-02-05T21:41:59.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:41:59.403+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:41:59.404581+0000) 2020-02-05T21:42:00.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:00.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:00.012525+0000) 2020-02-05T21:42:00.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:00.246+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:00.246640+0000) 2020-02-05T21:42:00.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:00.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:00.264925+0000) 2020-02-05T21:42:00.274 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:00.272+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:00.273520+0000) 2020-02-05T21:42:02.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:02.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.255936+0000) 2020-02-05T21:42:02.261 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:02.258+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.259806+0000) 2020-02-05T21:42:02.262 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:02.260+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.261344+0000) 2020-02-05T21:42:02.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:02.260+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.261917+0000) 2020-02-05T21:42:02.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:02.261+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.262366+0000) 2020-02-05T21:42:02.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:02.262+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.262582+0000) 2020-02-05T21:42:02.266 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:02.261+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.262815+0000) 2020-02-05T21:42:02.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:02.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.264314+0000) 2020-02-05T21:42:02.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:02.262+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.263198+0000) 2020-02-05T21:42:02.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:02.265+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.265559+0000) 2020-02-05T21:42:02.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:02.266+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.266482+0000) 2020-02-05T21:42:02.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:02.266+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.266987+0000) 2020-02-05T21:42:02.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:02.266+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.267120+0000) 2020-02-05T21:42:02.269 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:02.266+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.267283+0000) 2020-02-05T21:42:02.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:02.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.269688+0000) 2020-02-05T21:42:02.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:02.271+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:02.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:02.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:02.656047+0000) 2020-02-05T21:42:03.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:03.514+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:03.515123+0000) 2020-02-05T21:42:05.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:05.868+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:05.868794+0000) 2020-02-05T21:42:07.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:07.013+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.013190+0000) 2020-02-05T21:42:07.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:07.013+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.013211+0000) 2020-02-05T21:42:07.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:07.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.021857+0000) 2020-02-05T21:42:07.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:07.277+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.277899+0000) 2020-02-05T21:42:07.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:07.278+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.279647+0000) 2020-02-05T21:42:07.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:07.278+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.279299+0000) 2020-02-05T21:42:07.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:07.279+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.280453+0000) 2020-02-05T21:42:07.282 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:07.279+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.280946+0000) 2020-02-05T21:42:07.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:07.281+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.282210+0000) 2020-02-05T21:42:07.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:07.282+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.282497+0000) 2020-02-05T21:42:07.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:07.282+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.282762+0000) 2020-02-05T21:42:07.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:07.283+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.284349+0000) 2020-02-05T21:42:07.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:07.284+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.284477+0000) 2020-02-05T21:42:07.289 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:07.282+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.283750+0000) 2020-02-05T21:42:07.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:07.283+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.284282+0000) 2020-02-05T21:42:07.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:07.283+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.284650+0000) 2020-02-05T21:42:07.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:07.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.286115+0000) 2020-02-05T21:42:07.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:07.289+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:07.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:07.851+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:07.851722+0000) 2020-02-05T21:42:09.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:09.122+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:09.122363+0000) 2020-02-05T21:42:09.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:09.404+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:09.404811+0000) 2020-02-05T21:42:10.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:10.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:10.012748+0000) 2020-02-05T21:42:10.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:10.246+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:10.246897+0000) 2020-02-05T21:42:10.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:10.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:10.265163+0000) 2020-02-05T21:42:10.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:10.273+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:10.273750+0000) 2020-02-05T21:42:12.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:12.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.256229+0000) 2020-02-05T21:42:12.296 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:12.294+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.295401+0000) 2020-02-05T21:42:12.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:12.295+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.296403+0000) 2020-02-05T21:42:12.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:12.296+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.297591+0000) 2020-02-05T21:42:12.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:12.297+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.298081+0000) 2020-02-05T21:42:12.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:12.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.298263+0000) 2020-02-05T21:42:12.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:12.297+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.298524+0000) 2020-02-05T21:42:12.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:12.297+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.298902+0000) 2020-02-05T21:42:12.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:12.298+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.299467+0000) 2020-02-05T21:42:12.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:12.300+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.300848+0000) 2020-02-05T21:42:12.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:12.301+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.301567+0000) 2020-02-05T21:42:12.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:12.301+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.301920+0000) 2020-02-05T21:42:12.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:12.301+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.302185+0000) 2020-02-05T21:42:12.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:12.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.305651+0000) 2020-02-05T21:42:12.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:12.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.305998+0000) 2020-02-05T21:42:12.308 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:12.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:12.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:12.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:12.656289+0000) 2020-02-05T21:42:13.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:13.515+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:13.515393+0000) 2020-02-05T21:42:15.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:15.868+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:15.869019+0000) 2020-02-05T21:42:16.121 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:42:16.128 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:42:17.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:17.012+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.013482+0000) 2020-02-05T21:42:17.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:17.012+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.013502+0000) 2020-02-05T21:42:17.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:17.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.022124+0000) 2020-02-05T21:42:17.314 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:17.312+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.313149+0000) 2020-02-05T21:42:17.315 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:17.312+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.313796+0000) 2020-02-05T21:42:17.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:17.313+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.315067+0000) 2020-02-05T21:42:17.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:17.314+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.315798+0000) 2020-02-05T21:42:17.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:17.315+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.316288+0000) 2020-02-05T21:42:17.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:17.314+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.315833+0000) 2020-02-05T21:42:17.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:17.315+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.316626+0000) 2020-02-05T21:42:17.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:17.315+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.317050+0000) 2020-02-05T21:42:17.319 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:17.316+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.317814+0000) 2020-02-05T21:42:17.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:17.316+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.318245+0000) 2020-02-05T21:42:17.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:17.317+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.318841+0000) 2020-02-05T21:42:17.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:17.317+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.318561+0000) 2020-02-05T21:42:17.328 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:17.324+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:17.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:17.321+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.322881+0000) 2020-02-05T21:42:17.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:17.322+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.323160+0000) 2020-02-05T21:42:17.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:17.859+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:17.859597+0000) 2020-02-05T21:42:19.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:19.121+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:19.122689+0000) 2020-02-05T21:42:19.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:19.404+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:19.405043+0000) 2020-02-05T21:42:20.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:20.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:20.012909+0000) 2020-02-05T21:42:20.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:20.246+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:20.247130+0000) 2020-02-05T21:42:20.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:20.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:20.265405+0000) 2020-02-05T21:42:20.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:20.273+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:20.273986+0000) 2020-02-05T21:42:22.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:22.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.256485+0000) 2020-02-05T21:42:22.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:22.329+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.330908+0000) 2020-02-05T21:42:22.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:22.331+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.332435+0000) 2020-02-05T21:42:22.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:22.332+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.333147+0000) 2020-02-05T21:42:22.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:22.332+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.333522+0000) 2020-02-05T21:42:22.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:22.333+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.334396+0000) 2020-02-05T21:42:22.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:22.333+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.334295+0000) 2020-02-05T21:42:22.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:22.333+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.334790+0000) 2020-02-05T21:42:22.337 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:22.334+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.335455+0000) 2020-02-05T21:42:22.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:22.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.335808+0000) 2020-02-05T21:42:22.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:22.334+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.336364+0000) 2020-02-05T21:42:22.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:22.336+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.337520+0000) 2020-02-05T21:42:22.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:22.336+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.337990+0000) 2020-02-05T21:42:22.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:22.336+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.338356+0000) 2020-02-05T21:42:22.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:22.340+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.341475+0000) 2020-02-05T21:42:22.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:22.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:22.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:22.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:22.656448+0000) 2020-02-05T21:42:23.517 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:23.515+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:23.515703+0000) 2020-02-05T21:42:25.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:25.869+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:25.869290+0000) 2020-02-05T21:42:27.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:27.012+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.013690+0000) 2020-02-05T21:42:27.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:27.012+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.013715+0000) 2020-02-05T21:42:27.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:27.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.022361+0000) 2020-02-05T21:42:27.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:27.347+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.348792+0000) 2020-02-05T21:42:27.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:27.349+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.349878+0000) 2020-02-05T21:42:27.351 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:27.350+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.350251+0000) 2020-02-05T21:42:27.352 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:27.350+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.350627+0000) 2020-02-05T21:42:27.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:27.349+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.350930+0000) 2020-02-05T21:42:27.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:27.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.351032+0000) 2020-02-05T21:42:27.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:27.350+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.351647+0000) 2020-02-05T21:42:27.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:27.351+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.351704+0000) 2020-02-05T21:42:27.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:27.352+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.352428+0000) 2020-02-05T21:42:27.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:27.351+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.352764+0000) 2020-02-05T21:42:27.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:27.351+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.353203+0000) 2020-02-05T21:42:27.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:27.352+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.353854+0000) 2020-02-05T21:42:27.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:27.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.358024+0000) 2020-02-05T21:42:27.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:27.358+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.358280+0000) 2020-02-05T21:42:27.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:27.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:27.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:27.868+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:27.868322+0000) 2020-02-05T21:42:29.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:29.122+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:29.122960+0000) 2020-02-05T21:42:29.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:29.405+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:29.405333+0000) 2020-02-05T21:42:30.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:30.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:30.013152+0000) 2020-02-05T21:42:30.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:30.246+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:30.247393+0000) 2020-02-05T21:42:30.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:30.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:30.265682+0000) 2020-02-05T21:42:30.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:30.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:30.274217+0000) 2020-02-05T21:42:32.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:32.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.256730+0000) 2020-02-05T21:42:32.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:32.364+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.365700+0000) 2020-02-05T21:42:32.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:32.365+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.366811+0000) 2020-02-05T21:42:32.369 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:32.366+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.367103+0000) 2020-02-05T21:42:32.369 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:32.366+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.367567+0000) 2020-02-05T21:42:32.370 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:32.367+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.368143+0000) 2020-02-05T21:42:32.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:32.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.368710+0000) 2020-02-05T21:42:32.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:32.367+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.369304+0000) 2020-02-05T21:42:32.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:32.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.370046+0000) 2020-02-05T21:42:32.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:32.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.370570+0000) 2020-02-05T21:42:32.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:32.368+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.370069+0000) 2020-02-05T21:42:32.374 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:32.369+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.370612+0000) 2020-02-05T21:42:32.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:32.369+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.370876+0000) 2020-02-05T21:42:32.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:32.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.372009+0000) 2020-02-05T21:42:32.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:32.375+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.376226+0000) 2020-02-05T21:42:32.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:32.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:32.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:32.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:32.656685+0000) 2020-02-05T21:42:33.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:33.514+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:33.515934+0000) 2020-02-05T21:42:35.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:35.868+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:35.869548+0000) 2020-02-05T21:42:37.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:37.013+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.013925+0000) 2020-02-05T21:42:37.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:37.013+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.013998+0000) 2020-02-05T21:42:37.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:37.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.022652+0000) 2020-02-05T21:42:37.386 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:37.384+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.385388+0000) 2020-02-05T21:42:37.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:37.384+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.385975+0000) 2020-02-05T21:42:37.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:37.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.386547+0000) 2020-02-05T21:42:37.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:37.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.386864+0000) 2020-02-05T21:42:37.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:37.386+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.387192+0000) 2020-02-05T21:42:37.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:37.386+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.387656+0000) 2020-02-05T21:42:37.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:37.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.388510+0000) 2020-02-05T21:42:37.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:37.387+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.388577+0000) 2020-02-05T21:42:37.390 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:37.388+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.389440+0000) 2020-02-05T21:42:37.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:37.388+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.389604+0000) 2020-02-05T21:42:37.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:37.388+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.389769+0000) 2020-02-05T21:42:37.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:37.387+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.388908+0000) 2020-02-05T21:42:37.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:37.388+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.389406+0000) 2020-02-05T21:42:37.395 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:37.392+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:37.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:37.394+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.395110+0000) 2020-02-05T21:42:37.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:37.875+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:37.876426+0000) 2020-02-05T21:42:39.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:39.121+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:39.123207+0000) 2020-02-05T21:42:39.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:39.404+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:39.405587+0000) 2020-02-05T21:42:40.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:40.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:40.013428+0000) 2020-02-05T21:42:40.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:40.246+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:40.247705+0000) 2020-02-05T21:42:40.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:40.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:40.265908+0000) 2020-02-05T21:42:40.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:40.273+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:40.274445+0000) 2020-02-05T21:42:42.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:42.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.256986+0000) 2020-02-05T21:42:42.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:42.397+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.398790+0000) 2020-02-05T21:42:42.400 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:42.398+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.399664+0000) 2020-02-05T21:42:42.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:42.399+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.400766+0000) 2020-02-05T21:42:42.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:42.400+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.401549+0000) 2020-02-05T21:42:42.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:42.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.402302+0000) 2020-02-05T21:42:42.404 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:42.401+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.402764+0000) 2020-02-05T21:42:42.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:42.402+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.403638+0000) 2020-02-05T21:42:42.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:42.403+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.404530+0000) 2020-02-05T21:42:42.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:42.403+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.404146+0000) 2020-02-05T21:42:42.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:42.403+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.405165+0000) 2020-02-05T21:42:42.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:42.403+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.405349+0000) 2020-02-05T21:42:42.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:42.404+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.405730+0000) 2020-02-05T21:42:42.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:42.404+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.405327+0000) 2020-02-05T21:42:42.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:42.409+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.410110+0000) 2020-02-05T21:42:42.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:42.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:42.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:42.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:42.656895+0000) 2020-02-05T21:42:43.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:43.515+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:43.516193+0000) 2020-02-05T21:42:45.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:45.868+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:45.869761+0000) 2020-02-05T21:42:46.260 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:42:46.266 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:42:47.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:47.013+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.014249+0000) 2020-02-05T21:42:47.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:47.013+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.014188+0000) 2020-02-05T21:42:47.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:47.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.022889+0000) 2020-02-05T21:42:47.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:47.416+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.417473+0000) 2020-02-05T21:42:47.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:47.417+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.418828+0000) 2020-02-05T21:42:47.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:47.418+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.419834+0000) 2020-02-05T21:42:47.422 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:47.419+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.420738+0000) 2020-02-05T21:42:47.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:47.419+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.421252+0000) 2020-02-05T21:42:47.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:47.420+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.421810+0000) 2020-02-05T21:42:47.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:47.421+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.422281+0000) 2020-02-05T21:42:47.424 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:47.421+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.422735+0000) 2020-02-05T21:42:47.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:47.421+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.423125+0000) 2020-02-05T21:42:47.425 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:47.423+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.424164+0000) 2020-02-05T21:42:47.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:47.422+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.424357+0000) 2020-02-05T21:42:47.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:47.422+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.424409+0000) 2020-02-05T21:42:47.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:47.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93863 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:47.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:47.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.430941+0000) 2020-02-05T21:42:47.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:47.430+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.431483+0000) 2020-02-05T21:42:47.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:47.883+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:47.884560+0000) 2020-02-05T21:42:49.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:49.122+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:49.123498+0000) 2020-02-05T21:42:49.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:49.405+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:49.405828+0000) 2020-02-05T21:42:50.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:50.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:50.013703+0000) 2020-02-05T21:42:50.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:50.247+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:50.247942+0000) 2020-02-05T21:42:50.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:50.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:50.266137+0000) 2020-02-05T21:42:50.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:50.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:50.274746+0000) 2020-02-05T21:42:52.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:52.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.257231+0000) 2020-02-05T21:42:52.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:52.433+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.434720+0000) 2020-02-05T21:42:52.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:52.435+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.436304+0000) 2020-02-05T21:42:52.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:52.435+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.436789+0000) 2020-02-05T21:42:52.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:52.436+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.437652+0000) 2020-02-05T21:42:52.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:52.437+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.438501+0000) 2020-02-05T21:42:52.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:52.435+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.437292+0000) 2020-02-05T21:42:52.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:52.437+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.438883+0000) 2020-02-05T21:42:52.441 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:52.436+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.438312+0000) 2020-02-05T21:42:52.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:52.438+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.439617+0000) 2020-02-05T21:42:52.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:52.439+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.440512+0000) 2020-02-05T21:42:52.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:52.439+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.440712+0000) 2020-02-05T21:42:52.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:52.439+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.440877+0000) 2020-02-05T21:42:52.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:52.444+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.445853+0000) 2020-02-05T21:42:52.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:52.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.446249+0000) 2020-02-05T21:42:52.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:52.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:52.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:52.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:52.657131+0000) 2020-02-05T21:42:53.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:53.515+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:53.516430+0000) 2020-02-05T21:42:55.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:55.868+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:55.870008+0000) 2020-02-05T21:42:57.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:57.013+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.014487+0000) 2020-02-05T21:42:57.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:57.013+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.014482+0000) 2020-02-05T21:42:57.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:57.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.023125+0000) 2020-02-05T21:42:57.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:42:57.449+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.450830+0000) 2020-02-05T21:42:57.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:42:57.450+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.451484+0000) 2020-02-05T21:42:57.454 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:42:57.452+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.453262+0000) 2020-02-05T21:42:57.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:42:57.453+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.454319+0000) 2020-02-05T21:42:57.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:42:57.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.454874+0000) 2020-02-05T21:42:57.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:57.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.456249+0000) 2020-02-05T21:42:57.458 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:42:57.454+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.455928+0000) 2020-02-05T21:42:57.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:42:57.455+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.456494+0000) 2020-02-05T21:42:57.459 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:42:57.455+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.456828+0000) 2020-02-05T21:42:57.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:42:57.456+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.457531+0000) 2020-02-05T21:42:57.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:42:57.456+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.458110+0000) 2020-02-05T21:42:57.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:57.457+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.458449+0000) 2020-02-05T21:42:57.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:57.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.458011+0000) 2020-02-05T21:42:57.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:42:57.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:42:57.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:57.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.463235+0000) 2020-02-05T21:42:57.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:57.891+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:57.893036+0000) 2020-02-05T21:42:59.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:42:59.122+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:59.123695+0000) 2020-02-05T21:42:59.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:42:59.405+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:42:59.406056+0000) 2020-02-05T21:43:00.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:00.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:00.013989+0000) 2020-02-05T21:43:00.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:00.247+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:00.248197+0000) 2020-02-05T21:43:00.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:00.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:00.266387+0000) 2020-02-05T21:43:00.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:00.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:00.274973+0000) 2020-02-05T21:43:02.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:02.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.257490+0000) 2020-02-05T21:43:02.472 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:02.469+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.470609+0000) 2020-02-05T21:43:02.472 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:02.470+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.471281+0000) 2020-02-05T21:43:02.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:02.470+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.471799+0000) 2020-02-05T21:43:02.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:02.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.472294+0000) 2020-02-05T21:43:02.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:02.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.472860+0000) 2020-02-05T21:43:02.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:02.472+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.473309+0000) 2020-02-05T21:43:02.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:02.471+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.473338+0000) 2020-02-05T21:43:02.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:02.472+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.473979+0000) 2020-02-05T21:43:02.475 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:02.472+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.474311+0000) 2020-02-05T21:43:02.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:02.473+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.475208+0000) 2020-02-05T21:43:02.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:02.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.475637+0000) 2020-02-05T21:43:02.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:02.474+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.475978+0000) 2020-02-05T21:43:02.478 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:02.475+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.477019+0000) 2020-02-05T21:43:02.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:02.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 93988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:02.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:02.481+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.482211+0000) 2020-02-05T21:43:02.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:02.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:02.657335+0000) 2020-02-05T21:43:03.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:03.515+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:03.516685+0000) 2020-02-05T21:43:05.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:05.869+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:05.870246+0000) 2020-02-05T21:43:07.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:07.014+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.014742+0000) 2020-02-05T21:43:07.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:07.014+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.014741+0000) 2020-02-05T21:43:07.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:07.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.023351+0000) 2020-02-05T21:43:07.489 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:07.486+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.488015+0000) 2020-02-05T21:43:07.491 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:07.489+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.490066+0000) 2020-02-05T21:43:07.491 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:07.489+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.490667+0000) 2020-02-05T21:43:07.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:07.489+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.490988+0000) 2020-02-05T21:43:07.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:07.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.491859+0000) 2020-02-05T21:43:07.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:07.490+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.491398+0000) 2020-02-05T21:43:07.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:07.491+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.492292+0000) 2020-02-05T21:43:07.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:07.491+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.492649+0000) 2020-02-05T21:43:07.495 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:07.491+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.493004+0000) 2020-02-05T21:43:07.495 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:07.492+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.492956+0000) 2020-02-05T21:43:07.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:07.492+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.493223+0000) 2020-02-05T21:43:07.496 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:07.493+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.493678+0000) 2020-02-05T21:43:07.496 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:07.493+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.493892+0000) 2020-02-05T21:43:07.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:07.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.497969+0000) 2020-02-05T21:43:07.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:07.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:07.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:07.899+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:07.900066+0000) 2020-02-05T21:43:09.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:09.123+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:09.123952+0000) 2020-02-05T21:43:09.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:09.405+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:09.406355+0000) 2020-02-05T21:43:10.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:10.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:10.014268+0000) 2020-02-05T21:43:10.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:10.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:10.248475+0000) 2020-02-05T21:43:10.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:10.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:10.266684+0000) 2020-02-05T21:43:10.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:10.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:10.275211+0000) 2020-02-05T21:43:12.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:12.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.257746+0000) 2020-02-05T21:43:12.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:12.503+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.504979+0000) 2020-02-05T21:43:12.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:12.504+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.505469+0000) 2020-02-05T21:43:12.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:12.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.505824+0000) 2020-02-05T21:43:12.507 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:12.504+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.506014+0000) 2020-02-05T21:43:12.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:12.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.506563+0000) 2020-02-05T21:43:12.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:12.505+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.507029+0000) 2020-02-05T21:43:12.509 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:12.506+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.507141+0000) 2020-02-05T21:43:12.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:12.506+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.507357+0000) 2020-02-05T21:43:12.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:12.507+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.508250+0000) 2020-02-05T21:43:12.510 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:12.507+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.509010+0000) 2020-02-05T21:43:12.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:12.508+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.509183+0000) 2020-02-05T21:43:12.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:12.509+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.509411+0000) 2020-02-05T21:43:12.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:12.509+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.510022+0000) 2020-02-05T21:43:12.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:12.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.515089+0000) 2020-02-05T21:43:12.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:12.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:12.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:12.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:12.657515+0000) 2020-02-05T21:43:13.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:13.516+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:13.516924+0000) 2020-02-05T21:43:15.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:15.869+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:15.870472+0000) 2020-02-05T21:43:16.410 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:43:16.416 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:43:17.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:17.014+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.014997+0000) 2020-02-05T21:43:17.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:17.014+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.014980+0000) 2020-02-05T21:43:17.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:17.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.023613+0000) 2020-02-05T21:43:17.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:17.520+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.521165+0000) 2020-02-05T21:43:17.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:17.522+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.523686+0000) 2020-02-05T21:43:17.525 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:17.523+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.524221+0000) 2020-02-05T21:43:17.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:17.524+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.524675+0000) 2020-02-05T21:43:17.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:17.524+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.525697+0000) 2020-02-05T21:43:17.527 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:17.524+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.525783+0000) 2020-02-05T21:43:17.528 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:17.525+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.526029+0000) 2020-02-05T21:43:17.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:17.525+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.526297+0000) 2020-02-05T21:43:17.528 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:17.526+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.526440+0000) 2020-02-05T21:43:17.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:17.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.526454+0000) 2020-02-05T21:43:17.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:17.525+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.526506+0000) 2020-02-05T21:43:17.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:17.526+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.527142+0000) 2020-02-05T21:43:17.529 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:17.526+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.527392+0000) 2020-02-05T21:43:17.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:17.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.532404+0000) 2020-02-05T21:43:17.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:17.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:17.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:17.907+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:17.908023+0000) 2020-02-05T21:43:19.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:19.123+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:19.124220+0000) 2020-02-05T21:43:19.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:19.406+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:19.406616+0000) 2020-02-05T21:43:20.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:20.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:20.014532+0000) 2020-02-05T21:43:20.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:20.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:20.248713+0000) 2020-02-05T21:43:20.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:20.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:20.266928+0000) 2020-02-05T21:43:20.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:20.275+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:20.275489+0000) 2020-02-05T21:43:22.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:22.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.257950+0000) 2020-02-05T21:43:22.541 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:22.539+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.539919+0000) 2020-02-05T21:43:22.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:22.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.540913+0000) 2020-02-05T21:43:22.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:22.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.541111+0000) 2020-02-05T21:43:22.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:22.541+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.541837+0000) 2020-02-05T21:43:22.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:22.542+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.542756+0000) 2020-02-05T21:43:22.545 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:22.543+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.543620+0000) 2020-02-05T21:43:22.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:22.544+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.544200+0000) 2020-02-05T21:43:22.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:22.542+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.543322+0000) 2020-02-05T21:43:22.546 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:22.544+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.544398+0000) 2020-02-05T21:43:22.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:22.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.544795+0000) 2020-02-05T21:43:22.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:22.544+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.544486+0000) 2020-02-05T21:43:22.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:22.546+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.546268+0000) 2020-02-05T21:43:22.547 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:22.545+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.546076+0000) 2020-02-05T21:43:22.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:22.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.550447+0000) 2020-02-05T21:43:22.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:22.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:22.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:22.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:22.657722+0000) 2020-02-05T21:43:23.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:23.517+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:23.517274+0000) 2020-02-05T21:43:25.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:25.869+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:25.870706+0000) 2020-02-05T21:43:27.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:27.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.015284+0000) 2020-02-05T21:43:27.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:27.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.015291+0000) 2020-02-05T21:43:27.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:27.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.023857+0000) 2020-02-05T21:43:27.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:27.555+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.556624+0000) 2020-02-05T21:43:27.559 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:27.556+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.557976+0000) 2020-02-05T21:43:27.559 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:27.557+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.558356+0000) 2020-02-05T21:43:27.560 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:27.557+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.558716+0000) 2020-02-05T21:43:27.560 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:27.557+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.558750+0000) 2020-02-05T21:43:27.560 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:27.558+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.559153+0000) 2020-02-05T21:43:27.561 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:27.559+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.559564+0000) 2020-02-05T21:43:27.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:27.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.560443+0000) 2020-02-05T21:43:27.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:27.559+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.560656+0000) 2020-02-05T21:43:27.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:27.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.562061+0000) 2020-02-05T21:43:27.563 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:27.561+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.562193+0000) 2020-02-05T21:43:27.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:27.562+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.562517+0000) 2020-02-05T21:43:27.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:27.562+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.562672+0000) 2020-02-05T21:43:27.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:27.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.568405+0000) 2020-02-05T21:43:27.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:27.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94191 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:27.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:27.915+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:27.916392+0000) 2020-02-05T21:43:29.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:29.124+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:29.124506+0000) 2020-02-05T21:43:29.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:29.405+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:29.406893+0000) 2020-02-05T21:43:30.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:30.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:30.014757+0000) 2020-02-05T21:43:30.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:30.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:30.248928+0000) 2020-02-05T21:43:30.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:30.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:30.267178+0000) 2020-02-05T21:43:30.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:30.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:30.275698+0000) 2020-02-05T21:43:32.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:32.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.258217+0000) 2020-02-05T21:43:32.575 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:32.572+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.573762+0000) 2020-02-05T21:43:32.575 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:32.573+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.574404+0000) 2020-02-05T21:43:32.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:32.574+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.575164+0000) 2020-02-05T21:43:32.576 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:32.574+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.575598+0000) 2020-02-05T21:43:32.577 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:32.575+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.576272+0000) 2020-02-05T21:43:32.578 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:32.577+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.577470+0000) 2020-02-05T21:43:32.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:32.577+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.577779+0000) 2020-02-05T21:43:32.579 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:32.577+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.578168+0000) 2020-02-05T21:43:32.580 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:32.578+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.578565+0000) 2020-02-05T21:43:32.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:32.578+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.579161+0000) 2020-02-05T21:43:32.581 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:32.579+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.579483+0000) 2020-02-05T21:43:32.581 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:32.578+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.579930+0000) 2020-02-05T21:43:32.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:32.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.584406+0000) 2020-02-05T21:43:32.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:32.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.584688+0000) 2020-02-05T21:43:32.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:32.585+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:32.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:32.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:32.657908+0000) 2020-02-05T21:43:33.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:33.516+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:33.517510+0000) 2020-02-05T21:43:35.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:35.870+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:35.870932+0000) 2020-02-05T21:43:37.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:37.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.015543+0000) 2020-02-05T21:43:37.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:37.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.015554+0000) 2020-02-05T21:43:37.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:37.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.024115+0000) 2020-02-05T21:43:37.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:37.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.591305+0000) 2020-02-05T21:43:37.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:37.590+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.591472+0000) 2020-02-05T21:43:37.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:37.591+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.591815+0000) 2020-02-05T21:43:37.594 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:37.591+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.592887+0000) 2020-02-05T21:43:37.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:37.591+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593000+0000) 2020-02-05T21:43:37.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:37.592+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593247+0000) 2020-02-05T21:43:37.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:37.593+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593496+0000) 2020-02-05T21:43:37.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:37.592+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593601+0000) 2020-02-05T21:43:37.595 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:37.592+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593971+0000) 2020-02-05T21:43:37.596 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:37.593+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.593720+0000) 2020-02-05T21:43:37.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:37.594+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.595211+0000) 2020-02-05T21:43:37.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:37.594+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.595404+0000) 2020-02-05T21:43:37.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:37.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.595790+0000) 2020-02-05T21:43:37.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:37.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.600031+0000) 2020-02-05T21:43:37.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:37.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:37.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:37.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:37.924677+0000) 2020-02-05T21:43:39.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:39.124+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:39.124767+0000) 2020-02-05T21:43:39.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:39.406+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:39.407156+0000) 2020-02-05T21:43:40.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:40.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:40.014984+0000) 2020-02-05T21:43:40.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:40.249+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:40.249201+0000) 2020-02-05T21:43:40.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:40.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:40.267356+0000) 2020-02-05T21:43:40.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:40.274+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:40.275965+0000) 2020-02-05T21:43:42.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:42.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.258422+0000) 2020-02-05T21:43:42.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:42.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.609169+0000) 2020-02-05T21:43:42.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:42.608+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.609786+0000) 2020-02-05T21:43:42.612 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:42.609+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.610850+0000) 2020-02-05T21:43:42.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:42.610+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.611499+0000) 2020-02-05T21:43:42.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:42.611+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.611718+0000) 2020-02-05T21:43:42.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:42.611+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.612224+0000) 2020-02-05T21:43:42.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:42.611+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.612570+0000) 2020-02-05T21:43:42.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:42.612+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.613036+0000) 2020-02-05T21:43:42.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:42.612+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.613693+0000) 2020-02-05T21:43:42.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:42.613+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.614323+0000) 2020-02-05T21:43:42.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:42.614+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.615311+0000) 2020-02-05T21:43:42.617 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:42.615+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.615540+0000) 2020-02-05T21:43:42.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:42.615+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.616123+0000) 2020-02-05T21:43:42.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:42.618+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.619650+0000) 2020-02-05T21:43:42.623 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:42.621+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94315 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:42.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:42.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:42.658133+0000) 2020-02-05T21:43:43.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:43.516+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:43.517723+0000) 2020-02-05T21:43:45.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:45.870+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:45.871151+0000) 2020-02-05T21:43:46.556 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:43:46.569 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:43:47.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:47.014+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.015812+0000) 2020-02-05T21:43:47.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:47.014+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.015814+0000) 2020-02-05T21:43:47.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:47.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.024302+0000) 2020-02-05T21:43:47.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:47.625+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.626825+0000) 2020-02-05T21:43:47.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:47.626+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.627527+0000) 2020-02-05T21:43:47.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:47.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.628119+0000) 2020-02-05T21:43:47.632 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:47.627+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.628645+0000) 2020-02-05T21:43:47.633 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:47.628+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.630386+0000) 2020-02-05T21:43:47.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:47.629+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.630064+0000) 2020-02-05T21:43:47.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:47.629+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.630536+0000) 2020-02-05T21:43:47.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:47.630+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.631410+0000) 2020-02-05T21:43:47.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:47.630+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.632301+0000) 2020-02-05T21:43:47.635 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:47.631+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.632532+0000) 2020-02-05T21:43:47.635 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:47.631+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.633418+0000) 2020-02-05T21:43:47.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:47.632+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.633560+0000) 2020-02-05T21:43:47.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:47.632+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.633554+0000) 2020-02-05T21:43:47.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:47.636+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.637247+0000) 2020-02-05T21:43:47.641 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:47.638+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:47.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:47.935+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:47.936379+0000) 2020-02-05T21:43:49.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:49.123+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:49.125019+0000) 2020-02-05T21:43:49.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:49.406+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:49.407395+0000) 2020-02-05T21:43:50.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:50.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:50.015218+0000) 2020-02-05T21:43:50.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:50.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:50.249481+0000) 2020-02-05T21:43:50.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:50.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:50.267679+0000) 2020-02-05T21:43:50.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:50.275+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:50.276192+0000) 2020-02-05T21:43:52.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:52.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.258695+0000) 2020-02-05T21:43:52.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:52.645+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.646408+0000) 2020-02-05T21:43:52.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:52.646+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.647378+0000) 2020-02-05T21:43:52.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:52.646+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.647820+0000) 2020-02-05T21:43:52.650 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:52.646+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.648359+0000) 2020-02-05T21:43:52.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:52.647+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.648947+0000) 2020-02-05T21:43:52.650 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:52.646+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.648193+0000) 2020-02-05T21:43:52.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:52.648+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.649533+0000) 2020-02-05T21:43:52.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:52.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.650039+0000) 2020-02-05T21:43:52.652 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:52.649+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.650417+0000) 2020-02-05T21:43:52.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:52.650+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.651759+0000) 2020-02-05T21:43:52.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:52.650+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.651988+0000) 2020-02-05T21:43:52.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:52.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94397 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:52.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:52.649+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.651017+0000) 2020-02-05T21:43:52.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:52.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.651304+0000) 2020-02-05T21:43:52.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:52.655+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.656494+0000) 2020-02-05T21:43:52.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:52.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:52.658277+0000) 2020-02-05T21:43:53.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:53.517+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:53.517923+0000) 2020-02-05T21:43:55.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:55.870+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:55.871370+0000) 2020-02-05T21:43:57.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:57.014+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.016022+0000) 2020-02-05T21:43:57.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:57.014+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.016032+0000) 2020-02-05T21:43:57.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:57.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.024546+0000) 2020-02-05T21:43:57.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:43:57.663+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.664685+0000) 2020-02-05T21:43:57.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:57.663+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.665250+0000) 2020-02-05T21:43:57.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:57.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.666289+0000) 2020-02-05T21:43:57.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:43:57.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.666328+0000) 2020-02-05T21:43:57.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:43:57.665+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.666651+0000) 2020-02-05T21:43:57.669 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:43:57.665+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.667263+0000) 2020-02-05T21:43:57.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:57.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.667020+0000) 2020-02-05T21:43:57.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:43:57.666+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.667557+0000) 2020-02-05T21:43:57.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:43:57.666+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.668092+0000) 2020-02-05T21:43:57.670 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:43:57.667+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.668957+0000) 2020-02-05T21:43:57.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:43:57.667+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.668724+0000) 2020-02-05T21:43:57.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:43:57.668+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.669796+0000) 2020-02-05T21:43:57.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:43:57.668+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.670173+0000) 2020-02-05T21:43:57.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:57.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.674451+0000) 2020-02-05T21:43:57.677 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:43:57.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:43:57.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:57.944+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:57.945397+0000) 2020-02-05T21:43:59.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:43:59.124+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:59.125203+0000) 2020-02-05T21:43:59.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:43:59.407+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:43:59.407622+0000) 2020-02-05T21:44:00.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:00.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:00.015401+0000) 2020-02-05T21:44:00.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:00.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:00.249785+0000) 2020-02-05T21:44:00.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:00.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:00.267891+0000) 2020-02-05T21:44:00.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:00.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:00.276499+0000) 2020-02-05T21:44:02.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:02.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.258937+0000) 2020-02-05T21:44:02.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:02.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.658471+0000) 2020-02-05T21:44:02.682 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:02.679+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.681043+0000) 2020-02-05T21:44:02.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:02.680+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.681983+0000) 2020-02-05T21:44:02.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:02.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.682380+0000) 2020-02-05T21:44:02.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:02.681+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.682940+0000) 2020-02-05T21:44:02.684 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:02.682+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.683267+0000) 2020-02-05T21:44:02.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:02.682+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.683961+0000) 2020-02-05T21:44:02.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:02.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.684854+0000) 2020-02-05T21:44:02.686 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:02.684+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.685390+0000) 2020-02-05T21:44:02.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:02.685+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.686175+0000) 2020-02-05T21:44:02.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:02.684+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.686283+0000) 2020-02-05T21:44:02.688 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:02.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.686565+0000) 2020-02-05T21:44:02.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:02.685+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.687105+0000) 2020-02-05T21:44:02.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:02.686+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.687460+0000) 2020-02-05T21:44:02.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:02.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:02.691171+0000) 2020-02-05T21:44:02.695 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:02.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:03.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:03.517+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:03.518132+0000) 2020-02-05T21:44:05.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:05.871+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:05.871596+0000) 2020-02-05T21:44:07.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:07.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.016276+0000) 2020-02-05T21:44:07.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:07.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.016288+0000) 2020-02-05T21:44:07.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:07.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.024780+0000) 2020-02-05T21:44:07.701 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:07.699+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.700312+0000) 2020-02-05T21:44:07.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:07.699+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.700764+0000) 2020-02-05T21:44:07.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:07.700+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.701454+0000) 2020-02-05T21:44:07.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:07.701+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.702190+0000) 2020-02-05T21:44:07.704 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:07.701+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.702670+0000) 2020-02-05T21:44:07.704 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:07.701+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.702888+0000) 2020-02-05T21:44:07.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:07.702+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.703510+0000) 2020-02-05T21:44:07.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:07.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.703709+0000) 2020-02-05T21:44:07.705 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:07.702+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.703829+0000) 2020-02-05T21:44:07.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:07.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.704341+0000) 2020-02-05T21:44:07.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:07.702+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.704339+0000) 2020-02-05T21:44:07.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:07.703+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.704977+0000) 2020-02-05T21:44:07.712 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:07.709+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:07.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:07.710+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.711618+0000) 2020-02-05T21:44:07.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:07.710+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.711922+0000) 2020-02-05T21:44:07.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:07.953+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:07.953724+0000) 2020-02-05T21:44:09.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:09.124+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:09.125475+0000) 2020-02-05T21:44:09.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:09.407+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:09.407865+0000) 2020-02-05T21:44:10.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:10.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:10.015658+0000) 2020-02-05T21:44:10.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:10.249+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:10.250042+0000) 2020-02-05T21:44:10.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:10.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:10.268123+0000) 2020-02-05T21:44:10.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:10.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:10.276807+0000) 2020-02-05T21:44:12.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:12.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.259205+0000) 2020-02-05T21:44:12.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:12.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.658710+0000) 2020-02-05T21:44:12.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:12.716+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.717186+0000) 2020-02-05T21:44:12.719 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:12.716+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.717683+0000) 2020-02-05T21:44:12.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:12.717+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.718197+0000) 2020-02-05T21:44:12.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:12.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.718867+0000) 2020-02-05T21:44:12.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:12.718+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.719260+0000) 2020-02-05T21:44:12.721 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:12.718+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.719607+0000) 2020-02-05T21:44:12.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:12.718+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.720375+0000) 2020-02-05T21:44:12.722 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:12.719+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.721348+0000) 2020-02-05T21:44:12.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:12.720+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.721809+0000) 2020-02-05T21:44:12.723 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:12.720+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.722323+0000) 2020-02-05T21:44:12.724 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:12.721+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.722668+0000) 2020-02-05T21:44:12.724 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:12.721+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.723060+0000) 2020-02-05T21:44:12.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:12.722+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.723160+0000) 2020-02-05T21:44:12.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:12.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:12.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:12.726+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:12.727328+0000) 2020-02-05T21:44:13.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:13.518+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:13.518376+0000) 2020-02-05T21:44:15.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:15.871+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:15.871893+0000) 2020-02-05T21:44:16.707 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:44:16.714 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:44:17.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:17.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.016545+0000) 2020-02-05T21:44:17.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:17.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.016602+0000) 2020-02-05T21:44:17.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:17.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.025019+0000) 2020-02-05T21:44:17.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:17.734+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.735419+0000) 2020-02-05T21:44:17.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:17.736+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.736302+0000) 2020-02-05T21:44:17.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:17.736+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.736909+0000) 2020-02-05T21:44:17.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:17.734+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.735966+0000) 2020-02-05T21:44:17.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:17.737+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.737723+0000) 2020-02-05T21:44:17.739 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:17.735+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.736941+0000) 2020-02-05T21:44:17.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:17.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.737994+0000) 2020-02-05T21:44:17.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:17.738+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.738425+0000) 2020-02-05T21:44:17.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:17.736+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.738255+0000) 2020-02-05T21:44:17.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:17.738+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.738812+0000) 2020-02-05T21:44:17.741 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:17.739+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.739370+0000) 2020-02-05T21:44:17.741 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:17.737+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.739041+0000) 2020-02-05T21:44:17.742 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:17.738+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.740077+0000) 2020-02-05T21:44:17.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:17.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.745468+0000) 2020-02-05T21:44:17.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:17.744+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:17.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:17.961+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:17.961640+0000) 2020-02-05T21:44:19.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:19.124+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:19.125716+0000) 2020-02-05T21:44:19.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:19.408+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:19.408159+0000) 2020-02-05T21:44:20.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:20.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:20.015923+0000) 2020-02-05T21:44:20.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:20.248+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:20.250281+0000) 2020-02-05T21:44:20.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:20.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:20.268361+0000) 2020-02-05T21:44:20.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:20.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:20.277041+0000) 2020-02-05T21:44:22.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:22.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.259491+0000) 2020-02-05T21:44:22.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:22.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.658939+0000) 2020-02-05T21:44:22.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:22.751+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.753216+0000) 2020-02-05T21:44:22.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:22.752+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.754093+0000) 2020-02-05T21:44:22.764 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:22.753+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.754788+0000) 2020-02-05T21:44:22.764 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:22.756+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.758099+0000) 2020-02-05T21:44:22.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:22.756+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.757931+0000) 2020-02-05T21:44:22.764 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:22.756+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.757726+0000) 2020-02-05T21:44:22.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:22.751+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.752538+0000) 2020-02-05T21:44:22.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:22.754+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.755562+0000) 2020-02-05T21:44:22.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:22.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.761825+0000) 2020-02-05T21:44:22.765 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:22.755+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.756304+0000) 2020-02-05T21:44:22.765 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:22.756+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.757149+0000) 2020-02-05T21:44:22.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:22.756+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.757427+0000) 2020-02-05T21:44:22.766 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:22.757+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.758360+0000) 2020-02-05T21:44:22.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:22.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:22.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:22.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:22.762982+0000) 2020-02-05T21:44:23.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:23.517+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:23.518581+0000) 2020-02-05T21:44:25.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:25.871+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:25.872185+0000) 2020-02-05T21:44:27.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:27.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.016779+0000) 2020-02-05T21:44:27.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:27.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.016838+0000) 2020-02-05T21:44:27.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:27.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.025291+0000) 2020-02-05T21:44:27.770 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:27.767+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.768840+0000) 2020-02-05T21:44:27.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:27.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.769935+0000) 2020-02-05T21:44:27.772 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:27.769+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.771386+0000) 2020-02-05T21:44:27.773 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:27.770+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.772169+0000) 2020-02-05T21:44:27.774 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:27.771+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.772668+0000) 2020-02-05T21:44:27.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:27.772+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.773512+0000) 2020-02-05T21:44:27.775 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:27.772+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.773578+0000) 2020-02-05T21:44:27.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:27.773+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.774209+0000) 2020-02-05T21:44:27.776 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:27.773+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.774527+0000) 2020-02-05T21:44:27.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:27.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.774743+0000) 2020-02-05T21:44:27.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:27.773+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.774955+0000) 2020-02-05T21:44:27.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:27.774+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.776097+0000) 2020-02-05T21:44:27.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:27.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.775657+0000) 2020-02-05T21:44:27.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:27.779+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:27.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:27.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.782956+0000) 2020-02-05T21:44:27.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:27.968+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:27.969540+0000) 2020-02-05T21:44:29.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:29.125+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:29.125982+0000) 2020-02-05T21:44:29.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:29.407+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:29.408368+0000) 2020-02-05T21:44:30.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:30.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:30.016156+0000) 2020-02-05T21:44:30.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:30.249+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:30.250516+0000) 2020-02-05T21:44:30.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:30.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:30.268637+0000) 2020-02-05T21:44:30.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:30.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:30.277316+0000) 2020-02-05T21:44:32.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:32.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.259764+0000) 2020-02-05T21:44:32.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:32.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.659172+0000) 2020-02-05T21:44:32.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:32.783+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.785402+0000) 2020-02-05T21:44:32.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:32.785+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.786258+0000) 2020-02-05T21:44:32.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:32.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.786711+0000) 2020-02-05T21:44:32.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:32.786+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.787419+0000) 2020-02-05T21:44:32.790 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:32.787+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.788822+0000) 2020-02-05T21:44:32.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:32.788+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.789700+0000) 2020-02-05T21:44:32.791 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:32.789+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.790370+0000) 2020-02-05T21:44:32.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:32.790+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.791522+0000) 2020-02-05T21:44:32.793 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:32.790+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.791864+0000) 2020-02-05T21:44:32.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:32.791+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.792211+0000) 2020-02-05T21:44:32.794 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:32.791+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.792561+0000) 2020-02-05T21:44:32.794 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:32.791+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.792621+0000) 2020-02-05T21:44:32.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:32.792+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.793161+0000) 2020-02-05T21:44:32.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:32.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:32.797896+0000) 2020-02-05T21:44:32.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:32.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:33.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:33.518+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:33.518834+0000) 2020-02-05T21:44:35.874 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:35.871+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:35.872421+0000) 2020-02-05T21:44:37.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:37.015+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.016993+0000) 2020-02-05T21:44:37.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:37.015+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.017083+0000) 2020-02-05T21:44:37.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:37.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.025537+0000) 2020-02-05T21:44:37.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:37.802+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.803748+0000) 2020-02-05T21:44:37.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:37.803+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.804541+0000) 2020-02-05T21:44:37.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:37.803+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.805341+0000) 2020-02-05T21:44:37.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:37.804+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.805803+0000) 2020-02-05T21:44:37.809 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:37.805+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.807265+0000) 2020-02-05T21:44:37.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:37.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.808009+0000) 2020-02-05T21:44:37.810 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:37.806+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.808227+0000) 2020-02-05T21:44:37.810 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:37.807+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.808990+0000) 2020-02-05T21:44:37.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:37.808+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.809141+0000) 2020-02-05T21:44:37.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:37.808+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.809594+0000) 2020-02-05T21:44:37.812 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:37.808+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.809776+0000) 2020-02-05T21:44:37.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:37.809+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.810159+0000) 2020-02-05T21:44:37.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:37.809+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.810517+0000) 2020-02-05T21:44:37.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:37.813+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.814381+0000) 2020-02-05T21:44:37.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:37.814+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94769 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:37.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:37.976+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:37.977770+0000) 2020-02-05T21:44:39.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:39.125+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:39.126223+0000) 2020-02-05T21:44:39.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:39.407+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:39.408599+0000) 2020-02-05T21:44:40.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:40.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:40.016424+0000) 2020-02-05T21:44:40.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:40.250+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:40.250706+0000) 2020-02-05T21:44:40.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:40.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:40.268961+0000) 2020-02-05T21:44:40.278 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:40.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:40.277538+0000) 2020-02-05T21:44:42.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:42.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.260019+0000) 2020-02-05T21:44:42.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:42.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.659401+0000) 2020-02-05T21:44:42.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:42.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.821267+0000) 2020-02-05T21:44:42.823 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:42.821+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.821973+0000) 2020-02-05T21:44:42.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:42.822+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.823077+0000) 2020-02-05T21:44:42.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:42.822+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.823387+0000) 2020-02-05T21:44:42.826 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:42.824+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.824555+0000) 2020-02-05T21:44:42.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:42.823+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.824890+0000) 2020-02-05T21:44:42.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:42.825+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.826343+0000) 2020-02-05T21:44:42.828 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:42.825+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.825955+0000) 2020-02-05T21:44:42.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:42.825+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.827029+0000) 2020-02-05T21:44:42.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:42.826+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.827153+0000) 2020-02-05T21:44:42.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:42.826+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.827488+0000) 2020-02-05T21:44:42.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:42.826+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.827735+0000) 2020-02-05T21:44:42.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:42.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.830702+0000) 2020-02-05T21:44:42.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:42.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:42.830929+0000) 2020-02-05T21:44:42.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:42.834+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94811 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:43.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:43.518+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:43.519119+0000) 2020-02-05T21:44:45.874 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:45.871+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:45.872649+0000) 2020-02-05T21:44:46.845 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:44:46.852 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:44:47.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:47.016+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.017256+0000) 2020-02-05T21:44:47.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:47.016+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.017345+0000) 2020-02-05T21:44:47.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:47.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.025775+0000) 2020-02-05T21:44:47.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:47.838+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.839897+0000) 2020-02-05T21:44:47.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:47.839+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.840327+0000) 2020-02-05T21:44:47.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:47.839+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.840997+0000) 2020-02-05T21:44:47.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:47.841+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.841585+0000) 2020-02-05T21:44:47.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:47.841+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.842969+0000) 2020-02-05T21:44:47.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:47.842+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.843364+0000) 2020-02-05T21:44:47.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:47.842+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.843925+0000) 2020-02-05T21:44:47.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:47.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.844231+0000) 2020-02-05T21:44:47.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:47.844+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.844936+0000) 2020-02-05T21:44:47.847 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:47.845+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.845508+0000) 2020-02-05T21:44:47.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:47.846+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.846495+0000) 2020-02-05T21:44:47.848 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:47.845+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.846367+0000) 2020-02-05T21:44:47.848 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:47.846+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.846795+0000) 2020-02-05T21:44:47.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:47.849+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.850527+0000) 2020-02-05T21:44:47.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:47.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:47.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:47.983+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:47.983985+0000) 2020-02-05T21:44:49.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:49.126+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:49.126477+0000) 2020-02-05T21:44:49.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:49.408+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:49.408832+0000) 2020-02-05T21:44:50.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:50.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:50.016664+0000) 2020-02-05T21:44:50.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:50.250+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:50.250928+0000) 2020-02-05T21:44:50.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:50.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:50.269188+0000) 2020-02-05T21:44:50.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:50.276+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:50.277716+0000) 2020-02-05T21:44:52.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:52.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.260265+0000) 2020-02-05T21:44:52.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:52.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.659645+0000) 2020-02-05T21:44:52.859 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:52.857+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.858040+0000) 2020-02-05T21:44:52.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:52.857+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.858759+0000) 2020-02-05T21:44:52.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:52.858+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.859770+0000) 2020-02-05T21:44:52.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:52.859+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.859611+0000) 2020-02-05T21:44:52.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:52.860+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.861119+0000) 2020-02-05T21:44:52.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:52.860+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.861172+0000) 2020-02-05T21:44:52.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:52.860+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.861520+0000) 2020-02-05T21:44:52.863 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:52.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.862566+0000) 2020-02-05T21:44:52.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:52.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.862759+0000) 2020-02-05T21:44:52.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:52.862+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.863295+0000) 2020-02-05T21:44:52.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:52.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.863262+0000) 2020-02-05T21:44:52.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:52.862+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.863782+0000) 2020-02-05T21:44:52.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:52.863+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.864258+0000) 2020-02-05T21:44:52.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:52.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:52.868539+0000) 2020-02-05T21:44:52.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:52.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:53.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:53.518+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:53.519335+0000) 2020-02-05T21:44:55.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:55.872+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:55.872918+0000) 2020-02-05T21:44:57.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:57.017+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.017522+0000) 2020-02-05T21:44:57.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:57.017+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.017608+0000) 2020-02-05T21:44:57.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:57.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.026018+0000) 2020-02-05T21:44:57.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:44:57.874+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.876020+0000) 2020-02-05T21:44:57.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:44:57.875+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.876407+0000) 2020-02-05T21:44:57.879 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:44:57.877+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.877623+0000) 2020-02-05T21:44:57.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:44:57.876+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.877936+0000) 2020-02-05T21:44:57.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:57.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.878395+0000) 2020-02-05T21:44:57.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:57.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.878857+0000) 2020-02-05T21:44:57.880 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:44:57.877+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.878986+0000) 2020-02-05T21:44:57.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:44:57.879+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.880145+0000) 2020-02-05T21:44:57.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:44:57.879+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.880384+0000) 2020-02-05T21:44:57.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:44:57.879+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.880500+0000) 2020-02-05T21:44:57.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:44:57.880+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.880768+0000) 2020-02-05T21:44:57.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:57.881+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.881874+0000) 2020-02-05T21:44:57.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:44:57.881+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.882286+0000) 2020-02-05T21:44:57.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:57.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.886116+0000) 2020-02-05T21:44:57.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:44:57.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:44:57.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:57.992+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:57.992438+0000) 2020-02-05T21:44:59.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:44:59.126+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:59.126746+0000) 2020-02-05T21:44:59.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:44:59.408+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:44:59.409114+0000) 2020-02-05T21:45:00.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:00.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:00.016892+0000) 2020-02-05T21:45:00.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:00.250+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:00.251173+0000) 2020-02-05T21:45:00.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:00.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:00.269466+0000) 2020-02-05T21:45:00.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:00.277+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:00.277939+0000) 2020-02-05T21:45:02.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:02.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.260508+0000) 2020-02-05T21:45:02.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:02.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.659925+0000) 2020-02-05T21:45:02.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:02.891+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.892394+0000) 2020-02-05T21:45:02.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:02.892+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.893530+0000) 2020-02-05T21:45:02.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:02.893+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.894391+0000) 2020-02-05T21:45:02.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:02.895+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.896117+0000) 2020-02-05T21:45:02.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:02.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.896660+0000) 2020-02-05T21:45:02.898 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:02.895+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.896740+0000) 2020-02-05T21:45:02.899 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:02.896+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.896993+0000) 2020-02-05T21:45:02.899 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:02.896+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.897055+0000) 2020-02-05T21:45:02.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:02.897+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.897532+0000) 2020-02-05T21:45:02.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:02.897+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.897935+0000) 2020-02-05T21:45:02.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:02.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.897905+0000) 2020-02-05T21:45:02.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:02.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.898464+0000) 2020-02-05T21:45:02.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:02.898+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.898541+0000) 2020-02-05T21:45:02.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:02.903+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:02.904363+0000) 2020-02-05T21:45:02.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:02.904+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 94977 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:03.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:03.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:03.519541+0000) 2020-02-05T21:45:05.874 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:05.872+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:05.873178+0000) 2020-02-05T21:45:07.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:07.017+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.017829+0000) 2020-02-05T21:45:07.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:07.017+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.017778+0000) 2020-02-05T21:45:07.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:07.026+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.026282+0000) 2020-02-05T21:45:07.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:07.908+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.909557+0000) 2020-02-05T21:45:07.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:07.909+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.910312+0000) 2020-02-05T21:45:07.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:07.911+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.911858+0000) 2020-02-05T21:45:07.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:07.911+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.912330+0000) 2020-02-05T21:45:07.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:07.911+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.912628+0000) 2020-02-05T21:45:07.914 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:07.912+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.913029+0000) 2020-02-05T21:45:07.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:07.912+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.913320+0000) 2020-02-05T21:45:07.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:07.912+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.913607+0000) 2020-02-05T21:45:07.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:07.913+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.914003+0000) 2020-02-05T21:45:07.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:07.913+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.914572+0000) 2020-02-05T21:45:07.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:07.913+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.914591+0000) 2020-02-05T21:45:07.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:07.914+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.915201+0000) 2020-02-05T21:45:07.917 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:07.914+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.915400+0000) 2020-02-05T21:45:07.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:07.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:07.919267+0000) 2020-02-05T21:45:07.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:07.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:08.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:08.004+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:08.004603+0000) 2020-02-05T21:45:09.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:09.126+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:09.126991+0000) 2020-02-05T21:45:09.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:09.409+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:09.409380+0000) 2020-02-05T21:45:10.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:10.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:10.017150+0000) 2020-02-05T21:45:10.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:10.251+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:10.251407+0000) 2020-02-05T21:45:10.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:10.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:10.269696+0000) 2020-02-05T21:45:10.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:10.278+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:10.278210+0000) 2020-02-05T21:45:12.263 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:12.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.260764+0000) 2020-02-05T21:45:12.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:12.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.660162+0000) 2020-02-05T21:45:12.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:12.927+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.927253+0000) 2020-02-05T21:45:12.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:12.928+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.929060+0000) 2020-02-05T21:45:12.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:12.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.929540+0000) 2020-02-05T21:45:12.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:12.930+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.930165+0000) 2020-02-05T21:45:12.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:12.930+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.930572+0000) 2020-02-05T21:45:12.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:12.930+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.931116+0000) 2020-02-05T21:45:12.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:12.931+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.931854+0000) 2020-02-05T21:45:12.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:12.932+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.932276+0000) 2020-02-05T21:45:12.934 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:12.931+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.932162+0000) 2020-02-05T21:45:12.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:12.932+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.932661+0000) 2020-02-05T21:45:12.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:12.932+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.933049+0000) 2020-02-05T21:45:12.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:12.933+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.933667+0000) 2020-02-05T21:45:12.935 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:12.933+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.934055+0000) 2020-02-05T21:45:12.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:12.938+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:12.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:12.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:12.938967+0000) 2020-02-05T21:45:13.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:13.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:13.519752+0000) 2020-02-05T21:45:15.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:15.872+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:15.873418+0000) 2020-02-05T21:45:17.002 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:45:17.009 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:45:17.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:17.017+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.018044+0000) 2020-02-05T21:45:17.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:17.017+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.018052+0000) 2020-02-05T21:45:17.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:17.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.026518+0000) 2020-02-05T21:45:17.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:17.943+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.944723+0000) 2020-02-05T21:45:17.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:17.944+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.945188+0000) 2020-02-05T21:45:17.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:17.945+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.945668+0000) 2020-02-05T21:45:17.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:17.944+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.945996+0000) 2020-02-05T21:45:17.948 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:17.945+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.946343+0000) 2020-02-05T21:45:17.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:17.945+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.946791+0000) 2020-02-05T21:45:17.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:17.946+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.947228+0000) 2020-02-05T21:45:17.949 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:17.946+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.947378+0000) 2020-02-05T21:45:17.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:17.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.947769+0000) 2020-02-05T21:45:17.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:17.947+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.948433+0000) 2020-02-05T21:45:17.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:17.948+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.948423+0000) 2020-02-05T21:45:17.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:17.948+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.949167+0000) 2020-02-05T21:45:17.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:17.949+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.949680+0000) 2020-02-05T21:45:17.955 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:17.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:17.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:17.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:17.955401+0000) 2020-02-05T21:45:18.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:18.011+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:18.012741+0000) 2020-02-05T21:45:19.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:19.127+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:19.127273+0000) 2020-02-05T21:45:19.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:19.408+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:19.409610+0000) 2020-02-05T21:45:20.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:20.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:20.017334+0000) 2020-02-05T21:45:20.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:20.251+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:20.251718+0000) 2020-02-05T21:45:20.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:20.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:20.269943+0000) 2020-02-05T21:45:20.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:20.277+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:20.278488+0000) 2020-02-05T21:45:22.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:22.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.261021+0000) 2020-02-05T21:45:22.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:22.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.660401+0000) 2020-02-05T21:45:22.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:22.958+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.959071+0000) 2020-02-05T21:45:22.961 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:22.959+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.960109+0000) 2020-02-05T21:45:22.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:22.959+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.960654+0000) 2020-02-05T21:45:22.962 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:22.960+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.961235+0000) 2020-02-05T21:45:22.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:22.960+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.962231+0000) 2020-02-05T21:45:22.964 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:22.961+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.962751+0000) 2020-02-05T21:45:22.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:22.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.962966+0000) 2020-02-05T21:45:22.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:22.962+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.963945+0000) 2020-02-05T21:45:22.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:22.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.964019+0000) 2020-02-05T21:45:22.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:22.962+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.964000+0000) 2020-02-05T21:45:22.966 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:22.963+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.964722+0000) 2020-02-05T21:45:22.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:22.964+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.965739+0000) 2020-02-05T21:45:22.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:22.965+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.966870+0000) 2020-02-05T21:45:22.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:22.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:22.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:22.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:22.972090+0000) 2020-02-05T21:45:23.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:23.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:23.519966+0000) 2020-02-05T21:45:25.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:25.872+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:25.873722+0000) 2020-02-05T21:45:27.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:27.016+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.018328+0000) 2020-02-05T21:45:27.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:27.016+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.018355+0000) 2020-02-05T21:45:27.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:27.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.026773+0000) 2020-02-05T21:45:27.977 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:27.973+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.975227+0000) 2020-02-05T21:45:27.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:27.975+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.976507+0000) 2020-02-05T21:45:27.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:27.976+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.977538+0000) 2020-02-05T21:45:27.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:27.976+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.978344+0000) 2020-02-05T21:45:27.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:27.977+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.979006+0000) 2020-02-05T21:45:27.980 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:27.978+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.979508+0000) 2020-02-05T21:45:27.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:27.978+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.979791+0000) 2020-02-05T21:45:27.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:27.979+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.980266+0000) 2020-02-05T21:45:27.982 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:27.978+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.980230+0000) 2020-02-05T21:45:27.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:27.979+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.980726+0000) 2020-02-05T21:45:27.983 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:27.980+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.981796+0000) 2020-02-05T21:45:27.983 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:27.980+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.981418+0000) 2020-02-05T21:45:27.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:27.980+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.981997+0000) 2020-02-05T21:45:27.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:27.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:27.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:27.987+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:27.988589+0000) 2020-02-05T21:45:28.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:28.020+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:28.021317+0000) 2020-02-05T21:45:29.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:29.126+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:29.127534+0000) 2020-02-05T21:45:29.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:29.409+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:29.409853+0000) 2020-02-05T21:45:30.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:30.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:30.017569+0000) 2020-02-05T21:45:30.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:30.250+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:30.251962+0000) 2020-02-05T21:45:30.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:30.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:30.270184+0000) 2020-02-05T21:45:30.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:30.278+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:30.278787+0000) 2020-02-05T21:45:32.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:32.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.261279+0000) 2020-02-05T21:45:32.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:32.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.660632+0000) 2020-02-05T21:45:32.992 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:32.989+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.991026+0000) 2020-02-05T21:45:32.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:32.992+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.993163+0000) 2020-02-05T21:45:32.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:32.991+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.993414+0000) 2020-02-05T21:45:32.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:32.992+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.993778+0000) 2020-02-05T21:45:32.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:32.993+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.994159+0000) 2020-02-05T21:45:32.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:32.993+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.994508+0000) 2020-02-05T21:45:32.996 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:32.993+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.994751+0000) 2020-02-05T21:45:32.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:32.993+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.995379+0000) 2020-02-05T21:45:32.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:32.994+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.995799+0000) 2020-02-05T21:45:32.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:32.995+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.996765+0000) 2020-02-05T21:45:32.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:32.995+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.997034+0000) 2020-02-05T21:45:32.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:32.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:32.997391+0000) 2020-02-05T21:45:33.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:32.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:33.000199+0000) 2020-02-05T21:45:33.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:32.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:33.000541+0000) 2020-02-05T21:45:33.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:33.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:33.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:33.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:33.520175+0000) 2020-02-05T21:45:35.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:35.873+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:35.873945+0000) 2020-02-05T21:45:37.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:37.017+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:37.018645+0000) 2020-02-05T21:45:37.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:37.017+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:37.018705+0000) 2020-02-05T21:45:37.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:37.026+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:37.027042+0000) 2020-02-05T21:45:38.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:38.007+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.008713+0000) 2020-02-05T21:45:38.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:38.008+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.009261+0000) 2020-02-05T21:45:38.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:38.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.009968+0000) 2020-02-05T21:45:38.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:38.009+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.010502+0000) 2020-02-05T21:45:38.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:38.009+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.010977+0000) 2020-02-05T21:45:38.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:38.010+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.011518+0000) 2020-02-05T21:45:38.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:38.009+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.010857+0000) 2020-02-05T21:45:38.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:38.010+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.012118+0000) 2020-02-05T21:45:38.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:38.011+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.013265+0000) 2020-02-05T21:45:38.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:38.011+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.013315+0000) 2020-02-05T21:45:38.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:38.012+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.013700+0000) 2020-02-05T21:45:38.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:38.011+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.012850+0000) 2020-02-05T21:45:38.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:38.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.018019+0000) 2020-02-05T21:45:38.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:38.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.019152+0000) 2020-02-05T21:45:38.022 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:38.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:38.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:38.027+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:38.028109+0000) 2020-02-05T21:45:39.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:39.126+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:39.127814+0000) 2020-02-05T21:45:39.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:39.409+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:39.410121+0000) 2020-02-05T21:45:40.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:40.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:40.017794+0000) 2020-02-05T21:45:40.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:40.250+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:40.252213+0000) 2020-02-05T21:45:40.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:40.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:40.270463+0000) 2020-02-05T21:45:40.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:40.278+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:40.279019+0000) 2020-02-05T21:45:42.263 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:42.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:42.261538+0000) 2020-02-05T21:45:42.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:42.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:42.660862+0000) 2020-02-05T21:45:43.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:43.027+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.029022+0000) 2020-02-05T21:45:43.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:43.028+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.029521+0000) 2020-02-05T21:45:43.031 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:43.028+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.029984+0000) 2020-02-05T21:45:43.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:43.029+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.030574+0000) 2020-02-05T21:45:43.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:43.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.031072+0000) 2020-02-05T21:45:43.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:43.029+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.030986+0000) 2020-02-05T21:45:43.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:43.030+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.031383+0000) 2020-02-05T21:45:43.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:43.030+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.031501+0000) 2020-02-05T21:45:43.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:43.030+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.032220+0000) 2020-02-05T21:45:43.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:43.031+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.032614+0000) 2020-02-05T21:45:43.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:43.030+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.031634+0000) 2020-02-05T21:45:43.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:43.031+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.032206+0000) 2020-02-05T21:45:43.035 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:43.031+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.032813+0000) 2020-02-05T21:45:43.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:43.036+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:43.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:43.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.039450+0000) 2020-02-05T21:45:43.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:43.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:43.520382+0000) 2020-02-05T21:45:45.876 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:45.873+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:45.874170+0000) 2020-02-05T21:45:47.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:47.018+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:47.018878+0000) 2020-02-05T21:45:47.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:47.018+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:47.018935+0000) 2020-02-05T21:45:47.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:47.026+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:47.027332+0000) 2020-02-05T21:45:47.142 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:45:47.148 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:45:48.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:48.036+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.036942+0000) 2020-02-05T21:45:48.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:48.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.043801+0000) 2020-02-05T21:45:48.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:48.043+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.044893+0000) 2020-02-05T21:45:48.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:48.043+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.044737+0000) 2020-02-05T21:45:48.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:48.044+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.045746+0000) 2020-02-05T21:45:48.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:48.045+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046314+0000) 2020-02-05T21:45:48.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:48.045+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046461+0000) 2020-02-05T21:45:48.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:48.045+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046559+0000) 2020-02-05T21:45:48.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:48.045+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046873+0000) 2020-02-05T21:45:48.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:48.045+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046894+0000) 2020-02-05T21:45:48.049 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:48.045+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.047370+0000) 2020-02-05T21:45:48.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:48.046+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.047260+0000) 2020-02-05T21:45:48.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:48.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.046986+0000) 2020-02-05T21:45:48.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:48.047+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.049185+0000) 2020-02-05T21:45:48.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:48.052+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:48.053537+0000) 2020-02-05T21:45:48.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:48.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:49.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:49.127+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:49.128080+0000) 2020-02-05T21:45:49.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:49.409+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:49.410313+0000) 2020-02-05T21:45:50.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:50.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:50.018028+0000) 2020-02-05T21:45:50.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:50.251+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:50.252472+0000) 2020-02-05T21:45:50.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:50.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:50.270723+0000) 2020-02-05T21:45:50.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:50.278+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:50.279287+0000) 2020-02-05T21:45:52.263 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:52.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:52.261730+0000) 2020-02-05T21:45:52.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:52.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:52.661125+0000) 2020-02-05T21:45:53.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:53.059+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.060855+0000) 2020-02-05T21:45:53.063 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:53.061+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.062321+0000) 2020-02-05T21:45:53.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:53.060+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.062303+0000) 2020-02-05T21:45:53.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:53.062+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.063311+0000) 2020-02-05T21:45:53.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:53.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.063461+0000) 2020-02-05T21:45:53.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:53.062+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.064155+0000) 2020-02-05T21:45:53.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:53.063+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.064683+0000) 2020-02-05T21:45:53.066 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:53.063+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.064828+0000) 2020-02-05T21:45:53.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:53.064+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.065191+0000) 2020-02-05T21:45:53.067 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:53.063+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.065399+0000) 2020-02-05T21:45:53.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:53.064+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.066407+0000) 2020-02-05T21:45:53.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:53.065+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.066364+0000) 2020-02-05T21:45:53.068 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:53.065+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.067018+0000) 2020-02-05T21:45:53.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:53.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.069531+0000) 2020-02-05T21:45:53.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:53.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:53.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:53.520+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:53.520601+0000) 2020-02-05T21:45:55.876 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:55.874+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:55.874390+0000) 2020-02-05T21:45:57.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:57.018+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:57.019118+0000) 2020-02-05T21:45:57.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:57.018+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:57.019175+0000) 2020-02-05T21:45:57.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:57.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:57.027565+0000) 2020-02-05T21:45:58.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:58.047+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.048169+0000) 2020-02-05T21:45:58.080 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:45:58.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.079454+0000) 2020-02-05T21:45:58.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:45:58.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.080510+0000) 2020-02-05T21:45:58.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:58.079+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.080886+0000) 2020-02-05T21:45:58.083 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:45:58.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.081036+0000) 2020-02-05T21:45:58.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:45:58.078+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.079938+0000) 2020-02-05T21:45:58.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:58.079+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.081223+0000) 2020-02-05T21:45:58.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:45:58.080+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.081581+0000) 2020-02-05T21:45:58.084 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:45:58.081+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.082261+0000) 2020-02-05T21:45:58.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:45:58.081+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.083066+0000) 2020-02-05T21:45:58.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:58.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.082943+0000) 2020-02-05T21:45:58.085 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:45:58.082+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.083502+0000) 2020-02-05T21:45:58.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:45:58.082+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.084278+0000) 2020-02-05T21:45:58.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:45:58.083+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.084781+0000) 2020-02-05T21:45:58.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:58.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:58.087799+0000) 2020-02-05T21:45:58.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:45:58.089+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:45:59.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:45:59.127+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:59.128378+0000) 2020-02-05T21:45:59.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:45:59.410+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:45:59.410546+0000) 2020-02-05T21:46:00.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:00.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:00.018298+0000) 2020-02-05T21:46:00.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:00.252+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:00.252724+0000) 2020-02-05T21:46:00.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:00.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:00.270960+0000) 2020-02-05T21:46:00.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:00.279+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:00.279512+0000) 2020-02-05T21:46:02.263 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:02.261+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:02.262031+0000) 2020-02-05T21:46:02.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:02.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:02.661380+0000) 2020-02-05T21:46:03.099 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:03.096+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.097552+0000) 2020-02-05T21:46:03.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:03.096+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.097951+0000) 2020-02-05T21:46:03.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:03.097+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.098644+0000) 2020-02-05T21:46:03.100 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:03.097+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.099015+0000) 2020-02-05T21:46:03.101 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:03.097+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.099257+0000) 2020-02-05T21:46:03.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:03.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.099687+0000) 2020-02-05T21:46:03.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:03.099+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.100299+0000) 2020-02-05T21:46:03.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:03.099+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.100527+0000) 2020-02-05T21:46:03.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:03.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.101276+0000) 2020-02-05T21:46:03.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:03.099+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.101299+0000) 2020-02-05T21:46:03.103 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:03.100+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.101843+0000) 2020-02-05T21:46:03.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:03.099+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.100872+0000) 2020-02-05T21:46:03.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:03.100+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.101611+0000) 2020-02-05T21:46:03.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:03.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.107090+0000) 2020-02-05T21:46:03.108 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:03.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:03.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:03.520+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:03.520812+0000) 2020-02-05T21:46:05.876 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:05.874+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:05.874626+0000) 2020-02-05T21:46:07.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:07.018+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:07.019359+0000) 2020-02-05T21:46:07.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:07.018+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:07.019465+0000) 2020-02-05T21:46:07.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:07.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:07.027796+0000) 2020-02-05T21:46:08.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:08.056+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.056186+0000) 2020-02-05T21:46:08.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:08.113+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.113205+0000) 2020-02-05T21:46:08.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:08.112+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.113940+0000) 2020-02-05T21:46:08.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:08.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.114315+0000) 2020-02-05T21:46:08.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:08.114+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.115944+0000) 2020-02-05T21:46:08.117 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:08.116+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.116297+0000) 2020-02-05T21:46:08.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:08.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.116745+0000) 2020-02-05T21:46:08.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:08.115+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.117308+0000) 2020-02-05T21:46:08.119 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:08.117+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.117520+0000) 2020-02-05T21:46:08.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:08.116+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.118247+0000) 2020-02-05T21:46:08.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:08.117+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.119230+0000) 2020-02-05T21:46:08.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:08.117+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.119232+0000) 2020-02-05T21:46:08.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:08.119+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.119761+0000) 2020-02-05T21:46:08.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:08.120+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.120433+0000) 2020-02-05T21:46:08.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:08.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:08.124759+0000) 2020-02-05T21:46:08.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:08.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:09.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:09.128+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:09.128675+0000) 2020-02-05T21:46:09.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:09.410+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:09.410794+0000) 2020-02-05T21:46:10.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:10.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:10.018545+0000) 2020-02-05T21:46:10.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:10.252+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:10.252978+0000) 2020-02-05T21:46:10.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:10.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:10.271192+0000) 2020-02-05T21:46:10.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:10.278+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:10.279738+0000) 2020-02-05T21:46:12.263 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:12.261+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:12.262275+0000) 2020-02-05T21:46:12.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:12.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:12.661590+0000) 2020-02-05T21:46:13.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:13.130+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.132017+0000) 2020-02-05T21:46:13.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:13.130+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.132256+0000) 2020-02-05T21:46:13.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:13.131+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.132652+0000) 2020-02-05T21:46:13.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:13.131+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.133144+0000) 2020-02-05T21:46:13.135 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:13.131+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.132824+0000) 2020-02-05T21:46:13.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:13.132+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.134377+0000) 2020-02-05T21:46:13.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:13.132+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.133479+0000) 2020-02-05T21:46:13.136 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:13.132+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.133950+0000) 2020-02-05T21:46:13.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:13.133+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.134162+0000) 2020-02-05T21:46:13.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:13.134+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.136260+0000) 2020-02-05T21:46:13.138 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:13.135+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.136589+0000) 2020-02-05T21:46:13.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:13.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.141649+0000) 2020-02-05T21:46:13.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:13.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.142010+0000) 2020-02-05T21:46:13.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:13.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.142282+0000) 2020-02-05T21:46:13.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:13.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:13.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:13.519+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:13.521037+0000) 2020-02-05T21:46:15.876 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:15.873+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:15.874817+0000) 2020-02-05T21:46:17.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:17.019+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:17.019666+0000) 2020-02-05T21:46:17.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:17.019+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:17.019705+0000) 2020-02-05T21:46:17.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:17.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:17.028022+0000) 2020-02-05T21:46:17.292 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:46:17.298 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:46:18.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:18.063+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.064407+0000) 2020-02-05T21:46:18.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:18.147+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.148955+0000) 2020-02-05T21:46:18.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:18.148+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.149514+0000) 2020-02-05T21:46:18.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:18.150+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.150630+0000) 2020-02-05T21:46:18.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:18.150+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.151392+0000) 2020-02-05T21:46:18.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:18.150+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.151803+0000) 2020-02-05T21:46:18.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:18.151+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.152507+0000) 2020-02-05T21:46:18.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:18.151+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.152903+0000) 2020-02-05T21:46:18.154 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:18.153+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.153541+0000) 2020-02-05T21:46:18.155 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:18.152+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.153616+0000) 2020-02-05T21:46:18.155 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:18.153+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.153852+0000) 2020-02-05T21:46:18.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:18.153+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.154191+0000) 2020-02-05T21:46:18.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:18.153+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.154309+0000) 2020-02-05T21:46:18.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:18.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.159402+0000) 2020-02-05T21:46:18.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:18.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:18.159701+0000) 2020-02-05T21:46:18.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:18.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95597 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:19.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:19.128+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:19.128938+0000) 2020-02-05T21:46:19.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:19.409+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:19.411024+0000) 2020-02-05T21:46:20.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:20.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:20.018807+0000) 2020-02-05T21:46:20.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:20.252+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:20.253211+0000) 2020-02-05T21:46:20.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:20.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:20.271475+0000) 2020-02-05T21:46:20.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:20.279+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:20.279966+0000) 2020-02-05T21:46:22.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:22.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:22.262532+0000) 2020-02-05T21:46:22.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:22.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:22.661820+0000) 2020-02-05T21:46:23.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:23.164+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.165889+0000) 2020-02-05T21:46:23.168 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:23.165+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.166552+0000) 2020-02-05T21:46:23.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:23.166+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.167470+0000) 2020-02-05T21:46:23.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:23.167+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.168099+0000) 2020-02-05T21:46:23.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:23.167+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.168556+0000) 2020-02-05T21:46:23.170 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:23.168+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.169030+0000) 2020-02-05T21:46:23.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:23.168+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.169322+0000) 2020-02-05T21:46:23.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:23.168+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.170051+0000) 2020-02-05T21:46:23.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:23.170+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.170982+0000) 2020-02-05T21:46:23.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:23.170+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.171387+0000) 2020-02-05T21:46:23.173 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:23.171+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.171501+0000) 2020-02-05T21:46:23.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:23.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.172221+0000) 2020-02-05T21:46:23.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:23.172+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.172471+0000) 2020-02-05T21:46:23.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:23.176+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.177813+0000) 2020-02-05T21:46:23.179 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:23.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:23.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:23.520+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:23.521291+0000) 2020-02-05T21:46:25.876 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:25.874+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:25.875069+0000) 2020-02-05T21:46:27.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:27.019+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:27.019906+0000) 2020-02-05T21:46:27.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:27.019+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:27.019999+0000) 2020-02-05T21:46:27.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:27.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:27.028268+0000) 2020-02-05T21:46:28.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:28.075+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.076362+0000) 2020-02-05T21:46:28.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:28.182+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.183459+0000) 2020-02-05T21:46:28.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:28.183+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.184185+0000) 2020-02-05T21:46:28.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:28.183+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.184745+0000) 2020-02-05T21:46:28.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:28.184+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.185307+0000) 2020-02-05T21:46:28.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:28.184+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.185634+0000) 2020-02-05T21:46:28.187 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:28.185+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.186464+0000) 2020-02-05T21:46:28.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:28.186+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.187428+0000) 2020-02-05T21:46:28.189 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:28.187+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.188259+0000) 2020-02-05T21:46:28.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:28.187+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.187802+0000) 2020-02-05T21:46:28.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:28.187+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.188246+0000) 2020-02-05T21:46:28.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:28.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.188827+0000) 2020-02-05T21:46:28.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:28.189+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.189783+0000) 2020-02-05T21:46:28.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:28.190+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.191615+0000) 2020-02-05T21:46:28.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:28.190+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:28.191863+0000) 2020-02-05T21:46:28.197 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:28.196+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95679 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:29.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:29.128+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:29.129175+0000) 2020-02-05T21:46:29.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:29.410+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:29.411306+0000) 2020-02-05T21:46:30.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:30.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:30.019072+0000) 2020-02-05T21:46:30.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:30.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:30.253486+0000) 2020-02-05T21:46:30.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:30.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:30.271717+0000) 2020-02-05T21:46:30.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:30.279+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:30.280195+0000) 2020-02-05T21:46:32.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:32.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:32.262790+0000) 2020-02-05T21:46:32.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:32.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:32.662058+0000) 2020-02-05T21:46:33.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:33.200+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.201133+0000) 2020-02-05T21:46:33.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:33.202+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.202530+0000) 2020-02-05T21:46:33.204 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:33.201+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.203011+0000) 2020-02-05T21:46:33.205 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:33.203+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.204241+0000) 2020-02-05T21:46:33.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:33.203+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.204428+0000) 2020-02-05T21:46:33.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:33.203+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.204765+0000) 2020-02-05T21:46:33.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:33.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.205870+0000) 2020-02-05T21:46:33.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:33.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.206510+0000) 2020-02-05T21:46:33.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:33.205+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.206340+0000) 2020-02-05T21:46:33.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:33.206+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.207053+0000) 2020-02-05T21:46:33.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:33.206+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.207379+0000) 2020-02-05T21:46:33.209 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:33.207+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.207668+0000) 2020-02-05T21:46:33.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:33.207+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.208244+0000) 2020-02-05T21:46:33.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:33.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.211372+0000) 2020-02-05T21:46:33.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:33.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:33.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:33.520+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:33.521506+0000) 2020-02-05T21:46:35.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:35.874+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:35.875377+0000) 2020-02-05T21:46:37.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:37.019+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:37.020157+0000) 2020-02-05T21:46:37.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:37.019+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:37.020276+0000) 2020-02-05T21:46:37.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:37.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:37.028513+0000) 2020-02-05T21:46:38.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:38.083+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.084428+0000) 2020-02-05T21:46:38.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:38.219+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.220383+0000) 2020-02-05T21:46:38.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:38.220+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.220882+0000) 2020-02-05T21:46:38.223 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:38.220+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.221151+0000) 2020-02-05T21:46:38.223 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:38.220+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.221635+0000) 2020-02-05T21:46:38.224 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:38.221+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.222247+0000) 2020-02-05T21:46:38.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:38.221+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.223036+0000) 2020-02-05T21:46:38.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:38.222+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.223326+0000) 2020-02-05T21:46:38.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:38.223+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.224336+0000) 2020-02-05T21:46:38.226 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:38.223+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.224840+0000) 2020-02-05T21:46:38.227 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:38.223+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.225007+0000) 2020-02-05T21:46:38.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:38.224+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.225233+0000) 2020-02-05T21:46:38.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:38.224+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.225098+0000) 2020-02-05T21:46:38.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:38.225+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.225804+0000) 2020-02-05T21:46:38.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:38.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:38.229321+0000) 2020-02-05T21:46:38.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:38.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:39.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:39.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:39.129396+0000) 2020-02-05T21:46:39.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:39.411+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:39.411530+0000) 2020-02-05T21:46:40.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:40.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:40.019292+0000) 2020-02-05T21:46:40.255 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:40.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:40.253774+0000) 2020-02-05T21:46:40.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:40.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:40.271946+0000) 2020-02-05T21:46:40.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:40.279+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:40.280422+0000) 2020-02-05T21:46:42.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:42.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:42.263033+0000) 2020-02-05T21:46:42.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:42.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:42.662325+0000) 2020-02-05T21:46:43.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:43.237+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.237488+0000) 2020-02-05T21:46:43.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:43.236+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.238058+0000) 2020-02-05T21:46:43.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:43.237+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.238656+0000) 2020-02-05T21:46:43.240 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:43.237+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.238883+0000) 2020-02-05T21:46:43.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:43.238+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.239447+0000) 2020-02-05T21:46:43.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:43.238+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.239510+0000) 2020-02-05T21:46:43.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:43.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.240249+0000) 2020-02-05T21:46:43.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:43.240+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.240774+0000) 2020-02-05T21:46:43.242 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:43.240+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.241070+0000) 2020-02-05T21:46:43.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:43.241+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.242219+0000) 2020-02-05T21:46:43.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:43.242+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.243109+0000) 2020-02-05T21:46:43.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:43.242+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.243193+0000) 2020-02-05T21:46:43.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:43.243+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.243632+0000) 2020-02-05T21:46:43.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:43.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.247572+0000) 2020-02-05T21:46:43.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:43.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95804 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:43.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:43.521+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:43.521717+0000) 2020-02-05T21:46:45.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:45.875+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:45.875624+0000) 2020-02-05T21:46:47.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:47.020+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:47.020414+0000) 2020-02-05T21:46:47.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:47.020+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:47.020552+0000) 2020-02-05T21:46:47.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:47.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:47.028746+0000) 2020-02-05T21:46:47.431 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:46:47.437 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:46:48.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:48.092+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.092798+0000) 2020-02-05T21:46:48.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:48.253+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.254943+0000) 2020-02-05T21:46:48.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:48.255+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.256273+0000) 2020-02-05T21:46:48.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:48.256+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.256692+0000) 2020-02-05T21:46:48.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:48.257+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.257791+0000) 2020-02-05T21:46:48.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:48.257+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.258161+0000) 2020-02-05T21:46:48.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:48.257+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.258512+0000) 2020-02-05T21:46:48.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:48.258+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.258896+0000) 2020-02-05T21:46:48.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:48.257+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.258903+0000) 2020-02-05T21:46:48.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:48.259+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.259405+0000) 2020-02-05T21:46:48.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:48.259+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.259836+0000) 2020-02-05T21:46:48.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:48.259+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.260814+0000) 2020-02-05T21:46:48.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:48.260+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.260779+0000) 2020-02-05T21:46:48.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:48.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.264493+0000) 2020-02-05T21:46:48.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:48.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:48.264823+0000) 2020-02-05T21:46:48.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:48.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:49.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:49.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:49.129675+0000) 2020-02-05T21:46:49.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:49.411+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:49.411768+0000) 2020-02-05T21:46:50.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:50.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:50.019483+0000) 2020-02-05T21:46:50.255 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:50.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:50.254035+0000) 2020-02-05T21:46:50.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:50.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:50.272175+0000) 2020-02-05T21:46:50.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:50.280+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:50.280678+0000) 2020-02-05T21:46:52.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:52.263+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:52.263298+0000) 2020-02-05T21:46:52.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:52.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:52.662577+0000) 2020-02-05T21:46:53.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:53.272+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.273405+0000) 2020-02-05T21:46:53.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:53.273+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.275056+0000) 2020-02-05T21:46:53.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:53.274+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.275412+0000) 2020-02-05T21:46:53.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:53.275+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.275913+0000) 2020-02-05T21:46:53.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:53.274+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.275918+0000) 2020-02-05T21:46:53.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:53.275+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.276354+0000) 2020-02-05T21:46:53.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:53.275+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.276546+0000) 2020-02-05T21:46:53.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:53.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.277018+0000) 2020-02-05T21:46:53.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:53.277+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.277940+0000) 2020-02-05T21:46:53.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:53.276+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.277512+0000) 2020-02-05T21:46:53.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:53.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.278833+0000) 2020-02-05T21:46:53.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:53.278+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.278578+0000) 2020-02-05T21:46:53.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:53.278+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.279257+0000) 2020-02-05T21:46:53.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:53.284+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:53.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:53.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.285786+0000) 2020-02-05T21:46:53.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:53.521+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:53.521951+0000) 2020-02-05T21:46:55.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:55.875+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:55.875850+0000) 2020-02-05T21:46:57.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:57.019+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:57.020718+0000) 2020-02-05T21:46:57.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:57.019+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:57.020779+0000) 2020-02-05T21:46:57.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:57.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:57.028975+0000) 2020-02-05T21:46:58.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:58.101+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.101411+0000) 2020-02-05T21:46:58.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:46:58.288+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.289432+0000) 2020-02-05T21:46:58.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:46:58.289+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.290458+0000) 2020-02-05T21:46:58.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:58.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.291683+0000) 2020-02-05T21:46:58.294 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:46:58.291+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.292780+0000) 2020-02-05T21:46:58.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:46:58.292+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.293422+0000) 2020-02-05T21:46:58.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:46:58.292+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.293463+0000) 2020-02-05T21:46:58.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:58.293+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.294618+0000) 2020-02-05T21:46:58.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:46:58.293+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.294540+0000) 2020-02-05T21:46:58.297 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:46:58.295+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.296150+0000) 2020-02-05T21:46:58.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:46:58.294+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.295657+0000) 2020-02-05T21:46:58.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:58.294+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.296053+0000) 2020-02-05T21:46:58.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:46:58.295+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.296938+0000) 2020-02-05T21:46:58.299 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:46:58.295+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.296680+0000) 2020-02-05T21:46:58.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:58.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:58.301252+0000) 2020-02-05T21:46:58.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:46:58.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:46:59.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:46:59.128+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:59.129909+0000) 2020-02-05T21:46:59.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:46:59.411+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:46:59.412028+0000) 2020-02-05T21:47:00.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:00.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:00.019719+0000) 2020-02-05T21:47:00.255 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:00.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:00.254293+0000) 2020-02-05T21:47:00.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:00.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:00.272456+0000) 2020-02-05T21:47:00.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:00.280+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:00.280899+0000) 2020-02-05T21:47:02.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:02.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:02.263553+0000) 2020-02-05T21:47:02.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:02.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:02.662813+0000) 2020-02-05T21:47:03.310 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:03.309+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.309117+0000) 2020-02-05T21:47:03.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:03.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.309792+0000) 2020-02-05T21:47:03.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:03.310+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.310443+0000) 2020-02-05T21:47:03.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:03.309+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.310886+0000) 2020-02-05T21:47:03.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:03.312+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.312183+0000) 2020-02-05T21:47:03.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:03.310+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.312337+0000) 2020-02-05T21:47:03.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:03.311+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.312844+0000) 2020-02-05T21:47:03.314 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:03.312+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.312859+0000) 2020-02-05T21:47:03.315 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:03.313+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.313580+0000) 2020-02-05T21:47:03.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:03.313+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.313649+0000) 2020-02-05T21:47:03.315 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:03.312+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.314029+0000) 2020-02-05T21:47:03.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:03.313+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.314761+0000) 2020-02-05T21:47:03.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:03.313+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.315160+0000) 2020-02-05T21:47:03.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:03.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.317850+0000) 2020-02-05T21:47:03.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:03.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 95970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:03.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:03.522+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:03.522184+0000) 2020-02-05T21:47:05.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:05.875+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:05.876101+0000) 2020-02-05T21:47:07.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:07.020+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:07.020981+0000) 2020-02-05T21:47:07.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:07.020+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:07.021049+0000) 2020-02-05T21:47:07.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:07.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:07.029205+0000) 2020-02-05T21:47:08.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:08.108+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.109734+0000) 2020-02-05T21:47:08.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:08.325+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.326946+0000) 2020-02-05T21:47:08.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:08.327+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.328308+0000) 2020-02-05T21:47:08.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:08.327+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.328857+0000) 2020-02-05T21:47:08.330 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:08.328+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.329134+0000) 2020-02-05T21:47:08.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:08.328+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.330163+0000) 2020-02-05T21:47:08.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:08.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.330261+0000) 2020-02-05T21:47:08.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:08.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.330370+0000) 2020-02-05T21:47:08.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:08.329+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.330907+0000) 2020-02-05T21:47:08.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:08.329+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.331202+0000) 2020-02-05T21:47:08.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:08.330+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.331768+0000) 2020-02-05T21:47:08.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:08.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.332252+0000) 2020-02-05T21:47:08.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:08.331+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.332557+0000) 2020-02-05T21:47:08.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:08.331+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.333345+0000) 2020-02-05T21:47:08.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:08.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:08.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:08.336+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:08.337533+0000) 2020-02-05T21:47:09.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:09.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:09.130126+0000) 2020-02-05T21:47:09.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:09.411+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:09.412337+0000) 2020-02-05T21:47:10.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:10.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:10.019976+0000) 2020-02-05T21:47:10.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:10.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:10.254553+0000) 2020-02-05T21:47:10.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:10.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:10.272722+0000) 2020-02-05T21:47:10.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:10.280+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:10.281128+0000) 2020-02-05T21:47:12.265 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:12.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:12.263812+0000) 2020-02-05T21:47:12.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:12.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:12.663050+0000) 2020-02-05T21:47:13.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:13.343+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.344628+0000) 2020-02-05T21:47:13.346 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:13.344+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.345530+0000) 2020-02-05T21:47:13.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:13.344+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.345640+0000) 2020-02-05T21:47:13.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:13.345+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.346351+0000) 2020-02-05T21:47:13.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:13.345+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.346936+0000) 2020-02-05T21:47:13.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:13.345+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.347054+0000) 2020-02-05T21:47:13.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:13.346+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.347472+0000) 2020-02-05T21:47:13.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:13.346+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.348395+0000) 2020-02-05T21:47:13.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:13.346+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.348037+0000) 2020-02-05T21:47:13.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:13.348+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.349809+0000) 2020-02-05T21:47:13.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:13.347+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.349244+0000) 2020-02-05T21:47:13.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:13.347+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.349378+0000) 2020-02-05T21:47:13.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:13.348+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.350396+0000) 2020-02-05T21:47:13.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:13.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.354611+0000) 2020-02-05T21:47:13.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:13.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:13.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:13.521+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:13.522336+0000) 2020-02-05T21:47:15.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:15.875+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:15.876330+0000) 2020-02-05T21:47:17.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:17.020+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:17.021244+0000) 2020-02-05T21:47:17.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:17.020+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:17.021303+0000) 2020-02-05T21:47:17.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:17.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:17.029445+0000) 2020-02-05T21:47:17.571 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:47:17.577 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:47:18.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:18.116+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.117749+0000) 2020-02-05T21:47:18.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:18.361+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.362100+0000) 2020-02-05T21:47:18.364 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:18.362+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.363120+0000) 2020-02-05T21:47:18.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:18.361+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.363066+0000) 2020-02-05T21:47:18.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:18.363+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.364739+0000) 2020-02-05T21:47:18.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:18.364+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.365371+0000) 2020-02-05T21:47:18.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:18.364+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.365564+0000) 2020-02-05T21:47:18.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:18.365+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.366488+0000) 2020-02-05T21:47:18.368 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:18.364+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.365956+0000) 2020-02-05T21:47:18.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:18.365+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.366656+0000) 2020-02-05T21:47:18.369 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:18.366+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.367357+0000) 2020-02-05T21:47:18.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:18.366+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.368024+0000) 2020-02-05T21:47:18.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:18.366+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.368323+0000) 2020-02-05T21:47:18.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:18.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.368463+0000) 2020-02-05T21:47:18.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:18.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:18.374459+0000) 2020-02-05T21:47:18.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:18.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96094 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:19.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:19.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:19.130331+0000) 2020-02-05T21:47:19.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:19.411+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:19.412558+0000) 2020-02-05T21:47:20.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:20.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:20.020202+0000) 2020-02-05T21:47:20.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:20.253+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:20.254781+0000) 2020-02-05T21:47:20.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:20.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:20.272964+0000) 2020-02-05T21:47:20.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:20.280+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:20.281372+0000) 2020-02-05T21:47:22.265 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:22.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:22.264043+0000) 2020-02-05T21:47:22.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:22.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:22.663320+0000) 2020-02-05T21:47:23.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:23.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.380395+0000) 2020-02-05T21:47:23.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:23.380+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.381941+0000) 2020-02-05T21:47:23.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:23.381+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.382580+0000) 2020-02-05T21:47:23.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:23.381+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.383270+0000) 2020-02-05T21:47:23.385 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:23.383+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.384138+0000) 2020-02-05T21:47:23.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:23.383+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.384406+0000) 2020-02-05T21:47:23.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:23.383+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.385089+0000) 2020-02-05T21:47:23.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:23.384+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.385604+0000) 2020-02-05T21:47:23.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:23.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.386126+0000) 2020-02-05T21:47:23.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:23.385+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.386444+0000) 2020-02-05T21:47:23.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:23.385+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.386719+0000) 2020-02-05T21:47:23.389 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:23.385+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.387093+0000) 2020-02-05T21:47:23.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:23.385+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.387357+0000) 2020-02-05T21:47:23.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:23.389+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.390212+0000) 2020-02-05T21:47:23.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:23.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96136 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:23.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:23.521+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:23.522542+0000) 2020-02-05T21:47:25.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:25.875+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:25.876560+0000) 2020-02-05T21:47:27.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:27.020+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:27.021485+0000) 2020-02-05T21:47:27.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:27.020+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:27.021535+0000) 2020-02-05T21:47:27.097 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:27.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:27.029677+0000) 2020-02-05T21:47:28.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:28.128+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.129478+0000) 2020-02-05T21:47:28.400 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:28.396+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.397578+0000) 2020-02-05T21:47:28.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:28.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.400062+0000) 2020-02-05T21:47:28.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:28.399+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.400504+0000) 2020-02-05T21:47:28.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:28.399+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.400797+0000) 2020-02-05T21:47:28.402 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:28.400+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.401058+0000) 2020-02-05T21:47:28.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:28.399+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.400818+0000) 2020-02-05T21:47:28.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:28.399+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.401339+0000) 2020-02-05T21:47:28.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:28.400+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.401809+0000) 2020-02-05T21:47:28.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:28.401+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.402598+0000) 2020-02-05T21:47:28.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:28.401+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.402761+0000) 2020-02-05T21:47:28.405 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:28.401+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.403008+0000) 2020-02-05T21:47:28.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:28.402+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.403168+0000) 2020-02-05T21:47:28.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:28.403+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.404512+0000) 2020-02-05T21:47:28.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:28.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:28.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:28.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:28.409605+0000) 2020-02-05T21:47:29.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:29.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:29.130649+0000) 2020-02-05T21:47:29.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:29.412+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:29.412835+0000) 2020-02-05T21:47:30.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:30.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:30.020431+0000) 2020-02-05T21:47:30.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:30.254+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:30.255014+0000) 2020-02-05T21:47:30.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:30.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:30.273181+0000) 2020-02-05T21:47:30.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:30.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:30.281590+0000) 2020-02-05T21:47:32.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:32.262+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:32.264293+0000) 2020-02-05T21:47:32.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:32.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:32.663557+0000) 2020-02-05T21:47:33.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:33.414+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.416015+0000) 2020-02-05T21:47:33.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:33.414+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.415928+0000) 2020-02-05T21:47:33.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:33.415+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.416724+0000) 2020-02-05T21:47:33.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:33.415+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.416935+0000) 2020-02-05T21:47:33.419 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:33.416+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.417508+0000) 2020-02-05T21:47:33.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:33.416+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.418281+0000) 2020-02-05T21:47:33.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:33.417+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.418586+0000) 2020-02-05T21:47:33.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:33.418+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.419654+0000) 2020-02-05T21:47:33.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:33.418+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.419990+0000) 2020-02-05T21:47:33.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:33.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.420411+0000) 2020-02-05T21:47:33.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:33.419+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.421377+0000) 2020-02-05T21:47:33.423 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:33.420+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.421760+0000) 2020-02-05T21:47:33.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:33.420+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.421880+0000) 2020-02-05T21:47:33.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:33.425+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.426599+0000) 2020-02-05T21:47:33.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:33.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:33.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:33.521+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:33.522860+0000) 2020-02-05T21:47:35.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:35.876+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:35.876789+0000) 2020-02-05T21:47:37.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:37.021+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:37.021738+0000) 2020-02-05T21:47:37.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:37.021+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:37.021804+0000) 2020-02-05T21:47:37.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:37.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:37.029907+0000) 2020-02-05T21:47:38.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:38.136+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.136984+0000) 2020-02-05T21:47:38.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:38.432+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.433929+0000) 2020-02-05T21:47:38.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:38.433+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.434745+0000) 2020-02-05T21:47:38.436 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:38.434+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.435531+0000) 2020-02-05T21:47:38.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:38.434+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.435597+0000) 2020-02-05T21:47:38.438 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:38.435+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.436537+0000) 2020-02-05T21:47:38.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:38.436+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.437153+0000) 2020-02-05T21:47:38.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:38.436+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.438111+0000) 2020-02-05T21:47:38.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:38.437+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.438292+0000) 2020-02-05T21:47:38.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:38.437+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.438749+0000) 2020-02-05T21:47:38.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:38.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.439034+0000) 2020-02-05T21:47:38.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:38.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.439431+0000) 2020-02-05T21:47:38.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:38.438+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.440057+0000) 2020-02-05T21:47:38.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:38.438+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.440084+0000) 2020-02-05T21:47:38.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:38.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:38.442945+0000) 2020-02-05T21:47:38.447 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:38.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:39.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:39.129+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:39.130869+0000) 2020-02-05T21:47:39.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:39.412+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:39.413065+0000) 2020-02-05T21:47:40.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:40.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:40.020684+0000) 2020-02-05T21:47:40.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:40.254+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:40.255270+0000) 2020-02-05T21:47:40.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:40.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:40.273475+0000) 2020-02-05T21:47:40.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:40.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:40.281814+0000) 2020-02-05T21:47:42.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:42.263+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:42.264559+0000) 2020-02-05T21:47:42.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:42.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:42.663837+0000) 2020-02-05T21:47:43.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:43.450+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.451944+0000) 2020-02-05T21:47:43.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:43.450+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.451728+0000) 2020-02-05T21:47:43.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:43.451+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.453028+0000) 2020-02-05T21:47:43.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:43.452+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.453944+0000) 2020-02-05T21:47:43.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:43.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.454245+0000) 2020-02-05T21:47:43.457 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:43.454+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.455451+0000) 2020-02-05T21:47:43.457 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:43.454+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.455540+0000) 2020-02-05T21:47:43.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:43.454+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.455968+0000) 2020-02-05T21:47:43.458 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:43.455+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.456386+0000) 2020-02-05T21:47:43.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:43.455+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.456724+0000) 2020-02-05T21:47:43.459 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:43.456+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.457422+0000) 2020-02-05T21:47:43.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:43.456+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.458299+0000) 2020-02-05T21:47:43.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:43.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.458310+0000) 2020-02-05T21:47:43.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:43.461+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.462965+0000) 2020-02-05T21:47:43.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:43.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:43.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:43.522+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:43.523096+0000) 2020-02-05T21:47:45.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:45.876+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:45.877033+0000) 2020-02-05T21:47:47.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:47.021+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:47.022037+0000) 2020-02-05T21:47:47.026 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:47.021+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:47.022032+0000) 2020-02-05T21:47:47.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:47.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:47.030137+0000) 2020-02-05T21:47:47.708 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:47:47.714 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:47:48.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:48.144+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.145248+0000) 2020-02-05T21:47:48.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:48.470+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.471544+0000) 2020-02-05T21:47:48.473 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:48.469+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.471220+0000) 2020-02-05T21:47:48.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:48.470+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.472211+0000) 2020-02-05T21:47:48.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:48.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.472308+0000) 2020-02-05T21:47:48.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:48.471+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.472387+0000) 2020-02-05T21:47:48.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:48.472+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.473143+0000) 2020-02-05T21:47:48.475 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:48.471+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.472866+0000) 2020-02-05T21:47:48.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:48.472+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.473676+0000) 2020-02-05T21:47:48.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:48.473+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.474738+0000) 2020-02-05T21:47:48.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:48.473+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.475165+0000) 2020-02-05T21:47:48.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:48.474+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.475656+0000) 2020-02-05T21:47:48.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:48.474+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.475903+0000) 2020-02-05T21:47:48.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:48.478+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.479387+0000) 2020-02-05T21:47:48.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:48.478+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:48.479644+0000) 2020-02-05T21:47:48.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:48.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:49.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:49.130+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:49.131114+0000) 2020-02-05T21:47:49.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:49.413+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:49.413337+0000) 2020-02-05T21:47:50.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:50.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:50.020919+0000) 2020-02-05T21:47:50.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:50.255+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:50.255528+0000) 2020-02-05T21:47:50.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:50.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:50.273710+0000) 2020-02-05T21:47:50.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:50.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:50.282039+0000) 2020-02-05T21:47:52.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:52.264+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:52.264815+0000) 2020-02-05T21:47:52.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:52.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:52.664120+0000) 2020-02-05T21:47:53.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:53.485+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.485504+0000) 2020-02-05T21:47:53.489 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:53.487+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.488340+0000) 2020-02-05T21:47:53.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:53.487+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.488812+0000) 2020-02-05T21:47:53.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:53.487+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.488896+0000) 2020-02-05T21:47:53.491 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:53.489+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.489570+0000) 2020-02-05T21:47:53.491 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:53.488+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.489177+0000) 2020-02-05T21:47:53.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:53.489+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.489917+0000) 2020-02-05T21:47:53.494 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:53.489+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.490572+0000) 2020-02-05T21:47:53.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:53.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.491354+0000) 2020-02-05T21:47:53.495 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:53.491+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.492853+0000) 2020-02-05T21:47:53.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:53.491+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.491963+0000) 2020-02-05T21:47:53.495 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:53.491+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.492280+0000) 2020-02-05T21:47:53.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:53.491+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.492399+0000) 2020-02-05T21:47:53.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:53.493+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.494265+0000) 2020-02-05T21:47:53.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:53.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96385 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:53.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:53.523+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:53.523308+0000) 2020-02-05T21:47:55.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:55.877+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:55.877309+0000) 2020-02-05T21:47:57.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:57.021+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:57.022289+0000) 2020-02-05T21:47:57.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:57.021+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:57.022288+0000) 2020-02-05T21:47:57.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:57.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:57.030356+0000) 2020-02-05T21:47:58.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:58.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.154023+0000) 2020-02-05T21:47:58.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:58.502+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.502897+0000) 2020-02-05T21:47:58.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:58.504+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.504207+0000) 2020-02-05T21:47:58.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:47:58.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.504601+0000) 2020-02-05T21:47:58.506 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:47:58.504+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.504950+0000) 2020-02-05T21:47:58.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:47:58.505+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.505395+0000) 2020-02-05T21:47:58.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:58.505+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.505829+0000) 2020-02-05T21:47:58.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:47:58.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.506219+0000) 2020-02-05T21:47:58.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:47:58.507+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.507493+0000) 2020-02-05T21:47:58.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:47:58.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.508117+0000) 2020-02-05T21:47:58.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:47:58.508+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.508681+0000) 2020-02-05T21:47:58.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:47:58.508+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.509207+0000) 2020-02-05T21:47:58.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:47:58.509+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.509697+0000) 2020-02-05T21:47:58.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:47:58.509+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.509760+0000) 2020-02-05T21:47:58.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:58.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:58.514203+0000) 2020-02-05T21:47:58.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:47:58.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:47:59.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:47:59.130+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:59.131361+0000) 2020-02-05T21:47:59.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:47:59.412+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:47:59.413544+0000) 2020-02-05T21:48:00.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:00.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:00.021169+0000) 2020-02-05T21:48:00.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:00.255+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:00.255778+0000) 2020-02-05T21:48:00.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:00.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:00.273937+0000) 2020-02-05T21:48:00.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:00.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:00.282291+0000) 2020-02-05T21:48:02.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:02.264+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:02.265073+0000) 2020-02-05T21:48:02.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:02.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:02.664349+0000) 2020-02-05T21:48:03.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:03.521+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.522223+0000) 2020-02-05T21:48:03.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:03.521+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.522386+0000) 2020-02-05T21:48:03.524 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:03.521+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.522669+0000) 2020-02-05T21:48:03.524 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:03.522+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.523264+0000) 2020-02-05T21:48:03.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:03.523+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.523760+0000) 2020-02-05T21:48:03.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:03.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.524370+0000) 2020-02-05T21:48:03.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:03.524+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.524864+0000) 2020-02-05T21:48:03.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:03.523+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.524831+0000) 2020-02-05T21:48:03.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:03.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.525237+0000) 2020-02-05T21:48:03.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:03.526+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.526902+0000) 2020-02-05T21:48:03.528 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:03.526+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.526785+0000) 2020-02-05T21:48:03.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:03.526+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.527073+0000) 2020-02-05T21:48:03.529 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:03.527+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.528070+0000) 2020-02-05T21:48:03.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:03.529+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:03.530795+0000) 2020-02-05T21:48:03.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:03.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:05.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:05.876+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:05.877540+0000) 2020-02-05T21:48:07.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:07.022+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:07.022545+0000) 2020-02-05T21:48:07.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:07.022+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:07.022542+0000) 2020-02-05T21:48:07.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:07.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:07.030580+0000) 2020-02-05T21:48:08.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:08.161+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.162128+0000) 2020-02-05T21:48:08.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:08.538+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.539900+0000) 2020-02-05T21:48:08.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:08.539+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.540477+0000) 2020-02-05T21:48:08.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:08.539+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.540988+0000) 2020-02-05T21:48:08.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:08.541+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.541565+0000) 2020-02-05T21:48:08.544 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:08.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.542517+0000) 2020-02-05T21:48:08.544 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:08.542+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.542629+0000) 2020-02-05T21:48:08.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:08.544+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.544612+0000) 2020-02-05T21:48:08.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:08.543+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.544834+0000) 2020-02-05T21:48:08.546 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:08.544+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.545315+0000) 2020-02-05T21:48:08.547 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:08.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.545185+0000) 2020-02-05T21:48:08.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:08.545+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.545854+0000) 2020-02-05T21:48:08.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:08.545+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.545644+0000) 2020-02-05T21:48:08.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:08.546+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.547969+0000) 2020-02-05T21:48:08.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:08.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:08.549058+0000) 2020-02-05T21:48:08.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:08.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:09.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:09.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:09.131573+0000) 2020-02-05T21:48:09.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:09.412+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:09.413757+0000) 2020-02-05T21:48:10.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:10.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:10.021397+0000) 2020-02-05T21:48:10.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:10.255+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:10.256010+0000) 2020-02-05T21:48:10.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:10.273+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:10.274007+0000) 2020-02-05T21:48:10.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:10.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:10.274180+0000) 2020-02-05T21:48:10.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:10.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:10.282535+0000) 2020-02-05T21:48:12.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:12.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:12.265406+0000) 2020-02-05T21:48:12.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:12.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:12.664615+0000) 2020-02-05T21:48:13.559 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:13.556+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.557733+0000) 2020-02-05T21:48:13.559 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:13.557+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.558495+0000) 2020-02-05T21:48:13.560 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:13.558+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.559297+0000) 2020-02-05T21:48:13.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:13.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.560278+0000) 2020-02-05T21:48:13.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:13.559+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.560566+0000) 2020-02-05T21:48:13.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:13.560+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.561236+0000) 2020-02-05T21:48:13.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:13.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.561956+0000) 2020-02-05T21:48:13.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:13.562+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.562424+0000) 2020-02-05T21:48:13.564 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:13.562+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.563381+0000) 2020-02-05T21:48:13.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:13.562+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.563859+0000) 2020-02-05T21:48:13.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:13.563+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.564184+0000) 2020-02-05T21:48:13.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:13.564+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.564604+0000) 2020-02-05T21:48:13.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:13.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.567828+0000) 2020-02-05T21:48:13.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:13.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:13.568123+0000) 2020-02-05T21:48:13.572 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:13.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:15.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:15.877+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:15.877772+0000) 2020-02-05T21:48:17.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:17.022+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:17.022810+0000) 2020-02-05T21:48:17.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:17.022+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:17.022810+0000) 2020-02-05T21:48:17.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:17.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:17.030834+0000) 2020-02-05T21:48:17.846 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:48:17.852 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:48:18.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:18.169+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.170032+0000) 2020-02-05T21:48:18.578 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:18.575+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.576608+0000) 2020-02-05T21:48:18.579 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:18.576+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.577406+0000) 2020-02-05T21:48:18.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:18.576+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.577872+0000) 2020-02-05T21:48:18.581 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:18.579+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.580082+0000) 2020-02-05T21:48:18.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:18.579+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.580284+0000) 2020-02-05T21:48:18.582 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:18.580+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.581139+0000) 2020-02-05T21:48:18.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:18.580+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.581299+0000) 2020-02-05T21:48:18.583 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:18.581+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.582307+0000) 2020-02-05T21:48:18.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:18.581+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.582369+0000) 2020-02-05T21:48:18.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:18.581+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.582915+0000) 2020-02-05T21:48:18.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:18.582+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.583283+0000) 2020-02-05T21:48:18.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:18.583+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.583640+0000) 2020-02-05T21:48:18.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:18.586+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.587733+0000) 2020-02-05T21:48:18.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:18.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:18.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:18.588+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:18.589194+0000) 2020-02-05T21:48:19.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:19.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:19.131807+0000) 2020-02-05T21:48:19.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:19.413+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:19.413963+0000) 2020-02-05T21:48:20.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:20.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:20.021634+0000) 2020-02-05T21:48:20.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:20.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:20.256263+0000) 2020-02-05T21:48:20.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:20.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:20.274419+0000) 2020-02-05T21:48:20.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:20.273+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:20.274275+0000) 2020-02-05T21:48:20.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:20.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:20.282775+0000) 2020-02-05T21:48:22.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:22.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:22.265717+0000) 2020-02-05T21:48:22.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:22.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:22.664844+0000) 2020-02-05T21:48:23.597 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:23.595+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.595562+0000) 2020-02-05T21:48:23.597 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:23.594+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.595976+0000) 2020-02-05T21:48:23.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:23.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.595843+0000) 2020-02-05T21:48:23.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:23.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.596501+0000) 2020-02-05T21:48:23.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:23.596+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.597264+0000) 2020-02-05T21:48:23.599 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:23.596+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.597367+0000) 2020-02-05T21:48:23.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:23.597+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.598351+0000) 2020-02-05T21:48:23.600 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:23.597+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.598831+0000) 2020-02-05T21:48:23.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:23.598+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.599236+0000) 2020-02-05T21:48:23.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:23.599+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.600168+0000) 2020-02-05T21:48:23.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:23.600+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.600956+0000) 2020-02-05T21:48:23.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:23.600+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.601242+0000) 2020-02-05T21:48:23.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:23.599+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.601061+0000) 2020-02-05T21:48:23.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:23.605+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:23.606471+0000) 2020-02-05T21:48:23.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:23.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96636 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:25.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:25.877+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:25.877995+0000) 2020-02-05T21:48:27.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:27.022+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:27.023070+0000) 2020-02-05T21:48:27.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:27.022+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:27.023076+0000) 2020-02-05T21:48:27.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:27.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:27.031127+0000) 2020-02-05T21:48:28.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:28.177+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.178220+0000) 2020-02-05T21:48:28.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:28.611+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.613057+0000) 2020-02-05T21:48:28.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:28.613+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.614314+0000) 2020-02-05T21:48:28.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:28.614+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.615368+0000) 2020-02-05T21:48:28.617 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:28.616+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.616525+0000) 2020-02-05T21:48:28.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:28.616+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.617088+0000) 2020-02-05T21:48:28.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:28.616+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.617375+0000) 2020-02-05T21:48:28.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:28.617+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.618190+0000) 2020-02-05T21:48:28.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:28.617+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.618462+0000) 2020-02-05T21:48:28.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:28.618+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.619163+0000) 2020-02-05T21:48:28.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:28.618+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.619238+0000) 2020-02-05T21:48:28.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:28.619+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.619579+0000) 2020-02-05T21:48:28.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:28.619+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.619811+0000) 2020-02-05T21:48:28.621 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:28.618+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.619933+0000) 2020-02-05T21:48:28.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:28.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:28.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:28.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:28.626422+0000) 2020-02-05T21:48:29.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:29.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:29.132030+0000) 2020-02-05T21:48:29.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:29.413+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:29.414180+0000) 2020-02-05T21:48:30.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:30.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:30.021878+0000) 2020-02-05T21:48:30.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:30.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:30.256510+0000) 2020-02-05T21:48:30.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:30.273+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:30.274470+0000) 2020-02-05T21:48:30.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:30.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:30.274699+0000) 2020-02-05T21:48:30.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:30.281+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:30.283027+0000) 2020-02-05T21:48:32.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:32.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:32.265978+0000) 2020-02-05T21:48:32.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:32.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:32.665072+0000) 2020-02-05T21:48:33.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:33.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.632342+0000) 2020-02-05T21:48:33.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:33.632+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.633423+0000) 2020-02-05T21:48:33.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:33.632+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.632790+0000) 2020-02-05T21:48:33.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:33.633+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.634068+0000) 2020-02-05T21:48:33.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:33.633+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.634243+0000) 2020-02-05T21:48:33.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:33.633+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.634732+0000) 2020-02-05T21:48:33.636 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:33.635+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.635467+0000) 2020-02-05T21:48:33.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:33.635+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.635820+0000) 2020-02-05T21:48:33.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:33.635+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.636824+0000) 2020-02-05T21:48:33.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:33.636+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.636479+0000) 2020-02-05T21:48:33.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:33.636+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.637306+0000) 2020-02-05T21:48:33.639 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:33.637+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.638015+0000) 2020-02-05T21:48:33.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:33.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.638661+0000) 2020-02-05T21:48:33.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:33.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:33.642201+0000) 2020-02-05T21:48:33.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:33.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:35.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:35.877+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:35.878269+0000) 2020-02-05T21:48:37.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:37.021+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:37.023311+0000) 2020-02-05T21:48:37.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:37.021+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:37.023331+0000) 2020-02-05T21:48:37.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:37.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:37.031364+0000) 2020-02-05T21:48:38.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:38.185+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.186403+0000) 2020-02-05T21:48:38.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:38.649+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.650383+0000) 2020-02-05T21:48:38.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:38.649+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.650815+0000) 2020-02-05T21:48:38.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:38.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.651989+0000) 2020-02-05T21:48:38.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:38.651+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.652902+0000) 2020-02-05T21:48:38.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:38.652+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.654241+0000) 2020-02-05T21:48:38.656 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:38.653+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.654644+0000) 2020-02-05T21:48:38.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:38.653+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.654599+0000) 2020-02-05T21:48:38.656 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:38.654+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.655140+0000) 2020-02-05T21:48:38.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:38.654+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.656046+0000) 2020-02-05T21:48:38.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:38.654+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.655487+0000) 2020-02-05T21:48:38.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:38.654+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.656236+0000) 2020-02-05T21:48:38.658 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:38.655+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.656768+0000) 2020-02-05T21:48:38.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:38.655+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.656891+0000) 2020-02-05T21:48:38.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:38.659+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:38.660429+0000) 2020-02-05T21:48:38.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:38.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:39.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:39.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:39.132245+0000) 2020-02-05T21:48:39.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:39.413+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:39.414332+0000) 2020-02-05T21:48:40.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:40.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:40.022142+0000) 2020-02-05T21:48:40.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:40.255+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:40.256740+0000) 2020-02-05T21:48:40.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:40.274+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:40.274744+0000) 2020-02-05T21:48:40.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:40.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:40.274945+0000) 2020-02-05T21:48:40.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:40.283+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:40.283337+0000) 2020-02-05T21:48:42.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:42.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:42.266237+0000) 2020-02-05T21:48:42.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:42.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:42.665329+0000) 2020-02-05T21:48:43.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:43.667+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.668866+0000) 2020-02-05T21:48:43.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:43.668+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.670013+0000) 2020-02-05T21:48:43.672 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:43.668+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.670071+0000) 2020-02-05T21:48:43.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:43.669+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.670579+0000) 2020-02-05T21:48:43.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:43.670+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.671499+0000) 2020-02-05T21:48:43.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:43.670+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.671950+0000) 2020-02-05T21:48:43.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:43.671+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.672351+0000) 2020-02-05T21:48:43.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:43.671+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.673096+0000) 2020-02-05T21:48:43.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:43.671+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.673259+0000) 2020-02-05T21:48:43.679 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:43.672+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.674120+0000) 2020-02-05T21:48:43.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:43.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.674653+0000) 2020-02-05T21:48:43.679 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:43.673+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.674847+0000) 2020-02-05T21:48:43.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:43.673+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.674119+0000) 2020-02-05T21:48:43.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:43.677+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:43.678851+0000) 2020-02-05T21:48:43.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:43.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:45.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:45.878+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:45.878502+0000) 2020-02-05T21:48:47.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:47.022+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:47.023568+0000) 2020-02-05T21:48:47.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:47.022+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:47.023564+0000) 2020-02-05T21:48:47.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:47.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:47.031628+0000) 2020-02-05T21:48:47.982 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:48:47.989 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:48:48.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:48.194+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.194391+0000) 2020-02-05T21:48:48.688 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:48.685+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.686368+0000) 2020-02-05T21:48:48.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:48.686+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.687150+0000) 2020-02-05T21:48:48.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:48.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.687931+0000) 2020-02-05T21:48:48.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:48.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.688528+0000) 2020-02-05T21:48:48.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:48.687+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.689037+0000) 2020-02-05T21:48:48.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:48.688+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.689611+0000) 2020-02-05T21:48:48.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:48.689+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.691076+0000) 2020-02-05T21:48:48.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:48.690+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.691439+0000) 2020-02-05T21:48:48.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:48.690+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.691899+0000) 2020-02-05T21:48:48.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:48.691+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.692510+0000) 2020-02-05T21:48:48.694 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:48.691+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.692559+0000) 2020-02-05T21:48:48.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:48.691+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.693120+0000) 2020-02-05T21:48:48.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:48.691+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.693295+0000) 2020-02-05T21:48:48.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:48.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:48.696683+0000) 2020-02-05T21:48:48.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:48.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:49.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:49.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:49.132493+0000) 2020-02-05T21:48:49.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:49.414+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:49.414542+0000) 2020-02-05T21:48:50.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:50.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:50.022382+0000) 2020-02-05T21:48:50.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:50.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:50.256973+0000) 2020-02-05T21:48:50.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:50.274+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:50.275038+0000) 2020-02-05T21:48:50.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:50.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:50.275188+0000) 2020-02-05T21:48:50.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:50.283+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:50.283569+0000) 2020-02-05T21:48:52.269 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:52.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:52.266473+0000) 2020-02-05T21:48:52.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:52.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:52.665576+0000) 2020-02-05T21:48:53.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:53.704+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.705766+0000) 2020-02-05T21:48:53.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:53.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.706475+0000) 2020-02-05T21:48:53.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:53.706+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.706894+0000) 2020-02-05T21:48:53.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:53.706+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.707524+0000) 2020-02-05T21:48:53.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:53.707+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.707883+0000) 2020-02-05T21:48:53.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:53.708+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.708339+0000) 2020-02-05T21:48:53.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:53.707+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.709176+0000) 2020-02-05T21:48:53.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:53.709+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.709452+0000) 2020-02-05T21:48:53.711 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:53.708+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.710068+0000) 2020-02-05T21:48:53.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:53.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.710212+0000) 2020-02-05T21:48:53.711 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:53.710+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.710580+0000) 2020-02-05T21:48:53.712 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:53.709+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.710934+0000) 2020-02-05T21:48:53.712 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:53.709+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.711063+0000) 2020-02-05T21:48:53.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:53.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:53.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:53.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:53.716897+0000) 2020-02-05T21:48:55.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:55.877+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:55.878801+0000) 2020-02-05T21:48:57.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:57.023+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:57.023830+0000) 2020-02-05T21:48:57.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:57.023+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:57.023828+0000) 2020-02-05T21:48:57.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:57.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:57.031884+0000) 2020-02-05T21:48:58.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:58.201+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.202520+0000) 2020-02-05T21:48:58.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:48:58.723+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.724892+0000) 2020-02-05T21:48:58.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:58.723+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.725071+0000) 2020-02-05T21:48:58.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:48:58.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.725006+0000) 2020-02-05T21:48:58.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:48:58.724+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.725070+0000) 2020-02-05T21:48:58.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:58.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.725977+0000) 2020-02-05T21:48:58.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:48:58.725+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.726756+0000) 2020-02-05T21:48:58.730 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:48:58.726+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.727973+0000) 2020-02-05T21:48:58.732 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:48:58.726+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.727559+0000) 2020-02-05T21:48:58.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:48:58.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.727682+0000) 2020-02-05T21:48:58.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:48:58.727+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.728205+0000) 2020-02-05T21:48:58.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:48:58.727+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.728422+0000) 2020-02-05T21:48:58.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:48:58.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.729010+0000) 2020-02-05T21:48:58.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:58.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.732255+0000) 2020-02-05T21:48:58.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:58.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:58.733365+0000) 2020-02-05T21:48:58.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:48:58.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96927 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:48:59.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:48:59.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:59.132721+0000) 2020-02-05T21:48:59.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:48:59.413+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:48:59.414762+0000) 2020-02-05T21:49:00.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:00.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:00.022631+0000) 2020-02-05T21:49:00.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:00.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:00.257211+0000) 2020-02-05T21:49:00.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:00.274+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:00.275322+0000) 2020-02-05T21:49:00.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:00.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:00.275421+0000) 2020-02-05T21:49:00.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:00.282+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:00.283790+0000) 2020-02-05T21:49:02.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:02.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:02.266736+0000) 2020-02-05T21:49:02.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:02.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:02.665810+0000) 2020-02-05T21:49:03.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:03.739+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.740476+0000) 2020-02-05T21:49:03.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:03.739+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.740866+0000) 2020-02-05T21:49:03.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:03.740+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.741505+0000) 2020-02-05T21:49:03.743 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:03.741+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.742163+0000) 2020-02-05T21:49:03.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:03.741+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.742474+0000) 2020-02-05T21:49:03.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:03.741+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.742672+0000) 2020-02-05T21:49:03.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:03.742+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.743877+0000) 2020-02-05T21:49:03.745 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:03.742+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.743671+0000) 2020-02-05T21:49:03.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:03.742+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.744257+0000) 2020-02-05T21:49:03.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:03.743+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.744537+0000) 2020-02-05T21:49:03.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:03.743+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.744860+0000) 2020-02-05T21:49:03.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:03.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.745299+0000) 2020-02-05T21:49:03.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:03.744+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.746024+0000) 2020-02-05T21:49:03.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:03.747+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:03.748817+0000) 2020-02-05T21:49:03.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:03.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 96968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:05.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:05.878+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:05.879036+0000) 2020-02-05T21:49:07.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:07.023+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:07.024090+0000) 2020-02-05T21:49:07.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:07.023+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:07.024097+0000) 2020-02-05T21:49:07.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:07.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:07.032139+0000) 2020-02-05T21:49:08.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:08.209+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.210587+0000) 2020-02-05T21:49:08.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:08.755+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.756246+0000) 2020-02-05T21:49:08.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:08.756+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.757814+0000) 2020-02-05T21:49:08.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:08.756+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.758205+0000) 2020-02-05T21:49:08.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:08.756+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.758260+0000) 2020-02-05T21:49:08.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:08.757+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.758925+0000) 2020-02-05T21:49:08.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:08.759+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.760258+0000) 2020-02-05T21:49:08.761 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:08.759+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.760326+0000) 2020-02-05T21:49:08.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:08.759+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.760930+0000) 2020-02-05T21:49:08.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:08.760+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.761183+0000) 2020-02-05T21:49:08.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:08.760+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.761559+0000) 2020-02-05T21:49:08.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:08.760+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.761703+0000) 2020-02-05T21:49:08.763 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:08.760+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.762275+0000) 2020-02-05T21:49:08.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:08.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:08.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:08.768+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.770001+0000) 2020-02-05T21:49:08.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:08.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:08.770393+0000) 2020-02-05T21:49:09.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:09.132+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:09.132939+0000) 2020-02-05T21:49:09.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:09.414+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:09.414977+0000) 2020-02-05T21:49:10.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:10.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:10.022861+0000) 2020-02-05T21:49:10.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:10.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:10.257490+0000) 2020-02-05T21:49:10.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:10.274+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:10.275553+0000) 2020-02-05T21:49:10.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:10.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:10.275717+0000) 2020-02-05T21:49:10.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:10.283+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:10.284014+0000) 2020-02-05T21:49:12.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:12.265+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:12.266982+0000) 2020-02-05T21:49:12.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:12.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:12.666114+0000) 2020-02-05T21:49:13.775 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:13.772+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.773868+0000) 2020-02-05T21:49:13.777 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:13.774+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.776050+0000) 2020-02-05T21:49:13.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:13.775+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.776063+0000) 2020-02-05T21:49:13.778 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:13.775+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.776363+0000) 2020-02-05T21:49:13.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:13.775+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.777022+0000) 2020-02-05T21:49:13.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:13.776+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.777199+0000) 2020-02-05T21:49:13.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:13.776+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.777553+0000) 2020-02-05T21:49:13.781 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:13.777+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.778762+0000) 2020-02-05T21:49:13.781 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:13.777+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.778712+0000) 2020-02-05T21:49:13.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:13.777+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.778738+0000) 2020-02-05T21:49:13.782 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:13.777+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.779205+0000) 2020-02-05T21:49:13.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:13.776+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.777806+0000) 2020-02-05T21:49:13.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:13.778+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.779341+0000) 2020-02-05T21:49:13.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:13.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:13.783746+0000) 2020-02-05T21:49:13.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:13.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:15.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:15.878+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:15.879339+0000) 2020-02-05T21:49:17.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:17.023+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:17.024350+0000) 2020-02-05T21:49:17.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:17.023+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:17.024337+0000) 2020-02-05T21:49:17.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:17.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:17.032323+0000) 2020-02-05T21:49:18.131 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:49:18.138 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:49:18.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:18.221+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.222422+0000) 2020-02-05T21:49:18.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:18.790+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.791707+0000) 2020-02-05T21:49:18.794 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:18.791+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.792675+0000) 2020-02-05T21:49:18.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:18.792+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.793655+0000) 2020-02-05T21:49:18.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:18.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.794288+0000) 2020-02-05T21:49:18.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:18.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.794739+0000) 2020-02-05T21:49:18.798 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:18.793+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.795352+0000) 2020-02-05T21:49:18.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:18.794+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.795985+0000) 2020-02-05T21:49:18.798 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:18.795+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.796587+0000) 2020-02-05T21:49:18.798 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:18.795+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.796860+0000) 2020-02-05T21:49:18.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:18.794+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.795372+0000) 2020-02-05T21:49:18.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:18.794+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.795993+0000) 2020-02-05T21:49:18.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:18.796+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.797891+0000) 2020-02-05T21:49:18.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:18.796+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.797722+0000) 2020-02-05T21:49:18.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:18.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:18.803239+0000) 2020-02-05T21:49:18.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:18.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:19.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:19.131+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:19.133159+0000) 2020-02-05T21:49:19.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:19.414+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:19.415189+0000) 2020-02-05T21:49:20.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:20.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:20.023116+0000) 2020-02-05T21:49:20.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:20.256+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:20.257753+0000) 2020-02-05T21:49:20.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:20.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:20.275778+0000) 2020-02-05T21:49:20.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:20.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:20.275949+0000) 2020-02-05T21:49:20.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:20.283+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:20.284259+0000) 2020-02-05T21:49:22.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:22.266+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:22.267183+0000) 2020-02-05T21:49:22.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:22.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:22.666359+0000) 2020-02-05T21:49:23.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:23.807+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.808447+0000) 2020-02-05T21:49:23.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:23.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.809738+0000) 2020-02-05T21:49:23.813 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:23.809+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.811336+0000) 2020-02-05T21:49:23.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:23.809+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.810555+0000) 2020-02-05T21:49:23.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:23.809+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.810653+0000) 2020-02-05T21:49:23.814 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:23.810+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.811681+0000) 2020-02-05T21:49:23.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:23.811+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.812232+0000) 2020-02-05T21:49:23.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:23.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.813020+0000) 2020-02-05T21:49:23.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:23.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.812641+0000) 2020-02-05T21:49:23.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:23.812+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.814081+0000) 2020-02-05T21:49:23.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:23.813+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.814780+0000) 2020-02-05T21:49:23.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:23.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.815033+0000) 2020-02-05T21:49:23.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:23.814+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.815424+0000) 2020-02-05T21:49:23.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:23.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:23.819770+0000) 2020-02-05T21:49:23.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:23.820+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:25.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:25.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:25.879576+0000) 2020-02-05T21:49:27.026 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:27.024+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:27.024622+0000) 2020-02-05T21:49:27.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:27.024+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:27.024579+0000) 2020-02-05T21:49:27.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:27.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:27.032556+0000) 2020-02-05T21:49:28.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:28.232+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.233334+0000) 2020-02-05T21:49:28.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:28.826+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.827036+0000) 2020-02-05T21:49:28.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:28.826+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.828060+0000) 2020-02-05T21:49:28.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:28.827+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.828469+0000) 2020-02-05T21:49:28.830 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:28.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.829558+0000) 2020-02-05T21:49:28.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:28.828+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.829768+0000) 2020-02-05T21:49:28.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:28.829+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.830322+0000) 2020-02-05T21:49:28.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:28.829+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.830516+0000) 2020-02-05T21:49:28.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:28.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.830889+0000) 2020-02-05T21:49:28.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:28.831+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.832826+0000) 2020-02-05T21:49:28.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:28.832+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.832874+0000) 2020-02-05T21:49:28.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:28.832+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.833305+0000) 2020-02-05T21:49:28.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:28.833+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.833567+0000) 2020-02-05T21:49:28.835 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:28.833+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.834117+0000) 2020-02-05T21:49:28.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:28.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:28.837975+0000) 2020-02-05T21:49:28.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:28.839+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97175 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:29.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:29.133+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:29.133394+0000) 2020-02-05T21:49:29.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:29.414+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:29.415376+0000) 2020-02-05T21:49:30.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:30.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:30.023399+0000) 2020-02-05T21:49:30.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:30.257+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:30.258004+0000) 2020-02-05T21:49:30.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:30.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:30.276178+0000) 2020-02-05T21:49:30.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:30.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:30.276006+0000) 2020-02-05T21:49:30.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:30.284+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:30.284507+0000) 2020-02-05T21:49:32.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:32.267+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:32.267499+0000) 2020-02-05T21:49:32.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:32.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:32.666608+0000) 2020-02-05T21:49:33.846 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:33.843+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.844688+0000) 2020-02-05T21:49:33.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:33.844+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.845538+0000) 2020-02-05T21:49:33.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:33.844+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.845787+0000) 2020-02-05T21:49:33.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:33.845+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.846269+0000) 2020-02-05T21:49:33.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:33.845+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.846845+0000) 2020-02-05T21:49:33.848 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:33.846+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.846843+0000) 2020-02-05T21:49:33.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:33.846+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.847310+0000) 2020-02-05T21:49:33.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:33.846+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.847603+0000) 2020-02-05T21:49:33.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:33.846+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.847947+0000) 2020-02-05T21:49:33.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:33.847+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.848223+0000) 2020-02-05T21:49:33.850 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:33.848+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.849215+0000) 2020-02-05T21:49:33.851 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:33.849+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.849477+0000) 2020-02-05T21:49:33.851 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:33.849+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.849899+0000) 2020-02-05T21:49:33.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:33.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:33.853911+0000) 2020-02-05T21:49:33.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:33.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:35.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:35.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:35.879811+0000) 2020-02-05T21:49:37.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:37.024+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:37.024825+0000) 2020-02-05T21:49:37.026 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:37.024+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:37.024860+0000) 2020-02-05T21:49:37.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:37.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:37.032787+0000) 2020-02-05T21:49:38.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:38.241+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.241885+0000) 2020-02-05T21:49:38.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:38.860+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.861923+0000) 2020-02-05T21:49:38.864 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:38.862+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.863374+0000) 2020-02-05T21:49:38.867 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:38.863+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.863982+0000) 2020-02-05T21:49:38.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:38.862+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.864011+0000) 2020-02-05T21:49:38.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:38.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.864661+0000) 2020-02-05T21:49:38.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:38.864+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.865153+0000) 2020-02-05T21:49:38.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:38.866+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.866761+0000) 2020-02-05T21:49:38.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:38.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.866164+0000) 2020-02-05T21:49:38.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:38.867+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.867550+0000) 2020-02-05T21:49:38.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:38.867+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.867860+0000) 2020-02-05T21:49:38.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:38.867+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.868343+0000) 2020-02-05T21:49:38.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:38.866+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.867694+0000) 2020-02-05T21:49:38.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:38.866+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.868002+0000) 2020-02-05T21:49:38.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:38.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:38.872391+0000) 2020-02-05T21:49:38.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:38.873+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:39.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:39.133+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:39.133731+0000) 2020-02-05T21:49:39.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:39.415+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:39.415585+0000) 2020-02-05T21:49:40.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:40.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:40.023660+0000) 2020-02-05T21:49:40.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:40.257+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:40.258241+0000) 2020-02-05T21:49:40.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:40.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:40.276241+0000) 2020-02-05T21:49:40.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:40.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:40.276454+0000) 2020-02-05T21:49:40.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:40.284+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:40.284731+0000) 2020-02-05T21:49:42.269 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:42.267+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:42.267688+0000) 2020-02-05T21:49:42.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:42.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:42.666839+0000) 2020-02-05T21:49:43.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:43.878+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.879127+0000) 2020-02-05T21:49:43.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:43.880+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.880608+0000) 2020-02-05T21:49:43.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:43.879+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.881042+0000) 2020-02-05T21:49:43.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:43.881+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.881771+0000) 2020-02-05T21:49:43.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:43.880+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.881582+0000) 2020-02-05T21:49:43.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:43.880+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.882060+0000) 2020-02-05T21:49:43.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:43.881+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.882818+0000) 2020-02-05T21:49:43.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:43.882+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.883007+0000) 2020-02-05T21:49:43.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:43.882+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.883203+0000) 2020-02-05T21:49:43.885 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:43.883+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.883951+0000) 2020-02-05T21:49:43.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:43.882+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.883645+0000) 2020-02-05T21:49:43.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:43.884+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.884603+0000) 2020-02-05T21:49:43.886 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:43.884+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.884627+0000) 2020-02-05T21:49:43.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:43.888+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:43.889450+0000) 2020-02-05T21:49:43.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:43.891+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:45.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:45.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:45.880045+0000) 2020-02-05T21:49:47.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:47.024+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:47.025061+0000) 2020-02-05T21:49:47.026 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:47.024+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:47.025141+0000) 2020-02-05T21:49:47.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:47.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:47.033038+0000) 2020-02-05T21:49:48.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:48.250+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.250827+0000) 2020-02-05T21:49:48.276 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:49:48.283 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:49:48.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:48.897+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.898405+0000) 2020-02-05T21:49:48.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:48.898+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.899142+0000) 2020-02-05T21:49:48.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:48.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.899770+0000) 2020-02-05T21:49:48.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:48.899+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.899854+0000) 2020-02-05T21:49:48.902 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:48.900+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.900972+0000) 2020-02-05T21:49:48.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:48.900+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.900869+0000) 2020-02-05T21:49:48.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:48.900+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.901002+0000) 2020-02-05T21:49:48.903 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:48.901+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.901346+0000) 2020-02-05T21:49:48.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:48.901+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.901560+0000) 2020-02-05T21:49:48.905 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:48.901+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.902011+0000) 2020-02-05T21:49:48.905 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:48.902+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.902609+0000) 2020-02-05T21:49:48.906 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:48.902+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.902765+0000) 2020-02-05T21:49:48.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:48.902+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.902473+0000) 2020-02-05T21:49:48.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:48.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:48.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:48.909+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:48.909509+0000) 2020-02-05T21:49:49.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:49.133+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:49.133963+0000) 2020-02-05T21:49:49.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:49.415+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:49.415849+0000) 2020-02-05T21:49:50.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:50.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:50.023930+0000) 2020-02-05T21:49:50.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:50.258+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:50.258491+0000) 2020-02-05T21:49:50.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:50.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:50.276424+0000) 2020-02-05T21:49:50.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:50.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:50.276720+0000) 2020-02-05T21:49:50.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:50.283+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:50.284957+0000) 2020-02-05T21:49:52.269 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:52.267+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:52.267908+0000) 2020-02-05T21:49:52.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:52.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:52.667069+0000) 2020-02-05T21:49:53.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:53.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.913774+0000) 2020-02-05T21:49:53.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:53.913+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.914270+0000) 2020-02-05T21:49:53.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:53.914+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.914806+0000) 2020-02-05T21:49:53.916 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:53.915+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.915630+0000) 2020-02-05T21:49:53.917 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:53.914+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.915789+0000) 2020-02-05T21:49:53.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:53.914+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.916030+0000) 2020-02-05T21:49:53.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:53.915+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.916376+0000) 2020-02-05T21:49:53.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:53.915+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.916905+0000) 2020-02-05T21:49:53.918 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:53.916+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.917129+0000) 2020-02-05T21:49:53.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:53.917+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.917992+0000) 2020-02-05T21:49:53.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:53.917+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.917847+0000) 2020-02-05T21:49:53.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:53.917+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.918267+0000) 2020-02-05T21:49:53.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:53.917+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.917964+0000) 2020-02-05T21:49:53.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:53.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:53.923169+0000) 2020-02-05T21:49:53.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:53.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:55.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:55.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:55.880282+0000) 2020-02-05T21:49:57.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:57.025+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:57.025311+0000) 2020-02-05T21:49:57.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:57.025+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:57.025387+0000) 2020-02-05T21:49:57.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:57.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:57.033281+0000) 2020-02-05T21:49:58.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:58.258+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.259464+0000) 2020-02-05T21:49:58.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:49:58.928+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.930035+0000) 2020-02-05T21:49:58.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:58.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.930752+0000) 2020-02-05T21:49:58.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:58.931+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.931999+0000) 2020-02-05T21:49:58.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:58.931+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.932518+0000) 2020-02-05T21:49:58.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:49:58.931+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.932499+0000) 2020-02-05T21:49:58.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:49:58.931+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.932906+0000) 2020-02-05T21:49:58.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:49:58.932+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.933523+0000) 2020-02-05T21:49:58.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:49:58.932+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.933943+0000) 2020-02-05T21:49:58.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:49:58.934+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.934408+0000) 2020-02-05T21:49:58.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:49:58.935+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.935464+0000) 2020-02-05T21:49:58.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:49:58.935+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.935808+0000) 2020-02-05T21:49:58.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:49:58.936+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.936884+0000) 2020-02-05T21:49:58.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:49:58.936+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.937247+0000) 2020-02-05T21:49:58.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:58.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:58.939560+0000) 2020-02-05T21:49:58.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:49:58.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:49:59.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:49:59.134+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:59.134191+0000) 2020-02-05T21:49:59.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:49:59.415+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:49:59.416147+0000) 2020-02-05T21:50:00.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:00.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:00.024159+0000) 2020-02-05T21:50:00.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:00.258+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:00.258756+0000) 2020-02-05T21:50:00.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:00.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:00.276649+0000) 2020-02-05T21:50:00.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:00.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:00.276965+0000) 2020-02-05T21:50:00.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:00.284+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:00.285182+0000) 2020-02-05T21:50:02.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:02.268+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:02.268152+0000) 2020-02-05T21:50:02.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:02.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:02.667331+0000) 2020-02-05T21:50:03.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:03.946+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.947688+0000) 2020-02-05T21:50:03.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:03.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.948828+0000) 2020-02-05T21:50:03.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:03.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.950259+0000) 2020-02-05T21:50:03.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:03.949+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.950533+0000) 2020-02-05T21:50:03.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:03.950+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.950695+0000) 2020-02-05T21:50:03.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:03.950+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.951230+0000) 2020-02-05T21:50:03.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:03.950+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.951395+0000) 2020-02-05T21:50:03.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:03.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.950723+0000) 2020-02-05T21:50:03.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:03.949+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.951047+0000) 2020-02-05T21:50:03.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:03.951+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.951740+0000) 2020-02-05T21:50:03.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:03.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.952060+0000) 2020-02-05T21:50:03.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:03.951+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.952129+0000) 2020-02-05T21:50:03.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:03.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.955556+0000) 2020-02-05T21:50:03.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:03.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:03.955787+0000) 2020-02-05T21:50:03.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:03.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97463 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:05.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:05.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:05.880581+0000) 2020-02-05T21:50:07.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:07.024+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:07.025667+0000) 2020-02-05T21:50:07.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:07.024+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:07.025667+0000) 2020-02-05T21:50:07.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:07.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:07.033527+0000) 2020-02-05T21:50:08.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:08.266+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.267643+0000) 2020-02-05T21:50:08.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:08.963+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.964942+0000) 2020-02-05T21:50:08.967 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:08.964+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.966019+0000) 2020-02-05T21:50:08.968 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:08.965+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.966450+0000) 2020-02-05T21:50:08.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:08.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.966610+0000) 2020-02-05T21:50:08.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:08.965+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.967074+0000) 2020-02-05T21:50:08.969 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:08.965+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.966946+0000) 2020-02-05T21:50:08.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:08.966+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.967780+0000) 2020-02-05T21:50:08.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:08.966+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.967371+0000) 2020-02-05T21:50:08.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:08.966+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.968132+0000) 2020-02-05T21:50:08.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:08.967+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.969243+0000) 2020-02-05T21:50:08.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:08.967+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.968463+0000) 2020-02-05T21:50:08.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:08.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.968749+0000) 2020-02-05T21:50:08.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:08.969+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.970286+0000) 2020-02-05T21:50:08.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:08.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:08.975720+0000) 2020-02-05T21:50:08.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:08.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:09.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:09.133+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:09.134415+0000) 2020-02-05T21:50:09.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:09.415+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:09.416459+0000) 2020-02-05T21:50:10.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:10.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:10.024409+0000) 2020-02-05T21:50:10.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:10.258+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:10.258999+0000) 2020-02-05T21:50:10.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:10.275+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:10.276859+0000) 2020-02-05T21:50:10.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:10.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:10.277238+0000) 2020-02-05T21:50:10.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:10.284+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:10.285436+0000) 2020-02-05T21:50:12.269 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:12.266+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:12.268406+0000) 2020-02-05T21:50:12.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:12.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:12.667552+0000) 2020-02-05T21:50:13.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:13.981+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.982709+0000) 2020-02-05T21:50:13.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:13.982+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.983626+0000) 2020-02-05T21:50:13.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:13.982+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.983959+0000) 2020-02-05T21:50:13.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:13.983+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.984763+0000) 2020-02-05T21:50:13.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:13.983+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.985303+0000) 2020-02-05T21:50:13.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:13.984+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.985161+0000) 2020-02-05T21:50:13.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:13.984+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.985960+0000) 2020-02-05T21:50:13.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:13.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.984563+0000) 2020-02-05T21:50:13.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:13.985+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.986459+0000) 2020-02-05T21:50:13.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:13.985+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.986855+0000) 2020-02-05T21:50:13.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:13.986+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.987437+0000) 2020-02-05T21:50:13.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:13.986+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.987796+0000) 2020-02-05T21:50:13.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:13.986+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.987587+0000) 2020-02-05T21:50:13.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:13.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:13.991784+0000) 2020-02-05T21:50:13.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:13.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:15.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:15.879+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:15.880825+0000) 2020-02-05T21:50:17.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:17.024+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:17.025951+0000) 2020-02-05T21:50:17.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:17.024+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:17.025915+0000) 2020-02-05T21:50:17.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:17.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:17.033783+0000) 2020-02-05T21:50:18.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:18.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:18.275786+0000) 2020-02-05T21:50:18.444 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:50:18.451 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:50:19.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:18.998+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:18.999173+0000) 2020-02-05T21:50:19.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:18.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:18.999958+0000) 2020-02-05T21:50:19.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:18.999+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.000439+0000) 2020-02-05T21:50:19.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:18.999+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.000785+0000) 2020-02-05T21:50:19.003 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:19.000+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.001983+0000) 2020-02-05T21:50:19.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:19.000+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.002351+0000) 2020-02-05T21:50:19.004 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:19.001+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.002636+0000) 2020-02-05T21:50:19.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:19.001+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.002831+0000) 2020-02-05T21:50:19.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:19.002+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.003363+0000) 2020-02-05T21:50:19.005 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:19.002+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.003970+0000) 2020-02-05T21:50:19.006 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:19.002+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.004238+0000) 2020-02-05T21:50:19.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:19.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.004601+0000) 2020-02-05T21:50:19.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:19.003+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.004914+0000) 2020-02-05T21:50:19.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:19.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.006831+0000) 2020-02-05T21:50:19.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:19.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:19.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:19.133+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.134689+0000) 2020-02-05T21:50:19.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:19.416+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:19.416695+0000) 2020-02-05T21:50:20.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:20.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:20.024634+0000) 2020-02-05T21:50:20.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:20.257+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:20.259207+0000) 2020-02-05T21:50:20.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:20.276+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:20.277113+0000) 2020-02-05T21:50:20.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:20.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:20.277545+0000) 2020-02-05T21:50:20.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:20.285+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:20.285701+0000) 2020-02-05T21:50:22.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:22.267+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:22.268695+0000) 2020-02-05T21:50:22.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:22.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:22.667807+0000) 2020-02-05T21:50:24.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:24.016+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.017508+0000) 2020-02-05T21:50:24.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:24.017+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.018955+0000) 2020-02-05T21:50:24.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:24.017+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.018965+0000) 2020-02-05T21:50:24.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:24.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.019360+0000) 2020-02-05T21:50:24.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:24.018+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.019718+0000) 2020-02-05T21:50:24.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:24.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.020588+0000) 2020-02-05T21:50:24.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:24.019+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.020887+0000) 2020-02-05T21:50:24.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:24.020+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.021940+0000) 2020-02-05T21:50:24.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:24.021+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.022580+0000) 2020-02-05T21:50:24.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:24.022+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.023280+0000) 2020-02-05T21:50:24.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:24.022+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.023557+0000) 2020-02-05T21:50:24.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:24.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.023869+0000) 2020-02-05T21:50:24.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:24.022+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.024281+0000) 2020-02-05T21:50:24.029 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:24.027+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:24.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:24.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:24.029370+0000) 2020-02-05T21:50:25.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:25.880+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:25.881068+0000) 2020-02-05T21:50:27.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:27.025+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:27.026198+0000) 2020-02-05T21:50:27.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:27.025+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:27.026185+0000) 2020-02-05T21:50:27.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:27.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:27.034020+0000) 2020-02-05T21:50:28.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:28.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:28.284020+0000) 2020-02-05T21:50:29.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:29.033+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.034307+0000) 2020-02-05T21:50:29.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:29.033+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.035054+0000) 2020-02-05T21:50:29.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:29.035+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.036209+0000) 2020-02-05T21:50:29.038 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:29.035+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.036823+0000) 2020-02-05T21:50:29.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:29.035+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.036926+0000) 2020-02-05T21:50:29.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:29.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.037425+0000) 2020-02-05T21:50:29.039 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:29.036+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.037975+0000) 2020-02-05T21:50:29.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:29.037+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.038770+0000) 2020-02-05T21:50:29.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:29.037+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.039037+0000) 2020-02-05T21:50:29.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:29.038+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.039792+0000) 2020-02-05T21:50:29.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:29.038+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.039985+0000) 2020-02-05T21:50:29.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:29.038+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.040268+0000) 2020-02-05T21:50:29.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:29.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.040697+0000) 2020-02-05T21:50:29.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:29.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.045422+0000) 2020-02-05T21:50:29.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:29.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97670 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:29.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:29.134+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.134934+0000) 2020-02-05T21:50:29.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:29.416+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:29.416908+0000) 2020-02-05T21:50:30.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:30.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:30.024875+0000) 2020-02-05T21:50:30.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:30.258+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:30.259485+0000) 2020-02-05T21:50:30.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:30.277+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:30.277340+0000) 2020-02-05T21:50:30.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:30.277+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:30.277785+0000) 2020-02-05T21:50:30.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:30.284+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:30.285919+0000) 2020-02-05T21:50:32.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:32.268+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:32.268951+0000) 2020-02-05T21:50:32.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:32.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:32.668065+0000) 2020-02-05T21:50:34.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:34.050+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.051774+0000) 2020-02-05T21:50:34.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:34.051+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.052962+0000) 2020-02-05T21:50:34.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:34.052+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.054131+0000) 2020-02-05T21:50:34.056 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:34.054+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.055346+0000) 2020-02-05T21:50:34.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:34.053+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.055337+0000) 2020-02-05T21:50:34.057 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:34.054+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.056023+0000) 2020-02-05T21:50:34.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:34.055+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.056976+0000) 2020-02-05T21:50:34.059 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:34.056+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.057138+0000) 2020-02-05T21:50:34.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:34.056+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.057826+0000) 2020-02-05T21:50:34.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:34.057+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.058497+0000) 2020-02-05T21:50:34.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:34.057+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.058914+0000) 2020-02-05T21:50:34.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:34.057+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.059060+0000) 2020-02-05T21:50:34.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:34.057+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.058560+0000) 2020-02-05T21:50:34.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:34.057+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:34.059100+0000) 2020-02-05T21:50:34.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:34.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97713 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:35.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:35.880+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:35.881329+0000) 2020-02-05T21:50:37.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:37.025+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:37.026469+0000) 2020-02-05T21:50:37.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:37.025+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:37.026472+0000) 2020-02-05T21:50:37.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:37.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:37.034295+0000) 2020-02-05T21:50:38.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:38.293+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:38.294586+0000) 2020-02-05T21:50:39.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:39.069+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.070608+0000) 2020-02-05T21:50:39.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:39.069+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.071009+0000) 2020-02-05T21:50:39.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:39.070+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.071633+0000) 2020-02-05T21:50:39.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:39.070+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.071931+0000) 2020-02-05T21:50:39.073 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:39.071+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.072334+0000) 2020-02-05T21:50:39.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:39.071+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.072627+0000) 2020-02-05T21:50:39.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:39.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.072887+0000) 2020-02-05T21:50:39.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:39.072+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.073991+0000) 2020-02-05T21:50:39.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:39.073+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.074456+0000) 2020-02-05T21:50:39.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:39.073+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.074805+0000) 2020-02-05T21:50:39.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:39.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.075331+0000) 2020-02-05T21:50:39.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:39.074+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.076017+0000) 2020-02-05T21:50:39.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:39.074+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.075744+0000) 2020-02-05T21:50:39.082 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:39.079+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97753 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:39.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:39.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.081926+0000) 2020-02-05T21:50:39.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:39.134+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.135221+0000) 2020-02-05T21:50:39.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:39.416+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:39.417170+0000) 2020-02-05T21:50:40.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:40.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:40.025133+0000) 2020-02-05T21:50:40.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:40.258+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:40.259716+0000) 2020-02-05T21:50:40.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:40.277+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:40.277574+0000) 2020-02-05T21:50:40.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:40.277+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:40.278025+0000) 2020-02-05T21:50:40.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:40.286+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:40.286161+0000) 2020-02-05T21:50:42.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:42.267+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:42.269197+0000) 2020-02-05T21:50:42.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:42.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:42.668359+0000) 2020-02-05T21:50:44.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:44.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.087782+0000) 2020-02-05T21:50:44.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:44.087+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.088421+0000) 2020-02-05T21:50:44.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:44.089+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.089315+0000) 2020-02-05T21:50:44.091 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:44.089+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.089708+0000) 2020-02-05T21:50:44.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:44.088+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.090336+0000) 2020-02-05T21:50:44.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:44.089+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.090715+0000) 2020-02-05T21:50:44.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:44.089+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.091104+0000) 2020-02-05T21:50:44.092 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:44.091+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.091317+0000) 2020-02-05T21:50:44.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:44.090+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.091944+0000) 2020-02-05T21:50:44.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:44.092+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.092386+0000) 2020-02-05T21:50:44.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:44.092+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.092589+0000) 2020-02-05T21:50:44.094 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:44.091+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.092704+0000) 2020-02-05T21:50:44.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:44.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.093051+0000) 2020-02-05T21:50:44.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:44.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97795 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:44.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:44.098+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:44.098544+0000) 2020-02-05T21:50:45.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:45.880+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:45.881563+0000) 2020-02-05T21:50:47.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:47.026+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:47.026721+0000) 2020-02-05T21:50:47.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:47.026+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:47.026719+0000) 2020-02-05T21:50:47.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:47.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:47.034515+0000) 2020-02-05T21:50:48.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:48.301+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:48.302796+0000) 2020-02-05T21:50:48.592 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:50:48.598 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:50:49.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:49.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.103363+0000) 2020-02-05T21:50:49.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:49.104+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.105346+0000) 2020-02-05T21:50:49.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:49.103+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.105231+0000) 2020-02-05T21:50:49.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:49.104+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.105874+0000) 2020-02-05T21:50:49.107 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:49.105+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.106163+0000) 2020-02-05T21:50:49.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:49.105+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.106581+0000) 2020-02-05T21:50:49.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:49.105+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.107232+0000) 2020-02-05T21:50:49.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:49.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.107382+0000) 2020-02-05T21:50:49.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:49.106+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.108082+0000) 2020-02-05T21:50:49.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:49.106+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.107924+0000) 2020-02-05T21:50:49.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:49.107+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.108875+0000) 2020-02-05T21:50:49.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:49.107+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.109005+0000) 2020-02-05T21:50:49.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:49.107+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.109295+0000) 2020-02-05T21:50:49.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:49.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:49.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:49.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.113518+0000) 2020-02-05T21:50:49.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:49.134+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.135487+0000) 2020-02-05T21:50:49.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:49.416+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:49.417330+0000) 2020-02-05T21:50:50.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:50.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:50.025384+0000) 2020-02-05T21:50:50.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:50.259+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:50.260000+0000) 2020-02-05T21:50:50.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:50.276+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:50.277808+0000) 2020-02-05T21:50:50.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:50.277+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:50.278273+0000) 2020-02-05T21:50:50.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:50.285+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:50.286331+0000) 2020-02-05T21:50:52.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:52.268+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:52.269481+0000) 2020-02-05T21:50:52.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:52.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:52.668594+0000) 2020-02-05T21:50:54.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:54.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.119632+0000) 2020-02-05T21:50:54.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:54.120+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.121170+0000) 2020-02-05T21:50:54.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:54.120+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.121606+0000) 2020-02-05T21:50:54.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:54.120+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.121846+0000) 2020-02-05T21:50:54.124 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:54.121+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.122808+0000) 2020-02-05T21:50:54.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:54.122+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.123154+0000) 2020-02-05T21:50:54.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:54.122+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.124291+0000) 2020-02-05T21:50:54.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:54.123+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.124899+0000) 2020-02-05T21:50:54.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:54.124+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.125468+0000) 2020-02-05T21:50:54.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:54.124+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.125523+0000) 2020-02-05T21:50:54.127 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:54.124+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.125937+0000) 2020-02-05T21:50:54.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:54.125+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.126597+0000) 2020-02-05T21:50:54.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:54.125+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.126705+0000) 2020-02-05T21:50:54.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:54.129+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:54.130941+0000) 2020-02-05T21:50:54.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:54.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97879 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:55.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:55.880+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:55.881799+0000) 2020-02-05T21:50:57.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:57.026+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:57.027002+0000) 2020-02-05T21:50:57.030 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:57.026+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:57.027003+0000) 2020-02-05T21:50:57.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:57.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:57.034746+0000) 2020-02-05T21:50:58.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:58.312+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:58.313131+0000) 2020-02-05T21:50:59.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:50:59.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.138435+0000) 2020-02-05T21:50:59.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:59.137+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.138958+0000) 2020-02-05T21:50:59.141 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:50:59.138+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.139540+0000) 2020-02-05T21:50:59.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:50:59.138+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.139946+0000) 2020-02-05T21:50:59.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:50:59.140+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.140876+0000) 2020-02-05T21:50:59.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:50:59.141+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.141621+0000) 2020-02-05T21:50:59.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:59.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.141799+0000) 2020-02-05T21:50:59.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:50:59.141+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.142288+0000) 2020-02-05T21:50:59.143 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:50:59.142+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.142625+0000) 2020-02-05T21:50:59.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:50:59.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.142744+0000) 2020-02-05T21:50:59.144 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:50:59.142+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.143072+0000) 2020-02-05T21:50:59.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:50:59.143+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.144045+0000) 2020-02-05T21:50:59.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:50:59.144+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.144666+0000) 2020-02-05T21:50:59.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:59.148+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.149353+0000) 2020-02-05T21:50:59.151 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:50:59.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97919 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:50:59.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:50:59.416+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:50:59.417551+0000) 2020-02-05T21:51:00.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:00.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:00.025614+0000) 2020-02-05T21:51:00.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:00.259+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:00.260240+0000) 2020-02-05T21:51:00.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:00.277+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:00.278134+0000) 2020-02-05T21:51:00.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:00.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:00.278527+0000) 2020-02-05T21:51:00.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:00.285+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:00.286560+0000) 2020-02-05T21:51:02.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:02.269+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:02.269734+0000) 2020-02-05T21:51:02.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:02.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:02.668801+0000) 2020-02-05T21:51:04.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:04.154+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.155717+0000) 2020-02-05T21:51:04.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:04.155+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.156752+0000) 2020-02-05T21:51:04.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:04.156+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.157317+0000) 2020-02-05T21:51:04.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:04.157+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.157673+0000) 2020-02-05T21:51:04.159 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:04.157+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.158155+0000) 2020-02-05T21:51:04.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:04.158+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.158721+0000) 2020-02-05T21:51:04.161 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:04.159+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.159865+0000) 2020-02-05T21:51:04.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:04.160+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.160604+0000) 2020-02-05T21:51:04.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:04.159+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.160876+0000) 2020-02-05T21:51:04.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:04.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.161316+0000) 2020-02-05T21:51:04.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:04.160+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.161387+0000) 2020-02-05T21:51:04.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:04.160+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.161845+0000) 2020-02-05T21:51:04.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:04.162+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.163059+0000) 2020-02-05T21:51:04.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:04.166+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:04.167754+0000) 2020-02-05T21:51:04.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:04.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 97961 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:05.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:05.881+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:05.882052+0000) 2020-02-05T21:51:05.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:05.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:05.886999+0000) 2020-02-05T21:51:07.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:07.026+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:07.027257+0000) 2020-02-05T21:51:07.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:07.026+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:07.027257+0000) 2020-02-05T21:51:07.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:07.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:07.034965+0000) 2020-02-05T21:51:08.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:08.320+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:08.321459+0000) 2020-02-05T21:51:09.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:09.171+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.172201+0000) 2020-02-05T21:51:09.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:09.175+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.175476+0000) 2020-02-05T21:51:09.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:09.174+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.175743+0000) 2020-02-05T21:51:09.177 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:09.175+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.176196+0000) 2020-02-05T21:51:09.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:09.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.176554+0000) 2020-02-05T21:51:09.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:09.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.177046+0000) 2020-02-05T21:51:09.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:09.176+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.177862+0000) 2020-02-05T21:51:09.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:09.177+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.178937+0000) 2020-02-05T21:51:09.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:09.178+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.179321+0000) 2020-02-05T21:51:09.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:09.178+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.179626+0000) 2020-02-05T21:51:09.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:09.180+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.180769+0000) 2020-02-05T21:51:09.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:09.180+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.180896+0000) 2020-02-05T21:51:09.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:09.180+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.181026+0000) 2020-02-05T21:51:09.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:09.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.184870+0000) 2020-02-05T21:51:09.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:09.187+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98002 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:09.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:09.417+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:09.417766+0000) 2020-02-05T21:51:10.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:10.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:10.025859+0000) 2020-02-05T21:51:10.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:10.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:10.260502+0000) 2020-02-05T21:51:10.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:10.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:10.278799+0000) 2020-02-05T21:51:10.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:10.277+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:10.278366+0000) 2020-02-05T21:51:10.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:10.286+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:10.286793+0000) 2020-02-05T21:51:12.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:12.269+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:12.269971+0000) 2020-02-05T21:51:12.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:12.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:12.669054+0000) 2020-02-05T21:51:14.193 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:14.190+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.191976+0000) 2020-02-05T21:51:14.194 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:14.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.193314+0000) 2020-02-05T21:51:14.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:14.192+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.193802+0000) 2020-02-05T21:51:14.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:14.193+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.195059+0000) 2020-02-05T21:51:14.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:14.194+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.195151+0000) 2020-02-05T21:51:14.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:14.195+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.196136+0000) 2020-02-05T21:51:14.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:14.195+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.195788+0000) 2020-02-05T21:51:14.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:14.195+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.196586+0000) 2020-02-05T21:51:14.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:14.195+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.196910+0000) 2020-02-05T21:51:14.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:14.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.198235+0000) 2020-02-05T21:51:14.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:14.196+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.197407+0000) 2020-02-05T21:51:14.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:14.198+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.198932+0000) 2020-02-05T21:51:14.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:14.198+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.199239+0000) 2020-02-05T21:51:14.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:14.202+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:14.203909+0000) 2020-02-05T21:51:14.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:14.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:15.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:15.881+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:15.882364+0000) 2020-02-05T21:51:15.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:15.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:15.887251+0000) 2020-02-05T21:51:17.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:17.027+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:17.027515+0000) 2020-02-05T21:51:17.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:17.027+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:17.027515+0000) 2020-02-05T21:51:17.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:17.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:17.035197+0000) 2020-02-05T21:51:18.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:18.328+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:18.328990+0000) 2020-02-05T21:51:18.732 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:51:18.738 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:51:19.212 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:19.209+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.211049+0000) 2020-02-05T21:51:19.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:19.211+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.212738+0000) 2020-02-05T21:51:19.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:19.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.213589+0000) 2020-02-05T21:51:19.215 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:19.212+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.214024+0000) 2020-02-05T21:51:19.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:19.213+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.214467+0000) 2020-02-05T21:51:19.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:19.214+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.214837+0000) 2020-02-05T21:51:19.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:19.214+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.214888+0000) 2020-02-05T21:51:19.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:19.214+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.215398+0000) 2020-02-05T21:51:19.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:19.215+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.215462+0000) 2020-02-05T21:51:19.218 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:19.215+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.216140+0000) 2020-02-05T21:51:19.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:19.215+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.216817+0000) 2020-02-05T21:51:19.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:19.217+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.218081+0000) 2020-02-05T21:51:19.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:19.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:19.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:19.219+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.220103+0000) 2020-02-05T21:51:19.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:19.219+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.220360+0000) 2020-02-05T21:51:19.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:19.417+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:19.417965+0000) 2020-02-05T21:51:20.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:20.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:20.026069+0000) 2020-02-05T21:51:20.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:20.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:20.260761+0000) 2020-02-05T21:51:20.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:20.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:20.279075+0000) 2020-02-05T21:51:20.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:20.278+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:20.278590+0000) 2020-02-05T21:51:20.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:20.286+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:20.287028+0000) 2020-02-05T21:51:22.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:22.270+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:22.270239+0000) 2020-02-05T21:51:22.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:22.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:22.669356+0000) 2020-02-05T21:51:24.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:24.228+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.229359+0000) 2020-02-05T21:51:24.231 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:24.229+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.230362+0000) 2020-02-05T21:51:24.232 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:24.229+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.230889+0000) 2020-02-05T21:51:24.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:24.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.231363+0000) 2020-02-05T21:51:24.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:24.230+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.231563+0000) 2020-02-05T21:51:24.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:24.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.232199+0000) 2020-02-05T21:51:24.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:24.232+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.232496+0000) 2020-02-05T21:51:24.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:24.232+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.233302+0000) 2020-02-05T21:51:24.238 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:24.232+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.233176+0000) 2020-02-05T21:51:24.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:24.233+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.233823+0000) 2020-02-05T21:51:24.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:24.233+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.234086+0000) 2020-02-05T21:51:24.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:24.234+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.235022+0000) 2020-02-05T21:51:24.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:24.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.234325+0000) 2020-02-05T21:51:24.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:24.240+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:24.241169+0000) 2020-02-05T21:51:24.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:24.241+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:25.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:25.881+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:25.882560+0000) 2020-02-05T21:51:25.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:25.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:25.887497+0000) 2020-02-05T21:51:27.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:27.027+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:27.027774+0000) 2020-02-05T21:51:27.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:27.027+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:27.027773+0000) 2020-02-05T21:51:27.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:27.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:27.035435+0000) 2020-02-05T21:51:28.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:28.337+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:28.337445+0000) 2020-02-05T21:51:29.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:29.245+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.247034+0000) 2020-02-05T21:51:29.249 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:29.247+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.248128+0000) 2020-02-05T21:51:29.250 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:29.247+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.248922+0000) 2020-02-05T21:51:29.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:29.249+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.249731+0000) 2020-02-05T21:51:29.251 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:29.248+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.249990+0000) 2020-02-05T21:51:29.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:29.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.250486+0000) 2020-02-05T21:51:29.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:29.250+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.251228+0000) 2020-02-05T21:51:29.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:29.250+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.251442+0000) 2020-02-05T21:51:29.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:29.251+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.252456+0000) 2020-02-05T21:51:29.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:29.251+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.252049+0000) 2020-02-05T21:51:29.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:29.252+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.252931+0000) 2020-02-05T21:51:29.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:29.252+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.253196+0000) 2020-02-05T21:51:29.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:29.252+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.253343+0000) 2020-02-05T21:51:29.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:29.256+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.257450+0000) 2020-02-05T21:51:29.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:29.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:29.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:29.417+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:29.418189+0000) 2020-02-05T21:51:30.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:30.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:30.026318+0000) 2020-02-05T21:51:30.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:30.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:30.260997+0000) 2020-02-05T21:51:30.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:30.278+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:30.278825+0000) 2020-02-05T21:51:30.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:30.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:30.279335+0000) 2020-02-05T21:51:30.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:30.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:30.287312+0000) 2020-02-05T21:51:32.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:32.270+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:32.270480+0000) 2020-02-05T21:51:32.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:32.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:32.669597+0000) 2020-02-05T21:51:34.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:34.262+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.264063+0000) 2020-02-05T21:51:34.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:34.263+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.264916+0000) 2020-02-05T21:51:34.266 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:34.264+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.265688+0000) 2020-02-05T21:51:34.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:34.264+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.265527+0000) 2020-02-05T21:51:34.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:34.265+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.266235+0000) 2020-02-05T21:51:34.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:34.265+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.266845+0000) 2020-02-05T21:51:34.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:34.268+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.269411+0000) 2020-02-05T21:51:34.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:34.268+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.269907+0000) 2020-02-05T21:51:34.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:34.268+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.269920+0000) 2020-02-05T21:51:34.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:34.269+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.271018+0000) 2020-02-05T21:51:34.273 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:34.270+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.271661+0000) 2020-02-05T21:51:34.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:34.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.270662+0000) 2020-02-05T21:51:34.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:34.271+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.272768+0000) 2020-02-05T21:51:34.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:34.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:34.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:34.275+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:34.276313+0000) 2020-02-05T21:51:35.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:35.882+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:35.882873+0000) 2020-02-05T21:51:35.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:35.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:35.887737+0000) 2020-02-05T21:51:37.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:37.027+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:37.028049+0000) 2020-02-05T21:51:37.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:37.027+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:37.028030+0000) 2020-02-05T21:51:37.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:37.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:37.035713+0000) 2020-02-05T21:51:38.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:38.345+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:38.345731+0000) 2020-02-05T21:51:39.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:39.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.283747+0000) 2020-02-05T21:51:39.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:39.283+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.284970+0000) 2020-02-05T21:51:39.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:39.285+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.285789+0000) 2020-02-05T21:51:39.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:39.286+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.286104+0000) 2020-02-05T21:51:39.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:39.286+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.286575+0000) 2020-02-05T21:51:39.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:39.285+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.286848+0000) 2020-02-05T21:51:39.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:39.285+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.286912+0000) 2020-02-05T21:51:39.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:39.286+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.287757+0000) 2020-02-05T21:51:39.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:39.285+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.287283+0000) 2020-02-05T21:51:39.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:39.288+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.288478+0000) 2020-02-05T21:51:39.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:39.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.288991+0000) 2020-02-05T21:51:39.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:39.289+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.289302+0000) 2020-02-05T21:51:39.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:39.288+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.289411+0000) 2020-02-05T21:51:39.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:39.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:39.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:39.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.294509+0000) 2020-02-05T21:51:39.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:39.418+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:39.418398+0000) 2020-02-05T21:51:40.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:40.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:40.026488+0000) 2020-02-05T21:51:40.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:40.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:40.261230+0000) 2020-02-05T21:51:40.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:40.278+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:40.279057+0000) 2020-02-05T21:51:40.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:40.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:40.279609+0000) 2020-02-05T21:51:40.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:40.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:40.287534+0000) 2020-02-05T21:51:42.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:42.269+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:42.270721+0000) 2020-02-05T21:51:42.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:42.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:42.669834+0000) 2020-02-05T21:51:44.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:44.297+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.298694+0000) 2020-02-05T21:51:44.301 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:44.298+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.299815+0000) 2020-02-05T21:51:44.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:44.298+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.299981+0000) 2020-02-05T21:51:44.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:44.301+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.302563+0000) 2020-02-05T21:51:44.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:44.301+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.302817+0000) 2020-02-05T21:51:44.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:44.302+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.303435+0000) 2020-02-05T21:51:44.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:44.302+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.304287+0000) 2020-02-05T21:51:44.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:44.302+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.304221+0000) 2020-02-05T21:51:44.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:44.303+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.305030+0000) 2020-02-05T21:51:44.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:44.303+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.304885+0000) 2020-02-05T21:51:44.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:44.304+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.305665+0000) 2020-02-05T21:51:44.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:44.304+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.306315+0000) 2020-02-05T21:51:44.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:44.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.306429+0000) 2020-02-05T21:51:44.308 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:44.305+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:44.306460+0000) 2020-02-05T21:51:44.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:44.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:45.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:45.882+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:45.883139+0000) 2020-02-05T21:51:45.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:45.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:45.887985+0000) 2020-02-05T21:51:47.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:47.027+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:47.028278+0000) 2020-02-05T21:51:47.030 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:47.027+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:47.028279+0000) 2020-02-05T21:51:47.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:47.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:47.035951+0000) 2020-02-05T21:51:48.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:48.353+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:48.353969+0000) 2020-02-05T21:51:48.863 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:51:48.870 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:51:49.319 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:49.317+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.318356+0000) 2020-02-05T21:51:49.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:49.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.320126+0000) 2020-02-05T21:51:49.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:49.319+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.320782+0000) 2020-02-05T21:51:49.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:49.319+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.320874+0000) 2020-02-05T21:51:49.323 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:49.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.321271+0000) 2020-02-05T21:51:49.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:49.320+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.321760+0000) 2020-02-05T21:51:49.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:49.320+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.322320+0000) 2020-02-05T21:51:49.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:49.320+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.322213+0000) 2020-02-05T21:51:49.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:49.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.324382+0000) 2020-02-05T21:51:49.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:49.323+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.324797+0000) 2020-02-05T21:51:49.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:49.324+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.325150+0000) 2020-02-05T21:51:49.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:49.324+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.325133+0000) 2020-02-05T21:51:49.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:49.324+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.325611+0000) 2020-02-05T21:51:49.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:49.327+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.329058+0000) 2020-02-05T21:51:49.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:49.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:49.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:49.417+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:49.418650+0000) 2020-02-05T21:51:50.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:50.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:50.026693+0000) 2020-02-05T21:51:50.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:50.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:50.261472+0000) 2020-02-05T21:51:50.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:50.278+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:50.279301+0000) 2020-02-05T21:51:50.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:50.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:50.279840+0000) 2020-02-05T21:51:50.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:50.286+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:50.287718+0000) 2020-02-05T21:51:52.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:52.269+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:52.270957+0000) 2020-02-05T21:51:52.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:52.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:52.670104+0000) 2020-02-05T21:51:54.338 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:54.335+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.336736+0000) 2020-02-05T21:51:54.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:54.336+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.337263+0000) 2020-02-05T21:51:54.339 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:54.336+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.337742+0000) 2020-02-05T21:51:54.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:54.337+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.338434+0000) 2020-02-05T21:51:54.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:54.337+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.339099+0000) 2020-02-05T21:51:54.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:54.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.339480+0000) 2020-02-05T21:51:54.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:54.338+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.339862+0000) 2020-02-05T21:51:54.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:54.338+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.340381+0000) 2020-02-05T21:51:54.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:54.338+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.340184+0000) 2020-02-05T21:51:54.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:54.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.340576+0000) 2020-02-05T21:51:54.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:54.340+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.341642+0000) 2020-02-05T21:51:54.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:54.340+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.341931+0000) 2020-02-05T21:51:54.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:54.341+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.342545+0000) 2020-02-05T21:51:54.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:54.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:54.346652+0000) 2020-02-05T21:51:54.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:54.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:55.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:55.882+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:55.883362+0000) 2020-02-05T21:51:55.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:55.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:55.888270+0000) 2020-02-05T21:51:57.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:57.027+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:57.028573+0000) 2020-02-05T21:51:57.030 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:57.027+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:57.028578+0000) 2020-02-05T21:51:57.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:57.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:57.036235+0000) 2020-02-05T21:51:58.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:58.361+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:58.362034+0000) 2020-02-05T21:51:59.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:51:59.356+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.357257+0000) 2020-02-05T21:51:59.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:51:59.356+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.358116+0000) 2020-02-05T21:51:59.359 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:51:59.356+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.358134+0000) 2020-02-05T21:51:59.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:51:59.357+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.358349+0000) 2020-02-05T21:51:59.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:51:59.357+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.358944+0000) 2020-02-05T21:51:59.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:59.358+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.359248+0000) 2020-02-05T21:51:59.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:51:59.358+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.360079+0000) 2020-02-05T21:51:59.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:59.358+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.359650+0000) 2020-02-05T21:51:59.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:51:59.359+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.360612+0000) 2020-02-05T21:51:59.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:51:59.359+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.361054+0000) 2020-02-05T21:51:59.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:51:59.360+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.361313+0000) 2020-02-05T21:51:59.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:51:59.360+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.361695+0000) 2020-02-05T21:51:59.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:51:59.360+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.362045+0000) 2020-02-05T21:51:59.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:59.364+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.365336+0000) 2020-02-05T21:51:59.370 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:51:59.367+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:51:59.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:51:59.418+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:51:59.418859+0000) 2020-02-05T21:52:00.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:00.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:00.026897+0000) 2020-02-05T21:52:00.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:00.260+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:00.261725+0000) 2020-02-05T21:52:00.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:00.278+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:00.279507+0000) 2020-02-05T21:52:00.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:00.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:00.280038+0000) 2020-02-05T21:52:00.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:00.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:00.287924+0000) 2020-02-05T21:52:02.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:02.270+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:02.271256+0000) 2020-02-05T21:52:02.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:02.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:02.670362+0000) 2020-02-05T21:52:04.376 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:04.373+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.375049+0000) 2020-02-05T21:52:04.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:04.374+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.375782+0000) 2020-02-05T21:52:04.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:04.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.376146+0000) 2020-02-05T21:52:04.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:04.375+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.377104+0000) 2020-02-05T21:52:04.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:04.376+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.378286+0000) 2020-02-05T21:52:04.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:04.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.378572+0000) 2020-02-05T21:52:04.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:04.378+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380050+0000) 2020-02-05T21:52:04.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:04.378+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380168+0000) 2020-02-05T21:52:04.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:04.379+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380992+0000) 2020-02-05T21:52:04.382 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:04.379+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380280+0000) 2020-02-05T21:52:04.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:04.379+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380713+0000) 2020-02-05T21:52:04.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:04.379+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.380840+0000) 2020-02-05T21:52:04.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:04.380+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.381589+0000) 2020-02-05T21:52:04.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:04.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:04.385287+0000) 2020-02-05T21:52:04.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:04.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:05.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:05.882+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:05.883591+0000) 2020-02-05T21:52:05.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:05.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:05.888495+0000) 2020-02-05T21:52:07.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:07.028+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:07.028876+0000) 2020-02-05T21:52:07.030 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:07.028+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:07.028876+0000) 2020-02-05T21:52:07.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:07.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:07.036521+0000) 2020-02-05T21:52:08.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:08.369+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:08.369963+0000) 2020-02-05T21:52:09.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:09.391+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.393286+0000) 2020-02-05T21:52:09.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:09.393+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.394795+0000) 2020-02-05T21:52:09.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:09.394+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.395252+0000) 2020-02-05T21:52:09.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:09.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.395812+0000) 2020-02-05T21:52:09.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:09.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.396223+0000) 2020-02-05T21:52:09.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:09.395+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.396332+0000) 2020-02-05T21:52:09.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:09.395+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.396718+0000) 2020-02-05T21:52:09.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:09.396+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.397264+0000) 2020-02-05T21:52:09.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:09.396+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.397769+0000) 2020-02-05T21:52:09.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:09.396+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.398270+0000) 2020-02-05T21:52:09.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:09.397+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.398977+0000) 2020-02-05T21:52:09.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:09.398+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.399547+0000) 2020-02-05T21:52:09.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:09.397+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.399197+0000) 2020-02-05T21:52:09.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:09.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:09.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:09.405+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.406518+0000) 2020-02-05T21:52:09.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:09.418+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:09.419066+0000) 2020-02-05T21:52:10.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:10.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:10.027104+0000) 2020-02-05T21:52:10.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:10.261+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:10.261954+0000) 2020-02-05T21:52:10.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:10.279+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:10.279708+0000) 2020-02-05T21:52:10.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:10.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:10.280233+0000) 2020-02-05T21:52:10.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:10.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:10.288136+0000) 2020-02-05T21:52:12.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:12.270+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:12.271494+0000) 2020-02-05T21:52:12.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:12.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:12.670589+0000) 2020-02-05T21:52:14.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:14.408+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.410322+0000) 2020-02-05T21:52:14.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:14.409+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.411005+0000) 2020-02-05T21:52:14.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:14.410+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.412089+0000) 2020-02-05T21:52:14.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:14.411+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.412250+0000) 2020-02-05T21:52:14.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:14.411+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.412729+0000) 2020-02-05T21:52:14.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:14.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.413401+0000) 2020-02-05T21:52:14.415 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:14.412+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.413985+0000) 2020-02-05T21:52:14.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:14.413+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.415317+0000) 2020-02-05T21:52:14.417 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:14.414+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.415509+0000) 2020-02-05T21:52:14.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:14.414+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.415784+0000) 2020-02-05T21:52:14.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:14.415+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.416295+0000) 2020-02-05T21:52:14.418 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:14.415+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.416359+0000) 2020-02-05T21:52:14.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:14.415+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.416959+0000) 2020-02-05T21:52:14.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:14.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:14.420219+0000) 2020-02-05T21:52:14.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:14.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:15.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:15.883+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:15.883828+0000) 2020-02-05T21:52:15.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:15.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:15.888758+0000) 2020-02-05T21:52:17.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:17.028+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:17.029177+0000) 2020-02-05T21:52:17.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:17.028+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:17.029177+0000) 2020-02-05T21:52:17.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:17.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:17.036791+0000) 2020-02-05T21:52:18.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:18.377+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:18.378313+0000) 2020-02-05T21:52:19.018 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:52:19.024 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:52:19.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:19.418+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.419342+0000) 2020-02-05T21:52:19.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:19.427+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.428716+0000) 2020-02-05T21:52:19.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:19.427+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.429018+0000) 2020-02-05T21:52:19.431 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:19.429+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.430523+0000) 2020-02-05T21:52:19.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:19.429+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.430899+0000) 2020-02-05T21:52:19.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:19.430+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.431463+0000) 2020-02-05T21:52:19.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:19.430+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.431375+0000) 2020-02-05T21:52:19.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:19.430+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.431957+0000) 2020-02-05T21:52:19.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:19.430+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.431376+0000) 2020-02-05T21:52:19.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:19.430+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.431734+0000) 2020-02-05T21:52:19.434 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:19.431+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.432119+0000) 2020-02-05T21:52:19.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:19.431+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.432577+0000) 2020-02-05T21:52:19.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:19.431+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.432683+0000) 2020-02-05T21:52:19.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:19.431+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.432876+0000) 2020-02-05T21:52:19.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:19.437+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:19.438748+0000) 2020-02-05T21:52:19.441 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:19.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:20.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:20.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:20.027334+0000) 2020-02-05T21:52:20.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:20.261+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:20.262188+0000) 2020-02-05T21:52:20.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:20.279+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:20.279922+0000) 2020-02-05T21:52:20.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:20.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:20.280489+0000) 2020-02-05T21:52:20.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:20.288+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:20.288351+0000) 2020-02-05T21:52:22.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:22.271+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:22.271743+0000) 2020-02-05T21:52:22.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:22.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:22.670843+0000) 2020-02-05T21:52:24.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:24.444+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.446024+0000) 2020-02-05T21:52:24.448 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:24.445+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.446894+0000) 2020-02-05T21:52:24.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:24.446+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.447610+0000) 2020-02-05T21:52:24.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:24.446+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.447693+0000) 2020-02-05T21:52:24.450 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:24.447+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.448325+0000) 2020-02-05T21:52:24.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:24.446+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.448270+0000) 2020-02-05T21:52:24.450 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:24.447+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.448711+0000) 2020-02-05T21:52:24.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:24.448+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.449813+0000) 2020-02-05T21:52:24.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:24.448+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.449976+0000) 2020-02-05T21:52:24.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:24.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.450607+0000) 2020-02-05T21:52:24.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:24.449+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.450535+0000) 2020-02-05T21:52:24.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:24.449+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.451006+0000) 2020-02-05T21:52:24.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:24.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.451631+0000) 2020-02-05T21:52:24.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:24.454+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:24.455914+0000) 2020-02-05T21:52:24.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:24.456+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:25.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:25.883+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:25.884099+0000) 2020-02-05T21:52:25.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:25.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:25.889001+0000) 2020-02-05T21:52:27.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:27.029+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:27.029482+0000) 2020-02-05T21:52:27.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:27.029+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:27.029482+0000) 2020-02-05T21:52:27.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:27.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:27.037026+0000) 2020-02-05T21:52:28.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:28.386+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:28.386401+0000) 2020-02-05T21:52:29.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:29.419+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.419622+0000) 2020-02-05T21:52:29.464 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:29.461+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.462997+0000) 2020-02-05T21:52:29.465 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:29.463+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.464572+0000) 2020-02-05T21:52:29.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:29.464+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.465029+0000) 2020-02-05T21:52:29.466 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:29.465+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.465562+0000) 2020-02-05T21:52:29.467 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:29.464+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.465832+0000) 2020-02-05T21:52:29.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:29.465+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.465925+0000) 2020-02-05T21:52:29.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:29.465+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.466333+0000) 2020-02-05T21:52:29.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:29.466+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.467172+0000) 2020-02-05T21:52:29.469 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:29.466+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.466817+0000) 2020-02-05T21:52:29.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:29.467+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.467711+0000) 2020-02-05T21:52:29.470 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:29.466+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.467744+0000) 2020-02-05T21:52:29.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:29.467+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.468522+0000) 2020-02-05T21:52:29.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:29.467+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.469036+0000) 2020-02-05T21:52:29.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:29.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:29.474099+0000) 2020-02-05T21:52:29.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:29.475+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:30.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:30.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:30.027541+0000) 2020-02-05T21:52:30.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:30.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:30.262419+0000) 2020-02-05T21:52:30.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:30.280+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:30.280133+0000) 2020-02-05T21:52:30.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:30.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:30.280662+0000) 2020-02-05T21:52:30.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:30.288+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:30.288560+0000) 2020-02-05T21:52:32.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:32.271+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:32.271962+0000) 2020-02-05T21:52:32.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:32.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:32.671112+0000) 2020-02-05T21:52:34.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:34.481+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.481424+0000) 2020-02-05T21:52:34.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:34.481+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.481745+0000) 2020-02-05T21:52:34.483 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:34.482+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.482561+0000) 2020-02-05T21:52:34.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:34.483+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.483167+0000) 2020-02-05T21:52:34.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:34.483+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.483605+0000) 2020-02-05T21:52:34.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:34.483+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.483780+0000) 2020-02-05T21:52:34.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:34.483+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.484231+0000) 2020-02-05T21:52:34.486 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:34.485+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.485281+0000) 2020-02-05T21:52:34.487 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:34.485+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.486018+0000) 2020-02-05T21:52:34.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:34.486+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.486674+0000) 2020-02-05T21:52:34.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:34.486+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.486789+0000) 2020-02-05T21:52:34.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:34.486+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.486816+0000) 2020-02-05T21:52:34.489 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:34.486+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.486996+0000) 2020-02-05T21:52:34.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:34.489+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:34.490021+0000) 2020-02-05T21:52:34.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:34.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:35.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:35.883+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:35.884352+0000) 2020-02-05T21:52:35.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:35.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:35.889233+0000) 2020-02-05T21:52:37.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:37.029+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:37.029770+0000) 2020-02-05T21:52:37.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:37.029+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:37.029770+0000) 2020-02-05T21:52:37.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:37.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:37.037286+0000) 2020-02-05T21:52:38.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:38.393+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:38.394603+0000) 2020-02-05T21:52:39.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:39.418+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.419888+0000) 2020-02-05T21:52:39.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:39.497+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.498566+0000) 2020-02-05T21:52:39.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:39.499+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.499682+0000) 2020-02-05T21:52:39.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:39.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.499968+0000) 2020-02-05T21:52:39.501 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:39.499+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.500190+0000) 2020-02-05T21:52:39.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:39.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.500757+0000) 2020-02-05T21:52:39.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:39.501+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.502386+0000) 2020-02-05T21:52:39.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:39.501+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.502405+0000) 2020-02-05T21:52:39.504 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:39.501+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.502756+0000) 2020-02-05T21:52:39.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:39.502+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.503145+0000) 2020-02-05T21:52:39.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:39.503+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.503615+0000) 2020-02-05T21:52:39.505 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:39.502+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.503719+0000) 2020-02-05T21:52:39.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:39.504+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.504531+0000) 2020-02-05T21:52:39.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:39.504+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.505126+0000) 2020-02-05T21:52:39.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:39.509+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:39.510401+0000) 2020-02-05T21:52:39.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:39.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:40.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:40.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:40.027748+0000) 2020-02-05T21:52:40.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:40.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:40.262675+0000) 2020-02-05T21:52:40.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:40.279+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:40.280364+0000) 2020-02-05T21:52:40.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:40.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:40.280854+0000) 2020-02-05T21:52:40.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:40.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:40.288771+0000) 2020-02-05T21:52:42.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:42.271+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:42.272221+0000) 2020-02-05T21:52:42.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:42.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:42.671372+0000) 2020-02-05T21:52:44.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:44.515+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.516327+0000) 2020-02-05T21:52:44.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:44.515+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.516936+0000) 2020-02-05T21:52:44.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:44.518+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.518413+0000) 2020-02-05T21:52:44.520 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:44.518+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.519348+0000) 2020-02-05T21:52:44.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:44.518+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.519495+0000) 2020-02-05T21:52:44.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:44.518+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.520032+0000) 2020-02-05T21:52:44.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:44.520+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.520880+0000) 2020-02-05T21:52:44.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:44.520+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.521499+0000) 2020-02-05T21:52:44.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:44.520+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.521683+0000) 2020-02-05T21:52:44.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:44.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.522358+0000) 2020-02-05T21:52:44.524 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:44.522+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.523137+0000) 2020-02-05T21:52:44.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:44.523+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.523547+0000) 2020-02-05T21:52:44.525 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:44.523+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.523608+0000) 2020-02-05T21:52:44.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:44.525+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:44.526854+0000) 2020-02-05T21:52:44.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:44.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:45.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:45.883+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:45.884589+0000) 2020-02-05T21:52:45.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:45.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:45.889491+0000) 2020-02-05T21:52:47.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:47.029+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:47.030045+0000) 2020-02-05T21:52:47.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:47.029+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:47.030054+0000) 2020-02-05T21:52:47.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:47.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:47.037551+0000) 2020-02-05T21:52:48.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:48.403+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:48.404374+0000) 2020-02-05T21:52:49.151 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:52:49.157 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:52:49.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:49.419+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.420162+0000) 2020-02-05T21:52:49.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:49.535+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.536185+0000) 2020-02-05T21:52:49.538 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:49.535+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.536929+0000) 2020-02-05T21:52:49.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:49.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.537402+0000) 2020-02-05T21:52:49.539 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:49.536+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.537886+0000) 2020-02-05T21:52:49.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:49.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.538643+0000) 2020-02-05T21:52:49.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:49.538+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.539049+0000) 2020-02-05T21:52:49.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:49.538+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.539776+0000) 2020-02-05T21:52:49.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:49.539+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.539724+0000) 2020-02-05T21:52:49.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:49.539+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.540264+0000) 2020-02-05T21:52:49.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:49.540+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.541288+0000) 2020-02-05T21:52:49.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:49.541+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.541648+0000) 2020-02-05T21:52:49.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:49.542+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.542653+0000) 2020-02-05T21:52:49.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:49.545+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.546492+0000) 2020-02-05T21:52:49.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:49.545+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:49.546737+0000) 2020-02-05T21:52:49.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:49.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:50.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:50.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:50.027967+0000) 2020-02-05T21:52:50.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:50.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:50.262891+0000) 2020-02-05T21:52:50.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:50.279+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:50.280600+0000) 2020-02-05T21:52:50.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:50.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:50.281086+0000) 2020-02-05T21:52:50.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:50.287+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:50.288971+0000) 2020-02-05T21:52:52.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:52.272+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:52.272484+0000) 2020-02-05T21:52:52.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:52.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:52.671604+0000) 2020-02-05T21:52:54.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:54.554+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.555849+0000) 2020-02-05T21:52:54.558 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:54.555+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.556375+0000) 2020-02-05T21:52:54.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:54.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.556907+0000) 2020-02-05T21:52:54.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:54.557+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.557496+0000) 2020-02-05T21:52:54.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:54.556+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.557394+0000) 2020-02-05T21:52:54.560 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:54.557+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.557989+0000) 2020-02-05T21:52:54.560 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:54.557+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.558590+0000) 2020-02-05T21:52:54.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:54.558+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.559766+0000) 2020-02-05T21:52:54.561 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:54.560+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.560449+0000) 2020-02-05T21:52:54.562 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:54.560+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.560534+0000) 2020-02-05T21:52:54.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:54.560+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.561312+0000) 2020-02-05T21:52:54.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:54.560+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.561383+0000) 2020-02-05T21:52:54.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:54.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.561842+0000) 2020-02-05T21:52:54.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:54.564+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:54.565148+0000) 2020-02-05T21:52:54.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:54.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:52:55.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:55.884+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:55.884815+0000) 2020-02-05T21:52:55.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:55.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:55.889748+0000) 2020-02-05T21:52:57.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:57.030+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:57.030328+0000) 2020-02-05T21:52:57.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:57.030+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:57.030321+0000) 2020-02-05T21:52:57.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:57.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:57.037780+0000) 2020-02-05T21:52:58.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:58.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:58.412540+0000) 2020-02-05T21:52:59.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:59.419+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.420397+0000) 2020-02-05T21:52:59.575 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:52:59.572+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.573812+0000) 2020-02-05T21:52:59.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:52:59.574+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.574903+0000) 2020-02-05T21:52:59.576 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:52:59.574+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.575143+0000) 2020-02-05T21:52:59.577 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:52:59.574+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.575748+0000) 2020-02-05T21:52:59.577 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:52:59.574+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.575820+0000) 2020-02-05T21:52:59.578 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:52:59.576+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.576494+0000) 2020-02-05T21:52:59.578 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:52:59.576+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.577188+0000) 2020-02-05T21:52:59.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:59.575+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.576789+0000) 2020-02-05T21:52:59.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:59.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.577314+0000) 2020-02-05T21:52:59.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:52:59.577+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.578172+0000) 2020-02-05T21:52:59.579 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:52:59.576+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.577622+0000) 2020-02-05T21:52:59.580 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:52:59.578+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.578689+0000) 2020-02-05T21:52:59.580 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:52:59.578+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.579415+0000) 2020-02-05T21:52:59.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:52:59.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:52:59.584651+0000) 2020-02-05T21:52:59.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:52:59.585+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:00.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:00.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:00.028193+0000) 2020-02-05T21:53:00.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:00.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:00.263098+0000) 2020-02-05T21:53:00.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:00.280+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:00.280821+0000) 2020-02-05T21:53:00.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:00.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:00.281301+0000) 2020-02-05T21:53:00.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:00.288+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:00.289199+0000) 2020-02-05T21:53:02.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:02.272+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:02.272726+0000) 2020-02-05T21:53:02.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:02.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:02.671830+0000) 2020-02-05T21:53:04.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:04.589+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.590443+0000) 2020-02-05T21:53:04.592 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:04.590+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.591283+0000) 2020-02-05T21:53:04.593 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:04.590+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.592191+0000) 2020-02-05T21:53:04.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:04.591+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.593013+0000) 2020-02-05T21:53:04.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:04.593+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.594310+0000) 2020-02-05T21:53:04.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:04.593+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.595092+0000) 2020-02-05T21:53:04.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:04.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.595748+0000) 2020-02-05T21:53:04.597 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:04.594+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.595858+0000) 2020-02-05T21:53:04.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:04.594+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.596341+0000) 2020-02-05T21:53:04.598 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:04.595+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.596716+0000) 2020-02-05T21:53:04.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:04.595+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.596736+0000) 2020-02-05T21:53:04.599 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:04.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.597182+0000) 2020-02-05T21:53:04.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:04.595+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.596984+0000) 2020-02-05T21:53:04.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:04.601+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:04.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:04.598+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:04.599446+0000) 2020-02-05T21:53:05.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:05.884+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:05.885047+0000) 2020-02-05T21:53:05.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:05.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:05.890008+0000) 2020-02-05T21:53:07.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:07.029+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:07.030575+0000) 2020-02-05T21:53:07.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:07.029+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:07.030574+0000) 2020-02-05T21:53:07.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:07.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:07.038011+0000) 2020-02-05T21:53:08.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:08.420+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:08.421277+0000) 2020-02-05T21:53:09.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:09.419+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.420611+0000) 2020-02-05T21:53:09.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:09.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.608251+0000) 2020-02-05T21:53:09.612 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:09.608+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.610104+0000) 2020-02-05T21:53:09.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:09.608+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.610336+0000) 2020-02-05T21:53:09.613 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:09.609+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.611018+0000) 2020-02-05T21:53:09.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:09.610+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.611071+0000) 2020-02-05T21:53:09.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:09.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.611510+0000) 2020-02-05T21:53:09.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:09.610+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.611721+0000) 2020-02-05T21:53:09.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:09.611+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.612372+0000) 2020-02-05T21:53:09.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:09.612+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.613658+0000) 2020-02-05T21:53:09.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:09.613+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.614482+0000) 2020-02-05T21:53:09.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:09.613+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.614476+0000) 2020-02-05T21:53:09.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:09.613+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.614880+0000) 2020-02-05T21:53:09.617 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:09.613+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.614837+0000) 2020-02-05T21:53:09.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:09.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 98995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:09.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:09.617+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:09.618897+0000) 2020-02-05T21:53:10.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:10.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:10.028396+0000) 2020-02-05T21:53:10.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:10.261+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:10.263316+0000) 2020-02-05T21:53:10.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:10.280+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:10.281026+0000) 2020-02-05T21:53:10.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:10.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:10.281506+0000) 2020-02-05T21:53:10.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:10.288+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:10.289403+0000) 2020-02-05T21:53:12.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:12.272+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:12.272980+0000) 2020-02-05T21:53:12.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:12.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:12.672066+0000) 2020-02-05T21:53:14.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:14.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.627489+0000) 2020-02-05T21:53:14.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:14.626+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.627650+0000) 2020-02-05T21:53:14.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:14.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.628771+0000) 2020-02-05T21:53:14.630 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:14.627+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.629095+0000) 2020-02-05T21:53:14.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:14.628+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.629886+0000) 2020-02-05T21:53:14.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:14.630+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.631416+0000) 2020-02-05T21:53:14.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:14.630+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.631951+0000) 2020-02-05T21:53:14.633 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:14.630+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.632039+0000) 2020-02-05T21:53:14.634 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:14.631+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.633003+0000) 2020-02-05T21:53:14.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:14.631+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.633159+0000) 2020-02-05T21:53:14.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:14.632+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.633598+0000) 2020-02-05T21:53:14.635 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:14.632+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.633893+0000) 2020-02-05T21:53:14.635 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:14.632+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.633987+0000) 2020-02-05T21:53:14.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:14.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:14.639029+0000) 2020-02-05T21:53:14.640 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:14.638+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99037 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:15.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:15.884+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:15.885312+0000) 2020-02-05T21:53:15.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:15.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:15.890270+0000) 2020-02-05T21:53:17.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:17.029+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:17.030841+0000) 2020-02-05T21:53:17.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:17.029+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:17.030835+0000) 2020-02-05T21:53:17.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:17.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:17.038286+0000) 2020-02-05T21:53:18.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:18.428+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:18.429443+0000) 2020-02-05T21:53:19.286 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:53:19.292 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:53:19.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:19.419+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.420867+0000) 2020-02-05T21:53:19.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:19.645+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.646898+0000) 2020-02-05T21:53:19.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:19.647+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.648512+0000) 2020-02-05T21:53:19.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:19.647+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.649014+0000) 2020-02-05T21:53:19.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:19.648+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.650152+0000) 2020-02-05T21:53:19.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:19.648+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.649696+0000) 2020-02-05T21:53:19.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:19.648+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.650352+0000) 2020-02-05T21:53:19.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:19.655+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99078 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:19.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:19.645+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.646359+0000) 2020-02-05T21:53:19.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:19.646+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.647265+0000) 2020-02-05T21:53:19.667 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:19.646+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.647569+0000) 2020-02-05T21:53:19.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:19.646+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.647934+0000) 2020-02-05T21:53:19.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:19.648+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.649564+0000) 2020-02-05T21:53:19.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:19.654+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.655285+0000) 2020-02-05T21:53:19.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:19.647+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.648565+0000) 2020-02-05T21:53:19.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:19.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:19.649796+0000) 2020-02-05T21:53:20.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:20.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:20.028614+0000) 2020-02-05T21:53:20.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:20.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:20.263549+0000) 2020-02-05T21:53:20.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:20.281+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:20.281257+0000) 2020-02-05T21:53:20.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:20.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:20.281719+0000) 2020-02-05T21:53:20.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:20.289+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:20.289606+0000) 2020-02-05T21:53:22.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:22.272+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:22.273242+0000) 2020-02-05T21:53:22.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:22.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:22.672365+0000) 2020-02-05T21:53:24.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:24.659+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.660488+0000) 2020-02-05T21:53:24.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:24.662+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.664002+0000) 2020-02-05T21:53:24.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:24.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.664223+0000) 2020-02-05T21:53:24.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:24.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.664523+0000) 2020-02-05T21:53:24.666 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:24.663+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.664930+0000) 2020-02-05T21:53:24.667 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:24.665+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.665870+0000) 2020-02-05T21:53:24.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:24.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666161+0000) 2020-02-05T21:53:24.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:24.665+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666559+0000) 2020-02-05T21:53:24.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:24.665+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666843+0000) 2020-02-05T21:53:24.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:24.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666398+0000) 2020-02-05T21:53:24.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:24.665+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666913+0000) 2020-02-05T21:53:24.669 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:24.665+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.666896+0000) 2020-02-05T21:53:24.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:24.666+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.667577+0000) 2020-02-05T21:53:24.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:24.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:24.672506+0000) 2020-02-05T21:53:24.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:24.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:25.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:25.885+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:25.885584+0000) 2020-02-05T21:53:25.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:25.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:25.890533+0000) 2020-02-05T21:53:27.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:27.030+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:27.031086+0000) 2020-02-05T21:53:27.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:27.030+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:27.031076+0000) 2020-02-05T21:53:27.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:27.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:27.038522+0000) 2020-02-05T21:53:28.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:28.437+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:28.437801+0000) 2020-02-05T21:53:29.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:29.421+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.421136+0000) 2020-02-05T21:53:29.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:29.679+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.680492+0000) 2020-02-05T21:53:29.683 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:29.680+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.681837+0000) 2020-02-05T21:53:29.683 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:29.680+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.681640+0000) 2020-02-05T21:53:29.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:29.681+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.682277+0000) 2020-02-05T21:53:29.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:29.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.683013+0000) 2020-02-05T21:53:29.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:29.682+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.683385+0000) 2020-02-05T21:53:29.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:29.682+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.683808+0000) 2020-02-05T21:53:29.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:29.682+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.684199+0000) 2020-02-05T21:53:29.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:29.683+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.684530+0000) 2020-02-05T21:53:29.686 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:29.683+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.684461+0000) 2020-02-05T21:53:29.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:29.683+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.684791+0000) 2020-02-05T21:53:29.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:29.684+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.686264+0000) 2020-02-05T21:53:29.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:29.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.685547+0000) 2020-02-05T21:53:29.691 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:29.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:29.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:29.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:29.692431+0000) 2020-02-05T21:53:30.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:30.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:30.028834+0000) 2020-02-05T21:53:30.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:30.262+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:30.263763+0000) 2020-02-05T21:53:30.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:30.280+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:30.281413+0000) 2020-02-05T21:53:30.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:30.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:30.281932+0000) 2020-02-05T21:53:30.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:30.288+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:30.289820+0000) 2020-02-05T21:53:32.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:32.272+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:32.273509+0000) 2020-02-05T21:53:32.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:32.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:32.672603+0000) 2020-02-05T21:53:34.699 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:34.694+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.695626+0000) 2020-02-05T21:53:34.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:34.695+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.696253+0000) 2020-02-05T21:53:34.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:34.695+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.696761+0000) 2020-02-05T21:53:34.700 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:34.695+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.696638+0000) 2020-02-05T21:53:34.700 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:34.696+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.697827+0000) 2020-02-05T21:53:34.702 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:34.698+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.699703+0000) 2020-02-05T21:53:34.702 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:34.698+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.699884+0000) 2020-02-05T21:53:34.702 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:34.698+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.699987+0000) 2020-02-05T21:53:34.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:34.699+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.700455+0000) 2020-02-05T21:53:34.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:34.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.698153+0000) 2020-02-05T21:53:34.703 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:34.697+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.698250+0000) 2020-02-05T21:53:34.703 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:34.697+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.698763+0000) 2020-02-05T21:53:34.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:34.703+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.704791+0000) 2020-02-05T21:53:34.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:34.703+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:34.704987+0000) 2020-02-05T21:53:34.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:34.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:35.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:35.884+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:35.885875+0000) 2020-02-05T21:53:35.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:35.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:35.890773+0000) 2020-02-05T21:53:37.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:37.030+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:37.031339+0000) 2020-02-05T21:53:37.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:37.030+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:37.031322+0000) 2020-02-05T21:53:37.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:37.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:37.038755+0000) 2020-02-05T21:53:38.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:38.445+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:38.446548+0000) 2020-02-05T21:53:39.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:39.420+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.421374+0000) 2020-02-05T21:53:39.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:39.711+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.712601+0000) 2020-02-05T21:53:39.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:39.711+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.713170+0000) 2020-02-05T21:53:39.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:39.713+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.714713+0000) 2020-02-05T21:53:39.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:39.713+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.715381+0000) 2020-02-05T21:53:39.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:39.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.715529+0000) 2020-02-05T21:53:39.717 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:39.715+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.716122+0000) 2020-02-05T21:53:39.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:39.715+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.716285+0000) 2020-02-05T21:53:39.718 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:39.715+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.716310+0000) 2020-02-05T21:53:39.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:39.715+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.716515+0000) 2020-02-05T21:53:39.719 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:39.715+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.717163+0000) 2020-02-05T21:53:39.720 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:39.715+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.717251+0000) 2020-02-05T21:53:39.720 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:39.716+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.718258+0000) 2020-02-05T21:53:39.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:39.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.717462+0000) 2020-02-05T21:53:39.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:39.720+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:39.721863+0000) 2020-02-05T21:53:39.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:39.723+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:40.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:40.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:40.029117+0000) 2020-02-05T21:53:40.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:40.263+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:40.263988+0000) 2020-02-05T21:53:40.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:40.280+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:40.281622+0000) 2020-02-05T21:53:40.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:40.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:40.282150+0000) 2020-02-05T21:53:40.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:40.289+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:40.290030+0000) 2020-02-05T21:53:42.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:42.273+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:42.273740+0000) 2020-02-05T21:53:42.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:42.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:42.672837+0000) 2020-02-05T21:53:44.730 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:44.727+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.728617+0000) 2020-02-05T21:53:44.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:44.727+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.729258+0000) 2020-02-05T21:53:44.730 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:44.727+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.729380+0000) 2020-02-05T21:53:44.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:44.728+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.729695+0000) 2020-02-05T21:53:44.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:44.728+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.730091+0000) 2020-02-05T21:53:44.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:44.729+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.730878+0000) 2020-02-05T21:53:44.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:44.730+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.731224+0000) 2020-02-05T21:53:44.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:44.730+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.731522+0000) 2020-02-05T21:53:44.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:44.730+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.732016+0000) 2020-02-05T21:53:44.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:44.732+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.733433+0000) 2020-02-05T21:53:44.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:44.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.734369+0000) 2020-02-05T21:53:44.736 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:44.732+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.733952+0000) 2020-02-05T21:53:44.736 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:44.733+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.734557+0000) 2020-02-05T21:53:44.745 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:44.742+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:44.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:44.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:44.745731+0000) 2020-02-05T21:53:45.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:45.885+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:45.886137+0000) 2020-02-05T21:53:45.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:45.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:45.891043+0000) 2020-02-05T21:53:47.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:47.030+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:47.031600+0000) 2020-02-05T21:53:47.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:47.030+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:47.031537+0000) 2020-02-05T21:53:47.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:47.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:47.038983+0000) 2020-02-05T21:53:48.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:48.453+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:48.454591+0000) 2020-02-05T21:53:49.416 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:53:49.423 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:53:49.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:49.420+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.421630+0000) 2020-02-05T21:53:49.751 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:49.748+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.749899+0000) 2020-02-05T21:53:49.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:49.750+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.751895+0000) 2020-02-05T21:53:49.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:49.750+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.752012+0000) 2020-02-05T21:53:49.753 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:49.750+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.752366+0000) 2020-02-05T21:53:49.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:49.751+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.752573+0000) 2020-02-05T21:53:49.754 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:49.752+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.753165+0000) 2020-02-05T21:53:49.755 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:49.752+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.753647+0000) 2020-02-05T21:53:49.755 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:49.752+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.753812+0000) 2020-02-05T21:53:49.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:49.753+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.754396+0000) 2020-02-05T21:53:49.757 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:49.753+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.754600+0000) 2020-02-05T21:53:49.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:49.753+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.754905+0000) 2020-02-05T21:53:49.757 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:49.754+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.755601+0000) 2020-02-05T21:53:49.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:49.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.755304+0000) 2020-02-05T21:53:49.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:49.759+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:49.760205+0000) 2020-02-05T21:53:49.763 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:49.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:50.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:50.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:50.029286+0000) 2020-02-05T21:53:50.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:50.263+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:50.264222+0000) 2020-02-05T21:53:50.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:50.281+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:50.281829+0000) 2020-02-05T21:53:50.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:50.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:50.282376+0000) 2020-02-05T21:53:50.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:50.289+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:50.290328+0000) 2020-02-05T21:53:52.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:52.273+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:52.273981+0000) 2020-02-05T21:53:52.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:52.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:52.673122+0000) 2020-02-05T21:53:54.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:54.766+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.767741+0000) 2020-02-05T21:53:54.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:54.768+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.769588+0000) 2020-02-05T21:53:54.771 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:54.767+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.769295+0000) 2020-02-05T21:53:54.772 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:54.770+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.770894+0000) 2020-02-05T21:53:54.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:54.770+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.771629+0000) 2020-02-05T21:53:54.773 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:54.772+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.772506+0000) 2020-02-05T21:53:54.774 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:54.771+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.772894+0000) 2020-02-05T21:53:54.774 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:54.772+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.772784+0000) 2020-02-05T21:53:54.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:54.772+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.773484+0000) 2020-02-05T21:53:54.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:54.772+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.773647+0000) 2020-02-05T21:53:54.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:54.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.774346+0000) 2020-02-05T21:53:54.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:54.773+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.774513+0000) 2020-02-05T21:53:54.776 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:54.773+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.774924+0000) 2020-02-05T21:53:54.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:54.775+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:54.776657+0000) 2020-02-05T21:53:54.782 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:54.780+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99361 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:53:55.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:55.885+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:55.886375+0000) 2020-02-05T21:53:55.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:55.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:55.891318+0000) 2020-02-05T21:53:57.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:57.031+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:57.031834+0000) 2020-02-05T21:53:57.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:57.031+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:57.031829+0000) 2020-02-05T21:53:57.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:57.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:57.039206+0000) 2020-02-05T21:53:58.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:58.462+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:58.462875+0000) 2020-02-05T21:53:59.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:59.421+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.421866+0000) 2020-02-05T21:53:59.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:53:59.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.785912+0000) 2020-02-05T21:53:59.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:53:59.786+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.787531+0000) 2020-02-05T21:53:59.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:53:59.787+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.788544+0000) 2020-02-05T21:53:59.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:53:59.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.789073+0000) 2020-02-05T21:53:59.790 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:53:59.789+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.789536+0000) 2020-02-05T21:53:59.791 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:53:59.788+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.789845+0000) 2020-02-05T21:53:59.791 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:53:59.789+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.790015+0000) 2020-02-05T21:53:59.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:59.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.790481+0000) 2020-02-05T21:53:59.792 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:53:59.790+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.790713+0000) 2020-02-05T21:53:59.792 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:53:59.790+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.791146+0000) 2020-02-05T21:53:59.793 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:53:59.791+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.791564+0000) 2020-02-05T21:53:59.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:59.792+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.793379+0000) 2020-02-05T21:53:59.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:53:59.792+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.793651+0000) 2020-02-05T21:53:59.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:53:59.796+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:53:59.798032+0000) 2020-02-05T21:53:59.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:53:59.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:00.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:00.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:00.029505+0000) 2020-02-05T21:54:00.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:00.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:00.264465+0000) 2020-02-05T21:54:00.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:00.281+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:00.282040+0000) 2020-02-05T21:54:00.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:00.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:00.282638+0000) 2020-02-05T21:54:00.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:00.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:00.290552+0000) 2020-02-05T21:54:02.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:02.273+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:02.274233+0000) 2020-02-05T21:54:02.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:02.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:02.673335+0000) 2020-02-05T21:54:04.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:04.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.805953+0000) 2020-02-05T21:54:04.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:04.805+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.806226+0000) 2020-02-05T21:54:04.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:04.806+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.806925+0000) 2020-02-05T21:54:04.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:04.806+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.807238+0000) 2020-02-05T21:54:04.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:04.806+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.807715+0000) 2020-02-05T21:54:04.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:04.808+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.808473+0000) 2020-02-05T21:54:04.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:04.807+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.808537+0000) 2020-02-05T21:54:04.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:04.808+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.809220+0000) 2020-02-05T21:54:04.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:04.809+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.809569+0000) 2020-02-05T21:54:04.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:04.810+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.810740+0000) 2020-02-05T21:54:04.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:04.810+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.811849+0000) 2020-02-05T21:54:04.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:04.811+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.812291+0000) 2020-02-05T21:54:04.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:04.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.811721+0000) 2020-02-05T21:54:04.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:04.815+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:04.816488+0000) 2020-02-05T21:54:04.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:04.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:05.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:05.885+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:05.886606+0000) 2020-02-05T21:54:05.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:05.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:05.891545+0000) 2020-02-05T21:54:07.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:07.031+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:07.032081+0000) 2020-02-05T21:54:07.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:07.031+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:07.032082+0000) 2020-02-05T21:54:07.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:07.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:07.039438+0000) 2020-02-05T21:54:08.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:08.470+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:08.471288+0000) 2020-02-05T21:54:09.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:09.421+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.422137+0000) 2020-02-05T21:54:09.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:09.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.823309+0000) 2020-02-05T21:54:09.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:09.824+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.825147+0000) 2020-02-05T21:54:09.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:09.825+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.826386+0000) 2020-02-05T21:54:09.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:09.826+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.826654+0000) 2020-02-05T21:54:09.829 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:09.825+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.826923+0000) 2020-02-05T21:54:09.829 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:09.827+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.827909+0000) 2020-02-05T21:54:09.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:09.826+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.827996+0000) 2020-02-05T21:54:09.830 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:09.828+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.829147+0000) 2020-02-05T21:54:09.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:09.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.829329+0000) 2020-02-05T21:54:09.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:09.827+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.829034+0000) 2020-02-05T21:54:09.831 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:09.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.829939+0000) 2020-02-05T21:54:09.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:09.828+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.829511+0000) 2020-02-05T21:54:09.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:09.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:09.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:09.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.837794+0000) 2020-02-05T21:54:09.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:09.840+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:09.841120+0000) 2020-02-05T21:54:10.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:10.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:10.029718+0000) 2020-02-05T21:54:10.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:10.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:10.264632+0000) 2020-02-05T21:54:10.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:10.281+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:10.282303+0000) 2020-02-05T21:54:10.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:10.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:10.282838+0000) 2020-02-05T21:54:10.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:10.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:10.290762+0000) 2020-02-05T21:54:12.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:12.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:12.274544+0000) 2020-02-05T21:54:12.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:12.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:12.673593+0000) 2020-02-05T21:54:14.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:14.841+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.842416+0000) 2020-02-05T21:54:14.844 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:14.843+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.843464+0000) 2020-02-05T21:54:14.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:14.842+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.843572+0000) 2020-02-05T21:54:14.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:14.844+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.844770+0000) 2020-02-05T21:54:14.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:14.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.845217+0000) 2020-02-05T21:54:14.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:14.845+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.846307+0000) 2020-02-05T21:54:14.847 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:14.845+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.846149+0000) 2020-02-05T21:54:14.848 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:14.845+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.846843+0000) 2020-02-05T21:54:14.848 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:14.847+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.847503+0000) 2020-02-05T21:54:14.849 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:14.847+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.847588+0000) 2020-02-05T21:54:14.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:14.846+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.847731+0000) 2020-02-05T21:54:14.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:14.847+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.848185+0000) 2020-02-05T21:54:14.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:14.848+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.848597+0000) 2020-02-05T21:54:14.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:14.849+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:14.850962+0000) 2020-02-05T21:54:14.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:14.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:15.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:15.886+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:15.886864+0000) 2020-02-05T21:54:15.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:15.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:15.891798+0000) 2020-02-05T21:54:17.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:17.031+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:17.032331+0000) 2020-02-05T21:54:17.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:17.031+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:17.032345+0000) 2020-02-05T21:54:17.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:17.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:17.039675+0000) 2020-02-05T21:54:18.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:18.479+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:18.479990+0000) 2020-02-05T21:54:19.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:19.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.422389+0000) 2020-02-05T21:54:19.562 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:54:19.569 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:54:19.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:19.859+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.859535+0000) 2020-02-05T21:54:19.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:19.860+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.862032+0000) 2020-02-05T21:54:19.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:19.862+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.862680+0000) 2020-02-05T21:54:19.864 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:19.862+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.863037+0000) 2020-02-05T21:54:19.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:19.863+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.863403+0000) 2020-02-05T21:54:19.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:19.864+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.864346+0000) 2020-02-05T21:54:19.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:19.864+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.864482+0000) 2020-02-05T21:54:19.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:19.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.864683+0000) 2020-02-05T21:54:19.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:19.864+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.865021+0000) 2020-02-05T21:54:19.867 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:19.864+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.865265+0000) 2020-02-05T21:54:19.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:19.865+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.866339+0000) 2020-02-05T21:54:19.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:19.866+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.866609+0000) 2020-02-05T21:54:19.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:19.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.865542+0000) 2020-02-05T21:54:19.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:19.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:19.872447+0000) 2020-02-05T21:54:19.874 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:19.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99561 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:20.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:20.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:20.029948+0000) 2020-02-05T21:54:20.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:20.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:20.264860+0000) 2020-02-05T21:54:20.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:20.282+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:20.282498+0000) 2020-02-05T21:54:20.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:20.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:20.283015+0000) 2020-02-05T21:54:20.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:20.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:20.290973+0000) 2020-02-05T21:54:22.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:22.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:22.274806+0000) 2020-02-05T21:54:22.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:22.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:22.673828+0000) 2020-02-05T21:54:24.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:24.876+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.877487+0000) 2020-02-05T21:54:24.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:24.877+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.878396+0000) 2020-02-05T21:54:24.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:24.877+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.878834+0000) 2020-02-05T21:54:24.880 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:24.878+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.879197+0000) 2020-02-05T21:54:24.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:24.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.880550+0000) 2020-02-05T21:54:24.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:24.880+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.881138+0000) 2020-02-05T21:54:24.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:24.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.881053+0000) 2020-02-05T21:54:24.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:24.880+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.881599+0000) 2020-02-05T21:54:24.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:24.881+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.881717+0000) 2020-02-05T21:54:24.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:24.881+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.882148+0000) 2020-02-05T21:54:24.884 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:24.881+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.882576+0000) 2020-02-05T21:54:24.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:24.882+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.883133+0000) 2020-02-05T21:54:24.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:24.883+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.883421+0000) 2020-02-05T21:54:24.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:24.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:24.886409+0000) 2020-02-05T21:54:24.891 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:24.889+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99599 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:25.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:25.886+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:25.887124+0000) 2020-02-05T21:54:25.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:25.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:25.892110+0000) 2020-02-05T21:54:27.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:27.032+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:27.032586+0000) 2020-02-05T21:54:27.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:27.032+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:27.032663+0000) 2020-02-05T21:54:27.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:27.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:27.039929+0000) 2020-02-05T21:54:28.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:28.486+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:28.487715+0000) 2020-02-05T21:54:29.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:29.421+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.422624+0000) 2020-02-05T21:54:29.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:29.894+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.895489+0000) 2020-02-05T21:54:29.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:29.894+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.896049+0000) 2020-02-05T21:54:29.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:29.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.896871+0000) 2020-02-05T21:54:29.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:29.896+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.897246+0000) 2020-02-05T21:54:29.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:29.896+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.897690+0000) 2020-02-05T21:54:29.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:29.896+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.897638+0000) 2020-02-05T21:54:29.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:29.898+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.899180+0000) 2020-02-05T21:54:29.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:29.898+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.899578+0000) 2020-02-05T21:54:29.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:29.898+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.899384+0000) 2020-02-05T21:54:29.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:29.900+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.900630+0000) 2020-02-05T21:54:29.902 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:29.900+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.900857+0000) 2020-02-05T21:54:29.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:29.901+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.901625+0000) 2020-02-05T21:54:29.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:29.901+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.901861+0000) 2020-02-05T21:54:29.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:29.902+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:29.903950+0000) 2020-02-05T21:54:29.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:29.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99642 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:30.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:30.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:30.030191+0000) 2020-02-05T21:54:30.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:30.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:30.265072+0000) 2020-02-05T21:54:30.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:30.281+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:30.282700+0000) 2020-02-05T21:54:30.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:30.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:30.283252+0000) 2020-02-05T21:54:30.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:30.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:30.291211+0000) 2020-02-05T21:54:32.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:32.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:32.275060+0000) 2020-02-05T21:54:32.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:32.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:32.674116+0000) 2020-02-05T21:54:34.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:34.911+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.912786+0000) 2020-02-05T21:54:34.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:34.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.913534+0000) 2020-02-05T21:54:34.916 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:34.913+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.915044+0000) 2020-02-05T21:54:34.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:34.914+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.915841+0000) 2020-02-05T21:54:34.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:34.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.916269+0000) 2020-02-05T21:54:34.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:34.915+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.916851+0000) 2020-02-05T21:54:34.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:34.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.916981+0000) 2020-02-05T21:54:34.918 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:34.916+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.917754+0000) 2020-02-05T21:54:34.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:34.916+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.918055+0000) 2020-02-05T21:54:34.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:34.917+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.918450+0000) 2020-02-05T21:54:34.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:34.917+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.918909+0000) 2020-02-05T21:54:34.921 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:34.919+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.920574+0000) 2020-02-05T21:54:34.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:34.919+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.920869+0000) 2020-02-05T21:54:34.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:34.921+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:34.922113+0000) 2020-02-05T21:54:34.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:34.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99679 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:35.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:35.886+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:35.887318+0000) 2020-02-05T21:54:35.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:35.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:35.892354+0000) 2020-02-05T21:54:37.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:37.031+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:37.032824+0000) 2020-02-05T21:54:37.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:37.031+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:37.032936+0000) 2020-02-05T21:54:37.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:37.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:37.040167+0000) 2020-02-05T21:54:38.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:38.494+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:38.495469+0000) 2020-02-05T21:54:39.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:39.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.422844+0000) 2020-02-05T21:54:39.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:39.931+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.932181+0000) 2020-02-05T21:54:39.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:39.932+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.933136+0000) 2020-02-05T21:54:39.934 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:39.932+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.933795+0000) 2020-02-05T21:54:39.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:39.932+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.934348+0000) 2020-02-05T21:54:39.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:39.933+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.934506+0000) 2020-02-05T21:54:39.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:39.933+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.934898+0000) 2020-02-05T21:54:39.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:39.934+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.935305+0000) 2020-02-05T21:54:39.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:39.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.935434+0000) 2020-02-05T21:54:39.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:39.934+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.935925+0000) 2020-02-05T21:54:39.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:39.935+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.936414+0000) 2020-02-05T21:54:39.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:39.935+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.936677+0000) 2020-02-05T21:54:39.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:39.935+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.936213+0000) 2020-02-05T21:54:39.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:39.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.936310+0000) 2020-02-05T21:54:39.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:39.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:39.940816+0000) 2020-02-05T21:54:39.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:39.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:40.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:40.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:40.030406+0000) 2020-02-05T21:54:40.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:40.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:40.265302+0000) 2020-02-05T21:54:40.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:40.282+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:40.282941+0000) 2020-02-05T21:54:40.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:40.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:40.283494+0000) 2020-02-05T21:54:40.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:40.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:40.291416+0000) 2020-02-05T21:54:42.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:42.273+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:42.275300+0000) 2020-02-05T21:54:42.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:42.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:42.674385+0000) 2020-02-05T21:54:44.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:44.947+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.948717+0000) 2020-02-05T21:54:44.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:44.949+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.950339+0000) 2020-02-05T21:54:44.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:44.949+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.950828+0000) 2020-02-05T21:54:44.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:44.950+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.951142+0000) 2020-02-05T21:54:44.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:44.950+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.951423+0000) 2020-02-05T21:54:44.953 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:44.950+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.951929+0000) 2020-02-05T21:54:44.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:44.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.951700+0000) 2020-02-05T21:54:44.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:44.951+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.952271+0000) 2020-02-05T21:54:44.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:44.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.952626+0000) 2020-02-05T21:54:44.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:44.951+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.952917+0000) 2020-02-05T21:54:44.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:44.951+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.953197+0000) 2020-02-05T21:54:44.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:44.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.953569+0000) 2020-02-05T21:54:44.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:44.952+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.954005+0000) 2020-02-05T21:54:44.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:44.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:44.958650+0000) 2020-02-05T21:54:44.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:44.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:45.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:45.886+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:45.887523+0000) 2020-02-05T21:54:45.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:45.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:45.892615+0000) 2020-02-05T21:54:47.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:47.032+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:47.033076+0000) 2020-02-05T21:54:47.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:47.032+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:47.033203+0000) 2020-02-05T21:54:47.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:47.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:47.040399+0000) 2020-02-05T21:54:48.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:48.501+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:48.502105+0000) 2020-02-05T21:54:49.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:49.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.423105+0000) 2020-02-05T21:54:49.701 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:54:49.708 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:54:49.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:49.965+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.966601+0000) 2020-02-05T21:54:49.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:49.966+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.967662+0000) 2020-02-05T21:54:49.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:49.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.968216+0000) 2020-02-05T21:54:49.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:49.967+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.968966+0000) 2020-02-05T21:54:49.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:49.968+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.969588+0000) 2020-02-05T21:54:49.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:49.969+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.970542+0000) 2020-02-05T21:54:49.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:49.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.970904+0000) 2020-02-05T21:54:49.973 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:49.970+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.971678+0000) 2020-02-05T21:54:49.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:49.970+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.971770+0000) 2020-02-05T21:54:49.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:49.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.972744+0000) 2020-02-05T21:54:49.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:49.971+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.973254+0000) 2020-02-05T21:54:49.975 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:49.971+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.972975+0000) 2020-02-05T21:54:49.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:49.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.973964+0000) 2020-02-05T21:54:49.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:49.976+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:49.977684+0000) 2020-02-05T21:54:49.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:49.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:50.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:50.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:50.030594+0000) 2020-02-05T21:54:50.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:50.264+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:50.265525+0000) 2020-02-05T21:54:50.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:50.282+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:50.283130+0000) 2020-02-05T21:54:50.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:50.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:50.283660+0000) 2020-02-05T21:54:50.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:50.291+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:50.291620+0000) 2020-02-05T21:54:52.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:52.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:52.275538+0000) 2020-02-05T21:54:52.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:52.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:52.674613+0000) 2020-02-05T21:54:54.988 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:54:54.985+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.987121+0000) 2020-02-05T21:54:54.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:54.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.987967+0000) 2020-02-05T21:54:54.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:54.987+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.988576+0000) 2020-02-05T21:54:54.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:54.987+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.988971+0000) 2020-02-05T21:54:54.991 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:54:54.988+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.989610+0000) 2020-02-05T21:54:54.991 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:54:54.988+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.990238+0000) 2020-02-05T21:54:54.992 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:54:54.989+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.990752+0000) 2020-02-05T21:54:54.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:54.990+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.991895+0000) 2020-02-05T21:54:54.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:54:54.990+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.991660+0000) 2020-02-05T21:54:54.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:54.990+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.991449+0000) 2020-02-05T21:54:54.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:54:54.991+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.992185+0000) 2020-02-05T21:54:54.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:54.990+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.992236+0000) 2020-02-05T21:54:54.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:54.991+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.992570+0000) 2020-02-05T21:54:54.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:54.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:54.997093+0000) 2020-02-05T21:54:55.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:54:54.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:54:55.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:54:55.886+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:55.887711+0000) 2020-02-05T21:54:55.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:54:55.892+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:55.892832+0000) 2020-02-05T21:54:57.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:54:57.032+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:57.033328+0000) 2020-02-05T21:54:57.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:54:57.032+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:57.033492+0000) 2020-02-05T21:54:57.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:54:57.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:57.040623+0000) 2020-02-05T21:54:58.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:58.509+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:58.510354+0000) 2020-02-05T21:54:59.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:54:59.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:54:59.423345+0000) 2020-02-05T21:55:00.011 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:00.004+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.005791+0000) 2020-02-05T21:55:00.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:00.005+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.007136+0000) 2020-02-05T21:55:00.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:00.006+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.008322+0000) 2020-02-05T21:55:00.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:00.007+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.008524+0000) 2020-02-05T21:55:00.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:00.004+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.005073+0000) 2020-02-05T21:55:00.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:00.006+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.007361+0000) 2020-02-05T21:55:00.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:00.006+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.008023+0000) 2020-02-05T21:55:00.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:00.007+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.008478+0000) 2020-02-05T21:55:00.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:00.008+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.009122+0000) 2020-02-05T21:55:00.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:00.008+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.009982+0000) 2020-02-05T21:55:00.013 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:00.008+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.010128+0000) 2020-02-05T21:55:00.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:00.008+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.010261+0000) 2020-02-05T21:55:00.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:00.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.010411+0000) 2020-02-05T21:55:00.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:00.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.011584+0000) 2020-02-05T21:55:00.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:00.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:00.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:00.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.030795+0000) 2020-02-05T21:55:00.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:00.265+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.265768+0000) 2020-02-05T21:55:00.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:00.282+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.283313+0000) 2020-02-05T21:55:00.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:00.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.283869+0000) 2020-02-05T21:55:00.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:00.290+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:00.291849+0000) 2020-02-05T21:55:02.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:02.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:02.275743+0000) 2020-02-05T21:55:02.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:02.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:02.674854+0000) 2020-02-05T21:55:05.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:05.020+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.021968+0000) 2020-02-05T21:55:05.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:05.022+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.023409+0000) 2020-02-05T21:55:05.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:05.023+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.024065+0000) 2020-02-05T21:55:05.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:05.023+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.024642+0000) 2020-02-05T21:55:05.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:05.023+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.024796+0000) 2020-02-05T21:55:05.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:05.023+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.025120+0000) 2020-02-05T21:55:05.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:05.024+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.025566+0000) 2020-02-05T21:55:05.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:05.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.026003+0000) 2020-02-05T21:55:05.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:05.025+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.027344+0000) 2020-02-05T21:55:05.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:05.025+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.027254+0000) 2020-02-05T21:55:05.029 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:05.027+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.028683+0000) 2020-02-05T21:55:05.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:05.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.031211+0000) 2020-02-05T21:55:05.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:05.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.031587+0000) 2020-02-05T21:55:05.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:05.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.031896+0000) 2020-02-05T21:55:05.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:05.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99918 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:05.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:05.887+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.887956+0000) 2020-02-05T21:55:05.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:05.892+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:05.893052+0000) 2020-02-05T21:55:07.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:07.032+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:07.033567+0000) 2020-02-05T21:55:07.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:07.032+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:07.033795+0000) 2020-02-05T21:55:07.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:07.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:07.040861+0000) 2020-02-05T21:55:08.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:08.518+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:08.518484+0000) 2020-02-05T21:55:09.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:09.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:09.423567+0000) 2020-02-05T21:55:10.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:10.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.030969+0000) 2020-02-05T21:55:10.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:10.038+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.039126+0000) 2020-02-05T21:55:10.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:10.038+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.039951+0000) 2020-02-05T21:55:10.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:10.039+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.040423+0000) 2020-02-05T21:55:10.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:10.040+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.041131+0000) 2020-02-05T21:55:10.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:10.040+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.041559+0000) 2020-02-05T21:55:10.043 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:10.040+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.042278+0000) 2020-02-05T21:55:10.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:10.041+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.042788+0000) 2020-02-05T21:55:10.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:10.041+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.043136+0000) 2020-02-05T21:55:10.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:10.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.043667+0000) 2020-02-05T21:55:10.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:10.042+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.043861+0000) 2020-02-05T21:55:10.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:10.042+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.043929+0000) 2020-02-05T21:55:10.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:10.043+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.044774+0000) 2020-02-05T21:55:10.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:10.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.047902+0000) 2020-02-05T21:55:10.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:10.048+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.049069+0000) 2020-02-05T21:55:10.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:10.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:10.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:10.265+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.266002+0000) 2020-02-05T21:55:10.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:10.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.283548+0000) 2020-02-05T21:55:10.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:10.283+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.284075+0000) 2020-02-05T21:55:10.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:10.291+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:10.292110+0000) 2020-02-05T21:55:12.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:12.274+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:12.276014+0000) 2020-02-05T21:55:12.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:12.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:12.675143+0000) 2020-02-05T21:55:15.060 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:15.057+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.058778+0000) 2020-02-05T21:55:15.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:15.058+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.058886+0000) 2020-02-05T21:55:15.061 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:15.059+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.059498+0000) 2020-02-05T21:55:15.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:15.059+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.059558+0000) 2020-02-05T21:55:15.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:15.058+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.060036+0000) 2020-02-05T21:55:15.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:15.060+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.060306+0000) 2020-02-05T21:55:15.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:15.059+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.061010+0000) 2020-02-05T21:55:15.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:15.061+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.061470+0000) 2020-02-05T21:55:15.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:15.060+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.062077+0000) 2020-02-05T21:55:15.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:15.062+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.062400+0000) 2020-02-05T21:55:15.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:15.062+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.062410+0000) 2020-02-05T21:55:15.064 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:15.061+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.062640+0000) 2020-02-05T21:55:15.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:15.061+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.063035+0000) 2020-02-05T21:55:15.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:15.065+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.065942+0000) 2020-02-05T21:55:15.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:15.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 99998 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:15.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:15.888+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.888164+0000) 2020-02-05T21:55:15.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:15.892+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:15.893293+0000) 2020-02-05T21:55:17.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:17.033+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:17.033847+0000) 2020-02-05T21:55:17.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:17.033+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:17.034077+0000) 2020-02-05T21:55:17.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:17.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:17.041126+0000) 2020-02-05T21:55:18.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:18.530+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:18.530325+0000) 2020-02-05T21:55:19.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:19.422+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:19.423801+0000) 2020-02-05T21:55:19.863 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:55:19.869 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:55:20.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:20.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.031202+0000) 2020-02-05T21:55:20.077 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:20.074+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.075580+0000) 2020-02-05T21:55:20.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:20.074+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.076315+0000) 2020-02-05T21:55:20.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:20.076+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.077485+0000) 2020-02-05T21:55:20.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:20.077+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.078121+0000) 2020-02-05T21:55:20.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:20.077+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.078525+0000) 2020-02-05T21:55:20.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:20.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.079259+0000) 2020-02-05T21:55:20.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:20.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.079779+0000) 2020-02-05T21:55:20.081 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:20.078+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.080036+0000) 2020-02-05T21:55:20.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:20.078+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.080090+0000) 2020-02-05T21:55:20.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:20.079+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.080942+0000) 2020-02-05T21:55:20.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:20.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.081361+0000) 2020-02-05T21:55:20.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:20.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.081640+0000) 2020-02-05T21:55:20.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:20.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.082383+0000) 2020-02-05T21:55:20.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:20.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.087476+0000) 2020-02-05T21:55:20.089 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:20.086+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:20.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:20.265+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.266237+0000) 2020-02-05T21:55:20.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:20.282+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.283749+0000) 2020-02-05T21:55:20.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:20.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.284275+0000) 2020-02-05T21:55:20.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:20.291+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:20.292345+0000) 2020-02-05T21:55:22.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:22.275+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:22.276280+0000) 2020-02-05T21:55:22.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:22.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:22.675430+0000) 2020-02-05T21:55:25.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:25.092+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.093658+0000) 2020-02-05T21:55:25.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:25.094+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.094814+0000) 2020-02-05T21:55:25.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:25.094+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.095320+0000) 2020-02-05T21:55:25.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:25.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.095917+0000) 2020-02-05T21:55:25.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:25.095+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.096034+0000) 2020-02-05T21:55:25.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:25.095+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.096266+0000) 2020-02-05T21:55:25.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:25.096+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.096851+0000) 2020-02-05T21:55:25.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:25.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.097255+0000) 2020-02-05T21:55:25.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:25.096+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.097413+0000) 2020-02-05T21:55:25.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:25.096+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.097563+0000) 2020-02-05T21:55:25.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:25.098+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.099123+0000) 2020-02-05T21:55:25.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:25.099+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.100098+0000) 2020-02-05T21:55:25.101 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:25.099+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.100582+0000) 2020-02-05T21:55:25.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:25.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.105264+0000) 2020-02-05T21:55:25.107 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:25.105+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100078 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:25.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:25.887+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.888335+0000) 2020-02-05T21:55:25.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:25.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:25.893522+0000) 2020-02-05T21:55:27.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:27.033+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:27.034077+0000) 2020-02-05T21:55:27.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:27.033+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:27.034314+0000) 2020-02-05T21:55:27.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:27.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:27.041378+0000) 2020-02-05T21:55:28.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:28.537+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:28.538215+0000) 2020-02-05T21:55:29.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:29.423+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:29.424031+0000) 2020-02-05T21:55:30.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:30.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.031439+0000) 2020-02-05T21:55:30.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:30.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.112288+0000) 2020-02-05T21:55:30.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:30.112+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.113030+0000) 2020-02-05T21:55:30.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:30.113+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.114423+0000) 2020-02-05T21:55:30.116 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:30.114+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.115208+0000) 2020-02-05T21:55:30.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:30.115+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.115834+0000) 2020-02-05T21:55:30.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:30.115+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.115986+0000) 2020-02-05T21:55:30.117 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:30.115+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.116423+0000) 2020-02-05T21:55:30.118 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:30.117+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.117768+0000) 2020-02-05T21:55:30.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:30.116+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.117605+0000) 2020-02-05T21:55:30.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:30.117+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.118102+0000) 2020-02-05T21:55:30.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:30.117+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.118211+0000) 2020-02-05T21:55:30.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:30.118+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.118574+0000) 2020-02-05T21:55:30.120 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:30.117+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.118364+0000) 2020-02-05T21:55:30.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:30.121+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.122791+0000) 2020-02-05T21:55:30.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:30.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100122 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:30.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:30.266+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.266470+0000) 2020-02-05T21:55:30.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:30.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.283975+0000) 2020-02-05T21:55:30.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:30.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.284483+0000) 2020-02-05T21:55:30.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:30.291+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:30.292593+0000) 2020-02-05T21:55:32.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:32.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:32.276538+0000) 2020-02-05T21:55:32.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:32.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:32.675671+0000) 2020-02-05T21:55:35.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:35.129+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.130417+0000) 2020-02-05T21:55:35.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:35.131+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.132160+0000) 2020-02-05T21:55:35.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:35.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.132671+0000) 2020-02-05T21:55:35.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:35.133+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.133628+0000) 2020-02-05T21:55:35.136 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:35.133+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.134282+0000) 2020-02-05T21:55:35.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:35.133+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.133986+0000) 2020-02-05T21:55:35.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:35.134+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.135324+0000) 2020-02-05T21:55:35.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:35.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.135747+0000) 2020-02-05T21:55:35.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:35.135+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.136161+0000) 2020-02-05T21:55:35.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:35.136+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.136742+0000) 2020-02-05T21:55:35.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:35.135+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.136603+0000) 2020-02-05T21:55:35.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:35.136+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.137150+0000) 2020-02-05T21:55:35.139 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:35.136+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.137198+0000) 2020-02-05T21:55:35.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:35.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:35.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:35.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.145515+0000) 2020-02-05T21:55:35.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:35.887+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.888551+0000) 2020-02-05T21:55:35.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:35.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:35.893784+0000) 2020-02-05T21:55:37.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:37.033+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:37.034321+0000) 2020-02-05T21:55:37.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:37.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:37.034613+0000) 2020-02-05T21:55:37.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:37.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:37.041615+0000) 2020-02-05T21:55:38.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:38.545+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:38.546670+0000) 2020-02-05T21:55:39.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:39.423+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:39.424310+0000) 2020-02-05T21:55:40.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:40.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.031665+0000) 2020-02-05T21:55:40.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:40.147+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.149023+0000) 2020-02-05T21:55:40.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:40.149+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.149579+0000) 2020-02-05T21:55:40.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:40.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.149963+0000) 2020-02-05T21:55:40.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:40.150+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.150731+0000) 2020-02-05T21:55:40.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:40.149+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.150904+0000) 2020-02-05T21:55:40.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:40.151+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.151921+0000) 2020-02-05T21:55:40.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:40.150+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.152025+0000) 2020-02-05T21:55:40.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:40.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.152673+0000) 2020-02-05T21:55:40.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:40.152+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.153883+0000) 2020-02-05T21:55:40.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:40.153+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.154381+0000) 2020-02-05T21:55:40.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:40.154+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.154797+0000) 2020-02-05T21:55:40.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:40.155+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.155809+0000) 2020-02-05T21:55:40.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:40.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.159663+0000) 2020-02-05T21:55:40.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:40.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.160015+0000) 2020-02-05T21:55:40.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:40.160+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:40.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:40.266+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.266690+0000) 2020-02-05T21:55:40.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:40.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.284154+0000) 2020-02-05T21:55:40.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:40.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.284688+0000) 2020-02-05T21:55:40.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:40.292+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:40.292805+0000) 2020-02-05T21:55:42.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:42.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:42.276794+0000) 2020-02-05T21:55:42.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:42.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:42.675919+0000) 2020-02-05T21:55:45.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:45.165+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.166345+0000) 2020-02-05T21:55:45.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:45.165+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.166858+0000) 2020-02-05T21:55:45.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:45.166+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.167544+0000) 2020-02-05T21:55:45.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:45.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.169025+0000) 2020-02-05T21:55:45.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:45.168+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.169273+0000) 2020-02-05T21:55:45.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:45.169+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.170298+0000) 2020-02-05T21:55:45.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:45.170+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.171126+0000) 2020-02-05T21:55:45.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:45.170+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.171900+0000) 2020-02-05T21:55:45.173 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:45.171+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.171640+0000) 2020-02-05T21:55:45.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:45.171+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.171825+0000) 2020-02-05T21:55:45.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:45.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.172441+0000) 2020-02-05T21:55:45.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:45.171+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.172364+0000) 2020-02-05T21:55:45.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:45.172+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.172731+0000) 2020-02-05T21:55:45.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:45.175+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.176701+0000) 2020-02-05T21:55:45.179 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:45.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:45.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:45.888+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.888818+0000) 2020-02-05T21:55:45.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:45.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:45.894007+0000) 2020-02-05T21:55:47.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:47.034+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:47.034578+0000) 2020-02-05T21:55:47.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:47.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:47.034864+0000) 2020-02-05T21:55:47.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:47.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:47.041839+0000) 2020-02-05T21:55:48.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:48.554+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:48.555370+0000) 2020-02-05T21:55:49.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:49.423+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:49.424508+0000) 2020-02-05T21:55:50.014 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:55:50.020 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:55:50.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:50.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.031880+0000) 2020-02-05T21:55:50.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:50.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.184773+0000) 2020-02-05T21:55:50.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:50.185+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.185897+0000) 2020-02-05T21:55:50.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:50.185+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.186064+0000) 2020-02-05T21:55:50.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:50.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.186490+0000) 2020-02-05T21:55:50.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:50.185+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.186799+0000) 2020-02-05T21:55:50.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:50.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.187732+0000) 2020-02-05T21:55:50.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:50.187+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.188054+0000) 2020-02-05T21:55:50.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:50.187+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.187978+0000) 2020-02-05T21:55:50.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:50.187+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.188380+0000) 2020-02-05T21:55:50.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:50.187+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.188575+0000) 2020-02-05T21:55:50.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:50.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.189058+0000) 2020-02-05T21:55:50.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:50.189+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.189569+0000) 2020-02-05T21:55:50.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:50.190+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.190515+0000) 2020-02-05T21:55:50.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:50.192+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.193428+0000) 2020-02-05T21:55:50.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:50.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:50.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:50.266+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.266919+0000) 2020-02-05T21:55:50.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:50.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.284374+0000) 2020-02-05T21:55:50.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:50.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.284897+0000) 2020-02-05T21:55:50.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:50.292+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:50.293009+0000) 2020-02-05T21:55:52.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:52.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:52.277031+0000) 2020-02-05T21:55:52.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:52.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:52.676168+0000) 2020-02-05T21:55:55.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:55.201+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.202312+0000) 2020-02-05T21:55:55.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:55.203+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.203863+0000) 2020-02-05T21:55:55.205 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:55:55.203+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.204169+0000) 2020-02-05T21:55:55.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:55.205+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.206064+0000) 2020-02-05T21:55:55.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:55.205+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.206635+0000) 2020-02-05T21:55:55.208 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:55.205+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.206858+0000) 2020-02-05T21:55:55.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:55:55.206+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.207557+0000) 2020-02-05T21:55:55.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:55:55.207+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.207945+0000) 2020-02-05T21:55:55.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:55:55.207+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.208163+0000) 2020-02-05T21:55:55.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:55.207+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.208632+0000) 2020-02-05T21:55:55.210 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:55:55.207+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.208751+0000) 2020-02-05T21:55:55.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:55:55.208+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.208735+0000) 2020-02-05T21:55:55.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:55.208+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.208958+0000) 2020-02-05T21:55:55.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:55.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.212344+0000) 2020-02-05T21:55:55.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:55:55.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:55:55.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:55:55.888+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.889065+0000) 2020-02-05T21:55:55.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:55:55.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:55.894230+0000) 2020-02-05T21:55:57.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:55:57.034+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:57.034827+0000) 2020-02-05T21:55:57.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:55:57.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:57.035111+0000) 2020-02-05T21:55:57.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:55:57.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:57.042103+0000) 2020-02-05T21:55:58.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:58.563+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:58.564827+0000) 2020-02-05T21:55:59.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:55:59.424+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:55:59.424804+0000) 2020-02-05T21:56:00.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:00.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.032156+0000) 2020-02-05T21:56:00.224 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:00.221+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.222830+0000) 2020-02-05T21:56:00.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:00.222+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.223739+0000) 2020-02-05T21:56:00.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:00.223+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.224404+0000) 2020-02-05T21:56:00.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:00.223+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.224686+0000) 2020-02-05T21:56:00.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:00.225+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.225557+0000) 2020-02-05T21:56:00.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:00.225+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.226294+0000) 2020-02-05T21:56:00.228 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:00.226+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.227338+0000) 2020-02-05T21:56:00.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:00.227+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.227628+0000) 2020-02-05T21:56:00.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:00.226+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.227722+0000) 2020-02-05T21:56:00.229 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:00.226+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.227870+0000) 2020-02-05T21:56:00.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:00.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.228070+0000) 2020-02-05T21:56:00.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:00.227+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.228293+0000) 2020-02-05T21:56:00.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:00.228+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.228471+0000) 2020-02-05T21:56:00.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:00.232+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.234026+0000) 2020-02-05T21:56:00.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:00.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100361 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:00.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:00.267+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.267144+0000) 2020-02-05T21:56:00.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:00.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.284543+0000) 2020-02-05T21:56:00.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:00.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.285080+0000) 2020-02-05T21:56:00.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:00.292+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:00.293126+0000) 2020-02-05T21:56:02.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:02.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:02.277251+0000) 2020-02-05T21:56:02.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:02.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:02.676400+0000) 2020-02-05T21:56:05.242 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:05.240+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.241353+0000) 2020-02-05T21:56:05.243 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:05.241+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.242208+0000) 2020-02-05T21:56:05.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:05.241+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.242470+0000) 2020-02-05T21:56:05.244 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:05.241+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.242956+0000) 2020-02-05T21:56:05.245 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:05.242+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.243412+0000) 2020-02-05T21:56:05.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:05.242+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.244172+0000) 2020-02-05T21:56:05.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:05.243+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.244421+0000) 2020-02-05T21:56:05.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:05.243+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.245244+0000) 2020-02-05T21:56:05.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:05.243+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.245411+0000) 2020-02-05T21:56:05.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:05.244+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.246032+0000) 2020-02-05T21:56:05.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:05.245+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.246998+0000) 2020-02-05T21:56:05.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:05.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.246981+0000) 2020-02-05T21:56:05.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:05.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.249678+0000) 2020-02-05T21:56:05.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:05.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.250768+0000) 2020-02-05T21:56:05.254 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:05.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:05.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:05.889+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.889329+0000) 2020-02-05T21:56:05.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:05.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:05.894478+0000) 2020-02-05T21:56:07.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:07.034+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:07.035082+0000) 2020-02-05T21:56:07.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:07.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:07.035369+0000) 2020-02-05T21:56:07.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:07.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:07.042294+0000) 2020-02-05T21:56:08.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:08.575+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:08.576029+0000) 2020-02-05T21:56:09.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:09.424+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:09.425052+0000) 2020-02-05T21:56:10.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:10.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.032412+0000) 2020-02-05T21:56:10.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:10.258+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.258963+0000) 2020-02-05T21:56:10.261 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:10.260+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.260465+0000) 2020-02-05T21:56:10.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:10.258+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.260112+0000) 2020-02-05T21:56:10.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:10.259+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.261072+0000) 2020-02-05T21:56:10.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:10.262+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.262162+0000) 2020-02-05T21:56:10.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:10.260+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.262001+0000) 2020-02-05T21:56:10.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:10.260+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.262363+0000) 2020-02-05T21:56:10.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:10.261+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.262864+0000) 2020-02-05T21:56:10.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:10.262+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.262697+0000) 2020-02-05T21:56:10.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:10.263+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.263160+0000) 2020-02-05T21:56:10.265 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:10.263+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.263424+0000) 2020-02-05T21:56:10.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:10.262+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.264038+0000) 2020-02-05T21:56:10.266 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:10.263+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.263900+0000) 2020-02-05T21:56:10.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:10.265+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.267324+0000) 2020-02-05T21:56:10.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:10.270+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.270363+0000) 2020-02-05T21:56:10.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:10.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100442 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:10.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:10.284+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.284782+0000) 2020-02-05T21:56:10.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:10.283+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.285259+0000) 2020-02-05T21:56:10.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:10.293+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:10.293346+0000) 2020-02-05T21:56:12.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:12.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:12.277541+0000) 2020-02-05T21:56:12.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:12.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:12.676630+0000) 2020-02-05T21:56:15.277 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:15.274+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.275735+0000) 2020-02-05T21:56:15.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:15.275+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.276541+0000) 2020-02-05T21:56:15.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:15.276+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.277486+0000) 2020-02-05T21:56:15.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:15.276+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.277939+0000) 2020-02-05T21:56:15.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:15.276+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.278115+0000) 2020-02-05T21:56:15.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:15.277+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.279046+0000) 2020-02-05T21:56:15.280 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:15.277+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.279008+0000) 2020-02-05T21:56:15.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:15.278+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.279481+0000) 2020-02-05T21:56:15.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:15.278+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.279867+0000) 2020-02-05T21:56:15.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:15.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.280040+0000) 2020-02-05T21:56:15.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:15.279+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.280420+0000) 2020-02-05T21:56:15.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:15.279+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.281076+0000) 2020-02-05T21:56:15.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:15.279+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.281187+0000) 2020-02-05T21:56:15.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:15.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.285613+0000) 2020-02-05T21:56:15.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:15.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:15.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:15.888+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.889532+0000) 2020-02-05T21:56:15.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:15.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:15.894720+0000) 2020-02-05T21:56:17.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:17.033+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:17.035326+0000) 2020-02-05T21:56:17.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:17.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:17.035594+0000) 2020-02-05T21:56:17.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:17.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:17.042530+0000) 2020-02-05T21:56:18.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:18.586+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:18.587972+0000) 2020-02-05T21:56:19.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:19.424+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:19.425397+0000) 2020-02-05T21:56:20.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:20.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.032633+0000) 2020-02-05T21:56:20.156 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:56:20.162 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:56:20.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:20.266+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.267540+0000) 2020-02-05T21:56:20.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:20.283+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.284982+0000) 2020-02-05T21:56:20.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:20.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.285502+0000) 2020-02-05T21:56:20.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:20.291+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.292887+0000) 2020-02-05T21:56:20.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:20.293+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.294506+0000) 2020-02-05T21:56:20.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:20.292+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.294013+0000) 2020-02-05T21:56:20.297 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:20.294+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.295114+0000) 2020-02-05T21:56:20.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:20.294+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.295866+0000) 2020-02-05T21:56:20.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:20.295+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.297023+0000) 2020-02-05T21:56:20.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:20.296+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.297649+0000) 2020-02-05T21:56:20.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:20.296+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.297932+0000) 2020-02-05T21:56:20.299 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:20.296+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.298055+0000) 2020-02-05T21:56:20.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:20.296+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.298332+0000) 2020-02-05T21:56:20.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:20.297+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.299059+0000) 2020-02-05T21:56:20.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:20.297+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.298697+0000) 2020-02-05T21:56:20.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:20.297+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.299399+0000) 2020-02-05T21:56:20.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:20.298+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:20.299820+0000) 2020-02-05T21:56:20.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:20.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:22.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:22.276+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:22.277799+0000) 2020-02-05T21:56:22.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:22.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:22.676866+0000) 2020-02-05T21:56:25.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:25.309+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.310980+0000) 2020-02-05T21:56:25.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:25.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.311775+0000) 2020-02-05T21:56:25.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:25.311+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.312285+0000) 2020-02-05T21:56:25.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:25.310+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.311506+0000) 2020-02-05T21:56:25.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:25.311+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.312880+0000) 2020-02-05T21:56:25.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:25.311+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.313014+0000) 2020-02-05T21:56:25.315 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:25.312+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.313459+0000) 2020-02-05T21:56:25.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:25.312+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.313490+0000) 2020-02-05T21:56:25.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:25.312+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.314365+0000) 2020-02-05T21:56:25.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:25.313+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.314470+0000) 2020-02-05T21:56:25.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:25.313+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.315008+0000) 2020-02-05T21:56:25.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:25.313+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.315032+0000) 2020-02-05T21:56:25.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:25.313+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.315239+0000) 2020-02-05T21:56:25.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:25.318+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.319564+0000) 2020-02-05T21:56:25.323 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:25.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:25.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:25.888+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.889760+0000) 2020-02-05T21:56:25.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:25.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:25.895003+0000) 2020-02-05T21:56:27.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:27.034+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:27.035563+0000) 2020-02-05T21:56:27.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:27.034+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:27.035771+0000) 2020-02-05T21:56:27.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:27.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:27.042765+0000) 2020-02-05T21:56:27.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:27.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:27.043907+0000) 2020-02-05T21:56:28.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:28.598+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:28.599605+0000) 2020-02-05T21:56:29.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:29.424+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:29.425630+0000) 2020-02-05T21:56:30.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:30.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.032846+0000) 2020-02-05T21:56:30.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:30.267+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.267852+0000) 2020-02-05T21:56:30.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:30.284+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.285216+0000) 2020-02-05T21:56:30.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:30.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.285722+0000) 2020-02-05T21:56:30.330 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:30.327+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.328957+0000) 2020-02-05T21:56:30.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:30.328+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.329597+0000) 2020-02-05T21:56:30.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:30.328+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.329975+0000) 2020-02-05T21:56:30.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:30.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.330347+0000) 2020-02-05T21:56:30.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:30.329+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.330526+0000) 2020-02-05T21:56:30.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:30.329+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.331347+0000) 2020-02-05T21:56:30.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:30.330+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.331533+0000) 2020-02-05T21:56:30.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:30.330+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.331724+0000) 2020-02-05T21:56:30.334 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:30.329+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.330773+0000) 2020-02-05T21:56:30.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:30.330+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.331244+0000) 2020-02-05T21:56:30.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:30.331+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.332962+0000) 2020-02-05T21:56:30.337 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:30.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.332203+0000) 2020-02-05T21:56:30.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:30.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.332921+0000) 2020-02-05T21:56:30.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:30.337+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:30.338618+0000) 2020-02-05T21:56:30.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:30.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100602 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:32.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:32.277+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:32.278043+0000) 2020-02-05T21:56:32.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:32.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:32.677154+0000) 2020-02-05T21:56:35.347 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:35.344+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.345596+0000) 2020-02-05T21:56:35.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:35.345+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.346308+0000) 2020-02-05T21:56:35.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:35.345+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.346556+0000) 2020-02-05T21:56:35.348 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:35.345+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.347182+0000) 2020-02-05T21:56:35.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:35.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.347377+0000) 2020-02-05T21:56:35.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:35.346+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.347694+0000) 2020-02-05T21:56:35.349 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:35.347+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.348592+0000) 2020-02-05T21:56:35.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:35.347+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.348903+0000) 2020-02-05T21:56:35.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:35.347+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.349331+0000) 2020-02-05T21:56:35.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:35.348+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.349756+0000) 2020-02-05T21:56:35.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:35.348+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.350118+0000) 2020-02-05T21:56:35.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:35.349+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.350671+0000) 2020-02-05T21:56:35.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:35.354+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.356002+0000) 2020-02-05T21:56:35.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:35.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.356280+0000) 2020-02-05T21:56:35.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:35.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:35.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:35.889+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.890019+0000) 2020-02-05T21:56:35.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:35.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:35.895237+0000) 2020-02-05T21:56:37.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:37.035+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:37.035817+0000) 2020-02-05T21:56:37.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:37.035+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:37.035981+0000) 2020-02-05T21:56:37.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:37.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:37.043027+0000) 2020-02-05T21:56:37.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:37.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:37.044208+0000) 2020-02-05T21:56:38.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:38.607+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:38.607997+0000) 2020-02-05T21:56:39.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:39.425+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:39.425890+0000) 2020-02-05T21:56:40.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:40.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.033063+0000) 2020-02-05T21:56:40.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:40.267+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.268106+0000) 2020-02-05T21:56:40.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:40.284+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.285452+0000) 2020-02-05T21:56:40.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:40.285+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.285951+0000) 2020-02-05T21:56:40.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:40.361+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.362497+0000) 2020-02-05T21:56:40.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:40.362+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.363806+0000) 2020-02-05T21:56:40.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:40.362+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.364092+0000) 2020-02-05T21:56:40.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:40.363+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.364474+0000) 2020-02-05T21:56:40.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:40.363+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.364960+0000) 2020-02-05T21:56:40.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:40.363+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.364956+0000) 2020-02-05T21:56:40.367 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:40.364+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.365588+0000) 2020-02-05T21:56:40.367 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:40.364+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.365975+0000) 2020-02-05T21:56:40.368 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:40.365+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.366571+0000) 2020-02-05T21:56:40.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:40.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.367147+0000) 2020-02-05T21:56:40.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:40.366+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.367505+0000) 2020-02-05T21:56:40.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:40.366+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.368056+0000) 2020-02-05T21:56:40.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:40.367+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.368742+0000) 2020-02-05T21:56:40.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:40.372+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:40.373617+0000) 2020-02-05T21:56:40.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:40.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100679 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:42.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:42.277+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:42.278336+0000) 2020-02-05T21:56:42.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:42.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:42.677417+0000) 2020-02-05T21:56:45.381 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:45.378+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.379852+0000) 2020-02-05T21:56:45.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:45.378+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.380045+0000) 2020-02-05T21:56:45.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:45.379+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.380727+0000) 2020-02-05T21:56:45.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:45.380+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.381527+0000) 2020-02-05T21:56:45.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:45.380+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.381825+0000) 2020-02-05T21:56:45.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:45.381+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.382323+0000) 2020-02-05T21:56:45.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:45.381+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.382678+0000) 2020-02-05T21:56:45.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:45.381+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.383154+0000) 2020-02-05T21:56:45.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:45.382+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.383300+0000) 2020-02-05T21:56:45.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:45.382+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.383760+0000) 2020-02-05T21:56:45.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:45.383+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.385162+0000) 2020-02-05T21:56:45.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:45.384+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.385422+0000) 2020-02-05T21:56:45.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:45.384+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.386074+0000) 2020-02-05T21:56:45.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:45.390+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100716 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:45.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:45.391+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.392117+0000) 2020-02-05T21:56:45.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:45.889+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.890263+0000) 2020-02-05T21:56:45.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:45.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:45.895475+0000) 2020-02-05T21:56:47.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:47.035+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:47.036048+0000) 2020-02-05T21:56:47.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:47.035+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:47.036375+0000) 2020-02-05T21:56:47.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:47.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:47.043310+0000) 2020-02-05T21:56:47.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:47.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:47.044383+0000) 2020-02-05T21:56:48.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:48.614+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:48.616034+0000) 2020-02-05T21:56:49.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:49.425+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:49.426145+0000) 2020-02-05T21:56:50.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:50.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.033340+0000) 2020-02-05T21:56:50.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:50.267+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.268362+0000) 2020-02-05T21:56:50.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:50.285+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.285682+0000) 2020-02-05T21:56:50.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:50.285+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.286204+0000) 2020-02-05T21:56:50.298 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:56:50.304 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:56:50.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:50.394+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.395963+0000) 2020-02-05T21:56:50.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:50.395+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.396091+0000) 2020-02-05T21:56:50.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:50.395+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.397204+0000) 2020-02-05T21:56:50.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:50.397+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.398293+0000) 2020-02-05T21:56:50.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:50.397+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.398929+0000) 2020-02-05T21:56:50.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:50.397+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.399050+0000) 2020-02-05T21:56:50.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:50.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.399452+0000) 2020-02-05T21:56:50.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:50.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.399752+0000) 2020-02-05T21:56:50.402 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:50.399+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.400256+0000) 2020-02-05T21:56:50.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:50.399+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.400622+0000) 2020-02-05T21:56:50.402 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:50.399+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.401215+0000) 2020-02-05T21:56:50.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:50.400+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.402392+0000) 2020-02-05T21:56:50.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:50.401+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.403133+0000) 2020-02-05T21:56:50.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:50.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:50.404441+0000) 2020-02-05T21:56:50.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:50.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:52.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:52.278+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:52.278626+0000) 2020-02-05T21:56:52.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:52.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:52.677642+0000) 2020-02-05T21:56:55.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:55.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.412146+0000) 2020-02-05T21:56:55.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:56:55.411+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.412911+0000) 2020-02-05T21:56:55.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:55.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.413508+0000) 2020-02-05T21:56:55.415 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:55.412+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.414019+0000) 2020-02-05T21:56:55.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:56:55.413+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.414552+0000) 2020-02-05T21:56:55.416 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:55.414+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.415086+0000) 2020-02-05T21:56:55.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:55.414+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.415235+0000) 2020-02-05T21:56:55.416 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:55.414+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.415537+0000) 2020-02-05T21:56:55.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:56:55.415+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.415983+0000) 2020-02-05T21:56:55.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:56:55.415+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.416411+0000) 2020-02-05T21:56:55.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:56:55.416+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.416923+0000) 2020-02-05T21:56:55.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:55.415+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.416664+0000) 2020-02-05T21:56:55.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:55.417+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.417766+0000) 2020-02-05T21:56:55.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:55.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.421749+0000) 2020-02-05T21:56:55.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:56:55.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100794 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:56:55.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:56:55.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.890462+0000) 2020-02-05T21:56:55.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:56:55.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:55.895723+0000) 2020-02-05T21:56:57.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:56:57.035+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:57.036310+0000) 2020-02-05T21:56:57.038 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:56:57.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:57.036652+0000) 2020-02-05T21:56:57.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:56:57.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:57.043587+0000) 2020-02-05T21:56:57.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:56:57.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:57.044622+0000) 2020-02-05T21:56:58.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:58.623+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:58.624118+0000) 2020-02-05T21:56:59.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:56:59.426+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:56:59.426380+0000) 2020-02-05T21:57:00.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:00.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.033610+0000) 2020-02-05T21:57:00.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:00.268+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.268592+0000) 2020-02-05T21:57:00.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:00.284+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.285934+0000) 2020-02-05T21:57:00.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:00.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.286477+0000) 2020-02-05T21:57:00.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:00.430+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.430695+0000) 2020-02-05T21:57:00.432 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:00.429+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.430935+0000) 2020-02-05T21:57:00.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:00.430+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.431396+0000) 2020-02-05T21:57:00.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:00.430+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.431660+0000) 2020-02-05T21:57:00.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:00.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.432138+0000) 2020-02-05T21:57:00.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:00.432+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.433191+0000) 2020-02-05T21:57:00.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:00.433+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.433720+0000) 2020-02-05T21:57:00.435 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:00.433+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.434192+0000) 2020-02-05T21:57:00.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:00.434+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.434491+0000) 2020-02-05T21:57:00.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:00.434+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.434674+0000) 2020-02-05T21:57:00.436 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:00.434+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.435181+0000) 2020-02-05T21:57:00.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:00.434+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.435047+0000) 2020-02-05T21:57:00.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:00.435+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.436012+0000) 2020-02-05T21:57:00.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:00.441+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:00.442620+0000) 2020-02-05T21:57:00.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:00.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:02.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:02.278+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:02.278885+0000) 2020-02-05T21:57:02.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:02.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:02.677896+0000) 2020-02-05T21:57:05.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:05.448+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.448837+0000) 2020-02-05T21:57:05.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:05.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.449352+0000) 2020-02-05T21:57:05.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:05.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.450334+0000) 2020-02-05T21:57:05.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:05.451+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.451559+0000) 2020-02-05T21:57:05.453 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:05.451+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.451971+0000) 2020-02-05T21:57:05.453 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:05.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.452073+0000) 2020-02-05T21:57:05.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:05.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.452537+0000) 2020-02-05T21:57:05.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:05.452+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.452993+0000) 2020-02-05T21:57:05.454 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:05.453+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.453489+0000) 2020-02-05T21:57:05.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:05.453+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.453986+0000) 2020-02-05T21:57:05.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:05.454+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.454855+0000) 2020-02-05T21:57:05.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:05.454+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.454901+0000) 2020-02-05T21:57:05.456 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:05.454+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.455043+0000) 2020-02-05T21:57:05.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:05.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.456083+0000) 2020-02-05T21:57:05.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:05.461+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:05.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:05.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.890666+0000) 2020-02-05T21:57:05.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:05.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:05.895967+0000) 2020-02-05T21:57:07.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:07.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:07.036550+0000) 2020-02-05T21:57:07.038 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:07.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:07.036912+0000) 2020-02-05T21:57:07.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:07.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:07.043840+0000) 2020-02-05T21:57:07.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:07.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:07.044812+0000) 2020-02-05T21:57:08.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:08.631+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:08.632639+0000) 2020-02-05T21:57:09.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:09.425+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:09.426583+0000) 2020-02-05T21:57:10.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:10.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.033837+0000) 2020-02-05T21:57:10.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:10.268+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.268848+0000) 2020-02-05T21:57:10.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:10.285+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.286133+0000) 2020-02-05T21:57:10.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:10.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.286701+0000) 2020-02-05T21:57:10.471 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:10.469+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.470137+0000) 2020-02-05T21:57:10.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:10.470+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.470737+0000) 2020-02-05T21:57:10.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:10.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.471103+0000) 2020-02-05T21:57:10.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:10.470+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.471204+0000) 2020-02-05T21:57:10.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:10.470+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.471907+0000) 2020-02-05T21:57:10.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:10.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.472440+0000) 2020-02-05T21:57:10.474 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:10.471+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.472845+0000) 2020-02-05T21:57:10.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:10.473+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.473823+0000) 2020-02-05T21:57:10.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:10.473+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.473826+0000) 2020-02-05T21:57:10.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:10.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.474046+0000) 2020-02-05T21:57:10.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:10.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.474360+0000) 2020-02-05T21:57:10.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:10.472+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.473410+0000) 2020-02-05T21:57:10.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:10.472+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.473466+0000) 2020-02-05T21:57:10.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:10.477+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:10.478889+0000) 2020-02-05T21:57:10.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:10.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:12.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:12.278+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:12.279146+0000) 2020-02-05T21:57:12.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:12.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:12.678121+0000) 2020-02-05T21:57:15.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:15.487+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.488122+0000) 2020-02-05T21:57:15.491 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:15.487+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.488460+0000) 2020-02-05T21:57:15.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:15.487+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.488868+0000) 2020-02-05T21:57:15.491 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:15.487+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.487888+0000) 2020-02-05T21:57:15.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:15.490+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.490630+0000) 2020-02-05T21:57:15.493 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:15.490+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.491525+0000) 2020-02-05T21:57:15.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:15.491+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.492308+0000) 2020-02-05T21:57:15.494 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:15.492+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.492898+0000) 2020-02-05T21:57:15.495 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:15.490+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.492056+0000) 2020-02-05T21:57:15.495 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:15.491+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.492861+0000) 2020-02-05T21:57:15.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:15.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.493553+0000) 2020-02-05T21:57:15.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:15.493+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.493872+0000) 2020-02-05T21:57:15.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:15.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.494701+0000) 2020-02-05T21:57:15.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:15.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.494971+0000) 2020-02-05T21:57:15.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:15.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100951 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:15.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:15.889+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.890876+0000) 2020-02-05T21:57:15.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:15.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:15.896188+0000) 2020-02-05T21:57:17.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:17.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:17.036794+0000) 2020-02-05T21:57:17.038 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:17.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:17.037202+0000) 2020-02-05T21:57:17.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:17.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:17.044072+0000) 2020-02-05T21:57:17.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:17.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:17.045034+0000) 2020-02-05T21:57:18.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:18.640+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:18.641484+0000) 2020-02-05T21:57:19.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:19.425+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:19.426858+0000) 2020-02-05T21:57:20.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:20.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.034113+0000) 2020-02-05T21:57:20.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:20.268+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.269096+0000) 2020-02-05T21:57:20.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:20.285+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.286357+0000) 2020-02-05T21:57:20.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:20.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.286977+0000) 2020-02-05T21:57:20.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:20.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.504285+0000) 2020-02-05T21:57:20.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:20.503+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.504663+0000) 2020-02-05T21:57:20.506 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:20.504+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.505418+0000) 2020-02-05T21:57:20.508 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:20.505+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.505849+0000) 2020-02-05T21:57:20.509 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:20.507+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.508559+0000) 2020-02-05T21:57:20.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:20.508+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.508772+0000) 2020-02-05T21:57:20.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:20.508+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.509329+0000) 2020-02-05T21:57:20.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:20.507+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.508961+0000) 2020-02-05T21:57:20.511 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:20.508+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.509583+0000) 2020-02-05T21:57:20.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:20.509+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.510016+0000) 2020-02-05T21:57:20.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:20.509+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.510172+0000) 2020-02-05T21:57:20.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:20.510+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.510454+0000) 2020-02-05T21:57:20.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:20.510+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.511584+0000) 2020-02-05T21:57:20.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:20.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:20.512872+0000) 2020-02-05T21:57:20.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:20.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 100994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:20.639 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:57:20.646 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:57:22.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:22.278+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:22.279391+0000) 2020-02-05T21:57:22.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:22.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:22.678358+0000) 2020-02-05T21:57:25.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:25.522+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.523335+0000) 2020-02-05T21:57:25.525 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:25.522+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.523945+0000) 2020-02-05T21:57:25.525 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:25.523+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.524477+0000) 2020-02-05T21:57:25.526 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:25.523+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.525019+0000) 2020-02-05T21:57:25.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:25.526+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.526472+0000) 2020-02-05T21:57:25.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:25.525+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.526538+0000) 2020-02-05T21:57:25.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:25.525+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.526611+0000) 2020-02-05T21:57:25.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:25.526+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.527120+0000) 2020-02-05T21:57:25.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:25.526+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.527289+0000) 2020-02-05T21:57:25.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:25.526+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.527341+0000) 2020-02-05T21:57:25.530 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:25.527+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.527905+0000) 2020-02-05T21:57:25.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:25.527+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.528145+0000) 2020-02-05T21:57:25.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:25.530+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.531689+0000) 2020-02-05T21:57:25.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:25.530+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.531899+0000) 2020-02-05T21:57:25.536 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:25.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:25.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:25.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.891131+0000) 2020-02-05T21:57:25.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:25.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:25.896419+0000) 2020-02-05T21:57:27.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:27.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:27.037049+0000) 2020-02-05T21:57:27.038 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:27.037+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:27.037482+0000) 2020-02-05T21:57:27.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:27.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:27.044351+0000) 2020-02-05T21:57:27.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:27.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:27.045254+0000) 2020-02-05T21:57:28.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:28.648+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:28.649629+0000) 2020-02-05T21:57:29.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:29.426+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:29.427140+0000) 2020-02-05T21:57:30.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:30.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.034338+0000) 2020-02-05T21:57:30.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:30.269+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.269380+0000) 2020-02-05T21:57:30.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:30.285+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.286587+0000) 2020-02-05T21:57:30.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:30.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.287211+0000) 2020-02-05T21:57:30.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:30.539+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.540952+0000) 2020-02-05T21:57:30.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:30.540+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.541469+0000) 2020-02-05T21:57:30.543 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:30.542+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.542480+0000) 2020-02-05T21:57:30.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:30.542+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.543361+0000) 2020-02-05T21:57:30.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:30.542+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.542969+0000) 2020-02-05T21:57:30.545 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:30.542+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.543919+0000) 2020-02-05T21:57:30.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:30.544+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.544647+0000) 2020-02-05T21:57:30.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:30.543+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.544745+0000) 2020-02-05T21:57:30.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:30.543+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.545060+0000) 2020-02-05T21:57:30.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:30.544+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.545957+0000) 2020-02-05T21:57:30.548 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:30.546+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.547247+0000) 2020-02-05T21:57:30.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:30.546+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.547372+0000) 2020-02-05T21:57:30.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:30.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.552621+0000) 2020-02-05T21:57:30.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:30.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:30.552930+0000) 2020-02-05T21:57:30.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:30.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101073 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:32.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:32.279+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:32.279679+0000) 2020-02-05T21:57:32.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:32.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:32.678592+0000) 2020-02-05T21:57:35.560 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:35.558+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.559554+0000) 2020-02-05T21:57:35.561 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:35.559+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.560387+0000) 2020-02-05T21:57:35.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:35.559+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.560542+0000) 2020-02-05T21:57:35.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:35.559+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.561048+0000) 2020-02-05T21:57:35.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:35.560+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.562355+0000) 2020-02-05T21:57:35.564 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:35.561+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.562657+0000) 2020-02-05T21:57:35.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:35.562+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.563426+0000) 2020-02-05T21:57:35.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:35.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.564281+0000) 2020-02-05T21:57:35.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:35.563+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.565182+0000) 2020-02-05T21:57:35.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:35.563+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.565195+0000) 2020-02-05T21:57:35.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:35.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.564666+0000) 2020-02-05T21:57:35.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:35.563+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.564675+0000) 2020-02-05T21:57:35.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:35.564+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.565945+0000) 2020-02-05T21:57:35.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:35.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.568260+0000) 2020-02-05T21:57:35.572 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:35.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:35.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:35.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.891347+0000) 2020-02-05T21:57:35.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:35.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:35.896713+0000) 2020-02-05T21:57:37.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:37.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:37.037304+0000) 2020-02-05T21:57:37.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:37.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:37.037735+0000) 2020-02-05T21:57:37.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:37.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:37.044587+0000) 2020-02-05T21:57:37.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:37.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:37.045470+0000) 2020-02-05T21:57:38.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:38.655+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:38.656487+0000) 2020-02-05T21:57:39.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:39.426+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:39.427384+0000) 2020-02-05T21:57:40.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:40.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.034573+0000) 2020-02-05T21:57:40.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:40.268+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.269640+0000) 2020-02-05T21:57:40.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:40.286+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.286838+0000) 2020-02-05T21:57:40.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:40.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.287503+0000) 2020-02-05T21:57:40.578 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:40.576+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.577276+0000) 2020-02-05T21:57:40.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:40.576+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.577923+0000) 2020-02-05T21:57:40.581 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:40.578+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.579685+0000) 2020-02-05T21:57:40.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:40.577+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.578501+0000) 2020-02-05T21:57:40.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:40.578+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.579174+0000) 2020-02-05T21:57:40.581 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:40.579+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.580486+0000) 2020-02-05T21:57:40.583 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:40.580+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.581471+0000) 2020-02-05T21:57:40.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:40.581+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.582247+0000) 2020-02-05T21:57:40.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:40.581+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.582760+0000) 2020-02-05T21:57:40.584 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:40.581+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.583392+0000) 2020-02-05T21:57:40.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:40.582+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.583492+0000) 2020-02-05T21:57:40.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:40.582+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.583978+0000) 2020-02-05T21:57:40.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:40.581+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.583006+0000) 2020-02-05T21:57:40.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:40.585+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:40.586433+0000) 2020-02-05T21:57:40.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:40.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101153 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:42.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:42.279+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:42.279990+0000) 2020-02-05T21:57:42.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:42.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:42.678845+0000) 2020-02-05T21:57:45.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:45.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.596407+0000) 2020-02-05T21:57:45.598 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:45.595+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.596872+0000) 2020-02-05T21:57:45.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:45.596+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.597549+0000) 2020-02-05T21:57:45.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:45.596+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.597930+0000) 2020-02-05T21:57:45.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:45.596+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.597907+0000) 2020-02-05T21:57:45.600 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:45.597+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.598424+0000) 2020-02-05T21:57:45.600 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:45.597+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.599129+0000) 2020-02-05T21:57:45.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:45.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.599673+0000) 2020-02-05T21:57:45.601 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:45.599+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.600783+0000) 2020-02-05T21:57:45.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:45.600+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.601107+0000) 2020-02-05T21:57:45.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:45.600+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.602019+0000) 2020-02-05T21:57:45.603 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:45.600+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.601729+0000) 2020-02-05T21:57:45.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:45.601+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.602736+0000) 2020-02-05T21:57:45.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:45.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:45.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:45.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.606373+0000) 2020-02-05T21:57:45.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:45.891+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.891546+0000) 2020-02-05T21:57:45.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:45.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:45.896924+0000) 2020-02-05T21:57:47.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:47.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:47.037552+0000) 2020-02-05T21:57:47.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:47.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:47.038002+0000) 2020-02-05T21:57:47.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:47.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:47.044812+0000) 2020-02-05T21:57:47.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:47.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:47.045652+0000) 2020-02-05T21:57:48.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:48.666+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:48.667205+0000) 2020-02-05T21:57:49.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:49.427+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:49.427614+0000) 2020-02-05T21:57:50.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:50.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.034805+0000) 2020-02-05T21:57:50.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:50.269+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.269985+0000) 2020-02-05T21:57:50.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:50.286+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.287064+0000) 2020-02-05T21:57:50.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:50.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.287749+0000) 2020-02-05T21:57:50.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:50.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.613341+0000) 2020-02-05T21:57:50.615 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:50.612+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.613683+0000) 2020-02-05T21:57:50.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:50.612+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.614140+0000) 2020-02-05T21:57:50.616 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:50.613+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.614563+0000) 2020-02-05T21:57:50.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:50.613+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.615042+0000) 2020-02-05T21:57:50.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:50.614+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.615402+0000) 2020-02-05T21:57:50.617 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:50.614+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.615632+0000) 2020-02-05T21:57:50.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:50.615+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.616577+0000) 2020-02-05T21:57:50.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:50.615+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.617394+0000) 2020-02-05T21:57:50.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:50.615+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.617168+0000) 2020-02-05T21:57:50.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:50.617+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.618305+0000) 2020-02-05T21:57:50.620 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:50.617+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.618599+0000) 2020-02-05T21:57:50.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:50.617+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.618660+0000) 2020-02-05T21:57:50.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:50.622+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:50.624028+0000) 2020-02-05T21:57:50.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:50.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101232 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:50.857 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:57:50.864 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:57:52.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:52.279+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:52.280232+0000) 2020-02-05T21:57:52.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:52.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:52.679134+0000) 2020-02-05T21:57:55.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:55.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.629662+0000) 2020-02-05T21:57:55.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:55.630+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.631749+0000) 2020-02-05T21:57:55.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:57:55.631+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.632176+0000) 2020-02-05T21:57:55.634 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:55.631+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.632839+0000) 2020-02-05T21:57:55.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:57:55.631+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.632959+0000) 2020-02-05T21:57:55.635 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:55.631+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.633035+0000) 2020-02-05T21:57:55.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:57:55.632+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.633793+0000) 2020-02-05T21:57:55.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:55.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.634448+0000) 2020-02-05T21:57:55.636 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:57:55.633+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.634468+0000) 2020-02-05T21:57:55.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:57:55.633+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.635208+0000) 2020-02-05T21:57:55.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:55.634+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.635957+0000) 2020-02-05T21:57:55.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:55.635+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.636139+0000) 2020-02-05T21:57:55.638 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:55.635+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.637000+0000) 2020-02-05T21:57:55.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:55.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.639917+0000) 2020-02-05T21:57:55.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:57:55.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:57:55.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:57:55.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.891755+0000) 2020-02-05T21:57:55.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:57:55.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:55.897167+0000) 2020-02-05T21:57:57.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:57:57.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:57.037778+0000) 2020-02-05T21:57:57.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:57:57.036+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:57.038208+0000) 2020-02-05T21:57:57.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:57:57.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:57.045050+0000) 2020-02-05T21:57:57.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:57:57.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:57.045828+0000) 2020-02-05T21:57:58.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:58.674+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:58.674678+0000) 2020-02-05T21:57:59.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:57:59.427+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:57:59.427840+0000) 2020-02-05T21:58:00.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:00.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.035041+0000) 2020-02-05T21:58:00.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:00.268+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.270234+0000) 2020-02-05T21:58:00.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:00.287+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.287333+0000) 2020-02-05T21:58:00.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:00.287+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.287944+0000) 2020-02-05T21:58:00.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:00.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.650415+0000) 2020-02-05T21:58:00.652 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:00.650+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.650514+0000) 2020-02-05T21:58:00.653 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:00.649+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.651171+0000) 2020-02-05T21:58:00.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:00.652+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.652155+0000) 2020-02-05T21:58:00.654 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:00.651+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.652850+0000) 2020-02-05T21:58:00.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:00.653+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.653453+0000) 2020-02-05T21:58:00.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:00.652+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.654133+0000) 2020-02-05T21:58:00.657 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:00.654+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.654664+0000) 2020-02-05T21:58:00.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:00.654+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.654992+0000) 2020-02-05T21:58:00.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:00.655+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.655281+0000) 2020-02-05T21:58:00.658 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:00.654+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.655906+0000) 2020-02-05T21:58:00.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:00.654+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.656173+0000) 2020-02-05T21:58:00.659 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:00.654+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.656337+0000) 2020-02-05T21:58:00.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:00.660+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:00.660796+0000) 2020-02-05T21:58:00.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:00.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101311 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:02.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:02.279+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:02.280499+0000) 2020-02-05T21:58:02.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:02.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:02.679427+0000) 2020-02-05T21:58:05.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:05.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.667710+0000) 2020-02-05T21:58:05.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:05.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.668771+0000) 2020-02-05T21:58:05.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:05.668+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.669136+0000) 2020-02-05T21:58:05.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:05.669+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.670241+0000) 2020-02-05T21:58:05.671 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:05.669+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.670785+0000) 2020-02-05T21:58:05.672 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:05.669+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.671013+0000) 2020-02-05T21:58:05.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:05.670+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.671524+0000) 2020-02-05T21:58:05.673 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:05.670+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.671202+0000) 2020-02-05T21:58:05.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:05.670+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.671648+0000) 2020-02-05T21:58:05.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:05.670+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.671988+0000) 2020-02-05T21:58:05.674 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:05.670+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.672031+0000) 2020-02-05T21:58:05.674 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:05.671+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.672480+0000) 2020-02-05T21:58:05.674 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:05.671+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.672986+0000) 2020-02-05T21:58:05.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:05.676+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.677643+0000) 2020-02-05T21:58:05.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:05.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:05.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:05.890+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.891976+0000) 2020-02-05T21:58:05.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:05.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:05.897392+0000) 2020-02-05T21:58:07.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:07.037+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:07.038037+0000) 2020-02-05T21:58:07.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:07.037+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:07.038406+0000) 2020-02-05T21:58:07.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:07.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:07.045319+0000) 2020-02-05T21:58:07.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:07.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:07.046048+0000) 2020-02-05T21:58:08.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:08.684+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:08.685705+0000) 2020-02-05T21:58:09.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:09.427+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:09.428069+0000) 2020-02-05T21:58:10.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:10.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.035267+0000) 2020-02-05T21:58:10.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:10.269+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.270490+0000) 2020-02-05T21:58:10.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:10.286+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.287556+0000) 2020-02-05T21:58:10.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:10.287+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.288214+0000) 2020-02-05T21:58:10.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:10.683+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.685001+0000) 2020-02-05T21:58:10.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:10.683+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.685367+0000) 2020-02-05T21:58:10.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:10.684+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.685780+0000) 2020-02-05T21:58:10.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:10.685+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.686235+0000) 2020-02-05T21:58:10.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:10.685+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.686735+0000) 2020-02-05T21:58:10.689 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:10.685+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.687400+0000) 2020-02-05T21:58:10.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:10.686+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.687408+0000) 2020-02-05T21:58:10.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:10.686+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.687971+0000) 2020-02-05T21:58:10.690 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:10.687+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.688798+0000) 2020-02-05T21:58:10.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:10.687+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.688633+0000) 2020-02-05T21:58:10.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:10.688+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.689518+0000) 2020-02-05T21:58:10.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:10.688+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.689481+0000) 2020-02-05T21:58:10.691 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:10.688+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.690278+0000) 2020-02-05T21:58:10.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:10.693+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:10.694185+0000) 2020-02-05T21:58:10.707 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:10.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:12.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:12.280+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:12.280763+0000) 2020-02-05T21:58:12.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:12.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:12.679661+0000) 2020-02-05T21:58:15.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:15.700+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.702276+0000) 2020-02-05T21:58:15.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:15.702+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.703283+0000) 2020-02-05T21:58:15.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:15.703+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.704386+0000) 2020-02-05T21:58:15.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:15.703+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.704777+0000) 2020-02-05T21:58:15.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:15.704+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.705413+0000) 2020-02-05T21:58:15.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:15.704+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.705480+0000) 2020-02-05T21:58:15.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:15.704+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.705897+0000) 2020-02-05T21:58:15.707 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:15.705+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.706261+0000) 2020-02-05T21:58:15.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:15.705+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.706962+0000) 2020-02-05T21:58:15.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:15.706+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.707846+0000) 2020-02-05T21:58:15.710 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:15.706+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.707969+0000) 2020-02-05T21:58:15.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:15.707+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.708726+0000) 2020-02-05T21:58:15.710 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:15.707+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.708626+0000) 2020-02-05T21:58:15.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:15.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.714624+0000) 2020-02-05T21:58:15.717 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:15.715+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:15.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:15.891+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.892194+0000) 2020-02-05T21:58:15.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:15.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:15.897649+0000) 2020-02-05T21:58:17.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:17.036+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:17.038279+0000) 2020-02-05T21:58:17.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:17.037+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:17.038665+0000) 2020-02-05T21:58:17.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:17.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:17.045553+0000) 2020-02-05T21:58:17.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:17.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:17.046282+0000) 2020-02-05T21:58:18.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:18.692+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:18.693691+0000) 2020-02-05T21:58:19.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:19.427+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:19.428325+0000) 2020-02-05T21:58:20.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:20.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.035503+0000) 2020-02-05T21:58:20.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:20.269+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.270707+0000) 2020-02-05T21:58:20.289 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:20.287+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.287785+0000) 2020-02-05T21:58:20.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:20.287+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.288481+0000) 2020-02-05T21:58:20.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:20.721+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.722313+0000) 2020-02-05T21:58:20.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:20.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.723414+0000) 2020-02-05T21:58:20.725 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:20.722+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.724240+0000) 2020-02-05T21:58:20.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:20.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.724396+0000) 2020-02-05T21:58:20.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:20.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.724915+0000) 2020-02-05T21:58:20.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:20.724+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.725208+0000) 2020-02-05T21:58:20.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:20.724+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.725465+0000) 2020-02-05T21:58:20.727 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:20.724+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.726003+0000) 2020-02-05T21:58:20.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:20.725+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.726644+0000) 2020-02-05T21:58:20.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:20.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.726601+0000) 2020-02-05T21:58:20.728 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:20.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.727185+0000) 2020-02-05T21:58:20.729 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:20.725+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.727274+0000) 2020-02-05T21:58:20.729 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:20.726+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.728157+0000) 2020-02-05T21:58:20.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:20.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:20.732745+0000) 2020-02-05T21:58:20.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:20.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:21.253 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:58:21.259 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:58:22.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:22.280+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:22.281050+0000) 2020-02-05T21:58:22.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:22.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:22.679898+0000) 2020-02-05T21:58:25.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:25.738+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.739733+0000) 2020-02-05T21:58:25.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:25.739+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.740662+0000) 2020-02-05T21:58:25.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:25.739+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.741044+0000) 2020-02-05T21:58:25.743 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:25.741+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.742040+0000) 2020-02-05T21:58:25.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:25.741+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.742537+0000) 2020-02-05T21:58:25.744 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:25.742+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.743010+0000) 2020-02-05T21:58:25.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:25.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.743455+0000) 2020-02-05T21:58:25.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:25.743+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.744042+0000) 2020-02-05T21:58:25.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:25.744+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.744532+0000) 2020-02-05T21:58:25.747 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:25.744+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.745394+0000) 2020-02-05T21:58:25.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:25.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.745553+0000) 2020-02-05T21:58:25.747 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:25.744+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.745663+0000) 2020-02-05T21:58:25.748 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:25.745+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.746129+0000) 2020-02-05T21:58:25.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:25.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.751718+0000) 2020-02-05T21:58:25.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:25.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:25.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:25.891+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.892407+0000) 2020-02-05T21:58:25.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:25.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:25.897874+0000) 2020-02-05T21:58:27.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:27.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:27.038535+0000) 2020-02-05T21:58:27.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:27.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:27.038877+0000) 2020-02-05T21:58:27.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:27.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:27.045800+0000) 2020-02-05T21:58:27.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:27.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:27.046490+0000) 2020-02-05T21:58:28.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:28.702+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:28.702935+0000) 2020-02-05T21:58:29.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:29.427+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:29.428551+0000) 2020-02-05T21:58:30.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:30.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.035739+0000) 2020-02-05T21:58:30.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:30.270+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.271005+0000) 2020-02-05T21:58:30.289 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:30.286+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.288037+0000) 2020-02-05T21:58:30.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:30.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.288730+0000) 2020-02-05T21:58:30.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:30.757+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.758107+0000) 2020-02-05T21:58:30.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:30.757+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.758485+0000) 2020-02-05T21:58:30.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:30.757+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.758773+0000) 2020-02-05T21:58:30.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:30.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.759567+0000) 2020-02-05T21:58:30.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:30.759+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.760309+0000) 2020-02-05T21:58:30.762 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:30.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.761005+0000) 2020-02-05T21:58:30.762 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:30.760+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.761311+0000) 2020-02-05T21:58:30.763 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:30.761+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.761883+0000) 2020-02-05T21:58:30.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:30.761+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.762290+0000) 2020-02-05T21:58:30.764 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:30.762+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.762551+0000) 2020-02-05T21:58:30.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:30.762+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.763157+0000) 2020-02-05T21:58:30.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:30.763+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.763938+0000) 2020-02-05T21:58:30.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:30.763+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.764241+0000) 2020-02-05T21:58:30.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:30.765+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:30.766947+0000) 2020-02-05T21:58:30.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:30.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:32.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:32.280+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:32.281307+0000) 2020-02-05T21:58:32.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:32.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:32.680136+0000) 2020-02-05T21:58:35.777 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:35.775+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.775510+0000) 2020-02-05T21:58:35.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:35.775+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.776639+0000) 2020-02-05T21:58:35.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:35.776+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.777205+0000) 2020-02-05T21:58:35.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:35.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.777806+0000) 2020-02-05T21:58:35.779 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:35.778+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.778678+0000) 2020-02-05T21:58:35.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:35.777+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.778795+0000) 2020-02-05T21:58:35.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:35.779+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.779642+0000) 2020-02-05T21:58:35.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:35.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.779681+0000) 2020-02-05T21:58:35.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:35.779+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.780118+0000) 2020-02-05T21:58:35.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:35.780+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.780757+0000) 2020-02-05T21:58:35.782 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:35.780+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.781149+0000) 2020-02-05T21:58:35.783 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:35.781+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.781541+0000) 2020-02-05T21:58:35.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:35.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.782109+0000) 2020-02-05T21:58:35.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:35.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.787514+0000) 2020-02-05T21:58:35.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:35.789+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101590 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:35.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:35.892+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.892648+0000) 2020-02-05T21:58:35.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:35.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:35.898093+0000) 2020-02-05T21:58:37.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:37.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:37.038727+0000) 2020-02-05T21:58:37.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:37.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:37.039107+0000) 2020-02-05T21:58:37.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:37.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:37.046069+0000) 2020-02-05T21:58:37.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:37.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:37.046685+0000) 2020-02-05T21:58:38.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:38.711+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:38.711478+0000) 2020-02-05T21:58:39.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:39.428+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:39.428782+0000) 2020-02-05T21:58:40.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:40.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.035974+0000) 2020-02-05T21:58:40.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:40.270+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.271253+0000) 2020-02-05T21:58:40.289 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:40.287+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.288324+0000) 2020-02-05T21:58:40.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:40.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.288991+0000) 2020-02-05T21:58:40.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:40.793+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.794068+0000) 2020-02-05T21:58:40.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:40.794+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.795426+0000) 2020-02-05T21:58:40.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:40.795+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.796146+0000) 2020-02-05T21:58:40.798 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:40.797+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.797457+0000) 2020-02-05T21:58:40.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:40.796+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.797311+0000) 2020-02-05T21:58:40.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:40.796+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.797755+0000) 2020-02-05T21:58:40.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:40.797+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.798238+0000) 2020-02-05T21:58:40.800 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:40.798+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.798977+0000) 2020-02-05T21:58:40.800 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:40.798+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.799334+0000) 2020-02-05T21:58:40.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:40.798+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.799545+0000) 2020-02-05T21:58:40.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:40.800+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.800716+0000) 2020-02-05T21:58:40.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:40.799+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.800334+0000) 2020-02-05T21:58:40.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:40.800+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.801164+0000) 2020-02-05T21:58:40.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:40.804+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:40.805156+0000) 2020-02-05T21:58:40.808 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:40.806+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:42.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:42.281+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:42.281548+0000) 2020-02-05T21:58:42.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:42.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:42.680393+0000) 2020-02-05T21:58:45.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:45.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.813784+0000) 2020-02-05T21:58:45.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:45.814+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.814673+0000) 2020-02-05T21:58:45.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:45.814+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.814944+0000) 2020-02-05T21:58:45.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:45.814+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.815145+0000) 2020-02-05T21:58:45.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:45.813+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.814855+0000) 2020-02-05T21:58:45.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:45.814+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.815862+0000) 2020-02-05T21:58:45.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:45.814+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.815990+0000) 2020-02-05T21:58:45.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:45.815+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.816254+0000) 2020-02-05T21:58:45.818 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:45.815+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.816674+0000) 2020-02-05T21:58:45.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:45.816+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.817341+0000) 2020-02-05T21:58:45.819 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:45.817+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.818172+0000) 2020-02-05T21:58:45.819 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:45.816+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.817883+0000) 2020-02-05T21:58:45.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:45.818+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.818474+0000) 2020-02-05T21:58:45.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:45.821+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.822937+0000) 2020-02-05T21:58:45.824 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:45.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101674 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:45.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:45.892+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.892855+0000) 2020-02-05T21:58:45.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:45.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:45.898324+0000) 2020-02-05T21:58:47.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:47.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:47.038980+0000) 2020-02-05T21:58:47.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:47.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:47.039353+0000) 2020-02-05T21:58:47.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:47.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:47.046284+0000) 2020-02-05T21:58:47.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:47.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:47.046894+0000) 2020-02-05T21:58:48.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:48.719+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:48.719835+0000) 2020-02-05T21:58:49.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:49.428+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:49.428985+0000) 2020-02-05T21:58:50.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:50.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.036203+0000) 2020-02-05T21:58:50.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:50.271+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.271534+0000) 2020-02-05T21:58:50.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:50.288+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.288594+0000) 2020-02-05T21:58:50.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:50.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.289267+0000) 2020-02-05T21:58:50.831 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:50.829+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.830242+0000) 2020-02-05T21:58:50.832 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:50.829+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.830461+0000) 2020-02-05T21:58:50.833 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:50.831+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.832233+0000) 2020-02-05T21:58:50.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:50.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.833191+0000) 2020-02-05T21:58:50.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:50.832+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.833459+0000) 2020-02-05T21:58:50.835 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:50.833+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.833498+0000) 2020-02-05T21:58:50.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:50.833+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.834348+0000) 2020-02-05T21:58:50.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:50.834+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.835140+0000) 2020-02-05T21:58:50.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:50.835+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.835468+0000) 2020-02-05T21:58:50.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:50.834+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.835246+0000) 2020-02-05T21:58:50.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:50.833+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.834682+0000) 2020-02-05T21:58:50.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:50.833+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.834779+0000) 2020-02-05T21:58:50.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:50.835+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.836480+0000) 2020-02-05T21:58:50.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:50.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:50.837745+0000) 2020-02-05T21:58:50.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:50.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:51.445 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:58:51.452 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:58:52.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:52.281+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:52.281811+0000) 2020-02-05T21:58:52.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:52.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:52.680644+0000) 2020-02-05T21:58:55.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:55.846+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.847251+0000) 2020-02-05T21:58:55.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:58:55.849+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.849502+0000) 2020-02-05T21:58:55.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:55.849+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.850024+0000) 2020-02-05T21:58:55.852 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:58:55.851+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.851312+0000) 2020-02-05T21:58:55.852 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:58:55.850+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.851271+0000) 2020-02-05T21:58:55.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:58:55.851+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.851915+0000) 2020-02-05T21:58:55.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:55.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.852599+0000) 2020-02-05T21:58:55.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:55.852+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.852953+0000) 2020-02-05T21:58:55.854 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:55.853+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.853475+0000) 2020-02-05T21:58:55.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:55.853+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.853398+0000) 2020-02-05T21:58:55.855 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:55.853+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.853821+0000) 2020-02-05T21:58:55.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:58:55.853+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.854172+0000) 2020-02-05T21:58:55.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:55.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.856233+0000) 2020-02-05T21:58:55.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:55.857+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.857185+0000) 2020-02-05T21:58:55.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:58:55.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101756 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:58:55.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:58:55.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.893107+0000) 2020-02-05T21:58:55.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:58:55.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:55.898542+0000) 2020-02-05T21:58:57.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:58:57.039+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:57.039213+0000) 2020-02-05T21:58:57.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:58:57.039+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:57.039614+0000) 2020-02-05T21:58:57.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:58:57.046+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:57.046559+0000) 2020-02-05T21:58:57.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:58:57.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:57.047107+0000) 2020-02-05T21:58:58.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:58.726+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:58.727846+0000) 2020-02-05T21:58:59.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:58:59.428+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:58:59.429332+0000) 2020-02-05T21:59:00.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:00.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.036436+0000) 2020-02-05T21:59:00.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:00.271+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.271830+0000) 2020-02-05T21:59:00.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:00.287+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.288863+0000) 2020-02-05T21:59:00.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:00.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.289510+0000) 2020-02-05T21:59:00.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:00.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.866389+0000) 2020-02-05T21:59:00.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:00.866+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.866532+0000) 2020-02-05T21:59:00.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:00.865+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.866902+0000) 2020-02-05T21:59:00.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:00.866+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.867485+0000) 2020-02-05T21:59:00.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:00.867+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.868113+0000) 2020-02-05T21:59:00.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:00.867+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.868723+0000) 2020-02-05T21:59:00.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:00.868+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.869305+0000) 2020-02-05T21:59:00.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:00.869+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.870128+0000) 2020-02-05T21:59:00.871 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:00.869+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.870666+0000) 2020-02-05T21:59:00.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:00.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.871086+0000) 2020-02-05T21:59:00.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:00.871+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.871514+0000) 2020-02-05T21:59:00.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:00.871+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.872606+0000) 2020-02-05T21:59:00.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:00.872+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.872785+0000) 2020-02-05T21:59:00.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:00.876+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:00.877324+0000) 2020-02-05T21:59:00.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:00.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:02.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:02.281+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:02.282055+0000) 2020-02-05T21:59:02.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:02.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:02.680880+0000) 2020-02-05T21:59:05.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:05.883+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.884795+0000) 2020-02-05T21:59:05.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:05.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.886596+0000) 2020-02-05T21:59:05.888 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:05.885+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.887284+0000) 2020-02-05T21:59:05.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:05.886+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.887314+0000) 2020-02-05T21:59:05.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:05.886+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.887993+0000) 2020-02-05T21:59:05.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:05.887+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.888296+0000) 2020-02-05T21:59:05.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:05.887+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.888665+0000) 2020-02-05T21:59:05.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:05.888+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.889582+0000) 2020-02-05T21:59:05.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:05.888+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.889666+0000) 2020-02-05T21:59:05.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:05.888+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.889869+0000) 2020-02-05T21:59:05.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:05.889+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.890459+0000) 2020-02-05T21:59:05.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:05.889+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.890473+0000) 2020-02-05T21:59:05.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:05.889+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.891114+0000) 2020-02-05T21:59:05.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:05.892+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.893234+0000) 2020-02-05T21:59:05.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:05.895+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.896358+0000) 2020-02-05T21:59:05.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:05.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:05.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:05.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:05.898793+0000) 2020-02-05T21:59:07.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:07.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:07.039497+0000) 2020-02-05T21:59:07.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:07.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:07.039859+0000) 2020-02-05T21:59:07.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:07.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:07.046797+0000) 2020-02-05T21:59:07.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:07.046+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:07.047293+0000) 2020-02-05T21:59:08.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:08.735+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:08.735931+0000) 2020-02-05T21:59:09.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:09.428+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:09.429589+0000) 2020-02-05T21:59:10.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:10.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.036666+0000) 2020-02-05T21:59:10.273 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:10.271+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.272082+0000) 2020-02-05T21:59:10.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:10.288+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.289126+0000) 2020-02-05T21:59:10.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:10.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.289783+0000) 2020-02-05T21:59:10.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:10.901+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.902728+0000) 2020-02-05T21:59:10.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:10.903+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.904315+0000) 2020-02-05T21:59:10.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:10.904+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.905492+0000) 2020-02-05T21:59:10.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:10.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.906181+0000) 2020-02-05T21:59:10.908 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:10.905+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.906693+0000) 2020-02-05T21:59:10.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:10.905+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.906725+0000) 2020-02-05T21:59:10.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:10.906+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.907554+0000) 2020-02-05T21:59:10.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:10.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.907226+0000) 2020-02-05T21:59:10.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:10.906+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.907598+0000) 2020-02-05T21:59:10.911 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:10.906+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.908146+0000) 2020-02-05T21:59:10.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:10.907+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.908470+0000) 2020-02-05T21:59:10.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:10.908+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.909467+0000) 2020-02-05T21:59:10.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:10.907+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.908895+0000) 2020-02-05T21:59:10.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:10.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:10.913756+0000) 2020-02-05T21:59:10.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:10.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:12.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:12.280+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:12.282399+0000) 2020-02-05T21:59:12.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:12.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:12.681173+0000) 2020-02-05T21:59:15.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:15.892+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.893409+0000) 2020-02-05T21:59:15.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:15.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.898987+0000) 2020-02-05T21:59:15.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:15.919+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.920841+0000) 2020-02-05T21:59:15.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:15.920+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.921683+0000) 2020-02-05T21:59:15.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:15.921+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.922282+0000) 2020-02-05T21:59:15.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:15.921+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.922963+0000) 2020-02-05T21:59:15.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:15.923+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.924322+0000) 2020-02-05T21:59:15.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:15.922+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.924035+0000) 2020-02-05T21:59:15.926 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:15.922+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.924142+0000) 2020-02-05T21:59:15.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:15.923+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.924703+0000) 2020-02-05T21:59:15.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:15.923+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.924884+0000) 2020-02-05T21:59:15.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:15.924+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.926184+0000) 2020-02-05T21:59:15.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:15.925+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.926417+0000) 2020-02-05T21:59:15.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:15.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.926631+0000) 2020-02-05T21:59:15.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:15.925+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.927137+0000) 2020-02-05T21:59:15.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:15.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:15.931776+0000) 2020-02-05T21:59:15.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:15.933+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101922 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:17.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:17.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:17.039758+0000) 2020-02-05T21:59:17.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:17.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:17.040087+0000) 2020-02-05T21:59:17.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:17.046+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:17.047040+0000) 2020-02-05T21:59:17.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:17.046+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:17.047501+0000) 2020-02-05T21:59:18.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:18.742+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:18.743993+0000) 2020-02-05T21:59:19.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:19.429+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:19.429813+0000) 2020-02-05T21:59:20.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:20.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.036831+0000) 2020-02-05T21:59:20.273 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:20.271+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.272315+0000) 2020-02-05T21:59:20.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:20.288+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.289364+0000) 2020-02-05T21:59:20.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:20.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.290019+0000) 2020-02-05T21:59:20.941 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:20.938+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.939738+0000) 2020-02-05T21:59:20.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:20.939+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.940739+0000) 2020-02-05T21:59:20.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:20.940+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.941369+0000) 2020-02-05T21:59:20.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:20.940+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.941735+0000) 2020-02-05T21:59:20.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:20.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.942204+0000) 2020-02-05T21:59:20.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:20.941+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.942520+0000) 2020-02-05T21:59:20.945 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:20.941+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.943039+0000) 2020-02-05T21:59:20.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:20.942+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.943891+0000) 2020-02-05T21:59:20.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:20.942+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.943173+0000) 2020-02-05T21:59:20.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:20.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.943508+0000) 2020-02-05T21:59:20.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:20.942+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.944344+0000) 2020-02-05T21:59:20.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:20.943+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.944489+0000) 2020-02-05T21:59:20.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:20.949+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.950892+0000) 2020-02-05T21:59:20.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:20.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:20.951214+0000) 2020-02-05T21:59:20.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:20.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 101969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:21.578 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:59:21.584 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:59:22.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:22.281+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:22.282735+0000) 2020-02-05T21:59:22.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:22.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:22.681398+0000) 2020-02-05T21:59:25.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:25.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.893663+0000) 2020-02-05T21:59:25.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:25.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.899217+0000) 2020-02-05T21:59:25.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:25.956+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.957121+0000) 2020-02-05T21:59:25.959 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:25.956+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.957610+0000) 2020-02-05T21:59:25.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:25.956+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.958298+0000) 2020-02-05T21:59:25.960 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:25.957+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.958683+0000) 2020-02-05T21:59:25.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:25.957+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.959352+0000) 2020-02-05T21:59:25.961 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:25.958+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.960225+0000) 2020-02-05T21:59:25.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:25.959+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.960338+0000) 2020-02-05T21:59:25.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:25.959+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.961350+0000) 2020-02-05T21:59:25.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:25.960+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.961268+0000) 2020-02-05T21:59:25.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:25.960+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.962062+0000) 2020-02-05T21:59:25.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:25.961+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.962480+0000) 2020-02-05T21:59:25.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:25.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.961612+0000) 2020-02-05T21:59:25.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:25.961+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.962490+0000) 2020-02-05T21:59:25.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:25.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:25.967164+0000) 2020-02-05T21:59:25.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:25.967+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:27.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:27.038+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:27.039995+0000) 2020-02-05T21:59:27.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:27.038+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:27.040392+0000) 2020-02-05T21:59:27.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:27.046+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:27.047261+0000) 2020-02-05T21:59:27.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:27.046+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:27.047684+0000) 2020-02-05T21:59:28.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:28.751+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:28.752176+0000) 2020-02-05T21:59:29.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:29.429+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:29.430045+0000) 2020-02-05T21:59:30.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:30.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.037099+0000) 2020-02-05T21:59:30.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:30.271+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.272561+0000) 2020-02-05T21:59:30.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:30.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.289589+0000) 2020-02-05T21:59:30.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:30.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.290271+0000) 2020-02-05T21:59:30.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:30.974+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.975515+0000) 2020-02-05T21:59:30.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:30.974+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.975693+0000) 2020-02-05T21:59:30.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:30.975+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.976122+0000) 2020-02-05T21:59:30.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:30.975+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.976598+0000) 2020-02-05T21:59:30.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:30.975+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.976581+0000) 2020-02-05T21:59:30.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:30.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.977216+0000) 2020-02-05T21:59:30.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:30.975+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.976995+0000) 2020-02-05T21:59:30.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:30.976+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.977543+0000) 2020-02-05T21:59:30.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:30.977+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.978401+0000) 2020-02-05T21:59:30.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:30.977+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.978509+0000) 2020-02-05T21:59:30.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:30.977+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.978605+0000) 2020-02-05T21:59:30.981 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:30.977+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.978863+0000) 2020-02-05T21:59:30.981 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:30.977+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.979371+0000) 2020-02-05T21:59:30.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:30.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:30.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:30.985+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:30.986518+0000) 2020-02-05T21:59:32.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:32.282+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:32.282993+0000) 2020-02-05T21:59:32.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:32.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:32.681626+0000) 2020-02-05T21:59:35.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:35.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.893892+0000) 2020-02-05T21:59:35.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:35.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.899486+0000) 2020-02-05T21:59:35.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:35.989+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.991249+0000) 2020-02-05T21:59:35.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:35.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.991815+0000) 2020-02-05T21:59:35.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:35.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.992336+0000) 2020-02-05T21:59:35.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:35.991+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.993204+0000) 2020-02-05T21:59:35.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:35.992+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.993550+0000) 2020-02-05T21:59:35.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:35.993+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.994112+0000) 2020-02-05T21:59:35.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:35.993+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.994604+0000) 2020-02-05T21:59:35.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:35.994+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.995183+0000) 2020-02-05T21:59:35.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:35.994+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.995816+0000) 2020-02-05T21:59:35.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:35.995+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.996901+0000) 2020-02-05T21:59:35.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:35.996+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.997523+0000) 2020-02-05T21:59:35.999 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:35.995+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.997345+0000) 2020-02-05T21:59:35.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:35.996+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:35.997687+0000) 2020-02-05T21:59:36.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:36.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:36.001749+0000) 2020-02-05T21:59:36.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:36.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:37.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:37.039+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:37.040234+0000) 2020-02-05T21:59:37.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:37.039+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:37.040690+0000) 2020-02-05T21:59:37.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:37.047+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:37.047535+0000) 2020-02-05T21:59:37.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:37.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:37.047941+0000) 2020-02-05T21:59:38.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:38.760+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:38.760819+0000) 2020-02-05T21:59:39.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:39.430+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:39.430318+0000) 2020-02-05T21:59:40.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:40.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:40.037313+0000) 2020-02-05T21:59:40.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:40.272+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:40.272812+0000) 2020-02-05T21:59:40.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:40.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:40.289816+0000) 2020-02-05T21:59:40.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:40.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:40.290515+0000) 2020-02-05T21:59:41.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:41.008+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.009840+0000) 2020-02-05T21:59:41.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:41.009+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.010253+0000) 2020-02-05T21:59:41.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:41.009+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.010923+0000) 2020-02-05T21:59:41.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:41.010+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.011664+0000) 2020-02-05T21:59:41.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:41.010+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.011971+0000) 2020-02-05T21:59:41.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:41.010+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.012173+0000) 2020-02-05T21:59:41.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:41.011+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.012434+0000) 2020-02-05T21:59:41.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:41.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.012645+0000) 2020-02-05T21:59:41.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:41.011+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.012764+0000) 2020-02-05T21:59:41.015 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:41.011+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.012556+0000) 2020-02-05T21:59:41.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:41.011+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.013253+0000) 2020-02-05T21:59:41.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:41.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.013856+0000) 2020-02-05T21:59:41.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:41.012+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.013861+0000) 2020-02-05T21:59:41.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:41.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:41.019574+0000) 2020-02-05T21:59:41.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:41.018+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:42.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:42.282+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:42.283247+0000) 2020-02-05T21:59:42.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:42.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:42.681861+0000) 2020-02-05T21:59:45.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:45.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:45.894128+0000) 2020-02-05T21:59:45.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:45.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:45.899720+0000) 2020-02-05T21:59:46.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:46.024+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.025340+0000) 2020-02-05T21:59:46.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:46.024+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.026216+0000) 2020-02-05T21:59:46.029 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:46.026+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.027991+0000) 2020-02-05T21:59:46.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:46.027+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.028314+0000) 2020-02-05T21:59:46.029 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:46.027+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.028545+0000) 2020-02-05T21:59:46.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:46.027+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.028767+0000) 2020-02-05T21:59:46.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:46.027+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.029343+0000) 2020-02-05T21:59:46.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:46.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.029163+0000) 2020-02-05T21:59:46.031 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:46.028+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.029548+0000) 2020-02-05T21:59:46.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:46.028+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.029656+0000) 2020-02-05T21:59:46.032 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:46.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.030019+0000) 2020-02-05T21:59:46.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:46.028+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.029990+0000) 2020-02-05T21:59:46.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:46.034+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.035288+0000) 2020-02-05T21:59:46.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:46.034+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:46.035485+0000) 2020-02-05T21:59:46.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:46.036+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102172 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:47.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:47.039+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:47.040486+0000) 2020-02-05T21:59:47.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:47.039+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:47.040947+0000) 2020-02-05T21:59:47.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:47.047+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:47.047773+0000) 2020-02-05T21:59:47.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:47.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:47.048231+0000) 2020-02-05T21:59:48.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:48.769+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:48.769278+0000) 2020-02-05T21:59:49.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:49.430+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:49.430606+0000) 2020-02-05T21:59:50.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:50.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:50.037560+0000) 2020-02-05T21:59:50.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:50.272+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:50.273046+0000) 2020-02-05T21:59:50.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:50.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:50.290053+0000) 2020-02-05T21:59:50.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:50.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:50.290793+0000) 2020-02-05T21:59:51.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:51.042+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.042610+0000) 2020-02-05T21:59:51.044 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:51.043+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.043146+0000) 2020-02-05T21:59:51.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:51.042+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.043877+0000) 2020-02-05T21:59:51.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:51.043+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.044588+0000) 2020-02-05T21:59:51.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:51.044+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.045011+0000) 2020-02-05T21:59:51.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:51.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.045427+0000) 2020-02-05T21:59:51.047 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:51.045+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.045856+0000) 2020-02-05T21:59:51.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:51.045+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.046702+0000) 2020-02-05T21:59:51.048 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:51.045+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.047373+0000) 2020-02-05T21:59:51.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:51.047+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.047561+0000) 2020-02-05T21:59:51.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:51.047+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.048044+0000) 2020-02-05T21:59:51.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:51.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.048663+0000) 2020-02-05T21:59:51.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:51.047+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.048896+0000) 2020-02-05T21:59:51.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:51.053+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:51.053465+0000) 2020-02-05T21:59:51.057 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:51.054+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:51.768 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:59:51.774 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T21:59:52.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:52.282+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:52.283511+0000) 2020-02-05T21:59:52.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:52.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:52.682127+0000) 2020-02-05T21:59:55.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:55.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:55.894322+0000) 2020-02-05T21:59:55.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:55.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:55.899965+0000) 2020-02-05T21:59:56.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T21:59:56.060+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.061513+0000) 2020-02-05T21:59:56.063 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T21:59:56.060+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.061803+0000) 2020-02-05T21:59:56.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:56.063+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.064642+0000) 2020-02-05T21:59:56.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T21:59:56.064+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.065267+0000) 2020-02-05T21:59:56.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:56.064+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.065800+0000) 2020-02-05T21:59:56.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:56.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.067613+0000) 2020-02-05T21:59:56.072 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:56.065+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.066450+0000) 2020-02-05T21:59:56.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T21:59:56.063+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.063938+0000) 2020-02-05T21:59:56.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:56.064+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.065101+0000) 2020-02-05T21:59:56.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:56.066+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.066964+0000) 2020-02-05T21:59:56.072 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T21:59:56.066+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.067210+0000) 2020-02-05T21:59:56.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T21:59:56.067+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.067454+0000) 2020-02-05T21:59:56.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T21:59:56.067+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.068377+0000) 2020-02-05T21:59:56.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:56.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:56.071390+0000) 2020-02-05T21:59:56.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T21:59:56.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102254 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T21:59:57.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T21:59:57.040+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:57.040739+0000) 2020-02-05T21:59:57.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T21:59:57.040+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:57.041159+0000) 2020-02-05T21:59:57.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T21:59:57.046+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:57.048030+0000) 2020-02-05T21:59:57.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T21:59:57.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:57.048424+0000) 2020-02-05T21:59:58.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:58.778+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:58.779886+0000) 2020-02-05T21:59:59.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T21:59:59.429+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T20:59:59.430833+0000) 2020-02-05T22:00:00.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:00.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:00.037786+0000) 2020-02-05T22:00:00.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:00.272+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:00.273295+0000) 2020-02-05T22:00:00.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:00.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:00.290263+0000) 2020-02-05T22:00:00.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:00.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:00.290988+0000) 2020-02-05T22:00:01.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:01.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.079881+0000) 2020-02-05T22:00:01.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:01.079+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.080404+0000) 2020-02-05T22:00:01.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:01.079+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.080580+0000) 2020-02-05T22:00:01.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:01.079+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.080981+0000) 2020-02-05T22:00:01.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:01.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.081497+0000) 2020-02-05T22:00:01.083 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:01.080+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.081746+0000) 2020-02-05T22:00:01.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:01.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.082047+0000) 2020-02-05T22:00:01.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:01.082+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.082468+0000) 2020-02-05T22:00:01.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:01.083+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.083419+0000) 2020-02-05T22:00:01.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:01.083+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.083674+0000) 2020-02-05T22:00:01.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:01.083+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.084130+0000) 2020-02-05T22:00:01.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:01.083+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.083779+0000) 2020-02-05T22:00:01.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:01.084+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.085024+0000) 2020-02-05T22:00:01.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:01.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:01.088021+0000) 2020-02-05T22:00:01.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:01.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:02.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:02.283+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:02.283810+0000) 2020-02-05T22:00:02.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:02.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:02.682307+0000) 2020-02-05T22:00:05.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:05.893+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:05.894528+0000) 2020-02-05T22:00:05.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:05.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:05.900210+0000) 2020-02-05T22:00:06.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:06.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.098054+0000) 2020-02-05T22:00:06.100 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:06.098+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.099216+0000) 2020-02-05T22:00:06.100 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:06.098+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.099372+0000) 2020-02-05T22:00:06.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:06.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.100093+0000) 2020-02-05T22:00:06.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:06.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.100163+0000) 2020-02-05T22:00:06.102 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:06.099+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.100501+0000) 2020-02-05T22:00:06.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:06.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.101113+0000) 2020-02-05T22:00:06.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:06.100+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.101419+0000) 2020-02-05T22:00:06.103 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:06.100+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.101371+0000) 2020-02-05T22:00:06.103 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:06.100+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.101861+0000) 2020-02-05T22:00:06.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:06.101+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.102135+0000) 2020-02-05T22:00:06.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:06.103+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.103766+0000) 2020-02-05T22:00:06.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:06.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.108721+0000) 2020-02-05T22:00:06.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:06.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:06.109027+0000) 2020-02-05T22:00:06.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:06.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:07.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:07.040+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:07.040969+0000) 2020-02-05T22:00:07.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:07.040+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:07.041334+0000) 2020-02-05T22:00:07.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:07.047+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:07.048276+0000) 2020-02-05T22:00:07.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:07.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:07.048595+0000) 2020-02-05T22:00:08.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:08.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:08.788278+0000) 2020-02-05T22:00:09.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:09.430+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:09.431065+0000) 2020-02-05T22:00:10.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:10.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:10.038047+0000) 2020-02-05T22:00:10.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:10.273+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:10.273549+0000) 2020-02-05T22:00:10.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:10.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:10.290492+0000) 2020-02-05T22:00:10.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:10.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:10.291221+0000) 2020-02-05T22:00:11.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:11.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.116472+0000) 2020-02-05T22:00:11.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:11.117+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.117694+0000) 2020-02-05T22:00:11.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:11.116+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.117191+0000) 2020-02-05T22:00:11.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:11.118+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.119016+0000) 2020-02-05T22:00:11.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:11.118+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.119166+0000) 2020-02-05T22:00:11.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:11.118+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.119762+0000) 2020-02-05T22:00:11.121 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:11.119+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.120163+0000) 2020-02-05T22:00:11.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:11.120+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.120997+0000) 2020-02-05T22:00:11.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:11.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.121113+0000) 2020-02-05T22:00:11.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:11.120+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.121612+0000) 2020-02-05T22:00:11.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:11.121+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.122230+0000) 2020-02-05T22:00:11.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:11.121+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.122573+0000) 2020-02-05T22:00:11.124 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:11.122+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.123056+0000) 2020-02-05T22:00:11.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:11.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:11.125808+0000) 2020-02-05T22:00:11.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:11.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102384 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:12.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:12.283+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:12.284067+0000) 2020-02-05T22:00:12.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:12.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:12.682520+0000) 2020-02-05T22:00:15.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:15.894+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:15.894762+0000) 2020-02-05T22:00:15.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:15.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:15.900482+0000) 2020-02-05T22:00:16.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:16.133+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.134705+0000) 2020-02-05T22:00:16.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:16.134+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.134780+0000) 2020-02-05T22:00:16.137 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:16.134+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.135659+0000) 2020-02-05T22:00:16.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:16.134+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.136015+0000) 2020-02-05T22:00:16.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:16.135+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.136022+0000) 2020-02-05T22:00:16.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:16.135+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.136286+0000) 2020-02-05T22:00:16.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:16.135+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.136598+0000) 2020-02-05T22:00:16.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:16.135+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.136877+0000) 2020-02-05T22:00:16.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:16.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.137414+0000) 2020-02-05T22:00:16.140 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:16.137+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.137712+0000) 2020-02-05T22:00:16.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:16.138+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.138424+0000) 2020-02-05T22:00:16.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:16.137+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.138806+0000) 2020-02-05T22:00:16.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:16.139+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.139639+0000) 2020-02-05T22:00:16.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:16.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:16.144699+0000) 2020-02-05T22:00:16.147 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:16.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:17.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:17.041+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:17.041247+0000) 2020-02-05T22:00:17.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:17.041+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:17.041503+0000) 2020-02-05T22:00:17.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:17.047+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:17.048481+0000) 2020-02-05T22:00:17.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:17.047+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:17.048819+0000) 2020-02-05T22:00:18.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:18.795+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:18.795702+0000) 2020-02-05T22:00:19.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:19.430+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:19.431351+0000) 2020-02-05T22:00:20.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:20.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:20.038311+0000) 2020-02-05T22:00:20.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:20.273+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:20.273781+0000) 2020-02-05T22:00:20.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:20.289+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:20.290739+0000) 2020-02-05T22:00:20.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:20.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:20.291500+0000) 2020-02-05T22:00:21.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:21.152+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.153121+0000) 2020-02-05T22:00:21.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:21.153+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.153756+0000) 2020-02-05T22:00:21.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:21.153+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.154260+0000) 2020-02-05T22:00:21.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:21.154+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.155246+0000) 2020-02-05T22:00:21.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:21.154+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.155971+0000) 2020-02-05T22:00:21.157 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:21.155+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.156330+0000) 2020-02-05T22:00:21.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:21.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.156741+0000) 2020-02-05T22:00:21.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:21.156+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.156601+0000) 2020-02-05T22:00:21.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:21.156+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.157342+0000) 2020-02-05T22:00:21.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:21.156+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.157331+0000) 2020-02-05T22:00:21.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:21.157+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.158338+0000) 2020-02-05T22:00:21.160 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:21.157+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.158401+0000) 2020-02-05T22:00:21.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:21.158+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.158938+0000) 2020-02-05T22:00:21.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:21.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:21.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:21.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:21.163200+0000) 2020-02-05T22:00:21.956 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:00:21.962 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:00:22.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:22.284+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:22.284316+0000) 2020-02-05T22:00:22.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:22.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:22.682785+0000) 2020-02-05T22:00:25.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:25.894+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:25.894992+0000) 2020-02-05T22:00:25.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:25.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:25.900720+0000) 2020-02-05T22:00:26.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:26.170+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.171765+0000) 2020-02-05T22:00:26.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:26.172+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.172724+0000) 2020-02-05T22:00:26.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:26.171+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.173030+0000) 2020-02-05T22:00:26.174 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:26.172+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.173441+0000) 2020-02-05T22:00:26.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:26.173+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.174450+0000) 2020-02-05T22:00:26.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:26.173+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.174968+0000) 2020-02-05T22:00:26.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:26.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.175391+0000) 2020-02-05T22:00:26.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:26.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.175516+0000) 2020-02-05T22:00:26.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:26.175+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.176255+0000) 2020-02-05T22:00:26.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:26.176+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.176710+0000) 2020-02-05T22:00:26.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:26.176+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.177129+0000) 2020-02-05T22:00:26.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:26.176+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.177018+0000) 2020-02-05T22:00:26.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:26.175+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.176789+0000) 2020-02-05T22:00:26.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:26.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:26.178586+0000) 2020-02-05T22:00:26.184 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:26.182+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:27.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:27.041+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:27.041490+0000) 2020-02-05T22:00:27.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:27.041+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:27.041719+0000) 2020-02-05T22:00:27.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:27.048+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:27.048688+0000) 2020-02-05T22:00:27.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:27.048+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:27.049125+0000) 2020-02-05T22:00:28.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:28.804+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:28.804620+0000) 2020-02-05T22:00:29.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:29.430+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:29.431606+0000) 2020-02-05T22:00:30.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:30.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:30.038583+0000) 2020-02-05T22:00:30.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:30.273+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:30.274035+0000) 2020-02-05T22:00:30.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:30.290+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:30.290967+0000) 2020-02-05T22:00:30.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:30.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:30.291720+0000) 2020-02-05T22:00:31.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:31.187+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.188693+0000) 2020-02-05T22:00:31.191 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:31.189+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.190207+0000) 2020-02-05T22:00:31.192 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:31.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.190038+0000) 2020-02-05T22:00:31.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:31.189+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.190780+0000) 2020-02-05T22:00:31.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:31.190+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.191711+0000) 2020-02-05T22:00:31.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:31.191+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.191915+0000) 2020-02-05T22:00:31.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:31.192+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.192880+0000) 2020-02-05T22:00:31.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:31.193+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.193517+0000) 2020-02-05T22:00:31.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:31.192+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.193968+0000) 2020-02-05T22:00:31.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:31.193+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.194383+0000) 2020-02-05T22:00:31.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:31.193+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.194409+0000) 2020-02-05T22:00:31.196 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:31.194+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.194958+0000) 2020-02-05T22:00:31.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:31.193+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.195032+0000) 2020-02-05T22:00:31.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:31.198+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:31.199133+0000) 2020-02-05T22:00:31.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:31.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:32.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:32.284+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:32.284573+0000) 2020-02-05T22:00:32.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:32.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:32.683048+0000) 2020-02-05T22:00:35.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:35.894+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:35.895217+0000) 2020-02-05T22:00:35.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:35.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:35.900968+0000) 2020-02-05T22:00:36.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:36.205+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.207002+0000) 2020-02-05T22:00:36.209 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:36.205+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.206973+0000) 2020-02-05T22:00:36.209 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:36.206+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.207662+0000) 2020-02-05T22:00:36.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:36.207+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.208337+0000) 2020-02-05T22:00:36.210 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:36.208+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.209230+0000) 2020-02-05T22:00:36.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:36.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.209916+0000) 2020-02-05T22:00:36.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:36.210+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.211686+0000) 2020-02-05T22:00:36.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:36.211+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.212534+0000) 2020-02-05T22:00:36.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:36.211+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.212654+0000) 2020-02-05T22:00:36.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:36.211+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.212837+0000) 2020-02-05T22:00:36.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:36.211+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.213311+0000) 2020-02-05T22:00:36.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:36.212+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.214253+0000) 2020-02-05T22:00:36.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:36.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.218461+0000) 2020-02-05T22:00:36.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:36.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:36.218774+0000) 2020-02-05T22:00:36.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:36.218+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:37.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:37.040+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:37.041764+0000) 2020-02-05T22:00:37.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:37.040+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:37.041946+0000) 2020-02-05T22:00:37.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:37.048+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:37.048895+0000) 2020-02-05T22:00:37.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:37.048+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:37.049305+0000) 2020-02-05T22:00:38.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:38.811+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:38.812023+0000) 2020-02-05T22:00:39.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:39.431+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:39.431853+0000) 2020-02-05T22:00:40.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:40.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:40.038824+0000) 2020-02-05T22:00:40.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:40.273+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:40.274278+0000) 2020-02-05T22:00:40.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:40.291+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:40.291186+0000) 2020-02-05T22:00:40.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:40.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:40.291984+0000) 2020-02-05T22:00:41.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:41.224+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.225981+0000) 2020-02-05T22:00:41.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:41.225+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.226862+0000) 2020-02-05T22:00:41.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:41.225+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.226960+0000) 2020-02-05T22:00:41.228 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:41.226+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.227315+0000) 2020-02-05T22:00:41.229 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:41.226+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.227782+0000) 2020-02-05T22:00:41.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:41.227+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.228581+0000) 2020-02-05T22:00:41.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:41.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.228876+0000) 2020-02-05T22:00:41.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:41.227+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.229323+0000) 2020-02-05T22:00:41.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:41.227+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.229380+0000) 2020-02-05T22:00:41.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:41.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.229692+0000) 2020-02-05T22:00:41.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:41.229+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.230792+0000) 2020-02-05T22:00:41.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:41.229+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.231318+0000) 2020-02-05T22:00:41.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:41.230+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.231889+0000) 2020-02-05T22:00:41.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:41.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:41.234763+0000) 2020-02-05T22:00:41.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:41.237+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:42.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:42.283+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:42.284836+0000) 2020-02-05T22:00:42.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:42.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:42.683303+0000) 2020-02-05T22:00:45.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:45.895+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:45.895435+0000) 2020-02-05T22:00:45.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:45.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:45.901199+0000) 2020-02-05T22:00:46.244 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:46.242+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.243054+0000) 2020-02-05T22:00:46.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:46.243+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.243899+0000) 2020-02-05T22:00:46.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:46.243+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.244990+0000) 2020-02-05T22:00:46.246 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:46.245+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.245067+0000) 2020-02-05T22:00:46.247 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:46.244+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.246012+0000) 2020-02-05T22:00:46.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:46.244+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.246305+0000) 2020-02-05T22:00:46.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:46.246+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.246479+0000) 2020-02-05T22:00:46.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:46.245+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.247042+0000) 2020-02-05T22:00:46.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:46.247+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.247352+0000) 2020-02-05T22:00:46.249 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:46.248+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.248459+0000) 2020-02-05T22:00:46.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:46.247+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.248568+0000) 2020-02-05T22:00:46.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:46.247+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.248888+0000) 2020-02-05T22:00:46.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:46.249+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.249427+0000) 2020-02-05T22:00:46.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:46.253+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:46.253780+0000) 2020-02-05T22:00:46.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:46.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102670 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:47.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:47.041+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:47.042030+0000) 2020-02-05T22:00:47.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:47.041+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:47.042167+0000) 2020-02-05T22:00:47.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:47.048+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:47.049137+0000) 2020-02-05T22:00:47.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:47.048+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:47.049516+0000) 2020-02-05T22:00:48.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:48.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:48.820737+0000) 2020-02-05T22:00:49.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:49.431+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:49.432115+0000) 2020-02-05T22:00:50.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:50.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:50.039056+0000) 2020-02-05T22:00:50.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:50.273+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:50.274521+0000) 2020-02-05T22:00:50.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:50.290+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:50.291438+0000) 2020-02-05T22:00:50.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:50.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:50.292222+0000) 2020-02-05T22:00:51.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:51.261+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.262407+0000) 2020-02-05T22:00:51.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:51.261+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.262901+0000) 2020-02-05T22:00:51.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:51.262+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.263834+0000) 2020-02-05T22:00:51.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:51.263+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.264893+0000) 2020-02-05T22:00:51.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:51.264+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.266010+0000) 2020-02-05T22:00:51.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:51.264+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.265772+0000) 2020-02-05T22:00:51.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:51.265+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.266891+0000) 2020-02-05T22:00:51.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:51.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.267153+0000) 2020-02-05T22:00:51.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:51.266+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.267377+0000) 2020-02-05T22:00:51.269 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:51.266+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.268186+0000) 2020-02-05T22:00:51.269 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:51.266+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.267641+0000) 2020-02-05T22:00:51.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:51.267+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.268619+0000) 2020-02-05T22:00:51.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:51.267+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.268723+0000) 2020-02-05T22:00:51.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:51.273+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:51.274465+0000) 2020-02-05T22:00:51.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:51.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:52.170 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:00:52.176 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:00:52.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:52.283+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:52.285107+0000) 2020-02-05T22:00:52.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:52.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:52.683549+0000) 2020-02-05T22:00:55.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:55.894+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:55.895680+0000) 2020-02-05T22:00:55.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:55.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:55.901506+0000) 2020-02-05T22:00:56.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:56.280+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.282261+0000) 2020-02-05T22:00:56.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:00:56.281+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.283015+0000) 2020-02-05T22:00:56.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:00:56.282+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.284191+0000) 2020-02-05T22:00:56.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:00:56.283+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.284605+0000) 2020-02-05T22:00:56.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:00:56.283+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.284959+0000) 2020-02-05T22:00:56.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:00:56.284+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.285433+0000) 2020-02-05T22:00:56.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:56.284+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.285787+0000) 2020-02-05T22:00:56.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:00:56.285+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.286647+0000) 2020-02-05T22:00:56.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:56.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.287140+0000) 2020-02-05T22:00:56.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:56.286+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.287805+0000) 2020-02-05T22:00:56.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:56.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.287759+0000) 2020-02-05T22:00:56.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:56.286+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.287948+0000) 2020-02-05T22:00:56.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:00:56.286+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.288190+0000) 2020-02-05T22:00:56.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:56.290+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:56.292032+0000) 2020-02-05T22:00:56.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:00:56.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:00:57.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:00:57.041+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:57.042244+0000) 2020-02-05T22:00:57.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:00:57.041+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:57.042387+0000) 2020-02-05T22:00:57.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:00:57.048+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:57.049337+0000) 2020-02-05T22:00:57.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:00:57.048+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:57.049665+0000) 2020-02-05T22:00:58.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:58.831+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:58.832728+0000) 2020-02-05T22:00:59.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:00:59.431+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:00:59.432395+0000) 2020-02-05T22:01:00.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:00.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:00.039355+0000) 2020-02-05T22:01:00.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:00.274+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:00.274763+0000) 2020-02-05T22:01:00.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:00.290+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:00.291668+0000) 2020-02-05T22:01:00.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:00.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:00.292487+0000) 2020-02-05T22:01:01.302 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:01.299+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.300527+0000) 2020-02-05T22:01:01.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:01.300+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.301514+0000) 2020-02-05T22:01:01.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:01.300+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.302109+0000) 2020-02-05T22:01:01.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:01.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.303163+0000) 2020-02-05T22:01:01.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:01.301+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.303266+0000) 2020-02-05T22:01:01.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:01.302+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.303756+0000) 2020-02-05T22:01:01.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:01.303+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.304713+0000) 2020-02-05T22:01:01.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:01.303+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.304735+0000) 2020-02-05T22:01:01.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:01.304+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.305314+0000) 2020-02-05T22:01:01.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:01.304+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.305443+0000) 2020-02-05T22:01:01.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:01.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.305877+0000) 2020-02-05T22:01:01.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:01.305+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.306561+0000) 2020-02-05T22:01:01.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:01.308+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.309608+0000) 2020-02-05T22:01:01.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:01.308+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:01.309919+0000) 2020-02-05T22:01:01.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:01.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:02.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:02.284+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:02.285526+0000) 2020-02-05T22:01:02.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:02.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:02.683790+0000) 2020-02-05T22:01:05.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:05.894+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:05.895932+0000) 2020-02-05T22:01:05.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:05.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:05.901759+0000) 2020-02-05T22:01:06.319 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:06.316+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.317975+0000) 2020-02-05T22:01:06.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:06.318+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.320167+0000) 2020-02-05T22:01:06.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:06.319+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.320432+0000) 2020-02-05T22:01:06.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:06.319+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.321000+0000) 2020-02-05T22:01:06.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:06.320+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.321415+0000) 2020-02-05T22:01:06.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:06.320+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.321815+0000) 2020-02-05T22:01:06.324 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:06.320+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.321940+0000) 2020-02-05T22:01:06.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:06.321+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.322702+0000) 2020-02-05T22:01:06.325 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:06.322+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.323403+0000) 2020-02-05T22:01:06.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:06.322+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.324244+0000) 2020-02-05T22:01:06.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:06.323+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.325180+0000) 2020-02-05T22:01:06.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:06.324+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.325392+0000) 2020-02-05T22:01:06.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:06.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.330237+0000) 2020-02-05T22:01:06.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:06.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:06.330545+0000) 2020-02-05T22:01:06.333 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:06.330+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:07.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:07.041+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:07.042495+0000) 2020-02-05T22:01:07.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:07.041+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:07.042673+0000) 2020-02-05T22:01:07.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:07.048+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:07.049558+0000) 2020-02-05T22:01:07.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:07.048+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:07.049848+0000) 2020-02-05T22:01:08.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:08.840+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:08.841104+0000) 2020-02-05T22:01:09.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:09.432+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:09.432666+0000) 2020-02-05T22:01:10.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:10.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:10.039633+0000) 2020-02-05T22:01:10.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:10.274+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:10.275025+0000) 2020-02-05T22:01:10.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:10.291+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:10.291892+0000) 2020-02-05T22:01:10.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:10.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:10.292726+0000) 2020-02-05T22:01:11.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:11.336+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.337904+0000) 2020-02-05T22:01:11.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:11.337+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.339244+0000) 2020-02-05T22:01:11.341 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:11.338+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.339828+0000) 2020-02-05T22:01:11.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:11.338+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.339800+0000) 2020-02-05T22:01:11.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:11.338+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.340377+0000) 2020-02-05T22:01:11.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:11.339+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.340440+0000) 2020-02-05T22:01:11.342 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:11.339+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.340802+0000) 2020-02-05T22:01:11.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:11.340+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.341313+0000) 2020-02-05T22:01:11.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:11.339+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.341012+0000) 2020-02-05T22:01:11.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:11.340+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.341839+0000) 2020-02-05T22:01:11.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:11.340+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.341802+0000) 2020-02-05T22:01:11.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:11.341+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.342576+0000) 2020-02-05T22:01:11.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:11.341+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.342841+0000) 2020-02-05T22:01:11.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:11.344+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:11.345691+0000) 2020-02-05T22:01:11.351 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:11.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:12.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:12.285+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:12.285842+0000) 2020-02-05T22:01:12.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:12.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:12.684025+0000) 2020-02-05T22:01:15.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:15.895+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:15.896164+0000) 2020-02-05T22:01:15.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:15.901+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:15.901995+0000) 2020-02-05T22:01:16.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:16.354+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.355995+0000) 2020-02-05T22:01:16.359 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:16.355+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.357134+0000) 2020-02-05T22:01:16.359 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:16.357+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.358167+0000) 2020-02-05T22:01:16.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:16.357+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.358480+0000) 2020-02-05T22:01:16.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:16.357+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.358876+0000) 2020-02-05T22:01:16.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:16.358+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.359355+0000) 2020-02-05T22:01:16.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:16.358+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.359516+0000) 2020-02-05T22:01:16.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:16.358+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.359966+0000) 2020-02-05T22:01:16.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:16.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.361101+0000) 2020-02-05T22:01:16.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:16.360+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.361560+0000) 2020-02-05T22:01:16.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:16.359+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.360856+0000) 2020-02-05T22:01:16.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:16.360+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.361195+0000) 2020-02-05T22:01:16.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:16.361+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.362420+0000) 2020-02-05T22:01:16.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:16.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:16.366761+0000) 2020-02-05T22:01:16.368 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:16.366+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:17.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:17.042+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:17.042717+0000) 2020-02-05T22:01:17.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:17.042+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:17.042872+0000) 2020-02-05T22:01:17.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:17.049+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:17.049773+0000) 2020-02-05T22:01:17.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:17.049+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:17.050066+0000) 2020-02-05T22:01:18.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:18.849+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:18.849566+0000) 2020-02-05T22:01:19.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:19.432+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:19.432904+0000) 2020-02-05T22:01:20.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:20.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:20.039865+0000) 2020-02-05T22:01:20.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:20.274+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:20.275274+0000) 2020-02-05T22:01:20.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:20.291+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:20.292165+0000) 2020-02-05T22:01:20.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:20.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:20.292959+0000) 2020-02-05T22:01:21.375 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:21.372+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.373697+0000) 2020-02-05T22:01:21.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:21.373+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.374391+0000) 2020-02-05T22:01:21.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:21.373+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.374825+0000) 2020-02-05T22:01:21.376 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:21.373+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.374965+0000) 2020-02-05T22:01:21.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:21.374+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.375635+0000) 2020-02-05T22:01:21.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:21.374+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.375917+0000) 2020-02-05T22:01:21.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:21.375+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.376254+0000) 2020-02-05T22:01:21.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:21.375+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.377152+0000) 2020-02-05T22:01:21.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:21.376+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.377900+0000) 2020-02-05T22:01:21.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:21.379+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.380237+0000) 2020-02-05T22:01:21.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:21.378+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.379946+0000) 2020-02-05T22:01:21.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:21.378+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.379849+0000) 2020-02-05T22:01:21.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:21.383+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.385001+0000) 2020-02-05T22:01:21.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:21.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:21.385344+0000) 2020-02-05T22:01:21.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:21.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 102966 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:22.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:22.285+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:22.286065+0000) 2020-02-05T22:01:22.367 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:01:22.373 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:01:22.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:22.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:22.684255+0000) 2020-02-05T22:01:25.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:25.896+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:25.896390+0000) 2020-02-05T22:01:25.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:25.901+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:25.902257+0000) 2020-02-05T22:01:26.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:26.390+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.391610+0000) 2020-02-05T22:01:26.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:26.391+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.392658+0000) 2020-02-05T22:01:26.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:26.392+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.393306+0000) 2020-02-05T22:01:26.394 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:26.392+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.393369+0000) 2020-02-05T22:01:26.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:26.392+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.394002+0000) 2020-02-05T22:01:26.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:26.393+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.394017+0000) 2020-02-05T22:01:26.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:26.393+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.394465+0000) 2020-02-05T22:01:26.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:26.394+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.395682+0000) 2020-02-05T22:01:26.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:26.394+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.395360+0000) 2020-02-05T22:01:26.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:26.395+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.395918+0000) 2020-02-05T22:01:26.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:26.394+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.395212+0000) 2020-02-05T22:01:26.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:26.394+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.394933+0000) 2020-02-05T22:01:26.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:26.396+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.396772+0000) 2020-02-05T22:01:26.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:26.399+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:26.400433+0000) 2020-02-05T22:01:26.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:26.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103005 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:27.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:27.042+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:27.042958+0000) 2020-02-05T22:01:27.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:27.042+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:27.043150+0000) 2020-02-05T22:01:27.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:27.049+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:27.049993+0000) 2020-02-05T22:01:27.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:27.049+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:27.050302+0000) 2020-02-05T22:01:28.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:28.857+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:28.857668+0000) 2020-02-05T22:01:29.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:29.432+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:29.433131+0000) 2020-02-05T22:01:30.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:30.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:30.040124+0000) 2020-02-05T22:01:30.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:30.275+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:30.275528+0000) 2020-02-05T22:01:30.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:30.292+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:30.292397+0000) 2020-02-05T22:01:30.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:30.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:30.293209+0000) 2020-02-05T22:01:31.410 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:31.408+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.409269+0000) 2020-02-05T22:01:31.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:31.409+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.410074+0000) 2020-02-05T22:01:31.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:31.410+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.411281+0000) 2020-02-05T22:01:31.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:31.410+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.411349+0000) 2020-02-05T22:01:31.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:31.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.411745+0000) 2020-02-05T22:01:31.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:31.412+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.412532+0000) 2020-02-05T22:01:31.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:31.411+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.412885+0000) 2020-02-05T22:01:31.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:31.412+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.412969+0000) 2020-02-05T22:01:31.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:31.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.413394+0000) 2020-02-05T22:01:31.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:31.414+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.414628+0000) 2020-02-05T22:01:31.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:31.414+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.415159+0000) 2020-02-05T22:01:31.416 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:31.414+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.415371+0000) 2020-02-05T22:01:31.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:31.414+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.415194+0000) 2020-02-05T22:01:31.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:31.417+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:31.418970+0000) 2020-02-05T22:01:31.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:31.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103050 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:32.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:32.285+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:32.286275+0000) 2020-02-05T22:01:32.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:32.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:32.684496+0000) 2020-02-05T22:01:35.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:35.896+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:35.896608+0000) 2020-02-05T22:01:35.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:35.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:35.902508+0000) 2020-02-05T22:01:36.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:36.426+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.427213+0000) 2020-02-05T22:01:36.429 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:36.427+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.428095+0000) 2020-02-05T22:01:36.430 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:36.427+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.428589+0000) 2020-02-05T22:01:36.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:36.428+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.429127+0000) 2020-02-05T22:01:36.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:36.429+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.430554+0000) 2020-02-05T22:01:36.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:36.431+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.431838+0000) 2020-02-05T22:01:36.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:36.431+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.432210+0000) 2020-02-05T22:01:36.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:36.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.432482+0000) 2020-02-05T22:01:36.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:36.431+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.432122+0000) 2020-02-05T22:01:36.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:36.433+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.433520+0000) 2020-02-05T22:01:36.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:36.433+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.433841+0000) 2020-02-05T22:01:36.436 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:36.433+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.434342+0000) 2020-02-05T22:01:36.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:36.433+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.434387+0000) 2020-02-05T22:01:36.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:36.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:36.439138+0000) 2020-02-05T22:01:36.441 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:36.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:37.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:37.042+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:37.043231+0000) 2020-02-05T22:01:37.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:37.042+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:37.043378+0000) 2020-02-05T22:01:37.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:37.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:37.050208+0000) 2020-02-05T22:01:37.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:37.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:37.050479+0000) 2020-02-05T22:01:38.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:38.865+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:38.865741+0000) 2020-02-05T22:01:39.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:39.433+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:39.433362+0000) 2020-02-05T22:01:40.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:40.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:40.040315+0000) 2020-02-05T22:01:40.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:40.275+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:40.275762+0000) 2020-02-05T22:01:40.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:40.292+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:40.292647+0000) 2020-02-05T22:01:40.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:40.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:40.293418+0000) 2020-02-05T22:01:41.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:41.444+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.446057+0000) 2020-02-05T22:01:41.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:41.445+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.446662+0000) 2020-02-05T22:01:41.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:41.447+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.448824+0000) 2020-02-05T22:01:41.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:41.449+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.449616+0000) 2020-02-05T22:01:41.451 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:41.449+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.450415+0000) 2020-02-05T22:01:41.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:41.450+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.451004+0000) 2020-02-05T22:01:41.453 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:41.450+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.451272+0000) 2020-02-05T22:01:41.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:41.450+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.451630+0000) 2020-02-05T22:01:41.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:41.452+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.452673+0000) 2020-02-05T22:01:41.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:41.452+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.452959+0000) 2020-02-05T22:01:41.454 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:41.452+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.453212+0000) 2020-02-05T22:01:41.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:41.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.453378+0000) 2020-02-05T22:01:41.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:41.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.453892+0000) 2020-02-05T22:01:41.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:41.453+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:41.454181+0000) 2020-02-05T22:01:41.460 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:41.459+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:42.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:42.286+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:42.286506+0000) 2020-02-05T22:01:42.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:42.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:42.684735+0000) 2020-02-05T22:01:45.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:45.895+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:45.896836+0000) 2020-02-05T22:01:45.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:45.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:45.902733+0000) 2020-02-05T22:01:46.465 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:46.462+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.463726+0000) 2020-02-05T22:01:46.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:46.464+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.465782+0000) 2020-02-05T22:01:46.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:46.465+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.466291+0000) 2020-02-05T22:01:46.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:46.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.466688+0000) 2020-02-05T22:01:46.468 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:46.465+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.466885+0000) 2020-02-05T22:01:46.470 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:46.468+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.468540+0000) 2020-02-05T22:01:46.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:46.468+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.468970+0000) 2020-02-05T22:01:46.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:46.468+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.469063+0000) 2020-02-05T22:01:46.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:46.469+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.469564+0000) 2020-02-05T22:01:46.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:46.470+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.470582+0000) 2020-02-05T22:01:46.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:46.470+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.470955+0000) 2020-02-05T22:01:46.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:46.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.471129+0000) 2020-02-05T22:01:46.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:46.471+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.471773+0000) 2020-02-05T22:01:46.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:46.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:46.472753+0000) 2020-02-05T22:01:46.479 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:46.477+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103175 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:47.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:47.043+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:47.043496+0000) 2020-02-05T22:01:47.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:47.043+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:47.043820+0000) 2020-02-05T22:01:47.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:47.049+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:47.050435+0000) 2020-02-05T22:01:47.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:47.049+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:47.050679+0000) 2020-02-05T22:01:48.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:48.872+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:48.873723+0000) 2020-02-05T22:01:49.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:49.432+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:49.433588+0000) 2020-02-05T22:01:50.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:50.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:50.040551+0000) 2020-02-05T22:01:50.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:50.275+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:50.275996+0000) 2020-02-05T22:01:50.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:50.292+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:50.292950+0000) 2020-02-05T22:01:50.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:50.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:50.293714+0000) 2020-02-05T22:01:51.485 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:51.482+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.483864+0000) 2020-02-05T22:01:51.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:51.483+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.484384+0000) 2020-02-05T22:01:51.487 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:51.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.485868+0000) 2020-02-05T22:01:51.488 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:51.486+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.486946+0000) 2020-02-05T22:01:51.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:51.485+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.486909+0000) 2020-02-05T22:01:51.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:51.486+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.487448+0000) 2020-02-05T22:01:51.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:51.486+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.487809+0000) 2020-02-05T22:01:51.498 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:51.487+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.488962+0000) 2020-02-05T22:01:51.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:51.488+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.489980+0000) 2020-02-05T22:01:51.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:51.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.493891+0000) 2020-02-05T22:01:51.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:51.488+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.488759+0000) 2020-02-05T22:01:51.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:51.488+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.489173+0000) 2020-02-05T22:01:51.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:51.489+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.489935+0000) 2020-02-05T22:01:51.500 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:51.489+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:51.490197+0000) 2020-02-05T22:01:51.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:51.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:52.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:52.286+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:52.286788+0000) 2020-02-05T22:01:52.529 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:01:52.535 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:01:52.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:52.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:52.684978+0000) 2020-02-05T22:01:55.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:55.896+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:55.897133+0000) 2020-02-05T22:01:55.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:55.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:55.902994+0000) 2020-02-05T22:01:56.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:01:56.500+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.501510+0000) 2020-02-05T22:01:56.503 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:56.501+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.502187+0000) 2020-02-05T22:01:56.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:56.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.502711+0000) 2020-02-05T22:01:56.504 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:01:56.501+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.503012+0000) 2020-02-05T22:01:56.505 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:56.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.503643+0000) 2020-02-05T22:01:56.505 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:01:56.503+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.504146+0000) 2020-02-05T22:01:56.506 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:01:56.502+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.504011+0000) 2020-02-05T22:01:56.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:01:56.504+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.504816+0000) 2020-02-05T22:01:56.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:01:56.504+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.505059+0000) 2020-02-05T22:01:56.507 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:56.505+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.505858+0000) 2020-02-05T22:01:56.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:01:56.505+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.505869+0000) 2020-02-05T22:01:56.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:56.505+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.506300+0000) 2020-02-05T22:01:56.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:56.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.511121+0000) 2020-02-05T22:01:56.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:56.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:56.511376+0000) 2020-02-05T22:01:56.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:01:56.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:01:57.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:01:57.043+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:57.043707+0000) 2020-02-05T22:01:57.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:01:57.043+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:57.044032+0000) 2020-02-05T22:01:57.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:01:57.049+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:57.050651+0000) 2020-02-05T22:01:57.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:01:57.049+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:57.050847+0000) 2020-02-05T22:01:58.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:58.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:58.883567+0000) 2020-02-05T22:01:59.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:01:59.432+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:01:59.433806+0000) 2020-02-05T22:02:00.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:00.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:00.040775+0000) 2020-02-05T22:02:00.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:00.276+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:00.276244+0000) 2020-02-05T22:02:00.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:00.292+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:00.293189+0000) 2020-02-05T22:02:00.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:00.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:00.293954+0000) 2020-02-05T22:02:01.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:01.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.518934+0000) 2020-02-05T22:02:01.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:01.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.520340+0000) 2020-02-05T22:02:01.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:01.519+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.520749+0000) 2020-02-05T22:02:01.522 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:01.520+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.521211+0000) 2020-02-05T22:02:01.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:01.520+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.521524+0000) 2020-02-05T22:02:01.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:01.521+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.522213+0000) 2020-02-05T22:02:01.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:01.522+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.523301+0000) 2020-02-05T22:02:01.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:01.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.523668+0000) 2020-02-05T22:02:01.525 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:01.524+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.524622+0000) 2020-02-05T22:02:01.526 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:01.523+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.524462+0000) 2020-02-05T22:02:01.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:01.524+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.525071+0000) 2020-02-05T22:02:01.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:01.525+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.525449+0000) 2020-02-05T22:02:01.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:01.525+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.525661+0000) 2020-02-05T22:02:01.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:01.528+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:01.529121+0000) 2020-02-05T22:02:01.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:01.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:02.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:02.285+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:02.287003+0000) 2020-02-05T22:02:02.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:02.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:02.685229+0000) 2020-02-05T22:02:05.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:05.896+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:05.897364+0000) 2020-02-05T22:02:05.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:05.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:05.903260+0000) 2020-02-05T22:02:06.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:06.536+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.537727+0000) 2020-02-05T22:02:06.540 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:06.537+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.538601+0000) 2020-02-05T22:02:06.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:06.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.540330+0000) 2020-02-05T22:02:06.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:06.539+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.540605+0000) 2020-02-05T22:02:06.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:06.539+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.540880+0000) 2020-02-05T22:02:06.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:06.540+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.542013+0000) 2020-02-05T22:02:06.543 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:06.539+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.541401+0000) 2020-02-05T22:02:06.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:06.541+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.542384+0000) 2020-02-05T22:02:06.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:06.541+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.542923+0000) 2020-02-05T22:02:06.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:06.542+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.543866+0000) 2020-02-05T22:02:06.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:06.542+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.544045+0000) 2020-02-05T22:02:06.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:06.543+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.544938+0000) 2020-02-05T22:02:06.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:06.542+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.544169+0000) 2020-02-05T22:02:06.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:06.549+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:06.550598+0000) 2020-02-05T22:02:06.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:06.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:07.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:07.042+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:07.043976+0000) 2020-02-05T22:02:07.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:07.042+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:07.044231+0000) 2020-02-05T22:02:07.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:07.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:07.050971+0000) 2020-02-05T22:02:07.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:07.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:07.051124+0000) 2020-02-05T22:02:08.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:08.891+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:08.891584+0000) 2020-02-05T22:02:09.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:09.433+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:09.434057+0000) 2020-02-05T22:02:10.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:10.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:10.041022+0000) 2020-02-05T22:02:10.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:10.275+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:10.276502+0000) 2020-02-05T22:02:10.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:10.292+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:10.293438+0000) 2020-02-05T22:02:10.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:10.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:10.294160+0000) 2020-02-05T22:02:11.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:11.555+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.556783+0000) 2020-02-05T22:02:11.559 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:11.556+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.557505+0000) 2020-02-05T22:02:11.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:11.557+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.558282+0000) 2020-02-05T22:02:11.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:11.557+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.558773+0000) 2020-02-05T22:02:11.561 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:11.558+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.559784+0000) 2020-02-05T22:02:11.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:11.559+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.561048+0000) 2020-02-05T22:02:11.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:11.559+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.561402+0000) 2020-02-05T22:02:11.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:11.560+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.561637+0000) 2020-02-05T22:02:11.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:11.560+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.561829+0000) 2020-02-05T22:02:11.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:11.561+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.562616+0000) 2020-02-05T22:02:11.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:11.561+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.563021+0000) 2020-02-05T22:02:11.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:11.561+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.562493+0000) 2020-02-05T22:02:11.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:11.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.563157+0000) 2020-02-05T22:02:11.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:11.564+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:11.565558+0000) 2020-02-05T22:02:11.571 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:11.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:12.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:12.286+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:12.287244+0000) 2020-02-05T22:02:12.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:12.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:12.685454+0000) 2020-02-05T22:02:15.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:15.897+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:15.897588+0000) 2020-02-05T22:02:15.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:15.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:15.903512+0000) 2020-02-05T22:02:16.577 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:16.574+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.575818+0000) 2020-02-05T22:02:16.578 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:16.575+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.576869+0000) 2020-02-05T22:02:16.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:16.575+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.577385+0000) 2020-02-05T22:02:16.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:16.576+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.577942+0000) 2020-02-05T22:02:16.579 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:16.576+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.577962+0000) 2020-02-05T22:02:16.580 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:16.577+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.578817+0000) 2020-02-05T22:02:16.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:16.577+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.578891+0000) 2020-02-05T22:02:16.580 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:16.577+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.579122+0000) 2020-02-05T22:02:16.581 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:16.578+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.579164+0000) 2020-02-05T22:02:16.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:16.578+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.579517+0000) 2020-02-05T22:02:16.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:16.578+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.580240+0000) 2020-02-05T22:02:16.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:16.579+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.580414+0000) 2020-02-05T22:02:16.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:16.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.584703+0000) 2020-02-05T22:02:16.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:16.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:16.584971+0000) 2020-02-05T22:02:16.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:16.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:17.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:17.042+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:17.044221+0000) 2020-02-05T22:02:17.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:17.043+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:17.044483+0000) 2020-02-05T22:02:17.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:17.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:17.051185+0000) 2020-02-05T22:02:17.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:17.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:17.051329+0000) 2020-02-05T22:02:18.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:18.899+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:18.900180+0000) 2020-02-05T22:02:19.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:19.433+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:19.434324+0000) 2020-02-05T22:02:20.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:20.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:20.041285+0000) 2020-02-05T22:02:20.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:20.275+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:20.276737+0000) 2020-02-05T22:02:20.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:20.293+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:20.293657+0000) 2020-02-05T22:02:20.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:20.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:20.294373+0000) 2020-02-05T22:02:21.596 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:21.593+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.594871+0000) 2020-02-05T22:02:21.597 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:21.593+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.595254+0000) 2020-02-05T22:02:21.597 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:21.594+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.595778+0000) 2020-02-05T22:02:21.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:21.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.596430+0000) 2020-02-05T22:02:21.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:21.595+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.596688+0000) 2020-02-05T22:02:21.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:21.596+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.597236+0000) 2020-02-05T22:02:21.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:21.595+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.597406+0000) 2020-02-05T22:02:21.599 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:21.596+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.597806+0000) 2020-02-05T22:02:21.599 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:21.596+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.598384+0000) 2020-02-05T22:02:21.600 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:21.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.598752+0000) 2020-02-05T22:02:21.600 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:21.598+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.599614+0000) 2020-02-05T22:02:21.601 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:21.598+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.599409+0000) 2020-02-05T22:02:21.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:21.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.599577+0000) 2020-02-05T22:02:21.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:21.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:21.603368+0000) 2020-02-05T22:02:21.608 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:21.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103463 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:22.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:22.286+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:22.287532+0000) 2020-02-05T22:02:22.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:22.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:22.685684+0000) 2020-02-05T22:02:22.718 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:02:22.724 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:02:25.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:25.897+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:25.897806+0000) 2020-02-05T22:02:25.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:25.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:25.903738+0000) 2020-02-05T22:02:26.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:26.612+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.613251+0000) 2020-02-05T22:02:26.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:26.612+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.614025+0000) 2020-02-05T22:02:26.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:26.613+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.614885+0000) 2020-02-05T22:02:26.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:26.613+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.615019+0000) 2020-02-05T22:02:26.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:26.614+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.615485+0000) 2020-02-05T22:02:26.617 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:26.614+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.616373+0000) 2020-02-05T22:02:26.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:26.615+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.616694+0000) 2020-02-05T22:02:26.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:26.615+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.617202+0000) 2020-02-05T22:02:26.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:26.616+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.617652+0000) 2020-02-05T22:02:26.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:26.616+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.617772+0000) 2020-02-05T22:02:26.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:26.617+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.618862+0000) 2020-02-05T22:02:26.620 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:26.617+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.619251+0000) 2020-02-05T22:02:26.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:26.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.623107+0000) 2020-02-05T22:02:26.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:26.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:26.623355+0000) 2020-02-05T22:02:26.627 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:26.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:27.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:27.043+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:27.044458+0000) 2020-02-05T22:02:27.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:27.043+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:27.044672+0000) 2020-02-05T22:02:27.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:27.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:27.051343+0000) 2020-02-05T22:02:27.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:27.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:27.051573+0000) 2020-02-05T22:02:28.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:28.908+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:28.908517+0000) 2020-02-05T22:02:29.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:29.434+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:29.434547+0000) 2020-02-05T22:02:30.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:30.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:30.041566+0000) 2020-02-05T22:02:30.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:30.276+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:30.276993+0000) 2020-02-05T22:02:30.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:30.293+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:30.293881+0000) 2020-02-05T22:02:30.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:30.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:30.294653+0000) 2020-02-05T22:02:31.634 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:31.631+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.632496+0000) 2020-02-05T22:02:31.634 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:31.633+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.633428+0000) 2020-02-05T22:02:31.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:31.634+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.634451+0000) 2020-02-05T22:02:31.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:31.633+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.635147+0000) 2020-02-05T22:02:31.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:31.635+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.635541+0000) 2020-02-05T22:02:31.637 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:31.636+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.636399+0000) 2020-02-05T22:02:31.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:31.634+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.636332+0000) 2020-02-05T22:02:31.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:31.635+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.636549+0000) 2020-02-05T22:02:31.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:31.636+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.636989+0000) 2020-02-05T22:02:31.639 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:31.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.637460+0000) 2020-02-05T22:02:31.639 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:31.636+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.638059+0000) 2020-02-05T22:02:31.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:31.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.640165+0000) 2020-02-05T22:02:31.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:31.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.640414+0000) 2020-02-05T22:02:31.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:31.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:31.641523+0000) 2020-02-05T22:02:31.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:31.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:32.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:32.287+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:32.287774+0000) 2020-02-05T22:02:32.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:32.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:32.685925+0000) 2020-02-05T22:02:35.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:35.896+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:35.898041+0000) 2020-02-05T22:02:35.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:35.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:35.903972+0000) 2020-02-05T22:02:36.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:36.648+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.649793+0000) 2020-02-05T22:02:36.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:36.649+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.650641+0000) 2020-02-05T22:02:36.652 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:36.649+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.650875+0000) 2020-02-05T22:02:36.652 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:36.650+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.651324+0000) 2020-02-05T22:02:36.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:36.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.651929+0000) 2020-02-05T22:02:36.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:36.651+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.652735+0000) 2020-02-05T22:02:36.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:36.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.652902+0000) 2020-02-05T22:02:36.655 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:36.651+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.653263+0000) 2020-02-05T22:02:36.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:36.652+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.653705+0000) 2020-02-05T22:02:36.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:36.652+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.653499+0000) 2020-02-05T22:02:36.656 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:36.652+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.654127+0000) 2020-02-05T22:02:36.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:36.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.654141+0000) 2020-02-05T22:02:36.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:36.653+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.654829+0000) 2020-02-05T22:02:36.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:36.659+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:36.660175+0000) 2020-02-05T22:02:36.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:36.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:37.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:37.044+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:37.044708+0000) 2020-02-05T22:02:37.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:37.044+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:37.044873+0000) 2020-02-05T22:02:37.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:37.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:37.051549+0000) 2020-02-05T22:02:37.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:37.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:37.051741+0000) 2020-02-05T22:02:38.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:38.915+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:38.916674+0000) 2020-02-05T22:02:39.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:39.433+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:39.434770+0000) 2020-02-05T22:02:40.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:40.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:40.041780+0000) 2020-02-05T22:02:40.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:40.276+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:40.277277+0000) 2020-02-05T22:02:40.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:40.293+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:40.294140+0000) 2020-02-05T22:02:40.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:40.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:40.294881+0000) 2020-02-05T22:02:41.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:41.666+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.667984+0000) 2020-02-05T22:02:41.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:41.667+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.668772+0000) 2020-02-05T22:02:41.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:41.668+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.669337+0000) 2020-02-05T22:02:41.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:41.669+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.670545+0000) 2020-02-05T22:02:41.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:41.668+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.670247+0000) 2020-02-05T22:02:41.672 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:41.669+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.670896+0000) 2020-02-05T22:02:41.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:41.670+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.671548+0000) 2020-02-05T22:02:41.673 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:41.670+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.671767+0000) 2020-02-05T22:02:41.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:41.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.672549+0000) 2020-02-05T22:02:41.674 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:41.671+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.673096+0000) 2020-02-05T22:02:41.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:41.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.673212+0000) 2020-02-05T22:02:41.675 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:41.672+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.673782+0000) 2020-02-05T22:02:41.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:41.674+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.675467+0000) 2020-02-05T22:02:41.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:41.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:41.680291+0000) 2020-02-05T22:02:41.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:41.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:42.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:42.287+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:42.288018+0000) 2020-02-05T22:02:42.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:42.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:42.686166+0000) 2020-02-05T22:02:45.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:45.897+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:45.898272+0000) 2020-02-05T22:02:45.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:45.903+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:45.904202+0000) 2020-02-05T22:02:46.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:46.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.687778+0000) 2020-02-05T22:02:46.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:46.687+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.688198+0000) 2020-02-05T22:02:46.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:46.687+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.688727+0000) 2020-02-05T22:02:46.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:46.688+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.689169+0000) 2020-02-05T22:02:46.691 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:46.688+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.689860+0000) 2020-02-05T22:02:46.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:46.689+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.690269+0000) 2020-02-05T22:02:46.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:46.689+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.690693+0000) 2020-02-05T22:02:46.692 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:46.689+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.691182+0000) 2020-02-05T22:02:46.693 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:46.690+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.692111+0000) 2020-02-05T22:02:46.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:46.691+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.692603+0000) 2020-02-05T22:02:46.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:46.691+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.693163+0000) 2020-02-05T22:02:46.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:46.692+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.693671+0000) 2020-02-05T22:02:46.695 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:46.692+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.693389+0000) 2020-02-05T22:02:46.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:46.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:46.699085+0000) 2020-02-05T22:02:46.701 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:46.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:47.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:47.044+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:47.044941+0000) 2020-02-05T22:02:47.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:47.044+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:47.045070+0000) 2020-02-05T22:02:47.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:47.050+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:47.051773+0000) 2020-02-05T22:02:47.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:47.050+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:47.051993+0000) 2020-02-05T22:02:48.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:48.924+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:48.924926+0000) 2020-02-05T22:02:49.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:49.434+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:49.435025+0000) 2020-02-05T22:02:50.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:50.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:50.042012+0000) 2020-02-05T22:02:50.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:50.276+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:50.277514+0000) 2020-02-05T22:02:50.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:50.293+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:50.294450+0000) 2020-02-05T22:02:50.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:50.294+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:50.295141+0000) 2020-02-05T22:02:51.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:51.704+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.705067+0000) 2020-02-05T22:02:51.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:51.705+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.706926+0000) 2020-02-05T22:02:51.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:51.706+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.707903+0000) 2020-02-05T22:02:51.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:51.707+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.708224+0000) 2020-02-05T22:02:51.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:51.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.708802+0000) 2020-02-05T22:02:51.710 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:51.708+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.709211+0000) 2020-02-05T22:02:51.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:51.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.709747+0000) 2020-02-05T22:02:51.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:51.708+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.709788+0000) 2020-02-05T22:02:51.711 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:51.708+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.710171+0000) 2020-02-05T22:02:51.712 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:51.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.710917+0000) 2020-02-05T22:02:51.712 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:51.710+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.711415+0000) 2020-02-05T22:02:51.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:51.710+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.712357+0000) 2020-02-05T22:02:51.713 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:51.711+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.712767+0000) 2020-02-05T22:02:51.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:51.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:51.714535+0000) 2020-02-05T22:02:51.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:51.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:52.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:52.287+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:52.288252+0000) 2020-02-05T22:02:52.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:52.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:52.686389+0000) 2020-02-05T22:02:52.907 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:02:52.913 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:02:55.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:55.897+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:55.898500+0000) 2020-02-05T22:02:55.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:55.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:55.904469+0000) 2020-02-05T22:02:56.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:56.725+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.726470+0000) 2020-02-05T22:02:56.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:56.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.726900+0000) 2020-02-05T22:02:56.728 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:02:56.726+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.727067+0000) 2020-02-05T22:02:56.729 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:56.727+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.727449+0000) 2020-02-05T22:02:56.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:02:56.727+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.728324+0000) 2020-02-05T22:02:56.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:02:56.728+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.728551+0000) 2020-02-05T22:02:56.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:02:56.727+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.728715+0000) 2020-02-05T22:02:56.730 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:56.728+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.729170+0000) 2020-02-05T22:02:56.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:56.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.729804+0000) 2020-02-05T22:02:56.731 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:56.728+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.729921+0000) 2020-02-05T22:02:56.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:02:56.730+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.730782+0000) 2020-02-05T22:02:56.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:02:56.729+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.730652+0000) 2020-02-05T22:02:56.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:02:56.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.731151+0000) 2020-02-05T22:02:56.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:56.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:56.735101+0000) 2020-02-05T22:02:56.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:02:56.736+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:02:57.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:02:57.044+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:57.045159+0000) 2020-02-05T22:02:57.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:02:57.044+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:57.045288+0000) 2020-02-05T22:02:57.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:02:57.051+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:57.051988+0000) 2020-02-05T22:02:57.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:02:57.051+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:57.052265+0000) 2020-02-05T22:02:58.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:58.932+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:58.932991+0000) 2020-02-05T22:02:59.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:02:59.434+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:02:59.435322+0000) 2020-02-05T22:03:00.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:00.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:00.042306+0000) 2020-02-05T22:03:00.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:00.277+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:00.277768+0000) 2020-02-05T22:03:00.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:00.294+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:00.294685+0000) 2020-02-05T22:03:00.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:00.294+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:00.295383+0000) 2020-02-05T22:03:01.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:01.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.743303+0000) 2020-02-05T22:03:01.745 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:01.743+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.744200+0000) 2020-02-05T22:03:01.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:01.743+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.744535+0000) 2020-02-05T22:03:01.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:01.743+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.744951+0000) 2020-02-05T22:03:01.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:01.745+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.745500+0000) 2020-02-05T22:03:01.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:01.745+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.746396+0000) 2020-02-05T22:03:01.748 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:01.745+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.746510+0000) 2020-02-05T22:03:01.749 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:01.746+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.747639+0000) 2020-02-05T22:03:01.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:01.748+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.748536+0000) 2020-02-05T22:03:01.750 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:01.748+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.748794+0000) 2020-02-05T22:03:01.751 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:01.748+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.749272+0000) 2020-02-05T22:03:01.751 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:01.749+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.749450+0000) 2020-02-05T22:03:01.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:01.750+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.751237+0000) 2020-02-05T22:03:01.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:01.750+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:01.751466+0000) 2020-02-05T22:03:01.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:01.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:02.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:02.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:02.288492+0000) 2020-02-05T22:03:02.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:02.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:02.686610+0000) 2020-02-05T22:03:05.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:05.898+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:05.898767+0000) 2020-02-05T22:03:05.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:05.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:05.904715+0000) 2020-02-05T22:03:06.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:06.760+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.761169+0000) 2020-02-05T22:03:06.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:06.760+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.761815+0000) 2020-02-05T22:03:06.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:06.762+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.762480+0000) 2020-02-05T22:03:06.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:06.761+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.762580+0000) 2020-02-05T22:03:06.764 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:06.762+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.763189+0000) 2020-02-05T22:03:06.764 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:06.763+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.763545+0000) 2020-02-05T22:03:06.765 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:06.763+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.764229+0000) 2020-02-05T22:03:06.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:06.764+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.764550+0000) 2020-02-05T22:03:06.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:06.764+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.765089+0000) 2020-02-05T22:03:06.766 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:06.765+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.765664+0000) 2020-02-05T22:03:06.768 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:06.765+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.766365+0000) 2020-02-05T22:03:06.768 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:06.765+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.766820+0000) 2020-02-05T22:03:06.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:06.766+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.767284+0000) 2020-02-05T22:03:06.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:06.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:06.771713+0000) 2020-02-05T22:03:06.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:06.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:07.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:07.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:07.045448+0000) 2020-02-05T22:03:07.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:07.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:07.045535+0000) 2020-02-05T22:03:07.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:07.051+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:07.052200+0000) 2020-02-05T22:03:07.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:07.051+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:07.052452+0000) 2020-02-05T22:03:08.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:08.943+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:08.944418+0000) 2020-02-05T22:03:09.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:09.435+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:09.435568+0000) 2020-02-05T22:03:10.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:10.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:10.042593+0000) 2020-02-05T22:03:10.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:10.277+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:10.278021+0000) 2020-02-05T22:03:10.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:10.294+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:10.294908+0000) 2020-02-05T22:03:10.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:10.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:10.295638+0000) 2020-02-05T22:03:11.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:11.780+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.781523+0000) 2020-02-05T22:03:11.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:11.781+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.782382+0000) 2020-02-05T22:03:11.785 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:11.783+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.783610+0000) 2020-02-05T22:03:11.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:11.783+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.784887+0000) 2020-02-05T22:03:11.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:11.784+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.785449+0000) 2020-02-05T22:03:11.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:11.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.786020+0000) 2020-02-05T22:03:11.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:11.786+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.786838+0000) 2020-02-05T22:03:11.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:11.785+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.787051+0000) 2020-02-05T22:03:11.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:11.786+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.787407+0000) 2020-02-05T22:03:11.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:11.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.787843+0000) 2020-02-05T22:03:11.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:11.786+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.787919+0000) 2020-02-05T22:03:11.790 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:11.787+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.788345+0000) 2020-02-05T22:03:11.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:11.787+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.788772+0000) 2020-02-05T22:03:11.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:11.793+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:11.794533+0000) 2020-02-05T22:03:11.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:11.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:12.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:12.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:12.288710+0000) 2020-02-05T22:03:12.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:12.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:12.686841+0000) 2020-02-05T22:03:15.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:15.898+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:15.899000+0000) 2020-02-05T22:03:15.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:15.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:15.904951+0000) 2020-02-05T22:03:16.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:16.801+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.801885+0000) 2020-02-05T22:03:16.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:16.802+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.803560+0000) 2020-02-05T22:03:16.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:16.802+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.803872+0000) 2020-02-05T22:03:16.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:16.804+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.805354+0000) 2020-02-05T22:03:16.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:16.804+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.804997+0000) 2020-02-05T22:03:16.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:16.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.806183+0000) 2020-02-05T22:03:16.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:16.805+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.806196+0000) 2020-02-05T22:03:16.808 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:16.806+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.806625+0000) 2020-02-05T22:03:16.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:16.805+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.806916+0000) 2020-02-05T22:03:16.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:16.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.807516+0000) 2020-02-05T22:03:16.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:16.806+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.807733+0000) 2020-02-05T22:03:16.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:16.806+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.807386+0000) 2020-02-05T22:03:16.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:16.807+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.808126+0000) 2020-02-05T22:03:16.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:16.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:16.811344+0000) 2020-02-05T22:03:16.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:16.811+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:17.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:17.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:17.045662+0000) 2020-02-05T22:03:17.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:17.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:17.045750+0000) 2020-02-05T22:03:17.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:17.052+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:17.052407+0000) 2020-02-05T22:03:17.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:17.052+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:17.052638+0000) 2020-02-05T22:03:18.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:18.952+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:18.952706+0000) 2020-02-05T22:03:19.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:19.435+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:19.435805+0000) 2020-02-05T22:03:20.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:20.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:20.042838+0000) 2020-02-05T22:03:20.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:20.277+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:20.278314+0000) 2020-02-05T22:03:20.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:20.294+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:20.295136+0000) 2020-02-05T22:03:20.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:20.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:20.295890+0000) 2020-02-05T22:03:21.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:21.816+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.817466+0000) 2020-02-05T22:03:21.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:21.818+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.818514+0000) 2020-02-05T22:03:21.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:21.817+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.818970+0000) 2020-02-05T22:03:21.820 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:21.818+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.819374+0000) 2020-02-05T22:03:21.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:21.818+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.819954+0000) 2020-02-05T22:03:21.821 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:21.820+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.820595+0000) 2020-02-05T22:03:21.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:21.821+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.821637+0000) 2020-02-05T22:03:21.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:21.821+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.821800+0000) 2020-02-05T22:03:21.823 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:21.821+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.822190+0000) 2020-02-05T22:03:21.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:21.822+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.823655+0000) 2020-02-05T22:03:21.825 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:21.823+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.824266+0000) 2020-02-05T22:03:21.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:21.823+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.824361+0000) 2020-02-05T22:03:21.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:21.828+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.829158+0000) 2020-02-05T22:03:21.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:21.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:21.830432+0000) 2020-02-05T22:03:21.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:21.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103948 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:22.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:22.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:22.288940+0000) 2020-02-05T22:03:22.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:22.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:22.687048+0000) 2020-02-05T22:03:23.086 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:03:23.092 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:03:25.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:25.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:25.899340+0000) 2020-02-05T22:03:25.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:25.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:25.905190+0000) 2020-02-05T22:03:26.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:26.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.837812+0000) 2020-02-05T22:03:26.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:26.838+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.838200+0000) 2020-02-05T22:03:26.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:26.838+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.838823+0000) 2020-02-05T22:03:26.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:26.839+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.839252+0000) 2020-02-05T22:03:26.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:26.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.840583+0000) 2020-02-05T22:03:26.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:26.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.842262+0000) 2020-02-05T22:03:26.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:26.840+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.840818+0000) 2020-02-05T22:03:26.846 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:26.842+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.842486+0000) 2020-02-05T22:03:26.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:26.839+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.839952+0000) 2020-02-05T22:03:26.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:26.839+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.840269+0000) 2020-02-05T22:03:26.846 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:26.842+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.843032+0000) 2020-02-05T22:03:26.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:26.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.843214+0000) 2020-02-05T22:03:26.850 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:26.843+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.844297+0000) 2020-02-05T22:03:26.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:26.846+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:26.846949+0000) 2020-02-05T22:03:26.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:26.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 103986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:27.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:27.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:27.045882+0000) 2020-02-05T22:03:27.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:27.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:27.045964+0000) 2020-02-05T22:03:27.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:27.052+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:27.052677+0000) 2020-02-05T22:03:27.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:27.052+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:27.052810+0000) 2020-02-05T22:03:28.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:28.961+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:28.961072+0000) 2020-02-05T22:03:29.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:29.435+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:29.436032+0000) 2020-02-05T22:03:30.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:30.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:30.043113+0000) 2020-02-05T22:03:30.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:30.278+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:30.278547+0000) 2020-02-05T22:03:30.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:30.295+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:30.295367+0000) 2020-02-05T22:03:30.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:30.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:30.296132+0000) 2020-02-05T22:03:31.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:31.853+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.855343+0000) 2020-02-05T22:03:31.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:31.855+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.856900+0000) 2020-02-05T22:03:31.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:31.856+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.857295+0000) 2020-02-05T22:03:31.859 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:31.856+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.857773+0000) 2020-02-05T22:03:31.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:31.857+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.858388+0000) 2020-02-05T22:03:31.860 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:31.857+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.858780+0000) 2020-02-05T22:03:31.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:31.857+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.858836+0000) 2020-02-05T22:03:31.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:31.858+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.859417+0000) 2020-02-05T22:03:31.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:31.858+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.860149+0000) 2020-02-05T22:03:31.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:31.857+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.859040+0000) 2020-02-05T22:03:31.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:31.859+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.860409+0000) 2020-02-05T22:03:31.862 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:31.859+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.861120+0000) 2020-02-05T22:03:31.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:31.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.864426+0000) 2020-02-05T22:03:31.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:31.863+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:31.864639+0000) 2020-02-05T22:03:31.868 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:31.866+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:32.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:32.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:32.289167+0000) 2020-02-05T22:03:32.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:32.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:32.687289+0000) 2020-02-05T22:03:35.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:35.898+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:35.899566+0000) 2020-02-05T22:03:35.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:35.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:35.905487+0000) 2020-02-05T22:03:36.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:36.871+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.872589+0000) 2020-02-05T22:03:36.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:36.872+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.873443+0000) 2020-02-05T22:03:36.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:36.873+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.874796+0000) 2020-02-05T22:03:36.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:36.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.875215+0000) 2020-02-05T22:03:36.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:36.874+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.875658+0000) 2020-02-05T22:03:36.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:36.875+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.876214+0000) 2020-02-05T22:03:36.878 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:36.875+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.877071+0000) 2020-02-05T22:03:36.878 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:36.876+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.877266+0000) 2020-02-05T22:03:36.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:36.876+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.878278+0000) 2020-02-05T22:03:36.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:36.877+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.879151+0000) 2020-02-05T22:03:36.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:36.878+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.879598+0000) 2020-02-05T22:03:36.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:36.878+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.879781+0000) 2020-02-05T22:03:36.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:36.882+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.883130+0000) 2020-02-05T22:03:36.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:36.882+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:36.883377+0000) 2020-02-05T22:03:36.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:36.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:37.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:37.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:37.046118+0000) 2020-02-05T22:03:37.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:37.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:37.046219+0000) 2020-02-05T22:03:37.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:37.051+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:37.052884+0000) 2020-02-05T22:03:37.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:37.051+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:37.052996+0000) 2020-02-05T22:03:38.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:38.968+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:38.969540+0000) 2020-02-05T22:03:39.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:39.435+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:39.436344+0000) 2020-02-05T22:03:40.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:40.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:40.043373+0000) 2020-02-05T22:03:40.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:40.277+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:40.278786+0000) 2020-02-05T22:03:40.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:40.294+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:40.295649+0000) 2020-02-05T22:03:40.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:40.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:40.296348+0000) 2020-02-05T22:03:41.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:41.890+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.891894+0000) 2020-02-05T22:03:41.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:41.891+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.892606+0000) 2020-02-05T22:03:41.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:41.893+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.894383+0000) 2020-02-05T22:03:41.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:41.893+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.895164+0000) 2020-02-05T22:03:41.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:41.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.895348+0000) 2020-02-05T22:03:41.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:41.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.895604+0000) 2020-02-05T22:03:41.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:41.894+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.896278+0000) 2020-02-05T22:03:41.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:41.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.896677+0000) 2020-02-05T22:03:41.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:41.896+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.897726+0000) 2020-02-05T22:03:41.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:41.897+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.898097+0000) 2020-02-05T22:03:41.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:41.896+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.898318+0000) 2020-02-05T22:03:41.900 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:41.897+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.898535+0000) 2020-02-05T22:03:41.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:41.897+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.898500+0000) 2020-02-05T22:03:41.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:41.901+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:41.902738+0000) 2020-02-05T22:03:41.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:41.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:42.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:42.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:42.289395+0000) 2020-02-05T22:03:42.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:42.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:42.687501+0000) 2020-02-05T22:03:45.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:45.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:45.899800+0000) 2020-02-05T22:03:45.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:45.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:45.905772+0000) 2020-02-05T22:03:46.910 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:46.908+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.909175+0000) 2020-02-05T22:03:46.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:46.910+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.911639+0000) 2020-02-05T22:03:46.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:46.910+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.911921+0000) 2020-02-05T22:03:46.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:46.910+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.912383+0000) 2020-02-05T22:03:46.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:46.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.913276+0000) 2020-02-05T22:03:46.915 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:46.912+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.913795+0000) 2020-02-05T22:03:46.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:46.913+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.914502+0000) 2020-02-05T22:03:46.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:46.914+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.915409+0000) 2020-02-05T22:03:46.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:46.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.916449+0000) 2020-02-05T22:03:46.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:46.915+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.917032+0000) 2020-02-05T22:03:46.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:46.916+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.917455+0000) 2020-02-05T22:03:46.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:46.913+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.915193+0000) 2020-02-05T22:03:46.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:46.915+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.916699+0000) 2020-02-05T22:03:46.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:46.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:46.919464+0000) 2020-02-05T22:03:46.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:46.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:47.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:47.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:47.046347+0000) 2020-02-05T22:03:47.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:47.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:47.046523+0000) 2020-02-05T22:03:47.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:47.052+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:47.053122+0000) 2020-02-05T22:03:47.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:47.052+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:47.053233+0000) 2020-02-05T22:03:48.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:48.977+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:48.977748+0000) 2020-02-05T22:03:49.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:49.435+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:49.436579+0000) 2020-02-05T22:03:50.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:50.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:50.043598+0000) 2020-02-05T22:03:50.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:50.277+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:50.279009+0000) 2020-02-05T22:03:50.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:50.294+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:50.295869+0000) 2020-02-05T22:03:50.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:50.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:50.296631+0000) 2020-02-05T22:03:51.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:51.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.929001+0000) 2020-02-05T22:03:51.932 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:51.929+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.930841+0000) 2020-02-05T22:03:51.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:51.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.931492+0000) 2020-02-05T22:03:51.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:51.930+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.932135+0000) 2020-02-05T22:03:51.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:51.931+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.932866+0000) 2020-02-05T22:03:51.934 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:51.931+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.932890+0000) 2020-02-05T22:03:51.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:51.932+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.933695+0000) 2020-02-05T22:03:51.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:51.932+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.933873+0000) 2020-02-05T22:03:51.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:51.933+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.934251+0000) 2020-02-05T22:03:51.936 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:51.932+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.934017+0000) 2020-02-05T22:03:51.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:51.933+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.934791+0000) 2020-02-05T22:03:51.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:51.933+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.935050+0000) 2020-02-05T22:03:51.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:51.934+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.935577+0000) 2020-02-05T22:03:51.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:51.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:51.940226+0000) 2020-02-05T22:03:51.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:51.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:52.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:52.288+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:52.289683+0000) 2020-02-05T22:03:52.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:52.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:52.687707+0000) 2020-02-05T22:03:53.298 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:03:53.305 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:03:55.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:55.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:55.900027+0000) 2020-02-05T22:03:55.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:55.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:55.906023+0000) 2020-02-05T22:03:56.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:03:56.946+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.947772+0000) 2020-02-05T22:03:56.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:56.947+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.948887+0000) 2020-02-05T22:03:56.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:03:56.948+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.949552+0000) 2020-02-05T22:03:56.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:56.948+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.949987+0000) 2020-02-05T22:03:56.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:03:56.948+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.950032+0000) 2020-02-05T22:03:56.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:03:56.948+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.950402+0000) 2020-02-05T22:03:56.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:56.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.950587+0000) 2020-02-05T22:03:56.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:56.950+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.951604+0000) 2020-02-05T22:03:56.953 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:03:56.950+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.951800+0000) 2020-02-05T22:03:56.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:56.950+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.952042+0000) 2020-02-05T22:03:56.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:03:56.950+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.952142+0000) 2020-02-05T22:03:56.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:03:56.950+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.952047+0000) 2020-02-05T22:03:56.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:56.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.952978+0000) 2020-02-05T22:03:56.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:56.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:56.958018+0000) 2020-02-05T22:03:56.960 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:03:56.958+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104227 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:03:57.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:03:57.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:57.046568+0000) 2020-02-05T22:03:57.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:03:57.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:57.046758+0000) 2020-02-05T22:03:57.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:03:57.052+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:57.053359+0000) 2020-02-05T22:03:57.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:03:57.052+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:57.053414+0000) 2020-02-05T22:03:58.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:58.987+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:58.987575+0000) 2020-02-05T22:03:59.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:03:59.436+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:03:59.436803+0000) 2020-02-05T22:04:00.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:00.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:00.043844+0000) 2020-02-05T22:04:00.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:00.278+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:00.279234+0000) 2020-02-05T22:04:00.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:00.295+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:00.296143+0000) 2020-02-05T22:04:00.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:00.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:00.296945+0000) 2020-02-05T22:04:01.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:01.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.965782+0000) 2020-02-05T22:04:01.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:01.965+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.966114+0000) 2020-02-05T22:04:01.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:01.965+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.966957+0000) 2020-02-05T22:04:01.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:01.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.967287+0000) 2020-02-05T22:04:01.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:01.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.968708+0000) 2020-02-05T22:04:01.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:01.968+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.969728+0000) 2020-02-05T22:04:01.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:01.968+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.969504+0000) 2020-02-05T22:04:01.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:01.968+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.969965+0000) 2020-02-05T22:04:01.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:01.969+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.970290+0000) 2020-02-05T22:04:01.972 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:01.969+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.970715+0000) 2020-02-05T22:04:01.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:01.969+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.971199+0000) 2020-02-05T22:04:01.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:01.969+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.971348+0000) 2020-02-05T22:04:01.973 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:01.970+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.971606+0000) 2020-02-05T22:04:01.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:01.972+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:01.974015+0000) 2020-02-05T22:04:01.979 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:01.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104268 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:02.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:02.289+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:02.289919+0000) 2020-02-05T22:04:02.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:02.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:02.687941+0000) 2020-02-05T22:04:05.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:05.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:05.900293+0000) 2020-02-05T22:04:05.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:05.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:05.906275+0000) 2020-02-05T22:04:06.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:06.981+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.982835+0000) 2020-02-05T22:04:06.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:06.981+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.983045+0000) 2020-02-05T22:04:06.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:06.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.984707+0000) 2020-02-05T22:04:06.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:06.984+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.985646+0000) 2020-02-05T22:04:06.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:06.985+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.986286+0000) 2020-02-05T22:04:06.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:06.985+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.986807+0000) 2020-02-05T22:04:06.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:06.986+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.987412+0000) 2020-02-05T22:04:06.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:06.986+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.987951+0000) 2020-02-05T22:04:06.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:06.986+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.988030+0000) 2020-02-05T22:04:06.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:06.987+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.989108+0000) 2020-02-05T22:04:06.991 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:06.988+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.989732+0000) 2020-02-05T22:04:06.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:06.988+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.990007+0000) 2020-02-05T22:04:06.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:06.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.990691+0000) 2020-02-05T22:04:06.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:06.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:06.991762+0000) 2020-02-05T22:04:06.997 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:06.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:07.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:07.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:07.046724+0000) 2020-02-05T22:04:07.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:07.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:07.046972+0000) 2020-02-05T22:04:07.055 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:07.053+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:07.053599+0000) 2020-02-05T22:04:07.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:07.053+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:07.053601+0000) 2020-02-05T22:04:08.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:08.994+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:08.995593+0000) 2020-02-05T22:04:09.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:09.436+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:09.437036+0000) 2020-02-05T22:04:10.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:10.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:10.044109+0000) 2020-02-05T22:04:10.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:10.278+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:10.279480+0000) 2020-02-05T22:04:10.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:10.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:10.296373+0000) 2020-02-05T22:04:10.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:10.296+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:10.297202+0000) 2020-02-05T22:04:12.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:12.000+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.001475+0000) 2020-02-05T22:04:12.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:12.001+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.002125+0000) 2020-02-05T22:04:12.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:12.002+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.004054+0000) 2020-02-05T22:04:12.005 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:12.003+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.004523+0000) 2020-02-05T22:04:12.006 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:12.003+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.004503+0000) 2020-02-05T22:04:12.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:12.004+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.005221+0000) 2020-02-05T22:04:12.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:12.004+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.005141+0000) 2020-02-05T22:04:12.007 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:12.004+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.006188+0000) 2020-02-05T22:04:12.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:12.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.006388+0000) 2020-02-05T22:04:12.008 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:12.004+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.006075+0000) 2020-02-05T22:04:12.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:12.006+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.007179+0000) 2020-02-05T22:04:12.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:12.005+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.007043+0000) 2020-02-05T22:04:12.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:12.007+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.008748+0000) 2020-02-05T22:04:12.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:12.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.012021+0000) 2020-02-05T22:04:12.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:12.014+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:12.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:12.289+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.290126+0000) 2020-02-05T22:04:12.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:12.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:12.688175+0000) 2020-02-05T22:04:15.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:15.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:15.900476+0000) 2020-02-05T22:04:15.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:15.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:15.906508+0000) 2020-02-05T22:04:17.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:17.018+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.020038+0000) 2020-02-05T22:04:17.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:17.019+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.021000+0000) 2020-02-05T22:04:17.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:17.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.022286+0000) 2020-02-05T22:04:17.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:17.021+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.023213+0000) 2020-02-05T22:04:17.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:17.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.023835+0000) 2020-02-05T22:04:17.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:17.023+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.024790+0000) 2020-02-05T22:04:17.026 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:17.024+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.025481+0000) 2020-02-05T22:04:17.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:17.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.025349+0000) 2020-02-05T22:04:17.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:17.024+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.025743+0000) 2020-02-05T22:04:17.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:17.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.026343+0000) 2020-02-05T22:04:17.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:17.025+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.026878+0000) 2020-02-05T22:04:17.028 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:17.025+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.027079+0000) 2020-02-05T22:04:17.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:17.026+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.027478+0000) 2020-02-05T22:04:17.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:17.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.032562+0000) 2020-02-05T22:04:17.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:17.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104385 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:17.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:17.045+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.046918+0000) 2020-02-05T22:04:17.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:17.045+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.047172+0000) 2020-02-05T22:04:17.055 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:17.053+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.053852+0000) 2020-02-05T22:04:17.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:17.053+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:17.053851+0000) 2020-02-05T22:04:19.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:19.003+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:19.004026+0000) 2020-02-05T22:04:19.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:19.437+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:19.437347+0000) 2020-02-05T22:04:20.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:20.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:20.044351+0000) 2020-02-05T22:04:20.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:20.279+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:20.279682+0000) 2020-02-05T22:04:20.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:20.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:20.296622+0000) 2020-02-05T22:04:20.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:20.296+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:20.297484+0000) 2020-02-05T22:04:22.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:22.039+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.039496+0000) 2020-02-05T22:04:22.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:22.039+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.041373+0000) 2020-02-05T22:04:22.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:22.040+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.041755+0000) 2020-02-05T22:04:22.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:22.040+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.042368+0000) 2020-02-05T22:04:22.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:22.043+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.044584+0000) 2020-02-05T22:04:22.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:22.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.043255+0000) 2020-02-05T22:04:22.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:22.044+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.044200+0000) 2020-02-05T22:04:22.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:22.044+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.044473+0000) 2020-02-05T22:04:22.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:22.044+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.044413+0000) 2020-02-05T22:04:22.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:22.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.045010+0000) 2020-02-05T22:04:22.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:22.043+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.045143+0000) 2020-02-05T22:04:22.048 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:22.044+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.045531+0000) 2020-02-05T22:04:22.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:22.049+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.049298+0000) 2020-02-05T22:04:22.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:22.049+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.049548+0000) 2020-02-05T22:04:22.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:22.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:22.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:22.289+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.290350+0000) 2020-02-05T22:04:22.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:22.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:22.688401+0000) 2020-02-05T22:04:23.487 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:04:23.493 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:04:25.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:25.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:25.900724+0000) 2020-02-05T22:04:25.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:25.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:25.906819+0000) 2020-02-05T22:04:27.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:27.046+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.047103+0000) 2020-02-05T22:04:27.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:27.046+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.047303+0000) 2020-02-05T22:04:27.059 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:27.056+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.057900+0000) 2020-02-05T22:04:27.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:27.057+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.058641+0000) 2020-02-05T22:04:27.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:27.057+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.059045+0000) 2020-02-05T22:04:27.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:27.058+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.059439+0000) 2020-02-05T22:04:27.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:27.059+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.059901+0000) 2020-02-05T22:04:27.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:27.059+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.060202+0000) 2020-02-05T22:04:27.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:27.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.060713+0000) 2020-02-05T22:04:27.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:27.061+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.061605+0000) 2020-02-05T22:04:27.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:27.061+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.062292+0000) 2020-02-05T22:04:27.064 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:27.062+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.063229+0000) 2020-02-05T22:04:27.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:27.063+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.064003+0000) 2020-02-05T22:04:27.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:27.064+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.064499+0000) 2020-02-05T22:04:27.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:27.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.067151+0000) 2020-02-05T22:04:27.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:27.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:27.067399+0000) 2020-02-05T22:04:27.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:27.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:29.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:29.010+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:29.011780+0000) 2020-02-05T22:04:29.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:29.436+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:29.437578+0000) 2020-02-05T22:04:30.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:30.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:30.044586+0000) 2020-02-05T22:04:30.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:30.279+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:30.279917+0000) 2020-02-05T22:04:30.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:30.295+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:30.296846+0000) 2020-02-05T22:04:30.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:30.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:30.297737+0000) 2020-02-05T22:04:32.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:32.078+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.079323+0000) 2020-02-05T22:04:32.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:32.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.080266+0000) 2020-02-05T22:04:32.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:32.076+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.077193+0000) 2020-02-05T22:04:32.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:32.076+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.077895+0000) 2020-02-05T22:04:32.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:32.080+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.080588+0000) 2020-02-05T22:04:32.083 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:32.079+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.080914+0000) 2020-02-05T22:04:32.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:32.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.081163+0000) 2020-02-05T22:04:32.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:32.081+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.082338+0000) 2020-02-05T22:04:32.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:32.082+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.082610+0000) 2020-02-05T22:04:32.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:32.082+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.082774+0000) 2020-02-05T22:04:32.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:32.081+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.082544+0000) 2020-02-05T22:04:32.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:32.081+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.082958+0000) 2020-02-05T22:04:32.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:32.082+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.083245+0000) 2020-02-05T22:04:32.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:32.086+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.087595+0000) 2020-02-05T22:04:32.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:32.089+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:32.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:32.290+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.290638+0000) 2020-02-05T22:04:32.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:32.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:32.688616+0000) 2020-02-05T22:04:33.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:33.803+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:33.804828+0000) 2020-02-05T22:04:33.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:33.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:33.805615+0000) 2020-02-05T22:04:35.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:35.899+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:35.900983+0000) 2020-02-05T22:04:35.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:35.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:35.907072+0000) 2020-02-05T22:04:37.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:37.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.047359+0000) 2020-02-05T22:04:37.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:37.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.047531+0000) 2020-02-05T22:04:37.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:37.095+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.095987+0000) 2020-02-05T22:04:37.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:37.095+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.096372+0000) 2020-02-05T22:04:37.098 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:37.095+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.096603+0000) 2020-02-05T22:04:37.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:37.095+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.097057+0000) 2020-02-05T22:04:37.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:37.097+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.098327+0000) 2020-02-05T22:04:37.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:37.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.098506+0000) 2020-02-05T22:04:37.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:37.097+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.098832+0000) 2020-02-05T22:04:37.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:37.097+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.098975+0000) 2020-02-05T22:04:37.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:37.098+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.099352+0000) 2020-02-05T22:04:37.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:37.098+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.099242+0000) 2020-02-05T22:04:37.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:37.098+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.099336+0000) 2020-02-05T22:04:37.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:37.099+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.099890+0000) 2020-02-05T22:04:37.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:37.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.100499+0000) 2020-02-05T22:04:37.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:37.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:37.104638+0000) 2020-02-05T22:04:37.108 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:37.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:39.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:39.019+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:39.020600+0000) 2020-02-05T22:04:39.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:39.437+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:39.437836+0000) 2020-02-05T22:04:40.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:40.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:40.044835+0000) 2020-02-05T22:04:40.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:40.279+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:40.280163+0000) 2020-02-05T22:04:40.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:40.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:40.297072+0000) 2020-02-05T22:04:40.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:40.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:40.297978+0000) 2020-02-05T22:04:42.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:42.112+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.112625+0000) 2020-02-05T22:04:42.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:42.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.113232+0000) 2020-02-05T22:04:42.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:42.113+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.113981+0000) 2020-02-05T22:04:42.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:42.113+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.114134+0000) 2020-02-05T22:04:42.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:42.113+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.114613+0000) 2020-02-05T22:04:42.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:42.114+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.115335+0000) 2020-02-05T22:04:42.117 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:42.113+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.115023+0000) 2020-02-05T22:04:42.117 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:42.115+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.115920+0000) 2020-02-05T22:04:42.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:42.115+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.116341+0000) 2020-02-05T22:04:42.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:42.115+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.116522+0000) 2020-02-05T22:04:42.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:42.116+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.116693+0000) 2020-02-05T22:04:42.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:42.115+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.116757+0000) 2020-02-05T22:04:42.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:42.117+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.117626+0000) 2020-02-05T22:04:42.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:42.120+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.121234+0000) 2020-02-05T22:04:42.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:42.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:42.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:42.290+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.290955+0000) 2020-02-05T22:04:42.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:42.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:42.688826+0000) 2020-02-05T22:04:43.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:43.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:43.805129+0000) 2020-02-05T22:04:43.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:43.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:43.805776+0000) 2020-02-05T22:04:45.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:45.900+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:45.901272+0000) 2020-02-05T22:04:45.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:45.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:45.907322+0000) 2020-02-05T22:04:47.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:47.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.047635+0000) 2020-02-05T22:04:47.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:47.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.047824+0000) 2020-02-05T22:04:47.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:47.130+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.131351+0000) 2020-02-05T22:04:47.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:47.131+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.132282+0000) 2020-02-05T22:04:47.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:47.131+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.132842+0000) 2020-02-05T22:04:47.134 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:47.133+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.133526+0000) 2020-02-05T22:04:47.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:47.133+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.134061+0000) 2020-02-05T22:04:47.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:47.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.134644+0000) 2020-02-05T22:04:47.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:47.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.135262+0000) 2020-02-05T22:04:47.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:47.135+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.135897+0000) 2020-02-05T22:04:47.138 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:47.135+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.136579+0000) 2020-02-05T22:04:47.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:47.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.137198+0000) 2020-02-05T22:04:47.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:47.137+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.137548+0000) 2020-02-05T22:04:47.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:47.137+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.138286+0000) 2020-02-05T22:04:47.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:47.138+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.138444+0000) 2020-02-05T22:04:47.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:47.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:47.141878+0000) 2020-02-05T22:04:47.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:47.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:49.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:49.027+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:49.028456+0000) 2020-02-05T22:04:49.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:49.437+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:49.438064+0000) 2020-02-05T22:04:50.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:50.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:50.045105+0000) 2020-02-05T22:04:50.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:50.280+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:50.280352+0000) 2020-02-05T22:04:50.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:50.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:50.297371+0000) 2020-02-05T22:04:50.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:50.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:50.298208+0000) 2020-02-05T22:04:52.151 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:52.149+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.149600+0000) 2020-02-05T22:04:52.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:52.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.150461+0000) 2020-02-05T22:04:52.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:52.150+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.151143+0000) 2020-02-05T22:04:52.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:52.151+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.152380+0000) 2020-02-05T22:04:52.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:52.152+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.153672+0000) 2020-02-05T22:04:52.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:52.153+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.154293+0000) 2020-02-05T22:04:52.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:52.154+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.155034+0000) 2020-02-05T22:04:52.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:52.154+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.155399+0000) 2020-02-05T22:04:52.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:52.154+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.154980+0000) 2020-02-05T22:04:52.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:52.155+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.156763+0000) 2020-02-05T22:04:52.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:52.156+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.156747+0000) 2020-02-05T22:04:52.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:52.156+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.157378+0000) 2020-02-05T22:04:52.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:52.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.159677+0000) 2020-02-05T22:04:52.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:52.159+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.160108+0000) 2020-02-05T22:04:52.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:52.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:52.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:52.291+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.291225+0000) 2020-02-05T22:04:52.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:52.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:52.689042+0000) 2020-02-05T22:04:53.629 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:04:53.636 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:04:53.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:53.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:53.805364+0000) 2020-02-05T22:04:53.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:53.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:53.805930+0000) 2020-02-05T22:04:55.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:55.901+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:55.901502+0000) 2020-02-05T22:04:55.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:55.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:55.907570+0000) 2020-02-05T22:04:57.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:57.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.047868+0000) 2020-02-05T22:04:57.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:57.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.048093+0000) 2020-02-05T22:04:57.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:04:57.168+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.169214+0000) 2020-02-05T22:04:57.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:04:57.168+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.169993+0000) 2020-02-05T22:04:57.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:04:57.170+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.171151+0000) 2020-02-05T22:04:57.172 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:04:57.171+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.171612+0000) 2020-02-05T22:04:57.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:04:57.170+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.171983+0000) 2020-02-05T22:04:57.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:57.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.172630+0000) 2020-02-05T22:04:57.174 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:04:57.172+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.173510+0000) 2020-02-05T22:04:57.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:04:57.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.173563+0000) 2020-02-05T22:04:57.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:04:57.174+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.174734+0000) 2020-02-05T22:04:57.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:57.173+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.174517+0000) 2020-02-05T22:04:57.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:04:57.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.175221+0000) 2020-02-05T22:04:57.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:04:57.174+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.175801+0000) 2020-02-05T22:04:57.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:04:57.175+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.175695+0000) 2020-02-05T22:04:57.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:57.175+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:57.176762+0000) 2020-02-05T22:04:57.184 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:04:57.182+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:04:59.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:59.037+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:59.038039+0000) 2020-02-05T22:04:59.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:04:59.437+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:04:59.438358+0000) 2020-02-05T22:05:00.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:00.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:00.045329+0000) 2020-02-05T22:05:00.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:00.279+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:00.280639+0000) 2020-02-05T22:05:00.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:00.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:00.297584+0000) 2020-02-05T22:05:00.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:00.296+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:00.298399+0000) 2020-02-05T22:05:02.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:02.188+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.189554+0000) 2020-02-05T22:05:02.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:02.190+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.191056+0000) 2020-02-05T22:05:02.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:02.190+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.191421+0000) 2020-02-05T22:05:02.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:02.190+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.191918+0000) 2020-02-05T22:05:02.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:02.191+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.192175+0000) 2020-02-05T22:05:02.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:02.191+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.192638+0000) 2020-02-05T22:05:02.194 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:02.191+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.192946+0000) 2020-02-05T22:05:02.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:02.192+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.193759+0000) 2020-02-05T22:05:02.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:02.192+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.193871+0000) 2020-02-05T22:05:02.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:02.193+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.194237+0000) 2020-02-05T22:05:02.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:02.193+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.194700+0000) 2020-02-05T22:05:02.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:02.193+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.194800+0000) 2020-02-05T22:05:02.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:02.193+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.195060+0000) 2020-02-05T22:05:02.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:02.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.196343+0000) 2020-02-05T22:05:02.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:02.199+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:02.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:02.290+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.291497+0000) 2020-02-05T22:05:02.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:02.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:02.689277+0000) 2020-02-05T22:05:03.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:03.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:03.805573+0000) 2020-02-05T22:05:03.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:03.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:03.806117+0000) 2020-02-05T22:05:05.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:05.901+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:05.901738+0000) 2020-02-05T22:05:05.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:05.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:05.907825+0000) 2020-02-05T22:05:07.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:07.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.048114+0000) 2020-02-05T22:05:07.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:07.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.048362+0000) 2020-02-05T22:05:07.208 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:07.205+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.207300+0000) 2020-02-05T22:05:07.209 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:07.206+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.208028+0000) 2020-02-05T22:05:07.210 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:07.207+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.208804+0000) 2020-02-05T22:05:07.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:07.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.209953+0000) 2020-02-05T22:05:07.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:07.209+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.210405+0000) 2020-02-05T22:05:07.212 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:07.209+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.211003+0000) 2020-02-05T22:05:07.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:07.210+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.211748+0000) 2020-02-05T22:05:07.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:07.210+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.211879+0000) 2020-02-05T22:05:07.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:07.210+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.212363+0000) 2020-02-05T22:05:07.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:07.211+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.212804+0000) 2020-02-05T22:05:07.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:07.211+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.213070+0000) 2020-02-05T22:05:07.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:07.212+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.213447+0000) 2020-02-05T22:05:07.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:07.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.213782+0000) 2020-02-05T22:05:07.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:07.218+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104783 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:07.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:07.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:07.219511+0000) 2020-02-05T22:05:09.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:09.046+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:09.046599+0000) 2020-02-05T22:05:09.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:09.438+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:09.438532+0000) 2020-02-05T22:05:10.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:10.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:10.045590+0000) 2020-02-05T22:05:10.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:10.280+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:10.280938+0000) 2020-02-05T22:05:10.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:10.296+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:10.297851+0000) 2020-02-05T22:05:10.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:10.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:10.298685+0000) 2020-02-05T22:05:12.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:12.225+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.226417+0000) 2020-02-05T22:05:12.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:12.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.226253+0000) 2020-02-05T22:05:12.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:12.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.227221+0000) 2020-02-05T22:05:12.229 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:12.226+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.227301+0000) 2020-02-05T22:05:12.229 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:12.226+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.227572+0000) 2020-02-05T22:05:12.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:12.226+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.227847+0000) 2020-02-05T22:05:12.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:12.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.228316+0000) 2020-02-05T22:05:12.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:12.228+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.229555+0000) 2020-02-05T22:05:12.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:12.228+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.230098+0000) 2020-02-05T22:05:12.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:12.229+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.230633+0000) 2020-02-05T22:05:12.232 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:12.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.230504+0000) 2020-02-05T22:05:12.233 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:12.229+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.230437+0000) 2020-02-05T22:05:12.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:12.229+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.231241+0000) 2020-02-05T22:05:12.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:12.232+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.233804+0000) 2020-02-05T22:05:12.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:12.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104824 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:12.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:12.290+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.291714+0000) 2020-02-05T22:05:12.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:12.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:12.689488+0000) 2020-02-05T22:05:13.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:13.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:13.805803+0000) 2020-02-05T22:05:13.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:13.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:13.806299+0000) 2020-02-05T22:05:15.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:15.901+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:15.901964+0000) 2020-02-05T22:05:15.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:15.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:15.908077+0000) 2020-02-05T22:05:17.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:17.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.048364+0000) 2020-02-05T22:05:17.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:17.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.048636+0000) 2020-02-05T22:05:17.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:17.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.244103+0000) 2020-02-05T22:05:17.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:17.244+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.244315+0000) 2020-02-05T22:05:17.246 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:17.244+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.244719+0000) 2020-02-05T22:05:17.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:17.245+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.245202+0000) 2020-02-05T22:05:17.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:17.245+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.245858+0000) 2020-02-05T22:05:17.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:17.246+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.246501+0000) 2020-02-05T22:05:17.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:17.245+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.247210+0000) 2020-02-05T22:05:17.248 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:17.247+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.247420+0000) 2020-02-05T22:05:17.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:17.246+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.247945+0000) 2020-02-05T22:05:17.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:17.247+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.248546+0000) 2020-02-05T22:05:17.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:17.247+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.248847+0000) 2020-02-05T22:05:17.250 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:17.249+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.249183+0000) 2020-02-05T22:05:17.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:17.248+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.249423+0000) 2020-02-05T22:05:17.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:17.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:17.252381+0000) 2020-02-05T22:05:17.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:17.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:19.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:19.056+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:19.056997+0000) 2020-02-05T22:05:19.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:19.438+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:19.438761+0000) 2020-02-05T22:05:20.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:20.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:20.045827+0000) 2020-02-05T22:05:20.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:20.280+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:20.281187+0000) 2020-02-05T22:05:20.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:20.297+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:20.298072+0000) 2020-02-05T22:05:20.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:20.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:20.298930+0000) 2020-02-05T22:05:22.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:22.261+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.263344+0000) 2020-02-05T22:05:22.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:22.262+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.263681+0000) 2020-02-05T22:05:22.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:22.263+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.264244+0000) 2020-02-05T22:05:22.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:22.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.264759+0000) 2020-02-05T22:05:22.266 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:22.263+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.264921+0000) 2020-02-05T22:05:22.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:22.263+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.265340+0000) 2020-02-05T22:05:22.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:22.264+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.265565+0000) 2020-02-05T22:05:22.267 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:22.264+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.265918+0000) 2020-02-05T22:05:22.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:22.264+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.266316+0000) 2020-02-05T22:05:22.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:22.265+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.266556+0000) 2020-02-05T22:05:22.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:22.266+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.267613+0000) 2020-02-05T22:05:22.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:22.266+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.267221+0000) 2020-02-05T22:05:22.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:22.266+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.268012+0000) 2020-02-05T22:05:22.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:22.270+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.272002+0000) 2020-02-05T22:05:22.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:22.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104904 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:22.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:22.291+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.291947+0000) 2020-02-05T22:05:22.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:22.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:22.689696+0000) 2020-02-05T22:05:23.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:23.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:23.806039+0000) 2020-02-05T22:05:23.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:23.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:23.806487+0000) 2020-02-05T22:05:23.822 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:05:23.830 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:05:25.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:25.901+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:25.902189+0000) 2020-02-05T22:05:25.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:25.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:25.908330+0000) 2020-02-05T22:05:27.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:27.047+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.048642+0000) 2020-02-05T22:05:27.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:27.047+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.048836+0000) 2020-02-05T22:05:27.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:27.280+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.281514+0000) 2020-02-05T22:05:27.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:27.281+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.282300+0000) 2020-02-05T22:05:27.286 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:27.281+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.282707+0000) 2020-02-05T22:05:27.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:27.282+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.283421+0000) 2020-02-05T22:05:27.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:27.282+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.284034+0000) 2020-02-05T22:05:27.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:27.283+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.284562+0000) 2020-02-05T22:05:27.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:27.281+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.282802+0000) 2020-02-05T22:05:27.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:27.283+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.284642+0000) 2020-02-05T22:05:27.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:27.284+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.285757+0000) 2020-02-05T22:05:27.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:27.284+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.285423+0000) 2020-02-05T22:05:27.291 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:27.284+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.286294+0000) 2020-02-05T22:05:27.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:27.286+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.287528+0000) 2020-02-05T22:05:27.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:27.285+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.286103+0000) 2020-02-05T22:05:27.298 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:27.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104941 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:27.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:27.291+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:27.292584+0000) 2020-02-05T22:05:29.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:29.064+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:29.065225+0000) 2020-02-05T22:05:29.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:29.438+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:29.439014+0000) 2020-02-05T22:05:30.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:30.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:30.046067+0000) 2020-02-05T22:05:30.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:30.280+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:30.281477+0000) 2020-02-05T22:05:30.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:30.297+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:30.298397+0000) 2020-02-05T22:05:30.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:30.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:30.299133+0000) 2020-02-05T22:05:32.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:32.291+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.292169+0000) 2020-02-05T22:05:32.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:32.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.300623+0000) 2020-02-05T22:05:32.302 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:32.300+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.301395+0000) 2020-02-05T22:05:32.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:32.300+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.301424+0000) 2020-02-05T22:05:32.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:32.301+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.302308+0000) 2020-02-05T22:05:32.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:32.301+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.302800+0000) 2020-02-05T22:05:32.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:32.301+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.303402+0000) 2020-02-05T22:05:32.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:32.302+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.304013+0000) 2020-02-05T22:05:32.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:32.303+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.304310+0000) 2020-02-05T22:05:32.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:32.303+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.304570+0000) 2020-02-05T22:05:32.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:32.303+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.304910+0000) 2020-02-05T22:05:32.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:32.304+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.305564+0000) 2020-02-05T22:05:32.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:32.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.305652+0000) 2020-02-05T22:05:32.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:32.304+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.305841+0000) 2020-02-05T22:05:32.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:32.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.310341+0000) 2020-02-05T22:05:32.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:32.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 104982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:32.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:32.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:32.689939+0000) 2020-02-05T22:05:33.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:33.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:33.806345+0000) 2020-02-05T22:05:33.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:33.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:33.806700+0000) 2020-02-05T22:05:35.903 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:35.901+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:35.902381+0000) 2020-02-05T22:05:35.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:35.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:35.908584+0000) 2020-02-05T22:05:37.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:37.048+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.048887+0000) 2020-02-05T22:05:37.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:37.048+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.049080+0000) 2020-02-05T22:05:37.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:37.317+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.318733+0000) 2020-02-05T22:05:37.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:37.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.320393+0000) 2020-02-05T22:05:37.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:37.319+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.320464+0000) 2020-02-05T22:05:37.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:37.319+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.320990+0000) 2020-02-05T22:05:37.323 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:37.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.321378+0000) 2020-02-05T22:05:37.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:37.320+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.321778+0000) 2020-02-05T22:05:37.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:37.320+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.322359+0000) 2020-02-05T22:05:37.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:37.321+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.322638+0000) 2020-02-05T22:05:37.324 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:37.321+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.322960+0000) 2020-02-05T22:05:37.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:37.322+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.323519+0000) 2020-02-05T22:05:37.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:37.321+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.323127+0000) 2020-02-05T22:05:37.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:37.323+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.324830+0000) 2020-02-05T22:05:37.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:37.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.329513+0000) 2020-02-05T22:05:37.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:37.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:37.329948+0000) 2020-02-05T22:05:37.333 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:37.330+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:39.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:39.072+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:39.073521+0000) 2020-02-05T22:05:39.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:39.438+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:39.439270+0000) 2020-02-05T22:05:40.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:40.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:40.046365+0000) 2020-02-05T22:05:40.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:40.281+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:40.281722+0000) 2020-02-05T22:05:40.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:40.297+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:40.298671+0000) 2020-02-05T22:05:40.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:40.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:40.299335+0000) 2020-02-05T22:05:42.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:42.291+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.292456+0000) 2020-02-05T22:05:42.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:42.335+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.336707+0000) 2020-02-05T22:05:42.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:42.337+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.338322+0000) 2020-02-05T22:05:42.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:42.336+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.338192+0000) 2020-02-05T22:05:42.340 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:42.338+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.339386+0000) 2020-02-05T22:05:42.342 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:42.340+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.341506+0000) 2020-02-05T22:05:42.343 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:42.341+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.342140+0000) 2020-02-05T22:05:42.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:42.341+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.342910+0000) 2020-02-05T22:05:42.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:42.341+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.343017+0000) 2020-02-05T22:05:42.345 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:42.342+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.343459+0000) 2020-02-05T22:05:42.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:42.341+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.343302+0000) 2020-02-05T22:05:42.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:42.342+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.343635+0000) 2020-02-05T22:05:42.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:42.344+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.345155+0000) 2020-02-05T22:05:42.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:42.342+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.343834+0000) 2020-02-05T22:05:42.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:42.345+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.346286+0000) 2020-02-05T22:05:42.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:42.348+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:42.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:42.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:42.690177+0000) 2020-02-05T22:05:43.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:43.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:43.806609+0000) 2020-02-05T22:05:43.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:43.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:43.806896+0000) 2020-02-05T22:05:45.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:45.902+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:45.902641+0000) 2020-02-05T22:05:45.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:45.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:45.908831+0000) 2020-02-05T22:05:47.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:47.048+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.049143+0000) 2020-02-05T22:05:47.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:47.048+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.049304+0000) 2020-02-05T22:05:47.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:47.354+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.355976+0000) 2020-02-05T22:05:47.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:47.356+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.357276+0000) 2020-02-05T22:05:47.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:47.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.357955+0000) 2020-02-05T22:05:47.359 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:47.357+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.358058+0000) 2020-02-05T22:05:47.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:47.357+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.358341+0000) 2020-02-05T22:05:47.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:47.359+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.359853+0000) 2020-02-05T22:05:47.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:47.358+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.359786+0000) 2020-02-05T22:05:47.362 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:47.359+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.360719+0000) 2020-02-05T22:05:47.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:47.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.361204+0000) 2020-02-05T22:05:47.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:47.361+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.361813+0000) 2020-02-05T22:05:47.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:47.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.362184+0000) 2020-02-05T22:05:47.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:47.361+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.362598+0000) 2020-02-05T22:05:47.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:47.361+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.362747+0000) 2020-02-05T22:05:47.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:47.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:47.367373+0000) 2020-02-05T22:05:47.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:47.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:49.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:49.083+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:49.083620+0000) 2020-02-05T22:05:49.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:49.438+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:49.439510+0000) 2020-02-05T22:05:50.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:50.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:50.046603+0000) 2020-02-05T22:05:50.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:50.281+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:50.281979+0000) 2020-02-05T22:05:50.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:50.298+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:50.298939+0000) 2020-02-05T22:05:50.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:50.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:50.299683+0000) 2020-02-05T22:05:52.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:52.292+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.292714+0000) 2020-02-05T22:05:52.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:52.374+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.375330+0000) 2020-02-05T22:05:52.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:52.375+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.376668+0000) 2020-02-05T22:05:52.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:52.375+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.376963+0000) 2020-02-05T22:05:52.379 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:52.376+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.377717+0000) 2020-02-05T22:05:52.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:52.378+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.378450+0000) 2020-02-05T22:05:52.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:52.378+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.379000+0000) 2020-02-05T22:05:52.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:52.378+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.379473+0000) 2020-02-05T22:05:52.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:52.379+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.379937+0000) 2020-02-05T22:05:52.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:52.379+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.380464+0000) 2020-02-05T22:05:52.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:52.379+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.380912+0000) 2020-02-05T22:05:52.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:52.380+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.381289+0000) 2020-02-05T22:05:52.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:52.381+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.381472+0000) 2020-02-05T22:05:52.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:52.381+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.381549+0000) 2020-02-05T22:05:52.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:52.386+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.387613+0000) 2020-02-05T22:05:52.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:52.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:52.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:52.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:52.690406+0000) 2020-02-05T22:05:53.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:53.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:53.806841+0000) 2020-02-05T22:05:53.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:53.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:53.807140+0000) 2020-02-05T22:05:53.994 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:05:54.000 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:05:55.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:55.902+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:55.902869+0000) 2020-02-05T22:05:55.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:55.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:55.909082+0000) 2020-02-05T22:05:57.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:57.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.049400+0000) 2020-02-05T22:05:57.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:57.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.049522+0000) 2020-02-05T22:05:57.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:57.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.393262+0000) 2020-02-05T22:05:57.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:05:57.393+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.394800+0000) 2020-02-05T22:05:57.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:05:57.394+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.395630+0000) 2020-02-05T22:05:57.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:05:57.395+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.396249+0000) 2020-02-05T22:05:57.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:05:57.396+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.396625+0000) 2020-02-05T22:05:57.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:05:57.396+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.396836+0000) 2020-02-05T22:05:57.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:05:57.396+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.397282+0000) 2020-02-05T22:05:57.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:05:57.396+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.397626+0000) 2020-02-05T22:05:57.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:05:57.397+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.398187+0000) 2020-02-05T22:05:57.400 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:05:57.398+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.398620+0000) 2020-02-05T22:05:57.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:05:57.397+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.398899+0000) 2020-02-05T22:05:57.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:05:57.399+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.399576+0000) 2020-02-05T22:05:57.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:57.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.402772+0000) 2020-02-05T22:05:57.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:57.402+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:57.403896+0000) 2020-02-05T22:05:57.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:05:57.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:05:59.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:59.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:59.091503+0000) 2020-02-05T22:05:59.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:05:59.439+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:05:59.439738+0000) 2020-02-05T22:06:00.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:00.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:00.046844+0000) 2020-02-05T22:06:00.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:00.281+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:00.282232+0000) 2020-02-05T22:06:00.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:00.298+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:00.299170+0000) 2020-02-05T22:06:00.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:00.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:00.299918+0000) 2020-02-05T22:06:02.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:02.292+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.292955+0000) 2020-02-05T22:06:02.415 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:02.412+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.413600+0000) 2020-02-05T22:06:02.416 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:02.413+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.415046+0000) 2020-02-05T22:06:02.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:02.415+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.415810+0000) 2020-02-05T22:06:02.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:02.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.416540+0000) 2020-02-05T22:06:02.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:02.416+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.417214+0000) 2020-02-05T22:06:02.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:02.417+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.418033+0000) 2020-02-05T22:06:02.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:02.417+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.417722+0000) 2020-02-05T22:06:02.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:02.417+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.418205+0000) 2020-02-05T22:06:02.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:02.418+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.418827+0000) 2020-02-05T22:06:02.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:02.418+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.419078+0000) 2020-02-05T22:06:02.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:02.417+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.418648+0000) 2020-02-05T22:06:02.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:02.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.418792+0000) 2020-02-05T22:06:02.422 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:02.418+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.419470+0000) 2020-02-05T22:06:02.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:02.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.422133+0000) 2020-02-05T22:06:02.427 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:02.426+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:02.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:02.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:02.690618+0000) 2020-02-05T22:06:03.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:03.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:03.807137+0000) 2020-02-05T22:06:03.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:03.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:03.807356+0000) 2020-02-05T22:06:05.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:05.902+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:05.903123+0000) 2020-02-05T22:06:05.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:05.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:05.909333+0000) 2020-02-05T22:06:07.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:07.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.049697+0000) 2020-02-05T22:06:07.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:07.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.049756+0000) 2020-02-05T22:06:07.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:07.432+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.433899+0000) 2020-02-05T22:06:07.436 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:07.433+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.434399+0000) 2020-02-05T22:06:07.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:07.434+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.435072+0000) 2020-02-05T22:06:07.437 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:07.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.435458+0000) 2020-02-05T22:06:07.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:07.434+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.434974+0000) 2020-02-05T22:06:07.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:07.435+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.436288+0000) 2020-02-05T22:06:07.438 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:07.435+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.436349+0000) 2020-02-05T22:06:07.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:07.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.437200+0000) 2020-02-05T22:06:07.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:07.437+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.437595+0000) 2020-02-05T22:06:07.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:07.437+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.438196+0000) 2020-02-05T22:06:07.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:07.436+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.437831+0000) 2020-02-05T22:06:07.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:07.438+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.438676+0000) 2020-02-05T22:06:07.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:07.438+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.438694+0000) 2020-02-05T22:06:07.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:07.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:07.442476+0000) 2020-02-05T22:06:07.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:07.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:09.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:09.102+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:09.103130+0000) 2020-02-05T22:06:09.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:09.439+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:09.439960+0000) 2020-02-05T22:06:10.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:10.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:10.047124+0000) 2020-02-05T22:06:10.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:10.282+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:10.282482+0000) 2020-02-05T22:06:10.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:10.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:10.300186+0000) 2020-02-05T22:06:10.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:10.299+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:10.299393+0000) 2020-02-05T22:06:12.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:12.292+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.293194+0000) 2020-02-05T22:06:12.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:12.451+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.451132+0000) 2020-02-05T22:06:12.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:12.451+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.451614+0000) 2020-02-05T22:06:12.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:12.451+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.452041+0000) 2020-02-05T22:06:12.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:12.451+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.451999+0000) 2020-02-05T22:06:12.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:12.452+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.452995+0000) 2020-02-05T22:06:12.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:12.453+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.454309+0000) 2020-02-05T22:06:12.457 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:12.454+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.455203+0000) 2020-02-05T22:06:12.457 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:12.454+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.455140+0000) 2020-02-05T22:06:12.457 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:12.455+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.455839+0000) 2020-02-05T22:06:12.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:12.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.455885+0000) 2020-02-05T22:06:12.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:12.456+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.456590+0000) 2020-02-05T22:06:12.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:12.455+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.456244+0000) 2020-02-05T22:06:12.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:12.458+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.458991+0000) 2020-02-05T22:06:12.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:12.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.460102+0000) 2020-02-05T22:06:12.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:12.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:12.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:12.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:12.690832+0000) 2020-02-05T22:06:13.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:13.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:13.807376+0000) 2020-02-05T22:06:13.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:13.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:13.807565+0000) 2020-02-05T22:06:15.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:15.902+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:15.903355+0000) 2020-02-05T22:06:15.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:15.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:15.909581+0000) 2020-02-05T22:06:17.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:17.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.049960+0000) 2020-02-05T22:06:17.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:17.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.050027+0000) 2020-02-05T22:06:17.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:17.469+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.469809+0000) 2020-02-05T22:06:17.471 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:17.469+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.470193+0000) 2020-02-05T22:06:17.471 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:17.469+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.470396+0000) 2020-02-05T22:06:17.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:17.469+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.470755+0000) 2020-02-05T22:06:17.472 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:17.469+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.470971+0000) 2020-02-05T22:06:17.473 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:17.471+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.471423+0000) 2020-02-05T22:06:17.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:17.470+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.471839+0000) 2020-02-05T22:06:17.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:17.471+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.472184+0000) 2020-02-05T22:06:17.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:17.472+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.473203+0000) 2020-02-05T22:06:17.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:17.472+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.472639+0000) 2020-02-05T22:06:17.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:17.472+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.473059+0000) 2020-02-05T22:06:17.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:17.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.474050+0000) 2020-02-05T22:06:17.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:17.474+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.475023+0000) 2020-02-05T22:06:17.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:17.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:17.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:17.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:17.477419+0000) 2020-02-05T22:06:19.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:19.113+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:19.114768+0000) 2020-02-05T22:06:19.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:19.439+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:19.440178+0000) 2020-02-05T22:06:20.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:20.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:20.047349+0000) 2020-02-05T22:06:20.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:20.282+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:20.282716+0000) 2020-02-05T22:06:20.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:20.298+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:20.299583+0000) 2020-02-05T22:06:20.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:20.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:20.300367+0000) 2020-02-05T22:06:22.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:22.293+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.293486+0000) 2020-02-05T22:06:22.488 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:22.485+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.486532+0000) 2020-02-05T22:06:22.488 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:22.486+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.487659+0000) 2020-02-05T22:06:22.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:22.487+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.488178+0000) 2020-02-05T22:06:22.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:22.488+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.488897+0000) 2020-02-05T22:06:22.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:22.488+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.489242+0000) 2020-02-05T22:06:22.491 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:22.488+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.489776+0000) 2020-02-05T22:06:22.492 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:22.489+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.490821+0000) 2020-02-05T22:06:22.492 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:22.490+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.491333+0000) 2020-02-05T22:06:22.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:22.491+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.492113+0000) 2020-02-05T22:06:22.493 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:22.492+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.492557+0000) 2020-02-05T22:06:22.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:22.492+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.493089+0000) 2020-02-05T22:06:22.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:22.492+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.492982+0000) 2020-02-05T22:06:22.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:22.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.499514+0000) 2020-02-05T22:06:22.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:22.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.499823+0000) 2020-02-05T22:06:22.501 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:22.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:22.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:22.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:22.691036+0000) 2020-02-05T22:06:23.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:23.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:23.807634+0000) 2020-02-05T22:06:23.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:23.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:23.807799+0000) 2020-02-05T22:06:24.186 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:06:24.192 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:06:25.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:25.902+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:25.903627+0000) 2020-02-05T22:06:25.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:25.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:25.909859+0000) 2020-02-05T22:06:27.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:27.048+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.050259+0000) 2020-02-05T22:06:27.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:27.048+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.050265+0000) 2020-02-05T22:06:27.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:27.503+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.505033+0000) 2020-02-05T22:06:27.507 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:27.504+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.505452+0000) 2020-02-05T22:06:27.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:27.504+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.506033+0000) 2020-02-05T22:06:27.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:27.505+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.506446+0000) 2020-02-05T22:06:27.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:27.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.507871+0000) 2020-02-05T22:06:27.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:27.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.508843+0000) 2020-02-05T22:06:27.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:27.507+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.509316+0000) 2020-02-05T22:06:27.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:27.509+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.510591+0000) 2020-02-05T22:06:27.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:27.509+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.511003+0000) 2020-02-05T22:06:27.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:27.510+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.511468+0000) 2020-02-05T22:06:27.513 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:27.510+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.511600+0000) 2020-02-05T22:06:27.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:27.510+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.512404+0000) 2020-02-05T22:06:27.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:27.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.514193+0000) 2020-02-05T22:06:27.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:27.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:27.514410+0000) 2020-02-05T22:06:27.519 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:27.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:29.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:29.121+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:29.122892+0000) 2020-02-05T22:06:29.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:29.439+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:29.440407+0000) 2020-02-05T22:06:30.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:30.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:30.047534+0000) 2020-02-05T22:06:30.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:30.281+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:30.282957+0000) 2020-02-05T22:06:30.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:30.299+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:30.299814+0000) 2020-02-05T22:06:30.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:30.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:30.300703+0000) 2020-02-05T22:06:32.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:32.292+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.293773+0000) 2020-02-05T22:06:32.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:32.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.523332+0000) 2020-02-05T22:06:32.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:32.522+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.523850+0000) 2020-02-05T22:06:32.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:32.523+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.525397+0000) 2020-02-05T22:06:32.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:32.524+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.526058+0000) 2020-02-05T22:06:32.528 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:32.525+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.527267+0000) 2020-02-05T22:06:32.529 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:32.526+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.527994+0000) 2020-02-05T22:06:32.530 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:32.528+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.529329+0000) 2020-02-05T22:06:32.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:32.528+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.529394+0000) 2020-02-05T22:06:32.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:32.528+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.529731+0000) 2020-02-05T22:06:32.531 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:32.528+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.530047+0000) 2020-02-05T22:06:32.532 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:32.528+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.529759+0000) 2020-02-05T22:06:32.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:32.529+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.530241+0000) 2020-02-05T22:06:32.532 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:32.529+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.530785+0000) 2020-02-05T22:06:32.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:32.530+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.531237+0000) 2020-02-05T22:06:32.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:32.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:32.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:32.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:32.691268+0000) 2020-02-05T22:06:33.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:33.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:33.807881+0000) 2020-02-05T22:06:33.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:33.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:33.808041+0000) 2020-02-05T22:06:35.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:35.903+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:35.903841+0000) 2020-02-05T22:06:35.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:35.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:35.910097+0000) 2020-02-05T22:06:37.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:37.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.050525+0000) 2020-02-05T22:06:37.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:37.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.050541+0000) 2020-02-05T22:06:37.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:37.541+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.543029+0000) 2020-02-05T22:06:37.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:37.542+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.544031+0000) 2020-02-05T22:06:37.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:37.543+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.544975+0000) 2020-02-05T22:06:37.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:37.544+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.545410+0000) 2020-02-05T22:06:37.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:37.545+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.546488+0000) 2020-02-05T22:06:37.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:37.545+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.546553+0000) 2020-02-05T22:06:37.549 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:37.545+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.546781+0000) 2020-02-05T22:06:37.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:37.546+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.547666+0000) 2020-02-05T22:06:37.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:37.546+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.547961+0000) 2020-02-05T22:06:37.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:37.547+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.548622+0000) 2020-02-05T22:06:37.550 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:37.547+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.548781+0000) 2020-02-05T22:06:37.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:37.547+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.548987+0000) 2020-02-05T22:06:37.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:37.547+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.548953+0000) 2020-02-05T22:06:37.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:37.552+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:37.553292+0000) 2020-02-05T22:06:37.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:37.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:39.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:39.129+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:39.130188+0000) 2020-02-05T22:06:39.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:39.440+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:39.440640+0000) 2020-02-05T22:06:40.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:40.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:40.047765+0000) 2020-02-05T22:06:40.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:40.282+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:40.283206+0000) 2020-02-05T22:06:40.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:40.299+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:40.300035+0000) 2020-02-05T22:06:40.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:40.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:40.300913+0000) 2020-02-05T22:06:42.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:42.293+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.294007+0000) 2020-02-05T22:06:42.562 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:42.559+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.561028+0000) 2020-02-05T22:06:42.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:42.560+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.562191+0000) 2020-02-05T22:06:42.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:42.561+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.562546+0000) 2020-02-05T22:06:42.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:42.561+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.562597+0000) 2020-02-05T22:06:42.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:42.561+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.563071+0000) 2020-02-05T22:06:42.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:42.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.563641+0000) 2020-02-05T22:06:42.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:42.563+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.564501+0000) 2020-02-05T22:06:42.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:42.562+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.564090+0000) 2020-02-05T22:06:42.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:42.563+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.564966+0000) 2020-02-05T22:06:42.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:42.563+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.565288+0000) 2020-02-05T22:06:42.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:42.563+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.565415+0000) 2020-02-05T22:06:42.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:42.564+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.565643+0000) 2020-02-05T22:06:42.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:42.564+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.566009+0000) 2020-02-05T22:06:42.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:42.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.570499+0000) 2020-02-05T22:06:42.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:42.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:42.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:42.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:42.691434+0000) 2020-02-05T22:06:43.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:43.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:43.808132+0000) 2020-02-05T22:06:43.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:43.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:43.808332+0000) 2020-02-05T22:06:45.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:45.903+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:45.904066+0000) 2020-02-05T22:06:45.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:45.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:45.910361+0000) 2020-02-05T22:06:47.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:47.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.050771+0000) 2020-02-05T22:06:47.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:47.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.050780+0000) 2020-02-05T22:06:47.582 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:47.579+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.580797+0000) 2020-02-05T22:06:47.583 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:47.579+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.580937+0000) 2020-02-05T22:06:47.583 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:47.580+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.581319+0000) 2020-02-05T22:06:47.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:47.581+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.582718+0000) 2020-02-05T22:06:47.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:47.581+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.582174+0000) 2020-02-05T22:06:47.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:47.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.582495+0000) 2020-02-05T22:06:47.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:47.581+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.583218+0000) 2020-02-05T22:06:47.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:47.582+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.583937+0000) 2020-02-05T22:06:47.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:47.582+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.584297+0000) 2020-02-05T22:06:47.587 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:47.582+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.583526+0000) 2020-02-05T22:06:47.587 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:47.583+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.584619+0000) 2020-02-05T22:06:47.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:47.583+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.585281+0000) 2020-02-05T22:06:47.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:47.584+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.585928+0000) 2020-02-05T22:06:47.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:47.591+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:47.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:47.590+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:47.592052+0000) 2020-02-05T22:06:49.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:49.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:49.137007+0000) 2020-02-05T22:06:49.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:49.440+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:49.440861+0000) 2020-02-05T22:06:50.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:50.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:50.047983+0000) 2020-02-05T22:06:50.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:50.282+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:50.283474+0000) 2020-02-05T22:06:50.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:50.299+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:50.300320+0000) 2020-02-05T22:06:50.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:50.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:50.301144+0000) 2020-02-05T22:06:52.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:52.293+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.294259+0000) 2020-02-05T22:06:52.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:52.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.598309+0000) 2020-02-05T22:06:52.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:52.597+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.598963+0000) 2020-02-05T22:06:52.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:52.597+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.599303+0000) 2020-02-05T22:06:52.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:52.598+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.599945+0000) 2020-02-05T22:06:52.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:52.599+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.601163+0000) 2020-02-05T22:06:52.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:52.600+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.601417+0000) 2020-02-05T22:06:52.603 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:52.600+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.601539+0000) 2020-02-05T22:06:52.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:52.601+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.602238+0000) 2020-02-05T22:06:52.603 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:52.600+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.602148+0000) 2020-02-05T22:06:52.604 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:52.602+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.603221+0000) 2020-02-05T22:06:52.605 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:52.601+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.602948+0000) 2020-02-05T22:06:52.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:52.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.603630+0000) 2020-02-05T22:06:52.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:52.600+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.602162+0000) 2020-02-05T22:06:52.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:52.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.606208+0000) 2020-02-05T22:06:52.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:52.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:52.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:52.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:52.691645+0000) 2020-02-05T22:06:53.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:53.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:53.808335+0000) 2020-02-05T22:06:53.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:53.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:53.808573+0000) 2020-02-05T22:06:54.375 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:06:54.381 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:06:55.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:55.904+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:55.904344+0000) 2020-02-05T22:06:55.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:55.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:55.910625+0000) 2020-02-05T22:06:57.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:57.049+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.051019+0000) 2020-02-05T22:06:57.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:57.049+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.050991+0000) 2020-02-05T22:06:57.615 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:06:57.612+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.614009+0000) 2020-02-05T22:06:57.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:06:57.613+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.615023+0000) 2020-02-05T22:06:57.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:06:57.614+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.615389+0000) 2020-02-05T22:06:57.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:57.615+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.616503+0000) 2020-02-05T22:06:57.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:06:57.616+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.617642+0000) 2020-02-05T22:06:57.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:57.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.618193+0000) 2020-02-05T22:06:57.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:06:57.617+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.618665+0000) 2020-02-05T22:06:57.620 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:06:57.617+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.618612+0000) 2020-02-05T22:06:57.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:06:57.617+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.619246+0000) 2020-02-05T22:06:57.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:06:57.618+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.619542+0000) 2020-02-05T22:06:57.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:06:57.618+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.619860+0000) 2020-02-05T22:06:57.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:06:57.618+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.620235+0000) 2020-02-05T22:06:57.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:06:57.618+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.620411+0000) 2020-02-05T22:06:57.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:57.622+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:57.623571+0000) 2020-02-05T22:06:57.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:06:57.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:06:59.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:59.146+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:59.146859+0000) 2020-02-05T22:06:59.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:06:59.440+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:06:59.441127+0000) 2020-02-05T22:07:00.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:00.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:00.048207+0000) 2020-02-05T22:07:00.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:00.283+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:00.283724+0000) 2020-02-05T22:07:00.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:00.300+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:00.300542+0000) 2020-02-05T22:07:00.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:00.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:00.301344+0000) 2020-02-05T22:07:02.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:02.293+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.294502+0000) 2020-02-05T22:07:02.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:02.631+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.633056+0000) 2020-02-05T22:07:02.635 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:02.632+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.633638+0000) 2020-02-05T22:07:02.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:02.633+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.634598+0000) 2020-02-05T22:07:02.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:02.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.635182+0000) 2020-02-05T22:07:02.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:02.634+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.635792+0000) 2020-02-05T22:07:02.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:02.634+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.636269+0000) 2020-02-05T22:07:02.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:02.634+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.636398+0000) 2020-02-05T22:07:02.638 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:02.635+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.636881+0000) 2020-02-05T22:07:02.638 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:02.636+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.637146+0000) 2020-02-05T22:07:02.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:02.637+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.638639+0000) 2020-02-05T22:07:02.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:02.637+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.638946+0000) 2020-02-05T22:07:02.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:02.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.639523+0000) 2020-02-05T22:07:02.641 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:02.638+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.639528+0000) 2020-02-05T22:07:02.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:02.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.644451+0000) 2020-02-05T22:07:02.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:02.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:02.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:02.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:02.691865+0000) 2020-02-05T22:07:03.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:03.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:03.808650+0000) 2020-02-05T22:07:03.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:03.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:03.808792+0000) 2020-02-05T22:07:05.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:05.904+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:05.904576+0000) 2020-02-05T22:07:05.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:05.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:05.910878+0000) 2020-02-05T22:07:07.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:07.050+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.051274+0000) 2020-02-05T22:07:07.053 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:07.050+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.051275+0000) 2020-02-05T22:07:07.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:07.651+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.651244+0000) 2020-02-05T22:07:07.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:07.651+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.651763+0000) 2020-02-05T22:07:07.653 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:07.650+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.652177+0000) 2020-02-05T22:07:07.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:07.652+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.652189+0000) 2020-02-05T22:07:07.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:07.652+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.652616+0000) 2020-02-05T22:07:07.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:07.651+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.653192+0000) 2020-02-05T22:07:07.655 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:07.652+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.653562+0000) 2020-02-05T22:07:07.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:07.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.653839+0000) 2020-02-05T22:07:07.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:07.654+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.654209+0000) 2020-02-05T22:07:07.656 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:07.653+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.654728+0000) 2020-02-05T22:07:07.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:07.653+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.655283+0000) 2020-02-05T22:07:07.657 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:07.654+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.654640+0000) 2020-02-05T22:07:07.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:07.654+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.656076+0000) 2020-02-05T22:07:07.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:07.661+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:07.661993+0000) 2020-02-05T22:07:07.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:07.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:09.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:09.154+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:09.155191+0000) 2020-02-05T22:07:09.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:09.440+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:09.441350+0000) 2020-02-05T22:07:10.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:10.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:10.048436+0000) 2020-02-05T22:07:10.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:10.283+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:10.283973+0000) 2020-02-05T22:07:10.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:10.299+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:10.300771+0000) 2020-02-05T22:07:10.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:10.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:10.301586+0000) 2020-02-05T22:07:12.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:12.294+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.294751+0000) 2020-02-05T22:07:12.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:12.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.668021+0000) 2020-02-05T22:07:12.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:12.667+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.668621+0000) 2020-02-05T22:07:12.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:12.668+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.669813+0000) 2020-02-05T22:07:12.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:12.669+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.670460+0000) 2020-02-05T22:07:12.672 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:12.669+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.671107+0000) 2020-02-05T22:07:12.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:12.670+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.671568+0000) 2020-02-05T22:07:12.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:12.670+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.672340+0000) 2020-02-05T22:07:12.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:12.671+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.672415+0000) 2020-02-05T22:07:12.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:12.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.672980+0000) 2020-02-05T22:07:12.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:12.672+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.674234+0000) 2020-02-05T22:07:12.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:12.673+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.674323+0000) 2020-02-05T22:07:12.676 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:12.674+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.675254+0000) 2020-02-05T22:07:12.676 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:12.674+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.675596+0000) 2020-02-05T22:07:12.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:12.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.680193+0000) 2020-02-05T22:07:12.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:12.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:12.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:12.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:12.692053+0000) 2020-02-05T22:07:13.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:13.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:13.808920+0000) 2020-02-05T22:07:13.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:13.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:13.809009+0000) 2020-02-05T22:07:15.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:15.903+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:15.904787+0000) 2020-02-05T22:07:15.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:15.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:15.911128+0000) 2020-02-05T22:07:17.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:17.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.051534+0000) 2020-02-05T22:07:17.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:17.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.051553+0000) 2020-02-05T22:07:17.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:17.687+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.688157+0000) 2020-02-05T22:07:17.690 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:17.687+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.689045+0000) 2020-02-05T22:07:17.690 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:17.688+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.689193+0000) 2020-02-05T22:07:17.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:17.689+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.689562+0000) 2020-02-05T22:07:17.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:17.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.689939+0000) 2020-02-05T22:07:17.691 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:17.689+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.690167+0000) 2020-02-05T22:07:17.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:17.689+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.690689+0000) 2020-02-05T22:07:17.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:17.689+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.690941+0000) 2020-02-05T22:07:17.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:17.690+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.691177+0000) 2020-02-05T22:07:17.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:17.691+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.691917+0000) 2020-02-05T22:07:17.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:17.691+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.692173+0000) 2020-02-05T22:07:17.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:17.691+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.692867+0000) 2020-02-05T22:07:17.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:17.692+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.693197+0000) 2020-02-05T22:07:17.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:17.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:17.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:17.700+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:17.701703+0000) 2020-02-05T22:07:19.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:19.162+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:19.163260+0000) 2020-02-05T22:07:19.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:19.440+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:19.441586+0000) 2020-02-05T22:07:20.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:20.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:20.048666+0000) 2020-02-05T22:07:20.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:20.283+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:20.284215+0000) 2020-02-05T22:07:20.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:20.300+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:20.301007+0000) 2020-02-05T22:07:20.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:20.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:20.301782+0000) 2020-02-05T22:07:22.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:22.294+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.295000+0000) 2020-02-05T22:07:22.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:22.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.692234+0000) 2020-02-05T22:07:22.706 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:22.704+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.704664+0000) 2020-02-05T22:07:22.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:22.704+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.705111+0000) 2020-02-05T22:07:22.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:22.705+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.706463+0000) 2020-02-05T22:07:22.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:22.706+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.706939+0000) 2020-02-05T22:07:22.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:22.705+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.706940+0000) 2020-02-05T22:07:22.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:22.706+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.707405+0000) 2020-02-05T22:07:22.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:22.707+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.708230+0000) 2020-02-05T22:07:22.709 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:22.708+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.708716+0000) 2020-02-05T22:07:22.710 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:22.708+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.709032+0000) 2020-02-05T22:07:22.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:22.709+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.709913+0000) 2020-02-05T22:07:22.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:22.708+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.710050+0000) 2020-02-05T22:07:22.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:22.710+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.710696+0000) 2020-02-05T22:07:22.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:22.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.714500+0000) 2020-02-05T22:07:22.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:22.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:22.714769+0000) 2020-02-05T22:07:22.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:22.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:23.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:23.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:23.809200+0000) 2020-02-05T22:07:23.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:23.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:23.809291+0000) 2020-02-05T22:07:24.524 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:07:24.530 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:07:25.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:25.904+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:25.905055+0000) 2020-02-05T22:07:25.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:25.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:25.911362+0000) 2020-02-05T22:07:27.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:27.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.051773+0000) 2020-02-05T22:07:27.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:27.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.051785+0000) 2020-02-05T22:07:27.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:27.721+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.722464+0000) 2020-02-05T22:07:27.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:27.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.723155+0000) 2020-02-05T22:07:27.725 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:27.723+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.724071+0000) 2020-02-05T22:07:27.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:27.724+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.724940+0000) 2020-02-05T22:07:27.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:27.723+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.724972+0000) 2020-02-05T22:07:27.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:27.724+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.725473+0000) 2020-02-05T22:07:27.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:27.725+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.726114+0000) 2020-02-05T22:07:27.727 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:27.724+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.725683+0000) 2020-02-05T22:07:27.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:27.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.726466+0000) 2020-02-05T22:07:27.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:27.726+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.727111+0000) 2020-02-05T22:07:27.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:27.727+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.727821+0000) 2020-02-05T22:07:27.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:27.727+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.728316+0000) 2020-02-05T22:07:27.730 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:27.728+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.729159+0000) 2020-02-05T22:07:27.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:27.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:27.734540+0000) 2020-02-05T22:07:27.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:27.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:29.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:29.170+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:29.170835+0000) 2020-02-05T22:07:29.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:29.441+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:29.441817+0000) 2020-02-05T22:07:30.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:30.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:30.048922+0000) 2020-02-05T22:07:30.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:30.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:30.284468+0000) 2020-02-05T22:07:30.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:30.300+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:30.301273+0000) 2020-02-05T22:07:30.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:30.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:30.302004+0000) 2020-02-05T22:07:32.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:32.294+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.295257+0000) 2020-02-05T22:07:32.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:32.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.692494+0000) 2020-02-05T22:07:32.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:32.741+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.742409+0000) 2020-02-05T22:07:32.744 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:32.741+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.742789+0000) 2020-02-05T22:07:32.745 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:32.741+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.742166+0000) 2020-02-05T22:07:32.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:32.742+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.743778+0000) 2020-02-05T22:07:32.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:32.743+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.744156+0000) 2020-02-05T22:07:32.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:32.743+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.743683+0000) 2020-02-05T22:07:32.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:32.744+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.744583+0000) 2020-02-05T22:07:32.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:32.744+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.745129+0000) 2020-02-05T22:07:32.747 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:32.744+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.745511+0000) 2020-02-05T22:07:32.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:32.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.745646+0000) 2020-02-05T22:07:32.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:32.745+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.745821+0000) 2020-02-05T22:07:32.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:32.745+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.745975+0000) 2020-02-05T22:07:32.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:32.746+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.747070+0000) 2020-02-05T22:07:32.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:32.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:32.751326+0000) 2020-02-05T22:07:32.756 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:32.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:33.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:33.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:33.809435+0000) 2020-02-05T22:07:33.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:33.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:33.809565+0000) 2020-02-05T22:07:35.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:35.904+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:35.905310+0000) 2020-02-05T22:07:35.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:35.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:35.911587+0000) 2020-02-05T22:07:37.053 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:37.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.052010+0000) 2020-02-05T22:07:37.053 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:37.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.052023+0000) 2020-02-05T22:07:37.762 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:37.760+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.761382+0000) 2020-02-05T22:07:37.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:37.760+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.761818+0000) 2020-02-05T22:07:37.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:37.761+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.762222+0000) 2020-02-05T22:07:37.764 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:37.761+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.762996+0000) 2020-02-05T22:07:37.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:37.762+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.762913+0000) 2020-02-05T22:07:37.765 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:37.763+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.763750+0000) 2020-02-05T22:07:37.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:37.762+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.763732+0000) 2020-02-05T22:07:37.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:37.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.764324+0000) 2020-02-05T22:07:37.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:37.764+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.764722+0000) 2020-02-05T22:07:37.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:37.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.765066+0000) 2020-02-05T22:07:37.767 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:37.765+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.766347+0000) 2020-02-05T22:07:37.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:37.766+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.766954+0000) 2020-02-05T22:07:37.768 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:37.766+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.766892+0000) 2020-02-05T22:07:37.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:37.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:37.770048+0000) 2020-02-05T22:07:37.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:37.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 105999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:39.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:39.178+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:39.178851+0000) 2020-02-05T22:07:39.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:39.441+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:39.442047+0000) 2020-02-05T22:07:40.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:40.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:40.049150+0000) 2020-02-05T22:07:40.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:40.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:40.284716+0000) 2020-02-05T22:07:40.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:40.301+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:40.301517+0000) 2020-02-05T22:07:40.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:40.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:40.302203+0000) 2020-02-05T22:07:42.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:42.295+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.295519+0000) 2020-02-05T22:07:42.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:42.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.692732+0000) 2020-02-05T22:07:42.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:42.777+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.778685+0000) 2020-02-05T22:07:42.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:42.780+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.781369+0000) 2020-02-05T22:07:42.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:42.782+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.783013+0000) 2020-02-05T22:07:42.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:42.783+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.783761+0000) 2020-02-05T22:07:42.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:42.784+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.784699+0000) 2020-02-05T22:07:42.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:42.778+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.779975+0000) 2020-02-05T22:07:42.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:42.779+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.780740+0000) 2020-02-05T22:07:42.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:42.780+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.781873+0000) 2020-02-05T22:07:42.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:42.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.782279+0000) 2020-02-05T22:07:42.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:42.781+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.782491+0000) 2020-02-05T22:07:42.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:42.782+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.783230+0000) 2020-02-05T22:07:42.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:42.784+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.784850+0000) 2020-02-05T22:07:42.789 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:42.784+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.785018+0000) 2020-02-05T22:07:42.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:42.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106042 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:42.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:42.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:42.792539+0000) 2020-02-05T22:07:43.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:43.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:43.809666+0000) 2020-02-05T22:07:43.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:43.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:43.809805+0000) 2020-02-05T22:07:45.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:45.905+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:45.905532+0000) 2020-02-05T22:07:45.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:45.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:45.911856+0000) 2020-02-05T22:07:47.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:47.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.052264+0000) 2020-02-05T22:07:47.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:47.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.052235+0000) 2020-02-05T22:07:47.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:47.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.799234+0000) 2020-02-05T22:07:47.801 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:47.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.800220+0000) 2020-02-05T22:07:47.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:47.799+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.800367+0000) 2020-02-05T22:07:47.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:47.799+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.800749+0000) 2020-02-05T22:07:47.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:47.799+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.801017+0000) 2020-02-05T22:07:47.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:47.800+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.801535+0000) 2020-02-05T22:07:47.803 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:47.801+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.801797+0000) 2020-02-05T22:07:47.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:47.801+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.802310+0000) 2020-02-05T22:07:47.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:47.801+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.802331+0000) 2020-02-05T22:07:47.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:47.801+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.802657+0000) 2020-02-05T22:07:47.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:47.803+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.803681+0000) 2020-02-05T22:07:47.805 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:47.804+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.804635+0000) 2020-02-05T22:07:47.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:47.803+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.803580+0000) 2020-02-05T22:07:47.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:47.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:47.808320+0000) 2020-02-05T22:07:47.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:47.811+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:49.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:49.186+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:49.187044+0000) 2020-02-05T22:07:49.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:49.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:49.442337+0000) 2020-02-05T22:07:50.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:50.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:50.049409+0000) 2020-02-05T22:07:50.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:50.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:50.284948+0000) 2020-02-05T22:07:50.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:50.300+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:50.301742+0000) 2020-02-05T22:07:50.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:50.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:50.302473+0000) 2020-02-05T22:07:52.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:52.295+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.295785+0000) 2020-02-05T22:07:52.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:52.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.692962+0000) 2020-02-05T22:07:52.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:52.815+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.816425+0000) 2020-02-05T22:07:52.819 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:52.816+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.817258+0000) 2020-02-05T22:07:52.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:52.817+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.818394+0000) 2020-02-05T22:07:52.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:52.817+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.819050+0000) 2020-02-05T22:07:52.821 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:52.819+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.819991+0000) 2020-02-05T22:07:52.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:52.819+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.820026+0000) 2020-02-05T22:07:52.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:52.819+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.820393+0000) 2020-02-05T22:07:52.822 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:52.819+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.820818+0000) 2020-02-05T22:07:52.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:52.820+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.821658+0000) 2020-02-05T22:07:52.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:52.821+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.822274+0000) 2020-02-05T22:07:52.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:52.821+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.822797+0000) 2020-02-05T22:07:52.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:52.822+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.823119+0000) 2020-02-05T22:07:52.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:52.823+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.823482+0000) 2020-02-05T22:07:52.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:52.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:52.827517+0000) 2020-02-05T22:07:52.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:52.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106126 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:53.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:53.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:53.809896+0000) 2020-02-05T22:07:53.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:53.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:53.810036+0000) 2020-02-05T22:07:54.687 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:07:54.693 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:07:55.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:55.905+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:55.905834+0000) 2020-02-05T22:07:55.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:55.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:55.912119+0000) 2020-02-05T22:07:57.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:57.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.052516+0000) 2020-02-05T22:07:57.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:57.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.052497+0000) 2020-02-05T22:07:57.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:07:57.834+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.835515+0000) 2020-02-05T22:07:57.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:57.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.836611+0000) 2020-02-05T22:07:57.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:07:57.835+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.837223+0000) 2020-02-05T22:07:57.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:07:57.836+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.837934+0000) 2020-02-05T22:07:57.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:07:57.836+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.837971+0000) 2020-02-05T22:07:57.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:07:57.837+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.838322+0000) 2020-02-05T22:07:57.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:57.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.838845+0000) 2020-02-05T22:07:57.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:07:57.838+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.839320+0000) 2020-02-05T22:07:57.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:07:57.838+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.840282+0000) 2020-02-05T22:07:57.841 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:07:57.839+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.840431+0000) 2020-02-05T22:07:57.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:07:57.840+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.841496+0000) 2020-02-05T22:07:57.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:07:57.840+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.841835+0000) 2020-02-05T22:07:57.843 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:07:57.841+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.842524+0000) 2020-02-05T22:07:57.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:57.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:57.845656+0000) 2020-02-05T22:07:57.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:07:57.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106167 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:07:59.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:59.195+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:59.195535+0000) 2020-02-05T22:07:59.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:07:59.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:07:59.442564+0000) 2020-02-05T22:08:00.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:00.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:00.049637+0000) 2020-02-05T22:08:00.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:00.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:00.285200+0000) 2020-02-05T22:08:00.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:00.301+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:00.301975+0000) 2020-02-05T22:08:00.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:00.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:00.302683+0000) 2020-02-05T22:08:02.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:02.295+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.296054+0000) 2020-02-05T22:08:02.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:02.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.693213+0000) 2020-02-05T22:08:02.857 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:02.855+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.855969+0000) 2020-02-05T22:08:02.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:02.856+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.856511+0000) 2020-02-05T22:08:02.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:02.857+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.857611+0000) 2020-02-05T22:08:02.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:02.858+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.858698+0000) 2020-02-05T22:08:02.860 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:02.857+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.858699+0000) 2020-02-05T22:08:02.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:02.858+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.859913+0000) 2020-02-05T22:08:02.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:02.859+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.859908+0000) 2020-02-05T22:08:02.862 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:02.859+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.860672+0000) 2020-02-05T22:08:02.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:02.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.861221+0000) 2020-02-05T22:08:02.862 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:02.860+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.861861+0000) 2020-02-05T22:08:02.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:02.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.861852+0000) 2020-02-05T22:08:02.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:02.860+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.862278+0000) 2020-02-05T22:08:02.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:02.861+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.862441+0000) 2020-02-05T22:08:02.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:02.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:02.864205+0000) 2020-02-05T22:08:02.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:02.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106210 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:03.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:03.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:03.810138+0000) 2020-02-05T22:08:03.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:03.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:03.810261+0000) 2020-02-05T22:08:05.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:05.905+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:05.906065+0000) 2020-02-05T22:08:05.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:05.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:05.912403+0000) 2020-02-05T22:08:07.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:07.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.052756+0000) 2020-02-05T22:08:07.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:07.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.052739+0000) 2020-02-05T22:08:07.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:07.872+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.874057+0000) 2020-02-05T22:08:07.876 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:07.873+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.875043+0000) 2020-02-05T22:08:07.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:07.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.876171+0000) 2020-02-05T22:08:07.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:07.876+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.877516+0000) 2020-02-05T22:08:07.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:07.877+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.878668+0000) 2020-02-05T22:08:07.880 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:07.876+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.878256+0000) 2020-02-05T22:08:07.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:07.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.878756+0000) 2020-02-05T22:08:07.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:07.878+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.879188+0000) 2020-02-05T22:08:07.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:07.878+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.879568+0000) 2020-02-05T22:08:07.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:07.879+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.880497+0000) 2020-02-05T22:08:07.882 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:07.879+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.880467+0000) 2020-02-05T22:08:07.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:07.878+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.880177+0000) 2020-02-05T22:08:07.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:07.880+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.881454+0000) 2020-02-05T22:08:07.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:07.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:07.886509+0000) 2020-02-05T22:08:07.891 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:07.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106250 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:09.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:09.202+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:09.203537+0000) 2020-02-05T22:08:09.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:09.441+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:09.442784+0000) 2020-02-05T22:08:10.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:10.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:10.049853+0000) 2020-02-05T22:08:10.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:10.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:10.285491+0000) 2020-02-05T22:08:10.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:10.301+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:10.302184+0000) 2020-02-05T22:08:10.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:10.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:10.302877+0000) 2020-02-05T22:08:12.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:12.295+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.296315+0000) 2020-02-05T22:08:12.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:12.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.693443+0000) 2020-02-05T22:08:12.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:12.895+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.896406+0000) 2020-02-05T22:08:12.899 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:12.896+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.898078+0000) 2020-02-05T22:08:12.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:12.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.898469+0000) 2020-02-05T22:08:12.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:12.897+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.898973+0000) 2020-02-05T22:08:12.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:12.898+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.899994+0000) 2020-02-05T22:08:12.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:12.898+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.899748+0000) 2020-02-05T22:08:12.903 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:12.898+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.900080+0000) 2020-02-05T22:08:12.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:12.899+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.900647+0000) 2020-02-05T22:08:12.904 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:12.899+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.900695+0000) 2020-02-05T22:08:12.904 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:12.900+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.902197+0000) 2020-02-05T22:08:12.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:12.899+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.900805+0000) 2020-02-05T22:08:12.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:12.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.901630+0000) 2020-02-05T22:08:12.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:12.900+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.901903+0000) 2020-02-05T22:08:12.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:12.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:12.906757+0000) 2020-02-05T22:08:12.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:12.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:13.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:13.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:13.810360+0000) 2020-02-05T22:08:13.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:13.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:13.810427+0000) 2020-02-05T22:08:15.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:15.905+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:15.906321+0000) 2020-02-05T22:08:15.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:15.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:15.912689+0000) 2020-02-05T22:08:17.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:17.051+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.053009+0000) 2020-02-05T22:08:17.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:17.051+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.052986+0000) 2020-02-05T22:08:17.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:17.912+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.913724+0000) 2020-02-05T22:08:17.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:17.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.914459+0000) 2020-02-05T22:08:17.917 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:17.913+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.915013+0000) 2020-02-05T22:08:17.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:17.914+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.915455+0000) 2020-02-05T22:08:17.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:17.915+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.916921+0000) 2020-02-05T22:08:17.918 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:17.915+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.917340+0000) 2020-02-05T22:08:17.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:17.916+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.918105+0000) 2020-02-05T22:08:17.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:17.916+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.918325+0000) 2020-02-05T22:08:17.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:17.917+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.918928+0000) 2020-02-05T22:08:17.921 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:17.918+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.920182+0000) 2020-02-05T22:08:17.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:17.919+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.920623+0000) 2020-02-05T22:08:17.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:17.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.920655+0000) 2020-02-05T22:08:17.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:17.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.920920+0000) 2020-02-05T22:08:17.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:17.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:17.925478+0000) 2020-02-05T22:08:17.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:17.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:19.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:19.211+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:19.211973+0000) 2020-02-05T22:08:19.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:19.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:19.443007+0000) 2020-02-05T22:08:20.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:20.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:20.050109+0000) 2020-02-05T22:08:20.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:20.284+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:20.285688+0000) 2020-02-05T22:08:20.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:20.301+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:20.302415+0000) 2020-02-05T22:08:20.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:20.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:20.303099+0000) 2020-02-05T22:08:22.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:22.295+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.296578+0000) 2020-02-05T22:08:22.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:22.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.693673+0000) 2020-02-05T22:08:22.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:22.931+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.933009+0000) 2020-02-05T22:08:22.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:22.932+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.933607+0000) 2020-02-05T22:08:22.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:22.932+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.933890+0000) 2020-02-05T22:08:22.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:22.933+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.934313+0000) 2020-02-05T22:08:22.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:22.934+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.935724+0000) 2020-02-05T22:08:22.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:22.935+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.936129+0000) 2020-02-05T22:08:22.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:22.935+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.936655+0000) 2020-02-05T22:08:22.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:22.935+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.937035+0000) 2020-02-05T22:08:22.940 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:22.938+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.939601+0000) 2020-02-05T22:08:22.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:22.938+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.939636+0000) 2020-02-05T22:08:22.941 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:22.938+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.940071+0000) 2020-02-05T22:08:22.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:22.938+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.940055+0000) 2020-02-05T22:08:22.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:22.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.940433+0000) 2020-02-05T22:08:22.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:22.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:22.940719+0000) 2020-02-05T22:08:22.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:22.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:23.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:23.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:23.810595+0000) 2020-02-05T22:08:23.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:23.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:23.810633+0000) 2020-02-05T22:08:24.841 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:08:24.847 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:08:25.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:25.905+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:25.906552+0000) 2020-02-05T22:08:25.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:25.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:25.912941+0000) 2020-02-05T22:08:27.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:27.052+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.053219+0000) 2020-02-05T22:08:27.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:27.052+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.053256+0000) 2020-02-05T22:08:27.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:27.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.952328+0000) 2020-02-05T22:08:27.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:27.951+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.953318+0000) 2020-02-05T22:08:27.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:27.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.953909+0000) 2020-02-05T22:08:27.955 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:27.953+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.954353+0000) 2020-02-05T22:08:27.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:27.953+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.954854+0000) 2020-02-05T22:08:27.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:27.954+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.955583+0000) 2020-02-05T22:08:27.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:27.955+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.956674+0000) 2020-02-05T22:08:27.958 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:27.955+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.956562+0000) 2020-02-05T22:08:27.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:27.956+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.957415+0000) 2020-02-05T22:08:27.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:27.956+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.957644+0000) 2020-02-05T22:08:27.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:27.957+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.958480+0000) 2020-02-05T22:08:27.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:27.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.958778+0000) 2020-02-05T22:08:27.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:27.956+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.958243+0000) 2020-02-05T22:08:27.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:27.959+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:27.960883+0000) 2020-02-05T22:08:27.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:27.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:29.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:29.220+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:29.220865+0000) 2020-02-05T22:08:29.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:29.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:29.443339+0000) 2020-02-05T22:08:30.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:30.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:30.050363+0000) 2020-02-05T22:08:30.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:30.285+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:30.285981+0000) 2020-02-05T22:08:30.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:30.302+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:30.302649+0000) 2020-02-05T22:08:30.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:30.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:30.303338+0000) 2020-02-05T22:08:32.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:32.296+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.296837+0000) 2020-02-05T22:08:32.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:32.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.693930+0000) 2020-02-05T22:08:32.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:32.970+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.971181+0000) 2020-02-05T22:08:32.973 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:32.970+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.971405+0000) 2020-02-05T22:08:32.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:32.971+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.972715+0000) 2020-02-05T22:08:32.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:32.971+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.973003+0000) 2020-02-05T22:08:32.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:32.972+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.973427+0000) 2020-02-05T22:08:32.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:32.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.974107+0000) 2020-02-05T22:08:32.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:32.973+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.974455+0000) 2020-02-05T22:08:32.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:32.973+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.975286+0000) 2020-02-05T22:08:32.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:32.975+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.976515+0000) 2020-02-05T22:08:32.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:32.974+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.976199+0000) 2020-02-05T22:08:32.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:32.976+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.977123+0000) 2020-02-05T22:08:32.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:32.976+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.977624+0000) 2020-02-05T22:08:32.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:32.976+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.978056+0000) 2020-02-05T22:08:32.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:32.981+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:32.982644+0000) 2020-02-05T22:08:32.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:32.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106462 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:33.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:33.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:33.810831+0000) 2020-02-05T22:08:33.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:33.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:33.810833+0000) 2020-02-05T22:08:35.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:35.906+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:35.906776+0000) 2020-02-05T22:08:35.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:35.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:35.913223+0000) 2020-02-05T22:08:37.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:37.052+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.053504+0000) 2020-02-05T22:08:37.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:37.052+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.053479+0000) 2020-02-05T22:08:37.991 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:37.988+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.990081+0000) 2020-02-05T22:08:37.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:37.989+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.990835+0000) 2020-02-05T22:08:37.992 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:37.990+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.991480+0000) 2020-02-05T22:08:37.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:37.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.992065+0000) 2020-02-05T22:08:37.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:37.991+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.992823+0000) 2020-02-05T22:08:37.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:37.991+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.993383+0000) 2020-02-05T22:08:37.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:37.992+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.993529+0000) 2020-02-05T22:08:37.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:37.992+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.993769+0000) 2020-02-05T22:08:37.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:37.993+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.994287+0000) 2020-02-05T22:08:37.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:37.992+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.994352+0000) 2020-02-05T22:08:37.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:37.994+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.995658+0000) 2020-02-05T22:08:37.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:37.993+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.995048+0000) 2020-02-05T22:08:37.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:37.994+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:37.995534+0000) 2020-02-05T22:08:38.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:37.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:38.000991+0000) 2020-02-05T22:08:38.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:38.000+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:39.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:39.228+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:39.229266+0000) 2020-02-05T22:08:39.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:39.443+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:39.443599+0000) 2020-02-05T22:08:40.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:40.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:40.050593+0000) 2020-02-05T22:08:40.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:40.285+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:40.286231+0000) 2020-02-05T22:08:40.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:40.302+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:40.302881+0000) 2020-02-05T22:08:40.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:40.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:40.303617+0000) 2020-02-05T22:08:42.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:42.296+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:42.297096+0000) 2020-02-05T22:08:42.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:42.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:42.694188+0000) 2020-02-05T22:08:43.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:43.007+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.008615+0000) 2020-02-05T22:08:43.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:43.008+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.009536+0000) 2020-02-05T22:08:43.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:43.009+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.010340+0000) 2020-02-05T22:08:43.012 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:43.008+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.010099+0000) 2020-02-05T22:08:43.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:43.009+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.010959+0000) 2020-02-05T22:08:43.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:43.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.011544+0000) 2020-02-05T22:08:43.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:43.010+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.011969+0000) 2020-02-05T22:08:43.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:43.011+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.012599+0000) 2020-02-05T22:08:43.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:43.011+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.012811+0000) 2020-02-05T22:08:43.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:43.011+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.013360+0000) 2020-02-05T22:08:43.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:43.012+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.013435+0000) 2020-02-05T22:08:43.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:43.012+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.013822+0000) 2020-02-05T22:08:43.015 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:43.012+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.014391+0000) 2020-02-05T22:08:43.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:43.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.019890+0000) 2020-02-05T22:08:43.022 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:43.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:43.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:43.810+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.811136+0000) 2020-02-05T22:08:43.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:43.810+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:43.811137+0000) 2020-02-05T22:08:45.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:45.906+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:45.907064+0000) 2020-02-05T22:08:45.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:45.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:45.913505+0000) 2020-02-05T22:08:47.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:47.053+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:47.053676+0000) 2020-02-05T22:08:47.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:47.053+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:47.053736+0000) 2020-02-05T22:08:48.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:48.026+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.027194+0000) 2020-02-05T22:08:48.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:48.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.027901+0000) 2020-02-05T22:08:48.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:48.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.028543+0000) 2020-02-05T22:08:48.030 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:48.028+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.029181+0000) 2020-02-05T22:08:48.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:48.028+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.029661+0000) 2020-02-05T22:08:48.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:48.030+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.030740+0000) 2020-02-05T22:08:48.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:48.030+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.031264+0000) 2020-02-05T22:08:48.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:48.030+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.030677+0000) 2020-02-05T22:08:48.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:48.031+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.032326+0000) 2020-02-05T22:08:48.034 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:48.031+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.032255+0000) 2020-02-05T22:08:48.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:48.032+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.033209+0000) 2020-02-05T22:08:48.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:48.032+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.033326+0000) 2020-02-05T22:08:48.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:48.033+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.033771+0000) 2020-02-05T22:08:48.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:48.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:48.037204+0000) 2020-02-05T22:08:48.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:48.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:49.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:49.237+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:49.237403+0000) 2020-02-05T22:08:49.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:49.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:49.443817+0000) 2020-02-05T22:08:50.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:50.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:50.050812+0000) 2020-02-05T22:08:50.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:50.286+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:50.286485+0000) 2020-02-05T22:08:50.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:50.302+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:50.303194+0000) 2020-02-05T22:08:50.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:50.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:50.303845+0000) 2020-02-05T22:08:52.298 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:52.296+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:52.297363+0000) 2020-02-05T22:08:52.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:52.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:52.694381+0000) 2020-02-05T22:08:53.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:53.044+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.045773+0000) 2020-02-05T22:08:53.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:53.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.046617+0000) 2020-02-05T22:08:53.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:53.045+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.046712+0000) 2020-02-05T22:08:53.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:53.046+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.047288+0000) 2020-02-05T22:08:53.049 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:53.046+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.047864+0000) 2020-02-05T22:08:53.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:53.048+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.049208+0000) 2020-02-05T22:08:53.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:53.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.049563+0000) 2020-02-05T22:08:53.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:53.049+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.050085+0000) 2020-02-05T22:08:53.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:53.049+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.050271+0000) 2020-02-05T22:08:53.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:53.049+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.051049+0000) 2020-02-05T22:08:53.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:53.050+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.050834+0000) 2020-02-05T22:08:53.053 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:53.051+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.051736+0000) 2020-02-05T22:08:53.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:53.052+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.052618+0000) 2020-02-05T22:08:53.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:53.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.055433+0000) 2020-02-05T22:08:53.059 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:53.057+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106630 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:53.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:53.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.811392+0000) 2020-02-05T22:08:53.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:53.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:53.811394+0000) 2020-02-05T22:08:55.014 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:08:55.020 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:08:55.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:55.907+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:55.907355+0000) 2020-02-05T22:08:55.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:55.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:55.913805+0000) 2020-02-05T22:08:57.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:57.053+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:57.053930+0000) 2020-02-05T22:08:57.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:57.053+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:57.053982+0000) 2020-02-05T22:08:58.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:58.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.063055+0000) 2020-02-05T22:08:58.065 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:08:58.063+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.063959+0000) 2020-02-05T22:08:58.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:08:58.065+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.065897+0000) 2020-02-05T22:08:58.068 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:08:58.066+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.066134+0000) 2020-02-05T22:08:58.068 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:08:58.066+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.066807+0000) 2020-02-05T22:08:58.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:08:58.067+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.067763+0000) 2020-02-05T22:08:58.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:08:58.068+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.068063+0000) 2020-02-05T22:08:58.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:08:58.068+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.068237+0000) 2020-02-05T22:08:58.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:08:58.068+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.068833+0000) 2020-02-05T22:08:58.071 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:08:58.068+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.068603+0000) 2020-02-05T22:08:58.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:08:58.068+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.068604+0000) 2020-02-05T22:08:58.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:08:58.068+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.069104+0000) 2020-02-05T22:08:58.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:58.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.073154+0000) 2020-02-05T22:08:58.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:58.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:58.073502+0000) 2020-02-05T22:08:58.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:08:58.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106671 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:08:59.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:59.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:59.245774+0000) 2020-02-05T22:08:59.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:08:59.442+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:08:59.444048+0000) 2020-02-05T22:09:00.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:00.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:00.051071+0000) 2020-02-05T22:09:00.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:00.286+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:00.286683+0000) 2020-02-05T22:09:00.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:00.302+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:00.303416+0000) 2020-02-05T22:09:00.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:00.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:00.304083+0000) 2020-02-05T22:09:02.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:02.297+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:02.297658+0000) 2020-02-05T22:09:02.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:02.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:02.694615+0000) 2020-02-05T22:09:03.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:03.081+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.082081+0000) 2020-02-05T22:09:03.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:03.081+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.082429+0000) 2020-02-05T22:09:03.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:03.082+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.083178+0000) 2020-02-05T22:09:03.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:03.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.083592+0000) 2020-02-05T22:09:03.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:03.082+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.083903+0000) 2020-02-05T22:09:03.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:03.083+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.085009+0000) 2020-02-05T22:09:03.086 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:03.084+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.085769+0000) 2020-02-05T22:09:03.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:03.085+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.086305+0000) 2020-02-05T22:09:03.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:03.086+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.086501+0000) 2020-02-05T22:09:03.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:03.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.086446+0000) 2020-02-05T22:09:03.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:03.086+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.086852+0000) 2020-02-05T22:09:03.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:03.087+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.087805+0000) 2020-02-05T22:09:03.089 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:03.087+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.088048+0000) 2020-02-05T22:09:03.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:03.089+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.091052+0000) 2020-02-05T22:09:03.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:03.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106714 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:03.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:03.810+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.811639+0000) 2020-02-05T22:09:03.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:03.810+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:03.811640+0000) 2020-02-05T22:09:05.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:05.906+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:05.907577+0000) 2020-02-05T22:09:05.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:05.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:05.914067+0000) 2020-02-05T22:09:07.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:07.053+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:07.054162+0000) 2020-02-05T22:09:07.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:07.053+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:07.054265+0000) 2020-02-05T22:09:08.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:08.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.099472+0000) 2020-02-05T22:09:08.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:08.099+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.100507+0000) 2020-02-05T22:09:08.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:08.100+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.101442+0000) 2020-02-05T22:09:08.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:08.102+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.102478+0000) 2020-02-05T22:09:08.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:08.102+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.102512+0000) 2020-02-05T22:09:08.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:08.102+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.103066+0000) 2020-02-05T22:09:08.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:08.102+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.103259+0000) 2020-02-05T22:09:08.108 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:08.103+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.103601+0000) 2020-02-05T22:09:08.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:08.104+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.104602+0000) 2020-02-05T22:09:08.108 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:08.100+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.101926+0000) 2020-02-05T22:09:08.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:08.101+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.102066+0000) 2020-02-05T22:09:08.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:08.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.103944+0000) 2020-02-05T22:09:08.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:08.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.104307+0000) 2020-02-05T22:09:08.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:08.108+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:08.109507+0000) 2020-02-05T22:09:08.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:08.112+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:09.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:09.251+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:09.252720+0000) 2020-02-05T22:09:09.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:09.443+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:09.444348+0000) 2020-02-05T22:09:10.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:10.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:10.051331+0000) 2020-02-05T22:09:10.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:10.286+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:10.286927+0000) 2020-02-05T22:09:10.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:10.302+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:10.303672+0000) 2020-02-05T22:09:10.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:10.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:10.304323+0000) 2020-02-05T22:09:12.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:12.297+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:12.297906+0000) 2020-02-05T22:09:12.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:12.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:12.694852+0000) 2020-02-05T22:09:13.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:13.117+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.118271+0000) 2020-02-05T22:09:13.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:13.118+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.119002+0000) 2020-02-05T22:09:13.120 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:13.118+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.119497+0000) 2020-02-05T22:09:13.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:13.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.119730+0000) 2020-02-05T22:09:13.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:13.119+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.120184+0000) 2020-02-05T22:09:13.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:13.119+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.120303+0000) 2020-02-05T22:09:13.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:13.120+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.120926+0000) 2020-02-05T22:09:13.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:13.120+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.121197+0000) 2020-02-05T22:09:13.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:13.121+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.122190+0000) 2020-02-05T22:09:13.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:13.122+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.122573+0000) 2020-02-05T22:09:13.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:13.121+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.123050+0000) 2020-02-05T22:09:13.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:13.122+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.123009+0000) 2020-02-05T22:09:13.125 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:13.122+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.123389+0000) 2020-02-05T22:09:13.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:13.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.127718+0000) 2020-02-05T22:09:13.132 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:13.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:13.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:13.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.811894+0000) 2020-02-05T22:09:13.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:13.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:13.811892+0000) 2020-02-05T22:09:15.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:15.907+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:15.907854+0000) 2020-02-05T22:09:15.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:15.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:15.914350+0000) 2020-02-05T22:09:17.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:17.054+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:17.054394+0000) 2020-02-05T22:09:17.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:17.054+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:17.054504+0000) 2020-02-05T22:09:18.137 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:18.135+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.136505+0000) 2020-02-05T22:09:18.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:18.136+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.137104+0000) 2020-02-05T22:09:18.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:18.136+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.137977+0000) 2020-02-05T22:09:18.139 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:18.137+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.138332+0000) 2020-02-05T22:09:18.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:18.138+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.138684+0000) 2020-02-05T22:09:18.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:18.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.138903+0000) 2020-02-05T22:09:18.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:18.138+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.139572+0000) 2020-02-05T22:09:18.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:18.138+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.139993+0000) 2020-02-05T22:09:18.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:18.139+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.140145+0000) 2020-02-05T22:09:18.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:18.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.140461+0000) 2020-02-05T22:09:18.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:18.140+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.141025+0000) 2020-02-05T22:09:18.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:18.141+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.141804+0000) 2020-02-05T22:09:18.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:18.141+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.142254+0000) 2020-02-05T22:09:18.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:18.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:18.144904+0000) 2020-02-05T22:09:18.150 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:18.149+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:19.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:19.260+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:19.261109+0000) 2020-02-05T22:09:19.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:19.443+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:19.444585+0000) 2020-02-05T22:09:20.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:20.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:20.051565+0000) 2020-02-05T22:09:20.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:20.287+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:20.287229+0000) 2020-02-05T22:09:20.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:20.303+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:20.304111+0000) 2020-02-05T22:09:20.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:20.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:20.304517+0000) 2020-02-05T22:09:22.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:22.297+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:22.298149+0000) 2020-02-05T22:09:22.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:22.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:22.695122+0000) 2020-02-05T22:09:23.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:23.154+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.155828+0000) 2020-02-05T22:09:23.157 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:23.155+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.156444+0000) 2020-02-05T22:09:23.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:23.156+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.157515+0000) 2020-02-05T22:09:23.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:23.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.157740+0000) 2020-02-05T22:09:23.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:23.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.158288+0000) 2020-02-05T22:09:23.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:23.157+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.158328+0000) 2020-02-05T22:09:23.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:23.158+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.159135+0000) 2020-02-05T22:09:23.160 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:23.158+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.159369+0000) 2020-02-05T22:09:23.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:23.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.159476+0000) 2020-02-05T22:09:23.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:23.160+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.160421+0000) 2020-02-05T22:09:23.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:23.160+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.160618+0000) 2020-02-05T22:09:23.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:23.160+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.160804+0000) 2020-02-05T22:09:23.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:23.160+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.160666+0000) 2020-02-05T22:09:23.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:23.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.165575+0000) 2020-02-05T22:09:23.170 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:23.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:23.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:23.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.812133+0000) 2020-02-05T22:09:23.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:23.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:23.812133+0000) 2020-02-05T22:09:25.176 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:09:25.182 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:09:25.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:25.907+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:25.908142+0000) 2020-02-05T22:09:25.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:25.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:25.914630+0000) 2020-02-05T22:09:27.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:27.053+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:27.054608+0000) 2020-02-05T22:09:27.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:27.053+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:27.054745+0000) 2020-02-05T22:09:28.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:28.175+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.176568+0000) 2020-02-05T22:09:28.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:28.176+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.177442+0000) 2020-02-05T22:09:28.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:28.176+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.177836+0000) 2020-02-05T22:09:28.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:28.176+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178014+0000) 2020-02-05T22:09:28.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:28.177+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178280+0000) 2020-02-05T22:09:28.180 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:28.177+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178711+0000) 2020-02-05T22:09:28.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:28.177+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178931+0000) 2020-02-05T22:09:28.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:28.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178593+0000) 2020-02-05T22:09:28.181 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:28.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.178667+0000) 2020-02-05T22:09:28.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:28.178+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.179243+0000) 2020-02-05T22:09:28.182 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:28.178+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.179652+0000) 2020-02-05T22:09:28.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:28.178+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.179659+0000) 2020-02-05T22:09:28.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:28.178+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.179936+0000) 2020-02-05T22:09:28.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:28.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:28.184907+0000) 2020-02-05T22:09:28.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:28.184+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:29.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:29.268+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:29.269411+0000) 2020-02-05T22:09:29.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:29.443+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:29.444828+0000) 2020-02-05T22:09:30.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:30.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:30.051849+0000) 2020-02-05T22:09:30.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:30.286+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:30.287503+0000) 2020-02-05T22:09:30.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:30.303+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:30.304363+0000) 2020-02-05T22:09:30.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:30.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:30.304780+0000) 2020-02-05T22:09:32.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:32.297+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:32.298404+0000) 2020-02-05T22:09:32.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:32.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:32.695380+0000) 2020-02-05T22:09:33.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:33.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.192207+0000) 2020-02-05T22:09:33.194 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:33.190+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.192186+0000) 2020-02-05T22:09:33.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:33.191+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.192938+0000) 2020-02-05T22:09:33.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:33.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.193478+0000) 2020-02-05T22:09:33.195 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:33.192+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.193922+0000) 2020-02-05T22:09:33.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:33.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.194943+0000) 2020-02-05T22:09:33.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:33.193+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.195410+0000) 2020-02-05T22:09:33.197 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:33.194+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.195811+0000) 2020-02-05T22:09:33.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:33.195+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.196977+0000) 2020-02-05T22:09:33.198 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:33.195+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.197059+0000) 2020-02-05T22:09:33.199 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:33.196+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.197474+0000) 2020-02-05T22:09:33.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:33.196+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.197916+0000) 2020-02-05T22:09:33.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:33.196+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.198177+0000) 2020-02-05T22:09:33.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:33.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.202828+0000) 2020-02-05T22:09:33.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:33.203+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 106969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:33.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:33.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.812341+0000) 2020-02-05T22:09:33.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:33.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:33.812341+0000) 2020-02-05T22:09:35.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:35.907+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:35.908391+0000) 2020-02-05T22:09:35.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:35.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:35.914873+0000) 2020-02-05T22:09:37.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:37.053+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:37.054868+0000) 2020-02-05T22:09:37.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:37.053+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:37.055008+0000) 2020-02-05T22:09:38.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:38.208+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.210019+0000) 2020-02-05T22:09:38.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:38.209+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.210913+0000) 2020-02-05T22:09:38.213 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:38.211+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.212484+0000) 2020-02-05T22:09:38.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:38.210+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.212295+0000) 2020-02-05T22:09:38.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:38.211+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.212646+0000) 2020-02-05T22:09:38.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:38.212+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.213218+0000) 2020-02-05T22:09:38.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:38.212+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.213483+0000) 2020-02-05T22:09:38.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:38.212+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.213794+0000) 2020-02-05T22:09:38.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:38.212+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.213983+0000) 2020-02-05T22:09:38.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:38.213+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.214736+0000) 2020-02-05T22:09:38.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:38.214+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.216203+0000) 2020-02-05T22:09:38.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:38.215+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.216894+0000) 2020-02-05T22:09:38.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:38.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.216798+0000) 2020-02-05T22:09:38.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:38.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:38.221041+0000) 2020-02-05T22:09:38.225 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:38.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:39.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:39.279+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:39.279607+0000) 2020-02-05T22:09:39.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:39.444+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:39.445059+0000) 2020-02-05T22:09:40.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:40.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:40.052112+0000) 2020-02-05T22:09:40.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:40.286+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:40.287757+0000) 2020-02-05T22:09:40.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:40.304+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:40.304673+0000) 2020-02-05T22:09:40.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:40.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:40.305018+0000) 2020-02-05T22:09:42.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:42.297+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:42.298709+0000) 2020-02-05T22:09:42.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:42.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:42.695606+0000) 2020-02-05T22:09:43.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:43.229+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.230745+0000) 2020-02-05T22:09:43.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:43.229+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.230586+0000) 2020-02-05T22:09:43.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:43.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.232182+0000) 2020-02-05T22:09:43.234 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:43.231+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.232712+0000) 2020-02-05T22:09:43.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:43.232+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.233921+0000) 2020-02-05T22:09:43.235 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:43.233+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.234131+0000) 2020-02-05T22:09:43.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:43.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.234069+0000) 2020-02-05T22:09:43.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:43.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.234492+0000) 2020-02-05T22:09:43.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:43.233+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.234540+0000) 2020-02-05T22:09:43.237 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:43.234+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.235127+0000) 2020-02-05T22:09:43.237 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:43.234+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.235167+0000) 2020-02-05T22:09:43.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:43.234+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.235791+0000) 2020-02-05T22:09:43.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:43.234+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.235888+0000) 2020-02-05T22:09:43.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:43.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.240913+0000) 2020-02-05T22:09:43.245 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:43.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:43.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:43.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.812588+0000) 2020-02-05T22:09:43.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:43.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:43.812587+0000) 2020-02-05T22:09:45.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:45.908+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:45.908622+0000) 2020-02-05T22:09:45.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:45.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:45.915143+0000) 2020-02-05T22:09:47.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:47.054+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:47.055133+0000) 2020-02-05T22:09:47.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:47.054+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:47.055291+0000) 2020-02-05T22:09:48.251 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:48.248+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.250047+0000) 2020-02-05T22:09:48.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:48.249+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.251224+0000) 2020-02-05T22:09:48.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:48.250+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.251923+0000) 2020-02-05T22:09:48.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:48.250+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.252283+0000) 2020-02-05T22:09:48.254 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:48.251+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.252555+0000) 2020-02-05T22:09:48.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:48.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.253373+0000) 2020-02-05T22:09:48.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:48.252+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.253475+0000) 2020-02-05T22:09:48.255 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:48.252+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.253800+0000) 2020-02-05T22:09:48.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:48.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.254308+0000) 2020-02-05T22:09:48.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:48.253+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.254985+0000) 2020-02-05T22:09:48.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:48.253+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.255147+0000) 2020-02-05T22:09:48.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:48.253+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.255400+0000) 2020-02-05T22:09:48.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:48.255+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.256186+0000) 2020-02-05T22:09:48.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:48.256+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:48.257689+0000) 2020-02-05T22:09:48.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:48.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:49.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:49.291+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:49.291310+0000) 2020-02-05T22:09:49.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:49.445+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:49.445317+0000) 2020-02-05T22:09:50.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:50.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:50.052374+0000) 2020-02-05T22:09:50.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:50.287+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:50.288013+0000) 2020-02-05T22:09:50.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:50.304+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:50.304908+0000) 2020-02-05T22:09:50.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:50.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:50.305258+0000) 2020-02-05T22:09:52.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:52.298+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:52.298963+0000) 2020-02-05T22:09:52.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:52.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:52.695842+0000) 2020-02-05T22:09:53.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:53.267+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.267116+0000) 2020-02-05T22:09:53.269 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:53.268+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.268249+0000) 2020-02-05T22:09:53.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:53.267+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.268652+0000) 2020-02-05T22:09:53.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:53.269+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.269967+0000) 2020-02-05T22:09:53.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:53.269+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.270655+0000) 2020-02-05T22:09:53.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:53.271+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.271071+0000) 2020-02-05T22:09:53.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:53.270+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.271823+0000) 2020-02-05T22:09:53.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:53.272+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.272132+0000) 2020-02-05T22:09:53.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:53.271+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.273165+0000) 2020-02-05T22:09:53.274 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:53.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.273396+0000) 2020-02-05T22:09:53.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:53.272+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.273491+0000) 2020-02-05T22:09:53.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:53.272+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.273584+0000) 2020-02-05T22:09:53.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:53.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.277505+0000) 2020-02-05T22:09:53.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:53.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.277771+0000) 2020-02-05T22:09:53.282 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:53.279+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:53.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:53.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.812844+0000) 2020-02-05T22:09:53.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:53.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:53.812844+0000) 2020-02-05T22:09:55.357 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:09:55.363 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:09:55.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:55.908+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:55.908853+0000) 2020-02-05T22:09:55.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:55.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:55.915394+0000) 2020-02-05T22:09:57.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:57.054+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:57.055458+0000) 2020-02-05T22:09:57.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:57.054+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:57.055521+0000) 2020-02-05T22:09:58.287 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:09:58.285+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.286335+0000) 2020-02-05T22:09:58.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:09:58.287+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.288211+0000) 2020-02-05T22:09:58.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:58.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.288861+0000) 2020-02-05T22:09:58.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:09:58.288+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.289545+0000) 2020-02-05T22:09:58.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:09:58.288+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.289969+0000) 2020-02-05T22:09:58.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:09:58.288+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.289928+0000) 2020-02-05T22:09:58.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:58.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.291251+0000) 2020-02-05T22:09:58.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:09:58.290+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.291588+0000) 2020-02-05T22:09:58.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:09:58.289+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.291255+0000) 2020-02-05T22:09:58.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:09:58.290+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.292158+0000) 2020-02-05T22:09:58.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:09:58.290+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.292318+0000) 2020-02-05T22:09:58.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:09:58.291+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.292562+0000) 2020-02-05T22:09:58.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:58.293+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.294733+0000) 2020-02-05T22:09:58.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:09:58.291+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:58.293216+0000) 2020-02-05T22:09:58.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:09:58.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:09:59.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:59.298+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:59.299994+0000) 2020-02-05T22:09:59.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:09:59.444+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:09:59.445543+0000) 2020-02-05T22:10:00.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:00.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:00.052603+0000) 2020-02-05T22:10:00.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:00.287+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:00.288269+0000) 2020-02-05T22:10:00.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:00.304+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:00.305171+0000) 2020-02-05T22:10:00.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:00.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:00.305533+0000) 2020-02-05T22:10:02.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:02.298+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:02.299199+0000) 2020-02-05T22:10:02.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:02.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:02.696040+0000) 2020-02-05T22:10:03.307 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:03.304+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.305889+0000) 2020-02-05T22:10:03.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:03.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.306508+0000) 2020-02-05T22:10:03.308 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:03.306+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.307280+0000) 2020-02-05T22:10:03.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:03.306+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.307699+0000) 2020-02-05T22:10:03.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:03.307+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.308603+0000) 2020-02-05T22:10:03.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:03.307+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.309255+0000) 2020-02-05T22:10:03.311 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:03.309+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.310178+0000) 2020-02-05T22:10:03.313 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:03.310+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.311264+0000) 2020-02-05T22:10:03.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:03.310+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.311690+0000) 2020-02-05T22:10:03.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:03.310+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.312244+0000) 2020-02-05T22:10:03.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:03.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.312948+0000) 2020-02-05T22:10:03.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:03.311+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.313109+0000) 2020-02-05T22:10:03.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:03.314+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.316013+0000) 2020-02-05T22:10:03.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:03.315+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.316248+0000) 2020-02-05T22:10:03.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:03.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:03.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:03.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.813126+0000) 2020-02-05T22:10:03.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:03.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:03.813126+0000) 2020-02-05T22:10:05.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:05.908+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:05.909110+0000) 2020-02-05T22:10:05.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:05.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:05.915742+0000) 2020-02-05T22:10:07.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:07.055+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:07.055762+0000) 2020-02-05T22:10:07.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:07.055+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:07.055740+0000) 2020-02-05T22:10:08.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:08.326+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.327149+0000) 2020-02-05T22:10:08.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:08.326+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.327444+0000) 2020-02-05T22:10:08.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:08.327+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.328131+0000) 2020-02-05T22:10:08.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:08.327+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.328427+0000) 2020-02-05T22:10:08.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:08.327+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.329256+0000) 2020-02-05T22:10:08.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:08.327+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.328927+0000) 2020-02-05T22:10:08.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:08.328+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.329903+0000) 2020-02-05T22:10:08.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:08.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.330032+0000) 2020-02-05T22:10:08.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:08.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.330447+0000) 2020-02-05T22:10:08.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:08.329+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.331408+0000) 2020-02-05T22:10:08.333 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:08.330+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.331586+0000) 2020-02-05T22:10:08.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:08.330+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.332203+0000) 2020-02-05T22:10:08.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:08.331+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.332734+0000) 2020-02-05T22:10:08.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:08.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:08.340015+0000) 2020-02-05T22:10:08.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:08.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107259 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:09.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:09.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:09.308369+0000) 2020-02-05T22:10:09.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:09.444+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:09.445805+0000) 2020-02-05T22:10:10.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:10.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:10.052829+0000) 2020-02-05T22:10:10.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:10.287+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:10.288534+0000) 2020-02-05T22:10:10.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:10.304+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:10.305394+0000) 2020-02-05T22:10:10.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:10.305+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:10.305787+0000) 2020-02-05T22:10:12.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:12.298+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:12.299502+0000) 2020-02-05T22:10:12.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:12.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:12.696271+0000) 2020-02-05T22:10:13.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:13.344+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.345461+0000) 2020-02-05T22:10:13.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:13.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.346212+0000) 2020-02-05T22:10:13.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:13.346+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.347302+0000) 2020-02-05T22:10:13.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:13.346+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.347877+0000) 2020-02-05T22:10:13.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:13.347+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.348473+0000) 2020-02-05T22:10:13.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:13.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.348840+0000) 2020-02-05T22:10:13.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:13.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.349131+0000) 2020-02-05T22:10:13.350 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:13.348+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.349577+0000) 2020-02-05T22:10:13.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:13.348+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.350100+0000) 2020-02-05T22:10:13.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:13.349+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.350505+0000) 2020-02-05T22:10:13.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:13.350+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.351763+0000) 2020-02-05T22:10:13.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:13.350+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.351871+0000) 2020-02-05T22:10:13.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:13.351+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.352461+0000) 2020-02-05T22:10:13.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:13.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:13.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:13.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.357138+0000) 2020-02-05T22:10:13.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:13.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.813385+0000) 2020-02-05T22:10:13.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:13.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:13.813385+0000) 2020-02-05T22:10:15.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:15.908+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:15.909350+0000) 2020-02-05T22:10:15.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:15.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:15.915993+0000) 2020-02-05T22:10:17.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:17.055+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:17.055995+0000) 2020-02-05T22:10:17.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:17.055+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:17.055981+0000) 2020-02-05T22:10:18.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:18.363+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.364644+0000) 2020-02-05T22:10:18.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:18.364+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.365153+0000) 2020-02-05T22:10:18.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:18.366+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.366639+0000) 2020-02-05T22:10:18.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:18.366+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.366794+0000) 2020-02-05T22:10:18.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:18.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.367508+0000) 2020-02-05T22:10:18.369 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:18.366+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.367826+0000) 2020-02-05T22:10:18.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:18.367+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.368339+0000) 2020-02-05T22:10:18.370 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:18.367+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.368500+0000) 2020-02-05T22:10:18.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:18.367+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.368833+0000) 2020-02-05T22:10:18.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:18.368+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.369132+0000) 2020-02-05T22:10:18.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:18.369+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.369543+0000) 2020-02-05T22:10:18.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:18.369+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.370111+0000) 2020-02-05T22:10:18.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:18.369+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.370345+0000) 2020-02-05T22:10:18.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:18.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:18.374280+0000) 2020-02-05T22:10:18.377 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:18.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:19.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:19.316+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:19.316778+0000) 2020-02-05T22:10:19.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:19.445+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:19.446056+0000) 2020-02-05T22:10:20.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:20.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:20.053106+0000) 2020-02-05T22:10:20.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:20.288+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:20.288789+0000) 2020-02-05T22:10:20.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:20.305+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:20.305742+0000) 2020-02-05T22:10:20.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:20.305+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:20.306045+0000) 2020-02-05T22:10:22.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:22.299+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:22.299870+0000) 2020-02-05T22:10:22.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:22.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:22.696487+0000) 2020-02-05T22:10:23.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:23.380+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.381897+0000) 2020-02-05T22:10:23.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:23.382+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.382796+0000) 2020-02-05T22:10:23.384 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:23.382+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.383166+0000) 2020-02-05T22:10:23.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:23.383+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.383922+0000) 2020-02-05T22:10:23.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:23.384+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.385256+0000) 2020-02-05T22:10:23.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:23.384+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.385548+0000) 2020-02-05T22:10:23.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:23.384+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.385770+0000) 2020-02-05T22:10:23.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:23.385+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.386176+0000) 2020-02-05T22:10:23.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:23.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.386560+0000) 2020-02-05T22:10:23.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:23.386+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.387005+0000) 2020-02-05T22:10:23.388 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:23.387+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.387491+0000) 2020-02-05T22:10:23.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:23.386+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.387176+0000) 2020-02-05T22:10:23.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:23.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:23.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:23.388+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.390055+0000) 2020-02-05T22:10:23.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:23.389+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.390327+0000) 2020-02-05T22:10:23.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:23.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.813664+0000) 2020-02-05T22:10:23.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:23.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:23.813666+0000) 2020-02-05T22:10:25.545 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:10:25.551 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:10:25.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:25.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:25.909579+0000) 2020-02-05T22:10:25.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:25.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:25.916234+0000) 2020-02-05T22:10:27.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:27.055+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:27.056279+0000) 2020-02-05T22:10:27.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:27.055+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:27.056236+0000) 2020-02-05T22:10:28.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:28.400+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.400741+0000) 2020-02-05T22:10:28.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:28.400+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.401259+0000) 2020-02-05T22:10:28.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:28.400+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.401763+0000) 2020-02-05T22:10:28.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:28.401+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.402148+0000) 2020-02-05T22:10:28.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:28.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.402473+0000) 2020-02-05T22:10:28.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:28.402+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.402764+0000) 2020-02-05T22:10:28.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:28.401+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.403001+0000) 2020-02-05T22:10:28.405 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:28.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.404065+0000) 2020-02-05T22:10:28.405 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:28.403+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.403619+0000) 2020-02-05T22:10:28.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:28.403+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.403727+0000) 2020-02-05T22:10:28.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:28.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.404975+0000) 2020-02-05T22:10:28.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:28.404+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.404671+0000) 2020-02-05T22:10:28.407 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:28.403+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.404293+0000) 2020-02-05T22:10:28.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:28.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:28.411101+0000) 2020-02-05T22:10:28.415 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:28.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:29.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:29.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:29.325138+0000) 2020-02-05T22:10:29.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:29.445+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:29.446382+0000) 2020-02-05T22:10:30.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:30.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:30.053341+0000) 2020-02-05T22:10:30.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:30.288+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:30.289009+0000) 2020-02-05T22:10:30.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:30.305+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:30.305963+0000) 2020-02-05T22:10:30.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:30.305+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:30.306316+0000) 2020-02-05T22:10:32.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:32.299+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:32.300153+0000) 2020-02-05T22:10:32.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:32.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:32.696721+0000) 2020-02-05T22:10:33.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:33.419+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.420227+0000) 2020-02-05T22:10:33.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:33.420+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.421119+0000) 2020-02-05T22:10:33.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:33.421+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.421720+0000) 2020-02-05T22:10:33.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:33.421+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.422470+0000) 2020-02-05T22:10:33.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:33.422+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.422917+0000) 2020-02-05T22:10:33.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:33.422+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.423293+0000) 2020-02-05T22:10:33.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:33.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.423724+0000) 2020-02-05T22:10:33.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:33.423+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.424250+0000) 2020-02-05T22:10:33.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:33.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.425550+0000) 2020-02-05T22:10:33.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:33.424+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.425867+0000) 2020-02-05T22:10:33.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:33.425+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.426168+0000) 2020-02-05T22:10:33.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:33.426+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.426622+0000) 2020-02-05T22:10:33.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:33.425+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.426302+0000) 2020-02-05T22:10:33.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:33.428+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.429847+0000) 2020-02-05T22:10:33.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:33.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107470 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:33.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:33.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.813905+0000) 2020-02-05T22:10:33.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:33.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:33.813906+0000) 2020-02-05T22:10:35.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:35.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:35.909809+0000) 2020-02-05T22:10:35.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:35.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:35.916476+0000) 2020-02-05T22:10:37.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:37.056+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:37.056552+0000) 2020-02-05T22:10:37.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:37.056+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:37.056535+0000) 2020-02-05T22:10:38.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:38.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.439230+0000) 2020-02-05T22:10:38.441 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:38.438+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.439859+0000) 2020-02-05T22:10:38.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:38.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.440334+0000) 2020-02-05T22:10:38.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:38.439+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.440834+0000) 2020-02-05T22:10:38.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:38.440+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.441394+0000) 2020-02-05T22:10:38.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:38.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.441762+0000) 2020-02-05T22:10:38.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:38.442+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.443227+0000) 2020-02-05T22:10:38.445 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:38.443+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.444354+0000) 2020-02-05T22:10:38.446 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:38.444+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.444847+0000) 2020-02-05T22:10:38.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:38.444+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.445379+0000) 2020-02-05T22:10:38.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:38.445+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.445867+0000) 2020-02-05T22:10:38.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:38.445+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.446161+0000) 2020-02-05T22:10:38.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:38.446+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.446604+0000) 2020-02-05T22:10:38.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:38.448+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:38.449492+0000) 2020-02-05T22:10:38.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:38.453+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107509 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:39.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:39.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:39.332867+0000) 2020-02-05T22:10:39.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:39.446+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:39.446610+0000) 2020-02-05T22:10:40.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:40.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:40.053558+0000) 2020-02-05T22:10:40.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:40.289+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:40.289251+0000) 2020-02-05T22:10:40.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:40.305+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:40.306224+0000) 2020-02-05T22:10:40.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:40.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:40.306537+0000) 2020-02-05T22:10:42.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:42.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:42.300392+0000) 2020-02-05T22:10:42.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:42.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:42.696989+0000) 2020-02-05T22:10:43.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:43.459+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.460309+0000) 2020-02-05T22:10:43.463 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:43.460+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.461395+0000) 2020-02-05T22:10:43.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:43.461+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.462138+0000) 2020-02-05T22:10:43.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:43.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.462994+0000) 2020-02-05T22:10:43.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:43.462+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.463277+0000) 2020-02-05T22:10:43.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:43.463+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.463613+0000) 2020-02-05T22:10:43.465 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:43.463+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.464203+0000) 2020-02-05T22:10:43.465 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:43.464+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.464494+0000) 2020-02-05T22:10:43.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:43.463+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.464787+0000) 2020-02-05T22:10:43.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:43.464+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.465580+0000) 2020-02-05T22:10:43.467 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:43.464+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.465663+0000) 2020-02-05T22:10:43.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:43.464+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.464901+0000) 2020-02-05T22:10:43.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:43.464+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.465079+0000) 2020-02-05T22:10:43.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:43.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.471267+0000) 2020-02-05T22:10:43.474 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:43.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:43.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:43.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.814149+0000) 2020-02-05T22:10:43.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:43.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:43.814149+0000) 2020-02-05T22:10:45.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:45.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:45.910138+0000) 2020-02-05T22:10:45.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:45.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:45.916673+0000) 2020-02-05T22:10:47.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:47.056+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:47.056725+0000) 2020-02-05T22:10:47.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:47.056+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:47.056804+0000) 2020-02-05T22:10:48.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:48.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.479544+0000) 2020-02-05T22:10:48.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:48.479+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.480229+0000) 2020-02-05T22:10:48.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:48.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.480469+0000) 2020-02-05T22:10:48.482 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:48.480+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.481340+0000) 2020-02-05T22:10:48.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:48.480+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.481734+0000) 2020-02-05T22:10:48.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:48.481+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.482224+0000) 2020-02-05T22:10:48.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:48.481+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.482413+0000) 2020-02-05T22:10:48.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:48.482+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.482474+0000) 2020-02-05T22:10:48.484 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:48.482+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.483157+0000) 2020-02-05T22:10:48.485 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:48.482+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.482835+0000) 2020-02-05T22:10:48.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:48.483+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.483358+0000) 2020-02-05T22:10:48.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:48.483+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.484220+0000) 2020-02-05T22:10:48.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:48.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.487773+0000) 2020-02-05T22:10:48.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:48.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:48.487979+0000) 2020-02-05T22:10:48.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:48.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:49.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:49.341+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:49.341850+0000) 2020-02-05T22:10:49.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:49.446+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:49.446872+0000) 2020-02-05T22:10:50.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:50.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:50.053805+0000) 2020-02-05T22:10:50.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:50.289+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:50.289502+0000) 2020-02-05T22:10:50.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:50.306+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:50.306449+0000) 2020-02-05T22:10:50.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:50.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:50.306798+0000) 2020-02-05T22:10:52.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:52.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:52.300718+0000) 2020-02-05T22:10:52.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:52.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:52.697305+0000) 2020-02-05T22:10:53.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:53.495+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.497069+0000) 2020-02-05T22:10:53.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:53.498+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.498191+0000) 2020-02-05T22:10:53.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:53.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.498788+0000) 2020-02-05T22:10:53.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:53.498+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.499060+0000) 2020-02-05T22:10:53.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:53.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.499681+0000) 2020-02-05T22:10:53.501 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:53.499+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.500418+0000) 2020-02-05T22:10:53.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:53.501+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.501432+0000) 2020-02-05T22:10:53.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:53.499+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.500830+0000) 2020-02-05T22:10:53.503 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:53.499+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.501138+0000) 2020-02-05T22:10:53.503 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:53.500+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.501502+0000) 2020-02-05T22:10:53.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:53.501+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.501812+0000) 2020-02-05T22:10:53.504 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:53.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.502846+0000) 2020-02-05T22:10:53.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:53.500+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.501878+0000) 2020-02-05T22:10:53.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:53.506+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.506919+0000) 2020-02-05T22:10:53.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:53.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107636 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:53.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:53.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.814391+0000) 2020-02-05T22:10:53.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:53.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:53.814391+0000) 2020-02-05T22:10:55.725 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:10:55.733 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:10:55.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:55.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:55.910378+0000) 2020-02-05T22:10:55.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:55.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:55.916953+0000) 2020-02-05T22:10:57.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:57.055+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:57.056949+0000) 2020-02-05T22:10:57.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:57.055+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:57.057054+0000) 2020-02-05T22:10:58.518 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:10:58.514+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.515939+0000) 2020-02-05T22:10:58.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:58.516+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.517452+0000) 2020-02-05T22:10:58.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:10:58.515+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.517175+0000) 2020-02-05T22:10:58.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:10:58.517+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.518245+0000) 2020-02-05T22:10:58.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:10:58.517+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.518361+0000) 2020-02-05T22:10:58.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:10:58.516+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.518006+0000) 2020-02-05T22:10:58.521 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:10:58.517+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.518776+0000) 2020-02-05T22:10:58.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:10:58.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.518975+0000) 2020-02-05T22:10:58.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:10:58.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.519641+0000) 2020-02-05T22:10:58.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:10:58.518+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.519806+0000) 2020-02-05T22:10:58.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:10:58.519+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.520287+0000) 2020-02-05T22:10:58.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:58.519+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.520781+0000) 2020-02-05T22:10:58.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:10:58.520+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.522149+0000) 2020-02-05T22:10:58.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:58.527+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:58.528806+0000) 2020-02-05T22:10:58.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:10:58.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:10:59.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:59.348+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:59.349980+0000) 2020-02-05T22:10:59.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:10:59.446+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:10:59.447065+0000) 2020-02-05T22:11:00.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:00.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:00.054040+0000) 2020-02-05T22:11:00.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:00.288+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:00.289750+0000) 2020-02-05T22:11:00.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:00.305+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:00.306673+0000) 2020-02-05T22:11:00.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:00.305+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:00.307020+0000) 2020-02-05T22:11:02.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:02.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:02.300979+0000) 2020-02-05T22:11:02.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:02.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:02.697536+0000) 2020-02-05T22:11:03.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:03.533+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.534961+0000) 2020-02-05T22:11:03.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:03.534+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.535418+0000) 2020-02-05T22:11:03.537 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:03.534+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.535989+0000) 2020-02-05T22:11:03.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:03.535+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.536469+0000) 2020-02-05T22:11:03.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:03.535+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.536564+0000) 2020-02-05T22:11:03.538 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:03.536+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.537320+0000) 2020-02-05T22:11:03.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:03.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.537790+0000) 2020-02-05T22:11:03.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:03.536+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.537738+0000) 2020-02-05T22:11:03.539 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:03.536+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.538179+0000) 2020-02-05T22:11:03.540 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:03.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.538340+0000) 2020-02-05T22:11:03.540 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:03.537+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.538842+0000) 2020-02-05T22:11:03.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:03.537+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.539154+0000) 2020-02-05T22:11:03.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:03.538+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.539764+0000) 2020-02-05T22:11:03.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:03.542+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.544021+0000) 2020-02-05T22:11:03.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:03.546+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:03.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:03.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.814628+0000) 2020-02-05T22:11:03.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:03.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:03.814615+0000) 2020-02-05T22:11:05.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:05.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:05.910624+0000) 2020-02-05T22:11:05.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:05.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:05.917220+0000) 2020-02-05T22:11:07.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:07.056+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:07.057208+0000) 2020-02-05T22:11:07.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:07.056+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:07.057289+0000) 2020-02-05T22:11:08.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:08.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.553929+0000) 2020-02-05T22:11:08.555 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:08.552+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.554111+0000) 2020-02-05T22:11:08.556 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:08.553+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.554811+0000) 2020-02-05T22:11:08.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:08.554+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.555502+0000) 2020-02-05T22:11:08.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:08.554+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.556335+0000) 2020-02-05T22:11:08.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:08.558+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.559499+0000) 2020-02-05T22:11:08.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:08.558+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.559547+0000) 2020-02-05T22:11:08.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:08.554+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.555805+0000) 2020-02-05T22:11:08.565 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:08.556+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.557969+0000) 2020-02-05T22:11:08.565 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:08.557+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.558419+0000) 2020-02-05T22:11:08.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:08.557+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.558642+0000) 2020-02-05T22:11:08.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:08.558+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.559293+0000) 2020-02-05T22:11:08.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:08.558+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.559333+0000) 2020-02-05T22:11:08.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:08.563+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:08.564399+0000) 2020-02-05T22:11:08.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:08.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:09.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:09.357+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:09.358206+0000) 2020-02-05T22:11:09.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:09.446+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:09.447348+0000) 2020-02-05T22:11:10.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:10.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:10.054317+0000) 2020-02-05T22:11:10.291 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:10.289+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:10.289990+0000) 2020-02-05T22:11:10.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:10.306+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:10.306898+0000) 2020-02-05T22:11:10.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:10.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:10.307246+0000) 2020-02-05T22:11:12.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:12.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:12.301222+0000) 2020-02-05T22:11:12.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:12.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:12.697778+0000) 2020-02-05T22:11:13.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:13.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.572720+0000) 2020-02-05T22:11:13.574 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:13.570+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.572346+0000) 2020-02-05T22:11:13.575 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:13.572+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.573725+0000) 2020-02-05T22:11:13.576 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:13.574+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.575513+0000) 2020-02-05T22:11:13.577 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:13.574+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.575786+0000) 2020-02-05T22:11:13.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:13.575+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.576286+0000) 2020-02-05T22:11:13.577 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:13.575+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.576603+0000) 2020-02-05T22:11:13.578 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:13.576+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.577108+0000) 2020-02-05T22:11:13.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:13.575+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.577408+0000) 2020-02-05T22:11:13.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:13.576+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.577907+0000) 2020-02-05T22:11:13.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:13.576+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.578403+0000) 2020-02-05T22:11:13.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:13.577+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.578478+0000) 2020-02-05T22:11:13.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:13.578+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.579447+0000) 2020-02-05T22:11:13.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:13.583+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:13.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:13.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.585503+0000) 2020-02-05T22:11:13.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:13.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.814823+0000) 2020-02-05T22:11:13.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:13.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:13.814855+0000) 2020-02-05T22:11:15.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:15.910+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:15.910850+0000) 2020-02-05T22:11:15.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:15.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:15.917474+0000) 2020-02-05T22:11:17.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:17.056+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:17.057473+0000) 2020-02-05T22:11:17.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:17.056+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:17.057557+0000) 2020-02-05T22:11:18.592 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:18.589+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.590964+0000) 2020-02-05T22:11:18.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:18.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.592061+0000) 2020-02-05T22:11:18.594 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:18.591+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.593142+0000) 2020-02-05T22:11:18.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:18.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.593572+0000) 2020-02-05T22:11:18.595 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:18.593+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.594573+0000) 2020-02-05T22:11:18.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:18.593+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.594766+0000) 2020-02-05T22:11:18.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:18.593+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.594992+0000) 2020-02-05T22:11:18.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:18.594+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.595558+0000) 2020-02-05T22:11:18.597 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:18.594+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.595993+0000) 2020-02-05T22:11:18.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:18.595+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.596227+0000) 2020-02-05T22:11:18.598 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:18.595+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.596534+0000) 2020-02-05T22:11:18.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:18.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.596797+0000) 2020-02-05T22:11:18.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:18.596+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.597811+0000) 2020-02-05T22:11:18.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:18.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:18.599767+0000) 2020-02-05T22:11:18.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:18.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:19.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:19.365+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:19.366365+0000) 2020-02-05T22:11:19.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:19.446+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:19.447563+0000) 2020-02-05T22:11:20.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:20.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:20.054552+0000) 2020-02-05T22:11:20.291 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:20.289+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:20.290243+0000) 2020-02-05T22:11:20.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:20.306+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:20.307118+0000) 2020-02-05T22:11:20.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:20.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:20.307501+0000) 2020-02-05T22:11:22.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:22.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:22.301494+0000) 2020-02-05T22:11:22.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:22.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:22.698011+0000) 2020-02-05T22:11:23.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:23.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.610372+0000) 2020-02-05T22:11:23.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:23.611+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.612771+0000) 2020-02-05T22:11:23.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:23.611+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.613135+0000) 2020-02-05T22:11:23.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:23.612+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.613801+0000) 2020-02-05T22:11:23.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:23.612+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.613944+0000) 2020-02-05T22:11:23.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:23.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.615022+0000) 2020-02-05T22:11:23.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:23.614+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.615197+0000) 2020-02-05T22:11:23.617 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:23.614+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.615513+0000) 2020-02-05T22:11:23.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:23.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.615772+0000) 2020-02-05T22:11:23.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:23.615+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.616477+0000) 2020-02-05T22:11:23.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:23.615+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.616289+0000) 2020-02-05T22:11:23.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:23.615+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.616442+0000) 2020-02-05T22:11:23.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:23.615+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.616814+0000) 2020-02-05T22:11:23.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:23.622+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.623663+0000) 2020-02-05T22:11:23.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:23.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:23.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:23.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.815123+0000) 2020-02-05T22:11:23.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:23.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:23.815102+0000) 2020-02-05T22:11:25.861 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:11:25.868 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:11:25.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:25.909+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:25.911056+0000) 2020-02-05T22:11:25.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:25.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:25.917682+0000) 2020-02-05T22:11:27.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:27.057+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:27.057723+0000) 2020-02-05T22:11:27.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:27.057+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:27.057786+0000) 2020-02-05T22:11:28.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:28.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.628929+0000) 2020-02-05T22:11:28.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:28.628+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.629896+0000) 2020-02-05T22:11:28.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:28.629+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.630254+0000) 2020-02-05T22:11:28.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:28.629+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.630938+0000) 2020-02-05T22:11:28.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:28.630+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.631401+0000) 2020-02-05T22:11:28.633 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:28.630+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.631609+0000) 2020-02-05T22:11:28.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:28.630+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.632268+0000) 2020-02-05T22:11:28.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:28.631+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.632971+0000) 2020-02-05T22:11:28.634 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:28.632+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.633495+0000) 2020-02-05T22:11:28.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:28.631+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.633317+0000) 2020-02-05T22:11:28.635 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:28.631+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.632938+0000) 2020-02-05T22:11:28.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:28.633+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.634888+0000) 2020-02-05T22:11:28.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:28.634+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.635405+0000) 2020-02-05T22:11:28.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:28.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:28.639794+0000) 2020-02-05T22:11:28.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:28.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:29.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:29.374+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:29.374741+0000) 2020-02-05T22:11:29.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:29.447+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:29.447758+0000) 2020-02-05T22:11:30.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:30.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:30.054781+0000) 2020-02-05T22:11:30.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:30.289+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:30.290500+0000) 2020-02-05T22:11:30.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:30.306+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:30.307350+0000) 2020-02-05T22:11:30.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:30.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:30.307731+0000) 2020-02-05T22:11:32.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:32.300+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:32.301729+0000) 2020-02-05T22:11:32.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:32.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:32.698276+0000) 2020-02-05T22:11:33.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:33.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.649931+0000) 2020-02-05T22:11:33.652 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:33.648+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.650291+0000) 2020-02-05T22:11:33.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:33.649+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.650820+0000) 2020-02-05T22:11:33.652 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:33.650+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.651479+0000) 2020-02-05T22:11:33.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:33.650+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.651890+0000) 2020-02-05T22:11:33.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:33.651+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.652237+0000) 2020-02-05T22:11:33.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:33.650+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.652194+0000) 2020-02-05T22:11:33.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:33.651+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.652692+0000) 2020-02-05T22:11:33.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:33.651+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.653042+0000) 2020-02-05T22:11:33.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:33.652+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.653747+0000) 2020-02-05T22:11:33.656 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:33.652+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.653469+0000) 2020-02-05T22:11:33.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:33.653+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.654134+0000) 2020-02-05T22:11:33.656 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:33.652+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.654317+0000) 2020-02-05T22:11:33.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:33.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.659586+0000) 2020-02-05T22:11:33.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:33.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 107968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:33.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:33.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.815359+0000) 2020-02-05T22:11:33.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:33.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:33.815359+0000) 2020-02-05T22:11:35.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:35.910+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:35.911314+0000) 2020-02-05T22:11:35.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:35.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:35.917923+0000) 2020-02-05T22:11:37.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:37.056+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:37.057949+0000) 2020-02-05T22:11:37.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:37.056+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:37.058041+0000) 2020-02-05T22:11:38.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:38.668+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.669526+0000) 2020-02-05T22:11:38.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:38.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.669820+0000) 2020-02-05T22:11:38.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:38.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.670513+0000) 2020-02-05T22:11:38.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:38.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.670563+0000) 2020-02-05T22:11:38.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:38.669+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.670950+0000) 2020-02-05T22:11:38.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:38.670+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.671234+0000) 2020-02-05T22:11:38.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:38.670+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.671734+0000) 2020-02-05T22:11:38.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:38.670+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.672129+0000) 2020-02-05T22:11:38.674 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:38.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.672689+0000) 2020-02-05T22:11:38.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:38.672+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.673777+0000) 2020-02-05T22:11:38.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:38.672+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.674234+0000) 2020-02-05T22:11:38.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:38.673+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.674697+0000) 2020-02-05T22:11:38.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:38.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.675128+0000) 2020-02-05T22:11:38.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:38.678+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:38.679938+0000) 2020-02-05T22:11:38.684 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:38.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:39.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:39.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:39.383042+0000) 2020-02-05T22:11:39.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:39.447+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:39.447966+0000) 2020-02-05T22:11:40.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:40.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:40.055036+0000) 2020-02-05T22:11:40.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:40.290+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:40.290729+0000) 2020-02-05T22:11:40.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:40.307+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:40.307621+0000) 2020-02-05T22:11:40.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:40.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:40.307972+0000) 2020-02-05T22:11:42.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:42.301+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:42.301974+0000) 2020-02-05T22:11:42.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:42.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:42.698503+0000) 2020-02-05T22:11:43.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:43.685+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.686257+0000) 2020-02-05T22:11:43.690 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:43.688+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.688712+0000) 2020-02-05T22:11:43.690 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:43.688+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.689142+0000) 2020-02-05T22:11:43.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:43.688+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.689793+0000) 2020-02-05T22:11:43.691 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:43.689+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.689616+0000) 2020-02-05T22:11:43.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:43.689+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.690598+0000) 2020-02-05T22:11:43.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:43.690+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.691299+0000) 2020-02-05T22:11:43.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:43.691+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.691675+0000) 2020-02-05T22:11:43.693 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:43.691+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.692121+0000) 2020-02-05T22:11:43.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:43.692+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.692793+0000) 2020-02-05T22:11:43.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:43.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.692845+0000) 2020-02-05T22:11:43.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:43.693+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.693502+0000) 2020-02-05T22:11:43.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:43.693+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.693955+0000) 2020-02-05T22:11:43.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:43.697+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.698587+0000) 2020-02-05T22:11:43.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:43.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:43.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:43.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.815524+0000) 2020-02-05T22:11:43.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:43.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:43.815601+0000) 2020-02-05T22:11:45.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:45.911+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:45.911542+0000) 2020-02-05T22:11:45.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:45.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:45.918175+0000) 2020-02-05T22:11:47.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:47.057+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:47.058189+0000) 2020-02-05T22:11:47.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:47.057+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:47.058285+0000) 2020-02-05T22:11:48.709 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:48.708+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.708237+0000) 2020-02-05T22:11:48.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:48.708+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.709226+0000) 2020-02-05T22:11:48.711 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:48.709+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.709955+0000) 2020-02-05T22:11:48.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:48.710+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.710376+0000) 2020-02-05T22:11:48.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:48.711+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.711199+0000) 2020-02-05T22:11:48.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:48.711+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.711519+0000) 2020-02-05T22:11:48.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:48.711+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.711698+0000) 2020-02-05T22:11:48.713 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:48.711+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.712012+0000) 2020-02-05T22:11:48.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:48.712+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.712613+0000) 2020-02-05T22:11:48.714 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:48.712+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.713018+0000) 2020-02-05T22:11:48.714 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:48.712+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.713059+0000) 2020-02-05T22:11:48.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:48.713+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.713672+0000) 2020-02-05T22:11:48.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:48.714+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.714233+0000) 2020-02-05T22:11:48.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:48.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:48.718970+0000) 2020-02-05T22:11:48.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:48.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:49.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:49.391+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:49.391307+0000) 2020-02-05T22:11:49.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:49.448+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:49.448143+0000) 2020-02-05T22:11:50.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:50.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:50.055288+0000) 2020-02-05T22:11:50.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:50.290+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:50.290987+0000) 2020-02-05T22:11:50.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:50.306+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:50.307879+0000) 2020-02-05T22:11:50.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:50.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:50.308214+0000) 2020-02-05T22:11:52.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:52.301+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:52.302251+0000) 2020-02-05T22:11:52.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:52.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:52.698736+0000) 2020-02-05T22:11:53.725 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:53.723+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.724376+0000) 2020-02-05T22:11:53.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:53.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.725565+0000) 2020-02-05T22:11:53.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:53.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.727323+0000) 2020-02-05T22:11:53.729 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:53.727+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.728062+0000) 2020-02-05T22:11:53.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:53.727+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.728289+0000) 2020-02-05T22:11:53.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:53.727+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.727992+0000) 2020-02-05T22:11:53.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:53.727+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.728640+0000) 2020-02-05T22:11:53.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:53.727+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.728987+0000) 2020-02-05T22:11:53.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:53.728+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.728945+0000) 2020-02-05T22:11:53.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:53.728+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.729394+0000) 2020-02-05T22:11:53.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:53.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.729614+0000) 2020-02-05T22:11:53.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:53.729+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.729582+0000) 2020-02-05T22:11:53.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:53.729+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.730038+0000) 2020-02-05T22:11:53.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:53.734+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.735772+0000) 2020-02-05T22:11:53.740 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:53.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:53.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:53.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.815820+0000) 2020-02-05T22:11:53.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:53.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:53.815738+0000) 2020-02-05T22:11:55.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:55.910+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:55.911779+0000) 2020-02-05T22:11:55.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:55.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:55.918424+0000) 2020-02-05T22:11:56.000 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:11:56.006 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:11:57.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:57.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:57.058474+0000) 2020-02-05T22:11:57.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:57.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:57.058537+0000) 2020-02-05T22:11:58.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:11:58.744+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.745142+0000) 2020-02-05T22:11:58.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:58.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.746245+0000) 2020-02-05T22:11:58.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:11:58.745+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.746834+0000) 2020-02-05T22:11:58.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:11:58.746+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.748065+0000) 2020-02-05T22:11:58.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:11:58.747+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.747957+0000) 2020-02-05T22:11:58.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:11:58.747+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.749056+0000) 2020-02-05T22:11:58.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:58.747+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.749048+0000) 2020-02-05T22:11:58.750 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:11:58.749+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.749666+0000) 2020-02-05T22:11:58.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:11:58.749+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.749993+0000) 2020-02-05T22:11:58.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:11:58.750+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.750573+0000) 2020-02-05T22:11:58.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:11:58.749+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.751002+0000) 2020-02-05T22:11:58.752 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:11:58.750+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.751366+0000) 2020-02-05T22:11:58.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:58.750+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.751767+0000) 2020-02-05T22:11:58.753 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:11:58.751+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:58.751557+0000) 2020-02-05T22:11:58.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:11:58.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:11:59.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:59.402+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:59.403279+0000) 2020-02-05T22:11:59.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:11:59.447+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:11:59.448349+0000) 2020-02-05T22:12:00.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:00.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:00.055512+0000) 2020-02-05T22:12:00.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:00.290+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:00.291218+0000) 2020-02-05T22:12:00.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:00.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:00.308497+0000) 2020-02-05T22:12:00.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:00.307+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:00.308130+0000) 2020-02-05T22:12:02.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:02.302+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:02.302498+0000) 2020-02-05T22:12:02.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:02.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:02.698971+0000) 2020-02-05T22:12:03.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:03.761+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.762210+0000) 2020-02-05T22:12:03.764 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:03.761+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.762700+0000) 2020-02-05T22:12:03.766 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:03.764+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.764866+0000) 2020-02-05T22:12:03.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:03.764+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.765142+0000) 2020-02-05T22:12:03.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:03.764+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.765897+0000) 2020-02-05T22:12:03.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:03.765+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.766382+0000) 2020-02-05T22:12:03.767 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:03.765+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.766477+0000) 2020-02-05T22:12:03.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:03.765+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.766884+0000) 2020-02-05T22:12:03.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:03.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.767363+0000) 2020-02-05T22:12:03.769 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:03.767+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.767954+0000) 2020-02-05T22:12:03.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:03.767+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.767969+0000) 2020-02-05T22:12:03.770 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:03.768+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.769047+0000) 2020-02-05T22:12:03.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:03.769+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.770701+0000) 2020-02-05T22:12:03.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:03.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.771879+0000) 2020-02-05T22:12:03.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:03.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:03.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:03.815+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.815988+0000) 2020-02-05T22:12:03.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:03.815+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:03.816046+0000) 2020-02-05T22:12:05.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:05.911+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:05.912005+0000) 2020-02-05T22:12:05.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:05.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:05.918735+0000) 2020-02-05T22:12:07.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:07.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:07.058716+0000) 2020-02-05T22:12:07.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:07.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:07.058834+0000) 2020-02-05T22:12:08.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:08.782+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.783220+0000) 2020-02-05T22:12:08.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:08.783+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.784111+0000) 2020-02-05T22:12:08.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:08.783+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.784151+0000) 2020-02-05T22:12:08.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:08.784+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.784482+0000) 2020-02-05T22:12:08.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:08.784+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.784682+0000) 2020-02-05T22:12:08.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:08.783+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.785025+0000) 2020-02-05T22:12:08.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:08.785+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.785632+0000) 2020-02-05T22:12:08.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:08.785+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.785497+0000) 2020-02-05T22:12:08.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:08.784+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.785960+0000) 2020-02-05T22:12:08.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:08.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.786360+0000) 2020-02-05T22:12:08.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:08.786+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.786661+0000) 2020-02-05T22:12:08.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:08.786+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.787148+0000) 2020-02-05T22:12:08.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:08.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.786933+0000) 2020-02-05T22:12:08.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:08.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:08.792659+0000) 2020-02-05T22:12:08.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:08.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:09.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:09.410+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:09.411134+0000) 2020-02-05T22:12:09.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:09.447+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:09.448571+0000) 2020-02-05T22:12:10.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:10.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:10.055775+0000) 2020-02-05T22:12:10.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:10.291+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:10.291484+0000) 2020-02-05T22:12:10.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:10.307+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:10.308371+0000) 2020-02-05T22:12:10.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:10.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:10.308731+0000) 2020-02-05T22:12:12.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:12.302+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:12.302744+0000) 2020-02-05T22:12:12.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:12.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:12.699212+0000) 2020-02-05T22:12:13.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:13.800+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.801099+0000) 2020-02-05T22:12:13.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:13.800+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.801858+0000) 2020-02-05T22:12:13.803 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:13.801+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.802517+0000) 2020-02-05T22:12:13.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:13.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.803152+0000) 2020-02-05T22:12:13.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:13.803+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.804359+0000) 2020-02-05T22:12:13.806 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:13.804+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.805298+0000) 2020-02-05T22:12:13.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:13.805+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.805925+0000) 2020-02-05T22:12:13.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:13.805+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.806237+0000) 2020-02-05T22:12:13.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:13.806+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.807445+0000) 2020-02-05T22:12:13.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:13.806+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.807576+0000) 2020-02-05T22:12:13.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:13.807+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.807720+0000) 2020-02-05T22:12:13.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:13.807+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.808156+0000) 2020-02-05T22:12:13.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:13.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.811805+0000) 2020-02-05T22:12:13.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:13.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.812054+0000) 2020-02-05T22:12:13.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:13.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:13.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:13.815+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.816266+0000) 2020-02-05T22:12:13.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:13.815+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:13.816323+0000) 2020-02-05T22:12:15.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:15.911+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:15.912302+0000) 2020-02-05T22:12:15.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:15.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:15.918966+0000) 2020-02-05T22:12:17.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:17.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:17.058931+0000) 2020-02-05T22:12:17.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:17.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:17.059105+0000) 2020-02-05T22:12:18.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:18.821+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.822524+0000) 2020-02-05T22:12:18.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:18.821+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.821933+0000) 2020-02-05T22:12:18.825 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:18.823+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.824070+0000) 2020-02-05T22:12:18.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:18.822+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.824045+0000) 2020-02-05T22:12:18.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:18.823+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.824525+0000) 2020-02-05T22:12:18.826 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:18.824+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.825417+0000) 2020-02-05T22:12:18.826 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:18.825+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.825453+0000) 2020-02-05T22:12:18.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:18.824+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.825967+0000) 2020-02-05T22:12:18.827 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:18.826+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.826419+0000) 2020-02-05T22:12:18.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:18.826+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.827411+0000) 2020-02-05T22:12:18.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:18.826+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.827279+0000) 2020-02-05T22:12:18.829 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:18.827+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.827583+0000) 2020-02-05T22:12:18.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:18.826+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.828036+0000) 2020-02-05T22:12:18.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:18.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:18.830736+0000) 2020-02-05T22:12:18.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:18.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:19.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:19.422+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:19.422689+0000) 2020-02-05T22:12:19.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:19.447+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:19.448762+0000) 2020-02-05T22:12:20.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:20.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:20.056005+0000) 2020-02-05T22:12:20.293 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:20.291+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:20.291719+0000) 2020-02-05T22:12:20.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:20.308+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:20.308596+0000) 2020-02-05T22:12:20.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:20.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:20.308990+0000) 2020-02-05T22:12:22.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:22.301+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:22.303006+0000) 2020-02-05T22:12:22.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:22.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:22.699441+0000) 2020-02-05T22:12:23.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:23.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.816452+0000) 2020-02-05T22:12:23.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:23.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.816608+0000) 2020-02-05T22:12:23.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:23.838+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.839231+0000) 2020-02-05T22:12:23.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:23.839+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.840251+0000) 2020-02-05T22:12:23.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:23.839+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.841255+0000) 2020-02-05T22:12:23.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:23.840+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.841675+0000) 2020-02-05T22:12:23.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:23.840+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.841611+0000) 2020-02-05T22:12:23.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:23.840+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842283+0000) 2020-02-05T22:12:23.844 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:23.841+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842502+0000) 2020-02-05T22:12:23.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:23.841+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842368+0000) 2020-02-05T22:12:23.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:23.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842396+0000) 2020-02-05T22:12:23.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:23.841+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842687+0000) 2020-02-05T22:12:23.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:23.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.843104+0000) 2020-02-05T22:12:23.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:23.841+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.842962+0000) 2020-02-05T22:12:23.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:23.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.844035+0000) 2020-02-05T22:12:23.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:23.847+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:23.848499+0000) 2020-02-05T22:12:23.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:23.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:25.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:25.912+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:25.912643+0000) 2020-02-05T22:12:25.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:25.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:25.919204+0000) 2020-02-05T22:12:26.152 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:12:26.159 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:12:27.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:27.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:27.059178+0000) 2020-02-05T22:12:27.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:27.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:27.059367+0000) 2020-02-05T22:12:28.857 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:28.854+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.856257+0000) 2020-02-05T22:12:28.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:28.857+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.858666+0000) 2020-02-05T22:12:28.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:28.857+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.859007+0000) 2020-02-05T22:12:28.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:28.857+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.859280+0000) 2020-02-05T22:12:28.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:28.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.859600+0000) 2020-02-05T22:12:28.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:28.858+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.859982+0000) 2020-02-05T22:12:28.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:28.858+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.859928+0000) 2020-02-05T22:12:28.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:28.859+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.860573+0000) 2020-02-05T22:12:28.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:28.859+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.860834+0000) 2020-02-05T22:12:28.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:28.860+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.861393+0000) 2020-02-05T22:12:28.863 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:28.860+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.861830+0000) 2020-02-05T22:12:28.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:28.860+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.862074+0000) 2020-02-05T22:12:28.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:28.860+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.862332+0000) 2020-02-05T22:12:28.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:28.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:28.867058+0000) 2020-02-05T22:12:28.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:28.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:29.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:29.433+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:29.433880+0000) 2020-02-05T22:12:29.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:29.448+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:29.448997+0000) 2020-02-05T22:12:30.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:30.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:30.056271+0000) 2020-02-05T22:12:30.293 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:30.290+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:30.291953+0000) 2020-02-05T22:12:30.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:30.308+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:30.308844+0000) 2020-02-05T22:12:30.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:30.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:30.309215+0000) 2020-02-05T22:12:32.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:32.302+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:32.303282+0000) 2020-02-05T22:12:32.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:32.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:32.699649+0000) 2020-02-05T22:12:33.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:33.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.816674+0000) 2020-02-05T22:12:33.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:33.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.816838+0000) 2020-02-05T22:12:33.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:33.875+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.876358+0000) 2020-02-05T22:12:33.878 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:33.875+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.877013+0000) 2020-02-05T22:12:33.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:33.876+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.877390+0000) 2020-02-05T22:12:33.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:33.877+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.878288+0000) 2020-02-05T22:12:33.879 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:33.877+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.878640+0000) 2020-02-05T22:12:33.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:33.877+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.879001+0000) 2020-02-05T22:12:33.880 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:33.878+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.879247+0000) 2020-02-05T22:12:33.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:33.878+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.879406+0000) 2020-02-05T22:12:33.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:33.878+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.879919+0000) 2020-02-05T22:12:33.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:33.879+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.880488+0000) 2020-02-05T22:12:33.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:33.878+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.880403+0000) 2020-02-05T22:12:33.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:33.878+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.880341+0000) 2020-02-05T22:12:33.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:33.878+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.879905+0000) 2020-02-05T22:12:33.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:33.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:33.885001+0000) 2020-02-05T22:12:33.890 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:33.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:35.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:35.912+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:35.912863+0000) 2020-02-05T22:12:35.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:35.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:35.919491+0000) 2020-02-05T22:12:37.061 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:37.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:37.059422+0000) 2020-02-05T22:12:37.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:37.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:37.059593+0000) 2020-02-05T22:12:38.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:38.893+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.894510+0000) 2020-02-05T22:12:38.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:38.893+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.895038+0000) 2020-02-05T22:12:38.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:38.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.895612+0000) 2020-02-05T22:12:38.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:38.895+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.896658+0000) 2020-02-05T22:12:38.898 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:38.894+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.896292+0000) 2020-02-05T22:12:38.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:38.896+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.897145+0000) 2020-02-05T22:12:38.898 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:38.896+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.897448+0000) 2020-02-05T22:12:38.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:38.897+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.898463+0000) 2020-02-05T22:12:38.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:38.897+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.899041+0000) 2020-02-05T22:12:38.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:38.898+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.899429+0000) 2020-02-05T22:12:38.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:38.898+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.899700+0000) 2020-02-05T22:12:38.901 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:38.898+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.900145+0000) 2020-02-05T22:12:38.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:38.898+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.900212+0000) 2020-02-05T22:12:38.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:38.905+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:38.906360+0000) 2020-02-05T22:12:38.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:38.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:39.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:39.441+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:39.442167+0000) 2020-02-05T22:12:39.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:39.448+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:39.449218+0000) 2020-02-05T22:12:40.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:40.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:40.056536+0000) 2020-02-05T22:12:40.293 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:40.291+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:40.292204+0000) 2020-02-05T22:12:40.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:40.308+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:40.309059+0000) 2020-02-05T22:12:40.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:40.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:40.309493+0000) 2020-02-05T22:12:42.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:42.302+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:42.303524+0000) 2020-02-05T22:12:42.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:42.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:42.699889+0000) 2020-02-05T22:12:43.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:43.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.816900+0000) 2020-02-05T22:12:43.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:43.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.817057+0000) 2020-02-05T22:12:43.914 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:43.911+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.912792+0000) 2020-02-05T22:12:43.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:43.913+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.914748+0000) 2020-02-05T22:12:43.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:43.913+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.915241+0000) 2020-02-05T22:12:43.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:43.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.915832+0000) 2020-02-05T22:12:43.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:43.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.916047+0000) 2020-02-05T22:12:43.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:43.916+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.916417+0000) 2020-02-05T22:12:43.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:43.917+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.917165+0000) 2020-02-05T22:12:43.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:43.917+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.917541+0000) 2020-02-05T22:12:43.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:43.917+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.917870+0000) 2020-02-05T22:12:43.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:43.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.918324+0000) 2020-02-05T22:12:43.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:43.916+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.918383+0000) 2020-02-05T22:12:43.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:43.917+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.918839+0000) 2020-02-05T22:12:43.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:43.917+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.919193+0000) 2020-02-05T22:12:43.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:43.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:43.923215+0000) 2020-02-05T22:12:43.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:43.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:45.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:45.912+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:45.913118+0000) 2020-02-05T22:12:45.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:45.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:45.919748+0000) 2020-02-05T22:12:47.061 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:47.058+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:47.059712+0000) 2020-02-05T22:12:47.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:47.058+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:47.059841+0000) 2020-02-05T22:12:48.934 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:48.931+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.932951+0000) 2020-02-05T22:12:48.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:48.932+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.933484+0000) 2020-02-05T22:12:48.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:48.933+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.934234+0000) 2020-02-05T22:12:48.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:48.933+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.934318+0000) 2020-02-05T22:12:48.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:48.933+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.934898+0000) 2020-02-05T22:12:48.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:48.934+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.935870+0000) 2020-02-05T22:12:48.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:48.934+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.936037+0000) 2020-02-05T22:12:48.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:48.935+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.936660+0000) 2020-02-05T22:12:48.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:48.935+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.937007+0000) 2020-02-05T22:12:48.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:48.935+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.937397+0000) 2020-02-05T22:12:48.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:48.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.938499+0000) 2020-02-05T22:12:48.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:48.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.938689+0000) 2020-02-05T22:12:48.940 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:48.937+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.939118+0000) 2020-02-05T22:12:48.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:48.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:48.941054+0000) 2020-02-05T22:12:48.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:48.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:49.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:49.448+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:49.449452+0000) 2020-02-05T22:12:49.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:49.448+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:49.449991+0000) 2020-02-05T22:12:50.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:50.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:50.056774+0000) 2020-02-05T22:12:50.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:50.291+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:50.292481+0000) 2020-02-05T22:12:50.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:50.308+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:50.309331+0000) 2020-02-05T22:12:50.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:50.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:50.309735+0000) 2020-02-05T22:12:52.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:52.302+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:52.303759+0000) 2020-02-05T22:12:52.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:52.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:52.700143+0000) 2020-02-05T22:12:53.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:53.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.817158+0000) 2020-02-05T22:12:53.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:53.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.817349+0000) 2020-02-05T22:12:53.953 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:53.950+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.951862+0000) 2020-02-05T22:12:53.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:53.951+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.952741+0000) 2020-02-05T22:12:53.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:53.951+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.953338+0000) 2020-02-05T22:12:53.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:53.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.954303+0000) 2020-02-05T22:12:53.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:53.953+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.954594+0000) 2020-02-05T22:12:53.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:53.953+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.954647+0000) 2020-02-05T22:12:53.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:53.953+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.954953+0000) 2020-02-05T22:12:53.957 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:53.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.955269+0000) 2020-02-05T22:12:53.957 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:53.954+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.955417+0000) 2020-02-05T22:12:53.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:53.955+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.956416+0000) 2020-02-05T22:12:53.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:53.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.956271+0000) 2020-02-05T22:12:53.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:53.955+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.956511+0000) 2020-02-05T22:12:53.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:53.955+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.957035+0000) 2020-02-05T22:12:53.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:53.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:53.961113+0000) 2020-02-05T22:12:53.965 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:53.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:55.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:55.912+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:55.913347+0000) 2020-02-05T22:12:55.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:55.919+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:55.919993+0000) 2020-02-05T22:12:56.290 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:12:56.296 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:12:57.061 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:57.059+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:57.060016+0000) 2020-02-05T22:12:57.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:57.059+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:57.060082+0000) 2020-02-05T22:12:58.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:12:58.970+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.971172+0000) 2020-02-05T22:12:58.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:12:58.970+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.971925+0000) 2020-02-05T22:12:58.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:58.971+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.972481+0000) 2020-02-05T22:12:58.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:58.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.972888+0000) 2020-02-05T22:12:58.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:12:58.972+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.973063+0000) 2020-02-05T22:12:58.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:12:58.972+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.973417+0000) 2020-02-05T22:12:58.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:12:58.972+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.973933+0000) 2020-02-05T22:12:58.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:12:58.973+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.974549+0000) 2020-02-05T22:12:58.976 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:12:58.973+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.975116+0000) 2020-02-05T22:12:58.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:12:58.974+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.975691+0000) 2020-02-05T22:12:58.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:12:58.974+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.976061+0000) 2020-02-05T22:12:58.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:12:58.974+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.976062+0000) 2020-02-05T22:12:58.978 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:12:58.974+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.976304+0000) 2020-02-05T22:12:58.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:58.980+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:58.981788+0000) 2020-02-05T22:12:58.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:12:58.982+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:12:59.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:59.448+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:59.449661+0000) 2020-02-05T22:12:59.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:12:59.456+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:12:59.457785+0000) 2020-02-05T22:13:00.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:00.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:00.057136+0000) 2020-02-05T22:13:00.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:00.292+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:00.292721+0000) 2020-02-05T22:13:00.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:00.308+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:00.309555+0000) 2020-02-05T22:13:00.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:00.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:00.309955+0000) 2020-02-05T22:13:02.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:02.303+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:02.304024+0000) 2020-02-05T22:13:02.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:02.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:02.700369+0000) 2020-02-05T22:13:03.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:03.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.817401+0000) 2020-02-05T22:13:03.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:03.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.817591+0000) 2020-02-05T22:13:04.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:03.990+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.991549+0000) 2020-02-05T22:13:04.363 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:03.991+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.992476+0000) 2020-02-05T22:13:04.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:03.992+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.994080+0000) 2020-02-05T22:13:04.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:03.993+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.994597+0000) 2020-02-05T22:13:04.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:03.994+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.996023+0000) 2020-02-05T22:13:04.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:03.994+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.996152+0000) 2020-02-05T22:13:04.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:04.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:04.365 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:03.989+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.990178+0000) 2020-02-05T22:13:04.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:03.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.990796+0000) 2020-02-05T22:13:04.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:03.992+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.993403+0000) 2020-02-05T22:13:04.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:03.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.999406+0000) 2020-02-05T22:13:04.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:03.990+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.991448+0000) 2020-02-05T22:13:04.367 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:03.990+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.991942+0000) 2020-02-05T22:13:04.367 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:03.994+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.995864+0000) 2020-02-05T22:13:04.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:03.995+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:03.996333+0000) 2020-02-05T22:13:05.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:05.912+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:05.913572+0000) 2020-02-05T22:13:05.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:05.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:05.920236+0000) 2020-02-05T22:13:07.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:07.059+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:07.060276+0000) 2020-02-05T22:13:07.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:07.059+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:07.060335+0000) 2020-02-05T22:13:09.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:09.007+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.008645+0000) 2020-02-05T22:13:09.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:09.009+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.010638+0000) 2020-02-05T22:13:09.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:09.010+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.011318+0000) 2020-02-05T22:13:09.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:09.010+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.011691+0000) 2020-02-05T22:13:09.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:09.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.012197+0000) 2020-02-05T22:13:09.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:09.011+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.012607+0000) 2020-02-05T22:13:09.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:09.011+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.012996+0000) 2020-02-05T22:13:09.014 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:09.012+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.013619+0000) 2020-02-05T22:13:09.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:09.012+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.013794+0000) 2020-02-05T22:13:09.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:09.013+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.014540+0000) 2020-02-05T22:13:09.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:09.013+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.014676+0000) 2020-02-05T22:13:09.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:09.013+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.014916+0000) 2020-02-05T22:13:09.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:09.013+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.014970+0000) 2020-02-05T22:13:09.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:09.014+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.015860+0000) 2020-02-05T22:13:09.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:09.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:09.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:09.449+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.449884+0000) 2020-02-05T22:13:09.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:09.465+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:09.465677+0000) 2020-02-05T22:13:10.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:10.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:10.057385+0000) 2020-02-05T22:13:10.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:10.292+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:10.292980+0000) 2020-02-05T22:13:10.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:10.309+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:10.309820+0000) 2020-02-05T22:13:10.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:10.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:10.310225+0000) 2020-02-05T22:13:12.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:12.303+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:12.304272+0000) 2020-02-05T22:13:12.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:12.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:12.700611+0000) 2020-02-05T22:13:13.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:13.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:13.817653+0000) 2020-02-05T22:13:13.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:13.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:13.817856+0000) 2020-02-05T22:13:14.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:14.027+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.028600+0000) 2020-02-05T22:13:14.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:14.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.029997+0000) 2020-02-05T22:13:14.031 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:14.029+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.030565+0000) 2020-02-05T22:13:14.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:14.030+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.031171+0000) 2020-02-05T22:13:14.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:14.031+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.031783+0000) 2020-02-05T22:13:14.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:14.032+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.032679+0000) 2020-02-05T22:13:14.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:14.032+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.033387+0000) 2020-02-05T22:13:14.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:14.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.033680+0000) 2020-02-05T22:13:14.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:14.033+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.034022+0000) 2020-02-05T22:13:14.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:14.033+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.034288+0000) 2020-02-05T22:13:14.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:14.034+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.035165+0000) 2020-02-05T22:13:14.036 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:14.034+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.035264+0000) 2020-02-05T22:13:14.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:14.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.037333+0000) 2020-02-05T22:13:14.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:14.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:14.037614+0000) 2020-02-05T22:13:14.043 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:14.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:15.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:15.913+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:15.913809+0000) 2020-02-05T22:13:15.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:15.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:15.920486+0000) 2020-02-05T22:13:17.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:17.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:17.060514+0000) 2020-02-05T22:13:17.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:17.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:17.060606+0000) 2020-02-05T22:13:19.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:19.046+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.047376+0000) 2020-02-05T22:13:19.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:19.048+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.049116+0000) 2020-02-05T22:13:19.051 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:19.048+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.049772+0000) 2020-02-05T22:13:19.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:19.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.050027+0000) 2020-02-05T22:13:19.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:19.049+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.050306+0000) 2020-02-05T22:13:19.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:19.050+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.050560+0000) 2020-02-05T22:13:19.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:19.049+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.050978+0000) 2020-02-05T22:13:19.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:19.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.051582+0000) 2020-02-05T22:13:19.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:19.050+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.051269+0000) 2020-02-05T22:13:19.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:19.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.052277+0000) 2020-02-05T22:13:19.054 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:19.053+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.053605+0000) 2020-02-05T22:13:19.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:19.051+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.052907+0000) 2020-02-05T22:13:19.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:19.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.053778+0000) 2020-02-05T22:13:19.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:19.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.057996+0000) 2020-02-05T22:13:19.061 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:19.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:19.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:19.449+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.450126+0000) 2020-02-05T22:13:19.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:19.474+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:19.475105+0000) 2020-02-05T22:13:20.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:20.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:20.057657+0000) 2020-02-05T22:13:20.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:20.292+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:20.293210+0000) 2020-02-05T22:13:20.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:20.309+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:20.310049+0000) 2020-02-05T22:13:20.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:20.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:20.310487+0000) 2020-02-05T22:13:22.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:22.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:22.304520+0000) 2020-02-05T22:13:22.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:22.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:22.700843+0000) 2020-02-05T22:13:23.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:23.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:23.817918+0000) 2020-02-05T22:13:23.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:23.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:23.818126+0000) 2020-02-05T22:13:24.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:24.063+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.064830+0000) 2020-02-05T22:13:24.068 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:24.066+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.067099+0000) 2020-02-05T22:13:24.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:24.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.067784+0000) 2020-02-05T22:13:24.069 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:24.067+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.068146+0000) 2020-02-05T22:13:24.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:24.067+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.068654+0000) 2020-02-05T22:13:24.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:24.067+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.069045+0000) 2020-02-05T22:13:24.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:24.068+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.069131+0000) 2020-02-05T22:13:24.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:24.069+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.070108+0000) 2020-02-05T22:13:24.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:24.070+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.070729+0000) 2020-02-05T22:13:24.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:24.070+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.071213+0000) 2020-02-05T22:13:24.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:24.071+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.071719+0000) 2020-02-05T22:13:24.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:24.071+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.072152+0000) 2020-02-05T22:13:24.073 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:24.071+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.072347+0000) 2020-02-05T22:13:24.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:24.074+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:24.075774+0000) 2020-02-05T22:13:24.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:24.079+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:25.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:25.913+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:25.914037+0000) 2020-02-05T22:13:25.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:25.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:25.920732+0000) 2020-02-05T22:13:26.425 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:13:26.432 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:13:27.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:27.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:27.060716+0000) 2020-02-05T22:13:27.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:27.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:27.060875+0000) 2020-02-05T22:13:29.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:29.083+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.085020+0000) 2020-02-05T22:13:29.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:29.086+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.086472+0000) 2020-02-05T22:13:29.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:29.086+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.087177+0000) 2020-02-05T22:13:29.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:29.087+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.088166+0000) 2020-02-05T22:13:29.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:29.087+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.088643+0000) 2020-02-05T22:13:29.091 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:29.088+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.089256+0000) 2020-02-05T22:13:29.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:29.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.089542+0000) 2020-02-05T22:13:29.092 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:29.088+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.090050+0000) 2020-02-05T22:13:29.092 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:29.089+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.090500+0000) 2020-02-05T22:13:29.093 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:29.090+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.090611+0000) 2020-02-05T22:13:29.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:29.090+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.090971+0000) 2020-02-05T22:13:29.094 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:29.090+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.090789+0000) 2020-02-05T22:13:29.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:29.090+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.091417+0000) 2020-02-05T22:13:29.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:29.095+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.096828+0000) 2020-02-05T22:13:29.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:29.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:29.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:29.450+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.450363+0000) 2020-02-05T22:13:29.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:29.483+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:29.483491+0000) 2020-02-05T22:13:30.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:30.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:30.057930+0000) 2020-02-05T22:13:30.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:30.293+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:30.293507+0000) 2020-02-05T22:13:30.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:30.310+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:30.310330+0000) 2020-02-05T22:13:30.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:30.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:30.310729+0000) 2020-02-05T22:13:32.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:32.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:32.304760+0000) 2020-02-05T22:13:32.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:32.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:32.701126+0000) 2020-02-05T22:13:33.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:33.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:33.818143+0000) 2020-02-05T22:13:33.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:33.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:33.818370+0000) 2020-02-05T22:13:34.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:34.101+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.102858+0000) 2020-02-05T22:13:34.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:34.104+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.104813+0000) 2020-02-05T22:13:34.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:34.104+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.105734+0000) 2020-02-05T22:13:34.108 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:34.106+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.106658+0000) 2020-02-05T22:13:34.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:34.106+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.107413+0000) 2020-02-05T22:13:34.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:34.107+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.107778+0000) 2020-02-05T22:13:34.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:34.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.108385+0000) 2020-02-05T22:13:34.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:34.108+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.109083+0000) 2020-02-05T22:13:34.110 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:34.108+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.109318+0000) 2020-02-05T22:13:34.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:34.109+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.109568+0000) 2020-02-05T22:13:34.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:34.108+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.109958+0000) 2020-02-05T22:13:34.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:34.109+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.110357+0000) 2020-02-05T22:13:34.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:34.109+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.109975+0000) 2020-02-05T22:13:34.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:34.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:34.116009+0000) 2020-02-05T22:13:34.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:34.116+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108950 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:35.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:35.914+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:35.914371+0000) 2020-02-05T22:13:35.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:35.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:35.920984+0000) 2020-02-05T22:13:37.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:37.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:37.060948+0000) 2020-02-05T22:13:37.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:37.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:37.061132+0000) 2020-02-05T22:13:39.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:39.122+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.122734+0000) 2020-02-05T22:13:39.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:39.123+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.123358+0000) 2020-02-05T22:13:39.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:39.123+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.124052+0000) 2020-02-05T22:13:39.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:39.124+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.124777+0000) 2020-02-05T22:13:39.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:39.125+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.125717+0000) 2020-02-05T22:13:39.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:39.125+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.126267+0000) 2020-02-05T22:13:39.127 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:39.126+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.126677+0000) 2020-02-05T22:13:39.128 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:39.127+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.127179+0000) 2020-02-05T22:13:39.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:39.127+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.127577+0000) 2020-02-05T22:13:39.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:39.127+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.128337+0000) 2020-02-05T22:13:39.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:39.128+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.128707+0000) 2020-02-05T22:13:39.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:39.128+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.129021+0000) 2020-02-05T22:13:39.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:39.127+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.128105+0000) 2020-02-05T22:13:39.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:39.135+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.135191+0000) 2020-02-05T22:13:39.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:39.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 108989 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:39.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:39.450+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.450540+0000) 2020-02-05T22:13:39.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:39.491+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:39.491718+0000) 2020-02-05T22:13:40.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:40.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:40.058144+0000) 2020-02-05T22:13:40.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:40.293+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:40.293773+0000) 2020-02-05T22:13:40.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:40.310+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:40.310602+0000) 2020-02-05T22:13:40.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:40.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:40.310994+0000) 2020-02-05T22:13:42.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:42.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:42.305052+0000) 2020-02-05T22:13:42.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:42.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:42.701321+0000) 2020-02-05T22:13:43.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:43.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:43.818369+0000) 2020-02-05T22:13:43.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:43.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:43.818615+0000) 2020-02-05T22:13:44.144 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:44.141+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.142900+0000) 2020-02-05T22:13:44.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:44.143+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.143904+0000) 2020-02-05T22:13:44.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:44.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.145034+0000) 2020-02-05T22:13:44.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:44.144+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.145315+0000) 2020-02-05T22:13:44.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:44.145+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.146640+0000) 2020-02-05T22:13:44.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:44.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.147918+0000) 2020-02-05T22:13:44.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:44.147+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.148274+0000) 2020-02-05T22:13:44.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:44.147+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.148786+0000) 2020-02-05T22:13:44.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:44.147+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.148863+0000) 2020-02-05T22:13:44.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:44.145+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.146230+0000) 2020-02-05T22:13:44.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:44.146+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.147334+0000) 2020-02-05T22:13:44.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:44.148+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.148423+0000) 2020-02-05T22:13:44.152 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:44.148+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.149064+0000) 2020-02-05T22:13:44.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:44.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:44.151541+0000) 2020-02-05T22:13:44.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:44.156+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:45.916 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:45.913+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:45.914601+0000) 2020-02-05T22:13:45.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:45.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:45.921217+0000) 2020-02-05T22:13:47.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:47.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:47.061186+0000) 2020-02-05T22:13:47.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:47.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:47.061367+0000) 2020-02-05T22:13:49.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:49.162+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.163310+0000) 2020-02-05T22:13:49.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:49.163+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.163608+0000) 2020-02-05T22:13:49.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:49.163+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.164112+0000) 2020-02-05T22:13:49.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:49.163+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.164510+0000) 2020-02-05T22:13:49.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:49.163+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.164760+0000) 2020-02-05T22:13:49.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:49.165+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.165584+0000) 2020-02-05T22:13:49.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:49.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.165828+0000) 2020-02-05T22:13:49.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:49.165+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.166385+0000) 2020-02-05T22:13:49.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:49.165+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.166490+0000) 2020-02-05T22:13:49.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:49.166+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.166842+0000) 2020-02-05T22:13:49.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:49.166+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.167214+0000) 2020-02-05T22:13:49.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:49.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.168303+0000) 2020-02-05T22:13:49.170 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:49.167+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.167793+0000) 2020-02-05T22:13:49.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:49.169+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.170550+0000) 2020-02-05T22:13:49.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:49.176+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:49.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:49.449+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.450824+0000) 2020-02-05T22:13:49.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:49.499+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:49.500124+0000) 2020-02-05T22:13:50.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:50.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:50.058308+0000) 2020-02-05T22:13:50.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:50.293+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:50.294054+0000) 2020-02-05T22:13:50.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:50.309+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:50.310826+0000) 2020-02-05T22:13:50.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:50.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:50.311269+0000) 2020-02-05T22:13:52.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:52.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:52.305298+0000) 2020-02-05T22:13:52.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:52.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:52.701528+0000) 2020-02-05T22:13:53.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:53.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:53.818594+0000) 2020-02-05T22:13:53.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:53.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:53.818857+0000) 2020-02-05T22:13:54.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:54.182+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.183410+0000) 2020-02-05T22:13:54.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:54.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.184178+0000) 2020-02-05T22:13:54.185 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:54.183+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.184483+0000) 2020-02-05T22:13:54.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:54.183+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.184843+0000) 2020-02-05T22:13:54.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:54.184+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.185790+0000) 2020-02-05T22:13:54.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:54.184+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.186111+0000) 2020-02-05T22:13:54.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:54.185+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.186470+0000) 2020-02-05T22:13:54.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:54.185+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.186797+0000) 2020-02-05T22:13:54.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:54.186+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.187567+0000) 2020-02-05T22:13:54.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:54.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.187867+0000) 2020-02-05T22:13:54.189 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:54.186+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.188228+0000) 2020-02-05T22:13:54.189 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:54.187+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.188338+0000) 2020-02-05T22:13:54.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:54.187+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.188731+0000) 2020-02-05T22:13:54.197 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:54.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:54.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:54.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:54.196415+0000) 2020-02-05T22:13:55.916 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:55.913+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:55.914842+0000) 2020-02-05T22:13:55.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:55.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:55.921475+0000) 2020-02-05T22:13:56.571 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:13:56.578 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:13:57.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:57.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:57.061473+0000) 2020-02-05T22:13:57.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:57.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:57.061695+0000) 2020-02-05T22:13:59.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:13:59.200+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.201357+0000) 2020-02-05T22:13:59.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:13:59.201+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.202986+0000) 2020-02-05T22:13:59.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:59.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.203315+0000) 2020-02-05T22:13:59.205 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:13:59.202+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.203543+0000) 2020-02-05T22:13:59.205 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:13:59.202+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.203610+0000) 2020-02-05T22:13:59.205 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:13:59.202+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.203925+0000) 2020-02-05T22:13:59.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:13:59.203+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.204415+0000) 2020-02-05T22:13:59.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:13:59.202+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.204342+0000) 2020-02-05T22:13:59.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:13:59.203+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.205133+0000) 2020-02-05T22:13:59.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:59.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.205812+0000) 2020-02-05T22:13:59.208 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:13:59.205+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.206811+0000) 2020-02-05T22:13:59.209 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:13:59.205+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.207003+0000) 2020-02-05T22:13:59.209 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:13:59.206+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.207457+0000) 2020-02-05T22:13:59.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:59.209+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.210937+0000) 2020-02-05T22:13:59.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:13:59.214+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:13:59.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:59.450+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.451037+0000) 2020-02-05T22:13:59.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:13:59.507+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:13:59.508250+0000) 2020-02-05T22:14:00.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:00.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:00.058521+0000) 2020-02-05T22:14:00.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:00.293+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:00.294302+0000) 2020-02-05T22:14:00.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:00.310+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:00.311052+0000) 2020-02-05T22:14:00.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:00.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:00.311503+0000) 2020-02-05T22:14:02.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:02.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:02.305541+0000) 2020-02-05T22:14:02.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:02.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:02.701747+0000) 2020-02-05T22:14:03.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:03.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:03.818813+0000) 2020-02-05T22:14:03.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:03.818+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:03.819135+0000) 2020-02-05T22:14:04.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:04.220+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.221690+0000) 2020-02-05T22:14:04.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:04.222+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.223331+0000) 2020-02-05T22:14:04.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:04.221+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.222848+0000) 2020-02-05T22:14:04.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:04.222+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.223958+0000) 2020-02-05T22:14:04.225 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:04.223+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.224590+0000) 2020-02-05T22:14:04.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:04.224+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.225140+0000) 2020-02-05T22:14:04.226 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:04.224+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.225181+0000) 2020-02-05T22:14:04.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:04.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.225950+0000) 2020-02-05T22:14:04.227 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:04.224+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.225591+0000) 2020-02-05T22:14:04.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:04.225+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.227123+0000) 2020-02-05T22:14:04.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:04.226+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.227678+0000) 2020-02-05T22:14:04.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:04.226+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.227947+0000) 2020-02-05T22:14:04.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:04.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.233060+0000) 2020-02-05T22:14:04.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:04.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:04.233428+0000) 2020-02-05T22:14:04.236 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:04.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109192 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:05.916 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:05.914+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:05.915117+0000) 2020-02-05T22:14:05.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:05.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:05.921685+0000) 2020-02-05T22:14:07.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:07.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:07.061760+0000) 2020-02-05T22:14:07.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:07.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:07.061929+0000) 2020-02-05T22:14:09.241 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:09.238+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.239910+0000) 2020-02-05T22:14:09.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:09.237+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.239375+0000) 2020-02-05T22:14:09.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:09.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.240602+0000) 2020-02-05T22:14:09.243 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:09.239+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.240966+0000) 2020-02-05T22:14:09.243 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:09.240+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.241566+0000) 2020-02-05T22:14:09.244 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:09.242+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.243422+0000) 2020-02-05T22:14:09.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:09.243+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.244211+0000) 2020-02-05T22:14:09.245 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:09.243+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.244408+0000) 2020-02-05T22:14:09.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:09.243+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.244718+0000) 2020-02-05T22:14:09.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:09.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.245175+0000) 2020-02-05T22:14:09.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:09.244+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.245938+0000) 2020-02-05T22:14:09.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:09.245+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.246520+0000) 2020-02-05T22:14:09.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:09.245+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.246614+0000) 2020-02-05T22:14:09.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:09.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.252041+0000) 2020-02-05T22:14:09.256 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:09.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109232 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:09.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:09.450+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.451284+0000) 2020-02-05T22:14:09.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:09.515+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:09.516536+0000) 2020-02-05T22:14:10.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:10.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:10.058730+0000) 2020-02-05T22:14:10.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:10.293+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:10.294536+0000) 2020-02-05T22:14:10.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:10.310+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:10.311308+0000) 2020-02-05T22:14:10.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:10.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:10.311738+0000) 2020-02-05T22:14:12.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:12.304+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:12.305746+0000) 2020-02-05T22:14:12.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:12.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:12.702005+0000) 2020-02-05T22:14:13.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:13.818+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:13.819070+0000) 2020-02-05T22:14:13.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:13.818+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:13.819350+0000) 2020-02-05T22:14:14.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:14.261+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.262607+0000) 2020-02-05T22:14:14.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:14.260+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.262380+0000) 2020-02-05T22:14:14.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:14.262+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.263579+0000) 2020-02-05T22:14:14.265 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:14.262+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.264043+0000) 2020-02-05T22:14:14.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:14.263+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.264838+0000) 2020-02-05T22:14:14.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:14.263+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.265201+0000) 2020-02-05T22:14:14.268 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:14.264+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.265763+0000) 2020-02-05T22:14:14.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:14.265+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.266460+0000) 2020-02-05T22:14:14.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:14.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.266519+0000) 2020-02-05T22:14:14.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:14.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.267039+0000) 2020-02-05T22:14:14.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:14.265+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.266868+0000) 2020-02-05T22:14:14.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:14.265+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.267187+0000) 2020-02-05T22:14:14.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:14.265+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.267342+0000) 2020-02-05T22:14:14.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:14.271+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:14.272730+0000) 2020-02-05T22:14:14.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:14.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109274 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:15.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:15.914+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:15.915346+0000) 2020-02-05T22:14:15.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:15.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:15.921928+0000) 2020-02-05T22:14:17.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:17.060+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:17.062007+0000) 2020-02-05T22:14:17.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:17.060+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:17.062168+0000) 2020-02-05T22:14:19.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:19.278+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.279827+0000) 2020-02-05T22:14:19.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:19.280+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.281583+0000) 2020-02-05T22:14:19.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:19.280+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.281841+0000) 2020-02-05T22:14:19.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:19.281+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.282746+0000) 2020-02-05T22:14:19.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:19.281+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.283115+0000) 2020-02-05T22:14:19.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:19.282+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.283616+0000) 2020-02-05T22:14:19.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:19.282+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.283752+0000) 2020-02-05T22:14:19.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:19.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.284721+0000) 2020-02-05T22:14:19.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:19.283+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.284492+0000) 2020-02-05T22:14:19.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:19.283+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.284983+0000) 2020-02-05T22:14:19.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:19.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.285509+0000) 2020-02-05T22:14:19.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:19.284+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.285317+0000) 2020-02-05T22:14:19.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:19.284+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.285656+0000) 2020-02-05T22:14:19.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:19.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.288858+0000) 2020-02-05T22:14:19.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:19.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:19.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:19.451+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.451491+0000) 2020-02-05T22:14:19.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:19.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:19.524684+0000) 2020-02-05T22:14:20.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:20.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:20.058939+0000) 2020-02-05T22:14:20.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:20.294+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:20.294788+0000) 2020-02-05T22:14:20.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:20.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:20.311532+0000) 2020-02-05T22:14:20.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:20.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:20.311938+0000) 2020-02-05T22:14:22.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:22.305+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:22.306053+0000) 2020-02-05T22:14:22.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:22.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:22.702259+0000) 2020-02-05T22:14:23.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:23.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:23.819341+0000) 2020-02-05T22:14:23.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:23.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:23.819571+0000) 2020-02-05T22:14:24.301 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:24.297+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.299306+0000) 2020-02-05T22:14:24.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:24.299+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.300637+0000) 2020-02-05T22:14:24.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:24.300+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.301610+0000) 2020-02-05T22:14:24.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:24.301+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.302618+0000) 2020-02-05T22:14:24.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:24.301+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.303025+0000) 2020-02-05T22:14:24.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:24.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.303374+0000) 2020-02-05T22:14:24.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:24.302+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.303728+0000) 2020-02-05T22:14:24.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:24.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.304392+0000) 2020-02-05T22:14:24.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:24.303+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.304539+0000) 2020-02-05T22:14:24.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:24.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.305324+0000) 2020-02-05T22:14:24.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:24.303+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.304871+0000) 2020-02-05T22:14:24.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:24.304+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.305900+0000) 2020-02-05T22:14:24.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:24.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.309364+0000) 2020-02-05T22:14:24.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:24.309+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:24.310428+0000) 2020-02-05T22:14:24.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:24.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:25.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:25.915+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:25.915577+0000) 2020-02-05T22:14:25.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:25.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:25.922206+0000) 2020-02-05T22:14:26.710 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:14:26.716 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:14:27.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:27.061+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:27.062259+0000) 2020-02-05T22:14:27.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:27.061+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:27.062424+0000) 2020-02-05T22:14:29.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:29.319+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.320181+0000) 2020-02-05T22:14:29.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:29.319+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.321069+0000) 2020-02-05T22:14:29.322 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:29.319+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.321363+0000) 2020-02-05T22:14:29.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:29.320+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.321638+0000) 2020-02-05T22:14:29.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:29.320+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.322191+0000) 2020-02-05T22:14:29.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:29.321+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.322751+0000) 2020-02-05T22:14:29.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:29.322+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.323218+0000) 2020-02-05T22:14:29.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:29.322+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.323366+0000) 2020-02-05T22:14:29.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:29.322+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.323440+0000) 2020-02-05T22:14:29.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:29.322+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.324146+0000) 2020-02-05T22:14:29.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:29.322+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.323864+0000) 2020-02-05T22:14:29.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:29.323+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.324126+0000) 2020-02-05T22:14:29.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:29.323+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.324880+0000) 2020-02-05T22:14:29.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:29.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.330943+0000) 2020-02-05T22:14:29.335 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:29.332+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:29.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:29.451+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.451722+0000) 2020-02-05T22:14:29.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:29.533+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:29.534034+0000) 2020-02-05T22:14:30.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:30.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:30.059192+0000) 2020-02-05T22:14:30.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:30.294+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:30.295046+0000) 2020-02-05T22:14:30.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:30.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:30.311783+0000) 2020-02-05T22:14:30.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:30.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:30.312200+0000) 2020-02-05T22:14:32.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:32.305+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:32.306301+0000) 2020-02-05T22:14:32.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:32.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:32.702493+0000) 2020-02-05T22:14:33.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:33.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:33.819587+0000) 2020-02-05T22:14:33.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:33.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:33.819846+0000) 2020-02-05T22:14:34.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:34.339+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.339630+0000) 2020-02-05T22:14:34.341 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:34.339+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.340585+0000) 2020-02-05T22:14:34.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:34.340+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.341733+0000) 2020-02-05T22:14:34.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:34.340+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.342243+0000) 2020-02-05T22:14:34.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:34.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.342829+0000) 2020-02-05T22:14:34.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:34.343+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.343147+0000) 2020-02-05T22:14:34.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:34.341+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.343238+0000) 2020-02-05T22:14:34.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:34.341+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.343253+0000) 2020-02-05T22:14:34.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:34.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.343815+0000) 2020-02-05T22:14:34.346 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:34.343+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.343871+0000) 2020-02-05T22:14:34.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:34.342+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.344405+0000) 2020-02-05T22:14:34.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:34.344+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.344916+0000) 2020-02-05T22:14:34.347 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:34.346+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.346158+0000) 2020-02-05T22:14:34.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:34.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:34.348189+0000) 2020-02-05T22:14:34.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:34.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:35.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:35.914+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:35.915894+0000) 2020-02-05T22:14:35.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:35.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:35.922493+0000) 2020-02-05T22:14:37.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:37.061+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:37.062515+0000) 2020-02-05T22:14:37.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:37.061+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:37.062708+0000) 2020-02-05T22:14:39.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:39.358+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.359987+0000) 2020-02-05T22:14:39.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:39.359+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.360338+0000) 2020-02-05T22:14:39.362 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:39.359+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.360865+0000) 2020-02-05T22:14:39.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:39.360+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.361819+0000) 2020-02-05T22:14:39.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:39.361+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.362537+0000) 2020-02-05T22:14:39.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:39.360+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.362009+0000) 2020-02-05T22:14:39.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:39.361+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.363011+0000) 2020-02-05T22:14:39.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:39.361+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.363329+0000) 2020-02-05T22:14:39.366 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:39.363+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.364706+0000) 2020-02-05T22:14:39.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:39.364+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.365205+0000) 2020-02-05T22:14:39.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:39.363+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.365233+0000) 2020-02-05T22:14:39.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:39.363+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.365362+0000) 2020-02-05T22:14:39.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:39.364+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.366238+0000) 2020-02-05T22:14:39.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:39.368+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.369172+0000) 2020-02-05T22:14:39.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:39.372+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:39.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:39.450+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.451958+0000) 2020-02-05T22:14:39.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:39.543+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:39.544869+0000) 2020-02-05T22:14:40.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:40.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:40.059380+0000) 2020-02-05T22:14:40.296 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:40.294+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:40.295278+0000) 2020-02-05T22:14:40.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:40.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:40.311988+0000) 2020-02-05T22:14:40.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:40.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:40.312420+0000) 2020-02-05T22:14:42.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:42.306+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:42.306502+0000) 2020-02-05T22:14:42.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:42.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:42.702730+0000) 2020-02-05T22:14:43.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:43.818+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:43.819856+0000) 2020-02-05T22:14:43.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:43.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:43.820103+0000) 2020-02-05T22:14:44.381 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:44.378+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.379634+0000) 2020-02-05T22:14:44.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:44.379+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.380341+0000) 2020-02-05T22:14:44.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:44.379+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.381020+0000) 2020-02-05T22:14:44.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:44.380+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.380925+0000) 2020-02-05T22:14:44.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:44.382+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.382799+0000) 2020-02-05T22:14:44.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:44.382+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.383259+0000) 2020-02-05T22:14:44.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:44.383+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.383856+0000) 2020-02-05T22:14:44.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:44.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.384279+0000) 2020-02-05T22:14:44.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:44.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.384824+0000) 2020-02-05T22:14:44.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:44.383+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.385017+0000) 2020-02-05T22:14:44.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:44.385+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.385578+0000) 2020-02-05T22:14:44.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:44.385+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.385622+0000) 2020-02-05T22:14:44.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:44.385+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.385992+0000) 2020-02-05T22:14:44.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:44.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:44.389918+0000) 2020-02-05T22:14:44.395 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:44.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:45.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:45.915+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:45.916118+0000) 2020-02-05T22:14:45.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:45.922+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:45.922758+0000) 2020-02-05T22:14:47.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:47.062+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:47.062784+0000) 2020-02-05T22:14:47.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:47.062+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:47.062909+0000) 2020-02-05T22:14:49.401 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:49.398+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.399850+0000) 2020-02-05T22:14:49.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:49.400+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.400522+0000) 2020-02-05T22:14:49.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:49.400+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.401605+0000) 2020-02-05T22:14:49.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:49.401+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.402611+0000) 2020-02-05T22:14:49.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:49.401+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.402883+0000) 2020-02-05T22:14:49.405 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:49.402+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.403340+0000) 2020-02-05T22:14:49.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:49.402+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.403239+0000) 2020-02-05T22:14:49.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:49.403+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.403782+0000) 2020-02-05T22:14:49.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:49.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.404445+0000) 2020-02-05T22:14:49.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:49.404+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.404873+0000) 2020-02-05T22:14:49.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:49.404+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.405329+0000) 2020-02-05T22:14:49.407 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:49.404+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.405353+0000) 2020-02-05T22:14:49.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:49.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.406139+0000) 2020-02-05T22:14:49.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:49.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.408913+0000) 2020-02-05T22:14:49.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:49.412+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:49.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:49.451+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.452188+0000) 2020-02-05T22:14:49.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:49.552+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:49.552986+0000) 2020-02-05T22:14:50.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:50.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:50.059632+0000) 2020-02-05T22:14:50.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:50.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:50.295503+0000) 2020-02-05T22:14:50.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:50.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:50.312191+0000) 2020-02-05T22:14:50.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:50.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:50.312610+0000) 2020-02-05T22:14:52.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:52.306+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:52.306763+0000) 2020-02-05T22:14:52.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:52.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:52.702961+0000) 2020-02-05T22:14:53.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:53.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:53.820141+0000) 2020-02-05T22:14:53.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:53.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:53.820309+0000) 2020-02-05T22:14:54.421 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:54.419+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.419602+0000) 2020-02-05T22:14:54.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:54.419+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.420283+0000) 2020-02-05T22:14:54.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:54.419+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.420806+0000) 2020-02-05T22:14:54.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:54.420+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.421429+0000) 2020-02-05T22:14:54.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:54.420+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.421972+0000) 2020-02-05T22:14:54.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:54.421+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.422812+0000) 2020-02-05T22:14:54.424 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:54.422+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.423188+0000) 2020-02-05T22:14:54.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:54.423+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.423838+0000) 2020-02-05T22:14:54.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:54.422+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.424025+0000) 2020-02-05T22:14:54.426 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:54.423+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.424329+0000) 2020-02-05T22:14:54.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:54.423+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.423872+0000) 2020-02-05T22:14:54.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:54.423+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.423473+0000) 2020-02-05T22:14:54.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:54.424+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.424942+0000) 2020-02-05T22:14:54.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:54.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:54.430832+0000) 2020-02-05T22:14:54.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:54.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109606 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:55.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:55.915+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:55.916340+0000) 2020-02-05T22:14:55.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:55.922+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:55.922999+0000) 2020-02-05T22:14:56.847 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:14:56.854 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:14:57.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:57.062+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:57.063052+0000) 2020-02-05T22:14:57.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:57.062+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:57.063166+0000) 2020-02-05T22:14:59.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:14:59.437+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.437963+0000) 2020-02-05T22:14:59.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:14:59.437+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.438200+0000) 2020-02-05T22:14:59.440 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:14:59.437+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.438620+0000) 2020-02-05T22:14:59.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:59.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.439362+0000) 2020-02-05T22:14:59.441 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:14:59.439+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.440373+0000) 2020-02-05T22:14:59.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:14:59.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.440687+0000) 2020-02-05T22:14:59.443 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:14:59.441+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.442201+0000) 2020-02-05T22:14:59.443 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:14:59.441+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.442315+0000) 2020-02-05T22:14:59.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:14:59.442+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.442681+0000) 2020-02-05T22:14:59.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:14:59.442+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.442994+0000) 2020-02-05T22:14:59.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:14:59.442+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.443366+0000) 2020-02-05T22:14:59.445 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:14:59.441+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.442854+0000) 2020-02-05T22:14:59.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:59.447+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.448117+0000) 2020-02-05T22:14:59.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:59.447+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.448472+0000) 2020-02-05T22:14:59.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:14:59.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:14:59.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:59.451+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.452359+0000) 2020-02-05T22:14:59.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:14:59.563+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:14:59.564402+0000) 2020-02-05T22:15:00.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:00.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:00.059833+0000) 2020-02-05T22:15:00.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:00.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:00.295670+0000) 2020-02-05T22:15:00.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:00.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:00.312407+0000) 2020-02-05T22:15:00.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:00.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:00.312807+0000) 2020-02-05T22:15:02.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:02.306+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:02.307007+0000) 2020-02-05T22:15:02.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:02.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:02.703197+0000) 2020-02-05T22:15:03.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:03.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:03.820374+0000) 2020-02-05T22:15:03.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:03.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:03.820506+0000) 2020-02-05T22:15:04.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:04.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.456287+0000) 2020-02-05T22:15:04.458 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:04.456+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.457262+0000) 2020-02-05T22:15:04.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:04.456+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.458013+0000) 2020-02-05T22:15:04.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:04.457+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.458534+0000) 2020-02-05T22:15:04.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:04.458+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.459208+0000) 2020-02-05T22:15:04.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:04.458+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.458968+0000) 2020-02-05T22:15:04.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:04.458+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.460036+0000) 2020-02-05T22:15:04.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:04.459+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.460340+0000) 2020-02-05T22:15:04.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:04.460+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.460628+0000) 2020-02-05T22:15:04.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:04.460+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.460696+0000) 2020-02-05T22:15:04.463 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:04.460+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.461862+0000) 2020-02-05T22:15:04.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:04.460+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.461269+0000) 2020-02-05T22:15:04.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:04.461+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.462553+0000) 2020-02-05T22:15:04.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:04.466+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:04.467700+0000) 2020-02-05T22:15:04.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:04.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109689 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:05.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:05.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:05.916591+0000) 2020-02-05T22:15:05.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:05.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:05.923244+0000) 2020-02-05T22:15:07.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:07.063+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:07.063301+0000) 2020-02-05T22:15:07.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:07.063+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:07.063420+0000) 2020-02-05T22:15:09.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:09.452+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.453026+0000) 2020-02-05T22:15:09.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:09.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.475767+0000) 2020-02-05T22:15:09.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:09.475+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.476276+0000) 2020-02-05T22:15:09.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:09.475+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.476641+0000) 2020-02-05T22:15:09.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:09.476+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.476871+0000) 2020-02-05T22:15:09.478 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:09.476+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.477298+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:09.478+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.479703+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:09.479+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.481021+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:09.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.482050+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:09.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.487138+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:09.477+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.477902+0000) 2020-02-05T22:15:09.490 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:09.478+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.478616+0000) 2020-02-05T22:15:09.491 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:09.480+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.480693+0000) 2020-02-05T22:15:09.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:09.479+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.480348+0000) 2020-02-05T22:15:09.491 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:09.482+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.482452+0000) 2020-02-05T22:15:09.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:09.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:09.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:09.571+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:09.572661+0000) 2020-02-05T22:15:10.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:10.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:10.060130+0000) 2020-02-05T22:15:10.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:10.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:10.295846+0000) 2020-02-05T22:15:10.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:10.311+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:10.312558+0000) 2020-02-05T22:15:10.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:10.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:10.313010+0000) 2020-02-05T22:15:12.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:12.307+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:12.307271+0000) 2020-02-05T22:15:12.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:12.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:12.703379+0000) 2020-02-05T22:15:13.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:13.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:13.820697+0000) 2020-02-05T22:15:13.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:13.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:13.820631+0000) 2020-02-05T22:15:14.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:14.493+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.494732+0000) 2020-02-05T22:15:14.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:14.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.495247+0000) 2020-02-05T22:15:14.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:14.496+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.496456+0000) 2020-02-05T22:15:14.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:14.497+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.497519+0000) 2020-02-05T22:15:14.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:14.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.498123+0000) 2020-02-05T22:15:14.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:14.497+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.498398+0000) 2020-02-05T22:15:14.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:14.497+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.498457+0000) 2020-02-05T22:15:14.500 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:14.498+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.498760+0000) 2020-02-05T22:15:14.500 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:14.498+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.499286+0000) 2020-02-05T22:15:14.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:14.498+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.498881+0000) 2020-02-05T22:15:14.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:14.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.499658+0000) 2020-02-05T22:15:14.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:14.499+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.499943+0000) 2020-02-05T22:15:14.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:14.499+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.500725+0000) 2020-02-05T22:15:14.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:14.504+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:14.505365+0000) 2020-02-05T22:15:14.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:14.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:15.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:15.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:15.916833+0000) 2020-02-05T22:15:15.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:15.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:15.923495+0000) 2020-02-05T22:15:17.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:17.063+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:17.063535+0000) 2020-02-05T22:15:17.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:17.063+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:17.063684+0000) 2020-02-05T22:15:19.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:19.452+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.453319+0000) 2020-02-05T22:15:19.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:19.512+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.513291+0000) 2020-02-05T22:15:19.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:19.512+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.513967+0000) 2020-02-05T22:15:19.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:19.514+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.514473+0000) 2020-02-05T22:15:19.517 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:19.515+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.516422+0000) 2020-02-05T22:15:19.517 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:19.515+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.515804+0000) 2020-02-05T22:15:19.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:19.517+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.517703+0000) 2020-02-05T22:15:19.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:19.517+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.518281+0000) 2020-02-05T22:15:19.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:19.518+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.519210+0000) 2020-02-05T22:15:19.521 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:19.519+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.519844+0000) 2020-02-05T22:15:19.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:19.519+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.519896+0000) 2020-02-05T22:15:19.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:19.519+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.520391+0000) 2020-02-05T22:15:19.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:19.519+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.520726+0000) 2020-02-05T22:15:19.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:19.520+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.521167+0000) 2020-02-05T22:15:19.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:19.522+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.523184+0000) 2020-02-05T22:15:19.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:19.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:19.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:19.578+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:19.579159+0000) 2020-02-05T22:15:20.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:20.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:20.060395+0000) 2020-02-05T22:15:20.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:20.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:20.296082+0000) 2020-02-05T22:15:20.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:20.312+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:20.312792+0000) 2020-02-05T22:15:20.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:20.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:20.313202+0000) 2020-02-05T22:15:22.309 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:22.306+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:22.307527+0000) 2020-02-05T22:15:22.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:22.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:22.703614+0000) 2020-02-05T22:15:23.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:23.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:23.820919+0000) 2020-02-05T22:15:23.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:23.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:23.820908+0000) 2020-02-05T22:15:24.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:24.531+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.533024+0000) 2020-02-05T22:15:24.535 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:24.533+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.534425+0000) 2020-02-05T22:15:24.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:24.534+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.535485+0000) 2020-02-05T22:15:24.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:24.534+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.535946+0000) 2020-02-05T22:15:24.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:24.534+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.536265+0000) 2020-02-05T22:15:24.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:24.535+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.536531+0000) 2020-02-05T22:15:24.539 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:24.535+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.536973+0000) 2020-02-05T22:15:24.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:24.536+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.537649+0000) 2020-02-05T22:15:24.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:24.536+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.538059+0000) 2020-02-05T22:15:24.541 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:24.536+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.538284+0000) 2020-02-05T22:15:24.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:24.537+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.538716+0000) 2020-02-05T22:15:24.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:24.537+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.538159+0000) 2020-02-05T22:15:24.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:24.537+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.538457+0000) 2020-02-05T22:15:24.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:24.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:24.542160+0000) 2020-02-05T22:15:24.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:24.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:25.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:25.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:25.917063+0000) 2020-02-05T22:15:25.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:25.922+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:25.923742+0000) 2020-02-05T22:15:27.011 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:15:27.017 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:15:27.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:27.062+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:27.063738+0000) 2020-02-05T22:15:27.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:27.062+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:27.063914+0000) 2020-02-05T22:15:29.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:29.453+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.453557+0000) 2020-02-05T22:15:29.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:29.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.552537+0000) 2020-02-05T22:15:29.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:29.553+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.553340+0000) 2020-02-05T22:15:29.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:29.553+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.553727+0000) 2020-02-05T22:15:29.555 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:29.554+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.554062+0000) 2020-02-05T22:15:29.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:29.553+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.554667+0000) 2020-02-05T22:15:29.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:29.554+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.554986+0000) 2020-02-05T22:15:29.557 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:29.554+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.555496+0000) 2020-02-05T22:15:29.557 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:29.554+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.556266+0000) 2020-02-05T22:15:29.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:29.556+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.556612+0000) 2020-02-05T22:15:29.558 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:29.555+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.557291+0000) 2020-02-05T22:15:29.559 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:29.556+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.557927+0000) 2020-02-05T22:15:29.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:29.556+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.557836+0000) 2020-02-05T22:15:29.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:29.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.558590+0000) 2020-02-05T22:15:29.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:29.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.562269+0000) 2020-02-05T22:15:29.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:29.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:29.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:29.587+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:29.587274+0000) 2020-02-05T22:15:30.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:30.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:30.060612+0000) 2020-02-05T22:15:30.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:30.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:30.296314+0000) 2020-02-05T22:15:30.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:30.312+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:30.312988+0000) 2020-02-05T22:15:30.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:30.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:30.313412+0000) 2020-02-05T22:15:32.309 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:32.306+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:32.307785+0000) 2020-02-05T22:15:32.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:32.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:32.703880+0000) 2020-02-05T22:15:33.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:33.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:33.821179+0000) 2020-02-05T22:15:33.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:33.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:33.821179+0000) 2020-02-05T22:15:34.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:34.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.570656+0000) 2020-02-05T22:15:34.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:34.570+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.571225+0000) 2020-02-05T22:15:34.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:34.570+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.571976+0000) 2020-02-05T22:15:34.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:34.571+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.572450+0000) 2020-02-05T22:15:34.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:34.571+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.572644+0000) 2020-02-05T22:15:34.574 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:34.571+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.572908+0000) 2020-02-05T22:15:34.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:34.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.573408+0000) 2020-02-05T22:15:34.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:34.571+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.573395+0000) 2020-02-05T22:15:34.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:34.572+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.573795+0000) 2020-02-05T22:15:34.575 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:34.572+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.573697+0000) 2020-02-05T22:15:34.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:34.572+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.574389+0000) 2020-02-05T22:15:34.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:34.573+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.574559+0000) 2020-02-05T22:15:34.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:34.573+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.575292+0000) 2020-02-05T22:15:34.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:34.577+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:34.578320+0000) 2020-02-05T22:15:34.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:34.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:35.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:35.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:35.917413+0000) 2020-02-05T22:15:35.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:35.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:35.924005+0000) 2020-02-05T22:15:37.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:37.063+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:37.064023+0000) 2020-02-05T22:15:37.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:37.063+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:37.064119+0000) 2020-02-05T22:15:39.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:39.452+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.453807+0000) 2020-02-05T22:15:39.589 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:39.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.587738+0000) 2020-02-05T22:15:39.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:39.587+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.588329+0000) 2020-02-05T22:15:39.590 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:39.587+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.589018+0000) 2020-02-05T22:15:39.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:39.588+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.589617+0000) 2020-02-05T22:15:39.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:39.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.590793+0000) 2020-02-05T22:15:39.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:39.589+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.590992+0000) 2020-02-05T22:15:39.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:39.590+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.591711+0000) 2020-02-05T22:15:39.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:39.590+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.591714+0000) 2020-02-05T22:15:39.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:39.591+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.592659+0000) 2020-02-05T22:15:39.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:39.591+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.593038+0000) 2020-02-05T22:15:39.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:39.591+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.593213+0000) 2020-02-05T22:15:39.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:39.592+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.594316+0000) 2020-02-05T22:15:39.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:39.592+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.593501+0000) 2020-02-05T22:15:39.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:39.595+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.596270+0000) 2020-02-05T22:15:39.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:39.595+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:39.596419+0000) 2020-02-05T22:15:39.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:39.599+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 109978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:40.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:40.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:40.060842+0000) 2020-02-05T22:15:40.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:40.295+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:40.296550+0000) 2020-02-05T22:15:40.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:40.312+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:40.313204+0000) 2020-02-05T22:15:40.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:40.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:40.313661+0000) 2020-02-05T22:15:42.309 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:42.307+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:42.308029+0000) 2020-02-05T22:15:42.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:42.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:42.704134+0000) 2020-02-05T22:15:43.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:43.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:43.821450+0000) 2020-02-05T22:15:43.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:43.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:43.821449+0000) 2020-02-05T22:15:44.608 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:44.606+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.607220+0000) 2020-02-05T22:15:44.609 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:44.606+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.607980+0000) 2020-02-05T22:15:44.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:44.607+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.608508+0000) 2020-02-05T22:15:44.610 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:44.607+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.608814+0000) 2020-02-05T22:15:44.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:44.608+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.609267+0000) 2020-02-05T22:15:44.610 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:44.608+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.609352+0000) 2020-02-05T22:15:44.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:44.608+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.610003+0000) 2020-02-05T22:15:44.611 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:44.608+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.610257+0000) 2020-02-05T22:15:44.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:44.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.611010+0000) 2020-02-05T22:15:44.612 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:44.609+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.610958+0000) 2020-02-05T22:15:44.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:44.610+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.611914+0000) 2020-02-05T22:15:44.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:44.611+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.612291+0000) 2020-02-05T22:15:44.614 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:44.611+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.612803+0000) 2020-02-05T22:15:44.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:44.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:44.618654+0000) 2020-02-05T22:15:44.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:44.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:45.919 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:45.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:45.917649+0000) 2020-02-05T22:15:45.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:45.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:45.924302+0000) 2020-02-05T22:15:47.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:47.063+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:47.064264+0000) 2020-02-05T22:15:47.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:47.063+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:47.064373+0000) 2020-02-05T22:15:49.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:49.452+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.454056+0000) 2020-02-05T22:15:49.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:49.595+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.596624+0000) 2020-02-05T22:15:49.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:49.622+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.624036+0000) 2020-02-05T22:15:49.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:49.622+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.623674+0000) 2020-02-05T22:15:49.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:49.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.625936+0000) 2020-02-05T22:15:49.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:49.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.626545+0000) 2020-02-05T22:15:49.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:49.625+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.626674+0000) 2020-02-05T22:15:49.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:49.625+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.627004+0000) 2020-02-05T22:15:49.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:49.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.627439+0000) 2020-02-05T22:15:49.629 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:49.626+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.628094+0000) 2020-02-05T22:15:49.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:49.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.628611+0000) 2020-02-05T22:15:49.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:49.627+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.629045+0000) 2020-02-05T22:15:49.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:49.629+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.630822+0000) 2020-02-05T22:15:49.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:49.629+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.631193+0000) 2020-02-05T22:15:49.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:49.629+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.631323+0000) 2020-02-05T22:15:49.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:49.632+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:49.633617+0000) 2020-02-05T22:15:49.640 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:49.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:50.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:50.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:50.061058+0000) 2020-02-05T22:15:50.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:50.296+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:50.296782+0000) 2020-02-05T22:15:50.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:50.312+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:50.313404+0000) 2020-02-05T22:15:50.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:50.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:50.313813+0000) 2020-02-05T22:15:52.309 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:52.307+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:52.308373+0000) 2020-02-05T22:15:52.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:52.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:52.704366+0000) 2020-02-05T22:15:53.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:53.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:53.821709+0000) 2020-02-05T22:15:53.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:53.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:53.821709+0000) 2020-02-05T22:15:54.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:54.643+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.644707+0000) 2020-02-05T22:15:54.647 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:54.644+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.645931+0000) 2020-02-05T22:15:54.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:54.645+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.646490+0000) 2020-02-05T22:15:54.648 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:54.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.646826+0000) 2020-02-05T22:15:54.648 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:54.646+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.647382+0000) 2020-02-05T22:15:54.649 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:54.645+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.647120+0000) 2020-02-05T22:15:54.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:54.646+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.647517+0000) 2020-02-05T22:15:54.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:54.646+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.647843+0000) 2020-02-05T22:15:54.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:54.647+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.648278+0000) 2020-02-05T22:15:54.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:54.647+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.648632+0000) 2020-02-05T22:15:54.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:54.647+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.648842+0000) 2020-02-05T22:15:54.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:54.648+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.649741+0000) 2020-02-05T22:15:54.651 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:54.648+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.650161+0000) 2020-02-05T22:15:54.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:54.654+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:54.655767+0000) 2020-02-05T22:15:54.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:54.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:55.919 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:55.916+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:55.917894+0000) 2020-02-05T22:15:55.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:55.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:55.924550+0000) 2020-02-05T22:15:57.066 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:57.063+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:57.064503+0000) 2020-02-05T22:15:57.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:57.063+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:57.064649+0000) 2020-02-05T22:15:57.146 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:15:57.158 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:15:59.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:59.453+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.454279+0000) 2020-02-05T22:15:59.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:59.604+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.605170+0000) 2020-02-05T22:15:59.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:15:59.664+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.665790+0000) 2020-02-05T22:15:59.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:15:59.665+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.667164+0000) 2020-02-05T22:15:59.670 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:15:59.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.664841+0000) 2020-02-05T22:15:59.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:59.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.666546+0000) 2020-02-05T22:15:59.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:15:59.666+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.667672+0000) 2020-02-05T22:15:59.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:15:59.667+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.668215+0000) 2020-02-05T22:15:59.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:15:59.667+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.669203+0000) 2020-02-05T22:15:59.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:15:59.668+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.669859+0000) 2020-02-05T22:15:59.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:15:59.669+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.670726+0000) 2020-02-05T22:15:59.673 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:15:59.669+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.671031+0000) 2020-02-05T22:15:59.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:59.668+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.669921+0000) 2020-02-05T22:15:59.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:15:59.669+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.670349+0000) 2020-02-05T22:15:59.686 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:15:59.669+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.670492+0000) 2020-02-05T22:15:59.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:15:59.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:15:59.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:15:59.674+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:15:59.675537+0000) 2020-02-05T22:16:00.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:00.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:00.061317+0000) 2020-02-05T22:16:00.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:00.296+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:00.296997+0000) 2020-02-05T22:16:00.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:00.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:00.313612+0000) 2020-02-05T22:16:00.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:00.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:00.314026+0000) 2020-02-05T22:16:02.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:02.308+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:02.308665+0000) 2020-02-05T22:16:02.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:02.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:02.704618+0000) 2020-02-05T22:16:03.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:03.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:03.822003+0000) 2020-02-05T22:16:03.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:03.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:03.822002+0000) 2020-02-05T22:16:04.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:04.680+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.682342+0000) 2020-02-05T22:16:04.684 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:04.682+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.683743+0000) 2020-02-05T22:16:04.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:04.684+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.685350+0000) 2020-02-05T22:16:04.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:04.683+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.685272+0000) 2020-02-05T22:16:04.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:04.684+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.685793+0000) 2020-02-05T22:16:04.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:04.685+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.686121+0000) 2020-02-05T22:16:04.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:04.685+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.687059+0000) 2020-02-05T22:16:04.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:04.686+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.687405+0000) 2020-02-05T22:16:04.689 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:04.686+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.687876+0000) 2020-02-05T22:16:04.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:04.687+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.688263+0000) 2020-02-05T22:16:04.689 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:04.687+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.688313+0000) 2020-02-05T22:16:04.690 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:04.686+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.688401+0000) 2020-02-05T22:16:04.690 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:04.687+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.688483+0000) 2020-02-05T22:16:04.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:04.693+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:04.694481+0000) 2020-02-05T22:16:04.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:04.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:05.919 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:05.917+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:05.918130+0000) 2020-02-05T22:16:05.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:05.924+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:05.924856+0000) 2020-02-05T22:16:07.066 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:07.064+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:07.064746+0000) 2020-02-05T22:16:07.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:07.064+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:07.064931+0000) 2020-02-05T22:16:09.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:09.454+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.454511+0000) 2020-02-05T22:16:09.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:09.613+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.613414+0000) 2020-02-05T22:16:09.702 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:09.698+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.699922+0000) 2020-02-05T22:16:09.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:09.701+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.702603+0000) 2020-02-05T22:16:09.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:09.702+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.703705+0000) 2020-02-05T22:16:09.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:09.703+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.704496+0000) 2020-02-05T22:16:09.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:09.704+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.705141+0000) 2020-02-05T22:16:09.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:09.704+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.705582+0000) 2020-02-05T22:16:09.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:09.704+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.705905+0000) 2020-02-05T22:16:09.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:09.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.706348+0000) 2020-02-05T22:16:09.708 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:09.705+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.706610+0000) 2020-02-05T22:16:09.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:09.705+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.706799+0000) 2020-02-05T22:16:09.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:09.706+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.707798+0000) 2020-02-05T22:16:09.709 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:09.706+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.707932+0000) 2020-02-05T22:16:09.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:09.707+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.708494+0000) 2020-02-05T22:16:09.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:09.712+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:09.713313+0000) 2020-02-05T22:16:09.716 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:09.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:10.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:10.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:10.061612+0000) 2020-02-05T22:16:10.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:10.296+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:10.297192+0000) 2020-02-05T22:16:10.315 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:10.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:10.313861+0000) 2020-02-05T22:16:10.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:10.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:10.314237+0000) 2020-02-05T22:16:12.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:12.308+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:12.308902+0000) 2020-02-05T22:16:12.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:12.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:12.704895+0000) 2020-02-05T22:16:13.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:13.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:13.822335+0000) 2020-02-05T22:16:13.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:13.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:13.822335+0000) 2020-02-05T22:16:14.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:14.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.721207+0000) 2020-02-05T22:16:14.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:14.720+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.721604+0000) 2020-02-05T22:16:14.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:14.721+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.722262+0000) 2020-02-05T22:16:14.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:14.722+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.722797+0000) 2020-02-05T22:16:14.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:14.722+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.723620+0000) 2020-02-05T22:16:14.725 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:14.723+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.723925+0000) 2020-02-05T22:16:14.725 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:14.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.723912+0000) 2020-02-05T22:16:14.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:14.723+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.724449+0000) 2020-02-05T22:16:14.726 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:14.723+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.724129+0000) 2020-02-05T22:16:14.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:14.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.724921+0000) 2020-02-05T22:16:14.727 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:14.724+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.725166+0000) 2020-02-05T22:16:14.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:14.724+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.725378+0000) 2020-02-05T22:16:14.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:14.725+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.725806+0000) 2020-02-05T22:16:14.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:14.730+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:14.731496+0000) 2020-02-05T22:16:14.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:14.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110265 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:15.919 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:15.918+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:15.918361+0000) 2020-02-05T22:16:15.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:15.924+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:15.925116+0000) 2020-02-05T22:16:17.066 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:17.064+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:17.065020+0000) 2020-02-05T22:16:17.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:17.064+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:17.065174+0000) 2020-02-05T22:16:19.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:19.454+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.454748+0000) 2020-02-05T22:16:19.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:19.621+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.621873+0000) 2020-02-05T22:16:19.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:19.735+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.736991+0000) 2020-02-05T22:16:19.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:19.737+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.738270+0000) 2020-02-05T22:16:19.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:19.738+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.739360+0000) 2020-02-05T22:16:19.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:19.738+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.739985+0000) 2020-02-05T22:16:19.742 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:19.740+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.740975+0000) 2020-02-05T22:16:19.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:19.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.741381+0000) 2020-02-05T22:16:19.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:19.740+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.741738+0000) 2020-02-05T22:16:19.743 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:19.740+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.741874+0000) 2020-02-05T22:16:19.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:19.742+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.742485+0000) 2020-02-05T22:16:19.744 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:19.742+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.742753+0000) 2020-02-05T22:16:19.744 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:19.742+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.743250+0000) 2020-02-05T22:16:19.745 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:19.742+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.743362+0000) 2020-02-05T22:16:19.745 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:19.743+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.744167+0000) 2020-02-05T22:16:19.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:19.747+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:19.749003+0000) 2020-02-05T22:16:19.752 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:19.750+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:20.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:20.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:20.061858+0000) 2020-02-05T22:16:20.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:20.297+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:20.297415+0000) 2020-02-05T22:16:20.315 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:20.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:20.314136+0000) 2020-02-05T22:16:20.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:20.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:20.314512+0000) 2020-02-05T22:16:22.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:22.308+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:22.309186+0000) 2020-02-05T22:16:22.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:22.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:22.705133+0000) 2020-02-05T22:16:23.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:23.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:23.822619+0000) 2020-02-05T22:16:23.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:23.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:23.822618+0000) 2020-02-05T22:16:24.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:24.756+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.757325+0000) 2020-02-05T22:16:24.759 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:24.757+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.758215+0000) 2020-02-05T22:16:24.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:24.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.757750+0000) 2020-02-05T22:16:24.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:24.759+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.759957+0000) 2020-02-05T22:16:24.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:24.758+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.758210+0000) 2020-02-05T22:16:24.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:24.758+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.758703+0000) 2020-02-05T22:16:24.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:24.759+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.759113+0000) 2020-02-05T22:16:24.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:24.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.759669+0000) 2020-02-05T22:16:24.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:24.758+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.758793+0000) 2020-02-05T22:16:24.763 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:24.759+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.759552+0000) 2020-02-05T22:16:24.764 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:24.760+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.760526+0000) 2020-02-05T22:16:24.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:24.760+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.760649+0000) 2020-02-05T22:16:24.765 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:24.761+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.761763+0000) 2020-02-05T22:16:24.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:24.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:24.766177+0000) 2020-02-05T22:16:24.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:24.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110347 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:25.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:25.917+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:25.918594+0000) 2020-02-05T22:16:25.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:25.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:25.925376+0000) 2020-02-05T22:16:27.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:27.064+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:27.065254+0000) 2020-02-05T22:16:27.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:27.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:27.065419+0000) 2020-02-05T22:16:27.280 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:16:27.289 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:16:29.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:29.454+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.454988+0000) 2020-02-05T22:16:29.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:29.629+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.630250+0000) 2020-02-05T22:16:29.778 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:29.773+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.774352+0000) 2020-02-05T22:16:29.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:29.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.774563+0000) 2020-02-05T22:16:29.778 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:29.775+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.775733+0000) 2020-02-05T22:16:29.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:29.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.773761+0000) 2020-02-05T22:16:29.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:29.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.774848+0000) 2020-02-05T22:16:29.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:29.774+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.775701+0000) 2020-02-05T22:16:29.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:29.775+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.776279+0000) 2020-02-05T22:16:29.779 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:29.775+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.776875+0000) 2020-02-05T22:16:29.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:29.776+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.776663+0000) 2020-02-05T22:16:29.780 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:29.776+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.777103+0000) 2020-02-05T22:16:29.781 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:29.777+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.778215+0000) 2020-02-05T22:16:29.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:29.777+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.778148+0000) 2020-02-05T22:16:29.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:29.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.779145+0000) 2020-02-05T22:16:29.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:29.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:29.783344+0000) 2020-02-05T22:16:29.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:29.787+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:30.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:30.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:30.062056+0000) 2020-02-05T22:16:30.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:30.297+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:30.297666+0000) 2020-02-05T22:16:30.315 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:30.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:30.314349+0000) 2020-02-05T22:16:30.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:30.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:30.314733+0000) 2020-02-05T22:16:32.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:32.309+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:32.309471+0000) 2020-02-05T22:16:32.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:32.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:32.705370+0000) 2020-02-05T22:16:33.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:33.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:33.822896+0000) 2020-02-05T22:16:33.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:33.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:33.822896+0000) 2020-02-05T22:16:34.794 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:34.791+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.792829+0000) 2020-02-05T22:16:34.794 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:34.792+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.792520+0000) 2020-02-05T22:16:34.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:34.793+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.794109+0000) 2020-02-05T22:16:34.796 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:34.794+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.795802+0000) 2020-02-05T22:16:34.797 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:34.795+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.796276+0000) 2020-02-05T22:16:34.797 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:34.795+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.796400+0000) 2020-02-05T22:16:34.799 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:34.797+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.797852+0000) 2020-02-05T22:16:34.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:34.797+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.798232+0000) 2020-02-05T22:16:34.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:34.797+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.798536+0000) 2020-02-05T22:16:34.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:34.798+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.798851+0000) 2020-02-05T22:16:34.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:34.797+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.798406+0000) 2020-02-05T22:16:34.801 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:34.798+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.799233+0000) 2020-02-05T22:16:34.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:34.801+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.802467+0000) 2020-02-05T22:16:34.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:34.801+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:34.802705+0000) 2020-02-05T22:16:34.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:34.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110426 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:35.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:35.917+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:35.918839+0000) 2020-02-05T22:16:35.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:35.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:35.925674+0000) 2020-02-05T22:16:37.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:37.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:37.065477+0000) 2020-02-05T22:16:37.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:37.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:37.065651+0000) 2020-02-05T22:16:39.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:39.454+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.455257+0000) 2020-02-05T22:16:39.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:39.637+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.638683+0000) 2020-02-05T22:16:39.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:39.810+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.811972+0000) 2020-02-05T22:16:39.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:39.811+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.812282+0000) 2020-02-05T22:16:39.814 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:39.812+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.812728+0000) 2020-02-05T22:16:39.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:39.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.813224+0000) 2020-02-05T22:16:39.815 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:39.813+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.814143+0000) 2020-02-05T22:16:39.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:39.813+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.815048+0000) 2020-02-05T22:16:39.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:39.815+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.815520+0000) 2020-02-05T22:16:39.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:39.814+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.815686+0000) 2020-02-05T22:16:39.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:39.815+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.816242+0000) 2020-02-05T22:16:39.818 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:39.816+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.816878+0000) 2020-02-05T22:16:39.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:39.816+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.817297+0000) 2020-02-05T22:16:39.819 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:39.817+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.817756+0000) 2020-02-05T22:16:39.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:39.816+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.817478+0000) 2020-02-05T22:16:39.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:39.821+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:39.822796+0000) 2020-02-05T22:16:39.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:39.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:40.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:40.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:40.062335+0000) 2020-02-05T22:16:40.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:40.297+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:40.297903+0000) 2020-02-05T22:16:40.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:40.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:40.314920+0000) 2020-02-05T22:16:40.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:40.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:40.314562+0000) 2020-02-05T22:16:42.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:42.309+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:42.309713+0000) 2020-02-05T22:16:42.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:42.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:42.705562+0000) 2020-02-05T22:16:43.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:43.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:43.823211+0000) 2020-02-05T22:16:43.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:43.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:43.823199+0000) 2020-02-05T22:16:44.833 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:44.831+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.831660+0000) 2020-02-05T22:16:44.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:44.831+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.832447+0000) 2020-02-05T22:16:44.834 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:44.832+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.833391+0000) 2020-02-05T22:16:44.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:44.833+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.834217+0000) 2020-02-05T22:16:44.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:44.834+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.835111+0000) 2020-02-05T22:16:44.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:44.834+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.835426+0000) 2020-02-05T22:16:44.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:44.835+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.836168+0000) 2020-02-05T22:16:44.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:44.835+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.836472+0000) 2020-02-05T22:16:44.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:44.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.836622+0000) 2020-02-05T22:16:44.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:44.836+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.836944+0000) 2020-02-05T22:16:44.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:44.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.837190+0000) 2020-02-05T22:16:44.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:44.837+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.837751+0000) 2020-02-05T22:16:44.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:44.837+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.837576+0000) 2020-02-05T22:16:44.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:44.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:44.842990+0000) 2020-02-05T22:16:44.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:44.845+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:45.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:45.918+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:45.919070+0000) 2020-02-05T22:16:45.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:45.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:45.925942+0000) 2020-02-05T22:16:47.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:47.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:47.065803+0000) 2020-02-05T22:16:47.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:47.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:47.065925+0000) 2020-02-05T22:16:49.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:49.454+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.455491+0000) 2020-02-05T22:16:49.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:49.646+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.647187+0000) 2020-02-05T22:16:49.853 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:49.850+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.851655+0000) 2020-02-05T22:16:49.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:49.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.851161+0000) 2020-02-05T22:16:49.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:49.850+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.851730+0000) 2020-02-05T22:16:49.854 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:49.851+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.852421+0000) 2020-02-05T22:16:49.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:49.852+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.853520+0000) 2020-02-05T22:16:49.855 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:49.853+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.854553+0000) 2020-02-05T22:16:49.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:49.854+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.855139+0000) 2020-02-05T22:16:49.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:49.854+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.855378+0000) 2020-02-05T22:16:49.857 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:49.854+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.855967+0000) 2020-02-05T22:16:49.858 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:49.855+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.856791+0000) 2020-02-05T22:16:49.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:49.855+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.856777+0000) 2020-02-05T22:16:49.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:49.856+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.857225+0000) 2020-02-05T22:16:49.859 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:49.855+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.857190+0000) 2020-02-05T22:16:49.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:49.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:49.862971+0000) 2020-02-05T22:16:49.864 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:49.862+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:50.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:50.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:50.062498+0000) 2020-02-05T22:16:50.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:50.297+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:50.298119+0000) 2020-02-05T22:16:50.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:50.314+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:50.314798+0000) 2020-02-05T22:16:50.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:50.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:50.315216+0000) 2020-02-05T22:16:52.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:52.308+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:52.309972+0000) 2020-02-05T22:16:52.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:52.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:52.705789+0000) 2020-02-05T22:16:53.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:53.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:53.823470+0000) 2020-02-05T22:16:53.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:53.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:53.823470+0000) 2020-02-05T22:16:54.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:54.867+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.869168+0000) 2020-02-05T22:16:54.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:54.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.871434+0000) 2020-02-05T22:16:54.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:54.870+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.871666+0000) 2020-02-05T22:16:54.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:54.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.872296+0000) 2020-02-05T22:16:54.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:54.871+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.872747+0000) 2020-02-05T22:16:54.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:54.871+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.872996+0000) 2020-02-05T22:16:54.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:54.872+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.873466+0000) 2020-02-05T22:16:54.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:54.872+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.873658+0000) 2020-02-05T22:16:54.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:54.872+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.873863+0000) 2020-02-05T22:16:54.876 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:54.873+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.874666+0000) 2020-02-05T22:16:54.876 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:54.873+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.874910+0000) 2020-02-05T22:16:54.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:54.873+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.874699+0000) 2020-02-05T22:16:54.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:54.874+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.875373+0000) 2020-02-05T22:16:54.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:54.881+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:54.882313+0000) 2020-02-05T22:16:54.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:54.880+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:16:55.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:55.918+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:55.919336+0000) 2020-02-05T22:16:55.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:55.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:55.926188+0000) 2020-02-05T22:16:57.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:57.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:57.066037+0000) 2020-02-05T22:16:57.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:57.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:57.066173+0000) 2020-02-05T22:16:57.478 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:16:57.484 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:16:59.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:59.455+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.455756+0000) 2020-02-05T22:16:59.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:59.655+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.655551+0000) 2020-02-05T22:16:59.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:16:59.886+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.887270+0000) 2020-02-05T22:16:59.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:16:59.886+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.887853+0000) 2020-02-05T22:16:59.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:16:59.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.888569+0000) 2020-02-05T22:16:59.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:59.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.889807+0000) 2020-02-05T22:16:59.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:16:59.888+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.889860+0000) 2020-02-05T22:16:59.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:16:59.888+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.890088+0000) 2020-02-05T22:16:59.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:59.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.890754+0000) 2020-02-05T22:16:59.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:16:59.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.891356+0000) 2020-02-05T22:16:59.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:16:59.890+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.891743+0000) 2020-02-05T22:16:59.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:16:59.891+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.892522+0000) 2020-02-05T22:16:59.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:16:59.892+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.893451+0000) 2020-02-05T22:16:59.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:16:59.892+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.893701+0000) 2020-02-05T22:16:59.895 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:16:59.892+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.894393+0000) 2020-02-05T22:16:59.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:16:59.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:16:59.896935+0000) 2020-02-05T22:16:59.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:16:59.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110624 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:00.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:00.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:00.062711+0000) 2020-02-05T22:17:00.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:00.296+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:00.298352+0000) 2020-02-05T22:17:00.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:00.313+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:00.315026+0000) 2020-02-05T22:17:00.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:00.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:00.315401+0000) 2020-02-05T22:17:02.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:02.308+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:02.310210+0000) 2020-02-05T22:17:02.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:02.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:02.706023+0000) 2020-02-05T22:17:03.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:03.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:03.823749+0000) 2020-02-05T22:17:03.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:03.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:03.823749+0000) 2020-02-05T22:17:04.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:04.904+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.905680+0000) 2020-02-05T22:17:04.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:04.905+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.906605+0000) 2020-02-05T22:17:04.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:04.906+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.907249+0000) 2020-02-05T22:17:04.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:04.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.908135+0000) 2020-02-05T22:17:04.909 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:04.907+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.908327+0000) 2020-02-05T22:17:04.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:04.908+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.909580+0000) 2020-02-05T22:17:04.911 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:04.908+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.909922+0000) 2020-02-05T22:17:04.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:04.909+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.910261+0000) 2020-02-05T22:17:04.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:04.909+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.910674+0000) 2020-02-05T22:17:04.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:04.909+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.911320+0000) 2020-02-05T22:17:04.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:04.910+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.911868+0000) 2020-02-05T22:17:04.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:04.910+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.912109+0000) 2020-02-05T22:17:04.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:04.910+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.912325+0000) 2020-02-05T22:17:04.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:04.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:04.914920+0000) 2020-02-05T22:17:04.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:04.917+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:05.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:05.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:05.919567+0000) 2020-02-05T22:17:05.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:05.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:05.926473+0000) 2020-02-05T22:17:07.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:07.064+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:07.066294+0000) 2020-02-05T22:17:07.068 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:07.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:07.066425+0000) 2020-02-05T22:17:09.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:09.455+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.456007+0000) 2020-02-05T22:17:09.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:09.663+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.663741+0000) 2020-02-05T22:17:09.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:09.923+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.924746+0000) 2020-02-05T22:17:09.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:09.924+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.925463+0000) 2020-02-05T22:17:09.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:09.924+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.926075+0000) 2020-02-05T22:17:09.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:09.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.926701+0000) 2020-02-05T22:17:09.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:09.926+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.927140+0000) 2020-02-05T22:17:09.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:09.926+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.927537+0000) 2020-02-05T22:17:09.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:09.926+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.927561+0000) 2020-02-05T22:17:09.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:09.926+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.927832+0000) 2020-02-05T22:17:09.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:09.926+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.928281+0000) 2020-02-05T22:17:09.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:09.926+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.928166+0000) 2020-02-05T22:17:09.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:09.927+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.928479+0000) 2020-02-05T22:17:09.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:09.927+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.928846+0000) 2020-02-05T22:17:09.931 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:09.928+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.929182+0000) 2020-02-05T22:17:09.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:09.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:09.935290+0000) 2020-02-05T22:17:09.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:09.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:10.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:10.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:10.062943+0000) 2020-02-05T22:17:10.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:10.297+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:10.298575+0000) 2020-02-05T22:17:10.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:10.315+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:10.315283+0000) 2020-02-05T22:17:10.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:10.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:10.315687+0000) 2020-02-05T22:17:12.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:12.309+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:12.310486+0000) 2020-02-05T22:17:12.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:12.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:12.706291+0000) 2020-02-05T22:17:13.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:13.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:13.823967+0000) 2020-02-05T22:17:13.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:13.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:13.823967+0000) 2020-02-05T22:17:14.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:14.942+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.943422+0000) 2020-02-05T22:17:14.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:14.942+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.943918+0000) 2020-02-05T22:17:14.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:14.943+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.944970+0000) 2020-02-05T22:17:14.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:14.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.945300+0000) 2020-02-05T22:17:14.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:14.944+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.946042+0000) 2020-02-05T22:17:14.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:14.945+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.946903+0000) 2020-02-05T22:17:14.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:14.946+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.948042+0000) 2020-02-05T22:17:14.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:14.946+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.947579+0000) 2020-02-05T22:17:14.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:14.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.948462+0000) 2020-02-05T22:17:14.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:14.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.949002+0000) 2020-02-05T22:17:14.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:14.948+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.949606+0000) 2020-02-05T22:17:14.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:14.948+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.949656+0000) 2020-02-05T22:17:14.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:14.948+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.949619+0000) 2020-02-05T22:17:14.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:14.952+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:14.953141+0000) 2020-02-05T22:17:14.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:14.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110743 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:15.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:15.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:15.919793+0000) 2020-02-05T22:17:15.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:15.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:15.926709+0000) 2020-02-05T22:17:17.068 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:17.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:17.066548+0000) 2020-02-05T22:17:17.068 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:17.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:17.066685+0000) 2020-02-05T22:17:19.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:19.456+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.456231+0000) 2020-02-05T22:17:19.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:19.669+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.670795+0000) 2020-02-05T22:17:19.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:19.961+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.962917+0000) 2020-02-05T22:17:19.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:19.962+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.963678+0000) 2020-02-05T22:17:19.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:19.963+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.964499+0000) 2020-02-05T22:17:19.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:19.964+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.965488+0000) 2020-02-05T22:17:19.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:19.965+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.967000+0000) 2020-02-05T22:17:19.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:19.966+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.967600+0000) 2020-02-05T22:17:19.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:19.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.967979+0000) 2020-02-05T22:17:19.970 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:19.966+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.968321+0000) 2020-02-05T22:17:19.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:19.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.968651+0000) 2020-02-05T22:17:19.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:19.967+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.969356+0000) 2020-02-05T22:17:19.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:19.968+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.969693+0000) 2020-02-05T22:17:19.971 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:19.968+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.969793+0000) 2020-02-05T22:17:19.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:19.969+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.970144+0000) 2020-02-05T22:17:19.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:19.973+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:19.974959+0000) 2020-02-05T22:17:19.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:19.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:20.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:20.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:20.063183+0000) 2020-02-05T22:17:20.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:20.298+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:20.298800+0000) 2020-02-05T22:17:20.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:20.314+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:20.315485+0000) 2020-02-05T22:17:20.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:20.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:20.315892+0000) 2020-02-05T22:17:22.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:22.310+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:22.310790+0000) 2020-02-05T22:17:22.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:22.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:22.706544+0000) 2020-02-05T22:17:23.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:23.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:23.824172+0000) 2020-02-05T22:17:23.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:23.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:23.824266+0000) 2020-02-05T22:17:24.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:24.981+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.982174+0000) 2020-02-05T22:17:24.984 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:24.981+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.982832+0000) 2020-02-05T22:17:24.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:24.982+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.983436+0000) 2020-02-05T22:17:24.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:24.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.985263+0000) 2020-02-05T22:17:24.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:24.984+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.985914+0000) 2020-02-05T22:17:24.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:24.984+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.985633+0000) 2020-02-05T22:17:24.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:24.985+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.986643+0000) 2020-02-05T22:17:24.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:24.984+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.986366+0000) 2020-02-05T22:17:24.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:24.985+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.987058+0000) 2020-02-05T22:17:24.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:24.986+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.987781+0000) 2020-02-05T22:17:24.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:24.987+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.988206+0000) 2020-02-05T22:17:24.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:24.986+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.988239+0000) 2020-02-05T22:17:24.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:24.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.992087+0000) 2020-02-05T22:17:24.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:24.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:24.992402+0000) 2020-02-05T22:17:24.997 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:24.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:25.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:25.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:25.920004+0000) 2020-02-05T22:17:25.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:25.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:25.926946+0000) 2020-02-05T22:17:27.069 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:27.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:27.066856+0000) 2020-02-05T22:17:27.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:27.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:27.066922+0000) 2020-02-05T22:17:27.624 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:17:27.630 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:17:29.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:29.455+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:29.456492+0000) 2020-02-05T22:17:29.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:29.681+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:29.682314+0000) 2020-02-05T22:17:30.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:30.000+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.001663+0000) 2020-02-05T22:17:30.003 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:30.001+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.002183+0000) 2020-02-05T22:17:30.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:30.001+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.002674+0000) 2020-02-05T22:17:30.004 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:30.002+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.003525+0000) 2020-02-05T22:17:30.006 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:30.003+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.004609+0000) 2020-02-05T22:17:30.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:30.004+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.005846+0000) 2020-02-05T22:17:30.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:30.004+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.005680+0000) 2020-02-05T22:17:30.007 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:30.005+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.006339+0000) 2020-02-05T22:17:30.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:30.004+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.006361+0000) 2020-02-05T22:17:30.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:30.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.006978+0000) 2020-02-05T22:17:30.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:30.006+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.007731+0000) 2020-02-05T22:17:30.009 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:30.006+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.007878+0000) 2020-02-05T22:17:30.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:30.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.007406+0000) 2020-02-05T22:17:30.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:30.011+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.012836+0000) 2020-02-05T22:17:30.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:30.014+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:30.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:30.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.063372+0000) 2020-02-05T22:17:30.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:30.298+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.299013+0000) 2020-02-05T22:17:30.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:30.314+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.315711+0000) 2020-02-05T22:17:30.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:30.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:30.316066+0000) 2020-02-05T22:17:32.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:32.310+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:32.311032+0000) 2020-02-05T22:17:32.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:32.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:32.706781+0000) 2020-02-05T22:17:33.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:33.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:33.824423+0000) 2020-02-05T22:17:33.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:33.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:33.824510+0000) 2020-02-05T22:17:35.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:35.021+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.022481+0000) 2020-02-05T22:17:35.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:35.022+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.023188+0000) 2020-02-05T22:17:35.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:35.022+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.023949+0000) 2020-02-05T22:17:35.025 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:35.023+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.024369+0000) 2020-02-05T22:17:35.026 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:35.023+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.024451+0000) 2020-02-05T22:17:35.030 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:35.025+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.026509+0000) 2020-02-05T22:17:35.030 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:35.025+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.027224+0000) 2020-02-05T22:17:35.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:35.026+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.027422+0000) 2020-02-05T22:17:35.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:35.026+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.027943+0000) 2020-02-05T22:17:35.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:35.024+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.025403+0000) 2020-02-05T22:17:35.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:35.024+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.026023+0000) 2020-02-05T22:17:35.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:35.025+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.026186+0000) 2020-02-05T22:17:35.031 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:35.027+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.028322+0000) 2020-02-05T22:17:35.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:35.030+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.031895+0000) 2020-02-05T22:17:35.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:35.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:35.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:35.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.920211+0000) 2020-02-05T22:17:35.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:35.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:35.927160+0000) 2020-02-05T22:17:37.068 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:37.065+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:37.067091+0000) 2020-02-05T22:17:37.068 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:37.065+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:37.067200+0000) 2020-02-05T22:17:39.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:39.456+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:39.456751+0000) 2020-02-05T22:17:39.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:39.689+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:39.690668+0000) 2020-02-05T22:17:40.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:40.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.041228+0000) 2020-02-05T22:17:40.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:40.041+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.042652+0000) 2020-02-05T22:17:40.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:40.042+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.043624+0000) 2020-02-05T22:17:40.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:40.043+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.043526+0000) 2020-02-05T22:17:40.045 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:40.043+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.044245+0000) 2020-02-05T22:17:40.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:40.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.045429+0000) 2020-02-05T22:17:40.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:40.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.046156+0000) 2020-02-05T22:17:40.047 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:40.045+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.046426+0000) 2020-02-05T22:17:40.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:40.045+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.046048+0000) 2020-02-05T22:17:40.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:40.044+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.045237+0000) 2020-02-05T22:17:40.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:40.045+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.047026+0000) 2020-02-05T22:17:40.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:40.045+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.047341+0000) 2020-02-05T22:17:40.049 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:40.046+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.047621+0000) 2020-02-05T22:17:40.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:40.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.052798+0000) 2020-02-05T22:17:40.055 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:40.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:40.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:40.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.063612+0000) 2020-02-05T22:17:40.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:40.298+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.299217+0000) 2020-02-05T22:17:40.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:40.315+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.315920+0000) 2020-02-05T22:17:40.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:40.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:40.316290+0000) 2020-02-05T22:17:42.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:42.310+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:42.311299+0000) 2020-02-05T22:17:42.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:42.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:42.706984+0000) 2020-02-05T22:17:43.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:43.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:43.824653+0000) 2020-02-05T22:17:43.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:43.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:43.824794+0000) 2020-02-05T22:17:45.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:45.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.060624+0000) 2020-02-05T22:17:45.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:45.061+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.062132+0000) 2020-02-05T22:17:45.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:45.061+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.061872+0000) 2020-02-05T22:17:45.064 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:45.061+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.062589+0000) 2020-02-05T22:17:45.064 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:45.062+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.063642+0000) 2020-02-05T22:17:45.065 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:45.063+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.063419+0000) 2020-02-05T22:17:45.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:45.063+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.064306+0000) 2020-02-05T22:17:45.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:45.063+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.064194+0000) 2020-02-05T22:17:45.066 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:45.064+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.064560+0000) 2020-02-05T22:17:45.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:45.064+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.065150+0000) 2020-02-05T22:17:45.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:45.065+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.066028+0000) 2020-02-05T22:17:45.067 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:45.065+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.066431+0000) 2020-02-05T22:17:45.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:45.065+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.066718+0000) 2020-02-05T22:17:45.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:45.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 110982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:45.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:45.072+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.073415+0000) 2020-02-05T22:17:45.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:45.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.920417+0000) 2020-02-05T22:17:45.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:45.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:45.927388+0000) 2020-02-05T22:17:47.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:47.066+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:47.067347+0000) 2020-02-05T22:17:47.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:47.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:47.067478+0000) 2020-02-05T22:17:49.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:49.456+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:49.456992+0000) 2020-02-05T22:17:49.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:49.701+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:49.702465+0000) 2020-02-05T22:17:50.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:50.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.063824+0000) 2020-02-05T22:17:50.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:50.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.078388+0000) 2020-02-05T22:17:50.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:50.077+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.079054+0000) 2020-02-05T22:17:50.081 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:50.079+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.080361+0000) 2020-02-05T22:17:50.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:50.080+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.080777+0000) 2020-02-05T22:17:50.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:50.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.081614+0000) 2020-02-05T22:17:50.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:50.081+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.082003+0000) 2020-02-05T22:17:50.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:50.081+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.082284+0000) 2020-02-05T22:17:50.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:50.081+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.083027+0000) 2020-02-05T22:17:50.084 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:50.082+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.083816+0000) 2020-02-05T22:17:50.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:50.083+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.084101+0000) 2020-02-05T22:17:50.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:50.084+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.084714+0000) 2020-02-05T22:17:50.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:50.083+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.084595+0000) 2020-02-05T22:17:50.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:50.084+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.084929+0000) 2020-02-05T22:17:50.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:50.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.089354+0000) 2020-02-05T22:17:50.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:50.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:50.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:50.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.299450+0000) 2020-02-05T22:17:50.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:50.315+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.316156+0000) 2020-02-05T22:17:50.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:50.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:50.316505+0000) 2020-02-05T22:17:52.313 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:52.311+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:52.311677+0000) 2020-02-05T22:17:52.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:52.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:52.707217+0000) 2020-02-05T22:17:53.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:53.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:53.824919+0000) 2020-02-05T22:17:53.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:53.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:53.825128+0000) 2020-02-05T22:17:55.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:17:55.097+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.097517+0000) 2020-02-05T22:17:55.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:17:55.098+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.099279+0000) 2020-02-05T22:17:55.101 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:17:55.098+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.099654+0000) 2020-02-05T22:17:55.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:17:55.099+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.100361+0000) 2020-02-05T22:17:55.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:17:55.099+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.100542+0000) 2020-02-05T22:17:55.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:55.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.101062+0000) 2020-02-05T22:17:55.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:55.100+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.101373+0000) 2020-02-05T22:17:55.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:17:55.100+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.102023+0000) 2020-02-05T22:17:55.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:55.100+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.101904+0000) 2020-02-05T22:17:55.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:55.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.102012+0000) 2020-02-05T22:17:55.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:55.102+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.102984+0000) 2020-02-05T22:17:55.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:55.103+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.103750+0000) 2020-02-05T22:17:55.105 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:17:55.103+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.104103+0000) 2020-02-05T22:17:55.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:55.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.107120+0000) 2020-02-05T22:17:55.112 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:17:55.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:17:55.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:17:55.919+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.920641+0000) 2020-02-05T22:17:55.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:17:55.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:55.927704+0000) 2020-02-05T22:17:57.069 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:17:57.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:57.067633+0000) 2020-02-05T22:17:57.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:17:57.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:57.067741+0000) 2020-02-05T22:17:57.820 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:17:57.827 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:17:59.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:59.456+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:59.457285+0000) 2020-02-05T22:17:59.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:17:59.710+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:17:59.710710+0000) 2020-02-05T22:18:00.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:00.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.064122+0000) 2020-02-05T22:18:00.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:00.116+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.117132+0000) 2020-02-05T22:18:00.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:00.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.117961+0000) 2020-02-05T22:18:00.120 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:00.117+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.118767+0000) 2020-02-05T22:18:00.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:00.118+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.119205+0000) 2020-02-05T22:18:00.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:00.118+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.118567+0000) 2020-02-05T22:18:00.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:00.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.119672+0000) 2020-02-05T22:18:00.124 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:00.118+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.120027+0000) 2020-02-05T22:18:00.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:00.120+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.121265+0000) 2020-02-05T22:18:00.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:00.120+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.120457+0000) 2020-02-05T22:18:00.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:00.120+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.121124+0000) 2020-02-05T22:18:00.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:00.120+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.121810+0000) 2020-02-05T22:18:00.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:00.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.127058+0000) 2020-02-05T22:18:00.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:00.122+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.122497+0000) 2020-02-05T22:18:00.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:00.122+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.122532+0000) 2020-02-05T22:18:00.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:00.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:00.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:00.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.299683+0000) 2020-02-05T22:18:00.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:00.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.316366+0000) 2020-02-05T22:18:00.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:00.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:00.316708+0000) 2020-02-05T22:18:02.313 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:02.311+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:02.312017+0000) 2020-02-05T22:18:02.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:02.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:02.707527+0000) 2020-02-05T22:18:03.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:03.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:03.825146+0000) 2020-02-05T22:18:03.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:03.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:03.825359+0000) 2020-02-05T22:18:05.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:05.136+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.137185+0000) 2020-02-05T22:18:05.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:05.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.138716+0000) 2020-02-05T22:18:05.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:05.137+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.138377+0000) 2020-02-05T22:18:05.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:05.138+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.139267+0000) 2020-02-05T22:18:05.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:05.139+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.140255+0000) 2020-02-05T22:18:05.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:05.140+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.141015+0000) 2020-02-05T22:18:05.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:05.139+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.140743+0000) 2020-02-05T22:18:05.142 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:05.140+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.141280+0000) 2020-02-05T22:18:05.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:05.141+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.141941+0000) 2020-02-05T22:18:05.143 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:05.141+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.141895+0000) 2020-02-05T22:18:05.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:05.141+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.142392+0000) 2020-02-05T22:18:05.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:05.141+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.142645+0000) 2020-02-05T22:18:05.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:05.142+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.142931+0000) 2020-02-05T22:18:05.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:05.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.147726+0000) 2020-02-05T22:18:05.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:05.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:05.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:05.920+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.920830+0000) 2020-02-05T22:18:05.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:05.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:05.927922+0000) 2020-02-05T22:18:07.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:07.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:07.067870+0000) 2020-02-05T22:18:07.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:07.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:07.067961+0000) 2020-02-05T22:18:09.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:09.457+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:09.457538+0000) 2020-02-05T22:18:09.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:09.718+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:09.718517+0000) 2020-02-05T22:18:10.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:10.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.064318+0000) 2020-02-05T22:18:10.158 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:10.156+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.157473+0000) 2020-02-05T22:18:10.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:10.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.158529+0000) 2020-02-05T22:18:10.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:10.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.158969+0000) 2020-02-05T22:18:10.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:10.158+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.159248+0000) 2020-02-05T22:18:10.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:10.159+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.159611+0000) 2020-02-05T22:18:10.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:10.159+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.159855+0000) 2020-02-05T22:18:10.161 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:10.159+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.160451+0000) 2020-02-05T22:18:10.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:10.160+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.161066+0000) 2020-02-05T22:18:10.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:10.162+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.162807+0000) 2020-02-05T22:18:10.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:10.162+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.163309+0000) 2020-02-05T22:18:10.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:10.160+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.161070+0000) 2020-02-05T22:18:10.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:10.161+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.162422+0000) 2020-02-05T22:18:10.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:10.161+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.162539+0000) 2020-02-05T22:18:10.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:10.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.167059+0000) 2020-02-05T22:18:10.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:10.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:10.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:10.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.299920+0000) 2020-02-05T22:18:10.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:10.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.316585+0000) 2020-02-05T22:18:10.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:10.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:10.316999+0000) 2020-02-05T22:18:12.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:12.312+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:12.312276+0000) 2020-02-05T22:18:12.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:12.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:12.707765+0000) 2020-02-05T22:18:13.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:13.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:13.825329+0000) 2020-02-05T22:18:13.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:13.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:13.825546+0000) 2020-02-05T22:18:15.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:15.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.176510+0000) 2020-02-05T22:18:15.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:15.176+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.177118+0000) 2020-02-05T22:18:15.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:15.177+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.178469+0000) 2020-02-05T22:18:15.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:15.178+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.179024+0000) 2020-02-05T22:18:15.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:15.179+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.180129+0000) 2020-02-05T22:18:15.182 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:15.179+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.180632+0000) 2020-02-05T22:18:15.183 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:15.180+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.181270+0000) 2020-02-05T22:18:15.183 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:15.181+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.181687+0000) 2020-02-05T22:18:15.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:15.181+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.182583+0000) 2020-02-05T22:18:15.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:15.182+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.182867+0000) 2020-02-05T22:18:15.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:15.182+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.183260+0000) 2020-02-05T22:18:15.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:15.183+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.184152+0000) 2020-02-05T22:18:15.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:15.183+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.184241+0000) 2020-02-05T22:18:15.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:15.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.189264+0000) 2020-02-05T22:18:15.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:15.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:15.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:15.920+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.921045+0000) 2020-02-05T22:18:15.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:15.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:15.928149+0000) 2020-02-05T22:18:17.069 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:17.068+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:17.068134+0000) 2020-02-05T22:18:17.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:17.068+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:17.068233+0000) 2020-02-05T22:18:19.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:19.456+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:19.457800+0000) 2020-02-05T22:18:19.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:19.725+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:19.726693+0000) 2020-02-05T22:18:20.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:20.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.064558+0000) 2020-02-05T22:18:20.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:20.196+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.197456+0000) 2020-02-05T22:18:20.199 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:20.196+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.197938+0000) 2020-02-05T22:18:20.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:20.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.198514+0000) 2020-02-05T22:18:20.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:20.197+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.199026+0000) 2020-02-05T22:18:20.201 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:20.198+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.199234+0000) 2020-02-05T22:18:20.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:20.199+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.200289+0000) 2020-02-05T22:18:20.202 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:20.199+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.200926+0000) 2020-02-05T22:18:20.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:20.200+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.201488+0000) 2020-02-05T22:18:20.203 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:20.200+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.201964+0000) 2020-02-05T22:18:20.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:20.201+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.202729+0000) 2020-02-05T22:18:20.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:20.201+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.203016+0000) 2020-02-05T22:18:20.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:20.202+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.203541+0000) 2020-02-05T22:18:20.205 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:20.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.203670+0000) 2020-02-05T22:18:20.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:20.207+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.208387+0000) 2020-02-05T22:18:20.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:20.209+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:20.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:20.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.300161+0000) 2020-02-05T22:18:20.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:20.315+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.316768+0000) 2020-02-05T22:18:20.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:20.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:20.317210+0000) 2020-02-05T22:18:22.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:22.311+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:22.312542+0000) 2020-02-05T22:18:22.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:22.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:22.708004+0000) 2020-02-05T22:18:23.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:23.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:23.825608+0000) 2020-02-05T22:18:23.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:23.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:23.825821+0000) 2020-02-05T22:18:25.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:25.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.217506+0000) 2020-02-05T22:18:25.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:25.217+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.218837+0000) 2020-02-05T22:18:25.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:25.218+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.219556+0000) 2020-02-05T22:18:25.221 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:25.219+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.220155+0000) 2020-02-05T22:18:25.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:25.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.220706+0000) 2020-02-05T22:18:25.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:25.219+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.221248+0000) 2020-02-05T22:18:25.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:25.221+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.222339+0000) 2020-02-05T22:18:25.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:25.221+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.222992+0000) 2020-02-05T22:18:25.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:25.220+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.221821+0000) 2020-02-05T22:18:25.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:25.220+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.221737+0000) 2020-02-05T22:18:25.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:25.220+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.222208+0000) 2020-02-05T22:18:25.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:25.221+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.222998+0000) 2020-02-05T22:18:25.226 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:25.222+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.223369+0000) 2020-02-05T22:18:25.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:25.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.228887+0000) 2020-02-05T22:18:25.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:25.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:25.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:25.920+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.921311+0000) 2020-02-05T22:18:25.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:25.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:25.928498+0000) 2020-02-05T22:18:27.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:27.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:27.068466+0000) 2020-02-05T22:18:27.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:27.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:27.068500+0000) 2020-02-05T22:18:28.026 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:18:28.032 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:18:29.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:29.457+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:29.458070+0000) 2020-02-05T22:18:29.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:29.734+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:29.735159+0000) 2020-02-05T22:18:30.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:30.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.064787+0000) 2020-02-05T22:18:30.238 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:30.235+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.236763+0000) 2020-02-05T22:18:30.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:30.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.237418+0000) 2020-02-05T22:18:30.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:30.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.237964+0000) 2020-02-05T22:18:30.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:30.237+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.238310+0000) 2020-02-05T22:18:30.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:30.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.238958+0000) 2020-02-05T22:18:30.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:30.238+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.239891+0000) 2020-02-05T22:18:30.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:30.239+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.240387+0000) 2020-02-05T22:18:30.242 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:30.238+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.240159+0000) 2020-02-05T22:18:30.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:30.240+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.241438+0000) 2020-02-05T22:18:30.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:30.240+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.241947+0000) 2020-02-05T22:18:30.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:30.240+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.242333+0000) 2020-02-05T22:18:30.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:30.241+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.242507+0000) 2020-02-05T22:18:30.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:30.241+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.242999+0000) 2020-02-05T22:18:30.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:30.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.247652+0000) 2020-02-05T22:18:30.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:30.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:30.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:30.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.300393+0000) 2020-02-05T22:18:30.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:30.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.316996+0000) 2020-02-05T22:18:30.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:30.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:30.317380+0000) 2020-02-05T22:18:32.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:32.311+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:32.312802+0000) 2020-02-05T22:18:32.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:32.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:32.708274+0000) 2020-02-05T22:18:33.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:33.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:33.825853+0000) 2020-02-05T22:18:33.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:33.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:33.826029+0000) 2020-02-05T22:18:35.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:35.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.255835+0000) 2020-02-05T22:18:35.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:35.254+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.256326+0000) 2020-02-05T22:18:35.258 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:35.256+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.257617+0000) 2020-02-05T22:18:35.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:35.257+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.258530+0000) 2020-02-05T22:18:35.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:35.257+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.258893+0000) 2020-02-05T22:18:35.261 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:35.258+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.259954+0000) 2020-02-05T22:18:35.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:35.258+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.259667+0000) 2020-02-05T22:18:35.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:35.259+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.260661+0000) 2020-02-05T22:18:35.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:35.259+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.260821+0000) 2020-02-05T22:18:35.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:35.260+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.261160+0000) 2020-02-05T22:18:35.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:35.260+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.261532+0000) 2020-02-05T22:18:35.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:35.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.261681+0000) 2020-02-05T22:18:35.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:35.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.262042+0000) 2020-02-05T22:18:35.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:35.266+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.267374+0000) 2020-02-05T22:18:35.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:35.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:35.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:35.920+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.921511+0000) 2020-02-05T22:18:35.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:35.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:35.928765+0000) 2020-02-05T22:18:37.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:37.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:37.068731+0000) 2020-02-05T22:18:37.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:37.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:37.068730+0000) 2020-02-05T22:18:39.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:39.457+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:39.458366+0000) 2020-02-05T22:18:39.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:39.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:39.747471+0000) 2020-02-05T22:18:40.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:40.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.065051+0000) 2020-02-05T22:18:40.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:40.274+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.275709+0000) 2020-02-05T22:18:40.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:40.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.276626+0000) 2020-02-05T22:18:40.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:40.276+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.277772+0000) 2020-02-05T22:18:40.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:40.276+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.277701+0000) 2020-02-05T22:18:40.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:40.277+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.278636+0000) 2020-02-05T22:18:40.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:40.278+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.279219+0000) 2020-02-05T22:18:40.280 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:40.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.279565+0000) 2020-02-05T22:18:40.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:40.279+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.280275+0000) 2020-02-05T22:18:40.282 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:40.279+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.280921+0000) 2020-02-05T22:18:40.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:40.280+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.281423+0000) 2020-02-05T22:18:40.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:40.280+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.281529+0000) 2020-02-05T22:18:40.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:40.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.281927+0000) 2020-02-05T22:18:40.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:40.280+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.282128+0000) 2020-02-05T22:18:40.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:40.286+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.287141+0000) 2020-02-05T22:18:40.291 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:40.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:40.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:40.299+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.300579+0000) 2020-02-05T22:18:40.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:40.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.317217+0000) 2020-02-05T22:18:40.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:40.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:40.317669+0000) 2020-02-05T22:18:42.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:42.312+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:42.313063+0000) 2020-02-05T22:18:42.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:42.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:42.708513+0000) 2020-02-05T22:18:43.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:43.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:43.826073+0000) 2020-02-05T22:18:43.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:43.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:43.826256+0000) 2020-02-05T22:18:45.300 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:45.297+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.298338+0000) 2020-02-05T22:18:45.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:45.295+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.297014+0000) 2020-02-05T22:18:45.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:45.296+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.297534+0000) 2020-02-05T22:18:45.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:45.297+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.299188+0000) 2020-02-05T22:18:45.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:45.298+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.299772+0000) 2020-02-05T22:18:45.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:45.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.300196+0000) 2020-02-05T22:18:45.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:45.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.300691+0000) 2020-02-05T22:18:45.302 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:45.299+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.300971+0000) 2020-02-05T22:18:45.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:45.300+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.301667+0000) 2020-02-05T22:18:45.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:45.300+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.301667+0000) 2020-02-05T22:18:45.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:45.301+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.302132+0000) 2020-02-05T22:18:45.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:45.301+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.302638+0000) 2020-02-05T22:18:45.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:45.300+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.302158+0000) 2020-02-05T22:18:45.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:45.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.308528+0000) 2020-02-05T22:18:45.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:45.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:45.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:45.921+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.921722+0000) 2020-02-05T22:18:45.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:45.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:45.928987+0000) 2020-02-05T22:18:47.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:47.068+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:47.068933+0000) 2020-02-05T22:18:47.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:47.068+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:47.068928+0000) 2020-02-05T22:18:49.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:49.457+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:49.458625+0000) 2020-02-05T22:18:49.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:49.755+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:49.756145+0000) 2020-02-05T22:18:50.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:50.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.065381+0000) 2020-02-05T22:18:50.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:50.300+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.300790+0000) 2020-02-05T22:18:50.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:50.314+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.315666+0000) 2020-02-05T22:18:50.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:50.314+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.315635+0000) 2020-02-05T22:18:50.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:50.315+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.316420+0000) 2020-02-05T22:18:50.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:50.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.317376+0000) 2020-02-05T22:18:50.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:50.316+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.317750+0000) 2020-02-05T22:18:50.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:50.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.318395+0000) 2020-02-05T22:18:50.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:50.317+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.318562+0000) 2020-02-05T22:18:50.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:50.318+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.319406+0000) 2020-02-05T22:18:50.321 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:50.318+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.320136+0000) 2020-02-05T22:18:50.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:50.318+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.320332+0000) 2020-02-05T22:18:50.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:50.319+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.320797+0000) 2020-02-05T22:18:50.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:50.319+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.321016+0000) 2020-02-05T22:18:50.323 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:50.320+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.321932+0000) 2020-02-05T22:18:50.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:50.320+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.322417+0000) 2020-02-05T22:18:50.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:50.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:50.325001+0000) 2020-02-05T22:18:50.328 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:50.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:52.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:52.312+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:52.313325+0000) 2020-02-05T22:18:52.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:52.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:52.708750+0000) 2020-02-05T22:18:53.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:53.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:53.826358+0000) 2020-02-05T22:18:53.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:53.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:53.826474+0000) 2020-02-05T22:18:55.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:18:55.332+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.333759+0000) 2020-02-05T22:18:55.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:55.332+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.334245+0000) 2020-02-05T22:18:55.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:18:55.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.334754+0000) 2020-02-05T22:18:55.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:18:55.334+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.335264+0000) 2020-02-05T22:18:55.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:55.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.335877+0000) 2020-02-05T22:18:55.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:55.335+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.336475+0000) 2020-02-05T22:18:55.338 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:18:55.335+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.336509+0000) 2020-02-05T22:18:55.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:18:55.335+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.336916+0000) 2020-02-05T22:18:55.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:55.335+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.337122+0000) 2020-02-05T22:18:55.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:55.336+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.337844+0000) 2020-02-05T22:18:55.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:55.336+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.338376+0000) 2020-02-05T22:18:55.341 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:18:55.337+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.338406+0000) 2020-02-05T22:18:55.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:55.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.341504+0000) 2020-02-05T22:18:55.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:55.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.341779+0000) 2020-02-05T22:18:55.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:18:55.342+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111549 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:18:55.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:18:55.921+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.921937+0000) 2020-02-05T22:18:55.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:18:55.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:55.929206+0000) 2020-02-05T22:18:57.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:18:57.066+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:57.068229+0000) 2020-02-05T22:18:57.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:18:57.067+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:57.069191+0000) 2020-02-05T22:18:57.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:18:57.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:57.069176+0000) 2020-02-05T22:18:58.167 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:18:58.173 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:18:59.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:59.458+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:59.458874+0000) 2020-02-05T22:18:59.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:18:59.764+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:18:59.764717+0000) 2020-02-05T22:19:00.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:00.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.065644+0000) 2020-02-05T22:19:00.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:00.300+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.301022+0000) 2020-02-05T22:19:00.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:00.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.317799+0000) 2020-02-05T22:19:00.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:00.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.348990+0000) 2020-02-05T22:19:00.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:00.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.349637+0000) 2020-02-05T22:19:00.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:00.348+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.350062+0000) 2020-02-05T22:19:00.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:00.349+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.351329+0000) 2020-02-05T22:19:00.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:00.350+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.351719+0000) 2020-02-05T22:19:00.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:00.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.352300+0000) 2020-02-05T22:19:00.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:00.351+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.352677+0000) 2020-02-05T22:19:00.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:00.351+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.352993+0000) 2020-02-05T22:19:00.355 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:00.352+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.353991+0000) 2020-02-05T22:19:00.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:00.352+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.354164+0000) 2020-02-05T22:19:00.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:00.352+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.354270+0000) 2020-02-05T22:19:00.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:00.353+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.355040+0000) 2020-02-05T22:19:00.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:00.354+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.355321+0000) 2020-02-05T22:19:00.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:00.358+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:00.359995+0000) 2020-02-05T22:19:00.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:00.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111591 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:02.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:02.313+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:02.313663+0000) 2020-02-05T22:19:02.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:02.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:02.709016+0000) 2020-02-05T22:19:03.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:03.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:03.826582+0000) 2020-02-05T22:19:03.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:03.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:03.826676+0000) 2020-02-05T22:19:05.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:05.369+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.369981+0000) 2020-02-05T22:19:05.372 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:05.371+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.371431+0000) 2020-02-05T22:19:05.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:05.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.372361+0000) 2020-02-05T22:19:05.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:05.371+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.372720+0000) 2020-02-05T22:19:05.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:05.371+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.373377+0000) 2020-02-05T22:19:05.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:05.374+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.374414+0000) 2020-02-05T22:19:05.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:05.374+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.374517+0000) 2020-02-05T22:19:05.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:05.375+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.375150+0000) 2020-02-05T22:19:05.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:05.374+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.375472+0000) 2020-02-05T22:19:05.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:05.374+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.375624+0000) 2020-02-05T22:19:05.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:05.374+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.375868+0000) 2020-02-05T22:19:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:05.378+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.379949+0000) 2020-02-05T22:19:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:05.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.380240+0000) 2020-02-05T22:19:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:05.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.380486+0000) 2020-02-05T22:19:05.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:05.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111630 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:05.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:05.922+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.922166+0000) 2020-02-05T22:19:05.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:05.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:05.929465+0000) 2020-02-05T22:19:07.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:07.067+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:07.068484+0000) 2020-02-05T22:19:07.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:07.068+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:07.069472+0000) 2020-02-05T22:19:07.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:07.067+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:07.069387+0000) 2020-02-05T22:19:09.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:09.459+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:09.459138+0000) 2020-02-05T22:19:09.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:09.774+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:09.775139+0000) 2020-02-05T22:19:10.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:10.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.065876+0000) 2020-02-05T22:19:10.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:10.300+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.301278+0000) 2020-02-05T22:19:10.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:10.316+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.318046+0000) 2020-02-05T22:19:10.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:10.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.388065+0000) 2020-02-05T22:19:10.390 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:10.388+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.389142+0000) 2020-02-05T22:19:10.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:10.389+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.390018+0000) 2020-02-05T22:19:10.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:10.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.390408+0000) 2020-02-05T22:19:10.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:10.390+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.390891+0000) 2020-02-05T22:19:10.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:10.390+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.391537+0000) 2020-02-05T22:19:10.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:10.391+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.391645+0000) 2020-02-05T22:19:10.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:10.391+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.392377+0000) 2020-02-05T22:19:10.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:10.391+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.392806+0000) 2020-02-05T22:19:10.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:10.392+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.392662+0000) 2020-02-05T22:19:10.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:10.392+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.393393+0000) 2020-02-05T22:19:10.395 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:10.393+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.393616+0000) 2020-02-05T22:19:10.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:10.393+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.393959+0000) 2020-02-05T22:19:10.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:10.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:10.399494+0000) 2020-02-05T22:19:10.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:10.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111673 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:12.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:12.313+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:12.313920+0000) 2020-02-05T22:19:12.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:12.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:12.709283+0000) 2020-02-05T22:19:13.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:13.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:13.826800+0000) 2020-02-05T22:19:13.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:13.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:13.826918+0000) 2020-02-05T22:19:15.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:15.407+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.407726+0000) 2020-02-05T22:19:15.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:15.408+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.409189+0000) 2020-02-05T22:19:15.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:15.408+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.409728+0000) 2020-02-05T22:19:15.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:15.409+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.410393+0000) 2020-02-05T22:19:15.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:15.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.411256+0000) 2020-02-05T22:19:15.412 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:15.410+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.411492+0000) 2020-02-05T22:19:15.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:15.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.412006+0000) 2020-02-05T22:19:15.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:15.412+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.412702+0000) 2020-02-05T22:19:15.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:15.412+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.413056+0000) 2020-02-05T22:19:15.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:15.412+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.413345+0000) 2020-02-05T22:19:15.415 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:15.412+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.413738+0000) 2020-02-05T22:19:15.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:15.413+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.414200+0000) 2020-02-05T22:19:15.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:15.413+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.414217+0000) 2020-02-05T22:19:15.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:15.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.420161+0000) 2020-02-05T22:19:15.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:15.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:15.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:15.921+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.922374+0000) 2020-02-05T22:19:15.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:15.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:15.929693+0000) 2020-02-05T22:19:17.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:17.068+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:17.068662+0000) 2020-02-05T22:19:17.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:17.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:17.069602+0000) 2020-02-05T22:19:17.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:17.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:17.069667+0000) 2020-02-05T22:19:19.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:19.458+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:19.459366+0000) 2020-02-05T22:19:19.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:19.785+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:19.786586+0000) 2020-02-05T22:19:20.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:20.065+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.066133+0000) 2020-02-05T22:19:20.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:20.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.301481+0000) 2020-02-05T22:19:20.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:20.317+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.318317+0000) 2020-02-05T22:19:20.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:20.425+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.426181+0000) 2020-02-05T22:19:20.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:20.426+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.427766+0000) 2020-02-05T22:19:20.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:20.428+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.428706+0000) 2020-02-05T22:19:20.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:20.428+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.429229+0000) 2020-02-05T22:19:20.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:20.430+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.430519+0000) 2020-02-05T22:19:20.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:20.430+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.431182+0000) 2020-02-05T22:19:20.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:20.430+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.431808+0000) 2020-02-05T22:19:20.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:20.430+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.431388+0000) 2020-02-05T22:19:20.434 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:20.431+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.432193+0000) 2020-02-05T22:19:20.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:20.431+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.432620+0000) 2020-02-05T22:19:20.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:20.431+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.432561+0000) 2020-02-05T22:19:20.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:20.432+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.433306+0000) 2020-02-05T22:19:20.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:20.433+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.433552+0000) 2020-02-05T22:19:20.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:20.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:20.437348+0000) 2020-02-05T22:19:20.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:20.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:22.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:22.313+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:22.314176+0000) 2020-02-05T22:19:22.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:22.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:22.709518+0000) 2020-02-05T22:19:23.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:23.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:23.827058+0000) 2020-02-05T22:19:23.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:23.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:23.827161+0000) 2020-02-05T22:19:25.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:25.446+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.447379+0000) 2020-02-05T22:19:25.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:25.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.447898+0000) 2020-02-05T22:19:25.450 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:25.448+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.449735+0000) 2020-02-05T22:19:25.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:25.449+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.449561+0000) 2020-02-05T22:19:25.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:25.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.450301+0000) 2020-02-05T22:19:25.452 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:25.450+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.450839+0000) 2020-02-05T22:19:25.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:25.449+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.450959+0000) 2020-02-05T22:19:25.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:25.450+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.451413+0000) 2020-02-05T22:19:25.453 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:25.451+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.451701+0000) 2020-02-05T22:19:25.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:25.451+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.452213+0000) 2020-02-05T22:19:25.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:25.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.453015+0000) 2020-02-05T22:19:25.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:25.452+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.453959+0000) 2020-02-05T22:19:25.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:25.453+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.454273+0000) 2020-02-05T22:19:25.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:25.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.457229+0000) 2020-02-05T22:19:25.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:25.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:25.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:25.921+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.922579+0000) 2020-02-05T22:19:25.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:25.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:25.929919+0000) 2020-02-05T22:19:27.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:27.068+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:27.068935+0000) 2020-02-05T22:19:27.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:27.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:27.069883+0000) 2020-02-05T22:19:27.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:27.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:27.069850+0000) 2020-02-05T22:19:28.308 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:19:28.314 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:19:29.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:29.458+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:29.459600+0000) 2020-02-05T22:19:29.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:29.793+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:29.794696+0000) 2020-02-05T22:19:30.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:30.065+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.066375+0000) 2020-02-05T22:19:30.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:30.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.301763+0000) 2020-02-05T22:19:30.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:30.317+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.318577+0000) 2020-02-05T22:19:30.471 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:30.469+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.470152+0000) 2020-02-05T22:19:30.472 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:30.469+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.470895+0000) 2020-02-05T22:19:30.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:30.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.471507+0000) 2020-02-05T22:19:30.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:30.471+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.472414+0000) 2020-02-05T22:19:30.474 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:30.472+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.473068+0000) 2020-02-05T22:19:30.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:30.472+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.473492+0000) 2020-02-05T22:19:30.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:30.473+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.473814+0000) 2020-02-05T22:19:30.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:30.472+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.474013+0000) 2020-02-05T22:19:30.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:30.473+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.474014+0000) 2020-02-05T22:19:30.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:30.474+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.474460+0000) 2020-02-05T22:19:30.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:30.473+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.474721+0000) 2020-02-05T22:19:30.477 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:30.474+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.475211+0000) 2020-02-05T22:19:30.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:30.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.475687+0000) 2020-02-05T22:19:30.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:30.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:30.480639+0000) 2020-02-05T22:19:30.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:30.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:32.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:32.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:32.314423+0000) 2020-02-05T22:19:32.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:32.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:32.709751+0000) 2020-02-05T22:19:33.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:33.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:33.827338+0000) 2020-02-05T22:19:33.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:33.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:33.827394+0000) 2020-02-05T22:19:35.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:35.488+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.489779+0000) 2020-02-05T22:19:35.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:35.489+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.490196+0000) 2020-02-05T22:19:35.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:35.490+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.491265+0000) 2020-02-05T22:19:35.493 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:35.490+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.491336+0000) 2020-02-05T22:19:35.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:35.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.491111+0000) 2020-02-05T22:19:35.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:35.491+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.492000+0000) 2020-02-05T22:19:35.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:35.491+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.492719+0000) 2020-02-05T22:19:35.495 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:35.492+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.493223+0000) 2020-02-05T22:19:35.495 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:35.492+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.493249+0000) 2020-02-05T22:19:35.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:35.493+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.493853+0000) 2020-02-05T22:19:35.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:35.494+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.494619+0000) 2020-02-05T22:19:35.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:35.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.494605+0000) 2020-02-05T22:19:35.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:35.493+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.494648+0000) 2020-02-05T22:19:35.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:35.501+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111869 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:35.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:35.496+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.497491+0000) 2020-02-05T22:19:35.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:35.921+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.922812+0000) 2020-02-05T22:19:35.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:35.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:35.930136+0000) 2020-02-05T22:19:37.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:37.068+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:37.069194+0000) 2020-02-05T22:19:37.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:37.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:37.070050+0000) 2020-02-05T22:19:37.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:37.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:37.070113+0000) 2020-02-05T22:19:39.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:39.459+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:39.459845+0000) 2020-02-05T22:19:39.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:39.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:39.803050+0000) 2020-02-05T22:19:40.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:40.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.066610+0000) 2020-02-05T22:19:40.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:40.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.301995+0000) 2020-02-05T22:19:40.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:40.318+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.318801+0000) 2020-02-05T22:19:40.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:40.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.507822+0000) 2020-02-05T22:19:40.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:40.507+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.508627+0000) 2020-02-05T22:19:40.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:40.509+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.510399+0000) 2020-02-05T22:19:40.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:40.509+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.510672+0000) 2020-02-05T22:19:40.513 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:40.510+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.511028+0000) 2020-02-05T22:19:40.513 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:40.511+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.511694+0000) 2020-02-05T22:19:40.513 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:40.510+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.511133+0000) 2020-02-05T22:19:40.515 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:40.511+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.512317+0000) 2020-02-05T22:19:40.515 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:40.511+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.513019+0000) 2020-02-05T22:19:40.515 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:40.512+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.512623+0000) 2020-02-05T22:19:40.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:40.512+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.513287+0000) 2020-02-05T22:19:40.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:40.513+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.514400+0000) 2020-02-05T22:19:40.517 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:40.514+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.514465+0000) 2020-02-05T22:19:40.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:40.517+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:40.518870+0000) 2020-02-05T22:19:40.520 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:40.518+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:42.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:42.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:42.314740+0000) 2020-02-05T22:19:42.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:42.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:42.709988+0000) 2020-02-05T22:19:43.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:43.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:43.827577+0000) 2020-02-05T22:19:43.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:43.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:43.827605+0000) 2020-02-05T22:19:45.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:45.524+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.525534+0000) 2020-02-05T22:19:45.527 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:45.524+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.525931+0000) 2020-02-05T22:19:45.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:45.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.526398+0000) 2020-02-05T22:19:45.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:45.526+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.526755+0000) 2020-02-05T22:19:45.529 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:45.526+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.527198+0000) 2020-02-05T22:19:45.529 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:45.526+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.527584+0000) 2020-02-05T22:19:45.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:45.527+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.528053+0000) 2020-02-05T22:19:45.531 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:45.527+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.528276+0000) 2020-02-05T22:19:45.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:45.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.528222+0000) 2020-02-05T22:19:45.531 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:45.527+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.528790+0000) 2020-02-05T22:19:45.532 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:45.529+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.529595+0000) 2020-02-05T22:19:45.532 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:45.530+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.530512+0000) 2020-02-05T22:19:45.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:45.530+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.531056+0000) 2020-02-05T22:19:45.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:45.532+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.533901+0000) 2020-02-05T22:19:45.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:45.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111948 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:45.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:45.922+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.923015+0000) 2020-02-05T22:19:45.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:45.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:45.930345+0000) 2020-02-05T22:19:47.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:47.069+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:47.069460+0000) 2020-02-05T22:19:47.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:47.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:47.070265+0000) 2020-02-05T22:19:47.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:47.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:47.070374+0000) 2020-02-05T22:19:49.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:49.459+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:49.460112+0000) 2020-02-05T22:19:49.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:49.811+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:49.811422+0000) 2020-02-05T22:19:50.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:50.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.066842+0000) 2020-02-05T22:19:50.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:50.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.302269+0000) 2020-02-05T22:19:50.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:50.318+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.319028+0000) 2020-02-05T22:19:50.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:50.545+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.546533+0000) 2020-02-05T22:19:50.548 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:50.545+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.546968+0000) 2020-02-05T22:19:50.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:50.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.547514+0000) 2020-02-05T22:19:50.549 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:50.546+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.547829+0000) 2020-02-05T22:19:50.549 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:50.547+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.548275+0000) 2020-02-05T22:19:50.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:50.547+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.548616+0000) 2020-02-05T22:19:50.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:50.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.549180+0000) 2020-02-05T22:19:50.550 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:50.548+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.549717+0000) 2020-02-05T22:19:50.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:50.548+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.549674+0000) 2020-02-05T22:19:50.552 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:50.549+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.550618+0000) 2020-02-05T22:19:50.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:50.549+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.550860+0000) 2020-02-05T22:19:50.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:50.550+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.551838+0000) 2020-02-05T22:19:50.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:50.550+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.552100+0000) 2020-02-05T22:19:50.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:50.556+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:50.557150+0000) 2020-02-05T22:19:50.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:50.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 111989 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:52.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:52.313+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:52.315002+0000) 2020-02-05T22:19:52.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:52.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:52.710225+0000) 2020-02-05T22:19:53.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:53.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:53.827852+0000) 2020-02-05T22:19:53.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:53.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:53.827836+0000) 2020-02-05T22:19:55.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:19:55.562+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.563869+0000) 2020-02-05T22:19:55.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:55.564+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.565420+0000) 2020-02-05T22:19:55.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:55.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.566443+0000) 2020-02-05T22:19:55.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:19:55.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.566808+0000) 2020-02-05T22:19:55.568 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:19:55.566+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.567145+0000) 2020-02-05T22:19:55.568 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:19:55.566+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.567785+0000) 2020-02-05T22:19:55.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:55.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.568484+0000) 2020-02-05T22:19:55.570 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:55.567+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.568691+0000) 2020-02-05T22:19:55.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:55.567+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.569150+0000) 2020-02-05T22:19:55.570 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:19:55.567+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.569411+0000) 2020-02-05T22:19:55.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:55.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.570315+0000) 2020-02-05T22:19:55.571 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:55.569+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.570666+0000) 2020-02-05T22:19:55.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:19:55.569+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.570717+0000) 2020-02-05T22:19:55.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:19:55.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:19:55.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:55.576+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.577204+0000) 2020-02-05T22:19:55.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:19:55.923+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.923301+0000) 2020-02-05T22:19:55.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:19:55.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:55.930565+0000) 2020-02-05T22:19:57.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:19:57.068+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:57.069696+0000) 2020-02-05T22:19:57.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:19:57.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:57.070516+0000) 2020-02-05T22:19:57.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:19:57.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:57.070656+0000) 2020-02-05T22:19:58.455 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:19:58.462 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:19:59.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:59.460+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:59.460371+0000) 2020-02-05T22:19:59.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:19:59.818+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:19:59.819128+0000) 2020-02-05T22:20:00.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:00.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.067072+0000) 2020-02-05T22:20:00.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:00.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.302513+0000) 2020-02-05T22:20:00.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:00.319+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.319291+0000) 2020-02-05T22:20:00.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:00.580+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.582150+0000) 2020-02-05T22:20:00.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:00.583+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.583663+0000) 2020-02-05T22:20:00.585 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:00.583+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.584473+0000) 2020-02-05T22:20:00.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:00.583+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.584750+0000) 2020-02-05T22:20:00.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:00.584+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.585336+0000) 2020-02-05T22:20:00.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:00.583+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.585252+0000) 2020-02-05T22:20:00.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:00.584+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.585645+0000) 2020-02-05T22:20:00.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:00.585+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.586212+0000) 2020-02-05T22:20:00.590 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:00.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.587216+0000) 2020-02-05T22:20:00.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:00.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.588343+0000) 2020-02-05T22:20:00.591 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:00.585+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.586965+0000) 2020-02-05T22:20:00.591 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:00.586+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.587613+0000) 2020-02-05T22:20:00.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:00.593+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.594143+0000) 2020-02-05T22:20:00.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:00.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:00.595366+0000) 2020-02-05T22:20:00.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:00.594+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:02.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:02.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:02.315255+0000) 2020-02-05T22:20:02.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:02.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:02.710465+0000) 2020-02-05T22:20:03.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:03.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:03.828126+0000) 2020-02-05T22:20:03.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:03.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:03.828113+0000) 2020-02-05T22:20:05.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:05.601+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.603177+0000) 2020-02-05T22:20:05.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:05.603+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.604149+0000) 2020-02-05T22:20:05.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:05.602+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.603570+0000) 2020-02-05T22:20:05.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:05.603+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.605057+0000) 2020-02-05T22:20:05.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:05.604+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.605482+0000) 2020-02-05T22:20:05.607 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:05.605+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.606284+0000) 2020-02-05T22:20:05.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:05.606+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.607770+0000) 2020-02-05T22:20:05.609 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:05.606+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.607435+0000) 2020-02-05T22:20:05.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:05.607+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.608541+0000) 2020-02-05T22:20:05.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:05.607+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.608756+0000) 2020-02-05T22:20:05.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:05.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.608499+0000) 2020-02-05T22:20:05.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:05.607+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.609059+0000) 2020-02-05T22:20:05.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:05.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.612520+0000) 2020-02-05T22:20:05.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:05.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.612816+0000) 2020-02-05T22:20:05.631 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:05.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:05.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:05.922+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.923505+0000) 2020-02-05T22:20:05.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:05.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:05.930807+0000) 2020-02-05T22:20:07.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:07.069+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:07.069947+0000) 2020-02-05T22:20:07.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:07.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:07.070748+0000) 2020-02-05T22:20:07.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:07.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:07.070902+0000) 2020-02-05T22:20:09.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:09.459+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:09.460614+0000) 2020-02-05T22:20:09.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:09.829+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:09.830609+0000) 2020-02-05T22:20:10.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:10.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.067367+0000) 2020-02-05T22:20:10.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:10.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.302745+0000) 2020-02-05T22:20:10.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:10.318+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.319527+0000) 2020-02-05T22:20:10.622 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:10.620+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.621513+0000) 2020-02-05T22:20:10.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:10.622+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.623272+0000) 2020-02-05T22:20:10.625 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:10.622+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.623647+0000) 2020-02-05T22:20:10.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:10.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.624322+0000) 2020-02-05T22:20:10.626 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:10.623+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.624833+0000) 2020-02-05T22:20:10.626 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:10.623+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.624838+0000) 2020-02-05T22:20:10.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:10.624+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.625373+0000) 2020-02-05T22:20:10.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:10.625+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.626379+0000) 2020-02-05T22:20:10.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:10.624+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.625661+0000) 2020-02-05T22:20:10.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:10.624+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.625772+0000) 2020-02-05T22:20:10.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:10.625+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.626500+0000) 2020-02-05T22:20:10.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:10.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.627071+0000) 2020-02-05T22:20:10.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:10.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.627784+0000) 2020-02-05T22:20:10.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:10.629+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:10.630709+0000) 2020-02-05T22:20:10.637 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:10.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:12.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:12.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:12.315510+0000) 2020-02-05T22:20:12.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:12.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:12.710689+0000) 2020-02-05T22:20:13.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:13.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:13.828349+0000) 2020-02-05T22:20:13.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:13.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:13.828370+0000) 2020-02-05T22:20:15.646 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:15.641+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.643368+0000) 2020-02-05T22:20:15.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:15.642+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.643558+0000) 2020-02-05T22:20:15.647 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:15.641+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.642745+0000) 2020-02-05T22:20:15.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:15.643+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.644232+0000) 2020-02-05T22:20:15.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:15.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.644618+0000) 2020-02-05T22:20:15.649 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:15.643+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.644901+0000) 2020-02-05T22:20:15.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:15.643+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.645361+0000) 2020-02-05T22:20:15.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:15.644+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.646199+0000) 2020-02-05T22:20:15.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:15.645+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.646925+0000) 2020-02-05T22:20:15.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:15.644+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.645858+0000) 2020-02-05T22:20:15.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:15.645+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.646632+0000) 2020-02-05T22:20:15.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:15.646+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.647362+0000) 2020-02-05T22:20:15.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:15.652+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.653798+0000) 2020-02-05T22:20:15.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:15.653+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.654152+0000) 2020-02-05T22:20:15.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:15.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:15.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:15.922+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.923711+0000) 2020-02-05T22:20:15.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:15.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:15.931021+0000) 2020-02-05T22:20:17.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:17.069+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:17.070198+0000) 2020-02-05T22:20:17.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:17.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:17.070962+0000) 2020-02-05T22:20:17.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:17.069+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:17.071170+0000) 2020-02-05T22:20:19.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:19.460+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:19.460857+0000) 2020-02-05T22:20:19.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:19.838+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:19.839017+0000) 2020-02-05T22:20:20.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:20.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.067648+0000) 2020-02-05T22:20:20.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:20.301+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.302976+0000) 2020-02-05T22:20:20.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:20.318+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.319771+0000) 2020-02-05T22:20:20.661 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:20.659+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.660064+0000) 2020-02-05T22:20:20.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:20.659+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.660907+0000) 2020-02-05T22:20:20.662 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:20.659+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.661403+0000) 2020-02-05T22:20:20.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:20.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.661809+0000) 2020-02-05T22:20:20.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:20.661+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.662513+0000) 2020-02-05T22:20:20.664 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:20.660+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.662201+0000) 2020-02-05T22:20:20.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:20.662+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.663215+0000) 2020-02-05T22:20:20.664 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:20.662+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.663709+0000) 2020-02-05T22:20:20.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:20.662+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.663666+0000) 2020-02-05T22:20:20.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:20.662+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.664236+0000) 2020-02-05T22:20:20.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:20.663+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.664662+0000) 2020-02-05T22:20:20.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:20.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.665640+0000) 2020-02-05T22:20:20.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:20.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.670189+0000) 2020-02-05T22:20:20.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:20.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:20.670452+0000) 2020-02-05T22:20:20.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:20.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:22.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:22.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:22.315704+0000) 2020-02-05T22:20:22.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:22.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:22.710924+0000) 2020-02-05T22:20:23.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:23.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:23.828597+0000) 2020-02-05T22:20:23.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:23.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:23.828620+0000) 2020-02-05T22:20:25.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:25.678+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.679695+0000) 2020-02-05T22:20:25.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:25.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.680373+0000) 2020-02-05T22:20:25.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:25.679+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.680629+0000) 2020-02-05T22:20:25.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:25.679+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.680993+0000) 2020-02-05T22:20:25.683 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:25.679+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.681325+0000) 2020-02-05T22:20:25.683 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:25.680+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.681957+0000) 2020-02-05T22:20:25.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:25.680+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.682370+0000) 2020-02-05T22:20:25.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:25.681+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.682707+0000) 2020-02-05T22:20:25.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:25.681+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.683402+0000) 2020-02-05T22:20:25.685 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:25.682+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.684101+0000) 2020-02-05T22:20:25.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:25.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.683922+0000) 2020-02-05T22:20:25.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:25.683+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.685067+0000) 2020-02-05T22:20:25.688 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:25.684+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.685447+0000) 2020-02-05T22:20:25.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:25.691+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.692791+0000) 2020-02-05T22:20:25.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:25.691+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112268 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:25.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:25.923+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.923925+0000) 2020-02-05T22:20:25.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:25.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:25.931222+0000) 2020-02-05T22:20:27.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:27.069+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:27.070463+0000) 2020-02-05T22:20:27.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:27.069+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:27.071149+0000) 2020-02-05T22:20:27.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:27.070+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:27.071423+0000) 2020-02-05T22:20:28.620 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:20:28.626 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:20:29.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:29.460+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:29.461136+0000) 2020-02-05T22:20:29.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:29.846+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:29.847222+0000) 2020-02-05T22:20:30.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:30.067+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.067883+0000) 2020-02-05T22:20:30.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:30.302+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.303233+0000) 2020-02-05T22:20:30.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:30.318+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.320004+0000) 2020-02-05T22:20:30.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:30.697+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.698224+0000) 2020-02-05T22:20:30.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:30.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.699379+0000) 2020-02-05T22:20:30.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:30.698+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.699815+0000) 2020-02-05T22:20:30.701 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:30.699+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.700168+0000) 2020-02-05T22:20:30.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:30.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.700594+0000) 2020-02-05T22:20:30.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:30.700+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.701070+0000) 2020-02-05T22:20:30.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:30.700+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.701429+0000) 2020-02-05T22:20:30.702 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:30.700+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.701581+0000) 2020-02-05T22:20:30.706 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:30.700+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.702349+0000) 2020-02-05T22:20:30.707 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:30.701+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.703231+0000) 2020-02-05T22:20:30.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:30.702+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.703685+0000) 2020-02-05T22:20:30.707 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:30.702+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.704274+0000) 2020-02-05T22:20:30.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:30.703+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.704602+0000) 2020-02-05T22:20:30.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:30.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:30.708885+0000) 2020-02-05T22:20:30.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:30.710+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112309 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:32.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:32.314+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:32.315958+0000) 2020-02-05T22:20:32.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:32.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:32.711174+0000) 2020-02-05T22:20:33.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:33.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:33.828833+0000) 2020-02-05T22:20:33.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:33.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:33.828848+0000) 2020-02-05T22:20:35.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:35.717+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.718152+0000) 2020-02-05T22:20:35.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:35.717+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.718655+0000) 2020-02-05T22:20:35.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:35.718+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.719632+0000) 2020-02-05T22:20:35.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:35.718+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.719457+0000) 2020-02-05T22:20:35.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:35.719+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.720562+0000) 2020-02-05T22:20:35.722 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:35.719+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.721252+0000) 2020-02-05T22:20:35.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:35.720+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.721626+0000) 2020-02-05T22:20:35.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:35.722+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.723285+0000) 2020-02-05T22:20:35.724 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:35.722+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.723526+0000) 2020-02-05T22:20:35.725 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:35.722+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.723685+0000) 2020-02-05T22:20:35.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:35.722+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.723824+0000) 2020-02-05T22:20:35.726 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:35.722+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.724388+0000) 2020-02-05T22:20:35.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:35.726+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.728012+0000) 2020-02-05T22:20:35.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:35.727+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.728275+0000) 2020-02-05T22:20:35.733 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:35.730+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:35.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:35.923+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.924148+0000) 2020-02-05T22:20:35.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:35.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:35.931460+0000) 2020-02-05T22:20:37.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:37.070+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:37.070733+0000) 2020-02-05T22:20:37.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:37.070+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:37.071352+0000) 2020-02-05T22:20:37.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:37.070+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:37.071633+0000) 2020-02-05T22:20:39.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:39.460+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:39.461393+0000) 2020-02-05T22:20:39.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:39.854+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:39.855420+0000) 2020-02-05T22:20:40.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:40.067+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.068120+0000) 2020-02-05T22:20:40.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:40.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.303488+0000) 2020-02-05T22:20:40.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:40.319+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.320324+0000) 2020-02-05T22:20:40.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:40.737+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.738866+0000) 2020-02-05T22:20:40.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:40.738+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.739370+0000) 2020-02-05T22:20:40.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:40.738+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.739755+0000) 2020-02-05T22:20:40.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:40.739+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.740812+0000) 2020-02-05T22:20:40.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:40.740+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.741284+0000) 2020-02-05T22:20:40.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:40.740+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.742013+0000) 2020-02-05T22:20:40.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:40.741+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.742271+0000) 2020-02-05T22:20:40.744 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:40.741+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.740386+0000) 2020-02-05T22:20:40.744 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:40.742+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.742614+0000) 2020-02-05T22:20:40.744 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:40.742+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.743330+0000) 2020-02-05T22:20:40.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:40.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.743868+0000) 2020-02-05T22:20:40.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:40.743+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.744157+0000) 2020-02-05T22:20:40.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:40.746+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.747964+0000) 2020-02-05T22:20:40.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:40.747+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:40.748242+0000) 2020-02-05T22:20:40.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:40.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:42.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:42.315+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:42.316232+0000) 2020-02-05T22:20:42.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:42.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:42.711410+0000) 2020-02-05T22:20:43.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:43.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:43.829095+0000) 2020-02-05T22:20:43.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:43.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:43.829118+0000) 2020-02-05T22:20:45.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:45.757+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.758672+0000) 2020-02-05T22:20:45.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:45.758+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.759337+0000) 2020-02-05T22:20:45.760 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:45.759+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.759545+0000) 2020-02-05T22:20:45.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:45.758+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.760038+0000) 2020-02-05T22:20:45.761 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:45.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.760786+0000) 2020-02-05T22:20:45.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:45.760+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.761146+0000) 2020-02-05T22:20:45.762 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:45.761+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.761536+0000) 2020-02-05T22:20:45.764 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:45.762+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.762664+0000) 2020-02-05T22:20:45.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:45.762+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.762859+0000) 2020-02-05T22:20:45.764 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:45.762+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.763763+0000) 2020-02-05T22:20:45.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:45.764+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.764907+0000) 2020-02-05T22:20:45.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:45.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.765063+0000) 2020-02-05T22:20:45.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:45.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.767102+0000) 2020-02-05T22:20:45.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:45.767+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.768493+0000) 2020-02-05T22:20:45.773 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:45.771+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:45.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:45.924+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.924368+0000) 2020-02-05T22:20:45.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:45.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:45.931682+0000) 2020-02-05T22:20:47.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:47.070+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:47.070969+0000) 2020-02-05T22:20:47.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:47.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:47.071573+0000) 2020-02-05T22:20:47.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:47.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:47.071882+0000) 2020-02-05T22:20:49.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:49.461+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:49.461633+0000) 2020-02-05T22:20:49.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:49.866+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:49.867224+0000) 2020-02-05T22:20:50.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:50.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.068346+0000) 2020-02-05T22:20:50.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:50.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.303723+0000) 2020-02-05T22:20:50.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:50.320+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.320646+0000) 2020-02-05T22:20:50.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:50.776+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.777788+0000) 2020-02-05T22:20:50.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:50.777+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.778117+0000) 2020-02-05T22:20:50.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:50.777+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.778608+0000) 2020-02-05T22:20:50.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:50.780+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.780604+0000) 2020-02-05T22:20:50.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:50.780+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.781287+0000) 2020-02-05T22:20:50.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:50.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.782335+0000) 2020-02-05T22:20:50.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:50.782+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.783110+0000) 2020-02-05T22:20:50.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:50.782+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.783316+0000) 2020-02-05T22:20:50.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:50.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.783514+0000) 2020-02-05T22:20:50.785 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:50.783+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.783718+0000) 2020-02-05T22:20:50.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:50.783+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.784065+0000) 2020-02-05T22:20:50.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:50.783+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.784377+0000) 2020-02-05T22:20:50.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:50.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.788463+0000) 2020-02-05T22:20:50.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:50.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:50.788784+0000) 2020-02-05T22:20:50.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:50.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112470 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:52.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:52.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:52.316494+0000) 2020-02-05T22:20:52.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:52.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:52.711669+0000) 2020-02-05T22:20:53.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:53.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:53.829331+0000) 2020-02-05T22:20:53.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:53.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:53.829365+0000) 2020-02-05T22:20:55.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:20:55.796+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.796282+0000) 2020-02-05T22:20:55.798 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:20:55.796+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.796705+0000) 2020-02-05T22:20:55.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:55.797+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.798236+0000) 2020-02-05T22:20:55.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:20:55.797+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.798518+0000) 2020-02-05T22:20:55.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:55.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.798816+0000) 2020-02-05T22:20:55.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:55.798+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.799327+0000) 2020-02-05T22:20:55.801 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:20:55.799+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.799823+0000) 2020-02-05T22:20:55.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:55.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.800033+0000) 2020-02-05T22:20:55.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:55.800+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.800766+0000) 2020-02-05T22:20:55.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:55.800+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.800648+0000) 2020-02-05T22:20:55.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:55.800+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.801221+0000) 2020-02-05T22:20:55.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:20:55.800+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.801845+0000) 2020-02-05T22:20:55.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:20:55.801+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.801843+0000) 2020-02-05T22:20:55.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:55.808+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.808407+0000) 2020-02-05T22:20:55.810 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:20:55.809+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:20:55.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:20:55.924+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.924582+0000) 2020-02-05T22:20:55.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:20:55.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:55.931943+0000) 2020-02-05T22:20:57.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:20:57.070+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:57.071203+0000) 2020-02-05T22:20:57.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:20:57.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:57.071774+0000) 2020-02-05T22:20:57.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:20:57.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:57.072062+0000) 2020-02-05T22:20:58.759 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:20:58.765 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:20:59.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:59.461+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:59.461875+0000) 2020-02-05T22:20:59.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:20:59.876+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:20:59.876826+0000) 2020-02-05T22:21:00.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:00.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.068643+0000) 2020-02-05T22:21:00.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:00.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.303959+0000) 2020-02-05T22:21:00.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:00.320+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.320892+0000) 2020-02-05T22:21:00.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:00.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.815754+0000) 2020-02-05T22:21:00.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:00.815+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.816048+0000) 2020-02-05T22:21:00.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:00.817+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.817472+0000) 2020-02-05T22:21:00.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:00.817+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.817766+0000) 2020-02-05T22:21:00.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:00.818+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.818362+0000) 2020-02-05T22:21:00.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:00.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.818744+0000) 2020-02-05T22:21:00.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:00.819+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.819214+0000) 2020-02-05T22:21:00.821 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:00.818+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.819093+0000) 2020-02-05T22:21:00.821 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:00.819+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.819695+0000) 2020-02-05T22:21:00.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:00.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.819647+0000) 2020-02-05T22:21:00.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:00.819+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.819599+0000) 2020-02-05T22:21:00.822 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:00.820+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.820582+0000) 2020-02-05T22:21:00.822 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:00.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.820847+0000) 2020-02-05T22:21:00.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:00.826+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:00.826230+0000) 2020-02-05T22:21:00.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:00.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:02.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:02.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:02.316728+0000) 2020-02-05T22:21:02.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:02.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:02.711920+0000) 2020-02-05T22:21:03.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:03.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:03.829564+0000) 2020-02-05T22:21:03.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:03.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:03.829606+0000) 2020-02-05T22:21:05.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:05.836+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.836992+0000) 2020-02-05T22:21:05.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:05.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.837844+0000) 2020-02-05T22:21:05.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:05.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.838330+0000) 2020-02-05T22:21:05.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:05.838+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.838829+0000) 2020-02-05T22:21:05.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:05.837+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.838986+0000) 2020-02-05T22:21:05.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:05.838+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.839427+0000) 2020-02-05T22:21:05.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:05.839+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.839694+0000) 2020-02-05T22:21:05.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:05.839+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.840340+0000) 2020-02-05T22:21:05.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:05.839+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.840728+0000) 2020-02-05T22:21:05.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:05.840+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.841115+0000) 2020-02-05T22:21:05.844 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:05.841+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.841653+0000) 2020-02-05T22:21:05.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:05.841+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.842025+0000) 2020-02-05T22:21:05.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:05.840+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.841901+0000) 2020-02-05T22:21:05.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:05.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.847808+0000) 2020-02-05T22:21:05.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:05.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:05.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:05.923+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.924823+0000) 2020-02-05T22:21:05.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:05.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:05.932170+0000) 2020-02-05T22:21:07.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:07.071+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:07.071497+0000) 2020-02-05T22:21:07.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:07.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:07.071981+0000) 2020-02-05T22:21:07.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:07.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:07.072267+0000) 2020-02-05T22:21:09.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:09.461+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:09.462131+0000) 2020-02-05T22:21:09.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:09.884+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:09.884962+0000) 2020-02-05T22:21:10.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:10.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.068912+0000) 2020-02-05T22:21:10.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:10.304+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.304212+0000) 2020-02-05T22:21:10.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:10.320+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.321189+0000) 2020-02-05T22:21:10.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:10.855+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.855520+0000) 2020-02-05T22:21:10.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:10.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.857313+0000) 2020-02-05T22:21:10.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:10.856+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.857369+0000) 2020-02-05T22:21:10.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:10.857+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.858251+0000) 2020-02-05T22:21:10.859 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:10.858+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.858834+0000) 2020-02-05T22:21:10.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:10.858+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.859724+0000) 2020-02-05T22:21:10.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:10.858+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.859652+0000) 2020-02-05T22:21:10.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:10.859+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.860189+0000) 2020-02-05T22:21:10.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:10.859+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.860378+0000) 2020-02-05T22:21:10.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:10.859+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.860488+0000) 2020-02-05T22:21:10.862 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:10.860+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.860804+0000) 2020-02-05T22:21:10.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:10.860+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.861789+0000) 2020-02-05T22:21:10.863 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:10.861+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.862153+0000) 2020-02-05T22:21:10.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:10.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:10.867069+0000) 2020-02-05T22:21:10.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:10.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112636 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:12.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:12.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:12.316956+0000) 2020-02-05T22:21:12.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:12.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:12.712145+0000) 2020-02-05T22:21:13.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:13.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:13.829785+0000) 2020-02-05T22:21:13.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:13.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:13.829904+0000) 2020-02-05T22:21:15.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:15.873+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.874704+0000) 2020-02-05T22:21:15.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:15.874+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.875449+0000) 2020-02-05T22:21:15.877 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:15.875+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.876033+0000) 2020-02-05T22:21:15.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:15.875+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.876377+0000) 2020-02-05T22:21:15.878 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:15.876+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.876741+0000) 2020-02-05T22:21:15.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:15.875+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.876759+0000) 2020-02-05T22:21:15.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:15.876+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.877313+0000) 2020-02-05T22:21:15.879 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:15.876+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.877305+0000) 2020-02-05T22:21:15.879 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:15.876+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.877731+0000) 2020-02-05T22:21:15.880 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:15.878+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.878679+0000) 2020-02-05T22:21:15.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:15.878+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.879529+0000) 2020-02-05T22:21:15.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:15.879+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.879965+0000) 2020-02-05T22:21:15.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:15.880+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.880480+0000) 2020-02-05T22:21:15.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:15.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.886460+0000) 2020-02-05T22:21:15.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:15.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:15.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:15.923+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.924986+0000) 2020-02-05T22:21:15.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:15.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:15.932373+0000) 2020-02-05T22:21:17.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:17.071+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:17.071764+0000) 2020-02-05T22:21:17.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:17.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:17.072203+0000) 2020-02-05T22:21:17.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:17.072+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:17.072513+0000) 2020-02-05T22:21:19.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:19.461+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:19.462414+0000) 2020-02-05T22:21:19.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:19.892+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:19.893605+0000) 2020-02-05T22:21:20.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:20.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.069135+0000) 2020-02-05T22:21:20.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:20.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.304485+0000) 2020-02-05T22:21:20.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:20.320+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.321411+0000) 2020-02-05T22:21:20.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:20.893+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.895114+0000) 2020-02-05T22:21:20.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:20.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.896235+0000) 2020-02-05T22:21:20.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:20.895+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.896683+0000) 2020-02-05T22:21:20.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:20.895+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.896749+0000) 2020-02-05T22:21:20.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:20.896+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.897643+0000) 2020-02-05T22:21:20.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:20.896+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.897590+0000) 2020-02-05T22:21:20.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:20.897+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.898233+0000) 2020-02-05T22:21:20.899 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:20.896+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.897911+0000) 2020-02-05T22:21:20.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:20.897+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.898912+0000) 2020-02-05T22:21:20.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:20.898+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.899578+0000) 2020-02-05T22:21:20.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:20.898+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.899956+0000) 2020-02-05T22:21:20.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:20.898+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.900213+0000) 2020-02-05T22:21:20.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:20.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.900619+0000) 2020-02-05T22:21:20.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:20.905+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:20.906705+0000) 2020-02-05T22:21:20.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:20.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:22.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:22.315+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:22.317207+0000) 2020-02-05T22:21:22.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:22.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:22.712381+0000) 2020-02-05T22:21:23.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:23.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:23.830023+0000) 2020-02-05T22:21:23.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:23.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:23.830143+0000) 2020-02-05T22:21:25.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:25.913+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.915137+0000) 2020-02-05T22:21:25.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:25.914+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.915576+0000) 2020-02-05T22:21:25.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:25.915+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.916399+0000) 2020-02-05T22:21:25.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:25.914+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.916256+0000) 2020-02-05T22:21:25.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:25.915+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.916757+0000) 2020-02-05T22:21:25.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:25.917+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.918629+0000) 2020-02-05T22:21:25.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:25.916+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.917152+0000) 2020-02-05T22:21:25.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:25.916+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.917451+0000) 2020-02-05T22:21:25.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:25.917+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.918512+0000) 2020-02-05T22:21:25.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:25.917+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.918744+0000) 2020-02-05T22:21:25.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:25.917+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.918465+0000) 2020-02-05T22:21:25.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:25.917+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.919047+0000) 2020-02-05T22:21:25.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:25.917+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.918683+0000) 2020-02-05T22:21:25.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:25.924+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.925133+0000) 2020-02-05T22:21:25.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:25.924+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.925585+0000) 2020-02-05T22:21:25.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:25.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:25.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:25.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:25.932588+0000) 2020-02-05T22:21:27.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:27.070+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:27.071976+0000) 2020-02-05T22:21:27.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:27.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:27.072486+0000) 2020-02-05T22:21:27.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:27.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:27.072730+0000) 2020-02-05T22:21:28.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:21:28.898 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:21:29.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:29.462+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:29.462649+0000) 2020-02-05T22:21:29.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:29.900+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:29.901777+0000) 2020-02-05T22:21:30.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:30.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.069388+0000) 2020-02-05T22:21:30.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:30.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.304719+0000) 2020-02-05T22:21:30.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:30.321+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.321678+0000) 2020-02-05T22:21:30.936 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:30.933+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.934760+0000) 2020-02-05T22:21:30.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:30.934+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.935552+0000) 2020-02-05T22:21:30.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:30.934+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.935790+0000) 2020-02-05T22:21:30.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:30.935+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.936744+0000) 2020-02-05T22:21:30.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:30.936+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.937220+0000) 2020-02-05T22:21:30.939 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:30.935+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.936562+0000) 2020-02-05T22:21:30.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:30.936+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.937592+0000) 2020-02-05T22:21:30.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:30.937+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.938141+0000) 2020-02-05T22:21:30.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:30.936+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.938212+0000) 2020-02-05T22:21:30.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:30.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.939378+0000) 2020-02-05T22:21:30.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:30.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.939290+0000) 2020-02-05T22:21:30.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:30.937+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.939343+0000) 2020-02-05T22:21:30.943 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:30.938+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.939611+0000) 2020-02-05T22:21:30.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:30.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:30.943013+0000) 2020-02-05T22:21:30.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:30.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:32.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:32.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:32.317481+0000) 2020-02-05T22:21:32.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:32.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:32.712621+0000) 2020-02-05T22:21:33.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:33.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:33.830315+0000) 2020-02-05T22:21:33.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:33.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:33.830374+0000) 2020-02-05T22:21:35.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:35.924+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.925334+0000) 2020-02-05T22:21:35.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:35.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.932809+0000) 2020-02-05T22:21:35.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:35.951+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.952283+0000) 2020-02-05T22:21:35.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:35.951+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.952860+0000) 2020-02-05T22:21:35.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:35.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.953638+0000) 2020-02-05T22:21:35.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:35.953+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.954147+0000) 2020-02-05T22:21:35.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:35.953+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.954521+0000) 2020-02-05T22:21:35.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:35.953+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.954862+0000) 2020-02-05T22:21:35.956 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:35.953+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.955063+0000) 2020-02-05T22:21:35.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:35.954+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.955719+0000) 2020-02-05T22:21:35.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:35.955+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.956818+0000) 2020-02-05T22:21:35.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:35.956+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.957536+0000) 2020-02-05T22:21:35.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:35.956+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.957738+0000) 2020-02-05T22:21:35.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:35.956+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.957935+0000) 2020-02-05T22:21:35.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:35.956+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.957870+0000) 2020-02-05T22:21:35.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:35.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:35.962123+0000) 2020-02-05T22:21:35.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:35.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:37.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:37.070+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:37.072202+0000) 2020-02-05T22:21:37.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:37.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:37.072732+0000) 2020-02-05T22:21:37.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:37.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:37.072964+0000) 2020-02-05T22:21:39.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:39.462+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:39.462890+0000) 2020-02-05T22:21:39.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:39.909+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:39.909931+0000) 2020-02-05T22:21:40.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:40.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.069604+0000) 2020-02-05T22:21:40.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:40.303+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.304952+0000) 2020-02-05T22:21:40.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:40.320+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.321924+0000) 2020-02-05T22:21:40.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:40.971+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.972515+0000) 2020-02-05T22:21:40.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:40.971+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.972809+0000) 2020-02-05T22:21:40.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:40.971+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.973357+0000) 2020-02-05T22:21:40.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:40.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.973642+0000) 2020-02-05T22:21:40.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:40.973+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.974665+0000) 2020-02-05T22:21:40.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:40.974+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.975015+0000) 2020-02-05T22:21:40.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:40.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.975586+0000) 2020-02-05T22:21:40.977 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:40.974+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.975922+0000) 2020-02-05T22:21:40.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:40.975+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.976547+0000) 2020-02-05T22:21:40.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:40.974+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.976316+0000) 2020-02-05T22:21:40.978 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:40.975+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.976993+0000) 2020-02-05T22:21:40.978 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:40.976+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.977445+0000) 2020-02-05T22:21:40.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:40.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.980524+0000) 2020-02-05T22:21:40.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:40.980+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:40.981632+0000) 2020-02-05T22:21:40.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:40.982+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:42.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:42.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:42.317726+0000) 2020-02-05T22:21:42.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:42.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:42.712845+0000) 2020-02-05T22:21:43.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:43.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:43.830554+0000) 2020-02-05T22:21:43.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:43.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:43.830648+0000) 2020-02-05T22:21:45.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:45.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.925578+0000) 2020-02-05T22:21:45.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:45.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.933046+0000) 2020-02-05T22:21:45.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:45.987+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.988550+0000) 2020-02-05T22:21:45.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:45.988+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.989309+0000) 2020-02-05T22:21:45.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:45.989+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.990752+0000) 2020-02-05T22:21:45.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:45.990+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.991241+0000) 2020-02-05T22:21:45.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:45.990+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.991583+0000) 2020-02-05T22:21:45.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:45.990+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.991871+0000) 2020-02-05T22:21:45.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:45.991+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.993082+0000) 2020-02-05T22:21:45.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:45.992+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.993500+0000) 2020-02-05T22:21:45.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:45.993+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.994667+0000) 2020-02-05T22:21:45.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:45.993+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.995032+0000) 2020-02-05T22:21:45.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:45.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.995462+0000) 2020-02-05T22:21:45.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:45.994+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.995826+0000) 2020-02-05T22:21:45.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:45.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:45.996206+0000) 2020-02-05T22:21:46.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:46.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:46.001419+0000) 2020-02-05T22:21:46.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:46.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112926 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:47.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:47.071+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:47.072492+0000) 2020-02-05T22:21:47.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:47.071+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:47.072934+0000) 2020-02-05T22:21:47.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:47.071+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:47.073184+0000) 2020-02-05T22:21:49.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:49.462+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:49.463162+0000) 2020-02-05T22:21:49.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:49.917+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:49.918367+0000) 2020-02-05T22:21:50.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:50.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:50.069879+0000) 2020-02-05T22:21:50.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:50.304+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:50.305209+0000) 2020-02-05T22:21:50.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:50.321+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:50.322153+0000) 2020-02-05T22:21:51.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:51.008+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.009550+0000) 2020-02-05T22:21:51.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:51.008+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.009800+0000) 2020-02-05T22:21:51.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:51.009+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.010424+0000) 2020-02-05T22:21:51.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:51.010+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.011643+0000) 2020-02-05T22:21:51.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:51.012+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.013483+0000) 2020-02-05T22:21:51.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:51.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.012435+0000) 2020-02-05T22:21:51.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:51.012+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.014271+0000) 2020-02-05T22:21:51.016 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:51.013+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.014604+0000) 2020-02-05T22:21:51.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:51.014+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.015458+0000) 2020-02-05T22:21:51.017 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:51.013+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.015166+0000) 2020-02-05T22:21:51.017 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:51.015+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.016063+0000) 2020-02-05T22:21:51.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:51.015+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.016194+0000) 2020-02-05T22:21:51.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:51.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.016530+0000) 2020-02-05T22:21:51.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:51.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:51.018604+0000) 2020-02-05T22:21:51.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:51.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 112968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:52.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:52.316+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:52.318078+0000) 2020-02-05T22:21:52.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:52.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:52.713112+0000) 2020-02-05T22:21:53.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:53.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:53.830777+0000) 2020-02-05T22:21:53.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:53.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:53.830881+0000) 2020-02-05T22:21:55.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:55.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:55.925832+0000) 2020-02-05T22:21:55.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:55.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:55.933279+0000) 2020-02-05T22:21:56.030 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:21:56.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.029199+0000) 2020-02-05T22:21:56.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:21:56.029+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.029737+0000) 2020-02-05T22:21:56.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:21:56.031+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.031673+0000) 2020-02-05T22:21:56.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:56.030+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.031852+0000) 2020-02-05T22:21:56.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:21:56.032+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.032541+0000) 2020-02-05T22:21:56.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:56.032+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.033021+0000) 2020-02-05T22:21:56.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:21:56.031+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.033284+0000) 2020-02-05T22:21:56.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:21:56.032+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.033952+0000) 2020-02-05T22:21:56.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:56.032+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.034256+0000) 2020-02-05T22:21:56.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:21:56.035+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.035656+0000) 2020-02-05T22:21:56.037 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:21:56.033+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.035405+0000) 2020-02-05T22:21:56.037 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:56.034+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.035604+0000) 2020-02-05T22:21:56.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:56.040+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.040941+0000) 2020-02-05T22:21:56.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:56.041+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:56.041215+0000) 2020-02-05T22:21:56.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:21:56.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113008 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:21:57.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:21:57.072+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:57.072732+0000) 2020-02-05T22:21:57.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:21:57.072+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:57.073179+0000) 2020-02-05T22:21:57.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:21:57.072+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:57.073402+0000) 2020-02-05T22:21:59.049 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:21:59.055 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:21:59.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:59.462+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:59.463417+0000) 2020-02-05T22:21:59.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:21:59.924+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:21:59.925232+0000) 2020-02-05T22:22:00.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:00.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:00.070127+0000) 2020-02-05T22:22:00.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:00.304+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:00.305483+0000) 2020-02-05T22:22:00.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:00.321+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:00.322386+0000) 2020-02-05T22:22:01.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:01.047+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.048750+0000) 2020-02-05T22:22:01.050 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:01.048+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.049530+0000) 2020-02-05T22:22:01.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:01.049+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.050463+0000) 2020-02-05T22:22:01.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:01.050+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.051814+0000) 2020-02-05T22:22:01.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:01.051+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.052639+0000) 2020-02-05T22:22:01.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:01.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.051281+0000) 2020-02-05T22:22:01.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:01.050+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.052046+0000) 2020-02-05T22:22:01.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:01.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.053203+0000) 2020-02-05T22:22:01.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:01.052+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.053302+0000) 2020-02-05T22:22:01.055 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:01.052+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.053638+0000) 2020-02-05T22:22:01.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:01.052+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.054288+0000) 2020-02-05T22:22:01.056 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:01.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.055055+0000) 2020-02-05T22:22:01.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:01.054+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.055663+0000) 2020-02-05T22:22:01.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:01.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:01.058999+0000) 2020-02-05T22:22:01.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:01.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:02.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:02.317+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:02.318300+0000) 2020-02-05T22:22:02.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:02.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:02.713348+0000) 2020-02-05T22:22:03.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:03.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:03.831010+0000) 2020-02-05T22:22:03.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:03.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:03.831150+0000) 2020-02-05T22:22:05.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:05.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:05.926123+0000) 2020-02-05T22:22:05.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:05.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:05.933516+0000) 2020-02-05T22:22:06.070 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:06.068+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.069311+0000) 2020-02-05T22:22:06.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:06.069+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.070067+0000) 2020-02-05T22:22:06.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:06.069+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.070534+0000) 2020-02-05T22:22:06.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:06.069+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.071286+0000) 2020-02-05T22:22:06.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:06.070+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.071651+0000) 2020-02-05T22:22:06.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:06.071+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.072181+0000) 2020-02-05T22:22:06.074 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:06.071+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.072457+0000) 2020-02-05T22:22:06.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:06.071+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.072825+0000) 2020-02-05T22:22:06.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:06.071+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.072703+0000) 2020-02-05T22:22:06.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:06.072+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.073443+0000) 2020-02-05T22:22:06.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:06.072+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.073977+0000) 2020-02-05T22:22:06.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:06.073+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.074932+0000) 2020-02-05T22:22:06.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:06.073+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.075282+0000) 2020-02-05T22:22:06.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:06.077+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:06.078638+0000) 2020-02-05T22:22:06.084 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:06.082+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:07.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:07.072+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:07.072990+0000) 2020-02-05T22:22:07.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:07.072+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:07.073485+0000) 2020-02-05T22:22:07.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:07.072+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:07.073653+0000) 2020-02-05T22:22:09.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:09.462+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:09.463663+0000) 2020-02-05T22:22:09.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:09.933+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:09.933873+0000) 2020-02-05T22:22:10.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:10.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:10.070406+0000) 2020-02-05T22:22:10.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:10.305+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:10.305677+0000) 2020-02-05T22:22:10.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:10.321+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:10.322604+0000) 2020-02-05T22:22:11.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:11.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.088419+0000) 2020-02-05T22:22:11.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:11.088+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.089316+0000) 2020-02-05T22:22:11.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:11.089+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.090264+0000) 2020-02-05T22:22:11.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:11.089+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.090643+0000) 2020-02-05T22:22:11.092 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:11.089+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.090963+0000) 2020-02-05T22:22:11.092 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:11.090+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.091301+0000) 2020-02-05T22:22:11.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:11.090+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.091748+0000) 2020-02-05T22:22:11.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:11.091+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.092146+0000) 2020-02-05T22:22:11.094 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:11.091+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.092811+0000) 2020-02-05T22:22:11.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:11.092+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.092746+0000) 2020-02-05T22:22:11.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:11.092+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.093278+0000) 2020-02-05T22:22:11.095 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:11.093+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.093667+0000) 2020-02-05T22:22:11.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:11.093+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.094391+0000) 2020-02-05T22:22:11.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:11.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:11.101380+0000) 2020-02-05T22:22:11.104 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:11.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:12.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:12.318+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:12.318567+0000) 2020-02-05T22:22:12.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:12.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:12.713583+0000) 2020-02-05T22:22:13.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:13.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:13.831268+0000) 2020-02-05T22:22:13.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:13.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:13.831375+0000) 2020-02-05T22:22:15.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:15.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:15.926390+0000) 2020-02-05T22:22:15.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:15.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:15.933758+0000) 2020-02-05T22:22:16.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:16.108+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.109236+0000) 2020-02-05T22:22:16.111 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:16.108+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.110040+0000) 2020-02-05T22:22:16.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:16.110+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.110494+0000) 2020-02-05T22:22:16.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:16.109+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.110927+0000) 2020-02-05T22:22:16.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:16.110+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.111272+0000) 2020-02-05T22:22:16.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:16.110+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.111598+0000) 2020-02-05T22:22:16.113 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:16.110+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.111896+0000) 2020-02-05T22:22:16.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:16.111+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.111925+0000) 2020-02-05T22:22:16.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:16.112+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.112587+0000) 2020-02-05T22:22:16.114 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:16.111+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.112829+0000) 2020-02-05T22:22:16.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:16.112+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.113099+0000) 2020-02-05T22:22:16.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:16.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.113923+0000) 2020-02-05T22:22:16.115 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:16.113+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.113705+0000) 2020-02-05T22:22:16.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:16.117+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:16.118136+0000) 2020-02-05T22:22:16.124 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:16.122+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113174 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:17.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:17.072+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:17.073186+0000) 2020-02-05T22:22:17.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:17.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:17.073705+0000) 2020-02-05T22:22:17.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:17.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:17.073839+0000) 2020-02-05T22:22:19.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:19.463+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:19.463895+0000) 2020-02-05T22:22:19.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:19.941+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:19.942290+0000) 2020-02-05T22:22:20.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:20.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:20.070677+0000) 2020-02-05T22:22:20.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:20.305+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:20.305912+0000) 2020-02-05T22:22:20.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:20.322+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:20.322836+0000) 2020-02-05T22:22:21.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:21.128+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.129151+0000) 2020-02-05T22:22:21.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:21.129+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.129882+0000) 2020-02-05T22:22:21.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:21.130+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.130806+0000) 2020-02-05T22:22:21.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:21.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.131319+0000) 2020-02-05T22:22:21.133 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:21.131+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.132173+0000) 2020-02-05T22:22:21.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:21.131+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.132482+0000) 2020-02-05T22:22:21.134 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:21.132+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.132597+0000) 2020-02-05T22:22:21.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:21.132+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.133183+0000) 2020-02-05T22:22:21.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:21.133+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.133885+0000) 2020-02-05T22:22:21.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:21.133+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.134299+0000) 2020-02-05T22:22:21.135 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:21.133+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.134758+0000) 2020-02-05T22:22:21.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:21.134+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.134887+0000) 2020-02-05T22:22:21.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:21.134+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.135234+0000) 2020-02-05T22:22:21.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:21.137+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:21.138071+0000) 2020-02-05T22:22:21.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:21.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:22.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:22.318+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:22.318860+0000) 2020-02-05T22:22:22.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:22.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:22.713817+0000) 2020-02-05T22:22:23.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:23.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:23.831525+0000) 2020-02-05T22:22:23.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:23.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:23.831624+0000) 2020-02-05T22:22:25.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:25.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:25.926569+0000) 2020-02-05T22:22:25.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:25.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:25.934044+0000) 2020-02-05T22:22:26.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:26.148+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.149740+0000) 2020-02-05T22:22:26.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:26.149+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.150234+0000) 2020-02-05T22:22:26.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:26.149+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.150308+0000) 2020-02-05T22:22:26.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:26.150+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.150547+0000) 2020-02-05T22:22:26.154 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:26.150+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.151156+0000) 2020-02-05T22:22:26.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:26.150+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.151381+0000) 2020-02-05T22:22:26.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:26.151+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.151697+0000) 2020-02-05T22:22:26.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:26.149+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.150876+0000) 2020-02-05T22:22:26.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:26.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.150981+0000) 2020-02-05T22:22:26.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:26.150+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.151304+0000) 2020-02-05T22:22:26.155 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:26.151+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.152662+0000) 2020-02-05T22:22:26.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:26.152+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.152705+0000) 2020-02-05T22:22:26.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:26.152+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.153161+0000) 2020-02-05T22:22:26.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:26.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:26.159775+0000) 2020-02-05T22:22:26.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:26.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:27.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:27.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:27.073469+0000) 2020-02-05T22:22:27.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:27.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:27.073972+0000) 2020-02-05T22:22:27.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:27.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:27.074101+0000) 2020-02-05T22:22:29.208 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:22:29.215 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:22:29.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:29.463+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:29.464161+0000) 2020-02-05T22:22:29.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:29.950+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:29.950535+0000) 2020-02-05T22:22:30.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:30.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:30.070917+0000) 2020-02-05T22:22:30.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:30.305+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:30.306150+0000) 2020-02-05T22:22:30.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:30.322+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:30.323066+0000) 2020-02-05T22:22:31.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:31.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.167794+0000) 2020-02-05T22:22:31.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:31.167+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.168760+0000) 2020-02-05T22:22:31.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:31.169+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.169919+0000) 2020-02-05T22:22:31.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:31.169+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.170263+0000) 2020-02-05T22:22:31.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:31.169+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.170561+0000) 2020-02-05T22:22:31.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:31.169+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.170975+0000) 2020-02-05T22:22:31.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:31.170+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.171242+0000) 2020-02-05T22:22:31.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:31.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.171561+0000) 2020-02-05T22:22:31.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:31.171+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.171890+0000) 2020-02-05T22:22:31.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:31.171+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.172398+0000) 2020-02-05T22:22:31.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:31.172+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.172888+0000) 2020-02-05T22:22:31.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:31.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.174117+0000) 2020-02-05T22:22:31.175 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:31.173+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.174096+0000) 2020-02-05T22:22:31.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:31.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:31.178409+0000) 2020-02-05T22:22:31.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:31.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:32.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:32.318+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:32.319102+0000) 2020-02-05T22:22:32.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:32.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:32.714072+0000) 2020-02-05T22:22:33.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:33.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:33.831754+0000) 2020-02-05T22:22:33.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:33.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:33.831901+0000) 2020-02-05T22:22:35.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:35.925+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:35.926825+0000) 2020-02-05T22:22:35.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:35.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:35.934288+0000) 2020-02-05T22:22:36.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:36.184+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.185130+0000) 2020-02-05T22:22:36.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:36.184+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.185694+0000) 2020-02-05T22:22:36.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:36.185+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.186067+0000) 2020-02-05T22:22:36.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:36.185+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.186903+0000) 2020-02-05T22:22:36.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:36.186+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.187663+0000) 2020-02-05T22:22:36.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:36.187+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.188282+0000) 2020-02-05T22:22:36.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:36.187+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.189032+0000) 2020-02-05T22:22:36.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:36.188+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.189326+0000) 2020-02-05T22:22:36.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:36.188+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.188770+0000) 2020-02-05T22:22:36.191 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:36.189+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.190449+0000) 2020-02-05T22:22:36.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:36.190+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.190531+0000) 2020-02-05T22:22:36.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:36.189+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.190789+0000) 2020-02-05T22:22:36.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:36.190+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.191086+0000) 2020-02-05T22:22:36.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:36.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:36.196813+0000) 2020-02-05T22:22:36.199 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:36.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:37.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:37.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:37.073683+0000) 2020-02-05T22:22:37.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:37.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:37.074160+0000) 2020-02-05T22:22:37.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:37.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:37.074280+0000) 2020-02-05T22:22:39.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:39.463+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:39.464429+0000) 2020-02-05T22:22:39.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:39.957+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:39.958665+0000) 2020-02-05T22:22:40.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:40.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:40.071144+0000) 2020-02-05T22:22:40.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:40.306+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:40.306502+0000) 2020-02-05T22:22:40.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:40.323+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:40.323364+0000) 2020-02-05T22:22:41.205 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:41.203+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.203572+0000) 2020-02-05T22:22:41.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:41.204+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.205310+0000) 2020-02-05T22:22:41.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:41.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.205713+0000) 2020-02-05T22:22:41.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:41.205+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.205933+0000) 2020-02-05T22:22:41.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:41.206+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.206555+0000) 2020-02-05T22:22:41.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:41.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.207037+0000) 2020-02-05T22:22:41.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:41.206+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.207246+0000) 2020-02-05T22:22:41.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:41.207+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.208428+0000) 2020-02-05T22:22:41.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:41.207+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.208611+0000) 2020-02-05T22:22:41.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:41.208+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.208872+0000) 2020-02-05T22:22:41.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:41.208+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.208669+0000) 2020-02-05T22:22:41.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:41.208+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.209792+0000) 2020-02-05T22:22:41.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:41.209+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.209704+0000) 2020-02-05T22:22:41.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:41.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:41.213905+0000) 2020-02-05T22:22:41.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:41.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:42.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:42.319+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:42.319349+0000) 2020-02-05T22:22:42.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:42.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:42.714342+0000) 2020-02-05T22:22:43.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:43.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:43.831983+0000) 2020-02-05T22:22:43.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:43.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:43.832150+0000) 2020-02-05T22:22:45.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:45.926+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:45.927053+0000) 2020-02-05T22:22:45.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:45.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:45.934539+0000) 2020-02-05T22:22:46.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:46.222+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.223019+0000) 2020-02-05T22:22:46.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:46.225+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.226276+0000) 2020-02-05T22:22:46.228 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:46.225+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.226547+0000) 2020-02-05T22:22:46.228 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:46.225+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.226639+0000) 2020-02-05T22:22:46.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:46.224+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.224708+0000) 2020-02-05T22:22:46.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:46.226+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.226519+0000) 2020-02-05T22:22:46.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:46.225+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.225494+0000) 2020-02-05T22:22:46.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:46.226+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.227162+0000) 2020-02-05T22:22:46.230 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:46.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.227869+0000) 2020-02-05T22:22:46.230 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:46.227+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.228420+0000) 2020-02-05T22:22:46.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:46.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.228432+0000) 2020-02-05T22:22:46.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:46.226+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.227273+0000) 2020-02-05T22:22:46.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:46.227+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.228203+0000) 2020-02-05T22:22:46.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:46.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:46.233601+0000) 2020-02-05T22:22:46.244 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:46.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113423 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:47.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:47.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:47.073915+0000) 2020-02-05T22:22:47.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:47.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:47.074327+0000) 2020-02-05T22:22:47.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:47.074+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:47.074498+0000) 2020-02-05T22:22:49.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:49.464+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:49.464663+0000) 2020-02-05T22:22:49.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:49.966+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:49.966911+0000) 2020-02-05T22:22:50.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:50.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:50.071411+0000) 2020-02-05T22:22:50.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:50.305+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:50.306740+0000) 2020-02-05T22:22:50.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:50.323+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:50.323652+0000) 2020-02-05T22:22:51.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:51.240+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.241788+0000) 2020-02-05T22:22:51.243 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:51.241+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.242868+0000) 2020-02-05T22:22:51.244 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:51.242+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.243849+0000) 2020-02-05T22:22:51.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:51.242+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.244287+0000) 2020-02-05T22:22:51.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:51.243+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.244742+0000) 2020-02-05T22:22:51.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:51.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.244670+0000) 2020-02-05T22:22:51.246 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:51.244+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.245321+0000) 2020-02-05T22:22:51.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:51.245+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.246221+0000) 2020-02-05T22:22:51.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:51.245+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.246500+0000) 2020-02-05T22:22:51.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:51.245+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.247182+0000) 2020-02-05T22:22:51.249 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:51.246+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.247610+0000) 2020-02-05T22:22:51.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:51.246+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.248034+0000) 2020-02-05T22:22:51.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:51.246+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.248204+0000) 2020-02-05T22:22:51.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:51.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:51.251789+0000) 2020-02-05T22:22:51.256 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:51.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:52.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:52.318+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:52.319651+0000) 2020-02-05T22:22:52.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:52.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:52.714573+0000) 2020-02-05T22:22:53.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:53.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:53.832221+0000) 2020-02-05T22:22:53.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:53.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:53.832383+0000) 2020-02-05T22:22:55.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:55.926+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:55.927319+0000) 2020-02-05T22:22:55.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:55.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:55.934833+0000) 2020-02-05T22:22:56.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:56.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.261063+0000) 2020-02-05T22:22:56.263 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:22:56.260+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.261875+0000) 2020-02-05T22:22:56.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:22:56.262+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.263229+0000) 2020-02-05T22:22:56.265 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:22:56.263+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.264139+0000) 2020-02-05T22:22:56.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:22:56.262+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.263681+0000) 2020-02-05T22:22:56.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:56.262+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.263784+0000) 2020-02-05T22:22:56.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:22:56.263+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.264714+0000) 2020-02-05T22:22:56.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:22:56.263+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.264863+0000) 2020-02-05T22:22:56.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:56.263+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.264759+0000) 2020-02-05T22:22:56.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:22:56.263+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.264955+0000) 2020-02-05T22:22:56.268 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:22:56.264+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.265668+0000) 2020-02-05T22:22:56.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:56.264+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.265767+0000) 2020-02-05T22:22:56.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:56.264+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.266284+0000) 2020-02-05T22:22:56.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:56.274+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:56.275610+0000) 2020-02-05T22:22:56.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:22:56.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:22:57.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:22:57.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:57.074142+0000) 2020-02-05T22:22:57.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:22:57.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:57.074514+0000) 2020-02-05T22:22:57.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:22:57.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:57.074682+0000) 2020-02-05T22:22:59.388 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:22:59.394 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:22:59.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:59.463+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:59.464883+0000) 2020-02-05T22:22:59.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:22:59.974+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:22:59.975028+0000) 2020-02-05T22:23:00.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:00.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:00.071631+0000) 2020-02-05T22:23:00.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:00.306+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:00.307057+0000) 2020-02-05T22:23:00.325 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:00.322+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:00.323880+0000) 2020-02-05T22:23:01.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:01.280+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.281783+0000) 2020-02-05T22:23:01.283 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:01.281+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.282324+0000) 2020-02-05T22:23:01.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:01.281+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.282729+0000) 2020-02-05T22:23:01.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:01.282+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.283395+0000) 2020-02-05T22:23:01.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:01.282+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.283816+0000) 2020-02-05T22:23:01.285 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:01.283+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.284530+0000) 2020-02-05T22:23:01.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:01.283+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.284885+0000) 2020-02-05T22:23:01.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:01.284+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.285519+0000) 2020-02-05T22:23:01.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:01.284+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.286247+0000) 2020-02-05T22:23:01.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:01.285+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.287304+0000) 2020-02-05T22:23:01.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:01.286+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.288321+0000) 2020-02-05T22:23:01.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:01.286+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.288393+0000) 2020-02-05T22:23:01.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:01.286+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.288308+0000) 2020-02-05T22:23:01.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:01.293+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:01.294702+0000) 2020-02-05T22:23:01.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:01.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113549 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:02.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:02.318+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:02.319934+0000) 2020-02-05T22:23:02.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:02.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:02.714829+0000) 2020-02-05T22:23:03.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:03.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:03.832438+0000) 2020-02-05T22:23:03.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:03.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:03.832652+0000) 2020-02-05T22:23:05.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:05.926+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:05.927606+0000) 2020-02-05T22:23:05.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:05.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:05.935087+0000) 2020-02-05T22:23:06.302 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:06.300+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.301694+0000) 2020-02-05T22:23:06.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:06.301+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.302794+0000) 2020-02-05T22:23:06.304 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:06.302+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.303462+0000) 2020-02-05T22:23:06.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:06.301+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.303222+0000) 2020-02-05T22:23:06.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:06.303+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.304309+0000) 2020-02-05T22:23:06.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:06.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.304422+0000) 2020-02-05T22:23:06.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:06.303+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.304768+0000) 2020-02-05T22:23:06.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:06.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.305321+0000) 2020-02-05T22:23:06.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:06.304+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.305693+0000) 2020-02-05T22:23:06.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:06.304+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.306068+0000) 2020-02-05T22:23:06.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:06.305+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.307255+0000) 2020-02-05T22:23:06.309 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:06.305+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.307196+0000) 2020-02-05T22:23:06.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:06.306+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.307565+0000) 2020-02-05T22:23:06.319 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:06.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:06.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:06.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:06.311563+0000) 2020-02-05T22:23:07.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:07.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:07.074367+0000) 2020-02-05T22:23:07.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:07.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:07.074711+0000) 2020-02-05T22:23:07.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:07.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:07.074853+0000) 2020-02-05T22:23:09.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:09.464+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:09.465132+0000) 2020-02-05T22:23:09.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:09.982+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:09.983177+0000) 2020-02-05T22:23:10.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:10.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:10.071890+0000) 2020-02-05T22:23:10.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:10.306+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:10.307322+0000) 2020-02-05T22:23:10.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:10.323+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:10.324120+0000) 2020-02-05T22:23:11.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:11.319+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.320701+0000) 2020-02-05T22:23:11.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:11.320+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.321702+0000) 2020-02-05T22:23:11.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:11.321+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.322825+0000) 2020-02-05T22:23:11.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:11.321+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.323199+0000) 2020-02-05T22:23:11.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:11.322+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.323537+0000) 2020-02-05T22:23:11.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:11.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.323985+0000) 2020-02-05T22:23:11.325 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:11.323+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.324575+0000) 2020-02-05T22:23:11.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:11.322+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.324289+0000) 2020-02-05T22:23:11.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:11.324+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.325546+0000) 2020-02-05T22:23:11.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:11.325+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.326869+0000) 2020-02-05T22:23:11.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:11.325+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.326678+0000) 2020-02-05T22:23:11.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:11.324+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.325788+0000) 2020-02-05T22:23:11.331 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:11.324+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.325351+0000) 2020-02-05T22:23:11.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:11.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:11.330646+0000) 2020-02-05T22:23:11.335 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:11.333+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:12.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:12.319+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:12.320181+0000) 2020-02-05T22:23:12.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:12.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:12.715064+0000) 2020-02-05T22:23:13.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:13.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:13.832728+0000) 2020-02-05T22:23:13.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:13.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:13.832875+0000) 2020-02-05T22:23:15.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:15.927+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:15.927876+0000) 2020-02-05T22:23:15.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:15.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:15.935334+0000) 2020-02-05T22:23:16.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:16.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.339697+0000) 2020-02-05T22:23:16.341 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:16.339+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.340593+0000) 2020-02-05T22:23:16.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:16.340+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.341230+0000) 2020-02-05T22:23:16.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:16.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.342167+0000) 2020-02-05T22:23:16.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:16.342+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.343915+0000) 2020-02-05T22:23:16.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:16.343+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.344614+0000) 2020-02-05T22:23:16.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:16.341+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.342998+0000) 2020-02-05T22:23:16.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:16.342+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.344266+0000) 2020-02-05T22:23:16.347 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:16.343+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.345264+0000) 2020-02-05T22:23:16.347 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:16.344+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.345939+0000) 2020-02-05T22:23:16.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:16.344+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.346171+0000) 2020-02-05T22:23:16.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:16.345+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.346857+0000) 2020-02-05T22:23:16.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:16.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.348164+0000) 2020-02-05T22:23:16.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:16.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:16.348427+0000) 2020-02-05T22:23:16.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:16.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113671 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:17.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:17.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:17.074639+0000) 2020-02-05T22:23:17.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:17.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:17.074936+0000) 2020-02-05T22:23:17.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:17.073+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:17.075032+0000) 2020-02-05T22:23:19.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:19.464+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:19.465373+0000) 2020-02-05T22:23:19.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:19.992+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:19.993904+0000) 2020-02-05T22:23:20.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:20.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:20.072132+0000) 2020-02-05T22:23:20.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:20.306+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:20.307573+0000) 2020-02-05T22:23:20.325 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:20.323+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:20.324367+0000) 2020-02-05T22:23:21.361 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:21.358+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.359968+0000) 2020-02-05T22:23:21.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:21.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.360417+0000) 2020-02-05T22:23:21.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:21.360+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.361180+0000) 2020-02-05T22:23:21.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:21.360+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.361795+0000) 2020-02-05T22:23:21.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:21.360+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.362067+0000) 2020-02-05T22:23:21.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:21.361+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.362684+0000) 2020-02-05T22:23:21.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:21.362+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.363440+0000) 2020-02-05T22:23:21.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:21.362+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.363674+0000) 2020-02-05T22:23:21.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:21.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.364395+0000) 2020-02-05T22:23:21.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:21.364+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.365263+0000) 2020-02-05T22:23:21.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:21.364+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.365561+0000) 2020-02-05T22:23:21.367 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:21.364+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.365873+0000) 2020-02-05T22:23:21.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:21.364+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.365917+0000) 2020-02-05T22:23:21.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:21.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113715 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:21.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:21.368+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:21.369993+0000) 2020-02-05T22:23:22.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:22.319+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:22.320470+0000) 2020-02-05T22:23:22.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:22.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:22.715285+0000) 2020-02-05T22:23:23.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:23.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:23.832960+0000) 2020-02-05T22:23:23.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:23.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:23.833119+0000) 2020-02-05T22:23:25.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:25.927+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:25.928148+0000) 2020-02-05T22:23:25.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:25.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:25.935571+0000) 2020-02-05T22:23:26.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:26.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.378738+0000) 2020-02-05T22:23:26.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:26.378+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.379610+0000) 2020-02-05T22:23:26.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:26.379+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.380419+0000) 2020-02-05T22:23:26.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:26.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.380968+0000) 2020-02-05T22:23:26.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:26.379+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.381332+0000) 2020-02-05T22:23:26.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:26.381+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.382139+0000) 2020-02-05T22:23:26.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:26.381+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.382470+0000) 2020-02-05T22:23:26.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:26.382+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.383309+0000) 2020-02-05T22:23:26.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:26.382+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.383635+0000) 2020-02-05T22:23:26.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:26.382+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.383422+0000) 2020-02-05T22:23:26.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:26.382+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.384318+0000) 2020-02-05T22:23:26.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:26.383+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.384789+0000) 2020-02-05T22:23:26.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:26.383+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.385126+0000) 2020-02-05T22:23:26.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:26.388+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:26.389449+0000) 2020-02-05T22:23:26.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:26.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:27.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:27.074+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:27.074904+0000) 2020-02-05T22:23:27.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:27.074+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:27.075192+0000) 2020-02-05T22:23:27.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:27.074+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:27.075512+0000) 2020-02-05T22:23:29.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:29.465+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:29.465615+0000) 2020-02-05T22:23:29.528 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:23:29.535 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:23:30.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:30.001+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:30.002995+0000) 2020-02-05T22:23:30.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:30.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:30.072362+0000) 2020-02-05T22:23:30.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:30.307+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:30.307803+0000) 2020-02-05T22:23:30.325 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:30.324+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:30.324590+0000) 2020-02-05T22:23:31.411 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:31.399+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.401037+0000) 2020-02-05T22:23:31.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:31.400+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.402347+0000) 2020-02-05T22:23:31.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:31.401+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.403366+0000) 2020-02-05T22:23:31.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:31.402+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.403483+0000) 2020-02-05T22:23:31.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:31.402+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.404197+0000) 2020-02-05T22:23:31.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:31.403+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.404885+0000) 2020-02-05T22:23:31.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:31.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.399414+0000) 2020-02-05T22:23:31.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:31.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.404151+0000) 2020-02-05T22:23:31.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:31.406+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.407779+0000) 2020-02-05T22:23:31.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:31.398+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.399943+0000) 2020-02-05T22:23:31.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:31.399+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.400300+0000) 2020-02-05T22:23:31.413 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:31.400+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.401524+0000) 2020-02-05T22:23:31.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:31.401+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.402661+0000) 2020-02-05T22:23:31.414 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:31.402+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:31.403981+0000) 2020-02-05T22:23:31.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:31.409+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:32.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:32.320+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:32.320714+0000) 2020-02-05T22:23:32.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:32.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:32.715523+0000) 2020-02-05T22:23:33.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:33.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:33.833157+0000) 2020-02-05T22:23:33.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:33.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:33.833320+0000) 2020-02-05T22:23:35.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:35.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:35.928371+0000) 2020-02-05T22:23:35.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:35.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:35.935844+0000) 2020-02-05T22:23:36.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:36.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.416926+0000) 2020-02-05T22:23:36.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:36.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.417373+0000) 2020-02-05T22:23:36.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:36.416+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.417995+0000) 2020-02-05T22:23:36.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:36.417+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.418490+0000) 2020-02-05T22:23:36.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:36.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.419318+0000) 2020-02-05T22:23:36.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:36.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.419947+0000) 2020-02-05T22:23:36.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:36.419+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.420963+0000) 2020-02-05T22:23:36.423 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:36.420+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.422029+0000) 2020-02-05T22:23:36.423 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:36.420+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.422383+0000) 2020-02-05T22:23:36.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:36.421+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.422391+0000) 2020-02-05T22:23:36.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:36.421+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.422500+0000) 2020-02-05T22:23:36.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:36.420+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.421699+0000) 2020-02-05T22:23:36.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:36.421+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.422995+0000) 2020-02-05T22:23:36.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:36.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:36.427449+0000) 2020-02-05T22:23:36.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:36.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:37.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:37.073+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:37.075106+0000) 2020-02-05T22:23:37.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:37.073+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:37.075395+0000) 2020-02-05T22:23:37.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:37.074+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:37.075685+0000) 2020-02-05T22:23:39.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:39.465+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:39.465846+0000) 2020-02-05T22:23:40.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:40.012+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:40.012648+0000) 2020-02-05T22:23:40.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:40.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:40.072693+0000) 2020-02-05T22:23:40.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:40.307+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:40.308044+0000) 2020-02-05T22:23:40.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:40.324+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:40.324820+0000) 2020-02-05T22:23:41.437 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:41.435+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.436358+0000) 2020-02-05T22:23:41.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:41.437+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.437822+0000) 2020-02-05T22:23:41.439 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:41.438+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.438779+0000) 2020-02-05T22:23:41.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:41.439+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.439305+0000) 2020-02-05T22:23:41.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:41.439+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.439798+0000) 2020-02-05T22:23:41.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:41.439+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.439840+0000) 2020-02-05T22:23:41.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:41.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.440272+0000) 2020-02-05T22:23:41.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:41.439+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.440789+0000) 2020-02-05T22:23:41.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:41.440+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.441181+0000) 2020-02-05T22:23:41.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:41.440+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.440802+0000) 2020-02-05T22:23:41.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:41.440+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.441211+0000) 2020-02-05T22:23:41.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:41.440+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.441859+0000) 2020-02-05T22:23:41.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:41.441+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.442124+0000) 2020-02-05T22:23:41.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:41.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:41.444656+0000) 2020-02-05T22:23:41.450 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:41.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:42.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:42.320+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:42.320933+0000) 2020-02-05T22:23:42.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:42.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:42.715783+0000) 2020-02-05T22:23:43.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:43.833+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:43.833386+0000) 2020-02-05T22:23:43.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:43.833+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:43.833524+0000) 2020-02-05T22:23:45.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:45.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:45.928605+0000) 2020-02-05T22:23:45.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:45.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:45.936078+0000) 2020-02-05T22:23:46.456 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:46.454+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.455034+0000) 2020-02-05T22:23:46.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:46.455+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.455466+0000) 2020-02-05T22:23:46.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:46.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.456056+0000) 2020-02-05T22:23:46.457 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:46.455+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.456501+0000) 2020-02-05T22:23:46.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:46.455+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.456109+0000) 2020-02-05T22:23:46.458 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:46.455+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.456846+0000) 2020-02-05T22:23:46.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:46.456+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.457217+0000) 2020-02-05T22:23:46.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:46.456+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.456946+0000) 2020-02-05T22:23:46.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:46.456+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.458024+0000) 2020-02-05T22:23:46.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:46.457+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.458460+0000) 2020-02-05T22:23:46.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:46.457+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.457802+0000) 2020-02-05T22:23:46.460 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:46.458+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.459012+0000) 2020-02-05T22:23:46.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:46.458+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.459185+0000) 2020-02-05T22:23:46.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:46.463+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:46.464572+0000) 2020-02-05T22:23:46.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:46.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:47.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:47.074+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:47.075352+0000) 2020-02-05T22:23:47.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:47.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:47.075648+0000) 2020-02-05T22:23:47.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:47.075+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:47.075882+0000) 2020-02-05T22:23:49.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:49.465+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:49.466111+0000) 2020-02-05T22:23:50.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:50.019+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:50.020421+0000) 2020-02-05T22:23:50.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:50.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:50.072962+0000) 2020-02-05T22:23:50.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:50.307+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:50.308289+0000) 2020-02-05T22:23:50.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:50.324+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:50.325050+0000) 2020-02-05T22:23:51.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:51.474+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.475855+0000) 2020-02-05T22:23:51.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:51.475+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.475548+0000) 2020-02-05T22:23:51.478 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:51.475+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.476435+0000) 2020-02-05T22:23:51.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:51.475+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.477047+0000) 2020-02-05T22:23:51.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:51.476+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.477803+0000) 2020-02-05T22:23:51.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:51.477+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.477906+0000) 2020-02-05T22:23:51.479 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:51.477+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.478362+0000) 2020-02-05T22:23:51.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:51.477+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.478083+0000) 2020-02-05T22:23:51.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:51.478+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.479303+0000) 2020-02-05T22:23:51.480 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:51.478+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.479749+0000) 2020-02-05T22:23:51.482 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:51.480+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.481365+0000) 2020-02-05T22:23:51.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:51.480+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.481404+0000) 2020-02-05T22:23:51.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:51.481+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.481673+0000) 2020-02-05T22:23:51.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:51.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:51.487134+0000) 2020-02-05T22:23:51.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:51.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 113968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:52.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:52.320+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:52.321166+0000) 2020-02-05T22:23:52.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:52.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:52.716004+0000) 2020-02-05T22:23:53.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:53.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:53.833639+0000) 2020-02-05T22:23:53.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:53.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:53.833730+0000) 2020-02-05T22:23:55.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:55.927+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:55.928839+0000) 2020-02-05T22:23:55.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:55.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:55.936332+0000) 2020-02-05T22:23:56.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:56.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.496367+0000) 2020-02-05T22:23:56.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:23:56.495+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.496913+0000) 2020-02-05T22:23:56.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:23:56.497+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.497487+0000) 2020-02-05T22:23:56.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:56.497+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.498131+0000) 2020-02-05T22:23:56.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:23:56.497+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.498430+0000) 2020-02-05T22:23:56.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:23:56.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.498670+0000) 2020-02-05T22:23:56.500 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:56.498+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.499164+0000) 2020-02-05T22:23:56.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:23:56.498+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.499894+0000) 2020-02-05T22:23:56.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:23:56.499+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.500640+0000) 2020-02-05T22:23:56.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:56.500+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.501049+0000) 2020-02-05T22:23:56.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:23:56.500+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.500828+0000) 2020-02-05T22:23:56.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:23:56.500+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.500514+0000) 2020-02-05T22:23:56.503 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:56.500+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.501263+0000) 2020-02-05T22:23:56.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:56.504+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:56.506040+0000) 2020-02-05T22:23:56.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:23:56.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:23:57.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:23:57.075+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:57.075584+0000) 2020-02-05T22:23:57.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:23:57.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:57.075853+0000) 2020-02-05T22:23:57.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:23:57.075+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:57.076061+0000) 2020-02-05T22:23:59.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:23:59.465+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:23:59.466308+0000) 2020-02-05T22:23:59.715 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:23:59.722 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:24:00.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:00.027+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:00.028426+0000) 2020-02-05T22:24:00.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:00.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:00.073201+0000) 2020-02-05T22:24:00.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:00.308+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:00.308532+0000) 2020-02-05T22:24:00.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:00.324+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:00.325352+0000) 2020-02-05T22:24:01.516 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:01.514+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.515710+0000) 2020-02-05T22:24:01.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:01.515+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.516149+0000) 2020-02-05T22:24:01.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:01.515+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.516651+0000) 2020-02-05T22:24:01.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:01.516+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.517263+0000) 2020-02-05T22:24:01.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:01.516+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.517700+0000) 2020-02-05T22:24:01.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:01.517+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.518165+0000) 2020-02-05T22:24:01.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:01.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.518632+0000) 2020-02-05T22:24:01.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:01.517+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.519015+0000) 2020-02-05T22:24:01.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:01.519+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.519707+0000) 2020-02-05T22:24:01.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:01.518+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.520015+0000) 2020-02-05T22:24:01.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:01.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.520779+0000) 2020-02-05T22:24:01.522 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:01.520+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.520924+0000) 2020-02-05T22:24:01.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:01.521+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.521620+0000) 2020-02-05T22:24:01.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:01.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:01.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:01.528+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:01.529801+0000) 2020-02-05T22:24:02.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:02.321+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:02.321379+0000) 2020-02-05T22:24:02.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:02.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:02.716235+0000) 2020-02-05T22:24:03.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:03.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:03.833880+0000) 2020-02-05T22:24:03.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:03.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:03.833932+0000) 2020-02-05T22:24:05.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:05.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:05.929115+0000) 2020-02-05T22:24:05.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:05.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:05.936582+0000) 2020-02-05T22:24:06.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:06.532+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.534032+0000) 2020-02-05T22:24:06.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:06.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.535036+0000) 2020-02-05T22:24:06.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:06.535+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.535483+0000) 2020-02-05T22:24:06.537 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:06.534+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.535909+0000) 2020-02-05T22:24:06.537 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:06.535+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.536179+0000) 2020-02-05T22:24:06.538 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:06.535+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.536737+0000) 2020-02-05T22:24:06.538 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:06.536+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.536825+0000) 2020-02-05T22:24:06.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:06.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.537393+0000) 2020-02-05T22:24:06.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:06.536+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.537731+0000) 2020-02-05T22:24:06.539 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:06.537+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.538496+0000) 2020-02-05T22:24:06.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:06.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.539240+0000) 2020-02-05T22:24:06.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:06.539+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.540220+0000) 2020-02-05T22:24:06.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:06.539+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.539797+0000) 2020-02-05T22:24:06.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:06.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:06.545845+0000) 2020-02-05T22:24:06.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:06.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:07.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:07.075+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:07.075906+0000) 2020-02-05T22:24:07.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:07.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:07.076076+0000) 2020-02-05T22:24:07.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:07.075+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:07.076239+0000) 2020-02-05T22:24:09.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:09.465+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:09.466558+0000) 2020-02-05T22:24:10.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:10.035+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:10.036604+0000) 2020-02-05T22:24:10.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:10.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:10.073443+0000) 2020-02-05T22:24:10.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:10.308+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:10.308779+0000) 2020-02-05T22:24:10.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:10.324+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:10.325541+0000) 2020-02-05T22:24:11.556 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:11.554+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.554901+0000) 2020-02-05T22:24:11.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:11.554+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.555536+0000) 2020-02-05T22:24:11.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:11.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.556023+0000) 2020-02-05T22:24:11.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:11.555+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.556262+0000) 2020-02-05T22:24:11.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:11.556+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.556475+0000) 2020-02-05T22:24:11.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:11.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.557018+0000) 2020-02-05T22:24:11.559 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:11.556+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.557655+0000) 2020-02-05T22:24:11.559 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:11.556+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.557652+0000) 2020-02-05T22:24:11.559 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:11.557+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.558371+0000) 2020-02-05T22:24:11.560 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:11.557+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.558413+0000) 2020-02-05T22:24:11.560 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:11.558+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.559076+0000) 2020-02-05T22:24:11.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:11.558+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.559374+0000) 2020-02-05T22:24:11.561 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:11.559+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.560280+0000) 2020-02-05T22:24:11.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:11.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:11.563852+0000) 2020-02-05T22:24:11.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:11.566+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114137 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:12.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:12.321+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:12.321606+0000) 2020-02-05T22:24:12.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:12.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:12.716441+0000) 2020-02-05T22:24:13.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:13.833+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:13.834158+0000) 2020-02-05T22:24:13.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:13.833+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:13.834158+0000) 2020-02-05T22:24:15.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:15.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:15.929320+0000) 2020-02-05T22:24:15.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:15.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:15.936832+0000) 2020-02-05T22:24:16.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:16.570+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.571802+0000) 2020-02-05T22:24:16.574 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:16.572+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.573615+0000) 2020-02-05T22:24:16.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:16.573+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.573416+0000) 2020-02-05T22:24:16.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:16.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.574284+0000) 2020-02-05T22:24:16.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:16.574+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.574826+0000) 2020-02-05T22:24:16.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:16.574+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.574757+0000) 2020-02-05T22:24:16.577 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:16.574+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.575466+0000) 2020-02-05T22:24:16.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:16.574+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.575972+0000) 2020-02-05T22:24:16.577 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:16.575+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.576409+0000) 2020-02-05T22:24:16.578 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:16.576+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.577150+0000) 2020-02-05T22:24:16.578 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:16.576+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.577067+0000) 2020-02-05T22:24:16.578 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:16.577+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.577803+0000) 2020-02-05T22:24:16.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:16.577+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.578363+0000) 2020-02-05T22:24:16.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:16.582+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:16.583280+0000) 2020-02-05T22:24:16.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:16.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114179 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:17.077 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:17.076+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:17.076121+0000) 2020-02-05T22:24:17.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:17.076+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:17.076302+0000) 2020-02-05T22:24:17.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:17.076+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:17.076502+0000) 2020-02-05T22:24:19.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:19.466+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:19.466824+0000) 2020-02-05T22:24:20.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:20.043+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:20.044947+0000) 2020-02-05T22:24:20.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:20.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:20.073683+0000) 2020-02-05T22:24:20.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:20.307+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:20.309036+0000) 2020-02-05T22:24:20.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:20.325+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:20.325780+0000) 2020-02-05T22:24:21.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:21.592+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.594243+0000) 2020-02-05T22:24:21.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:21.593+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.594479+0000) 2020-02-05T22:24:21.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:21.593+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.594795+0000) 2020-02-05T22:24:21.596 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:21.594+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.595231+0000) 2020-02-05T22:24:21.597 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:21.594+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.595372+0000) 2020-02-05T22:24:21.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:21.594+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.595701+0000) 2020-02-05T22:24:21.597 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:21.595+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.596291+0000) 2020-02-05T22:24:21.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:21.595+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.597139+0000) 2020-02-05T22:24:21.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:21.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.597991+0000) 2020-02-05T22:24:21.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:21.596+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.597756+0000) 2020-02-05T22:24:21.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:21.595+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.596795+0000) 2020-02-05T22:24:21.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:21.596+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.598262+0000) 2020-02-05T22:24:21.600 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:21.596+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.598347+0000) 2020-02-05T22:24:21.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:21.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:21.601706+0000) 2020-02-05T22:24:21.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:21.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:22.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:22.320+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:22.321841+0000) 2020-02-05T22:24:22.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:22.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:22.716701+0000) 2020-02-05T22:24:23.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:23.833+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:23.834427+0000) 2020-02-05T22:24:23.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:23.833+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:23.834427+0000) 2020-02-05T22:24:25.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:25.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:25.929564+0000) 2020-02-05T22:24:25.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:25.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:25.937088+0000) 2020-02-05T22:24:26.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:26.614+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.615692+0000) 2020-02-05T22:24:26.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:26.614+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.616037+0000) 2020-02-05T22:24:26.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:26.614+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.616331+0000) 2020-02-05T22:24:26.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:26.615+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.616822+0000) 2020-02-05T22:24:26.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:26.616+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.617816+0000) 2020-02-05T22:24:26.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:26.617+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.618438+0000) 2020-02-05T22:24:26.625 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:26.618+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.619610+0000) 2020-02-05T22:24:26.625 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:26.618+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.619538+0000) 2020-02-05T22:24:26.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:26.619+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.620767+0000) 2020-02-05T22:24:26.625 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:26.619+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.621319+0000) 2020-02-05T22:24:26.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:26.619+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.621364+0000) 2020-02-05T22:24:26.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:26.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.619252+0000) 2020-02-05T22:24:26.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:26.618+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.619998+0000) 2020-02-05T22:24:26.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:26.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114261 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:26.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:26.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:26.626909+0000) 2020-02-05T22:24:27.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:27.075+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:27.076402+0000) 2020-02-05T22:24:27.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:27.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:27.076538+0000) 2020-02-05T22:24:27.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:27.075+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:27.076707+0000) 2020-02-05T22:24:29.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:29.466+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:29.467134+0000) 2020-02-05T22:24:29.880 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:24:29.894 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:24:30.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:30.053+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:30.053312+0000) 2020-02-05T22:24:30.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:30.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:30.073933+0000) 2020-02-05T22:24:30.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:30.308+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:30.309291+0000) 2020-02-05T22:24:30.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:30.325+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:30.326022+0000) 2020-02-05T22:24:31.635 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:31.633+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.634067+0000) 2020-02-05T22:24:31.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:31.633+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.634677+0000) 2020-02-05T22:24:31.636 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:31.634+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.635235+0000) 2020-02-05T22:24:31.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:31.634+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.635474+0000) 2020-02-05T22:24:31.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:31.634+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.635837+0000) 2020-02-05T22:24:31.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:31.634+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.636203+0000) 2020-02-05T22:24:31.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:31.635+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.636709+0000) 2020-02-05T22:24:31.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:31.636+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.637254+0000) 2020-02-05T22:24:31.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:31.636+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.637635+0000) 2020-02-05T22:24:31.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:31.637+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.638815+0000) 2020-02-05T22:24:31.640 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:31.637+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.639397+0000) 2020-02-05T22:24:31.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:31.638+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.639506+0000) 2020-02-05T22:24:31.641 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:31.638+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.639916+0000) 2020-02-05T22:24:31.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:31.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:31.644584+0000) 2020-02-05T22:24:31.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:31.646+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:32.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:32.321+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:32.322074+0000) 2020-02-05T22:24:32.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:32.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:32.716932+0000) 2020-02-05T22:24:33.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:33.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:33.834722+0000) 2020-02-05T22:24:33.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:33.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:33.834722+0000) 2020-02-05T22:24:35.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:35.928+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:35.929856+0000) 2020-02-05T22:24:35.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:35.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:35.937340+0000) 2020-02-05T22:24:36.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:36.653+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.653973+0000) 2020-02-05T22:24:36.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:36.653+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.654637+0000) 2020-02-05T22:24:36.657 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:36.654+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.656009+0000) 2020-02-05T22:24:36.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:36.656+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.656758+0000) 2020-02-05T22:24:36.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:36.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.657616+0000) 2020-02-05T22:24:36.659 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:36.655+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.657323+0000) 2020-02-05T22:24:36.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:36.658+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.658342+0000) 2020-02-05T22:24:36.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:36.658+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.658583+0000) 2020-02-05T22:24:36.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:36.659+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.659689+0000) 2020-02-05T22:24:36.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:36.658+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.659829+0000) 2020-02-05T22:24:36.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:36.658+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.660245+0000) 2020-02-05T22:24:36.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:36.659+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.660602+0000) 2020-02-05T22:24:36.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:36.660+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.660616+0000) 2020-02-05T22:24:36.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:36.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:36.666022+0000) 2020-02-05T22:24:36.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:36.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114343 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:37.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:37.075+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:37.076666+0000) 2020-02-05T22:24:37.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:37.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:37.076764+0000) 2020-02-05T22:24:37.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:37.075+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:37.076918+0000) 2020-02-05T22:24:39.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:39.467+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:39.467362+0000) 2020-02-05T22:24:40.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:40.065+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:40.065409+0000) 2020-02-05T22:24:40.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:40.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:40.074155+0000) 2020-02-05T22:24:40.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:40.308+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:40.309531+0000) 2020-02-05T22:24:40.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:40.326+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:40.326306+0000) 2020-02-05T22:24:41.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:41.674+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.675710+0000) 2020-02-05T22:24:41.677 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:41.675+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.676595+0000) 2020-02-05T22:24:41.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:41.675+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.676811+0000) 2020-02-05T22:24:41.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:41.676+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.677998+0000) 2020-02-05T22:24:41.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:41.677+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.678219+0000) 2020-02-05T22:24:41.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:41.677+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.679282+0000) 2020-02-05T22:24:41.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:41.678+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.679621+0000) 2020-02-05T22:24:41.681 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:41.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.680271+0000) 2020-02-05T22:24:41.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:41.679+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.680491+0000) 2020-02-05T22:24:41.682 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:41.679+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.680744+0000) 2020-02-05T22:24:41.682 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:41.679+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.681247+0000) 2020-02-05T22:24:41.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:41.680+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.681061+0000) 2020-02-05T22:24:41.683 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:41.680+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.681595+0000) 2020-02-05T22:24:41.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:41.681+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:41.682841+0000) 2020-02-05T22:24:41.690 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:41.687+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:42.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:42.321+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:42.322318+0000) 2020-02-05T22:24:42.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:42.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:42.717178+0000) 2020-02-05T22:24:43.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:43.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:43.835027+0000) 2020-02-05T22:24:43.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:43.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:43.835027+0000) 2020-02-05T22:24:45.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:45.929+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:45.930118+0000) 2020-02-05T22:24:45.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:45.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:45.937594+0000) 2020-02-05T22:24:46.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:46.693+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.695007+0000) 2020-02-05T22:24:46.697 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:46.695+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.696056+0000) 2020-02-05T22:24:46.697 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:46.694+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.696164+0000) 2020-02-05T22:24:46.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:46.695+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.696858+0000) 2020-02-05T22:24:46.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:46.696+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.697147+0000) 2020-02-05T22:24:46.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:46.696+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.697239+0000) 2020-02-05T22:24:46.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:46.696+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.697839+0000) 2020-02-05T22:24:46.700 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:46.697+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.698582+0000) 2020-02-05T22:24:46.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:46.696+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.697820+0000) 2020-02-05T22:24:46.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:46.696+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.697897+0000) 2020-02-05T22:24:46.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:46.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.698744+0000) 2020-02-05T22:24:46.701 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:46.697+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.699051+0000) 2020-02-05T22:24:46.701 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:46.699+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.700126+0000) 2020-02-05T22:24:46.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:46.702+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:46.703802+0000) 2020-02-05T22:24:46.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:46.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:47.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:47.076+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:47.076905+0000) 2020-02-05T22:24:47.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:47.076+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:47.077024+0000) 2020-02-05T22:24:47.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:47.076+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:47.077152+0000) 2020-02-05T22:24:49.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:49.466+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:49.467605+0000) 2020-02-05T22:24:50.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:50.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:50.074427+0000) 2020-02-05T22:24:50.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:50.073+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:50.074700+0000) 2020-02-05T22:24:50.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:50.309+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:50.309783+0000) 2020-02-05T22:24:50.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:50.325+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:50.326534+0000) 2020-02-05T22:24:51.714 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:51.711+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.712669+0000) 2020-02-05T22:24:51.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:51.711+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.713349+0000) 2020-02-05T22:24:51.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:51.713+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.714626+0000) 2020-02-05T22:24:51.717 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:51.714+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.716136+0000) 2020-02-05T22:24:51.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:51.715+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.716702+0000) 2020-02-05T22:24:51.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:51.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.718227+0000) 2020-02-05T22:24:51.719 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:51.716+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.718201+0000) 2020-02-05T22:24:51.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:51.717+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.718590+0000) 2020-02-05T22:24:51.720 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:51.717+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.719313+0000) 2020-02-05T22:24:51.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:51.718+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.719316+0000) 2020-02-05T22:24:51.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:51.718+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.719543+0000) 2020-02-05T22:24:51.721 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:51.718+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.719634+0000) 2020-02-05T22:24:51.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:51.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.722163+0000) 2020-02-05T22:24:51.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:51.722+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:51.723245+0000) 2020-02-05T22:24:51.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:51.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:52.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:52.321+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:52.322565+0000) 2020-02-05T22:24:52.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:52.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:52.717417+0000) 2020-02-05T22:24:53.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:53.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:53.835323+0000) 2020-02-05T22:24:53.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:53.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:53.835322+0000) 2020-02-05T22:24:55.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:55.929+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:55.930362+0000) 2020-02-05T22:24:55.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:55.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:55.937884+0000) 2020-02-05T22:24:56.734 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:56.731+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.732840+0000) 2020-02-05T22:24:56.734 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:56.731+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.733215+0000) 2020-02-05T22:24:56.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:24:56.732+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.733675+0000) 2020-02-05T22:24:56.735 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:24:56.733+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.734789+0000) 2020-02-05T22:24:56.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:56.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.736421+0000) 2020-02-05T22:24:56.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:24:56.735+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.737193+0000) 2020-02-05T22:24:56.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:24:56.736+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.737617+0000) 2020-02-05T22:24:56.739 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:24:56.737+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.738323+0000) 2020-02-05T22:24:56.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:24:56.737+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.738373+0000) 2020-02-05T22:24:56.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:56.737+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.738757+0000) 2020-02-05T22:24:56.740 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:24:56.737+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.739245+0000) 2020-02-05T22:24:56.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:24:56.738+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.739931+0000) 2020-02-05T22:24:56.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:56.739+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.740119+0000) 2020-02-05T22:24:56.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:56.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:56.741698+0000) 2020-02-05T22:24:56.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:24:56.746+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114507 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:24:57.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:24:57.075+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:57.077125+0000) 2020-02-05T22:24:57.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:24:57.075+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:57.077277+0000) 2020-02-05T22:24:57.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:24:57.076+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:57.077396+0000) 2020-02-05T22:24:59.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:24:59.466+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:24:59.467862+0000) 2020-02-05T22:25:00.068 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:25:00.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:00.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:00.074685+0000) 2020-02-05T22:25:00.081 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:25:00.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:00.082+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:00.083376+0000) 2020-02-05T22:25:00.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:00.309+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:00.310042+0000) 2020-02-05T22:25:00.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:00.325+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:00.326716+0000) 2020-02-05T22:25:01.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:01.752+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.753259+0000) 2020-02-05T22:25:01.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:01.752+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.753934+0000) 2020-02-05T22:25:01.755 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:01.753+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.754559+0000) 2020-02-05T22:25:01.756 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:01.754+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.755403+0000) 2020-02-05T22:25:01.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:01.754+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.755980+0000) 2020-02-05T22:25:01.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:01.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.756179+0000) 2020-02-05T22:25:01.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:01.754+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.756258+0000) 2020-02-05T22:25:01.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:01.755+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.756734+0000) 2020-02-05T22:25:01.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:01.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.757183+0000) 2020-02-05T22:25:01.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:01.756+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.757382+0000) 2020-02-05T22:25:01.759 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:01.756+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.757777+0000) 2020-02-05T22:25:01.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:01.757+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.758399+0000) 2020-02-05T22:25:01.760 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:01.757+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.759280+0000) 2020-02-05T22:25:01.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:01.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:01.763416+0000) 2020-02-05T22:25:01.768 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:01.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:02.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:02.322+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:02.322879+0000) 2020-02-05T22:25:02.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:02.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:02.717625+0000) 2020-02-05T22:25:03.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:03.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:03.835603+0000) 2020-02-05T22:25:03.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:03.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:03.835603+0000) 2020-02-05T22:25:05.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:05.929+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:05.930616+0000) 2020-02-05T22:25:05.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:05.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:05.938136+0000) 2020-02-05T22:25:06.775 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:06.772+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.774031+0000) 2020-02-05T22:25:06.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:06.772+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.773782+0000) 2020-02-05T22:25:06.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:06.773+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.774804+0000) 2020-02-05T22:25:06.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:06.774+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.775353+0000) 2020-02-05T22:25:06.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:06.774+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.775805+0000) 2020-02-05T22:25:06.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:06.774+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.775988+0000) 2020-02-05T22:25:06.777 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:06.774+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.776393+0000) 2020-02-05T22:25:06.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:06.776+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.777905+0000) 2020-02-05T22:25:06.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:06.777+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.778199+0000) 2020-02-05T22:25:06.780 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:06.777+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.778645+0000) 2020-02-05T22:25:06.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:06.777+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.778651+0000) 2020-02-05T22:25:06.780 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:06.777+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.778833+0000) 2020-02-05T22:25:06.781 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:06.777+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.779116+0000) 2020-02-05T22:25:06.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:06.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:06.782958+0000) 2020-02-05T22:25:06.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:06.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:07.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:07.076+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:07.077358+0000) 2020-02-05T22:25:07.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:07.076+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:07.077504+0000) 2020-02-05T22:25:07.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:07.076+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:07.077677+0000) 2020-02-05T22:25:09.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:09.467+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:09.468130+0000) 2020-02-05T22:25:10.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:10.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:10.074918+0000) 2020-02-05T22:25:10.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:10.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:10.091937+0000) 2020-02-05T22:25:10.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:10.309+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:10.310308+0000) 2020-02-05T22:25:10.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:10.326+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:10.326964+0000) 2020-02-05T22:25:11.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:11.792+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.793573+0000) 2020-02-05T22:25:11.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:11.794+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.795237+0000) 2020-02-05T22:25:11.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:11.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.796260+0000) 2020-02-05T22:25:11.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:11.795+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.796642+0000) 2020-02-05T22:25:11.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:11.795+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.796699+0000) 2020-02-05T22:25:11.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:11.797+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.797465+0000) 2020-02-05T22:25:11.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:11.796+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.797957+0000) 2020-02-05T22:25:11.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:11.797+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.797978+0000) 2020-02-05T22:25:11.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:11.798+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.798653+0000) 2020-02-05T22:25:11.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:11.797+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.798952+0000) 2020-02-05T22:25:11.803 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:11.799+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.799781+0000) 2020-02-05T22:25:11.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:11.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.799831+0000) 2020-02-05T22:25:11.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:11.799+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.800101+0000) 2020-02-05T22:25:11.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:11.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:11.803880+0000) 2020-02-05T22:25:11.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:11.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:12.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:12.322+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:12.323112+0000) 2020-02-05T22:25:12.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:12.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:12.717835+0000) 2020-02-05T22:25:13.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:13.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:13.835895+0000) 2020-02-05T22:25:13.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:13.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:13.835893+0000) 2020-02-05T22:25:15.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:15.929+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:15.930849+0000) 2020-02-05T22:25:15.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:15.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:15.938383+0000) 2020-02-05T22:25:16.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:16.813+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.814493+0000) 2020-02-05T22:25:16.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:16.813+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.814756+0000) 2020-02-05T22:25:16.817 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:16.815+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.816260+0000) 2020-02-05T22:25:16.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:16.815+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.816668+0000) 2020-02-05T22:25:16.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:16.817+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.817572+0000) 2020-02-05T22:25:16.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:16.817+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.817742+0000) 2020-02-05T22:25:16.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:16.817+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.818469+0000) 2020-02-05T22:25:16.819 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:16.818+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.818614+0000) 2020-02-05T22:25:16.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:16.818+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.819479+0000) 2020-02-05T22:25:16.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:16.819+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.819661+0000) 2020-02-05T22:25:16.821 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:16.819+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.819586+0000) 2020-02-05T22:25:16.821 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:16.819+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.819979+0000) 2020-02-05T22:25:16.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:16.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.820438+0000) 2020-02-05T22:25:16.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:16.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114670 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:16.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:16.825+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:16.826205+0000) 2020-02-05T22:25:17.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:17.077+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:17.077614+0000) 2020-02-05T22:25:17.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:17.077+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:17.077718+0000) 2020-02-05T22:25:17.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:17.077+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:17.077929+0000) 2020-02-05T22:25:19.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:19.468+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:19.468370+0000) 2020-02-05T22:25:20.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:20.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:20.075189+0000) 2020-02-05T22:25:20.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:20.099+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:20.100225+0000) 2020-02-05T22:25:20.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:20.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:20.310555+0000) 2020-02-05T22:25:20.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:20.326+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:20.327197+0000) 2020-02-05T22:25:21.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:21.829+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.830797+0000) 2020-02-05T22:25:21.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:21.831+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.832009+0000) 2020-02-05T22:25:21.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:21.832+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.833566+0000) 2020-02-05T22:25:21.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:21.833+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.833899+0000) 2020-02-05T22:25:21.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:21.833+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.834373+0000) 2020-02-05T22:25:21.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:21.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.835469+0000) 2020-02-05T22:25:21.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:21.835+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.835699+0000) 2020-02-05T22:25:21.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:21.835+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.836372+0000) 2020-02-05T22:25:21.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:21.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.837049+0000) 2020-02-05T22:25:21.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:21.836+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.837197+0000) 2020-02-05T22:25:21.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:21.837+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.837712+0000) 2020-02-05T22:25:21.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:21.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.837674+0000) 2020-02-05T22:25:21.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:21.837+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.838083+0000) 2020-02-05T22:25:21.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:21.843+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:21.844143+0000) 2020-02-05T22:25:21.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:21.844+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:22.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:22.322+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:22.323341+0000) 2020-02-05T22:25:22.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:22.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:22.718045+0000) 2020-02-05T22:25:23.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:23.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:23.836176+0000) 2020-02-05T22:25:23.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:23.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:23.836175+0000) 2020-02-05T22:25:25.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:25.930+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:25.931101+0000) 2020-02-05T22:25:25.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:25.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:25.938664+0000) 2020-02-05T22:25:26.853 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:26.851+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.852581+0000) 2020-02-05T22:25:26.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:26.851+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.852961+0000) 2020-02-05T22:25:26.854 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:26.852+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.853302+0000) 2020-02-05T22:25:26.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:26.852+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.853580+0000) 2020-02-05T22:25:26.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:26.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.853895+0000) 2020-02-05T22:25:26.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:26.854+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.854479+0000) 2020-02-05T22:25:26.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:26.854+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.854955+0000) 2020-02-05T22:25:26.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:26.854+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.855755+0000) 2020-02-05T22:25:26.857 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:26.855+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.855909+0000) 2020-02-05T22:25:26.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:26.855+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.856374+0000) 2020-02-05T22:25:26.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:26.856+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.856505+0000) 2020-02-05T22:25:26.858 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:26.855+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.857009+0000) 2020-02-05T22:25:26.858 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:26.856+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.857134+0000) 2020-02-05T22:25:26.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:26.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:26.862491+0000) 2020-02-05T22:25:26.864 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:26.862+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:27.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:27.077+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:27.077850+0000) 2020-02-05T22:25:27.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:27.077+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:27.077979+0000) 2020-02-05T22:25:27.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:27.077+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:27.078158+0000) 2020-02-05T22:25:29.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:29.468+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:29.468633+0000) 2020-02-05T22:25:30.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:30.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:30.075476+0000) 2020-02-05T22:25:30.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:30.107+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:30.108134+0000) 2020-02-05T22:25:30.272 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:25:30.279 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:25:30.312 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:30.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:30.310787+0000) 2020-02-05T22:25:30.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:30.327+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:30.327416+0000) 2020-02-05T22:25:31.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:31.869+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.869150+0000) 2020-02-05T22:25:31.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:31.869+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.869546+0000) 2020-02-05T22:25:31.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:31.869+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.870073+0000) 2020-02-05T22:25:31.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:31.871+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.871159+0000) 2020-02-05T22:25:31.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:31.871+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.871515+0000) 2020-02-05T22:25:31.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:31.871+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.871829+0000) 2020-02-05T22:25:31.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:31.872+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.872385+0000) 2020-02-05T22:25:31.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:31.871+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.872073+0000) 2020-02-05T22:25:31.875 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:31.872+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.873028+0000) 2020-02-05T22:25:31.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:31.873+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.873330+0000) 2020-02-05T22:25:31.876 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:31.873+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.874092+0000) 2020-02-05T22:25:31.876 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:31.874+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.874791+0000) 2020-02-05T22:25:31.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:31.874+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.874956+0000) 2020-02-05T22:25:31.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:31.880+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:31.880436+0000) 2020-02-05T22:25:31.884 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:31.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:32.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:32.323+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:32.323537+0000) 2020-02-05T22:25:32.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:32.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:32.718230+0000) 2020-02-05T22:25:33.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:33.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:33.836464+0000) 2020-02-05T22:25:33.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:33.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:33.836487+0000) 2020-02-05T22:25:35.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:35.930+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:35.931346+0000) 2020-02-05T22:25:35.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:35.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:35.938950+0000) 2020-02-05T22:25:36.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:36.887+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.887523+0000) 2020-02-05T22:25:36.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:36.889+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.890828+0000) 2020-02-05T22:25:36.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:36.890+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.891876+0000) 2020-02-05T22:25:36.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:36.891+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.892544+0000) 2020-02-05T22:25:36.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:36.891+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.892733+0000) 2020-02-05T22:25:36.895 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:36.892+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.893233+0000) 2020-02-05T22:25:36.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:36.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.893527+0000) 2020-02-05T22:25:36.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:36.892+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.893844+0000) 2020-02-05T22:25:36.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:36.893+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.894217+0000) 2020-02-05T22:25:36.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:36.893+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.893919+0000) 2020-02-05T22:25:36.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:36.894+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.894440+0000) 2020-02-05T22:25:36.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:36.894+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.894716+0000) 2020-02-05T22:25:36.898 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:36.894+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.895407+0000) 2020-02-05T22:25:36.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:36.899+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:36.900916+0000) 2020-02-05T22:25:36.905 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:36.902+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114832 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:37.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:37.077+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:37.078121+0000) 2020-02-05T22:25:37.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:37.077+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:37.078197+0000) 2020-02-05T22:25:37.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:37.077+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:37.078361+0000) 2020-02-05T22:25:39.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:39.467+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:39.468834+0000) 2020-02-05T22:25:40.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:40.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:40.075720+0000) 2020-02-05T22:25:40.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:40.114+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:40.115907+0000) 2020-02-05T22:25:40.312 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:40.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:40.311002+0000) 2020-02-05T22:25:40.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:40.326+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:40.327670+0000) 2020-02-05T22:25:41.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:41.908+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.909285+0000) 2020-02-05T22:25:41.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:41.909+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.909807+0000) 2020-02-05T22:25:41.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:41.910+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.911104+0000) 2020-02-05T22:25:41.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:41.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.911183+0000) 2020-02-05T22:25:41.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:41.910+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.911313+0000) 2020-02-05T22:25:41.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:41.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.911721+0000) 2020-02-05T22:25:41.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:41.910+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.911976+0000) 2020-02-05T22:25:41.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:41.911+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.912346+0000) 2020-02-05T22:25:41.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:41.912+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.912842+0000) 2020-02-05T22:25:41.915 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:41.912+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.912913+0000) 2020-02-05T22:25:41.915 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:41.913+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.913694+0000) 2020-02-05T22:25:41.915 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:41.912+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.913664+0000) 2020-02-05T22:25:41.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:41.913+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.914507+0000) 2020-02-05T22:25:41.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:41.919+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:41.920069+0000) 2020-02-05T22:25:41.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:41.920+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114873 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:42.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:42.323+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:42.323765+0000) 2020-02-05T22:25:42.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:42.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:42.718441+0000) 2020-02-05T22:25:43.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:43.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:43.836764+0000) 2020-02-05T22:25:43.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:43.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:43.836762+0000) 2020-02-05T22:25:45.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:45.930+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:45.931645+0000) 2020-02-05T22:25:45.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:45.939+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:45.939187+0000) 2020-02-05T22:25:46.927 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:46.925+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.926531+0000) 2020-02-05T22:25:46.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:46.926+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.927450+0000) 2020-02-05T22:25:46.929 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:46.927+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.928174+0000) 2020-02-05T22:25:46.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:46.928+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.928542+0000) 2020-02-05T22:25:46.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:46.928+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.929119+0000) 2020-02-05T22:25:46.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:46.929+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.930285+0000) 2020-02-05T22:25:46.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:46.929+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.930667+0000) 2020-02-05T22:25:46.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:46.929+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.930941+0000) 2020-02-05T22:25:46.932 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:46.930+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.931021+0000) 2020-02-05T22:25:46.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:46.931+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.931611+0000) 2020-02-05T22:25:46.933 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:46.931+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.932191+0000) 2020-02-05T22:25:46.933 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:46.932+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.932651+0000) 2020-02-05T22:25:46.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:46.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.936651+0000) 2020-02-05T22:25:46.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:46.936+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:46.937717+0000) 2020-02-05T22:25:46.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:46.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:47.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:47.078+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:47.078362+0000) 2020-02-05T22:25:47.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:47.078+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:47.078409+0000) 2020-02-05T22:25:47.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:47.078+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:47.078622+0000) 2020-02-05T22:25:49.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:49.468+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:49.469064+0000) 2020-02-05T22:25:50.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:50.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:50.075998+0000) 2020-02-05T22:25:50.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:50.123+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:50.124531+0000) 2020-02-05T22:25:50.312 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:50.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:50.311241+0000) 2020-02-05T22:25:50.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:50.327+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:50.327896+0000) 2020-02-05T22:25:51.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:51.945+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.946224+0000) 2020-02-05T22:25:51.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:51.945+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.947013+0000) 2020-02-05T22:25:51.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:51.947+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.948971+0000) 2020-02-05T22:25:51.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:51.948+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.949253+0000) 2020-02-05T22:25:51.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:51.948+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.950337+0000) 2020-02-05T22:25:51.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:51.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.950569+0000) 2020-02-05T22:25:51.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:51.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.950775+0000) 2020-02-05T22:25:51.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:51.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.951270+0000) 2020-02-05T22:25:51.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:51.950+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.951527+0000) 2020-02-05T22:25:51.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:51.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.951899+0000) 2020-02-05T22:25:51.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:51.951+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.952647+0000) 2020-02-05T22:25:51.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:51.951+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.952794+0000) 2020-02-05T22:25:51.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:51.951+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.953038+0000) 2020-02-05T22:25:51.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:51.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:51.957298+0000) 2020-02-05T22:25:51.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:51.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114953 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:52.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:52.322+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:52.324008+0000) 2020-02-05T22:25:52.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:52.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:52.718669+0000) 2020-02-05T22:25:53.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:53.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:53.837053+0000) 2020-02-05T22:25:53.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:53.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:53.837053+0000) 2020-02-05T22:25:55.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:55.931+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:55.931875+0000) 2020-02-05T22:25:55.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:55.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:55.939482+0000) 2020-02-05T22:25:56.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:25:56.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.968186+0000) 2020-02-05T22:25:56.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:25:56.967+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.968908+0000) 2020-02-05T22:25:56.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:25:56.969+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.970192+0000) 2020-02-05T22:25:56.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:25:56.969+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.970783+0000) 2020-02-05T22:25:56.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:56.969+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.971017+0000) 2020-02-05T22:25:56.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:56.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.970700+0000) 2020-02-05T22:25:56.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:25:56.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.971412+0000) 2020-02-05T22:25:56.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:25:56.970+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.971983+0000) 2020-02-05T22:25:56.975 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:25:56.970+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.972167+0000) 2020-02-05T22:25:56.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:56.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.973568+0000) 2020-02-05T22:25:56.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:56.972+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.973639+0000) 2020-02-05T22:25:56.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:25:56.971+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.972320+0000) 2020-02-05T22:25:56.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:56.971+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.972771+0000) 2020-02-05T22:25:56.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:56.977+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:56.978977+0000) 2020-02-05T22:25:56.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:25:56.980+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 114993 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:25:57.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:25:57.077+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:57.078606+0000) 2020-02-05T22:25:57.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:25:57.077+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:57.078675+0000) 2020-02-05T22:25:57.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:25:57.077+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:57.078809+0000) 2020-02-05T22:25:59.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:25:59.468+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:25:59.469375+0000) 2020-02-05T22:26:00.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:00.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:00.076276+0000) 2020-02-05T22:26:00.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:00.133+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:00.134332+0000) 2020-02-05T22:26:00.312 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:00.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:00.311473+0000) 2020-02-05T22:26:00.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:00.327+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:00.328117+0000) 2020-02-05T22:26:00.427 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:26:00.433 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:26:01.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:01.987+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.988538+0000) 2020-02-05T22:26:01.991 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:01.988+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.990022+0000) 2020-02-05T22:26:01.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:01.988+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.990020+0000) 2020-02-05T22:26:01.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:01.989+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.990449+0000) 2020-02-05T22:26:01.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:01.988+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.990042+0000) 2020-02-05T22:26:01.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:01.989+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.991019+0000) 2020-02-05T22:26:01.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:01.990+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.991519+0000) 2020-02-05T22:26:01.993 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:01.990+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.991879+0000) 2020-02-05T22:26:01.993 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:01.990+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.991690+0000) 2020-02-05T22:26:01.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:01.991+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.992466+0000) 2020-02-05T22:26:01.994 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:01.991+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.992939+0000) 2020-02-05T22:26:01.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:01.992+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.993694+0000) 2020-02-05T22:26:01.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:01.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.998009+0000) 2020-02-05T22:26:01.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:01.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:01.998288+0000) 2020-02-05T22:26:02.002 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:01.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115035 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:02.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:02.322+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:02.324247+0000) 2020-02-05T22:26:02.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:02.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:02.718870+0000) 2020-02-05T22:26:03.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:03.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:03.837373+0000) 2020-02-05T22:26:03.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:03.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:03.837373+0000) 2020-02-05T22:26:05.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:05.931+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:05.932157+0000) 2020-02-05T22:26:05.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:05.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:05.939706+0000) 2020-02-05T22:26:07.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:07.005+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.006896+0000) 2020-02-05T22:26:07.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:07.006+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.007596+0000) 2020-02-05T22:26:07.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:07.006+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.008022+0000) 2020-02-05T22:26:07.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:07.007+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.008416+0000) 2020-02-05T22:26:07.010 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:07.007+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.008842+0000) 2020-02-05T22:26:07.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:07.007+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.009311+0000) 2020-02-05T22:26:07.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:07.008+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.009696+0000) 2020-02-05T22:26:07.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:07.009+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.010465+0000) 2020-02-05T22:26:07.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:07.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.011588+0000) 2020-02-05T22:26:07.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:07.009+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.010848+0000) 2020-02-05T22:26:07.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:07.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.012285+0000) 2020-02-05T22:26:07.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:07.011+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.012526+0000) 2020-02-05T22:26:07.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:07.012+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.013909+0000) 2020-02-05T22:26:07.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:07.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.017890+0000) 2020-02-05T22:26:07.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:07.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:07.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:07.077+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.078869+0000) 2020-02-05T22:26:07.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:07.077+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.078887+0000) 2020-02-05T22:26:07.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:07.077+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:07.079030+0000) 2020-02-05T22:26:09.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:09.469+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:09.469633+0000) 2020-02-05T22:26:10.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:10.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:10.076494+0000) 2020-02-05T22:26:10.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:10.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:10.142489+0000) 2020-02-05T22:26:10.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:10.310+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:10.311715+0000) 2020-02-05T22:26:10.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:10.327+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:10.328341+0000) 2020-02-05T22:26:12.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:12.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.027565+0000) 2020-02-05T22:26:12.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:12.027+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.028995+0000) 2020-02-05T22:26:12.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:12.028+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.029396+0000) 2020-02-05T22:26:12.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:12.028+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.029579+0000) 2020-02-05T22:26:12.031 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:12.028+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.029836+0000) 2020-02-05T22:26:12.032 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:12.028+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.030308+0000) 2020-02-05T22:26:12.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:12.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.030746+0000) 2020-02-05T22:26:12.032 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:12.029+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.031402+0000) 2020-02-05T22:26:12.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:12.030+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.032112+0000) 2020-02-05T22:26:12.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:12.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.032705+0000) 2020-02-05T22:26:12.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:12.031+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.032827+0000) 2020-02-05T22:26:12.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:12.031+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.033214+0000) 2020-02-05T22:26:12.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:12.031+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.033253+0000) 2020-02-05T22:26:12.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:12.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.036727+0000) 2020-02-05T22:26:12.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:12.039+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:12.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:12.323+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.324496+0000) 2020-02-05T22:26:12.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:12.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:12.719114+0000) 2020-02-05T22:26:13.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:13.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:13.837639+0000) 2020-02-05T22:26:13.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:13.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:13.837638+0000) 2020-02-05T22:26:15.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:15.932+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:15.932390+0000) 2020-02-05T22:26:15.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:15.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:15.939941+0000) 2020-02-05T22:26:17.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:17.046+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.047343+0000) 2020-02-05T22:26:17.049 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:17.046+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.047746+0000) 2020-02-05T22:26:17.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:17.046+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.048315+0000) 2020-02-05T22:26:17.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:17.047+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.048701+0000) 2020-02-05T22:26:17.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:17.047+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.049111+0000) 2020-02-05T22:26:17.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:17.048+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.049442+0000) 2020-02-05T22:26:17.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:17.049+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.050710+0000) 2020-02-05T22:26:17.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:17.050+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.051204+0000) 2020-02-05T22:26:17.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:17.049+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.051097+0000) 2020-02-05T22:26:17.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:17.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.051629+0000) 2020-02-05T22:26:17.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:17.051+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.052227+0000) 2020-02-05T22:26:17.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:17.051+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.052992+0000) 2020-02-05T22:26:17.054 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:17.051+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.052910+0000) 2020-02-05T22:26:17.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:17.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.060383+0000) 2020-02-05T22:26:17.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:17.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:17.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:17.078+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.079104+0000) 2020-02-05T22:26:17.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:17.078+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.079100+0000) 2020-02-05T22:26:17.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:17.078+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:17.079258+0000) 2020-02-05T22:26:19.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:19.469+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:19.469871+0000) 2020-02-05T22:26:20.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:20.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:20.076730+0000) 2020-02-05T22:26:20.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:20.150+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:20.150384+0000) 2020-02-05T22:26:20.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:20.311+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:20.312006+0000) 2020-02-05T22:26:20.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:20.328+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:20.328573+0000) 2020-02-05T22:26:22.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:22.066+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.067306+0000) 2020-02-05T22:26:22.069 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:22.066+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.067867+0000) 2020-02-05T22:26:22.069 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:22.067+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.068302+0000) 2020-02-05T22:26:22.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:22.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.068730+0000) 2020-02-05T22:26:22.070 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:22.068+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.069385+0000) 2020-02-05T22:26:22.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:22.068+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.069618+0000) 2020-02-05T22:26:22.071 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:22.068+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.070021+0000) 2020-02-05T22:26:22.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:22.068+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.070319+0000) 2020-02-05T22:26:22.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:22.069+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.071067+0000) 2020-02-05T22:26:22.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:22.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.072867+0000) 2020-02-05T22:26:22.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:22.071+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.073042+0000) 2020-02-05T22:26:22.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:22.071+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.073288+0000) 2020-02-05T22:26:22.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:22.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.077127+0000) 2020-02-05T22:26:22.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:22.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.077382+0000) 2020-02-05T22:26:22.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:22.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:22.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:22.324+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.324724+0000) 2020-02-05T22:26:22.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:22.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:22.719336+0000) 2020-02-05T22:26:23.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:23.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:23.837923+0000) 2020-02-05T22:26:23.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:23.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:23.837923+0000) 2020-02-05T22:26:25.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:25.932+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:25.932659+0000) 2020-02-05T22:26:25.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:25.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:25.940222+0000) 2020-02-05T22:26:27.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:27.078+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.079336+0000) 2020-02-05T22:26:27.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:27.078+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.079356+0000) 2020-02-05T22:26:27.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:27.078+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.079540+0000) 2020-02-05T22:26:27.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:27.085+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.085714+0000) 2020-02-05T22:26:27.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:27.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.086415+0000) 2020-02-05T22:26:27.088 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:27.086+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.086799+0000) 2020-02-05T22:26:27.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:27.085+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.087310+0000) 2020-02-05T22:26:27.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:27.086+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.088248+0000) 2020-02-05T22:26:27.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:27.088+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.088763+0000) 2020-02-05T22:26:27.092 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:27.088+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.090409+0000) 2020-02-05T22:26:27.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:27.089+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.090559+0000) 2020-02-05T22:26:27.093 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:27.089+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.089446+0000) 2020-02-05T22:26:27.093 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:27.089+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.089519+0000) 2020-02-05T22:26:27.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:27.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.090070+0000) 2020-02-05T22:26:27.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:27.089+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.091282+0000) 2020-02-05T22:26:27.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:27.090+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.091943+0000) 2020-02-05T22:26:27.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:27.095+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:27.095921+0000) 2020-02-05T22:26:27.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:27.099+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:29.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:29.470+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:29.470128+0000) 2020-02-05T22:26:30.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:30.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:30.076943+0000) 2020-02-05T22:26:30.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:30.158+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:30.158484+0000) 2020-02-05T22:26:30.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:30.311+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:30.312260+0000) 2020-02-05T22:26:30.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:30.328+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:30.328823+0000) 2020-02-05T22:26:30.574 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:26:30.581 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:26:32.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:32.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.106755+0000) 2020-02-05T22:26:32.109 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:32.105+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.107155+0000) 2020-02-05T22:26:32.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:32.107+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.108569+0000) 2020-02-05T22:26:32.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:32.107+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.108728+0000) 2020-02-05T22:26:32.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:32.107+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.108908+0000) 2020-02-05T22:26:32.111 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:32.108+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.109586+0000) 2020-02-05T22:26:32.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:32.107+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.109301+0000) 2020-02-05T22:26:32.111 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:32.109+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.110172+0000) 2020-02-05T22:26:32.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:32.109+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.110904+0000) 2020-02-05T22:26:32.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:32.109+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.110985+0000) 2020-02-05T22:26:32.112 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:32.110+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.111209+0000) 2020-02-05T22:26:32.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:32.110+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.111782+0000) 2020-02-05T22:26:32.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:32.115+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.116892+0000) 2020-02-05T22:26:32.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:32.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.117236+0000) 2020-02-05T22:26:32.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:32.119+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:32.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:32.323+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.324957+0000) 2020-02-05T22:26:32.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:32.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:32.719563+0000) 2020-02-05T22:26:33.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:33.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:33.838195+0000) 2020-02-05T22:26:33.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:33.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:33.838194+0000) 2020-02-05T22:26:35.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:35.931+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:35.932857+0000) 2020-02-05T22:26:35.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:35.939+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:35.940481+0000) 2020-02-05T22:26:37.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:37.079+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.079608+0000) 2020-02-05T22:26:37.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:37.079+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.079674+0000) 2020-02-05T22:26:37.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:37.079+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.079794+0000) 2020-02-05T22:26:37.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:37.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.126291+0000) 2020-02-05T22:26:37.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:37.125+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.126901+0000) 2020-02-05T22:26:37.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:37.126+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.127749+0000) 2020-02-05T22:26:37.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:37.126+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.128063+0000) 2020-02-05T22:26:37.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:37.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.129230+0000) 2020-02-05T22:26:37.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:37.128+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.129566+0000) 2020-02-05T22:26:37.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:37.129+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.130512+0000) 2020-02-05T22:26:37.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:37.128+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.130275+0000) 2020-02-05T22:26:37.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:37.129+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.130511+0000) 2020-02-05T22:26:37.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:37.127+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.129203+0000) 2020-02-05T22:26:37.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:37.129+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.131011+0000) 2020-02-05T22:26:37.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:37.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.131210+0000) 2020-02-05T22:26:37.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:37.129+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.131183+0000) 2020-02-05T22:26:37.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:37.133+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:37.134544+0000) 2020-02-05T22:26:37.141 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:37.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115317 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:39.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:39.469+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:39.470363+0000) 2020-02-05T22:26:40.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:40.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:40.077183+0000) 2020-02-05T22:26:40.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:40.165+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:40.166490+0000) 2020-02-05T22:26:40.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:40.312+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:40.312515+0000) 2020-02-05T22:26:40.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:40.328+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:40.329052+0000) 2020-02-05T22:26:42.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:42.146+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.146714+0000) 2020-02-05T22:26:42.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:42.146+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.147353+0000) 2020-02-05T22:26:42.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:42.147+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.147865+0000) 2020-02-05T22:26:42.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:42.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.148338+0000) 2020-02-05T22:26:42.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:42.147+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.148700+0000) 2020-02-05T22:26:42.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:42.148+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.149241+0000) 2020-02-05T22:26:42.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:42.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.149709+0000) 2020-02-05T22:26:42.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:42.149+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.150183+0000) 2020-02-05T22:26:42.153 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:42.151+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.151482+0000) 2020-02-05T22:26:42.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:42.149+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.150841+0000) 2020-02-05T22:26:42.154 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:42.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.152621+0000) 2020-02-05T22:26:42.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:42.151+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.152467+0000) 2020-02-05T22:26:42.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:42.151+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.152064+0000) 2020-02-05T22:26:42.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:42.156+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.157445+0000) 2020-02-05T22:26:42.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:42.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:42.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:42.324+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.325197+0000) 2020-02-05T22:26:42.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:42.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:42.719763+0000) 2020-02-05T22:26:43.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:43.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:43.838450+0000) 2020-02-05T22:26:43.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:43.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:43.838451+0000) 2020-02-05T22:26:45.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:45.932+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:45.933132+0000) 2020-02-05T22:26:45.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:45.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:45.940734+0000) 2020-02-05T22:26:47.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:47.079+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.079850+0000) 2020-02-05T22:26:47.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:47.079+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.079924+0000) 2020-02-05T22:26:47.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:47.079+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.080048+0000) 2020-02-05T22:26:47.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:47.167+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.168548+0000) 2020-02-05T22:26:47.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:47.168+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.168595+0000) 2020-02-05T22:26:47.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:47.168+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.168916+0000) 2020-02-05T22:26:47.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:47.169+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.169888+0000) 2020-02-05T22:26:47.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:47.169+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.170575+0000) 2020-02-05T22:26:47.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:47.169+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.170932+0000) 2020-02-05T22:26:47.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:47.170+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.171186+0000) 2020-02-05T22:26:47.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:47.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.171899+0000) 2020-02-05T22:26:47.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:47.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.172513+0000) 2020-02-05T22:26:47.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:47.172+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.172722+0000) 2020-02-05T22:26:47.174 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:47.172+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.172961+0000) 2020-02-05T22:26:47.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:47.172+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.173552+0000) 2020-02-05T22:26:47.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:47.172+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.173794+0000) 2020-02-05T22:26:47.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:47.178+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:47.179451+0000) 2020-02-05T22:26:47.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:47.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:49.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:49.469+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:49.470603+0000) 2020-02-05T22:26:50.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:50.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:50.077381+0000) 2020-02-05T22:26:50.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:50.174+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:50.174798+0000) 2020-02-05T22:26:50.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:50.312+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:50.312767+0000) 2020-02-05T22:26:50.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:50.328+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:50.329317+0000) 2020-02-05T22:26:52.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:52.186+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.187887+0000) 2020-02-05T22:26:52.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:52.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.188576+0000) 2020-02-05T22:26:52.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:52.188+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.189075+0000) 2020-02-05T22:26:52.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:52.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.189722+0000) 2020-02-05T22:26:52.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:52.189+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.189894+0000) 2020-02-05T22:26:52.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:52.189+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.190626+0000) 2020-02-05T22:26:52.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:52.190+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.191207+0000) 2020-02-05T22:26:52.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:52.190+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.191601+0000) 2020-02-05T22:26:52.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:52.191+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.192397+0000) 2020-02-05T22:26:52.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:52.192+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.192994+0000) 2020-02-05T22:26:52.194 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:52.192+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.193233+0000) 2020-02-05T22:26:52.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:52.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.193951+0000) 2020-02-05T22:26:52.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:52.193+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.194053+0000) 2020-02-05T22:26:52.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:52.200+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.201143+0000) 2020-02-05T22:26:52.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:52.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:52.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:52.325+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.325469+0000) 2020-02-05T22:26:52.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:52.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:52.719972+0000) 2020-02-05T22:26:53.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:53.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:53.838712+0000) 2020-02-05T22:26:53.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:53.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:53.838768+0000) 2020-02-05T22:26:55.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:55.932+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:55.933359+0000) 2020-02-05T22:26:55.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:55.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:55.940964+0000) 2020-02-05T22:26:57.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:57.079+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.080106+0000) 2020-02-05T22:26:57.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:57.079+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.080210+0000) 2020-02-05T22:26:57.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:57.079+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.080308+0000) 2020-02-05T22:26:57.209 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:26:57.207+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.208582+0000) 2020-02-05T22:26:57.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:26:57.208+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.209280+0000) 2020-02-05T22:26:57.211 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:26:57.209+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.210292+0000) 2020-02-05T22:26:57.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:26:57.209+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.210094+0000) 2020-02-05T22:26:57.212 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:26:57.210+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.211183+0000) 2020-02-05T22:26:57.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:26:57.211+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.212081+0000) 2020-02-05T22:26:57.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:26:57.211+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.212977+0000) 2020-02-05T22:26:57.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:57.212+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.213849+0000) 2020-02-05T22:26:57.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:26:57.213+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.214340+0000) 2020-02-05T22:26:57.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:26:57.213+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.214570+0000) 2020-02-05T22:26:57.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:57.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.214481+0000) 2020-02-05T22:26:57.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:26:57.214+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.214462+0000) 2020-02-05T22:26:57.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:26:57.214+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.214612+0000) 2020-02-05T22:26:57.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:57.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:57.219293+0000) 2020-02-05T22:26:57.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:26:57.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:26:59.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:26:59.470+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:26:59.470857+0000) 2020-02-05T22:27:00.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:00.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:00.077603+0000) 2020-02-05T22:27:00.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:00.182+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:00.182747+0000) 2020-02-05T22:27:00.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:00.312+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:00.313027+0000) 2020-02-05T22:27:00.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:00.328+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:00.329512+0000) 2020-02-05T22:27:00.771 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:27:00.778 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:27:02.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:02.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.227628+0000) 2020-02-05T22:27:02.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:02.229+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.229640+0000) 2020-02-05T22:27:02.231 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:02.228+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.229849+0000) 2020-02-05T22:27:02.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:02.229+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.230301+0000) 2020-02-05T22:27:02.232 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:02.230+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.230814+0000) 2020-02-05T22:27:02.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:02.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.231865+0000) 2020-02-05T22:27:02.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:02.231+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.231886+0000) 2020-02-05T22:27:02.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:02.231+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.232020+0000) 2020-02-05T22:27:02.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:02.231+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.232453+0000) 2020-02-05T22:27:02.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:02.232+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.233263+0000) 2020-02-05T22:27:02.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:02.232+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.232676+0000) 2020-02-05T22:27:02.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:02.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.234590+0000) 2020-02-05T22:27:02.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:02.234+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.235130+0000) 2020-02-05T22:27:02.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:02.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.240466+0000) 2020-02-05T22:27:02.245 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:02.243+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:02.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:02.325+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.325692+0000) 2020-02-05T22:27:02.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:02.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:02.720208+0000) 2020-02-05T22:27:03.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:03.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:03.838977+0000) 2020-02-05T22:27:03.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:03.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:03.838981+0000) 2020-02-05T22:27:05.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:05.933+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:05.933588+0000) 2020-02-05T22:27:05.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:05.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:05.941267+0000) 2020-02-05T22:27:07.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:07.079+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.080299+0000) 2020-02-05T22:27:07.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:07.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.080599+0000) 2020-02-05T22:27:07.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:07.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.080465+0000) 2020-02-05T22:27:07.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:07.248+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.249496+0000) 2020-02-05T22:27:07.252 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:07.250+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.251471+0000) 2020-02-05T22:27:07.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:07.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.251513+0000) 2020-02-05T22:27:07.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:07.251+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.252134+0000) 2020-02-05T22:27:07.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:07.251+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.251823+0000) 2020-02-05T22:27:07.254 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:07.251+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.252350+0000) 2020-02-05T22:27:07.254 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:07.251+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.252787+0000) 2020-02-05T22:27:07.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:07.252+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.253294+0000) 2020-02-05T22:27:07.255 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:07.252+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.253448+0000) 2020-02-05T22:27:07.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:07.253+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.254004+0000) 2020-02-05T22:27:07.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:07.253+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.254013+0000) 2020-02-05T22:27:07.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:07.254+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.254733+0000) 2020-02-05T22:27:07.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:07.254+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.254784+0000) 2020-02-05T22:27:07.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:07.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:07.260620+0000) 2020-02-05T22:27:07.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:07.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:09.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:09.470+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:09.471149+0000) 2020-02-05T22:27:10.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:10.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:10.077827+0000) 2020-02-05T22:27:10.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:10.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:10.190584+0000) 2020-02-05T22:27:10.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:10.313+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:10.313301+0000) 2020-02-05T22:27:10.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:10.329+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:10.329746+0000) 2020-02-05T22:27:12.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:12.268+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.269285+0000) 2020-02-05T22:27:12.271 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:12.269+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.270342+0000) 2020-02-05T22:27:12.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:12.269+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.270623+0000) 2020-02-05T22:27:12.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:12.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.270074+0000) 2020-02-05T22:27:12.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:12.270+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.271240+0000) 2020-02-05T22:27:12.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:12.270+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.270862+0000) 2020-02-05T22:27:12.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:12.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.271222+0000) 2020-02-05T22:27:12.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:12.270+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.270929+0000) 2020-02-05T22:27:12.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:12.271+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.271889+0000) 2020-02-05T22:27:12.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:12.271+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.272028+0000) 2020-02-05T22:27:12.274 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:12.271+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.272504+0000) 2020-02-05T22:27:12.275 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:12.272+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.273430+0000) 2020-02-05T22:27:12.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:12.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.278135+0000) 2020-02-05T22:27:12.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:12.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.278347+0000) 2020-02-05T22:27:12.282 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:12.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:12.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:12.325+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.325918+0000) 2020-02-05T22:27:12.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:12.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:12.720410+0000) 2020-02-05T22:27:13.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:13.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:13.839277+0000) 2020-02-05T22:27:13.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:13.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:13.839277+0000) 2020-02-05T22:27:15.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:15.933+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:15.933821+0000) 2020-02-05T22:27:15.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:15.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:15.941518+0000) 2020-02-05T22:27:17.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:17.080+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.080546+0000) 2020-02-05T22:27:17.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:17.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.080732+0000) 2020-02-05T22:27:17.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:17.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.080846+0000) 2020-02-05T22:27:17.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:17.284+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.285745+0000) 2020-02-05T22:27:17.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:17.286+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.286979+0000) 2020-02-05T22:27:17.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:17.287+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.288568+0000) 2020-02-05T22:27:17.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:17.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.289304+0000) 2020-02-05T22:27:17.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:17.289+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.290116+0000) 2020-02-05T22:27:17.291 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:17.289+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.290674+0000) 2020-02-05T22:27:17.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:17.289+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.290971+0000) 2020-02-05T22:27:17.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:17.291+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.291585+0000) 2020-02-05T22:27:17.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:17.291+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.292234+0000) 2020-02-05T22:27:17.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:17.291+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.291795+0000) 2020-02-05T22:27:17.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:17.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.293750+0000) 2020-02-05T22:27:17.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:17.292+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.292850+0000) 2020-02-05T22:27:17.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:17.292+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.293289+0000) 2020-02-05T22:27:17.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:17.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:17.299320+0000) 2020-02-05T22:27:17.301 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:17.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:19.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:19.471+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:19.471384+0000) 2020-02-05T22:27:20.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:20.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:20.078063+0000) 2020-02-05T22:27:20.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:20.198+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:20.198541+0000) 2020-02-05T22:27:20.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:20.312+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:20.313546+0000) 2020-02-05T22:27:20.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:20.329+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:20.329970+0000) 2020-02-05T22:27:22.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:22.307+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.308896+0000) 2020-02-05T22:27:22.319 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:22.308+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.309681+0000) 2020-02-05T22:27:22.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:22.309+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.310724+0000) 2020-02-05T22:27:22.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:22.310+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.311869+0000) 2020-02-05T22:27:22.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:22.310+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.311715+0000) 2020-02-05T22:27:22.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:22.310+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.312063+0000) 2020-02-05T22:27:22.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:22.307+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.307508+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:22.308+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.308188+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:22.309+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.309207+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:22.309+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.310064+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:22.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.316369+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:22.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.316651+0000) 2020-02-05T22:27:22.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:22.310+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.310415+0000) 2020-02-05T22:27:22.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:22.311+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.311344+0000) 2020-02-05T22:27:22.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:22.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:22.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:22.324+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.326069+0000) 2020-02-05T22:27:22.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:22.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:22.720618+0000) 2020-02-05T22:27:23.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:23.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:23.839542+0000) 2020-02-05T22:27:23.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:23.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:23.839542+0000) 2020-02-05T22:27:25.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:25.933+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:25.934110+0000) 2020-02-05T22:27:25.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:25.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:25.941811+0000) 2020-02-05T22:27:27.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:27.079+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.080777+0000) 2020-02-05T22:27:27.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:27.079+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.080945+0000) 2020-02-05T22:27:27.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:27.079+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.081103+0000) 2020-02-05T22:27:27.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:27.327+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.328582+0000) 2020-02-05T22:27:27.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:27.328+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.329298+0000) 2020-02-05T22:27:27.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:27.328+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.329663+0000) 2020-02-05T22:27:27.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:27.329+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.330235+0000) 2020-02-05T22:27:27.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:27.329+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.330519+0000) 2020-02-05T22:27:27.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:27.329+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.331073+0000) 2020-02-05T22:27:27.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:27.330+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.331601+0000) 2020-02-05T22:27:27.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:27.330+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.332196+0000) 2020-02-05T22:27:27.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:27.331+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.332530+0000) 2020-02-05T22:27:27.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:27.331+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.332814+0000) 2020-02-05T22:27:27.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:27.331+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.333090+0000) 2020-02-05T22:27:27.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:27.332+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.333729+0000) 2020-02-05T22:27:27.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:27.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.335589+0000) 2020-02-05T22:27:27.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:27.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:27.335810+0000) 2020-02-05T22:27:27.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:27.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:29.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:29.470+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:29.471618+0000) 2020-02-05T22:27:30.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:30.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:30.078336+0000) 2020-02-05T22:27:30.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:30.205+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:30.206800+0000) 2020-02-05T22:27:30.315 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:30.312+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:30.313811+0000) 2020-02-05T22:27:30.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:30.329+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:30.330205+0000) 2020-02-05T22:27:30.976 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:27:30.982 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:27:32.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:32.325+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.326341+0000) 2020-02-05T22:27:32.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:32.344+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.345360+0000) 2020-02-05T22:27:32.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:32.344+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.346043+0000) 2020-02-05T22:27:32.348 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:32.345+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.346513+0000) 2020-02-05T22:27:32.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:32.345+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.346901+0000) 2020-02-05T22:27:32.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:32.346+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.347558+0000) 2020-02-05T22:27:32.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:32.347+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.348948+0000) 2020-02-05T22:27:32.350 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:32.348+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.349270+0000) 2020-02-05T22:27:32.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:32.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.349520+0000) 2020-02-05T22:27:32.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:32.348+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.349515+0000) 2020-02-05T22:27:32.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:32.349+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.350218+0000) 2020-02-05T22:27:32.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:32.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.350897+0000) 2020-02-05T22:27:32.352 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:32.350+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.351497+0000) 2020-02-05T22:27:32.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:32.350+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.352065+0000) 2020-02-05T22:27:32.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:32.354+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.355221+0000) 2020-02-05T22:27:32.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:32.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:32.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:32.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:32.720854+0000) 2020-02-05T22:27:33.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:33.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:33.839840+0000) 2020-02-05T22:27:33.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:33.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:33.839842+0000) 2020-02-05T22:27:35.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:35.933+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:35.934298+0000) 2020-02-05T22:27:35.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:35.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:35.942077+0000) 2020-02-05T22:27:37.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:37.080+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.081012+0000) 2020-02-05T22:27:37.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:37.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.081217+0000) 2020-02-05T22:27:37.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:37.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.081400+0000) 2020-02-05T22:27:37.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:37.364+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.365989+0000) 2020-02-05T22:27:37.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:37.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.366451+0000) 2020-02-05T22:27:37.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:37.365+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.367022+0000) 2020-02-05T22:27:37.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:37.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.367575+0000) 2020-02-05T22:27:37.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:37.366+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.367761+0000) 2020-02-05T22:27:37.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:37.367+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.368523+0000) 2020-02-05T22:27:37.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:37.367+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.368859+0000) 2020-02-05T22:27:37.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:37.368+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.369964+0000) 2020-02-05T22:27:37.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:37.369+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.370329+0000) 2020-02-05T22:27:37.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:37.369+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.370892+0000) 2020-02-05T22:27:37.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:37.369+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.371068+0000) 2020-02-05T22:27:37.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:37.370+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.371463+0000) 2020-02-05T22:27:37.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:37.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.371949+0000) 2020-02-05T22:27:37.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:37.374+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:37.375689+0000) 2020-02-05T22:27:37.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:37.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:39.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:39.471+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:39.471869+0000) 2020-02-05T22:27:40.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:40.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:40.078604+0000) 2020-02-05T22:27:40.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:40.214+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:40.215431+0000) 2020-02-05T22:27:40.315 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:40.313+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:40.314042+0000) 2020-02-05T22:27:40.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:40.329+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:40.330430+0000) 2020-02-05T22:27:42.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:42.325+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.326662+0000) 2020-02-05T22:27:42.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:42.385+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.386939+0000) 2020-02-05T22:27:42.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:42.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.387630+0000) 2020-02-05T22:27:42.389 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:42.386+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.387880+0000) 2020-02-05T22:27:42.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:42.387+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.388842+0000) 2020-02-05T22:27:42.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:42.387+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.388747+0000) 2020-02-05T22:27:42.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:42.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.389636+0000) 2020-02-05T22:27:42.391 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:42.388+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.389731+0000) 2020-02-05T22:27:42.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:42.389+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.390501+0000) 2020-02-05T22:27:42.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:42.389+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.390946+0000) 2020-02-05T22:27:42.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:42.389+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.391394+0000) 2020-02-05T22:27:42.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:42.390+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.391290+0000) 2020-02-05T22:27:42.394 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:42.390+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.391999+0000) 2020-02-05T22:27:42.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:42.390+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.392378+0000) 2020-02-05T22:27:42.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:42.391+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.392769+0000) 2020-02-05T22:27:42.401 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:42.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:42.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:42.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:42.721062+0000) 2020-02-05T22:27:43.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:43.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:43.840139+0000) 2020-02-05T22:27:43.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:43.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:43.840139+0000) 2020-02-05T22:27:45.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:45.933+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:45.934534+0000) 2020-02-05T22:27:45.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:45.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:45.942374+0000) 2020-02-05T22:27:47.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:47.080+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.081280+0000) 2020-02-05T22:27:47.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:47.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.081509+0000) 2020-02-05T22:27:47.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:47.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.081691+0000) 2020-02-05T22:27:47.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:47.403+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.404974+0000) 2020-02-05T22:27:47.407 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:47.404+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.406018+0000) 2020-02-05T22:27:47.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:47.405+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.407126+0000) 2020-02-05T22:27:47.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:47.405+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.407254+0000) 2020-02-05T22:27:47.409 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:47.406+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.407457+0000) 2020-02-05T22:27:47.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:47.406+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.407934+0000) 2020-02-05T22:27:47.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:47.407+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.408539+0000) 2020-02-05T22:27:47.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:47.407+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.409050+0000) 2020-02-05T22:27:47.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:47.407+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.409230+0000) 2020-02-05T22:27:47.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:47.409+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.410374+0000) 2020-02-05T22:27:47.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:47.409+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.410863+0000) 2020-02-05T22:27:47.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:47.410+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.411569+0000) 2020-02-05T22:27:47.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:47.410+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.411482+0000) 2020-02-05T22:27:47.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:47.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:47.416004+0000) 2020-02-05T22:27:47.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:47.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:49.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:49.471+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:49.472127+0000) 2020-02-05T22:27:50.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:50.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:50.078837+0000) 2020-02-05T22:27:50.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:50.222+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:50.223610+0000) 2020-02-05T22:27:50.315 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:50.313+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:50.314286+0000) 2020-02-05T22:27:50.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:50.330+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:50.330662+0000) 2020-02-05T22:27:52.328 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:52.326+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.326917+0000) 2020-02-05T22:27:52.426 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:52.423+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.425052+0000) 2020-02-05T22:27:52.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:52.425+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.426381+0000) 2020-02-05T22:27:52.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:52.424+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.426068+0000) 2020-02-05T22:27:52.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:52.426+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.427391+0000) 2020-02-05T22:27:52.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:52.426+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.427706+0000) 2020-02-05T22:27:52.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:52.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.427770+0000) 2020-02-05T22:27:52.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:52.427+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.428169+0000) 2020-02-05T22:27:52.429 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:52.427+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.428655+0000) 2020-02-05T22:27:52.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:52.427+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.428991+0000) 2020-02-05T22:27:52.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:52.427+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.429141+0000) 2020-02-05T22:27:52.431 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:52.428+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.429969+0000) 2020-02-05T22:27:52.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:52.429+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.430465+0000) 2020-02-05T22:27:52.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:52.428+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.430211+0000) 2020-02-05T22:27:52.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:52.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.436567+0000) 2020-02-05T22:27:52.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:52.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:52.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:52.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:52.721337+0000) 2020-02-05T22:27:53.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:53.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:53.840416+0000) 2020-02-05T22:27:53.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:53.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:53.840416+0000) 2020-02-05T22:27:55.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:55.934+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:55.934761+0000) 2020-02-05T22:27:55.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:55.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:55.942653+0000) 2020-02-05T22:27:57.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:57.080+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.081556+0000) 2020-02-05T22:27:57.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:57.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.081754+0000) 2020-02-05T22:27:57.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:57.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.082118+0000) 2020-02-05T22:27:57.444 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:27:57.442+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.443390+0000) 2020-02-05T22:27:57.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:27:57.443+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.444595+0000) 2020-02-05T22:27:57.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:57.444+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.445534+0000) 2020-02-05T22:27:57.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:27:57.444+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.445559+0000) 2020-02-05T22:27:57.447 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:27:57.444+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.446047+0000) 2020-02-05T22:27:57.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:57.445+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.446352+0000) 2020-02-05T22:27:57.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:27:57.445+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.446611+0000) 2020-02-05T22:27:57.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:27:57.445+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.446884+0000) 2020-02-05T22:27:57.448 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:27:57.445+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.447222+0000) 2020-02-05T22:27:57.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:27:57.447+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.448540+0000) 2020-02-05T22:27:57.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:27:57.446+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.447752+0000) 2020-02-05T22:27:57.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:27:57.446+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.447697+0000) 2020-02-05T22:27:57.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:27:57.447+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.448824+0000) 2020-02-05T22:27:57.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:57.453+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:57.454249+0000) 2020-02-05T22:27:57.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:27:57.456+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 115958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:27:59.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:27:59.471+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:27:59.472354+0000) 2020-02-05T22:28:00.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:00.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:00.079138+0000) 2020-02-05T22:28:00.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:00.231+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:00.231920+0000) 2020-02-05T22:28:00.315 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:00.313+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:00.314542+0000) 2020-02-05T22:28:00.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:00.329+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:00.330897+0000) 2020-02-05T22:28:01.131 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:28:01.137 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:28:02.328 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:02.326+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.327180+0000) 2020-02-05T22:28:02.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:02.462+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.463497+0000) 2020-02-05T22:28:02.465 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:02.463+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.463995+0000) 2020-02-05T22:28:02.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:02.463+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.464516+0000) 2020-02-05T22:28:02.466 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:02.463+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.465161+0000) 2020-02-05T22:28:02.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:02.464+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.465849+0000) 2020-02-05T22:28:02.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:02.464+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.466166+0000) 2020-02-05T22:28:02.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:02.465+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.466501+0000) 2020-02-05T22:28:02.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:02.465+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.466857+0000) 2020-02-05T22:28:02.468 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:02.466+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.467230+0000) 2020-02-05T22:28:02.469 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:02.466+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.467805+0000) 2020-02-05T22:28:02.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:02.466+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.468066+0000) 2020-02-05T22:28:02.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:02.467+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.468981+0000) 2020-02-05T22:28:02.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:02.467+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.469185+0000) 2020-02-05T22:28:02.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:02.473+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.474189+0000) 2020-02-05T22:28:02.479 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:02.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:02.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:02.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:02.721540+0000) 2020-02-05T22:28:03.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:03.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:03.840683+0000) 2020-02-05T22:28:03.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:03.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:03.840683+0000) 2020-02-05T22:28:05.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:05.934+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:05.935047+0000) 2020-02-05T22:28:05.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:05.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:05.942933+0000) 2020-02-05T22:28:07.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:07.080+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.081780+0000) 2020-02-05T22:28:07.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:07.080+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.082017+0000) 2020-02-05T22:28:07.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:07.080+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.082370+0000) 2020-02-05T22:28:07.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:07.481+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.482666+0000) 2020-02-05T22:28:07.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:07.482+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.483484+0000) 2020-02-05T22:28:07.485 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:07.482+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.484046+0000) 2020-02-05T22:28:07.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:07.483+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.485134+0000) 2020-02-05T22:28:07.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:07.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.485434+0000) 2020-02-05T22:28:07.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:07.485+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.486170+0000) 2020-02-05T22:28:07.487 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:07.485+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.486550+0000) 2020-02-05T22:28:07.488 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:07.485+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.487140+0000) 2020-02-05T22:28:07.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:07.485+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.487317+0000) 2020-02-05T22:28:07.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:07.486+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.487784+0000) 2020-02-05T22:28:07.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:07.488+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.490330+0000) 2020-02-05T22:28:07.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:07.488+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.489164+0000) 2020-02-05T22:28:07.495 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:07.488+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.489460+0000) 2020-02-05T22:28:07.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:07.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:07.494438+0000) 2020-02-05T22:28:07.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:07.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:09.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:09.472+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:09.472590+0000) 2020-02-05T22:28:10.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:10.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:10.079446+0000) 2020-02-05T22:28:10.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:10.239+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:10.240022+0000) 2020-02-05T22:28:10.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:10.314+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:10.314775+0000) 2020-02-05T22:28:10.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:10.330+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:10.331126+0000) 2020-02-05T22:28:12.328 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:12.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.327484+0000) 2020-02-05T22:28:12.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:12.502+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.504030+0000) 2020-02-05T22:28:12.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:12.504+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.505507+0000) 2020-02-05T22:28:12.507 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:12.505+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.506004+0000) 2020-02-05T22:28:12.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:12.505+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.506162+0000) 2020-02-05T22:28:12.507 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:12.505+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.506754+0000) 2020-02-05T22:28:12.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:12.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.507286+0000) 2020-02-05T22:28:12.508 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:12.506+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.507715+0000) 2020-02-05T22:28:12.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:12.507+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.507927+0000) 2020-02-05T22:28:12.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:12.509+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.509472+0000) 2020-02-05T22:28:12.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:12.509+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.509558+0000) 2020-02-05T22:28:12.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:12.508+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.509540+0000) 2020-02-05T22:28:12.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:12.509+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.510376+0000) 2020-02-05T22:28:12.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:12.509+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.510409+0000) 2020-02-05T22:28:12.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:12.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.514332+0000) 2020-02-05T22:28:12.519 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:12.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:12.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:12.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:12.721747+0000) 2020-02-05T22:28:13.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:13.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:13.840926+0000) 2020-02-05T22:28:13.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:13.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:13.840928+0000) 2020-02-05T22:28:15.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:15.935+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:15.935324+0000) 2020-02-05T22:28:15.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:15.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:15.943225+0000) 2020-02-05T22:28:17.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:17.081+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.082025+0000) 2020-02-05T22:28:17.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:17.081+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.082214+0000) 2020-02-05T22:28:17.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:17.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.082600+0000) 2020-02-05T22:28:17.525 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:17.523+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.524370+0000) 2020-02-05T22:28:17.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:17.524+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.524742+0000) 2020-02-05T22:28:17.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:17.525+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.525163+0000) 2020-02-05T22:28:17.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:17.525+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.526313+0000) 2020-02-05T22:28:17.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:17.526+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.526636+0000) 2020-02-05T22:28:17.528 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:17.527+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.527053+0000) 2020-02-05T22:28:17.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:17.527+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.527269+0000) 2020-02-05T22:28:17.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:17.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.527676+0000) 2020-02-05T22:28:17.529 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:17.527+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.527873+0000) 2020-02-05T22:28:17.530 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:17.527+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.528322+0000) 2020-02-05T22:28:17.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:17.528+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.528736+0000) 2020-02-05T22:28:17.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:17.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.528868+0000) 2020-02-05T22:28:17.531 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:17.529+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.530092+0000) 2020-02-05T22:28:17.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:17.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:17.533307+0000) 2020-02-05T22:28:17.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:17.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:19.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:19.472+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:19.472851+0000) 2020-02-05T22:28:20.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:20.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:20.079713+0000) 2020-02-05T22:28:20.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:20.248+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:20.248322+0000) 2020-02-05T22:28:20.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:20.314+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:20.315035+0000) 2020-02-05T22:28:20.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:20.330+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:20.331401+0000) 2020-02-05T22:28:22.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:22.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.327745+0000) 2020-02-05T22:28:22.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:22.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.545437+0000) 2020-02-05T22:28:22.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:22.544+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.545950+0000) 2020-02-05T22:28:22.547 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:22.545+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.546501+0000) 2020-02-05T22:28:22.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:22.545+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.546719+0000) 2020-02-05T22:28:22.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:22.546+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.547027+0000) 2020-02-05T22:28:22.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:22.546+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.547414+0000) 2020-02-05T22:28:22.549 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:22.547+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.548523+0000) 2020-02-05T22:28:22.549 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:22.547+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.548489+0000) 2020-02-05T22:28:22.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:22.548+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.548632+0000) 2020-02-05T22:28:22.550 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:22.548+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.548781+0000) 2020-02-05T22:28:22.551 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:22.549+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.549625+0000) 2020-02-05T22:28:22.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:22.548+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.549373+0000) 2020-02-05T22:28:22.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:22.549+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.549849+0000) 2020-02-05T22:28:22.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:22.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.553439+0000) 2020-02-05T22:28:22.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:22.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:22.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:22.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:22.721963+0000) 2020-02-05T22:28:23.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:23.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:23.841195+0000) 2020-02-05T22:28:23.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:23.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:23.841195+0000) 2020-02-05T22:28:25.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:25.934+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:25.935552+0000) 2020-02-05T22:28:25.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:25.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:25.943473+0000) 2020-02-05T22:28:27.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:27.081+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.082252+0000) 2020-02-05T22:28:27.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:27.081+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.082411+0000) 2020-02-05T22:28:27.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:27.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.082799+0000) 2020-02-05T22:28:27.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:27.562+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.563236+0000) 2020-02-05T22:28:27.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:27.564+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.565182+0000) 2020-02-05T22:28:27.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:27.564+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.565533+0000) 2020-02-05T22:28:27.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:27.565+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.565953+0000) 2020-02-05T22:28:27.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:27.564+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.565965+0000) 2020-02-05T22:28:27.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:27.564+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566041+0000) 2020-02-05T22:28:27.568 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:27.565+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566372+0000) 2020-02-05T22:28:27.568 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:27.565+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566580+0000) 2020-02-05T22:28:27.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:27.566+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566559+0000) 2020-02-05T22:28:27.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:27.566+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566699+0000) 2020-02-05T22:28:27.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:27.566+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.566975+0000) 2020-02-05T22:28:27.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:27.567+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.567919+0000) 2020-02-05T22:28:27.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:27.567+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.568033+0000) 2020-02-05T22:28:27.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:27.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:27.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:27.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:27.573777+0000) 2020-02-05T22:28:29.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:29.472+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:29.473121+0000) 2020-02-05T22:28:30.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:30.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:30.079941+0000) 2020-02-05T22:28:30.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:30.255+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:30.256663+0000) 2020-02-05T22:28:30.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:30.315+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:30.315288+0000) 2020-02-05T22:28:30.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:30.330+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:30.331684+0000) 2020-02-05T22:28:31.318 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:28:31.324 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:28:32.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:32.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.327984+0000) 2020-02-05T22:28:32.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:32.581+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.582787+0000) 2020-02-05T22:28:32.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:32.582+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.583313+0000) 2020-02-05T22:28:32.585 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:32.582+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.583543+0000) 2020-02-05T22:28:32.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:32.582+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.583816+0000) 2020-02-05T22:28:32.585 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:32.583+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.583910+0000) 2020-02-05T22:28:32.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:32.584+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.584576+0000) 2020-02-05T22:28:32.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:32.584+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.585136+0000) 2020-02-05T22:28:32.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:32.585+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.585752+0000) 2020-02-05T22:28:32.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:32.584+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.585805+0000) 2020-02-05T22:28:32.588 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:32.585+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.586337+0000) 2020-02-05T22:28:32.588 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:32.584+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.585338+0000) 2020-02-05T22:28:32.588 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:32.586+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.586615+0000) 2020-02-05T22:28:32.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:32.586+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.587291+0000) 2020-02-05T22:28:32.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:32.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.590877+0000) 2020-02-05T22:28:32.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:32.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:32.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:32.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:32.722197+0000) 2020-02-05T22:28:33.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:33.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:33.841464+0000) 2020-02-05T22:28:33.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:33.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:33.841464+0000) 2020-02-05T22:28:35.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:35.935+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:35.935780+0000) 2020-02-05T22:28:35.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:35.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:35.943707+0000) 2020-02-05T22:28:37.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:37.082+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.082532+0000) 2020-02-05T22:28:37.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:37.082+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.082726+0000) 2020-02-05T22:28:37.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:37.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.083036+0000) 2020-02-05T22:28:37.605 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:37.602+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.603738+0000) 2020-02-05T22:28:37.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:37.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.604225+0000) 2020-02-05T22:28:37.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:37.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.604555+0000) 2020-02-05T22:28:37.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:37.603+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.605012+0000) 2020-02-05T22:28:37.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:37.605+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.605593+0000) 2020-02-05T22:28:37.608 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:37.605+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.605855+0000) 2020-02-05T22:28:37.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:37.605+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.606321+0000) 2020-02-05T22:28:37.609 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:37.606+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.607069+0000) 2020-02-05T22:28:37.609 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:37.606+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.607411+0000) 2020-02-05T22:28:37.610 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:37.606+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.607131+0000) 2020-02-05T22:28:37.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:37.607+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.609061+0000) 2020-02-05T22:28:37.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:37.608+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.608732+0000) 2020-02-05T22:28:37.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:37.611+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.612413+0000) 2020-02-05T22:28:37.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:37.611+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:37.612765+0000) 2020-02-05T22:28:37.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:37.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:39.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:39.472+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:39.473357+0000) 2020-02-05T22:28:40.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:40.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:40.080176+0000) 2020-02-05T22:28:40.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:40.267+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:40.268368+0000) 2020-02-05T22:28:40.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:40.315+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:40.315524+0000) 2020-02-05T22:28:40.333 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:40.330+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:40.331929+0000) 2020-02-05T22:28:42.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:42.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.328229+0000) 2020-02-05T22:28:42.623 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:42.621+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.622597+0000) 2020-02-05T22:28:42.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:42.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.623721+0000) 2020-02-05T22:28:42.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:42.623+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.624493+0000) 2020-02-05T22:28:42.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:42.624+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.625212+0000) 2020-02-05T22:28:42.627 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:42.625+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.625979+0000) 2020-02-05T22:28:42.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:42.625+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.626927+0000) 2020-02-05T22:28:42.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:42.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.626815+0000) 2020-02-05T22:28:42.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:42.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.627583+0000) 2020-02-05T22:28:42.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:42.627+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.627691+0000) 2020-02-05T22:28:42.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:42.627+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.628244+0000) 2020-02-05T22:28:42.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:42.628+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.628800+0000) 2020-02-05T22:28:42.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:42.628+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.628738+0000) 2020-02-05T22:28:42.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:42.628+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.629124+0000) 2020-02-05T22:28:42.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:42.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.634371+0000) 2020-02-05T22:28:42.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:42.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:42.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:42.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:42.722409+0000) 2020-02-05T22:28:43.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:43.841+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:43.841739+0000) 2020-02-05T22:28:43.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:43.841+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:43.841739+0000) 2020-02-05T22:28:45.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:45.935+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:45.936127+0000) 2020-02-05T22:28:45.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:45.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:45.943955+0000) 2020-02-05T22:28:47.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:47.082+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.082789+0000) 2020-02-05T22:28:47.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:47.082+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.082958+0000) 2020-02-05T22:28:47.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:47.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.083267+0000) 2020-02-05T22:28:47.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:47.641+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.643047+0000) 2020-02-05T22:28:47.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:47.642+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.642971+0000) 2020-02-05T22:28:47.645 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:47.643+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.644165+0000) 2020-02-05T22:28:47.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:47.643+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.644519+0000) 2020-02-05T22:28:47.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:47.645+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.645631+0000) 2020-02-05T22:28:47.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:47.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.645930+0000) 2020-02-05T22:28:47.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:47.645+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.646937+0000) 2020-02-05T22:28:47.648 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:47.646+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.647473+0000) 2020-02-05T22:28:47.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:47.647+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.647844+0000) 2020-02-05T22:28:47.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:47.647+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.648170+0000) 2020-02-05T22:28:47.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:47.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.648857+0000) 2020-02-05T22:28:47.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:47.648+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.648827+0000) 2020-02-05T22:28:47.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:47.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.652569+0000) 2020-02-05T22:28:47.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:47.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:47.652869+0000) 2020-02-05T22:28:47.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:47.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116365 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:49.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:49.473+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:49.473591+0000) 2020-02-05T22:28:50.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:50.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:50.080404+0000) 2020-02-05T22:28:50.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:50.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:50.276197+0000) 2020-02-05T22:28:50.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:50.314+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:50.315759+0000) 2020-02-05T22:28:50.333 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:50.331+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:50.332189+0000) 2020-02-05T22:28:52.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:52.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.328496+0000) 2020-02-05T22:28:52.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:52.661+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.662642+0000) 2020-02-05T22:28:52.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:52.662+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.663533+0000) 2020-02-05T22:28:52.665 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:52.662+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.663922+0000) 2020-02-05T22:28:52.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:52.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.664917+0000) 2020-02-05T22:28:52.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:52.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.665549+0000) 2020-02-05T22:28:52.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:52.664+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.665993+0000) 2020-02-05T22:28:52.667 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:52.665+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.666312+0000) 2020-02-05T22:28:52.667 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:52.665+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.666741+0000) 2020-02-05T22:28:52.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:52.667+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.668206+0000) 2020-02-05T22:28:52.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:52.666+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.667551+0000) 2020-02-05T22:28:52.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:52.667+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.668507+0000) 2020-02-05T22:28:52.670 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:52.667+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.668940+0000) 2020-02-05T22:28:52.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:52.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.673411+0000) 2020-02-05T22:28:52.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:52.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.673716+0000) 2020-02-05T22:28:52.678 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:52.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:52.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:52.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:52.722645+0000) 2020-02-05T22:28:53.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:53.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:53.842027+0000) 2020-02-05T22:28:53.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:53.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:53.842026+0000) 2020-02-05T22:28:55.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:55.935+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:55.936358+0000) 2020-02-05T22:28:55.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:55.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:55.944188+0000) 2020-02-05T22:28:57.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:57.082+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.083046+0000) 2020-02-05T22:28:57.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:57.082+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.083237+0000) 2020-02-05T22:28:57.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:57.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.083499+0000) 2020-02-05T22:28:57.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:28:57.683+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.684419+0000) 2020-02-05T22:28:57.686 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:28:57.683+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.684887+0000) 2020-02-05T22:28:57.687 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:28:57.683+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.685242+0000) 2020-02-05T22:28:57.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:28:57.685+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.686722+0000) 2020-02-05T22:28:57.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:28:57.685+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.686677+0000) 2020-02-05T22:28:57.688 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:28:57.685+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.687094+0000) 2020-02-05T22:28:57.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:57.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.687853+0000) 2020-02-05T22:28:57.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:28:57.687+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.688118+0000) 2020-02-05T22:28:57.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:28:57.687+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.688588+0000) 2020-02-05T22:28:57.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:28:57.687+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.689205+0000) 2020-02-05T22:28:57.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:28:57.687+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.689385+0000) 2020-02-05T22:28:57.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:28:57.688+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.689734+0000) 2020-02-05T22:28:57.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:57.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.689541+0000) 2020-02-05T22:28:57.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:57.693+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:57.694654+0000) 2020-02-05T22:28:57.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:28:57.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:28:59.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:28:59.473+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:28:59.473819+0000) 2020-02-05T22:29:00.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:00.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:00.080646+0000) 2020-02-05T22:29:00.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:00.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:00.284549+0000) 2020-02-05T22:29:00.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:00.315+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:00.316015+0000) 2020-02-05T22:29:00.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:00.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:00.332456+0000) 2020-02-05T22:29:01.490 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:29:01.497 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:29:02.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:02.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.328768+0000) 2020-02-05T22:29:02.703 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:02.701+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.702233+0000) 2020-02-05T22:29:02.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:02.701+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.703028+0000) 2020-02-05T22:29:02.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:02.704+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.705214+0000) 2020-02-05T22:29:02.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:02.704+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.706030+0000) 2020-02-05T22:29:02.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:02.704+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.706377+0000) 2020-02-05T22:29:02.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:02.704+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.706298+0000) 2020-02-05T22:29:02.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:02.705+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.706988+0000) 2020-02-05T22:29:02.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:02.706+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.707575+0000) 2020-02-05T22:29:02.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:02.707+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.708270+0000) 2020-02-05T22:29:02.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:02.706+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.708182+0000) 2020-02-05T22:29:02.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:02.707+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.708657+0000) 2020-02-05T22:29:02.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:02.707+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.708623+0000) 2020-02-05T22:29:02.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:02.707+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.708902+0000) 2020-02-05T22:29:02.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:02.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.711892+0000) 2020-02-05T22:29:02.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:02.715+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:02.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:02.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:02.722808+0000) 2020-02-05T22:29:03.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:03.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:03.842338+0000) 2020-02-05T22:29:03.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:03.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:03.842338+0000) 2020-02-05T22:29:05.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:05.936+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:05.936583+0000) 2020-02-05T22:29:05.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:05.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:05.944384+0000) 2020-02-05T22:29:07.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:07.081+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.083331+0000) 2020-02-05T22:29:07.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:07.082+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.083510+0000) 2020-02-05T22:29:07.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:07.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.083727+0000) 2020-02-05T22:29:07.723 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:07.720+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.721801+0000) 2020-02-05T22:29:07.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:07.722+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.723448+0000) 2020-02-05T22:29:07.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:07.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.723750+0000) 2020-02-05T22:29:07.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:07.723+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.724177+0000) 2020-02-05T22:29:07.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:07.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.724969+0000) 2020-02-05T22:29:07.728 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:07.723+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.725253+0000) 2020-02-05T22:29:07.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:07.725+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.726459+0000) 2020-02-05T22:29:07.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:07.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.727251+0000) 2020-02-05T22:29:07.729 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:07.724+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.725907+0000) 2020-02-05T22:29:07.729 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:07.725+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.726873+0000) 2020-02-05T22:29:07.730 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:07.727+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.728768+0000) 2020-02-05T22:29:07.730 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:07.727+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.728938+0000) 2020-02-05T22:29:07.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:07.727+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.729169+0000) 2020-02-05T22:29:07.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:07.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:07.732870+0000) 2020-02-05T22:29:07.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:07.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:09.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:09.473+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:09.474072+0000) 2020-02-05T22:29:10.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:10.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:10.080882+0000) 2020-02-05T22:29:10.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:10.291+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:10.292061+0000) 2020-02-05T22:29:10.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:10.315+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:10.316289+0000) 2020-02-05T22:29:10.333 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:10.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:10.332685+0000) 2020-02-05T22:29:12.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:12.328+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.329072+0000) 2020-02-05T22:29:12.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:12.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.723027+0000) 2020-02-05T22:29:12.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:12.741+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.742003+0000) 2020-02-05T22:29:12.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:12.742+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.743052+0000) 2020-02-05T22:29:12.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:12.743+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.743461+0000) 2020-02-05T22:29:12.745 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:12.742+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.744122+0000) 2020-02-05T22:29:12.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:12.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.745035+0000) 2020-02-05T22:29:12.747 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:12.744+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.745796+0000) 2020-02-05T22:29:12.747 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:12.744+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.746108+0000) 2020-02-05T22:29:12.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:12.747+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.747208+0000) 2020-02-05T22:29:12.748 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:12.747+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.747463+0000) 2020-02-05T22:29:12.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:12.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.748174+0000) 2020-02-05T22:29:12.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:12.746+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.748274+0000) 2020-02-05T22:29:12.750 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:12.747+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.748675+0000) 2020-02-05T22:29:12.750 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:12.747+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.749050+0000) 2020-02-05T22:29:12.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:12.753+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:12.753270+0000) 2020-02-05T22:29:12.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:12.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:13.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:13.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:13.842615+0000) 2020-02-05T22:29:13.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:13.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:13.842617+0000) 2020-02-05T22:29:15.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:15.935+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:15.936828+0000) 2020-02-05T22:29:15.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:15.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:15.944692+0000) 2020-02-05T22:29:17.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:17.082+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.083517+0000) 2020-02-05T22:29:17.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:17.082+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.083706+0000) 2020-02-05T22:29:17.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:17.082+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.083974+0000) 2020-02-05T22:29:17.765 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:17.761+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.763302+0000) 2020-02-05T22:29:17.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:17.763+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.764381+0000) 2020-02-05T22:29:17.765 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:17.763+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.764741+0000) 2020-02-05T22:29:17.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:17.764+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.765947+0000) 2020-02-05T22:29:17.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:17.765+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.766620+0000) 2020-02-05T22:29:17.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:17.766+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.767315+0000) 2020-02-05T22:29:17.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:17.766+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.767946+0000) 2020-02-05T22:29:17.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:17.767+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.768487+0000) 2020-02-05T22:29:17.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:17.767+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.769114+0000) 2020-02-05T22:29:17.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:17.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.769865+0000) 2020-02-05T22:29:17.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:17.768+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.769813+0000) 2020-02-05T22:29:17.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:17.768+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.770181+0000) 2020-02-05T22:29:17.772 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:17.768+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.769996+0000) 2020-02-05T22:29:17.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:17.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:17.771188+0000) 2020-02-05T22:29:17.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:17.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:19.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:19.473+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:19.474342+0000) 2020-02-05T22:29:20.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:20.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:20.081122+0000) 2020-02-05T22:29:20.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:20.299+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:20.300071+0000) 2020-02-05T22:29:20.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:20.315+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:20.316539+0000) 2020-02-05T22:29:20.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:20.331+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:20.332919+0000) 2020-02-05T22:29:22.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:22.327+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.329321+0000) 2020-02-05T22:29:22.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:22.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.723298+0000) 2020-02-05T22:29:22.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:22.783+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.784761+0000) 2020-02-05T22:29:22.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:22.784+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.785373+0000) 2020-02-05T22:29:22.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:22.785+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.786220+0000) 2020-02-05T22:29:22.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:22.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.786486+0000) 2020-02-05T22:29:22.788 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:22.785+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.786661+0000) 2020-02-05T22:29:22.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:22.786+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.787672+0000) 2020-02-05T22:29:22.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:22.786+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.787824+0000) 2020-02-05T22:29:22.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:22.787+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.788415+0000) 2020-02-05T22:29:22.790 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:22.786+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.788364+0000) 2020-02-05T22:29:22.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:22.788+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.789331+0000) 2020-02-05T22:29:22.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:22.788+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.789824+0000) 2020-02-05T22:29:22.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:22.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.790743+0000) 2020-02-05T22:29:22.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:22.788+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.790176+0000) 2020-02-05T22:29:22.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:22.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:22.792991+0000) 2020-02-05T22:29:22.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:22.797+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:23.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:23.841+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:23.842885+0000) 2020-02-05T22:29:23.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:23.841+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:23.842887+0000) 2020-02-05T22:29:25.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:25.936+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:25.937061+0000) 2020-02-05T22:29:25.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:25.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:25.944930+0000) 2020-02-05T22:29:27.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:27.083+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.083782+0000) 2020-02-05T22:29:27.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:27.083+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.083949+0000) 2020-02-05T22:29:27.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:27.083+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.084211+0000) 2020-02-05T22:29:27.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:27.803+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.804510+0000) 2020-02-05T22:29:27.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:27.804+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.805505+0000) 2020-02-05T22:29:27.807 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:27.805+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.806606+0000) 2020-02-05T22:29:27.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:27.805+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.807184+0000) 2020-02-05T22:29:27.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:27.806+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.807677+0000) 2020-02-05T22:29:27.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:27.806+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.808059+0000) 2020-02-05T22:29:27.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:27.806+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.808406+0000) 2020-02-05T22:29:27.810 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:27.807+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.808989+0000) 2020-02-05T22:29:27.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:27.808+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.809448+0000) 2020-02-05T22:29:27.811 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:27.809+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.810552+0000) 2020-02-05T22:29:27.812 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:27.809+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.810494+0000) 2020-02-05T22:29:27.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:27.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.811471+0000) 2020-02-05T22:29:27.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:27.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.814788+0000) 2020-02-05T22:29:27.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:27.814+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:27.815113+0000) 2020-02-05T22:29:27.822 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:27.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:29.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:29.473+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:29.474598+0000) 2020-02-05T22:29:30.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:30.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:30.081351+0000) 2020-02-05T22:29:30.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:30.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:30.308169+0000) 2020-02-05T22:29:30.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:30.316+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:30.316778+0000) 2020-02-05T22:29:30.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:30.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:30.333164+0000) 2020-02-05T22:29:31.656 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:29:31.662 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:29:32.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:32.328+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.329622+0000) 2020-02-05T22:29:32.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:32.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.723541+0000) 2020-02-05T22:29:32.827 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:32.824+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.825812+0000) 2020-02-05T22:29:32.828 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:32.824+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.826363+0000) 2020-02-05T22:29:32.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:32.826+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.827391+0000) 2020-02-05T22:29:32.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:32.825+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.827047+0000) 2020-02-05T22:29:32.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:32.826+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.827841+0000) 2020-02-05T22:29:32.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:32.827+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.828159+0000) 2020-02-05T22:29:32.830 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:32.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.828489+0000) 2020-02-05T22:29:32.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:32.827+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.828752+0000) 2020-02-05T22:29:32.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:32.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.829770+0000) 2020-02-05T22:29:32.831 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:32.829+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.830228+0000) 2020-02-05T22:29:32.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:32.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.830223+0000) 2020-02-05T22:29:32.834 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:32.829+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.831248+0000) 2020-02-05T22:29:32.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:32.830+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.831287+0000) 2020-02-05T22:29:32.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:32.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:32.836627+0000) 2020-02-05T22:29:32.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:32.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:33.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:33.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:33.843159+0000) 2020-02-05T22:29:33.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:33.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:33.843157+0000) 2020-02-05T22:29:35.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:35.936+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:35.937323+0000) 2020-02-05T22:29:35.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:35.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:35.945219+0000) 2020-02-05T22:29:37.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:37.083+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.084035+0000) 2020-02-05T22:29:37.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:37.083+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.084188+0000) 2020-02-05T22:29:37.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:37.083+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.084554+0000) 2020-02-05T22:29:37.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:37.843+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.844866+0000) 2020-02-05T22:29:37.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:37.845+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.846685+0000) 2020-02-05T22:29:37.848 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:37.844+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.845954+0000) 2020-02-05T22:29:37.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:37.846+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.847493+0000) 2020-02-05T22:29:37.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:37.847+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.848200+0000) 2020-02-05T22:29:37.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:37.847+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.848832+0000) 2020-02-05T22:29:37.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:37.846+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.848329+0000) 2020-02-05T22:29:37.850 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:37.848+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.849300+0000) 2020-02-05T22:29:37.851 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:37.848+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.849659+0000) 2020-02-05T22:29:37.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:37.849+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.850503+0000) 2020-02-05T22:29:37.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:37.848+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.850073+0000) 2020-02-05T22:29:37.852 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:37.849+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.850486+0000) 2020-02-05T22:29:37.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:37.849+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.851021+0000) 2020-02-05T22:29:37.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:37.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:37.853604+0000) 2020-02-05T22:29:37.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:37.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:39.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:39.474+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:39.474828+0000) 2020-02-05T22:29:40.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:40.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:40.081583+0000) 2020-02-05T22:29:40.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:40.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:40.316479+0000) 2020-02-05T22:29:40.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:40.316+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:40.316998+0000) 2020-02-05T22:29:40.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:40.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:40.333427+0000) 2020-02-05T22:29:42.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:42.329+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.329869+0000) 2020-02-05T22:29:42.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:42.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.723779+0000) 2020-02-05T22:29:42.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:42.862+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.863365+0000) 2020-02-05T22:29:42.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:42.864+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.865350+0000) 2020-02-05T22:29:42.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:42.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.865840+0000) 2020-02-05T22:29:42.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:42.865+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.866666+0000) 2020-02-05T22:29:42.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:42.866+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.866914+0000) 2020-02-05T22:29:42.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:42.866+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.867090+0000) 2020-02-05T22:29:42.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:42.866+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.867272+0000) 2020-02-05T22:29:42.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:42.866+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.868008+0000) 2020-02-05T22:29:42.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:42.867+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.867780+0000) 2020-02-05T22:29:42.870 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:42.867+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.868003+0000) 2020-02-05T22:29:42.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:42.868+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.869526+0000) 2020-02-05T22:29:42.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:42.869+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.869580+0000) 2020-02-05T22:29:42.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:42.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.873043+0000) 2020-02-05T22:29:42.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:42.873+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:42.874304+0000) 2020-02-05T22:29:42.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:42.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:43.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:43.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:43.843428+0000) 2020-02-05T22:29:43.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:43.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:43.843430+0000) 2020-02-05T22:29:45.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:45.937+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:45.937555+0000) 2020-02-05T22:29:45.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:45.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:45.945490+0000) 2020-02-05T22:29:47.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:47.083+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.084266+0000) 2020-02-05T22:29:47.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:47.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.084511+0000) 2020-02-05T22:29:47.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:47.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.084769+0000) 2020-02-05T22:29:47.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:47.880+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.881420+0000) 2020-02-05T22:29:47.883 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:47.881+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.882423+0000) 2020-02-05T22:29:47.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:47.883+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.883481+0000) 2020-02-05T22:29:47.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:47.883+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.884363+0000) 2020-02-05T22:29:47.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:47.883+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.884413+0000) 2020-02-05T22:29:47.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:47.884+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.885498+0000) 2020-02-05T22:29:47.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:47.884+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.885703+0000) 2020-02-05T22:29:47.887 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:47.885+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.885675+0000) 2020-02-05T22:29:47.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:47.885+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.886329+0000) 2020-02-05T22:29:47.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:47.885+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.886761+0000) 2020-02-05T22:29:47.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:47.886+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.886727+0000) 2020-02-05T22:29:47.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:47.886+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.887192+0000) 2020-02-05T22:29:47.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:47.886+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.887337+0000) 2020-02-05T22:29:47.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:47.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:47.893516+0000) 2020-02-05T22:29:47.896 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:47.894+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:49.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:49.474+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:49.475067+0000) 2020-02-05T22:29:50.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:50.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:50.081815+0000) 2020-02-05T22:29:50.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:50.316+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:50.317247+0000) 2020-02-05T22:29:50.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:50.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:50.324671+0000) 2020-02-05T22:29:50.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:50.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:50.333646+0000) 2020-02-05T22:29:52.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:52.329+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.330128+0000) 2020-02-05T22:29:52.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:52.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.724018+0000) 2020-02-05T22:29:52.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:52.899+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.900702+0000) 2020-02-05T22:29:52.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:52.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.902055+0000) 2020-02-05T22:29:52.903 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:52.902+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.902632+0000) 2020-02-05T22:29:52.904 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:52.902+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.902854+0000) 2020-02-05T22:29:52.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:52.902+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.903319+0000) 2020-02-05T22:29:52.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:52.902+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.903933+0000) 2020-02-05T22:29:52.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:52.903+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.904134+0000) 2020-02-05T22:29:52.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:52.903+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.904370+0000) 2020-02-05T22:29:52.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:52.904+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.904958+0000) 2020-02-05T22:29:52.906 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:52.903+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.904816+0000) 2020-02-05T22:29:52.906 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:52.904+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.905181+0000) 2020-02-05T22:29:52.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:52.905+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.905975+0000) 2020-02-05T22:29:52.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:52.905+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.906635+0000) 2020-02-05T22:29:52.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:52.910+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:52.911336+0000) 2020-02-05T22:29:52.915 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:52.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116907 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:53.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:53.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:53.843670+0000) 2020-02-05T22:29:53.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:53.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:53.843670+0000) 2020-02-05T22:29:55.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:55.937+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:55.937795+0000) 2020-02-05T22:29:55.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:55.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:55.945725+0000) 2020-02-05T22:29:57.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:57.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.084516+0000) 2020-02-05T22:29:57.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:57.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.084726+0000) 2020-02-05T22:29:57.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:57.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.084964+0000) 2020-02-05T22:29:57.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:29:57.918+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.919574+0000) 2020-02-05T22:29:57.921 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:29:57.918+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.920009+0000) 2020-02-05T22:29:57.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:29:57.919+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.920777+0000) 2020-02-05T22:29:57.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:57.920+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.921548+0000) 2020-02-05T22:29:57.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:29:57.921+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.921696+0000) 2020-02-05T22:29:57.923 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:29:57.921+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.922350+0000) 2020-02-05T22:29:57.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:29:57.922+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.923572+0000) 2020-02-05T22:29:57.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:29:57.922+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.923171+0000) 2020-02-05T22:29:57.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:29:57.922+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.923908+0000) 2020-02-05T22:29:57.926 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:29:57.924+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.924680+0000) 2020-02-05T22:29:57.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:57.923+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.924191+0000) 2020-02-05T22:29:57.926 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:29:57.924+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.925221+0000) 2020-02-05T22:29:57.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:29:57.925+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.925708+0000) 2020-02-05T22:29:57.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:57.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:57.928365+0000) 2020-02-05T22:29:57.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:29:57.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:29:59.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:29:59.475+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:29:59.475384+0000) 2020-02-05T22:30:00.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:00.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:00.082042+0000) 2020-02-05T22:30:00.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:00.317+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:00.317482+0000) 2020-02-05T22:30:00.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:00.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:00.332896+0000) 2020-02-05T22:30:00.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:00.332+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:00.333853+0000) 2020-02-05T22:30:01.816 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:30:01.822 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:30:02.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:02.330+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.330421+0000) 2020-02-05T22:30:02.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:02.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.724242+0000) 2020-02-05T22:30:02.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:02.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.938839+0000) 2020-02-05T22:30:02.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:02.938+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.939523+0000) 2020-02-05T22:30:02.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:02.939+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.940053+0000) 2020-02-05T22:30:02.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:02.941+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.941549+0000) 2020-02-05T22:30:02.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:02.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.942288+0000) 2020-02-05T22:30:02.943 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:02.942+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.942566+0000) 2020-02-05T22:30:02.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:02.942+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.943172+0000) 2020-02-05T22:30:02.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:02.942+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.943463+0000) 2020-02-05T22:30:02.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:02.942+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.943996+0000) 2020-02-05T22:30:02.945 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:02.943+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.943934+0000) 2020-02-05T22:30:02.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:02.943+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.944039+0000) 2020-02-05T22:30:02.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:02.944+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.944805+0000) 2020-02-05T22:30:02.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:02.944+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.945022+0000) 2020-02-05T22:30:02.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:02.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:02.947337+0000) 2020-02-05T22:30:02.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:02.951+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 116991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:03.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:03.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:03.843915+0000) 2020-02-05T22:30:03.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:03.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:03.843915+0000) 2020-02-05T22:30:05.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:05.937+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:05.938024+0000) 2020-02-05T22:30:05.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:05.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:05.945955+0000) 2020-02-05T22:30:07.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:07.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.084771+0000) 2020-02-05T22:30:07.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:07.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.084929+0000) 2020-02-05T22:30:07.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:07.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.085215+0000) 2020-02-05T22:30:07.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:07.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.957341+0000) 2020-02-05T22:30:07.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:07.958+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.958644+0000) 2020-02-05T22:30:07.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:07.960+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.960239+0000) 2020-02-05T22:30:07.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:07.960+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.960640+0000) 2020-02-05T22:30:07.962 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:07.960+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.961020+0000) 2020-02-05T22:30:07.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:07.961+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.961577+0000) 2020-02-05T22:30:07.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:07.961+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.962140+0000) 2020-02-05T22:30:07.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:07.962+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.962555+0000) 2020-02-05T22:30:07.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:07.962+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.962854+0000) 2020-02-05T22:30:07.964 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:07.962+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.963377+0000) 2020-02-05T22:30:07.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:07.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.964001+0000) 2020-02-05T22:30:07.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:07.963+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.964152+0000) 2020-02-05T22:30:07.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:07.963+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.964345+0000) 2020-02-05T22:30:07.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:07.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:07.966635+0000) 2020-02-05T22:30:07.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:07.970+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:09.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:09.475+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:09.475619+0000) 2020-02-05T22:30:10.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:10.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:10.082308+0000) 2020-02-05T22:30:10.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:10.317+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:10.317704+0000) 2020-02-05T22:30:10.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:10.333+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:10.334117+0000) 2020-02-05T22:30:10.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:10.339+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:10.340894+0000) 2020-02-05T22:30:12.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:12.330+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.330662+0000) 2020-02-05T22:30:12.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:12.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.724444+0000) 2020-02-05T22:30:12.978 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:12.976+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.977202+0000) 2020-02-05T22:30:12.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:12.976+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.977582+0000) 2020-02-05T22:30:12.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:12.977+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.978065+0000) 2020-02-05T22:30:12.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:12.977+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.978367+0000) 2020-02-05T22:30:12.980 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:12.978+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.978783+0000) 2020-02-05T22:30:12.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:12.978+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.979066+0000) 2020-02-05T22:30:12.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:12.978+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.979576+0000) 2020-02-05T22:30:12.981 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:12.979+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.979937+0000) 2020-02-05T22:30:12.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:12.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.980419+0000) 2020-02-05T22:30:12.982 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:12.980+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.980698+0000) 2020-02-05T22:30:12.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:12.981+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.982484+0000) 2020-02-05T22:30:12.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:12.981+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.982329+0000) 2020-02-05T22:30:12.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:12.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.986594+0000) 2020-02-05T22:30:12.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:12.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:12.986885+0000) 2020-02-05T22:30:12.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:12.990+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117075 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:13.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:13.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:13.844158+0000) 2020-02-05T22:30:13.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:13.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:13.844156+0000) 2020-02-05T22:30:15.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:15.937+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:15.938293+0000) 2020-02-05T22:30:15.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:15.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:15.946204+0000) 2020-02-05T22:30:17.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:17.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.085008+0000) 2020-02-05T22:30:17.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:17.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.085142+0000) 2020-02-05T22:30:17.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:17.085+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.085491+0000) 2020-02-05T22:30:17.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:17.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.996592+0000) 2020-02-05T22:30:17.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:17.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.997465+0000) 2020-02-05T22:30:17.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:17.997+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.998452+0000) 2020-02-05T22:30:18.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:17.997+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.998945+0000) 2020-02-05T22:30:18.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:17.998+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:17.999312+0000) 2020-02-05T22:30:18.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:17.999+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.000133+0000) 2020-02-05T22:30:18.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:18.000+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.000563+0000) 2020-02-05T22:30:18.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:17.999+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.001048+0000) 2020-02-05T22:30:18.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:18.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.001177+0000) 2020-02-05T22:30:18.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:18.000+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.001274+0000) 2020-02-05T22:30:18.003 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:18.001+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.001479+0000) 2020-02-05T22:30:18.003 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:18.001+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.002883+0000) 2020-02-05T22:30:18.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:18.002+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.002701+0000) 2020-02-05T22:30:18.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:18.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:18.007194+0000) 2020-02-05T22:30:18.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:18.008+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:19.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:19.474+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:19.475882+0000) 2020-02-05T22:30:20.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:20.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:20.082538+0000) 2020-02-05T22:30:20.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:20.316+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:20.317957+0000) 2020-02-05T22:30:20.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:20.333+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:20.334338+0000) 2020-02-05T22:30:20.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:20.347+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:20.348276+0000) 2020-02-05T22:30:22.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:22.329+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:22.330859+0000) 2020-02-05T22:30:22.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:22.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:22.724694+0000) 2020-02-05T22:30:23.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:23.015+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.016829+0000) 2020-02-05T22:30:23.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:23.016+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.017153+0000) 2020-02-05T22:30:23.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:23.016+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.017355+0000) 2020-02-05T22:30:23.019 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:23.016+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.017563+0000) 2020-02-05T22:30:23.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:23.016+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.018116+0000) 2020-02-05T22:30:23.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:23.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.017969+0000) 2020-02-05T22:30:23.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:23.017+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.019134+0000) 2020-02-05T22:30:23.021 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:23.017+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.019132+0000) 2020-02-05T22:30:23.021 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:23.018+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.020271+0000) 2020-02-05T22:30:23.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:23.018+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.020139+0000) 2020-02-05T22:30:23.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:23.019+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.020707+0000) 2020-02-05T22:30:23.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:23.019+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.020851+0000) 2020-02-05T22:30:23.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:23.020+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.021306+0000) 2020-02-05T22:30:23.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:23.025+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.026482+0000) 2020-02-05T22:30:23.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:23.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:23.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:23.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.844399+0000) 2020-02-05T22:30:23.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:23.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:23.844401+0000) 2020-02-05T22:30:25.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:25.937+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:25.938525+0000) 2020-02-05T22:30:25.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:25.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:25.946482+0000) 2020-02-05T22:30:27.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:27.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:27.085270+0000) 2020-02-05T22:30:27.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:27.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:27.085413+0000) 2020-02-05T22:30:27.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:27.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:27.085756+0000) 2020-02-05T22:30:28.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:28.036+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.037605+0000) 2020-02-05T22:30:28.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:28.036+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.037522+0000) 2020-02-05T22:30:28.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:28.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.038160+0000) 2020-02-05T22:30:28.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:28.037+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.038459+0000) 2020-02-05T22:30:28.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:28.037+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.038709+0000) 2020-02-05T22:30:28.040 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:28.037+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.038908+0000) 2020-02-05T22:30:28.041 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:28.038+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.039673+0000) 2020-02-05T22:30:28.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:28.038+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.039474+0000) 2020-02-05T22:30:28.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:28.038+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.040310+0000) 2020-02-05T22:30:28.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:28.039+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.040933+0000) 2020-02-05T22:30:28.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:28.039+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.041281+0000) 2020-02-05T22:30:28.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:28.039+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.041357+0000) 2020-02-05T22:30:28.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:28.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.045318+0000) 2020-02-05T22:30:28.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:28.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:28.045582+0000) 2020-02-05T22:30:28.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:28.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:29.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:29.475+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:29.476139+0000) 2020-02-05T22:30:30.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:30.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:30.082780+0000) 2020-02-05T22:30:30.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:30.317+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:30.318211+0000) 2020-02-05T22:30:30.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:30.333+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:30.334610+0000) 2020-02-05T22:30:30.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:30.355+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:30.356413+0000) 2020-02-05T22:30:31.986 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:30:32.005 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:30:32.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:32.330+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:32.331164+0000) 2020-02-05T22:30:32.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:32.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:32.724954+0000) 2020-02-05T22:30:33.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:33.056+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.057064+0000) 2020-02-05T22:30:33.059 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:33.056+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.057773+0000) 2020-02-05T22:30:33.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:33.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.058443+0000) 2020-02-05T22:30:33.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:33.057+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.058740+0000) 2020-02-05T22:30:33.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:33.058+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.059631+0000) 2020-02-05T22:30:33.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:33.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.060158+0000) 2020-02-05T22:30:33.061 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:33.059+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.060597+0000) 2020-02-05T22:30:33.062 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:33.059+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.061317+0000) 2020-02-05T22:30:33.062 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:33.060+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.061807+0000) 2020-02-05T22:30:33.063 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:33.061+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.062382+0000) 2020-02-05T22:30:33.063 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:33.061+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.062685+0000) 2020-02-05T22:30:33.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:33.061+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.062940+0000) 2020-02-05T22:30:33.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:33.061+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.063033+0000) 2020-02-05T22:30:33.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:33.064+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.065843+0000) 2020-02-05T22:30:33.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:33.069+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:33.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:33.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.844657+0000) 2020-02-05T22:30:33.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:33.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:33.844657+0000) 2020-02-05T22:30:35.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:35.938+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:35.938750+0000) 2020-02-05T22:30:35.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:35.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:35.946680+0000) 2020-02-05T22:30:37.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:37.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:37.085567+0000) 2020-02-05T22:30:37.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:37.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:37.085723+0000) 2020-02-05T22:30:37.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:37.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:37.085991+0000) 2020-02-05T22:30:38.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:38.075+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.076637+0000) 2020-02-05T22:30:38.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:38.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.077317+0000) 2020-02-05T22:30:38.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:38.076+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.077514+0000) 2020-02-05T22:30:38.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:38.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.077925+0000) 2020-02-05T22:30:38.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:38.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.078117+0000) 2020-02-05T22:30:38.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:38.077+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.079003+0000) 2020-02-05T22:30:38.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:38.078+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.079526+0000) 2020-02-05T22:30:38.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:38.079+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.080169+0000) 2020-02-05T22:30:38.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:38.079+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.080421+0000) 2020-02-05T22:30:38.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:38.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.081229+0000) 2020-02-05T22:30:38.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:38.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.081592+0000) 2020-02-05T22:30:38.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:38.079+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.081007+0000) 2020-02-05T22:30:38.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:38.080+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.081852+0000) 2020-02-05T22:30:38.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:38.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:38.084452+0000) 2020-02-05T22:30:38.091 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:38.089+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:39.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:39.475+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:39.476427+0000) 2020-02-05T22:30:40.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:40.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:40.083029+0000) 2020-02-05T22:30:40.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:40.317+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:40.318486+0000) 2020-02-05T22:30:40.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:40.334+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:40.334838+0000) 2020-02-05T22:30:40.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:40.363+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:40.364472+0000) 2020-02-05T22:30:42.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:42.330+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:42.331460+0000) 2020-02-05T22:30:42.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:42.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:42.725196+0000) 2020-02-05T22:30:43.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:43.095+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.096787+0000) 2020-02-05T22:30:43.098 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:43.096+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.097594+0000) 2020-02-05T22:30:43.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:43.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.098786+0000) 2020-02-05T22:30:43.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:43.098+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.099700+0000) 2020-02-05T22:30:43.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:43.099+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.100500+0000) 2020-02-05T22:30:43.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:43.099+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.100552+0000) 2020-02-05T22:30:43.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:43.099+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.101056+0000) 2020-02-05T22:30:43.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:43.100+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.101572+0000) 2020-02-05T22:30:43.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:43.100+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.101801+0000) 2020-02-05T22:30:43.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:43.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.102460+0000) 2020-02-05T22:30:43.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:43.100+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.102323+0000) 2020-02-05T22:30:43.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:43.101+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.102751+0000) 2020-02-05T22:30:43.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:43.101+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.102893+0000) 2020-02-05T22:30:43.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:43.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.106060+0000) 2020-02-05T22:30:43.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:43.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:43.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:43.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.844903+0000) 2020-02-05T22:30:43.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:43.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:43.844903+0000) 2020-02-05T22:30:45.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:45.938+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:45.938989+0000) 2020-02-05T22:30:45.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:45.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:45.946921+0000) 2020-02-05T22:30:47.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:47.084+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:47.085794+0000) 2020-02-05T22:30:47.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:47.084+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:47.086019+0000) 2020-02-05T22:30:47.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:47.084+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:47.086207+0000) 2020-02-05T22:30:48.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:48.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.115431+0000) 2020-02-05T22:30:48.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:48.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.116353+0000) 2020-02-05T22:30:48.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:48.115+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.116683+0000) 2020-02-05T22:30:48.118 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:48.116+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.117241+0000) 2020-02-05T22:30:48.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:48.116+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.118077+0000) 2020-02-05T22:30:48.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:48.117+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.118976+0000) 2020-02-05T22:30:48.120 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:48.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.119638+0000) 2020-02-05T22:30:48.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:48.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.120470+0000) 2020-02-05T22:30:48.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:48.119+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.120730+0000) 2020-02-05T22:30:48.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:48.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.121509+0000) 2020-02-05T22:30:48.123 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:48.120+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.121642+0000) 2020-02-05T22:30:48.123 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:48.120+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.121724+0000) 2020-02-05T22:30:48.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:48.120+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.121913+0000) 2020-02-05T22:30:48.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:48.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:48.125635+0000) 2020-02-05T22:30:48.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:48.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:49.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:49.476+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:49.476657+0000) 2020-02-05T22:30:50.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:50.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:50.083294+0000) 2020-02-05T22:30:50.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:50.317+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:50.318718+0000) 2020-02-05T22:30:50.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:50.334+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:50.335070+0000) 2020-02-05T22:30:50.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:50.372+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:50.372619+0000) 2020-02-05T22:30:52.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:52.331+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:52.331721+0000) 2020-02-05T22:30:52.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:52.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:52.725463+0000) 2020-02-05T22:30:53.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:53.137+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.138629+0000) 2020-02-05T22:30:53.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:53.137+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.139082+0000) 2020-02-05T22:30:53.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:53.138+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.139523+0000) 2020-02-05T22:30:53.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:53.139+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.140118+0000) 2020-02-05T22:30:53.141 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:53.139+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.140770+0000) 2020-02-05T22:30:53.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:53.140+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.141212+0000) 2020-02-05T22:30:53.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:53.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.141654+0000) 2020-02-05T22:30:53.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:53.141+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.142428+0000) 2020-02-05T22:30:53.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:53.141+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.142756+0000) 2020-02-05T22:30:53.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:53.141+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.143149+0000) 2020-02-05T22:30:53.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:53.142+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.143856+0000) 2020-02-05T22:30:53.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:53.142+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.144194+0000) 2020-02-05T22:30:53.145 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:53.143+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.144459+0000) 2020-02-05T22:30:53.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:53.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.149227+0000) 2020-02-05T22:30:53.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:53.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117411 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:53.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:53.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.845154+0000) 2020-02-05T22:30:53.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:53.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:53.845153+0000) 2020-02-05T22:30:55.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:55.939+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:55.939308+0000) 2020-02-05T22:30:55.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:55.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:55.947155+0000) 2020-02-05T22:30:57.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:57.085+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:57.086060+0000) 2020-02-05T22:30:57.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:57.085+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:57.086264+0000) 2020-02-05T22:30:57.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:57.085+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:57.086493+0000) 2020-02-05T22:30:58.159 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:30:58.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.157645+0000) 2020-02-05T22:30:58.160 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:30:58.157+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.158942+0000) 2020-02-05T22:30:58.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:30:58.158+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.160228+0000) 2020-02-05T22:30:58.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:58.159+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.160706+0000) 2020-02-05T22:30:58.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:30:58.160+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.161452+0000) 2020-02-05T22:30:58.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:30:58.160+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.161994+0000) 2020-02-05T22:30:58.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:30:58.161+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.162368+0000) 2020-02-05T22:30:58.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:30:58.161+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.163065+0000) 2020-02-05T22:30:58.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:30:58.161+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.163381+0000) 2020-02-05T22:30:58.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:30:58.163+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.164165+0000) 2020-02-05T22:30:58.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:30:58.162+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.164068+0000) 2020-02-05T22:30:58.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:30:58.162+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.164359+0000) 2020-02-05T22:30:58.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:58.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.165158+0000) 2020-02-05T22:30:58.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:58.164+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:58.165575+0000) 2020-02-05T22:30:58.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:30:58.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117453 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:30:59.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:30:59.476+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:30:59.476909+0000) 2020-02-05T22:31:00.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:00.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:00.083528+0000) 2020-02-05T22:31:00.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:00.318+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:00.318982+0000) 2020-02-05T22:31:00.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:00.335+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:00.335395+0000) 2020-02-05T22:31:00.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:00.380+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:00.380717+0000) 2020-02-05T22:31:02.203 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:31:02.210 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:31:02.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:02.331+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:02.332011+0000) 2020-02-05T22:31:02.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:02.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:02.725716+0000) 2020-02-05T22:31:03.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:03.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.179453+0000) 2020-02-05T22:31:03.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:03.180+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.180106+0000) 2020-02-05T22:31:03.181 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:03.179+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.180587+0000) 2020-02-05T22:31:03.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:03.180+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.181049+0000) 2020-02-05T22:31:03.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:03.180+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.181622+0000) 2020-02-05T22:31:03.183 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:03.180+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.181981+0000) 2020-02-05T22:31:03.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:03.181+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.182527+0000) 2020-02-05T22:31:03.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:03.182+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.182959+0000) 2020-02-05T22:31:03.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:03.183+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.183598+0000) 2020-02-05T22:31:03.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:03.182+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.184117+0000) 2020-02-05T22:31:03.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:03.183+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.184045+0000) 2020-02-05T22:31:03.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:03.183+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.184806+0000) 2020-02-05T22:31:03.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:03.190+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.190834+0000) 2020-02-05T22:31:03.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:03.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.191158+0000) 2020-02-05T22:31:03.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:03.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:03.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:03.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.845403+0000) 2020-02-05T22:31:03.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:03.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:03.845403+0000) 2020-02-05T22:31:05.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:05.938+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:05.939535+0000) 2020-02-05T22:31:05.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:05.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:05.947398+0000) 2020-02-05T22:31:07.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:07.085+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:07.086311+0000) 2020-02-05T22:31:07.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:07.085+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:07.086549+0000) 2020-02-05T22:31:07.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:07.085+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:07.086771+0000) 2020-02-05T22:31:08.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:08.198+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.199602+0000) 2020-02-05T22:31:08.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:08.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.200450+0000) 2020-02-05T22:31:08.203 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:08.200+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.201481+0000) 2020-02-05T22:31:08.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:08.200+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.201672+0000) 2020-02-05T22:31:08.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:08.201+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.202387+0000) 2020-02-05T22:31:08.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:08.201+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.202642+0000) 2020-02-05T22:31:08.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:08.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.203544+0000) 2020-02-05T22:31:08.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:08.202+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.203845+0000) 2020-02-05T22:31:08.205 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:08.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.204308+0000) 2020-02-05T22:31:08.205 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:08.203+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.204754+0000) 2020-02-05T22:31:08.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:08.204+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.205542+0000) 2020-02-05T22:31:08.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:08.204+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.205742+0000) 2020-02-05T22:31:08.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:08.204+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.205596+0000) 2020-02-05T22:31:08.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:08.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:08.211392+0000) 2020-02-05T22:31:08.214 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:08.211+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117536 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:09.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:09.476+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:09.477147+0000) 2020-02-05T22:31:10.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:10.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:10.083773+0000) 2020-02-05T22:31:10.320 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:10.318+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:10.319276+0000) 2020-02-05T22:31:10.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:10.334+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:10.335623+0000) 2020-02-05T22:31:10.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:10.387+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:10.388803+0000) 2020-02-05T22:31:12.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:12.331+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:12.332299+0000) 2020-02-05T22:31:12.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:12.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:12.725965+0000) 2020-02-05T22:31:13.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:13.217+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.218268+0000) 2020-02-05T22:31:13.220 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:13.218+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.219352+0000) 2020-02-05T22:31:13.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:13.218+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.219846+0000) 2020-02-05T22:31:13.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:13.219+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.220298+0000) 2020-02-05T22:31:13.221 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:13.219+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.220606+0000) 2020-02-05T22:31:13.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:13.219+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.220938+0000) 2020-02-05T22:31:13.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:13.220+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.221301+0000) 2020-02-05T22:31:13.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:13.220+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.221589+0000) 2020-02-05T22:31:13.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:13.221+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.221802+0000) 2020-02-05T22:31:13.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:13.221+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.222363+0000) 2020-02-05T22:31:13.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:13.222+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.222726+0000) 2020-02-05T22:31:13.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:13.223+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.223643+0000) 2020-02-05T22:31:13.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:13.223+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.224112+0000) 2020-02-05T22:31:13.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:13.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.228405+0000) 2020-02-05T22:31:13.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:13.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:13.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:13.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.845644+0000) 2020-02-05T22:31:13.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:13.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:13.845644+0000) 2020-02-05T22:31:15.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:15.938+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:15.939751+0000) 2020-02-05T22:31:15.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:15.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:15.947715+0000) 2020-02-05T22:31:17.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:17.086+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:17.086631+0000) 2020-02-05T22:31:17.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:17.086+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:17.086805+0000) 2020-02-05T22:31:17.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:17.086+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:17.087132+0000) 2020-02-05T22:31:18.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:18.237+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.237862+0000) 2020-02-05T22:31:18.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:18.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.238159+0000) 2020-02-05T22:31:18.240 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:18.237+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.238714+0000) 2020-02-05T22:31:18.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:18.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.239732+0000) 2020-02-05T22:31:18.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:18.239+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.240628+0000) 2020-02-05T22:31:18.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:18.239+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.240319+0000) 2020-02-05T22:31:18.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:18.240+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.240935+0000) 2020-02-05T22:31:18.242 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:18.240+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.241375+0000) 2020-02-05T22:31:18.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:18.240+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.241964+0000) 2020-02-05T22:31:18.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:18.242+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.242551+0000) 2020-02-05T22:31:18.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:18.242+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.242993+0000) 2020-02-05T22:31:18.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:18.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.243269+0000) 2020-02-05T22:31:18.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:18.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.244506+0000) 2020-02-05T22:31:18.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:18.244+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:18.245490+0000) 2020-02-05T22:31:18.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:18.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:19.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:19.476+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:19.477364+0000) 2020-02-05T22:31:20.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:20.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:20.084023+0000) 2020-02-05T22:31:20.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:20.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:20.319533+0000) 2020-02-05T22:31:20.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:20.334+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:20.335844+0000) 2020-02-05T22:31:20.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:20.396+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:20.396923+0000) 2020-02-05T22:31:22.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:22.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:22.332559+0000) 2020-02-05T22:31:22.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:22.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:22.726194+0000) 2020-02-05T22:31:23.257 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:23.255+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.256327+0000) 2020-02-05T22:31:23.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:23.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.257248+0000) 2020-02-05T22:31:23.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:23.258+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.258670+0000) 2020-02-05T22:31:23.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:23.258+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.259131+0000) 2020-02-05T22:31:23.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:23.259+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.260104+0000) 2020-02-05T22:31:23.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:23.259+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.260296+0000) 2020-02-05T22:31:23.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:23.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.260802+0000) 2020-02-05T22:31:23.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:23.262+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.262770+0000) 2020-02-05T22:31:23.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:23.262+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.263321+0000) 2020-02-05T22:31:23.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:23.262+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.263958+0000) 2020-02-05T22:31:23.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:23.263+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.264522+0000) 2020-02-05T22:31:23.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:23.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.265749+0000) 2020-02-05T22:31:23.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:23.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.267118+0000) 2020-02-05T22:31:23.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:23.262+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.263288+0000) 2020-02-05T22:31:23.273 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:23.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:23.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:23.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.845884+0000) 2020-02-05T22:31:23.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:23.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:23.845886+0000) 2020-02-05T22:31:25.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:25.939+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:25.939976+0000) 2020-02-05T22:31:25.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:25.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:25.947973+0000) 2020-02-05T22:31:27.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:27.086+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:27.086892+0000) 2020-02-05T22:31:27.089 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:27.086+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:27.087087+0000) 2020-02-05T22:31:27.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:27.086+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:27.087374+0000) 2020-02-05T22:31:28.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:28.275+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.277045+0000) 2020-02-05T22:31:28.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:28.278+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.278640+0000) 2020-02-05T22:31:28.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:28.278+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.279422+0000) 2020-02-05T22:31:28.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:28.279+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.280374+0000) 2020-02-05T22:31:28.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:28.280+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.281070+0000) 2020-02-05T22:31:28.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:28.280+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.281330+0000) 2020-02-05T22:31:28.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:28.281+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.281538+0000) 2020-02-05T22:31:28.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:28.280+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.282015+0000) 2020-02-05T22:31:28.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:28.282+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.282565+0000) 2020-02-05T22:31:28.285 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:28.283+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.284234+0000) 2020-02-05T22:31:28.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:28.283+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.284250+0000) 2020-02-05T22:31:28.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:28.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.284593+0000) 2020-02-05T22:31:28.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:28.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.284771+0000) 2020-02-05T22:31:28.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:28.285+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:28.286589+0000) 2020-02-05T22:31:28.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:28.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:29.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:29.477+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:29.477621+0000) 2020-02-05T22:31:30.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:30.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:30.084302+0000) 2020-02-05T22:31:30.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:30.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:30.319767+0000) 2020-02-05T22:31:30.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:30.335+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:30.336119+0000) 2020-02-05T22:31:30.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:30.403+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:30.404489+0000) 2020-02-05T22:31:32.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:32.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:32.332813+0000) 2020-02-05T22:31:32.344 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:31:32.350 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:31:32.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:32.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:32.726400+0000) 2020-02-05T22:31:33.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:33.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.298352+0000) 2020-02-05T22:31:33.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:33.298+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.298748+0000) 2020-02-05T22:31:33.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:33.300+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.300518+0000) 2020-02-05T22:31:33.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:33.300+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.300493+0000) 2020-02-05T22:31:33.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:33.299+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.300177+0000) 2020-02-05T22:31:33.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:33.300+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.301446+0000) 2020-02-05T22:31:33.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:33.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.301572+0000) 2020-02-05T22:31:33.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:33.300+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.300650+0000) 2020-02-05T22:31:33.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:33.301+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.302356+0000) 2020-02-05T22:31:33.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:33.301+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.302526+0000) 2020-02-05T22:31:33.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:33.301+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.302876+0000) 2020-02-05T22:31:33.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:33.302+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.302956+0000) 2020-02-05T22:31:33.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:33.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.303637+0000) 2020-02-05T22:31:33.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:33.307+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.308088+0000) 2020-02-05T22:31:33.313 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:33.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:33.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:33.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.846145+0000) 2020-02-05T22:31:33.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:33.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:33.846145+0000) 2020-02-05T22:31:35.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:35.939+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:35.940125+0000) 2020-02-05T22:31:35.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:35.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:35.948262+0000) 2020-02-05T22:31:37.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:37.086+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:37.087164+0000) 2020-02-05T22:31:37.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:37.086+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:37.087333+0000) 2020-02-05T22:31:37.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:37.087+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:37.087746+0000) 2020-02-05T22:31:38.319 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:38.317+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.318727+0000) 2020-02-05T22:31:38.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:38.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.319946+0000) 2020-02-05T22:31:38.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:38.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.320702+0000) 2020-02-05T22:31:38.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:38.320+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.320659+0000) 2020-02-05T22:31:38.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:38.319+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.321043+0000) 2020-02-05T22:31:38.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:38.321+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.321853+0000) 2020-02-05T22:31:38.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:38.321+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.322032+0000) 2020-02-05T22:31:38.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:38.321+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.322812+0000) 2020-02-05T22:31:38.324 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:38.322+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.323586+0000) 2020-02-05T22:31:38.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:38.323+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.324229+0000) 2020-02-05T22:31:38.325 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:38.323+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.324198+0000) 2020-02-05T22:31:38.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:38.323+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.324350+0000) 2020-02-05T22:31:38.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:38.324+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.324924+0000) 2020-02-05T22:31:38.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:38.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:38.329500+0000) 2020-02-05T22:31:38.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:38.331+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:39.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:39.477+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:39.477871+0000) 2020-02-05T22:31:40.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:40.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:40.084537+0000) 2020-02-05T22:31:40.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:40.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:40.320017+0000) 2020-02-05T22:31:40.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:40.335+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:40.336399+0000) 2020-02-05T22:31:40.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:40.412+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:40.412611+0000) 2020-02-05T22:31:42.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:42.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:42.333066+0000) 2020-02-05T22:31:42.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:42.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:42.726613+0000) 2020-02-05T22:31:43.339 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:43.336+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.337988+0000) 2020-02-05T22:31:43.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:43.337+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.338523+0000) 2020-02-05T22:31:43.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:43.338+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.338772+0000) 2020-02-05T22:31:43.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:43.338+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.339263+0000) 2020-02-05T22:31:43.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:43.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.339707+0000) 2020-02-05T22:31:43.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:43.338+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.339850+0000) 2020-02-05T22:31:43.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:43.339+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.340261+0000) 2020-02-05T22:31:43.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:43.340+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.340597+0000) 2020-02-05T22:31:43.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:43.339+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.340222+0000) 2020-02-05T22:31:43.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:43.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.340383+0000) 2020-02-05T22:31:43.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:43.340+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.340951+0000) 2020-02-05T22:31:43.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:43.341+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.341698+0000) 2020-02-05T22:31:43.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:43.341+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.341917+0000) 2020-02-05T22:31:43.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:43.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.348003+0000) 2020-02-05T22:31:43.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:43.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:43.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:43.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.846387+0000) 2020-02-05T22:31:43.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:43.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:43.846387+0000) 2020-02-05T22:31:45.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:45.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:45.940348+0000) 2020-02-05T22:31:45.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:45.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:45.948508+0000) 2020-02-05T22:31:47.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:47.087+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:47.087393+0000) 2020-02-05T22:31:47.089 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:47.087+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:47.087642+0000) 2020-02-05T22:31:47.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:47.087+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:47.087993+0000) 2020-02-05T22:31:48.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:48.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.358470+0000) 2020-02-05T22:31:48.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:48.357+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.358830+0000) 2020-02-05T22:31:48.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:48.358+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.359370+0000) 2020-02-05T22:31:48.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:48.358+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.360374+0000) 2020-02-05T22:31:48.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:48.359+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.360688+0000) 2020-02-05T22:31:48.362 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:48.360+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.361066+0000) 2020-02-05T22:31:48.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:48.360+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.361568+0000) 2020-02-05T22:31:48.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:48.361+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.362058+0000) 2020-02-05T22:31:48.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:48.360+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.362332+0000) 2020-02-05T22:31:48.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:48.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.362661+0000) 2020-02-05T22:31:48.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:48.361+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.362916+0000) 2020-02-05T22:31:48.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:48.361+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.363233+0000) 2020-02-05T22:31:48.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:48.361+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.363297+0000) 2020-02-05T22:31:48.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:48.364+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:48.365654+0000) 2020-02-05T22:31:48.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:48.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:49.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:49.477+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:49.478131+0000) 2020-02-05T22:31:50.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:50.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:50.084768+0000) 2020-02-05T22:31:50.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:50.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:50.320265+0000) 2020-02-05T22:31:50.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:50.336+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:50.336632+0000) 2020-02-05T22:31:50.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:50.419+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:50.420844+0000) 2020-02-05T22:31:52.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:52.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:52.333317+0000) 2020-02-05T22:31:52.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:52.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:52.726896+0000) 2020-02-05T22:31:53.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:53.375+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.376662+0000) 2020-02-05T22:31:53.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:53.376+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.377778+0000) 2020-02-05T22:31:53.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:53.376+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.377614+0000) 2020-02-05T22:31:53.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:53.377+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.378678+0000) 2020-02-05T22:31:53.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:53.377+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.378476+0000) 2020-02-05T22:31:53.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:53.379+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.380112+0000) 2020-02-05T22:31:53.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:53.379+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.380483+0000) 2020-02-05T22:31:53.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:53.380+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.381054+0000) 2020-02-05T22:31:53.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:53.380+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.381506+0000) 2020-02-05T22:31:53.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:53.380+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.381605+0000) 2020-02-05T22:31:53.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:53.380+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.382004+0000) 2020-02-05T22:31:53.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:53.381+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.382319+0000) 2020-02-05T22:31:53.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:53.381+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.382556+0000) 2020-02-05T22:31:53.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:53.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.385148+0000) 2020-02-05T22:31:53.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:53.390+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117918 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:53.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:53.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.846642+0000) 2020-02-05T22:31:53.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:53.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:53.846642+0000) 2020-02-05T22:31:55.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:55.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:55.940599+0000) 2020-02-05T22:31:55.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:55.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:55.948746+0000) 2020-02-05T22:31:57.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:57.086+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:57.087680+0000) 2020-02-05T22:31:57.089 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:57.086+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:57.087871+0000) 2020-02-05T22:31:57.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:57.086+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:57.088235+0000) 2020-02-05T22:31:58.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:31:58.396+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.397917+0000) 2020-02-05T22:31:58.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:58.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.398379+0000) 2020-02-05T22:31:58.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:58.399+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.399255+0000) 2020-02-05T22:31:58.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:31:58.398+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.399490+0000) 2020-02-05T22:31:58.401 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:31:58.398+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.399920+0000) 2020-02-05T22:31:58.402 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:31:58.400+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.400266+0000) 2020-02-05T22:31:58.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:31:58.400+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.400807+0000) 2020-02-05T22:31:58.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:31:58.401+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.401183+0000) 2020-02-05T22:31:58.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:31:58.399+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.400650+0000) 2020-02-05T22:31:58.403 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:31:58.401+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.401588+0000) 2020-02-05T22:31:58.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:31:58.401+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.402600+0000) 2020-02-05T22:31:58.404 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:31:58.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.403577+0000) 2020-02-05T22:31:58.405 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:31:58.401+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.403136+0000) 2020-02-05T22:31:58.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:58.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:58.408063+0000) 2020-02-05T22:31:58.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:31:58.411+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 117959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:31:59.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:31:59.477+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:31:59.478378+0000) 2020-02-05T22:32:00.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:00.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:00.085001+0000) 2020-02-05T22:32:00.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:00.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:00.320506+0000) 2020-02-05T22:32:00.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:00.335+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:00.336860+0000) 2020-02-05T22:32:00.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:00.427+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:00.428964+0000) 2020-02-05T22:32:02.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:02.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:02.333575+0000) 2020-02-05T22:32:02.477 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:32:02.483 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:32:02.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:02.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:02.727166+0000) 2020-02-05T22:32:03.420 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:03.418+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.419634+0000) 2020-02-05T22:32:03.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:03.418+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.420128+0000) 2020-02-05T22:32:03.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:03.418+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.419974+0000) 2020-02-05T22:32:03.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:03.419+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.420797+0000) 2020-02-05T22:32:03.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:03.419+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.421098+0000) 2020-02-05T22:32:03.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:03.421+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.422371+0000) 2020-02-05T22:32:03.423 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:03.421+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.422836+0000) 2020-02-05T22:32:03.424 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:03.422+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.423363+0000) 2020-02-05T22:32:03.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:03.421+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.423323+0000) 2020-02-05T22:32:03.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:03.422+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.423833+0000) 2020-02-05T22:32:03.425 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:03.423+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.424490+0000) 2020-02-05T22:32:03.426 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:03.423+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.424805+0000) 2020-02-05T22:32:03.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:03.423+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.424929+0000) 2020-02-05T22:32:03.435 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:03.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:03.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:03.430+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.431465+0000) 2020-02-05T22:32:03.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:03.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.846889+0000) 2020-02-05T22:32:03.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:03.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:03.846887+0000) 2020-02-05T22:32:05.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:05.939+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:05.940843+0000) 2020-02-05T22:32:05.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:05.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:05.948996+0000) 2020-02-05T22:32:07.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:07.087+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:07.087914+0000) 2020-02-05T22:32:07.089 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:07.087+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:07.088122+0000) 2020-02-05T22:32:07.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:07.087+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:07.088477+0000) 2020-02-05T22:32:08.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:08.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.438569+0000) 2020-02-05T22:32:08.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:08.438+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.439109+0000) 2020-02-05T22:32:08.441 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:08.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.440139+0000) 2020-02-05T22:32:08.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:08.439+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.440552+0000) 2020-02-05T22:32:08.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:08.440+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.441239+0000) 2020-02-05T22:32:08.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:08.440+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.441401+0000) 2020-02-05T22:32:08.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:08.441+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.442306+0000) 2020-02-05T22:32:08.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:08.440+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.442052+0000) 2020-02-05T22:32:08.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:08.440+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.442247+0000) 2020-02-05T22:32:08.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:08.441+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.442760+0000) 2020-02-05T22:32:08.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:08.442+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.443424+0000) 2020-02-05T22:32:08.445 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:08.442+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.443421+0000) 2020-02-05T22:32:08.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:08.442+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.443833+0000) 2020-02-05T22:32:08.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:08.443+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:08.444962+0000) 2020-02-05T22:32:08.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:08.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:09.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:09.477+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:09.478611+0000) 2020-02-05T22:32:10.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:10.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:10.085268+0000) 2020-02-05T22:32:10.322 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:10.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:10.320704+0000) 2020-02-05T22:32:10.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:10.336+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:10.337126+0000) 2020-02-05T22:32:10.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:10.436+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:10.437303+0000) 2020-02-05T22:32:12.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:12.333+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:12.333831+0000) 2020-02-05T22:32:12.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:12.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:12.727375+0000) 2020-02-05T22:32:13.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:13.458+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.459682+0000) 2020-02-05T22:32:13.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:13.459+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.460897+0000) 2020-02-05T22:32:13.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:13.459+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.461391+0000) 2020-02-05T22:32:13.463 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:13.460+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.462039+0000) 2020-02-05T22:32:13.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:13.461+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.462299+0000) 2020-02-05T22:32:13.463 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:13.461+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.462688+0000) 2020-02-05T22:32:13.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:13.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.463143+0000) 2020-02-05T22:32:13.464 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:13.462+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.463542+0000) 2020-02-05T22:32:13.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:13.463+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.464224+0000) 2020-02-05T22:32:13.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:13.462+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.463752+0000) 2020-02-05T22:32:13.466 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:13.463+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.464424+0000) 2020-02-05T22:32:13.466 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:13.463+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.464525+0000) 2020-02-05T22:32:13.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:13.463+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.465332+0000) 2020-02-05T22:32:13.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:13.465+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.466544+0000) 2020-02-05T22:32:13.474 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:13.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:13.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:13.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.847163+0000) 2020-02-05T22:32:13.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:13.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:13.847163+0000) 2020-02-05T22:32:15.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:15.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:15.941141+0000) 2020-02-05T22:32:15.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:15.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:15.949229+0000) 2020-02-05T22:32:17.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:17.087+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:17.088175+0000) 2020-02-05T22:32:17.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:17.087+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:17.088360+0000) 2020-02-05T22:32:17.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:17.087+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:17.088681+0000) 2020-02-05T22:32:18.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:18.477+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.479294+0000) 2020-02-05T22:32:18.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:18.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.480239+0000) 2020-02-05T22:32:18.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:18.480+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.481239+0000) 2020-02-05T22:32:18.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:18.480+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.481757+0000) 2020-02-05T22:32:18.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:18.481+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.482172+0000) 2020-02-05T22:32:18.483 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:18.481+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.482884+0000) 2020-02-05T22:32:18.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:18.481+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.483300+0000) 2020-02-05T22:32:18.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:18.482+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.483700+0000) 2020-02-05T22:32:18.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:18.483+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.484593+0000) 2020-02-05T22:32:18.486 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:18.483+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.485001+0000) 2020-02-05T22:32:18.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:18.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.485425+0000) 2020-02-05T22:32:18.487 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:18.483+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.485343+0000) 2020-02-05T22:32:18.487 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:18.484+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.485570+0000) 2020-02-05T22:32:18.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:18.487+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:18.488314+0000) 2020-02-05T22:32:18.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:18.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:19.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:19.478+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:19.478840+0000) 2020-02-05T22:32:20.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:20.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:20.085496+0000) 2020-02-05T22:32:20.322 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:20.320+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:20.320952+0000) 2020-02-05T22:32:20.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:20.336+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:20.337355+0000) 2020-02-05T22:32:20.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:20.443+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:20.444550+0000) 2020-02-05T22:32:22.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:22.333+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:22.334088+0000) 2020-02-05T22:32:22.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:22.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:22.727677+0000) 2020-02-05T22:32:23.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:23.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.499165+0000) 2020-02-05T22:32:23.501 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:23.499+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.500400+0000) 2020-02-05T22:32:23.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:23.499+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.501341+0000) 2020-02-05T22:32:23.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:23.499+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.501213+0000) 2020-02-05T22:32:23.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:23.500+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.501988+0000) 2020-02-05T22:32:23.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:23.501+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.502337+0000) 2020-02-05T22:32:23.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:23.502+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.503493+0000) 2020-02-05T22:32:23.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:23.502+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.503598+0000) 2020-02-05T22:32:23.505 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:23.500+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.502407+0000) 2020-02-05T22:32:23.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:23.501+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.502952+0000) 2020-02-05T22:32:23.506 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:23.501+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.503082+0000) 2020-02-05T22:32:23.506 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:23.501+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.503169+0000) 2020-02-05T22:32:23.506 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:23.502+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.503961+0000) 2020-02-05T22:32:23.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:23.507+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.508883+0000) 2020-02-05T22:32:23.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:23.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:23.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:23.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.847407+0000) 2020-02-05T22:32:23.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:23.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:23.847409+0000) 2020-02-05T22:32:25.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:25.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:25.941350+0000) 2020-02-05T22:32:25.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:25.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:25.949465+0000) 2020-02-05T22:32:27.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:27.087+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:27.088458+0000) 2020-02-05T22:32:27.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:27.087+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:27.088607+0000) 2020-02-05T22:32:27.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:27.087+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:27.088919+0000) 2020-02-05T22:32:28.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:28.517+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.518611+0000) 2020-02-05T22:32:28.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:28.517+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.519115+0000) 2020-02-05T22:32:28.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:28.518+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.519957+0000) 2020-02-05T22:32:28.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:28.519+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.520389+0000) 2020-02-05T22:32:28.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:28.519+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.520421+0000) 2020-02-05T22:32:28.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:28.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.520783+0000) 2020-02-05T22:32:28.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:28.520+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.521073+0000) 2020-02-05T22:32:28.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:28.520+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.521758+0000) 2020-02-05T22:32:28.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:28.520+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.522032+0000) 2020-02-05T22:32:28.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:28.520+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.522200+0000) 2020-02-05T22:32:28.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:28.521+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.522811+0000) 2020-02-05T22:32:28.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:28.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.523089+0000) 2020-02-05T22:32:28.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:28.522+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.523449+0000) 2020-02-05T22:32:28.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:28.527+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:28.528591+0000) 2020-02-05T22:32:28.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:28.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118208 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:29.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:29.478+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:29.479044+0000) 2020-02-05T22:32:30.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:30.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:30.085729+0000) 2020-02-05T22:32:30.322 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:30.319+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:30.321193+0000) 2020-02-05T22:32:30.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:30.337+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:30.337581+0000) 2020-02-05T22:32:30.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:30.452+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:30.452999+0000) 2020-02-05T22:32:32.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:32.332+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:32.334333+0000) 2020-02-05T22:32:32.648 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:32:32.654 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:32:32.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:32.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:32.727921+0000) 2020-02-05T22:32:33.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:33.537+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.538115+0000) 2020-02-05T22:32:33.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:33.537+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.538708+0000) 2020-02-05T22:32:33.540 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:33.538+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.539149+0000) 2020-02-05T22:32:33.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:33.538+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.539410+0000) 2020-02-05T22:32:33.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:33.538+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.539848+0000) 2020-02-05T22:32:33.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:33.539+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.540641+0000) 2020-02-05T22:32:33.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:33.539+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.541030+0000) 2020-02-05T22:32:33.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:33.540+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.541761+0000) 2020-02-05T22:32:33.543 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:33.540+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.541752+0000) 2020-02-05T22:32:33.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:33.540+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.541888+0000) 2020-02-05T22:32:33.544 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:33.541+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.542495+0000) 2020-02-05T22:32:33.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:33.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.543686+0000) 2020-02-05T22:32:33.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:33.542+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.544296+0000) 2020-02-05T22:32:33.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:33.546+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.547185+0000) 2020-02-05T22:32:33.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:33.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118249 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:33.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:33.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.847649+0000) 2020-02-05T22:32:33.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:33.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:33.847650+0000) 2020-02-05T22:32:35.942 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:35.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:35.941620+0000) 2020-02-05T22:32:35.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:35.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:35.949745+0000) 2020-02-05T22:32:37.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:37.088+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:37.088735+0000) 2020-02-05T22:32:37.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:37.088+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:37.088922+0000) 2020-02-05T22:32:37.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:37.088+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:37.089184+0000) 2020-02-05T22:32:38.560 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:38.557+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.559255+0000) 2020-02-05T22:32:38.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:38.559+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.560392+0000) 2020-02-05T22:32:38.561 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:38.559+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.560744+0000) 2020-02-05T22:32:38.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:38.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.561253+0000) 2020-02-05T22:32:38.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:38.560+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.561429+0000) 2020-02-05T22:32:38.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:38.560+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.561790+0000) 2020-02-05T22:32:38.563 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:38.560+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.562292+0000) 2020-02-05T22:32:38.564 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:38.561+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.562905+0000) 2020-02-05T22:32:38.565 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:38.562+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.563887+0000) 2020-02-05T22:32:38.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:38.563+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.564365+0000) 2020-02-05T22:32:38.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:38.563+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.564638+0000) 2020-02-05T22:32:38.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:38.563+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.565043+0000) 2020-02-05T22:32:38.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:38.564+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.565788+0000) 2020-02-05T22:32:38.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:38.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:38.569140+0000) 2020-02-05T22:32:38.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:38.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:39.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:39.478+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:39.479310+0000) 2020-02-05T22:32:40.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:40.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:40.085951+0000) 2020-02-05T22:32:40.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:40.321+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:40.321504+0000) 2020-02-05T22:32:40.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:40.337+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:40.337803+0000) 2020-02-05T22:32:40.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:40.460+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:40.461197+0000) 2020-02-05T22:32:42.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:42.334+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:42.334573+0000) 2020-02-05T22:32:42.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:42.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:42.728155+0000) 2020-02-05T22:32:43.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:43.578+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.579121+0000) 2020-02-05T22:32:43.581 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:43.579+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.579953+0000) 2020-02-05T22:32:43.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:43.578+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.579842+0000) 2020-02-05T22:32:43.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:43.579+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.581031+0000) 2020-02-05T22:32:43.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:43.581+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.581951+0000) 2020-02-05T22:32:43.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:43.581+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.582841+0000) 2020-02-05T22:32:43.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:43.582+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.583334+0000) 2020-02-05T22:32:43.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:43.581+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.581799+0000) 2020-02-05T22:32:43.585 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:43.582+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.582408+0000) 2020-02-05T22:32:43.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:43.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.584563+0000) 2020-02-05T22:32:43.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:43.583+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.584286+0000) 2020-02-05T22:32:43.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:43.584+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.584852+0000) 2020-02-05T22:32:43.586 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:43.584+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.585338+0000) 2020-02-05T22:32:43.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:43.588+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.589802+0000) 2020-02-05T22:32:43.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:43.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:43.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:43.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.847914+0000) 2020-02-05T22:32:43.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:43.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:43.847914+0000) 2020-02-05T22:32:45.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:45.940+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:45.941894+0000) 2020-02-05T22:32:45.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:45.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:45.949988+0000) 2020-02-05T22:32:47.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:47.088+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:47.088991+0000) 2020-02-05T22:32:47.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:47.088+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:47.089155+0000) 2020-02-05T22:32:47.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:47.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:47.089486+0000) 2020-02-05T22:32:48.602 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:48.599+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.600787+0000) 2020-02-05T22:32:48.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:48.600+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.600725+0000) 2020-02-05T22:32:48.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:48.600+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.601352+0000) 2020-02-05T22:32:48.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:48.601+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.602266+0000) 2020-02-05T22:32:48.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:48.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.602672+0000) 2020-02-05T22:32:48.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:48.602+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.602803+0000) 2020-02-05T22:32:48.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:48.602+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.603070+0000) 2020-02-05T22:32:48.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:48.603+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.603826+0000) 2020-02-05T22:32:48.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:48.604+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.604978+0000) 2020-02-05T22:32:48.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:48.604+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.605674+0000) 2020-02-05T22:32:48.607 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:48.603+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.604347+0000) 2020-02-05T22:32:48.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:48.604+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.605149+0000) 2020-02-05T22:32:48.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:48.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.607850+0000) 2020-02-05T22:32:48.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:48.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:48.608959+0000) 2020-02-05T22:32:48.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:48.614+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:49.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:49.479+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:49.479535+0000) 2020-02-05T22:32:50.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:50.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:50.086187+0000) 2020-02-05T22:32:50.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:50.321+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:50.321754+0000) 2020-02-05T22:32:50.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:50.338+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:50.338063+0000) 2020-02-05T22:32:50.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:50.469+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:50.469290+0000) 2020-02-05T22:32:52.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:52.334+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:52.334816+0000) 2020-02-05T22:32:52.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:52.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:52.728390+0000) 2020-02-05T22:32:53.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:53.619+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.620820+0000) 2020-02-05T22:32:53.622 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:53.620+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.621465+0000) 2020-02-05T22:32:53.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:53.621+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.622308+0000) 2020-02-05T22:32:53.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:53.621+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.622366+0000) 2020-02-05T22:32:53.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:53.622+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.623473+0000) 2020-02-05T22:32:53.625 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:53.623+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.623663+0000) 2020-02-05T22:32:53.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:53.623+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.624509+0000) 2020-02-05T22:32:53.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:53.624+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.625400+0000) 2020-02-05T22:32:53.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:53.625+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.625985+0000) 2020-02-05T22:32:53.627 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:53.625+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.626054+0000) 2020-02-05T22:32:53.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:53.625+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.626619+0000) 2020-02-05T22:32:53.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:53.626+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.626629+0000) 2020-02-05T22:32:53.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:53.629+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.630116+0000) 2020-02-05T22:32:53.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:53.630+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.631619+0000) 2020-02-05T22:32:53.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:53.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:53.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:53.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.848170+0000) 2020-02-05T22:32:53.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:53.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:53.848168+0000) 2020-02-05T22:32:55.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:55.941+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:55.942117+0000) 2020-02-05T22:32:55.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:55.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:55.950181+0000) 2020-02-05T22:32:57.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:57.088+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:57.089249+0000) 2020-02-05T22:32:57.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:57.088+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:57.089405+0000) 2020-02-05T22:32:57.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:57.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:57.089736+0000) 2020-02-05T22:32:58.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:32:58.639+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.640446+0000) 2020-02-05T22:32:58.642 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:32:58.640+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.640997+0000) 2020-02-05T22:32:58.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:32:58.640+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.641682+0000) 2020-02-05T22:32:58.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:32:58.641+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.642283+0000) 2020-02-05T22:32:58.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:32:58.642+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.642766+0000) 2020-02-05T22:32:58.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:32:58.642+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.643161+0000) 2020-02-05T22:32:58.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:32:58.643+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.644529+0000) 2020-02-05T22:32:58.645 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:32:58.643+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.644641+0000) 2020-02-05T22:32:58.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:58.644+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.645105+0000) 2020-02-05T22:32:58.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:32:58.645+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.645639+0000) 2020-02-05T22:32:58.647 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:32:58.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.646335+0000) 2020-02-05T22:32:58.647 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:32:58.646+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.646523+0000) 2020-02-05T22:32:58.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:58.647+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.648810+0000) 2020-02-05T22:32:58.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:58.647+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:58.649054+0000) 2020-02-05T22:32:58.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:32:58.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:32:59.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:32:59.478+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:32:59.479751+0000) 2020-02-05T22:33:00.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:00.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:00.086412+0000) 2020-02-05T22:33:00.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:00.321+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:00.322007+0000) 2020-02-05T22:33:00.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:00.337+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:00.338336+0000) 2020-02-05T22:33:00.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:00.480+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:00.481690+0000) 2020-02-05T22:33:02.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:02.334+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:02.335115+0000) 2020-02-05T22:33:02.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:02.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:02.728624+0000) 2020-02-05T22:33:02.811 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:33:02.817 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:33:03.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:03.658+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.659617+0000) 2020-02-05T22:33:03.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:03.660+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.660911+0000) 2020-02-05T22:33:03.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:03.660+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.661392+0000) 2020-02-05T22:33:03.663 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:03.661+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.662171+0000) 2020-02-05T22:33:03.663 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:03.662+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.662863+0000) 2020-02-05T22:33:03.664 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:03.662+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.663299+0000) 2020-02-05T22:33:03.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:03.662+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.663877+0000) 2020-02-05T22:33:03.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:03.662+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.663968+0000) 2020-02-05T22:33:03.665 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:03.663+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.664490+0000) 2020-02-05T22:33:03.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:03.663+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.664922+0000) 2020-02-05T22:33:03.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:03.664+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.665570+0000) 2020-02-05T22:33:03.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:03.665+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.666066+0000) 2020-02-05T22:33:03.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:03.666+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.666826+0000) 2020-02-05T22:33:03.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:03.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.671027+0000) 2020-02-05T22:33:03.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:03.674+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:03.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:03.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.848427+0000) 2020-02-05T22:33:03.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:03.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:03.848428+0000) 2020-02-05T22:33:05.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:05.941+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:05.942301+0000) 2020-02-05T22:33:05.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:05.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:05.950459+0000) 2020-02-05T22:33:07.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:07.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:07.089499+0000) 2020-02-05T22:33:07.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:07.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:07.089710+0000) 2020-02-05T22:33:07.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:07.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:07.090126+0000) 2020-02-05T22:33:08.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:08.680+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.681267+0000) 2020-02-05T22:33:08.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:08.680+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.681390+0000) 2020-02-05T22:33:08.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:08.681+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.682110+0000) 2020-02-05T22:33:08.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:08.681+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.682239+0000) 2020-02-05T22:33:08.684 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:08.682+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.682827+0000) 2020-02-05T22:33:08.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:08.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.682944+0000) 2020-02-05T22:33:08.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:08.682+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.683215+0000) 2020-02-05T22:33:08.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:08.682+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.683541+0000) 2020-02-05T22:33:08.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:08.683+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.683424+0000) 2020-02-05T22:33:08.687 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:08.683+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.684165+0000) 2020-02-05T22:33:08.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:08.684+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.684682+0000) 2020-02-05T22:33:08.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:08.682+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.683775+0000) 2020-02-05T22:33:08.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:08.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.684244+0000) 2020-02-05T22:33:08.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:08.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:08.689722+0000) 2020-02-05T22:33:08.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:08.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:09.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:09.479+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:09.479983+0000) 2020-02-05T22:33:10.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:10.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:10.086684+0000) 2020-02-05T22:33:10.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:10.322+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:10.322259+0000) 2020-02-05T22:33:10.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:10.337+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:10.338567+0000) 2020-02-05T22:33:10.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:10.488+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:10.489828+0000) 2020-02-05T22:33:12.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:12.335+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:12.335360+0000) 2020-02-05T22:33:12.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:12.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:12.728869+0000) 2020-02-05T22:33:13.705 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:13.701+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.701542+0000) 2020-02-05T22:33:13.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:13.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.703068+0000) 2020-02-05T22:33:13.705 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:13.702+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.703247+0000) 2020-02-05T22:33:13.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:13.699+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.700645+0000) 2020-02-05T22:33:13.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:13.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.702626+0000) 2020-02-05T22:33:13.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:13.701+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.702732+0000) 2020-02-05T22:33:13.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:13.701+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.702985+0000) 2020-02-05T22:33:13.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:13.702+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.703502+0000) 2020-02-05T22:33:13.708 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:13.704+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.704905+0000) 2020-02-05T22:33:13.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:13.704+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.705084+0000) 2020-02-05T22:33:13.708 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:13.705+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.705906+0000) 2020-02-05T22:33:13.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:13.703+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.704144+0000) 2020-02-05T22:33:13.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:13.704+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.705548+0000) 2020-02-05T22:33:13.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:13.708+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.709136+0000) 2020-02-05T22:33:13.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:13.714+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118581 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:13.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:13.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.848686+0000) 2020-02-05T22:33:13.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:13.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:13.848687+0000) 2020-02-05T22:33:15.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:15.941+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:15.942530+0000) 2020-02-05T22:33:15.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:15.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:15.950715+0000) 2020-02-05T22:33:17.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:17.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:17.089792+0000) 2020-02-05T22:33:17.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:17.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:17.089951+0000) 2020-02-05T22:33:17.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:17.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:17.090330+0000) 2020-02-05T22:33:18.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:18.718+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.719069+0000) 2020-02-05T22:33:18.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:18.718+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.719920+0000) 2020-02-05T22:33:18.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:18.719+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.721042+0000) 2020-02-05T22:33:18.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:18.720+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.721473+0000) 2020-02-05T22:33:18.723 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:18.720+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.721775+0000) 2020-02-05T22:33:18.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:18.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.722260+0000) 2020-02-05T22:33:18.723 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:18.721+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.722927+0000) 2020-02-05T22:33:18.724 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:18.722+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.723466+0000) 2020-02-05T22:33:18.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:18.723+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.724190+0000) 2020-02-05T22:33:18.726 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:18.723+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.724812+0000) 2020-02-05T22:33:18.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:18.724+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.725481+0000) 2020-02-05T22:33:18.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:18.724+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.726297+0000) 2020-02-05T22:33:18.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:18.724+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.726394+0000) 2020-02-05T22:33:18.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:18.730+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:18.731118+0000) 2020-02-05T22:33:18.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:18.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:19.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:19.479+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:19.480207+0000) 2020-02-05T22:33:20.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:20.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:20.086922+0000) 2020-02-05T22:33:20.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:20.321+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:20.322546+0000) 2020-02-05T22:33:20.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:20.338+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:20.338798+0000) 2020-02-05T22:33:20.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:20.496+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:20.497192+0000) 2020-02-05T22:33:22.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:22.334+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:22.335591+0000) 2020-02-05T22:33:22.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:22.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:22.729127+0000) 2020-02-05T22:33:23.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:23.740+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.741541+0000) 2020-02-05T22:33:23.745 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:23.742+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.743580+0000) 2020-02-05T22:33:23.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:23.743+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.744593+0000) 2020-02-05T22:33:23.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:23.743+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.744869+0000) 2020-02-05T22:33:23.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:23.744+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.745589+0000) 2020-02-05T22:33:23.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:23.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.745483+0000) 2020-02-05T22:33:23.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:23.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.745654+0000) 2020-02-05T22:33:23.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:23.744+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.745903+0000) 2020-02-05T22:33:23.748 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:23.745+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.746449+0000) 2020-02-05T22:33:23.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:23.745+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.746784+0000) 2020-02-05T22:33:23.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:23.745+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.746617+0000) 2020-02-05T22:33:23.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:23.745+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.747145+0000) 2020-02-05T22:33:23.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:23.746+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.747610+0000) 2020-02-05T22:33:23.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:23.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.752949+0000) 2020-02-05T22:33:23.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:23.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:23.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:23.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.848933+0000) 2020-02-05T22:33:23.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:23.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:23.848933+0000) 2020-02-05T22:33:25.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:25.942+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:25.942761+0000) 2020-02-05T22:33:25.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:25.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:25.950966+0000) 2020-02-05T22:33:27.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:27.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:27.090027+0000) 2020-02-05T22:33:27.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:27.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:27.090230+0000) 2020-02-05T22:33:27.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:27.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:27.090609+0000) 2020-02-05T22:33:28.765 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:28.762+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.763986+0000) 2020-02-05T22:33:28.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:28.764+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.765158+0000) 2020-02-05T22:33:28.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:28.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.765867+0000) 2020-02-05T22:33:28.767 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:28.764+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.765788+0000) 2020-02-05T22:33:28.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:28.765+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.766634+0000) 2020-02-05T22:33:28.768 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:28.765+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.766722+0000) 2020-02-05T22:33:28.768 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:28.765+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.766967+0000) 2020-02-05T22:33:28.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:28.766+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.767346+0000) 2020-02-05T22:33:28.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:28.768+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.769742+0000) 2020-02-05T22:33:28.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:28.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.767830+0000) 2020-02-05T22:33:28.775 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:28.766+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.767542+0000) 2020-02-05T22:33:28.775 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:28.767+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.768910+0000) 2020-02-05T22:33:28.775 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:28.767+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.769193+0000) 2020-02-05T22:33:28.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:28.772+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:28.774091+0000) 2020-02-05T22:33:28.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:28.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:29.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:29.480+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:29.480448+0000) 2020-02-05T22:33:30.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:30.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:30.087144+0000) 2020-02-05T22:33:30.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:30.321+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:30.322785+0000) 2020-02-05T22:33:30.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:30.337+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:30.339018+0000) 2020-02-05T22:33:30.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:30.504+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:30.505128+0000) 2020-02-05T22:33:32.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:32.334+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:32.335856+0000) 2020-02-05T22:33:32.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:32.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:32.729357+0000) 2020-02-05T22:33:32.949 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:33:32.956 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:33:33.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:33.784+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.785612+0000) 2020-02-05T22:33:33.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:33.784+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.785871+0000) 2020-02-05T22:33:33.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:33.785+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.786347+0000) 2020-02-05T22:33:33.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:33.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.786875+0000) 2020-02-05T22:33:33.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:33.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.786889+0000) 2020-02-05T22:33:33.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:33.786+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.787251+0000) 2020-02-05T22:33:33.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:33.785+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.787289+0000) 2020-02-05T22:33:33.790 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:33.786+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.787561+0000) 2020-02-05T22:33:33.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:33.786+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.788023+0000) 2020-02-05T22:33:33.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:33.787+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.788511+0000) 2020-02-05T22:33:33.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:33.786+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.787629+0000) 2020-02-05T22:33:33.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:33.787+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.788571+0000) 2020-02-05T22:33:33.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:33.788+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.789263+0000) 2020-02-05T22:33:33.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:33.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.792427+0000) 2020-02-05T22:33:33.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:33.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118747 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:33.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:33.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.849175+0000) 2020-02-05T22:33:33.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:33.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:33.849176+0000) 2020-02-05T22:33:35.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:35.942+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:35.943014+0000) 2020-02-05T22:33:35.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:35.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:35.951216+0000) 2020-02-05T22:33:37.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:37.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:37.090345+0000) 2020-02-05T22:33:37.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:37.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:37.090513+0000) 2020-02-05T22:33:37.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:37.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:37.090858+0000) 2020-02-05T22:33:38.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:38.803+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.804765+0000) 2020-02-05T22:33:38.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:38.803+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.804617+0000) 2020-02-05T22:33:38.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:38.805+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.806255+0000) 2020-02-05T22:33:38.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:38.804+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.806333+0000) 2020-02-05T22:33:38.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:38.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.807217+0000) 2020-02-05T22:33:38.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:38.806+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.807509+0000) 2020-02-05T22:33:38.809 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:38.806+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.807910+0000) 2020-02-05T22:33:38.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:38.807+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.808148+0000) 2020-02-05T22:33:38.809 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:38.807+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.808496+0000) 2020-02-05T22:33:38.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:38.808+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.809375+0000) 2020-02-05T22:33:38.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:38.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.809286+0000) 2020-02-05T22:33:38.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:38.808+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.809813+0000) 2020-02-05T22:33:38.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:38.809+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.810631+0000) 2020-02-05T22:33:38.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:38.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:38.811617+0000) 2020-02-05T22:33:38.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:38.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:39.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:39.479+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:39.480679+0000) 2020-02-05T22:33:40.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:40.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:40.087381+0000) 2020-02-05T22:33:40.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:40.322+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:40.323079+0000) 2020-02-05T22:33:40.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:40.339+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:40.339275+0000) 2020-02-05T22:33:40.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:40.511+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:40.512124+0000) 2020-02-05T22:33:42.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:42.335+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:42.336116+0000) 2020-02-05T22:33:42.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:42.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:42.729578+0000) 2020-02-05T22:33:43.823 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:43.819+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.821144+0000) 2020-02-05T22:33:43.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:43.823+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.823062+0000) 2020-02-05T22:33:43.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:43.823+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.823962+0000) 2020-02-05T22:33:43.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:43.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.824508+0000) 2020-02-05T22:33:43.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:43.824+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.824900+0000) 2020-02-05T22:33:43.827 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:43.824+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.825880+0000) 2020-02-05T22:33:43.828 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:43.825+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.826870+0000) 2020-02-05T22:33:43.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:43.826+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.827544+0000) 2020-02-05T22:33:43.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:43.826+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.827720+0000) 2020-02-05T22:33:43.829 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:43.827+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.828755+0000) 2020-02-05T22:33:43.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:43.828+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.829039+0000) 2020-02-05T22:33:43.830 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:43.829+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.829291+0000) 2020-02-05T22:33:43.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:43.829+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.829336+0000) 2020-02-05T22:33:43.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:43.829+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.830453+0000) 2020-02-05T22:33:43.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:43.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:43.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:43.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.849319+0000) 2020-02-05T22:33:43.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:43.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:43.849319+0000) 2020-02-05T22:33:45.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:45.943+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:45.943305+0000) 2020-02-05T22:33:45.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:45.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:45.951466+0000) 2020-02-05T22:33:47.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:47.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:47.090603+0000) 2020-02-05T22:33:47.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:47.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:47.090785+0000) 2020-02-05T22:33:47.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:47.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:47.091114+0000) 2020-02-05T22:33:48.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:48.842+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.844023+0000) 2020-02-05T22:33:48.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:48.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.844587+0000) 2020-02-05T22:33:48.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:48.843+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.845382+0000) 2020-02-05T22:33:48.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:48.844+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.845946+0000) 2020-02-05T22:33:48.848 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:48.846+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847155+0000) 2020-02-05T22:33:48.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:48.846+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847285+0000) 2020-02-05T22:33:48.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:48.846+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847717+0000) 2020-02-05T22:33:48.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:48.846+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847604+0000) 2020-02-05T22:33:48.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:48.846+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847522+0000) 2020-02-05T22:33:48.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:48.846+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.847968+0000) 2020-02-05T22:33:48.850 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:48.846+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.848395+0000) 2020-02-05T22:33:48.850 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:48.847+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.848906+0000) 2020-02-05T22:33:48.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:48.850+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.852014+0000) 2020-02-05T22:33:48.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:48.851+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:48.852285+0000) 2020-02-05T22:33:48.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:48.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:49.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:49.479+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:49.480880+0000) 2020-02-05T22:33:50.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:50.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:50.087617+0000) 2020-02-05T22:33:50.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:50.322+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:50.323322+0000) 2020-02-05T22:33:50.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:50.338+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:50.339520+0000) 2020-02-05T22:33:50.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:50.522+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:50.523557+0000) 2020-02-05T22:33:52.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:52.335+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:52.336359+0000) 2020-02-05T22:33:52.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:52.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:52.729831+0000) 2020-02-05T22:33:53.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:53.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.849535+0000) 2020-02-05T22:33:53.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:53.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.849533+0000) 2020-02-05T22:33:53.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:53.861+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.862816+0000) 2020-02-05T22:33:53.864 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:53.862+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.863672+0000) 2020-02-05T22:33:53.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:53.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.864129+0000) 2020-02-05T22:33:53.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:53.862+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.864285+0000) 2020-02-05T22:33:53.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:53.864+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.865611+0000) 2020-02-05T22:33:53.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:53.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.866176+0000) 2020-02-05T22:33:53.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:53.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.866861+0000) 2020-02-05T22:33:53.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:53.865+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.866976+0000) 2020-02-05T22:33:53.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:53.866+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.867617+0000) 2020-02-05T22:33:53.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:53.866+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.867599+0000) 2020-02-05T22:33:53.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:53.867+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.868181+0000) 2020-02-05T22:33:53.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:53.866+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.867906+0000) 2020-02-05T22:33:53.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:53.867+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.868671+0000) 2020-02-05T22:33:53.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:53.876+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:53.877341+0000) 2020-02-05T22:33:53.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:53.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:55.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:55.942+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:55.943578+0000) 2020-02-05T22:33:55.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:55.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:55.951704+0000) 2020-02-05T22:33:57.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:57.089+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:57.090800+0000) 2020-02-05T22:33:57.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:57.089+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:57.091077+0000) 2020-02-05T22:33:57.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:57.089+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:57.091351+0000) 2020-02-05T22:33:58.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:33:58.882+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.883797+0000) 2020-02-05T22:33:58.886 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:33:58.883+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.884452+0000) 2020-02-05T22:33:58.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:33:58.883+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.884940+0000) 2020-02-05T22:33:58.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:33:58.884+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.885333+0000) 2020-02-05T22:33:58.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:58.884+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.885662+0000) 2020-02-05T22:33:58.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:33:58.885+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.886375+0000) 2020-02-05T22:33:58.889 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:33:58.884+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.886207+0000) 2020-02-05T22:33:58.889 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:33:58.885+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.886958+0000) 2020-02-05T22:33:58.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:33:58.885+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.887040+0000) 2020-02-05T22:33:58.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:58.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.887798+0000) 2020-02-05T22:33:58.891 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:33:58.886+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.888374+0000) 2020-02-05T22:33:58.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:33:58.887+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.888647+0000) 2020-02-05T22:33:58.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:33:58.887+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.889040+0000) 2020-02-05T22:33:58.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:58.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:58.895268+0000) 2020-02-05T22:33:58.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:33:58.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118955 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:33:59.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:33:59.480+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:33:59.481169+0000) 2020-02-05T22:34:00.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:00.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:00.087859+0000) 2020-02-05T22:34:00.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:00.322+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:00.323553+0000) 2020-02-05T22:34:00.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:00.338+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:00.339747+0000) 2020-02-05T22:34:00.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:00.532+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:00.533473+0000) 2020-02-05T22:34:02.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:02.336+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:02.336674+0000) 2020-02-05T22:34:02.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:02.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:02.730060+0000) 2020-02-05T22:34:03.146 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:34:03.152 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:34:03.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:03.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.849812+0000) 2020-02-05T22:34:03.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:03.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.849809+0000) 2020-02-05T22:34:03.907 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:03.903+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.904898+0000) 2020-02-05T22:34:03.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:03.906+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.907735+0000) 2020-02-05T22:34:03.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:03.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.908165+0000) 2020-02-05T22:34:03.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:03.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.908763+0000) 2020-02-05T22:34:03.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:03.907+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.909024+0000) 2020-02-05T22:34:03.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:03.908+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.909678+0000) 2020-02-05T22:34:03.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:03.908+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.909804+0000) 2020-02-05T22:34:03.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:03.908+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.910210+0000) 2020-02-05T22:34:03.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:03.909+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.910826+0000) 2020-02-05T22:34:03.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:03.909+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.910965+0000) 2020-02-05T22:34:03.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:03.910+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.911271+0000) 2020-02-05T22:34:03.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:03.910+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.912060+0000) 2020-02-05T22:34:03.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:03.910+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.912350+0000) 2020-02-05T22:34:03.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:03.916+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:03.917170+0000) 2020-02-05T22:34:03.920 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:03.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 118996 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:05.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:05.943+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:05.943835+0000) 2020-02-05T22:34:05.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:05.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:05.951958+0000) 2020-02-05T22:34:07.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:07.090+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:07.091036+0000) 2020-02-05T22:34:07.092 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:07.090+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:07.091331+0000) 2020-02-05T22:34:07.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:07.090+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:07.091589+0000) 2020-02-05T22:34:08.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:08.925+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.925703+0000) 2020-02-05T22:34:08.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:08.925+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.926725+0000) 2020-02-05T22:34:08.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:08.926+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.927205+0000) 2020-02-05T22:34:08.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:08.926+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.927547+0000) 2020-02-05T22:34:08.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:08.927+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.928274+0000) 2020-02-05T22:34:08.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:08.927+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.928644+0000) 2020-02-05T22:34:08.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:08.928+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.928858+0000) 2020-02-05T22:34:08.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:08.928+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.929186+0000) 2020-02-05T22:34:08.931 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:08.929+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.929479+0000) 2020-02-05T22:34:08.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:08.927+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.928968+0000) 2020-02-05T22:34:08.931 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:08.930+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.930631+0000) 2020-02-05T22:34:08.932 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:08.930+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.931334+0000) 2020-02-05T22:34:08.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:08.930+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.931419+0000) 2020-02-05T22:34:08.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:08.936+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:08.937726+0000) 2020-02-05T22:34:08.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:08.938+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:09.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:09.480+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:09.481358+0000) 2020-02-05T22:34:10.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:10.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:10.088122+0000) 2020-02-05T22:34:10.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:10.323+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:10.323791+0000) 2020-02-05T22:34:10.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:10.339+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:10.339973+0000) 2020-02-05T22:34:10.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:10.541+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:10.541878+0000) 2020-02-05T22:34:12.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:12.336+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:12.336931+0000) 2020-02-05T22:34:12.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:12.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:12.730356+0000) 2020-02-05T22:34:13.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:13.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.850139+0000) 2020-02-05T22:34:13.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:13.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.850139+0000) 2020-02-05T22:34:13.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:13.943+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.944257+0000) 2020-02-05T22:34:13.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:13.944+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.945858+0000) 2020-02-05T22:34:13.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:13.945+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.946754+0000) 2020-02-05T22:34:13.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:13.946+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.947888+0000) 2020-02-05T22:34:13.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:13.947+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.948397+0000) 2020-02-05T22:34:13.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:13.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.948372+0000) 2020-02-05T22:34:13.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:13.948+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.948673+0000) 2020-02-05T22:34:13.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:13.948+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.949260+0000) 2020-02-05T22:34:13.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:13.948+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.949697+0000) 2020-02-05T22:34:13.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:13.949+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.949771+0000) 2020-02-05T22:34:13.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:13.949+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.950038+0000) 2020-02-05T22:34:13.952 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:13.950+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.950452+0000) 2020-02-05T22:34:13.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:13.951+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.952137+0000) 2020-02-05T22:34:13.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:13.955+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:13.956608+0000) 2020-02-05T22:34:13.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:13.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:15.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:15.942+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:15.944058+0000) 2020-02-05T22:34:15.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:15.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:15.952210+0000) 2020-02-05T22:34:17.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:17.090+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:17.091279+0000) 2020-02-05T22:34:17.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:17.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:17.091608+0000) 2020-02-05T22:34:17.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:17.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:17.091818+0000) 2020-02-05T22:34:18.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:18.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.966268+0000) 2020-02-05T22:34:18.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:18.965+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.966048+0000) 2020-02-05T22:34:18.968 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:18.966+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.967206+0000) 2020-02-05T22:34:18.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:18.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.967813+0000) 2020-02-05T22:34:18.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:18.967+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.968212+0000) 2020-02-05T22:34:18.970 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:18.967+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.969058+0000) 2020-02-05T22:34:18.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:18.968+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.969237+0000) 2020-02-05T22:34:18.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:18.969+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.970123+0000) 2020-02-05T22:34:18.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:18.969+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.970407+0000) 2020-02-05T22:34:18.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:18.969+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.969673+0000) 2020-02-05T22:34:18.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:18.970+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.970779+0000) 2020-02-05T22:34:18.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:18.970+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.971272+0000) 2020-02-05T22:34:18.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:18.970+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.972031+0000) 2020-02-05T22:34:18.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:18.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:18.975208+0000) 2020-02-05T22:34:18.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:18.980+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:19.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:19.480+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:19.481644+0000) 2020-02-05T22:34:20.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:20.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:20.088356+0000) 2020-02-05T22:34:20.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:20.323+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:20.324026+0000) 2020-02-05T22:34:20.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:20.339+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:20.340189+0000) 2020-02-05T22:34:20.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:20.553+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:20.553653+0000) 2020-02-05T22:34:22.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:22.336+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:22.337174+0000) 2020-02-05T22:34:22.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:22.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:22.730618+0000) 2020-02-05T22:34:23.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:23.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.850412+0000) 2020-02-05T22:34:23.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:23.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.850412+0000) 2020-02-05T22:34:23.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:23.986+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.987200+0000) 2020-02-05T22:34:23.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:23.987+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.988446+0000) 2020-02-05T22:34:23.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:23.988+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.989021+0000) 2020-02-05T22:34:23.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:23.988+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.989243+0000) 2020-02-05T22:34:23.991 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:23.989+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.989784+0000) 2020-02-05T22:34:23.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:23.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.990539+0000) 2020-02-05T22:34:23.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:23.990+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.990722+0000) 2020-02-05T22:34:23.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:23.990+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.991464+0000) 2020-02-05T22:34:23.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:23.990+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.991956+0000) 2020-02-05T22:34:23.993 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:23.991+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.992444+0000) 2020-02-05T22:34:23.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:23.991+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.992600+0000) 2020-02-05T22:34:23.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:23.992+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.992527+0000) 2020-02-05T22:34:23.994 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:23.992+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.992692+0000) 2020-02-05T22:34:23.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:23.997+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:23.998932+0000) 2020-02-05T22:34:24.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:24.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:25.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:25.943+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:25.944347+0000) 2020-02-05T22:34:25.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:25.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:25.952503+0000) 2020-02-05T22:34:27.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:27.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:27.091510+0000) 2020-02-05T22:34:27.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:27.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:27.091870+0000) 2020-02-05T22:34:27.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:27.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:27.092067+0000) 2020-02-05T22:34:29.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:29.006+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.007988+0000) 2020-02-05T22:34:29.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:29.007+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.008837+0000) 2020-02-05T22:34:29.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:29.009+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.009543+0000) 2020-02-05T22:34:29.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:29.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.010051+0000) 2020-02-05T22:34:29.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:29.009+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.010535+0000) 2020-02-05T22:34:29.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:29.009+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.010927+0000) 2020-02-05T22:34:29.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:29.010+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.011463+0000) 2020-02-05T22:34:29.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:29.012+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.013142+0000) 2020-02-05T22:34:29.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:29.012+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.013303+0000) 2020-02-05T22:34:29.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:29.013+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.013706+0000) 2020-02-05T22:34:29.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:29.013+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.013816+0000) 2020-02-05T22:34:29.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:29.013+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.014378+0000) 2020-02-05T22:34:29.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:29.011+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.012849+0000) 2020-02-05T22:34:29.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:29.016+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.017257+0000) 2020-02-05T22:34:29.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:29.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:29.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:29.481+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:29.481852+0000) 2020-02-05T22:34:30.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:30.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:30.088603+0000) 2020-02-05T22:34:30.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:30.324+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:30.324279+0000) 2020-02-05T22:34:30.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:30.340+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:30.340416+0000) 2020-02-05T22:34:30.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:30.561+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:30.561988+0000) 2020-02-05T22:34:32.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:32.337+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:32.337480+0000) 2020-02-05T22:34:32.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:32.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:32.730850+0000) 2020-02-05T22:34:33.342 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:34:33.349 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:34:33.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:33.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:33.850685+0000) 2020-02-05T22:34:33.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:33.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:33.850686+0000) 2020-02-05T22:34:34.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:34.026+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.027629+0000) 2020-02-05T22:34:34.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:34.028+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.029823+0000) 2020-02-05T22:34:34.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:34.029+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.030350+0000) 2020-02-05T22:34:34.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:34.029+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.030767+0000) 2020-02-05T22:34:34.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:34.030+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.031486+0000) 2020-02-05T22:34:34.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:34.031+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.032474+0000) 2020-02-05T22:34:34.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:34.030+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.030989+0000) 2020-02-05T22:34:34.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:34.032+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.032502+0000) 2020-02-05T22:34:34.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:34.032+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.033168+0000) 2020-02-05T22:34:34.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:34.033+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.034055+0000) 2020-02-05T22:34:34.035 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:34.034+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.034618+0000) 2020-02-05T22:34:34.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:34.034+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.035080+0000) 2020-02-05T22:34:34.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:34.034+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.035476+0000) 2020-02-05T22:34:34.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:34.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:34.040794+0000) 2020-02-05T22:34:34.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:34.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119244 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:35.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:35.944+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:35.944580+0000) 2020-02-05T22:34:35.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:35.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:35.952772+0000) 2020-02-05T22:34:37.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:37.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:37.091813+0000) 2020-02-05T22:34:37.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:37.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:37.092123+0000) 2020-02-05T22:34:37.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:37.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:37.092345+0000) 2020-02-05T22:34:39.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:39.049+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.049316+0000) 2020-02-05T22:34:39.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:39.049+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.049974+0000) 2020-02-05T22:34:39.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:39.049+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.049598+0000) 2020-02-05T22:34:39.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:39.049+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.050092+0000) 2020-02-05T22:34:39.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:39.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.050795+0000) 2020-02-05T22:34:39.053 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:39.050+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.051158+0000) 2020-02-05T22:34:39.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:39.051+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.051483+0000) 2020-02-05T22:34:39.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:39.051+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.051699+0000) 2020-02-05T22:34:39.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:39.052+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.052549+0000) 2020-02-05T22:34:39.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:39.052+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.053143+0000) 2020-02-05T22:34:39.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:39.052+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.052315+0000) 2020-02-05T22:34:39.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:39.053+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.053710+0000) 2020-02-05T22:34:39.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:39.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.053663+0000) 2020-02-05T22:34:39.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:39.058+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.058109+0000) 2020-02-05T22:34:39.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:39.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:39.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:39.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:39.482108+0000) 2020-02-05T22:34:40.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:40.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:40.088846+0000) 2020-02-05T22:34:40.325 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:40.324+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:40.324555+0000) 2020-02-05T22:34:40.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:40.340+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:40.340637+0000) 2020-02-05T22:34:40.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:40.573+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:40.573463+0000) 2020-02-05T22:34:42.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:42.337+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:42.337746+0000) 2020-02-05T22:34:42.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:42.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:42.731073+0000) 2020-02-05T22:34:43.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:43.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:43.850981+0000) 2020-02-05T22:34:43.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:43.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:43.850980+0000) 2020-02-05T22:34:44.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:44.071+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.071533+0000) 2020-02-05T22:34:44.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:44.071+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.071872+0000) 2020-02-05T22:34:44.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:44.069+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.070847+0000) 2020-02-05T22:34:44.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:44.070+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.071321+0000) 2020-02-05T22:34:44.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:44.071+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.072665+0000) 2020-02-05T22:34:44.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:44.072+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.073435+0000) 2020-02-05T22:34:44.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:44.073+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.074267+0000) 2020-02-05T22:34:44.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:44.072+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.073910+0000) 2020-02-05T22:34:44.079 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:44.072+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.073846+0000) 2020-02-05T22:34:44.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:44.073+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.074334+0000) 2020-02-05T22:34:44.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:44.074+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.074591+0000) 2020-02-05T22:34:44.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:44.074+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.074783+0000) 2020-02-05T22:34:44.081 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:44.074+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.075152+0000) 2020-02-05T22:34:44.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:44.078+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:44.079287+0000) 2020-02-05T22:34:44.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:44.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:45.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:45.943+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:45.944813+0000) 2020-02-05T22:34:45.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:45.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:45.953030+0000) 2020-02-05T22:34:47.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:47.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:47.092068+0000) 2020-02-05T22:34:47.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:47.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:47.092347+0000) 2020-02-05T22:34:47.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:47.092+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:47.092616+0000) 2020-02-05T22:34:49.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:49.089+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.090457+0000) 2020-02-05T22:34:49.092 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:49.090+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.091134+0000) 2020-02-05T22:34:49.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:49.090+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.091479+0000) 2020-02-05T22:34:49.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:49.090+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.091954+0000) 2020-02-05T22:34:49.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:49.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.092406+0000) 2020-02-05T22:34:49.094 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:49.091+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.092638+0000) 2020-02-05T22:34:49.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:49.092+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.093772+0000) 2020-02-05T22:34:49.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:49.092+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.093793+0000) 2020-02-05T22:34:49.095 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:49.092+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.094077+0000) 2020-02-05T22:34:49.096 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:49.093+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.094575+0000) 2020-02-05T22:34:49.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:49.094+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.095234+0000) 2020-02-05T22:34:49.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:49.093+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.094874+0000) 2020-02-05T22:34:49.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:49.094+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.095698+0000) 2020-02-05T22:34:49.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:49.097+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.098476+0000) 2020-02-05T22:34:49.104 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:49.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:49.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:49.481+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:49.482326+0000) 2020-02-05T22:34:50.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:50.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:50.089128+0000) 2020-02-05T22:34:50.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:50.323+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:50.324809+0000) 2020-02-05T22:34:50.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:50.339+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:50.340897+0000) 2020-02-05T22:34:50.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:50.582+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:50.583140+0000) 2020-02-05T22:34:52.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:52.336+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:52.338009+0000) 2020-02-05T22:34:52.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:52.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:52.731333+0000) 2020-02-05T22:34:53.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:53.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:53.851285+0000) 2020-02-05T22:34:53.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:53.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:53.851285+0000) 2020-02-05T22:34:54.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:54.108+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.109945+0000) 2020-02-05T22:34:54.113 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:54.109+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.110572+0000) 2020-02-05T22:34:54.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:54.110+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.111312+0000) 2020-02-05T22:34:54.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:54.110+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.111452+0000) 2020-02-05T22:34:54.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:54.110+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.111978+0000) 2020-02-05T22:34:54.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:54.111+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.112624+0000) 2020-02-05T22:34:54.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:54.111+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.113371+0000) 2020-02-05T22:34:54.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:54.113+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.114252+0000) 2020-02-05T22:34:54.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:54.113+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.114616+0000) 2020-02-05T22:34:54.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:54.113+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.115017+0000) 2020-02-05T22:34:54.117 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:54.114+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.115703+0000) 2020-02-05T22:34:54.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:54.114+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.115722+0000) 2020-02-05T22:34:54.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:54.114+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.116000+0000) 2020-02-05T22:34:54.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:54.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:54.120152+0000) 2020-02-05T22:34:54.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:54.122+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119408 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:55.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:55.943+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:55.945058+0000) 2020-02-05T22:34:55.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:55.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:55.953272+0000) 2020-02-05T22:34:57.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:57.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:57.092321+0000) 2020-02-05T22:34:57.094 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:57.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:57.092635+0000) 2020-02-05T22:34:57.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:57.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:57.092860+0000) 2020-02-05T22:34:59.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:59.128+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.129264+0000) 2020-02-05T22:34:59.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:34:59.129+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.131261+0000) 2020-02-05T22:34:59.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:34:59.130+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.131726+0000) 2020-02-05T22:34:59.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:34:59.130+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.132257+0000) 2020-02-05T22:34:59.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:34:59.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.132533+0000) 2020-02-05T22:34:59.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:34:59.131+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.132798+0000) 2020-02-05T22:34:59.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:34:59.132+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.133606+0000) 2020-02-05T22:34:59.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:34:59.132+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.133483+0000) 2020-02-05T22:34:59.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:34:59.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.133687+0000) 2020-02-05T22:34:59.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:34:59.132+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.134070+0000) 2020-02-05T22:34:59.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:34:59.133+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.134108+0000) 2020-02-05T22:34:59.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:59.133+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.134185+0000) 2020-02-05T22:34:59.136 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:34:59.133+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.134673+0000) 2020-02-05T22:34:59.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:59.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.140434+0000) 2020-02-05T22:34:59.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:34:59.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:34:59.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:34:59.481+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:34:59.482567+0000) 2020-02-05T22:35:00.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:00.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:00.089382+0000) 2020-02-05T22:35:00.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:00.324+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:00.325051+0000) 2020-02-05T22:35:00.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:00.340+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:00.341167+0000) 2020-02-05T22:35:00.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:00.590+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:00.591400+0000) 2020-02-05T22:35:02.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:02.337+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:02.338364+0000) 2020-02-05T22:35:02.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:02.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:02.731572+0000) 2020-02-05T22:35:03.522 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:35:03.528 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:35:03.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:03.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:03.851560+0000) 2020-02-05T22:35:03.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:03.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:03.851560+0000) 2020-02-05T22:35:04.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:04.148+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.149247+0000) 2020-02-05T22:35:04.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:04.148+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.150100+0000) 2020-02-05T22:35:04.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:04.150+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.151249+0000) 2020-02-05T22:35:04.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:04.150+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.151608+0000) 2020-02-05T22:35:04.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:04.150+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.152249+0000) 2020-02-05T22:35:04.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:04.151+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.152667+0000) 2020-02-05T22:35:04.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:04.151+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.152907+0000) 2020-02-05T22:35:04.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:04.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.153598+0000) 2020-02-05T22:35:04.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:04.153+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.154142+0000) 2020-02-05T22:35:04.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:04.153+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.154515+0000) 2020-02-05T22:35:04.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:04.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.155108+0000) 2020-02-05T22:35:04.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:04.154+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.155542+0000) 2020-02-05T22:35:04.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:04.154+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.155883+0000) 2020-02-05T22:35:04.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:04.160+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:04.161454+0000) 2020-02-05T22:35:04.166 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:04.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:05.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:05.944+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:05.945323+0000) 2020-02-05T22:35:05.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:05.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:05.953526+0000) 2020-02-05T22:35:07.094 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:07.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:07.092661+0000) 2020-02-05T22:35:07.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:07.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:07.092873+0000) 2020-02-05T22:35:07.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:07.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:07.093093+0000) 2020-02-05T22:35:09.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:09.169+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.170539+0000) 2020-02-05T22:35:09.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:09.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.171221+0000) 2020-02-05T22:35:09.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:09.170+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.171655+0000) 2020-02-05T22:35:09.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:09.170+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.172039+0000) 2020-02-05T22:35:09.173 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:09.171+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.172736+0000) 2020-02-05T22:35:09.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:09.172+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.173151+0000) 2020-02-05T22:35:09.174 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:09.171+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.173394+0000) 2020-02-05T22:35:09.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:09.173+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.174423+0000) 2020-02-05T22:35:09.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:09.172+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.174256+0000) 2020-02-05T22:35:09.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:09.173+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.174459+0000) 2020-02-05T22:35:09.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:09.173+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.174502+0000) 2020-02-05T22:35:09.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:09.173+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.174987+0000) 2020-02-05T22:35:09.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:09.174+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.175547+0000) 2020-02-05T22:35:09.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:09.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.178540+0000) 2020-02-05T22:35:09.186 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:09.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:09.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:09.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:09.482776+0000) 2020-02-05T22:35:10.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:10.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:10.089625+0000) 2020-02-05T22:35:10.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:10.324+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:10.325289+0000) 2020-02-05T22:35:10.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:10.340+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:10.341404+0000) 2020-02-05T22:35:10.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:10.599+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:10.599786+0000) 2020-02-05T22:35:12.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:12.337+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:12.338671+0000) 2020-02-05T22:35:12.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:12.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:12.731819+0000) 2020-02-05T22:35:13.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:13.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:13.851833+0000) 2020-02-05T22:35:13.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:13.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:13.851831+0000) 2020-02-05T22:35:14.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:14.189+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.190779+0000) 2020-02-05T22:35:14.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:14.190+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.191629+0000) 2020-02-05T22:35:14.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:14.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.193224+0000) 2020-02-05T22:35:14.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:14.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.193711+0000) 2020-02-05T22:35:14.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:14.192+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.194297+0000) 2020-02-05T22:35:14.196 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:14.193+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.194805+0000) 2020-02-05T22:35:14.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:14.193+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.194558+0000) 2020-02-05T22:35:14.197 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:14.193+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.194999+0000) 2020-02-05T22:35:14.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:14.193+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.195267+0000) 2020-02-05T22:35:14.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:14.195+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.196364+0000) 2020-02-05T22:35:14.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:14.195+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.197033+0000) 2020-02-05T22:35:14.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:14.196+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.197565+0000) 2020-02-05T22:35:14.199 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:14.196+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.197285+0000) 2020-02-05T22:35:14.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:14.201+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:14.202434+0000) 2020-02-05T22:35:14.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:14.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:15.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:15.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:15.945579+0000) 2020-02-05T22:35:15.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:15.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:15.953812+0000) 2020-02-05T22:35:17.094 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:17.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:17.092888+0000) 2020-02-05T22:35:17.094 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:17.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:17.093116+0000) 2020-02-05T22:35:17.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:17.091+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:17.093345+0000) 2020-02-05T22:35:19.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:19.209+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.210660+0000) 2020-02-05T22:35:19.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:19.209+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.210886+0000) 2020-02-05T22:35:19.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:19.210+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.212008+0000) 2020-02-05T22:35:19.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:19.212+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.213131+0000) 2020-02-05T22:35:19.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:19.212+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.213377+0000) 2020-02-05T22:35:19.214 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:19.212+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.213567+0000) 2020-02-05T22:35:19.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:19.212+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.213951+0000) 2020-02-05T22:35:19.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:19.213+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.214401+0000) 2020-02-05T22:35:19.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:19.213+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.214709+0000) 2020-02-05T22:35:19.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:19.214+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.215693+0000) 2020-02-05T22:35:19.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:19.215+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.216614+0000) 2020-02-05T22:35:19.219 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:19.215+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.216806+0000) 2020-02-05T22:35:19.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:19.215+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.217143+0000) 2020-02-05T22:35:19.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:19.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.222229+0000) 2020-02-05T22:35:19.229 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:19.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:19.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:19.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:19.482992+0000) 2020-02-05T22:35:20.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:20.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:20.089847+0000) 2020-02-05T22:35:20.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:20.324+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:20.325530+0000) 2020-02-05T22:35:20.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:20.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:20.341636+0000) 2020-02-05T22:35:20.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:20.607+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:20.608138+0000) 2020-02-05T22:35:22.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:22.338+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:22.338938+0000) 2020-02-05T22:35:22.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:22.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:22.732055+0000) 2020-02-05T22:35:23.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:23.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:23.852136+0000) 2020-02-05T22:35:23.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:23.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:23.852136+0000) 2020-02-05T22:35:24.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:24.231+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.232787+0000) 2020-02-05T22:35:24.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:24.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.234371+0000) 2020-02-05T22:35:24.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:24.233+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.235041+0000) 2020-02-05T22:35:24.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:24.234+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.235672+0000) 2020-02-05T22:35:24.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:24.234+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.235763+0000) 2020-02-05T22:35:24.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:24.235+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.236632+0000) 2020-02-05T22:35:24.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:24.234+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.236204+0000) 2020-02-05T22:35:24.238 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:24.236+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.237194+0000) 2020-02-05T22:35:24.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:24.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.238694+0000) 2020-02-05T22:35:24.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:24.237+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.238799+0000) 2020-02-05T22:35:24.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:24.236+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.238133+0000) 2020-02-05T22:35:24.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:24.238+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.239765+0000) 2020-02-05T22:35:24.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:24.238+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.239873+0000) 2020-02-05T22:35:24.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:24.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:24.243156+0000) 2020-02-05T22:35:24.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:24.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:25.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:25.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:25.945818+0000) 2020-02-05T22:35:25.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:25.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:25.954070+0000) 2020-02-05T22:35:27.094 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:27.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:27.093131+0000) 2020-02-05T22:35:27.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:27.091+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:27.093354+0000) 2020-02-05T22:35:27.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:27.092+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:27.093668+0000) 2020-02-05T22:35:29.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:29.253+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.254457+0000) 2020-02-05T22:35:29.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:29.254+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.255200+0000) 2020-02-05T22:35:29.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:29.254+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.255610+0000) 2020-02-05T22:35:29.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:29.255+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.256666+0000) 2020-02-05T22:35:29.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:29.255+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.257110+0000) 2020-02-05T22:35:29.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:29.256+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.257515+0000) 2020-02-05T22:35:29.261 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:29.257+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.258471+0000) 2020-02-05T22:35:29.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:29.257+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.258892+0000) 2020-02-05T22:35:29.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:29.259+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.260264+0000) 2020-02-05T22:35:29.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:29.256+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.258017+0000) 2020-02-05T22:35:29.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:29.257+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.259301+0000) 2020-02-05T22:35:29.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:29.258+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.260060+0000) 2020-02-05T22:35:29.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:29.258+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.259778+0000) 2020-02-05T22:35:29.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:29.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.265542+0000) 2020-02-05T22:35:29.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:29.267+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:29.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:29.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:29.483202+0000) 2020-02-05T22:35:30.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:30.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:30.090119+0000) 2020-02-05T22:35:30.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:30.325+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:30.325784+0000) 2020-02-05T22:35:30.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:30.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:30.341867+0000) 2020-02-05T22:35:30.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:30.616+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:30.616461+0000) 2020-02-05T22:35:32.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:32.338+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:32.339239+0000) 2020-02-05T22:35:32.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:32.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:32.732339+0000) 2020-02-05T22:35:33.702 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:35:33.708 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:35:33.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:33.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:33.852367+0000) 2020-02-05T22:35:33.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:33.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:33.852368+0000) 2020-02-05T22:35:34.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:34.273+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.274615+0000) 2020-02-05T22:35:34.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:34.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.275480+0000) 2020-02-05T22:35:34.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:34.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.276029+0000) 2020-02-05T22:35:34.277 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:34.276+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.276511+0000) 2020-02-05T22:35:34.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:34.275+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.277229+0000) 2020-02-05T22:35:34.279 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:34.277+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.278057+0000) 2020-02-05T22:35:34.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:34.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.278326+0000) 2020-02-05T22:35:34.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:34.279+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.279297+0000) 2020-02-05T22:35:34.280 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:34.279+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.279345+0000) 2020-02-05T22:35:34.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:34.277+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.279359+0000) 2020-02-05T22:35:34.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:34.278+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.280083+0000) 2020-02-05T22:35:34.281 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:34.278+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.280321+0000) 2020-02-05T22:35:34.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:34.279+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.281163+0000) 2020-02-05T22:35:34.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:34.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:34.284202+0000) 2020-02-05T22:35:34.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:34.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:35.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:35.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:35.946045+0000) 2020-02-05T22:35:35.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:35.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:35.954328+0000) 2020-02-05T22:35:37.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:37.091+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:37.093375+0000) 2020-02-05T22:35:37.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:37.092+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:37.093589+0000) 2020-02-05T22:35:37.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:37.092+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:37.093885+0000) 2020-02-05T22:35:39.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:39.294+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.296200+0000) 2020-02-05T22:35:39.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:39.295+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.296636+0000) 2020-02-05T22:35:39.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:39.296+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.297401+0000) 2020-02-05T22:35:39.298 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:39.296+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.297864+0000) 2020-02-05T22:35:39.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:39.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.298597+0000) 2020-02-05T22:35:39.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:39.297+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.299145+0000) 2020-02-05T22:35:39.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:39.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.299567+0000) 2020-02-05T22:35:39.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:39.298+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.299696+0000) 2020-02-05T22:35:39.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:39.299+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.300218+0000) 2020-02-05T22:35:39.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:39.299+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.300516+0000) 2020-02-05T22:35:39.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:39.299+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.301040+0000) 2020-02-05T22:35:39.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:39.299+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.300806+0000) 2020-02-05T22:35:39.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:39.299+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.300920+0000) 2020-02-05T22:35:39.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:39.306+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.307282+0000) 2020-02-05T22:35:39.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:39.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:39.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:39.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:39.483457+0000) 2020-02-05T22:35:40.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:40.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:40.090345+0000) 2020-02-05T22:35:40.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:40.325+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:40.326039+0000) 2020-02-05T22:35:40.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:40.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:40.342142+0000) 2020-02-05T22:35:40.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:40.623+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:40.624238+0000) 2020-02-05T22:35:42.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:42.339+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:42.339499+0000) 2020-02-05T22:35:42.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:42.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:42.732578+0000) 2020-02-05T22:35:43.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:43.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:43.852677+0000) 2020-02-05T22:35:43.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:43.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:43.852678+0000) 2020-02-05T22:35:44.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:44.314+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.315096+0000) 2020-02-05T22:35:44.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:44.315+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.316350+0000) 2020-02-05T22:35:44.317 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:44.315+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.316742+0000) 2020-02-05T22:35:44.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:44.316+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.317483+0000) 2020-02-05T22:35:44.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:44.317+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.317590+0000) 2020-02-05T22:35:44.319 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:44.316+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.317989+0000) 2020-02-05T22:35:44.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:44.317+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.318233+0000) 2020-02-05T22:35:44.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:44.318+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.318939+0000) 2020-02-05T22:35:44.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:44.318+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.319177+0000) 2020-02-05T22:35:44.320 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:44.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.319815+0000) 2020-02-05T22:35:44.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:44.319+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.320509+0000) 2020-02-05T22:35:44.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:44.320+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.320609+0000) 2020-02-05T22:35:44.322 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:44.320+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.321278+0000) 2020-02-05T22:35:44.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:44.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:44.324237+0000) 2020-02-05T22:35:44.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:44.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:45.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:45.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:45.946315+0000) 2020-02-05T22:35:45.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:45.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:45.954556+0000) 2020-02-05T22:35:47.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:47.093+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:47.093596+0000) 2020-02-05T22:35:47.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:47.093+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:47.093845+0000) 2020-02-05T22:35:47.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:47.093+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:47.094128+0000) 2020-02-05T22:35:49.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:49.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.335022+0000) 2020-02-05T22:35:49.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:49.335+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.336714+0000) 2020-02-05T22:35:49.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:49.336+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.337452+0000) 2020-02-05T22:35:49.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:49.337+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.337791+0000) 2020-02-05T22:35:49.339 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:49.337+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.338278+0000) 2020-02-05T22:35:49.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:49.337+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.337679+0000) 2020-02-05T22:35:49.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:49.338+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.338740+0000) 2020-02-05T22:35:49.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:49.338+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.338955+0000) 2020-02-05T22:35:49.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:49.338+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.339243+0000) 2020-02-05T22:35:49.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:49.340+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.340953+0000) 2020-02-05T22:35:49.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:49.340+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.341176+0000) 2020-02-05T22:35:49.342 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:49.340+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.341341+0000) 2020-02-05T22:35:49.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:49.341+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.341501+0000) 2020-02-05T22:35:49.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:49.344+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.345789+0000) 2020-02-05T22:35:49.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:49.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:49.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:49.482+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:49.483678+0000) 2020-02-05T22:35:50.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:50.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:50.090571+0000) 2020-02-05T22:35:50.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:50.325+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:50.326298+0000) 2020-02-05T22:35:50.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:50.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:50.342373+0000) 2020-02-05T22:35:50.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:50.635+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:50.636274+0000) 2020-02-05T22:35:52.341 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:52.339+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:52.339769+0000) 2020-02-05T22:35:52.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:52.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:52.732813+0000) 2020-02-05T22:35:53.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:53.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:53.852969+0000) 2020-02-05T22:35:53.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:53.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:53.852968+0000) 2020-02-05T22:35:54.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:54.356+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.357237+0000) 2020-02-05T22:35:54.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:54.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.357717+0000) 2020-02-05T22:35:54.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:54.357+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.358514+0000) 2020-02-05T22:35:54.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:54.358+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.359372+0000) 2020-02-05T22:35:54.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:54.358+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.359896+0000) 2020-02-05T22:35:54.361 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:54.359+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.360344+0000) 2020-02-05T22:35:54.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:54.359+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.359768+0000) 2020-02-05T22:35:54.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:54.360+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.361120+0000) 2020-02-05T22:35:54.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:54.361+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.362074+0000) 2020-02-05T22:35:54.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:54.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.362606+0000) 2020-02-05T22:35:54.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:54.362+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.362578+0000) 2020-02-05T22:35:54.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:54.362+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.363510+0000) 2020-02-05T22:35:54.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:54.363+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.363674+0000) 2020-02-05T22:35:54.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:54.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:54.366395+0000) 2020-02-05T22:35:54.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:54.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:55.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:55.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:55.946531+0000) 2020-02-05T22:35:55.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:55.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:55.954820+0000) 2020-02-05T22:35:57.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:57.093+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:57.093879+0000) 2020-02-05T22:35:57.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:57.093+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:57.094093+0000) 2020-02-05T22:35:57.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:57.093+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:57.094345+0000) 2020-02-05T22:35:59.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:35:59.376+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.377340+0000) 2020-02-05T22:35:59.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:35:59.378+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.378824+0000) 2020-02-05T22:35:59.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:35:59.378+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.379349+0000) 2020-02-05T22:35:59.380 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:35:59.378+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.379698+0000) 2020-02-05T22:35:59.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:59.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.380419+0000) 2020-02-05T22:35:59.381 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:35:59.380+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.380417+0000) 2020-02-05T22:35:59.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:35:59.380+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.380697+0000) 2020-02-05T22:35:59.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:35:59.381+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.382057+0000) 2020-02-05T22:35:59.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:35:59.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.382163+0000) 2020-02-05T22:35:59.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:59.381+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.382402+0000) 2020-02-05T22:35:59.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:35:59.381+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.382682+0000) 2020-02-05T22:35:59.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:35:59.382+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.382611+0000) 2020-02-05T22:35:59.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:35:59.382+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.383251+0000) 2020-02-05T22:35:59.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:59.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.384720+0000) 2020-02-05T22:35:59.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:35:59.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:35:59.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:35:59.483+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:35:59.483894+0000) 2020-02-05T22:36:00.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:00.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:00.090822+0000) 2020-02-05T22:36:00.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:00.326+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:00.326524+0000) 2020-02-05T22:36:00.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:00.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:00.342602+0000) 2020-02-05T22:36:00.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:00.643+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:00.644618+0000) 2020-02-05T22:36:02.341 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:02.339+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:02.340039+0000) 2020-02-05T22:36:02.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:02.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:02.733104+0000) 2020-02-05T22:36:03.844 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:36:03.850 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:36:03.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:03.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:03.853237+0000) 2020-02-05T22:36:03.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:03.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:03.853235+0000) 2020-02-05T22:36:04.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:04.394+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.395832+0000) 2020-02-05T22:36:04.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:04.395+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.396510+0000) 2020-02-05T22:36:04.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:04.395+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.396935+0000) 2020-02-05T22:36:04.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:04.396+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.397615+0000) 2020-02-05T22:36:04.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:04.397+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.398428+0000) 2020-02-05T22:36:04.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:04.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.398907+0000) 2020-02-05T22:36:04.400 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:04.398+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.399078+0000) 2020-02-05T22:36:04.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:04.399+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.399715+0000) 2020-02-05T22:36:04.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:04.400+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.401022+0000) 2020-02-05T22:36:04.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:04.400+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.401046+0000) 2020-02-05T22:36:04.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:04.401+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.401421+0000) 2020-02-05T22:36:04.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:04.400+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.401384+0000) 2020-02-05T22:36:04.403 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:04.400+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.401382+0000) 2020-02-05T22:36:04.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:04.406+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:04.407538+0000) 2020-02-05T22:36:04.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:04.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 119985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:05.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:05.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:05.946759+0000) 2020-02-05T22:36:05.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:05.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:05.955079+0000) 2020-02-05T22:36:07.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:07.093+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:07.094122+0000) 2020-02-05T22:36:07.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:07.093+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:07.094360+0000) 2020-02-05T22:36:07.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:07.094+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:07.094628+0000) 2020-02-05T22:36:09.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:09.415+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.416767+0000) 2020-02-05T22:36:09.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:09.416+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.417584+0000) 2020-02-05T22:36:09.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:09.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.417775+0000) 2020-02-05T22:36:09.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:09.417+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.418285+0000) 2020-02-05T22:36:09.420 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:09.417+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.419021+0000) 2020-02-05T22:36:09.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:09.419+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.419574+0000) 2020-02-05T22:36:09.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:09.419+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.419747+0000) 2020-02-05T22:36:09.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:09.420+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.421362+0000) 2020-02-05T22:36:09.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:09.420+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.421298+0000) 2020-02-05T22:36:09.423 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:09.421+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.422254+0000) 2020-02-05T22:36:09.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:09.422+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.422584+0000) 2020-02-05T22:36:09.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:09.422+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.422851+0000) 2020-02-05T22:36:09.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:09.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.423575+0000) 2020-02-05T22:36:09.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:09.427+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.428313+0000) 2020-02-05T22:36:09.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:09.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120026 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:09.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:09.483+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:09.484134+0000) 2020-02-05T22:36:10.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:10.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:10.091035+0000) 2020-02-05T22:36:10.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:10.326+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:10.326846+0000) 2020-02-05T22:36:10.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:10.341+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:10.342827+0000) 2020-02-05T22:36:10.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:10.652+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:10.652712+0000) 2020-02-05T22:36:12.341 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:12.340+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:12.340305+0000) 2020-02-05T22:36:12.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:12.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:12.733371+0000) 2020-02-05T22:36:13.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:13.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:13.853461+0000) 2020-02-05T22:36:13.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:13.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:13.853461+0000) 2020-02-05T22:36:14.439 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:14.437+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.438541+0000) 2020-02-05T22:36:14.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:14.437+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.438967+0000) 2020-02-05T22:36:14.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:14.439+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.439492+0000) 2020-02-05T22:36:14.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:14.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.439656+0000) 2020-02-05T22:36:14.441 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:14.439+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.440180+0000) 2020-02-05T22:36:14.441 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:14.439+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.440419+0000) 2020-02-05T22:36:14.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:14.440+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.440730+0000) 2020-02-05T22:36:14.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:14.440+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.441629+0000) 2020-02-05T22:36:14.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:14.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.441995+0000) 2020-02-05T22:36:14.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:14.442+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.442877+0000) 2020-02-05T22:36:14.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:14.442+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.443387+0000) 2020-02-05T22:36:14.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:14.443+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.443599+0000) 2020-02-05T22:36:14.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:14.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.446269+0000) 2020-02-05T22:36:14.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:14.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:14.446517+0000) 2020-02-05T22:36:14.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:14.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:15.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:15.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:15.946992+0000) 2020-02-05T22:36:15.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:15.955+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:15.955334+0000) 2020-02-05T22:36:17.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:17.093+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:17.094369+0000) 2020-02-05T22:36:17.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:17.094+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:17.094618+0000) 2020-02-05T22:36:17.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:17.094+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:17.094909+0000) 2020-02-05T22:36:19.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:19.456+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.457656+0000) 2020-02-05T22:36:19.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:19.457+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.458916+0000) 2020-02-05T22:36:19.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:19.458+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.459332+0000) 2020-02-05T22:36:19.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:19.459+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.459925+0000) 2020-02-05T22:36:19.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:19.460+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.460814+0000) 2020-02-05T22:36:19.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:19.460+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.461389+0000) 2020-02-05T22:36:19.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:19.462+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.462881+0000) 2020-02-05T22:36:19.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:19.462+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.463145+0000) 2020-02-05T22:36:19.473 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:19.463+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.464150+0000) 2020-02-05T22:36:19.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:19.460+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.461894+0000) 2020-02-05T22:36:19.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:19.461+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.462338+0000) 2020-02-05T22:36:19.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:19.462+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.463396+0000) 2020-02-05T22:36:19.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:19.467+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.468142+0000) 2020-02-05T22:36:19.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:19.467+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.468429+0000) 2020-02-05T22:36:19.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:19.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120110 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:19.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:19.483+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:19.484276+0000) 2020-02-05T22:36:20.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:20.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:20.091270+0000) 2020-02-05T22:36:20.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:20.325+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:20.327030+0000) 2020-02-05T22:36:20.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:20.342+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:20.343011+0000) 2020-02-05T22:36:20.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:20.660+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:20.660607+0000) 2020-02-05T22:36:22.341 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:22.339+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:22.340559+0000) 2020-02-05T22:36:22.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:22.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:22.733607+0000) 2020-02-05T22:36:23.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:23.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:23.853730+0000) 2020-02-05T22:36:23.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:23.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:23.853726+0000) 2020-02-05T22:36:24.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:24.475+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.476742+0000) 2020-02-05T22:36:24.479 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:24.477+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.478065+0000) 2020-02-05T22:36:24.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:24.477+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.478780+0000) 2020-02-05T22:36:24.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:24.477+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.478881+0000) 2020-02-05T22:36:24.480 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:24.478+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.479601+0000) 2020-02-05T22:36:24.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:24.478+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.479726+0000) 2020-02-05T22:36:24.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:24.479+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.480373+0000) 2020-02-05T22:36:24.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:24.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.480857+0000) 2020-02-05T22:36:24.482 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:24.480+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.481455+0000) 2020-02-05T22:36:24.483 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:24.480+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.482362+0000) 2020-02-05T22:36:24.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:24.481+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.482940+0000) 2020-02-05T22:36:24.484 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:24.481+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.483182+0000) 2020-02-05T22:36:24.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:24.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.483790+0000) 2020-02-05T22:36:24.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:24.485+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:24.486844+0000) 2020-02-05T22:36:24.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:24.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120153 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:25.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:25.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:25.947273+0000) 2020-02-05T22:36:25.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:25.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:25.955628+0000) 2020-02-05T22:36:27.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:27.093+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:27.094654+0000) 2020-02-05T22:36:27.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:27.093+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:27.094841+0000) 2020-02-05T22:36:27.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:27.093+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:27.095150+0000) 2020-02-05T22:36:29.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:29.484+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.484473+0000) 2020-02-05T22:36:29.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:29.496+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.497643+0000) 2020-02-05T22:36:29.500 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:29.497+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.499402+0000) 2020-02-05T22:36:29.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:29.500+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.500151+0000) 2020-02-05T22:36:29.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:29.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.500934+0000) 2020-02-05T22:36:29.502 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:29.501+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.501286+0000) 2020-02-05T22:36:29.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:29.503+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.503134+0000) 2020-02-05T22:36:29.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:29.503+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.503344+0000) 2020-02-05T22:36:29.510 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:29.503+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.503987+0000) 2020-02-05T22:36:29.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:29.504+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.504498+0000) 2020-02-05T22:36:29.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:29.500+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.501618+0000) 2020-02-05T22:36:29.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:29.500+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.502257+0000) 2020-02-05T22:36:29.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:29.502+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.504286+0000) 2020-02-05T22:36:29.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:29.503+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.504716+0000) 2020-02-05T22:36:29.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:29.510+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:29.510103+0000) 2020-02-05T22:36:29.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:29.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:30.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:30.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:30.091503+0000) 2020-02-05T22:36:30.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:30.325+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:30.327243+0000) 2020-02-05T22:36:30.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:30.343+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:30.343212+0000) 2020-02-05T22:36:30.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:30.668+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:30.668676+0000) 2020-02-05T22:36:32.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:32.339+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:32.340805+0000) 2020-02-05T22:36:32.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:32.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:32.733859+0000) 2020-02-05T22:36:33.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:33.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:33.854001+0000) 2020-02-05T22:36:33.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:33.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:33.854004+0000) 2020-02-05T22:36:33.986 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:36:33.992 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:36:34.520 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:34.517+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.518860+0000) 2020-02-05T22:36:34.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:34.518+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.519395+0000) 2020-02-05T22:36:34.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:34.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.519873+0000) 2020-02-05T22:36:34.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:34.518+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.520215+0000) 2020-02-05T22:36:34.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:34.519+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.520660+0000) 2020-02-05T22:36:34.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:34.520+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.521212+0000) 2020-02-05T22:36:34.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:34.520+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.521667+0000) 2020-02-05T22:36:34.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:34.520+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.521871+0000) 2020-02-05T22:36:34.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:34.521+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.522743+0000) 2020-02-05T22:36:34.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:34.520+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.522348+0000) 2020-02-05T22:36:34.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:34.521+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.523094+0000) 2020-02-05T22:36:34.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:34.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.524020+0000) 2020-02-05T22:36:34.525 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:34.522+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.524331+0000) 2020-02-05T22:36:34.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:34.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:34.532289+0000) 2020-02-05T22:36:34.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:34.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:35.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:35.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:35.947499+0000) 2020-02-05T22:36:35.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:35.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:35.955826+0000) 2020-02-05T22:36:37.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:37.094+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:37.094904+0000) 2020-02-05T22:36:37.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:37.094+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:37.095107+0000) 2020-02-05T22:36:37.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:37.094+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:37.095384+0000) 2020-02-05T22:36:39.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:39.483+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.484707+0000) 2020-02-05T22:36:39.539 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:39.537+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.538432+0000) 2020-02-05T22:36:39.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:39.537+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.538676+0000) 2020-02-05T22:36:39.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:39.538+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.539885+0000) 2020-02-05T22:36:39.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:39.538+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.539911+0000) 2020-02-05T22:36:39.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:39.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.541233+0000) 2020-02-05T22:36:39.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:39.540+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.541647+0000) 2020-02-05T22:36:39.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:39.540+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.542242+0000) 2020-02-05T22:36:39.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:39.541+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.541276+0000) 2020-02-05T22:36:39.545 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:39.540+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.541807+0000) 2020-02-05T22:36:39.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:39.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.542752+0000) 2020-02-05T22:36:39.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:39.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.543532+0000) 2020-02-05T22:36:39.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:39.542+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.544041+0000) 2020-02-05T22:36:39.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:39.543+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.545048+0000) 2020-02-05T22:36:39.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:39.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:39.549672+0000) 2020-02-05T22:36:39.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:39.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:40.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:40.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:40.091708+0000) 2020-02-05T22:36:40.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:40.326+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:40.327484+0000) 2020-02-05T22:36:40.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:40.342+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:40.343445+0000) 2020-02-05T22:36:40.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:40.675+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:40.677036+0000) 2020-02-05T22:36:42.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:42.340+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:42.341068+0000) 2020-02-05T22:36:42.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:42.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:42.734148+0000) 2020-02-05T22:36:43.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:43.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:43.854340+0000) 2020-02-05T22:36:43.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:43.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:43.854340+0000) 2020-02-05T22:36:44.560 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:44.558+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.559487+0000) 2020-02-05T22:36:44.561 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:44.559+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.560290+0000) 2020-02-05T22:36:44.561 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:44.559+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.560508+0000) 2020-02-05T22:36:44.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:44.559+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.560872+0000) 2020-02-05T22:36:44.562 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:44.560+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.561316+0000) 2020-02-05T22:36:44.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:44.560+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.561807+0000) 2020-02-05T22:36:44.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:44.560+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.562235+0000) 2020-02-05T22:36:44.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:44.562+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.563696+0000) 2020-02-05T22:36:44.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:44.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.564296+0000) 2020-02-05T22:36:44.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:44.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.564571+0000) 2020-02-05T22:36:44.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:44.563+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.564693+0000) 2020-02-05T22:36:44.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:44.563+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.564983+0000) 2020-02-05T22:36:44.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:44.564+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.565654+0000) 2020-02-05T22:36:44.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:44.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:44.567896+0000) 2020-02-05T22:36:44.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:44.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:45.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:45.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:45.947736+0000) 2020-02-05T22:36:45.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:45.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:45.956089+0000) 2020-02-05T22:36:47.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:47.094+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:47.095152+0000) 2020-02-05T22:36:47.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:47.094+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:47.095346+0000) 2020-02-05T22:36:47.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:47.094+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:47.095659+0000) 2020-02-05T22:36:49.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:49.483+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.484958+0000) 2020-02-05T22:36:49.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:49.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.580492+0000) 2020-02-05T22:36:49.582 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:49.578+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.580091+0000) 2020-02-05T22:36:49.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:49.579+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.580703+0000) 2020-02-05T22:36:49.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:49.579+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.580987+0000) 2020-02-05T22:36:49.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:49.580+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.581331+0000) 2020-02-05T22:36:49.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:49.580+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.582385+0000) 2020-02-05T22:36:49.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:49.581+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.582069+0000) 2020-02-05T22:36:49.584 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:49.581+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.583070+0000) 2020-02-05T22:36:49.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:49.582+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.583407+0000) 2020-02-05T22:36:49.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:49.582+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.583812+0000) 2020-02-05T22:36:49.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:49.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.584438+0000) 2020-02-05T22:36:49.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:49.583+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.585017+0000) 2020-02-05T22:36:49.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:49.583+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.585270+0000) 2020-02-05T22:36:49.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:49.587+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:49.588559+0000) 2020-02-05T22:36:49.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:49.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120362 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:50.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:50.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:50.091919+0000) 2020-02-05T22:36:50.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:50.326+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:50.327660+0000) 2020-02-05T22:36:50.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:50.342+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:50.343654+0000) 2020-02-05T22:36:50.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:50.685+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:50.686181+0000) 2020-02-05T22:36:52.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:52.340+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:52.341319+0000) 2020-02-05T22:36:52.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:52.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:52.734379+0000) 2020-02-05T22:36:53.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:53.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:53.854623+0000) 2020-02-05T22:36:53.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:53.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:53.854623+0000) 2020-02-05T22:36:54.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:54.599+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.600656+0000) 2020-02-05T22:36:54.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:54.600+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.601370+0000) 2020-02-05T22:36:54.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:54.600+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.601803+0000) 2020-02-05T22:36:54.603 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:54.601+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.602646+0000) 2020-02-05T22:36:54.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:54.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.603157+0000) 2020-02-05T22:36:54.604 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:54.601+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.603199+0000) 2020-02-05T22:36:54.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:54.602+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.603766+0000) 2020-02-05T22:36:54.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:54.602+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.604078+0000) 2020-02-05T22:36:54.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:54.603+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.604495+0000) 2020-02-05T22:36:54.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:54.604+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.605060+0000) 2020-02-05T22:36:54.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:54.604+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.605448+0000) 2020-02-05T22:36:54.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:54.604+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.605573+0000) 2020-02-05T22:36:54.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:54.604+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.606127+0000) 2020-02-05T22:36:54.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:54.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:54.609939+0000) 2020-02-05T22:36:54.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:54.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:36:55.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:55.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:55.947936+0000) 2020-02-05T22:36:55.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:55.955+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:55.956331+0000) 2020-02-05T22:36:57.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:57.094+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:57.095383+0000) 2020-02-05T22:36:57.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:57.094+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:57.095614+0000) 2020-02-05T22:36:57.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:57.094+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:57.095899+0000) 2020-02-05T22:36:59.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:59.484+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.485184+0000) 2020-02-05T22:36:59.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:36:59.617+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.618874+0000) 2020-02-05T22:36:59.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:36:59.619+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.620963+0000) 2020-02-05T22:36:59.622 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:36:59.620+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.621725+0000) 2020-02-05T22:36:59.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:59.621+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.622293+0000) 2020-02-05T22:36:59.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:36:59.621+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.622463+0000) 2020-02-05T22:36:59.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:36:59.621+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.622898+0000) 2020-02-05T22:36:59.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:36:59.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.623298+0000) 2020-02-05T22:36:59.625 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:36:59.621+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.623294+0000) 2020-02-05T22:36:59.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:36:59.622+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.623647+0000) 2020-02-05T22:36:59.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:59.622+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.623893+0000) 2020-02-05T22:36:59.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:36:59.622+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.623958+0000) 2020-02-05T22:36:59.627 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:36:59.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.624288+0000) 2020-02-05T22:36:59.627 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:36:59.623+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.624566+0000) 2020-02-05T22:36:59.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:36:59.626+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:36:59.627300+0000) 2020-02-05T22:36:59.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:36:59.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120445 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:00.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:00.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:00.092181+0000) 2020-02-05T22:37:00.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:00.327+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:00.327932+0000) 2020-02-05T22:37:00.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:00.342+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:00.343961+0000) 2020-02-05T22:37:00.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:00.693+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:00.694492+0000) 2020-02-05T22:37:02.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:02.340+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:02.341574+0000) 2020-02-05T22:37:02.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:02.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:02.734610+0000) 2020-02-05T22:37:03.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:03.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:03.854918+0000) 2020-02-05T22:37:03.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:03.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:03.854930+0000) 2020-02-05T22:37:04.166 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:37:04.173 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:37:04.640 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:04.638+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.639399+0000) 2020-02-05T22:37:04.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:04.639+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.640941+0000) 2020-02-05T22:37:04.642 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:04.639+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.641245+0000) 2020-02-05T22:37:04.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:04.640+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.641662+0000) 2020-02-05T22:37:04.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:04.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.641960+0000) 2020-02-05T22:37:04.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:04.641+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.642374+0000) 2020-02-05T22:37:04.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:04.641+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.642734+0000) 2020-02-05T22:37:04.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:04.641+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.643386+0000) 2020-02-05T22:37:04.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:04.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.644180+0000) 2020-02-05T22:37:04.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:04.642+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.644382+0000) 2020-02-05T22:37:04.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:04.643+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.644596+0000) 2020-02-05T22:37:04.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:04.644+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.645343+0000) 2020-02-05T22:37:04.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:04.644+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.646207+0000) 2020-02-05T22:37:04.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:04.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:04.651816+0000) 2020-02-05T22:37:04.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:04.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:05.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:05.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:05.948198+0000) 2020-02-05T22:37:05.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:05.955+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:05.956567+0000) 2020-02-05T22:37:07.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:07.095+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:07.095662+0000) 2020-02-05T22:37:07.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:07.095+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:07.095827+0000) 2020-02-05T22:37:07.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:07.095+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:07.096165+0000) 2020-02-05T22:37:09.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:09.484+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.485406+0000) 2020-02-05T22:37:09.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:09.661+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.663135+0000) 2020-02-05T22:37:09.665 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:09.659+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.660939+0000) 2020-02-05T22:37:09.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:09.660+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.661654+0000) 2020-02-05T22:37:09.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:09.661+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.662409+0000) 2020-02-05T22:37:09.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:09.662+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.663919+0000) 2020-02-05T22:37:09.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:09.662+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.663790+0000) 2020-02-05T22:37:09.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:09.664+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.665196+0000) 2020-02-05T22:37:09.668 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:09.664+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.666368+0000) 2020-02-05T22:37:09.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:09.665+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.666455+0000) 2020-02-05T22:37:09.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:09.665+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.666499+0000) 2020-02-05T22:37:09.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:09.665+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.666963+0000) 2020-02-05T22:37:09.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:09.665+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.666980+0000) 2020-02-05T22:37:09.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:09.665+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.667068+0000) 2020-02-05T22:37:09.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:09.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:09.672396+0000) 2020-02-05T22:37:09.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:09.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:10.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:10.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:10.092390+0000) 2020-02-05T22:37:10.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:10.327+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:10.328146+0000) 2020-02-05T22:37:10.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:10.343+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:10.344180+0000) 2020-02-05T22:37:10.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:10.702+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:10.702472+0000) 2020-02-05T22:37:12.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:12.341+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:12.341826+0000) 2020-02-05T22:37:12.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:12.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:12.734849+0000) 2020-02-05T22:37:13.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:13.854+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:13.855193+0000) 2020-02-05T22:37:13.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:13.854+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:13.855190+0000) 2020-02-05T22:37:14.682 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:14.680+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.681559+0000) 2020-02-05T22:37:14.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:14.682+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.683067+0000) 2020-02-05T22:37:14.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:14.682+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.683940+0000) 2020-02-05T22:37:14.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:14.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.684590+0000) 2020-02-05T22:37:14.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:14.683+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.684846+0000) 2020-02-05T22:37:14.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:14.684+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.685223+0000) 2020-02-05T22:37:14.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:14.684+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.685483+0000) 2020-02-05T22:37:14.687 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:14.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.686007+0000) 2020-02-05T22:37:14.687 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:14.685+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.686540+0000) 2020-02-05T22:37:14.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:14.685+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.686898+0000) 2020-02-05T22:37:14.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:14.686+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.687292+0000) 2020-02-05T22:37:14.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:14.686+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.687219+0000) 2020-02-05T22:37:14.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:14.687+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.687868+0000) 2020-02-05T22:37:14.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:14.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:14.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:14.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:14.692300+0000) 2020-02-05T22:37:15.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:15.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:15.948431+0000) 2020-02-05T22:37:15.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:15.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:15.956813+0000) 2020-02-05T22:37:17.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:17.095+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:17.095887+0000) 2020-02-05T22:37:17.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:17.095+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:17.096079+0000) 2020-02-05T22:37:17.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:17.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:17.096474+0000) 2020-02-05T22:37:19.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:19.485+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.485655+0000) 2020-02-05T22:37:19.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:19.699+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.700144+0000) 2020-02-05T22:37:19.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:19.700+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.701420+0000) 2020-02-05T22:37:19.702 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:19.701+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.701890+0000) 2020-02-05T22:37:19.703 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:19.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.702424+0000) 2020-02-05T22:37:19.703 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:19.701+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.702704+0000) 2020-02-05T22:37:19.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:19.702+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.702880+0000) 2020-02-05T22:37:19.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:19.702+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.703598+0000) 2020-02-05T22:37:19.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:19.702+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.703678+0000) 2020-02-05T22:37:19.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:19.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.704107+0000) 2020-02-05T22:37:19.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:19.703+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.704348+0000) 2020-02-05T22:37:19.706 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:19.704+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.704616+0000) 2020-02-05T22:37:19.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:19.705+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.705523+0000) 2020-02-05T22:37:19.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:19.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.709534+0000) 2020-02-05T22:37:19.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:19.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:19.709767+0000) 2020-02-05T22:37:19.716 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:19.714+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:20.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:20.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:20.092626+0000) 2020-02-05T22:37:20.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:20.327+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:20.328375+0000) 2020-02-05T22:37:20.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:20.344+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:20.344432+0000) 2020-02-05T22:37:20.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:20.710+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:20.710243+0000) 2020-02-05T22:37:22.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:22.341+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:22.342066+0000) 2020-02-05T22:37:22.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:22.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:22.735171+0000) 2020-02-05T22:37:23.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:23.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:23.855461+0000) 2020-02-05T22:37:23.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:23.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:23.855461+0000) 2020-02-05T22:37:24.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:24.719+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.720041+0000) 2020-02-05T22:37:24.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:24.721+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.721432+0000) 2020-02-05T22:37:24.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:24.721+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.721832+0000) 2020-02-05T22:37:24.723 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:24.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.722293+0000) 2020-02-05T22:37:24.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:24.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.722972+0000) 2020-02-05T22:37:24.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:24.723+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.723705+0000) 2020-02-05T22:37:24.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:24.724+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.724918+0000) 2020-02-05T22:37:24.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:24.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.725484+0000) 2020-02-05T22:37:24.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:24.728+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.728892+0000) 2020-02-05T22:37:24.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:24.725+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.725668+0000) 2020-02-05T22:37:24.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:24.722+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.723324+0000) 2020-02-05T22:37:24.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:24.726+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.726646+0000) 2020-02-05T22:37:24.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:24.726+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.727031+0000) 2020-02-05T22:37:24.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:24.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:24.727565+0000) 2020-02-05T22:37:24.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:24.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:25.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:25.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:25.948657+0000) 2020-02-05T22:37:25.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:25.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:25.957066+0000) 2020-02-05T22:37:27.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:27.095+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:27.096156+0000) 2020-02-05T22:37:27.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:27.095+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:27.096312+0000) 2020-02-05T22:37:27.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:27.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:27.096712+0000) 2020-02-05T22:37:29.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:29.484+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.485892+0000) 2020-02-05T22:37:29.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:29.740+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.741038+0000) 2020-02-05T22:37:29.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:29.741+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.742281+0000) 2020-02-05T22:37:29.743 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:29.741+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.742669+0000) 2020-02-05T22:37:29.744 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:29.742+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.743432+0000) 2020-02-05T22:37:29.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:29.742+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.744035+0000) 2020-02-05T22:37:29.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:29.744+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.744476+0000) 2020-02-05T22:37:29.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:29.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.745149+0000) 2020-02-05T22:37:29.746 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:29.744+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.745580+0000) 2020-02-05T22:37:29.746 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:29.745+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.745559+0000) 2020-02-05T22:37:29.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:29.745+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.746314+0000) 2020-02-05T22:37:29.747 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:29.745+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.746097+0000) 2020-02-05T22:37:29.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:29.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.746699+0000) 2020-02-05T22:37:29.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:29.746+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.746909+0000) 2020-02-05T22:37:29.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:29.749+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:29.751041+0000) 2020-02-05T22:37:29.756 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:29.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:30.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:30.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:30.092881+0000) 2020-02-05T22:37:30.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:30.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:30.328633+0000) 2020-02-05T22:37:30.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:30.343+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:30.344731+0000) 2020-02-05T22:37:30.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:30.717+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:30.718272+0000) 2020-02-05T22:37:32.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:32.342+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:32.342329+0000) 2020-02-05T22:37:32.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:32.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:32.735375+0000) 2020-02-05T22:37:33.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:33.854+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:33.855714+0000) 2020-02-05T22:37:33.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:33.854+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:33.855714+0000) 2020-02-05T22:37:34.355 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:37:34.362 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:37:34.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:34.760+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.760708+0000) 2020-02-05T22:37:34.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:34.759+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.761020+0000) 2020-02-05T22:37:34.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:34.760+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.761646+0000) 2020-02-05T22:37:34.764 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:34.761+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.762935+0000) 2020-02-05T22:37:34.764 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:34.762+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.763324+0000) 2020-02-05T22:37:34.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:34.762+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.763653+0000) 2020-02-05T22:37:34.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:34.763+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.764532+0000) 2020-02-05T22:37:34.766 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:34.764+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.764647+0000) 2020-02-05T22:37:34.766 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:34.764+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.764787+0000) 2020-02-05T22:37:34.767 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:34.764+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.765455+0000) 2020-02-05T22:37:34.767 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:34.764+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.765926+0000) 2020-02-05T22:37:34.767 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:34.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.766605+0000) 2020-02-05T22:37:34.768 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:34.766+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.766971+0000) 2020-02-05T22:37:34.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:34.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:34.769787+0000) 2020-02-05T22:37:34.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:34.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:35.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:35.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:35.948931+0000) 2020-02-05T22:37:35.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:35.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:35.957328+0000) 2020-02-05T22:37:37.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:37.096+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:37.096410+0000) 2020-02-05T22:37:37.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:37.096+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:37.096522+0000) 2020-02-05T22:37:37.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:37.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:37.096907+0000) 2020-02-05T22:37:39.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:39.485+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.486143+0000) 2020-02-05T22:37:39.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:39.781+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.783017+0000) 2020-02-05T22:37:39.784 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:39.783+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.783508+0000) 2020-02-05T22:37:39.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:39.782+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.783850+0000) 2020-02-05T22:37:39.785 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:39.783+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.784370+0000) 2020-02-05T22:37:39.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:39.783+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.784770+0000) 2020-02-05T22:37:39.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:39.784+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.785362+0000) 2020-02-05T22:37:39.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:39.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.786074+0000) 2020-02-05T22:37:39.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:39.784+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.786034+0000) 2020-02-05T22:37:39.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:39.785+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.786272+0000) 2020-02-05T22:37:39.788 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:39.786+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.786692+0000) 2020-02-05T22:37:39.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:39.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.786957+0000) 2020-02-05T22:37:39.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:39.786+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.787207+0000) 2020-02-05T22:37:39.789 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:39.786+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.787266+0000) 2020-02-05T22:37:39.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:39.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:39.792134+0000) 2020-02-05T22:37:39.798 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:39.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:40.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:40.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:40.093119+0000) 2020-02-05T22:37:40.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:40.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:40.328838+0000) 2020-02-05T22:37:40.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:40.344+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:40.344942+0000) 2020-02-05T22:37:40.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:40.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:40.730132+0000) 2020-02-05T22:37:42.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:42.342+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:42.342589+0000) 2020-02-05T22:37:42.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:42.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:42.735610+0000) 2020-02-05T22:37:43.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:43.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:43.855989+0000) 2020-02-05T22:37:43.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:43.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:43.855991+0000) 2020-02-05T22:37:44.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:44.802+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.803365+0000) 2020-02-05T22:37:44.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:44.803+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.803804+0000) 2020-02-05T22:37:44.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:44.803+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.804310+0000) 2020-02-05T22:37:44.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:44.803+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.804985+0000) 2020-02-05T22:37:44.806 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:44.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.805330+0000) 2020-02-05T22:37:44.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:44.804+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.806025+0000) 2020-02-05T22:37:44.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:44.805+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.806078+0000) 2020-02-05T22:37:44.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:44.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.806512+0000) 2020-02-05T22:37:44.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:44.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.807875+0000) 2020-02-05T22:37:44.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:44.807+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.808221+0000) 2020-02-05T22:37:44.810 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:44.807+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.808022+0000) 2020-02-05T22:37:44.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:44.808+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.808850+0000) 2020-02-05T22:37:44.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:44.808+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.809093+0000) 2020-02-05T22:37:44.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:44.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:44.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:44.814+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:44.815762+0000) 2020-02-05T22:37:45.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:45.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:45.949166+0000) 2020-02-05T22:37:45.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:45.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:45.957611+0000) 2020-02-05T22:37:47.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:47.096+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:47.096747+0000) 2020-02-05T22:37:47.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:47.096+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:47.096739+0000) 2020-02-05T22:37:47.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:47.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:47.097153+0000) 2020-02-05T22:37:49.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:49.485+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.486364+0000) 2020-02-05T22:37:49.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:49.820+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.821535+0000) 2020-02-05T22:37:49.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:49.821+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.822844+0000) 2020-02-05T22:37:49.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:49.823+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.824572+0000) 2020-02-05T22:37:49.826 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:49.823+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.824153+0000) 2020-02-05T22:37:49.827 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:49.825+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.825979+0000) 2020-02-05T22:37:49.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:49.825+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.825610+0000) 2020-02-05T22:37:49.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:49.825+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.826219+0000) 2020-02-05T22:37:49.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:49.825+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.826741+0000) 2020-02-05T22:37:49.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:49.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.828317+0000) 2020-02-05T22:37:49.834 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:49.828+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.828650+0000) 2020-02-05T22:37:49.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:49.828+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.829127+0000) 2020-02-05T22:37:49.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:49.826+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.827810+0000) 2020-02-05T22:37:49.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:49.827+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.828919+0000) 2020-02-05T22:37:49.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:49.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:49.836727+0000) 2020-02-05T22:37:49.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:49.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:50.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:50.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:50.093364+0000) 2020-02-05T22:37:50.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:50.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:50.329055+0000) 2020-02-05T22:37:50.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:50.344+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:50.345198+0000) 2020-02-05T22:37:50.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:50.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:50.738134+0000) 2020-02-05T22:37:52.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:52.341+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:52.342828+0000) 2020-02-05T22:37:52.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:52.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:52.735840+0000) 2020-02-05T22:37:53.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:53.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:53.856300+0000) 2020-02-05T22:37:53.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:53.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:53.856300+0000) 2020-02-05T22:37:54.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:54.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.843161+0000) 2020-02-05T22:37:54.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:54.842+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.844017+0000) 2020-02-05T22:37:54.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:54.843+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.844375+0000) 2020-02-05T22:37:54.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:54.843+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.844624+0000) 2020-02-05T22:37:54.846 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:54.844+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.845492+0000) 2020-02-05T22:37:54.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:54.844+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.845656+0000) 2020-02-05T22:37:54.847 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:54.844+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.846174+0000) 2020-02-05T22:37:54.847 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:54.845+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.846407+0000) 2020-02-05T22:37:54.848 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:54.845+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.847058+0000) 2020-02-05T22:37:54.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:54.846+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.847425+0000) 2020-02-05T22:37:54.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:54.847+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.848067+0000) 2020-02-05T22:37:54.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:54.846+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.848251+0000) 2020-02-05T22:37:54.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:54.846+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.848308+0000) 2020-02-05T22:37:54.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:54.849+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:54.850535+0000) 2020-02-05T22:37:54.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:54.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:37:55.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:55.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:55.949417+0000) 2020-02-05T22:37:55.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:55.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:55.957894+0000) 2020-02-05T22:37:57.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:57.096+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:57.097035+0000) 2020-02-05T22:37:57.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:57.096+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:57.097031+0000) 2020-02-05T22:37:57.099 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:57.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:57.097463+0000) 2020-02-05T22:37:59.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:59.486+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.486600+0000) 2020-02-05T22:37:59.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:37:59.861+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.863246+0000) 2020-02-05T22:37:59.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:37:59.862+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.863629+0000) 2020-02-05T22:37:59.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:37:59.863+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.864416+0000) 2020-02-05T22:37:59.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:37:59.864+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.865546+0000) 2020-02-05T22:37:59.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:37:59.865+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.866402+0000) 2020-02-05T22:37:59.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:37:59.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.866839+0000) 2020-02-05T22:37:59.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:59.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.867439+0000) 2020-02-05T22:37:59.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:37:59.866+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.867920+0000) 2020-02-05T22:37:59.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:37:59.866+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.867977+0000) 2020-02-05T22:37:59.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:37:59.867+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.868542+0000) 2020-02-05T22:37:59.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:59.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.868745+0000) 2020-02-05T22:37:59.870 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:37:59.867+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.868934+0000) 2020-02-05T22:37:59.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:37:59.867+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.869267+0000) 2020-02-05T22:37:59.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:37:59.875+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:37:59.876152+0000) 2020-02-05T22:37:59.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:37:59.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:00.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:00.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:00.093618+0000) 2020-02-05T22:38:00.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:00.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:00.329286+0000) 2020-02-05T22:38:00.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:00.344+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:00.345360+0000) 2020-02-05T22:38:00.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:00.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:00.746600+0000) 2020-02-05T22:38:02.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:02.341+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:02.343075+0000) 2020-02-05T22:38:02.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:02.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:02.736108+0000) 2020-02-05T22:38:03.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:03.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:03.856549+0000) 2020-02-05T22:38:03.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:03.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:03.856547+0000) 2020-02-05T22:38:04.518 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:38:04.525 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:38:04.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:04.883+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.884138+0000) 2020-02-05T22:38:04.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:04.883+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.884540+0000) 2020-02-05T22:38:04.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:04.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.886378+0000) 2020-02-05T22:38:04.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:04.885+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.886585+0000) 2020-02-05T22:38:04.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:04.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.887545+0000) 2020-02-05T22:38:04.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:04.887+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.888444+0000) 2020-02-05T22:38:04.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:04.884+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.885681+0000) 2020-02-05T22:38:04.891 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:04.883+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.885239+0000) 2020-02-05T22:38:04.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:04.886+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.887804+0000) 2020-02-05T22:38:04.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:04.887+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.888842+0000) 2020-02-05T22:38:04.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:04.887+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.889117+0000) 2020-02-05T22:38:04.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:04.887+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.889220+0000) 2020-02-05T22:38:04.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:04.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.893408+0000) 2020-02-05T22:38:04.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:04.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:04.893615+0000) 2020-02-05T22:38:04.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:04.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 120984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:05.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:05.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:05.949650+0000) 2020-02-05T22:38:05.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:05.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:05.958154+0000) 2020-02-05T22:38:07.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:07.096+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:07.097335+0000) 2020-02-05T22:38:07.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:07.096+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:07.097333+0000) 2020-02-05T22:38:07.099 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:07.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:07.097712+0000) 2020-02-05T22:38:09.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:09.486+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.486836+0000) 2020-02-05T22:38:09.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:09.901+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.902702+0000) 2020-02-05T22:38:09.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:09.902+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.903313+0000) 2020-02-05T22:38:09.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:09.902+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.903752+0000) 2020-02-05T22:38:09.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:09.903+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.904573+0000) 2020-02-05T22:38:09.906 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:09.903+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.904752+0000) 2020-02-05T22:38:09.906 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:09.903+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.905201+0000) 2020-02-05T22:38:09.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:09.904+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.905769+0000) 2020-02-05T22:38:09.907 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:09.905+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.906466+0000) 2020-02-05T22:38:09.909 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:09.907+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.908202+0000) 2020-02-05T22:38:09.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:09.907+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.908614+0000) 2020-02-05T22:38:09.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:09.908+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.909402+0000) 2020-02-05T22:38:09.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:09.908+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.909949+0000) 2020-02-05T22:38:09.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:09.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.912469+0000) 2020-02-05T22:38:09.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:09.912+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:09.913361+0000) 2020-02-05T22:38:09.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:09.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:10.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:10.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:10.093835+0000) 2020-02-05T22:38:10.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:10.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:10.329500+0000) 2020-02-05T22:38:10.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:10.345+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:10.345570+0000) 2020-02-05T22:38:10.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:10.753+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:10.753460+0000) 2020-02-05T22:38:12.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:12.342+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:12.343316+0000) 2020-02-05T22:38:12.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:12.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:12.736375+0000) 2020-02-05T22:38:13.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:13.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:13.856815+0000) 2020-02-05T22:38:13.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:13.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:13.856815+0000) 2020-02-05T22:38:14.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:14.924+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.925147+0000) 2020-02-05T22:38:14.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:14.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.925552+0000) 2020-02-05T22:38:14.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:14.925+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.926578+0000) 2020-02-05T22:38:14.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:14.924+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.926306+0000) 2020-02-05T22:38:14.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:14.925+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.926881+0000) 2020-02-05T22:38:14.928 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:14.925+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.927309+0000) 2020-02-05T22:38:14.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:14.926+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.927612+0000) 2020-02-05T22:38:14.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:14.926+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.927923+0000) 2020-02-05T22:38:14.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:14.927+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.928543+0000) 2020-02-05T22:38:14.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:14.926+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.928258+0000) 2020-02-05T22:38:14.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:14.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.929421+0000) 2020-02-05T22:38:14.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:14.927+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.928921+0000) 2020-02-05T22:38:14.931 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:14.928+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.929994+0000) 2020-02-05T22:38:14.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:14.931+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:14.932200+0000) 2020-02-05T22:38:14.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:14.938+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121065 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:15.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:15.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:15.949880+0000) 2020-02-05T22:38:15.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:15.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:15.958419+0000) 2020-02-05T22:38:17.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:17.096+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:17.097669+0000) 2020-02-05T22:38:17.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:17.096+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:17.097669+0000) 2020-02-05T22:38:17.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:17.096+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:17.097921+0000) 2020-02-05T22:38:19.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:19.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.487070+0000) 2020-02-05T22:38:19.947 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:19.944+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.946170+0000) 2020-02-05T22:38:19.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:19.945+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.946811+0000) 2020-02-05T22:38:19.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:19.947+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.947309+0000) 2020-02-05T22:38:19.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:19.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.947651+0000) 2020-02-05T22:38:19.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:19.948+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.948158+0000) 2020-02-05T22:38:19.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:19.948+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.948286+0000) 2020-02-05T22:38:19.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:19.948+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.948621+0000) 2020-02-05T22:38:19.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:19.947+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.949115+0000) 2020-02-05T22:38:19.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:19.948+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.950200+0000) 2020-02-05T22:38:19.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:19.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.950367+0000) 2020-02-05T22:38:19.951 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:19.949+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.950944+0000) 2020-02-05T22:38:19.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:19.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.951403+0000) 2020-02-05T22:38:19.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:19.950+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.951790+0000) 2020-02-05T22:38:19.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:19.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:19.956604+0000) 2020-02-05T22:38:19.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:19.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:20.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:20.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:20.094050+0000) 2020-02-05T22:38:20.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:20.328+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:20.329685+0000) 2020-02-05T22:38:20.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:20.345+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:20.345816+0000) 2020-02-05T22:38:20.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:20.761+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:20.761581+0000) 2020-02-05T22:38:22.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:22.342+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:22.343574+0000) 2020-02-05T22:38:22.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:22.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:22.736611+0000) 2020-02-05T22:38:23.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:23.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:23.857164+0000) 2020-02-05T22:38:23.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:23.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:23.857158+0000) 2020-02-05T22:38:24.967 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:24.964+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.965406+0000) 2020-02-05T22:38:24.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:24.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.967320+0000) 2020-02-05T22:38:24.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:24.966+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.967696+0000) 2020-02-05T22:38:24.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:24.967+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.968285+0000) 2020-02-05T22:38:24.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:24.967+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.968769+0000) 2020-02-05T22:38:24.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:24.968+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.969361+0000) 2020-02-05T22:38:24.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:24.968+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.970170+0000) 2020-02-05T22:38:24.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:24.969+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.970521+0000) 2020-02-05T22:38:24.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:24.970+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.971482+0000) 2020-02-05T22:38:24.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:24.969+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.971284+0000) 2020-02-05T22:38:24.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:24.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.971925+0000) 2020-02-05T22:38:24.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:24.970+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.972158+0000) 2020-02-05T22:38:24.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:24.970+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.972242+0000) 2020-02-05T22:38:24.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:24.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:24.977945+0000) 2020-02-05T22:38:24.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:24.980+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121146 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:25.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:25.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:25.950143+0000) 2020-02-05T22:38:25.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:25.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:25.958730+0000) 2020-02-05T22:38:27.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:27.097+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:27.097977+0000) 2020-02-05T22:38:27.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:27.097+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:27.097972+0000) 2020-02-05T22:38:27.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:27.097+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:27.098199+0000) 2020-02-05T22:38:29.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:29.486+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.487366+0000) 2020-02-05T22:38:29.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:29.986+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.987778+0000) 2020-02-05T22:38:29.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:29.987+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.988959+0000) 2020-02-05T22:38:29.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:29.988+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.989615+0000) 2020-02-05T22:38:29.991 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:29.988+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.989617+0000) 2020-02-05T22:38:29.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:29.989+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.990217+0000) 2020-02-05T22:38:29.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:29.988+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.990315+0000) 2020-02-05T22:38:29.993 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:29.989+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.990907+0000) 2020-02-05T22:38:29.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:29.989+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.991152+0000) 2020-02-05T22:38:29.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:29.990+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.991232+0000) 2020-02-05T22:38:29.994 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:29.990+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.991923+0000) 2020-02-05T22:38:29.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:29.990+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.992214+0000) 2020-02-05T22:38:29.995 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:29.992+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.993667+0000) 2020-02-05T22:38:29.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:29.991+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.992530+0000) 2020-02-05T22:38:29.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:29.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:29.998949+0000) 2020-02-05T22:38:30.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:30.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:30.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:30.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:30.094298+0000) 2020-02-05T22:38:30.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:30.329+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:30.329910+0000) 2020-02-05T22:38:30.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:30.345+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:30.346027+0000) 2020-02-05T22:38:30.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:30.769+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:30.770879+0000) 2020-02-05T22:38:32.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:32.342+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:32.343780+0000) 2020-02-05T22:38:32.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:32.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:32.736850+0000) 2020-02-05T22:38:33.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:33.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:33.857434+0000) 2020-02-05T22:38:33.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:33.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:33.857433+0000) 2020-02-05T22:38:34.690 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:38:34.696 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:38:35.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:35.006+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.007554+0000) 2020-02-05T22:38:35.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:35.007+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.008505+0000) 2020-02-05T22:38:35.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:35.007+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.008888+0000) 2020-02-05T22:38:35.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:35.008+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.009276+0000) 2020-02-05T22:38:35.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:35.008+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.010371+0000) 2020-02-05T22:38:35.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:35.010+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.011247+0000) 2020-02-05T22:38:35.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:35.010+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.011816+0000) 2020-02-05T22:38:35.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:35.011+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.012472+0000) 2020-02-05T22:38:35.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:35.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.013382+0000) 2020-02-05T22:38:35.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:35.012+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.013658+0000) 2020-02-05T22:38:35.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:35.012+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.013764+0000) 2020-02-05T22:38:35.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:35.013+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.014215+0000) 2020-02-05T22:38:35.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:35.012+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.014319+0000) 2020-02-05T22:38:35.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:35.018+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.019284+0000) 2020-02-05T22:38:35.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:35.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:35.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:35.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.950305+0000) 2020-02-05T22:38:35.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:35.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:35.958966+0000) 2020-02-05T22:38:37.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:37.097+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:37.098277+0000) 2020-02-05T22:38:37.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:37.097+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:37.098275+0000) 2020-02-05T22:38:37.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:37.097+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:37.098487+0000) 2020-02-05T22:38:39.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:39.486+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:39.487596+0000) 2020-02-05T22:38:40.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:40.027+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.029004+0000) 2020-02-05T22:38:40.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:40.028+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.030011+0000) 2020-02-05T22:38:40.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:40.029+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.030436+0000) 2020-02-05T22:38:40.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:40.029+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.031054+0000) 2020-02-05T22:38:40.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:40.030+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.031569+0000) 2020-02-05T22:38:40.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:40.030+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.031797+0000) 2020-02-05T22:38:40.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:40.031+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.032399+0000) 2020-02-05T22:38:40.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:40.031+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.032978+0000) 2020-02-05T22:38:40.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:40.032+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.033748+0000) 2020-02-05T22:38:40.035 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:40.032+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.034253+0000) 2020-02-05T22:38:40.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:40.032+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.033551+0000) 2020-02-05T22:38:40.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:40.033+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.034439+0000) 2020-02-05T22:38:40.036 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:40.031+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.033246+0000) 2020-02-05T22:38:40.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:40.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121268 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:40.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:40.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.039819+0000) 2020-02-05T22:38:40.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:40.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.094518+0000) 2020-02-05T22:38:40.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:40.329+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.330144+0000) 2020-02-05T22:38:40.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:40.345+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.346272+0000) 2020-02-05T22:38:40.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:40.778+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:40.779229+0000) 2020-02-05T22:38:42.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:42.343+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:42.344042+0000) 2020-02-05T22:38:42.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:42.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:42.737138+0000) 2020-02-05T22:38:43.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:43.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:43.857724+0000) 2020-02-05T22:38:43.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:43.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:43.857724+0000) 2020-02-05T22:38:45.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:45.048+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.049552+0000) 2020-02-05T22:38:45.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:45.049+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.050473+0000) 2020-02-05T22:38:45.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:45.050+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.050781+0000) 2020-02-05T22:38:45.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:45.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.051922+0000) 2020-02-05T22:38:45.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:45.052+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.053150+0000) 2020-02-05T22:38:45.054 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:45.052+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.053704+0000) 2020-02-05T22:38:45.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:45.053+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.053600+0000) 2020-02-05T22:38:45.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:45.053+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.054365+0000) 2020-02-05T22:38:45.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:45.054+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.055058+0000) 2020-02-05T22:38:45.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:45.053+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.054948+0000) 2020-02-05T22:38:45.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:45.055+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.055471+0000) 2020-02-05T22:38:45.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:45.055+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.055948+0000) 2020-02-05T22:38:45.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:45.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.057591+0000) 2020-02-05T22:38:45.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:45.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.057860+0000) 2020-02-05T22:38:45.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:45.064+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:45.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:45.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.950512+0000) 2020-02-05T22:38:45.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:45.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:45.959213+0000) 2020-02-05T22:38:47.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:47.098+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:47.098569+0000) 2020-02-05T22:38:47.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:47.098+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:47.098568+0000) 2020-02-05T22:38:47.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:47.098+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:47.098743+0000) 2020-02-05T22:38:49.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:49.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:49.487829+0000) 2020-02-05T22:38:50.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:50.070+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.071273+0000) 2020-02-05T22:38:50.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:50.071+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.072629+0000) 2020-02-05T22:38:50.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:50.072+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.073247+0000) 2020-02-05T22:38:50.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:50.072+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.073503+0000) 2020-02-05T22:38:50.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:50.072+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.073969+0000) 2020-02-05T22:38:50.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:50.073+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.074241+0000) 2020-02-05T22:38:50.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:50.074+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.074803+0000) 2020-02-05T22:38:50.076 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:50.073+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.074989+0000) 2020-02-05T22:38:50.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:50.075+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.075423+0000) 2020-02-05T22:38:50.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:50.074+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.076010+0000) 2020-02-05T22:38:50.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:50.076+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.076480+0000) 2020-02-05T22:38:50.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:50.076+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.076736+0000) 2020-02-05T22:38:50.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:50.076+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.076772+0000) 2020-02-05T22:38:50.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:50.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.084120+0000) 2020-02-05T22:38:50.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:50.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:50.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:50.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.094678+0000) 2020-02-05T22:38:50.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:50.329+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.330372+0000) 2020-02-05T22:38:50.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:50.345+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.346451+0000) 2020-02-05T22:38:50.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:50.786+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:50.787547+0000) 2020-02-05T22:38:52.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:52.343+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:52.344306+0000) 2020-02-05T22:38:52.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:52.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:52.737380+0000) 2020-02-05T22:38:53.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:53.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:53.857995+0000) 2020-02-05T22:38:53.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:53.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:53.857994+0000) 2020-02-05T22:38:55.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:38:55.091+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.092399+0000) 2020-02-05T22:38:55.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:55.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.092822+0000) 2020-02-05T22:38:55.094 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:55.092+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.093316+0000) 2020-02-05T22:38:55.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:55.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.093951+0000) 2020-02-05T22:38:55.096 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:38:55.094+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.095219+0000) 2020-02-05T22:38:55.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:38:55.094+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.094993+0000) 2020-02-05T22:38:55.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:55.095+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.095518+0000) 2020-02-05T22:38:55.097 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:55.094+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.095789+0000) 2020-02-05T22:38:55.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:55.095+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.096349+0000) 2020-02-05T22:38:55.098 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:38:55.096+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.096776+0000) 2020-02-05T22:38:55.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:38:55.095+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.097007+0000) 2020-02-05T22:38:55.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:55.097+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.097799+0000) 2020-02-05T22:38:55.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:38:55.097+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.098065+0000) 2020-02-05T22:38:55.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:55.102+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.103516+0000) 2020-02-05T22:38:55.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:38:55.105+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121392 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:38:55.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:38:55.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.950743+0000) 2020-02-05T22:38:55.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:38:55.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:55.959461+0000) 2020-02-05T22:38:57.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:38:57.098+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:57.098830+0000) 2020-02-05T22:38:57.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:38:57.098+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:57.098829+0000) 2020-02-05T22:38:57.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:38:57.098+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:57.099017+0000) 2020-02-05T22:38:59.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:38:59.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:38:59.488068+0000) 2020-02-05T22:39:00.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:00.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.094882+0000) 2020-02-05T22:39:00.110 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:00.108+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.109767+0000) 2020-02-05T22:39:00.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:00.111+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.111408+0000) 2020-02-05T22:39:00.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:00.112+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.112562+0000) 2020-02-05T22:39:00.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:00.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.113032+0000) 2020-02-05T22:39:00.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:00.113+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.113599+0000) 2020-02-05T22:39:00.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:00.113+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.114136+0000) 2020-02-05T22:39:00.115 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:00.113+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.114932+0000) 2020-02-05T22:39:00.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:00.114+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.115383+0000) 2020-02-05T22:39:00.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:00.115+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.116313+0000) 2020-02-05T22:39:00.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:00.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.116779+0000) 2020-02-05T22:39:00.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:00.116+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.117403+0000) 2020-02-05T22:39:00.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:00.117+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.117625+0000) 2020-02-05T22:39:00.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:00.117+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.117754+0000) 2020-02-05T22:39:00.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:00.120+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.122013+0000) 2020-02-05T22:39:00.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:00.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121432 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:00.331 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:00.330+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.330590+0000) 2020-02-05T22:39:00.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:00.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.346661+0000) 2020-02-05T22:39:00.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:00.795+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:00.795510+0000) 2020-02-05T22:39:02.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:02.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:02.344704+0000) 2020-02-05T22:39:02.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:02.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:02.737637+0000) 2020-02-05T22:39:03.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:03.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:03.858291+0000) 2020-02-05T22:39:03.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:03.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:03.858291+0000) 2020-02-05T22:39:04.838 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:39:04.844 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:39:05.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:05.131+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.131614+0000) 2020-02-05T22:39:05.133 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:05.131+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.132838+0000) 2020-02-05T22:39:05.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:05.132+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.133149+0000) 2020-02-05T22:39:05.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:05.132+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.133538+0000) 2020-02-05T22:39:05.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:05.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.134056+0000) 2020-02-05T22:39:05.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:05.133+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.134222+0000) 2020-02-05T22:39:05.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:05.134+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.134757+0000) 2020-02-05T22:39:05.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:05.134+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.135009+0000) 2020-02-05T22:39:05.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:05.133+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.134532+0000) 2020-02-05T22:39:05.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:05.134+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.135318+0000) 2020-02-05T22:39:05.137 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:05.135+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.135515+0000) 2020-02-05T22:39:05.137 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:05.135+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.135823+0000) 2020-02-05T22:39:05.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:05.135+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.136065+0000) 2020-02-05T22:39:05.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:05.140+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.141877+0000) 2020-02-05T22:39:05.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:05.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:05.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:05.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.950971+0000) 2020-02-05T22:39:05.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:05.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:05.959706+0000) 2020-02-05T22:39:07.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:07.098+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:07.099116+0000) 2020-02-05T22:39:07.101 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:07.098+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:07.099114+0000) 2020-02-05T22:39:07.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:07.098+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:07.099311+0000) 2020-02-05T22:39:09.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:09.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:09.488302+0000) 2020-02-05T22:39:10.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:10.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.095137+0000) 2020-02-05T22:39:10.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:10.147+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.148844+0000) 2020-02-05T22:39:10.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:10.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.150910+0000) 2020-02-05T22:39:10.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:10.151+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.151739+0000) 2020-02-05T22:39:10.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:10.151+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.152168+0000) 2020-02-05T22:39:10.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:10.151+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.152601+0000) 2020-02-05T22:39:10.154 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:10.152+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.153183+0000) 2020-02-05T22:39:10.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:10.152+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.154006+0000) 2020-02-05T22:39:10.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:10.153+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.154401+0000) 2020-02-05T22:39:10.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:10.154+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.154498+0000) 2020-02-05T22:39:10.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:10.154+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.155129+0000) 2020-02-05T22:39:10.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:10.155+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.155836+0000) 2020-02-05T22:39:10.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:10.153+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.154962+0000) 2020-02-05T22:39:10.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:10.155+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.156293+0000) 2020-02-05T22:39:10.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:10.157+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.158804+0000) 2020-02-05T22:39:10.165 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:10.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:10.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:10.330+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.330850+0000) 2020-02-05T22:39:10.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:10.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.346868+0000) 2020-02-05T22:39:10.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:10.807+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:10.807204+0000) 2020-02-05T22:39:12.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:12.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:12.344960+0000) 2020-02-05T22:39:12.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:12.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:12.737889+0000) 2020-02-05T22:39:13.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:13.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:13.858563+0000) 2020-02-05T22:39:13.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:13.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:13.858563+0000) 2020-02-05T22:39:15.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:15.168+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.168700+0000) 2020-02-05T22:39:15.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:15.170+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.170471+0000) 2020-02-05T22:39:15.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:15.171+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.171308+0000) 2020-02-05T22:39:15.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:15.170+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.171069+0000) 2020-02-05T22:39:15.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:15.171+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.171455+0000) 2020-02-05T22:39:15.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:15.171+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.172367+0000) 2020-02-05T22:39:15.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:15.173+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.173498+0000) 2020-02-05T22:39:15.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:15.173+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.173480+0000) 2020-02-05T22:39:15.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:15.173+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.173888+0000) 2020-02-05T22:39:15.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:15.174+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.174297+0000) 2020-02-05T22:39:15.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:15.175+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.175342+0000) 2020-02-05T22:39:15.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:15.175+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.175491+0000) 2020-02-05T22:39:15.177 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:15.176+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.176597+0000) 2020-02-05T22:39:15.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:15.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.178105+0000) 2020-02-05T22:39:15.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:15.184+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121556 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:15.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:15.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.951182+0000) 2020-02-05T22:39:15.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:15.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:15.959927+0000) 2020-02-05T22:39:17.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:17.099+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:17.099412+0000) 2020-02-05T22:39:17.101 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:17.099+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:17.099412+0000) 2020-02-05T22:39:17.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:17.099+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:17.099658+0000) 2020-02-05T22:39:19.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:19.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:19.488542+0000) 2020-02-05T22:39:20.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:20.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.095404+0000) 2020-02-05T22:39:20.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:20.190+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.191246+0000) 2020-02-05T22:39:20.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:20.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.192187+0000) 2020-02-05T22:39:20.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:20.191+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.192746+0000) 2020-02-05T22:39:20.194 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:20.192+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.193088+0000) 2020-02-05T22:39:20.194 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:20.192+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.193502+0000) 2020-02-05T22:39:20.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:20.192+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.193725+0000) 2020-02-05T22:39:20.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:20.193+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.194256+0000) 2020-02-05T22:39:20.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:20.194+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.194477+0000) 2020-02-05T22:39:20.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:20.193+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.194942+0000) 2020-02-05T22:39:20.196 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:20.195+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.195908+0000) 2020-02-05T22:39:20.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:20.195+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.196280+0000) 2020-02-05T22:39:20.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:20.196+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.197269+0000) 2020-02-05T22:39:20.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:20.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.197630+0000) 2020-02-05T22:39:20.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:20.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.198541+0000) 2020-02-05T22:39:20.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:20.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:20.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:20.331+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.331109+0000) 2020-02-05T22:39:20.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:20.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.347117+0000) 2020-02-05T22:39:20.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:20.814+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:20.815120+0000) 2020-02-05T22:39:22.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:22.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:22.345197+0000) 2020-02-05T22:39:22.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:22.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:22.738143+0000) 2020-02-05T22:39:23.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:23.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:23.858831+0000) 2020-02-05T22:39:23.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:23.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:23.858831+0000) 2020-02-05T22:39:25.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:25.211+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.212141+0000) 2020-02-05T22:39:25.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:25.211+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.212519+0000) 2020-02-05T22:39:25.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:25.212+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.213218+0000) 2020-02-05T22:39:25.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:25.212+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.213696+0000) 2020-02-05T22:39:25.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:25.213+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.214672+0000) 2020-02-05T22:39:25.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:25.214+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.215603+0000) 2020-02-05T22:39:25.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:25.214+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.216080+0000) 2020-02-05T22:39:25.217 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:25.215+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.216176+0000) 2020-02-05T22:39:25.218 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:25.215+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.216668+0000) 2020-02-05T22:39:25.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:25.216+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.217166+0000) 2020-02-05T22:39:25.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:25.215+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.217351+0000) 2020-02-05T22:39:25.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:25.216+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.217902+0000) 2020-02-05T22:39:25.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:25.219+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.220754+0000) 2020-02-05T22:39:25.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:25.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.221875+0000) 2020-02-05T22:39:25.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:25.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:25.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:25.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.951400+0000) 2020-02-05T22:39:25.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:25.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:25.960145+0000) 2020-02-05T22:39:27.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:27.098+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:27.099768+0000) 2020-02-05T22:39:27.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:27.098+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:27.099766+0000) 2020-02-05T22:39:27.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:27.098+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:27.099892+0000) 2020-02-05T22:39:29.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:29.487+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:29.488781+0000) 2020-02-05T22:39:30.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:30.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.095630+0000) 2020-02-05T22:39:30.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:30.233+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.234541+0000) 2020-02-05T22:39:30.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:30.234+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.235783+0000) 2020-02-05T22:39:30.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:30.234+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.236001+0000) 2020-02-05T22:39:30.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:30.235+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.237258+0000) 2020-02-05T22:39:30.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:30.236+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.237977+0000) 2020-02-05T22:39:30.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:30.237+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.238624+0000) 2020-02-05T22:39:30.239 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:30.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.238984+0000) 2020-02-05T22:39:30.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:30.237+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.239341+0000) 2020-02-05T22:39:30.240 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:30.238+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.239849+0000) 2020-02-05T22:39:30.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:30.238+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.239930+0000) 2020-02-05T22:39:30.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:30.239+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.240635+0000) 2020-02-05T22:39:30.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:30.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.240736+0000) 2020-02-05T22:39:30.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:30.240+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.241418+0000) 2020-02-05T22:39:30.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:30.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.247330+0000) 2020-02-05T22:39:30.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:30.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:30.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:30.330+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.331338+0000) 2020-02-05T22:39:30.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:30.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.347353+0000) 2020-02-05T22:39:30.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:30.822+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:30.823145+0000) 2020-02-05T22:39:32.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:32.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:32.345475+0000) 2020-02-05T22:39:32.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:32.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:32.738384+0000) 2020-02-05T22:39:33.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:33.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:33.859131+0000) 2020-02-05T22:39:33.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:33.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:33.859132+0000) 2020-02-05T22:39:34.975 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:39:35.003 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:39:35.257 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:35.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.256015+0000) 2020-02-05T22:39:35.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:35.255+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.256622+0000) 2020-02-05T22:39:35.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:35.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.257177+0000) 2020-02-05T22:39:35.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:35.256+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.257612+0000) 2020-02-05T22:39:35.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:35.257+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.258423+0000) 2020-02-05T22:39:35.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:35.256+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.257886+0000) 2020-02-05T22:39:35.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:35.257+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.258888+0000) 2020-02-05T22:39:35.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:35.259+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.260330+0000) 2020-02-05T22:39:35.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:35.258+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.259949+0000) 2020-02-05T22:39:35.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:35.258+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.260017+0000) 2020-02-05T22:39:35.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:35.260+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.261486+0000) 2020-02-05T22:39:35.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:35.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.262041+0000) 2020-02-05T22:39:35.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:35.260+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.261948+0000) 2020-02-05T22:39:35.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:35.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.268199+0000) 2020-02-05T22:39:35.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:35.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:35.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:35.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.951609+0000) 2020-02-05T22:39:35.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:35.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:35.960389+0000) 2020-02-05T22:39:37.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:37.098+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:37.100045+0000) 2020-02-05T22:39:37.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:37.098+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:37.100049+0000) 2020-02-05T22:39:37.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:37.098+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:37.100220+0000) 2020-02-05T22:39:39.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:39.488+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:39.489017+0000) 2020-02-05T22:39:40.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:40.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.095871+0000) 2020-02-05T22:39:40.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:40.274+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.275271+0000) 2020-02-05T22:39:40.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:40.274+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.276119+0000) 2020-02-05T22:39:40.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:40.275+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.276602+0000) 2020-02-05T22:39:40.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:40.276+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.277704+0000) 2020-02-05T22:39:40.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:40.277+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.278297+0000) 2020-02-05T22:39:40.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:40.277+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.278972+0000) 2020-02-05T22:39:40.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:40.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.279470+0000) 2020-02-05T22:39:40.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:40.278+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.279846+0000) 2020-02-05T22:39:40.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:40.279+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.280150+0000) 2020-02-05T22:39:40.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:40.279+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.280455+0000) 2020-02-05T22:39:40.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:40.279+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.280516+0000) 2020-02-05T22:39:40.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:40.279+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.280908+0000) 2020-02-05T22:39:40.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:40.280+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.281487+0000) 2020-02-05T22:39:40.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:40.285+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.286886+0000) 2020-02-05T22:39:40.289 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:40.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:40.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:40.330+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.331530+0000) 2020-02-05T22:39:40.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:40.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.347551+0000) 2020-02-05T22:39:40.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:40.833+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:40.834418+0000) 2020-02-05T22:39:42.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:42.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:42.345733+0000) 2020-02-05T22:39:42.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:42.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:42.738609+0000) 2020-02-05T22:39:43.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:43.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:43.859418+0000) 2020-02-05T22:39:43.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:43.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:43.859419+0000) 2020-02-05T22:39:45.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:45.293+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.294558+0000) 2020-02-05T22:39:45.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:45.294+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.295384+0000) 2020-02-05T22:39:45.296 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:45.295+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.296122+0000) 2020-02-05T22:39:45.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:45.295+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.296397+0000) 2020-02-05T22:39:45.297 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:45.295+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.296921+0000) 2020-02-05T22:39:45.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:45.295+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.296821+0000) 2020-02-05T22:39:45.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:45.296+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.297387+0000) 2020-02-05T22:39:45.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:45.295+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.297235+0000) 2020-02-05T22:39:45.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:45.296+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.297856+0000) 2020-02-05T22:39:45.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:45.296+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.298392+0000) 2020-02-05T22:39:45.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:45.297+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.298763+0000) 2020-02-05T22:39:45.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:45.298+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.299811+0000) 2020-02-05T22:39:45.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:45.304+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.306068+0000) 2020-02-05T22:39:45.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:45.299+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.300668+0000) 2020-02-05T22:39:45.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:45.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121800 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:45.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:45.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.951859+0000) 2020-02-05T22:39:45.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:45.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:45.960650+0000) 2020-02-05T22:39:47.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:47.099+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:47.100344+0000) 2020-02-05T22:39:47.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:47.099+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:47.100344+0000) 2020-02-05T22:39:47.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:47.099+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:47.100507+0000) 2020-02-05T22:39:49.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:49.488+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:49.489296+0000) 2020-02-05T22:39:50.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:50.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.096052+0000) 2020-02-05T22:39:50.316 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:50.313+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.314909+0000) 2020-02-05T22:39:50.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:50.314+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.315475+0000) 2020-02-05T22:39:50.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:50.316+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.317396+0000) 2020-02-05T22:39:50.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:50.316+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.317835+0000) 2020-02-05T22:39:50.319 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:50.317+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.318374+0000) 2020-02-05T22:39:50.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:50.317+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.318500+0000) 2020-02-05T22:39:50.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:50.317+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.318545+0000) 2020-02-05T22:39:50.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:50.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.318861+0000) 2020-02-05T22:39:50.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:50.317+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.319212+0000) 2020-02-05T22:39:50.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:50.317+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.319311+0000) 2020-02-05T22:39:50.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:50.318+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.319811+0000) 2020-02-05T22:39:50.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:50.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.320774+0000) 2020-02-05T22:39:50.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:50.318+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.320193+0000) 2020-02-05T22:39:50.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:50.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.324820+0000) 2020-02-05T22:39:50.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:50.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:50.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:50.330+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.331683+0000) 2020-02-05T22:39:50.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:50.347+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.347768+0000) 2020-02-05T22:39:50.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:50.840+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:50.841432+0000) 2020-02-05T22:39:52.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:52.344+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:52.345975+0000) 2020-02-05T22:39:52.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:52.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:52.738898+0000) 2020-02-05T22:39:53.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:53.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:53.859684+0000) 2020-02-05T22:39:53.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:53.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:53.859684+0000) 2020-02-05T22:39:55.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:39:55.334+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.335974+0000) 2020-02-05T22:39:55.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:39:55.335+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.336295+0000) 2020-02-05T22:39:55.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:55.335+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.337279+0000) 2020-02-05T22:39:55.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:55.335+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.337022+0000) 2020-02-05T22:39:55.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:55.337+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.338450+0000) 2020-02-05T22:39:55.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:39:55.338+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.339924+0000) 2020-02-05T22:39:55.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:55.339+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.341112+0000) 2020-02-05T22:39:55.342 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:55.340+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.341314+0000) 2020-02-05T22:39:55.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:55.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.341710+0000) 2020-02-05T22:39:55.343 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:39:55.341+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.342135+0000) 2020-02-05T22:39:55.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:55.340+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.342221+0000) 2020-02-05T22:39:55.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:39:55.341+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.342687+0000) 2020-02-05T22:39:55.344 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:39:55.341+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.342408+0000) 2020-02-05T22:39:55.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:55.344+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.345937+0000) 2020-02-05T22:39:55.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:39:55.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:39:55.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:39:55.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.952071+0000) 2020-02-05T22:39:55.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:39:55.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:55.960886+0000) 2020-02-05T22:39:57.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:39:57.100+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:57.100686+0000) 2020-02-05T22:39:57.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:39:57.100+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:57.100769+0000) 2020-02-05T22:39:57.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:39:57.100+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:57.100685+0000) 2020-02-05T22:39:59.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:39:59.489+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:39:59.489528+0000) 2020-02-05T22:40:00.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:00.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.096333+0000) 2020-02-05T22:40:00.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:00.331+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.331892+0000) 2020-02-05T22:40:00.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:00.346+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.347990+0000) 2020-02-05T22:40:00.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:00.356+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.357987+0000) 2020-02-05T22:40:00.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:00.357+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.358471+0000) 2020-02-05T22:40:00.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:00.358+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.359246+0000) 2020-02-05T22:40:00.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:00.358+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.359609+0000) 2020-02-05T22:40:00.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:00.358+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.359517+0000) 2020-02-05T22:40:00.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:00.359+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.360353+0000) 2020-02-05T22:40:00.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:00.359+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.360521+0000) 2020-02-05T22:40:00.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:00.358+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.360137+0000) 2020-02-05T22:40:00.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:00.359+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.361307+0000) 2020-02-05T22:40:00.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:00.360+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.361892+0000) 2020-02-05T22:40:00.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:00.361+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.362152+0000) 2020-02-05T22:40:00.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:00.359+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.361310+0000) 2020-02-05T22:40:00.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:00.361+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.363170+0000) 2020-02-05T22:40:00.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:00.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.369287+0000) 2020-02-05T22:40:00.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:00.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:00.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:00.849+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:00.849536+0000) 2020-02-05T22:40:02.347 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:02.345+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:02.346271+0000) 2020-02-05T22:40:02.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:02.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:02.739167+0000) 2020-02-05T22:40:03.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:03.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:03.859955+0000) 2020-02-05T22:40:03.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:03.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:03.859958+0000) 2020-02-05T22:40:05.152 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:40:05.158 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:40:05.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:05.376+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.377938+0000) 2020-02-05T22:40:05.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:05.377+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.378397+0000) 2020-02-05T22:40:05.380 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:05.377+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.378628+0000) 2020-02-05T22:40:05.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:05.377+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.379316+0000) 2020-02-05T22:40:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:05.378+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.379153+0000) 2020-02-05T22:40:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:05.378+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.379640+0000) 2020-02-05T22:40:05.381 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:05.378+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.379985+0000) 2020-02-05T22:40:05.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:05.379+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.380424+0000) 2020-02-05T22:40:05.382 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:05.379+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.381068+0000) 2020-02-05T22:40:05.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:05.380+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.381933+0000) 2020-02-05T22:40:05.383 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:05.380+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.382109+0000) 2020-02-05T22:40:05.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:05.381+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.382542+0000) 2020-02-05T22:40:05.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:05.381+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.383177+0000) 2020-02-05T22:40:05.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:05.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.387245+0000) 2020-02-05T22:40:05.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:05.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 121962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:05.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:05.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.952284+0000) 2020-02-05T22:40:05.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:05.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:05.961120+0000) 2020-02-05T22:40:07.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:07.100+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:07.100988+0000) 2020-02-05T22:40:07.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:07.100+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:07.101023+0000) 2020-02-05T22:40:07.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:07.100+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:07.100989+0000) 2020-02-05T22:40:09.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:09.489+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:09.489760+0000) 2020-02-05T22:40:10.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:10.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.096562+0000) 2020-02-05T22:40:10.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:10.331+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.332175+0000) 2020-02-05T22:40:10.349 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:10.348+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.348218+0000) 2020-02-05T22:40:10.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:10.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.397450+0000) 2020-02-05T22:40:10.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:10.397+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.398797+0000) 2020-02-05T22:40:10.400 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:10.399+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.399354+0000) 2020-02-05T22:40:10.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:10.397+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.399146+0000) 2020-02-05T22:40:10.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:10.400+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.400344+0000) 2020-02-05T22:40:10.401 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:10.400+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.400441+0000) 2020-02-05T22:40:10.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:10.400+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.400770+0000) 2020-02-05T22:40:10.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:10.401+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.401286+0000) 2020-02-05T22:40:10.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:10.401+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.402899+0000) 2020-02-05T22:40:10.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:10.401+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.403222+0000) 2020-02-05T22:40:10.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:10.402+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.403667+0000) 2020-02-05T22:40:10.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:10.404+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.404278+0000) 2020-02-05T22:40:10.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:10.402+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.404378+0000) 2020-02-05T22:40:10.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:10.407+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.407834+0000) 2020-02-05T22:40:10.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:10.412+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122002 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:10.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:10.857+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:10.857499+0000) 2020-02-05T22:40:12.347 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:12.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:12.346517+0000) 2020-02-05T22:40:12.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:12.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:12.739450+0000) 2020-02-05T22:40:13.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:13.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:13.860230+0000) 2020-02-05T22:40:13.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:13.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:13.860232+0000) 2020-02-05T22:40:15.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:15.419+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.420139+0000) 2020-02-05T22:40:15.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:15.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.421493+0000) 2020-02-05T22:40:15.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:15.421+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.421525+0000) 2020-02-05T22:40:15.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:15.420+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.421832+0000) 2020-02-05T22:40:15.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:15.421+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.422364+0000) 2020-02-05T22:40:15.424 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:15.421+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.422747+0000) 2020-02-05T22:40:15.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:15.422+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.422946+0000) 2020-02-05T22:40:15.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:15.422+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.423252+0000) 2020-02-05T22:40:15.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:15.422+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.423369+0000) 2020-02-05T22:40:15.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:15.423+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.424194+0000) 2020-02-05T22:40:15.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:15.423+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.424125+0000) 2020-02-05T22:40:15.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:15.424+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.425526+0000) 2020-02-05T22:40:15.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:15.424+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.425403+0000) 2020-02-05T22:40:15.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:15.426+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.428014+0000) 2020-02-05T22:40:15.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:15.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:15.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:15.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.952488+0000) 2020-02-05T22:40:15.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:15.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:15.961290+0000) 2020-02-05T22:40:17.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:17.100+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:17.101269+0000) 2020-02-05T22:40:17.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:17.100+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:17.101275+0000) 2020-02-05T22:40:17.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:17.100+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:17.101275+0000) 2020-02-05T22:40:19.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:19.488+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:19.489985+0000) 2020-02-05T22:40:20.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:20.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.096776+0000) 2020-02-05T22:40:20.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:20.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.332473+0000) 2020-02-05T22:40:20.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:20.347+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.348447+0000) 2020-02-05T22:40:20.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:20.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.437666+0000) 2020-02-05T22:40:20.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:20.438+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.438414+0000) 2020-02-05T22:40:20.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:20.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.439023+0000) 2020-02-05T22:40:20.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:20.439+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.439817+0000) 2020-02-05T22:40:20.441 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:20.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.440172+0000) 2020-02-05T22:40:20.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:20.440+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.441304+0000) 2020-02-05T22:40:20.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:20.440+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.441550+0000) 2020-02-05T22:40:20.443 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:20.440+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.441978+0000) 2020-02-05T22:40:20.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:20.441+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.442296+0000) 2020-02-05T22:40:20.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:20.441+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.442648+0000) 2020-02-05T22:40:20.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:20.442+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.443085+0000) 2020-02-05T22:40:20.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:20.443+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.444368+0000) 2020-02-05T22:40:20.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:20.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.444624+0000) 2020-02-05T22:40:20.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:20.448+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.449201+0000) 2020-02-05T22:40:20.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:20.452+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:20.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:20.864+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:20.865667+0000) 2020-02-05T22:40:22.348 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:22.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:22.346754+0000) 2020-02-05T22:40:22.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:22.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:22.739675+0000) 2020-02-05T22:40:23.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:23.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:23.860497+0000) 2020-02-05T22:40:23.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:23.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:23.860498+0000) 2020-02-05T22:40:25.459 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:25.456+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.458033+0000) 2020-02-05T22:40:25.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:25.459+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.460448+0000) 2020-02-05T22:40:25.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:25.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.461049+0000) 2020-02-05T22:40:25.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:25.460+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.461256+0000) 2020-02-05T22:40:25.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:25.460+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.461628+0000) 2020-02-05T22:40:25.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:25.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.462039+0000) 2020-02-05T22:40:25.463 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:25.462+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.462645+0000) 2020-02-05T22:40:25.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:25.462+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.462901+0000) 2020-02-05T22:40:25.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:25.462+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.463228+0000) 2020-02-05T22:40:25.465 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:25.463+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.464223+0000) 2020-02-05T22:40:25.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:25.463+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.464321+0000) 2020-02-05T22:40:25.466 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:25.463+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.464995+0000) 2020-02-05T22:40:25.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:25.463+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.465039+0000) 2020-02-05T22:40:25.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:25.468+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.469296+0000) 2020-02-05T22:40:25.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:25.474+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122124 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:25.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:25.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.952736+0000) 2020-02-05T22:40:25.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:25.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:25.961503+0000) 2020-02-05T22:40:27.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:27.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:27.101547+0000) 2020-02-05T22:40:27.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:27.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:27.101519+0000) 2020-02-05T22:40:27.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:27.101+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:27.101549+0000) 2020-02-05T22:40:29.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:29.489+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:29.490234+0000) 2020-02-05T22:40:30.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:30.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.097011+0000) 2020-02-05T22:40:30.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:30.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.332708+0000) 2020-02-05T22:40:30.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:30.348+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.348760+0000) 2020-02-05T22:40:30.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:30.479+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.480127+0000) 2020-02-05T22:40:30.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:30.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.480743+0000) 2020-02-05T22:40:30.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:30.481+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.482180+0000) 2020-02-05T22:40:30.484 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:30.480+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.481234+0000) 2020-02-05T22:40:30.484 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:30.482+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.483272+0000) 2020-02-05T22:40:30.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:30.483+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.484729+0000) 2020-02-05T22:40:30.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:30.484+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.485155+0000) 2020-02-05T22:40:30.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:30.484+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.485032+0000) 2020-02-05T22:40:30.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:30.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.485635+0000) 2020-02-05T22:40:30.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:30.485+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.486266+0000) 2020-02-05T22:40:30.488 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:30.485+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.485745+0000) 2020-02-05T22:40:30.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:30.486+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.486508+0000) 2020-02-05T22:40:30.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:30.486+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.486606+0000) 2020-02-05T22:40:30.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:30.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.491427+0000) 2020-02-05T22:40:30.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:30.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:30.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:30.876+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:30.877434+0000) 2020-02-05T22:40:32.348 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:32.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:32.347039+0000) 2020-02-05T22:40:32.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:32.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:32.739908+0000) 2020-02-05T22:40:33.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:33.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:33.860766+0000) 2020-02-05T22:40:33.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:33.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:33.860766+0000) 2020-02-05T22:40:35.311 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:40:35.317 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:40:35.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:35.498+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.499250+0000) 2020-02-05T22:40:35.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:35.498+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.499853+0000) 2020-02-05T22:40:35.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:35.500+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.501348+0000) 2020-02-05T22:40:35.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:35.500+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.501878+0000) 2020-02-05T22:40:35.503 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:35.500+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.501146+0000) 2020-02-05T22:40:35.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:35.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.502787+0000) 2020-02-05T22:40:35.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:35.502+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.503387+0000) 2020-02-05T22:40:35.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:35.503+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.503911+0000) 2020-02-05T22:40:35.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:35.503+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.504062+0000) 2020-02-05T22:40:35.506 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:35.504+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.505102+0000) 2020-02-05T22:40:35.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:35.504+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.505396+0000) 2020-02-05T22:40:35.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:35.505+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.505755+0000) 2020-02-05T22:40:35.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:35.506+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.506480+0000) 2020-02-05T22:40:35.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:35.512+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.513161+0000) 2020-02-05T22:40:35.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:35.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:35.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:35.952+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.952918+0000) 2020-02-05T22:40:35.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:35.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:35.961753+0000) 2020-02-05T22:40:37.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:37.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:37.101755+0000) 2020-02-05T22:40:37.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:37.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:37.101825+0000) 2020-02-05T22:40:37.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:37.101+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:37.101828+0000) 2020-02-05T22:40:39.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:39.489+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:39.490469+0000) 2020-02-05T22:40:40.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:40.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.097283+0000) 2020-02-05T22:40:40.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:40.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.332962+0000) 2020-02-05T22:40:40.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:40.347+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.349007+0000) 2020-02-05T22:40:40.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:40.519+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.520470+0000) 2020-02-05T22:40:40.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:40.520+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.521194+0000) 2020-02-05T22:40:40.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:40.521+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.521706+0000) 2020-02-05T22:40:40.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:40.520+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.521995+0000) 2020-02-05T22:40:40.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:40.522+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.522516+0000) 2020-02-05T22:40:40.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:40.522+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.523512+0000) 2020-02-05T22:40:40.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:40.523+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.524333+0000) 2020-02-05T22:40:40.527 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:40.525+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.525912+0000) 2020-02-05T22:40:40.527 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:40.526+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.526494+0000) 2020-02-05T22:40:40.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:40.526+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.526788+0000) 2020-02-05T22:40:40.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:40.526+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.527233+0000) 2020-02-05T22:40:40.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:40.526+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.527208+0000) 2020-02-05T22:40:40.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:40.527+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.528726+0000) 2020-02-05T22:40:40.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:40.528+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.529808+0000) 2020-02-05T22:40:40.536 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:40.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:40.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:40.884+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:40.885512+0000) 2020-02-05T22:40:42.348 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:42.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:42.347348+0000) 2020-02-05T22:40:42.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:42.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:42.740127+0000) 2020-02-05T22:40:43.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:43.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:43.861046+0000) 2020-02-05T22:40:43.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:43.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:43.861046+0000) 2020-02-05T22:40:45.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:45.539+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.540216+0000) 2020-02-05T22:40:45.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:45.540+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.541209+0000) 2020-02-05T22:40:45.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:45.540+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.542020+0000) 2020-02-05T22:40:45.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:45.541+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.542850+0000) 2020-02-05T22:40:45.544 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:45.543+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.543511+0000) 2020-02-05T22:40:45.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:45.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.544604+0000) 2020-02-05T22:40:45.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:45.543+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.544360+0000) 2020-02-05T22:40:45.546 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:45.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.545157+0000) 2020-02-05T22:40:45.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:45.545+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.546268+0000) 2020-02-05T22:40:45.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:45.545+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.546362+0000) 2020-02-05T22:40:45.548 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:45.546+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.546928+0000) 2020-02-05T22:40:45.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:45.546+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.547008+0000) 2020-02-05T22:40:45.549 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:45.547+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.547687+0000) 2020-02-05T22:40:45.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:45.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.552866+0000) 2020-02-05T22:40:45.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:45.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:45.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:45.952+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.953132+0000) 2020-02-05T22:40:45.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:45.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:45.961955+0000) 2020-02-05T22:40:47.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:47.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:47.102025+0000) 2020-02-05T22:40:47.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:47.101+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:47.102100+0000) 2020-02-05T22:40:47.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:47.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:47.102100+0000) 2020-02-05T22:40:49.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:49.490+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:49.490703+0000) 2020-02-05T22:40:50.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:50.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.097505+0000) 2020-02-05T22:40:50.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:50.333+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.333213+0000) 2020-02-05T22:40:50.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:50.348+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.349267+0000) 2020-02-05T22:40:50.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:50.560+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.561115+0000) 2020-02-05T22:40:50.562 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:50.560+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.561725+0000) 2020-02-05T22:40:50.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:50.562+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.562712+0000) 2020-02-05T22:40:50.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:50.562+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.563208+0000) 2020-02-05T22:40:50.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:50.563+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.564162+0000) 2020-02-05T22:40:50.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:50.563+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.564945+0000) 2020-02-05T22:40:50.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:50.564+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.565394+0000) 2020-02-05T22:40:50.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:50.565+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.565565+0000) 2020-02-05T22:40:50.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:50.564+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.565737+0000) 2020-02-05T22:40:50.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:50.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.566340+0000) 2020-02-05T22:40:50.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:50.566+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.566517+0000) 2020-02-05T22:40:50.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:50.566+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.566588+0000) 2020-02-05T22:40:50.571 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:50.566+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.567137+0000) 2020-02-05T22:40:50.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:50.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.570890+0000) 2020-02-05T22:40:50.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:50.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:50.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:50.893+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:50.894058+0000) 2020-02-05T22:40:52.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:52.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:52.347762+0000) 2020-02-05T22:40:52.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:52.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:52.740380+0000) 2020-02-05T22:40:53.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:53.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:53.861363+0000) 2020-02-05T22:40:53.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:53.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:53.861362+0000) 2020-02-05T22:40:55.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:55.581+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.582303+0000) 2020-02-05T22:40:55.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:40:55.581+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.582807+0000) 2020-02-05T22:40:55.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:55.582+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.583675+0000) 2020-02-05T22:40:55.585 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:40:55.583+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.584263+0000) 2020-02-05T22:40:55.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:55.583+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.584707+0000) 2020-02-05T22:40:55.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:55.584+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.585495+0000) 2020-02-05T22:40:55.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:40:55.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.587215+0000) 2020-02-05T22:40:55.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:55.586+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.587473+0000) 2020-02-05T22:40:55.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:40:55.586+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.587633+0000) 2020-02-05T22:40:55.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:40:55.586+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.588102+0000) 2020-02-05T22:40:55.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:40:55.587+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.588524+0000) 2020-02-05T22:40:55.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:55.587+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.588570+0000) 2020-02-05T22:40:55.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:55.588+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.589675+0000) 2020-02-05T22:40:55.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:55.592+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.593150+0000) 2020-02-05T22:40:55.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:40:55.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122361 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:40:55.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:40:55.953+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.953342+0000) 2020-02-05T22:40:55.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:40:55.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:55.962181+0000) 2020-02-05T22:40:57.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:40:57.100+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:57.102235+0000) 2020-02-05T22:40:57.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:40:57.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:57.102420+0000) 2020-02-05T22:40:57.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:40:57.100+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:57.102367+0000) 2020-02-05T22:40:59.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:40:59.490+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:40:59.490939+0000) 2020-02-05T22:41:00.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:00.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.097775+0000) 2020-02-05T22:41:00.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:00.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.333520+0000) 2020-02-05T22:41:00.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:00.349+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.349496+0000) 2020-02-05T22:41:00.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:00.603+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.604686+0000) 2020-02-05T22:41:00.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:00.603+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.605409+0000) 2020-02-05T22:41:00.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:00.604+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.605163+0000) 2020-02-05T22:41:00.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:00.605+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.606427+0000) 2020-02-05T22:41:00.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:00.605+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.606648+0000) 2020-02-05T22:41:00.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:00.606+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.607679+0000) 2020-02-05T22:41:00.609 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:00.606+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.608166+0000) 2020-02-05T22:41:00.609 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:00.607+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.608591+0000) 2020-02-05T22:41:00.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:00.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.609189+0000) 2020-02-05T22:41:00.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:00.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.609536+0000) 2020-02-05T22:41:00.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:00.608+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.609557+0000) 2020-02-05T22:41:00.611 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:00.608+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.609818+0000) 2020-02-05T22:41:00.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:00.609+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.610446+0000) 2020-02-05T22:41:00.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:00.614+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.615891+0000) 2020-02-05T22:41:00.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:00.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:00.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:00.905+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:00.905431+0000) 2020-02-05T22:41:02.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:02.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:02.348001+0000) 2020-02-05T22:41:02.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:02.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:02.740626+0000) 2020-02-05T22:41:03.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:03.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:03.861631+0000) 2020-02-05T22:41:03.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:03.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:03.861635+0000) 2020-02-05T22:41:05.440 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:41:05.446 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:41:05.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:05.622+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.622662+0000) 2020-02-05T22:41:05.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:05.623+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.623898+0000) 2020-02-05T22:41:05.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:05.624+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.625994+0000) 2020-02-05T22:41:05.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:05.626+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.626538+0000) 2020-02-05T22:41:05.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:05.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.627171+0000) 2020-02-05T22:41:05.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:05.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.627860+0000) 2020-02-05T22:41:05.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:05.628+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.628863+0000) 2020-02-05T22:41:05.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:05.627+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.629342+0000) 2020-02-05T22:41:05.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:05.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.629341+0000) 2020-02-05T22:41:05.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:05.629+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.629525+0000) 2020-02-05T22:41:05.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:05.629+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.629757+0000) 2020-02-05T22:41:05.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:05.628+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.629992+0000) 2020-02-05T22:41:05.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:05.628+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.630286+0000) 2020-02-05T22:41:05.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:05.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.634288+0000) 2020-02-05T22:41:05.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:05.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:05.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:05.953+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.953572+0000) 2020-02-05T22:41:05.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:05.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:05.962469+0000) 2020-02-05T22:41:07.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:07.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:07.102482+0000) 2020-02-05T22:41:07.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:07.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:07.102700+0000) 2020-02-05T22:41:07.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:07.101+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:07.102685+0000) 2020-02-05T22:41:09.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:09.490+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:09.491176+0000) 2020-02-05T22:41:10.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:10.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.098107+0000) 2020-02-05T22:41:10.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:10.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.333758+0000) 2020-02-05T22:41:10.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:10.348+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.349716+0000) 2020-02-05T22:41:10.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:10.642+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.643546+0000) 2020-02-05T22:41:10.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:10.643+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.644157+0000) 2020-02-05T22:41:10.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:10.643+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.645065+0000) 2020-02-05T22:41:10.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:10.644+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.645610+0000) 2020-02-05T22:41:10.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:10.645+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.646755+0000) 2020-02-05T22:41:10.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:10.646+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.647565+0000) 2020-02-05T22:41:10.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:10.647+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.648583+0000) 2020-02-05T22:41:10.650 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:10.646+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.648362+0000) 2020-02-05T22:41:10.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:10.648+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.649130+0000) 2020-02-05T22:41:10.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:10.648+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.649552+0000) 2020-02-05T22:41:10.651 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:10.648+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.649927+0000) 2020-02-05T22:41:10.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:10.648+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.650041+0000) 2020-02-05T22:41:10.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:10.649+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.650443+0000) 2020-02-05T22:41:10.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:10.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.650904+0000) 2020-02-05T22:41:10.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:10.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122480 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:10.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:10.912+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:10.913666+0000) 2020-02-05T22:41:12.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:12.346+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:12.348241+0000) 2020-02-05T22:41:12.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:12.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:12.740883+0000) 2020-02-05T22:41:13.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:13.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:13.861903+0000) 2020-02-05T22:41:13.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:13.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:13.861903+0000) 2020-02-05T22:41:15.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:15.662+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.663871+0000) 2020-02-05T22:41:15.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:15.664+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.665724+0000) 2020-02-05T22:41:15.667 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:15.665+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.666226+0000) 2020-02-05T22:41:15.668 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:15.665+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.666509+0000) 2020-02-05T22:41:15.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:15.665+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.666899+0000) 2020-02-05T22:41:15.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:15.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.667421+0000) 2020-02-05T22:41:15.669 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:15.666+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.667801+0000) 2020-02-05T22:41:15.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:15.667+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.668300+0000) 2020-02-05T22:41:15.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:15.667+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.668515+0000) 2020-02-05T22:41:15.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:15.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.670190+0000) 2020-02-05T22:41:15.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:15.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.670207+0000) 2020-02-05T22:41:15.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:15.668+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.670407+0000) 2020-02-05T22:41:15.672 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:15.669+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.670491+0000) 2020-02-05T22:41:15.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:15.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.676979+0000) 2020-02-05T22:41:15.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:15.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:15.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:15.952+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.953777+0000) 2020-02-05T22:41:15.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:15.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:15.962725+0000) 2020-02-05T22:41:17.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:17.102+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:17.102781+0000) 2020-02-05T22:41:17.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:17.102+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:17.102983+0000) 2020-02-05T22:41:17.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:17.102+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:17.102983+0000) 2020-02-05T22:41:19.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:19.490+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:19.491370+0000) 2020-02-05T22:41:20.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:20.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.098301+0000) 2020-02-05T22:41:20.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:20.333+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.334098+0000) 2020-02-05T22:41:20.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:20.349+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.349949+0000) 2020-02-05T22:41:20.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:20.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.684322+0000) 2020-02-05T22:41:20.686 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:20.684+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.685415+0000) 2020-02-05T22:41:20.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:20.684+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.685855+0000) 2020-02-05T22:41:20.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:20.684+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.686253+0000) 2020-02-05T22:41:20.687 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:20.685+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.686916+0000) 2020-02-05T22:41:20.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:20.685+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.687082+0000) 2020-02-05T22:41:20.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:20.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.687912+0000) 2020-02-05T22:41:20.689 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:20.686+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.688360+0000) 2020-02-05T22:41:20.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:20.688+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.689241+0000) 2020-02-05T22:41:20.690 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:20.688+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.689661+0000) 2020-02-05T22:41:20.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:20.688+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.689681+0000) 2020-02-05T22:41:20.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:20.688+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.690274+0000) 2020-02-05T22:41:20.691 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:20.689+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.690584+0000) 2020-02-05T22:41:20.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:20.695+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.696353+0000) 2020-02-05T22:41:20.699 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:20.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122560 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:20.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:20.920+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:20.921730+0000) 2020-02-05T22:41:22.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:22.347+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:22.348508+0000) 2020-02-05T22:41:22.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:22.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:22.741145+0000) 2020-02-05T22:41:23.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:23.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:23.862198+0000) 2020-02-05T22:41:23.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:23.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:23.862194+0000) 2020-02-05T22:41:25.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:25.705+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.706413+0000) 2020-02-05T22:41:25.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:25.706+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.707291+0000) 2020-02-05T22:41:25.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:25.706+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.707711+0000) 2020-02-05T22:41:25.709 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:25.707+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.708448+0000) 2020-02-05T22:41:25.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:25.707+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.708563+0000) 2020-02-05T22:41:25.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:25.707+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.709041+0000) 2020-02-05T22:41:25.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:25.708+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.710002+0000) 2020-02-05T22:41:25.711 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:25.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.710487+0000) 2020-02-05T22:41:25.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:25.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.710843+0000) 2020-02-05T22:41:25.712 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:25.709+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.710778+0000) 2020-02-05T22:41:25.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:25.709+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.711002+0000) 2020-02-05T22:41:25.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:25.710+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.711537+0000) 2020-02-05T22:41:25.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:25.710+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.711643+0000) 2020-02-05T22:41:25.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:25.714+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.715617+0000) 2020-02-05T22:41:25.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:25.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122601 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:25.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:25.953+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.954001+0000) 2020-02-05T22:41:25.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:25.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:25.962957+0000) 2020-02-05T22:41:27.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:27.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:27.102987+0000) 2020-02-05T22:41:27.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:27.101+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:27.103269+0000) 2020-02-05T22:41:27.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:27.101+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:27.103273+0000) 2020-02-05T22:41:29.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:29.490+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:29.491607+0000) 2020-02-05T22:41:30.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:30.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.098543+0000) 2020-02-05T22:41:30.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:30.332+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.334388+0000) 2020-02-05T22:41:30.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:30.349+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.350179+0000) 2020-02-05T22:41:30.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:30.723+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.724793+0000) 2020-02-05T22:41:30.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:30.724+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.726405+0000) 2020-02-05T22:41:30.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:30.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.727371+0000) 2020-02-05T22:41:30.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:30.726+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.727862+0000) 2020-02-05T22:41:30.729 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:30.727+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.728264+0000) 2020-02-05T22:41:30.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:30.727+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.728632+0000) 2020-02-05T22:41:30.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:30.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.729169+0000) 2020-02-05T22:41:30.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:30.728+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.729614+0000) 2020-02-05T22:41:30.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:30.728+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.729800+0000) 2020-02-05T22:41:30.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:30.728+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.729885+0000) 2020-02-05T22:41:30.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:30.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.730146+0000) 2020-02-05T22:41:30.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:30.729+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.730833+0000) 2020-02-05T22:41:30.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:30.729+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.731325+0000) 2020-02-05T22:41:30.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:30.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.737990+0000) 2020-02-05T22:41:30.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:30.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:30.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:30.928+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:30.929572+0000) 2020-02-05T22:41:32.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:32.348+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:32.348808+0000) 2020-02-05T22:41:32.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:32.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:32.741349+0000) 2020-02-05T22:41:33.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:33.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:33.862480+0000) 2020-02-05T22:41:33.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:33.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:33.862481+0000) 2020-02-05T22:41:35.569 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:41:35.576 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:41:35.748 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:35.745+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.747394+0000) 2020-02-05T22:41:35.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:35.746+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.747955+0000) 2020-02-05T22:41:35.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:35.747+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.748854+0000) 2020-02-05T22:41:35.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:35.748+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.749701+0000) 2020-02-05T22:41:35.751 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:35.747+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.749367+0000) 2020-02-05T22:41:35.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:35.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.750132+0000) 2020-02-05T22:41:35.752 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:35.749+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.750629+0000) 2020-02-05T22:41:35.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:35.749+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.750840+0000) 2020-02-05T22:41:35.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:35.750+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.751259+0000) 2020-02-05T22:41:35.753 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:35.750+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.751464+0000) 2020-02-05T22:41:35.753 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:35.750+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.751851+0000) 2020-02-05T22:41:35.753 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:35.751+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.752608+0000) 2020-02-05T22:41:35.754 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:35.751+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.752576+0000) 2020-02-05T22:41:35.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:35.759+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.760468+0000) 2020-02-05T22:41:35.763 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:35.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:35.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:35.953+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.954233+0000) 2020-02-05T22:41:35.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:35.962+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:35.963190+0000) 2020-02-05T22:41:37.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:37.101+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:37.103198+0000) 2020-02-05T22:41:37.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:37.102+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:37.103533+0000) 2020-02-05T22:41:37.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:37.102+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:37.103606+0000) 2020-02-05T22:41:39.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:39.491+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:39.491851+0000) 2020-02-05T22:41:40.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:40.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.098766+0000) 2020-02-05T22:41:40.336 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:40.334+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.334768+0000) 2020-02-05T22:41:40.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:40.349+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.350445+0000) 2020-02-05T22:41:40.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:40.765+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.766756+0000) 2020-02-05T22:41:40.769 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:40.767+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.768788+0000) 2020-02-05T22:41:40.770 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:40.767+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.768973+0000) 2020-02-05T22:41:40.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:40.768+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.769816+0000) 2020-02-05T22:41:40.771 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:40.769+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.770420+0000) 2020-02-05T22:41:40.771 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:40.769+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.770697+0000) 2020-02-05T22:41:40.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:40.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.771358+0000) 2020-02-05T22:41:40.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:40.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.771608+0000) 2020-02-05T22:41:40.773 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:40.770+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.772241+0000) 2020-02-05T22:41:40.774 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:40.770+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.772123+0000) 2020-02-05T22:41:40.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:40.770+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.772044+0000) 2020-02-05T22:41:40.774 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:40.771+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.773384+0000) 2020-02-05T22:41:40.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:40.772+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.773611+0000) 2020-02-05T22:41:40.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:40.776+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.777880+0000) 2020-02-05T22:41:40.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:40.780+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:40.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:40.937+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:40.937817+0000) 2020-02-05T22:41:42.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:42.348+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:42.349076+0000) 2020-02-05T22:41:42.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:42.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:42.741626+0000) 2020-02-05T22:41:43.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:43.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:43.862766+0000) 2020-02-05T22:41:43.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:43.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:43.862767+0000) 2020-02-05T22:41:45.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:45.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.788458+0000) 2020-02-05T22:41:45.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:45.788+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.789421+0000) 2020-02-05T22:41:45.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:45.788+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.789713+0000) 2020-02-05T22:41:45.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:45.789+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.790042+0000) 2020-02-05T22:41:45.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:45.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.790457+0000) 2020-02-05T22:41:45.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:45.790+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.791368+0000) 2020-02-05T22:41:45.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:45.790+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.791165+0000) 2020-02-05T22:41:45.793 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:45.791+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.791636+0000) 2020-02-05T22:41:45.794 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:45.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.792684+0000) 2020-02-05T22:41:45.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:45.791+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.792888+0000) 2020-02-05T22:41:45.795 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:45.793+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.793532+0000) 2020-02-05T22:41:45.795 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:45.793+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.793452+0000) 2020-02-05T22:41:45.795 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:45.793+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.794385+0000) 2020-02-05T22:41:45.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:45.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.798593+0000) 2020-02-05T22:41:45.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:45.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:45.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:45.954+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.954459+0000) 2020-02-05T22:41:45.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:45.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:45.963414+0000) 2020-02-05T22:41:47.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:47.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:47.103543+0000) 2020-02-05T22:41:47.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:47.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:47.103850+0000) 2020-02-05T22:41:47.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:47.103+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:47.103876+0000) 2020-02-05T22:41:49.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:49.491+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:49.492117+0000) 2020-02-05T22:41:50.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:50.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.099018+0000) 2020-02-05T22:41:50.336 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:50.334+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.335021+0000) 2020-02-05T22:41:50.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:50.350+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.350674+0000) 2020-02-05T22:41:50.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:50.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.807463+0000) 2020-02-05T22:41:50.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:50.807+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.808421+0000) 2020-02-05T22:41:50.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:50.809+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.810616+0000) 2020-02-05T22:41:50.812 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:50.810+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.810418+0000) 2020-02-05T22:41:50.812 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:50.810+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.811120+0000) 2020-02-05T22:41:50.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:50.810+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.811681+0000) 2020-02-05T22:41:50.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:50.811+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.812145+0000) 2020-02-05T22:41:50.813 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:50.812+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.812588+0000) 2020-02-05T22:41:50.814 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:50.812+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.813213+0000) 2020-02-05T22:41:50.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:50.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.813493+0000) 2020-02-05T22:41:50.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:50.813+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.813742+0000) 2020-02-05T22:41:50.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:50.813+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.814264+0000) 2020-02-05T22:41:50.815 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:50.813+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.814304+0000) 2020-02-05T22:41:50.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:50.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.818720+0000) 2020-02-05T22:41:50.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:50.822+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122800 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:50.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:50.944+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:50.944757+0000) 2020-02-05T22:41:52.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:52.348+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:52.349314+0000) 2020-02-05T22:41:52.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:52.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:52.741863+0000) 2020-02-05T22:41:53.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:53.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:53.863037+0000) 2020-02-05T22:41:53.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:53.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:53.863037+0000) 2020-02-05T22:41:55.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:55.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.827396+0000) 2020-02-05T22:41:55.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:41:55.827+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.828667+0000) 2020-02-05T22:41:55.830 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:41:55.828+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.829278+0000) 2020-02-05T22:41:55.831 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:41:55.830+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.830419+0000) 2020-02-05T22:41:55.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:55.829+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.830489+0000) 2020-02-05T22:41:55.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:55.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.830922+0000) 2020-02-05T22:41:55.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:41:55.830+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831198+0000) 2020-02-05T22:41:55.833 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:41:55.830+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831445+0000) 2020-02-05T22:41:55.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:41:55.830+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831739+0000) 2020-02-05T22:41:55.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:55.830+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831258+0000) 2020-02-05T22:41:55.835 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:55.831+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831686+0000) 2020-02-05T22:41:55.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:55.831+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.831925+0000) 2020-02-05T22:41:55.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:55.832+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.832633+0000) 2020-02-05T22:41:55.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:55.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.835855+0000) 2020-02-05T22:41:55.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:41:55.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:41:55.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:41:55.954+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.954672+0000) 2020-02-05T22:41:55.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:41:55.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:55.963638+0000) 2020-02-05T22:41:57.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:41:57.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:57.103785+0000) 2020-02-05T22:41:57.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:41:57.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:57.104162+0000) 2020-02-05T22:41:57.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:41:57.103+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:57.104114+0000) 2020-02-05T22:41:59.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:41:59.492+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:41:59.492386+0000) 2020-02-05T22:42:00.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:00.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.099280+0000) 2020-02-05T22:42:00.336 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:00.334+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.335268+0000) 2020-02-05T22:42:00.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:00.350+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.350907+0000) 2020-02-05T22:42:00.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:00.849+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.849801+0000) 2020-02-05T22:42:00.851 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:00.849+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.849716+0000) 2020-02-05T22:42:00.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:00.851+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.851491+0000) 2020-02-05T22:42:00.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:00.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.852049+0000) 2020-02-05T22:42:00.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:00.851+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.852253+0000) 2020-02-05T22:42:00.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:00.852+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.852798+0000) 2020-02-05T22:42:00.854 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:00.853+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.853251+0000) 2020-02-05T22:42:00.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:00.853+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.853650+0000) 2020-02-05T22:42:00.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:00.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.854063+0000) 2020-02-05T22:42:00.856 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:00.853+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.854265+0000) 2020-02-05T22:42:00.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:00.853+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.854235+0000) 2020-02-05T22:42:00.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:00.854+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.854736+0000) 2020-02-05T22:42:00.856 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:00.854+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.855394+0000) 2020-02-05T22:42:00.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:00.860+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.860227+0000) 2020-02-05T22:42:00.866 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:00.864+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122882 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:00.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:00.951+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:00.951831+0000) 2020-02-05T22:42:02.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:02.349+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:02.349565+0000) 2020-02-05T22:42:02.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:02.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:02.742141+0000) 2020-02-05T22:42:03.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:03.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:03.863346+0000) 2020-02-05T22:42:03.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:03.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:03.863345+0000) 2020-02-05T22:42:05.717 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:42:05.750 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:42:05.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:05.869+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.870136+0000) 2020-02-05T22:42:05.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:05.869+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.870660+0000) 2020-02-05T22:42:05.879 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:05.869+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.870029+0000) 2020-02-05T22:42:05.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:05.870+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.871479+0000) 2020-02-05T22:42:05.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:05.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.872894+0000) 2020-02-05T22:42:05.880 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:05.870+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.871595+0000) 2020-02-05T22:42:05.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:05.870+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.871771+0000) 2020-02-05T22:42:05.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:05.872+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.872679+0000) 2020-02-05T22:42:05.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:05.872+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.873271+0000) 2020-02-05T22:42:05.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:05.873+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.874398+0000) 2020-02-05T22:42:05.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:05.873+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.874305+0000) 2020-02-05T22:42:05.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:05.874+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.874767+0000) 2020-02-05T22:42:05.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:05.872+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.873596+0000) 2020-02-05T22:42:05.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:05.878+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.879345+0000) 2020-02-05T22:42:05.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:05.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:05.966 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:05.953+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.954896+0000) 2020-02-05T22:42:05.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:05.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:05.963863+0000) 2020-02-05T22:42:07.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:07.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:07.104039+0000) 2020-02-05T22:42:07.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:07.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:07.104402+0000) 2020-02-05T22:42:07.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:07.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:07.104568+0000) 2020-02-05T22:42:09.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:09.491+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:09.492657+0000) 2020-02-05T22:42:10.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:10.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.099523+0000) 2020-02-05T22:42:10.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:10.335+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.335523+0000) 2020-02-05T22:42:10.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:10.350+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.351138+0000) 2020-02-05T22:42:10.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:10.890+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.891659+0000) 2020-02-05T22:42:10.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:10.890+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.891244+0000) 2020-02-05T22:42:10.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:10.892+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.893130+0000) 2020-02-05T22:42:10.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:10.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.893535+0000) 2020-02-05T22:42:10.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:10.892+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.893799+0000) 2020-02-05T22:42:10.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:10.893+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.894326+0000) 2020-02-05T22:42:10.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:10.893+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.894628+0000) 2020-02-05T22:42:10.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:10.895+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.895975+0000) 2020-02-05T22:42:10.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:10.896+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.896600+0000) 2020-02-05T22:42:10.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:10.896+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.896888+0000) 2020-02-05T22:42:10.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:10.896+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.897516+0000) 2020-02-05T22:42:10.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:10.897+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.897424+0000) 2020-02-05T22:42:10.899 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:10.897+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.897800+0000) 2020-02-05T22:42:10.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:10.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.901014+0000) 2020-02-05T22:42:10.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:10.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 122966 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:10.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:10.959+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:10.960297+0000) 2020-02-05T22:42:12.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:12.349+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:12.349827+0000) 2020-02-05T22:42:12.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:12.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:12.742397+0000) 2020-02-05T22:42:13.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:13.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:13.863617+0000) 2020-02-05T22:42:13.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:13.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:13.863614+0000) 2020-02-05T22:42:15.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:15.912+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.912547+0000) 2020-02-05T22:42:15.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:15.912+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.913465+0000) 2020-02-05T22:42:15.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:15.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.914289+0000) 2020-02-05T22:42:15.915 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:15.913+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.914999+0000) 2020-02-05T22:42:15.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:15.914+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.915210+0000) 2020-02-05T22:42:15.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:15.914+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.914669+0000) 2020-02-05T22:42:15.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:15.915+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.915991+0000) 2020-02-05T22:42:15.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:15.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.916503+0000) 2020-02-05T22:42:15.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:15.915+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.916701+0000) 2020-02-05T22:42:15.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:15.916+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.917530+0000) 2020-02-05T22:42:15.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:15.917+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.917852+0000) 2020-02-05T22:42:15.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:15.917+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.918352+0000) 2020-02-05T22:42:15.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:15.918+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.918815+0000) 2020-02-05T22:42:15.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:15.923+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.924356+0000) 2020-02-05T22:42:15.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:15.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123005 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:15.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:15.954+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.955128+0000) 2020-02-05T22:42:15.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:15.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:15.964085+0000) 2020-02-05T22:42:17.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:17.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:17.104251+0000) 2020-02-05T22:42:17.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:17.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:17.104844+0000) 2020-02-05T22:42:17.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:17.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:17.104937+0000) 2020-02-05T22:42:19.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:19.492+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:19.492899+0000) 2020-02-05T22:42:20.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:20.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.099751+0000) 2020-02-05T22:42:20.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:20.335+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.335775+0000) 2020-02-05T22:42:20.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:20.350+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.351363+0000) 2020-02-05T22:42:20.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:20.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.936027+0000) 2020-02-05T22:42:20.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:20.936+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.936617+0000) 2020-02-05T22:42:20.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:20.937+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.937494+0000) 2020-02-05T22:42:20.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:20.937+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.937978+0000) 2020-02-05T22:42:20.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:20.937+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.938347+0000) 2020-02-05T22:42:20.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:20.937+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.938666+0000) 2020-02-05T22:42:20.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:20.938+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.939067+0000) 2020-02-05T22:42:20.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:20.938+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.939437+0000) 2020-02-05T22:42:20.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:20.939+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.939804+0000) 2020-02-05T22:42:20.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:20.939+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.940310+0000) 2020-02-05T22:42:20.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:20.939+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.940595+0000) 2020-02-05T22:42:20.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:20.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.940995+0000) 2020-02-05T22:42:20.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:20.940+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.941168+0000) 2020-02-05T22:42:20.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:20.945+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.946642+0000) 2020-02-05T22:42:20.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:20.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123048 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:20.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:20.967+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:20.967793+0000) 2020-02-05T22:42:22.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:22.349+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:22.350066+0000) 2020-02-05T22:42:22.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:22.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:22.742657+0000) 2020-02-05T22:42:23.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:23.863+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:23.863834+0000) 2020-02-05T22:42:23.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:23.863+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:23.863835+0000) 2020-02-05T22:42:25.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:25.954+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.956133+0000) 2020-02-05T22:42:25.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:25.955+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.956739+0000) 2020-02-05T22:42:25.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:25.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.958025+0000) 2020-02-05T22:42:25.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:25.956+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.957952+0000) 2020-02-05T22:42:25.959 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:25.957+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.958343+0000) 2020-02-05T22:42:25.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:25.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.958757+0000) 2020-02-05T22:42:25.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:25.958+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.959391+0000) 2020-02-05T22:42:25.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:25.958+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.959868+0000) 2020-02-05T22:42:25.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:25.959+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.960516+0000) 2020-02-05T22:42:25.962 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:25.958+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.960249+0000) 2020-02-05T22:42:25.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:25.959+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.960744+0000) 2020-02-05T22:42:25.962 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:25.960+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.961751+0000) 2020-02-05T22:42:25.963 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:25.960+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.961866+0000) 2020-02-05T22:42:25.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:25.962+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.964260+0000) 2020-02-05T22:42:25.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:25.965+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:25.966236+0000) 2020-02-05T22:42:25.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:25.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:27.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:27.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:27.104572+0000) 2020-02-05T22:42:27.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:27.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:27.105121+0000) 2020-02-05T22:42:27.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:27.103+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:27.105157+0000) 2020-02-05T22:42:29.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:29.492+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:29.493151+0000) 2020-02-05T22:42:30.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:30.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.099984+0000) 2020-02-05T22:42:30.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:30.335+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.336034+0000) 2020-02-05T22:42:30.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:30.350+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.351550+0000) 2020-02-05T22:42:30.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:30.974+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.976328+0000) 2020-02-05T22:42:30.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:30.976+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.977331+0000) 2020-02-05T22:42:30.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:30.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.977862+0000) 2020-02-05T22:42:30.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:30.977+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.978213+0000) 2020-02-05T22:42:30.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:30.977+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.978750+0000) 2020-02-05T22:42:30.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:30.978+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.979176+0000) 2020-02-05T22:42:30.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:30.977+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.979267+0000) 2020-02-05T22:42:30.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:30.979+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.980653+0000) 2020-02-05T22:42:30.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:30.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.981567+0000) 2020-02-05T22:42:30.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:30.978+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.979831+0000) 2020-02-05T22:42:30.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:30.978+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.979868+0000) 2020-02-05T22:42:30.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:30.979+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.981061+0000) 2020-02-05T22:42:30.990 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:30.980+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.982096+0000) 2020-02-05T22:42:30.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:30.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:30.986651+0000) 2020-02-05T22:42:30.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:30.990+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123130 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:32.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:32.349+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:32.350322+0000) 2020-02-05T22:42:32.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:32.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:32.706184+0000) 2020-02-05T22:42:32.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:32.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:32.742883+0000) 2020-02-05T22:42:33.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:33.863+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:33.864132+0000) 2020-02-05T22:42:33.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:33.863+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:33.864130+0000) 2020-02-05T22:42:35.893 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:42:35.900 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:42:35.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:35.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.964469+0000) 2020-02-05T22:42:35.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:35.995+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.996603+0000) 2020-02-05T22:42:35.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:35.996+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.997561+0000) 2020-02-05T22:42:35.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:35.995+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.997268+0000) 2020-02-05T22:42:35.999 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:35.996+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.997827+0000) 2020-02-05T22:42:35.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:35.997+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.998178+0000) 2020-02-05T22:42:36.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:35.997+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.998593+0000) 2020-02-05T22:42:36.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:35.997+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.999256+0000) 2020-02-05T22:42:36.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:35.998+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:35.999763+0000) 2020-02-05T22:42:36.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:35.998+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.000048+0000) 2020-02-05T22:42:36.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:35.999+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.000573+0000) 2020-02-05T22:42:36.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:35.999+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.000876+0000) 2020-02-05T22:42:36.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:36.000+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.001445+0000) 2020-02-05T22:42:36.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:36.000+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.001790+0000) 2020-02-05T22:42:36.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:36.004+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:36.005268+0000) 2020-02-05T22:42:36.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:36.008+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:37.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:37.103+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:37.104836+0000) 2020-02-05T22:42:37.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:37.103+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:37.105306+0000) 2020-02-05T22:42:37.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:37.103+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:37.105382+0000) 2020-02-05T22:42:37.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:37.726+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:37.726481+0000) 2020-02-05T22:42:39.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:39.492+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:39.493382+0000) 2020-02-05T22:42:40.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:40.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:40.100240+0000) 2020-02-05T22:42:40.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:40.334+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:40.336310+0000) 2020-02-05T22:42:40.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:40.351+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:40.351823+0000) 2020-02-05T22:42:41.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:41.013+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.014330+0000) 2020-02-05T22:42:41.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:41.014+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.015599+0000) 2020-02-05T22:42:41.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:41.015+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.017039+0000) 2020-02-05T22:42:41.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:41.016+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.017421+0000) 2020-02-05T22:42:41.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:41.015+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.017019+0000) 2020-02-05T22:42:41.019 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:41.016+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.017959+0000) 2020-02-05T22:42:41.019 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:41.017+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.018377+0000) 2020-02-05T22:42:41.020 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:41.017+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.018951+0000) 2020-02-05T22:42:41.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:41.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.018873+0000) 2020-02-05T22:42:41.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:41.019+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.020161+0000) 2020-02-05T22:42:41.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:41.019+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.021166+0000) 2020-02-05T22:42:41.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:41.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.021317+0000) 2020-02-05T22:42:41.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:41.020+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.021586+0000) 2020-02-05T22:42:41.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:41.025+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:41.026160+0000) 2020-02-05T22:42:41.032 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:41.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:42.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:42.349+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:42.350589+0000) 2020-02-05T22:42:42.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:42.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:42.706419+0000) 2020-02-05T22:42:42.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:42.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:42.743165+0000) 2020-02-05T22:42:43.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:43.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:43.864436+0000) 2020-02-05T22:42:43.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:43.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:43.864436+0000) 2020-02-05T22:42:45.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:45.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:45.964734+0000) 2020-02-05T22:42:46.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:46.036+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.038267+0000) 2020-02-05T22:42:46.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:46.038+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.039206+0000) 2020-02-05T22:42:46.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:46.038+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.039514+0000) 2020-02-05T22:42:46.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:46.038+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.040149+0000) 2020-02-05T22:42:46.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:46.039+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.040539+0000) 2020-02-05T22:42:46.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:46.040+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.041223+0000) 2020-02-05T22:42:46.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:46.040+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.042027+0000) 2020-02-05T22:42:46.044 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:46.041+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.043033+0000) 2020-02-05T22:42:46.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:46.041+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.042469+0000) 2020-02-05T22:42:46.044 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:46.041+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.042565+0000) 2020-02-05T22:42:46.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:46.042+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.043649+0000) 2020-02-05T22:42:46.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:46.042+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.043687+0000) 2020-02-05T22:42:46.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:46.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.044435+0000) 2020-02-05T22:42:46.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:46.046+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:46.047732+0000) 2020-02-05T22:42:46.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:46.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:47.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:47.104+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:47.105101+0000) 2020-02-05T22:42:47.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:47.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:47.105527+0000) 2020-02-05T22:42:47.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:47.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:47.105617+0000) 2020-02-05T22:42:47.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:47.734+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:47.734526+0000) 2020-02-05T22:42:49.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:49.493+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:49.493624+0000) 2020-02-05T22:42:50.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:50.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:50.100466+0000) 2020-02-05T22:42:50.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:50.335+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:50.336574+0000) 2020-02-05T22:42:50.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:50.351+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:50.352071+0000) 2020-02-05T22:42:51.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:51.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.056175+0000) 2020-02-05T22:42:51.059 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:51.056+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.058094+0000) 2020-02-05T22:42:51.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:51.057+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.058706+0000) 2020-02-05T22:42:51.060 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:51.057+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.059219+0000) 2020-02-05T22:42:51.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:51.058+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.059970+0000) 2020-02-05T22:42:51.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:51.058+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.060174+0000) 2020-02-05T22:42:51.061 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:51.059+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.060610+0000) 2020-02-05T22:42:51.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:51.060+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.061507+0000) 2020-02-05T22:42:51.062 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:51.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.061725+0000) 2020-02-05T22:42:51.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:51.061+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.062244+0000) 2020-02-05T22:42:51.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:51.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.062242+0000) 2020-02-05T22:42:51.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:51.061+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.062407+0000) 2020-02-05T22:42:51.064 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:51.061+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.062733+0000) 2020-02-05T22:42:51.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:51.067+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:51.068959+0000) 2020-02-05T22:42:51.073 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:51.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:52.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:52.350+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:52.350846+0000) 2020-02-05T22:42:52.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:52.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:52.706659+0000) 2020-02-05T22:42:52.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:52.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:52.743407+0000) 2020-02-05T22:42:53.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:53.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:53.864707+0000) 2020-02-05T22:42:53.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:53.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:53.864707+0000) 2020-02-05T22:42:55.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:55.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:55.964951+0000) 2020-02-05T22:42:56.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:42:56.079+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.079341+0000) 2020-02-05T22:42:56.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:56.079+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.079843+0000) 2020-02-05T22:42:56.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:42:56.079+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.079998+0000) 2020-02-05T22:42:56.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:42:56.080+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.080426+0000) 2020-02-05T22:42:56.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:42:56.079+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.080832+0000) 2020-02-05T22:42:56.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:42:56.080+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.081515+0000) 2020-02-05T22:42:56.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:56.081+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.081765+0000) 2020-02-05T22:42:56.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:42:56.080+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.082122+0000) 2020-02-05T22:42:56.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:56.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.082371+0000) 2020-02-05T22:42:56.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:56.081+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.082598+0000) 2020-02-05T22:42:56.084 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:42:56.082+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.082352+0000) 2020-02-05T22:42:56.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:56.081+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.083058+0000) 2020-02-05T22:42:56.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:42:56.083+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.083996+0000) 2020-02-05T22:42:56.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:56.089+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:56.089991+0000) 2020-02-05T22:42:56.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:42:56.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:42:57.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:42:57.104+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:57.105391+0000) 2020-02-05T22:42:57.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:42:57.104+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:57.105852+0000) 2020-02-05T22:42:57.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:42:57.104+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:57.105784+0000) 2020-02-05T22:42:57.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:57.741+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:57.742560+0000) 2020-02-05T22:42:59.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:42:59.492+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:42:59.493896+0000) 2020-02-05T22:43:00.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:00.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:00.100721+0000) 2020-02-05T22:43:00.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:00.336+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:00.336803+0000) 2020-02-05T22:43:00.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:00.351+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:00.352404+0000) 2020-02-05T22:43:01.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:01.098+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.099211+0000) 2020-02-05T22:43:01.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:01.098+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.099909+0000) 2020-02-05T22:43:01.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:01.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.100520+0000) 2020-02-05T22:43:01.102 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:01.099+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.100659+0000) 2020-02-05T22:43:01.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:01.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.101079+0000) 2020-02-05T22:43:01.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:01.100+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.101532+0000) 2020-02-05T22:43:01.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:01.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.102261+0000) 2020-02-05T22:43:01.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:01.101+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.102712+0000) 2020-02-05T22:43:01.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:01.102+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.103821+0000) 2020-02-05T22:43:01.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:01.102+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.103824+0000) 2020-02-05T22:43:01.105 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:01.103+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.104840+0000) 2020-02-05T22:43:01.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:01.103+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.105017+0000) 2020-02-05T22:43:01.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:01.103+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.105225+0000) 2020-02-05T22:43:01.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:01.108+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:01.109395+0000) 2020-02-05T22:43:01.115 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:01.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:02.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:02.350+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:02.351095+0000) 2020-02-05T22:43:02.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:02.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:02.706890+0000) 2020-02-05T22:43:02.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:02.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:02.743660+0000) 2020-02-05T22:43:03.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:03.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:03.864984+0000) 2020-02-05T22:43:03.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:03.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:03.864986+0000) 2020-02-05T22:43:05.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:05.964+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:05.965212+0000) 2020-02-05T22:43:06.031 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:43:06.043 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:43:06.119 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:06.116+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.118005+0000) 2020-02-05T22:43:06.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:06.116+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.118123+0000) 2020-02-05T22:43:06.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:06.117+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.118938+0000) 2020-02-05T22:43:06.120 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:06.118+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.119575+0000) 2020-02-05T22:43:06.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:06.120+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.121430+0000) 2020-02-05T22:43:06.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:06.119+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.120666+0000) 2020-02-05T22:43:06.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:06.119+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.120849+0000) 2020-02-05T22:43:06.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:06.120+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.122291+0000) 2020-02-05T22:43:06.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:06.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.122392+0000) 2020-02-05T22:43:06.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:06.121+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.122862+0000) 2020-02-05T22:43:06.125 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:06.121+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.123171+0000) 2020-02-05T22:43:06.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:06.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.125162+0000) 2020-02-05T22:43:06.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:06.127+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.128523+0000) 2020-02-05T22:43:06.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:06.128+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:06.129735+0000) 2020-02-05T22:43:06.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:06.135+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:07.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:07.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:07.105678+0000) 2020-02-05T22:43:07.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:07.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:07.106107+0000) 2020-02-05T22:43:07.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:07.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:07.106067+0000) 2020-02-05T22:43:07.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:07.749+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:07.750618+0000) 2020-02-05T22:43:09.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:09.493+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:09.494135+0000) 2020-02-05T22:43:10.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:10.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:10.100953+0000) 2020-02-05T22:43:10.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:10.336+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:10.337051+0000) 2020-02-05T22:43:10.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:10.351+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:10.352635+0000) 2020-02-05T22:43:11.145 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:11.143+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.144343+0000) 2020-02-05T22:43:11.145 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:11.143+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.144814+0000) 2020-02-05T22:43:11.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:11.144+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.145212+0000) 2020-02-05T22:43:11.147 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:11.144+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.146158+0000) 2020-02-05T22:43:11.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:11.146+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.147182+0000) 2020-02-05T22:43:11.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:11.146+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.147577+0000) 2020-02-05T22:43:11.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:11.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.147917+0000) 2020-02-05T22:43:11.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:11.146+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.147849+0000) 2020-02-05T22:43:11.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:11.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.148647+0000) 2020-02-05T22:43:11.150 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:11.147+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.148961+0000) 2020-02-05T22:43:11.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:11.147+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.149151+0000) 2020-02-05T22:43:11.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:11.147+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.149378+0000) 2020-02-05T22:43:11.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:11.148+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.149622+0000) 2020-02-05T22:43:11.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:11.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:11.151636+0000) 2020-02-05T22:43:11.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:11.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:12.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:12.350+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:12.351388+0000) 2020-02-05T22:43:12.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:12.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:12.707159+0000) 2020-02-05T22:43:12.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:12.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:12.743898+0000) 2020-02-05T22:43:13.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:13.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:13.865305+0000) 2020-02-05T22:43:13.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:13.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:13.865303+0000) 2020-02-05T22:43:15.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:15.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:15.965495+0000) 2020-02-05T22:43:16.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:16.163+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.164590+0000) 2020-02-05T22:43:16.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:16.164+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.165288+0000) 2020-02-05T22:43:16.166 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:16.165+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.165589+0000) 2020-02-05T22:43:16.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:16.165+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.166520+0000) 2020-02-05T22:43:16.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:16.167+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.168363+0000) 2020-02-05T22:43:16.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:16.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.168863+0000) 2020-02-05T22:43:16.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:16.168+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.169294+0000) 2020-02-05T22:43:16.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:16.168+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.168952+0000) 2020-02-05T22:43:16.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:16.169+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.170459+0000) 2020-02-05T22:43:16.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:16.169+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.170088+0000) 2020-02-05T22:43:16.172 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:16.169+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.170153+0000) 2020-02-05T22:43:16.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:16.170+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.170664+0000) 2020-02-05T22:43:16.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:16.172+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.173363+0000) 2020-02-05T22:43:16.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:16.172+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:16.173654+0000) 2020-02-05T22:43:16.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:16.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:17.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:17.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:17.105912+0000) 2020-02-05T22:43:17.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:17.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:17.106239+0000) 2020-02-05T22:43:17.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:17.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:17.106322+0000) 2020-02-05T22:43:17.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:17.758+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:17.759053+0000) 2020-02-05T22:43:19.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:19.493+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:19.494367+0000) 2020-02-05T22:43:20.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:20.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:20.101185+0000) 2020-02-05T22:43:20.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:20.336+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:20.337350+0000) 2020-02-05T22:43:20.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:20.351+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:20.352885+0000) 2020-02-05T22:43:21.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:21.185+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.186200+0000) 2020-02-05T22:43:21.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:21.186+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.187177+0000) 2020-02-05T22:43:21.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:21.187+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.187644+0000) 2020-02-05T22:43:21.189 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:21.186+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.188014+0000) 2020-02-05T22:43:21.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:21.187+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.188593+0000) 2020-02-05T22:43:21.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:21.188+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.189122+0000) 2020-02-05T22:43:21.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:21.189+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.189549+0000) 2020-02-05T22:43:21.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:21.188+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.189504+0000) 2020-02-05T22:43:21.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:21.189+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.189988+0000) 2020-02-05T22:43:21.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:21.190+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.190480+0000) 2020-02-05T22:43:21.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:21.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.190184+0000) 2020-02-05T22:43:21.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:21.190+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.191167+0000) 2020-02-05T22:43:21.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:21.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.195108+0000) 2020-02-05T22:43:21.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:21.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:21.195345+0000) 2020-02-05T22:43:21.201 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:21.199+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123539 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:22.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:22.351+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:22.351702+0000) 2020-02-05T22:43:22.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:22.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:22.707356+0000) 2020-02-05T22:43:22.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:22.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:22.744156+0000) 2020-02-05T22:43:23.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:23.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:23.865572+0000) 2020-02-05T22:43:23.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:23.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:23.865572+0000) 2020-02-05T22:43:25.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:25.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:25.965723+0000) 2020-02-05T22:43:26.206 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:26.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.205820+0000) 2020-02-05T22:43:26.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:26.205+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.206370+0000) 2020-02-05T22:43:26.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:26.206+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.207202+0000) 2020-02-05T22:43:26.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:26.206+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.207291+0000) 2020-02-05T22:43:26.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:26.206+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.207379+0000) 2020-02-05T22:43:26.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:26.206+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.208046+0000) 2020-02-05T22:43:26.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:26.207+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.208704+0000) 2020-02-05T22:43:26.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:26.208+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.208964+0000) 2020-02-05T22:43:26.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:26.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.209603+0000) 2020-02-05T22:43:26.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:26.209+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.209536+0000) 2020-02-05T22:43:26.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:26.209+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.210512+0000) 2020-02-05T22:43:26.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:26.209+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.209849+0000) 2020-02-05T22:43:26.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:26.210+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.211324+0000) 2020-02-05T22:43:26.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:26.212+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:26.214059+0000) 2020-02-05T22:43:26.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:26.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:27.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:27.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:27.106140+0000) 2020-02-05T22:43:27.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:27.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:27.106418+0000) 2020-02-05T22:43:27.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:27.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:27.106592+0000) 2020-02-05T22:43:27.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:27.766+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:27.767333+0000) 2020-02-05T22:43:29.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:29.493+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:29.494602+0000) 2020-02-05T22:43:30.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:30.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:30.101420+0000) 2020-02-05T22:43:30.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:30.337+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:30.337715+0000) 2020-02-05T22:43:30.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:30.352+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:30.353144+0000) 2020-02-05T22:43:31.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:31.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.227319+0000) 2020-02-05T22:43:31.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:31.227+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.227985+0000) 2020-02-05T22:43:31.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:31.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.229291+0000) 2020-02-05T22:43:31.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:31.229+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.229515+0000) 2020-02-05T22:43:31.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:31.228+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.229821+0000) 2020-02-05T22:43:31.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:31.229+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.230319+0000) 2020-02-05T22:43:31.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:31.230+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.231058+0000) 2020-02-05T22:43:31.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:31.230+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.231247+0000) 2020-02-05T22:43:31.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:31.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.231789+0000) 2020-02-05T22:43:31.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:31.231+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.232008+0000) 2020-02-05T22:43:31.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:31.231+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.232211+0000) 2020-02-05T22:43:31.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:31.231+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.232339+0000) 2020-02-05T22:43:31.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:31.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.232860+0000) 2020-02-05T22:43:31.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:31.234+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:31.235319+0000) 2020-02-05T22:43:31.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:31.241+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:32.353 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:32.351+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:32.351959+0000) 2020-02-05T22:43:32.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:32.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:32.707608+0000) 2020-02-05T22:43:32.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:32.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:32.744411+0000) 2020-02-05T22:43:33.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:33.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:33.865839+0000) 2020-02-05T22:43:33.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:33.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:33.865839+0000) 2020-02-05T22:43:35.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:35.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:35.965956+0000) 2020-02-05T22:43:36.182 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:43:36.189 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:43:36.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:36.248+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.248681+0000) 2020-02-05T22:43:36.250 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:36.248+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.249210+0000) 2020-02-05T22:43:36.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:36.249+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.249880+0000) 2020-02-05T22:43:36.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:36.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.250372+0000) 2020-02-05T22:43:36.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:36.249+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.250542+0000) 2020-02-05T22:43:36.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:36.250+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.250991+0000) 2020-02-05T22:43:36.252 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:36.250+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.251072+0000) 2020-02-05T22:43:36.252 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:36.250+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.251358+0000) 2020-02-05T22:43:36.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:36.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.252377+0000) 2020-02-05T22:43:36.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:36.251+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.251742+0000) 2020-02-05T22:43:36.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:36.251+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.251960+0000) 2020-02-05T22:43:36.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:36.252+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.252719+0000) 2020-02-05T22:43:36.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:36.252+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.253995+0000) 2020-02-05T22:43:36.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:36.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:36.259319+0000) 2020-02-05T22:43:36.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:36.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:37.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:37.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:37.106366+0000) 2020-02-05T22:43:37.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:37.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:37.106684+0000) 2020-02-05T22:43:37.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:37.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:37.106785+0000) 2020-02-05T22:43:37.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:37.775+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:37.775382+0000) 2020-02-05T22:43:39.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:39.493+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:39.494854+0000) 2020-02-05T22:43:40.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:40.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:40.101659+0000) 2020-02-05T22:43:40.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:40.337+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:40.337979+0000) 2020-02-05T22:43:40.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:40.353+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:40.353428+0000) 2020-02-05T22:43:41.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:41.267+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.268154+0000) 2020-02-05T22:43:41.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:41.267+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.268618+0000) 2020-02-05T22:43:41.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:41.268+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.269059+0000) 2020-02-05T22:43:41.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:41.269+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.270231+0000) 2020-02-05T22:43:41.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:41.269+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.270405+0000) 2020-02-05T22:43:41.272 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:41.269+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.270910+0000) 2020-02-05T22:43:41.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:41.271+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.271460+0000) 2020-02-05T22:43:41.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:41.270+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.271482+0000) 2020-02-05T22:43:41.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:41.271+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.271756+0000) 2020-02-05T22:43:41.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:41.271+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.272357+0000) 2020-02-05T22:43:41.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:41.271+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.272950+0000) 2020-02-05T22:43:41.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:41.272+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.272965+0000) 2020-02-05T22:43:41.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:41.272+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.273149+0000) 2020-02-05T22:43:41.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:41.279+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:41.280354+0000) 2020-02-05T22:43:41.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:41.282+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:42.353 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:42.352+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:42.352223+0000) 2020-02-05T22:43:42.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:42.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:42.707834+0000) 2020-02-05T22:43:42.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:42.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:42.744658+0000) 2020-02-05T22:43:43.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:43.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:43.866144+0000) 2020-02-05T22:43:43.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:43.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:43.866144+0000) 2020-02-05T22:43:45.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:45.966+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:45.966192+0000) 2020-02-05T22:43:46.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:46.288+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.289152+0000) 2020-02-05T22:43:46.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:46.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.289715+0000) 2020-02-05T22:43:46.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:46.288+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.289743+0000) 2020-02-05T22:43:46.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:46.289+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.290448+0000) 2020-02-05T22:43:46.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:46.290+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.290960+0000) 2020-02-05T22:43:46.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:46.291+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.291489+0000) 2020-02-05T22:43:46.293 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:46.291+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.292385+0000) 2020-02-05T22:43:46.293 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:46.291+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.292211+0000) 2020-02-05T22:43:46.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:46.291+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.292923+0000) 2020-02-05T22:43:46.294 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:46.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.292568+0000) 2020-02-05T22:43:46.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:46.292+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.293908+0000) 2020-02-05T22:43:46.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:46.293+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.293526+0000) 2020-02-05T22:43:46.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:46.294+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.294772+0000) 2020-02-05T22:43:46.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:46.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:46.300817+0000) 2020-02-05T22:43:46.305 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:46.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123742 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:47.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:47.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:47.106573+0000) 2020-02-05T22:43:47.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:47.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:47.106845+0000) 2020-02-05T22:43:47.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:47.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:47.107008+0000) 2020-02-05T22:43:47.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:47.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:47.787483+0000) 2020-02-05T22:43:49.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:49.495+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:49.495140+0000) 2020-02-05T22:43:50.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:50.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:50.101899+0000) 2020-02-05T22:43:50.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:50.338+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:50.338257+0000) 2020-02-05T22:43:50.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:50.353+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:50.353650+0000) 2020-02-05T22:43:51.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:51.310+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.310667+0000) 2020-02-05T22:43:51.312 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:51.311+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.311233+0000) 2020-02-05T22:43:51.312 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:51.311+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.311662+0000) 2020-02-05T22:43:51.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:51.312+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.312293+0000) 2020-02-05T22:43:51.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:51.311+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.312305+0000) 2020-02-05T22:43:51.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:51.312+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.312819+0000) 2020-02-05T22:43:51.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:51.313+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.313132+0000) 2020-02-05T22:43:51.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:51.312+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.313310+0000) 2020-02-05T22:43:51.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:51.314+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.314940+0000) 2020-02-05T22:43:51.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:51.314+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.315184+0000) 2020-02-05T22:43:51.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:51.313+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.313804+0000) 2020-02-05T22:43:51.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:51.314+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.314771+0000) 2020-02-05T22:43:51.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:51.316+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.316306+0000) 2020-02-05T22:43:51.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:51.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:51.320966+0000) 2020-02-05T22:43:51.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:51.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:52.353 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:52.352+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:52.352514+0000) 2020-02-05T22:43:52.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:52.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:52.708065+0000) 2020-02-05T22:43:52.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:52.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:52.744888+0000) 2020-02-05T22:43:53.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:53.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:53.866417+0000) 2020-02-05T22:43:53.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:53.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:53.866417+0000) 2020-02-05T22:43:55.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:55.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:55.966482+0000) 2020-02-05T22:43:56.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:43:56.328+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.329575+0000) 2020-02-05T22:43:56.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:56.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.330589+0000) 2020-02-05T22:43:56.332 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:43:56.330+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.331411+0000) 2020-02-05T22:43:56.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:43:56.330+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.331668+0000) 2020-02-05T22:43:56.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:43:56.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.332571+0000) 2020-02-05T22:43:56.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:56.330+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.332033+0000) 2020-02-05T22:43:56.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:56.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.332569+0000) 2020-02-05T22:43:56.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:43:56.332+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.333181+0000) 2020-02-05T22:43:56.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:43:56.332+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.333875+0000) 2020-02-05T22:43:56.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:43:56.332+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.333813+0000) 2020-02-05T22:43:56.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:56.333+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.334597+0000) 2020-02-05T22:43:56.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:43:56.334+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.335252+0000) 2020-02-05T22:43:56.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:56.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.335505+0000) 2020-02-05T22:43:56.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:56.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:56.340804+0000) 2020-02-05T22:43:56.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:43:56.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:43:57.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:43:57.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:57.106797+0000) 2020-02-05T22:43:57.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:43:57.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:57.107050+0000) 2020-02-05T22:43:57.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:43:57.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:57.107210+0000) 2020-02-05T22:43:57.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:57.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:57.795840+0000) 2020-02-05T22:43:59.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:43:59.494+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:43:59.495351+0000) 2020-02-05T22:44:00.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:00.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:00.102133+0000) 2020-02-05T22:44:00.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:00.337+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:00.338534+0000) 2020-02-05T22:44:00.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:00.352+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:00.353914+0000) 2020-02-05T22:44:01.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:01.349+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.350504+0000) 2020-02-05T22:44:01.353 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:01.351+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.352188+0000) 2020-02-05T22:44:01.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:01.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.352855+0000) 2020-02-05T22:44:01.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:01.352+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.353834+0000) 2020-02-05T22:44:01.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:01.352+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.353958+0000) 2020-02-05T22:44:01.356 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:01.353+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.355026+0000) 2020-02-05T22:44:01.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:01.354+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.355790+0000) 2020-02-05T22:44:01.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:01.355+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.356176+0000) 2020-02-05T22:44:01.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:01.354+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.356401+0000) 2020-02-05T22:44:01.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:01.355+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.357007+0000) 2020-02-05T22:44:01.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:01.355+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.357351+0000) 2020-02-05T22:44:01.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:01.356+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.357810+0000) 2020-02-05T22:44:01.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:01.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.358189+0000) 2020-02-05T22:44:01.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:01.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:01.361523+0000) 2020-02-05T22:44:01.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:01.364+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:02.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:02.351+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:02.352771+0000) 2020-02-05T22:44:02.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:02.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:02.708377+0000) 2020-02-05T22:44:02.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:02.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:02.745146+0000) 2020-02-05T22:44:03.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:03.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:03.866682+0000) 2020-02-05T22:44:03.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:03.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:03.866677+0000) 2020-02-05T22:44:05.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:05.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:05.966667+0000) 2020-02-05T22:44:06.313 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:44:06.326 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:44:06.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:06.369+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.371031+0000) 2020-02-05T22:44:06.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:06.370+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.371686+0000) 2020-02-05T22:44:06.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:06.371+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.372427+0000) 2020-02-05T22:44:06.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:06.371+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.372676+0000) 2020-02-05T22:44:06.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:06.372+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.374017+0000) 2020-02-05T22:44:06.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:06.372+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.374112+0000) 2020-02-05T22:44:06.375 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:06.372+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.373419+0000) 2020-02-05T22:44:06.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:06.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.373790+0000) 2020-02-05T22:44:06.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:06.373+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.374729+0000) 2020-02-05T22:44:06.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:06.373+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.374908+0000) 2020-02-05T22:44:06.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:06.374+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.375849+0000) 2020-02-05T22:44:06.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:06.374+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.376184+0000) 2020-02-05T22:44:06.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:06.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.378597+0000) 2020-02-05T22:44:06.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:06.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:06.378829+0000) 2020-02-05T22:44:06.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:06.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:07.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:07.105+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:07.107093+0000) 2020-02-05T22:44:07.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:07.105+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:07.107231+0000) 2020-02-05T22:44:07.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:07.105+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:07.107398+0000) 2020-02-05T22:44:07.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:07.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:07.803261+0000) 2020-02-05T22:44:09.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:09.494+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:09.495594+0000) 2020-02-05T22:44:10.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:10.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:10.102392+0000) 2020-02-05T22:44:10.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:10.337+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:10.338782+0000) 2020-02-05T22:44:10.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:10.353+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:10.354165+0000) 2020-02-05T22:44:11.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:11.391+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.392881+0000) 2020-02-05T22:44:11.395 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:11.393+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.394236+0000) 2020-02-05T22:44:11.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:11.394+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.395447+0000) 2020-02-05T22:44:11.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:11.393+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.395092+0000) 2020-02-05T22:44:11.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:11.395+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.396486+0000) 2020-02-05T22:44:11.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:11.394+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.395998+0000) 2020-02-05T22:44:11.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:11.395+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.397254+0000) 2020-02-05T22:44:11.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:11.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.397618+0000) 2020-02-05T22:44:11.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:11.396+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.398039+0000) 2020-02-05T22:44:11.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:11.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.398096+0000) 2020-02-05T22:44:11.400 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:11.397+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.398628+0000) 2020-02-05T22:44:11.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:11.396+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.398148+0000) 2020-02-05T22:44:11.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:11.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.405094+0000) 2020-02-05T22:44:11.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:11.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:11.405347+0000) 2020-02-05T22:44:11.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:11.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:12.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:12.352+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:12.353094+0000) 2020-02-05T22:44:12.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:12.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:12.708608+0000) 2020-02-05T22:44:12.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:12.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:12.745337+0000) 2020-02-05T22:44:13.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:13.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:13.866979+0000) 2020-02-05T22:44:13.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:13.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:13.866979+0000) 2020-02-05T22:44:15.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:15.966+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:15.966935+0000) 2020-02-05T22:44:16.416 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:16.414+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.415529+0000) 2020-02-05T22:44:16.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:16.414+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.415958+0000) 2020-02-05T22:44:16.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:16.415+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.416617+0000) 2020-02-05T22:44:16.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:16.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.417160+0000) 2020-02-05T22:44:16.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:16.417+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.418599+0000) 2020-02-05T22:44:16.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:16.416+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.418223+0000) 2020-02-05T22:44:16.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:16.417+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.419121+0000) 2020-02-05T22:44:16.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:16.418+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.419776+0000) 2020-02-05T22:44:16.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:16.419+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.420197+0000) 2020-02-05T22:44:16.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:16.418+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.420092+0000) 2020-02-05T22:44:16.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:16.419+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.420736+0000) 2020-02-05T22:44:16.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:16.419+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.420598+0000) 2020-02-05T22:44:16.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:16.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.421200+0000) 2020-02-05T22:44:16.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:16.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:16.423559+0000) 2020-02-05T22:44:16.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:16.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 123987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:17.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:17.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:17.107306+0000) 2020-02-05T22:44:17.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:17.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:17.107481+0000) 2020-02-05T22:44:17.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:17.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:17.107623+0000) 2020-02-05T22:44:17.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:17.810+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:17.811355+0000) 2020-02-05T22:44:19.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:19.495+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:19.495824+0000) 2020-02-05T22:44:20.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:20.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:20.102625+0000) 2020-02-05T22:44:20.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:20.337+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:20.339024+0000) 2020-02-05T22:44:20.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:20.353+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:20.354409+0000) 2020-02-05T22:44:21.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:21.435+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.436823+0000) 2020-02-05T22:44:21.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:21.436+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.437536+0000) 2020-02-05T22:44:21.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:21.436+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.437872+0000) 2020-02-05T22:44:21.439 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:21.437+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.438157+0000) 2020-02-05T22:44:21.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:21.437+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.438565+0000) 2020-02-05T22:44:21.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:21.437+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.438601+0000) 2020-02-05T22:44:21.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:21.437+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.439140+0000) 2020-02-05T22:44:21.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:21.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.439729+0000) 2020-02-05T22:44:21.441 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:21.438+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.439826+0000) 2020-02-05T22:44:21.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:21.438+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.439818+0000) 2020-02-05T22:44:21.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:21.439+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.440423+0000) 2020-02-05T22:44:21.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:21.439+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.441132+0000) 2020-02-05T22:44:21.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:21.440+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.442295+0000) 2020-02-05T22:44:21.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:21.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:21.444588+0000) 2020-02-05T22:44:21.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:21.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124031 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:22.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:22.352+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:22.353332+0000) 2020-02-05T22:44:22.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:22.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:22.708839+0000) 2020-02-05T22:44:22.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:22.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:22.745544+0000) 2020-02-05T22:44:23.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:23.866+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:23.867287+0000) 2020-02-05T22:44:23.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:23.866+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:23.867284+0000) 2020-02-05T22:44:25.968 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:25.966+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:25.967182+0000) 2020-02-05T22:44:26.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:26.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.458408+0000) 2020-02-05T22:44:26.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:26.457+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.458680+0000) 2020-02-05T22:44:26.460 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:26.458+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.459253+0000) 2020-02-05T22:44:26.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:26.458+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.459414+0000) 2020-02-05T22:44:26.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:26.458+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.460023+0000) 2020-02-05T22:44:26.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:26.458+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.460331+0000) 2020-02-05T22:44:26.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:26.459+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.460678+0000) 2020-02-05T22:44:26.462 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:26.460+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.461451+0000) 2020-02-05T22:44:26.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:26.461+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.462481+0000) 2020-02-05T22:44:26.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:26.461+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.462839+0000) 2020-02-05T22:44:26.464 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:26.461+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.463127+0000) 2020-02-05T22:44:26.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:26.462+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.463956+0000) 2020-02-05T22:44:26.465 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:26.462+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.463915+0000) 2020-02-05T22:44:26.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:26.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:26.466967+0000) 2020-02-05T22:44:26.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:26.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:27.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:27.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:27.107534+0000) 2020-02-05T22:44:27.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:27.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:27.107709+0000) 2020-02-05T22:44:27.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:27.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:27.107890+0000) 2020-02-05T22:44:27.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:27.818+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:27.819371+0000) 2020-02-05T22:44:29.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:29.495+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:29.496073+0000) 2020-02-05T22:44:30.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:30.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:30.102861+0000) 2020-02-05T22:44:30.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:30.338+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:30.339284+0000) 2020-02-05T22:44:30.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:30.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:30.354673+0000) 2020-02-05T22:44:31.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:31.477+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.478790+0000) 2020-02-05T22:44:31.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:31.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.479696+0000) 2020-02-05T22:44:31.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:31.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.485509+0000) 2020-02-05T22:44:31.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:31.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.485698+0000) 2020-02-05T22:44:31.599 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:31.479+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.480500+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:31.479+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.480667+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:31.481+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.482062+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:31.481+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.482141+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:31.482+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.483217+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:31.479+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.481270+0000) 2020-02-05T22:44:31.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:31.479+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.480924+0000) 2020-02-05T22:44:31.601 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:31.480+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.482082+0000) 2020-02-05T22:44:31.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:31.480+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.482241+0000) 2020-02-05T22:44:31.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:31.481+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:31.483170+0000) 2020-02-05T22:44:31.601 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:31.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:32.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:32.352+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:32.353571+0000) 2020-02-05T22:44:32.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:32.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:32.709063+0000) 2020-02-05T22:44:32.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:32.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:32.745768+0000) 2020-02-05T22:44:33.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:33.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:33.867558+0000) 2020-02-05T22:44:33.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:33.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:33.867560+0000) 2020-02-05T22:44:35.968 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:35.966+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:35.967466+0000) 2020-02-05T22:44:36.459 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:44:36.465 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:44:36.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:36.496+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.498231+0000) 2020-02-05T22:44:36.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:36.497+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.498784+0000) 2020-02-05T22:44:36.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:36.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.498912+0000) 2020-02-05T22:44:36.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:36.498+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.499434+0000) 2020-02-05T22:44:36.502 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:36.498+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.499484+0000) 2020-02-05T22:44:36.503 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:36.498+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.499943+0000) 2020-02-05T22:44:36.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:36.497+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.499147+0000) 2020-02-05T22:44:36.504 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:36.497+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.499212+0000) 2020-02-05T22:44:36.504 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:36.499+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.500465+0000) 2020-02-05T22:44:36.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:36.500+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.501885+0000) 2020-02-05T22:44:36.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:36.500+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.502240+0000) 2020-02-05T22:44:36.505 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:36.502+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.503680+0000) 2020-02-05T22:44:36.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:36.503+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.504336+0000) 2020-02-05T22:44:36.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:36.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:36.504470+0000) 2020-02-05T22:44:36.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:36.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:37.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:37.106+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:37.107741+0000) 2020-02-05T22:44:37.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:37.106+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:37.107963+0000) 2020-02-05T22:44:37.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:37.106+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:37.108160+0000) 2020-02-05T22:44:37.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:37.826+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:37.826885+0000) 2020-02-05T22:44:39.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:39.496+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:39.496377+0000) 2020-02-05T22:44:40.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:40.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:40.103119+0000) 2020-02-05T22:44:40.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:40.338+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:40.339534+0000) 2020-02-05T22:44:40.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:40.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:40.354897+0000) 2020-02-05T22:44:41.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:41.517+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.518231+0000) 2020-02-05T22:44:41.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:41.517+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.518444+0000) 2020-02-05T22:44:41.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:41.517+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.519139+0000) 2020-02-05T22:44:41.520 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:41.518+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.519618+0000) 2020-02-05T22:44:41.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:41.519+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.520147+0000) 2020-02-05T22:44:41.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:41.519+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.520426+0000) 2020-02-05T22:44:41.522 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:41.519+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.520773+0000) 2020-02-05T22:44:41.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:41.520+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.521399+0000) 2020-02-05T22:44:41.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:41.520+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.521597+0000) 2020-02-05T22:44:41.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:41.520+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.522083+0000) 2020-02-05T22:44:41.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:41.521+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.522505+0000) 2020-02-05T22:44:41.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:41.521+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.522518+0000) 2020-02-05T22:44:41.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:41.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.523474+0000) 2020-02-05T22:44:41.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:41.527+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:41.528664+0000) 2020-02-05T22:44:41.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:41.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:42.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:42.353+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:42.353801+0000) 2020-02-05T22:44:42.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:42.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:42.709315+0000) 2020-02-05T22:44:42.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:42.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:42.746026+0000) 2020-02-05T22:44:43.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:43.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:43.867832+0000) 2020-02-05T22:44:43.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:43.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:43.867832+0000) 2020-02-05T22:44:45.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:45.967+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:45.967724+0000) 2020-02-05T22:44:46.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:46.538+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.538122+0000) 2020-02-05T22:44:46.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:46.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.539323+0000) 2020-02-05T22:44:46.541 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:46.540+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.540227+0000) 2020-02-05T22:44:46.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:46.540+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.540897+0000) 2020-02-05T22:44:46.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:46.541+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.541069+0000) 2020-02-05T22:44:46.542 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:46.540+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.540956+0000) 2020-02-05T22:44:46.543 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:46.541+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.541741+0000) 2020-02-05T22:44:46.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:46.541+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.541528+0000) 2020-02-05T22:44:46.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:46.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.541705+0000) 2020-02-05T22:44:46.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:46.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.542478+0000) 2020-02-05T22:44:46.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:46.543+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.543886+0000) 2020-02-05T22:44:46.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:46.544+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.544632+0000) 2020-02-05T22:44:46.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:46.548+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.548972+0000) 2020-02-05T22:44:46.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:46.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:46.550289+0000) 2020-02-05T22:44:46.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:46.552+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:47.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:47.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:47.107986+0000) 2020-02-05T22:44:47.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:47.107+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:47.108179+0000) 2020-02-05T22:44:47.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:47.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:47.108407+0000) 2020-02-05T22:44:47.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:47.834+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:47.835124+0000) 2020-02-05T22:44:49.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:49.495+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:49.496608+0000) 2020-02-05T22:44:50.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:50.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:50.103354+0000) 2020-02-05T22:44:50.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:50.339+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:50.339756+0000) 2020-02-05T22:44:50.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:50.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:50.355131+0000) 2020-02-05T22:44:51.558 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:51.556+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.557712+0000) 2020-02-05T22:44:51.560 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:51.559+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.559496+0000) 2020-02-05T22:44:51.561 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:51.559+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.560329+0000) 2020-02-05T22:44:51.561 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:51.559+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.560588+0000) 2020-02-05T22:44:51.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:51.560+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.560766+0000) 2020-02-05T22:44:51.562 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:51.560+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.561062+0000) 2020-02-05T22:44:51.562 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:51.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.560922+0000) 2020-02-05T22:44:51.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:51.561+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.561847+0000) 2020-02-05T22:44:51.563 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:51.561+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.562171+0000) 2020-02-05T22:44:51.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:51.562+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.562518+0000) 2020-02-05T22:44:51.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:51.560+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.561821+0000) 2020-02-05T22:44:51.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:51.561+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.562258+0000) 2020-02-05T22:44:51.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:51.564+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.565951+0000) 2020-02-05T22:44:51.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:51.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:51.567250+0000) 2020-02-05T22:44:51.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:51.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:52.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:52.353+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:52.354028+0000) 2020-02-05T22:44:52.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:52.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:52.709571+0000) 2020-02-05T22:44:52.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:52.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:52.746295+0000) 2020-02-05T22:44:53.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:53.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:53.868140+0000) 2020-02-05T22:44:53.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:53.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:53.868140+0000) 2020-02-05T22:44:55.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:55.967+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:55.968028+0000) 2020-02-05T22:44:56.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:56.576+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.578060+0000) 2020-02-05T22:44:56.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:44:56.577+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.578869+0000) 2020-02-05T22:44:56.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:56.578+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.579399+0000) 2020-02-05T22:44:56.581 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:44:56.579+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.579979+0000) 2020-02-05T22:44:56.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:56.579+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.580529+0000) 2020-02-05T22:44:56.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:44:56.579+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.580958+0000) 2020-02-05T22:44:56.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:44:56.580+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.581282+0000) 2020-02-05T22:44:56.582 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:56.581+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.582015+0000) 2020-02-05T22:44:56.583 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:56.581+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.582340+0000) 2020-02-05T22:44:56.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:44:56.582+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.583222+0000) 2020-02-05T22:44:56.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:44:56.583+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.584283+0000) 2020-02-05T22:44:56.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:44:56.583+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.584870+0000) 2020-02-05T22:44:56.586 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:44:56.584+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.585183+0000) 2020-02-05T22:44:56.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:56.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:56.588928+0000) 2020-02-05T22:44:56.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:44:56.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:44:57.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:44:57.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:57.108222+0000) 2020-02-05T22:44:57.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:44:57.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:57.108488+0000) 2020-02-05T22:44:57.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:44:57.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:57.108694+0000) 2020-02-05T22:44:57.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:57.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:57.843459+0000) 2020-02-05T22:44:59.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:44:59.495+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:44:59.496787+0000) 2020-02-05T22:45:00.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:00.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:00.103600+0000) 2020-02-05T22:45:00.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:00.339+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:00.340021+0000) 2020-02-05T22:45:00.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:00.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:00.355383+0000) 2020-02-05T22:45:01.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:01.598+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.599848+0000) 2020-02-05T22:45:01.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:01.600+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.600484+0000) 2020-02-05T22:45:01.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:01.600+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.601239+0000) 2020-02-05T22:45:01.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:01.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.601480+0000) 2020-02-05T22:45:01.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:01.601+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.603008+0000) 2020-02-05T22:45:01.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:01.602+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.602772+0000) 2020-02-05T22:45:01.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:01.602+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.603446+0000) 2020-02-05T22:45:01.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:01.604+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.605546+0000) 2020-02-05T22:45:01.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:01.604+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.605713+0000) 2020-02-05T22:45:01.608 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:01.605+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.605718+0000) 2020-02-05T22:45:01.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:01.605+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.605934+0000) 2020-02-05T22:45:01.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:01.605+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.606086+0000) 2020-02-05T22:45:01.608 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:01.605+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.606369+0000) 2020-02-05T22:45:01.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:01.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:01.609480+0000) 2020-02-05T22:45:01.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:01.613+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:02.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:02.353+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:02.354227+0000) 2020-02-05T22:45:02.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:02.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:02.709820+0000) 2020-02-05T22:45:02.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:02.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:02.746532+0000) 2020-02-05T22:45:03.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:03.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:03.868422+0000) 2020-02-05T22:45:03.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:03.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:03.868422+0000) 2020-02-05T22:45:05.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:05.968+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:05.968277+0000) 2020-02-05T22:45:06.606 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:45:06.612 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:45:06.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:06.618+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.619923+0000) 2020-02-05T22:45:06.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:06.619+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.620772+0000) 2020-02-05T22:45:06.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:06.620+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.620731+0000) 2020-02-05T22:45:06.622 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:06.620+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.621594+0000) 2020-02-05T22:45:06.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:06.620+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.621272+0000) 2020-02-05T22:45:06.623 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:06.621+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.621640+0000) 2020-02-05T22:45:06.623 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:06.621+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.622327+0000) 2020-02-05T22:45:06.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:06.621+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.622170+0000) 2020-02-05T22:45:06.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:06.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.622734+0000) 2020-02-05T22:45:06.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:06.622+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.622748+0000) 2020-02-05T22:45:06.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:06.623+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.623858+0000) 2020-02-05T22:45:06.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:06.622+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.623615+0000) 2020-02-05T22:45:06.625 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:06.624+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.624876+0000) 2020-02-05T22:45:06.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:06.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:06.628472+0000) 2020-02-05T22:45:06.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:06.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124408 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:07.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:07.108+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:07.108471+0000) 2020-02-05T22:45:07.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:07.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:07.108715+0000) 2020-02-05T22:45:07.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:07.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:07.108882+0000) 2020-02-05T22:45:07.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:07.850+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:07.851301+0000) 2020-02-05T22:45:09.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:09.496+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:09.497022+0000) 2020-02-05T22:45:10.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:10.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:10.103850+0000) 2020-02-05T22:45:10.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:10.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:10.340270+0000) 2020-02-05T22:45:10.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:10.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:10.355612+0000) 2020-02-05T22:45:11.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:11.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.642323+0000) 2020-02-05T22:45:11.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:11.641+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.642583+0000) 2020-02-05T22:45:11.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:11.641+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.642934+0000) 2020-02-05T22:45:11.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:11.643+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.643655+0000) 2020-02-05T22:45:11.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:11.642+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.643373+0000) 2020-02-05T22:45:11.645 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:11.643+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.644098+0000) 2020-02-05T22:45:11.645 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:11.644+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.644668+0000) 2020-02-05T22:45:11.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:11.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.645051+0000) 2020-02-05T22:45:11.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:11.644+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.645226+0000) 2020-02-05T22:45:11.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:11.646+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.646640+0000) 2020-02-05T22:45:11.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:11.645+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.646791+0000) 2020-02-05T22:45:11.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:11.646+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.647014+0000) 2020-02-05T22:45:11.648 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:11.646+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.647294+0000) 2020-02-05T22:45:11.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:11.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:11.650481+0000) 2020-02-05T22:45:11.657 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:11.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:12.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:12.354+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:12.354501+0000) 2020-02-05T22:45:12.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:12.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:12.710111+0000) 2020-02-05T22:45:12.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:12.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:12.746762+0000) 2020-02-05T22:45:13.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:13.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:13.868692+0000) 2020-02-05T22:45:13.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:13.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:13.868693+0000) 2020-02-05T22:45:15.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:15.968+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:15.968538+0000) 2020-02-05T22:45:16.663 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:16.661+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.662309+0000) 2020-02-05T22:45:16.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:16.661+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.662854+0000) 2020-02-05T22:45:16.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:16.662+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.663943+0000) 2020-02-05T22:45:16.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:16.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.664411+0000) 2020-02-05T22:45:16.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:16.664+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.664540+0000) 2020-02-05T22:45:16.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:16.664+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.664764+0000) 2020-02-05T22:45:16.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:16.664+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.665611+0000) 2020-02-05T22:45:16.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:16.665+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.666126+0000) 2020-02-05T22:45:16.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:16.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.666424+0000) 2020-02-05T22:45:16.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:16.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.666767+0000) 2020-02-05T22:45:16.668 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:16.666+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.666643+0000) 2020-02-05T22:45:16.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:16.667+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.668145+0000) 2020-02-05T22:45:16.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:16.666+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.667725+0000) 2020-02-05T22:45:16.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:16.673+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:16.674839+0000) 2020-02-05T22:45:16.677 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:16.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:17.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:17.108+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:17.108698+0000) 2020-02-05T22:45:17.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:17.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:17.108902+0000) 2020-02-05T22:45:17.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:17.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:17.109096+0000) 2020-02-05T22:45:17.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:17.858+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:17.859458+0000) 2020-02-05T22:45:19.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:19.496+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:19.497314+0000) 2020-02-05T22:45:20.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:20.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:20.104103+0000) 2020-02-05T22:45:20.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:20.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:20.340512+0000) 2020-02-05T22:45:20.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:20.354+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:20.355902+0000) 2020-02-05T22:45:21.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:21.680+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.681838+0000) 2020-02-05T22:45:21.683 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:21.681+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.682261+0000) 2020-02-05T22:45:21.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:21.683+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.683670+0000) 2020-02-05T22:45:21.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:21.682+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.683975+0000) 2020-02-05T22:45:21.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:21.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.684693+0000) 2020-02-05T22:45:21.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:21.683+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.684824+0000) 2020-02-05T22:45:21.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:21.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.685614+0000) 2020-02-05T22:45:21.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:21.685+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.686462+0000) 2020-02-05T22:45:21.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:21.685+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.686693+0000) 2020-02-05T22:45:21.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:21.686+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.686760+0000) 2020-02-05T22:45:21.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:21.686+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.687085+0000) 2020-02-05T22:45:21.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:21.687+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.687484+0000) 2020-02-05T22:45:21.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:21.687+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.687728+0000) 2020-02-05T22:45:21.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:21.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:21.694366+0000) 2020-02-05T22:45:21.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:21.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:22.357 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:22.354+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:22.354724+0000) 2020-02-05T22:45:22.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:22.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:22.710367+0000) 2020-02-05T22:45:22.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:22.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:22.746999+0000) 2020-02-05T22:45:23.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:23.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:23.868970+0000) 2020-02-05T22:45:23.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:23.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:23.868970+0000) 2020-02-05T22:45:25.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:25.967+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:25.968788+0000) 2020-02-05T22:45:26.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:26.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.700344+0000) 2020-02-05T22:45:26.702 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:26.699+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.701145+0000) 2020-02-05T22:45:26.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:26.701+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.702817+0000) 2020-02-05T22:45:26.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:26.702+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.703687+0000) 2020-02-05T22:45:26.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:26.703+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.704235+0000) 2020-02-05T22:45:26.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:26.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.704797+0000) 2020-02-05T22:45:26.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:26.704+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.705802+0000) 2020-02-05T22:45:26.707 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:26.704+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.706003+0000) 2020-02-05T22:45:26.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:26.705+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.706856+0000) 2020-02-05T22:45:26.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:26.705+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.707028+0000) 2020-02-05T22:45:26.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:26.706+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.707642+0000) 2020-02-05T22:45:26.708 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:26.705+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.707129+0000) 2020-02-05T22:45:26.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:26.710+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.712006+0000) 2020-02-05T22:45:26.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:26.711+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:26.712384+0000) 2020-02-05T22:45:26.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:26.715+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:27.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:27.107+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:27.108940+0000) 2020-02-05T22:45:27.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:27.107+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:27.109087+0000) 2020-02-05T22:45:27.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:27.107+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:27.109288+0000) 2020-02-05T22:45:27.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:27.867+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:27.867656+0000) 2020-02-05T22:45:29.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:29.497+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:29.497571+0000) 2020-02-05T22:45:30.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:30.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:30.104358+0000) 2020-02-05T22:45:30.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:30.339+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:30.340761+0000) 2020-02-05T22:45:30.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:30.355+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:30.356136+0000) 2020-02-05T22:45:31.723 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:31.720+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.722048+0000) 2020-02-05T22:45:31.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:31.722+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.723136+0000) 2020-02-05T22:45:31.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:31.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.723711+0000) 2020-02-05T22:45:31.725 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:31.722+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.724192+0000) 2020-02-05T22:45:31.726 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:31.723+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.724654+0000) 2020-02-05T22:45:31.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:31.723+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.725013+0000) 2020-02-05T22:45:31.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:31.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.726551+0000) 2020-02-05T22:45:31.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:31.725+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.726626+0000) 2020-02-05T22:45:31.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:31.725+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.727058+0000) 2020-02-05T22:45:31.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:31.726+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.727502+0000) 2020-02-05T22:45:31.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:31.726+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.728102+0000) 2020-02-05T22:45:31.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:31.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.728786+0000) 2020-02-05T22:45:31.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:31.729+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.730498+0000) 2020-02-05T22:45:31.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:31.730+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:31.731380+0000) 2020-02-05T22:45:31.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:31.736+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:32.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:32.353+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:32.354964+0000) 2020-02-05T22:45:32.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:32.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:32.710598+0000) 2020-02-05T22:45:32.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:32.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:32.747281+0000) 2020-02-05T22:45:33.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:33.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:33.869277+0000) 2020-02-05T22:45:33.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:33.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:33.869277+0000) 2020-02-05T22:45:35.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:35.968+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:35.969037+0000) 2020-02-05T22:45:36.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:36.743+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.744741+0000) 2020-02-05T22:45:36.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:36.742+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.744373+0000) 2020-02-05T22:45:36.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:36.744+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.745134+0000) 2020-02-05T22:45:36.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:36.744+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.745713+0000) 2020-02-05T22:45:36.747 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:36.744+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.746176+0000) 2020-02-05T22:45:36.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:36.745+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.746135+0000) 2020-02-05T22:45:36.748 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:36.745+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.746921+0000) 2020-02-05T22:45:36.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:36.747+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.748729+0000) 2020-02-05T22:45:36.750 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:36.746+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.748107+0000) 2020-02-05T22:45:36.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:36.748+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.750320+0000) 2020-02-05T22:45:36.751 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:36.749+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.750497+0000) 2020-02-05T22:45:36.752 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:36.749+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.750647+0000) 2020-02-05T22:45:36.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:36.753+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.754597+0000) 2020-02-05T22:45:36.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:36.753+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:36.754816+0000) 2020-02-05T22:45:36.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:36.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:36.777 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:45:36.782 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:45:37.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:37.108+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:37.109168+0000) 2020-02-05T22:45:37.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:37.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:37.109281+0000) 2020-02-05T22:45:37.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:37.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:37.109492+0000) 2020-02-05T22:45:37.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:37.875+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:37.875535+0000) 2020-02-05T22:45:39.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:39.497+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:39.497829+0000) 2020-02-05T22:45:40.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:40.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:40.104593+0000) 2020-02-05T22:45:40.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:40.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:40.341026+0000) 2020-02-05T22:45:40.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:40.356+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:40.356361+0000) 2020-02-05T22:45:41.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:41.763+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.764931+0000) 2020-02-05T22:45:41.767 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:41.765+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.765495+0000) 2020-02-05T22:45:41.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:41.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.766197+0000) 2020-02-05T22:45:41.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:41.766+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.766612+0000) 2020-02-05T22:45:41.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:41.765+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.767247+0000) 2020-02-05T22:45:41.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:41.768+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.768556+0000) 2020-02-05T22:45:41.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:41.766+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.768285+0000) 2020-02-05T22:45:41.770 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:41.769+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.769452+0000) 2020-02-05T22:45:41.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:41.770+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.770214+0000) 2020-02-05T22:45:41.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:41.769+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.770639+0000) 2020-02-05T22:45:41.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:41.769+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.770869+0000) 2020-02-05T22:45:41.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:41.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.771457+0000) 2020-02-05T22:45:41.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:41.769+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.771311+0000) 2020-02-05T22:45:41.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:41.776+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:41.776828+0000) 2020-02-05T22:45:41.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:41.779+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:42.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:42.354+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:42.355214+0000) 2020-02-05T22:45:42.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:42.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:42.710802+0000) 2020-02-05T22:45:42.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:42.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:42.747521+0000) 2020-02-05T22:45:43.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:43.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:43.869550+0000) 2020-02-05T22:45:43.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:43.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:43.869550+0000) 2020-02-05T22:45:45.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:45.968+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:45.969304+0000) 2020-02-05T22:45:46.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:46.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.786685+0000) 2020-02-05T22:45:46.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:46.786+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.787142+0000) 2020-02-05T22:45:46.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:46.786+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.787693+0000) 2020-02-05T22:45:46.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:46.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.788161+0000) 2020-02-05T22:45:46.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:46.787+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.788407+0000) 2020-02-05T22:45:46.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:46.787+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.788879+0000) 2020-02-05T22:45:46.790 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:46.787+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.788892+0000) 2020-02-05T22:45:46.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:46.788+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.789350+0000) 2020-02-05T22:45:46.791 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:46.788+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.789871+0000) 2020-02-05T22:45:46.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:46.788+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.790279+0000) 2020-02-05T22:45:46.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:46.789+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.790640+0000) 2020-02-05T22:45:46.792 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:46.789+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.791098+0000) 2020-02-05T22:45:46.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:46.790+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.791776+0000) 2020-02-05T22:45:46.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:46.794+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:46.795233+0000) 2020-02-05T22:45:46.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:46.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:47.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:47.108+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:47.109502+0000) 2020-02-05T22:45:47.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:47.108+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:47.109415+0000) 2020-02-05T22:45:47.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:47.108+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:47.109667+0000) 2020-02-05T22:45:47.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:47.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:47.883816+0000) 2020-02-05T22:45:49.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:49.497+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:49.498135+0000) 2020-02-05T22:45:50.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:50.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:50.104821+0000) 2020-02-05T22:45:50.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:50.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:50.341259+0000) 2020-02-05T22:45:50.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:50.355+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:50.356595+0000) 2020-02-05T22:45:51.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:51.805+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.806281+0000) 2020-02-05T22:45:51.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:51.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.806803+0000) 2020-02-05T22:45:51.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:51.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.807326+0000) 2020-02-05T22:45:51.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:51.806+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.807785+0000) 2020-02-05T22:45:51.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:51.807+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.808367+0000) 2020-02-05T22:45:51.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:51.807+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.808761+0000) 2020-02-05T22:45:51.810 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:51.807+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.808937+0000) 2020-02-05T22:45:51.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:51.808+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.809699+0000) 2020-02-05T22:45:51.811 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:51.808+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.809823+0000) 2020-02-05T22:45:51.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:51.809+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.810579+0000) 2020-02-05T22:45:51.811 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:51.809+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.810792+0000) 2020-02-05T22:45:51.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:51.809+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.811100+0000) 2020-02-05T22:45:51.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:51.810+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.811445+0000) 2020-02-05T22:45:51.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:51.815+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:51.816124+0000) 2020-02-05T22:45:51.820 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:51.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:52.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:52.354+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:52.355504+0000) 2020-02-05T22:45:52.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:52.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:52.711009+0000) 2020-02-05T22:45:52.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:52.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:52.747748+0000) 2020-02-05T22:45:53.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:53.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:53.869837+0000) 2020-02-05T22:45:53.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:53.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:53.869837+0000) 2020-02-05T22:45:55.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:55.968+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:55.969553+0000) 2020-02-05T22:45:56.826 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:45:56.823+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.825136+0000) 2020-02-05T22:45:56.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:56.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.825835+0000) 2020-02-05T22:45:56.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:45:56.826+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.827197+0000) 2020-02-05T22:45:56.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:45:56.826+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.827635+0000) 2020-02-05T22:45:56.829 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:45:56.827+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.828520+0000) 2020-02-05T22:45:56.829 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:45:56.827+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.828748+0000) 2020-02-05T22:45:56.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:56.828+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.829222+0000) 2020-02-05T22:45:56.831 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:56.828+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.829522+0000) 2020-02-05T22:45:56.831 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:45:56.828+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.829761+0000) 2020-02-05T22:45:56.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:56.828+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.830083+0000) 2020-02-05T22:45:56.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:45:56.830+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.831360+0000) 2020-02-05T22:45:56.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:45:56.830+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.831503+0000) 2020-02-05T22:45:56.832 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:56.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.831392+0000) 2020-02-05T22:45:56.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:56.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:56.836968+0000) 2020-02-05T22:45:56.842 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:45:56.839+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124818 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:45:57.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:45:57.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:57.109756+0000) 2020-02-05T22:45:57.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:45:57.109+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:57.109755+0000) 2020-02-05T22:45:57.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:45:57.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:57.109909+0000) 2020-02-05T22:45:57.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:57.891+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:57.891913+0000) 2020-02-05T22:45:59.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:45:59.497+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:45:59.498373+0000) 2020-02-05T22:46:00.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:00.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:00.105040+0000) 2020-02-05T22:46:00.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:00.340+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:00.341519+0000) 2020-02-05T22:46:00.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:00.355+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:00.356780+0000) 2020-02-05T22:46:01.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:01.845+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.846388+0000) 2020-02-05T22:46:01.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:01.845+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.847200+0000) 2020-02-05T22:46:01.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:01.847+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.848564+0000) 2020-02-05T22:46:01.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:01.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.849503+0000) 2020-02-05T22:46:01.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:01.849+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.850647+0000) 2020-02-05T22:46:01.852 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:01.850+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.851443+0000) 2020-02-05T22:46:01.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:01.850+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.851864+0000) 2020-02-05T22:46:01.853 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:01.851+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.852549+0000) 2020-02-05T22:46:01.853 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:01.850+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.852363+0000) 2020-02-05T22:46:01.854 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:01.851+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.852870+0000) 2020-02-05T22:46:01.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:01.852+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.853088+0000) 2020-02-05T22:46:01.854 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:01.851+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.853192+0000) 2020-02-05T22:46:01.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:01.855+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.856140+0000) 2020-02-05T22:46:01.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:01.855+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:01.856344+0000) 2020-02-05T22:46:01.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:01.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:02.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:02.355+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:02.355744+0000) 2020-02-05T22:46:02.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:02.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:02.711239+0000) 2020-02-05T22:46:02.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:02.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:02.747962+0000) 2020-02-05T22:46:03.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:03.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:03.870127+0000) 2020-02-05T22:46:03.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:03.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:03.870127+0000) 2020-02-05T22:46:05.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:05.969+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:05.969781+0000) 2020-02-05T22:46:06.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:06.865+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.866738+0000) 2020-02-05T22:46:06.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:06.867+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.868260+0000) 2020-02-05T22:46:06.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:06.868+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.869175+0000) 2020-02-05T22:46:06.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:06.868+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.869827+0000) 2020-02-05T22:46:06.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:06.868+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.869855+0000) 2020-02-05T22:46:06.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:06.869+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.870531+0000) 2020-02-05T22:46:06.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:06.869+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.871164+0000) 2020-02-05T22:46:06.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:06.869+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.871378+0000) 2020-02-05T22:46:06.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:06.870+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.872240+0000) 2020-02-05T22:46:06.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:06.871+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.872660+0000) 2020-02-05T22:46:06.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:06.872+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.873410+0000) 2020-02-05T22:46:06.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:06.872+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.873453+0000) 2020-02-05T22:46:06.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:06.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.875306+0000) 2020-02-05T22:46:06.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:06.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:06.875577+0000) 2020-02-05T22:46:06.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:06.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124899 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:06.940 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:46:06.946 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:46:07.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:07.109+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:07.110011+0000) 2020-02-05T22:46:07.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:07.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:07.110010+0000) 2020-02-05T22:46:07.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:07.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:07.110180+0000) 2020-02-05T22:46:07.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:07.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:07.903497+0000) 2020-02-05T22:46:09.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:09.498+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:09.498607+0000) 2020-02-05T22:46:10.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:10.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:10.105347+0000) 2020-02-05T22:46:10.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:10.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:10.341768+0000) 2020-02-05T22:46:10.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:10.356+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:10.357034+0000) 2020-02-05T22:46:11.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:11.884+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.885633+0000) 2020-02-05T22:46:11.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:11.885+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.887247+0000) 2020-02-05T22:46:11.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:11.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.887669+0000) 2020-02-05T22:46:11.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:11.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.888333+0000) 2020-02-05T22:46:11.889 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:11.887+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.888751+0000) 2020-02-05T22:46:11.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:11.889+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.890291+0000) 2020-02-05T22:46:11.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:11.889+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.890758+0000) 2020-02-05T22:46:11.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:11.889+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.890805+0000) 2020-02-05T22:46:11.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:11.890+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.891633+0000) 2020-02-05T22:46:11.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:11.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.891887+0000) 2020-02-05T22:46:11.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:11.890+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.892232+0000) 2020-02-05T22:46:11.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:11.891+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.892649+0000) 2020-02-05T22:46:11.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:11.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.893754+0000) 2020-02-05T22:46:11.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:11.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:11.894057+0000) 2020-02-05T22:46:11.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:11.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124940 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:12.357 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:12.354+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:12.355986+0000) 2020-02-05T22:46:12.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:12.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:12.711406+0000) 2020-02-05T22:46:12.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:12.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:12.748203+0000) 2020-02-05T22:46:13.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:13.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:13.870407+0000) 2020-02-05T22:46:13.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:13.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:13.870449+0000) 2020-02-05T22:46:15.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:15.969+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:15.970026+0000) 2020-02-05T22:46:16.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:16.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.907574+0000) 2020-02-05T22:46:16.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:16.907+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.908352+0000) 2020-02-05T22:46:16.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:16.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.908830+0000) 2020-02-05T22:46:16.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:16.908+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.909919+0000) 2020-02-05T22:46:16.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:16.908+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.910263+0000) 2020-02-05T22:46:16.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:16.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.911027+0000) 2020-02-05T22:46:16.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:16.910+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.911242+0000) 2020-02-05T22:46:16.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:16.911+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.912679+0000) 2020-02-05T22:46:16.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:16.912+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.913479+0000) 2020-02-05T22:46:16.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:16.910+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.912248+0000) 2020-02-05T22:46:16.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:16.911+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.912854+0000) 2020-02-05T22:46:16.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:16.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.913958+0000) 2020-02-05T22:46:16.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:16.917+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.918166+0000) 2020-02-05T22:46:16.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:16.918+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:16.919404+0000) 2020-02-05T22:46:16.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:16.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 124980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:17.111 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:17.109+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:17.110295+0000) 2020-02-05T22:46:17.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:17.109+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:17.110293+0000) 2020-02-05T22:46:17.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:17.109+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:17.110495+0000) 2020-02-05T22:46:17.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:17.911+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:17.911882+0000) 2020-02-05T22:46:19.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:19.497+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:19.498876+0000) 2020-02-05T22:46:20.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:20.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:20.105582+0000) 2020-02-05T22:46:20.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:20.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:20.342024+0000) 2020-02-05T22:46:20.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:20.356+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:20.357301+0000) 2020-02-05T22:46:21.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:21.927+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.928555+0000) 2020-02-05T22:46:21.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:21.928+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.929124+0000) 2020-02-05T22:46:21.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:21.929+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.930627+0000) 2020-02-05T22:46:21.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:21.930+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.931112+0000) 2020-02-05T22:46:21.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:21.931+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.931857+0000) 2020-02-05T22:46:21.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:21.931+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.932699+0000) 2020-02-05T22:46:21.934 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:21.932+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.933122+0000) 2020-02-05T22:46:21.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:21.932+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.933574+0000) 2020-02-05T22:46:21.935 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:21.932+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.933052+0000) 2020-02-05T22:46:21.935 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:21.933+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.934450+0000) 2020-02-05T22:46:21.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:21.933+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.934469+0000) 2020-02-05T22:46:21.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:21.933+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.934888+0000) 2020-02-05T22:46:21.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:21.934+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.935040+0000) 2020-02-05T22:46:21.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:21.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:21.939742+0000) 2020-02-05T22:46:21.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:21.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:22.357 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:22.355+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:22.356217+0000) 2020-02-05T22:46:22.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:22.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:22.711616+0000) 2020-02-05T22:46:22.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:22.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:22.748450+0000) 2020-02-05T22:46:23.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:23.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:23.870681+0000) 2020-02-05T22:46:23.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:23.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:23.870665+0000) 2020-02-05T22:46:25.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:25.969+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:25.970259+0000) 2020-02-05T22:46:26.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:26.949+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.949520+0000) 2020-02-05T22:46:26.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:26.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.950046+0000) 2020-02-05T22:46:26.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:26.950+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.950920+0000) 2020-02-05T22:46:26.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:26.950+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.951882+0000) 2020-02-05T22:46:26.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:26.951+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.951930+0000) 2020-02-05T22:46:26.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:26.952+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.953083+0000) 2020-02-05T22:46:26.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:26.953+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.954281+0000) 2020-02-05T22:46:26.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:26.953+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.954691+0000) 2020-02-05T22:46:26.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:26.955+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.955588+0000) 2020-02-05T22:46:26.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:26.955+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.956477+0000) 2020-02-05T22:46:26.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:26.955+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.956542+0000) 2020-02-05T22:46:26.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:26.955+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.956706+0000) 2020-02-05T22:46:26.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:26.955+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.956915+0000) 2020-02-05T22:46:26.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:26.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:26.958856+0000) 2020-02-05T22:46:26.968 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:26.967+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:27.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:27.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:27.110561+0000) 2020-02-05T22:46:27.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:27.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:27.110559+0000) 2020-02-05T22:46:27.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:27.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:27.110742+0000) 2020-02-05T22:46:27.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:27.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:27.923735+0000) 2020-02-05T22:46:29.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:29.498+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:29.499136+0000) 2020-02-05T22:46:30.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:30.105+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:30.105825+0000) 2020-02-05T22:46:30.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:30.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:30.342262+0000) 2020-02-05T22:46:30.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:30.356+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:30.357528+0000) 2020-02-05T22:46:31.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:31.972+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.973192+0000) 2020-02-05T22:46:31.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:31.972+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.974044+0000) 2020-02-05T22:46:31.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:31.974+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.975280+0000) 2020-02-05T22:46:31.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:31.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.975823+0000) 2020-02-05T22:46:31.977 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:31.975+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.976245+0000) 2020-02-05T22:46:31.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:31.975+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.976543+0000) 2020-02-05T22:46:31.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:31.976+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.977032+0000) 2020-02-05T22:46:31.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:31.976+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.977346+0000) 2020-02-05T22:46:31.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:31.977+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.977780+0000) 2020-02-05T22:46:31.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:31.977+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.978403+0000) 2020-02-05T22:46:31.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:31.978+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.979245+0000) 2020-02-05T22:46:31.980 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:31.978+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.978790+0000) 2020-02-05T22:46:31.980 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:31.978+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.978816+0000) 2020-02-05T22:46:31.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:31.986+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:31.986860+0000) 2020-02-05T22:46:31.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:31.988+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:32.357 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:32.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:32.356490+0000) 2020-02-05T22:46:32.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:32.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:32.711853+0000) 2020-02-05T22:46:32.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:32.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:32.748658+0000) 2020-02-05T22:46:33.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:33.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:33.870911+0000) 2020-02-05T22:46:33.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:33.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:33.870954+0000) 2020-02-05T22:46:35.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:35.970+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:35.970516+0000) 2020-02-05T22:46:36.995 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:36.993+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.993584+0000) 2020-02-05T22:46:36.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:36.994+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.994349+0000) 2020-02-05T22:46:36.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:36.994+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.995020+0000) 2020-02-05T22:46:36.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:36.996+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.996118+0000) 2020-02-05T22:46:36.997 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:36.996+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.996533+0000) 2020-02-05T22:46:36.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:36.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.997008+0000) 2020-02-05T22:46:36.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:36.997+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.997299+0000) 2020-02-05T22:46:36.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:36.997+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.998235+0000) 2020-02-05T22:46:36.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:36.997+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.998130+0000) 2020-02-05T22:46:37.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:36.998+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:36.998907+0000) 2020-02-05T22:46:37.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:36.999+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.000206+0000) 2020-02-05T22:46:37.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:37.000+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.000423+0000) 2020-02-05T22:46:37.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:37.000+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.000794+0000) 2020-02-05T22:46:37.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:37.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.005144+0000) 2020-02-05T22:46:37.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:37.009+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:37.087 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:46:37.093 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:46:37.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:37.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.110777+0000) 2020-02-05T22:46:37.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:37.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.110779+0000) 2020-02-05T22:46:37.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:37.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.110938+0000) 2020-02-05T22:46:37.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:37.932+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:37.933645+0000) 2020-02-05T22:46:39.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:39.498+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:39.499371+0000) 2020-02-05T22:46:40.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:40.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:40.106052+0000) 2020-02-05T22:46:40.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:40.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:40.342498+0000) 2020-02-05T22:46:40.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:40.356+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:40.357759+0000) 2020-02-05T22:46:42.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:42.016+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.016958+0000) 2020-02-05T22:46:42.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:42.016+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.016902+0000) 2020-02-05T22:46:42.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:42.016+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.017324+0000) 2020-02-05T22:46:42.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:42.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.017892+0000) 2020-02-05T22:46:42.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:42.017+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.018386+0000) 2020-02-05T22:46:42.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:42.017+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.017779+0000) 2020-02-05T22:46:42.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:42.017+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.018560+0000) 2020-02-05T22:46:42.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:42.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.018280+0000) 2020-02-05T22:46:42.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:42.017+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.019038+0000) 2020-02-05T22:46:42.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:42.018+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.018542+0000) 2020-02-05T22:46:42.021 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:42.018+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.019371+0000) 2020-02-05T22:46:42.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:42.019+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.020325+0000) 2020-02-05T22:46:42.022 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:42.021+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.021466+0000) 2020-02-05T22:46:42.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:42.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.023950+0000) 2020-02-05T22:46:42.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:42.029+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:42.357 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:42.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.356663+0000) 2020-02-05T22:46:42.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:42.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.712068+0000) 2020-02-05T22:46:42.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:42.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:42.748869+0000) 2020-02-05T22:46:43.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:43.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:43.871176+0000) 2020-02-05T22:46:43.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:43.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:43.871144+0000) 2020-02-05T22:46:45.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:45.970+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:45.970769+0000) 2020-02-05T22:46:47.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:47.033+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.034887+0000) 2020-02-05T22:46:47.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:47.035+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.036653+0000) 2020-02-05T22:46:47.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:47.036+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.036702+0000) 2020-02-05T22:46:47.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:47.036+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.037193+0000) 2020-02-05T22:46:47.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:47.036+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.037468+0000) 2020-02-05T22:46:47.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:47.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.038170+0000) 2020-02-05T22:46:47.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:47.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.038576+0000) 2020-02-05T22:46:47.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:47.038+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.038865+0000) 2020-02-05T22:46:47.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:47.039+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.039762+0000) 2020-02-05T22:46:47.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:47.039+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.040401+0000) 2020-02-05T22:46:47.041 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:47.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.040112+0000) 2020-02-05T22:46:47.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:47.040+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.040853+0000) 2020-02-05T22:46:47.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:47.040+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.041620+0000) 2020-02-05T22:46:47.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:47.045+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.046687+0000) 2020-02-05T22:46:47.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:47.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:47.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:47.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.110963+0000) 2020-02-05T22:46:47.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:47.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.111029+0000) 2020-02-05T22:46:47.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:47.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.111085+0000) 2020-02-05T22:46:47.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:47.939+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:47.940594+0000) 2020-02-05T22:46:49.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:49.498+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:49.499648+0000) 2020-02-05T22:46:50.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:50.105+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:50.106329+0000) 2020-02-05T22:46:50.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:50.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:50.342734+0000) 2020-02-05T22:46:50.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:50.357+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:50.357987+0000) 2020-02-05T22:46:52.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:52.053+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.054474+0000) 2020-02-05T22:46:52.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:52.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.056520+0000) 2020-02-05T22:46:52.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:52.056+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.056744+0000) 2020-02-05T22:46:52.058 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:52.056+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.057344+0000) 2020-02-05T22:46:52.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:52.058+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.059292+0000) 2020-02-05T22:46:52.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:52.058+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.059825+0000) 2020-02-05T22:46:52.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:52.059+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.060142+0000) 2020-02-05T22:46:52.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:52.059+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.060389+0000) 2020-02-05T22:46:52.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:52.060+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.061187+0000) 2020-02-05T22:46:52.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:52.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.061764+0000) 2020-02-05T22:46:52.063 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:52.061+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.062294+0000) 2020-02-05T22:46:52.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:52.061+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.062090+0000) 2020-02-05T22:46:52.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:52.059+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.059605+0000) 2020-02-05T22:46:52.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:52.063+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.064491+0000) 2020-02-05T22:46:52.073 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:52.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:52.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:52.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.356880+0000) 2020-02-05T22:46:52.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:52.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.712342+0000) 2020-02-05T22:46:52.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:52.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:52.749070+0000) 2020-02-05T22:46:53.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:53.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:53.871409+0000) 2020-02-05T22:46:53.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:53.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:53.871400+0000) 2020-02-05T22:46:55.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:55.970+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:55.971008+0000) 2020-02-05T22:46:57.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:57.076+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.077644+0000) 2020-02-05T22:46:57.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:57.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.078373+0000) 2020-02-05T22:46:57.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:46:57.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.079054+0000) 2020-02-05T22:46:57.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:46:57.077+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.079341+0000) 2020-02-05T22:46:57.080 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:46:57.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.080009+0000) 2020-02-05T22:46:57.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:57.079+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.080620+0000) 2020-02-05T22:46:57.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:46:57.079+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.080588+0000) 2020-02-05T22:46:57.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:57.079+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.081316+0000) 2020-02-05T22:46:57.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:46:57.080+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.081708+0000) 2020-02-05T22:46:57.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:46:57.080+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.081696+0000) 2020-02-05T22:46:57.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:46:57.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.082281+0000) 2020-02-05T22:46:57.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:46:57.081+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.082957+0000) 2020-02-05T22:46:57.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:57.082+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.083686+0000) 2020-02-05T22:46:57.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:57.088+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.089244+0000) 2020-02-05T22:46:57.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:46:57.090+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:46:57.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:46:57.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.111178+0000) 2020-02-05T22:46:57.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:46:57.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.111278+0000) 2020-02-05T22:46:57.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:46:57.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.111313+0000) 2020-02-05T22:46:57.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:57.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:57.949130+0000) 2020-02-05T22:46:59.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:46:59.498+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:46:59.499916+0000) 2020-02-05T22:47:00.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:00.105+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:00.106556+0000) 2020-02-05T22:47:00.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:00.341+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:00.342976+0000) 2020-02-05T22:47:00.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:00.357+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:00.358276+0000) 2020-02-05T22:47:02.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:02.097+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.098227+0000) 2020-02-05T22:47:02.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:02.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.099426+0000) 2020-02-05T22:47:02.101 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:02.099+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.100505+0000) 2020-02-05T22:47:02.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:02.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.100885+0000) 2020-02-05T22:47:02.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:02.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.101308+0000) 2020-02-05T22:47:02.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:02.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.101943+0000) 2020-02-05T22:47:02.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:02.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.102549+0000) 2020-02-05T22:47:02.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:02.101+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.102518+0000) 2020-02-05T22:47:02.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:02.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.103182+0000) 2020-02-05T22:47:02.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:02.102+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.103667+0000) 2020-02-05T22:47:02.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:02.102+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.103941+0000) 2020-02-05T22:47:02.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:02.103+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.104540+0000) 2020-02-05T22:47:02.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:02.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.104373+0000) 2020-02-05T22:47:02.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:02.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.110494+0000) 2020-02-05T22:47:02.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:02.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:02.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:02.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.357130+0000) 2020-02-05T22:47:02.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:02.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.712537+0000) 2020-02-05T22:47:02.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:02.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:02.749303+0000) 2020-02-05T22:47:03.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:03.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:03.871668+0000) 2020-02-05T22:47:03.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:03.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:03.871676+0000) 2020-02-05T22:47:05.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:05.969+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:05.971245+0000) 2020-02-05T22:47:07.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:07.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.111493+0000) 2020-02-05T22:47:07.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:07.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.111502+0000) 2020-02-05T22:47:07.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:07.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.111544+0000) 2020-02-05T22:47:07.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:07.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.119211+0000) 2020-02-05T22:47:07.120 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:07.118+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.119766+0000) 2020-02-05T22:47:07.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:07.119+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.120153+0000) 2020-02-05T22:47:07.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:07.119+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.120654+0000) 2020-02-05T22:47:07.122 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:07.120+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.121485+0000) 2020-02-05T22:47:07.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:07.120+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.121896+0000) 2020-02-05T22:47:07.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:07.121+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.122497+0000) 2020-02-05T22:47:07.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:07.120+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.122367+0000) 2020-02-05T22:47:07.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:07.121+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.122930+0000) 2020-02-05T22:47:07.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:07.121+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.123176+0000) 2020-02-05T22:47:07.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:07.121+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.123020+0000) 2020-02-05T22:47:07.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:07.121+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.123371+0000) 2020-02-05T22:47:07.125 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:07.122+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.124006+0000) 2020-02-05T22:47:07.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:07.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.129252+0000) 2020-02-05T22:47:07.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:07.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125385 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:07.283 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:47:07.290 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:47:07.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:07.957+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:07.957575+0000) 2020-02-05T22:47:09.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:09.499+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:09.500152+0000) 2020-02-05T22:47:10.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:10.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:10.106782+0000) 2020-02-05T22:47:10.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:10.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:10.343227+0000) 2020-02-05T22:47:10.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:10.358+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:10.358504+0000) 2020-02-05T22:47:12.139 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:12.137+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.138213+0000) 2020-02-05T22:47:12.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:12.137+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.138659+0000) 2020-02-05T22:47:12.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:12.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.139264+0000) 2020-02-05T22:47:12.140 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:12.138+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.139789+0000) 2020-02-05T22:47:12.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:12.138+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.139805+0000) 2020-02-05T22:47:12.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:12.138+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.140324+0000) 2020-02-05T22:47:12.141 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:12.139+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.140609+0000) 2020-02-05T22:47:12.142 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:12.140+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.141404+0000) 2020-02-05T22:47:12.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:12.139+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.141136+0000) 2020-02-05T22:47:12.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:12.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.142168+0000) 2020-02-05T22:47:12.143 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:12.141+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.142674+0000) 2020-02-05T22:47:12.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:12.141+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.142502+0000) 2020-02-05T22:47:12.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:12.142+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.143474+0000) 2020-02-05T22:47:12.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:12.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.148029+0000) 2020-02-05T22:47:12.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:12.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125426 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:12.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:12.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.357407+0000) 2020-02-05T22:47:12.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:12.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.712743+0000) 2020-02-05T22:47:12.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:12.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:12.749522+0000) 2020-02-05T22:47:13.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:13.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:13.871930+0000) 2020-02-05T22:47:13.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:13.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:13.871952+0000) 2020-02-05T22:47:15.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:15.970+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:15.971496+0000) 2020-02-05T22:47:17.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:17.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.111759+0000) 2020-02-05T22:47:17.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:17.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.111767+0000) 2020-02-05T22:47:17.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:17.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.111830+0000) 2020-02-05T22:47:17.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:17.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.157127+0000) 2020-02-05T22:47:17.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:17.157+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.158792+0000) 2020-02-05T22:47:17.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:17.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.159498+0000) 2020-02-05T22:47:17.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:17.158+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.159814+0000) 2020-02-05T22:47:17.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:17.159+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.160775+0000) 2020-02-05T22:47:17.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:17.160+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.161738+0000) 2020-02-05T22:47:17.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:17.160+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.161553+0000) 2020-02-05T22:47:17.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:17.160+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.162021+0000) 2020-02-05T22:47:17.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:17.161+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.162508+0000) 2020-02-05T22:47:17.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:17.161+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.163130+0000) 2020-02-05T22:47:17.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:17.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.163411+0000) 2020-02-05T22:47:17.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:17.162+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.163649+0000) 2020-02-05T22:47:17.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:17.162+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.163874+0000) 2020-02-05T22:47:17.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:17.168+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.169440+0000) 2020-02-05T22:47:17.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:17.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:17.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:17.969+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:17.969566+0000) 2020-02-05T22:47:19.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:19.500+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:19.500381+0000) 2020-02-05T22:47:20.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:20.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:20.107035+0000) 2020-02-05T22:47:20.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:20.342+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:20.343485+0000) 2020-02-05T22:47:20.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:20.358+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:20.358730+0000) 2020-02-05T22:47:22.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:22.178+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.179171+0000) 2020-02-05T22:47:22.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:22.178+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.179630+0000) 2020-02-05T22:47:22.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:22.178+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.180101+0000) 2020-02-05T22:47:22.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:22.179+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.180437+0000) 2020-02-05T22:47:22.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:22.179+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.180905+0000) 2020-02-05T22:47:22.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:22.180+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.181215+0000) 2020-02-05T22:47:22.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:22.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.183958+0000) 2020-02-05T22:47:22.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:22.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.184844+0000) 2020-02-05T22:47:22.187 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:22.183+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.184380+0000) 2020-02-05T22:47:22.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:22.184+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.185437+0000) 2020-02-05T22:47:22.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:22.180+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.181809+0000) 2020-02-05T22:47:22.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:22.181+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.182416+0000) 2020-02-05T22:47:22.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:22.182+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.184374+0000) 2020-02-05T22:47:22.195 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:22.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:22.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:22.194+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.195126+0000) 2020-02-05T22:47:22.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:22.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.357667+0000) 2020-02-05T22:47:22.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:22.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.712961+0000) 2020-02-05T22:47:22.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:22.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:22.749756+0000) 2020-02-05T22:47:23.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:23.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:23.872190+0000) 2020-02-05T22:47:23.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:23.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:23.872189+0000) 2020-02-05T22:47:25.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:25.971+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:25.971741+0000) 2020-02-05T22:47:27.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:27.110+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.112010+0000) 2020-02-05T22:47:27.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:27.110+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.112019+0000) 2020-02-05T22:47:27.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:27.110+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.112126+0000) 2020-02-05T22:47:27.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:27.199+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.200160+0000) 2020-02-05T22:47:27.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:27.199+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.200584+0000) 2020-02-05T22:47:27.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:27.199+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.200625+0000) 2020-02-05T22:47:27.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:27.200+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.201291+0000) 2020-02-05T22:47:27.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:27.200+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.201936+0000) 2020-02-05T22:47:27.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:27.201+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.202395+0000) 2020-02-05T22:47:27.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:27.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.203065+0000) 2020-02-05T22:47:27.204 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:27.202+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.203445+0000) 2020-02-05T22:47:27.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:27.202+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.203865+0000) 2020-02-05T22:47:27.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:27.203+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.204775+0000) 2020-02-05T22:47:27.206 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:27.203+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.204695+0000) 2020-02-05T22:47:27.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:27.204+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.205770+0000) 2020-02-05T22:47:27.206 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:27.204+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.205922+0000) 2020-02-05T22:47:27.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:27.205+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.206696+0000) 2020-02-05T22:47:27.216 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:27.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:27.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:27.978+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:27.978140+0000) 2020-02-05T22:47:29.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:29.500+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:29.500624+0000) 2020-02-05T22:47:30.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:30.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:30.107301+0000) 2020-02-05T22:47:30.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:30.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:30.343716+0000) 2020-02-05T22:47:30.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:30.358+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:30.358955+0000) 2020-02-05T22:47:32.222 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:32.220+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.221384+0000) 2020-02-05T22:47:32.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:32.220+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.222259+0000) 2020-02-05T22:47:32.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:32.221+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.222802+0000) 2020-02-05T22:47:32.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:32.222+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.223739+0000) 2020-02-05T22:47:32.225 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:32.223+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.224545+0000) 2020-02-05T22:47:32.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:32.223+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.225202+0000) 2020-02-05T22:47:32.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:32.224+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.225607+0000) 2020-02-05T22:47:32.227 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:32.225+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.226361+0000) 2020-02-05T22:47:32.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:32.225+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.226882+0000) 2020-02-05T22:47:32.228 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:32.225+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.226735+0000) 2020-02-05T22:47:32.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:32.226+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.227210+0000) 2020-02-05T22:47:32.229 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:32.226+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.227358+0000) 2020-02-05T22:47:32.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:32.226+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.227747+0000) 2020-02-05T22:47:32.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:32.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.232037+0000) 2020-02-05T22:47:32.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:32.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:32.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:32.356+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.357936+0000) 2020-02-05T22:47:32.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:32.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.713175+0000) 2020-02-05T22:47:32.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:32.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:32.749964+0000) 2020-02-05T22:47:33.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:33.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:33.872470+0000) 2020-02-05T22:47:33.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:33.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:33.872448+0000) 2020-02-05T22:47:35.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:35.971+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:35.971988+0000) 2020-02-05T22:47:37.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:37.111+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.112260+0000) 2020-02-05T22:47:37.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:37.111+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.112296+0000) 2020-02-05T22:47:37.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:37.111+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.112394+0000) 2020-02-05T22:47:37.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:37.241+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.242435+0000) 2020-02-05T22:47:37.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:37.240+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.242242+0000) 2020-02-05T22:47:37.246 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:37.243+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.244929+0000) 2020-02-05T22:47:37.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:37.243+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.245273+0000) 2020-02-05T22:47:37.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:37.244+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.245968+0000) 2020-02-05T22:47:37.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:37.245+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.246415+0000) 2020-02-05T22:47:37.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:37.245+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.246674+0000) 2020-02-05T22:47:37.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:37.245+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.246862+0000) 2020-02-05T22:47:37.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:37.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.247466+0000) 2020-02-05T22:47:37.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:37.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.248654+0000) 2020-02-05T22:47:37.250 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:37.247+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.248126+0000) 2020-02-05T22:47:37.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:37.246+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.247814+0000) 2020-02-05T22:47:37.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:37.246+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.248145+0000) 2020-02-05T22:47:37.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:37.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.250377+0000) 2020-02-05T22:47:37.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:37.256+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:37.484 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:47:37.490 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:47:37.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:37.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:37.989603+0000) 2020-02-05T22:47:39.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:39.499+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:39.500857+0000) 2020-02-05T22:47:40.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:40.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:40.107530+0000) 2020-02-05T22:47:40.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:40.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:40.343966+0000) 2020-02-05T22:47:40.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:40.358+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:40.359296+0000) 2020-02-05T22:47:42.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:42.262+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.263777+0000) 2020-02-05T22:47:42.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:42.263+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.264266+0000) 2020-02-05T22:47:42.266 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:42.263+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.264689+0000) 2020-02-05T22:47:42.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:42.263+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.265331+0000) 2020-02-05T22:47:42.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:42.265+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.266484+0000) 2020-02-05T22:47:42.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:42.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.267149+0000) 2020-02-05T22:47:42.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:42.266+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.267153+0000) 2020-02-05T22:47:42.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:42.266+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.267973+0000) 2020-02-05T22:47:42.270 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:42.268+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.269158+0000) 2020-02-05T22:47:42.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:42.268+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.269597+0000) 2020-02-05T22:47:42.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:42.268+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.269703+0000) 2020-02-05T22:47:42.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:42.268+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.269877+0000) 2020-02-05T22:47:42.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:42.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.274015+0000) 2020-02-05T22:47:42.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:42.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.274347+0000) 2020-02-05T22:47:42.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:42.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:42.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:42.357+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.358156+0000) 2020-02-05T22:47:42.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:42.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.713410+0000) 2020-02-05T22:47:42.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:42.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:42.750182+0000) 2020-02-05T22:47:43.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:43.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:43.872705+0000) 2020-02-05T22:47:43.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:43.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:43.872688+0000) 2020-02-05T22:47:45.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:45.971+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:45.972219+0000) 2020-02-05T22:47:47.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:47.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.112512+0000) 2020-02-05T22:47:47.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:47.112+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.112603+0000) 2020-02-05T22:47:47.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:47.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.112697+0000) 2020-02-05T22:47:47.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:47.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.285464+0000) 2020-02-05T22:47:47.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:47.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.285963+0000) 2020-02-05T22:47:47.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:47.285+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.286475+0000) 2020-02-05T22:47:47.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:47.285+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.286935+0000) 2020-02-05T22:47:47.288 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:47.286+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.287207+0000) 2020-02-05T22:47:47.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:47.285+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.286571+0000) 2020-02-05T22:47:47.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:47.285+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.287287+0000) 2020-02-05T22:47:47.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:47.286+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.287510+0000) 2020-02-05T22:47:47.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:47.287+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.288504+0000) 2020-02-05T22:47:47.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:47.287+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.289356+0000) 2020-02-05T22:47:47.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:47.288+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.289660+0000) 2020-02-05T22:47:47.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:47.288+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.289573+0000) 2020-02-05T22:47:47.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:47.289+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.290263+0000) 2020-02-05T22:47:47.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:47.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.295664+0000) 2020-02-05T22:47:47.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:47.297+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:47.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:47.996+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:47.997028+0000) 2020-02-05T22:47:49.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:49.500+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:49.501129+0000) 2020-02-05T22:47:50.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:50.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:50.107759+0000) 2020-02-05T22:47:50.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:50.343+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:50.344217+0000) 2020-02-05T22:47:50.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:50.358+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:50.359527+0000) 2020-02-05T22:47:52.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:52.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.304315+0000) 2020-02-05T22:47:52.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:52.303+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.304839+0000) 2020-02-05T22:47:52.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:52.304+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.305279+0000) 2020-02-05T22:47:52.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:52.305+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.305752+0000) 2020-02-05T22:47:52.307 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:52.304+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.306016+0000) 2020-02-05T22:47:52.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:52.306+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.307123+0000) 2020-02-05T22:47:52.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:52.307+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.308036+0000) 2020-02-05T22:47:52.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:52.308+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.308520+0000) 2020-02-05T22:47:52.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:52.307+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.308749+0000) 2020-02-05T22:47:52.310 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:52.308+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.309142+0000) 2020-02-05T22:47:52.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:52.308+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.309671+0000) 2020-02-05T22:47:52.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:52.310+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.310846+0000) 2020-02-05T22:47:52.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:52.310+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.311401+0000) 2020-02-05T22:47:52.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:52.312+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.314059+0000) 2020-02-05T22:47:52.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:52.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125747 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:52.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:52.357+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.358383+0000) 2020-02-05T22:47:52.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:52.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.713628+0000) 2020-02-05T22:47:52.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:52.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:52.750410+0000) 2020-02-05T22:47:53.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:53.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:53.872946+0000) 2020-02-05T22:47:53.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:53.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:53.872955+0000) 2020-02-05T22:47:55.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:55.972+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:55.972493+0000) 2020-02-05T22:47:57.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:57.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.112763+0000) 2020-02-05T22:47:57.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:57.112+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.112851+0000) 2020-02-05T22:47:57.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:57.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.112969+0000) 2020-02-05T22:47:57.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:47:57.323+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.324968+0000) 2020-02-05T22:47:57.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:57.325+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.326371+0000) 2020-02-05T22:47:57.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:57.325+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.326671+0000) 2020-02-05T22:47:57.328 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:47:57.326+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.327130+0000) 2020-02-05T22:47:57.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:47:57.326+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.326939+0000) 2020-02-05T22:47:57.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:47:57.327+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.327668+0000) 2020-02-05T22:47:57.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:47:57.327+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.328065+0000) 2020-02-05T22:47:57.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:47:57.327+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.328339+0000) 2020-02-05T22:47:57.330 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:47:57.328+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.329137+0000) 2020-02-05T22:47:57.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:47:57.328+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.329199+0000) 2020-02-05T22:47:57.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:47:57.328+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.329371+0000) 2020-02-05T22:47:57.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:47:57.330+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.330479+0000) 2020-02-05T22:47:57.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:47:57.329+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.331045+0000) 2020-02-05T22:47:57.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:57.335+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:57.336712+0000) 2020-02-05T22:47:57.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:47:57.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:47:58.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:58.004+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:58.005543+0000) 2020-02-05T22:47:59.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:47:59.500+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:47:59.501373+0000) 2020-02-05T22:48:00.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:00.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:00.108008+0000) 2020-02-05T22:48:00.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:00.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:00.344493+0000) 2020-02-05T22:48:00.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:00.359+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:00.359751+0000) 2020-02-05T22:48:02.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:02.346+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.347175+0000) 2020-02-05T22:48:02.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:02.346+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.347770+0000) 2020-02-05T22:48:02.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:02.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.349123+0000) 2020-02-05T22:48:02.350 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:02.348+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.349419+0000) 2020-02-05T22:48:02.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:02.348+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.349647+0000) 2020-02-05T22:48:02.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:02.348+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.349952+0000) 2020-02-05T22:48:02.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:02.348+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.350024+0000) 2020-02-05T22:48:02.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:02.350+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.350621+0000) 2020-02-05T22:48:02.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:02.350+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.350606+0000) 2020-02-05T22:48:02.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:02.350+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.351158+0000) 2020-02-05T22:48:02.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:02.351+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.351805+0000) 2020-02-05T22:48:02.353 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:02.351+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.352001+0000) 2020-02-05T22:48:02.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:02.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.354702+0000) 2020-02-05T22:48:02.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:02.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.354921+0000) 2020-02-05T22:48:02.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:02.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.358571+0000) 2020-02-05T22:48:02.362 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:02.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:02.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:02.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.713843+0000) 2020-02-05T22:48:02.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:02.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:02.750638+0000) 2020-02-05T22:48:03.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:03.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:03.873199+0000) 2020-02-05T22:48:03.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:03.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:03.873262+0000) 2020-02-05T22:48:05.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:05.972+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:05.972673+0000) 2020-02-05T22:48:07.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:07.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.113018+0000) 2020-02-05T22:48:07.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:07.112+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.113109+0000) 2020-02-05T22:48:07.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:07.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.113236+0000) 2020-02-05T22:48:07.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:07.367+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.368281+0000) 2020-02-05T22:48:07.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:07.367+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.368861+0000) 2020-02-05T22:48:07.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:07.368+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.369337+0000) 2020-02-05T22:48:07.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:07.368+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.369813+0000) 2020-02-05T22:48:07.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:07.370+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.370586+0000) 2020-02-05T22:48:07.372 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:07.369+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.370806+0000) 2020-02-05T22:48:07.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:07.370+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.371440+0000) 2020-02-05T22:48:07.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:07.371+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.371646+0000) 2020-02-05T22:48:07.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:07.371+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.371969+0000) 2020-02-05T22:48:07.374 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:07.371+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.372233+0000) 2020-02-05T22:48:07.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:07.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.373269+0000) 2020-02-05T22:48:07.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:07.372+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.373046+0000) 2020-02-05T22:48:07.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:07.372+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.373315+0000) 2020-02-05T22:48:07.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:07.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:07.378399+0000) 2020-02-05T22:48:07.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:07.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:07.644 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:48:07.651 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:48:08.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:08.013+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:08.013962+0000) 2020-02-05T22:48:09.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:09.501+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:09.501604+0000) 2020-02-05T22:48:10.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:10.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:10.108270+0000) 2020-02-05T22:48:10.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:10.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:10.344754+0000) 2020-02-05T22:48:10.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:10.359+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:10.359980+0000) 2020-02-05T22:48:12.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:12.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.358817+0000) 2020-02-05T22:48:12.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:12.386+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.387631+0000) 2020-02-05T22:48:12.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:12.387+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.388731+0000) 2020-02-05T22:48:12.390 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:12.388+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.389462+0000) 2020-02-05T22:48:12.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:12.389+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.389702+0000) 2020-02-05T22:48:12.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:12.389+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.390135+0000) 2020-02-05T22:48:12.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:12.390+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.391056+0000) 2020-02-05T22:48:12.392 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:12.390+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.391501+0000) 2020-02-05T22:48:12.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:12.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.391655+0000) 2020-02-05T22:48:12.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:12.391+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.392364+0000) 2020-02-05T22:48:12.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:12.391+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.392988+0000) 2020-02-05T22:48:12.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:12.392+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.392621+0000) 2020-02-05T22:48:12.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:12.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.394429+0000) 2020-02-05T22:48:12.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:12.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.397966+0000) 2020-02-05T22:48:12.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:12.397+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.398292+0000) 2020-02-05T22:48:12.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:12.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:12.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:12.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.714105+0000) 2020-02-05T22:48:12.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:12.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:12.750846+0000) 2020-02-05T22:48:13.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:13.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:13.873482+0000) 2020-02-05T22:48:13.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:13.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:13.873492+0000) 2020-02-05T22:48:15.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:15.972+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:15.972924+0000) 2020-02-05T22:48:17.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:17.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.113320+0000) 2020-02-05T22:48:17.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:17.113+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.113352+0000) 2020-02-05T22:48:17.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:17.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.113520+0000) 2020-02-05T22:48:17.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:17.407+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.408349+0000) 2020-02-05T22:48:17.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:17.409+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.410867+0000) 2020-02-05T22:48:17.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:17.410+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.411458+0000) 2020-02-05T22:48:17.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:17.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.412006+0000) 2020-02-05T22:48:17.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:17.411+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.412557+0000) 2020-02-05T22:48:17.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:17.411+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.413021+0000) 2020-02-05T22:48:17.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:17.412+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.413251+0000) 2020-02-05T22:48:17.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:17.413+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.413653+0000) 2020-02-05T22:48:17.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:17.413+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.413740+0000) 2020-02-05T22:48:17.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:17.413+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.414080+0000) 2020-02-05T22:48:17.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:17.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.414671+0000) 2020-02-05T22:48:17.416 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:17.414+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.414500+0000) 2020-02-05T22:48:17.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:17.414+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.414569+0000) 2020-02-05T22:48:17.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:17.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:17.416672+0000) 2020-02-05T22:48:17.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:17.423+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:18.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:18.021+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:18.021924+0000) 2020-02-05T22:48:19.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:19.501+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:19.501823+0000) 2020-02-05T22:48:20.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:20.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:20.108503+0000) 2020-02-05T22:48:20.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:20.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:20.345005+0000) 2020-02-05T22:48:20.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:20.360+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:20.360206+0000) 2020-02-05T22:48:22.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:22.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.359049+0000) 2020-02-05T22:48:22.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:22.430+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.430189+0000) 2020-02-05T22:48:22.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:22.430+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.430998+0000) 2020-02-05T22:48:22.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:22.430+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.431405+0000) 2020-02-05T22:48:22.433 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:22.431+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.431797+0000) 2020-02-05T22:48:22.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:22.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.432168+0000) 2020-02-05T22:48:22.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:22.432+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.432484+0000) 2020-02-05T22:48:22.434 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:22.432+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.432883+0000) 2020-02-05T22:48:22.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:22.433+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.433584+0000) 2020-02-05T22:48:22.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:22.433+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.433874+0000) 2020-02-05T22:48:22.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:22.433+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.434089+0000) 2020-02-05T22:48:22.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:22.433+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.434402+0000) 2020-02-05T22:48:22.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:22.434+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.435222+0000) 2020-02-05T22:48:22.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:22.435+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.435835+0000) 2020-02-05T22:48:22.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:22.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.441013+0000) 2020-02-05T22:48:22.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:22.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 125984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:22.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:22.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.714357+0000) 2020-02-05T22:48:22.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:22.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:22.751048+0000) 2020-02-05T22:48:23.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:23.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:23.873705+0000) 2020-02-05T22:48:23.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:23.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:23.873720+0000) 2020-02-05T22:48:25.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:25.973+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:25.973171+0000) 2020-02-05T22:48:27.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:27.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.113568+0000) 2020-02-05T22:48:27.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:27.113+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.113631+0000) 2020-02-05T22:48:27.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:27.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.113779+0000) 2020-02-05T22:48:27.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:27.449+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.450176+0000) 2020-02-05T22:48:27.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:27.449+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.450568+0000) 2020-02-05T22:48:27.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:27.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.452458+0000) 2020-02-05T22:48:27.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:27.452+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.453266+0000) 2020-02-05T22:48:27.454 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:27.452+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.453695+0000) 2020-02-05T22:48:27.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:27.453+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.454412+0000) 2020-02-05T22:48:27.456 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:27.454+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.455057+0000) 2020-02-05T22:48:27.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:27.454+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.454626+0000) 2020-02-05T22:48:27.458 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:27.454+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.455143+0000) 2020-02-05T22:48:27.458 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:27.454+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.455487+0000) 2020-02-05T22:48:27.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:27.455+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.456229+0000) 2020-02-05T22:48:27.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:27.456+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.456614+0000) 2020-02-05T22:48:27.459 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:27.456+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.457228+0000) 2020-02-05T22:48:27.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:27.463+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:27.464429+0000) 2020-02-05T22:48:27.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:27.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:28.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:28.029+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:28.030497+0000) 2020-02-05T22:48:29.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:29.501+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:29.502127+0000) 2020-02-05T22:48:30.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:30.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:30.108738+0000) 2020-02-05T22:48:30.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:30.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:30.345258+0000) 2020-02-05T22:48:30.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:30.359+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:30.360486+0000) 2020-02-05T22:48:32.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:32.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.359304+0000) 2020-02-05T22:48:32.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:32.471+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.472781+0000) 2020-02-05T22:48:32.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:32.472+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.473946+0000) 2020-02-05T22:48:32.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:32.472+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.473723+0000) 2020-02-05T22:48:32.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:32.472+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.473472+0000) 2020-02-05T22:48:32.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:32.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.474603+0000) 2020-02-05T22:48:32.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:32.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.475557+0000) 2020-02-05T22:48:32.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:32.473+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.475126+0000) 2020-02-05T22:48:32.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:32.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.475242+0000) 2020-02-05T22:48:32.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:32.475+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.476458+0000) 2020-02-05T22:48:32.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:32.475+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.476406+0000) 2020-02-05T22:48:32.478 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:32.475+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.476816+0000) 2020-02-05T22:48:32.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:32.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.477340+0000) 2020-02-05T22:48:32.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:32.476+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.477309+0000) 2020-02-05T22:48:32.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:32.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.481917+0000) 2020-02-05T22:48:32.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:32.485+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:32.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:32.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.714590+0000) 2020-02-05T22:48:32.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:32.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:32.751242+0000) 2020-02-05T22:48:33.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:33.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:33.873975+0000) 2020-02-05T22:48:33.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:33.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:33.873979+0000) 2020-02-05T22:48:35.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:35.971+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:35.973394+0000) 2020-02-05T22:48:37.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:37.112+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.113779+0000) 2020-02-05T22:48:37.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:37.112+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.113876+0000) 2020-02-05T22:48:37.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:37.112+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.114048+0000) 2020-02-05T22:48:37.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:37.490+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.492148+0000) 2020-02-05T22:48:37.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:37.491+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.492583+0000) 2020-02-05T22:48:37.494 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:37.492+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.493682+0000) 2020-02-05T22:48:37.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:37.493+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.494950+0000) 2020-02-05T22:48:37.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:37.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.495383+0000) 2020-02-05T22:48:37.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:37.494+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.495677+0000) 2020-02-05T22:48:37.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:37.494+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.495995+0000) 2020-02-05T22:48:37.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:37.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.496190+0000) 2020-02-05T22:48:37.497 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:37.495+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.496481+0000) 2020-02-05T22:48:37.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:37.495+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.496976+0000) 2020-02-05T22:48:37.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:37.496+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.497210+0000) 2020-02-05T22:48:37.499 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:37.496+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.498243+0000) 2020-02-05T22:48:37.499 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:37.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.498311+0000) 2020-02-05T22:48:37.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:37.504+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:37.505716+0000) 2020-02-05T22:48:37.508 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:37.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:37.840 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:48:37.847 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:48:38.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:38.037+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:38.038499+0000) 2020-02-05T22:48:39.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:39.501+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:39.502350+0000) 2020-02-05T22:48:40.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:40.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:40.108999+0000) 2020-02-05T22:48:40.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:40.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:40.345516+0000) 2020-02-05T22:48:40.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:40.359+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:40.360744+0000) 2020-02-05T22:48:42.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:42.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.359517+0000) 2020-02-05T22:48:42.514 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:42.511+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.513394+0000) 2020-02-05T22:48:42.515 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:42.513+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.514208+0000) 2020-02-05T22:48:42.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:42.513+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.514412+0000) 2020-02-05T22:48:42.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:42.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.515011+0000) 2020-02-05T22:48:42.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:42.514+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.515160+0000) 2020-02-05T22:48:42.516 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:42.514+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.515443+0000) 2020-02-05T22:48:42.517 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:42.514+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.515933+0000) 2020-02-05T22:48:42.517 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:42.514+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.515795+0000) 2020-02-05T22:48:42.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:42.514+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.516053+0000) 2020-02-05T22:48:42.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:42.515+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.516380+0000) 2020-02-05T22:48:42.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:42.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.516698+0000) 2020-02-05T22:48:42.518 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:42.515+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.517135+0000) 2020-02-05T22:48:42.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:42.516+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.517449+0000) 2020-02-05T22:48:42.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:42.522+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.523389+0000) 2020-02-05T22:48:42.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:42.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:42.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:42.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.714844+0000) 2020-02-05T22:48:42.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:42.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:42.751397+0000) 2020-02-05T22:48:43.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:43.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:43.874222+0000) 2020-02-05T22:48:43.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:43.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:43.874216+0000) 2020-02-05T22:48:45.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:45.972+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:45.973622+0000) 2020-02-05T22:48:47.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:47.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.114019+0000) 2020-02-05T22:48:47.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:47.113+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.114121+0000) 2020-02-05T22:48:47.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:47.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.114301+0000) 2020-02-05T22:48:47.535 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:47.532+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.534413+0000) 2020-02-05T22:48:47.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:47.533+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.534908+0000) 2020-02-05T22:48:47.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:47.534+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.535223+0000) 2020-02-05T22:48:47.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:47.534+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.535838+0000) 2020-02-05T22:48:47.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:47.533+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.535396+0000) 2020-02-05T22:48:47.537 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:47.534+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.536220+0000) 2020-02-05T22:48:47.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:47.535+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.536379+0000) 2020-02-05T22:48:47.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:47.535+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.536481+0000) 2020-02-05T22:48:47.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:47.536+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.537666+0000) 2020-02-05T22:48:47.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:47.537+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.538428+0000) 2020-02-05T22:48:47.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:47.537+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.538933+0000) 2020-02-05T22:48:47.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:47.537+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.539337+0000) 2020-02-05T22:48:47.540 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:47.538+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.539200+0000) 2020-02-05T22:48:47.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:47.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126177 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:47.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:47.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:47.545626+0000) 2020-02-05T22:48:48.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:48.045+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:48.046620+0000) 2020-02-05T22:48:49.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:49.501+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:49.502594+0000) 2020-02-05T22:48:50.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:50.108+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:50.109266+0000) 2020-02-05T22:48:50.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:50.344+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:50.345753+0000) 2020-02-05T22:48:50.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:50.360+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:50.360984+0000) 2020-02-05T22:48:52.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:52.358+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.359773+0000) 2020-02-05T22:48:52.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:52.550+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.551647+0000) 2020-02-05T22:48:52.553 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:52.551+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.552644+0000) 2020-02-05T22:48:52.554 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:52.552+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.553608+0000) 2020-02-05T22:48:52.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:52.553+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.554598+0000) 2020-02-05T22:48:52.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:52.554+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.555251+0000) 2020-02-05T22:48:52.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:52.554+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.555750+0000) 2020-02-05T22:48:52.556 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:52.555+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.556117+0000) 2020-02-05T22:48:52.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:52.554+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.556401+0000) 2020-02-05T22:48:52.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:52.556+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.557153+0000) 2020-02-05T22:48:52.558 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:52.556+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.557319+0000) 2020-02-05T22:48:52.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:52.555+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.557126+0000) 2020-02-05T22:48:52.559 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:52.557+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.558227+0000) 2020-02-05T22:48:52.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:52.556+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.558284+0000) 2020-02-05T22:48:52.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:52.562+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.563399+0000) 2020-02-05T22:48:52.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:52.566+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:52.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:52.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.715073+0000) 2020-02-05T22:48:52.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:52.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:52.751602+0000) 2020-02-05T22:48:53.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:53.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:53.874491+0000) 2020-02-05T22:48:53.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:53.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:53.874491+0000) 2020-02-05T22:48:55.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:55.973+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:55.973879+0000) 2020-02-05T22:48:57.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:57.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.114271+0000) 2020-02-05T22:48:57.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:57.113+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.114366+0000) 2020-02-05T22:48:57.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:57.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.114515+0000) 2020-02-05T22:48:57.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:48:57.571+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.573323+0000) 2020-02-05T22:48:57.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:57.572+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.573811+0000) 2020-02-05T22:48:57.575 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:48:57.573+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.574244+0000) 2020-02-05T22:48:57.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:57.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.574934+0000) 2020-02-05T22:48:57.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:48:57.573+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.575286+0000) 2020-02-05T22:48:57.576 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:48:57.574+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.575938+0000) 2020-02-05T22:48:57.577 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:48:57.575+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.576298+0000) 2020-02-05T22:48:57.577 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:48:57.575+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.576914+0000) 2020-02-05T22:48:57.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:48:57.577+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.578222+0000) 2020-02-05T22:48:57.580 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:48:57.577+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.579004+0000) 2020-02-05T22:48:57.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:48:57.577+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.579022+0000) 2020-02-05T22:48:57.580 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:48:57.577+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.579388+0000) 2020-02-05T22:48:57.580 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:48:57.578+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.579339+0000) 2020-02-05T22:48:57.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:57.580+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:57.581691+0000) 2020-02-05T22:48:57.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:48:57.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:48:58.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:58.057+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:58.058451+0000) 2020-02-05T22:48:59.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:48:59.502+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:48:59.502826+0000) 2020-02-05T22:49:00.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:00.108+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:00.109449+0000) 2020-02-05T22:49:00.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:00.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:00.346009+0000) 2020-02-05T22:49:00.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:00.360+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:00.361231+0000) 2020-02-05T22:49:02.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:02.359+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.359972+0000) 2020-02-05T22:49:02.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:02.592+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.594200+0000) 2020-02-05T22:49:02.596 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:02.593+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.594791+0000) 2020-02-05T22:49:02.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:02.594+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.595647+0000) 2020-02-05T22:49:02.597 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:02.594+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.595558+0000) 2020-02-05T22:49:02.597 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:02.594+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.595910+0000) 2020-02-05T22:49:02.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:02.595+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.597013+0000) 2020-02-05T22:49:02.598 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:02.596+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.597526+0000) 2020-02-05T22:49:02.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:02.596+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.597930+0000) 2020-02-05T22:49:02.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:02.596+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.597960+0000) 2020-02-05T22:49:02.599 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:02.596+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.598054+0000) 2020-02-05T22:49:02.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:02.597+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.598520+0000) 2020-02-05T22:49:02.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:02.597+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.598812+0000) 2020-02-05T22:49:02.600 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:02.597+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.599009+0000) 2020-02-05T22:49:02.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:02.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.604141+0000) 2020-02-05T22:49:02.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:02.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:02.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:02.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.715346+0000) 2020-02-05T22:49:02.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:02.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:02.751827+0000) 2020-02-05T22:49:03.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:03.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:03.874732+0000) 2020-02-05T22:49:03.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:03.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:03.874729+0000) 2020-02-05T22:49:05.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:05.973+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:05.974142+0000) 2020-02-05T22:49:07.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:07.113+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.114525+0000) 2020-02-05T22:49:07.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:07.113+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.114660+0000) 2020-02-05T22:49:07.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:07.113+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.114762+0000) 2020-02-05T22:49:07.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:07.613+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.615023+0000) 2020-02-05T22:49:07.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:07.615+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.616421+0000) 2020-02-05T22:49:07.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:07.615+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.616769+0000) 2020-02-05T22:49:07.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:07.616+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.617488+0000) 2020-02-05T22:49:07.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:07.616+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.617904+0000) 2020-02-05T22:49:07.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:07.617+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.618419+0000) 2020-02-05T22:49:07.619 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:07.617+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.618563+0000) 2020-02-05T22:49:07.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:07.618+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.619699+0000) 2020-02-05T22:49:07.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:07.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.620153+0000) 2020-02-05T22:49:07.622 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:07.619+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.620335+0000) 2020-02-05T22:49:07.622 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:07.619+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.620580+0000) 2020-02-05T22:49:07.622 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:07.619+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.620872+0000) 2020-02-05T22:49:07.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:07.619+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.620838+0000) 2020-02-05T22:49:07.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:07.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:07.626961+0000) 2020-02-05T22:49:07.631 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:07.629+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:07.994 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:49:08.001 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:49:08.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:08.065+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:08.066393+0000) 2020-02-05T22:49:09.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:09.502+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:09.503104+0000) 2020-02-05T22:49:10.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:10.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:10.109692+0000) 2020-02-05T22:49:10.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:10.345+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:10.346267+0000) 2020-02-05T22:49:10.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:10.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:10.361478+0000) 2020-02-05T22:49:12.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:12.359+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.360237+0000) 2020-02-05T22:49:12.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:12.635+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.637277+0000) 2020-02-05T22:49:12.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:12.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.638286+0000) 2020-02-05T22:49:12.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:12.637+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.639015+0000) 2020-02-05T22:49:12.640 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:12.637+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.638931+0000) 2020-02-05T22:49:12.641 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:12.637+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.639380+0000) 2020-02-05T22:49:12.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:12.639+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.640584+0000) 2020-02-05T22:49:12.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:12.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.641017+0000) 2020-02-05T22:49:12.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:12.640+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.641404+0000) 2020-02-05T22:49:12.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:12.640+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.641706+0000) 2020-02-05T22:49:12.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:12.640+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.641748+0000) 2020-02-05T22:49:12.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:12.640+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.642415+0000) 2020-02-05T22:49:12.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:12.641+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.642179+0000) 2020-02-05T22:49:12.644 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:12.641+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.642631+0000) 2020-02-05T22:49:12.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:12.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.647568+0000) 2020-02-05T22:49:12.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:12.650+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:12.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:12.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.715606+0000) 2020-02-05T22:49:12.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:12.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:12.752038+0000) 2020-02-05T22:49:13.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:13.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:13.874974+0000) 2020-02-05T22:49:13.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:13.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:13.875001+0000) 2020-02-05T22:49:15.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:15.973+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:15.974376+0000) 2020-02-05T22:49:17.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:17.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.114766+0000) 2020-02-05T22:49:17.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:17.114+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.114880+0000) 2020-02-05T22:49:17.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:17.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.114962+0000) 2020-02-05T22:49:17.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:17.658+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.658335+0000) 2020-02-05T22:49:17.660 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:17.658+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.658752+0000) 2020-02-05T22:49:17.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:17.659+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.659134+0000) 2020-02-05T22:49:17.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:17.659+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.659701+0000) 2020-02-05T22:49:17.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:17.659+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.659721+0000) 2020-02-05T22:49:17.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:17.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.660300+0000) 2020-02-05T22:49:17.662 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:17.659+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.660795+0000) 2020-02-05T22:49:17.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:17.659+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.661122+0000) 2020-02-05T22:49:17.663 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:17.660+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.662163+0000) 2020-02-05T22:49:17.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:17.661+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.662602+0000) 2020-02-05T22:49:17.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:17.661+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.663086+0000) 2020-02-05T22:49:17.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:17.662+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.663576+0000) 2020-02-05T22:49:17.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:17.664+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.664118+0000) 2020-02-05T22:49:17.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:17.667+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:17.667409+0000) 2020-02-05T22:49:17.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:17.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:18.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:18.073+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:18.073117+0000) 2020-02-05T22:49:19.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:19.503+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:19.503333+0000) 2020-02-05T22:49:20.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:20.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:20.109953+0000) 2020-02-05T22:49:20.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:20.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:20.346501+0000) 2020-02-05T22:49:20.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:20.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:20.361697+0000) 2020-02-05T22:49:22.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:22.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.360492+0000) 2020-02-05T22:49:22.680 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:22.678+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.679260+0000) 2020-02-05T22:49:22.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:22.679+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.680574+0000) 2020-02-05T22:49:22.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:22.680+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.681163+0000) 2020-02-05T22:49:22.683 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:22.681+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.681734+0000) 2020-02-05T22:49:22.683 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:22.681+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.681994+0000) 2020-02-05T22:49:22.684 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:22.681+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.682816+0000) 2020-02-05T22:49:22.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:22.682+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.683126+0000) 2020-02-05T22:49:22.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:22.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.683980+0000) 2020-02-05T22:49:22.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:22.683+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.684703+0000) 2020-02-05T22:49:22.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:22.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.684862+0000) 2020-02-05T22:49:22.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:22.684+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.685083+0000) 2020-02-05T22:49:22.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:22.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.685379+0000) 2020-02-05T22:49:22.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:22.685+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.685610+0000) 2020-02-05T22:49:22.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:22.689+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.690113+0000) 2020-02-05T22:49:22.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:22.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:22.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:22.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.715819+0000) 2020-02-05T22:49:22.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:22.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:22.752306+0000) 2020-02-05T22:49:23.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:23.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:23.875190+0000) 2020-02-05T22:49:23.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:23.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:23.875275+0000) 2020-02-05T22:49:25.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:25.974+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:25.974668+0000) 2020-02-05T22:49:27.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:27.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.114999+0000) 2020-02-05T22:49:27.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:27.114+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.115164+0000) 2020-02-05T22:49:27.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:27.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.115191+0000) 2020-02-05T22:49:27.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:27.699+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.700650+0000) 2020-02-05T22:49:27.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:27.700+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.701286+0000) 2020-02-05T22:49:27.703 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:27.701+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.702173+0000) 2020-02-05T22:49:27.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:27.702+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.702527+0000) 2020-02-05T22:49:27.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:27.702+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.703745+0000) 2020-02-05T22:49:27.705 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:27.703+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.703748+0000) 2020-02-05T22:49:27.705 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:27.703+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.704081+0000) 2020-02-05T22:49:27.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:27.704+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.705283+0000) 2020-02-05T22:49:27.706 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:27.704+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.705867+0000) 2020-02-05T22:49:27.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:27.705+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.706295+0000) 2020-02-05T22:49:27.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:27.706+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.706475+0000) 2020-02-05T22:49:27.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:27.705+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.706288+0000) 2020-02-05T22:49:27.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:27.706+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.707394+0000) 2020-02-05T22:49:27.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:27.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:27.710698+0000) 2020-02-05T22:49:27.717 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:27.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:28.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:28.080+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:28.081124+0000) 2020-02-05T22:49:29.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:29.502+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:29.503564+0000) 2020-02-05T22:49:30.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:30.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:30.110187+0000) 2020-02-05T22:49:30.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:30.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:30.346751+0000) 2020-02-05T22:49:30.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:30.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:30.361929+0000) 2020-02-05T22:49:32.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:32.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.360733+0000) 2020-02-05T22:49:32.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:32.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.716026+0000) 2020-02-05T22:49:32.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:32.721+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.722356+0000) 2020-02-05T22:49:32.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:32.721+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.722797+0000) 2020-02-05T22:49:32.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:32.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.723489+0000) 2020-02-05T22:49:32.724 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:32.722+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.723793+0000) 2020-02-05T22:49:32.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:32.723+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.724199+0000) 2020-02-05T22:49:32.725 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:32.723+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.724404+0000) 2020-02-05T22:49:32.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:32.723+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.724701+0000) 2020-02-05T22:49:32.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:32.724+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.725296+0000) 2020-02-05T22:49:32.726 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:32.725+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.725704+0000) 2020-02-05T22:49:32.727 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:32.725+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.726328+0000) 2020-02-05T22:49:32.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:32.726+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.726581+0000) 2020-02-05T22:49:32.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:32.726+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.727183+0000) 2020-02-05T22:49:32.729 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:32.727+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.727880+0000) 2020-02-05T22:49:32.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:32.730+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.731778+0000) 2020-02-05T22:49:32.737 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:32.736+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126539 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:32.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:32.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:32.752500+0000) 2020-02-05T22:49:33.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:33.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:33.875425+0000) 2020-02-05T22:49:33.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:33.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:33.875520+0000) 2020-02-05T22:49:35.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:35.974+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:35.974968+0000) 2020-02-05T22:49:37.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:37.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.115252+0000) 2020-02-05T22:49:37.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:37.115+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.115484+0000) 2020-02-05T22:49:37.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:37.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.115492+0000) 2020-02-05T22:49:37.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:37.740+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.741593+0000) 2020-02-05T22:49:37.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:37.742+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.743179+0000) 2020-02-05T22:49:37.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:37.744+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.744685+0000) 2020-02-05T22:49:37.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:37.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.745322+0000) 2020-02-05T22:49:37.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:37.744+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.745914+0000) 2020-02-05T22:49:37.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:37.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.746514+0000) 2020-02-05T22:49:37.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:37.746+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.746986+0000) 2020-02-05T22:49:37.748 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:37.747+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.747915+0000) 2020-02-05T22:49:37.749 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:37.747+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.748094+0000) 2020-02-05T22:49:37.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:37.746+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.747074+0000) 2020-02-05T22:49:37.750 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:37.748+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.749242+0000) 2020-02-05T22:49:37.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:37.748+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.749309+0000) 2020-02-05T22:49:37.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:37.748+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.749550+0000) 2020-02-05T22:49:37.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:37.753+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:37.754155+0000) 2020-02-05T22:49:37.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:37.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126579 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:38.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:38.088+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:38.089147+0000) 2020-02-05T22:49:38.134 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:49:38.141 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:49:39.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:39.503+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:39.503797+0000) 2020-02-05T22:49:40.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:40.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:40.110406+0000) 2020-02-05T22:49:40.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:40.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:40.346995+0000) 2020-02-05T22:49:40.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:40.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:40.362166+0000) 2020-02-05T22:49:42.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:42.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.361020+0000) 2020-02-05T22:49:42.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:42.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.716207+0000) 2020-02-05T22:49:42.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:42.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.752717+0000) 2020-02-05T22:49:42.765 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:42.763+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.764141+0000) 2020-02-05T22:49:42.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:42.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.764937+0000) 2020-02-05T22:49:42.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:42.764+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.765395+0000) 2020-02-05T22:49:42.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:42.765+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.765718+0000) 2020-02-05T22:49:42.767 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:42.765+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.766244+0000) 2020-02-05T22:49:42.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:42.765+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.766295+0000) 2020-02-05T22:49:42.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:42.765+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.766858+0000) 2020-02-05T22:49:42.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:42.766+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.766871+0000) 2020-02-05T22:49:42.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:42.766+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.767309+0000) 2020-02-05T22:49:42.769 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:42.767+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.768205+0000) 2020-02-05T22:49:42.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:42.768+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.768925+0000) 2020-02-05T22:49:42.770 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:42.768+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.769064+0000) 2020-02-05T22:49:42.770 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:42.767+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.768380+0000) 2020-02-05T22:49:42.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:42.772+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:42.773930+0000) 2020-02-05T22:49:42.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:42.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126620 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:43.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:43.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:43.875693+0000) 2020-02-05T22:49:43.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:43.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:43.875745+0000) 2020-02-05T22:49:45.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:45.975+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:45.975222+0000) 2020-02-05T22:49:47.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:47.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.115513+0000) 2020-02-05T22:49:47.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:47.115+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.115772+0000) 2020-02-05T22:49:47.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:47.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.115725+0000) 2020-02-05T22:49:47.785 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:47.783+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.784281+0000) 2020-02-05T22:49:47.786 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:47.784+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.784812+0000) 2020-02-05T22:49:47.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:47.784+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.785925+0000) 2020-02-05T22:49:47.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:47.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.786486+0000) 2020-02-05T22:49:47.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:47.785+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.786563+0000) 2020-02-05T22:49:47.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:47.786+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.787038+0000) 2020-02-05T22:49:47.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:47.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.787271+0000) 2020-02-05T22:49:47.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:47.786+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.787687+0000) 2020-02-05T22:49:47.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:47.787+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.788257+0000) 2020-02-05T22:49:47.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:47.788+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.789013+0000) 2020-02-05T22:49:47.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:47.788+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.789246+0000) 2020-02-05T22:49:47.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:47.789+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.789581+0000) 2020-02-05T22:49:47.791 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:47.789+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.790090+0000) 2020-02-05T22:49:47.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:47.792+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:47.793107+0000) 2020-02-05T22:49:47.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:47.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126660 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:48.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:48.096+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:48.097168+0000) 2020-02-05T22:49:49.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:49.502+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:49.504034+0000) 2020-02-05T22:49:50.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:50.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:50.110640+0000) 2020-02-05T22:49:50.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:50.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:50.347287+0000) 2020-02-05T22:49:50.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:50.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:50.362385+0000) 2020-02-05T22:49:52.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:52.361+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.361277+0000) 2020-02-05T22:49:52.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:52.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.716441+0000) 2020-02-05T22:49:52.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:52.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.752968+0000) 2020-02-05T22:49:52.806 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:52.803+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.804875+0000) 2020-02-05T22:49:52.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:52.804+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.805536+0000) 2020-02-05T22:49:52.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:52.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.807102+0000) 2020-02-05T22:49:52.808 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:52.807+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.807519+0000) 2020-02-05T22:49:52.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:52.807+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.808174+0000) 2020-02-05T22:49:52.809 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:52.808+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.808514+0000) 2020-02-05T22:49:52.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:52.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.808888+0000) 2020-02-05T22:49:52.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:52.808+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.809272+0000) 2020-02-05T22:49:52.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:52.808+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.809504+0000) 2020-02-05T22:49:52.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:52.809+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.809748+0000) 2020-02-05T22:49:52.812 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:52.810+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.811251+0000) 2020-02-05T22:49:52.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:52.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.811616+0000) 2020-02-05T22:49:52.812 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:52.811+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.811873+0000) 2020-02-05T22:49:52.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:52.814+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:52.815365+0000) 2020-02-05T22:49:52.821 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:52.820+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126701 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:53.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:53.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:53.875928+0000) 2020-02-05T22:49:53.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:53.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:53.875990+0000) 2020-02-05T22:49:55.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:55.975+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:55.975514+0000) 2020-02-05T22:49:57.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:57.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.115741+0000) 2020-02-05T22:49:57.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:57.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.115960+0000) 2020-02-05T22:49:57.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:57.115+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.116041+0000) 2020-02-05T22:49:57.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:49:57.824+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.825124+0000) 2020-02-05T22:49:57.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:57.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.826040+0000) 2020-02-05T22:49:57.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:49:57.827+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.827877+0000) 2020-02-05T22:49:57.829 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:49:57.827+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.828402+0000) 2020-02-05T22:49:57.830 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:49:57.828+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.829182+0000) 2020-02-05T22:49:57.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:49:57.828+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.829697+0000) 2020-02-05T22:49:57.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:49:57.830+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.830546+0000) 2020-02-05T22:49:57.832 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:49:57.831+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.831677+0000) 2020-02-05T22:49:57.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:49:57.831+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.832098+0000) 2020-02-05T22:49:57.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:57.831+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.832070+0000) 2020-02-05T22:49:57.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:49:57.831+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.832299+0000) 2020-02-05T22:49:57.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:49:57.831+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.831912+0000) 2020-02-05T22:49:57.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:49:57.831+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.832560+0000) 2020-02-05T22:49:57.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:57.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:57.837292+0000) 2020-02-05T22:49:57.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:49:57.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126741 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:49:58.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:58.105+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:58.105592+0000) 2020-02-05T22:49:59.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:49:59.503+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:49:59.504260+0000) 2020-02-05T22:50:00.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:00.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:00.110848+0000) 2020-02-05T22:50:00.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:00.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:00.347562+0000) 2020-02-05T22:50:00.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:00.361+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:00.362628+0000) 2020-02-05T22:50:02.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:02.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.361536+0000) 2020-02-05T22:50:02.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:02.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.716640+0000) 2020-02-05T22:50:02.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:02.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.753241+0000) 2020-02-05T22:50:02.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:02.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.849352+0000) 2020-02-05T22:50:02.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:02.848+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.849854+0000) 2020-02-05T22:50:02.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:02.849+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.850463+0000) 2020-02-05T22:50:02.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:02.849+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.850714+0000) 2020-02-05T22:50:02.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:02.849+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.850997+0000) 2020-02-05T22:50:02.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:02.850+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.851543+0000) 2020-02-05T22:50:02.853 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:02.850+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.852071+0000) 2020-02-05T22:50:02.853 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:02.850+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.852279+0000) 2020-02-05T22:50:02.854 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:02.850+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.852181+0000) 2020-02-05T22:50:02.854 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:02.851+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.852990+0000) 2020-02-05T22:50:02.855 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:02.852+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.853418+0000) 2020-02-05T22:50:02.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:02.852+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.853669+0000) 2020-02-05T22:50:02.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:02.852+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.854061+0000) 2020-02-05T22:50:02.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:02.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:02.858948+0000) 2020-02-05T22:50:02.864 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:02.862+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126782 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:03.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:03.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:03.876159+0000) 2020-02-05T22:50:03.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:03.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:03.876256+0000) 2020-02-05T22:50:05.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:05.974+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:05.975746+0000) 2020-02-05T22:50:07.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:07.114+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.115985+0000) 2020-02-05T22:50:07.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:07.114+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.116148+0000) 2020-02-05T22:50:07.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:07.114+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.116293+0000) 2020-02-05T22:50:07.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:07.869+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.870811+0000) 2020-02-05T22:50:07.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:07.870+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.871272+0000) 2020-02-05T22:50:07.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:07.870+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.871935+0000) 2020-02-05T22:50:07.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:07.871+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.872497+0000) 2020-02-05T22:50:07.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:07.871+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.872914+0000) 2020-02-05T22:50:07.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:07.871+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.873144+0000) 2020-02-05T22:50:07.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:07.872+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.873710+0000) 2020-02-05T22:50:07.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:07.873+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.874210+0000) 2020-02-05T22:50:07.875 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:07.873+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.874482+0000) 2020-02-05T22:50:07.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:07.873+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.874657+0000) 2020-02-05T22:50:07.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:07.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.875759+0000) 2020-02-05T22:50:07.877 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:07.874+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.875768+0000) 2020-02-05T22:50:07.877 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:07.874+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.876094+0000) 2020-02-05T22:50:07.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:07.879+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:07.880797+0000) 2020-02-05T22:50:07.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:07.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:08.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:08.113+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:08.113986+0000) 2020-02-05T22:50:08.265 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:50:08.271 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:50:09.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:09.504+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:09.504512+0000) 2020-02-05T22:50:10.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:10.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:10.111123+0000) 2020-02-05T22:50:10.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:10.346+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:10.347783+0000) 2020-02-05T22:50:10.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:10.362+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:10.362852+0000) 2020-02-05T22:50:12.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:12.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.361795+0000) 2020-02-05T22:50:12.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:12.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.716879+0000) 2020-02-05T22:50:12.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:12.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.753513+0000) 2020-02-05T22:50:12.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:12.889+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.891224+0000) 2020-02-05T22:50:12.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:12.891+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.891755+0000) 2020-02-05T22:50:12.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:12.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.892436+0000) 2020-02-05T22:50:12.894 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:12.891+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.893131+0000) 2020-02-05T22:50:12.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:12.892+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.893679+0000) 2020-02-05T22:50:12.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:12.893+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894013+0000) 2020-02-05T22:50:12.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:12.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894233+0000) 2020-02-05T22:50:12.896 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:12.892+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894079+0000) 2020-02-05T22:50:12.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:12.892+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894276+0000) 2020-02-05T22:50:12.896 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:12.893+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894975+0000) 2020-02-05T22:50:12.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:12.894+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894680+0000) 2020-02-05T22:50:12.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:12.894+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.894814+0000) 2020-02-05T22:50:12.897 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:12.895+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.895569+0000) 2020-02-05T22:50:12.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:12.901+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:12.901589+0000) 2020-02-05T22:50:12.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:12.904+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:13.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:13.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:13.876394+0000) 2020-02-05T22:50:13.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:13.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:13.876527+0000) 2020-02-05T22:50:15.977 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:15.975+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:15.976009+0000) 2020-02-05T22:50:17.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:17.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.116250+0000) 2020-02-05T22:50:17.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:17.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.116378+0000) 2020-02-05T22:50:17.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:17.115+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.116587+0000) 2020-02-05T22:50:17.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:17.910+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.911364+0000) 2020-02-05T22:50:17.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:17.910+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.911697+0000) 2020-02-05T22:50:17.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:17.910+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.912246+0000) 2020-02-05T22:50:17.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:17.911+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.912644+0000) 2020-02-05T22:50:17.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:17.911+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.913071+0000) 2020-02-05T22:50:17.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:17.912+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.913703+0000) 2020-02-05T22:50:17.915 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:17.913+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.914416+0000) 2020-02-05T22:50:17.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:17.913+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.914519+0000) 2020-02-05T22:50:17.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:17.913+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.914683+0000) 2020-02-05T22:50:17.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:17.913+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.914926+0000) 2020-02-05T22:50:17.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:17.914+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.915316+0000) 2020-02-05T22:50:17.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:17.914+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.915471+0000) 2020-02-05T22:50:17.917 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:17.915+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.917079+0000) 2020-02-05T22:50:17.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:17.921+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:17.922625+0000) 2020-02-05T22:50:17.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:17.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:18.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:18.121+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:18.122275+0000) 2020-02-05T22:50:19.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:19.503+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:19.504771+0000) 2020-02-05T22:50:20.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:20.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:20.111354+0000) 2020-02-05T22:50:20.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:20.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:20.348037+0000) 2020-02-05T22:50:20.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:20.362+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:20.363127+0000) 2020-02-05T22:50:22.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:22.361+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.362056+0000) 2020-02-05T22:50:22.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:22.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.717127+0000) 2020-02-05T22:50:22.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:22.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.753772+0000) 2020-02-05T22:50:22.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:22.929+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.931339+0000) 2020-02-05T22:50:22.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:22.931+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.932252+0000) 2020-02-05T22:50:22.933 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:22.931+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.932817+0000) 2020-02-05T22:50:22.935 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:22.932+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.934348+0000) 2020-02-05T22:50:22.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:22.933+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.934881+0000) 2020-02-05T22:50:22.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:22.934+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.935408+0000) 2020-02-05T22:50:22.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:22.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.935467+0000) 2020-02-05T22:50:22.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:22.934+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.935668+0000) 2020-02-05T22:50:22.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:22.934+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.935909+0000) 2020-02-05T22:50:22.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:22.934+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.936222+0000) 2020-02-05T22:50:22.938 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:22.935+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.936847+0000) 2020-02-05T22:50:22.938 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:22.936+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.937896+0000) 2020-02-05T22:50:22.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:22.940+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.942027+0000) 2020-02-05T22:50:22.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:22.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:22.943156+0000) 2020-02-05T22:50:22.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:22.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:23.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:23.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:23.876601+0000) 2020-02-05T22:50:23.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:23.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:23.876748+0000) 2020-02-05T22:50:25.977 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:25.975+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:25.976267+0000) 2020-02-05T22:50:27.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:27.115+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.116560+0000) 2020-02-05T22:50:27.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:27.115+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.116672+0000) 2020-02-05T22:50:27.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:27.115+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.116813+0000) 2020-02-05T22:50:27.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:27.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.953586+0000) 2020-02-05T22:50:27.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:27.953+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.954400+0000) 2020-02-05T22:50:27.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:27.953+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.954998+0000) 2020-02-05T22:50:27.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:27.954+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.955763+0000) 2020-02-05T22:50:27.957 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:27.955+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.956287+0000) 2020-02-05T22:50:27.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:27.955+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.956557+0000) 2020-02-05T22:50:27.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:27.955+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.957058+0000) 2020-02-05T22:50:27.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:27.955+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.957225+0000) 2020-02-05T22:50:27.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:27.956+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.957677+0000) 2020-02-05T22:50:27.959 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:27.956+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.958015+0000) 2020-02-05T22:50:27.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:27.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.958513+0000) 2020-02-05T22:50:27.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:27.957+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.958955+0000) 2020-02-05T22:50:27.961 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:27.957+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.958989+0000) 2020-02-05T22:50:27.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:27.963+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:27.964963+0000) 2020-02-05T22:50:27.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:27.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 126986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:28.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:28.129+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:28.130550+0000) 2020-02-05T22:50:29.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:29.504+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:29.504999+0000) 2020-02-05T22:50:30.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:30.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:30.111612+0000) 2020-02-05T22:50:30.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:30.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:30.348272+0000) 2020-02-05T22:50:30.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:30.362+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:30.363386+0000) 2020-02-05T22:50:32.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:32.360+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.362302+0000) 2020-02-05T22:50:32.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:32.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.717358+0000) 2020-02-05T22:50:32.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:32.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.754005+0000) 2020-02-05T22:50:32.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:32.974+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.975197+0000) 2020-02-05T22:50:32.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:32.975+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.976169+0000) 2020-02-05T22:50:32.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:32.975+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.976597+0000) 2020-02-05T22:50:32.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:32.976+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.977187+0000) 2020-02-05T22:50:32.979 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:32.977+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.978395+0000) 2020-02-05T22:50:32.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:32.976+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.977927+0000) 2020-02-05T22:50:32.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:32.976+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.977711+0000) 2020-02-05T22:50:32.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:32.978+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.979308+0000) 2020-02-05T22:50:32.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:32.977+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.979046+0000) 2020-02-05T22:50:32.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:32.979+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.980325+0000) 2020-02-05T22:50:32.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:32.979+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.980984+0000) 2020-02-05T22:50:32.982 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:32.980+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.981630+0000) 2020-02-05T22:50:32.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:32.980+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.981813+0000) 2020-02-05T22:50:32.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:32.986+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:32.987586+0000) 2020-02-05T22:50:32.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:32.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:33.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:33.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:33.876828+0000) 2020-02-05T22:50:33.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:33.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:33.876981+0000) 2020-02-05T22:50:35.977 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:35.975+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:35.976500+0000) 2020-02-05T22:50:37.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:37.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.116805+0000) 2020-02-05T22:50:37.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:37.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.116896+0000) 2020-02-05T22:50:37.118 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:37.116+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.117070+0000) 2020-02-05T22:50:37.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:37.996+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.997703+0000) 2020-02-05T22:50:37.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:37.997+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.998217+0000) 2020-02-05T22:50:38.000 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:37.998+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.999232+0000) 2020-02-05T22:50:38.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:37.998+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.999503+0000) 2020-02-05T22:50:38.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:37.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:37.999933+0000) 2020-02-05T22:50:38.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:37.999+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.000160+0000) 2020-02-05T22:50:38.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:37.998+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.000350+0000) 2020-02-05T22:50:38.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:38.000+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.001400+0000) 2020-02-05T22:50:38.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:37.999+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.001157+0000) 2020-02-05T22:50:38.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:38.001+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.002579+0000) 2020-02-05T22:50:38.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:38.001+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.002609+0000) 2020-02-05T22:50:38.004 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:38.001+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.003032+0000) 2020-02-05T22:50:38.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:38.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.007547+0000) 2020-02-05T22:50:38.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:38.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.007867+0000) 2020-02-05T22:50:38.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:38.011+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:38.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:38.136+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:38.137507+0000) 2020-02-05T22:50:38.416 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:50:38.423 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:50:39.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:39.504+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:39.505281+0000) 2020-02-05T22:50:40.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:40.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:40.111872+0000) 2020-02-05T22:50:40.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:40.347+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:40.348506+0000) 2020-02-05T22:50:40.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:40.363+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:40.363637+0000) 2020-02-05T22:50:42.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:42.361+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:42.362563+0000) 2020-02-05T22:50:42.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:42.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:42.717619+0000) 2020-02-05T22:50:42.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:42.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:42.754263+0000) 2020-02-05T22:50:43.019 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:43.017+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.018734+0000) 2020-02-05T22:50:43.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:43.018+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.019308+0000) 2020-02-05T22:50:43.020 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:43.019+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.020140+0000) 2020-02-05T22:50:43.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:43.019+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.021028+0000) 2020-02-05T22:50:43.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:43.020+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.021735+0000) 2020-02-05T22:50:43.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:43.020+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.022042+0000) 2020-02-05T22:50:43.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:43.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.022525+0000) 2020-02-05T22:50:43.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:43.021+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.022833+0000) 2020-02-05T22:50:43.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:43.022+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.023865+0000) 2020-02-05T22:50:43.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:43.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.024150+0000) 2020-02-05T22:50:43.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:43.023+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.024845+0000) 2020-02-05T22:50:43.026 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:43.024+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.025637+0000) 2020-02-05T22:50:43.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:43.026+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.027627+0000) 2020-02-05T22:50:43.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:43.028+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.029383+0000) 2020-02-05T22:50:43.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:43.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:43.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:43.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.877022+0000) 2020-02-05T22:50:43.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:43.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:43.877226+0000) 2020-02-05T22:50:45.977 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:45.976+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:45.976751+0000) 2020-02-05T22:50:47.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:47.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:47.117051+0000) 2020-02-05T22:50:47.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:47.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:47.117137+0000) 2020-02-05T22:50:47.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:47.116+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:47.117311+0000) 2020-02-05T22:50:48.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:48.039+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.040684+0000) 2020-02-05T22:50:48.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:48.039+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.040740+0000) 2020-02-05T22:50:48.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:48.039+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.041206+0000) 2020-02-05T22:50:48.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:48.040+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.041937+0000) 2020-02-05T22:50:48.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:48.041+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.042146+0000) 2020-02-05T22:50:48.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:48.041+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.042407+0000) 2020-02-05T22:50:48.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:48.042+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.043272+0000) 2020-02-05T22:50:48.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:48.043+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.044499+0000) 2020-02-05T22:50:48.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:48.043+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.044246+0000) 2020-02-05T22:50:48.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:48.044+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.045674+0000) 2020-02-05T22:50:48.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:48.044+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.045794+0000) 2020-02-05T22:50:48.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:48.044+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.045999+0000) 2020-02-05T22:50:48.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:48.044+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.046136+0000) 2020-02-05T22:50:48.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:48.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.051359+0000) 2020-02-05T22:50:48.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:48.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:48.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:48.145+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:48.145903+0000) 2020-02-05T22:50:49.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:49.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:49.505518+0000) 2020-02-05T22:50:50.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:50.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:50.112135+0000) 2020-02-05T22:50:50.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:50.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:50.348703+0000) 2020-02-05T22:50:50.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:50.363+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:50.363854+0000) 2020-02-05T22:50:52.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:52.362+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:52.362802+0000) 2020-02-05T22:50:52.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:52.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:52.717845+0000) 2020-02-05T22:50:52.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:52.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:52.754520+0000) 2020-02-05T22:50:53.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:53.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.060524+0000) 2020-02-05T22:50:53.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:53.061+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.061991+0000) 2020-02-05T22:50:53.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:53.062+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.062624+0000) 2020-02-05T22:50:53.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:53.062+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.063311+0000) 2020-02-05T22:50:53.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:53.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.064053+0000) 2020-02-05T22:50:53.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:53.064+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.064713+0000) 2020-02-05T22:50:53.066 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:53.064+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.065124+0000) 2020-02-05T22:50:53.067 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:53.064+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.065401+0000) 2020-02-05T22:50:53.067 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:53.064+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.065672+0000) 2020-02-05T22:50:53.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:53.066+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.066724+0000) 2020-02-05T22:50:53.068 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:53.065+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.066133+0000) 2020-02-05T22:50:53.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:53.066+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.067232+0000) 2020-02-05T22:50:53.069 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:53.067+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.067665+0000) 2020-02-05T22:50:53.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:53.071+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.072621+0000) 2020-02-05T22:50:53.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:53.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127191 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:53.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:53.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.877259+0000) 2020-02-05T22:50:53.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:53.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:53.877436+0000) 2020-02-05T22:50:55.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:55.976+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:55.976993+0000) 2020-02-05T22:50:57.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:57.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:57.117307+0000) 2020-02-05T22:50:57.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:57.116+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:57.117368+0000) 2020-02-05T22:50:57.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:57.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:57.117599+0000) 2020-02-05T22:50:58.082 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:50:58.080+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.081410+0000) 2020-02-05T22:50:58.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:50:58.081+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.082270+0000) 2020-02-05T22:50:58.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:50:58.082+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.083075+0000) 2020-02-05T22:50:58.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:50:58.083+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.083732+0000) 2020-02-05T22:50:58.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:50:58.083+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.084337+0000) 2020-02-05T22:50:58.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:58.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.084876+0000) 2020-02-05T22:50:58.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:50:58.084+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.085167+0000) 2020-02-05T22:50:58.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:50:58.085+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.086142+0000) 2020-02-05T22:50:58.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:58.085+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.086479+0000) 2020-02-05T22:50:58.087 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:50:58.085+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.086964+0000) 2020-02-05T22:50:58.088 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:50:58.086+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.087085+0000) 2020-02-05T22:50:58.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:50:58.087+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.087502+0000) 2020-02-05T22:50:58.089 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:50:58.087+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.088179+0000) 2020-02-05T22:50:58.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:58.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.093272+0000) 2020-02-05T22:50:58.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:50:58.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:50:58.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:58.157+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:58.157622+0000) 2020-02-05T22:50:59.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:50:59.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:50:59.505754+0000) 2020-02-05T22:51:00.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:00.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:00.112346+0000) 2020-02-05T22:51:00.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:00.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:00.348955+0000) 2020-02-05T22:51:00.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:00.363+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:00.364108+0000) 2020-02-05T22:51:02.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:02.362+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:02.363058+0000) 2020-02-05T22:51:02.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:02.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:02.718062+0000) 2020-02-05T22:51:02.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:02.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:02.754789+0000) 2020-02-05T22:51:03.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:03.102+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.103909+0000) 2020-02-05T22:51:03.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:03.104+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.104477+0000) 2020-02-05T22:51:03.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:03.103+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.104750+0000) 2020-02-05T22:51:03.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:03.104+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.105099+0000) 2020-02-05T22:51:03.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:03.105+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.105890+0000) 2020-02-05T22:51:03.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:03.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.105417+0000) 2020-02-05T22:51:03.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:03.106+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.106974+0000) 2020-02-05T22:51:03.108 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:03.105+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.106449+0000) 2020-02-05T22:51:03.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:03.107+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.107841+0000) 2020-02-05T22:51:03.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:03.106+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.107989+0000) 2020-02-05T22:51:03.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:03.108+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.108885+0000) 2020-02-05T22:51:03.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:03.107+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.108792+0000) 2020-02-05T22:51:03.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:03.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.113830+0000) 2020-02-05T22:51:03.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:03.113+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.114158+0000) 2020-02-05T22:51:03.119 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:03.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:03.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:03.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.877516+0000) 2020-02-05T22:51:03.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:03.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:03.877661+0000) 2020-02-05T22:51:05.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:05.976+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:05.977252+0000) 2020-02-05T22:51:07.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:07.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:07.117583+0000) 2020-02-05T22:51:07.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:07.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:07.117647+0000) 2020-02-05T22:51:07.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:07.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:07.117860+0000) 2020-02-05T22:51:08.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:08.122+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.122914+0000) 2020-02-05T22:51:08.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:08.123+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.123542+0000) 2020-02-05T22:51:08.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:08.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.124427+0000) 2020-02-05T22:51:08.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:08.125+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.125380+0000) 2020-02-05T22:51:08.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:08.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.125564+0000) 2020-02-05T22:51:08.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:08.125+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.126031+0000) 2020-02-05T22:51:08.127 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:08.126+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.126760+0000) 2020-02-05T22:51:08.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:08.127+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.127216+0000) 2020-02-05T22:51:08.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:08.127+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.127390+0000) 2020-02-05T22:51:08.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:08.125+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.126175+0000) 2020-02-05T22:51:08.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:08.127+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.127803+0000) 2020-02-05T22:51:08.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:08.127+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.128051+0000) 2020-02-05T22:51:08.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:08.128+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.129027+0000) 2020-02-05T22:51:08.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:08.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.133729+0000) 2020-02-05T22:51:08.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:08.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127315 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:08.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:08.165+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:08.165682+0000) 2020-02-05T22:51:08.572 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:51:08.578 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:51:09.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:09.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:09.505978+0000) 2020-02-05T22:51:10.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:10.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:10.112574+0000) 2020-02-05T22:51:10.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:10.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:10.349211+0000) 2020-02-05T22:51:10.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:10.364+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:10.364309+0000) 2020-02-05T22:51:12.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:12.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:12.363326+0000) 2020-02-05T22:51:12.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:12.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:12.718341+0000) 2020-02-05T22:51:12.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:12.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:12.755047+0000) 2020-02-05T22:51:13.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:13.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.144112+0000) 2020-02-05T22:51:13.145 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:13.143+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.144398+0000) 2020-02-05T22:51:13.145 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:13.143+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.144844+0000) 2020-02-05T22:51:13.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:13.144+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.145398+0000) 2020-02-05T22:51:13.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:13.144+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.145682+0000) 2020-02-05T22:51:13.147 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:13.145+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.146250+0000) 2020-02-05T22:51:13.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:13.146+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.146586+0000) 2020-02-05T22:51:13.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:13.145+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.146733+0000) 2020-02-05T22:51:13.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:13.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.147288+0000) 2020-02-05T22:51:13.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:13.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.147356+0000) 2020-02-05T22:51:13.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:13.147+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.147621+0000) 2020-02-05T22:51:13.149 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:13.148+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.148891+0000) 2020-02-05T22:51:13.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:13.148+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.149275+0000) 2020-02-05T22:51:13.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:13.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.153816+0000) 2020-02-05T22:51:13.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:13.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:13.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:13.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.877729+0000) 2020-02-05T22:51:13.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:13.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:13.877877+0000) 2020-02-05T22:51:15.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:15.977+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:15.977505+0000) 2020-02-05T22:51:17.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:17.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:17.117815+0000) 2020-02-05T22:51:17.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:17.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:17.118000+0000) 2020-02-05T22:51:17.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:17.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:17.118309+0000) 2020-02-05T22:51:18.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:18.164+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.165177+0000) 2020-02-05T22:51:18.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:18.166+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.166691+0000) 2020-02-05T22:51:18.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:18.166+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.166858+0000) 2020-02-05T22:51:18.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:18.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.167154+0000) 2020-02-05T22:51:18.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:18.166+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.167474+0000) 2020-02-05T22:51:18.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:18.166+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.167667+0000) 2020-02-05T22:51:18.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:18.167+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.168113+0000) 2020-02-05T22:51:18.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:18.168+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.168644+0000) 2020-02-05T22:51:18.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:18.167+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.168452+0000) 2020-02-05T22:51:18.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:18.169+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.169893+0000) 2020-02-05T22:51:18.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:18.169+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.170678+0000) 2020-02-05T22:51:18.172 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:18.170+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.170885+0000) 2020-02-05T22:51:18.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:18.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.172716+0000) 2020-02-05T22:51:18.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:18.171+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.172748+0000) 2020-02-05T22:51:18.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:18.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:18.172992+0000) 2020-02-05T22:51:18.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:18.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:19.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:19.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:19.506223+0000) 2020-02-05T22:51:20.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:20.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:20.112820+0000) 2020-02-05T22:51:20.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:20.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:20.349486+0000) 2020-02-05T22:51:20.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:20.363+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:20.364527+0000) 2020-02-05T22:51:22.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:22.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:22.363589+0000) 2020-02-05T22:51:22.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:22.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:22.718563+0000) 2020-02-05T22:51:22.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:22.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:22.755325+0000) 2020-02-05T22:51:23.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:23.185+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.185801+0000) 2020-02-05T22:51:23.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:23.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.186876+0000) 2020-02-05T22:51:23.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:23.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.187303+0000) 2020-02-05T22:51:23.188 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:23.186+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.187932+0000) 2020-02-05T22:51:23.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:23.188+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.188605+0000) 2020-02-05T22:51:23.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:23.187+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.188750+0000) 2020-02-05T22:51:23.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:23.188+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.189169+0000) 2020-02-05T22:51:23.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:23.189+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.189410+0000) 2020-02-05T22:51:23.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:23.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.189686+0000) 2020-02-05T22:51:23.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:23.189+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.190144+0000) 2020-02-05T22:51:23.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:23.190+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.191047+0000) 2020-02-05T22:51:23.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:23.190+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.191205+0000) 2020-02-05T22:51:23.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:23.190+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.191612+0000) 2020-02-05T22:51:23.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:23.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.195420+0000) 2020-02-05T22:51:23.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:23.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:23.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:23.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.877937+0000) 2020-02-05T22:51:23.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:23.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:23.878114+0000) 2020-02-05T22:51:25.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:25.977+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:25.977742+0000) 2020-02-05T22:51:27.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:27.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:27.118061+0000) 2020-02-05T22:51:27.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:27.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:27.118241+0000) 2020-02-05T22:51:27.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:27.118+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:27.118597+0000) 2020-02-05T22:51:28.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:28.172+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.172924+0000) 2020-02-05T22:51:28.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:28.205+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.206224+0000) 2020-02-05T22:51:28.207 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:28.205+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.206965+0000) 2020-02-05T22:51:28.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:28.207+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.208403+0000) 2020-02-05T22:51:28.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:28.207+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.209057+0000) 2020-02-05T22:51:28.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:28.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.210507+0000) 2020-02-05T22:51:28.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:28.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.210947+0000) 2020-02-05T22:51:28.216 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:28.211+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.212329+0000) 2020-02-05T22:51:28.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:28.205+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.206109+0000) 2020-02-05T22:51:28.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:28.209+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.209594+0000) 2020-02-05T22:51:28.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:28.209+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.209944+0000) 2020-02-05T22:51:28.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:28.210+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.210512+0000) 2020-02-05T22:51:28.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:28.212+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.213003+0000) 2020-02-05T22:51:28.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:28.213+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.213842+0000) 2020-02-05T22:51:28.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:28.215+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:28.216631+0000) 2020-02-05T22:51:28.223 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:28.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:29.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:29.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:29.506454+0000) 2020-02-05T22:51:30.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:30.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:30.113020+0000) 2020-02-05T22:51:30.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:30.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:30.349675+0000) 2020-02-05T22:51:30.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:30.363+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:30.364723+0000) 2020-02-05T22:51:32.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:32.362+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:32.363866+0000) 2020-02-05T22:51:32.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:32.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:32.718806+0000) 2020-02-05T22:51:32.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:32.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:32.755561+0000) 2020-02-05T22:51:33.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:33.224+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.225564+0000) 2020-02-05T22:51:33.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:33.226+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.227602+0000) 2020-02-05T22:51:33.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:33.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.227769+0000) 2020-02-05T22:51:33.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:33.227+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.228581+0000) 2020-02-05T22:51:33.229 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:33.227+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.228950+0000) 2020-02-05T22:51:33.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:33.228+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.229314+0000) 2020-02-05T22:51:33.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:33.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.230686+0000) 2020-02-05T22:51:33.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:33.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.231170+0000) 2020-02-05T22:51:33.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:33.230+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.231853+0000) 2020-02-05T22:51:33.232 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:33.230+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.231663+0000) 2020-02-05T22:51:33.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:33.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.232781+0000) 2020-02-05T22:51:33.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:33.231+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.233252+0000) 2020-02-05T22:51:33.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:33.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.237147+0000) 2020-02-05T22:51:33.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:33.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.237472+0000) 2020-02-05T22:51:33.243 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:33.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:33.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:33.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.878144+0000) 2020-02-05T22:51:33.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:33.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:33.878299+0000) 2020-02-05T22:51:35.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:35.976+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:35.977981+0000) 2020-02-05T22:51:37.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:37.116+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:37.118301+0000) 2020-02-05T22:51:37.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:37.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:37.118499+0000) 2020-02-05T22:51:37.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:37.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:37.118839+0000) 2020-02-05T22:51:38.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:38.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.184637+0000) 2020-02-05T22:51:38.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:38.246+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.247947+0000) 2020-02-05T22:51:38.250 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:38.246+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.248056+0000) 2020-02-05T22:51:38.250 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:38.247+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.248515+0000) 2020-02-05T22:51:38.251 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:38.248+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.249489+0000) 2020-02-05T22:51:38.251 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:38.248+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.249868+0000) 2020-02-05T22:51:38.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:38.249+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.250622+0000) 2020-02-05T22:51:38.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:38.250+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.251675+0000) 2020-02-05T22:51:38.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:38.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.252188+0000) 2020-02-05T22:51:38.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:38.251+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.252539+0000) 2020-02-05T22:51:38.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:38.251+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.253408+0000) 2020-02-05T22:51:38.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:38.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.253597+0000) 2020-02-05T22:51:38.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:38.252+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.254384+0000) 2020-02-05T22:51:38.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:38.253+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.254621+0000) 2020-02-05T22:51:38.265 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:38.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:38.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:38.261+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:38.262162+0000) 2020-02-05T22:51:38.719 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:51:38.725 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:51:39.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:39.506+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:39.506683+0000) 2020-02-05T22:51:40.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:40.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:40.113328+0000) 2020-02-05T22:51:40.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:40.348+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:40.349900+0000) 2020-02-05T22:51:40.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:40.364+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:40.364954+0000) 2020-02-05T22:51:42.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:42.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:42.364122+0000) 2020-02-05T22:51:42.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:42.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:42.719035+0000) 2020-02-05T22:51:42.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:42.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:42.755823+0000) 2020-02-05T22:51:43.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:43.268+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.269358+0000) 2020-02-05T22:51:43.271 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:43.269+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.270396+0000) 2020-02-05T22:51:43.271 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:43.269+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.270975+0000) 2020-02-05T22:51:43.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:43.270+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.271595+0000) 2020-02-05T22:51:43.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:43.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.271919+0000) 2020-02-05T22:51:43.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:43.271+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.272800+0000) 2020-02-05T22:51:43.274 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:43.271+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.272948+0000) 2020-02-05T22:51:43.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:43.271+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.272503+0000) 2020-02-05T22:51:43.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:43.271+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.273276+0000) 2020-02-05T22:51:43.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:43.273+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.274699+0000) 2020-02-05T22:51:43.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:43.273+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.274975+0000) 2020-02-05T22:51:43.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:43.273+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.275296+0000) 2020-02-05T22:51:43.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:43.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.278655+0000) 2020-02-05T22:51:43.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:43.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.278946+0000) 2020-02-05T22:51:43.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:43.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127605 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:43.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:43.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.878381+0000) 2020-02-05T22:51:43.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:43.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:43.878537+0000) 2020-02-05T22:51:45.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:45.977+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:45.978239+0000) 2020-02-05T22:51:47.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:47.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:47.118534+0000) 2020-02-05T22:51:47.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:47.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:47.118727+0000) 2020-02-05T22:51:47.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:47.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:47.119063+0000) 2020-02-05T22:51:48.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:48.192+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.192775+0000) 2020-02-05T22:51:48.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:48.288+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.289913+0000) 2020-02-05T22:51:48.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:48.289+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.290604+0000) 2020-02-05T22:51:48.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:48.289+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.291345+0000) 2020-02-05T22:51:48.292 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:48.290+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.291578+0000) 2020-02-05T22:51:48.293 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:48.291+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.292389+0000) 2020-02-05T22:51:48.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:48.291+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.292602+0000) 2020-02-05T22:51:48.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:48.291+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.292946+0000) 2020-02-05T22:51:48.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:48.292+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.293658+0000) 2020-02-05T22:51:48.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:48.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.294220+0000) 2020-02-05T22:51:48.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:48.293+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.294875+0000) 2020-02-05T22:51:48.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:48.293+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.295024+0000) 2020-02-05T22:51:48.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:48.293+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.295414+0000) 2020-02-05T22:51:48.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:48.294+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.296066+0000) 2020-02-05T22:51:48.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:48.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:48.300615+0000) 2020-02-05T22:51:48.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:48.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127646 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:49.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:49.505+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:49.506933+0000) 2020-02-05T22:51:50.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:50.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:50.113558+0000) 2020-02-05T22:51:50.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:50.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:50.350164+0000) 2020-02-05T22:51:50.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:50.364+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:50.365186+0000) 2020-02-05T22:51:52.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:52.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:52.364370+0000) 2020-02-05T22:51:52.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:52.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:52.719335+0000) 2020-02-05T22:51:52.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:52.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:52.756057+0000) 2020-02-05T22:51:53.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:53.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.311604+0000) 2020-02-05T22:51:53.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:53.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.312295+0000) 2020-02-05T22:51:53.313 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:53.311+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.312645+0000) 2020-02-05T22:51:53.314 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:53.311+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.312952+0000) 2020-02-05T22:51:53.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:53.312+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.313460+0000) 2020-02-05T22:51:53.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:53.313+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.314569+0000) 2020-02-05T22:51:53.316 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:53.314+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.315099+0000) 2020-02-05T22:51:53.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:53.314+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.315430+0000) 2020-02-05T22:51:53.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:53.314+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.315869+0000) 2020-02-05T22:51:53.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:53.315+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.316719+0000) 2020-02-05T22:51:53.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:53.315+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.316919+0000) 2020-02-05T22:51:53.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:53.315+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.317183+0000) 2020-02-05T22:51:53.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:53.316+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.317414+0000) 2020-02-05T22:51:53.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:53.320+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.321165+0000) 2020-02-05T22:51:53.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:53.324+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127689 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:53.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:53.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.878597+0000) 2020-02-05T22:51:53.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:53.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:53.878746+0000) 2020-02-05T22:51:55.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:55.977+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:55.978494+0000) 2020-02-05T22:51:57.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:57.117+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:57.118758+0000) 2020-02-05T22:51:57.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:57.117+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:57.118965+0000) 2020-02-05T22:51:57.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:57.117+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:57.119293+0000) 2020-02-05T22:51:58.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:58.201+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.201380+0000) 2020-02-05T22:51:58.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:51:58.330+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.331729+0000) 2020-02-05T22:51:58.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:51:58.331+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.332880+0000) 2020-02-05T22:51:58.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:58.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.333005+0000) 2020-02-05T22:51:58.334 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:51:58.332+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.333382+0000) 2020-02-05T22:51:58.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:58.332+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.333952+0000) 2020-02-05T22:51:58.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:51:58.333+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.334486+0000) 2020-02-05T22:51:58.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:51:58.333+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.334776+0000) 2020-02-05T22:51:58.336 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:51:58.333+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.335138+0000) 2020-02-05T22:51:58.337 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:51:58.334+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.335580+0000) 2020-02-05T22:51:58.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:51:58.334+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.335790+0000) 2020-02-05T22:51:58.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:51:58.334+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.335657+0000) 2020-02-05T22:51:58.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:51:58.335+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.336397+0000) 2020-02-05T22:51:58.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:51:58.335+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.336928+0000) 2020-02-05T22:51:58.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:58.341+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:58.342213+0000) 2020-02-05T22:51:58.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:51:58.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127731 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:51:59.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:51:59.506+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:51:59.507158+0000) 2020-02-05T22:52:00.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:00.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:00.113806+0000) 2020-02-05T22:52:00.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:00.349+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:00.350467+0000) 2020-02-05T22:52:00.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:00.365+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:00.365484+0000) 2020-02-05T22:52:02.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:02.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:02.364661+0000) 2020-02-05T22:52:02.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:02.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:02.719565+0000) 2020-02-05T22:52:02.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:02.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:02.756320+0000) 2020-02-05T22:52:03.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:03.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.351720+0000) 2020-02-05T22:52:03.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:03.352+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.352914+0000) 2020-02-05T22:52:03.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:03.353+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.353487+0000) 2020-02-05T22:52:03.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:03.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.354046+0000) 2020-02-05T22:52:03.356 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:03.355+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.355206+0000) 2020-02-05T22:52:03.356 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:03.353+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.355359+0000) 2020-02-05T22:52:03.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:03.355+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.355764+0000) 2020-02-05T22:52:03.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:03.354+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.356043+0000) 2020-02-05T22:52:03.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:03.353+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.355121+0000) 2020-02-05T22:52:03.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:03.356+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.356579+0000) 2020-02-05T22:52:03.358 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:03.355+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.357172+0000) 2020-02-05T22:52:03.358 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:03.356+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.357691+0000) 2020-02-05T22:52:03.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:03.356+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.358372+0000) 2020-02-05T22:52:03.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:03.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.363215+0000) 2020-02-05T22:52:03.368 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:03.366+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:03.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:03.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.878804+0000) 2020-02-05T22:52:03.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:03.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:03.878947+0000) 2020-02-05T22:52:05.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:05.978+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:05.978747+0000) 2020-02-05T22:52:07.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:07.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:07.118991+0000) 2020-02-05T22:52:07.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:07.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:07.119149+0000) 2020-02-05T22:52:07.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:07.118+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:07.119552+0000) 2020-02-05T22:52:08.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:08.207+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.208072+0000) 2020-02-05T22:52:08.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:08.373+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.375349+0000) 2020-02-05T22:52:08.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:08.374+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.375508+0000) 2020-02-05T22:52:08.377 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:08.374+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.375832+0000) 2020-02-05T22:52:08.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:08.375+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.376164+0000) 2020-02-05T22:52:08.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:08.375+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.376610+0000) 2020-02-05T22:52:08.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:08.375+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.376874+0000) 2020-02-05T22:52:08.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:08.376+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.377222+0000) 2020-02-05T22:52:08.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:08.375+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.376915+0000) 2020-02-05T22:52:08.379 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:08.375+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.377411+0000) 2020-02-05T22:52:08.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:08.376+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.377848+0000) 2020-02-05T22:52:08.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:08.376+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.377847+0000) 2020-02-05T22:52:08.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:08.377+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.378960+0000) 2020-02-05T22:52:08.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:08.377+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.379240+0000) 2020-02-05T22:52:08.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:08.383+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:08.384173+0000) 2020-02-05T22:52:08.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:08.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127814 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:08.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:52:08.897 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:52:09.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:09.506+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:09.507393+0000) 2020-02-05T22:52:10.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:10.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:10.114033+0000) 2020-02-05T22:52:10.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:10.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:10.350751+0000) 2020-02-05T22:52:10.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:10.364+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:10.365720+0000) 2020-02-05T22:52:12.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:12.363+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:12.364905+0000) 2020-02-05T22:52:12.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:12.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:12.719800+0000) 2020-02-05T22:52:12.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:12.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:12.756589+0000) 2020-02-05T22:52:13.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:13.392+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.394218+0000) 2020-02-05T22:52:13.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:13.394+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.395227+0000) 2020-02-05T22:52:13.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:13.394+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.395773+0000) 2020-02-05T22:52:13.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:13.394+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.396046+0000) 2020-02-05T22:52:13.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:13.395+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.396588+0000) 2020-02-05T22:52:13.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:13.395+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.397379+0000) 2020-02-05T22:52:13.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:13.395+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.397191+0000) 2020-02-05T22:52:13.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:13.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.398237+0000) 2020-02-05T22:52:13.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:13.397+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.398290+0000) 2020-02-05T22:52:13.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:13.397+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.398856+0000) 2020-02-05T22:52:13.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:13.397+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.399009+0000) 2020-02-05T22:52:13.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:13.398+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.399493+0000) 2020-02-05T22:52:13.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:13.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.404406+0000) 2020-02-05T22:52:13.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:13.404+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.405754+0000) 2020-02-05T22:52:13.410 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:13.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:13.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:13.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.879004+0000) 2020-02-05T22:52:13.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:13.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:13.879208+0000) 2020-02-05T22:52:15.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:15.978+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:15.978998+0000) 2020-02-05T22:52:17.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:17.118+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:17.119246+0000) 2020-02-05T22:52:17.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:17.118+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:17.119355+0000) 2020-02-05T22:52:17.121 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:17.118+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:17.119809+0000) 2020-02-05T22:52:18.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:18.215+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.216371+0000) 2020-02-05T22:52:18.417 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:18.415+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.416197+0000) 2020-02-05T22:52:18.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:18.415+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.416666+0000) 2020-02-05T22:52:18.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:18.415+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.417180+0000) 2020-02-05T22:52:18.418 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:18.417+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.418207+0000) 2020-02-05T22:52:18.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:18.417+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.418521+0000) 2020-02-05T22:52:18.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:18.417+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.418903+0000) 2020-02-05T22:52:18.421 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:18.418+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.420018+0000) 2020-02-05T22:52:18.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:18.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.420508+0000) 2020-02-05T22:52:18.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:18.418+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.420392+0000) 2020-02-05T22:52:18.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:18.420+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.421582+0000) 2020-02-05T22:52:18.422 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:18.420+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.421654+0000) 2020-02-05T22:52:18.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:18.420+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.421953+0000) 2020-02-05T22:52:18.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:18.420+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.422186+0000) 2020-02-05T22:52:18.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:18.426+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:18.428020+0000) 2020-02-05T22:52:18.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:18.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127897 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:19.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:19.506+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:19.507626+0000) 2020-02-05T22:52:20.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:20.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:20.114319+0000) 2020-02-05T22:52:20.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:20.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:20.350998+0000) 2020-02-05T22:52:20.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:20.364+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:20.365963+0000) 2020-02-05T22:52:22.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:22.364+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:22.365152+0000) 2020-02-05T22:52:22.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:22.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:22.720068+0000) 2020-02-05T22:52:22.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:22.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:22.756818+0000) 2020-02-05T22:52:23.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:23.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.435140+0000) 2020-02-05T22:52:23.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:23.436+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.437587+0000) 2020-02-05T22:52:23.439 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:23.437+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.438382+0000) 2020-02-05T22:52:23.439 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:23.437+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.438305+0000) 2020-02-05T22:52:23.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:23.438+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.439204+0000) 2020-02-05T22:52:23.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:23.439+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.439785+0000) 2020-02-05T22:52:23.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:23.439+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.440282+0000) 2020-02-05T22:52:23.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:23.440+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.441231+0000) 2020-02-05T22:52:23.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:23.441+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.441611+0000) 2020-02-05T22:52:23.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:23.440+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.441974+0000) 2020-02-05T22:52:23.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:23.441+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.442083+0000) 2020-02-05T22:52:23.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:23.441+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.442179+0000) 2020-02-05T22:52:23.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:23.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.446109+0000) 2020-02-05T22:52:23.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:23.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.446379+0000) 2020-02-05T22:52:23.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:23.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:23.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:23.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.879210+0000) 2020-02-05T22:52:23.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:23.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:23.879422+0000) 2020-02-05T22:52:25.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:25.978+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:25.979233+0000) 2020-02-05T22:52:27.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:27.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:27.119559+0000) 2020-02-05T22:52:27.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:27.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:27.119611+0000) 2020-02-05T22:52:27.121 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:27.119+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:27.120072+0000) 2020-02-05T22:52:28.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:28.223+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.224557+0000) 2020-02-05T22:52:28.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:28.456+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.457976+0000) 2020-02-05T22:52:28.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:28.458+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.459025+0000) 2020-02-05T22:52:28.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:28.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.459686+0000) 2020-02-05T22:52:28.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:28.459+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.460324+0000) 2020-02-05T22:52:28.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:28.459+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.460551+0000) 2020-02-05T22:52:28.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:28.461+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.461751+0000) 2020-02-05T22:52:28.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:28.461+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.462383+0000) 2020-02-05T22:52:28.463 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:28.461+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.462035+0000) 2020-02-05T22:52:28.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:28.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.462891+0000) 2020-02-05T22:52:28.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:28.462+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.463137+0000) 2020-02-05T22:52:28.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:28.462+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.463159+0000) 2020-02-05T22:52:28.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:28.463+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.463697+0000) 2020-02-05T22:52:28.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:28.463+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.464111+0000) 2020-02-05T22:52:28.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:28.469+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:28.470690+0000) 2020-02-05T22:52:28.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:28.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 127978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:29.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:29.507+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:29.507867+0000) 2020-02-05T22:52:30.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:30.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:30.114563+0000) 2020-02-05T22:52:30.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:30.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:30.351233+0000) 2020-02-05T22:52:30.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:30.365+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:30.366185+0000) 2020-02-05T22:52:32.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:32.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:32.365403+0000) 2020-02-05T22:52:32.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:32.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:32.720351+0000) 2020-02-05T22:52:32.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:32.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:32.757051+0000) 2020-02-05T22:52:33.479 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:33.477+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.478557+0000) 2020-02-05T22:52:33.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:33.478+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.480022+0000) 2020-02-05T22:52:33.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:33.479+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.480388+0000) 2020-02-05T22:52:33.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:33.479+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.480392+0000) 2020-02-05T22:52:33.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:33.480+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.481565+0000) 2020-02-05T22:52:33.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:33.481+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.481898+0000) 2020-02-05T22:52:33.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:33.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.482037+0000) 2020-02-05T22:52:33.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:33.481+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.482202+0000) 2020-02-05T22:52:33.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:33.482+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.482460+0000) 2020-02-05T22:52:33.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:33.482+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.482620+0000) 2020-02-05T22:52:33.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:33.482+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.482852+0000) 2020-02-05T22:52:33.485 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:33.482+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.483259+0000) 2020-02-05T22:52:33.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:33.483+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.483507+0000) 2020-02-05T22:52:33.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:33.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.487201+0000) 2020-02-05T22:52:33.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:33.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:33.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:33.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.879371+0000) 2020-02-05T22:52:33.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:33.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:33.879606+0000) 2020-02-05T22:52:35.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:35.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:35.979523+0000) 2020-02-05T22:52:37.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:37.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:37.119808+0000) 2020-02-05T22:52:37.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:37.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:37.119842+0000) 2020-02-05T22:52:37.121 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:37.119+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:37.120321+0000) 2020-02-05T22:52:38.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:38.232+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.232650+0000) 2020-02-05T22:52:38.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:38.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.498501+0000) 2020-02-05T22:52:38.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:38.497+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.498839+0000) 2020-02-05T22:52:38.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:38.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.499310+0000) 2020-02-05T22:52:38.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:38.498+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.499742+0000) 2020-02-05T22:52:38.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:38.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.500366+0000) 2020-02-05T22:52:38.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:38.499+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.500702+0000) 2020-02-05T22:52:38.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:38.502+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.502471+0000) 2020-02-05T22:52:38.504 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:38.502+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.502508+0000) 2020-02-05T22:52:38.504 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:38.502+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.503142+0000) 2020-02-05T22:52:38.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:38.502+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.503168+0000) 2020-02-05T22:52:38.505 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:38.503+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.503751+0000) 2020-02-05T22:52:38.506 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:38.504+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.504473+0000) 2020-02-05T22:52:38.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:38.504+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.504862+0000) 2020-02-05T22:52:38.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:38.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:38.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:38.509+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:38.510863+0000) 2020-02-05T22:52:39.022 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:52:39.028 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:52:39.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:39.507+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:39.508130+0000) 2020-02-05T22:52:40.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:40.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:40.114803+0000) 2020-02-05T22:52:40.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:40.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:40.351479+0000) 2020-02-05T22:52:40.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:40.366+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:40.366453+0000) 2020-02-05T22:52:42.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:42.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:42.365698+0000) 2020-02-05T22:52:42.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:42.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:42.720607+0000) 2020-02-05T22:52:42.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:42.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:42.757343+0000) 2020-02-05T22:52:43.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:43.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.518593+0000) 2020-02-05T22:52:43.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:43.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.520339+0000) 2020-02-05T22:52:43.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:43.520+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.520943+0000) 2020-02-05T22:52:43.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:43.520+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.521145+0000) 2020-02-05T22:52:43.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:43.520+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.521606+0000) 2020-02-05T22:52:43.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:43.521+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.522308+0000) 2020-02-05T22:52:43.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:43.521+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.522476+0000) 2020-02-05T22:52:43.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:43.524+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.524485+0000) 2020-02-05T22:52:43.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:43.524+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.524873+0000) 2020-02-05T22:52:43.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:43.524+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.525397+0000) 2020-02-05T22:52:43.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:43.525+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.525798+0000) 2020-02-05T22:52:43.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:43.526+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.526459+0000) 2020-02-05T22:52:43.527 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:43.525+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.526435+0000) 2020-02-05T22:52:43.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:43.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.529517+0000) 2020-02-05T22:52:43.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:43.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:43.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:43.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.879601+0000) 2020-02-05T22:52:43.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:43.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:43.879786+0000) 2020-02-05T22:52:45.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:45.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:45.979765+0000) 2020-02-05T22:52:47.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:47.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:47.120057+0000) 2020-02-05T22:52:47.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:47.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:47.120086+0000) 2020-02-05T22:52:47.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:47.120+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:47.120564+0000) 2020-02-05T22:52:48.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:48.240+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.241024+0000) 2020-02-05T22:52:48.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:48.541+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.542245+0000) 2020-02-05T22:52:48.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:48.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.543626+0000) 2020-02-05T22:52:48.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:48.543+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.544069+0000) 2020-02-05T22:52:48.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:48.544+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.544527+0000) 2020-02-05T22:52:48.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:48.543+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.545033+0000) 2020-02-05T22:52:48.546 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:48.545+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.545725+0000) 2020-02-05T22:52:48.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:48.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.546251+0000) 2020-02-05T22:52:48.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:48.545+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.546508+0000) 2020-02-05T22:52:48.548 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:48.545+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.546821+0000) 2020-02-05T22:52:48.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:48.547+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.547526+0000) 2020-02-05T22:52:48.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:48.546+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.547380+0000) 2020-02-05T22:52:48.549 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:48.547+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.548154+0000) 2020-02-05T22:52:48.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:48.547+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.548338+0000) 2020-02-05T22:52:48.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:48.552+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:48.553378+0000) 2020-02-05T22:52:48.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:48.558+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:49.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:49.507+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:49.508360+0000) 2020-02-05T22:52:50.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:50.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:50.115035+0000) 2020-02-05T22:52:50.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:50.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:50.351751+0000) 2020-02-05T22:52:50.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:50.366+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:50.366671+0000) 2020-02-05T22:52:52.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:52.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:52.365950+0000) 2020-02-05T22:52:52.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:52.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:52.720826+0000) 2020-02-05T22:52:52.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:52.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:52.757600+0000) 2020-02-05T22:52:53.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:53.563+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.564454+0000) 2020-02-05T22:52:53.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:53.563+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.564928+0000) 2020-02-05T22:52:53.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:53.564+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.565537+0000) 2020-02-05T22:52:53.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:53.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.566138+0000) 2020-02-05T22:52:53.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:53.566+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.567157+0000) 2020-02-05T22:52:53.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:53.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.567651+0000) 2020-02-05T22:52:53.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:53.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.568339+0000) 2020-02-05T22:52:53.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:53.568+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.568520+0000) 2020-02-05T22:52:53.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:53.567+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.569048+0000) 2020-02-05T22:52:53.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:53.569+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.569701+0000) 2020-02-05T22:52:53.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:53.568+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.569696+0000) 2020-02-05T22:52:53.571 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:53.569+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.570350+0000) 2020-02-05T22:52:53.571 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:53.570+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.570687+0000) 2020-02-05T22:52:53.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:53.574+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.576065+0000) 2020-02-05T22:52:53.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:53.578+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:53.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:53.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.879845+0000) 2020-02-05T22:52:53.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:53.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:53.879981+0000) 2020-02-05T22:52:55.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:55.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:55.980019+0000) 2020-02-05T22:52:57.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:57.120+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:57.120284+0000) 2020-02-05T22:52:57.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:57.120+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:57.120325+0000) 2020-02-05T22:52:57.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:57.120+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:57.120785+0000) 2020-02-05T22:52:58.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:58.249+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.249067+0000) 2020-02-05T22:52:58.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:52:58.583+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.583692+0000) 2020-02-05T22:52:58.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:58.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.584754+0000) 2020-02-05T22:52:58.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:52:58.584+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.585946+0000) 2020-02-05T22:52:58.587 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:52:58.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.586235+0000) 2020-02-05T22:52:58.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:52:58.586+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.586410+0000) 2020-02-05T22:52:58.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:52:58.586+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.586727+0000) 2020-02-05T22:52:58.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:52:58.585+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.586802+0000) 2020-02-05T22:52:58.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:52:58.585+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.587293+0000) 2020-02-05T22:52:58.589 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:52:58.586+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.587040+0000) 2020-02-05T22:52:58.589 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:52:58.587+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.588486+0000) 2020-02-05T22:52:58.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:52:58.587+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.589140+0000) 2020-02-05T22:52:58.590 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:52:58.588+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.589716+0000) 2020-02-05T22:52:58.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:58.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.594855+0000) 2020-02-05T22:52:58.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:58.595+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:58.595116+0000) 2020-02-05T22:52:58.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:52:58.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:52:59.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:52:59.508+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:52:59.508569+0000) 2020-02-05T22:53:00.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:00.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:00.115303+0000) 2020-02-05T22:53:00.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:00.350+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:00.352028+0000) 2020-02-05T22:53:00.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:00.365+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:00.366901+0000) 2020-02-05T22:53:02.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:02.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:02.366202+0000) 2020-02-05T22:53:02.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:02.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:02.721050+0000) 2020-02-05T22:53:02.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:02.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:02.757835+0000) 2020-02-05T22:53:03.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:03.604+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.605520+0000) 2020-02-05T22:53:03.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:03.605+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.606371+0000) 2020-02-05T22:53:03.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:03.605+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.606698+0000) 2020-02-05T22:53:03.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:03.606+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.607225+0000) 2020-02-05T22:53:03.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:03.605+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.607193+0000) 2020-02-05T22:53:03.608 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:03.606+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.607689+0000) 2020-02-05T22:53:03.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:03.606+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.608012+0000) 2020-02-05T22:53:03.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:03.607+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.608513+0000) 2020-02-05T22:53:03.610 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:03.607+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.608770+0000) 2020-02-05T22:53:03.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:03.608+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.609120+0000) 2020-02-05T22:53:03.610 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:03.608+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.609899+0000) 2020-02-05T22:53:03.611 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:03.608+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.610277+0000) 2020-02-05T22:53:03.611 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:03.609+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.610800+0000) 2020-02-05T22:53:03.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:03.613+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.614382+0000) 2020-02-05T22:53:03.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:03.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:03.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:03.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.880068+0000) 2020-02-05T22:53:03.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:03.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:03.880189+0000) 2020-02-05T22:53:05.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:05.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:05.980282+0000) 2020-02-05T22:53:07.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:07.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:07.120552+0000) 2020-02-05T22:53:07.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:07.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:07.120630+0000) 2020-02-05T22:53:07.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:07.119+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:07.121031+0000) 2020-02-05T22:53:08.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:08.257+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.258060+0000) 2020-02-05T22:53:08.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:08.623+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.624830+0000) 2020-02-05T22:53:08.626 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:08.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.625503+0000) 2020-02-05T22:53:08.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:08.624+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.625951+0000) 2020-02-05T22:53:08.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:08.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.626484+0000) 2020-02-05T22:53:08.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:08.626+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.627354+0000) 2020-02-05T22:53:08.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:08.626+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.628029+0000) 2020-02-05T22:53:08.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:08.627+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.628718+0000) 2020-02-05T22:53:08.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:08.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.629257+0000) 2020-02-05T22:53:08.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:08.628+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.629419+0000) 2020-02-05T22:53:08.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:08.628+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.629807+0000) 2020-02-05T22:53:08.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:08.628+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.630029+0000) 2020-02-05T22:53:08.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:08.629+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.630524+0000) 2020-02-05T22:53:08.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:08.630+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.631619+0000) 2020-02-05T22:53:08.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:08.633+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:08.635022+0000) 2020-02-05T22:53:08.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:08.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:09.158 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:53:09.164 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:53:09.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:09.507+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:09.508786+0000) 2020-02-05T22:53:10.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:10.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:10.115569+0000) 2020-02-05T22:53:10.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:10.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:10.352266+0000) 2020-02-05T22:53:10.368 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:10.366+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:10.367135+0000) 2020-02-05T22:53:12.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:12.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:12.366474+0000) 2020-02-05T22:53:12.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:12.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:12.721267+0000) 2020-02-05T22:53:12.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:12.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:12.758073+0000) 2020-02-05T22:53:13.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:13.646+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.647264+0000) 2020-02-05T22:53:13.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:13.646+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.647585+0000) 2020-02-05T22:53:13.649 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:13.647+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.648579+0000) 2020-02-05T22:53:13.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:13.647+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.649000+0000) 2020-02-05T22:53:13.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:13.648+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.649329+0000) 2020-02-05T22:53:13.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:13.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.649590+0000) 2020-02-05T22:53:13.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:13.649+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.650342+0000) 2020-02-05T22:53:13.651 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:13.648+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.650213+0000) 2020-02-05T22:53:13.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:13.649+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.650917+0000) 2020-02-05T22:53:13.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:13.650+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.652388+0000) 2020-02-05T22:53:13.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:13.651+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.652608+0000) 2020-02-05T22:53:13.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:13.651+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.652600+0000) 2020-02-05T22:53:13.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:13.651+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.652975+0000) 2020-02-05T22:53:13.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:13.655+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.656509+0000) 2020-02-05T22:53:13.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:13.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:13.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:13.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.880325+0000) 2020-02-05T22:53:13.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:13.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:13.880428+0000) 2020-02-05T22:53:15.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:15.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:15.980537+0000) 2020-02-05T22:53:17.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:17.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:17.120813+0000) 2020-02-05T22:53:17.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:17.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:17.120926+0000) 2020-02-05T22:53:17.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:17.119+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:17.121254+0000) 2020-02-05T22:53:18.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:18.265+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.266362+0000) 2020-02-05T22:53:18.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:18.665+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.666952+0000) 2020-02-05T22:53:18.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:18.667+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.668740+0000) 2020-02-05T22:53:18.670 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:18.667+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.668673+0000) 2020-02-05T22:53:18.670 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:18.668+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.669268+0000) 2020-02-05T22:53:18.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:18.668+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.669755+0000) 2020-02-05T22:53:18.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:18.669+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.670481+0000) 2020-02-05T22:53:18.672 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:18.669+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.671325+0000) 2020-02-05T22:53:18.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:18.670+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.671751+0000) 2020-02-05T22:53:18.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:18.671+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.672303+0000) 2020-02-05T22:53:18.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:18.671+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.672778+0000) 2020-02-05T22:53:18.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:18.672+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.673220+0000) 2020-02-05T22:53:18.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:18.672+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.673955+0000) 2020-02-05T22:53:18.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:18.672+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.674213+0000) 2020-02-05T22:53:18.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:18.678+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:18.679116+0000) 2020-02-05T22:53:18.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:18.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:19.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:19.508+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:19.509023+0000) 2020-02-05T22:53:20.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:20.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:20.115855+0000) 2020-02-05T22:53:20.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:20.351+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:20.352520+0000) 2020-02-05T22:53:20.368 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:20.366+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:20.367392+0000) 2020-02-05T22:53:22.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:22.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:22.366714+0000) 2020-02-05T22:53:22.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:22.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:22.721495+0000) 2020-02-05T22:53:22.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:22.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:22.758372+0000) 2020-02-05T22:53:23.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:23.686+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.687772+0000) 2020-02-05T22:53:23.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:23.687+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.688502+0000) 2020-02-05T22:53:23.690 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:23.688+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.689269+0000) 2020-02-05T22:53:23.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:23.688+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.689420+0000) 2020-02-05T22:53:23.691 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:23.688+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.690326+0000) 2020-02-05T22:53:23.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:23.690+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.691144+0000) 2020-02-05T22:53:23.692 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:23.690+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.691389+0000) 2020-02-05T22:53:23.692 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:23.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.691751+0000) 2020-02-05T22:53:23.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:23.690+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.691905+0000) 2020-02-05T22:53:23.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:23.691+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.692443+0000) 2020-02-05T22:53:23.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:23.691+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.693248+0000) 2020-02-05T22:53:23.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:23.692+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.693616+0000) 2020-02-05T22:53:23.695 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:23.693+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.694458+0000) 2020-02-05T22:53:23.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:23.697+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.698236+0000) 2020-02-05T22:53:23.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:23.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128430 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:23.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:23.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.880550+0000) 2020-02-05T22:53:23.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:23.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:23.880670+0000) 2020-02-05T22:53:25.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:25.979+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:25.980738+0000) 2020-02-05T22:53:27.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:27.119+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:27.121031+0000) 2020-02-05T22:53:27.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:27.119+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:27.121167+0000) 2020-02-05T22:53:27.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:27.120+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:27.121495+0000) 2020-02-05T22:53:28.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:28.274+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.274881+0000) 2020-02-05T22:53:28.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:28.708+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.709879+0000) 2020-02-05T22:53:28.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:28.709+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.710585+0000) 2020-02-05T22:53:28.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:28.710+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.711545+0000) 2020-02-05T22:53:28.712 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:28.710+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.711813+0000) 2020-02-05T22:53:28.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:28.711+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.712345+0000) 2020-02-05T22:53:28.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:28.711+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.712740+0000) 2020-02-05T22:53:28.714 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:28.711+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.712617+0000) 2020-02-05T22:53:28.714 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:28.711+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.713143+0000) 2020-02-05T22:53:28.714 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:28.712+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.713800+0000) 2020-02-05T22:53:28.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:28.712+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.714011+0000) 2020-02-05T22:53:28.715 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:28.713+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.714570+0000) 2020-02-05T22:53:28.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:28.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.714283+0000) 2020-02-05T22:53:28.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:28.713+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.714921+0000) 2020-02-05T22:53:28.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:28.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:28.718874+0000) 2020-02-05T22:53:28.724 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:28.722+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:29.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:29.508+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:29.509261+0000) 2020-02-05T22:53:30.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:30.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:30.116114+0000) 2020-02-05T22:53:30.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:30.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:30.352756+0000) 2020-02-05T22:53:30.368 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:30.366+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:30.367642+0000) 2020-02-05T22:53:32.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:32.366+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:32.366981+0000) 2020-02-05T22:53:32.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:32.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:32.721728+0000) 2020-02-05T22:53:32.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:32.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:32.758623+0000) 2020-02-05T22:53:33.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:33.730+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.731158+0000) 2020-02-05T22:53:33.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:33.730+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.731660+0000) 2020-02-05T22:53:33.733 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:33.731+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.732069+0000) 2020-02-05T22:53:33.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:33.731+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.732525+0000) 2020-02-05T22:53:33.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:33.732+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.733303+0000) 2020-02-05T22:53:33.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:33.732+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.733863+0000) 2020-02-05T22:53:33.735 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:33.732+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.733806+0000) 2020-02-05T22:53:33.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:33.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.734505+0000) 2020-02-05T22:53:33.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:33.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.735009+0000) 2020-02-05T22:53:33.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:33.733+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.735082+0000) 2020-02-05T22:53:33.736 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:33.734+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.735546+0000) 2020-02-05T22:53:33.737 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:33.734+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.735675+0000) 2020-02-05T22:53:33.737 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:33.734+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.736001+0000) 2020-02-05T22:53:33.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:33.742+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.743740+0000) 2020-02-05T22:53:33.745 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:33.742+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:33.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:33.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.880746+0000) 2020-02-05T22:53:33.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:33.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:33.880881+0000) 2020-02-05T22:53:35.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:35.980+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:35.980991+0000) 2020-02-05T22:53:37.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:37.120+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:37.121296+0000) 2020-02-05T22:53:37.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:37.120+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:37.121422+0000) 2020-02-05T22:53:37.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:37.120+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:37.121751+0000) 2020-02-05T22:53:38.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:38.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.283582+0000) 2020-02-05T22:53:38.749 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:38.747+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.748789+0000) 2020-02-05T22:53:38.750 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:38.748+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.749745+0000) 2020-02-05T22:53:38.752 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:38.750+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.751236+0000) 2020-02-05T22:53:38.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:38.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.752616+0000) 2020-02-05T22:53:38.754 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:38.751+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.752497+0000) 2020-02-05T22:53:38.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:38.752+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.753259+0000) 2020-02-05T22:53:38.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:38.752+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.753552+0000) 2020-02-05T22:53:38.755 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:38.751+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.752964+0000) 2020-02-05T22:53:38.755 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:38.752+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.754299+0000) 2020-02-05T22:53:38.755 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:38.753+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.754554+0000) 2020-02-05T22:53:38.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:38.753+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.754868+0000) 2020-02-05T22:53:38.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:38.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.755576+0000) 2020-02-05T22:53:38.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:38.755+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.756566+0000) 2020-02-05T22:53:38.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:38.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:38.763441+0000) 2020-02-05T22:53:38.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:38.764+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:39.286 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:53:39.293 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:53:39.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:39.509+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:39.509483+0000) 2020-02-05T22:53:40.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:40.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:40.116358+0000) 2020-02-05T22:53:40.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:40.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:40.353003+0000) 2020-02-05T22:53:40.368 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:40.367+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:40.367902+0000) 2020-02-05T22:53:42.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:42.365+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:42.367271+0000) 2020-02-05T22:53:42.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:42.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:42.721952+0000) 2020-02-05T22:53:42.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:42.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:42.758802+0000) 2020-02-05T22:53:43.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:43.770+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.771887+0000) 2020-02-05T22:53:43.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:43.771+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.772307+0000) 2020-02-05T22:53:43.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:43.772+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.773892+0000) 2020-02-05T22:53:43.775 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:43.773+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.774641+0000) 2020-02-05T22:53:43.776 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:43.774+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.775065+0000) 2020-02-05T22:53:43.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:43.774+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.775806+0000) 2020-02-05T22:53:43.777 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:43.775+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.776574+0000) 2020-02-05T22:53:43.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:43.775+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.776503+0000) 2020-02-05T22:53:43.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:43.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.776823+0000) 2020-02-05T22:53:43.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:43.775+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.777258+0000) 2020-02-05T22:53:43.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:43.777+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.778334+0000) 2020-02-05T22:53:43.779 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:43.777+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.778769+0000) 2020-02-05T22:53:43.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:43.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.779658+0000) 2020-02-05T22:53:43.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:43.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.779905+0000) 2020-02-05T22:53:43.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:43.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:43.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:43.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.880958+0000) 2020-02-05T22:53:43.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:43.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:43.881133+0000) 2020-02-05T22:53:45.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:45.980+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:45.981280+0000) 2020-02-05T22:53:47.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:47.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:47.121630+0000) 2020-02-05T22:53:47.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:47.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:47.121750+0000) 2020-02-05T22:53:47.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:47.121+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:47.122017+0000) 2020-02-05T22:53:48.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:48.291+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.291795+0000) 2020-02-05T22:53:48.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:48.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.792118+0000) 2020-02-05T22:53:48.793 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:48.792+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.792642+0000) 2020-02-05T22:53:48.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:48.791+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.792965+0000) 2020-02-05T22:53:48.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:48.792+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.793302+0000) 2020-02-05T22:53:48.795 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:48.793+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.794345+0000) 2020-02-05T22:53:48.795 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:48.794+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.794640+0000) 2020-02-05T22:53:48.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:48.794+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.795329+0000) 2020-02-05T22:53:48.796 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:48.795+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.795731+0000) 2020-02-05T22:53:48.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:48.795+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.796762+0000) 2020-02-05T22:53:48.798 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:48.796+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.796986+0000) 2020-02-05T22:53:48.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:48.796+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.797377+0000) 2020-02-05T22:53:48.798 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:48.797+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.797504+0000) 2020-02-05T22:53:48.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:48.796+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.798027+0000) 2020-02-05T22:53:48.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:48.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:48.798660+0000) 2020-02-05T22:53:48.808 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:48.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:49.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:49.509+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:49.509729+0000) 2020-02-05T22:53:50.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:50.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:50.116592+0000) 2020-02-05T22:53:50.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:50.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:50.353240+0000) 2020-02-05T22:53:50.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:50.368+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:50.368137+0000) 2020-02-05T22:53:52.368 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:52.367+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:52.367545+0000) 2020-02-05T22:53:52.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:52.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:52.722178+0000) 2020-02-05T22:53:52.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:52.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:52.759036+0000) 2020-02-05T22:53:53.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:53.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.813692+0000) 2020-02-05T22:53:53.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:53.812+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.813802+0000) 2020-02-05T22:53:53.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:53.813+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.814376+0000) 2020-02-05T22:53:53.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:53.813+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.814554+0000) 2020-02-05T22:53:53.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:53.814+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.814924+0000) 2020-02-05T22:53:53.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:53.814+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.815220+0000) 2020-02-05T22:53:53.816 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:53.814+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.815747+0000) 2020-02-05T22:53:53.817 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:53.815+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.816089+0000) 2020-02-05T22:53:53.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:53.814+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.816015+0000) 2020-02-05T22:53:53.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:53.816+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.816679+0000) 2020-02-05T22:53:53.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:53.816+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.817270+0000) 2020-02-05T22:53:53.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:53.817+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.817951+0000) 2020-02-05T22:53:53.820 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:53.818+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.819256+0000) 2020-02-05T22:53:53.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:53.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.823659+0000) 2020-02-05T22:53:53.830 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:53.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:53.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:53.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.881196+0000) 2020-02-05T22:53:53.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:53.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:53.881349+0000) 2020-02-05T22:53:55.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:55.981+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:55.981636+0000) 2020-02-05T22:53:57.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:57.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:57.121855+0000) 2020-02-05T22:53:57.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:57.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:57.121967+0000) 2020-02-05T22:53:57.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:57.121+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:57.122271+0000) 2020-02-05T22:53:58.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:58.299+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.300225+0000) 2020-02-05T22:53:58.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:53:58.833+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.834912+0000) 2020-02-05T22:53:58.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:53:58.834+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.835320+0000) 2020-02-05T22:53:58.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:58.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.836000+0000) 2020-02-05T22:53:58.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:53:58.835+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.836620+0000) 2020-02-05T22:53:58.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:53:58.836+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.837385+0000) 2020-02-05T22:53:58.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:53:58.836+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.837386+0000) 2020-02-05T22:53:58.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:53:58.836+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.837704+0000) 2020-02-05T22:53:58.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:53:58.837+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.838029+0000) 2020-02-05T22:53:58.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:53:58.837+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.838338+0000) 2020-02-05T22:53:58.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:53:58.837+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.838377+0000) 2020-02-05T22:53:58.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:58.838+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.839383+0000) 2020-02-05T22:53:58.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:53:58.838+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.838754+0000) 2020-02-05T22:53:58.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:53:58.839+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.840082+0000) 2020-02-05T22:53:58.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:58.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:58.843396+0000) 2020-02-05T22:53:58.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:53:58.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:53:59.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:53:59.508+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:53:59.509937+0000) 2020-02-05T22:54:00.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:00.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:00.116822+0000) 2020-02-05T22:54:00.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:00.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:00.353473+0000) 2020-02-05T22:54:00.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:00.367+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:00.368321+0000) 2020-02-05T22:54:02.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:02.367+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:02.367817+0000) 2020-02-05T22:54:02.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:02.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:02.722395+0000) 2020-02-05T22:54:02.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:02.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:02.759308+0000) 2020-02-05T22:54:03.855 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:03.853+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.854188+0000) 2020-02-05T22:54:03.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:03.854+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.855437+0000) 2020-02-05T22:54:03.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:03.855+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.856325+0000) 2020-02-05T22:54:03.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:03.855+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.856509+0000) 2020-02-05T22:54:03.858 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:03.856+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.856909+0000) 2020-02-05T22:54:03.858 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:03.857+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.857725+0000) 2020-02-05T22:54:03.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:03.857+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.858259+0000) 2020-02-05T22:54:03.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:03.857+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.858664+0000) 2020-02-05T22:54:03.860 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:03.859+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.859418+0000) 2020-02-05T22:54:03.860 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:03.859+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.859490+0000) 2020-02-05T22:54:03.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:03.858+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.859850+0000) 2020-02-05T22:54:03.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:03.859+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.859995+0000) 2020-02-05T22:54:03.861 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:03.859+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.860688+0000) 2020-02-05T22:54:03.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:03.864+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.866043+0000) 2020-02-05T22:54:03.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:03.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:03.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:03.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.881373+0000) 2020-02-05T22:54:03.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:03.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:03.881517+0000) 2020-02-05T22:54:05.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:05.981+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:05.981896+0000) 2020-02-05T22:54:07.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:07.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:07.122092+0000) 2020-02-05T22:54:07.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:07.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:07.122220+0000) 2020-02-05T22:54:07.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:07.122+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:07.122498+0000) 2020-02-05T22:54:08.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:08.306+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.307181+0000) 2020-02-05T22:54:08.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:08.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.874898+0000) 2020-02-05T22:54:08.876 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:08.875+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.875657+0000) 2020-02-05T22:54:08.877 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:08.876+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.876705+0000) 2020-02-05T22:54:08.878 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:08.876+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.877066+0000) 2020-02-05T22:54:08.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:08.876+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.877508+0000) 2020-02-05T22:54:08.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:08.876+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.877805+0000) 2020-02-05T22:54:08.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:08.877+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.877958+0000) 2020-02-05T22:54:08.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:08.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.878475+0000) 2020-02-05T22:54:08.881 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:08.877+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.879057+0000) 2020-02-05T22:54:08.881 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:08.878+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.879225+0000) 2020-02-05T22:54:08.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:08.877+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.879051+0000) 2020-02-05T22:54:08.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:08.878+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.878866+0000) 2020-02-05T22:54:08.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:08.879+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.879997+0000) 2020-02-05T22:54:08.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:08.883+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:08.884732+0000) 2020-02-05T22:54:08.891 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:08.889+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:09.434 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:54:09.440 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:54:09.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:09.509+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:09.510152+0000) 2020-02-05T22:54:10.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:10.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:10.117054+0000) 2020-02-05T22:54:10.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:10.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:10.353701+0000) 2020-02-05T22:54:10.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:10.367+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:10.368564+0000) 2020-02-05T22:54:12.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:12.367+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:12.368105+0000) 2020-02-05T22:54:12.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:12.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:12.722623+0000) 2020-02-05T22:54:12.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:12.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:12.759539+0000) 2020-02-05T22:54:13.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:13.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.881592+0000) 2020-02-05T22:54:13.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:13.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.881669+0000) 2020-02-05T22:54:13.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:13.895+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.896675+0000) 2020-02-05T22:54:13.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:13.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.897438+0000) 2020-02-05T22:54:13.898 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:13.897+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.897494+0000) 2020-02-05T22:54:13.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:13.897+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.898778+0000) 2020-02-05T22:54:13.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:13.897+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.898094+0000) 2020-02-05T22:54:13.900 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:13.898+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.898821+0000) 2020-02-05T22:54:13.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:13.898+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.898616+0000) 2020-02-05T22:54:13.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:13.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.899121+0000) 2020-02-05T22:54:13.901 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:13.898+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.899702+0000) 2020-02-05T22:54:13.901 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:13.899+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.900192+0000) 2020-02-05T22:54:13.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:13.899+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.899697+0000) 2020-02-05T22:54:13.902 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:13.900+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.900896+0000) 2020-02-05T22:54:13.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:13.899+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.900625+0000) 2020-02-05T22:54:13.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:13.905+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:13.906752+0000) 2020-02-05T22:54:13.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:13.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:15.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:15.982+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:15.982200+0000) 2020-02-05T22:54:17.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:17.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:17.122331+0000) 2020-02-05T22:54:17.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:17.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:17.122528+0000) 2020-02-05T22:54:17.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:17.122+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:17.122782+0000) 2020-02-05T22:54:18.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:18.314+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.315486+0000) 2020-02-05T22:54:18.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:18.915+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.916541+0000) 2020-02-05T22:54:18.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:18.917+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.917413+0000) 2020-02-05T22:54:18.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:18.916+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.917810+0000) 2020-02-05T22:54:18.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:18.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.918445+0000) 2020-02-05T22:54:18.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:18.918+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.919393+0000) 2020-02-05T22:54:18.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:18.918+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.919921+0000) 2020-02-05T22:54:18.921 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:18.920+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.920571+0000) 2020-02-05T22:54:18.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:18.920+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.921293+0000) 2020-02-05T22:54:18.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:18.920+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.921686+0000) 2020-02-05T22:54:18.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:18.919+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.920298+0000) 2020-02-05T22:54:18.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:18.921+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.921900+0000) 2020-02-05T22:54:18.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:18.921+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.922222+0000) 2020-02-05T22:54:18.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:18.922+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.922921+0000) 2020-02-05T22:54:18.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:18.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:18.926572+0000) 2020-02-05T22:54:18.934 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:18.933+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:19.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:19.509+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:19.510354+0000) 2020-02-05T22:54:20.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:20.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:20.117272+0000) 2020-02-05T22:54:20.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:20.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:20.353949+0000) 2020-02-05T22:54:20.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:20.368+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:20.368801+0000) 2020-02-05T22:54:22.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:22.368+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:22.368355+0000) 2020-02-05T22:54:22.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:22.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:22.722854+0000) 2020-02-05T22:54:22.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:22.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:22.759770+0000) 2020-02-05T22:54:23.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:23.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.881831+0000) 2020-02-05T22:54:23.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:23.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.881891+0000) 2020-02-05T22:54:23.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:23.937+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.938239+0000) 2020-02-05T22:54:23.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:23.937+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.938816+0000) 2020-02-05T22:54:23.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:23.938+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.939780+0000) 2020-02-05T22:54:23.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:23.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.940369+0000) 2020-02-05T22:54:23.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:23.941+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.941789+0000) 2020-02-05T22:54:23.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:23.941+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.942247+0000) 2020-02-05T22:54:23.943 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:23.942+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.942469+0000) 2020-02-05T22:54:23.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:23.941+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.942750+0000) 2020-02-05T22:54:23.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:23.942+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.943137+0000) 2020-02-05T22:54:23.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:23.942+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.942687+0000) 2020-02-05T22:54:23.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:23.942+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.943201+0000) 2020-02-05T22:54:23.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:23.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.943652+0000) 2020-02-05T22:54:23.946 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:23.943+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.944323+0000) 2020-02-05T22:54:23.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:23.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:23.949189+0000) 2020-02-05T22:54:23.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:23.954+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128927 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:25.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:25.982+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:25.982481+0000) 2020-02-05T22:54:27.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:27.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:27.122659+0000) 2020-02-05T22:54:27.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:27.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:27.122804+0000) 2020-02-05T22:54:27.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:27.121+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:27.123034+0000) 2020-02-05T22:54:28.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:28.323+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.323841+0000) 2020-02-05T22:54:28.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:28.959+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.960640+0000) 2020-02-05T22:54:28.963 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:28.960+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.962256+0000) 2020-02-05T22:54:28.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:28.961+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.962750+0000) 2020-02-05T22:54:28.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:28.962+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.963508+0000) 2020-02-05T22:54:28.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:28.962+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.963741+0000) 2020-02-05T22:54:28.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:28.963+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.964653+0000) 2020-02-05T22:54:28.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:28.963+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.965091+0000) 2020-02-05T22:54:28.966 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:28.964+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.965406+0000) 2020-02-05T22:54:28.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:28.965+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.966131+0000) 2020-02-05T22:54:28.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:28.965+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.966642+0000) 2020-02-05T22:54:28.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:28.966+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.967817+0000) 2020-02-05T22:54:28.969 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:28.966+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.968085+0000) 2020-02-05T22:54:28.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:28.967+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.968426+0000) 2020-02-05T22:54:28.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:28.972+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:28.973749+0000) 2020-02-05T22:54:28.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:28.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 128969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:29.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:29.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:29.510568+0000) 2020-02-05T22:54:30.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:30.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:30.117509+0000) 2020-02-05T22:54:30.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:30.352+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:30.354158+0000) 2020-02-05T22:54:30.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:30.367+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:30.369042+0000) 2020-02-05T22:54:32.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:32.367+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:32.368662+0000) 2020-02-05T22:54:32.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:32.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:32.723108+0000) 2020-02-05T22:54:32.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:32.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:32.759999+0000) 2020-02-05T22:54:33.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:33.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.882125+0000) 2020-02-05T22:54:33.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:33.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.882125+0000) 2020-02-05T22:54:33.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:33.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.984605+0000) 2020-02-05T22:54:33.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:33.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.984963+0000) 2020-02-05T22:54:33.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:33.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.985458+0000) 2020-02-05T22:54:33.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:33.984+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.985604+0000) 2020-02-05T22:54:33.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:33.984+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.985917+0000) 2020-02-05T22:54:33.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:33.984+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.986342+0000) 2020-02-05T22:54:33.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:33.985+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.986722+0000) 2020-02-05T22:54:33.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:33.986+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.987311+0000) 2020-02-05T22:54:33.988 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:33.986+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.987653+0000) 2020-02-05T22:54:33.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:33.987+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.988623+0000) 2020-02-05T22:54:33.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:33.987+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.988851+0000) 2020-02-05T22:54:33.990 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:33.988+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.989562+0000) 2020-02-05T22:54:33.990 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:33.987+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.988681+0000) 2020-02-05T22:54:33.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:33.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:33.994977+0000) 2020-02-05T22:54:33.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:33.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129011 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:35.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:35.981+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:35.982723+0000) 2020-02-05T22:54:37.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:37.121+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:37.122911+0000) 2020-02-05T22:54:37.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:37.121+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:37.123089+0000) 2020-02-05T22:54:37.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:37.121+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:37.123251+0000) 2020-02-05T22:54:38.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:38.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:38.332330+0000) 2020-02-05T22:54:39.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:39.002+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.003727+0000) 2020-02-05T22:54:39.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:39.004+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.005505+0000) 2020-02-05T22:54:39.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:39.004+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.005979+0000) 2020-02-05T22:54:39.007 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:39.005+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.006452+0000) 2020-02-05T22:54:39.007 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:39.005+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.006603+0000) 2020-02-05T22:54:39.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:39.006+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.007179+0000) 2020-02-05T22:54:39.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:39.006+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.007952+0000) 2020-02-05T22:54:39.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:39.007+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.008495+0000) 2020-02-05T22:54:39.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:39.008+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.009933+0000) 2020-02-05T22:54:39.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:39.008+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.010222+0000) 2020-02-05T22:54:39.014 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:39.009+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.010762+0000) 2020-02-05T22:54:39.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:39.009+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.010934+0000) 2020-02-05T22:54:39.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:39.008+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.009165+0000) 2020-02-05T22:54:39.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:39.015+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.016617+0000) 2020-02-05T22:54:39.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:39.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:39.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:39.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:39.510772+0000) 2020-02-05T22:54:39.572 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:54:39.579 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:54:40.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:40.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:40.117759+0000) 2020-02-05T22:54:40.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:40.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:40.354472+0000) 2020-02-05T22:54:40.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:40.369+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:40.369306+0000) 2020-02-05T22:54:42.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:42.368+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:42.369004+0000) 2020-02-05T22:54:42.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:42.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:42.723370+0000) 2020-02-05T22:54:42.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:42.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:42.760294+0000) 2020-02-05T22:54:43.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:43.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:43.882390+0000) 2020-02-05T22:54:43.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:43.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:43.882388+0000) 2020-02-05T22:54:44.027 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:44.026+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.026378+0000) 2020-02-05T22:54:44.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:44.025+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.026603+0000) 2020-02-05T22:54:44.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:44.028+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.028219+0000) 2020-02-05T22:54:44.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:44.028+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.028556+0000) 2020-02-05T22:54:44.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:44.029+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.029110+0000) 2020-02-05T22:54:44.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:44.028+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.029799+0000) 2020-02-05T22:54:44.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:44.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.030219+0000) 2020-02-05T22:54:44.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:44.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.030677+0000) 2020-02-05T22:54:44.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:44.030+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.030895+0000) 2020-02-05T22:54:44.032 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:44.030+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.031528+0000) 2020-02-05T22:54:44.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:44.030+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.031729+0000) 2020-02-05T22:54:44.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:44.032+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.032431+0000) 2020-02-05T22:54:44.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:44.030+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.032342+0000) 2020-02-05T22:54:44.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:44.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:44.034470+0000) 2020-02-05T22:54:44.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:44.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129093 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:45.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:45.982+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:45.982971+0000) 2020-02-05T22:54:47.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:47.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:47.123169+0000) 2020-02-05T22:54:47.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:47.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:47.123309+0000) 2020-02-05T22:54:47.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:47.122+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:47.123551+0000) 2020-02-05T22:54:48.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:48.340+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:48.340398+0000) 2020-02-05T22:54:49.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:49.046+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.047662+0000) 2020-02-05T22:54:49.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:49.047+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.048567+0000) 2020-02-05T22:54:49.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:49.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.049143+0000) 2020-02-05T22:54:49.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:49.048+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.049779+0000) 2020-02-05T22:54:49.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:49.049+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.050161+0000) 2020-02-05T22:54:49.051 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:49.049+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.050285+0000) 2020-02-05T22:54:49.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:49.049+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.050541+0000) 2020-02-05T22:54:49.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:49.049+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.050781+0000) 2020-02-05T22:54:49.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:49.049+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.051026+0000) 2020-02-05T22:54:49.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:49.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.051473+0000) 2020-02-05T22:54:49.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:49.050+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.051723+0000) 2020-02-05T22:54:49.053 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:49.050+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.051919+0000) 2020-02-05T22:54:49.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:49.051+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.053030+0000) 2020-02-05T22:54:49.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:49.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.057636+0000) 2020-02-05T22:54:49.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:49.061+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:49.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:49.509+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:49.510985+0000) 2020-02-05T22:54:50.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:50.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:50.117988+0000) 2020-02-05T22:54:50.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:50.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:50.354732+0000) 2020-02-05T22:54:50.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:50.368+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:50.369560+0000) 2020-02-05T22:54:52.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:52.367+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:52.369266+0000) 2020-02-05T22:54:52.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:52.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:52.723594+0000) 2020-02-05T22:54:52.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:52.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:52.760554+0000) 2020-02-05T22:54:53.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:53.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:53.882629+0000) 2020-02-05T22:54:53.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:53.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:53.882627+0000) 2020-02-05T22:54:54.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:54.065+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.067093+0000) 2020-02-05T22:54:54.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:54.069+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.070440+0000) 2020-02-05T22:54:54.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:54.069+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.070721+0000) 2020-02-05T22:54:54.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:54.069+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.070839+0000) 2020-02-05T22:54:54.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:54.070+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.071333+0000) 2020-02-05T22:54:54.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:54.070+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.071633+0000) 2020-02-05T22:54:54.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:54.071+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.072572+0000) 2020-02-05T22:54:54.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:54.071+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.072803+0000) 2020-02-05T22:54:54.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:54.071+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.073056+0000) 2020-02-05T22:54:54.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:54.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.073892+0000) 2020-02-05T22:54:54.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:54.072+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.074086+0000) 2020-02-05T22:54:54.075 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:54.072+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.074177+0000) 2020-02-05T22:54:54.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:54.073+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.075122+0000) 2020-02-05T22:54:54.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:54.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:54.078882+0000) 2020-02-05T22:54:54.084 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:54.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:55.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:55.982+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:55.983228+0000) 2020-02-05T22:54:57.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:57.122+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:57.123384+0000) 2020-02-05T22:54:57.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:57.122+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:57.123515+0000) 2020-02-05T22:54:57.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:57.122+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:57.123772+0000) 2020-02-05T22:54:58.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:58.347+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:58.348703+0000) 2020-02-05T22:54:59.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:54:59.087+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.088962+0000) 2020-02-05T22:54:59.090 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:54:59.088+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.089572+0000) 2020-02-05T22:54:59.091 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:54:59.088+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.089870+0000) 2020-02-05T22:54:59.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:54:59.089+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.090825+0000) 2020-02-05T22:54:59.092 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:54:59.089+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.090976+0000) 2020-02-05T22:54:59.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:59.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.091248+0000) 2020-02-05T22:54:59.093 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:54:59.090+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.091442+0000) 2020-02-05T22:54:59.093 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:54:59.090+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.091474+0000) 2020-02-05T22:54:59.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:54:59.090+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.092346+0000) 2020-02-05T22:54:59.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:59.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.092835+0000) 2020-02-05T22:54:59.094 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:54:59.091+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.093092+0000) 2020-02-05T22:54:59.095 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:54:59.092+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.093826+0000) 2020-02-05T22:54:59.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:54:59.092+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.094149+0000) 2020-02-05T22:54:59.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:59.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.098324+0000) 2020-02-05T22:54:59.103 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:54:59.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:54:59.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:54:59.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:54:59.511193+0000) 2020-02-05T22:55:00.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:00.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:00.118245+0000) 2020-02-05T22:55:00.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:00.353+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:00.354971+0000) 2020-02-05T22:55:00.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:00.368+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:00.369804+0000) 2020-02-05T22:55:02.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:02.368+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:02.369506+0000) 2020-02-05T22:55:02.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:02.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:02.723821+0000) 2020-02-05T22:55:02.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:02.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:02.760794+0000) 2020-02-05T22:55:03.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:03.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:03.882889+0000) 2020-02-05T22:55:03.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:03.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:03.882889+0000) 2020-02-05T22:55:04.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:04.106+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.107437+0000) 2020-02-05T22:55:04.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:04.107+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.109279+0000) 2020-02-05T22:55:04.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:04.108+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.109483+0000) 2020-02-05T22:55:04.111 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:04.109+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.110822+0000) 2020-02-05T22:55:04.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:04.110+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.111528+0000) 2020-02-05T22:55:04.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:04.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.112587+0000) 2020-02-05T22:55:04.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:04.111+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.112604+0000) 2020-02-05T22:55:04.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:04.112+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.113188+0000) 2020-02-05T22:55:04.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:04.112+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.113569+0000) 2020-02-05T22:55:04.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:04.111+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.112876+0000) 2020-02-05T22:55:04.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:04.112+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.113541+0000) 2020-02-05T22:55:04.115 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:04.113+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.114133+0000) 2020-02-05T22:55:04.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:04.113+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.114961+0000) 2020-02-05T22:55:04.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:04.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:04.117620+0000) 2020-02-05T22:55:04.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:04.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:05.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:05.982+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:05.983483+0000) 2020-02-05T22:55:07.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:07.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:07.123787+0000) 2020-02-05T22:55:07.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:07.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:07.123678+0000) 2020-02-05T22:55:07.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:07.123+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:07.124016+0000) 2020-02-05T22:55:08.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:08.356+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:08.357347+0000) 2020-02-05T22:55:09.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:09.128+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.129180+0000) 2020-02-05T22:55:09.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:09.128+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.129693+0000) 2020-02-05T22:55:09.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:09.128+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.130301+0000) 2020-02-05T22:55:09.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:09.129+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.130529+0000) 2020-02-05T22:55:09.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:09.129+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.131021+0000) 2020-02-05T22:55:09.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:09.130+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.131420+0000) 2020-02-05T22:55:09.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:09.131+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.132552+0000) 2020-02-05T22:55:09.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:09.131+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.132651+0000) 2020-02-05T22:55:09.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:09.133+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.134596+0000) 2020-02-05T22:55:09.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:09.133+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.134686+0000) 2020-02-05T22:55:09.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:09.133+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.134655+0000) 2020-02-05T22:55:09.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:09.134+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.135378+0000) 2020-02-05T22:55:09.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:09.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.135665+0000) 2020-02-05T22:55:09.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:09.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.140236+0000) 2020-02-05T22:55:09.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:09.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:09.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:09.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:09.511392+0000) 2020-02-05T22:55:09.695 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:55:09.701 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:55:10.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:10.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:10.118441+0000) 2020-02-05T22:55:10.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:10.354+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:10.355228+0000) 2020-02-05T22:55:10.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:10.369+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:10.370069+0000) 2020-02-05T22:55:12.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:12.369+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:12.369745+0000) 2020-02-05T22:55:12.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:12.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:12.724041+0000) 2020-02-05T22:55:12.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:12.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:12.761041+0000) 2020-02-05T22:55:13.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:13.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:13.883184+0000) 2020-02-05T22:55:13.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:13.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:13.883184+0000) 2020-02-05T22:55:14.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:14.149+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.150162+0000) 2020-02-05T22:55:14.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:14.149+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.150908+0000) 2020-02-05T22:55:14.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:14.151+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.153050+0000) 2020-02-05T22:55:14.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:14.152+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.153891+0000) 2020-02-05T22:55:14.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:14.153+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.154226+0000) 2020-02-05T22:55:14.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:14.153+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.154503+0000) 2020-02-05T22:55:14.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:14.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.155221+0000) 2020-02-05T22:55:14.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:14.153+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.155185+0000) 2020-02-05T22:55:14.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:14.154+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.155512+0000) 2020-02-05T22:55:14.157 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:14.155+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.156550+0000) 2020-02-05T22:55:14.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:14.154+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.156174+0000) 2020-02-05T22:55:14.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:14.154+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.155911+0000) 2020-02-05T22:55:14.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:14.155+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.156372+0000) 2020-02-05T22:55:14.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:14.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:14.159178+0000) 2020-02-05T22:55:14.167 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:14.165+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:15.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:15.983+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:15.983769+0000) 2020-02-05T22:55:17.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:17.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:17.124030+0000) 2020-02-05T22:55:17.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:17.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:17.124022+0000) 2020-02-05T22:55:17.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:17.123+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:17.124206+0000) 2020-02-05T22:55:18.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:18.365+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:18.365670+0000) 2020-02-05T22:55:19.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:19.171+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.172422+0000) 2020-02-05T22:55:19.175 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:19.171+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.172813+0000) 2020-02-05T22:55:19.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:19.174+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.174563+0000) 2020-02-05T22:55:19.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:19.172+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.173831+0000) 2020-02-05T22:55:19.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:19.173+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.174512+0000) 2020-02-05T22:55:19.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:19.175+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.175671+0000) 2020-02-05T22:55:19.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:19.176+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.176569+0000) 2020-02-05T22:55:19.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:19.176+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.176858+0000) 2020-02-05T22:55:19.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:19.176+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.177492+0000) 2020-02-05T22:55:19.179 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:19.177+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.178255+0000) 2020-02-05T22:55:19.179 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:19.177+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.178556+0000) 2020-02-05T22:55:19.179 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:19.177+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.178291+0000) 2020-02-05T22:55:19.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:19.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.180866+0000) 2020-02-05T22:55:19.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:19.180+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.181963+0000) 2020-02-05T22:55:19.189 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:19.187+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:19.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:19.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:19.511590+0000) 2020-02-05T22:55:20.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:20.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:20.118659+0000) 2020-02-05T22:55:20.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:20.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:20.355558+0000) 2020-02-05T22:55:20.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:20.369+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:20.370334+0000) 2020-02-05T22:55:22.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:22.369+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:22.370003+0000) 2020-02-05T22:55:22.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:22.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:22.724304+0000) 2020-02-05T22:55:22.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:22.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:22.761311+0000) 2020-02-05T22:55:23.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:23.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:23.883454+0000) 2020-02-05T22:55:23.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:23.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:23.883454+0000) 2020-02-05T22:55:24.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:24.193+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.194385+0000) 2020-02-05T22:55:24.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:24.194+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.195214+0000) 2020-02-05T22:55:24.196 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:24.194+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.195709+0000) 2020-02-05T22:55:24.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:24.194+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.196022+0000) 2020-02-05T22:55:24.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:24.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.196346+0000) 2020-02-05T22:55:24.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:24.196+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.196804+0000) 2020-02-05T22:55:24.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:24.196+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.197306+0000) 2020-02-05T22:55:24.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:24.197+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.197581+0000) 2020-02-05T22:55:24.199 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:24.197+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.198858+0000) 2020-02-05T22:55:24.200 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:24.198+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.199365+0000) 2020-02-05T22:55:24.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:24.199+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.199779+0000) 2020-02-05T22:55:24.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:24.199+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.199814+0000) 2020-02-05T22:55:24.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:24.202+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.203420+0000) 2020-02-05T22:55:24.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:24.202+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:24.203703+0000) 2020-02-05T22:55:24.209 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:24.208+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:25.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:25.983+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:25.984015+0000) 2020-02-05T22:55:27.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:27.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:27.124284+0000) 2020-02-05T22:55:27.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:27.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:27.124285+0000) 2020-02-05T22:55:27.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:27.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:27.124488+0000) 2020-02-05T22:55:28.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:28.373+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:28.373955+0000) 2020-02-05T22:55:29.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:29.213+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.214392+0000) 2020-02-05T22:55:29.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:29.214+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.215099+0000) 2020-02-05T22:55:29.216 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:29.214+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.215394+0000) 2020-02-05T22:55:29.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:29.216+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.216506+0000) 2020-02-05T22:55:29.217 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:29.215+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.215788+0000) 2020-02-05T22:55:29.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:29.216+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.217316+0000) 2020-02-05T22:55:29.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:29.217+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.218932+0000) 2020-02-05T22:55:29.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:29.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.220409+0000) 2020-02-05T22:55:29.223 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:29.219+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.220589+0000) 2020-02-05T22:55:29.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:29.219+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.220936+0000) 2020-02-05T22:55:29.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:29.218+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.219120+0000) 2020-02-05T22:55:29.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:29.219+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.219926+0000) 2020-02-05T22:55:29.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:29.220+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.220471+0000) 2020-02-05T22:55:29.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:29.223+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.224752+0000) 2020-02-05T22:55:29.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:29.229+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:29.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:29.510+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:29.511818+0000) 2020-02-05T22:55:30.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:30.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:30.118872+0000) 2020-02-05T22:55:30.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:30.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:30.355851+0000) 2020-02-05T22:55:30.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:30.370+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:30.370622+0000) 2020-02-05T22:55:32.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:32.369+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:32.370255+0000) 2020-02-05T22:55:32.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:32.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:32.724529+0000) 2020-02-05T22:55:32.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:32.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:32.761532+0000) 2020-02-05T22:55:33.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:33.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:33.883709+0000) 2020-02-05T22:55:33.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:33.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:33.883707+0000) 2020-02-05T22:55:34.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:34.236+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.237206+0000) 2020-02-05T22:55:34.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:34.236+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.237924+0000) 2020-02-05T22:55:34.239 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:34.237+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.238422+0000) 2020-02-05T22:55:34.239 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:34.238+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.239120+0000) 2020-02-05T22:55:34.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:34.239+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.240225+0000) 2020-02-05T22:55:34.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:34.239+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.240416+0000) 2020-02-05T22:55:34.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:34.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.240913+0000) 2020-02-05T22:55:34.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:34.240+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.241185+0000) 2020-02-05T22:55:34.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:34.241+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.241653+0000) 2020-02-05T22:55:34.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:34.241+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.241701+0000) 2020-02-05T22:55:34.243 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:34.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.242445+0000) 2020-02-05T22:55:34.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:34.241+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.242330+0000) 2020-02-05T22:55:34.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:34.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.247341+0000) 2020-02-05T22:55:34.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:34.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:34.247824+0000) 2020-02-05T22:55:34.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:34.252+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:35.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:35.983+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:35.984265+0000) 2020-02-05T22:55:37.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:37.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:37.124528+0000) 2020-02-05T22:55:37.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:37.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:37.124535+0000) 2020-02-05T22:55:37.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:37.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:37.124738+0000) 2020-02-05T22:55:38.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:38.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:38.382290+0000) 2020-02-05T22:55:39.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:39.259+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.259057+0000) 2020-02-05T22:55:39.260 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:39.260+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.260126+0000) 2020-02-05T22:55:39.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:39.260+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.260561+0000) 2020-02-05T22:55:39.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:39.261+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.261998+0000) 2020-02-05T22:55:39.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:39.262+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.262653+0000) 2020-02-05T22:55:39.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:39.261+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.262210+0000) 2020-02-05T22:55:39.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:39.262+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.262939+0000) 2020-02-05T22:55:39.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:39.263+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.263639+0000) 2020-02-05T22:55:39.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:39.263+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.264016+0000) 2020-02-05T22:55:39.265 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:39.263+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.263837+0000) 2020-02-05T22:55:39.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:39.264+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.264424+0000) 2020-02-05T22:55:39.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:39.264+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.264920+0000) 2020-02-05T22:55:39.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:39.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.266787+0000) 2020-02-05T22:55:39.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:39.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.268060+0000) 2020-02-05T22:55:39.273 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:39.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:39.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:39.511+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:39.512032+0000) 2020-02-05T22:55:39.852 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:55:39.859 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:55:40.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:40.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:40.119110+0000) 2020-02-05T22:55:40.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:40.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:40.356106+0000) 2020-02-05T22:55:40.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:40.370+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:40.370855+0000) 2020-02-05T22:55:42.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:42.370+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:42.370499+0000) 2020-02-05T22:55:42.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:42.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:42.724753+0000) 2020-02-05T22:55:42.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:42.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:42.761777+0000) 2020-02-05T22:55:43.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:43.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:43.883996+0000) 2020-02-05T22:55:43.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:43.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:43.883996+0000) 2020-02-05T22:55:44.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:44.279+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.280701+0000) 2020-02-05T22:55:44.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:44.280+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.280932+0000) 2020-02-05T22:55:44.283 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:44.280+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.282052+0000) 2020-02-05T22:55:44.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:44.281+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.281609+0000) 2020-02-05T22:55:44.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:44.281+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.282125+0000) 2020-02-05T22:55:44.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:44.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.282467+0000) 2020-02-05T22:55:44.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:44.281+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.282653+0000) 2020-02-05T22:55:44.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:44.282+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.283084+0000) 2020-02-05T22:55:44.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:44.282+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.283859+0000) 2020-02-05T22:55:44.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:44.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.284241+0000) 2020-02-05T22:55:44.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:44.283+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.284015+0000) 2020-02-05T22:55:44.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:44.283+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.284229+0000) 2020-02-05T22:55:44.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:44.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.285059+0000) 2020-02-05T22:55:44.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:44.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:44.289440+0000) 2020-02-05T22:55:44.296 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:44.294+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129585 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:45.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:45.984+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:45.984522+0000) 2020-02-05T22:55:47.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:47.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:47.124789+0000) 2020-02-05T22:55:47.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:47.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:47.124789+0000) 2020-02-05T22:55:47.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:47.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:47.125013+0000) 2020-02-05T22:55:48.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:48.390+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:48.390999+0000) 2020-02-05T22:55:49.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:49.300+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.301591+0000) 2020-02-05T22:55:49.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:49.301+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.302392+0000) 2020-02-05T22:55:49.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:49.302+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.302942+0000) 2020-02-05T22:55:49.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:49.302+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.303322+0000) 2020-02-05T22:55:49.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:49.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.303802+0000) 2020-02-05T22:55:49.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:49.303+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.304289+0000) 2020-02-05T22:55:49.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:49.303+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.304822+0000) 2020-02-05T22:55:49.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:49.304+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.305241+0000) 2020-02-05T22:55:49.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:49.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.305570+0000) 2020-02-05T22:55:49.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:49.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.305785+0000) 2020-02-05T22:55:49.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:49.305+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.306102+0000) 2020-02-05T22:55:49.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:49.306+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.306979+0000) 2020-02-05T22:55:49.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:49.306+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.307141+0000) 2020-02-05T22:55:49.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:49.310+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.311676+0000) 2020-02-05T22:55:49.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:49.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129628 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:49.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:49.511+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:49.512284+0000) 2020-02-05T22:55:50.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:50.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:50.119325+0000) 2020-02-05T22:55:50.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:50.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:50.356355+0000) 2020-02-05T22:55:50.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:50.370+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:50.371108+0000) 2020-02-05T22:55:52.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:52.370+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:52.370851+0000) 2020-02-05T22:55:52.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:52.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:52.724940+0000) 2020-02-05T22:55:52.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:52.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:52.762051+0000) 2020-02-05T22:55:53.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:53.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:53.884303+0000) 2020-02-05T22:55:53.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:53.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:53.884304+0000) 2020-02-05T22:55:54.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:54.322+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.323202+0000) 2020-02-05T22:55:54.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:54.324+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.325470+0000) 2020-02-05T22:55:54.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:54.325+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.325470+0000) 2020-02-05T22:55:54.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:54.325+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.326001+0000) 2020-02-05T22:55:54.327 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:54.325+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.326302+0000) 2020-02-05T22:55:54.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:54.325+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.326737+0000) 2020-02-05T22:55:54.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:54.326+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.327304+0000) 2020-02-05T22:55:54.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:54.327+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.327629+0000) 2020-02-05T22:55:54.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:54.327+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.328325+0000) 2020-02-05T22:55:54.342 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:54.328+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.328463+0000) 2020-02-05T22:55:54.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:54.327+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.328667+0000) 2020-02-05T22:55:54.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:54.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.333038+0000) 2020-02-05T22:55:54.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:54.332+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.333346+0000) 2020-02-05T22:55:54.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:54.328+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:54.328904+0000) 2020-02-05T22:55:54.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:54.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:55.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:55.984+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:55.984784+0000) 2020-02-05T22:55:57.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:57.123+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:57.125019+0000) 2020-02-05T22:55:57.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:57.123+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:57.124985+0000) 2020-02-05T22:55:57.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:57.123+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:57.125203+0000) 2020-02-05T22:55:58.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:58.398+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:58.399124+0000) 2020-02-05T22:55:59.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:55:59.341+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.342924+0000) 2020-02-05T22:55:59.344 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:55:59.342+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.343207+0000) 2020-02-05T22:55:59.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:55:59.342+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.343537+0000) 2020-02-05T22:55:59.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:55:59.342+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.343810+0000) 2020-02-05T22:55:59.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:55:59.342+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.343967+0000) 2020-02-05T22:55:59.346 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:55:59.343+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.344438+0000) 2020-02-05T22:55:59.346 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:55:59.343+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.344716+0000) 2020-02-05T22:55:59.347 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:55:59.345+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.346674+0000) 2020-02-05T22:55:59.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:55:59.345+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.346876+0000) 2020-02-05T22:55:59.348 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:55:59.345+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.347136+0000) 2020-02-05T22:55:59.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:55:59.346+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.347933+0000) 2020-02-05T22:55:59.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:59.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.348447+0000) 2020-02-05T22:55:59.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:59.352+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.353828+0000) 2020-02-05T22:55:59.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:59.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.354888+0000) 2020-02-05T22:55:59.358 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:55:59.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:55:59.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:55:59.511+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:55:59.512500+0000) 2020-02-05T22:56:00.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:00.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:00.119531+0000) 2020-02-05T22:56:00.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:00.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:00.356631+0000) 2020-02-05T22:56:00.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:00.370+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:00.371346+0000) 2020-02-05T22:56:02.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:02.370+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:02.371115+0000) 2020-02-05T22:56:02.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:02.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:02.725171+0000) 2020-02-05T22:56:02.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:02.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:02.762309+0000) 2020-02-05T22:56:03.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:03.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:03.884609+0000) 2020-02-05T22:56:03.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:03.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:03.884609+0000) 2020-02-05T22:56:04.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:04.362+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.363990+0000) 2020-02-05T22:56:04.366 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:04.363+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.365399+0000) 2020-02-05T22:56:04.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:04.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.366286+0000) 2020-02-05T22:56:04.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:04.365+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.366661+0000) 2020-02-05T22:56:04.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:04.366+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.367615+0000) 2020-02-05T22:56:04.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:04.367+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.368118+0000) 2020-02-05T22:56:04.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:04.367+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.368707+0000) 2020-02-05T22:56:04.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:04.367+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.368972+0000) 2020-02-05T22:56:04.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:04.368+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.369358+0000) 2020-02-05T22:56:04.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:04.368+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.369817+0000) 2020-02-05T22:56:04.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:04.368+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.369614+0000) 2020-02-05T22:56:04.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:04.369+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.370587+0000) 2020-02-05T22:56:04.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:04.368+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.370013+0000) 2020-02-05T22:56:04.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:04.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:04.375027+0000) 2020-02-05T22:56:04.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:04.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:05.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:05.983+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:05.985019+0000) 2020-02-05T22:56:07.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:07.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:07.125275+0000) 2020-02-05T22:56:07.126 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:07.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:07.125264+0000) 2020-02-05T22:56:07.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:07.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:07.125482+0000) 2020-02-05T22:56:08.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:08.406+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:08.407579+0000) 2020-02-05T22:56:09.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:09.384+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.385162+0000) 2020-02-05T22:56:09.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:09.384+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.386085+0000) 2020-02-05T22:56:09.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:09.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.386715+0000) 2020-02-05T22:56:09.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:09.386+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.387391+0000) 2020-02-05T22:56:09.389 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:09.386+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.387902+0000) 2020-02-05T22:56:09.390 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:09.387+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.389189+0000) 2020-02-05T22:56:09.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:09.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.389459+0000) 2020-02-05T22:56:09.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:09.388+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.389994+0000) 2020-02-05T22:56:09.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:09.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.390378+0000) 2020-02-05T22:56:09.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:09.389+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.391024+0000) 2020-02-05T22:56:09.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:09.389+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.390860+0000) 2020-02-05T22:56:09.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:09.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.391230+0000) 2020-02-05T22:56:09.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:09.390+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.391531+0000) 2020-02-05T22:56:09.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:09.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.397233+0000) 2020-02-05T22:56:09.401 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:09.399+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:09.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:09.512+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:09.512732+0000) 2020-02-05T22:56:09.989 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:56:10.000 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:56:10.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:10.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:10.119692+0000) 2020-02-05T22:56:10.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:10.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:10.356874+0000) 2020-02-05T22:56:10.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:10.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:10.371572+0000) 2020-02-05T22:56:12.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:12.369+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:12.371362+0000) 2020-02-05T22:56:12.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:12.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:12.725408+0000) 2020-02-05T22:56:12.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:12.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:12.762541+0000) 2020-02-05T22:56:13.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:13.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:13.884901+0000) 2020-02-05T22:56:13.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:13.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:13.884901+0000) 2020-02-05T22:56:14.407 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:14.405+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.406779+0000) 2020-02-05T22:56:14.408 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:14.406+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.407633+0000) 2020-02-05T22:56:14.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:14.406+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.408092+0000) 2020-02-05T22:56:14.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:14.407+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.408494+0000) 2020-02-05T22:56:14.410 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:14.407+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.409000+0000) 2020-02-05T22:56:14.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:14.408+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.409231+0000) 2020-02-05T22:56:14.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:14.408+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.409463+0000) 2020-02-05T22:56:14.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:14.409+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.410295+0000) 2020-02-05T22:56:14.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:14.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.410445+0000) 2020-02-05T22:56:14.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:14.410+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.411415+0000) 2020-02-05T22:56:14.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:14.410+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.411649+0000) 2020-02-05T22:56:14.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:14.409+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.411348+0000) 2020-02-05T22:56:14.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:14.411+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.412548+0000) 2020-02-05T22:56:14.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:14.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:14.415654+0000) 2020-02-05T22:56:14.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:14.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:15.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:15.984+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:15.985274+0000) 2020-02-05T22:56:17.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:17.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:17.125520+0000) 2020-02-05T22:56:17.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:17.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:17.125520+0000) 2020-02-05T22:56:17.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:17.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:17.125724+0000) 2020-02-05T22:56:18.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:18.415+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:18.415966+0000) 2020-02-05T22:56:19.429 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:19.426+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.427959+0000) 2020-02-05T22:56:19.429 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:19.427+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.428695+0000) 2020-02-05T22:56:19.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:19.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.429277+0000) 2020-02-05T22:56:19.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:19.428+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.429581+0000) 2020-02-05T22:56:19.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:19.428+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.430042+0000) 2020-02-05T22:56:19.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:19.429+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.430652+0000) 2020-02-05T22:56:19.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:19.429+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.430774+0000) 2020-02-05T22:56:19.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:19.430+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.431871+0000) 2020-02-05T22:56:19.433 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:19.431+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.432362+0000) 2020-02-05T22:56:19.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:19.431+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.432920+0000) 2020-02-05T22:56:19.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:19.431+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.433135+0000) 2020-02-05T22:56:19.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:19.431+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.433154+0000) 2020-02-05T22:56:19.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:19.436+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.437838+0000) 2020-02-05T22:56:19.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:19.437+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.438145+0000) 2020-02-05T22:56:19.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:19.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129876 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:19.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:19.511+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:19.512964+0000) 2020-02-05T22:56:20.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:20.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:20.119899+0000) 2020-02-05T22:56:20.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:20.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:20.357085+0000) 2020-02-05T22:56:20.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:20.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:20.371816+0000) 2020-02-05T22:56:22.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:22.370+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:22.371704+0000) 2020-02-05T22:56:22.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:22.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:22.725642+0000) 2020-02-05T22:56:22.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:22.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:22.762779+0000) 2020-02-05T22:56:23.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:23.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:23.885173+0000) 2020-02-05T22:56:23.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:23.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:23.885171+0000) 2020-02-05T22:56:24.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:24.449+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.450138+0000) 2020-02-05T22:56:24.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:24.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.450565+0000) 2020-02-05T22:56:24.452 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:24.449+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.450852+0000) 2020-02-05T22:56:24.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:24.449+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.451044+0000) 2020-02-05T22:56:24.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:24.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.451806+0000) 2020-02-05T22:56:24.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:24.451+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.452136+0000) 2020-02-05T22:56:24.453 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:24.451+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.452507+0000) 2020-02-05T22:56:24.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:24.451+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.453307+0000) 2020-02-05T22:56:24.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:24.452+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.453701+0000) 2020-02-05T22:56:24.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:24.451+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.453250+0000) 2020-02-05T22:56:24.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:24.452+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.453861+0000) 2020-02-05T22:56:24.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:24.453+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.454648+0000) 2020-02-05T22:56:24.456 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:24.453+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.455395+0000) 2020-02-05T22:56:24.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:24.457+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:24.458698+0000) 2020-02-05T22:56:24.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:24.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129917 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:25.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:25.984+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:25.985568+0000) 2020-02-05T22:56:27.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:27.125+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:27.125784+0000) 2020-02-05T22:56:27.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:27.125+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:27.125776+0000) 2020-02-05T22:56:27.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:27.125+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:27.125952+0000) 2020-02-05T22:56:28.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:28.423+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:28.423355+0000) 2020-02-05T22:56:29.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:29.470+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.471236+0000) 2020-02-05T22:56:29.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:29.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.472043+0000) 2020-02-05T22:56:29.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:29.470+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.472221+0000) 2020-02-05T22:56:29.473 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:29.471+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.472770+0000) 2020-02-05T22:56:29.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:29.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.472676+0000) 2020-02-05T22:56:29.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:29.471+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.473318+0000) 2020-02-05T22:56:29.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:29.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.473714+0000) 2020-02-05T22:56:29.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:29.472+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.474030+0000) 2020-02-05T22:56:29.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:29.473+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.474471+0000) 2020-02-05T22:56:29.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:29.473+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.474784+0000) 2020-02-05T22:56:29.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:29.474+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.475429+0000) 2020-02-05T22:56:29.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:29.473+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.475265+0000) 2020-02-05T22:56:29.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:29.474+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.475800+0000) 2020-02-05T22:56:29.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:29.479+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.481014+0000) 2020-02-05T22:56:29.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:29.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 129960 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:29.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:29.512+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:29.513172+0000) 2020-02-05T22:56:30.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:30.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:30.120130+0000) 2020-02-05T22:56:30.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:30.355+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:30.357302+0000) 2020-02-05T22:56:30.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:30.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:30.372023+0000) 2020-02-05T22:56:32.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:32.371+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:32.371961+0000) 2020-02-05T22:56:32.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:32.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:32.725904+0000) 2020-02-05T22:56:32.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:32.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:32.763033+0000) 2020-02-05T22:56:33.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:33.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:33.885434+0000) 2020-02-05T22:56:33.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:33.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:33.885434+0000) 2020-02-05T22:56:34.493 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:34.491+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.491947+0000) 2020-02-05T22:56:34.493 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:34.490+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.492387+0000) 2020-02-05T22:56:34.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:34.493+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.493793+0000) 2020-02-05T22:56:34.494 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:34.491+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.493198+0000) 2020-02-05T22:56:34.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:34.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.494348+0000) 2020-02-05T22:56:34.495 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:34.491+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.493157+0000) 2020-02-05T22:56:34.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:34.495+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.495148+0000) 2020-02-05T22:56:34.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:34.495+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.495890+0000) 2020-02-05T22:56:34.497 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:34.496+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.496321+0000) 2020-02-05T22:56:34.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:34.495+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.496610+0000) 2020-02-05T22:56:34.498 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:34.497+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.497379+0000) 2020-02-05T22:56:34.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:34.496+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.497522+0000) 2020-02-05T22:56:34.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:34.496+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.497585+0000) 2020-02-05T22:56:34.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:34.502+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:34.502999+0000) 2020-02-05T22:56:34.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:34.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130002 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:35.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:35.984+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:35.985805+0000) 2020-02-05T22:56:37.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:37.124+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:37.126036+0000) 2020-02-05T22:56:37.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:37.124+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:37.126024+0000) 2020-02-05T22:56:37.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:37.124+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:37.126259+0000) 2020-02-05T22:56:38.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:38.430+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:38.431945+0000) 2020-02-05T22:56:39.515 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:39.512+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.514343+0000) 2020-02-05T22:56:39.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:39.514+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.515614+0000) 2020-02-05T22:56:39.517 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:39.513+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.515368+0000) 2020-02-05T22:56:39.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:39.515+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.516197+0000) 2020-02-05T22:56:39.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:39.516+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.517166+0000) 2020-02-05T22:56:39.519 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:39.516+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.517662+0000) 2020-02-05T22:56:39.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:39.515+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.516604+0000) 2020-02-05T22:56:39.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:39.516+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.517512+0000) 2020-02-05T22:56:39.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:39.517+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.518199+0000) 2020-02-05T22:56:39.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:39.518+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.519228+0000) 2020-02-05T22:56:39.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:39.518+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.519714+0000) 2020-02-05T22:56:39.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:39.518+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.520255+0000) 2020-02-05T22:56:39.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:39.519+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.520220+0000) 2020-02-05T22:56:39.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:39.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:39.525448+0000) 2020-02-05T22:56:39.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:39.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130044 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:40.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:40.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:40.120343+0000) 2020-02-05T22:56:40.127 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:56:40.133 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:56:40.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:40.356+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:40.357542+0000) 2020-02-05T22:56:40.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:40.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:40.372297+0000) 2020-02-05T22:56:42.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:42.370+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:42.372209+0000) 2020-02-05T22:56:42.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:42.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:42.726131+0000) 2020-02-05T22:56:42.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:42.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:42.763295+0000) 2020-02-05T22:56:43.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:43.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:43.885707+0000) 2020-02-05T22:56:43.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:43.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:43.885710+0000) 2020-02-05T22:56:44.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:44.535+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.536573+0000) 2020-02-05T22:56:44.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:44.535+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.537258+0000) 2020-02-05T22:56:44.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:44.536+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.537969+0000) 2020-02-05T22:56:44.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:44.538+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.539478+0000) 2020-02-05T22:56:44.541 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:44.538+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.540061+0000) 2020-02-05T22:56:44.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:44.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.541037+0000) 2020-02-05T22:56:44.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:44.540+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.541626+0000) 2020-02-05T22:56:44.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:44.540+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.542003+0000) 2020-02-05T22:56:44.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:44.541+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.542484+0000) 2020-02-05T22:56:44.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:44.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.542547+0000) 2020-02-05T22:56:44.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:44.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.542519+0000) 2020-02-05T22:56:44.544 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:44.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.542988+0000) 2020-02-05T22:56:44.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:44.541+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.543036+0000) 2020-02-05T22:56:44.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:44.546+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:44.547813+0000) 2020-02-05T22:56:44.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:44.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:45.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:45.985+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:45.986087+0000) 2020-02-05T22:56:46.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:46.262+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:46.263796+0000) 2020-02-05T22:56:47.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:47.125+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:47.126289+0000) 2020-02-05T22:56:47.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:47.125+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:47.126292+0000) 2020-02-05T22:56:47.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:47.125+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:47.126671+0000) 2020-02-05T22:56:48.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:48.439+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:48.440395+0000) 2020-02-05T22:56:49.559 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:49.557+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.558616+0000) 2020-02-05T22:56:49.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:49.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.559272+0000) 2020-02-05T22:56:49.560 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:49.559+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.559955+0000) 2020-02-05T22:56:49.561 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:49.559+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.560719+0000) 2020-02-05T22:56:49.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:49.560+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.561316+0000) 2020-02-05T22:56:49.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:49.561+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.562035+0000) 2020-02-05T22:56:49.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:49.560+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.562045+0000) 2020-02-05T22:56:49.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:49.561+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.562402+0000) 2020-02-05T22:56:49.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:49.562+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.563067+0000) 2020-02-05T22:56:49.564 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:49.561+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.562965+0000) 2020-02-05T22:56:49.564 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:49.562+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.563314+0000) 2020-02-05T22:56:49.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:49.562+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.563865+0000) 2020-02-05T22:56:49.565 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:49.563+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.563835+0000) 2020-02-05T22:56:49.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:49.567+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:49.568230+0000) 2020-02-05T22:56:49.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:49.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:50.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:50.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:50.120600+0000) 2020-02-05T22:56:50.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:50.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:50.357780+0000) 2020-02-05T22:56:50.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:50.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:50.372529+0000) 2020-02-05T22:56:52.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:52.372+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:52.372497+0000) 2020-02-05T22:56:52.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:52.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:52.726365+0000) 2020-02-05T22:56:52.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:52.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:52.763539+0000) 2020-02-05T22:56:53.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:53.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:53.885989+0000) 2020-02-05T22:56:53.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:53.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:53.885989+0000) 2020-02-05T22:56:54.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:54.579+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.580388+0000) 2020-02-05T22:56:54.582 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:54.580+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.581209+0000) 2020-02-05T22:56:54.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:54.580+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.581959+0000) 2020-02-05T22:56:54.583 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:54.581+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.582081+0000) 2020-02-05T22:56:54.583 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:54.582+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.583053+0000) 2020-02-05T22:56:54.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:54.583+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.583662+0000) 2020-02-05T22:56:54.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:54.583+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.583952+0000) 2020-02-05T22:56:54.585 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:54.583+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.584384+0000) 2020-02-05T22:56:54.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:54.584+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.585031+0000) 2020-02-05T22:56:54.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:54.585+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.586375+0000) 2020-02-05T22:56:54.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:54.586+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.586610+0000) 2020-02-05T22:56:54.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:54.586+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.586593+0000) 2020-02-05T22:56:54.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:54.586+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.587317+0000) 2020-02-05T22:56:54.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:54.594+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:56:54.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:54.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:54.595442+0000) 2020-02-05T22:56:55.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:55.985+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:55.986339+0000) 2020-02-05T22:56:56.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:56.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:56.264032+0000) 2020-02-05T22:56:57.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:57.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:57.126557+0000) 2020-02-05T22:56:57.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:57.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:57.126677+0000) 2020-02-05T22:56:57.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:57.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:57.126927+0000) 2020-02-05T22:56:58.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:58.447+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:58.448325+0000) 2020-02-05T22:56:59.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:56:59.599+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.600551+0000) 2020-02-05T22:56:59.602 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:56:59.600+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.601987+0000) 2020-02-05T22:56:59.603 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:56:59.602+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.602579+0000) 2020-02-05T22:56:59.604 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:56:59.601+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.602724+0000) 2020-02-05T22:56:59.604 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:56:59.602+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.603234+0000) 2020-02-05T22:56:59.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:56:59.603+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.603625+0000) 2020-02-05T22:56:59.605 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:56:59.603+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.604197+0000) 2020-02-05T22:56:59.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:59.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.604699+0000) 2020-02-05T22:56:59.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:56:59.604+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.605305+0000) 2020-02-05T22:56:59.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:56:59.604+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.605743+0000) 2020-02-05T22:56:59.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:56:59.605+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.605797+0000) 2020-02-05T22:56:59.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:59.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.606422+0000) 2020-02-05T22:56:59.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:56:59.606+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.606798+0000) 2020-02-05T22:56:59.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:56:59.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:56:59.611008+0000) 2020-02-05T22:56:59.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:56:59.614+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:00.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:00.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:00.120833+0000) 2020-02-05T22:57:00.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:00.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:00.358058+0000) 2020-02-05T22:57:00.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:00.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:00.372755+0000) 2020-02-05T22:57:02.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:02.372+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:02.372741+0000) 2020-02-05T22:57:02.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:02.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:02.726590+0000) 2020-02-05T22:57:02.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:02.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:02.763772+0000) 2020-02-05T22:57:03.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:03.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:03.886326+0000) 2020-02-05T22:57:03.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:03.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:03.886326+0000) 2020-02-05T22:57:04.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:04.621+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.621975+0000) 2020-02-05T22:57:04.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:04.621+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.622326+0000) 2020-02-05T22:57:04.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:04.622+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.623073+0000) 2020-02-05T22:57:04.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:04.624+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.624479+0000) 2020-02-05T22:57:04.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:04.625+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.625800+0000) 2020-02-05T22:57:04.628 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:04.620+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.621560+0000) 2020-02-05T22:57:04.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:04.622+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.623496+0000) 2020-02-05T22:57:04.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:04.624+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.625284+0000) 2020-02-05T22:57:04.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:04.624+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.625514+0000) 2020-02-05T22:57:04.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:04.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.626358+0000) 2020-02-05T22:57:04.629 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:04.625+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.626702+0000) 2020-02-05T22:57:04.630 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:04.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.627169+0000) 2020-02-05T22:57:04.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:04.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.627052+0000) 2020-02-05T22:57:04.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:04.629+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:04.630229+0000) 2020-02-05T22:57:04.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:04.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130254 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:05.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:05.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:05.986641+0000) 2020-02-05T22:57:06.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:06.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:06.264292+0000) 2020-02-05T22:57:07.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:07.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:07.126782+0000) 2020-02-05T22:57:07.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:07.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:07.126961+0000) 2020-02-05T22:57:07.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:07.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:07.127136+0000) 2020-02-05T22:57:08.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:08.456+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:08.456656+0000) 2020-02-05T22:57:09.641 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:09.639+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.640197+0000) 2020-02-05T22:57:09.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:09.641+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.641504+0000) 2020-02-05T22:57:09.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:09.641+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.641734+0000) 2020-02-05T22:57:09.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:09.643+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.643523+0000) 2020-02-05T22:57:09.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:09.642+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.643737+0000) 2020-02-05T22:57:09.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:09.643+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.644575+0000) 2020-02-05T22:57:09.645 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:09.644+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.644908+0000) 2020-02-05T22:57:09.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:09.644+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.645430+0000) 2020-02-05T22:57:09.646 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:09.645+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.646016+0000) 2020-02-05T22:57:09.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:09.646+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.646585+0000) 2020-02-05T22:57:09.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:09.645+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.646608+0000) 2020-02-05T22:57:09.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:09.645+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.646822+0000) 2020-02-05T22:57:09.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:09.649+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.650506+0000) 2020-02-05T22:57:09.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:09.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:09.652057+0000) 2020-02-05T22:57:09.657 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:09.655+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:10.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:10.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:10.121048+0000) 2020-02-05T22:57:10.289 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:57:10.295 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:57:10.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:10.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:10.358214+0000) 2020-02-05T22:57:10.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:10.371+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:10.372954+0000) 2020-02-05T22:57:12.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:12.372+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:12.373096+0000) 2020-02-05T22:57:12.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:12.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:12.726825+0000) 2020-02-05T22:57:12.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:12.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:12.764002+0000) 2020-02-05T22:57:13.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:13.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:13.886600+0000) 2020-02-05T22:57:13.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:13.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:13.886601+0000) 2020-02-05T22:57:14.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:14.660+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.661362+0000) 2020-02-05T22:57:14.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:14.660+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.661819+0000) 2020-02-05T22:57:14.664 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:14.662+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.663334+0000) 2020-02-05T22:57:14.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:14.662+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.663143+0000) 2020-02-05T22:57:14.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:14.662+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.663566+0000) 2020-02-05T22:57:14.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:14.663+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.664134+0000) 2020-02-05T22:57:14.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:14.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.665398+0000) 2020-02-05T22:57:14.667 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:14.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.664403+0000) 2020-02-05T22:57:14.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:14.663+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.664994+0000) 2020-02-05T22:57:14.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:14.665+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.666173+0000) 2020-02-05T22:57:14.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:14.665+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.666260+0000) 2020-02-05T22:57:14.690 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:14.666+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.666967+0000) 2020-02-05T22:57:14.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:14.666+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.667071+0000) 2020-02-05T22:57:14.690 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:14.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:14.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:14.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:14.672905+0000) 2020-02-05T22:57:15.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:15.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:15.986883+0000) 2020-02-05T22:57:16.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:16.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:16.264562+0000) 2020-02-05T22:57:17.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:17.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:17.127030+0000) 2020-02-05T22:57:17.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:17.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:17.127206+0000) 2020-02-05T22:57:17.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:17.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:17.127341+0000) 2020-02-05T22:57:18.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:18.464+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:18.464810+0000) 2020-02-05T22:57:19.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:19.680+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.681796+0000) 2020-02-05T22:57:19.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:19.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.682197+0000) 2020-02-05T22:57:19.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:19.682+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.683979+0000) 2020-02-05T22:57:19.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:19.683+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.684493+0000) 2020-02-05T22:57:19.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:19.683+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.685038+0000) 2020-02-05T22:57:19.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:19.684+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.685249+0000) 2020-02-05T22:57:19.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:19.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.685619+0000) 2020-02-05T22:57:19.687 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:19.685+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.686035+0000) 2020-02-05T22:57:19.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:19.685+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.686576+0000) 2020-02-05T22:57:19.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:19.686+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.687279+0000) 2020-02-05T22:57:19.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:19.687+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.687546+0000) 2020-02-05T22:57:19.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:19.687+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.688050+0000) 2020-02-05T22:57:19.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:19.688+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.688827+0000) 2020-02-05T22:57:19.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:19.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:19.691386+0000) 2020-02-05T22:57:19.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:19.696+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:20.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:20.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:20.121300+0000) 2020-02-05T22:57:20.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:20.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:20.358458+0000) 2020-02-05T22:57:20.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:20.372+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:20.373153+0000) 2020-02-05T22:57:22.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:22.373+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:22.373371+0000) 2020-02-05T22:57:22.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:22.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:22.727111+0000) 2020-02-05T22:57:22.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:22.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:22.764267+0000) 2020-02-05T22:57:23.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:23.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:23.886896+0000) 2020-02-05T22:57:23.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:23.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:23.886892+0000) 2020-02-05T22:57:24.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:24.700+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.701758+0000) 2020-02-05T22:57:24.703 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:24.701+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.702726+0000) 2020-02-05T22:57:24.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:24.703+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.704054+0000) 2020-02-05T22:57:24.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:24.704+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.705433+0000) 2020-02-05T22:57:24.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:24.704+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.705338+0000) 2020-02-05T22:57:24.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:24.705+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.705579+0000) 2020-02-05T22:57:24.707 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:24.705+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.706393+0000) 2020-02-05T22:57:24.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:24.706+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.706795+0000) 2020-02-05T22:57:24.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:24.706+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.707340+0000) 2020-02-05T22:57:24.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:24.707+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.707562+0000) 2020-02-05T22:57:24.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:24.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.707944+0000) 2020-02-05T22:57:24.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:24.707+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.707986+0000) 2020-02-05T22:57:24.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:24.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.711565+0000) 2020-02-05T22:57:24.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:24.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:24.711769+0000) 2020-02-05T22:57:24.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:24.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:25.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:25.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:25.987143+0000) 2020-02-05T22:57:26.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:26.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:26.264819+0000) 2020-02-05T22:57:27.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:27.125+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:27.127358+0000) 2020-02-05T22:57:27.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:27.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:27.127488+0000) 2020-02-05T22:57:27.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:27.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:27.127642+0000) 2020-02-05T22:57:28.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:28.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:28.472878+0000) 2020-02-05T22:57:29.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:29.722+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.723009+0000) 2020-02-05T22:57:29.724 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:29.723+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.723877+0000) 2020-02-05T22:57:29.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:29.724+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.724443+0000) 2020-02-05T22:57:29.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:29.723+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.725184+0000) 2020-02-05T22:57:29.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:29.724+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.726095+0000) 2020-02-05T22:57:29.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:29.726+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.727672+0000) 2020-02-05T22:57:29.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:29.727+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.728512+0000) 2020-02-05T22:57:29.732 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:29.727+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.728825+0000) 2020-02-05T22:57:29.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:29.727+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.729101+0000) 2020-02-05T22:57:29.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:29.726+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.726131+0000) 2020-02-05T22:57:29.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:29.726+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.726562+0000) 2020-02-05T22:57:29.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:29.728+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.729025+0000) 2020-02-05T22:57:29.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:29.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.733555+0000) 2020-02-05T22:57:29.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:29.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:29.733810+0000) 2020-02-05T22:57:29.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:29.736+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130462 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:30.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:30.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:30.121520+0000) 2020-02-05T22:57:30.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:30.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:30.358713+0000) 2020-02-05T22:57:30.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:30.373+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:30.373362+0000) 2020-02-05T22:57:32.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:32.372+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:32.373664+0000) 2020-02-05T22:57:32.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:32.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:32.727368+0000) 2020-02-05T22:57:32.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:32.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:32.764523+0000) 2020-02-05T22:57:33.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:33.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:33.887166+0000) 2020-02-05T22:57:33.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:33.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:33.887166+0000) 2020-02-05T22:57:34.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:34.741+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.742477+0000) 2020-02-05T22:57:34.745 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:34.743+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.744654+0000) 2020-02-05T22:57:34.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:34.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.744885+0000) 2020-02-05T22:57:34.746 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:34.744+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.745264+0000) 2020-02-05T22:57:34.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:34.744+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.745907+0000) 2020-02-05T22:57:34.748 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:34.746+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.747224+0000) 2020-02-05T22:57:34.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:34.746+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.747703+0000) 2020-02-05T22:57:34.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:34.745+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.746572+0000) 2020-02-05T22:57:34.750 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:34.748+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.749448+0000) 2020-02-05T22:57:34.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:34.748+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.749971+0000) 2020-02-05T22:57:34.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:34.749+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.750274+0000) 2020-02-05T22:57:34.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:34.748+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.750280+0000) 2020-02-05T22:57:34.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:34.751+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.752142+0000) 2020-02-05T22:57:34.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:34.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:34.753217+0000) 2020-02-05T22:57:34.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:34.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:35.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:35.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:35.987419+0000) 2020-02-05T22:57:36.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:36.263+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:36.265054+0000) 2020-02-05T22:57:37.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:37.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:37.127618+0000) 2020-02-05T22:57:37.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:37.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:37.127778+0000) 2020-02-05T22:57:37.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:37.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:37.127891+0000) 2020-02-05T22:57:38.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:38.480+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:38.481825+0000) 2020-02-05T22:57:39.773 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:39.763+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.764874+0000) 2020-02-05T22:57:39.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:39.764+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.765683+0000) 2020-02-05T22:57:39.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:39.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.766324+0000) 2020-02-05T22:57:39.777 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:39.765+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.767201+0000) 2020-02-05T22:57:39.777 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:39.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.768345+0000) 2020-02-05T22:57:39.777 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:39.766+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.767774+0000) 2020-02-05T22:57:39.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:39.767+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.769279+0000) 2020-02-05T22:57:39.778 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:39.768+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.769778+0000) 2020-02-05T22:57:39.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:39.767+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.768813+0000) 2020-02-05T22:57:39.780 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:39.769+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.770694+0000) 2020-02-05T22:57:39.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:39.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.770514+0000) 2020-02-05T22:57:39.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:39.770+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.771148+0000) 2020-02-05T22:57:39.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:39.770+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.771451+0000) 2020-02-05T22:57:39.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:39.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:39.776863+0000) 2020-02-05T22:57:39.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:39.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:40.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:40.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:40.121762+0000) 2020-02-05T22:57:40.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:40.357+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:40.358924+0000) 2020-02-05T22:57:40.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:40.372+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:40.373578+0000) 2020-02-05T22:57:40.428 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:57:40.434 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:57:42.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:42.372+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:42.373856+0000) 2020-02-05T22:57:42.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:42.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:42.727577+0000) 2020-02-05T22:57:42.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:42.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:42.764753+0000) 2020-02-05T22:57:43.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:43.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:43.887443+0000) 2020-02-05T22:57:43.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:43.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:43.887443+0000) 2020-02-05T22:57:44.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:44.784+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.785492+0000) 2020-02-05T22:57:44.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:44.785+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.786339+0000) 2020-02-05T22:57:44.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:44.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.786613+0000) 2020-02-05T22:57:44.789 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:44.786+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.787572+0000) 2020-02-05T22:57:44.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:44.787+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.788527+0000) 2020-02-05T22:57:44.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:44.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.788988+0000) 2020-02-05T22:57:44.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:44.787+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.789130+0000) 2020-02-05T22:57:44.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:44.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.789172+0000) 2020-02-05T22:57:44.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:44.787+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.789399+0000) 2020-02-05T22:57:44.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:44.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.790388+0000) 2020-02-05T22:57:44.792 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:44.789+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.790845+0000) 2020-02-05T22:57:44.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:44.788+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.789960+0000) 2020-02-05T22:57:44.792 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:44.788+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.790230+0000) 2020-02-05T22:57:44.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:44.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:44.794304+0000) 2020-02-05T22:57:44.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:44.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:45.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:45.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:45.987727+0000) 2020-02-05T22:57:46.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:46.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:46.265358+0000) 2020-02-05T22:57:47.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:47.127+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:47.127854+0000) 2020-02-05T22:57:47.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:47.127+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:47.128046+0000) 2020-02-05T22:57:47.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:47.127+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:47.128138+0000) 2020-02-05T22:57:48.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:48.489+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:48.490243+0000) 2020-02-05T22:57:49.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:49.805+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.806503+0000) 2020-02-05T22:57:49.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:49.806+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.807260+0000) 2020-02-05T22:57:49.809 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:49.806+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.808006+0000) 2020-02-05T22:57:49.809 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:49.807+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.808422+0000) 2020-02-05T22:57:49.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:49.808+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.809329+0000) 2020-02-05T22:57:49.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:49.807+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.809138+0000) 2020-02-05T22:57:49.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:49.808+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.810008+0000) 2020-02-05T22:57:49.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:49.808+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.810308+0000) 2020-02-05T22:57:49.811 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:49.809+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.810521+0000) 2020-02-05T22:57:49.812 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:49.809+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.811207+0000) 2020-02-05T22:57:49.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:49.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.811727+0000) 2020-02-05T22:57:49.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:49.810+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.811752+0000) 2020-02-05T22:57:49.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:49.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.816588+0000) 2020-02-05T22:57:49.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:49.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:49.816916+0000) 2020-02-05T22:57:49.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:49.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:50.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:50.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:50.122000+0000) 2020-02-05T22:57:50.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:50.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:50.359169+0000) 2020-02-05T22:57:50.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:50.373+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:50.373780+0000) 2020-02-05T22:57:52.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:52.373+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:52.374109+0000) 2020-02-05T22:57:52.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:52.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:52.727804+0000) 2020-02-05T22:57:52.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:52.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:52.765002+0000) 2020-02-05T22:57:53.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:53.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:53.887716+0000) 2020-02-05T22:57:53.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:53.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:53.887716+0000) 2020-02-05T22:57:54.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:54.825+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.827011+0000) 2020-02-05T22:57:54.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:54.826+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.827356+0000) 2020-02-05T22:57:54.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:54.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.827759+0000) 2020-02-05T22:57:54.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:54.828+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.829170+0000) 2020-02-05T22:57:54.831 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:54.827+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.828695+0000) 2020-02-05T22:57:54.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:54.828+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.829767+0000) 2020-02-05T22:57:54.832 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:54.828+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.830139+0000) 2020-02-05T22:57:54.832 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:54.829+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.830620+0000) 2020-02-05T22:57:54.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:54.829+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.830831+0000) 2020-02-05T22:57:54.833 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:54.829+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.831344+0000) 2020-02-05T22:57:54.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:54.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.830708+0000) 2020-02-05T22:57:54.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:54.830+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.831520+0000) 2020-02-05T22:57:54.834 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:54.830+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.831859+0000) 2020-02-05T22:57:54.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:54.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:54.836313+0000) 2020-02-05T22:57:54.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:54.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:57:55.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:55.986+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:55.987935+0000) 2020-02-05T22:57:56.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:56.264+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:56.265614+0000) 2020-02-05T22:57:57.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:57.126+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:57.128126+0000) 2020-02-05T22:57:57.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:57.126+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:57.128300+0000) 2020-02-05T22:57:57.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:57.126+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:57.128364+0000) 2020-02-05T22:57:58.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:58.501+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:58.502234+0000) 2020-02-05T22:57:59.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:57:59.843+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.844797+0000) 2020-02-05T22:57:59.846 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:57:59.844+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.845256+0000) 2020-02-05T22:57:59.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:57:59.846+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.847165+0000) 2020-02-05T22:57:59.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:59.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.847782+0000) 2020-02-05T22:57:59.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:57:59.846+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.848240+0000) 2020-02-05T22:57:59.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:59.847+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.848361+0000) 2020-02-05T22:57:59.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:57:59.847+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.848683+0000) 2020-02-05T22:57:59.851 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:57:59.847+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.848611+0000) 2020-02-05T22:57:59.851 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:57:59.847+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.848837+0000) 2020-02-05T22:57:59.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:57:59.847+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.849019+0000) 2020-02-05T22:57:59.851 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:57:59.848+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.849489+0000) 2020-02-05T22:57:59.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:57:59.848+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.849654+0000) 2020-02-05T22:57:59.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:57:59.848+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.849506+0000) 2020-02-05T22:57:59.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:57:59.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:57:59.853362+0000) 2020-02-05T22:57:59.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:57:59.857+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:00.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:00.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:00.122280+0000) 2020-02-05T22:58:00.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:00.358+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:00.359401+0000) 2020-02-05T22:58:00.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:00.373+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:00.373987+0000) 2020-02-05T22:58:02.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:02.373+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:02.374380+0000) 2020-02-05T22:58:02.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:02.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:02.728056+0000) 2020-02-05T22:58:02.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:02.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:02.765275+0000) 2020-02-05T22:58:03.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:03.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:03.888009+0000) 2020-02-05T22:58:03.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:03.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:03.888010+0000) 2020-02-05T22:58:04.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:04.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.863690+0000) 2020-02-05T22:58:04.865 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:04.863+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.864543+0000) 2020-02-05T22:58:04.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:04.863+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.864719+0000) 2020-02-05T22:58:04.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:04.864+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.865165+0000) 2020-02-05T22:58:04.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:04.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.865632+0000) 2020-02-05T22:58:04.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:04.865+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.866379+0000) 2020-02-05T22:58:04.867 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:04.864+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.866023+0000) 2020-02-05T22:58:04.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:04.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.867234+0000) 2020-02-05T22:58:04.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:04.866+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.867742+0000) 2020-02-05T22:58:04.869 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:04.866+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.868353+0000) 2020-02-05T22:58:04.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:04.867+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.869352+0000) 2020-02-05T22:58:04.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:04.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.869155+0000) 2020-02-05T22:58:04.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:04.868+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.869640+0000) 2020-02-05T22:58:04.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:04.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:04.872773+0000) 2020-02-05T22:58:04.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:04.877+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:05.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:05.987+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:05.988208+0000) 2020-02-05T22:58:06.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:06.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:06.265849+0000) 2020-02-05T22:58:07.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:07.127+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:07.128371+0000) 2020-02-05T22:58:07.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:07.127+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:07.128519+0000) 2020-02-05T22:58:07.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:07.127+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:07.128677+0000) 2020-02-05T22:58:08.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:08.508+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:08.509318+0000) 2020-02-05T22:58:09.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:09.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.884627+0000) 2020-02-05T22:58:09.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:09.884+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.885703+0000) 2020-02-05T22:58:09.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:09.885+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.886267+0000) 2020-02-05T22:58:09.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:09.885+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.886671+0000) 2020-02-05T22:58:09.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:09.886+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.887196+0000) 2020-02-05T22:58:09.888 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:09.886+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.887494+0000) 2020-02-05T22:58:09.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:09.887+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.888245+0000) 2020-02-05T22:58:09.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:09.887+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.888559+0000) 2020-02-05T22:58:09.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:09.887+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.888660+0000) 2020-02-05T22:58:09.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:09.888+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.890026+0000) 2020-02-05T22:58:09.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:09.888+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.889133+0000) 2020-02-05T22:58:09.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:09.888+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.890229+0000) 2020-02-05T22:58:09.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:09.889+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.891100+0000) 2020-02-05T22:58:09.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:09.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:09.896790+0000) 2020-02-05T22:58:09.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:09.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:10.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:10.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:10.122508+0000) 2020-02-05T22:58:10.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:10.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:10.359645+0000) 2020-02-05T22:58:10.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:10.373+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:10.374208+0000) 2020-02-05T22:58:10.567 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:58:10.573 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:58:12.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:12.373+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:12.374609+0000) 2020-02-05T22:58:12.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:12.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:12.728364+0000) 2020-02-05T22:58:12.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:12.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:12.765513+0000) 2020-02-05T22:58:13.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:13.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:13.888336+0000) 2020-02-05T22:58:13.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:13.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:13.888336+0000) 2020-02-05T22:58:14.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:14.901+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.902547+0000) 2020-02-05T22:58:14.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:14.902+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.903744+0000) 2020-02-05T22:58:14.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:14.903+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.904167+0000) 2020-02-05T22:58:14.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:14.904+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.905345+0000) 2020-02-05T22:58:14.906 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:14.904+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.905404+0000) 2020-02-05T22:58:14.907 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:14.904+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.906259+0000) 2020-02-05T22:58:14.909 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:14.906+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.907586+0000) 2020-02-05T22:58:14.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:14.905+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.906681+0000) 2020-02-05T22:58:14.909 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:14.906+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.907419+0000) 2020-02-05T22:58:14.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:14.907+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.908870+0000) 2020-02-05T22:58:14.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:14.908+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.909815+0000) 2020-02-05T22:58:14.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:14.908+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.909952+0000) 2020-02-05T22:58:14.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:14.908+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.909989+0000) 2020-02-05T22:58:14.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:14.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:14.914860+0000) 2020-02-05T22:58:14.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:14.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:15.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:15.987+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:15.988488+0000) 2020-02-05T22:58:16.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:16.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:16.266123+0000) 2020-02-05T22:58:17.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:17.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:17.128657+0000) 2020-02-05T22:58:17.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:17.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:17.128753+0000) 2020-02-05T22:58:17.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:17.128+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:17.128927+0000) 2020-02-05T22:58:18.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:18.516+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:18.517538+0000) 2020-02-05T22:58:19.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:19.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.922984+0000) 2020-02-05T22:58:19.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:19.923+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.924319+0000) 2020-02-05T22:58:19.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:19.923+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.924698+0000) 2020-02-05T22:58:19.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:19.924+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.925155+0000) 2020-02-05T22:58:19.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:19.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.925833+0000) 2020-02-05T22:58:19.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:19.925+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.925849+0000) 2020-02-05T22:58:19.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:19.925+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.926297+0000) 2020-02-05T22:58:19.928 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:19.926+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.927124+0000) 2020-02-05T22:58:19.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:19.926+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.927672+0000) 2020-02-05T22:58:19.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:19.927+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.928098+0000) 2020-02-05T22:58:19.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:19.928+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.929231+0000) 2020-02-05T22:58:19.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:19.929+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.929595+0000) 2020-02-05T22:58:19.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:19.929+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.930058+0000) 2020-02-05T22:58:19.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:19.933+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:19.934460+0000) 2020-02-05T22:58:19.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:19.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:20.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:20.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:20.122720+0000) 2020-02-05T22:58:20.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:20.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:20.359938+0000) 2020-02-05T22:58:20.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:20.374+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:20.374412+0000) 2020-02-05T22:58:22.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:22.374+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:22.374870+0000) 2020-02-05T22:58:22.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:22.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:22.728600+0000) 2020-02-05T22:58:22.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:22.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:22.765782+0000) 2020-02-05T22:58:23.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:23.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:23.888621+0000) 2020-02-05T22:58:23.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:23.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:23.888625+0000) 2020-02-05T22:58:24.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:24.945+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.945352+0000) 2020-02-05T22:58:24.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:24.945+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.945903+0000) 2020-02-05T22:58:24.948 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:24.947+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.947298+0000) 2020-02-05T22:58:24.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:24.948+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.948319+0000) 2020-02-05T22:58:24.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:24.947+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.948284+0000) 2020-02-05T22:58:24.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:24.948+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.948541+0000) 2020-02-05T22:58:24.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:24.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.949565+0000) 2020-02-05T22:58:24.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:24.949+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.949812+0000) 2020-02-05T22:58:24.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:24.950+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.950279+0000) 2020-02-05T22:58:24.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:24.950+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.950585+0000) 2020-02-05T22:58:24.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:24.950+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.951074+0000) 2020-02-05T22:58:24.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:24.951+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.951419+0000) 2020-02-05T22:58:24.953 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:24.951+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.952036+0000) 2020-02-05T22:58:24.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:24.954+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:24.955044+0000) 2020-02-05T22:58:24.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:24.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:25.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:25.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:25.988766+0000) 2020-02-05T22:58:26.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:26.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:26.266359+0000) 2020-02-05T22:58:27.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:27.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:27.128930+0000) 2020-02-05T22:58:27.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:27.128+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:27.129159+0000) 2020-02-05T22:58:27.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:27.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:27.129016+0000) 2020-02-05T22:58:28.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:28.525+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:28.526045+0000) 2020-02-05T22:58:29.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:29.964+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.965979+0000) 2020-02-05T22:58:29.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:29.965+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.966371+0000) 2020-02-05T22:58:29.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:29.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.967135+0000) 2020-02-05T22:58:29.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:29.967+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.968029+0000) 2020-02-05T22:58:29.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:29.967+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.968225+0000) 2020-02-05T22:58:29.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:29.968+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.968641+0000) 2020-02-05T22:58:29.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:29.968+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.969714+0000) 2020-02-05T22:58:29.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:29.968+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.969973+0000) 2020-02-05T22:58:29.971 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:29.968+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.969087+0000) 2020-02-05T22:58:29.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:29.970+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.970627+0000) 2020-02-05T22:58:29.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:29.970+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.970911+0000) 2020-02-05T22:58:29.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:29.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.971574+0000) 2020-02-05T22:58:29.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:29.971+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.972080+0000) 2020-02-05T22:58:29.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:29.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:29.977721+0000) 2020-02-05T22:58:29.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:29.981+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:30.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:30.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:30.122917+0000) 2020-02-05T22:58:30.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:30.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:30.360154+0000) 2020-02-05T22:58:30.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:30.373+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:30.374642+0000) 2020-02-05T22:58:32.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:32.374+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:32.375151+0000) 2020-02-05T22:58:32.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:32.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:32.728838+0000) 2020-02-05T22:58:32.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:32.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:32.766014+0000) 2020-02-05T22:58:33.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:33.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:33.888894+0000) 2020-02-05T22:58:33.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:33.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:33.888894+0000) 2020-02-05T22:58:34.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:34.987+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.987924+0000) 2020-02-05T22:58:34.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:34.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.988966+0000) 2020-02-05T22:58:34.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:34.988+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.989394+0000) 2020-02-05T22:58:34.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:34.988+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.989747+0000) 2020-02-05T22:58:34.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:34.989+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.990262+0000) 2020-02-05T22:58:34.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:34.989+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.990624+0000) 2020-02-05T22:58:34.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:34.990+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.991431+0000) 2020-02-05T22:58:34.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:34.992+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.992727+0000) 2020-02-05T22:58:34.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:34.992+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.993321+0000) 2020-02-05T22:58:34.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:34.991+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.992388+0000) 2020-02-05T22:58:34.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:34.992+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.993557+0000) 2020-02-05T22:58:34.996 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:34.993+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.993581+0000) 2020-02-05T22:58:34.996 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:34.993+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.994090+0000) 2020-02-05T22:58:34.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:34.997+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:34.998088+0000) 2020-02-05T22:58:35.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:35.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 130995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:35.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:35.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:35.989012+0000) 2020-02-05T22:58:36.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:36.265+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:36.266594+0000) 2020-02-05T22:58:37.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:37.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:37.129191+0000) 2020-02-05T22:58:37.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:37.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:37.129248+0000) 2020-02-05T22:58:37.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:37.129+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:37.129395+0000) 2020-02-05T22:58:38.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:38.533+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:38.534128+0000) 2020-02-05T22:58:40.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:40.006+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.007794+0000) 2020-02-05T22:58:40.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:40.008+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.009282+0000) 2020-02-05T22:58:40.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:40.008+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.009698+0000) 2020-02-05T22:58:40.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:40.009+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.010160+0000) 2020-02-05T22:58:40.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:40.009+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.010718+0000) 2020-02-05T22:58:40.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:40.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.010502+0000) 2020-02-05T22:58:40.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:40.010+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.011319+0000) 2020-02-05T22:58:40.012 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:40.011+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.011770+0000) 2020-02-05T22:58:40.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:40.012+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.012511+0000) 2020-02-05T22:58:40.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:40.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.013115+0000) 2020-02-05T22:58:40.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:40.012+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.013414+0000) 2020-02-05T22:58:40.015 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:40.012+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.013212+0000) 2020-02-05T22:58:40.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:40.013+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.013585+0000) 2020-02-05T22:58:40.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:40.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.016964+0000) 2020-02-05T22:58:40.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:40.024+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:40.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:40.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.123132+0000) 2020-02-05T22:58:40.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:40.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.360395+0000) 2020-02-05T22:58:40.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:40.374+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:40.374852+0000) 2020-02-05T22:58:40.705 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:58:40.711 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:58:42.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:42.375+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:42.375403+0000) 2020-02-05T22:58:42.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:42.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:42.729114+0000) 2020-02-05T22:58:42.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:42.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:42.766280+0000) 2020-02-05T22:58:43.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:43.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:43.889199+0000) 2020-02-05T22:58:43.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:43.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:43.889199+0000) 2020-02-05T22:58:45.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:45.029+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.030332+0000) 2020-02-05T22:58:45.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:45.029+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.030672+0000) 2020-02-05T22:58:45.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:45.030+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.031120+0000) 2020-02-05T22:58:45.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:45.030+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.031463+0000) 2020-02-05T22:58:45.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:45.031+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.032510+0000) 2020-02-05T22:58:45.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:45.032+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.032836+0000) 2020-02-05T22:58:45.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:45.032+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.032984+0000) 2020-02-05T22:58:45.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:45.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.033655+0000) 2020-02-05T22:58:45.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:45.032+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.033720+0000) 2020-02-05T22:58:45.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:45.033+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.033861+0000) 2020-02-05T22:58:45.036 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:45.034+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.034739+0000) 2020-02-05T22:58:45.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:45.033+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.034409+0000) 2020-02-05T22:58:45.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:45.035+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.035794+0000) 2020-02-05T22:58:45.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:45.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.037159+0000) 2020-02-05T22:58:45.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:45.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:45.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:45.989+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:45.989254+0000) 2020-02-05T22:58:46.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:46.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:46.266827+0000) 2020-02-05T22:58:47.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:47.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:47.129483+0000) 2020-02-05T22:58:47.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:47.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:47.129483+0000) 2020-02-05T22:58:47.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:47.129+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:47.129702+0000) 2020-02-05T22:58:48.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:48.541+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:48.542124+0000) 2020-02-05T22:58:50.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:50.051+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.052067+0000) 2020-02-05T22:58:50.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:50.051+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.052484+0000) 2020-02-05T22:58:50.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:50.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.053251+0000) 2020-02-05T22:58:50.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:50.052+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.053460+0000) 2020-02-05T22:58:50.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:50.052+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.053879+0000) 2020-02-05T22:58:50.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:50.053+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.054281+0000) 2020-02-05T22:58:50.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:50.053+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.054558+0000) 2020-02-05T22:58:50.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:50.054+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.055400+0000) 2020-02-05T22:58:50.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:50.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.056220+0000) 2020-02-05T22:58:50.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:50.056+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.056832+0000) 2020-02-05T22:58:50.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:50.056+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.056933+0000) 2020-02-05T22:58:50.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:50.056+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.057344+0000) 2020-02-05T22:58:50.059 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:50.057+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.057542+0000) 2020-02-05T22:58:50.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:50.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.063234+0000) 2020-02-05T22:58:50.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:50.066+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:50.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:50.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.123347+0000) 2020-02-05T22:58:50.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:50.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.360697+0000) 2020-02-05T22:58:50.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:50.374+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:50.375184+0000) 2020-02-05T22:58:52.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:52.375+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:52.375790+0000) 2020-02-05T22:58:52.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:52.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:52.729323+0000) 2020-02-05T22:58:52.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:52.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:52.766517+0000) 2020-02-05T22:58:53.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:53.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:53.889467+0000) 2020-02-05T22:58:53.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:53.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:53.889471+0000) 2020-02-05T22:58:55.074 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:58:55.072+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.073123+0000) 2020-02-05T22:58:55.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:55.073+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.074275+0000) 2020-02-05T22:58:55.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:58:55.073+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.074898+0000) 2020-02-05T22:58:55.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:58:55.074+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.075668+0000) 2020-02-05T22:58:55.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:55.074+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.075920+0000) 2020-02-05T22:58:55.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:58:55.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.077179+0000) 2020-02-05T22:58:55.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:55.076+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.077153+0000) 2020-02-05T22:58:55.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:55.075+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.076639+0000) 2020-02-05T22:58:55.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:55.076+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.077473+0000) 2020-02-05T22:58:55.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:55.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.078044+0000) 2020-02-05T22:58:55.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:55.076+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.077984+0000) 2020-02-05T22:58:55.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:58:55.077+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.078899+0000) 2020-02-05T22:58:55.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:58:55.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.078677+0000) 2020-02-05T22:58:55.080 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:58:55.078+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.079755+0000) 2020-02-05T22:58:55.089 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:58:55.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:58:55.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:58:55.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:55.989495+0000) 2020-02-05T22:58:56.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:56.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:56.267053+0000) 2020-02-05T22:58:57.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:58:57.128+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:57.129713+0000) 2020-02-05T22:58:57.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:58:57.128+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:57.129714+0000) 2020-02-05T22:58:57.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:58:57.128+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:57.129985+0000) 2020-02-05T22:58:58.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:58:58.549+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:58:58.550240+0000) 2020-02-05T22:59:00.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:00.093+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.094907+0000) 2020-02-05T22:59:00.096 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:00.094+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.095495+0000) 2020-02-05T22:59:00.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:00.095+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.096613+0000) 2020-02-05T22:59:00.097 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:00.095+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.096777+0000) 2020-02-05T22:59:00.098 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:00.096+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.097603+0000) 2020-02-05T22:59:00.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:00.096+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.098044+0000) 2020-02-05T22:59:00.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:00.097+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.098679+0000) 2020-02-05T22:59:00.099 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:00.097+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.098560+0000) 2020-02-05T22:59:00.100 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:00.098+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.099175+0000) 2020-02-05T22:59:00.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:00.098+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.099852+0000) 2020-02-05T22:59:00.101 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:00.098+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.100313+0000) 2020-02-05T22:59:00.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:00.098+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.100339+0000) 2020-02-05T22:59:00.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:00.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.102172+0000) 2020-02-05T22:59:00.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:00.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.102448+0000) 2020-02-05T22:59:00.112 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:00.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:00.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:00.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.123575+0000) 2020-02-05T22:59:00.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:00.359+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.360906+0000) 2020-02-05T22:59:00.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:00.375+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:00.375415+0000) 2020-02-05T22:59:02.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:02.375+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:02.376089+0000) 2020-02-05T22:59:02.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:02.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:02.729558+0000) 2020-02-05T22:59:02.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:02.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:02.766796+0000) 2020-02-05T22:59:03.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:03.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:03.889739+0000) 2020-02-05T22:59:03.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:03.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:03.889739+0000) 2020-02-05T22:59:05.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:05.116+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.117828+0000) 2020-02-05T22:59:05.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:05.116+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.118390+0000) 2020-02-05T22:59:05.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:05.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.119740+0000) 2020-02-05T22:59:05.120 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:05.118+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.119969+0000) 2020-02-05T22:59:05.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:05.119+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.120839+0000) 2020-02-05T22:59:05.122 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:05.119+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.121291+0000) 2020-02-05T22:59:05.122 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:05.120+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.121552+0000) 2020-02-05T22:59:05.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:05.120+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.122209+0000) 2020-02-05T22:59:05.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:05.121+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.122520+0000) 2020-02-05T22:59:05.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:05.121+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.122992+0000) 2020-02-05T22:59:05.124 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:05.122+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.123117+0000) 2020-02-05T22:59:05.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:05.121+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.123356+0000) 2020-02-05T22:59:05.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:05.122+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.123774+0000) 2020-02-05T22:59:05.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:05.124+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.125339+0000) 2020-02-05T22:59:05.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:05.132+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:05.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:05.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:05.989749+0000) 2020-02-05T22:59:06.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:06.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:06.267318+0000) 2020-02-05T22:59:07.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:07.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:07.129956+0000) 2020-02-05T22:59:07.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:07.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:07.129957+0000) 2020-02-05T22:59:07.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:07.129+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:07.130202+0000) 2020-02-05T22:59:08.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:08.558+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:08.558620+0000) 2020-02-05T22:59:10.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:10.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.123807+0000) 2020-02-05T22:59:10.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:10.138+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.139439+0000) 2020-02-05T22:59:10.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:10.138+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.139941+0000) 2020-02-05T22:59:10.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:10.139+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.140757+0000) 2020-02-05T22:59:10.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:10.140+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.141949+0000) 2020-02-05T22:59:10.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:10.141+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.142554+0000) 2020-02-05T22:59:10.144 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:10.142+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.143324+0000) 2020-02-05T22:59:10.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:10.142+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.143756+0000) 2020-02-05T22:59:10.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:10.143+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.144278+0000) 2020-02-05T22:59:10.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:10.142+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.143975+0000) 2020-02-05T22:59:10.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:10.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.144614+0000) 2020-02-05T22:59:10.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:10.143+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.144894+0000) 2020-02-05T22:59:10.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:10.143+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.145093+0000) 2020-02-05T22:59:10.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:10.143+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.145292+0000) 2020-02-05T22:59:10.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:10.149+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.150369+0000) 2020-02-05T22:59:10.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:10.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:10.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:10.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.361121+0000) 2020-02-05T22:59:10.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:10.375+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:10.375630+0000) 2020-02-05T22:59:10.843 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:59:10.850 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:59:12.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:12.375+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:12.376348+0000) 2020-02-05T22:59:12.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:12.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:12.729820+0000) 2020-02-05T22:59:12.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:12.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:12.767051+0000) 2020-02-05T22:59:13.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:13.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:13.890010+0000) 2020-02-05T22:59:13.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:13.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:13.890009+0000) 2020-02-05T22:59:15.161 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:15.159+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.159969+0000) 2020-02-05T22:59:15.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:15.161+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.161586+0000) 2020-02-05T22:59:15.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:15.161+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.162899+0000) 2020-02-05T22:59:15.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:15.163+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.163447+0000) 2020-02-05T22:59:15.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:15.162+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.164008+0000) 2020-02-05T22:59:15.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:15.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.164258+0000) 2020-02-05T22:59:15.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:15.164+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.164562+0000) 2020-02-05T22:59:15.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:15.164+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.164904+0000) 2020-02-05T22:59:15.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:15.165+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.165227+0000) 2020-02-05T22:59:15.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:15.164+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.165658+0000) 2020-02-05T22:59:15.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:15.164+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.165839+0000) 2020-02-05T22:59:15.167 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:15.165+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.167168+0000) 2020-02-05T22:59:15.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:15.166+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.167615+0000) 2020-02-05T22:59:15.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:15.172+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.173948+0000) 2020-02-05T22:59:15.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:15.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:15.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:15.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:15.989952+0000) 2020-02-05T22:59:16.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:16.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:16.267548+0000) 2020-02-05T22:59:17.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:17.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:17.130220+0000) 2020-02-05T22:59:17.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:17.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:17.130249+0000) 2020-02-05T22:59:17.132 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:17.129+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:17.130405+0000) 2020-02-05T22:59:18.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:18.567+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:18.567120+0000) 2020-02-05T22:59:20.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:20.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.124052+0000) 2020-02-05T22:59:20.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:20.182+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.183339+0000) 2020-02-05T22:59:20.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:20.183+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.184390+0000) 2020-02-05T22:59:20.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:20.182+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.184322+0000) 2020-02-05T22:59:20.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:20.183+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.184788+0000) 2020-02-05T22:59:20.186 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:20.184+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.185421+0000) 2020-02-05T22:59:20.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:20.185+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.186186+0000) 2020-02-05T22:59:20.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:20.185+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.186393+0000) 2020-02-05T22:59:20.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:20.185+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.187326+0000) 2020-02-05T22:59:20.189 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:20.186+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.187856+0000) 2020-02-05T22:59:20.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:20.186+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.187950+0000) 2020-02-05T22:59:20.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:20.186+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.188344+0000) 2020-02-05T22:59:20.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:20.187+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.188738+0000) 2020-02-05T22:59:20.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:20.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.189224+0000) 2020-02-05T22:59:20.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:20.191+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.192515+0000) 2020-02-05T22:59:20.199 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:20.196+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:20.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:20.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.361359+0000) 2020-02-05T22:59:20.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:20.374+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:20.375884+0000) 2020-02-05T22:59:22.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:22.375+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:22.376656+0000) 2020-02-05T22:59:22.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:22.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:22.730066+0000) 2020-02-05T22:59:22.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:22.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:22.767322+0000) 2020-02-05T22:59:23.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:23.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:23.890323+0000) 2020-02-05T22:59:23.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:23.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:23.890323+0000) 2020-02-05T22:59:25.204 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:25.202+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.203671+0000) 2020-02-05T22:59:25.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:25.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.204126+0000) 2020-02-05T22:59:25.205 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:25.203+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.204578+0000) 2020-02-05T22:59:25.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:25.203+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.204949+0000) 2020-02-05T22:59:25.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:25.204+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.205777+0000) 2020-02-05T22:59:25.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:25.204+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.205806+0000) 2020-02-05T22:59:25.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:25.205+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.206510+0000) 2020-02-05T22:59:25.210 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:25.207+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.208675+0000) 2020-02-05T22:59:25.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:25.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.207776+0000) 2020-02-05T22:59:25.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:25.207+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.208487+0000) 2020-02-05T22:59:25.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:25.207+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.209343+0000) 2020-02-05T22:59:25.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:25.208+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.209789+0000) 2020-02-05T22:59:25.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:25.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.214624+0000) 2020-02-05T22:59:25.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:25.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.214931+0000) 2020-02-05T22:59:25.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:25.218+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131404 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:25.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:25.988+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:25.990160+0000) 2020-02-05T22:59:26.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:26.266+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:26.267782+0000) 2020-02-05T22:59:27.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:27.129+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:27.130481+0000) 2020-02-05T22:59:27.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:27.129+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:27.130513+0000) 2020-02-05T22:59:27.132 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:27.129+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:27.130685+0000) 2020-02-05T22:59:28.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:28.574+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:28.575570+0000) 2020-02-05T22:59:30.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:30.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.124318+0000) 2020-02-05T22:59:30.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:30.225+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.226453+0000) 2020-02-05T22:59:30.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:30.225+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.227011+0000) 2020-02-05T22:59:30.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:30.226+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.227862+0000) 2020-02-05T22:59:30.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:30.227+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.228425+0000) 2020-02-05T22:59:30.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:30.228+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.229611+0000) 2020-02-05T22:59:30.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:30.228+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.229779+0000) 2020-02-05T22:59:30.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:30.229+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.230450+0000) 2020-02-05T22:59:30.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:30.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.230989+0000) 2020-02-05T22:59:30.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:30.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.231144+0000) 2020-02-05T22:59:30.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:30.230+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.231394+0000) 2020-02-05T22:59:30.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:30.230+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.231736+0000) 2020-02-05T22:59:30.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:30.230+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.232176+0000) 2020-02-05T22:59:30.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:30.231+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.232312+0000) 2020-02-05T22:59:30.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:30.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.237165+0000) 2020-02-05T22:59:30.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:30.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:30.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:30.360+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.361630+0000) 2020-02-05T22:59:30.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:30.375+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:30.376134+0000) 2020-02-05T22:59:32.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:32.376+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:32.376904+0000) 2020-02-05T22:59:32.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:32.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:32.730354+0000) 2020-02-05T22:59:32.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:32.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:32.767550+0000) 2020-02-05T22:59:33.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:33.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:33.890598+0000) 2020-02-05T22:59:33.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:33.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:33.890598+0000) 2020-02-05T22:59:35.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:35.246+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.247338+0000) 2020-02-05T22:59:35.249 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:35.247+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.248816+0000) 2020-02-05T22:59:35.250 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:35.248+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.249698+0000) 2020-02-05T22:59:35.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:35.248+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.249553+0000) 2020-02-05T22:59:35.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:35.248+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.250002+0000) 2020-02-05T22:59:35.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:35.248+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.250283+0000) 2020-02-05T22:59:35.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:35.249+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.250663+0000) 2020-02-05T22:59:35.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:35.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.251525+0000) 2020-02-05T22:59:35.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:35.250+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.252094+0000) 2020-02-05T22:59:35.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:35.251+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.252681+0000) 2020-02-05T22:59:35.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:35.251+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.253099+0000) 2020-02-05T22:59:35.254 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:35.252+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.253592+0000) 2020-02-05T22:59:35.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:35.252+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.253295+0000) 2020-02-05T22:59:35.262 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:35.259+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131486 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:35.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:35.254+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.255175+0000) 2020-02-05T22:59:35.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:35.989+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:35.990399+0000) 2020-02-05T22:59:36.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:36.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:36.268023+0000) 2020-02-05T22:59:37.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:37.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:37.130737+0000) 2020-02-05T22:59:37.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:37.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:37.130746+0000) 2020-02-05T22:59:37.132 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:37.130+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:37.130908+0000) 2020-02-05T22:59:38.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:38.582+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:38.583816+0000) 2020-02-05T22:59:40.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:40.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.124549+0000) 2020-02-05T22:59:40.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:40.264+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.265622+0000) 2020-02-05T22:59:40.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:40.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.267006+0000) 2020-02-05T22:59:40.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:40.266+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.267554+0000) 2020-02-05T22:59:40.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:40.267+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.268235+0000) 2020-02-05T22:59:40.270 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:40.268+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.269120+0000) 2020-02-05T22:59:40.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:40.269+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.270200+0000) 2020-02-05T22:59:40.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:40.268+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.269816+0000) 2020-02-05T22:59:40.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:40.270+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.271602+0000) 2020-02-05T22:59:40.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:40.269+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.271173+0000) 2020-02-05T22:59:40.273 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:40.271+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.272069+0000) 2020-02-05T22:59:40.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:40.271+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.272595+0000) 2020-02-05T22:59:40.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:40.271+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.273072+0000) 2020-02-05T22:59:40.274 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:40.271+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.273045+0000) 2020-02-05T22:59:40.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:40.274+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.275679+0000) 2020-02-05T22:59:40.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:40.279+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:40.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:40.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.361860+0000) 2020-02-05T22:59:40.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:40.375+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:40.376368+0000) 2020-02-05T22:59:40.977 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:59:40.984 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T22:59:42.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:42.376+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:42.377103+0000) 2020-02-05T22:59:42.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:42.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:42.730585+0000) 2020-02-05T22:59:42.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:42.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:42.767826+0000) 2020-02-05T22:59:43.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:43.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:43.890887+0000) 2020-02-05T22:59:43.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:43.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:43.890889+0000) 2020-02-05T22:59:45.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:45.286+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.287474+0000) 2020-02-05T22:59:45.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:45.287+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.287913+0000) 2020-02-05T22:59:45.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:45.287+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.288443+0000) 2020-02-05T22:59:45.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:45.287+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.288944+0000) 2020-02-05T22:59:45.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:45.288+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.289850+0000) 2020-02-05T22:59:45.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:45.289+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.289511+0000) 2020-02-05T22:59:45.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:45.289+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.290475+0000) 2020-02-05T22:59:45.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:45.289+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.290967+0000) 2020-02-05T22:59:45.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:45.290+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.291590+0000) 2020-02-05T22:59:45.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:45.291+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.292035+0000) 2020-02-05T22:59:45.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:45.291+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.292257+0000) 2020-02-05T22:59:45.308 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:45.292+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.292421+0000) 2020-02-05T22:59:45.308 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:45.292+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.293240+0000) 2020-02-05T22:59:45.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:45.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.298267+0000) 2020-02-05T22:59:45.316 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:45.301+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131569 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:45.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:45.990+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:45.990650+0000) 2020-02-05T22:59:46.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:46.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:46.268299+0000) 2020-02-05T22:59:47.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:47.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:47.131011+0000) 2020-02-05T22:59:47.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:47.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:47.130994+0000) 2020-02-05T22:59:47.132 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:47.130+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:47.131169+0000) 2020-02-05T22:59:48.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:48.591+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:48.592036+0000) 2020-02-05T22:59:50.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:50.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.124833+0000) 2020-02-05T22:59:50.309 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:50.307+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.308419+0000) 2020-02-05T22:59:50.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:50.308+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.309592+0000) 2020-02-05T22:59:50.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:50.309+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.309971+0000) 2020-02-05T22:59:50.311 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:50.309+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.310278+0000) 2020-02-05T22:59:50.311 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:50.309+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.310756+0000) 2020-02-05T22:59:50.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:50.311+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.311565+0000) 2020-02-05T22:59:50.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:50.311+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.311827+0000) 2020-02-05T22:59:50.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:50.311+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.312296+0000) 2020-02-05T22:59:50.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:50.312+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.312922+0000) 2020-02-05T22:59:50.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:50.313+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.313424+0000) 2020-02-05T22:59:50.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:50.312+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.313378+0000) 2020-02-05T22:59:50.315 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:50.313+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.314249+0000) 2020-02-05T22:59:50.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:50.312+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.314011+0000) 2020-02-05T22:59:50.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:50.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.317732+0000) 2020-02-05T22:59:50.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:50.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:50.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:50.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.362088+0000) 2020-02-05T22:59:50.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:50.376+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:50.376577+0000) 2020-02-05T22:59:52.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:52.376+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:52.377352+0000) 2020-02-05T22:59:52.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:52.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:52.730857+0000) 2020-02-05T22:59:52.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:52.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:52.768113+0000) 2020-02-05T22:59:53.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:53.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:53.891189+0000) 2020-02-05T22:59:53.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:53.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:53.891186+0000) 2020-02-05T22:59:55.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T22:59:55.327+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.328962+0000) 2020-02-05T22:59:55.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T22:59:55.329+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.330625+0000) 2020-02-05T22:59:55.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T22:59:55.330+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.331383+0000) 2020-02-05T22:59:55.332 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T22:59:55.330+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.331225+0000) 2020-02-05T22:59:55.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:55.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.331786+0000) 2020-02-05T22:59:55.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:55.331+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.331868+0000) 2020-02-05T22:59:55.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:55.331+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.332817+0000) 2020-02-05T22:59:55.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:55.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.332400+0000) 2020-02-05T22:59:55.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:55.332+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.332944+0000) 2020-02-05T22:59:55.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T22:59:55.332+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.333764+0000) 2020-02-05T22:59:55.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T22:59:55.333+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.334198+0000) 2020-02-05T22:59:55.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T22:59:55.333+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.333417+0000) 2020-02-05T22:59:55.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:55.333+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.333928+0000) 2020-02-05T22:59:55.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:55.336+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.337478+0000) 2020-02-05T22:59:55.345 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T22:59:55.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T22:59:55.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T22:59:55.990+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:55.990874+0000) 2020-02-05T22:59:56.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:56.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:56.268506+0000) 2020-02-05T22:59:57.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T22:59:57.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:57.131269+0000) 2020-02-05T22:59:57.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T22:59:57.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:57.131254+0000) 2020-02-05T22:59:57.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T22:59:57.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:57.131489+0000) 2020-02-05T22:59:58.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T22:59:58.598+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T21:59:58.598970+0000) 2020-02-05T23:00:00.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:00.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.125115+0000) 2020-02-05T23:00:00.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:00.349+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.350289+0000) 2020-02-05T23:00:00.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:00.350+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.351016+0000) 2020-02-05T23:00:00.352 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:00.350+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.351208+0000) 2020-02-05T23:00:00.353 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:00.350+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.351615+0000) 2020-02-05T23:00:00.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:00.351+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.352276+0000) 2020-02-05T23:00:00.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:00.351+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.352508+0000) 2020-02-05T23:00:00.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:00.351+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.352342+0000) 2020-02-05T23:00:00.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:00.353+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.353468+0000) 2020-02-05T23:00:00.355 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:00.353+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.354062+0000) 2020-02-05T23:00:00.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:00.353+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.353817+0000) 2020-02-05T23:00:00.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:00.354+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.354817+0000) 2020-02-05T23:00:00.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:00.354+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.355166+0000) 2020-02-05T23:00:00.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:00.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.360539+0000) 2020-02-05T23:00:00.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:00.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.360810+0000) 2020-02-05T23:00:00.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:00.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.362253+0000) 2020-02-05T23:00:00.366 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:00.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:00.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:00.375+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:00.376801+0000) 2020-02-05T23:00:02.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:02.377+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:02.377646+0000) 2020-02-05T23:00:02.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:02.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:02.731148+0000) 2020-02-05T23:00:02.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:02.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:02.768395+0000) 2020-02-05T23:00:03.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:03.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:03.891417+0000) 2020-02-05T23:00:03.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:03.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:03.891419+0000) 2020-02-05T23:00:05.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:05.371+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.372294+0000) 2020-02-05T23:00:05.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:05.371+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.372550+0000) 2020-02-05T23:00:05.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:05.372+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.373211+0000) 2020-02-05T23:00:05.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:05.372+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.373354+0000) 2020-02-05T23:00:05.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:05.373+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.373649+0000) 2020-02-05T23:00:05.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:05.373+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.374255+0000) 2020-02-05T23:00:05.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:05.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.374186+0000) 2020-02-05T23:00:05.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:05.374+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.374701+0000) 2020-02-05T23:00:05.376 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:05.374+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.375697+0000) 2020-02-05T23:00:05.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:05.376+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.376905+0000) 2020-02-05T23:00:05.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:05.376+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.377267+0000) 2020-02-05T23:00:05.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:05.377+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.377619+0000) 2020-02-05T23:00:05.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:05.379+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.381060+0000) 2020-02-05T23:00:05.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:05.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.381300+0000) 2020-02-05T23:00:05.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:05.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:05.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:05.990+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:05.991154+0000) 2020-02-05T23:00:06.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:06.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:06.268720+0000) 2020-02-05T23:00:07.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:07.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:07.131527+0000) 2020-02-05T23:00:07.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:07.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:07.131524+0000) 2020-02-05T23:00:07.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:07.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:07.131723+0000) 2020-02-05T23:00:08.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:08.606+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:08.607073+0000) 2020-02-05T23:00:10.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:10.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.125366+0000) 2020-02-05T23:00:10.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:10.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.362480+0000) 2020-02-05T23:00:10.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:10.376+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.377012+0000) 2020-02-05T23:00:10.394 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:10.393+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.393276+0000) 2020-02-05T23:00:10.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:10.393+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.394272+0000) 2020-02-05T23:00:10.396 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:10.395+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.395226+0000) 2020-02-05T23:00:10.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:10.395+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.395585+0000) 2020-02-05T23:00:10.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:10.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.395955+0000) 2020-02-05T23:00:10.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:10.396+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.396266+0000) 2020-02-05T23:00:10.397 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:10.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.396702+0000) 2020-02-05T23:00:10.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:10.396+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.396979+0000) 2020-02-05T23:00:10.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:10.396+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.397389+0000) 2020-02-05T23:00:10.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:10.398+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.398549+0000) 2020-02-05T23:00:10.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:10.398+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.398797+0000) 2020-02-05T23:00:10.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:10.398+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.399279+0000) 2020-02-05T23:00:10.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:10.399+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.399337+0000) 2020-02-05T23:00:10.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:10.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:10.403782+0000) 2020-02-05T23:00:10.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:10.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:11.120 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:00:11.126 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:00:12.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:12.377+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:12.377911+0000) 2020-02-05T23:00:12.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:12.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:12.731393+0000) 2020-02-05T23:00:12.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:12.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:12.768677+0000) 2020-02-05T23:00:13.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:13.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:13.891711+0000) 2020-02-05T23:00:13.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:13.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:13.891711+0000) 2020-02-05T23:00:15.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:15.412+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.413829+0000) 2020-02-05T23:00:15.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:15.414+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.415179+0000) 2020-02-05T23:00:15.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:15.414+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.415688+0000) 2020-02-05T23:00:15.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:15.415+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.416224+0000) 2020-02-05T23:00:15.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:15.416+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.417215+0000) 2020-02-05T23:00:15.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:15.416+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.417289+0000) 2020-02-05T23:00:15.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:15.417+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.418135+0000) 2020-02-05T23:00:15.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:15.417+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.417913+0000) 2020-02-05T23:00:15.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:15.418+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.419265+0000) 2020-02-05T23:00:15.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:15.418+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.419341+0000) 2020-02-05T23:00:15.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:15.419+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.419826+0000) 2020-02-05T23:00:15.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:15.419+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.420218+0000) 2020-02-05T23:00:15.421 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:15.419+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.420571+0000) 2020-02-05T23:00:15.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:15.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.422834+0000) 2020-02-05T23:00:15.428 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:15.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:15.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:15.991+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:15.991487+0000) 2020-02-05T23:00:16.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:16.267+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:16.268948+0000) 2020-02-05T23:00:17.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:17.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:17.131763+0000) 2020-02-05T23:00:17.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:17.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:17.131760+0000) 2020-02-05T23:00:17.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:17.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:17.131974+0000) 2020-02-05T23:00:18.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:18.614+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:18.615294+0000) 2020-02-05T23:00:20.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:20.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.125616+0000) 2020-02-05T23:00:20.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:20.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.362710+0000) 2020-02-05T23:00:20.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:20.376+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.377277+0000) 2020-02-05T23:00:20.434 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:20.432+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.433758+0000) 2020-02-05T23:00:20.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:20.432+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.433079+0000) 2020-02-05T23:00:20.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:20.433+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.434415+0000) 2020-02-05T23:00:20.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:20.434+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.434969+0000) 2020-02-05T23:00:20.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:20.434+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.435448+0000) 2020-02-05T23:00:20.438 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:20.435+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.436448+0000) 2020-02-05T23:00:20.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:20.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.437218+0000) 2020-02-05T23:00:20.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:20.436+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.437196+0000) 2020-02-05T23:00:20.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:20.437+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.437531+0000) 2020-02-05T23:00:20.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:20.437+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.437803+0000) 2020-02-05T23:00:20.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:20.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.437759+0000) 2020-02-05T23:00:20.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:20.438+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.438918+0000) 2020-02-05T23:00:20.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:20.439+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.439582+0000) 2020-02-05T23:00:20.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:20.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:20.443477+0000) 2020-02-05T23:00:20.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:20.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131856 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:22.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:22.378+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:22.378225+0000) 2020-02-05T23:00:22.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:22.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:22.731661+0000) 2020-02-05T23:00:22.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:22.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:22.768945+0000) 2020-02-05T23:00:23.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:23.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:23.891995+0000) 2020-02-05T23:00:23.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:23.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:23.891995+0000) 2020-02-05T23:00:25.457 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:25.455+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.456592+0000) 2020-02-05T23:00:25.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:25.456+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.457850+0000) 2020-02-05T23:00:25.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:25.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.458419+0000) 2020-02-05T23:00:25.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:25.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.458652+0000) 2020-02-05T23:00:25.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:25.457+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.458932+0000) 2020-02-05T23:00:25.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:25.457+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.459317+0000) 2020-02-05T23:00:25.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:25.458+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.460259+0000) 2020-02-05T23:00:25.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:25.459+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.460421+0000) 2020-02-05T23:00:25.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:25.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.460970+0000) 2020-02-05T23:00:25.463 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:25.460+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.462154+0000) 2020-02-05T23:00:25.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:25.460+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.462204+0000) 2020-02-05T23:00:25.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:25.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.462200+0000) 2020-02-05T23:00:25.464 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:25.461+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.463010+0000) 2020-02-05T23:00:25.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:25.464+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.465222+0000) 2020-02-05T23:00:25.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:25.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131897 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:25.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:25.990+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:25.991761+0000) 2020-02-05T23:00:26.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:26.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:26.269158+0000) 2020-02-05T23:00:27.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:27.130+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:27.131948+0000) 2020-02-05T23:00:27.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:27.130+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:27.131979+0000) 2020-02-05T23:00:27.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:27.130+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:27.132176+0000) 2020-02-05T23:00:28.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:28.622+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:28.623381+0000) 2020-02-05T23:00:30.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:30.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.125850+0000) 2020-02-05T23:00:30.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:30.361+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.362960+0000) 2020-02-05T23:00:30.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:30.376+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.377510+0000) 2020-02-05T23:00:30.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:30.476+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.477736+0000) 2020-02-05T23:00:30.479 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:30.477+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.478317+0000) 2020-02-05T23:00:30.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:30.477+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.478934+0000) 2020-02-05T23:00:30.480 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:30.478+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.479298+0000) 2020-02-05T23:00:30.480 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:30.478+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.479364+0000) 2020-02-05T23:00:30.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:30.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.479738+0000) 2020-02-05T23:00:30.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:30.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.479983+0000) 2020-02-05T23:00:30.481 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:30.478+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.480238+0000) 2020-02-05T23:00:30.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:30.479+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.480483+0000) 2020-02-05T23:00:30.482 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:30.479+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.481047+0000) 2020-02-05T23:00:30.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:30.479+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.481027+0000) 2020-02-05T23:00:30.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:30.480+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.481654+0000) 2020-02-05T23:00:30.484 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:30.481+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.483248+0000) 2020-02-05T23:00:30.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:30.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:30.488641+0000) 2020-02-05T23:00:30.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:30.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:32.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:32.377+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:32.378510+0000) 2020-02-05T23:00:32.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:32.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:32.731933+0000) 2020-02-05T23:00:32.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:32.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:32.769165+0000) 2020-02-05T23:00:33.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:33.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:33.892251+0000) 2020-02-05T23:00:33.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:33.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:33.892252+0000) 2020-02-05T23:00:35.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:35.496+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.498048+0000) 2020-02-05T23:00:35.501 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:35.497+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.498603+0000) 2020-02-05T23:00:35.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:35.497+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.499024+0000) 2020-02-05T23:00:35.501 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:35.499+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.500305+0000) 2020-02-05T23:00:35.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:35.499+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.501024+0000) 2020-02-05T23:00:35.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:35.500+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.501414+0000) 2020-02-05T23:00:35.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:35.500+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.501802+0000) 2020-02-05T23:00:35.503 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:35.500+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.502217+0000) 2020-02-05T23:00:35.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:35.501+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.502509+0000) 2020-02-05T23:00:35.504 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:35.501+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.502899+0000) 2020-02-05T23:00:35.504 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:35.502+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.503529+0000) 2020-02-05T23:00:35.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:35.502+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.503775+0000) 2020-02-05T23:00:35.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:35.503+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.504565+0000) 2020-02-05T23:00:35.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:35.505+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.506145+0000) 2020-02-05T23:00:35.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:35.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 131979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:35.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:35.991+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:35.991971+0000) 2020-02-05T23:00:36.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:36.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:36.269316+0000) 2020-02-05T23:00:37.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:37.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:37.132204+0000) 2020-02-05T23:00:37.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:37.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:37.132256+0000) 2020-02-05T23:00:37.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:37.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:37.132485+0000) 2020-02-05T23:00:38.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:38.630+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:38.631342+0000) 2020-02-05T23:00:40.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:40.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.126114+0000) 2020-02-05T23:00:40.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:40.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.363196+0000) 2020-02-05T23:00:40.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:40.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.377733+0000) 2020-02-05T23:00:40.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:40.517+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.518953+0000) 2020-02-05T23:00:40.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:40.518+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.519738+0000) 2020-02-05T23:00:40.521 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:40.518+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.519420+0000) 2020-02-05T23:00:40.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:40.519+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.520268+0000) 2020-02-05T23:00:40.522 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:40.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.520759+0000) 2020-02-05T23:00:40.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:40.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.521264+0000) 2020-02-05T23:00:40.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:40.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.522099+0000) 2020-02-05T23:00:40.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:40.521+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.522398+0000) 2020-02-05T23:00:40.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:40.521+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.522729+0000) 2020-02-05T23:00:40.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:40.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.524056+0000) 2020-02-05T23:00:40.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:40.522+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.523985+0000) 2020-02-05T23:00:40.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:40.522+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.523675+0000) 2020-02-05T23:00:40.526 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:40.523+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.524392+0000) 2020-02-05T23:00:40.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:40.527+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:40.528224+0000) 2020-02-05T23:00:40.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:40.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:41.333 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:00:41.339 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:00:42.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:42.377+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:42.378713+0000) 2020-02-05T23:00:42.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:42.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:42.732161+0000) 2020-02-05T23:00:42.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:42.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:42.769414+0000) 2020-02-05T23:00:43.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:43.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:43.892485+0000) 2020-02-05T23:00:43.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:43.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:43.892487+0000) 2020-02-05T23:00:45.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:45.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.540239+0000) 2020-02-05T23:00:45.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:45.539+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.540774+0000) 2020-02-05T23:00:45.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:45.539+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.540962+0000) 2020-02-05T23:00:45.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:45.540+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.541414+0000) 2020-02-05T23:00:45.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:45.540+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.541924+0000) 2020-02-05T23:00:45.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:45.541+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.542904+0000) 2020-02-05T23:00:45.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:45.541+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.542927+0000) 2020-02-05T23:00:45.545 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:45.542+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.543853+0000) 2020-02-05T23:00:45.545 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:45.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.543635+0000) 2020-02-05T23:00:45.546 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:45.543+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.544493+0000) 2020-02-05T23:00:45.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:45.543+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.544814+0000) 2020-02-05T23:00:45.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:45.543+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.545029+0000) 2020-02-05T23:00:45.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:45.544+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.545672+0000) 2020-02-05T23:00:45.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:45.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.549152+0000) 2020-02-05T23:00:45.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:45.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:45.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:45.991+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:45.992184+0000) 2020-02-05T23:00:46.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:46.268+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:46.269471+0000) 2020-02-05T23:00:47.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:47.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:47.132514+0000) 2020-02-05T23:00:47.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:47.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:47.132496+0000) 2020-02-05T23:00:47.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:47.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:47.132701+0000) 2020-02-05T23:00:48.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:48.639+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:48.639590+0000) 2020-02-05T23:00:50.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:50.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.126343+0000) 2020-02-05T23:00:50.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:50.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.363474+0000) 2020-02-05T23:00:50.378 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:50.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.377957+0000) 2020-02-05T23:00:50.560 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:50.558+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.559778+0000) 2020-02-05T23:00:50.561 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:50.559+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.560348+0000) 2020-02-05T23:00:50.563 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:50.560+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.561484+0000) 2020-02-05T23:00:50.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:50.560+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.561922+0000) 2020-02-05T23:00:50.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:50.559+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.561161+0000) 2020-02-05T23:00:50.564 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:50.561+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.563019+0000) 2020-02-05T23:00:50.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:50.561+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.562908+0000) 2020-02-05T23:00:50.565 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:50.562+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.563669+0000) 2020-02-05T23:00:50.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:50.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.564641+0000) 2020-02-05T23:00:50.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:50.563+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.565002+0000) 2020-02-05T23:00:50.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:50.564+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.565381+0000) 2020-02-05T23:00:50.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:50.564+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.565765+0000) 2020-02-05T23:00:50.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:50.564+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.566054+0000) 2020-02-05T23:00:50.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:50.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:50.568978+0000) 2020-02-05T23:00:50.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:50.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:52.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:52.377+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:52.378948+0000) 2020-02-05T23:00:52.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:52.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:52.732468+0000) 2020-02-05T23:00:52.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:52.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:52.769671+0000) 2020-02-05T23:00:53.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:53.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:53.892755+0000) 2020-02-05T23:00:53.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:53.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:53.892757+0000) 2020-02-05T23:00:55.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:00:55.581+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.582536+0000) 2020-02-05T23:00:55.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:00:55.581+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.583053+0000) 2020-02-05T23:00:55.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:55.582+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.583744+0000) 2020-02-05T23:00:55.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:00:55.583+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.584626+0000) 2020-02-05T23:00:55.586 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:00:55.583+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.584227+0000) 2020-02-05T23:00:55.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:00:55.583+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.584670+0000) 2020-02-05T23:00:55.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:55.583+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.584971+0000) 2020-02-05T23:00:55.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:55.583+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.585328+0000) 2020-02-05T23:00:55.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:55.584+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.586060+0000) 2020-02-05T23:00:55.588 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:00:55.584+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.585697+0000) 2020-02-05T23:00:55.588 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:00:55.585+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.587070+0000) 2020-02-05T23:00:55.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:55.585+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.587322+0000) 2020-02-05T23:00:55.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:55.591+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.592350+0000) 2020-02-05T23:00:55.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:55.592+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.593490+0000) 2020-02-05T23:00:55.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:00:55.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:00:55.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:00:55.991+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:55.992419+0000) 2020-02-05T23:00:56.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:56.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:56.269679+0000) 2020-02-05T23:00:57.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:00:57.131+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:57.132716+0000) 2020-02-05T23:00:57.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:00:57.131+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:57.132754+0000) 2020-02-05T23:00:57.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:00:57.131+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:57.132956+0000) 2020-02-05T23:00:58.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:00:58.648+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:00:58.648785+0000) 2020-02-05T23:01:00.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:00.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.126545+0000) 2020-02-05T23:01:00.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:00.362+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.363703+0000) 2020-02-05T23:01:00.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:00.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.378189+0000) 2020-02-05T23:01:00.605 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:00.603+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.604246+0000) 2020-02-05T23:01:00.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:00.603+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.604931+0000) 2020-02-05T23:01:00.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:00.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.605595+0000) 2020-02-05T23:01:00.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:00.605+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.606279+0000) 2020-02-05T23:01:00.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:00.605+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.606622+0000) 2020-02-05T23:01:00.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:00.605+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.607108+0000) 2020-02-05T23:01:00.608 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:00.606+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.608052+0000) 2020-02-05T23:01:00.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:00.607+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.608478+0000) 2020-02-05T23:01:00.610 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:00.607+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.609080+0000) 2020-02-05T23:01:00.610 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:00.608+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.609523+0000) 2020-02-05T23:01:00.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:00.608+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.609768+0000) 2020-02-05T23:01:00.611 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:00.609+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.610667+0000) 2020-02-05T23:01:00.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:00.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.614877+0000) 2020-02-05T23:01:00.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:00.614+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:00.615186+0000) 2020-02-05T23:01:00.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:00.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132181 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:02.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:02.378+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:02.379262+0000) 2020-02-05T23:01:02.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:02.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:02.732719+0000) 2020-02-05T23:01:02.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:02.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:02.769930+0000) 2020-02-05T23:01:03.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:03.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:03.893050+0000) 2020-02-05T23:01:03.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:03.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:03.893052+0000) 2020-02-05T23:01:05.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:05.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.628231+0000) 2020-02-05T23:01:05.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:05.628+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.629954+0000) 2020-02-05T23:01:05.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:05.630+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.631504+0000) 2020-02-05T23:01:05.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:05.630+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.632071+0000) 2020-02-05T23:01:05.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:05.630+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.632034+0000) 2020-02-05T23:01:05.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:05.630+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.632279+0000) 2020-02-05T23:01:05.638 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:05.626+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.626669+0000) 2020-02-05T23:01:05.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:05.627+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.627690+0000) 2020-02-05T23:01:05.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:05.628+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.628783+0000) 2020-02-05T23:01:05.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:05.629+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.629656+0000) 2020-02-05T23:01:05.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:05.630+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.630874+0000) 2020-02-05T23:01:05.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:05.631+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.631246+0000) 2020-02-05T23:01:05.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:05.631+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.631543+0000) 2020-02-05T23:01:05.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:05.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:05.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:05.636+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.636968+0000) 2020-02-05T23:01:05.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:05.992+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:05.992710+0000) 2020-02-05T23:01:06.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:06.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:06.269871+0000) 2020-02-05T23:01:07.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:07.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:07.132950+0000) 2020-02-05T23:01:07.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:07.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:07.133007+0000) 2020-02-05T23:01:07.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:07.132+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:07.133216+0000) 2020-02-05T23:01:08.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:08.656+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:08.657340+0000) 2020-02-05T23:01:10.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:10.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.126794+0000) 2020-02-05T23:01:10.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:10.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.363938+0000) 2020-02-05T23:01:10.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:10.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.378416+0000) 2020-02-05T23:01:10.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:10.646+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.647313+0000) 2020-02-05T23:01:10.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:10.647+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.648483+0000) 2020-02-05T23:01:10.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:10.647+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.648948+0000) 2020-02-05T23:01:10.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:10.647+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.648945+0000) 2020-02-05T23:01:10.651 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:10.648+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.649972+0000) 2020-02-05T23:01:10.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:10.649+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.650224+0000) 2020-02-05T23:01:10.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:10.648+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.650035+0000) 2020-02-05T23:01:10.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:10.649+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.650685+0000) 2020-02-05T23:01:10.652 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:10.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.651179+0000) 2020-02-05T23:01:10.652 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:10.650+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.651454+0000) 2020-02-05T23:01:10.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:10.650+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.652111+0000) 2020-02-05T23:01:10.653 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:10.650+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.652396+0000) 2020-02-05T23:01:10.656 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:10.651+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.652799+0000) 2020-02-05T23:01:10.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:10.654+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:10.655764+0000) 2020-02-05T23:01:10.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:10.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:11.479 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:01:11.485 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:01:12.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:12.379+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:12.379657+0000) 2020-02-05T23:01:12.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:12.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:12.732912+0000) 2020-02-05T23:01:12.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:12.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:12.770193+0000) 2020-02-05T23:01:13.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:13.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:13.893372+0000) 2020-02-05T23:01:13.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:13.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:13.893372+0000) 2020-02-05T23:01:15.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:15.671+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.671750+0000) 2020-02-05T23:01:15.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:15.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.672148+0000) 2020-02-05T23:01:15.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:15.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.673218+0000) 2020-02-05T23:01:15.675 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:15.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.672392+0000) 2020-02-05T23:01:15.675 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:15.671+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.672689+0000) 2020-02-05T23:01:15.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:15.672+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.673389+0000) 2020-02-05T23:01:15.676 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:15.673+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.674081+0000) 2020-02-05T23:01:15.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:15.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.674069+0000) 2020-02-05T23:01:15.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:15.673+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.674695+0000) 2020-02-05T23:01:15.677 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:15.674+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.675271+0000) 2020-02-05T23:01:15.678 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:15.675+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.675929+0000) 2020-02-05T23:01:15.679 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:15.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.676205+0000) 2020-02-05T23:01:15.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:15.676+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.676841+0000) 2020-02-05T23:01:15.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:15.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.682035+0000) 2020-02-05T23:01:15.685 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:15.683+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132304 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:15.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:15.992+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:15.992927+0000) 2020-02-05T23:01:16.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:16.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:16.270120+0000) 2020-02-05T23:01:17.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:17.132+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:17.133206+0000) 2020-02-05T23:01:17.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:17.132+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:17.133235+0000) 2020-02-05T23:01:17.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:17.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:17.133484+0000) 2020-02-05T23:01:18.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:18.665+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:18.666172+0000) 2020-02-05T23:01:20.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:20.126+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.127059+0000) 2020-02-05T23:01:20.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:20.363+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.364193+0000) 2020-02-05T23:01:20.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:20.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.378642+0000) 2020-02-05T23:01:20.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:20.690+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.690678+0000) 2020-02-05T23:01:20.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:20.690+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.691594+0000) 2020-02-05T23:01:20.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:20.691+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.692482+0000) 2020-02-05T23:01:20.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:20.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.692771+0000) 2020-02-05T23:01:20.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:20.692+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.693475+0000) 2020-02-05T23:01:20.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:20.692+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.693743+0000) 2020-02-05T23:01:20.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:20.694+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.694956+0000) 2020-02-05T23:01:20.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:20.694+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.694992+0000) 2020-02-05T23:01:20.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:20.694+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.695106+0000) 2020-02-05T23:01:20.697 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:20.695+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.696317+0000) 2020-02-05T23:01:20.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:20.695+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.696512+0000) 2020-02-05T23:01:20.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:20.695+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.696367+0000) 2020-02-05T23:01:20.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:20.696+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.697061+0000) 2020-02-05T23:01:20.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:20.698+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:20.699940+0000) 2020-02-05T23:01:20.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:20.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:22.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:22.379+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:22.379888+0000) 2020-02-05T23:01:22.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:22.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:22.733141+0000) 2020-02-05T23:01:22.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:22.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:22.770421+0000) 2020-02-05T23:01:23.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:23.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:23.893646+0000) 2020-02-05T23:01:23.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:23.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:23.893646+0000) 2020-02-05T23:01:25.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:25.711+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.712608+0000) 2020-02-05T23:01:25.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:25.712+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.713268+0000) 2020-02-05T23:01:25.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:25.713+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.714862+0000) 2020-02-05T23:01:25.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:25.714+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.715646+0000) 2020-02-05T23:01:25.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:25.715+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.715732+0000) 2020-02-05T23:01:25.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:25.715+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.716178+0000) 2020-02-05T23:01:25.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:25.715+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.716366+0000) 2020-02-05T23:01:25.718 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:25.716+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.716825+0000) 2020-02-05T23:01:25.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:25.715+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.716958+0000) 2020-02-05T23:01:25.719 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:25.716+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.717266+0000) 2020-02-05T23:01:25.719 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:25.717+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.717720+0000) 2020-02-05T23:01:25.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:25.717+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.717854+0000) 2020-02-05T23:01:25.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:25.718+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.718938+0000) 2020-02-05T23:01:25.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:25.723+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.724711+0000) 2020-02-05T23:01:25.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:25.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:25.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:25.992+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:25.993141+0000) 2020-02-05T23:01:26.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:26.269+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:26.270331+0000) 2020-02-05T23:01:27.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:27.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:27.133498+0000) 2020-02-05T23:01:27.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:27.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:27.133495+0000) 2020-02-05T23:01:27.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:27.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:27.133831+0000) 2020-02-05T23:01:28.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:28.673+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:28.674454+0000) 2020-02-05T23:01:30.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:30.126+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.127277+0000) 2020-02-05T23:01:30.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:30.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.364481+0000) 2020-02-05T23:01:30.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:30.377+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.378913+0000) 2020-02-05T23:01:30.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:30.733+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.734361+0000) 2020-02-05T23:01:30.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:30.733+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.734829+0000) 2020-02-05T23:01:30.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:30.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.735290+0000) 2020-02-05T23:01:30.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:30.734+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.735654+0000) 2020-02-05T23:01:30.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:30.735+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.736388+0000) 2020-02-05T23:01:30.737 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:30.736+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.737150+0000) 2020-02-05T23:01:30.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:30.737+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.738069+0000) 2020-02-05T23:01:30.739 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:30.737+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.738231+0000) 2020-02-05T23:01:30.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:30.738+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.738604+0000) 2020-02-05T23:01:30.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:30.738+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.739804+0000) 2020-02-05T23:01:30.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:30.739+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.739756+0000) 2020-02-05T23:01:30.741 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:30.740+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.740413+0000) 2020-02-05T23:01:30.741 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:30.740+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.740810+0000) 2020-02-05T23:01:30.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:30.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:30.746156+0000) 2020-02-05T23:01:30.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:30.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:32.381 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:32.379+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:32.380139+0000) 2020-02-05T23:01:32.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:32.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:32.733328+0000) 2020-02-05T23:01:32.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:32.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:32.770650+0000) 2020-02-05T23:01:33.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:33.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:33.893938+0000) 2020-02-05T23:01:33.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:33.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:33.893941+0000) 2020-02-05T23:01:35.756 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:35.754+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.755077+0000) 2020-02-05T23:01:35.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:35.754+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.755603+0000) 2020-02-05T23:01:35.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:35.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.755993+0000) 2020-02-05T23:01:35.757 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:35.755+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.756200+0000) 2020-02-05T23:01:35.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:35.755+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.756426+0000) 2020-02-05T23:01:35.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:35.756+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.756675+0000) 2020-02-05T23:01:35.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:35.756+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.756693+0000) 2020-02-05T23:01:35.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:35.756+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.757255+0000) 2020-02-05T23:01:35.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:35.756+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.757268+0000) 2020-02-05T23:01:35.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:35.756+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.757994+0000) 2020-02-05T23:01:35.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:35.757+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.758557+0000) 2020-02-05T23:01:35.760 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:35.758+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.759154+0000) 2020-02-05T23:01:35.760 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:35.759+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.760091+0000) 2020-02-05T23:01:35.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:35.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.764533+0000) 2020-02-05T23:01:35.769 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:35.768+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:35.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:35.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:35.993372+0000) 2020-02-05T23:01:36.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:36.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:36.270567+0000) 2020-02-05T23:01:37.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:37.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:37.133733+0000) 2020-02-05T23:01:37.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:37.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:37.133728+0000) 2020-02-05T23:01:37.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:37.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:37.134125+0000) 2020-02-05T23:01:38.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:38.682+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:38.682646+0000) 2020-02-05T23:01:40.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:40.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.127513+0000) 2020-02-05T23:01:40.365 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:40.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.364739+0000) 2020-02-05T23:01:40.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:40.378+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.379161+0000) 2020-02-05T23:01:40.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:40.772+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.772990+0000) 2020-02-05T23:01:40.775 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:40.773+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.774732+0000) 2020-02-05T23:01:40.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:40.774+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.775553+0000) 2020-02-05T23:01:40.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:40.775+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.776232+0000) 2020-02-05T23:01:40.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:40.776+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.776715+0000) 2020-02-05T23:01:40.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:40.775+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.776865+0000) 2020-02-05T23:01:40.779 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:40.777+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.777559+0000) 2020-02-05T23:01:40.780 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:40.777+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.777738+0000) 2020-02-05T23:01:40.780 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:40.778+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.778717+0000) 2020-02-05T23:01:40.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:40.778+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.779637+0000) 2020-02-05T23:01:40.781 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:40.778+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.780032+0000) 2020-02-05T23:01:40.781 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:40.779+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.780262+0000) 2020-02-05T23:01:40.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:40.779+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.780488+0000) 2020-02-05T23:01:40.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:40.784+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:40.785768+0000) 2020-02-05T23:01:40.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:40.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132509 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:41.652 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:01:41.658 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:01:42.381 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:42.380+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:42.380485+0000) 2020-02-05T23:01:42.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:42.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:42.733575+0000) 2020-02-05T23:01:42.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:42.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:42.770886+0000) 2020-02-05T23:01:43.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:43.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:43.894212+0000) 2020-02-05T23:01:43.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:43.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:43.894209+0000) 2020-02-05T23:01:45.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:45.797+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.797762+0000) 2020-02-05T23:01:45.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:45.797+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.798422+0000) 2020-02-05T23:01:45.801 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:45.798+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.799349+0000) 2020-02-05T23:01:45.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:45.798+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.799905+0000) 2020-02-05T23:01:45.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:45.799+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.800278+0000) 2020-02-05T23:01:45.803 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:45.800+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.800544+0000) 2020-02-05T23:01:45.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:45.800+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.801342+0000) 2020-02-05T23:01:45.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:45.801+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.801658+0000) 2020-02-05T23:01:45.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:45.799+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.800919+0000) 2020-02-05T23:01:45.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:45.799+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.800971+0000) 2020-02-05T23:01:45.804 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:45.800+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.801492+0000) 2020-02-05T23:01:45.804 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:45.801+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.802302+0000) 2020-02-05T23:01:45.804 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:45.801+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.802152+0000) 2020-02-05T23:01:45.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:45.807+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.808660+0000) 2020-02-05T23:01:45.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:45.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:45.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:45.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:45.993575+0000) 2020-02-05T23:01:46.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:46.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:46.270816+0000) 2020-02-05T23:01:47.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:47.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:47.134002+0000) 2020-02-05T23:01:47.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:47.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:47.133981+0000) 2020-02-05T23:01:47.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:47.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:47.134329+0000) 2020-02-05T23:01:48.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:48.691+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:48.691460+0000) 2020-02-05T23:01:50.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:50.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.127782+0000) 2020-02-05T23:01:50.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:50.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.364991+0000) 2020-02-05T23:01:50.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:50.379+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.379389+0000) 2020-02-05T23:01:50.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:50.817+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.818579+0000) 2020-02-05T23:01:50.820 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:50.818+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.819268+0000) 2020-02-05T23:01:50.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:50.817+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.819004+0000) 2020-02-05T23:01:50.821 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:50.819+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.820046+0000) 2020-02-05T23:01:50.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:50.819+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.820109+0000) 2020-02-05T23:01:50.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:50.820+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.821205+0000) 2020-02-05T23:01:50.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:50.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.821423+0000) 2020-02-05T23:01:50.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:50.821+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.822130+0000) 2020-02-05T23:01:50.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:50.822+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.822483+0000) 2020-02-05T23:01:50.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:50.822+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.822876+0000) 2020-02-05T23:01:50.824 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:50.822+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.823412+0000) 2020-02-05T23:01:50.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:50.822+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.823357+0000) 2020-02-05T23:01:50.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:50.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.823656+0000) 2020-02-05T23:01:50.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:50.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:50.827421+0000) 2020-02-05T23:01:50.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:50.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132590 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:52.381 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:52.380+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:52.380736+0000) 2020-02-05T23:01:52.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:52.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:52.733798+0000) 2020-02-05T23:01:52.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:52.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:52.771131+0000) 2020-02-05T23:01:53.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:53.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:53.894492+0000) 2020-02-05T23:01:53.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:53.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:53.894492+0000) 2020-02-05T23:01:55.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:55.835+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.837287+0000) 2020-02-05T23:01:55.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:01:55.837+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.838588+0000) 2020-02-05T23:01:55.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:55.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.840509+0000) 2020-02-05T23:01:55.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:01:55.839+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.840933+0000) 2020-02-05T23:01:55.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:55.840+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.841924+0000) 2020-02-05T23:01:55.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:01:55.841+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.842256+0000) 2020-02-05T23:01:55.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:01:55.842+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.843122+0000) 2020-02-05T23:01:55.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:55.842+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.843222+0000) 2020-02-05T23:01:55.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:01:55.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.843687+0000) 2020-02-05T23:01:55.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:55.842+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.843632+0000) 2020-02-05T23:01:55.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:01:55.842+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.844151+0000) 2020-02-05T23:01:55.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:55.843+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.844626+0000) 2020-02-05T23:01:55.846 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:01:55.842+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.843927+0000) 2020-02-05T23:01:55.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:55.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.846601+0000) 2020-02-05T23:01:55.853 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:01:55.851+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132631 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:01:55.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:01:55.992+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:55.993791+0000) 2020-02-05T23:01:56.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:56.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:56.271151+0000) 2020-02-05T23:01:57.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:01:57.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:57.134248+0000) 2020-02-05T23:01:57.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:01:57.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:57.134248+0000) 2020-02-05T23:01:57.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:01:57.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:57.134526+0000) 2020-02-05T23:01:58.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:01:58.699+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:01:58.699936+0000) 2020-02-05T23:02:00.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:00.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.128051+0000) 2020-02-05T23:02:00.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:00.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.365245+0000) 2020-02-05T23:02:00.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:00.379+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.379639+0000) 2020-02-05T23:02:00.859 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:00.857+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.858765+0000) 2020-02-05T23:02:00.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:00.859+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.859241+0000) 2020-02-05T23:02:00.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:00.859+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.859736+0000) 2020-02-05T23:02:00.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:00.859+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.860422+0000) 2020-02-05T23:02:00.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:00.861+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.861432+0000) 2020-02-05T23:02:00.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:00.860+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.861781+0000) 2020-02-05T23:02:00.863 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:00.861+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.863144+0000) 2020-02-05T23:02:00.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:00.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.863123+0000) 2020-02-05T23:02:00.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:00.862+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.863793+0000) 2020-02-05T23:02:00.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:00.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.864173+0000) 2020-02-05T23:02:00.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:00.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.864431+0000) 2020-02-05T23:02:00.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:00.863+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.864579+0000) 2020-02-05T23:02:00.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:00.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.867156+0000) 2020-02-05T23:02:00.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:00.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:00.867424+0000) 2020-02-05T23:02:00.876 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:00.873+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132671 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:02.382 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:02.380+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:02.381001+0000) 2020-02-05T23:02:02.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:02.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:02.734028+0000) 2020-02-05T23:02:02.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:02.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:02.771366+0000) 2020-02-05T23:02:03.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:03.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:03.894747+0000) 2020-02-05T23:02:03.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:03.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:03.894754+0000) 2020-02-05T23:02:05.881 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:05.879+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.880299+0000) 2020-02-05T23:02:05.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:05.878+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.880170+0000) 2020-02-05T23:02:05.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:05.881+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.882460+0000) 2020-02-05T23:02:05.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:05.881+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.882867+0000) 2020-02-05T23:02:05.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:05.882+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.883404+0000) 2020-02-05T23:02:05.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:05.882+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.883807+0000) 2020-02-05T23:02:05.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:05.883+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.884150+0000) 2020-02-05T23:02:05.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:05.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.884704+0000) 2020-02-05T23:02:05.886 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:05.883+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.885144+0000) 2020-02-05T23:02:05.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:05.884+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.885811+0000) 2020-02-05T23:02:05.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:05.884+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.886077+0000) 2020-02-05T23:02:05.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:05.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.886785+0000) 2020-02-05T23:02:05.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:05.885+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.886883+0000) 2020-02-05T23:02:05.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:05.890+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.891463+0000) 2020-02-05T23:02:05.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:05.894+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132712 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:05.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:05.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:05.994051+0000) 2020-02-05T23:02:06.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:06.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:06.271365+0000) 2020-02-05T23:02:07.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:07.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:07.134498+0000) 2020-02-05T23:02:07.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:07.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:07.134495+0000) 2020-02-05T23:02:07.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:07.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:07.134787+0000) 2020-02-05T23:02:08.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:08.708+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:08.708990+0000) 2020-02-05T23:02:10.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:10.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.128303+0000) 2020-02-05T23:02:10.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:10.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.365486+0000) 2020-02-05T23:02:10.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:10.378+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.379888+0000) 2020-02-05T23:02:10.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:10.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.901709+0000) 2020-02-05T23:02:10.903 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:10.901+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.902647+0000) 2020-02-05T23:02:10.904 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:10.901+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.903055+0000) 2020-02-05T23:02:10.905 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:10.902+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.904129+0000) 2020-02-05T23:02:10.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:10.902+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.903826+0000) 2020-02-05T23:02:10.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:10.903+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.904411+0000) 2020-02-05T23:02:10.906 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:10.903+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.904855+0000) 2020-02-05T23:02:10.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:10.904+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.905275+0000) 2020-02-05T23:02:10.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:10.904+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.905713+0000) 2020-02-05T23:02:10.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:10.904+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.906050+0000) 2020-02-05T23:02:10.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:10.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.907815+0000) 2020-02-05T23:02:10.909 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:10.905+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.907128+0000) 2020-02-05T23:02:10.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:10.907+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.908401+0000) 2020-02-05T23:02:10.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:10.913+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:10.914446+0000) 2020-02-05T23:02:10.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:10.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:11.823 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:02:11.830 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:02:12.382 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:12.379+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:12.381349+0000) 2020-02-05T23:02:12.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:12.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:12.734327+0000) 2020-02-05T23:02:12.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:12.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:12.771619+0000) 2020-02-05T23:02:13.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:13.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:13.895035+0000) 2020-02-05T23:02:13.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:13.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:13.895038+0000) 2020-02-05T23:02:15.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:15.921+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.922958+0000) 2020-02-05T23:02:15.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:15.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.923596+0000) 2020-02-05T23:02:15.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:15.923+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.924329+0000) 2020-02-05T23:02:15.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:15.923+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.924847+0000) 2020-02-05T23:02:15.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:15.923+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.924944+0000) 2020-02-05T23:02:15.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:15.923+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.925342+0000) 2020-02-05T23:02:15.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:15.924+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.925640+0000) 2020-02-05T23:02:15.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:15.925+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.926546+0000) 2020-02-05T23:02:15.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:15.925+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.927220+0000) 2020-02-05T23:02:15.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:15.926+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.927549+0000) 2020-02-05T23:02:15.929 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:15.927+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.928668+0000) 2020-02-05T23:02:15.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:15.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.929232+0000) 2020-02-05T23:02:15.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:15.928+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.929538+0000) 2020-02-05T23:02:15.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:15.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.935784+0000) 2020-02-05T23:02:15.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:15.937+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132795 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:15.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:15.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:15.994288+0000) 2020-02-05T23:02:16.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:16.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:16.271593+0000) 2020-02-05T23:02:17.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:17.133+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:17.134737+0000) 2020-02-05T23:02:17.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:17.133+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:17.134723+0000) 2020-02-05T23:02:17.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:17.133+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:17.135031+0000) 2020-02-05T23:02:18.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:18.719+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:18.720474+0000) 2020-02-05T23:02:20.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:20.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.128604+0000) 2020-02-05T23:02:20.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:20.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.365721+0000) 2020-02-05T23:02:20.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:20.379+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.380119+0000) 2020-02-05T23:02:20.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:20.942+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.943637+0000) 2020-02-05T23:02:20.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:20.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.945441+0000) 2020-02-05T23:02:20.946 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:20.944+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.945929+0000) 2020-02-05T23:02:20.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:20.945+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.946236+0000) 2020-02-05T23:02:20.947 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:20.945+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.946925+0000) 2020-02-05T23:02:20.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:20.946+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.947235+0000) 2020-02-05T23:02:20.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:20.946+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.947633+0000) 2020-02-05T23:02:20.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:20.946+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.948207+0000) 2020-02-05T23:02:20.949 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:20.947+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.948765+0000) 2020-02-05T23:02:20.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:20.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.949447+0000) 2020-02-05T23:02:20.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:20.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.950224+0000) 2020-02-05T23:02:20.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:20.949+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.950370+0000) 2020-02-05T23:02:20.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:20.948+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.950368+0000) 2020-02-05T23:02:20.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:20.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:20.955595+0000) 2020-02-05T23:02:20.960 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:20.957+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:22.382 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:22.380+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:22.381660+0000) 2020-02-05T23:02:22.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:22.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:22.734568+0000) 2020-02-05T23:02:22.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:22.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:22.771852+0000) 2020-02-05T23:02:23.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:23.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:23.895289+0000) 2020-02-05T23:02:23.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:23.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:23.895305+0000) 2020-02-05T23:02:25.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:25.964+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.965987+0000) 2020-02-05T23:02:25.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:25.965+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.966641+0000) 2020-02-05T23:02:25.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:25.965+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.967123+0000) 2020-02-05T23:02:25.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:25.966+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.968283+0000) 2020-02-05T23:02:25.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:25.967+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.968748+0000) 2020-02-05T23:02:25.970 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:25.967+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.968713+0000) 2020-02-05T23:02:25.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:25.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.969342+0000) 2020-02-05T23:02:25.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:25.968+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.969433+0000) 2020-02-05T23:02:25.971 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:25.968+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.969860+0000) 2020-02-05T23:02:25.971 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:25.968+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.970050+0000) 2020-02-05T23:02:25.971 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:25.969+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.970912+0000) 2020-02-05T23:02:25.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:25.970+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.971381+0000) 2020-02-05T23:02:25.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:25.970+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.971444+0000) 2020-02-05T23:02:25.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:25.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.975455+0000) 2020-02-05T23:02:25.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:25.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132879 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:25.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:25.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:25.994522+0000) 2020-02-05T23:02:26.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:26.270+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:26.271820+0000) 2020-02-05T23:02:27.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:27.134+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:27.135000+0000) 2020-02-05T23:02:27.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:27.134+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:27.134997+0000) 2020-02-05T23:02:27.136 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:27.134+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:27.135282+0000) 2020-02-05T23:02:28.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:28.726+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:28.727336+0000) 2020-02-05T23:02:30.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:30.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.128920+0000) 2020-02-05T23:02:30.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:30.364+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.365977+0000) 2020-02-05T23:02:30.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:30.379+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.380345+0000) 2020-02-05T23:02:30.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:30.985+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.986707+0000) 2020-02-05T23:02:30.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:30.986+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.987455+0000) 2020-02-05T23:02:30.988 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:30.986+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.987799+0000) 2020-02-05T23:02:30.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:30.987+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.988686+0000) 2020-02-05T23:02:30.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:30.988+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.989505+0000) 2020-02-05T23:02:30.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:30.988+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.989689+0000) 2020-02-05T23:02:30.991 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:30.988+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.990119+0000) 2020-02-05T23:02:30.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:30.989+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.991008+0000) 2020-02-05T23:02:30.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:30.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.991671+0000) 2020-02-05T23:02:30.993 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:30.990+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.992036+0000) 2020-02-05T23:02:30.993 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:30.990+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.992412+0000) 2020-02-05T23:02:30.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:30.990+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.991986+0000) 2020-02-05T23:02:30.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:30.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.992595+0000) 2020-02-05T23:02:30.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:30.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:30.996500+0000) 2020-02-05T23:02:31.003 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:31.000+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:32.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:32.381+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:32.381920+0000) 2020-02-05T23:02:32.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:32.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:32.734792+0000) 2020-02-05T23:02:32.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:32.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:32.772108+0000) 2020-02-05T23:02:33.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:33.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:33.895573+0000) 2020-02-05T23:02:33.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:33.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:33.895573+0000) 2020-02-05T23:02:35.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:35.993+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:35.994717+0000) 2020-02-05T23:02:36.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:36.008+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.009400+0000) 2020-02-05T23:02:36.010 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:36.008+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.009986+0000) 2020-02-05T23:02:36.011 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:36.009+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.010436+0000) 2020-02-05T23:02:36.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:36.009+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.010890+0000) 2020-02-05T23:02:36.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:36.010+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.011597+0000) 2020-02-05T23:02:36.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:36.011+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.012413+0000) 2020-02-05T23:02:36.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:36.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.012562+0000) 2020-02-05T23:02:36.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:36.012+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.013140+0000) 2020-02-05T23:02:36.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:36.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.013488+0000) 2020-02-05T23:02:36.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:36.012+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.014017+0000) 2020-02-05T23:02:36.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:36.013+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.014560+0000) 2020-02-05T23:02:36.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:36.013+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.014550+0000) 2020-02-05T23:02:36.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:36.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.014059+0000) 2020-02-05T23:02:36.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:36.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.020011+0000) 2020-02-05T23:02:36.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:36.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 132961 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:36.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:36.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:36.272036+0000) 2020-02-05T23:02:37.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:37.134+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:37.135242+0000) 2020-02-05T23:02:37.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:37.134+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:37.135239+0000) 2020-02-05T23:02:37.137 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:37.134+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:37.135499+0000) 2020-02-05T23:02:38.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:38.735+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:38.735621+0000) 2020-02-05T23:02:40.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:40.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:40.129156+0000) 2020-02-05T23:02:40.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:40.365+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:40.366227+0000) 2020-02-05T23:02:40.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:40.380+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:40.380616+0000) 2020-02-05T23:02:41.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:41.026+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.027314+0000) 2020-02-05T23:02:41.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:41.027+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.028295+0000) 2020-02-05T23:02:41.030 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:41.028+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.029782+0000) 2020-02-05T23:02:41.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:41.029+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.030150+0000) 2020-02-05T23:02:41.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:41.028+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.030089+0000) 2020-02-05T23:02:41.031 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:41.028+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.030411+0000) 2020-02-05T23:02:41.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:41.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.030939+0000) 2020-02-05T23:02:41.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:41.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.031342+0000) 2020-02-05T23:02:41.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:41.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.030903+0000) 2020-02-05T23:02:41.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:41.029+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.031343+0000) 2020-02-05T23:02:41.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:41.030+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.031779+0000) 2020-02-05T23:02:41.034 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:41.030+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.032000+0000) 2020-02-05T23:02:41.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:41.030+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.032250+0000) 2020-02-05T23:02:41.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:41.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:41.037581+0000) 2020-02-05T23:02:41.044 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:41.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:42.012 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:02:42.018 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:02:42.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:42.381+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:42.382173+0000) 2020-02-05T23:02:42.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:42.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:42.735049+0000) 2020-02-05T23:02:42.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:42.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:42.772338+0000) 2020-02-05T23:02:43.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:43.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:43.895802+0000) 2020-02-05T23:02:43.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:43.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:43.895812+0000) 2020-02-05T23:02:45.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:45.994+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:45.994952+0000) 2020-02-05T23:02:46.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:46.047+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.048344+0000) 2020-02-05T23:02:46.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:46.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.049142+0000) 2020-02-05T23:02:46.050 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:46.048+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.049653+0000) 2020-02-05T23:02:46.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:46.049+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.050274+0000) 2020-02-05T23:02:46.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:46.049+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.050796+0000) 2020-02-05T23:02:46.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:46.050+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.051104+0000) 2020-02-05T23:02:46.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:46.050+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.051563+0000) 2020-02-05T23:02:46.052 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:46.050+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.051871+0000) 2020-02-05T23:02:46.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:46.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.052234+0000) 2020-02-05T23:02:46.053 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:46.051+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.052393+0000) 2020-02-05T23:02:46.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:46.052+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.053046+0000) 2020-02-05T23:02:46.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:46.053+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.053813+0000) 2020-02-05T23:02:46.055 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:46.053+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.053881+0000) 2020-02-05T23:02:46.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:46.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.057591+0000) 2020-02-05T23:02:46.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:46.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:46.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:46.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:46.272294+0000) 2020-02-05T23:02:47.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:47.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:47.135511+0000) 2020-02-05T23:02:47.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:47.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:47.135516+0000) 2020-02-05T23:02:47.137 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:47.135+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:47.135737+0000) 2020-02-05T23:02:48.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:48.743+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:48.743510+0000) 2020-02-05T23:02:50.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:50.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:50.129391+0000) 2020-02-05T23:02:50.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:50.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:50.366528+0000) 2020-02-05T23:02:50.381 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:50.380+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:50.380850+0000) 2020-02-05T23:02:51.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:51.070+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.071258+0000) 2020-02-05T23:02:51.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:51.071+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.073000+0000) 2020-02-05T23:02:51.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:51.073+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.074138+0000) 2020-02-05T23:02:51.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:51.073+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.074481+0000) 2020-02-05T23:02:51.075 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:51.074+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.074857+0000) 2020-02-05T23:02:51.076 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:51.074+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.075340+0000) 2020-02-05T23:02:51.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:51.074+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.075687+0000) 2020-02-05T23:02:51.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:51.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.076336+0000) 2020-02-05T23:02:51.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:51.075+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.076474+0000) 2020-02-05T23:02:51.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:51.076+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.077594+0000) 2020-02-05T23:02:51.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:51.077+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.077512+0000) 2020-02-05T23:02:51.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:51.077+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.077956+0000) 2020-02-05T23:02:51.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:51.078+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.078478+0000) 2020-02-05T23:02:51.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:51.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:51.081980+0000) 2020-02-05T23:02:51.089 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:51.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133086 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:52.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:52.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:52.382489+0000) 2020-02-05T23:02:52.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:52.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:52.735312+0000) 2020-02-05T23:02:52.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:52.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:52.772586+0000) 2020-02-05T23:02:53.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:53.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:53.896124+0000) 2020-02-05T23:02:53.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:53.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:53.896124+0000) 2020-02-05T23:02:55.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:55.994+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:55.995209+0000) 2020-02-05T23:02:56.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:02:56.093+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.093550+0000) 2020-02-05T23:02:56.095 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:02:56.094+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.094871+0000) 2020-02-05T23:02:56.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:56.095+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.095868+0000) 2020-02-05T23:02:56.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:56.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.096450+0000) 2020-02-05T23:02:56.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:02:56.096+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.096602+0000) 2020-02-05T23:02:56.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:56.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.097367+0000) 2020-02-05T23:02:56.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:56.097+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.097620+0000) 2020-02-05T23:02:56.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:02:56.097+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.097927+0000) 2020-02-05T23:02:56.099 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:02:56.098+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.098340+0000) 2020-02-05T23:02:56.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:02:56.098+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.098560+0000) 2020-02-05T23:02:56.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:02:56.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.099680+0000) 2020-02-05T23:02:56.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:56.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.100230+0000) 2020-02-05T23:02:56.101 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:02:56.100+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.100453+0000) 2020-02-05T23:02:56.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:56.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.104662+0000) 2020-02-05T23:02:56.109 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:02:56.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:02:56.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:56.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:56.272501+0000) 2020-02-05T23:02:57.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:02:57.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:57.135756+0000) 2020-02-05T23:02:57.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:02:57.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:57.135761+0000) 2020-02-05T23:02:57.137 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:02:57.135+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:57.135984+0000) 2020-02-05T23:02:58.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:02:58.750+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:02:58.751782+0000) 2020-02-05T23:03:00.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:00.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:00.129630+0000) 2020-02-05T23:03:00.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:00.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:00.366765+0000) 2020-02-05T23:03:00.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:00.380+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:00.381109+0000) 2020-02-05T23:03:01.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:01.114+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.115569+0000) 2020-02-05T23:03:01.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:01.114+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.116060+0000) 2020-02-05T23:03:01.117 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:01.115+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.116509+0000) 2020-02-05T23:03:01.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:01.116+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.116988+0000) 2020-02-05T23:03:01.118 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:01.116+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.117315+0000) 2020-02-05T23:03:01.119 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:01.117+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.118398+0000) 2020-02-05T23:03:01.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:01.117+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.117918+0000) 2020-02-05T23:03:01.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:01.118+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.118415+0000) 2020-02-05T23:03:01.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:01.118+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.119176+0000) 2020-02-05T23:03:01.120 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:01.118+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.119499+0000) 2020-02-05T23:03:01.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:01.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.119640+0000) 2020-02-05T23:03:01.121 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:01.119+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.120331+0000) 2020-02-05T23:03:01.121 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:01.119+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.120190+0000) 2020-02-05T23:03:01.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:01.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:01.125400+0000) 2020-02-05T23:03:01.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:01.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:02.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:02.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:02.382727+0000) 2020-02-05T23:03:02.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:02.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:02.735545+0000) 2020-02-05T23:03:02.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:02.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:02.772814+0000) 2020-02-05T23:03:03.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:03.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:03.896412+0000) 2020-02-05T23:03:03.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:03.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:03.896413+0000) 2020-02-05T23:03:05.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:05.995+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:05.995490+0000) 2020-02-05T23:03:06.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:06.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.137795+0000) 2020-02-05T23:03:06.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:06.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.142415+0000) 2020-02-05T23:03:06.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:06.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.138358+0000) 2020-02-05T23:03:06.146 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:06.137+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.138852+0000) 2020-02-05T23:03:06.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:06.138+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.139802+0000) 2020-02-05T23:03:06.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:06.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.142251+0000) 2020-02-05T23:03:06.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:06.141+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.142637+0000) 2020-02-05T23:03:06.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:06.135+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.135458+0000) 2020-02-05T23:03:06.147 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:06.136+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.136797+0000) 2020-02-05T23:03:06.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:06.138+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.138798+0000) 2020-02-05T23:03:06.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:06.140+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.140818+0000) 2020-02-05T23:03:06.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:06.140+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.140682+0000) 2020-02-05T23:03:06.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:06.141+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.142109+0000) 2020-02-05T23:03:06.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:06.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.148017+0000) 2020-02-05T23:03:06.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:06.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:06.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:06.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:06.272703+0000) 2020-02-05T23:03:07.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:07.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:07.135993+0000) 2020-02-05T23:03:07.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:07.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:07.136006+0000) 2020-02-05T23:03:07.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:07.135+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:07.136192+0000) 2020-02-05T23:03:08.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:08.759+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:08.759910+0000) 2020-02-05T23:03:10.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:10.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:10.129859+0000) 2020-02-05T23:03:10.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:10.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:10.367005+0000) 2020-02-05T23:03:10.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:10.380+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:10.381349+0000) 2020-02-05T23:03:11.159 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:11.157+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.157998+0000) 2020-02-05T23:03:11.159 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:11.157+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.158293+0000) 2020-02-05T23:03:11.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:11.158+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.159200+0000) 2020-02-05T23:03:11.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:11.159+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.159958+0000) 2020-02-05T23:03:11.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:11.158+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.159973+0000) 2020-02-05T23:03:11.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:11.159+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.160336+0000) 2020-02-05T23:03:11.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:11.160+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.160781+0000) 2020-02-05T23:03:11.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:11.160+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.161804+0000) 2020-02-05T23:03:11.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:11.161+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.161462+0000) 2020-02-05T23:03:11.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:11.161+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.162454+0000) 2020-02-05T23:03:11.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:11.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.163280+0000) 2020-02-05T23:03:11.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:11.162+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.163308+0000) 2020-02-05T23:03:11.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:11.162+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.163761+0000) 2020-02-05T23:03:11.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:11.168+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:11.169827+0000) 2020-02-05T23:03:11.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:11.172+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:12.175 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:03:12.181 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:03:12.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:12.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:12.382998+0000) 2020-02-05T23:03:12.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:12.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:12.735783+0000) 2020-02-05T23:03:12.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:12.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:12.773050+0000) 2020-02-05T23:03:13.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:13.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:13.896690+0000) 2020-02-05T23:03:13.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:13.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:13.896690+0000) 2020-02-05T23:03:15.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:15.995+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:15.995763+0000) 2020-02-05T23:03:16.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:16.178+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.179231+0000) 2020-02-05T23:03:16.181 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:16.179+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.179499+0000) 2020-02-05T23:03:16.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:16.178+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.179963+0000) 2020-02-05T23:03:16.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:16.179+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.180429+0000) 2020-02-05T23:03:16.185 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:16.179+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.180206+0000) 2020-02-05T23:03:16.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:16.180+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.180861+0000) 2020-02-05T23:03:16.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:16.181+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.181683+0000) 2020-02-05T23:03:16.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:16.181+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.182078+0000) 2020-02-05T23:03:16.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:16.180+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.181358+0000) 2020-02-05T23:03:16.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:16.181+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.182895+0000) 2020-02-05T23:03:16.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:16.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.183438+0000) 2020-02-05T23:03:16.187 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:16.182+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.183656+0000) 2020-02-05T23:03:16.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:16.183+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.184052+0000) 2020-02-05T23:03:16.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:16.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.189613+0000) 2020-02-05T23:03:16.195 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:16.193+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:16.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:16.271+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:16.272920+0000) 2020-02-05T23:03:17.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:17.136+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:17.136252+0000) 2020-02-05T23:03:17.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:17.136+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:17.136268+0000) 2020-02-05T23:03:17.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:17.136+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:17.136419+0000) 2020-02-05T23:03:18.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:18.767+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:18.768267+0000) 2020-02-05T23:03:20.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:20.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:20.130137+0000) 2020-02-05T23:03:20.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:20.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:20.367299+0000) 2020-02-05T23:03:20.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:20.380+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:20.381573+0000) 2020-02-05T23:03:21.200 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:21.198+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.199472+0000) 2020-02-05T23:03:21.201 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:21.199+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.199879+0000) 2020-02-05T23:03:21.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:21.200+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.201539+0000) 2020-02-05T23:03:21.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:21.201+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.202149+0000) 2020-02-05T23:03:21.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:21.201+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.202335+0000) 2020-02-05T23:03:21.204 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:21.201+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.202649+0000) 2020-02-05T23:03:21.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:21.202+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.203106+0000) 2020-02-05T23:03:21.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:21.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.203356+0000) 2020-02-05T23:03:21.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:21.203+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.204130+0000) 2020-02-05T23:03:21.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:21.203+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.204246+0000) 2020-02-05T23:03:21.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:21.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.204518+0000) 2020-02-05T23:03:21.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:21.204+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.205267+0000) 2020-02-05T23:03:21.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:21.205+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.205720+0000) 2020-02-05T23:03:21.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:21.208+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:21.209392+0000) 2020-02-05T23:03:21.214 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:21.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:22.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:22.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:22.383322+0000) 2020-02-05T23:03:22.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:22.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:22.736017+0000) 2020-02-05T23:03:22.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:22.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:22.773317+0000) 2020-02-05T23:03:23.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:23.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:23.896994+0000) 2020-02-05T23:03:23.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:23.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:23.896994+0000) 2020-02-05T23:03:25.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:25.994+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:25.996008+0000) 2020-02-05T23:03:26.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:26.219+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.220498+0000) 2020-02-05T23:03:26.222 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:26.218+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.219725+0000) 2020-02-05T23:03:26.222 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:26.219+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.220981+0000) 2020-02-05T23:03:26.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:26.220+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.221740+0000) 2020-02-05T23:03:26.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:26.220+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.221955+0000) 2020-02-05T23:03:26.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:26.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.222661+0000) 2020-02-05T23:03:26.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:26.221+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.222619+0000) 2020-02-05T23:03:26.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:26.221+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.222961+0000) 2020-02-05T23:03:26.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:26.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.223456+0000) 2020-02-05T23:03:26.225 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:26.223+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.224470+0000) 2020-02-05T23:03:26.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:26.223+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.224933+0000) 2020-02-05T23:03:26.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:26.223+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.225323+0000) 2020-02-05T23:03:26.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:26.223+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.225350+0000) 2020-02-05T23:03:26.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:26.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.229184+0000) 2020-02-05T23:03:26.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:26.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:26.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:26.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:26.273131+0000) 2020-02-05T23:03:27.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:27.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:27.136503+0000) 2020-02-05T23:03:27.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:27.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:27.136482+0000) 2020-02-05T23:03:27.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:27.135+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:27.136716+0000) 2020-02-05T23:03:28.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:28.779+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:28.780207+0000) 2020-02-05T23:03:30.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:30.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:30.130391+0000) 2020-02-05T23:03:30.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:30.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:30.367546+0000) 2020-02-05T23:03:30.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:30.381+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:30.381804+0000) 2020-02-05T23:03:31.241 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:31.239+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.240210+0000) 2020-02-05T23:03:31.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:31.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.241199+0000) 2020-02-05T23:03:31.242 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:31.240+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.241459+0000) 2020-02-05T23:03:31.243 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:31.241+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.242159+0000) 2020-02-05T23:03:31.243 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:31.241+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.242397+0000) 2020-02-05T23:03:31.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:31.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.243458+0000) 2020-02-05T23:03:31.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:31.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.243905+0000) 2020-02-05T23:03:31.255 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:31.242+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.244365+0000) 2020-02-05T23:03:31.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:31.243+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.244712+0000) 2020-02-05T23:03:31.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:31.243+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.244926+0000) 2020-02-05T23:03:31.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:31.241+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.242968+0000) 2020-02-05T23:03:31.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:31.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.244464+0000) 2020-02-05T23:03:31.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:31.249+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.250964+0000) 2020-02-05T23:03:31.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:31.242+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:31.243717+0000) 2020-02-05T23:03:31.256 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:31.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133418 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:32.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:32.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:32.383565+0000) 2020-02-05T23:03:32.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:32.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:32.736289+0000) 2020-02-05T23:03:32.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:32.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:32.773556+0000) 2020-02-05T23:03:33.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:33.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:33.897250+0000) 2020-02-05T23:03:33.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:33.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:33.897250+0000) 2020-02-05T23:03:35.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:35.995+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:35.996304+0000) 2020-02-05T23:03:36.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:36.261+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.262156+0000) 2020-02-05T23:03:36.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:36.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.262901+0000) 2020-02-05T23:03:36.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:36.263+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.264277+0000) 2020-02-05T23:03:36.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:36.263+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.265297+0000) 2020-02-05T23:03:36.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:36.264+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.265696+0000) 2020-02-05T23:03:36.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:36.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.266399+0000) 2020-02-05T23:03:36.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:36.264+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.265949+0000) 2020-02-05T23:03:36.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:36.265+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.267022+0000) 2020-02-05T23:03:36.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:36.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.267291+0000) 2020-02-05T23:03:36.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:36.265+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.266969+0000) 2020-02-05T23:03:36.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:36.265+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.267365+0000) 2020-02-05T23:03:36.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:36.266+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.267416+0000) 2020-02-05T23:03:36.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:36.266+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.267853+0000) 2020-02-05T23:03:36.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:36.268+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.269713+0000) 2020-02-05T23:03:36.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:36.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:36.273299+0000) 2020-02-05T23:03:36.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:36.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:37.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:37.135+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:37.136726+0000) 2020-02-05T23:03:37.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:37.135+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:37.136726+0000) 2020-02-05T23:03:37.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:37.135+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:37.136907+0000) 2020-02-05T23:03:38.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:38.788+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:38.788864+0000) 2020-02-05T23:03:40.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:40.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:40.130622+0000) 2020-02-05T23:03:40.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:40.366+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:40.367806+0000) 2020-02-05T23:03:40.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:40.381+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:40.381986+0000) 2020-02-05T23:03:41.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:41.283+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.284129+0000) 2020-02-05T23:03:41.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:41.283+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.284417+0000) 2020-02-05T23:03:41.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:41.283+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.285185+0000) 2020-02-05T23:03:41.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:41.284+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.285610+0000) 2020-02-05T23:03:41.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:41.284+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.285880+0000) 2020-02-05T23:03:41.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:41.284+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.286149+0000) 2020-02-05T23:03:41.288 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:41.285+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.287036+0000) 2020-02-05T23:03:41.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:41.285+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.287356+0000) 2020-02-05T23:03:41.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:41.286+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.287569+0000) 2020-02-05T23:03:41.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:41.286+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.288264+0000) 2020-02-05T23:03:41.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:41.286+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.288267+0000) 2020-02-05T23:03:41.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:41.287+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.289082+0000) 2020-02-05T23:03:41.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:41.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.290561+0000) 2020-02-05T23:03:41.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:41.290+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:41.291519+0000) 2020-02-05T23:03:41.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:41.297+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133501 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:42.328 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:03:42.334 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:03:42.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:42.382+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:42.383780+0000) 2020-02-05T23:03:42.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:42.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:42.736539+0000) 2020-02-05T23:03:42.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:42.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:42.773786+0000) 2020-02-05T23:03:43.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:43.897+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:43.897495+0000) 2020-02-05T23:03:43.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:43.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:43.897495+0000) 2020-02-05T23:03:45.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:45.995+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:45.996537+0000) 2020-02-05T23:03:46.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:46.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.273530+0000) 2020-02-05T23:03:46.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:46.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.304105+0000) 2020-02-05T23:03:46.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:46.303+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.304947+0000) 2020-02-05T23:03:46.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:46.304+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.305617+0000) 2020-02-05T23:03:46.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:46.305+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.306352+0000) 2020-02-05T23:03:46.308 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:46.307+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.308110+0000) 2020-02-05T23:03:46.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:46.307+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.308547+0000) 2020-02-05T23:03:46.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:46.307+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.309309+0000) 2020-02-05T23:03:46.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:46.307+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.309081+0000) 2020-02-05T23:03:46.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:46.308+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.310019+0000) 2020-02-05T23:03:46.311 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:46.309+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.310756+0000) 2020-02-05T23:03:46.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:46.309+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.310922+0000) 2020-02-05T23:03:46.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:46.309+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.311031+0000) 2020-02-05T23:03:46.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:46.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.311597+0000) 2020-02-05T23:03:46.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:46.314+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:46.315521+0000) 2020-02-05T23:03:46.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:46.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133543 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:47.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:47.136+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:47.136974+0000) 2020-02-05T23:03:47.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:47.136+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:47.136958+0000) 2020-02-05T23:03:47.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:47.136+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:47.137126+0000) 2020-02-05T23:03:48.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:48.797+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:48.797135+0000) 2020-02-05T23:03:50.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:50.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:50.130895+0000) 2020-02-05T23:03:50.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:50.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:50.368105+0000) 2020-02-05T23:03:50.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:50.382+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:50.382258+0000) 2020-02-05T23:03:51.327 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:51.326+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.326177+0000) 2020-02-05T23:03:51.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:51.326+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.326768+0000) 2020-02-05T23:03:51.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:51.327+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.327257+0000) 2020-02-05T23:03:51.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:51.326+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.327835+0000) 2020-02-05T23:03:51.329 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:51.328+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.328300+0000) 2020-02-05T23:03:51.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:51.328+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.328559+0000) 2020-02-05T23:03:51.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:51.328+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.328801+0000) 2020-02-05T23:03:51.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:51.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.329403+0000) 2020-02-05T23:03:51.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:51.328+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.330241+0000) 2020-02-05T23:03:51.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:51.329+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.330747+0000) 2020-02-05T23:03:51.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:51.330+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.331940+0000) 2020-02-05T23:03:51.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:51.330+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.332224+0000) 2020-02-05T23:03:51.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:51.331+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.332455+0000) 2020-02-05T23:03:51.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:51.336+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:51.336650+0000) 2020-02-05T23:03:51.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:51.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:52.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:52.383+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:52.384031+0000) 2020-02-05T23:03:52.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:52.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:52.736795+0000) 2020-02-05T23:03:52.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:52.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:52.774031+0000) 2020-02-05T23:03:53.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:53.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:53.897766+0000) 2020-02-05T23:03:53.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:53.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:53.897766+0000) 2020-02-05T23:03:55.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:55.996+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:55.996815+0000) 2020-02-05T23:03:56.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:56.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.273773+0000) 2020-02-05T23:03:56.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:03:56.346+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.347522+0000) 2020-02-05T23:03:56.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:56.346+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.347891+0000) 2020-02-05T23:03:56.349 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:03:56.347+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.348411+0000) 2020-02-05T23:03:56.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:56.347+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.348808+0000) 2020-02-05T23:03:56.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:56.348+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.349160+0000) 2020-02-05T23:03:56.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:03:56.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.349414+0000) 2020-02-05T23:03:56.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:03:56.349+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.350364+0000) 2020-02-05T23:03:56.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:03:56.349+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.350581+0000) 2020-02-05T23:03:56.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:03:56.348+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.350407+0000) 2020-02-05T23:03:56.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:56.349+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.350680+0000) 2020-02-05T23:03:56.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:03:56.350+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.351199+0000) 2020-02-05T23:03:56.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:56.350+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.351579+0000) 2020-02-05T23:03:56.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:03:56.351+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.352793+0000) 2020-02-05T23:03:56.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:56.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:56.356397+0000) 2020-02-05T23:03:56.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:03:56.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:03:57.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:03:57.136+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:57.137178+0000) 2020-02-05T23:03:57.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:03:57.136+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:57.137168+0000) 2020-02-05T23:03:57.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:03:57.136+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:57.137322+0000) 2020-02-05T23:03:58.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:03:58.804+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:03:58.805439+0000) 2020-02-05T23:04:00.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:00.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:00.131130+0000) 2020-02-05T23:04:00.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:00.367+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:00.368472+0000) 2020-02-05T23:04:00.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:00.381+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:00.382491+0000) 2020-02-05T23:04:01.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:01.364+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.365523+0000) 2020-02-05T23:04:01.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:01.364+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.365886+0000) 2020-02-05T23:04:01.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:01.365+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.366902+0000) 2020-02-05T23:04:01.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:01.365+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.366655+0000) 2020-02-05T23:04:01.368 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:01.366+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.367376+0000) 2020-02-05T23:04:01.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:01.368+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.369559+0000) 2020-02-05T23:04:01.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:01.368+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.370337+0000) 2020-02-05T23:04:01.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:01.369+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.370765+0000) 2020-02-05T23:04:01.373 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:01.370+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.371417+0000) 2020-02-05T23:04:01.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:01.370+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.371924+0000) 2020-02-05T23:04:01.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:01.370+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.372246+0000) 2020-02-05T23:04:01.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:01.371+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.373040+0000) 2020-02-05T23:04:01.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:01.371+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.372821+0000) 2020-02-05T23:04:01.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:01.379+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:01.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:01.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:01.377640+0000) 2020-02-05T23:04:02.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:02.383+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:02.384293+0000) 2020-02-05T23:04:02.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:02.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:02.737026+0000) 2020-02-05T23:04:02.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:02.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:02.774299+0000) 2020-02-05T23:04:03.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:03.897+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:03.898033+0000) 2020-02-05T23:04:03.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:03.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:03.898033+0000) 2020-02-05T23:04:05.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:05.996+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:05.997060+0000) 2020-02-05T23:04:06.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:06.272+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.274027+0000) 2020-02-05T23:04:06.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:06.386+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.387290+0000) 2020-02-05T23:04:06.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:06.386+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.387826+0000) 2020-02-05T23:04:06.390 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:06.387+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.388741+0000) 2020-02-05T23:04:06.390 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:06.387+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.388339+0000) 2020-02-05T23:04:06.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:06.388+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.389204+0000) 2020-02-05T23:04:06.391 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:06.388+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.389612+0000) 2020-02-05T23:04:06.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:06.388+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.390208+0000) 2020-02-05T23:04:06.391 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:06.389+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.390225+0000) 2020-02-05T23:04:06.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:06.389+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.390594+0000) 2020-02-05T23:04:06.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:06.389+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.391239+0000) 2020-02-05T23:04:06.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:06.390+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.391631+0000) 2020-02-05T23:04:06.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:06.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.392652+0000) 2020-02-05T23:04:06.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:06.391+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.392812+0000) 2020-02-05T23:04:06.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:06.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:06.397726+0000) 2020-02-05T23:04:06.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:06.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:07.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:07.136+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:07.137415+0000) 2020-02-05T23:04:07.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:07.136+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:07.137422+0000) 2020-02-05T23:04:07.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:07.136+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:07.137597+0000) 2020-02-05T23:04:08.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:08.812+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:08.813546+0000) 2020-02-05T23:04:10.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:10.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:10.131357+0000) 2020-02-05T23:04:10.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:10.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:10.368655+0000) 2020-02-05T23:04:10.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:10.381+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:10.382716+0000) 2020-02-05T23:04:11.407 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:11.405+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.406647+0000) 2020-02-05T23:04:11.409 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:11.407+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.408373+0000) 2020-02-05T23:04:11.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:11.408+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.409149+0000) 2020-02-05T23:04:11.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:11.408+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.409825+0000) 2020-02-05T23:04:11.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:11.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.409946+0000) 2020-02-05T23:04:11.411 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:11.409+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.410427+0000) 2020-02-05T23:04:11.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:11.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.410746+0000) 2020-02-05T23:04:11.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:11.410+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.411490+0000) 2020-02-05T23:04:11.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:11.411+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.412501+0000) 2020-02-05T23:04:11.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:11.412+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.413181+0000) 2020-02-05T23:04:11.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:11.411+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.413228+0000) 2020-02-05T23:04:11.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:11.412+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.413580+0000) 2020-02-05T23:04:11.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:11.412+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.413571+0000) 2020-02-05T23:04:11.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:11.419+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:11.420274+0000) 2020-02-05T23:04:11.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:11.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:12.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:12.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:12.384543+0000) 2020-02-05T23:04:12.476 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:04:12.482 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:04:12.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:12.736+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:12.737309+0000) 2020-02-05T23:04:12.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:12.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:12.774570+0000) 2020-02-05T23:04:13.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:13.897+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:13.898350+0000) 2020-02-05T23:04:13.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:13.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:13.898348+0000) 2020-02-05T23:04:15.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:15.996+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:15.997309+0000) 2020-02-05T23:04:16.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:16.273+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.274326+0000) 2020-02-05T23:04:16.429 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:16.426+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.427949+0000) 2020-02-05T23:04:16.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:16.427+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.428428+0000) 2020-02-05T23:04:16.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:16.428+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.428991+0000) 2020-02-05T23:04:16.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:16.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.429192+0000) 2020-02-05T23:04:16.431 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:16.428+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.429445+0000) 2020-02-05T23:04:16.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:16.429+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.430163+0000) 2020-02-05T23:04:16.432 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:16.431+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.431763+0000) 2020-02-05T23:04:16.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:16.432+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.432856+0000) 2020-02-05T23:04:16.435 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:16.433+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.434188+0000) 2020-02-05T23:04:16.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:16.434+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.434654+0000) 2020-02-05T23:04:16.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:16.434+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.434863+0000) 2020-02-05T23:04:16.436 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:16.433+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.434789+0000) 2020-02-05T23:04:16.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:16.437+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.438941+0000) 2020-02-05T23:04:16.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:16.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:16.439232+0000) 2020-02-05T23:04:16.445 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:16.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:17.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:17.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:17.137726+0000) 2020-02-05T23:04:17.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:17.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:17.137724+0000) 2020-02-05T23:04:17.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:17.137+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:17.137844+0000) 2020-02-05T23:04:18.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:18.820+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:18.822050+0000) 2020-02-05T23:04:20.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:20.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:20.131588+0000) 2020-02-05T23:04:20.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:20.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:20.368925+0000) 2020-02-05T23:04:20.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:20.382+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:20.382942+0000) 2020-02-05T23:04:21.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:21.449+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.450372+0000) 2020-02-05T23:04:21.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:21.450+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.451256+0000) 2020-02-05T23:04:21.452 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:21.450+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.451662+0000) 2020-02-05T23:04:21.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:21.451+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.452989+0000) 2020-02-05T23:04:21.454 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:21.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.453222+0000) 2020-02-05T23:04:21.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:21.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.454413+0000) 2020-02-05T23:04:21.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:21.453+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.453804+0000) 2020-02-05T23:04:21.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:21.453+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.454116+0000) 2020-02-05T23:04:21.456 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:21.454+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.455142+0000) 2020-02-05T23:04:21.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:21.454+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.455279+0000) 2020-02-05T23:04:21.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:21.455+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.455907+0000) 2020-02-05T23:04:21.458 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:21.455+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.456059+0000) 2020-02-05T23:04:21.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:21.455+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.456369+0000) 2020-02-05T23:04:21.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:21.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:21.460368+0000) 2020-02-05T23:04:21.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:21.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:22.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:22.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:22.384779+0000) 2020-02-05T23:04:22.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:22.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:22.737586+0000) 2020-02-05T23:04:22.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:22.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:22.774821+0000) 2020-02-05T23:04:23.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:23.898+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:23.898622+0000) 2020-02-05T23:04:23.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:23.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:23.898622+0000) 2020-02-05T23:04:25.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:25.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:25.997541+0000) 2020-02-05T23:04:26.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:26.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.274557+0000) 2020-02-05T23:04:26.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:26.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.472573+0000) 2020-02-05T23:04:26.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:26.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.472945+0000) 2020-02-05T23:04:26.474 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:26.472+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.473374+0000) 2020-02-05T23:04:26.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:26.472+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.474020+0000) 2020-02-05T23:04:26.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:26.473+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.474067+0000) 2020-02-05T23:04:26.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:26.474+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.474594+0000) 2020-02-05T23:04:26.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:26.473+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.475017+0000) 2020-02-05T23:04:26.476 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:26.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.475414+0000) 2020-02-05T23:04:26.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:26.474+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.475608+0000) 2020-02-05T23:04:26.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:26.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.476056+0000) 2020-02-05T23:04:26.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:26.476+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.476483+0000) 2020-02-05T23:04:26.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:26.476+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.476592+0000) 2020-02-05T23:04:26.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:26.476+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.477168+0000) 2020-02-05T23:04:26.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:26.481+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:26.483055+0000) 2020-02-05T23:04:26.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:26.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:27.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:27.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:27.137954+0000) 2020-02-05T23:04:27.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:27.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:27.137955+0000) 2020-02-05T23:04:27.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:27.137+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:27.138097+0000) 2020-02-05T23:04:28.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:28.828+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:28.828968+0000) 2020-02-05T23:04:30.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:30.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:30.131821+0000) 2020-02-05T23:04:30.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:30.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:30.369211+0000) 2020-02-05T23:04:30.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:30.382+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:30.383193+0000) 2020-02-05T23:04:31.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:31.494+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.494531+0000) 2020-02-05T23:04:31.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:31.493+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.494768+0000) 2020-02-05T23:04:31.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:31.494+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.495379+0000) 2020-02-05T23:04:31.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:31.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.495806+0000) 2020-02-05T23:04:31.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:31.495+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.496189+0000) 2020-02-05T23:04:31.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:31.496+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.496934+0000) 2020-02-05T23:04:31.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:31.496+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.497145+0000) 2020-02-05T23:04:31.498 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:31.497+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.497872+0000) 2020-02-05T23:04:31.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:31.498+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.498535+0000) 2020-02-05T23:04:31.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:31.498+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.499409+0000) 2020-02-05T23:04:31.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:31.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.499575+0000) 2020-02-05T23:04:31.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:31.500+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.500611+0000) 2020-02-05T23:04:31.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:31.502+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.504048+0000) 2020-02-05T23:04:31.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:31.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:31.504300+0000) 2020-02-05T23:04:31.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:31.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133917 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:32.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:32.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:32.385008+0000) 2020-02-05T23:04:32.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:32.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:32.737813+0000) 2020-02-05T23:04:32.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:32.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:32.775105+0000) 2020-02-05T23:04:33.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:33.898+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:33.898890+0000) 2020-02-05T23:04:33.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:33.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:33.898890+0000) 2020-02-05T23:04:35.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:35.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:35.997775+0000) 2020-02-05T23:04:36.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:36.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.274806+0000) 2020-02-05T23:04:36.518 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:36.516+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.516665+0000) 2020-02-05T23:04:36.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:36.516+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.517229+0000) 2020-02-05T23:04:36.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:36.517+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.517747+0000) 2020-02-05T23:04:36.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:36.517+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.518726+0000) 2020-02-05T23:04:36.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:36.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.519574+0000) 2020-02-05T23:04:36.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:36.519+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.519523+0000) 2020-02-05T23:04:36.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:36.519+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.520194+0000) 2020-02-05T23:04:36.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:36.520+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.520635+0000) 2020-02-05T23:04:36.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:36.519+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.520883+0000) 2020-02-05T23:04:36.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:36.519+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.521020+0000) 2020-02-05T23:04:36.522 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:36.520+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.521177+0000) 2020-02-05T23:04:36.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:36.521+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.521853+0000) 2020-02-05T23:04:36.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:36.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.522504+0000) 2020-02-05T23:04:36.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:36.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:36.526018+0000) 2020-02-05T23:04:36.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:36.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 133958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:37.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:37.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:37.138191+0000) 2020-02-05T23:04:37.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:37.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:37.138191+0000) 2020-02-05T23:04:37.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:37.137+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:37.138330+0000) 2020-02-05T23:04:38.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:38.837+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:38.837357+0000) 2020-02-05T23:04:40.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:40.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:40.132135+0000) 2020-02-05T23:04:40.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:40.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:40.369395+0000) 2020-02-05T23:04:40.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:40.383+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:40.383424+0000) 2020-02-05T23:04:41.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:41.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.536452+0000) 2020-02-05T23:04:41.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:41.537+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.538681+0000) 2020-02-05T23:04:41.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:41.538+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.539251+0000) 2020-02-05T23:04:41.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:41.539+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.540011+0000) 2020-02-05T23:04:41.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:41.540+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.540416+0000) 2020-02-05T23:04:41.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:41.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.541372+0000) 2020-02-05T23:04:41.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:41.540+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.541643+0000) 2020-02-05T23:04:41.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:41.541+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.542066+0000) 2020-02-05T23:04:41.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:41.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.542820+0000) 2020-02-05T23:04:41.543 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:41.542+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.543122+0000) 2020-02-05T23:04:41.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:41.542+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.543393+0000) 2020-02-05T23:04:41.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:41.543+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.543624+0000) 2020-02-05T23:04:41.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:41.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.543824+0000) 2020-02-05T23:04:41.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:41.545+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:41.546590+0000) 2020-02-05T23:04:41.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:41.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:42.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:42.385+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:42.385232+0000) 2020-02-05T23:04:42.640 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:04:42.646 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:04:42.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:42.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:42.738017+0000) 2020-02-05T23:04:42.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:42.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:42.775384+0000) 2020-02-05T23:04:43.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:43.899+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:43.899186+0000) 2020-02-05T23:04:43.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:43.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:43.899183+0000) 2020-02-05T23:04:45.999 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:45.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:45.998029+0000) 2020-02-05T23:04:46.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:46.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.275048+0000) 2020-02-05T23:04:46.560 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:46.559+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.559283+0000) 2020-02-05T23:04:46.561 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:46.559+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.559958+0000) 2020-02-05T23:04:46.561 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:46.559+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.560406+0000) 2020-02-05T23:04:46.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:46.560+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.560868+0000) 2020-02-05T23:04:46.562 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:46.561+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.561283+0000) 2020-02-05T23:04:46.562 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:46.561+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.561939+0000) 2020-02-05T23:04:46.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:46.562+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.562303+0000) 2020-02-05T23:04:46.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:46.562+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.562815+0000) 2020-02-05T23:04:46.564 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:46.563+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.563188+0000) 2020-02-05T23:04:46.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:46.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.563998+0000) 2020-02-05T23:04:46.565 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:46.564+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.564578+0000) 2020-02-05T23:04:46.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:46.564+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.564911+0000) 2020-02-05T23:04:46.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:46.564+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.565253+0000) 2020-02-05T23:04:46.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:46.568+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:46.568994+0000) 2020-02-05T23:04:46.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:46.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:47.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:47.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:47.138364+0000) 2020-02-05T23:04:47.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:47.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:47.138403+0000) 2020-02-05T23:04:47.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:47.138+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:47.138602+0000) 2020-02-05T23:04:48.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:48.848+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:48.849800+0000) 2020-02-05T23:04:50.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:50.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:50.132325+0000) 2020-02-05T23:04:50.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:50.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:50.369629+0000) 2020-02-05T23:04:50.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:50.382+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:50.383659+0000) 2020-02-05T23:04:51.579 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:51.577+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.578379+0000) 2020-02-05T23:04:51.580 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:51.578+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.579708+0000) 2020-02-05T23:04:51.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:51.579+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.580120+0000) 2020-02-05T23:04:51.581 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:51.579+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.580686+0000) 2020-02-05T23:04:51.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:51.579+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.581393+0000) 2020-02-05T23:04:51.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:51.580+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.581977+0000) 2020-02-05T23:04:51.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:51.581+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.582334+0000) 2020-02-05T23:04:51.583 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:51.581+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.582702+0000) 2020-02-05T23:04:51.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:51.581+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.583154+0000) 2020-02-05T23:04:51.584 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:51.582+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.583685+0000) 2020-02-05T23:04:51.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:51.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.584456+0000) 2020-02-05T23:04:51.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:51.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.584969+0000) 2020-02-05T23:04:51.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:51.584+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.585540+0000) 2020-02-05T23:04:51.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:51.588+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:51.589672+0000) 2020-02-05T23:04:51.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:51.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:52.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:52.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:52.385489+0000) 2020-02-05T23:04:52.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:52.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:52.738253+0000) 2020-02-05T23:04:52.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:52.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:52.775643+0000) 2020-02-05T23:04:53.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:53.898+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:53.899461+0000) 2020-02-05T23:04:53.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:53.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:53.899461+0000) 2020-02-05T23:04:55.999 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:55.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:55.998281+0000) 2020-02-05T23:04:56.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:56.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.275345+0000) 2020-02-05T23:04:56.600 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:04:56.598+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.599224+0000) 2020-02-05T23:04:56.602 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:04:56.600+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.601405+0000) 2020-02-05T23:04:56.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:04:56.599+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.600943+0000) 2020-02-05T23:04:56.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:04:56.600+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.601727+0000) 2020-02-05T23:04:56.603 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:04:56.600+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.601911+0000) 2020-02-05T23:04:56.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:56.600+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.602194+0000) 2020-02-05T23:04:56.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:04:56.601+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.602387+0000) 2020-02-05T23:04:56.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:56.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.603221+0000) 2020-02-05T23:04:56.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:56.602+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.603632+0000) 2020-02-05T23:04:56.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:04:56.602+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.604207+0000) 2020-02-05T23:04:56.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:56.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.604959+0000) 2020-02-05T23:04:56.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:04:56.603+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.605044+0000) 2020-02-05T23:04:56.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:56.603+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.605248+0000) 2020-02-05T23:04:56.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:56.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:56.610786+0000) 2020-02-05T23:04:56.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:04:56.613+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134126 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:04:57.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:04:57.137+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:57.138626+0000) 2020-02-05T23:04:57.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:04:57.137+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:57.138675+0000) 2020-02-05T23:04:57.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:04:57.137+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:57.138886+0000) 2020-02-05T23:04:58.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:04:58.857+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:04:58.858066+0000) 2020-02-05T23:05:00.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:00.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:00.132585+0000) 2020-02-05T23:05:00.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:00.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:00.369877+0000) 2020-02-05T23:05:00.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:00.383+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:00.383884+0000) 2020-02-05T23:05:01.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:01.620+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.622048+0000) 2020-02-05T23:05:01.623 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:01.621+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.622577+0000) 2020-02-05T23:05:01.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:01.622+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.623657+0000) 2020-02-05T23:05:01.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:01.623+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.624433+0000) 2020-02-05T23:05:01.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:01.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.625020+0000) 2020-02-05T23:05:01.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:01.624+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.625486+0000) 2020-02-05T23:05:01.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:01.624+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.626188+0000) 2020-02-05T23:05:01.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:01.625+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.626589+0000) 2020-02-05T23:05:01.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:01.625+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.626861+0000) 2020-02-05T23:05:01.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:01.625+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.627182+0000) 2020-02-05T23:05:01.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:01.625+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.627041+0000) 2020-02-05T23:05:01.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:01.626+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.627519+0000) 2020-02-05T23:05:01.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:01.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.627019+0000) 2020-02-05T23:05:01.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:01.629+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:01.630554+0000) 2020-02-05T23:05:01.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:01.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:02.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:02.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:02.385735+0000) 2020-02-05T23:05:02.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:02.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:02.738472+0000) 2020-02-05T23:05:02.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:02.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:02.775860+0000) 2020-02-05T23:05:03.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:03.898+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:03.899726+0000) 2020-02-05T23:05:03.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:03.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:03.899726+0000) 2020-02-05T23:05:06.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:05.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:05.998573+0000) 2020-02-05T23:05:06.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:06.274+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.275579+0000) 2020-02-05T23:05:06.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:06.641+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.642423+0000) 2020-02-05T23:05:06.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:06.641+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.643377+0000) 2020-02-05T23:05:06.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:06.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.644622+0000) 2020-02-05T23:05:06.646 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:06.644+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.645112+0000) 2020-02-05T23:05:06.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:06.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.645604+0000) 2020-02-05T23:05:06.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:06.644+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.645937+0000) 2020-02-05T23:05:06.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:06.644+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.646264+0000) 2020-02-05T23:05:06.647 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:06.645+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.646809+0000) 2020-02-05T23:05:06.659 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:06.646+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.647565+0000) 2020-02-05T23:05:06.660 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:06.646+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.648383+0000) 2020-02-05T23:05:06.660 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:06.647+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.648679+0000) 2020-02-05T23:05:06.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:06.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134210 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:06.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:06.646+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.647388+0000) 2020-02-05T23:05:06.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:06.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.650339+0000) 2020-02-05T23:05:06.660 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:06.647+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:06.648174+0000) 2020-02-05T23:05:07.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:07.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:07.138900+0000) 2020-02-05T23:05:07.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:07.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:07.138934+0000) 2020-02-05T23:05:07.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:07.138+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:07.139083+0000) 2020-02-05T23:05:08.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:08.865+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:08.866390+0000) 2020-02-05T23:05:10.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:10.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:10.132841+0000) 2020-02-05T23:05:10.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:10.368+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:10.370088+0000) 2020-02-05T23:05:10.385 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:10.383+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:10.384118+0000) 2020-02-05T23:05:11.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:11.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.664253+0000) 2020-02-05T23:05:11.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:11.663+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.664490+0000) 2020-02-05T23:05:11.666 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:11.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.664924+0000) 2020-02-05T23:05:11.666 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:11.664+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.665880+0000) 2020-02-05T23:05:11.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:11.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.666164+0000) 2020-02-05T23:05:11.667 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:11.665+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.666577+0000) 2020-02-05T23:05:11.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:11.666+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.667419+0000) 2020-02-05T23:05:11.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:11.666+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.667522+0000) 2020-02-05T23:05:11.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:11.666+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.668049+0000) 2020-02-05T23:05:11.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:11.666+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.668394+0000) 2020-02-05T23:05:11.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:11.667+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.669179+0000) 2020-02-05T23:05:11.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:11.668+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.669356+0000) 2020-02-05T23:05:11.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:11.668+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.669775+0000) 2020-02-05T23:05:11.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:11.674+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:11.675243+0000) 2020-02-05T23:05:11.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:11.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:12.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:12.385+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:12.385960+0000) 2020-02-05T23:05:12.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:12.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:12.738722+0000) 2020-02-05T23:05:12.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:12.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:12.776122+0000) 2020-02-05T23:05:12.853 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:05:12.859 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:05:13.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:13.899+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:13.900001+0000) 2020-02-05T23:05:13.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:13.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:13.900001+0000) 2020-02-05T23:05:16.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:15.998+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:15.998809+0000) 2020-02-05T23:05:16.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:16.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.275809+0000) 2020-02-05T23:05:16.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:16.684+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.685736+0000) 2020-02-05T23:05:16.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:16.685+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.686928+0000) 2020-02-05T23:05:16.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:16.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.687891+0000) 2020-02-05T23:05:16.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:16.687+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.688538+0000) 2020-02-05T23:05:16.690 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:16.686+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.688345+0000) 2020-02-05T23:05:16.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:16.688+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.689425+0000) 2020-02-05T23:05:16.690 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:16.688+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.689777+0000) 2020-02-05T23:05:16.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:16.688+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.690186+0000) 2020-02-05T23:05:16.692 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:16.689+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.690728+0000) 2020-02-05T23:05:16.693 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:16.689+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.690852+0000) 2020-02-05T23:05:16.693 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:16.690+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.692365+0000) 2020-02-05T23:05:16.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:16.691+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.692492+0000) 2020-02-05T23:05:16.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:16.693+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.694844+0000) 2020-02-05T23:05:16.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:16.693+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:16.695056+0000) 2020-02-05T23:05:16.700 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:16.698+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134292 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:17.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:17.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:17.139122+0000) 2020-02-05T23:05:17.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:17.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:17.139223+0000) 2020-02-05T23:05:17.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:17.138+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:17.139333+0000) 2020-02-05T23:05:18.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:18.873+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:18.874799+0000) 2020-02-05T23:05:20.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:20.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:20.133068+0000) 2020-02-05T23:05:20.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:20.369+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:20.370344+0000) 2020-02-05T23:05:20.385 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:20.383+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:20.384380+0000) 2020-02-05T23:05:21.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:21.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.705937+0000) 2020-02-05T23:05:21.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:21.705+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.706326+0000) 2020-02-05T23:05:21.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:21.706+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.707217+0000) 2020-02-05T23:05:21.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:21.706+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.707661+0000) 2020-02-05T23:05:21.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:21.706+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.708026+0000) 2020-02-05T23:05:21.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:21.707+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.708344+0000) 2020-02-05T23:05:21.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:21.707+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.709014+0000) 2020-02-05T23:05:21.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:21.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.709455+0000) 2020-02-05T23:05:21.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:21.708+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.710246+0000) 2020-02-05T23:05:21.711 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:21.709+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.711102+0000) 2020-02-05T23:05:21.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:21.710+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.711631+0000) 2020-02-05T23:05:21.712 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:21.710+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.711909+0000) 2020-02-05T23:05:21.713 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:21.710+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.712174+0000) 2020-02-05T23:05:21.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:21.717+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:21.718204+0000) 2020-02-05T23:05:21.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:21.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:22.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:22.384+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:22.386156+0000) 2020-02-05T23:05:22.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:22.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:22.738964+0000) 2020-02-05T23:05:22.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:22.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:22.776375+0000) 2020-02-05T23:05:23.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:23.899+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:23.900318+0000) 2020-02-05T23:05:23.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:23.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:23.900316+0000) 2020-02-05T23:05:26.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:25.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:25.999097+0000) 2020-02-05T23:05:26.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:26.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.276047+0000) 2020-02-05T23:05:26.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:26.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.724879+0000) 2020-02-05T23:05:26.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:26.724+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.725396+0000) 2020-02-05T23:05:26.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:26.726+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.727197+0000) 2020-02-05T23:05:26.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:26.726+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.727736+0000) 2020-02-05T23:05:26.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:26.726+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.728306+0000) 2020-02-05T23:05:26.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:26.727+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.728728+0000) 2020-02-05T23:05:26.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:26.728+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.729913+0000) 2020-02-05T23:05:26.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:26.728+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.730247+0000) 2020-02-05T23:05:26.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:26.729+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.731112+0000) 2020-02-05T23:05:26.732 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:26.730+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.731286+0000) 2020-02-05T23:05:26.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:26.729+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.731321+0000) 2020-02-05T23:05:26.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:26.730+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.731603+0000) 2020-02-05T23:05:26.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:26.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.734596+0000) 2020-02-05T23:05:26.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:26.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:26.734810+0000) 2020-02-05T23:05:26.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:26.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:27.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:27.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:27.139334+0000) 2020-02-05T23:05:27.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:27.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:27.139513+0000) 2020-02-05T23:05:27.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:27.138+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:27.139591+0000) 2020-02-05T23:05:28.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:28.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:28.883283+0000) 2020-02-05T23:05:30.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:30.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:30.133327+0000) 2020-02-05T23:05:30.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:30.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:30.370680+0000) 2020-02-05T23:05:30.385 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:30.384+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:30.384606+0000) 2020-02-05T23:05:31.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:31.745+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.746779+0000) 2020-02-05T23:05:31.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:31.746+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.747422+0000) 2020-02-05T23:05:31.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:31.746+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.748061+0000) 2020-02-05T23:05:31.749 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:31.747+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.748631+0000) 2020-02-05T23:05:31.759 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:31.748+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.750258+0000) 2020-02-05T23:05:31.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:31.749+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.750846+0000) 2020-02-05T23:05:31.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:31.750+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.751680+0000) 2020-02-05T23:05:31.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:31.750+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.751958+0000) 2020-02-05T23:05:31.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:31.751+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.752532+0000) 2020-02-05T23:05:31.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:31.748+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.749338+0000) 2020-02-05T23:05:31.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:31.748+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.749802+0000) 2020-02-05T23:05:31.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:31.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.751264+0000) 2020-02-05T23:05:31.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:31.756+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.757310+0000) 2020-02-05T23:05:31.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:31.750+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:31.752029+0000) 2020-02-05T23:05:31.761 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:31.759+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:32.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:32.385+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:32.386388+0000) 2020-02-05T23:05:32.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:32.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:32.739197+0000) 2020-02-05T23:05:32.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:32.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:32.776607+0000) 2020-02-05T23:05:33.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:33.899+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:33.900614+0000) 2020-02-05T23:05:33.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:33.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:33.900612+0000) 2020-02-05T23:05:36.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:35.997+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:35.999336+0000) 2020-02-05T23:05:36.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:36.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.276334+0000) 2020-02-05T23:05:36.767 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:36.764+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.765672+0000) 2020-02-05T23:05:36.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:36.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.766455+0000) 2020-02-05T23:05:36.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:36.767+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.767152+0000) 2020-02-05T23:05:36.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:36.767+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.767799+0000) 2020-02-05T23:05:36.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:36.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.769195+0000) 2020-02-05T23:05:36.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:36.768+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.769695+0000) 2020-02-05T23:05:36.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:36.767+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.768701+0000) 2020-02-05T23:05:36.771 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:36.768+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.770192+0000) 2020-02-05T23:05:36.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:36.769+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.770276+0000) 2020-02-05T23:05:36.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:36.769+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.770454+0000) 2020-02-05T23:05:36.773 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:36.770+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.771772+0000) 2020-02-05T23:05:36.773 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:36.770+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.772147+0000) 2020-02-05T23:05:36.773 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:36.771+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.772485+0000) 2020-02-05T23:05:36.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:36.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:36.775198+0000) 2020-02-05T23:05:36.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:36.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:37.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:37.138+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:37.139540+0000) 2020-02-05T23:05:37.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:37.138+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:37.139773+0000) 2020-02-05T23:05:37.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:37.138+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:37.139820+0000) 2020-02-05T23:05:38.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:38.895+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:38.895429+0000) 2020-02-05T23:05:40.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:40.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:40.133584+0000) 2020-02-05T23:05:40.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:40.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:40.370880+0000) 2020-02-05T23:05:40.385 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:40.384+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:40.384839+0000) 2020-02-05T23:05:41.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:41.787+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.788592+0000) 2020-02-05T23:05:41.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:41.788+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.789194+0000) 2020-02-05T23:05:41.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:41.789+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.790264+0000) 2020-02-05T23:05:41.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:41.789+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.790642+0000) 2020-02-05T23:05:41.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:41.789+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.790880+0000) 2020-02-05T23:05:41.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:41.790+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.791552+0000) 2020-02-05T23:05:41.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:41.790+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.791898+0000) 2020-02-05T23:05:41.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:41.791+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.792467+0000) 2020-02-05T23:05:41.794 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:41.791+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.793251+0000) 2020-02-05T23:05:41.794 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:41.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.793419+0000) 2020-02-05T23:05:41.795 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:41.792+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.793963+0000) 2020-02-05T23:05:41.795 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:41.792+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.794365+0000) 2020-02-05T23:05:41.795 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:41.793+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.794756+0000) 2020-02-05T23:05:41.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:41.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:41.798335+0000) 2020-02-05T23:05:41.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:41.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:42.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:42.386+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:42.386662+0000) 2020-02-05T23:05:42.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:42.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:42.739450+0000) 2020-02-05T23:05:42.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:42.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:42.776842+0000) 2020-02-05T23:05:43.095 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:05:43.101 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:05:43.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:43.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:43.900905+0000) 2020-02-05T23:05:43.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:43.899+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:43.900906+0000) 2020-02-05T23:05:46.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:45.999+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:45.999589+0000) 2020-02-05T23:05:46.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:46.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.276594+0000) 2020-02-05T23:05:46.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:46.808+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.810012+0000) 2020-02-05T23:05:46.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:46.809+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.810492+0000) 2020-02-05T23:05:46.812 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:46.810+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.811176+0000) 2020-02-05T23:05:46.813 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:46.811+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.811919+0000) 2020-02-05T23:05:46.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:46.811+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.812499+0000) 2020-02-05T23:05:46.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:46.811+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.812891+0000) 2020-02-05T23:05:46.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:46.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.813688+0000) 2020-02-05T23:05:46.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:46.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.813954+0000) 2020-02-05T23:05:46.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:46.813+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.814500+0000) 2020-02-05T23:05:46.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:46.814+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.815309+0000) 2020-02-05T23:05:46.816 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:46.815+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.815768+0000) 2020-02-05T23:05:46.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:46.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.815988+0000) 2020-02-05T23:05:46.817 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:46.816+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.816885+0000) 2020-02-05T23:05:46.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:46.819+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:46.820621+0000) 2020-02-05T23:05:46.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:46.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134536 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:47.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:47.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:47.139762+0000) 2020-02-05T23:05:47.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:47.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:47.139998+0000) 2020-02-05T23:05:47.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:47.139+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:47.140116+0000) 2020-02-05T23:05:48.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:48.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:48.903855+0000) 2020-02-05T23:05:50.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:50.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:50.133858+0000) 2020-02-05T23:05:50.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:50.370+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:50.371119+0000) 2020-02-05T23:05:50.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:50.383+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:50.385056+0000) 2020-02-05T23:05:51.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:51.831+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.832280+0000) 2020-02-05T23:05:51.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:51.832+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.833108+0000) 2020-02-05T23:05:51.835 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:51.833+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.833975+0000) 2020-02-05T23:05:51.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:51.834+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.835126+0000) 2020-02-05T23:05:51.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:51.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.835515+0000) 2020-02-05T23:05:51.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:51.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.836532+0000) 2020-02-05T23:05:51.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:51.835+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.836106+0000) 2020-02-05T23:05:51.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:51.836+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.836768+0000) 2020-02-05T23:05:51.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:51.836+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.837128+0000) 2020-02-05T23:05:51.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:51.836+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.837467+0000) 2020-02-05T23:05:51.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:51.837+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.837624+0000) 2020-02-05T23:05:51.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:51.837+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.838009+0000) 2020-02-05T23:05:51.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:51.838+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.839323+0000) 2020-02-05T23:05:51.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:51.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:51.842754+0000) 2020-02-05T23:05:51.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:51.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:52.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:52.386+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:52.386904+0000) 2020-02-05T23:05:52.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:52.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:52.739688+0000) 2020-02-05T23:05:52.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:52.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:52.777142+0000) 2020-02-05T23:05:53.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:53.900+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:53.901154+0000) 2020-02-05T23:05:53.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:53.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:53.901156+0000) 2020-02-05T23:05:56.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:55.999+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:55.999859+0000) 2020-02-05T23:05:56.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:56.275+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.276848+0000) 2020-02-05T23:05:56.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:56.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.854796+0000) 2020-02-05T23:05:56.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:05:56.854+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.856053+0000) 2020-02-05T23:05:56.857 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:05:56.855+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.856358+0000) 2020-02-05T23:05:56.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:56.855+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.856704+0000) 2020-02-05T23:05:56.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:56.855+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.857063+0000) 2020-02-05T23:05:56.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:05:56.857+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.857532+0000) 2020-02-05T23:05:56.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:05:56.857+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.858331+0000) 2020-02-05T23:05:56.860 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:56.858+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.858712+0000) 2020-02-05T23:05:56.860 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:05:56.858+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.859310+0000) 2020-02-05T23:05:56.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:05:56.858+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.859731+0000) 2020-02-05T23:05:56.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:56.859+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.860229+0000) 2020-02-05T23:05:56.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:05:56.860+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.860475+0000) 2020-02-05T23:05:56.861 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:05:56.858+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.860028+0000) 2020-02-05T23:05:56.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:56.863+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:56.864346+0000) 2020-02-05T23:05:56.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:05:56.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:05:57.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:05:57.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:57.139993+0000) 2020-02-05T23:05:57.141 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:05:57.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:57.140197+0000) 2020-02-05T23:05:57.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:05:57.139+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:57.140308+0000) 2020-02-05T23:05:58.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:05:58.910+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:05:58.911838+0000) 2020-02-05T23:06:00.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:00.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:00.134122+0000) 2020-02-05T23:06:00.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:00.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:00.371356+0000) 2020-02-05T23:06:00.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:00.384+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:00.385327+0000) 2020-02-05T23:06:01.878 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:01.876+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.876980+0000) 2020-02-05T23:06:01.878 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:01.877+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.877846+0000) 2020-02-05T23:06:01.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:01.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.878222+0000) 2020-02-05T23:06:01.879 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:01.877+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.878643+0000) 2020-02-05T23:06:01.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:01.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.878976+0000) 2020-02-05T23:06:01.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:01.878+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.879356+0000) 2020-02-05T23:06:01.881 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:01.878+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.879080+0000) 2020-02-05T23:06:01.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:01.879+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.879856+0000) 2020-02-05T23:06:01.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:01.879+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.880142+0000) 2020-02-05T23:06:01.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:01.879+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.880506+0000) 2020-02-05T23:06:01.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:01.880+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.881319+0000) 2020-02-05T23:06:01.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:01.880+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.881603+0000) 2020-02-05T23:06:01.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:01.881+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.881706+0000) 2020-02-05T23:06:01.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:01.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:01.886381+0000) 2020-02-05T23:06:01.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:01.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:02.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:02.386+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:02.387129+0000) 2020-02-05T23:06:02.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:02.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:02.739915+0000) 2020-02-05T23:06:02.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:02.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:02.777399+0000) 2020-02-05T23:06:03.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:03.900+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:03.901417+0000) 2020-02-05T23:06:03.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:03.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:03.901417+0000) 2020-02-05T23:06:06.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:05.999+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.000098+0000) 2020-02-05T23:06:06.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:06.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.277120+0000) 2020-02-05T23:06:06.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:06.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.897964+0000) 2020-02-05T23:06:06.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:06.897+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.898496+0000) 2020-02-05T23:06:06.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:06.897+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.898274+0000) 2020-02-05T23:06:06.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:06.898+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.899650+0000) 2020-02-05T23:06:06.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:06.899+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.900314+0000) 2020-02-05T23:06:06.901 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:06.899+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.900535+0000) 2020-02-05T23:06:06.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:06.900+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.901069+0000) 2020-02-05T23:06:06.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:06.901+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.901589+0000) 2020-02-05T23:06:06.902 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:06.901+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.901769+0000) 2020-02-05T23:06:06.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:06.901+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.902232+0000) 2020-02-05T23:06:06.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:06.901+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.902809+0000) 2020-02-05T23:06:06.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:06.902+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.902710+0000) 2020-02-05T23:06:06.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:06.901+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.902297+0000) 2020-02-05T23:06:06.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:06.904+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:06.905231+0000) 2020-02-05T23:06:06.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:06.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134700 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:07.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:07.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:07.140216+0000) 2020-02-05T23:06:07.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:07.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:07.140404+0000) 2020-02-05T23:06:07.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:07.140+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:07.140501+0000) 2020-02-05T23:06:08.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:08.919+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:08.920220+0000) 2020-02-05T23:06:10.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:10.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:10.134313+0000) 2020-02-05T23:06:10.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:10.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:10.371642+0000) 2020-02-05T23:06:10.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:10.384+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:10.385552+0000) 2020-02-05T23:06:11.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:11.917+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.918946+0000) 2020-02-05T23:06:11.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:11.918+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.919724+0000) 2020-02-05T23:06:11.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:11.919+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.920526+0000) 2020-02-05T23:06:11.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:11.920+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.921214+0000) 2020-02-05T23:06:11.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:11.921+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.921948+0000) 2020-02-05T23:06:11.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:11.921+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.922386+0000) 2020-02-05T23:06:11.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:11.922+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.923243+0000) 2020-02-05T23:06:11.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:11.922+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.923754+0000) 2020-02-05T23:06:11.925 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:11.923+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.923825+0000) 2020-02-05T23:06:11.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:11.923+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.924067+0000) 2020-02-05T23:06:11.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:11.924+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.924545+0000) 2020-02-05T23:06:11.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:11.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.924120+0000) 2020-02-05T23:06:11.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:11.924+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.925206+0000) 2020-02-05T23:06:11.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:11.927+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:11.928990+0000) 2020-02-05T23:06:11.934 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:11.933+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:12.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:12.387+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:12.387367+0000) 2020-02-05T23:06:12.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:12.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:12.740154+0000) 2020-02-05T23:06:12.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:12.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:12.777625+0000) 2020-02-05T23:06:13.233 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:06:13.239 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:06:13.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:13.901+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:13.901688+0000) 2020-02-05T23:06:13.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:13.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:13.901686+0000) 2020-02-05T23:06:16.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:16.000+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.000352+0000) 2020-02-05T23:06:16.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:16.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.277426+0000) 2020-02-05T23:06:16.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:16.938+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.938592+0000) 2020-02-05T23:06:16.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:16.939+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.940219+0000) 2020-02-05T23:06:16.941 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:16.939+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.940563+0000) 2020-02-05T23:06:16.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:16.940+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.941130+0000) 2020-02-05T23:06:16.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:16.941+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.941926+0000) 2020-02-05T23:06:16.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:16.941+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.942229+0000) 2020-02-05T23:06:16.943 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:16.942+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.942575+0000) 2020-02-05T23:06:16.945 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:16.943+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.943682+0000) 2020-02-05T23:06:16.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:16.943+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.944298+0000) 2020-02-05T23:06:16.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:16.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.943137+0000) 2020-02-05T23:06:16.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:16.943+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.944311+0000) 2020-02-05T23:06:16.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:16.942+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.943419+0000) 2020-02-05T23:06:16.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:16.945+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.945838+0000) 2020-02-05T23:06:16.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:16.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:16.947571+0000) 2020-02-05T23:06:16.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:16.955+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:17.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:17.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:17.140493+0000) 2020-02-05T23:06:17.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:17.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:17.140621+0000) 2020-02-05T23:06:17.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:17.140+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:17.140703+0000) 2020-02-05T23:06:18.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:18.928+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:18.928570+0000) 2020-02-05T23:06:20.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:20.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:20.134544+0000) 2020-02-05T23:06:20.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:20.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:20.371907+0000) 2020-02-05T23:06:20.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:20.385+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:20.385779+0000) 2020-02-05T23:06:21.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:21.960+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.961760+0000) 2020-02-05T23:06:21.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:21.961+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.962671+0000) 2020-02-05T23:06:21.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:21.962+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.963273+0000) 2020-02-05T23:06:21.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:21.962+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.963611+0000) 2020-02-05T23:06:21.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:21.963+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.964611+0000) 2020-02-05T23:06:21.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:21.963+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.965231+0000) 2020-02-05T23:06:21.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:21.963+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.965170+0000) 2020-02-05T23:06:21.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:21.964+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.965411+0000) 2020-02-05T23:06:21.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:21.964+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.965680+0000) 2020-02-05T23:06:21.967 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:21.964+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.966028+0000) 2020-02-05T23:06:21.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:21.965+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.967044+0000) 2020-02-05T23:06:21.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:21.965+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.966759+0000) 2020-02-05T23:06:21.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:21.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.969253+0000) 2020-02-05T23:06:21.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:21.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:21.970113+0000) 2020-02-05T23:06:21.979 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:21.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:22.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:22.386+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:22.387635+0000) 2020-02-05T23:06:22.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:22.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:22.740315+0000) 2020-02-05T23:06:22.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:22.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:22.777836+0000) 2020-02-05T23:06:23.903 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:23.901+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:23.901954+0000) 2020-02-05T23:06:23.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:23.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:23.901954+0000) 2020-02-05T23:06:26.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:25.999+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.000642+0000) 2020-02-05T23:06:26.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:26.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.277655+0000) 2020-02-05T23:06:26.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:26.982+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.983564+0000) 2020-02-05T23:06:26.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:26.984+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.985129+0000) 2020-02-05T23:06:26.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:26.982+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.984383+0000) 2020-02-05T23:06:26.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:26.984+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.985506+0000) 2020-02-05T23:06:26.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:26.984+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.985810+0000) 2020-02-05T23:06:26.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:26.985+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.986286+0000) 2020-02-05T23:06:26.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:26.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.986976+0000) 2020-02-05T23:06:26.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:26.985+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.986597+0000) 2020-02-05T23:06:26.988 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:26.985+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.987002+0000) 2020-02-05T23:06:26.989 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:26.986+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.987905+0000) 2020-02-05T23:06:26.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:26.986+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.988019+0000) 2020-02-05T23:06:26.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:26.986+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.988392+0000) 2020-02-05T23:06:26.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:26.987+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.988834+0000) 2020-02-05T23:06:26.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:26.992+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:26.993387+0000) 2020-02-05T23:06:26.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:26.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:27.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:27.139+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:27.140722+0000) 2020-02-05T23:06:27.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:27.139+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:27.140841+0000) 2020-02-05T23:06:27.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:27.139+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:27.140920+0000) 2020-02-05T23:06:28.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:28.940+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:28.940326+0000) 2020-02-05T23:06:30.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:30.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:30.134809+0000) 2020-02-05T23:06:30.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:30.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:30.372168+0000) 2020-02-05T23:06:30.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:30.385+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:30.386009+0000) 2020-02-05T23:06:32.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:32.002+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.004368+0000) 2020-02-05T23:06:32.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:32.005+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.005172+0000) 2020-02-05T23:06:32.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:32.005+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.005395+0000) 2020-02-05T23:06:32.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:32.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.005105+0000) 2020-02-05T23:06:32.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:32.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.005983+0000) 2020-02-05T23:06:32.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:32.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.006203+0000) 2020-02-05T23:06:32.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:32.006+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.006713+0000) 2020-02-05T23:06:32.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:32.005+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.006628+0000) 2020-02-05T23:06:32.008 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:32.005+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.007215+0000) 2020-02-05T23:06:32.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:32.007+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.008020+0000) 2020-02-05T23:06:32.009 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:32.006+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.007955+0000) 2020-02-05T23:06:32.009 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:32.007+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.008595+0000) 2020-02-05T23:06:32.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:32.008+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.008548+0000) 2020-02-05T23:06:32.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:32.014+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.014236+0000) 2020-02-05T23:06:32.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:32.019+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134901 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:32.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:32.386+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.387877+0000) 2020-02-05T23:06:32.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:32.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.740529+0000) 2020-02-05T23:06:32.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:32.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:32.778072+0000) 2020-02-05T23:06:33.903 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:33.902+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:33.902206+0000) 2020-02-05T23:06:33.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:33.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:33.902206+0000) 2020-02-05T23:06:36.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:36.000+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:36.000895+0000) 2020-02-05T23:06:36.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:36.276+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:36.277889+0000) 2020-02-05T23:06:37.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:37.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.025852+0000) 2020-02-05T23:06:37.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:37.026+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.027857+0000) 2020-02-05T23:06:37.029 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:37.027+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.028148+0000) 2020-02-05T23:06:37.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:37.026+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.028282+0000) 2020-02-05T23:06:37.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:37.027+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.028685+0000) 2020-02-05T23:06:37.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:37.027+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.028972+0000) 2020-02-05T23:06:37.030 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:37.027+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.029206+0000) 2020-02-05T23:06:37.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:37.028+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.029655+0000) 2020-02-05T23:06:37.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:37.028+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.029598+0000) 2020-02-05T23:06:37.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:37.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.030083+0000) 2020-02-05T23:06:37.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:37.029+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.030217+0000) 2020-02-05T23:06:37.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:37.029+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.031014+0000) 2020-02-05T23:06:37.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:37.028+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.030378+0000) 2020-02-05T23:06:37.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:37.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.032657+0000) 2020-02-05T23:06:37.043 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:37.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:37.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:37.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.140970+0000) 2020-02-05T23:06:37.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:37.140+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.141124+0000) 2020-02-05T23:06:37.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:37.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:37.141077+0000) 2020-02-05T23:06:38.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:38.947+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:38.948455+0000) 2020-02-05T23:06:40.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:40.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:40.135131+0000) 2020-02-05T23:06:40.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:40.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:40.372370+0000) 2020-02-05T23:06:40.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:40.385+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:40.386273+0000) 2020-02-05T23:06:42.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:42.047+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.048469+0000) 2020-02-05T23:06:42.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:42.048+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.049358+0000) 2020-02-05T23:06:42.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:42.048+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.049972+0000) 2020-02-05T23:06:42.051 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:42.049+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.050450+0000) 2020-02-05T23:06:42.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:42.049+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.050925+0000) 2020-02-05T23:06:42.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:42.050+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.051738+0000) 2020-02-05T23:06:42.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:42.051+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.052612+0000) 2020-02-05T23:06:42.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:42.051+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.052954+0000) 2020-02-05T23:06:42.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:42.051+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.053377+0000) 2020-02-05T23:06:42.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:42.052+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.053544+0000) 2020-02-05T23:06:42.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:42.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.054331+0000) 2020-02-05T23:06:42.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:42.052+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.054275+0000) 2020-02-05T23:06:42.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:42.053+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.054626+0000) 2020-02-05T23:06:42.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:42.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.057823+0000) 2020-02-05T23:06:42.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:42.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 134983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:42.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:42.387+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.388127+0000) 2020-02-05T23:06:42.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:42.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.740755+0000) 2020-02-05T23:06:42.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:42.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:42.778335+0000) 2020-02-05T23:06:43.376 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:06:43.383 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:06:43.903 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:43.901+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:43.902474+0000) 2020-02-05T23:06:43.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:43.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:43.902474+0000) 2020-02-05T23:06:46.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:46.000+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:46.001154+0000) 2020-02-05T23:06:46.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:46.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:46.278138+0000) 2020-02-05T23:06:47.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:47.068+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.069196+0000) 2020-02-05T23:06:47.070 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:47.068+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.069807+0000) 2020-02-05T23:06:47.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:47.069+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.070410+0000) 2020-02-05T23:06:47.071 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:47.069+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.070954+0000) 2020-02-05T23:06:47.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:47.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.071840+0000) 2020-02-05T23:06:47.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:47.072+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.073338+0000) 2020-02-05T23:06:47.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:47.072+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.073944+0000) 2020-02-05T23:06:47.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:47.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.073134+0000) 2020-02-05T23:06:47.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:47.072+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.073956+0000) 2020-02-05T23:06:47.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:47.072+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.074359+0000) 2020-02-05T23:06:47.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:47.074+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.075470+0000) 2020-02-05T23:06:47.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:47.073+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.075043+0000) 2020-02-05T23:06:47.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:47.076+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.077926+0000) 2020-02-05T23:06:47.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:47.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.078836+0000) 2020-02-05T23:06:47.084 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:47.082+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:47.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:47.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.141144+0000) 2020-02-05T23:06:47.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:47.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.141251+0000) 2020-02-05T23:06:47.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:47.140+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:47.141396+0000) 2020-02-05T23:06:48.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:48.955+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:48.956561+0000) 2020-02-05T23:06:50.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:50.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:50.135356+0000) 2020-02-05T23:06:50.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:50.371+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:50.372661+0000) 2020-02-05T23:06:50.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:50.385+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:50.386506+0000) 2020-02-05T23:06:52.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:52.088+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.089132+0000) 2020-02-05T23:06:52.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:52.088+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.089571+0000) 2020-02-05T23:06:52.091 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:52.088+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.090059+0000) 2020-02-05T23:06:52.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:52.089+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.090847+0000) 2020-02-05T23:06:52.092 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:52.089+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.091261+0000) 2020-02-05T23:06:52.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:52.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.091519+0000) 2020-02-05T23:06:52.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:52.090+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.092040+0000) 2020-02-05T23:06:52.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:52.091+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.092695+0000) 2020-02-05T23:06:52.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:52.091+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.092834+0000) 2020-02-05T23:06:52.095 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:52.092+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.093267+0000) 2020-02-05T23:06:52.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:52.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.093663+0000) 2020-02-05T23:06:52.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:52.093+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.094126+0000) 2020-02-05T23:06:52.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:52.093+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.095117+0000) 2020-02-05T23:06:52.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:52.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.098854+0000) 2020-02-05T23:06:52.105 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:52.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:52.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:52.387+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.388377+0000) 2020-02-05T23:06:52.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:52.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.740960+0000) 2020-02-05T23:06:52.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:52.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:52.778543+0000) 2020-02-05T23:06:53.904 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:53.901+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:53.902719+0000) 2020-02-05T23:06:53.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:53.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:53.902736+0000) 2020-02-05T23:06:56.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:56.000+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:56.001384+0000) 2020-02-05T23:06:56.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:56.277+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:56.278392+0000) 2020-02-05T23:06:57.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:06:57.108+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.109299+0000) 2020-02-05T23:06:57.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:57.108+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.109693+0000) 2020-02-05T23:06:57.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:57.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.110172+0000) 2020-02-05T23:06:57.112 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:06:57.110+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.111158+0000) 2020-02-05T23:06:57.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:06:57.110+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.111421+0000) 2020-02-05T23:06:57.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:57.110+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.112013+0000) 2020-02-05T23:06:57.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:57.111+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.112270+0000) 2020-02-05T23:06:57.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:06:57.111+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.112818+0000) 2020-02-05T23:06:57.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:57.112+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.113761+0000) 2020-02-05T23:06:57.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:06:57.111+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.113185+0000) 2020-02-05T23:06:57.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:06:57.112+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.114242+0000) 2020-02-05T23:06:57.116 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:06:57.113+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.114512+0000) 2020-02-05T23:06:57.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:06:57.114+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.115225+0000) 2020-02-05T23:06:57.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:57.117+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.118459+0000) 2020-02-05T23:06:57.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:06:57.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:06:57.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:06:57.140+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.141371+0000) 2020-02-05T23:06:57.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:06:57.140+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.141472+0000) 2020-02-05T23:06:57.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:06:57.140+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:57.141652+0000) 2020-02-05T23:06:58.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:06:58.967+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:06:58.968203+0000) 2020-02-05T23:07:00.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:00.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:00.135591+0000) 2020-02-05T23:07:00.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:00.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:00.372910+0000) 2020-02-05T23:07:00.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:00.385+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:00.386763+0000) 2020-02-05T23:07:02.133 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:02.129+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.130977+0000) 2020-02-05T23:07:02.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:02.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.131934+0000) 2020-02-05T23:07:02.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:02.131+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.133243+0000) 2020-02-05T23:07:02.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:02.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.133838+0000) 2020-02-05T23:07:02.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:02.132+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.134389+0000) 2020-02-05T23:07:02.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:02.133+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.135011+0000) 2020-02-05T23:07:02.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:02.134+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.135340+0000) 2020-02-05T23:07:02.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:02.134+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.135522+0000) 2020-02-05T23:07:02.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:02.135+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.136921+0000) 2020-02-05T23:07:02.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:02.136+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.137496+0000) 2020-02-05T23:07:02.138 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:02.136+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.137649+0000) 2020-02-05T23:07:02.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:02.136+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.137943+0000) 2020-02-05T23:07:02.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:02.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.138222+0000) 2020-02-05T23:07:02.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:02.141+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.142208+0000) 2020-02-05T23:07:02.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:02.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135145 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:02.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:02.388+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.388651+0000) 2020-02-05T23:07:02.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:02.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.741208+0000) 2020-02-05T23:07:02.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:02.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:02.778747+0000) 2020-02-05T23:07:03.904 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:03.902+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:03.903010+0000) 2020-02-05T23:07:03.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:03.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:03.903011+0000) 2020-02-05T23:07:06.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:06.001+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:06.001668+0000) 2020-02-05T23:07:06.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:06.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:06.278628+0000) 2020-02-05T23:07:07.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:07.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.141556+0000) 2020-02-05T23:07:07.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:07.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.141681+0000) 2020-02-05T23:07:07.143 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:07.141+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.141828+0000) 2020-02-05T23:07:07.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:07.152+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.153483+0000) 2020-02-05T23:07:07.155 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:07.153+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.154338+0000) 2020-02-05T23:07:07.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:07.153+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.154685+0000) 2020-02-05T23:07:07.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:07.153+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.155046+0000) 2020-02-05T23:07:07.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:07.154+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.155386+0000) 2020-02-05T23:07:07.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:07.154+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.155614+0000) 2020-02-05T23:07:07.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:07.154+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.155675+0000) 2020-02-05T23:07:07.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:07.154+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.156241+0000) 2020-02-05T23:07:07.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:07.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.156755+0000) 2020-02-05T23:07:07.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:07.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.157458+0000) 2020-02-05T23:07:07.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:07.156+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.157885+0000) 2020-02-05T23:07:07.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:07.156+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.157856+0000) 2020-02-05T23:07:07.159 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:07.156+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.158376+0000) 2020-02-05T23:07:07.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:07.160+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:07.162050+0000) 2020-02-05T23:07:07.171 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:07.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:08.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:08.975+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:08.976415+0000) 2020-02-05T23:07:10.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:10.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:10.135824+0000) 2020-02-05T23:07:10.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:10.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:10.373140+0000) 2020-02-05T23:07:10.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:10.386+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:10.386998+0000) 2020-02-05T23:07:12.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:12.175+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.176394+0000) 2020-02-05T23:07:12.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:12.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.176845+0000) 2020-02-05T23:07:12.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:12.176+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.177612+0000) 2020-02-05T23:07:12.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:12.177+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.177534+0000) 2020-02-05T23:07:12.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:12.177+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.178336+0000) 2020-02-05T23:07:12.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:12.177+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.178749+0000) 2020-02-05T23:07:12.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:12.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.179024+0000) 2020-02-05T23:07:12.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:12.178+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.179363+0000) 2020-02-05T23:07:12.182 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:12.179+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.179627+0000) 2020-02-05T23:07:12.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:12.179+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.179909+0000) 2020-02-05T23:07:12.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:12.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.180216+0000) 2020-02-05T23:07:12.183 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:12.179+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.180659+0000) 2020-02-05T23:07:12.183 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:12.181+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.181634+0000) 2020-02-05T23:07:12.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:12.185+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.186933+0000) 2020-02-05T23:07:12.192 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:12.190+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:12.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:12.388+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.388889+0000) 2020-02-05T23:07:12.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:12.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.741410+0000) 2020-02-05T23:07:12.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:12.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:12.778985+0000) 2020-02-05T23:07:13.521 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:07:13.527 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:07:13.904 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:13.902+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:13.903320+0000) 2020-02-05T23:07:13.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:13.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:13.903322+0000) 2020-02-05T23:07:16.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:16.001+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:16.001919+0000) 2020-02-05T23:07:16.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:16.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:16.278863+0000) 2020-02-05T23:07:17.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:17.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.141810+0000) 2020-02-05T23:07:17.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:17.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.141942+0000) 2020-02-05T23:07:17.143 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:17.141+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.142045+0000) 2020-02-05T23:07:17.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:17.196+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.197115+0000) 2020-02-05T23:07:17.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:17.196+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.196778+0000) 2020-02-05T23:07:17.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:17.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.197557+0000) 2020-02-05T23:07:17.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:17.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.198504+0000) 2020-02-05T23:07:17.200 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:17.198+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.199313+0000) 2020-02-05T23:07:17.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:17.199+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.199822+0000) 2020-02-05T23:07:17.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:17.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.200420+0000) 2020-02-05T23:07:17.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:17.200+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.200961+0000) 2020-02-05T23:07:17.202 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:17.200+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.201175+0000) 2020-02-05T23:07:17.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:17.200+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.201584+0000) 2020-02-05T23:07:17.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:17.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.202469+0000) 2020-02-05T23:07:17.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:17.201+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.202377+0000) 2020-02-05T23:07:17.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:17.201+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.202241+0000) 2020-02-05T23:07:17.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:17.206+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:17.207552+0000) 2020-02-05T23:07:17.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:17.211+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:18.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:18.985+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:18.985318+0000) 2020-02-05T23:07:20.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:20.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:20.136114+0000) 2020-02-05T23:07:20.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:20.372+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:20.373397+0000) 2020-02-05T23:07:20.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:20.387+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:20.387269+0000) 2020-02-05T23:07:22.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:22.218+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.219176+0000) 2020-02-05T23:07:22.220 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:22.218+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.219518+0000) 2020-02-05T23:07:22.221 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:22.218+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.219779+0000) 2020-02-05T23:07:22.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:22.219+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.219489+0000) 2020-02-05T23:07:22.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:22.219+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.220063+0000) 2020-02-05T23:07:22.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:22.219+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.220345+0000) 2020-02-05T23:07:22.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:22.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.220707+0000) 2020-02-05T23:07:22.222 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:22.220+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.221180+0000) 2020-02-05T23:07:22.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:22.220+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.221465+0000) 2020-02-05T23:07:22.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:22.222+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.222653+0000) 2020-02-05T23:07:22.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:22.221+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.222279+0000) 2020-02-05T23:07:22.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:22.222+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.223039+0000) 2020-02-05T23:07:22.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:22.223+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.223898+0000) 2020-02-05T23:07:22.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:22.226+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.227454+0000) 2020-02-05T23:07:22.236 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:22.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:22.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:22.388+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.389116+0000) 2020-02-05T23:07:22.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:22.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.741609+0000) 2020-02-05T23:07:22.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:22.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:22.779200+0000) 2020-02-05T23:07:23.904 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:23.903+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:23.903594+0000) 2020-02-05T23:07:23.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:23.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:23.903594+0000) 2020-02-05T23:07:26.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:26.001+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:26.002159+0000) 2020-02-05T23:07:26.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:26.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:26.279141+0000) 2020-02-05T23:07:27.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:27.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.142064+0000) 2020-02-05T23:07:27.143 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:27.141+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.142273+0000) 2020-02-05T23:07:27.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:27.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.142243+0000) 2020-02-05T23:07:27.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:27.240+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.240417+0000) 2020-02-05T23:07:27.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:27.241+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.242325+0000) 2020-02-05T23:07:27.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:27.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.243149+0000) 2020-02-05T23:07:27.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:27.244+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.244351+0000) 2020-02-05T23:07:27.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:27.244+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.244602+0000) 2020-02-05T23:07:27.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:27.243+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.243582+0000) 2020-02-05T23:07:27.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:27.243+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.244024+0000) 2020-02-05T23:07:27.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:27.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.245063+0000) 2020-02-05T23:07:27.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:27.245+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.245210+0000) 2020-02-05T23:07:27.248 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:27.245+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.245635+0000) 2020-02-05T23:07:27.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:27.244+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.245165+0000) 2020-02-05T23:07:27.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:27.245+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.245905+0000) 2020-02-05T23:07:27.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:27.246+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.246577+0000) 2020-02-05T23:07:27.255 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:27.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135347 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:27.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:27.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:27.251071+0000) 2020-02-05T23:07:28.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:28.993+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:28.993801+0000) 2020-02-05T23:07:30.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:30.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:30.136364+0000) 2020-02-05T23:07:30.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:30.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:30.373724+0000) 2020-02-05T23:07:30.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:30.386+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:30.387512+0000) 2020-02-05T23:07:32.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:32.259+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.259746+0000) 2020-02-05T23:07:32.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:32.259+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.260278+0000) 2020-02-05T23:07:32.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:32.260+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.261262+0000) 2020-02-05T23:07:32.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:32.260+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.261629+0000) 2020-02-05T23:07:32.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:32.261+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.262150+0000) 2020-02-05T23:07:32.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:32.262+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.262549+0000) 2020-02-05T23:07:32.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:32.262+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.262841+0000) 2020-02-05T23:07:32.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:32.262+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.263414+0000) 2020-02-05T23:07:32.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:32.262+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.263766+0000) 2020-02-05T23:07:32.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:32.263+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.263913+0000) 2020-02-05T23:07:32.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:32.263+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.264319+0000) 2020-02-05T23:07:32.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:32.264+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.264806+0000) 2020-02-05T23:07:32.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:32.264+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.265167+0000) 2020-02-05T23:07:32.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:32.269+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.270866+0000) 2020-02-05T23:07:32.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:32.275+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:32.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:32.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.389335+0000) 2020-02-05T23:07:32.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:32.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.741838+0000) 2020-02-05T23:07:32.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:32.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:32.779410+0000) 2020-02-05T23:07:33.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:33.902+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:33.903867+0000) 2020-02-05T23:07:33.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:33.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:33.903868+0000) 2020-02-05T23:07:36.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:36.002+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:36.002399+0000) 2020-02-05T23:07:36.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:36.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:36.279373+0000) 2020-02-05T23:07:37.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:37.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.142359+0000) 2020-02-05T23:07:37.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:37.142+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.142549+0000) 2020-02-05T23:07:37.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:37.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.142552+0000) 2020-02-05T23:07:37.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:37.280+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.280903+0000) 2020-02-05T23:07:37.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:37.281+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.282832+0000) 2020-02-05T23:07:37.284 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:37.282+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.283495+0000) 2020-02-05T23:07:37.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:37.283+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.284254+0000) 2020-02-05T23:07:37.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:37.284+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.284762+0000) 2020-02-05T23:07:37.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:37.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.285142+0000) 2020-02-05T23:07:37.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:37.284+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.285391+0000) 2020-02-05T23:07:37.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:37.285+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.286319+0000) 2020-02-05T23:07:37.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:37.285+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.286765+0000) 2020-02-05T23:07:37.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:37.286+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.286992+0000) 2020-02-05T23:07:37.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:37.286+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.287309+0000) 2020-02-05T23:07:37.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:37.286+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.287776+0000) 2020-02-05T23:07:37.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:37.287+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.288340+0000) 2020-02-05T23:07:37.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:37.291+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:37.292439+0000) 2020-02-05T23:07:37.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:37.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:39.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:39.000+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:39.001935+0000) 2020-02-05T23:07:40.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:40.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:40.136614+0000) 2020-02-05T23:07:40.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:40.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:40.373984+0000) 2020-02-05T23:07:40.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:40.386+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:40.387746+0000) 2020-02-05T23:07:42.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:42.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.302525+0000) 2020-02-05T23:07:42.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:42.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.302959+0000) 2020-02-05T23:07:42.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:42.303+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.304164+0000) 2020-02-05T23:07:42.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:42.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.304668+0000) 2020-02-05T23:07:42.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:42.304+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.305256+0000) 2020-02-05T23:07:42.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:42.305+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.305421+0000) 2020-02-05T23:07:42.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:42.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.305812+0000) 2020-02-05T23:07:42.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:42.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.306533+0000) 2020-02-05T23:07:42.308 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:42.306+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.307193+0000) 2020-02-05T23:07:42.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:42.306+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.307105+0000) 2020-02-05T23:07:42.309 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:42.307+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.308202+0000) 2020-02-05T23:07:42.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:42.307+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.308259+0000) 2020-02-05T23:07:42.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:42.308+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.308690+0000) 2020-02-05T23:07:42.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:42.312+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.313789+0000) 2020-02-05T23:07:42.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:42.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:42.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:42.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.389577+0000) 2020-02-05T23:07:42.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:42.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.742071+0000) 2020-02-05T23:07:42.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:42.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:42.779631+0000) 2020-02-05T23:07:43.660 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:07:43.666 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:07:43.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:43.903+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:43.904182+0000) 2020-02-05T23:07:43.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:43.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:43.904178+0000) 2020-02-05T23:07:46.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:46.002+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:46.002737+0000) 2020-02-05T23:07:46.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:46.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:46.279582+0000) 2020-02-05T23:07:47.143 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:47.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.142649+0000) 2020-02-05T23:07:47.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:47.142+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.142841+0000) 2020-02-05T23:07:47.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:47.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.142817+0000) 2020-02-05T23:07:47.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:47.324+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.325162+0000) 2020-02-05T23:07:47.327 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:47.325+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.326694+0000) 2020-02-05T23:07:47.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:47.326+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.327250+0000) 2020-02-05T23:07:47.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:47.327+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.327988+0000) 2020-02-05T23:07:47.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:47.327+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.328587+0000) 2020-02-05T23:07:47.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:47.327+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.328158+0000) 2020-02-05T23:07:47.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:47.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.329295+0000) 2020-02-05T23:07:47.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:47.328+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.329978+0000) 2020-02-05T23:07:47.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:47.329+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.330665+0000) 2020-02-05T23:07:47.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:47.330+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.331403+0000) 2020-02-05T23:07:47.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:47.331+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.331804+0000) 2020-02-05T23:07:47.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:47.331+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.332188+0000) 2020-02-05T23:07:47.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:47.331+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.332188+0000) 2020-02-05T23:07:47.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:47.334+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:47.335681+0000) 2020-02-05T23:07:47.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:47.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:49.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:49.009+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:49.010600+0000) 2020-02-05T23:07:50.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:50.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:50.136879+0000) 2020-02-05T23:07:50.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:50.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:50.374274+0000) 2020-02-05T23:07:50.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:50.386+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:50.387967+0000) 2020-02-05T23:07:52.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:52.346+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.347708+0000) 2020-02-05T23:07:52.349 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:52.347+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.348336+0000) 2020-02-05T23:07:52.350 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:52.348+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.349252+0000) 2020-02-05T23:07:52.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:52.348+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.350137+0000) 2020-02-05T23:07:52.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:52.349+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.350532+0000) 2020-02-05T23:07:52.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:52.349+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.350840+0000) 2020-02-05T23:07:52.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:52.349+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.351121+0000) 2020-02-05T23:07:52.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:52.350+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.351418+0000) 2020-02-05T23:07:52.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:52.350+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.352050+0000) 2020-02-05T23:07:52.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:52.350+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.352412+0000) 2020-02-05T23:07:52.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:52.351+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.352343+0000) 2020-02-05T23:07:52.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:52.351+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.353092+0000) 2020-02-05T23:07:52.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:52.351+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.352921+0000) 2020-02-05T23:07:52.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:52.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.357475+0000) 2020-02-05T23:07:52.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:52.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:52.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:52.388+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.389815+0000) 2020-02-05T23:07:52.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:52.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.742305+0000) 2020-02-05T23:07:52.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:52.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:52.779834+0000) 2020-02-05T23:07:53.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:53.903+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:53.904452+0000) 2020-02-05T23:07:53.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:53.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:53.904452+0000) 2020-02-05T23:07:56.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:56.001+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:56.002996+0000) 2020-02-05T23:07:56.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:56.278+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:56.279844+0000) 2020-02-05T23:07:57.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:57.141+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.142886+0000) 2020-02-05T23:07:57.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:57.141+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.143109+0000) 2020-02-05T23:07:57.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:57.141+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.143116+0000) 2020-02-05T23:07:57.370 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:07:57.368+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.369238+0000) 2020-02-05T23:07:57.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:07:57.369+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.370311+0000) 2020-02-05T23:07:57.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:07:57.369+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.370875+0000) 2020-02-05T23:07:57.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:07:57.370+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.371489+0000) 2020-02-05T23:07:57.372 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:07:57.370+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.371842+0000) 2020-02-05T23:07:57.373 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:07:57.371+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.372229+0000) 2020-02-05T23:07:57.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:07:57.371+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.372564+0000) 2020-02-05T23:07:57.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:07:57.371+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.373064+0000) 2020-02-05T23:07:57.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:57.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.373465+0000) 2020-02-05T23:07:57.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:07:57.372+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.373822+0000) 2020-02-05T23:07:57.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:07:57.373+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.375084+0000) 2020-02-05T23:07:57.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:07:57.374+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.375167+0000) 2020-02-05T23:07:57.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:57.379+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.380494+0000) 2020-02-05T23:07:57.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:57.379+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:57.380742+0000) 2020-02-05T23:07:57.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:07:57.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135585 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:07:59.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:07:59.018+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:07:59.018734+0000) 2020-02-05T23:08:00.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:00.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:00.137135+0000) 2020-02-05T23:08:00.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:00.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:00.374523+0000) 2020-02-05T23:08:00.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:00.387+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:00.388287+0000) 2020-02-05T23:08:02.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:02.388+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.390062+0000) 2020-02-05T23:08:02.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:02.391+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.392236+0000) 2020-02-05T23:08:02.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:02.392+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.393279+0000) 2020-02-05T23:08:02.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:02.392+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.393817+0000) 2020-02-05T23:08:02.395 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:02.392+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.394071+0000) 2020-02-05T23:08:02.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:02.393+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.394638+0000) 2020-02-05T23:08:02.396 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:02.393+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.394890+0000) 2020-02-05T23:08:02.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:02.394+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.395966+0000) 2020-02-05T23:08:02.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:02.394+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.396039+0000) 2020-02-05T23:08:02.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:02.394+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.396120+0000) 2020-02-05T23:08:02.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:02.394+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.396296+0000) 2020-02-05T23:08:02.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:02.396+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.397638+0000) 2020-02-05T23:08:02.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:02.397+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.398130+0000) 2020-02-05T23:08:02.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:02.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.402328+0000) 2020-02-05T23:08:02.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:02.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.402567+0000) 2020-02-05T23:08:02.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:02.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:02.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:02.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.742585+0000) 2020-02-05T23:08:02.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:02.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:02.780057+0000) 2020-02-05T23:08:03.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:03.904+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:03.904706+0000) 2020-02-05T23:08:03.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:03.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:03.904706+0000) 2020-02-05T23:08:06.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:06.002+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:06.003244+0000) 2020-02-05T23:08:06.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:06.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:06.280113+0000) 2020-02-05T23:08:07.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:07.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.143143+0000) 2020-02-05T23:08:07.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:07.142+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.143394+0000) 2020-02-05T23:08:07.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:07.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.143399+0000) 2020-02-05T23:08:07.416 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:07.414+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.415127+0000) 2020-02-05T23:08:07.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:07.414+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.415661+0000) 2020-02-05T23:08:07.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:07.415+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.416492+0000) 2020-02-05T23:08:07.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:07.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.416733+0000) 2020-02-05T23:08:07.419 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:07.416+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.417463+0000) 2020-02-05T23:08:07.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:07.416+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.417403+0000) 2020-02-05T23:08:07.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:07.415+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.416996+0000) 2020-02-05T23:08:07.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:07.416+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.417444+0000) 2020-02-05T23:08:07.420 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:07.416+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.417962+0000) 2020-02-05T23:08:07.421 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:07.416+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.417965+0000) 2020-02-05T23:08:07.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:07.416+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.418138+0000) 2020-02-05T23:08:07.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:07.417+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.418410+0000) 2020-02-05T23:08:07.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:07.417+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.418664+0000) 2020-02-05T23:08:07.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:07.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:07.422083+0000) 2020-02-05T23:08:07.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:07.428+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:09.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:09.026+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:09.026921+0000) 2020-02-05T23:08:10.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:10.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:10.137315+0000) 2020-02-05T23:08:10.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:10.373+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:10.374766+0000) 2020-02-05T23:08:10.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:10.388+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:10.388512+0000) 2020-02-05T23:08:12.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:12.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.390362+0000) 2020-02-05T23:08:12.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:12.434+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.435973+0000) 2020-02-05T23:08:12.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:12.435+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.436524+0000) 2020-02-05T23:08:12.437 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:12.435+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.437013+0000) 2020-02-05T23:08:12.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:12.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.438160+0000) 2020-02-05T23:08:12.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:12.436+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.437958+0000) 2020-02-05T23:08:12.439 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:12.437+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.438631+0000) 2020-02-05T23:08:12.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:12.437+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.438656+0000) 2020-02-05T23:08:12.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:12.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.439128+0000) 2020-02-05T23:08:12.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:12.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.439297+0000) 2020-02-05T23:08:12.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:12.438+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.440121+0000) 2020-02-05T23:08:12.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:12.438+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.439874+0000) 2020-02-05T23:08:12.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:12.439+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.440476+0000) 2020-02-05T23:08:12.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:12.439+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.441250+0000) 2020-02-05T23:08:12.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:12.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.444905+0000) 2020-02-05T23:08:12.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:12.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:12.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:12.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.742831+0000) 2020-02-05T23:08:12.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:12.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:12.780318+0000) 2020-02-05T23:08:13.800 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:08:13.807 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:08:13.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:13.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:13.904910+0000) 2020-02-05T23:08:13.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:13.903+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:13.904923+0000) 2020-02-05T23:08:16.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:16.002+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:16.003478+0000) 2020-02-05T23:08:16.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:16.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:16.280349+0000) 2020-02-05T23:08:17.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:17.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.143376+0000) 2020-02-05T23:08:17.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:17.142+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.143681+0000) 2020-02-05T23:08:17.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:17.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.143674+0000) 2020-02-05T23:08:17.457 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:17.455+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.456183+0000) 2020-02-05T23:08:17.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:17.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.458133+0000) 2020-02-05T23:08:17.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:17.457+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.458569+0000) 2020-02-05T23:08:17.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:17.458+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.459214+0000) 2020-02-05T23:08:17.460 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:17.458+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.459482+0000) 2020-02-05T23:08:17.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:17.459+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.460632+0000) 2020-02-05T23:08:17.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:17.460+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.461455+0000) 2020-02-05T23:08:17.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:17.460+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.461470+0000) 2020-02-05T23:08:17.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:17.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.461866+0000) 2020-02-05T23:08:17.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:17.461+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.462385+0000) 2020-02-05T23:08:17.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:17.461+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.462637+0000) 2020-02-05T23:08:17.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:17.461+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.463091+0000) 2020-02-05T23:08:17.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:17.461+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.463220+0000) 2020-02-05T23:08:17.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:17.464+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:17.465918+0000) 2020-02-05T23:08:17.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:17.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:19.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:19.034+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:19.034966+0000) 2020-02-05T23:08:20.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:20.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:20.137545+0000) 2020-02-05T23:08:20.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:20.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:20.375012+0000) 2020-02-05T23:08:20.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:20.388+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:20.388741+0000) 2020-02-05T23:08:22.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:22.389+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.390652+0000) 2020-02-05T23:08:22.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:22.480+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.480071+0000) 2020-02-05T23:08:22.482 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:22.481+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.481168+0000) 2020-02-05T23:08:22.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:22.480+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.481535+0000) 2020-02-05T23:08:22.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:22.481+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.482020+0000) 2020-02-05T23:08:22.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:22.481+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.483080+0000) 2020-02-05T23:08:22.484 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:22.483+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.483350+0000) 2020-02-05T23:08:22.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:22.482+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.484116+0000) 2020-02-05T23:08:22.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:22.484+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.484406+0000) 2020-02-05T23:08:22.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:22.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.484427+0000) 2020-02-05T23:08:22.486 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:22.483+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.484764+0000) 2020-02-05T23:08:22.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:22.483+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.484556+0000) 2020-02-05T23:08:22.486 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:22.483+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.485362+0000) 2020-02-05T23:08:22.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:22.485+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.485127+0000) 2020-02-05T23:08:22.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:22.491+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.491109+0000) 2020-02-05T23:08:22.496 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:22.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:22.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:22.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.743033+0000) 2020-02-05T23:08:22.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:22.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:22.780521+0000) 2020-02-05T23:08:23.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:23.905+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:23.905152+0000) 2020-02-05T23:08:23.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:23.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:23.905149+0000) 2020-02-05T23:08:26.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:26.002+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:26.003706+0000) 2020-02-05T23:08:26.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:26.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:26.280617+0000) 2020-02-05T23:08:27.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:27.142+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.143642+0000) 2020-02-05T23:08:27.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:27.142+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.143946+0000) 2020-02-05T23:08:27.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:27.142+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.143902+0000) 2020-02-05T23:08:27.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:27.500+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.501430+0000) 2020-02-05T23:08:27.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:27.500+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.501983+0000) 2020-02-05T23:08:27.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:27.501+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.502193+0000) 2020-02-05T23:08:27.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:27.501+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.502646+0000) 2020-02-05T23:08:27.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:27.501+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.502738+0000) 2020-02-05T23:08:27.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:27.502+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.503200+0000) 2020-02-05T23:08:27.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:27.502+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.503538+0000) 2020-02-05T23:08:27.505 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:27.502+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.503928+0000) 2020-02-05T23:08:27.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:27.503+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.504504+0000) 2020-02-05T23:08:27.505 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:27.503+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.505024+0000) 2020-02-05T23:08:27.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:27.504+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.505477+0000) 2020-02-05T23:08:27.506 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:27.504+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.505944+0000) 2020-02-05T23:08:27.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:27.504+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.505974+0000) 2020-02-05T23:08:27.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:27.508+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:27.509789+0000) 2020-02-05T23:08:27.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:27.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:29.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:29.042+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:29.043282+0000) 2020-02-05T23:08:30.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:30.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:30.137772+0000) 2020-02-05T23:08:30.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:30.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:30.375248+0000) 2020-02-05T23:08:30.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:30.388+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:30.388984+0000) 2020-02-05T23:08:32.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:32.390+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.390947+0000) 2020-02-05T23:08:32.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:32.519+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.520383+0000) 2020-02-05T23:08:32.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:32.520+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.521240+0000) 2020-02-05T23:08:32.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:32.521+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.522945+0000) 2020-02-05T23:08:32.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:32.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.523400+0000) 2020-02-05T23:08:32.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:32.522+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.523616+0000) 2020-02-05T23:08:32.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:32.522+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.523887+0000) 2020-02-05T23:08:32.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:32.523+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.524306+0000) 2020-02-05T23:08:32.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:32.523+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.524662+0000) 2020-02-05T23:08:32.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:32.523+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.525162+0000) 2020-02-05T23:08:32.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:32.524+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.525744+0000) 2020-02-05T23:08:32.527 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:32.524+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.526211+0000) 2020-02-05T23:08:32.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:32.525+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.526508+0000) 2020-02-05T23:08:32.528 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:32.525+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.527287+0000) 2020-02-05T23:08:32.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:32.530+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.531674+0000) 2020-02-05T23:08:32.539 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:32.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:32.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:32.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.743380+0000) 2020-02-05T23:08:32.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:32.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:32.780742+0000) 2020-02-05T23:08:33.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:33.904+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:33.905406+0000) 2020-02-05T23:08:33.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:33.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:33.905406+0000) 2020-02-05T23:08:36.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:36.003+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:36.004001+0000) 2020-02-05T23:08:36.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:36.279+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:36.280841+0000) 2020-02-05T23:08:37.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:37.143+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.143885+0000) 2020-02-05T23:08:37.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:37.143+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.144111+0000) 2020-02-05T23:08:37.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:37.143+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.144276+0000) 2020-02-05T23:08:37.546 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:37.544+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.545237+0000) 2020-02-05T23:08:37.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:37.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.545964+0000) 2020-02-05T23:08:37.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:37.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.546356+0000) 2020-02-05T23:08:37.547 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:37.545+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.546610+0000) 2020-02-05T23:08:37.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:37.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.547118+0000) 2020-02-05T23:08:37.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:37.546+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.547515+0000) 2020-02-05T23:08:37.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:37.546+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.547847+0000) 2020-02-05T23:08:37.549 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:37.547+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.548216+0000) 2020-02-05T23:08:37.549 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:37.547+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.548749+0000) 2020-02-05T23:08:37.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:37.547+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.549385+0000) 2020-02-05T23:08:37.550 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:37.547+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.549357+0000) 2020-02-05T23:08:37.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:37.548+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.550054+0000) 2020-02-05T23:08:37.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:37.548+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.550309+0000) 2020-02-05T23:08:37.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:37.553+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:37.554190+0000) 2020-02-05T23:08:37.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:37.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:39.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:39.050+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:39.051265+0000) 2020-02-05T23:08:40.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:40.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:40.138039+0000) 2020-02-05T23:08:40.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:40.374+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:40.375492+0000) 2020-02-05T23:08:40.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:40.388+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:40.389214+0000) 2020-02-05T23:08:42.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:42.390+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.391200+0000) 2020-02-05T23:08:42.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:42.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.563647+0000) 2020-02-05T23:08:42.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:42.564+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.565193+0000) 2020-02-05T23:08:42.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:42.564+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.565996+0000) 2020-02-05T23:08:42.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:42.565+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.566844+0000) 2020-02-05T23:08:42.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:42.566+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.567296+0000) 2020-02-05T23:08:42.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:42.567+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.567878+0000) 2020-02-05T23:08:42.569 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:42.567+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.568100+0000) 2020-02-05T23:08:42.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:42.568+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.568541+0000) 2020-02-05T23:08:42.570 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:42.568+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.568934+0000) 2020-02-05T23:08:42.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:42.568+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.569208+0000) 2020-02-05T23:08:42.570 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:42.568+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.569858+0000) 2020-02-05T23:08:42.571 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:42.568+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.570047+0000) 2020-02-05T23:08:42.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:42.568+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.570192+0000) 2020-02-05T23:08:42.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:42.574+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.575250+0000) 2020-02-05T23:08:42.579 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:42.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:42.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:42.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.743565+0000) 2020-02-05T23:08:42.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:42.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:42.780978+0000) 2020-02-05T23:08:43.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:43.904+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:43.905659+0000) 2020-02-05T23:08:43.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:43.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:43.905660+0000) 2020-02-05T23:08:43.954 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:08:43.961 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:08:46.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:46.003+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:46.004237+0000) 2020-02-05T23:08:46.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:46.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:46.281071+0000) 2020-02-05T23:08:47.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:47.143+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.144122+0000) 2020-02-05T23:08:47.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:47.143+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.144362+0000) 2020-02-05T23:08:47.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:47.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.144538+0000) 2020-02-05T23:08:47.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:47.582+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.583729+0000) 2020-02-05T23:08:47.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:47.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.584227+0000) 2020-02-05T23:08:47.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:47.584+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.585839+0000) 2020-02-05T23:08:47.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:47.586+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.587253+0000) 2020-02-05T23:08:47.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:47.586+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.587129+0000) 2020-02-05T23:08:47.588 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:47.587+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.587778+0000) 2020-02-05T23:08:47.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:47.587+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.588156+0000) 2020-02-05T23:08:47.589 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:47.587+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.588365+0000) 2020-02-05T23:08:47.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:47.588+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.588804+0000) 2020-02-05T23:08:47.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:47.588+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.589498+0000) 2020-02-05T23:08:47.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:47.588+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.589189+0000) 2020-02-05T23:08:47.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:47.589+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.589670+0000) 2020-02-05T23:08:47.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:47.588+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.589934+0000) 2020-02-05T23:08:47.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:47.593+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:47.594677+0000) 2020-02-05T23:08:47.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:47.598+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 135980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:49.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:49.059+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:49.060015+0000) 2020-02-05T23:08:50.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:50.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:50.138256+0000) 2020-02-05T23:08:50.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:50.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:50.375745+0000) 2020-02-05T23:08:50.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:50.388+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:50.389390+0000) 2020-02-05T23:08:52.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:52.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.391496+0000) 2020-02-05T23:08:52.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:52.604+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.606010+0000) 2020-02-05T23:08:52.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:52.605+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.606382+0000) 2020-02-05T23:08:52.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:52.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.606896+0000) 2020-02-05T23:08:52.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:52.606+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.607303+0000) 2020-02-05T23:08:52.608 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:52.606+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.607394+0000) 2020-02-05T23:08:52.610 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:52.606+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.607688+0000) 2020-02-05T23:08:52.610 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:52.607+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.608313+0000) 2020-02-05T23:08:52.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:52.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.608576+0000) 2020-02-05T23:08:52.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:52.608+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.609159+0000) 2020-02-05T23:08:52.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:52.607+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.607925+0000) 2020-02-05T23:08:52.611 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:52.608+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.608863+0000) 2020-02-05T23:08:52.611 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:52.609+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.609849+0000) 2020-02-05T23:08:52.612 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:52.609+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.610221+0000) 2020-02-05T23:08:52.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:52.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.614694+0000) 2020-02-05T23:08:52.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:52.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:52.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:52.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.743780+0000) 2020-02-05T23:08:52.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:52.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:52.781185+0000) 2020-02-05T23:08:53.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:53.905+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:53.905899+0000) 2020-02-05T23:08:53.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:53.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:53.905901+0000) 2020-02-05T23:08:56.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:56.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:56.004493+0000) 2020-02-05T23:08:56.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:56.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:56.281357+0000) 2020-02-05T23:08:57.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:57.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.144400+0000) 2020-02-05T23:08:57.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:57.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.144607+0000) 2020-02-05T23:08:57.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:57.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.144786+0000) 2020-02-05T23:08:57.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:08:57.625+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.626411+0000) 2020-02-05T23:08:57.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:08:57.626+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.627465+0000) 2020-02-05T23:08:57.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:08:57.628+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.628972+0000) 2020-02-05T23:08:57.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:08:57.628+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.629334+0000) 2020-02-05T23:08:57.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:08:57.629+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.629979+0000) 2020-02-05T23:08:57.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:08:57.630+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.630622+0000) 2020-02-05T23:08:57.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:08:57.630+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.631518+0000) 2020-02-05T23:08:57.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:08:57.631+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.632164+0000) 2020-02-05T23:08:57.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:08:57.631+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.632238+0000) 2020-02-05T23:08:57.633 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:08:57.632+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.632690+0000) 2020-02-05T23:08:57.634 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:08:57.632+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.633071+0000) 2020-02-05T23:08:57.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:57.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.636027+0000) 2020-02-05T23:08:57.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:57.635+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.636412+0000) 2020-02-05T23:08:57.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:57.635+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:57.636746+0000) 2020-02-05T23:08:57.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:08:57.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:08:59.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:08:59.066+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:08:59.067869+0000) 2020-02-05T23:09:00.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:00.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:00.138517+0000) 2020-02-05T23:09:00.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:00.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:00.375993+0000) 2020-02-05T23:09:00.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:00.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:00.389626+0000) 2020-02-05T23:09:02.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:02.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.391829+0000) 2020-02-05T23:09:02.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:02.646+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.647764+0000) 2020-02-05T23:09:02.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:02.647+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.648668+0000) 2020-02-05T23:09:02.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:02.648+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.649433+0000) 2020-02-05T23:09:02.652 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:02.648+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.649247+0000) 2020-02-05T23:09:02.653 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:02.650+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.650516+0000) 2020-02-05T23:09:02.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:02.648+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.650056+0000) 2020-02-05T23:09:02.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:02.649+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.650712+0000) 2020-02-05T23:09:02.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:02.651+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.651747+0000) 2020-02-05T23:09:02.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:02.651+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.651854+0000) 2020-02-05T23:09:02.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:02.652+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.652589+0000) 2020-02-05T23:09:02.657 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:02.653+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.653821+0000) 2020-02-05T23:09:02.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:02.652+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.653333+0000) 2020-02-05T23:09:02.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:02.652+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.653455+0000) 2020-02-05T23:09:02.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:02.655+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.656714+0000) 2020-02-05T23:09:02.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:02.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:02.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:02.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.744011+0000) 2020-02-05T23:09:02.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:02.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:02.781399+0000) 2020-02-05T23:09:03.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:03.905+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:03.906139+0000) 2020-02-05T23:09:03.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:03.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:03.906148+0000) 2020-02-05T23:09:06.005 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:06.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:06.004678+0000) 2020-02-05T23:09:06.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:06.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:06.281572+0000) 2020-02-05T23:09:07.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:07.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.144673+0000) 2020-02-05T23:09:07.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:07.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.144860+0000) 2020-02-05T23:09:07.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:07.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.145029+0000) 2020-02-05T23:09:07.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:07.668+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.669787+0000) 2020-02-05T23:09:07.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:07.669+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.670665+0000) 2020-02-05T23:09:07.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:07.670+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.671382+0000) 2020-02-05T23:09:07.672 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:07.670+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.671333+0000) 2020-02-05T23:09:07.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:07.670+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.671450+0000) 2020-02-05T23:09:07.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:07.671+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.671737+0000) 2020-02-05T23:09:07.674 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:07.671+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.671884+0000) 2020-02-05T23:09:07.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:07.671+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.672328+0000) 2020-02-05T23:09:07.674 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:07.672+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.673068+0000) 2020-02-05T23:09:07.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:07.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.673419+0000) 2020-02-05T23:09:07.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:07.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.673904+0000) 2020-02-05T23:09:07.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:07.673+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.673965+0000) 2020-02-05T23:09:07.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:07.674+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.674539+0000) 2020-02-05T23:09:07.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:07.677+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:07.678914+0000) 2020-02-05T23:09:07.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:07.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:09.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:09.077+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:09.077422+0000) 2020-02-05T23:09:10.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:10.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:10.138787+0000) 2020-02-05T23:09:10.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:10.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:10.376235+0000) 2020-02-05T23:09:10.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:10.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:10.389858+0000) 2020-02-05T23:09:12.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:12.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.392093+0000) 2020-02-05T23:09:12.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:12.690+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.691149+0000) 2020-02-05T23:09:12.692 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:12.690+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.691436+0000) 2020-02-05T23:09:12.693 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:12.691+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.692216+0000) 2020-02-05T23:09:12.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:12.690+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.691907+0000) 2020-02-05T23:09:12.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:12.692+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.692717+0000) 2020-02-05T23:09:12.694 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:12.693+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.693475+0000) 2020-02-05T23:09:12.695 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:12.694+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.694408+0000) 2020-02-05T23:09:12.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:12.694+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.695262+0000) 2020-02-05T23:09:12.697 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:12.695+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.696297+0000) 2020-02-05T23:09:12.698 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:12.695+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.696934+0000) 2020-02-05T23:09:12.698 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:12.696+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.697068+0000) 2020-02-05T23:09:12.698 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:12.697+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.697712+0000) 2020-02-05T23:09:12.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:12.696+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.697486+0000) 2020-02-05T23:09:12.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:12.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.702581+0000) 2020-02-05T23:09:12.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:12.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:12.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:12.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.744267+0000) 2020-02-05T23:09:12.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:12.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:12.781588+0000) 2020-02-05T23:09:13.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:13.906+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:13.906406+0000) 2020-02-05T23:09:13.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:13.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:13.906406+0000) 2020-02-05T23:09:14.088 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:09:14.095 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:09:16.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:16.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:16.004924+0000) 2020-02-05T23:09:16.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:16.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:16.281791+0000) 2020-02-05T23:09:17.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:17.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.144912+0000) 2020-02-05T23:09:17.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:17.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.145133+0000) 2020-02-05T23:09:17.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:17.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.145273+0000) 2020-02-05T23:09:17.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:17.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.710613+0000) 2020-02-05T23:09:17.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:17.711+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.711126+0000) 2020-02-05T23:09:17.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:17.712+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.712856+0000) 2020-02-05T23:09:17.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:17.713+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.713376+0000) 2020-02-05T23:09:17.714 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:17.713+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.713691+0000) 2020-02-05T23:09:17.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:17.714+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.714135+0000) 2020-02-05T23:09:17.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:17.714+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.714682+0000) 2020-02-05T23:09:17.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:17.715+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.715762+0000) 2020-02-05T23:09:17.717 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:17.715+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.716193+0000) 2020-02-05T23:09:17.717 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:17.716+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.716682+0000) 2020-02-05T23:09:17.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:17.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.717475+0000) 2020-02-05T23:09:17.718 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:17.716+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.717225+0000) 2020-02-05T23:09:17.719 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:17.717+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.717751+0000) 2020-02-05T23:09:17.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:17.720+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:17.721068+0000) 2020-02-05T23:09:17.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:17.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:19.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:19.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:19.085674+0000) 2020-02-05T23:09:20.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:20.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:20.139027+0000) 2020-02-05T23:09:20.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:20.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:20.376527+0000) 2020-02-05T23:09:20.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:20.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:20.390133+0000) 2020-02-05T23:09:22.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:22.390+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.392336+0000) 2020-02-05T23:09:22.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:22.731+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.732688+0000) 2020-02-05T23:09:22.734 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:22.732+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.733214+0000) 2020-02-05T23:09:22.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:22.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.733791+0000) 2020-02-05T23:09:22.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:22.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.734327+0000) 2020-02-05T23:09:22.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:22.733+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.734518+0000) 2020-02-05T23:09:22.736 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:22.733+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.734998+0000) 2020-02-05T23:09:22.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:22.734+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.735484+0000) 2020-02-05T23:09:22.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:22.735+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.736718+0000) 2020-02-05T23:09:22.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:22.736+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.737453+0000) 2020-02-05T23:09:22.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:22.735+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.736562+0000) 2020-02-05T23:09:22.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:22.735+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.736909+0000) 2020-02-05T23:09:22.739 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:22.735+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.737234+0000) 2020-02-05T23:09:22.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:22.736+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.738088+0000) 2020-02-05T23:09:22.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:22.739+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.740873+0000) 2020-02-05T23:09:22.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:22.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.744409+0000) 2020-02-05T23:09:22.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:22.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:22.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:22.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:22.781793+0000) 2020-02-05T23:09:23.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:23.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:23.906641+0000) 2020-02-05T23:09:23.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:23.905+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:23.906680+0000) 2020-02-05T23:09:26.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:26.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:26.005179+0000) 2020-02-05T23:09:26.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:26.280+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:26.282058+0000) 2020-02-05T23:09:27.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:27.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.145150+0000) 2020-02-05T23:09:27.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:27.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.145471+0000) 2020-02-05T23:09:27.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:27.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.145523+0000) 2020-02-05T23:09:27.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:27.755+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.756697+0000) 2020-02-05T23:09:27.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:27.756+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.757371+0000) 2020-02-05T23:09:27.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:27.756+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.757957+0000) 2020-02-05T23:09:27.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:27.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.758478+0000) 2020-02-05T23:09:27.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:27.757+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.758532+0000) 2020-02-05T23:09:27.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:27.757+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.758878+0000) 2020-02-05T23:09:27.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:27.758+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.759464+0000) 2020-02-05T23:09:27.761 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:27.758+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.759829+0000) 2020-02-05T23:09:27.761 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:27.758+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.759848+0000) 2020-02-05T23:09:27.761 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:27.759+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.760338+0000) 2020-02-05T23:09:27.762 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:27.758+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.760235+0000) 2020-02-05T23:09:27.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:27.759+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.760939+0000) 2020-02-05T23:09:27.762 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:27.760+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.761907+0000) 2020-02-05T23:09:27.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:27.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:27.767774+0000) 2020-02-05T23:09:27.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:27.768+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136305 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:29.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:29.092+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:29.093385+0000) 2020-02-05T23:09:30.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:30.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:30.139299+0000) 2020-02-05T23:09:30.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:30.375+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:30.376763+0000) 2020-02-05T23:09:30.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:30.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:30.390384+0000) 2020-02-05T23:09:32.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:32.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.392638+0000) 2020-02-05T23:09:32.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:32.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.744586+0000) 2020-02-05T23:09:32.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:32.774+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.775532+0000) 2020-02-05T23:09:32.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:32.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.777393+0000) 2020-02-05T23:09:32.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:32.776+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.777400+0000) 2020-02-05T23:09:32.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:32.776+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.777980+0000) 2020-02-05T23:09:32.779 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:32.777+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.778484+0000) 2020-02-05T23:09:32.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:32.777+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.779372+0000) 2020-02-05T23:09:32.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:32.778+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.779815+0000) 2020-02-05T23:09:32.782 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:32.777+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.779191+0000) 2020-02-05T23:09:32.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:32.779+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.780550+0000) 2020-02-05T23:09:32.782 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:32.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.781070+0000) 2020-02-05T23:09:32.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:32.780+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.782044+0000) 2020-02-05T23:09:32.783 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:32.780+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.781809+0000) 2020-02-05T23:09:32.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:32.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.788854+0000) 2020-02-05T23:09:32.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:32.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:32.790814+0000) 2020-02-05T23:09:32.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:32.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:33.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:33.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:33.906883+0000) 2020-02-05T23:09:33.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:33.906+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:33.906942+0000) 2020-02-05T23:09:36.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:36.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:36.005475+0000) 2020-02-05T23:09:36.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:36.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:36.282314+0000) 2020-02-05T23:09:37.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:37.143+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.145393+0000) 2020-02-05T23:09:37.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:37.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.145711+0000) 2020-02-05T23:09:37.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:37.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.145750+0000) 2020-02-05T23:09:37.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:37.796+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.797567+0000) 2020-02-05T23:09:37.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:37.796+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.798233+0000) 2020-02-05T23:09:37.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:37.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.799412+0000) 2020-02-05T23:09:37.800 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:37.798+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.800017+0000) 2020-02-05T23:09:37.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:37.799+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.800733+0000) 2020-02-05T23:09:37.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:37.800+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.801221+0000) 2020-02-05T23:09:37.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:37.800+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.801286+0000) 2020-02-05T23:09:37.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:37.799+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.800989+0000) 2020-02-05T23:09:37.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:37.800+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.801678+0000) 2020-02-05T23:09:37.804 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:37.800+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.802400+0000) 2020-02-05T23:09:37.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:37.801+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.802977+0000) 2020-02-05T23:09:37.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:37.802+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.803154+0000) 2020-02-05T23:09:37.805 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:37.802+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.803601+0000) 2020-02-05T23:09:37.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:37.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:37.808908+0000) 2020-02-05T23:09:37.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:37.813+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:39.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:39.100+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:39.101638+0000) 2020-02-05T23:09:39.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:39.531+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:39.532775+0000) 2020-02-05T23:09:40.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:40.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:40.139548+0000) 2020-02-05T23:09:40.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:40.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:40.377006+0000) 2020-02-05T23:09:40.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:40.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:40.390620+0000) 2020-02-05T23:09:42.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:42.391+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.392886+0000) 2020-02-05T23:09:42.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:42.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.744851+0000) 2020-02-05T23:09:42.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:42.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.819848+0000) 2020-02-05T23:09:42.821 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:42.819+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.820464+0000) 2020-02-05T23:09:42.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:42.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.821048+0000) 2020-02-05T23:09:42.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:42.820+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.821420+0000) 2020-02-05T23:09:42.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:42.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.822433+0000) 2020-02-05T23:09:42.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:42.820+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.822252+0000) 2020-02-05T23:09:42.824 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:42.821+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.822763+0000) 2020-02-05T23:09:42.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:42.822+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.823523+0000) 2020-02-05T23:09:42.825 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:42.822+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.823597+0000) 2020-02-05T23:09:42.825 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:42.822+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.823895+0000) 2020-02-05T23:09:42.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:42.822+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.824139+0000) 2020-02-05T23:09:42.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:42.823+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.824288+0000) 2020-02-05T23:09:42.826 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:42.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.825876+0000) 2020-02-05T23:09:42.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:42.828+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:42.829535+0000) 2020-02-05T23:09:42.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:42.834+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:43.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:43.906+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:43.907167+0000) 2020-02-05T23:09:43.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:43.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:43.907139+0000) 2020-02-05T23:09:44.239 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:09:44.246 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:09:46.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:46.005+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:46.005764+0000) 2020-02-05T23:09:46.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:46.281+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:46.282536+0000) 2020-02-05T23:09:47.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:47.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.145658+0000) 2020-02-05T23:09:47.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:47.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.145932+0000) 2020-02-05T23:09:47.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:47.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.145983+0000) 2020-02-05T23:09:47.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:47.840+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.842303+0000) 2020-02-05T23:09:47.844 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:47.841+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.843265+0000) 2020-02-05T23:09:47.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:47.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.843699+0000) 2020-02-05T23:09:47.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:47.842+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.844059+0000) 2020-02-05T23:09:47.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:47.843+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.844462+0000) 2020-02-05T23:09:47.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:47.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.844750+0000) 2020-02-05T23:09:47.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:47.843+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.845046+0000) 2020-02-05T23:09:47.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:47.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.845484+0000) 2020-02-05T23:09:47.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:47.845+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.846259+0000) 2020-02-05T23:09:47.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:47.845+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.846796+0000) 2020-02-05T23:09:47.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:47.846+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.847279+0000) 2020-02-05T23:09:47.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:47.846+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.848169+0000) 2020-02-05T23:09:47.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:47.847+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.848500+0000) 2020-02-05T23:09:47.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:47.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:47.852207+0000) 2020-02-05T23:09:47.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:47.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:49.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:49.109+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:49.109974+0000) 2020-02-05T23:09:49.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:49.531+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:49.533029+0000) 2020-02-05T23:09:50.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:50.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:50.139778+0000) 2020-02-05T23:09:50.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:50.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:50.377252+0000) 2020-02-05T23:09:50.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:50.389+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:50.390869+0000) 2020-02-05T23:09:52.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:52.392+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.393138+0000) 2020-02-05T23:09:52.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:52.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.745106+0000) 2020-02-05T23:09:52.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:52.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.863955+0000) 2020-02-05T23:09:52.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:52.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.864897+0000) 2020-02-05T23:09:52.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:52.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.865704+0000) 2020-02-05T23:09:52.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:52.865+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.866125+0000) 2020-02-05T23:09:52.867 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:52.865+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.866604+0000) 2020-02-05T23:09:52.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:52.865+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.866848+0000) 2020-02-05T23:09:52.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:52.866+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.867256+0000) 2020-02-05T23:09:52.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:52.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.867572+0000) 2020-02-05T23:09:52.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:52.866+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.868240+0000) 2020-02-05T23:09:52.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:52.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.868803+0000) 2020-02-05T23:09:52.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:52.867+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.869147+0000) 2020-02-05T23:09:52.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:52.869+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.870115+0000) 2020-02-05T23:09:52.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:52.868+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.870118+0000) 2020-02-05T23:09:52.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:52.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:52.874942+0000) 2020-02-05T23:09:52.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:52.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:53.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:53.906+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:53.907412+0000) 2020-02-05T23:09:53.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:53.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:53.907365+0000) 2020-02-05T23:09:56.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:56.005+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:56.006017+0000) 2020-02-05T23:09:56.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:56.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:56.282767+0000) 2020-02-05T23:09:57.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:57.144+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.145837+0000) 2020-02-05T23:09:57.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:57.144+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.146245+0000) 2020-02-05T23:09:57.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:57.144+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.146171+0000) 2020-02-05T23:09:57.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:57.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.885949+0000) 2020-02-05T23:09:57.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:09:57.884+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.886116+0000) 2020-02-05T23:09:57.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:09:57.885+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.886576+0000) 2020-02-05T23:09:57.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:57.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.887281+0000) 2020-02-05T23:09:57.888 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:09:57.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.887869+0000) 2020-02-05T23:09:57.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:09:57.886+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.887772+0000) 2020-02-05T23:09:57.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:09:57.887+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.888645+0000) 2020-02-05T23:09:57.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:09:57.888+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.889128+0000) 2020-02-05T23:09:57.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:09:57.888+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.889566+0000) 2020-02-05T23:09:57.891 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:09:57.889+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.890824+0000) 2020-02-05T23:09:57.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:09:57.890+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.891422+0000) 2020-02-05T23:09:57.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:09:57.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.891605+0000) 2020-02-05T23:09:57.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:09:57.890+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.891987+0000) 2020-02-05T23:09:57.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:57.894+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:57.895106+0000) 2020-02-05T23:09:57.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:09:57.899+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136551 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:09:59.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:59.121+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:59.121434+0000) 2020-02-05T23:09:59.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:09:59.532+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:09:59.533292+0000) 2020-02-05T23:10:00.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:00.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:00.140004+0000) 2020-02-05T23:10:00.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:00.376+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:00.377507+0000) 2020-02-05T23:10:00.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:00.390+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:00.391133+0000) 2020-02-05T23:10:02.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:02.392+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.393415+0000) 2020-02-05T23:10:02.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:02.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.745347+0000) 2020-02-05T23:10:02.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:02.904+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.905237+0000) 2020-02-05T23:10:02.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:02.905+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.906568+0000) 2020-02-05T23:10:02.908 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:02.905+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.907154+0000) 2020-02-05T23:10:02.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:02.906+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.907633+0000) 2020-02-05T23:10:02.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:02.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.908020+0000) 2020-02-05T23:10:02.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:02.907+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909145+0000) 2020-02-05T23:10:02.910 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:02.907+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909094+0000) 2020-02-05T23:10:02.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:02.908+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909284+0000) 2020-02-05T23:10:02.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:02.908+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909500+0000) 2020-02-05T23:10:02.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:02.908+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909950+0000) 2020-02-05T23:10:02.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:02.908+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.909903+0000) 2020-02-05T23:10:02.912 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:02.908+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.910166+0000) 2020-02-05T23:10:02.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:02.909+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.910742+0000) 2020-02-05T23:10:02.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:02.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:02.916835+0000) 2020-02-05T23:10:02.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:02.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:03.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:03.907+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:03.907645+0000) 2020-02-05T23:10:03.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:03.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:03.907643+0000) 2020-02-05T23:10:06.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:06.004+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:06.006247+0000) 2020-02-05T23:10:06.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:06.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:06.283015+0000) 2020-02-05T23:10:07.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:07.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.146076+0000) 2020-02-05T23:10:07.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:07.145+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.146422+0000) 2020-02-05T23:10:07.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:07.145+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.146512+0000) 2020-02-05T23:10:07.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:07.926+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.927127+0000) 2020-02-05T23:10:07.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:07.926+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.927671+0000) 2020-02-05T23:10:07.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:07.926+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.927810+0000) 2020-02-05T23:10:07.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:07.927+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.928291+0000) 2020-02-05T23:10:07.929 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:07.927+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.928728+0000) 2020-02-05T23:10:07.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:07.927+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.929221+0000) 2020-02-05T23:10:07.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:07.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.930208+0000) 2020-02-05T23:10:07.931 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:07.929+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.930656+0000) 2020-02-05T23:10:07.932 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:07.929+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.931039+0000) 2020-02-05T23:10:07.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:07.930+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.931463+0000) 2020-02-05T23:10:07.932 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:07.930+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.932003+0000) 2020-02-05T23:10:07.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:07.931+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.932732+0000) 2020-02-05T23:10:07.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:07.931+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.932713+0000) 2020-02-05T23:10:07.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:07.934+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:07.935747+0000) 2020-02-05T23:10:07.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:07.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:09.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:09.129+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:09.129516+0000) 2020-02-05T23:10:09.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:09.533+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:09.533518+0000) 2020-02-05T23:10:10.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:10.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:10.140263+0000) 2020-02-05T23:10:10.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:10.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:10.377822+0000) 2020-02-05T23:10:10.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:10.391+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:10.391358+0000) 2020-02-05T23:10:12.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:12.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.393693+0000) 2020-02-05T23:10:12.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:12.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.745586+0000) 2020-02-05T23:10:12.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:12.948+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.948454+0000) 2020-02-05T23:10:12.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:12.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.949496+0000) 2020-02-05T23:10:12.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:12.950+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.950272+0000) 2020-02-05T23:10:12.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:12.951+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.951237+0000) 2020-02-05T23:10:12.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:12.950+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.951277+0000) 2020-02-05T23:10:12.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:12.951+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.951877+0000) 2020-02-05T23:10:12.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:12.952+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.952418+0000) 2020-02-05T23:10:12.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:12.952+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.952832+0000) 2020-02-05T23:10:12.954 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:12.953+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.953541+0000) 2020-02-05T23:10:12.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:12.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.953350+0000) 2020-02-05T23:10:12.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:12.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.954014+0000) 2020-02-05T23:10:12.955 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:12.954+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.954751+0000) 2020-02-05T23:10:12.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:12.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.958058+0000) 2020-02-05T23:10:12.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:12.958+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:12.958385+0000) 2020-02-05T23:10:12.965 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:12.964+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:13.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:13.906+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:13.907871+0000) 2020-02-05T23:10:13.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:13.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:13.907873+0000) 2020-02-05T23:10:14.469 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:10:14.475 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:10:16.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:16.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:16.006488+0000) 2020-02-05T23:10:16.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:16.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:16.283298+0000) 2020-02-05T23:10:17.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:17.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.146308+0000) 2020-02-05T23:10:17.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:17.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.146732+0000) 2020-02-05T23:10:17.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:17.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.146745+0000) 2020-02-05T23:10:17.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:17.968+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.969890+0000) 2020-02-05T23:10:17.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:17.969+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.970402+0000) 2020-02-05T23:10:17.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:17.970+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.972032+0000) 2020-02-05T23:10:17.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:17.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.972697+0000) 2020-02-05T23:10:17.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:17.972+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.972678+0000) 2020-02-05T23:10:17.974 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:17.972+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.972954+0000) 2020-02-05T23:10:17.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:17.972+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.973820+0000) 2020-02-05T23:10:17.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:17.973+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.973537+0000) 2020-02-05T23:10:17.975 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:17.973+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.974215+0000) 2020-02-05T23:10:17.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:17.974+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.974854+0000) 2020-02-05T23:10:17.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:17.975+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.975521+0000) 2020-02-05T23:10:17.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:17.975+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.975631+0000) 2020-02-05T23:10:17.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:17.975+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.976374+0000) 2020-02-05T23:10:17.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:17.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:17.979890+0000) 2020-02-05T23:10:17.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:17.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136717 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:19.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:19.136+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:19.137860+0000) 2020-02-05T23:10:19.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:19.532+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:19.533705+0000) 2020-02-05T23:10:20.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:20.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:20.140498+0000) 2020-02-05T23:10:20.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:20.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:20.378087+0000) 2020-02-05T23:10:20.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:20.390+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:20.391587+0000) 2020-02-05T23:10:22.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:22.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.393956+0000) 2020-02-05T23:10:22.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:22.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.745834+0000) 2020-02-05T23:10:22.993 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:22.991+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.991803+0000) 2020-02-05T23:10:22.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:22.991+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.991771+0000) 2020-02-05T23:10:22.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:22.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.992418+0000) 2020-02-05T23:10:22.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:22.991+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.992959+0000) 2020-02-05T23:10:22.994 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:22.993+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.993405+0000) 2020-02-05T23:10:22.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:22.993+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.994433+0000) 2020-02-05T23:10:22.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:22.993+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.994350+0000) 2020-02-05T23:10:22.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:22.993+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.994532+0000) 2020-02-05T23:10:22.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:22.993+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.994806+0000) 2020-02-05T23:10:22.996 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:22.993+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.994367+0000) 2020-02-05T23:10:22.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:22.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.995542+0000) 2020-02-05T23:10:22.997 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:22.995+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.995927+0000) 2020-02-05T23:10:22.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:22.996+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:22.996500+0000) 2020-02-05T23:10:23.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:22.999+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:23.000390+0000) 2020-02-05T23:10:23.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:23.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:23.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:23.907+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:23.908127+0000) 2020-02-05T23:10:23.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:23.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:23.908117+0000) 2020-02-05T23:10:26.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:26.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:26.006751+0000) 2020-02-05T23:10:26.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:26.282+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:26.283522+0000) 2020-02-05T23:10:27.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:27.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:27.146534+0000) 2020-02-05T23:10:27.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:27.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:27.146960+0000) 2020-02-05T23:10:27.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:27.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:27.146955+0000) 2020-02-05T23:10:28.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:28.010+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.011236+0000) 2020-02-05T23:10:28.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:28.011+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.012428+0000) 2020-02-05T23:10:28.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:28.012+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.013215+0000) 2020-02-05T23:10:28.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:28.012+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.013488+0000) 2020-02-05T23:10:28.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:28.013+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.013722+0000) 2020-02-05T23:10:28.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:28.014+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.014534+0000) 2020-02-05T23:10:28.015 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:28.013+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.014968+0000) 2020-02-05T23:10:28.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:28.014+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.015301+0000) 2020-02-05T23:10:28.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:28.015+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.016314+0000) 2020-02-05T23:10:28.017 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:28.015+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.016316+0000) 2020-02-05T23:10:28.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:28.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.016821+0000) 2020-02-05T23:10:28.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:28.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.017447+0000) 2020-02-05T23:10:28.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:28.017+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.017686+0000) 2020-02-05T23:10:28.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:28.023+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:28.024231+0000) 2020-02-05T23:10:28.029 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:28.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:29.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:29.145+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:29.146212+0000) 2020-02-05T23:10:29.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:29.532+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:29.533886+0000) 2020-02-05T23:10:30.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:30.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:30.140752+0000) 2020-02-05T23:10:30.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:30.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:30.378350+0000) 2020-02-05T23:10:30.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:30.391+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:30.391831+0000) 2020-02-05T23:10:32.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:32.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:32.394210+0000) 2020-02-05T23:10:32.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:32.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:32.746053+0000) 2020-02-05T23:10:33.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:33.033+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.034231+0000) 2020-02-05T23:10:33.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:33.034+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.035316+0000) 2020-02-05T23:10:33.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:33.034+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.035711+0000) 2020-02-05T23:10:33.037 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:33.035+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.036138+0000) 2020-02-05T23:10:33.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:33.036+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.036647+0000) 2020-02-05T23:10:33.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:33.036+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.037259+0000) 2020-02-05T23:10:33.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:33.037+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.038226+0000) 2020-02-05T23:10:33.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:33.038+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.039136+0000) 2020-02-05T23:10:33.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:33.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.040340+0000) 2020-02-05T23:10:33.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:33.040+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.041111+0000) 2020-02-05T23:10:33.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:33.039+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.040757+0000) 2020-02-05T23:10:33.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:33.038+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.038710+0000) 2020-02-05T23:10:33.045 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:33.039+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.039903+0000) 2020-02-05T23:10:33.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:33.044+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.045276+0000) 2020-02-05T23:10:33.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:33.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:33.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:33.907+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.908332+0000) 2020-02-05T23:10:33.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:33.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:33.908324+0000) 2020-02-05T23:10:36.009 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:36.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:36.007134+0000) 2020-02-05T23:10:36.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:36.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:36.283759+0000) 2020-02-05T23:10:37.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:37.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:37.146726+0000) 2020-02-05T23:10:37.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:37.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:37.147199+0000) 2020-02-05T23:10:37.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:37.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:37.147245+0000) 2020-02-05T23:10:38.056 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:38.054+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.055595+0000) 2020-02-05T23:10:38.057 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:38.055+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.056106+0000) 2020-02-05T23:10:38.057 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:38.055+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.056805+0000) 2020-02-05T23:10:38.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:38.056+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.057533+0000) 2020-02-05T23:10:38.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:38.057+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.057934+0000) 2020-02-05T23:10:38.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:38.057+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.058532+0000) 2020-02-05T23:10:38.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:38.058+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.059132+0000) 2020-02-05T23:10:38.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:38.058+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.059415+0000) 2020-02-05T23:10:38.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:38.058+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.059931+0000) 2020-02-05T23:10:38.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:38.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.060947+0000) 2020-02-05T23:10:38.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:38.060+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.061311+0000) 2020-02-05T23:10:38.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:38.060+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.061317+0000) 2020-02-05T23:10:38.063 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:38.061+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.062208+0000) 2020-02-05T23:10:38.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:38.064+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:38.065660+0000) 2020-02-05T23:10:38.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:38.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:39.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:39.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:39.154261+0000) 2020-02-05T23:10:39.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:39.533+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:39.534149+0000) 2020-02-05T23:10:40.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:40.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:40.140998+0000) 2020-02-05T23:10:40.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:40.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:40.378618+0000) 2020-02-05T23:10:40.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:40.391+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:40.392056+0000) 2020-02-05T23:10:42.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:42.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:42.394485+0000) 2020-02-05T23:10:42.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:42.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:42.746319+0000) 2020-02-05T23:10:43.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:43.076+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.077256+0000) 2020-02-05T23:10:43.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:43.077+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.078378+0000) 2020-02-05T23:10:43.080 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:43.078+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.079224+0000) 2020-02-05T23:10:43.080 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:43.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.079407+0000) 2020-02-05T23:10:43.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:43.078+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.078863+0000) 2020-02-05T23:10:43.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:43.078+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.079661+0000) 2020-02-05T23:10:43.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:43.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.080260+0000) 2020-02-05T23:10:43.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:43.079+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.080754+0000) 2020-02-05T23:10:43.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:43.081+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.081505+0000) 2020-02-05T23:10:43.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:43.081+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.081864+0000) 2020-02-05T23:10:43.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:43.081+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.082118+0000) 2020-02-05T23:10:43.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:43.082+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.082653+0000) 2020-02-05T23:10:43.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:43.082+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.083553+0000) 2020-02-05T23:10:43.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:43.085+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.086638+0000) 2020-02-05T23:10:43.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:43.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:43.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:43.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.908558+0000) 2020-02-05T23:10:43.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:43.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:43.908557+0000) 2020-02-05T23:10:44.625 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:10:44.631 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:10:46.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:46.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:46.007468+0000) 2020-02-05T23:10:46.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:46.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:46.283997+0000) 2020-02-05T23:10:47.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:47.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:47.147002+0000) 2020-02-05T23:10:47.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:47.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:47.147453+0000) 2020-02-05T23:10:47.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:47.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:47.147514+0000) 2020-02-05T23:10:48.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:48.097+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.098406+0000) 2020-02-05T23:10:48.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:48.098+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.099723+0000) 2020-02-05T23:10:48.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:48.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.099767+0000) 2020-02-05T23:10:48.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:48.099+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.100642+0000) 2020-02-05T23:10:48.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:48.099+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.100647+0000) 2020-02-05T23:10:48.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:48.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.101311+0000) 2020-02-05T23:10:48.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:48.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.101724+0000) 2020-02-05T23:10:48.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:48.100+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.101847+0000) 2020-02-05T23:10:48.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:48.101+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.102924+0000) 2020-02-05T23:10:48.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:48.101+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.102538+0000) 2020-02-05T23:10:48.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:48.101+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.102938+0000) 2020-02-05T23:10:48.106 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:48.102+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.104068+0000) 2020-02-05T23:10:48.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:48.102+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.104331+0000) 2020-02-05T23:10:48.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:48.108+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:48.109484+0000) 2020-02-05T23:10:48.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:48.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 136965 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:49.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:49.161+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:49.162674+0000) 2020-02-05T23:10:49.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:49.533+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:49.534343+0000) 2020-02-05T23:10:50.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:50.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:50.141251+0000) 2020-02-05T23:10:50.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:50.377+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:50.378809+0000) 2020-02-05T23:10:50.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:50.391+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:50.392277+0000) 2020-02-05T23:10:52.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:52.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:52.394750+0000) 2020-02-05T23:10:52.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:52.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:52.746602+0000) 2020-02-05T23:10:53.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:53.117+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.118886+0000) 2020-02-05T23:10:53.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:53.118+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.119729+0000) 2020-02-05T23:10:53.121 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:53.119+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.120430+0000) 2020-02-05T23:10:53.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:53.119+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.120711+0000) 2020-02-05T23:10:53.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:53.120+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.121656+0000) 2020-02-05T23:10:53.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:53.120+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.122127+0000) 2020-02-05T23:10:53.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:53.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.122331+0000) 2020-02-05T23:10:53.124 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:53.121+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.122366+0000) 2020-02-05T23:10:53.125 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:53.121+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.122980+0000) 2020-02-05T23:10:53.125 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:53.122+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.123592+0000) 2020-02-05T23:10:53.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:53.122+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.124032+0000) 2020-02-05T23:10:53.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:53.122+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.124148+0000) 2020-02-05T23:10:53.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:53.123+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.124620+0000) 2020-02-05T23:10:53.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:53.127+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.128632+0000) 2020-02-05T23:10:53.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:53.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137007 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:53.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:53.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.908783+0000) 2020-02-05T23:10:53.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:53.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:53.908789+0000) 2020-02-05T23:10:56.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:56.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:56.007768+0000) 2020-02-05T23:10:56.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:56.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:56.284268+0000) 2020-02-05T23:10:57.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:57.145+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:57.147240+0000) 2020-02-05T23:10:57.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:57.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:57.147694+0000) 2020-02-05T23:10:57.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:57.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:57.147751+0000) 2020-02-05T23:10:58.142 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:10:58.140+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.141429+0000) 2020-02-05T23:10:58.143 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:10:58.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.141660+0000) 2020-02-05T23:10:58.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:10:58.140+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.142124+0000) 2020-02-05T23:10:58.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:10:58.140+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.141975+0000) 2020-02-05T23:10:58.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:10:58.141+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.143107+0000) 2020-02-05T23:10:58.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:58.141+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.142562+0000) 2020-02-05T23:10:58.145 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:10:58.141+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.142855+0000) 2020-02-05T23:10:58.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:10:58.142+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.143153+0000) 2020-02-05T23:10:58.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:10:58.142+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.144352+0000) 2020-02-05T23:10:58.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:10:58.143+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.144549+0000) 2020-02-05T23:10:58.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:10:58.143+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.144974+0000) 2020-02-05T23:10:58.147 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:10:58.143+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.144120+0000) 2020-02-05T23:10:58.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:58.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.144663+0000) 2020-02-05T23:10:58.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:58.147+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:58.149032+0000) 2020-02-05T23:10:58.158 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:10:58.156+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137049 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:10:59.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:59.173+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:59.174184+0000) 2020-02-05T23:10:59.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:10:59.534+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:10:59.534585+0000) 2020-02-05T23:11:00.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:00.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:00.141504+0000) 2020-02-05T23:11:00.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:00.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:00.379045+0000) 2020-02-05T23:11:00.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:00.392+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:00.392504+0000) 2020-02-05T23:11:02.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:02.393+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:02.394942+0000) 2020-02-05T23:11:02.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:02.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:02.746839+0000) 2020-02-05T23:11:03.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:03.161+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.162747+0000) 2020-02-05T23:11:03.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:03.162+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.163604+0000) 2020-02-05T23:11:03.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:03.162+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.164056+0000) 2020-02-05T23:11:03.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:03.164+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.165277+0000) 2020-02-05T23:11:03.166 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:03.164+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.165945+0000) 2020-02-05T23:11:03.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:03.165+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.166923+0000) 2020-02-05T23:11:03.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:03.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.167508+0000) 2020-02-05T23:11:03.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:03.166+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.167878+0000) 2020-02-05T23:11:03.169 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:03.166+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.168150+0000) 2020-02-05T23:11:03.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:03.167+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.168730+0000) 2020-02-05T23:11:03.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:03.168+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.169721+0000) 2020-02-05T23:11:03.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:03.169+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.170472+0000) 2020-02-05T23:11:03.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:03.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.170629+0000) 2020-02-05T23:11:03.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:03.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.175438+0000) 2020-02-05T23:11:03.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:03.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:03.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:03.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.908998+0000) 2020-02-05T23:11:03.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:03.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:03.908996+0000) 2020-02-05T23:11:06.009 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:06.006+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:06.007988+0000) 2020-02-05T23:11:06.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:06.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:06.284510+0000) 2020-02-05T23:11:07.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:07.146+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:07.147481+0000) 2020-02-05T23:11:07.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:07.146+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:07.147920+0000) 2020-02-05T23:11:07.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:07.146+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:07.148030+0000) 2020-02-05T23:11:08.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:08.183+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.184842+0000) 2020-02-05T23:11:08.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:08.185+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.186534+0000) 2020-02-05T23:11:08.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:08.186+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.187915+0000) 2020-02-05T23:11:08.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:08.187+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.189357+0000) 2020-02-05T23:11:08.195 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:08.188+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.189699+0000) 2020-02-05T23:11:08.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:08.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.190089+0000) 2020-02-05T23:11:08.196 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:08.188+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.190237+0000) 2020-02-05T23:11:08.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:08.189+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.190819+0000) 2020-02-05T23:11:08.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:08.186+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.187118+0000) 2020-02-05T23:11:08.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:08.191+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.192811+0000) 2020-02-05T23:11:08.196 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:08.186+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.187174+0000) 2020-02-05T23:11:08.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:08.187+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.188469+0000) 2020-02-05T23:11:08.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:08.188+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.189072+0000) 2020-02-05T23:11:08.197 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:08.188+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:08.189704+0000) 2020-02-05T23:11:08.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:08.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137131 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:09.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:09.180+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:09.181742+0000) 2020-02-05T23:11:09.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:09.533+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:09.534813+0000) 2020-02-05T23:11:10.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:10.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:10.141738+0000) 2020-02-05T23:11:10.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:10.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:10.379305+0000) 2020-02-05T23:11:10.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:10.391+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:10.392733+0000) 2020-02-05T23:11:12.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:12.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:12.395201+0000) 2020-02-05T23:11:12.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:12.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:12.747112+0000) 2020-02-05T23:11:13.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:13.205+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.206601+0000) 2020-02-05T23:11:13.209 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:13.207+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.208273+0000) 2020-02-05T23:11:13.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:13.208+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.209437+0000) 2020-02-05T23:11:13.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:13.207+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.209127+0000) 2020-02-05T23:11:13.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:13.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.210049+0000) 2020-02-05T23:11:13.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:13.209+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.211122+0000) 2020-02-05T23:11:13.213 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:13.210+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.211411+0000) 2020-02-05T23:11:13.213 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:13.209+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.211237+0000) 2020-02-05T23:11:13.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:13.211+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.212298+0000) 2020-02-05T23:11:13.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:13.211+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.212534+0000) 2020-02-05T23:11:13.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:13.211+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.212812+0000) 2020-02-05T23:11:13.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:13.211+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.212976+0000) 2020-02-05T23:11:13.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:13.211+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.213415+0000) 2020-02-05T23:11:13.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:13.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.217009+0000) 2020-02-05T23:11:13.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:13.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:13.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:13.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.909216+0000) 2020-02-05T23:11:13.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:13.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:13.909219+0000) 2020-02-05T23:11:14.763 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:11:14.770 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:11:16.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:16.007+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:16.008239+0000) 2020-02-05T23:11:16.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:16.283+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:16.284741+0000) 2020-02-05T23:11:17.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:17.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:17.147740+0000) 2020-02-05T23:11:17.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:17.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:17.148175+0000) 2020-02-05T23:11:17.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:17.147+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:17.148273+0000) 2020-02-05T23:11:18.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:18.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.231537+0000) 2020-02-05T23:11:18.233 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:18.231+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.232366+0000) 2020-02-05T23:11:18.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:18.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.232935+0000) 2020-02-05T23:11:18.235 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:18.232+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.234326+0000) 2020-02-05T23:11:18.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:18.232+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.234124+0000) 2020-02-05T23:11:18.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:18.234+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.235243+0000) 2020-02-05T23:11:18.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:18.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.235672+0000) 2020-02-05T23:11:18.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:18.235+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.236153+0000) 2020-02-05T23:11:18.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:18.235+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.236722+0000) 2020-02-05T23:11:18.238 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:18.235+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.237034+0000) 2020-02-05T23:11:18.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:18.235+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.237221+0000) 2020-02-05T23:11:18.239 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:18.236+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.237892+0000) 2020-02-05T23:11:18.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:18.236+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.237761+0000) 2020-02-05T23:11:18.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:18.243+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:18.244162+0000) 2020-02-05T23:11:18.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:18.245+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:19.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:19.192+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:19.193662+0000) 2020-02-05T23:11:19.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:19.534+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:19.535039+0000) 2020-02-05T23:11:20.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:20.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:20.141974+0000) 2020-02-05T23:11:20.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:20.378+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:20.379541+0000) 2020-02-05T23:11:20.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:20.392+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:20.392961+0000) 2020-02-05T23:11:22.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:22.394+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:22.395489+0000) 2020-02-05T23:11:22.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:22.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:22.747339+0000) 2020-02-05T23:11:23.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:23.251+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.252318+0000) 2020-02-05T23:11:23.254 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:23.252+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.253124+0000) 2020-02-05T23:11:23.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:23.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.254155+0000) 2020-02-05T23:11:23.255 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:23.253+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.254701+0000) 2020-02-05T23:11:23.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:23.253+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.254734+0000) 2020-02-05T23:11:23.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:23.254+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.255262+0000) 2020-02-05T23:11:23.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:23.253+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.255000+0000) 2020-02-05T23:11:23.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:23.254+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.255756+0000) 2020-02-05T23:11:23.257 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:23.255+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.256541+0000) 2020-02-05T23:11:23.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:23.255+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.256770+0000) 2020-02-05T23:11:23.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:23.255+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.256548+0000) 2020-02-05T23:11:23.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:23.256+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.257907+0000) 2020-02-05T23:11:23.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:23.256+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.257889+0000) 2020-02-05T23:11:23.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:23.261+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.262219+0000) 2020-02-05T23:11:23.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:23.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:23.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:23.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.909459+0000) 2020-02-05T23:11:23.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:23.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:23.909423+0000) 2020-02-05T23:11:26.009 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:26.007+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:26.008533+0000) 2020-02-05T23:11:26.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:26.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:26.284968+0000) 2020-02-05T23:11:27.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:27.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:27.147998+0000) 2020-02-05T23:11:27.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:27.147+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:27.148503+0000) 2020-02-05T23:11:27.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:27.147+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:27.148485+0000) 2020-02-05T23:11:28.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:28.273+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.274062+0000) 2020-02-05T23:11:28.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:28.274+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.275393+0000) 2020-02-05T23:11:28.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:28.275+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.276163+0000) 2020-02-05T23:11:28.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:28.274+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.276043+0000) 2020-02-05T23:11:28.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:28.275+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.276685+0000) 2020-02-05T23:11:28.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:28.275+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.277048+0000) 2020-02-05T23:11:28.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:28.275+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.277292+0000) 2020-02-05T23:11:28.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:28.276+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.278024+0000) 2020-02-05T23:11:28.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:28.276+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.277856+0000) 2020-02-05T23:11:28.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:28.276+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.278186+0000) 2020-02-05T23:11:28.280 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:28.277+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.278467+0000) 2020-02-05T23:11:28.280 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:28.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.279412+0000) 2020-02-05T23:11:28.280 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:28.277+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.278617+0000) 2020-02-05T23:11:28.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:28.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:28.284674+0000) 2020-02-05T23:11:28.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:28.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:29.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:29.201+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:29.201812+0000) 2020-02-05T23:11:29.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:29.534+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:29.535302+0000) 2020-02-05T23:11:30.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:30.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:30.142209+0000) 2020-02-05T23:11:30.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:30.379+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:30.379800+0000) 2020-02-05T23:11:30.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:30.392+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:30.393185+0000) 2020-02-05T23:11:32.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:32.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:32.395761+0000) 2020-02-05T23:11:32.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:32.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:32.747551+0000) 2020-02-05T23:11:33.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:33.295+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.296543+0000) 2020-02-05T23:11:33.298 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:33.296+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.297136+0000) 2020-02-05T23:11:33.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:33.296+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.297544+0000) 2020-02-05T23:11:33.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:33.297+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.298185+0000) 2020-02-05T23:11:33.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:33.297+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.298962+0000) 2020-02-05T23:11:33.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:33.297+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.298871+0000) 2020-02-05T23:11:33.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:33.298+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.299908+0000) 2020-02-05T23:11:33.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:33.299+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.300752+0000) 2020-02-05T23:11:33.302 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:33.300+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.301063+0000) 2020-02-05T23:11:33.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:33.299+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.301272+0000) 2020-02-05T23:11:33.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:33.300+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.301897+0000) 2020-02-05T23:11:33.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:33.300+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.301716+0000) 2020-02-05T23:11:33.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:33.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.303808+0000) 2020-02-05T23:11:33.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:33.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.304045+0000) 2020-02-05T23:11:33.313 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:33.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:33.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:33.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.909601+0000) 2020-02-05T23:11:33.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:33.908+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:33.909757+0000) 2020-02-05T23:11:36.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:36.008+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:36.008787+0000) 2020-02-05T23:11:36.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:36.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:36.285195+0000) 2020-02-05T23:11:37.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:37.147+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:37.148254+0000) 2020-02-05T23:11:37.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:37.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:37.148734+0000) 2020-02-05T23:11:37.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:37.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:37.148731+0000) 2020-02-05T23:11:38.320 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:38.318+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.319465+0000) 2020-02-05T23:11:38.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:38.318+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.320042+0000) 2020-02-05T23:11:38.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:38.319+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.320496+0000) 2020-02-05T23:11:38.322 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:38.319+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.320954+0000) 2020-02-05T23:11:38.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:38.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.321197+0000) 2020-02-05T23:11:38.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:38.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.321756+0000) 2020-02-05T23:11:38.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:38.322+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.322770+0000) 2020-02-05T23:11:38.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:38.322+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.322977+0000) 2020-02-05T23:11:38.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:38.323+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.323724+0000) 2020-02-05T23:11:38.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:38.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.324589+0000) 2020-02-05T23:11:38.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:38.324+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.324536+0000) 2020-02-05T23:11:38.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:38.324+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.324615+0000) 2020-02-05T23:11:38.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:38.324+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.325140+0000) 2020-02-05T23:11:38.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:38.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:38.329411+0000) 2020-02-05T23:11:38.335 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:38.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137382 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:39.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:39.209+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:39.210024+0000) 2020-02-05T23:11:39.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:39.535+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:39.535530+0000) 2020-02-05T23:11:40.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:40.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:40.142460+0000) 2020-02-05T23:11:40.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:40.379+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:40.380055+0000) 2020-02-05T23:11:40.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:40.392+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:40.393414+0000) 2020-02-05T23:11:42.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:42.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:42.395995+0000) 2020-02-05T23:11:42.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:42.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:42.747778+0000) 2020-02-05T23:11:43.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:43.338+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.339805+0000) 2020-02-05T23:11:43.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:43.340+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.341236+0000) 2020-02-05T23:11:43.343 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:43.341+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.342469+0000) 2020-02-05T23:11:43.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:43.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.342640+0000) 2020-02-05T23:11:43.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:43.342+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.342985+0000) 2020-02-05T23:11:43.344 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:43.342+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.343335+0000) 2020-02-05T23:11:43.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:43.343+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.343982+0000) 2020-02-05T23:11:43.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:43.343+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.344423+0000) 2020-02-05T23:11:43.345 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:43.343+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.344320+0000) 2020-02-05T23:11:43.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:43.344+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.345294+0000) 2020-02-05T23:11:43.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:43.345+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.345644+0000) 2020-02-05T23:11:43.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:43.346+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.347227+0000) 2020-02-05T23:11:43.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:43.348+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.349147+0000) 2020-02-05T23:11:43.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:43.345+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.346313+0000) 2020-02-05T23:11:43.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:43.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:43.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:43.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.909802+0000) 2020-02-05T23:11:43.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:43.909+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:43.909944+0000) 2020-02-05T23:11:44.902 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:11:44.909 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:11:46.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:46.008+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:46.009042+0000) 2020-02-05T23:11:46.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:46.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:46.285426+0000) 2020-02-05T23:11:47.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:47.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:47.148510+0000) 2020-02-05T23:11:47.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:47.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:47.148949+0000) 2020-02-05T23:11:47.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:47.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:47.148993+0000) 2020-02-05T23:11:48.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:48.360+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.361881+0000) 2020-02-05T23:11:48.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:48.362+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.362831+0000) 2020-02-05T23:11:48.364 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:48.362+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.363243+0000) 2020-02-05T23:11:48.364 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:48.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.363629+0000) 2020-02-05T23:11:48.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:48.363+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.364417+0000) 2020-02-05T23:11:48.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:48.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.364440+0000) 2020-02-05T23:11:48.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:48.364+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.364989+0000) 2020-02-05T23:11:48.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:48.364+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.365146+0000) 2020-02-05T23:11:48.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:48.364+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.365747+0000) 2020-02-05T23:11:48.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:48.365+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.366162+0000) 2020-02-05T23:11:48.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:48.365+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.365910+0000) 2020-02-05T23:11:48.367 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:48.366+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.366994+0000) 2020-02-05T23:11:48.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:48.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.368030+0000) 2020-02-05T23:11:48.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:48.370+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:48.371180+0000) 2020-02-05T23:11:48.381 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:48.379+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:49.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:49.217+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:49.218285+0000) 2020-02-05T23:11:49.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:49.534+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:49.535777+0000) 2020-02-05T23:11:50.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:50.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:50.142695+0000) 2020-02-05T23:11:50.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:50.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:50.380354+0000) 2020-02-05T23:11:50.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:50.393+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:50.393626+0000) 2020-02-05T23:11:52.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:52.395+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:52.396260+0000) 2020-02-05T23:11:52.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:52.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:52.748035+0000) 2020-02-05T23:11:53.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:53.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.386927+0000) 2020-02-05T23:11:53.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:53.387+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.387483+0000) 2020-02-05T23:11:53.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:53.386+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.387799+0000) 2020-02-05T23:11:53.389 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:53.387+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.388361+0000) 2020-02-05T23:11:53.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:53.387+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.388621+0000) 2020-02-05T23:11:53.390 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:53.387+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.388976+0000) 2020-02-05T23:11:53.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:53.389+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.389544+0000) 2020-02-05T23:11:53.391 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:53.388+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.389683+0000) 2020-02-05T23:11:53.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:53.389+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.390222+0000) 2020-02-05T23:11:53.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:53.390+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.390896+0000) 2020-02-05T23:11:53.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:53.391+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.391479+0000) 2020-02-05T23:11:53.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:53.390+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.391757+0000) 2020-02-05T23:11:53.393 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:53.391+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.391745+0000) 2020-02-05T23:11:53.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:53.394+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.395607+0000) 2020-02-05T23:11:53.401 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:53.400+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:53.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:53.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.910025+0000) 2020-02-05T23:11:53.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:53.909+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:53.910160+0000) 2020-02-05T23:11:56.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:56.009+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:56.009286+0000) 2020-02-05T23:11:56.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:56.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:56.285655+0000) 2020-02-05T23:11:57.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:57.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:57.148697+0000) 2020-02-05T23:11:57.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:57.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:57.149174+0000) 2020-02-05T23:11:57.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:57.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:57.149235+0000) 2020-02-05T23:11:58.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:11:58.405+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.405871+0000) 2020-02-05T23:11:58.408 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:11:58.407+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.407256+0000) 2020-02-05T23:11:58.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:11:58.407+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.408335+0000) 2020-02-05T23:11:58.409 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:11:58.408+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.409048+0000) 2020-02-05T23:11:58.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:11:58.409+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.409867+0000) 2020-02-05T23:11:58.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:11:58.409+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.410179+0000) 2020-02-05T23:11:58.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:58.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.410179+0000) 2020-02-05T23:11:58.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:11:58.411+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.411459+0000) 2020-02-05T23:11:58.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:11:58.411+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.411800+0000) 2020-02-05T23:11:58.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:11:58.412+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.412136+0000) 2020-02-05T23:11:58.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:11:58.411+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.412351+0000) 2020-02-05T23:11:58.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:11:58.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.412768+0000) 2020-02-05T23:11:58.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:58.416+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.416623+0000) 2020-02-05T23:11:58.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:58.416+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:58.416901+0000) 2020-02-05T23:11:58.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:11:58.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:11:59.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:59.224+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:59.224930+0000) 2020-02-05T23:11:59.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:11:59.535+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:11:59.536020+0000) 2020-02-05T23:12:00.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:00.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:00.142922+0000) 2020-02-05T23:12:00.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:00.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:00.380627+0000) 2020-02-05T23:12:00.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:00.393+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:00.393871+0000) 2020-02-05T23:12:02.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:02.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:02.396521+0000) 2020-02-05T23:12:02.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:02.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:02.748261+0000) 2020-02-05T23:12:03.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:03.425+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.426847+0000) 2020-02-05T23:12:03.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:03.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.428046+0000) 2020-02-05T23:12:03.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:03.428+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.428808+0000) 2020-02-05T23:12:03.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:03.429+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.430024+0000) 2020-02-05T23:12:03.431 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:03.428+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.429631+0000) 2020-02-05T23:12:03.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:03.429+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.430357+0000) 2020-02-05T23:12:03.432 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:03.430+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.431347+0000) 2020-02-05T23:12:03.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:03.430+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.431343+0000) 2020-02-05T23:12:03.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:03.429+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.430967+0000) 2020-02-05T23:12:03.434 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:03.430+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.431221+0000) 2020-02-05T23:12:03.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:03.430+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.431942+0000) 2020-02-05T23:12:03.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:03.431+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.432335+0000) 2020-02-05T23:12:03.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:03.431+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.432222+0000) 2020-02-05T23:12:03.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:03.436+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.437185+0000) 2020-02-05T23:12:03.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:03.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:03.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:03.909+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.910375+0000) 2020-02-05T23:12:03.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:03.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:03.910347+0000) 2020-02-05T23:12:06.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:06.009+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:06.009546+0000) 2020-02-05T23:12:06.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:06.284+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:06.285920+0000) 2020-02-05T23:12:07.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:07.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:07.148949+0000) 2020-02-05T23:12:07.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:07.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:07.149389+0000) 2020-02-05T23:12:07.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:07.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:07.149494+0000) 2020-02-05T23:12:08.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:08.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.448791+0000) 2020-02-05T23:12:08.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:08.448+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.449349+0000) 2020-02-05T23:12:08.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:08.448+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.449857+0000) 2020-02-05T23:12:08.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:08.449+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.450334+0000) 2020-02-05T23:12:08.452 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:08.450+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.451042+0000) 2020-02-05T23:12:08.453 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:08.450+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.451460+0000) 2020-02-05T23:12:08.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:08.450+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.452049+0000) 2020-02-05T23:12:08.454 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:08.451+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.452536+0000) 2020-02-05T23:12:08.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:08.452+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.452844+0000) 2020-02-05T23:12:08.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:08.453+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.453776+0000) 2020-02-05T23:12:08.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:08.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.453699+0000) 2020-02-05T23:12:08.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:08.453+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.454323+0000) 2020-02-05T23:12:08.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:08.454+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.454905+0000) 2020-02-05T23:12:08.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:08.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:08.461313+0000) 2020-02-05T23:12:08.464 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:08.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137635 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:09.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:09.230+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:09.231761+0000) 2020-02-05T23:12:09.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:09.535+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:09.536328+0000) 2020-02-05T23:12:10.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:10.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:10.143146+0000) 2020-02-05T23:12:10.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:10.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:10.380918+0000) 2020-02-05T23:12:10.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:10.393+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:10.394141+0000) 2020-02-05T23:12:12.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:12.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:12.396761+0000) 2020-02-05T23:12:12.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:12.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:12.748489+0000) 2020-02-05T23:12:13.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:13.467+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.468891+0000) 2020-02-05T23:12:13.470 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:13.468+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.469350+0000) 2020-02-05T23:12:13.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:13.470+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.470513+0000) 2020-02-05T23:12:13.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:13.470+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.470843+0000) 2020-02-05T23:12:13.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:13.471+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.471513+0000) 2020-02-05T23:12:13.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:13.471+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.471800+0000) 2020-02-05T23:12:13.473 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:13.471+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.472337+0000) 2020-02-05T23:12:13.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:13.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.472604+0000) 2020-02-05T23:12:13.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:13.471+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.472753+0000) 2020-02-05T23:12:13.474 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:13.471+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.472959+0000) 2020-02-05T23:12:13.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:13.472+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.473049+0000) 2020-02-05T23:12:13.475 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:13.472+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.473508+0000) 2020-02-05T23:12:13.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:13.474+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.474823+0000) 2020-02-05T23:12:13.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:13.477+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.478250+0000) 2020-02-05T23:12:13.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:13.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137679 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:13.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:13.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.910557+0000) 2020-02-05T23:12:13.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:13.909+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:13.910584+0000) 2020-02-05T23:12:15.043 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:12:15.050 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:12:16.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:16.008+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:16.009841+0000) 2020-02-05T23:12:16.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:16.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:16.286156+0000) 2020-02-05T23:12:17.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:17.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:17.149258+0000) 2020-02-05T23:12:17.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:17.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:17.149662+0000) 2020-02-05T23:12:17.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:17.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:17.149746+0000) 2020-02-05T23:12:18.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:18.489+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.490512+0000) 2020-02-05T23:12:18.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:18.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.491306+0000) 2020-02-05T23:12:18.493 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:18.491+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.492283+0000) 2020-02-05T23:12:18.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:18.491+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.492966+0000) 2020-02-05T23:12:18.494 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:18.492+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.493384+0000) 2020-02-05T23:12:18.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:18.492+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.493821+0000) 2020-02-05T23:12:18.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:18.493+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.494286+0000) 2020-02-05T23:12:18.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:18.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.495199+0000) 2020-02-05T23:12:18.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:18.493+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.494695+0000) 2020-02-05T23:12:18.497 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:18.494+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.495784+0000) 2020-02-05T23:12:18.497 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:18.494+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.495955+0000) 2020-02-05T23:12:18.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:18.494+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.496084+0000) 2020-02-05T23:12:18.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:18.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.496646+0000) 2020-02-05T23:12:18.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:18.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:18.499611+0000) 2020-02-05T23:12:18.506 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:18.503+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:19.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:19.239+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:19.239896+0000) 2020-02-05T23:12:19.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:19.535+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:19.536553+0000) 2020-02-05T23:12:20.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:20.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:20.143386+0000) 2020-02-05T23:12:20.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:20.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:20.381151+0000) 2020-02-05T23:12:20.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:20.393+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:20.394359+0000) 2020-02-05T23:12:22.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:22.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:22.397001+0000) 2020-02-05T23:12:22.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:22.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:22.748787+0000) 2020-02-05T23:12:23.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:23.510+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.511283+0000) 2020-02-05T23:12:23.513 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:23.511+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.512550+0000) 2020-02-05T23:12:23.514 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:23.511+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.512948+0000) 2020-02-05T23:12:23.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:23.512+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.513561+0000) 2020-02-05T23:12:23.515 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:23.511+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.513294+0000) 2020-02-05T23:12:23.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:23.512+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.514021+0000) 2020-02-05T23:12:23.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:23.513+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.514531+0000) 2020-02-05T23:12:23.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:23.513+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.514785+0000) 2020-02-05T23:12:23.518 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:23.513+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.515011+0000) 2020-02-05T23:12:23.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:23.514+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.515991+0000) 2020-02-05T23:12:23.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:23.514+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.516043+0000) 2020-02-05T23:12:23.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:23.514+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.515316+0000) 2020-02-05T23:12:23.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:23.515+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.516805+0000) 2020-02-05T23:12:23.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:23.520+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.521241+0000) 2020-02-05T23:12:23.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:23.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137762 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:23.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:23.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.910765+0000) 2020-02-05T23:12:23.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:23.910+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:23.910827+0000) 2020-02-05T23:12:26.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:26.009+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:26.010100+0000) 2020-02-05T23:12:26.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:26.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:26.286387+0000) 2020-02-05T23:12:27.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:27.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:27.149495+0000) 2020-02-05T23:12:27.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:27.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:27.149945+0000) 2020-02-05T23:12:27.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:27.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:27.149985+0000) 2020-02-05T23:12:28.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:28.532+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.533645+0000) 2020-02-05T23:12:28.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:28.532+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.534220+0000) 2020-02-05T23:12:28.536 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:28.533+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.534714+0000) 2020-02-05T23:12:28.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:28.534+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.535621+0000) 2020-02-05T23:12:28.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:28.534+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.535964+0000) 2020-02-05T23:12:28.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:28.535+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.536501+0000) 2020-02-05T23:12:28.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:28.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.536810+0000) 2020-02-05T23:12:28.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:28.535+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.537372+0000) 2020-02-05T23:12:28.539 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:28.536+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.537942+0000) 2020-02-05T23:12:28.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:28.537+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.538620+0000) 2020-02-05T23:12:28.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:28.537+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.539000+0000) 2020-02-05T23:12:28.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:28.537+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.539275+0000) 2020-02-05T23:12:28.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:28.537+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.539188+0000) 2020-02-05T23:12:28.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:28.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:28.540953+0000) 2020-02-05T23:12:28.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:28.546+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:29.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:29.247+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:29.248012+0000) 2020-02-05T23:12:29.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:29.536+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:29.536806+0000) 2020-02-05T23:12:30.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:30.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:30.143572+0000) 2020-02-05T23:12:30.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:30.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:30.381403+0000) 2020-02-05T23:12:30.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:30.394+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:30.394584+0000) 2020-02-05T23:12:32.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:32.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:32.397257+0000) 2020-02-05T23:12:32.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:32.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:32.749030+0000) 2020-02-05T23:12:33.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:33.551+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.552951+0000) 2020-02-05T23:12:33.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:33.552+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.553589+0000) 2020-02-05T23:12:33.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:33.552+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.554030+0000) 2020-02-05T23:12:33.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:33.554+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.555841+0000) 2020-02-05T23:12:33.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:33.555+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.556314+0000) 2020-02-05T23:12:33.558 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:33.555+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.556947+0000) 2020-02-05T23:12:33.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:33.555+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.557205+0000) 2020-02-05T23:12:33.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:33.557+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.558512+0000) 2020-02-05T23:12:33.559 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:33.557+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.558576+0000) 2020-02-05T23:12:33.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:33.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.559244+0000) 2020-02-05T23:12:33.560 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:33.558+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.559361+0000) 2020-02-05T23:12:33.561 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:33.558+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.559968+0000) 2020-02-05T23:12:33.561 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:33.559+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.561013+0000) 2020-02-05T23:12:33.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:33.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.564908+0000) 2020-02-05T23:12:33.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:33.570+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:33.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:33.910+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.911032+0000) 2020-02-05T23:12:33.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:33.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:33.911018+0000) 2020-02-05T23:12:36.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:36.009+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:36.010420+0000) 2020-02-05T23:12:36.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:36.285+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:36.286625+0000) 2020-02-05T23:12:37.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:37.148+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:37.149781+0000) 2020-02-05T23:12:37.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:37.148+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:37.150182+0000) 2020-02-05T23:12:37.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:37.148+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:37.150191+0000) 2020-02-05T23:12:38.578 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:38.576+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.577458+0000) 2020-02-05T23:12:38.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:38.577+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.578554+0000) 2020-02-05T23:12:38.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:38.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.579189+0000) 2020-02-05T23:12:38.580 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:38.578+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.579710+0000) 2020-02-05T23:12:38.581 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:38.579+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.580065+0000) 2020-02-05T23:12:38.581 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:38.579+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.580740+0000) 2020-02-05T23:12:38.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:38.579+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.580937+0000) 2020-02-05T23:12:38.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:38.580+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.581329+0000) 2020-02-05T23:12:38.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:38.579+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.581390+0000) 2020-02-05T23:12:38.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:38.580+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.581809+0000) 2020-02-05T23:12:38.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:38.581+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.582703+0000) 2020-02-05T23:12:38.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:38.581+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.582512+0000) 2020-02-05T23:12:38.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:38.582+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.583097+0000) 2020-02-05T23:12:38.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:38.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:38.584455+0000) 2020-02-05T23:12:38.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:38.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137885 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:39.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:39.259+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:39.259664+0000) 2020-02-05T23:12:39.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:39.536+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:39.537059+0000) 2020-02-05T23:12:40.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:40.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:40.143790+0000) 2020-02-05T23:12:40.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:40.380+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:40.381659+0000) 2020-02-05T23:12:40.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:40.394+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:40.394809+0000) 2020-02-05T23:12:42.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:42.396+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:42.397509+0000) 2020-02-05T23:12:42.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:42.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:42.749271+0000) 2020-02-05T23:12:43.601 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:43.599+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.600334+0000) 2020-02-05T23:12:43.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:43.599+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.601157+0000) 2020-02-05T23:12:43.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:43.600+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.602127+0000) 2020-02-05T23:12:43.603 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:43.601+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.602579+0000) 2020-02-05T23:12:43.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:43.601+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.603075+0000) 2020-02-05T23:12:43.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:43.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.603698+0000) 2020-02-05T23:12:43.605 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:43.602+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.603976+0000) 2020-02-05T23:12:43.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:43.603+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.604317+0000) 2020-02-05T23:12:43.607 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:43.603+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.605228+0000) 2020-02-05T23:12:43.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:43.603+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.605318+0000) 2020-02-05T23:12:43.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:43.603+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.604734+0000) 2020-02-05T23:12:43.608 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:43.605+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.606242+0000) 2020-02-05T23:12:43.608 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:43.604+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.606242+0000) 2020-02-05T23:12:43.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:43.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.611160+0000) 2020-02-05T23:12:43.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:43.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:43.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:43.910+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.911292+0000) 2020-02-05T23:12:43.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:43.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:43.911295+0000) 2020-02-05T23:12:45.187 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:12:45.194 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:12:46.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:46.010+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:46.010732+0000) 2020-02-05T23:12:46.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:46.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:46.286861+0000) 2020-02-05T23:12:47.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:47.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:47.150042+0000) 2020-02-05T23:12:47.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:47.149+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:47.150424+0000) 2020-02-05T23:12:47.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:47.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:47.150424+0000) 2020-02-05T23:12:48.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:48.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.623719+0000) 2020-02-05T23:12:48.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:48.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.624145+0000) 2020-02-05T23:12:48.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:48.623+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.624398+0000) 2020-02-05T23:12:48.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:48.623+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.624813+0000) 2020-02-05T23:12:48.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:48.623+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.625027+0000) 2020-02-05T23:12:48.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:48.624+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.625712+0000) 2020-02-05T23:12:48.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:48.624+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.625614+0000) 2020-02-05T23:12:48.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:48.625+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.626295+0000) 2020-02-05T23:12:48.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:48.625+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.626541+0000) 2020-02-05T23:12:48.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:48.625+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.627363+0000) 2020-02-05T23:12:48.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:48.626+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.627662+0000) 2020-02-05T23:12:48.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:48.627+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.628365+0000) 2020-02-05T23:12:48.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:48.627+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.628619+0000) 2020-02-05T23:12:48.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:48.633+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:48.634149+0000) 2020-02-05T23:12:48.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:48.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 137967 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:49.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:49.267+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:49.267849+0000) 2020-02-05T23:12:49.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:49.537+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:49.537310+0000) 2020-02-05T23:12:50.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:50.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:50.144029+0000) 2020-02-05T23:12:50.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:50.381+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:50.381901+0000) 2020-02-05T23:12:50.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:50.394+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:50.395038+0000) 2020-02-05T23:12:52.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:52.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:52.397745+0000) 2020-02-05T23:12:52.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:52.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:52.749498+0000) 2020-02-05T23:12:53.645 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:53.642+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.643512+0000) 2020-02-05T23:12:53.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:53.644+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.644514+0000) 2020-02-05T23:12:53.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:53.645+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.645160+0000) 2020-02-05T23:12:53.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:53.645+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.645596+0000) 2020-02-05T23:12:53.646 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:53.645+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.645742+0000) 2020-02-05T23:12:53.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:53.646+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.646151+0000) 2020-02-05T23:12:53.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:53.645+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.647082+0000) 2020-02-05T23:12:53.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:53.646+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.647779+0000) 2020-02-05T23:12:53.649 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:53.647+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.647878+0000) 2020-02-05T23:12:53.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:53.648+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.648322+0000) 2020-02-05T23:12:53.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:53.647+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.648948+0000) 2020-02-05T23:12:53.650 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:53.647+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.648844+0000) 2020-02-05T23:12:53.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:53.647+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.649365+0000) 2020-02-05T23:12:53.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:53.652+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.652682+0000) 2020-02-05T23:12:53.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:53.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138011 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:53.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:53.911+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.911511+0000) 2020-02-05T23:12:53.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:53.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:53.911502+0000) 2020-02-05T23:12:56.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:56.010+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:56.010997+0000) 2020-02-05T23:12:56.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:56.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:56.287133+0000) 2020-02-05T23:12:57.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:57.149+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:57.150305+0000) 2020-02-05T23:12:57.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:57.149+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:57.150687+0000) 2020-02-05T23:12:57.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:57.149+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:57.150718+0000) 2020-02-05T23:12:58.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:12:58.663+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.664597+0000) 2020-02-05T23:12:58.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:12:58.663+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.664819+0000) 2020-02-05T23:12:58.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:58.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.666554+0000) 2020-02-05T23:12:58.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:12:58.664+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.666303+0000) 2020-02-05T23:12:58.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:58.665+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.667045+0000) 2020-02-05T23:12:58.669 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:12:58.666+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.667762+0000) 2020-02-05T23:12:58.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:12:58.666+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.668040+0000) 2020-02-05T23:12:58.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:12:58.667+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.668430+0000) 2020-02-05T23:12:58.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:12:58.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.668881+0000) 2020-02-05T23:12:58.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:12:58.667+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.669231+0000) 2020-02-05T23:12:58.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:12:58.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.669750+0000) 2020-02-05T23:12:58.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:12:58.668+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.670046+0000) 2020-02-05T23:12:58.672 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:12:58.669+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.671289+0000) 2020-02-05T23:12:58.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:58.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:58.676440+0000) 2020-02-05T23:12:58.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:12:58.679+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:12:59.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:59.274+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:59.275955+0000) 2020-02-05T23:12:59.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:12:59.536+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:12:59.537524+0000) 2020-02-05T23:13:00.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:00.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:00.144312+0000) 2020-02-05T23:13:00.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:00.381+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:00.382148+0000) 2020-02-05T23:13:00.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:00.394+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:00.395292+0000) 2020-02-05T23:13:02.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:02.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:02.398007+0000) 2020-02-05T23:13:02.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:02.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:02.749751+0000) 2020-02-05T23:13:03.696 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:03.686+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.688223+0000) 2020-02-05T23:13:03.696 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:03.687+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.689270+0000) 2020-02-05T23:13:03.697 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:03.688+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.690125+0000) 2020-02-05T23:13:03.697 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:03.689+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.690889+0000) 2020-02-05T23:13:03.697 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:03.689+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.691241+0000) 2020-02-05T23:13:03.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:03.691+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.692690+0000) 2020-02-05T23:13:03.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:03.685+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.686714+0000) 2020-02-05T23:13:03.707 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:03.686+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.687465+0000) 2020-02-05T23:13:03.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:03.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.689165+0000) 2020-02-05T23:13:03.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:03.688+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.689748+0000) 2020-02-05T23:13:03.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:03.689+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.690484+0000) 2020-02-05T23:13:03.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:03.690+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.691722+0000) 2020-02-05T23:13:03.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:03.691+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.692433+0000) 2020-02-05T23:13:03.711 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:03.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138094 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:03.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:03.697+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.698299+0000) 2020-02-05T23:13:03.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:03.910+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.911745+0000) 2020-02-05T23:13:03.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:03.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:03.911722+0000) 2020-02-05T23:13:06.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:06.010+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:06.011259+0000) 2020-02-05T23:13:06.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:06.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:06.287366+0000) 2020-02-05T23:13:07.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:07.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:07.150539+0000) 2020-02-05T23:13:07.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:07.150+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:07.150966+0000) 2020-02-05T23:13:07.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:07.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:07.150976+0000) 2020-02-05T23:13:08.713 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:08.711+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.712492+0000) 2020-02-05T23:13:08.714 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:08.712+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.713411+0000) 2020-02-05T23:13:08.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:08.712+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.713826+0000) 2020-02-05T23:13:08.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:08.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.714523+0000) 2020-02-05T23:13:08.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:08.714+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.714924+0000) 2020-02-05T23:13:08.716 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:08.715+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.715491+0000) 2020-02-05T23:13:08.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:08.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.716250+0000) 2020-02-05T23:13:08.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:08.716+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.716415+0000) 2020-02-05T23:13:08.718 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:08.716+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.717045+0000) 2020-02-05T23:13:08.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:08.716+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.717699+0000) 2020-02-05T23:13:08.719 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:08.717+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.717688+0000) 2020-02-05T23:13:08.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:08.717+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.718154+0000) 2020-02-05T23:13:08.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:08.717+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.718234+0000) 2020-02-05T23:13:08.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:08.722+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:08.723605+0000) 2020-02-05T23:13:08.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:08.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:09.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:09.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:09.284156+0000) 2020-02-05T23:13:09.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:09.536+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:09.537724+0000) 2020-02-05T23:13:10.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:10.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:10.144569+0000) 2020-02-05T23:13:10.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:10.381+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:10.382394+0000) 2020-02-05T23:13:10.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:10.394+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:10.395533+0000) 2020-02-05T23:13:12.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:12.397+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:12.398248+0000) 2020-02-05T23:13:12.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:12.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:12.749994+0000) 2020-02-05T23:13:13.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:13.732+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.733771+0000) 2020-02-05T23:13:13.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:13.733+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.734246+0000) 2020-02-05T23:13:13.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:13.734+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.735337+0000) 2020-02-05T23:13:13.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:13.735+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.736461+0000) 2020-02-05T23:13:13.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:13.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.736923+0000) 2020-02-05T23:13:13.738 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:13.736+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.737375+0000) 2020-02-05T23:13:13.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:13.736+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.737775+0000) 2020-02-05T23:13:13.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:13.737+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.738304+0000) 2020-02-05T23:13:13.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:13.738+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.738817+0000) 2020-02-05T23:13:13.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:13.738+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.739262+0000) 2020-02-05T23:13:13.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:13.739+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.740308+0000) 2020-02-05T23:13:13.741 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:13.739+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.740056+0000) 2020-02-05T23:13:13.741 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:13.740+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.740691+0000) 2020-02-05T23:13:13.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:13.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.746832+0000) 2020-02-05T23:13:13.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:13.750+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:13.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:13.911+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.911970+0000) 2020-02-05T23:13:13.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:13.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:13.911965+0000) 2020-02-05T23:13:15.317 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:13:15.323 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:13:16.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:16.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:16.011490+0000) 2020-02-05T23:13:16.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:16.286+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:16.287600+0000) 2020-02-05T23:13:17.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:17.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:17.150777+0000) 2020-02-05T23:13:17.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:17.150+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:17.151202+0000) 2020-02-05T23:13:17.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:17.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:17.151254+0000) 2020-02-05T23:13:18.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:18.755+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.756509+0000) 2020-02-05T23:13:18.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:18.756+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.757438+0000) 2020-02-05T23:13:18.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:18.757+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.758327+0000) 2020-02-05T23:13:18.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:18.758+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.758855+0000) 2020-02-05T23:13:18.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:18.757+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.758865+0000) 2020-02-05T23:13:18.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:18.757+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.758795+0000) 2020-02-05T23:13:18.761 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:18.758+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.759157+0000) 2020-02-05T23:13:18.761 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:18.759+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.759899+0000) 2020-02-05T23:13:18.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:18.759+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.759844+0000) 2020-02-05T23:13:18.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:18.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.759540+0000) 2020-02-05T23:13:18.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:18.760+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.760796+0000) 2020-02-05T23:13:18.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:18.760+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.760950+0000) 2020-02-05T23:13:18.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:18.760+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.761277+0000) 2020-02-05T23:13:18.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:18.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:18.763692+0000) 2020-02-05T23:13:18.773 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:18.771+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138216 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:19.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:19.290+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:19.291523+0000) 2020-02-05T23:13:19.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:19.537+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:19.537921+0000) 2020-02-05T23:13:20.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:20.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:20.144809+0000) 2020-02-05T23:13:20.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:20.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:20.382621+0000) 2020-02-05T23:13:20.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:20.395+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:20.395776+0000) 2020-02-05T23:13:22.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:22.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:22.398532+0000) 2020-02-05T23:13:22.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:22.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:22.750192+0000) 2020-02-05T23:13:23.778 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:23.777+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.777579+0000) 2020-02-05T23:13:23.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:23.778+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.779265+0000) 2020-02-05T23:13:23.780 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:23.778+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.779401+0000) 2020-02-05T23:13:23.781 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:23.779+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.780307+0000) 2020-02-05T23:13:23.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:23.779+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.780956+0000) 2020-02-05T23:13:23.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:23.781+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.781466+0000) 2020-02-05T23:13:23.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:23.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.781298+0000) 2020-02-05T23:13:23.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:23.780+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.781715+0000) 2020-02-05T23:13:23.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:23.781+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.782367+0000) 2020-02-05T23:13:23.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:23.782+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.783097+0000) 2020-02-05T23:13:23.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:23.782+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.783078+0000) 2020-02-05T23:13:23.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:23.783+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.783483+0000) 2020-02-05T23:13:23.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:23.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.786780+0000) 2020-02-05T23:13:23.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:23.786+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.787123+0000) 2020-02-05T23:13:23.795 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:23.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:23.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:23.911+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.912202+0000) 2020-02-05T23:13:23.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:23.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:23.912182+0000) 2020-02-05T23:13:26.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:26.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:26.011787+0000) 2020-02-05T23:13:26.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:26.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:26.287853+0000) 2020-02-05T23:13:27.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:27.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:27.151034+0000) 2020-02-05T23:13:27.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:27.151+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:27.151483+0000) 2020-02-05T23:13:27.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:27.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:27.151540+0000) 2020-02-05T23:13:28.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:28.799+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.800551+0000) 2020-02-05T23:13:28.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:28.799+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.800779+0000) 2020-02-05T23:13:28.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:28.802+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.802473+0000) 2020-02-05T23:13:28.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:28.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.803230+0000) 2020-02-05T23:13:28.804 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:28.802+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.802968+0000) 2020-02-05T23:13:28.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:28.802+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.803707+0000) 2020-02-05T23:13:28.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:28.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.804531+0000) 2020-02-05T23:13:28.806 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:28.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.804883+0000) 2020-02-05T23:13:28.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:28.804+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.805296+0000) 2020-02-05T23:13:28.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:28.804+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.805836+0000) 2020-02-05T23:13:28.807 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:28.805+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.805844+0000) 2020-02-05T23:13:28.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:28.805+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.806249+0000) 2020-02-05T23:13:28.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:28.805+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.806961+0000) 2020-02-05T23:13:28.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:28.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:28.809203+0000) 2020-02-05T23:13:28.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:28.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:29.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:29.299+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:29.299759+0000) 2020-02-05T23:13:29.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:29.537+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:29.538126+0000) 2020-02-05T23:13:30.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:30.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:30.145035+0000) 2020-02-05T23:13:30.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:30.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:30.382867+0000) 2020-02-05T23:13:30.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:30.395+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:30.396000+0000) 2020-02-05T23:13:32.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:32.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:32.398771+0000) 2020-02-05T23:13:32.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:32.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:32.750451+0000) 2020-02-05T23:13:33.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:33.822+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.823969+0000) 2020-02-05T23:13:33.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:33.824+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.825197+0000) 2020-02-05T23:13:33.826 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:33.825+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.825688+0000) 2020-02-05T23:13:33.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:33.824+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.825821+0000) 2020-02-05T23:13:33.827 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:33.825+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.826353+0000) 2020-02-05T23:13:33.827 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:33.826+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.826784+0000) 2020-02-05T23:13:33.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:33.826+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.827446+0000) 2020-02-05T23:13:33.829 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:33.826+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.826979+0000) 2020-02-05T23:13:33.829 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:33.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.827976+0000) 2020-02-05T23:13:33.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:33.827+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.828735+0000) 2020-02-05T23:13:33.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:33.828+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.828830+0000) 2020-02-05T23:13:33.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:33.827+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.828950+0000) 2020-02-05T23:13:33.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:33.828+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.829586+0000) 2020-02-05T23:13:33.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:33.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.835231+0000) 2020-02-05T23:13:33.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:33.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:33.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:33.912+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.912419+0000) 2020-02-05T23:13:33.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:33.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:33.912416+0000) 2020-02-05T23:13:36.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:36.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:36.012037+0000) 2020-02-05T23:13:36.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:36.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:36.288112+0000) 2020-02-05T23:13:37.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:37.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:37.151285+0000) 2020-02-05T23:13:37.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:37.151+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:37.151732+0000) 2020-02-05T23:13:37.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:37.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:37.151769+0000) 2020-02-05T23:13:38.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:38.842+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.843631+0000) 2020-02-05T23:13:38.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:38.843+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.844987+0000) 2020-02-05T23:13:38.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:38.845+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.845413+0000) 2020-02-05T23:13:38.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:38.845+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.846368+0000) 2020-02-05T23:13:38.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:38.846+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.846807+0000) 2020-02-05T23:13:38.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:38.846+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.847373+0000) 2020-02-05T23:13:38.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:38.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.847833+0000) 2020-02-05T23:13:38.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:38.847+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.848082+0000) 2020-02-05T23:13:38.849 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:38.847+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.848407+0000) 2020-02-05T23:13:38.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:38.849+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.849512+0000) 2020-02-05T23:13:38.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:38.848+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.850025+0000) 2020-02-05T23:13:38.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:38.849+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.850213+0000) 2020-02-05T23:13:38.851 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:38.849+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.850082+0000) 2020-02-05T23:13:38.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:38.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:38.853909+0000) 2020-02-05T23:13:38.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:38.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:39.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:39.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:39.307635+0000) 2020-02-05T23:13:39.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:39.538+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:39.538330+0000) 2020-02-05T23:13:40.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:40.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:40.145296+0000) 2020-02-05T23:13:40.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:40.383+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:40.383107+0000) 2020-02-05T23:13:40.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:40.396+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:40.396272+0000) 2020-02-05T23:13:42.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:42.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:42.399029+0000) 2020-02-05T23:13:42.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:42.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:42.750670+0000) 2020-02-05T23:13:43.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:43.864+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.865777+0000) 2020-02-05T23:13:43.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:43.865+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.866432+0000) 2020-02-05T23:13:43.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:43.866+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.868088+0000) 2020-02-05T23:13:43.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:43.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.868382+0000) 2020-02-05T23:13:43.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:43.867+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.868777+0000) 2020-02-05T23:13:43.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:43.867+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.868673+0000) 2020-02-05T23:13:43.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:43.867+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.869127+0000) 2020-02-05T23:13:43.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:43.868+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.869489+0000) 2020-02-05T23:13:43.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:43.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.869848+0000) 2020-02-05T23:13:43.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:43.868+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.870201+0000) 2020-02-05T23:13:43.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:43.870+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.871742+0000) 2020-02-05T23:13:43.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:43.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.872058+0000) 2020-02-05T23:13:43.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:43.870+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.872167+0000) 2020-02-05T23:13:43.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:43.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.874525+0000) 2020-02-05T23:13:43.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:43.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:43.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:43.912+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.912642+0000) 2020-02-05T23:13:43.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:43.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:43.912670+0000) 2020-02-05T23:13:45.481 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:13:45.487 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:13:46.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:46.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:46.012306+0000) 2020-02-05T23:13:46.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:46.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:46.288347+0000) 2020-02-05T23:13:47.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:47.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:47.151537+0000) 2020-02-05T23:13:47.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:47.150+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:47.151959+0000) 2020-02-05T23:13:47.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:47.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:47.152018+0000) 2020-02-05T23:13:48.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:48.886+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.886247+0000) 2020-02-05T23:13:48.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:48.887+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.887358+0000) 2020-02-05T23:13:48.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:48.886+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.887739+0000) 2020-02-05T23:13:48.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:48.888+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.888125+0000) 2020-02-05T23:13:48.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:48.887+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.889155+0000) 2020-02-05T23:13:48.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:48.889+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.889447+0000) 2020-02-05T23:13:48.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:48.888+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.889996+0000) 2020-02-05T23:13:48.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:48.890+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.890441+0000) 2020-02-05T23:13:48.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:48.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.890929+0000) 2020-02-05T23:13:48.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:48.890+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.891532+0000) 2020-02-05T23:13:48.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:48.890+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.892041+0000) 2020-02-05T23:13:48.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:48.892+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.892374+0000) 2020-02-05T23:13:48.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:48.890+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.892276+0000) 2020-02-05T23:13:48.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:48.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:48.896558+0000) 2020-02-05T23:13:48.903 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:48.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138462 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:49.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:49.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:49.315716+0000) 2020-02-05T23:13:49.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:49.538+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:49.538534+0000) 2020-02-05T23:13:50.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:50.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:50.145528+0000) 2020-02-05T23:13:50.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:50.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:50.383364+0000) 2020-02-05T23:13:50.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:50.396+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:50.396540+0000) 2020-02-05T23:13:52.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:52.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:52.399301+0000) 2020-02-05T23:13:52.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:52.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:52.750900+0000) 2020-02-05T23:13:53.908 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:53.906+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.907383+0000) 2020-02-05T23:13:53.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:53.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.907959+0000) 2020-02-05T23:13:53.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:53.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.909017+0000) 2020-02-05T23:13:53.910 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:53.908+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.909599+0000) 2020-02-05T23:13:53.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:53.909+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.910780+0000) 2020-02-05T23:13:53.912 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:53.909+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.910501+0000) 2020-02-05T23:13:53.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:53.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.911329+0000) 2020-02-05T23:13:53.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:53.910+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.911627+0000) 2020-02-05T23:13:53.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:53.911+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.912831+0000) 2020-02-05T23:13:53.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:53.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.912885+0000) 2020-02-05T23:13:53.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:53.911+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.913075+0000) 2020-02-05T23:13:53.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:53.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.913540+0000) 2020-02-05T23:13:53.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:53.912+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.913726+0000) 2020-02-05T23:13:53.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:53.913+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.914566+0000) 2020-02-05T23:13:53.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:53.913+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.914343+0000) 2020-02-05T23:13:53.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:53.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:53.916943+0000) 2020-02-05T23:13:53.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:53.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:56.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:56.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:56.012552+0000) 2020-02-05T23:13:56.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:56.287+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:56.288592+0000) 2020-02-05T23:13:57.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:57.150+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:57.151785+0000) 2020-02-05T23:13:57.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:57.150+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:57.152224+0000) 2020-02-05T23:13:57.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:57.150+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:57.152240+0000) 2020-02-05T23:13:58.932 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:13:58.930+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.931794+0000) 2020-02-05T23:13:58.933 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:13:58.931+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.932363+0000) 2020-02-05T23:13:58.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:13:58.931+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.932668+0000) 2020-02-05T23:13:58.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:13:58.932+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.933660+0000) 2020-02-05T23:13:58.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:13:58.932+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.933973+0000) 2020-02-05T23:13:58.935 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:13:58.932+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.934133+0000) 2020-02-05T23:13:58.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:58.933+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.934071+0000) 2020-02-05T23:13:58.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:13:58.933+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.934832+0000) 2020-02-05T23:13:58.936 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:13:58.933+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.935155+0000) 2020-02-05T23:13:58.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:13:58.933+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.935280+0000) 2020-02-05T23:13:58.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:13:58.935+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.936530+0000) 2020-02-05T23:13:58.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:13:58.934+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.936263+0000) 2020-02-05T23:13:58.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:58.935+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.936617+0000) 2020-02-05T23:13:58.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:58.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:58.940345+0000) 2020-02-05T23:13:58.946 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:13:58.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:13:59.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:59.322+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:59.323608+0000) 2020-02-05T23:13:59.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:13:59.537+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:13:59.538736+0000) 2020-02-05T23:14:00.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:00.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:00.145754+0000) 2020-02-05T23:14:00.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:00.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:00.383743+0000) 2020-02-05T23:14:00.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:00.395+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:00.396770+0000) 2020-02-05T23:14:02.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:02.398+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:02.399556+0000) 2020-02-05T23:14:02.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:02.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:02.751149+0000) 2020-02-05T23:14:03.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:03.912+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.913050+0000) 2020-02-05T23:14:03.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:03.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.913130+0000) 2020-02-05T23:14:03.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:03.950+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.951654+0000) 2020-02-05T23:14:03.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:03.951+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.952156+0000) 2020-02-05T23:14:03.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:03.952+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.953302+0000) 2020-02-05T23:14:03.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:03.952+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.953713+0000) 2020-02-05T23:14:03.955 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:03.953+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.954266+0000) 2020-02-05T23:14:03.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:03.953+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.955064+0000) 2020-02-05T23:14:03.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:03.954+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.955161+0000) 2020-02-05T23:14:03.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:03.954+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.955349+0000) 2020-02-05T23:14:03.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:03.954+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.955852+0000) 2020-02-05T23:14:03.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:03.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.956363+0000) 2020-02-05T23:14:03.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:03.955+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.957156+0000) 2020-02-05T23:14:03.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:03.956+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.957612+0000) 2020-02-05T23:14:03.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:03.956+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.957759+0000) 2020-02-05T23:14:03.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:03.963+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:03.964535+0000) 2020-02-05T23:14:03.968 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:03.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:06.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:06.012+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:06.012889+0000) 2020-02-05T23:14:06.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:06.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:06.288823+0000) 2020-02-05T23:14:07.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:07.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:07.152033+0000) 2020-02-05T23:14:07.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:07.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:07.152473+0000) 2020-02-05T23:14:07.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:07.151+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:07.152531+0000) 2020-02-05T23:14:08.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:08.973+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.974762+0000) 2020-02-05T23:14:08.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:08.974+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.975504+0000) 2020-02-05T23:14:08.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:08.974+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.975923+0000) 2020-02-05T23:14:08.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:08.974+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.976010+0000) 2020-02-05T23:14:08.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:08.975+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.976654+0000) 2020-02-05T23:14:08.978 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:08.975+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.977150+0000) 2020-02-05T23:14:08.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:08.976+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.977593+0000) 2020-02-05T23:14:08.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:08.977+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.978243+0000) 2020-02-05T23:14:08.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:08.977+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.978224+0000) 2020-02-05T23:14:08.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:08.976+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.978174+0000) 2020-02-05T23:14:08.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:08.977+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.978604+0000) 2020-02-05T23:14:08.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:08.977+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.979022+0000) 2020-02-05T23:14:08.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:08.978+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.979369+0000) 2020-02-05T23:14:08.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:08.984+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:08.985384+0000) 2020-02-05T23:14:08.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:08.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:09.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:09.329+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:09.330486+0000) 2020-02-05T23:14:09.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:09.537+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:09.538969+0000) 2020-02-05T23:14:10.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:10.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:10.145990+0000) 2020-02-05T23:14:10.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:10.383+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:10.383983+0000) 2020-02-05T23:14:10.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:10.396+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:10.396994+0000) 2020-02-05T23:14:12.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:12.399+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:12.399804+0000) 2020-02-05T23:14:12.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:12.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:12.751421+0000) 2020-02-05T23:14:13.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:13.912+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.913332+0000) 2020-02-05T23:14:13.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:13.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.913381+0000) 2020-02-05T23:14:13.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:13.992+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.993470+0000) 2020-02-05T23:14:13.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:13.993+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.994268+0000) 2020-02-05T23:14:13.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:13.993+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.994925+0000) 2020-02-05T23:14:13.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:13.994+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.995405+0000) 2020-02-05T23:14:13.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:13.994+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.995788+0000) 2020-02-05T23:14:13.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:13.994+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.996125+0000) 2020-02-05T23:14:13.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:13.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.996477+0000) 2020-02-05T23:14:13.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:13.995+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.996875+0000) 2020-02-05T23:14:13.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:13.995+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.996673+0000) 2020-02-05T23:14:13.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:13.996+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.997480+0000) 2020-02-05T23:14:13.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:13.996+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.997677+0000) 2020-02-05T23:14:13.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:13.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.998606+0000) 2020-02-05T23:14:14.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:13.998+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:13.999493+0000) 2020-02-05T23:14:14.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:14.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:14.005286+0000) 2020-02-05T23:14:14.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:14.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:15.620 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:14:15.626 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:14:16.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:16.012+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:16.013126+0000) 2020-02-05T23:14:16.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:16.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:16.289053+0000) 2020-02-05T23:14:17.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:17.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:17.152278+0000) 2020-02-05T23:14:17.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:17.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:17.152692+0000) 2020-02-05T23:14:17.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:17.151+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:17.152802+0000) 2020-02-05T23:14:19.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:19.011+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.012661+0000) 2020-02-05T23:14:19.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:19.012+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.013781+0000) 2020-02-05T23:14:19.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:19.014+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.015482+0000) 2020-02-05T23:14:19.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:19.015+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.016533+0000) 2020-02-05T23:14:19.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:19.015+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.016749+0000) 2020-02-05T23:14:19.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:19.015+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.017057+0000) 2020-02-05T23:14:19.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:19.016+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.017416+0000) 2020-02-05T23:14:19.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:19.017+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.018502+0000) 2020-02-05T23:14:19.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:19.017+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.018854+0000) 2020-02-05T23:14:19.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:19.017+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.019401+0000) 2020-02-05T23:14:19.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:19.019+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.020257+0000) 2020-02-05T23:14:19.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:19.018+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.019915+0000) 2020-02-05T23:14:19.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:19.018+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.020305+0000) 2020-02-05T23:14:19.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:19.020+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.021670+0000) 2020-02-05T23:14:19.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:19.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:19.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:19.338+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.338959+0000) 2020-02-05T23:14:19.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:19.538+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:19.539176+0000) 2020-02-05T23:14:20.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:20.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:20.146259+0000) 2020-02-05T23:14:20.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:20.382+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:20.384193+0000) 2020-02-05T23:14:20.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:20.396+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:20.397249+0000) 2020-02-05T23:14:22.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:22.399+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:22.400082+0000) 2020-02-05T23:14:22.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:22.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:22.751654+0000) 2020-02-05T23:14:23.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:23.913+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:23.913559+0000) 2020-02-05T23:14:23.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:23.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:23.913664+0000) 2020-02-05T23:14:24.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:24.033+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.034654+0000) 2020-02-05T23:14:24.036 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:24.034+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.035322+0000) 2020-02-05T23:14:24.036 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:24.034+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.035891+0000) 2020-02-05T23:14:24.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:24.035+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.036800+0000) 2020-02-05T23:14:24.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:24.036+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.037278+0000) 2020-02-05T23:14:24.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:24.036+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.038235+0000) 2020-02-05T23:14:24.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:24.037+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.038642+0000) 2020-02-05T23:14:24.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:24.037+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.039210+0000) 2020-02-05T23:14:24.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:24.038+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.039520+0000) 2020-02-05T23:14:24.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:24.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.039827+0000) 2020-02-05T23:14:24.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:24.040+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.041452+0000) 2020-02-05T23:14:24.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:24.039+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.040866+0000) 2020-02-05T23:14:24.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:24.040+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.041698+0000) 2020-02-05T23:14:24.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:24.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:24.044317+0000) 2020-02-05T23:14:24.054 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:24.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138750 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:26.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:26.011+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:26.013344+0000) 2020-02-05T23:14:26.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:26.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:26.289336+0000) 2020-02-05T23:14:27.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:27.151+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:27.152510+0000) 2020-02-05T23:14:27.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:27.151+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:27.152918+0000) 2020-02-05T23:14:27.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:27.151+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:27.153182+0000) 2020-02-05T23:14:29.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:29.057+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.058930+0000) 2020-02-05T23:14:29.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:29.058+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.059753+0000) 2020-02-05T23:14:29.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:29.059+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.061142+0000) 2020-02-05T23:14:29.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:29.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.061364+0000) 2020-02-05T23:14:29.062 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:29.060+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.061573+0000) 2020-02-05T23:14:29.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:29.060+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.062108+0000) 2020-02-05T23:14:29.063 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:29.060+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.062308+0000) 2020-02-05T23:14:29.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:29.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.061953+0000) 2020-02-05T23:14:29.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:29.061+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.062217+0000) 2020-02-05T23:14:29.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:29.061+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.062612+0000) 2020-02-05T23:14:29.064 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:29.061+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.062744+0000) 2020-02-05T23:14:29.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:29.062+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.063556+0000) 2020-02-05T23:14:29.067 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:29.064+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.066139+0000) 2020-02-05T23:14:29.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:29.068+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.069624+0000) 2020-02-05T23:14:29.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:29.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:29.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:29.347+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.347419+0000) 2020-02-05T23:14:29.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:29.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:29.539386+0000) 2020-02-05T23:14:30.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:30.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:30.146522+0000) 2020-02-05T23:14:30.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:30.383+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:30.384491+0000) 2020-02-05T23:14:30.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:30.397+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:30.397516+0000) 2020-02-05T23:14:32.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:32.399+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:32.400276+0000) 2020-02-05T23:14:32.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:32.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:32.751895+0000) 2020-02-05T23:14:33.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:33.913+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:33.913789+0000) 2020-02-05T23:14:33.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:33.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:33.913895+0000) 2020-02-05T23:14:34.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:34.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.079105+0000) 2020-02-05T23:14:34.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:34.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.079783+0000) 2020-02-05T23:14:34.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:34.079+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.080457+0000) 2020-02-05T23:14:34.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:34.079+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.080941+0000) 2020-02-05T23:14:34.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:34.080+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.081411+0000) 2020-02-05T23:14:34.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:34.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.081978+0000) 2020-02-05T23:14:34.083 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:34.081+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.082487+0000) 2020-02-05T23:14:34.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:34.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.082766+0000) 2020-02-05T23:14:34.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:34.081+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.083000+0000) 2020-02-05T23:14:34.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:34.082+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.083318+0000) 2020-02-05T23:14:34.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:34.083+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.084453+0000) 2020-02-05T23:14:34.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:34.083+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.084617+0000) 2020-02-05T23:14:34.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:34.083+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.085370+0000) 2020-02-05T23:14:34.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:34.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:34.089207+0000) 2020-02-05T23:14:34.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:34.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:36.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:36.013+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:36.013613+0000) 2020-02-05T23:14:36.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:36.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:36.289578+0000) 2020-02-05T23:14:37.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:37.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:37.152768+0000) 2020-02-05T23:14:37.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:37.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:37.153107+0000) 2020-02-05T23:14:37.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:37.152+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:37.153390+0000) 2020-02-05T23:14:39.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:39.098+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.099613+0000) 2020-02-05T23:14:39.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:39.100+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.100965+0000) 2020-02-05T23:14:39.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:39.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.100892+0000) 2020-02-05T23:14:39.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:39.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.101416+0000) 2020-02-05T23:14:39.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:39.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.102319+0000) 2020-02-05T23:14:39.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:39.101+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.101946+0000) 2020-02-05T23:14:39.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:39.102+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.102722+0000) 2020-02-05T23:14:39.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:39.102+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.103347+0000) 2020-02-05T23:14:39.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:39.102+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.103570+0000) 2020-02-05T23:14:39.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:39.102+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.103857+0000) 2020-02-05T23:14:39.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:39.103+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.104523+0000) 2020-02-05T23:14:39.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:39.104+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.105228+0000) 2020-02-05T23:14:39.110 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:39.104+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.105545+0000) 2020-02-05T23:14:39.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:39.106+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.107998+0000) 2020-02-05T23:14:39.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:39.114+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:39.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:39.355+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.355817+0000) 2020-02-05T23:14:39.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:39.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:39.539597+0000) 2020-02-05T23:14:40.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:40.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:40.146698+0000) 2020-02-05T23:14:40.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:40.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:40.384743+0000) 2020-02-05T23:14:40.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:40.397+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:40.397745+0000) 2020-02-05T23:14:42.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:42.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:42.400507+0000) 2020-02-05T23:14:42.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:42.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:42.752124+0000) 2020-02-05T23:14:43.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:43.914+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:43.914064+0000) 2020-02-05T23:14:43.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:43.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:43.914121+0000) 2020-02-05T23:14:44.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:44.121+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.121971+0000) 2020-02-05T23:14:44.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:44.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.122275+0000) 2020-02-05T23:14:44.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:44.122+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.122665+0000) 2020-02-05T23:14:44.124 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:44.123+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.123419+0000) 2020-02-05T23:14:44.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:44.122+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.123121+0000) 2020-02-05T23:14:44.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:44.123+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.123941+0000) 2020-02-05T23:14:44.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:44.124+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.124205+0000) 2020-02-05T23:14:44.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:44.123+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.124395+0000) 2020-02-05T23:14:44.126 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:44.124+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.125044+0000) 2020-02-05T23:14:44.126 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:44.125+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.125653+0000) 2020-02-05T23:14:44.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:44.125+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.126020+0000) 2020-02-05T23:14:44.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:44.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.126962+0000) 2020-02-05T23:14:44.128 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:44.126+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.127120+0000) 2020-02-05T23:14:44.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:44.132+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:44.132356+0000) 2020-02-05T23:14:44.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:44.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138913 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:45.758 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:14:45.764 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:14:46.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:46.013+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:46.013853+0000) 2020-02-05T23:14:46.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:46.288+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:46.289847+0000) 2020-02-05T23:14:47.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:47.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:47.153129+0000) 2020-02-05T23:14:47.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:47.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:47.153364+0000) 2020-02-05T23:14:47.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:47.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:47.153658+0000) 2020-02-05T23:14:49.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:49.141+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.142252+0000) 2020-02-05T23:14:49.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:49.143+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.143865+0000) 2020-02-05T23:14:49.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:49.143+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.144384+0000) 2020-02-05T23:14:49.145 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:49.143+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.144739+0000) 2020-02-05T23:14:49.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:49.144+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.145041+0000) 2020-02-05T23:14:49.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:49.145+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.145803+0000) 2020-02-05T23:14:49.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:49.146+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.146555+0000) 2020-02-05T23:14:49.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:49.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.147187+0000) 2020-02-05T23:14:49.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:49.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.147925+0000) 2020-02-05T23:14:49.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:49.147+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.148209+0000) 2020-02-05T23:14:49.149 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:49.147+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.148568+0000) 2020-02-05T23:14:49.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:49.148+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.149425+0000) 2020-02-05T23:14:49.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:49.151+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.152939+0000) 2020-02-05T23:14:49.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:49.152+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.153189+0000) 2020-02-05T23:14:49.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:49.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138953 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:49.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:49.363+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.364192+0000) 2020-02-05T23:14:49.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:49.538+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:49.539802+0000) 2020-02-05T23:14:50.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:50.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:50.146905+0000) 2020-02-05T23:14:50.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:50.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:50.384974+0000) 2020-02-05T23:14:50.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:50.396+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:50.397969+0000) 2020-02-05T23:14:52.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:52.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:52.400766+0000) 2020-02-05T23:14:52.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:52.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:52.752372+0000) 2020-02-05T23:14:53.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:53.913+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:53.914343+0000) 2020-02-05T23:14:53.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:53.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:53.914356+0000) 2020-02-05T23:14:54.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:54.162+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.163770+0000) 2020-02-05T23:14:54.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:54.164+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.165233+0000) 2020-02-05T23:14:54.166 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:54.165+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.165920+0000) 2020-02-05T23:14:54.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:54.165+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.166330+0000) 2020-02-05T23:14:54.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:54.166+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.166695+0000) 2020-02-05T23:14:54.168 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:54.165+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.167038+0000) 2020-02-05T23:14:54.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:54.166+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.167536+0000) 2020-02-05T23:14:54.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:54.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.167924+0000) 2020-02-05T23:14:54.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:54.168+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.168498+0000) 2020-02-05T23:14:54.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:54.167+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.168669+0000) 2020-02-05T23:14:54.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:54.168+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.169311+0000) 2020-02-05T23:14:54.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:54.169+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.169887+0000) 2020-02-05T23:14:54.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:54.169+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.170163+0000) 2020-02-05T23:14:54.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:54.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:54.171848+0000) 2020-02-05T23:14:54.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:54.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 138995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:56.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:56.013+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:56.014117+0000) 2020-02-05T23:14:56.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:56.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:56.290120+0000) 2020-02-05T23:14:57.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:57.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:57.153385+0000) 2020-02-05T23:14:57.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:57.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:57.153664+0000) 2020-02-05T23:14:57.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:57.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:57.153889+0000) 2020-02-05T23:14:59.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:14:59.186+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.186633+0000) 2020-02-05T23:14:59.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:14:59.186+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.187096+0000) 2020-02-05T23:14:59.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:14:59.186+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.187656+0000) 2020-02-05T23:14:59.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:14:59.187+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.188056+0000) 2020-02-05T23:14:59.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:14:59.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.188731+0000) 2020-02-05T23:14:59.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:14:59.188+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.189174+0000) 2020-02-05T23:14:59.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:14:59.188+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.189453+0000) 2020-02-05T23:14:59.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:14:59.189+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.190183+0000) 2020-02-05T23:14:59.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:14:59.190+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.191104+0000) 2020-02-05T23:14:59.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:14:59.190+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.191503+0000) 2020-02-05T23:14:59.192 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:14:59.190+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.191402+0000) 2020-02-05T23:14:59.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:59.190+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.191944+0000) 2020-02-05T23:14:59.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:59.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.195066+0000) 2020-02-05T23:14:59.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:59.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.195341+0000) 2020-02-05T23:14:59.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:14:59.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139034 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:14:59.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:59.371+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.372423+0000) 2020-02-05T23:14:59.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:14:59.538+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:14:59.539954+0000) 2020-02-05T23:15:00.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:00.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:00.147169+0000) 2020-02-05T23:15:00.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:00.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:00.385227+0000) 2020-02-05T23:15:00.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:00.397+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:00.398228+0000) 2020-02-05T23:15:02.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:02.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:02.401030+0000) 2020-02-05T23:15:02.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:02.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:02.752607+0000) 2020-02-05T23:15:03.915 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:03.913+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:03.914615+0000) 2020-02-05T23:15:03.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:03.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:03.914620+0000) 2020-02-05T23:15:04.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:04.207+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.208169+0000) 2020-02-05T23:15:04.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:04.207+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.208792+0000) 2020-02-05T23:15:04.211 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:04.209+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.210505+0000) 2020-02-05T23:15:04.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:04.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.211257+0000) 2020-02-05T23:15:04.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:04.211+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.211520+0000) 2020-02-05T23:15:04.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:04.211+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.211795+0000) 2020-02-05T23:15:04.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:04.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.212402+0000) 2020-02-05T23:15:04.213 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:04.211+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.212475+0000) 2020-02-05T23:15:04.214 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:04.211+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.212762+0000) 2020-02-05T23:15:04.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:04.212+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.213116+0000) 2020-02-05T23:15:04.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:04.212+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.213176+0000) 2020-02-05T23:15:04.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:04.213+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.213924+0000) 2020-02-05T23:15:04.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:04.213+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.214094+0000) 2020-02-05T23:15:04.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:04.220+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:04.221131+0000) 2020-02-05T23:15:04.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:04.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139075 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:06.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:06.014+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:06.014369+0000) 2020-02-05T23:15:06.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:06.289+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:06.290373+0000) 2020-02-05T23:15:07.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:07.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:07.153679+0000) 2020-02-05T23:15:07.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:07.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:07.153920+0000) 2020-02-05T23:15:07.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:07.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:07.154123+0000) 2020-02-05T23:15:09.232 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:09.230+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.231433+0000) 2020-02-05T23:15:09.233 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:09.231+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.232530+0000) 2020-02-05T23:15:09.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:09.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.232963+0000) 2020-02-05T23:15:09.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:09.232+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.233349+0000) 2020-02-05T23:15:09.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:09.233+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.233798+0000) 2020-02-05T23:15:09.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:09.234+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.235415+0000) 2020-02-05T23:15:09.236 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:09.234+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.235513+0000) 2020-02-05T23:15:09.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:09.234+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.235376+0000) 2020-02-05T23:15:09.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:09.235+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.236042+0000) 2020-02-05T23:15:09.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:09.235+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.236308+0000) 2020-02-05T23:15:09.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:09.235+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.236722+0000) 2020-02-05T23:15:09.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:09.236+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.237213+0000) 2020-02-05T23:15:09.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:09.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.239961+0000) 2020-02-05T23:15:09.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:09.239+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.240214+0000) 2020-02-05T23:15:09.246 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:09.244+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139115 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:09.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:09.379+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.380540+0000) 2020-02-05T23:15:09.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:09.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:09.540187+0000) 2020-02-05T23:15:10.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:10.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:10.147375+0000) 2020-02-05T23:15:10.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:10.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:10.385476+0000) 2020-02-05T23:15:10.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:10.397+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:10.398454+0000) 2020-02-05T23:15:12.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:12.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:12.401266+0000) 2020-02-05T23:15:12.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:12.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:12.752870+0000) 2020-02-05T23:15:13.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:13.914+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:13.914849+0000) 2020-02-05T23:15:13.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:13.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:13.914844+0000) 2020-02-05T23:15:14.251 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:14.249+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.250685+0000) 2020-02-05T23:15:14.252 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:14.250+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.251359+0000) 2020-02-05T23:15:14.252 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:14.250+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.251820+0000) 2020-02-05T23:15:14.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:14.251+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.252870+0000) 2020-02-05T23:15:14.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:14.252+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.254031+0000) 2020-02-05T23:15:14.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:14.252+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.254373+0000) 2020-02-05T23:15:14.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:14.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.254760+0000) 2020-02-05T23:15:14.256 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:14.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.255248+0000) 2020-02-05T23:15:14.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:14.255+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.256294+0000) 2020-02-05T23:15:14.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:14.255+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.256519+0000) 2020-02-05T23:15:14.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:14.255+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.256836+0000) 2020-02-05T23:15:14.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:14.256+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.257461+0000) 2020-02-05T23:15:14.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:14.256+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.257483+0000) 2020-02-05T23:15:14.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:14.261+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:14.262660+0000) 2020-02-05T23:15:14.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:14.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:15.896 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:15:15.902 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:15:16.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:16.013+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:16.014598+0000) 2020-02-05T23:15:16.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:16.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:16.290619+0000) 2020-02-05T23:15:17.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:17.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:17.153929+0000) 2020-02-05T23:15:17.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:17.152+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:17.154285+0000) 2020-02-05T23:15:17.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:17.152+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:17.154349+0000) 2020-02-05T23:15:19.272 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:19.269+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.270860+0000) 2020-02-05T23:15:19.272 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:19.270+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.271900+0000) 2020-02-05T23:15:19.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:19.271+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.272335+0000) 2020-02-05T23:15:19.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:19.272+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.273108+0000) 2020-02-05T23:15:19.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:19.272+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.273781+0000) 2020-02-05T23:15:19.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:19.272+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.274033+0000) 2020-02-05T23:15:19.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:19.273+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.274522+0000) 2020-02-05T23:15:19.276 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:19.274+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.275469+0000) 2020-02-05T23:15:19.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:19.274+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.276080+0000) 2020-02-05T23:15:19.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:19.275+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.276743+0000) 2020-02-05T23:15:19.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:19.275+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.277209+0000) 2020-02-05T23:15:19.278 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:19.276+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.277881+0000) 2020-02-05T23:15:19.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:19.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.278294+0000) 2020-02-05T23:15:19.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:19.280+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.281071+0000) 2020-02-05T23:15:19.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:19.285+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139196 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:19.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:19.388+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.389018+0000) 2020-02-05T23:15:19.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:19.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:19.540388+0000) 2020-02-05T23:15:20.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:20.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:20.147589+0000) 2020-02-05T23:15:20.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:20.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:20.385724+0000) 2020-02-05T23:15:20.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:20.398+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:20.398682+0000) 2020-02-05T23:15:22.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:22.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:22.401532+0000) 2020-02-05T23:15:22.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:22.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:22.753134+0000) 2020-02-05T23:15:23.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:23.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:23.915125+0000) 2020-02-05T23:15:23.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:23.914+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:23.915132+0000) 2020-02-05T23:15:24.295 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:24.292+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.293948+0000) 2020-02-05T23:15:24.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:24.292+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.294304+0000) 2020-02-05T23:15:24.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:24.294+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.295476+0000) 2020-02-05T23:15:24.297 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:24.294+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.295881+0000) 2020-02-05T23:15:24.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:24.295+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.296443+0000) 2020-02-05T23:15:24.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:24.296+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.297151+0000) 2020-02-05T23:15:24.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:24.296+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.297658+0000) 2020-02-05T23:15:24.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:24.296+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.297851+0000) 2020-02-05T23:15:24.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:24.296+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.297998+0000) 2020-02-05T23:15:24.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:24.297+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.298524+0000) 2020-02-05T23:15:24.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:24.297+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.298703+0000) 2020-02-05T23:15:24.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:24.298+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.299469+0000) 2020-02-05T23:15:24.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:24.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.300790+0000) 2020-02-05T23:15:24.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:24.301+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:24.302498+0000) 2020-02-05T23:15:24.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:24.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:26.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:26.013+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:26.014882+0000) 2020-02-05T23:15:26.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:26.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:26.290849+0000) 2020-02-05T23:15:27.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:27.152+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:27.154168+0000) 2020-02-05T23:15:27.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:27.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:27.154636+0000) 2020-02-05T23:15:27.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:27.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:27.154608+0000) 2020-02-05T23:15:29.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:29.313+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.314477+0000) 2020-02-05T23:15:29.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:29.313+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.315083+0000) 2020-02-05T23:15:29.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:29.314+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.315486+0000) 2020-02-05T23:15:29.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:29.315+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.316057+0000) 2020-02-05T23:15:29.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:29.315+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.316734+0000) 2020-02-05T23:15:29.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:29.316+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.317368+0000) 2020-02-05T23:15:29.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:29.316+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.317839+0000) 2020-02-05T23:15:29.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:29.316+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.318400+0000) 2020-02-05T23:15:29.320 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:29.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.319459+0000) 2020-02-05T23:15:29.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:29.317+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.319172+0000) 2020-02-05T23:15:29.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:29.318+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.320094+0000) 2020-02-05T23:15:29.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:29.319+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.320527+0000) 2020-02-05T23:15:29.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:29.319+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.320873+0000) 2020-02-05T23:15:29.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:29.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.323392+0000) 2020-02-05T23:15:29.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:29.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:29.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:29.397+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.397356+0000) 2020-02-05T23:15:29.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:29.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:29.540564+0000) 2020-02-05T23:15:30.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:30.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:30.147802+0000) 2020-02-05T23:15:30.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:30.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:30.385970+0000) 2020-02-05T23:15:30.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:30.398+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:30.398910+0000) 2020-02-05T23:15:32.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:32.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:32.401789+0000) 2020-02-05T23:15:32.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:32.753+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:32.753368+0000) 2020-02-05T23:15:33.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:33.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:33.915368+0000) 2020-02-05T23:15:33.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:33.915+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:33.915410+0000) 2020-02-05T23:15:34.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:34.335+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.337232+0000) 2020-02-05T23:15:34.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:34.336+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.338084+0000) 2020-02-05T23:15:34.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:34.336+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.338237+0000) 2020-02-05T23:15:34.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:34.337+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.338484+0000) 2020-02-05T23:15:34.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:34.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.339241+0000) 2020-02-05T23:15:34.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:34.339+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.340067+0000) 2020-02-05T23:15:34.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:34.339+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.340554+0000) 2020-02-05T23:15:34.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:34.339+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.340417+0000) 2020-02-05T23:15:34.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:34.339+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.340865+0000) 2020-02-05T23:15:34.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:34.339+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.341205+0000) 2020-02-05T23:15:34.342 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:34.340+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.341432+0000) 2020-02-05T23:15:34.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:34.340+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.341768+0000) 2020-02-05T23:15:34.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:34.344+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.345322+0000) 2020-02-05T23:15:34.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:34.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:34.346323+0000) 2020-02-05T23:15:34.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:34.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:36.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:36.014+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:36.015137+0000) 2020-02-05T23:15:36.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:36.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:36.291108+0000) 2020-02-05T23:15:37.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:37.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:37.154420+0000) 2020-02-05T23:15:37.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:37.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:37.154855+0000) 2020-02-05T23:15:37.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:37.153+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:37.154913+0000) 2020-02-05T23:15:39.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:39.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.358252+0000) 2020-02-05T23:15:39.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:39.358+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.358965+0000) 2020-02-05T23:15:39.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:39.359+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.359996+0000) 2020-02-05T23:15:39.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:39.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.360585+0000) 2020-02-05T23:15:39.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:39.360+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.360955+0000) 2020-02-05T23:15:39.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:39.360+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.361947+0000) 2020-02-05T23:15:39.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:39.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.363623+0000) 2020-02-05T23:15:39.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:39.362+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.363704+0000) 2020-02-05T23:15:39.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:39.364+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.364335+0000) 2020-02-05T23:15:39.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:39.364+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.364355+0000) 2020-02-05T23:15:39.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:39.363+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.364699+0000) 2020-02-05T23:15:39.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:39.363+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.365135+0000) 2020-02-05T23:15:39.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:39.365+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.365273+0000) 2020-02-05T23:15:39.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:39.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.369258+0000) 2020-02-05T23:15:39.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:39.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:39.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:39.404+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.404577+0000) 2020-02-05T23:15:39.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:39.540+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:39.540783+0000) 2020-02-05T23:15:40.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:40.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:40.148130+0000) 2020-02-05T23:15:40.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:40.384+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:40.386216+0000) 2020-02-05T23:15:40.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:40.398+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:40.399186+0000) 2020-02-05T23:15:42.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:42.400+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:42.402011+0000) 2020-02-05T23:15:42.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:42.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:42.753604+0000) 2020-02-05T23:15:43.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:43.914+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:43.915646+0000) 2020-02-05T23:15:43.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:43.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:43.915587+0000) 2020-02-05T23:15:44.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:44.379+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.380371+0000) 2020-02-05T23:15:44.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:44.380+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.381413+0000) 2020-02-05T23:15:44.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:44.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.381998+0000) 2020-02-05T23:15:44.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:44.381+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.382344+0000) 2020-02-05T23:15:44.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:44.382+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.383143+0000) 2020-02-05T23:15:44.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:44.382+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.383386+0000) 2020-02-05T23:15:44.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:44.382+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.383634+0000) 2020-02-05T23:15:44.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:44.383+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.384254+0000) 2020-02-05T23:15:44.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:44.383+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.385099+0000) 2020-02-05T23:15:44.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:44.384+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.385839+0000) 2020-02-05T23:15:44.388 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:44.384+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.386016+0000) 2020-02-05T23:15:44.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:44.385+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.386683+0000) 2020-02-05T23:15:44.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:44.384+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.386092+0000) 2020-02-05T23:15:44.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:44.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:44.391263+0000) 2020-02-05T23:15:44.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:44.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139399 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:46.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:46.014+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:46.015456+0000) 2020-02-05T23:15:46.109 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:15:46.116 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:15:46.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:46.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:46.291341+0000) 2020-02-05T23:15:47.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:47.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:47.154707+0000) 2020-02-05T23:15:47.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:47.154+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:47.155064+0000) 2020-02-05T23:15:47.156 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:47.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:47.155199+0000) 2020-02-05T23:15:49.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:49.401+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.402507+0000) 2020-02-05T23:15:49.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:49.402+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.403371+0000) 2020-02-05T23:15:49.404 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:49.402+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.403596+0000) 2020-02-05T23:15:49.405 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:49.402+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.404041+0000) 2020-02-05T23:15:49.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:49.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.404764+0000) 2020-02-05T23:15:49.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:49.403+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.404815+0000) 2020-02-05T23:15:49.406 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:49.404+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.405609+0000) 2020-02-05T23:15:49.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:49.404+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.406254+0000) 2020-02-05T23:15:49.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:49.405+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.406566+0000) 2020-02-05T23:15:49.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:49.406+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.407516+0000) 2020-02-05T23:15:49.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:49.406+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.407907+0000) 2020-02-05T23:15:49.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:49.407+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.408451+0000) 2020-02-05T23:15:49.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:49.407+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.408697+0000) 2020-02-05T23:15:49.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:49.411+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.412332+0000) 2020-02-05T23:15:49.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:49.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.412401+0000) 2020-02-05T23:15:49.419 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:49.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139440 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:49.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:49.539+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:49.540990+0000) 2020-02-05T23:15:50.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:50.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:50.148410+0000) 2020-02-05T23:15:50.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:50.385+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:50.386473+0000) 2020-02-05T23:15:50.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:50.398+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:50.399424+0000) 2020-02-05T23:15:52.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:52.401+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:52.402269+0000) 2020-02-05T23:15:52.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:52.753+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:52.753909+0000) 2020-02-05T23:15:53.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:53.914+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:53.915875+0000) 2020-02-05T23:15:53.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:53.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:53.915829+0000) 2020-02-05T23:15:54.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:54.424+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.426023+0000) 2020-02-05T23:15:54.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:54.425+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.426363+0000) 2020-02-05T23:15:54.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:54.425+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.426529+0000) 2020-02-05T23:15:54.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:54.425+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.426622+0000) 2020-02-05T23:15:54.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:54.425+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.427028+0000) 2020-02-05T23:15:54.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:54.424+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.425838+0000) 2020-02-05T23:15:54.429 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:54.427+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.428604+0000) 2020-02-05T23:15:54.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:54.426+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.427732+0000) 2020-02-05T23:15:54.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:54.426+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.428231+0000) 2020-02-05T23:15:54.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:54.427+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.428999+0000) 2020-02-05T23:15:54.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:54.428+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.429166+0000) 2020-02-05T23:15:54.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:54.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.429463+0000) 2020-02-05T23:15:54.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:54.428+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.430271+0000) 2020-02-05T23:15:54.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:54.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:54.435258+0000) 2020-02-05T23:15:54.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:54.439+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139481 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:56.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:56.015+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:56.015721+0000) 2020-02-05T23:15:56.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:56.290+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:56.291598+0000) 2020-02-05T23:15:57.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:57.153+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:57.154927+0000) 2020-02-05T23:15:57.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:57.153+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:57.155306+0000) 2020-02-05T23:15:57.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:57.154+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:57.155478+0000) 2020-02-05T23:15:59.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:59.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.412593+0000) 2020-02-05T23:15:59.449 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:15:59.447+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.448252+0000) 2020-02-05T23:15:59.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:15:59.447+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.449066+0000) 2020-02-05T23:15:59.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:59.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.449318+0000) 2020-02-05T23:15:59.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:15:59.448+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.449690+0000) 2020-02-05T23:15:59.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:15:59.449+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.450172+0000) 2020-02-05T23:15:59.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:15:59.449+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.450456+0000) 2020-02-05T23:15:59.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:59.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.450777+0000) 2020-02-05T23:15:59.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:15:59.449+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.450807+0000) 2020-02-05T23:15:59.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:15:59.449+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.451291+0000) 2020-02-05T23:15:59.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:15:59.450+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.451562+0000) 2020-02-05T23:15:59.455 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:15:59.450+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.452129+0000) 2020-02-05T23:15:59.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:15:59.450+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.451781+0000) 2020-02-05T23:15:59.456 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:15:59.451+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.452722+0000) 2020-02-05T23:15:59.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:59.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.455098+0000) 2020-02-05T23:15:59.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:15:59.461+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:15:59.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:15:59.540+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:15:59.541219+0000) 2020-02-05T23:16:00.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:00.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:00.148658+0000) 2020-02-05T23:16:00.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:00.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:00.386616+0000) 2020-02-05T23:16:00.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:00.398+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:00.399620+0000) 2020-02-05T23:16:02.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:02.401+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:02.402514+0000) 2020-02-05T23:16:02.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:02.753+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:02.754176+0000) 2020-02-05T23:16:03.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:03.915+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:03.916140+0000) 2020-02-05T23:16:03.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:03.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:03.916136+0000) 2020-02-05T23:16:04.469 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:04.467+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.468752+0000) 2020-02-05T23:16:04.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:04.469+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.469628+0000) 2020-02-05T23:16:04.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:04.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.470748+0000) 2020-02-05T23:16:04.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:04.471+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.471868+0000) 2020-02-05T23:16:04.475 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:04.471+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.472263+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:04.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.473811+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:04.468+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.469317+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:04.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.470122+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:04.469+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.470407+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:04.472+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.473298+0000) 2020-02-05T23:16:04.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:04.473+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.474233+0000) 2020-02-05T23:16:04.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:04.473+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.474338+0000) 2020-02-05T23:16:04.477 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:04.474+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.474774+0000) 2020-02-05T23:16:04.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:04.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:04.481679+0000) 2020-02-05T23:16:04.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:04.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:06.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:06.015+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:06.015988+0000) 2020-02-05T23:16:06.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:06.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:06.291862+0000) 2020-02-05T23:16:07.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:07.154+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:07.155171+0000) 2020-02-05T23:16:07.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:07.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:07.155532+0000) 2020-02-05T23:16:07.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:07.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:07.155725+0000) 2020-02-05T23:16:09.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:09.420+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.421054+0000) 2020-02-05T23:16:09.493 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:09.491+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.491722+0000) 2020-02-05T23:16:09.493 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:09.491+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.492323+0000) 2020-02-05T23:16:09.494 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:09.492+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.493431+0000) 2020-02-05T23:16:09.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:09.492+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.493177+0000) 2020-02-05T23:16:09.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:09.493+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.494162+0000) 2020-02-05T23:16:09.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:09.493+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.494617+0000) 2020-02-05T23:16:09.495 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:09.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.494897+0000) 2020-02-05T23:16:09.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:09.495+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.495513+0000) 2020-02-05T23:16:09.497 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:09.495+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.496199+0000) 2020-02-05T23:16:09.497 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:09.495+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.496580+0000) 2020-02-05T23:16:09.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:09.495+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.496687+0000) 2020-02-05T23:16:09.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:09.496+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.496810+0000) 2020-02-05T23:16:09.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:09.497+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.497538+0000) 2020-02-05T23:16:09.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:09.501+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.502123+0000) 2020-02-05T23:16:09.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:09.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139604 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:09.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:09.540+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:09.541448+0000) 2020-02-05T23:16:10.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:10.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:10.148978+0000) 2020-02-05T23:16:10.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:10.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:10.386877+0000) 2020-02-05T23:16:10.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:10.399+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:10.399839+0000) 2020-02-05T23:16:12.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:12.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:12.402771+0000) 2020-02-05T23:16:12.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:12.753+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:12.754403+0000) 2020-02-05T23:16:13.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:13.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:13.916322+0000) 2020-02-05T23:16:13.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:13.915+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:13.916367+0000) 2020-02-05T23:16:14.513 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:14.511+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.512630+0000) 2020-02-05T23:16:14.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:14.513+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.514678+0000) 2020-02-05T23:16:14.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:14.513+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.515036+0000) 2020-02-05T23:16:14.516 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:14.515+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.515739+0000) 2020-02-05T23:16:14.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:14.515+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.515791+0000) 2020-02-05T23:16:14.518 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:14.516+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.516961+0000) 2020-02-05T23:16:14.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:14.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.517012+0000) 2020-02-05T23:16:14.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:14.517+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.517454+0000) 2020-02-05T23:16:14.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:14.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.517683+0000) 2020-02-05T23:16:14.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:14.516+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.517379+0000) 2020-02-05T23:16:14.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:14.518+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.518588+0000) 2020-02-05T23:16:14.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:14.517+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.518349+0000) 2020-02-05T23:16:14.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:14.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.524431+0000) 2020-02-05T23:16:14.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:14.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:14.525606+0000) 2020-02-05T23:16:14.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:14.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139647 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:16.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:16.015+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:16.016275+0000) 2020-02-05T23:16:16.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:16.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:16.292130+0000) 2020-02-05T23:16:16.305 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:16:16.312 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:16:17.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:17.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:17.155424+0000) 2020-02-05T23:16:17.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:17.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:17.155775+0000) 2020-02-05T23:16:17.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:17.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:17.155990+0000) 2020-02-05T23:16:19.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:19.428+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.428634+0000) 2020-02-05T23:16:19.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:19.534+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.535907+0000) 2020-02-05T23:16:19.537 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:19.535+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.536467+0000) 2020-02-05T23:16:19.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:19.535+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.536704+0000) 2020-02-05T23:16:19.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:19.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.537009+0000) 2020-02-05T23:16:19.538 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:19.536+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.537395+0000) 2020-02-05T23:16:19.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:19.536+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.537978+0000) 2020-02-05T23:16:19.539 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:19.537+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.538115+0000) 2020-02-05T23:16:19.539 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:19.538+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.538617+0000) 2020-02-05T23:16:19.541 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:19.539+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.540002+0000) 2020-02-05T23:16:19.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:19.538+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.540013+0000) 2020-02-05T23:16:19.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:19.539+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.540317+0000) 2020-02-05T23:16:19.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:19.540+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.541047+0000) 2020-02-05T23:16:19.542 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:19.540+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.540917+0000) 2020-02-05T23:16:19.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:19.540+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.541672+0000) 2020-02-05T23:16:19.555 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:19.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139687 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:19.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:19.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:19.543752+0000) 2020-02-05T23:16:20.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:20.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:20.149194+0000) 2020-02-05T23:16:20.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:20.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:20.387113+0000) 2020-02-05T23:16:20.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:20.399+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:20.400046+0000) 2020-02-05T23:16:22.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:22.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:22.403023+0000) 2020-02-05T23:16:22.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:22.754+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:22.754637+0000) 2020-02-05T23:16:23.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:23.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:23.916588+0000) 2020-02-05T23:16:23.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:23.915+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:23.916640+0000) 2020-02-05T23:16:24.557 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:24.555+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.556776+0000) 2020-02-05T23:16:24.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:24.557+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.558331+0000) 2020-02-05T23:16:24.560 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:24.558+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.559154+0000) 2020-02-05T23:16:24.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:24.558+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.559471+0000) 2020-02-05T23:16:24.560 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:24.558+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.559694+0000) 2020-02-05T23:16:24.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:24.560+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.560958+0000) 2020-02-05T23:16:24.562 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:24.560+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.560647+0000) 2020-02-05T23:16:24.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:24.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.561556+0000) 2020-02-05T23:16:24.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:24.562+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.563230+0000) 2020-02-05T23:16:24.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:24.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.563711+0000) 2020-02-05T23:16:24.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:24.563+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.563682+0000) 2020-02-05T23:16:24.565 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:24.563+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.563872+0000) 2020-02-05T23:16:24.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:24.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.564881+0000) 2020-02-05T23:16:24.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:24.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:24.566369+0000) 2020-02-05T23:16:24.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:24.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:26.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:26.016+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:26.016541+0000) 2020-02-05T23:16:26.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:26.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:26.292389+0000) 2020-02-05T23:16:27.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:27.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:27.155724+0000) 2020-02-05T23:16:27.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:27.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:27.156066+0000) 2020-02-05T23:16:27.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:27.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:27.156229+0000) 2020-02-05T23:16:29.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:29.436+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.436304+0000) 2020-02-05T23:16:29.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:29.541+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.541885+0000) 2020-02-05T23:16:29.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:29.581+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.581772+0000) 2020-02-05T23:16:29.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:29.582+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.582855+0000) 2020-02-05T23:16:29.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:29.583+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.583673+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:29.583+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.584240+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:29.584+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.585203+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:29.584+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.585234+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:29.580+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.580863+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:29.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.581228+0000) 2020-02-05T23:16:29.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:29.582+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.582367+0000) 2020-02-05T23:16:29.589 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:29.582+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.583020+0000) 2020-02-05T23:16:29.589 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:29.583+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.583425+0000) 2020-02-05T23:16:29.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:29.584+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.584378+0000) 2020-02-05T23:16:29.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:29.584+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.584634+0000) 2020-02-05T23:16:29.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:29.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:29.591952+0000) 2020-02-05T23:16:29.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:29.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139770 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:30.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:30.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:30.149400+0000) 2020-02-05T23:16:30.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:30.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:30.387338+0000) 2020-02-05T23:16:30.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:30.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:30.400292+0000) 2020-02-05T23:16:32.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:32.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:32.403333+0000) 2020-02-05T23:16:32.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:32.754+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:32.754918+0000) 2020-02-05T23:16:33.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:33.915+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:33.916865+0000) 2020-02-05T23:16:33.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:33.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:33.916808+0000) 2020-02-05T23:16:34.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:34.603+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.604127+0000) 2020-02-05T23:16:34.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:34.603+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.604931+0000) 2020-02-05T23:16:34.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:34.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.605306+0000) 2020-02-05T23:16:34.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:34.604+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.605739+0000) 2020-02-05T23:16:34.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:34.605+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.606281+0000) 2020-02-05T23:16:34.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:34.606+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.606957+0000) 2020-02-05T23:16:34.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:34.606+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.606956+0000) 2020-02-05T23:16:34.608 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:34.606+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.607195+0000) 2020-02-05T23:16:34.609 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:34.606+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.607667+0000) 2020-02-05T23:16:34.609 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:34.606+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.607728+0000) 2020-02-05T23:16:34.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:34.608+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.608589+0000) 2020-02-05T23:16:34.610 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:34.608+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.608926+0000) 2020-02-05T23:16:34.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:34.608+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.609234+0000) 2020-02-05T23:16:34.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:34.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:34.614901+0000) 2020-02-05T23:16:34.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:34.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:36.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:36.016+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:36.016769+0000) 2020-02-05T23:16:36.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:36.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:36.292643+0000) 2020-02-05T23:16:37.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:37.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:37.155989+0000) 2020-02-05T23:16:37.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:37.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:37.156262+0000) 2020-02-05T23:16:37.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:37.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:37.156493+0000) 2020-02-05T23:16:39.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:39.443+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.444403+0000) 2020-02-05T23:16:39.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:39.541+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.542118+0000) 2020-02-05T23:16:39.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:39.624+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.625187+0000) 2020-02-05T23:16:39.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:39.626+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.626416+0000) 2020-02-05T23:16:39.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:39.625+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.626620+0000) 2020-02-05T23:16:39.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:39.626+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.627063+0000) 2020-02-05T23:16:39.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:39.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.627590+0000) 2020-02-05T23:16:39.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:39.626+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.627773+0000) 2020-02-05T23:16:39.629 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:39.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.627842+0000) 2020-02-05T23:16:39.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:39.627+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.628102+0000) 2020-02-05T23:16:39.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:39.627+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.628379+0000) 2020-02-05T23:16:39.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:39.627+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.628726+0000) 2020-02-05T23:16:39.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:39.628+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.629348+0000) 2020-02-05T23:16:39.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:39.628+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.628614+0000) 2020-02-05T23:16:39.631 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:39.628+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.628728+0000) 2020-02-05T23:16:39.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:39.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:39.635771+0000) 2020-02-05T23:16:39.640 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:39.638+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:40.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:40.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:40.149618+0000) 2020-02-05T23:16:40.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:40.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:40.387565+0000) 2020-02-05T23:16:40.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:40.399+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:40.400526+0000) 2020-02-05T23:16:42.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:42.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:42.403596+0000) 2020-02-05T23:16:42.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:42.754+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:42.755208+0000) 2020-02-05T23:16:43.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:43.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:43.917036+0000) 2020-02-05T23:16:43.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:43.916+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:43.917136+0000) 2020-02-05T23:16:44.647 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:44.645+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.646666+0000) 2020-02-05T23:16:44.648 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:44.646+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.647062+0000) 2020-02-05T23:16:44.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:44.646+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.647709+0000) 2020-02-05T23:16:44.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:44.646+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.647639+0000) 2020-02-05T23:16:44.649 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:44.647+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.648241+0000) 2020-02-05T23:16:44.650 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:44.647+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.648850+0000) 2020-02-05T23:16:44.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:44.647+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.648657+0000) 2020-02-05T23:16:44.651 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:44.648+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.650192+0000) 2020-02-05T23:16:44.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:44.648+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.650384+0000) 2020-02-05T23:16:44.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:44.649+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.650638+0000) 2020-02-05T23:16:44.652 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:44.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.650925+0000) 2020-02-05T23:16:44.652 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:44.649+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.651171+0000) 2020-02-05T23:16:44.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:44.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.651827+0000) 2020-02-05T23:16:44.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:44.656+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:44.657263+0000) 2020-02-05T23:16:44.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:44.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:46.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:46.015+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:46.017027+0000) 2020-02-05T23:16:46.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:46.291+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:46.292878+0000) 2020-02-05T23:16:46.494 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:16:46.500 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:16:47.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:47.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:47.156226+0000) 2020-02-05T23:16:47.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:47.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:47.156511+0000) 2020-02-05T23:16:47.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:47.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:47.156886+0000) 2020-02-05T23:16:49.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:49.451+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.452790+0000) 2020-02-05T23:16:49.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:49.541+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.542354+0000) 2020-02-05T23:16:49.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:49.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.666480+0000) 2020-02-05T23:16:49.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:49.666+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.667409+0000) 2020-02-05T23:16:49.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:49.667+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.668629+0000) 2020-02-05T23:16:49.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:49.668+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.669501+0000) 2020-02-05T23:16:49.673 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:49.668+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.670331+0000) 2020-02-05T23:16:49.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:49.670+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.671525+0000) 2020-02-05T23:16:49.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:49.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.670556+0000) 2020-02-05T23:16:49.675 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:49.669+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.670918+0000) 2020-02-05T23:16:49.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:49.670+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.671240+0000) 2020-02-05T23:16:49.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:49.670+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.672146+0000) 2020-02-05T23:16:49.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:49.671+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.672965+0000) 2020-02-05T23:16:49.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:49.671+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.673110+0000) 2020-02-05T23:16:49.677 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:49.671+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.673358+0000) 2020-02-05T23:16:49.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:49.676+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:49.677453+0000) 2020-02-05T23:16:49.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:49.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:50.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:50.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:50.149809+0000) 2020-02-05T23:16:50.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:50.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:50.387805+0000) 2020-02-05T23:16:50.401 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:50.399+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:50.400728+0000) 2020-02-05T23:16:52.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:52.402+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:52.403859+0000) 2020-02-05T23:16:52.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:52.754+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:52.755477+0000) 2020-02-05T23:16:53.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:53.916+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:53.917318+0000) 2020-02-05T23:16:53.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:53.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:53.917289+0000) 2020-02-05T23:16:54.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:54.685+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.686875+0000) 2020-02-05T23:16:54.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:54.686+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.687840+0000) 2020-02-05T23:16:54.690 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:54.687+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.688858+0000) 2020-02-05T23:16:54.690 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:54.687+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.689217+0000) 2020-02-05T23:16:54.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:54.689+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.690363+0000) 2020-02-05T23:16:54.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:54.689+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.690623+0000) 2020-02-05T23:16:54.692 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:54.689+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.691221+0000) 2020-02-05T23:16:54.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:54.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.692274+0000) 2020-02-05T23:16:54.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:54.691+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.692835+0000) 2020-02-05T23:16:54.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:54.691+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.692710+0000) 2020-02-05T23:16:54.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:54.692+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.693506+0000) 2020-02-05T23:16:54.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:54.692+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.693726+0000) 2020-02-05T23:16:54.695 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:54.692+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.693520+0000) 2020-02-05T23:16:54.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:54.697+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:54.698091+0000) 2020-02-05T23:16:54.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:54.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 139978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:16:56.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:56.016+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:56.017279+0000) 2020-02-05T23:16:56.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:56.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:56.293156+0000) 2020-02-05T23:16:57.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:57.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:57.156506+0000) 2020-02-05T23:16:57.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:57.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:57.156697+0000) 2020-02-05T23:16:57.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:57.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:57.157099+0000) 2020-02-05T23:16:59.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:59.463+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.463858+0000) 2020-02-05T23:16:59.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:59.541+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.542568+0000) 2020-02-05T23:16:59.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:16:59.708+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.709650+0000) 2020-02-05T23:16:59.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:16:59.709+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.710330+0000) 2020-02-05T23:16:59.712 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:16:59.709+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.711224+0000) 2020-02-05T23:16:59.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:16:59.710+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.711996+0000) 2020-02-05T23:16:59.714 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:16:59.711+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.712784+0000) 2020-02-05T23:16:59.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:16:59.712+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.713478+0000) 2020-02-05T23:16:59.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:59.713+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.714112+0000) 2020-02-05T23:16:59.715 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:16:59.713+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.714394+0000) 2020-02-05T23:16:59.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:16:59.712+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.714118+0000) 2020-02-05T23:16:59.717 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:16:59.713+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.715315+0000) 2020-02-05T23:16:59.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:16:59.713+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.715350+0000) 2020-02-05T23:16:59.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:59.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.714899+0000) 2020-02-05T23:16:59.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:16:59.714+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.715582+0000) 2020-02-05T23:16:59.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:16:59.719+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:16:59.720517+0000) 2020-02-05T23:16:59.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:16:59.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:00.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:00.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:00.150055+0000) 2020-02-05T23:17:00.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:00.386+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:00.388107+0000) 2020-02-05T23:17:00.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:00.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:00.400939+0000) 2020-02-05T23:17:02.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:02.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:02.404099+0000) 2020-02-05T23:17:02.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:02.755+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:02.755743+0000) 2020-02-05T23:17:03.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:03.917+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:03.917588+0000) 2020-02-05T23:17:03.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:03.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:03.917559+0000) 2020-02-05T23:17:04.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:04.729+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.730644+0000) 2020-02-05T23:17:04.732 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:04.730+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.731641+0000) 2020-02-05T23:17:04.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:04.731+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.733034+0000) 2020-02-05T23:17:04.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:04.732+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.733626+0000) 2020-02-05T23:17:04.735 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:04.732+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.733655+0000) 2020-02-05T23:17:04.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:04.732+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.733997+0000) 2020-02-05T23:17:04.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:04.733+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.734563+0000) 2020-02-05T23:17:04.736 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:04.733+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.735103+0000) 2020-02-05T23:17:04.736 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:04.734+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.735657+0000) 2020-02-05T23:17:04.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:04.734+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.735815+0000) 2020-02-05T23:17:04.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:04.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.736216+0000) 2020-02-05T23:17:04.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:04.735+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.736697+0000) 2020-02-05T23:17:04.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:04.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.736860+0000) 2020-02-05T23:17:04.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:04.739+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:04.741063+0000) 2020-02-05T23:17:04.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:04.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:06.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:06.016+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:06.017530+0000) 2020-02-05T23:17:06.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:06.292+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:06.293336+0000) 2020-02-05T23:17:07.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:07.155+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:07.156750+0000) 2020-02-05T23:17:07.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:07.155+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:07.156974+0000) 2020-02-05T23:17:07.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:07.155+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:07.157334+0000) 2020-02-05T23:17:09.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:09.470+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.471988+0000) 2020-02-05T23:17:09.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:09.542+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.542803+0000) 2020-02-05T23:17:09.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:09.753+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.754809+0000) 2020-02-05T23:17:09.756 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:09.753+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.755267+0000) 2020-02-05T23:17:09.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:09.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.756322+0000) 2020-02-05T23:17:09.757 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:09.754+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.756270+0000) 2020-02-05T23:17:09.758 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:09.756+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.757155+0000) 2020-02-05T23:17:09.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:09.756+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.757614+0000) 2020-02-05T23:17:09.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:09.757+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.758460+0000) 2020-02-05T23:17:09.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:09.757+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.758899+0000) 2020-02-05T23:17:09.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:09.757+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.758399+0000) 2020-02-05T23:17:09.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:09.758+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.759738+0000) 2020-02-05T23:17:09.761 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:09.758+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.759297+0000) 2020-02-05T23:17:09.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:09.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.759372+0000) 2020-02-05T23:17:09.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:09.759+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.760571+0000) 2020-02-05T23:17:09.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:09.766+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:09.767570+0000) 2020-02-05T23:17:09.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:09.769+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:10.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:10.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:10.150307+0000) 2020-02-05T23:17:10.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:10.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:10.388372+0000) 2020-02-05T23:17:10.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:10.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:10.401146+0000) 2020-02-05T23:17:12.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:12.403+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:12.404344+0000) 2020-02-05T23:17:12.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:12.754+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:12.755967+0000) 2020-02-05T23:17:13.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:13.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:13.917795+0000) 2020-02-05T23:17:13.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:13.916+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:13.917851+0000) 2020-02-05T23:17:14.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:14.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.777664+0000) 2020-02-05T23:17:14.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:14.777+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.778504+0000) 2020-02-05T23:17:14.779 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:14.777+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.778419+0000) 2020-02-05T23:17:14.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:14.777+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.778976+0000) 2020-02-05T23:17:14.780 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:14.777+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.779194+0000) 2020-02-05T23:17:14.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:14.778+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.779535+0000) 2020-02-05T23:17:14.781 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:14.778+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.779873+0000) 2020-02-05T23:17:14.781 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:14.778+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.780338+0000) 2020-02-05T23:17:14.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:14.779+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.780211+0000) 2020-02-05T23:17:14.782 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:14.779+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.780728+0000) 2020-02-05T23:17:14.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:14.779+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.781106+0000) 2020-02-05T23:17:14.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:14.781+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.782528+0000) 2020-02-05T23:17:14.783 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:14.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.782577+0000) 2020-02-05T23:17:14.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:14.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:14.788257+0000) 2020-02-05T23:17:14.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:14.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:16.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:16.017+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:16.017815+0000) 2020-02-05T23:17:16.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:16.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:16.293630+0000) 2020-02-05T23:17:16.648 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:17:16.655 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:17:17.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:17.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:17.157063+0000) 2020-02-05T23:17:17.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:17.156+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:17.157253+0000) 2020-02-05T23:17:17.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:17.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:17.157592+0000) 2020-02-05T23:17:19.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:19.483+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.483347+0000) 2020-02-05T23:17:19.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:19.542+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.543040+0000) 2020-02-05T23:17:19.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:19.796+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.797515+0000) 2020-02-05T23:17:19.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:19.798+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.799766+0000) 2020-02-05T23:17:19.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:19.799+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.800399+0000) 2020-02-05T23:17:19.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:19.799+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.800405+0000) 2020-02-05T23:17:19.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:19.799+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.800954+0000) 2020-02-05T23:17:19.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:19.800+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.801299+0000) 2020-02-05T23:17:19.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:19.800+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.801663+0000) 2020-02-05T23:17:19.803 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:19.800+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.801589+0000) 2020-02-05T23:17:19.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:19.801+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.802212+0000) 2020-02-05T23:17:19.804 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:19.801+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.802779+0000) 2020-02-05T23:17:19.804 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:19.801+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.802873+0000) 2020-02-05T23:17:19.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:19.803+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.804183+0000) 2020-02-05T23:17:19.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:19.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.809048+0000) 2020-02-05T23:17:19.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:19.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:19.809349+0000) 2020-02-05T23:17:19.814 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:19.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:20.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:20.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:20.150507+0000) 2020-02-05T23:17:20.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:20.387+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:20.388603+0000) 2020-02-05T23:17:20.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:20.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:20.401305+0000) 2020-02-05T23:17:22.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:22.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:22.404632+0000) 2020-02-05T23:17:22.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:22.756+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:22.756190+0000) 2020-02-05T23:17:23.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:23.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:23.918043+0000) 2020-02-05T23:17:23.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:23.918+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:23.918134+0000) 2020-02-05T23:17:24.821 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:24.820+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.820285+0000) 2020-02-05T23:17:24.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:24.820+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.820587+0000) 2020-02-05T23:17:24.822 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:24.819+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.820899+0000) 2020-02-05T23:17:24.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:24.821+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.821923+0000) 2020-02-05T23:17:24.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:24.822+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.822343+0000) 2020-02-05T23:17:24.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:24.822+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.822607+0000) 2020-02-05T23:17:24.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:24.822+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.822946+0000) 2020-02-05T23:17:24.825 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:24.822+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.823706+0000) 2020-02-05T23:17:24.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:24.823+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.824637+0000) 2020-02-05T23:17:24.826 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:24.823+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.825025+0000) 2020-02-05T23:17:24.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:24.823+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.825174+0000) 2020-02-05T23:17:24.826 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:24.824+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.825751+0000) 2020-02-05T23:17:24.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:24.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.829121+0000) 2020-02-05T23:17:24.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:24.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:24.829341+0000) 2020-02-05T23:17:24.835 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:24.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:26.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:26.017+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:26.018124+0000) 2020-02-05T23:17:26.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:26.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:26.293871+0000) 2020-02-05T23:17:27.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:27.156+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:27.157326+0000) 2020-02-05T23:17:27.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:27.156+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:27.157510+0000) 2020-02-05T23:17:27.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:27.156+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:27.157843+0000) 2020-02-05T23:17:29.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:29.490+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.491863+0000) 2020-02-05T23:17:29.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:29.542+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.543266+0000) 2020-02-05T23:17:29.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:29.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.840519+0000) 2020-02-05T23:17:29.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:29.840+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.841542+0000) 2020-02-05T23:17:29.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:29.841+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.842873+0000) 2020-02-05T23:17:29.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:29.841+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.842999+0000) 2020-02-05T23:17:29.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:29.842+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.843682+0000) 2020-02-05T23:17:29.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:29.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.843749+0000) 2020-02-05T23:17:29.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:29.842+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.844047+0000) 2020-02-05T23:17:29.846 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:29.842+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.844356+0000) 2020-02-05T23:17:29.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:29.843+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.845004+0000) 2020-02-05T23:17:29.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:29.843+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.845002+0000) 2020-02-05T23:17:29.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:29.843+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.845110+0000) 2020-02-05T23:17:29.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:29.844+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.845589+0000) 2020-02-05T23:17:29.847 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:29.844+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.845690+0000) 2020-02-05T23:17:29.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:29.847+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:29.848096+0000) 2020-02-05T23:17:29.856 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:29.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:30.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:30.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:30.150719+0000) 2020-02-05T23:17:30.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:30.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:30.388822+0000) 2020-02-05T23:17:30.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:30.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:30.401511+0000) 2020-02-05T23:17:32.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:32.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:32.404879+0000) 2020-02-05T23:17:32.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:32.755+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:32.756397+0000) 2020-02-05T23:17:33.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:33.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:33.918310+0000) 2020-02-05T23:17:33.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:33.917+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:33.918402+0000) 2020-02-05T23:17:34.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:34.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.860324+0000) 2020-02-05T23:17:34.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:34.862+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.862484+0000) 2020-02-05T23:17:34.864 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:34.861+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.862708+0000) 2020-02-05T23:17:34.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:34.862+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.863300+0000) 2020-02-05T23:17:34.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:34.862+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.863056+0000) 2020-02-05T23:17:34.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:34.863+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.863654+0000) 2020-02-05T23:17:34.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:34.863+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.863654+0000) 2020-02-05T23:17:34.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:34.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.863967+0000) 2020-02-05T23:17:34.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:34.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.864448+0000) 2020-02-05T23:17:34.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:34.863+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.864332+0000) 2020-02-05T23:17:34.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:34.863+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.864999+0000) 2020-02-05T23:17:34.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:34.864+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.865334+0000) 2020-02-05T23:17:34.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:34.865+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.866186+0000) 2020-02-05T23:17:34.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:34.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:34.871332+0000) 2020-02-05T23:17:34.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:34.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:36.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:36.017+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:36.018398+0000) 2020-02-05T23:17:36.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:36.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:36.294141+0000) 2020-02-05T23:17:37.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:37.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:37.157590+0000) 2020-02-05T23:17:37.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:37.157+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:37.157755+0000) 2020-02-05T23:17:37.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:37.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:37.158081+0000) 2020-02-05T23:17:39.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:39.499+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.500051+0000) 2020-02-05T23:17:39.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:39.542+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.543494+0000) 2020-02-05T23:17:39.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:39.880+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.881445+0000) 2020-02-05T23:17:39.885 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:39.883+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.883818+0000) 2020-02-05T23:17:39.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:39.883+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.884370+0000) 2020-02-05T23:17:39.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:39.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.884020+0000) 2020-02-05T23:17:39.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:39.883+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.884846+0000) 2020-02-05T23:17:39.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:39.884+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.885143+0000) 2020-02-05T23:17:39.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:39.884+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.885626+0000) 2020-02-05T23:17:39.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:39.885+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.886279+0000) 2020-02-05T23:17:39.887 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:39.886+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.886773+0000) 2020-02-05T23:17:39.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:39.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.887011+0000) 2020-02-05T23:17:39.888 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:39.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.887535+0000) 2020-02-05T23:17:39.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:39.888+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.888680+0000) 2020-02-05T23:17:39.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:39.888+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.888621+0000) 2020-02-05T23:17:39.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:39.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:39.893744+0000) 2020-02-05T23:17:39.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:39.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:40.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:40.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:40.150929+0000) 2020-02-05T23:17:40.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:40.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:40.389062+0000) 2020-02-05T23:17:40.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:40.400+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:40.401757+0000) 2020-02-05T23:17:42.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:42.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:42.405119+0000) 2020-02-05T23:17:42.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:42.755+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:42.756623+0000) 2020-02-05T23:17:43.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:43.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:43.918537+0000) 2020-02-05T23:17:43.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:43.917+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:43.918653+0000) 2020-02-05T23:17:44.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:44.901+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.902655+0000) 2020-02-05T23:17:44.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:44.903+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.903936+0000) 2020-02-05T23:17:44.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:44.903+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.904178+0000) 2020-02-05T23:17:44.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:44.903+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.904927+0000) 2020-02-05T23:17:44.906 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:44.904+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.905593+0000) 2020-02-05T23:17:44.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:44.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.907145+0000) 2020-02-05T23:17:44.908 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:44.907+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.907507+0000) 2020-02-05T23:17:44.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:44.906+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.907761+0000) 2020-02-05T23:17:44.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:44.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.908653+0000) 2020-02-05T23:17:44.910 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:44.908+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.909030+0000) 2020-02-05T23:17:44.910 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:44.908+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.909397+0000) 2020-02-05T23:17:44.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:44.909+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.909744+0000) 2020-02-05T23:17:44.911 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:44.910+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.910543+0000) 2020-02-05T23:17:44.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:44.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:44.913726+0000) 2020-02-05T23:17:44.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:44.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:46.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:46.018+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:46.018722+0000) 2020-02-05T23:17:46.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:46.293+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:46.294338+0000) 2020-02-05T23:17:46.811 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:17:46.817 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:17:47.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:47.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:47.157842+0000) 2020-02-05T23:17:47.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:47.157+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:47.157998+0000) 2020-02-05T23:17:47.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:47.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:47.158315+0000) 2020-02-05T23:17:49.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:49.506+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.508058+0000) 2020-02-05T23:17:49.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:49.542+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.543737+0000) 2020-02-05T23:17:49.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:49.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.923369+0000) 2020-02-05T23:17:49.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:49.923+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.924991+0000) 2020-02-05T23:17:49.926 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:49.924+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.925572+0000) 2020-02-05T23:17:49.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:49.925+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.926436+0000) 2020-02-05T23:17:49.927 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:49.925+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.926906+0000) 2020-02-05T23:17:49.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:49.926+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.927489+0000) 2020-02-05T23:17:49.928 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:49.927+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.927951+0000) 2020-02-05T23:17:49.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:49.928+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.928732+0000) 2020-02-05T23:17:49.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:49.928+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.928735+0000) 2020-02-05T23:17:49.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:49.929+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.929748+0000) 2020-02-05T23:17:49.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:49.930+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.930489+0000) 2020-02-05T23:17:49.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:49.929+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.930526+0000) 2020-02-05T23:17:49.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:49.930+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.930934+0000) 2020-02-05T23:17:49.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:49.933+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:49.934420+0000) 2020-02-05T23:17:49.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:49.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:50.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:50.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:50.151140+0000) 2020-02-05T23:17:50.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:50.388+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:50.389290+0000) 2020-02-05T23:17:50.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:50.401+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:50.401969+0000) 2020-02-05T23:17:52.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:52.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:52.405388+0000) 2020-02-05T23:17:52.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:52.756+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:52.756855+0000) 2020-02-05T23:17:53.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:53.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:53.918771+0000) 2020-02-05T23:17:53.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:53.917+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:53.918887+0000) 2020-02-05T23:17:54.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:54.947+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.948156+0000) 2020-02-05T23:17:54.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:54.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.950215+0000) 2020-02-05T23:17:54.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:54.949+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.950650+0000) 2020-02-05T23:17:54.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:54.949+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.950928+0000) 2020-02-05T23:17:54.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:54.950+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.951073+0000) 2020-02-05T23:17:54.953 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:54.951+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.951572+0000) 2020-02-05T23:17:54.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:54.951+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.951418+0000) 2020-02-05T23:17:54.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:54.951+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.952132+0000) 2020-02-05T23:17:54.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:54.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.951946+0000) 2020-02-05T23:17:54.954 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:54.951+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.952189+0000) 2020-02-05T23:17:54.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:54.952+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.952624+0000) 2020-02-05T23:17:54.955 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:54.952+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.953005+0000) 2020-02-05T23:17:54.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:54.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.953199+0000) 2020-02-05T23:17:54.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:54.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:54.958262+0000) 2020-02-05T23:17:54.965 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:54.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:17:56.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:56.018+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:56.018982+0000) 2020-02-05T23:17:56.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:56.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:56.294568+0000) 2020-02-05T23:17:57.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:57.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:57.158083+0000) 2020-02-05T23:17:57.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:57.157+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:57.158276+0000) 2020-02-05T23:17:57.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:57.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:57.158606+0000) 2020-02-05T23:17:59.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:59.515+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.516150+0000) 2020-02-05T23:17:59.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:59.543+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.543974+0000) 2020-02-05T23:17:59.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:17:59.968+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.969924+0000) 2020-02-05T23:17:59.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:17:59.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.971005+0000) 2020-02-05T23:17:59.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:17:59.970+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.971352+0000) 2020-02-05T23:17:59.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:17:59.970+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.971726+0000) 2020-02-05T23:17:59.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:17:59.971+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.972185+0000) 2020-02-05T23:17:59.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:17:59.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.972660+0000) 2020-02-05T23:17:59.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:17:59.971+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.972846+0000) 2020-02-05T23:17:59.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:59.972+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.973676+0000) 2020-02-05T23:17:59.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:59.972+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.973799+0000) 2020-02-05T23:17:59.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:17:59.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.973984+0000) 2020-02-05T23:17:59.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:17:59.974+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.974893+0000) 2020-02-05T23:17:59.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:17:59.975+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.975699+0000) 2020-02-05T23:17:59.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:17:59.975+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.975898+0000) 2020-02-05T23:17:59.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:17:59.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:17:59.980081+0000) 2020-02-05T23:17:59.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:17:59.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:00.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:00.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:00.151385+0000) 2020-02-05T23:18:00.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:00.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:00.389522+0000) 2020-02-05T23:18:00.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:00.401+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:00.402182+0000) 2020-02-05T23:18:02.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:02.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:02.405690+0000) 2020-02-05T23:18:02.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:02.756+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:02.757118+0000) 2020-02-05T23:18:03.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:03.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:03.919011+0000) 2020-02-05T23:18:03.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:03.918+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:03.919166+0000) 2020-02-05T23:18:04.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:04.991+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.992323+0000) 2020-02-05T23:18:04.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:04.991+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.992458+0000) 2020-02-05T23:18:04.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:04.993+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.993986+0000) 2020-02-05T23:18:04.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:04.993+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.994310+0000) 2020-02-05T23:18:04.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:04.993+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.994670+0000) 2020-02-05T23:18:04.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:04.994+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.995236+0000) 2020-02-05T23:18:04.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:04.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.995793+0000) 2020-02-05T23:18:04.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:04.995+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.996373+0000) 2020-02-05T23:18:04.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:04.995+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.996859+0000) 2020-02-05T23:18:04.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:04.996+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.997007+0000) 2020-02-05T23:18:04.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:04.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.997470+0000) 2020-02-05T23:18:04.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:04.997+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.998130+0000) 2020-02-05T23:18:05.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:04.998+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:04.998652+0000) 2020-02-05T23:18:05.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:05.002+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:05.003249+0000) 2020-02-05T23:18:05.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:05.009+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140548 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:06.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:06.019+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:06.019232+0000) 2020-02-05T23:18:06.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:06.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:06.294787+0000) 2020-02-05T23:18:07.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:07.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:07.158306+0000) 2020-02-05T23:18:07.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:07.158+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:07.158517+0000) 2020-02-05T23:18:07.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:07.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:07.158847+0000) 2020-02-05T23:18:09.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:09.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:09.524248+0000) 2020-02-05T23:18:09.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:09.543+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:09.544219+0000) 2020-02-05T23:18:10.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:10.012+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.013486+0000) 2020-02-05T23:18:10.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:10.012+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.014004+0000) 2020-02-05T23:18:10.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:10.014+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.015913+0000) 2020-02-05T23:18:10.017 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:10.015+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.016619+0000) 2020-02-05T23:18:10.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:10.015+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.016278+0000) 2020-02-05T23:18:10.018 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:10.016+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.017390+0000) 2020-02-05T23:18:10.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:10.017+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.017906+0000) 2020-02-05T23:18:10.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:10.018+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.018768+0000) 2020-02-05T23:18:10.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:10.018+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.019392+0000) 2020-02-05T23:18:10.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:10.018+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.019935+0000) 2020-02-05T23:18:10.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:10.020+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.021016+0000) 2020-02-05T23:18:10.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:10.020+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.021368+0000) 2020-02-05T23:18:10.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:10.021+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.022179+0000) 2020-02-05T23:18:10.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:10.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.028638+0000) 2020-02-05T23:18:10.032 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:10.030+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:10.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:10.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.151606+0000) 2020-02-05T23:18:10.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:10.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.389741+0000) 2020-02-05T23:18:10.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:10.402+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:10.402394+0000) 2020-02-05T23:18:12.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:12.404+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:12.405945+0000) 2020-02-05T23:18:12.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:12.757+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:12.757352+0000) 2020-02-05T23:18:13.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:13.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:13.919264+0000) 2020-02-05T23:18:13.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:13.918+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:13.919359+0000) 2020-02-05T23:18:15.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:15.035+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.036481+0000) 2020-02-05T23:18:15.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:15.036+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.037440+0000) 2020-02-05T23:18:15.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:15.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.038273+0000) 2020-02-05T23:18:15.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:15.038+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.039408+0000) 2020-02-05T23:18:15.040 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:15.037+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.039077+0000) 2020-02-05T23:18:15.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:15.039+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.040352+0000) 2020-02-05T23:18:15.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:15.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.041416+0000) 2020-02-05T23:18:15.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:15.039+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.040848+0000) 2020-02-05T23:18:15.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:15.039+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.041046+0000) 2020-02-05T23:18:15.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:15.040+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.042076+0000) 2020-02-05T23:18:15.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:15.040+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.042292+0000) 2020-02-05T23:18:15.045 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:15.041+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.042936+0000) 2020-02-05T23:18:15.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:15.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.043114+0000) 2020-02-05T23:18:15.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:15.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:15.045178+0000) 2020-02-05T23:18:15.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:15.051+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140629 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:16.020 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:16.018+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:16.019510+0000) 2020-02-05T23:18:16.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:16.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:16.295070+0000) 2020-02-05T23:18:16.999 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:18:17.005 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:18:17.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:17.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:17.158549+0000) 2020-02-05T23:18:17.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:17.157+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:17.158733+0000) 2020-02-05T23:18:17.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:17.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:17.159046+0000) 2020-02-05T23:18:19.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:19.536+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:19.536383+0000) 2020-02-05T23:18:19.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:19.544+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:19.544451+0000) 2020-02-05T23:18:20.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:20.057+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.058746+0000) 2020-02-05T23:18:20.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:20.059+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.059844+0000) 2020-02-05T23:18:20.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:20.060+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.060300+0000) 2020-02-05T23:18:20.061 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:20.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.060838+0000) 2020-02-05T23:18:20.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:20.061+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.061199+0000) 2020-02-05T23:18:20.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:20.062+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.062272+0000) 2020-02-05T23:18:20.064 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:20.062+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.062856+0000) 2020-02-05T23:18:20.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:20.060+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.061473+0000) 2020-02-05T23:18:20.066 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:20.060+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.062077+0000) 2020-02-05T23:18:20.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:20.061+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.062699+0000) 2020-02-05T23:18:20.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:20.062+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.063408+0000) 2020-02-05T23:18:20.066 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:20.062+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.064012+0000) 2020-02-05T23:18:20.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:20.063+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.063424+0000) 2020-02-05T23:18:20.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:20.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.070557+0000) 2020-02-05T23:18:20.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:20.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:20.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:20.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.151837+0000) 2020-02-05T23:18:20.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:20.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.390020+0000) 2020-02-05T23:18:20.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:20.402+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:20.402663+0000) 2020-02-05T23:18:22.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:22.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:22.406214+0000) 2020-02-05T23:18:22.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:22.756+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:22.757584+0000) 2020-02-05T23:18:23.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:23.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:23.919496+0000) 2020-02-05T23:18:23.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:23.918+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:23.919637+0000) 2020-02-05T23:18:25.081 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:25.078+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.080336+0000) 2020-02-05T23:18:25.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:25.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.080901+0000) 2020-02-05T23:18:25.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:25.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.081231+0000) 2020-02-05T23:18:25.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:25.080+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.081681+0000) 2020-02-05T23:18:25.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:25.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.082256+0000) 2020-02-05T23:18:25.083 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:25.081+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.082742+0000) 2020-02-05T23:18:25.086 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:25.082+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.083766+0000) 2020-02-05T23:18:25.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:25.083+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.084139+0000) 2020-02-05T23:18:25.086 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:25.084+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.085394+0000) 2020-02-05T23:18:25.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:25.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.085626+0000) 2020-02-05T23:18:25.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:25.084+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.085949+0000) 2020-02-05T23:18:25.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:25.084+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.086331+0000) 2020-02-05T23:18:25.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:25.089+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.090361+0000) 2020-02-05T23:18:25.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:25.089+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:25.090650+0000) 2020-02-05T23:18:25.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:25.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:26.020 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:26.018+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:26.019738+0000) 2020-02-05T23:18:26.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:26.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:26.295311+0000) 2020-02-05T23:18:27.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:27.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:27.158809+0000) 2020-02-05T23:18:27.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:27.157+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:27.158984+0000) 2020-02-05T23:18:27.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:27.157+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:27.159311+0000) 2020-02-05T23:18:29.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:29.543+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:29.544575+0000) 2020-02-05T23:18:29.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:29.543+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:29.544703+0000) 2020-02-05T23:18:30.103 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:30.101+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.102706+0000) 2020-02-05T23:18:30.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:30.102+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.103657+0000) 2020-02-05T23:18:30.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:30.103+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.104844+0000) 2020-02-05T23:18:30.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:30.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.105410+0000) 2020-02-05T23:18:30.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:30.104+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.105643+0000) 2020-02-05T23:18:30.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:30.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.106222+0000) 2020-02-05T23:18:30.107 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:30.105+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.106426+0000) 2020-02-05T23:18:30.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:30.105+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.107250+0000) 2020-02-05T23:18:30.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:30.106+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.108178+0000) 2020-02-05T23:18:30.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:30.107+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.108163+0000) 2020-02-05T23:18:30.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:30.106+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.108415+0000) 2020-02-05T23:18:30.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:30.107+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.108854+0000) 2020-02-05T23:18:30.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:30.107+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.109034+0000) 2020-02-05T23:18:30.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:30.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.112163+0000) 2020-02-05T23:18:30.117 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:30.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:30.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:30.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.152013+0000) 2020-02-05T23:18:30.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:30.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.390256+0000) 2020-02-05T23:18:30.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:30.401+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:30.402843+0000) 2020-02-05T23:18:32.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:32.405+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:32.406499+0000) 2020-02-05T23:18:32.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:32.756+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:32.757805+0000) 2020-02-05T23:18:33.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:33.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:33.919725+0000) 2020-02-05T23:18:33.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:33.918+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:33.919880+0000) 2020-02-05T23:18:35.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:35.122+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.123543+0000) 2020-02-05T23:18:35.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:35.122+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.123972+0000) 2020-02-05T23:18:35.125 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:35.123+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.124490+0000) 2020-02-05T23:18:35.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:35.123+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.125086+0000) 2020-02-05T23:18:35.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:35.125+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.126071+0000) 2020-02-05T23:18:35.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:35.125+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.126429+0000) 2020-02-05T23:18:35.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:35.125+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.126879+0000) 2020-02-05T23:18:35.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:35.125+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.127402+0000) 2020-02-05T23:18:35.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:35.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.127775+0000) 2020-02-05T23:18:35.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:35.126+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.128177+0000) 2020-02-05T23:18:35.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:35.127+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.128731+0000) 2020-02-05T23:18:35.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:35.127+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.128493+0000) 2020-02-05T23:18:35.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:35.127+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.129207+0000) 2020-02-05T23:18:35.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:35.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:35.134166+0000) 2020-02-05T23:18:35.139 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:35.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:36.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:36.019+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:36.020007+0000) 2020-02-05T23:18:36.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:36.294+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:36.295548+0000) 2020-02-05T23:18:37.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:37.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:37.159042+0000) 2020-02-05T23:18:37.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:37.158+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:37.159255+0000) 2020-02-05T23:18:37.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:37.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:37.159620+0000) 2020-02-05T23:18:39.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:39.544+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:39.545007+0000) 2020-02-05T23:18:39.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:39.552+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:39.553208+0000) 2020-02-05T23:18:40.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:40.141+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.143366+0000) 2020-02-05T23:18:40.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:40.143+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.144864+0000) 2020-02-05T23:18:40.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:40.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.145243+0000) 2020-02-05T23:18:40.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:40.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.145764+0000) 2020-02-05T23:18:40.147 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:40.145+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.146262+0000) 2020-02-05T23:18:40.148 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:40.146+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.147294+0000) 2020-02-05T23:18:40.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:40.145+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.146993+0000) 2020-02-05T23:18:40.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:40.146+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.147657+0000) 2020-02-05T23:18:40.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:40.147+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.148264+0000) 2020-02-05T23:18:40.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:40.147+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.148626+0000) 2020-02-05T23:18:40.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:40.148+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.149073+0000) 2020-02-05T23:18:40.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:40.148+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.149538+0000) 2020-02-05T23:18:40.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:40.148+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.149852+0000) 2020-02-05T23:18:40.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:40.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.152151+0000) 2020-02-05T23:18:40.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:40.153+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.154429+0000) 2020-02-05T23:18:40.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:40.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:40.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:40.389+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.390492+0000) 2020-02-05T23:18:40.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:40.402+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:40.403069+0000) 2020-02-05T23:18:42.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:42.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:42.406761+0000) 2020-02-05T23:18:42.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:42.757+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:42.758059+0000) 2020-02-05T23:18:43.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:43.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:43.919956+0000) 2020-02-05T23:18:43.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:43.919+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:43.920144+0000) 2020-02-05T23:18:45.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:45.164+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.165649+0000) 2020-02-05T23:18:45.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:45.166+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.167469+0000) 2020-02-05T23:18:45.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:45.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.168351+0000) 2020-02-05T23:18:45.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:45.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.168472+0000) 2020-02-05T23:18:45.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:45.168+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.169302+0000) 2020-02-05T23:18:45.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:45.167+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.169069+0000) 2020-02-05T23:18:45.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:45.169+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.170886+0000) 2020-02-05T23:18:45.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:45.169+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.171327+0000) 2020-02-05T23:18:45.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:45.169+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.170785+0000) 2020-02-05T23:18:45.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:45.170+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.171776+0000) 2020-02-05T23:18:45.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:45.171+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.172577+0000) 2020-02-05T23:18:45.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:45.170+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.172150+0000) 2020-02-05T23:18:45.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:45.171+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.172560+0000) 2020-02-05T23:18:45.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:45.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:45.178234+0000) 2020-02-05T23:18:45.182 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:45.180+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:46.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:46.019+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:46.020224+0000) 2020-02-05T23:18:46.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:46.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:46.295804+0000) 2020-02-05T23:18:47.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:47.157+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:47.159279+0000) 2020-02-05T23:18:47.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:47.158+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:47.159503+0000) 2020-02-05T23:18:47.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:47.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:47.159828+0000) 2020-02-05T23:18:47.196 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:18:47.205 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:18:49.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:49.544+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:49.545229+0000) 2020-02-05T23:18:49.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:49.560+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:49.561244+0000) 2020-02-05T23:18:50.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:50.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.152345+0000) 2020-02-05T23:18:50.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:50.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.187711+0000) 2020-02-05T23:18:50.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:50.187+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.188141+0000) 2020-02-05T23:18:50.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:50.187+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.188701+0000) 2020-02-05T23:18:50.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:50.188+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.189965+0000) 2020-02-05T23:18:50.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:50.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.189803+0000) 2020-02-05T23:18:50.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:50.189+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.190425+0000) 2020-02-05T23:18:50.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:50.189+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.190979+0000) 2020-02-05T23:18:50.192 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:50.190+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.191730+0000) 2020-02-05T23:18:50.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:50.190+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.191728+0000) 2020-02-05T23:18:50.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:50.191+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.192416+0000) 2020-02-05T23:18:50.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:50.190+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.192240+0000) 2020-02-05T23:18:50.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:50.191+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.193034+0000) 2020-02-05T23:18:50.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:50.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.193499+0000) 2020-02-05T23:18:50.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:50.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.197213+0000) 2020-02-05T23:18:50.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:50.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140919 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:50.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:50.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.390730+0000) 2020-02-05T23:18:50.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:50.402+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:50.403348+0000) 2020-02-05T23:18:52.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:52.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:52.407023+0000) 2020-02-05T23:18:52.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:52.757+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:52.758363+0000) 2020-02-05T23:18:53.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:53.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:53.920188+0000) 2020-02-05T23:18:53.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:53.919+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:53.920354+0000) 2020-02-05T23:18:55.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:18:55.208+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.209897+0000) 2020-02-05T23:18:55.211 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:18:55.209+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.210638+0000) 2020-02-05T23:18:55.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:55.210+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.211843+0000) 2020-02-05T23:18:55.213 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:18:55.211+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.212489+0000) 2020-02-05T23:18:55.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:55.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.212947+0000) 2020-02-05T23:18:55.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:55.211+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.213358+0000) 2020-02-05T23:18:55.214 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:18:55.212+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.213932+0000) 2020-02-05T23:18:55.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:18:55.213+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.214555+0000) 2020-02-05T23:18:55.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:18:55.213+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.214583+0000) 2020-02-05T23:18:55.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:55.213+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.215240+0000) 2020-02-05T23:18:55.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:55.214+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.215486+0000) 2020-02-05T23:18:55.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:18:55.214+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.215783+0000) 2020-02-05T23:18:55.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:55.214+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.216358+0000) 2020-02-05T23:18:55.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:55.222+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:55.223259+0000) 2020-02-05T23:18:55.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:18:55.225+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 140956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:18:56.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:18:56.019+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:56.020389+0000) 2020-02-05T23:18:56.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:56.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:56.296036+0000) 2020-02-05T23:18:57.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:18:57.158+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:57.159514+0000) 2020-02-05T23:18:57.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:18:57.158+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:57.159781+0000) 2020-02-05T23:18:57.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:18:57.158+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:57.160059+0000) 2020-02-05T23:18:59.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:59.545+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:59.545462+0000) 2020-02-05T23:18:59.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:18:59.569+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:18:59.569756+0000) 2020-02-05T23:19:00.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:00.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.152567+0000) 2020-02-05T23:19:00.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:00.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.231967+0000) 2020-02-05T23:19:00.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:00.232+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.233694+0000) 2020-02-05T23:19:00.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:00.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.234122+0000) 2020-02-05T23:19:00.235 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:00.233+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.234440+0000) 2020-02-05T23:19:00.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:00.233+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.235227+0000) 2020-02-05T23:19:00.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:00.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.235457+0000) 2020-02-05T23:19:00.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:00.234+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.235641+0000) 2020-02-05T23:19:00.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:00.234+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.236162+0000) 2020-02-05T23:19:00.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:00.235+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.236562+0000) 2020-02-05T23:19:00.238 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:00.235+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.236721+0000) 2020-02-05T23:19:00.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:00.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.237821+0000) 2020-02-05T23:19:00.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:00.236+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.237261+0000) 2020-02-05T23:19:00.239 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:00.236+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.238027+0000) 2020-02-05T23:19:00.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:00.242+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.243143+0000) 2020-02-05T23:19:00.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:00.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141000 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:00.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:00.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.390945+0000) 2020-02-05T23:19:00.407 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:00.403+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:00.403554+0000) 2020-02-05T23:19:02.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:02.406+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:02.407264+0000) 2020-02-05T23:19:02.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:02.758+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:02.758641+0000) 2020-02-05T23:19:03.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:03.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:03.920444+0000) 2020-02-05T23:19:03.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:03.920+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:03.920610+0000) 2020-02-05T23:19:05.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:05.255+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.257007+0000) 2020-02-05T23:19:05.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:05.257+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.257563+0000) 2020-02-05T23:19:05.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:05.258+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.258737+0000) 2020-02-05T23:19:05.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:05.257+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.258380+0000) 2020-02-05T23:19:05.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:05.259+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.260062+0000) 2020-02-05T23:19:05.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:05.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.260426+0000) 2020-02-05T23:19:05.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:05.260+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.261089+0000) 2020-02-05T23:19:05.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:05.260+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.261395+0000) 2020-02-05T23:19:05.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:05.260+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.261326+0000) 2020-02-05T23:19:05.263 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:05.260+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.261800+0000) 2020-02-05T23:19:05.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:05.261+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.262323+0000) 2020-02-05T23:19:05.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:05.261+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.262154+0000) 2020-02-05T23:19:05.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:05.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.262552+0000) 2020-02-05T23:19:05.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:05.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:05.266007+0000) 2020-02-05T23:19:05.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:05.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:06.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:06.020+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:06.020683+0000) 2020-02-05T23:19:06.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:06.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:06.296346+0000) 2020-02-05T23:19:07.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:07.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:07.159769+0000) 2020-02-05T23:19:07.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:07.159+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:07.159986+0000) 2020-02-05T23:19:07.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:07.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:07.160286+0000) 2020-02-05T23:19:09.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:09.545+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:09.545721+0000) 2020-02-05T23:19:09.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:09.578+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:09.578220+0000) 2020-02-05T23:19:10.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:10.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.152820+0000) 2020-02-05T23:19:10.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:10.275+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.276552+0000) 2020-02-05T23:19:10.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:10.276+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.277316+0000) 2020-02-05T23:19:10.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:10.276+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.277905+0000) 2020-02-05T23:19:10.280 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:10.278+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.279516+0000) 2020-02-05T23:19:10.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:10.279+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.279715+0000) 2020-02-05T23:19:10.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:10.279+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.280208+0000) 2020-02-05T23:19:10.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:10.279+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.280365+0000) 2020-02-05T23:19:10.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:10.279+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.280856+0000) 2020-02-05T23:19:10.282 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:10.280+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.281232+0000) 2020-02-05T23:19:10.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:10.281+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.281727+0000) 2020-02-05T23:19:10.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:10.280+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.281419+0000) 2020-02-05T23:19:10.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:10.280+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.281627+0000) 2020-02-05T23:19:10.284 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:10.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.282115+0000) 2020-02-05T23:19:10.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:10.285+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.286924+0000) 2020-02-05T23:19:10.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:10.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141083 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:10.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:10.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.391225+0000) 2020-02-05T23:19:10.404 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:10.403+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:10.403777+0000) 2020-02-05T23:19:12.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:12.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:12.407530+0000) 2020-02-05T23:19:12.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:12.758+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:12.758908+0000) 2020-02-05T23:19:13.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:13.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:13.920677+0000) 2020-02-05T23:19:13.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:13.920+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:13.920849+0000) 2020-02-05T23:19:15.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:15.299+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.299391+0000) 2020-02-05T23:19:15.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:15.299+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.299998+0000) 2020-02-05T23:19:15.301 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:15.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.300522+0000) 2020-02-05T23:19:15.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:15.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.301149+0000) 2020-02-05T23:19:15.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:15.301+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.301461+0000) 2020-02-05T23:19:15.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:15.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.302400+0000) 2020-02-05T23:19:15.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:15.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.303001+0000) 2020-02-05T23:19:15.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:15.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.303212+0000) 2020-02-05T23:19:15.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:15.303+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.303914+0000) 2020-02-05T23:19:15.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:15.303+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.303702+0000) 2020-02-05T23:19:15.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:15.304+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.304457+0000) 2020-02-05T23:19:15.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:15.304+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.304724+0000) 2020-02-05T23:19:15.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:15.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.304753+0000) 2020-02-05T23:19:15.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:15.309+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:15.309560+0000) 2020-02-05T23:19:15.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:15.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:16.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:16.020+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:16.020903+0000) 2020-02-05T23:19:16.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:16.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:16.296586+0000) 2020-02-05T23:19:17.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:17.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:17.160024+0000) 2020-02-05T23:19:17.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:17.159+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:17.160228+0000) 2020-02-05T23:19:17.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:17.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:17.160532+0000) 2020-02-05T23:19:17.392 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:19:17.398 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:19:19.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:19.544+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:19.545953+0000) 2020-02-05T23:19:19.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:19.584+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:19.585333+0000) 2020-02-05T23:19:20.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:20.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.153052+0000) 2020-02-05T23:19:20.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:20.318+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.319326+0000) 2020-02-05T23:19:20.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:20.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.321440+0000) 2020-02-05T23:19:20.322 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:20.320+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.321765+0000) 2020-02-05T23:19:20.323 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:20.321+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.322531+0000) 2020-02-05T23:19:20.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:20.322+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.322543+0000) 2020-02-05T23:19:20.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:20.322+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.323033+0000) 2020-02-05T23:19:20.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:20.323+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.324184+0000) 2020-02-05T23:19:20.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:20.324+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.324495+0000) 2020-02-05T23:19:20.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:20.324+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.325157+0000) 2020-02-05T23:19:20.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:20.324+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.325576+0000) 2020-02-05T23:19:20.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:20.325+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.325969+0000) 2020-02-05T23:19:20.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:20.325+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.326015+0000) 2020-02-05T23:19:20.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:20.326+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.326706+0000) 2020-02-05T23:19:20.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:20.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.332242+0000) 2020-02-05T23:19:20.336 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:20.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141164 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:20.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:20.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.391476+0000) 2020-02-05T23:19:20.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:20.403+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:20.404009+0000) 2020-02-05T23:19:22.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:22.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:22.407722+0000) 2020-02-05T23:19:22.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:22.758+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:22.759147+0000) 2020-02-05T23:19:23.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:23.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:23.920915+0000) 2020-02-05T23:19:23.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:23.920+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:23.921113+0000) 2020-02-05T23:19:25.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:25.340+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.341495+0000) 2020-02-05T23:19:25.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:25.341+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.342308+0000) 2020-02-05T23:19:25.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:25.342+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.342873+0000) 2020-02-05T23:19:25.344 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:25.342+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.343576+0000) 2020-02-05T23:19:25.352 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:25.344+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.344905+0000) 2020-02-05T23:19:25.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:25.344+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.345301+0000) 2020-02-05T23:19:25.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:25.345+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.346001+0000) 2020-02-05T23:19:25.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:25.346+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.346477+0000) 2020-02-05T23:19:25.353 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:25.346+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.347009+0000) 2020-02-05T23:19:25.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:25.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.344173+0000) 2020-02-05T23:19:25.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:25.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.350017+0000) 2020-02-05T23:19:25.354 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:25.343+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.344373+0000) 2020-02-05T23:19:25.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:25.344+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.345636+0000) 2020-02-05T23:19:25.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:25.344+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:25.346022+0000) 2020-02-05T23:19:25.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:25.357+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141202 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:26.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:26.020+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:26.021212+0000) 2020-02-05T23:19:26.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:26.295+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:26.296791+0000) 2020-02-05T23:19:27.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:27.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:27.160261+0000) 2020-02-05T23:19:27.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:27.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:27.160511+0000) 2020-02-05T23:19:27.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:27.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:27.160777+0000) 2020-02-05T23:19:29.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:29.545+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:29.546191+0000) 2020-02-05T23:19:29.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:29.594+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:29.595189+0000) 2020-02-05T23:19:30.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:30.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.153340+0000) 2020-02-05T23:19:30.363 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:30.361+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.362282+0000) 2020-02-05T23:19:30.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:30.362+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.363471+0000) 2020-02-05T23:19:30.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:30.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.366419+0000) 2020-02-05T23:19:30.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:30.365+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.365951+0000) 2020-02-05T23:19:30.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:30.365+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.366157+0000) 2020-02-05T23:19:30.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:30.366+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.367360+0000) 2020-02-05T23:19:30.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:30.366+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.367594+0000) 2020-02-05T23:19:30.369 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:30.367+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.367475+0000) 2020-02-05T23:19:30.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:30.367+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.367869+0000) 2020-02-05T23:19:30.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:30.367+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.368097+0000) 2020-02-05T23:19:30.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:30.367+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.368518+0000) 2020-02-05T23:19:30.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:30.367+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.368980+0000) 2020-02-05T23:19:30.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:30.369+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.369481+0000) 2020-02-05T23:19:30.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:30.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.372910+0000) 2020-02-05T23:19:30.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:30.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141247 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:30.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:30.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.391680+0000) 2020-02-05T23:19:30.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:30.403+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:30.404259+0000) 2020-02-05T23:19:32.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:32.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:32.407994+0000) 2020-02-05T23:19:32.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:32.758+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:32.759411+0000) 2020-02-05T23:19:33.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:33.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:33.921185+0000) 2020-02-05T23:19:33.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:33.920+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:33.921350+0000) 2020-02-05T23:19:35.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:35.382+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.383323+0000) 2020-02-05T23:19:35.385 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:35.383+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.384460+0000) 2020-02-05T23:19:35.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:35.384+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.385127+0000) 2020-02-05T23:19:35.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:35.385+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.385923+0000) 2020-02-05T23:19:35.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:35.385+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.386527+0000) 2020-02-05T23:19:35.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:35.386+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.386519+0000) 2020-02-05T23:19:35.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:35.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.387055+0000) 2020-02-05T23:19:35.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:35.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.387697+0000) 2020-02-05T23:19:35.389 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:35.386+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.387317+0000) 2020-02-05T23:19:35.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:35.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.388648+0000) 2020-02-05T23:19:35.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:35.387+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.388863+0000) 2020-02-05T23:19:35.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:35.388+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.388910+0000) 2020-02-05T23:19:35.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:35.389+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.389662+0000) 2020-02-05T23:19:35.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:35.393+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:35.394478+0000) 2020-02-05T23:19:35.400 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:35.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:36.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:36.021+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:36.021475+0000) 2020-02-05T23:19:36.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:36.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:36.297014+0000) 2020-02-05T23:19:37.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:37.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:37.160507+0000) 2020-02-05T23:19:37.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:37.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:37.160760+0000) 2020-02-05T23:19:37.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:37.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:37.161002+0000) 2020-02-05T23:19:39.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:39.545+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:39.546422+0000) 2020-02-05T23:19:39.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:39.602+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:39.603313+0000) 2020-02-05T23:19:40.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:40.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.153582+0000) 2020-02-05T23:19:40.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:40.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.391850+0000) 2020-02-05T23:19:40.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:40.404+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.405200+0000) 2020-02-05T23:19:40.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:40.405+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.405878+0000) 2020-02-05T23:19:40.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:40.405+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.406378+0000) 2020-02-05T23:19:40.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:40.406+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.407579+0000) 2020-02-05T23:19:40.408 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:40.406+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.407209+0000) 2020-02-05T23:19:40.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:40.408+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.408636+0000) 2020-02-05T23:19:40.410 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:40.407+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.409057+0000) 2020-02-05T23:19:40.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:40.408+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.409593+0000) 2020-02-05T23:19:40.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:40.409+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.409981+0000) 2020-02-05T23:19:40.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:40.409+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.410481+0000) 2020-02-05T23:19:40.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:40.409+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.410288+0000) 2020-02-05T23:19:40.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:40.409+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.410998+0000) 2020-02-05T23:19:40.412 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:40.410+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.411247+0000) 2020-02-05T23:19:40.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:40.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:40.415510+0000) 2020-02-05T23:19:40.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:40.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141328 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:42.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:42.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:42.408300+0000) 2020-02-05T23:19:42.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:42.759+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:42.759643+0000) 2020-02-05T23:19:43.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:43.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:43.921475+0000) 2020-02-05T23:19:43.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:43.920+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:43.921740+0000) 2020-02-05T23:19:45.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:45.425+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.426229+0000) 2020-02-05T23:19:45.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:45.426+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.427369+0000) 2020-02-05T23:19:45.429 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:45.427+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.428066+0000) 2020-02-05T23:19:45.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:45.427+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.429205+0000) 2020-02-05T23:19:45.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:45.427+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.429388+0000) 2020-02-05T23:19:45.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:45.428+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.429840+0000) 2020-02-05T23:19:45.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:45.429+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.430375+0000) 2020-02-05T23:19:45.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:45.429+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.430485+0000) 2020-02-05T23:19:45.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:45.429+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.430895+0000) 2020-02-05T23:19:45.432 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:45.430+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.431584+0000) 2020-02-05T23:19:45.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:45.430+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.431645+0000) 2020-02-05T23:19:45.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:45.430+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.432227+0000) 2020-02-05T23:19:45.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:45.431+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.432491+0000) 2020-02-05T23:19:45.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:45.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:45.436338+0000) 2020-02-05T23:19:45.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:45.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:46.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:46.020+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:46.021723+0000) 2020-02-05T23:19:46.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:46.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:46.297242+0000) 2020-02-05T23:19:47.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:47.154+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:47.154967+0000) 2020-02-05T23:19:47.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:47.159+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:47.160692+0000) 2020-02-05T23:19:47.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:47.159+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:47.161018+0000) 2020-02-05T23:19:47.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:47.159+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:47.161264+0000) 2020-02-05T23:19:47.598 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:19:47.604 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:19:49.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:49.546+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:49.546656+0000) 2020-02-05T23:19:49.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:49.610+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:49.611411+0000) 2020-02-05T23:19:50.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:50.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.153800+0000) 2020-02-05T23:19:50.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:50.390+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.392088+0000) 2020-02-05T23:19:50.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:50.448+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.449696+0000) 2020-02-05T23:19:50.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:50.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.450194+0000) 2020-02-05T23:19:50.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:50.449+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.450483+0000) 2020-02-05T23:19:50.452 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:50.450+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.451169+0000) 2020-02-05T23:19:50.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:50.450+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.452135+0000) 2020-02-05T23:19:50.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:50.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.451924+0000) 2020-02-05T23:19:50.454 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:50.451+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.453229+0000) 2020-02-05T23:19:50.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:50.451+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.453404+0000) 2020-02-05T23:19:50.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:50.453+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.454493+0000) 2020-02-05T23:19:50.456 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:50.453+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.454838+0000) 2020-02-05T23:19:50.456 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:50.454+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.455128+0000) 2020-02-05T23:19:50.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:50.454+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.455194+0000) 2020-02-05T23:19:50.457 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:50.453+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.455388+0000) 2020-02-05T23:19:50.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:50.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141411 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:50.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:50.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:50.460649+0000) 2020-02-05T23:19:52.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:52.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:52.408536+0000) 2020-02-05T23:19:52.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:52.759+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:52.759881+0000) 2020-02-05T23:19:53.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:53.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:53.921728+0000) 2020-02-05T23:19:53.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:53.921+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:53.921973+0000) 2020-02-05T23:19:55.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:19:55.468+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.469790+0000) 2020-02-05T23:19:55.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:55.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.471587+0000) 2020-02-05T23:19:55.473 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:19:55.470+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.471944+0000) 2020-02-05T23:19:55.473 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:55.471+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.472415+0000) 2020-02-05T23:19:55.474 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:19:55.471+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.473212+0000) 2020-02-05T23:19:55.474 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:19:55.472+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.473789+0000) 2020-02-05T23:19:55.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:19:55.472+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.473723+0000) 2020-02-05T23:19:55.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:55.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.474378+0000) 2020-02-05T23:19:55.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:55.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.474501+0000) 2020-02-05T23:19:55.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:19:55.474+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.475187+0000) 2020-02-05T23:19:55.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:55.473+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.475169+0000) 2020-02-05T23:19:55.477 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:55.474+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.475735+0000) 2020-02-05T23:19:55.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:55.474+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.476247+0000) 2020-02-05T23:19:55.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:55.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:55.480640+0000) 2020-02-05T23:19:55.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:19:55.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:19:56.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:19:56.020+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:56.021946+0000) 2020-02-05T23:19:56.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:56.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:56.297449+0000) 2020-02-05T23:19:57.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:19:57.154+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:57.155197+0000) 2020-02-05T23:19:57.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:19:57.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:57.160952+0000) 2020-02-05T23:19:57.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:19:57.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:57.161272+0000) 2020-02-05T23:19:57.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:19:57.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:57.161501+0000) 2020-02-05T23:19:59.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:59.546+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:59.546891+0000) 2020-02-05T23:19:59.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:19:59.619+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:19:59.619452+0000) 2020-02-05T23:20:00.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:00.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.154038+0000) 2020-02-05T23:20:00.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:00.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.392333+0000) 2020-02-05T23:20:00.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:00.490+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.491591+0000) 2020-02-05T23:20:00.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:00.491+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.493163+0000) 2020-02-05T23:20:00.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:00.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.493657+0000) 2020-02-05T23:20:00.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:00.492+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.493942+0000) 2020-02-05T23:20:00.496 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:00.493+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.494415+0000) 2020-02-05T23:20:00.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:00.493+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.494772+0000) 2020-02-05T23:20:00.497 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:00.493+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.495215+0000) 2020-02-05T23:20:00.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:00.494+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.495786+0000) 2020-02-05T23:20:00.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:00.494+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.495799+0000) 2020-02-05T23:20:00.498 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:00.495+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.496396+0000) 2020-02-05T23:20:00.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:00.495+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.496987+0000) 2020-02-05T23:20:00.500 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:00.496+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.497521+0000) 2020-02-05T23:20:00.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:00.496+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.497342+0000) 2020-02-05T23:20:00.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:00.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141492 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:00.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:00.501+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:00.502717+0000) 2020-02-05T23:20:02.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:02.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:02.408772+0000) 2020-02-05T23:20:02.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:02.759+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:02.760130+0000) 2020-02-05T23:20:03.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:03.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:03.921971+0000) 2020-02-05T23:20:03.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:03.921+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:03.922304+0000) 2020-02-05T23:20:05.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:05.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.514493+0000) 2020-02-05T23:20:05.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:05.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.515648+0000) 2020-02-05T23:20:05.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:05.513+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.515003+0000) 2020-02-05T23:20:05.518 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:05.514+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.515195+0000) 2020-02-05T23:20:05.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:05.514+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.515950+0000) 2020-02-05T23:20:05.520 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:05.515+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.516907+0000) 2020-02-05T23:20:05.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:05.516+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.517610+0000) 2020-02-05T23:20:05.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:05.516+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.518003+0000) 2020-02-05T23:20:05.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:05.518+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.519114+0000) 2020-02-05T23:20:05.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:05.517+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.518811+0000) 2020-02-05T23:20:05.522 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:05.518+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.519553+0000) 2020-02-05T23:20:05.523 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:05.518+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.519911+0000) 2020-02-05T23:20:05.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:05.518+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.520391+0000) 2020-02-05T23:20:05.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:05.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:05.525656+0000) 2020-02-05T23:20:05.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:05.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:06.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:06.021+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:06.022167+0000) 2020-02-05T23:20:06.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:06.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:06.297664+0000) 2020-02-05T23:20:07.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:07.155+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:07.155462+0000) 2020-02-05T23:20:07.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:07.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:07.161220+0000) 2020-02-05T23:20:07.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:07.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:07.161522+0000) 2020-02-05T23:20:07.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:07.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:07.161721+0000) 2020-02-05T23:20:09.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:09.547+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:09.547126+0000) 2020-02-05T23:20:09.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:09.628+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:09.628285+0000) 2020-02-05T23:20:10.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:10.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.154294+0000) 2020-02-05T23:20:10.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:10.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.392593+0000) 2020-02-05T23:20:10.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:10.533+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.535076+0000) 2020-02-05T23:20:10.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:10.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.536283+0000) 2020-02-05T23:20:10.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:10.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.536873+0000) 2020-02-05T23:20:10.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:10.535+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.537269+0000) 2020-02-05T23:20:10.538 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:10.536+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.537584+0000) 2020-02-05T23:20:10.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:10.538+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.538230+0000) 2020-02-05T23:20:10.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:10.538+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.538776+0000) 2020-02-05T23:20:10.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:10.537+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.539233+0000) 2020-02-05T23:20:10.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:10.540+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.540138+0000) 2020-02-05T23:20:10.541 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:10.540+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.540411+0000) 2020-02-05T23:20:10.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:10.540+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.540390+0000) 2020-02-05T23:20:10.542 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:10.539+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.540744+0000) 2020-02-05T23:20:10.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:10.539+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.541392+0000) 2020-02-05T23:20:10.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:10.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:10.545005+0000) 2020-02-05T23:20:10.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:10.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141574 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:12.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:12.408+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:12.409029+0000) 2020-02-05T23:20:12.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:12.759+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:12.760382+0000) 2020-02-05T23:20:13.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:13.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:13.922206+0000) 2020-02-05T23:20:13.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:13.921+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:13.922531+0000) 2020-02-05T23:20:15.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:15.556+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.557714+0000) 2020-02-05T23:20:15.559 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:15.557+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.558666+0000) 2020-02-05T23:20:15.560 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:15.558+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.559425+0000) 2020-02-05T23:20:15.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:15.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.559441+0000) 2020-02-05T23:20:15.561 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:15.558+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.559805+0000) 2020-02-05T23:20:15.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:15.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.560924+0000) 2020-02-05T23:20:15.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:15.559+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.561297+0000) 2020-02-05T23:20:15.562 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:15.559+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.560887+0000) 2020-02-05T23:20:15.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:15.559+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.561356+0000) 2020-02-05T23:20:15.563 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:15.560+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.561883+0000) 2020-02-05T23:20:15.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:15.561+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.562414+0000) 2020-02-05T23:20:15.564 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:15.561+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.563040+0000) 2020-02-05T23:20:15.564 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:15.561+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.563017+0000) 2020-02-05T23:20:15.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:15.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:15.567831+0000) 2020-02-05T23:20:15.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:15.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:16.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:16.021+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:16.022401+0000) 2020-02-05T23:20:16.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:16.296+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:16.297869+0000) 2020-02-05T23:20:17.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:17.154+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:17.155688+0000) 2020-02-05T23:20:17.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:17.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:17.161472+0000) 2020-02-05T23:20:17.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:17.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:17.161740+0000) 2020-02-05T23:20:17.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:17.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:17.161961+0000) 2020-02-05T23:20:17.798 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:20:17.804 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:20:19.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:19.546+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:19.547353+0000) 2020-02-05T23:20:19.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:19.635+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:19.636824+0000) 2020-02-05T23:20:20.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:20.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.154547+0000) 2020-02-05T23:20:20.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:20.391+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.392776+0000) 2020-02-05T23:20:20.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:20.579+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.580031+0000) 2020-02-05T23:20:20.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:20.579+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.581175+0000) 2020-02-05T23:20:20.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:20.580+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.581893+0000) 2020-02-05T23:20:20.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:20.580+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.581613+0000) 2020-02-05T23:20:20.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:20.581+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.582933+0000) 2020-02-05T23:20:20.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:20.582+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.583585+0000) 2020-02-05T23:20:20.586 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:20.584+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.585127+0000) 2020-02-05T23:20:20.586 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:20.584+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.585120+0000) 2020-02-05T23:20:20.588 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:20.584+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.585891+0000) 2020-02-05T23:20:20.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:20.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.585598+0000) 2020-02-05T23:20:20.589 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:20.585+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.586917+0000) 2020-02-05T23:20:20.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:20.585+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.586720+0000) 2020-02-05T23:20:20.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:20.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.592674+0000) 2020-02-05T23:20:20.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:20.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:20.592933+0000) 2020-02-05T23:20:20.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:20.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:22.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:22.407+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:22.409284+0000) 2020-02-05T23:20:22.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:22.759+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:22.760638+0000) 2020-02-05T23:20:23.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:23.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:23.922435+0000) 2020-02-05T23:20:23.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:23.921+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:23.922726+0000) 2020-02-05T23:20:25.602 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:25.600+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.601558+0000) 2020-02-05T23:20:25.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:25.600+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.602058+0000) 2020-02-05T23:20:25.603 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:25.601+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.602819+0000) 2020-02-05T23:20:25.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:25.601+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.602530+0000) 2020-02-05T23:20:25.605 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:25.603+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.605029+0000) 2020-02-05T23:20:25.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:25.604+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.605533+0000) 2020-02-05T23:20:25.606 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:25.604+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.605872+0000) 2020-02-05T23:20:25.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:25.605+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.606144+0000) 2020-02-05T23:20:25.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:25.605+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.606262+0000) 2020-02-05T23:20:25.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:25.605+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.606524+0000) 2020-02-05T23:20:25.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:25.605+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.607058+0000) 2020-02-05T23:20:25.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:25.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.609643+0000) 2020-02-05T23:20:25.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:25.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.610778+0000) 2020-02-05T23:20:25.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:25.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:25.611017+0000) 2020-02-05T23:20:25.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:25.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:26.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:26.022+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:26.022664+0000) 2020-02-05T23:20:26.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:26.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:26.298073+0000) 2020-02-05T23:20:27.157 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:27.155+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:27.155924+0000) 2020-02-05T23:20:27.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:27.160+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:27.161701+0000) 2020-02-05T23:20:27.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:27.160+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:27.161956+0000) 2020-02-05T23:20:27.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:27.160+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:27.162209+0000) 2020-02-05T23:20:29.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:29.546+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:29.547584+0000) 2020-02-05T23:20:29.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:29.647+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:29.648655+0000) 2020-02-05T23:20:30.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:30.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.154785+0000) 2020-02-05T23:20:30.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:30.392+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.393035+0000) 2020-02-05T23:20:30.625 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:30.623+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.624420+0000) 2020-02-05T23:20:30.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:30.625+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.626597+0000) 2020-02-05T23:20:30.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:30.626+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.627159+0000) 2020-02-05T23:20:30.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:30.626+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.627876+0000) 2020-02-05T23:20:30.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:30.627+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.628392+0000) 2020-02-05T23:20:30.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:30.626+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.627805+0000) 2020-02-05T23:20:30.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:30.628+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.629505+0000) 2020-02-05T23:20:30.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:30.628+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.629844+0000) 2020-02-05T23:20:30.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:30.627+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.628675+0000) 2020-02-05T23:20:30.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:30.628+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.629554+0000) 2020-02-05T23:20:30.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:30.628+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.629737+0000) 2020-02-05T23:20:30.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:30.628+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.630089+0000) 2020-02-05T23:20:30.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:30.629+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.630596+0000) 2020-02-05T23:20:30.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:30.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:30.635761+0000) 2020-02-05T23:20:30.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:30.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:32.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:32.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:32.409531+0000) 2020-02-05T23:20:32.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:32.760+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:32.760881+0000) 2020-02-05T23:20:33.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:33.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:33.922656+0000) 2020-02-05T23:20:33.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:33.921+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:33.922957+0000) 2020-02-05T23:20:35.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:35.647+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.647778+0000) 2020-02-05T23:20:35.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:35.647+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.648124+0000) 2020-02-05T23:20:35.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:35.647+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.648742+0000) 2020-02-05T23:20:35.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:35.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.649616+0000) 2020-02-05T23:20:35.650 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:35.649+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.649574+0000) 2020-02-05T23:20:35.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:35.649+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.650401+0000) 2020-02-05T23:20:35.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:35.650+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.651185+0000) 2020-02-05T23:20:35.652 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:35.650+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.651481+0000) 2020-02-05T23:20:35.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:35.651+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.651592+0000) 2020-02-05T23:20:35.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:35.651+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.651801+0000) 2020-02-05T23:20:35.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:35.650+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.650907+0000) 2020-02-05T23:20:35.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:35.650+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.651736+0000) 2020-02-05T23:20:35.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:35.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.654530+0000) 2020-02-05T23:20:35.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:35.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:35.654782+0000) 2020-02-05T23:20:35.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:35.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:36.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:36.022+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:36.022902+0000) 2020-02-05T23:20:36.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:36.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:36.298310+0000) 2020-02-05T23:20:37.157 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:37.155+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:37.156173+0000) 2020-02-05T23:20:37.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:37.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:37.161981+0000) 2020-02-05T23:20:37.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:37.161+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:37.162222+0000) 2020-02-05T23:20:37.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:37.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:37.162495+0000) 2020-02-05T23:20:39.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:39.547+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:39.547816+0000) 2020-02-05T23:20:39.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:39.659+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:39.660472+0000) 2020-02-05T23:20:40.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:40.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.155019+0000) 2020-02-05T23:20:40.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:40.392+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.393287+0000) 2020-02-05T23:20:40.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:40.668+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.669155+0000) 2020-02-05T23:20:40.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:40.669+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.670283+0000) 2020-02-05T23:20:40.672 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:40.670+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.671900+0000) 2020-02-05T23:20:40.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:40.671+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.672213+0000) 2020-02-05T23:20:40.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:40.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.672537+0000) 2020-02-05T23:20:40.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:40.671+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.672930+0000) 2020-02-05T23:20:40.674 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:40.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.673488+0000) 2020-02-05T23:20:40.674 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:40.673+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.673699+0000) 2020-02-05T23:20:40.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:40.674+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.674423+0000) 2020-02-05T23:20:40.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:40.674+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.674696+0000) 2020-02-05T23:20:40.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:40.674+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.675317+0000) 2020-02-05T23:20:40.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:40.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.675400+0000) 2020-02-05T23:20:40.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:40.676+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.677098+0000) 2020-02-05T23:20:40.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:40.676+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:40.677429+0000) 2020-02-05T23:20:40.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:40.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:42.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:42.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:42.409784+0000) 2020-02-05T23:20:42.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:42.760+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:42.761144+0000) 2020-02-05T23:20:43.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:43.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:43.922894+0000) 2020-02-05T23:20:43.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:43.922+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:43.923288+0000) 2020-02-05T23:20:45.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:45.690+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.691626+0000) 2020-02-05T23:20:45.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:45.691+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.692196+0000) 2020-02-05T23:20:45.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:45.693+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.694001+0000) 2020-02-05T23:20:45.695 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:45.693+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.694345+0000) 2020-02-05T23:20:45.695 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:45.694+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.694765+0000) 2020-02-05T23:20:45.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:45.694+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.695302+0000) 2020-02-05T23:20:45.696 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:45.695+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.695858+0000) 2020-02-05T23:20:45.697 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:45.694+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.695985+0000) 2020-02-05T23:20:45.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:45.695+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.696638+0000) 2020-02-05T23:20:45.697 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:45.696+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.696725+0000) 2020-02-05T23:20:45.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:45.697+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.697494+0000) 2020-02-05T23:20:45.698 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:45.697+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.697819+0000) 2020-02-05T23:20:45.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:45.697+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.698731+0000) 2020-02-05T23:20:45.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:45.698+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:45.699496+0000) 2020-02-05T23:20:45.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:45.707+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:46.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:46.022+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:46.023192+0000) 2020-02-05T23:20:46.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:46.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:46.298514+0000) 2020-02-05T23:20:47.157 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:47.156+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:47.156409+0000) 2020-02-05T23:20:47.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:47.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:47.162221+0000) 2020-02-05T23:20:47.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:47.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:47.162471+0000) 2020-02-05T23:20:47.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:47.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:47.162750+0000) 2020-02-05T23:20:47.983 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:20:47.989 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:20:49.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:49.547+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:49.548054+0000) 2020-02-05T23:20:49.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:49.671+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:49.672379+0000) 2020-02-05T23:20:50.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:50.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.155282+0000) 2020-02-05T23:20:50.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:50.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.393575+0000) 2020-02-05T23:20:50.714 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:50.712+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.713343+0000) 2020-02-05T23:20:50.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:50.713+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.714973+0000) 2020-02-05T23:20:50.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:50.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.715611+0000) 2020-02-05T23:20:50.717 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:50.715+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.715719+0000) 2020-02-05T23:20:50.717 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:50.715+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.716168+0000) 2020-02-05T23:20:50.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:50.715+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.716887+0000) 2020-02-05T23:20:50.718 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:50.716+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.717300+0000) 2020-02-05T23:20:50.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:50.717+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.718072+0000) 2020-02-05T23:20:50.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:50.717+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.717796+0000) 2020-02-05T23:20:50.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:50.718+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.719485+0000) 2020-02-05T23:20:50.720 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:50.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.719532+0000) 2020-02-05T23:20:50.721 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:50.719+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.719529+0000) 2020-02-05T23:20:50.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:50.719+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.720873+0000) 2020-02-05T23:20:50.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:50.724+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:50.725322+0000) 2020-02-05T23:20:50.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:50.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:52.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:52.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:52.410032+0000) 2020-02-05T23:20:52.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:52.760+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:52.761369+0000) 2020-02-05T23:20:53.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:53.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:53.923123+0000) 2020-02-05T23:20:53.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:53.922+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:53.923559+0000) 2020-02-05T23:20:55.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:20:55.734+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.735466+0000) 2020-02-05T23:20:55.737 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:20:55.734+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.735972+0000) 2020-02-05T23:20:55.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:55.735+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.736532+0000) 2020-02-05T23:20:55.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:20:55.735+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.736980+0000) 2020-02-05T23:20:55.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:55.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.737480+0000) 2020-02-05T23:20:55.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:55.737+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.738208+0000) 2020-02-05T23:20:55.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:20:55.738+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.738829+0000) 2020-02-05T23:20:55.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:20:55.738+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.738931+0000) 2020-02-05T23:20:55.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:20:55.739+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.740425+0000) 2020-02-05T23:20:55.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:55.739+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.740076+0000) 2020-02-05T23:20:55.742 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:55.740+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.740697+0000) 2020-02-05T23:20:55.743 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:55.740+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.741006+0000) 2020-02-05T23:20:55.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:55.740+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.741201+0000) 2020-02-05T23:20:55.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:55.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:55.744649+0000) 2020-02-05T23:20:55.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:20:55.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:20:56.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:20:56.023+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:56.023425+0000) 2020-02-05T23:20:56.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:56.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:56.298717+0000) 2020-02-05T23:20:57.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:20:57.156+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:57.156622+0000) 2020-02-05T23:20:57.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:20:57.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:57.162477+0000) 2020-02-05T23:20:57.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:20:57.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:57.162697+0000) 2020-02-05T23:20:57.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:20:57.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:57.162978+0000) 2020-02-05T23:20:59.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:59.548+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:59.548321+0000) 2020-02-05T23:20:59.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:20:59.680+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:20:59.680476+0000) 2020-02-05T23:21:00.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:00.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.155536+0000) 2020-02-05T23:21:00.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:00.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.393807+0000) 2020-02-05T23:21:00.756 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:00.754+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.755219+0000) 2020-02-05T23:21:00.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:00.754+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.755891+0000) 2020-02-05T23:21:00.758 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:00.756+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.757851+0000) 2020-02-05T23:21:00.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:00.757+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.757907+0000) 2020-02-05T23:21:00.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:00.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.758283+0000) 2020-02-05T23:21:00.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:00.757+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.758996+0000) 2020-02-05T23:21:00.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:00.758+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.759014+0000) 2020-02-05T23:21:00.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:00.759+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.760364+0000) 2020-02-05T23:21:00.761 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:00.760+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.760821+0000) 2020-02-05T23:21:00.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:00.760+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.761259+0000) 2020-02-05T23:21:00.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:00.761+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.761858+0000) 2020-02-05T23:21:00.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:00.761+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.762267+0000) 2020-02-05T23:21:00.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:00.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.764736+0000) 2020-02-05T23:21:00.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:00.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:00.765015+0000) 2020-02-05T23:21:00.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:00.771+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 141980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:02.411 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:02.410+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:02.410276+0000) 2020-02-05T23:21:02.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:02.761+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:02.761608+0000) 2020-02-05T23:21:03.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:03.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:03.923350+0000) 2020-02-05T23:21:03.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:03.923+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:03.923754+0000) 2020-02-05T23:21:05.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:05.777+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.777923+0000) 2020-02-05T23:21:05.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:05.778+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.778760+0000) 2020-02-05T23:21:05.780 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:05.779+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.779277+0000) 2020-02-05T23:21:05.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:05.779+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.779589+0000) 2020-02-05T23:21:05.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:05.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.780522+0000) 2020-02-05T23:21:05.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:05.779+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.780321+0000) 2020-02-05T23:21:05.782 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:05.781+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.781421+0000) 2020-02-05T23:21:05.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:05.782+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.782243+0000) 2020-02-05T23:21:05.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:05.782+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.782597+0000) 2020-02-05T23:21:05.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:05.782+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.782735+0000) 2020-02-05T23:21:05.784 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:05.782+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.783080+0000) 2020-02-05T23:21:05.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:05.782+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.783247+0000) 2020-02-05T23:21:05.785 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:05.782+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.783299+0000) 2020-02-05T23:21:05.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:05.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:05.787581+0000) 2020-02-05T23:21:05.795 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:05.794+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:06.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:06.023+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:06.023678+0000) 2020-02-05T23:21:06.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:06.297+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:06.298928+0000) 2020-02-05T23:21:07.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:07.155+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:07.156868+0000) 2020-02-05T23:21:07.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:07.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:07.162734+0000) 2020-02-05T23:21:07.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:07.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:07.162979+0000) 2020-02-05T23:21:07.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:07.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:07.163221+0000) 2020-02-05T23:21:09.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:09.547+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:09.548600+0000) 2020-02-05T23:21:09.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:09.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:09.688812+0000) 2020-02-05T23:21:10.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:10.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.155765+0000) 2020-02-05T23:21:10.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:10.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.394038+0000) 2020-02-05T23:21:10.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:10.798+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.799473+0000) 2020-02-05T23:21:10.817 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:10.801+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.801672+0000) 2020-02-05T23:21:10.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:10.802+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.802687+0000) 2020-02-05T23:21:10.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:10.803+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.803642+0000) 2020-02-05T23:21:10.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:10.804+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.804817+0000) 2020-02-05T23:21:10.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:10.804+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.805212+0000) 2020-02-05T23:21:10.818 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:10.805+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.805896+0000) 2020-02-05T23:21:10.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:10.799+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.800944+0000) 2020-02-05T23:21:10.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:10.802+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.803198+0000) 2020-02-05T23:21:10.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:10.802+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.803898+0000) 2020-02-05T23:21:10.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:10.803+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.804661+0000) 2020-02-05T23:21:10.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:10.804+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.805197+0000) 2020-02-05T23:21:10.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:10.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.805996+0000) 2020-02-05T23:21:10.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:10.808+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:10.809208+0000) 2020-02-05T23:21:10.821 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:10.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:12.411 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:12.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:12.410529+0000) 2020-02-05T23:21:12.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:12.760+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:12.761858+0000) 2020-02-05T23:21:13.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:13.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:13.923583+0000) 2020-02-05T23:21:13.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:13.922+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:13.923941+0000) 2020-02-05T23:21:15.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:15.821+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.822972+0000) 2020-02-05T23:21:15.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:15.822+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.823667+0000) 2020-02-05T23:21:15.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:15.824+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.825133+0000) 2020-02-05T23:21:15.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:15.824+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.825536+0000) 2020-02-05T23:21:15.827 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:15.825+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.826554+0000) 2020-02-05T23:21:15.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:15.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.825880+0000) 2020-02-05T23:21:15.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:15.826+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.827424+0000) 2020-02-05T23:21:15.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:15.825+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.827127+0000) 2020-02-05T23:21:15.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:15.827+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.828430+0000) 2020-02-05T23:21:15.829 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:15.827+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.829084+0000) 2020-02-05T23:21:15.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:15.828+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.829491+0000) 2020-02-05T23:21:15.830 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:15.828+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.830060+0000) 2020-02-05T23:21:15.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:15.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.830480+0000) 2020-02-05T23:21:15.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:15.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:15.830715+0000) 2020-02-05T23:21:15.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:15.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:16.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:16.022+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:16.023911+0000) 2020-02-05T23:21:16.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:16.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:16.299137+0000) 2020-02-05T23:21:17.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:17.156+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:17.157129+0000) 2020-02-05T23:21:17.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:17.161+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:17.162991+0000) 2020-02-05T23:21:17.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:17.161+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:17.163200+0000) 2020-02-05T23:21:17.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:17.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:17.163512+0000) 2020-02-05T23:21:18.180 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:21:18.186 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:21:19.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:19.548+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:19.548846+0000) 2020-02-05T23:21:19.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:19.696+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:19.697239+0000) 2020-02-05T23:21:20.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:20.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.155995+0000) 2020-02-05T23:21:20.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:20.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.394276+0000) 2020-02-05T23:21:20.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:20.844+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.845960+0000) 2020-02-05T23:21:20.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:20.845+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.846344+0000) 2020-02-05T23:21:20.848 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:20.846+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.847904+0000) 2020-02-05T23:21:20.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:20.847+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.849012+0000) 2020-02-05T23:21:20.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:20.848+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.849547+0000) 2020-02-05T23:21:20.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:20.848+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.850313+0000) 2020-02-05T23:21:20.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:20.849+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.850544+0000) 2020-02-05T23:21:20.852 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:20.848+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.850193+0000) 2020-02-05T23:21:20.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:20.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.851297+0000) 2020-02-05T23:21:20.853 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:20.850+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.851763+0000) 2020-02-05T23:21:20.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:20.849+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.851281+0000) 2020-02-05T23:21:20.853 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:20.850+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.851846+0000) 2020-02-05T23:21:20.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:20.853+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.854927+0000) 2020-02-05T23:21:20.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:20.854+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:20.855163+0000) 2020-02-05T23:21:20.863 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:20.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:22.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:22.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:22.410808+0000) 2020-02-05T23:21:22.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:22.761+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:22.762122+0000) 2020-02-05T23:21:23.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:23.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:23.923818+0000) 2020-02-05T23:21:23.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:23.923+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:23.924144+0000) 2020-02-05T23:21:25.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:25.867+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.868628+0000) 2020-02-05T23:21:25.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:25.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.869048+0000) 2020-02-05T23:21:25.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:25.869+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.870190+0000) 2020-02-05T23:21:25.874 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:25.868+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.869604+0000) 2020-02-05T23:21:25.874 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:25.869+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.870568+0000) 2020-02-05T23:21:25.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:25.869+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.871006+0000) 2020-02-05T23:21:25.875 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:25.871+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.872461+0000) 2020-02-05T23:21:25.875 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:25.870+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.871431+0000) 2020-02-05T23:21:25.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:25.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.871919+0000) 2020-02-05T23:21:25.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:25.872+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.873609+0000) 2020-02-05T23:21:25.877 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:25.872+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.874199+0000) 2020-02-05T23:21:25.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:25.873+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.875135+0000) 2020-02-05T23:21:25.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:25.873+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.874886+0000) 2020-02-05T23:21:25.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:25.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:25.878347+0000) 2020-02-05T23:21:25.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:25.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:26.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:26.022+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:26.024133+0000) 2020-02-05T23:21:26.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:26.298+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:26.299372+0000) 2020-02-05T23:21:27.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:27.156+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:27.157362+0000) 2020-02-05T23:21:27.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:27.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:27.163228+0000) 2020-02-05T23:21:27.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:27.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:27.163473+0000) 2020-02-05T23:21:27.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:27.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:27.163766+0000) 2020-02-05T23:21:29.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:29.548+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:29.549114+0000) 2020-02-05T23:21:29.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:29.704+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:29.705552+0000) 2020-02-05T23:21:30.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:30.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.156258+0000) 2020-02-05T23:21:30.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:30.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.394555+0000) 2020-02-05T23:21:30.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:30.888+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.889899+0000) 2020-02-05T23:21:30.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:30.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.890531+0000) 2020-02-05T23:21:30.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:30.889+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.890859+0000) 2020-02-05T23:21:30.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:30.890+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.891888+0000) 2020-02-05T23:21:30.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:30.891+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.892239+0000) 2020-02-05T23:21:30.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:30.891+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.893387+0000) 2020-02-05T23:21:30.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:30.892+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.894017+0000) 2020-02-05T23:21:30.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:30.892+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.893818+0000) 2020-02-05T23:21:30.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:30.893+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.894560+0000) 2020-02-05T23:21:30.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:30.894+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.895491+0000) 2020-02-05T23:21:30.896 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:30.894+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.895805+0000) 2020-02-05T23:21:30.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:30.894+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.895592+0000) 2020-02-05T23:21:30.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:30.894+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.896306+0000) 2020-02-05T23:21:30.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:30.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:30.900129+0000) 2020-02-05T23:21:30.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:30.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:32.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:32.409+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:32.411100+0000) 2020-02-05T23:21:32.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:32.761+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:32.762358+0000) 2020-02-05T23:21:33.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:33.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:33.924048+0000) 2020-02-05T23:21:33.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:33.923+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:33.924334+0000) 2020-02-05T23:21:35.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:35.910+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.912205+0000) 2020-02-05T23:21:35.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:35.912+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.913946+0000) 2020-02-05T23:21:35.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:35.913+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.914636+0000) 2020-02-05T23:21:35.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:35.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.915330+0000) 2020-02-05T23:21:35.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:35.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.915721+0000) 2020-02-05T23:21:35.917 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:35.915+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.916531+0000) 2020-02-05T23:21:35.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:35.915+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.916693+0000) 2020-02-05T23:21:35.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:35.915+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.917042+0000) 2020-02-05T23:21:35.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:35.916+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.917654+0000) 2020-02-05T23:21:35.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:35.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.917950+0000) 2020-02-05T23:21:35.919 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:35.916+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.918294+0000) 2020-02-05T23:21:35.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:35.917+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.918839+0000) 2020-02-05T23:21:35.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:35.917+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.919214+0000) 2020-02-05T23:21:35.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:35.920+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:35.921867+0000) 2020-02-05T23:21:35.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:35.927+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:36.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:36.023+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:36.024370+0000) 2020-02-05T23:21:36.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:36.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:36.299582+0000) 2020-02-05T23:21:37.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:37.157+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:37.157603+0000) 2020-02-05T23:21:37.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:37.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:37.163486+0000) 2020-02-05T23:21:37.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:37.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:37.163660+0000) 2020-02-05T23:21:37.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:37.162+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:37.164033+0000) 2020-02-05T23:21:39.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:39.548+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:39.549339+0000) 2020-02-05T23:21:39.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:39.713+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:39.713782+0000) 2020-02-05T23:21:40.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:40.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.156489+0000) 2020-02-05T23:21:40.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:40.394+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.394849+0000) 2020-02-05T23:21:40.936 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:40.933+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.934954+0000) 2020-02-05T23:21:40.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:40.934+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.935963+0000) 2020-02-05T23:21:40.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:40.935+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.936487+0000) 2020-02-05T23:21:40.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:40.935+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.936784+0000) 2020-02-05T23:21:40.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:40.936+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.937401+0000) 2020-02-05T23:21:40.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:40.937+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.938560+0000) 2020-02-05T23:21:40.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:40.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.938922+0000) 2020-02-05T23:21:40.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:40.938+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.939339+0000) 2020-02-05T23:21:40.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:40.938+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.939668+0000) 2020-02-05T23:21:40.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:40.938+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.939875+0000) 2020-02-05T23:21:40.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:40.939+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.940340+0000) 2020-02-05T23:21:40.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:40.939+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.941126+0000) 2020-02-05T23:21:40.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:40.940+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.941297+0000) 2020-02-05T23:21:40.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:40.945+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:40.946625+0000) 2020-02-05T23:21:40.950 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:40.947+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:42.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:42.410+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:42.411346+0000) 2020-02-05T23:21:42.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:42.761+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:42.762548+0000) 2020-02-05T23:21:43.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:43.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:43.924314+0000) 2020-02-05T23:21:43.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:43.923+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:43.924572+0000) 2020-02-05T23:21:45.958 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:45.955+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.956847+0000) 2020-02-05T23:21:45.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:45.956+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.957175+0000) 2020-02-05T23:21:45.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:45.956+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.957962+0000) 2020-02-05T23:21:45.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:45.957+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.958686+0000) 2020-02-05T23:21:45.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:45.957+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.958647+0000) 2020-02-05T23:21:45.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:45.958+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.959368+0000) 2020-02-05T23:21:45.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:45.958+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.959699+0000) 2020-02-05T23:21:45.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:45.959+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.960488+0000) 2020-02-05T23:21:45.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:45.959+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.960725+0000) 2020-02-05T23:21:45.963 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:45.959+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.960737+0000) 2020-02-05T23:21:45.963 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:45.959+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.961400+0000) 2020-02-05T23:21:45.963 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:45.960+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.962173+0000) 2020-02-05T23:21:45.964 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:45.961+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.962418+0000) 2020-02-05T23:21:45.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:45.967+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:45.969016+0000) 2020-02-05T23:21:45.973 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:45.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:46.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:46.023+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:46.024573+0000) 2020-02-05T23:21:46.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:46.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:46.299810+0000) 2020-02-05T23:21:47.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:47.157+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:47.157830+0000) 2020-02-05T23:21:47.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:47.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:47.163713+0000) 2020-02-05T23:21:47.165 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:47.163+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:47.163895+0000) 2020-02-05T23:21:47.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:47.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:47.164278+0000) 2020-02-05T23:21:48.594 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:21:48.606 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:21:49.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:49.549+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:49.549572+0000) 2020-02-05T23:21:49.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:49.721+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:49.722199+0000) 2020-02-05T23:21:50.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:50.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.156731+0000) 2020-02-05T23:21:50.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:50.393+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.395089+0000) 2020-02-05T23:21:50.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:50.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.979071+0000) 2020-02-05T23:21:50.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:50.977+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.978921+0000) 2020-02-05T23:21:50.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:50.977+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.979277+0000) 2020-02-05T23:21:50.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:50.978+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.979990+0000) 2020-02-05T23:21:50.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:50.979+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.980998+0000) 2020-02-05T23:21:50.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:50.979+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.980985+0000) 2020-02-05T23:21:50.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:50.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.981604+0000) 2020-02-05T23:21:50.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:50.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.982379+0000) 2020-02-05T23:21:50.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:50.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.982812+0000) 2020-02-05T23:21:50.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:50.982+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.983298+0000) 2020-02-05T23:21:50.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:50.980+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.982218+0000) 2020-02-05T23:21:50.985 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:50.981+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.982862+0000) 2020-02-05T23:21:50.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:50.981+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.983254+0000) 2020-02-05T23:21:50.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:50.987+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:50.989022+0000) 2020-02-05T23:21:50.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:50.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:52.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:52.410+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:52.411645+0000) 2020-02-05T23:21:52.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:52.761+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:52.762799+0000) 2020-02-05T23:21:53.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:53.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:53.924568+0000) 2020-02-05T23:21:53.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:53.924+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:53.924770+0000) 2020-02-05T23:21:55.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:55.994+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.995493+0000) 2020-02-05T23:21:55.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:55.996+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.997533+0000) 2020-02-05T23:21:55.999 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:21:55.996+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.998152+0000) 2020-02-05T23:21:55.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:21:55.998+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.998808+0000) 2020-02-05T23:21:56.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:21:55.999+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.999136+0000) 2020-02-05T23:21:56.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:55.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:55.999890+0000) 2020-02-05T23:21:56.001 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:56.000+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.000330+0000) 2020-02-05T23:21:56.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:21:56.000+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.000723+0000) 2020-02-05T23:21:56.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:21:56.001+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.001395+0000) 2020-02-05T23:21:56.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:56.000+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.001975+0000) 2020-02-05T23:21:56.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:21:56.000+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.002400+0000) 2020-02-05T23:21:56.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:56.001+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.003040+0000) 2020-02-05T23:21:56.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:56.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.005667+0000) 2020-02-05T23:21:56.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:56.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.005918+0000) 2020-02-05T23:21:56.011 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:21:56.009+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:21:56.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:21:56.023+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.024751+0000) 2020-02-05T23:21:56.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:56.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:56.300019+0000) 2020-02-05T23:21:57.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:21:57.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:57.158120+0000) 2020-02-05T23:21:57.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:21:57.162+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:57.163928+0000) 2020-02-05T23:21:57.165 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:21:57.162+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:57.164160+0000) 2020-02-05T23:21:57.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:21:57.163+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:57.164484+0000) 2020-02-05T23:21:59.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:59.549+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:59.549811+0000) 2020-02-05T23:21:59.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:21:59.734+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:21:59.734258+0000) 2020-02-05T23:22:00.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:00.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:00.156979+0000) 2020-02-05T23:22:00.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:00.394+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:00.395347+0000) 2020-02-05T23:22:01.018 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:01.016+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.017302+0000) 2020-02-05T23:22:01.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:01.016+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.017699+0000) 2020-02-05T23:22:01.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:01.017+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.018432+0000) 2020-02-05T23:22:01.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:01.016+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.018302+0000) 2020-02-05T23:22:01.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:01.017+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.018772+0000) 2020-02-05T23:22:01.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:01.018+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.019526+0000) 2020-02-05T23:22:01.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:01.018+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.020308+0000) 2020-02-05T23:22:01.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:01.019+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.020394+0000) 2020-02-05T23:22:01.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:01.019+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.021291+0000) 2020-02-05T23:22:01.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:01.020+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.021441+0000) 2020-02-05T23:22:01.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:01.020+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.021831+0000) 2020-02-05T23:22:01.023 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:01.020+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.022197+0000) 2020-02-05T23:22:01.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:01.021+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.022447+0000) 2020-02-05T23:22:01.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:01.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:01.026069+0000) 2020-02-05T23:22:01.033 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:01.031+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:02.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:02.411+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:02.411919+0000) 2020-02-05T23:22:02.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:02.762+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:02.763034+0000) 2020-02-05T23:22:03.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:03.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:03.924763+0000) 2020-02-05T23:22:03.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:03.923+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:03.924966+0000) 2020-02-05T23:22:06.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:06.024+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.024950+0000) 2020-02-05T23:22:06.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:06.038+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.039263+0000) 2020-02-05T23:22:06.040 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:06.038+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.039689+0000) 2020-02-05T23:22:06.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:06.039+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.040285+0000) 2020-02-05T23:22:06.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:06.040+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.041283+0000) 2020-02-05T23:22:06.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:06.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.041706+0000) 2020-02-05T23:22:06.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:06.041+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.042141+0000) 2020-02-05T23:22:06.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:06.041+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.042455+0000) 2020-02-05T23:22:06.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:06.042+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.043189+0000) 2020-02-05T23:22:06.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:06.042+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.043397+0000) 2020-02-05T23:22:06.045 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:06.043+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.043864+0000) 2020-02-05T23:22:06.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:06.044+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.045000+0000) 2020-02-05T23:22:06.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:06.045+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.045928+0000) 2020-02-05T23:22:06.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:06.046+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.046622+0000) 2020-02-05T23:22:06.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:06.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.048944+0000) 2020-02-05T23:22:06.054 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:06.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142512 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:06.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:06.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:06.300281+0000) 2020-02-05T23:22:07.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:07.157+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:07.158391+0000) 2020-02-05T23:22:07.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:07.163+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:07.164208+0000) 2020-02-05T23:22:07.165 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:07.163+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:07.164345+0000) 2020-02-05T23:22:07.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:07.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:07.164731+0000) 2020-02-05T23:22:09.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:09.548+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:09.550044+0000) 2020-02-05T23:22:09.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:09.741+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:09.742598+0000) 2020-02-05T23:22:10.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:10.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:10.157202+0000) 2020-02-05T23:22:10.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:10.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:10.395635+0000) 2020-02-05T23:22:11.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:11.057+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.058702+0000) 2020-02-05T23:22:11.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:11.058+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.059260+0000) 2020-02-05T23:22:11.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:11.058+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.060012+0000) 2020-02-05T23:22:11.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:11.059+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.060468+0000) 2020-02-05T23:22:11.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:11.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.060994+0000) 2020-02-05T23:22:11.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:11.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.062661+0000) 2020-02-05T23:22:11.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:11.062+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.063339+0000) 2020-02-05T23:22:11.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:11.062+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.063005+0000) 2020-02-05T23:22:11.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:11.063+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.063813+0000) 2020-02-05T23:22:11.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:11.063+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.064216+0000) 2020-02-05T23:22:11.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:11.065+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.065420+0000) 2020-02-05T23:22:11.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:11.064+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.065833+0000) 2020-02-05T23:22:11.067 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:11.065+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.066217+0000) 2020-02-05T23:22:11.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:11.068+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:11.070035+0000) 2020-02-05T23:22:11.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:11.075+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:12.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:12.411+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:12.412123+0000) 2020-02-05T23:22:12.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:12.762+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:12.763441+0000) 2020-02-05T23:22:13.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:13.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:13.924995+0000) 2020-02-05T23:22:13.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:13.924+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:13.925157+0000) 2020-02-05T23:22:16.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:16.024+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.025189+0000) 2020-02-05T23:22:16.083 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:16.080+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.081921+0000) 2020-02-05T23:22:16.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:16.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.082371+0000) 2020-02-05T23:22:16.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:16.083+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.083649+0000) 2020-02-05T23:22:16.085 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:16.083+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.084042+0000) 2020-02-05T23:22:16.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:16.083+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.084644+0000) 2020-02-05T23:22:16.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:16.083+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.084764+0000) 2020-02-05T23:22:16.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:16.084+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.084667+0000) 2020-02-05T23:22:16.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:16.085+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.085517+0000) 2020-02-05T23:22:16.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:16.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.086217+0000) 2020-02-05T23:22:16.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:16.085+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.086145+0000) 2020-02-05T23:22:16.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:16.085+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.086666+0000) 2020-02-05T23:22:16.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:16.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.087244+0000) 2020-02-05T23:22:16.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:16.086+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.087112+0000) 2020-02-05T23:22:16.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:16.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.091858+0000) 2020-02-05T23:22:16.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:16.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:16.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:16.299+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:16.300560+0000) 2020-02-05T23:22:17.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:17.157+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:17.158656+0000) 2020-02-05T23:22:17.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:17.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:17.164513+0000) 2020-02-05T23:22:17.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:17.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:17.164641+0000) 2020-02-05T23:22:17.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:17.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:17.164981+0000) 2020-02-05T23:22:18.739 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:22:18.746 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:22:19.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:19.549+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:19.550351+0000) 2020-02-05T23:22:19.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:19.750+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:19.750883+0000) 2020-02-05T23:22:20.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:20.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:20.157427+0000) 2020-02-05T23:22:20.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:20.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:20.395932+0000) 2020-02-05T23:22:21.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:21.102+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.103748+0000) 2020-02-05T23:22:21.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:21.103+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.104323+0000) 2020-02-05T23:22:21.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:21.104+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.104917+0000) 2020-02-05T23:22:21.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:21.104+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.105283+0000) 2020-02-05T23:22:21.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:21.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.105767+0000) 2020-02-05T23:22:21.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:21.105+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.106189+0000) 2020-02-05T23:22:21.107 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:21.105+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.106498+0000) 2020-02-05T23:22:21.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:21.106+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.107977+0000) 2020-02-05T23:22:21.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:21.107+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.107767+0000) 2020-02-05T23:22:21.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:21.107+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.108392+0000) 2020-02-05T23:22:21.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:21.108+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.109412+0000) 2020-02-05T23:22:21.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:21.109+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.110066+0000) 2020-02-05T23:22:21.111 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:21.109+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.110000+0000) 2020-02-05T23:22:21.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:21.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:21.113900+0000) 2020-02-05T23:22:21.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:21.119+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:22.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:22.412+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:22.412358+0000) 2020-02-05T23:22:22.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:22.762+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:22.763706+0000) 2020-02-05T23:22:23.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:23.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:23.925267+0000) 2020-02-05T23:22:23.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:23.924+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:23.925413+0000) 2020-02-05T23:22:26.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:26.025+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.025532+0000) 2020-02-05T23:22:26.126 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:26.123+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.124993+0000) 2020-02-05T23:22:26.127 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:26.124+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.125760+0000) 2020-02-05T23:22:26.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:26.125+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.126265+0000) 2020-02-05T23:22:26.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:26.125+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.126517+0000) 2020-02-05T23:22:26.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:26.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.127246+0000) 2020-02-05T23:22:26.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:26.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.127626+0000) 2020-02-05T23:22:26.129 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:26.127+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.128530+0000) 2020-02-05T23:22:26.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:26.128+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.128760+0000) 2020-02-05T23:22:26.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:26.128+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.129003+0000) 2020-02-05T23:22:26.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:26.128+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.129537+0000) 2020-02-05T23:22:26.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:26.129+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.130115+0000) 2020-02-05T23:22:26.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:26.129+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.130630+0000) 2020-02-05T23:22:26.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:26.130+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.131371+0000) 2020-02-05T23:22:26.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:26.133+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.134765+0000) 2020-02-05T23:22:26.143 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:26.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:26.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:26.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:26.300771+0000) 2020-02-05T23:22:27.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:27.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:27.158892+0000) 2020-02-05T23:22:27.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:27.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:27.164739+0000) 2020-02-05T23:22:27.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:27.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:27.164899+0000) 2020-02-05T23:22:27.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:27.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:27.165239+0000) 2020-02-05T23:22:29.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:29.550+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:29.550588+0000) 2020-02-05T23:22:29.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:29.758+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:29.759106+0000) 2020-02-05T23:22:30.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:30.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:30.157678+0000) 2020-02-05T23:22:30.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:30.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:30.396183+0000) 2020-02-05T23:22:31.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:31.146+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.147481+0000) 2020-02-05T23:22:31.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:31.147+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.148513+0000) 2020-02-05T23:22:31.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:31.148+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.148995+0000) 2020-02-05T23:22:31.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:31.148+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.149405+0000) 2020-02-05T23:22:31.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:31.149+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.149791+0000) 2020-02-05T23:22:31.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:31.149+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.150183+0000) 2020-02-05T23:22:31.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:31.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.150670+0000) 2020-02-05T23:22:31.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:31.150+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.150974+0000) 2020-02-05T23:22:31.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:31.150+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.151312+0000) 2020-02-05T23:22:31.153 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:31.151+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.152119+0000) 2020-02-05T23:22:31.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:31.152+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.152638+0000) 2020-02-05T23:22:31.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:31.152+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.152886+0000) 2020-02-05T23:22:31.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:31.152+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.153715+0000) 2020-02-05T23:22:31.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:31.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:31.159744+0000) 2020-02-05T23:22:31.166 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:31.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:32.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:32.412+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:32.412642+0000) 2020-02-05T23:22:32.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:32.762+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:32.763906+0000) 2020-02-05T23:22:33.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:33.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:33.925458+0000) 2020-02-05T23:22:33.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:33.924+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:33.925660+0000) 2020-02-05T23:22:36.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:36.025+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.025752+0000) 2020-02-05T23:22:36.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:36.169+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.170860+0000) 2020-02-05T23:22:36.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:36.171+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.172343+0000) 2020-02-05T23:22:36.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:36.171+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.172089+0000) 2020-02-05T23:22:36.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:36.171+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.172954+0000) 2020-02-05T23:22:36.175 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:36.173+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.173601+0000) 2020-02-05T23:22:36.175 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:36.173+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.173990+0000) 2020-02-05T23:22:36.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:36.173+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.174298+0000) 2020-02-05T23:22:36.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:36.174+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.175051+0000) 2020-02-05T23:22:36.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:36.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.175621+0000) 2020-02-05T23:22:36.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:36.175+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.175801+0000) 2020-02-05T23:22:36.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:36.174+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.176000+0000) 2020-02-05T23:22:36.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:36.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.176286+0000) 2020-02-05T23:22:36.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:36.176+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.176514+0000) 2020-02-05T23:22:36.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:36.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.181891+0000) 2020-02-05T23:22:36.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:36.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:36.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:36.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:36.301042+0000) 2020-02-05T23:22:37.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:37.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:37.159070+0000) 2020-02-05T23:22:37.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:37.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:37.164996+0000) 2020-02-05T23:22:37.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:37.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:37.165176+0000) 2020-02-05T23:22:37.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:37.165+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:37.165518+0000) 2020-02-05T23:22:39.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:39.550+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:39.550831+0000) 2020-02-05T23:22:39.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:39.766+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:39.766567+0000) 2020-02-05T23:22:40.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:40.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:40.157937+0000) 2020-02-05T23:22:40.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:40.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:40.396418+0000) 2020-02-05T23:22:41.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:41.190+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.191629+0000) 2020-02-05T23:22:41.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:41.191+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.193268+0000) 2020-02-05T23:22:41.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:41.192+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.193885+0000) 2020-02-05T23:22:41.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:41.192+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.193650+0000) 2020-02-05T23:22:41.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:41.193+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.194351+0000) 2020-02-05T23:22:41.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:41.194+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.196015+0000) 2020-02-05T23:22:41.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:41.194+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.195323+0000) 2020-02-05T23:22:41.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:41.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.195765+0000) 2020-02-05T23:22:41.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:41.195+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.196218+0000) 2020-02-05T23:22:41.198 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:41.195+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.196827+0000) 2020-02-05T23:22:41.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:41.196+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.197586+0000) 2020-02-05T23:22:41.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:41.196+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.197969+0000) 2020-02-05T23:22:41.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:41.196+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.197933+0000) 2020-02-05T23:22:41.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:41.202+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:41.203093+0000) 2020-02-05T23:22:41.208 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:41.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142804 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:42.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:42.411+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:42.412887+0000) 2020-02-05T23:22:42.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:42.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:42.764156+0000) 2020-02-05T23:22:43.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:43.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:43.925688+0000) 2020-02-05T23:22:43.927 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:43.925+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:43.925859+0000) 2020-02-05T23:22:46.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:46.025+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.026010+0000) 2020-02-05T23:22:46.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:46.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.214035+0000) 2020-02-05T23:22:46.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:46.214+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.215154+0000) 2020-02-05T23:22:46.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:46.213+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.214962+0000) 2020-02-05T23:22:46.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:46.214+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.216050+0000) 2020-02-05T23:22:46.217 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:46.215+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.216442+0000) 2020-02-05T23:22:46.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:46.215+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.216810+0000) 2020-02-05T23:22:46.218 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:46.216+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.217211+0000) 2020-02-05T23:22:46.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:46.216+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.217635+0000) 2020-02-05T23:22:46.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:46.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.218027+0000) 2020-02-05T23:22:46.219 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:46.216+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.218236+0000) 2020-02-05T23:22:46.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:46.217+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.218394+0000) 2020-02-05T23:22:46.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:46.217+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.218928+0000) 2020-02-05T23:22:46.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:46.217+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.219364+0000) 2020-02-05T23:22:46.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:46.226+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.227538+0000) 2020-02-05T23:22:46.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:46.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:46.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:46.301+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:46.301327+0000) 2020-02-05T23:22:47.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:47.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:47.159303+0000) 2020-02-05T23:22:47.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:47.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:47.165223+0000) 2020-02-05T23:22:47.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:47.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:47.165416+0000) 2020-02-05T23:22:47.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:47.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:47.165762+0000) 2020-02-05T23:22:48.882 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:22:48.888 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:22:49.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:49.550+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:49.551066+0000) 2020-02-05T23:22:49.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:49.774+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:49.774994+0000) 2020-02-05T23:22:50.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:50.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:50.158179+0000) 2020-02-05T23:22:50.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:50.395+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:50.396753+0000) 2020-02-05T23:22:51.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:51.234+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.236187+0000) 2020-02-05T23:22:51.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:51.237+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.237168+0000) 2020-02-05T23:22:51.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:51.235+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.236817+0000) 2020-02-05T23:22:51.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:51.236+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.237743+0000) 2020-02-05T23:22:51.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:51.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.238124+0000) 2020-02-05T23:22:51.240 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:51.238+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.238452+0000) 2020-02-05T23:22:51.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:51.238+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.238876+0000) 2020-02-05T23:22:51.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:51.237+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.238962+0000) 2020-02-05T23:22:51.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:51.237+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.239228+0000) 2020-02-05T23:22:51.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:51.238+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.239990+0000) 2020-02-05T23:22:51.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:51.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.239810+0000) 2020-02-05T23:22:51.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:51.240+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.240299+0000) 2020-02-05T23:22:51.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:51.240+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.240733+0000) 2020-02-05T23:22:51.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:51.245+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:51.245218+0000) 2020-02-05T23:22:51.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:51.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:52.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:52.412+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:52.413171+0000) 2020-02-05T23:22:52.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:52.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:52.764329+0000) 2020-02-05T23:22:53.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:53.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:53.925936+0000) 2020-02-05T23:22:53.927 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:53.926+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:53.926202+0000) 2020-02-05T23:22:56.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:56.025+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.026306+0000) 2020-02-05T23:22:56.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:56.256+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.257375+0000) 2020-02-05T23:22:56.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:56.256+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.257845+0000) 2020-02-05T23:22:56.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:22:56.257+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.258344+0000) 2020-02-05T23:22:56.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:56.257+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.258850+0000) 2020-02-05T23:22:56.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:22:56.257+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.258948+0000) 2020-02-05T23:22:56.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:22:56.258+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.259682+0000) 2020-02-05T23:22:56.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:56.258+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.260132+0000) 2020-02-05T23:22:56.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:56.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.260022+0000) 2020-02-05T23:22:56.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:56.258+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.260406+0000) 2020-02-05T23:22:56.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:22:56.260+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.261562+0000) 2020-02-05T23:22:56.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:22:56.260+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.262030+0000) 2020-02-05T23:22:56.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:22:56.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.262148+0000) 2020-02-05T23:22:56.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:22:56.261+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.262960+0000) 2020-02-05T23:22:56.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:56.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.266655+0000) 2020-02-05T23:22:56.274 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:22:56.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142925 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:22:56.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:56.300+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:56.301566+0000) 2020-02-05T23:22:57.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:22:57.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:57.159544+0000) 2020-02-05T23:22:57.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:22:57.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:57.165480+0000) 2020-02-05T23:22:57.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:22:57.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:57.165662+0000) 2020-02-05T23:22:57.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:22:57.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:57.165944+0000) 2020-02-05T23:22:59.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:59.550+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:59.551346+0000) 2020-02-05T23:22:59.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:22:59.782+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:22:59.783828+0000) 2020-02-05T23:23:00.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:00.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:00.158409+0000) 2020-02-05T23:23:00.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:00.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:00.397008+0000) 2020-02-05T23:23:01.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:01.278+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.279502+0000) 2020-02-05T23:23:01.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:01.279+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.280531+0000) 2020-02-05T23:23:01.281 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:01.279+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.280958+0000) 2020-02-05T23:23:01.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:01.280+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.281366+0000) 2020-02-05T23:23:01.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:01.281+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.282271+0000) 2020-02-05T23:23:01.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:01.280+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.282077+0000) 2020-02-05T23:23:01.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:01.281+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.282838+0000) 2020-02-05T23:23:01.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:01.282+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.283643+0000) 2020-02-05T23:23:01.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:01.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.284264+0000) 2020-02-05T23:23:01.285 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:01.282+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.284237+0000) 2020-02-05T23:23:01.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:01.283+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.284830+0000) 2020-02-05T23:23:01.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:01.284+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.285155+0000) 2020-02-05T23:23:01.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:01.284+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.285454+0000) 2020-02-05T23:23:01.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:01.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:01.290651+0000) 2020-02-05T23:23:01.296 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:01.294+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 142969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:02.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:02.412+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:02.413468+0000) 2020-02-05T23:23:02.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:02.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:02.764593+0000) 2020-02-05T23:23:03.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:03.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:03.926166+0000) 2020-02-05T23:23:03.927 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:03.925+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:03.926430+0000) 2020-02-05T23:23:06.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:06.025+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.026551+0000) 2020-02-05T23:23:06.302 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:06.300+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.301435+0000) 2020-02-05T23:23:06.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:06.301+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.303030+0000) 2020-02-05T23:23:06.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:06.302+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.303580+0000) 2020-02-05T23:23:06.305 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:06.303+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.304310+0000) 2020-02-05T23:23:06.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:06.302+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.303986+0000) 2020-02-05T23:23:06.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:06.303+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.304947+0000) 2020-02-05T23:23:06.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:06.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.305536+0000) 2020-02-05T23:23:06.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:06.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.305664+0000) 2020-02-05T23:23:06.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:06.304+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.306294+0000) 2020-02-05T23:23:06.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:06.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.306632+0000) 2020-02-05T23:23:06.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:06.305+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.307285+0000) 2020-02-05T23:23:06.308 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:06.306+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.307410+0000) 2020-02-05T23:23:06.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:06.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.311812+0000) 2020-02-05T23:23:06.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:06.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:06.312856+0000) 2020-02-05T23:23:06.319 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:06.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:07.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:07.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:07.159808+0000) 2020-02-05T23:23:07.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:07.164+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:07.165710+0000) 2020-02-05T23:23:07.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:07.164+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:07.165890+0000) 2020-02-05T23:23:07.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:07.164+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:07.166142+0000) 2020-02-05T23:23:09.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:09.550+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:09.551581+0000) 2020-02-05T23:23:09.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:09.791+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:09.792161+0000) 2020-02-05T23:23:10.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:10.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:10.158665+0000) 2020-02-05T23:23:10.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:10.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:10.397260+0000) 2020-02-05T23:23:11.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:11.324+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.325240+0000) 2020-02-05T23:23:11.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:11.324+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.325549+0000) 2020-02-05T23:23:11.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:11.324+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.325823+0000) 2020-02-05T23:23:11.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:11.325+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.326449+0000) 2020-02-05T23:23:11.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:11.325+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.327129+0000) 2020-02-05T23:23:11.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:11.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.327268+0000) 2020-02-05T23:23:11.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:11.326+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.327670+0000) 2020-02-05T23:23:11.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:11.326+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.327736+0000) 2020-02-05T23:23:11.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:11.326+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.327750+0000) 2020-02-05T23:23:11.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:11.327+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.328127+0000) 2020-02-05T23:23:11.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:11.327+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.328480+0000) 2020-02-05T23:23:11.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:11.328+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.329606+0000) 2020-02-05T23:23:11.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:11.328+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.329740+0000) 2020-02-05T23:23:11.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:11.332+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:11.333353+0000) 2020-02-05T23:23:11.340 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:11.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:12.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:12.413+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:12.413718+0000) 2020-02-05T23:23:12.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:12.763+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:12.764817+0000) 2020-02-05T23:23:13.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:13.051+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:13.052185+0000) 2020-02-05T23:23:13.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:13.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:13.926416+0000) 2020-02-05T23:23:13.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:13.925+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:13.926622+0000) 2020-02-05T23:23:16.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:16.026+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.026785+0000) 2020-02-05T23:23:16.346 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:16.343+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.345035+0000) 2020-02-05T23:23:16.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:16.344+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.345328+0000) 2020-02-05T23:23:16.347 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:16.344+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.346082+0000) 2020-02-05T23:23:16.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:16.345+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.346997+0000) 2020-02-05T23:23:16.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:16.346+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.347301+0000) 2020-02-05T23:23:16.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:16.346+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.347487+0000) 2020-02-05T23:23:16.349 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:16.347+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.349004+0000) 2020-02-05T23:23:16.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:16.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.348970+0000) 2020-02-05T23:23:16.350 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:16.347+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.348512+0000) 2020-02-05T23:23:16.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:16.348+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.349574+0000) 2020-02-05T23:23:16.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:16.348+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.350025+0000) 2020-02-05T23:23:16.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:16.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.350499+0000) 2020-02-05T23:23:16.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:16.349+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.350560+0000) 2020-02-05T23:23:16.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:16.354+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:16.355418+0000) 2020-02-05T23:23:16.360 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:16.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:17.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:17.158+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:17.159984+0000) 2020-02-05T23:23:17.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:17.165+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:17.165949+0000) 2020-02-05T23:23:17.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:17.165+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:17.166099+0000) 2020-02-05T23:23:17.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:17.165+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:17.166376+0000) 2020-02-05T23:23:19.049 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:23:19.056 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:23:19.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:19.551+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:19.551816+0000) 2020-02-05T23:23:19.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:19.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:19.804010+0000) 2020-02-05T23:23:20.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:20.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:20.158938+0000) 2020-02-05T23:23:20.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:20.396+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:20.397494+0000) 2020-02-05T23:23:21.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:21.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.366132+0000) 2020-02-05T23:23:21.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:21.366+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.367552+0000) 2020-02-05T23:23:21.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:21.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.367890+0000) 2020-02-05T23:23:21.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:21.367+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.368249+0000) 2020-02-05T23:23:21.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:21.367+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.368420+0000) 2020-02-05T23:23:21.370 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:21.367+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.368995+0000) 2020-02-05T23:23:21.370 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:21.368+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.369602+0000) 2020-02-05T23:23:21.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:21.368+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.369733+0000) 2020-02-05T23:23:21.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:21.369+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.370536+0000) 2020-02-05T23:23:21.371 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:21.369+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.370913+0000) 2020-02-05T23:23:21.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:21.370+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.371484+0000) 2020-02-05T23:23:21.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:21.370+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.371842+0000) 2020-02-05T23:23:21.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:21.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.372088+0000) 2020-02-05T23:23:21.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:21.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:21.375530+0000) 2020-02-05T23:23:21.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:21.380+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143135 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:22.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:22.412+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:22.413967+0000) 2020-02-05T23:23:22.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:22.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:22.765048+0000) 2020-02-05T23:23:23.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:23.051+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:23.052409+0000) 2020-02-05T23:23:23.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:23.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:23.926654+0000) 2020-02-05T23:23:23.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:23.926+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:23.926845+0000) 2020-02-05T23:23:26.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:26.026+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.027039+0000) 2020-02-05T23:23:26.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:26.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.387806+0000) 2020-02-05T23:23:26.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:26.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.388383+0000) 2020-02-05T23:23:26.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:26.388+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.389835+0000) 2020-02-05T23:23:26.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:26.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.390383+0000) 2020-02-05T23:23:26.392 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:26.389+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.390546+0000) 2020-02-05T23:23:26.392 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:26.390+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.391096+0000) 2020-02-05T23:23:26.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:26.390+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.391538+0000) 2020-02-05T23:23:26.393 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:26.390+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.391912+0000) 2020-02-05T23:23:26.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:26.391+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.392305+0000) 2020-02-05T23:23:26.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:26.392+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.393283+0000) 2020-02-05T23:23:26.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:26.391+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.393381+0000) 2020-02-05T23:23:26.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:26.392+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.393639+0000) 2020-02-05T23:23:26.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:26.391+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.393114+0000) 2020-02-05T23:23:26.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:26.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:26.399488+0000) 2020-02-05T23:23:26.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:26.401+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143175 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:27.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:27.159+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:27.160193+0000) 2020-02-05T23:23:27.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:27.165+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:27.166215+0000) 2020-02-05T23:23:27.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:27.165+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:27.166362+0000) 2020-02-05T23:23:27.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:27.165+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:27.166658+0000) 2020-02-05T23:23:29.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:29.551+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:29.552063+0000) 2020-02-05T23:23:29.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:29.812+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:29.812553+0000) 2020-02-05T23:23:30.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:30.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:30.159192+0000) 2020-02-05T23:23:30.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:30.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:30.397735+0000) 2020-02-05T23:23:31.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:31.408+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.409767+0000) 2020-02-05T23:23:31.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:31.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.408480+0000) 2020-02-05T23:23:31.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:31.409+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.410926+0000) 2020-02-05T23:23:31.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:31.407+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.408782+0000) 2020-02-05T23:23:31.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:31.408+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.409213+0000) 2020-02-05T23:23:31.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:31.409+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.410179+0000) 2020-02-05T23:23:31.414 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:31.410+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.411338+0000) 2020-02-05T23:23:31.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:31.410+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.411765+0000) 2020-02-05T23:23:31.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:31.411+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.412192+0000) 2020-02-05T23:23:31.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:31.412+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.413896+0000) 2020-02-05T23:23:31.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:31.412+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.413576+0000) 2020-02-05T23:23:31.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:31.412+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.414075+0000) 2020-02-05T23:23:31.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:31.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.414537+0000) 2020-02-05T23:23:31.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:31.415+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:31.416867+0000) 2020-02-05T23:23:31.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:31.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:32.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:32.413+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:32.414238+0000) 2020-02-05T23:23:32.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:32.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:32.765326+0000) 2020-02-05T23:23:33.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:33.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:33.052641+0000) 2020-02-05T23:23:33.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:33.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:33.926913+0000) 2020-02-05T23:23:33.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:33.925+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:33.927044+0000) 2020-02-05T23:23:36.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:36.026+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.027324+0000) 2020-02-05T23:23:36.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:36.429+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.430456+0000) 2020-02-05T23:23:36.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:36.430+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.431464+0000) 2020-02-05T23:23:36.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:36.432+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.433369+0000) 2020-02-05T23:23:36.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:36.432+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.433820+0000) 2020-02-05T23:23:36.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:36.433+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.434411+0000) 2020-02-05T23:23:36.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:36.434+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.435131+0000) 2020-02-05T23:23:36.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:36.434+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.434745+0000) 2020-02-05T23:23:36.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:36.435+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.436081+0000) 2020-02-05T23:23:36.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:36.435+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.437005+0000) 2020-02-05T23:23:36.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:36.436+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.437299+0000) 2020-02-05T23:23:36.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:36.437+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.437636+0000) 2020-02-05T23:23:36.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:36.437+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.437760+0000) 2020-02-05T23:23:36.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:36.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.441338+0000) 2020-02-05T23:23:36.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:36.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:36.441597+0000) 2020-02-05T23:23:36.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:36.447+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:37.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:37.160+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:37.160412+0000) 2020-02-05T23:23:37.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:37.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:37.166468+0000) 2020-02-05T23:23:37.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:37.166+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:37.166604+0000) 2020-02-05T23:23:37.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:37.166+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:37.166828+0000) 2020-02-05T23:23:39.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:39.552+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:39.552362+0000) 2020-02-05T23:23:39.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:39.820+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:39.820886+0000) 2020-02-05T23:23:40.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:40.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:40.159425+0000) 2020-02-05T23:23:40.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:40.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:40.397996+0000) 2020-02-05T23:23:41.454 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:41.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.453677+0000) 2020-02-05T23:23:41.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:41.453+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.454201+0000) 2020-02-05T23:23:41.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:41.454+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.454769+0000) 2020-02-05T23:23:41.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:41.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.455249+0000) 2020-02-05T23:23:41.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:41.455+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.455670+0000) 2020-02-05T23:23:41.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:41.456+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.456751+0000) 2020-02-05T23:23:41.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:41.456+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.457836+0000) 2020-02-05T23:23:41.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:41.457+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.458248+0000) 2020-02-05T23:23:41.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:41.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.458594+0000) 2020-02-05T23:23:41.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:41.457+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.459000+0000) 2020-02-05T23:23:41.460 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:41.458+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.459229+0000) 2020-02-05T23:23:41.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:41.458+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.459193+0000) 2020-02-05T23:23:41.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:41.459+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.459442+0000) 2020-02-05T23:23:41.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:41.463+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:41.464349+0000) 2020-02-05T23:23:41.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:41.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:42.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:42.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:42.414503+0000) 2020-02-05T23:23:42.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:42.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:42.765557+0000) 2020-02-05T23:23:43.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:43.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:43.052899+0000) 2020-02-05T23:23:43.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:43.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:43.927153+0000) 2020-02-05T23:23:43.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:43.926+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:43.927308+0000) 2020-02-05T23:23:46.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:46.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.027547+0000) 2020-02-05T23:23:46.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:46.474+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.475236+0000) 2020-02-05T23:23:46.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:46.476+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.477033+0000) 2020-02-05T23:23:46.478 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:46.477+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.477629+0000) 2020-02-05T23:23:46.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:46.477+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.478052+0000) 2020-02-05T23:23:46.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:46.478+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.478702+0000) 2020-02-05T23:23:46.480 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:46.478+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.479055+0000) 2020-02-05T23:23:46.480 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:46.479+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.479839+0000) 2020-02-05T23:23:46.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:46.480+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.480306+0000) 2020-02-05T23:23:46.481 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:46.480+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.480584+0000) 2020-02-05T23:23:46.482 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:46.480+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.481019+0000) 2020-02-05T23:23:46.482 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:46.480+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.481322+0000) 2020-02-05T23:23:46.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:46.481+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.481934+0000) 2020-02-05T23:23:46.483 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:46.481+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.481985+0000) 2020-02-05T23:23:46.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:46.487+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:46.488044+0000) 2020-02-05T23:23:46.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:46.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:47.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:47.160+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:47.160620+0000) 2020-02-05T23:23:47.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:47.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:47.166720+0000) 2020-02-05T23:23:47.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:47.166+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:47.166873+0000) 2020-02-05T23:23:47.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:47.166+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:47.167035+0000) 2020-02-05T23:23:49.246 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:23:49.252 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:23:49.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:49.551+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:49.552604+0000) 2020-02-05T23:23:49.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:49.828+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:49.829122+0000) 2020-02-05T23:23:50.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:50.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:50.159655+0000) 2020-02-05T23:23:50.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:50.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:50.398247+0000) 2020-02-05T23:23:51.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:51.497+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.497693+0000) 2020-02-05T23:23:51.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:51.497+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.498661+0000) 2020-02-05T23:23:51.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:51.498+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.499104+0000) 2020-02-05T23:23:51.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:51.498+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.499859+0000) 2020-02-05T23:23:51.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:51.499+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.500275+0000) 2020-02-05T23:23:51.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:51.500+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.500562+0000) 2020-02-05T23:23:51.502 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:51.500+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.500913+0000) 2020-02-05T23:23:51.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:51.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.501596+0000) 2020-02-05T23:23:51.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:51.501+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.502298+0000) 2020-02-05T23:23:51.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:51.502+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.502475+0000) 2020-02-05T23:23:51.504 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:51.502+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.503234+0000) 2020-02-05T23:23:51.504 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:51.502+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.503605+0000) 2020-02-05T23:23:51.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:51.502+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.504047+0000) 2020-02-05T23:23:51.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:51.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:51.508662+0000) 2020-02-05T23:23:51.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:51.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143384 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:52.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:52.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:52.414717+0000) 2020-02-05T23:23:52.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:52.764+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:52.765772+0000) 2020-02-05T23:23:53.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:53.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:53.053156+0000) 2020-02-05T23:23:53.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:53.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:53.927351+0000) 2020-02-05T23:23:53.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:53.926+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:53.927518+0000) 2020-02-05T23:23:56.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:56.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.027797+0000) 2020-02-05T23:23:56.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:23:56.519+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.520230+0000) 2020-02-05T23:23:56.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:23:56.520+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.521152+0000) 2020-02-05T23:23:56.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:56.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.521623+0000) 2020-02-05T23:23:56.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:56.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.522113+0000) 2020-02-05T23:23:56.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:56.521+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.522420+0000) 2020-02-05T23:23:56.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:23:56.521+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.522600+0000) 2020-02-05T23:23:56.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:23:56.522+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.523307+0000) 2020-02-05T23:23:56.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:56.522+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.523230+0000) 2020-02-05T23:23:56.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:23:56.523+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.524160+0000) 2020-02-05T23:23:56.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:56.523+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.524171+0000) 2020-02-05T23:23:56.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:23:56.523+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.524408+0000) 2020-02-05T23:23:56.525 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:56.524+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.524540+0000) 2020-02-05T23:23:56.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:23:56.525+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.525418+0000) 2020-02-05T23:23:56.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:56.529+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:56.530111+0000) 2020-02-05T23:23:56.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:23:56.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143423 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:23:57.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:23:57.159+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:57.160816+0000) 2020-02-05T23:23:57.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:23:57.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:57.166940+0000) 2020-02-05T23:23:57.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:23:57.166+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:57.167110+0000) 2020-02-05T23:23:57.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:23:57.166+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:57.167203+0000) 2020-02-05T23:23:59.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:59.551+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:59.552840+0000) 2020-02-05T23:23:59.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:23:59.836+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:23:59.837272+0000) 2020-02-05T23:24:00.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:00.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:00.159889+0000) 2020-02-05T23:24:00.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:00.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:00.398504+0000) 2020-02-05T23:24:01.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:01.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.542640+0000) 2020-02-05T23:24:01.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:01.542+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.543403+0000) 2020-02-05T23:24:01.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:01.543+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.543631+0000) 2020-02-05T23:24:01.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:01.542+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.543929+0000) 2020-02-05T23:24:01.546 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:01.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.544592+0000) 2020-02-05T23:24:01.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:01.543+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.544806+0000) 2020-02-05T23:24:01.546 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:01.544+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.545304+0000) 2020-02-05T23:24:01.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:01.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.545835+0000) 2020-02-05T23:24:01.547 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:01.545+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.546497+0000) 2020-02-05T23:24:01.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:01.545+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.545985+0000) 2020-02-05T23:24:01.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:01.546+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.546983+0000) 2020-02-05T23:24:01.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:01.546+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.547000+0000) 2020-02-05T23:24:01.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:01.547+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.547889+0000) 2020-02-05T23:24:01.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:01.553+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:01.554626+0000) 2020-02-05T23:24:01.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:01.559+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:02.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:02.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:02.414915+0000) 2020-02-05T23:24:02.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:02.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:02.766069+0000) 2020-02-05T23:24:03.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:03.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:03.053384+0000) 2020-02-05T23:24:03.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:03.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:03.927610+0000) 2020-02-05T23:24:03.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:03.926+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:03.927733+0000) 2020-02-05T23:24:06.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:06.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.028022+0000) 2020-02-05T23:24:06.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:06.565+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.567056+0000) 2020-02-05T23:24:06.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:06.567+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.567523+0000) 2020-02-05T23:24:06.569 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:06.567+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.568164+0000) 2020-02-05T23:24:06.569 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:06.568+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.568495+0000) 2020-02-05T23:24:06.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:06.568+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.569290+0000) 2020-02-05T23:24:06.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:06.569+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.569516+0000) 2020-02-05T23:24:06.571 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:06.568+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.570049+0000) 2020-02-05T23:24:06.571 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:06.569+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.570321+0000) 2020-02-05T23:24:06.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:06.571+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.571422+0000) 2020-02-05T23:24:06.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:06.570+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.571422+0000) 2020-02-05T23:24:06.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:06.570+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.571650+0000) 2020-02-05T23:24:06.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:06.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.572214+0000) 2020-02-05T23:24:06.574 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:06.571+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.572365+0000) 2020-02-05T23:24:06.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:06.575+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:06.576234+0000) 2020-02-05T23:24:06.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:06.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:07.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:07.160+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:07.161028+0000) 2020-02-05T23:24:07.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:07.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:07.167156+0000) 2020-02-05T23:24:07.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:07.167+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:07.167349+0000) 2020-02-05T23:24:07.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:07.167+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:07.167487+0000) 2020-02-05T23:24:09.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:09.552+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:09.553117+0000) 2020-02-05T23:24:09.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:09.844+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:09.845340+0000) 2020-02-05T23:24:10.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:10.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:10.160126+0000) 2020-02-05T23:24:10.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:10.397+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:10.398751+0000) 2020-02-05T23:24:11.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:11.587+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.588501+0000) 2020-02-05T23:24:11.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:11.587+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.588862+0000) 2020-02-05T23:24:11.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:11.587+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.589018+0000) 2020-02-05T23:24:11.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:11.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.589505+0000) 2020-02-05T23:24:11.591 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:11.588+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.589730+0000) 2020-02-05T23:24:11.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:11.588+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.589862+0000) 2020-02-05T23:24:11.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:11.588+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.590146+0000) 2020-02-05T23:24:11.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:11.589+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.590503+0000) 2020-02-05T23:24:11.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:11.590+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.591475+0000) 2020-02-05T23:24:11.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:11.590+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.591421+0000) 2020-02-05T23:24:11.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:11.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.592331+0000) 2020-02-05T23:24:11.593 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:11.591+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.592681+0000) 2020-02-05T23:24:11.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:11.591+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.592922+0000) 2020-02-05T23:24:11.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:11.599+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:11.601059+0000) 2020-02-05T23:24:11.604 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:11.602+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143551 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:12.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:12.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:12.415132+0000) 2020-02-05T23:24:12.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:12.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:12.766326+0000) 2020-02-05T23:24:13.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:13.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:13.053658+0000) 2020-02-05T23:24:13.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:13.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:13.927849+0000) 2020-02-05T23:24:13.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:13.927+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:13.927998+0000) 2020-02-05T23:24:16.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:16.026+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.028267+0000) 2020-02-05T23:24:16.610 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:16.607+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.608897+0000) 2020-02-05T23:24:16.610 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:16.608+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.609862+0000) 2020-02-05T23:24:16.611 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:16.609+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.610189+0000) 2020-02-05T23:24:16.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:16.609+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.610572+0000) 2020-02-05T23:24:16.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:16.609+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.610844+0000) 2020-02-05T23:24:16.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:16.610+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.611488+0000) 2020-02-05T23:24:16.612 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:16.610+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.611895+0000) 2020-02-05T23:24:16.613 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:16.611+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.612113+0000) 2020-02-05T23:24:16.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:16.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.612672+0000) 2020-02-05T23:24:16.615 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:16.612+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.613883+0000) 2020-02-05T23:24:16.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:16.613+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.614540+0000) 2020-02-05T23:24:16.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:16.613+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.615027+0000) 2020-02-05T23:24:16.616 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:16.613+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.615357+0000) 2020-02-05T23:24:16.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:16.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:16.619880+0000) 2020-02-05T23:24:16.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:16.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143591 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:17.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:17.160+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:17.161281+0000) 2020-02-05T23:24:17.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:17.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:17.167372+0000) 2020-02-05T23:24:17.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:17.166+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:17.167535+0000) 2020-02-05T23:24:17.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:17.166+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:17.167734+0000) 2020-02-05T23:24:19.435 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:24:19.442 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:24:19.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:19.552+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:19.553371+0000) 2020-02-05T23:24:19.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:19.853+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:19.853793+0000) 2020-02-05T23:24:20.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:20.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:20.160361+0000) 2020-02-05T23:24:20.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:20.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:20.399011+0000) 2020-02-05T23:24:21.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:21.629+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.630803+0000) 2020-02-05T23:24:21.632 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:21.630+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.631526+0000) 2020-02-05T23:24:21.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:21.631+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.632244+0000) 2020-02-05T23:24:21.633 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:21.631+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.632593+0000) 2020-02-05T23:24:21.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:21.632+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.633422+0000) 2020-02-05T23:24:21.634 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:21.632+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.633871+0000) 2020-02-05T23:24:21.635 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:21.633+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.634376+0000) 2020-02-05T23:24:21.636 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:21.633+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.635250+0000) 2020-02-05T23:24:21.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:21.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.635564+0000) 2020-02-05T23:24:21.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:21.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.635843+0000) 2020-02-05T23:24:21.637 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:21.634+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.636228+0000) 2020-02-05T23:24:21.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:21.635+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.637227+0000) 2020-02-05T23:24:21.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:21.636+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.637502+0000) 2020-02-05T23:24:21.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:21.642+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:21.643239+0000) 2020-02-05T23:24:21.648 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:21.646+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:22.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:22.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:22.415347+0000) 2020-02-05T23:24:22.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:22.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:22.766569+0000) 2020-02-05T23:24:23.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:23.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:23.053888+0000) 2020-02-05T23:24:23.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:23.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:23.928145+0000) 2020-02-05T23:24:23.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:23.927+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:23.928305+0000) 2020-02-05T23:24:26.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:26.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.028522+0000) 2020-02-05T23:24:26.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:26.653+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.654357+0000) 2020-02-05T23:24:26.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:26.652+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.654151+0000) 2020-02-05T23:24:26.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:26.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.654872+0000) 2020-02-05T23:24:26.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:26.654+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.655332+0000) 2020-02-05T23:24:26.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:26.654+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.655675+0000) 2020-02-05T23:24:26.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:26.653+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.654825+0000) 2020-02-05T23:24:26.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:26.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.656149+0000) 2020-02-05T23:24:26.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:26.655+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.657130+0000) 2020-02-05T23:24:26.658 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:26.656+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.657555+0000) 2020-02-05T23:24:26.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:26.656+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.657936+0000) 2020-02-05T23:24:26.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:26.657+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.658344+0000) 2020-02-05T23:24:26.659 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:26.657+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.658522+0000) 2020-02-05T23:24:26.659 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:26.657+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.658719+0000) 2020-02-05T23:24:26.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:26.660+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:26.661827+0000) 2020-02-05T23:24:26.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:26.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:27.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:27.161+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:27.161491+0000) 2020-02-05T23:24:27.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:27.166+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:27.167606+0000) 2020-02-05T23:24:27.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:27.166+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:27.167737+0000) 2020-02-05T23:24:27.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:27.166+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:27.167940+0000) 2020-02-05T23:24:29.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:29.553+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:29.553628+0000) 2020-02-05T23:24:29.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:29.861+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:29.861918+0000) 2020-02-05T23:24:30.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:30.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:30.160594+0000) 2020-02-05T23:24:30.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:30.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:30.399266+0000) 2020-02-05T23:24:31.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:31.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.675325+0000) 2020-02-05T23:24:31.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:31.674+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.675840+0000) 2020-02-05T23:24:31.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:31.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.676379+0000) 2020-02-05T23:24:31.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:31.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.676787+0000) 2020-02-05T23:24:31.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:31.675+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.677378+0000) 2020-02-05T23:24:31.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:31.677+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.678279+0000) 2020-02-05T23:24:31.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:31.677+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.678829+0000) 2020-02-05T23:24:31.679 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:31.677+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.678746+0000) 2020-02-05T23:24:31.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:31.677+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.679217+0000) 2020-02-05T23:24:31.680 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:31.678+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.679697+0000) 2020-02-05T23:24:31.681 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:31.679+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.680247+0000) 2020-02-05T23:24:31.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:31.679+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.680605+0000) 2020-02-05T23:24:31.681 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:31.679+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.681000+0000) 2020-02-05T23:24:31.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:31.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:31.685618+0000) 2020-02-05T23:24:31.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:31.691+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:32.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:32.414+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:32.415666+0000) 2020-02-05T23:24:32.767 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:32.765+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:32.766769+0000) 2020-02-05T23:24:33.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:33.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:33.054124+0000) 2020-02-05T23:24:33.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:33.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:33.928430+0000) 2020-02-05T23:24:33.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:33.928+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:33.928537+0000) 2020-02-05T23:24:36.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:36.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.028702+0000) 2020-02-05T23:24:36.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:36.698+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.699066+0000) 2020-02-05T23:24:36.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:36.698+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.699715+0000) 2020-02-05T23:24:36.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:36.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.700730+0000) 2020-02-05T23:24:36.701 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:36.699+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.700987+0000) 2020-02-05T23:24:36.702 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:36.700+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.701565+0000) 2020-02-05T23:24:36.703 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:36.700+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.702182+0000) 2020-02-05T23:24:36.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:36.702+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.702416+0000) 2020-02-05T23:24:36.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:36.701+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.702694+0000) 2020-02-05T23:24:36.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:36.702+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.703025+0000) 2020-02-05T23:24:36.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:36.702+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.703334+0000) 2020-02-05T23:24:36.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:36.702+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.704037+0000) 2020-02-05T23:24:36.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:36.703+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.704414+0000) 2020-02-05T23:24:36.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:36.703+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.705006+0000) 2020-02-05T23:24:36.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:36.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:36.708492+0000) 2020-02-05T23:24:36.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:36.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:37.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:37.161+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:37.161691+0000) 2020-02-05T23:24:37.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:37.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:37.167840+0000) 2020-02-05T23:24:37.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:37.167+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:37.168014+0000) 2020-02-05T23:24:37.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:37.167+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:37.168221+0000) 2020-02-05T23:24:39.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:39.553+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:39.553860+0000) 2020-02-05T23:24:39.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:39.869+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:39.869810+0000) 2020-02-05T23:24:40.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:40.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:40.160833+0000) 2020-02-05T23:24:40.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:40.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:40.399502+0000) 2020-02-05T23:24:41.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:41.721+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.721102+0000) 2020-02-05T23:24:41.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:41.721+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.721733+0000) 2020-02-05T23:24:41.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:41.721+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.722001+0000) 2020-02-05T23:24:41.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:41.721+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.722550+0000) 2020-02-05T23:24:41.723 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:41.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.722690+0000) 2020-02-05T23:24:41.724 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:41.721+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.723191+0000) 2020-02-05T23:24:41.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:41.723+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.724862+0000) 2020-02-05T23:24:41.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:41.723+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.725141+0000) 2020-02-05T23:24:41.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:41.723+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.725283+0000) 2020-02-05T23:24:41.728 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:41.724+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.725794+0000) 2020-02-05T23:24:41.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:41.724+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.724406+0000) 2020-02-05T23:24:41.728 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:41.724+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.724651+0000) 2020-02-05T23:24:41.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:41.731+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.731594+0000) 2020-02-05T23:24:41.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:41.731+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:41.731872+0000) 2020-02-05T23:24:41.737 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:41.735+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:42.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:42.415+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:42.415919+0000) 2020-02-05T23:24:42.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:42.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:42.767067+0000) 2020-02-05T23:24:43.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:43.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:43.054346+0000) 2020-02-05T23:24:43.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:43.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:43.928716+0000) 2020-02-05T23:24:43.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:43.927+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:43.928769+0000) 2020-02-05T23:24:46.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:46.027+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.028929+0000) 2020-02-05T23:24:46.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:46.739+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.741255+0000) 2020-02-05T23:24:46.743 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:46.741+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.742472+0000) 2020-02-05T23:24:46.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:46.742+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.743911+0000) 2020-02-05T23:24:46.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:46.743+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.744475+0000) 2020-02-05T23:24:46.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:46.743+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.744872+0000) 2020-02-05T23:24:46.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:46.744+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.745159+0000) 2020-02-05T23:24:46.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:46.743+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.745379+0000) 2020-02-05T23:24:46.747 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:46.744+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.745698+0000) 2020-02-05T23:24:46.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:46.745+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.746214+0000) 2020-02-05T23:24:46.747 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:46.745+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.746695+0000) 2020-02-05T23:24:46.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:46.744+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.746320+0000) 2020-02-05T23:24:46.748 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:46.745+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.747383+0000) 2020-02-05T23:24:46.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:46.746+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.747774+0000) 2020-02-05T23:24:46.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:46.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:46.751533+0000) 2020-02-05T23:24:46.761 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:46.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143840 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:47.163 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:47.160+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:47.161884+0000) 2020-02-05T23:24:47.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:47.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:47.168063+0000) 2020-02-05T23:24:47.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:47.167+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:47.168247+0000) 2020-02-05T23:24:47.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:47.167+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:47.168489+0000) 2020-02-05T23:24:49.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:49.553+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:49.554146+0000) 2020-02-05T23:24:49.572 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:24:49.578 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:24:49.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:49.877+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:49.878032+0000) 2020-02-05T23:24:50.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:50.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:50.161067+0000) 2020-02-05T23:24:50.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:50.398+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:50.399729+0000) 2020-02-05T23:24:51.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:51.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.764580+0000) 2020-02-05T23:24:51.766 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:51.764+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.765216+0000) 2020-02-05T23:24:51.767 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:51.764+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.766353+0000) 2020-02-05T23:24:51.769 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:51.766+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.767821+0000) 2020-02-05T23:24:51.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:51.767+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.768495+0000) 2020-02-05T23:24:51.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:51.767+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.768402+0000) 2020-02-05T23:24:51.771 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:51.767+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.768951+0000) 2020-02-05T23:24:51.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:51.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.769680+0000) 2020-02-05T23:24:51.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:51.768+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.769679+0000) 2020-02-05T23:24:51.771 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:51.768+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.770355+0000) 2020-02-05T23:24:51.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:51.769+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.770964+0000) 2020-02-05T23:24:51.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:51.770+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.771452+0000) 2020-02-05T23:24:51.772 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:51.769+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.771149+0000) 2020-02-05T23:24:51.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:51.771+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:51.773059+0000) 2020-02-05T23:24:51.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:51.781+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:52.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:52.415+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:52.416146+0000) 2020-02-05T23:24:52.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:52.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:52.767347+0000) 2020-02-05T23:24:53.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:53.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:53.054574+0000) 2020-02-05T23:24:53.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:53.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:53.928950+0000) 2020-02-05T23:24:53.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:53.928+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:53.929019+0000) 2020-02-05T23:24:56.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:56.028+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.029213+0000) 2020-02-05T23:24:56.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:24:56.786+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.787533+0000) 2020-02-05T23:24:56.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:56.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.788227+0000) 2020-02-05T23:24:56.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:56.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.789144+0000) 2020-02-05T23:24:56.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:56.788+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.789508+0000) 2020-02-05T23:24:56.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:56.788+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.790037+0000) 2020-02-05T23:24:56.791 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:24:56.789+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.790572+0000) 2020-02-05T23:24:56.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:24:56.790+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.791721+0000) 2020-02-05T23:24:56.793 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:56.790+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.791905+0000) 2020-02-05T23:24:56.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:24:56.791+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.792404+0000) 2020-02-05T23:24:56.793 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:24:56.791+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.792654+0000) 2020-02-05T23:24:56.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:24:56.792+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.793470+0000) 2020-02-05T23:24:56.794 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:56.791+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.793081+0000) 2020-02-05T23:24:56.794 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:24:56.791+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.793108+0000) 2020-02-05T23:24:56.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:56.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:56.797851+0000) 2020-02-05T23:24:56.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:24:56.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:24:57.163 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:24:57.161+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:57.162129+0000) 2020-02-05T23:24:57.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:24:57.167+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:57.168283+0000) 2020-02-05T23:24:57.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:24:57.167+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:57.168498+0000) 2020-02-05T23:24:57.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:24:57.167+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:57.168674+0000) 2020-02-05T23:24:59.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:59.553+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:59.554394+0000) 2020-02-05T23:24:59.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:24:59.886+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:24:59.887049+0000) 2020-02-05T23:25:00.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:00.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:00.161348+0000) 2020-02-05T23:25:00.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:00.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:00.399971+0000) 2020-02-05T23:25:01.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:01.810+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.811523+0000) 2020-02-05T23:25:01.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:01.810+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.811993+0000) 2020-02-05T23:25:01.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:01.811+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.812425+0000) 2020-02-05T23:25:01.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:01.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.812839+0000) 2020-02-05T23:25:01.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:01.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.813326+0000) 2020-02-05T23:25:01.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:01.812+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.813519+0000) 2020-02-05T23:25:01.815 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:01.813+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.813638+0000) 2020-02-05T23:25:01.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:01.813+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.814051+0000) 2020-02-05T23:25:01.815 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:01.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.814283+0000) 2020-02-05T23:25:01.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:01.814+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.814920+0000) 2020-02-05T23:25:01.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:01.814+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.815525+0000) 2020-02-05T23:25:01.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:01.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.815853+0000) 2020-02-05T23:25:01.817 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:01.816+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.816823+0000) 2020-02-05T23:25:01.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:01.819+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:01.820850+0000) 2020-02-05T23:25:01.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:01.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 143959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:02.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:02.415+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:02.416332+0000) 2020-02-05T23:25:02.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:02.766+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:02.767650+0000) 2020-02-05T23:25:03.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:03.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:03.054806+0000) 2020-02-05T23:25:03.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:03.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:03.929268+0000) 2020-02-05T23:25:03.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:03.928+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:03.929333+0000) 2020-02-05T23:25:06.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:06.029+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.029499+0000) 2020-02-05T23:25:06.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:06.831+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.832253+0000) 2020-02-05T23:25:06.835 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:06.833+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.834096+0000) 2020-02-05T23:25:06.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:06.833+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.834419+0000) 2020-02-05T23:25:06.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:06.834+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.835066+0000) 2020-02-05T23:25:06.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:06.834+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.835635+0000) 2020-02-05T23:25:06.837 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:06.835+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.836208+0000) 2020-02-05T23:25:06.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:06.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.836173+0000) 2020-02-05T23:25:06.838 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:06.835+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.836696+0000) 2020-02-05T23:25:06.838 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:06.836+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.837344+0000) 2020-02-05T23:25:06.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:06.836+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.837620+0000) 2020-02-05T23:25:06.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:06.837+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.838001+0000) 2020-02-05T23:25:06.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:06.838+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.838614+0000) 2020-02-05T23:25:06.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:06.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.838501+0000) 2020-02-05T23:25:06.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:06.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:06.843908+0000) 2020-02-05T23:25:06.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:06.847+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144000 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:07.163 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:07.161+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:07.162336+0000) 2020-02-05T23:25:07.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:07.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:07.168511+0000) 2020-02-05T23:25:07.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:07.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:07.168711+0000) 2020-02-05T23:25:07.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:07.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:07.168886+0000) 2020-02-05T23:25:09.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:09.553+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:09.554615+0000) 2020-02-05T23:25:09.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:09.894+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:09.895480+0000) 2020-02-05T23:25:10.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:10.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:10.161616+0000) 2020-02-05T23:25:10.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:10.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:10.400261+0000) 2020-02-05T23:25:11.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:11.852+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.853926+0000) 2020-02-05T23:25:11.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:11.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.854506+0000) 2020-02-05T23:25:11.856 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:11.854+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.854915+0000) 2020-02-05T23:25:11.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:11.854+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.855208+0000) 2020-02-05T23:25:11.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:11.855+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.856223+0000) 2020-02-05T23:25:11.858 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:11.855+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.855942+0000) 2020-02-05T23:25:11.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:11.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.858483+0000) 2020-02-05T23:25:11.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:11.857+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.858723+0000) 2020-02-05T23:25:11.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:11.858+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.858722+0000) 2020-02-05T23:25:11.862 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:11.858+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.859054+0000) 2020-02-05T23:25:11.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:11.859+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.860000+0000) 2020-02-05T23:25:11.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:11.858+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.859338+0000) 2020-02-05T23:25:11.863 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:11.858+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.859579+0000) 2020-02-05T23:25:11.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:11.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:11.862512+0000) 2020-02-05T23:25:11.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:11.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:12.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:12.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:12.416553+0000) 2020-02-05T23:25:12.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:12.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:12.767862+0000) 2020-02-05T23:25:13.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:13.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:13.055058+0000) 2020-02-05T23:25:13.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:13.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:13.929542+0000) 2020-02-05T23:25:13.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:13.929+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:13.929596+0000) 2020-02-05T23:25:16.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:16.029+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.029724+0000) 2020-02-05T23:25:16.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:16.878+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.878703+0000) 2020-02-05T23:25:16.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:16.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.880317+0000) 2020-02-05T23:25:16.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:16.880+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.880462+0000) 2020-02-05T23:25:16.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:16.879+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.880927+0000) 2020-02-05T23:25:16.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:16.880+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.881494+0000) 2020-02-05T23:25:16.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:16.881+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.882200+0000) 2020-02-05T23:25:16.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:16.881+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.882731+0000) 2020-02-05T23:25:16.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:16.882+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.883291+0000) 2020-02-05T23:25:16.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:16.882+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.883588+0000) 2020-02-05T23:25:16.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:16.883+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.883985+0000) 2020-02-05T23:25:16.885 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:16.884+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.884472+0000) 2020-02-05T23:25:16.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:16.884+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.884935+0000) 2020-02-05T23:25:16.886 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:16.884+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.885211+0000) 2020-02-05T23:25:16.886 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:16.884+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:16.885725+0000) 2020-02-05T23:25:16.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:16.894+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:17.163 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:17.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:17.162550+0000) 2020-02-05T23:25:17.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:17.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:17.168739+0000) 2020-02-05T23:25:17.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:17.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:17.168921+0000) 2020-02-05T23:25:17.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:17.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:17.169072+0000) 2020-02-05T23:25:19.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:19.554+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:19.554856+0000) 2020-02-05T23:25:19.710 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:25:19.716 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:25:19.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:19.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:19.903603+0000) 2020-02-05T23:25:20.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:20.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:20.161845+0000) 2020-02-05T23:25:20.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:20.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:20.400499+0000) 2020-02-05T23:25:21.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:21.899+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.900690+0000) 2020-02-05T23:25:21.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:21.901+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.901513+0000) 2020-02-05T23:25:21.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:21.900+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.901957+0000) 2020-02-05T23:25:21.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:21.901+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.902306+0000) 2020-02-05T23:25:21.904 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:21.902+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.902567+0000) 2020-02-05T23:25:21.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:21.901+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.902303+0000) 2020-02-05T23:25:21.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:21.903+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.903614+0000) 2020-02-05T23:25:21.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:21.901+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.903030+0000) 2020-02-05T23:25:21.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:21.902+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.903232+0000) 2020-02-05T23:25:21.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:21.902+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.903481+0000) 2020-02-05T23:25:21.906 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:21.903+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.904136+0000) 2020-02-05T23:25:21.906 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:21.904+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.905325+0000) 2020-02-05T23:25:21.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:21.903+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.904527+0000) 2020-02-05T23:25:21.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:21.908+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:21.909437+0000) 2020-02-05T23:25:21.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:21.915+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:22.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:22.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:22.416775+0000) 2020-02-05T23:25:22.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:22.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:22.768118+0000) 2020-02-05T23:25:23.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:23.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:23.055318+0000) 2020-02-05T23:25:23.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:23.929+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:23.929834+0000) 2020-02-05T23:25:23.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:23.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:23.929817+0000) 2020-02-05T23:25:26.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:26.029+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.029991+0000) 2020-02-05T23:25:26.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:26.920+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.921540+0000) 2020-02-05T23:25:26.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:26.922+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.922643+0000) 2020-02-05T23:25:26.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:26.922+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.922680+0000) 2020-02-05T23:25:26.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:26.922+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.923140+0000) 2020-02-05T23:25:26.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:26.923+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.923883+0000) 2020-02-05T23:25:26.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:26.923+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.924104+0000) 2020-02-05T23:25:26.926 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:26.923+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.924930+0000) 2020-02-05T23:25:26.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:26.924+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.925457+0000) 2020-02-05T23:25:26.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:26.924+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.925721+0000) 2020-02-05T23:25:26.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:26.924+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.925339+0000) 2020-02-05T23:25:26.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:26.926+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.926683+0000) 2020-02-05T23:25:26.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:26.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.926234+0000) 2020-02-05T23:25:26.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:26.925+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.926398+0000) 2020-02-05T23:25:26.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:26.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:26.930519+0000) 2020-02-05T23:25:26.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:26.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:27.163 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:27.161+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:27.162763+0000) 2020-02-05T23:25:27.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:27.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:27.168968+0000) 2020-02-05T23:25:27.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:27.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:27.169127+0000) 2020-02-05T23:25:27.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:27.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:27.169254+0000) 2020-02-05T23:25:29.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:29.554+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:29.555142+0000) 2020-02-05T23:25:29.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:29.910+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:29.911072+0000) 2020-02-05T23:25:30.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:30.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:30.162127+0000) 2020-02-05T23:25:30.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:30.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:30.400754+0000) 2020-02-05T23:25:31.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:31.943+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.944886+0000) 2020-02-05T23:25:31.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:31.945+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.946379+0000) 2020-02-05T23:25:31.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:31.946+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.947151+0000) 2020-02-05T23:25:31.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:31.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.947808+0000) 2020-02-05T23:25:31.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:31.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.948258+0000) 2020-02-05T23:25:31.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:31.947+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.948386+0000) 2020-02-05T23:25:31.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:31.947+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.948798+0000) 2020-02-05T23:25:31.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:31.948+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.949178+0000) 2020-02-05T23:25:31.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:31.949+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.949975+0000) 2020-02-05T23:25:31.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:31.950+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.950946+0000) 2020-02-05T23:25:31.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:31.950+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.950952+0000) 2020-02-05T23:25:31.952 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:31.950+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.951286+0000) 2020-02-05T23:25:31.952 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:31.949+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.950323+0000) 2020-02-05T23:25:31.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:31.953+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:31.954698+0000) 2020-02-05T23:25:31.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:31.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:32.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:32.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:32.417016+0000) 2020-02-05T23:25:32.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:32.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:32.768333+0000) 2020-02-05T23:25:33.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:33.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:33.055558+0000) 2020-02-05T23:25:33.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:33.929+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:33.930071+0000) 2020-02-05T23:25:33.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:33.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:33.930110+0000) 2020-02-05T23:25:36.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:36.030+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.030305+0000) 2020-02-05T23:25:36.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:36.968+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.968472+0000) 2020-02-05T23:25:36.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:36.967+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.967593+0000) 2020-02-05T23:25:36.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:36.969+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.969373+0000) 2020-02-05T23:25:36.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:36.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.970735+0000) 2020-02-05T23:25:36.972 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:36.971+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.971162+0000) 2020-02-05T23:25:36.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:36.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.971679+0000) 2020-02-05T23:25:36.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:36.972+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.972294+0000) 2020-02-05T23:25:36.973 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:36.971+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.972087+0000) 2020-02-05T23:25:36.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:36.972+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.972625+0000) 2020-02-05T23:25:36.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:36.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.973737+0000) 2020-02-05T23:25:36.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:36.973+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.974252+0000) 2020-02-05T23:25:36.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:36.974+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.974693+0000) 2020-02-05T23:25:36.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:36.974+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.975219+0000) 2020-02-05T23:25:36.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:36.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:36.978638+0000) 2020-02-05T23:25:36.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:36.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:37.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:37.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:37.163001+0000) 2020-02-05T23:25:37.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:37.169+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:37.169179+0000) 2020-02-05T23:25:37.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:37.169+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:37.169352+0000) 2020-02-05T23:25:37.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:37.169+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:37.169509+0000) 2020-02-05T23:25:39.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:39.554+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:39.555383+0000) 2020-02-05T23:25:39.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:39.918+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:39.919349+0000) 2020-02-05T23:25:40.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:40.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:40.162385+0000) 2020-02-05T23:25:40.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:40.399+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:40.401005+0000) 2020-02-05T23:25:41.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:41.989+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.990409+0000) 2020-02-05T23:25:41.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:41.990+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.991640+0000) 2020-02-05T23:25:41.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:41.990+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.991891+0000) 2020-02-05T23:25:41.993 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:41.991+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.992477+0000) 2020-02-05T23:25:41.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:41.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.993027+0000) 2020-02-05T23:25:41.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:41.992+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.993516+0000) 2020-02-05T23:25:41.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:41.993+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.994361+0000) 2020-02-05T23:25:41.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:41.993+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.994507+0000) 2020-02-05T23:25:41.996 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:41.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.995622+0000) 2020-02-05T23:25:41.997 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:41.994+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.995879+0000) 2020-02-05T23:25:41.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:41.994+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.996101+0000) 2020-02-05T23:25:41.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:41.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.995953+0000) 2020-02-05T23:25:41.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:41.994+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:41.996372+0000) 2020-02-05T23:25:42.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:41.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:42.000017+0000) 2020-02-05T23:25:42.008 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:42.005+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:42.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:42.415+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:42.417248+0000) 2020-02-05T23:25:42.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:42.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:42.768554+0000) 2020-02-05T23:25:43.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:43.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:43.055791+0000) 2020-02-05T23:25:43.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:43.929+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:43.930356+0000) 2020-02-05T23:25:43.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:43.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:43.930356+0000) 2020-02-05T23:25:46.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:46.029+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:46.030557+0000) 2020-02-05T23:25:47.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:47.010+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.011559+0000) 2020-02-05T23:25:47.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:47.011+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.012602+0000) 2020-02-05T23:25:47.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:47.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.013361+0000) 2020-02-05T23:25:47.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:47.012+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.013995+0000) 2020-02-05T23:25:47.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:47.013+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.014414+0000) 2020-02-05T23:25:47.016 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:47.013+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.015301+0000) 2020-02-05T23:25:47.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:47.014+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.015617+0000) 2020-02-05T23:25:47.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:47.015+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.016861+0000) 2020-02-05T23:25:47.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:47.016+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.017212+0000) 2020-02-05T23:25:47.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:47.016+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.017845+0000) 2020-02-05T23:25:47.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:47.016+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.018252+0000) 2020-02-05T23:25:47.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:47.016+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.018396+0000) 2020-02-05T23:25:47.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:47.017+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.019011+0000) 2020-02-05T23:25:47.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:47.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.022396+0000) 2020-02-05T23:25:47.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:47.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:47.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:47.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.163208+0000) 2020-02-05T23:25:47.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:47.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.169414+0000) 2020-02-05T23:25:47.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:47.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.169629+0000) 2020-02-05T23:25:47.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:47.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:47.169742+0000) 2020-02-05T23:25:49.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:49.554+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:49.555623+0000) 2020-02-05T23:25:49.890 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:25:49.896 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:25:49.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:49.927+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:49.928972+0000) 2020-02-05T23:25:50.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:50.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:50.162640+0000) 2020-02-05T23:25:50.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:50.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:50.401261+0000) 2020-02-05T23:25:52.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:52.034+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.035622+0000) 2020-02-05T23:25:52.037 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:52.035+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.036171+0000) 2020-02-05T23:25:52.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:52.035+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.036732+0000) 2020-02-05T23:25:52.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:52.035+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.037021+0000) 2020-02-05T23:25:52.038 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:52.036+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.037616+0000) 2020-02-05T23:25:52.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:52.036+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.037769+0000) 2020-02-05T23:25:52.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:52.036+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.038044+0000) 2020-02-05T23:25:52.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:52.038+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.039143+0000) 2020-02-05T23:25:52.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:52.038+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.039976+0000) 2020-02-05T23:25:52.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:52.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.040442+0000) 2020-02-05T23:25:52.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:52.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.040782+0000) 2020-02-05T23:25:52.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:52.039+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.040928+0000) 2020-02-05T23:25:52.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:52.039+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.041274+0000) 2020-02-05T23:25:52.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:52.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.043132+0000) 2020-02-05T23:25:52.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:52.051+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144362 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:52.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:52.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.417492+0000) 2020-02-05T23:25:52.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:52.767+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:52.768765+0000) 2020-02-05T23:25:53.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:53.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:53.056047+0000) 2020-02-05T23:25:53.931 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:53.929+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:53.930657+0000) 2020-02-05T23:25:53.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:53.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:53.930657+0000) 2020-02-05T23:25:56.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:56.029+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:56.030810+0000) 2020-02-05T23:25:57.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:25:57.058+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.059954+0000) 2020-02-05T23:25:57.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:57.060+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.061666+0000) 2020-02-05T23:25:57.072 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:25:57.060+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.062176+0000) 2020-02-05T23:25:57.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:25:57.062+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.063542+0000) 2020-02-05T23:25:57.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:57.062+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.063744+0000) 2020-02-05T23:25:57.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:57.062+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.064249+0000) 2020-02-05T23:25:57.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:25:57.056+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.057780+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:57.057+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.058530+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:57.062+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.063948+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:57.065+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.066218+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:57.058+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.059114+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:25:57.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.061274+0000) 2020-02-05T23:25:57.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:25:57.061+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.062391+0000) 2020-02-05T23:25:57.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:25:57.061+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.062871+0000) 2020-02-05T23:25:57.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:25:57.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144403 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:25:57.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:25:57.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.163421+0000) 2020-02-05T23:25:57.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:25:57.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.169695+0000) 2020-02-05T23:25:57.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:25:57.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.169862+0000) 2020-02-05T23:25:57.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:25:57.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:57.169947+0000) 2020-02-05T23:25:59.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:59.555+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:59.555848+0000) 2020-02-05T23:25:59.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:25:59.932+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:25:59.932989+0000) 2020-02-05T23:26:00.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:00.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:00.162876+0000) 2020-02-05T23:26:00.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:00.400+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:00.401551+0000) 2020-02-05T23:26:02.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:02.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.080336+0000) 2020-02-05T23:26:02.081 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:02.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.080452+0000) 2020-02-05T23:26:02.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:02.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.081510+0000) 2020-02-05T23:26:02.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:02.081+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.082137+0000) 2020-02-05T23:26:02.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:02.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.082432+0000) 2020-02-05T23:26:02.084 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:02.081+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.083108+0000) 2020-02-05T23:26:02.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:02.082+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.083193+0000) 2020-02-05T23:26:02.084 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:02.082+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.083764+0000) 2020-02-05T23:26:02.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:02.082+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.083926+0000) 2020-02-05T23:26:02.085 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:02.082+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.083900+0000) 2020-02-05T23:26:02.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:02.083+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.084458+0000) 2020-02-05T23:26:02.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:02.083+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.084606+0000) 2020-02-05T23:26:02.086 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:02.084+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.085561+0000) 2020-02-05T23:26:02.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:02.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.092880+0000) 2020-02-05T23:26:02.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:02.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:02.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:02.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.417737+0000) 2020-02-05T23:26:02.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:02.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:02.769028+0000) 2020-02-05T23:26:03.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:03.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:03.056332+0000) 2020-02-05T23:26:03.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:03.930+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:03.930926+0000) 2020-02-05T23:26:03.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:03.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:03.930926+0000) 2020-02-05T23:26:06.032 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:06.030+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:06.031086+0000) 2020-02-05T23:26:07.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:07.101+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.102132+0000) 2020-02-05T23:26:07.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:07.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.102729+0000) 2020-02-05T23:26:07.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:07.101+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.102996+0000) 2020-02-05T23:26:07.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:07.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.103464+0000) 2020-02-05T23:26:07.105 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:07.102+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.103579+0000) 2020-02-05T23:26:07.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:07.102+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.104339+0000) 2020-02-05T23:26:07.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:07.103+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.104608+0000) 2020-02-05T23:26:07.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:07.103+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.104940+0000) 2020-02-05T23:26:07.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:07.103+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.105183+0000) 2020-02-05T23:26:07.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:07.104+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.105441+0000) 2020-02-05T23:26:07.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:07.104+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.105824+0000) 2020-02-05T23:26:07.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:07.104+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.105285+0000) 2020-02-05T23:26:07.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:07.104+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.105931+0000) 2020-02-05T23:26:07.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:07.110+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.111119+0000) 2020-02-05T23:26:07.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:07.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:07.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:07.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.163651+0000) 2020-02-05T23:26:07.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:07.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.169890+0000) 2020-02-05T23:26:07.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:07.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.170038+0000) 2020-02-05T23:26:07.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:07.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:07.170135+0000) 2020-02-05T23:26:09.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:09.555+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:09.556071+0000) 2020-02-05T23:26:09.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:09.940+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:09.940735+0000) 2020-02-05T23:26:10.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:10.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:10.163138+0000) 2020-02-05T23:26:10.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:10.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:10.401807+0000) 2020-02-05T23:26:12.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:12.122+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.123942+0000) 2020-02-05T23:26:12.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:12.123+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.124971+0000) 2020-02-05T23:26:12.126 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:12.124+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.125400+0000) 2020-02-05T23:26:12.127 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:12.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.126051+0000) 2020-02-05T23:26:12.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:12.125+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.126166+0000) 2020-02-05T23:26:12.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:12.125+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.126604+0000) 2020-02-05T23:26:12.128 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:12.125+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.127379+0000) 2020-02-05T23:26:12.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:12.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.127369+0000) 2020-02-05T23:26:12.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:12.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.127937+0000) 2020-02-05T23:26:12.129 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:12.126+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.128035+0000) 2020-02-05T23:26:12.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:12.126+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.128326+0000) 2020-02-05T23:26:12.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:12.127+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.128797+0000) 2020-02-05T23:26:12.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:12.127+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.128928+0000) 2020-02-05T23:26:12.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:12.131+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.132634+0000) 2020-02-05T23:26:12.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:12.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:12.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:12.416+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.417959+0000) 2020-02-05T23:26:12.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:12.768+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:12.769271+0000) 2020-02-05T23:26:13.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:13.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:13.056518+0000) 2020-02-05T23:26:13.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:13.930+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:13.931210+0000) 2020-02-05T23:26:13.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:13.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:13.931210+0000) 2020-02-05T23:26:16.032 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:16.030+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:16.031339+0000) 2020-02-05T23:26:17.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:17.143+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.144710+0000) 2020-02-05T23:26:17.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:17.144+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.145119+0000) 2020-02-05T23:26:17.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:17.144+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.145981+0000) 2020-02-05T23:26:17.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:17.145+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.146595+0000) 2020-02-05T23:26:17.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:17.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.147644+0000) 2020-02-05T23:26:17.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:17.146+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.148051+0000) 2020-02-05T23:26:17.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:17.147+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.148706+0000) 2020-02-05T23:26:17.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:17.147+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.149340+0000) 2020-02-05T23:26:17.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:17.148+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.149863+0000) 2020-02-05T23:26:17.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:17.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.150720+0000) 2020-02-05T23:26:17.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:17.149+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.151029+0000) 2020-02-05T23:26:17.152 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:17.150+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.151158+0000) 2020-02-05T23:26:17.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:17.150+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.151329+0000) 2020-02-05T23:26:17.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:17.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.159492+0000) 2020-02-05T23:26:17.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:17.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:17.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:17.162+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.163916+0000) 2020-02-05T23:26:17.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:17.168+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.170044+0000) 2020-02-05T23:26:17.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:17.168+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.170223+0000) 2020-02-05T23:26:17.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:17.168+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:17.170302+0000) 2020-02-05T23:26:19.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:19.555+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:19.556303+0000) 2020-02-05T23:26:19.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:19.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:19.948424+0000) 2020-02-05T23:26:20.034 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:26:20.040 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:26:20.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:20.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:20.163329+0000) 2020-02-05T23:26:20.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:20.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:20.402067+0000) 2020-02-05T23:26:22.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:22.165+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.166445+0000) 2020-02-05T23:26:22.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:22.166+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.167176+0000) 2020-02-05T23:26:22.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:22.167+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.168903+0000) 2020-02-05T23:26:22.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:22.168+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.169593+0000) 2020-02-05T23:26:22.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:22.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.170550+0000) 2020-02-05T23:26:22.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:22.168+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.169975+0000) 2020-02-05T23:26:22.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:22.169+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.171062+0000) 2020-02-05T23:26:22.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:22.170+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.171297+0000) 2020-02-05T23:26:22.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:22.170+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.171871+0000) 2020-02-05T23:26:22.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:22.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.172806+0000) 2020-02-05T23:26:22.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:22.171+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.172766+0000) 2020-02-05T23:26:22.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:22.172+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.173069+0000) 2020-02-05T23:26:22.174 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:22.172+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.173130+0000) 2020-02-05T23:26:22.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:22.176+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.177805+0000) 2020-02-05T23:26:22.184 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:22.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:22.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:22.417+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.418180+0000) 2020-02-05T23:26:22.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:22.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:22.769508+0000) 2020-02-05T23:26:23.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:23.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:23.056729+0000) 2020-02-05T23:26:23.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:23.931+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:23.931483+0000) 2020-02-05T23:26:23.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:23.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:23.931481+0000) 2020-02-05T23:26:26.032 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:26.031+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:26.031635+0000) 2020-02-05T23:26:27.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:27.163+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.164139+0000) 2020-02-05T23:26:27.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:27.169+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.170261+0000) 2020-02-05T23:26:27.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:27.169+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.170404+0000) 2020-02-05T23:26:27.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:27.169+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.170503+0000) 2020-02-05T23:26:27.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:27.187+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.188557+0000) 2020-02-05T23:26:27.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:27.187+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.189177+0000) 2020-02-05T23:26:27.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:27.189+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.190874+0000) 2020-02-05T23:26:27.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:27.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.192277+0000) 2020-02-05T23:26:27.193 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:27.191+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.192538+0000) 2020-02-05T23:26:27.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:27.192+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.193630+0000) 2020-02-05T23:26:27.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:27.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.193071+0000) 2020-02-05T23:26:27.195 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:27.192+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.193335+0000) 2020-02-05T23:26:27.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:27.192+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.194058+0000) 2020-02-05T23:26:27.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:27.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.194703+0000) 2020-02-05T23:26:27.196 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:27.193+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.195038+0000) 2020-02-05T23:26:27.196 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:27.193+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.195070+0000) 2020-02-05T23:26:27.196 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:27.193+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.195155+0000) 2020-02-05T23:26:27.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:27.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:27.196408+0000) 2020-02-05T23:26:27.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:27.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144643 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:29.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:29.556+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:29.556547+0000) 2020-02-05T23:26:29.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:29.956+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:29.956714+0000) 2020-02-05T23:26:30.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:30.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:30.163581+0000) 2020-02-05T23:26:30.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:30.401+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:30.402320+0000) 2020-02-05T23:26:32.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:32.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.212763+0000) 2020-02-05T23:26:32.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:32.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.213503+0000) 2020-02-05T23:26:32.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:32.213+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.214396+0000) 2020-02-05T23:26:32.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:32.214+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.214790+0000) 2020-02-05T23:26:32.216 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:32.215+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.215286+0000) 2020-02-05T23:26:32.216 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:32.215+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.215509+0000) 2020-02-05T23:26:32.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:32.215+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.215688+0000) 2020-02-05T23:26:32.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:32.215+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.216114+0000) 2020-02-05T23:26:32.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:32.216+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.216580+0000) 2020-02-05T23:26:32.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:32.215+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.216049+0000) 2020-02-05T23:26:32.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:32.216+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.216389+0000) 2020-02-05T23:26:32.219 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:32.217+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.217592+0000) 2020-02-05T23:26:32.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:32.217+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.217212+0000) 2020-02-05T23:26:32.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:32.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.222627+0000) 2020-02-05T23:26:32.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:32.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:32.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:32.417+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.418406+0000) 2020-02-05T23:26:32.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:32.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:32.769775+0000) 2020-02-05T23:26:33.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:33.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:33.056942+0000) 2020-02-05T23:26:33.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:33.930+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:33.931769+0000) 2020-02-05T23:26:33.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:33.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:33.931769+0000) 2020-02-05T23:26:36.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:36.031+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:36.031889+0000) 2020-02-05T23:26:37.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:37.163+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.164372+0000) 2020-02-05T23:26:37.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:37.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.170517+0000) 2020-02-05T23:26:37.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:37.170+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.170682+0000) 2020-02-05T23:26:37.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:37.170+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.170767+0000) 2020-02-05T23:26:37.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:37.230+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.231852+0000) 2020-02-05T23:26:37.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:37.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.234543+0000) 2020-02-05T23:26:37.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:37.234+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.234818+0000) 2020-02-05T23:26:37.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:37.234+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.235432+0000) 2020-02-05T23:26:37.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:37.235+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.235814+0000) 2020-02-05T23:26:37.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:37.234+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.236038+0000) 2020-02-05T23:26:37.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:37.237+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.237514+0000) 2020-02-05T23:26:37.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:37.236+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.237617+0000) 2020-02-05T23:26:37.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:37.237+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.238205+0000) 2020-02-05T23:26:37.239 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:37.237+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.238497+0000) 2020-02-05T23:26:37.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:37.238+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.238902+0000) 2020-02-05T23:26:37.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:37.238+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.239205+0000) 2020-02-05T23:26:37.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:37.238+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.239305+0000) 2020-02-05T23:26:37.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:37.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:37.245018+0000) 2020-02-05T23:26:37.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:37.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:39.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:39.555+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:39.556784+0000) 2020-02-05T23:26:39.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:39.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:39.965034+0000) 2020-02-05T23:26:40.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:40.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:40.163823+0000) 2020-02-05T23:26:40.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:40.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:40.402572+0000) 2020-02-05T23:26:42.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:42.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.254982+0000) 2020-02-05T23:26:42.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:42.256+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.256490+0000) 2020-02-05T23:26:42.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:42.256+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.257730+0000) 2020-02-05T23:26:42.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:42.257+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.257473+0000) 2020-02-05T23:26:42.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:42.257+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.258447+0000) 2020-02-05T23:26:42.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:42.257+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.258703+0000) 2020-02-05T23:26:42.260 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:42.257+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.258948+0000) 2020-02-05T23:26:42.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:42.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.259432+0000) 2020-02-05T23:26:42.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:42.258+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.259607+0000) 2020-02-05T23:26:42.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:42.259+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.259816+0000) 2020-02-05T23:26:42.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:42.259+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.260278+0000) 2020-02-05T23:26:42.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:42.260+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.260928+0000) 2020-02-05T23:26:42.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:42.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.260504+0000) 2020-02-05T23:26:42.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:42.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.268051+0000) 2020-02-05T23:26:42.273 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:42.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:42.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:42.418+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.418738+0000) 2020-02-05T23:26:42.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:42.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:42.770046+0000) 2020-02-05T23:26:43.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:43.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:43.057152+0000) 2020-02-05T23:26:43.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:43.931+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:43.932147+0000) 2020-02-05T23:26:43.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:43.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:43.932118+0000) 2020-02-05T23:26:46.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:46.031+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:46.032109+0000) 2020-02-05T23:26:47.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:47.163+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.164638+0000) 2020-02-05T23:26:47.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:47.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.170776+0000) 2020-02-05T23:26:47.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:47.170+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.170921+0000) 2020-02-05T23:26:47.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:47.170+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.171019+0000) 2020-02-05T23:26:47.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:47.279+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.279768+0000) 2020-02-05T23:26:47.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:47.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.279978+0000) 2020-02-05T23:26:47.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:47.279+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.280517+0000) 2020-02-05T23:26:47.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:47.279+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.280827+0000) 2020-02-05T23:26:47.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:47.281+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.282147+0000) 2020-02-05T23:26:47.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:47.281+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.282597+0000) 2020-02-05T23:26:47.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:47.282+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.283393+0000) 2020-02-05T23:26:47.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:47.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.283482+0000) 2020-02-05T23:26:47.285 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:47.283+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.284311+0000) 2020-02-05T23:26:47.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:47.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.284465+0000) 2020-02-05T23:26:47.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:47.283+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.284266+0000) 2020-02-05T23:26:47.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:47.283+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.284224+0000) 2020-02-05T23:26:47.286 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:47.284+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.284675+0000) 2020-02-05T23:26:47.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:47.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:47.287312+0000) 2020-02-05T23:26:47.296 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:47.294+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:49.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:49.556+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:49.557010+0000) 2020-02-05T23:26:49.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:49.972+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:49.972992+0000) 2020-02-05T23:26:50.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:50.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:50.164049+0000) 2020-02-05T23:26:50.174 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:26:50.179 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:26:50.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:50.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:50.402889+0000) 2020-02-05T23:26:52.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:52.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.302189+0000) 2020-02-05T23:26:52.305 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:52.301+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.302989+0000) 2020-02-05T23:26:52.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:52.302+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.303395+0000) 2020-02-05T23:26:52.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:52.301+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.301499+0000) 2020-02-05T23:26:52.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:52.303+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.303595+0000) 2020-02-05T23:26:52.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:52.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.304550+0000) 2020-02-05T23:26:52.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:52.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.304987+0000) 2020-02-05T23:26:52.307 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:52.304+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.305350+0000) 2020-02-05T23:26:52.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:52.304+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.305587+0000) 2020-02-05T23:26:52.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:52.304+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.304648+0000) 2020-02-05T23:26:52.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:52.305+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.306295+0000) 2020-02-05T23:26:52.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:52.306+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.307099+0000) 2020-02-05T23:26:52.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:52.307+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.307549+0000) 2020-02-05T23:26:52.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:52.310+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.311793+0000) 2020-02-05T23:26:52.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:52.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:52.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:52.418+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.418959+0000) 2020-02-05T23:26:52.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:52.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:52.770310+0000) 2020-02-05T23:26:53.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:53.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:53.057375+0000) 2020-02-05T23:26:53.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:53.931+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:53.932381+0000) 2020-02-05T23:26:53.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:53.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:53.932344+0000) 2020-02-05T23:26:56.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:56.032+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:56.032373+0000) 2020-02-05T23:26:57.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:57.164+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.164875+0000) 2020-02-05T23:26:57.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:57.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.171035+0000) 2020-02-05T23:26:57.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:57.170+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.171136+0000) 2020-02-05T23:26:57.177 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:57.170+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.171302+0000) 2020-02-05T23:26:57.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:26:57.323+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.324305+0000) 2020-02-05T23:26:57.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:26:57.323+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.324980+0000) 2020-02-05T23:26:57.327 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:26:57.325+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.326609+0000) 2020-02-05T23:26:57.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:26:57.326+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.327232+0000) 2020-02-05T23:26:57.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:26:57.327+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.327773+0000) 2020-02-05T23:26:57.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:57.326+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.328051+0000) 2020-02-05T23:26:57.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:26:57.328+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.328531+0000) 2020-02-05T23:26:57.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:26:57.328+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.328688+0000) 2020-02-05T23:26:57.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:26:57.328+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.329319+0000) 2020-02-05T23:26:57.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:57.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.329950+0000) 2020-02-05T23:26:57.331 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:26:57.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.330135+0000) 2020-02-05T23:26:57.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:26:57.329+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.330395+0000) 2020-02-05T23:26:57.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:26:57.329+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.330411+0000) 2020-02-05T23:26:57.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:57.332+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:57.333939+0000) 2020-02-05T23:26:57.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:26:57.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144882 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:26:59.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:59.556+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:59.557273+0000) 2020-02-05T23:26:59.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:26:59.979+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:26:59.980262+0000) 2020-02-05T23:27:00.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:00.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:00.164265+0000) 2020-02-05T23:27:00.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:00.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:00.403140+0000) 2020-02-05T23:27:02.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:02.346+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.346559+0000) 2020-02-05T23:27:02.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:02.346+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.347555+0000) 2020-02-05T23:27:02.350 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:02.346+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.347907+0000) 2020-02-05T23:27:02.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:02.347+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.348327+0000) 2020-02-05T23:27:02.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:02.349+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.349750+0000) 2020-02-05T23:27:02.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:02.348+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.349188+0000) 2020-02-05T23:27:02.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:02.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.349461+0000) 2020-02-05T23:27:02.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:02.349+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.350214+0000) 2020-02-05T23:27:02.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:02.350+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.351083+0000) 2020-02-05T23:27:02.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:02.350+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.350705+0000) 2020-02-05T23:27:02.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:02.351+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.352219+0000) 2020-02-05T23:27:02.354 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:02.352+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.352609+0000) 2020-02-05T23:27:02.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:02.350+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.351361+0000) 2020-02-05T23:27:02.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:02.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.357718+0000) 2020-02-05T23:27:02.366 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:02.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144922 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:02.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:02.419+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.419187+0000) 2020-02-05T23:27:02.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:02.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:02.770531+0000) 2020-02-05T23:27:03.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:03.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:03.057575+0000) 2020-02-05T23:27:03.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:03.931+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:03.932624+0000) 2020-02-05T23:27:03.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:03.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:03.932602+0000) 2020-02-05T23:27:06.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:06.032+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:06.032683+0000) 2020-02-05T23:27:07.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:07.164+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.165137+0000) 2020-02-05T23:27:07.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:07.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.171289+0000) 2020-02-05T23:27:07.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:07.171+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.171389+0000) 2020-02-05T23:27:07.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:07.171+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.171502+0000) 2020-02-05T23:27:07.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:07.369+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.369930+0000) 2020-02-05T23:27:07.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:07.370+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.371291+0000) 2020-02-05T23:27:07.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:07.370+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.371986+0000) 2020-02-05T23:27:07.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:07.372+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.372748+0000) 2020-02-05T23:27:07.374 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:07.372+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.373355+0000) 2020-02-05T23:27:07.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:07.372+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.373275+0000) 2020-02-05T23:27:07.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:07.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.374176+0000) 2020-02-05T23:27:07.375 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:07.373+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.374176+0000) 2020-02-05T23:27:07.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:07.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.374723+0000) 2020-02-05T23:27:07.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:07.374+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.374849+0000) 2020-02-05T23:27:07.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:07.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.375799+0000) 2020-02-05T23:27:07.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:07.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.376518+0000) 2020-02-05T23:27:07.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:07.375+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.375823+0000) 2020-02-05T23:27:07.378 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:07.375+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:07.375995+0000) 2020-02-05T23:27:07.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:07.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 144961 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:09.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:09.557+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:09.557505+0000) 2020-02-05T23:27:09.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:09.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:09.988405+0000) 2020-02-05T23:27:10.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:10.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:10.164483+0000) 2020-02-05T23:27:10.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:10.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:10.403403+0000) 2020-02-05T23:27:12.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:12.390+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.391934+0000) 2020-02-05T23:27:12.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:12.391+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.393408+0000) 2020-02-05T23:27:12.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:12.393+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.394429+0000) 2020-02-05T23:27:12.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:12.393+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.394773+0000) 2020-02-05T23:27:12.396 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:12.394+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.395959+0000) 2020-02-05T23:27:12.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:12.395+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.396376+0000) 2020-02-05T23:27:12.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:12.395+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.396645+0000) 2020-02-05T23:27:12.398 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:12.395+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.396945+0000) 2020-02-05T23:27:12.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:12.395+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.397320+0000) 2020-02-05T23:27:12.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:12.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.397763+0000) 2020-02-05T23:27:12.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:12.396+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.397481+0000) 2020-02-05T23:27:12.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:12.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.398194+0000) 2020-02-05T23:27:12.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:12.399+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.400900+0000) 2020-02-05T23:27:12.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:12.400+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.401154+0000) 2020-02-05T23:27:12.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:12.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145000 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:12.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:12.417+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.419396+0000) 2020-02-05T23:27:12.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:12.769+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:12.770784+0000) 2020-02-05T23:27:13.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:13.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:13.057782+0000) 2020-02-05T23:27:13.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:13.932+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:13.932864+0000) 2020-02-05T23:27:13.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:13.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:13.932844+0000) 2020-02-05T23:27:16.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:16.031+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:16.032898+0000) 2020-02-05T23:27:17.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:17.165+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.165405+0000) 2020-02-05T23:27:17.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:17.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.171537+0000) 2020-02-05T23:27:17.173 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:17.170+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.171711+0000) 2020-02-05T23:27:17.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:17.170+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.171701+0000) 2020-02-05T23:27:17.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:17.412+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.413903+0000) 2020-02-05T23:27:17.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:17.413+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.414999+0000) 2020-02-05T23:27:17.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:17.414+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.416060+0000) 2020-02-05T23:27:17.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:17.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.416352+0000) 2020-02-05T23:27:17.417 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:17.415+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.416900+0000) 2020-02-05T23:27:17.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:17.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.417461+0000) 2020-02-05T23:27:17.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:17.416+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.417535+0000) 2020-02-05T23:27:17.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:17.416+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.417941+0000) 2020-02-05T23:27:17.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:17.416+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.418207+0000) 2020-02-05T23:27:17.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:17.416+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.418247+0000) 2020-02-05T23:27:17.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:17.417+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.418842+0000) 2020-02-05T23:27:17.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:17.417+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.419266+0000) 2020-02-05T23:27:17.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:17.418+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.419834+0000) 2020-02-05T23:27:17.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:17.425+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:17.426335+0000) 2020-02-05T23:27:17.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:17.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:19.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:19.557+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:19.557758+0000) 2020-02-05T23:27:19.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:19.996+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:19.996457+0000) 2020-02-05T23:27:20.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:20.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:20.164708+0000) 2020-02-05T23:27:20.347 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:27:20.353 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:27:20.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:20.402+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:20.403624+0000) 2020-02-05T23:27:22.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:22.418+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.419631+0000) 2020-02-05T23:27:22.437 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:22.435+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.436343+0000) 2020-02-05T23:27:22.437 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:22.435+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.436848+0000) 2020-02-05T23:27:22.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:22.436+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.437258+0000) 2020-02-05T23:27:22.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:22.436+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.437681+0000) 2020-02-05T23:27:22.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:22.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.439227+0000) 2020-02-05T23:27:22.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:22.438+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.439717+0000) 2020-02-05T23:27:22.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:22.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.440148+0000) 2020-02-05T23:27:22.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:22.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.440449+0000) 2020-02-05T23:27:22.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:22.439+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.441262+0000) 2020-02-05T23:27:22.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:22.440+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.441790+0000) 2020-02-05T23:27:22.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:22.440+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.442096+0000) 2020-02-05T23:27:22.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:22.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.442465+0000) 2020-02-05T23:27:22.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:22.441+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.442928+0000) 2020-02-05T23:27:22.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:22.444+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.445517+0000) 2020-02-05T23:27:22.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:22.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:22.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:22.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:22.771028+0000) 2020-02-05T23:27:23.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:23.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:23.057986+0000) 2020-02-05T23:27:23.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:23.933+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:23.933128+0000) 2020-02-05T23:27:23.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:23.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:23.933129+0000) 2020-02-05T23:27:26.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:26.032+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:26.033135+0000) 2020-02-05T23:27:27.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:27.165+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.165638+0000) 2020-02-05T23:27:27.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:27.170+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.171773+0000) 2020-02-05T23:27:27.173 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:27.170+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.171953+0000) 2020-02-05T23:27:27.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:27.170+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.171930+0000) 2020-02-05T23:27:27.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:27.458+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.458695+0000) 2020-02-05T23:27:27.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:27.459+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.459163+0000) 2020-02-05T23:27:27.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:27.459+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.459817+0000) 2020-02-05T23:27:27.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:27.460+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.460208+0000) 2020-02-05T23:27:27.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:27.460+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.460784+0000) 2020-02-05T23:27:27.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:27.460+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.461458+0000) 2020-02-05T23:27:27.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:27.460+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.462329+0000) 2020-02-05T23:27:27.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:27.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.462700+0000) 2020-02-05T23:27:27.464 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:27.461+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.462893+0000) 2020-02-05T23:27:27.464 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:27.461+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.463337+0000) 2020-02-05T23:27:27.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:27.462+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.463908+0000) 2020-02-05T23:27:27.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:27.462+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.464125+0000) 2020-02-05T23:27:27.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:27.464+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.464964+0000) 2020-02-05T23:27:27.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:27.468+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:27.468158+0000) 2020-02-05T23:27:27.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:27.474+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:29.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:29.556+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:29.557996+0000) 2020-02-05T23:27:30.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:30.003+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:30.004588+0000) 2020-02-05T23:27:30.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:30.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:30.164948+0000) 2020-02-05T23:27:30.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:30.403+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:30.403893+0000) 2020-02-05T23:27:32.421 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:32.419+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.419871+0000) 2020-02-05T23:27:32.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:32.479+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.480830+0000) 2020-02-05T23:27:32.482 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:32.480+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.481238+0000) 2020-02-05T23:27:32.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:32.480+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.481863+0000) 2020-02-05T23:27:32.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:32.481+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.482546+0000) 2020-02-05T23:27:32.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:32.481+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.483032+0000) 2020-02-05T23:27:32.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:32.482+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.483601+0000) 2020-02-05T23:27:32.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:32.483+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.484394+0000) 2020-02-05T23:27:32.486 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:32.484+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.485943+0000) 2020-02-05T23:27:32.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:32.485+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.486437+0000) 2020-02-05T23:27:32.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:32.485+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.486805+0000) 2020-02-05T23:27:32.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:32.485+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.487346+0000) 2020-02-05T23:27:32.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:32.486+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.487945+0000) 2020-02-05T23:27:32.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:32.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.488018+0000) 2020-02-05T23:27:32.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:32.492+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.493558+0000) 2020-02-05T23:27:32.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:32.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:32.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:32.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:32.771268+0000) 2020-02-05T23:27:33.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:33.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:33.058189+0000) 2020-02-05T23:27:33.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:33.932+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:33.933408+0000) 2020-02-05T23:27:33.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:33.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:33.933408+0000) 2020-02-05T23:27:36.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:36.032+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:36.033367+0000) 2020-02-05T23:27:37.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:37.165+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.165887+0000) 2020-02-05T23:27:37.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:37.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.172057+0000) 2020-02-05T23:27:37.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:37.171+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.172192+0000) 2020-02-05T23:27:37.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:37.171+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.172212+0000) 2020-02-05T23:27:37.505 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:37.503+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.504428+0000) 2020-02-05T23:27:37.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:37.504+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.505942+0000) 2020-02-05T23:27:37.507 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:37.505+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.506246+0000) 2020-02-05T23:27:37.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:37.505+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.506900+0000) 2020-02-05T23:27:37.508 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:37.506+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.507596+0000) 2020-02-05T23:27:37.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:37.506+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.508005+0000) 2020-02-05T23:27:37.509 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:37.507+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.508528+0000) 2020-02-05T23:27:37.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:37.507+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.508881+0000) 2020-02-05T23:27:37.510 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:37.508+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.509368+0000) 2020-02-05T23:27:37.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:37.508+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.509927+0000) 2020-02-05T23:27:37.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:37.508+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.509774+0000) 2020-02-05T23:27:37.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:37.510+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.511389+0000) 2020-02-05T23:27:37.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:37.510+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.511635+0000) 2020-02-05T23:27:37.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:37.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:37.515648+0000) 2020-02-05T23:27:37.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:37.520+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:39.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:39.557+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:39.558281+0000) 2020-02-05T23:27:40.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:40.011+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:40.012854+0000) 2020-02-05T23:27:40.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:40.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:40.165208+0000) 2020-02-05T23:27:40.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:40.403+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:40.404135+0000) 2020-02-05T23:27:42.421 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:42.419+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.420127+0000) 2020-02-05T23:27:42.527 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:42.525+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.526135+0000) 2020-02-05T23:27:42.528 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:42.526+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.527553+0000) 2020-02-05T23:27:42.532 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:42.527+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.528457+0000) 2020-02-05T23:27:42.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:42.527+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.529007+0000) 2020-02-05T23:27:42.532 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:42.528+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.529651+0000) 2020-02-05T23:27:42.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:42.527+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.528404+0000) 2020-02-05T23:27:42.533 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:42.527+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.528723+0000) 2020-02-05T23:27:42.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:42.528+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.529312+0000) 2020-02-05T23:27:42.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:42.529+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.530195+0000) 2020-02-05T23:27:42.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:42.528+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.529634+0000) 2020-02-05T23:27:42.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:42.529+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.531008+0000) 2020-02-05T23:27:42.548 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:42.529+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.530774+0000) 2020-02-05T23:27:42.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:42.530+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.531626+0000) 2020-02-05T23:27:42.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:42.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.536394+0000) 2020-02-05T23:27:42.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:42.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:42.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:42.770+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:42.771483+0000) 2020-02-05T23:27:43.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:43.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:43.058395+0000) 2020-02-05T23:27:43.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:43.932+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:43.933644+0000) 2020-02-05T23:27:43.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:43.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:43.933643+0000) 2020-02-05T23:27:46.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:46.033+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:46.033645+0000) 2020-02-05T23:27:47.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:47.165+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.166122+0000) 2020-02-05T23:27:47.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:47.171+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.172313+0000) 2020-02-05T23:27:47.173 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:47.171+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.172484+0000) 2020-02-05T23:27:47.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:47.171+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.172503+0000) 2020-02-05T23:27:47.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:47.548+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.549644+0000) 2020-02-05T23:27:47.552 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:47.550+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.551128+0000) 2020-02-05T23:27:47.553 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:47.550+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.551959+0000) 2020-02-05T23:27:47.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:47.551+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.552758+0000) 2020-02-05T23:27:47.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:47.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.552908+0000) 2020-02-05T23:27:47.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:47.552+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.553552+0000) 2020-02-05T23:27:47.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:47.552+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.553444+0000) 2020-02-05T23:27:47.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:47.552+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.554032+0000) 2020-02-05T23:27:47.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:47.552+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.553975+0000) 2020-02-05T23:27:47.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:47.553+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.554790+0000) 2020-02-05T23:27:47.556 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:47.553+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.555088+0000) 2020-02-05T23:27:47.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:47.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.555231+0000) 2020-02-05T23:27:47.557 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:47.554+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.555390+0000) 2020-02-05T23:27:47.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:47.560+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:47.561245+0000) 2020-02-05T23:27:47.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:47.564+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:49.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:49.557+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:49.558535+0000) 2020-02-05T23:27:50.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:50.020+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:50.021648+0000) 2020-02-05T23:27:50.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:50.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:50.165452+0000) 2020-02-05T23:27:50.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:50.403+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:50.404388+0000) 2020-02-05T23:27:50.527 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:27:50.533 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:27:52.421 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:52.419+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.420368+0000) 2020-02-05T23:27:52.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:52.569+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.570917+0000) 2020-02-05T23:27:52.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:52.570+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.571676+0000) 2020-02-05T23:27:52.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:52.571+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.572835+0000) 2020-02-05T23:27:52.574 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:52.572+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.573459+0000) 2020-02-05T23:27:52.575 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:52.572+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.573918+0000) 2020-02-05T23:27:52.575 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:52.572+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.573839+0000) 2020-02-05T23:27:52.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:52.572+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.574308+0000) 2020-02-05T23:27:52.576 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:52.573+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.574730+0000) 2020-02-05T23:27:52.578 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:52.574+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.575174+0000) 2020-02-05T23:27:52.578 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:52.575+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.577044+0000) 2020-02-05T23:27:52.579 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:52.576+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.577685+0000) 2020-02-05T23:27:52.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:52.576+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.577475+0000) 2020-02-05T23:27:52.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:52.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.580929+0000) 2020-02-05T23:27:52.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:52.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.581205+0000) 2020-02-05T23:27:52.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:52.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:52.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:52.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:52.771697+0000) 2020-02-05T23:27:53.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:53.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:53.058600+0000) 2020-02-05T23:27:53.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:53.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:53.933873+0000) 2020-02-05T23:27:53.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:53.933+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:53.933917+0000) 2020-02-05T23:27:56.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:56.032+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:56.033823+0000) 2020-02-05T23:27:57.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:57.165+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.166355+0000) 2020-02-05T23:27:57.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:57.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.172584+0000) 2020-02-05T23:27:57.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:57.172+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.172823+0000) 2020-02-05T23:27:57.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:57.172+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.172780+0000) 2020-02-05T23:27:57.596 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:27:57.594+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.595066+0000) 2020-02-05T23:27:57.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:27:57.594+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.595914+0000) 2020-02-05T23:27:57.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:27:57.595+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.596633+0000) 2020-02-05T23:27:57.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:57.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.596932+0000) 2020-02-05T23:27:57.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:27:57.596+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.597507+0000) 2020-02-05T23:27:57.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:27:57.596+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.598161+0000) 2020-02-05T23:27:57.599 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:27:57.597+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.598474+0000) 2020-02-05T23:27:57.600 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:27:57.598+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.599822+0000) 2020-02-05T23:27:57.601 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:27:57.599+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.600521+0000) 2020-02-05T23:27:57.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:57.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.600796+0000) 2020-02-05T23:27:57.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:27:57.599+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.601046+0000) 2020-02-05T23:27:57.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:27:57.600+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.601537+0000) 2020-02-05T23:27:57.603 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:27:57.600+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.601779+0000) 2020-02-05T23:27:57.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:57.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:57.604624+0000) 2020-02-05T23:27:57.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:27:57.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:27:59.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:27:59.558+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:27:59.558768+0000) 2020-02-05T23:28:00.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:00.028+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:00.029831+0000) 2020-02-05T23:28:00.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:00.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:00.165717+0000) 2020-02-05T23:28:00.405 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:00.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:00.404669+0000) 2020-02-05T23:28:02.421 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:02.420+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.420643+0000) 2020-02-05T23:28:02.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:02.615+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.616495+0000) 2020-02-05T23:28:02.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:02.616+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.616714+0000) 2020-02-05T23:28:02.618 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:02.616+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.617602+0000) 2020-02-05T23:28:02.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:02.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.618233+0000) 2020-02-05T23:28:02.619 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:02.618+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.618505+0000) 2020-02-05T23:28:02.620 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:02.617+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.618972+0000) 2020-02-05T23:28:02.620 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:02.618+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.619425+0000) 2020-02-05T23:28:02.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:02.619+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.619782+0000) 2020-02-05T23:28:02.621 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:02.619+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.620008+0000) 2020-02-05T23:28:02.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:02.618+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.619820+0000) 2020-02-05T23:28:02.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:02.620+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.620499+0000) 2020-02-05T23:28:02.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:02.620+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.620882+0000) 2020-02-05T23:28:02.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:02.620+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.621486+0000) 2020-02-05T23:28:02.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:02.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.626926+0000) 2020-02-05T23:28:02.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:02.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:02.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:02.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:02.771909+0000) 2020-02-05T23:28:03.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:03.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:03.058809+0000) 2020-02-05T23:28:03.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:03.933+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:03.934156+0000) 2020-02-05T23:28:03.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:03.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:03.934130+0000) 2020-02-05T23:28:06.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:06.033+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:06.034089+0000) 2020-02-05T23:28:07.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:07.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.166580+0000) 2020-02-05T23:28:07.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:07.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.172826+0000) 2020-02-05T23:28:07.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:07.172+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.173063+0000) 2020-02-05T23:28:07.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:07.172+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.173031+0000) 2020-02-05T23:28:07.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:07.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.638148+0000) 2020-02-05T23:28:07.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:07.637+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.638507+0000) 2020-02-05T23:28:07.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:07.638+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.639170+0000) 2020-02-05T23:28:07.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:07.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.639801+0000) 2020-02-05T23:28:07.641 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:07.639+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.639921+0000) 2020-02-05T23:28:07.641 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:07.640+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.640817+0000) 2020-02-05T23:28:07.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:07.640+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.641208+0000) 2020-02-05T23:28:07.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:07.640+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.641637+0000) 2020-02-05T23:28:07.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:07.641+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.642137+0000) 2020-02-05T23:28:07.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:07.641+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.642314+0000) 2020-02-05T23:28:07.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:07.642+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.643074+0000) 2020-02-05T23:28:07.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:07.642+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.643302+0000) 2020-02-05T23:28:07.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:07.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.643346+0000) 2020-02-05T23:28:07.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:07.645+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:07.646270+0000) 2020-02-05T23:28:07.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:07.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:09.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:09.558+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:09.558997+0000) 2020-02-05T23:28:10.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:10.037+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:10.037618+0000) 2020-02-05T23:28:10.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:10.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:10.166005+0000) 2020-02-05T23:28:10.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:10.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:10.404904+0000) 2020-02-05T23:28:12.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:12.420+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.420904+0000) 2020-02-05T23:28:12.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:12.659+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.660978+0000) 2020-02-05T23:28:12.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:12.660+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.661913+0000) 2020-02-05T23:28:12.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:12.661+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.662534+0000) 2020-02-05T23:28:12.664 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:12.662+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.663229+0000) 2020-02-05T23:28:12.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:12.663+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.663527+0000) 2020-02-05T23:28:12.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:12.663+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.664143+0000) 2020-02-05T23:28:12.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:12.663+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.663924+0000) 2020-02-05T23:28:12.665 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:12.663+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.664919+0000) 2020-02-05T23:28:12.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:12.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.664803+0000) 2020-02-05T23:28:12.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:12.664+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.665270+0000) 2020-02-05T23:28:12.667 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:12.665+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.665666+0000) 2020-02-05T23:28:12.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:12.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.666351+0000) 2020-02-05T23:28:12.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:12.665+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.666393+0000) 2020-02-05T23:28:12.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:12.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.674367+0000) 2020-02-05T23:28:12.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:12.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145473 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:12.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:12.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:12.772146+0000) 2020-02-05T23:28:13.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:13.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:13.059019+0000) 2020-02-05T23:28:13.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:13.933+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:13.934396+0000) 2020-02-05T23:28:13.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:13.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:13.934382+0000) 2020-02-05T23:28:16.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:16.033+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:16.034317+0000) 2020-02-05T23:28:17.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:17.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.166797+0000) 2020-02-05T23:28:17.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:17.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.173121+0000) 2020-02-05T23:28:17.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:17.172+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.173285+0000) 2020-02-05T23:28:17.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:17.172+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.173294+0000) 2020-02-05T23:28:17.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:17.682+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.683362+0000) 2020-02-05T23:28:17.685 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:17.683+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.684146+0000) 2020-02-05T23:28:17.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:17.684+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.684775+0000) 2020-02-05T23:28:17.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:17.685+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.685358+0000) 2020-02-05T23:28:17.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:17.685+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.685608+0000) 2020-02-05T23:28:17.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:17.686+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.686640+0000) 2020-02-05T23:28:17.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:17.686+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.686992+0000) 2020-02-05T23:28:17.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:17.686+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.687145+0000) 2020-02-05T23:28:17.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:17.686+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.687433+0000) 2020-02-05T23:28:17.689 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:17.687+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.687530+0000) 2020-02-05T23:28:17.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:17.687+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.687726+0000) 2020-02-05T23:28:17.689 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:17.687+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.688038+0000) 2020-02-05T23:28:17.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:17.686+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.687993+0000) 2020-02-05T23:28:17.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:17.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:17.691390+0000) 2020-02-05T23:28:17.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:17.696+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145512 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:19.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:19.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:19.559266+0000) 2020-02-05T23:28:20.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:20.045+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:20.045505+0000) 2020-02-05T23:28:20.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:20.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:20.166270+0000) 2020-02-05T23:28:20.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:20.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:20.405197+0000) 2020-02-05T23:28:20.664 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:28:20.670 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:28:22.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:22.420+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.421170+0000) 2020-02-05T23:28:22.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:22.702+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.703179+0000) 2020-02-05T23:28:22.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:22.703+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.704175+0000) 2020-02-05T23:28:22.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:22.704+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.704966+0000) 2020-02-05T23:28:22.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:22.705+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.705700+0000) 2020-02-05T23:28:22.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:22.704+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.705887+0000) 2020-02-05T23:28:22.707 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:22.705+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.706286+0000) 2020-02-05T23:28:22.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:22.705+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.706372+0000) 2020-02-05T23:28:22.708 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:22.706+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.706655+0000) 2020-02-05T23:28:22.708 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:22.706+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.707183+0000) 2020-02-05T23:28:22.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:22.706+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.707937+0000) 2020-02-05T23:28:22.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:22.706+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.706651+0000) 2020-02-05T23:28:22.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:22.707+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.708006+0000) 2020-02-05T23:28:22.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:22.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.717200+0000) 2020-02-05T23:28:22.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:22.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.717539+0000) 2020-02-05T23:28:22.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:22.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:22.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:22.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:22.772357+0000) 2020-02-05T23:28:23.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:23.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:23.059257+0000) 2020-02-05T23:28:23.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:23.933+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:23.934645+0000) 2020-02-05T23:28:23.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:23.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:23.934633+0000) 2020-02-05T23:28:26.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:26.034+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:26.034652+0000) 2020-02-05T23:28:27.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:27.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.167023+0000) 2020-02-05T23:28:27.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:27.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.173375+0000) 2020-02-05T23:28:27.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:27.173+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.173544+0000) 2020-02-05T23:28:27.175 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:27.173+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.173750+0000) 2020-02-05T23:28:27.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:27.726+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.726990+0000) 2020-02-05T23:28:27.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:27.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.727345+0000) 2020-02-05T23:28:27.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:27.727+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.728164+0000) 2020-02-05T23:28:27.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:27.727+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.728161+0000) 2020-02-05T23:28:27.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:27.728+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.729165+0000) 2020-02-05T23:28:27.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:27.729+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.730255+0000) 2020-02-05T23:28:27.740 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:27.730+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.730738+0000) 2020-02-05T23:28:27.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:27.727+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.728725+0000) 2020-02-05T23:28:27.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:27.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.728852+0000) 2020-02-05T23:28:27.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:27.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.730722+0000) 2020-02-05T23:28:27.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:27.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.734748+0000) 2020-02-05T23:28:27.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:27.728+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.729389+0000) 2020-02-05T23:28:27.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:27.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.729527+0000) 2020-02-05T23:28:27.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:27.729+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:27.730189+0000) 2020-02-05T23:28:27.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:27.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:29.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:29.558+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:29.559497+0000) 2020-02-05T23:28:30.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:30.052+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:30.053659+0000) 2020-02-05T23:28:30.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:30.165+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:30.166500+0000) 2020-02-05T23:28:30.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:30.405+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:30.405461+0000) 2020-02-05T23:28:32.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:32.421+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.421414+0000) 2020-02-05T23:28:32.749 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:32.747+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.748242+0000) 2020-02-05T23:28:32.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:32.747+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.748695+0000) 2020-02-05T23:28:32.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:32.748+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.749659+0000) 2020-02-05T23:28:32.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:32.749+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.750309+0000) 2020-02-05T23:28:32.752 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:32.750+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.751198+0000) 2020-02-05T23:28:32.753 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:32.751+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.752113+0000) 2020-02-05T23:28:32.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:32.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.752637+0000) 2020-02-05T23:28:32.754 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:32.751+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.752968+0000) 2020-02-05T23:28:32.754 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:32.752+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.753252+0000) 2020-02-05T23:28:32.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:32.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.753613+0000) 2020-02-05T23:28:32.755 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:32.753+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.754296+0000) 2020-02-05T23:28:32.755 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:32.754+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.754459+0000) 2020-02-05T23:28:32.756 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:32.754+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.754805+0000) 2020-02-05T23:28:32.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:32.758+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.759407+0000) 2020-02-05T23:28:32.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:32.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:32.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:32.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:32.772564+0000) 2020-02-05T23:28:33.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:33.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:33.059470+0000) 2020-02-05T23:28:33.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:33.934+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:33.934899+0000) 2020-02-05T23:28:33.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:33.934+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:33.934870+0000) 2020-02-05T23:28:36.036 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:36.034+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:36.034918+0000) 2020-02-05T23:28:37.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:37.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.167246+0000) 2020-02-05T23:28:37.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:37.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.173652+0000) 2020-02-05T23:28:37.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:37.173+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.173761+0000) 2020-02-05T23:28:37.175 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:37.173+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.174144+0000) 2020-02-05T23:28:37.773 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:37.771+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.772103+0000) 2020-02-05T23:28:37.773 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:37.771+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.772781+0000) 2020-02-05T23:28:37.774 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:37.772+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.772526+0000) 2020-02-05T23:28:37.774 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:37.772+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.773196+0000) 2020-02-05T23:28:37.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:37.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.773547+0000) 2020-02-05T23:28:37.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:37.772+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.773833+0000) 2020-02-05T23:28:37.775 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:37.773+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.774131+0000) 2020-02-05T23:28:37.776 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:37.773+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.774340+0000) 2020-02-05T23:28:37.776 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:37.774+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.774919+0000) 2020-02-05T23:28:37.776 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:37.773+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.774600+0000) 2020-02-05T23:28:37.777 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:37.775+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.775681+0000) 2020-02-05T23:28:37.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:37.774+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.775529+0000) 2020-02-05T23:28:37.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:37.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.775537+0000) 2020-02-05T23:28:37.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:37.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:37.776404+0000) 2020-02-05T23:28:37.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:37.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:39.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:39.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:39.559777+0000) 2020-02-05T23:28:40.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:40.061+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:40.061847+0000) 2020-02-05T23:28:40.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:40.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:40.166730+0000) 2020-02-05T23:28:40.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:40.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:40.405729+0000) 2020-02-05T23:28:42.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:42.420+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.421693+0000) 2020-02-05T23:28:42.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:42.771+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.772773+0000) 2020-02-05T23:28:42.794 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:42.791+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.792984+0000) 2020-02-05T23:28:42.795 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:42.793+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.794535+0000) 2020-02-05T23:28:42.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:42.793+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.795008+0000) 2020-02-05T23:28:42.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:42.794+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.795414+0000) 2020-02-05T23:28:42.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:42.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.795719+0000) 2020-02-05T23:28:42.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:42.795+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.796196+0000) 2020-02-05T23:28:42.797 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:42.795+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.796576+0000) 2020-02-05T23:28:42.798 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:42.796+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.797238+0000) 2020-02-05T23:28:42.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:42.796+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.797706+0000) 2020-02-05T23:28:42.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:42.796+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.798394+0000) 2020-02-05T23:28:42.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:42.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.798683+0000) 2020-02-05T23:28:42.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:42.797+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.799030+0000) 2020-02-05T23:28:42.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:42.797+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.799390+0000) 2020-02-05T23:28:42.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:42.803+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:42.804141+0000) 2020-02-05T23:28:42.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:42.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145716 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:43.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:43.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:43.059709+0000) 2020-02-05T23:28:43.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:43.934+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:43.935130+0000) 2020-02-05T23:28:43.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:43.934+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:43.935126+0000) 2020-02-05T23:28:46.036 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:46.034+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:46.035243+0000) 2020-02-05T23:28:47.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:47.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.167501+0000) 2020-02-05T23:28:47.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:47.172+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.173919+0000) 2020-02-05T23:28:47.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:47.172+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.173975+0000) 2020-02-05T23:28:47.175 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:47.172+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.174371+0000) 2020-02-05T23:28:47.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:47.816+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.817542+0000) 2020-02-05T23:28:47.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:47.817+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.818139+0000) 2020-02-05T23:28:47.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:47.817+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.818635+0000) 2020-02-05T23:28:47.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:47.818+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.819260+0000) 2020-02-05T23:28:47.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:47.818+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.819720+0000) 2020-02-05T23:28:47.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:47.819+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.820478+0000) 2020-02-05T23:28:47.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:47.820+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.821273+0000) 2020-02-05T23:28:47.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:47.820+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.821452+0000) 2020-02-05T23:28:47.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:47.820+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.822116+0000) 2020-02-05T23:28:47.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:47.820+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.822134+0000) 2020-02-05T23:28:47.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:47.820+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.822194+0000) 2020-02-05T23:28:47.825 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:47.821+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.822957+0000) 2020-02-05T23:28:47.825 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:47.821+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.823008+0000) 2020-02-05T23:28:47.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:47.825+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:47.827039+0000) 2020-02-05T23:28:47.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:47.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145757 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:49.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:49.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:49.560016+0000) 2020-02-05T23:28:50.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:50.069+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:50.070137+0000) 2020-02-05T23:28:50.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:50.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:50.166962+0000) 2020-02-05T23:28:50.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:50.404+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:50.406010+0000) 2020-02-05T23:28:50.801 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:28:50.808 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:28:52.423 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:52.421+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.421940+0000) 2020-02-05T23:28:52.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:52.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.773001+0000) 2020-02-05T23:28:52.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:52.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.838335+0000) 2020-02-05T23:28:52.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:52.837+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.838725+0000) 2020-02-05T23:28:52.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:52.838+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.839660+0000) 2020-02-05T23:28:52.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:52.839+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.840525+0000) 2020-02-05T23:28:52.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:52.839+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.840865+0000) 2020-02-05T23:28:52.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:52.839+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.840969+0000) 2020-02-05T23:28:52.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:52.840+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.841819+0000) 2020-02-05T23:28:52.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:52.840+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.842389+0000) 2020-02-05T23:28:52.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:52.841+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.842619+0000) 2020-02-05T23:28:52.844 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:52.842+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.843788+0000) 2020-02-05T23:28:52.846 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:52.843+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.844282+0000) 2020-02-05T23:28:52.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:52.842+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.844135+0000) 2020-02-05T23:28:52.846 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:52.843+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.845154+0000) 2020-02-05T23:28:52.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:52.846+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:52.847194+0000) 2020-02-05T23:28:52.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:52.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145798 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:53.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:53.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:53.059936+0000) 2020-02-05T23:28:53.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:53.934+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:53.935310+0000) 2020-02-05T23:28:53.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:53.934+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:53.935395+0000) 2020-02-05T23:28:56.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:56.034+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:56.035495+0000) 2020-02-05T23:28:57.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:57.166+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.167720+0000) 2020-02-05T23:28:57.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:57.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.174128+0000) 2020-02-05T23:28:57.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:57.173+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.174208+0000) 2020-02-05T23:28:57.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:57.173+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.174662+0000) 2020-02-05T23:28:57.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:57.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.863423+0000) 2020-02-05T23:28:57.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:28:57.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.864082+0000) 2020-02-05T23:28:57.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:28:57.863+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.864387+0000) 2020-02-05T23:28:57.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:28:57.863+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.864760+0000) 2020-02-05T23:28:57.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:28:57.864+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.865626+0000) 2020-02-05T23:28:57.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:28:57.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.865998+0000) 2020-02-05T23:28:57.869 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:28:57.865+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.867179+0000) 2020-02-05T23:28:57.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:28:57.866+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.867903+0000) 2020-02-05T23:28:57.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:28:57.866+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.868129+0000) 2020-02-05T23:28:57.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:28:57.867+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.868867+0000) 2020-02-05T23:28:57.871 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:28:57.868+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.869191+0000) 2020-02-05T23:28:57.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:28:57.868+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.869452+0000) 2020-02-05T23:28:57.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:57.870+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.871131+0000) 2020-02-05T23:28:57.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:57.870+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:57.871335+0000) 2020-02-05T23:28:57.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:28:57.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:28:59.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:28:59.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:28:59.560280+0000) 2020-02-05T23:29:00.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:00.077+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:00.078207+0000) 2020-02-05T23:29:00.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:00.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:00.167285+0000) 2020-02-05T23:29:00.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:00.405+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:00.406249+0000) 2020-02-05T23:29:02.423 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:02.421+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.422182+0000) 2020-02-05T23:29:02.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:02.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.773268+0000) 2020-02-05T23:29:02.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:02.883+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.885352+0000) 2020-02-05T23:29:02.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:02.885+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.886668+0000) 2020-02-05T23:29:02.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:02.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.887737+0000) 2020-02-05T23:29:02.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:02.887+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.888483+0000) 2020-02-05T23:29:02.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:02.887+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.888873+0000) 2020-02-05T23:29:02.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:02.888+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.889749+0000) 2020-02-05T23:29:02.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:02.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.890112+0000) 2020-02-05T23:29:02.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:02.888+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.890335+0000) 2020-02-05T23:29:02.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:02.889+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.891138+0000) 2020-02-05T23:29:02.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:02.890+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.891962+0000) 2020-02-05T23:29:02.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:02.890+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.892281+0000) 2020-02-05T23:29:02.894 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:02.891+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.892329+0000) 2020-02-05T23:29:02.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:02.891+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.892988+0000) 2020-02-05T23:29:02.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:02.897+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:02.898327+0000) 2020-02-05T23:29:02.905 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:02.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145880 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:03.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:03.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:03.060150+0000) 2020-02-05T23:29:03.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:03.935+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:03.935579+0000) 2020-02-05T23:29:03.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:03.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:03.935681+0000) 2020-02-05T23:29:06.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:06.035+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:06.035790+0000) 2020-02-05T23:29:07.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:07.167+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.167947+0000) 2020-02-05T23:29:07.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:07.173+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.174409+0000) 2020-02-05T23:29:07.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:07.173+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.174477+0000) 2020-02-05T23:29:07.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:07.173+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.174947+0000) 2020-02-05T23:29:07.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:07.908+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.909754+0000) 2020-02-05T23:29:07.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:07.908+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.910278+0000) 2020-02-05T23:29:07.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:07.909+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.910769+0000) 2020-02-05T23:29:07.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:07.910+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.911538+0000) 2020-02-05T23:29:07.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:07.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.912014+0000) 2020-02-05T23:29:07.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:07.911+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.912355+0000) 2020-02-05T23:29:07.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:07.911+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.912470+0000) 2020-02-05T23:29:07.914 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:07.911+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.912857+0000) 2020-02-05T23:29:07.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:07.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.913319+0000) 2020-02-05T23:29:07.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:07.912+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.913457+0000) 2020-02-05T23:29:07.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:07.912+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.913662+0000) 2020-02-05T23:29:07.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:07.912+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.913895+0000) 2020-02-05T23:29:07.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:07.913+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.914471+0000) 2020-02-05T23:29:07.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:07.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:07.917262+0000) 2020-02-05T23:29:07.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:07.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:09.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:09.560+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:09.560536+0000) 2020-02-05T23:29:10.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:10.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:10.085294+0000) 2020-02-05T23:29:10.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:10.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:10.167544+0000) 2020-02-05T23:29:10.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:10.405+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:10.406533+0000) 2020-02-05T23:29:12.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:12.421+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.422466+0000) 2020-02-05T23:29:12.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:12.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.773501+0000) 2020-02-05T23:29:12.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:12.932+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.932578+0000) 2020-02-05T23:29:12.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:12.934+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.934296+0000) 2020-02-05T23:29:12.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:12.933+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.934889+0000) 2020-02-05T23:29:12.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:12.935+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.935351+0000) 2020-02-05T23:29:12.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:12.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.936371+0000) 2020-02-05T23:29:12.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:12.934+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.936403+0000) 2020-02-05T23:29:12.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:12.936+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.936850+0000) 2020-02-05T23:29:12.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:12.937+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.937233+0000) 2020-02-05T23:29:12.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:12.937+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.937953+0000) 2020-02-05T23:29:12.946 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:12.936+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.937580+0000) 2020-02-05T23:29:12.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:12.937+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.938770+0000) 2020-02-05T23:29:12.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:12.937+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.939125+0000) 2020-02-05T23:29:12.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:12.937+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.939247+0000) 2020-02-05T23:29:12.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:12.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:12.941751+0000) 2020-02-05T23:29:12.957 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:12.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 145962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:13.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:13.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:13.060386+0000) 2020-02-05T23:29:13.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:13.935+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:13.935794+0000) 2020-02-05T23:29:13.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:13.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:13.935889+0000) 2020-02-05T23:29:16.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:16.035+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:16.036030+0000) 2020-02-05T23:29:17.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:17.167+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.168175+0000) 2020-02-05T23:29:17.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:17.174+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.174749+0000) 2020-02-05T23:29:17.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:17.174+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.174748+0000) 2020-02-05T23:29:17.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:17.174+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.175195+0000) 2020-02-05T23:29:17.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:17.955+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.956612+0000) 2020-02-05T23:29:17.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:17.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.957568+0000) 2020-02-05T23:29:17.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:17.956+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.958045+0000) 2020-02-05T23:29:17.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:17.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.958594+0000) 2020-02-05T23:29:17.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:17.958+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.959444+0000) 2020-02-05T23:29:17.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:17.957+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.959231+0000) 2020-02-05T23:29:17.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:17.958+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.959910+0000) 2020-02-05T23:29:17.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:17.958+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.960250+0000) 2020-02-05T23:29:17.962 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:17.959+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.960698+0000) 2020-02-05T23:29:17.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:17.959+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.961078+0000) 2020-02-05T23:29:17.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:17.960+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.961941+0000) 2020-02-05T23:29:17.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:17.960+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.961831+0000) 2020-02-05T23:29:17.963 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:17.961+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.962544+0000) 2020-02-05T23:29:17.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:17.964+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:17.965183+0000) 2020-02-05T23:29:17.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:17.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:19.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:19.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:19.560790+0000) 2020-02-05T23:29:20.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:20.092+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:20.093631+0000) 2020-02-05T23:29:20.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:20.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:20.167778+0000) 2020-02-05T23:29:20.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:20.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:20.406792+0000) 2020-02-05T23:29:20.937 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:29:20.944 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:29:22.423 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:22.422+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.422725+0000) 2020-02-05T23:29:22.774 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:22.772+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.773729+0000) 2020-02-05T23:29:22.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:22.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.979941+0000) 2020-02-05T23:29:22.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:22.979+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.980279+0000) 2020-02-05T23:29:22.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:22.980+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.981618+0000) 2020-02-05T23:29:22.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:22.979+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.981293+0000) 2020-02-05T23:29:22.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:22.981+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.982276+0000) 2020-02-05T23:29:22.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:22.981+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.982666+0000) 2020-02-05T23:29:22.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:22.981+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.982850+0000) 2020-02-05T23:29:22.984 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:22.982+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.983658+0000) 2020-02-05T23:29:22.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:22.982+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.984008+0000) 2020-02-05T23:29:22.985 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:22.982+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.984294+0000) 2020-02-05T23:29:22.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:22.983+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.984843+0000) 2020-02-05T23:29:22.986 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:22.983+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.984541+0000) 2020-02-05T23:29:22.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:22.984+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.985362+0000) 2020-02-05T23:29:22.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:22.987+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:22.988263+0000) 2020-02-05T23:29:22.995 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:22.993+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:23.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:23.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:23.060594+0000) 2020-02-05T23:29:23.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:23.934+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:23.936029+0000) 2020-02-05T23:29:23.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:23.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:23.936144+0000) 2020-02-05T23:29:26.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:26.035+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:26.036284+0000) 2020-02-05T23:29:27.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:27.167+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:27.168402+0000) 2020-02-05T23:29:27.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:27.174+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:27.175053+0000) 2020-02-05T23:29:27.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:27.174+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:27.175055+0000) 2020-02-05T23:29:27.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:27.174+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:27.175502+0000) 2020-02-05T23:29:28.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:27.998+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.000142+0000) 2020-02-05T23:29:28.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:27.999+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.000753+0000) 2020-02-05T23:29:28.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:28.000+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.001345+0000) 2020-02-05T23:29:28.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:28.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.001780+0000) 2020-02-05T23:29:28.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:28.001+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.002223+0000) 2020-02-05T23:29:28.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:28.001+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.002545+0000) 2020-02-05T23:29:28.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:28.002+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.003620+0000) 2020-02-05T23:29:28.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:28.002+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.003818+0000) 2020-02-05T23:29:28.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:28.003+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.004654+0000) 2020-02-05T23:29:28.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:28.004+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.005841+0000) 2020-02-05T23:29:28.007 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:28.004+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.006287+0000) 2020-02-05T23:29:28.007 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:28.004+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.006165+0000) 2020-02-05T23:29:28.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:28.005+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.006960+0000) 2020-02-05T23:29:28.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:28.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:28.010384+0000) 2020-02-05T23:29:28.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:28.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146086 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:29.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:29.559+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:29.561038+0000) 2020-02-05T23:29:30.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:30.101+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:30.102921+0000) 2020-02-05T23:29:30.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:30.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:30.168007+0000) 2020-02-05T23:29:30.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:30.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:30.407027+0000) 2020-02-05T23:29:32.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:32.422+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:32.422984+0000) 2020-02-05T23:29:32.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:32.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:32.773962+0000) 2020-02-05T23:29:33.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:33.021+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.021620+0000) 2020-02-05T23:29:33.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:33.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.024203+0000) 2020-02-05T23:29:33.025 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:33.023+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.024915+0000) 2020-02-05T23:29:33.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:33.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.025622+0000) 2020-02-05T23:29:33.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:33.025+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.026113+0000) 2020-02-05T23:29:33.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:33.026+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.026386+0000) 2020-02-05T23:29:33.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:33.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.026743+0000) 2020-02-05T23:29:33.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:33.026+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.027824+0000) 2020-02-05T23:29:33.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:33.027+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.028341+0000) 2020-02-05T23:29:33.030 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:33.028+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.029280+0000) 2020-02-05T23:29:33.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:33.028+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.029277+0000) 2020-02-05T23:29:33.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:33.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.028977+0000) 2020-02-05T23:29:33.031 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:33.029+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.030296+0000) 2020-02-05T23:29:33.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:33.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.033757+0000) 2020-02-05T23:29:33.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:33.039+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146128 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:33.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:33.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.060812+0000) 2020-02-05T23:29:33.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:33.935+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.936270+0000) 2020-02-05T23:29:33.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:33.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:33.936392+0000) 2020-02-05T23:29:36.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:36.036+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:36.036517+0000) 2020-02-05T23:29:37.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:37.167+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:37.168628+0000) 2020-02-05T23:29:37.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:37.174+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:37.175367+0000) 2020-02-05T23:29:37.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:37.174+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:37.175363+0000) 2020-02-05T23:29:37.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:37.175+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:37.175755+0000) 2020-02-05T23:29:38.046 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:38.044+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.045466+0000) 2020-02-05T23:29:38.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:38.044+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.045208+0000) 2020-02-05T23:29:38.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:38.045+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.046148+0000) 2020-02-05T23:29:38.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:38.045+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.046779+0000) 2020-02-05T23:29:38.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:38.046+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.047970+0000) 2020-02-05T23:29:38.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:38.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.048354+0000) 2020-02-05T23:29:38.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:38.048+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.048822+0000) 2020-02-05T23:29:38.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:38.048+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.048815+0000) 2020-02-05T23:29:38.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:38.049+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.050701+0000) 2020-02-05T23:29:38.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:38.050+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.051187+0000) 2020-02-05T23:29:38.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:38.050+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.051542+0000) 2020-02-05T23:29:38.052 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:38.051+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.051853+0000) 2020-02-05T23:29:38.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:38.052+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.053777+0000) 2020-02-05T23:29:38.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:38.052+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:38.054020+0000) 2020-02-05T23:29:38.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:38.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146171 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:39.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:39.560+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:39.561260+0000) 2020-02-05T23:29:40.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:40.110+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:40.111196+0000) 2020-02-05T23:29:40.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:40.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:40.168263+0000) 2020-02-05T23:29:40.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:40.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:40.407287+0000) 2020-02-05T23:29:42.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:42.422+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:42.423244+0000) 2020-02-05T23:29:42.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:42.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:42.774270+0000) 2020-02-05T23:29:43.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:43.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.061039+0000) 2020-02-05T23:29:43.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:43.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.069429+0000) 2020-02-05T23:29:43.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:43.069+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.070435+0000) 2020-02-05T23:29:43.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:43.070+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.071341+0000) 2020-02-05T23:29:43.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:43.071+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.072396+0000) 2020-02-05T23:29:43.076 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:43.072+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.073354+0000) 2020-02-05T23:29:43.076 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:43.073+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.074071+0000) 2020-02-05T23:29:43.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:43.072+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.073217+0000) 2020-02-05T23:29:43.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:43.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.074315+0000) 2020-02-05T23:29:43.077 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:43.074+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.075141+0000) 2020-02-05T23:29:43.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:43.075+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.075659+0000) 2020-02-05T23:29:43.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:43.074+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.075407+0000) 2020-02-05T23:29:43.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:43.074+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.075593+0000) 2020-02-05T23:29:43.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:43.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.075493+0000) 2020-02-05T23:29:43.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:43.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.081120+0000) 2020-02-05T23:29:43.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:43.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:43.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:43.935+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.936486+0000) 2020-02-05T23:29:43.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:43.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:43.936631+0000) 2020-02-05T23:29:46.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:46.036+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:46.036795+0000) 2020-02-05T23:29:47.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:47.168+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:47.168842+0000) 2020-02-05T23:29:47.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:47.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:47.175710+0000) 2020-02-05T23:29:47.177 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:47.175+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:47.175710+0000) 2020-02-05T23:29:47.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:47.175+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:47.175964+0000) 2020-02-05T23:29:48.092 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:48.090+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.091375+0000) 2020-02-05T23:29:48.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:48.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.092891+0000) 2020-02-05T23:29:48.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:48.092+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.093502+0000) 2020-02-05T23:29:48.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:48.093+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.094587+0000) 2020-02-05T23:29:48.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:48.094+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.094679+0000) 2020-02-05T23:29:48.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:48.094+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.095038+0000) 2020-02-05T23:29:48.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:48.094+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.095858+0000) 2020-02-05T23:29:48.098 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:48.095+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.095688+0000) 2020-02-05T23:29:48.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:48.095+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.096016+0000) 2020-02-05T23:29:48.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:48.095+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.096579+0000) 2020-02-05T23:29:48.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:48.095+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.096773+0000) 2020-02-05T23:29:48.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:48.096+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.097152+0000) 2020-02-05T23:29:48.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:48.096+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.097116+0000) 2020-02-05T23:29:48.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:48.102+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:48.103535+0000) 2020-02-05T23:29:48.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:48.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:49.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:49.561+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:49.561504+0000) 2020-02-05T23:29:50.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:50.119+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:50.119473+0000) 2020-02-05T23:29:50.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:50.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:50.168498+0000) 2020-02-05T23:29:50.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:50.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:50.407526+0000) 2020-02-05T23:29:51.076 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:29:51.082 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:29:52.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:52.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:52.423505+0000) 2020-02-05T23:29:52.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:52.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:52.774524+0000) 2020-02-05T23:29:53.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:53.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.061338+0000) 2020-02-05T23:29:53.116 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:53.113+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.115037+0000) 2020-02-05T23:29:53.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:53.114+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.115985+0000) 2020-02-05T23:29:53.117 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:53.116+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.116513+0000) 2020-02-05T23:29:53.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:53.115+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.116752+0000) 2020-02-05T23:29:53.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:53.117+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.118157+0000) 2020-02-05T23:29:53.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:53.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.119170+0000) 2020-02-05T23:29:53.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:53.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.119641+0000) 2020-02-05T23:29:53.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:53.118+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.119815+0000) 2020-02-05T23:29:53.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:53.119+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.119993+0000) 2020-02-05T23:29:53.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:53.120+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.120664+0000) 2020-02-05T23:29:53.123 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:53.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.120810+0000) 2020-02-05T23:29:53.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:53.120+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.121152+0000) 2020-02-05T23:29:53.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:53.119+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.120187+0000) 2020-02-05T23:29:53.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:53.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.123790+0000) 2020-02-05T23:29:53.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:53.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:53.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:53.935+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.936671+0000) 2020-02-05T23:29:53.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:53.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:53.936843+0000) 2020-02-05T23:29:56.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:56.036+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:56.037039+0000) 2020-02-05T23:29:57.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:57.168+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:57.169144+0000) 2020-02-05T23:29:57.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:57.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:57.176007+0000) 2020-02-05T23:29:57.177 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:57.175+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:57.176011+0000) 2020-02-05T23:29:57.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:57.175+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:57.176279+0000) 2020-02-05T23:29:58.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:58.136+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.137303+0000) 2020-02-05T23:29:58.139 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:29:58.138+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.138748+0000) 2020-02-05T23:29:58.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:29:58.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.139010+0000) 2020-02-05T23:29:58.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:29:58.139+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.140517+0000) 2020-02-05T23:29:58.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:58.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.141324+0000) 2020-02-05T23:29:58.142 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:29:58.140+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.141921+0000) 2020-02-05T23:29:58.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:29:58.141+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.142366+0000) 2020-02-05T23:29:58.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:29:58.142+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.143163+0000) 2020-02-05T23:29:58.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:29:58.142+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.143188+0000) 2020-02-05T23:29:58.145 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:29:58.143+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.143701+0000) 2020-02-05T23:29:58.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:29:58.142+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.143570+0000) 2020-02-05T23:29:58.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:29:58.143+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.144132+0000) 2020-02-05T23:29:58.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:29:58.143+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.144385+0000) 2020-02-05T23:29:58.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:58.149+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:58.150742+0000) 2020-02-05T23:29:58.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:29:58.155+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:29:59.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:29:59.561+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:29:59.561748+0000) 2020-02-05T23:30:00.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:00.127+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:00.127832+0000) 2020-02-05T23:30:00.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:00.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:00.168734+0000) 2020-02-05T23:30:00.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:00.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:00.407811+0000) 2020-02-05T23:30:02.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:02.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:02.423768+0000) 2020-02-05T23:30:02.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:02.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:02.774770+0000) 2020-02-05T23:30:03.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:03.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.061604+0000) 2020-02-05T23:30:03.161 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:03.159+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.160279+0000) 2020-02-05T23:30:03.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:03.160+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.161243+0000) 2020-02-05T23:30:03.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:03.160+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.160834+0000) 2020-02-05T23:30:03.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:03.160+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.162042+0000) 2020-02-05T23:30:03.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:03.161+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.163007+0000) 2020-02-05T23:30:03.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:03.164+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.165139+0000) 2020-02-05T23:30:03.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:03.166+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.166615+0000) 2020-02-05T23:30:03.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:03.165+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.166373+0000) 2020-02-05T23:30:03.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:03.167+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.167513+0000) 2020-02-05T23:30:03.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:03.167+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.167802+0000) 2020-02-05T23:30:03.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:03.167+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.167861+0000) 2020-02-05T23:30:03.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:03.166+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.167497+0000) 2020-02-05T23:30:03.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:03.169+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.170694+0000) 2020-02-05T23:30:03.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:03.169+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.170949+0000) 2020-02-05T23:30:03.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:03.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:03.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:03.936+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.936883+0000) 2020-02-05T23:30:03.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:03.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:03.937126+0000) 2020-02-05T23:30:06.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:06.037+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:06.037292+0000) 2020-02-05T23:30:07.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:07.169+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:07.169372+0000) 2020-02-05T23:30:07.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:07.176+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:07.176288+0000) 2020-02-05T23:30:07.177 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:07.176+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:07.176290+0000) 2020-02-05T23:30:07.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:07.176+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:07.176560+0000) 2020-02-05T23:30:08.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:08.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.185495+0000) 2020-02-05T23:30:08.187 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:08.186+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.186403+0000) 2020-02-05T23:30:08.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:08.186+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.186981+0000) 2020-02-05T23:30:08.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:08.187+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.187377+0000) 2020-02-05T23:30:08.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:08.186+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.187697+0000) 2020-02-05T23:30:08.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:08.187+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.188523+0000) 2020-02-05T23:30:08.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:08.188+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.188983+0000) 2020-02-05T23:30:08.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:08.189+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.189423+0000) 2020-02-05T23:30:08.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:08.188+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.189800+0000) 2020-02-05T23:30:08.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:08.188+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.190128+0000) 2020-02-05T23:30:08.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:08.189+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.190674+0000) 2020-02-05T23:30:08.192 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:08.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.190602+0000) 2020-02-05T23:30:08.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:08.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.195391+0000) 2020-02-05T23:30:08.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:08.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:08.195684+0000) 2020-02-05T23:30:08.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:08.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:09.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:09.561+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:09.562025+0000) 2020-02-05T23:30:10.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:10.136+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:10.136351+0000) 2020-02-05T23:30:10.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:10.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:10.168968+0000) 2020-02-05T23:30:10.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:10.406+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:10.408049+0000) 2020-02-05T23:30:12.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:12.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:12.424030+0000) 2020-02-05T23:30:12.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:12.773+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:12.775032+0000) 2020-02-05T23:30:13.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:13.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.061874+0000) 2020-02-05T23:30:13.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:13.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.206761+0000) 2020-02-05T23:30:13.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:13.207+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.208524+0000) 2020-02-05T23:30:13.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:13.208+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.209421+0000) 2020-02-05T23:30:13.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:13.208+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.209791+0000) 2020-02-05T23:30:13.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:13.208+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.209970+0000) 2020-02-05T23:30:13.212 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:13.209+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.210546+0000) 2020-02-05T23:30:13.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:13.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.211460+0000) 2020-02-05T23:30:13.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:13.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.211963+0000) 2020-02-05T23:30:13.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:13.210+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.212346+0000) 2020-02-05T23:30:13.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:13.211+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.212732+0000) 2020-02-05T23:30:13.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:13.211+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.212953+0000) 2020-02-05T23:30:13.214 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:13.211+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.213331+0000) 2020-02-05T23:30:13.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:13.211+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.213235+0000) 2020-02-05T23:30:13.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:13.218+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.219361+0000) 2020-02-05T23:30:13.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:13.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:13.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:13.936+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.937112+0000) 2020-02-05T23:30:13.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:13.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:13.937344+0000) 2020-02-05T23:30:16.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:16.036+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:16.037521+0000) 2020-02-05T23:30:17.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:17.168+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:17.169600+0000) 2020-02-05T23:30:17.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:17.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:17.176566+0000) 2020-02-05T23:30:17.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:17.175+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:17.176566+0000) 2020-02-05T23:30:17.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:17.175+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:17.176884+0000) 2020-02-05T23:30:18.230 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:18.228+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.229381+0000) 2020-02-05T23:30:18.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:18.228+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.229817+0000) 2020-02-05T23:30:18.231 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:18.228+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.230184+0000) 2020-02-05T23:30:18.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:18.229+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.230668+0000) 2020-02-05T23:30:18.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:18.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.232456+0000) 2020-02-05T23:30:18.233 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:18.231+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.232643+0000) 2020-02-05T23:30:18.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:18.231+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.232511+0000) 2020-02-05T23:30:18.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:18.231+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.232942+0000) 2020-02-05T23:30:18.235 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:18.232+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.233638+0000) 2020-02-05T23:30:18.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:18.233+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.234275+0000) 2020-02-05T23:30:18.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:18.233+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.234848+0000) 2020-02-05T23:30:18.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:18.233+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.234719+0000) 2020-02-05T23:30:18.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:18.234+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.235876+0000) 2020-02-05T23:30:18.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:18.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:18.239646+0000) 2020-02-05T23:30:18.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:18.244+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:19.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:19.561+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:19.562295+0000) 2020-02-05T23:30:20.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:20.143+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:20.144183+0000) 2020-02-05T23:30:20.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:20.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:20.169233+0000) 2020-02-05T23:30:20.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:20.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:20.408300+0000) 2020-02-05T23:30:21.208 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:30:21.215 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:30:22.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:22.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:22.424296+0000) 2020-02-05T23:30:22.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:22.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:22.775297+0000) 2020-02-05T23:30:23.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:23.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.062136+0000) 2020-02-05T23:30:23.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:23.250+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.251605+0000) 2020-02-05T23:30:23.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:23.251+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.252418+0000) 2020-02-05T23:30:23.254 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:23.251+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.253006+0000) 2020-02-05T23:30:23.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:23.252+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.253446+0000) 2020-02-05T23:30:23.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:23.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.253921+0000) 2020-02-05T23:30:23.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:23.252+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.254377+0000) 2020-02-05T23:30:23.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:23.254+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.255378+0000) 2020-02-05T23:30:23.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:23.254+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.256020+0000) 2020-02-05T23:30:23.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:23.254+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.255891+0000) 2020-02-05T23:30:23.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:23.255+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.256954+0000) 2020-02-05T23:30:23.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:23.256+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.257972+0000) 2020-02-05T23:30:23.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:23.256+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.257983+0000) 2020-02-05T23:30:23.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:23.257+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.258671+0000) 2020-02-05T23:30:23.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:23.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.260903+0000) 2020-02-05T23:30:23.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:23.268+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146543 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:23.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:23.936+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.937328+0000) 2020-02-05T23:30:23.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:23.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:23.937525+0000) 2020-02-05T23:30:26.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:26.036+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:26.037748+0000) 2020-02-05T23:30:27.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:27.169+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:27.169827+0000) 2020-02-05T23:30:27.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:27.176+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:27.176866+0000) 2020-02-05T23:30:27.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:27.176+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:27.176866+0000) 2020-02-05T23:30:27.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:27.176+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:27.177135+0000) 2020-02-05T23:30:28.277 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:28.274+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.275892+0000) 2020-02-05T23:30:28.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:28.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.276632+0000) 2020-02-05T23:30:28.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:28.276+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.277242+0000) 2020-02-05T23:30:28.278 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:28.276+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.277769+0000) 2020-02-05T23:30:28.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:28.277+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.278217+0000) 2020-02-05T23:30:28.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:28.276+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.278121+0000) 2020-02-05T23:30:28.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:28.277+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.278866+0000) 2020-02-05T23:30:28.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:28.277+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.278670+0000) 2020-02-05T23:30:28.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:28.277+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.279222+0000) 2020-02-05T23:30:28.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:28.277+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.278992+0000) 2020-02-05T23:30:28.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:28.278+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.279429+0000) 2020-02-05T23:30:28.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:28.278+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.279971+0000) 2020-02-05T23:30:28.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:28.279+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.280600+0000) 2020-02-05T23:30:28.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:28.290+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146585 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:28.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:28.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:28.285609+0000) 2020-02-05T23:30:29.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:29.561+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:29.562544+0000) 2020-02-05T23:30:30.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:30.151+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:30.152255+0000) 2020-02-05T23:30:30.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:30.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:30.169505+0000) 2020-02-05T23:30:30.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:30.407+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:30.408558+0000) 2020-02-05T23:30:32.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:32.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:32.424574+0000) 2020-02-05T23:30:32.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:32.774+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:32.775513+0000) 2020-02-05T23:30:33.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:33.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.062378+0000) 2020-02-05T23:30:33.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:33.298+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.300186+0000) 2020-02-05T23:30:33.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:33.299+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.300615+0000) 2020-02-05T23:30:33.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:33.300+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.301162+0000) 2020-02-05T23:30:33.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:33.300+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.301758+0000) 2020-02-05T23:30:33.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:33.300+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.301672+0000) 2020-02-05T23:30:33.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:33.300+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.302044+0000) 2020-02-05T23:30:33.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:33.301+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.302517+0000) 2020-02-05T23:30:33.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:33.301+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.302673+0000) 2020-02-05T23:30:33.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:33.302+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.304122+0000) 2020-02-05T23:30:33.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:33.302+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.304323+0000) 2020-02-05T23:30:33.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:33.303+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.304798+0000) 2020-02-05T23:30:33.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:33.304+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.305269+0000) 2020-02-05T23:30:33.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:33.304+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.306075+0000) 2020-02-05T23:30:33.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:33.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.309798+0000) 2020-02-05T23:30:33.316 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:33.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:33.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:33.936+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.937535+0000) 2020-02-05T23:30:33.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:33.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:33.937718+0000) 2020-02-05T23:30:36.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:36.037+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:36.037998+0000) 2020-02-05T23:30:37.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:37.169+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:37.170056+0000) 2020-02-05T23:30:37.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:37.175+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:37.177165+0000) 2020-02-05T23:30:37.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:37.175+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:37.177194+0000) 2020-02-05T23:30:37.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:37.175+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:37.177381+0000) 2020-02-05T23:30:38.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:38.319+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.320804+0000) 2020-02-05T23:30:38.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:38.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.321696+0000) 2020-02-05T23:30:38.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:38.321+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.322517+0000) 2020-02-05T23:30:38.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:38.322+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.323198+0000) 2020-02-05T23:30:38.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:38.322+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.323894+0000) 2020-02-05T23:30:38.326 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:38.323+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.325038+0000) 2020-02-05T23:30:38.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:38.324+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.325431+0000) 2020-02-05T23:30:38.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:38.324+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.325696+0000) 2020-02-05T23:30:38.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:38.325+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.326471+0000) 2020-02-05T23:30:38.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:38.325+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.326625+0000) 2020-02-05T23:30:38.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:38.325+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.326471+0000) 2020-02-05T23:30:38.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:38.326+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.327145+0000) 2020-02-05T23:30:38.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:38.326+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.327717+0000) 2020-02-05T23:30:38.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:38.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:38.330603+0000) 2020-02-05T23:30:38.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:38.336+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:39.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:39.562+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:39.562771+0000) 2020-02-05T23:30:40.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:40.159+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:40.160456+0000) 2020-02-05T23:30:40.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:40.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:40.169728+0000) 2020-02-05T23:30:40.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:40.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:40.408817+0000) 2020-02-05T23:30:42.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:42.424+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:42.424831+0000) 2020-02-05T23:30:42.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:42.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:42.775742+0000) 2020-02-05T23:30:43.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:43.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.062613+0000) 2020-02-05T23:30:43.345 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:43.342+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.343923+0000) 2020-02-05T23:30:43.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:43.342+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.343868+0000) 2020-02-05T23:30:43.346 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:43.343+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.344441+0000) 2020-02-05T23:30:43.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:43.343+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.344838+0000) 2020-02-05T23:30:43.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:43.344+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.345578+0000) 2020-02-05T23:30:43.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:43.344+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.346218+0000) 2020-02-05T23:30:43.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:43.345+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.346846+0000) 2020-02-05T23:30:43.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:43.346+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.347642+0000) 2020-02-05T23:30:43.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:43.346+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.348104+0000) 2020-02-05T23:30:43.350 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:43.347+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.348813+0000) 2020-02-05T23:30:43.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:43.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.349828+0000) 2020-02-05T23:30:43.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:43.349+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.350476+0000) 2020-02-05T23:30:43.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:43.349+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.350577+0000) 2020-02-05T23:30:43.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:43.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.352524+0000) 2020-02-05T23:30:43.363 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:43.360+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:43.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:43.937+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.937745+0000) 2020-02-05T23:30:43.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:43.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:43.937937+0000) 2020-02-05T23:30:46.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:46.037+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:46.038242+0000) 2020-02-05T23:30:47.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:47.169+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:47.170346+0000) 2020-02-05T23:30:47.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:47.176+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:47.177476+0000) 2020-02-05T23:30:47.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:47.176+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:47.177495+0000) 2020-02-05T23:30:47.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:47.176+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:47.177663+0000) 2020-02-05T23:30:48.368 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:48.366+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.367575+0000) 2020-02-05T23:30:48.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:48.366+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.367901+0000) 2020-02-05T23:30:48.369 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:48.367+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.368313+0000) 2020-02-05T23:30:48.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:48.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.368773+0000) 2020-02-05T23:30:48.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:48.368+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.369887+0000) 2020-02-05T23:30:48.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:48.369+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.370378+0000) 2020-02-05T23:30:48.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:48.369+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.370931+0000) 2020-02-05T23:30:48.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:48.370+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.371343+0000) 2020-02-05T23:30:48.372 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:48.370+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.371682+0000) 2020-02-05T23:30:48.374 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:48.371+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.372753+0000) 2020-02-05T23:30:48.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:48.371+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.372739+0000) 2020-02-05T23:30:48.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:48.371+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.373015+0000) 2020-02-05T23:30:48.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:48.372+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.373650+0000) 2020-02-05T23:30:48.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:48.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:48.378742+0000) 2020-02-05T23:30:48.385 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:48.383+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:49.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:49.562+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:49.562998+0000) 2020-02-05T23:30:50.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:50.168+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:50.168402+0000) 2020-02-05T23:30:50.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:50.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:50.169926+0000) 2020-02-05T23:30:50.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:50.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:50.409058+0000) 2020-02-05T23:30:51.424 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:30:51.430 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:30:52.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:52.423+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:52.425075+0000) 2020-02-05T23:30:52.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:52.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:52.775972+0000) 2020-02-05T23:30:53.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:53.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.062845+0000) 2020-02-05T23:30:53.390 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:53.387+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.388941+0000) 2020-02-05T23:30:53.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:53.388+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.389869+0000) 2020-02-05T23:30:53.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:53.389+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.390644+0000) 2020-02-05T23:30:53.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:53.391+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.392298+0000) 2020-02-05T23:30:53.393 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:53.391+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.392570+0000) 2020-02-05T23:30:53.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:53.391+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.392934+0000) 2020-02-05T23:30:53.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:53.391+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.392794+0000) 2020-02-05T23:30:53.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:53.392+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.393250+0000) 2020-02-05T23:30:53.395 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:53.392+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.393691+0000) 2020-02-05T23:30:53.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:53.393+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.394724+0000) 2020-02-05T23:30:53.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:53.393+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.395025+0000) 2020-02-05T23:30:53.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:53.394+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.395771+0000) 2020-02-05T23:30:53.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:53.398+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.399791+0000) 2020-02-05T23:30:53.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:53.399+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.400818+0000) 2020-02-05T23:30:53.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:53.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146794 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:53.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:53.937+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.938006+0000) 2020-02-05T23:30:53.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:53.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:53.938160+0000) 2020-02-05T23:30:56.039 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:56.037+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:56.038501+0000) 2020-02-05T23:30:57.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:57.170+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:57.170570+0000) 2020-02-05T23:30:57.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:57.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:57.177752+0000) 2020-02-05T23:30:57.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:57.177+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:57.177758+0000) 2020-02-05T23:30:57.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:57.177+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:57.177931+0000) 2020-02-05T23:30:58.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:30:58.410+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.411309+0000) 2020-02-05T23:30:58.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:58.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.412361+0000) 2020-02-05T23:30:58.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:30:58.411+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.412685+0000) 2020-02-05T23:30:58.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:30:58.412+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.413147+0000) 2020-02-05T23:30:58.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:30:58.412+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.413586+0000) 2020-02-05T23:30:58.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:30:58.413+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.414614+0000) 2020-02-05T23:30:58.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:30:58.413+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.415079+0000) 2020-02-05T23:30:58.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:30:58.414+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.415217+0000) 2020-02-05T23:30:58.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:30:58.415+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.415586+0000) 2020-02-05T23:30:58.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:30:58.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.416416+0000) 2020-02-05T23:30:58.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:30:58.415+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.416108+0000) 2020-02-05T23:30:58.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:30:58.415+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.416986+0000) 2020-02-05T23:30:58.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:58.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.417427+0000) 2020-02-05T23:30:58.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:58.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:58.423650+0000) 2020-02-05T23:30:58.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:30:58.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:30:59.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:30:59.563+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:30:59.563281+0000) 2020-02-05T23:31:00.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:00.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:00.170209+0000) 2020-02-05T23:31:00.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:00.176+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:00.176623+0000) 2020-02-05T23:31:00.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:00.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:00.409310+0000) 2020-02-05T23:31:02.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:02.424+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:02.425346+0000) 2020-02-05T23:31:02.777 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:02.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:02.776283+0000) 2020-02-05T23:31:03.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:03.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.063118+0000) 2020-02-05T23:31:03.436 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:03.434+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.434421+0000) 2020-02-05T23:31:03.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:03.435+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.435276+0000) 2020-02-05T23:31:03.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:03.436+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.436575+0000) 2020-02-05T23:31:03.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:03.437+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.437242+0000) 2020-02-05T23:31:03.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:03.436+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.437007+0000) 2020-02-05T23:31:03.439 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:03.437+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.437631+0000) 2020-02-05T23:31:03.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:03.437+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.437991+0000) 2020-02-05T23:31:03.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:03.438+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.438813+0000) 2020-02-05T23:31:03.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:03.438+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.439101+0000) 2020-02-05T23:31:03.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:03.439+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.439774+0000) 2020-02-05T23:31:03.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:03.440+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.440309+0000) 2020-02-05T23:31:03.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:03.439+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.440175+0000) 2020-02-05T23:31:03.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:03.440+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.440614+0000) 2020-02-05T23:31:03.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:03.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.445530+0000) 2020-02-05T23:31:03.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:03.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:03.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:03.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.938223+0000) 2020-02-05T23:31:03.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:03.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:03.938376+0000) 2020-02-05T23:31:06.040 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:06.038+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:06.038741+0000) 2020-02-05T23:31:07.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:07.169+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:07.170838+0000) 2020-02-05T23:31:07.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:07.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:07.177943+0000) 2020-02-05T23:31:07.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:07.177+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:07.177969+0000) 2020-02-05T23:31:07.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:07.177+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:07.178158+0000) 2020-02-05T23:31:08.456 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:08.454+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.455466+0000) 2020-02-05T23:31:08.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:08.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.456455+0000) 2020-02-05T23:31:08.458 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:08.456+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.457232+0000) 2020-02-05T23:31:08.458 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:08.457+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.457632+0000) 2020-02-05T23:31:08.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:08.457+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.458145+0000) 2020-02-05T23:31:08.459 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:08.457+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.458365+0000) 2020-02-05T23:31:08.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:08.457+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.458421+0000) 2020-02-05T23:31:08.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:08.457+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.458908+0000) 2020-02-05T23:31:08.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:08.458+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.458587+0000) 2020-02-05T23:31:08.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:08.458+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.459260+0000) 2020-02-05T23:31:08.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:08.458+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.459273+0000) 2020-02-05T23:31:08.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:08.459+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.460155+0000) 2020-02-05T23:31:08.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:08.459+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.460307+0000) 2020-02-05T23:31:08.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:08.463+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:08.465006+0000) 2020-02-05T23:31:08.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:08.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:09.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:09.562+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:09.563515+0000) 2020-02-05T23:31:10.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:10.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:10.170445+0000) 2020-02-05T23:31:10.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:10.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:10.184912+0000) 2020-02-05T23:31:10.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:10.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:10.409536+0000) 2020-02-05T23:31:12.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:12.425+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:12.425643+0000) 2020-02-05T23:31:12.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:12.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:12.776522+0000) 2020-02-05T23:31:13.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:13.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.063351+0000) 2020-02-05T23:31:13.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:13.475+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.476297+0000) 2020-02-05T23:31:13.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:13.478+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.479255+0000) 2020-02-05T23:31:13.480 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:13.478+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.479448+0000) 2020-02-05T23:31:13.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:13.478+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.479963+0000) 2020-02-05T23:31:13.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:13.479+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.480155+0000) 2020-02-05T23:31:13.482 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:13.480+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.480580+0000) 2020-02-05T23:31:13.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:13.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.480864+0000) 2020-02-05T23:31:13.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:13.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.481466+0000) 2020-02-05T23:31:13.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:13.482+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.482470+0000) 2020-02-05T23:31:13.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:13.482+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.482607+0000) 2020-02-05T23:31:13.484 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:13.481+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.482081+0000) 2020-02-05T23:31:13.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:13.482+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.483086+0000) 2020-02-05T23:31:13.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:13.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.487294+0000) 2020-02-05T23:31:13.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:13.484+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.484819+0000) 2020-02-05T23:31:13.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:13.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 146962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:13.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:13.937+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.938550+0000) 2020-02-05T23:31:13.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:13.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:13.938564+0000) 2020-02-05T23:31:16.040 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:16.038+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:16.039032+0000) 2020-02-05T23:31:17.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:17.170+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:17.171127+0000) 2020-02-05T23:31:17.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:17.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:17.178182+0000) 2020-02-05T23:31:17.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:17.177+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:17.178249+0000) 2020-02-05T23:31:17.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:17.177+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:17.178392+0000) 2020-02-05T23:31:18.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:18.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.499709+0000) 2020-02-05T23:31:18.501 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:18.499+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.499503+0000) 2020-02-05T23:31:18.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:18.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.501114+0000) 2020-02-05T23:31:18.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:18.501+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.502153+0000) 2020-02-05T23:31:18.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:18.501+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.502546+0000) 2020-02-05T23:31:18.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:18.501+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.503053+0000) 2020-02-05T23:31:18.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:18.502+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.503826+0000) 2020-02-05T23:31:18.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:18.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.504688+0000) 2020-02-05T23:31:18.506 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:18.504+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.505080+0000) 2020-02-05T23:31:18.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:18.504+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.505273+0000) 2020-02-05T23:31:18.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:18.505+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.505581+0000) 2020-02-05T23:31:18.507 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:18.505+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.505968+0000) 2020-02-05T23:31:18.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:18.506+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.506723+0000) 2020-02-05T23:31:18.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:18.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:18.509136+0000) 2020-02-05T23:31:18.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:18.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147004 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:19.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:19.562+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:19.563774+0000) 2020-02-05T23:31:20.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:20.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:20.170718+0000) 2020-02-05T23:31:20.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:20.191+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:20.192726+0000) 2020-02-05T23:31:20.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:20.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:20.409803+0000) 2020-02-05T23:31:21.577 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:31:21.583 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:31:22.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:22.425+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:22.425892+0000) 2020-02-05T23:31:22.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:22.775+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:22.776784+0000) 2020-02-05T23:31:23.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:23.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.063581+0000) 2020-02-05T23:31:23.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:23.521+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.522750+0000) 2020-02-05T23:31:23.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:23.522+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.523927+0000) 2020-02-05T23:31:23.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:23.523+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.524294+0000) 2020-02-05T23:31:23.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:23.523+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.523541+0000) 2020-02-05T23:31:23.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:23.523+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.524764+0000) 2020-02-05T23:31:23.526 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:23.524+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.525209+0000) 2020-02-05T23:31:23.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:23.525+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.526024+0000) 2020-02-05T23:31:23.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:23.525+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.526222+0000) 2020-02-05T23:31:23.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:23.526+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.527465+0000) 2020-02-05T23:31:23.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:23.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.528196+0000) 2020-02-05T23:31:23.529 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:23.526+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.527033+0000) 2020-02-05T23:31:23.530 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:23.527+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.527567+0000) 2020-02-05T23:31:23.530 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:23.527+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.528215+0000) 2020-02-05T23:31:23.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:23.532+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.533129+0000) 2020-02-05T23:31:23.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:23.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:23.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:23.937+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.938746+0000) 2020-02-05T23:31:23.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:23.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:23.938730+0000) 2020-02-05T23:31:26.040 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:26.039+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:26.039267+0000) 2020-02-05T23:31:27.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:27.170+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:27.171314+0000) 2020-02-05T23:31:27.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:27.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:27.178473+0000) 2020-02-05T23:31:27.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:27.178+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:27.178515+0000) 2020-02-05T23:31:27.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:27.178+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:27.178723+0000) 2020-02-05T23:31:28.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:28.544+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.545454+0000) 2020-02-05T23:31:28.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:28.547+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.547512+0000) 2020-02-05T23:31:28.549 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:28.548+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.548535+0000) 2020-02-05T23:31:28.550 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:28.547+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.548779+0000) 2020-02-05T23:31:28.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:28.548+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.548487+0000) 2020-02-05T23:31:28.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:28.548+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.549501+0000) 2020-02-05T23:31:28.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:28.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.550045+0000) 2020-02-05T23:31:28.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:28.549+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.550507+0000) 2020-02-05T23:31:28.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:28.550+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.550793+0000) 2020-02-05T23:31:28.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:28.550+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.551374+0000) 2020-02-05T23:31:28.552 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:28.551+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.551850+0000) 2020-02-05T23:31:28.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:28.552+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.552671+0000) 2020-02-05T23:31:28.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:28.551+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.552536+0000) 2020-02-05T23:31:28.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:28.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:28.556929+0000) 2020-02-05T23:31:28.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:28.562+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:29.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:29.563+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:29.564010+0000) 2020-02-05T23:31:30.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:30.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:30.170954+0000) 2020-02-05T23:31:30.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:30.199+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:30.201027+0000) 2020-02-05T23:31:30.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:30.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:30.410063+0000) 2020-02-05T23:31:32.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:32.426+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:32.426163+0000) 2020-02-05T23:31:32.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:32.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:32.776970+0000) 2020-02-05T23:31:33.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:33.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.063831+0000) 2020-02-05T23:31:33.570 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:33.567+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.568903+0000) 2020-02-05T23:31:33.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:33.569+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.570428+0000) 2020-02-05T23:31:33.572 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:33.570+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.571069+0000) 2020-02-05T23:31:33.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:33.570+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.571137+0000) 2020-02-05T23:31:33.572 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:33.570+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.571500+0000) 2020-02-05T23:31:33.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:33.571+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.572582+0000) 2020-02-05T23:31:33.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:33.572+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.572473+0000) 2020-02-05T23:31:33.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:33.572+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.573253+0000) 2020-02-05T23:31:33.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:33.573+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.574252+0000) 2020-02-05T23:31:33.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:33.574+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.574930+0000) 2020-02-05T23:31:33.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:33.574+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.575054+0000) 2020-02-05T23:31:33.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:33.574+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.575169+0000) 2020-02-05T23:31:33.577 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:33.573+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.574535+0000) 2020-02-05T23:31:33.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:33.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.579245+0000) 2020-02-05T23:31:33.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:33.585+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:33.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:33.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.938952+0000) 2020-02-05T23:31:33.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:33.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:33.938946+0000) 2020-02-05T23:31:36.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:36.039+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:36.039519+0000) 2020-02-05T23:31:37.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:37.171+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:37.171545+0000) 2020-02-05T23:31:37.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:37.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:37.178751+0000) 2020-02-05T23:31:37.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:37.177+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:37.178761+0000) 2020-02-05T23:31:37.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:37.177+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:37.178968+0000) 2020-02-05T23:31:38.592 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:38.590+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.591541+0000) 2020-02-05T23:31:38.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:38.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.592013+0000) 2020-02-05T23:31:38.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:38.592+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.593985+0000) 2020-02-05T23:31:38.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:38.593+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.594782+0000) 2020-02-05T23:31:38.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:38.593+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.595371+0000) 2020-02-05T23:31:38.596 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:38.594+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.595755+0000) 2020-02-05T23:31:38.597 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:38.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.596139+0000) 2020-02-05T23:31:38.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:38.595+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.596564+0000) 2020-02-05T23:31:38.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:38.595+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.596789+0000) 2020-02-05T23:31:38.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:38.596+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.597602+0000) 2020-02-05T23:31:38.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:38.596+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.597777+0000) 2020-02-05T23:31:38.599 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:38.596+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.597874+0000) 2020-02-05T23:31:38.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:38.597+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.598579+0000) 2020-02-05T23:31:38.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:38.604+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:38.605562+0000) 2020-02-05T23:31:38.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:38.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:39.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:39.563+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:39.564282+0000) 2020-02-05T23:31:40.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:40.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:40.171258+0000) 2020-02-05T23:31:40.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:40.210+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:40.211962+0000) 2020-02-05T23:31:40.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:40.408+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:40.410304+0000) 2020-02-05T23:31:42.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:42.425+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:42.426453+0000) 2020-02-05T23:31:42.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:42.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:42.777191+0000) 2020-02-05T23:31:43.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:43.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.064062+0000) 2020-02-05T23:31:43.615 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:43.613+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.614432+0000) 2020-02-05T23:31:43.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:43.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.615234+0000) 2020-02-05T23:31:43.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:43.614+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.615963+0000) 2020-02-05T23:31:43.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:43.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.616867+0000) 2020-02-05T23:31:43.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:43.615+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.616571+0000) 2020-02-05T23:31:43.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:43.616+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.617295+0000) 2020-02-05T23:31:43.619 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:43.616+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.617961+0000) 2020-02-05T23:31:43.620 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:43.617+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.618590+0000) 2020-02-05T23:31:43.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:43.616+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.618390+0000) 2020-02-05T23:31:43.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:43.617+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.619267+0000) 2020-02-05T23:31:43.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:43.618+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.619489+0000) 2020-02-05T23:31:43.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:43.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.619581+0000) 2020-02-05T23:31:43.622 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:43.618+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.620301+0000) 2020-02-05T23:31:43.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:43.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.623366+0000) 2020-02-05T23:31:43.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:43.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147210 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:43.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:43.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.939163+0000) 2020-02-05T23:31:43.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:43.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:43.939170+0000) 2020-02-05T23:31:46.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:46.038+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:46.039775+0000) 2020-02-05T23:31:47.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:47.171+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:47.171776+0000) 2020-02-05T23:31:47.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:47.177+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:47.178998+0000) 2020-02-05T23:31:47.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:47.177+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:47.178995+0000) 2020-02-05T23:31:47.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:47.177+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:47.179179+0000) 2020-02-05T23:31:48.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:48.637+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.638654+0000) 2020-02-05T23:31:48.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:48.638+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.639144+0000) 2020-02-05T23:31:48.641 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:48.639+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.640167+0000) 2020-02-05T23:31:48.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:48.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.641436+0000) 2020-02-05T23:31:48.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:48.639+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.641340+0000) 2020-02-05T23:31:48.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:48.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.641888+0000) 2020-02-05T23:31:48.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:48.640+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.642022+0000) 2020-02-05T23:31:48.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:48.641+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.642647+0000) 2020-02-05T23:31:48.644 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:48.641+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.643135+0000) 2020-02-05T23:31:48.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:48.642+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.643576+0000) 2020-02-05T23:31:48.645 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:48.642+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.643994+0000) 2020-02-05T23:31:48.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:48.643+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.644311+0000) 2020-02-05T23:31:48.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:48.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.644671+0000) 2020-02-05T23:31:48.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:48.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:48.648018+0000) 2020-02-05T23:31:48.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:48.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147250 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:49.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:49.564+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:49.564516+0000) 2020-02-05T23:31:50.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:50.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:50.171489+0000) 2020-02-05T23:31:50.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:50.219+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:50.220179+0000) 2020-02-05T23:31:50.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:50.409+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:50.410556+0000) 2020-02-05T23:31:51.765 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:31:51.771 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:31:52.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:52.425+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:52.426723+0000) 2020-02-05T23:31:52.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:52.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:52.777415+0000) 2020-02-05T23:31:53.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:53.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.064319+0000) 2020-02-05T23:31:53.661 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:53.658+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.660039+0000) 2020-02-05T23:31:53.661 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:53.659+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.660673+0000) 2020-02-05T23:31:53.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:53.660+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.661316+0000) 2020-02-05T23:31:53.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:53.660+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.661559+0000) 2020-02-05T23:31:53.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:53.660+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.661819+0000) 2020-02-05T23:31:53.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:53.661+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.662321+0000) 2020-02-05T23:31:53.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:53.661+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.662924+0000) 2020-02-05T23:31:53.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:53.662+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.663853+0000) 2020-02-05T23:31:53.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:53.663+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.664958+0000) 2020-02-05T23:31:53.670 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:53.663+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.664909+0000) 2020-02-05T23:31:53.670 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:53.663+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.665347+0000) 2020-02-05T23:31:53.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:53.664+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.665901+0000) 2020-02-05T23:31:53.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:53.668+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.669193+0000) 2020-02-05T23:31:53.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:53.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.670235+0000) 2020-02-05T23:31:53.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:53.676+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:53.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:53.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.939330+0000) 2020-02-05T23:31:53.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:53.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:53.939363+0000) 2020-02-05T23:31:56.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:56.038+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:56.040030+0000) 2020-02-05T23:31:57.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:57.171+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:57.171998+0000) 2020-02-05T23:31:57.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:57.178+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:57.179234+0000) 2020-02-05T23:31:57.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:57.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:57.179327+0000) 2020-02-05T23:31:57.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:57.178+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:57.179418+0000) 2020-02-05T23:31:58.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:31:58.683+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.683944+0000) 2020-02-05T23:31:58.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:31:58.683+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.684894+0000) 2020-02-05T23:31:58.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:58.685+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.685371+0000) 2020-02-05T23:31:58.690 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:31:58.685+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.687192+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:31:58.686+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.688044+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:31:58.687+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.688487+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:31:58.686+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.688251+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:31:58.687+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.689293+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:31:58.686+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.686176+0000) 2020-02-05T23:31:58.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:31:58.686+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.686600+0000) 2020-02-05T23:31:58.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:31:58.687+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.687326+0000) 2020-02-05T23:31:58.692 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:31:58.687+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.687401+0000) 2020-02-05T23:31:58.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:58.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.692372+0000) 2020-02-05T23:31:58.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:58.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:58.692649+0000) 2020-02-05T23:31:58.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:31:58.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:31:59.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:31:59.564+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:31:59.564721+0000) 2020-02-05T23:32:00.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:00.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:00.171717+0000) 2020-02-05T23:32:00.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:00.227+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:00.228349+0000) 2020-02-05T23:32:00.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:00.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:00.410805+0000) 2020-02-05T23:32:02.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:02.426+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:02.426995+0000) 2020-02-05T23:32:02.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:02.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:02.777647+0000) 2020-02-05T23:32:03.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:03.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.064548+0000) 2020-02-05T23:32:03.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:03.705+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.707167+0000) 2020-02-05T23:32:03.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:03.707+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.709044+0000) 2020-02-05T23:32:03.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:03.708+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.709691+0000) 2020-02-05T23:32:03.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:03.709+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.710151+0000) 2020-02-05T23:32:03.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:03.709+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.710694+0000) 2020-02-05T23:32:03.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:03.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.711264+0000) 2020-02-05T23:32:03.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:03.709+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.711347+0000) 2020-02-05T23:32:03.713 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:03.711+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.712263+0000) 2020-02-05T23:32:03.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:03.711+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.712666+0000) 2020-02-05T23:32:03.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:03.712+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.713203+0000) 2020-02-05T23:32:03.715 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:03.711+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.713375+0000) 2020-02-05T23:32:03.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:03.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.713868+0000) 2020-02-05T23:32:03.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:03.712+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.714082+0000) 2020-02-05T23:32:03.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:03.716+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.717153+0000) 2020-02-05T23:32:03.724 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:03.722+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:03.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:03.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.939571+0000) 2020-02-05T23:32:03.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:03.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:03.939570+0000) 2020-02-05T23:32:06.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:06.038+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:06.040304+0000) 2020-02-05T23:32:07.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:07.171+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:07.172291+0000) 2020-02-05T23:32:07.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:07.178+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:07.179480+0000) 2020-02-05T23:32:07.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:07.178+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:07.179722+0000) 2020-02-05T23:32:07.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:07.178+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:07.179661+0000) 2020-02-05T23:32:08.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:08.729+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.730402+0000) 2020-02-05T23:32:08.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:08.729+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.730963+0000) 2020-02-05T23:32:08.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:08.730+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.731649+0000) 2020-02-05T23:32:08.733 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:08.730+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.731857+0000) 2020-02-05T23:32:08.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:08.731+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.732858+0000) 2020-02-05T23:32:08.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:08.731+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.732945+0000) 2020-02-05T23:32:08.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:08.732+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.733253+0000) 2020-02-05T23:32:08.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:08.732+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.733564+0000) 2020-02-05T23:32:08.736 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:08.732+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.734264+0000) 2020-02-05T23:32:08.737 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:08.733+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.734835+0000) 2020-02-05T23:32:08.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:08.733+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.734813+0000) 2020-02-05T23:32:08.738 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:08.734+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.735550+0000) 2020-02-05T23:32:08.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:08.734+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.735754+0000) 2020-02-05T23:32:08.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:08.735+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:08.736553+0000) 2020-02-05T23:32:08.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:08.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147416 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:09.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:09.563+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:09.564944+0000) 2020-02-05T23:32:10.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:10.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:10.171975+0000) 2020-02-05T23:32:10.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:10.237+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:10.238364+0000) 2020-02-05T23:32:10.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:10.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:10.411035+0000) 2020-02-05T23:32:12.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:12.425+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:12.427231+0000) 2020-02-05T23:32:12.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:12.776+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:12.777897+0000) 2020-02-05T23:32:13.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:13.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.064799+0000) 2020-02-05T23:32:13.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:13.751+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.752828+0000) 2020-02-05T23:32:13.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:13.752+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.753934+0000) 2020-02-05T23:32:13.755 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:13.752+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.753950+0000) 2020-02-05T23:32:13.756 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:13.753+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.755184+0000) 2020-02-05T23:32:13.756 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:13.754+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.755715+0000) 2020-02-05T23:32:13.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:13.754+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.755907+0000) 2020-02-05T23:32:13.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:13.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.756408+0000) 2020-02-05T23:32:13.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:13.755+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.756678+0000) 2020-02-05T23:32:13.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:13.756+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.757868+0000) 2020-02-05T23:32:13.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:13.757+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.758142+0000) 2020-02-05T23:32:13.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:13.757+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.758816+0000) 2020-02-05T23:32:13.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:13.757+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.759246+0000) 2020-02-05T23:32:13.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:13.758+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.759195+0000) 2020-02-05T23:32:13.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:13.762+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.763508+0000) 2020-02-05T23:32:13.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:13.768+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:13.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:13.938+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.939839+0000) 2020-02-05T23:32:13.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:13.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:13.939825+0000) 2020-02-05T23:32:16.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:16.039+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:16.040626+0000) 2020-02-05T23:32:17.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:17.171+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:17.172517+0000) 2020-02-05T23:32:17.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:17.179+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:17.179733+0000) 2020-02-05T23:32:17.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:17.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:17.179918+0000) 2020-02-05T23:32:17.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:17.179+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:17.179943+0000) 2020-02-05T23:32:18.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:18.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.775230+0000) 2020-02-05T23:32:18.777 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:18.774+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.775661+0000) 2020-02-05T23:32:18.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:18.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.777066+0000) 2020-02-05T23:32:18.778 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:18.776+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.777591+0000) 2020-02-05T23:32:18.778 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:18.775+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.777161+0000) 2020-02-05T23:32:18.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:18.776+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.778322+0000) 2020-02-05T23:32:18.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:18.777+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.778896+0000) 2020-02-05T23:32:18.780 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:18.777+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.779253+0000) 2020-02-05T23:32:18.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:18.778+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.779943+0000) 2020-02-05T23:32:18.781 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:18.779+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.780329+0000) 2020-02-05T23:32:18.781 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:18.779+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.780853+0000) 2020-02-05T23:32:18.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:18.780+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.781292+0000) 2020-02-05T23:32:18.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:18.780+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.781683+0000) 2020-02-05T23:32:18.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:18.782+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:18.783465+0000) 2020-02-05T23:32:18.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:18.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:19.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:19.564+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:19.565152+0000) 2020-02-05T23:32:20.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:20.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:20.172189+0000) 2020-02-05T23:32:20.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:20.250+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:20.250863+0000) 2020-02-05T23:32:20.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:20.410+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:20.411271+0000) 2020-02-05T23:32:21.970 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:32:21.976 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:32:22.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:22.426+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:22.427517+0000) 2020-02-05T23:32:22.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:22.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:22.778136+0000) 2020-02-05T23:32:23.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:23.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.065030+0000) 2020-02-05T23:32:23.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:23.797+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.798568+0000) 2020-02-05T23:32:23.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:23.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.799109+0000) 2020-02-05T23:32:23.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:23.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.799928+0000) 2020-02-05T23:32:23.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:23.799+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.800833+0000) 2020-02-05T23:32:23.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:23.799+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.801350+0000) 2020-02-05T23:32:23.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:23.800+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.801764+0000) 2020-02-05T23:32:23.803 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:23.800+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.801776+0000) 2020-02-05T23:32:23.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:23.800+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.802405+0000) 2020-02-05T23:32:23.804 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:23.801+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.802613+0000) 2020-02-05T23:32:23.804 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:23.801+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.802815+0000) 2020-02-05T23:32:23.805 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:23.801+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.803240+0000) 2020-02-05T23:32:23.805 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:23.802+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.803857+0000) 2020-02-05T23:32:23.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:23.802+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.803871+0000) 2020-02-05T23:32:23.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:23.805+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.806312+0000) 2020-02-05T23:32:23.815 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:23.813+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:23.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:23.939+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.940123+0000) 2020-02-05T23:32:23.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:23.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:23.940136+0000) 2020-02-05T23:32:26.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:26.040+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:26.040890+0000) 2020-02-05T23:32:27.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:27.172+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:27.172755+0000) 2020-02-05T23:32:27.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:27.179+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:27.179982+0000) 2020-02-05T23:32:27.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:27.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:27.180197+0000) 2020-02-05T23:32:27.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:27.179+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:27.180183+0000) 2020-02-05T23:32:28.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:28.820+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.821078+0000) 2020-02-05T23:32:28.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:28.820+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.821570+0000) 2020-02-05T23:32:28.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:28.820+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.821757+0000) 2020-02-05T23:32:28.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:28.821+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.821949+0000) 2020-02-05T23:32:28.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:28.822+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.822770+0000) 2020-02-05T23:32:28.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:28.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.822515+0000) 2020-02-05T23:32:28.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:28.821+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.822852+0000) 2020-02-05T23:32:28.825 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:28.822+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.822983+0000) 2020-02-05T23:32:28.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:28.822+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.823346+0000) 2020-02-05T23:32:28.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:28.822+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.823812+0000) 2020-02-05T23:32:28.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:28.823+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.824212+0000) 2020-02-05T23:32:28.826 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:28.824+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.825157+0000) 2020-02-05T23:32:28.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:28.824+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.825206+0000) 2020-02-05T23:32:28.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:28.828+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:28.829526+0000) 2020-02-05T23:32:28.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:28.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:29.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:29.564+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:29.565359+0000) 2020-02-05T23:32:30.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:30.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:30.172412+0000) 2020-02-05T23:32:30.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:30.259+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:30.260542+0000) 2020-02-05T23:32:30.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:30.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:30.411525+0000) 2020-02-05T23:32:32.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:32.427+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:32.427794+0000) 2020-02-05T23:32:32.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:32.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:32.778368+0000) 2020-02-05T23:32:33.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:33.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.065254+0000) 2020-02-05T23:32:33.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:33.840+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.841948+0000) 2020-02-05T23:32:33.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:33.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.842771+0000) 2020-02-05T23:32:33.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:33.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.843293+0000) 2020-02-05T23:32:33.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:33.842+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.844012+0000) 2020-02-05T23:32:33.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:33.843+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.843874+0000) 2020-02-05T23:32:33.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:33.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.844545+0000) 2020-02-05T23:32:33.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:33.844+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.845211+0000) 2020-02-05T23:32:33.847 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:33.845+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.845505+0000) 2020-02-05T23:32:33.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:33.845+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.845802+0000) 2020-02-05T23:32:33.847 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:33.844+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.845219+0000) 2020-02-05T23:32:33.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:33.845+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.846137+0000) 2020-02-05T23:32:33.848 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:33.844+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.845658+0000) 2020-02-05T23:32:33.848 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:33.846+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.846649+0000) 2020-02-05T23:32:33.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:33.851+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.852792+0000) 2020-02-05T23:32:33.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:33.857+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:33.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:33.939+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.940352+0000) 2020-02-05T23:32:33.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:33.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:33.940355+0000) 2020-02-05T23:32:36.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:36.040+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:36.041128+0000) 2020-02-05T23:32:37.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:37.172+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:37.172998+0000) 2020-02-05T23:32:37.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:37.179+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:37.180232+0000) 2020-02-05T23:32:37.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:37.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:37.180492+0000) 2020-02-05T23:32:37.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:37.180+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:37.180406+0000) 2020-02-05T23:32:38.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:38.863+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.864044+0000) 2020-02-05T23:32:38.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:38.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.865036+0000) 2020-02-05T23:32:38.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:38.864+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.865468+0000) 2020-02-05T23:32:38.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:38.864+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.865930+0000) 2020-02-05T23:32:38.867 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:38.866+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.866619+0000) 2020-02-05T23:32:38.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:38.866+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.867140+0000) 2020-02-05T23:32:38.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:38.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.867603+0000) 2020-02-05T23:32:38.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:38.867+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.868232+0000) 2020-02-05T23:32:38.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:38.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.868948+0000) 2020-02-05T23:32:38.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:38.868+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.869534+0000) 2020-02-05T23:32:38.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:38.869+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.869840+0000) 2020-02-05T23:32:38.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:38.869+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.870167+0000) 2020-02-05T23:32:38.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:38.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.871247+0000) 2020-02-05T23:32:38.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:38.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:38.871487+0000) 2020-02-05T23:32:38.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:38.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:39.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:39.565+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:39.565569+0000) 2020-02-05T23:32:40.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:40.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:40.172643+0000) 2020-02-05T23:32:40.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:40.268+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:40.268579+0000) 2020-02-05T23:32:40.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:40.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:40.411787+0000) 2020-02-05T23:32:42.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:42.427+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:42.428042+0000) 2020-02-05T23:32:42.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:42.777+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:42.778629+0000) 2020-02-05T23:32:43.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:43.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.065446+0000) 2020-02-05T23:32:43.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:43.883+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.884993+0000) 2020-02-05T23:32:43.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:43.884+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.885450+0000) 2020-02-05T23:32:43.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:43.885+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.885743+0000) 2020-02-05T23:32:43.887 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:43.885+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.886353+0000) 2020-02-05T23:32:43.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:43.885+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.886992+0000) 2020-02-05T23:32:43.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:43.887+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.887739+0000) 2020-02-05T23:32:43.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:43.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.888155+0000) 2020-02-05T23:32:43.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:43.888+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.889213+0000) 2020-02-05T23:32:43.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:43.889+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.890280+0000) 2020-02-05T23:32:43.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:43.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.890636+0000) 2020-02-05T23:32:43.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:43.890+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.891014+0000) 2020-02-05T23:32:43.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:43.891+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.891647+0000) 2020-02-05T23:32:43.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:43.891+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.892521+0000) 2020-02-05T23:32:43.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:43.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.897065+0000) 2020-02-05T23:32:43.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:43.902+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:43.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:43.939+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.940632+0000) 2020-02-05T23:32:43.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:43.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:43.940625+0000) 2020-02-05T23:32:46.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:46.040+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:46.041364+0000) 2020-02-05T23:32:47.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:47.173+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:47.173267+0000) 2020-02-05T23:32:47.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:47.180+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:47.180492+0000) 2020-02-05T23:32:47.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:47.180+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:47.180673+0000) 2020-02-05T23:32:47.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:47.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:47.180762+0000) 2020-02-05T23:32:48.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:48.909+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.909807+0000) 2020-02-05T23:32:48.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:48.909+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.910607+0000) 2020-02-05T23:32:48.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:48.910+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.911049+0000) 2020-02-05T23:32:48.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:48.910+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.911420+0000) 2020-02-05T23:32:48.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:48.912+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.912670+0000) 2020-02-05T23:32:48.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:48.911+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.912755+0000) 2020-02-05T23:32:48.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:48.912+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.912965+0000) 2020-02-05T23:32:48.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:48.912+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.913328+0000) 2020-02-05T23:32:48.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:48.912+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.913931+0000) 2020-02-05T23:32:48.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:48.913+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.913752+0000) 2020-02-05T23:32:48.916 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:48.913+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.914373+0000) 2020-02-05T23:32:48.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:48.913+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.914152+0000) 2020-02-05T23:32:48.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:48.913+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.914458+0000) 2020-02-05T23:32:48.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:48.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:48.920958+0000) 2020-02-05T23:32:48.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:48.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:49.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:49.565+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:49.565742+0000) 2020-02-05T23:32:50.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:50.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:50.172931+0000) 2020-02-05T23:32:50.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:50.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:50.275731+0000) 2020-02-05T23:32:50.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:50.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:50.412043+0000) 2020-02-05T23:32:52.158 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:32:52.165 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:32:52.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:52.428+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:52.428312+0000) 2020-02-05T23:32:52.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:52.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:52.778850+0000) 2020-02-05T23:32:53.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:53.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.065679+0000) 2020-02-05T23:32:53.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:53.930+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.931014+0000) 2020-02-05T23:32:53.933 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:53.932+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.932500+0000) 2020-02-05T23:32:53.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:53.932+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.933338+0000) 2020-02-05T23:32:53.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:53.933+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.933540+0000) 2020-02-05T23:32:53.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:53.934+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.934157+0000) 2020-02-05T23:32:53.935 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:53.934+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.934677+0000) 2020-02-05T23:32:53.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:53.934+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.934720+0000) 2020-02-05T23:32:53.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:53.935+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.935385+0000) 2020-02-05T23:32:53.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:53.935+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.935789+0000) 2020-02-05T23:32:53.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:53.935+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.936020+0000) 2020-02-05T23:32:53.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:53.935+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.935859+0000) 2020-02-05T23:32:53.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:53.936+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.936662+0000) 2020-02-05T23:32:53.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:53.936+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.936842+0000) 2020-02-05T23:32:53.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:53.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.940069+0000) 2020-02-05T23:32:53.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:53.940+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.940797+0000) 2020-02-05T23:32:53.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:53.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:53.940829+0000) 2020-02-05T23:32:53.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:53.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:56.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:56.041+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:56.041649+0000) 2020-02-05T23:32:57.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:57.172+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:57.173504+0000) 2020-02-05T23:32:57.181 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:57.180+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:57.180722+0000) 2020-02-05T23:32:57.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:57.180+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:57.180908+0000) 2020-02-05T23:32:57.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:57.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:57.181086+0000) 2020-02-05T23:32:58.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:58.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.953423+0000) 2020-02-05T23:32:58.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:32:58.953+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.953965+0000) 2020-02-05T23:32:58.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:32:58.953+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.954512+0000) 2020-02-05T23:32:58.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:32:58.953+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.954842+0000) 2020-02-05T23:32:58.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:58.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.955809+0000) 2020-02-05T23:32:58.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:32:58.955+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.955911+0000) 2020-02-05T23:32:58.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:32:58.955+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.956509+0000) 2020-02-05T23:32:58.958 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:32:58.956+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.956654+0000) 2020-02-05T23:32:58.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:32:58.956+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.957458+0000) 2020-02-05T23:32:58.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:32:58.957+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.957601+0000) 2020-02-05T23:32:58.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:32:58.957+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.957738+0000) 2020-02-05T23:32:58.959 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:32:58.957+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.958453+0000) 2020-02-05T23:32:58.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:32:58.958+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.958761+0000) 2020-02-05T23:32:58.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:58.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:58.963265+0000) 2020-02-05T23:32:58.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:32:58.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:32:59.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:32:59.564+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:32:59.565949+0000) 2020-02-05T23:33:00.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:00.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:00.173164+0000) 2020-02-05T23:33:00.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:00.282+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:00.283646+0000) 2020-02-05T23:33:00.413 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:00.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:00.412294+0000) 2020-02-05T23:33:02.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:02.428+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:02.428566+0000) 2020-02-05T23:33:02.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:02.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:02.779139+0000) 2020-02-05T23:33:03.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:03.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.065911+0000) 2020-02-05T23:33:03.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:03.940+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.940978+0000) 2020-02-05T23:33:03.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:03.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.941031+0000) 2020-02-05T23:33:03.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:03.973+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.974122+0000) 2020-02-05T23:33:03.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:03.973+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.974570+0000) 2020-02-05T23:33:03.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:03.974+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.974928+0000) 2020-02-05T23:33:03.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:03.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.975561+0000) 2020-02-05T23:33:03.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:03.976+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.977350+0000) 2020-02-05T23:33:03.979 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:03.975+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.976058+0000) 2020-02-05T23:33:03.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:03.977+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.978318+0000) 2020-02-05T23:33:03.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:03.978+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.978965+0000) 2020-02-05T23:33:03.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:03.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.979359+0000) 2020-02-05T23:33:03.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:03.979+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.979648+0000) 2020-02-05T23:33:03.982 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:03.980+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.980605+0000) 2020-02-05T23:33:03.982 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:03.980+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.980745+0000) 2020-02-05T23:33:03.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:03.979+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.980258+0000) 2020-02-05T23:33:03.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:03.984+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:03.985820+0000) 2020-02-05T23:33:03.992 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:03.990+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:06.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:06.041+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:06.041904+0000) 2020-02-05T23:33:07.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:07.172+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:07.173738+0000) 2020-02-05T23:33:07.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:07.179+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:07.180958+0000) 2020-02-05T23:33:07.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:07.179+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:07.181137+0000) 2020-02-05T23:33:07.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:07.179+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:07.181358+0000) 2020-02-05T23:33:08.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:08.995+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.996327+0000) 2020-02-05T23:33:08.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:08.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.997137+0000) 2020-02-05T23:33:08.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:08.996+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.997601+0000) 2020-02-05T23:33:08.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:08.997+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.998297+0000) 2020-02-05T23:33:08.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:08.997+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.998610+0000) 2020-02-05T23:33:09.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:08.996+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.998339+0000) 2020-02-05T23:33:09.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:08.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.999266+0000) 2020-02-05T23:33:09.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:08.997+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.999307+0000) 2020-02-05T23:33:09.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:08.997+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:08.999407+0000) 2020-02-05T23:33:09.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:08.998+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.000026+0000) 2020-02-05T23:33:09.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:08.999+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.000224+0000) 2020-02-05T23:33:09.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:08.999+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.000479+0000) 2020-02-05T23:33:09.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:08.999+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.000963+0000) 2020-02-05T23:33:09.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:09.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.005479+0000) 2020-02-05T23:33:09.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:09.013+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:09.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:09.565+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:09.566156+0000) 2020-02-05T23:33:10.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:10.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:10.173437+0000) 2020-02-05T23:33:10.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:10.290+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:10.291990+0000) 2020-02-05T23:33:10.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:10.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:10.412542+0000) 2020-02-05T23:33:12.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:12.427+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:12.428854+0000) 2020-02-05T23:33:12.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:12.778+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:12.779365+0000) 2020-02-05T23:33:13.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:13.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:13.066161+0000) 2020-02-05T23:33:13.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:13.940+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:13.941219+0000) 2020-02-05T23:33:13.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:13.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:13.941336+0000) 2020-02-05T23:33:14.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:14.017+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.018725+0000) 2020-02-05T23:33:14.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:14.019+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.020459+0000) 2020-02-05T23:33:14.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:14.020+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.021557+0000) 2020-02-05T23:33:14.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:14.020+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.021269+0000) 2020-02-05T23:33:14.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:14.020+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.021910+0000) 2020-02-05T23:33:14.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:14.021+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.022483+0000) 2020-02-05T23:33:14.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:14.022+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.023597+0000) 2020-02-05T23:33:14.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:14.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.023020+0000) 2020-02-05T23:33:14.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:14.022+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.024130+0000) 2020-02-05T23:33:14.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:14.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.024663+0000) 2020-02-05T23:33:14.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:14.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.025257+0000) 2020-02-05T23:33:14.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:14.024+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.025649+0000) 2020-02-05T23:33:14.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:14.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.026233+0000) 2020-02-05T23:33:14.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:14.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:14.032183+0000) 2020-02-05T23:33:14.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:14.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147957 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:16.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:16.040+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:16.042140+0000) 2020-02-05T23:33:17.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:17.173+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:17.173966+0000) 2020-02-05T23:33:17.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:17.179+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:17.181218+0000) 2020-02-05T23:33:17.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:17.179+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:17.181383+0000) 2020-02-05T23:33:17.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:17.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:17.181640+0000) 2020-02-05T23:33:19.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:19.042+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.043574+0000) 2020-02-05T23:33:19.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:19.044+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.045156+0000) 2020-02-05T23:33:19.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:19.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.046974+0000) 2020-02-05T23:33:19.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:19.047+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.048469+0000) 2020-02-05T23:33:19.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:19.046+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.047396+0000) 2020-02-05T23:33:19.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:19.044+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.046043+0000) 2020-02-05T23:33:19.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:19.044+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.045825+0000) 2020-02-05T23:33:19.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:19.045+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.046681+0000) 2020-02-05T23:33:19.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:19.046+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.047897+0000) 2020-02-05T23:33:19.052 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:19.048+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.049524+0000) 2020-02-05T23:33:19.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:19.048+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.049947+0000) 2020-02-05T23:33:19.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:19.047+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.049237+0000) 2020-02-05T23:33:19.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:19.048+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.049560+0000) 2020-02-05T23:33:19.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:19.054+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.055519+0000) 2020-02-05T23:33:19.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:19.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 147997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:19.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:19.565+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:19.566363+0000) 2020-02-05T23:33:20.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:20.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:20.173703+0000) 2020-02-05T23:33:20.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:20.298+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:20.299029+0000) 2020-02-05T23:33:20.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:20.411+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:20.412738+0000) 2020-02-05T23:33:22.363 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:33:22.370 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:33:22.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:22.427+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:22.429115+0000) 2020-02-05T23:33:22.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:22.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:22.779608+0000) 2020-02-05T23:33:23.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:23.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:23.066387+0000) 2020-02-05T23:33:23.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:23.940+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:23.941472+0000) 2020-02-05T23:33:23.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:23.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:23.941562+0000) 2020-02-05T23:33:24.068 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:24.066+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.067304+0000) 2020-02-05T23:33:24.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:24.066+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.068109+0000) 2020-02-05T23:33:24.069 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:24.067+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.068350+0000) 2020-02-05T23:33:24.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:24.067+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.068894+0000) 2020-02-05T23:33:24.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:24.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.069378+0000) 2020-02-05T23:33:24.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:24.068+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.069807+0000) 2020-02-05T23:33:24.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:24.068+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.070036+0000) 2020-02-05T23:33:24.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:24.069+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.070649+0000) 2020-02-05T23:33:24.073 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:24.070+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.071873+0000) 2020-02-05T23:33:24.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:24.070+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.072247+0000) 2020-02-05T23:33:24.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:24.071+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.072479+0000) 2020-02-05T23:33:24.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:24.071+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.072718+0000) 2020-02-05T23:33:24.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:24.074+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.075672+0000) 2020-02-05T23:33:24.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:24.071+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:24.073017+0000) 2020-02-05T23:33:24.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:24.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:26.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:26.041+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:26.042388+0000) 2020-02-05T23:33:27.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:27.173+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:27.174149+0000) 2020-02-05T23:33:27.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:27.180+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:27.181487+0000) 2020-02-05T23:33:27.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:27.180+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:27.181660+0000) 2020-02-05T23:33:27.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:27.180+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:27.181954+0000) 2020-02-05T23:33:29.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:29.087+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.088894+0000) 2020-02-05T23:33:29.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:29.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.089417+0000) 2020-02-05T23:33:29.091 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:29.088+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.089781+0000) 2020-02-05T23:33:29.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:29.088+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.090344+0000) 2020-02-05T23:33:29.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:29.089+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.091038+0000) 2020-02-05T23:33:29.092 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:29.089+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.091087+0000) 2020-02-05T23:33:29.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:29.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.091572+0000) 2020-02-05T23:33:29.094 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:29.091+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.092704+0000) 2020-02-05T23:33:29.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:29.091+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.093060+0000) 2020-02-05T23:33:29.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:29.092+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.093467+0000) 2020-02-05T23:33:29.095 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:29.092+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.093801+0000) 2020-02-05T23:33:29.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:29.092+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.093979+0000) 2020-02-05T23:33:29.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:29.092+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.094254+0000) 2020-02-05T23:33:29.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:29.097+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.098818+0000) 2020-02-05T23:33:29.105 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:29.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:29.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:29.566+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:29.566576+0000) 2020-02-05T23:33:30.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:30.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:30.173945+0000) 2020-02-05T23:33:30.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:30.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:30.307683+0000) 2020-02-05T23:33:30.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:30.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:30.413000+0000) 2020-02-05T23:33:32.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:32.428+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:32.429411+0000) 2020-02-05T23:33:32.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:32.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:32.779854+0000) 2020-02-05T23:33:33.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:33.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:33.066648+0000) 2020-02-05T23:33:33.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:33.941+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:33.941697+0000) 2020-02-05T23:33:33.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:33.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:33.941797+0000) 2020-02-05T23:33:34.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:34.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.110713+0000) 2020-02-05T23:33:34.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:34.111+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.112746+0000) 2020-02-05T23:33:34.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:34.112+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.113695+0000) 2020-02-05T23:33:34.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:34.114+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.115466+0000) 2020-02-05T23:33:34.124 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:34.114+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.115970+0000) 2020-02-05T23:33:34.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:34.114+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.116048+0000) 2020-02-05T23:33:34.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:34.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.117011+0000) 2020-02-05T23:33:34.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:34.116+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.117290+0000) 2020-02-05T23:33:34.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:34.110+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.111810+0000) 2020-02-05T23:33:34.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:34.111+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.113272+0000) 2020-02-05T23:33:34.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:34.112+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.113705+0000) 2020-02-05T23:33:34.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:34.113+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.114612+0000) 2020-02-05T23:33:34.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:34.113+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.114865+0000) 2020-02-05T23:33:34.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:34.114+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:34.116129+0000) 2020-02-05T23:33:34.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:34.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148124 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:36.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:36.041+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:36.042664+0000) 2020-02-05T23:33:37.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:37.174+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:37.174377+0000) 2020-02-05T23:33:37.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:37.181+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:37.181775+0000) 2020-02-05T23:33:37.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:37.181+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:37.181893+0000) 2020-02-05T23:33:37.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:37.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:37.182219+0000) 2020-02-05T23:33:39.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:39.132+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.133620+0000) 2020-02-05T23:33:39.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:39.132+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.133908+0000) 2020-02-05T23:33:39.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:39.133+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.134165+0000) 2020-02-05T23:33:39.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:39.133+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.134474+0000) 2020-02-05T23:33:39.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:39.133+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.134934+0000) 2020-02-05T23:33:39.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:39.133+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.135322+0000) 2020-02-05T23:33:39.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:39.134+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.135615+0000) 2020-02-05T23:33:39.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:39.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.136473+0000) 2020-02-05T23:33:39.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:39.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.136551+0000) 2020-02-05T23:33:39.139 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:39.135+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.137032+0000) 2020-02-05T23:33:39.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:39.136+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.137562+0000) 2020-02-05T23:33:39.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:39.136+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.138181+0000) 2020-02-05T23:33:39.140 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:39.136+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.137221+0000) 2020-02-05T23:33:39.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:39.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.144497+0000) 2020-02-05T23:33:39.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:39.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:39.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:39.566+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:39.566782+0000) 2020-02-05T23:33:40.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:40.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:40.174180+0000) 2020-02-05T23:33:40.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:40.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:40.315800+0000) 2020-02-05T23:33:40.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:40.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:40.413241+0000) 2020-02-05T23:33:42.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:42.429+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:42.429686+0000) 2020-02-05T23:33:42.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:42.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:42.780107+0000) 2020-02-05T23:33:43.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:43.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:43.066918+0000) 2020-02-05T23:33:43.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:43.941+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:43.941924+0000) 2020-02-05T23:33:43.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:43.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:43.942041+0000) 2020-02-05T23:33:44.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:44.155+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.156488+0000) 2020-02-05T23:33:44.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:44.155+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.156679+0000) 2020-02-05T23:33:44.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:44.156+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.157593+0000) 2020-02-05T23:33:44.159 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:44.157+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.158126+0000) 2020-02-05T23:33:44.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:44.155+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.157274+0000) 2020-02-05T23:33:44.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:44.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.158716+0000) 2020-02-05T23:33:44.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:44.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.159207+0000) 2020-02-05T23:33:44.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:44.158+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.159650+0000) 2020-02-05T23:33:44.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:44.158+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.159851+0000) 2020-02-05T23:33:44.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:44.157+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.158505+0000) 2020-02-05T23:33:44.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:44.158+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.159813+0000) 2020-02-05T23:33:44.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:44.158+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.160309+0000) 2020-02-05T23:33:44.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:44.159+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.160279+0000) 2020-02-05T23:33:44.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:44.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:44.166380+0000) 2020-02-05T23:33:44.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:44.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148205 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:46.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:46.042+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:46.042950+0000) 2020-02-05T23:33:47.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:47.174+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:47.174607+0000) 2020-02-05T23:33:47.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:47.181+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:47.182033+0000) 2020-02-05T23:33:47.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:47.181+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:47.182130+0000) 2020-02-05T23:33:47.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:47.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:47.182485+0000) 2020-02-05T23:33:49.179 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:49.175+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.176948+0000) 2020-02-05T23:33:49.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:49.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.178208+0000) 2020-02-05T23:33:49.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:49.177+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.179018+0000) 2020-02-05T23:33:49.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:49.177+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.178607+0000) 2020-02-05T23:33:49.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:49.178+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.179563+0000) 2020-02-05T23:33:49.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:49.179+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.180524+0000) 2020-02-05T23:33:49.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:49.179+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.180606+0000) 2020-02-05T23:33:49.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:49.180+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.181197+0000) 2020-02-05T23:33:49.183 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:49.179+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.181288+0000) 2020-02-05T23:33:49.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:49.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.181626+0000) 2020-02-05T23:33:49.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:49.181+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.182176+0000) 2020-02-05T23:33:49.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:49.181+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.183287+0000) 2020-02-05T23:33:49.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:49.182+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.183264+0000) 2020-02-05T23:33:49.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:49.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.189352+0000) 2020-02-05T23:33:49.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:49.193+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148245 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:49.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:49.565+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:49.566977+0000) 2020-02-05T23:33:50.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:50.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:50.174462+0000) 2020-02-05T23:33:50.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:50.322+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:50.323759+0000) 2020-02-05T23:33:50.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:50.412+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:50.413520+0000) 2020-02-05T23:33:52.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:52.429+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:52.429946+0000) 2020-02-05T23:33:52.547 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:33:52.553 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:33:52.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:52.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:52.780408+0000) 2020-02-05T23:33:53.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:53.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:53.067187+0000) 2020-02-05T23:33:53.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:53.941+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:53.942165+0000) 2020-02-05T23:33:53.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:53.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:53.942338+0000) 2020-02-05T23:33:54.201 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:54.199+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.200526+0000) 2020-02-05T23:33:54.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:54.200+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.201843+0000) 2020-02-05T23:33:54.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:54.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.202547+0000) 2020-02-05T23:33:54.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:54.201+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.202787+0000) 2020-02-05T23:33:54.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:54.202+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.203334+0000) 2020-02-05T23:33:54.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:54.202+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.204181+0000) 2020-02-05T23:33:54.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:54.203+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.204561+0000) 2020-02-05T23:33:54.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:54.204+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.205119+0000) 2020-02-05T23:33:54.206 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:54.204+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.205477+0000) 2020-02-05T23:33:54.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:54.204+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.205869+0000) 2020-02-05T23:33:54.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:54.204+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.206361+0000) 2020-02-05T23:33:54.208 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:54.205+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.206897+0000) 2020-02-05T23:33:54.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:54.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.207534+0000) 2020-02-05T23:33:54.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:54.210+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:54.211362+0000) 2020-02-05T23:33:54.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:54.216+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148288 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:56.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:56.042+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:56.043198+0000) 2020-02-05T23:33:57.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:57.173+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:57.174830+0000) 2020-02-05T23:33:57.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:57.181+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:57.182291+0000) 2020-02-05T23:33:57.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:57.181+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:57.182390+0000) 2020-02-05T23:33:57.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:57.181+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:57.182707+0000) 2020-02-05T23:33:59.226 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:33:59.224+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.225261+0000) 2020-02-05T23:33:59.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:33:59.224+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.225736+0000) 2020-02-05T23:33:59.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:33:59.226+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.227012+0000) 2020-02-05T23:33:59.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:33:59.226+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.227161+0000) 2020-02-05T23:33:59.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:59.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.227569+0000) 2020-02-05T23:33:59.229 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:33:59.227+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.227958+0000) 2020-02-05T23:33:59.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:33:59.227+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.228985+0000) 2020-02-05T23:33:59.230 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:33:59.228+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.229210+0000) 2020-02-05T23:33:59.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:59.228+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.229229+0000) 2020-02-05T23:33:59.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:33:59.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.229460+0000) 2020-02-05T23:33:59.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:33:59.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.229587+0000) 2020-02-05T23:33:59.231 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:33:59.229+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.229827+0000) 2020-02-05T23:33:59.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:33:59.229+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.230238+0000) 2020-02-05T23:33:59.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:59.233+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.234488+0000) 2020-02-05T23:33:59.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:33:59.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:33:59.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:33:59.566+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:33:59.567179+0000) 2020-02-05T23:34:00.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:00.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:00.174715+0000) 2020-02-05T23:34:00.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:00.331+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:00.331962+0000) 2020-02-05T23:34:00.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:00.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:00.413815+0000) 2020-02-05T23:34:02.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:02.429+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:02.430187+0000) 2020-02-05T23:34:02.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:02.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:02.780666+0000) 2020-02-05T23:34:03.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:03.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:03.067422+0000) 2020-02-05T23:34:03.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:03.941+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:03.942394+0000) 2020-02-05T23:34:03.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:03.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:03.942618+0000) 2020-02-05T23:34:04.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:04.244+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.245684+0000) 2020-02-05T23:34:04.247 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:04.245+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.246455+0000) 2020-02-05T23:34:04.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:04.246+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.246912+0000) 2020-02-05T23:34:04.248 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:04.246+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.247431+0000) 2020-02-05T23:34:04.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:04.247+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.247934+0000) 2020-02-05T23:34:04.249 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:04.247+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.248504+0000) 2020-02-05T23:34:04.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:04.248+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.248569+0000) 2020-02-05T23:34:04.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:04.248+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.249139+0000) 2020-02-05T23:34:04.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:04.248+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.249369+0000) 2020-02-05T23:34:04.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:04.249+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.249701+0000) 2020-02-05T23:34:04.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:04.248+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.250027+0000) 2020-02-05T23:34:04.251 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:04.249+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.250460+0000) 2020-02-05T23:34:04.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:04.250+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.251119+0000) 2020-02-05T23:34:04.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:04.254+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:04.255827+0000) 2020-02-05T23:34:04.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:04.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148370 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:06.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:06.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:06.043476+0000) 2020-02-05T23:34:07.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:07.174+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:07.175068+0000) 2020-02-05T23:34:07.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:07.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:07.182543+0000) 2020-02-05T23:34:07.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:07.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:07.182664+0000) 2020-02-05T23:34:07.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:07.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:07.182945+0000) 2020-02-05T23:34:09.269 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:09.267+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.268420+0000) 2020-02-05T23:34:09.270 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:09.268+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.269405+0000) 2020-02-05T23:34:09.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:09.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.270295+0000) 2020-02-05T23:34:09.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:09.270+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.270761+0000) 2020-02-05T23:34:09.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:09.270+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.271335+0000) 2020-02-05T23:34:09.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:09.270+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.271691+0000) 2020-02-05T23:34:09.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:09.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.271957+0000) 2020-02-05T23:34:09.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:09.271+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.272559+0000) 2020-02-05T23:34:09.274 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:09.272+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.273563+0000) 2020-02-05T23:34:09.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:09.273+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.274100+0000) 2020-02-05T23:34:09.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:09.274+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.274626+0000) 2020-02-05T23:34:09.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:09.274+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.274764+0000) 2020-02-05T23:34:09.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:09.274+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.275403+0000) 2020-02-05T23:34:09.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:09.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.278757+0000) 2020-02-05T23:34:09.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:09.285+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:09.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:09.566+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:09.567396+0000) 2020-02-05T23:34:10.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:10.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:10.174922+0000) 2020-02-05T23:34:10.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:10.339+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:10.339802+0000) 2020-02-05T23:34:10.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:10.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:10.414070+0000) 2020-02-05T23:34:12.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:12.430+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:12.430472+0000) 2020-02-05T23:34:12.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:12.779+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:12.780924+0000) 2020-02-05T23:34:13.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:13.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:13.067706+0000) 2020-02-05T23:34:13.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:13.942+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:13.942627+0000) 2020-02-05T23:34:13.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:13.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:13.942808+0000) 2020-02-05T23:34:14.292 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:14.289+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.290747+0000) 2020-02-05T23:34:14.292 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:14.290+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.291731+0000) 2020-02-05T23:34:14.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:14.291+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.292451+0000) 2020-02-05T23:34:14.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:14.291+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.292822+0000) 2020-02-05T23:34:14.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:14.292+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.293443+0000) 2020-02-05T23:34:14.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:14.293+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.293851+0000) 2020-02-05T23:34:14.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:14.293+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.294571+0000) 2020-02-05T23:34:14.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:14.294+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.294714+0000) 2020-02-05T23:34:14.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:14.294+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.295048+0000) 2020-02-05T23:34:14.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:14.294+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.295399+0000) 2020-02-05T23:34:14.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:14.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.295842+0000) 2020-02-05T23:34:14.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:14.295+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.296028+0000) 2020-02-05T23:34:14.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:14.296+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.296640+0000) 2020-02-05T23:34:14.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:14.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:14.299935+0000) 2020-02-05T23:34:14.308 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:14.307+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148453 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:16.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:16.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:16.043733+0000) 2020-02-05T23:34:17.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:17.174+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:17.175319+0000) 2020-02-05T23:34:17.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:17.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:17.182795+0000) 2020-02-05T23:34:17.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:17.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:17.182892+0000) 2020-02-05T23:34:17.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:17.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:17.183201+0000) 2020-02-05T23:34:19.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:19.313+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.313520+0000) 2020-02-05T23:34:19.316 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:19.313+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.314525+0000) 2020-02-05T23:34:19.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:19.313+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.314331+0000) 2020-02-05T23:34:19.316 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:19.314+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.315437+0000) 2020-02-05T23:34:19.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:19.315+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.316436+0000) 2020-02-05T23:34:19.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:19.316+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.317182+0000) 2020-02-05T23:34:19.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:19.316+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.317652+0000) 2020-02-05T23:34:19.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:19.317+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.318123+0000) 2020-02-05T23:34:19.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:19.318+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.318526+0000) 2020-02-05T23:34:19.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:19.317+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.318911+0000) 2020-02-05T23:34:19.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:19.318+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.319210+0000) 2020-02-05T23:34:19.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:19.318+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.319346+0000) 2020-02-05T23:34:19.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:19.320+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.321582+0000) 2020-02-05T23:34:19.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:19.320+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.321859+0000) 2020-02-05T23:34:19.330 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:19.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148491 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:19.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:19.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:19.567612+0000) 2020-02-05T23:34:20.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:20.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:20.175136+0000) 2020-02-05T23:34:20.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:20.347+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:20.347904+0000) 2020-02-05T23:34:20.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:20.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:20.414325+0000) 2020-02-05T23:34:22.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:22.430+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:22.430708+0000) 2020-02-05T23:34:22.739 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:34:22.745 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:34:22.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:22.780+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:22.781268+0000) 2020-02-05T23:34:23.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:23.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:23.067978+0000) 2020-02-05T23:34:23.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:23.942+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:23.942859+0000) 2020-02-05T23:34:23.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:23.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:23.943009+0000) 2020-02-05T23:34:24.339 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:24.334+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.335541+0000) 2020-02-05T23:34:24.339 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:24.335+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.336368+0000) 2020-02-05T23:34:24.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:24.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.336986+0000) 2020-02-05T23:34:24.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:24.336+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.337199+0000) 2020-02-05T23:34:24.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:24.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.334694+0000) 2020-02-05T23:34:24.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:24.337+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.337877+0000) 2020-02-05T23:34:24.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:24.337+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.338676+0000) 2020-02-05T23:34:24.342 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:24.338+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.339959+0000) 2020-02-05T23:34:24.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:24.337+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.338368+0000) 2020-02-05T23:34:24.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:24.339+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.339617+0000) 2020-02-05T23:34:24.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:24.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.340682+0000) 2020-02-05T23:34:24.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:24.340+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.341351+0000) 2020-02-05T23:34:24.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:24.341+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.341567+0000) 2020-02-05T23:34:24.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:24.345+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:24.346991+0000) 2020-02-05T23:34:24.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:24.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:26.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:26.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:26.043988+0000) 2020-02-05T23:34:27.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:27.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:27.175556+0000) 2020-02-05T23:34:27.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:27.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:27.183077+0000) 2020-02-05T23:34:27.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:27.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:27.183166+0000) 2020-02-05T23:34:27.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:27.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:27.183469+0000) 2020-02-05T23:34:29.359 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:29.356+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.357779+0000) 2020-02-05T23:34:29.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:29.357+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.358574+0000) 2020-02-05T23:34:29.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:29.357+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.358385+0000) 2020-02-05T23:34:29.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:29.359+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.360228+0000) 2020-02-05T23:34:29.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:29.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.360126+0000) 2020-02-05T23:34:29.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:29.360+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.361403+0000) 2020-02-05T23:34:29.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:29.360+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.361775+0000) 2020-02-05T23:34:29.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:29.361+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.362165+0000) 2020-02-05T23:34:29.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:29.361+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.362925+0000) 2020-02-05T23:34:29.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:29.362+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.363395+0000) 2020-02-05T23:34:29.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:29.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.364091+0000) 2020-02-05T23:34:29.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:29.363+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.364181+0000) 2020-02-05T23:34:29.365 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:29.364+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.364469+0000) 2020-02-05T23:34:29.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:29.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.366799+0000) 2020-02-05T23:34:29.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:29.374+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148573 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:29.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:29.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:29.567851+0000) 2020-02-05T23:34:30.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:30.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:30.175350+0000) 2020-02-05T23:34:30.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:30.355+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:30.355887+0000) 2020-02-05T23:34:30.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:30.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:30.414499+0000) 2020-02-05T23:34:32.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:32.430+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:32.431001+0000) 2020-02-05T23:34:32.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:32.780+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:32.781518+0000) 2020-02-05T23:34:33.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:33.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:33.068286+0000) 2020-02-05T23:34:33.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:33.942+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:33.943112+0000) 2020-02-05T23:34:33.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:33.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:33.943258+0000) 2020-02-05T23:34:34.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:34.380+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.381500+0000) 2020-02-05T23:34:34.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:34.381+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.382145+0000) 2020-02-05T23:34:34.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:34.382+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.383227+0000) 2020-02-05T23:34:34.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:34.382+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.383708+0000) 2020-02-05T23:34:34.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:34.382+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.383658+0000) 2020-02-05T23:34:34.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:34.383+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.384289+0000) 2020-02-05T23:34:34.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:34.383+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.384484+0000) 2020-02-05T23:34:34.386 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:34.383+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.384627+0000) 2020-02-05T23:34:34.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:34.384+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.384875+0000) 2020-02-05T23:34:34.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:34.385+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.385451+0000) 2020-02-05T23:34:34.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:34.385+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.385573+0000) 2020-02-05T23:34:34.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:34.384+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.385243+0000) 2020-02-05T23:34:34.388 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:34.385+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.386219+0000) 2020-02-05T23:34:34.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:34.389+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:34.390610+0000) 2020-02-05T23:34:34.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:34.397+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148617 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:36.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:36.044+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:36.044243+0000) 2020-02-05T23:34:37.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:37.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:37.175790+0000) 2020-02-05T23:34:37.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:37.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:37.183307+0000) 2020-02-05T23:34:37.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:37.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:37.183371+0000) 2020-02-05T23:34:37.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:37.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:37.183696+0000) 2020-02-05T23:34:39.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:39.402+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.402579+0000) 2020-02-05T23:34:39.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:39.403+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.403281+0000) 2020-02-05T23:34:39.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:39.403+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.403680+0000) 2020-02-05T23:34:39.405 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:39.403+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.403943+0000) 2020-02-05T23:34:39.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:39.404+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.404699+0000) 2020-02-05T23:34:39.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:39.405+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.405459+0000) 2020-02-05T23:34:39.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:39.404+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.406209+0000) 2020-02-05T23:34:39.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:39.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.406344+0000) 2020-02-05T23:34:39.408 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:39.407+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.407420+0000) 2020-02-05T23:34:39.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:39.406+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.408044+0000) 2020-02-05T23:34:39.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:39.407+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.408853+0000) 2020-02-05T23:34:39.410 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:39.407+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.408946+0000) 2020-02-05T23:34:39.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:39.408+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.409471+0000) 2020-02-05T23:34:39.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:39.413+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.413274+0000) 2020-02-05T23:34:39.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:39.419+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:39.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:39.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:39.568035+0000) 2020-02-05T23:34:40.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:40.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:40.175513+0000) 2020-02-05T23:34:40.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:40.364+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:40.364128+0000) 2020-02-05T23:34:40.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:40.413+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:40.414758+0000) 2020-02-05T23:34:42.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:42.429+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:42.431242+0000) 2020-02-05T23:34:42.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:42.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:42.781760+0000) 2020-02-05T23:34:43.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:43.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:43.068495+0000) 2020-02-05T23:34:43.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:43.942+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:43.943374+0000) 2020-02-05T23:34:43.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:43.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:43.943480+0000) 2020-02-05T23:34:44.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:44.425+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.426916+0000) 2020-02-05T23:34:44.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:44.426+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.427606+0000) 2020-02-05T23:34:44.429 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:44.426+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.427922+0000) 2020-02-05T23:34:44.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:44.427+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.428592+0000) 2020-02-05T23:34:44.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:44.427+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.429235+0000) 2020-02-05T23:34:44.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:44.428+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.429834+0000) 2020-02-05T23:34:44.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:44.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.429773+0000) 2020-02-05T23:34:44.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:44.428+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.430092+0000) 2020-02-05T23:34:44.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:44.429+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.430626+0000) 2020-02-05T23:34:44.432 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:44.429+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.430957+0000) 2020-02-05T23:34:44.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:44.430+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.431457+0000) 2020-02-05T23:34:44.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:44.429+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.431107+0000) 2020-02-05T23:34:44.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:44.430+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.431879+0000) 2020-02-05T23:34:44.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:44.432+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:44.434041+0000) 2020-02-05T23:34:44.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:44.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:46.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:46.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:46.044478+0000) 2020-02-05T23:34:47.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:47.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:47.176063+0000) 2020-02-05T23:34:47.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:47.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:47.183542+0000) 2020-02-05T23:34:47.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:47.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:47.183657+0000) 2020-02-05T23:34:47.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:47.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:47.183948+0000) 2020-02-05T23:34:49.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:49.448+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.449462+0000) 2020-02-05T23:34:49.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:49.448+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.449959+0000) 2020-02-05T23:34:49.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:49.448+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.450035+0000) 2020-02-05T23:34:49.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:49.449+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.450596+0000) 2020-02-05T23:34:49.452 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:49.449+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.450931+0000) 2020-02-05T23:34:49.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:49.450+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.451616+0000) 2020-02-05T23:34:49.453 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:49.450+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.452050+0000) 2020-02-05T23:34:49.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:49.451+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.452497+0000) 2020-02-05T23:34:49.454 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:49.452+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.453179+0000) 2020-02-05T23:34:49.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:49.452+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.453933+0000) 2020-02-05T23:34:49.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:49.453+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.454832+0000) 2020-02-05T23:34:49.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:49.453+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.455176+0000) 2020-02-05T23:34:49.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:49.454+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.455666+0000) 2020-02-05T23:34:49.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:49.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.457148+0000) 2020-02-05T23:34:49.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:49.464+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148737 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:49.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:49.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:49.568315+0000) 2020-02-05T23:34:50.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:50.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:50.175724+0000) 2020-02-05T23:34:50.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:50.371+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:50.372428+0000) 2020-02-05T23:34:50.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:50.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:50.414999+0000) 2020-02-05T23:34:52.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:52.430+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:52.431534+0000) 2020-02-05T23:34:52.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:52.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:52.781997+0000) 2020-02-05T23:34:52.935 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:34:52.942 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:34:53.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:53.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:53.068773+0000) 2020-02-05T23:34:53.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:53.942+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:53.943631+0000) 2020-02-05T23:34:53.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:53.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:53.943680+0000) 2020-02-05T23:34:54.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:54.469+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.471369+0000) 2020-02-05T23:34:54.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:54.471+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.472609+0000) 2020-02-05T23:34:54.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:54.472+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.473342+0000) 2020-02-05T23:34:54.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:54.472+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.473640+0000) 2020-02-05T23:34:54.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:54.472+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.474268+0000) 2020-02-05T23:34:54.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:54.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.474576+0000) 2020-02-05T23:34:54.477 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:54.474+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.475699+0000) 2020-02-05T23:34:54.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:54.474+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.476219+0000) 2020-02-05T23:34:54.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:54.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.476833+0000) 2020-02-05T23:34:54.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:54.475+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.477403+0000) 2020-02-05T23:34:54.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:54.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.477546+0000) 2020-02-05T23:34:54.479 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:54.476+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.477914+0000) 2020-02-05T23:34:54.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:54.476+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.478016+0000) 2020-02-05T23:34:54.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:54.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:54.483013+0000) 2020-02-05T23:34:54.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:54.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:56.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:56.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:56.044709+0000) 2020-02-05T23:34:57.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:57.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:57.176315+0000) 2020-02-05T23:34:57.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:57.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:57.183758+0000) 2020-02-05T23:34:57.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:57.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:57.183950+0000) 2020-02-05T23:34:57.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:57.182+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:57.184184+0000) 2020-02-05T23:34:59.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:34:59.494+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.496236+0000) 2020-02-05T23:34:59.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:34:59.495+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.496630+0000) 2020-02-05T23:34:59.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:34:59.495+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.497320+0000) 2020-02-05T23:34:59.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:34:59.496+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.497915+0000) 2020-02-05T23:34:59.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:34:59.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.498270+0000) 2020-02-05T23:34:59.499 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:34:59.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.498713+0000) 2020-02-05T23:34:59.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:34:59.497+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.499263+0000) 2020-02-05T23:34:59.501 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:34:59.498+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.499918+0000) 2020-02-05T23:34:59.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:59.499+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.500955+0000) 2020-02-05T23:34:59.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:34:59.499+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.500776+0000) 2020-02-05T23:34:59.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:59.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.501757+0000) 2020-02-05T23:34:59.503 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:34:59.500+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.502158+0000) 2020-02-05T23:34:59.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:34:59.501+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.502702+0000) 2020-02-05T23:34:59.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:59.506+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.507869+0000) 2020-02-05T23:34:59.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:34:59.512+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:34:59.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:34:59.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:34:59.568543+0000) 2020-02-05T23:35:00.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:00.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:00.175913+0000) 2020-02-05T23:35:00.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:00.381+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:00.381911+0000) 2020-02-05T23:35:00.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:00.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:00.415267+0000) 2020-02-05T23:35:02.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:02.431+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:02.431801+0000) 2020-02-05T23:35:02.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:02.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:02.782265+0000) 2020-02-05T23:35:03.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:03.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:03.069022+0000) 2020-02-05T23:35:03.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:03.943+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:03.943916+0000) 2020-02-05T23:35:03.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:03.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:03.943916+0000) 2020-02-05T23:35:04.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:04.518+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.520058+0000) 2020-02-05T23:35:04.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:04.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.521168+0000) 2020-02-05T23:35:04.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:04.520+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.522193+0000) 2020-02-05T23:35:04.523 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:04.520+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.522244+0000) 2020-02-05T23:35:04.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:04.521+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.523036+0000) 2020-02-05T23:35:04.524 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:04.522+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.523387+0000) 2020-02-05T23:35:04.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:04.523+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.524109+0000) 2020-02-05T23:35:04.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:04.523+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.524528+0000) 2020-02-05T23:35:04.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:04.523+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.524936+0000) 2020-02-05T23:35:04.526 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:04.524+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.525156+0000) 2020-02-05T23:35:04.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:04.523+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.525296+0000) 2020-02-05T23:35:04.528 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:04.524+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.526098+0000) 2020-02-05T23:35:04.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:04.524+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.526124+0000) 2020-02-05T23:35:04.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:04.530+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:04.531449+0000) 2020-02-05T23:35:04.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:04.535+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148864 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:06.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:06.043+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:06.044904+0000) 2020-02-05T23:35:07.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:07.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:07.176553+0000) 2020-02-05T23:35:07.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:07.183+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:07.184009+0000) 2020-02-05T23:35:07.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:07.183+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:07.184164+0000) 2020-02-05T23:35:07.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:07.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:07.184481+0000) 2020-02-05T23:35:09.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:09.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.541839+0000) 2020-02-05T23:35:09.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:09.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.542801+0000) 2020-02-05T23:35:09.545 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:09.542+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.543964+0000) 2020-02-05T23:35:09.545 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:09.543+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.544218+0000) 2020-02-05T23:35:09.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:09.543+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.545318+0000) 2020-02-05T23:35:09.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:09.544+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.546150+0000) 2020-02-05T23:35:09.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:09.545+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.546617+0000) 2020-02-05T23:35:09.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:09.545+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.547240+0000) 2020-02-05T23:35:09.548 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:09.546+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.547247+0000) 2020-02-05T23:35:09.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:09.546+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.547800+0000) 2020-02-05T23:35:09.549 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:09.546+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.548219+0000) 2020-02-05T23:35:09.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:09.550+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.551841+0000) 2020-02-05T23:35:09.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:09.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.552179+0000) 2020-02-05T23:35:09.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:09.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.553284+0000) 2020-02-05T23:35:09.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:09.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:09.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:09.567+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:09.568757+0000) 2020-02-05T23:35:10.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:10.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:10.176119+0000) 2020-02-05T23:35:10.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:10.389+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:10.389966+0000) 2020-02-05T23:35:10.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:10.414+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:10.415501+0000) 2020-02-05T23:35:12.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:12.431+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:12.432066+0000) 2020-02-05T23:35:12.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:12.781+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:12.782525+0000) 2020-02-05T23:35:13.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:13.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:13.069290+0000) 2020-02-05T23:35:13.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:13.943+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:13.944186+0000) 2020-02-05T23:35:13.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:13.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:13.944187+0000) 2020-02-05T23:35:14.565 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:14.563+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.564131+0000) 2020-02-05T23:35:14.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:14.563+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.564618+0000) 2020-02-05T23:35:14.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:14.564+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.565419+0000) 2020-02-05T23:35:14.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:14.564+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.565397+0000) 2020-02-05T23:35:14.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:14.564+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.565925+0000) 2020-02-05T23:35:14.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:14.565+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.566644+0000) 2020-02-05T23:35:14.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:14.565+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.567179+0000) 2020-02-05T23:35:14.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:14.565+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.567389+0000) 2020-02-05T23:35:14.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:14.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.567877+0000) 2020-02-05T23:35:14.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:14.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.568236+0000) 2020-02-05T23:35:14.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:14.567+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.569118+0000) 2020-02-05T23:35:14.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:14.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.568687+0000) 2020-02-05T23:35:14.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:14.568+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.570070+0000) 2020-02-05T23:35:14.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:14.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:14.574482+0000) 2020-02-05T23:35:14.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:14.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:16.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:16.044+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:16.045154+0000) 2020-02-05T23:35:17.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:17.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:17.176778+0000) 2020-02-05T23:35:17.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:17.182+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:17.184244+0000) 2020-02-05T23:35:17.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:17.182+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:17.184402+0000) 2020-02-05T23:35:17.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:17.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:17.184715+0000) 2020-02-05T23:35:19.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:19.568+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.568996+0000) 2020-02-05T23:35:19.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:19.586+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.587517+0000) 2020-02-05T23:35:19.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:19.587+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.588391+0000) 2020-02-05T23:35:19.590 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:19.588+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.589547+0000) 2020-02-05T23:35:19.591 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:19.589+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.590253+0000) 2020-02-05T23:35:19.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:19.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.590688+0000) 2020-02-05T23:35:19.592 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:19.590+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.591244+0000) 2020-02-05T23:35:19.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:19.590+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.591785+0000) 2020-02-05T23:35:19.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:19.591+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.592335+0000) 2020-02-05T23:35:19.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:19.591+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.592844+0000) 2020-02-05T23:35:19.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:19.591+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.592991+0000) 2020-02-05T23:35:19.594 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:19.591+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.593169+0000) 2020-02-05T23:35:19.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:19.591+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.593409+0000) 2020-02-05T23:35:19.595 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:19.592+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.593776+0000) 2020-02-05T23:35:19.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:19.599+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:19.601025+0000) 2020-02-05T23:35:19.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:19.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 148987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:20.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:20.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:20.176318+0000) 2020-02-05T23:35:20.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:20.397+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:20.398135+0000) 2020-02-05T23:35:20.416 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:20.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:20.415717+0000) 2020-02-05T23:35:22.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:22.431+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:22.432332+0000) 2020-02-05T23:35:22.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:22.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:22.782762+0000) 2020-02-05T23:35:23.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:23.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:23.069539+0000) 2020-02-05T23:35:23.125 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:35:23.135 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:35:23.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:23.943+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:23.944452+0000) 2020-02-05T23:35:23.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:23.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:23.944454+0000) 2020-02-05T23:35:24.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:24.610+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.611601+0000) 2020-02-05T23:35:24.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:24.611+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.612836+0000) 2020-02-05T23:35:24.614 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:24.611+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.613338+0000) 2020-02-05T23:35:24.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:24.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.613630+0000) 2020-02-05T23:35:24.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:24.612+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.613951+0000) 2020-02-05T23:35:24.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:24.612+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.614287+0000) 2020-02-05T23:35:24.615 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:24.613+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.614491+0000) 2020-02-05T23:35:24.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:24.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615195+0000) 2020-02-05T23:35:24.616 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:24.614+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615463+0000) 2020-02-05T23:35:24.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:24.614+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615879+0000) 2020-02-05T23:35:24.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:24.614+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615996+0000) 2020-02-05T23:35:24.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:24.614+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615802+0000) 2020-02-05T23:35:24.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:24.614+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.615858+0000) 2020-02-05T23:35:24.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:24.619+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:24.620888+0000) 2020-02-05T23:35:24.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:24.626+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:26.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:26.044+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:26.045387+0000) 2020-02-05T23:35:27.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:27.175+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:27.177025+0000) 2020-02-05T23:35:27.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:27.183+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:27.184479+0000) 2020-02-05T23:35:27.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:27.183+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:27.184688+0000) 2020-02-05T23:35:27.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:27.183+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:27.184925+0000) 2020-02-05T23:35:29.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:29.569+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.569290+0000) 2020-02-05T23:35:29.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:29.632+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.633445+0000) 2020-02-05T23:35:29.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:29.633+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.634411+0000) 2020-02-05T23:35:29.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:29.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.635790+0000) 2020-02-05T23:35:29.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:29.635+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.636327+0000) 2020-02-05T23:35:29.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:29.635+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.637023+0000) 2020-02-05T23:35:29.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:29.636+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.636903+0000) 2020-02-05T23:35:29.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:29.636+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.637471+0000) 2020-02-05T23:35:29.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:29.636+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.637869+0000) 2020-02-05T23:35:29.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:29.638+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.638464+0000) 2020-02-05T23:35:29.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:29.638+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.638938+0000) 2020-02-05T23:35:29.641 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:29.639+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.640152+0000) 2020-02-05T23:35:29.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:29.639+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.640379+0000) 2020-02-05T23:35:29.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:29.640+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.640561+0000) 2020-02-05T23:35:29.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:29.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:29.644989+0000) 2020-02-05T23:35:29.651 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:29.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149071 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:30.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:30.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:30.176562+0000) 2020-02-05T23:35:30.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:30.405+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:30.406111+0000) 2020-02-05T23:35:30.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:30.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:30.415888+0000) 2020-02-05T23:35:32.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:32.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:32.432592+0000) 2020-02-05T23:35:32.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:32.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:32.783019+0000) 2020-02-05T23:35:33.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:33.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:33.069770+0000) 2020-02-05T23:35:33.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:33.944+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:33.944700+0000) 2020-02-05T23:35:33.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:33.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:33.944711+0000) 2020-02-05T23:35:34.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:34.656+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.656599+0000) 2020-02-05T23:35:34.658 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:34.656+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.656973+0000) 2020-02-05T23:35:34.659 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:34.657+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.657957+0000) 2020-02-05T23:35:34.659 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:34.658+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.658341+0000) 2020-02-05T23:35:34.660 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:34.658+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.658988+0000) 2020-02-05T23:35:34.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:34.659+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.659461+0000) 2020-02-05T23:35:34.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:34.659+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.659984+0000) 2020-02-05T23:35:34.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:34.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.660448+0000) 2020-02-05T23:35:34.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:34.661+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.661334+0000) 2020-02-05T23:35:34.663 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:34.661+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.662103+0000) 2020-02-05T23:35:34.664 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:34.662+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.662282+0000) 2020-02-05T23:35:34.664 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:34.661+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.662407+0000) 2020-02-05T23:35:34.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:34.662+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.662674+0000) 2020-02-05T23:35:34.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:34.666+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:34.666887+0000) 2020-02-05T23:35:34.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:34.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:36.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:36.045+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:36.045713+0000) 2020-02-05T23:35:37.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:37.177+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:37.177293+0000) 2020-02-05T23:35:37.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:37.184+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:37.184748+0000) 2020-02-05T23:35:37.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:37.184+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:37.184935+0000) 2020-02-05T23:35:37.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:37.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:37.185156+0000) 2020-02-05T23:35:39.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:39.568+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.569547+0000) 2020-02-05T23:35:39.680 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:39.677+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.678890+0000) 2020-02-05T23:35:39.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:39.680+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.681074+0000) 2020-02-05T23:35:39.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:39.680+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.681359+0000) 2020-02-05T23:35:39.683 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:39.681+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.681769+0000) 2020-02-05T23:35:39.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:39.681+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.682461+0000) 2020-02-05T23:35:39.684 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:39.681+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.682925+0000) 2020-02-05T23:35:39.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:39.682+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.682520+0000) 2020-02-05T23:35:39.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:39.682+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.683322+0000) 2020-02-05T23:35:39.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:39.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.684005+0000) 2020-02-05T23:35:39.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:39.684+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.684806+0000) 2020-02-05T23:35:39.686 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:39.684+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.684408+0000) 2020-02-05T23:35:39.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:39.684+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.685156+0000) 2020-02-05T23:35:39.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:39.684+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.685403+0000) 2020-02-05T23:35:39.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:39.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:39.692705+0000) 2020-02-05T23:35:39.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:39.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149155 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:40.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:40.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:40.176807+0000) 2020-02-05T23:35:40.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:40.413+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:40.414390+0000) 2020-02-05T23:35:40.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:40.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:40.416119+0000) 2020-02-05T23:35:42.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:42.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:42.432852+0000) 2020-02-05T23:35:42.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:42.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:42.783278+0000) 2020-02-05T23:35:43.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:43.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:43.070019+0000) 2020-02-05T23:35:43.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:43.944+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:43.944973+0000) 2020-02-05T23:35:43.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:43.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:43.944976+0000) 2020-02-05T23:35:44.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:44.701+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.701789+0000) 2020-02-05T23:35:44.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:44.701+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.702642+0000) 2020-02-05T23:35:44.704 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:44.702+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.703204+0000) 2020-02-05T23:35:44.705 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:44.702+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.703635+0000) 2020-02-05T23:35:44.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:44.702+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.704056+0000) 2020-02-05T23:35:44.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:44.704+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.704606+0000) 2020-02-05T23:35:44.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:44.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.705132+0000) 2020-02-05T23:35:44.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:44.705+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.706153+0000) 2020-02-05T23:35:44.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:44.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.706321+0000) 2020-02-05T23:35:44.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:44.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.706693+0000) 2020-02-05T23:35:44.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:44.705+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.706937+0000) 2020-02-05T23:35:44.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:44.706+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.707350+0000) 2020-02-05T23:35:44.709 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:44.707+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.708061+0000) 2020-02-05T23:35:44.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:44.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:44.710698+0000) 2020-02-05T23:35:44.719 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:44.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:46.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:46.045+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:46.045947+0000) 2020-02-05T23:35:47.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:47.176+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:47.177548+0000) 2020-02-05T23:35:47.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:47.184+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:47.185033+0000) 2020-02-05T23:35:47.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:47.184+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:47.185189+0000) 2020-02-05T23:35:47.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:47.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:47.185532+0000) 2020-02-05T23:35:49.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:49.568+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.569764+0000) 2020-02-05T23:35:49.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:49.722+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.723970+0000) 2020-02-05T23:35:49.726 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:49.724+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.725035+0000) 2020-02-05T23:35:49.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:49.724+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.725315+0000) 2020-02-05T23:35:49.726 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:49.724+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.725622+0000) 2020-02-05T23:35:49.727 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:49.725+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.726301+0000) 2020-02-05T23:35:49.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:49.726+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.726846+0000) 2020-02-05T23:35:49.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:49.726+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.727190+0000) 2020-02-05T23:35:49.728 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:49.727+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.727412+0000) 2020-02-05T23:35:49.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:49.727+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.727550+0000) 2020-02-05T23:35:49.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:49.727+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.727976+0000) 2020-02-05T23:35:49.729 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:49.727+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.728436+0000) 2020-02-05T23:35:49.730 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:49.727+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.728738+0000) 2020-02-05T23:35:49.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:49.731+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.732162+0000) 2020-02-05T23:35:49.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:49.731+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:49.732450+0000) 2020-02-05T23:35:49.740 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:49.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149239 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:50.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:50.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:50.177011+0000) 2020-02-05T23:35:50.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:50.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:50.416364+0000) 2020-02-05T23:35:50.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:50.421+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:50.422646+0000) 2020-02-05T23:35:52.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:52.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:52.433096+0000) 2020-02-05T23:35:52.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:52.782+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:52.783553+0000) 2020-02-05T23:35:53.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:53.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:53.070261+0000) 2020-02-05T23:35:53.294 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:35:53.300 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:35:53.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:53.944+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:53.945240+0000) 2020-02-05T23:35:53.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:53.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:53.945201+0000) 2020-02-05T23:35:54.745 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:54.743+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.744089+0000) 2020-02-05T23:35:54.746 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:54.743+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.744881+0000) 2020-02-05T23:35:54.746 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:54.745+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.745557+0000) 2020-02-05T23:35:54.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:54.744+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.745837+0000) 2020-02-05T23:35:54.747 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:54.745+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.746940+0000) 2020-02-05T23:35:54.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:54.746+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.747713+0000) 2020-02-05T23:35:54.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:54.747+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.748488+0000) 2020-02-05T23:35:54.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:54.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.749177+0000) 2020-02-05T23:35:54.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:54.748+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.749863+0000) 2020-02-05T23:35:54.751 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:54.749+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.749812+0000) 2020-02-05T23:35:54.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:54.750+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.750683+0000) 2020-02-05T23:35:54.752 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:54.750+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.751001+0000) 2020-02-05T23:35:54.752 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:54.751+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.751608+0000) 2020-02-05T23:35:54.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:54.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:54.753818+0000) 2020-02-05T23:35:54.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:54.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:35:56.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:56.046+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:56.046194+0000) 2020-02-05T23:35:57.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:57.177+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:57.177769+0000) 2020-02-05T23:35:57.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:57.185+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:57.185269+0000) 2020-02-05T23:35:57.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:57.185+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:57.185393+0000) 2020-02-05T23:35:57.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:57.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:57.185763+0000) 2020-02-05T23:35:59.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:59.569+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.570006+0000) 2020-02-05T23:35:59.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:35:59.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.766972+0000) 2020-02-05T23:35:59.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:35:59.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.767208+0000) 2020-02-05T23:35:59.769 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:35:59.766+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.767662+0000) 2020-02-05T23:35:59.769 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:35:59.767+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.768562+0000) 2020-02-05T23:35:59.770 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:35:59.768+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.768543+0000) 2020-02-05T23:35:59.770 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:35:59.767+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.768972+0000) 2020-02-05T23:35:59.770 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:35:59.768+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.769286+0000) 2020-02-05T23:35:59.771 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:35:59.768+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.769761+0000) 2020-02-05T23:35:59.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:59.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.769951+0000) 2020-02-05T23:35:59.772 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:35:59.769+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.770392+0000) 2020-02-05T23:35:59.772 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:35:59.770+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.771021+0000) 2020-02-05T23:35:59.772 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:35:59.770+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.771234+0000) 2020-02-05T23:35:59.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:59.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.771970+0000) 2020-02-05T23:35:59.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:35:59.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:35:59.776274+0000) 2020-02-05T23:35:59.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:35:59.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:00.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:00.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:00.177246+0000) 2020-02-05T23:36:00.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:00.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:00.416580+0000) 2020-02-05T23:36:00.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:00.429+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:00.430439+0000) 2020-02-05T23:36:02.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:02.433+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:02.433353+0000) 2020-02-05T23:36:02.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:02.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:02.783840+0000) 2020-02-05T23:36:03.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:03.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:03.070482+0000) 2020-02-05T23:36:03.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:03.944+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:03.945473+0000) 2020-02-05T23:36:03.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:03.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:03.945473+0000) 2020-02-05T23:36:04.791 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:04.788+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.789578+0000) 2020-02-05T23:36:04.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:04.790+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.790458+0000) 2020-02-05T23:36:04.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:04.789+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.790743+0000) 2020-02-05T23:36:04.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:04.790+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.791322+0000) 2020-02-05T23:36:04.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:04.791+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.792551+0000) 2020-02-05T23:36:04.794 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:04.792+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.793361+0000) 2020-02-05T23:36:04.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:04.792+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.793545+0000) 2020-02-05T23:36:04.795 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:04.793+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.794366+0000) 2020-02-05T23:36:04.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:04.794+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.794480+0000) 2020-02-05T23:36:04.796 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:04.794+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.794908+0000) 2020-02-05T23:36:04.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:04.794+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.795255+0000) 2020-02-05T23:36:04.797 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:04.794+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.795233+0000) 2020-02-05T23:36:04.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:04.796+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.797600+0000) 2020-02-05T23:36:04.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:04.796+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:04.797871+0000) 2020-02-05T23:36:04.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:04.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:06.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:06.045+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:06.046493+0000) 2020-02-05T23:36:07.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:07.176+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:07.178021+0000) 2020-02-05T23:36:07.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:07.184+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:07.185533+0000) 2020-02-05T23:36:07.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:07.184+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:07.185658+0000) 2020-02-05T23:36:07.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:07.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:07.186011+0000) 2020-02-05T23:36:09.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:09.569+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.570275+0000) 2020-02-05T23:36:09.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:09.811+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.812460+0000) 2020-02-05T23:36:09.814 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:09.811+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.813247+0000) 2020-02-05T23:36:09.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:09.813+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.814433+0000) 2020-02-05T23:36:09.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:09.813+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.814846+0000) 2020-02-05T23:36:09.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:09.814+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.815282+0000) 2020-02-05T23:36:09.818 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:09.814+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.816101+0000) 2020-02-05T23:36:09.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:09.815+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.817025+0000) 2020-02-05T23:36:09.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:09.815+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.816691+0000) 2020-02-05T23:36:09.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:09.816+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.817583+0000) 2020-02-05T23:36:09.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:09.815+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.816878+0000) 2020-02-05T23:36:09.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:09.815+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.817368+0000) 2020-02-05T23:36:09.820 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:09.816+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.817930+0000) 2020-02-05T23:36:09.821 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:09.816+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.818298+0000) 2020-02-05T23:36:09.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:09.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:09.821858+0000) 2020-02-05T23:36:09.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:09.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:10.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:10.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:10.177453+0000) 2020-02-05T23:36:10.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:10.415+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:10.416795+0000) 2020-02-05T23:36:10.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:10.438+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:10.440024+0000) 2020-02-05T23:36:12.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:12.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:12.433620+0000) 2020-02-05T23:36:12.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:12.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:12.784124+0000) 2020-02-05T23:36:13.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:13.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:13.070759+0000) 2020-02-05T23:36:13.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:13.945+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:13.945738+0000) 2020-02-05T23:36:13.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:13.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:13.945738+0000) 2020-02-05T23:36:14.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:14.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.835184+0000) 2020-02-05T23:36:14.837 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:14.834+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.835865+0000) 2020-02-05T23:36:14.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:14.835+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.836530+0000) 2020-02-05T23:36:14.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:14.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.836693+0000) 2020-02-05T23:36:14.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:14.836+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.837142+0000) 2020-02-05T23:36:14.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:14.837+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.838706+0000) 2020-02-05T23:36:14.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:14.838+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.839570+0000) 2020-02-05T23:36:14.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:14.838+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.840052+0000) 2020-02-05T23:36:14.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:14.839+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.840922+0000) 2020-02-05T23:36:14.842 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:14.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.840820+0000) 2020-02-05T23:36:14.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:14.840+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.841202+0000) 2020-02-05T23:36:14.843 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:14.839+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.841362+0000) 2020-02-05T23:36:14.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:14.841+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.842616+0000) 2020-02-05T23:36:14.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:14.841+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:14.843021+0000) 2020-02-05T23:36:14.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:14.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:16.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:16.045+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:16.046723+0000) 2020-02-05T23:36:17.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:17.177+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:17.178306+0000) 2020-02-05T23:36:17.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:17.184+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:17.185725+0000) 2020-02-05T23:36:17.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:17.184+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:17.185863+0000) 2020-02-05T23:36:17.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:17.184+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:17.186266+0000) 2020-02-05T23:36:19.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:19.570+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.570511+0000) 2020-02-05T23:36:19.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:19.859+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.860183+0000) 2020-02-05T23:36:19.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:19.859+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.860650+0000) 2020-02-05T23:36:19.865 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:19.860+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.861415+0000) 2020-02-05T23:36:19.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:19.860+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.861561+0000) 2020-02-05T23:36:19.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:19.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.862421+0000) 2020-02-05T23:36:19.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:19.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.862914+0000) 2020-02-05T23:36:19.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:19.862+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.863718+0000) 2020-02-05T23:36:19.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:19.863+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.864836+0000) 2020-02-05T23:36:19.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:19.864+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.865685+0000) 2020-02-05T23:36:19.871 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:19.864+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.865907+0000) 2020-02-05T23:36:19.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:19.862+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.864029+0000) 2020-02-05T23:36:19.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:19.863+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.864350+0000) 2020-02-05T23:36:19.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:19.863+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.864964+0000) 2020-02-05T23:36:19.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:19.872+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:19.873107+0000) 2020-02-05T23:36:19.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:19.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:20.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:20.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:20.177666+0000) 2020-02-05T23:36:20.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:20.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:20.417034+0000) 2020-02-05T23:36:20.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:20.447+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:20.448026+0000) 2020-02-05T23:36:22.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:22.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:22.433833+0000) 2020-02-05T23:36:22.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:22.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:22.784362+0000) 2020-02-05T23:36:23.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:23.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:23.070994+0000) 2020-02-05T23:36:23.474 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:36:23.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:36:23.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:23.945+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:23.946004+0000) 2020-02-05T23:36:23.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:23.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:23.946004+0000) 2020-02-05T23:36:24.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:24.881+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.882973+0000) 2020-02-05T23:36:24.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:24.882+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.883418+0000) 2020-02-05T23:36:24.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:24.883+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.884573+0000) 2020-02-05T23:36:24.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:24.883+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.884635+0000) 2020-02-05T23:36:24.886 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:24.884+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.885533+0000) 2020-02-05T23:36:24.887 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:24.884+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.885801+0000) 2020-02-05T23:36:24.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:24.885+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.886525+0000) 2020-02-05T23:36:24.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:24.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.887124+0000) 2020-02-05T23:36:24.888 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:24.885+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.887091+0000) 2020-02-05T23:36:24.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:24.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.887426+0000) 2020-02-05T23:36:24.889 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:24.886+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.888173+0000) 2020-02-05T23:36:24.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:24.886+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.887842+0000) 2020-02-05T23:36:24.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:24.887+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.888735+0000) 2020-02-05T23:36:24.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:24.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:24.893234+0000) 2020-02-05T23:36:24.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:24.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:26.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:26.046+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:26.046982+0000) 2020-02-05T23:36:27.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:27.178+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:27.178543+0000) 2020-02-05T23:36:27.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:27.185+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:27.185977+0000) 2020-02-05T23:36:27.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:27.185+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:27.186088+0000) 2020-02-05T23:36:27.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:27.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:27.186487+0000) 2020-02-05T23:36:29.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:29.570+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.570764+0000) 2020-02-05T23:36:29.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:29.905+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.905720+0000) 2020-02-05T23:36:29.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:29.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.906289+0000) 2020-02-05T23:36:29.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:29.906+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.906658+0000) 2020-02-05T23:36:29.908 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:29.907+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.907599+0000) 2020-02-05T23:36:29.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:29.906+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.908226+0000) 2020-02-05T23:36:29.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:29.908+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.909024+0000) 2020-02-05T23:36:29.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:29.908+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.909566+0000) 2020-02-05T23:36:29.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:29.909+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.909908+0000) 2020-02-05T23:36:29.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:29.909+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.910512+0000) 2020-02-05T23:36:29.912 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:29.910+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.911558+0000) 2020-02-05T23:36:29.912 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:29.910+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.911613+0000) 2020-02-05T23:36:29.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:29.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.912287+0000) 2020-02-05T23:36:29.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:29.911+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.912648+0000) 2020-02-05T23:36:29.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:29.917+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:29.917163+0000) 2020-02-05T23:36:29.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:29.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:30.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:30.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:30.177879+0000) 2020-02-05T23:36:30.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:30.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:30.417258+0000) 2020-02-05T23:36:30.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:30.455+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:30.455556+0000) 2020-02-05T23:36:32.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:32.433+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:32.434093+0000) 2020-02-05T23:36:32.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:32.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:32.784594+0000) 2020-02-05T23:36:33.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:33.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:33.071281+0000) 2020-02-05T23:36:33.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:33.945+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:33.946325+0000) 2020-02-05T23:36:33.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:33.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:33.946324+0000) 2020-02-05T23:36:34.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:34.926+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.927772+0000) 2020-02-05T23:36:34.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:34.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.928460+0000) 2020-02-05T23:36:34.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:34.928+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.929207+0000) 2020-02-05T23:36:34.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:34.928+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.929729+0000) 2020-02-05T23:36:34.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:34.929+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.930207+0000) 2020-02-05T23:36:34.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:34.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.930848+0000) 2020-02-05T23:36:34.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:34.930+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.931515+0000) 2020-02-05T23:36:34.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:34.930+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.931688+0000) 2020-02-05T23:36:34.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:34.931+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.932408+0000) 2020-02-05T23:36:34.935 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:34.932+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.933271+0000) 2020-02-05T23:36:34.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:34.932+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.933725+0000) 2020-02-05T23:36:34.936 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:34.933+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.934525+0000) 2020-02-05T23:36:34.936 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:34.933+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.934808+0000) 2020-02-05T23:36:34.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:34.938+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:34.939756+0000) 2020-02-05T23:36:34.954 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:34.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:36.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:36.046+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:36.047246+0000) 2020-02-05T23:36:37.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:37.177+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:37.178774+0000) 2020-02-05T23:36:37.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:37.185+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:37.186240+0000) 2020-02-05T23:36:37.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:37.185+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:37.186344+0000) 2020-02-05T23:36:37.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:37.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:37.186752+0000) 2020-02-05T23:36:39.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:39.570+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.571000+0000) 2020-02-05T23:36:39.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:39.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.952326+0000) 2020-02-05T23:36:39.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:39.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.952735+0000) 2020-02-05T23:36:39.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:39.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.953803+0000) 2020-02-05T23:36:39.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:39.953+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.954207+0000) 2020-02-05T23:36:39.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:39.953+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.954644+0000) 2020-02-05T23:36:39.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:39.953+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.955138+0000) 2020-02-05T23:36:39.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:39.954+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.955588+0000) 2020-02-05T23:36:39.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:39.954+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.955889+0000) 2020-02-05T23:36:39.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:39.954+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.956188+0000) 2020-02-05T23:36:39.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:39.955+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.956827+0000) 2020-02-05T23:36:39.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:39.955+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.957198+0000) 2020-02-05T23:36:39.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:39.956+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.957681+0000) 2020-02-05T23:36:39.960 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:39.956+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.958227+0000) 2020-02-05T23:36:39.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:39.961+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:39.962746+0000) 2020-02-05T23:36:39.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:39.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:40.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:40.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:40.178121+0000) 2020-02-05T23:36:40.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:40.416+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:40.417490+0000) 2020-02-05T23:36:40.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:40.463+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:40.464539+0000) 2020-02-05T23:36:42.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:42.432+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:42.434367+0000) 2020-02-05T23:36:42.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:42.783+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:42.784861+0000) 2020-02-05T23:36:43.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:43.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:43.071507+0000) 2020-02-05T23:36:43.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:43.945+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:43.946595+0000) 2020-02-05T23:36:43.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:43.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:43.946595+0000) 2020-02-05T23:36:44.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:44.972+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.973928+0000) 2020-02-05T23:36:44.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:44.973+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.974360+0000) 2020-02-05T23:36:44.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:44.972+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.973809+0000) 2020-02-05T23:36:44.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:44.974+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.975271+0000) 2020-02-05T23:36:44.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:44.975+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.976956+0000) 2020-02-05T23:36:44.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:44.976+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.977448+0000) 2020-02-05T23:36:44.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:44.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.977996+0000) 2020-02-05T23:36:44.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:44.977+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.978481+0000) 2020-02-05T23:36:44.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:44.977+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.979071+0000) 2020-02-05T23:36:44.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:44.978+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.980035+0000) 2020-02-05T23:36:44.981 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:44.978+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.980194+0000) 2020-02-05T23:36:44.981 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:44.979+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.980589+0000) 2020-02-05T23:36:44.982 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:44.979+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.980903+0000) 2020-02-05T23:36:44.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:44.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:44.984326+0000) 2020-02-05T23:36:44.992 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:44.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149697 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:46.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:46.046+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:46.047562+0000) 2020-02-05T23:36:47.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:47.178+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:47.179012+0000) 2020-02-05T23:36:47.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:47.185+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:47.186497+0000) 2020-02-05T23:36:47.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:47.185+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:47.186616+0000) 2020-02-05T23:36:47.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:47.185+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:47.186974+0000) 2020-02-05T23:36:49.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:49.570+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.571285+0000) 2020-02-05T23:36:49.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:49.997+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.998328+0000) 2020-02-05T23:36:50.000 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:49.997+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.998658+0000) 2020-02-05T23:36:50.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:49.997+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.999111+0000) 2020-02-05T23:36:50.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:49.998+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.999133+0000) 2020-02-05T23:36:50.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:49.998+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.999478+0000) 2020-02-05T23:36:50.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:49.998+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:49.999462+0000) 2020-02-05T23:36:50.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:49.998+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.000017+0000) 2020-02-05T23:36:50.002 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:49.999+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.000782+0000) 2020-02-05T23:36:50.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:50.000+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.001406+0000) 2020-02-05T23:36:50.003 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:50.000+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.001810+0000) 2020-02-05T23:36:50.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:49.999+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.000816+0000) 2020-02-05T23:36:50.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:50.000+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.002400+0000) 2020-02-05T23:36:50.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:50.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.005349+0000) 2020-02-05T23:36:50.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:50.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.005560+0000) 2020-02-05T23:36:50.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:50.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149735 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:50.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:50.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.178325+0000) 2020-02-05T23:36:50.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:50.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.417757+0000) 2020-02-05T23:36:50.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:50.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:50.472888+0000) 2020-02-05T23:36:52.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:52.434+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:52.434666+0000) 2020-02-05T23:36:52.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:52.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:52.785107+0000) 2020-02-05T23:36:53.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:53.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:53.071737+0000) 2020-02-05T23:36:53.672 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:36:53.678 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:36:53.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:53.945+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:53.946891+0000) 2020-02-05T23:36:53.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:53.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:53.946891+0000) 2020-02-05T23:36:55.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:36:55.018+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.019221+0000) 2020-02-05T23:36:55.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:55.018+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.019494+0000) 2020-02-05T23:36:55.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:36:55.018+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.020293+0000) 2020-02-05T23:36:55.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:55.019+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.020569+0000) 2020-02-05T23:36:55.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:36:55.020+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.021432+0000) 2020-02-05T23:36:55.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:55.020+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.021998+0000) 2020-02-05T23:36:55.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:36:55.021+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.022224+0000) 2020-02-05T23:36:55.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:55.021+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.022785+0000) 2020-02-05T23:36:55.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:55.022+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.023305+0000) 2020-02-05T23:36:55.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:36:55.022+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.023775+0000) 2020-02-05T23:36:55.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:55.022+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.023728+0000) 2020-02-05T23:36:55.025 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:36:55.023+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.024068+0000) 2020-02-05T23:36:55.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:55.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.026186+0000) 2020-02-05T23:36:55.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:55.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:55.026395+0000) 2020-02-05T23:36:55.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:36:55.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:36:56.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:36:56.047+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:56.047828+0000) 2020-02-05T23:36:57.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:36:57.178+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:57.179363+0000) 2020-02-05T23:36:57.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:36:57.186+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:57.186748+0000) 2020-02-05T23:36:57.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:36:57.186+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:57.186902+0000) 2020-02-05T23:36:57.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:36:57.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:57.187268+0000) 2020-02-05T23:36:59.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:36:59.570+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:36:59.571545+0000) 2020-02-05T23:37:00.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:00.038+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.039445+0000) 2020-02-05T23:37:00.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:00.039+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.040067+0000) 2020-02-05T23:37:00.042 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:00.039+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.040232+0000) 2020-02-05T23:37:00.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:00.039+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.040478+0000) 2020-02-05T23:37:00.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:00.040+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.041126+0000) 2020-02-05T23:37:00.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:00.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.041485+0000) 2020-02-05T23:37:00.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:00.041+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.042025+0000) 2020-02-05T23:37:00.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:00.041+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.042294+0000) 2020-02-05T23:37:00.044 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:00.042+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.043004+0000) 2020-02-05T23:37:00.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:00.044+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.044914+0000) 2020-02-05T23:37:00.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:00.044+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.045169+0000) 2020-02-05T23:37:00.046 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:00.044+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.045318+0000) 2020-02-05T23:37:00.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:00.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.049284+0000) 2020-02-05T23:37:00.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:00.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.049570+0000) 2020-02-05T23:37:00.057 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:00.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:00.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:00.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.178580+0000) 2020-02-05T23:37:00.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:00.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.417977+0000) 2020-02-05T23:37:00.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:00.480+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:00.481034+0000) 2020-02-05T23:37:02.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:02.434+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:02.434927+0000) 2020-02-05T23:37:02.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:02.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:02.785343+0000) 2020-02-05T23:37:03.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:03.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:03.071971+0000) 2020-02-05T23:37:03.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:03.946+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:03.947184+0000) 2020-02-05T23:37:03.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:03.946+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:03.947185+0000) 2020-02-05T23:37:05.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:05.063+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.063651+0000) 2020-02-05T23:37:05.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:05.063+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.064187+0000) 2020-02-05T23:37:05.065 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:05.063+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.064510+0000) 2020-02-05T23:37:05.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:05.063+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.064932+0000) 2020-02-05T23:37:05.066 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:05.064+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.065544+0000) 2020-02-05T23:37:05.066 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:05.064+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.065593+0000) 2020-02-05T23:37:05.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:05.065+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.065773+0000) 2020-02-05T23:37:05.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:05.065+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.066240+0000) 2020-02-05T23:37:05.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:05.066+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.066877+0000) 2020-02-05T23:37:05.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:05.066+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.067027+0000) 2020-02-05T23:37:05.068 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:05.066+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.067253+0000) 2020-02-05T23:37:05.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:05.067+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.067959+0000) 2020-02-05T23:37:05.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:05.068+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.068647+0000) 2020-02-05T23:37:05.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:05.070+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:05.071533+0000) 2020-02-05T23:37:05.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:05.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:06.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:06.047+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:06.048079+0000) 2020-02-05T23:37:07.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:07.179+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:07.179589+0000) 2020-02-05T23:37:07.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:07.186+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:07.187000+0000) 2020-02-05T23:37:07.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:07.186+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:07.187160+0000) 2020-02-05T23:37:07.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:07.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:07.187506+0000) 2020-02-05T23:37:09.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:09.571+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:09.571786+0000) 2020-02-05T23:37:10.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:10.083+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.084753+0000) 2020-02-05T23:37:10.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:10.084+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.085282+0000) 2020-02-05T23:37:10.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:10.086+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.086469+0000) 2020-02-05T23:37:10.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:10.086+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.087158+0000) 2020-02-05T23:37:10.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:10.087+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.087600+0000) 2020-02-05T23:37:10.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:10.087+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.087745+0000) 2020-02-05T23:37:10.089 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:10.087+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.088554+0000) 2020-02-05T23:37:10.090 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:10.088+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.088975+0000) 2020-02-05T23:37:10.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:10.088+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.089142+0000) 2020-02-05T23:37:10.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:10.088+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.089451+0000) 2020-02-05T23:37:10.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:10.089+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.089797+0000) 2020-02-05T23:37:10.091 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:10.089+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.090257+0000) 2020-02-05T23:37:10.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:10.090+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.091971+0000) 2020-02-05T23:37:10.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:10.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.092915+0000) 2020-02-05T23:37:10.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:10.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:10.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:10.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.178783+0000) 2020-02-05T23:37:10.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:10.417+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.418206+0000) 2020-02-05T23:37:10.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:10.489+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:10.489464+0000) 2020-02-05T23:37:12.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:12.434+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:12.435187+0000) 2020-02-05T23:37:12.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:12.784+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:12.785569+0000) 2020-02-05T23:37:13.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:13.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:13.072199+0000) 2020-02-05T23:37:13.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:13.947+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:13.947456+0000) 2020-02-05T23:37:13.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:13.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:13.947456+0000) 2020-02-05T23:37:15.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:15.106+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.107324+0000) 2020-02-05T23:37:15.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:15.106+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.107581+0000) 2020-02-05T23:37:15.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:15.107+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.108507+0000) 2020-02-05T23:37:15.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:15.108+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.108707+0000) 2020-02-05T23:37:15.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:15.107+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.108968+0000) 2020-02-05T23:37:15.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:15.109+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.110040+0000) 2020-02-05T23:37:15.112 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:15.110+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.111400+0000) 2020-02-05T23:37:15.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:15.111+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.111796+0000) 2020-02-05T23:37:15.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:15.111+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.112143+0000) 2020-02-05T23:37:15.114 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:15.112+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.112933+0000) 2020-02-05T23:37:15.115 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:15.113+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.113839+0000) 2020-02-05T23:37:15.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:15.113+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.114333+0000) 2020-02-05T23:37:15.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:15.114+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.115830+0000) 2020-02-05T23:37:15.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:15.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:15.116960+0000) 2020-02-05T23:37:15.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:15.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:16.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:16.048+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:16.048312+0000) 2020-02-05T23:37:17.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:17.179+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:17.179820+0000) 2020-02-05T23:37:17.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:17.187+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:17.187275+0000) 2020-02-05T23:37:17.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:17.187+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:17.187488+0000) 2020-02-05T23:37:17.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:17.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:17.187747+0000) 2020-02-05T23:37:19.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:19.571+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:19.572022+0000) 2020-02-05T23:37:20.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:20.130+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.131233+0000) 2020-02-05T23:37:20.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:20.130+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.131497+0000) 2020-02-05T23:37:20.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:20.131+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.132014+0000) 2020-02-05T23:37:20.134 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:20.130+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.131900+0000) 2020-02-05T23:37:20.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:20.131+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.132690+0000) 2020-02-05T23:37:20.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:20.132+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.133602+0000) 2020-02-05T23:37:20.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:20.133+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.134191+0000) 2020-02-05T23:37:20.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:20.133+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.134381+0000) 2020-02-05T23:37:20.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:20.134+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.134668+0000) 2020-02-05T23:37:20.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:20.135+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.135411+0000) 2020-02-05T23:37:20.137 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:20.135+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.135765+0000) 2020-02-05T23:37:20.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:20.135+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.136064+0000) 2020-02-05T23:37:20.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:20.135+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.135604+0000) 2020-02-05T23:37:20.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:20.139+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.140444+0000) 2020-02-05T23:37:20.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:20.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 149979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:20.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:20.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.179037+0000) 2020-02-05T23:37:20.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:20.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.418481+0000) 2020-02-05T23:37:20.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:20.497+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:20.497668+0000) 2020-02-05T23:37:22.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:22.435+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:22.435498+0000) 2020-02-05T23:37:22.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:22.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:22.785812+0000) 2020-02-05T23:37:23.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:23.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:23.072466+0000) 2020-02-05T23:37:23.826 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:37:23.832 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:37:23.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:23.947+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:23.947732+0000) 2020-02-05T23:37:23.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:23.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:23.947734+0000) 2020-02-05T23:37:25.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:25.152+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.152715+0000) 2020-02-05T23:37:25.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:25.153+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.153931+0000) 2020-02-05T23:37:25.156 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:25.154+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.154790+0000) 2020-02-05T23:37:25.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:25.154+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.155055+0000) 2020-02-05T23:37:25.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:25.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.155513+0000) 2020-02-05T23:37:25.157 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:25.155+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.155839+0000) 2020-02-05T23:37:25.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:25.155+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.155715+0000) 2020-02-05T23:37:25.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:25.155+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.156356+0000) 2020-02-05T23:37:25.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:25.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.156812+0000) 2020-02-05T23:37:25.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:25.156+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.157098+0000) 2020-02-05T23:37:25.159 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:25.156+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.157280+0000) 2020-02-05T23:37:25.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:25.158+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.158698+0000) 2020-02-05T23:37:25.160 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:25.159+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.159531+0000) 2020-02-05T23:37:25.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:25.163+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:25.163203+0000) 2020-02-05T23:37:25.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:25.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:26.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:26.048+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:26.048539+0000) 2020-02-05T23:37:27.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:27.178+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:27.180045+0000) 2020-02-05T23:37:27.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:27.187+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:27.187530+0000) 2020-02-05T23:37:27.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:27.187+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:27.187718+0000) 2020-02-05T23:37:27.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:27.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:27.187973+0000) 2020-02-05T23:37:29.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:29.571+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:29.572310+0000) 2020-02-05T23:37:30.177 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:30.175+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.176426+0000) 2020-02-05T23:37:30.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:30.175+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.176978+0000) 2020-02-05T23:37:30.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:30.176+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.177541+0000) 2020-02-05T23:37:30.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:30.178+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.178628+0000) 2020-02-05T23:37:30.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:30.177+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.178979+0000) 2020-02-05T23:37:30.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:30.178+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.179502+0000) 2020-02-05T23:37:30.181 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:30.178+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.179728+0000) 2020-02-05T23:37:30.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:30.180+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.180475+0000) 2020-02-05T23:37:30.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:30.180+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.181015+0000) 2020-02-05T23:37:30.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:30.181+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.181551+0000) 2020-02-05T23:37:30.183 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:30.181+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.182305+0000) 2020-02-05T23:37:30.183 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:30.181+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.182162+0000) 2020-02-05T23:37:30.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:30.182+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.183073+0000) 2020-02-05T23:37:30.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:30.186+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.187354+0000) 2020-02-05T23:37:30.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:30.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:30.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:30.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.418776+0000) 2020-02-05T23:37:30.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:30.504+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:30.505932+0000) 2020-02-05T23:37:32.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:32.435+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:32.435775+0000) 2020-02-05T23:37:32.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:32.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:32.786045+0000) 2020-02-05T23:37:33.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:33.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:33.072707+0000) 2020-02-05T23:37:33.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:33.947+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:33.948027+0000) 2020-02-05T23:37:33.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:33.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:33.948026+0000) 2020-02-05T23:37:35.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:35.198+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.198824+0000) 2020-02-05T23:37:35.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:35.199+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.200476+0000) 2020-02-05T23:37:35.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:35.201+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.201458+0000) 2020-02-05T23:37:35.218 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:35.201+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.202151+0000) 2020-02-05T23:37:35.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:35.201+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.202611+0000) 2020-02-05T23:37:35.219 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:35.202+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.203335+0000) 2020-02-05T23:37:35.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:35.203+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.204720+0000) 2020-02-05T23:37:35.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:35.203+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.204807+0000) 2020-02-05T23:37:35.220 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:35.202+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.202780+0000) 2020-02-05T23:37:35.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:35.202+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.203396+0000) 2020-02-05T23:37:35.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:35.203+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.204109+0000) 2020-02-05T23:37:35.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:35.203+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.204102+0000) 2020-02-05T23:37:35.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:35.207+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.208909+0000) 2020-02-05T23:37:35.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:35.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:35.209189+0000) 2020-02-05T23:37:35.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:35.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:36.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:36.047+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:36.048768+0000) 2020-02-05T23:37:36.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:36.929+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:36.930211+0000) 2020-02-05T23:37:37.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:37.179+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:37.180322+0000) 2020-02-05T23:37:37.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:37.186+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:37.187767+0000) 2020-02-05T23:37:37.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:37.186+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:37.187997+0000) 2020-02-05T23:37:37.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:37.186+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:37.188223+0000) 2020-02-05T23:37:39.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:39.571+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:39.572537+0000) 2020-02-05T23:37:40.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:40.220+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.222037+0000) 2020-02-05T23:37:40.224 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:40.222+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.223231+0000) 2020-02-05T23:37:40.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:40.222+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.224185+0000) 2020-02-05T23:37:40.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:40.223+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.224352+0000) 2020-02-05T23:37:40.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:40.223+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.224838+0000) 2020-02-05T23:37:40.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:40.224+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.225185+0000) 2020-02-05T23:37:40.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:40.224+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.225857+0000) 2020-02-05T23:37:40.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:40.225+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.227142+0000) 2020-02-05T23:37:40.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:40.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.227286+0000) 2020-02-05T23:37:40.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:40.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.227770+0000) 2020-02-05T23:37:40.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:40.226+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.228091+0000) 2020-02-05T23:37:40.229 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:40.227+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.228250+0000) 2020-02-05T23:37:40.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:40.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.231683+0000) 2020-02-05T23:37:40.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:40.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.231949+0000) 2020-02-05T23:37:40.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:40.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:40.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:40.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.419020+0000) 2020-02-05T23:37:40.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:40.513+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:40.514379+0000) 2020-02-05T23:37:42.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:42.435+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:42.436035+0000) 2020-02-05T23:37:42.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:42.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:42.786312+0000) 2020-02-05T23:37:43.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:43.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:43.072936+0000) 2020-02-05T23:37:43.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:43.947+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:43.948336+0000) 2020-02-05T23:37:43.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:43.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:43.948336+0000) 2020-02-05T23:37:45.246 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:45.243+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.244973+0000) 2020-02-05T23:37:45.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:45.246+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.247689+0000) 2020-02-05T23:37:45.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:45.246+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.248144+0000) 2020-02-05T23:37:45.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:45.247+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.248405+0000) 2020-02-05T23:37:45.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:45.247+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.248993+0000) 2020-02-05T23:37:45.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:45.248+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.249678+0000) 2020-02-05T23:37:45.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:45.248+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.250204+0000) 2020-02-05T23:37:45.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:45.249+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.250809+0000) 2020-02-05T23:37:45.252 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:45.250+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.251143+0000) 2020-02-05T23:37:45.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:45.249+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.251244+0000) 2020-02-05T23:37:45.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:45.250+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.251924+0000) 2020-02-05T23:37:45.253 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:45.250+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.251815+0000) 2020-02-05T23:37:45.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:45.253+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.255006+0000) 2020-02-05T23:37:45.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:45.255+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:45.256114+0000) 2020-02-05T23:37:45.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:45.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:46.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:46.047+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:46.049072+0000) 2020-02-05T23:37:46.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:46.929+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:46.930448+0000) 2020-02-05T23:37:47.181 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:47.179+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:47.180536+0000) 2020-02-05T23:37:47.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:47.187+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:47.188068+0000) 2020-02-05T23:37:47.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:47.187+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:47.188266+0000) 2020-02-05T23:37:47.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:47.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:47.188513+0000) 2020-02-05T23:37:49.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:49.572+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:49.572766+0000) 2020-02-05T23:37:50.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:50.267+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.268489+0000) 2020-02-05T23:37:50.270 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:50.267+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.269038+0000) 2020-02-05T23:37:50.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:50.268+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.269719+0000) 2020-02-05T23:37:50.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:50.269+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.270463+0000) 2020-02-05T23:37:50.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:50.269+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.271205+0000) 2020-02-05T23:37:50.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:50.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.271972+0000) 2020-02-05T23:37:50.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:50.271+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.272206+0000) 2020-02-05T23:37:50.274 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:50.271+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.272350+0000) 2020-02-05T23:37:50.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:50.271+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.273219+0000) 2020-02-05T23:37:50.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:50.273+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.274423+0000) 2020-02-05T23:37:50.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:50.272+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.274393+0000) 2020-02-05T23:37:50.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:50.273+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.274886+0000) 2020-02-05T23:37:50.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:50.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.278638+0000) 2020-02-05T23:37:50.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:50.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.278878+0000) 2020-02-05T23:37:50.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:50.285+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150226 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:50.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:50.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.419255+0000) 2020-02-05T23:37:50.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:50.521+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:50.522501+0000) 2020-02-05T23:37:52.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:52.434+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:52.436289+0000) 2020-02-05T23:37:52.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:52.785+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:52.786545+0000) 2020-02-05T23:37:53.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:53.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:53.073192+0000) 2020-02-05T23:37:53.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:53.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:53.948614+0000) 2020-02-05T23:37:53.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:53.948+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:53.948610+0000) 2020-02-05T23:37:54.027 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:37:54.034 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:37:55.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:37:55.292+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.293330+0000) 2020-02-05T23:37:55.295 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:55.293+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.294163+0000) 2020-02-05T23:37:55.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:55.294+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.295462+0000) 2020-02-05T23:37:55.297 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:37:55.294+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.295919+0000) 2020-02-05T23:37:55.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:55.295+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.296498+0000) 2020-02-05T23:37:55.298 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:37:55.295+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.296997+0000) 2020-02-05T23:37:55.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:55.295+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.297096+0000) 2020-02-05T23:37:55.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:55.295+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.296982+0000) 2020-02-05T23:37:55.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:37:55.296+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.297444+0000) 2020-02-05T23:37:55.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:55.296+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.297613+0000) 2020-02-05T23:37:55.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:55.297+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.298488+0000) 2020-02-05T23:37:55.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:37:55.297+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.299223+0000) 2020-02-05T23:37:55.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:55.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.298448+0000) 2020-02-05T23:37:55.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:55.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:55.302682+0000) 2020-02-05T23:37:55.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:37:55.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:37:56.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:37:56.048+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:56.049320+0000) 2020-02-05T23:37:56.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:37:56.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:56.930690+0000) 2020-02-05T23:37:57.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:37:57.180+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:57.180796+0000) 2020-02-05T23:37:57.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:37:57.187+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:57.188318+0000) 2020-02-05T23:37:57.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:37:57.187+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:57.188505+0000) 2020-02-05T23:37:57.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:37:57.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:57.188773+0000) 2020-02-05T23:37:59.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:37:59.571+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:37:59.573035+0000) 2020-02-05T23:38:00.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:00.314+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.315614+0000) 2020-02-05T23:38:00.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:00.315+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.316241+0000) 2020-02-05T23:38:00.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:00.316+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.317953+0000) 2020-02-05T23:38:00.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:00.317+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.318592+0000) 2020-02-05T23:38:00.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:00.317+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.319043+0000) 2020-02-05T23:38:00.320 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:00.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.319278+0000) 2020-02-05T23:38:00.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:00.318+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.319874+0000) 2020-02-05T23:38:00.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:00.318+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.320093+0000) 2020-02-05T23:38:00.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:00.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.321121+0000) 2020-02-05T23:38:00.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:00.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.321858+0000) 2020-02-05T23:38:00.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:00.320+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.321874+0000) 2020-02-05T23:38:00.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:00.320+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.321967+0000) 2020-02-05T23:38:00.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:00.320+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.322144+0000) 2020-02-05T23:38:00.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:00.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.323899+0000) 2020-02-05T23:38:00.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:00.332+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:00.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:00.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.419492+0000) 2020-02-05T23:38:00.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:00.529+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:00.530439+0000) 2020-02-05T23:38:02.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:02.435+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:02.436549+0000) 2020-02-05T23:38:02.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:02.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:02.786777+0000) 2020-02-05T23:38:03.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:03.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:03.073427+0000) 2020-02-05T23:38:03.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:03.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:03.948874+0000) 2020-02-05T23:38:03.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:03.948+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:03.948872+0000) 2020-02-05T23:38:05.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:05.338+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.339225+0000) 2020-02-05T23:38:05.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:05.338+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.339900+0000) 2020-02-05T23:38:05.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:05.340+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.341100+0000) 2020-02-05T23:38:05.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:05.340+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.341521+0000) 2020-02-05T23:38:05.343 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:05.340+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.341517+0000) 2020-02-05T23:38:05.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:05.340+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.342281+0000) 2020-02-05T23:38:05.343 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:05.341+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.342511+0000) 2020-02-05T23:38:05.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:05.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.343553+0000) 2020-02-05T23:38:05.345 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:05.342+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.343566+0000) 2020-02-05T23:38:05.345 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:05.342+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.343925+0000) 2020-02-05T23:38:05.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:05.343+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.344705+0000) 2020-02-05T23:38:05.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:05.343+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.345096+0000) 2020-02-05T23:38:05.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:05.347+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.348613+0000) 2020-02-05T23:38:05.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:05.347+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:05.348940+0000) 2020-02-05T23:38:05.357 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:05.355+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:06.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:06.048+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:06.049542+0000) 2020-02-05T23:38:06.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:06.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:06.931005+0000) 2020-02-05T23:38:07.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:07.179+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:07.181039+0000) 2020-02-05T23:38:07.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:07.187+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:07.188573+0000) 2020-02-05T23:38:07.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:07.187+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:07.188776+0000) 2020-02-05T23:38:07.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:07.187+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:07.189039+0000) 2020-02-05T23:38:09.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:09.573+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:09.573325+0000) 2020-02-05T23:38:10.364 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:10.361+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.363032+0000) 2020-02-05T23:38:10.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:10.362+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.363638+0000) 2020-02-05T23:38:10.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:10.362+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.364195+0000) 2020-02-05T23:38:10.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:10.363+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.364893+0000) 2020-02-05T23:38:10.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:10.364+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.365255+0000) 2020-02-05T23:38:10.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:10.363+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.364866+0000) 2020-02-05T23:38:10.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:10.365+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.366237+0000) 2020-02-05T23:38:10.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:10.364+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.365892+0000) 2020-02-05T23:38:10.368 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:10.364+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.366304+0000) 2020-02-05T23:38:10.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:10.365+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.367394+0000) 2020-02-05T23:38:10.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:10.366+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.367636+0000) 2020-02-05T23:38:10.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:10.366+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.368220+0000) 2020-02-05T23:38:10.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:10.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.368140+0000) 2020-02-05T23:38:10.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:10.371+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.372388+0000) 2020-02-05T23:38:10.380 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:10.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:10.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:10.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.419766+0000) 2020-02-05T23:38:10.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:10.538+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:10.538464+0000) 2020-02-05T23:38:12.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:12.436+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:12.436819+0000) 2020-02-05T23:38:12.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:12.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:12.787008+0000) 2020-02-05T23:38:13.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:13.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:13.073663+0000) 2020-02-05T23:38:13.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:13.948+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:13.949140+0000) 2020-02-05T23:38:13.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:13.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:13.949140+0000) 2020-02-05T23:38:15.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:15.384+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.385706+0000) 2020-02-05T23:38:15.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:15.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.386389+0000) 2020-02-05T23:38:15.388 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:15.385+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.386808+0000) 2020-02-05T23:38:15.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:15.386+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.387314+0000) 2020-02-05T23:38:15.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:15.386+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.387607+0000) 2020-02-05T23:38:15.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:15.386+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.388280+0000) 2020-02-05T23:38:15.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:15.388+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.389175+0000) 2020-02-05T23:38:15.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:15.387+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.389259+0000) 2020-02-05T23:38:15.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:15.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.389603+0000) 2020-02-05T23:38:15.391 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:15.389+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.390227+0000) 2020-02-05T23:38:15.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:15.389+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.390674+0000) 2020-02-05T23:38:15.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:15.390+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.391319+0000) 2020-02-05T23:38:15.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:15.389+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.391042+0000) 2020-02-05T23:38:15.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:15.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:15.396371+0000) 2020-02-05T23:38:15.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:15.401+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:16.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:16.049+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:16.049817+0000) 2020-02-05T23:38:16.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:16.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:16.931271+0000) 2020-02-05T23:38:17.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:17.181+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:17.181313+0000) 2020-02-05T23:38:17.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:17.188+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:17.188833+0000) 2020-02-05T23:38:17.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:17.188+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:17.189056+0000) 2020-02-05T23:38:17.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:17.188+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:17.189359+0000) 2020-02-05T23:38:19.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:19.573+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:19.573587+0000) 2020-02-05T23:38:20.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:20.407+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.407594+0000) 2020-02-05T23:38:20.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:20.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.408789+0000) 2020-02-05T23:38:20.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:20.406+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.408272+0000) 2020-02-05T23:38:20.410 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:20.409+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.409371+0000) 2020-02-05T23:38:20.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:20.408+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.409899+0000) 2020-02-05T23:38:20.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:20.410+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.410955+0000) 2020-02-05T23:38:20.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:20.411+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.411864+0000) 2020-02-05T23:38:20.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:20.412+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.412418+0000) 2020-02-05T23:38:20.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:20.412+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.412886+0000) 2020-02-05T23:38:20.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:20.411+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.413351+0000) 2020-02-05T23:38:20.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:20.412+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.414311+0000) 2020-02-05T23:38:20.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:20.412+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.414338+0000) 2020-02-05T23:38:20.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:20.413+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.414662+0000) 2020-02-05T23:38:20.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:20.418+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.418104+0000) 2020-02-05T23:38:20.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:20.418+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.420041+0000) 2020-02-05T23:38:20.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:20.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:20.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:20.550+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:20.550518+0000) 2020-02-05T23:38:22.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:22.436+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:22.437107+0000) 2020-02-05T23:38:22.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:22.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:22.787270+0000) 2020-02-05T23:38:23.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:23.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:23.073894+0000) 2020-02-05T23:38:23.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:23.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:23.949369+0000) 2020-02-05T23:38:23.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:23.948+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:23.949365+0000) 2020-02-05T23:38:24.186 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:38:24.192 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:38:25.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:25.429+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.431090+0000) 2020-02-05T23:38:25.433 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:25.430+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.431403+0000) 2020-02-05T23:38:25.433 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:25.431+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.432216+0000) 2020-02-05T23:38:25.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:25.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.432682+0000) 2020-02-05T23:38:25.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:25.433+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.435045+0000) 2020-02-05T23:38:25.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:25.434+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.435497+0000) 2020-02-05T23:38:25.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:25.434+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.435736+0000) 2020-02-05T23:38:25.437 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:25.435+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.436294+0000) 2020-02-05T23:38:25.437 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:25.435+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.436596+0000) 2020-02-05T23:38:25.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:25.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.437584+0000) 2020-02-05T23:38:25.439 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:25.436+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.437923+0000) 2020-02-05T23:38:25.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:25.436+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.438076+0000) 2020-02-05T23:38:25.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:25.436+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.438219+0000) 2020-02-05T23:38:25.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:25.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:25.443231+0000) 2020-02-05T23:38:25.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:25.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:26.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:26.049+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:26.050043+0000) 2020-02-05T23:38:26.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:26.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:26.931496+0000) 2020-02-05T23:38:27.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:27.180+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:27.181570+0000) 2020-02-05T23:38:27.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:27.188+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:27.189106+0000) 2020-02-05T23:38:27.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:27.188+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:27.189260+0000) 2020-02-05T23:38:27.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:27.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:27.189645+0000) 2020-02-05T23:38:29.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:29.572+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:29.573825+0000) 2020-02-05T23:38:30.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:30.419+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.420243+0000) 2020-02-05T23:38:30.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:30.453+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.454260+0000) 2020-02-05T23:38:30.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:30.454+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.455116+0000) 2020-02-05T23:38:30.456 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:30.454+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.455788+0000) 2020-02-05T23:38:30.457 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:30.455+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.456356+0000) 2020-02-05T23:38:30.458 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:30.457+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.457757+0000) 2020-02-05T23:38:30.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:30.457+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.458587+0000) 2020-02-05T23:38:30.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:30.458+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.458703+0000) 2020-02-05T23:38:30.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:30.457+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.458791+0000) 2020-02-05T23:38:30.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:30.458+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.459331+0000) 2020-02-05T23:38:30.461 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:30.459+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.460270+0000) 2020-02-05T23:38:30.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:30.459+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.460205+0000) 2020-02-05T23:38:30.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:30.459+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.460077+0000) 2020-02-05T23:38:30.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:30.459+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.460732+0000) 2020-02-05T23:38:30.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:30.464+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.465357+0000) 2020-02-05T23:38:30.474 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:30.472+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:30.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:30.557+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:30.558573+0000) 2020-02-05T23:38:32.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:32.436+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:32.437362+0000) 2020-02-05T23:38:32.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:32.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:32.787506+0000) 2020-02-05T23:38:33.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:33.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:33.074130+0000) 2020-02-05T23:38:33.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:33.948+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:33.949665+0000) 2020-02-05T23:38:33.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:33.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:33.949665+0000) 2020-02-05T23:38:35.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:35.477+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.478749+0000) 2020-02-05T23:38:35.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:35.478+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.479858+0000) 2020-02-05T23:38:35.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:35.480+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.480659+0000) 2020-02-05T23:38:35.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:35.480+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.481124+0000) 2020-02-05T23:38:35.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:35.480+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.482006+0000) 2020-02-05T23:38:35.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:35.481+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.482064+0000) 2020-02-05T23:38:35.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:35.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.482383+0000) 2020-02-05T23:38:35.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:35.482+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.482737+0000) 2020-02-05T23:38:35.484 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:35.481+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.482851+0000) 2020-02-05T23:38:35.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:35.482+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.483438+0000) 2020-02-05T23:38:35.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:35.484+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.484789+0000) 2020-02-05T23:38:35.486 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:35.485+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.485460+0000) 2020-02-05T23:38:35.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:35.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.485302+0000) 2020-02-05T23:38:35.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:35.488+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:35.489665+0000) 2020-02-05T23:38:35.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:35.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:36.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:36.049+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:36.050296+0000) 2020-02-05T23:38:36.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:36.930+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:36.931720+0000) 2020-02-05T23:38:37.183 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:37.181+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:37.181794+0000) 2020-02-05T23:38:37.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:37.188+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:37.189352+0000) 2020-02-05T23:38:37.190 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:37.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:37.189495+0000) 2020-02-05T23:38:37.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:37.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:37.189856+0000) 2020-02-05T23:38:39.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:39.573+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:39.574070+0000) 2020-02-05T23:38:40.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:40.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.420475+0000) 2020-02-05T23:38:40.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:40.501+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.502191+0000) 2020-02-05T23:38:40.504 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:40.502+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.503523+0000) 2020-02-05T23:38:40.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:40.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.504729+0000) 2020-02-05T23:38:40.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:40.504+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.505968+0000) 2020-02-05T23:38:40.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:40.505+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.506207+0000) 2020-02-05T23:38:40.509 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:40.505+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.505625+0000) 2020-02-05T23:38:40.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:40.506+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.506699+0000) 2020-02-05T23:38:40.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:40.506+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.507238+0000) 2020-02-05T23:38:40.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:40.507+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.507711+0000) 2020-02-05T23:38:40.510 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:40.507+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.508240+0000) 2020-02-05T23:38:40.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:40.508+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.509128+0000) 2020-02-05T23:38:40.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:40.509+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.509486+0000) 2020-02-05T23:38:40.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:40.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.512149+0000) 2020-02-05T23:38:40.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:40.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.512445+0000) 2020-02-05T23:38:40.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:40.520+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:40.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:40.570+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:40.570963+0000) 2020-02-05T23:38:42.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:42.437+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:42.437645+0000) 2020-02-05T23:38:42.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:42.786+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:42.787834+0000) 2020-02-05T23:38:43.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:43.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:43.074378+0000) 2020-02-05T23:38:43.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:43.949+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:43.949924+0000) 2020-02-05T23:38:43.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:43.949+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:43.949926+0000) 2020-02-05T23:38:45.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:45.524+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.526040+0000) 2020-02-05T23:38:45.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:45.526+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.526822+0000) 2020-02-05T23:38:45.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:45.526+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.527228+0000) 2020-02-05T23:38:45.529 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:45.528+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.528489+0000) 2020-02-05T23:38:45.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:45.527+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.528579+0000) 2020-02-05T23:38:45.530 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:45.528+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.529457+0000) 2020-02-05T23:38:45.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:45.529+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.529539+0000) 2020-02-05T23:38:45.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:45.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.529912+0000) 2020-02-05T23:38:45.532 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:45.530+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.531335+0000) 2020-02-05T23:38:45.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:45.531+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.531833+0000) 2020-02-05T23:38:45.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:45.531+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.532179+0000) 2020-02-05T23:38:45.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:45.532+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.532519+0000) 2020-02-05T23:38:45.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:45.532+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.532606+0000) 2020-02-05T23:38:45.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:45.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:45.537418+0000) 2020-02-05T23:38:45.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:45.543+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150677 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:46.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:46.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:46.050520+0000) 2020-02-05T23:38:46.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:46.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:46.931949+0000) 2020-02-05T23:38:47.183 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:47.181+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:47.182070+0000) 2020-02-05T23:38:47.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:47.189+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:47.189630+0000) 2020-02-05T23:38:47.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:47.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:47.189718+0000) 2020-02-05T23:38:47.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:47.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:47.190104+0000) 2020-02-05T23:38:49.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:49.573+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:49.574366+0000) 2020-02-05T23:38:50.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:50.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.420663+0000) 2020-02-05T23:38:50.549 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:50.547+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.548542+0000) 2020-02-05T23:38:50.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:50.548+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.549273+0000) 2020-02-05T23:38:50.551 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:50.548+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.549916+0000) 2020-02-05T23:38:50.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:50.550+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.550762+0000) 2020-02-05T23:38:50.553 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:50.551+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.551698+0000) 2020-02-05T23:38:50.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:50.551+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.552559+0000) 2020-02-05T23:38:50.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:50.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.552598+0000) 2020-02-05T23:38:50.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:50.552+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.553269+0000) 2020-02-05T23:38:50.554 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:50.552+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.553179+0000) 2020-02-05T23:38:50.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:50.552+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.553966+0000) 2020-02-05T23:38:50.556 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:50.554+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.554894+0000) 2020-02-05T23:38:50.556 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:50.554+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.555009+0000) 2020-02-05T23:38:50.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:50.553+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.554937+0000) 2020-02-05T23:38:50.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:50.558+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.559494+0000) 2020-02-05T23:38:50.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:50.564+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:50.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:50.579+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:50.579540+0000) 2020-02-05T23:38:52.439 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:52.437+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:52.437902+0000) 2020-02-05T23:38:52.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:52.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:52.788126+0000) 2020-02-05T23:38:53.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:53.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:53.074611+0000) 2020-02-05T23:38:53.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:53.949+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:53.950208+0000) 2020-02-05T23:38:53.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:53.949+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:53.950208+0000) 2020-02-05T23:38:54.348 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:38:54.355 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:38:55.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:55.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.570107+0000) 2020-02-05T23:38:55.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:55.570+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.571220+0000) 2020-02-05T23:38:55.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:55.570+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.571442+0000) 2020-02-05T23:38:55.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:55.570+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.572030+0000) 2020-02-05T23:38:55.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:55.571+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.572204+0000) 2020-02-05T23:38:55.573 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:38:55.571+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.572535+0000) 2020-02-05T23:38:55.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:38:55.572+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.573285+0000) 2020-02-05T23:38:55.574 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:38:55.572+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.573185+0000) 2020-02-05T23:38:55.575 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:55.573+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.574040+0000) 2020-02-05T23:38:55.575 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:55.573+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.574145+0000) 2020-02-05T23:38:55.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:55.574+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.574422+0000) 2020-02-05T23:38:55.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:38:55.574+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.574825+0000) 2020-02-05T23:38:55.577 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:38:55.574+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.576026+0000) 2020-02-05T23:38:55.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:55.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:55.578380+0000) 2020-02-05T23:38:55.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:38:55.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:38:56.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:38:56.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:56.050691+0000) 2020-02-05T23:38:56.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:38:56.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:56.932186+0000) 2020-02-05T23:38:57.183 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:38:57.181+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:57.182321+0000) 2020-02-05T23:38:57.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:38:57.189+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:57.189889+0000) 2020-02-05T23:38:57.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:38:57.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:57.189982+0000) 2020-02-05T23:38:57.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:38:57.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:57.190316+0000) 2020-02-05T23:38:59.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:38:59.573+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:38:59.574618+0000) 2020-02-05T23:39:00.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:00.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.420914+0000) 2020-02-05T23:39:00.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:00.584+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.584756+0000) 2020-02-05T23:39:00.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:00.593+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.593522+0000) 2020-02-05T23:39:00.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:00.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.594317+0000) 2020-02-05T23:39:00.596 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:00.594+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.595124+0000) 2020-02-05T23:39:00.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:00.594+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.595671+0000) 2020-02-05T23:39:00.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:00.594+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.596012+0000) 2020-02-05T23:39:00.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:00.596+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.596446+0000) 2020-02-05T23:39:00.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:00.596+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.596449+0000) 2020-02-05T23:39:00.598 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:00.595+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.596805+0000) 2020-02-05T23:39:00.599 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:00.596+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.597154+0000) 2020-02-05T23:39:00.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:00.596+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.596932+0000) 2020-02-05T23:39:00.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:00.597+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.597502+0000) 2020-02-05T23:39:00.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:00.597+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.598385+0000) 2020-02-05T23:39:00.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:00.597+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.598103+0000) 2020-02-05T23:39:00.616 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:00.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:00.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:00.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:00.603613+0000) 2020-02-05T23:39:02.439 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:02.438+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:02.438140+0000) 2020-02-05T23:39:02.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:02.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:02.788351+0000) 2020-02-05T23:39:03.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:03.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:03.074848+0000) 2020-02-05T23:39:03.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:03.950+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:03.950465+0000) 2020-02-05T23:39:03.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:03.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:03.950467+0000) 2020-02-05T23:39:05.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:05.615+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.616808+0000) 2020-02-05T23:39:05.619 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:05.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.617059+0000) 2020-02-05T23:39:05.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:05.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.617361+0000) 2020-02-05T23:39:05.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:05.616+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.617705+0000) 2020-02-05T23:39:05.621 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:05.618+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.620116+0000) 2020-02-05T23:39:05.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:05.619+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.620218+0000) 2020-02-05T23:39:05.622 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:05.619+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.620792+0000) 2020-02-05T23:39:05.622 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:05.620+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.621205+0000) 2020-02-05T23:39:05.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:05.620+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.622055+0000) 2020-02-05T23:39:05.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:05.621+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.622209+0000) 2020-02-05T23:39:05.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:05.621+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.622847+0000) 2020-02-05T23:39:05.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:05.621+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.622839+0000) 2020-02-05T23:39:05.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:05.621+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.623013+0000) 2020-02-05T23:39:05.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:05.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:05.626223+0000) 2020-02-05T23:39:05.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:05.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:06.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:06.049+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:06.050907+0000) 2020-02-05T23:39:06.933 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:06.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:06.932424+0000) 2020-02-05T23:39:07.183 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:07.182+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:07.182592+0000) 2020-02-05T23:39:07.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:07.189+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:07.190149+0000) 2020-02-05T23:39:07.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:07.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:07.190247+0000) 2020-02-05T23:39:07.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:07.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:07.190602+0000) 2020-02-05T23:39:09.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:09.574+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:09.574877+0000) 2020-02-05T23:39:10.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:10.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.421167+0000) 2020-02-05T23:39:10.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:10.594+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.594310+0000) 2020-02-05T23:39:10.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:10.636+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.637375+0000) 2020-02-05T23:39:10.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:10.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.639561+0000) 2020-02-05T23:39:10.641 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:10.638+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.639541+0000) 2020-02-05T23:39:10.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:10.638+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.639863+0000) 2020-02-05T23:39:10.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:10.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.640380+0000) 2020-02-05T23:39:10.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:10.639+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.640831+0000) 2020-02-05T23:39:10.642 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:10.640+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.641340+0000) 2020-02-05T23:39:10.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:10.640+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.642057+0000) 2020-02-05T23:39:10.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:10.641+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.642549+0000) 2020-02-05T23:39:10.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:10.640+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.641958+0000) 2020-02-05T23:39:10.644 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:10.641+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.642521+0000) 2020-02-05T23:39:10.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:10.641+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.642852+0000) 2020-02-05T23:39:10.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:10.641+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.643099+0000) 2020-02-05T23:39:10.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:10.647+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:10.648117+0000) 2020-02-05T23:39:10.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:10.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:12.439 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:12.437+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:12.438382+0000) 2020-02-05T23:39:12.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:12.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:12.788611+0000) 2020-02-05T23:39:13.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:13.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:13.075112+0000) 2020-02-05T23:39:13.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:13.950+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:13.950739+0000) 2020-02-05T23:39:13.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:13.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:13.950739+0000) 2020-02-05T23:39:15.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:15.663+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.663893+0000) 2020-02-05T23:39:15.666 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:15.664+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.664772+0000) 2020-02-05T23:39:15.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:15.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.664941+0000) 2020-02-05T23:39:15.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:15.664+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.665647+0000) 2020-02-05T23:39:15.667 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:15.665+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.665908+0000) 2020-02-05T23:39:15.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:15.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.666484+0000) 2020-02-05T23:39:15.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:15.665+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.666848+0000) 2020-02-05T23:39:15.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:15.666+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.666838+0000) 2020-02-05T23:39:15.668 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:15.665+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.667370+0000) 2020-02-05T23:39:15.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:15.667+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.667993+0000) 2020-02-05T23:39:15.670 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:15.666+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.668378+0000) 2020-02-05T23:39:15.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:15.667+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.668540+0000) 2020-02-05T23:39:15.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:15.667+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.669224+0000) 2020-02-05T23:39:15.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:15.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:15.674009+0000) 2020-02-05T23:39:15.681 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:15.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150922 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:16.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:16.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:16.051149+0000) 2020-02-05T23:39:16.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:16.931+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:16.932690+0000) 2020-02-05T23:39:17.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:17.181+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:17.182867+0000) 2020-02-05T23:39:17.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:17.188+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:17.190355+0000) 2020-02-05T23:39:17.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:17.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:17.190496+0000) 2020-02-05T23:39:17.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:17.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:17.190838+0000) 2020-02-05T23:39:19.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:19.574+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:19.575137+0000) 2020-02-05T23:39:20.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:20.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.421416+0000) 2020-02-05T23:39:20.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:20.600+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.601798+0000) 2020-02-05T23:39:20.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:20.684+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.685837+0000) 2020-02-05T23:39:20.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:20.685+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.686257+0000) 2020-02-05T23:39:20.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:20.686+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.687454+0000) 2020-02-05T23:39:20.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:20.687+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.688795+0000) 2020-02-05T23:39:20.690 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:20.687+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.689398+0000) 2020-02-05T23:39:20.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:20.689+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.690218+0000) 2020-02-05T23:39:20.692 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:20.688+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.690340+0000) 2020-02-05T23:39:20.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:20.689+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.690595+0000) 2020-02-05T23:39:20.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:20.689+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.690715+0000) 2020-02-05T23:39:20.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:20.689+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.690932+0000) 2020-02-05T23:39:20.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:20.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.691583+0000) 2020-02-05T23:39:20.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:20.689+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.691404+0000) 2020-02-05T23:39:20.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:20.690+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.691734+0000) 2020-02-05T23:39:20.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:20.694+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:20.695567+0000) 2020-02-05T23:39:20.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:20.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 150963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:22.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:22.437+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:22.438684+0000) 2020-02-05T23:39:22.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:22.787+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:22.788836+0000) 2020-02-05T23:39:23.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:23.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:23.075383+0000) 2020-02-05T23:39:23.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:23.950+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:23.950974+0000) 2020-02-05T23:39:23.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:23.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:23.950978+0000) 2020-02-05T23:39:24.545 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:39:24.552 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:39:25.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:25.705+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.706908+0000) 2020-02-05T23:39:25.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:25.706+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.707513+0000) 2020-02-05T23:39:25.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:25.707+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.708228+0000) 2020-02-05T23:39:25.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:25.707+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.708589+0000) 2020-02-05T23:39:25.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:25.707+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.709019+0000) 2020-02-05T23:39:25.711 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:25.708+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.709706+0000) 2020-02-05T23:39:25.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:25.708+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.709653+0000) 2020-02-05T23:39:25.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:25.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.710459+0000) 2020-02-05T23:39:25.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:25.708+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.709850+0000) 2020-02-05T23:39:25.715 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:25.709+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.710581+0000) 2020-02-05T23:39:25.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:25.709+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.710960+0000) 2020-02-05T23:39:25.715 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:25.709+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.711402+0000) 2020-02-05T23:39:25.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:25.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.715172+0000) 2020-02-05T23:39:25.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:25.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:25.716713+0000) 2020-02-05T23:39:25.724 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:25.721+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:26.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:26.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:26.051360+0000) 2020-02-05T23:39:26.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:26.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:26.932938+0000) 2020-02-05T23:39:27.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:27.182+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:27.183123+0000) 2020-02-05T23:39:27.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:27.189+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:27.190616+0000) 2020-02-05T23:39:27.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:27.189+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:27.190749+0000) 2020-02-05T23:39:27.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:27.189+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:27.191093+0000) 2020-02-05T23:39:29.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:29.574+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:29.575380+0000) 2020-02-05T23:39:30.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:30.420+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.421690+0000) 2020-02-05T23:39:30.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:30.609+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.610625+0000) 2020-02-05T23:39:30.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:30.727+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.728656+0000) 2020-02-05T23:39:30.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:30.728+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.729455+0000) 2020-02-05T23:39:30.731 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:30.728+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.729980+0000) 2020-02-05T23:39:30.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:30.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.730265+0000) 2020-02-05T23:39:30.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:30.729+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.730594+0000) 2020-02-05T23:39:30.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:30.729+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.730929+0000) 2020-02-05T23:39:30.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:30.730+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.731353+0000) 2020-02-05T23:39:30.733 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:30.730+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.732237+0000) 2020-02-05T23:39:30.734 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:30.731+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.733390+0000) 2020-02-05T23:39:30.735 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:30.732+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.733955+0000) 2020-02-05T23:39:30.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:30.732+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.734229+0000) 2020-02-05T23:39:30.735 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:30.732+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.734381+0000) 2020-02-05T23:39:30.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:30.734+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.735250+0000) 2020-02-05T23:39:30.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:30.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:30.741436+0000) 2020-02-05T23:39:30.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:30.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:32.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:32.437+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:32.438922+0000) 2020-02-05T23:39:32.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:32.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:32.789065+0000) 2020-02-05T23:39:33.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:33.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:33.075600+0000) 2020-02-05T23:39:33.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:33.950+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:33.951314+0000) 2020-02-05T23:39:33.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:33.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:33.951314+0000) 2020-02-05T23:39:35.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:35.752+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.754383+0000) 2020-02-05T23:39:35.756 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:35.754+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.755279+0000) 2020-02-05T23:39:35.756 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:35.753+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.755006+0000) 2020-02-05T23:39:35.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:35.754+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.755714+0000) 2020-02-05T23:39:35.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:35.754+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.756311+0000) 2020-02-05T23:39:35.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:35.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.756421+0000) 2020-02-05T23:39:35.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:35.755+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.756748+0000) 2020-02-05T23:39:35.758 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:35.755+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.757122+0000) 2020-02-05T23:39:35.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:35.757+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.758307+0000) 2020-02-05T23:39:35.760 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:35.757+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.759078+0000) 2020-02-05T23:39:35.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:35.758+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.759531+0000) 2020-02-05T23:39:35.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:35.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.760058+0000) 2020-02-05T23:39:35.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:35.758+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.760087+0000) 2020-02-05T23:39:35.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:35.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:35.763949+0000) 2020-02-05T23:39:35.773 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:35.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:36.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:36.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:36.051588+0000) 2020-02-05T23:39:36.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:36.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:36.933169+0000) 2020-02-05T23:39:37.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:37.182+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:37.183357+0000) 2020-02-05T23:39:37.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:37.190+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:37.190860+0000) 2020-02-05T23:39:37.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:37.190+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:37.191001+0000) 2020-02-05T23:39:37.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:37.190+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:37.191356+0000) 2020-02-05T23:39:39.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:39.574+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:39.575565+0000) 2020-02-05T23:39:40.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:40.421+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.421931+0000) 2020-02-05T23:39:40.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:40.621+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.622421+0000) 2020-02-05T23:39:40.779 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:40.776+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.777882+0000) 2020-02-05T23:39:40.779 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:40.776+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.778308+0000) 2020-02-05T23:39:40.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:40.778+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.779657+0000) 2020-02-05T23:39:40.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:40.779+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.780159+0000) 2020-02-05T23:39:40.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:40.779+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.780938+0000) 2020-02-05T23:39:40.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:40.780+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.781149+0000) 2020-02-05T23:39:40.782 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:40.780+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.781669+0000) 2020-02-05T23:39:40.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:40.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.781724+0000) 2020-02-05T23:39:40.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:40.781+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.782070+0000) 2020-02-05T23:39:40.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:40.781+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.782939+0000) 2020-02-05T23:39:40.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:40.781+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.783181+0000) 2020-02-05T23:39:40.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:40.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.783337+0000) 2020-02-05T23:39:40.785 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:40.782+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.783943+0000) 2020-02-05T23:39:40.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:40.788+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:40.789301+0000) 2020-02-05T23:39:40.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:40.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:42.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:42.438+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:42.439173+0000) 2020-02-05T23:39:42.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:42.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:42.789344+0000) 2020-02-05T23:39:43.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:43.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:43.075805+0000) 2020-02-05T23:39:43.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:43.951+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:43.951572+0000) 2020-02-05T23:39:43.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:43.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:43.951572+0000) 2020-02-05T23:39:45.803 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:45.800+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.802131+0000) 2020-02-05T23:39:45.804 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:45.801+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.802709+0000) 2020-02-05T23:39:45.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:45.801+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.802991+0000) 2020-02-05T23:39:45.805 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:45.801+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.803379+0000) 2020-02-05T23:39:45.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:45.802+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.803873+0000) 2020-02-05T23:39:45.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:45.802+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.803945+0000) 2020-02-05T23:39:45.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:45.803+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.804455+0000) 2020-02-05T23:39:45.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:45.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.804430+0000) 2020-02-05T23:39:45.807 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:45.803+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.805076+0000) 2020-02-05T23:39:45.807 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:45.804+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.806018+0000) 2020-02-05T23:39:45.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:45.804+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.805458+0000) 2020-02-05T23:39:45.807 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:45.805+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.806133+0000) 2020-02-05T23:39:45.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:45.809+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.811067+0000) 2020-02-05T23:39:45.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:45.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:45.811313+0000) 2020-02-05T23:39:45.818 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:45.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:46.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:46.050+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:46.051800+0000) 2020-02-05T23:39:46.934 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:46.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:46.933438+0000) 2020-02-05T23:39:47.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:47.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:47.183586+0000) 2020-02-05T23:39:47.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:47.190+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:47.191100+0000) 2020-02-05T23:39:47.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:47.190+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:47.191277+0000) 2020-02-05T23:39:47.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:47.190+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:47.191644+0000) 2020-02-05T23:39:49.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:49.575+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:49.575838+0000) 2020-02-05T23:39:50.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:50.421+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.422188+0000) 2020-02-05T23:39:50.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:50.629+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.630332+0000) 2020-02-05T23:39:50.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:50.822+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.823978+0000) 2020-02-05T23:39:50.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:50.823+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.824391+0000) 2020-02-05T23:39:50.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:50.823+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.824802+0000) 2020-02-05T23:39:50.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:50.824+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.825330+0000) 2020-02-05T23:39:50.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:50.826+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.827398+0000) 2020-02-05T23:39:50.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:50.826+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.828047+0000) 2020-02-05T23:39:50.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:50.828+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.829350+0000) 2020-02-05T23:39:50.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:50.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.825892+0000) 2020-02-05T23:39:50.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:50.825+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.826558+0000) 2020-02-05T23:39:50.840 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:50.826+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.827578+0000) 2020-02-05T23:39:50.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:50.827+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.828644+0000) 2020-02-05T23:39:50.840 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:50.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.828839+0000) 2020-02-05T23:39:50.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:50.827+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.828879+0000) 2020-02-05T23:39:50.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:50.833+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:50.834219+0000) 2020-02-05T23:39:50.841 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:50.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151209 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:52.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:52.438+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:52.439454+0000) 2020-02-05T23:39:52.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:52.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:52.789567+0000) 2020-02-05T23:39:53.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:53.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:53.076046+0000) 2020-02-05T23:39:53.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:53.950+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:53.951863+0000) 2020-02-05T23:39:53.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:53.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:53.951866+0000) 2020-02-05T23:39:54.733 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:39:54.739 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:39:55.847 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:39:55.845+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.845973+0000) 2020-02-05T23:39:55.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:55.845+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.846636+0000) 2020-02-05T23:39:55.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:55.846+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.847207+0000) 2020-02-05T23:39:55.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:39:55.846+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.847376+0000) 2020-02-05T23:39:55.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:39:55.846+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.847848+0000) 2020-02-05T23:39:55.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:55.847+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.848313+0000) 2020-02-05T23:39:55.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:39:55.847+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.848862+0000) 2020-02-05T23:39:55.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:55.847+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.849004+0000) 2020-02-05T23:39:55.851 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:55.848+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.849435+0000) 2020-02-05T23:39:55.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:55.849+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.849912+0000) 2020-02-05T23:39:55.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:39:55.850+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.850606+0000) 2020-02-05T23:39:55.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:55.850+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.850465+0000) 2020-02-05T23:39:55.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:55.851+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.851679+0000) 2020-02-05T23:39:55.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:55.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:55.853979+0000) 2020-02-05T23:39:55.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:39:55.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151248 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:39:56.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:39:56.051+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:56.052077+0000) 2020-02-05T23:39:56.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:39:56.932+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:56.933676+0000) 2020-02-05T23:39:57.185 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:39:57.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:57.183815+0000) 2020-02-05T23:39:57.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:39:57.190+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:57.191350+0000) 2020-02-05T23:39:57.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:39:57.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:57.191517+0000) 2020-02-05T23:39:57.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:39:57.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:57.191866+0000) 2020-02-05T23:39:59.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:39:59.575+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:39:59.576166+0000) 2020-02-05T23:40:00.423 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:00.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.422419+0000) 2020-02-05T23:40:00.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:00.638+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.638309+0000) 2020-02-05T23:40:00.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:00.865+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.866586+0000) 2020-02-05T23:40:00.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:00.866+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.866461+0000) 2020-02-05T23:40:00.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:00.866+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.867110+0000) 2020-02-05T23:40:00.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:00.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.867475+0000) 2020-02-05T23:40:00.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:00.868+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.869343+0000) 2020-02-05T23:40:00.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:00.869+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.870248+0000) 2020-02-05T23:40:00.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:00.870+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.871159+0000) 2020-02-05T23:40:00.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:00.870+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.871453+0000) 2020-02-05T23:40:00.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:00.871+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.872024+0000) 2020-02-05T23:40:00.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:00.871+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.872080+0000) 2020-02-05T23:40:00.874 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:00.870+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.871890+0000) 2020-02-05T23:40:00.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:00.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.871960+0000) 2020-02-05T23:40:00.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:00.872+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.872604+0000) 2020-02-05T23:40:00.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:00.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:00.878334+0000) 2020-02-05T23:40:00.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:00.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:02.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:02.439+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:02.439712+0000) 2020-02-05T23:40:02.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:02.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:02.789787+0000) 2020-02-05T23:40:03.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:03.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:03.076316+0000) 2020-02-05T23:40:03.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:03.951+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:03.952139+0000) 2020-02-05T23:40:03.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:03.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:03.952139+0000) 2020-02-05T23:40:05.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:05.888+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.888982+0000) 2020-02-05T23:40:05.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:05.888+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.888572+0000) 2020-02-05T23:40:05.891 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:05.889+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.889903+0000) 2020-02-05T23:40:05.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:05.890+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.890734+0000) 2020-02-05T23:40:05.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:05.890+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.890906+0000) 2020-02-05T23:40:05.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:05.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.891088+0000) 2020-02-05T23:40:05.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:05.891+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.891591+0000) 2020-02-05T23:40:05.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:05.891+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.891720+0000) 2020-02-05T23:40:05.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:05.891+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.892052+0000) 2020-02-05T23:40:05.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:05.892+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.892715+0000) 2020-02-05T23:40:05.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:05.892+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.892548+0000) 2020-02-05T23:40:05.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:05.893+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.893618+0000) 2020-02-05T23:40:05.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:05.892+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.893159+0000) 2020-02-05T23:40:05.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:05.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:05.899071+0000) 2020-02-05T23:40:05.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:05.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151330 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:06.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:06.051+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:06.052299+0000) 2020-02-05T23:40:06.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:06.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:06.933910+0000) 2020-02-05T23:40:07.185 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:07.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:07.184050+0000) 2020-02-05T23:40:07.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:07.191+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:07.191639+0000) 2020-02-05T23:40:07.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:07.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:07.191755+0000) 2020-02-05T23:40:07.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:07.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:07.192107+0000) 2020-02-05T23:40:09.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:09.576+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:09.576428+0000) 2020-02-05T23:40:10.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:10.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.422726+0000) 2020-02-05T23:40:10.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:10.645+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.646546+0000) 2020-02-05T23:40:10.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:10.909+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.910996+0000) 2020-02-05T23:40:10.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:10.910+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.911601+0000) 2020-02-05T23:40:10.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:10.911+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.912558+0000) 2020-02-05T23:40:10.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:10.911+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.912904+0000) 2020-02-05T23:40:10.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:10.912+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.913210+0000) 2020-02-05T23:40:10.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:10.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.913639+0000) 2020-02-05T23:40:10.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:10.913+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.914209+0000) 2020-02-05T23:40:10.916 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:10.913+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.914820+0000) 2020-02-05T23:40:10.916 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:10.914+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.915234+0000) 2020-02-05T23:40:10.917 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:10.915+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.915791+0000) 2020-02-05T23:40:10.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:10.916+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.916853+0000) 2020-02-05T23:40:10.918 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:10.917+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.917629+0000) 2020-02-05T23:40:10.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:10.916+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.917500+0000) 2020-02-05T23:40:10.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:10.920+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:10.921935+0000) 2020-02-05T23:40:10.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:10.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151373 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:12.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:12.439+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:12.440024+0000) 2020-02-05T23:40:12.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:12.788+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:12.790031+0000) 2020-02-05T23:40:13.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:13.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:13.076575+0000) 2020-02-05T23:40:13.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:13.951+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:13.952399+0000) 2020-02-05T23:40:13.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:13.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:13.952399+0000) 2020-02-05T23:40:15.935 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:15.933+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.933968+0000) 2020-02-05T23:40:15.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:15.933+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.934719+0000) 2020-02-05T23:40:15.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:15.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.935288+0000) 2020-02-05T23:40:15.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:15.934+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.935561+0000) 2020-02-05T23:40:15.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:15.935+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.936444+0000) 2020-02-05T23:40:15.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:15.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.937239+0000) 2020-02-05T23:40:15.938 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:15.936+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.937354+0000) 2020-02-05T23:40:15.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:15.936+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.937966+0000) 2020-02-05T23:40:15.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:15.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.938822+0000) 2020-02-05T23:40:15.940 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:15.938+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.939264+0000) 2020-02-05T23:40:15.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:15.939+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.939618+0000) 2020-02-05T23:40:15.941 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:15.939+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.940178+0000) 2020-02-05T23:40:15.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:15.939+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.940232+0000) 2020-02-05T23:40:15.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:15.943+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:15.944149+0000) 2020-02-05T23:40:15.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:15.951+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151412 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:16.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:16.052+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:16.052517+0000) 2020-02-05T23:40:16.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:16.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:16.934133+0000) 2020-02-05T23:40:17.185 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:17.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:17.184329+0000) 2020-02-05T23:40:17.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:17.191+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:17.191872+0000) 2020-02-05T23:40:17.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:17.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:17.192040+0000) 2020-02-05T23:40:17.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:17.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:17.192329+0000) 2020-02-05T23:40:19.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:19.575+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:19.576673+0000) 2020-02-05T23:40:20.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:20.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.422970+0000) 2020-02-05T23:40:20.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:20.654+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.654972+0000) 2020-02-05T23:40:20.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:20.955+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.956668+0000) 2020-02-05T23:40:20.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:20.957+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.957552+0000) 2020-02-05T23:40:20.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:20.958+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.959307+0000) 2020-02-05T23:40:20.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:20.958+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.959800+0000) 2020-02-05T23:40:20.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:20.959+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.959600+0000) 2020-02-05T23:40:20.962 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:20.960+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.961147+0000) 2020-02-05T23:40:20.962 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:20.960+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.960934+0000) 2020-02-05T23:40:20.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:20.960+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.961496+0000) 2020-02-05T23:40:20.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:20.960+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.962014+0000) 2020-02-05T23:40:20.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:20.962+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.962720+0000) 2020-02-05T23:40:20.964 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:20.961+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.962570+0000) 2020-02-05T23:40:20.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:20.962+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.963203+0000) 2020-02-05T23:40:20.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:20.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.969027+0000) 2020-02-05T23:40:20.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:20.969+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:20.970298+0000) 2020-02-05T23:40:20.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:20.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151455 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:22.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:22.440+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:22.440280+0000) 2020-02-05T23:40:22.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:22.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:22.790283+0000) 2020-02-05T23:40:23.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:23.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:23.076802+0000) 2020-02-05T23:40:23.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:23.951+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:23.952682+0000) 2020-02-05T23:40:23.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:23.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:23.952681+0000) 2020-02-05T23:40:24.867 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:40:24.873 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:40:25.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:25.978+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.979364+0000) 2020-02-05T23:40:25.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:25.978+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.979938+0000) 2020-02-05T23:40:25.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:25.980+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.981263+0000) 2020-02-05T23:40:25.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:25.981+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.982730+0000) 2020-02-05T23:40:25.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:25.982+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.982994+0000) 2020-02-05T23:40:25.984 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:25.982+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.983530+0000) 2020-02-05T23:40:25.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:25.983+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.983595+0000) 2020-02-05T23:40:25.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:25.983+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.984412+0000) 2020-02-05T23:40:25.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:25.983+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.984082+0000) 2020-02-05T23:40:25.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:25.984+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.985292+0000) 2020-02-05T23:40:25.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:25.985+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.985419+0000) 2020-02-05T23:40:25.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:25.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.986196+0000) 2020-02-05T23:40:25.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:25.985+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.986133+0000) 2020-02-05T23:40:25.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:25.989+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:25.990733+0000) 2020-02-05T23:40:25.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:25.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:26.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:26.052+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:26.052731+0000) 2020-02-05T23:40:26.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:26.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:26.934345+0000) 2020-02-05T23:40:27.185 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:27.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:27.184575+0000) 2020-02-05T23:40:27.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:27.191+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:27.192128+0000) 2020-02-05T23:40:27.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:27.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:27.192317+0000) 2020-02-05T23:40:27.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:27.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:27.192677+0000) 2020-02-05T23:40:29.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:29.575+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:29.576944+0000) 2020-02-05T23:40:30.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:30.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:30.423229+0000) 2020-02-05T23:40:30.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:30.662+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:30.663182+0000) 2020-02-05T23:40:31.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:31.002+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.003768+0000) 2020-02-05T23:40:31.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:31.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.004619+0000) 2020-02-05T23:40:31.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:31.003+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.004927+0000) 2020-02-05T23:40:31.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:31.005+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.006056+0000) 2020-02-05T23:40:31.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:31.006+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.007514+0000) 2020-02-05T23:40:31.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:31.007+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.008490+0000) 2020-02-05T23:40:31.010 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:31.007+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.008039+0000) 2020-02-05T23:40:31.010 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:31.008+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.009344+0000) 2020-02-05T23:40:31.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:31.009+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.009831+0000) 2020-02-05T23:40:31.011 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:31.009+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.010115+0000) 2020-02-05T23:40:31.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:31.010+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.010565+0000) 2020-02-05T23:40:31.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:31.010+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.010966+0000) 2020-02-05T23:40:31.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:31.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.011299+0000) 2020-02-05T23:40:31.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:31.012+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:31.013086+0000) 2020-02-05T23:40:31.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:31.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:32.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:32.440+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:32.440514+0000) 2020-02-05T23:40:32.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:32.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:32.790521+0000) 2020-02-05T23:40:33.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:33.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:33.077051+0000) 2020-02-05T23:40:33.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:33.952+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:33.952985+0000) 2020-02-05T23:40:33.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:33.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:33.952985+0000) 2020-02-05T23:40:36.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:36.026+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.027263+0000) 2020-02-05T23:40:36.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:36.026+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.028085+0000) 2020-02-05T23:40:36.029 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:36.027+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.028447+0000) 2020-02-05T23:40:36.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:36.028+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.029201+0000) 2020-02-05T23:40:36.030 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:36.028+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.029679+0000) 2020-02-05T23:40:36.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:36.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.030392+0000) 2020-02-05T23:40:36.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:36.029+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.030649+0000) 2020-02-05T23:40:36.032 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:36.030+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.031422+0000) 2020-02-05T23:40:36.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:36.031+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.032411+0000) 2020-02-05T23:40:36.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:36.031+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.032738+0000) 2020-02-05T23:40:36.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:36.032+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.033615+0000) 2020-02-05T23:40:36.034 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:36.032+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.033687+0000) 2020-02-05T23:40:36.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:36.032+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.033978+0000) 2020-02-05T23:40:36.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:36.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.037095+0000) 2020-02-05T23:40:36.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:36.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:36.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:36.051+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.052898+0000) 2020-02-05T23:40:36.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:36.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:36.934557+0000) 2020-02-05T23:40:37.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:37.183+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:37.184815+0000) 2020-02-05T23:40:37.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:37.191+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:37.192390+0000) 2020-02-05T23:40:37.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:37.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:37.192541+0000) 2020-02-05T23:40:37.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:37.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:37.192923+0000) 2020-02-05T23:40:39.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:39.576+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:39.577180+0000) 2020-02-05T23:40:40.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:40.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:40.423526+0000) 2020-02-05T23:40:40.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:40.670+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:40.671066+0000) 2020-02-05T23:40:41.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:41.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.052278+0000) 2020-02-05T23:40:41.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:41.051+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.052632+0000) 2020-02-05T23:40:41.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:41.052+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.053290+0000) 2020-02-05T23:40:41.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:41.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.053673+0000) 2020-02-05T23:40:41.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:41.052+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.053956+0000) 2020-02-05T23:40:41.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:41.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.054431+0000) 2020-02-05T23:40:41.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:41.053+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.054721+0000) 2020-02-05T23:40:41.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:41.053+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.055024+0000) 2020-02-05T23:40:41.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:41.054+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.055787+0000) 2020-02-05T23:40:41.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:41.054+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.055905+0000) 2020-02-05T23:40:41.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:41.054+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.055896+0000) 2020-02-05T23:40:41.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:41.054+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.056333+0000) 2020-02-05T23:40:41.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:41.055+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.057076+0000) 2020-02-05T23:40:41.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:41.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:41.062396+0000) 2020-02-05T23:40:41.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:41.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:42.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:42.439+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:42.440782+0000) 2020-02-05T23:40:42.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:42.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:42.790759+0000) 2020-02-05T23:40:43.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:43.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:43.077346+0000) 2020-02-05T23:40:43.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:43.952+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:43.953302+0000) 2020-02-05T23:40:43.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:43.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:43.953302+0000) 2020-02-05T23:40:46.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:46.051+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.053117+0000) 2020-02-05T23:40:46.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:46.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.074336+0000) 2020-02-05T23:40:46.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:46.073+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.074669+0000) 2020-02-05T23:40:46.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:46.074+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.075223+0000) 2020-02-05T23:40:46.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:46.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.075716+0000) 2020-02-05T23:40:46.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:46.075+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.076790+0000) 2020-02-05T23:40:46.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:46.075+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.076935+0000) 2020-02-05T23:40:46.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:46.075+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.077349+0000) 2020-02-05T23:40:46.079 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:46.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.077903+0000) 2020-02-05T23:40:46.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:46.076+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.078268+0000) 2020-02-05T23:40:46.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:46.077+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.078533+0000) 2020-02-05T23:40:46.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:46.077+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.078771+0000) 2020-02-05T23:40:46.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:46.078+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.079098+0000) 2020-02-05T23:40:46.081 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:46.078+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.079788+0000) 2020-02-05T23:40:46.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:46.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.081741+0000) 2020-02-05T23:40:46.091 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:46.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:46.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:46.933+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:46.934778+0000) 2020-02-05T23:40:47.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:47.184+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:47.185042+0000) 2020-02-05T23:40:47.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:47.191+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:47.192672+0000) 2020-02-05T23:40:47.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:47.191+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:47.192779+0000) 2020-02-05T23:40:47.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:47.191+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:47.193163+0000) 2020-02-05T23:40:49.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:49.577+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:49.577412+0000) 2020-02-05T23:40:50.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:50.422+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:50.423780+0000) 2020-02-05T23:40:50.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:50.679+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:50.680193+0000) 2020-02-05T23:40:51.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:51.094+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.096031+0000) 2020-02-05T23:40:51.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:51.095+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.096703+0000) 2020-02-05T23:40:51.098 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:51.095+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.096990+0000) 2020-02-05T23:40:51.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:51.096+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.097445+0000) 2020-02-05T23:40:51.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:51.096+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.098243+0000) 2020-02-05T23:40:51.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:51.098+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.099496+0000) 2020-02-05T23:40:51.101 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:51.098+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.099720+0000) 2020-02-05T23:40:51.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:51.099+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.100485+0000) 2020-02-05T23:40:51.102 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:51.099+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.100325+0000) 2020-02-05T23:40:51.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:51.100+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.101526+0000) 2020-02-05T23:40:51.102 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:51.100+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.101153+0000) 2020-02-05T23:40:51.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:51.100+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.101892+0000) 2020-02-05T23:40:51.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:51.100+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.102147+0000) 2020-02-05T23:40:51.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:51.103+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:51.104624+0000) 2020-02-05T23:40:51.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:51.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:52.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:52.440+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:52.441047+0000) 2020-02-05T23:40:52.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:52.789+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:52.791005+0000) 2020-02-05T23:40:53.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:53.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:53.077582+0000) 2020-02-05T23:40:53.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:53.952+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:53.953564+0000) 2020-02-05T23:40:53.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:53.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:53.953570+0000) 2020-02-05T23:40:55.025 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:40:55.031 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:40:56.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:56.052+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.053387+0000) 2020-02-05T23:40:56.118 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:40:56.116+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.117539+0000) 2020-02-05T23:40:56.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:56.117+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.118704+0000) 2020-02-05T23:40:56.120 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:56.118+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.119682+0000) 2020-02-05T23:40:56.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:56.119+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.120517+0000) 2020-02-05T23:40:56.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:56.119+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.120857+0000) 2020-02-05T23:40:56.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:56.119+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.121115+0000) 2020-02-05T23:40:56.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:40:56.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.121300+0000) 2020-02-05T23:40:56.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:56.120+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.121579+0000) 2020-02-05T23:40:56.123 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:40:56.120+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.122081+0000) 2020-02-05T23:40:56.123 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:56.120+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.121765+0000) 2020-02-05T23:40:56.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:40:56.121+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.122754+0000) 2020-02-05T23:40:56.124 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:40:56.121+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.122822+0000) 2020-02-05T23:40:56.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:40:56.122+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.123224+0000) 2020-02-05T23:40:56.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:56.124+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.125255+0000) 2020-02-05T23:40:56.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:40:56.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151743 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:40:56.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:40:56.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:56.935019+0000) 2020-02-05T23:40:57.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:40:57.185+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:57.185325+0000) 2020-02-05T23:40:57.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:40:57.192+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:57.192931+0000) 2020-02-05T23:40:57.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:40:57.192+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:57.193061+0000) 2020-02-05T23:40:57.194 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:40:57.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:57.193512+0000) 2020-02-05T23:40:59.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:40:59.577+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:40:59.577642+0000) 2020-02-05T23:41:00.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:00.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:00.424035+0000) 2020-02-05T23:41:00.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:00.688+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:00.688254+0000) 2020-02-05T23:41:01.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:01.139+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.139805+0000) 2020-02-05T23:41:01.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:01.140+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.141714+0000) 2020-02-05T23:41:01.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:01.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.141972+0000) 2020-02-05T23:41:01.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:01.142+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.142463+0000) 2020-02-05T23:41:01.144 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:01.142+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.142898+0000) 2020-02-05T23:41:01.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:01.143+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.143966+0000) 2020-02-05T23:41:01.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:01.144+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.144433+0000) 2020-02-05T23:41:01.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:01.143+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.144838+0000) 2020-02-05T23:41:01.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:01.144+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.145952+0000) 2020-02-05T23:41:01.147 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:01.144+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.146376+0000) 2020-02-05T23:41:01.148 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:01.146+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.146580+0000) 2020-02-05T23:41:01.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:01.145+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.146805+0000) 2020-02-05T23:41:01.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:01.145+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.147170+0000) 2020-02-05T23:41:01.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:01.150+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:01.150946+0000) 2020-02-05T23:41:01.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:01.156+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151787 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:02.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:02.440+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:02.441312+0000) 2020-02-05T23:41:02.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:02.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:02.791318+0000) 2020-02-05T23:41:03.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:03.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:03.077810+0000) 2020-02-05T23:41:03.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:03.953+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:03.953839+0000) 2020-02-05T23:41:03.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:03.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:03.953840+0000) 2020-02-05T23:41:06.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:06.053+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.053711+0000) 2020-02-05T23:41:06.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:06.163+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.164237+0000) 2020-02-05T23:41:06.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:06.162+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.164356+0000) 2020-02-05T23:41:06.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:06.164+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.165176+0000) 2020-02-05T23:41:06.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:06.164+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.165629+0000) 2020-02-05T23:41:06.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:06.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.165708+0000) 2020-02-05T23:41:06.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:06.164+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.165374+0000) 2020-02-05T23:41:06.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:06.165+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.166244+0000) 2020-02-05T23:41:06.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:06.165+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.166428+0000) 2020-02-05T23:41:06.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:06.165+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.167031+0000) 2020-02-05T23:41:06.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:06.166+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.167447+0000) 2020-02-05T23:41:06.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:06.166+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.167298+0000) 2020-02-05T23:41:06.169 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:06.166+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.168107+0000) 2020-02-05T23:41:06.169 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:06.166+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.168107+0000) 2020-02-05T23:41:06.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:06.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.172479+0000) 2020-02-05T23:41:06.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:06.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151827 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:06.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:06.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:06.935273+0000) 2020-02-05T23:41:07.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:07.184+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:07.185561+0000) 2020-02-05T23:41:07.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:07.192+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:07.193173+0000) 2020-02-05T23:41:07.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:07.192+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:07.193310+0000) 2020-02-05T23:41:07.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:07.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:07.193740+0000) 2020-02-05T23:41:09.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:09.576+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:09.577871+0000) 2020-02-05T23:41:10.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:10.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:10.424286+0000) 2020-02-05T23:41:10.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:10.695+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:10.696646+0000) 2020-02-05T23:41:11.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:11.183+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.185221+0000) 2020-02-05T23:41:11.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:11.185+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.186408+0000) 2020-02-05T23:41:11.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:11.185+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.187126+0000) 2020-02-05T23:41:11.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:11.187+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.188756+0000) 2020-02-05T23:41:11.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:11.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.189215+0000) 2020-02-05T23:41:11.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:11.188+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.189302+0000) 2020-02-05T23:41:11.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:11.188+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.190062+0000) 2020-02-05T23:41:11.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:11.188+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.189730+0000) 2020-02-05T23:41:11.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:11.189+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.190571+0000) 2020-02-05T23:41:11.192 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:11.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.191137+0000) 2020-02-05T23:41:11.193 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:11.190+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.191510+0000) 2020-02-05T23:41:11.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:11.190+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.191722+0000) 2020-02-05T23:41:11.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:11.190+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.192024+0000) 2020-02-05T23:41:11.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:11.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:11.198172+0000) 2020-02-05T23:41:11.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:11.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:12.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:12.440+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:12.441552+0000) 2020-02-05T23:41:12.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:12.790+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:12.791543+0000) 2020-02-05T23:41:13.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:13.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:13.078060+0000) 2020-02-05T23:41:13.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:13.953+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:13.954144+0000) 2020-02-05T23:41:13.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:13.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:13.954144+0000) 2020-02-05T23:41:16.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:16.053+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.053950+0000) 2020-02-05T23:41:16.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:16.206+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.208153+0000) 2020-02-05T23:41:16.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:16.207+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.209161+0000) 2020-02-05T23:41:16.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:16.208+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.209834+0000) 2020-02-05T23:41:16.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:16.209+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.210437+0000) 2020-02-05T23:41:16.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:16.210+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.211224+0000) 2020-02-05T23:41:16.212 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:16.210+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.211589+0000) 2020-02-05T23:41:16.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:16.210+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.211894+0000) 2020-02-05T23:41:16.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:16.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.213683+0000) 2020-02-05T23:41:16.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:16.212+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.214184+0000) 2020-02-05T23:41:16.215 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:16.213+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.214197+0000) 2020-02-05T23:41:16.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:16.213+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.214653+0000) 2020-02-05T23:41:16.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:16.213+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.214816+0000) 2020-02-05T23:41:16.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:16.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.218716+0000) 2020-02-05T23:41:16.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:16.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.219007+0000) 2020-02-05T23:41:16.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:16.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:16.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:16.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:16.935472+0000) 2020-02-05T23:41:17.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:17.184+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:17.185788+0000) 2020-02-05T23:41:17.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:17.192+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:17.193464+0000) 2020-02-05T23:41:17.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:17.192+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:17.193614+0000) 2020-02-05T23:41:17.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:17.192+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:17.193963+0000) 2020-02-05T23:41:19.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:19.577+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:19.578130+0000) 2020-02-05T23:41:20.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:20.423+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:20.424546+0000) 2020-02-05T23:41:20.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:20.704+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:20.705049+0000) 2020-02-05T23:41:21.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:21.231+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.232742+0000) 2020-02-05T23:41:21.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:21.232+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.233365+0000) 2020-02-05T23:41:21.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:21.232+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.233705+0000) 2020-02-05T23:41:21.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:21.232+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.233873+0000) 2020-02-05T23:41:21.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:21.232+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.234325+0000) 2020-02-05T23:41:21.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:21.233+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.234464+0000) 2020-02-05T23:41:21.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:21.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.235014+0000) 2020-02-05T23:41:21.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:21.234+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.235490+0000) 2020-02-05T23:41:21.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:21.234+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.235216+0000) 2020-02-05T23:41:21.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:21.234+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.235685+0000) 2020-02-05T23:41:21.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:21.234+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.236003+0000) 2020-02-05T23:41:21.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:21.235+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.236831+0000) 2020-02-05T23:41:21.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:21.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.236968+0000) 2020-02-05T23:41:21.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:21.240+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:21.241999+0000) 2020-02-05T23:41:21.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:21.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151955 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:22.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:22.441+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:22.441820+0000) 2020-02-05T23:41:22.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:22.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:22.791775+0000) 2020-02-05T23:41:23.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:23.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:23.078318+0000) 2020-02-05T23:41:23.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:23.953+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:23.954434+0000) 2020-02-05T23:41:23.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:23.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:23.954434+0000) 2020-02-05T23:41:25.156 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:41:25.162 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:41:26.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:26.053+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.054203+0000) 2020-02-05T23:41:26.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:26.256+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.257044+0000) 2020-02-05T23:41:26.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:26.256+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.257114+0000) 2020-02-05T23:41:26.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:26.254+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.255520+0000) 2020-02-05T23:41:26.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:26.257+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.258361+0000) 2020-02-05T23:41:26.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:26.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.258760+0000) 2020-02-05T23:41:26.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:26.258+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.259056+0000) 2020-02-05T23:41:26.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:26.258+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.259131+0000) 2020-02-05T23:41:26.261 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:26.258+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.260044+0000) 2020-02-05T23:41:26.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:26.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.260537+0000) 2020-02-05T23:41:26.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:26.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.260879+0000) 2020-02-05T23:41:26.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:26.260+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.261485+0000) 2020-02-05T23:41:26.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:26.260+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.261647+0000) 2020-02-05T23:41:26.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:26.260+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.261937+0000) 2020-02-05T23:41:26.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:26.262+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.264068+0000) 2020-02-05T23:41:26.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:26.269+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 151994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:26.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:26.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:26.935681+0000) 2020-02-05T23:41:27.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:27.184+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:27.185952+0000) 2020-02-05T23:41:27.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:27.193+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:27.193727+0000) 2020-02-05T23:41:27.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:27.193+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:27.193834+0000) 2020-02-05T23:41:27.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:27.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:27.194132+0000) 2020-02-05T23:41:29.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:29.577+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:29.578392+0000) 2020-02-05T23:41:30.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:30.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:30.424734+0000) 2020-02-05T23:41:30.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:30.712+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:30.713133+0000) 2020-02-05T23:41:31.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:31.275+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.276133+0000) 2020-02-05T23:41:31.277 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:31.275+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.276355+0000) 2020-02-05T23:41:31.278 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:31.275+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.276752+0000) 2020-02-05T23:41:31.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:31.277+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.277647+0000) 2020-02-05T23:41:31.279 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:31.276+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.277906+0000) 2020-02-05T23:41:31.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:31.277+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.278117+0000) 2020-02-05T23:41:31.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:31.278+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.278549+0000) 2020-02-05T23:41:31.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:31.278+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.279137+0000) 2020-02-05T23:41:31.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:31.278+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.279258+0000) 2020-02-05T23:41:31.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:31.279+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.280204+0000) 2020-02-05T23:41:31.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:31.280+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.280664+0000) 2020-02-05T23:41:31.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:31.281+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.281493+0000) 2020-02-05T23:41:31.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:31.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.284130+0000) 2020-02-05T23:41:31.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:31.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:31.285265+0000) 2020-02-05T23:41:31.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:31.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:32.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:32.441+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:32.442086+0000) 2020-02-05T23:41:32.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:32.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:32.791997+0000) 2020-02-05T23:41:33.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:33.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:33.078585+0000) 2020-02-05T23:41:33.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:33.954+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:33.954704+0000) 2020-02-05T23:41:33.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:33.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:33.954704+0000) 2020-02-05T23:41:36.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:36.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.054469+0000) 2020-02-05T23:41:36.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:36.298+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.299265+0000) 2020-02-05T23:41:36.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:36.299+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.300005+0000) 2020-02-05T23:41:36.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:36.299+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.300643+0000) 2020-02-05T23:41:36.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:36.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.302502+0000) 2020-02-05T23:41:36.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:36.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.302972+0000) 2020-02-05T23:41:36.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:36.302+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.303344+0000) 2020-02-05T23:41:36.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:36.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.303830+0000) 2020-02-05T23:41:36.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:36.304+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.304511+0000) 2020-02-05T23:41:36.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:36.304+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.304823+0000) 2020-02-05T23:41:36.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:36.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.305007+0000) 2020-02-05T23:41:36.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:36.304+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.305364+0000) 2020-02-05T23:41:36.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:36.305+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.305461+0000) 2020-02-05T23:41:36.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:36.306+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.307724+0000) 2020-02-05T23:41:36.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:36.306+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.307975+0000) 2020-02-05T23:41:36.319 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:36.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:36.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:36.934+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:36.935910+0000) 2020-02-05T23:41:37.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:37.185+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:37.186281+0000) 2020-02-05T23:41:37.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:37.193+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:37.193940+0000) 2020-02-05T23:41:37.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:37.193+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:37.194078+0000) 2020-02-05T23:41:37.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:37.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:37.194303+0000) 2020-02-05T23:41:39.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:39.578+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:39.578622+0000) 2020-02-05T23:41:40.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:40.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:40.424985+0000) 2020-02-05T23:41:40.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:40.721+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:40.721508+0000) 2020-02-05T23:41:41.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:41.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.323910+0000) 2020-02-05T23:41:41.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:41.323+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.324447+0000) 2020-02-05T23:41:41.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:41.324+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.325292+0000) 2020-02-05T23:41:41.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:41.324+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.325797+0000) 2020-02-05T23:41:41.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:41.325+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.326363+0000) 2020-02-05T23:41:41.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:41.326+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.326802+0000) 2020-02-05T23:41:41.328 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:41.325+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.327023+0000) 2020-02-05T23:41:41.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:41.328+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.328850+0000) 2020-02-05T23:41:41.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:41.329+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.329520+0000) 2020-02-05T23:41:41.331 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:41.329+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.330117+0000) 2020-02-05T23:41:41.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:41.329+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.330206+0000) 2020-02-05T23:41:41.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:41.329+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.330030+0000) 2020-02-05T23:41:41.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:41.330+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.331092+0000) 2020-02-05T23:41:41.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:41.332+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:41.333936+0000) 2020-02-05T23:41:41.343 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:41.342+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:42.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:42.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:42.442351+0000) 2020-02-05T23:41:42.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:42.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:42.792261+0000) 2020-02-05T23:41:43.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:43.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:43.078844+0000) 2020-02-05T23:41:43.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:43.953+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:43.954981+0000) 2020-02-05T23:41:43.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:43.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:43.954980+0000) 2020-02-05T23:41:46.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:46.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.054722+0000) 2020-02-05T23:41:46.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:46.346+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.347277+0000) 2020-02-05T23:41:46.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:46.349+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.349689+0000) 2020-02-05T23:41:46.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:46.348+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.349952+0000) 2020-02-05T23:41:46.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:46.351+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.351685+0000) 2020-02-05T23:41:46.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:46.351+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.352021+0000) 2020-02-05T23:41:46.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:46.351+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.352455+0000) 2020-02-05T23:41:46.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:46.351+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.352910+0000) 2020-02-05T23:41:46.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:46.351+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.353004+0000) 2020-02-05T23:41:46.355 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:46.352+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.353320+0000) 2020-02-05T23:41:46.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:46.352+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.353493+0000) 2020-02-05T23:41:46.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:46.352+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.353963+0000) 2020-02-05T23:41:46.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:46.353+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.354184+0000) 2020-02-05T23:41:46.356 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:46.354+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.354622+0000) 2020-02-05T23:41:46.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:46.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.361789+0000) 2020-02-05T23:41:46.366 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:46.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:46.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:46.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:46.936117+0000) 2020-02-05T23:41:47.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:47.186+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:47.186483+0000) 2020-02-05T23:41:47.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:47.193+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:47.194174+0000) 2020-02-05T23:41:47.195 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:47.193+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:47.194310+0000) 2020-02-05T23:41:47.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:47.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:47.194516+0000) 2020-02-05T23:41:49.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:49.578+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:49.578854+0000) 2020-02-05T23:41:50.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:50.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:50.425232+0000) 2020-02-05T23:41:50.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:50.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:50.729659+0000) 2020-02-05T23:41:51.372 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:51.371+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.371606+0000) 2020-02-05T23:41:51.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:51.371+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.372291+0000) 2020-02-05T23:41:51.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:51.371+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.372421+0000) 2020-02-05T23:41:51.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:51.373+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.373650+0000) 2020-02-05T23:41:51.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:51.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.374171+0000) 2020-02-05T23:41:51.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:51.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.374751+0000) 2020-02-05T23:41:51.376 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:51.374+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.375528+0000) 2020-02-05T23:41:51.377 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:51.375+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.375714+0000) 2020-02-05T23:41:51.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:51.375+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.376054+0000) 2020-02-05T23:41:51.377 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:51.375+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.376404+0000) 2020-02-05T23:41:51.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:51.375+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.376591+0000) 2020-02-05T23:41:51.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:51.376+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.376776+0000) 2020-02-05T23:41:51.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:51.376+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.377336+0000) 2020-02-05T23:41:51.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:51.379+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:51.380979+0000) 2020-02-05T23:41:51.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:51.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152207 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:52.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:52.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:52.442648+0000) 2020-02-05T23:41:52.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:52.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:52.792488+0000) 2020-02-05T23:41:53.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:53.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:53.079127+0000) 2020-02-05T23:41:53.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:53.955+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:53.955248+0000) 2020-02-05T23:41:53.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:53.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:53.955248+0000) 2020-02-05T23:41:55.294 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:41:55.301 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:41:56.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:56.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.054983+0000) 2020-02-05T23:41:56.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:56.394+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.394174+0000) 2020-02-05T23:41:56.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:41:56.395+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.395898+0000) 2020-02-05T23:41:56.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:41:56.395+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.396372+0000) 2020-02-05T23:41:56.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:56.396+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.396903+0000) 2020-02-05T23:41:56.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:41:56.397+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.397803+0000) 2020-02-05T23:41:56.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:41:56.399+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.399756+0000) 2020-02-05T23:41:56.401 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:56.399+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.399973+0000) 2020-02-05T23:41:56.401 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:41:56.400+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.400146+0000) 2020-02-05T23:41:56.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:56.399+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.400203+0000) 2020-02-05T23:41:56.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:41:56.401+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.401237+0000) 2020-02-05T23:41:56.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:56.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.401434+0000) 2020-02-05T23:41:56.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:56.400+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.401329+0000) 2020-02-05T23:41:56.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:56.401+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.401562+0000) 2020-02-05T23:41:56.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:56.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.404951+0000) 2020-02-05T23:41:56.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:41:56.411+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152245 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:41:56.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:41:56.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:56.936325+0000) 2020-02-05T23:41:57.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:41:57.186+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:57.186696+0000) 2020-02-05T23:41:57.195 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:41:57.194+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:57.194479+0000) 2020-02-05T23:41:57.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:41:57.194+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:57.194543+0000) 2020-02-05T23:41:57.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:41:57.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:57.194869+0000) 2020-02-05T23:41:59.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:41:59.578+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:41:59.579123+0000) 2020-02-05T23:42:00.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:00.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:00.425498+0000) 2020-02-05T23:42:00.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:00.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:00.738284+0000) 2020-02-05T23:42:01.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:01.415+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.416427+0000) 2020-02-05T23:42:01.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:01.418+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.419057+0000) 2020-02-05T23:42:01.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:01.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.419179+0000) 2020-02-05T23:42:01.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:01.418+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.419921+0000) 2020-02-05T23:42:01.421 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:01.419+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.420486+0000) 2020-02-05T23:42:01.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:01.420+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.420845+0000) 2020-02-05T23:42:01.422 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:01.420+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.421251+0000) 2020-02-05T23:42:01.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:01.420+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.421635+0000) 2020-02-05T23:42:01.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:01.421+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.422166+0000) 2020-02-05T23:42:01.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:01.423+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.423539+0000) 2020-02-05T23:42:01.425 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:01.422+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.423889+0000) 2020-02-05T23:42:01.425 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:01.423+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.424138+0000) 2020-02-05T23:42:01.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:01.423+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.424144+0000) 2020-02-05T23:42:01.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:01.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:01.427597+0000) 2020-02-05T23:42:01.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:01.434+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:02.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:02.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:02.442908+0000) 2020-02-05T23:42:02.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:02.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:02.792764+0000) 2020-02-05T23:42:03.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:03.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:03.079362+0000) 2020-02-05T23:42:03.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:03.954+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:03.955512+0000) 2020-02-05T23:42:03.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:03.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:03.955512+0000) 2020-02-05T23:42:06.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:06.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.055277+0000) 2020-02-05T23:42:06.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:06.439+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.440548+0000) 2020-02-05T23:42:06.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:06.441+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.442476+0000) 2020-02-05T23:42:06.444 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:06.441+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.442966+0000) 2020-02-05T23:42:06.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:06.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.443445+0000) 2020-02-05T23:42:06.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:06.442+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.443767+0000) 2020-02-05T23:42:06.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:06.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.443887+0000) 2020-02-05T23:42:06.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:06.443+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.444220+0000) 2020-02-05T23:42:06.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:06.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.444474+0000) 2020-02-05T23:42:06.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:06.443+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.445085+0000) 2020-02-05T23:42:06.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:06.443+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.444874+0000) 2020-02-05T23:42:06.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:06.444+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.445484+0000) 2020-02-05T23:42:06.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:06.445+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.446495+0000) 2020-02-05T23:42:06.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:06.445+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.446872+0000) 2020-02-05T23:42:06.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:06.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.450540+0000) 2020-02-05T23:42:06.461 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:06.458+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152330 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:06.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:06.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:06.936535+0000) 2020-02-05T23:42:07.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:07.185+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:07.186910+0000) 2020-02-05T23:42:07.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:07.193+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:07.194761+0000) 2020-02-05T23:42:07.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:07.193+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:07.194721+0000) 2020-02-05T23:42:07.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:07.193+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:07.195089+0000) 2020-02-05T23:42:09.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:09.578+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:09.579389+0000) 2020-02-05T23:42:10.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:10.424+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:10.425747+0000) 2020-02-05T23:42:10.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:10.749+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:10.750300+0000) 2020-02-05T23:42:11.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:11.464+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.465925+0000) 2020-02-05T23:42:11.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:11.465+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.466602+0000) 2020-02-05T23:42:11.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:11.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.466883+0000) 2020-02-05T23:42:11.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:11.465+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.467384+0000) 2020-02-05T23:42:11.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:11.467+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.468170+0000) 2020-02-05T23:42:11.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:11.467+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.468766+0000) 2020-02-05T23:42:11.470 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:11.468+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.469253+0000) 2020-02-05T23:42:11.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:11.468+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.469658+0000) 2020-02-05T23:42:11.471 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:11.468+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.470150+0000) 2020-02-05T23:42:11.471 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:11.469+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.470663+0000) 2020-02-05T23:42:11.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:11.469+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.471022+0000) 2020-02-05T23:42:11.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:11.470+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.471523+0000) 2020-02-05T23:42:11.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:11.470+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.471447+0000) 2020-02-05T23:42:11.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:11.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:11.476645+0000) 2020-02-05T23:42:11.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:11.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:12.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:12.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:12.443175+0000) 2020-02-05T23:42:12.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:12.791+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:12.792984+0000) 2020-02-05T23:42:13.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:13.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:13.079611+0000) 2020-02-05T23:42:13.957 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:13.954+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:13.955796+0000) 2020-02-05T23:42:13.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:13.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:13.955798+0000) 2020-02-05T23:42:16.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:16.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.055573+0000) 2020-02-05T23:42:16.487 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:16.484+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.486338+0000) 2020-02-05T23:42:16.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:16.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.487499+0000) 2020-02-05T23:42:16.489 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:16.486+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.487726+0000) 2020-02-05T23:42:16.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:16.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.488225+0000) 2020-02-05T23:42:16.490 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:16.487+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.488507+0000) 2020-02-05T23:42:16.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:16.488+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.489293+0000) 2020-02-05T23:42:16.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:16.488+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.489717+0000) 2020-02-05T23:42:16.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:16.488+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.490273+0000) 2020-02-05T23:42:16.492 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:16.489+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.490603+0000) 2020-02-05T23:42:16.492 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:16.489+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.490907+0000) 2020-02-05T23:42:16.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:16.489+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.491041+0000) 2020-02-05T23:42:16.493 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:16.490+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.492120+0000) 2020-02-05T23:42:16.493 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:16.491+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.492234+0000) 2020-02-05T23:42:16.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:16.497+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.498993+0000) 2020-02-05T23:42:16.505 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:16.503+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:16.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:16.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:16.936759+0000) 2020-02-05T23:42:17.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:17.186+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:17.187118+0000) 2020-02-05T23:42:17.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:17.194+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:17.194984+0000) 2020-02-05T23:42:17.196 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:17.194+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:17.194949+0000) 2020-02-05T23:42:17.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:17.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:17.195297+0000) 2020-02-05T23:42:19.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:19.578+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:19.579638+0000) 2020-02-05T23:42:20.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:20.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:20.425983+0000) 2020-02-05T23:42:20.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:20.757+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:20.758445+0000) 2020-02-05T23:42:21.512 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:21.509+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.510993+0000) 2020-02-05T23:42:21.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:21.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.511913+0000) 2020-02-05T23:42:21.514 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:21.511+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.512402+0000) 2020-02-05T23:42:21.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:21.511+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.512878+0000) 2020-02-05T23:42:21.514 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:21.512+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.513794+0000) 2020-02-05T23:42:21.515 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:21.512+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.514137+0000) 2020-02-05T23:42:21.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:21.512+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.514394+0000) 2020-02-05T23:42:21.516 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:21.513+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.515010+0000) 2020-02-05T23:42:21.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:21.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.515652+0000) 2020-02-05T23:42:21.518 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:21.514+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.516368+0000) 2020-02-05T23:42:21.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:21.515+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.516453+0000) 2020-02-05T23:42:21.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:21.515+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.516217+0000) 2020-02-05T23:42:21.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:21.515+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.516254+0000) 2020-02-05T23:42:21.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:21.518+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:21.519216+0000) 2020-02-05T23:42:21.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:21.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:22.444 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:22.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:22.443410+0000) 2020-02-05T23:42:22.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:22.792+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:22.793312+0000) 2020-02-05T23:42:23.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:23.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:23.079839+0000) 2020-02-05T23:42:23.957 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:23.955+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:23.956056+0000) 2020-02-05T23:42:23.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:23.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:23.956057+0000) 2020-02-05T23:42:25.430 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:42:25.436 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:42:26.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:26.055+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.055823+0000) 2020-02-05T23:42:26.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:26.534+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.535187+0000) 2020-02-05T23:42:26.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:26.534+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.535942+0000) 2020-02-05T23:42:26.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:26.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.536941+0000) 2020-02-05T23:42:26.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:26.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.537632+0000) 2020-02-05T23:42:26.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:26.537+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.538263+0000) 2020-02-05T23:42:26.539 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:26.536+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.537957+0000) 2020-02-05T23:42:26.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:26.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.539538+0000) 2020-02-05T23:42:26.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:26.538+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.539745+0000) 2020-02-05T23:42:26.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:26.539+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.540173+0000) 2020-02-05T23:42:26.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:26.539+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.540419+0000) 2020-02-05T23:42:26.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:26.539+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.540584+0000) 2020-02-05T23:42:26.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:26.539+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.541408+0000) 2020-02-05T23:42:26.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:26.539+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.540918+0000) 2020-02-05T23:42:26.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:26.543+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.544298+0000) 2020-02-05T23:42:26.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:26.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:26.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:26.935+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:26.936918+0000) 2020-02-05T23:42:27.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:27.186+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:27.187325+0000) 2020-02-05T23:42:27.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:27.194+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:27.195211+0000) 2020-02-05T23:42:27.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:27.194+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:27.195178+0000) 2020-02-05T23:42:27.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:27.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:27.195555+0000) 2020-02-05T23:42:29.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:29.579+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:29.579873+0000) 2020-02-05T23:42:30.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:30.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:30.426230+0000) 2020-02-05T23:42:30.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:30.766+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:30.766814+0000) 2020-02-05T23:42:31.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:31.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.556286+0000) 2020-02-05T23:42:31.559 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:31.556+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.558119+0000) 2020-02-05T23:42:31.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:31.557+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.558610+0000) 2020-02-05T23:42:31.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:31.558+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.559067+0000) 2020-02-05T23:42:31.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:31.558+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.559947+0000) 2020-02-05T23:42:31.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:31.559+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.560841+0000) 2020-02-05T23:42:31.562 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:31.559+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.560335+0000) 2020-02-05T23:42:31.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:31.559+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.561085+0000) 2020-02-05T23:42:31.563 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:31.560+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.561775+0000) 2020-02-05T23:42:31.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:31.560+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.561368+0000) 2020-02-05T23:42:31.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:31.561+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.562064+0000) 2020-02-05T23:42:31.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:31.561+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.562853+0000) 2020-02-05T23:42:31.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:31.562+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.563608+0000) 2020-02-05T23:42:31.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:31.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:31.570615+0000) 2020-02-05T23:42:31.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:31.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:32.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:32.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:32.443663+0000) 2020-02-05T23:42:32.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:32.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:32.793550+0000) 2020-02-05T23:42:33.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:33.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:33.080113+0000) 2020-02-05T23:42:33.957 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:33.955+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:33.956385+0000) 2020-02-05T23:42:33.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:33.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:33.956385+0000) 2020-02-05T23:42:36.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:36.054+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.056091+0000) 2020-02-05T23:42:36.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:36.578+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.579659+0000) 2020-02-05T23:42:36.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:36.579+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.580339+0000) 2020-02-05T23:42:36.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:36.580+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.581329+0000) 2020-02-05T23:42:36.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:36.579+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.581272+0000) 2020-02-05T23:42:36.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:36.581+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.582103+0000) 2020-02-05T23:42:36.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:36.581+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.582672+0000) 2020-02-05T23:42:36.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:36.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.582866+0000) 2020-02-05T23:42:36.585 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:36.582+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.583373+0000) 2020-02-05T23:42:36.586 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:36.583+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.584461+0000) 2020-02-05T23:42:36.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:36.583+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.585194+0000) 2020-02-05T23:42:36.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:36.584+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.586062+0000) 2020-02-05T23:42:36.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:36.584+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.586371+0000) 2020-02-05T23:42:36.588 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:36.585+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.586521+0000) 2020-02-05T23:42:36.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:36.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.590965+0000) 2020-02-05T23:42:36.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:36.597+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152579 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:36.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:36.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:36.937166+0000) 2020-02-05T23:42:37.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:37.187+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:37.187534+0000) 2020-02-05T23:42:37.196 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:37.194+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:37.195385+0000) 2020-02-05T23:42:37.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:37.194+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:37.195471+0000) 2020-02-05T23:42:37.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:37.194+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:37.195742+0000) 2020-02-05T23:42:39.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:39.579+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:39.580137+0000) 2020-02-05T23:42:40.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:40.425+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:40.426475+0000) 2020-02-05T23:42:40.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:40.777+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:40.778279+0000) 2020-02-05T23:42:41.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:41.603+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.604877+0000) 2020-02-05T23:42:41.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:41.604+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.605407+0000) 2020-02-05T23:42:41.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:41.604+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.606277+0000) 2020-02-05T23:42:41.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:41.605+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.606869+0000) 2020-02-05T23:42:41.608 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:41.605+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.606971+0000) 2020-02-05T23:42:41.608 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:41.605+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.607312+0000) 2020-02-05T23:42:41.609 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:41.606+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.607843+0000) 2020-02-05T23:42:41.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:41.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.607786+0000) 2020-02-05T23:42:41.609 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:41.606+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.608096+0000) 2020-02-05T23:42:41.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:41.607+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.608499+0000) 2020-02-05T23:42:41.610 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:41.607+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.609158+0000) 2020-02-05T23:42:41.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:41.608+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.609510+0000) 2020-02-05T23:42:41.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:41.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.613288+0000) 2020-02-05T23:42:41.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:41.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:41.613542+0000) 2020-02-05T23:42:41.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:41.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152624 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:42.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:42.442+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:42.443865+0000) 2020-02-05T23:42:42.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:42.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:42.793790+0000) 2020-02-05T23:42:43.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:43.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:43.080381+0000) 2020-02-05T23:42:43.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:43.956+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:43.956655+0000) 2020-02-05T23:42:43.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:43.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:43.956655+0000) 2020-02-05T23:42:46.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:46.055+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.056343+0000) 2020-02-05T23:42:46.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:46.625+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.626306+0000) 2020-02-05T23:42:46.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:46.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.627357+0000) 2020-02-05T23:42:46.628 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:46.626+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.627711+0000) 2020-02-05T23:42:46.629 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:46.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.628330+0000) 2020-02-05T23:42:46.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:46.627+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.628418+0000) 2020-02-05T23:42:46.630 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:46.627+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.628741+0000) 2020-02-05T23:42:46.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:46.628+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.629122+0000) 2020-02-05T23:42:46.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:46.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.634809+0000) 2020-02-05T23:42:46.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:46.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.636000+0000) 2020-02-05T23:42:46.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:46.629+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.630568+0000) 2020-02-05T23:42:46.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:46.629+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.631242+0000) 2020-02-05T23:42:46.643 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:46.628+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.630310+0000) 2020-02-05T23:42:46.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:46.630+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.631788+0000) 2020-02-05T23:42:46.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:46.630+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.631647+0000) 2020-02-05T23:42:46.654 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:46.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:46.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:46.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:46.937375+0000) 2020-02-05T23:42:47.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:47.187+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:47.187784+0000) 2020-02-05T23:42:47.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:47.195+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:47.195651+0000) 2020-02-05T23:42:47.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:47.195+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:47.195738+0000) 2020-02-05T23:42:47.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:47.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:47.195944+0000) 2020-02-05T23:42:49.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:49.580+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:49.580371+0000) 2020-02-05T23:42:50.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:50.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:50.426715+0000) 2020-02-05T23:42:50.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:50.786+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:50.786402+0000) 2020-02-05T23:42:51.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:51.650+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.650345+0000) 2020-02-05T23:42:51.652 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:51.649+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.651373+0000) 2020-02-05T23:42:51.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:51.652+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.652517+0000) 2020-02-05T23:42:51.654 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:51.651+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.652991+0000) 2020-02-05T23:42:51.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:51.652+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.653541+0000) 2020-02-05T23:42:51.655 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:51.653+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.653969+0000) 2020-02-05T23:42:51.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:51.653+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.653860+0000) 2020-02-05T23:42:51.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:51.653+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.654459+0000) 2020-02-05T23:42:51.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:51.653+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.654918+0000) 2020-02-05T23:42:51.656 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:51.654+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.654313+0000) 2020-02-05T23:42:51.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:51.654+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.654535+0000) 2020-02-05T23:42:51.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:51.654+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.654684+0000) 2020-02-05T23:42:51.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:51.654+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.655422+0000) 2020-02-05T23:42:51.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:51.661+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:51.661136+0000) 2020-02-05T23:42:51.668 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:51.666+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:52.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:52.443+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:52.444169+0000) 2020-02-05T23:42:52.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:52.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:52.794000+0000) 2020-02-05T23:42:53.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:53.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:53.080625+0000) 2020-02-05T23:42:53.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:53.955+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:53.956927+0000) 2020-02-05T23:42:53.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:53.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:53.956923+0000) 2020-02-05T23:42:55.567 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:42:55.573 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:42:56.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:56.056+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.056613+0000) 2020-02-05T23:42:56.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:42:56.671+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.672469+0000) 2020-02-05T23:42:56.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:56.672+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.673963+0000) 2020-02-05T23:42:56.676 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:42:56.673+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.674819+0000) 2020-02-05T23:42:56.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:56.674+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.674714+0000) 2020-02-05T23:42:56.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:42:56.674+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.675331+0000) 2020-02-05T23:42:56.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:56.675+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.676106+0000) 2020-02-05T23:42:56.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:42:56.675+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.676555+0000) 2020-02-05T23:42:56.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:56.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.676614+0000) 2020-02-05T23:42:56.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:56.676+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.677108+0000) 2020-02-05T23:42:56.679 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:56.677+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.677418+0000) 2020-02-05T23:42:56.679 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:42:56.677+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.677635+0000) 2020-02-05T23:42:56.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:42:56.677+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.678215+0000) 2020-02-05T23:42:56.680 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:56.676+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.677401+0000) 2020-02-05T23:42:56.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:56.682+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.683562+0000) 2020-02-05T23:42:56.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:42:56.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152747 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:42:56.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:42:56.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:56.937591+0000) 2020-02-05T23:42:57.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:42:57.187+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:57.188025+0000) 2020-02-05T23:42:57.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:42:57.195+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:57.195860+0000) 2020-02-05T23:42:57.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:42:57.195+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:57.195971+0000) 2020-02-05T23:42:57.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:42:57.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:57.196192+0000) 2020-02-05T23:42:59.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:42:59.579+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:42:59.580614+0000) 2020-02-05T23:43:00.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:00.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:00.426954+0000) 2020-02-05T23:43:00.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:00.793+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:00.794967+0000) 2020-02-05T23:43:01.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:01.696+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.696741+0000) 2020-02-05T23:43:01.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:01.697+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.697923+0000) 2020-02-05T23:43:01.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:01.697+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.698357+0000) 2020-02-05T23:43:01.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:01.698+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.699250+0000) 2020-02-05T23:43:01.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:01.698+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.699649+0000) 2020-02-05T23:43:01.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:01.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.700237+0000) 2020-02-05T23:43:01.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:01.699+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.700601+0000) 2020-02-05T23:43:01.702 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:01.701+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.701538+0000) 2020-02-05T23:43:01.703 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:01.700+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.701750+0000) 2020-02-05T23:43:01.703 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:01.701+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.701588+0000) 2020-02-05T23:43:01.703 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:01.701+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.702229+0000) 2020-02-05T23:43:01.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:01.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.702484+0000) 2020-02-05T23:43:01.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:01.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.702898+0000) 2020-02-05T23:43:01.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:01.708+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:01.709279+0000) 2020-02-05T23:43:01.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:01.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:02.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:02.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:02.444505+0000) 2020-02-05T23:43:02.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:02.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:02.794264+0000) 2020-02-05T23:43:03.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:03.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:03.080882+0000) 2020-02-05T23:43:03.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:03.956+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:03.957187+0000) 2020-02-05T23:43:03.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:03.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:03.957187+0000) 2020-02-05T23:43:06.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:06.056+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.056885+0000) 2020-02-05T23:43:06.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:06.718+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.719238+0000) 2020-02-05T23:43:06.721 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:06.718+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.719740+0000) 2020-02-05T23:43:06.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:06.719+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.720371+0000) 2020-02-05T23:43:06.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:06.721+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.721612+0000) 2020-02-05T23:43:06.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:06.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.721692+0000) 2020-02-05T23:43:06.724 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:06.722+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.722897+0000) 2020-02-05T23:43:06.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:06.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.723283+0000) 2020-02-05T23:43:06.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:06.722+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.723485+0000) 2020-02-05T23:43:06.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:06.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.724122+0000) 2020-02-05T23:43:06.725 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:06.724+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.724640+0000) 2020-02-05T23:43:06.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:06.724+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.725344+0000) 2020-02-05T23:43:06.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:06.724+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.725324+0000) 2020-02-05T23:43:06.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:06.725+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.725459+0000) 2020-02-05T23:43:06.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:06.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.730725+0000) 2020-02-05T23:43:06.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:06.737+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152833 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:06.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:06.936+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:06.937823+0000) 2020-02-05T23:43:07.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:07.187+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:07.188265+0000) 2020-02-05T23:43:07.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:07.195+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:07.196185+0000) 2020-02-05T23:43:07.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:07.195+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:07.196087+0000) 2020-02-05T23:43:07.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:07.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:07.196408+0000) 2020-02-05T23:43:09.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:09.580+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:09.580849+0000) 2020-02-05T23:43:10.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:10.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:10.427193+0000) 2020-02-05T23:43:10.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:10.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:10.803315+0000) 2020-02-05T23:43:11.744 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:11.742+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.743203+0000) 2020-02-05T23:43:11.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:11.742+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.743896+0000) 2020-02-05T23:43:11.746 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:11.744+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.744848+0000) 2020-02-05T23:43:11.746 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:11.744+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.745250+0000) 2020-02-05T23:43:11.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:11.745+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.745823+0000) 2020-02-05T23:43:11.747 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:11.744+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.745871+0000) 2020-02-05T23:43:11.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:11.746+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.747172+0000) 2020-02-05T23:43:11.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:11.747+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.747661+0000) 2020-02-05T23:43:11.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:11.746+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.747954+0000) 2020-02-05T23:43:11.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:11.747+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.748358+0000) 2020-02-05T23:43:11.750 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:11.748+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.748815+0000) 2020-02-05T23:43:11.750 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:11.748+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.748895+0000) 2020-02-05T23:43:11.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:11.749+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.749643+0000) 2020-02-05T23:43:11.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:11.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:11.753386+0000) 2020-02-05T23:43:11.763 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:11.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152876 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:12.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:12.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:12.444730+0000) 2020-02-05T23:43:12.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:12.793+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:12.794493+0000) 2020-02-05T23:43:13.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:13.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:13.081140+0000) 2020-02-05T23:43:13.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:13.956+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:13.957460+0000) 2020-02-05T23:43:13.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:13.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:13.957460+0000) 2020-02-05T23:43:16.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:16.056+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.057133+0000) 2020-02-05T23:43:16.769 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:16.767+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.767845+0000) 2020-02-05T23:43:16.770 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:16.768+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.768759+0000) 2020-02-05T23:43:16.771 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:16.769+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.769846+0000) 2020-02-05T23:43:16.771 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:16.768+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.769937+0000) 2020-02-05T23:43:16.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:16.769+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.770536+0000) 2020-02-05T23:43:16.772 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:16.770+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.771348+0000) 2020-02-05T23:43:16.773 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:16.771+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.772029+0000) 2020-02-05T23:43:16.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:16.771+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.772244+0000) 2020-02-05T23:43:16.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:16.771+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.772622+0000) 2020-02-05T23:43:16.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:16.772+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.773070+0000) 2020-02-05T23:43:16.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:16.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.773104+0000) 2020-02-05T23:43:16.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:16.772+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.773386+0000) 2020-02-05T23:43:16.775 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:16.773+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.773543+0000) 2020-02-05T23:43:16.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:16.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.775653+0000) 2020-02-05T23:43:16.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:16.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:16.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:16.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:16.938027+0000) 2020-02-05T23:43:17.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:17.187+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:17.188469+0000) 2020-02-05T23:43:17.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:17.196+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:17.196357+0000) 2020-02-05T23:43:17.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:17.196+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:17.196409+0000) 2020-02-05T23:43:17.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:17.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:17.196672+0000) 2020-02-05T23:43:19.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:19.580+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:19.581120+0000) 2020-02-05T23:43:20.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:20.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:20.427487+0000) 2020-02-05T23:43:20.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:20.812+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:20.813120+0000) 2020-02-05T23:43:21.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:21.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.790872+0000) 2020-02-05T23:43:21.793 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:21.790+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.791228+0000) 2020-02-05T23:43:21.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:21.790+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.791785+0000) 2020-02-05T23:43:21.793 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:21.791+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.791552+0000) 2020-02-05T23:43:21.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:21.791+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.792844+0000) 2020-02-05T23:43:21.794 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:21.793+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.793500+0000) 2020-02-05T23:43:21.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:21.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.794180+0000) 2020-02-05T23:43:21.795 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:21.794+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.794582+0000) 2020-02-05T23:43:21.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:21.794+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.795069+0000) 2020-02-05T23:43:21.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:21.795+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.795501+0000) 2020-02-05T23:43:21.797 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:21.795+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.795992+0000) 2020-02-05T23:43:21.797 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:21.794+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.796047+0000) 2020-02-05T23:43:21.797 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:21.795+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.796388+0000) 2020-02-05T23:43:21.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:21.801+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:21.802985+0000) 2020-02-05T23:43:21.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:21.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:22.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:22.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:22.444965+0000) 2020-02-05T23:43:22.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:22.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:22.794720+0000) 2020-02-05T23:43:23.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:23.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:23.081372+0000) 2020-02-05T23:43:23.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:23.957+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:23.957713+0000) 2020-02-05T23:43:23.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:23.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:23.957714+0000) 2020-02-05T23:43:25.691 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:43:25.697 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:43:26.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:26.057+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.057460+0000) 2020-02-05T23:43:26.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:26.816+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.816613+0000) 2020-02-05T23:43:26.818 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:26.816+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.817362+0000) 2020-02-05T23:43:26.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:26.816+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.818057+0000) 2020-02-05T23:43:26.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:26.817+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.818684+0000) 2020-02-05T23:43:26.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:26.818+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.819343+0000) 2020-02-05T23:43:26.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:26.818+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.819957+0000) 2020-02-05T23:43:26.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:26.819+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.820549+0000) 2020-02-05T23:43:26.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:26.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.821064+0000) 2020-02-05T23:43:26.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:26.820+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.821729+0000) 2020-02-05T23:43:26.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:26.821+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.822099+0000) 2020-02-05T23:43:26.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:26.821+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.822310+0000) 2020-02-05T23:43:26.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:26.822+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.822481+0000) 2020-02-05T23:43:26.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:26.821+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.822274+0000) 2020-02-05T23:43:26.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:26.825+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.826645+0000) 2020-02-05T23:43:26.835 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:26.833+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 152997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:26.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:26.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:26.938269+0000) 2020-02-05T23:43:27.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:27.188+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:27.188698+0000) 2020-02-05T23:43:27.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:27.196+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:27.196616+0000) 2020-02-05T23:43:27.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:27.196+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:27.196710+0000) 2020-02-05T23:43:27.478 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:27.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:27.196882+0000) 2020-02-05T23:43:29.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:29.580+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:29.581351+0000) 2020-02-05T23:43:30.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:30.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:30.427733+0000) 2020-02-05T23:43:30.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:30.821+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:30.822277+0000) 2020-02-05T23:43:31.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:31.840+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.840488+0000) 2020-02-05T23:43:31.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:31.840+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.841434+0000) 2020-02-05T23:43:31.843 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:31.841+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.841613+0000) 2020-02-05T23:43:31.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:31.841+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.842252+0000) 2020-02-05T23:43:31.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:31.841+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.842371+0000) 2020-02-05T23:43:31.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:31.841+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.842889+0000) 2020-02-05T23:43:31.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:31.842+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.843167+0000) 2020-02-05T23:43:31.845 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:31.843+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.843821+0000) 2020-02-05T23:43:31.845 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:31.843+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.844322+0000) 2020-02-05T23:43:31.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:31.843+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.844495+0000) 2020-02-05T23:43:31.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:31.844+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.845199+0000) 2020-02-05T23:43:31.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:31.845+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.846084+0000) 2020-02-05T23:43:31.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:31.848+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.849591+0000) 2020-02-05T23:43:31.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:31.848+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:31.849826+0000) 2020-02-05T23:43:31.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:31.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:32.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:32.445+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:32.445250+0000) 2020-02-05T23:43:32.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:32.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:32.794968+0000) 2020-02-05T23:43:33.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:33.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:33.081613+0000) 2020-02-05T23:43:33.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:33.956+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:33.958005+0000) 2020-02-05T23:43:33.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:33.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:33.958005+0000) 2020-02-05T23:43:36.059 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:36.056+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.057718+0000) 2020-02-05T23:43:36.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:36.860+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.861071+0000) 2020-02-05T23:43:36.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:36.861+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.862331+0000) 2020-02-05T23:43:36.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:36.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.863015+0000) 2020-02-05T23:43:36.865 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:36.862+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.863616+0000) 2020-02-05T23:43:36.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:36.862+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.863620+0000) 2020-02-05T23:43:36.865 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:36.863+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.864615+0000) 2020-02-05T23:43:36.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:36.863+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.865260+0000) 2020-02-05T23:43:36.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:36.864+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.866086+0000) 2020-02-05T23:43:36.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:36.865+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.866154+0000) 2020-02-05T23:43:36.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:36.865+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.866521+0000) 2020-02-05T23:43:36.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:36.865+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.867310+0000) 2020-02-05T23:43:36.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:36.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.867484+0000) 2020-02-05T23:43:36.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:36.866+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.867905+0000) 2020-02-05T23:43:36.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:36.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.870996+0000) 2020-02-05T23:43:36.880 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:36.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:36.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:36.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:36.938472+0000) 2020-02-05T23:43:37.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:37.188+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:37.188988+0000) 2020-02-05T23:43:37.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:37.195+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:37.196830+0000) 2020-02-05T23:43:37.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:37.195+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:37.196949+0000) 2020-02-05T23:43:37.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:37.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:37.197092+0000) 2020-02-05T23:43:39.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:39.581+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:39.581590+0000) 2020-02-05T23:43:40.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:40.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:40.427978+0000) 2020-02-05T23:43:40.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:40.830+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:40.831038+0000) 2020-02-05T23:43:41.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:41.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.886425+0000) 2020-02-05T23:43:41.888 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:41.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.886826+0000) 2020-02-05T23:43:41.889 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:41.886+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.887639+0000) 2020-02-05T23:43:41.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:41.886+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.888294+0000) 2020-02-05T23:43:41.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:41.887+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.889028+0000) 2020-02-05T23:43:41.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:41.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.890465+0000) 2020-02-05T23:43:41.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:41.888+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.890391+0000) 2020-02-05T23:43:41.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:41.890+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891067+0000) 2020-02-05T23:43:41.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:41.889+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891001+0000) 2020-02-05T23:43:41.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:41.890+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891337+0000) 2020-02-05T23:43:41.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:41.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891779+0000) 2020-02-05T23:43:41.895 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:41.890+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891568+0000) 2020-02-05T23:43:41.895 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:41.890+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.891891+0000) 2020-02-05T23:43:41.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:41.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:41.896914+0000) 2020-02-05T23:43:41.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:41.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153116 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:42.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:42.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:42.445495+0000) 2020-02-05T23:43:42.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:42.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:42.795186+0000) 2020-02-05T23:43:43.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:43.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:43.081848+0000) 2020-02-05T23:43:43.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:43.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:43.958314+0000) 2020-02-05T23:43:43.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:43.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:43.958314+0000) 2020-02-05T23:43:46.059 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:46.057+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.057955+0000) 2020-02-05T23:43:46.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:46.908+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.908896+0000) 2020-02-05T23:43:46.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:46.911+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.911216+0000) 2020-02-05T23:43:46.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:46.911+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.911330+0000) 2020-02-05T23:43:46.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:46.911+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.911849+0000) 2020-02-05T23:43:46.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:46.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.912311+0000) 2020-02-05T23:43:46.914 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:46.911+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.912607+0000) 2020-02-05T23:43:46.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:46.912+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.912807+0000) 2020-02-05T23:43:46.915 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:46.911+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.913368+0000) 2020-02-05T23:43:46.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:46.912+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.913841+0000) 2020-02-05T23:43:46.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:46.913+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.913412+0000) 2020-02-05T23:43:46.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:46.912+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.914253+0000) 2020-02-05T23:43:46.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:46.913+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.914680+0000) 2020-02-05T23:43:46.916 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:46.913+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.914960+0000) 2020-02-05T23:43:46.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:46.916+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.916984+0000) 2020-02-05T23:43:46.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:46.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:46.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:46.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:46.938620+0000) 2020-02-05T23:43:47.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:47.189+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:47.189199+0000) 2020-02-05T23:43:47.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:47.195+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:47.197053+0000) 2020-02-05T23:43:47.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:47.195+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:47.197232+0000) 2020-02-05T23:43:47.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:47.195+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:47.197302+0000) 2020-02-05T23:43:49.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:49.580+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:49.581824+0000) 2020-02-05T23:43:50.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:50.426+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:50.428166+0000) 2020-02-05T23:43:50.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:50.837+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:50.838920+0000) 2020-02-05T23:43:51.933 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:51.931+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.932156+0000) 2020-02-05T23:43:51.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:51.932+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.933622+0000) 2020-02-05T23:43:51.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:51.932+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.934282+0000) 2020-02-05T23:43:51.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:51.933+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.934321+0000) 2020-02-05T23:43:51.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:51.933+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.934910+0000) 2020-02-05T23:43:51.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:51.934+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.935472+0000) 2020-02-05T23:43:51.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:51.934+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.935964+0000) 2020-02-05T23:43:51.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:51.935+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.936539+0000) 2020-02-05T23:43:51.938 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:51.935+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.936888+0000) 2020-02-05T23:43:51.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:51.936+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.937521+0000) 2020-02-05T23:43:51.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:51.935+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.937369+0000) 2020-02-05T23:43:51.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:51.936+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.938066+0000) 2020-02-05T23:43:51.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:51.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.937796+0000) 2020-02-05T23:43:51.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:51.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:51.941546+0000) 2020-02-05T23:43:51.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:51.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153196 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:52.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:52.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:52.445737+0000) 2020-02-05T23:43:52.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:52.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:52.795392+0000) 2020-02-05T23:43:53.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:53.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:53.082104+0000) 2020-02-05T23:43:53.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:53.957+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:53.958587+0000) 2020-02-05T23:43:53.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:53.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:53.958602+0000) 2020-02-05T23:43:55.829 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:43:55.835 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:43:56.059 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:56.057+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.058284+0000) 2020-02-05T23:43:56.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:56.937+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.938829+0000) 2020-02-05T23:43:56.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:43:56.955+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.956631+0000) 2020-02-05T23:43:56.958 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:56.954+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.956203+0000) 2020-02-05T23:43:56.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:43:56.956+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.957742+0000) 2020-02-05T23:43:56.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:56.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.958660+0000) 2020-02-05T23:43:56.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:43:56.958+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.959256+0000) 2020-02-05T23:43:56.961 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:43:56.958+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.960318+0000) 2020-02-05T23:43:56.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:43:56.959+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.960294+0000) 2020-02-05T23:43:56.962 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:56.959+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.960835+0000) 2020-02-05T23:43:56.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:56.959+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.960963+0000) 2020-02-05T23:43:56.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:56.960+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.961153+0000) 2020-02-05T23:43:56.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:56.960+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.961646+0000) 2020-02-05T23:43:56.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:43:56.960+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.962339+0000) 2020-02-05T23:43:56.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:43:56.961+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.963003+0000) 2020-02-05T23:43:56.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:56.963+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:56.964996+0000) 2020-02-05T23:43:56.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:43:56.972+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:43:57.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:43:57.188+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:57.189411+0000) 2020-02-05T23:43:57.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:43:57.196+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:57.197262+0000) 2020-02-05T23:43:57.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:43:57.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:57.197480+0000) 2020-02-05T23:43:57.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:43:57.196+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:57.197497+0000) 2020-02-05T23:43:59.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:43:59.581+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:43:59.582044+0000) 2020-02-05T23:44:00.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:00.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:00.428424+0000) 2020-02-05T23:44:00.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:00.847+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:00.848134+0000) 2020-02-05T23:44:01.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:01.977+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.979129+0000) 2020-02-05T23:44:01.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:01.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.980397+0000) 2020-02-05T23:44:01.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:01.980+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.981188+0000) 2020-02-05T23:44:01.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:01.980+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.981510+0000) 2020-02-05T23:44:01.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:01.980+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.981914+0000) 2020-02-05T23:44:01.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:01.980+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.982060+0000) 2020-02-05T23:44:01.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:01.981+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.982519+0000) 2020-02-05T23:44:01.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:01.981+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.983046+0000) 2020-02-05T23:44:01.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:01.982+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.983692+0000) 2020-02-05T23:44:01.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:01.983+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.984719+0000) 2020-02-05T23:44:01.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:01.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.985063+0000) 2020-02-05T23:44:01.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:01.983+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.985327+0000) 2020-02-05T23:44:01.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:01.984+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.985514+0000) 2020-02-05T23:44:01.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:01.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:01.988511+0000) 2020-02-05T23:44:01.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:01.996+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153276 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:02.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:02.445+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:02.446047+0000) 2020-02-05T23:44:02.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:02.794+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:02.795635+0000) 2020-02-05T23:44:03.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:03.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:03.082360+0000) 2020-02-05T23:44:03.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:03.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:03.958876+0000) 2020-02-05T23:44:03.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:03.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:03.958886+0000) 2020-02-05T23:44:06.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:06.057+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:06.058547+0000) 2020-02-05T23:44:06.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:06.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:06.939059+0000) 2020-02-05T23:44:07.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:07.001+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.002602+0000) 2020-02-05T23:44:07.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:07.002+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.003552+0000) 2020-02-05T23:44:07.005 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:07.002+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.004139+0000) 2020-02-05T23:44:07.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:07.003+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.004218+0000) 2020-02-05T23:44:07.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:07.003+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.004566+0000) 2020-02-05T23:44:07.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:07.003+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.004688+0000) 2020-02-05T23:44:07.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:07.004+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.005313+0000) 2020-02-05T23:44:07.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:07.004+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.005421+0000) 2020-02-05T23:44:07.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:07.004+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.005720+0000) 2020-02-05T23:44:07.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:07.005+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.006177+0000) 2020-02-05T23:44:07.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:07.005+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.006640+0000) 2020-02-05T23:44:07.008 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:07.005+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.006912+0000) 2020-02-05T23:44:07.009 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:07.006+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.007842+0000) 2020-02-05T23:44:07.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:07.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.013545+0000) 2020-02-05T23:44:07.020 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:07.018+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153317 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:07.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:07.188+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.189634+0000) 2020-02-05T23:44:07.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:07.196+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.197485+0000) 2020-02-05T23:44:07.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:07.196+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.197683+0000) 2020-02-05T23:44:07.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:07.196+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:07.197784+0000) 2020-02-05T23:44:09.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:09.581+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:09.582326+0000) 2020-02-05T23:44:10.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:10.427+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:10.428753+0000) 2020-02-05T23:44:10.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:10.856+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:10.856775+0000) 2020-02-05T23:44:12.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:12.023+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.024657+0000) 2020-02-05T23:44:12.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:12.024+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.025464+0000) 2020-02-05T23:44:12.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:12.025+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.026398+0000) 2020-02-05T23:44:12.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:12.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.026955+0000) 2020-02-05T23:44:12.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:12.026+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.027570+0000) 2020-02-05T23:44:12.029 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:12.026+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.027757+0000) 2020-02-05T23:44:12.029 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:12.027+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.028361+0000) 2020-02-05T23:44:12.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:12.027+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.028693+0000) 2020-02-05T23:44:12.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:12.028+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.029615+0000) 2020-02-05T23:44:12.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:12.028+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.029725+0000) 2020-02-05T23:44:12.032 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:12.029+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.030759+0000) 2020-02-05T23:44:12.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:12.029+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.031017+0000) 2020-02-05T23:44:12.032 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:12.029+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.030874+0000) 2020-02-05T23:44:12.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:12.034+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.035420+0000) 2020-02-05T23:44:12.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:12.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:12.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:12.444+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.446256+0000) 2020-02-05T23:44:12.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:12.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:12.795844+0000) 2020-02-05T23:44:13.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:13.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:13.082621+0000) 2020-02-05T23:44:13.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:13.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:13.959123+0000) 2020-02-05T23:44:13.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:13.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:13.959124+0000) 2020-02-05T23:44:16.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:16.058+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:16.058791+0000) 2020-02-05T23:44:16.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:16.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:16.939338+0000) 2020-02-05T23:44:17.051 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:17.049+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.050679+0000) 2020-02-05T23:44:17.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:17.049+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.050837+0000) 2020-02-05T23:44:17.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:17.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.051345+0000) 2020-02-05T23:44:17.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:17.050+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.051705+0000) 2020-02-05T23:44:17.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:17.052+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.053069+0000) 2020-02-05T23:44:17.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:17.052+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.053651+0000) 2020-02-05T23:44:17.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:17.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.054168+0000) 2020-02-05T23:44:17.055 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:17.053+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.054440+0000) 2020-02-05T23:44:17.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:17.053+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.054625+0000) 2020-02-05T23:44:17.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:17.053+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.055135+0000) 2020-02-05T23:44:17.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:17.054+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.055670+0000) 2020-02-05T23:44:17.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:17.054+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.056146+0000) 2020-02-05T23:44:17.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:17.055+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.056673+0000) 2020-02-05T23:44:17.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:17.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.060117+0000) 2020-02-05T23:44:17.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:17.066+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:17.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:17.189+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.189844+0000) 2020-02-05T23:44:17.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:17.197+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.197713+0000) 2020-02-05T23:44:17.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:17.197+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.197876+0000) 2020-02-05T23:44:17.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:17.197+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:17.198011+0000) 2020-02-05T23:44:19.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:19.582+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:19.582562+0000) 2020-02-05T23:44:20.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:20.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:20.429005+0000) 2020-02-05T23:44:20.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:20.864+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:20.865120+0000) 2020-02-05T23:44:22.074 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:22.072+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.073568+0000) 2020-02-05T23:44:22.075 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:22.073+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.074476+0000) 2020-02-05T23:44:22.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:22.073+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.075229+0000) 2020-02-05T23:44:22.076 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:22.074+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.075582+0000) 2020-02-05T23:44:22.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:22.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.076136+0000) 2020-02-05T23:44:22.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:22.075+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.076629+0000) 2020-02-05T23:44:22.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:22.075+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.076963+0000) 2020-02-05T23:44:22.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:22.075+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.077337+0000) 2020-02-05T23:44:22.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:22.076+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.077458+0000) 2020-02-05T23:44:22.079 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:22.076+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.078016+0000) 2020-02-05T23:44:22.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:22.076+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.077790+0000) 2020-02-05T23:44:22.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:22.077+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.078571+0000) 2020-02-05T23:44:22.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:22.082+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.083106+0000) 2020-02-05T23:44:22.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:22.082+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.083363+0000) 2020-02-05T23:44:22.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:22.090+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:22.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:22.445+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.446484+0000) 2020-02-05T23:44:22.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:22.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:22.796071+0000) 2020-02-05T23:44:23.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:23.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:23.082866+0000) 2020-02-05T23:44:23.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:23.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:23.959383+0000) 2020-02-05T23:44:23.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:23.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:23.959386+0000) 2020-02-05T23:44:26.018 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:44:26.025 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:44:26.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:26.058+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:26.058998+0000) 2020-02-05T23:44:26.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:26.938+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:26.939521+0000) 2020-02-05T23:44:27.099 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:27.097+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.098106+0000) 2020-02-05T23:44:27.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:27.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.098544+0000) 2020-02-05T23:44:27.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:27.097+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.098880+0000) 2020-02-05T23:44:27.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:27.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.099214+0000) 2020-02-05T23:44:27.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:27.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.099458+0000) 2020-02-05T23:44:27.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:27.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.100045+0000) 2020-02-05T23:44:27.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:27.099+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.100668+0000) 2020-02-05T23:44:27.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:27.100+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.101181+0000) 2020-02-05T23:44:27.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:27.100+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.101417+0000) 2020-02-05T23:44:27.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:27.101+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.101860+0000) 2020-02-05T23:44:27.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:27.101+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.102320+0000) 2020-02-05T23:44:27.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:27.103+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.103469+0000) 2020-02-05T23:44:27.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:27.103+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.103819+0000) 2020-02-05T23:44:27.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:27.106+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.107629+0000) 2020-02-05T23:44:27.117 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:27.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:27.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:27.189+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.190060+0000) 2020-02-05T23:44:27.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:27.197+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.197932+0000) 2020-02-05T23:44:27.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:27.197+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.198132+0000) 2020-02-05T23:44:27.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:27.197+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:27.198228+0000) 2020-02-05T23:44:29.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:29.582+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:29.582796+0000) 2020-02-05T23:44:30.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:30.428+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:30.429234+0000) 2020-02-05T23:44:30.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:30.873+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:30.873715+0000) 2020-02-05T23:44:32.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:32.120+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.121159+0000) 2020-02-05T23:44:32.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:32.121+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.122275+0000) 2020-02-05T23:44:32.126 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:32.121+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.123060+0000) 2020-02-05T23:44:32.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:32.122+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.123535+0000) 2020-02-05T23:44:32.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:32.123+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.124204+0000) 2020-02-05T23:44:32.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:32.123+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.124462+0000) 2020-02-05T23:44:32.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:32.122+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.122718+0000) 2020-02-05T23:44:32.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:32.124+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.124691+0000) 2020-02-05T23:44:32.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:32.125+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.125422+0000) 2020-02-05T23:44:32.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:32.125+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.126116+0000) 2020-02-05T23:44:32.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:32.125+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.126557+0000) 2020-02-05T23:44:32.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:32.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.126790+0000) 2020-02-05T23:44:32.128 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:32.127+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.127884+0000) 2020-02-05T23:44:32.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:32.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.134119+0000) 2020-02-05T23:44:32.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:32.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153514 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:32.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:32.446+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.446771+0000) 2020-02-05T23:44:32.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:32.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:32.796367+0000) 2020-02-05T23:44:33.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:33.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:33.083120+0000) 2020-02-05T23:44:33.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:33.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:33.959622+0000) 2020-02-05T23:44:33.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:33.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:33.959625+0000) 2020-02-05T23:44:36.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:36.058+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:36.059217+0000) 2020-02-05T23:44:36.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:36.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:36.939784+0000) 2020-02-05T23:44:37.146 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:37.144+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.145439+0000) 2020-02-05T23:44:37.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:37.144+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.145876+0000) 2020-02-05T23:44:37.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:37.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.146906+0000) 2020-02-05T23:44:37.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:37.146+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.147199+0000) 2020-02-05T23:44:37.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:37.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.147539+0000) 2020-02-05T23:44:37.149 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:37.147+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.148006+0000) 2020-02-05T23:44:37.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:37.148+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.148660+0000) 2020-02-05T23:44:37.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:37.148+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.149016+0000) 2020-02-05T23:44:37.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:37.148+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.149481+0000) 2020-02-05T23:44:37.151 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:37.148+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.149319+0000) 2020-02-05T23:44:37.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:37.149+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.150282+0000) 2020-02-05T23:44:37.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:37.149+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.150670+0000) 2020-02-05T23:44:37.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:37.155+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.156897+0000) 2020-02-05T23:44:37.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:37.156+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.157174+0000) 2020-02-05T23:44:37.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:37.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:37.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:37.190+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.190315+0000) 2020-02-05T23:44:37.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:37.197+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.198150+0000) 2020-02-05T23:44:37.199 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:37.197+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.198306+0000) 2020-02-05T23:44:37.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:37.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:37.198410+0000) 2020-02-05T23:44:39.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:39.582+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:39.583055+0000) 2020-02-05T23:44:40.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:40.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:40.429482+0000) 2020-02-05T23:44:40.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:40.881+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:40.881797+0000) 2020-02-05T23:44:42.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:42.169+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.169440+0000) 2020-02-05T23:44:42.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:42.170+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.170537+0000) 2020-02-05T23:44:42.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:42.171+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.171170+0000) 2020-02-05T23:44:42.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:42.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.171547+0000) 2020-02-05T23:44:42.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:42.171+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.171868+0000) 2020-02-05T23:44:42.173 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:42.172+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.172543+0000) 2020-02-05T23:44:42.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:42.172+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.172536+0000) 2020-02-05T23:44:42.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:42.172+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.173034+0000) 2020-02-05T23:44:42.175 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:42.173+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.173842+0000) 2020-02-05T23:44:42.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:42.174+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.174792+0000) 2020-02-05T23:44:42.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:42.174+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.175133+0000) 2020-02-05T23:44:42.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:42.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.174953+0000) 2020-02-05T23:44:42.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:42.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.179669+0000) 2020-02-05T23:44:42.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:42.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.179900+0000) 2020-02-05T23:44:42.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:42.187+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:42.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:42.446+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.447015+0000) 2020-02-05T23:44:42.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:42.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:42.796585+0000) 2020-02-05T23:44:43.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:43.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:43.083282+0000) 2020-02-05T23:44:43.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:43.958+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:43.959860+0000) 2020-02-05T23:44:43.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:43.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:43.959868+0000) 2020-02-05T23:44:46.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:46.059+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:46.059485+0000) 2020-02-05T23:44:46.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:46.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:46.940007+0000) 2020-02-05T23:44:47.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:47.191+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.192979+0000) 2020-02-05T23:44:47.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:47.192+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.193640+0000) 2020-02-05T23:44:47.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:47.193+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.194396+0000) 2020-02-05T23:44:47.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:47.194+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.195621+0000) 2020-02-05T23:44:47.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:47.195+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.195610+0000) 2020-02-05T23:44:47.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:47.196+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.196513+0000) 2020-02-05T23:44:47.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:47.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.196625+0000) 2020-02-05T23:44:47.198 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:47.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.196973+0000) 2020-02-05T23:44:47.198 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:47.197+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.197782+0000) 2020-02-05T23:44:47.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:47.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.198712+0000) 2020-02-05T23:44:47.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:47.198+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.198512+0000) 2020-02-05T23:44:47.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:47.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.199025+0000) 2020-02-05T23:44:47.200 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:47.198+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.199100+0000) 2020-02-05T23:44:47.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:47.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.198507+0000) 2020-02-05T23:44:47.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:47.198+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.199651+0000) 2020-02-05T23:44:47.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:47.199+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.199961+0000) 2020-02-05T23:44:47.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:47.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:47.204961+0000) 2020-02-05T23:44:47.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:47.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153633 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:49.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:49.582+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:49.583354+0000) 2020-02-05T23:44:50.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:50.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:50.429768+0000) 2020-02-05T23:44:50.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:50.889+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:50.889988+0000) 2020-02-05T23:44:52.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:52.215+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.216399+0000) 2020-02-05T23:44:52.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:52.217+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.217952+0000) 2020-02-05T23:44:52.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:52.217+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.218442+0000) 2020-02-05T23:44:52.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:52.217+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.218244+0000) 2020-02-05T23:44:52.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:52.217+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.218810+0000) 2020-02-05T23:44:52.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:52.218+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.219309+0000) 2020-02-05T23:44:52.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:52.219+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.220336+0000) 2020-02-05T23:44:52.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:52.220+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.221249+0000) 2020-02-05T23:44:52.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:52.220+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.220858+0000) 2020-02-05T23:44:52.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:52.220+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.221849+0000) 2020-02-05T23:44:52.223 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:52.221+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.222399+0000) 2020-02-05T23:44:52.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:52.222+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.222682+0000) 2020-02-05T23:44:52.224 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:52.222+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.222907+0000) 2020-02-05T23:44:52.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:52.226+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.227489+0000) 2020-02-05T23:44:52.233 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:52.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:52.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:52.446+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.447230+0000) 2020-02-05T23:44:52.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:52.795+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:52.796824+0000) 2020-02-05T23:44:53.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:53.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:53.083488+0000) 2020-02-05T23:44:53.942 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:53.940+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:53.941177+0000) 2020-02-05T23:44:53.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:53.959+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:53.960111+0000) 2020-02-05T23:44:53.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:53.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:53.960132+0000) 2020-02-05T23:44:56.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:56.059+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:56.059661+0000) 2020-02-05T23:44:56.148 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:44:56.155 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:44:56.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:56.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:56.940270+0000) 2020-02-05T23:44:57.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:57.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.198733+0000) 2020-02-05T23:44:57.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:57.198+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.198733+0000) 2020-02-05T23:44:57.200 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:57.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.198919+0000) 2020-02-05T23:44:57.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:44:57.239+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.240130+0000) 2020-02-05T23:44:57.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:44:57.240+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.240834+0000) 2020-02-05T23:44:57.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:44:57.241+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.241524+0000) 2020-02-05T23:44:57.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:44:57.240+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.241797+0000) 2020-02-05T23:44:57.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:57.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.242364+0000) 2020-02-05T23:44:57.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:44:57.242+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.242716+0000) 2020-02-05T23:44:57.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:57.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.243199+0000) 2020-02-05T23:44:57.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:44:57.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.244042+0000) 2020-02-05T23:44:57.245 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:44:57.243+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.244439+0000) 2020-02-05T23:44:57.246 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:44:57.243+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.244863+0000) 2020-02-05T23:44:57.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:44:57.245+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.245856+0000) 2020-02-05T23:44:57.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:44:57.245+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.245989+0000) 2020-02-05T23:44:57.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:44:57.246+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.246544+0000) 2020-02-05T23:44:57.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:57.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:57.251336+0000) 2020-02-05T23:44:57.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:44:57.256+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153712 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:44:59.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:44:59.582+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:44:59.583590+0000) 2020-02-05T23:45:00.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:00.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:00.430020+0000) 2020-02-05T23:45:00.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:00.897+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:00.898776+0000) 2020-02-05T23:45:02.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:02.260+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.261576+0000) 2020-02-05T23:45:02.263 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:02.261+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.262397+0000) 2020-02-05T23:45:02.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:02.261+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.263053+0000) 2020-02-05T23:45:02.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:02.262+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.263772+0000) 2020-02-05T23:45:02.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:02.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.264303+0000) 2020-02-05T23:45:02.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:02.264+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.264994+0000) 2020-02-05T23:45:02.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:02.264+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.265180+0000) 2020-02-05T23:45:02.267 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:02.265+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.265622+0000) 2020-02-05T23:45:02.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:02.265+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.266394+0000) 2020-02-05T23:45:02.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:02.266+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.266485+0000) 2020-02-05T23:45:02.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:02.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.266910+0000) 2020-02-05T23:45:02.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:02.265+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.267008+0000) 2020-02-05T23:45:02.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:02.268+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.269002+0000) 2020-02-05T23:45:02.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:02.271+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.272946+0000) 2020-02-05T23:45:02.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:02.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:02.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:02.447+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.447495+0000) 2020-02-05T23:45:02.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:02.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:02.797043+0000) 2020-02-05T23:45:03.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:03.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:03.083699+0000) 2020-02-05T23:45:03.942 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:03.940+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:03.941421+0000) 2020-02-05T23:45:03.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:03.959+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:03.960370+0000) 2020-02-05T23:45:03.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:03.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:03.960370+0000) 2020-02-05T23:45:06.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:06.058+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:06.059915+0000) 2020-02-05T23:45:06.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:06.939+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:06.940500+0000) 2020-02-05T23:45:07.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:07.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.199032+0000) 2020-02-05T23:45:07.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:07.198+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.199032+0000) 2020-02-05T23:45:07.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:07.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.199192+0000) 2020-02-05T23:45:07.284 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:07.282+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.283193+0000) 2020-02-05T23:45:07.285 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:07.283+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.284072+0000) 2020-02-05T23:45:07.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:07.283+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.284673+0000) 2020-02-05T23:45:07.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:07.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.286248+0000) 2020-02-05T23:45:07.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:07.285+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.286563+0000) 2020-02-05T23:45:07.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:07.285+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.287066+0000) 2020-02-05T23:45:07.289 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:07.286+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.288099+0000) 2020-02-05T23:45:07.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:07.286+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.288222+0000) 2020-02-05T23:45:07.290 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:07.287+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.288548+0000) 2020-02-05T23:45:07.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:07.287+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.288887+0000) 2020-02-05T23:45:07.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:07.287+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.289068+0000) 2020-02-05T23:45:07.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:07.288+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.289862+0000) 2020-02-05T23:45:07.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:07.288+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.289984+0000) 2020-02-05T23:45:07.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:07.291+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:07.292928+0000) 2020-02-05T23:45:07.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:07.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:09.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:09.583+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:09.583807+0000) 2020-02-05T23:45:10.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:10.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:10.430352+0000) 2020-02-05T23:45:10.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:10.905+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:10.906113+0000) 2020-02-05T23:45:12.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:12.305+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.306938+0000) 2020-02-05T23:45:12.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:12.305+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.307042+0000) 2020-02-05T23:45:12.309 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:12.306+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.307443+0000) 2020-02-05T23:45:12.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:12.306+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.307851+0000) 2020-02-05T23:45:12.309 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:12.306+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.308002+0000) 2020-02-05T23:45:12.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:12.307+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.308801+0000) 2020-02-05T23:45:12.310 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:12.308+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.309414+0000) 2020-02-05T23:45:12.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:12.307+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.309402+0000) 2020-02-05T23:45:12.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:12.308+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.310191+0000) 2020-02-05T23:45:12.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:12.309+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.310612+0000) 2020-02-05T23:45:12.312 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:12.309+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.310623+0000) 2020-02-05T23:45:12.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:12.310+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.311841+0000) 2020-02-05T23:45:12.313 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:12.310+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.312352+0000) 2020-02-05T23:45:12.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:12.313+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.314650+0000) 2020-02-05T23:45:12.325 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:12.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:12.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:12.446+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.447740+0000) 2020-02-05T23:45:12.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:12.796+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:12.797250+0000) 2020-02-05T23:45:13.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:13.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:13.083925+0000) 2020-02-05T23:45:13.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:13.941+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:13.941691+0000) 2020-02-05T23:45:13.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:13.960+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:13.960667+0000) 2020-02-05T23:45:13.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:13.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:13.960675+0000) 2020-02-05T23:45:16.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:16.059+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:16.060158+0000) 2020-02-05T23:45:16.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:16.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:16.940738+0000) 2020-02-05T23:45:17.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:17.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.199335+0000) 2020-02-05T23:45:17.201 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:17.198+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.199335+0000) 2020-02-05T23:45:17.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:17.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.199524+0000) 2020-02-05T23:45:17.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:17.328+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.329472+0000) 2020-02-05T23:45:17.331 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:17.328+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.330013+0000) 2020-02-05T23:45:17.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:17.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.330542+0000) 2020-02-05T23:45:17.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:17.329+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.331204+0000) 2020-02-05T23:45:17.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:17.330+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.331466+0000) 2020-02-05T23:45:17.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:17.331+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.332125+0000) 2020-02-05T23:45:17.333 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:17.331+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.332720+0000) 2020-02-05T23:45:17.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:17.332+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.333632+0000) 2020-02-05T23:45:17.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:17.333+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.334504+0000) 2020-02-05T23:45:17.335 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:17.333+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.334465+0000) 2020-02-05T23:45:17.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:17.333+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.334812+0000) 2020-02-05T23:45:17.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:17.334+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.335512+0000) 2020-02-05T23:45:17.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:17.339+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.340407+0000) 2020-02-05T23:45:17.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:17.339+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:17.340712+0000) 2020-02-05T23:45:17.348 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:17.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153870 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:19.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:19.583+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:19.584043+0000) 2020-02-05T23:45:20.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:20.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:20.430685+0000) 2020-02-05T23:45:20.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:20.913+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:20.913686+0000) 2020-02-05T23:45:22.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:22.351+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.352238+0000) 2020-02-05T23:45:22.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:22.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.352928+0000) 2020-02-05T23:45:22.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:22.352+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.353485+0000) 2020-02-05T23:45:22.356 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:22.353+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.354980+0000) 2020-02-05T23:45:22.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:22.354+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.355621+0000) 2020-02-05T23:45:22.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:22.355+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.356697+0000) 2020-02-05T23:45:22.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:22.356+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.357300+0000) 2020-02-05T23:45:22.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:22.356+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.358244+0000) 2020-02-05T23:45:22.359 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:22.357+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.358462+0000) 2020-02-05T23:45:22.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:22.357+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.358792+0000) 2020-02-05T23:45:22.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:22.357+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.358700+0000) 2020-02-05T23:45:22.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:22.357+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.358772+0000) 2020-02-05T23:45:22.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:22.357+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.359077+0000) 2020-02-05T23:45:22.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:22.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.364287+0000) 2020-02-05T23:45:22.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:22.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:22.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:22.446+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.447955+0000) 2020-02-05T23:45:22.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:22.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:22.797497+0000) 2020-02-05T23:45:23.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:23.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:23.084138+0000) 2020-02-05T23:45:23.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:23.941+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:23.941930+0000) 2020-02-05T23:45:23.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:23.960+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:23.960930+0000) 2020-02-05T23:45:23.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:23.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:23.960921+0000) 2020-02-05T23:45:26.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:26.059+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:26.060415+0000) 2020-02-05T23:45:26.297 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:45:26.303 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:45:26.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:26.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:26.940974+0000) 2020-02-05T23:45:27.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:27.198+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.199682+0000) 2020-02-05T23:45:27.201 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:27.198+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.199682+0000) 2020-02-05T23:45:27.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:27.198+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.199831+0000) 2020-02-05T23:45:27.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:27.375+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.376618+0000) 2020-02-05T23:45:27.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:27.375+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.377008+0000) 2020-02-05T23:45:27.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:27.376+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.377860+0000) 2020-02-05T23:45:27.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:27.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.378423+0000) 2020-02-05T23:45:27.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:27.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.379041+0000) 2020-02-05T23:45:27.381 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:27.378+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.379871+0000) 2020-02-05T23:45:27.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:27.379+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.380317+0000) 2020-02-05T23:45:27.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:27.379+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.381001+0000) 2020-02-05T23:45:27.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:27.380+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.381496+0000) 2020-02-05T23:45:27.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:27.380+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.381461+0000) 2020-02-05T23:45:27.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:27.380+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.382258+0000) 2020-02-05T23:45:27.384 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:27.382+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.383655+0000) 2020-02-05T23:45:27.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:27.381+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.382918+0000) 2020-02-05T23:45:27.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:27.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:27.387903+0000) 2020-02-05T23:45:27.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:27.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:29.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:29.584+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:29.584338+0000) 2020-02-05T23:45:30.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:30.430+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:30.430928+0000) 2020-02-05T23:45:30.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:30.921+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:30.921378+0000) 2020-02-05T23:45:32.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:32.400+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.401141+0000) 2020-02-05T23:45:32.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:32.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.402179+0000) 2020-02-05T23:45:32.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:32.401+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.402573+0000) 2020-02-05T23:45:32.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:32.401+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.402947+0000) 2020-02-05T23:45:32.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:32.402+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.403927+0000) 2020-02-05T23:45:32.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:32.402+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.403873+0000) 2020-02-05T23:45:32.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:32.403+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.404677+0000) 2020-02-05T23:45:32.406 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:32.403+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.404625+0000) 2020-02-05T23:45:32.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:32.404+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.405937+0000) 2020-02-05T23:45:32.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:32.405+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.406190+0000) 2020-02-05T23:45:32.407 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:32.405+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.406561+0000) 2020-02-05T23:45:32.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:32.405+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.407172+0000) 2020-02-05T23:45:32.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:32.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.407370+0000) 2020-02-05T23:45:32.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:32.411+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.412904+0000) 2020-02-05T23:45:32.419 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:32.416+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 153988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:32.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:32.447+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.448186+0000) 2020-02-05T23:45:32.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:32.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:32.797714+0000) 2020-02-05T23:45:33.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:33.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:33.084373+0000) 2020-02-05T23:45:33.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:33.942+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:33.942185+0000) 2020-02-05T23:45:33.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:33.961+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:33.961160+0000) 2020-02-05T23:45:33.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:33.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:33.961155+0000) 2020-02-05T23:45:36.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:36.060+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:36.060714+0000) 2020-02-05T23:45:36.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:36.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:36.941228+0000) 2020-02-05T23:45:37.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:37.199+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.199968+0000) 2020-02-05T23:45:37.201 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:37.199+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.199985+0000) 2020-02-05T23:45:37.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:37.199+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.200115+0000) 2020-02-05T23:45:37.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:37.422+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.423161+0000) 2020-02-05T23:45:37.425 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:37.423+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.424452+0000) 2020-02-05T23:45:37.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:37.423+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.424766+0000) 2020-02-05T23:45:37.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:37.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.425368+0000) 2020-02-05T23:45:37.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:37.423+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.425314+0000) 2020-02-05T23:45:37.428 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:37.424+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.425598+0000) 2020-02-05T23:45:37.428 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:37.424+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.426279+0000) 2020-02-05T23:45:37.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:37.425+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.427058+0000) 2020-02-05T23:45:37.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:37.425+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.427396+0000) 2020-02-05T23:45:37.429 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:37.427+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.428309+0000) 2020-02-05T23:45:37.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:37.427+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.428551+0000) 2020-02-05T23:45:37.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:37.427+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.429335+0000) 2020-02-05T23:45:37.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:37.428+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.429757+0000) 2020-02-05T23:45:37.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:37.431+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:37.432304+0000) 2020-02-05T23:45:37.441 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:37.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:39.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:39.583+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:39.584577+0000) 2020-02-05T23:45:40.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:40.429+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:40.431192+0000) 2020-02-05T23:45:40.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:40.928+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:40.929360+0000) 2020-02-05T23:45:42.446 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:42.443+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.444617+0000) 2020-02-05T23:45:42.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:42.444+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.445460+0000) 2020-02-05T23:45:42.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:42.445+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.446938+0000) 2020-02-05T23:45:42.448 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:42.446+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.447670+0000) 2020-02-05T23:45:42.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:42.446+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.448002+0000) 2020-02-05T23:45:42.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:42.447+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.448529+0000) 2020-02-05T23:45:42.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:42.447+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.448748+0000) 2020-02-05T23:45:42.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:42.447+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.449004+0000) 2020-02-05T23:45:42.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:42.448+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.449754+0000) 2020-02-05T23:45:42.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:42.449+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.450247+0000) 2020-02-05T23:45:42.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:42.449+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.450646+0000) 2020-02-05T23:45:42.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:42.449+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.451181+0000) 2020-02-05T23:45:42.453 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:42.450+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.452319+0000) 2020-02-05T23:45:42.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:42.454+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.455187+0000) 2020-02-05T23:45:42.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:42.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:42.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:42.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:42.797966+0000) 2020-02-05T23:45:43.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:43.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:43.084598+0000) 2020-02-05T23:45:43.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:43.941+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:43.942410+0000) 2020-02-05T23:45:43.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:43.960+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:43.961399+0000) 2020-02-05T23:45:43.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:43.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:43.961391+0000) 2020-02-05T23:45:46.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:46.060+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:46.060969+0000) 2020-02-05T23:45:46.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:46.940+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:46.941458+0000) 2020-02-05T23:45:47.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:47.199+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.200279+0000) 2020-02-05T23:45:47.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:47.199+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.200279+0000) 2020-02-05T23:45:47.202 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:47.199+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.200476+0000) 2020-02-05T23:45:47.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:47.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.471667+0000) 2020-02-05T23:45:47.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:47.471+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.472148+0000) 2020-02-05T23:45:47.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:47.471+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.472647+0000) 2020-02-05T23:45:47.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:47.472+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.473431+0000) 2020-02-05T23:45:47.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:47.471+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.473187+0000) 2020-02-05T23:45:47.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:47.472+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.474005+0000) 2020-02-05T23:45:47.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:47.473+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.474405+0000) 2020-02-05T23:45:47.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:47.473+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.474679+0000) 2020-02-05T23:45:47.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:47.473+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.475220+0000) 2020-02-05T23:45:47.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:47.474+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.475506+0000) 2020-02-05T23:45:47.477 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:47.474+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.476315+0000) 2020-02-05T23:45:47.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:47.475+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.476688+0000) 2020-02-05T23:45:47.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:47.475+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.476840+0000) 2020-02-05T23:45:47.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:47.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:47.481608+0000) 2020-02-05T23:45:47.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:47.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:49.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:49.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:49.198933+0000) 2020-02-05T23:45:49.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:49.583+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:49.584807+0000) 2020-02-05T23:45:50.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:50.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:50.431550+0000) 2020-02-05T23:45:50.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:50.936+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:50.937825+0000) 2020-02-05T23:45:52.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:52.492+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.493654+0000) 2020-02-05T23:45:52.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:52.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.494277+0000) 2020-02-05T23:45:52.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:52.493+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.494714+0000) 2020-02-05T23:45:52.496 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:52.493+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.494951+0000) 2020-02-05T23:45:52.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:52.494+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.495460+0000) 2020-02-05T23:45:52.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:52.494+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.495485+0000) 2020-02-05T23:45:52.497 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:52.494+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.495641+0000) 2020-02-05T23:45:52.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:52.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.495958+0000) 2020-02-05T23:45:52.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:52.495+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.497387+0000) 2020-02-05T23:45:52.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:52.496+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.497873+0000) 2020-02-05T23:45:52.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:52.496+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.498024+0000) 2020-02-05T23:45:52.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:52.496+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.498365+0000) 2020-02-05T23:45:52.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:52.497+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.498820+0000) 2020-02-05T23:45:52.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:52.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.501878+0000) 2020-02-05T23:45:52.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:52.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:52.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:52.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:52.798172+0000) 2020-02-05T23:45:53.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:53.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:53.084809+0000) 2020-02-05T23:45:53.943 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:53.941+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:53.942642+0000) 2020-02-05T23:45:53.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:53.960+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:53.961574+0000) 2020-02-05T23:45:53.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:53.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:53.961619+0000) 2020-02-05T23:45:56.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:56.060+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:56.061219+0000) 2020-02-05T23:45:56.454 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:45:56.460 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:45:56.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:56.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:56.941710+0000) 2020-02-05T23:45:57.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:57.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.200582+0000) 2020-02-05T23:45:57.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:57.200+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.200582+0000) 2020-02-05T23:45:57.202 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:57.200+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.200767+0000) 2020-02-05T23:45:57.516 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:45:57.513+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.514880+0000) 2020-02-05T23:45:57.517 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:45:57.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.516131+0000) 2020-02-05T23:45:57.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:57.515+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.516305+0000) 2020-02-05T23:45:57.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:45:57.515+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.516520+0000) 2020-02-05T23:45:57.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:45:57.516+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.516962+0000) 2020-02-05T23:45:57.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:57.517+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.517484+0000) 2020-02-05T23:45:57.519 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:45:57.515+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.516744+0000) 2020-02-05T23:45:57.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:57.516+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.517770+0000) 2020-02-05T23:45:57.520 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:45:57.517+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.517845+0000) 2020-02-05T23:45:57.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:45:57.518+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.518517+0000) 2020-02-05T23:45:57.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:45:57.518+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.518617+0000) 2020-02-05T23:45:57.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:45:57.517+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.518655+0000) 2020-02-05T23:45:57.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:45:57.519+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.519951+0000) 2020-02-05T23:45:57.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:57.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:57.523810+0000) 2020-02-05T23:45:57.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:45:57.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:45:59.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:45:59.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:59.199192+0000) 2020-02-05T23:45:59.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:45:59.584+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:45:59.585041+0000) 2020-02-05T23:46:00.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:00.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:00.431783+0000) 2020-02-05T23:46:00.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:00.945+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:00.946369+0000) 2020-02-05T23:46:02.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:02.535+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.536374+0000) 2020-02-05T23:46:02.539 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:02.537+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.537858+0000) 2020-02-05T23:46:02.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:02.537+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.538331+0000) 2020-02-05T23:46:02.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:02.538+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.538970+0000) 2020-02-05T23:46:02.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:02.538+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.539236+0000) 2020-02-05T23:46:02.541 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:02.539+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.539666+0000) 2020-02-05T23:46:02.541 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:02.538+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.540007+0000) 2020-02-05T23:46:02.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:02.539+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.541020+0000) 2020-02-05T23:46:02.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:02.540+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.541577+0000) 2020-02-05T23:46:02.542 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:02.540+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.541530+0000) 2020-02-05T23:46:02.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:02.541+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.541693+0000) 2020-02-05T23:46:02.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:02.541+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.542190+0000) 2020-02-05T23:46:02.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:02.540+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.541874+0000) 2020-02-05T23:46:02.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:02.548+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.549207+0000) 2020-02-05T23:46:02.556 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:02.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:02.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:02.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:02.798396+0000) 2020-02-05T23:46:03.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:03.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:03.085044+0000) 2020-02-05T23:46:03.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:03.941+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:03.942900+0000) 2020-02-05T23:46:03.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:03.960+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:03.961813+0000) 2020-02-05T23:46:03.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:03.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:03.961922+0000) 2020-02-05T23:46:06.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:06.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:06.061474+0000) 2020-02-05T23:46:06.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:06.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:06.941947+0000) 2020-02-05T23:46:07.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:07.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.200882+0000) 2020-02-05T23:46:07.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:07.200+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.200882+0000) 2020-02-05T23:46:07.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:07.200+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.201066+0000) 2020-02-05T23:46:07.561 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:07.559+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.560038+0000) 2020-02-05T23:46:07.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:07.561+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.561913+0000) 2020-02-05T23:46:07.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:07.560+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.562045+0000) 2020-02-05T23:46:07.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:07.561+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.562396+0000) 2020-02-05T23:46:07.564 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:07.562+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.563005+0000) 2020-02-05T23:46:07.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:07.562+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.563711+0000) 2020-02-05T23:46:07.565 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:07.563+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.564118+0000) 2020-02-05T23:46:07.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:07.563+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.564325+0000) 2020-02-05T23:46:07.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:07.563+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.564803+0000) 2020-02-05T23:46:07.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:07.565+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.566389+0000) 2020-02-05T23:46:07.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:07.565+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.566163+0000) 2020-02-05T23:46:07.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:07.566+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.566746+0000) 2020-02-05T23:46:07.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:07.566+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.567355+0000) 2020-02-05T23:46:07.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:07.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:07.570068+0000) 2020-02-05T23:46:07.579 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:07.577+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:09.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:09.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:09.199494+0000) 2020-02-05T23:46:09.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:09.584+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:09.585333+0000) 2020-02-05T23:46:10.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:10.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:10.432044+0000) 2020-02-05T23:46:10.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:10.954+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:10.954636+0000) 2020-02-05T23:46:12.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:12.582+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.583392+0000) 2020-02-05T23:46:12.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:12.582+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.583893+0000) 2020-02-05T23:46:12.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:12.583+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.584796+0000) 2020-02-05T23:46:12.586 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:12.584+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.585555+0000) 2020-02-05T23:46:12.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:12.585+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.586118+0000) 2020-02-05T23:46:12.588 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:12.585+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.586295+0000) 2020-02-05T23:46:12.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:12.586+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.586842+0000) 2020-02-05T23:46:12.589 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:12.586+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.586846+0000) 2020-02-05T23:46:12.590 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:12.586+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.587454+0000) 2020-02-05T23:46:12.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:12.587+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.588061+0000) 2020-02-05T23:46:12.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:12.587+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.587610+0000) 2020-02-05T23:46:12.591 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:12.587+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.588207+0000) 2020-02-05T23:46:12.591 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:12.589+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.589813+0000) 2020-02-05T23:46:12.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:12.590+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.591451+0000) 2020-02-05T23:46:12.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:12.600+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:12.799 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:12.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:12.798598+0000) 2020-02-05T23:46:13.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:13.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:13.085329+0000) 2020-02-05T23:46:13.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:13.942+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:13.943190+0000) 2020-02-05T23:46:13.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:13.961+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:13.962038+0000) 2020-02-05T23:46:13.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:13.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:13.962156+0000) 2020-02-05T23:46:16.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:16.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:16.061726+0000) 2020-02-05T23:46:16.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:16.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:16.942255+0000) 2020-02-05T23:46:17.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:17.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.201162+0000) 2020-02-05T23:46:17.203 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:17.200+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.201162+0000) 2020-02-05T23:46:17.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:17.200+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.201359+0000) 2020-02-05T23:46:17.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:17.605+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.606416+0000) 2020-02-05T23:46:17.608 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:17.605+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.606908+0000) 2020-02-05T23:46:17.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:17.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.607997+0000) 2020-02-05T23:46:17.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:17.608+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.608748+0000) 2020-02-05T23:46:17.610 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:17.608+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.609427+0000) 2020-02-05T23:46:17.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:17.609+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.609812+0000) 2020-02-05T23:46:17.611 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:17.609+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.610367+0000) 2020-02-05T23:46:17.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:17.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.610522+0000) 2020-02-05T23:46:17.612 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:17.610+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.611143+0000) 2020-02-05T23:46:17.613 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:17.611+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.611913+0000) 2020-02-05T23:46:17.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:17.611+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.612216+0000) 2020-02-05T23:46:17.613 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:17.611+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.612389+0000) 2020-02-05T23:46:17.614 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:17.612+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.612560+0000) 2020-02-05T23:46:17.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:17.619+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:17.620901+0000) 2020-02-05T23:46:17.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:17.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154342 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:19.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:19.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:19.199818+0000) 2020-02-05T23:46:19.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:19.585+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:19.585565+0000) 2020-02-05T23:46:20.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:20.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:20.432298+0000) 2020-02-05T23:46:20.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:20.961+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:20.962793+0000) 2020-02-05T23:46:22.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:22.628+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.629736+0000) 2020-02-05T23:46:22.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:22.629+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.630424+0000) 2020-02-05T23:46:22.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:22.630+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.631141+0000) 2020-02-05T23:46:22.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:22.630+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.631645+0000) 2020-02-05T23:46:22.633 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:22.631+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.632106+0000) 2020-02-05T23:46:22.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:22.631+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.632377+0000) 2020-02-05T23:46:22.635 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:22.633+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.633548+0000) 2020-02-05T23:46:22.635 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:22.632+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.634038+0000) 2020-02-05T23:46:22.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:22.633+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.634391+0000) 2020-02-05T23:46:22.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:22.633+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.634665+0000) 2020-02-05T23:46:22.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:22.634+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.635236+0000) 2020-02-05T23:46:22.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:22.634+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.635361+0000) 2020-02-05T23:46:22.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:22.635+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.635777+0000) 2020-02-05T23:46:22.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:22.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.639333+0000) 2020-02-05T23:46:22.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:22.648+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:22.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:22.797+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:22.798847+0000) 2020-02-05T23:46:23.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:23.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:23.085519+0000) 2020-02-05T23:46:23.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:23.943+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:23.943424+0000) 2020-02-05T23:46:23.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:23.961+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:23.962297+0000) 2020-02-05T23:46:23.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:23.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:23.962347+0000) 2020-02-05T23:46:26.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:26.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:26.061984+0000) 2020-02-05T23:46:26.602 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:46:26.608 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:46:26.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:26.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:26.942488+0000) 2020-02-05T23:46:27.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:27.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.201499+0000) 2020-02-05T23:46:27.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:27.201+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.201499+0000) 2020-02-05T23:46:27.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:27.201+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.201689+0000) 2020-02-05T23:46:27.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:27.653+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.653796+0000) 2020-02-05T23:46:27.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:27.654+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.654561+0000) 2020-02-05T23:46:27.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:27.655+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.655372+0000) 2020-02-05T23:46:27.657 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:27.655+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.655808+0000) 2020-02-05T23:46:27.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:27.656+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.656622+0000) 2020-02-05T23:46:27.659 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:27.658+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.658123+0000) 2020-02-05T23:46:27.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:27.658+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.658782+0000) 2020-02-05T23:46:27.660 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:27.658+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.659374+0000) 2020-02-05T23:46:27.661 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:27.659+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.659558+0000) 2020-02-05T23:46:27.661 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:27.659+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.660334+0000) 2020-02-05T23:46:27.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:27.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.660193+0000) 2020-02-05T23:46:27.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:27.660+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.660749+0000) 2020-02-05T23:46:27.662 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:27.660+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.660813+0000) 2020-02-05T23:46:27.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:27.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:27.664416+0000) 2020-02-05T23:46:27.673 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:27.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:29.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:29.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:29.200054+0000) 2020-02-05T23:46:29.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:29.585+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:29.585812+0000) 2020-02-05T23:46:30.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:30.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:30.432533+0000) 2020-02-05T23:46:30.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:30.974+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:30.974625+0000) 2020-02-05T23:46:32.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:32.677+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.678379+0000) 2020-02-05T23:46:32.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:32.678+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.679500+0000) 2020-02-05T23:46:32.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:32.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.681024+0000) 2020-02-05T23:46:32.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:32.680+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.681129+0000) 2020-02-05T23:46:32.683 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:32.680+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.681516+0000) 2020-02-05T23:46:32.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:32.680+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.681953+0000) 2020-02-05T23:46:32.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:32.680+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.682390+0000) 2020-02-05T23:46:32.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:32.680+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.682375+0000) 2020-02-05T23:46:32.684 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:32.682+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.683382+0000) 2020-02-05T23:46:32.685 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:32.682+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.683994+0000) 2020-02-05T23:46:32.685 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:32.682+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.684263+0000) 2020-02-05T23:46:32.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:32.683+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.684667+0000) 2020-02-05T23:46:32.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:32.683+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.684389+0000) 2020-02-05T23:46:32.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:32.685+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.686239+0000) 2020-02-05T23:46:32.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:32.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:32.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:32.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:32.799061+0000) 2020-02-05T23:46:33.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:33.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:33.085730+0000) 2020-02-05T23:46:33.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:33.942+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:33.943653+0000) 2020-02-05T23:46:33.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:33.961+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:33.962531+0000) 2020-02-05T23:46:33.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:33.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:33.962575+0000) 2020-02-05T23:46:36.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:36.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:36.062234+0000) 2020-02-05T23:46:36.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:36.941+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:36.942721+0000) 2020-02-05T23:46:37.203 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:37.200+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.201789+0000) 2020-02-05T23:46:37.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:37.200+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.201784+0000) 2020-02-05T23:46:37.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:37.200+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.201938+0000) 2020-02-05T23:46:37.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:37.699+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.700857+0000) 2020-02-05T23:46:37.703 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:37.700+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.702023+0000) 2020-02-05T23:46:37.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:37.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.702603+0000) 2020-02-05T23:46:37.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:37.702+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.703499+0000) 2020-02-05T23:46:37.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:37.702+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.703690+0000) 2020-02-05T23:46:37.705 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:37.702+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.704035+0000) 2020-02-05T23:46:37.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:37.703+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.704404+0000) 2020-02-05T23:46:37.706 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:37.703+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.704823+0000) 2020-02-05T23:46:37.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:37.704+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.705478+0000) 2020-02-05T23:46:37.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:37.704+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.705415+0000) 2020-02-05T23:46:37.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:37.704+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.705998+0000) 2020-02-05T23:46:37.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:37.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.705905+0000) 2020-02-05T23:46:37.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:37.705+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.706775+0000) 2020-02-05T23:46:37.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:37.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:37.712034+0000) 2020-02-05T23:46:37.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:37.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:39.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:39.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:39.200307+0000) 2020-02-05T23:46:39.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:39.585+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:39.586051+0000) 2020-02-05T23:46:40.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:40.431+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:40.432764+0000) 2020-02-05T23:46:40.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:40.981+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:40.982649+0000) 2020-02-05T23:46:42.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:42.724+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.726018+0000) 2020-02-05T23:46:42.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:42.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.726544+0000) 2020-02-05T23:46:42.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:42.725+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.727050+0000) 2020-02-05T23:46:42.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:42.726+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.727856+0000) 2020-02-05T23:46:42.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:42.726+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.727691+0000) 2020-02-05T23:46:42.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:42.727+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.728402+0000) 2020-02-05T23:46:42.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:42.728+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.729233+0000) 2020-02-05T23:46:42.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:42.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.729300+0000) 2020-02-05T23:46:42.731 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:42.728+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.729543+0000) 2020-02-05T23:46:42.732 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:42.728+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.730127+0000) 2020-02-05T23:46:42.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:42.728+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.730267+0000) 2020-02-05T23:46:42.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:42.729+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.731048+0000) 2020-02-05T23:46:42.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:42.730+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.731495+0000) 2020-02-05T23:46:42.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:42.734+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.735629+0000) 2020-02-05T23:46:42.744 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:42.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:42.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:42.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:42.799336+0000) 2020-02-05T23:46:43.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:43.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:43.085939+0000) 2020-02-05T23:46:43.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:43.943+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:43.943883+0000) 2020-02-05T23:46:43.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:43.961+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:43.962761+0000) 2020-02-05T23:46:43.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:43.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:43.962814+0000) 2020-02-05T23:46:46.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:46.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:46.062491+0000) 2020-02-05T23:46:46.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:46.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:46.942954+0000) 2020-02-05T23:46:47.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:47.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.202064+0000) 2020-02-05T23:46:47.203 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:47.201+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.202064+0000) 2020-02-05T23:46:47.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:47.201+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.202255+0000) 2020-02-05T23:46:47.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:47.747+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.749009+0000) 2020-02-05T23:46:47.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:47.748+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.749733+0000) 2020-02-05T23:46:47.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:47.748+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.749979+0000) 2020-02-05T23:46:47.751 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:47.749+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.750308+0000) 2020-02-05T23:46:47.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:47.749+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.750789+0000) 2020-02-05T23:46:47.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:47.750+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.751235+0000) 2020-02-05T23:46:47.753 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:47.750+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.751962+0000) 2020-02-05T23:46:47.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:47.750+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.751911+0000) 2020-02-05T23:46:47.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:47.751+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.752977+0000) 2020-02-05T23:46:47.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:47.753+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.754572+0000) 2020-02-05T23:46:47.756 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:47.753+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.754581+0000) 2020-02-05T23:46:47.756 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:47.753+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.754786+0000) 2020-02-05T23:46:47.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:47.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.761407+0000) 2020-02-05T23:46:47.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:47.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:47.761716+0000) 2020-02-05T23:46:47.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:47.764+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:49.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:49.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:49.200598+0000) 2020-02-05T23:46:49.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:49.585+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:49.586295+0000) 2020-02-05T23:46:50.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:50.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:50.433035+0000) 2020-02-05T23:46:50.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:50.989+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:50.990588+0000) 2020-02-05T23:46:52.773 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:52.770+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.772048+0000) 2020-02-05T23:46:52.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:52.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.772941+0000) 2020-02-05T23:46:52.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:52.772+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.773949+0000) 2020-02-05T23:46:52.776 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:52.773+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.774574+0000) 2020-02-05T23:46:52.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:52.773+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.774816+0000) 2020-02-05T23:46:52.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:52.774+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.775682+0000) 2020-02-05T23:46:52.777 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:52.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.776179+0000) 2020-02-05T23:46:52.777 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:52.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.776361+0000) 2020-02-05T23:46:52.778 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:52.775+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.776973+0000) 2020-02-05T23:46:52.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:52.776+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.778016+0000) 2020-02-05T23:46:52.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:52.777+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.778256+0000) 2020-02-05T23:46:52.780 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:52.776+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.778385+0000) 2020-02-05T23:46:52.780 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:52.776+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.778236+0000) 2020-02-05T23:46:52.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:52.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.779682+0000) 2020-02-05T23:46:52.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:52.789+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:52.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:52.798+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:52.799532+0000) 2020-02-05T23:46:53.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:53.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:53.086143+0000) 2020-02-05T23:46:53.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:53.943+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:53.944156+0000) 2020-02-05T23:46:53.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:53.962+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:53.962998+0000) 2020-02-05T23:46:53.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:53.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:53.963041+0000) 2020-02-05T23:46:56.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:56.061+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:56.062746+0000) 2020-02-05T23:46:56.746 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:46:56.755 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:46:56.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:56.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:56.943201+0000) 2020-02-05T23:46:57.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:57.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.202323+0000) 2020-02-05T23:46:57.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:57.201+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.202305+0000) 2020-02-05T23:46:57.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:57.201+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.202526+0000) 2020-02-05T23:46:57.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:57.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.796200+0000) 2020-02-05T23:46:57.798 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:46:57.795+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.796553+0000) 2020-02-05T23:46:57.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:46:57.795+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.796912+0000) 2020-02-05T23:46:57.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:57.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.797775+0000) 2020-02-05T23:46:57.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:46:57.796+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.797935+0000) 2020-02-05T23:46:57.799 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:46:57.796+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.798368+0000) 2020-02-05T23:46:57.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:46:57.797+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.798578+0000) 2020-02-05T23:46:57.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:46:57.797+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.798869+0000) 2020-02-05T23:46:57.802 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:46:57.797+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.799216+0000) 2020-02-05T23:46:57.802 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:46:57.798+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.799660+0000) 2020-02-05T23:46:57.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:57.798+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.800168+0000) 2020-02-05T23:46:57.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:46:57.798+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.800378+0000) 2020-02-05T23:46:57.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:46:57.800+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.801552+0000) 2020-02-05T23:46:57.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:57.805+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:57.806992+0000) 2020-02-05T23:46:57.812 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:46:57.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:46:59.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:46:59.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:59.200944+0000) 2020-02-05T23:46:59.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:46:59.585+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:46:59.586518+0000) 2020-02-05T23:47:00.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:00.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:00.433287+0000) 2020-02-05T23:47:01.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:00.998+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:00.998780+0000) 2020-02-05T23:47:02.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:02.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.799750+0000) 2020-02-05T23:47:02.820 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:02.817+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.818652+0000) 2020-02-05T23:47:02.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:02.818+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.819382+0000) 2020-02-05T23:47:02.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:02.818+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.819896+0000) 2020-02-05T23:47:02.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:02.819+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.820383+0000) 2020-02-05T23:47:02.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:02.819+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.820485+0000) 2020-02-05T23:47:02.822 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:02.820+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.821152+0000) 2020-02-05T23:47:02.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:02.819+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.821176+0000) 2020-02-05T23:47:02.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:02.820+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.821816+0000) 2020-02-05T23:47:02.823 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:02.820+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.822186+0000) 2020-02-05T23:47:02.823 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:02.821+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.822625+0000) 2020-02-05T23:47:02.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:02.821+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.822546+0000) 2020-02-05T23:47:02.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:02.821+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.823357+0000) 2020-02-05T23:47:02.825 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:02.822+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.823533+0000) 2020-02-05T23:47:02.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:02.828+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:02.829614+0000) 2020-02-05T23:47:02.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:02.833+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:03.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:03.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:03.086347+0000) 2020-02-05T23:47:03.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:03.943+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:03.944392+0000) 2020-02-05T23:47:03.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:03.962+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:03.963268+0000) 2020-02-05T23:47:03.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:03.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:03.963296+0000) 2020-02-05T23:47:06.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:06.062+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:06.062984+0000) 2020-02-05T23:47:06.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:06.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:06.943444+0000) 2020-02-05T23:47:07.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:07.201+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.202562+0000) 2020-02-05T23:47:07.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:07.201+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.202563+0000) 2020-02-05T23:47:07.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:07.201+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.202786+0000) 2020-02-05T23:47:07.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:07.840+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.841432+0000) 2020-02-05T23:47:07.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:07.840+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.841590+0000) 2020-02-05T23:47:07.843 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:07.841+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.842516+0000) 2020-02-05T23:47:07.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:07.841+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.843001+0000) 2020-02-05T23:47:07.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:07.842+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.843373+0000) 2020-02-05T23:47:07.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:07.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.843500+0000) 2020-02-05T23:47:07.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:07.843+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.844115+0000) 2020-02-05T23:47:07.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:07.843+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.844575+0000) 2020-02-05T23:47:07.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:07.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.844605+0000) 2020-02-05T23:47:07.846 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:07.843+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.845015+0000) 2020-02-05T23:47:07.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:07.844+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.846326+0000) 2020-02-05T23:47:07.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:07.845+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.846651+0000) 2020-02-05T23:47:07.848 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:07.845+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.847044+0000) 2020-02-05T23:47:07.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:07.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:07.851552+0000) 2020-02-05T23:47:07.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:07.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154741 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:09.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:09.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:09.201207+0000) 2020-02-05T23:47:09.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:09.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:09.586744+0000) 2020-02-05T23:47:10.434 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:10.432+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:10.433544+0000) 2020-02-05T23:47:11.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:11.006+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:11.007049+0000) 2020-02-05T23:47:12.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:12.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.799976+0000) 2020-02-05T23:47:12.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:12.864+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.865981+0000) 2020-02-05T23:47:12.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:12.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.866851+0000) 2020-02-05T23:47:12.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:12.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.866908+0000) 2020-02-05T23:47:12.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:12.866+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.867460+0000) 2020-02-05T23:47:12.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:12.866+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.867695+0000) 2020-02-05T23:47:12.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:12.866+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.867939+0000) 2020-02-05T23:47:12.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:12.867+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.868475+0000) 2020-02-05T23:47:12.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:12.867+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.868324+0000) 2020-02-05T23:47:12.870 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:12.867+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.869057+0000) 2020-02-05T23:47:12.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:12.867+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.869373+0000) 2020-02-05T23:47:12.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:12.869+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.870468+0000) 2020-02-05T23:47:12.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:12.869+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.870704+0000) 2020-02-05T23:47:12.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:12.869+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.870957+0000) 2020-02-05T23:47:12.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:12.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:12.875437+0000) 2020-02-05T23:47:12.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:12.880+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:13.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:13.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:13.086568+0000) 2020-02-05T23:47:13.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:13.944+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:13.944625+0000) 2020-02-05T23:47:13.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:13.963+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:13.963532+0000) 2020-02-05T23:47:13.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:13.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:13.963542+0000) 2020-02-05T23:47:16.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:16.062+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:16.063248+0000) 2020-02-05T23:47:16.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:16.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:16.943678+0000) 2020-02-05T23:47:17.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:17.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.202821+0000) 2020-02-05T23:47:17.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:17.202+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.202822+0000) 2020-02-05T23:47:17.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:17.202+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.203022+0000) 2020-02-05T23:47:17.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:17.885+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.886340+0000) 2020-02-05T23:47:17.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:17.885+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.886830+0000) 2020-02-05T23:47:17.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:17.886+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.887965+0000) 2020-02-05T23:47:17.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:17.888+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.889497+0000) 2020-02-05T23:47:17.891 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:17.887+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.889285+0000) 2020-02-05T23:47:17.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:17.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.890166+0000) 2020-02-05T23:47:17.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:17.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.890395+0000) 2020-02-05T23:47:17.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:17.889+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.890984+0000) 2020-02-05T23:47:17.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:17.890+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.891538+0000) 2020-02-05T23:47:17.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:17.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.892015+0000) 2020-02-05T23:47:17.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:17.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.892395+0000) 2020-02-05T23:47:17.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:17.891+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.892662+0000) 2020-02-05T23:47:17.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:17.891+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.892919+0000) 2020-02-05T23:47:17.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:17.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:17.899207+0000) 2020-02-05T23:47:17.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:17.902+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154823 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:19.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:19.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:19.201499+0000) 2020-02-05T23:47:19.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:19.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:19.587025+0000) 2020-02-05T23:47:20.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:20.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:20.433773+0000) 2020-02-05T23:47:21.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:21.018+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:21.018693+0000) 2020-02-05T23:47:22.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:22.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.800205+0000) 2020-02-05T23:47:22.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:22.909+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.909803+0000) 2020-02-05T23:47:22.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:22.910+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.910557+0000) 2020-02-05T23:47:22.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:22.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.910936+0000) 2020-02-05T23:47:22.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:22.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.911263+0000) 2020-02-05T23:47:22.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:22.911+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.911606+0000) 2020-02-05T23:47:22.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:22.911+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.912559+0000) 2020-02-05T23:47:22.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:22.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.912936+0000) 2020-02-05T23:47:22.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:22.912+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.913764+0000) 2020-02-05T23:47:22.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:22.913+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.914630+0000) 2020-02-05T23:47:22.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:22.913+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.914802+0000) 2020-02-05T23:47:22.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:22.913+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.915186+0000) 2020-02-05T23:47:22.918 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:22.917+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.917070+0000) 2020-02-05T23:47:22.918 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:22.915+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.917409+0000) 2020-02-05T23:47:22.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:22.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:22.919645+0000) 2020-02-05T23:47:22.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:22.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:23.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:23.086+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:23.086786+0000) 2020-02-05T23:47:23.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:23.944+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:23.944860+0000) 2020-02-05T23:47:23.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:23.963+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:23.963801+0000) 2020-02-05T23:47:23.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:23.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:23.963755+0000) 2020-02-05T23:47:26.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:26.063+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:26.063498+0000) 2020-02-05T23:47:26.883 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:47:26.893 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:47:26.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:26.942+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:26.943894+0000) 2020-02-05T23:47:27.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:27.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.203078+0000) 2020-02-05T23:47:27.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:27.202+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.203249+0000) 2020-02-05T23:47:27.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:27.202+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.203154+0000) 2020-02-05T23:47:27.936 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:27.933+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.934806+0000) 2020-02-05T23:47:27.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:27.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.935587+0000) 2020-02-05T23:47:27.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:27.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.935822+0000) 2020-02-05T23:47:27.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:27.935+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.936398+0000) 2020-02-05T23:47:27.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:27.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.936367+0000) 2020-02-05T23:47:27.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:27.935+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.936823+0000) 2020-02-05T23:47:27.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:27.938+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.938793+0000) 2020-02-05T23:47:27.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:27.937+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.938404+0000) 2020-02-05T23:47:27.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:27.938+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.939389+0000) 2020-02-05T23:47:27.941 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:27.938+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.939193+0000) 2020-02-05T23:47:27.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:27.938+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.939145+0000) 2020-02-05T23:47:27.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:27.939+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.939912+0000) 2020-02-05T23:47:27.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:27.939+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.940055+0000) 2020-02-05T23:47:27.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:27.941+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:27.942473+0000) 2020-02-05T23:47:27.952 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:27.950+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:29.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:29.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:29.201792+0000) 2020-02-05T23:47:29.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:29.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:29.587337+0000) 2020-02-05T23:47:30.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:30.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:30.434008+0000) 2020-02-05T23:47:31.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:31.025+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:31.026730+0000) 2020-02-05T23:47:32.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:32.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.800432+0000) 2020-02-05T23:47:32.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:32.955+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.956611+0000) 2020-02-05T23:47:32.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:32.957+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.957990+0000) 2020-02-05T23:47:32.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:32.957+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.958357+0000) 2020-02-05T23:47:32.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:32.958+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.958911+0000) 2020-02-05T23:47:32.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:32.958+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.959225+0000) 2020-02-05T23:47:32.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:32.959+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.959690+0000) 2020-02-05T23:47:32.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:32.959+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.960141+0000) 2020-02-05T23:47:32.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:32.960+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.961104+0000) 2020-02-05T23:47:32.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:32.960+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.961408+0000) 2020-02-05T23:47:32.963 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:32.961+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.961731+0000) 2020-02-05T23:47:32.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:32.961+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.962488+0000) 2020-02-05T23:47:32.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:32.961+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.962682+0000) 2020-02-05T23:47:32.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:32.962+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.963084+0000) 2020-02-05T23:47:32.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:32.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:32.967534+0000) 2020-02-05T23:47:32.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:32.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:33.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:33.086+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:33.086995+0000) 2020-02-05T23:47:33.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:33.944+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:33.945123+0000) 2020-02-05T23:47:33.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:33.962+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:33.964042+0000) 2020-02-05T23:47:33.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:33.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:33.964021+0000) 2020-02-05T23:47:36.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:36.063+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:36.063749+0000) 2020-02-05T23:47:36.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:36.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:36.944121+0000) 2020-02-05T23:47:37.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:37.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.203313+0000) 2020-02-05T23:47:37.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:37.202+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.203403+0000) 2020-02-05T23:47:37.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:37.203+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.203497+0000) 2020-02-05T23:47:37.982 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:37.980+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.981377+0000) 2020-02-05T23:47:37.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:37.981+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.981621+0000) 2020-02-05T23:47:37.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:37.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.982127+0000) 2020-02-05T23:47:37.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:37.981+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.982466+0000) 2020-02-05T23:47:37.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:37.981+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.983052+0000) 2020-02-05T23:47:37.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:37.983+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.983651+0000) 2020-02-05T23:47:37.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:37.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.984578+0000) 2020-02-05T23:47:37.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:37.984+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.984983+0000) 2020-02-05T23:47:37.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:37.984+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.985328+0000) 2020-02-05T23:47:37.986 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:37.984+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.985233+0000) 2020-02-05T23:47:37.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:37.985+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.986004+0000) 2020-02-05T23:47:37.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:37.985+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.986701+0000) 2020-02-05T23:47:37.988 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:37.986+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.986563+0000) 2020-02-05T23:47:37.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:37.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:37.990311+0000) 2020-02-05T23:47:38.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:37.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 154987 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:39.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:39.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:39.202015+0000) 2020-02-05T23:47:39.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:39.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:39.587574+0000) 2020-02-05T23:47:40.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:40.433+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:40.434255+0000) 2020-02-05T23:47:41.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:41.033+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:41.034321+0000) 2020-02-05T23:47:42.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:42.799+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:42.800711+0000) 2020-02-05T23:47:43.006 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:43.004+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.005387+0000) 2020-02-05T23:47:43.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:43.005+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.006278+0000) 2020-02-05T23:47:43.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:43.005+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.006639+0000) 2020-02-05T23:47:43.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:43.007+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.007753+0000) 2020-02-05T23:47:43.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:43.008+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.008714+0000) 2020-02-05T23:47:43.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:43.008+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.009417+0000) 2020-02-05T23:47:43.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:43.008+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.009649+0000) 2020-02-05T23:47:43.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:43.008+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.009235+0000) 2020-02-05T23:47:43.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:43.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.010018+0000) 2020-02-05T23:47:43.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:43.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.010462+0000) 2020-02-05T23:47:43.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:43.010+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.010893+0000) 2020-02-05T23:47:43.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:43.010+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.010849+0000) 2020-02-05T23:47:43.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:43.011+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.012057+0000) 2020-02-05T23:47:43.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:43.016+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.017655+0000) 2020-02-05T23:47:43.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:43.021+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:43.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:43.086+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.087187+0000) 2020-02-05T23:47:43.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:43.944+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.945335+0000) 2020-02-05T23:47:43.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:43.963+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.964328+0000) 2020-02-05T23:47:43.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:43.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:43.964310+0000) 2020-02-05T23:47:46.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:46.063+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:46.064006+0000) 2020-02-05T23:47:46.945 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:46.943+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:46.944297+0000) 2020-02-05T23:47:47.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:47.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:47.203555+0000) 2020-02-05T23:47:47.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:47.203+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:47.203691+0000) 2020-02-05T23:47:47.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:47.203+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:47.203748+0000) 2020-02-05T23:47:48.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:48.025+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.026725+0000) 2020-02-05T23:47:48.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:48.026+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.027593+0000) 2020-02-05T23:47:48.029 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:48.026+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.027285+0000) 2020-02-05T23:47:48.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:48.027+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.028937+0000) 2020-02-05T23:47:48.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:48.029+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.030908+0000) 2020-02-05T23:47:48.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:48.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.031389+0000) 2020-02-05T23:47:48.034 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:48.030+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.031724+0000) 2020-02-05T23:47:48.036 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:48.031+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.032208+0000) 2020-02-05T23:47:48.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:48.032+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.032812+0000) 2020-02-05T23:47:48.036 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:48.032+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.032695+0000) 2020-02-05T23:47:48.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:48.032+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.033268+0000) 2020-02-05T23:47:48.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:48.032+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.033268+0000) 2020-02-05T23:47:48.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:48.031+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.032621+0000) 2020-02-05T23:47:48.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:48.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:48.036100+0000) 2020-02-05T23:47:48.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:48.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:49.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:49.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:49.202271+0000) 2020-02-05T23:47:49.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:49.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:49.587789+0000) 2020-02-05T23:47:50.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:50.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:50.434501+0000) 2020-02-05T23:47:51.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:51.041+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:51.042568+0000) 2020-02-05T23:47:52.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:52.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:52.800928+0000) 2020-02-05T23:47:53.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:53.050+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.051610+0000) 2020-02-05T23:47:53.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:53.050+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.051913+0000) 2020-02-05T23:47:53.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:53.051+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.051786+0000) 2020-02-05T23:47:53.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:53.051+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.052952+0000) 2020-02-05T23:47:53.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:53.053+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.053574+0000) 2020-02-05T23:47:53.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:53.053+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.053943+0000) 2020-02-05T23:47:53.056 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:53.053+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.054408+0000) 2020-02-05T23:47:53.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:53.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.054969+0000) 2020-02-05T23:47:53.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:53.053+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.054372+0000) 2020-02-05T23:47:53.057 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:53.054+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.055488+0000) 2020-02-05T23:47:53.057 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:53.054+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.055983+0000) 2020-02-05T23:47:53.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:53.055+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.055793+0000) 2020-02-05T23:47:53.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:53.055+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.056038+0000) 2020-02-05T23:47:53.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:53.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.060276+0000) 2020-02-05T23:47:53.067 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:53.066+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155110 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:53.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:53.086+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.087406+0000) 2020-02-05T23:47:53.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:53.944+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.945546+0000) 2020-02-05T23:47:53.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:53.963+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.964482+0000) 2020-02-05T23:47:53.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:53.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:53.964524+0000) 2020-02-05T23:47:56.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:56.064+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:56.064339+0000) 2020-02-05T23:47:56.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:56.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:56.944584+0000) 2020-02-05T23:47:57.081 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:47:57.087 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:47:57.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:57.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:57.203750+0000) 2020-02-05T23:47:57.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:57.203+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:57.204003+0000) 2020-02-05T23:47:57.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:57.203+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:57.203914+0000) 2020-02-05T23:47:58.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:47:58.070+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.071716+0000) 2020-02-05T23:47:58.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:58.073+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.073905+0000) 2020-02-05T23:47:58.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:47:58.073+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.074222+0000) 2020-02-05T23:47:58.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:58.073+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.074861+0000) 2020-02-05T23:47:58.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:47:58.074+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.075337+0000) 2020-02-05T23:47:58.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:58.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.075582+0000) 2020-02-05T23:47:58.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:47:58.074+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.075873+0000) 2020-02-05T23:47:58.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:47:58.075+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.075653+0000) 2020-02-05T23:47:58.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:47:58.075+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.076075+0000) 2020-02-05T23:47:58.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:47:58.076+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.076699+0000) 2020-02-05T23:47:58.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:47:58.076+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.077286+0000) 2020-02-05T23:47:58.079 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:47:58.075+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.076359+0000) 2020-02-05T23:47:58.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:47:58.076+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.077372+0000) 2020-02-05T23:47:58.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:58.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:58.082345+0000) 2020-02-05T23:47:58.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:47:58.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:47:59.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:47:59.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:59.202535+0000) 2020-02-05T23:47:59.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:47:59.586+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:47:59.588008+0000) 2020-02-05T23:48:00.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:00.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:00.434742+0000) 2020-02-05T23:48:01.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:01.049+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:01.050579+0000) 2020-02-05T23:48:02.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:02.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:02.801167+0000) 2020-02-05T23:48:03.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:03.087+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.087631+0000) 2020-02-05T23:48:03.098 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:03.092+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.094229+0000) 2020-02-05T23:48:03.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:03.095+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.096488+0000) 2020-02-05T23:48:03.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:03.093+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.094562+0000) 2020-02-05T23:48:03.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:03.093+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.094853+0000) 2020-02-05T23:48:03.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:03.094+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.095708+0000) 2020-02-05T23:48:03.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:03.095+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.096572+0000) 2020-02-05T23:48:03.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:03.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.098588+0000) 2020-02-05T23:48:03.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:03.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.099528+0000) 2020-02-05T23:48:03.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:03.098+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.099986+0000) 2020-02-05T23:48:03.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:03.098+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.099624+0000) 2020-02-05T23:48:03.102 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:03.098+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.100326+0000) 2020-02-05T23:48:03.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:03.099+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.100586+0000) 2020-02-05T23:48:03.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:03.099+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.100655+0000) 2020-02-05T23:48:03.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:03.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.103503+0000) 2020-02-05T23:48:03.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:03.110+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:03.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:03.945+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.945791+0000) 2020-02-05T23:48:03.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:03.964+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.964714+0000) 2020-02-05T23:48:03.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:03.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:03.964778+0000) 2020-02-05T23:48:06.065 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:06.063+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:06.064591+0000) 2020-02-05T23:48:06.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:06.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:06.944857+0000) 2020-02-05T23:48:07.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:07.202+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:07.203993+0000) 2020-02-05T23:48:07.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:07.202+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:07.204178+0000) 2020-02-05T23:48:07.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:07.202+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:07.204191+0000) 2020-02-05T23:48:08.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:08.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.118450+0000) 2020-02-05T23:48:08.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:08.117+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.119077+0000) 2020-02-05T23:48:08.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:08.118+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.119199+0000) 2020-02-05T23:48:08.121 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:08.118+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.120129+0000) 2020-02-05T23:48:08.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:08.118+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.119737+0000) 2020-02-05T23:48:08.122 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:08.119+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.120406+0000) 2020-02-05T23:48:08.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:08.119+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.120856+0000) 2020-02-05T23:48:08.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:08.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.121113+0000) 2020-02-05T23:48:08.123 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:08.120+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.121600+0000) 2020-02-05T23:48:08.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:08.120+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.122193+0000) 2020-02-05T23:48:08.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:08.121+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.122493+0000) 2020-02-05T23:48:08.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:08.121+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.122833+0000) 2020-02-05T23:48:08.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:08.122+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.123867+0000) 2020-02-05T23:48:08.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:08.126+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:08.127769+0000) 2020-02-05T23:48:08.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:08.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155235 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:09.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:09.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:09.202806+0000) 2020-02-05T23:48:09.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:09.587+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:09.588260+0000) 2020-02-05T23:48:10.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:10.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:10.434987+0000) 2020-02-05T23:48:11.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:11.057+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:11.057744+0000) 2020-02-05T23:48:12.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:12.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:12.801435+0000) 2020-02-05T23:48:13.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:13.087+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.087861+0000) 2020-02-05T23:48:13.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:13.138+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.140091+0000) 2020-02-05T23:48:13.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:13.139+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.141207+0000) 2020-02-05T23:48:13.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:13.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.141858+0000) 2020-02-05T23:48:13.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:13.141+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.142921+0000) 2020-02-05T23:48:13.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:13.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.143020+0000) 2020-02-05T23:48:13.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:13.142+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.143449+0000) 2020-02-05T23:48:13.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:13.142+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.143878+0000) 2020-02-05T23:48:13.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:13.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.144396+0000) 2020-02-05T23:48:13.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:13.143+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.144491+0000) 2020-02-05T23:48:13.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:13.143+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.145126+0000) 2020-02-05T23:48:13.147 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:13.144+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.145255+0000) 2020-02-05T23:48:13.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:13.144+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.145719+0000) 2020-02-05T23:48:13.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:13.145+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.146711+0000) 2020-02-05T23:48:13.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:13.154+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.155833+0000) 2020-02-05T23:48:13.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:13.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155276 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:13.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:13.945+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.946040+0000) 2020-02-05T23:48:13.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:13.963+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.964937+0000) 2020-02-05T23:48:13.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:13.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:13.965000+0000) 2020-02-05T23:48:16.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:16.063+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:16.064839+0000) 2020-02-05T23:48:16.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:16.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:16.945127+0000) 2020-02-05T23:48:17.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:17.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:17.204243+0000) 2020-02-05T23:48:17.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:17.203+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:17.204420+0000) 2020-02-05T23:48:17.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:17.203+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:17.204415+0000) 2020-02-05T23:48:18.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:18.165+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.165556+0000) 2020-02-05T23:48:18.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:18.166+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.166145+0000) 2020-02-05T23:48:18.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:18.165+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.166833+0000) 2020-02-05T23:48:18.168 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:18.165+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.167130+0000) 2020-02-05T23:48:18.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:18.166+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.168312+0000) 2020-02-05T23:48:18.169 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:18.168+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.168394+0000) 2020-02-05T23:48:18.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:18.167+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.169306+0000) 2020-02-05T23:48:18.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:18.169+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.169949+0000) 2020-02-05T23:48:18.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:18.169+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.170674+0000) 2020-02-05T23:48:18.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:18.171+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.171150+0000) 2020-02-05T23:48:18.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:18.170+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.170972+0000) 2020-02-05T23:48:18.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:18.169+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.171207+0000) 2020-02-05T23:48:18.173 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:18.171+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.171665+0000) 2020-02-05T23:48:18.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:18.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:18.177488+0000) 2020-02-05T23:48:18.183 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:18.181+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:19.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:19.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:19.203084+0000) 2020-02-05T23:48:19.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:19.588+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:19.588495+0000) 2020-02-05T23:48:20.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:20.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:20.435230+0000) 2020-02-05T23:48:21.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:21.065+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:21.066059+0000) 2020-02-05T23:48:22.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:22.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:22.801676+0000) 2020-02-05T23:48:23.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:23.087+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.088111+0000) 2020-02-05T23:48:23.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:23.185+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.186915+0000) 2020-02-05T23:48:23.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:23.187+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.188880+0000) 2020-02-05T23:48:23.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:23.187+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.189195+0000) 2020-02-05T23:48:23.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:23.188+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.190032+0000) 2020-02-05T23:48:23.193 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:23.188+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.189573+0000) 2020-02-05T23:48:23.194 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:23.188+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.189991+0000) 2020-02-05T23:48:23.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:23.189+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.190314+0000) 2020-02-05T23:48:23.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:23.190+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.191511+0000) 2020-02-05T23:48:23.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:23.189+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.190612+0000) 2020-02-05T23:48:23.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:23.189+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.190737+0000) 2020-02-05T23:48:23.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:23.190+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.191820+0000) 2020-02-05T23:48:23.195 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:23.190+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.191938+0000) 2020-02-05T23:48:23.196 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:23.191+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.192643+0000) 2020-02-05T23:48:23.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:23.198+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.199114+0000) 2020-02-05T23:48:23.210 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:23.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:23.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:23.945+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.946279+0000) 2020-02-05T23:48:23.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:23.964+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.965210+0000) 2020-02-05T23:48:23.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:23.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:23.965217+0000) 2020-02-05T23:48:26.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:26.064+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:26.065178+0000) 2020-02-05T23:48:26.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:26.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:26.945357+0000) 2020-02-05T23:48:27.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:27.203+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:27.204564+0000) 2020-02-05T23:48:27.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:27.203+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:27.204691+0000) 2020-02-05T23:48:27.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:27.203+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:27.204692+0000) 2020-02-05T23:48:27.268 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:48:27.274 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:48:28.212 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:28.210+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.211308+0000) 2020-02-05T23:48:28.213 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:28.211+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.212071+0000) 2020-02-05T23:48:28.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:28.211+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.213004+0000) 2020-02-05T23:48:28.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:28.212+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.213411+0000) 2020-02-05T23:48:28.215 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:28.212+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.214050+0000) 2020-02-05T23:48:28.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:28.213+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.214965+0000) 2020-02-05T23:48:28.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:28.214+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.216048+0000) 2020-02-05T23:48:28.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:28.214+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.216068+0000) 2020-02-05T23:48:28.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:28.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.216149+0000) 2020-02-05T23:48:28.218 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:28.215+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.216763+0000) 2020-02-05T23:48:28.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:28.215+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.217179+0000) 2020-02-05T23:48:28.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:28.215+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.217024+0000) 2020-02-05T23:48:28.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:28.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155401 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:28.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:28.217+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.219043+0000) 2020-02-05T23:48:28.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:28.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:28.219307+0000) 2020-02-05T23:48:29.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:29.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:29.203356+0000) 2020-02-05T23:48:29.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:29.587+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:29.588741+0000) 2020-02-05T23:48:30.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:30.434+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:30.435498+0000) 2020-02-05T23:48:31.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:31.074+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:31.074891+0000) 2020-02-05T23:48:32.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:32.800+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:32.801957+0000) 2020-02-05T23:48:33.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:33.087+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.088356+0000) 2020-02-05T23:48:33.233 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:33.231+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.232543+0000) 2020-02-05T23:48:33.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:33.232+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.233195+0000) 2020-02-05T23:48:33.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:33.232+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.233781+0000) 2020-02-05T23:48:33.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:33.233+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.235094+0000) 2020-02-05T23:48:33.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:33.234+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.235635+0000) 2020-02-05T23:48:33.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:33.235+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.236605+0000) 2020-02-05T23:48:33.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:33.235+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.236788+0000) 2020-02-05T23:48:33.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:33.235+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.237164+0000) 2020-02-05T23:48:33.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:33.236+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.237410+0000) 2020-02-05T23:48:33.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:33.236+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.237567+0000) 2020-02-05T23:48:33.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:33.236+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.238084+0000) 2020-02-05T23:48:33.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:33.237+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.238827+0000) 2020-02-05T23:48:33.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:33.239+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.240950+0000) 2020-02-05T23:48:33.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:33.240+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.241801+0000) 2020-02-05T23:48:33.254 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:33.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155442 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:33.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:33.945+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.946479+0000) 2020-02-05T23:48:33.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:33.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.965376+0000) 2020-02-05T23:48:33.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:33.964+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:33.965413+0000) 2020-02-05T23:48:36.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:36.064+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:36.065461+0000) 2020-02-05T23:48:36.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:36.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:36.945588+0000) 2020-02-05T23:48:37.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:37.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:37.204855+0000) 2020-02-05T23:48:37.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:37.204+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:37.204992+0000) 2020-02-05T23:48:37.208 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:37.204+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:37.204987+0000) 2020-02-05T23:48:38.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:38.256+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.257119+0000) 2020-02-05T23:48:38.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:38.256+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.257493+0000) 2020-02-05T23:48:38.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:38.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.257991+0000) 2020-02-05T23:48:38.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:38.257+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.258976+0000) 2020-02-05T23:48:38.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:38.258+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.260032+0000) 2020-02-05T23:48:38.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:38.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.260361+0000) 2020-02-05T23:48:38.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:38.259+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.260570+0000) 2020-02-05T23:48:38.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:38.259+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.260908+0000) 2020-02-05T23:48:38.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:38.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.260952+0000) 2020-02-05T23:48:38.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:38.260+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.261752+0000) 2020-02-05T23:48:38.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:38.260+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.261152+0000) 2020-02-05T23:48:38.263 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:38.260+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.261579+0000) 2020-02-05T23:48:38.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:38.259+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.261337+0000) 2020-02-05T23:48:38.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:38.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:38.264834+0000) 2020-02-05T23:48:38.274 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:38.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155485 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:39.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:39.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:39.203601+0000) 2020-02-05T23:48:39.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:39.588+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:39.588997+0000) 2020-02-05T23:48:40.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:40.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:40.435746+0000) 2020-02-05T23:48:41.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:41.082+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:41.083441+0000) 2020-02-05T23:48:42.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:42.801+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:42.802193+0000) 2020-02-05T23:48:43.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:43.087+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.088578+0000) 2020-02-05T23:48:43.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:43.277+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.278353+0000) 2020-02-05T23:48:43.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:43.278+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.279619+0000) 2020-02-05T23:48:43.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:43.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.280905+0000) 2020-02-05T23:48:43.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:43.280+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.281364+0000) 2020-02-05T23:48:43.283 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:43.280+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.281653+0000) 2020-02-05T23:48:43.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:43.281+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.282598+0000) 2020-02-05T23:48:43.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:43.281+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.282762+0000) 2020-02-05T23:48:43.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:43.281+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.282867+0000) 2020-02-05T23:48:43.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:43.282+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.283464+0000) 2020-02-05T23:48:43.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:43.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.284268+0000) 2020-02-05T23:48:43.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:43.282+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.284039+0000) 2020-02-05T23:48:43.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:43.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.285176+0000) 2020-02-05T23:48:43.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:43.284+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.285889+0000) 2020-02-05T23:48:43.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:43.289+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.290886+0000) 2020-02-05T23:48:43.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:43.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:43.948 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:43.946+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.946707+0000) 2020-02-05T23:48:43.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:43.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.965591+0000) 2020-02-05T23:48:43.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:43.964+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:43.965619+0000) 2020-02-05T23:48:46.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:46.065+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:46.065726+0000) 2020-02-05T23:48:46.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:46.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:46.945812+0000) 2020-02-05T23:48:47.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:47.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:47.205153+0000) 2020-02-05T23:48:47.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:47.204+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:47.205274+0000) 2020-02-05T23:48:47.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:47.204+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:47.205252+0000) 2020-02-05T23:48:48.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:48.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.302749+0000) 2020-02-05T23:48:48.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:48.303+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.304111+0000) 2020-02-05T23:48:48.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:48.303+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.304835+0000) 2020-02-05T23:48:48.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:48.303+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.305364+0000) 2020-02-05T23:48:48.308 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:48.305+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.306841+0000) 2020-02-05T23:48:48.308 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:48.306+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.307509+0000) 2020-02-05T23:48:48.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:48.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.307442+0000) 2020-02-05T23:48:48.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:48.306+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.308123+0000) 2020-02-05T23:48:48.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:48.307+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.308744+0000) 2020-02-05T23:48:48.310 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:48.307+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.308960+0000) 2020-02-05T23:48:48.311 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:48.308+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.309549+0000) 2020-02-05T23:48:48.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:48.309+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.310581+0000) 2020-02-05T23:48:48.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:48.309+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.310810+0000) 2020-02-05T23:48:48.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:48.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:48.310871+0000) 2020-02-05T23:48:48.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:48.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155567 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:49.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:49.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:49.203841+0000) 2020-02-05T23:48:49.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:49.588+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:49.589257+0000) 2020-02-05T23:48:50.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:50.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:50.435987+0000) 2020-02-05T23:48:51.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:51.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:51.092124+0000) 2020-02-05T23:48:52.803 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:52.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:52.802483+0000) 2020-02-05T23:48:53.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:53.088+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.088828+0000) 2020-02-05T23:48:53.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:53.326+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.326642+0000) 2020-02-05T23:48:53.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:53.326+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.327350+0000) 2020-02-05T23:48:53.329 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:53.326+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.327662+0000) 2020-02-05T23:48:53.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:53.327+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.328369+0000) 2020-02-05T23:48:53.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:53.328+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.329544+0000) 2020-02-05T23:48:53.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:53.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.330100+0000) 2020-02-05T23:48:53.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:53.330+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.330776+0000) 2020-02-05T23:48:53.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:53.330+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.330941+0000) 2020-02-05T23:48:53.333 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:53.331+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.332182+0000) 2020-02-05T23:48:53.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:53.332+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.332558+0000) 2020-02-05T23:48:53.334 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:53.332+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.333289+0000) 2020-02-05T23:48:53.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:53.333+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.333489+0000) 2020-02-05T23:48:53.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:53.333+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.333821+0000) 2020-02-05T23:48:53.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:53.335+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.337053+0000) 2020-02-05T23:48:53.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:53.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155608 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:53.948 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:53.946+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.946909+0000) 2020-02-05T23:48:53.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:53.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.965801+0000) 2020-02-05T23:48:53.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:53.965+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:53.965871+0000) 2020-02-05T23:48:56.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:56.065+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:56.065970+0000) 2020-02-05T23:48:56.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:56.944+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:56.946042+0000) 2020-02-05T23:48:57.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:57.204+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:57.205402+0000) 2020-02-05T23:48:57.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:57.205+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:57.205552+0000) 2020-02-05T23:48:57.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:57.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:57.205528+0000) 2020-02-05T23:48:57.449 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:48:57.455 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:48:58.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:48:58.349+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.350480+0000) 2020-02-05T23:48:58.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:58.352+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.353495+0000) 2020-02-05T23:48:58.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:48:58.352+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.353230+0000) 2020-02-05T23:48:58.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:48:58.353+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.354113+0000) 2020-02-05T23:48:58.356 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:48:58.353+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.354662+0000) 2020-02-05T23:48:58.356 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:48:58.354+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.355194+0000) 2020-02-05T23:48:58.356 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:48:58.355+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.355584+0000) 2020-02-05T23:48:58.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:48:58.355+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.356082+0000) 2020-02-05T23:48:58.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:48:58.355+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.356448+0000) 2020-02-05T23:48:58.358 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:48:58.356+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.357079+0000) 2020-02-05T23:48:58.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:48:58.355+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.356865+0000) 2020-02-05T23:48:58.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:58.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.357350+0000) 2020-02-05T23:48:58.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:58.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.358996+0000) 2020-02-05T23:48:58.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:58.358+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:58.359233+0000) 2020-02-05T23:48:58.370 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:48:58.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:48:59.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:48:59.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:59.204062+0000) 2020-02-05T23:48:59.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:48:59.589+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:48:59.589498+0000) 2020-02-05T23:49:00.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:00.435+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:00.436236+0000) 2020-02-05T23:49:01.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:01.101+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:01.101584+0000) 2020-02-05T23:49:02.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:02.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:02.802715+0000) 2020-02-05T23:49:03.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:03.088+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.089059+0000) 2020-02-05T23:49:03.375 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:03.373+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.374420+0000) 2020-02-05T23:49:03.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:03.374+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.375038+0000) 2020-02-05T23:49:03.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:03.376+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.376617+0000) 2020-02-05T23:49:03.378 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:03.376+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.377124+0000) 2020-02-05T23:49:03.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:03.377+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.377950+0000) 2020-02-05T23:49:03.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:03.378+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.378806+0000) 2020-02-05T23:49:03.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:03.379+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.379836+0000) 2020-02-05T23:49:03.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:03.378+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.379930+0000) 2020-02-05T23:49:03.381 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:03.379+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.379506+0000) 2020-02-05T23:49:03.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:03.379+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.380886+0000) 2020-02-05T23:49:03.382 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:03.381+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.381488+0000) 2020-02-05T23:49:03.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:03.380+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.381420+0000) 2020-02-05T23:49:03.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:03.381+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.382354+0000) 2020-02-05T23:49:03.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:03.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.386519+0000) 2020-02-05T23:49:03.394 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:03.392+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:03.948 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:03.946+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.947119+0000) 2020-02-05T23:49:03.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:03.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.966016+0000) 2020-02-05T23:49:03.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:03.965+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:03.966119+0000) 2020-02-05T23:49:06.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:06.065+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:06.066220+0000) 2020-02-05T23:49:06.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:06.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:06.946327+0000) 2020-02-05T23:49:07.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:07.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:07.205695+0000) 2020-02-05T23:49:07.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:07.205+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:07.205790+0000) 2020-02-05T23:49:07.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:07.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:07.205876+0000) 2020-02-05T23:49:08.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:08.397+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.398193+0000) 2020-02-05T23:49:08.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:08.399+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.399600+0000) 2020-02-05T23:49:08.401 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:08.399+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.400579+0000) 2020-02-05T23:49:08.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:08.399+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.400253+0000) 2020-02-05T23:49:08.402 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:08.400+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.401130+0000) 2020-02-05T23:49:08.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:08.400+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.401646+0000) 2020-02-05T23:49:08.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:08.402+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.402407+0000) 2020-02-05T23:49:08.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:08.402+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.403044+0000) 2020-02-05T23:49:08.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:08.403+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.403859+0000) 2020-02-05T23:49:08.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:08.402+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.404028+0000) 2020-02-05T23:49:08.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:08.404+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.404541+0000) 2020-02-05T23:49:08.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:08.404+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.405380+0000) 2020-02-05T23:49:08.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:08.406+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.407674+0000) 2020-02-05T23:49:08.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:08.406+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:08.407961+0000) 2020-02-05T23:49:08.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:08.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:09.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:09.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:09.204310+0000) 2020-02-05T23:49:09.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:09.589+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:09.589765+0000) 2020-02-05T23:49:10.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:10.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:10.436490+0000) 2020-02-05T23:49:11.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:11.110+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:11.110382+0000) 2020-02-05T23:49:12.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:12.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:12.802962+0000) 2020-02-05T23:49:13.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:13.089+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.089327+0000) 2020-02-05T23:49:13.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:13.422+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.422378+0000) 2020-02-05T23:49:13.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:13.423+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.423913+0000) 2020-02-05T23:49:13.425 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:13.424+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.424442+0000) 2020-02-05T23:49:13.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:13.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.424900+0000) 2020-02-05T23:49:13.427 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:13.426+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.426267+0000) 2020-02-05T23:49:13.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:13.425+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.426240+0000) 2020-02-05T23:49:13.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:13.426+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.426937+0000) 2020-02-05T23:49:13.429 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:13.427+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.427783+0000) 2020-02-05T23:49:13.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:13.428+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.429145+0000) 2020-02-05T23:49:13.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:13.429+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.429152+0000) 2020-02-05T23:49:13.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:13.428+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.429374+0000) 2020-02-05T23:49:13.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:13.429+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.429559+0000) 2020-02-05T23:49:13.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:13.429+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.430137+0000) 2020-02-05T23:49:13.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:13.432+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.432181+0000) 2020-02-05T23:49:13.442 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:13.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:13.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:13.947+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.947333+0000) 2020-02-05T23:49:13.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:13.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.966287+0000) 2020-02-05T23:49:13.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:13.966+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:13.966351+0000) 2020-02-05T23:49:16.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:16.066+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:16.066502+0000) 2020-02-05T23:49:16.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:16.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:16.946557+0000) 2020-02-05T23:49:17.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:17.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:17.205957+0000) 2020-02-05T23:49:17.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:17.205+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:17.206138+0000) 2020-02-05T23:49:17.208 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:17.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:17.206163+0000) 2020-02-05T23:49:18.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:18.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.448576+0000) 2020-02-05T23:49:18.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:18.447+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.448182+0000) 2020-02-05T23:49:18.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:18.447+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.449029+0000) 2020-02-05T23:49:18.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:18.448+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.449653+0000) 2020-02-05T23:49:18.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:18.449+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.449716+0000) 2020-02-05T23:49:18.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:18.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.450054+0000) 2020-02-05T23:49:18.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:18.450+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.451537+0000) 2020-02-05T23:49:18.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:18.450+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.451224+0000) 2020-02-05T23:49:18.453 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:18.450+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.452006+0000) 2020-02-05T23:49:18.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:18.452+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.452817+0000) 2020-02-05T23:49:18.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:18.452+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.453393+0000) 2020-02-05T23:49:18.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:18.453+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.453708+0000) 2020-02-05T23:49:18.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:18.453+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.453923+0000) 2020-02-05T23:49:18.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:18.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:18.456090+0000) 2020-02-05T23:49:18.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:18.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:19.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:19.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:19.204535+0000) 2020-02-05T23:49:19.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:19.588+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:19.589992+0000) 2020-02-05T23:49:20.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:20.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:20.436722+0000) 2020-02-05T23:49:21.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:21.118+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:21.119131+0000) 2020-02-05T23:49:22.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:22.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:22.803137+0000) 2020-02-05T23:49:23.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:23.088+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.089555+0000) 2020-02-05T23:49:23.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:23.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.472233+0000) 2020-02-05T23:49:23.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:23.471+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.472776+0000) 2020-02-05T23:49:23.475 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:23.472+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.474007+0000) 2020-02-05T23:49:23.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:23.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.474604+0000) 2020-02-05T23:49:23.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:23.474+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.475305+0000) 2020-02-05T23:49:23.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:23.474+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.475368+0000) 2020-02-05T23:49:23.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:23.474+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.475947+0000) 2020-02-05T23:49:23.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:23.475+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.475685+0000) 2020-02-05T23:49:23.478 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:23.475+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.476465+0000) 2020-02-05T23:49:23.488 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:23.477+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.477543+0000) 2020-02-05T23:49:23.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:23.477+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.478046+0000) 2020-02-05T23:49:23.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:23.477+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.478271+0000) 2020-02-05T23:49:23.488 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:23.478+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.478670+0000) 2020-02-05T23:49:23.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:23.480+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.481563+0000) 2020-02-05T23:49:23.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:23.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:23.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:23.946+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.947502+0000) 2020-02-05T23:49:23.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:23.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.966494+0000) 2020-02-05T23:49:23.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:23.965+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:23.966567+0000) 2020-02-05T23:49:26.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:26.066+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:26.066733+0000) 2020-02-05T23:49:26.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:26.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:26.946799+0000) 2020-02-05T23:49:27.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:27.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:27.206218+0000) 2020-02-05T23:49:27.208 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:27.206+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:27.206407+0000) 2020-02-05T23:49:27.208 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:27.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:27.206389+0000) 2020-02-05T23:49:27.597 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:49:27.604 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:49:28.497 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:28.495+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.496217+0000) 2020-02-05T23:49:28.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:28.496+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.497099+0000) 2020-02-05T23:49:28.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:28.497+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.497752+0000) 2020-02-05T23:49:28.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:28.497+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.498348+0000) 2020-02-05T23:49:28.500 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:28.497+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.498642+0000) 2020-02-05T23:49:28.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:28.497+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.498881+0000) 2020-02-05T23:49:28.501 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:28.499+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.499587+0000) 2020-02-05T23:49:28.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:28.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.499383+0000) 2020-02-05T23:49:28.502 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:28.499+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.500257+0000) 2020-02-05T23:49:28.502 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:28.500+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.500497+0000) 2020-02-05T23:49:28.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:28.499+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.500510+0000) 2020-02-05T23:49:28.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:28.500+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.501130+0000) 2020-02-05T23:49:28.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:28.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.504668+0000) 2020-02-05T23:49:28.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:28.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:28.504943+0000) 2020-02-05T23:49:28.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:28.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:29.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:29.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:29.204781+0000) 2020-02-05T23:49:29.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:29.589+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:29.590225+0000) 2020-02-05T23:49:30.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:30.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:30.436969+0000) 2020-02-05T23:49:31.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:31.126+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:31.127562+0000) 2020-02-05T23:49:32.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:32.802+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:32.803351+0000) 2020-02-05T23:49:33.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:33.089+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.089803+0000) 2020-02-05T23:49:33.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:33.519+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.520286+0000) 2020-02-05T23:49:33.522 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:33.519+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.520786+0000) 2020-02-05T23:49:33.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:33.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.521511+0000) 2020-02-05T23:49:33.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:33.520+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.522241+0000) 2020-02-05T23:49:33.524 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:33.521+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.522901+0000) 2020-02-05T23:49:33.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:33.522+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.523233+0000) 2020-02-05T23:49:33.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:33.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.523483+0000) 2020-02-05T23:49:33.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:33.522+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.523965+0000) 2020-02-05T23:49:33.525 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:33.522+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.524279+0000) 2020-02-05T23:49:33.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:33.523+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.524922+0000) 2020-02-05T23:49:33.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:33.524+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.525543+0000) 2020-02-05T23:49:33.526 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:33.524+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.525755+0000) 2020-02-05T23:49:33.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:33.528+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.529989+0000) 2020-02-05T23:49:33.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:33.529+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.530280+0000) 2020-02-05T23:49:33.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:33.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:33.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:33.946+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.947709+0000) 2020-02-05T23:49:33.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:33.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.966719+0000) 2020-02-05T23:49:33.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:33.965+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:33.966805+0000) 2020-02-05T23:49:36.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:36.065+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:36.066974+0000) 2020-02-05T23:49:36.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:36.945+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:36.947053+0000) 2020-02-05T23:49:37.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:37.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:37.206470+0000) 2020-02-05T23:49:37.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:37.205+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:37.206683+0000) 2020-02-05T23:49:37.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:37.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:37.206672+0000) 2020-02-05T23:49:38.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:38.542+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.544367+0000) 2020-02-05T23:49:38.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:38.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.544691+0000) 2020-02-05T23:49:38.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:38.544+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.545158+0000) 2020-02-05T23:49:38.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:38.544+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.545820+0000) 2020-02-05T23:49:38.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:38.545+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.546216+0000) 2020-02-05T23:49:38.548 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:38.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.546390+0000) 2020-02-05T23:49:38.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:38.545+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.546912+0000) 2020-02-05T23:49:38.549 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:38.546+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.547560+0000) 2020-02-05T23:49:38.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:38.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.547884+0000) 2020-02-05T23:49:38.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:38.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.548355+0000) 2020-02-05T23:49:38.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:38.547+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.548291+0000) 2020-02-05T23:49:38.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:38.547+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.548532+0000) 2020-02-05T23:49:38.551 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:38.547+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.548995+0000) 2020-02-05T23:49:38.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:38.553+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:38.554701+0000) 2020-02-05T23:49:38.563 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:38.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 155986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:39.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:39.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:39.205027+0000) 2020-02-05T23:49:39.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:39.589+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:39.590477+0000) 2020-02-05T23:49:40.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:40.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:40.437225+0000) 2020-02-05T23:49:41.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:41.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:41.136529+0000) 2020-02-05T23:49:42.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:42.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:42.803586+0000) 2020-02-05T23:49:43.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:43.088+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.090019+0000) 2020-02-05T23:49:43.569 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:43.567+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.568282+0000) 2020-02-05T23:49:43.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:43.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.568995+0000) 2020-02-05T23:49:43.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:43.568+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.569786+0000) 2020-02-05T23:49:43.571 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:43.568+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.570278+0000) 2020-02-05T23:49:43.572 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:43.569+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.570961+0000) 2020-02-05T23:49:43.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:43.569+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.570892+0000) 2020-02-05T23:49:43.573 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:43.570+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.571635+0000) 2020-02-05T23:49:43.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:43.570+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.571851+0000) 2020-02-05T23:49:43.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:43.571+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.572072+0000) 2020-02-05T23:49:43.574 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:43.571+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.572491+0000) 2020-02-05T23:49:43.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:43.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.573285+0000) 2020-02-05T23:49:43.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:43.571+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.572603+0000) 2020-02-05T23:49:43.575 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:43.572+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.573610+0000) 2020-02-05T23:49:43.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:43.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.577370+0000) 2020-02-05T23:49:43.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:43.584+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:43.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:43.947+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.947917+0000) 2020-02-05T23:49:43.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:43.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.966939+0000) 2020-02-05T23:49:43.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:43.966+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:43.967140+0000) 2020-02-05T23:49:46.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:46.066+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:46.067233+0000) 2020-02-05T23:49:46.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:46.946+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:46.947323+0000) 2020-02-05T23:49:47.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:47.205+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:47.206702+0000) 2020-02-05T23:49:47.208 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:47.205+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:47.206926+0000) 2020-02-05T23:49:47.208 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:47.205+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:47.206909+0000) 2020-02-05T23:49:48.594 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:48.591+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.592826+0000) 2020-02-05T23:49:48.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:48.592+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.593459+0000) 2020-02-05T23:49:48.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:48.592+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.594099+0000) 2020-02-05T23:49:48.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:48.593+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.594577+0000) 2020-02-05T23:49:48.596 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:48.593+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.595105+0000) 2020-02-05T23:49:48.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:48.593+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.594998+0000) 2020-02-05T23:49:48.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:48.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.595477+0000) 2020-02-05T23:49:48.597 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:48.594+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.595553+0000) 2020-02-05T23:49:48.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:48.594+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.595741+0000) 2020-02-05T23:49:48.598 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:48.595+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.596697+0000) 2020-02-05T23:49:48.598 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:48.595+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.596924+0000) 2020-02-05T23:49:48.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:48.595+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.597385+0000) 2020-02-05T23:49:48.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:48.597+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.598227+0000) 2020-02-05T23:49:48.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:48.603+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:48.604288+0000) 2020-02-05T23:49:48.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:48.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156066 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:49.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:49.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:49.205269+0000) 2020-02-05T23:49:49.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:49.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:49.590708+0000) 2020-02-05T23:49:50.438 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:50.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:50.437485+0000) 2020-02-05T23:49:51.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:51.144+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:51.145196+0000) 2020-02-05T23:49:52.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:52.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:52.803819+0000) 2020-02-05T23:49:53.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:53.089+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.090284+0000) 2020-02-05T23:49:53.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:53.613+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.614856+0000) 2020-02-05T23:49:53.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:53.614+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.615322+0000) 2020-02-05T23:49:53.617 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:53.614+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.616304+0000) 2020-02-05T23:49:53.618 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:53.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.616778+0000) 2020-02-05T23:49:53.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:53.615+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.617106+0000) 2020-02-05T23:49:53.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:53.616+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.617536+0000) 2020-02-05T23:49:53.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:53.617+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.618894+0000) 2020-02-05T23:49:53.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:53.617+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.618719+0000) 2020-02-05T23:49:53.621 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:53.618+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.619406+0000) 2020-02-05T23:49:53.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:53.618+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.620005+0000) 2020-02-05T23:49:53.621 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:53.619+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.620453+0000) 2020-02-05T23:49:53.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:53.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.620984+0000) 2020-02-05T23:49:53.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:53.619+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.621084+0000) 2020-02-05T23:49:53.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:53.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.626323+0000) 2020-02-05T23:49:53.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:53.633+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:53.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:53.947+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.948128+0000) 2020-02-05T23:49:53.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:53.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.967148+0000) 2020-02-05T23:49:53.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:53.966+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:53.967408+0000) 2020-02-05T23:49:56.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:56.066+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:56.067493+0000) 2020-02-05T23:49:56.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:56.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:56.947595+0000) 2020-02-05T23:49:57.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:57.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:57.206958+0000) 2020-02-05T23:49:57.208 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:57.206+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:57.207145+0000) 2020-02-05T23:49:57.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:57.206+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:57.207133+0000) 2020-02-05T23:49:57.749 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:49:57.756 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:49:58.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:58.640+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.641120+0000) 2020-02-05T23:49:58.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:49:58.640+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.641485+0000) 2020-02-05T23:49:58.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:49:58.639+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.641320+0000) 2020-02-05T23:49:58.644 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:49:58.640+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.641956+0000) 2020-02-05T23:49:58.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:58.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.642481+0000) 2020-02-05T23:49:58.645 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:49:58.641+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.642690+0000) 2020-02-05T23:49:58.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:49:58.642+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.643458+0000) 2020-02-05T23:49:58.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:49:58.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.643899+0000) 2020-02-05T23:49:58.646 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:49:58.643+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.644569+0000) 2020-02-05T23:49:58.647 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:49:58.644+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.645655+0000) 2020-02-05T23:49:58.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:49:58.644+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.646242+0000) 2020-02-05T23:49:58.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:49:58.645+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.646765+0000) 2020-02-05T23:49:58.648 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:58.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.647290+0000) 2020-02-05T23:49:58.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:58.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:58.650799+0000) 2020-02-05T23:49:58.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:49:58.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156148 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:49:59.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:49:59.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:59.205510+0000) 2020-02-05T23:49:59.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:49:59.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:49:59.590927+0000) 2020-02-05T23:50:00.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:00.436+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:00.437728+0000) 2020-02-05T23:50:01.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:01.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:01.154215+0000) 2020-02-05T23:50:02.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:02.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:02.804123+0000) 2020-02-05T23:50:03.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:03.090+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.090520+0000) 2020-02-05T23:50:03.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:03.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.664864+0000) 2020-02-05T23:50:03.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:03.663+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.665342+0000) 2020-02-05T23:50:03.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:03.665+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.666436+0000) 2020-02-05T23:50:03.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:03.666+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.667471+0000) 2020-02-05T23:50:03.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:03.666+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.668023+0000) 2020-02-05T23:50:03.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:03.667+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.668373+0000) 2020-02-05T23:50:03.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:03.668+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.669492+0000) 2020-02-05T23:50:03.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:03.667+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.668991+0000) 2020-02-05T23:50:03.672 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:03.668+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.669944+0000) 2020-02-05T23:50:03.672 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:03.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.669596+0000) 2020-02-05T23:50:03.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:03.668+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.669942+0000) 2020-02-05T23:50:03.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:03.669+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.670440+0000) 2020-02-05T23:50:03.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:03.669+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.670774+0000) 2020-02-05T23:50:03.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:03.671+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.672618+0000) 2020-02-05T23:50:03.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:03.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156190 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:03.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:03.948+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.948336+0000) 2020-02-05T23:50:03.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:03.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.967349+0000) 2020-02-05T23:50:03.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:03.967+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:03.967614+0000) 2020-02-05T23:50:06.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:06.067+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:06.067742+0000) 2020-02-05T23:50:06.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:06.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:06.947824+0000) 2020-02-05T23:50:07.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:07.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:07.207208+0000) 2020-02-05T23:50:07.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:07.206+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:07.207387+0000) 2020-02-05T23:50:07.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:07.206+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:07.207384+0000) 2020-02-05T23:50:08.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:08.687+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.687863+0000) 2020-02-05T23:50:08.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:08.688+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.688345+0000) 2020-02-05T23:50:08.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:08.688+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.688983+0000) 2020-02-05T23:50:08.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:08.689+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.689320+0000) 2020-02-05T23:50:08.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:08.689+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.689709+0000) 2020-02-05T23:50:08.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:08.688+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.690135+0000) 2020-02-05T23:50:08.692 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:08.689+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.691265+0000) 2020-02-05T23:50:08.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:08.691+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.691334+0000) 2020-02-05T23:50:08.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:08.691+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.691665+0000) 2020-02-05T23:50:08.693 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:08.690+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.692068+0000) 2020-02-05T23:50:08.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:08.689+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.690798+0000) 2020-02-05T23:50:08.694 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:08.691+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.692839+0000) 2020-02-05T23:50:08.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:08.692+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.693565+0000) 2020-02-05T23:50:08.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:08.701+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:08.701951+0000) 2020-02-05T23:50:08.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:08.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:09.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:09.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:09.205758+0000) 2020-02-05T23:50:09.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:09.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:09.591230+0000) 2020-02-05T23:50:10.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:10.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:10.437990+0000) 2020-02-05T23:50:11.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:11.162+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:11.163590+0000) 2020-02-05T23:50:12.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:12.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:12.804390+0000) 2020-02-05T23:50:13.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:13.089+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.090756+0000) 2020-02-05T23:50:13.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:13.710+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.711904+0000) 2020-02-05T23:50:13.715 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:13.712+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.713540+0000) 2020-02-05T23:50:13.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:13.712+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.713929+0000) 2020-02-05T23:50:13.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:13.713+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.714795+0000) 2020-02-05T23:50:13.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:13.713+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.715183+0000) 2020-02-05T23:50:13.717 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:13.714+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.715836+0000) 2020-02-05T23:50:13.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:13.714+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.715810+0000) 2020-02-05T23:50:13.718 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:13.715+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.716536+0000) 2020-02-05T23:50:13.718 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:13.715+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.716294+0000) 2020-02-05T23:50:13.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:13.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.716292+0000) 2020-02-05T23:50:13.718 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:13.715+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.717313+0000) 2020-02-05T23:50:13.719 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:13.716+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.718329+0000) 2020-02-05T23:50:13.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:13.717+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.718667+0000) 2020-02-05T23:50:13.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:13.721+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.722719+0000) 2020-02-05T23:50:13.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:13.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156272 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:13.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:13.947+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.948547+0000) 2020-02-05T23:50:13.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:13.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.967557+0000) 2020-02-05T23:50:13.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:13.966+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:13.967830+0000) 2020-02-05T23:50:16.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:16.067+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:16.067976+0000) 2020-02-05T23:50:16.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:16.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:16.948107+0000) 2020-02-05T23:50:17.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:17.206+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:17.207493+0000) 2020-02-05T23:50:17.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:17.206+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:17.207715+0000) 2020-02-05T23:50:17.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:17.206+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:17.207661+0000) 2020-02-05T23:50:18.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:18.735+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.736574+0000) 2020-02-05T23:50:18.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:18.736+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.737259+0000) 2020-02-05T23:50:18.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:18.736+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.737632+0000) 2020-02-05T23:50:18.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:18.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.738165+0000) 2020-02-05T23:50:18.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:18.737+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.738377+0000) 2020-02-05T23:50:18.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:18.737+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.738551+0000) 2020-02-05T23:50:18.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:18.737+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.739135+0000) 2020-02-05T23:50:18.741 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:18.738+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.739511+0000) 2020-02-05T23:50:18.741 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:18.738+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.739173+0000) 2020-02-05T23:50:18.741 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:18.738+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.739634+0000) 2020-02-05T23:50:18.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:18.738+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.740194+0000) 2020-02-05T23:50:18.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:18.739+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.740420+0000) 2020-02-05T23:50:18.742 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:18.739+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.741161+0000) 2020-02-05T23:50:18.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:18.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:18.745136+0000) 2020-02-05T23:50:18.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:18.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156312 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:19.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:19.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:19.206036+0000) 2020-02-05T23:50:19.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:19.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:19.591463+0000) 2020-02-05T23:50:20.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:20.437+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:20.438294+0000) 2020-02-05T23:50:21.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:21.171+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:21.172745+0000) 2020-02-05T23:50:22.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:22.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:22.804681+0000) 2020-02-05T23:50:23.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:23.090+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.090988+0000) 2020-02-05T23:50:23.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:23.756+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.757390+0000) 2020-02-05T23:50:23.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:23.756+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.757751+0000) 2020-02-05T23:50:23.761 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:23.759+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.759572+0000) 2020-02-05T23:50:23.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:23.760+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.761041+0000) 2020-02-05T23:50:23.762 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:23.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.761048+0000) 2020-02-05T23:50:23.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:23.760+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.761474+0000) 2020-02-05T23:50:23.763 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:23.761+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.761798+0000) 2020-02-05T23:50:23.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:23.760+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.761899+0000) 2020-02-05T23:50:23.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:23.761+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.762259+0000) 2020-02-05T23:50:23.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:23.763+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.763600+0000) 2020-02-05T23:50:23.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:23.763+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.763412+0000) 2020-02-05T23:50:23.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:23.763+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.763919+0000) 2020-02-05T23:50:23.765 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:23.763+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.764208+0000) 2020-02-05T23:50:23.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:23.767+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.768338+0000) 2020-02-05T23:50:23.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:23.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:23.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:23.947+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.948760+0000) 2020-02-05T23:50:23.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:23.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.967836+0000) 2020-02-05T23:50:23.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:23.966+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:23.968006+0000) 2020-02-05T23:50:26.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:26.067+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:26.068224+0000) 2020-02-05T23:50:26.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:26.947+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:26.948372+0000) 2020-02-05T23:50:27.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:27.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:27.207682+0000) 2020-02-05T23:50:27.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:27.207+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:27.207953+0000) 2020-02-05T23:50:27.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:27.207+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:27.207896+0000) 2020-02-05T23:50:27.886 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:50:27.892 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:50:28.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:28.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.783182+0000) 2020-02-05T23:50:28.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:28.782+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.783273+0000) 2020-02-05T23:50:28.785 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:28.783+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.784327+0000) 2020-02-05T23:50:28.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:28.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.784820+0000) 2020-02-05T23:50:28.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:28.785+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.785542+0000) 2020-02-05T23:50:28.787 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:28.784+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.785753+0000) 2020-02-05T23:50:28.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:28.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.786763+0000) 2020-02-05T23:50:28.789 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:28.787+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.787612+0000) 2020-02-05T23:50:28.789 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:28.787+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.788195+0000) 2020-02-05T23:50:28.789 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:28.787+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.788075+0000) 2020-02-05T23:50:28.790 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:28.787+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.788957+0000) 2020-02-05T23:50:28.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:28.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.788811+0000) 2020-02-05T23:50:28.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:28.788+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.789584+0000) 2020-02-05T23:50:28.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:28.794+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:28.795517+0000) 2020-02-05T23:50:28.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:28.800+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156393 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:29.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:29.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:29.206261+0000) 2020-02-05T23:50:29.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:29.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:29.591668+0000) 2020-02-05T23:50:30.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:30.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:30.438547+0000) 2020-02-05T23:50:31.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:31.180+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:31.181203+0000) 2020-02-05T23:50:32.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:32.803+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:32.804925+0000) 2020-02-05T23:50:33.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:33.090+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.091241+0000) 2020-02-05T23:50:33.808 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:33.806+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.806988+0000) 2020-02-05T23:50:33.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:33.805+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.807042+0000) 2020-02-05T23:50:33.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:33.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.808128+0000) 2020-02-05T23:50:33.809 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:33.808+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.808581+0000) 2020-02-05T23:50:33.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:33.807+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.808909+0000) 2020-02-05T23:50:33.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:33.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.809585+0000) 2020-02-05T23:50:33.811 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:33.809+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.809559+0000) 2020-02-05T23:50:33.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:33.809+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.810183+0000) 2020-02-05T23:50:33.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:33.809+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.810962+0000) 2020-02-05T23:50:33.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:33.810+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.811212+0000) 2020-02-05T23:50:33.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:33.810+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.811364+0000) 2020-02-05T23:50:33.813 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:33.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.812032+0000) 2020-02-05T23:50:33.813 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:33.811+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.812417+0000) 2020-02-05T23:50:33.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:33.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.819328+0000) 2020-02-05T23:50:33.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:33.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156435 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:33.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:33.948+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.948949+0000) 2020-02-05T23:50:33.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:33.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.968026+0000) 2020-02-05T23:50:33.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:33.967+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:33.968187+0000) 2020-02-05T23:50:36.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:36.068+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:36.068477+0000) 2020-02-05T23:50:36.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:36.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:36.948605+0000) 2020-02-05T23:50:37.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:37.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:37.207932+0000) 2020-02-05T23:50:37.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:37.207+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:37.208119+0000) 2020-02-05T23:50:37.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:37.207+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:37.208240+0000) 2020-02-05T23:50:38.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:38.831+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.831789+0000) 2020-02-05T23:50:38.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:38.831+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.832762+0000) 2020-02-05T23:50:38.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:38.831+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.832987+0000) 2020-02-05T23:50:38.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:38.832+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.833994+0000) 2020-02-05T23:50:38.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:38.833+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.834728+0000) 2020-02-05T23:50:38.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:38.834+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.835208+0000) 2020-02-05T23:50:38.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:38.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.835067+0000) 2020-02-05T23:50:38.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:38.834+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.835535+0000) 2020-02-05T23:50:38.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:38.835+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.835877+0000) 2020-02-05T23:50:38.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:38.835+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.836030+0000) 2020-02-05T23:50:38.838 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:38.836+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.836553+0000) 2020-02-05T23:50:38.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:38.836+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.836904+0000) 2020-02-05T23:50:38.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:38.835+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.836366+0000) 2020-02-05T23:50:38.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:38.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:38.842287+0000) 2020-02-05T23:50:38.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:38.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:39.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:39.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:39.206498+0000) 2020-02-05T23:50:39.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:39.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:39.591900+0000) 2020-02-05T23:50:40.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:40.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:40.438804+0000) 2020-02-05T23:50:41.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:41.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:41.189598+0000) 2020-02-05T23:50:42.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:42.804+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:42.805155+0000) 2020-02-05T23:50:43.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:43.090+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.091465+0000) 2020-02-05T23:50:43.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:43.853+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.854265+0000) 2020-02-05T23:50:43.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:43.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.855296+0000) 2020-02-05T23:50:43.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:43.854+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.855371+0000) 2020-02-05T23:50:43.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:43.854+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.856029+0000) 2020-02-05T23:50:43.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:43.855+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.856319+0000) 2020-02-05T23:50:43.867 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:43.855+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.856518+0000) 2020-02-05T23:50:43.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:43.856+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.857518+0000) 2020-02-05T23:50:43.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:43.857+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.857832+0000) 2020-02-05T23:50:43.869 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:43.857+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.858242+0000) 2020-02-05T23:50:43.870 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:43.858+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.858895+0000) 2020-02-05T23:50:43.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:43.858+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.859656+0000) 2020-02-05T23:50:43.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:43.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.865542+0000) 2020-02-05T23:50:43.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:43.858+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.859143+0000) 2020-02-05T23:50:43.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:43.859+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.859806+0000) 2020-02-05T23:50:43.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:43.874+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:43.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:43.948+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.949161+0000) 2020-02-05T23:50:43.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:43.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.968220+0000) 2020-02-05T23:50:43.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:43.967+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:43.968369+0000) 2020-02-05T23:50:46.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:46.068+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:46.068743+0000) 2020-02-05T23:50:46.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:46.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:46.948873+0000) 2020-02-05T23:50:47.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:47.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:47.208173+0000) 2020-02-05T23:50:47.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:47.207+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:47.208336+0000) 2020-02-05T23:50:47.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:47.208+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:47.208540+0000) 2020-02-05T23:50:48.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:48.882+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.882630+0000) 2020-02-05T23:50:48.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:48.882+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.883477+0000) 2020-02-05T23:50:48.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:48.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.883646+0000) 2020-02-05T23:50:48.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:48.882+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.883770+0000) 2020-02-05T23:50:48.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:48.883+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.884441+0000) 2020-02-05T23:50:48.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:48.883+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.884226+0000) 2020-02-05T23:50:48.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:48.883+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.884929+0000) 2020-02-05T23:50:48.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:48.884+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.885486+0000) 2020-02-05T23:50:48.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:48.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.885520+0000) 2020-02-05T23:50:48.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:48.885+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.885666+0000) 2020-02-05T23:50:48.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:48.885+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.886222+0000) 2020-02-05T23:50:48.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:48.885+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.886201+0000) 2020-02-05T23:50:48.888 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:48.886+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.886744+0000) 2020-02-05T23:50:48.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:48.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:48.890469+0000) 2020-02-05T23:50:48.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:48.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156556 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:49.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:49.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:49.206728+0000) 2020-02-05T23:50:49.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:49.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:49.592144+0000) 2020-02-05T23:50:50.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:50.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:50.439030+0000) 2020-02-05T23:50:51.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:51.198+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:51.198936+0000) 2020-02-05T23:50:52.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:52.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:52.805396+0000) 2020-02-05T23:50:53.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:53.090+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.091716+0000) 2020-02-05T23:50:53.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:53.903+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.903420+0000) 2020-02-05T23:50:53.906 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:53.904+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.904523+0000) 2020-02-05T23:50:53.906 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:53.903+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.904579+0000) 2020-02-05T23:50:53.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:53.904+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.905472+0000) 2020-02-05T23:50:53.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:53.904+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.906016+0000) 2020-02-05T23:50:53.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:53.904+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.905910+0000) 2020-02-05T23:50:53.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:53.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.906535+0000) 2020-02-05T23:50:53.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:53.906+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.907139+0000) 2020-02-05T23:50:53.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:53.908+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.908611+0000) 2020-02-05T23:50:53.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:53.908+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.908985+0000) 2020-02-05T23:50:53.910 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:53.909+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.909580+0000) 2020-02-05T23:50:53.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:53.908+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.909802+0000) 2020-02-05T23:50:53.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:53.909+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.910137+0000) 2020-02-05T23:50:53.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:53.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.914954+0000) 2020-02-05T23:50:53.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:53.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:53.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:53.948+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.949317+0000) 2020-02-05T23:50:53.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:53.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.968413+0000) 2020-02-05T23:50:53.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:53.967+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:53.968547+0000) 2020-02-05T23:50:56.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:56.068+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:56.069010+0000) 2020-02-05T23:50:56.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:56.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:56.949068+0000) 2020-02-05T23:50:57.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:57.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:57.208369+0000) 2020-02-05T23:50:57.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:57.208+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:57.208590+0000) 2020-02-05T23:50:57.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:57.208+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:57.208793+0000) 2020-02-05T23:50:58.035 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:50:58.042 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:50:58.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:50:58.925+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.926794+0000) 2020-02-05T23:50:58.929 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:50:58.927+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.927782+0000) 2020-02-05T23:50:58.929 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:58.927+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.928379+0000) 2020-02-05T23:50:58.930 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:50:58.928+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.929029+0000) 2020-02-05T23:50:58.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:50:58.928+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.929252+0000) 2020-02-05T23:50:58.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:50:58.929+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.929952+0000) 2020-02-05T23:50:58.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:58.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.930251+0000) 2020-02-05T23:50:58.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:50:58.930+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.930578+0000) 2020-02-05T23:50:58.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:50:58.929+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.930961+0000) 2020-02-05T23:50:58.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:50:58.930+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.931414+0000) 2020-02-05T23:50:58.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:58.931+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.932383+0000) 2020-02-05T23:50:58.935 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:50:58.931+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.932987+0000) 2020-02-05T23:50:58.935 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:50:58.933+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.934268+0000) 2020-02-05T23:50:58.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:58.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:58.936314+0000) 2020-02-05T23:50:58.946 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:50:58.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:50:59.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:50:59.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:59.206957+0000) 2020-02-05T23:50:59.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:50:59.592+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:50:59.592374+0000) 2020-02-05T23:51:00.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:00.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:00.439282+0000) 2020-02-05T23:51:01.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:01.207+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:01.207674+0000) 2020-02-05T23:51:02.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:02.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:02.805689+0000) 2020-02-05T23:51:03.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:03.091+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.091963+0000) 2020-02-05T23:51:03.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:03.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.949957+0000) 2020-02-05T23:51:03.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:03.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.952175+0000) 2020-02-05T23:51:03.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:03.952+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.952394+0000) 2020-02-05T23:51:03.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:03.953+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.953451+0000) 2020-02-05T23:51:03.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:03.951+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.953149+0000) 2020-02-05T23:51:03.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:03.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.954284+0000) 2020-02-05T23:51:03.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:03.953+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.954571+0000) 2020-02-05T23:51:03.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:03.955+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.955235+0000) 2020-02-05T23:51:03.957 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:03.954+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.955849+0000) 2020-02-05T23:51:03.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:03.955+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.956557+0000) 2020-02-05T23:51:03.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:03.956+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.956854+0000) 2020-02-05T23:51:03.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:03.955+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.957314+0000) 2020-02-05T23:51:03.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:03.956+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.957049+0000) 2020-02-05T23:51:03.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:03.961+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.961891+0000) 2020-02-05T23:51:03.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:03.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156677 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:03.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:03.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.968622+0000) 2020-02-05T23:51:03.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:03.968+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:03.968734+0000) 2020-02-05T23:51:06.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:06.067+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:06.069254+0000) 2020-02-05T23:51:06.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:06.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:06.949365+0000) 2020-02-05T23:51:07.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:07.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:07.208638+0000) 2020-02-05T23:51:07.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:07.207+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:07.208817+0000) 2020-02-05T23:51:07.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:07.207+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:07.209021+0000) 2020-02-05T23:51:08.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:08.973+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.974491+0000) 2020-02-05T23:51:08.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:08.974+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.975586+0000) 2020-02-05T23:51:08.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:08.973+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.975318+0000) 2020-02-05T23:51:08.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:08.975+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.976831+0000) 2020-02-05T23:51:08.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:08.976+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.978162+0000) 2020-02-05T23:51:08.980 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:08.978+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.979148+0000) 2020-02-05T23:51:08.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:08.977+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.979043+0000) 2020-02-05T23:51:08.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:08.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.979618+0000) 2020-02-05T23:51:08.981 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:08.978+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.980130+0000) 2020-02-05T23:51:08.981 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:08.979+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.980293+0000) 2020-02-05T23:51:08.982 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:08.979+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.981286+0000) 2020-02-05T23:51:08.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:08.979+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.980841+0000) 2020-02-05T23:51:08.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:08.980+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.981564+0000) 2020-02-05T23:51:08.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:08.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:08.985928+0000) 2020-02-05T23:51:08.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:08.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:09.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:09.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:09.207235+0000) 2020-02-05T23:51:09.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:09.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:09.592588+0000) 2020-02-05T23:51:10.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:10.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:10.439534+0000) 2020-02-05T23:51:10.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:10.699+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:10.700705+0000) 2020-02-05T23:51:11.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:11.215+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:11.216154+0000) 2020-02-05T23:51:12.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:12.804+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:12.805922+0000) 2020-02-05T23:51:13.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:13.091+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:13.092226+0000) 2020-02-05T23:51:13.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:13.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:13.968826+0000) 2020-02-05T23:51:13.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:13.967+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:13.968913+0000) 2020-02-05T23:51:13.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:13.997+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:13.998130+0000) 2020-02-05T23:51:14.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:13.997+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:13.998544+0000) 2020-02-05T23:51:14.001 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:13.998+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.000053+0000) 2020-02-05T23:51:14.002 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:13.999+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.000841+0000) 2020-02-05T23:51:14.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:14.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.001416+0000) 2020-02-05T23:51:14.003 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:14.000+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.001753+0000) 2020-02-05T23:51:14.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:14.000+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.002116+0000) 2020-02-05T23:51:14.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:14.001+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.002561+0000) 2020-02-05T23:51:14.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:14.001+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.002923+0000) 2020-02-05T23:51:14.005 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:14.002+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.004367+0000) 2020-02-05T23:51:14.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:14.003+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.004405+0000) 2020-02-05T23:51:14.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:14.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.005274+0000) 2020-02-05T23:51:14.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:14.004+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.005578+0000) 2020-02-05T23:51:14.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:14.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:14.007648+0000) 2020-02-05T23:51:14.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:14.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:16.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:16.068+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:16.069505+0000) 2020-02-05T23:51:16.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:16.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:16.949621+0000) 2020-02-05T23:51:17.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:17.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:17.208893+0000) 2020-02-05T23:51:17.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:17.208+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:17.209025+0000) 2020-02-05T23:51:17.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:17.208+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:17.209396+0000) 2020-02-05T23:51:19.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:19.022+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.023390+0000) 2020-02-05T23:51:19.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:19.023+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.024140+0000) 2020-02-05T23:51:19.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:19.023+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.024635+0000) 2020-02-05T23:51:19.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:19.023+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.024884+0000) 2020-02-05T23:51:19.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:19.024+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.025275+0000) 2020-02-05T23:51:19.027 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:19.024+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.025862+0000) 2020-02-05T23:51:19.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:19.024+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.025959+0000) 2020-02-05T23:51:19.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:19.025+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.026626+0000) 2020-02-05T23:51:19.029 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:19.026+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.027845+0000) 2020-02-05T23:51:19.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:19.027+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.028529+0000) 2020-02-05T23:51:19.030 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:19.026+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.028250+0000) 2020-02-05T23:51:19.030 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:19.027+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.028823+0000) 2020-02-05T23:51:19.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:19.027+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.028888+0000) 2020-02-05T23:51:19.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:19.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.034011+0000) 2020-02-05T23:51:19.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:19.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:19.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:19.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.207493+0000) 2020-02-05T23:51:19.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:19.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:19.592816+0000) 2020-02-05T23:51:20.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:20.438+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:20.439775+0000) 2020-02-05T23:51:20.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:20.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:20.700940+0000) 2020-02-05T23:51:21.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:21.223+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:21.224478+0000) 2020-02-05T23:51:22.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:22.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:22.806154+0000) 2020-02-05T23:51:23.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:23.091+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:23.092460+0000) 2020-02-05T23:51:23.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:23.968+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:23.969122+0000) 2020-02-05T23:51:23.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:23.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:23.969115+0000) 2020-02-05T23:51:24.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:24.043+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.044885+0000) 2020-02-05T23:51:24.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:24.044+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.045427+0000) 2020-02-05T23:51:24.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:24.044+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.045869+0000) 2020-02-05T23:51:24.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:24.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.046260+0000) 2020-02-05T23:51:24.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:24.045+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.046883+0000) 2020-02-05T23:51:24.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:24.045+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.047191+0000) 2020-02-05T23:51:24.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:24.046+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.048372+0000) 2020-02-05T23:51:24.050 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:24.047+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.048922+0000) 2020-02-05T23:51:24.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:24.047+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.049188+0000) 2020-02-05T23:51:24.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:24.048+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.049537+0000) 2020-02-05T23:51:24.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:24.048+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.050394+0000) 2020-02-05T23:51:24.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:24.048+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.050369+0000) 2020-02-05T23:51:24.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:24.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.056431+0000) 2020-02-05T23:51:24.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:24.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:24.056744+0000) 2020-02-05T23:51:24.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:24.061+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156839 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:26.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:26.068+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:26.069738+0000) 2020-02-05T23:51:26.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:26.948+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:26.949854+0000) 2020-02-05T23:51:27.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:27.208+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:27.209132+0000) 2020-02-05T23:51:27.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:27.208+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:27.209244+0000) 2020-02-05T23:51:27.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:27.208+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:27.209640+0000) 2020-02-05T23:51:28.165 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:51:28.171 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:51:29.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:29.068+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.069286+0000) 2020-02-05T23:51:29.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:29.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.069727+0000) 2020-02-05T23:51:29.071 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:29.068+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.070078+0000) 2020-02-05T23:51:29.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:29.068+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.070382+0000) 2020-02-05T23:51:29.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:29.069+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.071029+0000) 2020-02-05T23:51:29.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:29.070+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.071159+0000) 2020-02-05T23:51:29.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:29.070+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.071548+0000) 2020-02-05T23:51:29.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:29.070+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.071818+0000) 2020-02-05T23:51:29.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:29.070+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.071986+0000) 2020-02-05T23:51:29.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:29.071+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.072690+0000) 2020-02-05T23:51:29.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:29.070+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.072063+0000) 2020-02-05T23:51:29.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:29.071+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.072936+0000) 2020-02-05T23:51:29.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:29.072+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.074104+0000) 2020-02-05T23:51:29.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:29.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.080031+0000) 2020-02-05T23:51:29.087 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:29.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156880 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:29.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:29.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.207756+0000) 2020-02-05T23:51:29.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:29.592+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:29.593026+0000) 2020-02-05T23:51:30.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:30.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:30.440029+0000) 2020-02-05T23:51:30.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:30.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:30.701170+0000) 2020-02-05T23:51:31.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:31.232+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:31.232973+0000) 2020-02-05T23:51:32.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:32.805+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:32.806337+0000) 2020-02-05T23:51:33.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:33.091+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:33.092708+0000) 2020-02-05T23:51:33.970 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:33.968+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:33.969394+0000) 2020-02-05T23:51:33.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:33.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:33.969394+0000) 2020-02-05T23:51:34.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:34.089+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.091284+0000) 2020-02-05T23:51:34.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:34.091+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.092482+0000) 2020-02-05T23:51:34.094 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:34.091+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.093055+0000) 2020-02-05T23:51:34.094 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:34.092+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.093825+0000) 2020-02-05T23:51:34.095 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:34.092+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.094183+0000) 2020-02-05T23:51:34.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:34.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.093924+0000) 2020-02-05T23:51:34.096 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:34.093+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.094430+0000) 2020-02-05T23:51:34.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:34.093+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.094718+0000) 2020-02-05T23:51:34.097 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:34.093+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.095046+0000) 2020-02-05T23:51:34.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:34.094+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.095275+0000) 2020-02-05T23:51:34.097 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:34.094+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.095588+0000) 2020-02-05T23:51:34.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:34.094+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.095878+0000) 2020-02-05T23:51:34.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:34.095+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.096560+0000) 2020-02-05T23:51:34.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:34.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:34.103015+0000) 2020-02-05T23:51:34.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:34.107+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:36.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:36.069+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:36.069982+0000) 2020-02-05T23:51:36.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:36.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:36.950114+0000) 2020-02-05T23:51:37.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:37.207+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:37.209390+0000) 2020-02-05T23:51:37.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:37.208+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:37.209489+0000) 2020-02-05T23:51:37.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:37.208+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:37.209859+0000) 2020-02-05T23:51:39.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:39.113+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.114953+0000) 2020-02-05T23:51:39.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:39.114+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.115740+0000) 2020-02-05T23:51:39.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:39.115+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.116427+0000) 2020-02-05T23:51:39.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:39.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.117149+0000) 2020-02-05T23:51:39.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:39.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.119221+0000) 2020-02-05T23:51:39.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:39.116+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.117739+0000) 2020-02-05T23:51:39.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:39.116+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.117775+0000) 2020-02-05T23:51:39.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:39.117+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.118907+0000) 2020-02-05T23:51:39.129 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:39.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.119686+0000) 2020-02-05T23:51:39.130 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:39.119+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.120246+0000) 2020-02-05T23:51:39.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:39.123+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.124258+0000) 2020-02-05T23:51:39.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:39.119+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.120772+0000) 2020-02-05T23:51:39.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:39.119+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.120975+0000) 2020-02-05T23:51:39.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:39.119+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.121306+0000) 2020-02-05T23:51:39.135 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:39.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 156961 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:39.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:39.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.208034+0000) 2020-02-05T23:51:39.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:39.592+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:39.593268+0000) 2020-02-05T23:51:40.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:40.439+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:40.440271+0000) 2020-02-05T23:51:40.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:40.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:40.701418+0000) 2020-02-05T23:51:41.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:41.241+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:41.241800+0000) 2020-02-05T23:51:42.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:42.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:42.806569+0000) 2020-02-05T23:51:43.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:43.092+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:43.092939+0000) 2020-02-05T23:51:43.971 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:43.969+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:43.969667+0000) 2020-02-05T23:51:43.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:43.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:43.969665+0000) 2020-02-05T23:51:44.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:44.139+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.140687+0000) 2020-02-05T23:51:44.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:44.139+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.140990+0000) 2020-02-05T23:51:44.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:44.140+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.141692+0000) 2020-02-05T23:51:44.148 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:44.140+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.142017+0000) 2020-02-05T23:51:44.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:44.142+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.143498+0000) 2020-02-05T23:51:44.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:44.142+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.143748+0000) 2020-02-05T23:51:44.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:44.136+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.137419+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:44.137+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.138258+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:44.137+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.139014+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:44.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.141110+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:44.138+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.139868+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:44.142+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.143161+0000) 2020-02-05T23:51:44.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:44.142+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.143433+0000) 2020-02-05T23:51:44.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:44.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:44.149430+0000) 2020-02-05T23:51:44.155 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:44.152+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:46.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:46.069+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:46.070210+0000) 2020-02-05T23:51:46.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:46.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:46.950306+0000) 2020-02-05T23:51:47.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:47.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:47.209725+0000) 2020-02-05T23:51:47.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:47.209+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:47.209725+0000) 2020-02-05T23:51:47.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:47.209+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:47.210123+0000) 2020-02-05T23:51:49.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:49.159+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.160638+0000) 2020-02-05T23:51:49.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:49.161+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.162158+0000) 2020-02-05T23:51:49.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:49.162+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.163158+0000) 2020-02-05T23:51:49.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:49.162+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.163594+0000) 2020-02-05T23:51:49.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:49.162+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.163803+0000) 2020-02-05T23:51:49.165 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:49.163+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.164363+0000) 2020-02-05T23:51:49.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:49.163+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.165016+0000) 2020-02-05T23:51:49.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:49.163+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.164920+0000) 2020-02-05T23:51:49.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:49.164+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.165560+0000) 2020-02-05T23:51:49.167 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:49.164+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.166047+0000) 2020-02-05T23:51:49.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:49.164+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.166216+0000) 2020-02-05T23:51:49.168 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:49.164+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.166362+0000) 2020-02-05T23:51:49.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:49.165+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.166635+0000) 2020-02-05T23:51:49.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:49.167+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.168693+0000) 2020-02-05T23:51:49.178 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:49.176+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:49.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:49.207+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.208257+0000) 2020-02-05T23:51:49.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:49.593+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:49.593477+0000) 2020-02-05T23:51:50.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:50.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:50.440504+0000) 2020-02-05T23:51:50.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:50.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:50.701649+0000) 2020-02-05T23:51:51.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:51.249+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:51.250128+0000) 2020-02-05T23:51:52.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:52.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:52.806802+0000) 2020-02-05T23:51:53.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:53.092+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:53.093201+0000) 2020-02-05T23:51:53.971 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:53.969+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:53.969930+0000) 2020-02-05T23:51:53.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:53.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:53.969933+0000) 2020-02-05T23:51:54.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:54.181+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.182581+0000) 2020-02-05T23:51:54.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:54.182+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.183148+0000) 2020-02-05T23:51:54.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:54.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.183660+0000) 2020-02-05T23:51:54.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:54.184+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.185154+0000) 2020-02-05T23:51:54.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:54.184+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.186005+0000) 2020-02-05T23:51:54.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:54.186+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.186480+0000) 2020-02-05T23:51:54.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:54.186+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.187182+0000) 2020-02-05T23:51:54.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:54.186+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.187471+0000) 2020-02-05T23:51:54.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:54.187+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.187724+0000) 2020-02-05T23:51:54.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:54.188+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.188973+0000) 2020-02-05T23:51:54.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:54.187+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.189043+0000) 2020-02-05T23:51:54.190 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:54.188+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.189396+0000) 2020-02-05T23:51:54.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:54.190+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.191928+0000) 2020-02-05T23:51:54.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:54.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:54.192342+0000) 2020-02-05T23:51:54.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:54.199+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157082 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:56.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:56.070+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:56.070474+0000) 2020-02-05T23:51:56.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:56.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:56.950508+0000) 2020-02-05T23:51:57.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:57.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:57.209990+0000) 2020-02-05T23:51:57.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:57.209+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:57.209990+0000) 2020-02-05T23:51:57.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:57.209+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:57.210375+0000) 2020-02-05T23:51:58.310 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:51:58.320 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:51:59.206 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:51:59.205+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.205337+0000) 2020-02-05T23:51:59.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:59.206+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.206662+0000) 2020-02-05T23:51:59.208 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:51:59.206+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.206916+0000) 2020-02-05T23:51:59.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:51:59.207+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.208050+0000) 2020-02-05T23:51:59.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:59.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.208479+0000) 2020-02-05T23:51:59.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:51:59.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.208422+0000) 2020-02-05T23:51:59.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:51:59.208+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.209028+0000) 2020-02-05T23:51:59.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:51:59.209+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.209543+0000) 2020-02-05T23:51:59.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:51:59.210+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.210491+0000) 2020-02-05T23:51:59.213 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:51:59.209+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.209651+0000) 2020-02-05T23:51:59.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:51:59.210+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.210524+0000) 2020-02-05T23:51:59.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:59.211+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.211217+0000) 2020-02-05T23:51:59.214 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:51:59.211+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.211647+0000) 2020-02-05T23:51:59.214 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:51:59.211+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.211629+0000) 2020-02-05T23:51:59.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:59.216+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.216509+0000) 2020-02-05T23:51:59.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:51:59.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:51:59.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:51:59.593+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:51:59.593714+0000) 2020-02-05T23:52:00.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:00.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:00.440754+0000) 2020-02-05T23:52:00.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:00.700+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:00.701872+0000) 2020-02-05T23:52:01.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:01.260+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:01.261684+0000) 2020-02-05T23:52:02.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:02.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:02.807045+0000) 2020-02-05T23:52:03.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:03.092+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:03.093521+0000) 2020-02-05T23:52:03.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:03.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:03.970196+0000) 2020-02-05T23:52:03.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:03.969+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:03.970193+0000) 2020-02-05T23:52:04.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:04.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.228662+0000) 2020-02-05T23:52:04.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:04.229+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.229529+0000) 2020-02-05T23:52:04.231 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:04.228+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.229866+0000) 2020-02-05T23:52:04.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:04.229+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.230339+0000) 2020-02-05T23:52:04.232 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:04.229+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.230908+0000) 2020-02-05T23:52:04.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:04.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.231620+0000) 2020-02-05T23:52:04.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:04.232+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.232630+0000) 2020-02-05T23:52:04.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:04.232+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.232886+0000) 2020-02-05T23:52:04.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:04.232+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.233290+0000) 2020-02-05T23:52:04.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:04.233+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.234432+0000) 2020-02-05T23:52:04.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:04.233+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.234040+0000) 2020-02-05T23:52:04.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:04.233+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.234143+0000) 2020-02-05T23:52:04.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:04.235+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.235459+0000) 2020-02-05T23:52:04.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:04.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:04.238608+0000) 2020-02-05T23:52:04.248 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:04.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:06.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:06.070+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:06.070727+0000) 2020-02-05T23:52:06.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:06.949+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:06.950749+0000) 2020-02-05T23:52:07.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:07.209+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:07.210270+0000) 2020-02-05T23:52:07.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:07.209+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:07.210270+0000) 2020-02-05T23:52:07.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:07.210+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:07.210616+0000) 2020-02-05T23:52:09.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:09.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.208663+0000) 2020-02-05T23:52:09.254 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:09.251+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.252977+0000) 2020-02-05T23:52:09.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:09.251+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.252047+0000) 2020-02-05T23:52:09.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:09.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.254146+0000) 2020-02-05T23:52:09.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:09.254+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.254688+0000) 2020-02-05T23:52:09.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:09.253+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.254882+0000) 2020-02-05T23:52:09.257 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:09.254+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.255751+0000) 2020-02-05T23:52:09.257 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:09.255+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.256346+0000) 2020-02-05T23:52:09.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:09.255+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.256906+0000) 2020-02-05T23:52:09.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:09.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.257807+0000) 2020-02-05T23:52:09.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:09.257+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.258341+0000) 2020-02-05T23:52:09.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:09.258+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.258712+0000) 2020-02-05T23:52:09.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:09.259+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.259788+0000) 2020-02-05T23:52:09.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:09.260+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.260937+0000) 2020-02-05T23:52:09.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:09.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.264762+0000) 2020-02-05T23:52:09.273 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:09.271+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:09.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:09.593+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:09.593928+0000) 2020-02-05T23:52:10.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:10.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:10.441000+0000) 2020-02-05T23:52:10.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:10.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:10.702128+0000) 2020-02-05T23:52:11.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:11.270+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:11.271422+0000) 2020-02-05T23:52:12.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:12.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:12.807327+0000) 2020-02-05T23:52:13.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:13.092+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:13.093772+0000) 2020-02-05T23:52:13.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:13.969+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:13.970449+0000) 2020-02-05T23:52:13.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:13.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:13.970449+0000) 2020-02-05T23:52:14.280 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:14.277+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.278802+0000) 2020-02-05T23:52:14.280 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:14.278+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.279152+0000) 2020-02-05T23:52:14.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:14.279+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.279551+0000) 2020-02-05T23:52:14.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:14.279+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.280027+0000) 2020-02-05T23:52:14.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:14.279+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.280391+0000) 2020-02-05T23:52:14.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:14.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.280716+0000) 2020-02-05T23:52:14.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:14.280+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.281344+0000) 2020-02-05T23:52:14.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:14.280+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.281897+0000) 2020-02-05T23:52:14.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:14.281+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.281822+0000) 2020-02-05T23:52:14.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:14.281+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.282294+0000) 2020-02-05T23:52:14.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:14.282+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.282759+0000) 2020-02-05T23:52:14.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:14.282+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.283153+0000) 2020-02-05T23:52:14.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:14.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.288328+0000) 2020-02-05T23:52:14.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:14.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:14.288588+0000) 2020-02-05T23:52:14.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:14.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157242 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:16.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:16.070+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:16.070910+0000) 2020-02-05T23:52:16.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:16.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:16.950998+0000) 2020-02-05T23:52:17.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:17.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:17.210577+0000) 2020-02-05T23:52:17.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:17.210+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:17.210577+0000) 2020-02-05T23:52:17.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:17.210+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:17.210859+0000) 2020-02-05T23:52:19.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:19.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.208932+0000) 2020-02-05T23:52:19.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:19.300+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.301417+0000) 2020-02-05T23:52:19.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:19.301+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.302069+0000) 2020-02-05T23:52:19.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:19.301+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.302530+0000) 2020-02-05T23:52:19.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:19.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.304195+0000) 2020-02-05T23:52:19.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:19.304+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.304762+0000) 2020-02-05T23:52:19.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:19.304+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.305115+0000) 2020-02-05T23:52:19.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:19.304+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.305478+0000) 2020-02-05T23:52:19.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:19.306+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.306691+0000) 2020-02-05T23:52:19.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:19.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.306602+0000) 2020-02-05T23:52:19.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:19.306+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.307381+0000) 2020-02-05T23:52:19.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:19.307+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.307856+0000) 2020-02-05T23:52:19.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:19.307+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.308276+0000) 2020-02-05T23:52:19.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:19.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.312244+0000) 2020-02-05T23:52:19.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:19.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.312547+0000) 2020-02-05T23:52:19.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:19.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:19.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:19.593+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:19.594143+0000) 2020-02-05T23:52:20.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:20.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:20.441256+0000) 2020-02-05T23:52:20.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:20.701+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:20.702400+0000) 2020-02-05T23:52:21.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:21.279+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:21.280352+0000) 2020-02-05T23:52:22.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:22.806+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:22.807558+0000) 2020-02-05T23:52:23.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:23.092+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:23.094042+0000) 2020-02-05T23:52:23.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:23.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:23.970749+0000) 2020-02-05T23:52:23.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:23.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:23.970749+0000) 2020-02-05T23:52:24.327 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:24.325+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.326536+0000) 2020-02-05T23:52:24.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:24.326+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.327282+0000) 2020-02-05T23:52:24.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:24.327+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.327896+0000) 2020-02-05T23:52:24.329 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:24.327+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.328260+0000) 2020-02-05T23:52:24.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:24.327+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.328811+0000) 2020-02-05T23:52:24.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:24.328+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.329817+0000) 2020-02-05T23:52:24.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:24.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.330213+0000) 2020-02-05T23:52:24.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:24.330+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.330913+0000) 2020-02-05T23:52:24.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:24.330+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.331207+0000) 2020-02-05T23:52:24.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:24.330+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.331441+0000) 2020-02-05T23:52:24.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:24.332+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.332508+0000) 2020-02-05T23:52:24.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:24.331+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.332075+0000) 2020-02-05T23:52:24.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:24.332+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.332989+0000) 2020-02-05T23:52:24.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:24.334+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:24.335979+0000) 2020-02-05T23:52:24.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:24.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157321 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:26.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:26.071+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:26.071153+0000) 2020-02-05T23:52:26.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:26.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:26.951263+0000) 2020-02-05T23:52:27.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:27.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:27.210833+0000) 2020-02-05T23:52:27.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:27.210+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:27.210836+0000) 2020-02-05T23:52:27.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:27.210+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:27.211071+0000) 2020-02-05T23:52:28.452 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:52:28.459 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:52:29.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:29.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.209203+0000) 2020-02-05T23:52:29.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:29.350+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.351948+0000) 2020-02-05T23:52:29.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:29.351+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.352115+0000) 2020-02-05T23:52:29.354 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:29.351+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.352535+0000) 2020-02-05T23:52:29.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:29.352+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.353451+0000) 2020-02-05T23:52:29.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:29.353+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.354251+0000) 2020-02-05T23:52:29.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:29.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.354629+0000) 2020-02-05T23:52:29.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:29.354+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.355094+0000) 2020-02-05T23:52:29.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:29.354+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.355353+0000) 2020-02-05T23:52:29.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:29.357+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.357930+0000) 2020-02-05T23:52:29.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:29.356+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.358045+0000) 2020-02-05T23:52:29.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:29.357+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.358307+0000) 2020-02-05T23:52:29.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:29.358+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.358598+0000) 2020-02-05T23:52:29.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:29.358+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.358650+0000) 2020-02-05T23:52:29.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:29.361+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.362445+0000) 2020-02-05T23:52:29.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:29.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:29.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:29.593+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:29.594351+0000) 2020-02-05T23:52:30.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:30.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:30.441576+0000) 2020-02-05T23:52:30.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:30.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:30.702658+0000) 2020-02-05T23:52:31.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:31.288+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:31.289025+0000) 2020-02-05T23:52:32.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:32.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:32.807781+0000) 2020-02-05T23:52:33.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:33.093+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:33.094300+0000) 2020-02-05T23:52:33.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:33.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:33.971031+0000) 2020-02-05T23:52:33.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:33.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:33.971031+0000) 2020-02-05T23:52:34.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:34.375+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.376398+0000) 2020-02-05T23:52:34.378 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:34.375+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.376947+0000) 2020-02-05T23:52:34.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:34.376+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.378027+0000) 2020-02-05T23:52:34.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:34.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.378485+0000) 2020-02-05T23:52:34.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:34.377+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.379351+0000) 2020-02-05T23:52:34.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:34.378+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.379636+0000) 2020-02-05T23:52:34.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:34.379+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.380689+0000) 2020-02-05T23:52:34.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:34.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.380541+0000) 2020-02-05T23:52:34.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:34.380+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.381176+0000) 2020-02-05T23:52:34.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:34.380+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.381301+0000) 2020-02-05T23:52:34.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:34.380+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.381814+0000) 2020-02-05T23:52:34.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:34.380+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.381864+0000) 2020-02-05T23:52:34.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:34.381+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.382411+0000) 2020-02-05T23:52:34.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:34.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:34.385250+0000) 2020-02-05T23:52:34.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:34.393+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:36.072 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:36.069+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:36.071393+0000) 2020-02-05T23:52:36.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:36.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:36.951498+0000) 2020-02-05T23:52:37.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:37.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:37.211145+0000) 2020-02-05T23:52:37.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:37.210+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:37.211143+0000) 2020-02-05T23:52:37.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:37.210+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:37.211346+0000) 2020-02-05T23:52:39.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:39.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.209480+0000) 2020-02-05T23:52:39.402 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:39.400+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.401397+0000) 2020-02-05T23:52:39.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:39.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.402115+0000) 2020-02-05T23:52:39.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:39.401+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.402500+0000) 2020-02-05T23:52:39.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:39.401+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.402853+0000) 2020-02-05T23:52:39.404 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:39.402+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.403670+0000) 2020-02-05T23:52:39.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:39.403+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.404487+0000) 2020-02-05T23:52:39.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:39.403+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.404699+0000) 2020-02-05T23:52:39.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:39.404+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.405245+0000) 2020-02-05T23:52:39.406 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:39.404+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.405608+0000) 2020-02-05T23:52:39.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:39.404+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.406277+0000) 2020-02-05T23:52:39.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:39.404+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.406369+0000) 2020-02-05T23:52:39.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:39.405+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.406680+0000) 2020-02-05T23:52:39.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:39.405+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.406935+0000) 2020-02-05T23:52:39.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:39.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.411175+0000) 2020-02-05T23:52:39.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:39.416+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:39.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:39.594+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:39.594565+0000) 2020-02-05T23:52:40.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:40.440+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:40.441814+0000) 2020-02-05T23:52:40.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:40.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:40.702892+0000) 2020-02-05T23:52:41.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:41.296+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:41.297272+0000) 2020-02-05T23:52:42.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:42.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:42.808045+0000) 2020-02-05T23:52:43.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:43.094+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:43.094556+0000) 2020-02-05T23:52:43.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:43.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:43.971310+0000) 2020-02-05T23:52:43.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:43.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:43.971310+0000) 2020-02-05T23:52:44.425 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:44.422+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.423780+0000) 2020-02-05T23:52:44.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:44.423+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.424294+0000) 2020-02-05T23:52:44.426 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:44.423+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.424994+0000) 2020-02-05T23:52:44.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:44.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.425424+0000) 2020-02-05T23:52:44.427 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:44.424+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.425514+0000) 2020-02-05T23:52:44.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:44.424+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.425982+0000) 2020-02-05T23:52:44.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:44.425+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.426708+0000) 2020-02-05T23:52:44.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:44.425+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.426749+0000) 2020-02-05T23:52:44.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:44.425+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.426831+0000) 2020-02-05T23:52:44.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:44.425+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.427331+0000) 2020-02-05T23:52:44.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:44.426+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.427945+0000) 2020-02-05T23:52:44.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:44.427+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.428530+0000) 2020-02-05T23:52:44.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:44.428+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.429557+0000) 2020-02-05T23:52:44.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:44.431+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:44.432946+0000) 2020-02-05T23:52:44.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:44.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157481 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:46.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:46.070+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:46.071683+0000) 2020-02-05T23:52:46.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:46.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:46.951750+0000) 2020-02-05T23:52:47.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:47.210+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:47.211491+0000) 2020-02-05T23:52:47.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:47.210+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:47.211491+0000) 2020-02-05T23:52:47.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:47.210+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:47.211629+0000) 2020-02-05T23:52:49.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:49.208+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.209740+0000) 2020-02-05T23:52:49.446 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:49.444+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.445071+0000) 2020-02-05T23:52:49.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:49.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.446501+0000) 2020-02-05T23:52:49.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:49.444+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.446276+0000) 2020-02-05T23:52:49.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:49.447+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.447267+0000) 2020-02-05T23:52:49.449 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:49.447+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.447779+0000) 2020-02-05T23:52:49.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:49.446+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.448262+0000) 2020-02-05T23:52:49.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:49.447+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.448851+0000) 2020-02-05T23:52:49.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:49.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.449164+0000) 2020-02-05T23:52:49.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:49.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.449580+0000) 2020-02-05T23:52:49.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:49.448+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.450151+0000) 2020-02-05T23:52:49.451 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:49.449+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.450781+0000) 2020-02-05T23:52:49.452 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:49.449+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.451133+0000) 2020-02-05T23:52:49.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:49.450+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.451656+0000) 2020-02-05T23:52:49.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:49.454+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.455721+0000) 2020-02-05T23:52:49.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:49.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157523 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:49.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:49.594+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:49.594778+0000) 2020-02-05T23:52:50.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:50.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:50.442116+0000) 2020-02-05T23:52:50.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:50.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:50.703127+0000) 2020-02-05T23:52:51.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:51.305+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:51.305298+0000) 2020-02-05T23:52:52.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:52.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:52.808313+0000) 2020-02-05T23:52:53.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:53.094+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:53.094792+0000) 2020-02-05T23:52:53.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:53.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:53.971598+0000) 2020-02-05T23:52:53.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:53.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:53.971600+0000) 2020-02-05T23:52:54.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:54.467+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.467204+0000) 2020-02-05T23:52:54.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:54.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.466692+0000) 2020-02-05T23:52:54.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:54.467+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.467841+0000) 2020-02-05T23:52:54.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:54.468+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.468274+0000) 2020-02-05T23:52:54.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:54.469+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.469650+0000) 2020-02-05T23:52:54.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:54.469+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.470680+0000) 2020-02-05T23:52:54.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:54.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.471012+0000) 2020-02-05T23:52:54.473 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:54.469+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.471277+0000) 2020-02-05T23:52:54.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:54.471+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.471653+0000) 2020-02-05T23:52:54.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:54.470+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.472338+0000) 2020-02-05T23:52:54.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:54.471+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.472942+0000) 2020-02-05T23:52:54.474 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:54.473+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.473464+0000) 2020-02-05T23:52:54.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:54.472+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.473836+0000) 2020-02-05T23:52:54.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:54.475+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:54.475662+0000) 2020-02-05T23:52:54.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:54.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:56.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:56.071+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:56.071941+0000) 2020-02-05T23:52:56.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:56.950+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:56.951975+0000) 2020-02-05T23:52:57.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:57.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:57.211812+0000) 2020-02-05T23:52:57.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:57.211+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:57.211812+0000) 2020-02-05T23:52:57.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:57.211+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:57.211883+0000) 2020-02-05T23:52:58.630 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:52:58.636 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:52:59.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:59.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.210009+0000) 2020-02-05T23:52:59.493 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:52:59.490+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.491331+0000) 2020-02-05T23:52:59.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:52:59.490+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.491705+0000) 2020-02-05T23:52:59.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:52:59.491+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.492620+0000) 2020-02-05T23:52:59.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:52:59.492+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.493407+0000) 2020-02-05T23:52:59.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:52:59.492+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.493975+0000) 2020-02-05T23:52:59.495 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:52:59.493+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.494438+0000) 2020-02-05T23:52:59.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:52:59.493+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.494997+0000) 2020-02-05T23:52:59.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:59.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.495368+0000) 2020-02-05T23:52:59.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:52:59.494+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.496038+0000) 2020-02-05T23:52:59.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:59.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.496325+0000) 2020-02-05T23:52:59.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:52:59.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.496836+0000) 2020-02-05T23:52:59.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:52:59.496+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.497507+0000) 2020-02-05T23:52:59.499 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:52:59.496+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.497062+0000) 2020-02-05T23:52:59.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:59.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.498894+0000) 2020-02-05T23:52:59.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:52:59.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:52:59.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:52:59.594+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:52:59.595011+0000) 2020-02-05T23:53:00.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:00.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:00.442328+0000) 2020-02-05T23:53:00.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:00.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:00.703350+0000) 2020-02-05T23:53:01.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:01.312+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:01.313852+0000) 2020-02-05T23:53:02.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:02.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:02.808547+0000) 2020-02-05T23:53:03.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:03.094+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:03.095025+0000) 2020-02-05T23:53:03.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:03.970+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:03.971844+0000) 2020-02-05T23:53:03.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:03.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:03.971844+0000) 2020-02-05T23:53:04.518 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:04.515+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.516846+0000) 2020-02-05T23:53:04.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:04.515+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.516957+0000) 2020-02-05T23:53:04.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:04.516+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.517580+0000) 2020-02-05T23:53:04.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:04.516+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.517824+0000) 2020-02-05T23:53:04.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:04.517+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.518716+0000) 2020-02-05T23:53:04.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:04.517+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.519123+0000) 2020-02-05T23:53:04.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:04.518+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.519328+0000) 2020-02-05T23:53:04.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:04.518+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.520226+0000) 2020-02-05T23:53:04.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:04.519+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.520556+0000) 2020-02-05T23:53:04.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:04.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.521533+0000) 2020-02-05T23:53:04.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:04.520+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.521655+0000) 2020-02-05T23:53:04.523 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:04.520+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.522104+0000) 2020-02-05T23:53:04.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:04.520+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.521950+0000) 2020-02-05T23:53:04.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:04.524+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:04.525545+0000) 2020-02-05T23:53:04.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:04.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157641 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:06.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:06.071+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:06.072205+0000) 2020-02-05T23:53:06.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:06.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:06.952210+0000) 2020-02-05T23:53:07.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:07.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:07.212115+0000) 2020-02-05T23:53:07.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:07.211+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:07.212147+0000) 2020-02-05T23:53:07.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:07.211+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:07.212115+0000) 2020-02-05T23:53:09.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:09.209+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.210261+0000) 2020-02-05T23:53:09.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:09.539+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.540658+0000) 2020-02-05T23:53:09.542 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:09.539+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.541347+0000) 2020-02-05T23:53:09.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:09.540+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.541581+0000) 2020-02-05T23:53:09.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:09.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.542001+0000) 2020-02-05T23:53:09.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:09.541+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.542545+0000) 2020-02-05T23:53:09.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:09.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.542626+0000) 2020-02-05T23:53:09.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:09.542+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.543320+0000) 2020-02-05T23:53:09.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:09.542+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.543756+0000) 2020-02-05T23:53:09.545 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:09.543+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.544367+0000) 2020-02-05T23:53:09.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:09.543+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.544862+0000) 2020-02-05T23:53:09.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:09.544+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.545617+0000) 2020-02-05T23:53:09.547 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:09.544+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.545876+0000) 2020-02-05T23:53:09.547 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:09.544+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.546245+0000) 2020-02-05T23:53:09.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:09.550+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.551331+0000) 2020-02-05T23:53:09.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:09.555+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157683 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:09.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:09.594+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:09.595269+0000) 2020-02-05T23:53:10.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:10.441+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:10.442549+0000) 2020-02-05T23:53:10.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:10.702+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:10.703582+0000) 2020-02-05T23:53:11.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:11.323+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:11.323965+0000) 2020-02-05T23:53:12.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:12.807+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:12.808727+0000) 2020-02-05T23:53:13.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:13.094+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:13.095327+0000) 2020-02-05T23:53:13.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:13.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:13.972130+0000) 2020-02-05T23:53:13.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:13.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:13.972130+0000) 2020-02-05T23:53:14.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:14.561+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.562712+0000) 2020-02-05T23:53:14.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:14.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.564036+0000) 2020-02-05T23:53:14.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:14.563+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.564376+0000) 2020-02-05T23:53:14.567 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:14.561+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.563305+0000) 2020-02-05T23:53:14.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:14.563+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.564832+0000) 2020-02-05T23:53:14.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:14.564+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.566331+0000) 2020-02-05T23:53:14.569 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:14.565+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.566651+0000) 2020-02-05T23:53:14.569 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:14.566+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.567460+0000) 2020-02-05T23:53:14.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:14.565+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.567107+0000) 2020-02-05T23:53:14.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:14.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.567268+0000) 2020-02-05T23:53:14.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:14.567+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.568524+0000) 2020-02-05T23:53:14.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:14.567+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.569102+0000) 2020-02-05T23:53:14.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:14.569+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.570876+0000) 2020-02-05T23:53:14.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:14.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:14.572228+0000) 2020-02-05T23:53:14.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:14.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157725 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:16.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:16.071+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:16.072467+0000) 2020-02-05T23:53:16.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:16.951+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:16.952436+0000) 2020-02-05T23:53:17.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:17.211+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:17.212423+0000) 2020-02-05T23:53:17.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:17.211+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:17.212403+0000) 2020-02-05T23:53:17.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:17.211+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:17.212422+0000) 2020-02-05T23:53:19.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:19.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.210524+0000) 2020-02-05T23:53:19.587 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:19.585+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.586199+0000) 2020-02-05T23:53:19.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:19.585+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.587149+0000) 2020-02-05T23:53:19.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:19.586+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.587573+0000) 2020-02-05T23:53:19.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:19.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.588253+0000) 2020-02-05T23:53:19.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:19.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.588750+0000) 2020-02-05T23:53:19.590 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:19.587+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.588688+0000) 2020-02-05T23:53:19.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:19.588+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.589294+0000) 2020-02-05T23:53:19.591 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:19.588+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.590028+0000) 2020-02-05T23:53:19.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:19.589+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.590900+0000) 2020-02-05T23:53:19.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:19.589+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.591276+0000) 2020-02-05T23:53:19.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:19.590+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.592077+0000) 2020-02-05T23:53:19.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:19.591+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.592448+0000) 2020-02-05T23:53:19.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:19.591+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.592400+0000) 2020-02-05T23:53:19.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:19.594+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.595458+0000) 2020-02-05T23:53:19.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:19.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:19.597678+0000) 2020-02-05T23:53:19.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:19.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:20.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:20.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:20.442783+0000) 2020-02-05T23:53:20.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:20.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:20.703817+0000) 2020-02-05T23:53:21.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:21.331+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:21.332205+0000) 2020-02-05T23:53:22.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:22.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:22.808976+0000) 2020-02-05T23:53:23.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:23.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:23.095554+0000) 2020-02-05T23:53:23.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:23.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:23.972394+0000) 2020-02-05T23:53:23.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:23.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:23.972393+0000) 2020-02-05T23:53:24.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:24.608+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.609844+0000) 2020-02-05T23:53:24.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:24.609+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.610582+0000) 2020-02-05T23:53:24.612 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:24.609+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.611042+0000) 2020-02-05T23:53:24.612 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:24.610+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.611348+0000) 2020-02-05T23:53:24.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:24.610+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.611255+0000) 2020-02-05T23:53:24.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:24.610+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.611932+0000) 2020-02-05T23:53:24.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:24.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.612347+0000) 2020-02-05T23:53:24.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:24.612+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.612625+0000) 2020-02-05T23:53:24.614 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:24.612+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.613144+0000) 2020-02-05T23:53:24.615 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:24.612+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.613244+0000) 2020-02-05T23:53:24.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:24.613+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.612855+0000) 2020-02-05T23:53:24.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:24.612+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.613848+0000) 2020-02-05T23:53:24.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:24.614+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.614419+0000) 2020-02-05T23:53:24.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:24.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:24.619925+0000) 2020-02-05T23:53:24.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:24.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:26.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:26.072+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:26.072715+0000) 2020-02-05T23:53:26.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:26.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:26.952684+0000) 2020-02-05T23:53:27.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:27.212+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:27.212747+0000) 2020-02-05T23:53:27.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:27.212+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:27.212737+0000) 2020-02-05T23:53:27.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:27.212+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:27.212747+0000) 2020-02-05T23:53:28.767 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:53:28.774 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:53:29.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:29.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.210786+0000) 2020-02-05T23:53:29.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:29.595+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.595651+0000) 2020-02-05T23:53:29.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:29.629+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.630521+0000) 2020-02-05T23:53:29.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:29.630+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.631939+0000) 2020-02-05T23:53:29.634 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:29.631+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.632799+0000) 2020-02-05T23:53:29.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:29.633+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.633756+0000) 2020-02-05T23:53:29.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:29.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.634484+0000) 2020-02-05T23:53:29.636 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:29.633+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.634982+0000) 2020-02-05T23:53:29.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:29.635+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.635862+0000) 2020-02-05T23:53:29.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:29.634+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.636056+0000) 2020-02-05T23:53:29.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:29.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.636770+0000) 2020-02-05T23:53:29.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:29.637+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.637713+0000) 2020-02-05T23:53:29.639 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:29.637+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.637988+0000) 2020-02-05T23:53:29.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:29.637+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.638326+0000) 2020-02-05T23:53:29.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:29.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.640051+0000) 2020-02-05T23:53:29.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:29.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:29.640317+0000) 2020-02-05T23:53:29.650 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:29.648+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157849 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:30.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:30.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:30.442994+0000) 2020-02-05T23:53:30.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:30.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:30.704053+0000) 2020-02-05T23:53:31.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:31.339+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:31.340515+0000) 2020-02-05T23:53:32.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:32.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:32.809225+0000) 2020-02-05T23:53:33.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:33.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:33.095744+0000) 2020-02-05T23:53:33.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:33.971+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:33.972679+0000) 2020-02-05T23:53:33.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:33.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:33.972679+0000) 2020-02-05T23:53:34.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:34.655+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.655684+0000) 2020-02-05T23:53:34.657 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:34.655+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.656168+0000) 2020-02-05T23:53:34.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:34.655+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.656074+0000) 2020-02-05T23:53:34.658 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:34.655+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.656802+0000) 2020-02-05T23:53:34.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:34.656+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.657527+0000) 2020-02-05T23:53:34.659 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:34.656+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.657904+0000) 2020-02-05T23:53:34.659 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:34.657+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.658235+0000) 2020-02-05T23:53:34.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:34.657+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.658503+0000) 2020-02-05T23:53:34.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:34.658+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.659507+0000) 2020-02-05T23:53:34.661 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:34.659+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.659917+0000) 2020-02-05T23:53:34.661 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:34.659+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.660282+0000) 2020-02-05T23:53:34.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:34.659+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.660750+0000) 2020-02-05T23:53:34.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:34.660+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.661043+0000) 2020-02-05T23:53:34.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:34.663+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:34.665006+0000) 2020-02-05T23:53:34.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:34.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157891 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:36.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:36.072+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:36.072964+0000) 2020-02-05T23:53:36.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:36.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:36.952927+0000) 2020-02-05T23:53:37.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:37.212+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:37.212982+0000) 2020-02-05T23:53:37.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:37.212+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:37.213029+0000) 2020-02-05T23:53:37.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:37.212+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:37.213024+0000) 2020-02-05T23:53:39.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:39.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.211036+0000) 2020-02-05T23:53:39.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:39.595+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.595893+0000) 2020-02-05T23:53:39.680 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:39.677+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.678835+0000) 2020-02-05T23:53:39.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:39.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.679656+0000) 2020-02-05T23:53:39.681 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:39.679+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.680210+0000) 2020-02-05T23:53:39.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:39.679+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.680359+0000) 2020-02-05T23:53:39.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:39.680+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.680883+0000) 2020-02-05T23:53:39.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:39.680+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.681230+0000) 2020-02-05T23:53:39.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:39.680+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.681333+0000) 2020-02-05T23:53:39.683 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:39.680+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.681721+0000) 2020-02-05T23:53:39.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:39.681+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.682286+0000) 2020-02-05T23:53:39.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:39.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.683280+0000) 2020-02-05T23:53:39.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:39.683+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.683828+0000) 2020-02-05T23:53:39.685 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:39.683+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.684093+0000) 2020-02-05T23:53:39.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:39.684+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.685286+0000) 2020-02-05T23:53:39.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:39.687+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:39.688136+0000) 2020-02-05T23:53:39.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:39.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:40.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:40.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:40.443225+0000) 2020-02-05T23:53:40.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:40.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:40.704297+0000) 2020-02-05T23:53:41.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:41.348+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:41.348699+0000) 2020-02-05T23:53:42.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:42.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:42.809453+0000) 2020-02-05T23:53:43.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:43.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:43.095974+0000) 2020-02-05T23:53:43.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:43.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:43.972946+0000) 2020-02-05T23:53:43.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:43.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:43.972946+0000) 2020-02-05T23:53:44.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:44.699+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.700811+0000) 2020-02-05T23:53:44.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:44.700+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.701449+0000) 2020-02-05T23:53:44.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:44.701+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.702118+0000) 2020-02-05T23:53:44.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:44.703+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.703430+0000) 2020-02-05T23:53:44.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:44.703+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.704128+0000) 2020-02-05T23:53:44.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:44.704+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.705381+0000) 2020-02-05T23:53:44.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:44.703+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.703945+0000) 2020-02-05T23:53:44.710 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:44.704+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.704543+0000) 2020-02-05T23:53:44.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:44.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.705104+0000) 2020-02-05T23:53:44.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:44.705+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.706105+0000) 2020-02-05T23:53:44.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:44.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.706881+0000) 2020-02-05T23:53:44.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:44.706+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.707393+0000) 2020-02-05T23:53:44.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:44.707+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.707801+0000) 2020-02-05T23:53:44.732 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:44.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 157973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:44.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:44.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:44.710794+0000) 2020-02-05T23:53:46.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:46.072+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:46.073207+0000) 2020-02-05T23:53:46.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:46.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:46.953181+0000) 2020-02-05T23:53:47.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:47.213+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:47.213225+0000) 2020-02-05T23:53:47.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:47.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:47.213298+0000) 2020-02-05T23:53:47.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:47.213+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:47.213316+0000) 2020-02-05T23:53:49.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:49.210+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.211301+0000) 2020-02-05T23:53:49.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:49.595+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.596180+0000) 2020-02-05T23:53:49.727 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:49.724+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.725683+0000) 2020-02-05T23:53:49.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:49.725+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.726332+0000) 2020-02-05T23:53:49.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:49.725+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.726656+0000) 2020-02-05T23:53:49.729 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:49.727+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.728245+0000) 2020-02-05T23:53:49.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:49.727+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.728400+0000) 2020-02-05T23:53:49.730 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:49.728+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.728953+0000) 2020-02-05T23:53:49.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:49.727+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.729054+0000) 2020-02-05T23:53:49.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:49.729+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.729644+0000) 2020-02-05T23:53:49.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:49.729+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.730070+0000) 2020-02-05T23:53:49.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:49.730+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.730968+0000) 2020-02-05T23:53:49.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:49.730+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.730975+0000) 2020-02-05T23:53:49.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:49.730+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.731604+0000) 2020-02-05T23:53:49.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:49.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.733760+0000) 2020-02-05T23:53:49.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:49.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:49.733947+0000) 2020-02-05T23:53:49.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:49.744+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158015 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:50.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:50.443+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:50.443502+0000) 2020-02-05T23:53:50.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:50.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:50.704528+0000) 2020-02-05T23:53:51.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:51.355+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:51.356586+0000) 2020-02-05T23:53:52.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:52.808+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:52.809682+0000) 2020-02-05T23:53:53.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:53.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:53.096179+0000) 2020-02-05T23:53:53.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:53.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:53.973248+0000) 2020-02-05T23:53:53.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:53.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:53.973249+0000) 2020-02-05T23:53:54.752 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:54.749+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.750712+0000) 2020-02-05T23:53:54.752 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:54.750+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.751454+0000) 2020-02-05T23:53:54.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:54.750+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.751880+0000) 2020-02-05T23:53:54.754 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:54.752+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.752712+0000) 2020-02-05T23:53:54.754 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:54.752+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.753152+0000) 2020-02-05T23:53:54.754 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:54.753+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.753530+0000) 2020-02-05T23:53:54.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:54.753+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.754450+0000) 2020-02-05T23:53:54.756 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:54.753+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.754271+0000) 2020-02-05T23:53:54.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:54.754+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.755494+0000) 2020-02-05T23:53:54.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:54.755+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.756300+0000) 2020-02-05T23:53:54.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:54.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.756657+0000) 2020-02-05T23:53:54.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:54.757+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.757861+0000) 2020-02-05T23:53:54.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:54.759+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.760565+0000) 2020-02-05T23:53:54.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:54.759+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:54.760808+0000) 2020-02-05T23:53:54.767 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:54.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:53:56.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:56.073+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:56.073488+0000) 2020-02-05T23:53:56.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:56.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:56.953416+0000) 2020-02-05T23:53:57.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:57.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:57.213556+0000) 2020-02-05T23:53:57.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:57.213+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:57.213493+0000) 2020-02-05T23:53:57.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:57.213+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:57.213620+0000) 2020-02-05T23:53:58.905 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:53:58.911 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:53:59.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:59.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.211588+0000) 2020-02-05T23:53:59.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:59.595+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.596452+0000) 2020-02-05T23:53:59.773 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:53:59.770+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.771528+0000) 2020-02-05T23:53:59.774 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:53:59.771+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.772394+0000) 2020-02-05T23:53:59.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:59.772+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.773649+0000) 2020-02-05T23:53:59.776 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:53:59.772+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.774279+0000) 2020-02-05T23:53:59.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:53:59.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.775053+0000) 2020-02-05T23:53:59.777 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:53:59.774+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.775707+0000) 2020-02-05T23:53:59.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:59.775+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.776166+0000) 2020-02-05T23:53:59.777 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:53:59.775+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.776598+0000) 2020-02-05T23:53:59.778 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:53:59.775+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.776739+0000) 2020-02-05T23:53:59.779 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:53:59.776+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.777156+0000) 2020-02-05T23:53:59.779 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:53:59.776+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.777682+0000) 2020-02-05T23:53:59.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:53:59.777+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.778328+0000) 2020-02-05T23:53:59.780 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:53:59.777+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.779077+0000) 2020-02-05T23:53:59.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:53:59.778+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:53:59.780024+0000) 2020-02-05T23:53:59.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:53:59.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:00.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:00.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:00.443741+0000) 2020-02-05T23:54:00.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:00.703+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:00.704731+0000) 2020-02-05T23:54:01.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:01.363+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:01.364188+0000) 2020-02-05T23:54:02.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:02.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:02.809909+0000) 2020-02-05T23:54:03.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:03.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:03.096435+0000) 2020-02-05T23:54:03.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:03.972+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:03.973499+0000) 2020-02-05T23:54:03.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:03.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:03.973499+0000) 2020-02-05T23:54:04.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:04.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.795474+0000) 2020-02-05T23:54:04.798 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:04.795+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.796669+0000) 2020-02-05T23:54:04.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:04.796+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.798103+0000) 2020-02-05T23:54:04.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:04.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.798831+0000) 2020-02-05T23:54:04.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:04.798+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.799430+0000) 2020-02-05T23:54:04.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:04.798+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.800069+0000) 2020-02-05T23:54:04.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:04.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.800708+0000) 2020-02-05T23:54:04.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:04.799+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.800541+0000) 2020-02-05T23:54:04.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:04.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.801581+0000) 2020-02-05T23:54:04.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:04.800+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.801979+0000) 2020-02-05T23:54:04.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:04.800+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.802385+0000) 2020-02-05T23:54:04.804 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:04.801+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.802989+0000) 2020-02-05T23:54:04.804 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:04.802+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.803606+0000) 2020-02-05T23:54:04.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:04.806+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:04.807830+0000) 2020-02-05T23:54:04.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:04.814+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:06.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:06.072+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:06.073681+0000) 2020-02-05T23:54:06.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:06.952+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:06.953646+0000) 2020-02-05T23:54:07.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:07.212+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:07.213799+0000) 2020-02-05T23:54:07.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:07.212+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:07.213782+0000) 2020-02-05T23:54:07.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:07.212+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:07.213904+0000) 2020-02-05T23:54:09.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:09.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.211963+0000) 2020-02-05T23:54:09.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:09.595+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.596674+0000) 2020-02-05T23:54:09.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:09.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.821809+0000) 2020-02-05T23:54:09.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:09.821+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.822456+0000) 2020-02-05T23:54:09.824 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:09.821+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.822825+0000) 2020-02-05T23:54:09.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:09.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.823211+0000) 2020-02-05T23:54:09.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:09.822+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.823555+0000) 2020-02-05T23:54:09.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:09.822+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.824330+0000) 2020-02-05T23:54:09.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:09.823+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.824812+0000) 2020-02-05T23:54:09.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:09.824+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.825119+0000) 2020-02-05T23:54:09.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:09.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.825244+0000) 2020-02-05T23:54:09.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:09.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.825718+0000) 2020-02-05T23:54:09.827 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:09.824+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.826041+0000) 2020-02-05T23:54:09.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:09.825+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.827116+0000) 2020-02-05T23:54:09.828 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:09.825+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.827275+0000) 2020-02-05T23:54:09.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:09.830+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:09.831676+0000) 2020-02-05T23:54:09.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:09.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:10.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:10.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:10.443942+0000) 2020-02-05T23:54:10.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:10.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:10.704944+0000) 2020-02-05T23:54:11.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:11.373+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:11.374434+0000) 2020-02-05T23:54:12.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:12.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:12.810137+0000) 2020-02-05T23:54:13.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:13.096+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:13.096671+0000) 2020-02-05T23:54:13.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:13.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:13.973758+0000) 2020-02-05T23:54:13.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:13.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:13.973758+0000) 2020-02-05T23:54:14.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:14.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.845179+0000) 2020-02-05T23:54:14.848 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:14.845+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.846301+0000) 2020-02-05T23:54:14.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:14.845+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.846769+0000) 2020-02-05T23:54:14.849 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:14.846+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.847413+0000) 2020-02-05T23:54:14.849 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:14.846+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.847742+0000) 2020-02-05T23:54:14.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:14.847+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.848312+0000) 2020-02-05T23:54:14.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:14.847+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.848412+0000) 2020-02-05T23:54:14.850 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:14.848+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.849593+0000) 2020-02-05T23:54:14.851 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:14.847+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.849181+0000) 2020-02-05T23:54:14.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:14.849+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.850702+0000) 2020-02-05T23:54:14.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:14.850+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.851137+0000) 2020-02-05T23:54:14.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:14.850+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.851658+0000) 2020-02-05T23:54:14.853 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:14.850+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.851875+0000) 2020-02-05T23:54:14.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:14.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:14.857571+0000) 2020-02-05T23:54:14.864 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:14.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:16.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:16.073+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:16.073921+0000) 2020-02-05T23:54:16.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:16.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:16.953884+0000) 2020-02-05T23:54:17.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:17.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:17.214040+0000) 2020-02-05T23:54:17.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:17.213+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:17.214031+0000) 2020-02-05T23:54:17.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:17.213+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:17.214170+0000) 2020-02-05T23:54:19.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:19.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.212225+0000) 2020-02-05T23:54:19.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:19.596+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.596934+0000) 2020-02-05T23:54:19.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:19.868+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.869774+0000) 2020-02-05T23:54:19.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:19.868+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.870289+0000) 2020-02-05T23:54:19.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:19.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.870528+0000) 2020-02-05T23:54:19.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:19.869+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.871168+0000) 2020-02-05T23:54:19.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:19.870+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.871541+0000) 2020-02-05T23:54:19.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:19.871+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.872251+0000) 2020-02-05T23:54:19.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:19.871+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.872493+0000) 2020-02-05T23:54:19.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:19.871+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.872768+0000) 2020-02-05T23:54:19.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:19.872+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.873422+0000) 2020-02-05T23:54:19.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:19.872+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.873940+0000) 2020-02-05T23:54:19.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:19.873+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.874200+0000) 2020-02-05T23:54:19.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:19.874+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.875155+0000) 2020-02-05T23:54:19.876 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:19.874+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.875615+0000) 2020-02-05T23:54:19.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:19.880+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:19.881259+0000) 2020-02-05T23:54:19.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:19.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:20.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:20.443+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:20.444178+0000) 2020-02-05T23:54:20.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:20.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:20.705150+0000) 2020-02-05T23:54:21.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:21.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:21.382937+0000) 2020-02-05T23:54:22.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:22.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:22.810407+0000) 2020-02-05T23:54:23.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:23.095+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:23.096895+0000) 2020-02-05T23:54:23.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:23.973+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:23.974034+0000) 2020-02-05T23:54:23.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:23.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:23.974034+0000) 2020-02-05T23:54:24.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:24.890+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.892157+0000) 2020-02-05T23:54:24.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:24.892+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.893139+0000) 2020-02-05T23:54:24.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:24.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.893557+0000) 2020-02-05T23:54:24.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:24.894+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.895715+0000) 2020-02-05T23:54:24.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:24.894+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.896243+0000) 2020-02-05T23:54:24.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:24.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.897180+0000) 2020-02-05T23:54:24.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:24.895+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.896937+0000) 2020-02-05T23:54:24.899 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:24.896+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.897902+0000) 2020-02-05T23:54:24.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:24.897+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.898305+0000) 2020-02-05T23:54:24.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:24.897+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.898833+0000) 2020-02-05T23:54:24.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:24.897+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.899268+0000) 2020-02-05T23:54:24.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:24.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.899150+0000) 2020-02-05T23:54:24.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:24.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.899484+0000) 2020-02-05T23:54:24.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:24.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:24.899605+0000) 2020-02-05T23:54:24.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:24.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:26.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:26.073+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:26.074225+0000) 2020-02-05T23:54:26.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:26.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:26.954155+0000) 2020-02-05T23:54:27.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:27.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:27.214332+0000) 2020-02-05T23:54:27.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:27.213+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:27.214345+0000) 2020-02-05T23:54:27.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:27.213+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:27.214501+0000) 2020-02-05T23:54:29.061 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:54:29.067 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:54:29.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:29.211+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.212490+0000) 2020-02-05T23:54:29.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:29.596+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.597205+0000) 2020-02-05T23:54:29.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:29.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.916958+0000) 2020-02-05T23:54:29.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:29.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.918389+0000) 2020-02-05T23:54:29.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:29.917+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.918825+0000) 2020-02-05T23:54:29.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:29.918+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.919487+0000) 2020-02-05T23:54:29.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:29.918+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.919741+0000) 2020-02-05T23:54:29.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:29.918+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.920326+0000) 2020-02-05T23:54:29.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:29.919+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.920852+0000) 2020-02-05T23:54:29.922 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:29.920+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.921695+0000) 2020-02-05T23:54:29.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:29.921+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.922690+0000) 2020-02-05T23:54:29.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:29.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.923177+0000) 2020-02-05T23:54:29.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:29.922+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.923352+0000) 2020-02-05T23:54:29.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:29.921+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.923357+0000) 2020-02-05T23:54:29.925 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:29.922+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.923796+0000) 2020-02-05T23:54:29.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:29.926+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:29.927517+0000) 2020-02-05T23:54:29.934 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:29.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158350 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:30.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:30.442+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:30.444402+0000) 2020-02-05T23:54:30.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:30.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:30.705364+0000) 2020-02-05T23:54:31.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:31.390+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:31.390871+0000) 2020-02-05T23:54:32.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:32.809+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:32.810652+0000) 2020-02-05T23:54:33.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:33.096+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:33.097125+0000) 2020-02-05T23:54:33.975 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:33.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:33.974356+0000) 2020-02-05T23:54:33.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:33.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:33.974356+0000) 2020-02-05T23:54:34.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:34.937+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.938615+0000) 2020-02-05T23:54:34.941 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:34.938+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.939913+0000) 2020-02-05T23:54:34.941 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:34.939+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.940421+0000) 2020-02-05T23:54:34.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:34.940+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.941201+0000) 2020-02-05T23:54:34.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:34.940+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.941728+0000) 2020-02-05T23:54:34.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:34.941+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.942329+0000) 2020-02-05T23:54:34.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:34.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.942371+0000) 2020-02-05T23:54:34.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:34.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.942622+0000) 2020-02-05T23:54:34.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:34.942+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.943397+0000) 2020-02-05T23:54:34.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:34.941+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.943267+0000) 2020-02-05T23:54:34.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:34.942+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.943740+0000) 2020-02-05T23:54:34.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:34.942+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.944369+0000) 2020-02-05T23:54:34.946 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:34.943+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.944577+0000) 2020-02-05T23:54:34.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:34.949+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:34.950634+0000) 2020-02-05T23:54:34.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:34.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158392 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:36.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:36.073+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:36.074485+0000) 2020-02-05T23:54:36.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:36.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:36.954338+0000) 2020-02-05T23:54:37.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:37.213+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:37.214566+0000) 2020-02-05T23:54:37.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:37.213+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:37.214597+0000) 2020-02-05T23:54:37.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:37.213+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:37.214741+0000) 2020-02-05T23:54:39.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:39.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.212757+0000) 2020-02-05T23:54:39.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:39.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.597445+0000) 2020-02-05T23:54:39.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:39.962+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.962835+0000) 2020-02-05T23:54:39.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:39.961+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.962723+0000) 2020-02-05T23:54:39.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:39.963+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.963985+0000) 2020-02-05T23:54:39.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:39.963+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.965071+0000) 2020-02-05T23:54:39.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:39.965+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.965484+0000) 2020-02-05T23:54:39.967 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:39.964+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.965754+0000) 2020-02-05T23:54:39.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:39.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.966006+0000) 2020-02-05T23:54:39.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:39.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.966487+0000) 2020-02-05T23:54:39.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:39.964+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.966203+0000) 2020-02-05T23:54:39.968 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:39.966+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.966876+0000) 2020-02-05T23:54:39.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:39.967+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.967820+0000) 2020-02-05T23:54:39.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:39.966+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.968200+0000) 2020-02-05T23:54:39.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:39.967+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.969282+0000) 2020-02-05T23:54:39.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:39.972+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:39.973971+0000) 2020-02-05T23:54:39.981 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:39.978+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:40.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:40.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:40.444676+0000) 2020-02-05T23:54:40.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:40.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:40.705571+0000) 2020-02-05T23:54:41.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:41.399+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:41.399339+0000) 2020-02-05T23:54:42.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:42.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:42.810878+0000) 2020-02-05T23:54:43.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:43.097+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:43.097396+0000) 2020-02-05T23:54:43.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:43.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:43.974634+0000) 2020-02-05T23:54:43.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:43.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:43.974633+0000) 2020-02-05T23:54:44.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:44.983+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.984767+0000) 2020-02-05T23:54:44.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:44.984+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.985405+0000) 2020-02-05T23:54:44.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:44.984+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.985935+0000) 2020-02-05T23:54:44.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:44.986+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.987990+0000) 2020-02-05T23:54:44.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:44.987+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.988746+0000) 2020-02-05T23:54:44.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:44.988+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.989219+0000) 2020-02-05T23:54:44.990 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:44.988+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.989637+0000) 2020-02-05T23:54:44.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:44.989+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.990286+0000) 2020-02-05T23:54:44.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:44.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.990642+0000) 2020-02-05T23:54:44.992 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:44.989+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.990867+0000) 2020-02-05T23:54:44.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:44.989+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.991336+0000) 2020-02-05T23:54:44.993 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:44.989+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.991097+0000) 2020-02-05T23:54:44.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:44.990+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.992039+0000) 2020-02-05T23:54:44.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:44.992+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:44.993933+0000) 2020-02-05T23:54:45.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:45.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158476 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:46.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:46.074+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:46.074738+0000) 2020-02-05T23:54:46.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:46.953+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:46.954604+0000) 2020-02-05T23:54:47.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:47.214+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:47.214814+0000) 2020-02-05T23:54:47.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:47.214+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:47.214846+0000) 2020-02-05T23:54:47.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:47.214+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:47.215085+0000) 2020-02-05T23:54:49.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:49.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:49.213026+0000) 2020-02-05T23:54:49.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:49.596+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:49.597694+0000) 2020-02-05T23:54:50.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:50.010+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.011149+0000) 2020-02-05T23:54:50.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:50.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.012138+0000) 2020-02-05T23:54:50.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:50.011+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.012662+0000) 2020-02-05T23:54:50.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:50.014+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.014676+0000) 2020-02-05T23:54:50.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:50.014+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.015203+0000) 2020-02-05T23:54:50.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:50.014+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.015272+0000) 2020-02-05T23:54:50.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:50.015+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.015816+0000) 2020-02-05T23:54:50.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:50.015+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.016219+0000) 2020-02-05T23:54:50.022 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:50.015+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.017117+0000) 2020-02-05T23:54:50.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:50.015+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.017196+0000) 2020-02-05T23:54:50.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:50.016+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.017576+0000) 2020-02-05T23:54:50.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:50.015+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.016739+0000) 2020-02-05T23:54:50.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:50.015+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.016972+0000) 2020-02-05T23:54:50.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:50.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.020001+0000) 2020-02-05T23:54:50.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:50.027+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:50.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:50.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.444911+0000) 2020-02-05T23:54:50.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:50.704+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:50.705817+0000) 2020-02-05T23:54:51.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:51.406+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:51.407665+0000) 2020-02-05T23:54:52.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:52.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:52.811146+0000) 2020-02-05T23:54:53.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:53.096+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:53.097637+0000) 2020-02-05T23:54:53.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:53.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:53.974907+0000) 2020-02-05T23:54:53.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:53.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:53.974907+0000) 2020-02-05T23:54:55.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:55.034+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.034648+0000) 2020-02-05T23:54:55.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:55.035+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.035583+0000) 2020-02-05T23:54:55.038 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:54:55.035+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.036231+0000) 2020-02-05T23:54:55.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:55.035+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.036843+0000) 2020-02-05T23:54:55.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:54:55.036+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.037199+0000) 2020-02-05T23:54:55.040 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:54:55.037+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.037742+0000) 2020-02-05T23:54:55.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:54:55.036+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.037892+0000) 2020-02-05T23:54:55.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:55.037+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.038695+0000) 2020-02-05T23:54:55.041 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:54:55.038+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.039213+0000) 2020-02-05T23:54:55.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:55.038+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.040005+0000) 2020-02-05T23:54:55.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:55.040+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.040640+0000) 2020-02-05T23:54:55.043 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:55.040+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.041066+0000) 2020-02-05T23:54:55.043 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:55.040+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.041293+0000) 2020-02-05T23:54:55.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:55.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:55.044340+0000) 2020-02-05T23:54:55.053 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:54:55.051+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:54:56.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:54:56.074+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:56.075010+0000) 2020-02-05T23:54:56.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:54:56.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:56.954837+0000) 2020-02-05T23:54:57.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:54:57.214+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:57.215051+0000) 2020-02-05T23:54:57.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:54:57.214+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:57.215088+0000) 2020-02-05T23:54:57.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:54:57.214+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:57.215385+0000) 2020-02-05T23:54:59.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:54:59.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:59.213288+0000) 2020-02-05T23:54:59.256 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:54:59.263 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:54:59.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:54:59.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:54:59.597929+0000) 2020-02-05T23:55:00.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:00.056+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.057199+0000) 2020-02-05T23:55:00.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:00.057+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.058946+0000) 2020-02-05T23:55:00.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:00.058+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.059921+0000) 2020-02-05T23:55:00.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:00.060+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.060881+0000) 2020-02-05T23:55:00.062 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:00.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.061135+0000) 2020-02-05T23:55:00.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:00.060+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.061595+0000) 2020-02-05T23:55:00.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:00.061+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.061646+0000) 2020-02-05T23:55:00.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:00.060+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.061959+0000) 2020-02-05T23:55:00.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:00.062+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.062804+0000) 2020-02-05T23:55:00.064 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:00.063+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.063486+0000) 2020-02-05T23:55:00.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:00.062+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.063344+0000) 2020-02-05T23:55:00.065 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:00.063+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.063911+0000) 2020-02-05T23:55:00.065 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:00.063+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.064229+0000) 2020-02-05T23:55:00.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:00.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.068530+0000) 2020-02-05T23:55:00.078 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:00.077+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:00.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:00.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.445126+0000) 2020-02-05T23:55:00.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:00.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:00.706023+0000) 2020-02-05T23:55:01.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:01.414+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:01.415897+0000) 2020-02-05T23:55:02.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:02.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:02.811409+0000) 2020-02-05T23:55:03.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:03.097+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:03.097874+0000) 2020-02-05T23:55:03.976 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:03.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:03.975175+0000) 2020-02-05T23:55:03.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:03.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:03.975175+0000) 2020-02-05T23:55:05.084 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:05.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.083381+0000) 2020-02-05T23:55:05.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:05.084+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.084717+0000) 2020-02-05T23:55:05.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:05.084+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.085224+0000) 2020-02-05T23:55:05.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:05.084+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.085931+0000) 2020-02-05T23:55:05.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:05.085+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.086139+0000) 2020-02-05T23:55:05.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:05.085+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.086499+0000) 2020-02-05T23:55:05.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:05.086+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.087337+0000) 2020-02-05T23:55:05.088 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:05.086+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.087525+0000) 2020-02-05T23:55:05.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:05.087+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.088227+0000) 2020-02-05T23:55:05.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:05.087+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.088249+0000) 2020-02-05T23:55:05.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:05.087+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.088738+0000) 2020-02-05T23:55:05.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:05.088+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.089129+0000) 2020-02-05T23:55:05.090 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:05.088+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.089218+0000) 2020-02-05T23:55:05.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:05.091+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:05.092287+0000) 2020-02-05T23:55:05.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:05.100+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158641 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:06.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:06.074+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:06.075273+0000) 2020-02-05T23:55:06.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:06.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:06.955119+0000) 2020-02-05T23:55:07.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:07.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:07.215309+0000) 2020-02-05T23:55:07.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:07.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:07.215330+0000) 2020-02-05T23:55:07.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:07.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:07.215667+0000) 2020-02-05T23:55:09.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:09.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:09.213534+0000) 2020-02-05T23:55:09.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:09.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:09.598169+0000) 2020-02-05T23:55:10.107 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:10.104+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.105633+0000) 2020-02-05T23:55:10.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:10.105+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.106273+0000) 2020-02-05T23:55:10.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:10.107+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.108850+0000) 2020-02-05T23:55:10.111 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:10.109+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.110064+0000) 2020-02-05T23:55:10.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:10.110+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.111019+0000) 2020-02-05T23:55:10.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:10.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.110560+0000) 2020-02-05T23:55:10.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:10.111+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.112003+0000) 2020-02-05T23:55:10.113 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:10.111+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.112538+0000) 2020-02-05T23:55:10.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:10.111+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.112611+0000) 2020-02-05T23:55:10.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:10.111+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.112780+0000) 2020-02-05T23:55:10.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:10.112+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.112937+0000) 2020-02-05T23:55:10.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:10.113+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.113678+0000) 2020-02-05T23:55:10.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:10.111+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.113054+0000) 2020-02-05T23:55:10.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:10.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.118157+0000) 2020-02-05T23:55:10.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:10.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158683 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:10.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:10.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.445340+0000) 2020-02-05T23:55:10.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:10.705+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:10.706260+0000) 2020-02-05T23:55:11.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:11.425+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:11.425789+0000) 2020-02-05T23:55:12.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:12.810+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:12.811598+0000) 2020-02-05T23:55:13.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:13.097+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:13.098135+0000) 2020-02-05T23:55:13.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:13.974+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:13.975443+0000) 2020-02-05T23:55:13.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:13.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:13.975443+0000) 2020-02-05T23:55:15.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:15.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.129619+0000) 2020-02-05T23:55:15.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:15.129+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.130641+0000) 2020-02-05T23:55:15.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:15.130+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.131062+0000) 2020-02-05T23:55:15.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:15.131+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.131719+0000) 2020-02-05T23:55:15.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:15.131+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.132087+0000) 2020-02-05T23:55:15.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:15.132+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.132740+0000) 2020-02-05T23:55:15.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:15.132+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.133388+0000) 2020-02-05T23:55:15.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:15.132+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.133916+0000) 2020-02-05T23:55:15.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:15.133+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.134068+0000) 2020-02-05T23:55:15.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:15.134+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.134467+0000) 2020-02-05T23:55:15.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:15.134+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.134775+0000) 2020-02-05T23:55:15.137 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:15.134+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.135116+0000) 2020-02-05T23:55:15.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:15.134+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.135511+0000) 2020-02-05T23:55:15.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:15.137+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:15.139026+0000) 2020-02-05T23:55:15.150 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:15.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158724 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:16.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:16.075+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:16.075502+0000) 2020-02-05T23:55:16.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:16.954+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:16.955353+0000) 2020-02-05T23:55:17.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:17.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:17.215591+0000) 2020-02-05T23:55:17.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:17.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:17.215658+0000) 2020-02-05T23:55:17.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:17.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:17.215906+0000) 2020-02-05T23:55:19.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:19.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:19.213785+0000) 2020-02-05T23:55:19.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:19.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:19.598394+0000) 2020-02-05T23:55:20.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:20.152+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.153567+0000) 2020-02-05T23:55:20.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:20.153+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.154656+0000) 2020-02-05T23:55:20.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:20.154+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.154921+0000) 2020-02-05T23:55:20.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:20.155+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.155565+0000) 2020-02-05T23:55:20.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:20.155+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.156204+0000) 2020-02-05T23:55:20.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:20.155+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.156304+0000) 2020-02-05T23:55:20.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:20.156+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.157967+0000) 2020-02-05T23:55:20.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:20.157+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.158942+0000) 2020-02-05T23:55:20.161 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:20.158+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.158654+0000) 2020-02-05T23:55:20.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:20.158+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.158774+0000) 2020-02-05T23:55:20.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:20.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.160133+0000) 2020-02-05T23:55:20.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:20.159+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.160442+0000) 2020-02-05T23:55:20.162 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:20.159+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.159810+0000) 2020-02-05T23:55:20.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:20.161+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.163004+0000) 2020-02-05T23:55:20.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:20.172+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:20.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:20.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.445558+0000) 2020-02-05T23:55:20.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:20.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:20.706473+0000) 2020-02-05T23:55:21.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:21.434+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:21.434407+0000) 2020-02-05T23:55:22.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:22.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:22.811865+0000) 2020-02-05T23:55:23.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:23.098+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:23.098360+0000) 2020-02-05T23:55:23.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:23.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:23.975712+0000) 2020-02-05T23:55:23.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:23.975+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:23.975714+0000) 2020-02-05T23:55:25.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:25.176+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.177547+0000) 2020-02-05T23:55:25.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:25.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.178997+0000) 2020-02-05T23:55:25.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:25.178+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.179562+0000) 2020-02-05T23:55:25.181 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:25.179+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.180385+0000) 2020-02-05T23:55:25.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:25.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.180876+0000) 2020-02-05T23:55:25.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:25.180+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.180796+0000) 2020-02-05T23:55:25.183 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:25.181+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.181498+0000) 2020-02-05T23:55:25.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:25.182+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.182464+0000) 2020-02-05T23:55:25.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:25.182+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.182971+0000) 2020-02-05T23:55:25.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:25.183+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.185028+0000) 2020-02-05T23:55:25.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:25.183+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.184986+0000) 2020-02-05T23:55:25.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:25.184+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.185216+0000) 2020-02-05T23:55:25.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:25.185+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.185483+0000) 2020-02-05T23:55:25.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:25.187+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:25.188914+0000) 2020-02-05T23:55:25.198 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:25.196+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:26.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:26.075+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:26.075747+0000) 2020-02-05T23:55:26.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:26.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:26.955614+0000) 2020-02-05T23:55:27.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:27.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:27.215886+0000) 2020-02-05T23:55:27.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:27.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:27.215851+0000) 2020-02-05T23:55:27.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:27.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:27.216213+0000) 2020-02-05T23:55:29.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:29.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:29.214044+0000) 2020-02-05T23:55:29.444 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:55:29.451 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:55:29.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:29.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:29.598601+0000) 2020-02-05T23:55:30.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:30.199+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.200468+0000) 2020-02-05T23:55:30.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:30.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.201031+0000) 2020-02-05T23:55:30.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:30.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.202732+0000) 2020-02-05T23:55:30.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:30.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.203282+0000) 2020-02-05T23:55:30.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:30.203+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.204736+0000) 2020-02-05T23:55:30.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:30.203+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.205061+0000) 2020-02-05T23:55:30.206 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:30.203+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.204961+0000) 2020-02-05T23:55:30.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:30.204+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.205580+0000) 2020-02-05T23:55:30.207 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:30.205+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.206149+0000) 2020-02-05T23:55:30.208 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:30.205+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.206791+0000) 2020-02-05T23:55:30.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:30.206+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.207111+0000) 2020-02-05T23:55:30.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:30.206+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.207820+0000) 2020-02-05T23:55:30.209 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:30.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.208380+0000) 2020-02-05T23:55:30.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:30.211+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.212803+0000) 2020-02-05T23:55:30.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:30.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:30.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:30.444+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.445779+0000) 2020-02-05T23:55:30.708 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:30.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:30.706688+0000) 2020-02-05T23:55:31.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:31.442+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:31.442353+0000) 2020-02-05T23:55:32.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:32.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:32.812119+0000) 2020-02-05T23:55:33.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:33.098+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:33.098613+0000) 2020-02-05T23:55:33.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:33.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:33.975971+0000) 2020-02-05T23:55:33.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:33.975+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:33.975971+0000) 2020-02-05T23:55:35.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:35.226+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.226379+0000) 2020-02-05T23:55:35.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:35.226+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.228157+0000) 2020-02-05T23:55:35.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:35.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.228866+0000) 2020-02-05T23:55:35.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:35.228+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.230008+0000) 2020-02-05T23:55:35.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:35.228+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.230278+0000) 2020-02-05T23:55:35.232 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:35.230+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.230884+0000) 2020-02-05T23:55:35.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:35.229+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.230715+0000) 2020-02-05T23:55:35.232 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:35.231+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.231220+0000) 2020-02-05T23:55:35.233 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:35.231+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.231979+0000) 2020-02-05T23:55:35.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:35.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.232648+0000) 2020-02-05T23:55:35.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:35.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.232781+0000) 2020-02-05T23:55:35.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:35.231+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.232529+0000) 2020-02-05T23:55:35.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:35.233+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.233218+0000) 2020-02-05T23:55:35.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:35.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:35.238612+0000) 2020-02-05T23:55:35.245 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:35.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158891 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:36.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:36.075+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:36.075987+0000) 2020-02-05T23:55:36.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:36.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:36.955891+0000) 2020-02-05T23:55:37.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:37.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:37.216123+0000) 2020-02-05T23:55:37.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:37.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:37.216118+0000) 2020-02-05T23:55:37.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:37.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:37.216518+0000) 2020-02-05T23:55:39.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:39.212+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:39.214328+0000) 2020-02-05T23:55:39.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:39.597+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:39.598838+0000) 2020-02-05T23:55:40.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:40.249+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.250674+0000) 2020-02-05T23:55:40.252 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:40.250+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.251359+0000) 2020-02-05T23:55:40.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:40.250+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.251856+0000) 2020-02-05T23:55:40.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:40.251+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.252200+0000) 2020-02-05T23:55:40.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:40.251+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.253096+0000) 2020-02-05T23:55:40.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:40.252+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.253327+0000) 2020-02-05T23:55:40.255 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:40.252+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.253827+0000) 2020-02-05T23:55:40.255 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:40.253+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.254145+0000) 2020-02-05T23:55:40.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:40.253+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.254695+0000) 2020-02-05T23:55:40.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:40.255+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.256234+0000) 2020-02-05T23:55:40.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:40.253+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.254978+0000) 2020-02-05T23:55:40.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:40.255+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.256449+0000) 2020-02-05T23:55:40.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:40.255+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.256618+0000) 2020-02-05T23:55:40.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:40.259+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.260397+0000) 2020-02-05T23:55:40.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:40.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158933 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:40.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:40.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.445998+0000) 2020-02-05T23:55:40.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:40.706+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:40.707757+0000) 2020-02-05T23:55:41.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:41.449+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:41.450749+0000) 2020-02-05T23:55:42.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:42.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:42.812350+0000) 2020-02-05T23:55:43.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:43.097+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:43.098846+0000) 2020-02-05T23:55:43.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:43.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:43.976261+0000) 2020-02-05T23:55:43.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:43.975+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:43.976261+0000) 2020-02-05T23:55:45.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:45.271+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.272379+0000) 2020-02-05T23:55:45.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:45.270+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.272307+0000) 2020-02-05T23:55:45.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:45.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.273694+0000) 2020-02-05T23:55:45.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:45.272+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.273937+0000) 2020-02-05T23:55:45.275 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:45.273+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.274492+0000) 2020-02-05T23:55:45.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:45.273+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.274988+0000) 2020-02-05T23:55:45.276 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:45.274+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.275398+0000) 2020-02-05T23:55:45.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:45.274+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.275685+0000) 2020-02-05T23:55:45.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:45.274+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.275893+0000) 2020-02-05T23:55:45.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:45.275+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.276577+0000) 2020-02-05T23:55:45.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:45.275+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.276897+0000) 2020-02-05T23:55:45.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:45.276+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.277711+0000) 2020-02-05T23:55:45.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:45.276+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.277835+0000) 2020-02-05T23:55:45.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:45.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:45.283310+0000) 2020-02-05T23:55:45.292 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:45.289+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 158975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:46.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:46.075+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:46.076212+0000) 2020-02-05T23:55:46.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:46.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:46.956120+0000) 2020-02-05T23:55:47.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:47.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:47.216395+0000) 2020-02-05T23:55:47.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:47.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:47.216384+0000) 2020-02-05T23:55:47.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:47.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:47.216766+0000) 2020-02-05T23:55:49.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:49.213+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:49.214574+0000) 2020-02-05T23:55:49.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:49.598+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:49.599072+0000) 2020-02-05T23:55:50.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:50.296+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.297369+0000) 2020-02-05T23:55:50.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:50.296+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.297799+0000) 2020-02-05T23:55:50.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:50.297+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.299074+0000) 2020-02-05T23:55:50.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:50.298+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.299375+0000) 2020-02-05T23:55:50.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:50.298+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.299716+0000) 2020-02-05T23:55:50.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:50.299+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.300333+0000) 2020-02-05T23:55:50.302 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:50.299+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.300524+0000) 2020-02-05T23:55:50.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:50.298+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.300332+0000) 2020-02-05T23:55:50.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:50.299+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.301206+0000) 2020-02-05T23:55:50.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:50.299+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.301392+0000) 2020-02-05T23:55:50.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:50.300+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.301586+0000) 2020-02-05T23:55:50.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:50.301+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.302368+0000) 2020-02-05T23:55:50.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:50.303+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.304528+0000) 2020-02-05T23:55:50.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:50.304+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.305362+0000) 2020-02-05T23:55:50.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:50.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:50.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:50.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.446203+0000) 2020-02-05T23:55:50.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:50.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:50.707966+0000) 2020-02-05T23:55:51.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:51.457+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:51.458672+0000) 2020-02-05T23:55:52.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:52.811+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:52.812591+0000) 2020-02-05T23:55:53.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:53.098+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:53.099130+0000) 2020-02-05T23:55:53.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:53.975+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:53.976515+0000) 2020-02-05T23:55:53.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:53.975+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:53.976515+0000) 2020-02-05T23:55:55.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:55.316+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.317640+0000) 2020-02-05T23:55:55.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:55.318+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.319361+0000) 2020-02-05T23:55:55.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:55:55.318+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.319824+0000) 2020-02-05T23:55:55.321 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:55:55.319+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.320522+0000) 2020-02-05T23:55:55.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:55.320+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.321599+0000) 2020-02-05T23:55:55.323 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:55:55.320+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.321637+0000) 2020-02-05T23:55:55.323 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:55:55.320+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.321944+0000) 2020-02-05T23:55:55.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:55.321+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.322705+0000) 2020-02-05T23:55:55.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:55.322+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.323873+0000) 2020-02-05T23:55:55.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:55.323+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.324296+0000) 2020-02-05T23:55:55.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:55.323+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.324617+0000) 2020-02-05T23:55:55.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:55.323+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.324590+0000) 2020-02-05T23:55:55.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:55:55.323+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.325253+0000) 2020-02-05T23:55:55.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:55.327+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:55.328108+0000) 2020-02-05T23:55:55.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:55:55.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159058 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:55:56.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:55:56.075+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:56.076473+0000) 2020-02-05T23:55:56.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:55:56.955+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:56.956343+0000) 2020-02-05T23:55:57.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:55:57.215+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:57.216714+0000) 2020-02-05T23:55:57.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:55:57.215+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:57.216712+0000) 2020-02-05T23:55:57.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:55:57.215+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:57.217132+0000) 2020-02-05T23:55:59.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:55:59.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:59.214833+0000) 2020-02-05T23:55:59.594 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:55:59.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:55:59.598+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:55:59.599372+0000) 2020-02-05T23:55:59.601 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:56:00.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:00.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.342385+0000) 2020-02-05T23:56:00.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:00.341+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.343010+0000) 2020-02-05T23:56:00.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:00.342+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.343802+0000) 2020-02-05T23:56:00.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:00.343+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.344329+0000) 2020-02-05T23:56:00.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:00.343+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.344769+0000) 2020-02-05T23:56:00.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:00.343+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.345262+0000) 2020-02-05T23:56:00.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:00.344+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.345929+0000) 2020-02-05T23:56:00.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:00.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.346591+0000) 2020-02-05T23:56:00.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:00.346+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.348070+0000) 2020-02-05T23:56:00.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:00.347+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.348529+0000) 2020-02-05T23:56:00.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:00.346+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.348271+0000) 2020-02-05T23:56:00.350 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:00.347+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.348458+0000) 2020-02-05T23:56:00.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:00.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.348855+0000) 2020-02-05T23:56:00.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:00.351+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.352734+0000) 2020-02-05T23:56:00.360 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:00.357+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:00.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:00.445+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.446414+0000) 2020-02-05T23:56:00.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:00.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:00.708181+0000) 2020-02-05T23:56:01.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:01.465+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:01.467026+0000) 2020-02-05T23:56:02.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:02.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:02.812838+0000) 2020-02-05T23:56:03.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:03.098+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:03.099386+0000) 2020-02-05T23:56:03.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:03.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:03.976780+0000) 2020-02-05T23:56:03.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:03.976+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:03.976780+0000) 2020-02-05T23:56:05.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:05.364+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.365198+0000) 2020-02-05T23:56:05.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:05.364+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.366291+0000) 2020-02-05T23:56:05.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:05.365+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.366796+0000) 2020-02-05T23:56:05.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:05.366+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.367678+0000) 2020-02-05T23:56:05.369 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:05.366+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.367929+0000) 2020-02-05T23:56:05.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:05.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.368514+0000) 2020-02-05T23:56:05.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:05.367+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.368823+0000) 2020-02-05T23:56:05.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:05.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.368830+0000) 2020-02-05T23:56:05.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:05.367+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.369306+0000) 2020-02-05T23:56:05.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:05.368+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.369893+0000) 2020-02-05T23:56:05.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:05.369+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.370647+0000) 2020-02-05T23:56:05.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:05.369+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.370629+0000) 2020-02-05T23:56:05.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:05.369+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.371180+0000) 2020-02-05T23:56:05.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:05.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:05.376268+0000) 2020-02-05T23:56:05.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:05.380+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:06.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:06.076+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:06.076689+0000) 2020-02-05T23:56:06.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:06.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:06.956574+0000) 2020-02-05T23:56:07.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:07.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:07.216974+0000) 2020-02-05T23:56:07.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:07.216+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:07.216969+0000) 2020-02-05T23:56:07.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:07.216+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:07.217347+0000) 2020-02-05T23:56:09.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:09.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:09.215100+0000) 2020-02-05T23:56:09.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:09.599+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:09.599633+0000) 2020-02-05T23:56:10.390 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:10.386+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.387626+0000) 2020-02-05T23:56:10.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:10.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.388177+0000) 2020-02-05T23:56:10.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:10.387+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.388616+0000) 2020-02-05T23:56:10.391 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:10.388+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.389310+0000) 2020-02-05T23:56:10.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:10.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.390443+0000) 2020-02-05T23:56:10.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:10.389+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.390514+0000) 2020-02-05T23:56:10.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:10.389+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.391069+0000) 2020-02-05T23:56:10.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:10.390+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.391650+0000) 2020-02-05T23:56:10.393 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:10.390+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.391949+0000) 2020-02-05T23:56:10.393 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:10.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.392788+0000) 2020-02-05T23:56:10.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:10.391+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.393098+0000) 2020-02-05T23:56:10.394 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:10.391+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.393131+0000) 2020-02-05T23:56:10.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:10.392+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.394396+0000) 2020-02-05T23:56:10.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:10.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.399281+0000) 2020-02-05T23:56:10.407 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:10.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:10.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:10.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.446736+0000) 2020-02-05T23:56:10.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:10.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:10.708396+0000) 2020-02-05T23:56:11.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:11.474+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:11.475129+0000) 2020-02-05T23:56:12.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:12.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:12.813131+0000) 2020-02-05T23:56:13.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:13.099+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:13.099638+0000) 2020-02-05T23:56:13.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:13.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:13.977011+0000) 2020-02-05T23:56:13.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:13.976+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:13.977012+0000) 2020-02-05T23:56:15.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:15.411+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.412571+0000) 2020-02-05T23:56:15.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:15.412+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.413206+0000) 2020-02-05T23:56:15.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:15.412+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.413869+0000) 2020-02-05T23:56:15.415 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:15.413+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.414025+0000) 2020-02-05T23:56:15.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:15.413+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.414728+0000) 2020-02-05T23:56:15.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:15.414+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.415632+0000) 2020-02-05T23:56:15.417 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:15.414+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.416147+0000) 2020-02-05T23:56:15.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:15.414+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.415756+0000) 2020-02-05T23:56:15.418 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:15.416+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.417261+0000) 2020-02-05T23:56:15.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:15.416+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.417540+0000) 2020-02-05T23:56:15.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:15.416+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.418172+0000) 2020-02-05T23:56:15.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:15.417+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.418605+0000) 2020-02-05T23:56:15.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:15.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.418872+0000) 2020-02-05T23:56:15.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:15.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:15.423317+0000) 2020-02-05T23:56:15.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:15.428+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:16.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:16.076+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:16.076910+0000) 2020-02-05T23:56:16.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:16.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:16.956809+0000) 2020-02-05T23:56:17.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:17.216+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:17.217243+0000) 2020-02-05T23:56:17.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:17.216+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:17.217235+0000) 2020-02-05T23:56:17.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:17.216+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:17.217682+0000) 2020-02-05T23:56:19.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:19.214+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:19.215287+0000) 2020-02-05T23:56:19.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:19.599+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:19.599869+0000) 2020-02-05T23:56:20.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:20.435+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.436627+0000) 2020-02-05T23:56:20.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:20.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.436778+0000) 2020-02-05T23:56:20.439 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:20.436+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.437807+0000) 2020-02-05T23:56:20.441 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:20.438+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.439209+0000) 2020-02-05T23:56:20.441 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:20.438+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.439495+0000) 2020-02-05T23:56:20.442 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:20.439+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.440307+0000) 2020-02-05T23:56:20.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:20.440+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.440316+0000) 2020-02-05T23:56:20.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:20.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.440870+0000) 2020-02-05T23:56:20.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:20.441+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.441488+0000) 2020-02-05T23:56:20.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:20.441+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.442074+0000) 2020-02-05T23:56:20.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:20.441+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.442262+0000) 2020-02-05T23:56:20.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:20.440+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.441996+0000) 2020-02-05T23:56:20.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:20.442+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.442915+0000) 2020-02-05T23:56:20.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:20.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.446919+0000) 2020-02-05T23:56:20.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:20.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.447185+0000) 2020-02-05T23:56:20.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:20.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:20.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:20.707+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:20.708710+0000) 2020-02-05T23:56:21.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:21.482+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:21.483163+0000) 2020-02-05T23:56:22.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:22.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:22.813356+0000) 2020-02-05T23:56:23.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:23.099+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:23.099865+0000) 2020-02-05T23:56:23.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:23.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:23.977319+0000) 2020-02-05T23:56:23.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:23.977+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:23.977319+0000) 2020-02-05T23:56:25.461 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:25.459+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.460409+0000) 2020-02-05T23:56:25.462 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:25.460+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.461520+0000) 2020-02-05T23:56:25.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:25.460+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.462006+0000) 2020-02-05T23:56:25.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:25.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.462782+0000) 2020-02-05T23:56:25.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:25.462+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.463120+0000) 2020-02-05T23:56:25.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:25.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.463840+0000) 2020-02-05T23:56:25.465 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:25.463+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.463715+0000) 2020-02-05T23:56:25.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:25.464+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.464738+0000) 2020-02-05T23:56:25.467 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:25.464+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.465681+0000) 2020-02-05T23:56:25.467 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:25.464+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.465301+0000) 2020-02-05T23:56:25.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:25.465+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.465634+0000) 2020-02-05T23:56:25.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:25.466+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.466529+0000) 2020-02-05T23:56:25.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:25.466+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.466721+0000) 2020-02-05T23:56:25.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:25.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:25.471881+0000) 2020-02-05T23:56:25.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:25.480+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:26.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:26.076+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:26.077141+0000) 2020-02-05T23:56:26.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:26.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:26.957124+0000) 2020-02-05T23:56:27.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:27.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:27.217520+0000) 2020-02-05T23:56:27.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:27.217+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:27.217493+0000) 2020-02-05T23:56:27.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:27.217+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:27.217932+0000) 2020-02-05T23:56:29.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:29.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:29.215523+0000) 2020-02-05T23:56:29.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:29.600+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:29.600138+0000) 2020-02-05T23:56:29.780 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:56:29.786 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:56:30.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:30.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.447136+0000) 2020-02-05T23:56:30.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:30.486+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.486461+0000) 2020-02-05T23:56:30.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:30.487+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.487424+0000) 2020-02-05T23:56:30.489 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:30.487+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.487860+0000) 2020-02-05T23:56:30.490 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:30.488+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.488569+0000) 2020-02-05T23:56:30.490 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:30.488+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.489003+0000) 2020-02-05T23:56:30.491 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:30.490+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.490378+0000) 2020-02-05T23:56:30.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:30.490+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.490620+0000) 2020-02-05T23:56:30.492 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:30.490+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.491030+0000) 2020-02-05T23:56:30.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:30.491+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.492012+0000) 2020-02-05T23:56:30.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:30.492+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.492405+0000) 2020-02-05T23:56:30.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:30.493+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.493238+0000) 2020-02-05T23:56:30.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:30.492+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.492490+0000) 2020-02-05T23:56:30.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:30.492+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.492928+0000) 2020-02-05T23:56:30.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:30.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.497220+0000) 2020-02-05T23:56:30.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:30.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159350 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:30.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:30.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:30.708926+0000) 2020-02-05T23:56:31.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:31.491+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:31.491208+0000) 2020-02-05T23:56:32.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:32.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:32.813580+0000) 2020-02-05T23:56:33.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:33.100+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:33.100123+0000) 2020-02-05T23:56:33.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:33.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:33.977602+0000) 2020-02-05T23:56:33.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:33.976+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:33.977600+0000) 2020-02-05T23:56:35.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:35.510+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.511105+0000) 2020-02-05T23:56:35.513 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:35.511+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.512732+0000) 2020-02-05T23:56:35.514 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:35.512+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.513055+0000) 2020-02-05T23:56:35.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:35.513+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.513485+0000) 2020-02-05T23:56:35.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:35.513+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.513807+0000) 2020-02-05T23:56:35.516 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:35.514+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.514751+0000) 2020-02-05T23:56:35.516 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:35.514+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.515208+0000) 2020-02-05T23:56:35.517 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:35.514+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.514581+0000) 2020-02-05T23:56:35.517 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:35.515+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.515696+0000) 2020-02-05T23:56:35.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:35.515+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.516401+0000) 2020-02-05T23:56:35.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:35.515+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.516575+0000) 2020-02-05T23:56:35.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:35.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.516974+0000) 2020-02-05T23:56:35.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:35.516+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.517731+0000) 2020-02-05T23:56:35.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:35.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:35.524988+0000) 2020-02-05T23:56:35.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:35.526+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:36.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:36.077+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:36.077379+0000) 2020-02-05T23:56:36.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:36.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:36.957386+0000) 2020-02-05T23:56:37.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:37.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:37.217765+0000) 2020-02-05T23:56:37.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:37.217+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:37.217764+0000) 2020-02-05T23:56:37.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:37.217+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:37.218166+0000) 2020-02-05T23:56:39.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:39.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:39.215718+0000) 2020-02-05T23:56:39.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:39.599+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:39.600371+0000) 2020-02-05T23:56:40.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:40.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.447386+0000) 2020-02-05T23:56:40.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:40.531+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.532793+0000) 2020-02-05T23:56:40.534 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:40.532+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.533775+0000) 2020-02-05T23:56:40.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:40.534+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.534676+0000) 2020-02-05T23:56:40.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:40.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.535044+0000) 2020-02-05T23:56:40.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:40.535+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.535856+0000) 2020-02-05T23:56:40.537 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:40.535+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.536706+0000) 2020-02-05T23:56:40.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:40.536+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.537116+0000) 2020-02-05T23:56:40.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:40.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.537492+0000) 2020-02-05T23:56:40.539 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:40.537+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.537878+0000) 2020-02-05T23:56:40.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:40.537+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.538396+0000) 2020-02-05T23:56:40.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:40.538+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.538745+0000) 2020-02-05T23:56:40.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:40.538+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.539965+0000) 2020-02-05T23:56:40.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:40.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.542862+0000) 2020-02-05T23:56:40.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:40.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.543153+0000) 2020-02-05T23:56:40.551 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:40.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159432 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:40.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:40.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:40.709140+0000) 2020-02-05T23:56:41.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:41.498+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:41.499297+0000) 2020-02-05T23:56:42.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:42.812+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:42.813812+0000) 2020-02-05T23:56:43.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:43.099+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:43.100362+0000) 2020-02-05T23:56:43.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:43.976+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:43.977870+0000) 2020-02-05T23:56:43.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:43.976+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:43.977870+0000) 2020-02-05T23:56:45.557 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:45.554+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.555715+0000) 2020-02-05T23:56:45.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:45.555+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.556454+0000) 2020-02-05T23:56:45.558 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:45.556+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.557368+0000) 2020-02-05T23:56:45.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:45.557+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.558089+0000) 2020-02-05T23:56:45.559 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:45.557+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.558251+0000) 2020-02-05T23:56:45.560 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:45.559+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.559415+0000) 2020-02-05T23:56:45.561 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:45.558+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.559488+0000) 2020-02-05T23:56:45.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:45.558+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.559567+0000) 2020-02-05T23:56:45.561 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:45.560+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.560564+0000) 2020-02-05T23:56:45.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:45.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.560494+0000) 2020-02-05T23:56:45.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:45.560+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.560691+0000) 2020-02-05T23:56:45.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:45.561+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.562361+0000) 2020-02-05T23:56:45.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:45.561+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.562191+0000) 2020-02-05T23:56:45.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:45.564+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:45.565882+0000) 2020-02-05T23:56:45.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:45.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:46.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:46.077+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:46.077597+0000) 2020-02-05T23:56:46.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:46.956+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:46.957619+0000) 2020-02-05T23:56:47.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:47.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:47.218030+0000) 2020-02-05T23:56:47.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:47.217+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:47.218024+0000) 2020-02-05T23:56:47.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:47.217+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:47.218366+0000) 2020-02-05T23:56:49.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:49.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:49.215964+0000) 2020-02-05T23:56:49.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:49.599+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:49.600610+0000) 2020-02-05T23:56:50.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:50.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.447667+0000) 2020-02-05T23:56:50.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:50.577+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.578773+0000) 2020-02-05T23:56:50.581 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:50.579+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.580176+0000) 2020-02-05T23:56:50.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:50.580+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.580614+0000) 2020-02-05T23:56:50.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:50.579+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.580690+0000) 2020-02-05T23:56:50.582 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:50.580+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.581265+0000) 2020-02-05T23:56:50.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:50.581+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.582200+0000) 2020-02-05T23:56:50.583 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:50.581+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.582288+0000) 2020-02-05T23:56:50.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:50.581+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.582632+0000) 2020-02-05T23:56:50.585 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:50.582+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.583272+0000) 2020-02-05T23:56:50.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:50.581+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.582922+0000) 2020-02-05T23:56:50.585 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:50.583+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.583884+0000) 2020-02-05T23:56:50.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:50.582+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.584063+0000) 2020-02-05T23:56:50.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:50.584+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.585103+0000) 2020-02-05T23:56:50.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:50.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.588647+0000) 2020-02-05T23:56:50.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:50.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:50.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:50.708+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:50.709373+0000) 2020-02-05T23:56:51.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:51.506+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:51.507396+0000) 2020-02-05T23:56:52.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:52.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:52.814044+0000) 2020-02-05T23:56:53.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:53.099+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:53.100592+0000) 2020-02-05T23:56:53.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:53.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:53.978179+0000) 2020-02-05T23:56:53.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:53.977+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:53.978180+0000) 2020-02-05T23:56:55.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:56:55.600+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.601708+0000) 2020-02-05T23:56:55.603 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:55.602+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.602516+0000) 2020-02-05T23:56:55.604 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:56:55.601+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.602754+0000) 2020-02-05T23:56:55.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:55.602+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.603278+0000) 2020-02-05T23:56:55.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:56:55.602+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.603395+0000) 2020-02-05T23:56:55.605 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:55.603+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.604232+0000) 2020-02-05T23:56:55.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:55.603+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.604711+0000) 2020-02-05T23:56:55.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:56:55.605+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.606333+0000) 2020-02-05T23:56:55.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:55.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.606624+0000) 2020-02-05T23:56:55.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:56:55.606+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.607235+0000) 2020-02-05T23:56:55.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:55.605+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.607034+0000) 2020-02-05T23:56:55.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:55.607+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.607828+0000) 2020-02-05T23:56:55.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:55.607+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.607921+0000) 2020-02-05T23:56:55.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:55.610+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:55.611252+0000) 2020-02-05T23:56:55.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:56:55.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:56:56.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:56:56.077+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:56.077817+0000) 2020-02-05T23:56:56.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:56:56.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:56.957856+0000) 2020-02-05T23:56:57.219 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:56:57.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:57.218283+0000) 2020-02-05T23:56:57.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:56:57.218+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:57.218283+0000) 2020-02-05T23:56:57.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:56:57.218+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:57.218623+0000) 2020-02-05T23:56:59.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:56:59.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:59.216233+0000) 2020-02-05T23:56:59.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:56:59.600+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:56:59.600875+0000) 2020-02-05T23:56:59.943 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:56:59.950 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:57:00.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:00.446+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.447918+0000) 2020-02-05T23:57:00.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:00.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.625948+0000) 2020-02-05T23:57:00.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:00.625+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.626435+0000) 2020-02-05T23:57:00.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:00.627+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.628592+0000) 2020-02-05T23:57:00.630 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:00.627+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.628930+0000) 2020-02-05T23:57:00.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:00.628+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.629599+0000) 2020-02-05T23:57:00.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:00.628+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.629713+0000) 2020-02-05T23:57:00.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:00.628+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.630115+0000) 2020-02-05T23:57:00.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:00.629+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.630641+0000) 2020-02-05T23:57:00.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:00.629+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.630998+0000) 2020-02-05T23:57:00.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:00.630+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.631221+0000) 2020-02-05T23:57:00.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:00.630+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.631641+0000) 2020-02-05T23:57:00.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:00.631+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.632345+0000) 2020-02-05T23:57:00.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:00.631+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.632815+0000) 2020-02-05T23:57:00.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:00.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.638206+0000) 2020-02-05T23:57:00.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:00.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:00.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:00.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:00.709677+0000) 2020-02-05T23:57:01.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:01.515+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:01.515733+0000) 2020-02-05T23:57:02.815 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:02.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:02.814308+0000) 2020-02-05T23:57:03.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:03.100+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:03.100824+0000) 2020-02-05T23:57:03.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:03.977+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:03.978449+0000) 2020-02-05T23:57:03.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:03.977+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:03.978449+0000) 2020-02-05T23:57:05.651 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:05.649+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.650147+0000) 2020-02-05T23:57:05.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:05.650+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.651772+0000) 2020-02-05T23:57:05.653 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:05.650+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.651772+0000) 2020-02-05T23:57:05.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:05.650+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.652301+0000) 2020-02-05T23:57:05.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:05.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.652986+0000) 2020-02-05T23:57:05.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:05.652+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.653216+0000) 2020-02-05T23:57:05.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:05.652+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.653415+0000) 2020-02-05T23:57:05.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:05.652+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.653700+0000) 2020-02-05T23:57:05.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:05.652+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.654058+0000) 2020-02-05T23:57:05.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:05.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.654579+0000) 2020-02-05T23:57:05.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:05.654+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.655854+0000) 2020-02-05T23:57:05.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:05.654+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.656204+0000) 2020-02-05T23:57:05.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:05.655+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.656607+0000) 2020-02-05T23:57:05.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:05.660+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:05.661764+0000) 2020-02-05T23:57:05.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:05.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:06.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:06.076+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:06.078039+0000) 2020-02-05T23:57:06.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:06.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:06.958114+0000) 2020-02-05T23:57:07.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:07.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:07.218539+0000) 2020-02-05T23:57:07.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:07.217+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:07.218535+0000) 2020-02-05T23:57:07.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:07.217+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:07.218871+0000) 2020-02-05T23:57:09.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:09.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:09.216515+0000) 2020-02-05T23:57:09.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:09.600+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:09.601138+0000) 2020-02-05T23:57:10.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:10.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.448175+0000) 2020-02-05T23:57:10.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:10.674+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.675395+0000) 2020-02-05T23:57:10.677 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:10.674+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.676048+0000) 2020-02-05T23:57:10.678 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:10.675+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.677006+0000) 2020-02-05T23:57:10.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:10.676+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.677555+0000) 2020-02-05T23:57:10.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:10.676+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.677591+0000) 2020-02-05T23:57:10.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:10.676+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678039+0000) 2020-02-05T23:57:10.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:10.677+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678116+0000) 2020-02-05T23:57:10.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:10.676+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678126+0000) 2020-02-05T23:57:10.680 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:10.677+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678597+0000) 2020-02-05T23:57:10.680 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:10.677+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678905+0000) 2020-02-05T23:57:10.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:10.677+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.678520+0000) 2020-02-05T23:57:10.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:10.677+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.679308+0000) 2020-02-05T23:57:10.681 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:10.678+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.679880+0000) 2020-02-05T23:57:10.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:10.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.683266+0000) 2020-02-05T23:57:10.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:10.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:10.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:10.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:10.709914+0000) 2020-02-05T23:57:11.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:11.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:11.524249+0000) 2020-02-05T23:57:12.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:12.813+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:12.814570+0000) 2020-02-05T23:57:13.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:13.100+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:13.101057+0000) 2020-02-05T23:57:13.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:13.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:13.978722+0000) 2020-02-05T23:57:13.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:13.978+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:13.978722+0000) 2020-02-05T23:57:15.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:15.696+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.698000+0000) 2020-02-05T23:57:15.700 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:15.697+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.698785+0000) 2020-02-05T23:57:15.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:15.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.699423+0000) 2020-02-05T23:57:15.701 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:15.698+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.700118+0000) 2020-02-05T23:57:15.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:15.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.700221+0000) 2020-02-05T23:57:15.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:15.699+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.700618+0000) 2020-02-05T23:57:15.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:15.699+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.701044+0000) 2020-02-05T23:57:15.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:15.700+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.701417+0000) 2020-02-05T23:57:15.703 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:15.700+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.702126+0000) 2020-02-05T23:57:15.703 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:15.701+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.702398+0000) 2020-02-05T23:57:15.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:15.701+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.703118+0000) 2020-02-05T23:57:15.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:15.702+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.703916+0000) 2020-02-05T23:57:15.705 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:15.702+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.703953+0000) 2020-02-05T23:57:15.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:15.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:15.708507+0000) 2020-02-05T23:57:15.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:15.716+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159722 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:16.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:16.077+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:16.078259+0000) 2020-02-05T23:57:16.959 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:16.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:16.958358+0000) 2020-02-05T23:57:17.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:17.217+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:17.218777+0000) 2020-02-05T23:57:17.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:17.217+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:17.218777+0000) 2020-02-05T23:57:17.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:17.217+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:17.219112+0000) 2020-02-05T23:57:19.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:19.215+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:19.216784+0000) 2020-02-05T23:57:19.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:19.601+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:19.601379+0000) 2020-02-05T23:57:20.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:20.447+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.448424+0000) 2020-02-05T23:57:20.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:20.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.710127+0000) 2020-02-05T23:57:20.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:20.722+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.723548+0000) 2020-02-05T23:57:20.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:20.722+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.723580+0000) 2020-02-05T23:57:20.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:20.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.723996+0000) 2020-02-05T23:57:20.726 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:20.723+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.724658+0000) 2020-02-05T23:57:20.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:20.723+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.724712+0000) 2020-02-05T23:57:20.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:20.724+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.725337+0000) 2020-02-05T23:57:20.727 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:20.724+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.725618+0000) 2020-02-05T23:57:20.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:20.724+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.725833+0000) 2020-02-05T23:57:20.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:20.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.726299+0000) 2020-02-05T23:57:20.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:20.725+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.727017+0000) 2020-02-05T23:57:20.728 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:20.726+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.727287+0000) 2020-02-05T23:57:20.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:20.726+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.727756+0000) 2020-02-05T23:57:20.729 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:20.726+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.727543+0000) 2020-02-05T23:57:20.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:20.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:20.732111+0000) 2020-02-05T23:57:20.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:20.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159762 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:21.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:21.531+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:21.532403+0000) 2020-02-05T23:57:22.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:22.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:22.814872+0000) 2020-02-05T23:57:23.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:23.101+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:23.101355+0000) 2020-02-05T23:57:23.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:23.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:23.978982+0000) 2020-02-05T23:57:23.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:23.978+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:23.978982+0000) 2020-02-05T23:57:25.747 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:25.745+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.745782+0000) 2020-02-05T23:57:25.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:25.746+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.747017+0000) 2020-02-05T23:57:25.749 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:25.746+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.747962+0000) 2020-02-05T23:57:25.749 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:25.748+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.748508+0000) 2020-02-05T23:57:25.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:25.749+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.749452+0000) 2020-02-05T23:57:25.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:25.749+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.749681+0000) 2020-02-05T23:57:25.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:25.750+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.750184+0000) 2020-02-05T23:57:25.752 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:25.748+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.750282+0000) 2020-02-05T23:57:25.752 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:25.748+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.750047+0000) 2020-02-05T23:57:25.752 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:25.749+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.751032+0000) 2020-02-05T23:57:25.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:25.750+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.750940+0000) 2020-02-05T23:57:25.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:25.751+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.751676+0000) 2020-02-05T23:57:25.753 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:25.750+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.751974+0000) 2020-02-05T23:57:25.754 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:25.750+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:25.752041+0000) 2020-02-05T23:57:25.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:25.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159804 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:26.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:26.077+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:26.078496+0000) 2020-02-05T23:57:26.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:26.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:26.958599+0000) 2020-02-05T23:57:27.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:27.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:27.219043+0000) 2020-02-05T23:57:27.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:27.218+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:27.219041+0000) 2020-02-05T23:57:27.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:27.218+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:27.219335+0000) 2020-02-05T23:57:29.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:29.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:29.217056+0000) 2020-02-05T23:57:29.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:29.600+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:29.601657+0000) 2020-02-05T23:57:30.122 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:57:30.129 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:57:30.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:30.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.448715+0000) 2020-02-05T23:57:30.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:30.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.710386+0000) 2020-02-05T23:57:30.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:30.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.769650+0000) 2020-02-05T23:57:30.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:30.769+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.770268+0000) 2020-02-05T23:57:30.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:30.769+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.770565+0000) 2020-02-05T23:57:30.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:30.769+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.770936+0000) 2020-02-05T23:57:30.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:30.770+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.771490+0000) 2020-02-05T23:57:30.785 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:30.774+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.775707+0000) 2020-02-05T23:57:30.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:30.774+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.775964+0000) 2020-02-05T23:57:30.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:30.776+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.777534+0000) 2020-02-05T23:57:30.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:30.771+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.773052+0000) 2020-02-05T23:57:30.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:30.771+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.773377+0000) 2020-02-05T23:57:30.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:30.772+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.774018+0000) 2020-02-05T23:57:30.787 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:30.772+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.774285+0000) 2020-02-05T23:57:30.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:30.773+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.774940+0000) 2020-02-05T23:57:30.787 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:30.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:30.775721+0000) 2020-02-05T23:57:30.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:30.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:31.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:31.538+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:31.539521+0000) 2020-02-05T23:57:32.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:32.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:32.815135+0000) 2020-02-05T23:57:33.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:33.100+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:33.101590+0000) 2020-02-05T23:57:33.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:33.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:33.979250+0000) 2020-02-05T23:57:33.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:33.978+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:33.979249+0000) 2020-02-05T23:57:35.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:35.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.796349+0000) 2020-02-05T23:57:35.798 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:35.796+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.797325+0000) 2020-02-05T23:57:35.798 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:35.796+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.797629+0000) 2020-02-05T23:57:35.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:35.796+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.797868+0000) 2020-02-05T23:57:35.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:35.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.798330+0000) 2020-02-05T23:57:35.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:35.797+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.798562+0000) 2020-02-05T23:57:35.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:35.797+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.799034+0000) 2020-02-05T23:57:35.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:35.798+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.799606+0000) 2020-02-05T23:57:35.801 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:35.797+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.799259+0000) 2020-02-05T23:57:35.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:35.798+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.800282+0000) 2020-02-05T23:57:35.802 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:35.799+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.800867+0000) 2020-02-05T23:57:35.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:35.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.800668+0000) 2020-02-05T23:57:35.803 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:35.800+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.801415+0000) 2020-02-05T23:57:35.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:35.805+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:35.806275+0000) 2020-02-05T23:57:35.814 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:35.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:36.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:36.078+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:36.078731+0000) 2020-02-05T23:57:36.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:36.957+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:36.958832+0000) 2020-02-05T23:57:37.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:37.218+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:37.219296+0000) 2020-02-05T23:57:37.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:37.218+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:37.219298+0000) 2020-02-05T23:57:37.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:37.218+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:37.219633+0000) 2020-02-05T23:57:39.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:39.216+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:39.217326+0000) 2020-02-05T23:57:39.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:39.601+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:39.601893+0000) 2020-02-05T23:57:40.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:40.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.448946+0000) 2020-02-05T23:57:40.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:40.709+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.710610+0000) 2020-02-05T23:57:40.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:40.818+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.819578+0000) 2020-02-05T23:57:40.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:40.819+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.820501+0000) 2020-02-05T23:57:40.822 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:40.819+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.821132+0000) 2020-02-05T23:57:40.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:40.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.821583+0000) 2020-02-05T23:57:40.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:40.821+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.822458+0000) 2020-02-05T23:57:40.824 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:40.821+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.823366+0000) 2020-02-05T23:57:40.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:40.822+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.823705+0000) 2020-02-05T23:57:40.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:40.822+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.824013+0000) 2020-02-05T23:57:40.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:40.823+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.824381+0000) 2020-02-05T23:57:40.826 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:40.823+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.824782+0000) 2020-02-05T23:57:40.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:40.823+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.825029+0000) 2020-02-05T23:57:40.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:40.824+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.825487+0000) 2020-02-05T23:57:40.827 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:40.823+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.825229+0000) 2020-02-05T23:57:40.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:40.826+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:40.827890+0000) 2020-02-05T23:57:40.838 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:40.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159926 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:41.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:41.549+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:41.550488+0000) 2020-02-05T23:57:42.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:42.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:42.815317+0000) 2020-02-05T23:57:43.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:43.101+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:43.101820+0000) 2020-02-05T23:57:43.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:43.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:43.979472+0000) 2020-02-05T23:57:43.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:43.978+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:43.979472+0000) 2020-02-05T23:57:45.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:45.841+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.842753+0000) 2020-02-05T23:57:45.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:45.842+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.843473+0000) 2020-02-05T23:57:45.857 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:45.843+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.844063+0000) 2020-02-05T23:57:45.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:45.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.844557+0000) 2020-02-05T23:57:45.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:45.844+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.845564+0000) 2020-02-05T23:57:45.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:45.850+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.851814+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:45.845+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.846137+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:45.845+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.846892+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:45.846+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.847758+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:45.843+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.845199+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:45.844+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.845538+0000) 2020-02-05T23:57:45.859 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:45.845+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.846870+0000) 2020-02-05T23:57:45.860 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:45.846+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.847736+0000) 2020-02-05T23:57:45.860 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:45.846+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:45.847865+0000) 2020-02-05T23:57:45.865 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:45.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 159968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:46.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:46.078+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:46.078953+0000) 2020-02-05T23:57:46.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:46.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:46.959060+0000) 2020-02-05T23:57:47.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:47.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:47.219561+0000) 2020-02-05T23:57:47.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:47.219+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:47.219560+0000) 2020-02-05T23:57:47.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:47.219+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:47.219852+0000) 2020-02-05T23:57:49.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:49.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:49.217661+0000) 2020-02-05T23:57:49.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:49.601+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:49.602134+0000) 2020-02-05T23:57:50.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:50.448+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.449196+0000) 2020-02-05T23:57:50.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:50.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.710842+0000) 2020-02-05T23:57:50.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:50.865+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.866584+0000) 2020-02-05T23:57:50.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:50.867+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.867574+0000) 2020-02-05T23:57:50.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:50.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.868024+0000) 2020-02-05T23:57:50.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:50.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.868502+0000) 2020-02-05T23:57:50.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:50.868+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.869143+0000) 2020-02-05T23:57:50.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:50.869+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.870165+0000) 2020-02-05T23:57:50.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:50.870+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.871253+0000) 2020-02-05T23:57:50.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:50.871+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872404+0000) 2020-02-05T23:57:50.874 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:50.871+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872392+0000) 2020-02-05T23:57:50.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:50.872+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872513+0000) 2020-02-05T23:57:50.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:50.872+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872897+0000) 2020-02-05T23:57:50.874 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:50.872+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872769+0000) 2020-02-05T23:57:50.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:50.871+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.872960+0000) 2020-02-05T23:57:50.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:50.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:50.875318+0000) 2020-02-05T23:57:50.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:50.884+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:51.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:51.558+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:51.558892+0000) 2020-02-05T23:57:52.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:52.814+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:52.815522+0000) 2020-02-05T23:57:53.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:53.101+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:53.102063+0000) 2020-02-05T23:57:53.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:53.978+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:53.979761+0000) 2020-02-05T23:57:53.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:53.978+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:53.979762+0000) 2020-02-05T23:57:55.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:55.888+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.889249+0000) 2020-02-05T23:57:55.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:55.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.890776+0000) 2020-02-05T23:57:55.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:55.891+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.891565+0000) 2020-02-05T23:57:55.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:55.893+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.893572+0000) 2020-02-05T23:57:55.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:55.893+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.893884+0000) 2020-02-05T23:57:55.895 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:55.894+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.894678+0000) 2020-02-05T23:57:55.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:57:55.893+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.894590+0000) 2020-02-05T23:57:55.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:57:55.893+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.894935+0000) 2020-02-05T23:57:55.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:57:55.894+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.895177+0000) 2020-02-05T23:57:55.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:57:55.894+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.895443+0000) 2020-02-05T23:57:55.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:55.894+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.895128+0000) 2020-02-05T23:57:55.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:57:55.895+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.895571+0000) 2020-02-05T23:57:55.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:55.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.898597+0000) 2020-02-05T23:57:55.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:55.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:55.898863+0000) 2020-02-05T23:57:55.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:57:55.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160050 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:57:56.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:57:56.078+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:56.079191+0000) 2020-02-05T23:57:56.960 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:57:56.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:56.959284+0000) 2020-02-05T23:57:57.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:57:57.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:57.219794+0000) 2020-02-05T23:57:57.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:57:57.219+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:57.219843+0000) 2020-02-05T23:57:57.221 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:57:57.219+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:57.220063+0000) 2020-02-05T23:57:59.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:57:59.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:59.217857+0000) 2020-02-05T23:57:59.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:57:59.601+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:57:59.602391+0000) 2020-02-05T23:58:00.302 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:58:00.308 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:58:00.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:00.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.449447+0000) 2020-02-05T23:58:00.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:00.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.711135+0000) 2020-02-05T23:58:00.913 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:00.911+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.912180+0000) 2020-02-05T23:58:00.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:00.912+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.913425+0000) 2020-02-05T23:58:00.915 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:00.913+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.913870+0000) 2020-02-05T23:58:00.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:00.913+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.914213+0000) 2020-02-05T23:58:00.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:00.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.915146+0000) 2020-02-05T23:58:00.916 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:00.914+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.915331+0000) 2020-02-05T23:58:00.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:00.915+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.916105+0000) 2020-02-05T23:58:00.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:00.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.916269+0000) 2020-02-05T23:58:00.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:00.915+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.916413+0000) 2020-02-05T23:58:00.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:00.915+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.916790+0000) 2020-02-05T23:58:00.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:00.916+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.917379+0000) 2020-02-05T23:58:00.919 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:00.917+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.917452+0000) 2020-02-05T23:58:00.919 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:00.917+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.918105+0000) 2020-02-05T23:58:00.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:00.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:00.923011+0000) 2020-02-05T23:58:00.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:00.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:01.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:01.569+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:01.570309+0000) 2020-02-05T23:58:02.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:02.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:02.815788+0000) 2020-02-05T23:58:03.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:03.101+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:03.102361+0000) 2020-02-05T23:58:03.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:03.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:03.980015+0000) 2020-02-05T23:58:03.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:03.979+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:03.980015+0000) 2020-02-05T23:58:05.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:05.933+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.934350+0000) 2020-02-05T23:58:05.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:05.935+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.936904+0000) 2020-02-05T23:58:05.938 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:05.936+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.937214+0000) 2020-02-05T23:58:05.939 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:05.936+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.936589+0000) 2020-02-05T23:58:05.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:05.938+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.938488+0000) 2020-02-05T23:58:05.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:05.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.938765+0000) 2020-02-05T23:58:05.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:05.938+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.939382+0000) 2020-02-05T23:58:05.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:05.938+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.939666+0000) 2020-02-05T23:58:05.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:05.938+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.939959+0000) 2020-02-05T23:58:05.943 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:05.941+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.941813+0000) 2020-02-05T23:58:05.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:05.941+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.942014+0000) 2020-02-05T23:58:05.943 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:05.941+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.942273+0000) 2020-02-05T23:58:05.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:05.941+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.942025+0000) 2020-02-05T23:58:05.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:05.943+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:05.945031+0000) 2020-02-05T23:58:05.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:05.951+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:06.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:06.079+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:06.079411+0000) 2020-02-05T23:58:06.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:06.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:06.959515+0000) 2020-02-05T23:58:07.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:07.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:07.220045+0000) 2020-02-05T23:58:07.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:07.219+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:07.220090+0000) 2020-02-05T23:58:07.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:07.219+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:07.220297+0000) 2020-02-05T23:58:09.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:09.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:09.218094+0000) 2020-02-05T23:58:09.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:09.602+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:09.602586+0000) 2020-02-05T23:58:10.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:10.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.449702+0000) 2020-02-05T23:58:10.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:10.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.711318+0000) 2020-02-05T23:58:10.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:10.958+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.958781+0000) 2020-02-05T23:58:10.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:10.958+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.959577+0000) 2020-02-05T23:58:10.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:10.959+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.960183+0000) 2020-02-05T23:58:10.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:10.959+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.960514+0000) 2020-02-05T23:58:10.962 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:10.959+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.961017+0000) 2020-02-05T23:58:10.962 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:10.960+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.961154+0000) 2020-02-05T23:58:10.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:10.961+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.961842+0000) 2020-02-05T23:58:10.963 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:10.961+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.962190+0000) 2020-02-05T23:58:10.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:10.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.962546+0000) 2020-02-05T23:58:10.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:10.963+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.963634+0000) 2020-02-05T23:58:10.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:10.963+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.963717+0000) 2020-02-05T23:58:10.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:10.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.964022+0000) 2020-02-05T23:58:10.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:10.963+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.964224+0000) 2020-02-05T23:58:10.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:10.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:10.969044+0000) 2020-02-05T23:58:10.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:10.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:11.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:11.578+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:11.578439+0000) 2020-02-05T23:58:12.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:12.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:12.816014+0000) 2020-02-05T23:58:13.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:13.102+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:13.102587+0000) 2020-02-05T23:58:13.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:13.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:13.980330+0000) 2020-02-05T23:58:13.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:13.980+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:13.980331+0000) 2020-02-05T23:58:15.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:15.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.982070+0000) 2020-02-05T23:58:15.984 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:15.982+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.983105+0000) 2020-02-05T23:58:15.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:15.982+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.983207+0000) 2020-02-05T23:58:15.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:15.983+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.983538+0000) 2020-02-05T23:58:15.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:15.983+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.983899+0000) 2020-02-05T23:58:15.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:15.983+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.984116+0000) 2020-02-05T23:58:15.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:15.982+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.983824+0000) 2020-02-05T23:58:15.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:15.983+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.984855+0000) 2020-02-05T23:58:15.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:15.985+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.985522+0000) 2020-02-05T23:58:15.987 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:15.984+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.985429+0000) 2020-02-05T23:58:15.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:15.984+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.985597+0000) 2020-02-05T23:58:15.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:15.985+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.986262+0000) 2020-02-05T23:58:15.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:15.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.986443+0000) 2020-02-05T23:58:15.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:15.990+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:15.991559+0000) 2020-02-05T23:58:16.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:15.998+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160214 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:16.081 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:16.079+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:16.079684+0000) 2020-02-05T23:58:16.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:16.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:16.959752+0000) 2020-02-05T23:58:17.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:17.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:17.220302+0000) 2020-02-05T23:58:17.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:17.220+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:17.220339+0000) 2020-02-05T23:58:17.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:17.220+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:17.220539+0000) 2020-02-05T23:58:19.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:19.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:19.218353+0000) 2020-02-05T23:58:19.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:19.602+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:19.602830+0000) 2020-02-05T23:58:20.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:20.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:20.449931+0000) 2020-02-05T23:58:20.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:20.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:20.711502+0000) 2020-02-05T23:58:21.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:21.004+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.004870+0000) 2020-02-05T23:58:21.006 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:21.005+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.005547+0000) 2020-02-05T23:58:21.007 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:21.005+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.006205+0000) 2020-02-05T23:58:21.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:21.006+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.006463+0000) 2020-02-05T23:58:21.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:21.007+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.007201+0000) 2020-02-05T23:58:21.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:21.008+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.008570+0000) 2020-02-05T23:58:21.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:21.008+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.009205+0000) 2020-02-05T23:58:21.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:21.009+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.009445+0000) 2020-02-05T23:58:21.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:21.010+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.010353+0000) 2020-02-05T23:58:21.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:21.010+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.010889+0000) 2020-02-05T23:58:21.012 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:21.011+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.011650+0000) 2020-02-05T23:58:21.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:21.011+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.011711+0000) 2020-02-05T23:58:21.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:21.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.015737+0000) 2020-02-05T23:58:21.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:21.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.016009+0000) 2020-02-05T23:58:21.025 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:21.024+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:21.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:21.586+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:21.586690+0000) 2020-02-05T23:58:22.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:22.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:22.816284+0000) 2020-02-05T23:58:23.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:23.101+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:23.102818+0000) 2020-02-05T23:58:23.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:23.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:23.980593+0000) 2020-02-05T23:58:23.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:23.979+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:23.980592+0000) 2020-02-05T23:58:26.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:26.028+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.030042+0000) 2020-02-05T23:58:26.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:26.029+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.030696+0000) 2020-02-05T23:58:26.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:26.029+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.030972+0000) 2020-02-05T23:58:26.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:26.031+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.031831+0000) 2020-02-05T23:58:26.033 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:26.031+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.032099+0000) 2020-02-05T23:58:26.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:26.032+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.033368+0000) 2020-02-05T23:58:26.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:26.032+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.033726+0000) 2020-02-05T23:58:26.036 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:26.033+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.034348+0000) 2020-02-05T23:58:26.036 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:26.033+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.034366+0000) 2020-02-05T23:58:26.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:26.034+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.035035+0000) 2020-02-05T23:58:26.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:26.034+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.035255+0000) 2020-02-05T23:58:26.037 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:26.035+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.036297+0000) 2020-02-05T23:58:26.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:26.037+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.038717+0000) 2020-02-05T23:58:26.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:26.038+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.039782+0000) 2020-02-05T23:58:26.050 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:26.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:26.081 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:26.079+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.079914+0000) 2020-02-05T23:58:26.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:26.958+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:26.960002+0000) 2020-02-05T23:58:27.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:27.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:27.220538+0000) 2020-02-05T23:58:27.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:27.219+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:27.220652+0000) 2020-02-05T23:58:27.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:27.219+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:27.220804+0000) 2020-02-05T23:58:29.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:29.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:29.218554+0000) 2020-02-05T23:58:29.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:29.602+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:29.603063+0000) 2020-02-05T23:58:30.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:30.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:30.450186+0000) 2020-02-05T23:58:30.474 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:58:30.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:58:30.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:30.710+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:30.711755+0000) 2020-02-05T23:58:31.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:31.054+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.055496+0000) 2020-02-05T23:58:31.057 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:31.055+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.056609+0000) 2020-02-05T23:58:31.058 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:31.055+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.057173+0000) 2020-02-05T23:58:31.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:31.056+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.057367+0000) 2020-02-05T23:58:31.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:31.056+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.057792+0000) 2020-02-05T23:58:31.059 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:31.057+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.058294+0000) 2020-02-05T23:58:31.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:31.058+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.059193+0000) 2020-02-05T23:58:31.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:31.058+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.059257+0000) 2020-02-05T23:58:31.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:31.058+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.059776+0000) 2020-02-05T23:58:31.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:31.058+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.060357+0000) 2020-02-05T23:58:31.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:31.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.060668+0000) 2020-02-05T23:58:31.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:31.059+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.061077+0000) 2020-02-05T23:58:31.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:31.059+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.061319+0000) 2020-02-05T23:58:31.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:31.064+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.066025+0000) 2020-02-05T23:58:31.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:31.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:31.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:31.594+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:31.595135+0000) 2020-02-05T23:58:32.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:32.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:32.816553+0000) 2020-02-05T23:58:33.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:33.102+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:33.103049+0000) 2020-02-05T23:58:33.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:33.979+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:33.980883+0000) 2020-02-05T23:58:33.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:33.979+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:33.980879+0000) 2020-02-05T23:58:36.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:36.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.080072+0000) 2020-02-05T23:58:36.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:36.078+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.079877+0000) 2020-02-05T23:58:36.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:36.080+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.081377+0000) 2020-02-05T23:58:36.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:36.082+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.083072+0000) 2020-02-05T23:58:36.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:36.081+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.082920+0000) 2020-02-05T23:58:36.085 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:36.082+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.083398+0000) 2020-02-05T23:58:36.085 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:36.082+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.083911+0000) 2020-02-05T23:58:36.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:36.083+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.084294+0000) 2020-02-05T23:58:36.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:36.083+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.084456+0000) 2020-02-05T23:58:36.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:36.083+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.084674+0000) 2020-02-05T23:58:36.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:36.083+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.084848+0000) 2020-02-05T23:58:36.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:36.083+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.084612+0000) 2020-02-05T23:58:36.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:36.084+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.085717+0000) 2020-02-05T23:58:36.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:36.087+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.088729+0000) 2020-02-05T23:58:36.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:36.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:36.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:36.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:36.960229+0000) 2020-02-05T23:58:37.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:37.219+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:37.220793+0000) 2020-02-05T23:58:37.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:37.219+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:37.220885+0000) 2020-02-05T23:58:37.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:37.219+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:37.221036+0000) 2020-02-05T23:58:39.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:39.217+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:39.218796+0000) 2020-02-05T23:58:39.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:39.602+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:39.603362+0000) 2020-02-05T23:58:40.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:40.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:40.450493+0000) 2020-02-05T23:58:40.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:40.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:40.711999+0000) 2020-02-05T23:58:41.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:41.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.102157+0000) 2020-02-05T23:58:41.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:41.101+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.102506+0000) 2020-02-05T23:58:41.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:41.101+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.102980+0000) 2020-02-05T23:58:41.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:41.102+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.103408+0000) 2020-02-05T23:58:41.105 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:41.102+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.103849+0000) 2020-02-05T23:58:41.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:41.102+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.103977+0000) 2020-02-05T23:58:41.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:41.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.105014+0000) 2020-02-05T23:58:41.106 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:41.104+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.105576+0000) 2020-02-05T23:58:41.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:41.105+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.106738+0000) 2020-02-05T23:58:41.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:41.105+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.107221+0000) 2020-02-05T23:58:41.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:41.106+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.107679+0000) 2020-02-05T23:58:41.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:41.106+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.107814+0000) 2020-02-05T23:58:41.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:41.106+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.108331+0000) 2020-02-05T23:58:41.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:41.110+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.111743+0000) 2020-02-05T23:58:41.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:41.118+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:41.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:41.602+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:41.603428+0000) 2020-02-05T23:58:42.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:42.815+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:42.816802+0000) 2020-02-05T23:58:42.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:42.830+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:42.831019+0000) 2020-02-05T23:58:43.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:43.102+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:43.103312+0000) 2020-02-05T23:58:43.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:43.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:43.981150+0000) 2020-02-05T23:58:43.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:43.980+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:43.981150+0000) 2020-02-05T23:58:46.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:46.123+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.125012+0000) 2020-02-05T23:58:46.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:46.123+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.124931+0000) 2020-02-05T23:58:46.127 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:46.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.126038+0000) 2020-02-05T23:58:46.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:46.125+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.126562+0000) 2020-02-05T23:58:46.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:46.126+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.127161+0000) 2020-02-05T23:58:46.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:46.126+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.128151+0000) 2020-02-05T23:58:46.140 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:46.127+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.129396+0000) 2020-02-05T23:58:46.140 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:46.128+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.130152+0000) 2020-02-05T23:58:46.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:46.129+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.131111+0000) 2020-02-05T23:58:46.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:46.130+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.131949+0000) 2020-02-05T23:58:46.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:46.126+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.127545+0000) 2020-02-05T23:58:46.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:46.127+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.128692+0000) 2020-02-05T23:58:46.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:46.134+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.135486+0000) 2020-02-05T23:58:46.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:46.129+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.130447+0000) 2020-02-05T23:58:46.143 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:46.140+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:46.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:46.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:46.960463+0000) 2020-02-05T23:58:47.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:47.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:47.221126+0000) 2020-02-05T23:58:47.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:47.220+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:47.221126+0000) 2020-02-05T23:58:47.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:47.220+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:47.221298+0000) 2020-02-05T23:58:49.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:49.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:49.219045+0000) 2020-02-05T23:58:49.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:49.603+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:49.603603+0000) 2020-02-05T23:58:50.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:50.449+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:50.450759+0000) 2020-02-05T23:58:50.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:50.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:50.712255+0000) 2020-02-05T23:58:51.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:51.146+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.147988+0000) 2020-02-05T23:58:51.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:51.146+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.147754+0000) 2020-02-05T23:58:51.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:51.147+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.148523+0000) 2020-02-05T23:58:51.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:51.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.148943+0000) 2020-02-05T23:58:51.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:51.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.150004+0000) 2020-02-05T23:58:51.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:51.149+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.150699+0000) 2020-02-05T23:58:51.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:51.149+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.150836+0000) 2020-02-05T23:58:51.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:51.150+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.151368+0000) 2020-02-05T23:58:51.153 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:51.150+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.151838+0000) 2020-02-05T23:58:51.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:51.150+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.152100+0000) 2020-02-05T23:58:51.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:51.151+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.152375+0000) 2020-02-05T23:58:51.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:51.151+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.152923+0000) 2020-02-05T23:58:51.154 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:51.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.153802+0000) 2020-02-05T23:58:51.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:51.157+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.158560+0000) 2020-02-05T23:58:51.166 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:51.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:51.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:51.611+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:51.611547+0000) 2020-02-05T23:58:52.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:52.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:52.817113+0000) 2020-02-05T23:58:52.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:52.830+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:52.831284+0000) 2020-02-05T23:58:53.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:53.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:53.103542+0000) 2020-02-05T23:58:53.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:53.980+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:53.981423+0000) 2020-02-05T23:58:53.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:53.980+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:53.981425+0000) 2020-02-05T23:58:56.171 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:58:56.169+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.170639+0000) 2020-02-05T23:58:56.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:58:56.171+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.172206+0000) 2020-02-05T23:58:56.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:56.171+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.172567+0000) 2020-02-05T23:58:56.174 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:58:56.171+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.173229+0000) 2020-02-05T23:58:56.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:58:56.172+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.173668+0000) 2020-02-05T23:58:56.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:56.172+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.174195+0000) 2020-02-05T23:58:56.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:56.173+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.174431+0000) 2020-02-05T23:58:56.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:56.173+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.175167+0000) 2020-02-05T23:58:56.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:56.174+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.175593+0000) 2020-02-05T23:58:56.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:56.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.176124+0000) 2020-02-05T23:58:56.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:58:56.175+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.176229+0000) 2020-02-05T23:58:56.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:56.175+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.176683+0000) 2020-02-05T23:58:56.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:58:56.175+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.176906+0000) 2020-02-05T23:58:56.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:56.179+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.180791+0000) 2020-02-05T23:58:56.189 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:58:56.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:58:56.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:58:56.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:56.960672+0000) 2020-02-05T23:58:57.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:58:57.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:57.221391+0000) 2020-02-05T23:58:57.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:58:57.220+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:57.221395+0000) 2020-02-05T23:58:57.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:58:57.220+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:57.221531+0000) 2020-02-05T23:58:59.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:58:59.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:59.219296+0000) 2020-02-05T23:58:59.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:58:59.603+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:58:59.603865+0000) 2020-02-05T23:59:00.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:00.450+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:00.450995+0000) 2020-02-05T23:59:00.615 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:59:00.621 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:59:00.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:00.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:00.712454+0000) 2020-02-05T23:59:01.194 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:01.191+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.193027+0000) 2020-02-05T23:59:01.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:01.194+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.195971+0000) 2020-02-05T23:59:01.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:01.195+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.196442+0000) 2020-02-05T23:59:01.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:01.195+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.196919+0000) 2020-02-05T23:59:01.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:01.195+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.197332+0000) 2020-02-05T23:59:01.199 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:01.196+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.197526+0000) 2020-02-05T23:59:01.199 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:01.197+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.198287+0000) 2020-02-05T23:59:01.199 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:01.197+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.198459+0000) 2020-02-05T23:59:01.200 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:01.197+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.199120+0000) 2020-02-05T23:59:01.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:01.198+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.199083+0000) 2020-02-05T23:59:01.201 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:01.198+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.199757+0000) 2020-02-05T23:59:01.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:01.206+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.207511+0000) 2020-02-05T23:59:01.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:01.206+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.207887+0000) 2020-02-05T23:59:01.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:01.207+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.208250+0000) 2020-02-05T23:59:01.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:01.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:01.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:01.619+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:01.619616+0000) 2020-02-05T23:59:02.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:02.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:02.817335+0000) 2020-02-05T23:59:02.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:02.830+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:02.831554+0000) 2020-02-05T23:59:03.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:03.102+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:03.103765+0000) 2020-02-05T23:59:03.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:03.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:03.981689+0000) 2020-02-05T23:59:03.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:03.981+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:03.981690+0000) 2020-02-05T23:59:06.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:06.216+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.217425+0000) 2020-02-05T23:59:06.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:06.217+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.218111+0000) 2020-02-05T23:59:06.224 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:06.218+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.219466+0000) 2020-02-05T23:59:06.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:06.217+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.218687+0000) 2020-02-05T23:59:06.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:06.219+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.220541+0000) 2020-02-05T23:59:06.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:06.220+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.221857+0000) 2020-02-05T23:59:06.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:06.220+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.221843+0000) 2020-02-05T23:59:06.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:06.220+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.222180+0000) 2020-02-05T23:59:06.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:06.221+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.222981+0000) 2020-02-05T23:59:06.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:06.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.223236+0000) 2020-02-05T23:59:06.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:06.223+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.224388+0000) 2020-02-05T23:59:06.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:06.222+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.223427+0000) 2020-02-05T23:59:06.228 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:06.222+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.223632+0000) 2020-02-05T23:59:06.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:06.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.227629+0000) 2020-02-05T23:59:06.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:06.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:06.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:06.959+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:06.960926+0000) 2020-02-05T23:59:07.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:07.220+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:07.221648+0000) 2020-02-05T23:59:07.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:07.220+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:07.221654+0000) 2020-02-05T23:59:07.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:07.220+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:07.221774+0000) 2020-02-05T23:59:09.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:09.218+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:09.219577+0000) 2020-02-05T23:59:09.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:09.603+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:09.604136+0000) 2020-02-05T23:59:10.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:10.450+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:10.451225+0000) 2020-02-05T23:59:10.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:10.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:10.712708+0000) 2020-02-05T23:59:11.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:11.238+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.239121+0000) 2020-02-05T23:59:11.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:11.238+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.239520+0000) 2020-02-05T23:59:11.241 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:11.239+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.240301+0000) 2020-02-05T23:59:11.243 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:11.241+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.242465+0000) 2020-02-05T23:59:11.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:11.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.242372+0000) 2020-02-05T23:59:11.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:11.242+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.243523+0000) 2020-02-05T23:59:11.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:11.242+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.243886+0000) 2020-02-05T23:59:11.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:11.243+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.244678+0000) 2020-02-05T23:59:11.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:11.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.244843+0000) 2020-02-05T23:59:11.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:11.243+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.245042+0000) 2020-02-05T23:59:11.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:11.244+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.245789+0000) 2020-02-05T23:59:11.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:11.244+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.245883+0000) 2020-02-05T23:59:11.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:11.244+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.246361+0000) 2020-02-05T23:59:11.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:11.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.252513+0000) 2020-02-05T23:59:11.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:11.256+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:11.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:11.627+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:11.627852+0000) 2020-02-05T23:59:12.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:12.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:12.817565+0000) 2020-02-05T23:59:12.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:12.831+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:12.831816+0000) 2020-02-05T23:59:13.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:13.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:13.104016+0000) 2020-02-05T23:59:13.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:13.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:13.981957+0000) 2020-02-05T23:59:13.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:13.981+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:13.981957+0000) 2020-02-05T23:59:16.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:16.263+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.263464+0000) 2020-02-05T23:59:16.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:16.263+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.264496+0000) 2020-02-05T23:59:16.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:16.264+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.264818+0000) 2020-02-05T23:59:16.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:16.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.265426+0000) 2020-02-05T23:59:16.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:16.266+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.266122+0000) 2020-02-05T23:59:16.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:16.267+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.267489+0000) 2020-02-05T23:59:16.269 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:16.266+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.267509+0000) 2020-02-05T23:59:16.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:16.266+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.268093+0000) 2020-02-05T23:59:16.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:16.268+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.268427+0000) 2020-02-05T23:59:16.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:16.267+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.269050+0000) 2020-02-05T23:59:16.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:16.267+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.269192+0000) 2020-02-05T23:59:16.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:16.268+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.269714+0000) 2020-02-05T23:59:16.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:16.269+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.269648+0000) 2020-02-05T23:59:16.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:16.274+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.274380+0000) 2020-02-05T23:59:16.280 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:16.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:16.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:16.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:16.961117+0000) 2020-02-05T23:59:17.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:17.221+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:17.221893+0000) 2020-02-05T23:59:17.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:17.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:17.221895+0000) 2020-02-05T23:59:17.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:17.221+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:17.222025+0000) 2020-02-05T23:59:19.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:19.219+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:19.219860+0000) 2020-02-05T23:59:19.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:19.603+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:19.604372+0000) 2020-02-05T23:59:20.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:20.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:20.451470+0000) 2020-02-05T23:59:20.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:20.711+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:20.712939+0000) 2020-02-05T23:59:21.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:21.285+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.286280+0000) 2020-02-05T23:59:21.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:21.285+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.286976+0000) 2020-02-05T23:59:21.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:21.286+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.286846+0000) 2020-02-05T23:59:21.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:21.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.287557+0000) 2020-02-05T23:59:21.289 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:21.286+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.287865+0000) 2020-02-05T23:59:21.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:21.287+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.288065+0000) 2020-02-05T23:59:21.290 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:21.287+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.288515+0000) 2020-02-05T23:59:21.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:21.288+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.289168+0000) 2020-02-05T23:59:21.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:21.288+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.289374+0000) 2020-02-05T23:59:21.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:21.289+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.289674+0000) 2020-02-05T23:59:21.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:21.289+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.289921+0000) 2020-02-05T23:59:21.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:21.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.289506+0000) 2020-02-05T23:59:21.292 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:21.289+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.290322+0000) 2020-02-05T23:59:21.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:21.293+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.294758+0000) 2020-02-05T23:59:21.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:21.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:21.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:21.635+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:21.636613+0000) 2020-02-05T23:59:22.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:22.816+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:22.817794+0000) 2020-02-05T23:59:22.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:22.831+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:22.832061+0000) 2020-02-05T23:59:23.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:23.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:23.104287+0000) 2020-02-05T23:59:23.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:23.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:23.982214+0000) 2020-02-05T23:59:23.984 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:23.981+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:23.982234+0000) 2020-02-05T23:59:26.311 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:26.308+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.309138+0000) 2020-02-05T23:59:26.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:26.308+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.309918+0000) 2020-02-05T23:59:26.311 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:26.309+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.310417+0000) 2020-02-05T23:59:26.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:26.309+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.311013+0000) 2020-02-05T23:59:26.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:26.311+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.311787+0000) 2020-02-05T23:59:26.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:26.312+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.312419+0000) 2020-02-05T23:59:26.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:26.311+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.312050+0000) 2020-02-05T23:59:26.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:26.312+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.313214+0000) 2020-02-05T23:59:26.314 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:26.312+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.313483+0000) 2020-02-05T23:59:26.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:26.313+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.313638+0000) 2020-02-05T23:59:26.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:26.313+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.314055+0000) 2020-02-05T23:59:26.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:26.313+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.314173+0000) 2020-02-05T23:59:26.316 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:26.313+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.315032+0000) 2020-02-05T23:59:26.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:26.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.317170+0000) 2020-02-05T23:59:26.329 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:26.327+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:26.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:26.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:26.961324+0000) 2020-02-05T23:59:27.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:27.221+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:27.222106+0000) 2020-02-05T23:59:27.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:27.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:27.222178+0000) 2020-02-05T23:59:27.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:27.221+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:27.222252+0000) 2020-02-05T23:59:29.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:29.219+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:29.220123+0000) 2020-02-05T23:59:29.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:29.603+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:29.604598+0000) 2020-02-05T23:59:30.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:30.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:30.451719+0000) 2020-02-05T23:59:30.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:30.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:30.713215+0000) 2020-02-05T23:59:30.801 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:59:30.807 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-05T23:59:31.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:31.332+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.333604+0000) 2020-02-05T23:59:31.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:31.333+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.334513+0000) 2020-02-05T23:59:31.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:31.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.334950+0000) 2020-02-05T23:59:31.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:31.334+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.335311+0000) 2020-02-05T23:59:31.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:31.334+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.335459+0000) 2020-02-05T23:59:31.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:31.335+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.336047+0000) 2020-02-05T23:59:31.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:31.335+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.336626+0000) 2020-02-05T23:59:31.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:31.336+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.336731+0000) 2020-02-05T23:59:31.339 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:31.336+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.337190+0000) 2020-02-05T23:59:31.339 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:31.336+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.337221+0000) 2020-02-05T23:59:31.339 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:31.337+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.337592+0000) 2020-02-05T23:59:31.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:31.337+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.337945+0000) 2020-02-05T23:59:31.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:31.337+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.338240+0000) 2020-02-05T23:59:31.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:31.341+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.342935+0000) 2020-02-05T23:59:31.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:31.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160832 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:31.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:31.644+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:31.645314+0000) 2020-02-05T23:59:32.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:32.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:32.818014+0000) 2020-02-05T23:59:32.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:32.831+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:32.832328+0000) 2020-02-05T23:59:33.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:33.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:33.104511+0000) 2020-02-05T23:59:33.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:33.981+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:33.982494+0000) 2020-02-05T23:59:33.984 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:33.981+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:33.982494+0000) 2020-02-05T23:59:36.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:36.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.357877+0000) 2020-02-05T23:59:36.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:36.357+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.358571+0000) 2020-02-05T23:59:36.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:36.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.360251+0000) 2020-02-05T23:59:36.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:36.360+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.360984+0000) 2020-02-05T23:59:36.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:36.360+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.361071+0000) 2020-02-05T23:59:36.363 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:36.360+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.361378+0000) 2020-02-05T23:59:36.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:36.360+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.361870+0000) 2020-02-05T23:59:36.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:36.360+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.362026+0000) 2020-02-05T23:59:36.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:36.361+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.362756+0000) 2020-02-05T23:59:36.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:36.363+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.363699+0000) 2020-02-05T23:59:36.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:36.363+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.363783+0000) 2020-02-05T23:59:36.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:36.362+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.363944+0000) 2020-02-05T23:59:36.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:36.363+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.364308+0000) 2020-02-05T23:59:36.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:36.366+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.367660+0000) 2020-02-05T23:59:36.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:36.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:36.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:36.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:36.961527+0000) 2020-02-05T23:59:37.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:37.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:37.222336+0000) 2020-02-05T23:59:37.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:37.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:37.222544+0000) 2020-02-05T23:59:37.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:37.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:37.222496+0000) 2020-02-05T23:59:39.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:39.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:39.220379+0000) 2020-02-05T23:59:39.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:39.604+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:39.604861+0000) 2020-02-05T23:59:40.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:40.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:40.451977+0000) 2020-02-05T23:59:40.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:40.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:40.713405+0000) 2020-02-05T23:59:41.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:41.379+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.380920+0000) 2020-02-05T23:59:41.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:41.381+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.383037+0000) 2020-02-05T23:59:41.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:41.382+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.383378+0000) 2020-02-05T23:59:41.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:41.383+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.383896+0000) 2020-02-05T23:59:41.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:41.382+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.383720+0000) 2020-02-05T23:59:41.386 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:41.382+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.383385+0000) 2020-02-05T23:59:41.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:41.383+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.384300+0000) 2020-02-05T23:59:41.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:41.384+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.384882+0000) 2020-02-05T23:59:41.387 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:41.384+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.385031+0000) 2020-02-05T23:59:41.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:41.385+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.385476+0000) 2020-02-05T23:59:41.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:41.385+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.386602+0000) 2020-02-05T23:59:41.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:41.386+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.387002+0000) 2020-02-05T23:59:41.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:41.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.387558+0000) 2020-02-05T23:59:41.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:41.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.389565+0000) 2020-02-05T23:59:41.399 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:41.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:41.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:41.652+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:41.653238+0000) 2020-02-05T23:59:42.819 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:42.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:42.818270+0000) 2020-02-05T23:59:42.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:42.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:42.832600+0000) 2020-02-05T23:59:43.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:43.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:43.104725+0000) 2020-02-05T23:59:43.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:43.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:43.982768+0000) 2020-02-05T23:59:43.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:43.982+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:43.982768+0000) 2020-02-05T23:59:46.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:46.403+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.404423+0000) 2020-02-05T23:59:46.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:46.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.405956+0000) 2020-02-05T23:59:46.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:46.406+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.406772+0000) 2020-02-05T23:59:46.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:46.406+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.406631+0000) 2020-02-05T23:59:46.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:46.406+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.407120+0000) 2020-02-05T23:59:46.409 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:46.406+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.407516+0000) 2020-02-05T23:59:46.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:46.407+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.408438+0000) 2020-02-05T23:59:46.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:46.407+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.408958+0000) 2020-02-05T23:59:46.410 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:46.409+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.409755+0000) 2020-02-05T23:59:46.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:46.408+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.409396+0000) 2020-02-05T23:59:46.411 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:46.409+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.410079+0000) 2020-02-05T23:59:46.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:46.409+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.410690+0000) 2020-02-05T23:59:46.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:46.410+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.410840+0000) 2020-02-05T23:59:46.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:46.413+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.414434+0000) 2020-02-05T23:59:46.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:46.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160956 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:46.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:46.960+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:46.961793+0000) 2020-02-05T23:59:47.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:47.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:47.222560+0000) 2020-02-05T23:59:47.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:47.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:47.222714+0000) 2020-02-05T23:59:47.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:47.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:47.222796+0000) 2020-02-05T23:59:49.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:49.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:49.220661+0000) 2020-02-05T23:59:49.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:49.604+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:49.605058+0000) 2020-02-05T23:59:50.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:50.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:50.452232+0000) 2020-02-05T23:59:50.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:50.712+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:50.713688+0000) 2020-02-05T23:59:51.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:51.428+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.429918+0000) 2020-02-05T23:59:51.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:51.430+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.430414+0000) 2020-02-05T23:59:51.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:51.429+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.430870+0000) 2020-02-05T23:59:51.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:51.431+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.431763+0000) 2020-02-05T23:59:51.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:51.430+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.432040+0000) 2020-02-05T23:59:51.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:51.431+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.432370+0000) 2020-02-05T23:59:51.434 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:51.431+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.432689+0000) 2020-02-05T23:59:51.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:51.433+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.433414+0000) 2020-02-05T23:59:51.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:51.433+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.433715+0000) 2020-02-05T23:59:51.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:51.432+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.433907+0000) 2020-02-05T23:59:51.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:51.433+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.434513+0000) 2020-02-05T23:59:51.436 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:51.434+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.434829+0000) 2020-02-05T23:59:51.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:51.434+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.435359+0000) 2020-02-05T23:59:51.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:51.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.439172+0000) 2020-02-05T23:59:51.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:51.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 160998 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:51.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:51.660+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:51.661246+0000) 2020-02-05T23:59:52.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:52.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:52.818502+0000) 2020-02-05T23:59:52.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:52.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:52.832942+0000) 2020-02-05T23:59:53.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:53.103+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:53.104911+0000) 2020-02-05T23:59:53.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:53.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:53.983041+0000) 2020-02-05T23:59:53.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:53.982+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:53.983041+0000) 2020-02-05T23:59:56.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-05T23:59:56.448+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.449978+0000) 2020-02-05T23:59:56.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-05T23:59:56.449+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.451049+0000) 2020-02-05T23:59:56.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:56.450+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.451608+0000) 2020-02-05T23:59:56.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:56.452+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.452491+0000) 2020-02-05T23:59:56.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:56.451+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.453018+0000) 2020-02-05T23:59:56.454 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-05T23:59:56.452+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.453803+0000) 2020-02-05T23:59:56.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-05T23:59:56.454+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.454462+0000) 2020-02-05T23:59:56.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:56.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.455409+0000) 2020-02-05T23:59:56.457 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:56.455+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.455928+0000) 2020-02-05T23:59:56.458 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-05T23:59:56.455+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.456596+0000) 2020-02-05T23:59:56.458 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-05T23:59:56.456+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.456462+0000) 2020-02-05T23:59:56.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:56.456+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.457095+0000) 2020-02-05T23:59:56.459 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:56.456+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.457262+0000) 2020-02-05T23:59:56.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:56.459+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.460683+0000) 2020-02-05T23:59:56.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-05T23:59:56.469+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161038 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-05T23:59:56.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-05T23:59:56.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:56.962036+0000) 2020-02-05T23:59:57.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-05T23:59:57.221+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:57.222732+0000) 2020-02-05T23:59:57.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-05T23:59:57.221+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:57.222914+0000) 2020-02-05T23:59:57.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-05T23:59:57.221+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:57.223009+0000) 2020-02-05T23:59:59.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-05T23:59:59.219+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:59.220902+0000) 2020-02-05T23:59:59.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-05T23:59:59.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T22:59:59.605365+0000) 2020-02-06T00:00:00.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:00.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:00.452564+0000) 2020-02-06T00:00:00.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:00.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:00.713920+0000) 2020-02-06T00:00:00.980 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:00:00.987 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:00:01.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:01.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.474688+0000) 2020-02-06T00:00:01.476 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:01.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.475456+0000) 2020-02-06T00:00:01.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:01.474+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.475997+0000) 2020-02-06T00:00:01.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:01.475+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.476151+0000) 2020-02-06T00:00:01.478 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:01.475+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.476752+0000) 2020-02-06T00:00:01.478 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:01.476+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.477744+0000) 2020-02-06T00:00:01.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:01.477+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.479145+0000) 2020-02-06T00:00:01.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:01.478+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.479926+0000) 2020-02-06T00:00:01.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:01.479+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.480698+0000) 2020-02-06T00:00:01.482 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:01.479+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.480861+0000) 2020-02-06T00:00:01.482 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:01.480+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.481562+0000) 2020-02-06T00:00:01.483 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:01.479+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.481089+0000) 2020-02-06T00:00:01.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:01.482+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.483963+0000) 2020-02-06T00:00:01.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:01.483+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.484197+0000) 2020-02-06T00:00:01.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:01.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:01.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:01.669+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:01.669403+0000) 2020-02-06T00:00:02.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:02.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:02.818739+0000) 2020-02-06T00:00:02.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:02.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:02.833211+0000) 2020-02-06T00:00:03.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:03.104+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:03.105196+0000) 2020-02-06T00:00:03.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:03.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:03.983314+0000) 2020-02-06T00:00:03.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:03.982+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:03.983314+0000) 2020-02-06T00:00:06.501 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:06.498+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.500000+0000) 2020-02-06T00:00:06.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:06.499+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.500427+0000) 2020-02-06T00:00:06.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:06.500+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.501606+0000) 2020-02-06T00:00:06.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:06.499+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.501371+0000) 2020-02-06T00:00:06.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:06.501+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.502210+0000) 2020-02-06T00:00:06.504 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:06.501+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.502628+0000) 2020-02-06T00:00:06.504 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:06.501+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.502997+0000) 2020-02-06T00:00:06.505 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:06.502+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.503978+0000) 2020-02-06T00:00:06.506 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:06.503+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.505064+0000) 2020-02-06T00:00:06.507 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:06.503+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.505180+0000) 2020-02-06T00:00:06.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:06.504+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.505485+0000) 2020-02-06T00:00:06.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:06.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.504892+0000) 2020-02-06T00:00:06.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:06.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.505387+0000) 2020-02-06T00:00:06.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:06.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.511841+0000) 2020-02-06T00:00:06.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:06.515+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161122 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:06.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:06.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:06.962249+0000) 2020-02-06T00:00:07.224 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:07.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:07.222969+0000) 2020-02-06T00:00:07.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:07.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:07.223124+0000) 2020-02-06T00:00:07.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:07.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:07.223215+0000) 2020-02-06T00:00:09.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:09.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:09.221164+0000) 2020-02-06T00:00:09.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:09.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:09.605605+0000) 2020-02-06T00:00:10.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:10.451+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:10.452796+0000) 2020-02-06T00:00:10.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:10.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:10.714137+0000) 2020-02-06T00:00:11.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:11.522+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.522550+0000) 2020-02-06T00:00:11.525 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:11.523+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.524031+0000) 2020-02-06T00:00:11.526 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:11.523+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.525027+0000) 2020-02-06T00:00:11.526 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:11.525+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.525455+0000) 2020-02-06T00:00:11.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:11.524+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.525600+0000) 2020-02-06T00:00:11.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:11.525+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.526029+0000) 2020-02-06T00:00:11.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:11.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.526446+0000) 2020-02-06T00:00:11.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:11.525+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.526924+0000) 2020-02-06T00:00:11.528 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:11.525+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.527264+0000) 2020-02-06T00:00:11.529 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:11.527+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.527908+0000) 2020-02-06T00:00:11.529 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:11.526+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.528214+0000) 2020-02-06T00:00:11.529 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:11.527+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.528630+0000) 2020-02-06T00:00:11.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:11.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.528768+0000) 2020-02-06T00:00:11.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:11.534+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.534411+0000) 2020-02-06T00:00:11.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:11.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:11.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:11.680+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:11.680402+0000) 2020-02-06T00:00:12.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:12.817+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:12.818978+0000) 2020-02-06T00:00:12.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:12.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:12.833504+0000) 2020-02-06T00:00:13.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:13.104+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:13.105447+0000) 2020-02-06T00:00:13.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:13.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:13.983589+0000) 2020-02-06T00:00:13.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:13.982+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:13.983589+0000) 2020-02-06T00:00:16.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:16.544+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.545800+0000) 2020-02-06T00:00:16.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:16.546+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.547177+0000) 2020-02-06T00:00:16.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:16.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.547874+0000) 2020-02-06T00:00:16.549 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:16.547+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.548618+0000) 2020-02-06T00:00:16.550 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:16.547+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.548547+0000) 2020-02-06T00:00:16.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:16.548+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.549619+0000) 2020-02-06T00:00:16.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:16.549+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.550314+0000) 2020-02-06T00:00:16.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:16.548+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.550112+0000) 2020-02-06T00:00:16.552 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:16.549+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.550933+0000) 2020-02-06T00:00:16.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:16.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.551398+0000) 2020-02-06T00:00:16.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:16.550+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.551924+0000) 2020-02-06T00:00:16.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:16.551+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.552649+0000) 2020-02-06T00:00:16.554 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:16.551+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.552762+0000) 2020-02-06T00:00:16.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:16.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:16.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:16.558+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.559544+0000) 2020-02-06T00:00:16.963 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:16.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:16.962461+0000) 2020-02-06T00:00:17.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:17.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:17.223216+0000) 2020-02-06T00:00:17.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:17.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:17.223375+0000) 2020-02-06T00:00:17.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:17.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:17.223473+0000) 2020-02-06T00:00:19.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:19.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:19.221406+0000) 2020-02-06T00:00:19.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:19.604+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:19.605833+0000) 2020-02-06T00:00:20.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:20.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:20.453051+0000) 2020-02-06T00:00:20.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:20.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:20.714444+0000) 2020-02-06T00:00:21.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:21.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.568906+0000) 2020-02-06T00:00:21.570 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:21.568+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.569500+0000) 2020-02-06T00:00:21.572 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:21.569+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.570939+0000) 2020-02-06T00:00:21.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:21.569+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.571403+0000) 2020-02-06T00:00:21.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:21.570+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.571618+0000) 2020-02-06T00:00:21.573 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:21.569+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.571280+0000) 2020-02-06T00:00:21.573 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:21.570+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.571894+0000) 2020-02-06T00:00:21.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:21.571+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.572346+0000) 2020-02-06T00:00:21.574 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:21.572+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.573217+0000) 2020-02-06T00:00:21.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:21.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.573370+0000) 2020-02-06T00:00:21.575 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:21.572+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.573823+0000) 2020-02-06T00:00:21.575 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:21.572+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.573949+0000) 2020-02-06T00:00:21.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:21.573+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.574667+0000) 2020-02-06T00:00:21.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:21.578+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.579869+0000) 2020-02-06T00:00:21.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:21.585+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161245 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:21.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:21.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:21.688677+0000) 2020-02-06T00:00:22.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:22.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:22.819205+0000) 2020-02-06T00:00:22.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:22.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:22.833751+0000) 2020-02-06T00:00:23.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:23.104+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:23.105683+0000) 2020-02-06T00:00:23.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:23.982+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:23.983844+0000) 2020-02-06T00:00:23.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:23.982+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:23.983840+0000) 2020-02-06T00:00:26.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:26.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.592539+0000) 2020-02-06T00:00:26.594 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:26.592+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.593568+0000) 2020-02-06T00:00:26.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:26.592+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.594001+0000) 2020-02-06T00:00:26.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:26.593+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.594194+0000) 2020-02-06T00:00:26.596 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:26.593+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.595213+0000) 2020-02-06T00:00:26.596 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:26.594+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.595208+0000) 2020-02-06T00:00:26.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:26.594+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.595516+0000) 2020-02-06T00:00:26.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:26.594+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.595935+0000) 2020-02-06T00:00:26.598 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:26.595+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.596577+0000) 2020-02-06T00:00:26.598 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:26.595+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.596506+0000) 2020-02-06T00:00:26.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:26.595+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.597030+0000) 2020-02-06T00:00:26.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:26.595+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.597327+0000) 2020-02-06T00:00:26.599 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:26.596+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.597278+0000) 2020-02-06T00:00:26.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:26.601+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.602404+0000) 2020-02-06T00:00:26.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:26.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161287 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:26.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:26.961+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:26.962669+0000) 2020-02-06T00:00:27.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:27.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:27.223477+0000) 2020-02-06T00:00:27.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:27.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:27.223604+0000) 2020-02-06T00:00:27.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:27.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:27.223716+0000) 2020-02-06T00:00:29.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:29.220+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:29.221700+0000) 2020-02-06T00:00:29.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:29.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:29.606048+0000) 2020-02-06T00:00:30.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:30.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:30.453292+0000) 2020-02-06T00:00:30.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:30.713+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:30.714680+0000) 2020-02-06T00:00:31.212 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:00:31.222 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:00:31.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:31.613+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.614784+0000) 2020-02-06T00:00:31.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:31.614+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.615710+0000) 2020-02-06T00:00:31.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:31.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.617550+0000) 2020-02-06T00:00:31.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:31.616+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.617625+0000) 2020-02-06T00:00:31.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:31.616+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.618103+0000) 2020-02-06T00:00:31.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:31.617+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.619030+0000) 2020-02-06T00:00:31.621 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:31.618+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.620040+0000) 2020-02-06T00:00:31.622 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:31.618+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.619907+0000) 2020-02-06T00:00:31.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:31.619+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.620995+0000) 2020-02-06T00:00:31.623 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:31.619+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.621290+0000) 2020-02-06T00:00:31.623 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:31.620+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.621694+0000) 2020-02-06T00:00:31.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:31.620+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.621769+0000) 2020-02-06T00:00:31.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:31.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.624223+0000) 2020-02-06T00:00:31.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:31.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.624470+0000) 2020-02-06T00:00:31.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:31.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161329 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:31.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:31.696+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:31.697032+0000) 2020-02-06T00:00:32.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:32.818+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:32.819442+0000) 2020-02-06T00:00:32.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:32.833+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:32.834073+0000) 2020-02-06T00:00:33.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:33.105+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:33.105911+0000) 2020-02-06T00:00:33.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:33.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:33.984143+0000) 2020-02-06T00:00:33.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:33.983+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:33.984143+0000) 2020-02-06T00:00:36.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:36.638+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.639272+0000) 2020-02-06T00:00:36.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:36.639+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.640319+0000) 2020-02-06T00:00:36.642 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:36.639+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.641012+0000) 2020-02-06T00:00:36.642 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:36.640+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.641355+0000) 2020-02-06T00:00:36.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:36.640+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.641827+0000) 2020-02-06T00:00:36.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:36.641+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.642538+0000) 2020-02-06T00:00:36.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:36.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.642863+0000) 2020-02-06T00:00:36.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:36.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.643534+0000) 2020-02-06T00:00:36.645 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:36.642+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.643708+0000) 2020-02-06T00:00:36.645 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:36.642+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.644051+0000) 2020-02-06T00:00:36.646 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:36.643+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.644766+0000) 2020-02-06T00:00:36.646 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:36.643+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.645193+0000) 2020-02-06T00:00:36.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:36.644+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.645962+0000) 2020-02-06T00:00:36.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:36.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.651466+0000) 2020-02-06T00:00:36.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:36.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:36.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:36.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:36.962886+0000) 2020-02-06T00:00:37.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:37.222+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:37.223653+0000) 2020-02-06T00:00:37.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:37.222+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:37.223769+0000) 2020-02-06T00:00:37.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:37.222+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:37.223990+0000) 2020-02-06T00:00:39.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:39.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:39.221955+0000) 2020-02-06T00:00:39.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:39.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:39.606338+0000) 2020-02-06T00:00:40.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:40.452+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:40.453521+0000) 2020-02-06T00:00:40.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:40.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:40.714945+0000) 2020-02-06T00:00:41.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:41.662+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.664094+0000) 2020-02-06T00:00:41.666 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:41.664+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.665731+0000) 2020-02-06T00:00:41.667 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:41.664+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.666023+0000) 2020-02-06T00:00:41.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:41.665+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.666455+0000) 2020-02-06T00:00:41.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:41.665+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.666664+0000) 2020-02-06T00:00:41.668 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:41.665+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.667101+0000) 2020-02-06T00:00:41.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:41.666+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.667558+0000) 2020-02-06T00:00:41.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:41.666+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.667976+0000) 2020-02-06T00:00:41.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:41.667+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.668490+0000) 2020-02-06T00:00:41.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:41.668+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.669215+0000) 2020-02-06T00:00:41.670 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:41.668+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.669817+0000) 2020-02-06T00:00:41.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:41.668+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.669829+0000) 2020-02-06T00:00:41.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:41.668+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.669984+0000) 2020-02-06T00:00:41.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:41.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.673909+0000) 2020-02-06T00:00:41.684 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:41.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161411 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:41.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:41.705+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:41.706228+0000) 2020-02-06T00:00:42.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:42.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:42.819669+0000) 2020-02-06T00:00:42.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:42.832+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:42.834322+0000) 2020-02-06T00:00:43.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:43.105+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:43.106137+0000) 2020-02-06T00:00:43.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:43.983+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:43.984372+0000) 2020-02-06T00:00:43.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:43.983+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:43.984372+0000) 2020-02-06T00:00:46.690 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:46.688+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.689425+0000) 2020-02-06T00:00:46.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:46.688+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.689691+0000) 2020-02-06T00:00:46.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:46.689+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.690224+0000) 2020-02-06T00:00:46.692 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:46.689+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.691088+0000) 2020-02-06T00:00:46.692 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:46.690+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.691507+0000) 2020-02-06T00:00:46.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:46.690+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.691546+0000) 2020-02-06T00:00:46.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:46.690+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.691863+0000) 2020-02-06T00:00:46.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:46.690+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.692111+0000) 2020-02-06T00:00:46.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:46.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.692492+0000) 2020-02-06T00:00:46.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:46.691+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.692882+0000) 2020-02-06T00:00:46.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:46.691+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.693192+0000) 2020-02-06T00:00:46.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:46.692+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.693543+0000) 2020-02-06T00:00:46.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:46.693+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.694896+0000) 2020-02-06T00:00:46.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:46.697+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.698936+0000) 2020-02-06T00:00:46.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:46.706+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161452 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:46.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:46.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:46.963129+0000) 2020-02-06T00:00:47.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:47.223+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:47.223882+0000) 2020-02-06T00:00:47.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:47.223+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:47.223988+0000) 2020-02-06T00:00:47.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:47.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:47.224298+0000) 2020-02-06T00:00:49.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:49.221+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:49.222209+0000) 2020-02-06T00:00:49.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:49.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:49.606527+0000) 2020-02-06T00:00:50.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:50.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:50.453750+0000) 2020-02-06T00:00:50.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:50.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:50.715181+0000) 2020-02-06T00:00:51.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:51.710+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.711921+0000) 2020-02-06T00:00:51.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:51.713+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.714268+0000) 2020-02-06T00:00:51.716 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:51.714+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.715253+0000) 2020-02-06T00:00:51.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:51.715+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.716063+0000) 2020-02-06T00:00:51.717 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:51.715+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.716032+0000) 2020-02-06T00:00:51.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:51.715+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.716757+0000) 2020-02-06T00:00:51.718 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:51.716+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.717110+0000) 2020-02-06T00:00:51.718 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:51.716+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.717307+0000) 2020-02-06T00:00:51.720 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:51.718+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.719058+0000) 2020-02-06T00:00:51.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:51.718+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.719440+0000) 2020-02-06T00:00:51.721 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:51.718+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.719771+0000) 2020-02-06T00:00:51.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:51.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.720005+0000) 2020-02-06T00:00:51.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:51.719+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.720131+0000) 2020-02-06T00:00:51.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:51.720+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.720538+0000) 2020-02-06T00:00:51.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:51.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:51.728091+0000) 2020-02-06T00:00:51.733 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:51.731+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:52.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:52.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:52.819897+0000) 2020-02-06T00:00:52.835 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:52.834+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:52.834579+0000) 2020-02-06T00:00:53.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:53.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:53.106355+0000) 2020-02-06T00:00:53.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:53.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:53.984636+0000) 2020-02-06T00:00:53.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:53.984+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:53.984637+0000) 2020-02-06T00:00:56.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:56.738+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.738929+0000) 2020-02-06T00:00:56.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:00:56.739+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.740368+0000) 2020-02-06T00:00:56.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:00:56.740+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.741155+0000) 2020-02-06T00:00:56.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:56.739+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.740175+0000) 2020-02-06T00:00:56.743 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:00:56.740+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.740898+0000) 2020-02-06T00:00:56.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:56.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.741919+0000) 2020-02-06T00:00:56.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:00:56.741+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.742267+0000) 2020-02-06T00:00:56.744 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:56.741+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.741768+0000) 2020-02-06T00:00:56.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:56.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.743387+0000) 2020-02-06T00:00:56.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:56.742+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.743924+0000) 2020-02-06T00:00:56.745 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:00:56.743+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.744216+0000) 2020-02-06T00:00:56.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:00:56.743+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.743976+0000) 2020-02-06T00:00:56.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:56.743+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.744243+0000) 2020-02-06T00:00:56.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:56.746+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.748068+0000) 2020-02-06T00:00:56.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:00:56.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:00:56.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:00:56.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:56.963348+0000) 2020-02-06T00:00:57.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:00:57.223+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:57.224114+0000) 2020-02-06T00:00:57.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:00:57.223+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:57.224250+0000) 2020-02-06T00:00:57.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:00:57.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:57.224497+0000) 2020-02-06T00:00:59.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:00:59.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:59.222473+0000) 2020-02-06T00:00:59.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:00:59.606+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:00:59.606774+0000) 2020-02-06T00:01:00.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:00.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:00.454009+0000) 2020-02-06T00:01:00.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:00.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:00.715411+0000) 2020-02-06T00:01:01.433 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:01:01.439 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:01:01.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:01.722+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.722223+0000) 2020-02-06T00:01:01.763 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:01.761+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.761806+0000) 2020-02-06T00:01:01.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:01.761+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.762149+0000) 2020-02-06T00:01:01.764 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:01.762+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.762913+0000) 2020-02-06T00:01:01.768 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:01.763+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.764167+0000) 2020-02-06T00:01:01.770 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:01.763+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.764339+0000) 2020-02-06T00:01:01.770 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:01.765+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.765593+0000) 2020-02-06T00:01:01.770 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:01.765+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.765857+0000) 2020-02-06T00:01:01.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:01.766+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.766701+0000) 2020-02-06T00:01:01.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:01.763+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.764628+0000) 2020-02-06T00:01:01.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:01.764+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.765420+0000) 2020-02-06T00:01:01.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:01.765+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.766685+0000) 2020-02-06T00:01:01.771 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:01.767+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.767157+0000) 2020-02-06T00:01:01.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:01.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.771240+0000) 2020-02-06T00:01:01.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:01.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:01.771525+0000) 2020-02-06T00:01:01.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:01.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:02.821 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:02.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:02.820149+0000) 2020-02-06T00:01:02.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:02.834+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:02.834814+0000) 2020-02-06T00:01:03.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:03.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:03.106575+0000) 2020-02-06T00:01:03.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:03.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:03.984935+0000) 2020-02-06T00:01:03.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:03.984+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:03.984935+0000) 2020-02-06T00:01:06.784 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:06.782+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.783140+0000) 2020-02-06T00:01:06.785 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:06.783+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.784090+0000) 2020-02-06T00:01:06.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:06.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.785152+0000) 2020-02-06T00:01:06.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:06.784+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.784665+0000) 2020-02-06T00:01:06.788 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:06.786+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.787147+0000) 2020-02-06T00:01:06.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:06.786+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.787391+0000) 2020-02-06T00:01:06.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:06.786+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.787578+0000) 2020-02-06T00:01:06.789 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:06.786+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.787977+0000) 2020-02-06T00:01:06.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:06.787+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.788364+0000) 2020-02-06T00:01:06.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:06.788+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.788973+0000) 2020-02-06T00:01:06.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:06.788+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.789242+0000) 2020-02-06T00:01:06.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:06.788+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.789528+0000) 2020-02-06T00:01:06.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:06.789+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.789852+0000) 2020-02-06T00:01:06.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:06.792+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.793593+0000) 2020-02-06T00:01:06.802 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:06.800+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:06.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:06.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:06.963555+0000) 2020-02-06T00:01:07.225 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:07.224+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:07.224326+0000) 2020-02-06T00:01:07.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:07.224+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:07.224532+0000) 2020-02-06T00:01:07.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:07.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:07.224727+0000) 2020-02-06T00:01:09.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:09.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:09.222740+0000) 2020-02-06T00:01:09.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:09.605+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:09.607021+0000) 2020-02-06T00:01:10.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:10.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:10.454238+0000) 2020-02-06T00:01:10.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:10.714+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:10.715708+0000) 2020-02-06T00:01:11.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:11.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.730541+0000) 2020-02-06T00:01:11.808 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:11.805+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.806265+0000) 2020-02-06T00:01:11.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:11.807+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.807409+0000) 2020-02-06T00:01:11.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:11.806+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.807601+0000) 2020-02-06T00:01:11.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:11.807+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.808457+0000) 2020-02-06T00:01:11.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:11.807+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.808934+0000) 2020-02-06T00:01:11.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:11.808+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.809295+0000) 2020-02-06T00:01:11.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:11.809+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.809921+0000) 2020-02-06T00:01:11.811 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:11.809+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.810391+0000) 2020-02-06T00:01:11.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:11.809+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.810413+0000) 2020-02-06T00:01:11.812 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:11.810+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.810849+0000) 2020-02-06T00:01:11.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:11.810+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.811229+0000) 2020-02-06T00:01:11.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:11.811+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.811562+0000) 2020-02-06T00:01:11.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:11.812+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.813006+0000) 2020-02-06T00:01:11.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:11.816+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:11.817537+0000) 2020-02-06T00:01:11.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:11.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161659 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:12.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:12.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:12.820382+0000) 2020-02-06T00:01:12.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:12.834+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:12.835076+0000) 2020-02-06T00:01:13.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:13.105+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:13.106816+0000) 2020-02-06T00:01:13.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:13.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:13.985208+0000) 2020-02-06T00:01:13.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:13.984+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:13.985208+0000) 2020-02-06T00:01:16.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:16.832+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.833247+0000) 2020-02-06T00:01:16.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:16.832+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.833513+0000) 2020-02-06T00:01:16.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:16.833+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.834067+0000) 2020-02-06T00:01:16.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:16.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.835113+0000) 2020-02-06T00:01:16.836 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:16.834+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.835801+0000) 2020-02-06T00:01:16.837 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:16.835+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.836356+0000) 2020-02-06T00:01:16.837 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:16.836+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.836581+0000) 2020-02-06T00:01:16.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:16.836+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.836950+0000) 2020-02-06T00:01:16.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:16.836+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.837312+0000) 2020-02-06T00:01:16.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:16.837+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.838014+0000) 2020-02-06T00:01:16.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:16.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.838490+0000) 2020-02-06T00:01:16.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:16.837+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.838666+0000) 2020-02-06T00:01:16.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:16.837+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.838921+0000) 2020-02-06T00:01:16.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:16.844+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.845638+0000) 2020-02-06T00:01:16.851 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:16.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:16.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:16.962+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:16.963780+0000) 2020-02-06T00:01:17.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:17.224+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:17.224557+0000) 2020-02-06T00:01:17.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:17.224+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:17.224746+0000) 2020-02-06T00:01:17.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:17.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:17.224926+0000) 2020-02-06T00:01:19.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:19.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:19.223024+0000) 2020-02-06T00:01:19.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:19.606+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:19.607291+0000) 2020-02-06T00:01:20.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:20.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:20.454479+0000) 2020-02-06T00:01:20.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:20.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:20.715937+0000) 2020-02-06T00:01:21.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:21.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.738765+0000) 2020-02-06T00:01:21.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:21.854+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.855117+0000) 2020-02-06T00:01:21.858 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:21.856+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.857117+0000) 2020-02-06T00:01:21.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:21.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.857483+0000) 2020-02-06T00:01:21.859 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:21.857+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.858367+0000) 2020-02-06T00:01:21.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:21.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.858180+0000) 2020-02-06T00:01:21.860 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:21.857+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.858697+0000) 2020-02-06T00:01:21.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:21.858+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.858968+0000) 2020-02-06T00:01:21.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:21.858+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.859486+0000) 2020-02-06T00:01:21.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:21.859+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.859827+0000) 2020-02-06T00:01:21.861 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:21.859+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.860175+0000) 2020-02-06T00:01:21.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:21.859+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.860832+0000) 2020-02-06T00:01:21.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:21.860+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.860991+0000) 2020-02-06T00:01:21.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:21.860+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.861302+0000) 2020-02-06T00:01:21.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:21.863+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:21.865037+0000) 2020-02-06T00:01:21.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:21.874+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:22.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:22.819+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:22.820611+0000) 2020-02-06T00:01:22.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:22.835+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:22.835343+0000) 2020-02-06T00:01:23.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:23.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:23.107049+0000) 2020-02-06T00:01:23.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:23.984+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:23.985459+0000) 2020-02-06T00:01:23.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:23.984+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:23.985459+0000) 2020-02-06T00:01:26.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:26.878+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.879807+0000) 2020-02-06T00:01:26.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:26.879+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.880120+0000) 2020-02-06T00:01:26.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:26.879+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.880474+0000) 2020-02-06T00:01:26.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:26.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.881566+0000) 2020-02-06T00:01:26.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:26.880+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.882022+0000) 2020-02-06T00:01:26.883 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:26.881+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.882417+0000) 2020-02-06T00:01:26.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:26.881+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.882968+0000) 2020-02-06T00:01:26.885 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:26.882+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.884069+0000) 2020-02-06T00:01:26.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:26.882+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.884295+0000) 2020-02-06T00:01:26.886 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:26.883+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.884966+0000) 2020-02-06T00:01:26.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:26.884+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.885821+0000) 2020-02-06T00:01:26.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:26.884+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.885977+0000) 2020-02-06T00:01:26.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:26.885+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.886639+0000) 2020-02-06T00:01:26.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:26.887+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.888485+0000) 2020-02-06T00:01:26.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:26.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:26.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:26.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:26.964001+0000) 2020-02-06T00:01:27.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:27.223+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:27.224727+0000) 2020-02-06T00:01:27.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:27.223+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:27.224954+0000) 2020-02-06T00:01:27.226 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:27.223+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:27.225155+0000) 2020-02-06T00:01:29.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:29.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:29.223280+0000) 2020-02-06T00:01:29.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:29.606+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:29.607561+0000) 2020-02-06T00:01:30.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:30.453+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:30.454748+0000) 2020-02-06T00:01:30.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:30.715+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:30.716168+0000) 2020-02-06T00:01:31.606 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:01:31.612 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:01:31.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:31.745+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.746923+0000) 2020-02-06T00:01:31.905 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:31.902+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.903768+0000) 2020-02-06T00:01:31.907 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:31.904+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.906038+0000) 2020-02-06T00:01:31.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:31.906+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.907155+0000) 2020-02-06T00:01:31.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:31.906+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.907471+0000) 2020-02-06T00:01:31.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:31.906+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.907695+0000) 2020-02-06T00:01:31.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:31.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.908401+0000) 2020-02-06T00:01:31.911 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:31.907+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.908849+0000) 2020-02-06T00:01:31.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:31.907+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.909336+0000) 2020-02-06T00:01:31.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:31.908+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.909713+0000) 2020-02-06T00:01:31.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:31.908+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.909796+0000) 2020-02-06T00:01:31.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:31.909+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.911190+0000) 2020-02-06T00:01:31.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:31.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.911520+0000) 2020-02-06T00:01:31.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:31.910+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.912231+0000) 2020-02-06T00:01:31.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:31.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:31.916884+0000) 2020-02-06T00:01:31.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:31.922+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:32.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:32.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:32.820864+0000) 2020-02-06T00:01:32.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:32.834+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:32.835582+0000) 2020-02-06T00:01:33.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:33.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:33.107314+0000) 2020-02-06T00:01:33.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:33.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:33.985734+0000) 2020-02-06T00:01:33.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:33.985+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:33.985735+0000) 2020-02-06T00:01:36.930 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:36.927+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.928889+0000) 2020-02-06T00:01:36.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:36.928+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.929469+0000) 2020-02-06T00:01:36.931 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:36.929+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.930404+0000) 2020-02-06T00:01:36.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:36.929+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.931039+0000) 2020-02-06T00:01:36.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:36.930+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.931718+0000) 2020-02-06T00:01:36.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:36.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.931996+0000) 2020-02-06T00:01:36.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:36.931+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.932356+0000) 2020-02-06T00:01:36.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:36.931+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.932372+0000) 2020-02-06T00:01:36.935 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:36.931+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.933388+0000) 2020-02-06T00:01:36.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:36.931+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.932583+0000) 2020-02-06T00:01:36.935 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:36.932+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.933662+0000) 2020-02-06T00:01:36.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:36.932+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.934251+0000) 2020-02-06T00:01:36.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:36.933+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.935341+0000) 2020-02-06T00:01:36.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:36.940+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.941153+0000) 2020-02-06T00:01:36.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:36.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:36.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:36.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:36.964182+0000) 2020-02-06T00:01:37.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:37.224+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:37.224948+0000) 2020-02-06T00:01:37.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:37.224+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:37.225195+0000) 2020-02-06T00:01:37.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:37.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:37.225362+0000) 2020-02-06T00:01:39.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:39.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:39.223576+0000) 2020-02-06T00:01:39.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:39.607+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:39.607791+0000) 2020-02-06T00:01:40.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:40.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:40.455012+0000) 2020-02-06T00:01:40.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:40.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:40.716437+0000) 2020-02-06T00:01:41.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:41.755+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.755821+0000) 2020-02-06T00:01:41.953 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:41.950+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.951581+0000) 2020-02-06T00:01:41.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:41.951+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.952455+0000) 2020-02-06T00:01:41.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:41.951+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.952506+0000) 2020-02-06T00:01:41.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:41.952+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.953177+0000) 2020-02-06T00:01:41.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:41.952+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.953647+0000) 2020-02-06T00:01:41.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:41.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.954167+0000) 2020-02-06T00:01:41.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:41.953+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.954372+0000) 2020-02-06T00:01:41.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:41.953+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.955232+0000) 2020-02-06T00:01:41.957 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:41.953+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.955147+0000) 2020-02-06T00:01:41.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:41.954+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.956169+0000) 2020-02-06T00:01:41.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:41.955+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.956796+0000) 2020-02-06T00:01:41.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:41.956+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.957460+0000) 2020-02-06T00:01:41.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:41.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.957669+0000) 2020-02-06T00:01:41.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:41.959+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:41.960804+0000) 2020-02-06T00:01:41.972 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:41.969+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161899 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:42.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:42.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:42.821115+0000) 2020-02-06T00:01:42.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:42.834+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:42.835814+0000) 2020-02-06T00:01:43.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:43.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:43.107546+0000) 2020-02-06T00:01:43.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:43.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:43.986010+0000) 2020-02-06T00:01:43.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:43.985+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:43.986009+0000) 2020-02-06T00:01:46.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:46.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.964384+0000) 2020-02-06T00:01:46.977 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:46.975+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.976269+0000) 2020-02-06T00:01:46.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:46.976+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.977599+0000) 2020-02-06T00:01:46.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:46.976+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.978051+0000) 2020-02-06T00:01:46.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:46.977+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.978561+0000) 2020-02-06T00:01:46.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:46.977+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.978381+0000) 2020-02-06T00:01:46.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:46.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.979428+0000) 2020-02-06T00:01:46.981 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:46.978+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.979643+0000) 2020-02-06T00:01:46.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:46.977+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.979268+0000) 2020-02-06T00:01:46.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:46.979+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.980143+0000) 2020-02-06T00:01:46.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:46.979+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.980815+0000) 2020-02-06T00:01:46.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:46.980+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.981435+0000) 2020-02-06T00:01:46.983 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:46.980+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.981755+0000) 2020-02-06T00:01:46.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:46.980+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.982001+0000) 2020-02-06T00:01:46.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:46.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:46.988609+0000) 2020-02-06T00:01:46.998 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:46.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161940 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:47.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:47.224+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:47.225171+0000) 2020-02-06T00:01:47.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:47.224+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:47.225378+0000) 2020-02-06T00:01:47.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:47.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:47.225514+0000) 2020-02-06T00:01:49.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:49.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:49.223840+0000) 2020-02-06T00:01:49.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:49.607+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:49.608019+0000) 2020-02-06T00:01:50.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:50.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:50.455272+0000) 2020-02-06T00:01:50.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:50.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:50.716676+0000) 2020-02-06T00:01:51.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:51.764+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:51.764285+0000) 2020-02-06T00:01:52.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:52.002+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.003623+0000) 2020-02-06T00:01:52.005 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:52.003+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.004473+0000) 2020-02-06T00:01:52.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:52.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.005258+0000) 2020-02-06T00:01:52.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:52.004+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.005578+0000) 2020-02-06T00:01:52.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:52.004+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.005984+0000) 2020-02-06T00:01:52.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:52.005+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.006324+0000) 2020-02-06T00:01:52.008 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:52.005+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.006496+0000) 2020-02-06T00:01:52.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:52.005+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.006859+0000) 2020-02-06T00:01:52.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:52.005+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.006787+0000) 2020-02-06T00:01:52.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:52.006+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.007411+0000) 2020-02-06T00:01:52.009 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:52.006+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.007638+0000) 2020-02-06T00:01:52.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:52.006+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.007852+0000) 2020-02-06T00:01:52.010 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:52.006+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.008243+0000) 2020-02-06T00:01:52.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:52.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.013404+0000) 2020-02-06T00:01:52.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:52.018+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 161979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:52.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:52.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.821321+0000) 2020-02-06T00:01:52.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:52.835+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:52.836033+0000) 2020-02-06T00:01:53.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:53.107+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:53.107786+0000) 2020-02-06T00:01:53.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:53.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:53.986355+0000) 2020-02-06T00:01:53.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:53.986+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:53.986355+0000) 2020-02-06T00:01:56.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:56.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:56.964620+0000) 2020-02-06T00:01:57.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:01:57.024+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.024981+0000) 2020-02-06T00:01:57.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:57.025+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.026893+0000) 2020-02-06T00:01:57.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:57.025+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.027138+0000) 2020-02-06T00:01:57.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:57.027+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.027506+0000) 2020-02-06T00:01:57.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:57.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.027956+0000) 2020-02-06T00:01:57.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:01:57.027+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.027474+0000) 2020-02-06T00:01:57.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:01:57.028+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.028738+0000) 2020-02-06T00:01:57.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:57.028+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.029730+0000) 2020-02-06T00:01:57.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:01:57.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.030026+0000) 2020-02-06T00:01:57.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:01:57.029+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.029979+0000) 2020-02-06T00:01:57.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:01:57.030+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.030283+0000) 2020-02-06T00:01:57.032 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:01:57.029+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.030748+0000) 2020-02-06T00:01:57.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:57.029+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.030991+0000) 2020-02-06T00:01:57.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:57.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.033768+0000) 2020-02-06T00:01:57.043 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:01:57.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:01:57.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:01:57.224+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.225400+0000) 2020-02-06T00:01:57.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:01:57.224+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.225586+0000) 2020-02-06T00:01:57.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:01:57.224+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:57.225707+0000) 2020-02-06T00:01:59.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:01:59.223+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:59.224100+0000) 2020-02-06T00:01:59.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:01:59.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:01:59.608283+0000) 2020-02-06T00:02:00.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:00.454+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:00.455517+0000) 2020-02-06T00:02:00.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:00.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:00.716934+0000) 2020-02-06T00:02:01.750 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:02:01.756 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:02:01.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:01.771+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:01.772659+0000) 2020-02-06T00:02:02.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:02.048+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.049510+0000) 2020-02-06T00:02:02.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:02.049+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.051053+0000) 2020-02-06T00:02:02.054 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:02.050+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.052060+0000) 2020-02-06T00:02:02.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:02.051+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.052827+0000) 2020-02-06T00:02:02.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:02.049+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.050208+0000) 2020-02-06T00:02:02.055 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:02.049+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.050471+0000) 2020-02-06T00:02:02.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:02.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.051309+0000) 2020-02-06T00:02:02.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:02.050+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.051550+0000) 2020-02-06T00:02:02.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:02.051+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.052412+0000) 2020-02-06T00:02:02.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:02.052+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.054269+0000) 2020-02-06T00:02:02.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:02.053+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.054476+0000) 2020-02-06T00:02:02.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:02.053+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.055054+0000) 2020-02-06T00:02:02.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:02.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.054033+0000) 2020-02-06T00:02:02.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:02.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.057998+0000) 2020-02-06T00:02:02.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:02.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:02.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:02.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.821533+0000) 2020-02-06T00:02:02.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:02.835+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:02.836259+0000) 2020-02-06T00:02:03.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:03.106+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:03.108049+0000) 2020-02-06T00:02:03.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:03.985+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:03.986637+0000) 2020-02-06T00:02:03.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:03.985+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:03.986637+0000) 2020-02-06T00:02:06.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:06.963+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:06.964852+0000) 2020-02-06T00:02:07.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:07.071+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.073012+0000) 2020-02-06T00:02:07.076 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:07.073+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.074810+0000) 2020-02-06T00:02:07.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:07.074+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.075422+0000) 2020-02-06T00:02:07.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:07.074+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.076045+0000) 2020-02-06T00:02:07.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:07.075+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.076284+0000) 2020-02-06T00:02:07.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:07.075+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.076627+0000) 2020-02-06T00:02:07.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:07.075+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.077217+0000) 2020-02-06T00:02:07.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:07.076+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.077543+0000) 2020-02-06T00:02:07.079 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:07.076+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.078215+0000) 2020-02-06T00:02:07.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:07.076+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.077880+0000) 2020-02-06T00:02:07.081 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:07.077+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.078271+0000) 2020-02-06T00:02:07.081 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:07.077+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.078667+0000) 2020-02-06T00:02:07.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:07.077+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.078764+0000) 2020-02-06T00:02:07.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:07.082+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.083784+0000) 2020-02-06T00:02:07.091 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:07.089+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:07.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:07.225+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.225680+0000) 2020-02-06T00:02:07.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:07.225+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.225788+0000) 2020-02-06T00:02:07.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:07.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:07.225904+0000) 2020-02-06T00:02:09.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:09.222+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:09.224348+0000) 2020-02-06T00:02:09.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:09.607+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:09.608547+0000) 2020-02-06T00:02:10.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:10.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:10.455774+0000) 2020-02-06T00:02:10.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:10.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:10.717173+0000) 2020-02-06T00:02:11.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:11.779+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:11.780751+0000) 2020-02-06T00:02:12.096 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:12.094+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.095533+0000) 2020-02-06T00:02:12.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:12.096+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.097528+0000) 2020-02-06T00:02:12.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:12.096+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.097989+0000) 2020-02-06T00:02:12.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:12.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.098564+0000) 2020-02-06T00:02:12.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:12.097+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.098695+0000) 2020-02-06T00:02:12.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:12.097+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.099268+0000) 2020-02-06T00:02:12.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:12.099+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.100304+0000) 2020-02-06T00:02:12.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:12.099+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.100640+0000) 2020-02-06T00:02:12.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:12.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.101670+0000) 2020-02-06T00:02:12.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:12.100+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.101951+0000) 2020-02-06T00:02:12.104 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:12.101+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.102563+0000) 2020-02-06T00:02:12.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:12.100+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.102216+0000) 2020-02-06T00:02:12.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:12.101+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.102828+0000) 2020-02-06T00:02:12.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:12.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.105590+0000) 2020-02-06T00:02:12.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:12.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:12.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:12.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.821802+0000) 2020-02-06T00:02:12.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:12.835+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:12.836496+0000) 2020-02-06T00:02:13.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:13.107+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:13.108358+0000) 2020-02-06T00:02:13.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:13.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:13.986909+0000) 2020-02-06T00:02:13.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:13.986+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:13.986909+0000) 2020-02-06T00:02:16.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:16.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:16.965127+0000) 2020-02-06T00:02:17.121 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:17.118+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.119966+0000) 2020-02-06T00:02:17.122 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:17.120+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.121584+0000) 2020-02-06T00:02:17.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:17.121+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.122129+0000) 2020-02-06T00:02:17.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:17.121+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.122500+0000) 2020-02-06T00:02:17.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:17.122+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.122771+0000) 2020-02-06T00:02:17.124 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:17.122+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.123459+0000) 2020-02-06T00:02:17.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:17.123+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.123566+0000) 2020-02-06T00:02:17.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:17.124+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.124821+0000) 2020-02-06T00:02:17.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:17.124+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.125105+0000) 2020-02-06T00:02:17.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:17.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.125424+0000) 2020-02-06T00:02:17.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:17.126+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.126470+0000) 2020-02-06T00:02:17.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:17.125+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.126774+0000) 2020-02-06T00:02:17.128 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:17.126+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.126894+0000) 2020-02-06T00:02:17.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:17.130+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.131199+0000) 2020-02-06T00:02:17.139 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:17.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162180 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:17.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:17.225+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.225925+0000) 2020-02-06T00:02:17.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:17.225+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.226003+0000) 2020-02-06T00:02:17.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:17.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:17.226192+0000) 2020-02-06T00:02:19.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:19.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:19.224638+0000) 2020-02-06T00:02:19.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:19.607+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:19.608770+0000) 2020-02-06T00:02:20.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:20.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:20.455995+0000) 2020-02-06T00:02:20.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:20.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:20.717415+0000) 2020-02-06T00:02:21.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:21.788+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:21.789212+0000) 2020-02-06T00:02:22.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:22.143+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.144847+0000) 2020-02-06T00:02:22.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:22.144+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.145535+0000) 2020-02-06T00:02:22.147 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:22.144+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.145793+0000) 2020-02-06T00:02:22.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:22.145+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.146456+0000) 2020-02-06T00:02:22.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:22.146+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.146979+0000) 2020-02-06T00:02:22.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:22.146+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.147433+0000) 2020-02-06T00:02:22.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:22.146+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.147365+0000) 2020-02-06T00:02:22.149 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:22.146+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.147725+0000) 2020-02-06T00:02:22.149 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:22.148+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.148717+0000) 2020-02-06T00:02:22.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:22.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.148638+0000) 2020-02-06T00:02:22.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:22.148+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.148646+0000) 2020-02-06T00:02:22.151 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:22.150+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.150524+0000) 2020-02-06T00:02:22.151 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:22.150+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.150595+0000) 2020-02-06T00:02:22.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:22.152+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.154011+0000) 2020-02-06T00:02:22.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:22.163+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:22.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:22.820+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.822029+0000) 2020-02-06T00:02:22.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:22.836+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:22.836735+0000) 2020-02-06T00:02:23.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:23.107+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:23.108635+0000) 2020-02-06T00:02:23.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:23.986+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:23.987204+0000) 2020-02-06T00:02:23.989 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:23.986+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:23.987207+0000) 2020-02-06T00:02:26.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:26.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:26.965323+0000) 2020-02-06T00:02:27.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:27.168+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.169239+0000) 2020-02-06T00:02:27.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:27.169+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.170420+0000) 2020-02-06T00:02:27.172 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:27.170+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.171316+0000) 2020-02-06T00:02:27.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:27.170+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.171616+0000) 2020-02-06T00:02:27.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:27.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.172191+0000) 2020-02-06T00:02:27.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:27.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.172540+0000) 2020-02-06T00:02:27.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:27.171+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.172813+0000) 2020-02-06T00:02:27.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:27.172+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.172864+0000) 2020-02-06T00:02:27.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:27.173+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.173584+0000) 2020-02-06T00:02:27.175 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:27.173+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.173890+0000) 2020-02-06T00:02:27.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:27.173+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.174826+0000) 2020-02-06T00:02:27.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:27.174+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.174548+0000) 2020-02-06T00:02:27.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:27.174+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.175248+0000) 2020-02-06T00:02:27.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:27.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.179623+0000) 2020-02-06T00:02:27.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:27.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:27.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:27.225+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.226244+0000) 2020-02-06T00:02:27.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:27.225+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.226246+0000) 2020-02-06T00:02:27.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:27.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:27.226415+0000) 2020-02-06T00:02:29.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:29.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:29.224878+0000) 2020-02-06T00:02:29.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:29.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:29.609000+0000) 2020-02-06T00:02:30.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:30.455+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:30.456248+0000) 2020-02-06T00:02:30.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:30.716+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:30.717648+0000) 2020-02-06T00:02:31.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:31.798+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:31.799249+0000) 2020-02-06T00:02:31.949 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:02:31.955 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:02:32.195 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:32.193+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.194368+0000) 2020-02-06T00:02:32.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:32.194+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.194663+0000) 2020-02-06T00:02:32.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:32.193+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.194838+0000) 2020-02-06T00:02:32.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:32.194+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.195167+0000) 2020-02-06T00:02:32.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:32.195+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.195653+0000) 2020-02-06T00:02:32.198 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:32.195+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.196305+0000) 2020-02-06T00:02:32.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:32.194+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.195692+0000) 2020-02-06T00:02:32.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:32.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.196446+0000) 2020-02-06T00:02:32.198 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:32.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.196446+0000) 2020-02-06T00:02:32.199 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:32.195+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.196512+0000) 2020-02-06T00:02:32.199 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:32.196+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.197030+0000) 2020-02-06T00:02:32.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:32.196+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.196874+0000) 2020-02-06T00:02:32.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:32.196+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.197358+0000) 2020-02-06T00:02:32.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:32.200+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.201875+0000) 2020-02-06T00:02:32.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:32.211+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:32.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:32.821+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.822265+0000) 2020-02-06T00:02:32.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:32.836+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:32.836977+0000) 2020-02-06T00:02:33.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:33.107+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:33.108851+0000) 2020-02-06T00:02:33.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:33.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:33.987496+0000) 2020-02-06T00:02:33.989 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:33.987+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:33.987497+0000) 2020-02-06T00:02:36.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:36.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:36.965552+0000) 2020-02-06T00:02:37.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:37.215+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.216421+0000) 2020-02-06T00:02:37.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:37.216+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.217139+0000) 2020-02-06T00:02:37.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:37.217+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.218111+0000) 2020-02-06T00:02:37.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:37.218+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.218461+0000) 2020-02-06T00:02:37.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:37.218+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.218947+0000) 2020-02-06T00:02:37.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:37.218+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.219379+0000) 2020-02-06T00:02:37.221 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:37.219+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.220366+0000) 2020-02-06T00:02:37.222 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:37.220+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.221241+0000) 2020-02-06T00:02:37.222 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:37.220+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.221141+0000) 2020-02-06T00:02:37.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:37.221+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.222050+0000) 2020-02-06T00:02:37.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:37.222+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.223008+0000) 2020-02-06T00:02:37.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:37.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.223174+0000) 2020-02-06T00:02:37.227 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:37.223+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.223975+0000) 2020-02-06T00:02:37.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:37.225+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.226391+0000) 2020-02-06T00:02:37.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:37.226+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.226470+0000) 2020-02-06T00:02:37.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:37.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.226638+0000) 2020-02-06T00:02:37.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:37.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:37.228379+0000) 2020-02-06T00:02:37.238 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:37.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:39.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:39.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:39.225119+0000) 2020-02-06T00:02:39.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:39.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:39.609268+0000) 2020-02-06T00:02:40.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:40.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:40.456497+0000) 2020-02-06T00:02:40.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:40.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:40.717888+0000) 2020-02-06T00:02:41.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:41.807+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:41.807503+0000) 2020-02-06T00:02:42.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:42.241+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.242198+0000) 2020-02-06T00:02:42.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:42.242+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.243870+0000) 2020-02-06T00:02:42.246 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:42.243+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.244822+0000) 2020-02-06T00:02:42.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:42.244+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.244648+0000) 2020-02-06T00:02:42.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:42.244+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.245411+0000) 2020-02-06T00:02:42.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:42.245+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.246400+0000) 2020-02-06T00:02:42.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:42.246+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.246564+0000) 2020-02-06T00:02:42.248 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:42.245+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.247018+0000) 2020-02-06T00:02:42.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:42.247+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.247914+0000) 2020-02-06T00:02:42.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:42.247+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.247802+0000) 2020-02-06T00:02:42.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:42.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.248254+0000) 2020-02-06T00:02:42.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:42.248+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.248601+0000) 2020-02-06T00:02:42.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:42.248+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.249089+0000) 2020-02-06T00:02:42.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:42.249+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.250094+0000) 2020-02-06T00:02:42.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:42.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:42.823 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:42.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.822514+0000) 2020-02-06T00:02:42.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:42.836+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:42.837199+0000) 2020-02-06T00:02:43.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:43.108+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:43.109113+0000) 2020-02-06T00:02:43.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:43.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:43.987771+0000) 2020-02-06T00:02:43.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:43.987+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:43.987771+0000) 2020-02-06T00:02:46.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:46.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:46.965796+0000) 2020-02-06T00:02:47.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:47.226+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.226610+0000) 2020-02-06T00:02:47.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:47.226+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.226659+0000) 2020-02-06T00:02:47.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:47.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.226828+0000) 2020-02-06T00:02:47.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:47.267+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.268515+0000) 2020-02-06T00:02:47.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:47.267+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.268556+0000) 2020-02-06T00:02:47.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:47.268+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.269118+0000) 2020-02-06T00:02:47.271 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:47.268+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.270016+0000) 2020-02-06T00:02:47.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:47.269+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.270176+0000) 2020-02-06T00:02:47.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:47.269+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.270932+0000) 2020-02-06T00:02:47.272 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:47.270+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.271330+0000) 2020-02-06T00:02:47.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:47.270+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.270903+0000) 2020-02-06T00:02:47.274 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:47.271+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.272180+0000) 2020-02-06T00:02:47.274 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:47.270+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.271392+0000) 2020-02-06T00:02:47.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:47.272+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.272649+0000) 2020-02-06T00:02:47.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:47.272+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.272705+0000) 2020-02-06T00:02:47.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:47.275+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.276770+0000) 2020-02-06T00:02:47.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:47.275+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:47.276974+0000) 2020-02-06T00:02:47.287 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:47.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:49.226 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:49.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:49.225373+0000) 2020-02-06T00:02:49.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:49.609+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:49.609502+0000) 2020-02-06T00:02:50.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:50.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:50.456778+0000) 2020-02-06T00:02:50.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:50.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:50.718115+0000) 2020-02-06T00:02:51.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:51.815+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:51.815564+0000) 2020-02-06T00:02:52.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:52.293+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.293316+0000) 2020-02-06T00:02:52.295 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:52.293+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.293937+0000) 2020-02-06T00:02:52.295 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:52.294+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.294362+0000) 2020-02-06T00:02:52.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:52.294+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.294828+0000) 2020-02-06T00:02:52.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:52.295+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.295857+0000) 2020-02-06T00:02:52.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:52.295+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.295805+0000) 2020-02-06T00:02:52.298 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:52.296+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.296434+0000) 2020-02-06T00:02:52.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:52.295+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.296297+0000) 2020-02-06T00:02:52.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:52.296+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.297321+0000) 2020-02-06T00:02:52.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:52.297+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.297757+0000) 2020-02-06T00:02:52.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:52.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.297980+0000) 2020-02-06T00:02:52.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:52.298+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.298654+0000) 2020-02-06T00:02:52.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:52.298+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.298477+0000) 2020-02-06T00:02:52.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:52.303+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.303500+0000) 2020-02-06T00:02:52.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:52.310+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:52.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:52.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.822725+0000) 2020-02-06T00:02:52.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:52.837+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:52.837468+0000) 2020-02-06T00:02:53.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:53.109+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:53.109306+0000) 2020-02-06T00:02:53.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:53.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:53.988061+0000) 2020-02-06T00:02:53.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:53.988+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:53.988063+0000) 2020-02-06T00:02:56.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:56.964+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:56.966025+0000) 2020-02-06T00:02:57.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:57.225+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.226895+0000) 2020-02-06T00:02:57.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:57.225+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.226956+0000) 2020-02-06T00:02:57.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:57.225+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.227101+0000) 2020-02-06T00:02:57.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:02:57.315+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.316486+0000) 2020-02-06T00:02:57.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:02:57.316+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.317072+0000) 2020-02-06T00:02:57.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:57.316+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.317593+0000) 2020-02-06T00:02:57.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:02:57.316+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.318139+0000) 2020-02-06T00:02:57.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:57.317+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.318714+0000) 2020-02-06T00:02:57.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:02:57.318+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.319375+0000) 2020-02-06T00:02:57.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:02:57.317+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.319189+0000) 2020-02-06T00:02:57.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:02:57.320+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.321389+0000) 2020-02-06T00:02:57.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:02:57.320+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.322159+0000) 2020-02-06T00:02:57.323 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:02:57.320+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.321971+0000) 2020-02-06T00:02:57.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:02:57.321+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.322459+0000) 2020-02-06T00:02:57.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:02:57.321+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.322879+0000) 2020-02-06T00:02:57.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:57.321+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.322541+0000) 2020-02-06T00:02:57.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:57.328+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:57.329191+0000) 2020-02-06T00:02:57.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:02:57.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:02:59.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:02:59.224+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:59.225677+0000) 2020-02-06T00:02:59.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:02:59.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:02:59.609735+0000) 2020-02-06T00:03:00.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:00.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:00.457059+0000) 2020-02-06T00:03:00.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:00.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:00.718337+0000) 2020-02-06T00:03:01.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:01.823+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:01.824040+0000) 2020-02-06T00:03:02.128 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:03:02.135 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:03:02.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:02.340+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.341351+0000) 2020-02-06T00:03:02.343 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:02.340+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.341627+0000) 2020-02-06T00:03:02.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:02.342+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.343332+0000) 2020-02-06T00:03:02.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:02.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.343819+0000) 2020-02-06T00:03:02.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:02.343+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.344467+0000) 2020-02-06T00:03:02.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:02.343+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.344576+0000) 2020-02-06T00:03:02.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:02.343+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.344625+0000) 2020-02-06T00:03:02.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:02.343+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.344931+0000) 2020-02-06T00:03:02.351 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:02.343+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.345239+0000) 2020-02-06T00:03:02.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:02.343+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.345233+0000) 2020-02-06T00:03:02.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:02.344+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.346097+0000) 2020-02-06T00:03:02.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:02.344+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.346312+0000) 2020-02-06T00:03:02.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:02.345+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.346584+0000) 2020-02-06T00:03:02.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:02.352+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.353406+0000) 2020-02-06T00:03:02.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:02.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:02.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:02.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.822944+0000) 2020-02-06T00:03:02.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:02.836+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:02.837645+0000) 2020-02-06T00:03:03.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:03.108+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:03.109533+0000) 2020-02-06T00:03:03.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:03.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:03.988372+0000) 2020-02-06T00:03:03.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:03.987+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:03.988372+0000) 2020-02-06T00:03:06.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:06.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:06.966292+0000) 2020-02-06T00:03:07.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:07.226+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.227164+0000) 2020-02-06T00:03:07.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:07.226+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.227188+0000) 2020-02-06T00:03:07.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:07.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.227340+0000) 2020-02-06T00:03:07.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:07.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.363447+0000) 2020-02-06T00:03:07.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:07.364+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.365168+0000) 2020-02-06T00:03:07.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:07.364+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.365983+0000) 2020-02-06T00:03:07.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:07.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.366795+0000) 2020-02-06T00:03:07.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:07.366+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.367467+0000) 2020-02-06T00:03:07.369 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:07.367+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.367849+0000) 2020-02-06T00:03:07.369 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:07.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.368862+0000) 2020-02-06T00:03:07.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:07.367+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.369210+0000) 2020-02-06T00:03:07.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:07.368+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.369453+0000) 2020-02-06T00:03:07.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:07.368+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.370322+0000) 2020-02-06T00:03:07.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:07.369+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.370832+0000) 2020-02-06T00:03:07.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:07.369+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.371217+0000) 2020-02-06T00:03:07.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:07.369+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.371275+0000) 2020-02-06T00:03:07.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:07.375+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:07.376520+0000) 2020-02-06T00:03:07.385 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:07.383+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:09.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:09.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:09.225943+0000) 2020-02-06T00:03:09.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:09.608+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:09.609970+0000) 2020-02-06T00:03:10.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:10.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:10.457295+0000) 2020-02-06T00:03:10.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:10.717+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:10.718613+0000) 2020-02-06T00:03:11.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:11.831+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:11.832714+0000) 2020-02-06T00:03:12.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:12.389+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.390850+0000) 2020-02-06T00:03:12.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:12.390+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.392250+0000) 2020-02-06T00:03:12.393 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:12.391+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.392867+0000) 2020-02-06T00:03:12.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:12.391+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.392877+0000) 2020-02-06T00:03:12.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:12.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.393352+0000) 2020-02-06T00:03:12.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:12.392+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.393654+0000) 2020-02-06T00:03:12.395 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:12.393+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.394108+0000) 2020-02-06T00:03:12.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:12.392+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.394293+0000) 2020-02-06T00:03:12.396 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:12.393+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.394792+0000) 2020-02-06T00:03:12.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:12.394+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.395241+0000) 2020-02-06T00:03:12.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:12.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.395147+0000) 2020-02-06T00:03:12.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:12.394+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.396058+0000) 2020-02-06T00:03:12.397 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:12.394+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.396386+0000) 2020-02-06T00:03:12.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:12.398+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.399664+0000) 2020-02-06T00:03:12.410 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:12.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162620 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:12.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:12.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.823156+0000) 2020-02-06T00:03:12.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:12.836+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:12.837887+0000) 2020-02-06T00:03:13.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:13.108+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:13.109744+0000) 2020-02-06T00:03:13.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:13.987+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:13.988645+0000) 2020-02-06T00:03:13.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:13.987+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:13.988645+0000) 2020-02-06T00:03:16.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:16.965+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:16.966520+0000) 2020-02-06T00:03:17.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:17.226+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.227416+0000) 2020-02-06T00:03:17.229 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:17.226+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.227474+0000) 2020-02-06T00:03:17.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:17.226+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.227628+0000) 2020-02-06T00:03:17.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:17.415+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.416556+0000) 2020-02-06T00:03:17.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:17.416+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.417252+0000) 2020-02-06T00:03:17.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:17.415+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.416845+0000) 2020-02-06T00:03:17.419 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:17.416+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.418086+0000) 2020-02-06T00:03:17.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:17.417+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.418332+0000) 2020-02-06T00:03:17.420 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:17.417+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.418733+0000) 2020-02-06T00:03:17.420 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:17.418+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.419180+0000) 2020-02-06T00:03:17.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:17.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.419619+0000) 2020-02-06T00:03:17.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:17.418+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.420188+0000) 2020-02-06T00:03:17.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:17.419+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.420792+0000) 2020-02-06T00:03:17.422 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:17.419+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.420636+0000) 2020-02-06T00:03:17.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:17.419+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.421208+0000) 2020-02-06T00:03:17.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:17.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.421772+0000) 2020-02-06T00:03:17.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:17.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:17.427358+0000) 2020-02-06T00:03:17.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:17.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:19.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:19.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:19.226187+0000) 2020-02-06T00:03:19.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:19.609+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:19.610280+0000) 2020-02-06T00:03:20.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:20.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:20.457532+0000) 2020-02-06T00:03:20.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:20.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:20.718851+0000) 2020-02-06T00:03:21.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:21.839+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:21.840362+0000) 2020-02-06T00:03:22.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:22.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.438173+0000) 2020-02-06T00:03:22.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:22.437+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.438707+0000) 2020-02-06T00:03:22.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:22.439+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.440635+0000) 2020-02-06T00:03:22.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:22.439+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.440918+0000) 2020-02-06T00:03:22.442 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:22.440+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.441394+0000) 2020-02-06T00:03:22.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:22.440+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.441827+0000) 2020-02-06T00:03:22.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:22.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.442448+0000) 2020-02-06T00:03:22.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:22.442+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.443139+0000) 2020-02-06T00:03:22.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:22.442+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.444220+0000) 2020-02-06T00:03:22.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:22.443+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.444650+0000) 2020-02-06T00:03:22.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:22.443+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.444723+0000) 2020-02-06T00:03:22.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:22.444+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.445403+0000) 2020-02-06T00:03:22.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:22.444+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.445486+0000) 2020-02-06T00:03:22.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:22.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.447979+0000) 2020-02-06T00:03:22.457 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:22.455+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162702 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:22.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:22.822+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.823412+0000) 2020-02-06T00:03:22.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:22.837+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:22.838102+0000) 2020-02-06T00:03:23.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:23.109+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:23.109942+0000) 2020-02-06T00:03:23.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:23.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:23.988925+0000) 2020-02-06T00:03:23.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:23.988+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:23.988925+0000) 2020-02-06T00:03:26.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:26.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:26.966756+0000) 2020-02-06T00:03:27.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:27.227+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.227763+0000) 2020-02-06T00:03:27.229 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:27.227+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.227764+0000) 2020-02-06T00:03:27.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:27.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.227949+0000) 2020-02-06T00:03:27.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:27.460+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.461593+0000) 2020-02-06T00:03:27.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:27.461+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.462387+0000) 2020-02-06T00:03:27.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:27.462+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.463212+0000) 2020-02-06T00:03:27.465 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:27.463+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.464261+0000) 2020-02-06T00:03:27.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:27.462+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.464096+0000) 2020-02-06T00:03:27.466 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:27.463+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.464806+0000) 2020-02-06T00:03:27.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:27.464+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.465449+0000) 2020-02-06T00:03:27.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:27.464+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.465952+0000) 2020-02-06T00:03:27.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:27.464+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.465722+0000) 2020-02-06T00:03:27.467 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:27.465+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.466553+0000) 2020-02-06T00:03:27.468 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:27.465+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.467136+0000) 2020-02-06T00:03:27.468 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:27.466+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.467419+0000) 2020-02-06T00:03:27.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:27.466+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.467780+0000) 2020-02-06T00:03:27.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:27.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:27.472148+0000) 2020-02-06T00:03:27.482 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:27.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:29.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:29.225+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:29.226398+0000) 2020-02-06T00:03:29.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:29.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:29.610511+0000) 2020-02-06T00:03:30.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:30.456+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:30.457723+0000) 2020-02-06T00:03:30.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:30.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:30.719120+0000) 2020-02-06T00:03:31.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:31.848+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:31.848876+0000) 2020-02-06T00:03:32.334 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:03:32.340 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:03:32.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:32.485+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.486129+0000) 2020-02-06T00:03:32.488 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:32.485+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.486731+0000) 2020-02-06T00:03:32.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:32.485+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.487057+0000) 2020-02-06T00:03:32.489 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:32.486+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.487601+0000) 2020-02-06T00:03:32.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:32.486+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.488373+0000) 2020-02-06T00:03:32.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:32.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.487612+0000) 2020-02-06T00:03:32.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:32.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.487819+0000) 2020-02-06T00:03:32.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:32.487+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.489274+0000) 2020-02-06T00:03:32.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:32.488+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.489511+0000) 2020-02-06T00:03:32.492 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:32.488+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.489607+0000) 2020-02-06T00:03:32.492 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:32.488+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.490192+0000) 2020-02-06T00:03:32.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:32.488+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.489178+0000) 2020-02-06T00:03:32.493 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:32.489+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.490401+0000) 2020-02-06T00:03:32.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:32.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.495606+0000) 2020-02-06T00:03:32.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:32.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:32.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:32.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.823574+0000) 2020-02-06T00:03:32.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:32.837+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:32.838326+0000) 2020-02-06T00:03:33.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:33.109+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:33.110150+0000) 2020-02-06T00:03:33.990 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:33.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:33.989201+0000) 2020-02-06T00:03:33.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:33.988+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:33.989201+0000) 2020-02-06T00:03:36.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:36.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:36.966985+0000) 2020-02-06T00:03:37.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:37.227+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.228025+0000) 2020-02-06T00:03:37.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:37.227+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.228018+0000) 2020-02-06T00:03:37.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:37.227+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.228265+0000) 2020-02-06T00:03:37.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:37.510+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.511205+0000) 2020-02-06T00:03:37.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:37.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.511789+0000) 2020-02-06T00:03:37.513 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:37.511+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.512596+0000) 2020-02-06T00:03:37.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:37.512+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.513137+0000) 2020-02-06T00:03:37.514 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:37.512+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.513220+0000) 2020-02-06T00:03:37.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:37.512+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.513620+0000) 2020-02-06T00:03:37.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:37.513+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.514923+0000) 2020-02-06T00:03:37.527 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:37.514+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.515369+0000) 2020-02-06T00:03:37.527 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:37.514+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.516026+0000) 2020-02-06T00:03:37.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:37.519+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.520910+0000) 2020-02-06T00:03:37.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:37.512+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.514183+0000) 2020-02-06T00:03:37.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:37.514+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.515660+0000) 2020-02-06T00:03:37.528 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:37.515+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.516564+0000) 2020-02-06T00:03:37.528 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:37.515+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:37.516419+0000) 2020-02-06T00:03:37.531 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:37.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:39.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:39.226+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:39.226790+0000) 2020-02-06T00:03:39.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:39.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:39.610746+0000) 2020-02-06T00:03:40.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:40.457+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:40.457983+0000) 2020-02-06T00:03:40.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:40.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:40.719346+0000) 2020-02-06T00:03:41.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:41.857+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:41.857690+0000) 2020-02-06T00:03:42.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:42.534+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.535869+0000) 2020-02-06T00:03:42.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:42.535+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.536904+0000) 2020-02-06T00:03:42.539 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:42.536+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.537237+0000) 2020-02-06T00:03:42.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:42.535+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.536624+0000) 2020-02-06T00:03:42.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:42.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.537789+0000) 2020-02-06T00:03:42.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:42.537+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.538410+0000) 2020-02-06T00:03:42.540 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:42.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.538234+0000) 2020-02-06T00:03:42.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:42.538+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.539410+0000) 2020-02-06T00:03:42.541 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:42.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.539639+0000) 2020-02-06T00:03:42.541 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:42.538+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.540273+0000) 2020-02-06T00:03:42.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:42.539+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.541012+0000) 2020-02-06T00:03:42.542 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:42.540+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.541791+0000) 2020-02-06T00:03:42.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:42.540+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.541643+0000) 2020-02-06T00:03:42.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:42.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.547962+0000) 2020-02-06T00:03:42.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:42.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:42.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:42.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.823798+0000) 2020-02-06T00:03:42.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:42.838+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:42.838563+0000) 2020-02-06T00:03:43.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:43.110+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:43.110365+0000) 2020-02-06T00:03:43.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:43.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:43.989484+0000) 2020-02-06T00:03:43.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:43.989+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:43.989482+0000) 2020-02-06T00:03:46.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:46.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:46.967289+0000) 2020-02-06T00:03:47.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:47.227+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.228281+0000) 2020-02-06T00:03:47.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:47.227+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.228262+0000) 2020-02-06T00:03:47.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:47.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.228522+0000) 2020-02-06T00:03:47.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:47.560+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.560754+0000) 2020-02-06T00:03:47.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:47.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.562492+0000) 2020-02-06T00:03:47.564 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:47.563+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.563180+0000) 2020-02-06T00:03:47.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:47.563+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.563936+0000) 2020-02-06T00:03:47.565 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:47.563+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.564263+0000) 2020-02-06T00:03:47.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:47.565+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.565188+0000) 2020-02-06T00:03:47.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:47.565+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.565809+0000) 2020-02-06T00:03:47.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:47.566+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.566229+0000) 2020-02-06T00:03:47.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:47.566+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.566667+0000) 2020-02-06T00:03:47.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:47.566+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.567031+0000) 2020-02-06T00:03:47.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:47.567+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.567471+0000) 2020-02-06T00:03:47.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:47.567+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.568298+0000) 2020-02-06T00:03:47.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:47.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.571948+0000) 2020-02-06T00:03:47.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:47.572+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:47.572266+0000) 2020-02-06T00:03:47.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:47.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162899 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:49.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:49.226+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:49.227052+0000) 2020-02-06T00:03:49.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:49.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:49.610997+0000) 2020-02-06T00:03:50.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:50.457+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:50.458245+0000) 2020-02-06T00:03:50.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:50.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:50.719578+0000) 2020-02-06T00:03:51.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:51.865+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:51.866986+0000) 2020-02-06T00:03:52.588 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:52.585+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.586574+0000) 2020-02-06T00:03:52.589 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:52.587+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.588226+0000) 2020-02-06T00:03:52.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:52.588+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.589264+0000) 2020-02-06T00:03:52.591 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:52.588+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.589979+0000) 2020-02-06T00:03:52.593 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:52.590+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.590657+0000) 2020-02-06T00:03:52.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:52.589+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.590509+0000) 2020-02-06T00:03:52.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:52.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.591164+0000) 2020-02-06T00:03:52.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:52.590+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.591471+0000) 2020-02-06T00:03:52.594 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:52.591+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.592172+0000) 2020-02-06T00:03:52.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:52.592+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.592852+0000) 2020-02-06T00:03:52.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:52.592+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.593184+0000) 2020-02-06T00:03:52.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:52.591+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.592667+0000) 2020-02-06T00:03:52.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:52.591+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.592741+0000) 2020-02-06T00:03:52.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:52.597+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.598695+0000) 2020-02-06T00:03:52.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:52.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:52.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:52.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.824015+0000) 2020-02-06T00:03:52.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:52.838+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:52.838813+0000) 2020-02-06T00:03:53.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:53.109+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:53.110571+0000) 2020-02-06T00:03:53.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:53.988+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:53.989748+0000) 2020-02-06T00:03:53.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:53.988+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:53.989748+0000) 2020-02-06T00:03:56.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:56.966+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:56.967522+0000) 2020-02-06T00:03:57.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:57.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.228545+0000) 2020-02-06T00:03:57.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:57.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.228541+0000) 2020-02-06T00:03:57.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:57.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.228764+0000) 2020-02-06T00:03:57.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:03:57.611+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.612443+0000) 2020-02-06T00:03:57.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:03:57.612+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.613005+0000) 2020-02-06T00:03:57.614 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:03:57.612+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.613447+0000) 2020-02-06T00:03:57.616 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:03:57.614+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.614929+0000) 2020-02-06T00:03:57.617 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:03:57.615+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.615747+0000) 2020-02-06T00:03:57.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:03:57.615+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.616331+0000) 2020-02-06T00:03:57.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:03:57.616+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.616987+0000) 2020-02-06T00:03:57.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:57.617+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.617528+0000) 2020-02-06T00:03:57.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:03:57.617+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.618553+0000) 2020-02-06T00:03:57.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:03:57.618+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.618799+0000) 2020-02-06T00:03:57.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:03:57.617+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.618877+0000) 2020-02-06T00:03:57.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:57.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.619426+0000) 2020-02-06T00:03:57.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:57.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.624001+0000) 2020-02-06T00:03:57.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:57.624+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:57.625182+0000) 2020-02-06T00:03:57.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:03:57.630+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 162979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:03:59.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:03:59.226+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:59.227300+0000) 2020-02-06T00:03:59.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:03:59.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:03:59.611269+0000) 2020-02-06T00:04:00.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:00.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:00.458496+0000) 2020-02-06T00:04:00.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:00.718+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:00.719806+0000) 2020-02-06T00:04:01.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:01.874+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:01.875191+0000) 2020-02-06T00:04:02.495 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:04:02.501 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:04:02.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:02.635+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.636903+0000) 2020-02-06T00:04:02.638 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:02.636+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.637421+0000) 2020-02-06T00:04:02.639 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:02.637+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.637621+0000) 2020-02-06T00:04:02.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:02.637+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.637979+0000) 2020-02-06T00:04:02.639 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:02.637+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.638351+0000) 2020-02-06T00:04:02.640 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:02.637+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.638583+0000) 2020-02-06T00:04:02.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:02.638+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.638863+0000) 2020-02-06T00:04:02.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:02.637+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.639040+0000) 2020-02-06T00:04:02.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:02.638+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.639141+0000) 2020-02-06T00:04:02.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:02.639+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.639541+0000) 2020-02-06T00:04:02.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:02.639+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.639867+0000) 2020-02-06T00:04:02.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:02.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.639964+0000) 2020-02-06T00:04:02.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:02.640+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.641253+0000) 2020-02-06T00:04:02.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:02.643+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.644421+0000) 2020-02-06T00:04:02.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:02.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:02.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:02.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.824267+0000) 2020-02-06T00:04:02.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:02.838+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:02.839037+0000) 2020-02-06T00:04:03.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:03.109+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:03.110787+0000) 2020-02-06T00:04:03.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:03.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:03.990015+0000) 2020-02-06T00:04:03.991 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:03.989+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:03.990015+0000) 2020-02-06T00:04:06.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:06.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:06.967756+0000) 2020-02-06T00:04:07.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:07.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.228739+0000) 2020-02-06T00:04:07.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:07.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.228775+0000) 2020-02-06T00:04:07.230 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:07.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.229029+0000) 2020-02-06T00:04:07.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:07.659+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.660515+0000) 2020-02-06T00:04:07.663 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:07.661+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.661990+0000) 2020-02-06T00:04:07.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:07.660+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.662048+0000) 2020-02-06T00:04:07.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:07.661+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.662352+0000) 2020-02-06T00:04:07.664 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:07.662+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.662736+0000) 2020-02-06T00:04:07.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:07.662+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.663497+0000) 2020-02-06T00:04:07.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:07.662+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.663763+0000) 2020-02-06T00:04:07.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:07.663+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.663983+0000) 2020-02-06T00:04:07.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:07.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.664123+0000) 2020-02-06T00:04:07.666 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:07.663+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.664566+0000) 2020-02-06T00:04:07.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:07.664+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.664420+0000) 2020-02-06T00:04:07.667 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:07.664+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.664545+0000) 2020-02-06T00:04:07.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:07.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.665300+0000) 2020-02-06T00:04:07.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:07.668+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:07.669809+0000) 2020-02-06T00:04:07.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:07.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:09.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:09.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:09.227570+0000) 2020-02-06T00:04:09.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:09.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:09.611531+0000) 2020-02-06T00:04:10.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:10.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:10.458737+0000) 2020-02-06T00:04:10.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:10.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:10.720032+0000) 2020-02-06T00:04:11.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:11.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:11.883532+0000) 2020-02-06T00:04:12.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:12.683+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.684167+0000) 2020-02-06T00:04:12.686 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:12.683+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.684644+0000) 2020-02-06T00:04:12.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:12.685+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.686508+0000) 2020-02-06T00:04:12.688 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:12.686+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.686589+0000) 2020-02-06T00:04:12.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:12.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.687180+0000) 2020-02-06T00:04:12.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:12.686+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.688031+0000) 2020-02-06T00:04:12.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:12.688+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.688664+0000) 2020-02-06T00:04:12.690 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:12.688+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.689334+0000) 2020-02-06T00:04:12.690 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:12.688+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.689559+0000) 2020-02-06T00:04:12.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:12.689+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.690205+0000) 2020-02-06T00:04:12.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:12.689+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.690407+0000) 2020-02-06T00:04:12.692 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:12.689+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.690369+0000) 2020-02-06T00:04:12.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:12.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.693945+0000) 2020-02-06T00:04:12.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:12.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.694217+0000) 2020-02-06T00:04:12.703 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:12.702+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:12.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:12.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.824483+0000) 2020-02-06T00:04:12.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:12.839+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:12.839275+0000) 2020-02-06T00:04:13.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:13.110+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:13.110994+0000) 2020-02-06T00:04:13.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:13.989+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:13.990325+0000) 2020-02-06T00:04:13.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:13.989+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:13.990325+0000) 2020-02-06T00:04:16.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:16.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:16.967983+0000) 2020-02-06T00:04:17.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:17.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.229032+0000) 2020-02-06T00:04:17.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:17.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.229025+0000) 2020-02-06T00:04:17.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:17.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.229286+0000) 2020-02-06T00:04:17.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:17.707+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.708320+0000) 2020-02-06T00:04:17.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:17.708+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.709339+0000) 2020-02-06T00:04:17.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:17.709+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.709739+0000) 2020-02-06T00:04:17.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:17.709+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.710712+0000) 2020-02-06T00:04:17.712 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:17.710+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.710624+0000) 2020-02-06T00:04:17.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:17.711+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.712004+0000) 2020-02-06T00:04:17.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:17.711+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.712348+0000) 2020-02-06T00:04:17.714 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:17.711+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.712588+0000) 2020-02-06T00:04:17.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:17.712+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.713335+0000) 2020-02-06T00:04:17.715 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:17.713+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.714136+0000) 2020-02-06T00:04:17.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:17.713+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.714476+0000) 2020-02-06T00:04:17.716 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:17.713+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.714577+0000) 2020-02-06T00:04:17.716 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:17.714+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.714488+0000) 2020-02-06T00:04:17.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:17.717+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:17.718541+0000) 2020-02-06T00:04:17.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:17.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:19.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:19.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:19.227764+0000) 2020-02-06T00:04:19.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:19.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:19.611764+0000) 2020-02-06T00:04:20.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:20.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:20.458978+0000) 2020-02-06T00:04:20.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:20.719+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:20.720302+0000) 2020-02-06T00:04:21.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:21.890+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:21.891912+0000) 2020-02-06T00:04:22.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:22.731+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.732268+0000) 2020-02-06T00:04:22.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:22.732+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.733658+0000) 2020-02-06T00:04:22.735 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:22.733+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.734317+0000) 2020-02-06T00:04:22.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:22.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.734658+0000) 2020-02-06T00:04:22.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:22.735+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.735782+0000) 2020-02-06T00:04:22.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:22.735+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.735774+0000) 2020-02-06T00:04:22.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:22.735+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.736876+0000) 2020-02-06T00:04:22.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:22.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.737595+0000) 2020-02-06T00:04:22.739 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:22.737+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.738148+0000) 2020-02-06T00:04:22.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:22.737+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.738265+0000) 2020-02-06T00:04:22.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:22.738+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.738675+0000) 2020-02-06T00:04:22.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:22.738+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.738732+0000) 2020-02-06T00:04:22.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:22.738+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.739144+0000) 2020-02-06T00:04:22.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:22.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.745006+0000) 2020-02-06T00:04:22.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:22.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163180 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:22.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:22.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.824726+0000) 2020-02-06T00:04:22.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:22.839+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:22.839484+0000) 2020-02-06T00:04:23.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:23.110+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:23.111208+0000) 2020-02-06T00:04:23.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:23.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:23.990574+0000) 2020-02-06T00:04:23.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:23.990+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:23.990576+0000) 2020-02-06T00:04:26.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:26.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:26.968290+0000) 2020-02-06T00:04:27.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:27.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.229282+0000) 2020-02-06T00:04:27.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:27.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.229285+0000) 2020-02-06T00:04:27.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:27.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.229547+0000) 2020-02-06T00:04:27.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:27.756+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.757532+0000) 2020-02-06T00:04:27.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:27.757+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.758437+0000) 2020-02-06T00:04:27.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:27.758+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.759263+0000) 2020-02-06T00:04:27.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:27.758+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.759704+0000) 2020-02-06T00:04:27.761 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:27.758+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.759786+0000) 2020-02-06T00:04:27.761 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:27.759+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.760182+0000) 2020-02-06T00:04:27.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:27.759+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.761008+0000) 2020-02-06T00:04:27.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:27.760+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.761991+0000) 2020-02-06T00:04:27.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:27.761+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.762117+0000) 2020-02-06T00:04:27.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:27.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.761777+0000) 2020-02-06T00:04:27.764 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:27.761+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.762427+0000) 2020-02-06T00:04:27.764 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:27.761+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.762929+0000) 2020-02-06T00:04:27.764 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:27.761+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.762797+0000) 2020-02-06T00:04:27.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:27.767+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:27.768686+0000) 2020-02-06T00:04:27.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:27.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:29.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:29.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:29.228031+0000) 2020-02-06T00:04:29.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:29.610+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:29.612018+0000) 2020-02-06T00:04:30.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:30.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:30.459234+0000) 2020-02-06T00:04:30.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:30.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:30.720532+0000) 2020-02-06T00:04:31.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:31.899+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:31.900008+0000) 2020-02-06T00:04:32.692 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:04:32.698 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:04:32.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:32.778+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.779791+0000) 2020-02-06T00:04:32.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:32.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.780120+0000) 2020-02-06T00:04:32.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:32.780+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.781479+0000) 2020-02-06T00:04:32.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:32.780+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.781624+0000) 2020-02-06T00:04:32.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:32.780+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.781980+0000) 2020-02-06T00:04:32.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:32.781+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.782831+0000) 2020-02-06T00:04:32.784 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:32.781+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.782712+0000) 2020-02-06T00:04:32.785 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:32.782+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.783159+0000) 2020-02-06T00:04:32.785 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:32.782+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.783819+0000) 2020-02-06T00:04:32.785 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:32.781+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.783363+0000) 2020-02-06T00:04:32.785 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:32.782+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.783932+0000) 2020-02-06T00:04:32.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:32.783+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.784530+0000) 2020-02-06T00:04:32.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:32.783+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.784891+0000) 2020-02-06T00:04:32.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:32.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.788420+0000) 2020-02-06T00:04:32.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:32.797+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:32.826 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:32.823+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.824959+0000) 2020-02-06T00:04:32.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:32.838+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:32.839727+0000) 2020-02-06T00:04:33.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:33.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:33.111442+0000) 2020-02-06T00:04:33.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:33.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:33.990845+0000) 2020-02-06T00:04:33.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:33.990+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:33.990847+0000) 2020-02-06T00:04:36.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:36.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:36.968479+0000) 2020-02-06T00:04:37.231 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:37.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.229526+0000) 2020-02-06T00:04:37.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:37.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.229565+0000) 2020-02-06T00:04:37.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:37.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.229793+0000) 2020-02-06T00:04:37.805 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:37.803+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.804634+0000) 2020-02-06T00:04:37.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:37.803+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.804513+0000) 2020-02-06T00:04:37.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:37.803+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.804919+0000) 2020-02-06T00:04:37.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:37.804+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.805675+0000) 2020-02-06T00:04:37.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:37.804+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.806305+0000) 2020-02-06T00:04:37.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:37.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.805812+0000) 2020-02-06T00:04:37.808 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:37.805+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.806689+0000) 2020-02-06T00:04:37.808 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:37.805+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.807300+0000) 2020-02-06T00:04:37.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:37.807+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.808491+0000) 2020-02-06T00:04:37.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:37.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.808464+0000) 2020-02-06T00:04:37.810 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:37.807+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.808772+0000) 2020-02-06T00:04:37.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:37.808+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.809311+0000) 2020-02-06T00:04:37.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:37.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.809364+0000) 2020-02-06T00:04:37.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:37.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:37.814332+0000) 2020-02-06T00:04:37.824 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:37.821+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:39.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:39.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:39.228274+0000) 2020-02-06T00:04:39.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:39.612+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:39.612295+0000) 2020-02-06T00:04:40.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:40.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:40.459522+0000) 2020-02-06T00:04:40.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:40.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:40.720767+0000) 2020-02-06T00:04:41.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:41.908+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:41.908223+0000) 2020-02-06T00:04:42.831 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:42.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.829758+0000) 2020-02-06T00:04:42.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:42.828+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.828710+0000) 2020-02-06T00:04:42.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:42.828+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.828998+0000) 2020-02-06T00:04:42.832 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:42.828+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.830387+0000) 2020-02-06T00:04:42.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:42.829+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.830784+0000) 2020-02-06T00:04:42.833 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:42.831+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.831539+0000) 2020-02-06T00:04:42.833 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:42.830+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.832323+0000) 2020-02-06T00:04:42.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:42.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.832173+0000) 2020-02-06T00:04:42.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:42.832+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.832585+0000) 2020-02-06T00:04:42.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:42.831+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.832678+0000) 2020-02-06T00:04:42.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:42.831+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.833152+0000) 2020-02-06T00:04:42.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:42.833+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.833413+0000) 2020-02-06T00:04:42.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:42.833+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.833762+0000) 2020-02-06T00:04:42.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:42.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.837018+0000) 2020-02-06T00:04:42.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:42.838+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:42.839900+0000) 2020-02-06T00:04:42.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:42.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:43.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:43.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:43.111687+0000) 2020-02-06T00:04:43.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:43.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:43.991129+0000) 2020-02-06T00:04:43.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:43.991+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:43.991130+0000) 2020-02-06T00:04:46.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:46.967+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:46.968754+0000) 2020-02-06T00:04:47.231 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:47.228+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.229793+0000) 2020-02-06T00:04:47.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:47.228+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.229801+0000) 2020-02-06T00:04:47.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:47.228+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.230072+0000) 2020-02-06T00:04:47.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:47.848+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.849430+0000) 2020-02-06T00:04:47.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:47.849+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.850691+0000) 2020-02-06T00:04:47.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:47.850+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.851680+0000) 2020-02-06T00:04:47.854 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:47.851+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.852859+0000) 2020-02-06T00:04:47.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:47.852+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.853199+0000) 2020-02-06T00:04:47.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:47.852+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.853951+0000) 2020-02-06T00:04:47.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:47.852+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.854312+0000) 2020-02-06T00:04:47.856 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:47.853+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.854594+0000) 2020-02-06T00:04:47.856 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:47.853+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.855100+0000) 2020-02-06T00:04:47.857 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:47.854+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.855898+0000) 2020-02-06T00:04:47.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:47.855+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.856353+0000) 2020-02-06T00:04:47.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:47.855+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.856485+0000) 2020-02-06T00:04:47.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:47.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.860210+0000) 2020-02-06T00:04:47.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:47.860+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:47.861492+0000) 2020-02-06T00:04:47.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:47.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:49.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:49.227+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:49.228542+0000) 2020-02-06T00:04:49.578 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:49.576+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:49.577263+0000) 2020-02-06T00:04:49.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:49.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:49.612502+0000) 2020-02-06T00:04:50.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:50.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:50.459782+0000) 2020-02-06T00:04:50.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:50.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:50.720985+0000) 2020-02-06T00:04:51.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:51.914+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:51.915327+0000) 2020-02-06T00:04:52.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:52.839+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.840103+0000) 2020-02-06T00:04:52.877 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:52.874+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.875959+0000) 2020-02-06T00:04:52.877 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:52.875+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.876393+0000) 2020-02-06T00:04:52.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:52.875+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.876783+0000) 2020-02-06T00:04:52.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:52.876+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.877292+0000) 2020-02-06T00:04:52.879 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:52.876+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.878010+0000) 2020-02-06T00:04:52.880 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:52.877+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.878829+0000) 2020-02-06T00:04:52.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:52.877+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.879169+0000) 2020-02-06T00:04:52.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:52.878+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.879411+0000) 2020-02-06T00:04:52.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:52.879+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.880128+0000) 2020-02-06T00:04:52.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:52.879+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.880341+0000) 2020-02-06T00:04:52.882 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:52.879+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.880850+0000) 2020-02-06T00:04:52.882 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:52.880+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.881168+0000) 2020-02-06T00:04:52.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:52.880+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.881452+0000) 2020-02-06T00:04:52.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:52.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:52.886200+0000) 2020-02-06T00:04:52.893 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:52.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:53.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:53.110+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:53.111900+0000) 2020-02-06T00:04:53.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:53.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:53.991393+0000) 2020-02-06T00:04:53.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:53.990+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:53.991392+0000) 2020-02-06T00:04:56.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:56.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:56.968993+0000) 2020-02-06T00:04:57.231 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:57.229+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.230039+0000) 2020-02-06T00:04:57.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:57.229+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.230043+0000) 2020-02-06T00:04:57.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:57.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.230309+0000) 2020-02-06T00:04:57.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:57.896+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.898005+0000) 2020-02-06T00:04:57.902 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:57.897+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.899065+0000) 2020-02-06T00:04:57.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:04:57.899+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.900645+0000) 2020-02-06T00:04:57.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:04:57.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.899488+0000) 2020-02-06T00:04:57.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:57.899+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.900522+0000) 2020-02-06T00:04:57.903 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:04:57.900+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.901123+0000) 2020-02-06T00:04:57.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:57.900+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.901584+0000) 2020-02-06T00:04:57.903 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:04:57.900+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.902010+0000) 2020-02-06T00:04:57.904 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:04:57.901+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.902554+0000) 2020-02-06T00:04:57.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:04:57.902+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.903666+0000) 2020-02-06T00:04:57.905 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:04:57.902+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.903971+0000) 2020-02-06T00:04:57.905 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:04:57.902+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.904139+0000) 2020-02-06T00:04:57.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:04:57.902+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.904380+0000) 2020-02-06T00:04:57.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:57.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:57.908805+0000) 2020-02-06T00:04:57.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:04:57.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163462 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:04:59.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:04:59.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:59.228805+0000) 2020-02-06T00:04:59.578 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:04:59.576+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:59.577519+0000) 2020-02-06T00:04:59.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:04:59.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:04:59.612751+0000) 2020-02-06T00:05:00.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:00.458+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:00.460035+0000) 2020-02-06T00:05:00.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:00.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:00.721217+0000) 2020-02-06T00:05:01.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:01.921+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:01.921866+0000) 2020-02-06T00:05:02.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:02.839+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.840369+0000) 2020-02-06T00:05:02.857 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:05:02.863 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:05:02.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:02.919+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.920527+0000) 2020-02-06T00:05:02.922 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:02.919+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.921042+0000) 2020-02-06T00:05:02.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:02.920+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.921531+0000) 2020-02-06T00:05:02.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:02.920+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.921778+0000) 2020-02-06T00:05:02.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:02.921+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.922153+0000) 2020-02-06T00:05:02.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:02.922+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.924096+0000) 2020-02-06T00:05:02.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:02.922+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.924262+0000) 2020-02-06T00:05:02.926 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:02.923+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.924674+0000) 2020-02-06T00:05:02.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:02.923+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.925062+0000) 2020-02-06T00:05:02.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:02.924+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.925383+0000) 2020-02-06T00:05:02.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:02.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.925584+0000) 2020-02-06T00:05:02.927 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:02.925+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.926565+0000) 2020-02-06T00:05:02.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:02.925+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.927106+0000) 2020-02-06T00:05:02.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:02.931+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:02.932192+0000) 2020-02-06T00:05:02.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:02.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:03.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:03.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:03.112152+0000) 2020-02-06T00:05:03.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:03.990+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:03.991657+0000) 2020-02-06T00:05:03.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:03.990+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:03.991657+0000) 2020-02-06T00:05:06.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:06.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:06.969294+0000) 2020-02-06T00:05:07.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:07.229+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.230281+0000) 2020-02-06T00:05:07.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:07.229+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.230284+0000) 2020-02-06T00:05:07.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:07.229+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.230544+0000) 2020-02-06T00:05:07.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:07.945+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.946951+0000) 2020-02-06T00:05:07.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:07.946+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.947373+0000) 2020-02-06T00:05:07.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:07.946+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.947884+0000) 2020-02-06T00:05:07.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:07.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.948328+0000) 2020-02-06T00:05:07.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:07.947+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.948780+0000) 2020-02-06T00:05:07.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:07.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.949792+0000) 2020-02-06T00:05:07.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:07.949+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.950528+0000) 2020-02-06T00:05:07.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:07.949+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.951066+0000) 2020-02-06T00:05:07.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:07.950+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.951194+0000) 2020-02-06T00:05:07.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:07.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.952034+0000) 2020-02-06T00:05:07.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:07.951+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.952614+0000) 2020-02-06T00:05:07.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:07.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.953421+0000) 2020-02-06T00:05:07.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:07.952+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.953759+0000) 2020-02-06T00:05:07.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:07.953+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:07.954326+0000) 2020-02-06T00:05:07.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:07.965+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:09.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:09.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:09.229071+0000) 2020-02-06T00:05:09.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:09.577+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:09.577784+0000) 2020-02-06T00:05:09.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:09.611+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:09.613001+0000) 2020-02-06T00:05:10.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:10.459+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:10.460334+0000) 2020-02-06T00:05:10.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:10.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:10.721480+0000) 2020-02-06T00:05:11.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:11.929+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:11.930260+0000) 2020-02-06T00:05:12.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:12.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.840641+0000) 2020-02-06T00:05:12.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:12.972+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.973588+0000) 2020-02-06T00:05:12.975 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:12.972+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.973459+0000) 2020-02-06T00:05:12.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:12.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.974713+0000) 2020-02-06T00:05:12.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:12.974+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.975232+0000) 2020-02-06T00:05:12.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:12.973+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.974947+0000) 2020-02-06T00:05:12.977 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:12.974+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.975791+0000) 2020-02-06T00:05:12.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:12.975+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.976526+0000) 2020-02-06T00:05:12.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:12.975+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.976679+0000) 2020-02-06T00:05:12.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:12.976+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.977636+0000) 2020-02-06T00:05:12.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:12.977+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.978347+0000) 2020-02-06T00:05:12.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:12.977+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.978697+0000) 2020-02-06T00:05:12.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:12.977+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.978675+0000) 2020-02-06T00:05:12.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:12.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.979289+0000) 2020-02-06T00:05:12.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:12.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:12.987286+0000) 2020-02-06T00:05:12.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:12.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:13.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:13.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:13.112368+0000) 2020-02-06T00:05:13.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:13.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:13.991925+0000) 2020-02-06T00:05:13.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:13.991+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:13.991925+0000) 2020-02-06T00:05:16.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:16.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:16.969538+0000) 2020-02-06T00:05:17.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:17.230+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.230548+0000) 2020-02-06T00:05:17.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:17.230+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.230546+0000) 2020-02-06T00:05:17.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:17.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.230765+0000) 2020-02-06T00:05:17.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:17.994+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.995441+0000) 2020-02-06T00:05:17.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:17.996+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.997382+0000) 2020-02-06T00:05:17.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:17.996+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.997720+0000) 2020-02-06T00:05:17.999 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:17.997+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.998251+0000) 2020-02-06T00:05:18.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:17.998+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.999193+0000) 2020-02-06T00:05:18.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:17.998+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.999355+0000) 2020-02-06T00:05:18.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:17.998+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:17.999815+0000) 2020-02-06T00:05:18.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:17.999+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.000182+0000) 2020-02-06T00:05:18.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:18.000+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.000825+0000) 2020-02-06T00:05:18.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:18.000+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.000922+0000) 2020-02-06T00:05:18.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:18.000+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.001312+0000) 2020-02-06T00:05:18.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:18.000+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.001395+0000) 2020-02-06T00:05:18.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:18.001+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.001821+0000) 2020-02-06T00:05:18.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:18.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:18.007087+0000) 2020-02-06T00:05:18.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:18.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:19.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:19.228+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:19.229337+0000) 2020-02-06T00:05:19.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:19.577+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:19.578012+0000) 2020-02-06T00:05:19.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:19.612+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:19.613261+0000) 2020-02-06T00:05:20.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:20.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:20.460667+0000) 2020-02-06T00:05:20.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:20.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:20.721708+0000) 2020-02-06T00:05:21.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:21.937+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:21.938383+0000) 2020-02-06T00:05:22.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:22.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:22.840905+0000) 2020-02-06T00:05:23.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:23.017+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.018622+0000) 2020-02-06T00:05:23.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:23.019+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.020394+0000) 2020-02-06T00:05:23.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:23.020+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.020858+0000) 2020-02-06T00:05:23.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:23.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.021319+0000) 2020-02-06T00:05:23.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:23.020+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.021781+0000) 2020-02-06T00:05:23.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:23.020+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.021972+0000) 2020-02-06T00:05:23.024 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:23.022+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.022657+0000) 2020-02-06T00:05:23.024 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:23.022+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.022644+0000) 2020-02-06T00:05:23.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:23.022+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.023455+0000) 2020-02-06T00:05:23.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:23.022+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.023339+0000) 2020-02-06T00:05:23.025 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:23.023+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.023973+0000) 2020-02-06T00:05:23.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:23.023+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.024167+0000) 2020-02-06T00:05:23.026 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:23.024+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.024689+0000) 2020-02-06T00:05:23.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:23.029+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.030296+0000) 2020-02-06T00:05:23.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:23.035+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163666 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:23.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:23.112+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.112582+0000) 2020-02-06T00:05:23.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:23.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.992210+0000) 2020-02-06T00:05:23.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:23.991+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:23.992203+0000) 2020-02-06T00:05:26.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:26.968+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:26.969731+0000) 2020-02-06T00:05:27.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:27.230+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:27.230789+0000) 2020-02-06T00:05:27.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:27.230+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:27.230786+0000) 2020-02-06T00:05:27.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:27.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:27.231264+0000) 2020-02-06T00:05:28.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:28.040+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.041774+0000) 2020-02-06T00:05:28.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:28.042+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.043163+0000) 2020-02-06T00:05:28.045 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:28.042+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.044059+0000) 2020-02-06T00:05:28.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:28.043+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.043947+0000) 2020-02-06T00:05:28.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:28.043+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.044527+0000) 2020-02-06T00:05:28.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:28.044+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.044984+0000) 2020-02-06T00:05:28.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:28.045+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.046319+0000) 2020-02-06T00:05:28.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:28.045+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.045948+0000) 2020-02-06T00:05:28.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:28.045+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.046972+0000) 2020-02-06T00:05:28.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:28.046+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.046486+0000) 2020-02-06T00:05:28.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:28.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.047565+0000) 2020-02-06T00:05:28.049 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:28.047+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.047819+0000) 2020-02-06T00:05:28.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:28.048+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.049607+0000) 2020-02-06T00:05:28.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:28.048+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:28.049858+0000) 2020-02-06T00:05:28.060 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:28.058+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163707 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:29.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:29.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:29.229592+0000) 2020-02-06T00:05:29.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:29.578+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:29.578299+0000) 2020-02-06T00:05:29.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:29.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:29.613499+0000) 2020-02-06T00:05:30.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:30.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:30.460921+0000) 2020-02-06T00:05:30.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:30.720+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:30.721928+0000) 2020-02-06T00:05:31.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:31.946+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:31.947548+0000) 2020-02-06T00:05:32.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:32.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:32.841167+0000) 2020-02-06T00:05:33.011 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:05:33.017 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:05:33.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:33.063+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.064872+0000) 2020-02-06T00:05:33.066 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:33.063+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.065011+0000) 2020-02-06T00:05:33.066 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:33.064+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.065248+0000) 2020-02-06T00:05:33.067 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:33.064+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.065456+0000) 2020-02-06T00:05:33.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:33.065+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.066329+0000) 2020-02-06T00:05:33.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:33.065+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.066845+0000) 2020-02-06T00:05:33.068 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:33.066+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.067107+0000) 2020-02-06T00:05:33.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:33.067+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.067492+0000) 2020-02-06T00:05:33.069 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:33.067+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.067609+0000) 2020-02-06T00:05:33.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:33.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.068551+0000) 2020-02-06T00:05:33.070 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:33.068+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.068871+0000) 2020-02-06T00:05:33.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:33.068+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.069392+0000) 2020-02-06T00:05:33.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:33.069+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.069521+0000) 2020-02-06T00:05:33.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:33.071+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.072131+0000) 2020-02-06T00:05:33.082 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:33.081+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:33.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:33.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.112735+0000) 2020-02-06T00:05:33.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:33.992+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.992446+0000) 2020-02-06T00:05:33.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:33.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:33.992447+0000) 2020-02-06T00:05:36.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:36.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:36.969976+0000) 2020-02-06T00:05:37.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:37.230+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:37.231036+0000) 2020-02-06T00:05:37.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:37.230+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:37.231040+0000) 2020-02-06T00:05:37.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:37.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:37.231510+0000) 2020-02-06T00:05:38.088 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:38.087+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.087225+0000) 2020-02-06T00:05:38.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:38.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.088105+0000) 2020-02-06T00:05:38.090 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:38.088+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.088722+0000) 2020-02-06T00:05:38.090 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:38.089+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.089105+0000) 2020-02-06T00:05:38.090 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:38.089+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.089543+0000) 2020-02-06T00:05:38.091 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:38.090+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.090471+0000) 2020-02-06T00:05:38.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:38.090+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.091138+0000) 2020-02-06T00:05:38.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:38.090+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.091248+0000) 2020-02-06T00:05:38.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:38.091+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.091440+0000) 2020-02-06T00:05:38.093 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:38.091+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.091879+0000) 2020-02-06T00:05:38.093 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:38.090+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.090703+0000) 2020-02-06T00:05:38.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:38.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.092421+0000) 2020-02-06T00:05:38.094 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:38.091+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.092083+0000) 2020-02-06T00:05:38.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:38.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:38.096247+0000) 2020-02-06T00:05:38.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:38.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:39.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:39.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:39.229853+0000) 2020-02-06T00:05:39.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:39.577+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:39.578537+0000) 2020-02-06T00:05:39.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:39.612+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:39.613742+0000) 2020-02-06T00:05:40.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:40.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:40.461185+0000) 2020-02-06T00:05:40.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:40.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:40.722198+0000) 2020-02-06T00:05:41.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:41.956+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:41.957362+0000) 2020-02-06T00:05:42.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:42.841+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:42.841458+0000) 2020-02-06T00:05:43.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:43.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.110326+0000) 2020-02-06T00:05:43.112 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:43.109+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.110624+0000) 2020-02-06T00:05:43.112 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:43.110+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.111125+0000) 2020-02-06T00:05:43.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:43.110+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.111270+0000) 2020-02-06T00:05:43.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:43.111+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.111839+0000) 2020-02-06T00:05:43.113 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:43.110+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.111952+0000) 2020-02-06T00:05:43.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:43.111+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.112316+0000) 2020-02-06T00:05:43.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:43.111+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.112853+0000) 2020-02-06T00:05:43.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:43.111+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.112901+0000) 2020-02-06T00:05:43.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:43.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.113166+0000) 2020-02-06T00:05:43.115 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:43.113+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.113641+0000) 2020-02-06T00:05:43.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:43.113+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.114077+0000) 2020-02-06T00:05:43.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:43.114+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.115036+0000) 2020-02-06T00:05:43.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:43.114+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.115155+0000) 2020-02-06T00:05:43.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:43.120+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.121657+0000) 2020-02-06T00:05:43.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:43.129+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:43.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:43.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.992718+0000) 2020-02-06T00:05:43.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:43.991+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:43.992715+0000) 2020-02-06T00:05:46.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:46.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:46.970222+0000) 2020-02-06T00:05:47.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:47.231+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:47.231290+0000) 2020-02-06T00:05:47.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:47.231+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:47.231285+0000) 2020-02-06T00:05:47.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:47.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:47.231711+0000) 2020-02-06T00:05:48.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:48.134+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.135450+0000) 2020-02-06T00:05:48.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:48.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.136263+0000) 2020-02-06T00:05:48.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:48.135+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.136670+0000) 2020-02-06T00:05:48.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:48.136+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.137163+0000) 2020-02-06T00:05:48.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:48.136+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.137488+0000) 2020-02-06T00:05:48.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:48.136+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.137777+0000) 2020-02-06T00:05:48.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:48.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.138346+0000) 2020-02-06T00:05:48.139 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:48.137+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.138711+0000) 2020-02-06T00:05:48.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:48.138+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.139331+0000) 2020-02-06T00:05:48.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:48.139+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.140005+0000) 2020-02-06T00:05:48.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:48.139+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.140314+0000) 2020-02-06T00:05:48.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:48.140+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.141338+0000) 2020-02-06T00:05:48.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:48.140+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.141035+0000) 2020-02-06T00:05:48.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:48.144+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:48.145113+0000) 2020-02-06T00:05:48.157 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:48.155+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:49.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:49.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:49.230117+0000) 2020-02-06T00:05:49.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:49.577+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:49.578768+0000) 2020-02-06T00:05:49.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:49.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:49.613980+0000) 2020-02-06T00:05:50.462 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:50.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:50.461496+0000) 2020-02-06T00:05:50.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:50.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:50.722425+0000) 2020-02-06T00:05:51.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:51.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:51.965527+0000) 2020-02-06T00:05:52.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:52.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:52.841729+0000) 2020-02-06T00:05:53.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:53.112+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.113134+0000) 2020-02-06T00:05:53.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:53.161+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.162309+0000) 2020-02-06T00:05:53.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:53.161+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.162712+0000) 2020-02-06T00:05:53.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:53.162+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.163187+0000) 2020-02-06T00:05:53.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:53.162+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.163502+0000) 2020-02-06T00:05:53.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:53.162+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.163884+0000) 2020-02-06T00:05:53.165 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:53.163+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.164559+0000) 2020-02-06T00:05:53.166 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:53.163+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.164859+0000) 2020-02-06T00:05:53.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:53.164+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.165239+0000) 2020-02-06T00:05:53.166 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:53.164+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.165785+0000) 2020-02-06T00:05:53.167 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:53.164+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.165802+0000) 2020-02-06T00:05:53.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:53.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.165876+0000) 2020-02-06T00:05:53.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:53.165+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.166610+0000) 2020-02-06T00:05:53.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:53.165+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.166741+0000) 2020-02-06T00:05:53.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:53.170+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.171681+0000) 2020-02-06T00:05:53.180 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:53.177+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:53.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:53.991+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.992977+0000) 2020-02-06T00:05:53.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:53.991+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:53.992977+0000) 2020-02-06T00:05:56.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:56.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:56.970446+0000) 2020-02-06T00:05:57.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:57.230+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:57.231535+0000) 2020-02-06T00:05:57.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:57.230+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:57.231528+0000) 2020-02-06T00:05:57.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:57.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:57.231951+0000) 2020-02-06T00:05:58.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:58.183+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.184442+0000) 2020-02-06T00:05:58.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:05:58.183+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.185021+0000) 2020-02-06T00:05:58.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:05:58.184+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.185654+0000) 2020-02-06T00:05:58.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:58.185+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.186631+0000) 2020-02-06T00:05:58.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:05:58.184+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.186344+0000) 2020-02-06T00:05:58.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:58.186+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.187499+0000) 2020-02-06T00:05:58.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:05:58.187+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.188109+0000) 2020-02-06T00:05:58.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:05:58.186+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.187846+0000) 2020-02-06T00:05:58.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:05:58.187+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.188788+0000) 2020-02-06T00:05:58.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:05:58.189+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.190567+0000) 2020-02-06T00:05:58.192 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:05:58.189+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.190677+0000) 2020-02-06T00:05:58.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:05:58.189+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.190298+0000) 2020-02-06T00:05:58.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:58.194+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.195337+0000) 2020-02-06T00:05:58.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:58.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:58.196628+0000) 2020-02-06T00:05:58.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:05:58.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:05:59.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:05:59.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:59.230379+0000) 2020-02-06T00:05:59.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:05:59.578+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:59.578983+0000) 2020-02-06T00:05:59.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:05:59.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:05:59.614211+0000) 2020-02-06T00:06:00.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:00.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:00.461740+0000) 2020-02-06T00:06:00.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:00.721+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:00.722672+0000) 2020-02-06T00:06:01.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:01.971+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:01.972193+0000) 2020-02-06T00:06:02.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:02.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:02.841974+0000) 2020-02-06T00:06:03.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:03.112+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.113357+0000) 2020-02-06T00:06:03.167 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:06:03.174 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:06:03.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:03.205+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.206488+0000) 2020-02-06T00:06:03.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:03.205+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.206693+0000) 2020-02-06T00:06:03.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:03.206+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.207604+0000) 2020-02-06T00:06:03.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:03.207+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.209161+0000) 2020-02-06T00:06:03.210 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:03.208+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.209450+0000) 2020-02-06T00:06:03.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:03.208+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.210003+0000) 2020-02-06T00:06:03.211 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:03.208+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.210124+0000) 2020-02-06T00:06:03.213 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:03.209+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.210418+0000) 2020-02-06T00:06:03.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:03.209+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.210943+0000) 2020-02-06T00:06:03.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:03.209+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.211100+0000) 2020-02-06T00:06:03.213 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:03.210+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.211744+0000) 2020-02-06T00:06:03.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:03.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.211842+0000) 2020-02-06T00:06:03.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:03.210+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.211986+0000) 2020-02-06T00:06:03.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:03.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.213015+0000) 2020-02-06T00:06:03.225 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:03.222+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 163994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:03.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:03.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.993217+0000) 2020-02-06T00:06:03.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:03.992+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:03.993214+0000) 2020-02-06T00:06:06.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:06.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:06.970681+0000) 2020-02-06T00:06:07.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:07.230+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:07.231800+0000) 2020-02-06T00:06:07.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:07.230+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:07.231803+0000) 2020-02-06T00:06:07.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:07.230+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:07.232184+0000) 2020-02-06T00:06:08.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:08.228+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.229549+0000) 2020-02-06T00:06:08.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:08.229+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.230871+0000) 2020-02-06T00:06:08.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:08.230+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.231897+0000) 2020-02-06T00:06:08.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:08.231+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.232628+0000) 2020-02-06T00:06:08.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:08.231+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.233240+0000) 2020-02-06T00:06:08.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:08.232+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.234002+0000) 2020-02-06T00:06:08.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:08.232+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.233956+0000) 2020-02-06T00:06:08.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:08.233+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.234484+0000) 2020-02-06T00:06:08.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:08.234+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.235501+0000) 2020-02-06T00:06:08.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:08.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.235229+0000) 2020-02-06T00:06:08.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:08.234+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.236010+0000) 2020-02-06T00:06:08.237 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:08.235+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.236118+0000) 2020-02-06T00:06:08.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:08.235+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.236345+0000) 2020-02-06T00:06:08.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:08.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:08.239780+0000) 2020-02-06T00:06:08.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:08.249+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164035 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:09.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:09.229+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:09.230655+0000) 2020-02-06T00:06:09.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:09.578+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:09.579222+0000) 2020-02-06T00:06:09.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:09.613+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:09.614437+0000) 2020-02-06T00:06:10.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:10.460+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:10.461922+0000) 2020-02-06T00:06:10.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:10.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:10.722904+0000) 2020-02-06T00:06:11.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:11.983+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:11.984307+0000) 2020-02-06T00:06:12.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:12.840+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:12.842204+0000) 2020-02-06T00:06:13.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:13.112+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.113589+0000) 2020-02-06T00:06:13.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:13.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.256671+0000) 2020-02-06T00:06:13.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:13.256+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.257181+0000) 2020-02-06T00:06:13.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:13.257+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.258391+0000) 2020-02-06T00:06:13.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:13.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.258834+0000) 2020-02-06T00:06:13.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:13.257+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.259135+0000) 2020-02-06T00:06:13.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:13.258+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.259921+0000) 2020-02-06T00:06:13.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:13.260+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.261472+0000) 2020-02-06T00:06:13.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:13.260+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.261805+0000) 2020-02-06T00:06:13.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:13.261+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.262175+0000) 2020-02-06T00:06:13.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:13.261+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.262870+0000) 2020-02-06T00:06:13.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:13.261+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.263022+0000) 2020-02-06T00:06:13.264 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:13.261+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.262573+0000) 2020-02-06T00:06:13.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:13.261+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.262884+0000) 2020-02-06T00:06:13.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:13.265+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.266723+0000) 2020-02-06T00:06:13.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:13.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:13.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:13.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.993407+0000) 2020-02-06T00:06:13.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:13.992+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:13.993409+0000) 2020-02-06T00:06:16.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:16.969+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:16.970979+0000) 2020-02-06T00:06:17.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:17.231+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:17.232042+0000) 2020-02-06T00:06:17.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:17.231+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:17.232047+0000) 2020-02-06T00:06:17.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:17.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:17.232393+0000) 2020-02-06T00:06:18.282 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:18.279+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.280953+0000) 2020-02-06T00:06:18.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:18.280+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.281383+0000) 2020-02-06T00:06:18.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:18.280+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.281772+0000) 2020-02-06T00:06:18.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:18.281+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.283321+0000) 2020-02-06T00:06:18.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:18.282+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.283914+0000) 2020-02-06T00:06:18.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:18.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.284456+0000) 2020-02-06T00:06:18.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:18.283+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.284920+0000) 2020-02-06T00:06:18.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:18.284+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.286038+0000) 2020-02-06T00:06:18.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:18.285+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.286406+0000) 2020-02-06T00:06:18.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:18.285+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.286829+0000) 2020-02-06T00:06:18.288 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:18.285+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.286767+0000) 2020-02-06T00:06:18.288 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:18.286+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.287198+0000) 2020-02-06T00:06:18.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:18.286+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.287660+0000) 2020-02-06T00:06:18.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:18.293+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:18.294188+0000) 2020-02-06T00:06:18.300 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:18.298+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164118 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:19.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:19.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:19.230868+0000) 2020-02-06T00:06:19.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:19.579+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:19.579456+0000) 2020-02-06T00:06:19.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:19.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:19.614702+0000) 2020-02-06T00:06:20.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:20.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:20.462163+0000) 2020-02-06T00:06:20.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:20.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:20.723178+0000) 2020-02-06T00:06:21.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:21.992+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:21.992530+0000) 2020-02-06T00:06:22.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:22.841+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:22.842503+0000) 2020-02-06T00:06:23.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:23.113+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.113818+0000) 2020-02-06T00:06:23.305 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:23.303+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.304343+0000) 2020-02-06T00:06:23.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:23.304+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.305358+0000) 2020-02-06T00:06:23.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:23.305+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.306416+0000) 2020-02-06T00:06:23.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:23.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.306810+0000) 2020-02-06T00:06:23.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:23.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.307192+0000) 2020-02-06T00:06:23.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:23.306+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.308008+0000) 2020-02-06T00:06:23.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:23.307+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.308878+0000) 2020-02-06T00:06:23.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:23.308+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.309596+0000) 2020-02-06T00:06:23.311 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:23.309+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.310247+0000) 2020-02-06T00:06:23.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:23.309+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.310828+0000) 2020-02-06T00:06:23.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:23.310+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.311528+0000) 2020-02-06T00:06:23.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:23.311+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.312573+0000) 2020-02-06T00:06:23.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:23.311+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.312823+0000) 2020-02-06T00:06:23.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:23.315+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.316295+0000) 2020-02-06T00:06:23.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:23.321+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:23.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:23.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.993647+0000) 2020-02-06T00:06:23.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:23.993+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:23.993647+0000) 2020-02-06T00:06:26.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:26.971+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:26.971284+0000) 2020-02-06T00:06:27.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:27.231+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:27.232301+0000) 2020-02-06T00:06:27.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:27.231+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:27.232314+0000) 2020-02-06T00:06:27.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:27.231+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:27.232704+0000) 2020-02-06T00:06:28.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:28.327+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.328356+0000) 2020-02-06T00:06:28.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:28.328+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.329212+0000) 2020-02-06T00:06:28.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:28.329+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.331027+0000) 2020-02-06T00:06:28.333 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:28.330+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.331391+0000) 2020-02-06T00:06:28.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:28.330+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.332011+0000) 2020-02-06T00:06:28.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:28.330+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.332046+0000) 2020-02-06T00:06:28.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:28.329+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.331282+0000) 2020-02-06T00:06:28.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:28.330+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.332111+0000) 2020-02-06T00:06:28.335 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:28.331+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.332789+0000) 2020-02-06T00:06:28.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:28.331+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.333302+0000) 2020-02-06T00:06:28.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:28.332+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.333470+0000) 2020-02-06T00:06:28.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:28.332+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.333999+0000) 2020-02-06T00:06:28.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:28.335+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.336091+0000) 2020-02-06T00:06:28.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:28.335+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:28.336999+0000) 2020-02-06T00:06:28.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:28.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164199 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:29.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:29.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:29.231096+0000) 2020-02-06T00:06:29.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:29.579+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:29.579692+0000) 2020-02-06T00:06:29.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:29.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:29.614924+0000) 2020-02-06T00:06:30.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:30.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:30.462394+0000) 2020-02-06T00:06:30.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:30.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:30.723407+0000) 2020-02-06T00:06:32.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:32.001+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:32.001587+0000) 2020-02-06T00:06:32.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:32.841+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:32.842741+0000) 2020-02-06T00:06:33.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:33.113+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.114049+0000) 2020-02-06T00:06:33.331 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:06:33.337 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:06:33.352 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:33.351+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.351133+0000) 2020-02-06T00:06:33.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:33.350+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.351965+0000) 2020-02-06T00:06:33.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:33.352+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.352150+0000) 2020-02-06T00:06:33.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:33.352+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.352861+0000) 2020-02-06T00:06:33.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:33.353+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.353681+0000) 2020-02-06T00:06:33.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:33.352+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.353636+0000) 2020-02-06T00:06:33.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:33.352+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.353663+0000) 2020-02-06T00:06:33.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:33.353+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.354049+0000) 2020-02-06T00:06:33.356 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:33.353+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.355060+0000) 2020-02-06T00:06:33.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:33.355+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.355833+0000) 2020-02-06T00:06:33.357 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:33.354+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.355456+0000) 2020-02-06T00:06:33.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:33.354+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.355878+0000) 2020-02-06T00:06:33.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:33.362+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.363051+0000) 2020-02-06T00:06:33.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:33.363+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.363264+0000) 2020-02-06T00:06:33.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:33.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:33.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:33.992+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.993888+0000) 2020-02-06T00:06:33.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:33.992+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:33.993889+0000) 2020-02-06T00:06:36.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:36.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:36.971538+0000) 2020-02-06T00:06:37.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:37.232+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:37.232548+0000) 2020-02-06T00:06:37.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:37.232+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:37.232555+0000) 2020-02-06T00:06:37.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:37.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:37.232922+0000) 2020-02-06T00:06:38.377 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:38.375+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.376412+0000) 2020-02-06T00:06:38.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:38.375+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.376871+0000) 2020-02-06T00:06:38.378 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:38.376+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.377503+0000) 2020-02-06T00:06:38.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:38.376+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.377935+0000) 2020-02-06T00:06:38.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:38.377+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.378374+0000) 2020-02-06T00:06:38.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:38.377+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.379048+0000) 2020-02-06T00:06:38.381 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:38.378+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.380041+0000) 2020-02-06T00:06:38.381 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:38.379+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.380403+0000) 2020-02-06T00:06:38.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:38.379+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.381275+0000) 2020-02-06T00:06:38.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:38.379+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.381241+0000) 2020-02-06T00:06:38.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:38.378+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.380088+0000) 2020-02-06T00:06:38.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:38.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.381555+0000) 2020-02-06T00:06:38.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:38.380+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.381781+0000) 2020-02-06T00:06:38.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:38.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:38.386950+0000) 2020-02-06T00:06:38.393 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:38.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:39.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:39.230+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:39.231327+0000) 2020-02-06T00:06:39.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:39.578+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:39.579922+0000) 2020-02-06T00:06:39.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:39.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:39.615159+0000) 2020-02-06T00:06:40.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:40.461+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:40.462612+0000) 2020-02-06T00:06:40.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:40.722+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:40.723627+0000) 2020-02-06T00:06:42.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:42.009+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:42.010300+0000) 2020-02-06T00:06:42.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:42.842+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:42.842985+0000) 2020-02-06T00:06:43.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:43.113+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.114361+0000) 2020-02-06T00:06:43.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:43.396+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.397602+0000) 2020-02-06T00:06:43.400 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:43.397+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.398836+0000) 2020-02-06T00:06:43.401 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:43.399+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.399830+0000) 2020-02-06T00:06:43.401 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:43.399+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.400658+0000) 2020-02-06T00:06:43.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:43.400+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.400816+0000) 2020-02-06T00:06:43.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:43.400+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.401169+0000) 2020-02-06T00:06:43.402 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:43.400+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.401453+0000) 2020-02-06T00:06:43.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:43.401+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.401717+0000) 2020-02-06T00:06:43.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:43.401+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.402162+0000) 2020-02-06T00:06:43.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:43.402+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.403613+0000) 2020-02-06T00:06:43.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:43.403+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.403990+0000) 2020-02-06T00:06:43.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:43.403+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.404626+0000) 2020-02-06T00:06:43.406 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:43.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.404773+0000) 2020-02-06T00:06:43.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:43.408+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.409337+0000) 2020-02-06T00:06:43.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:43.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:43.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:43.993+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.994136+0000) 2020-02-06T00:06:43.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:43.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:43.994139+0000) 2020-02-06T00:06:46.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:46.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:46.971773+0000) 2020-02-06T00:06:47.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:47.232+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:47.232794+0000) 2020-02-06T00:06:47.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:47.232+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:47.232796+0000) 2020-02-06T00:06:47.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:47.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:47.233145+0000) 2020-02-06T00:06:48.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:48.422+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.423600+0000) 2020-02-06T00:06:48.426 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:48.423+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.424177+0000) 2020-02-06T00:06:48.426 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:48.423+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.424604+0000) 2020-02-06T00:06:48.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:48.425+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.425874+0000) 2020-02-06T00:06:48.427 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:48.425+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.426240+0000) 2020-02-06T00:06:48.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:48.426+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.426745+0000) 2020-02-06T00:06:48.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:48.425+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.426914+0000) 2020-02-06T00:06:48.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:48.426+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.428004+0000) 2020-02-06T00:06:48.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:48.427+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.428508+0000) 2020-02-06T00:06:48.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:48.428+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.428826+0000) 2020-02-06T00:06:48.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:48.428+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.429299+0000) 2020-02-06T00:06:48.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:48.429+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.429811+0000) 2020-02-06T00:06:48.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:48.429+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.429736+0000) 2020-02-06T00:06:48.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:48.432+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:48.433860+0000) 2020-02-06T00:06:48.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:48.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:49.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:49.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:49.231559+0000) 2020-02-06T00:06:49.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:49.579+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:49.580161+0000) 2020-02-06T00:06:49.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:49.614+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:49.615383+0000) 2020-02-06T00:06:50.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:50.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:50.462866+0000) 2020-02-06T00:06:50.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:50.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:50.723854+0000) 2020-02-06T00:06:52.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:52.016+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:52.017172+0000) 2020-02-06T00:06:52.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:52.842+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:52.843241+0000) 2020-02-06T00:06:53.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:53.113+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.114590+0000) 2020-02-06T00:06:53.448 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:53.445+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.447020+0000) 2020-02-06T00:06:53.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:53.448+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.449415+0000) 2020-02-06T00:06:53.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:53.449+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.449597+0000) 2020-02-06T00:06:53.451 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:53.449+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.448757+0000) 2020-02-06T00:06:53.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:53.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.450586+0000) 2020-02-06T00:06:53.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:53.450+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.451117+0000) 2020-02-06T00:06:53.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:53.450+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.451649+0000) 2020-02-06T00:06:53.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:53.450+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.451808+0000) 2020-02-06T00:06:53.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:53.453+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.453873+0000) 2020-02-06T00:06:53.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:53.453+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.454016+0000) 2020-02-06T00:06:53.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:53.454+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.454757+0000) 2020-02-06T00:06:53.456 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:53.454+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.454616+0000) 2020-02-06T00:06:53.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:53.456+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.457710+0000) 2020-02-06T00:06:53.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:53.456+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.458040+0000) 2020-02-06T00:06:53.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:53.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:53.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:53.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.994375+0000) 2020-02-06T00:06:53.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:53.993+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:53.994375+0000) 2020-02-06T00:06:56.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:56.970+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:56.972025+0000) 2020-02-06T00:06:57.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:57.232+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:57.233012+0000) 2020-02-06T00:06:57.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:57.232+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:57.233074+0000) 2020-02-06T00:06:57.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:57.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:57.233360+0000) 2020-02-06T00:06:58.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:58.471+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.472064+0000) 2020-02-06T00:06:58.474 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:06:58.471+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.472937+0000) 2020-02-06T00:06:58.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:58.472+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.473632+0000) 2020-02-06T00:06:58.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:06:58.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.474044+0000) 2020-02-06T00:06:58.475 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:06:58.473+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.474296+0000) 2020-02-06T00:06:58.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:58.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.474937+0000) 2020-02-06T00:06:58.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:06:58.474+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.475149+0000) 2020-02-06T00:06:58.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:06:58.474+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.475610+0000) 2020-02-06T00:06:58.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:06:58.475+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.475908+0000) 2020-02-06T00:06:58.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:06:58.475+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.476358+0000) 2020-02-06T00:06:58.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:06:58.476+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.477123+0000) 2020-02-06T00:06:58.478 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:06:58.476+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.476928+0000) 2020-02-06T00:06:58.479 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:58.476+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.477317+0000) 2020-02-06T00:06:58.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:58.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:58.481261+0000) 2020-02-06T00:06:58.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:06:58.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:06:59.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:06:59.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:59.231789+0000) 2020-02-06T00:06:59.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:06:59.579+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:59.580344+0000) 2020-02-06T00:06:59.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:06:59.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:06:59.615619+0000) 2020-02-06T00:07:00.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:00.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:00.463126+0000) 2020-02-06T00:07:00.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:00.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:00.724116+0000) 2020-02-06T00:07:02.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:02.025+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:02.025803+0000) 2020-02-06T00:07:02.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:02.843+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:02.843539+0000) 2020-02-06T00:07:03.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:03.114+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.114831+0000) 2020-02-06T00:07:03.453 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:07:03.459 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:07:03.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:03.496+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.497121+0000) 2020-02-06T00:07:03.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:03.494+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.495202+0000) 2020-02-06T00:07:03.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:03.497+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.498628+0000) 2020-02-06T00:07:03.500 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:03.497+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.498942+0000) 2020-02-06T00:07:03.501 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:03.498+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.499422+0000) 2020-02-06T00:07:03.502 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:03.498+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.499255+0000) 2020-02-06T00:07:03.502 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:03.499+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.499988+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:03.499+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.499920+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:03.499+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.500206+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:03.500+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.501279+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:03.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.499810+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:03.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.501096+0000) 2020-02-06T00:07:03.503 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:03.499+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.500873+0000) 2020-02-06T00:07:03.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:03.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.504305+0000) 2020-02-06T00:07:03.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:03.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:03.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:03.993+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.994608+0000) 2020-02-06T00:07:03.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:03.993+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:03.994605+0000) 2020-02-06T00:07:06.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:06.971+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:06.972286+0000) 2020-02-06T00:07:07.235 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:07.233+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:07.233271+0000) 2020-02-06T00:07:07.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:07.233+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:07.233344+0000) 2020-02-06T00:07:07.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:07.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:07.233657+0000) 2020-02-06T00:07:08.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:08.518+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.519867+0000) 2020-02-06T00:07:08.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:08.519+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.520629+0000) 2020-02-06T00:07:08.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:08.520+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.521349+0000) 2020-02-06T00:07:08.522 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:08.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.521735+0000) 2020-02-06T00:07:08.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:08.520+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.522015+0000) 2020-02-06T00:07:08.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:08.522+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.522592+0000) 2020-02-06T00:07:08.524 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:08.521+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.522724+0000) 2020-02-06T00:07:08.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:08.523+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.523547+0000) 2020-02-06T00:07:08.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:08.522+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.523836+0000) 2020-02-06T00:07:08.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:08.523+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.523575+0000) 2020-02-06T00:07:08.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:08.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.524815+0000) 2020-02-06T00:07:08.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:08.523+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.524389+0000) 2020-02-06T00:07:08.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:08.525+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.525931+0000) 2020-02-06T00:07:08.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:08.531+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:08.532969+0000) 2020-02-06T00:07:08.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:08.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:09.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:09.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:09.232019+0000) 2020-02-06T00:07:09.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:09.579+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:09.580572+0000) 2020-02-06T00:07:09.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:09.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:09.615847+0000) 2020-02-06T00:07:10.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:10.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:10.463386+0000) 2020-02-06T00:07:10.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:10.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:10.724350+0000) 2020-02-06T00:07:12.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:12.033+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:12.033869+0000) 2020-02-06T00:07:12.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:12.843+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:12.843795+0000) 2020-02-06T00:07:13.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:13.114+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.115102+0000) 2020-02-06T00:07:13.548 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:13.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.546864+0000) 2020-02-06T00:07:13.548 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:13.546+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.547287+0000) 2020-02-06T00:07:13.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:13.547+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.547830+0000) 2020-02-06T00:07:13.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:13.547+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.548310+0000) 2020-02-06T00:07:13.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:13.547+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.548799+0000) 2020-02-06T00:07:13.550 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:13.547+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.548989+0000) 2020-02-06T00:07:13.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:13.548+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.549146+0000) 2020-02-06T00:07:13.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:13.549+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.549464+0000) 2020-02-06T00:07:13.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:13.548+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.549849+0000) 2020-02-06T00:07:13.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:13.549+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.550208+0000) 2020-02-06T00:07:13.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:13.549+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.550650+0000) 2020-02-06T00:07:13.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:13.550+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.550974+0000) 2020-02-06T00:07:13.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:13.550+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.551243+0000) 2020-02-06T00:07:13.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:13.555+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.556400+0000) 2020-02-06T00:07:13.562 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:13.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164574 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:13.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:13.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.994845+0000) 2020-02-06T00:07:13.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:13.994+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:13.994845+0000) 2020-02-06T00:07:16.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:16.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:16.972557+0000) 2020-02-06T00:07:17.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:17.233+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:17.233617+0000) 2020-02-06T00:07:17.235 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:17.233+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:17.233562+0000) 2020-02-06T00:07:17.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:17.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:17.233913+0000) 2020-02-06T00:07:18.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:18.567+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.568562+0000) 2020-02-06T00:07:18.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:18.568+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.568566+0000) 2020-02-06T00:07:18.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:18.568+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.569269+0000) 2020-02-06T00:07:18.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:18.569+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.570205+0000) 2020-02-06T00:07:18.571 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:18.569+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.570549+0000) 2020-02-06T00:07:18.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:18.571+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.571478+0000) 2020-02-06T00:07:18.573 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:18.571+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.572305+0000) 2020-02-06T00:07:18.573 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:18.571+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.571906+0000) 2020-02-06T00:07:18.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:18.572+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.573041+0000) 2020-02-06T00:07:18.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:18.572+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.573244+0000) 2020-02-06T00:07:18.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:18.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.573714+0000) 2020-02-06T00:07:18.575 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:18.573+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.573937+0000) 2020-02-06T00:07:18.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:18.576+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.577095+0000) 2020-02-06T00:07:18.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:18.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:18.578296+0000) 2020-02-06T00:07:18.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:18.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:19.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:19.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:19.232250+0000) 2020-02-06T00:07:19.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:19.580+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:19.580849+0000) 2020-02-06T00:07:19.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:19.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:19.616071+0000) 2020-02-06T00:07:20.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:20.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:20.463682+0000) 2020-02-06T00:07:20.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:20.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:20.724585+0000) 2020-02-06T00:07:22.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:22.041+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:22.041751+0000) 2020-02-06T00:07:22.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:22.843+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:22.844092+0000) 2020-02-06T00:07:23.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:23.115+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.115339+0000) 2020-02-06T00:07:23.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:23.592+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.593462+0000) 2020-02-06T00:07:23.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:23.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.594631+0000) 2020-02-06T00:07:23.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:23.596+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.596536+0000) 2020-02-06T00:07:23.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:23.596+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.597275+0000) 2020-02-06T00:07:23.599 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:23.594+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.595587+0000) 2020-02-06T00:07:23.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:23.594+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.596228+0000) 2020-02-06T00:07:23.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:23.595+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.596549+0000) 2020-02-06T00:07:23.600 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:23.596+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.597653+0000) 2020-02-06T00:07:23.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:23.596+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.597841+0000) 2020-02-06T00:07:23.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:23.597+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.598952+0000) 2020-02-06T00:07:23.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:23.598+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.599384+0000) 2020-02-06T00:07:23.601 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:23.596+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.598337+0000) 2020-02-06T00:07:23.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:23.598+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.599498+0000) 2020-02-06T00:07:23.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:23.604+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.605462+0000) 2020-02-06T00:07:23.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:23.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164658 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:23.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:23.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.995129+0000) 2020-02-06T00:07:23.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:23.995+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:23.995129+0000) 2020-02-06T00:07:26.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:26.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:26.972791+0000) 2020-02-06T00:07:27.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:27.232+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:27.233850+0000) 2020-02-06T00:07:27.235 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:27.232+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:27.233836+0000) 2020-02-06T00:07:27.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:27.232+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:27.234148+0000) 2020-02-06T00:07:28.619 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:28.616+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.617843+0000) 2020-02-06T00:07:28.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:28.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.618413+0000) 2020-02-06T00:07:28.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:28.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.618914+0000) 2020-02-06T00:07:28.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:28.618+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.619483+0000) 2020-02-06T00:07:28.621 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:28.618+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.619836+0000) 2020-02-06T00:07:28.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:28.619+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.621022+0000) 2020-02-06T00:07:28.622 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:28.620+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.621429+0000) 2020-02-06T00:07:28.623 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:28.620+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.621768+0000) 2020-02-06T00:07:28.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:28.621+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.622656+0000) 2020-02-06T00:07:28.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:28.621+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.623111+0000) 2020-02-06T00:07:28.624 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:28.621+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.622983+0000) 2020-02-06T00:07:28.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:28.621+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.622789+0000) 2020-02-06T00:07:28.625 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:28.622+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.623883+0000) 2020-02-06T00:07:28.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:28.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:28.626490+0000) 2020-02-06T00:07:28.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:28.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164697 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:29.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:29.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:29.232485+0000) 2020-02-06T00:07:29.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:29.580+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:29.581148+0000) 2020-02-06T00:07:29.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:29.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:29.616358+0000) 2020-02-06T00:07:30.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:30.462+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:30.463911+0000) 2020-02-06T00:07:30.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:30.723+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:30.724817+0000) 2020-02-06T00:07:32.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:32.048+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:32.049878+0000) 2020-02-06T00:07:32.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:32.843+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:32.844333+0000) 2020-02-06T00:07:33.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:33.114+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.115565+0000) 2020-02-06T00:07:33.587 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:07:33.593 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:07:33.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:33.642+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.643510+0000) 2020-02-06T00:07:33.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:33.641+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.642665+0000) 2020-02-06T00:07:33.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:33.643+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.644553+0000) 2020-02-06T00:07:33.647 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:33.643+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.644984+0000) 2020-02-06T00:07:33.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:33.644+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.645495+0000) 2020-02-06T00:07:33.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:33.644+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.646289+0000) 2020-02-06T00:07:33.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:33.645+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.646170+0000) 2020-02-06T00:07:33.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:33.645+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.646572+0000) 2020-02-06T00:07:33.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:33.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.647025+0000) 2020-02-06T00:07:33.649 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:33.646+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.647584+0000) 2020-02-06T00:07:33.649 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:33.646+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.648244+0000) 2020-02-06T00:07:33.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:33.646+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.647672+0000) 2020-02-06T00:07:33.650 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:33.647+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.648594+0000) 2020-02-06T00:07:33.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:33.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.649693+0000) 2020-02-06T00:07:33.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:33.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:33.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:33.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.995380+0000) 2020-02-06T00:07:33.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:33.994+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:33.995382+0000) 2020-02-06T00:07:36.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:36.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:36.973027+0000) 2020-02-06T00:07:37.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:37.233+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:37.234130+0000) 2020-02-06T00:07:37.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:37.233+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:37.234101+0000) 2020-02-06T00:07:37.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:37.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:37.234366+0000) 2020-02-06T00:07:38.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:38.666+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.667687+0000) 2020-02-06T00:07:38.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:38.666+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.668040+0000) 2020-02-06T00:07:38.669 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:38.667+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.668432+0000) 2020-02-06T00:07:38.670 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:38.667+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.668773+0000) 2020-02-06T00:07:38.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:38.667+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.668988+0000) 2020-02-06T00:07:38.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:38.668+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.669158+0000) 2020-02-06T00:07:38.671 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:38.668+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.669639+0000) 2020-02-06T00:07:38.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:38.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.669882+0000) 2020-02-06T00:07:38.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:38.668+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.670354+0000) 2020-02-06T00:07:38.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:38.669+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.670448+0000) 2020-02-06T00:07:38.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:38.669+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.670549+0000) 2020-02-06T00:07:38.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:38.670+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.671139+0000) 2020-02-06T00:07:38.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:38.669+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.671135+0000) 2020-02-06T00:07:38.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:38.674+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:38.675111+0000) 2020-02-06T00:07:38.685 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:38.683+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:39.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:39.231+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:39.232712+0000) 2020-02-06T00:07:39.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:39.580+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:39.581375+0000) 2020-02-06T00:07:39.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:39.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:39.616559+0000) 2020-02-06T00:07:40.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:40.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:40.464169+0000) 2020-02-06T00:07:40.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:40.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:40.725052+0000) 2020-02-06T00:07:42.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:42.056+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:42.057803+0000) 2020-02-06T00:07:42.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:42.843+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:42.844574+0000) 2020-02-06T00:07:43.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:43.114+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.115786+0000) 2020-02-06T00:07:43.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:43.690+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.691470+0000) 2020-02-06T00:07:43.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:43.690+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.691640+0000) 2020-02-06T00:07:43.693 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:43.691+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.692428+0000) 2020-02-06T00:07:43.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:43.691+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.692654+0000) 2020-02-06T00:07:43.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:43.691+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.693020+0000) 2020-02-06T00:07:43.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:43.691+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.693181+0000) 2020-02-06T00:07:43.695 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:43.692+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.693462+0000) 2020-02-06T00:07:43.695 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:43.692+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.694010+0000) 2020-02-06T00:07:43.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:43.693+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.694203+0000) 2020-02-06T00:07:43.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:43.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.694580+0000) 2020-02-06T00:07:43.698 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:43.694+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.695842+0000) 2020-02-06T00:07:43.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:43.694+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.696155+0000) 2020-02-06T00:07:43.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:43.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.700417+0000) 2020-02-06T00:07:43.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:43.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.700683+0000) 2020-02-06T00:07:43.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:43.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:43.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:43.994+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.995635+0000) 2020-02-06T00:07:43.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:43.994+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:43.995630+0000) 2020-02-06T00:07:46.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:46.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:46.973296+0000) 2020-02-06T00:07:47.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:47.232+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:47.234379+0000) 2020-02-06T00:07:47.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:47.232+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:47.234374+0000) 2020-02-06T00:07:47.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:47.233+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:47.234675+0000) 2020-02-06T00:07:48.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:48.713+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.714532+0000) 2020-02-06T00:07:48.716 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:48.714+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.715556+0000) 2020-02-06T00:07:48.717 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:48.714+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.715891+0000) 2020-02-06T00:07:48.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:48.715+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.716319+0000) 2020-02-06T00:07:48.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:48.715+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.716623+0000) 2020-02-06T00:07:48.718 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:48.716+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.717313+0000) 2020-02-06T00:07:48.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:48.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.718555+0000) 2020-02-06T00:07:48.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:48.716+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.718249+0000) 2020-02-06T00:07:48.720 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:48.717+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.718974+0000) 2020-02-06T00:07:48.720 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:48.718+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.719661+0000) 2020-02-06T00:07:48.721 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:48.718+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.719902+0000) 2020-02-06T00:07:48.721 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:48.718+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.720192+0000) 2020-02-06T00:07:48.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:48.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.721904+0000) 2020-02-06T00:07:48.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:48.721+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:48.722176+0000) 2020-02-06T00:07:48.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:48.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:49.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:49.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:49.232943+0000) 2020-02-06T00:07:49.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:49.580+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:49.581590+0000) 2020-02-06T00:07:49.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:49.615+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:49.616766+0000) 2020-02-06T00:07:50.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:50.463+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:50.464407+0000) 2020-02-06T00:07:50.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:50.724+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:50.725380+0000) 2020-02-06T00:07:52.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:52.064+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:52.065343+0000) 2020-02-06T00:07:52.846 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:52.844+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:52.844832+0000) 2020-02-06T00:07:53.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:53.114+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.116019+0000) 2020-02-06T00:07:53.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:53.737+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.739016+0000) 2020-02-06T00:07:53.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:53.739+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.740409+0000) 2020-02-06T00:07:53.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:53.739+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.740918+0000) 2020-02-06T00:07:53.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:53.740+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.741552+0000) 2020-02-06T00:07:53.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:53.740+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.741785+0000) 2020-02-06T00:07:53.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:53.741+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.742371+0000) 2020-02-06T00:07:53.744 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:53.741+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.742830+0000) 2020-02-06T00:07:53.744 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:53.741+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.742843+0000) 2020-02-06T00:07:53.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:53.743+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.744139+0000) 2020-02-06T00:07:53.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:53.743+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.744494+0000) 2020-02-06T00:07:53.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:53.743+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.744512+0000) 2020-02-06T00:07:53.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:53.742+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.743878+0000) 2020-02-06T00:07:53.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:53.743+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.745009+0000) 2020-02-06T00:07:53.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:53.747+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.748843+0000) 2020-02-06T00:07:53.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:53.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:53.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:53.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.995873+0000) 2020-02-06T00:07:53.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:53.995+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:53.995873+0000) 2020-02-06T00:07:56.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:56.972+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:56.973534+0000) 2020-02-06T00:07:57.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:57.234+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:57.234669+0000) 2020-02-06T00:07:57.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:57.234+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:57.234723+0000) 2020-02-06T00:07:57.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:57.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:57.234910+0000) 2020-02-06T00:07:58.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:58.761+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.762986+0000) 2020-02-06T00:07:58.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:58.763+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.764196+0000) 2020-02-06T00:07:58.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:58.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.765007+0000) 2020-02-06T00:07:58.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:07:58.763+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.764975+0000) 2020-02-06T00:07:58.767 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:58.764+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.766163+0000) 2020-02-06T00:07:58.767 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:07:58.765+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.766342+0000) 2020-02-06T00:07:58.768 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:07:58.765+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.766940+0000) 2020-02-06T00:07:58.768 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:07:58.766+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.767755+0000) 2020-02-06T00:07:58.770 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:07:58.767+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.769023+0000) 2020-02-06T00:07:58.770 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:07:58.767+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.768838+0000) 2020-02-06T00:07:58.770 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:07:58.767+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.769088+0000) 2020-02-06T00:07:58.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:07:58.768+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.769476+0000) 2020-02-06T00:07:58.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:07:58.768+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.769748+0000) 2020-02-06T00:07:58.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:58.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:58.774821+0000) 2020-02-06T00:07:58.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:07:58.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:07:59.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:07:59.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:59.233171+0000) 2020-02-06T00:07:59.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:07:59.581+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:59.581804+0000) 2020-02-06T00:07:59.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:07:59.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:07:59.616990+0000) 2020-02-06T00:08:00.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:00.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:00.464671+0000) 2020-02-06T00:08:00.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:00.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:00.725608+0000) 2020-02-06T00:08:02.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:02.076+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:02.077267+0000) 2020-02-06T00:08:02.846 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:02.844+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:02.845088+0000) 2020-02-06T00:08:03.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:03.115+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.116272+0000) 2020-02-06T00:08:03.722 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:08:03.728 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:08:03.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:03.782+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.783700+0000) 2020-02-06T00:08:03.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:03.784+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.785479+0000) 2020-02-06T00:08:03.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:03.784+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.785934+0000) 2020-02-06T00:08:03.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:03.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.786799+0000) 2020-02-06T00:08:03.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:03.785+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.786675+0000) 2020-02-06T00:08:03.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:03.786+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.787136+0000) 2020-02-06T00:08:03.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:03.786+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.787453+0000) 2020-02-06T00:08:03.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:03.786+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.787733+0000) 2020-02-06T00:08:03.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:03.786+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.787852+0000) 2020-02-06T00:08:03.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:03.787+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.788735+0000) 2020-02-06T00:08:03.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:03.787+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.788848+0000) 2020-02-06T00:08:03.791 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:03.788+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.789753+0000) 2020-02-06T00:08:03.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:03.788+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.790076+0000) 2020-02-06T00:08:03.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:03.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.794952+0000) 2020-02-06T00:08:03.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:03.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 164988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:03.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:03.995+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.996137+0000) 2020-02-06T00:08:03.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:03.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:03.996164+0000) 2020-02-06T00:08:06.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:06.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:06.973766+0000) 2020-02-06T00:08:07.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:07.234+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:07.234977+0000) 2020-02-06T00:08:07.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:07.234+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:07.234922+0000) 2020-02-06T00:08:07.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:07.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:07.235147+0000) 2020-02-06T00:08:08.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:08.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.809802+0000) 2020-02-06T00:08:08.812 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:08.809+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.810817+0000) 2020-02-06T00:08:08.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:08.810+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.811564+0000) 2020-02-06T00:08:08.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:08.810+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.812028+0000) 2020-02-06T00:08:08.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:08.811+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.812460+0000) 2020-02-06T00:08:08.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:08.811+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.812589+0000) 2020-02-06T00:08:08.815 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:08.812+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.813707+0000) 2020-02-06T00:08:08.815 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:08.813+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.813414+0000) 2020-02-06T00:08:08.815 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:08.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.814139+0000) 2020-02-06T00:08:08.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:08.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.815249+0000) 2020-02-06T00:08:08.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:08.815+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.815996+0000) 2020-02-06T00:08:08.817 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:08.815+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.816261+0000) 2020-02-06T00:08:08.817 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:08.816+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.816808+0000) 2020-02-06T00:08:08.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:08.817+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:08.818382+0000) 2020-02-06T00:08:08.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:08.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:09.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:09.232+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:09.233409+0000) 2020-02-06T00:08:09.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:09.581+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:09.582055+0000) 2020-02-06T00:08:09.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:09.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:09.617199+0000) 2020-02-06T00:08:10.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:10.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:10.464922+0000) 2020-02-06T00:08:10.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:10.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:10.725810+0000) 2020-02-06T00:08:12.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:12.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:12.085709+0000) 2020-02-06T00:08:12.846 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:12.844+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:12.845323+0000) 2020-02-06T00:08:13.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:13.116+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.116529+0000) 2020-02-06T00:08:13.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:13.833+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.834398+0000) 2020-02-06T00:08:13.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:13.833+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.835031+0000) 2020-02-06T00:08:13.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:13.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.835649+0000) 2020-02-06T00:08:13.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:13.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.835783+0000) 2020-02-06T00:08:13.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:13.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.836321+0000) 2020-02-06T00:08:13.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:13.835+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.836512+0000) 2020-02-06T00:08:13.838 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:13.837+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.837464+0000) 2020-02-06T00:08:13.839 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:13.837+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.837698+0000) 2020-02-06T00:08:13.841 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:13.837+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.838154+0000) 2020-02-06T00:08:13.841 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:13.837+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.838219+0000) 2020-02-06T00:08:13.841 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:13.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.838357+0000) 2020-02-06T00:08:13.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:13.838+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.838833+0000) 2020-02-06T00:08:13.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:13.839+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.839580+0000) 2020-02-06T00:08:13.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:13.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.843929+0000) 2020-02-06T00:08:13.853 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:13.851+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:13.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:13.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.996393+0000) 2020-02-06T00:08:13.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:13.996+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:13.996392+0000) 2020-02-06T00:08:16.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:16.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:16.974017+0000) 2020-02-06T00:08:17.236 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:17.234+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:17.235227+0000) 2020-02-06T00:08:17.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:17.234+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:17.235234+0000) 2020-02-06T00:08:17.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:17.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:17.235374+0000) 2020-02-06T00:08:18.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:18.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.857354+0000) 2020-02-06T00:08:18.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:18.857+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.857802+0000) 2020-02-06T00:08:18.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:18.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.859817+0000) 2020-02-06T00:08:18.862 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:18.860+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.860633+0000) 2020-02-06T00:08:18.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:18.861+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.861268+0000) 2020-02-06T00:08:18.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:18.861+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.861510+0000) 2020-02-06T00:08:18.863 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:18.862+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.862321+0000) 2020-02-06T00:08:18.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:18.862+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.862721+0000) 2020-02-06T00:08:18.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:18.863+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.863421+0000) 2020-02-06T00:08:18.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:18.863+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.863887+0000) 2020-02-06T00:08:18.865 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:18.863+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.864114+0000) 2020-02-06T00:08:18.865 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:18.863+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.864390+0000) 2020-02-06T00:08:18.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:18.864+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.864673+0000) 2020-02-06T00:08:18.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:18.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:18.867267+0000) 2020-02-06T00:08:18.880 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:18.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165110 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:19.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:19.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:19.233678+0000) 2020-02-06T00:08:19.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:19.582+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:19.582357+0000) 2020-02-06T00:08:19.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:19.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:19.617430+0000) 2020-02-06T00:08:20.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:20.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:20.465159+0000) 2020-02-06T00:08:20.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:20.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:20.726068+0000) 2020-02-06T00:08:22.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:22.093+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:22.093755+0000) 2020-02-06T00:08:22.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:22.845+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:22.845565+0000) 2020-02-06T00:08:23.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:23.115+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.116770+0000) 2020-02-06T00:08:23.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:23.884+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.884498+0000) 2020-02-06T00:08:23.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:23.884+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.885985+0000) 2020-02-06T00:08:23.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:23.885+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.887009+0000) 2020-02-06T00:08:23.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:23.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.887425+0000) 2020-02-06T00:08:23.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:23.886+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.888002+0000) 2020-02-06T00:08:23.889 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:23.887+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.888377+0000) 2020-02-06T00:08:23.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:23.887+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.888630+0000) 2020-02-06T00:08:23.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:23.887+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.888348+0000) 2020-02-06T00:08:23.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:23.888+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.889341+0000) 2020-02-06T00:08:23.891 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:23.889+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.889757+0000) 2020-02-06T00:08:23.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:23.889+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.890473+0000) 2020-02-06T00:08:23.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:23.890+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.891042+0000) 2020-02-06T00:08:23.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:23.889+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.890029+0000) 2020-02-06T00:08:23.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:23.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.898073+0000) 2020-02-06T00:08:23.903 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:23.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:23.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:23.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.996646+0000) 2020-02-06T00:08:23.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:23.995+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:23.996647+0000) 2020-02-06T00:08:26.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:26.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:26.974269+0000) 2020-02-06T00:08:27.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:27.235+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:27.235492+0000) 2020-02-06T00:08:27.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:27.235+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:27.235490+0000) 2020-02-06T00:08:27.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:27.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:27.235668+0000) 2020-02-06T00:08:28.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:28.908+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.908600+0000) 2020-02-06T00:08:28.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:28.907+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.908814+0000) 2020-02-06T00:08:28.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:28.908+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.909333+0000) 2020-02-06T00:08:28.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:28.909+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.910382+0000) 2020-02-06T00:08:28.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:28.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.909938+0000) 2020-02-06T00:08:28.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:28.910+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.911193+0000) 2020-02-06T00:08:28.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:28.910+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.911601+0000) 2020-02-06T00:08:28.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:28.910+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.911719+0000) 2020-02-06T00:08:28.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:28.911+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.912220+0000) 2020-02-06T00:08:28.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:28.912+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.912815+0000) 2020-02-06T00:08:28.914 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:28.913+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.913819+0000) 2020-02-06T00:08:28.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:28.913+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.913930+0000) 2020-02-06T00:08:28.915 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:28.913+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.914197+0000) 2020-02-06T00:08:28.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:28.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:28.919773+0000) 2020-02-06T00:08:28.928 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:28.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165192 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:29.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:29.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:29.233886+0000) 2020-02-06T00:08:29.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:29.581+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:29.582588+0000) 2020-02-06T00:08:29.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:29.616+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:29.617643+0000) 2020-02-06T00:08:30.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:30.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:30.465453+0000) 2020-02-06T00:08:30.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:30.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:30.726300+0000) 2020-02-06T00:08:32.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:32.101+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:32.101950+0000) 2020-02-06T00:08:32.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:32.845+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:32.845809+0000) 2020-02-06T00:08:33.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:33.115+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.117017+0000) 2020-02-06T00:08:33.867 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:08:33.873 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:08:33.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:33.932+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.933371+0000) 2020-02-06T00:08:33.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:33.933+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.934137+0000) 2020-02-06T00:08:33.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:33.933+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.934769+0000) 2020-02-06T00:08:33.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:33.934+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.935406+0000) 2020-02-06T00:08:33.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:33.935+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.935938+0000) 2020-02-06T00:08:33.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:33.934+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.935819+0000) 2020-02-06T00:08:33.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:33.935+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.936348+0000) 2020-02-06T00:08:33.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:33.935+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.936477+0000) 2020-02-06T00:08:33.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:33.936+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.936811+0000) 2020-02-06T00:08:33.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:33.937+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.937505+0000) 2020-02-06T00:08:33.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:33.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.938082+0000) 2020-02-06T00:08:33.939 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:33.937+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.938370+0000) 2020-02-06T00:08:33.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:33.940+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.941313+0000) 2020-02-06T00:08:33.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:33.940+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.941672+0000) 2020-02-06T00:08:33.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:33.951+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:33.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:33.995+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.996889+0000) 2020-02-06T00:08:33.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:33.995+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:33.996888+0000) 2020-02-06T00:08:36.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:36.973+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:36.974502+0000) 2020-02-06T00:08:37.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:37.235+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:37.235727+0000) 2020-02-06T00:08:37.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:37.235+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:37.235725+0000) 2020-02-06T00:08:37.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:37.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:37.235901+0000) 2020-02-06T00:08:38.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:38.956+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.956673+0000) 2020-02-06T00:08:38.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:38.957+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.957819+0000) 2020-02-06T00:08:38.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:38.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.958333+0000) 2020-02-06T00:08:38.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:38.957+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.958790+0000) 2020-02-06T00:08:38.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:38.958+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.959571+0000) 2020-02-06T00:08:38.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:38.959+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.960152+0000) 2020-02-06T00:08:38.962 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:38.960+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.961117+0000) 2020-02-06T00:08:38.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:38.961+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.961995+0000) 2020-02-06T00:08:38.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:38.961+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.962252+0000) 2020-02-06T00:08:38.963 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:38.962+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.962514+0000) 2020-02-06T00:08:38.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:38.961+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.963004+0000) 2020-02-06T00:08:38.965 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:38.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.964029+0000) 2020-02-06T00:08:38.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:38.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.967154+0000) 2020-02-06T00:08:38.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:38.967+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:38.968149+0000) 2020-02-06T00:08:38.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:38.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165274 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:39.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:39.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:39.234109+0000) 2020-02-06T00:08:39.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:39.582+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:39.582800+0000) 2020-02-06T00:08:39.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:39.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:39.617858+0000) 2020-02-06T00:08:40.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:40.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:40.465739+0000) 2020-02-06T00:08:40.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:40.725+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:40.726568+0000) 2020-02-06T00:08:42.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:42.109+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:42.110179+0000) 2020-02-06T00:08:42.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:42.845+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:42.846046+0000) 2020-02-06T00:08:43.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:43.116+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.117285+0000) 2020-02-06T00:08:43.983 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:43.981+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.982004+0000) 2020-02-06T00:08:43.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:43.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.984139+0000) 2020-02-06T00:08:43.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:43.983+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.984646+0000) 2020-02-06T00:08:43.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:43.985+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.985682+0000) 2020-02-06T00:08:43.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:43.986+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.986697+0000) 2020-02-06T00:08:43.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:43.985+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.986494+0000) 2020-02-06T00:08:43.988 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:43.986+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.987155+0000) 2020-02-06T00:08:43.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:43.987+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.987625+0000) 2020-02-06T00:08:43.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:43.987+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.988009+0000) 2020-02-06T00:08:43.989 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:43.986+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.987579+0000) 2020-02-06T00:08:43.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:43.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.988023+0000) 2020-02-06T00:08:43.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:43.988+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.988519+0000) 2020-02-06T00:08:43.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:43.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.990519+0000) 2020-02-06T00:08:43.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:43.989+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.990776+0000) 2020-02-06T00:08:43.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:43.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.997147+0000) 2020-02-06T00:08:43.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:43.996+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:43.997148+0000) 2020-02-06T00:08:44.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:43.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:46.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:46.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:46.974729+0000) 2020-02-06T00:08:47.237 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:47.234+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:47.235968+0000) 2020-02-06T00:08:47.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:47.234+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:47.235963+0000) 2020-02-06T00:08:47.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:47.234+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:47.236167+0000) 2020-02-06T00:08:49.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:49.004+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.005487+0000) 2020-02-06T00:08:49.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:49.005+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.006301+0000) 2020-02-06T00:08:49.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:49.006+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.007563+0000) 2020-02-06T00:08:49.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:49.007+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.008383+0000) 2020-02-06T00:08:49.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:49.008+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.009909+0000) 2020-02-06T00:08:49.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:49.008+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.009966+0000) 2020-02-06T00:08:49.019 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:49.008+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.010366+0000) 2020-02-06T00:08:49.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:49.009+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.011106+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:49.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.011570+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:49.011+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.012826+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:49.008+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.009429+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:49.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.011793+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:49.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.014386+0000) 2020-02-06T00:08:49.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:49.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.014600+0000) 2020-02-06T00:08:49.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:49.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:49.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:49.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.234347+0000) 2020-02-06T00:08:49.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:49.582+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.583009+0000) 2020-02-06T00:08:49.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:49.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:49.618064+0000) 2020-02-06T00:08:50.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:50.464+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:50.465973+0000) 2020-02-06T00:08:50.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:50.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:50.726801+0000) 2020-02-06T00:08:52.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:52.117+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:52.118684+0000) 2020-02-06T00:08:52.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:52.845+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:52.846289+0000) 2020-02-06T00:08:53.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:53.117+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:53.117545+0000) 2020-02-06T00:08:53.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:53.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:53.997412+0000) 2020-02-06T00:08:53.999 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:53.996+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:53.997412+0000) 2020-02-06T00:08:54.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:54.030+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.031382+0000) 2020-02-06T00:08:54.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:54.031+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.032773+0000) 2020-02-06T00:08:54.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:54.032+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.033289+0000) 2020-02-06T00:08:54.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:54.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.033706+0000) 2020-02-06T00:08:54.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:54.032+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.033983+0000) 2020-02-06T00:08:54.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:54.033+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.034337+0000) 2020-02-06T00:08:54.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:54.034+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.035602+0000) 2020-02-06T00:08:54.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:54.033+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.034930+0000) 2020-02-06T00:08:54.038 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:54.035+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.036664+0000) 2020-02-06T00:08:54.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:54.035+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.036843+0000) 2020-02-06T00:08:54.039 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:54.035+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.037351+0000) 2020-02-06T00:08:54.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:54.035+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.036958+0000) 2020-02-06T00:08:54.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:54.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.040227+0000) 2020-02-06T00:08:54.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:54.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:54.040469+0000) 2020-02-06T00:08:54.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:54.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:56.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:56.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:56.974951+0000) 2020-02-06T00:08:57.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:57.235+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:57.236290+0000) 2020-02-06T00:08:57.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:57.235+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:57.236228+0000) 2020-02-06T00:08:57.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:57.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:57.236412+0000) 2020-02-06T00:08:59.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:08:59.054+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.055481+0000) 2020-02-06T00:08:59.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:59.053+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.055317+0000) 2020-02-06T00:08:59.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:59.055+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.056634+0000) 2020-02-06T00:08:59.059 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:59.057+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.058297+0000) 2020-02-06T00:08:59.060 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:08:59.057+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.058901+0000) 2020-02-06T00:08:59.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:08:59.058+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.059329+0000) 2020-02-06T00:08:59.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:08:59.058+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.059740+0000) 2020-02-06T00:08:59.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:08:59.058+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.060187+0000) 2020-02-06T00:08:59.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:08:59.059+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.060737+0000) 2020-02-06T00:08:59.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:08:59.060+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.061598+0000) 2020-02-06T00:08:59.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:08:59.060+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.061986+0000) 2020-02-06T00:08:59.063 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:08:59.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.062016+0000) 2020-02-06T00:08:59.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:59.066+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.067344+0000) 2020-02-06T00:08:59.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:59.067+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.068816+0000) 2020-02-06T00:08:59.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:08:59.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:08:59.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:08:59.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.234594+0000) 2020-02-06T00:08:59.584 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:08:59.582+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.583238+0000) 2020-02-06T00:08:59.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:08:59.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:08:59.618355+0000) 2020-02-06T00:09:00.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:00.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:00.466227+0000) 2020-02-06T00:09:00.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:00.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:00.727033+0000) 2020-02-06T00:09:02.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:02.126+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:02.127239+0000) 2020-02-06T00:09:02.847 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:02.845+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:02.846550+0000) 2020-02-06T00:09:03.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:03.117+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:03.117776+0000) 2020-02-06T00:09:03.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:03.996+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:03.997679+0000) 2020-02-06T00:09:03.999 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:03.996+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:03.997679+0000) 2020-02-06T00:09:04.006 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:09:04.011 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:09:04.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:04.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.079431+0000) 2020-02-06T00:09:04.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:04.080+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.081416+0000) 2020-02-06T00:09:04.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:04.081+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.082135+0000) 2020-02-06T00:09:04.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:04.081+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.082327+0000) 2020-02-06T00:09:04.083 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:04.081+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.082418+0000) 2020-02-06T00:09:04.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:04.081+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.082559+0000) 2020-02-06T00:09:04.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:04.081+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.082843+0000) 2020-02-06T00:09:04.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:04.081+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.083036+0000) 2020-02-06T00:09:04.085 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:04.081+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.083283+0000) 2020-02-06T00:09:04.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:04.081+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.083046+0000) 2020-02-06T00:09:04.085 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:04.082+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.083149+0000) 2020-02-06T00:09:04.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:04.082+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.084175+0000) 2020-02-06T00:09:04.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:04.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.089614+0000) 2020-02-06T00:09:04.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:04.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:04.089875+0000) 2020-02-06T00:09:04.100 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:04.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165481 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:06.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:06.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:06.975152+0000) 2020-02-06T00:09:07.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:07.235+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:07.236491+0000) 2020-02-06T00:09:07.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:07.235+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:07.236515+0000) 2020-02-06T00:09:07.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:07.235+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:07.236721+0000) 2020-02-06T00:09:09.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:09.102+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.103855+0000) 2020-02-06T00:09:09.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:09.104+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.105796+0000) 2020-02-06T00:09:09.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:09.105+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.106512+0000) 2020-02-06T00:09:09.108 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:09.105+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.106549+0000) 2020-02-06T00:09:09.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:09.105+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.107007+0000) 2020-02-06T00:09:09.109 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:09.106+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.107434+0000) 2020-02-06T00:09:09.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:09.106+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.108209+0000) 2020-02-06T00:09:09.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:09.107+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.108975+0000) 2020-02-06T00:09:09.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:09.108+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.110040+0000) 2020-02-06T00:09:09.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:09.108+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.110035+0000) 2020-02-06T00:09:09.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:09.109+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.110767+0000) 2020-02-06T00:09:09.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:09.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.110950+0000) 2020-02-06T00:09:09.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:09.110+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.111387+0000) 2020-02-06T00:09:09.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:09.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.115448+0000) 2020-02-06T00:09:09.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:09.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165520 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:09.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:09.233+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.234767+0000) 2020-02-06T00:09:09.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:09.583+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.583451+0000) 2020-02-06T00:09:09.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:09.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:09.618578+0000) 2020-02-06T00:09:10.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:10.465+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:10.466483+0000) 2020-02-06T00:09:10.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:10.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:10.727263+0000) 2020-02-06T00:09:12.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:12.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:12.135582+0000) 2020-02-06T00:09:12.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:12.846+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:12.846804+0000) 2020-02-06T00:09:13.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:13.117+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:13.118029+0000) 2020-02-06T00:09:13.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:13.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:13.997955+0000) 2020-02-06T00:09:13.999 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:13.997+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:13.997955+0000) 2020-02-06T00:09:14.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:14.129+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.129136+0000) 2020-02-06T00:09:14.130 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:14.129+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.129551+0000) 2020-02-06T00:09:14.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:14.127+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.128909+0000) 2020-02-06T00:09:14.131 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:14.129+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.130019+0000) 2020-02-06T00:09:14.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:14.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.130480+0000) 2020-02-06T00:09:14.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:14.129+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.130625+0000) 2020-02-06T00:09:14.132 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:14.130+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.130997+0000) 2020-02-06T00:09:14.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:14.131+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.132501+0000) 2020-02-06T00:09:14.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:14.133+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.133140+0000) 2020-02-06T00:09:14.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:14.132+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.133670+0000) 2020-02-06T00:09:14.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:14.132+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.134004+0000) 2020-02-06T00:09:14.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:14.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.134298+0000) 2020-02-06T00:09:14.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:14.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.134470+0000) 2020-02-06T00:09:14.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:14.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:14.140255+0000) 2020-02-06T00:09:14.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:14.145+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:16.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:16.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:16.975380+0000) 2020-02-06T00:09:17.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:17.236+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:17.236732+0000) 2020-02-06T00:09:17.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:17.236+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:17.236810+0000) 2020-02-06T00:09:17.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:17.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:17.236954+0000) 2020-02-06T00:09:19.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:19.152+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.153433+0000) 2020-02-06T00:09:19.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:19.152+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.154153+0000) 2020-02-06T00:09:19.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:19.153+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.154503+0000) 2020-02-06T00:09:19.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:19.153+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.154846+0000) 2020-02-06T00:09:19.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:19.154+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.155601+0000) 2020-02-06T00:09:19.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:19.154+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.155693+0000) 2020-02-06T00:09:19.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:19.154+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.156219+0000) 2020-02-06T00:09:19.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:19.155+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.156616+0000) 2020-02-06T00:09:19.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:19.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.156265+0000) 2020-02-06T00:09:19.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:19.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.157180+0000) 2020-02-06T00:09:19.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:19.156+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.157593+0000) 2020-02-06T00:09:19.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:19.155+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.157403+0000) 2020-02-06T00:09:19.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:19.156+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.158192+0000) 2020-02-06T00:09:19.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:19.161+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.162512+0000) 2020-02-06T00:09:19.172 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:19.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165602 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:19.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:19.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.235004+0000) 2020-02-06T00:09:19.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:19.582+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.583712+0000) 2020-02-06T00:09:19.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:19.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:19.618791+0000) 2020-02-06T00:09:20.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:20.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:20.466713+0000) 2020-02-06T00:09:20.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:20.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:20.727522+0000) 2020-02-06T00:09:22.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:22.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:22.143815+0000) 2020-02-06T00:09:22.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:22.846+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:22.847066+0000) 2020-02-06T00:09:23.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:23.117+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:23.118253+0000) 2020-02-06T00:09:23.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:23.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:23.998224+0000) 2020-02-06T00:09:24.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:23.997+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:23.998226+0000) 2020-02-06T00:09:24.180 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:24.177+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.178897+0000) 2020-02-06T00:09:24.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:24.178+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.179380+0000) 2020-02-06T00:09:24.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:24.178+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.179954+0000) 2020-02-06T00:09:24.181 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:24.178+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.180328+0000) 2020-02-06T00:09:24.182 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:24.179+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.180439+0000) 2020-02-06T00:09:24.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:24.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.180964+0000) 2020-02-06T00:09:24.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:24.180+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.181344+0000) 2020-02-06T00:09:24.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:24.180+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.181502+0000) 2020-02-06T00:09:24.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:24.181+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.182713+0000) 2020-02-06T00:09:24.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:24.181+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.183080+0000) 2020-02-06T00:09:24.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:24.182+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.183560+0000) 2020-02-06T00:09:24.184 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:24.182+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.183599+0000) 2020-02-06T00:09:24.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:24.186+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.187615+0000) 2020-02-06T00:09:24.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:24.186+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:24.187838+0000) 2020-02-06T00:09:24.199 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:24.196+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:26.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:26.974+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:26.975581+0000) 2020-02-06T00:09:27.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:27.236+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:27.236967+0000) 2020-02-06T00:09:27.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:27.236+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:27.237078+0000) 2020-02-06T00:09:27.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:27.236+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:27.237178+0000) 2020-02-06T00:09:29.204 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:29.201+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.202454+0000) 2020-02-06T00:09:29.205 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:29.202+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.203807+0000) 2020-02-06T00:09:29.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:29.202+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.203975+0000) 2020-02-06T00:09:29.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:29.203+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.205081+0000) 2020-02-06T00:09:29.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:29.204+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.205491+0000) 2020-02-06T00:09:29.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:29.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.206422+0000) 2020-02-06T00:09:29.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:29.206+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.207541+0000) 2020-02-06T00:09:29.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:29.205+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.207150+0000) 2020-02-06T00:09:29.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:29.205+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.207155+0000) 2020-02-06T00:09:29.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:29.206+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.207938+0000) 2020-02-06T00:09:29.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:29.207+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.208922+0000) 2020-02-06T00:09:29.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:29.207+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.208859+0000) 2020-02-06T00:09:29.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:29.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.211412+0000) 2020-02-06T00:09:29.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:29.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.212632+0000) 2020-02-06T00:09:29.224 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:29.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165684 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:29.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:29.234+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.235232+0000) 2020-02-06T00:09:29.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:29.583+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.583922+0000) 2020-02-06T00:09:29.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:29.617+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:29.618989+0000) 2020-02-06T00:09:30.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:30.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:30.466973+0000) 2020-02-06T00:09:30.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:30.726+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:30.727752+0000) 2020-02-06T00:09:32.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:32.154+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:32.155489+0000) 2020-02-06T00:09:32.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:32.846+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:32.847306+0000) 2020-02-06T00:09:33.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:33.117+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:33.118482+0000) 2020-02-06T00:09:34.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:33.997+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:33.998491+0000) 2020-02-06T00:09:34.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:33.997+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:33.998491+0000) 2020-02-06T00:09:34.161 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:09:34.168 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:09:34.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:34.225+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.226293+0000) 2020-02-06T00:09:34.228 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:34.225+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.226328+0000) 2020-02-06T00:09:34.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:34.226+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.227525+0000) 2020-02-06T00:09:34.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:34.226+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.228271+0000) 2020-02-06T00:09:34.230 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:34.228+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.229558+0000) 2020-02-06T00:09:34.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:34.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.229914+0000) 2020-02-06T00:09:34.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:34.228+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.230331+0000) 2020-02-06T00:09:34.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:34.229+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.230298+0000) 2020-02-06T00:09:34.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:34.230+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.231184+0000) 2020-02-06T00:09:34.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:34.230+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.231498+0000) 2020-02-06T00:09:34.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:34.230+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.231505+0000) 2020-02-06T00:09:34.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:34.230+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.231745+0000) 2020-02-06T00:09:34.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:34.231+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.232683+0000) 2020-02-06T00:09:34.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:34.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:34.234500+0000) 2020-02-06T00:09:34.246 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:34.243+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:36.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:36.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:36.975817+0000) 2020-02-06T00:09:37.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:37.236+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:37.237221+0000) 2020-02-06T00:09:37.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:37.236+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:37.237324+0000) 2020-02-06T00:09:37.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:37.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:37.237479+0000) 2020-02-06T00:09:39.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:39.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.235497+0000) 2020-02-06T00:09:39.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:39.250+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.251482+0000) 2020-02-06T00:09:39.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:39.251+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.252260+0000) 2020-02-06T00:09:39.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:39.252+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.253277+0000) 2020-02-06T00:09:39.255 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:39.252+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.253870+0000) 2020-02-06T00:09:39.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:39.254+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.255311+0000) 2020-02-06T00:09:39.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:39.255+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.255705+0000) 2020-02-06T00:09:39.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:39.255+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.256032+0000) 2020-02-06T00:09:39.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:39.253+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.254583+0000) 2020-02-06T00:09:39.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:39.254+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.255576+0000) 2020-02-06T00:09:39.258 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:39.255+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.256653+0000) 2020-02-06T00:09:39.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:39.257+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.257423+0000) 2020-02-06T00:09:39.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:39.256+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.257344+0000) 2020-02-06T00:09:39.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:39.256+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.257614+0000) 2020-02-06T00:09:39.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:39.262+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.263420+0000) 2020-02-06T00:09:39.270 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:39.269+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165764 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:39.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:39.583+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.584133+0000) 2020-02-06T00:09:39.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:39.618+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:39.619265+0000) 2020-02-06T00:09:40.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:40.466+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:40.467212+0000) 2020-02-06T00:09:40.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:40.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:40.727983+0000) 2020-02-06T00:09:42.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:42.162+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:42.163714+0000) 2020-02-06T00:09:42.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:42.847+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:42.847567+0000) 2020-02-06T00:09:43.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:43.118+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:43.118702+0000) 2020-02-06T00:09:44.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:43.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:43.998773+0000) 2020-02-06T00:09:44.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:43.998+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:43.998773+0000) 2020-02-06T00:09:44.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:44.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.274752+0000) 2020-02-06T00:09:44.277 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:44.275+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.276140+0000) 2020-02-06T00:09:44.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:44.276+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.277253+0000) 2020-02-06T00:09:44.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:44.276+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.277852+0000) 2020-02-06T00:09:44.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:44.278+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.278559+0000) 2020-02-06T00:09:44.280 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:44.277+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.278584+0000) 2020-02-06T00:09:44.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:44.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.279018+0000) 2020-02-06T00:09:44.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:44.278+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.279138+0000) 2020-02-06T00:09:44.281 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:44.278+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.279319+0000) 2020-02-06T00:09:44.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:44.279+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.279606+0000) 2020-02-06T00:09:44.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:44.279+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.280010+0000) 2020-02-06T00:09:44.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:44.280+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.280639+0000) 2020-02-06T00:09:44.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:44.280+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.281869+0000) 2020-02-06T00:09:44.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:44.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:44.288570+0000) 2020-02-06T00:09:44.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:44.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165805 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:46.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:46.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:46.976060+0000) 2020-02-06T00:09:47.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:47.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:47.237476+0000) 2020-02-06T00:09:47.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:47.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:47.237738+0000) 2020-02-06T00:09:47.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:47.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:47.237651+0000) 2020-02-06T00:09:49.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:49.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.235761+0000) 2020-02-06T00:09:49.301 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:49.299+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.300289+0000) 2020-02-06T00:09:49.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:49.299+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.299612+0000) 2020-02-06T00:09:49.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:49.301+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.302199+0000) 2020-02-06T00:09:49.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:49.301+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.302550+0000) 2020-02-06T00:09:49.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:49.302+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.302989+0000) 2020-02-06T00:09:49.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:49.303+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.303601+0000) 2020-02-06T00:09:49.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:49.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.303844+0000) 2020-02-06T00:09:49.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:49.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.305212+0000) 2020-02-06T00:09:49.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:49.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.305875+0000) 2020-02-06T00:09:49.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:49.305+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.306210+0000) 2020-02-06T00:09:49.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:49.305+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.306263+0000) 2020-02-06T00:09:49.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:49.305+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.306416+0000) 2020-02-06T00:09:49.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:49.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.310292+0000) 2020-02-06T00:09:49.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:49.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.310607+0000) 2020-02-06T00:09:49.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:49.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:49.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:49.583+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.584438+0000) 2020-02-06T00:09:49.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:49.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:49.619473+0000) 2020-02-06T00:09:50.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:50.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:50.467508+0000) 2020-02-06T00:09:50.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:50.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:50.728289+0000) 2020-02-06T00:09:52.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:52.171+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:52.172133+0000) 2020-02-06T00:09:52.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:52.847+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:52.847825+0000) 2020-02-06T00:09:53.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:53.118+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:53.118946+0000) 2020-02-06T00:09:54.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:53.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:53.999047+0000) 2020-02-06T00:09:54.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:53.998+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:53.999047+0000) 2020-02-06T00:09:54.323 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:54.321+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.322464+0000) 2020-02-06T00:09:54.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:54.323+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.324267+0000) 2020-02-06T00:09:54.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:54.324+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.324677+0000) 2020-02-06T00:09:54.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:54.324+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.325303+0000) 2020-02-06T00:09:54.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:54.324+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.325122+0000) 2020-02-06T00:09:54.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:54.324+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.325903+0000) 2020-02-06T00:09:54.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:54.324+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.325995+0000) 2020-02-06T00:09:54.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:54.325+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.326308+0000) 2020-02-06T00:09:54.328 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:54.325+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.326586+0000) 2020-02-06T00:09:54.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:54.326+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.327118+0000) 2020-02-06T00:09:54.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:54.327+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.328173+0000) 2020-02-06T00:09:54.329 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:54.328+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.328592+0000) 2020-02-06T00:09:54.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:54.327+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.328690+0000) 2020-02-06T00:09:54.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:54.330+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:54.331368+0000) 2020-02-06T00:09:54.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:54.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:56.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:56.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:56.976345+0000) 2020-02-06T00:09:57.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:57.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:57.237690+0000) 2020-02-06T00:09:57.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:57.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:57.237894+0000) 2020-02-06T00:09:57.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:57.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:57.237967+0000) 2020-02-06T00:09:59.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:59.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.236018+0000) 2020-02-06T00:09:59.348 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:09:59.345+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.346545+0000) 2020-02-06T00:09:59.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:09:59.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.347619+0000) 2020-02-06T00:09:59.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:09:59.347+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.348804+0000) 2020-02-06T00:09:59.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:09:59.348+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.349397+0000) 2020-02-06T00:09:59.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:59.349+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.350147+0000) 2020-02-06T00:09:59.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:09:59.349+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.350335+0000) 2020-02-06T00:09:59.352 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:09:59.349+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.350603+0000) 2020-02-06T00:09:59.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:59.349+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.350896+0000) 2020-02-06T00:09:59.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:59.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.351121+0000) 2020-02-06T00:09:59.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:09:59.351+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.352256+0000) 2020-02-06T00:09:59.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:09:59.350+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.351392+0000) 2020-02-06T00:09:59.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:09:59.352+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.352556+0000) 2020-02-06T00:09:59.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:09:59.352+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.352933+0000) 2020-02-06T00:09:59.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:59.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.357706+0000) 2020-02-06T00:09:59.367 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:09:59.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165924 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:09:59.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:09:59.584+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.584660+0000) 2020-02-06T00:09:59.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:09:59.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:09:59.619676+0000) 2020-02-06T00:10:00.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:00.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:00.467742+0000) 2020-02-06T00:10:00.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:00.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:00.728516+0000) 2020-02-06T00:10:02.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:02.180+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:02.180258+0000) 2020-02-06T00:10:02.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:02.847+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:02.848087+0000) 2020-02-06T00:10:03.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:03.118+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:03.119183+0000) 2020-02-06T00:10:04.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:03.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:03.999353+0000) 2020-02-06T00:10:04.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:03.999+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:03.999353+0000) 2020-02-06T00:10:04.284 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:10:04.291 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:10:04.371 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:04.369+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.370342+0000) 2020-02-06T00:10:04.372 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:04.369+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.370868+0000) 2020-02-06T00:10:04.372 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:04.370+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.371245+0000) 2020-02-06T00:10:04.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:04.370+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.371512+0000) 2020-02-06T00:10:04.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:04.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.372403+0000) 2020-02-06T00:10:04.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:04.371+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.371704+0000) 2020-02-06T00:10:04.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:04.372+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.372806+0000) 2020-02-06T00:10:04.374 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:04.372+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.372750+0000) 2020-02-06T00:10:04.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:04.372+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.373106+0000) 2020-02-06T00:10:04.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:04.372+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.373238+0000) 2020-02-06T00:10:04.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:04.372+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.373764+0000) 2020-02-06T00:10:04.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:04.373+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.374410+0000) 2020-02-06T00:10:04.376 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:04.373+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.374211+0000) 2020-02-06T00:10:04.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:04.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:04.380513+0000) 2020-02-06T00:10:04.390 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:04.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 165965 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:06.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:06.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:06.976618+0000) 2020-02-06T00:10:07.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:07.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:07.237944+0000) 2020-02-06T00:10:07.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:07.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:07.238175+0000) 2020-02-06T00:10:07.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:07.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:07.238222+0000) 2020-02-06T00:10:09.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:09.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.236262+0000) 2020-02-06T00:10:09.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:09.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.395050+0000) 2020-02-06T00:10:09.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:09.395+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.395816+0000) 2020-02-06T00:10:09.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:09.396+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.396268+0000) 2020-02-06T00:10:09.398 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:09.396+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.396514+0000) 2020-02-06T00:10:09.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:09.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.397341+0000) 2020-02-06T00:10:09.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:09.396+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.397183+0000) 2020-02-06T00:10:09.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:09.397+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.397930+0000) 2020-02-06T00:10:09.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:09.398+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.398271+0000) 2020-02-06T00:10:09.400 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:09.398+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.398851+0000) 2020-02-06T00:10:09.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:09.399+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.399286+0000) 2020-02-06T00:10:09.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:09.399+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.399726+0000) 2020-02-06T00:10:09.401 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:09.399+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.399535+0000) 2020-02-06T00:10:09.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:09.399+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.400013+0000) 2020-02-06T00:10:09.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:09.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.403502+0000) 2020-02-06T00:10:09.415 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:09.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166004 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:09.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:09.584+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.584870+0000) 2020-02-06T00:10:09.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:09.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:09.619883+0000) 2020-02-06T00:10:10.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:10.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:10.468002+0000) 2020-02-06T00:10:10.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:10.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:10.728760+0000) 2020-02-06T00:10:12.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:12.187+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:12.188144+0000) 2020-02-06T00:10:12.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:12.846+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:12.848340+0000) 2020-02-06T00:10:13.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:13.118+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:13.119427+0000) 2020-02-06T00:10:14.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:13.998+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:13.999623+0000) 2020-02-06T00:10:14.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:13.998+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:13.999623+0000) 2020-02-06T00:10:14.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:14.419+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.420270+0000) 2020-02-06T00:10:14.422 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:14.420+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.421337+0000) 2020-02-06T00:10:14.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:14.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.421279+0000) 2020-02-06T00:10:14.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:14.420+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.421818+0000) 2020-02-06T00:10:14.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:14.421+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.422487+0000) 2020-02-06T00:10:14.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:14.421+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.422907+0000) 2020-02-06T00:10:14.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:14.421+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.423210+0000) 2020-02-06T00:10:14.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:14.422+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.424172+0000) 2020-02-06T00:10:14.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:14.423+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.424739+0000) 2020-02-06T00:10:14.427 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:14.424+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.425687+0000) 2020-02-06T00:10:14.427 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:14.424+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.425934+0000) 2020-02-06T00:10:14.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:14.424+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.425621+0000) 2020-02-06T00:10:14.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:14.424+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.425857+0000) 2020-02-06T00:10:14.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:14.426+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:14.427381+0000) 2020-02-06T00:10:14.439 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:14.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166046 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:16.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:16.975+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:16.976879+0000) 2020-02-06T00:10:17.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:17.236+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:17.238127+0000) 2020-02-06T00:10:17.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:17.236+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:17.238367+0000) 2020-02-06T00:10:17.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:17.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:17.238481+0000) 2020-02-06T00:10:19.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:19.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.236523+0000) 2020-02-06T00:10:19.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:19.442+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.443880+0000) 2020-02-06T00:10:19.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:19.446+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.447529+0000) 2020-02-06T00:10:19.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:19.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.447783+0000) 2020-02-06T00:10:19.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:19.446+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.447467+0000) 2020-02-06T00:10:19.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:19.446+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.447972+0000) 2020-02-06T00:10:19.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:19.448+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.449238+0000) 2020-02-06T00:10:19.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:19.448+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.449468+0000) 2020-02-06T00:10:19.451 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:19.447+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.448966+0000) 2020-02-06T00:10:19.452 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:19.448+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.449910+0000) 2020-02-06T00:10:19.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:19.449+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.450603+0000) 2020-02-06T00:10:19.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:19.448+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.450237+0000) 2020-02-06T00:10:19.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:19.449+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.450857+0000) 2020-02-06T00:10:19.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:19.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.451979+0000) 2020-02-06T00:10:19.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:19.451+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.452874+0000) 2020-02-06T00:10:19.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:19.462+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166083 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:19.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:19.584+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.585120+0000) 2020-02-06T00:10:19.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:19.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:19.620137+0000) 2020-02-06T00:10:20.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:20.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:20.468238+0000) 2020-02-06T00:10:20.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:20.727+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:20.728950+0000) 2020-02-06T00:10:22.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:22.195+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:22.196558+0000) 2020-02-06T00:10:22.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:22.847+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:22.848630+0000) 2020-02-06T00:10:23.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:23.118+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:23.119658+0000) 2020-02-06T00:10:24.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:23.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:23.999892+0000) 2020-02-06T00:10:24.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:23.999+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:23.999893+0000) 2020-02-06T00:10:24.470 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:24.467+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.468848+0000) 2020-02-06T00:10:24.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:24.468+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.469684+0000) 2020-02-06T00:10:24.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:24.469+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.470834+0000) 2020-02-06T00:10:24.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:24.469+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.470883+0000) 2020-02-06T00:10:24.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:24.469+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.470917+0000) 2020-02-06T00:10:24.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:24.470+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.472019+0000) 2020-02-06T00:10:24.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:24.471+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.472592+0000) 2020-02-06T00:10:24.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:24.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.472577+0000) 2020-02-06T00:10:24.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:24.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.473111+0000) 2020-02-06T00:10:24.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:24.472+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.473503+0000) 2020-02-06T00:10:24.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:24.472+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.473365+0000) 2020-02-06T00:10:24.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:24.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.474285+0000) 2020-02-06T00:10:24.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:24.472+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.473839+0000) 2020-02-06T00:10:24.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:24.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:24.480861+0000) 2020-02-06T00:10:24.488 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:24.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:26.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:26.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:26.977120+0000) 2020-02-06T00:10:27.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:27.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:27.238389+0000) 2020-02-06T00:10:27.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:27.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:27.238745+0000) 2020-02-06T00:10:27.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:27.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:27.238683+0000) 2020-02-06T00:10:29.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:29.235+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.236780+0000) 2020-02-06T00:10:29.492 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:29.490+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.491560+0000) 2020-02-06T00:10:29.493 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:29.490+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.492008+0000) 2020-02-06T00:10:29.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:29.491+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.492964+0000) 2020-02-06T00:10:29.495 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:29.492+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.493545+0000) 2020-02-06T00:10:29.496 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:29.493+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.494876+0000) 2020-02-06T00:10:29.496 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:29.493+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.495234+0000) 2020-02-06T00:10:29.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:29.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.495626+0000) 2020-02-06T00:10:29.497 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:29.495+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.496174+0000) 2020-02-06T00:10:29.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:29.495+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.497026+0000) 2020-02-06T00:10:29.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:29.496+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.497729+0000) 2020-02-06T00:10:29.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:29.496+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.497973+0000) 2020-02-06T00:10:29.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:29.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.498321+0000) 2020-02-06T00:10:29.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:29.497+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.498835+0000) 2020-02-06T00:10:29.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:29.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.502667+0000) 2020-02-06T00:10:29.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:29.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:29.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:29.584+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.585352+0000) 2020-02-06T00:10:29.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:29.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:29.620344+0000) 2020-02-06T00:10:30.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:30.467+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:30.468520+0000) 2020-02-06T00:10:30.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:30.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:30.729165+0000) 2020-02-06T00:10:32.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:32.204+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:32.205404+0000) 2020-02-06T00:10:32.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:32.848+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:32.848889+0000) 2020-02-06T00:10:33.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:33.119+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:33.119919+0000) 2020-02-06T00:10:34.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:33.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.000148+0000) 2020-02-06T00:10:34.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:33.999+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.000148+0000) 2020-02-06T00:10:34.464 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:10:34.470 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:10:34.517 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:34.515+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.516440+0000) 2020-02-06T00:10:34.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:34.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.516915+0000) 2020-02-06T00:10:34.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:34.516+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.517707+0000) 2020-02-06T00:10:34.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:34.516+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.517654+0000) 2020-02-06T00:10:34.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:34.517+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.518115+0000) 2020-02-06T00:10:34.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:34.517+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.518463+0000) 2020-02-06T00:10:34.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:34.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.518483+0000) 2020-02-06T00:10:34.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:34.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.519000+0000) 2020-02-06T00:10:34.521 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:34.517+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.519023+0000) 2020-02-06T00:10:34.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:34.518+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.519832+0000) 2020-02-06T00:10:34.521 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:34.518+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.520350+0000) 2020-02-06T00:10:34.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:34.521+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.522778+0000) 2020-02-06T00:10:34.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:34.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.527904+0000) 2020-02-06T00:10:34.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:34.527+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:34.528155+0000) 2020-02-06T00:10:34.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:34.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166206 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:36.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:36.976+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:36.977391+0000) 2020-02-06T00:10:37.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:37.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:37.238663+0000) 2020-02-06T00:10:37.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:37.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:37.238990+0000) 2020-02-06T00:10:37.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:37.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:37.238995+0000) 2020-02-06T00:10:39.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:39.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.237016+0000) 2020-02-06T00:10:39.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:39.537+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.539008+0000) 2020-02-06T00:10:39.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:39.540+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.541150+0000) 2020-02-06T00:10:39.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:39.540+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.541450+0000) 2020-02-06T00:10:39.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:39.540+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.541638+0000) 2020-02-06T00:10:39.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:39.541+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.542357+0000) 2020-02-06T00:10:39.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:39.541+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.542800+0000) 2020-02-06T00:10:39.544 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:39.541+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.542774+0000) 2020-02-06T00:10:39.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:39.542+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.543910+0000) 2020-02-06T00:10:39.545 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:39.542+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.544228+0000) 2020-02-06T00:10:39.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:39.543+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.544721+0000) 2020-02-06T00:10:39.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:39.543+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.545202+0000) 2020-02-06T00:10:39.546 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:39.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.545565+0000) 2020-02-06T00:10:39.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:39.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.546564+0000) 2020-02-06T00:10:39.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:39.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.548231+0000) 2020-02-06T00:10:39.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:39.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:39.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:39.584+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.585573+0000) 2020-02-06T00:10:39.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:39.619+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:39.620575+0000) 2020-02-06T00:10:40.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:40.468+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:40.468792+0000) 2020-02-06T00:10:40.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:40.728+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:40.729397+0000) 2020-02-06T00:10:42.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:42.213+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:42.213703+0000) 2020-02-06T00:10:42.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:42.848+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:42.849154+0000) 2020-02-06T00:10:43.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:43.119+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:43.120137+0000) 2020-02-06T00:10:44.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:43.999+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.000414+0000) 2020-02-06T00:10:44.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:43.999+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.000415+0000) 2020-02-06T00:10:44.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:44.563+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.564202+0000) 2020-02-06T00:10:44.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:44.563+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.564839+0000) 2020-02-06T00:10:44.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:44.564+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.565117+0000) 2020-02-06T00:10:44.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:44.564+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.565635+0000) 2020-02-06T00:10:44.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:44.564+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.565632+0000) 2020-02-06T00:10:44.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:44.564+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.566086+0000) 2020-02-06T00:10:44.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:44.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.567197+0000) 2020-02-06T00:10:44.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:44.565+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.567038+0000) 2020-02-06T00:10:44.569 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:44.567+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.568533+0000) 2020-02-06T00:10:44.570 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:44.567+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.568979+0000) 2020-02-06T00:10:44.570 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:44.568+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.569538+0000) 2020-02-06T00:10:44.571 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:44.568+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.569570+0000) 2020-02-06T00:10:44.571 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:44.568+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.569898+0000) 2020-02-06T00:10:44.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:44.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:44.572826+0000) 2020-02-06T00:10:44.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:44.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:46.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:46.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:46.977619+0000) 2020-02-06T00:10:47.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:47.237+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:47.238878+0000) 2020-02-06T00:10:47.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:47.237+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:47.239235+0000) 2020-02-06T00:10:47.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:47.237+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:47.239243+0000) 2020-02-06T00:10:49.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:49.236+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.237279+0000) 2020-02-06T00:10:49.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:49.586+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.587893+0000) 2020-02-06T00:10:49.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:49.587+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.588796+0000) 2020-02-06T00:10:49.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:49.587+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.589372+0000) 2020-02-06T00:10:49.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:49.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.589890+0000) 2020-02-06T00:10:49.591 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:49.589+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.590207+0000) 2020-02-06T00:10:49.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:49.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.590381+0000) 2020-02-06T00:10:49.592 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:49.589+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.590702+0000) 2020-02-06T00:10:49.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:49.589+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.590797+0000) 2020-02-06T00:10:49.593 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:49.590+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.591847+0000) 2020-02-06T00:10:49.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:49.590+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.592209+0000) 2020-02-06T00:10:49.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:49.592+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.593747+0000) 2020-02-06T00:10:49.595 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:49.592+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.593660+0000) 2020-02-06T00:10:49.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:49.592+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.593657+0000) 2020-02-06T00:10:49.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:49.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.599087+0000) 2020-02-06T00:10:49.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:49.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166323 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:49.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:49.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:49.620817+0000) 2020-02-06T00:10:50.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:50.468+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:50.469024+0000) 2020-02-06T00:10:50.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:50.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:50.729667+0000) 2020-02-06T00:10:52.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:52.222+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:52.223063+0000) 2020-02-06T00:10:52.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:52.848+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:52.849414+0000) 2020-02-06T00:10:53.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:53.120+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:53.120396+0000) 2020-02-06T00:10:54.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:54.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.000689+0000) 2020-02-06T00:10:54.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:54.000+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.000689+0000) 2020-02-06T00:10:54.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:54.610+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.611299+0000) 2020-02-06T00:10:54.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:54.611+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.612274+0000) 2020-02-06T00:10:54.613 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:54.610+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.612114+0000) 2020-02-06T00:10:54.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:54.610+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.612340+0000) 2020-02-06T00:10:54.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:54.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.613058+0000) 2020-02-06T00:10:54.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:54.612+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.613427+0000) 2020-02-06T00:10:54.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:54.612+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.613563+0000) 2020-02-06T00:10:54.616 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:54.613+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.614519+0000) 2020-02-06T00:10:54.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:54.614+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.615560+0000) 2020-02-06T00:10:54.617 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:54.614+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.615722+0000) 2020-02-06T00:10:54.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:54.614+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.616093+0000) 2020-02-06T00:10:54.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:54.615+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.616600+0000) 2020-02-06T00:10:54.618 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:54.615+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.616453+0000) 2020-02-06T00:10:54.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:54.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:54.622326+0000) 2020-02-06T00:10:54.631 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:54.628+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:10:56.337 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:56.336+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:56.336254+0000) 2020-02-06T00:10:56.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:56.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:56.977848+0000) 2020-02-06T00:10:57.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:57.238+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:57.239124+0000) 2020-02-06T00:10:57.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:57.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:57.239489+0000) 2020-02-06T00:10:57.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:57.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:57.239537+0000) 2020-02-06T00:10:59.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:59.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.237541+0000) 2020-02-06T00:10:59.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:59.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.621042+0000) 2020-02-06T00:10:59.636 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:10:59.633+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.634527+0000) 2020-02-06T00:10:59.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:10:59.635+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.636393+0000) 2020-02-06T00:10:59.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:59.635+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.637036+0000) 2020-02-06T00:10:59.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:10:59.636+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.637367+0000) 2020-02-06T00:10:59.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:10:59.637+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.637761+0000) 2020-02-06T00:10:59.640 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:10:59.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.638240+0000) 2020-02-06T00:10:59.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:59.637+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.638907+0000) 2020-02-06T00:10:59.641 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:10:59.639+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.639858+0000) 2020-02-06T00:10:59.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:10:59.638+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.639997+0000) 2020-02-06T00:10:59.641 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:10:59.640+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.640650+0000) 2020-02-06T00:10:59.642 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:10:59.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.640922+0000) 2020-02-06T00:10:59.642 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:10:59.640+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.641364+0000) 2020-02-06T00:10:59.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:10:59.641+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.641840+0000) 2020-02-06T00:10:59.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:10:59.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:10:59.644293+0000) 2020-02-06T00:10:59.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:10:59.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:00.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:00.468+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:00.469248+0000) 2020-02-06T00:11:00.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:00.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:00.729899+0000) 2020-02-06T00:11:02.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:02.232+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:02.232920+0000) 2020-02-06T00:11:02.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:02.849+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:02.849672+0000) 2020-02-06T00:11:03.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:03.120+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:03.120630+0000) 2020-02-06T00:11:04.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:04.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.000951+0000) 2020-02-06T00:11:04.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:04.000+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.000951+0000) 2020-02-06T00:11:04.615 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:11:04.622 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:11:04.660 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:04.658+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.658868+0000) 2020-02-06T00:11:04.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:04.659+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.659543+0000) 2020-02-06T00:11:04.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:04.659+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.660046+0000) 2020-02-06T00:11:04.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:04.660+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.660981+0000) 2020-02-06T00:11:04.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:04.660+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.660965+0000) 2020-02-06T00:11:04.663 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:04.660+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.661160+0000) 2020-02-06T00:11:04.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:04.661+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.661312+0000) 2020-02-06T00:11:04.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:04.661+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.661677+0000) 2020-02-06T00:11:04.664 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:04.662+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.662882+0000) 2020-02-06T00:11:04.664 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:04.662+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.663356+0000) 2020-02-06T00:11:04.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:04.664+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.664241+0000) 2020-02-06T00:11:04.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:04.664+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.664653+0000) 2020-02-06T00:11:04.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:04.664+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.664926+0000) 2020-02-06T00:11:04.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:04.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:04.669384+0000) 2020-02-06T00:11:04.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:04.678+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:06.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:06.335+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:06.336487+0000) 2020-02-06T00:11:06.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:06.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:06.978105+0000) 2020-02-06T00:11:07.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:07.238+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:07.239400+0000) 2020-02-06T00:11:07.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:07.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:07.239692+0000) 2020-02-06T00:11:07.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:07.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:07.239773+0000) 2020-02-06T00:11:09.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:09.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.237826+0000) 2020-02-06T00:11:09.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:09.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.621343+0000) 2020-02-06T00:11:09.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:09.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.685129+0000) 2020-02-06T00:11:09.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:09.685+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.686392+0000) 2020-02-06T00:11:09.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:09.686+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.687180+0000) 2020-02-06T00:11:09.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:09.686+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.686641+0000) 2020-02-06T00:11:09.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:09.686+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.687574+0000) 2020-02-06T00:11:09.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:09.686+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.688016+0000) 2020-02-06T00:11:09.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:09.687+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.688763+0000) 2020-02-06T00:11:09.691 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:09.689+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.690132+0000) 2020-02-06T00:11:09.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:09.689+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.690354+0000) 2020-02-06T00:11:09.692 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:09.690+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.690690+0000) 2020-02-06T00:11:09.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:09.690+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.690998+0000) 2020-02-06T00:11:09.693 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:09.690+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.691323+0000) 2020-02-06T00:11:09.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:09.690+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.691070+0000) 2020-02-06T00:11:09.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:09.693+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:09.694641+0000) 2020-02-06T00:11:09.705 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:09.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166487 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:10.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:10.469+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:10.469483+0000) 2020-02-06T00:11:10.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:10.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:10.730137+0000) 2020-02-06T00:11:12.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:12.243+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:12.244486+0000) 2020-02-06T00:11:12.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:12.849+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:12.849938+0000) 2020-02-06T00:11:13.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:13.119+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:13.120880+0000) 2020-02-06T00:11:14.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:14.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.001230+0000) 2020-02-06T00:11:14.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:14.000+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.001230+0000) 2020-02-06T00:11:14.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:14.708+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.709818+0000) 2020-02-06T00:11:14.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:14.710+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.711230+0000) 2020-02-06T00:11:14.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:14.710+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.711680+0000) 2020-02-06T00:11:14.714 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:14.712+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.712942+0000) 2020-02-06T00:11:14.715 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:14.712+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.714041+0000) 2020-02-06T00:11:14.715 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:14.714+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.714657+0000) 2020-02-06T00:11:14.716 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:14.713+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.714224+0000) 2020-02-06T00:11:14.717 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:14.713+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.715030+0000) 2020-02-06T00:11:14.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:14.714+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.715420+0000) 2020-02-06T00:11:14.717 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:14.714+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.715083+0000) 2020-02-06T00:11:14.717 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:14.715+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.715683+0000) 2020-02-06T00:11:14.718 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:14.715+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.716252+0000) 2020-02-06T00:11:14.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:14.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.718396+0000) 2020-02-06T00:11:14.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:14.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:14.718625+0000) 2020-02-06T00:11:14.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:14.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:16.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:16.335+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:16.336757+0000) 2020-02-06T00:11:16.979 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:16.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:16.978300+0000) 2020-02-06T00:11:17.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:17.239+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:17.239690+0000) 2020-02-06T00:11:17.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:17.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:17.239883+0000) 2020-02-06T00:11:17.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:17.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:17.240012+0000) 2020-02-06T00:11:19.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:19.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.238086+0000) 2020-02-06T00:11:19.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:19.620+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.621573+0000) 2020-02-06T00:11:19.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:19.734+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.734817+0000) 2020-02-06T00:11:19.737 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:19.735+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.735834+0000) 2020-02-06T00:11:19.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:19.735+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.736465+0000) 2020-02-06T00:11:19.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:19.735+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.736697+0000) 2020-02-06T00:11:19.739 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:19.736+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.737039+0000) 2020-02-06T00:11:19.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:19.737+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.737554+0000) 2020-02-06T00:11:19.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:19.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.737827+0000) 2020-02-06T00:11:19.741 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:19.737+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.738122+0000) 2020-02-06T00:11:19.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:19.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.738439+0000) 2020-02-06T00:11:19.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:19.737+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.738952+0000) 2020-02-06T00:11:19.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:19.738+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.739346+0000) 2020-02-06T00:11:19.742 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:19.738+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.739555+0000) 2020-02-06T00:11:19.742 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:19.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.739499+0000) 2020-02-06T00:11:19.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:19.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:19.745482+0000) 2020-02-06T00:11:19.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:19.751+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166569 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:20.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:20.469+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:20.469708+0000) 2020-02-06T00:11:20.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:20.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:20.730369+0000) 2020-02-06T00:11:22.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:22.251+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:22.252653+0000) 2020-02-06T00:11:22.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:22.849+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:22.850183+0000) 2020-02-06T00:11:23.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:23.120+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:23.121132+0000) 2020-02-06T00:11:24.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:24.000+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.001533+0000) 2020-02-06T00:11:24.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:24.000+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.001533+0000) 2020-02-06T00:11:24.760 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:24.759+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.759544+0000) 2020-02-06T00:11:24.761 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:24.758+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.759976+0000) 2020-02-06T00:11:24.762 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:24.760+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.761023+0000) 2020-02-06T00:11:24.762 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:24.760+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.760586+0000) 2020-02-06T00:11:24.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:24.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.761256+0000) 2020-02-06T00:11:24.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:24.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.761707+0000) 2020-02-06T00:11:24.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:24.760+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.761810+0000) 2020-02-06T00:11:24.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:24.761+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.762363+0000) 2020-02-06T00:11:24.764 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:24.761+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.762857+0000) 2020-02-06T00:11:24.764 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:24.762+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.763103+0000) 2020-02-06T00:11:24.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:24.763+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.763613+0000) 2020-02-06T00:11:24.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:24.762+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.763949+0000) 2020-02-06T00:11:24.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:24.763+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.764134+0000) 2020-02-06T00:11:24.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:24.766+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:24.767838+0000) 2020-02-06T00:11:24.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:24.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166613 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:26.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:26.335+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:26.337008+0000) 2020-02-06T00:11:26.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:26.977+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:26.978542+0000) 2020-02-06T00:11:27.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:27.239+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:27.239944+0000) 2020-02-06T00:11:27.242 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:27.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:27.240117+0000) 2020-02-06T00:11:27.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:27.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:27.240342+0000) 2020-02-06T00:11:29.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:29.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.238333+0000) 2020-02-06T00:11:29.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:29.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.621810+0000) 2020-02-06T00:11:29.782 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:29.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.781297+0000) 2020-02-06T00:11:29.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:29.782+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.782513+0000) 2020-02-06T00:11:29.784 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:29.781+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.782744+0000) 2020-02-06T00:11:29.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:29.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.783855+0000) 2020-02-06T00:11:29.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:29.783+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.784522+0000) 2020-02-06T00:11:29.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:29.783+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.784951+0000) 2020-02-06T00:11:29.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:29.783+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.784410+0000) 2020-02-06T00:11:29.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:29.785+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.785393+0000) 2020-02-06T00:11:29.787 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:29.784+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.785432+0000) 2020-02-06T00:11:29.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:29.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.785843+0000) 2020-02-06T00:11:29.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:29.785+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.786774+0000) 2020-02-06T00:11:29.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:29.786+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.787038+0000) 2020-02-06T00:11:29.789 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:29.787+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.788036+0000) 2020-02-06T00:11:29.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:29.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:29.791911+0000) 2020-02-06T00:11:29.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:29.800+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:30.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:30.469+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:30.469947+0000) 2020-02-06T00:11:30.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:30.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:30.730586+0000) 2020-02-06T00:11:32.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:32.259+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:32.259905+0000) 2020-02-06T00:11:32.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:32.850+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:32.850462+0000) 2020-02-06T00:11:33.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:33.120+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:33.121397+0000) 2020-02-06T00:11:34.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:34.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.001805+0000) 2020-02-06T00:11:34.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:34.001+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.001805+0000) 2020-02-06T00:11:34.740 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:11:34.747 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:11:34.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:34.804+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.805630+0000) 2020-02-06T00:11:34.807 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:34.805+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.806296+0000) 2020-02-06T00:11:34.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:34.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.806720+0000) 2020-02-06T00:11:34.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:34.806+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.806921+0000) 2020-02-06T00:11:34.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:34.807+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.807420+0000) 2020-02-06T00:11:34.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:34.806+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.807237+0000) 2020-02-06T00:11:34.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:34.806+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.807896+0000) 2020-02-06T00:11:34.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:34.807+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.808176+0000) 2020-02-06T00:11:34.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:34.808+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.808496+0000) 2020-02-06T00:11:34.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:34.808+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.809413+0000) 2020-02-06T00:11:34.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:34.809+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.809671+0000) 2020-02-06T00:11:34.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:34.809+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.810364+0000) 2020-02-06T00:11:34.811 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:34.809+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.810341+0000) 2020-02-06T00:11:34.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:34.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:34.812000+0000) 2020-02-06T00:11:34.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:34.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:36.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:36.336+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:36.337312+0000) 2020-02-06T00:11:36.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:36.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:36.978801+0000) 2020-02-06T00:11:37.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:37.239+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:37.240192+0000) 2020-02-06T00:11:37.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:37.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:37.240349+0000) 2020-02-06T00:11:37.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:37.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:37.240723+0000) 2020-02-06T00:11:39.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:39.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.238577+0000) 2020-02-06T00:11:39.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:39.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.622034+0000) 2020-02-06T00:11:39.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:39.830+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.831671+0000) 2020-02-06T00:11:39.833 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:39.830+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.832166+0000) 2020-02-06T00:11:39.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:39.831+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.832682+0000) 2020-02-06T00:11:39.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:39.831+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.832942+0000) 2020-02-06T00:11:39.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:39.832+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.833493+0000) 2020-02-06T00:11:39.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:39.832+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.833985+0000) 2020-02-06T00:11:39.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:39.833+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.834947+0000) 2020-02-06T00:11:39.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:39.833+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.834782+0000) 2020-02-06T00:11:39.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:39.833+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.835083+0000) 2020-02-06T00:11:39.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:39.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.835737+0000) 2020-02-06T00:11:39.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:39.835+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.836764+0000) 2020-02-06T00:11:39.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:39.835+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.836972+0000) 2020-02-06T00:11:39.838 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:39.835+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.837214+0000) 2020-02-06T00:11:39.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:39.839+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:39.840103+0000) 2020-02-06T00:11:39.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:39.847+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:40.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:40.469+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:40.470177+0000) 2020-02-06T00:11:40.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:40.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:40.730822+0000) 2020-02-06T00:11:42.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:42.267+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:42.268218+0000) 2020-02-06T00:11:42.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:42.849+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:42.850735+0000) 2020-02-06T00:11:43.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:43.121+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:43.121623+0000) 2020-02-06T00:11:44.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:44.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.002113+0000) 2020-02-06T00:11:44.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:44.001+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.002115+0000) 2020-02-06T00:11:44.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:44.854+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.855111+0000) 2020-02-06T00:11:44.857 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:44.855+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.856412+0000) 2020-02-06T00:11:44.858 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:44.855+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.857282+0000) 2020-02-06T00:11:44.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:44.857+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.858278+0000) 2020-02-06T00:11:44.859 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:44.857+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.858626+0000) 2020-02-06T00:11:44.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:44.858+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.859144+0000) 2020-02-06T00:11:44.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:44.858+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.859417+0000) 2020-02-06T00:11:44.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:44.858+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.859715+0000) 2020-02-06T00:11:44.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:44.858+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.860102+0000) 2020-02-06T00:11:44.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:44.859+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.860373+0000) 2020-02-06T00:11:44.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:44.859+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.860672+0000) 2020-02-06T00:11:44.862 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:44.860+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.861483+0000) 2020-02-06T00:11:44.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:44.860+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.861590+0000) 2020-02-06T00:11:44.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:44.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:44.866127+0000) 2020-02-06T00:11:44.875 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:44.872+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:46.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:46.337+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:46.337568+0000) 2020-02-06T00:11:46.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:46.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:46.979114+0000) 2020-02-06T00:11:47.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:47.239+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:47.240489+0000) 2020-02-06T00:11:47.242 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:47.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:47.240637+0000) 2020-02-06T00:11:47.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:47.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:47.241052+0000) 2020-02-06T00:11:49.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:49.237+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.238837+0000) 2020-02-06T00:11:49.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:49.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.622340+0000) 2020-02-06T00:11:49.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:49.878+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.879505+0000) 2020-02-06T00:11:49.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:49.878+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.879757+0000) 2020-02-06T00:11:49.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:49.879+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.880444+0000) 2020-02-06T00:11:49.882 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:49.880+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.881462+0000) 2020-02-06T00:11:49.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:49.880+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.881805+0000) 2020-02-06T00:11:49.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:49.880+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.881900+0000) 2020-02-06T00:11:49.883 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:49.880+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.882135+0000) 2020-02-06T00:11:49.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:49.881+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.882659+0000) 2020-02-06T00:11:49.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:49.882+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.883797+0000) 2020-02-06T00:11:49.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:49.883+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.884286+0000) 2020-02-06T00:11:49.885 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:49.883+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.884590+0000) 2020-02-06T00:11:49.886 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:49.883+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.885018+0000) 2020-02-06T00:11:49.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:49.884+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.885419+0000) 2020-02-06T00:11:49.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:49.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:49.889432+0000) 2020-02-06T00:11:49.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:49.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166817 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:50.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:50.468+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:50.470395+0000) 2020-02-06T00:11:50.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:50.729+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:50.731050+0000) 2020-02-06T00:11:52.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:52.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:52.276812+0000) 2020-02-06T00:11:52.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:52.850+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:52.850980+0000) 2020-02-06T00:11:53.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:53.120+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:53.121853+0000) 2020-02-06T00:11:54.003 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:54.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.002411+0000) 2020-02-06T00:11:54.004 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:54.002+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.002411+0000) 2020-02-06T00:11:54.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:54.903+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.904584+0000) 2020-02-06T00:11:54.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:54.904+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.905261+0000) 2020-02-06T00:11:54.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:54.904+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.905730+0000) 2020-02-06T00:11:54.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:54.905+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.906376+0000) 2020-02-06T00:11:54.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:54.905+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.907323+0000) 2020-02-06T00:11:54.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:54.906+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.907837+0000) 2020-02-06T00:11:54.909 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:54.906+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.908011+0000) 2020-02-06T00:11:54.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:54.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.908639+0000) 2020-02-06T00:11:54.910 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:54.908+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.909591+0000) 2020-02-06T00:11:54.911 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:54.907+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.909217+0000) 2020-02-06T00:11:54.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:54.908+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.910048+0000) 2020-02-06T00:11:54.911 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:54.909+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.910567+0000) 2020-02-06T00:11:54.912 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:54.909+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.911082+0000) 2020-02-06T00:11:54.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:54.913+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:54.914508+0000) 2020-02-06T00:11:54.924 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:54.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:11:56.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:56.337+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:56.337831+0000) 2020-02-06T00:11:56.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:56.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:56.979351+0000) 2020-02-06T00:11:57.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:57.239+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:57.240777+0000) 2020-02-06T00:11:57.242 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:57.239+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:57.240895+0000) 2020-02-06T00:11:57.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:57.239+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:57.241268+0000) 2020-02-06T00:11:59.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:59.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.239095+0000) 2020-02-06T00:11:59.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:59.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.622621+0000) 2020-02-06T00:11:59.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:59.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.928827+0000) 2020-02-06T00:11:59.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:59.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.930483+0000) 2020-02-06T00:11:59.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:11:59.930+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.931744+0000) 2020-02-06T00:11:59.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:11:59.930+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.932322+0000) 2020-02-06T00:11:59.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:11:59.932+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.932594+0000) 2020-02-06T00:11:59.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:11:59.933+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.933265+0000) 2020-02-06T00:11:59.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:11:59.933+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.933626+0000) 2020-02-06T00:11:59.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:11:59.932+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.934351+0000) 2020-02-06T00:11:59.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:11:59.934+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.934895+0000) 2020-02-06T00:11:59.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:11:59.935+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.935119+0000) 2020-02-06T00:11:59.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:11:59.933+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.935404+0000) 2020-02-06T00:11:59.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:11:59.934+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.935519+0000) 2020-02-06T00:11:59.937 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:11:59.933+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.935309+0000) 2020-02-06T00:11:59.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:11:59.938+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:11:59.938479+0000) 2020-02-06T00:11:59.949 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:11:59.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166900 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:00.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:00.469+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:00.470670+0000) 2020-02-06T00:12:00.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:00.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:00.731302+0000) 2020-02-06T00:12:02.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:02.283+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:02.285055+0000) 2020-02-06T00:12:02.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:02.850+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:02.851226+0000) 2020-02-06T00:12:03.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:03.121+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:03.122120+0000) 2020-02-06T00:12:04.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:04.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.002692+0000) 2020-02-06T00:12:04.004 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:04.001+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.002693+0000) 2020-02-06T00:12:04.896 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:12:04.902 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:12:04.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:04.952+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.953597+0000) 2020-02-06T00:12:04.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:04.953+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.954659+0000) 2020-02-06T00:12:04.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:04.954+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.955561+0000) 2020-02-06T00:12:04.956 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:04.954+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.955601+0000) 2020-02-06T00:12:04.957 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:04.955+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.956184+0000) 2020-02-06T00:12:04.957 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:04.954+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.956355+0000) 2020-02-06T00:12:04.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:04.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.956828+0000) 2020-02-06T00:12:04.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:04.956+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.957224+0000) 2020-02-06T00:12:04.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:04.956+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.957514+0000) 2020-02-06T00:12:04.959 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:04.957+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.958263+0000) 2020-02-06T00:12:04.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:04.956+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.958310+0000) 2020-02-06T00:12:04.960 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:04.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.958546+0000) 2020-02-06T00:12:04.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:04.957+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.958910+0000) 2020-02-06T00:12:04.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:04.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:04.961563+0000) 2020-02-06T00:12:04.973 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:04.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:06.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:06.337+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:06.338065+0000) 2020-02-06T00:12:06.980 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:06.978+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:06.979574+0000) 2020-02-06T00:12:07.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:07.240+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:07.241026+0000) 2020-02-06T00:12:07.243 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:07.240+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:07.241128+0000) 2020-02-06T00:12:07.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:07.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:07.241506+0000) 2020-02-06T00:12:09.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:09.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.239377+0000) 2020-02-06T00:12:09.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:09.621+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.622875+0000) 2020-02-06T00:12:09.978 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:09.975+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.977312+0000) 2020-02-06T00:12:09.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:09.978+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.979065+0000) 2020-02-06T00:12:09.980 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:09.978+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.979615+0000) 2020-02-06T00:12:09.981 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:09.979+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.980205+0000) 2020-02-06T00:12:09.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:09.979+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.980402+0000) 2020-02-06T00:12:09.982 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:09.979+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.981213+0000) 2020-02-06T00:12:09.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:09.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.981478+0000) 2020-02-06T00:12:09.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:09.981+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.982670+0000) 2020-02-06T00:12:09.984 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:09.981+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.983302+0000) 2020-02-06T00:12:09.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:09.982+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.983727+0000) 2020-02-06T00:12:09.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:09.982+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.983537+0000) 2020-02-06T00:12:09.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:09.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.984321+0000) 2020-02-06T00:12:09.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:09.982+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.984152+0000) 2020-02-06T00:12:09.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:09.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:09.990804+0000) 2020-02-06T00:12:09.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:09.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 166983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:10.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:10.470+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:10.470882+0000) 2020-02-06T00:12:10.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:10.730+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:10.731536+0000) 2020-02-06T00:12:12.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:12.292+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:12.293190+0000) 2020-02-06T00:12:12.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:12.850+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:12.851513+0000) 2020-02-06T00:12:13.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:13.121+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:13.122400+0000) 2020-02-06T00:12:14.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:14.001+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:14.002954+0000) 2020-02-06T00:12:14.004 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:14.001+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:14.002954+0000) 2020-02-06T00:12:15.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:15.004+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.005320+0000) 2020-02-06T00:12:15.007 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:15.004+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.005974+0000) 2020-02-06T00:12:15.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:15.005+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.007098+0000) 2020-02-06T00:12:15.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:15.006+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.007343+0000) 2020-02-06T00:12:15.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:15.006+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.007819+0000) 2020-02-06T00:12:15.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:15.007+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.008429+0000) 2020-02-06T00:12:15.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:15.007+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.008922+0000) 2020-02-06T00:12:15.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:15.008+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.009512+0000) 2020-02-06T00:12:15.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:15.008+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.009625+0000) 2020-02-06T00:12:15.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:15.008+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.009591+0000) 2020-02-06T00:12:15.012 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:15.008+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.010070+0000) 2020-02-06T00:12:15.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:15.008+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.010342+0000) 2020-02-06T00:12:15.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:15.008+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.009955+0000) 2020-02-06T00:12:15.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:15.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:15.012594+0000) 2020-02-06T00:12:15.023 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:15.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:16.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:16.337+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:16.338391+0000) 2020-02-06T00:12:16.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:16.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:16.979803+0000) 2020-02-06T00:12:17.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:17.240+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:17.241282+0000) 2020-02-06T00:12:17.243 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:17.240+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:17.241399+0000) 2020-02-06T00:12:17.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:17.240+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:17.241915+0000) 2020-02-06T00:12:19.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:19.238+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:19.239650+0000) 2020-02-06T00:12:19.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:19.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:19.623116+0000) 2020-02-06T00:12:20.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:20.025+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.026591+0000) 2020-02-06T00:12:20.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:20.026+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.027839+0000) 2020-02-06T00:12:20.030 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:20.027+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.028815+0000) 2020-02-06T00:12:20.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:20.028+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.029958+0000) 2020-02-06T00:12:20.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:20.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.030398+0000) 2020-02-06T00:12:20.032 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:20.029+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.030739+0000) 2020-02-06T00:12:20.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:20.030+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.031667+0000) 2020-02-06T00:12:20.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:20.031+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.032777+0000) 2020-02-06T00:12:20.034 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:20.032+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.033286+0000) 2020-02-06T00:12:20.034 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:20.031+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.033277+0000) 2020-02-06T00:12:20.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:20.032+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.033823+0000) 2020-02-06T00:12:20.035 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:20.033+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.034460+0000) 2020-02-06T00:12:20.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:20.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.034992+0000) 2020-02-06T00:12:20.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:20.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.037143+0000) 2020-02-06T00:12:20.048 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:20.045+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167066 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:20.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:20.470+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.471112+0000) 2020-02-06T00:12:20.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:20.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:20.731773+0000) 2020-02-06T00:12:22.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:22.303+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:22.304042+0000) 2020-02-06T00:12:22.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:22.851+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:22.851758+0000) 2020-02-06T00:12:23.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:23.121+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:23.122640+0000) 2020-02-06T00:12:24.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:24.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:24.003222+0000) 2020-02-06T00:12:24.005 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:24.002+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:24.003222+0000) 2020-02-06T00:12:25.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:25.052+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.053172+0000) 2020-02-06T00:12:25.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:25.052+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.053976+0000) 2020-02-06T00:12:25.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:25.053+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.054753+0000) 2020-02-06T00:12:25.056 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:25.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.054980+0000) 2020-02-06T00:12:25.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:25.053+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.054993+0000) 2020-02-06T00:12:25.057 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:25.054+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.055516+0000) 2020-02-06T00:12:25.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:25.055+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.056270+0000) 2020-02-06T00:12:25.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:25.055+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.056390+0000) 2020-02-06T00:12:25.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:25.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.056737+0000) 2020-02-06T00:12:25.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:25.055+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.056311+0000) 2020-02-06T00:12:25.059 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:25.055+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.056831+0000) 2020-02-06T00:12:25.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:25.056+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.057721+0000) 2020-02-06T00:12:25.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:25.056+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.058174+0000) 2020-02-06T00:12:25.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:25.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:25.063035+0000) 2020-02-06T00:12:25.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:25.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167111 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:26.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:26.338+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:26.338691+0000) 2020-02-06T00:12:26.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:26.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:26.980033+0000) 2020-02-06T00:12:27.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:27.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:27.241513+0000) 2020-02-06T00:12:27.243 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:27.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:27.241729+0000) 2020-02-06T00:12:27.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:27.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:27.242110+0000) 2020-02-06T00:12:29.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:29.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:29.239892+0000) 2020-02-06T00:12:29.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:29.622+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:29.623335+0000) 2020-02-06T00:12:30.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:30.076+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.077393+0000) 2020-02-06T00:12:30.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:30.077+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.077640+0000) 2020-02-06T00:12:30.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:30.078+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.078511+0000) 2020-02-06T00:12:30.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:30.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.079936+0000) 2020-02-06T00:12:30.081 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:30.079+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.079963+0000) 2020-02-06T00:12:30.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:30.079+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.080480+0000) 2020-02-06T00:12:30.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:30.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.081072+0000) 2020-02-06T00:12:30.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:30.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.081606+0000) 2020-02-06T00:12:30.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:30.080+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.081924+0000) 2020-02-06T00:12:30.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:30.081+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.082275+0000) 2020-02-06T00:12:30.083 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:30.082+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.082781+0000) 2020-02-06T00:12:30.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:30.083+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.083612+0000) 2020-02-06T00:12:30.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:30.083+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.083498+0000) 2020-02-06T00:12:30.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:30.087+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.088177+0000) 2020-02-06T00:12:30.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:30.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:30.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:30.470+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.471392+0000) 2020-02-06T00:12:30.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:30.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:30.731976+0000) 2020-02-06T00:12:32.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:32.312+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:32.312504+0000) 2020-02-06T00:12:32.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:32.851+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:32.852021+0000) 2020-02-06T00:12:33.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:33.122+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:33.122879+0000) 2020-02-06T00:12:34.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:34.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:34.003486+0000) 2020-02-06T00:12:34.005 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:34.003+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:34.003485+0000) 2020-02-06T00:12:35.043 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:12:35.050 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:12:35.103 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:35.100+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.101292+0000) 2020-02-06T00:12:35.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:35.101+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.102024+0000) 2020-02-06T00:12:35.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:35.101+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.102373+0000) 2020-02-06T00:12:35.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:35.101+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.102952+0000) 2020-02-06T00:12:35.105 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:35.102+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.103332+0000) 2020-02-06T00:12:35.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:35.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.103762+0000) 2020-02-06T00:12:35.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:35.104+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.104703+0000) 2020-02-06T00:12:35.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:35.104+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.104565+0000) 2020-02-06T00:12:35.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:35.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.105207+0000) 2020-02-06T00:12:35.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:35.104+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.105367+0000) 2020-02-06T00:12:35.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:35.105+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.106116+0000) 2020-02-06T00:12:35.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:35.105+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.105446+0000) 2020-02-06T00:12:35.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:35.105+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.105825+0000) 2020-02-06T00:12:35.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:35.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:35.114056+0000) 2020-02-06T00:12:35.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:35.118+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:36.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:36.338+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:36.338910+0000) 2020-02-06T00:12:36.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:36.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:36.980296+0000) 2020-02-06T00:12:37.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:37.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:37.241757+0000) 2020-02-06T00:12:37.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:37.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:37.241980+0000) 2020-02-06T00:12:37.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:37.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:37.242342+0000) 2020-02-06T00:12:39.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:39.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:39.240153+0000) 2020-02-06T00:12:39.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:39.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:39.623577+0000) 2020-02-06T00:12:40.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:40.124+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.124743+0000) 2020-02-06T00:12:40.126 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:40.125+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.125555+0000) 2020-02-06T00:12:40.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:40.125+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.126521+0000) 2020-02-06T00:12:40.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:40.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.126767+0000) 2020-02-06T00:12:40.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:40.126+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.127682+0000) 2020-02-06T00:12:40.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:40.127+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.127933+0000) 2020-02-06T00:12:40.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:40.127+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.128355+0000) 2020-02-06T00:12:40.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:40.127+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.128576+0000) 2020-02-06T00:12:40.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:40.127+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.128942+0000) 2020-02-06T00:12:40.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:40.128+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.129336+0000) 2020-02-06T00:12:40.131 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:40.128+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.129408+0000) 2020-02-06T00:12:40.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:40.129+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.130145+0000) 2020-02-06T00:12:40.132 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:40.130+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.130652+0000) 2020-02-06T00:12:40.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:40.134+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.135713+0000) 2020-02-06T00:12:40.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:40.144+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167231 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:40.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:40.471+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.471640+0000) 2020-02-06T00:12:40.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:40.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:40.732202+0000) 2020-02-06T00:12:42.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:42.319+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:42.320180+0000) 2020-02-06T00:12:42.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:42.851+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:42.852268+0000) 2020-02-06T00:12:43.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:43.122+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:43.123133+0000) 2020-02-06T00:12:44.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:44.002+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:44.003745+0000) 2020-02-06T00:12:44.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:44.002+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:44.003715+0000) 2020-02-06T00:12:45.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:45.149+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.150070+0000) 2020-02-06T00:12:45.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:45.150+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.151653+0000) 2020-02-06T00:12:45.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:45.151+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.152444+0000) 2020-02-06T00:12:45.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:45.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.152768+0000) 2020-02-06T00:12:45.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:45.152+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.153159+0000) 2020-02-06T00:12:45.155 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:45.153+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.153682+0000) 2020-02-06T00:12:45.155 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:45.153+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.153918+0000) 2020-02-06T00:12:45.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:45.153+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.154487+0000) 2020-02-06T00:12:45.156 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:45.154+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.154934+0000) 2020-02-06T00:12:45.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:45.155+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.155749+0000) 2020-02-06T00:12:45.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:45.155+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.155748+0000) 2020-02-06T00:12:45.157 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:45.155+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.156627+0000) 2020-02-06T00:12:45.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:45.155+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.156153+0000) 2020-02-06T00:12:45.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:45.161+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:45.162241+0000) 2020-02-06T00:12:45.171 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:45.168+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:46.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:46.338+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:46.339174+0000) 2020-02-06T00:12:46.981 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:46.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:46.980529+0000) 2020-02-06T00:12:47.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:47.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:47.241997+0000) 2020-02-06T00:12:47.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:47.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:47.242215+0000) 2020-02-06T00:12:47.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:47.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:47.242592+0000) 2020-02-06T00:12:49.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:49.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:49.240403+0000) 2020-02-06T00:12:49.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:49.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:49.623821+0000) 2020-02-06T00:12:50.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:50.172+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.173619+0000) 2020-02-06T00:12:50.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:50.173+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.174717+0000) 2020-02-06T00:12:50.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:50.174+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.175711+0000) 2020-02-06T00:12:50.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:50.175+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.176260+0000) 2020-02-06T00:12:50.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:50.175+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.176518+0000) 2020-02-06T00:12:50.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:50.176+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.176754+0000) 2020-02-06T00:12:50.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:50.178+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.178749+0000) 2020-02-06T00:12:50.182 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:50.179+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.179824+0000) 2020-02-06T00:12:50.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:50.179+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.179757+0000) 2020-02-06T00:12:50.182 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:50.180+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.180711+0000) 2020-02-06T00:12:50.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:50.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.178125+0000) 2020-02-06T00:12:50.183 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:50.178+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.179054+0000) 2020-02-06T00:12:50.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:50.179+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.180905+0000) 2020-02-06T00:12:50.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:50.186+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.187292+0000) 2020-02-06T00:12:50.195 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:50.193+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:50.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:50.471+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.471865+0000) 2020-02-06T00:12:50.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:50.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:50.732496+0000) 2020-02-06T00:12:52.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:52.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:52.332262+0000) 2020-02-06T00:12:52.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:52.852+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:52.852535+0000) 2020-02-06T00:12:53.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:53.123+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:53.123367+0000) 2020-02-06T00:12:54.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:54.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:54.003987+0000) 2020-02-06T00:12:54.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:54.003+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:54.003973+0000) 2020-02-06T00:12:55.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:12:55.199+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.200327+0000) 2020-02-06T00:12:55.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:12:55.199+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.201056+0000) 2020-02-06T00:12:55.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:12:55.201+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.201749+0000) 2020-02-06T00:12:55.203 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:12:55.201+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.202096+0000) 2020-02-06T00:12:55.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:55.202+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.202727+0000) 2020-02-06T00:12:55.204 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:55.202+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.203356+0000) 2020-02-06T00:12:55.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:55.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.203873+0000) 2020-02-06T00:12:55.205 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:55.203+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.203420+0000) 2020-02-06T00:12:55.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:55.202+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.203303+0000) 2020-02-06T00:12:55.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:55.203+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.204726+0000) 2020-02-06T00:12:55.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:12:55.203+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.204768+0000) 2020-02-06T00:12:55.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:55.204+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.205265+0000) 2020-02-06T00:12:55.207 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:55.204+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.205292+0000) 2020-02-06T00:12:55.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:55.209+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:55.210940+0000) 2020-02-06T00:12:55.219 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:12:55.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:12:56.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:12:56.338+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:56.339403+0000) 2020-02-06T00:12:56.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:12:56.979+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:56.980761+0000) 2020-02-06T00:12:57.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:12:57.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:57.242235+0000) 2020-02-06T00:12:57.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:12:57.242+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:57.242497+0000) 2020-02-06T00:12:57.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:12:57.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:57.242819+0000) 2020-02-06T00:12:59.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:12:59.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:59.240704+0000) 2020-02-06T00:12:59.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:12:59.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:12:59.624057+0000) 2020-02-06T00:13:00.225 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:00.222+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.223627+0000) 2020-02-06T00:13:00.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:00.223+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.224492+0000) 2020-02-06T00:13:00.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:00.223+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.224967+0000) 2020-02-06T00:13:00.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:00.225+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.225897+0000) 2020-02-06T00:13:00.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:00.225+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.226234+0000) 2020-02-06T00:13:00.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:00.226+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.226708+0000) 2020-02-06T00:13:00.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:00.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.227279+0000) 2020-02-06T00:13:00.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:00.228+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.228689+0000) 2020-02-06T00:13:00.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:00.228+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.229282+0000) 2020-02-06T00:13:00.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:00.228+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.229599+0000) 2020-02-06T00:13:00.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:00.228+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.229044+0000) 2020-02-06T00:13:00.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:00.229+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.229746+0000) 2020-02-06T00:13:00.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:00.229+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.230335+0000) 2020-02-06T00:13:00.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:00.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.236371+0000) 2020-02-06T00:13:00.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:00.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167397 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:00.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:00.471+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.472077+0000) 2020-02-06T00:13:00.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:00.731+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:00.732707+0000) 2020-02-06T00:13:02.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:02.343+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:02.344161+0000) 2020-02-06T00:13:02.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:02.852+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:02.852819+0000) 2020-02-06T00:13:03.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:03.122+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:03.123600+0000) 2020-02-06T00:13:04.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:04.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:04.004206+0000) 2020-02-06T00:13:04.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:04.003+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:04.004277+0000) 2020-02-06T00:13:05.190 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:13:05.197 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:13:05.245 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:05.243+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.244449+0000) 2020-02-06T00:13:05.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:05.245+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.246356+0000) 2020-02-06T00:13:05.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:05.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.247219+0000) 2020-02-06T00:13:05.248 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:05.246+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.247616+0000) 2020-02-06T00:13:05.249 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:05.247+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.248632+0000) 2020-02-06T00:13:05.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:05.247+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.248733+0000) 2020-02-06T00:13:05.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:05.247+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.248949+0000) 2020-02-06T00:13:05.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:05.249+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.250502+0000) 2020-02-06T00:13:05.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:05.248+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.249665+0000) 2020-02-06T00:13:05.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:05.249+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.250456+0000) 2020-02-06T00:13:05.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:05.249+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.250580+0000) 2020-02-06T00:13:05.253 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:05.250+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.251420+0000) 2020-02-06T00:13:05.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:05.250+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.251486+0000) 2020-02-06T00:13:05.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:05.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:05.260860+0000) 2020-02-06T00:13:05.265 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:05.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:06.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:06.338+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:06.339639+0000) 2020-02-06T00:13:06.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:06.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:06.980991+0000) 2020-02-06T00:13:07.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:07.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:07.242475+0000) 2020-02-06T00:13:07.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:07.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:07.242733+0000) 2020-02-06T00:13:07.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:07.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:07.243043+0000) 2020-02-06T00:13:09.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:09.239+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:09.240993+0000) 2020-02-06T00:13:09.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:09.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:09.624328+0000) 2020-02-06T00:13:10.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:10.267+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.268664+0000) 2020-02-06T00:13:10.271 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:10.268+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.269674+0000) 2020-02-06T00:13:10.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:10.270+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.271572+0000) 2020-02-06T00:13:10.273 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:10.271+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.272122+0000) 2020-02-06T00:13:10.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:10.271+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.272559+0000) 2020-02-06T00:13:10.274 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:10.271+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.272847+0000) 2020-02-06T00:13:10.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:10.271+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.273361+0000) 2020-02-06T00:13:10.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:10.272+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.273814+0000) 2020-02-06T00:13:10.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:10.273+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.274424+0000) 2020-02-06T00:13:10.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:10.273+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.275094+0000) 2020-02-06T00:13:10.277 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:10.274+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.275560+0000) 2020-02-06T00:13:10.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:10.274+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.275462+0000) 2020-02-06T00:13:10.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:10.274+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.275711+0000) 2020-02-06T00:13:10.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:10.279+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.280212+0000) 2020-02-06T00:13:10.292 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:10.289+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167483 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:10.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:10.470+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.472300+0000) 2020-02-06T00:13:10.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:10.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:10.732922+0000) 2020-02-06T00:13:12.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:12.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:12.352950+0000) 2020-02-06T00:13:12.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:12.852+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:12.853083+0000) 2020-02-06T00:13:13.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:13.123+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:13.123819+0000) 2020-02-06T00:13:14.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:14.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:14.004432+0000) 2020-02-06T00:13:14.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:14.003+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:14.004521+0000) 2020-02-06T00:13:15.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:15.296+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.297154+0000) 2020-02-06T00:13:15.298 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:15.296+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.297696+0000) 2020-02-06T00:13:15.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:15.296+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.298368+0000) 2020-02-06T00:13:15.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:15.297+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.298713+0000) 2020-02-06T00:13:15.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:15.297+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.298986+0000) 2020-02-06T00:13:15.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:15.297+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.299031+0000) 2020-02-06T00:13:15.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:15.298+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.299567+0000) 2020-02-06T00:13:15.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:15.298+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.299960+0000) 2020-02-06T00:13:15.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:15.298+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.299838+0000) 2020-02-06T00:13:15.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:15.299+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.300557+0000) 2020-02-06T00:13:15.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:15.300+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.301699+0000) 2020-02-06T00:13:15.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:15.300+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.302180+0000) 2020-02-06T00:13:15.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:15.300+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.302326+0000) 2020-02-06T00:13:15.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:15.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:15.308735+0000) 2020-02-06T00:13:15.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:15.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:16.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:16.339+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:16.339879+0000) 2020-02-06T00:13:16.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:16.980+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:16.981263+0000) 2020-02-06T00:13:17.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:17.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:17.242720+0000) 2020-02-06T00:13:17.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:17.241+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:17.243241+0000) 2020-02-06T00:13:17.244 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:17.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:17.242983+0000) 2020-02-06T00:13:19.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:19.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:19.241256+0000) 2020-02-06T00:13:19.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:19.623+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:19.624564+0000) 2020-02-06T00:13:20.321 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:20.319+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.320636+0000) 2020-02-06T00:13:20.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:20.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.321479+0000) 2020-02-06T00:13:20.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:20.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.321990+0000) 2020-02-06T00:13:20.323 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:20.320+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.322200+0000) 2020-02-06T00:13:20.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:20.321+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.322760+0000) 2020-02-06T00:13:20.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:20.322+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.323317+0000) 2020-02-06T00:13:20.324 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:20.322+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.323741+0000) 2020-02-06T00:13:20.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:20.322+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.324194+0000) 2020-02-06T00:13:20.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:20.323+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.324553+0000) 2020-02-06T00:13:20.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:20.323+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.324731+0000) 2020-02-06T00:13:20.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:20.324+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.325788+0000) 2020-02-06T00:13:20.327 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:20.325+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.326361+0000) 2020-02-06T00:13:20.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:20.324+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.326337+0000) 2020-02-06T00:13:20.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:20.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.332639+0000) 2020-02-06T00:13:20.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:20.338+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:20.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:20.471+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.472532+0000) 2020-02-06T00:13:20.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:20.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:20.733121+0000) 2020-02-06T00:13:22.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:22.363+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:22.363857+0000) 2020-02-06T00:13:22.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:22.852+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:22.853343+0000) 2020-02-06T00:13:23.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:23.123+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:23.124053+0000) 2020-02-06T00:13:24.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:24.003+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:24.004687+0000) 2020-02-06T00:13:24.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:24.003+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:24.004778+0000) 2020-02-06T00:13:25.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:25.344+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.345594+0000) 2020-02-06T00:13:25.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:25.344+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.346202+0000) 2020-02-06T00:13:25.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:25.346+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.347379+0000) 2020-02-06T00:13:25.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:25.346+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.347536+0000) 2020-02-06T00:13:25.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:25.347+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.348459+0000) 2020-02-06T00:13:25.349 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:25.347+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.348715+0000) 2020-02-06T00:13:25.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:25.347+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.349031+0000) 2020-02-06T00:13:25.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:25.348+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.349592+0000) 2020-02-06T00:13:25.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:25.346+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.347770+0000) 2020-02-06T00:13:25.351 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:25.348+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.349679+0000) 2020-02-06T00:13:25.351 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:25.349+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.350247+0000) 2020-02-06T00:13:25.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:25.349+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.350693+0000) 2020-02-06T00:13:25.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:25.349+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.350959+0000) 2020-02-06T00:13:25.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:25.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:25.355052+0000) 2020-02-06T00:13:25.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:25.362+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167608 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:26.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:26.339+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:26.340134+0000) 2020-02-06T00:13:26.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:26.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:26.981501+0000) 2020-02-06T00:13:27.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:27.241+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:27.242933+0000) 2020-02-06T00:13:27.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:27.241+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:27.243203+0000) 2020-02-06T00:13:27.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:27.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:27.243511+0000) 2020-02-06T00:13:29.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:29.240+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:29.241518+0000) 2020-02-06T00:13:29.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:29.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:29.624821+0000) 2020-02-06T00:13:30.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:30.368+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.369260+0000) 2020-02-06T00:13:30.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:30.368+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.369883+0000) 2020-02-06T00:13:30.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:30.369+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.370548+0000) 2020-02-06T00:13:30.372 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:30.369+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.370885+0000) 2020-02-06T00:13:30.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:30.369+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.371227+0000) 2020-02-06T00:13:30.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:30.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.371724+0000) 2020-02-06T00:13:30.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:30.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.372112+0000) 2020-02-06T00:13:30.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:30.371+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.372934+0000) 2020-02-06T00:13:30.374 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:30.372+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.373526+0000) 2020-02-06T00:13:30.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:30.372+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.374104+0000) 2020-02-06T00:13:30.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:30.373+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.374820+0000) 2020-02-06T00:13:30.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:30.373+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.375167+0000) 2020-02-06T00:13:30.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:30.374+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.375821+0000) 2020-02-06T00:13:30.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:30.378+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.379414+0000) 2020-02-06T00:13:30.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:30.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:30.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:30.472+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.472762+0000) 2020-02-06T00:13:30.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:30.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:30.733330+0000) 2020-02-06T00:13:32.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:32.374+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:32.375029+0000) 2020-02-06T00:13:32.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:32.852+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:32.853660+0000) 2020-02-06T00:13:33.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:33.123+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:33.124314+0000) 2020-02-06T00:13:34.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:34.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:34.004922+0000) 2020-02-06T00:13:34.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:34.004+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:34.005035+0000) 2020-02-06T00:13:35.345 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:13:35.352 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:13:35.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:35.392+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.393969+0000) 2020-02-06T00:13:35.395 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:35.393+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.394478+0000) 2020-02-06T00:13:35.396 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:35.393+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.394503+0000) 2020-02-06T00:13:35.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:35.393+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.395104+0000) 2020-02-06T00:13:35.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:35.394+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.395707+0000) 2020-02-06T00:13:35.397 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:35.394+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.395868+0000) 2020-02-06T00:13:35.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:35.395+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.396262+0000) 2020-02-06T00:13:35.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:35.394+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.395709+0000) 2020-02-06T00:13:35.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:35.395+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.396691+0000) 2020-02-06T00:13:35.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:35.396+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.397202+0000) 2020-02-06T00:13:35.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:35.395+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.396974+0000) 2020-02-06T00:13:35.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:35.396+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.397562+0000) 2020-02-06T00:13:35.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:35.395+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.396535+0000) 2020-02-06T00:13:35.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:35.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:35.402658+0000) 2020-02-06T00:13:35.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:35.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:36.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:36.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:36.340338+0000) 2020-02-06T00:13:36.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:36.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:36.981782+0000) 2020-02-06T00:13:37.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:37.242+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:37.243187+0000) 2020-02-06T00:13:37.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:37.242+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:37.243483+0000) 2020-02-06T00:13:37.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:37.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:37.243753+0000) 2020-02-06T00:13:39.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:39.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:39.241781+0000) 2020-02-06T00:13:39.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:39.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:39.625056+0000) 2020-02-06T00:13:40.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:40.415+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.416877+0000) 2020-02-06T00:13:40.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:40.416+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.417509+0000) 2020-02-06T00:13:40.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:40.416+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.417895+0000) 2020-02-06T00:13:40.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:40.417+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.418708+0000) 2020-02-06T00:13:40.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:40.419+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.420189+0000) 2020-02-06T00:13:40.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:40.419+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.420852+0000) 2020-02-06T00:13:40.422 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:40.420+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.421106+0000) 2020-02-06T00:13:40.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:40.420+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.421485+0000) 2020-02-06T00:13:40.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:40.420+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.422057+0000) 2020-02-06T00:13:40.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:40.421+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.422531+0000) 2020-02-06T00:13:40.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:40.421+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.423209+0000) 2020-02-06T00:13:40.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:40.422+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.423516+0000) 2020-02-06T00:13:40.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:40.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.423797+0000) 2020-02-06T00:13:40.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:40.427+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.428265+0000) 2020-02-06T00:13:40.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:40.435+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:40.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:40.472+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.472979+0000) 2020-02-06T00:13:40.734 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:40.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:40.733529+0000) 2020-02-06T00:13:42.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:42.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:42.383191+0000) 2020-02-06T00:13:42.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:42.853+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:42.853921+0000) 2020-02-06T00:13:43.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:43.124+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:43.124544+0000) 2020-02-06T00:13:44.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:44.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:44.005136+0000) 2020-02-06T00:13:44.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:44.004+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:44.005265+0000) 2020-02-06T00:13:45.444 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:45.442+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.442957+0000) 2020-02-06T00:13:45.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:45.441+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.443179+0000) 2020-02-06T00:13:45.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:45.443+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.443595+0000) 2020-02-06T00:13:45.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:45.443+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.444046+0000) 2020-02-06T00:13:45.445 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:45.444+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.444404+0000) 2020-02-06T00:13:45.446 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:45.444+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.444602+0000) 2020-02-06T00:13:45.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:45.443+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.445062+0000) 2020-02-06T00:13:45.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:45.444+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.445608+0000) 2020-02-06T00:13:45.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:45.446+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.446206+0000) 2020-02-06T00:13:45.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:45.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.446626+0000) 2020-02-06T00:13:45.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:45.445+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.446762+0000) 2020-02-06T00:13:45.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:45.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.447613+0000) 2020-02-06T00:13:45.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:45.446+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.447684+0000) 2020-02-06T00:13:45.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:45.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:45.451086+0000) 2020-02-06T00:13:45.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:45.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:46.341 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:46.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:46.340591+0000) 2020-02-06T00:13:46.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:46.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:46.982044+0000) 2020-02-06T00:13:47.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:47.242+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:47.243400+0000) 2020-02-06T00:13:47.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:47.242+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:47.243716+0000) 2020-02-06T00:13:47.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:47.242+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:47.243987+0000) 2020-02-06T00:13:49.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:49.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:49.242030+0000) 2020-02-06T00:13:49.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:49.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:49.625322+0000) 2020-02-06T00:13:50.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:50.467+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.468743+0000) 2020-02-06T00:13:50.470 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:50.468+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.469212+0000) 2020-02-06T00:13:50.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:50.468+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.469530+0000) 2020-02-06T00:13:50.471 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:50.468+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.470054+0000) 2020-02-06T00:13:50.471 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:50.469+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.470703+0000) 2020-02-06T00:13:50.472 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:50.469+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.470965+0000) 2020-02-06T00:13:50.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:50.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.471425+0000) 2020-02-06T00:13:50.473 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:50.470+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.472247+0000) 2020-02-06T00:13:50.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:50.470+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.472355+0000) 2020-02-06T00:13:50.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:50.471+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.473120+0000) 2020-02-06T00:13:50.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:50.471+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.473312+0000) 2020-02-06T00:13:50.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:50.472+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.473527+0000) 2020-02-06T00:13:50.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:50.472+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.473784+0000) 2020-02-06T00:13:50.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:50.472+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.473740+0000) 2020-02-06T00:13:50.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:50.474+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.475295+0000) 2020-02-06T00:13:50.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:50.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:50.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:50.732+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:50.733765+0000) 2020-02-06T00:13:52.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:52.390+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:52.391432+0000) 2020-02-06T00:13:52.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:52.853+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:52.854183+0000) 2020-02-06T00:13:53.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:53.123+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:53.124796+0000) 2020-02-06T00:13:54.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:54.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:54.005388+0000) 2020-02-06T00:13:54.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:54.004+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:54.005494+0000) 2020-02-06T00:13:55.492 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:55.489+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.490919+0000) 2020-02-06T00:13:55.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:13:55.490+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.491608+0000) 2020-02-06T00:13:55.493 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:13:55.491+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.492504+0000) 2020-02-06T00:13:55.494 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:55.492+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.493319+0000) 2020-02-06T00:13:55.495 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:55.493+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.494001+0000) 2020-02-06T00:13:55.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:13:55.494+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.494493+0000) 2020-02-06T00:13:55.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:13:55.495+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.495625+0000) 2020-02-06T00:13:55.497 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:13:55.495+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.496408+0000) 2020-02-06T00:13:55.499 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:55.496+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.496505+0000) 2020-02-06T00:13:55.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:55.496+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.496766+0000) 2020-02-06T00:13:55.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:55.496+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.496909+0000) 2020-02-06T00:13:55.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:55.495+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.496760+0000) 2020-02-06T00:13:55.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:55.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.497173+0000) 2020-02-06T00:13:55.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:55.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:55.497422+0000) 2020-02-06T00:13:55.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:13:55.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:13:56.342 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:13:56.339+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:56.340803+0000) 2020-02-06T00:13:56.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:13:56.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:56.982304+0000) 2020-02-06T00:13:57.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:13:57.243+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:57.243704+0000) 2020-02-06T00:13:57.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:13:57.243+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:57.243961+0000) 2020-02-06T00:13:57.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:13:57.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:57.244215+0000) 2020-02-06T00:13:59.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:13:59.241+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:59.242305+0000) 2020-02-06T00:13:59.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:13:59.624+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:13:59.625558+0000) 2020-02-06T00:14:00.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:00.472+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.473327+0000) 2020-02-06T00:14:00.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:00.515+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.516438+0000) 2020-02-06T00:14:00.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:00.515+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.516772+0000) 2020-02-06T00:14:00.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:00.517+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.517786+0000) 2020-02-06T00:14:00.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:00.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.518899+0000) 2020-02-06T00:14:00.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:00.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.520135+0000) 2020-02-06T00:14:00.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:00.519+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.520345+0000) 2020-02-06T00:14:00.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:00.520+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.521185+0000) 2020-02-06T00:14:00.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:00.520+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.520930+0000) 2020-02-06T00:14:00.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:00.521+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.521463+0000) 2020-02-06T00:14:00.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:00.521+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.522231+0000) 2020-02-06T00:14:00.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:00.522+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.523217+0000) 2020-02-06T00:14:00.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:00.523+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.523781+0000) 2020-02-06T00:14:00.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:00.524+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.524728+0000) 2020-02-06T00:14:00.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:00.526+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.528002+0000) 2020-02-06T00:14:00.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:00.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:00.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:00.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:00.733985+0000) 2020-02-06T00:14:02.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:02.398+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:02.399521+0000) 2020-02-06T00:14:02.855 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:02.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:02.854462+0000) 2020-02-06T00:14:03.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:03.124+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:03.125058+0000) 2020-02-06T00:14:04.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:04.004+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:04.005621+0000) 2020-02-06T00:14:04.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:04.004+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:04.005720+0000) 2020-02-06T00:14:05.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:05.540+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.541423+0000) 2020-02-06T00:14:05.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:05.542+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.542582+0000) 2020-02-06T00:14:05.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:05.542+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.543525+0000) 2020-02-06T00:14:05.545 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:05.542+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.544005+0000) 2020-02-06T00:14:05.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:05.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.544188+0000) 2020-02-06T00:14:05.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:05.544+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.544692+0000) 2020-02-06T00:14:05.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:05.545+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.545607+0000) 2020-02-06T00:14:05.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:05.545+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.546131+0000) 2020-02-06T00:14:05.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:05.545+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.546642+0000) 2020-02-06T00:14:05.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:05.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.546873+0000) 2020-02-06T00:14:05.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:05.547+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.547731+0000) 2020-02-06T00:14:05.549 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:05.547+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.547894+0000) 2020-02-06T00:14:05.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:05.548+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.548764+0000) 2020-02-06T00:14:05.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:05.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:05.553840+0000) 2020-02-06T00:14:05.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:05.562+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167940 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:05.567 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:14:05.571 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:14:06.342 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:06.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:06.341013+0000) 2020-02-06T00:14:06.983 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:06.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:06.982549+0000) 2020-02-06T00:14:07.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:07.243+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:07.243962+0000) 2020-02-06T00:14:07.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:07.243+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:07.244227+0000) 2020-02-06T00:14:07.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:07.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:07.244469+0000) 2020-02-06T00:14:09.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:09.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:09.242565+0000) 2020-02-06T00:14:09.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:09.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:09.625793+0000) 2020-02-06T00:14:10.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:10.473+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.473575+0000) 2020-02-06T00:14:10.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:10.568+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.569228+0000) 2020-02-06T00:14:10.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:10.568+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.569926+0000) 2020-02-06T00:14:10.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:10.570+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.570702+0000) 2020-02-06T00:14:10.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:10.569+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.570946+0000) 2020-02-06T00:14:10.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:10.570+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.571910+0000) 2020-02-06T00:14:10.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:10.571+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.572313+0000) 2020-02-06T00:14:10.574 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:10.572+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.572688+0000) 2020-02-06T00:14:10.574 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:10.571+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.572845+0000) 2020-02-06T00:14:10.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:10.572+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.573465+0000) 2020-02-06T00:14:10.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:10.572+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.573976+0000) 2020-02-06T00:14:10.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:10.574+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.574869+0000) 2020-02-06T00:14:10.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:10.574+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.574981+0000) 2020-02-06T00:14:10.577 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:10.575+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.575470+0000) 2020-02-06T00:14:10.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:10.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.581184+0000) 2020-02-06T00:14:10.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:10.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 167982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:10.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:10.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:10.734199+0000) 2020-02-06T00:14:12.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:12.406+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:12.407855+0000) 2020-02-06T00:14:12.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:12.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:12.854715+0000) 2020-02-06T00:14:13.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:13.124+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:13.125385+0000) 2020-02-06T00:14:14.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:14.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:14.005861+0000) 2020-02-06T00:14:14.007 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:14.005+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:14.005981+0000) 2020-02-06T00:14:15.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:15.590+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.591491+0000) 2020-02-06T00:14:15.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:15.591+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.592706+0000) 2020-02-06T00:14:15.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:15.592+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.593235+0000) 2020-02-06T00:14:15.595 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:15.593+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.593726+0000) 2020-02-06T00:14:15.596 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:15.593+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.594234+0000) 2020-02-06T00:14:15.596 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:15.594+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.595621+0000) 2020-02-06T00:14:15.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:15.595+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.595638+0000) 2020-02-06T00:14:15.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:15.595+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.595967+0000) 2020-02-06T00:14:15.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:15.595+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.596118+0000) 2020-02-06T00:14:15.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:15.594+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.595893+0000) 2020-02-06T00:14:15.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:15.596+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.597296+0000) 2020-02-06T00:14:15.599 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:15.596+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.596868+0000) 2020-02-06T00:14:15.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:15.597+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.597551+0000) 2020-02-06T00:14:15.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:15.599+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:15.600613+0000) 2020-02-06T00:14:15.612 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:15.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:16.342 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:16.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:16.341261+0000) 2020-02-06T00:14:16.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:16.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:16.982801+0000) 2020-02-06T00:14:17.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:17.244+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:17.244248+0000) 2020-02-06T00:14:17.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:17.244+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:17.244475+0000) 2020-02-06T00:14:17.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:17.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:17.244744+0000) 2020-02-06T00:14:19.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:19.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:19.242797+0000) 2020-02-06T00:14:19.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:19.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:19.626050+0000) 2020-02-06T00:14:20.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:20.473+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.473805+0000) 2020-02-06T00:14:20.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:20.614+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.615734+0000) 2020-02-06T00:14:20.617 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:20.615+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.616371+0000) 2020-02-06T00:14:20.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:20.616+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.617589+0000) 2020-02-06T00:14:20.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:20.616+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.617993+0000) 2020-02-06T00:14:20.619 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:20.617+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.618484+0000) 2020-02-06T00:14:20.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:20.618+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.619129+0000) 2020-02-06T00:14:20.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:20.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.619541+0000) 2020-02-06T00:14:20.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:20.620+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.620580+0000) 2020-02-06T00:14:20.622 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:20.619+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.621005+0000) 2020-02-06T00:14:20.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:20.620+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.621339+0000) 2020-02-06T00:14:20.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:20.622+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.622646+0000) 2020-02-06T00:14:20.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:20.622+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.623199+0000) 2020-02-06T00:14:20.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:20.623+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.623522+0000) 2020-02-06T00:14:20.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:20.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.628147+0000) 2020-02-06T00:14:20.637 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:20.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168064 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:20.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:20.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:20.734431+0000) 2020-02-06T00:14:22.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:22.414+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:22.415435+0000) 2020-02-06T00:14:22.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:22.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:22.854959+0000) 2020-02-06T00:14:23.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:23.125+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:23.125627+0000) 2020-02-06T00:14:24.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:24.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:24.006130+0000) 2020-02-06T00:14:24.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:24.005+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:24.006297+0000) 2020-02-06T00:14:25.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:25.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.639775+0000) 2020-02-06T00:14:25.642 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:25.639+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.640840+0000) 2020-02-06T00:14:25.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:25.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.642585+0000) 2020-02-06T00:14:25.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:25.642+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.642925+0000) 2020-02-06T00:14:25.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:25.642+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.643373+0000) 2020-02-06T00:14:25.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:25.643+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.644338+0000) 2020-02-06T00:14:25.646 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:25.643+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.644414+0000) 2020-02-06T00:14:25.646 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:25.643+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.644822+0000) 2020-02-06T00:14:25.646 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:25.644+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.645518+0000) 2020-02-06T00:14:25.647 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:25.645+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.645784+0000) 2020-02-06T00:14:25.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:25.645+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.646395+0000) 2020-02-06T00:14:25.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:25.646+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.646673+0000) 2020-02-06T00:14:25.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:25.647+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.647548+0000) 2020-02-06T00:14:25.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:25.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:25.650898+0000) 2020-02-06T00:14:25.661 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:25.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168106 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:26.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:26.341+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:26.341472+0000) 2020-02-06T00:14:26.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:26.981+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:26.983033+0000) 2020-02-06T00:14:27.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:27.244+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:27.244480+0000) 2020-02-06T00:14:27.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:27.244+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:27.244691+0000) 2020-02-06T00:14:27.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:27.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:27.244951+0000) 2020-02-06T00:14:29.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:29.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:29.243058+0000) 2020-02-06T00:14:29.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:29.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:29.626327+0000) 2020-02-06T00:14:30.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:30.473+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.474044+0000) 2020-02-06T00:14:30.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:30.663+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.664935+0000) 2020-02-06T00:14:30.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:30.665+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.666469+0000) 2020-02-06T00:14:30.668 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:30.666+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.667370+0000) 2020-02-06T00:14:30.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:30.667+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.667682+0000) 2020-02-06T00:14:30.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:30.667+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.668286+0000) 2020-02-06T00:14:30.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:30.667+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.668897+0000) 2020-02-06T00:14:30.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:30.668+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.669167+0000) 2020-02-06T00:14:30.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:30.668+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.669392+0000) 2020-02-06T00:14:30.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:30.668+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.669978+0000) 2020-02-06T00:14:30.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:30.669+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.669749+0000) 2020-02-06T00:14:30.672 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:30.670+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.670521+0000) 2020-02-06T00:14:30.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:30.670+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.671041+0000) 2020-02-06T00:14:30.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:30.671+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.672137+0000) 2020-02-06T00:14:30.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:30.676+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.677817+0000) 2020-02-06T00:14:30.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:30.682+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168148 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:30.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:30.733+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:30.734659+0000) 2020-02-06T00:14:32.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:32.423+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:32.423552+0000) 2020-02-06T00:14:32.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:32.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:32.855223+0000) 2020-02-06T00:14:33.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:33.124+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:33.125887+0000) 2020-02-06T00:14:34.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:34.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:34.006315+0000) 2020-02-06T00:14:34.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:34.005+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:34.006557+0000) 2020-02-06T00:14:35.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:35.686+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.687693+0000) 2020-02-06T00:14:35.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:35.687+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.688815+0000) 2020-02-06T00:14:35.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:35.689+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.690602+0000) 2020-02-06T00:14:35.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:35.690+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.691062+0000) 2020-02-06T00:14:35.692 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:35.689+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.691290+0000) 2020-02-06T00:14:35.693 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:35.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.691470+0000) 2020-02-06T00:14:35.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:35.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.691909+0000) 2020-02-06T00:14:35.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:35.691+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.692385+0000) 2020-02-06T00:14:35.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:35.691+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.692481+0000) 2020-02-06T00:14:35.694 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:35.691+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.693052+0000) 2020-02-06T00:14:35.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:35.692+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.693574+0000) 2020-02-06T00:14:35.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:35.692+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.694122+0000) 2020-02-06T00:14:35.695 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:35.693+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.694172+0000) 2020-02-06T00:14:35.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:35.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:35.696087+0000) 2020-02-06T00:14:35.708 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:35.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:35.786 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:14:35.792 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:14:36.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:36.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:36.341697+0000) 2020-02-06T00:14:36.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:36.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:36.983323+0000) 2020-02-06T00:14:37.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:37.243+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:37.244707+0000) 2020-02-06T00:14:37.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:37.243+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:37.244929+0000) 2020-02-06T00:14:37.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:37.243+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:37.245200+0000) 2020-02-06T00:14:39.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:39.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:39.243319+0000) 2020-02-06T00:14:39.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:39.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:39.626566+0000) 2020-02-06T00:14:40.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:40.473+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.474294+0000) 2020-02-06T00:14:40.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:40.711+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.711409+0000) 2020-02-06T00:14:40.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:40.710+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.712223+0000) 2020-02-06T00:14:40.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:40.712+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.713025+0000) 2020-02-06T00:14:40.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:40.711+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.713385+0000) 2020-02-06T00:14:40.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:40.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.713685+0000) 2020-02-06T00:14:40.716 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:40.715+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.715843+0000) 2020-02-06T00:14:40.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:40.717+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.717123+0000) 2020-02-06T00:14:40.718 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:40.715+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.717035+0000) 2020-02-06T00:14:40.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:40.716+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.718164+0000) 2020-02-06T00:14:40.719 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:40.717+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.718608+0000) 2020-02-06T00:14:40.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:40.717+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.718830+0000) 2020-02-06T00:14:40.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:40.719+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.719225+0000) 2020-02-06T00:14:40.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:40.719+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.719738+0000) 2020-02-06T00:14:40.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:40.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.722795+0000) 2020-02-06T00:14:40.732 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:40.729+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:40.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:40.734+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:40.734816+0000) 2020-02-06T00:14:42.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:42.431+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:42.431903+0000) 2020-02-06T00:14:42.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:42.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:42.855487+0000) 2020-02-06T00:14:43.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:43.126+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:43.126121+0000) 2020-02-06T00:14:44.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:44.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:44.006534+0000) 2020-02-06T00:14:44.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:44.006+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:44.006767+0000) 2020-02-06T00:14:45.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:45.734+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.735644+0000) 2020-02-06T00:14:45.738 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:45.735+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.737083+0000) 2020-02-06T00:14:45.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:45.736+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.737666+0000) 2020-02-06T00:14:45.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:45.737+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.738306+0000) 2020-02-06T00:14:45.739 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:45.737+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.738642+0000) 2020-02-06T00:14:45.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:45.738+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.739200+0000) 2020-02-06T00:14:45.740 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:45.738+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.739675+0000) 2020-02-06T00:14:45.741 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:45.739+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.740461+0000) 2020-02-06T00:14:45.742 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:45.739+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.740853+0000) 2020-02-06T00:14:45.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:45.739+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.740974+0000) 2020-02-06T00:14:45.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:45.740+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.741138+0000) 2020-02-06T00:14:45.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:45.739+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.741377+0000) 2020-02-06T00:14:45.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:45.740+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.741594+0000) 2020-02-06T00:14:45.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:45.744+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:45.745345+0000) 2020-02-06T00:14:45.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:45.754+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168271 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:46.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:46.340+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:46.341955+0000) 2020-02-06T00:14:46.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:46.982+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:46.983557+0000) 2020-02-06T00:14:47.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:47.244+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:47.244943+0000) 2020-02-06T00:14:47.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:47.244+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:47.245163+0000) 2020-02-06T00:14:47.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:47.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:47.245464+0000) 2020-02-06T00:14:49.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:49.242+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:49.243518+0000) 2020-02-06T00:14:49.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:49.625+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:49.626829+0000) 2020-02-06T00:14:50.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:50.473+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.474545+0000) 2020-02-06T00:14:50.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:50.734+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.735035+0000) 2020-02-06T00:14:50.761 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:50.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.760374+0000) 2020-02-06T00:14:50.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:50.759+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.761107+0000) 2020-02-06T00:14:50.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:50.760+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.761625+0000) 2020-02-06T00:14:50.763 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:50.761+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.762507+0000) 2020-02-06T00:14:50.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:50.761+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.762730+0000) 2020-02-06T00:14:50.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:50.763+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.764292+0000) 2020-02-06T00:14:50.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:50.763+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.764745+0000) 2020-02-06T00:14:50.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:50.764+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.765152+0000) 2020-02-06T00:14:50.767 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:50.763+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.765151+0000) 2020-02-06T00:14:50.767 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:50.763+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.765382+0000) 2020-02-06T00:14:50.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:50.765+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.766173+0000) 2020-02-06T00:14:50.768 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:50.765+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.766460+0000) 2020-02-06T00:14:50.769 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:50.765+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.767204+0000) 2020-02-06T00:14:50.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:50.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:50.772464+0000) 2020-02-06T00:14:50.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:50.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:52.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:52.438+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:52.439280+0000) 2020-02-06T00:14:52.857 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:52.854+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:52.855733+0000) 2020-02-06T00:14:53.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:53.125+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:53.126341+0000) 2020-02-06T00:14:54.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:54.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:54.006776+0000) 2020-02-06T00:14:54.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:54.005+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:54.007030+0000) 2020-02-06T00:14:55.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:14:55.783+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.784791+0000) 2020-02-06T00:14:55.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:14:55.784+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.785556+0000) 2020-02-06T00:14:55.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:55.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.786374+0000) 2020-02-06T00:14:55.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:14:55.785+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.786784+0000) 2020-02-06T00:14:55.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:14:55.786+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.787166+0000) 2020-02-06T00:14:55.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:55.787+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.789183+0000) 2020-02-06T00:14:55.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:55.788+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.789883+0000) 2020-02-06T00:14:55.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:55.789+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.790146+0000) 2020-02-06T00:14:55.792 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:55.789+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.790514+0000) 2020-02-06T00:14:55.792 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:55.789+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.790870+0000) 2020-02-06T00:14:55.793 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:55.789+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.791279+0000) 2020-02-06T00:14:55.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:55.790+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.791394+0000) 2020-02-06T00:14:55.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:14:55.790+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.791631+0000) 2020-02-06T00:14:55.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:55.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:55.797032+0000) 2020-02-06T00:14:55.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:14:55.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:14:56.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:14:56.341+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:56.342170+0000) 2020-02-06T00:14:56.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:14:56.982+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:56.983788+0000) 2020-02-06T00:14:57.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:14:57.243+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:57.245194+0000) 2020-02-06T00:14:57.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:14:57.243+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:57.245402+0000) 2020-02-06T00:14:57.247 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:14:57.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:57.245707+0000) 2020-02-06T00:14:59.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:14:59.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:59.243770+0000) 2020-02-06T00:14:59.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:14:59.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:14:59.627059+0000) 2020-02-06T00:15:00.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:00.474+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.474798+0000) 2020-02-06T00:15:00.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:00.734+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.735327+0000) 2020-02-06T00:15:00.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:00.809+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.810731+0000) 2020-02-06T00:15:00.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:00.810+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.811178+0000) 2020-02-06T00:15:00.812 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:00.810+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.811581+0000) 2020-02-06T00:15:00.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:00.810+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.811737+0000) 2020-02-06T00:15:00.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:00.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.812395+0000) 2020-02-06T00:15:00.814 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:00.811+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.813133+0000) 2020-02-06T00:15:00.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:00.812+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.813654+0000) 2020-02-06T00:15:00.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:00.812+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.813667+0000) 2020-02-06T00:15:00.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:00.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.814143+0000) 2020-02-06T00:15:00.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:00.812+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.814380+0000) 2020-02-06T00:15:00.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:00.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.815105+0000) 2020-02-06T00:15:00.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:00.814+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.815490+0000) 2020-02-06T00:15:00.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:00.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.816202+0000) 2020-02-06T00:15:00.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:00.819+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:00.820424+0000) 2020-02-06T00:15:00.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:00.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:02.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:02.446+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:02.447545+0000) 2020-02-06T00:15:02.857 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:02.855+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:02.856039+0000) 2020-02-06T00:15:03.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:03.125+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:03.126575+0000) 2020-02-06T00:15:04.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:04.005+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:04.007024+0000) 2020-02-06T00:15:04.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:04.006+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:04.007288+0000) 2020-02-06T00:15:05.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:05.832+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.833859+0000) 2020-02-06T00:15:05.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:05.833+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.834650+0000) 2020-02-06T00:15:05.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:05.834+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.835327+0000) 2020-02-06T00:15:05.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:05.834+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.835950+0000) 2020-02-06T00:15:05.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:05.835+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.837039+0000) 2020-02-06T00:15:05.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:05.836+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.837500+0000) 2020-02-06T00:15:05.839 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:05.837+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.838284+0000) 2020-02-06T00:15:05.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:05.836+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.838029+0000) 2020-02-06T00:15:05.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:05.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.838713+0000) 2020-02-06T00:15:05.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:05.837+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.838861+0000) 2020-02-06T00:15:05.840 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:05.837+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.839334+0000) 2020-02-06T00:15:05.841 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:05.838+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.839785+0000) 2020-02-06T00:15:05.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:05.838+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.840297+0000) 2020-02-06T00:15:05.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:05.846+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:05.847230+0000) 2020-02-06T00:15:05.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:05.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168437 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:05.960 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:15:05.967 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:15:06.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:06.341+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:06.342381+0000) 2020-02-06T00:15:06.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:06.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:06.984019+0000) 2020-02-06T00:15:07.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:07.244+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:07.245494+0000) 2020-02-06T00:15:07.247 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:07.244+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:07.245675+0000) 2020-02-06T00:15:07.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:07.244+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:07.245952+0000) 2020-02-06T00:15:09.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:09.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:09.244055+0000) 2020-02-06T00:15:09.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:09.626+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:09.627334+0000) 2020-02-06T00:15:10.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:10.474+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.475045+0000) 2020-02-06T00:15:10.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:10.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.735579+0000) 2020-02-06T00:15:10.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:10.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.860996+0000) 2020-02-06T00:15:10.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:10.860+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.861610+0000) 2020-02-06T00:15:10.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:10.860+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.861722+0000) 2020-02-06T00:15:10.863 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:10.861+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.862203+0000) 2020-02-06T00:15:10.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:10.861+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.862791+0000) 2020-02-06T00:15:10.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:10.861+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.863115+0000) 2020-02-06T00:15:10.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:10.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.864164+0000) 2020-02-06T00:15:10.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:10.863+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.864922+0000) 2020-02-06T00:15:10.867 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:10.863+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.864648+0000) 2020-02-06T00:15:10.867 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:10.863+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.865206+0000) 2020-02-06T00:15:10.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:10.863+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.865315+0000) 2020-02-06T00:15:10.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:10.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.866467+0000) 2020-02-06T00:15:10.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:10.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.866359+0000) 2020-02-06T00:15:10.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:10.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:10.872473+0000) 2020-02-06T00:15:10.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:10.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:12.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:12.455+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:12.456017+0000) 2020-02-06T00:15:12.857 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:12.855+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:12.856305+0000) 2020-02-06T00:15:13.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:13.126+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:13.126842+0000) 2020-02-06T00:15:14.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:14.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:14.007291+0000) 2020-02-06T00:15:14.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:14.006+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:14.007486+0000) 2020-02-06T00:15:15.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:15.882+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.883456+0000) 2020-02-06T00:15:15.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:15.883+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.885399+0000) 2020-02-06T00:15:15.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:15.884+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.885551+0000) 2020-02-06T00:15:15.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:15.885+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.886473+0000) 2020-02-06T00:15:15.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:15.884+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.886406+0000) 2020-02-06T00:15:15.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:15.885+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.887203+0000) 2020-02-06T00:15:15.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:15.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.887275+0000) 2020-02-06T00:15:15.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:15.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.887533+0000) 2020-02-06T00:15:15.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:15.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.887772+0000) 2020-02-06T00:15:15.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:15.886+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.887996+0000) 2020-02-06T00:15:15.890 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:15.886+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.888379+0000) 2020-02-06T00:15:15.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:15.887+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.888227+0000) 2020-02-06T00:15:15.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:15.888+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.889196+0000) 2020-02-06T00:15:15.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:15.892+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:15.893708+0000) 2020-02-06T00:15:15.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:15.899+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168520 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:16.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:16.342+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:16.342697+0000) 2020-02-06T00:15:16.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:16.983+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:16.984284+0000) 2020-02-06T00:15:17.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:17.245+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:17.245760+0000) 2020-02-06T00:15:17.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:17.245+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:17.245924+0000) 2020-02-06T00:15:17.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:17.245+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:17.246231+0000) 2020-02-06T00:15:19.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:19.243+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:19.244296+0000) 2020-02-06T00:15:19.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:19.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:19.627573+0000) 2020-02-06T00:15:20.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:20.474+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.475325+0000) 2020-02-06T00:15:20.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:20.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.735820+0000) 2020-02-06T00:15:20.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:20.906+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.907642+0000) 2020-02-06T00:15:20.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:20.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.908281+0000) 2020-02-06T00:15:20.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:20.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.908868+0000) 2020-02-06T00:15:20.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:20.907+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.908999+0000) 2020-02-06T00:15:20.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:20.907+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.909013+0000) 2020-02-06T00:15:20.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:20.907+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.909196+0000) 2020-02-06T00:15:20.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:20.908+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.909750+0000) 2020-02-06T00:15:20.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:20.908+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.909968+0000) 2020-02-06T00:15:20.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:20.908+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.910019+0000) 2020-02-06T00:15:20.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:20.909+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.910558+0000) 2020-02-06T00:15:20.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:20.908+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.910173+0000) 2020-02-06T00:15:20.913 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:20.909+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.910633+0000) 2020-02-06T00:15:20.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:20.914+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.915593+0000) 2020-02-06T00:15:20.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:20.914+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:20.915857+0000) 2020-02-06T00:15:20.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:20.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168561 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:22.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:22.467+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:22.467746+0000) 2020-02-06T00:15:22.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:22.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:22.856562+0000) 2020-02-06T00:15:23.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:23.126+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:23.127116+0000) 2020-02-06T00:15:24.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:24.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:24.007527+0000) 2020-02-06T00:15:24.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:24.007+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:24.007729+0000) 2020-02-06T00:15:25.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:25.929+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.930063+0000) 2020-02-06T00:15:25.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:25.930+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.931139+0000) 2020-02-06T00:15:25.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:25.930+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.931976+0000) 2020-02-06T00:15:25.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:25.932+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.932517+0000) 2020-02-06T00:15:25.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:25.932+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.932887+0000) 2020-02-06T00:15:25.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:25.932+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.933562+0000) 2020-02-06T00:15:25.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:25.933+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.934850+0000) 2020-02-06T00:15:25.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:25.934+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.935457+0000) 2020-02-06T00:15:25.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:25.934+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.935576+0000) 2020-02-06T00:15:25.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:25.935+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.935840+0000) 2020-02-06T00:15:25.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:25.936+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.936974+0000) 2020-02-06T00:15:25.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:25.937+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.937463+0000) 2020-02-06T00:15:25.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:25.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.942097+0000) 2020-02-06T00:15:25.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:25.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:25.942372+0000) 2020-02-06T00:15:25.951 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:25.949+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:26.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:26.342+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:26.342917+0000) 2020-02-06T00:15:26.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:26.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:26.984512+0000) 2020-02-06T00:15:27.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:27.245+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:27.246049+0000) 2020-02-06T00:15:27.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:27.245+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:27.246208+0000) 2020-02-06T00:15:27.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:27.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:27.246513+0000) 2020-02-06T00:15:29.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:29.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:29.244558+0000) 2020-02-06T00:15:29.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:29.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:29.627810+0000) 2020-02-06T00:15:30.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:30.475+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.475561+0000) 2020-02-06T00:15:30.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:30.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.736048+0000) 2020-02-06T00:15:30.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:30.953+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.954885+0000) 2020-02-06T00:15:30.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:30.954+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.955272+0000) 2020-02-06T00:15:30.957 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:30.954+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.955919+0000) 2020-02-06T00:15:30.958 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:30.956+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.956540+0000) 2020-02-06T00:15:30.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:30.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.956924+0000) 2020-02-06T00:15:30.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:30.957+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.957685+0000) 2020-02-06T00:15:30.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:30.957+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.958149+0000) 2020-02-06T00:15:30.960 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:30.957+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.958950+0000) 2020-02-06T00:15:30.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:30.958+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.959330+0000) 2020-02-06T00:15:30.961 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:30.959+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.959946+0000) 2020-02-06T00:15:30.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:30.959+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.960263+0000) 2020-02-06T00:15:30.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:30.959+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.960893+0000) 2020-02-06T00:15:30.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:30.960+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.961769+0000) 2020-02-06T00:15:30.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:30.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:30.963401+0000) 2020-02-06T00:15:30.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:30.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168644 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:32.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:32.476+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:32.476321+0000) 2020-02-06T00:15:32.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:32.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:32.856770+0000) 2020-02-06T00:15:33.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:33.127+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:33.127346+0000) 2020-02-06T00:15:34.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:34.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:34.007748+0000) 2020-02-06T00:15:34.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:34.007+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:34.007960+0000) 2020-02-06T00:15:35.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:35.981+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.981111+0000) 2020-02-06T00:15:35.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:35.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.981729+0000) 2020-02-06T00:15:35.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:35.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.983280+0000) 2020-02-06T00:15:35.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:35.980+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.981003+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:35.982+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.982483+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:35.982+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.982649+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:35.983+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.983493+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:35.982+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.983383+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:35.984+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.984237+0000) 2020-02-06T00:15:35.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:35.984+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.984476+0000) 2020-02-06T00:15:35.988 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:35.985+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.985272+0000) 2020-02-06T00:15:35.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:35.985+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.985874+0000) 2020-02-06T00:15:35.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:35.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.985876+0000) 2020-02-06T00:15:35.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:35.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:35.990973+0000) 2020-02-06T00:15:35.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:35.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:36.131 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:15:36.137 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:15:36.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:36.343+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:36.343130+0000) 2020-02-06T00:15:36.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:36.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:36.984765+0000) 2020-02-06T00:15:37.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:37.245+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:37.246294+0000) 2020-02-06T00:15:37.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:37.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:37.246506+0000) 2020-02-06T00:15:37.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:37.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:37.246750+0000) 2020-02-06T00:15:39.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:39.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:39.244824+0000) 2020-02-06T00:15:39.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:39.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:39.628045+0000) 2020-02-06T00:15:40.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:40.475+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:40.475892+0000) 2020-02-06T00:15:40.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:40.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:40.736312+0000) 2020-02-06T00:15:41.004 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:41.002+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.003256+0000) 2020-02-06T00:15:41.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:41.003+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.004168+0000) 2020-02-06T00:15:41.006 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:41.004+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.004878+0000) 2020-02-06T00:15:41.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:41.004+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.005276+0000) 2020-02-06T00:15:41.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:41.005+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.005706+0000) 2020-02-06T00:15:41.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:41.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.006131+0000) 2020-02-06T00:15:41.008 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:41.005+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.006114+0000) 2020-02-06T00:15:41.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:41.005+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.006695+0000) 2020-02-06T00:15:41.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:41.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.007606+0000) 2020-02-06T00:15:41.009 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:41.007+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.008282+0000) 2020-02-06T00:15:41.009 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:41.007+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.008197+0000) 2020-02-06T00:15:41.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:41.008+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.008621+0000) 2020-02-06T00:15:41.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:41.008+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.009235+0000) 2020-02-06T00:15:41.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:41.014+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:41.015356+0000) 2020-02-06T00:15:41.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:41.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168726 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:42.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:42.482+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:42.483374+0000) 2020-02-06T00:15:42.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:42.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:42.857024+0000) 2020-02-06T00:15:43.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:43.126+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:43.127572+0000) 2020-02-06T00:15:44.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:44.006+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:44.007982+0000) 2020-02-06T00:15:44.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:44.007+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:44.008172+0000) 2020-02-06T00:15:46.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:46.028+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.029706+0000) 2020-02-06T00:15:46.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:46.029+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.030197+0000) 2020-02-06T00:15:46.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:46.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.030494+0000) 2020-02-06T00:15:46.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:46.029+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.030832+0000) 2020-02-06T00:15:46.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:46.030+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.031214+0000) 2020-02-06T00:15:46.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:46.031+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.031763+0000) 2020-02-06T00:15:46.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:46.031+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.032103+0000) 2020-02-06T00:15:46.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:46.032+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.032856+0000) 2020-02-06T00:15:46.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:46.033+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.033524+0000) 2020-02-06T00:15:46.035 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:46.033+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.033615+0000) 2020-02-06T00:15:46.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:46.033+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.033994+0000) 2020-02-06T00:15:46.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:46.033+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.034913+0000) 2020-02-06T00:15:46.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:46.038+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.039425+0000) 2020-02-06T00:15:46.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:46.038+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.039747+0000) 2020-02-06T00:15:46.048 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:46.046+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:46.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:46.342+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.343349+0000) 2020-02-06T00:15:46.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:46.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:46.985004+0000) 2020-02-06T00:15:47.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:47.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:47.246570+0000) 2020-02-06T00:15:47.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:47.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:47.246807+0000) 2020-02-06T00:15:47.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:47.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:47.246991+0000) 2020-02-06T00:15:49.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:49.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:49.245114+0000) 2020-02-06T00:15:49.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:49.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:49.628315+0000) 2020-02-06T00:15:50.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:50.475+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:50.476125+0000) 2020-02-06T00:15:50.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:50.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:50.736546+0000) 2020-02-06T00:15:51.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:51.052+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.053705+0000) 2020-02-06T00:15:51.055 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:51.053+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.054071+0000) 2020-02-06T00:15:51.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:51.053+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.054457+0000) 2020-02-06T00:15:51.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:51.054+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.054826+0000) 2020-02-06T00:15:51.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:51.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.055606+0000) 2020-02-06T00:15:51.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:51.054+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.055264+0000) 2020-02-06T00:15:51.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:51.055+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.056392+0000) 2020-02-06T00:15:51.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:51.056+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.057481+0000) 2020-02-06T00:15:51.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:51.056+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.057257+0000) 2020-02-06T00:15:51.059 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:51.056+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.057252+0000) 2020-02-06T00:15:51.059 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:51.057+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.058044+0000) 2020-02-06T00:15:51.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:51.057+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.058238+0000) 2020-02-06T00:15:51.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:51.057+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.058657+0000) 2020-02-06T00:15:51.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:51.063+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:51.064320+0000) 2020-02-06T00:15:51.073 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:51.072+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168808 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:52.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:52.490+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:52.491237+0000) 2020-02-06T00:15:52.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:52.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:52.857283+0000) 2020-02-06T00:15:53.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:53.127+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:53.127840+0000) 2020-02-06T00:15:54.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:54.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:54.008189+0000) 2020-02-06T00:15:54.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:54.007+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:54.008417+0000) 2020-02-06T00:15:56.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:56.078+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.078424+0000) 2020-02-06T00:15:56.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:56.078+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.079616+0000) 2020-02-06T00:15:56.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:56.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.080257+0000) 2020-02-06T00:15:56.081 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:56.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.080465+0000) 2020-02-06T00:15:56.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:15:56.080+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.081161+0000) 2020-02-06T00:15:56.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:56.080+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.080995+0000) 2020-02-06T00:15:56.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:15:56.081+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.081584+0000) 2020-02-06T00:15:56.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:56.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.082027+0000) 2020-02-06T00:15:56.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:56.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.082785+0000) 2020-02-06T00:15:56.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:56.082+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.083360+0000) 2020-02-06T00:15:56.085 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:15:56.083+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.084518+0000) 2020-02-06T00:15:56.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:15:56.084+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.084570+0000) 2020-02-06T00:15:56.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:15:56.084+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.085364+0000) 2020-02-06T00:15:56.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:56.087+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.088814+0000) 2020-02-06T00:15:56.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:15:56.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168848 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:15:56.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:15:56.342+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.343629+0000) 2020-02-06T00:15:56.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:15:56.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:56.985213+0000) 2020-02-06T00:15:57.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:15:57.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:57.246833+0000) 2020-02-06T00:15:57.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:15:57.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:57.247056+0000) 2020-02-06T00:15:57.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:15:57.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:57.247257+0000) 2020-02-06T00:15:59.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:15:59.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:59.245416+0000) 2020-02-06T00:15:59.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:15:59.627+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:15:59.628587+0000) 2020-02-06T00:16:00.477 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:00.475+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:00.476403+0000) 2020-02-06T00:16:00.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:00.735+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:00.736799+0000) 2020-02-06T00:16:01.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:01.101+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.102618+0000) 2020-02-06T00:16:01.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:01.103+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.104168+0000) 2020-02-06T00:16:01.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:01.103+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.104870+0000) 2020-02-06T00:16:01.107 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:01.104+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.105274+0000) 2020-02-06T00:16:01.107 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:01.104+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.106015+0000) 2020-02-06T00:16:01.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:01.105+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.106398+0000) 2020-02-06T00:16:01.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:01.106+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.107429+0000) 2020-02-06T00:16:01.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:01.106+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.107400+0000) 2020-02-06T00:16:01.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:01.106+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.107835+0000) 2020-02-06T00:16:01.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:01.106+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.108230+0000) 2020-02-06T00:16:01.110 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:01.107+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.108897+0000) 2020-02-06T00:16:01.110 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:01.107+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.109332+0000) 2020-02-06T00:16:01.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:01.110+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.111138+0000) 2020-02-06T00:16:01.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:01.111+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:01.112120+0000) 2020-02-06T00:16:01.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:01.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168891 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:02.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:02.498+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:02.499381+0000) 2020-02-06T00:16:02.859 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:02.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:02.857597+0000) 2020-02-06T00:16:03.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:03.127+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:03.128063+0000) 2020-02-06T00:16:04.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:04.007+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:04.008376+0000) 2020-02-06T00:16:04.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:04.007+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:04.008649+0000) 2020-02-06T00:16:06.130 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:06.128+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.129317+0000) 2020-02-06T00:16:06.131 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:06.129+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.130479+0000) 2020-02-06T00:16:06.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:06.129+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.130926+0000) 2020-02-06T00:16:06.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:06.130+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.131383+0000) 2020-02-06T00:16:06.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:06.129+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.131197+0000) 2020-02-06T00:16:06.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:06.130+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.132183+0000) 2020-02-06T00:16:06.134 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:06.131+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.132702+0000) 2020-02-06T00:16:06.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:06.132+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.133585+0000) 2020-02-06T00:16:06.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:06.132+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.133872+0000) 2020-02-06T00:16:06.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:06.133+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.135194+0000) 2020-02-06T00:16:06.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:06.134+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.135219+0000) 2020-02-06T00:16:06.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:06.134+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.135533+0000) 2020-02-06T00:16:06.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:06.134+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.135613+0000) 2020-02-06T00:16:06.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:06.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.141590+0000) 2020-02-06T00:16:06.149 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:06.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168931 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:06.296 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:16:06.302 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:16:06.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:06.342+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.343821+0000) 2020-02-06T00:16:06.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:06.984+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:06.985426+0000) 2020-02-06T00:16:07.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:07.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:07.247105+0000) 2020-02-06T00:16:07.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:07.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:07.247281+0000) 2020-02-06T00:16:07.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:07.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:07.247514+0000) 2020-02-06T00:16:09.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:09.244+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:09.245716+0000) 2020-02-06T00:16:09.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:09.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:09.628824+0000) 2020-02-06T00:16:10.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:10.475+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:10.476628+0000) 2020-02-06T00:16:10.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:10.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:10.737042+0000) 2020-02-06T00:16:11.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:11.152+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.153560+0000) 2020-02-06T00:16:11.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:11.152+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.154049+0000) 2020-02-06T00:16:11.155 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:11.153+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.154436+0000) 2020-02-06T00:16:11.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:11.153+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.154933+0000) 2020-02-06T00:16:11.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:11.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.155117+0000) 2020-02-06T00:16:11.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:11.154+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.155638+0000) 2020-02-06T00:16:11.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:11.154+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.155874+0000) 2020-02-06T00:16:11.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:11.156+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.157420+0000) 2020-02-06T00:16:11.159 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:11.157+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.158189+0000) 2020-02-06T00:16:11.159 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:11.156+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.158281+0000) 2020-02-06T00:16:11.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:11.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.158602+0000) 2020-02-06T00:16:11.160 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:11.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.158936+0000) 2020-02-06T00:16:11.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:11.157+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.159286+0000) 2020-02-06T00:16:11.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:11.162+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:11.163286+0000) 2020-02-06T00:16:11.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:11.172+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 168974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:12.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:12.506+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:12.507559+0000) 2020-02-06T00:16:12.859 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:12.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:12.857880+0000) 2020-02-06T00:16:13.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:13.127+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:13.128345+0000) 2020-02-06T00:16:14.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:14.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:14.008608+0000) 2020-02-06T00:16:14.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:14.008+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:14.008973+0000) 2020-02-06T00:16:16.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:16.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.181487+0000) 2020-02-06T00:16:16.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:16.181+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.182568+0000) 2020-02-06T00:16:16.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:16.181+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.182770+0000) 2020-02-06T00:16:16.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:16.182+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.183268+0000) 2020-02-06T00:16:16.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:16.182+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.183821+0000) 2020-02-06T00:16:16.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:16.183+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.184278+0000) 2020-02-06T00:16:16.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:16.183+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.184721+0000) 2020-02-06T00:16:16.186 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:16.183+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.184873+0000) 2020-02-06T00:16:16.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:16.184+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.185567+0000) 2020-02-06T00:16:16.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:16.184+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.185746+0000) 2020-02-06T00:16:16.188 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:16.185+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.186331+0000) 2020-02-06T00:16:16.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:16.184+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.185977+0000) 2020-02-06T00:16:16.188 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:16.185+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.186470+0000) 2020-02-06T00:16:16.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:16.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.190871+0000) 2020-02-06T00:16:16.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:16.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169014 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:16.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:16.343+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.344026+0000) 2020-02-06T00:16:16.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:16.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:16.985655+0000) 2020-02-06T00:16:17.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:17.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:17.247339+0000) 2020-02-06T00:16:17.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:17.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:17.247522+0000) 2020-02-06T00:16:17.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:17.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:17.247850+0000) 2020-02-06T00:16:19.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:19.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:19.245960+0000) 2020-02-06T00:16:19.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:19.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:19.629056+0000) 2020-02-06T00:16:20.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:20.476+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:20.476894+0000) 2020-02-06T00:16:20.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:20.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:20.737316+0000) 2020-02-06T00:16:21.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:21.205+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.206743+0000) 2020-02-06T00:16:21.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:21.206+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.207982+0000) 2020-02-06T00:16:21.209 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:21.207+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.208562+0000) 2020-02-06T00:16:21.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:21.207+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.208778+0000) 2020-02-06T00:16:21.210 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:21.208+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.209138+0000) 2020-02-06T00:16:21.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:21.208+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.209662+0000) 2020-02-06T00:16:21.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:21.208+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.209683+0000) 2020-02-06T00:16:21.212 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:21.208+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.209993+0000) 2020-02-06T00:16:21.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:21.208+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.210400+0000) 2020-02-06T00:16:21.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:21.209+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.210377+0000) 2020-02-06T00:16:21.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:21.210+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.211530+0000) 2020-02-06T00:16:21.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:21.210+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.211825+0000) 2020-02-06T00:16:21.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:21.210+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.211200+0000) 2020-02-06T00:16:21.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:21.215+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:21.216160+0000) 2020-02-06T00:16:21.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:21.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:22.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:22.515+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:22.515882+0000) 2020-02-06T00:16:22.859 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:22.856+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:22.858130+0000) 2020-02-06T00:16:23.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:23.128+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:23.128610+0000) 2020-02-06T00:16:24.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:24.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:24.008879+0000) 2020-02-06T00:16:24.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:24.008+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:24.009193+0000) 2020-02-06T00:16:26.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:26.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.232557+0000) 2020-02-06T00:16:26.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:26.230+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.232113+0000) 2020-02-06T00:16:26.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:26.231+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.233255+0000) 2020-02-06T00:16:26.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:26.232+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.233876+0000) 2020-02-06T00:16:26.235 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:26.233+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.234339+0000) 2020-02-06T00:16:26.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:26.233+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.234751+0000) 2020-02-06T00:16:26.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:26.233+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.235055+0000) 2020-02-06T00:16:26.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:26.234+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.235370+0000) 2020-02-06T00:16:26.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:26.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.236123+0000) 2020-02-06T00:16:26.238 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:26.235+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.237004+0000) 2020-02-06T00:16:26.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:26.235+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.237062+0000) 2020-02-06T00:16:26.239 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:26.235+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.237323+0000) 2020-02-06T00:16:26.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:26.236+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.237587+0000) 2020-02-06T00:16:26.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:26.241+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.242236+0000) 2020-02-06T00:16:26.250 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:26.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:26.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:26.343+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.344285+0000) 2020-02-06T00:16:26.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:26.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:26.985850+0000) 2020-02-06T00:16:27.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:27.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:27.247617+0000) 2020-02-06T00:16:27.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:27.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:27.247724+0000) 2020-02-06T00:16:27.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:27.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:27.248094+0000) 2020-02-06T00:16:29.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:29.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:29.246227+0000) 2020-02-06T00:16:29.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:29.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:29.629351+0000) 2020-02-06T00:16:30.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:30.476+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:30.477190+0000) 2020-02-06T00:16:30.738 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:30.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:30.737549+0000) 2020-02-06T00:16:31.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:31.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.256168+0000) 2020-02-06T00:16:31.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:31.256+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.257017+0000) 2020-02-06T00:16:31.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:31.255+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.256867+0000) 2020-02-06T00:16:31.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:31.256+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.258049+0000) 2020-02-06T00:16:31.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:31.257+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.257617+0000) 2020-02-06T00:16:31.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:31.258+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.258480+0000) 2020-02-06T00:16:31.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:31.258+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.258682+0000) 2020-02-06T00:16:31.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:31.257+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.259348+0000) 2020-02-06T00:16:31.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:31.259+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.260768+0000) 2020-02-06T00:16:31.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:31.260+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.260959+0000) 2020-02-06T00:16:31.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:31.259+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.261022+0000) 2020-02-06T00:16:31.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:31.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.261896+0000) 2020-02-06T00:16:31.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:31.261+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.261565+0000) 2020-02-06T00:16:31.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:31.267+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:31.267621+0000) 2020-02-06T00:16:31.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:31.272+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:32.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:32.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:32.524055+0000) 2020-02-06T00:16:32.859 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:32.857+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:32.858379+0000) 2020-02-06T00:16:33.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:33.127+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:33.128871+0000) 2020-02-06T00:16:34.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:34.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:34.009187+0000) 2020-02-06T00:16:34.017 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:34.008+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:34.009437+0000) 2020-02-06T00:16:36.281 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:36.279+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.280200+0000) 2020-02-06T00:16:36.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:36.279+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.280681+0000) 2020-02-06T00:16:36.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:36.280+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.281582+0000) 2020-02-06T00:16:36.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:36.279+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.281372+0000) 2020-02-06T00:16:36.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:36.281+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.282399+0000) 2020-02-06T00:16:36.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:36.282+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.284175+0000) 2020-02-06T00:16:36.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:36.283+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.284721+0000) 2020-02-06T00:16:36.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:36.283+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.285074+0000) 2020-02-06T00:16:36.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:36.283+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.285389+0000) 2020-02-06T00:16:36.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:36.284+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.285576+0000) 2020-02-06T00:16:36.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:36.284+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.285726+0000) 2020-02-06T00:16:36.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:36.284+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.285612+0000) 2020-02-06T00:16:36.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:36.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.289870+0000) 2020-02-06T00:16:36.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:36.289+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.290183+0000) 2020-02-06T00:16:36.299 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:36.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169180 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:36.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:36.343+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.344499+0000) 2020-02-06T00:16:36.443 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:16:36.449 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:16:36.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:36.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:36.986101+0000) 2020-02-06T00:16:37.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:37.246+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:37.247809+0000) 2020-02-06T00:16:37.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:37.246+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:37.248099+0000) 2020-02-06T00:16:37.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:37.246+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:37.248315+0000) 2020-02-06T00:16:39.247 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:39.245+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:39.246482+0000) 2020-02-06T00:16:39.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:39.628+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:39.629546+0000) 2020-02-06T00:16:40.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:40.476+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:40.477469+0000) 2020-02-06T00:16:40.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:40.736+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:40.737784+0000) 2020-02-06T00:16:41.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:41.301+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.303021+0000) 2020-02-06T00:16:41.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:41.303+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.304465+0000) 2020-02-06T00:16:41.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:41.303+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.304715+0000) 2020-02-06T00:16:41.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:41.304+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.305210+0000) 2020-02-06T00:16:41.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:41.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.305489+0000) 2020-02-06T00:16:41.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:41.304+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.305622+0000) 2020-02-06T00:16:41.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:41.304+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.306358+0000) 2020-02-06T00:16:41.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:41.305+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.306655+0000) 2020-02-06T00:16:41.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:41.305+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.307243+0000) 2020-02-06T00:16:41.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:41.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.307255+0000) 2020-02-06T00:16:41.309 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:41.307+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.308171+0000) 2020-02-06T00:16:41.309 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:41.306+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.308252+0000) 2020-02-06T00:16:41.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:41.307+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.308937+0000) 2020-02-06T00:16:41.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:41.312+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:41.313226+0000) 2020-02-06T00:16:41.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:41.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:42.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:42.531+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:42.532798+0000) 2020-02-06T00:16:42.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:42.858+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:42.858660+0000) 2020-02-06T00:16:43.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:43.128+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:43.129054+0000) 2020-02-06T00:16:44.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:44.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:44.009411+0000) 2020-02-06T00:16:44.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:44.008+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:44.009652+0000) 2020-02-06T00:16:46.330 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:46.326+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.327897+0000) 2020-02-06T00:16:46.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:46.327+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.328901+0000) 2020-02-06T00:16:46.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:46.328+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.329407+0000) 2020-02-06T00:16:46.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:46.328+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.329653+0000) 2020-02-06T00:16:46.331 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:46.328+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.330042+0000) 2020-02-06T00:16:46.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:46.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.330860+0000) 2020-02-06T00:16:46.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:46.330+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.331310+0000) 2020-02-06T00:16:46.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:46.330+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.331797+0000) 2020-02-06T00:16:46.333 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:46.331+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.332232+0000) 2020-02-06T00:16:46.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:46.331+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.332616+0000) 2020-02-06T00:16:46.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:46.331+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.332773+0000) 2020-02-06T00:16:46.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:46.331+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.333230+0000) 2020-02-06T00:16:46.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:46.332+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.333728+0000) 2020-02-06T00:16:46.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:46.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.339194+0000) 2020-02-06T00:16:46.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:46.343+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.344717+0000) 2020-02-06T00:16:46.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:46.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:46.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:46.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:46.986315+0000) 2020-02-06T00:16:47.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:47.247+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:47.248026+0000) 2020-02-06T00:16:47.249 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:47.247+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:47.248275+0000) 2020-02-06T00:16:47.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:47.247+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:47.248517+0000) 2020-02-06T00:16:49.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:49.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:49.246682+0000) 2020-02-06T00:16:49.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:49.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:49.629786+0000) 2020-02-06T00:16:50.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:50.477+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:50.477729+0000) 2020-02-06T00:16:50.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:50.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:50.738014+0000) 2020-02-06T00:16:51.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:51.350+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.351344+0000) 2020-02-06T00:16:51.353 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:51.351+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.352787+0000) 2020-02-06T00:16:51.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:51.352+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.353258+0000) 2020-02-06T00:16:51.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:51.352+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.353826+0000) 2020-02-06T00:16:51.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:51.352+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.354025+0000) 2020-02-06T00:16:51.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:51.354+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.354586+0000) 2020-02-06T00:16:51.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:51.354+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.355692+0000) 2020-02-06T00:16:51.357 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:51.354+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.355315+0000) 2020-02-06T00:16:51.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:51.354+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.354874+0000) 2020-02-06T00:16:51.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:51.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.357106+0000) 2020-02-06T00:16:51.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:51.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.357476+0000) 2020-02-06T00:16:51.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:51.357+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.357424+0000) 2020-02-06T00:16:51.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:51.357+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.357649+0000) 2020-02-06T00:16:51.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:51.364+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:51.365285+0000) 2020-02-06T00:16:51.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:51.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:52.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:52.540+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:52.541225+0000) 2020-02-06T00:16:52.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:52.858+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:52.858923+0000) 2020-02-06T00:16:53.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:53.128+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:53.129323+0000) 2020-02-06T00:16:54.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:54.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:54.009659+0000) 2020-02-06T00:16:54.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:54.008+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:54.009860+0000) 2020-02-06T00:16:56.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:56.344+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.344922+0000) 2020-02-06T00:16:56.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:56.375+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.376351+0000) 2020-02-06T00:16:56.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:16:56.375+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.376894+0000) 2020-02-06T00:16:56.379 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:16:56.376+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.377640+0000) 2020-02-06T00:16:56.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:56.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.378023+0000) 2020-02-06T00:16:56.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:56.378+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.378615+0000) 2020-02-06T00:16:56.380 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:16:56.377+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.378450+0000) 2020-02-06T00:16:56.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:56.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.378421+0000) 2020-02-06T00:16:56.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:56.378+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.378901+0000) 2020-02-06T00:16:56.382 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:16:56.378+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.379374+0000) 2020-02-06T00:16:56.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:16:56.378+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.379741+0000) 2020-02-06T00:16:56.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:56.379+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.380234+0000) 2020-02-06T00:16:56.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:16:56.379+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.380285+0000) 2020-02-06T00:16:56.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:56.380+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.380674+0000) 2020-02-06T00:16:56.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:56.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.387985+0000) 2020-02-06T00:16:56.396 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:16:56.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169347 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:16:56.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:16:56.985+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:56.986526+0000) 2020-02-06T00:16:57.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:16:57.247+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:57.248235+0000) 2020-02-06T00:16:57.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:16:57.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:57.248484+0000) 2020-02-06T00:16:57.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:16:57.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:57.248746+0000) 2020-02-06T00:16:59.248 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:16:59.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:59.246928+0000) 2020-02-06T00:16:59.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:16:59.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:16:59.630019+0000) 2020-02-06T00:17:00.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:00.477+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:00.477984+0000) 2020-02-06T00:17:00.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:00.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:00.738284+0000) 2020-02-06T00:17:01.402 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:01.399+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.400616+0000) 2020-02-06T00:17:01.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:01.401+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.401561+0000) 2020-02-06T00:17:01.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:01.400+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.401976+0000) 2020-02-06T00:17:01.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:01.401+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.402454+0000) 2020-02-06T00:17:01.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:01.401+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.402729+0000) 2020-02-06T00:17:01.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:01.402+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.403390+0000) 2020-02-06T00:17:01.405 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:01.403+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.404245+0000) 2020-02-06T00:17:01.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:01.403+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.403411+0000) 2020-02-06T00:17:01.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:01.404+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.405321+0000) 2020-02-06T00:17:01.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:01.404+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.405288+0000) 2020-02-06T00:17:01.407 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:01.404+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.406026+0000) 2020-02-06T00:17:01.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:01.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.405706+0000) 2020-02-06T00:17:01.408 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:01.406+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.407046+0000) 2020-02-06T00:17:01.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:01.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:01.410470+0000) 2020-02-06T00:17:01.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:01.418+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:02.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:02.548+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:02.549437+0000) 2020-02-06T00:17:02.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:02.858+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:02.859180+0000) 2020-02-06T00:17:03.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:03.128+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:03.129549+0000) 2020-02-06T00:17:04.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:04.008+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:04.009919+0000) 2020-02-06T00:17:04.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:04.009+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:04.010118+0000) 2020-02-06T00:17:06.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:06.344+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.345159+0000) 2020-02-06T00:17:06.425 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:06.423+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.424190+0000) 2020-02-06T00:17:06.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:06.423+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.424566+0000) 2020-02-06T00:17:06.427 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:06.425+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.425647+0000) 2020-02-06T00:17:06.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:06.425+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.425752+0000) 2020-02-06T00:17:06.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:06.425+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.426420+0000) 2020-02-06T00:17:06.428 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:06.425+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.426860+0000) 2020-02-06T00:17:06.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:06.425+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.426713+0000) 2020-02-06T00:17:06.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:06.426+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.427282+0000) 2020-02-06T00:17:06.429 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:06.426+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.427680+0000) 2020-02-06T00:17:06.429 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:06.426+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.427990+0000) 2020-02-06T00:17:06.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:06.427+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.427995+0000) 2020-02-06T00:17:06.430 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:06.428+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.428603+0000) 2020-02-06T00:17:06.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:06.428+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.428804+0000) 2020-02-06T00:17:06.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:06.431+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.432601+0000) 2020-02-06T00:17:06.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:06.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169432 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:06.639 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:17:06.645 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:17:06.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:06.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:06.986745+0000) 2020-02-06T00:17:07.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:07.248+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:07.248488+0000) 2020-02-06T00:17:07.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:07.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:07.248658+0000) 2020-02-06T00:17:07.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:07.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:07.248927+0000) 2020-02-06T00:17:09.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:09.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:09.247195+0000) 2020-02-06T00:17:09.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:09.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:09.630285+0000) 2020-02-06T00:17:10.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:10.477+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:10.478232+0000) 2020-02-06T00:17:10.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:10.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:10.738509+0000) 2020-02-06T00:17:11.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:11.447+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.448984+0000) 2020-02-06T00:17:11.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:11.449+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.450128+0000) 2020-02-06T00:17:11.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:11.449+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.450557+0000) 2020-02-06T00:17:11.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:11.449+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.450784+0000) 2020-02-06T00:17:11.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:11.450+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.451132+0000) 2020-02-06T00:17:11.453 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:11.450+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.451349+0000) 2020-02-06T00:17:11.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:11.451+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.452100+0000) 2020-02-06T00:17:11.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:11.452+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.452763+0000) 2020-02-06T00:17:11.454 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:11.452+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.453062+0000) 2020-02-06T00:17:11.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:11.452+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.453603+0000) 2020-02-06T00:17:11.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:11.453+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.453687+0000) 2020-02-06T00:17:11.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:11.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.454386+0000) 2020-02-06T00:17:11.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:11.454+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.454480+0000) 2020-02-06T00:17:11.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:11.459+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:11.460858+0000) 2020-02-06T00:17:11.467 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:11.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169475 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:12.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:12.557+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:12.557631+0000) 2020-02-06T00:17:12.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:12.859+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:12.859477+0000) 2020-02-06T00:17:13.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:13.129+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:13.129781+0000) 2020-02-06T00:17:14.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:14.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:14.010144+0000) 2020-02-06T00:17:14.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:14.009+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:14.010345+0000) 2020-02-06T00:17:16.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:16.344+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.345374+0000) 2020-02-06T00:17:16.473 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:16.470+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.471846+0000) 2020-02-06T00:17:16.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:16.471+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.471859+0000) 2020-02-06T00:17:16.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:16.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.472558+0000) 2020-02-06T00:17:16.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:16.472+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.473266+0000) 2020-02-06T00:17:16.474 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:16.472+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.473254+0000) 2020-02-06T00:17:16.475 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:16.473+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.473413+0000) 2020-02-06T00:17:16.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:16.473+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.474708+0000) 2020-02-06T00:17:16.476 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:16.473+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.475051+0000) 2020-02-06T00:17:16.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:16.474+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.475809+0000) 2020-02-06T00:17:16.477 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:16.474+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.475364+0000) 2020-02-06T00:17:16.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:16.476+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.476721+0000) 2020-02-06T00:17:16.478 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:16.476+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.477048+0000) 2020-02-06T00:17:16.478 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:16.476+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.477273+0000) 2020-02-06T00:17:16.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:16.479+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.480496+0000) 2020-02-06T00:17:16.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:16.491+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:16.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:16.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:16.986964+0000) 2020-02-06T00:17:17.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:17.248+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:17.248723+0000) 2020-02-06T00:17:17.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:17.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:17.248876+0000) 2020-02-06T00:17:17.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:17.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:17.249136+0000) 2020-02-06T00:17:19.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:19.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:19.247479+0000) 2020-02-06T00:17:19.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:19.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:19.630548+0000) 2020-02-06T00:17:20.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:20.478+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:20.478481+0000) 2020-02-06T00:17:20.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:20.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:20.738729+0000) 2020-02-06T00:17:21.499 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:21.496+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.498049+0000) 2020-02-06T00:17:21.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:21.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.499644+0000) 2020-02-06T00:17:21.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:21.499+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.500111+0000) 2020-02-06T00:17:21.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:21.499+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.500569+0000) 2020-02-06T00:17:21.502 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:21.500+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.500609+0000) 2020-02-06T00:17:21.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:21.500+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.501151+0000) 2020-02-06T00:17:21.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:21.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.501548+0000) 2020-02-06T00:17:21.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:21.501+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.502022+0000) 2020-02-06T00:17:21.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:21.502+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.503195+0000) 2020-02-06T00:17:21.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:21.503+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.503509+0000) 2020-02-06T00:17:21.505 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:21.503+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.503909+0000) 2020-02-06T00:17:21.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:21.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.503926+0000) 2020-02-06T00:17:21.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:21.503+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.504342+0000) 2020-02-06T00:17:21.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:21.506+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:21.507171+0000) 2020-02-06T00:17:21.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:21.516+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169559 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:22.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:22.565+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:22.565979+0000) 2020-02-06T00:17:22.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:22.859+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:22.859716+0000) 2020-02-06T00:17:23.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:23.129+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:23.130001+0000) 2020-02-06T00:17:24.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:24.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:24.010329+0000) 2020-02-06T00:17:24.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:24.010+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:24.010552+0000) 2020-02-06T00:17:26.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:26.345+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.345617+0000) 2020-02-06T00:17:26.524 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:26.523+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.523213+0000) 2020-02-06T00:17:26.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:26.523+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.523788+0000) 2020-02-06T00:17:26.526 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:26.524+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.525018+0000) 2020-02-06T00:17:26.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:26.523+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.525199+0000) 2020-02-06T00:17:26.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:26.525+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.525434+0000) 2020-02-06T00:17:26.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:26.524+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.525742+0000) 2020-02-06T00:17:26.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:26.526+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.526802+0000) 2020-02-06T00:17:26.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:26.527+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.527368+0000) 2020-02-06T00:17:26.529 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:26.526+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.527500+0000) 2020-02-06T00:17:26.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:26.526+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.527855+0000) 2020-02-06T00:17:26.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:26.526+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.527558+0000) 2020-02-06T00:17:26.529 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:26.526+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.528164+0000) 2020-02-06T00:17:26.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:26.531+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.532019+0000) 2020-02-06T00:17:26.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:26.532+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.532453+0000) 2020-02-06T00:17:26.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:26.540+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169601 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:26.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:26.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:26.987172+0000) 2020-02-06T00:17:27.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:27.247+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:27.248937+0000) 2020-02-06T00:17:27.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:27.247+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:27.249089+0000) 2020-02-06T00:17:27.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:27.247+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:27.249337+0000) 2020-02-06T00:17:29.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:29.246+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:29.247682+0000) 2020-02-06T00:17:29.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:29.629+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:29.630816+0000) 2020-02-06T00:17:30.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:30.477+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:30.478760+0000) 2020-02-06T00:17:30.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:30.737+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:30.738973+0000) 2020-02-06T00:17:31.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:31.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.548645+0000) 2020-02-06T00:17:31.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:31.548+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.549436+0000) 2020-02-06T00:17:31.553 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:31.547+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.548946+0000) 2020-02-06T00:17:31.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:31.550+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.552089+0000) 2020-02-06T00:17:31.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:31.548+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.549918+0000) 2020-02-06T00:17:31.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:31.549+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.550653+0000) 2020-02-06T00:17:31.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:31.549+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.550657+0000) 2020-02-06T00:17:31.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:31.550+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.551460+0000) 2020-02-06T00:17:31.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:31.550+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.552379+0000) 2020-02-06T00:17:31.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:31.551+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.553286+0000) 2020-02-06T00:17:31.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:31.552+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.554187+0000) 2020-02-06T00:17:31.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:31.552+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.553663+0000) 2020-02-06T00:17:31.556 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:31.553+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.554618+0000) 2020-02-06T00:17:31.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:31.557+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:31.558421+0000) 2020-02-06T00:17:31.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:31.564+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169643 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:32.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:32.573+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:32.574142+0000) 2020-02-06T00:17:32.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:32.858+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:32.860010+0000) 2020-02-06T00:17:33.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:33.129+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:33.130275+0000) 2020-02-06T00:17:34.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:34.009+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:34.010550+0000) 2020-02-06T00:17:34.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:34.009+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:34.010795+0000) 2020-02-06T00:17:36.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:36.344+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.345887+0000) 2020-02-06T00:17:36.575 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:36.572+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.573795+0000) 2020-02-06T00:17:36.575 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:36.573+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.574631+0000) 2020-02-06T00:17:36.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:36.572+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.574381+0000) 2020-02-06T00:17:36.576 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:36.574+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.575133+0000) 2020-02-06T00:17:36.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:36.574+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.575417+0000) 2020-02-06T00:17:36.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:36.574+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.575646+0000) 2020-02-06T00:17:36.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:36.575+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.576156+0000) 2020-02-06T00:17:36.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:36.575+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.576653+0000) 2020-02-06T00:17:36.578 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:36.574+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.576404+0000) 2020-02-06T00:17:36.579 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:36.576+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.577509+0000) 2020-02-06T00:17:36.579 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:36.576+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.577826+0000) 2020-02-06T00:17:36.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:36.577+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.578365+0000) 2020-02-06T00:17:36.580 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:36.577+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.578389+0000) 2020-02-06T00:17:36.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:36.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.580357+0000) 2020-02-06T00:17:36.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:36.591+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169684 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:36.785 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:17:36.791 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:17:36.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:36.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:36.987386+0000) 2020-02-06T00:17:37.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:37.248+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:37.249169+0000) 2020-02-06T00:17:37.251 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:37.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:37.249290+0000) 2020-02-06T00:17:37.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:37.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:37.249533+0000) 2020-02-06T00:17:39.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:39.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:39.247931+0000) 2020-02-06T00:17:39.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:39.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:39.631109+0000) 2020-02-06T00:17:40.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:40.478+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:40.479008+0000) 2020-02-06T00:17:40.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:40.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:40.739201+0000) 2020-02-06T00:17:41.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:41.598+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.599633+0000) 2020-02-06T00:17:41.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:41.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.600248+0000) 2020-02-06T00:17:41.602 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:41.599+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.600535+0000) 2020-02-06T00:17:41.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:41.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.600904+0000) 2020-02-06T00:17:41.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:41.599+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.601355+0000) 2020-02-06T00:17:41.603 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:41.600+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.601728+0000) 2020-02-06T00:17:41.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:41.600+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.602086+0000) 2020-02-06T00:17:41.605 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:41.601+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.602961+0000) 2020-02-06T00:17:41.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:41.601+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.602784+0000) 2020-02-06T00:17:41.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:41.602+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.604025+0000) 2020-02-06T00:17:41.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:41.603+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.604674+0000) 2020-02-06T00:17:41.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:41.602+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.604219+0000) 2020-02-06T00:17:41.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:41.603+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.604579+0000) 2020-02-06T00:17:41.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:41.608+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:41.609497+0000) 2020-02-06T00:17:41.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:41.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169727 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:42.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:42.581+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:42.582261+0000) 2020-02-06T00:17:42.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:42.859+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:42.860300+0000) 2020-02-06T00:17:43.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:43.129+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:43.130514+0000) 2020-02-06T00:17:44.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:44.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:44.010806+0000) 2020-02-06T00:17:44.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:44.010+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:44.011059+0000) 2020-02-06T00:17:46.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:46.345+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.346179+0000) 2020-02-06T00:17:46.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:46.620+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.621452+0000) 2020-02-06T00:17:46.624 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:46.620+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.622293+0000) 2020-02-06T00:17:46.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:46.621+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.622974+0000) 2020-02-06T00:17:46.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:46.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.624325+0000) 2020-02-06T00:17:46.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:46.623+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.624652+0000) 2020-02-06T00:17:46.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:46.623+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.625218+0000) 2020-02-06T00:17:46.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:46.625+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.626106+0000) 2020-02-06T00:17:46.627 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:46.625+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.626489+0000) 2020-02-06T00:17:46.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:46.625+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.627312+0000) 2020-02-06T00:17:46.629 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:46.626+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.627953+0000) 2020-02-06T00:17:46.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:46.626+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.628342+0000) 2020-02-06T00:17:46.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:46.627+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.628646+0000) 2020-02-06T00:17:46.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:46.627+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.628902+0000) 2020-02-06T00:17:46.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:46.631+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.632310+0000) 2020-02-06T00:17:46.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:46.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169769 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:46.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:46.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:46.987630+0000) 2020-02-06T00:17:47.251 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:47.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:47.249475+0000) 2020-02-06T00:17:47.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:47.248+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:47.249400+0000) 2020-02-06T00:17:47.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:47.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:47.249733+0000) 2020-02-06T00:17:49.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:49.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:49.248190+0000) 2020-02-06T00:17:49.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:49.630+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:49.631379+0000) 2020-02-06T00:17:50.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:50.478+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:50.479241+0000) 2020-02-06T00:17:50.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:50.738+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:50.739430+0000) 2020-02-06T00:17:51.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:51.645+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.646549+0000) 2020-02-06T00:17:51.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:51.646+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.647622+0000) 2020-02-06T00:17:51.649 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:51.646+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.647498+0000) 2020-02-06T00:17:51.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:51.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.649506+0000) 2020-02-06T00:17:51.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:51.648+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.649765+0000) 2020-02-06T00:17:51.651 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:51.649+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.650063+0000) 2020-02-06T00:17:51.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:51.649+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.650500+0000) 2020-02-06T00:17:51.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:51.650+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.651247+0000) 2020-02-06T00:17:51.653 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:51.650+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.652079+0000) 2020-02-06T00:17:51.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:51.651+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.652162+0000) 2020-02-06T00:17:51.654 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:51.651+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.652420+0000) 2020-02-06T00:17:51.654 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:51.651+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.652841+0000) 2020-02-06T00:17:51.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:51.651+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.652902+0000) 2020-02-06T00:17:51.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:51.657+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:51.658203+0000) 2020-02-06T00:17:51.668 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:51.666+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169810 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:52.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:52.589+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:52.590404+0000) 2020-02-06T00:17:52.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:52.859+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:52.860552+0000) 2020-02-06T00:17:53.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:53.129+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:53.130785+0000) 2020-02-06T00:17:54.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:54.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:54.011043+0000) 2020-02-06T00:17:54.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:54.010+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:54.011306+0000) 2020-02-06T00:17:56.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:56.345+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.346433+0000) 2020-02-06T00:17:56.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:56.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.672889+0000) 2020-02-06T00:17:56.676 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:17:56.674+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.675441+0000) 2020-02-06T00:17:56.677 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:56.674+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.675460+0000) 2020-02-06T00:17:56.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:17:56.674+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.675889+0000) 2020-02-06T00:17:56.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:56.675+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.676510+0000) 2020-02-06T00:17:56.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:56.674+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.676317+0000) 2020-02-06T00:17:56.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:56.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.676701+0000) 2020-02-06T00:17:56.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:56.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.676916+0000) 2020-02-06T00:17:56.679 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:56.676+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.677835+0000) 2020-02-06T00:17:56.679 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:17:56.675+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.677397+0000) 2020-02-06T00:17:56.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:17:56.677+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.678456+0000) 2020-02-06T00:17:56.680 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:17:56.677+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.678960+0000) 2020-02-06T00:17:56.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:17:56.677+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.679007+0000) 2020-02-06T00:17:56.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:56.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.681197+0000) 2020-02-06T00:17:56.691 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:17:56.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:17:56.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:17:56.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:56.987866+0000) 2020-02-06T00:17:57.251 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:17:57.249+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:57.249723+0000) 2020-02-06T00:17:57.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:17:57.249+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:57.249724+0000) 2020-02-06T00:17:57.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:17:57.249+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:57.249992+0000) 2020-02-06T00:17:59.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:17:59.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:59.248494+0000) 2020-02-06T00:17:59.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:17:59.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:17:59.631612+0000) 2020-02-06T00:18:00.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:00.478+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:00.479473+0000) 2020-02-06T00:18:00.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:00.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:00.739663+0000) 2020-02-06T00:18:01.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:01.695+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.696534+0000) 2020-02-06T00:18:01.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:01.695+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.697195+0000) 2020-02-06T00:18:01.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:01.697+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.698144+0000) 2020-02-06T00:18:01.699 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:01.697+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.698488+0000) 2020-02-06T00:18:01.700 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:01.697+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.699053+0000) 2020-02-06T00:18:01.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:01.698+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.699594+0000) 2020-02-06T00:18:01.701 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:01.698+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.700222+0000) 2020-02-06T00:18:01.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:01.700+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.701305+0000) 2020-02-06T00:18:01.702 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:01.700+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.701516+0000) 2020-02-06T00:18:01.703 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:01.700+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.701467+0000) 2020-02-06T00:18:01.703 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:01.700+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.701627+0000) 2020-02-06T00:18:01.703 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:01.700+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.701739+0000) 2020-02-06T00:18:01.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:01.701+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.702210+0000) 2020-02-06T00:18:01.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:01.704+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:01.705330+0000) 2020-02-06T00:18:01.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:01.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:02.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:02.598+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:02.598716+0000) 2020-02-06T00:18:02.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:02.859+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:02.860786+0000) 2020-02-06T00:18:03.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:03.130+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:03.131020+0000) 2020-02-06T00:18:04.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:04.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:04.011297+0000) 2020-02-06T00:18:04.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:04.010+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:04.011471+0000) 2020-02-06T00:18:06.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:06.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.346667+0000) 2020-02-06T00:18:06.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:06.719+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.720863+0000) 2020-02-06T00:18:06.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:06.719+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.721331+0000) 2020-02-06T00:18:06.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:06.721+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.722447+0000) 2020-02-06T00:18:06.724 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:06.720+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.721869+0000) 2020-02-06T00:18:06.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:06.722+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.723744+0000) 2020-02-06T00:18:06.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:06.722+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.723819+0000) 2020-02-06T00:18:06.726 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:06.722+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724289+0000) 2020-02-06T00:18:06.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:06.723+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724561+0000) 2020-02-06T00:18:06.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:06.723+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724158+0000) 2020-02-06T00:18:06.726 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:06.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724018+0000) 2020-02-06T00:18:06.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:06.723+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724347+0000) 2020-02-06T00:18:06.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:06.723+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.724990+0000) 2020-02-06T00:18:06.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:06.724+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.725534+0000) 2020-02-06T00:18:06.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:06.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.730928+0000) 2020-02-06T00:18:06.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:06.736+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:06.919 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:18:06.926 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:18:06.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:06.986+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:06.988044+0000) 2020-02-06T00:18:07.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:07.248+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:07.249998+0000) 2020-02-06T00:18:07.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:07.248+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:07.250014+0000) 2020-02-06T00:18:07.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:07.248+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:07.250238+0000) 2020-02-06T00:18:09.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:09.247+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:09.248701+0000) 2020-02-06T00:18:09.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:09.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:09.631844+0000) 2020-02-06T00:18:10.481 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:10.478+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:10.479664+0000) 2020-02-06T00:18:10.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:10.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:10.739943+0000) 2020-02-06T00:18:11.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:11.741+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.742655+0000) 2020-02-06T00:18:11.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:11.742+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.743343+0000) 2020-02-06T00:18:11.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:11.743+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.744783+0000) 2020-02-06T00:18:11.747 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:11.744+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.745864+0000) 2020-02-06T00:18:11.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:11.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.746220+0000) 2020-02-06T00:18:11.748 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:11.745+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.747039+0000) 2020-02-06T00:18:11.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:11.746+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.747414+0000) 2020-02-06T00:18:11.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:11.746+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.747830+0000) 2020-02-06T00:18:11.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:11.748+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.749133+0000) 2020-02-06T00:18:11.750 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:11.747+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.749105+0000) 2020-02-06T00:18:11.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:11.748+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.749650+0000) 2020-02-06T00:18:11.751 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:11.748+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.750002+0000) 2020-02-06T00:18:11.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:11.749+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.750523+0000) 2020-02-06T00:18:11.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:11.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:11.755956+0000) 2020-02-06T00:18:11.763 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:11.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 169976 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:12.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:12.606+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:12.606848+0000) 2020-02-06T00:18:12.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:12.860+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:12.861058+0000) 2020-02-06T00:18:13.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:13.131+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:13.131295+0000) 2020-02-06T00:18:14.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:14.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:14.011516+0000) 2020-02-06T00:18:14.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:14.011+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:14.011695+0000) 2020-02-06T00:18:16.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:16.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.346917+0000) 2020-02-06T00:18:16.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:16.767+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.767170+0000) 2020-02-06T00:18:16.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:16.767+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.767529+0000) 2020-02-06T00:18:16.770 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:16.767+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.768886+0000) 2020-02-06T00:18:16.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:16.769+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.769702+0000) 2020-02-06T00:18:16.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:16.769+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.770307+0000) 2020-02-06T00:18:16.773 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:16.770+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.771224+0000) 2020-02-06T00:18:16.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:16.770+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.771260+0000) 2020-02-06T00:18:16.773 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:16.771+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.771628+0000) 2020-02-06T00:18:16.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:16.770+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.771803+0000) 2020-02-06T00:18:16.774 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:16.771+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.772365+0000) 2020-02-06T00:18:16.774 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:16.772+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.772782+0000) 2020-02-06T00:18:16.774 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:16.772+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.773535+0000) 2020-02-06T00:18:16.775 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:16.773+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.774272+0000) 2020-02-06T00:18:16.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:16.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.776148+0000) 2020-02-06T00:18:16.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:16.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:16.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:16.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:16.988274+0000) 2020-02-06T00:18:17.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:17.249+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:17.250256+0000) 2020-02-06T00:18:17.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:17.249+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:17.250255+0000) 2020-02-06T00:18:17.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:17.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:17.250491+0000) 2020-02-06T00:18:19.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:19.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:19.248953+0000) 2020-02-06T00:18:19.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:19.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:19.632126+0000) 2020-02-06T00:18:20.481 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:20.479+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:20.479914+0000) 2020-02-06T00:18:20.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:20.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:20.740176+0000) 2020-02-06T00:18:21.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:21.794+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.795304+0000) 2020-02-06T00:18:21.797 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:21.794+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.795702+0000) 2020-02-06T00:18:21.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:21.795+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.795997+0000) 2020-02-06T00:18:21.799 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:21.796+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.796579+0000) 2020-02-06T00:18:21.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:21.797+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.797700+0000) 2020-02-06T00:18:21.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:21.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.798239+0000) 2020-02-06T00:18:21.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:21.797+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.798425+0000) 2020-02-06T00:18:21.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:21.798+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.799177+0000) 2020-02-06T00:18:21.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:21.798+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.799215+0000) 2020-02-06T00:18:21.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:21.799+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.800018+0000) 2020-02-06T00:18:21.802 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:21.800+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.800540+0000) 2020-02-06T00:18:21.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:21.798+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.799659+0000) 2020-02-06T00:18:21.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:21.799+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.800384+0000) 2020-02-06T00:18:21.815 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:21.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:21.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:21.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:21.803980+0000) 2020-02-06T00:18:22.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:22.613+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:22.615028+0000) 2020-02-06T00:18:22.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:22.860+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:22.861333+0000) 2020-02-06T00:18:23.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:23.130+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:23.131571+0000) 2020-02-06T00:18:24.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:24.010+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:24.011787+0000) 2020-02-06T00:18:24.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:24.010+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:24.011951+0000) 2020-02-06T00:18:26.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:26.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.347146+0000) 2020-02-06T00:18:26.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:26.818+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.819286+0000) 2020-02-06T00:18:26.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:26.819+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.820550+0000) 2020-02-06T00:18:26.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:26.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.820546+0000) 2020-02-06T00:18:26.822 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:26.820+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.820815+0000) 2020-02-06T00:18:26.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:26.820+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.821418+0000) 2020-02-06T00:18:26.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:26.821+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.822165+0000) 2020-02-06T00:18:26.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:26.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.821995+0000) 2020-02-06T00:18:26.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:26.821+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.822220+0000) 2020-02-06T00:18:26.824 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:26.822+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.822596+0000) 2020-02-06T00:18:26.825 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:26.823+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.823591+0000) 2020-02-06T00:18:26.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:26.821+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.822991+0000) 2020-02-06T00:18:26.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:26.823+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.824378+0000) 2020-02-06T00:18:26.826 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:26.824+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.824918+0000) 2020-02-06T00:18:26.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:26.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.828036+0000) 2020-02-06T00:18:26.837 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:26.836+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:26.989 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:26.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:26.988479+0000) 2020-02-06T00:18:27.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:27.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:27.250527+0000) 2020-02-06T00:18:27.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:27.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:27.250530+0000) 2020-02-06T00:18:27.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:27.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:27.250698+0000) 2020-02-06T00:18:29.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:29.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:29.249217+0000) 2020-02-06T00:18:29.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:29.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:29.632321+0000) 2020-02-06T00:18:30.481 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:30.479+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:30.480166+0000) 2020-02-06T00:18:30.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:30.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:30.740415+0000) 2020-02-06T00:18:31.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:31.841+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.842254+0000) 2020-02-06T00:18:31.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:31.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.843225+0000) 2020-02-06T00:18:31.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:31.842+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.843818+0000) 2020-02-06T00:18:31.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:31.843+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.844063+0000) 2020-02-06T00:18:31.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:31.843+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.844594+0000) 2020-02-06T00:18:31.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:31.844+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.845213+0000) 2020-02-06T00:18:31.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:31.844+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.845861+0000) 2020-02-06T00:18:31.847 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:31.845+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.845772+0000) 2020-02-06T00:18:31.848 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:31.845+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.846408+0000) 2020-02-06T00:18:31.848 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:31.845+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.846347+0000) 2020-02-06T00:18:31.848 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:31.845+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.846889+0000) 2020-02-06T00:18:31.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:31.847+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.847480+0000) 2020-02-06T00:18:31.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:31.847+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.848167+0000) 2020-02-06T00:18:31.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:31.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:31.851985+0000) 2020-02-06T00:18:31.863 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:31.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:32.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:32.624+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:32.625153+0000) 2020-02-06T00:18:32.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:32.861+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:32.861639+0000) 2020-02-06T00:18:33.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:33.130+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:33.131811+0000) 2020-02-06T00:18:34.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:34.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:34.012008+0000) 2020-02-06T00:18:34.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:34.011+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:34.012263+0000) 2020-02-06T00:18:36.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:36.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.347331+0000) 2020-02-06T00:18:36.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:36.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.869188+0000) 2020-02-06T00:18:36.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:36.868+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.869804+0000) 2020-02-06T00:18:36.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:36.869+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.870354+0000) 2020-02-06T00:18:36.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:36.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.870659+0000) 2020-02-06T00:18:36.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:36.870+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.870490+0000) 2020-02-06T00:18:36.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:36.870+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.871542+0000) 2020-02-06T00:18:36.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:36.870+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.871122+0000) 2020-02-06T00:18:36.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:36.871+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.871671+0000) 2020-02-06T00:18:36.873 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:36.871+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.872247+0000) 2020-02-06T00:18:36.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:36.872+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.872634+0000) 2020-02-06T00:18:36.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:36.872+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.873456+0000) 2020-02-06T00:18:36.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:36.872+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.873581+0000) 2020-02-06T00:18:36.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:36.872+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.873485+0000) 2020-02-06T00:18:36.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:36.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.878794+0000) 2020-02-06T00:18:36.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:36.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:36.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:36.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:36.988690+0000) 2020-02-06T00:18:37.059 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:18:37.066 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:18:37.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:37.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:37.250735+0000) 2020-02-06T00:18:37.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:37.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:37.250737+0000) 2020-02-06T00:18:37.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:37.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:37.250938+0000) 2020-02-06T00:18:39.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:39.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:39.249483+0000) 2020-02-06T00:18:39.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:39.631+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:39.632532+0000) 2020-02-06T00:18:40.481 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:40.480+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:40.480420+0000) 2020-02-06T00:18:40.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:40.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:40.740648+0000) 2020-02-06T00:18:41.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:41.892+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.893282+0000) 2020-02-06T00:18:41.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:41.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.894015+0000) 2020-02-06T00:18:41.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:41.894+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.894424+0000) 2020-02-06T00:18:41.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:41.894+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.895510+0000) 2020-02-06T00:18:41.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:41.895+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.895961+0000) 2020-02-06T00:18:41.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:41.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.896588+0000) 2020-02-06T00:18:41.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:41.895+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.896995+0000) 2020-02-06T00:18:41.898 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:41.896+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.897174+0000) 2020-02-06T00:18:41.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:41.896+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.897482+0000) 2020-02-06T00:18:41.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:41.897+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.898261+0000) 2020-02-06T00:18:41.900 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:41.898+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.898471+0000) 2020-02-06T00:18:41.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:41.897+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.898150+0000) 2020-02-06T00:18:41.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:41.897+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.898409+0000) 2020-02-06T00:18:41.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:41.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:41.901747+0000) 2020-02-06T00:18:41.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:41.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:42.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:42.632+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:42.633543+0000) 2020-02-06T00:18:42.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:42.861+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:42.861894+0000) 2020-02-06T00:18:43.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:43.131+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:43.132045+0000) 2020-02-06T00:18:44.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:44.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:44.012271+0000) 2020-02-06T00:18:44.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:44.011+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:44.012512+0000) 2020-02-06T00:18:46.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:46.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.347553+0000) 2020-02-06T00:18:46.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:46.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.918821+0000) 2020-02-06T00:18:46.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:46.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.919492+0000) 2020-02-06T00:18:46.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:46.918+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.919767+0000) 2020-02-06T00:18:46.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:46.919+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.920180+0000) 2020-02-06T00:18:46.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:46.919+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.920568+0000) 2020-02-06T00:18:46.923 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:46.920+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.921306+0000) 2020-02-06T00:18:46.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:46.920+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.921035+0000) 2020-02-06T00:18:46.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:46.920+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.921643+0000) 2020-02-06T00:18:46.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:46.921+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.922771+0000) 2020-02-06T00:18:46.924 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:46.921+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.922901+0000) 2020-02-06T00:18:46.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:46.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.923290+0000) 2020-02-06T00:18:46.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:46.923+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.923848+0000) 2020-02-06T00:18:46.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:46.923+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.924161+0000) 2020-02-06T00:18:46.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:46.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.929719+0000) 2020-02-06T00:18:46.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:46.937+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:46.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:46.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:46.988904+0000) 2020-02-06T00:18:47.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:47.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:47.251004+0000) 2020-02-06T00:18:47.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:47.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:47.251004+0000) 2020-02-06T00:18:47.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:47.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:47.251195+0000) 2020-02-06T00:18:49.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:49.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:49.249730+0000) 2020-02-06T00:18:49.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:49.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:49.632804+0000) 2020-02-06T00:18:50.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:50.480+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:50.480734+0000) 2020-02-06T00:18:50.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:50.739+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:50.740892+0000) 2020-02-06T00:18:51.944 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:51.942+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.943132+0000) 2020-02-06T00:18:51.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:51.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.945643+0000) 2020-02-06T00:18:51.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:51.945+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.946268+0000) 2020-02-06T00:18:51.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:51.945+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.946681+0000) 2020-02-06T00:18:51.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:51.944+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.944741+0000) 2020-02-06T00:18:51.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:51.946+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.947137+0000) 2020-02-06T00:18:51.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:51.947+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.947682+0000) 2020-02-06T00:18:51.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:51.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.948493+0000) 2020-02-06T00:18:51.950 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:51.947+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.948825+0000) 2020-02-06T00:18:51.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:51.948+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.948832+0000) 2020-02-06T00:18:51.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:51.949+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.950071+0000) 2020-02-06T00:18:51.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:51.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.950163+0000) 2020-02-06T00:18:51.952 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:51.949+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.950081+0000) 2020-02-06T00:18:51.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:51.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:51.954386+0000) 2020-02-06T00:18:51.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:51.962+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:52.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:52.644+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:52.645380+0000) 2020-02-06T00:18:52.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:52.862+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:52.862114+0000) 2020-02-06T00:18:53.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:53.131+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:53.132323+0000) 2020-02-06T00:18:54.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:54.011+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:54.012453+0000) 2020-02-06T00:18:54.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:54.011+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:54.012767+0000) 2020-02-06T00:18:56.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:56.346+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.347781+0000) 2020-02-06T00:18:56.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:56.967+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.968719+0000) 2020-02-06T00:18:56.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:18:56.968+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.969711+0000) 2020-02-06T00:18:56.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:56.968+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.970324+0000) 2020-02-06T00:18:56.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:18:56.969+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.970865+0000) 2020-02-06T00:18:56.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:18:56.970+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.971359+0000) 2020-02-06T00:18:56.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:18:56.970+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.971874+0000) 2020-02-06T00:18:56.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:56.971+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.972296+0000) 2020-02-06T00:18:56.973 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:18:56.970+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.971928+0000) 2020-02-06T00:18:56.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:56.971+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.972890+0000) 2020-02-06T00:18:56.974 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:56.971+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.973269+0000) 2020-02-06T00:18:56.975 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:18:56.973+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.974205+0000) 2020-02-06T00:18:56.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:56.973+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.974886+0000) 2020-02-06T00:18:56.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:56.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.979227+0000) 2020-02-06T00:18:56.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:56.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.979587+0000) 2020-02-06T00:18:56.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:18:56.985+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:18:56.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:18:56.987+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:56.989058+0000) 2020-02-06T00:18:57.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:18:57.249+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:57.251254+0000) 2020-02-06T00:18:57.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:18:57.249+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:57.251253+0000) 2020-02-06T00:18:57.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:18:57.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:57.251468+0000) 2020-02-06T00:18:59.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:18:59.248+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:59.249969+0000) 2020-02-06T00:18:59.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:18:59.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:18:59.633069+0000) 2020-02-06T00:19:00.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:00.479+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:00.480963+0000) 2020-02-06T00:19:00.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:00.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:00.741151+0000) 2020-02-06T00:19:01.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:01.991+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.992599+0000) 2020-02-06T00:19:01.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:01.992+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.993518+0000) 2020-02-06T00:19:01.995 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:01.992+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.994169+0000) 2020-02-06T00:19:01.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:01.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.994601+0000) 2020-02-06T00:19:01.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:01.994+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.995209+0000) 2020-02-06T00:19:01.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:01.994+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.995471+0000) 2020-02-06T00:19:01.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:01.994+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.995722+0000) 2020-02-06T00:19:01.997 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:01.994+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.996134+0000) 2020-02-06T00:19:01.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:01.994+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.995853+0000) 2020-02-06T00:19:01.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:01.995+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.996608+0000) 2020-02-06T00:19:01.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:01.996+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.997118+0000) 2020-02-06T00:19:01.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:01.995+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.997047+0000) 2020-02-06T00:19:01.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:01.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:01.998018+0000) 2020-02-06T00:19:02.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:02.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:02.001696+0000) 2020-02-06T00:19:02.013 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:02.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:02.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:02.653+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:02.653641+0000) 2020-02-06T00:19:02.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:02.861+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:02.862339+0000) 2020-02-06T00:19:03.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:03.132+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:03.132546+0000) 2020-02-06T00:19:04.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:04.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:04.012712+0000) 2020-02-06T00:19:04.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:04.012+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:04.012959+0000) 2020-02-06T00:19:06.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:06.347+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:06.348067+0000) 2020-02-06T00:19:06.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:06.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:06.989273+0000) 2020-02-06T00:19:07.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:07.015+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.017012+0000) 2020-02-06T00:19:07.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:07.017+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.018206+0000) 2020-02-06T00:19:07.019 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:07.017+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.018666+0000) 2020-02-06T00:19:07.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:07.017+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.019229+0000) 2020-02-06T00:19:07.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:07.019+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.020153+0000) 2020-02-06T00:19:07.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:07.018+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.019973+0000) 2020-02-06T00:19:07.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:07.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.021186+0000) 2020-02-06T00:19:07.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:07.020+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.021561+0000) 2020-02-06T00:19:07.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:07.021+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.022500+0000) 2020-02-06T00:19:07.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:07.021+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.023139+0000) 2020-02-06T00:19:07.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:07.021+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.023384+0000) 2020-02-06T00:19:07.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:07.023+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.024171+0000) 2020-02-06T00:19:07.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:07.023+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.025012+0000) 2020-02-06T00:19:07.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:07.028+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.029415+0000) 2020-02-06T00:19:07.038 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:07.035+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170426 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:07.213 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:19:07.220 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:19:07.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:07.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.251475+0000) 2020-02-06T00:19:07.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:07.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.251480+0000) 2020-02-06T00:19:07.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:07.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:07.251684+0000) 2020-02-06T00:19:09.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:09.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:09.250240+0000) 2020-02-06T00:19:09.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:09.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:09.633344+0000) 2020-02-06T00:19:10.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:10.480+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:10.481215+0000) 2020-02-06T00:19:10.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:10.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:10.741441+0000) 2020-02-06T00:19:12.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:12.042+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.042940+0000) 2020-02-06T00:19:12.044 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:12.043+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.043717+0000) 2020-02-06T00:19:12.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:12.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.044000+0000) 2020-02-06T00:19:12.046 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:12.043+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.045133+0000) 2020-02-06T00:19:12.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:12.044+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.046146+0000) 2020-02-06T00:19:12.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:12.046+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.046687+0000) 2020-02-06T00:19:12.048 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:12.047+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.047067+0000) 2020-02-06T00:19:12.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:12.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.047417+0000) 2020-02-06T00:19:12.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:12.046+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.048003+0000) 2020-02-06T00:19:12.049 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:12.046+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.048119+0000) 2020-02-06T00:19:12.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:12.046+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.048181+0000) 2020-02-06T00:19:12.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:12.047+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.048482+0000) 2020-02-06T00:19:12.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:12.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.050282+0000) 2020-02-06T00:19:12.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:12.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.050490+0000) 2020-02-06T00:19:12.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:12.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:12.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:12.661+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.661947+0000) 2020-02-06T00:19:12.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:12.861+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:12.862556+0000) 2020-02-06T00:19:13.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:13.132+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:13.132777+0000) 2020-02-06T00:19:14.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:14.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:14.012940+0000) 2020-02-06T00:19:14.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:14.013+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:14.013164+0000) 2020-02-06T00:19:16.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:16.347+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:16.348356+0000) 2020-02-06T00:19:16.990 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:16.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:16.989497+0000) 2020-02-06T00:19:17.068 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:17.065+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.066920+0000) 2020-02-06T00:19:17.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:17.066+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.067758+0000) 2020-02-06T00:19:17.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:17.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.068272+0000) 2020-02-06T00:19:17.069 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:17.067+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.068795+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:17.069+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.070309+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:17.070+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.071561+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:17.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.073057+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:17.068+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.069479+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:17.068+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.070107+0000) 2020-02-06T00:19:17.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:17.070+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.071930+0000) 2020-02-06T00:19:17.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:17.071+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.073165+0000) 2020-02-06T00:19:17.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:17.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.072986+0000) 2020-02-06T00:19:17.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:17.071+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.073261+0000) 2020-02-06T00:19:17.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:17.076+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.077972+0000) 2020-02-06T00:19:17.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:17.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170507 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:17.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:17.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.251732+0000) 2020-02-06T00:19:17.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:17.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.251732+0000) 2020-02-06T00:19:17.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:17.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:17.251979+0000) 2020-02-06T00:19:19.251 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:19.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:19.250506+0000) 2020-02-06T00:19:19.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:19.632+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:19.633588+0000) 2020-02-06T00:19:20.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:20.480+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:20.481469+0000) 2020-02-06T00:19:20.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:20.740+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:20.741676+0000) 2020-02-06T00:19:22.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:22.089+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.090980+0000) 2020-02-06T00:19:22.093 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:22.090+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.091435+0000) 2020-02-06T00:19:22.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:22.091+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.092230+0000) 2020-02-06T00:19:22.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:22.091+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.092467+0000) 2020-02-06T00:19:22.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:22.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.093013+0000) 2020-02-06T00:19:22.095 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:22.092+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.093852+0000) 2020-02-06T00:19:22.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:22.093+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.094363+0000) 2020-02-06T00:19:22.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:22.093+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.094870+0000) 2020-02-06T00:19:22.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:22.094+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.096232+0000) 2020-02-06T00:19:22.098 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:22.095+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.096451+0000) 2020-02-06T00:19:22.098 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:22.095+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.096954+0000) 2020-02-06T00:19:22.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:22.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.097257+0000) 2020-02-06T00:19:22.099 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:22.096+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.097370+0000) 2020-02-06T00:19:22.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:22.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.101125+0000) 2020-02-06T00:19:22.111 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:22.108+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170545 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:22.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:22.669+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.670428+0000) 2020-02-06T00:19:22.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:22.862+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:22.862782+0000) 2020-02-06T00:19:23.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:23.132+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:23.132999+0000) 2020-02-06T00:19:24.014 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:24.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:24.013171+0000) 2020-02-06T00:19:24.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:24.012+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:24.013421+0000) 2020-02-06T00:19:26.349 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:26.347+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:26.348586+0000) 2020-02-06T00:19:26.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:26.988+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:26.989736+0000) 2020-02-06T00:19:27.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:27.114+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.115940+0000) 2020-02-06T00:19:27.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:27.115+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.117015+0000) 2020-02-06T00:19:27.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:27.115+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.116774+0000) 2020-02-06T00:19:27.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:27.116+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.117521+0000) 2020-02-06T00:19:27.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:27.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.118347+0000) 2020-02-06T00:19:27.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:27.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.119562+0000) 2020-02-06T00:19:27.121 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:27.118+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.119605+0000) 2020-02-06T00:19:27.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:27.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.120025+0000) 2020-02-06T00:19:27.122 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:27.119+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.120544+0000) 2020-02-06T00:19:27.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:27.119+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.121200+0000) 2020-02-06T00:19:27.123 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:27.121+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.122197+0000) 2020-02-06T00:19:27.123 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:27.120+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.122205+0000) 2020-02-06T00:19:27.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:27.121+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.122727+0000) 2020-02-06T00:19:27.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:27.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.125771+0000) 2020-02-06T00:19:27.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:27.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:27.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:27.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.251990+0000) 2020-02-06T00:19:27.253 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:27.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.251990+0000) 2020-02-06T00:19:27.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:27.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:27.252171+0000) 2020-02-06T00:19:29.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:29.250+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:29.250765+0000) 2020-02-06T00:19:29.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:29.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:29.633850+0000) 2020-02-06T00:19:30.483 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:30.480+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:30.481705+0000) 2020-02-06T00:19:30.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:30.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:30.741884+0000) 2020-02-06T00:19:32.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:32.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.140771+0000) 2020-02-06T00:19:32.142 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:32.140+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.141230+0000) 2020-02-06T00:19:32.143 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:32.141+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.142293+0000) 2020-02-06T00:19:32.144 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:32.141+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.142777+0000) 2020-02-06T00:19:32.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:32.142+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.143376+0000) 2020-02-06T00:19:32.145 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:32.142+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.143565+0000) 2020-02-06T00:19:32.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:32.143+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.144916+0000) 2020-02-06T00:19:32.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:32.143+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.144722+0000) 2020-02-06T00:19:32.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:32.143+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.145165+0000) 2020-02-06T00:19:32.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:32.143+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.145415+0000) 2020-02-06T00:19:32.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:32.144+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.146045+0000) 2020-02-06T00:19:32.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:32.145+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.146539+0000) 2020-02-06T00:19:32.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:32.144+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.146226+0000) 2020-02-06T00:19:32.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:32.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.152141+0000) 2020-02-06T00:19:32.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:32.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:32.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:32.678+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.679176+0000) 2020-02-06T00:19:32.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:32.862+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:32.863015+0000) 2020-02-06T00:19:33.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:33.132+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:33.133270+0000) 2020-02-06T00:19:34.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:34.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:34.013421+0000) 2020-02-06T00:19:34.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:34.012+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:34.013641+0000) 2020-02-06T00:19:36.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:36.347+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:36.348836+0000) 2020-02-06T00:19:36.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:36.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:36.989967+0000) 2020-02-06T00:19:37.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:37.162+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.163650+0000) 2020-02-06T00:19:37.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:37.164+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.165182+0000) 2020-02-06T00:19:37.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:37.164+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.165672+0000) 2020-02-06T00:19:37.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:37.164+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.166022+0000) 2020-02-06T00:19:37.167 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:37.164+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.166314+0000) 2020-02-06T00:19:37.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:37.165+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.166518+0000) 2020-02-06T00:19:37.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:37.165+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.166813+0000) 2020-02-06T00:19:37.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:37.166+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.167794+0000) 2020-02-06T00:19:37.169 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:37.166+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.167978+0000) 2020-02-06T00:19:37.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:37.166+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.168234+0000) 2020-02-06T00:19:37.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:37.168+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.169327+0000) 2020-02-06T00:19:37.170 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:37.167+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.169103+0000) 2020-02-06T00:19:37.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:37.168+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.169813+0000) 2020-02-06T00:19:37.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:37.175+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.176140+0000) 2020-02-06T00:19:37.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:37.182+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:37.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:37.250+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.252218+0000) 2020-02-06T00:19:37.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:37.250+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.252218+0000) 2020-02-06T00:19:37.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:37.250+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:37.252381+0000) 2020-02-06T00:19:37.346 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:19:37.352 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:19:39.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:39.249+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:39.251011+0000) 2020-02-06T00:19:39.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:39.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:39.634136+0000) 2020-02-06T00:19:40.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:40.481+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:40.481971+0000) 2020-02-06T00:19:40.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:40.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:40.742148+0000) 2020-02-06T00:19:42.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:42.188+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.189131+0000) 2020-02-06T00:19:42.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:42.189+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.190710+0000) 2020-02-06T00:19:42.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:42.190+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.191596+0000) 2020-02-06T00:19:42.193 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:42.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.191260+0000) 2020-02-06T00:19:42.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:42.191+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.192279+0000) 2020-02-06T00:19:42.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:42.191+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.193000+0000) 2020-02-06T00:19:42.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:42.192+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.193491+0000) 2020-02-06T00:19:42.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:42.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.193910+0000) 2020-02-06T00:19:42.195 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:42.193+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.194327+0000) 2020-02-06T00:19:42.196 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:42.193+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.195007+0000) 2020-02-06T00:19:42.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:42.194+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.195480+0000) 2020-02-06T00:19:42.197 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:42.194+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.195822+0000) 2020-02-06T00:19:42.197 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:42.194+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.196218+0000) 2020-02-06T00:19:42.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:42.200+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.201714+0000) 2020-02-06T00:19:42.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:42.209+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170707 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:42.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:42.685+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.686524+0000) 2020-02-06T00:19:42.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:42.862+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:42.863246+0000) 2020-02-06T00:19:43.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:43.132+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:43.133477+0000) 2020-02-06T00:19:44.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:44.012+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:44.013653+0000) 2020-02-06T00:19:44.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:44.012+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:44.013900+0000) 2020-02-06T00:19:46.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:46.348+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:46.349130+0000) 2020-02-06T00:19:46.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:46.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:46.990194+0000) 2020-02-06T00:19:47.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:47.215+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.216194+0000) 2020-02-06T00:19:47.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:47.217+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.217785+0000) 2020-02-06T00:19:47.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:47.218+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.218487+0000) 2020-02-06T00:19:47.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:47.218+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.218794+0000) 2020-02-06T00:19:47.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:47.218+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.219295+0000) 2020-02-06T00:19:47.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:47.218+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.219628+0000) 2020-02-06T00:19:47.221 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:47.218+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.219924+0000) 2020-02-06T00:19:47.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:47.219+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.220299+0000) 2020-02-06T00:19:47.222 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:47.220+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.221529+0000) 2020-02-06T00:19:47.223 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:47.220+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.221916+0000) 2020-02-06T00:19:47.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:47.221+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.222069+0000) 2020-02-06T00:19:47.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:47.221+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.222335+0000) 2020-02-06T00:19:47.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:47.221+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.222928+0000) 2020-02-06T00:19:47.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:47.225+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.226638+0000) 2020-02-06T00:19:47.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:47.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:47.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:47.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.252496+0000) 2020-02-06T00:19:47.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:47.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.252496+0000) 2020-02-06T00:19:47.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:47.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:47.252627+0000) 2020-02-06T00:19:49.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:49.250+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:49.251284+0000) 2020-02-06T00:19:49.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:49.633+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:49.634324+0000) 2020-02-06T00:19:50.483 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:50.481+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:50.482221+0000) 2020-02-06T00:19:50.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:50.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:50.742335+0000) 2020-02-06T00:19:52.242 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:52.239+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.240560+0000) 2020-02-06T00:19:52.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:52.240+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.241192+0000) 2020-02-06T00:19:52.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:52.240+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.241664+0000) 2020-02-06T00:19:52.243 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:52.241+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.242633+0000) 2020-02-06T00:19:52.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:52.241+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.242282+0000) 2020-02-06T00:19:52.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:52.242+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.243434+0000) 2020-02-06T00:19:52.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:52.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.243654+0000) 2020-02-06T00:19:52.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:52.243+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.243929+0000) 2020-02-06T00:19:52.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:52.244+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.245064+0000) 2020-02-06T00:19:52.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:52.244+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.245418+0000) 2020-02-06T00:19:52.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:52.245+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.245716+0000) 2020-02-06T00:19:52.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:52.245+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.246010+0000) 2020-02-06T00:19:52.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:52.245+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.246320+0000) 2020-02-06T00:19:52.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:52.249+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.250801+0000) 2020-02-06T00:19:52.262 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:52.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170787 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:52.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:52.694+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.694948+0000) 2020-02-06T00:19:52.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:52.863+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:52.863501+0000) 2020-02-06T00:19:53.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:53.133+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:53.133687+0000) 2020-02-06T00:19:54.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:54.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:54.013898+0000) 2020-02-06T00:19:54.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:54.013+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:54.014135+0000) 2020-02-06T00:19:56.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:56.349+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:56.349410+0000) 2020-02-06T00:19:56.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:56.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:56.990421+0000) 2020-02-06T00:19:57.254 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:57.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.252810+0000) 2020-02-06T00:19:57.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:57.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.252766+0000) 2020-02-06T00:19:57.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:57.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.252767+0000) 2020-02-06T00:19:57.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:57.262+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.263840+0000) 2020-02-06T00:19:57.265 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:19:57.263+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.264617+0000) 2020-02-06T00:19:57.266 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:19:57.264+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.265154+0000) 2020-02-06T00:19:57.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:19:57.264+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.265923+0000) 2020-02-06T00:19:57.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:19:57.265+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.266411+0000) 2020-02-06T00:19:57.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:19:57.266+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.267060+0000) 2020-02-06T00:19:57.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:57.266+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.267238+0000) 2020-02-06T00:19:57.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:19:57.266+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.267776+0000) 2020-02-06T00:19:57.269 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:19:57.267+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.268195+0000) 2020-02-06T00:19:57.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:19:57.268+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.268784+0000) 2020-02-06T00:19:57.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:19:57.268+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.268661+0000) 2020-02-06T00:19:57.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:57.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.269243+0000) 2020-02-06T00:19:57.271 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:19:57.269+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.270294+0000) 2020-02-06T00:19:57.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:57.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:57.274930+0000) 2020-02-06T00:19:57.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:19:57.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170829 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:19:59.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:19:59.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:59.251527+0000) 2020-02-06T00:19:59.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:19:59.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:19:59.634591+0000) 2020-02-06T00:20:00.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:00.482+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:00.482519+0000) 2020-02-06T00:20:00.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:00.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:00.742571+0000) 2020-02-06T00:20:02.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:02.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.290833+0000) 2020-02-06T00:20:02.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:02.290+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.291660+0000) 2020-02-06T00:20:02.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:02.291+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.292084+0000) 2020-02-06T00:20:02.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:02.292+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.292664+0000) 2020-02-06T00:20:02.294 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:02.292+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.293310+0000) 2020-02-06T00:20:02.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:02.292+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.293655+0000) 2020-02-06T00:20:02.295 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:02.292+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.294007+0000) 2020-02-06T00:20:02.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:02.294+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.294656+0000) 2020-02-06T00:20:02.296 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:02.293+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.294919+0000) 2020-02-06T00:20:02.296 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:02.294+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.295645+0000) 2020-02-06T00:20:02.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:02.295+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.295815+0000) 2020-02-06T00:20:02.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:02.295+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.296013+0000) 2020-02-06T00:20:02.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:02.295+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.296174+0000) 2020-02-06T00:20:02.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:02.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.300720+0000) 2020-02-06T00:20:02.310 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:02.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:02.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:02.703+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.703668+0000) 2020-02-06T00:20:02.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:02.863+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:02.863749+0000) 2020-02-06T00:20:03.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:03.133+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:03.133901+0000) 2020-02-06T00:20:04.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:04.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:04.014136+0000) 2020-02-06T00:20:04.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:04.013+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:04.014401+0000) 2020-02-06T00:20:06.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:06.349+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:06.349675+0000) 2020-02-06T00:20:06.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:06.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:06.990644+0000) 2020-02-06T00:20:07.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:07.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.253060+0000) 2020-02-06T00:20:07.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:07.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.253076+0000) 2020-02-06T00:20:07.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:07.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.253077+0000) 2020-02-06T00:20:07.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:07.314+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.314956+0000) 2020-02-06T00:20:07.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:07.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.315486+0000) 2020-02-06T00:20:07.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:07.316+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.316430+0000) 2020-02-06T00:20:07.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:07.316+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.317178+0000) 2020-02-06T00:20:07.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:07.317+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.317504+0000) 2020-02-06T00:20:07.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:07.317+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.317922+0000) 2020-02-06T00:20:07.319 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:07.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.318469+0000) 2020-02-06T00:20:07.320 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:07.318+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.318644+0000) 2020-02-06T00:20:07.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:07.318+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.318718+0000) 2020-02-06T00:20:07.321 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:07.318+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.319338+0000) 2020-02-06T00:20:07.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:07.319+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.319424+0000) 2020-02-06T00:20:07.323 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:07.319+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.319746+0000) 2020-02-06T00:20:07.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:07.319+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.319728+0000) 2020-02-06T00:20:07.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:07.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:07.323742+0000) 2020-02-06T00:20:07.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:07.332+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:07.503 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:20:07.509 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:20:09.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:09.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:09.251768+0000) 2020-02-06T00:20:09.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:09.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:09.634843+0000) 2020-02-06T00:20:10.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:10.482+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:10.482767+0000) 2020-02-06T00:20:10.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:10.741+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:10.742898+0000) 2020-02-06T00:20:12.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:12.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.338475+0000) 2020-02-06T00:20:12.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:12.339+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.339666+0000) 2020-02-06T00:20:12.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:12.338+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.339992+0000) 2020-02-06T00:20:12.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:12.340+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.340489+0000) 2020-02-06T00:20:12.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:12.339+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.341034+0000) 2020-02-06T00:20:12.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:12.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.341588+0000) 2020-02-06T00:20:12.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:12.341+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.342175+0000) 2020-02-06T00:20:12.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:12.342+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.342464+0000) 2020-02-06T00:20:12.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:12.342+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.343157+0000) 2020-02-06T00:20:12.344 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:12.342+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.343440+0000) 2020-02-06T00:20:12.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:12.343+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.343964+0000) 2020-02-06T00:20:12.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:12.342+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.343781+0000) 2020-02-06T00:20:12.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:12.344+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.345367+0000) 2020-02-06T00:20:12.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:12.345+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.346289+0000) 2020-02-06T00:20:12.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:12.357+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170947 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:12.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:12.713+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.714374+0000) 2020-02-06T00:20:12.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:12.863+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:12.864032+0000) 2020-02-06T00:20:13.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:13.133+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:13.134131+0000) 2020-02-06T00:20:14.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:14.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:14.014413+0000) 2020-02-06T00:20:14.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:14.013+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:14.014610+0000) 2020-02-06T00:20:16.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:16.348+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:16.349909+0000) 2020-02-06T00:20:16.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:16.989+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:16.990877+0000) 2020-02-06T00:20:17.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:17.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.253320+0000) 2020-02-06T00:20:17.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:17.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.253360+0000) 2020-02-06T00:20:17.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:17.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.253357+0000) 2020-02-06T00:20:17.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:17.363+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.364144+0000) 2020-02-06T00:20:17.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:17.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.364738+0000) 2020-02-06T00:20:17.366 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:17.364+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.365239+0000) 2020-02-06T00:20:17.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:17.364+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.365970+0000) 2020-02-06T00:20:17.367 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:17.366+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.366621+0000) 2020-02-06T00:20:17.368 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:17.365+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.366843+0000) 2020-02-06T00:20:17.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:17.366+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.367127+0000) 2020-02-06T00:20:17.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:17.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.367473+0000) 2020-02-06T00:20:17.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:17.367+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.368160+0000) 2020-02-06T00:20:17.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:17.368+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.368656+0000) 2020-02-06T00:20:17.370 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:17.367+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.368886+0000) 2020-02-06T00:20:17.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:17.368+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.369246+0000) 2020-02-06T00:20:17.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:17.369+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.369796+0000) 2020-02-06T00:20:17.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:17.377+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:17.378501+0000) 2020-02-06T00:20:17.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:17.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 170989 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:19.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:19.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:19.252038+0000) 2020-02-06T00:20:19.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:19.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:19.635137+0000) 2020-02-06T00:20:20.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:20.482+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:20.482967+0000) 2020-02-06T00:20:20.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:20.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:20.743138+0000) 2020-02-06T00:20:22.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:22.386+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.387126+0000) 2020-02-06T00:20:22.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:22.388+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.389061+0000) 2020-02-06T00:20:22.390 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:22.388+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.389491+0000) 2020-02-06T00:20:22.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:22.388+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.389951+0000) 2020-02-06T00:20:22.391 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:22.389+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.390489+0000) 2020-02-06T00:20:22.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:22.389+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.390909+0000) 2020-02-06T00:20:22.392 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:22.391+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.391502+0000) 2020-02-06T00:20:22.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:22.391+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.392172+0000) 2020-02-06T00:20:22.393 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:22.391+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.391894+0000) 2020-02-06T00:20:22.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:22.392+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.392929+0000) 2020-02-06T00:20:22.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:22.392+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.393306+0000) 2020-02-06T00:20:22.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:22.393+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.393844+0000) 2020-02-06T00:20:22.395 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:22.394+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.394633+0000) 2020-02-06T00:20:22.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:22.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.397177+0000) 2020-02-06T00:20:22.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:22.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:22.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:22.721+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.722794+0000) 2020-02-06T00:20:22.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:22.864+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:22.864262+0000) 2020-02-06T00:20:23.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:23.133+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:23.134343+0000) 2020-02-06T00:20:24.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:24.013+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:24.014639+0000) 2020-02-06T00:20:24.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:24.013+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:24.014800+0000) 2020-02-06T00:20:26.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:26.349+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:26.350147+0000) 2020-02-06T00:20:26.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:26.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:26.991140+0000) 2020-02-06T00:20:27.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:27.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.253620+0000) 2020-02-06T00:20:27.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:27.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.253685+0000) 2020-02-06T00:20:27.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:27.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.253690+0000) 2020-02-06T00:20:27.413 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:27.411+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.412153+0000) 2020-02-06T00:20:27.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:27.411+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.412648+0000) 2020-02-06T00:20:27.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:27.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.413543+0000) 2020-02-06T00:20:27.415 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:27.412+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.414010+0000) 2020-02-06T00:20:27.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:27.413+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.414409+0000) 2020-02-06T00:20:27.416 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:27.413+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.415261+0000) 2020-02-06T00:20:27.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:27.414+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.415559+0000) 2020-02-06T00:20:27.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:27.414+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.416263+0000) 2020-02-06T00:20:27.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:27.414+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.416322+0000) 2020-02-06T00:20:27.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:27.415+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.416863+0000) 2020-02-06T00:20:27.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:27.415+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.417211+0000) 2020-02-06T00:20:27.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:27.416+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.417751+0000) 2020-02-06T00:20:27.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:27.416+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.418299+0000) 2020-02-06T00:20:27.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:27.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:27.423374+0000) 2020-02-06T00:20:27.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:27.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171069 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:29.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:29.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:29.252298+0000) 2020-02-06T00:20:29.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:29.634+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:29.635397+0000) 2020-02-06T00:20:30.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:30.481+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:30.483174+0000) 2020-02-06T00:20:30.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:30.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:30.743335+0000) 2020-02-06T00:20:32.439 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:32.437+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.438170+0000) 2020-02-06T00:20:32.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:32.437+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.438792+0000) 2020-02-06T00:20:32.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:32.438+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.439514+0000) 2020-02-06T00:20:32.441 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:32.439+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.440290+0000) 2020-02-06T00:20:32.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:32.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.440806+0000) 2020-02-06T00:20:32.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:32.440+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.441427+0000) 2020-02-06T00:20:32.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:32.440+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.441600+0000) 2020-02-06T00:20:32.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:32.440+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.442304+0000) 2020-02-06T00:20:32.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:32.441+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.442980+0000) 2020-02-06T00:20:32.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:32.441+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.442986+0000) 2020-02-06T00:20:32.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:32.442+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.444020+0000) 2020-02-06T00:20:32.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:32.442+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.443949+0000) 2020-02-06T00:20:32.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:32.443+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.444648+0000) 2020-02-06T00:20:32.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:32.447+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.448819+0000) 2020-02-06T00:20:32.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:32.456+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:32.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:32.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.730154+0000) 2020-02-06T00:20:32.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:32.863+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:32.864528+0000) 2020-02-06T00:20:33.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:33.133+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:33.134548+0000) 2020-02-06T00:20:34.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:34.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:34.014887+0000) 2020-02-06T00:20:34.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:34.014+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:34.015012+0000) 2020-02-06T00:20:36.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:36.349+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:36.350383+0000) 2020-02-06T00:20:36.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:36.990+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:36.991377+0000) 2020-02-06T00:20:37.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:37.252+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.253878+0000) 2020-02-06T00:20:37.255 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:37.252+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.253974+0000) 2020-02-06T00:20:37.255 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:37.252+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.253958+0000) 2020-02-06T00:20:37.464 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:37.462+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.463233+0000) 2020-02-06T00:20:37.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:37.463+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.464749+0000) 2020-02-06T00:20:37.466 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:37.463+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.465147+0000) 2020-02-06T00:20:37.467 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:37.464+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.465630+0000) 2020-02-06T00:20:37.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:37.464+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.466045+0000) 2020-02-06T00:20:37.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:37.465+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.466613+0000) 2020-02-06T00:20:37.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:37.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.466740+0000) 2020-02-06T00:20:37.469 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:37.465+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.466857+0000) 2020-02-06T00:20:37.469 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:37.466+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.467397+0000) 2020-02-06T00:20:37.469 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:37.466+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.467738+0000) 2020-02-06T00:20:37.470 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:37.466+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.467776+0000) 2020-02-06T00:20:37.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:37.466+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.468336+0000) 2020-02-06T00:20:37.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:37.467+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.468422+0000) 2020-02-06T00:20:37.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:37.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:37.472383+0000) 2020-02-06T00:20:37.484 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:37.481+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:37.651 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:20:37.657 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:20:39.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:39.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:39.252556+0000) 2020-02-06T00:20:39.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:39.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:39.635637+0000) 2020-02-06T00:20:40.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:40.482+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:40.483457+0000) 2020-02-06T00:20:40.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:40.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:40.743574+0000) 2020-02-06T00:20:42.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:42.486+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.487324+0000) 2020-02-06T00:20:42.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:42.486+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.487915+0000) 2020-02-06T00:20:42.491 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:42.488+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.489860+0000) 2020-02-06T00:20:42.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:42.489+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.490463+0000) 2020-02-06T00:20:42.492 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:42.490+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.491115+0000) 2020-02-06T00:20:42.492 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:42.489+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.491086+0000) 2020-02-06T00:20:42.493 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:42.490+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.492249+0000) 2020-02-06T00:20:42.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:42.491+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.492835+0000) 2020-02-06T00:20:42.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:42.492+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.493162+0000) 2020-02-06T00:20:42.495 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:42.492+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.494000+0000) 2020-02-06T00:20:42.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:42.493+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.494867+0000) 2020-02-06T00:20:42.496 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:42.493+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.494985+0000) 2020-02-06T00:20:42.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:42.493+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.494608+0000) 2020-02-06T00:20:42.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:42.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.500003+0000) 2020-02-06T00:20:42.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:42.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:42.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:42.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.738555+0000) 2020-02-06T00:20:42.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:42.863+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:42.864768+0000) 2020-02-06T00:20:43.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:43.134+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:43.134760+0000) 2020-02-06T00:20:44.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:44.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:44.015134+0000) 2020-02-06T00:20:44.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:44.014+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:44.015255+0000) 2020-02-06T00:20:46.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:46.349+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:46.350571+0000) 2020-02-06T00:20:46.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:46.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:46.991602+0000) 2020-02-06T00:20:47.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:47.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.254125+0000) 2020-02-06T00:20:47.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:47.253+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.254219+0000) 2020-02-06T00:20:47.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:47.253+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.254298+0000) 2020-02-06T00:20:47.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:47.511+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.512807+0000) 2020-02-06T00:20:47.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:47.512+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.514344+0000) 2020-02-06T00:20:47.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:47.513+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.514759+0000) 2020-02-06T00:20:47.519 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:47.514+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.516207+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:47.515+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.516915+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:47.509+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.510763+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:47.512+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.513206+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:47.512+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.513966+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:47.514+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.515196+0000) 2020-02-06T00:20:47.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:47.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.515745+0000) 2020-02-06T00:20:47.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:47.515+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.516606+0000) 2020-02-06T00:20:47.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:47.516+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.517438+0000) 2020-02-06T00:20:47.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:47.516+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.518364+0000) 2020-02-06T00:20:47.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:47.522+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:47.523823+0000) 2020-02-06T00:20:47.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:47.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:49.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:49.251+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:49.252797+0000) 2020-02-06T00:20:49.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:49.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:49.635876+0000) 2020-02-06T00:20:50.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:50.482+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:50.483702+0000) 2020-02-06T00:20:50.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:50.742+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:50.743806+0000) 2020-02-06T00:20:52.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:52.535+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.536423+0000) 2020-02-06T00:20:52.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:52.536+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.538194+0000) 2020-02-06T00:20:52.540 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:52.537+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.538885+0000) 2020-02-06T00:20:52.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:52.538+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.539909+0000) 2020-02-06T00:20:52.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:52.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.540418+0000) 2020-02-06T00:20:52.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:52.539+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.540581+0000) 2020-02-06T00:20:52.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:52.539+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.540847+0000) 2020-02-06T00:20:52.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:52.540+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.541280+0000) 2020-02-06T00:20:52.543 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:52.540+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.541535+0000) 2020-02-06T00:20:52.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:52.540+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.542028+0000) 2020-02-06T00:20:52.543 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:52.541+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.542491+0000) 2020-02-06T00:20:52.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:52.542+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.543596+0000) 2020-02-06T00:20:52.545 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:52.542+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.544139+0000) 2020-02-06T00:20:52.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:52.546+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.547447+0000) 2020-02-06T00:20:52.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:52.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:52.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:52.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.747138+0000) 2020-02-06T00:20:52.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:52.864+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:52.865002+0000) 2020-02-06T00:20:53.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:53.134+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:53.134991+0000) 2020-02-06T00:20:54.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:54.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:54.015366+0000) 2020-02-06T00:20:54.017 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:54.015+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:54.015481+0000) 2020-02-06T00:20:56.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:56.350+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:56.350825+0000) 2020-02-06T00:20:56.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:56.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:56.991858+0000) 2020-02-06T00:20:57.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:57.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.254366+0000) 2020-02-06T00:20:57.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:57.253+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.254509+0000) 2020-02-06T00:20:57.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:57.253+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.254609+0000) 2020-02-06T00:20:57.565 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:20:57.561+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.562800+0000) 2020-02-06T00:20:57.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:20:57.562+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.563892+0000) 2020-02-06T00:20:57.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:57.564+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.565693+0000) 2020-02-06T00:20:57.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:20:57.564+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.566011+0000) 2020-02-06T00:20:57.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:20:57.565+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.566454+0000) 2020-02-06T00:20:57.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:20:57.565+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.566972+0000) 2020-02-06T00:20:57.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:20:57.565+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.567402+0000) 2020-02-06T00:20:57.569 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:57.566+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.568269+0000) 2020-02-06T00:20:57.569 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:20:57.567+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.568734+0000) 2020-02-06T00:20:57.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:20:57.567+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.569378+0000) 2020-02-06T00:20:57.570 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:20:57.568+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.569578+0000) 2020-02-06T00:20:57.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:20:57.569+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.570570+0000) 2020-02-06T00:20:57.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:57.569+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.570648+0000) 2020-02-06T00:20:57.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:57.570+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:57.571344+0000) 2020-02-06T00:20:57.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:20:57.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171306 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:20:59.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:20:59.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:59.253070+0000) 2020-02-06T00:20:59.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:20:59.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:20:59.636146+0000) 2020-02-06T00:21:00.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:00.483+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:00.483951+0000) 2020-02-06T00:21:00.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:00.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:00.744036+0000) 2020-02-06T00:21:02.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:02.589+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.589654+0000) 2020-02-06T00:21:02.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:02.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.590638+0000) 2020-02-06T00:21:02.592 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:02.590+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.590768+0000) 2020-02-06T00:21:02.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:02.590+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.591479+0000) 2020-02-06T00:21:02.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:02.590+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.591762+0000) 2020-02-06T00:21:02.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:02.590+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.592210+0000) 2020-02-06T00:21:02.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:02.592+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.592430+0000) 2020-02-06T00:21:02.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:02.591+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.593086+0000) 2020-02-06T00:21:02.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:02.593+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.593621+0000) 2020-02-06T00:21:02.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:02.593+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.593556+0000) 2020-02-06T00:21:02.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:02.592+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.594026+0000) 2020-02-06T00:21:02.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:02.592+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.594284+0000) 2020-02-06T00:21:02.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:02.594+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.594129+0000) 2020-02-06T00:21:02.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:02.598+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.598139+0000) 2020-02-06T00:21:02.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:02.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:02.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:02.755+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.755530+0000) 2020-02-06T00:21:02.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:02.864+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:02.865251+0000) 2020-02-06T00:21:03.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:03.135+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:03.135196+0000) 2020-02-06T00:21:04.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:04.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:04.015612+0000) 2020-02-06T00:21:04.017 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:04.015+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:04.015699+0000) 2020-02-06T00:21:06.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:06.350+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:06.351062+0000) 2020-02-06T00:21:06.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:06.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:06.992120+0000) 2020-02-06T00:21:07.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:07.253+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.254576+0000) 2020-02-06T00:21:07.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:07.253+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.254719+0000) 2020-02-06T00:21:07.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:07.253+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.254872+0000) 2020-02-06T00:21:07.613 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:07.611+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.612188+0000) 2020-02-06T00:21:07.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:07.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.612698+0000) 2020-02-06T00:21:07.614 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:07.612+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.613164+0000) 2020-02-06T00:21:07.614 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:07.612+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.613572+0000) 2020-02-06T00:21:07.615 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:07.613+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.614420+0000) 2020-02-06T00:21:07.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:07.613+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.615019+0000) 2020-02-06T00:21:07.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:07.613+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.615376+0000) 2020-02-06T00:21:07.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:07.614+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.616047+0000) 2020-02-06T00:21:07.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:07.615+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.616015+0000) 2020-02-06T00:21:07.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:07.615+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.616219+0000) 2020-02-06T00:21:07.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:07.615+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.616599+0000) 2020-02-06T00:21:07.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:07.615+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.616522+0000) 2020-02-06T00:21:07.619 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:07.615+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.617343+0000) 2020-02-06T00:21:07.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:07.620+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:07.621988+0000) 2020-02-06T00:21:07.632 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:07.630+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:07.808 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:21:07.814 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:21:09.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:09.252+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:09.253360+0000) 2020-02-06T00:21:09.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:09.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:09.636369+0000) 2020-02-06T00:21:10.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:10.483+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:10.484203+0000) 2020-02-06T00:21:10.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:10.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:10.744298+0000) 2020-02-06T00:21:12.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:12.636+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.637703+0000) 2020-02-06T00:21:12.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:12.637+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.638741+0000) 2020-02-06T00:21:12.640 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:12.637+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.639128+0000) 2020-02-06T00:21:12.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:12.638+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.639398+0000) 2020-02-06T00:21:12.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:12.638+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.639783+0000) 2020-02-06T00:21:12.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:12.639+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.640752+0000) 2020-02-06T00:21:12.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:12.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.640674+0000) 2020-02-06T00:21:12.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:12.639+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.640908+0000) 2020-02-06T00:21:12.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:12.639+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.641110+0000) 2020-02-06T00:21:12.643 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:12.640+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.641481+0000) 2020-02-06T00:21:12.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:12.640+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.642015+0000) 2020-02-06T00:21:12.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:12.640+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.642020+0000) 2020-02-06T00:21:12.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:12.641+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.642454+0000) 2020-02-06T00:21:12.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:12.646+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.647342+0000) 2020-02-06T00:21:12.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:12.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:12.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:12.763+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.764457+0000) 2020-02-06T00:21:12.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:12.864+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:12.865554+0000) 2020-02-06T00:21:13.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:13.134+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:13.135401+0000) 2020-02-06T00:21:14.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:14.014+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:14.015874+0000) 2020-02-06T00:21:14.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:14.014+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:14.015909+0000) 2020-02-06T00:21:16.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:16.350+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:16.351377+0000) 2020-02-06T00:21:16.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:16.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:16.992369+0000) 2020-02-06T00:21:17.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:17.254+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.254818+0000) 2020-02-06T00:21:17.256 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:17.254+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.254932+0000) 2020-02-06T00:21:17.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:17.254+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.255102+0000) 2020-02-06T00:21:17.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:17.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.659752+0000) 2020-02-06T00:21:17.662 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:17.659+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.660485+0000) 2020-02-06T00:21:17.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:17.659+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.660587+0000) 2020-02-06T00:21:17.662 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:17.660+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.661369+0000) 2020-02-06T00:21:17.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:17.660+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.661734+0000) 2020-02-06T00:21:17.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:17.661+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.662439+0000) 2020-02-06T00:21:17.664 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:17.662+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.663247+0000) 2020-02-06T00:21:17.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:17.662+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.663849+0000) 2020-02-06T00:21:17.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:17.662+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.664108+0000) 2020-02-06T00:21:17.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:17.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.664354+0000) 2020-02-06T00:21:17.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:17.663+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.665201+0000) 2020-02-06T00:21:17.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:17.663+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.665155+0000) 2020-02-06T00:21:17.667 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:17.664+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.665563+0000) 2020-02-06T00:21:17.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:17.667+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:17.668437+0000) 2020-02-06T00:21:17.680 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:17.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171466 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:19.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:19.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:19.253635+0000) 2020-02-06T00:21:19.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:19.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:19.636636+0000) 2020-02-06T00:21:20.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:20.484+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:20.484476+0000) 2020-02-06T00:21:20.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:20.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:20.744554+0000) 2020-02-06T00:21:22.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:22.683+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.684365+0000) 2020-02-06T00:21:22.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:22.684+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.685019+0000) 2020-02-06T00:21:22.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:22.684+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.685754+0000) 2020-02-06T00:21:22.687 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:22.685+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.686384+0000) 2020-02-06T00:21:22.688 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:22.686+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.687492+0000) 2020-02-06T00:21:22.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:22.686+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.688010+0000) 2020-02-06T00:21:22.689 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:22.687+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.688236+0000) 2020-02-06T00:21:22.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:22.688+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.688967+0000) 2020-02-06T00:21:22.694 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:22.689+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.689729+0000) 2020-02-06T00:21:22.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:22.689+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.690310+0000) 2020-02-06T00:21:22.695 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:22.688+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.689466+0000) 2020-02-06T00:21:22.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:22.688+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.689916+0000) 2020-02-06T00:21:22.695 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:22.689+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.690225+0000) 2020-02-06T00:21:22.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:22.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.694298+0000) 2020-02-06T00:21:22.705 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:22.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:22.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:22.772+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.773464+0000) 2020-02-06T00:21:22.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:22.865+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:22.865773+0000) 2020-02-06T00:21:23.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:23.134+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:23.135615+0000) 2020-02-06T00:21:24.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:24.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:24.016158+0000) 2020-02-06T00:21:24.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:24.015+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:24.016156+0000) 2020-02-06T00:21:26.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:26.350+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:26.351600+0000) 2020-02-06T00:21:26.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:26.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:26.992598+0000) 2020-02-06T00:21:27.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:27.254+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.255052+0000) 2020-02-06T00:21:27.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:27.254+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.255216+0000) 2020-02-06T00:21:27.257 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:27.254+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.255335+0000) 2020-02-06T00:21:27.713 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:27.710+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.711862+0000) 2020-02-06T00:21:27.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:27.711+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.712247+0000) 2020-02-06T00:21:27.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:27.711+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.712459+0000) 2020-02-06T00:21:27.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:27.712+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.713428+0000) 2020-02-06T00:21:27.715 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:27.713+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.713569+0000) 2020-02-06T00:21:27.716 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:27.714+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.714739+0000) 2020-02-06T00:21:27.716 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:27.714+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.714827+0000) 2020-02-06T00:21:27.717 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:27.714+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.715368+0000) 2020-02-06T00:21:27.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:27.714+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.715804+0000) 2020-02-06T00:21:27.717 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:27.716+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.716773+0000) 2020-02-06T00:21:27.718 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:27.716+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.716671+0000) 2020-02-06T00:21:27.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:27.716+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.717137+0000) 2020-02-06T00:21:27.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:27.718+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.719228+0000) 2020-02-06T00:21:27.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:27.718+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:27.719475+0000) 2020-02-06T00:21:27.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:27.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:29.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:29.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:29.253895+0000) 2020-02-06T00:21:29.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:29.635+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:29.636912+0000) 2020-02-06T00:21:30.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:30.484+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:30.484728+0000) 2020-02-06T00:21:30.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:30.743+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:30.744807+0000) 2020-02-06T00:21:32.737 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:32.732+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.734040+0000) 2020-02-06T00:21:32.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:32.734+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.734993+0000) 2020-02-06T00:21:32.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:32.735+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.735792+0000) 2020-02-06T00:21:32.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:32.736+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.737448+0000) 2020-02-06T00:21:32.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:32.736+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.737790+0000) 2020-02-06T00:21:32.740 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:32.737+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.738140+0000) 2020-02-06T00:21:32.741 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:32.738+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.739136+0000) 2020-02-06T00:21:32.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:32.737+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.738875+0000) 2020-02-06T00:21:32.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:32.738+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.739137+0000) 2020-02-06T00:21:32.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:32.738+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.739779+0000) 2020-02-06T00:21:32.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:32.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.740079+0000) 2020-02-06T00:21:32.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:32.740+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.741138+0000) 2020-02-06T00:21:32.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:32.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.741236+0000) 2020-02-06T00:21:32.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:32.741+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.742915+0000) 2020-02-06T00:21:32.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:32.753+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:32.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:32.781+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.781606+0000) 2020-02-06T00:21:32.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:32.865+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:32.865990+0000) 2020-02-06T00:21:33.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:33.135+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:33.135820+0000) 2020-02-06T00:21:34.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:34.015+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:34.016431+0000) 2020-02-06T00:21:34.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:34.015+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:34.016433+0000) 2020-02-06T00:21:36.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:36.351+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:36.351829+0000) 2020-02-06T00:21:36.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:36.991+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:36.992858+0000) 2020-02-06T00:21:37.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:37.254+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.255308+0000) 2020-02-06T00:21:37.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:37.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.255471+0000) 2020-02-06T00:21:37.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:37.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.255620+0000) 2020-02-06T00:21:37.759 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:37.757+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.758376+0000) 2020-02-06T00:21:37.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:37.758+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.759292+0000) 2020-02-06T00:21:37.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:37.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.760725+0000) 2020-02-06T00:21:37.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:37.760+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.761194+0000) 2020-02-06T00:21:37.762 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:37.760+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.761596+0000) 2020-02-06T00:21:37.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:37.761+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.762002+0000) 2020-02-06T00:21:37.763 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:37.761+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.762383+0000) 2020-02-06T00:21:37.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:37.762+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.763133+0000) 2020-02-06T00:21:37.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:37.763+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.763695+0000) 2020-02-06T00:21:37.765 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:37.763+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.763724+0000) 2020-02-06T00:21:37.765 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:37.763+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.764028+0000) 2020-02-06T00:21:37.766 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:37.764+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.764953+0000) 2020-02-06T00:21:37.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:37.766+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.767437+0000) 2020-02-06T00:21:37.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:37.766+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:37.767648+0000) 2020-02-06T00:21:37.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:37.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:37.947 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:21:37.954 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:21:39.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:39.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:39.254134+0000) 2020-02-06T00:21:39.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:39.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:39.637170+0000) 2020-02-06T00:21:40.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:40.484+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:40.484990+0000) 2020-02-06T00:21:40.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:40.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:40.745130+0000) 2020-02-06T00:21:42.784 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:42.781+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.783040+0000) 2020-02-06T00:21:42.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:42.782+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.783926+0000) 2020-02-06T00:21:42.785 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:42.783+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.784659+0000) 2020-02-06T00:21:42.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:42.784+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.785370+0000) 2020-02-06T00:21:42.787 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:42.784+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.785224+0000) 2020-02-06T00:21:42.787 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:42.785+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.785664+0000) 2020-02-06T00:21:42.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:42.785+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.786210+0000) 2020-02-06T00:21:42.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:42.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.786312+0000) 2020-02-06T00:21:42.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:42.786+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.786778+0000) 2020-02-06T00:21:42.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:42.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.788051+0000) 2020-02-06T00:21:42.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:42.786+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.787996+0000) 2020-02-06T00:21:42.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:42.788+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.788912+0000) 2020-02-06T00:21:42.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:42.791+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.793003+0000) 2020-02-06T00:21:42.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:42.791+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.793030+0000) 2020-02-06T00:21:42.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:42.792+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.793281+0000) 2020-02-06T00:21:42.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:42.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171667 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:42.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:42.865+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:42.866218+0000) 2020-02-06T00:21:43.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:43.135+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:43.136046+0000) 2020-02-06T00:21:44.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:44.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:44.016700+0000) 2020-02-06T00:21:44.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:44.016+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:44.016700+0000) 2020-02-06T00:21:46.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:46.351+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:46.352073+0000) 2020-02-06T00:21:46.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:46.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:46.993124+0000) 2020-02-06T00:21:47.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:47.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.255552+0000) 2020-02-06T00:21:47.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:47.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.255710+0000) 2020-02-06T00:21:47.257 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:47.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.255882+0000) 2020-02-06T00:21:47.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:47.808+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.810042+0000) 2020-02-06T00:21:47.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:47.810+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.810569+0000) 2020-02-06T00:21:47.813 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:47.811+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.811622+0000) 2020-02-06T00:21:47.813 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:47.811+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.811925+0000) 2020-02-06T00:21:47.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:47.811+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.812259+0000) 2020-02-06T00:21:47.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:47.811+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.812582+0000) 2020-02-06T00:21:47.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:47.812+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.813202+0000) 2020-02-06T00:21:47.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:47.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.813375+0000) 2020-02-06T00:21:47.815 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:47.813+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.814105+0000) 2020-02-06T00:21:47.815 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:47.813+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.814069+0000) 2020-02-06T00:21:47.816 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:47.814+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.815068+0000) 2020-02-06T00:21:47.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:47.814+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.814796+0000) 2020-02-06T00:21:47.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:47.814+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.815451+0000) 2020-02-06T00:21:47.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:47.819+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:47.821004+0000) 2020-02-06T00:21:47.830 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:47.828+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:49.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:49.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:49.254368+0000) 2020-02-06T00:21:49.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:49.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:49.637412+0000) 2020-02-06T00:21:50.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:50.485+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:50.485221+0000) 2020-02-06T00:21:50.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:50.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:50.745352+0000) 2020-02-06T00:21:52.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:52.793+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.793206+0000) 2020-02-06T00:21:52.836 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:52.834+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.834944+0000) 2020-02-06T00:21:52.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:52.834+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.835486+0000) 2020-02-06T00:21:52.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:52.835+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.836262+0000) 2020-02-06T00:21:52.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:52.835+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.836210+0000) 2020-02-06T00:21:52.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:52.836+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.836421+0000) 2020-02-06T00:21:52.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:52.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.837047+0000) 2020-02-06T00:21:52.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:52.836+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.837004+0000) 2020-02-06T00:21:52.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:52.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.836764+0000) 2020-02-06T00:21:52.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:52.836+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.837390+0000) 2020-02-06T00:21:52.840 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:52.837+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.837678+0000) 2020-02-06T00:21:52.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:52.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.838676+0000) 2020-02-06T00:21:52.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:52.837+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.838988+0000) 2020-02-06T00:21:52.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:52.839+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.839632+0000) 2020-02-06T00:21:52.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:52.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.843393+0000) 2020-02-06T00:21:52.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:52.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:52.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:52.866+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:52.866463+0000) 2020-02-06T00:21:53.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:53.136+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:53.136303+0000) 2020-02-06T00:21:54.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:54.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:54.016920+0000) 2020-02-06T00:21:54.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:54.016+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:54.016925+0000) 2020-02-06T00:21:56.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:56.352+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:56.352352+0000) 2020-02-06T00:21:56.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:56.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:56.993381+0000) 2020-02-06T00:21:57.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:57.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.255814+0000) 2020-02-06T00:21:57.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:57.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.255979+0000) 2020-02-06T00:21:57.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:57.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.256133+0000) 2020-02-06T00:21:57.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:21:57.858+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.858343+0000) 2020-02-06T00:21:57.860 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:21:57.858+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.858937+0000) 2020-02-06T00:21:57.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:21:57.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.859109+0000) 2020-02-06T00:21:57.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:21:57.859+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.859648+0000) 2020-02-06T00:21:57.861 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:21:57.859+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.859984+0000) 2020-02-06T00:21:57.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:21:57.860+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.860219+0000) 2020-02-06T00:21:57.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:21:57.860+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.860475+0000) 2020-02-06T00:21:57.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:57.860+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.860855+0000) 2020-02-06T00:21:57.862 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:21:57.860+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.861257+0000) 2020-02-06T00:21:57.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:57.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.861697+0000) 2020-02-06T00:21:57.863 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:21:57.861+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.861659+0000) 2020-02-06T00:21:57.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:21:57.861+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.861968+0000) 2020-02-06T00:21:57.864 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:57.862+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.862845+0000) 2020-02-06T00:21:57.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:57.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:57.867492+0000) 2020-02-06T00:21:57.879 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:21:57.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171786 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:21:59.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:21:59.254+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:59.254649+0000) 2020-02-06T00:21:59.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:21:59.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:21:59.637640+0000) 2020-02-06T00:22:00.486 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:00.484+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:00.485488+0000) 2020-02-06T00:22:00.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:00.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:00.745580+0000) 2020-02-06T00:22:02.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:02.800+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.801188+0000) 2020-02-06T00:22:02.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:02.865+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.866716+0000) 2020-02-06T00:22:02.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:02.883+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.884580+0000) 2020-02-06T00:22:02.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:02.884+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.885126+0000) 2020-02-06T00:22:02.887 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:02.884+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.885961+0000) 2020-02-06T00:22:02.887 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:02.885+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.886780+0000) 2020-02-06T00:22:02.888 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:02.885+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.887059+0000) 2020-02-06T00:22:02.889 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:02.886+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.887988+0000) 2020-02-06T00:22:02.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:02.887+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.888324+0000) 2020-02-06T00:22:02.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:02.887+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.888781+0000) 2020-02-06T00:22:02.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:02.886+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.888102+0000) 2020-02-06T00:22:02.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:02.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.889430+0000) 2020-02-06T00:22:02.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:02.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.889755+0000) 2020-02-06T00:22:02.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:02.887+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.889370+0000) 2020-02-06T00:22:02.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:02.888+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.890008+0000) 2020-02-06T00:22:02.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:02.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:02.896395+0000) 2020-02-06T00:22:02.906 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:02.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:03.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:03.135+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:03.136522+0000) 2020-02-06T00:22:04.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:04.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:04.017178+0000) 2020-02-06T00:22:04.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:04.016+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:04.017178+0000) 2020-02-06T00:22:06.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:06.351+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:06.352590+0000) 2020-02-06T00:22:06.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:06.992+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:06.993609+0000) 2020-02-06T00:22:07.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:07.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.256062+0000) 2020-02-06T00:22:07.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:07.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.256215+0000) 2020-02-06T00:22:07.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:07.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.256410+0000) 2020-02-06T00:22:07.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:07.909+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.910053+0000) 2020-02-06T00:22:07.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:07.909+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.910680+0000) 2020-02-06T00:22:07.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:07.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.911253+0000) 2020-02-06T00:22:07.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:07.910+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.911708+0000) 2020-02-06T00:22:07.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:07.911+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.913121+0000) 2020-02-06T00:22:07.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:07.912+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.913385+0000) 2020-02-06T00:22:07.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:07.912+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.913967+0000) 2020-02-06T00:22:07.916 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:07.913+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.915258+0000) 2020-02-06T00:22:07.916 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:07.914+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.915721+0000) 2020-02-06T00:22:07.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:07.914+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.915965+0000) 2020-02-06T00:22:07.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:07.915+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.916319+0000) 2020-02-06T00:22:07.918 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:07.915+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.916491+0000) 2020-02-06T00:22:07.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:07.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.916941+0000) 2020-02-06T00:22:07.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:07.919+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:07.920370+0000) 2020-02-06T00:22:07.931 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:07.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:08.087 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:22:08.092 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:22:09.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:09.253+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:09.254918+0000) 2020-02-06T00:22:09.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:09.636+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:09.637868+0000) 2020-02-06T00:22:10.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:10.484+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:10.485742+0000) 2020-02-06T00:22:10.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:10.744+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:10.745827+0000) 2020-02-06T00:22:12.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:12.808+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.809236+0000) 2020-02-06T00:22:12.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:12.866+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.866986+0000) 2020-02-06T00:22:12.936 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:12.933+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.934893+0000) 2020-02-06T00:22:12.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:12.935+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.936263+0000) 2020-02-06T00:22:12.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:12.935+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.936787+0000) 2020-02-06T00:22:12.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:12.935+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.936778+0000) 2020-02-06T00:22:12.938 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:12.936+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.937235+0000) 2020-02-06T00:22:12.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:12.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.937536+0000) 2020-02-06T00:22:12.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:12.937+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.938691+0000) 2020-02-06T00:22:12.940 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:12.938+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.939494+0000) 2020-02-06T00:22:12.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:12.939+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.940492+0000) 2020-02-06T00:22:12.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:12.938+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.940126+0000) 2020-02-06T00:22:12.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:12.939+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.940684+0000) 2020-02-06T00:22:12.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:12.939+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.940838+0000) 2020-02-06T00:22:12.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:12.939+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.941293+0000) 2020-02-06T00:22:12.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:12.944+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:12.945104+0000) 2020-02-06T00:22:12.955 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:12.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171909 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:13.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:13.135+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:13.136772+0000) 2020-02-06T00:22:14.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:14.016+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:14.017430+0000) 2020-02-06T00:22:14.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:14.016+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:14.017430+0000) 2020-02-06T00:22:16.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:16.352+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:16.352817+0000) 2020-02-06T00:22:16.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:16.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:16.993872+0000) 2020-02-06T00:22:17.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:17.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.256318+0000) 2020-02-06T00:22:17.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:17.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.256503+0000) 2020-02-06T00:22:17.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:17.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.256700+0000) 2020-02-06T00:22:17.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:17.960+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.961922+0000) 2020-02-06T00:22:17.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:17.961+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.962637+0000) 2020-02-06T00:22:17.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:17.962+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.963144+0000) 2020-02-06T00:22:17.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:17.962+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.963864+0000) 2020-02-06T00:22:17.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:17.962+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.964041+0000) 2020-02-06T00:22:17.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:17.963+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.964555+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:17.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.964958+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:17.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.965876+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:17.964+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.965523+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:17.964+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.965698+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:17.964+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.966323+0000) 2020-02-06T00:22:17.969 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:17.964+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.966015+0000) 2020-02-06T00:22:17.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:17.965+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.966805+0000) 2020-02-06T00:22:17.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:17.970+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:17.971676+0000) 2020-02-06T00:22:17.980 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:17.977+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171950 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:19.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:19.254+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:19.255202+0000) 2020-02-06T00:22:19.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:19.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:19.638141+0000) 2020-02-06T00:22:20.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:20.485+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:20.485992+0000) 2020-02-06T00:22:20.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:20.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:20.746145+0000) 2020-02-06T00:22:22.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:22.816+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.817153+0000) 2020-02-06T00:22:22.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:22.866+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.867230+0000) 2020-02-06T00:22:22.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:22.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.986788+0000) 2020-02-06T00:22:22.988 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:22.985+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.986997+0000) 2020-02-06T00:22:22.989 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:22.986+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.987736+0000) 2020-02-06T00:22:22.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:22.986+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.987657+0000) 2020-02-06T00:22:22.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:22.986+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.988317+0000) 2020-02-06T00:22:22.990 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:22.988+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.989322+0000) 2020-02-06T00:22:22.991 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:22.988+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.989819+0000) 2020-02-06T00:22:22.991 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:22.988+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.989729+0000) 2020-02-06T00:22:22.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:22.989+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.990552+0000) 2020-02-06T00:22:22.992 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:22.989+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.991251+0000) 2020-02-06T00:22:22.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:22.989+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.990452+0000) 2020-02-06T00:22:22.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:22.989+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.991058+0000) 2020-02-06T00:22:22.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:22.990+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.991218+0000) 2020-02-06T00:22:22.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:22.993+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:22.994683+0000) 2020-02-06T00:22:23.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:23.001+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 171992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:23.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:23.136+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:23.137026+0000) 2020-02-06T00:22:24.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:24.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:24.017718+0000) 2020-02-06T00:22:24.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:24.017+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:24.017715+0000) 2020-02-06T00:22:26.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:26.352+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:26.353110+0000) 2020-02-06T00:22:26.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:26.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:26.994125+0000) 2020-02-06T00:22:27.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:27.255+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:27.256555+0000) 2020-02-06T00:22:27.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:27.255+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:27.256743+0000) 2020-02-06T00:22:27.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:27.255+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:27.257059+0000) 2020-02-06T00:22:28.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:28.007+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.009045+0000) 2020-02-06T00:22:28.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:28.009+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.010763+0000) 2020-02-06T00:22:28.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:28.010+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.011209+0000) 2020-02-06T00:22:28.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:28.010+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.011694+0000) 2020-02-06T00:22:28.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:28.010+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.012158+0000) 2020-02-06T00:22:28.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:28.011+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.012381+0000) 2020-02-06T00:22:28.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:28.011+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.012645+0000) 2020-02-06T00:22:28.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:28.011+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.013003+0000) 2020-02-06T00:22:28.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:28.012+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.013568+0000) 2020-02-06T00:22:28.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:28.012+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.014238+0000) 2020-02-06T00:22:28.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:28.013+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.014564+0000) 2020-02-06T00:22:28.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:28.014+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.015381+0000) 2020-02-06T00:22:28.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:28.014+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.016167+0000) 2020-02-06T00:22:28.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:28.021+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:28.022959+0000) 2020-02-06T00:22:28.029 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:28.027+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:29.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:29.254+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:29.255496+0000) 2020-02-06T00:22:29.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:29.637+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:29.638382+0000) 2020-02-06T00:22:30.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:30.485+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:30.486229+0000) 2020-02-06T00:22:30.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:30.745+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:30.746417+0000) 2020-02-06T00:22:32.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:32.824+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:32.825407+0000) 2020-02-06T00:22:32.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:32.866+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:32.867497+0000) 2020-02-06T00:22:33.035 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:33.032+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.033418+0000) 2020-02-06T00:22:33.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:33.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.034028+0000) 2020-02-06T00:22:33.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:33.033+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.034915+0000) 2020-02-06T00:22:33.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:33.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.034995+0000) 2020-02-06T00:22:33.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:33.034+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.035776+0000) 2020-02-06T00:22:33.039 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:33.036+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.038016+0000) 2020-02-06T00:22:33.040 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:33.037+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.038532+0000) 2020-02-06T00:22:33.040 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:33.037+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.038618+0000) 2020-02-06T00:22:33.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:33.038+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.039606+0000) 2020-02-06T00:22:33.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:33.038+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.039596+0000) 2020-02-06T00:22:33.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:33.038+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.039933+0000) 2020-02-06T00:22:33.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:33.039+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.040395+0000) 2020-02-06T00:22:33.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:33.038+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.040370+0000) 2020-02-06T00:22:33.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:33.044+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.045258+0000) 2020-02-06T00:22:33.055 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:33.052+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:33.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:33.136+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:33.137307+0000) 2020-02-06T00:22:34.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:34.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:34.017991+0000) 2020-02-06T00:22:34.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:34.017+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:34.017991+0000) 2020-02-06T00:22:36.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:36.352+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:36.353378+0000) 2020-02-06T00:22:36.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:36.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:36.994357+0000) 2020-02-06T00:22:37.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:37.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:37.256788+0000) 2020-02-06T00:22:37.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:37.256+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:37.256948+0000) 2020-02-06T00:22:37.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:37.256+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:37.257298+0000) 2020-02-06T00:22:38.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:38.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.060567+0000) 2020-02-06T00:22:38.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:38.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.061020+0000) 2020-02-06T00:22:38.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:38.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.061269+0000) 2020-02-06T00:22:38.063 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:38.060+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.061413+0000) 2020-02-06T00:22:38.063 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:38.060+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.061833+0000) 2020-02-06T00:22:38.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:38.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.062162+0000) 2020-02-06T00:22:38.064 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:38.061+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.063058+0000) 2020-02-06T00:22:38.065 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:38.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.063971+0000) 2020-02-06T00:22:38.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:38.063+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.064865+0000) 2020-02-06T00:22:38.066 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:38.064+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.065223+0000) 2020-02-06T00:22:38.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:38.064+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.065975+0000) 2020-02-06T00:22:38.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:38.064+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.066155+0000) 2020-02-06T00:22:38.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:38.064+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.066406+0000) 2020-02-06T00:22:38.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:38.069+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:38.071042+0000) 2020-02-06T00:22:38.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:38.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:38.225 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:22:38.231 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:22:39.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:39.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:39.255761+0000) 2020-02-06T00:22:39.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:39.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:39.638616+0000) 2020-02-06T00:22:40.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:40.485+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:40.486500+0000) 2020-02-06T00:22:40.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:40.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:40.746650+0000) 2020-02-06T00:22:42.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:42.833+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:42.834177+0000) 2020-02-06T00:22:42.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:42.866+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:42.867699+0000) 2020-02-06T00:22:43.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:43.084+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.085415+0000) 2020-02-06T00:22:43.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:43.085+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.086478+0000) 2020-02-06T00:22:43.087 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:43.085+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.086633+0000) 2020-02-06T00:22:43.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:43.085+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.086836+0000) 2020-02-06T00:22:43.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:43.085+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.086755+0000) 2020-02-06T00:22:43.089 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:43.086+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.087606+0000) 2020-02-06T00:22:43.089 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:43.085+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.087404+0000) 2020-02-06T00:22:43.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:43.086+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.088160+0000) 2020-02-06T00:22:43.090 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:43.087+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.088410+0000) 2020-02-06T00:22:43.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:43.087+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.089039+0000) 2020-02-06T00:22:43.090 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:43.087+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.088636+0000) 2020-02-06T00:22:43.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:43.088+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.089675+0000) 2020-02-06T00:22:43.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:43.088+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.089576+0000) 2020-02-06T00:22:43.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:43.091+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.092898+0000) 2020-02-06T00:22:43.105 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:43.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172156 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:43.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:43.137+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:43.137520+0000) 2020-02-06T00:22:44.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:44.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:44.018237+0000) 2020-02-06T00:22:44.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:44.017+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:44.018238+0000) 2020-02-06T00:22:46.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:46.353+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:46.353607+0000) 2020-02-06T00:22:46.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:46.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:46.994614+0000) 2020-02-06T00:22:47.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:47.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:47.257041+0000) 2020-02-06T00:22:47.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:47.256+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:47.257187+0000) 2020-02-06T00:22:47.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:47.256+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:47.257572+0000) 2020-02-06T00:22:48.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:48.108+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.109777+0000) 2020-02-06T00:22:48.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:48.109+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.110812+0000) 2020-02-06T00:22:48.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:48.110+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.111810+0000) 2020-02-06T00:22:48.113 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:48.111+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.112314+0000) 2020-02-06T00:22:48.114 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:48.111+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.113194+0000) 2020-02-06T00:22:48.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:48.112+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.113561+0000) 2020-02-06T00:22:48.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:48.112+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.113721+0000) 2020-02-06T00:22:48.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:48.113+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.114701+0000) 2020-02-06T00:22:48.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:48.113+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.115085+0000) 2020-02-06T00:22:48.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:48.114+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.115483+0000) 2020-02-06T00:22:48.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:48.114+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.115512+0000) 2020-02-06T00:22:48.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:48.113+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.114809+0000) 2020-02-06T00:22:48.117 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:48.114+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.115860+0000) 2020-02-06T00:22:48.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:48.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:48.120474+0000) 2020-02-06T00:22:48.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:48.126+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172196 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:49.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:49.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:49.256054+0000) 2020-02-06T00:22:49.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:49.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:49.638868+0000) 2020-02-06T00:22:50.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:50.486+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:50.486754+0000) 2020-02-06T00:22:50.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:50.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:50.746886+0000) 2020-02-06T00:22:52.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:52.842+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:52.842162+0000) 2020-02-06T00:22:52.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:52.867+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:52.867951+0000) 2020-02-06T00:22:53.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:53.133+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.133937+0000) 2020-02-06T00:22:53.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:53.134+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.134790+0000) 2020-02-06T00:22:53.137 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:53.136+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.136474+0000) 2020-02-06T00:22:53.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:53.135+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.136384+0000) 2020-02-06T00:22:53.138 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:53.137+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.137244+0000) 2020-02-06T00:22:53.138 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:53.136+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.137107+0000) 2020-02-06T00:22:53.139 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:53.137+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.137632+0000) 2020-02-06T00:22:53.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:53.137+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.137982+0000) 2020-02-06T00:22:53.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:53.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.138182+0000) 2020-02-06T00:22:53.140 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:53.138+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.138579+0000) 2020-02-06T00:22:53.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:53.139+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.139714+0000) 2020-02-06T00:22:53.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:53.139+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.140059+0000) 2020-02-06T00:22:53.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:53.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.145468+0000) 2020-02-06T00:22:53.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:53.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:53.145750+0000) 2020-02-06T00:22:53.155 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:53.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172238 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:54.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:54.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:54.018495+0000) 2020-02-06T00:22:54.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:54.017+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:54.018495+0000) 2020-02-06T00:22:56.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:56.352+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:56.353871+0000) 2020-02-06T00:22:56.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:56.993+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:56.994888+0000) 2020-02-06T00:22:57.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:57.256+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:57.257314+0000) 2020-02-06T00:22:57.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:57.257+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:57.257492+0000) 2020-02-06T00:22:57.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:57.257+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:57.257815+0000) 2020-02-06T00:22:58.162 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:22:58.160+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.161261+0000) 2020-02-06T00:22:58.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:22:58.160+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.161806+0000) 2020-02-06T00:22:58.163 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:22:58.161+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.162228+0000) 2020-02-06T00:22:58.164 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:22:58.162+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.162913+0000) 2020-02-06T00:22:58.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:58.163+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.163559+0000) 2020-02-06T00:22:58.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:58.163+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.164635+0000) 2020-02-06T00:22:58.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:22:58.164+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.165047+0000) 2020-02-06T00:22:58.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:22:58.164+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.165452+0000) 2020-02-06T00:22:58.167 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:22:58.165+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.165683+0000) 2020-02-06T00:22:58.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:58.165+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.166246+0000) 2020-02-06T00:22:58.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:22:58.166+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.166784+0000) 2020-02-06T00:22:58.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:22:58.166+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.167373+0000) 2020-02-06T00:22:58.168 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:22:58.166+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.167276+0000) 2020-02-06T00:22:58.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:58.172+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:58.173212+0000) 2020-02-06T00:22:58.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:22:58.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172278 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:22:59.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:22:59.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:59.256311+0000) 2020-02-06T00:22:59.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:59.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:59.639142+0000) 2020-02-06T00:22:59.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:22:59.887+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:22:59.888193+0000) 2020-02-06T00:23:00.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:00.486+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:00.487009+0000) 2020-02-06T00:23:00.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:00.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:00.747116+0000) 2020-02-06T00:23:02.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:02.849+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:02.849978+0000) 2020-02-06T00:23:02.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:02.867+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:02.868181+0000) 2020-02-06T00:23:03.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:03.184+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.185532+0000) 2020-02-06T00:23:03.188 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:03.186+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.187298+0000) 2020-02-06T00:23:03.189 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:03.186+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.187688+0000) 2020-02-06T00:23:03.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:03.188+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.188521+0000) 2020-02-06T00:23:03.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:03.187+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.188395+0000) 2020-02-06T00:23:03.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:03.187+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.188742+0000) 2020-02-06T00:23:03.191 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:03.188+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.189118+0000) 2020-02-06T00:23:03.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:03.188+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.189667+0000) 2020-02-06T00:23:03.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:03.189+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.190402+0000) 2020-02-06T00:23:03.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:03.190+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.190856+0000) 2020-02-06T00:23:03.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:03.189+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.190916+0000) 2020-02-06T00:23:03.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:03.190+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.191307+0000) 2020-02-06T00:23:03.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:03.190+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.191351+0000) 2020-02-06T00:23:03.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:03.195+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:03.196634+0000) 2020-02-06T00:23:03.205 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:03.203+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172320 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:04.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:04.017+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:04.018700+0000) 2020-02-06T00:23:04.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:04.017+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:04.018737+0000) 2020-02-06T00:23:06.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:06.353+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:06.354149+0000) 2020-02-06T00:23:06.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:06.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:06.995098+0000) 2020-02-06T00:23:07.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:07.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:07.257622+0000) 2020-02-06T00:23:07.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:07.257+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:07.257757+0000) 2020-02-06T00:23:07.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:07.257+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:07.258085+0000) 2020-02-06T00:23:08.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:08.208+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.209693+0000) 2020-02-06T00:23:08.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:08.209+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.210183+0000) 2020-02-06T00:23:08.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:08.209+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.210777+0000) 2020-02-06T00:23:08.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:08.210+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.211794+0000) 2020-02-06T00:23:08.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:08.211+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.212764+0000) 2020-02-06T00:23:08.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:08.212+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.212570+0000) 2020-02-06T00:23:08.214 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:08.212+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.213285+0000) 2020-02-06T00:23:08.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:08.214+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.214583+0000) 2020-02-06T00:23:08.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:08.214+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.214564+0000) 2020-02-06T00:23:08.216 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:08.214+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.215323+0000) 2020-02-06T00:23:08.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:08.214+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.215753+0000) 2020-02-06T00:23:08.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:08.214+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.215932+0000) 2020-02-06T00:23:08.217 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:08.215+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.216047+0000) 2020-02-06T00:23:08.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:08.220+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:08.221927+0000) 2020-02-06T00:23:08.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:08.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:08.382 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:23:08.388 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:23:09.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:09.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:09.256558+0000) 2020-02-06T00:23:09.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:09.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:09.639379+0000) 2020-02-06T00:23:09.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:09.887+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:09.888414+0000) 2020-02-06T00:23:10.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:10.487+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:10.487264+0000) 2020-02-06T00:23:10.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:10.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:10.747338+0000) 2020-02-06T00:23:12.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:12.857+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:12.858242+0000) 2020-02-06T00:23:12.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:12.868+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:12.868409+0000) 2020-02-06T00:23:13.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:13.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.232468+0000) 2020-02-06T00:23:13.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:13.232+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.233224+0000) 2020-02-06T00:23:13.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:13.233+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.234215+0000) 2020-02-06T00:23:13.236 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:13.234+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.235583+0000) 2020-02-06T00:23:13.237 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:13.235+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.236206+0000) 2020-02-06T00:23:13.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:13.236+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.236720+0000) 2020-02-06T00:23:13.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:13.236+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.237058+0000) 2020-02-06T00:23:13.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:13.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.237647+0000) 2020-02-06T00:23:13.239 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:13.236+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.238050+0000) 2020-02-06T00:23:13.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:13.237+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.238074+0000) 2020-02-06T00:23:13.240 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:13.237+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.238491+0000) 2020-02-06T00:23:13.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:13.237+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.238829+0000) 2020-02-06T00:23:13.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:13.238+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.239304+0000) 2020-02-06T00:23:13.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:13.244+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:13.245190+0000) 2020-02-06T00:23:13.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:13.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:14.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:14.018+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:14.018909+0000) 2020-02-06T00:23:14.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:14.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:14.018978+0000) 2020-02-06T00:23:16.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:16.353+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:16.354384+0000) 2020-02-06T00:23:16.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:16.994+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:16.995340+0000) 2020-02-06T00:23:17.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:17.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:17.257858+0000) 2020-02-06T00:23:17.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:17.257+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:17.258035+0000) 2020-02-06T00:23:17.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:17.257+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:17.258345+0000) 2020-02-06T00:23:18.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:18.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.257387+0000) 2020-02-06T00:23:18.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:18.257+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.258133+0000) 2020-02-06T00:23:18.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:18.257+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.258715+0000) 2020-02-06T00:23:18.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:18.258+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.258644+0000) 2020-02-06T00:23:18.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:18.259+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.260075+0000) 2020-02-06T00:23:18.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:18.260+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.260641+0000) 2020-02-06T00:23:18.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:18.260+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.260926+0000) 2020-02-06T00:23:18.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:18.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.260910+0000) 2020-02-06T00:23:18.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:18.260+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.261418+0000) 2020-02-06T00:23:18.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:18.261+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.262241+0000) 2020-02-06T00:23:18.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:18.262+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.262737+0000) 2020-02-06T00:23:18.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:18.261+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.262112+0000) 2020-02-06T00:23:18.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:18.261+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.262406+0000) 2020-02-06T00:23:18.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:18.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:18.267889+0000) 2020-02-06T00:23:18.278 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:18.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:19.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:19.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:19.256820+0000) 2020-02-06T00:23:19.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:19.638+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:19.639595+0000) 2020-02-06T00:23:19.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:19.888+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:19.888641+0000) 2020-02-06T00:23:20.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:20.487+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:20.487509+0000) 2020-02-06T00:23:20.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:20.746+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:20.747562+0000) 2020-02-06T00:23:22.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:22.865+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:22.866395+0000) 2020-02-06T00:23:22.869 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:22.868+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:22.868688+0000) 2020-02-06T00:23:23.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:23.282+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.283170+0000) 2020-02-06T00:23:23.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:23.282+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.283711+0000) 2020-02-06T00:23:23.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:23.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.283948+0000) 2020-02-06T00:23:23.285 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:23.283+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.284177+0000) 2020-02-06T00:23:23.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:23.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.284481+0000) 2020-02-06T00:23:23.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:23.283+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.285001+0000) 2020-02-06T00:23:23.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:23.284+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.285286+0000) 2020-02-06T00:23:23.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:23.285+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.285903+0000) 2020-02-06T00:23:23.287 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:23.285+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.286122+0000) 2020-02-06T00:23:23.288 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:23.285+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.286925+0000) 2020-02-06T00:23:23.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:23.286+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.287138+0000) 2020-02-06T00:23:23.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:23.287+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.287412+0000) 2020-02-06T00:23:23.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:23.287+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.287720+0000) 2020-02-06T00:23:23.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:23.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:23.291470+0000) 2020-02-06T00:23:23.301 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:23.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:24.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:24.018+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:24.019114+0000) 2020-02-06T00:23:24.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:24.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:24.019161+0000) 2020-02-06T00:23:26.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:26.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:26.354612+0000) 2020-02-06T00:23:26.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:26.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:26.995573+0000) 2020-02-06T00:23:27.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:27.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:27.258110+0000) 2020-02-06T00:23:27.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:27.258+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:27.258309+0000) 2020-02-06T00:23:27.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:27.258+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:27.258630+0000) 2020-02-06T00:23:28.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:28.305+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.306319+0000) 2020-02-06T00:23:28.308 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:28.305+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.306608+0000) 2020-02-06T00:23:28.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:28.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.307198+0000) 2020-02-06T00:23:28.308 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:28.306+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.307440+0000) 2020-02-06T00:23:28.309 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:28.306+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.307854+0000) 2020-02-06T00:23:28.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:28.306+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.308220+0000) 2020-02-06T00:23:28.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:28.307+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.308687+0000) 2020-02-06T00:23:28.310 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:28.308+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.309299+0000) 2020-02-06T00:23:28.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:28.307+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.309209+0000) 2020-02-06T00:23:28.312 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:28.309+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.310803+0000) 2020-02-06T00:23:28.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:28.308+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.310387+0000) 2020-02-06T00:23:28.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:28.309+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.311179+0000) 2020-02-06T00:23:28.313 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:28.310+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.311422+0000) 2020-02-06T00:23:28.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:28.314+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:28.315170+0000) 2020-02-06T00:23:28.327 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:28.324+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172524 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:29.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:29.255+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:29.257039+0000) 2020-02-06T00:23:29.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:29.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:29.639796+0000) 2020-02-06T00:23:29.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:29.888+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:29.888877+0000) 2020-02-06T00:23:30.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:30.486+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:30.487759+0000) 2020-02-06T00:23:30.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:30.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:30.747801+0000) 2020-02-06T00:23:32.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:32.867+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:32.868942+0000) 2020-02-06T00:23:32.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:32.874+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:32.874547+0000) 2020-02-06T00:23:33.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:33.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.332023+0000) 2020-02-06T00:23:33.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:33.331+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.332465+0000) 2020-02-06T00:23:33.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:33.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.332970+0000) 2020-02-06T00:23:33.335 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:33.332+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.333884+0000) 2020-02-06T00:23:33.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:33.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.334498+0000) 2020-02-06T00:23:33.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:33.333+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.334651+0000) 2020-02-06T00:23:33.336 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:33.333+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.334901+0000) 2020-02-06T00:23:33.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:33.333+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.335072+0000) 2020-02-06T00:23:33.337 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:33.334+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.335772+0000) 2020-02-06T00:23:33.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:33.334+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.336210+0000) 2020-02-06T00:23:33.338 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:33.335+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.336857+0000) 2020-02-06T00:23:33.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:33.335+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.336726+0000) 2020-02-06T00:23:33.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:33.336+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.337919+0000) 2020-02-06T00:23:33.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:33.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:33.340131+0000) 2020-02-06T00:23:33.351 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:33.348+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:34.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:34.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:34.019342+0000) 2020-02-06T00:23:34.021 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:34.018+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:34.019354+0000) 2020-02-06T00:23:36.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:36.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:36.354835+0000) 2020-02-06T00:23:36.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:36.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:36.995807+0000) 2020-02-06T00:23:37.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:37.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:37.258363+0000) 2020-02-06T00:23:37.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:37.257+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:37.258540+0000) 2020-02-06T00:23:37.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:37.257+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:37.258884+0000) 2020-02-06T00:23:38.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:38.355+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.356286+0000) 2020-02-06T00:23:38.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:38.355+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.356880+0000) 2020-02-06T00:23:38.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:38.356+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.358152+0000) 2020-02-06T00:23:38.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:38.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.358663+0000) 2020-02-06T00:23:38.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:38.357+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.359059+0000) 2020-02-06T00:23:38.360 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:38.358+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.359425+0000) 2020-02-06T00:23:38.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:38.359+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.360402+0000) 2020-02-06T00:23:38.362 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:38.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.360938+0000) 2020-02-06T00:23:38.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:38.360+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.361422+0000) 2020-02-06T00:23:38.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:38.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.360672+0000) 2020-02-06T00:23:38.363 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:38.360+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.361883+0000) 2020-02-06T00:23:38.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:38.360+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.362169+0000) 2020-02-06T00:23:38.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:38.360+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.362362+0000) 2020-02-06T00:23:38.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:38.366+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:38.367220+0000) 2020-02-06T00:23:38.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:38.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172607 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:38.526 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:23:38.532 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:23:39.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:39.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:39.257282+0000) 2020-02-06T00:23:39.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:39.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:39.640035+0000) 2020-02-06T00:23:39.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:39.888+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:39.889125+0000) 2020-02-06T00:23:40.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:40.487+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:40.487998+0000) 2020-02-06T00:23:40.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:40.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:40.748033+0000) 2020-02-06T00:23:42.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:42.868+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:42.869186+0000) 2020-02-06T00:23:42.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:42.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:42.882937+0000) 2020-02-06T00:23:43.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:43.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.380854+0000) 2020-02-06T00:23:43.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:43.380+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.381058+0000) 2020-02-06T00:23:43.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:43.380+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.381518+0000) 2020-02-06T00:23:43.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:43.380+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.381841+0000) 2020-02-06T00:23:43.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:43.380+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.382009+0000) 2020-02-06T00:23:43.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:43.381+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.382513+0000) 2020-02-06T00:23:43.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:43.381+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.382489+0000) 2020-02-06T00:23:43.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:43.382+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.383664+0000) 2020-02-06T00:23:43.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:43.382+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.384305+0000) 2020-02-06T00:23:43.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:43.382+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.384405+0000) 2020-02-06T00:23:43.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:43.383+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.384935+0000) 2020-02-06T00:23:43.386 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:43.384+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.385264+0000) 2020-02-06T00:23:43.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:43.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.386120+0000) 2020-02-06T00:23:43.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:43.391+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:43.392285+0000) 2020-02-06T00:23:43.400 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:43.397+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172647 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:44.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:44.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:44.019612+0000) 2020-02-06T00:23:44.021 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:44.019+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:44.019616+0000) 2020-02-06T00:23:46.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:46.355+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:46.355066+0000) 2020-02-06T00:23:46.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:46.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:46.996034+0000) 2020-02-06T00:23:47.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:47.257+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:47.258622+0000) 2020-02-06T00:23:47.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:47.257+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:47.258784+0000) 2020-02-06T00:23:47.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:47.257+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:47.259103+0000) 2020-02-06T00:23:48.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:48.404+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.404314+0000) 2020-02-06T00:23:48.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:48.405+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.405497+0000) 2020-02-06T00:23:48.407 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:48.406+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.406314+0000) 2020-02-06T00:23:48.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:48.406+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.406726+0000) 2020-02-06T00:23:48.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:48.405+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.407349+0000) 2020-02-06T00:23:48.410 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:48.407+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.408698+0000) 2020-02-06T00:23:48.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:48.406+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.408142+0000) 2020-02-06T00:23:48.411 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:48.410+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.410345+0000) 2020-02-06T00:23:48.412 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:48.409+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.410664+0000) 2020-02-06T00:23:48.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:48.409+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.410900+0000) 2020-02-06T00:23:48.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:48.409+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.411402+0000) 2020-02-06T00:23:48.413 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:48.411+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.411994+0000) 2020-02-06T00:23:48.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:48.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.412332+0000) 2020-02-06T00:23:48.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:48.415+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:48.416370+0000) 2020-02-06T00:23:48.424 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:48.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172687 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:49.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:49.256+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:49.257530+0000) 2020-02-06T00:23:49.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:49.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:49.640242+0000) 2020-02-06T00:23:49.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:49.888+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:49.889369+0000) 2020-02-06T00:23:50.489 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:50.487+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:50.488224+0000) 2020-02-06T00:23:50.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:50.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:50.748333+0000) 2020-02-06T00:23:52.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:52.868+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:52.869473+0000) 2020-02-06T00:23:52.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:52.890+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:52.891130+0000) 2020-02-06T00:23:53.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:53.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.429721+0000) 2020-02-06T00:23:53.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:53.428+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.429709+0000) 2020-02-06T00:23:53.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:53.429+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.430421+0000) 2020-02-06T00:23:53.432 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:53.430+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.431331+0000) 2020-02-06T00:23:53.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:53.430+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.431896+0000) 2020-02-06T00:23:53.433 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:53.431+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.432408+0000) 2020-02-06T00:23:53.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:53.431+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.433100+0000) 2020-02-06T00:23:53.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:53.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.433435+0000) 2020-02-06T00:23:53.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:53.432+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.434063+0000) 2020-02-06T00:23:53.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:53.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.434451+0000) 2020-02-06T00:23:53.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:53.433+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.434918+0000) 2020-02-06T00:23:53.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:53.434+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.435550+0000) 2020-02-06T00:23:53.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:53.434+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.435541+0000) 2020-02-06T00:23:53.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:53.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:53.441379+0000) 2020-02-06T00:23:53.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:53.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172730 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:54.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:54.018+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:54.019840+0000) 2020-02-06T00:23:54.021 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:54.018+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:54.019859+0000) 2020-02-06T00:23:56.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:56.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:56.355346+0000) 2020-02-06T00:23:56.997 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:56.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:56.996348+0000) 2020-02-06T00:23:57.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:57.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:57.258894+0000) 2020-02-06T00:23:57.260 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:57.258+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:57.259055+0000) 2020-02-06T00:23:57.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:57.258+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:57.259491+0000) 2020-02-06T00:23:58.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:23:58.455+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.456557+0000) 2020-02-06T00:23:58.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:23:58.455+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.457031+0000) 2020-02-06T00:23:58.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:23:58.456+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.457956+0000) 2020-02-06T00:23:58.460 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:58.457+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.458866+0000) 2020-02-06T00:23:58.460 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:23:58.458+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.459273+0000) 2020-02-06T00:23:58.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:23:58.458+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.459622+0000) 2020-02-06T00:23:58.461 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:23:58.458+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.460214+0000) 2020-02-06T00:23:58.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:58.459+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.460723+0000) 2020-02-06T00:23:58.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:58.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.461396+0000) 2020-02-06T00:23:58.463 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:23:58.461+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.462069+0000) 2020-02-06T00:23:58.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:23:58.460+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.462385+0000) 2020-02-06T00:23:58.464 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:23:58.461+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.462934+0000) 2020-02-06T00:23:58.464 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:23:58.462+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.463279+0000) 2020-02-06T00:23:58.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:58.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:58.467509+0000) 2020-02-06T00:23:58.476 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:23:58.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:23:59.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:23:59.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:59.257775+0000) 2020-02-06T00:23:59.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:59.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:59.640479+0000) 2020-02-06T00:23:59.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:23:59.888+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:23:59.889601+0000) 2020-02-06T00:24:00.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:00.487+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:00.488480+0000) 2020-02-06T00:24:00.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:00.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:00.748604+0000) 2020-02-06T00:24:02.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:02.869+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:02.869725+0000) 2020-02-06T00:24:02.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:02.900+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:02.901361+0000) 2020-02-06T00:24:03.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:03.479+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.480661+0000) 2020-02-06T00:24:03.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:03.480+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.481272+0000) 2020-02-06T00:24:03.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:03.480+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.481926+0000) 2020-02-06T00:24:03.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:03.481+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.482431+0000) 2020-02-06T00:24:03.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:03.481+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.483393+0000) 2020-02-06T00:24:03.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:03.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.483692+0000) 2020-02-06T00:24:03.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:03.482+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.484028+0000) 2020-02-06T00:24:03.486 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:03.483+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.484358+0000) 2020-02-06T00:24:03.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:03.483+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.484700+0000) 2020-02-06T00:24:03.487 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:03.484+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.485475+0000) 2020-02-06T00:24:03.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:03.484+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.485560+0000) 2020-02-06T00:24:03.487 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:03.485+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.486499+0000) 2020-02-06T00:24:03.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:03.485+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.486858+0000) 2020-02-06T00:24:03.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:03.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:03.491543+0000) 2020-02-06T00:24:03.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:03.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172814 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:04.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:04.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:04.020132+0000) 2020-02-06T00:24:04.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:04.019+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:04.020151+0000) 2020-02-06T00:24:06.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:06.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:06.355566+0000) 2020-02-06T00:24:06.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:06.995+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:06.996566+0000) 2020-02-06T00:24:07.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:07.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:07.259145+0000) 2020-02-06T00:24:07.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:07.258+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:07.259290+0000) 2020-02-06T00:24:07.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:07.258+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:07.259726+0000) 2020-02-06T00:24:08.504 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:08.502+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.503255+0000) 2020-02-06T00:24:08.505 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:08.503+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.504693+0000) 2020-02-06T00:24:08.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:08.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.505400+0000) 2020-02-06T00:24:08.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:08.504+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.505844+0000) 2020-02-06T00:24:08.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:08.504+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.506114+0000) 2020-02-06T00:24:08.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:08.505+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.506593+0000) 2020-02-06T00:24:08.508 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:08.506+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.507530+0000) 2020-02-06T00:24:08.509 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:08.507+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.508567+0000) 2020-02-06T00:24:08.510 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:08.507+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.509024+0000) 2020-02-06T00:24:08.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:08.507+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.508918+0000) 2020-02-06T00:24:08.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:08.508+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.509791+0000) 2020-02-06T00:24:08.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:08.508+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.510032+0000) 2020-02-06T00:24:08.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:08.508+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.510192+0000) 2020-02-06T00:24:08.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:08.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:08.515204+0000) 2020-02-06T00:24:08.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:08.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:08.653 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:24:08.659 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:24:09.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:09.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:09.258022+0000) 2020-02-06T00:24:09.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:09.639+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:09.640694+0000) 2020-02-06T00:24:09.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:09.889+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:09.889824+0000) 2020-02-06T00:24:10.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:10.488+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:10.488744+0000) 2020-02-06T00:24:10.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:10.747+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:10.748834+0000) 2020-02-06T00:24:12.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:12.869+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:12.869982+0000) 2020-02-06T00:24:12.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:12.909+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:12.909620+0000) 2020-02-06T00:24:13.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:13.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.528655+0000) 2020-02-06T00:24:13.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:13.528+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.529408+0000) 2020-02-06T00:24:13.531 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:13.528+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.530089+0000) 2020-02-06T00:24:13.532 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:13.529+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.530689+0000) 2020-02-06T00:24:13.532 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:13.530+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.531401+0000) 2020-02-06T00:24:13.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:13.530+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.532153+0000) 2020-02-06T00:24:13.534 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:13.531+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.533220+0000) 2020-02-06T00:24:13.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:13.532+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.533619+0000) 2020-02-06T00:24:13.535 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:13.532+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.533938+0000) 2020-02-06T00:24:13.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:13.533+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.534469+0000) 2020-02-06T00:24:13.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:13.533+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.535010+0000) 2020-02-06T00:24:13.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:13.533+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.534724+0000) 2020-02-06T00:24:13.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:13.533+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.535323+0000) 2020-02-06T00:24:13.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:13.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:13.542857+0000) 2020-02-06T00:24:13.548 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:13.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:14.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:14.019+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:14.020371+0000) 2020-02-06T00:24:14.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:14.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:14.020370+0000) 2020-02-06T00:24:16.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:16.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:16.355806+0000) 2020-02-06T00:24:16.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:16.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:16.996799+0000) 2020-02-06T00:24:17.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:17.258+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:17.259404+0000) 2020-02-06T00:24:17.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:17.259+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:17.259545+0000) 2020-02-06T00:24:17.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:17.259+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:17.259962+0000) 2020-02-06T00:24:18.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:18.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.551909+0000) 2020-02-06T00:24:18.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:18.551+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.552816+0000) 2020-02-06T00:24:18.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:18.552+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.553770+0000) 2020-02-06T00:24:18.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:18.553+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.554197+0000) 2020-02-06T00:24:18.557 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:18.554+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.555643+0000) 2020-02-06T00:24:18.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:18.555+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.556293+0000) 2020-02-06T00:24:18.557 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:18.555+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.556339+0000) 2020-02-06T00:24:18.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:18.556+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.556770+0000) 2020-02-06T00:24:18.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:18.557+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.557728+0000) 2020-02-06T00:24:18.559 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:18.557+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.558378+0000) 2020-02-06T00:24:18.560 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:18.558+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.559127+0000) 2020-02-06T00:24:18.561 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:18.559+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.559771+0000) 2020-02-06T00:24:18.561 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:18.559+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.559705+0000) 2020-02-06T00:24:18.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:18.561+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:18.562216+0000) 2020-02-06T00:24:18.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:18.572+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:19.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:19.257+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:19.258274+0000) 2020-02-06T00:24:19.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:19.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:19.640925+0000) 2020-02-06T00:24:19.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:19.889+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:19.890054+0000) 2020-02-06T00:24:20.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:20.488+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:20.488975+0000) 2020-02-06T00:24:20.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:20.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:20.749132+0000) 2020-02-06T00:24:22.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:22.869+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:22.870241+0000) 2020-02-06T00:24:22.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:22.917+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:22.918181+0000) 2020-02-06T00:24:23.578 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:23.575+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.576871+0000) 2020-02-06T00:24:23.579 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:23.577+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.578565+0000) 2020-02-06T00:24:23.580 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:23.578+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.579269+0000) 2020-02-06T00:24:23.581 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:23.579+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.580032+0000) 2020-02-06T00:24:23.581 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:23.579+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.580499+0000) 2020-02-06T00:24:23.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:23.580+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.581236+0000) 2020-02-06T00:24:23.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:23.580+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.581391+0000) 2020-02-06T00:24:23.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:23.581+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.582156+0000) 2020-02-06T00:24:23.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:23.581+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.582317+0000) 2020-02-06T00:24:23.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:23.582+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.583254+0000) 2020-02-06T00:24:23.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:23.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.584259+0000) 2020-02-06T00:24:23.585 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:23.584+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.584662+0000) 2020-02-06T00:24:23.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:23.584+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.585265+0000) 2020-02-06T00:24:23.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:23.589+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:23.590345+0000) 2020-02-06T00:24:23.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:23.598+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 172982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:24.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:24.020+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:24.020581+0000) 2020-02-06T00:24:24.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:24.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:24.020620+0000) 2020-02-06T00:24:26.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:26.354+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:26.356027+0000) 2020-02-06T00:24:26.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:26.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:26.997042+0000) 2020-02-06T00:24:27.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:27.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:27.259668+0000) 2020-02-06T00:24:27.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:27.259+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:27.259826+0000) 2020-02-06T00:24:27.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:27.259+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:27.260173+0000) 2020-02-06T00:24:28.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:28.603+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.604720+0000) 2020-02-06T00:24:28.607 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:28.605+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.606237+0000) 2020-02-06T00:24:28.608 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:28.605+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.606921+0000) 2020-02-06T00:24:28.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:28.607+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.607718+0000) 2020-02-06T00:24:28.609 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:28.606+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.608025+0000) 2020-02-06T00:24:28.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:28.608+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.608816+0000) 2020-02-06T00:24:28.610 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:28.608+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.609250+0000) 2020-02-06T00:24:28.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:28.608+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.609604+0000) 2020-02-06T00:24:28.612 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:28.608+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.609992+0000) 2020-02-06T00:24:28.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:28.609+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.609572+0000) 2020-02-06T00:24:28.612 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:28.610+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.610478+0000) 2020-02-06T00:24:28.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:28.609+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.610769+0000) 2020-02-06T00:24:28.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:28.614+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.615359+0000) 2020-02-06T00:24:28.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:28.614+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:28.615617+0000) 2020-02-06T00:24:28.624 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:28.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173022 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:29.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:29.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:29.258503+0000) 2020-02-06T00:24:29.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:29.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:29.641159+0000) 2020-02-06T00:24:29.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:29.889+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:29.890330+0000) 2020-02-06T00:24:30.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:30.488+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:30.489212+0000) 2020-02-06T00:24:30.750 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:30.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:30.749320+0000) 2020-02-06T00:24:32.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:32.870+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:32.870577+0000) 2020-02-06T00:24:32.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:32.926+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:32.926381+0000) 2020-02-06T00:24:33.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:33.627+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.628235+0000) 2020-02-06T00:24:33.630 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:33.627+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.628632+0000) 2020-02-06T00:24:33.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:33.628+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.629072+0000) 2020-02-06T00:24:33.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:33.628+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.629681+0000) 2020-02-06T00:24:33.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:33.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.629728+0000) 2020-02-06T00:24:33.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:33.629+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.630148+0000) 2020-02-06T00:24:33.632 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:33.630+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.631111+0000) 2020-02-06T00:24:33.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:33.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.632273+0000) 2020-02-06T00:24:33.633 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:33.631+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.632340+0000) 2020-02-06T00:24:33.634 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:33.632+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.633322+0000) 2020-02-06T00:24:33.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:33.633+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.634405+0000) 2020-02-06T00:24:33.635 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:33.633+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.634111+0000) 2020-02-06T00:24:33.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:33.634+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.635141+0000) 2020-02-06T00:24:33.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:33.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:33.640706+0000) 2020-02-06T00:24:33.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:33.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173066 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:34.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:34.020+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:34.020795+0000) 2020-02-06T00:24:34.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:34.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:34.020848+0000) 2020-02-06T00:24:36.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:36.355+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:36.356237+0000) 2020-02-06T00:24:36.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:36.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:36.997334+0000) 2020-02-06T00:24:37.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:37.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:37.259910+0000) 2020-02-06T00:24:37.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:37.259+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:37.260079+0000) 2020-02-06T00:24:37.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:37.260+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:37.260411+0000) 2020-02-06T00:24:38.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:38.652+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.653014+0000) 2020-02-06T00:24:38.656 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:38.655+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.655510+0000) 2020-02-06T00:24:38.657 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:38.655+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.655714+0000) 2020-02-06T00:24:38.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:38.656+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.656388+0000) 2020-02-06T00:24:38.658 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:38.657+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.657237+0000) 2020-02-06T00:24:38.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:38.656+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.656852+0000) 2020-02-06T00:24:38.659 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:38.657+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.658205+0000) 2020-02-06T00:24:38.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:38.658+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.658570+0000) 2020-02-06T00:24:38.660 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:38.658+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.659045+0000) 2020-02-06T00:24:38.660 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:38.658+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.659269+0000) 2020-02-06T00:24:38.661 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:38.659+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.659649+0000) 2020-02-06T00:24:38.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:38.659+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.660037+0000) 2020-02-06T00:24:38.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:38.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.665249+0000) 2020-02-06T00:24:38.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:38.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:38.665524+0000) 2020-02-06T00:24:38.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:38.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173105 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:38.788 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:24:38.795 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:24:39.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:39.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:39.258734+0000) 2020-02-06T00:24:39.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:39.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:39.641397+0000) 2020-02-06T00:24:39.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:39.890+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:39.890537+0000) 2020-02-06T00:24:40.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:40.489+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:40.489481+0000) 2020-02-06T00:24:40.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:40.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:40.749559+0000) 2020-02-06T00:24:42.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:42.870+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:42.870841+0000) 2020-02-06T00:24:42.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:42.933+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:42.934384+0000) 2020-02-06T00:24:43.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:43.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.681974+0000) 2020-02-06T00:24:43.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:43.682+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.682521+0000) 2020-02-06T00:24:43.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:43.683+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.683865+0000) 2020-02-06T00:24:43.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:43.683+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.684187+0000) 2020-02-06T00:24:43.696 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:43.684+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.684638+0000) 2020-02-06T00:24:43.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:43.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.684899+0000) 2020-02-06T00:24:43.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:43.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.680230+0000) 2020-02-06T00:24:43.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:43.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.684701+0000) 2020-02-06T00:24:43.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:43.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.691320+0000) 2020-02-06T00:24:43.697 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:43.679+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.680449+0000) 2020-02-06T00:24:43.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:43.680+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.681182+0000) 2020-02-06T00:24:43.697 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:43.681+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.682997+0000) 2020-02-06T00:24:43.698 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:43.682+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.683288+0000) 2020-02-06T00:24:43.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:43.683+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:43.684410+0000) 2020-02-06T00:24:43.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:43.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173148 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:44.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:44.019+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:44.021053+0000) 2020-02-06T00:24:44.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:44.019+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:44.021030+0000) 2020-02-06T00:24:46.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:46.355+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:46.356467+0000) 2020-02-06T00:24:46.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:46.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:46.997557+0000) 2020-02-06T00:24:47.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:47.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:47.260146+0000) 2020-02-06T00:24:47.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:47.260+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:47.260289+0000) 2020-02-06T00:24:47.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:47.260+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:47.260715+0000) 2020-02-06T00:24:48.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:48.706+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.706933+0000) 2020-02-06T00:24:48.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:48.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.707709+0000) 2020-02-06T00:24:48.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:48.707+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.708292+0000) 2020-02-06T00:24:48.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:48.707+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.708872+0000) 2020-02-06T00:24:48.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:48.708+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.709460+0000) 2020-02-06T00:24:48.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:48.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.709769+0000) 2020-02-06T00:24:48.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:48.709+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.710066+0000) 2020-02-06T00:24:48.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:48.709+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.710491+0000) 2020-02-06T00:24:48.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:48.711+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.711675+0000) 2020-02-06T00:24:48.713 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:48.711+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.712296+0000) 2020-02-06T00:24:48.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:48.711+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.712094+0000) 2020-02-06T00:24:48.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:48.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.712451+0000) 2020-02-06T00:24:48.714 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:48.712+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.712741+0000) 2020-02-06T00:24:48.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:48.715+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:48.716485+0000) 2020-02-06T00:24:48.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:48.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173186 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:49.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:49.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:49.258968+0000) 2020-02-06T00:24:49.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:49.640+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:49.641612+0000) 2020-02-06T00:24:49.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:49.889+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:49.890768+0000) 2020-02-06T00:24:50.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:50.489+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:50.489727+0000) 2020-02-06T00:24:50.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:50.748+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:50.749812+0000) 2020-02-06T00:24:52.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:52.870+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:52.871113+0000) 2020-02-06T00:24:52.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:52.941+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:52.942429+0000) 2020-02-06T00:24:53.733 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:53.730+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.731239+0000) 2020-02-06T00:24:53.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:53.730+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.731533+0000) 2020-02-06T00:24:53.733 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:53.728+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.730062+0000) 2020-02-06T00:24:53.734 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:53.729+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.731003+0000) 2020-02-06T00:24:53.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:53.731+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.732311+0000) 2020-02-06T00:24:53.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:53.731+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.732981+0000) 2020-02-06T00:24:53.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:53.732+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.733553+0000) 2020-02-06T00:24:53.736 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:53.732+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.733934+0000) 2020-02-06T00:24:53.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:53.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.733752+0000) 2020-02-06T00:24:53.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:53.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.734959+0000) 2020-02-06T00:24:53.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:53.733+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.734436+0000) 2020-02-06T00:24:53.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:53.734+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.735911+0000) 2020-02-06T00:24:53.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:53.734+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.736208+0000) 2020-02-06T00:24:53.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:53.739+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:53.740833+0000) 2020-02-06T00:24:53.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:53.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173230 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:54.022 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:54.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:54.021309+0000) 2020-02-06T00:24:54.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:54.020+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:54.021287+0000) 2020-02-06T00:24:56.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:56.355+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:56.356724+0000) 2020-02-06T00:24:56.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:56.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:56.997826+0000) 2020-02-06T00:24:57.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:57.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:57.260409+0000) 2020-02-06T00:24:57.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:57.259+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:57.260542+0000) 2020-02-06T00:24:57.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:57.259+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:57.260951+0000) 2020-02-06T00:24:58.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:24:58.754+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.755897+0000) 2020-02-06T00:24:58.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:24:58.753+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.755268+0000) 2020-02-06T00:24:58.758 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:24:58.755+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.756315+0000) 2020-02-06T00:24:58.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:24:58.755+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.757253+0000) 2020-02-06T00:24:58.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:24:58.756+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.757858+0000) 2020-02-06T00:24:58.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:58.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.758564+0000) 2020-02-06T00:24:58.760 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:24:58.757+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.758925+0000) 2020-02-06T00:24:58.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:58.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.759249+0000) 2020-02-06T00:24:58.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:24:58.758+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.759663+0000) 2020-02-06T00:24:58.761 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:24:58.758+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.759882+0000) 2020-02-06T00:24:58.761 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:24:58.759+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.760611+0000) 2020-02-06T00:24:58.762 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:58.760+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.761627+0000) 2020-02-06T00:24:58.763 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:24:58.760+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.761694+0000) 2020-02-06T00:24:58.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:58.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:58.767687+0000) 2020-02-06T00:24:58.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:24:58.773+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173269 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:24:59.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:24:59.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:59.259190+0000) 2020-02-06T00:24:59.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:59.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:59.641821+0000) 2020-02-06T00:24:59.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:24:59.890+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:24:59.891002+0000) 2020-02-06T00:25:00.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:00.489+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:00.489982+0000) 2020-02-06T00:25:00.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:00.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:00.750049+0000) 2020-02-06T00:25:02.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:02.870+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:02.871370+0000) 2020-02-06T00:25:02.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:02.948+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:02.949452+0000) 2020-02-06T00:25:03.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:03.780+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.781556+0000) 2020-02-06T00:25:03.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:03.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.782240+0000) 2020-02-06T00:25:03.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:03.781+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.782867+0000) 2020-02-06T00:25:03.784 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:03.781+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.782733+0000) 2020-02-06T00:25:03.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:03.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.783184+0000) 2020-02-06T00:25:03.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:03.782+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.783564+0000) 2020-02-06T00:25:03.785 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:03.782+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.784154+0000) 2020-02-06T00:25:03.786 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:03.784+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.785370+0000) 2020-02-06T00:25:03.787 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:03.783+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.784820+0000) 2020-02-06T00:25:03.787 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:03.783+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.785049+0000) 2020-02-06T00:25:03.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:03.784+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.786016+0000) 2020-02-06T00:25:03.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:03.785+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.786573+0000) 2020-02-06T00:25:03.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:03.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.786504+0000) 2020-02-06T00:25:03.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:03.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:03.791568+0000) 2020-02-06T00:25:03.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:03.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:04.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:04.020+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:04.021563+0000) 2020-02-06T00:25:04.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:04.020+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:04.021558+0000) 2020-02-06T00:25:06.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:06.355+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:06.356974+0000) 2020-02-06T00:25:06.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:06.996+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:06.998030+0000) 2020-02-06T00:25:07.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:07.259+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:07.260764+0000) 2020-02-06T00:25:07.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:07.259+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:07.260780+0000) 2020-02-06T00:25:07.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:07.259+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:07.261168+0000) 2020-02-06T00:25:08.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:08.805+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.806321+0000) 2020-02-06T00:25:08.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:08.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.807258+0000) 2020-02-06T00:25:08.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:08.806+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.807777+0000) 2020-02-06T00:25:08.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:08.806+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.808280+0000) 2020-02-06T00:25:08.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:08.807+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.808511+0000) 2020-02-06T00:25:08.811 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:08.807+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.809294+0000) 2020-02-06T00:25:08.811 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:08.809+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.810145+0000) 2020-02-06T00:25:08.812 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:08.808+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.810236+0000) 2020-02-06T00:25:08.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:08.809+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.810953+0000) 2020-02-06T00:25:08.813 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:08.809+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.811327+0000) 2020-02-06T00:25:08.813 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:08.810+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.812377+0000) 2020-02-06T00:25:08.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:08.811+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.812476+0000) 2020-02-06T00:25:08.814 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:08.811+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.812813+0000) 2020-02-06T00:25:08.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:08.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:08.816620+0000) 2020-02-06T00:25:08.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:08.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:08.941 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:25:08.947 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:25:09.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:09.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:09.259423+0000) 2020-02-06T00:25:09.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:09.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:09.642030+0000) 2020-02-06T00:25:09.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:09.890+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:09.891213+0000) 2020-02-06T00:25:10.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:10.489+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:10.490281+0000) 2020-02-06T00:25:10.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:10.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:10.750272+0000) 2020-02-06T00:25:12.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:12.870+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:12.871651+0000) 2020-02-06T00:25:12.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:12.957+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:12.957776+0000) 2020-02-06T00:25:13.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:13.829+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.830153+0000) 2020-02-06T00:25:13.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:13.830+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.831235+0000) 2020-02-06T00:25:13.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:13.830+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.831321+0000) 2020-02-06T00:25:13.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:13.830+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.832031+0000) 2020-02-06T00:25:13.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:13.831+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.832460+0000) 2020-02-06T00:25:13.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:13.830+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.831898+0000) 2020-02-06T00:25:13.836 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:13.831+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.833168+0000) 2020-02-06T00:25:13.836 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:13.831+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.833414+0000) 2020-02-06T00:25:13.836 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:13.831+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.833141+0000) 2020-02-06T00:25:13.837 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:13.832+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.834168+0000) 2020-02-06T00:25:13.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:13.832+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.833145+0000) 2020-02-06T00:25:13.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:13.832+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.833825+0000) 2020-02-06T00:25:13.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:13.838+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.839960+0000) 2020-02-06T00:25:13.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:13.839+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:13.840959+0000) 2020-02-06T00:25:13.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:13.846+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:14.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:14.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:14.021774+0000) 2020-02-06T00:25:14.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:14.021+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:14.021771+0000) 2020-02-06T00:25:16.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:16.356+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:16.357205+0000) 2020-02-06T00:25:16.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:16.997+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:16.998295+0000) 2020-02-06T00:25:17.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:17.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:17.261058+0000) 2020-02-06T00:25:17.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:17.260+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:17.261058+0000) 2020-02-06T00:25:17.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:17.260+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:17.261396+0000) 2020-02-06T00:25:18.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:18.852+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.853543+0000) 2020-02-06T00:25:18.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:18.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.854141+0000) 2020-02-06T00:25:18.856 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:18.853+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.855372+0000) 2020-02-06T00:25:18.857 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:18.854+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.855985+0000) 2020-02-06T00:25:18.857 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:18.854+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.856256+0000) 2020-02-06T00:25:18.858 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:18.855+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.857033+0000) 2020-02-06T00:25:18.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:18.856+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.857499+0000) 2020-02-06T00:25:18.859 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:18.856+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.857882+0000) 2020-02-06T00:25:18.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:18.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.858246+0000) 2020-02-06T00:25:18.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:18.857+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.858671+0000) 2020-02-06T00:25:18.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:18.858+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.859173+0000) 2020-02-06T00:25:18.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:18.858+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.859622+0000) 2020-02-06T00:25:18.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:18.858+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.859662+0000) 2020-02-06T00:25:18.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:18.864+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:18.865417+0000) 2020-02-06T00:25:18.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:18.870+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173435 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:19.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:19.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:19.259742+0000) 2020-02-06T00:25:19.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:19.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:19.642269+0000) 2020-02-06T00:25:19.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:19.891+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:19.891428+0000) 2020-02-06T00:25:20.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:20.489+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:20.490528+0000) 2020-02-06T00:25:20.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:20.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:20.750556+0000) 2020-02-06T00:25:22.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:22.871+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:22.871910+0000) 2020-02-06T00:25:22.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:22.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:22.964923+0000) 2020-02-06T00:25:23.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:23.877+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.878587+0000) 2020-02-06T00:25:23.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:23.878+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.879526+0000) 2020-02-06T00:25:23.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:23.879+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.880150+0000) 2020-02-06T00:25:23.881 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:23.879+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.880605+0000) 2020-02-06T00:25:23.882 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:23.879+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.880734+0000) 2020-02-06T00:25:23.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:23.879+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.881194+0000) 2020-02-06T00:25:23.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:23.880+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.881829+0000) 2020-02-06T00:25:23.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:23.881+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.882419+0000) 2020-02-06T00:25:23.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:23.881+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.882622+0000) 2020-02-06T00:25:23.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:23.881+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.883048+0000) 2020-02-06T00:25:23.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:23.882+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.883290+0000) 2020-02-06T00:25:23.885 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:23.882+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.884124+0000) 2020-02-06T00:25:23.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:23.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.887673+0000) 2020-02-06T00:25:23.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:23.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:23.887952+0000) 2020-02-06T00:25:23.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:23.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:24.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:24.021+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:24.021992+0000) 2020-02-06T00:25:24.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:24.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:24.022018+0000) 2020-02-06T00:25:26.358 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:26.356+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:26.357430+0000) 2020-02-06T00:25:26.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:26.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:26.998534+0000) 2020-02-06T00:25:27.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:27.260+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:27.261363+0000) 2020-02-06T00:25:27.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:27.260+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:27.261364+0000) 2020-02-06T00:25:27.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:27.260+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:27.261642+0000) 2020-02-06T00:25:28.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:28.903+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.904225+0000) 2020-02-06T00:25:28.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:28.904+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.905115+0000) 2020-02-06T00:25:28.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:28.904+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.905535+0000) 2020-02-06T00:25:28.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:28.904+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.906069+0000) 2020-02-06T00:25:28.907 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:28.905+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.906763+0000) 2020-02-06T00:25:28.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:28.906+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.907260+0000) 2020-02-06T00:25:28.908 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:28.905+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.907122+0000) 2020-02-06T00:25:28.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:28.906+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.907837+0000) 2020-02-06T00:25:28.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:28.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.908463+0000) 2020-02-06T00:25:28.910 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:28.907+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.908650+0000) 2020-02-06T00:25:28.911 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:28.907+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.908985+0000) 2020-02-06T00:25:28.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:28.907+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.909076+0000) 2020-02-06T00:25:28.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:28.907+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.909270+0000) 2020-02-06T00:25:28.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:28.912+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:28.913240+0000) 2020-02-06T00:25:28.925 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:28.922+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:29.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:29.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:29.259968+0000) 2020-02-06T00:25:29.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:29.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:29.642472+0000) 2020-02-06T00:25:29.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:29.891+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:29.891656+0000) 2020-02-06T00:25:30.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:30.490+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:30.490765+0000) 2020-02-06T00:25:30.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:30.749+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:30.750806+0000) 2020-02-06T00:25:32.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:32.871+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:32.872173+0000) 2020-02-06T00:25:32.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:32.972+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:32.972598+0000) 2020-02-06T00:25:33.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:33.928+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.930316+0000) 2020-02-06T00:25:33.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:33.931+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.931404+0000) 2020-02-06T00:25:33.933 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:33.930+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.932014+0000) 2020-02-06T00:25:33.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:33.930+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.932257+0000) 2020-02-06T00:25:33.934 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:33.933+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.933199+0000) 2020-02-06T00:25:33.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:33.932+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.932930+0000) 2020-02-06T00:25:33.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:33.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.934148+0000) 2020-02-06T00:25:33.936 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:33.933+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.934506+0000) 2020-02-06T00:25:33.936 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:33.933+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.934835+0000) 2020-02-06T00:25:33.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:33.933+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.934999+0000) 2020-02-06T00:25:33.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:33.935+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.936028+0000) 2020-02-06T00:25:33.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:33.936+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.936123+0000) 2020-02-06T00:25:33.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:33.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.939294+0000) 2020-02-06T00:25:33.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:33.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:33.939528+0000) 2020-02-06T00:25:33.951 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:33.948+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173560 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:34.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:34.022+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:34.022209+0000) 2020-02-06T00:25:34.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:34.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:34.022312+0000) 2020-02-06T00:25:36.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:36.357+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:36.357661+0000) 2020-02-06T00:25:37.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:36.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:36.998768+0000) 2020-02-06T00:25:37.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:37.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:37.261706+0000) 2020-02-06T00:25:37.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:37.261+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:37.261707+0000) 2020-02-06T00:25:37.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:37.261+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:37.261927+0000) 2020-02-06T00:25:38.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:38.954+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.955919+0000) 2020-02-06T00:25:38.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:38.955+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.956390+0000) 2020-02-06T00:25:38.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:38.955+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.956986+0000) 2020-02-06T00:25:38.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:38.956+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.957651+0000) 2020-02-06T00:25:38.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:38.956+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.957835+0000) 2020-02-06T00:25:38.959 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:38.957+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.958142+0000) 2020-02-06T00:25:38.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:38.957+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.958663+0000) 2020-02-06T00:25:38.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:38.958+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.959633+0000) 2020-02-06T00:25:38.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:38.958+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.959982+0000) 2020-02-06T00:25:38.961 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:38.958+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.960365+0000) 2020-02-06T00:25:38.962 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:38.960+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.961166+0000) 2020-02-06T00:25:38.962 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:38.959+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.960932+0000) 2020-02-06T00:25:38.963 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:38.960+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.961581+0000) 2020-02-06T00:25:38.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:38.964+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:38.965744+0000) 2020-02-06T00:25:38.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:38.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:39.090 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:25:39.095 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:25:39.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:39.258+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:39.260204+0000) 2020-02-06T00:25:39.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:39.641+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:39.642724+0000) 2020-02-06T00:25:39.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:39.890+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:39.891877+0000) 2020-02-06T00:25:40.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:40.490+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:40.491004+0000) 2020-02-06T00:25:40.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:40.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:40.751043+0000) 2020-02-06T00:25:42.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:42.872+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:42.872498+0000) 2020-02-06T00:25:42.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:42.979+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:42.980886+0000) 2020-02-06T00:25:43.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:43.979+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.980524+0000) 2020-02-06T00:25:43.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:43.980+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.981259+0000) 2020-02-06T00:25:43.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:43.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.981828+0000) 2020-02-06T00:25:43.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:43.981+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.982436+0000) 2020-02-06T00:25:43.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:43.982+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.983416+0000) 2020-02-06T00:25:43.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:43.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.983529+0000) 2020-02-06T00:25:43.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:43.983+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.984297+0000) 2020-02-06T00:25:43.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:43.983+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.983963+0000) 2020-02-06T00:25:43.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:43.983+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.984188+0000) 2020-02-06T00:25:43.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:43.984+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.984646+0000) 2020-02-06T00:25:43.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:43.984+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.984535+0000) 2020-02-06T00:25:43.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:43.984+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.985078+0000) 2020-02-06T00:25:43.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:43.985+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.985743+0000) 2020-02-06T00:25:43.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:43.988+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:43.989665+0000) 2020-02-06T00:25:44.001 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:43.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173642 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:44.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:44.021+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:44.022425+0000) 2020-02-06T00:25:44.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:44.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:44.022585+0000) 2020-02-06T00:25:46.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:46.356+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:46.357941+0000) 2020-02-06T00:25:47.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:46.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:46.999005+0000) 2020-02-06T00:25:47.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:47.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:47.262004+0000) 2020-02-06T00:25:47.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:47.261+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:47.262004+0000) 2020-02-06T00:25:47.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:47.261+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:47.262206+0000) 2020-02-06T00:25:49.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:49.003+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.004621+0000) 2020-02-06T00:25:49.007 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:49.004+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.005957+0000) 2020-02-06T00:25:49.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:49.006+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.006967+0000) 2020-02-06T00:25:49.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:49.006+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.008055+0000) 2020-02-06T00:25:49.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:49.007+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.008553+0000) 2020-02-06T00:25:49.010 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:49.008+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.008829+0000) 2020-02-06T00:25:49.010 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:49.008+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.009354+0000) 2020-02-06T00:25:49.011 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:49.008+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.009213+0000) 2020-02-06T00:25:49.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:49.009+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.010440+0000) 2020-02-06T00:25:49.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:49.009+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.010244+0000) 2020-02-06T00:25:49.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:49.010+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.011756+0000) 2020-02-06T00:25:49.013 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:49.011+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.011915+0000) 2020-02-06T00:25:49.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:49.011+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.012147+0000) 2020-02-06T00:25:49.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:49.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.013486+0000) 2020-02-06T00:25:49.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:49.024+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:49.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:49.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.260489+0000) 2020-02-06T00:25:49.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:49.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.642942+0000) 2020-02-06T00:25:49.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:49.891+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:49.892143+0000) 2020-02-06T00:25:50.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:50.490+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:50.491248+0000) 2020-02-06T00:25:50.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:50.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:50.751310+0000) 2020-02-06T00:25:52.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:52.872+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:52.872681+0000) 2020-02-06T00:25:52.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:52.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:52.989234+0000) 2020-02-06T00:25:54.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:54.021+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.022656+0000) 2020-02-06T00:25:54.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:54.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.022759+0000) 2020-02-06T00:25:54.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:54.030+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.031118+0000) 2020-02-06T00:25:54.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:54.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.032001+0000) 2020-02-06T00:25:54.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:54.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.032925+0000) 2020-02-06T00:25:54.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:54.032+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.033452+0000) 2020-02-06T00:25:54.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:54.032+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.033380+0000) 2020-02-06T00:25:54.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:54.033+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.034247+0000) 2020-02-06T00:25:54.036 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:54.034+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.034401+0000) 2020-02-06T00:25:54.036 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:54.034+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.035003+0000) 2020-02-06T00:25:54.037 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:54.033+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.034762+0000) 2020-02-06T00:25:54.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:54.034+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.035440+0000) 2020-02-06T00:25:54.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:54.035+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.036145+0000) 2020-02-06T00:25:54.038 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:54.036+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.036719+0000) 2020-02-06T00:25:54.038 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:54.035+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.036449+0000) 2020-02-06T00:25:54.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:54.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:54.041028+0000) 2020-02-06T00:25:54.050 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:54.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173724 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:56.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:56.357+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:56.358202+0000) 2020-02-06T00:25:57.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:56.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:56.999239+0000) 2020-02-06T00:25:57.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:57.261+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:57.262300+0000) 2020-02-06T00:25:57.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:57.261+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:57.262300+0000) 2020-02-06T00:25:57.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:57.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:57.262507+0000) 2020-02-06T00:25:59.055 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:25:59.052+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.054055+0000) 2020-02-06T00:25:59.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:59.054+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.055271+0000) 2020-02-06T00:25:59.057 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:25:59.055+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.056190+0000) 2020-02-06T00:25:59.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:25:59.055+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.056577+0000) 2020-02-06T00:25:59.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:59.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.057413+0000) 2020-02-06T00:25:59.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:25:59.057+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.057674+0000) 2020-02-06T00:25:59.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:25:59.058+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.059282+0000) 2020-02-06T00:25:59.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:59.058+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.059134+0000) 2020-02-06T00:25:59.061 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:25:59.060+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.060541+0000) 2020-02-06T00:25:59.062 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:25:59.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.060723+0000) 2020-02-06T00:25:59.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:25:59.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.061231+0000) 2020-02-06T00:25:59.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:25:59.060+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.061416+0000) 2020-02-06T00:25:59.063 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:25:59.060+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.061061+0000) 2020-02-06T00:25:59.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:59.064+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.065424+0000) 2020-02-06T00:25:59.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:25:59.074+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173764 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:25:59.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:25:59.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.260733+0000) 2020-02-06T00:25:59.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:59.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.643161+0000) 2020-02-06T00:25:59.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:25:59.891+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:25:59.892350+0000) 2020-02-06T00:26:00.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:00.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:00.491483+0000) 2020-02-06T00:26:00.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:00.750+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:00.751582+0000) 2020-02-06T00:26:02.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:02.872+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:02.872945+0000) 2020-02-06T00:26:02.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:02.997+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:02.997618+0000) 2020-02-06T00:26:04.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:04.021+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.022943+0000) 2020-02-06T00:26:04.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:04.021+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.023008+0000) 2020-02-06T00:26:04.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:04.078+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.079283+0000) 2020-02-06T00:26:04.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:04.079+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.080468+0000) 2020-02-06T00:26:04.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:04.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.081300+0000) 2020-02-06T00:26:04.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:04.080+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.081695+0000) 2020-02-06T00:26:04.083 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:04.081+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.082347+0000) 2020-02-06T00:26:04.084 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:04.081+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.082470+0000) 2020-02-06T00:26:04.084 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:04.082+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.083231+0000) 2020-02-06T00:26:04.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:04.083+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.084142+0000) 2020-02-06T00:26:04.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:04.084+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.084724+0000) 2020-02-06T00:26:04.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:04.085+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.085679+0000) 2020-02-06T00:26:04.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:04.085+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.086159+0000) 2020-02-06T00:26:04.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:04.086+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.086444+0000) 2020-02-06T00:26:04.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:04.085+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.086172+0000) 2020-02-06T00:26:04.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:04.090+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:04.091163+0000) 2020-02-06T00:26:04.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:04.098+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:06.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:06.357+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:06.358389+0000) 2020-02-06T00:26:07.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:06.998+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:06.999473+0000) 2020-02-06T00:26:07.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:07.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:07.262553+0000) 2020-02-06T00:26:07.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:07.262+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:07.262553+0000) 2020-02-06T00:26:07.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:07.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:07.262766+0000) 2020-02-06T00:26:09.105 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:09.102+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.103528+0000) 2020-02-06T00:26:09.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:09.105+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.105683+0000) 2020-02-06T00:26:09.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:09.105+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.106162+0000) 2020-02-06T00:26:09.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:09.106+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.106854+0000) 2020-02-06T00:26:09.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:09.106+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.107150+0000) 2020-02-06T00:26:09.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:09.107+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.107748+0000) 2020-02-06T00:26:09.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:09.107+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.108265+0000) 2020-02-06T00:26:09.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:09.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.108744+0000) 2020-02-06T00:26:09.110 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:09.108+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.108980+0000) 2020-02-06T00:26:09.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:09.108+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.109293+0000) 2020-02-06T00:26:09.111 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:09.109+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.110044+0000) 2020-02-06T00:26:09.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:09.108+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.109653+0000) 2020-02-06T00:26:09.112 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:09.110+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.110456+0000) 2020-02-06T00:26:09.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:09.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.115222+0000) 2020-02-06T00:26:09.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:09.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:09.255 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:26:09.261 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:26:09.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:09.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.260930+0000) 2020-02-06T00:26:09.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:09.642+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.643398+0000) 2020-02-06T00:26:09.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:09.892+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:09.892578+0000) 2020-02-06T00:26:10.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:10.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:10.491717+0000) 2020-02-06T00:26:10.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:10.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:10.751817+0000) 2020-02-06T00:26:12.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:12.873+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:12.873188+0000) 2020-02-06T00:26:13.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:13.005+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:13.006018+0000) 2020-02-06T00:26:14.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:14.022+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.023182+0000) 2020-02-06T00:26:14.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:14.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.023280+0000) 2020-02-06T00:26:14.131 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:14.128+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.129991+0000) 2020-02-06T00:26:14.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:14.129+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.130647+0000) 2020-02-06T00:26:14.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:14.130+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.131376+0000) 2020-02-06T00:26:14.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:14.130+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.131783+0000) 2020-02-06T00:26:14.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:14.131+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.131879+0000) 2020-02-06T00:26:14.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:14.131+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.132211+0000) 2020-02-06T00:26:14.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:14.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.132723+0000) 2020-02-06T00:26:14.134 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:14.131+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.132607+0000) 2020-02-06T00:26:14.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:14.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.133201+0000) 2020-02-06T00:26:14.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:14.132+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.133105+0000) 2020-02-06T00:26:14.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:14.132+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.133327+0000) 2020-02-06T00:26:14.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:14.133+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.134208+0000) 2020-02-06T00:26:14.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:14.136+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.138055+0000) 2020-02-06T00:26:14.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:14.137+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:14.138267+0000) 2020-02-06T00:26:14.149 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:14.147+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173888 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:16.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:16.357+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:16.358636+0000) 2020-02-06T00:26:17.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:16.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:16.999713+0000) 2020-02-06T00:26:17.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:17.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:17.262856+0000) 2020-02-06T00:26:17.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:17.262+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:17.262856+0000) 2020-02-06T00:26:17.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:17.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:17.263026+0000) 2020-02-06T00:26:19.154 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:19.151+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.152545+0000) 2020-02-06T00:26:19.155 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:19.153+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.154357+0000) 2020-02-06T00:26:19.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:19.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.155158+0000) 2020-02-06T00:26:19.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:19.154+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.155732+0000) 2020-02-06T00:26:19.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:19.156+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.156498+0000) 2020-02-06T00:26:19.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:19.156+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.157323+0000) 2020-02-06T00:26:19.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:19.157+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.158050+0000) 2020-02-06T00:26:19.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:19.158+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.158742+0000) 2020-02-06T00:26:19.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:19.157+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.158504+0000) 2020-02-06T00:26:19.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:19.159+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.160114+0000) 2020-02-06T00:26:19.161 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:19.159+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.159958+0000) 2020-02-06T00:26:19.162 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:19.159+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.160507+0000) 2020-02-06T00:26:19.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:19.159+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.160593+0000) 2020-02-06T00:26:19.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:19.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.166660+0000) 2020-02-06T00:26:19.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:19.172+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:19.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:19.260+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.261106+0000) 2020-02-06T00:26:19.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:19.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.643609+0000) 2020-02-06T00:26:19.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:19.892+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:19.892794+0000) 2020-02-06T00:26:20.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:20.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:20.491939+0000) 2020-02-06T00:26:20.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:20.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:20.752056+0000) 2020-02-06T00:26:22.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:22.872+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:22.873468+0000) 2020-02-06T00:26:23.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:23.014+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:23.014535+0000) 2020-02-06T00:26:24.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:24.023+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.023394+0000) 2020-02-06T00:26:24.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:24.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.023543+0000) 2020-02-06T00:26:24.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:24.178+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.179131+0000) 2020-02-06T00:26:24.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:24.179+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.180328+0000) 2020-02-06T00:26:24.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:24.179+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.180883+0000) 2020-02-06T00:26:24.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:24.179+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.180793+0000) 2020-02-06T00:26:24.182 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:24.180+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.181274+0000) 2020-02-06T00:26:24.183 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:24.180+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.181616+0000) 2020-02-06T00:26:24.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:24.181+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.182292+0000) 2020-02-06T00:26:24.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:24.182+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.183280+0000) 2020-02-06T00:26:24.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:24.182+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.184034+0000) 2020-02-06T00:26:24.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:24.182+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.184340+0000) 2020-02-06T00:26:24.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:24.183+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.184797+0000) 2020-02-06T00:26:24.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:24.183+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.185069+0000) 2020-02-06T00:26:24.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:24.183+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.185263+0000) 2020-02-06T00:26:24.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:24.187+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:24.188833+0000) 2020-02-06T00:26:24.201 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:24.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 173970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:26.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:26.358+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:26.358885+0000) 2020-02-06T00:26:27.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:26.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:26.999945+0000) 2020-02-06T00:26:27.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:27.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:27.263137+0000) 2020-02-06T00:26:27.265 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:27.262+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:27.263137+0000) 2020-02-06T00:26:27.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:27.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:27.263297+0000) 2020-02-06T00:26:29.206 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:29.205+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.205211+0000) 2020-02-06T00:26:29.207 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:29.205+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.205615+0000) 2020-02-06T00:26:29.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:29.204+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.206086+0000) 2020-02-06T00:26:29.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:29.205+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.206037+0000) 2020-02-06T00:26:29.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:29.206+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.206807+0000) 2020-02-06T00:26:29.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:29.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.208775+0000) 2020-02-06T00:26:29.210 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:29.209+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.209145+0000) 2020-02-06T00:26:29.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:29.209+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.209727+0000) 2020-02-06T00:26:29.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:29.208+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.210139+0000) 2020-02-06T00:26:29.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:29.208+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.210244+0000) 2020-02-06T00:26:29.212 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:29.209+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.211013+0000) 2020-02-06T00:26:29.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:29.209+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.211329+0000) 2020-02-06T00:26:29.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:29.210+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.211491+0000) 2020-02-06T00:26:29.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:29.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.217072+0000) 2020-02-06T00:26:29.226 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:29.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174010 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:29.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:29.259+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.261318+0000) 2020-02-06T00:26:29.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:29.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.643816+0000) 2020-02-06T00:26:29.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:29.892+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:29.893012+0000) 2020-02-06T00:26:30.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:30.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:30.492185+0000) 2020-02-06T00:26:30.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:30.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:30.752362+0000) 2020-02-06T00:26:32.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:32.872+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:32.873733+0000) 2020-02-06T00:26:33.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:33.021+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:33.022925+0000) 2020-02-06T00:26:34.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:34.022+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.023656+0000) 2020-02-06T00:26:34.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:34.022+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.023796+0000) 2020-02-06T00:26:34.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:34.229+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.230852+0000) 2020-02-06T00:26:34.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:34.230+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.231675+0000) 2020-02-06T00:26:34.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:34.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.232858+0000) 2020-02-06T00:26:34.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:34.232+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.233685+0000) 2020-02-06T00:26:34.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:34.232+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.233451+0000) 2020-02-06T00:26:34.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:34.233+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.234070+0000) 2020-02-06T00:26:34.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:34.233+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.234604+0000) 2020-02-06T00:26:34.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:34.234+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.235370+0000) 2020-02-06T00:26:34.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:34.233+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.235219+0000) 2020-02-06T00:26:34.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:34.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.235670+0000) 2020-02-06T00:26:34.238 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:34.234+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.236039+0000) 2020-02-06T00:26:34.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:34.235+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.236562+0000) 2020-02-06T00:26:34.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:34.235+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.236800+0000) 2020-02-06T00:26:34.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:34.237+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:34.238686+0000) 2020-02-06T00:26:34.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:34.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174053 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:36.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:36.358+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:36.359134+0000) 2020-02-06T00:26:37.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:36.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:37.000181+0000) 2020-02-06T00:26:37.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:37.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:37.263478+0000) 2020-02-06T00:26:37.265 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:37.262+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:37.263477+0000) 2020-02-06T00:26:37.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:37.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:37.263599+0000) 2020-02-06T00:26:39.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:39.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.254983+0000) 2020-02-06T00:26:39.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:39.254+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.256390+0000) 2020-02-06T00:26:39.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:39.255+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.257003+0000) 2020-02-06T00:26:39.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:39.256+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.257562+0000) 2020-02-06T00:26:39.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:39.257+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.258102+0000) 2020-02-06T00:26:39.259 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:39.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.258380+0000) 2020-02-06T00:26:39.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:39.258+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.259597+0000) 2020-02-06T00:26:39.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:39.258+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.260110+0000) 2020-02-06T00:26:39.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:39.259+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.261006+0000) 2020-02-06T00:26:39.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:39.261+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.262791+0000) 2020-02-06T00:26:39.268 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:39.258+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.259589+0000) 2020-02-06T00:26:39.268 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:39.258+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.259954+0000) 2020-02-06T00:26:39.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:39.259+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.261005+0000) 2020-02-06T00:26:39.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:39.264+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.265597+0000) 2020-02-06T00:26:39.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:39.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:39.393 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:26:39.399 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:26:39.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:39.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.644030+0000) 2020-02-06T00:26:39.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:39.892+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:39.893214+0000) 2020-02-06T00:26:40.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:40.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:40.492419+0000) 2020-02-06T00:26:40.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:40.751+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:40.752597+0000) 2020-02-06T00:26:42.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:42.873+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:42.873976+0000) 2020-02-06T00:26:43.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:43.031+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:43.032710+0000) 2020-02-06T00:26:44.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:44.023+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.023883+0000) 2020-02-06T00:26:44.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:44.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.024065+0000) 2020-02-06T00:26:44.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:44.279+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.280447+0000) 2020-02-06T00:26:44.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:44.280+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.281326+0000) 2020-02-06T00:26:44.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:44.280+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.282103+0000) 2020-02-06T00:26:44.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:44.280+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.282276+0000) 2020-02-06T00:26:44.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:44.281+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.283028+0000) 2020-02-06T00:26:44.284 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:44.281+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.283347+0000) 2020-02-06T00:26:44.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:44.282+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.283745+0000) 2020-02-06T00:26:44.285 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:44.282+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.283846+0000) 2020-02-06T00:26:44.286 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:44.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.284750+0000) 2020-02-06T00:26:44.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:44.283+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.284706+0000) 2020-02-06T00:26:44.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:44.283+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.284997+0000) 2020-02-06T00:26:44.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:44.283+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.285039+0000) 2020-02-06T00:26:44.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:44.284+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.285783+0000) 2020-02-06T00:26:44.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:44.288+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:44.289785+0000) 2020-02-06T00:26:44.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:44.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:46.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:46.011+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:46.011849+0000) 2020-02-06T00:26:46.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:46.358+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:46.359323+0000) 2020-02-06T00:26:47.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:46.999+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:47.000429+0000) 2020-02-06T00:26:47.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:47.263+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:47.263807+0000) 2020-02-06T00:26:47.265 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:47.263+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:47.263807+0000) 2020-02-06T00:26:47.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:47.263+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:47.263880+0000) 2020-02-06T00:26:49.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:49.305+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.306510+0000) 2020-02-06T00:26:49.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:49.306+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.307850+0000) 2020-02-06T00:26:49.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:49.307+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.308662+0000) 2020-02-06T00:26:49.310 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:49.307+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.308803+0000) 2020-02-06T00:26:49.311 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:49.307+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.308998+0000) 2020-02-06T00:26:49.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:49.308+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.309998+0000) 2020-02-06T00:26:49.312 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:49.309+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.310380+0000) 2020-02-06T00:26:49.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:49.309+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.310909+0000) 2020-02-06T00:26:49.312 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:49.308+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.309898+0000) 2020-02-06T00:26:49.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:49.310+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.311994+0000) 2020-02-06T00:26:49.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:49.310+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.312375+0000) 2020-02-06T00:26:49.314 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:49.311+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.312238+0000) 2020-02-06T00:26:49.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:49.311+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.312903+0000) 2020-02-06T00:26:49.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:49.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.316926+0000) 2020-02-06T00:26:49.328 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:49.325+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:49.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:49.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.644245+0000) 2020-02-06T00:26:49.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:49.892+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:49.893423+0000) 2020-02-06T00:26:50.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:50.491+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:50.492690+0000) 2020-02-06T00:26:50.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:50.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:50.752832+0000) 2020-02-06T00:26:52.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:52.873+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:52.874217+0000) 2020-02-06T00:26:53.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:53.039+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:53.040375+0000) 2020-02-06T00:26:54.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:54.023+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.024127+0000) 2020-02-06T00:26:54.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:54.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.024321+0000) 2020-02-06T00:26:54.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:54.332+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.334194+0000) 2020-02-06T00:26:54.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:54.333+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.334719+0000) 2020-02-06T00:26:54.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:54.333+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.334717+0000) 2020-02-06T00:26:54.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:54.334+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.335202+0000) 2020-02-06T00:26:54.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:54.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.335871+0000) 2020-02-06T00:26:54.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:54.335+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.336386+0000) 2020-02-06T00:26:54.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:54.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.336555+0000) 2020-02-06T00:26:54.338 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:54.335+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.336979+0000) 2020-02-06T00:26:54.339 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:54.336+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.337465+0000) 2020-02-06T00:26:54.339 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:54.336+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.337799+0000) 2020-02-06T00:26:54.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:54.336+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.338263+0000) 2020-02-06T00:26:54.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:54.336+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.338208+0000) 2020-02-06T00:26:54.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:54.337+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.338659+0000) 2020-02-06T00:26:54.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:54.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:54.343358+0000) 2020-02-06T00:26:54.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:54.351+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174214 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:56.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:56.011+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:56.012142+0000) 2020-02-06T00:26:56.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:56.358+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:56.359571+0000) 2020-02-06T00:26:57.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:57.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:57.000658+0000) 2020-02-06T00:26:57.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:57.262+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:57.264054+0000) 2020-02-06T00:26:57.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:57.262+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:57.264058+0000) 2020-02-06T00:26:57.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:57.262+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:57.264376+0000) 2020-02-06T00:26:59.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:26:59.358+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.359134+0000) 2020-02-06T00:26:59.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:26:59.358+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.359698+0000) 2020-02-06T00:26:59.361 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:26:59.359+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.360063+0000) 2020-02-06T00:26:59.362 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:26:59.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.360740+0000) 2020-02-06T00:26:59.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:59.359+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.360649+0000) 2020-02-06T00:26:59.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:26:59.359+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.361056+0000) 2020-02-06T00:26:59.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:59.361+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.362733+0000) 2020-02-06T00:26:59.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:26:59.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.362812+0000) 2020-02-06T00:26:59.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:26:59.362+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.363463+0000) 2020-02-06T00:26:59.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:26:59.362+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.363584+0000) 2020-02-06T00:26:59.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:26:59.362+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.363949+0000) 2020-02-06T00:26:59.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:26:59.363+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.364489+0000) 2020-02-06T00:26:59.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:26:59.363+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.365277+0000) 2020-02-06T00:26:59.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:59.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.370850+0000) 2020-02-06T00:26:59.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:26:59.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174254 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:26:59.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:59.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.644415+0000) 2020-02-06T00:26:59.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:26:59.893+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:26:59.893633+0000) 2020-02-06T00:27:00.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:00.492+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:00.492938+0000) 2020-02-06T00:27:00.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:00.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:00.753120+0000) 2020-02-06T00:27:02.875 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:02.873+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:02.874492+0000) 2020-02-06T00:27:03.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:03.048+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:03.049056+0000) 2020-02-06T00:27:04.025 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:04.023+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.024352+0000) 2020-02-06T00:27:04.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:04.023+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.024513+0000) 2020-02-06T00:27:04.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:04.383+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.384436+0000) 2020-02-06T00:27:04.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:04.384+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.385361+0000) 2020-02-06T00:27:04.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:04.384+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.385954+0000) 2020-02-06T00:27:04.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:04.385+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.386786+0000) 2020-02-06T00:27:04.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:04.386+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.387204+0000) 2020-02-06T00:27:04.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:04.386+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.387535+0000) 2020-02-06T00:27:04.389 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:04.386+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.387739+0000) 2020-02-06T00:27:04.389 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:04.386+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.388039+0000) 2020-02-06T00:27:04.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:04.387+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.388747+0000) 2020-02-06T00:27:04.390 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:04.387+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.388629+0000) 2020-02-06T00:27:04.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:04.388+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.389624+0000) 2020-02-06T00:27:04.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:04.388+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.390396+0000) 2020-02-06T00:27:04.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:04.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.394610+0000) 2020-02-06T00:27:04.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:04.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:04.394900+0000) 2020-02-06T00:27:04.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:04.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:06.013 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:06.011+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:06.012399+0000) 2020-02-06T00:27:06.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:06.358+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:06.359815+0000) 2020-02-06T00:27:07.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:07.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:07.000884+0000) 2020-02-06T00:27:07.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:07.263+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:07.264333+0000) 2020-02-06T00:27:07.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:07.263+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:07.264332+0000) 2020-02-06T00:27:07.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:07.263+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:07.264631+0000) 2020-02-06T00:27:09.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:09.409+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.410852+0000) 2020-02-06T00:27:09.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:09.409+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.411311+0000) 2020-02-06T00:27:09.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:09.411+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.412919+0000) 2020-02-06T00:27:09.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:09.412+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.413435+0000) 2020-02-06T00:27:09.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:09.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.413490+0000) 2020-02-06T00:27:09.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:09.412+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.413940+0000) 2020-02-06T00:27:09.415 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:09.412+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.413995+0000) 2020-02-06T00:27:09.416 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:09.413+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.414480+0000) 2020-02-06T00:27:09.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:09.413+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.414896+0000) 2020-02-06T00:27:09.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:09.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.415003+0000) 2020-02-06T00:27:09.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:09.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.415240+0000) 2020-02-06T00:27:09.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:09.414+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.415711+0000) 2020-02-06T00:27:09.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:09.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.418693+0000) 2020-02-06T00:27:09.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:09.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.418918+0000) 2020-02-06T00:27:09.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:09.426+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:09.532 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:27:09.538 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:27:09.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:09.643+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.644593+0000) 2020-02-06T00:27:09.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:09.893+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:09.893847+0000) 2020-02-06T00:27:10.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:10.492+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:10.493176+0000) 2020-02-06T00:27:10.754 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:10.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:10.753356+0000) 2020-02-06T00:27:12.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:12.874+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:12.874773+0000) 2020-02-06T00:27:13.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:13.057+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:13.057432+0000) 2020-02-06T00:27:14.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:14.024+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.024584+0000) 2020-02-06T00:27:14.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:14.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.024765+0000) 2020-02-06T00:27:14.436 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:14.434+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.435119+0000) 2020-02-06T00:27:14.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:14.434+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.434602+0000) 2020-02-06T00:27:14.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:14.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.435656+0000) 2020-02-06T00:27:14.437 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:14.435+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.436274+0000) 2020-02-06T00:27:14.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:14.436+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.436990+0000) 2020-02-06T00:27:14.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:14.435+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.436797+0000) 2020-02-06T00:27:14.438 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:14.436+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.437293+0000) 2020-02-06T00:27:14.439 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:14.437+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.437591+0000) 2020-02-06T00:27:14.439 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:14.437+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.438065+0000) 2020-02-06T00:27:14.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:14.437+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.438360+0000) 2020-02-06T00:27:14.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:14.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.439443+0000) 2020-02-06T00:27:14.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:14.439+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.440182+0000) 2020-02-06T00:27:14.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:14.440+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.440717+0000) 2020-02-06T00:27:14.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:14.444+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:14.445803+0000) 2020-02-06T00:27:14.455 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:14.453+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:16.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:16.012+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:16.012691+0000) 2020-02-06T00:27:16.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:16.359+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:16.360065+0000) 2020-02-06T00:27:17.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:17.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:17.001118+0000) 2020-02-06T00:27:17.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:17.264+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:17.264674+0000) 2020-02-06T00:27:17.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:17.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:17.264678+0000) 2020-02-06T00:27:17.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:17.264+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:17.264882+0000) 2020-02-06T00:27:19.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:19.459+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.460141+0000) 2020-02-06T00:27:19.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:19.460+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.460667+0000) 2020-02-06T00:27:19.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:19.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.461529+0000) 2020-02-06T00:27:19.463 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:19.460+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.461753+0000) 2020-02-06T00:27:19.463 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:19.461+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.462351+0000) 2020-02-06T00:27:19.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:19.461+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.462477+0000) 2020-02-06T00:27:19.464 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:19.463+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.463496+0000) 2020-02-06T00:27:19.465 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:19.462+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.463840+0000) 2020-02-06T00:27:19.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:19.463+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.464344+0000) 2020-02-06T00:27:19.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:19.463+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.464307+0000) 2020-02-06T00:27:19.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:19.464+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.464724+0000) 2020-02-06T00:27:19.467 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:19.464+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.465416+0000) 2020-02-06T00:27:19.467 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:19.464+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.465378+0000) 2020-02-06T00:27:19.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:19.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.467244+0000) 2020-02-06T00:27:19.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:19.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174416 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:19.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:19.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.644840+0000) 2020-02-06T00:27:19.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:19.893+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:19.894057+0000) 2020-02-06T00:27:20.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:20.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:20.493412+0000) 2020-02-06T00:27:20.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:20.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:20.753590+0000) 2020-02-06T00:27:22.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:22.874+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:22.875016+0000) 2020-02-06T00:27:23.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:23.065+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:23.065874+0000) 2020-02-06T00:27:24.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:24.024+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.024826+0000) 2020-02-06T00:27:24.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:24.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.025007+0000) 2020-02-06T00:27:24.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:24.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.484450+0000) 2020-02-06T00:27:24.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:24.485+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.485388+0000) 2020-02-06T00:27:24.487 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:24.486+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.486472+0000) 2020-02-06T00:27:24.488 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:24.487+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.487293+0000) 2020-02-06T00:27:24.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:24.486+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.487266+0000) 2020-02-06T00:27:24.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:24.487+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.487694+0000) 2020-02-06T00:27:24.489 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:24.488+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.488544+0000) 2020-02-06T00:27:24.490 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:24.488+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.488506+0000) 2020-02-06T00:27:24.490 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:24.488+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.489021+0000) 2020-02-06T00:27:24.491 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:24.489+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.489778+0000) 2020-02-06T00:27:24.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:24.490+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.490419+0000) 2020-02-06T00:27:24.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:24.490+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.490862+0000) 2020-02-06T00:27:24.492 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:24.491+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.491128+0000) 2020-02-06T00:27:24.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:24.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:24.497233+0000) 2020-02-06T00:27:24.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:24.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:26.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:26.012+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:26.012928+0000) 2020-02-06T00:27:26.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:26.360+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:26.360293+0000) 2020-02-06T00:27:27.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:27.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:27.001350+0000) 2020-02-06T00:27:27.266 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:27.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:27.264965+0000) 2020-02-06T00:27:27.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:27.264+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:27.264967+0000) 2020-02-06T00:27:27.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:27.264+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:27.265173+0000) 2020-02-06T00:27:29.513 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:29.510+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.511896+0000) 2020-02-06T00:27:29.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:29.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.512698+0000) 2020-02-06T00:27:29.514 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:29.512+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.513058+0000) 2020-02-06T00:27:29.515 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:29.513+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.514149+0000) 2020-02-06T00:27:29.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:29.513+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.514682+0000) 2020-02-06T00:27:29.516 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:29.514+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.515350+0000) 2020-02-06T00:27:29.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:29.515+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.516024+0000) 2020-02-06T00:27:29.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:29.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.515950+0000) 2020-02-06T00:27:29.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:29.515+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.516426+0000) 2020-02-06T00:27:29.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:29.516+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.516998+0000) 2020-02-06T00:27:29.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:29.516+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.516725+0000) 2020-02-06T00:27:29.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:29.517+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.518251+0000) 2020-02-06T00:27:29.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:29.518+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.518622+0000) 2020-02-06T00:27:29.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:29.517+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.518965+0000) 2020-02-06T00:27:29.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:29.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174497 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:29.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:29.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.645052+0000) 2020-02-06T00:27:29.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:29.893+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:29.894331+0000) 2020-02-06T00:27:30.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:30.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:30.493715+0000) 2020-02-06T00:27:30.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:30.752+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:30.753856+0000) 2020-02-06T00:27:32.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:32.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:32.875276+0000) 2020-02-06T00:27:33.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:33.073+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:33.074610+0000) 2020-02-06T00:27:34.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:34.024+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.025129+0000) 2020-02-06T00:27:34.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:34.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.025260+0000) 2020-02-06T00:27:34.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:34.536+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.537419+0000) 2020-02-06T00:27:34.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:34.536+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.537732+0000) 2020-02-06T00:27:34.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:34.537+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.537935+0000) 2020-02-06T00:27:34.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:34.537+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.538381+0000) 2020-02-06T00:27:34.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:34.538+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.539352+0000) 2020-02-06T00:27:34.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:34.539+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.539982+0000) 2020-02-06T00:27:34.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:34.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.540306+0000) 2020-02-06T00:27:34.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:34.540+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.541004+0000) 2020-02-06T00:27:34.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:34.541+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.541777+0000) 2020-02-06T00:27:34.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:34.541+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.542193+0000) 2020-02-06T00:27:34.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:34.541+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.542781+0000) 2020-02-06T00:27:34.544 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:34.542+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.543207+0000) 2020-02-06T00:27:34.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:34.543+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.543911+0000) 2020-02-06T00:27:34.548 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:34.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:34.546996+0000) 2020-02-06T00:27:34.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:34.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174537 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:36.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:36.012+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:36.013219+0000) 2020-02-06T00:27:36.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:36.359+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:36.360560+0000) 2020-02-06T00:27:37.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:37.000+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:37.001588+0000) 2020-02-06T00:27:37.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:37.265+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:37.265269+0000) 2020-02-06T00:27:37.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:37.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:37.265270+0000) 2020-02-06T00:27:37.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:37.265+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:37.265498+0000) 2020-02-06T00:27:39.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:39.562+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.563144+0000) 2020-02-06T00:27:39.565 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:39.562+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.563624+0000) 2020-02-06T00:27:39.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:39.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.564397+0000) 2020-02-06T00:27:39.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:39.564+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.564492+0000) 2020-02-06T00:27:39.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:39.564+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.565258+0000) 2020-02-06T00:27:39.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:39.565+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.565523+0000) 2020-02-06T00:27:39.567 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:39.565+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.565940+0000) 2020-02-06T00:27:39.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:39.565+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.566287+0000) 2020-02-06T00:27:39.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:39.565+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.566648+0000) 2020-02-06T00:27:39.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:39.566+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.567022+0000) 2020-02-06T00:27:39.568 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:39.566+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.567393+0000) 2020-02-06T00:27:39.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:39.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.567845+0000) 2020-02-06T00:27:39.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:39.567+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.568261+0000) 2020-02-06T00:27:39.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:39.572+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.573240+0000) 2020-02-06T00:27:39.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:39.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:39.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:39.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.645257+0000) 2020-02-06T00:27:39.669 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:27:39.675 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:27:39.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:39.893+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:39.894542+0000) 2020-02-06T00:27:40.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:40.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:40.493946+0000) 2020-02-06T00:27:40.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:40.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:40.754108+0000) 2020-02-06T00:27:42.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:42.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:42.875543+0000) 2020-02-06T00:27:43.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:43.082+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:43.083410+0000) 2020-02-06T00:27:44.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:44.024+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.025354+0000) 2020-02-06T00:27:44.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:44.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.025461+0000) 2020-02-06T00:27:44.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:44.588+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.588786+0000) 2020-02-06T00:27:44.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:44.588+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.589149+0000) 2020-02-06T00:27:44.591 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:44.588+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.589960+0000) 2020-02-06T00:27:44.591 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:44.589+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.590544+0000) 2020-02-06T00:27:44.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:44.590+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.590831+0000) 2020-02-06T00:27:44.592 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:44.590+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.591317+0000) 2020-02-06T00:27:44.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:44.590+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.591790+0000) 2020-02-06T00:27:44.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:44.591+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.592264+0000) 2020-02-06T00:27:44.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:44.592+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.592958+0000) 2020-02-06T00:27:44.594 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:44.592+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.593118+0000) 2020-02-06T00:27:44.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:44.592+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.593807+0000) 2020-02-06T00:27:44.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:44.592+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.593841+0000) 2020-02-06T00:27:44.596 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:44.593+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.594276+0000) 2020-02-06T00:27:44.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:44.599+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:44.600371+0000) 2020-02-06T00:27:44.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:44.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:46.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:46.013+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:46.013501+0000) 2020-02-06T00:27:46.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:46.359+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:46.360820+0000) 2020-02-06T00:27:47.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:47.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:47.001846+0000) 2020-02-06T00:27:47.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:47.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:47.265556+0000) 2020-02-06T00:27:47.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:47.265+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:47.265557+0000) 2020-02-06T00:27:47.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:47.265+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:47.265754+0000) 2020-02-06T00:27:49.615 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:49.613+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.614467+0000) 2020-02-06T00:27:49.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:49.613+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.615074+0000) 2020-02-06T00:27:49.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:49.613+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.615166+0000) 2020-02-06T00:27:49.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:49.614+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.615728+0000) 2020-02-06T00:27:49.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:49.615+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.616721+0000) 2020-02-06T00:27:49.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:49.616+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.617502+0000) 2020-02-06T00:27:49.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:49.616+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.617973+0000) 2020-02-06T00:27:49.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:49.616+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.618024+0000) 2020-02-06T00:27:49.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:49.617+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.618704+0000) 2020-02-06T00:27:49.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:49.617+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.618513+0000) 2020-02-06T00:27:49.621 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:49.618+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.619557+0000) 2020-02-06T00:27:49.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:49.618+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.619505+0000) 2020-02-06T00:27:49.621 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:49.618+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.619501+0000) 2020-02-06T00:27:49.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:49.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.625035+0000) 2020-02-06T00:27:49.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:49.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174657 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:49.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:49.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.645494+0000) 2020-02-06T00:27:49.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:49.894+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:49.894735+0000) 2020-02-06T00:27:50.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:50.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:50.494185+0000) 2020-02-06T00:27:50.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:50.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:50.754353+0000) 2020-02-06T00:27:52.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:52.874+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:52.875799+0000) 2020-02-06T00:27:53.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:53.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:53.092021+0000) 2020-02-06T00:27:54.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:54.025+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.025580+0000) 2020-02-06T00:27:54.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:54.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.025730+0000) 2020-02-06T00:27:54.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:54.637+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.638396+0000) 2020-02-06T00:27:54.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:54.637+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.638665+0000) 2020-02-06T00:27:54.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:54.638+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.640040+0000) 2020-02-06T00:27:54.641 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:54.639+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.640429+0000) 2020-02-06T00:27:54.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:54.639+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.640938+0000) 2020-02-06T00:27:54.642 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:54.640+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.641487+0000) 2020-02-06T00:27:54.649 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:54.641+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.642942+0000) 2020-02-06T00:27:54.649 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:54.642+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.643954+0000) 2020-02-06T00:27:54.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:54.643+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.644837+0000) 2020-02-06T00:27:54.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:54.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.642984+0000) 2020-02-06T00:27:54.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:54.642+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.643615+0000) 2020-02-06T00:27:54.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:54.643+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.644652+0000) 2020-02-06T00:27:54.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:54.643+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.644775+0000) 2020-02-06T00:27:54.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:54.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:54.648197+0000) 2020-02-06T00:27:54.661 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:54.658+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:56.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:56.012+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:56.013752+0000) 2020-02-06T00:27:56.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:56.360+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:56.361112+0000) 2020-02-06T00:27:57.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:57.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:57.002103+0000) 2020-02-06T00:27:57.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:57.264+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:57.265846+0000) 2020-02-06T00:27:57.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:57.264+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:57.265850+0000) 2020-02-06T00:27:57.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:57.264+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:57.266042+0000) 2020-02-06T00:27:59.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:59.644+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.645753+0000) 2020-02-06T00:27:59.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:59.664+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.666025+0000) 2020-02-06T00:27:59.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:59.666+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.667223+0000) 2020-02-06T00:27:59.669 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:27:59.666+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.667806+0000) 2020-02-06T00:27:59.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:27:59.666+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.668055+0000) 2020-02-06T00:27:59.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:27:59.667+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.668338+0000) 2020-02-06T00:27:59.670 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:27:59.667+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.668628+0000) 2020-02-06T00:27:59.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:27:59.667+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.668980+0000) 2020-02-06T00:27:59.670 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:27:59.668+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.669309+0000) 2020-02-06T00:27:59.671 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:27:59.668+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.670168+0000) 2020-02-06T00:27:59.671 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:27:59.669+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.670588+0000) 2020-02-06T00:27:59.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:27:59.669+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.670906+0000) 2020-02-06T00:27:59.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:27:59.669+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.670425+0000) 2020-02-06T00:27:59.673 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:27:59.670+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.671518+0000) 2020-02-06T00:27:59.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:59.674+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.675245+0000) 2020-02-06T00:27:59.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:27:59.683+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:27:59.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:27:59.894+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:27:59.894947+0000) 2020-02-06T00:28:00.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:00.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:00.494405+0000) 2020-02-06T00:28:00.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:00.753+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:00.754598+0000) 2020-02-06T00:28:02.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:02.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:02.876061+0000) 2020-02-06T00:28:03.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:03.099+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:03.100570+0000) 2020-02-06T00:28:04.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:04.024+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.025806+0000) 2020-02-06T00:28:04.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:04.024+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.025970+0000) 2020-02-06T00:28:04.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:04.689+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.690979+0000) 2020-02-06T00:28:04.693 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:04.690+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.691624+0000) 2020-02-06T00:28:04.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:04.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.691775+0000) 2020-02-06T00:28:04.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:04.690+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.692193+0000) 2020-02-06T00:28:04.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:04.691+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.692641+0000) 2020-02-06T00:28:04.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:04.692+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.693901+0000) 2020-02-06T00:28:04.695 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:04.693+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.694254+0000) 2020-02-06T00:28:04.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:04.693+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.694533+0000) 2020-02-06T00:28:04.696 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:04.694+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.695476+0000) 2020-02-06T00:28:04.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:04.694+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.696077+0000) 2020-02-06T00:28:04.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:04.695+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.696327+0000) 2020-02-06T00:28:04.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:04.695+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.696775+0000) 2020-02-06T00:28:04.698 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:04.695+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.696493+0000) 2020-02-06T00:28:04.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:04.699+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:04.700524+0000) 2020-02-06T00:28:04.712 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:04.709+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:06.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:06.012+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:06.014006+0000) 2020-02-06T00:28:06.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:06.360+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:06.361336+0000) 2020-02-06T00:28:07.003 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:07.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:07.002361+0000) 2020-02-06T00:28:07.267 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:07.265+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:07.266135+0000) 2020-02-06T00:28:07.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:07.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:07.266137+0000) 2020-02-06T00:28:07.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:07.265+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:07.266333+0000) 2020-02-06T00:28:09.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:09.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.646019+0000) 2020-02-06T00:28:09.718 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:09.715+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.716623+0000) 2020-02-06T00:28:09.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:09.716+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.717367+0000) 2020-02-06T00:28:09.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:09.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.718460+0000) 2020-02-06T00:28:09.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:09.717+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.719015+0000) 2020-02-06T00:28:09.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:09.718+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.719516+0000) 2020-02-06T00:28:09.721 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:09.718+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.719450+0000) 2020-02-06T00:28:09.721 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:09.719+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.720144+0000) 2020-02-06T00:28:09.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:09.719+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.720178+0000) 2020-02-06T00:28:09.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:09.719+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.721019+0000) 2020-02-06T00:28:09.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:09.720+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.721230+0000) 2020-02-06T00:28:09.723 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:09.720+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.721523+0000) 2020-02-06T00:28:09.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:09.720+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.722333+0000) 2020-02-06T00:28:09.723 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:09.721+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.722620+0000) 2020-02-06T00:28:09.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:09.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.726397+0000) 2020-02-06T00:28:09.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:09.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:09.790 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:28:09.795 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:28:09.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:09.894+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:09.895120+0000) 2020-02-06T00:28:10.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:10.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:10.494677+0000) 2020-02-06T00:28:10.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:10.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:10.754829+0000) 2020-02-06T00:28:12.877 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:12.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:12.876317+0000) 2020-02-06T00:28:13.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:13.107+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:13.108911+0000) 2020-02-06T00:28:14.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:14.025+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.026035+0000) 2020-02-06T00:28:14.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:14.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.026239+0000) 2020-02-06T00:28:14.743 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:14.740+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.741583+0000) 2020-02-06T00:28:14.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:14.741+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.742293+0000) 2020-02-06T00:28:14.744 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:14.741+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.743088+0000) 2020-02-06T00:28:14.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:14.741+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.742944+0000) 2020-02-06T00:28:14.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:14.742+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.743668+0000) 2020-02-06T00:28:14.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:14.742+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.744027+0000) 2020-02-06T00:28:14.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:14.743+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.744557+0000) 2020-02-06T00:28:14.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:14.743+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.744608+0000) 2020-02-06T00:28:14.746 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:14.744+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.745228+0000) 2020-02-06T00:28:14.746 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:14.743+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.745286+0000) 2020-02-06T00:28:14.747 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:14.744+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.745837+0000) 2020-02-06T00:28:14.747 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:14.745+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.746657+0000) 2020-02-06T00:28:14.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:14.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.746753+0000) 2020-02-06T00:28:14.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:14.749+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:14.750111+0000) 2020-02-06T00:28:14.764 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:14.761+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174863 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:16.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:16.013+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:16.014265+0000) 2020-02-06T00:28:16.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:16.360+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:16.361568+0000) 2020-02-06T00:28:17.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:17.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:17.002593+0000) 2020-02-06T00:28:17.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:17.265+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:17.266479+0000) 2020-02-06T00:28:17.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:17.265+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:17.266484+0000) 2020-02-06T00:28:17.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:17.265+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:17.266671+0000) 2020-02-06T00:28:19.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:19.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.646285+0000) 2020-02-06T00:28:19.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:19.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.766957+0000) 2020-02-06T00:28:19.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:19.769+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.769948+0000) 2020-02-06T00:28:19.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:19.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.770507+0000) 2020-02-06T00:28:19.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:19.771+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.771068+0000) 2020-02-06T00:28:19.772 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:19.769+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.771026+0000) 2020-02-06T00:28:19.773 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:19.770+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.771641+0000) 2020-02-06T00:28:19.773 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:19.770+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.771775+0000) 2020-02-06T00:28:19.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:19.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.773122+0000) 2020-02-06T00:28:19.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:19.773+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.773780+0000) 2020-02-06T00:28:19.775 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:19.772+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.773577+0000) 2020-02-06T00:28:19.776 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:19.772+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.774216+0000) 2020-02-06T00:28:19.776 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:19.775+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.775452+0000) 2020-02-06T00:28:19.776 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:19.774+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.775580+0000) 2020-02-06T00:28:19.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:19.777+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.777495+0000) 2020-02-06T00:28:19.789 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:19.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:19.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:19.895+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:19.895308+0000) 2020-02-06T00:28:20.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:20.493+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:20.494899+0000) 2020-02-06T00:28:20.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:20.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:20.755028+0000) 2020-02-06T00:28:22.878 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:22.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:22.876564+0000) 2020-02-06T00:28:23.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:23.116+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:23.117508+0000) 2020-02-06T00:28:24.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:24.025+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.026300+0000) 2020-02-06T00:28:24.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:24.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.026462+0000) 2020-02-06T00:28:24.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:24.794+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.795392+0000) 2020-02-06T00:28:24.797 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:24.793+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.795287+0000) 2020-02-06T00:28:24.797 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:24.795+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.796062+0000) 2020-02-06T00:28:24.798 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:24.795+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.796539+0000) 2020-02-06T00:28:24.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:24.795+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.796969+0000) 2020-02-06T00:28:24.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:24.796+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.797489+0000) 2020-02-06T00:28:24.800 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:24.796+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.797960+0000) 2020-02-06T00:28:24.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:24.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.798333+0000) 2020-02-06T00:28:24.801 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:24.797+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.798638+0000) 2020-02-06T00:28:24.801 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:24.797+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.799048+0000) 2020-02-06T00:28:24.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:24.798+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.799760+0000) 2020-02-06T00:28:24.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:24.798+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.800213+0000) 2020-02-06T00:28:24.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:24.798+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.799566+0000) 2020-02-06T00:28:24.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:24.804+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:24.805390+0000) 2020-02-06T00:28:24.813 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:24.810+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:26.015 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:26.013+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:26.014521+0000) 2020-02-06T00:28:26.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:26.360+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:26.361794+0000) 2020-02-06T00:28:27.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:27.001+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:27.002851+0000) 2020-02-06T00:28:27.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:27.266+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:27.266763+0000) 2020-02-06T00:28:27.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:27.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:27.266747+0000) 2020-02-06T00:28:27.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:27.266+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:27.266880+0000) 2020-02-06T00:28:29.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:29.645+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.646519+0000) 2020-02-06T00:28:29.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:29.816+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.817484+0000) 2020-02-06T00:28:29.820 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:29.817+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.818816+0000) 2020-02-06T00:28:29.820 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:29.817+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.818960+0000) 2020-02-06T00:28:29.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:29.819+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.820236+0000) 2020-02-06T00:28:29.822 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:29.819+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.820554+0000) 2020-02-06T00:28:29.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:29.819+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.820927+0000) 2020-02-06T00:28:29.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:29.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.821888+0000) 2020-02-06T00:28:29.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:29.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.822352+0000) 2020-02-06T00:28:29.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:29.821+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.822586+0000) 2020-02-06T00:28:29.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:29.822+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.823587+0000) 2020-02-06T00:28:29.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:29.822+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.823956+0000) 2020-02-06T00:28:29.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:29.822+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.823776+0000) 2020-02-06T00:28:29.826 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:29.822+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.824320+0000) 2020-02-06T00:28:29.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:29.830+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.831246+0000) 2020-02-06T00:28:29.839 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:29.836+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 174985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:29.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:29.894+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:29.895519+0000) 2020-02-06T00:28:30.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:30.494+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:30.495099+0000) 2020-02-06T00:28:30.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:30.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:30.755261+0000) 2020-02-06T00:28:32.878 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:32.875+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:32.876793+0000) 2020-02-06T00:28:33.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:33.124+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:33.124910+0000) 2020-02-06T00:28:34.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:34.025+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.026540+0000) 2020-02-06T00:28:34.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:34.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.026733+0000) 2020-02-06T00:28:34.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:34.842+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.843235+0000) 2020-02-06T00:28:34.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:34.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.843468+0000) 2020-02-06T00:28:34.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:34.843+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.844868+0000) 2020-02-06T00:28:34.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:34.843+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.845305+0000) 2020-02-06T00:28:34.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:34.844+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.845473+0000) 2020-02-06T00:28:34.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:34.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.846122+0000) 2020-02-06T00:28:34.847 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:34.845+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.846442+0000) 2020-02-06T00:28:34.848 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:34.846+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.847463+0000) 2020-02-06T00:28:34.849 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:34.846+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.848177+0000) 2020-02-06T00:28:34.850 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:34.847+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.848726+0000) 2020-02-06T00:28:34.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:34.848+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.849318+0000) 2020-02-06T00:28:34.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:34.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.849433+0000) 2020-02-06T00:28:34.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:34.848+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.849461+0000) 2020-02-06T00:28:34.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:34.851+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:34.852210+0000) 2020-02-06T00:28:34.864 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:34.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:36.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:36.014+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:36.014784+0000) 2020-02-06T00:28:36.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:36.361+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:36.362023+0000) 2020-02-06T00:28:37.004 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:37.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:37.003127+0000) 2020-02-06T00:28:37.268 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:37.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:37.267058+0000) 2020-02-06T00:28:37.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:37.266+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:37.267058+0000) 2020-02-06T00:28:37.269 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:37.266+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:37.267162+0000) 2020-02-06T00:28:39.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:39.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.646752+0000) 2020-02-06T00:28:39.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:39.868+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.868935+0000) 2020-02-06T00:28:39.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:39.868+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.869379+0000) 2020-02-06T00:28:39.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:39.869+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.870463+0000) 2020-02-06T00:28:39.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:39.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.871297+0000) 2020-02-06T00:28:39.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:39.870+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.871939+0000) 2020-02-06T00:28:39.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:39.871+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.872309+0000) 2020-02-06T00:28:39.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:39.871+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.872656+0000) 2020-02-06T00:28:39.874 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:39.871+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.873129+0000) 2020-02-06T00:28:39.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:39.872+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.873622+0000) 2020-02-06T00:28:39.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:39.873+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.874202+0000) 2020-02-06T00:28:39.875 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:39.872+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.873989+0000) 2020-02-06T00:28:39.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:39.873+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.874940+0000) 2020-02-06T00:28:39.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:39.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.875361+0000) 2020-02-06T00:28:39.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:39.880+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.882051+0000) 2020-02-06T00:28:39.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:39.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:39.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:39.894+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:39.895678+0000) 2020-02-06T00:28:39.926 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:28:39.931 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:28:40.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:40.494+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:40.495315+0000) 2020-02-06T00:28:40.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:40.754+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:40.755473+0000) 2020-02-06T00:28:42.878 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:42.876+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:42.877086+0000) 2020-02-06T00:28:43.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:43.133+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:43.133806+0000) 2020-02-06T00:28:44.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:44.025+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.026786+0000) 2020-02-06T00:28:44.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:44.025+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.026944+0000) 2020-02-06T00:28:44.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:44.891+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.892343+0000) 2020-02-06T00:28:44.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:44.893+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.894442+0000) 2020-02-06T00:28:44.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:44.895+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.895517+0000) 2020-02-06T00:28:44.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:44.895+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.895819+0000) 2020-02-06T00:28:44.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:44.895+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.896571+0000) 2020-02-06T00:28:44.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:44.895+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.896663+0000) 2020-02-06T00:28:44.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:44.895+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.897019+0000) 2020-02-06T00:28:44.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:44.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.897659+0000) 2020-02-06T00:28:44.900 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:44.897+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.898375+0000) 2020-02-06T00:28:44.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:44.898+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.898660+0000) 2020-02-06T00:28:44.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:44.898+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.899111+0000) 2020-02-06T00:28:44.901 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:44.897+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.898920+0000) 2020-02-06T00:28:44.901 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:44.898+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.899322+0000) 2020-02-06T00:28:44.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:44.903+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:44.904613+0000) 2020-02-06T00:28:44.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:44.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:46.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:46.014+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:46.015054+0000) 2020-02-06T00:28:46.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:46.361+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:46.362336+0000) 2020-02-06T00:28:47.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:47.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:47.003357+0000) 2020-02-06T00:28:47.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:47.266+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:47.267356+0000) 2020-02-06T00:28:47.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:47.266+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:47.267356+0000) 2020-02-06T00:28:47.269 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:47.266+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:47.267400+0000) 2020-02-06T00:28:49.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:49.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.646984+0000) 2020-02-06T00:28:49.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:49.895+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.895871+0000) 2020-02-06T00:28:49.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:49.915+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.916845+0000) 2020-02-06T00:28:49.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:49.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.917591+0000) 2020-02-06T00:28:49.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:49.918+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.919059+0000) 2020-02-06T00:28:49.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:49.919+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.920129+0000) 2020-02-06T00:28:49.921 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:49.919+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.920028+0000) 2020-02-06T00:28:49.922 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:49.920+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.920907+0000) 2020-02-06T00:28:49.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:49.920+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.921303+0000) 2020-02-06T00:28:49.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:49.921+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.922204+0000) 2020-02-06T00:28:49.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:49.922+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.922550+0000) 2020-02-06T00:28:49.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:49.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.923165+0000) 2020-02-06T00:28:49.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:49.922+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.923287+0000) 2020-02-06T00:28:49.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:49.922+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.923351+0000) 2020-02-06T00:28:49.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:49.923+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.923870+0000) 2020-02-06T00:28:49.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:49.926+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:49.927156+0000) 2020-02-06T00:28:49.938 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:49.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:50.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:50.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:50.495533+0000) 2020-02-06T00:28:50.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:50.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:50.755685+0000) 2020-02-06T00:28:52.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:52.876+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:52.877320+0000) 2020-02-06T00:28:53.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:53.141+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:53.142520+0000) 2020-02-06T00:28:54.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:54.026+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.027003+0000) 2020-02-06T00:28:54.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:54.026+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.027169+0000) 2020-02-06T00:28:54.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:54.940+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.941916+0000) 2020-02-06T00:28:54.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:54.942+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.943753+0000) 2020-02-06T00:28:54.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:54.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.944302+0000) 2020-02-06T00:28:54.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:54.944+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.946021+0000) 2020-02-06T00:28:54.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:54.945+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.945414+0000) 2020-02-06T00:28:54.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:54.945+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.946483+0000) 2020-02-06T00:28:54.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:54.945+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.946739+0000) 2020-02-06T00:28:54.948 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:54.946+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.947177+0000) 2020-02-06T00:28:54.949 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:54.947+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.947953+0000) 2020-02-06T00:28:54.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:54.948+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.948775+0000) 2020-02-06T00:28:54.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:54.948+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.948911+0000) 2020-02-06T00:28:54.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:54.947+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.948186+0000) 2020-02-06T00:28:54.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:54.947+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.948948+0000) 2020-02-06T00:28:54.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:54.951+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:54.952821+0000) 2020-02-06T00:28:54.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:54.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175192 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:28:56.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:56.014+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:56.015311+0000) 2020-02-06T00:28:56.364 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:56.361+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:56.362589+0000) 2020-02-06T00:28:57.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:57.002+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:57.003605+0000) 2020-02-06T00:28:57.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:57.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:57.267707+0000) 2020-02-06T00:28:57.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:57.267+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:57.267708+0000) 2020-02-06T00:28:57.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:57.267+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:57.267698+0000) 2020-02-06T00:28:59.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:59.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.647262+0000) 2020-02-06T00:28:59.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:59.895+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.896132+0000) 2020-02-06T00:28:59.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:59.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.968041+0000) 2020-02-06T00:28:59.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:59.967+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.968732+0000) 2020-02-06T00:28:59.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:28:59.968+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.969235+0000) 2020-02-06T00:28:59.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:28:59.968+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.969751+0000) 2020-02-06T00:28:59.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:28:59.970+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.970533+0000) 2020-02-06T00:28:59.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:28:59.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.970929+0000) 2020-02-06T00:28:59.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:28:59.969+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.970984+0000) 2020-02-06T00:28:59.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:28:59.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.971594+0000) 2020-02-06T00:28:59.973 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:28:59.971+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.971587+0000) 2020-02-06T00:28:59.973 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:28:59.970+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.971940+0000) 2020-02-06T00:28:59.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:28:59.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.972612+0000) 2020-02-06T00:28:59.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:28:59.972+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.972634+0000) 2020-02-06T00:28:59.974 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:28:59.972+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.973258+0000) 2020-02-06T00:28:59.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:28:59.975+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:28:59.976109+0000) 2020-02-06T00:28:59.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:28:59.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175233 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:00.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:00.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:00.495745+0000) 2020-02-06T00:29:00.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:00.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:00.755927+0000) 2020-02-06T00:29:02.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:02.877+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:02.877587+0000) 2020-02-06T00:29:03.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:03.150+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:03.151037+0000) 2020-02-06T00:29:04.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:04.027+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.027279+0000) 2020-02-06T00:29:04.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:04.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.027405+0000) 2020-02-06T00:29:04.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:04.993+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.994868+0000) 2020-02-06T00:29:04.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:04.994+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.995389+0000) 2020-02-06T00:29:04.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:04.997+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.997528+0000) 2020-02-06T00:29:04.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:04.997+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.998368+0000) 2020-02-06T00:29:04.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:04.997+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.998358+0000) 2020-02-06T00:29:05.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:04.998+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.998608+0000) 2020-02-06T00:29:05.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:04.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.999219+0000) 2020-02-06T00:29:05.001 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:04.998+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.999627+0000) 2020-02-06T00:29:05.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:04.998+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:04.999331+0000) 2020-02-06T00:29:05.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:04.999+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:05.000355+0000) 2020-02-06T00:29:05.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:04.999+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:05.000163+0000) 2020-02-06T00:29:05.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:04.999+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:05.000059+0000) 2020-02-06T00:29:05.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:05.001+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:05.002804+0000) 2020-02-06T00:29:05.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:05.002+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:05.003072+0000) 2020-02-06T00:29:05.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:05.013+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:06.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:06.015+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:06.015563+0000) 2020-02-06T00:29:06.364 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:06.362+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:06.362811+0000) 2020-02-06T00:29:07.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:07.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:07.003842+0000) 2020-02-06T00:29:07.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:07.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:07.267995+0000) 2020-02-06T00:29:07.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:07.267+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:07.267987+0000) 2020-02-06T00:29:07.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:07.267+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:07.267992+0000) 2020-02-06T00:29:09.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:09.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:09.647500+0000) 2020-02-06T00:29:09.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:09.896+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:09.896390+0000) 2020-02-06T00:29:10.020 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:10.017+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.018807+0000) 2020-02-06T00:29:10.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:10.018+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.019853+0000) 2020-02-06T00:29:10.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:10.019+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.020383+0000) 2020-02-06T00:29:10.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:10.019+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.020696+0000) 2020-02-06T00:29:10.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:10.021+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.021801+0000) 2020-02-06T00:29:10.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:10.021+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.022267+0000) 2020-02-06T00:29:10.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:10.022+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.023072+0000) 2020-02-06T00:29:10.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:10.023+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.023608+0000) 2020-02-06T00:29:10.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:10.023+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.024137+0000) 2020-02-06T00:29:10.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:10.023+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.024166+0000) 2020-02-06T00:29:10.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:10.023+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.024368+0000) 2020-02-06T00:29:10.026 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:10.024+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.024885+0000) 2020-02-06T00:29:10.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:10.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.029742+0000) 2020-02-06T00:29:10.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:10.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.029963+0000) 2020-02-06T00:29:10.038 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:10.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:10.056 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:29:10.061 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:29:10.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:10.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.495952+0000) 2020-02-06T00:29:10.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:10.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:10.756143+0000) 2020-02-06T00:29:12.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:12.877+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:12.877889+0000) 2020-02-06T00:29:13.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:13.159+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:13.159599+0000) 2020-02-06T00:29:14.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:14.027+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:14.027566+0000) 2020-02-06T00:29:14.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:14.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:14.027660+0000) 2020-02-06T00:29:15.044 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:15.043+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.043401+0000) 2020-02-06T00:29:15.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:15.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.044541+0000) 2020-02-06T00:29:15.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:15.044+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.044978+0000) 2020-02-06T00:29:15.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:15.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.045254+0000) 2020-02-06T00:29:15.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:15.045+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.045765+0000) 2020-02-06T00:29:15.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:15.046+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.047136+0000) 2020-02-06T00:29:15.048 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:15.046+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.046840+0000) 2020-02-06T00:29:15.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:15.047+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.047893+0000) 2020-02-06T00:29:15.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:15.048+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.048404+0000) 2020-02-06T00:29:15.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:15.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.048271+0000) 2020-02-06T00:29:15.050 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:15.048+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.048916+0000) 2020-02-06T00:29:15.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:15.049+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.049421+0000) 2020-02-06T00:29:15.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:15.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.054323+0000) 2020-02-06T00:29:15.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:15.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:15.054616+0000) 2020-02-06T00:29:15.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:15.061+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175358 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:16.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:16.015+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:16.015817+0000) 2020-02-06T00:29:16.364 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:16.362+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:16.363068+0000) 2020-02-06T00:29:17.005 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:17.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:17.004133+0000) 2020-02-06T00:29:17.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:17.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:17.268283+0000) 2020-02-06T00:29:17.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:17.268+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:17.268283+0000) 2020-02-06T00:29:17.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:17.268+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:17.268257+0000) 2020-02-06T00:29:19.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:19.646+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:19.647720+0000) 2020-02-06T00:29:19.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:19.895+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:19.896646+0000) 2020-02-06T00:29:20.068 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:20.065+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.067015+0000) 2020-02-06T00:29:20.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:20.066+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.067874+0000) 2020-02-06T00:29:20.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:20.067+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.068238+0000) 2020-02-06T00:29:20.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:20.067+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.069062+0000) 2020-02-06T00:29:20.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:20.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.069554+0000) 2020-02-06T00:29:20.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:20.069+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.070184+0000) 2020-02-06T00:29:20.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:20.069+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.070567+0000) 2020-02-06T00:29:20.072 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:20.070+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.071146+0000) 2020-02-06T00:29:20.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:20.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.072452+0000) 2020-02-06T00:29:20.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:20.071+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.072832+0000) 2020-02-06T00:29:20.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:20.071+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.073014+0000) 2020-02-06T00:29:20.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:20.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.073763+0000) 2020-02-06T00:29:20.075 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:20.072+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.073901+0000) 2020-02-06T00:29:20.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:20.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.078789+0000) 2020-02-06T00:29:20.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:20.086+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175399 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:20.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:20.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.496196+0000) 2020-02-06T00:29:20.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:20.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:20.756350+0000) 2020-02-06T00:29:22.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:22.876+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:22.878141+0000) 2020-02-06T00:29:23.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:23.166+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:23.167319+0000) 2020-02-06T00:29:24.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:24.026+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:24.027801+0000) 2020-02-06T00:29:24.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:24.026+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:24.027925+0000) 2020-02-06T00:29:25.094 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:25.092+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.093435+0000) 2020-02-06T00:29:25.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:25.093+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.094639+0000) 2020-02-06T00:29:25.096 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:25.093+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.095154+0000) 2020-02-06T00:29:25.096 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:25.094+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.095417+0000) 2020-02-06T00:29:25.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:25.094+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.095995+0000) 2020-02-06T00:29:25.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:25.095+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.096512+0000) 2020-02-06T00:29:25.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:25.096+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.097394+0000) 2020-02-06T00:29:25.098 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:25.095+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.097044+0000) 2020-02-06T00:29:25.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:25.096+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.098129+0000) 2020-02-06T00:29:25.099 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:25.097+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.098184+0000) 2020-02-06T00:29:25.100 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:25.097+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.098699+0000) 2020-02-06T00:29:25.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:25.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.099364+0000) 2020-02-06T00:29:25.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:25.097+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.099402+0000) 2020-02-06T00:29:25.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:25.101+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:25.102126+0000) 2020-02-06T00:29:25.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:25.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175442 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:26.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:26.015+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:26.016053+0000) 2020-02-06T00:29:26.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:26.362+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:26.363398+0000) 2020-02-06T00:29:27.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:27.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:27.004366+0000) 2020-02-06T00:29:27.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:27.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:27.268592+0000) 2020-02-06T00:29:27.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:27.267+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:27.268589+0000) 2020-02-06T00:29:27.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:27.267+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:27.268579+0000) 2020-02-06T00:29:29.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:29.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:29.647995+0000) 2020-02-06T00:29:29.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:29.896+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:29.896889+0000) 2020-02-06T00:29:30.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:30.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.117867+0000) 2020-02-06T00:29:30.119 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:30.117+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.118654+0000) 2020-02-06T00:29:30.120 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:30.118+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.119446+0000) 2020-02-06T00:29:30.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:30.119+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.120601+0000) 2020-02-06T00:29:30.122 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:30.120+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.121445+0000) 2020-02-06T00:29:30.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:30.120+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.122396+0000) 2020-02-06T00:29:30.124 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:30.120+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.122377+0000) 2020-02-06T00:29:30.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:30.121+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.123177+0000) 2020-02-06T00:29:30.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:30.121+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.122703+0000) 2020-02-06T00:29:30.125 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:30.122+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.123642+0000) 2020-02-06T00:29:30.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:30.123+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.124128+0000) 2020-02-06T00:29:30.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:30.122+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.123977+0000) 2020-02-06T00:29:30.126 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:30.122+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.123522+0000) 2020-02-06T00:29:30.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:30.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.126111+0000) 2020-02-06T00:29:30.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:30.136+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:30.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:30.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.496455+0000) 2020-02-06T00:29:30.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:30.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:30.756581+0000) 2020-02-06T00:29:32.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:32.876+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:32.878384+0000) 2020-02-06T00:29:33.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:33.174+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:33.174851+0000) 2020-02-06T00:29:34.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:34.027+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:34.028037+0000) 2020-02-06T00:29:34.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:34.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:34.028154+0000) 2020-02-06T00:29:35.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:35.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.143481+0000) 2020-02-06T00:29:35.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:35.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.143981+0000) 2020-02-06T00:29:35.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:35.143+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.144943+0000) 2020-02-06T00:29:35.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:35.144+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.145615+0000) 2020-02-06T00:29:35.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:35.144+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.145749+0000) 2020-02-06T00:29:35.148 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:35.145+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.147055+0000) 2020-02-06T00:29:35.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:35.146+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.147322+0000) 2020-02-06T00:29:35.149 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:35.146+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.147707+0000) 2020-02-06T00:29:35.150 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:35.146+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.148078+0000) 2020-02-06T00:29:35.150 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:35.146+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.148122+0000) 2020-02-06T00:29:35.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:35.147+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.148723+0000) 2020-02-06T00:29:35.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:35.147+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.148971+0000) 2020-02-06T00:29:35.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:35.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.149494+0000) 2020-02-06T00:29:35.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:35.154+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:35.155356+0000) 2020-02-06T00:29:35.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:35.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175524 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:36.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:36.015+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:36.016276+0000) 2020-02-06T00:29:36.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:36.362+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:36.363659+0000) 2020-02-06T00:29:37.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:37.003+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:37.004602+0000) 2020-02-06T00:29:37.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:37.267+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:37.268880+0000) 2020-02-06T00:29:37.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:37.267+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:37.268883+0000) 2020-02-06T00:29:37.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:37.267+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:37.268861+0000) 2020-02-06T00:29:39.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:39.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:39.648267+0000) 2020-02-06T00:29:39.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:39.896+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:39.897122+0000) 2020-02-06T00:29:40.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:40.166+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.167422+0000) 2020-02-06T00:29:40.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:40.167+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.168566+0000) 2020-02-06T00:29:40.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:40.168+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.169271+0000) 2020-02-06T00:29:40.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:40.168+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.170008+0000) 2020-02-06T00:29:40.171 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:40.168+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.170130+0000) 2020-02-06T00:29:40.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:40.169+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.170786+0000) 2020-02-06T00:29:40.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:40.169+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.171291+0000) 2020-02-06T00:29:40.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:40.170+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.172026+0000) 2020-02-06T00:29:40.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:40.170+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.172166+0000) 2020-02-06T00:29:40.174 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:40.171+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.172453+0000) 2020-02-06T00:29:40.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:40.171+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.172969+0000) 2020-02-06T00:29:40.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:40.172+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.173602+0000) 2020-02-06T00:29:40.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:40.173+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.174310+0000) 2020-02-06T00:29:40.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:40.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.180676+0000) 2020-02-06T00:29:40.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:40.185+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175564 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:40.265 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:29:40.272 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:29:40.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:40.495+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.496708+0000) 2020-02-06T00:29:40.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:40.755+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:40.756851+0000) 2020-02-06T00:29:42.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:42.877+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:42.878736+0000) 2020-02-06T00:29:43.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:43.182+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:43.183462+0000) 2020-02-06T00:29:44.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:44.027+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:44.028304+0000) 2020-02-06T00:29:44.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:44.027+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:44.028382+0000) 2020-02-06T00:29:45.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:45.194+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.195893+0000) 2020-02-06T00:29:45.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:45.195+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.197149+0000) 2020-02-06T00:29:45.200 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:45.191+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.192803+0000) 2020-02-06T00:29:45.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:45.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.194293+0000) 2020-02-06T00:29:45.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:45.196+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.197578+0000) 2020-02-06T00:29:45.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:45.194+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.195154+0000) 2020-02-06T00:29:45.201 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:45.194+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.195426+0000) 2020-02-06T00:29:45.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:45.195+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.196149+0000) 2020-02-06T00:29:45.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:45.197+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.198283+0000) 2020-02-06T00:29:45.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:45.197+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.199067+0000) 2020-02-06T00:29:45.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:45.197+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.199336+0000) 2020-02-06T00:29:45.202 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:45.198+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.200009+0000) 2020-02-06T00:29:45.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:45.198+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.199559+0000) 2020-02-06T00:29:45.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:45.201+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:45.202318+0000) 2020-02-06T00:29:45.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:45.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175606 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:46.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:46.015+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:46.016497+0000) 2020-02-06T00:29:46.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:46.363+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:46.363884+0000) 2020-02-06T00:29:47.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:47.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:47.004841+0000) 2020-02-06T00:29:47.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:47.268+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:47.269118+0000) 2020-02-06T00:29:47.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:47.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:47.269159+0000) 2020-02-06T00:29:47.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:47.268+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:47.269165+0000) 2020-02-06T00:29:49.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:49.647+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:49.648533+0000) 2020-02-06T00:29:49.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:49.896+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:49.897399+0000) 2020-02-06T00:29:50.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:50.216+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.217236+0000) 2020-02-06T00:29:50.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:50.217+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.218490+0000) 2020-02-06T00:29:50.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:50.217+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.218690+0000) 2020-02-06T00:29:50.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:50.218+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.220245+0000) 2020-02-06T00:29:50.223 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:50.217+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.218889+0000) 2020-02-06T00:29:50.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:50.218+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.219351+0000) 2020-02-06T00:29:50.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:50.218+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.219935+0000) 2020-02-06T00:29:50.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:50.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.220596+0000) 2020-02-06T00:29:50.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:50.220+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.221521+0000) 2020-02-06T00:29:50.225 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:50.220+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.221839+0000) 2020-02-06T00:29:50.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:50.221+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.222897+0000) 2020-02-06T00:29:50.225 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:50.221+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.222936+0000) 2020-02-06T00:29:50.225 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:50.222+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.223919+0000) 2020-02-06T00:29:50.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:50.225+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.227026+0000) 2020-02-06T00:29:50.237 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:50.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175646 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:50.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:50.496+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.496875+0000) 2020-02-06T00:29:50.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:50.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:50.757131+0000) 2020-02-06T00:29:52.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:52.878+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:52.878992+0000) 2020-02-06T00:29:53.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:53.191+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:53.191920+0000) 2020-02-06T00:29:54.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:54.028+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:54.028534+0000) 2020-02-06T00:29:54.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:54.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:54.028631+0000) 2020-02-06T00:29:55.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:55.242+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.243762+0000) 2020-02-06T00:29:55.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:55.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.244388+0000) 2020-02-06T00:29:55.246 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:29:55.243+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.244741+0000) 2020-02-06T00:29:55.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:29:55.243+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.244959+0000) 2020-02-06T00:29:55.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:29:55.244+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.245321+0000) 2020-02-06T00:29:55.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:29:55.244+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.245698+0000) 2020-02-06T00:29:55.247 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:29:55.244+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.245999+0000) 2020-02-06T00:29:55.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:55.245+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.246488+0000) 2020-02-06T00:29:55.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:55.245+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.247205+0000) 2020-02-06T00:29:55.248 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:55.246+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.247469+0000) 2020-02-06T00:29:55.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:55.246+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.247957+0000) 2020-02-06T00:29:55.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:55.246+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.248243+0000) 2020-02-06T00:29:55.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:55.248+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.249402+0000) 2020-02-06T00:29:55.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:55.249+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:55.250544+0000) 2020-02-06T00:29:55.265 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:29:55.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175688 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:29:56.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:29:56.016+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:56.016722+0000) 2020-02-06T00:29:56.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:29:56.363+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:56.364117+0000) 2020-02-06T00:29:57.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:29:57.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:57.005101+0000) 2020-02-06T00:29:57.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:29:57.268+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:57.269317+0000) 2020-02-06T00:29:57.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:29:57.268+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:57.269408+0000) 2020-02-06T00:29:57.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:29:57.268+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:57.269491+0000) 2020-02-06T00:29:59.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:59.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:59.648735+0000) 2020-02-06T00:29:59.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:29:59.897+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:29:59.897633+0000) 2020-02-06T00:30:00.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:00.269+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.270517+0000) 2020-02-06T00:30:00.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:00.269+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.270983+0000) 2020-02-06T00:30:00.273 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:00.270+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.271566+0000) 2020-02-06T00:30:00.273 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:00.271+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.272104+0000) 2020-02-06T00:30:00.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:00.272+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.273165+0000) 2020-02-06T00:30:00.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:00.272+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.273478+0000) 2020-02-06T00:30:00.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:00.272+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.273820+0000) 2020-02-06T00:30:00.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:00.271+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.272784+0000) 2020-02-06T00:30:00.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:00.272+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.274234+0000) 2020-02-06T00:30:00.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:00.273+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.274607+0000) 2020-02-06T00:30:00.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:00.273+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.274716+0000) 2020-02-06T00:30:00.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:00.273+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.275103+0000) 2020-02-06T00:30:00.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:00.274+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.275722+0000) 2020-02-06T00:30:00.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:00.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.278302+0000) 2020-02-06T00:30:00.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:00.287+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:00.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:00.496+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.497105+0000) 2020-02-06T00:30:00.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:00.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:00.757358+0000) 2020-02-06T00:30:02.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:02.878+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:02.879259+0000) 2020-02-06T00:30:03.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:03.200+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:03.200533+0000) 2020-02-06T00:30:04.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:04.028+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:04.028763+0000) 2020-02-06T00:30:04.030 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:04.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:04.028871+0000) 2020-02-06T00:30:05.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:05.293+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.294786+0000) 2020-02-06T00:30:05.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:05.294+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.296093+0000) 2020-02-06T00:30:05.299 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:05.296+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.297627+0000) 2020-02-06T00:30:05.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:05.295+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.297091+0000) 2020-02-06T00:30:05.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:05.297+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.298366+0000) 2020-02-06T00:30:05.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:05.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.298612+0000) 2020-02-06T00:30:05.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:05.296+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.297992+0000) 2020-02-06T00:30:05.300 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:05.298+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.299119+0000) 2020-02-06T00:30:05.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:05.298+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.299552+0000) 2020-02-06T00:30:05.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:05.298+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.300190+0000) 2020-02-06T00:30:05.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:05.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.300176+0000) 2020-02-06T00:30:05.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:05.299+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.300572+0000) 2020-02-06T00:30:05.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:05.298+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.300364+0000) 2020-02-06T00:30:05.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:05.304+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:05.305755+0000) 2020-02-06T00:30:05.316 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:05.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:06.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:06.016+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:06.016938+0000) 2020-02-06T00:30:06.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:06.364+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:06.364328+0000) 2020-02-06T00:30:07.006 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:07.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:07.005295+0000) 2020-02-06T00:30:07.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:07.269+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:07.269580+0000) 2020-02-06T00:30:07.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:07.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:07.269717+0000) 2020-02-06T00:30:07.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:07.269+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:07.269748+0000) 2020-02-06T00:30:09.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:09.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:09.648965+0000) 2020-02-06T00:30:09.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:09.896+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:09.897861+0000) 2020-02-06T00:30:10.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:10.319+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.320246+0000) 2020-02-06T00:30:10.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:10.319+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.320948+0000) 2020-02-06T00:30:10.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:10.321+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.321602+0000) 2020-02-06T00:30:10.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:10.323+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.323900+0000) 2020-02-06T00:30:10.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:10.323+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.324113+0000) 2020-02-06T00:30:10.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:10.323+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.324854+0000) 2020-02-06T00:30:10.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:10.323+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.324946+0000) 2020-02-06T00:30:10.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:10.324+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.324863+0000) 2020-02-06T00:30:10.327 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:10.324+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.325429+0000) 2020-02-06T00:30:10.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:10.325+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.325881+0000) 2020-02-06T00:30:10.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:10.325+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.325965+0000) 2020-02-06T00:30:10.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:10.325+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.326306+0000) 2020-02-06T00:30:10.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:10.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.327925+0000) 2020-02-06T00:30:10.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:10.327+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.328217+0000) 2020-02-06T00:30:10.342 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:10.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:10.446 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:30:10.452 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:30:10.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:10.496+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.497277+0000) 2020-02-06T00:30:10.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:10.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:10.757575+0000) 2020-02-06T00:30:12.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:12.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:12.879533+0000) 2020-02-06T00:30:13.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:13.208+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:13.209412+0000) 2020-02-06T00:30:14.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:14.027+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:14.028988+0000) 2020-02-06T00:30:14.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:14.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:14.029135+0000) 2020-02-06T00:30:15.347 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:15.345+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.345940+0000) 2020-02-06T00:30:15.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:15.346+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.347275+0000) 2020-02-06T00:30:15.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:15.346+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.347505+0000) 2020-02-06T00:30:15.349 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:15.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.348194+0000) 2020-02-06T00:30:15.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:15.347+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.348878+0000) 2020-02-06T00:30:15.351 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:15.349+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.350226+0000) 2020-02-06T00:30:15.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:15.349+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.349931+0000) 2020-02-06T00:30:15.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:15.350+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.350836+0000) 2020-02-06T00:30:15.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:15.351+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.352115+0000) 2020-02-06T00:30:15.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:15.351+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.352098+0000) 2020-02-06T00:30:15.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:15.351+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.352218+0000) 2020-02-06T00:30:15.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:15.352+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.352635+0000) 2020-02-06T00:30:15.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:15.351+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.352365+0000) 2020-02-06T00:30:15.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:15.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:15.357275+0000) 2020-02-06T00:30:15.366 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:15.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:16.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:16.016+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:16.017155+0000) 2020-02-06T00:30:16.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:16.363+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:16.364559+0000) 2020-02-06T00:30:17.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:17.004+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:17.005533+0000) 2020-02-06T00:30:17.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:17.269+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:17.269831+0000) 2020-02-06T00:30:17.271 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:17.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:17.269998+0000) 2020-02-06T00:30:17.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:17.269+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:17.269993+0000) 2020-02-06T00:30:19.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:19.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:19.649201+0000) 2020-02-06T00:30:19.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:19.897+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:19.898123+0000) 2020-02-06T00:30:20.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:20.369+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.370830+0000) 2020-02-06T00:30:20.373 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:20.370+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.371794+0000) 2020-02-06T00:30:20.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:20.372+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.373383+0000) 2020-02-06T00:30:20.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:20.372+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.373734+0000) 2020-02-06T00:30:20.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:20.373+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.374518+0000) 2020-02-06T00:30:20.376 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:20.374+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.374970+0000) 2020-02-06T00:30:20.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:20.375+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.375644+0000) 2020-02-06T00:30:20.377 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:20.374+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.375300+0000) 2020-02-06T00:30:20.377 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:20.375+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.376271+0000) 2020-02-06T00:30:20.378 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:20.376+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.376516+0000) 2020-02-06T00:30:20.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:20.376+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.376680+0000) 2020-02-06T00:30:20.379 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:20.375+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.376971+0000) 2020-02-06T00:30:20.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:20.376+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.377229+0000) 2020-02-06T00:30:20.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:20.380+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.382051+0000) 2020-02-06T00:30:20.392 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:20.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175895 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:20.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:20.497+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.497504+0000) 2020-02-06T00:30:20.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:20.756+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:20.757787+0000) 2020-02-06T00:30:22.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:22.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:22.879739+0000) 2020-02-06T00:30:23.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:23.217+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:23.218060+0000) 2020-02-06T00:30:24.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:24.028+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:24.029256+0000) 2020-02-06T00:30:24.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:24.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:24.029366+0000) 2020-02-06T00:30:25.398 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:25.395+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.396623+0000) 2020-02-06T00:30:25.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:25.396+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.397489+0000) 2020-02-06T00:30:25.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:25.397+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.398027+0000) 2020-02-06T00:30:25.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:25.397+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.398223+0000) 2020-02-06T00:30:25.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:25.397+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.398950+0000) 2020-02-06T00:30:25.400 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:25.398+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.399268+0000) 2020-02-06T00:30:25.401 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:25.398+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.399952+0000) 2020-02-06T00:30:25.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:25.400+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.401134+0000) 2020-02-06T00:30:25.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:25.401+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.401653+0000) 2020-02-06T00:30:25.403 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:25.401+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.401748+0000) 2020-02-06T00:30:25.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:25.401+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.402222+0000) 2020-02-06T00:30:25.403 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:25.402+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.402594+0000) 2020-02-06T00:30:25.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:25.405+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.406832+0000) 2020-02-06T00:30:25.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:25.406+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:25.407847+0000) 2020-02-06T00:30:25.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:25.416+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:26.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:26.016+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:26.017357+0000) 2020-02-06T00:30:26.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:26.364+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:26.364794+0000) 2020-02-06T00:30:27.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:27.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:27.005768+0000) 2020-02-06T00:30:27.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:27.269+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:27.270068+0000) 2020-02-06T00:30:27.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:27.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:27.270216+0000) 2020-02-06T00:30:27.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:27.269+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:27.270238+0000) 2020-02-06T00:30:29.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:29.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:29.649433+0000) 2020-02-06T00:30:29.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:29.897+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:29.898353+0000) 2020-02-06T00:30:30.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:30.423+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.424429+0000) 2020-02-06T00:30:30.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:30.423+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.424670+0000) 2020-02-06T00:30:30.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:30.424+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.425456+0000) 2020-02-06T00:30:30.428 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:30.424+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.425226+0000) 2020-02-06T00:30:30.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:30.426+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.426553+0000) 2020-02-06T00:30:30.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:30.426+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.426746+0000) 2020-02-06T00:30:30.429 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:30.426+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.427622+0000) 2020-02-06T00:30:30.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:30.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.427824+0000) 2020-02-06T00:30:30.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:30.427+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.428311+0000) 2020-02-06T00:30:30.430 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:30.427+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.428381+0000) 2020-02-06T00:30:30.430 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:30.427+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.429016+0000) 2020-02-06T00:30:30.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:30.428+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.428919+0000) 2020-02-06T00:30:30.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:30.429+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.430016+0000) 2020-02-06T00:30:30.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:30.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.435886+0000) 2020-02-06T00:30:30.444 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:30.442+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 175978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:30.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:30.497+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.497795+0000) 2020-02-06T00:30:30.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:30.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:30.757996+0000) 2020-02-06T00:30:32.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:32.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:32.879952+0000) 2020-02-06T00:30:33.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:33.226+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:33.226684+0000) 2020-02-06T00:30:34.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:34.028+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:34.029491+0000) 2020-02-06T00:30:34.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:34.028+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:34.029606+0000) 2020-02-06T00:30:35.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:35.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.450150+0000) 2020-02-06T00:30:35.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:35.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.450708+0000) 2020-02-06T00:30:35.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:35.449+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.450468+0000) 2020-02-06T00:30:35.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:35.450+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.451240+0000) 2020-02-06T00:30:35.453 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:35.451+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.451641+0000) 2020-02-06T00:30:35.453 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:35.451+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.452140+0000) 2020-02-06T00:30:35.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:35.451+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.452296+0000) 2020-02-06T00:30:35.454 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:35.451+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.452985+0000) 2020-02-06T00:30:35.454 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:35.452+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.453398+0000) 2020-02-06T00:30:35.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:35.453+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.453461+0000) 2020-02-06T00:30:35.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:35.452+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.454029+0000) 2020-02-06T00:30:35.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:35.453+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.454213+0000) 2020-02-06T00:30:35.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:35.453+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.454216+0000) 2020-02-06T00:30:35.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:35.456+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:35.457637+0000) 2020-02-06T00:30:35.469 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:35.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:36.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:36.017+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:36.017577+0000) 2020-02-06T00:30:36.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:36.364+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:36.365010+0000) 2020-02-06T00:30:37.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:37.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:37.005999+0000) 2020-02-06T00:30:37.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:37.270+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:37.270304+0000) 2020-02-06T00:30:37.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:37.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:37.270496+0000) 2020-02-06T00:30:37.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:37.270+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:37.270527+0000) 2020-02-06T00:30:39.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:39.648+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:39.649624+0000) 2020-02-06T00:30:39.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:39.897+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:39.898616+0000) 2020-02-06T00:30:40.475 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:40.472+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.473759+0000) 2020-02-06T00:30:40.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:40.473+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.474567+0000) 2020-02-06T00:30:40.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:40.473+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.474884+0000) 2020-02-06T00:30:40.476 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:40.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.475502+0000) 2020-02-06T00:30:40.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:40.475+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.475578+0000) 2020-02-06T00:30:40.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:40.474+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.476027+0000) 2020-02-06T00:30:40.478 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:40.476+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.476600+0000) 2020-02-06T00:30:40.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:40.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.476999+0000) 2020-02-06T00:30:40.478 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:40.476+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.477221+0000) 2020-02-06T00:30:40.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:40.477+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.477422+0000) 2020-02-06T00:30:40.479 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:40.477+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.477552+0000) 2020-02-06T00:30:40.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:40.477+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.478109+0000) 2020-02-06T00:30:40.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:40.477+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.477861+0000) 2020-02-06T00:30:40.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:40.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.481814+0000) 2020-02-06T00:30:40.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:40.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:40.499 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:40.497+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.497935+0000) 2020-02-06T00:30:40.625 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:30:40.632 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:30:40.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:40.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:40.758176+0000) 2020-02-06T00:30:42.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:42.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:42.880239+0000) 2020-02-06T00:30:43.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:43.234+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:43.235196+0000) 2020-02-06T00:30:44.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:44.029+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:44.029727+0000) 2020-02-06T00:30:44.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:44.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:44.029861+0000) 2020-02-06T00:30:45.500 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:45.498+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.498878+0000) 2020-02-06T00:30:45.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:45.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.500197+0000) 2020-02-06T00:30:45.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:45.500+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.501367+0000) 2020-02-06T00:30:45.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:45.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.501658+0000) 2020-02-06T00:30:45.503 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:45.501+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.502413+0000) 2020-02-06T00:30:45.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:45.501+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.502295+0000) 2020-02-06T00:30:45.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:45.501+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.502838+0000) 2020-02-06T00:30:45.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:45.502+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.503317+0000) 2020-02-06T00:30:45.505 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:45.502+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.503792+0000) 2020-02-06T00:30:45.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:45.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.504137+0000) 2020-02-06T00:30:45.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:45.504+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.504492+0000) 2020-02-06T00:30:45.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:45.504+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.504895+0000) 2020-02-06T00:30:45.506 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:45.504+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.504841+0000) 2020-02-06T00:30:45.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:45.509+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:45.510352+0000) 2020-02-06T00:30:45.518 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:45.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176100 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:46.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:46.017+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:46.017791+0000) 2020-02-06T00:30:46.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:46.364+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:46.365263+0000) 2020-02-06T00:30:47.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:47.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:47.006303+0000) 2020-02-06T00:30:47.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:47.269+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:47.270575+0000) 2020-02-06T00:30:47.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:47.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:47.270787+0000) 2020-02-06T00:30:47.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:47.269+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:47.270797+0000) 2020-02-06T00:30:49.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:49.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:49.649876+0000) 2020-02-06T00:30:49.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:49.898+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:49.898841+0000) 2020-02-06T00:30:50.500 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:50.498+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.499297+0000) 2020-02-06T00:30:50.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:50.522+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.523191+0000) 2020-02-06T00:30:50.525 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:50.522+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.523692+0000) 2020-02-06T00:30:50.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:50.523+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.524439+0000) 2020-02-06T00:30:50.526 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:50.523+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.524856+0000) 2020-02-06T00:30:50.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:50.523+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.525398+0000) 2020-02-06T00:30:50.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:50.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.525748+0000) 2020-02-06T00:30:50.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:50.525+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.526302+0000) 2020-02-06T00:30:50.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:50.525+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.526506+0000) 2020-02-06T00:30:50.528 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:50.525+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.526832+0000) 2020-02-06T00:30:50.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:50.526+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.527681+0000) 2020-02-06T00:30:50.529 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:50.527+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.528481+0000) 2020-02-06T00:30:50.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:50.527+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.528630+0000) 2020-02-06T00:30:50.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:50.527+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.529211+0000) 2020-02-06T00:30:50.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:50.531+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.532165+0000) 2020-02-06T00:30:50.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:50.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:50.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:50.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:50.758413+0000) 2020-02-06T00:30:52.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:52.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:52.880490+0000) 2020-02-06T00:30:53.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:53.243+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:53.243931+0000) 2020-02-06T00:30:54.031 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:54.028+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:54.029967+0000) 2020-02-06T00:30:54.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:54.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:54.030138+0000) 2020-02-06T00:30:55.553 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:55.550+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.551841+0000) 2020-02-06T00:30:55.553 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:30:55.551+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.552226+0000) 2020-02-06T00:30:55.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:30:55.551+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.552908+0000) 2020-02-06T00:30:55.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:55.552+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.553528+0000) 2020-02-06T00:30:55.555 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:55.552+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.554113+0000) 2020-02-06T00:30:55.555 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:55.552+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.554058+0000) 2020-02-06T00:30:55.556 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:30:55.553+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.554407+0000) 2020-02-06T00:30:55.556 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:55.553+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.555051+0000) 2020-02-06T00:30:55.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:55.554+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.555696+0000) 2020-02-06T00:30:55.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:55.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.556327+0000) 2020-02-06T00:30:55.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:30:55.553+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.555286+0000) 2020-02-06T00:30:55.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:30:55.555+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.556928+0000) 2020-02-06T00:30:55.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:55.556+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.557718+0000) 2020-02-06T00:30:55.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:55.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:55.561434+0000) 2020-02-06T00:30:55.570 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:30:55.567+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176183 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:30:56.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:30:56.017+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:56.018037+0000) 2020-02-06T00:30:56.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:30:56.365+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:56.365418+0000) 2020-02-06T00:30:57.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:30:57.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:57.006557+0000) 2020-02-06T00:30:57.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:30:57.269+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:57.270840+0000) 2020-02-06T00:30:57.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:30:57.269+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:57.271014+0000) 2020-02-06T00:30:57.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:30:57.269+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:57.271076+0000) 2020-02-06T00:30:59.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:59.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:59.650143+0000) 2020-02-06T00:30:59.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:30:59.898+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:30:59.899124+0000) 2020-02-06T00:31:00.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:00.498+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.499538+0000) 2020-02-06T00:31:00.577 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:00.573+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.575373+0000) 2020-02-06T00:31:00.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:00.574+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.575761+0000) 2020-02-06T00:31:00.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:00.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.576518+0000) 2020-02-06T00:31:00.578 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:00.577+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.577620+0000) 2020-02-06T00:31:00.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:00.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.578552+0000) 2020-02-06T00:31:00.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:00.577+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.578753+0000) 2020-02-06T00:31:00.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:00.579+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.579320+0000) 2020-02-06T00:31:00.582 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:00.579+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.579748+0000) 2020-02-06T00:31:00.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:00.580+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.580208+0000) 2020-02-06T00:31:00.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:00.580+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.580358+0000) 2020-02-06T00:31:00.583 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:00.579+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.580617+0000) 2020-02-06T00:31:00.583 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:00.579+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.580855+0000) 2020-02-06T00:31:00.584 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:00.579+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.580731+0000) 2020-02-06T00:31:00.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:00.582+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.582874+0000) 2020-02-06T00:31:00.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:00.594+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:00.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:00.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:00.758650+0000) 2020-02-06T00:31:02.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:02.879+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:02.880728+0000) 2020-02-06T00:31:03.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:03.252+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:03.252779+0000) 2020-02-06T00:31:04.037 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:04.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:04.030154+0000) 2020-02-06T00:31:04.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:04.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:04.030401+0000) 2020-02-06T00:31:05.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:05.599+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.600401+0000) 2020-02-06T00:31:05.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:05.600+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.601487+0000) 2020-02-06T00:31:05.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:05.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.602352+0000) 2020-02-06T00:31:05.604 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:05.602+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.603157+0000) 2020-02-06T00:31:05.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:05.601+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.602982+0000) 2020-02-06T00:31:05.605 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:05.602+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.603528+0000) 2020-02-06T00:31:05.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:05.602+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.603551+0000) 2020-02-06T00:31:05.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:05.602+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.603840+0000) 2020-02-06T00:31:05.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:05.602+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.604096+0000) 2020-02-06T00:31:05.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:05.603+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.604590+0000) 2020-02-06T00:31:05.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:05.603+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.605012+0000) 2020-02-06T00:31:05.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:05.603+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.605353+0000) 2020-02-06T00:31:05.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:05.605+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.606404+0000) 2020-02-06T00:31:05.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:05.609+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:05.610154+0000) 2020-02-06T00:31:05.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:05.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:06.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:06.017+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:06.018282+0000) 2020-02-06T00:31:06.367 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:06.364+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:06.365640+0000) 2020-02-06T00:31:07.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:07.005+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:07.006792+0000) 2020-02-06T00:31:07.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:07.270+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:07.271093+0000) 2020-02-06T00:31:07.272 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:07.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:07.271243+0000) 2020-02-06T00:31:07.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:07.270+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:07.271371+0000) 2020-02-06T00:31:09.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:09.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:09.650410+0000) 2020-02-06T00:31:09.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:09.898+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:09.899381+0000) 2020-02-06T00:31:10.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:10.498+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.499756+0000) 2020-02-06T00:31:10.627 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:10.625+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.626421+0000) 2020-02-06T00:31:10.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:10.626+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.627820+0000) 2020-02-06T00:31:10.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:10.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.628515+0000) 2020-02-06T00:31:10.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:10.627+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.628636+0000) 2020-02-06T00:31:10.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:10.627+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.628905+0000) 2020-02-06T00:31:10.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:10.628+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.629169+0000) 2020-02-06T00:31:10.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:10.628+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.629503+0000) 2020-02-06T00:31:10.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:10.627+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.629291+0000) 2020-02-06T00:31:10.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:10.628+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.630147+0000) 2020-02-06T00:31:10.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:10.628+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.630354+0000) 2020-02-06T00:31:10.632 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:10.628+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.630035+0000) 2020-02-06T00:31:10.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:10.630+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.631537+0000) 2020-02-06T00:31:10.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:10.629+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.630949+0000) 2020-02-06T00:31:10.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:10.634+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.635275+0000) 2020-02-06T00:31:10.647 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:10.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176309 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:10.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:10.757+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:10.758861+0000) 2020-02-06T00:31:10.788 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:31:10.795 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:31:12.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:12.880+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:12.881006+0000) 2020-02-06T00:31:13.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:13.260+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:13.261250+0000) 2020-02-06T00:31:14.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:14.029+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:14.030412+0000) 2020-02-06T00:31:14.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:14.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:14.030588+0000) 2020-02-06T00:31:15.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:15.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.651745+0000) 2020-02-06T00:31:15.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:15.651+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.652500+0000) 2020-02-06T00:31:15.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:15.652+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.653212+0000) 2020-02-06T00:31:15.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:15.652+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.653482+0000) 2020-02-06T00:31:15.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:15.652+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.653782+0000) 2020-02-06T00:31:15.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:15.652+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.654057+0000) 2020-02-06T00:31:15.656 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:15.652+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.654310+0000) 2020-02-06T00:31:15.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:15.654+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.655594+0000) 2020-02-06T00:31:15.657 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:15.654+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.655602+0000) 2020-02-06T00:31:15.657 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:15.655+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.656566+0000) 2020-02-06T00:31:15.658 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:15.655+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.656708+0000) 2020-02-06T00:31:15.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:15.656+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.657065+0000) 2020-02-06T00:31:15.659 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:15.655+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.657189+0000) 2020-02-06T00:31:15.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:15.663+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:15.664501+0000) 2020-02-06T00:31:15.673 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:15.671+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176349 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:16.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:16.017+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:16.018532+0000) 2020-02-06T00:31:16.367 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:16.365+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:16.365911+0000) 2020-02-06T00:31:17.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:17.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:17.007024+0000) 2020-02-06T00:31:17.272 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:17.270+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:17.271295+0000) 2020-02-06T00:31:17.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:17.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:17.271512+0000) 2020-02-06T00:31:17.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:17.270+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:17.271755+0000) 2020-02-06T00:31:19.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:19.649+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:19.650614+0000) 2020-02-06T00:31:19.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:19.898+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:19.899626+0000) 2020-02-06T00:31:20.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:20.499+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.500015+0000) 2020-02-06T00:31:20.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:20.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.679165+0000) 2020-02-06T00:31:20.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:20.678+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.679758+0000) 2020-02-06T00:31:20.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:20.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.680385+0000) 2020-02-06T00:31:20.682 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:20.679+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.681104+0000) 2020-02-06T00:31:20.683 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:20.680+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.681514+0000) 2020-02-06T00:31:20.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:20.681+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.682224+0000) 2020-02-06T00:31:20.684 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:20.681+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.682472+0000) 2020-02-06T00:31:20.684 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:20.681+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.682608+0000) 2020-02-06T00:31:20.684 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:20.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.683244+0000) 2020-02-06T00:31:20.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:20.682+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.683476+0000) 2020-02-06T00:31:20.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:20.682+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.683930+0000) 2020-02-06T00:31:20.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:20.682+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.684270+0000) 2020-02-06T00:31:20.686 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:20.683+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.685076+0000) 2020-02-06T00:31:20.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:20.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.687463+0000) 2020-02-06T00:31:20.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:20.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:20.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:20.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:20.759072+0000) 2020-02-06T00:31:22.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:22.880+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:22.881315+0000) 2020-02-06T00:31:23.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:23.268+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:23.269473+0000) 2020-02-06T00:31:24.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:24.029+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:24.030641+0000) 2020-02-06T00:31:24.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:24.029+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:24.030851+0000) 2020-02-06T00:31:25.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:25.701+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.702769+0000) 2020-02-06T00:31:25.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:25.702+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.703684+0000) 2020-02-06T00:31:25.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:25.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.704627+0000) 2020-02-06T00:31:25.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:25.703+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.704392+0000) 2020-02-06T00:31:25.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:25.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.705611+0000) 2020-02-06T00:31:25.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:25.703+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.705346+0000) 2020-02-06T00:31:25.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:25.705+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.706377+0000) 2020-02-06T00:31:25.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:25.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.706915+0000) 2020-02-06T00:31:25.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:25.706+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.707224+0000) 2020-02-06T00:31:25.710 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:25.706+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.707410+0000) 2020-02-06T00:31:25.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:25.706+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.707947+0000) 2020-02-06T00:31:25.711 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:25.706+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.707596+0000) 2020-02-06T00:31:25.711 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:25.706+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.708067+0000) 2020-02-06T00:31:25.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:25.710+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:25.711071+0000) 2020-02-06T00:31:25.723 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:25.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:26.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:26.018+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:26.018771+0000) 2020-02-06T00:31:26.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:26.365+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:26.366177+0000) 2020-02-06T00:31:27.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:27.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:27.007293+0000) 2020-02-06T00:31:27.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:27.270+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:27.271538+0000) 2020-02-06T00:31:27.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:27.270+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:27.271733+0000) 2020-02-06T00:31:27.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:27.270+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:27.272003+0000) 2020-02-06T00:31:29.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:29.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:29.650883+0000) 2020-02-06T00:31:29.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:29.899+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:29.899854+0000) 2020-02-06T00:31:30.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:30.499+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.500273+0000) 2020-02-06T00:31:30.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:30.726+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.727706+0000) 2020-02-06T00:31:30.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:30.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.729412+0000) 2020-02-06T00:31:30.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:30.728+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.729768+0000) 2020-02-06T00:31:30.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:30.728+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.729972+0000) 2020-02-06T00:31:30.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:30.729+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.730477+0000) 2020-02-06T00:31:30.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:30.730+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.731388+0000) 2020-02-06T00:31:30.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:30.730+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.731870+0000) 2020-02-06T00:31:30.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:30.729+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.731374+0000) 2020-02-06T00:31:30.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:30.731+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.732280+0000) 2020-02-06T00:31:30.736 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:30.731+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.732365+0000) 2020-02-06T00:31:30.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:30.732+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.733748+0000) 2020-02-06T00:31:30.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:30.731+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.733030+0000) 2020-02-06T00:31:30.736 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:30.732+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.733591+0000) 2020-02-06T00:31:30.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:30.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.738844+0000) 2020-02-06T00:31:30.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:30.746+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176473 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:30.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:30.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:30.759312+0000) 2020-02-06T00:31:32.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:32.881+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:32.881574+0000) 2020-02-06T00:31:33.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:33.277+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:33.277809+0000) 2020-02-06T00:31:34.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:34.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:34.030899+0000) 2020-02-06T00:31:34.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:34.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:34.031124+0000) 2020-02-06T00:31:35.755 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:35.752+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.752716+0000) 2020-02-06T00:31:35.756 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:35.753+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.754465+0000) 2020-02-06T00:31:35.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:35.754+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.755868+0000) 2020-02-06T00:31:35.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:35.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.756237+0000) 2020-02-06T00:31:35.758 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:35.755+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.756633+0000) 2020-02-06T00:31:35.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:35.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.757027+0000) 2020-02-06T00:31:35.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:35.756+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.757150+0000) 2020-02-06T00:31:35.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:35.756+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.757751+0000) 2020-02-06T00:31:35.760 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:35.756+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.758068+0000) 2020-02-06T00:31:35.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:35.756+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.757544+0000) 2020-02-06T00:31:35.760 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:35.757+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.758226+0000) 2020-02-06T00:31:35.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:35.757+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.758735+0000) 2020-02-06T00:31:35.761 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:35.758+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.759761+0000) 2020-02-06T00:31:35.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:35.761+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:35.762868+0000) 2020-02-06T00:31:35.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:35.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:36.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:36.018+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:36.019019+0000) 2020-02-06T00:31:36.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:36.365+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:36.366383+0000) 2020-02-06T00:31:37.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:37.006+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:37.007507+0000) 2020-02-06T00:31:37.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:37.271+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:37.271795+0000) 2020-02-06T00:31:37.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:37.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:37.272009+0000) 2020-02-06T00:31:37.274 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:37.271+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:37.272294+0000) 2020-02-06T00:31:39.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:39.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:39.651146+0000) 2020-02-06T00:31:39.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:39.899+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:39.900114+0000) 2020-02-06T00:31:40.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:40.500+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.500532+0000) 2020-02-06T00:31:40.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:40.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.759518+0000) 2020-02-06T00:31:40.779 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:40.777+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.778540+0000) 2020-02-06T00:31:40.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:40.779+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.780734+0000) 2020-02-06T00:31:40.782 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:40.780+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.781002+0000) 2020-02-06T00:31:40.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:40.781+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.781648+0000) 2020-02-06T00:31:40.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:40.781+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.782318+0000) 2020-02-06T00:31:40.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:40.781+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.782464+0000) 2020-02-06T00:31:40.784 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:40.782+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.782900+0000) 2020-02-06T00:31:40.784 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:40.782+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.783216+0000) 2020-02-06T00:31:40.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:40.782+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.783755+0000) 2020-02-06T00:31:40.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:40.783+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.784211+0000) 2020-02-06T00:31:40.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:40.784+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.784762+0000) 2020-02-06T00:31:40.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:40.784+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.785182+0000) 2020-02-06T00:31:40.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:40.785+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.785556+0000) 2020-02-06T00:31:40.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:40.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:40.790972+0000) 2020-02-06T00:31:40.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:40.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176554 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:40.927 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:31:40.934 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:31:42.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:42.881+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:42.881840+0000) 2020-02-06T00:31:43.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:43.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:43.287132+0000) 2020-02-06T00:31:44.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:44.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:44.031140+0000) 2020-02-06T00:31:44.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:44.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:44.031359+0000) 2020-02-06T00:31:45.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:45.803+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.804586+0000) 2020-02-06T00:31:45.807 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:45.804+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.805368+0000) 2020-02-06T00:31:45.807 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:45.805+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.806105+0000) 2020-02-06T00:31:45.807 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:45.806+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.806468+0000) 2020-02-06T00:31:45.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:45.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.807135+0000) 2020-02-06T00:31:45.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:45.806+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.807390+0000) 2020-02-06T00:31:45.817 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:45.809+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.809464+0000) 2020-02-06T00:31:45.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:45.809+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.810085+0000) 2020-02-06T00:31:45.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:45.810+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.810538+0000) 2020-02-06T00:31:45.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:45.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.808136+0000) 2020-02-06T00:31:45.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:45.807+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.808439+0000) 2020-02-06T00:31:45.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:45.807+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.808924+0000) 2020-02-06T00:31:45.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:45.809+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.810222+0000) 2020-02-06T00:31:45.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:45.817+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:45.818585+0000) 2020-02-06T00:31:45.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:45.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:46.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:46.018+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:46.019253+0000) 2020-02-06T00:31:46.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:46.366+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:46.366592+0000) 2020-02-06T00:31:47.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:47.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:47.007732+0000) 2020-02-06T00:31:47.273 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:47.271+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:47.272090+0000) 2020-02-06T00:31:47.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:47.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:47.272287+0000) 2020-02-06T00:31:47.274 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:47.272+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:47.272504+0000) 2020-02-06T00:31:49.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:49.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:49.651385+0000) 2020-02-06T00:31:49.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:49.899+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:49.900345+0000) 2020-02-06T00:31:50.502 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:50.500+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.500832+0000) 2020-02-06T00:31:50.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:50.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.759747+0000) 2020-02-06T00:31:50.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:50.832+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.833360+0000) 2020-02-06T00:31:50.835 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:50.833+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.833911+0000) 2020-02-06T00:31:50.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:50.833+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.834484+0000) 2020-02-06T00:31:50.837 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:50.835+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.835632+0000) 2020-02-06T00:31:50.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:50.835+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.836461+0000) 2020-02-06T00:31:50.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:50.835+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.836659+0000) 2020-02-06T00:31:50.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:50.835+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.836102+0000) 2020-02-06T00:31:50.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:50.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.837367+0000) 2020-02-06T00:31:50.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:50.836+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.837897+0000) 2020-02-06T00:31:50.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:50.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.837562+0000) 2020-02-06T00:31:50.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:50.837+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.837554+0000) 2020-02-06T00:31:50.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:50.837+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.838264+0000) 2020-02-06T00:31:50.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:50.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.838512+0000) 2020-02-06T00:31:50.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:50.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:50.843516+0000) 2020-02-06T00:31:50.852 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:50.851+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:52.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:52.881+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:52.882103+0000) 2020-02-06T00:31:53.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:53.295+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:53.295591+0000) 2020-02-06T00:31:54.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:54.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:54.031332+0000) 2020-02-06T00:31:54.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:54.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:54.031621+0000) 2020-02-06T00:31:55.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:31:55.857+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.857448+0000) 2020-02-06T00:31:55.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:31:55.858+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.858323+0000) 2020-02-06T00:31:55.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:55.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.859049+0000) 2020-02-06T00:31:55.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:55.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.859723+0000) 2020-02-06T00:31:55.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:55.860+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.860395+0000) 2020-02-06T00:31:55.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:31:55.860+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.860603+0000) 2020-02-06T00:31:55.862 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:55.860+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.861221+0000) 2020-02-06T00:31:55.863 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:55.862+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.862470+0000) 2020-02-06T00:31:55.864 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:55.862+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.862826+0000) 2020-02-06T00:31:55.864 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:55.862+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.863090+0000) 2020-02-06T00:31:55.865 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:55.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.863341+0000) 2020-02-06T00:31:55.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:31:55.863+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.863643+0000) 2020-02-06T00:31:55.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:31:55.863+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.864274+0000) 2020-02-06T00:31:55.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:55.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:55.870887+0000) 2020-02-06T00:31:55.878 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:31:55.876+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176677 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:31:56.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:31:56.019+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:56.019481+0000) 2020-02-06T00:31:56.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:31:56.366+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:56.366818+0000) 2020-02-06T00:31:57.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:31:57.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:57.007964+0000) 2020-02-06T00:31:57.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:31:57.271+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:57.272319+0000) 2020-02-06T00:31:57.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:31:57.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:57.272507+0000) 2020-02-06T00:31:57.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:31:57.272+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:57.272748+0000) 2020-02-06T00:31:59.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:59.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:59.651638+0000) 2020-02-06T00:31:59.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:31:59.899+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:31:59.900571+0000) 2020-02-06T00:32:00.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:00.500+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.501049+0000) 2020-02-06T00:32:00.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:00.758+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.760013+0000) 2020-02-06T00:32:00.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:00.882+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.883770+0000) 2020-02-06T00:32:00.886 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:00.883+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.884482+0000) 2020-02-06T00:32:00.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:00.882+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.882995+0000) 2020-02-06T00:32:00.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:00.884+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.885437+0000) 2020-02-06T00:32:00.887 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:00.884+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.885574+0000) 2020-02-06T00:32:00.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:00.885+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.886192+0000) 2020-02-06T00:32:00.889 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:00.885+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.885977+0000) 2020-02-06T00:32:00.889 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:00.886+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.886758+0000) 2020-02-06T00:32:00.890 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:00.886+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.887177+0000) 2020-02-06T00:32:00.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:00.886+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.886992+0000) 2020-02-06T00:32:00.890 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:00.887+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.888023+0000) 2020-02-06T00:32:00.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:00.885+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.886658+0000) 2020-02-06T00:32:00.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:00.887+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.888243+0000) 2020-02-06T00:32:00.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:00.892+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:00.893435+0000) 2020-02-06T00:32:00.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:00.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:02.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:02.882+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:02.882372+0000) 2020-02-06T00:32:03.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:03.303+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:03.304203+0000) 2020-02-06T00:32:04.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:04.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:04.031552+0000) 2020-02-06T00:32:04.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:04.030+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:04.031846+0000) 2020-02-06T00:32:05.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:05.906+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.907173+0000) 2020-02-06T00:32:06.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:05.906+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.907615+0000) 2020-02-06T00:32:06.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:05.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.908702+0000) 2020-02-06T00:32:06.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:05.908+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.909919+0000) 2020-02-06T00:32:06.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:05.909+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.910529+0000) 2020-02-06T00:32:06.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:05.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.912281+0000) 2020-02-06T00:32:06.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:05.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.917684+0000) 2020-02-06T00:32:06.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:05.909+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.910600+0000) 2020-02-06T00:32:06.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:05.910+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.911808+0000) 2020-02-06T00:32:06.567 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:06.366+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:06.367038+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:05.909+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.909714+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:06.019+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:06.019706+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:05.910+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.911105+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:05.911+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.911582+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:05.912+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.912625+0000) 2020-02-06T00:32:06.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:05.913+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:05.913760+0000) 2020-02-06T00:32:06.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:05.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:07.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:07.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:07.008195+0000) 2020-02-06T00:32:07.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:07.272+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:07.272580+0000) 2020-02-06T00:32:07.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:07.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:07.272772+0000) 2020-02-06T00:32:07.274 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:07.272+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:07.272996+0000) 2020-02-06T00:32:09.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:09.650+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:09.651889+0000) 2020-02-06T00:32:09.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:09.899+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:09.900802+0000) 2020-02-06T00:32:10.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:10.501+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.501298+0000) 2020-02-06T00:32:10.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:10.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.760290+0000) 2020-02-06T00:32:10.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:10.930+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.931784+0000) 2020-02-06T00:32:10.934 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:10.931+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.932783+0000) 2020-02-06T00:32:10.935 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:10.933+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.934070+0000) 2020-02-06T00:32:10.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:10.934+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.934488+0000) 2020-02-06T00:32:10.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:10.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.935473+0000) 2020-02-06T00:32:10.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:10.934+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.935148+0000) 2020-02-06T00:32:10.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:10.935+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.936186+0000) 2020-02-06T00:32:10.937 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:10.935+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.935925+0000) 2020-02-06T00:32:10.938 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:10.936+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.936480+0000) 2020-02-06T00:32:10.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:10.936+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.937379+0000) 2020-02-06T00:32:10.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:10.937+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.937684+0000) 2020-02-06T00:32:10.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:10.937+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.937853+0000) 2020-02-06T00:32:10.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:10.937+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.938422+0000) 2020-02-06T00:32:10.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:10.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:10.944765+0000) 2020-02-06T00:32:10.953 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:10.951+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:11.090 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:32:11.097 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:32:12.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:12.882+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:12.882676+0000) 2020-02-06T00:32:13.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:13.311+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:13.312741+0000) 2020-02-06T00:32:14.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:14.031+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:14.031782+0000) 2020-02-06T00:32:14.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:14.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:14.032040+0000) 2020-02-06T00:32:15.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:15.954+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.955614+0000) 2020-02-06T00:32:15.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:15.957+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.958606+0000) 2020-02-06T00:32:15.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:15.957+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.958993+0000) 2020-02-06T00:32:15.961 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:15.958+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.959886+0000) 2020-02-06T00:32:15.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:15.959+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.960211+0000) 2020-02-06T00:32:15.962 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:15.959+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.960698+0000) 2020-02-06T00:32:15.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:15.959+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.961050+0000) 2020-02-06T00:32:15.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:15.960+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.961700+0000) 2020-02-06T00:32:15.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:15.960+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.961764+0000) 2020-02-06T00:32:15.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:15.961+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.962791+0000) 2020-02-06T00:32:15.964 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:15.960+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.962231+0000) 2020-02-06T00:32:15.964 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:15.961+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.962694+0000) 2020-02-06T00:32:15.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:15.961+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.963289+0000) 2020-02-06T00:32:15.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:15.968+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:15.969267+0000) 2020-02-06T00:32:15.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:15.974+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:16.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:16.018+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:16.019972+0000) 2020-02-06T00:32:16.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:16.366+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:16.367224+0000) 2020-02-06T00:32:17.009 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:17.007+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:17.008430+0000) 2020-02-06T00:32:17.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:17.271+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:17.272823+0000) 2020-02-06T00:32:17.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:17.271+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:17.273032+0000) 2020-02-06T00:32:17.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:17.271+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:17.273251+0000) 2020-02-06T00:32:19.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:19.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:19.652149+0000) 2020-02-06T00:32:19.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:19.900+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:19.901044+0000) 2020-02-06T00:32:20.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:20.500+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.501552+0000) 2020-02-06T00:32:20.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:20.759+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.760598+0000) 2020-02-06T00:32:20.983 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:20.980+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.981803+0000) 2020-02-06T00:32:20.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:20.981+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.982568+0000) 2020-02-06T00:32:20.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:20.982+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.983098+0000) 2020-02-06T00:32:20.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:20.982+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.983538+0000) 2020-02-06T00:32:20.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:20.982+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.983751+0000) 2020-02-06T00:32:20.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:20.982+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.984028+0000) 2020-02-06T00:32:20.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:20.983+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.984757+0000) 2020-02-06T00:32:20.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:20.983+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.984489+0000) 2020-02-06T00:32:20.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:20.984+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.985636+0000) 2020-02-06T00:32:20.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:20.985+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.986462+0000) 2020-02-06T00:32:20.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:20.985+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.986703+0000) 2020-02-06T00:32:20.989 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:20.985+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.987166+0000) 2020-02-06T00:32:20.989 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:20.986+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.987858+0000) 2020-02-06T00:32:20.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:20.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:20.991012+0000) 2020-02-06T00:32:21.002 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:20.999+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:22.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:22.881+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:22.882953+0000) 2020-02-06T00:32:23.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:23.320+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:23.321292+0000) 2020-02-06T00:32:24.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:24.030+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:24.031999+0000) 2020-02-06T00:32:24.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:24.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:24.032257+0000) 2020-02-06T00:32:26.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:26.004+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.005892+0000) 2020-02-06T00:32:26.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:26.005+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.006639+0000) 2020-02-06T00:32:26.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:26.007+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.008617+0000) 2020-02-06T00:32:26.010 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:26.007+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.009135+0000) 2020-02-06T00:32:26.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:26.008+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.009821+0000) 2020-02-06T00:32:26.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:26.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.010048+0000) 2020-02-06T00:32:26.011 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:26.008+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.010168+0000) 2020-02-06T00:32:26.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:26.009+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.010446+0000) 2020-02-06T00:32:26.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:26.009+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.010966+0000) 2020-02-06T00:32:26.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:26.010+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.011154+0000) 2020-02-06T00:32:26.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:26.010+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.011353+0000) 2020-02-06T00:32:26.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:26.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.011511+0000) 2020-02-06T00:32:26.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:26.011+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.012059+0000) 2020-02-06T00:32:26.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:26.017+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.018136+0000) 2020-02-06T00:32:26.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:26.018+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.020158+0000) 2020-02-06T00:32:26.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:26.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176922 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:26.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:26.366+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:26.367429+0000) 2020-02-06T00:32:27.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:27.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:27.008668+0000) 2020-02-06T00:32:27.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:27.272+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:27.273140+0000) 2020-02-06T00:32:27.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:27.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:27.273279+0000) 2020-02-06T00:32:27.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:27.272+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:27.273511+0000) 2020-02-06T00:32:29.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:29.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:29.652415+0000) 2020-02-06T00:32:29.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:29.900+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:29.901310+0000) 2020-02-06T00:32:30.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:30.500+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:30.501771+0000) 2020-02-06T00:32:30.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:30.760+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:30.760830+0000) 2020-02-06T00:32:31.031 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:31.029+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.030553+0000) 2020-02-06T00:32:31.033 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:31.030+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.031815+0000) 2020-02-06T00:32:31.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:31.031+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.032379+0000) 2020-02-06T00:32:31.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:31.031+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.033181+0000) 2020-02-06T00:32:31.034 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:31.032+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.033566+0000) 2020-02-06T00:32:31.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:31.033+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.034485+0000) 2020-02-06T00:32:31.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:31.033+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.034880+0000) 2020-02-06T00:32:31.036 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:31.034+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.035266+0000) 2020-02-06T00:32:31.037 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:31.034+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.036288+0000) 2020-02-06T00:32:31.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:31.035+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.036620+0000) 2020-02-06T00:32:31.038 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:31.034+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.036396+0000) 2020-02-06T00:32:31.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:31.035+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.037074+0000) 2020-02-06T00:32:31.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:31.036+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.037358+0000) 2020-02-06T00:32:31.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:31.039+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:31.041042+0000) 2020-02-06T00:32:31.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:31.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 176964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:32.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:32.882+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:32.883237+0000) 2020-02-06T00:32:33.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:33.328+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:33.329200+0000) 2020-02-06T00:32:34.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:34.031+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:34.032275+0000) 2020-02-06T00:32:34.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:34.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:34.032517+0000) 2020-02-06T00:32:36.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:36.019+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.020375+0000) 2020-02-06T00:32:36.057 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:36.054+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.055668+0000) 2020-02-06T00:32:36.057 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:36.055+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.056484+0000) 2020-02-06T00:32:36.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:36.056+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.057549+0000) 2020-02-06T00:32:36.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:36.057+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.058152+0000) 2020-02-06T00:32:36.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:36.056+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.057977+0000) 2020-02-06T00:32:36.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:36.057+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.058609+0000) 2020-02-06T00:32:36.062 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:36.057+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.059205+0000) 2020-02-06T00:32:36.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:36.059+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.060311+0000) 2020-02-06T00:32:36.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:36.059+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.060883+0000) 2020-02-06T00:32:36.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:36.060+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.061428+0000) 2020-02-06T00:32:36.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:36.058+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.060023+0000) 2020-02-06T00:32:36.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:36.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.061480+0000) 2020-02-06T00:32:36.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:36.060+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.062162+0000) 2020-02-06T00:32:36.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:36.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.067662+0000) 2020-02-06T00:32:36.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:36.074+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:36.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:36.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:36.367652+0000) 2020-02-06T00:32:37.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:37.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:37.008922+0000) 2020-02-06T00:32:37.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:37.272+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:37.273462+0000) 2020-02-06T00:32:37.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:37.272+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:37.273509+0000) 2020-02-06T00:32:37.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:37.272+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:37.273796+0000) 2020-02-06T00:32:39.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:39.651+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:39.652669+0000) 2020-02-06T00:32:39.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:39.901+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:39.901550+0000) 2020-02-06T00:32:40.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:40.501+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:40.502009+0000) 2020-02-06T00:32:40.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:40.760+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:40.761065+0000) 2020-02-06T00:32:41.084 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:41.081+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.082919+0000) 2020-02-06T00:32:41.084 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:41.082+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.083467+0000) 2020-02-06T00:32:41.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:41.081+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.083300+0000) 2020-02-06T00:32:41.085 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:41.083+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.084414+0000) 2020-02-06T00:32:41.086 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:41.083+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.084720+0000) 2020-02-06T00:32:41.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:41.084+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.085066+0000) 2020-02-06T00:32:41.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:41.083+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.084118+0000) 2020-02-06T00:32:41.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:41.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.085684+0000) 2020-02-06T00:32:41.087 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:41.084+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.086056+0000) 2020-02-06T00:32:41.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:41.085+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.086978+0000) 2020-02-06T00:32:41.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:41.085+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.087211+0000) 2020-02-06T00:32:41.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:41.086+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.087753+0000) 2020-02-06T00:32:41.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:41.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.088162+0000) 2020-02-06T00:32:41.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:41.091+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:41.092356+0000) 2020-02-06T00:32:41.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:41.098+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:41.279 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:32:41.286 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:32:42.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:42.882+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:42.883512+0000) 2020-02-06T00:32:43.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:43.337+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:43.337287+0000) 2020-02-06T00:32:44.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:44.032+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:44.032557+0000) 2020-02-06T00:32:44.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:44.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:44.032742+0000) 2020-02-06T00:32:46.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:46.019+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.020667+0000) 2020-02-06T00:32:46.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:46.103+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.104482+0000) 2020-02-06T00:32:46.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:46.105+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.106839+0000) 2020-02-06T00:32:46.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:46.106+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.107516+0000) 2020-02-06T00:32:46.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:46.106+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.108006+0000) 2020-02-06T00:32:46.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:46.107+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.108330+0000) 2020-02-06T00:32:46.110 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:46.108+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.109202+0000) 2020-02-06T00:32:46.111 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:46.108+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.110057+0000) 2020-02-06T00:32:46.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:46.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.110822+0000) 2020-02-06T00:32:46.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:46.109+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.110964+0000) 2020-02-06T00:32:46.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:46.109+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.111250+0000) 2020-02-06T00:32:46.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:46.109+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.111368+0000) 2020-02-06T00:32:46.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:46.110+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.111960+0000) 2020-02-06T00:32:46.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:46.112+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.113906+0000) 2020-02-06T00:32:46.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:46.113+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.114920+0000) 2020-02-06T00:32:46.126 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:46.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:46.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:46.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:46.367862+0000) 2020-02-06T00:32:47.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:47.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:47.009169+0000) 2020-02-06T00:32:47.275 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:47.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:47.273753+0000) 2020-02-06T00:32:47.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:47.273+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:47.273740+0000) 2020-02-06T00:32:47.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:47.273+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:47.274044+0000) 2020-02-06T00:32:49.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:49.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:49.652930+0000) 2020-02-06T00:32:49.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:49.901+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:49.901787+0000) 2020-02-06T00:32:50.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:50.501+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:50.502251+0000) 2020-02-06T00:32:50.762 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:50.761+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:50.761332+0000) 2020-02-06T00:32:51.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:51.129+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.131206+0000) 2020-02-06T00:32:51.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:51.131+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.131479+0000) 2020-02-06T00:32:51.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:51.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.131973+0000) 2020-02-06T00:32:51.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:51.130+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.132332+0000) 2020-02-06T00:32:51.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:51.132+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.132793+0000) 2020-02-06T00:32:51.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:51.132+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.133941+0000) 2020-02-06T00:32:51.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:51.134+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.134259+0000) 2020-02-06T00:32:51.136 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:51.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.134695+0000) 2020-02-06T00:32:51.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:51.134+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.135017+0000) 2020-02-06T00:32:51.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:51.134+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.135928+0000) 2020-02-06T00:32:51.137 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:51.134+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.136114+0000) 2020-02-06T00:32:51.137 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:51.135+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.136605+0000) 2020-02-06T00:32:51.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:51.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.138989+0000) 2020-02-06T00:32:51.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:51.140+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:51.140254+0000) 2020-02-06T00:32:51.150 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:51.147+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:52.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:52.883+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:52.883871+0000) 2020-02-06T00:32:53.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:53.344+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:53.345461+0000) 2020-02-06T00:32:54.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:54.031+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:54.032789+0000) 2020-02-06T00:32:54.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:54.031+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:54.032978+0000) 2020-02-06T00:32:56.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:56.020+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.020922+0000) 2020-02-06T00:32:56.155 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:32:56.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.153515+0000) 2020-02-06T00:32:56.157 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:32:56.155+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.156712+0000) 2020-02-06T00:32:56.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:56.156+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.157602+0000) 2020-02-06T00:32:56.159 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:32:56.157+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.158196+0000) 2020-02-06T00:32:56.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:56.158+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.160224+0000) 2020-02-06T00:32:56.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:32:56.158+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.160382+0000) 2020-02-06T00:32:56.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:56.159+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.160971+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:32:56.159+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.161389+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:56.160+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.161655+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:56.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.158892+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:56.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.159705+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:32:56.158+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.159201+0000) 2020-02-06T00:32:56.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:56.158+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.159874+0000) 2020-02-06T00:32:56.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:56.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.163415+0000) 2020-02-06T00:32:56.175 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:32:56.172+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:32:56.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:32:56.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:56.368119+0000) 2020-02-06T00:32:57.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:32:57.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:57.009384+0000) 2020-02-06T00:32:57.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:32:57.273+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:57.274008+0000) 2020-02-06T00:32:57.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:32:57.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:57.274011+0000) 2020-02-06T00:32:57.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:32:57.273+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:57.274306+0000) 2020-02-06T00:32:59.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:59.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:59.653150+0000) 2020-02-06T00:32:59.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:32:59.901+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:32:59.902014+0000) 2020-02-06T00:33:00.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:00.501+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:00.502486+0000) 2020-02-06T00:33:00.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:00.760+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:00.761568+0000) 2020-02-06T00:33:01.181 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:01.178+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.179529+0000) 2020-02-06T00:33:01.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:01.178+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.180244+0000) 2020-02-06T00:33:01.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:01.179+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.180880+0000) 2020-02-06T00:33:01.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:01.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.181621+0000) 2020-02-06T00:33:01.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:01.180+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.181893+0000) 2020-02-06T00:33:01.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:01.181+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.182296+0000) 2020-02-06T00:33:01.184 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:01.182+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.183524+0000) 2020-02-06T00:33:01.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:01.182+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.183724+0000) 2020-02-06T00:33:01.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:01.182+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.184195+0000) 2020-02-06T00:33:01.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:01.183+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.184706+0000) 2020-02-06T00:33:01.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:01.184+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.185342+0000) 2020-02-06T00:33:01.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:01.184+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.185429+0000) 2020-02-06T00:33:01.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:01.184+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.186069+0000) 2020-02-06T00:33:01.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:01.187+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:01.188648+0000) 2020-02-06T00:33:01.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:01.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177209 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:02.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:02.883+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:02.884148+0000) 2020-02-06T00:33:03.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:03.353+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:03.353886+0000) 2020-02-06T00:33:04.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:04.032+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:04.033015+0000) 2020-02-06T00:33:04.034 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:04.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:04.033265+0000) 2020-02-06T00:33:06.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:06.020+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.021185+0000) 2020-02-06T00:33:06.204 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:06.202+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.203045+0000) 2020-02-06T00:33:06.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:06.203+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.204161+0000) 2020-02-06T00:33:06.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:06.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.204852+0000) 2020-02-06T00:33:06.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:06.204+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.205295+0000) 2020-02-06T00:33:06.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:06.204+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.205736+0000) 2020-02-06T00:33:06.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:06.205+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.206405+0000) 2020-02-06T00:33:06.208 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:06.205+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.206664+0000) 2020-02-06T00:33:06.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:06.206+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.207890+0000) 2020-02-06T00:33:06.209 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:06.207+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.207700+0000) 2020-02-06T00:33:06.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:06.206+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.206986+0000) 2020-02-06T00:33:06.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:06.207+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.208073+0000) 2020-02-06T00:33:06.210 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:06.207+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.208257+0000) 2020-02-06T00:33:06.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:06.208+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.209282+0000) 2020-02-06T00:33:06.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:06.212+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.214062+0000) 2020-02-06T00:33:06.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:06.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177248 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:06.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:06.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:06.368337+0000) 2020-02-06T00:33:07.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:07.008+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:07.009586+0000) 2020-02-06T00:33:07.276 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:07.273+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:07.274265+0000) 2020-02-06T00:33:07.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:07.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:07.274314+0000) 2020-02-06T00:33:07.276 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:07.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:07.274608+0000) 2020-02-06T00:33:09.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:09.652+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:09.653381+0000) 2020-02-06T00:33:09.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:09.901+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:09.902274+0000) 2020-02-06T00:33:10.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:10.502+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:10.502733+0000) 2020-02-06T00:33:10.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:10.761+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:10.761800+0000) 2020-02-06T00:33:11.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:11.228+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.229568+0000) 2020-02-06T00:33:11.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:11.229+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.230301+0000) 2020-02-06T00:33:11.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:11.230+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.230984+0000) 2020-02-06T00:33:11.232 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:11.230+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.231332+0000) 2020-02-06T00:33:11.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:11.230+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.231817+0000) 2020-02-06T00:33:11.233 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:11.230+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.231922+0000) 2020-02-06T00:33:11.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:11.231+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.232232+0000) 2020-02-06T00:33:11.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:11.232+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.232850+0000) 2020-02-06T00:33:11.234 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:11.232+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.233238+0000) 2020-02-06T00:33:11.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:11.233+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.233740+0000) 2020-02-06T00:33:11.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:11.232+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.233984+0000) 2020-02-06T00:33:11.236 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:11.234+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.234727+0000) 2020-02-06T00:33:11.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:11.234+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.235150+0000) 2020-02-06T00:33:11.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:11.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:11.239386+0000) 2020-02-06T00:33:11.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:11.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:11.451 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:33:11.457 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:33:12.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:12.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:12.884422+0000) 2020-02-06T00:33:13.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:13.364+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:13.365579+0000) 2020-02-06T00:33:14.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:14.032+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:14.033298+0000) 2020-02-06T00:33:14.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:14.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:14.033481+0000) 2020-02-06T00:33:16.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:16.021+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.021482+0000) 2020-02-06T00:33:16.260 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:16.258+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.258618+0000) 2020-02-06T00:33:16.260 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:16.258+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.259251+0000) 2020-02-06T00:33:16.261 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:16.258+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.259647+0000) 2020-02-06T00:33:16.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:16.259+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.260147+0000) 2020-02-06T00:33:16.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:16.259+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.260154+0000) 2020-02-06T00:33:16.262 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:16.259+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.260488+0000) 2020-02-06T00:33:16.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:16.261+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.261877+0000) 2020-02-06T00:33:16.264 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:16.262+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.262821+0000) 2020-02-06T00:33:16.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:16.262+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.263255+0000) 2020-02-06T00:33:16.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:16.262+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.263586+0000) 2020-02-06T00:33:16.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:16.263+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.263655+0000) 2020-02-06T00:33:16.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:16.263+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.264216+0000) 2020-02-06T00:33:16.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:16.264+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.264463+0000) 2020-02-06T00:33:16.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:16.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.269127+0000) 2020-02-06T00:33:16.280 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:16.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177330 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:16.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:16.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:16.368547+0000) 2020-02-06T00:33:17.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:17.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:17.009796+0000) 2020-02-06T00:33:17.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:17.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:17.274569+0000) 2020-02-06T00:33:17.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:17.274+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:17.274566+0000) 2020-02-06T00:33:17.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:17.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:17.274870+0000) 2020-02-06T00:33:19.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:19.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:19.653618+0000) 2020-02-06T00:33:19.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:19.901+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:19.902511+0000) 2020-02-06T00:33:20.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:20.502+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:20.502973+0000) 2020-02-06T00:33:20.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:20.761+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:20.762036+0000) 2020-02-06T00:33:21.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:21.283+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.284280+0000) 2020-02-06T00:33:21.286 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:21.283+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.284746+0000) 2020-02-06T00:33:21.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:21.284+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.285327+0000) 2020-02-06T00:33:21.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:21.284+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.285557+0000) 2020-02-06T00:33:21.287 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:21.285+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.286374+0000) 2020-02-06T00:33:21.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:21.285+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.286250+0000) 2020-02-06T00:33:21.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:21.286+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.286553+0000) 2020-02-06T00:33:21.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:21.286+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.287273+0000) 2020-02-06T00:33:21.289 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:21.286+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.288002+0000) 2020-02-06T00:33:21.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:21.287+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.288465+0000) 2020-02-06T00:33:21.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:21.288+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.288761+0000) 2020-02-06T00:33:21.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:21.288+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.289129+0000) 2020-02-06T00:33:21.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:21.288+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.289133+0000) 2020-02-06T00:33:21.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:21.292+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:21.293833+0000) 2020-02-06T00:33:21.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:21.302+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:22.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:22.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:22.884717+0000) 2020-02-06T00:33:23.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:23.373+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:23.373806+0000) 2020-02-06T00:33:24.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:24.032+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:24.033541+0000) 2020-02-06T00:33:24.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:24.032+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:24.033685+0000) 2020-02-06T00:33:26.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:26.021+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.021721+0000) 2020-02-06T00:33:26.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:26.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.308998+0000) 2020-02-06T00:33:26.310 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:26.308+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.309686+0000) 2020-02-06T00:33:26.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:26.311+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.311724+0000) 2020-02-06T00:33:26.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:26.310+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.311873+0000) 2020-02-06T00:33:26.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:26.311+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.313014+0000) 2020-02-06T00:33:26.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:26.312+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.313676+0000) 2020-02-06T00:33:26.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:26.313+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.313625+0000) 2020-02-06T00:33:26.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:26.313+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.314406+0000) 2020-02-06T00:33:26.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:26.313+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.314698+0000) 2020-02-06T00:33:26.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:26.314+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.314711+0000) 2020-02-06T00:33:26.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:26.314+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.314912+0000) 2020-02-06T00:33:26.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:26.314+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.315253+0000) 2020-02-06T00:33:26.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:26.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.317999+0000) 2020-02-06T00:33:26.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:26.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.318359+0000) 2020-02-06T00:33:26.328 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:26.326+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:26.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:26.368+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:26.368757+0000) 2020-02-06T00:33:27.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:27.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:27.010003+0000) 2020-02-06T00:33:27.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:27.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:27.274824+0000) 2020-02-06T00:33:27.276 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:27.274+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:27.274818+0000) 2020-02-06T00:33:27.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:27.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:27.275102+0000) 2020-02-06T00:33:29.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:29.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:29.653816+0000) 2020-02-06T00:33:29.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:29.902+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:29.902743+0000) 2020-02-06T00:33:30.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:30.503+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:30.503282+0000) 2020-02-06T00:33:30.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:30.761+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:30.762300+0000) 2020-02-06T00:33:31.334 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:31.331+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.332873+0000) 2020-02-06T00:33:31.335 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:31.333+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.333786+0000) 2020-02-06T00:33:31.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:31.333+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.334470+0000) 2020-02-06T00:33:31.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:31.334+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.335530+0000) 2020-02-06T00:33:31.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:31.334+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.335843+0000) 2020-02-06T00:33:31.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:31.335+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.336295+0000) 2020-02-06T00:33:31.340 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:31.336+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.337349+0000) 2020-02-06T00:33:31.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:31.336+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.337600+0000) 2020-02-06T00:33:31.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:31.336+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.336947+0000) 2020-02-06T00:33:31.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:31.336+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.337087+0000) 2020-02-06T00:33:31.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:31.336+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.336598+0000) 2020-02-06T00:33:31.341 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:31.338+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.338746+0000) 2020-02-06T00:33:31.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:31.338+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.338864+0000) 2020-02-06T00:33:31.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:31.339+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:31.340612+0000) 2020-02-06T00:33:31.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:31.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:32.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:32.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:32.884967+0000) 2020-02-06T00:33:33.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:33.380+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:33.380950+0000) 2020-02-06T00:33:34.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:34.033+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:34.033777+0000) 2020-02-06T00:33:34.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:34.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:34.033920+0000) 2020-02-06T00:33:36.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:36.021+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.021966+0000) 2020-02-06T00:33:36.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:36.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.359486+0000) 2020-02-06T00:33:36.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:36.359+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.360351+0000) 2020-02-06T00:33:36.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:36.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.359843+0000) 2020-02-06T00:33:36.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:36.360+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.360576+0000) 2020-02-06T00:33:36.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:36.359+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.360811+0000) 2020-02-06T00:33:36.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:36.360+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.361241+0000) 2020-02-06T00:33:36.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:36.360+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.361618+0000) 2020-02-06T00:33:36.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:36.361+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.362060+0000) 2020-02-06T00:33:36.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:36.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.362147+0000) 2020-02-06T00:33:36.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:36.361+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.362171+0000) 2020-02-06T00:33:36.364 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:36.362+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.363087+0000) 2020-02-06T00:33:36.365 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:36.362+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.363586+0000) 2020-02-06T00:33:36.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:36.363+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.363971+0000) 2020-02-06T00:33:36.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:36.367+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.368335+0000) 2020-02-06T00:33:36.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:36.367+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:36.368956+0000) 2020-02-06T00:33:36.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:36.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:37.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:37.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:37.010211+0000) 2020-02-06T00:33:37.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:37.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:37.275078+0000) 2020-02-06T00:33:37.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:37.274+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:37.275032+0000) 2020-02-06T00:33:37.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:37.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:37.275296+0000) 2020-02-06T00:33:39.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:39.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:39.654065+0000) 2020-02-06T00:33:39.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:39.902+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:39.902970+0000) 2020-02-06T00:33:40.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:40.502+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:40.503516+0000) 2020-02-06T00:33:40.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:40.762+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:40.762534+0000) 2020-02-06T00:33:41.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:41.381+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.383244+0000) 2020-02-06T00:33:41.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:41.383+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.384680+0000) 2020-02-06T00:33:41.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:41.384+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.385056+0000) 2020-02-06T00:33:41.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:41.384+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.385763+0000) 2020-02-06T00:33:41.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:41.386+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.387203+0000) 2020-02-06T00:33:41.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:41.386+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.387548+0000) 2020-02-06T00:33:41.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:41.386+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.387663+0000) 2020-02-06T00:33:41.390 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:41.387+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.388812+0000) 2020-02-06T00:33:41.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:41.387+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.388201+0000) 2020-02-06T00:33:41.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:41.388+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.389416+0000) 2020-02-06T00:33:41.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:41.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.389651+0000) 2020-02-06T00:33:41.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:41.388+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.390100+0000) 2020-02-06T00:33:41.392 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:41.389+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.390337+0000) 2020-02-06T00:33:41.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:41.393+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:41.394429+0000) 2020-02-06T00:33:41.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:41.401+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177539 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:41.631 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:33:41.638 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:33:42.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:42.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:42.885211+0000) 2020-02-06T00:33:43.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:43.392+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:43.392383+0000) 2020-02-06T00:33:44.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:44.033+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:44.034048+0000) 2020-02-06T00:33:44.035 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:44.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:44.034162+0000) 2020-02-06T00:33:46.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:46.021+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.022261+0000) 2020-02-06T00:33:46.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:46.369+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.369160+0000) 2020-02-06T00:33:46.410 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:46.407+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.408612+0000) 2020-02-06T00:33:46.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:46.408+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.408757+0000) 2020-02-06T00:33:46.410 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:46.409+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.409411+0000) 2020-02-06T00:33:46.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:46.409+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.409952+0000) 2020-02-06T00:33:46.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:46.411+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.411446+0000) 2020-02-06T00:33:46.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:46.410+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.411666+0000) 2020-02-06T00:33:46.413 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:46.412+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.412220+0000) 2020-02-06T00:33:46.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:46.411+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.412484+0000) 2020-02-06T00:33:46.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:46.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.413694+0000) 2020-02-06T00:33:46.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:46.413+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.414884+0000) 2020-02-06T00:33:46.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:46.413+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.414926+0000) 2020-02-06T00:33:46.418 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:46.413+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.413501+0000) 2020-02-06T00:33:46.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:46.414+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.414681+0000) 2020-02-06T00:33:46.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:46.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:46.418263+0000) 2020-02-06T00:33:46.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:46.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177578 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:47.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:47.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:47.010425+0000) 2020-02-06T00:33:47.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:47.273+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:47.275302+0000) 2020-02-06T00:33:47.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:47.273+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:47.275289+0000) 2020-02-06T00:33:47.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:47.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:47.275550+0000) 2020-02-06T00:33:49.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:49.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:49.654340+0000) 2020-02-06T00:33:49.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:49.902+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:49.903204+0000) 2020-02-06T00:33:50.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:50.502+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:50.503769+0000) 2020-02-06T00:33:50.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:50.761+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:50.762764+0000) 2020-02-06T00:33:51.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:51.432+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.434348+0000) 2020-02-06T00:33:51.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:51.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.435036+0000) 2020-02-06T00:33:51.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:51.434+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.436066+0000) 2020-02-06T00:33:51.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:51.436+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.437267+0000) 2020-02-06T00:33:51.438 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:51.436+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.437716+0000) 2020-02-06T00:33:51.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:51.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.438498+0000) 2020-02-06T00:33:51.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:51.437+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.438859+0000) 2020-02-06T00:33:51.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:51.438+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.439818+0000) 2020-02-06T00:33:51.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:51.438+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.440078+0000) 2020-02-06T00:33:51.441 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:51.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.440452+0000) 2020-02-06T00:33:51.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:51.439+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.440680+0000) 2020-02-06T00:33:51.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:51.439+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.441279+0000) 2020-02-06T00:33:51.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:51.440+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.441530+0000) 2020-02-06T00:33:51.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:51.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:51.447353+0000) 2020-02-06T00:33:51.456 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:51.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:52.887 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:52.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:52.885530+0000) 2020-02-06T00:33:53.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:53.399+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:53.400313+0000) 2020-02-06T00:33:54.035 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:54.033+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:54.034322+0000) 2020-02-06T00:33:54.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:54.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:54.034423+0000) 2020-02-06T00:33:56.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:56.021+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.022521+0000) 2020-02-06T00:33:56.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:56.368+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.369387+0000) 2020-02-06T00:33:56.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:33:56.458+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.460023+0000) 2020-02-06T00:33:56.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:56.460+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.462209+0000) 2020-02-06T00:33:56.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:33:56.461+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.462910+0000) 2020-02-06T00:33:56.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:33:56.461+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.463282+0000) 2020-02-06T00:33:56.465 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:56.462+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.463860+0000) 2020-02-06T00:33:56.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:56.463+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.464489+0000) 2020-02-06T00:33:56.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:33:56.464+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.465128+0000) 2020-02-06T00:33:56.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:56.464+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.465622+0000) 2020-02-06T00:33:56.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:56.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.466221+0000) 2020-02-06T00:33:56.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:56.464+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.466238+0000) 2020-02-06T00:33:56.468 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:33:56.465+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.466874+0000) 2020-02-06T00:33:56.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:33:56.466+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.467209+0000) 2020-02-06T00:33:56.469 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:33:56.466+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.467872+0000) 2020-02-06T00:33:56.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:56.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:56.471781+0000) 2020-02-06T00:33:56.481 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:33:56.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177660 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:33:57.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:33:57.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:57.010631+0000) 2020-02-06T00:33:57.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:33:57.274+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:57.275505+0000) 2020-02-06T00:33:57.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:33:57.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:57.275597+0000) 2020-02-06T00:33:57.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:33:57.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:57.275752+0000) 2020-02-06T00:33:59.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:59.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:59.654581+0000) 2020-02-06T00:33:59.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:33:59.902+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:33:59.903435+0000) 2020-02-06T00:34:00.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:00.503+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:00.504025+0000) 2020-02-06T00:34:00.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:00.762+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:00.763022+0000) 2020-02-06T00:34:01.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:01.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.485470+0000) 2020-02-06T00:34:01.487 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:01.485+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.486252+0000) 2020-02-06T00:34:01.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:01.485+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.486674+0000) 2020-02-06T00:34:01.488 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:01.485+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.486983+0000) 2020-02-06T00:34:01.489 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:01.486+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.487765+0000) 2020-02-06T00:34:01.489 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:01.487+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.488599+0000) 2020-02-06T00:34:01.490 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:01.487+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.488492+0000) 2020-02-06T00:34:01.490 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:01.488+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.489457+0000) 2020-02-06T00:34:01.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:01.489+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.490515+0000) 2020-02-06T00:34:01.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:01.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.490921+0000) 2020-02-06T00:34:01.492 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:01.489+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.491026+0000) 2020-02-06T00:34:01.493 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:01.490+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.492021+0000) 2020-02-06T00:34:01.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:01.490+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.492219+0000) 2020-02-06T00:34:01.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:01.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:01.497292+0000) 2020-02-06T00:34:01.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:01.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:02.887 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:02.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:02.885783+0000) 2020-02-06T00:34:03.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:03.406+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:03.407568+0000) 2020-02-06T00:34:04.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:04.033+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:04.034547+0000) 2020-02-06T00:34:04.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:04.033+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:04.034614+0000) 2020-02-06T00:34:06.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:06.022+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.022780+0000) 2020-02-06T00:34:06.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:06.368+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.369637+0000) 2020-02-06T00:34:06.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:06.510+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.511468+0000) 2020-02-06T00:34:06.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:06.512+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.513458+0000) 2020-02-06T00:34:06.515 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:06.513+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.514144+0000) 2020-02-06T00:34:06.515 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:06.512+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.513880+0000) 2020-02-06T00:34:06.516 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:06.513+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.514453+0000) 2020-02-06T00:34:06.516 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:06.513+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.514644+0000) 2020-02-06T00:34:06.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:06.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.515242+0000) 2020-02-06T00:34:06.517 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:06.514+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.515569+0000) 2020-02-06T00:34:06.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:06.514+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.515420+0000) 2020-02-06T00:34:06.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:06.515+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.516716+0000) 2020-02-06T00:34:06.518 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:06.515+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.516877+0000) 2020-02-06T00:34:06.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:06.515+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.517404+0000) 2020-02-06T00:34:06.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:06.516+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.517823+0000) 2020-02-06T00:34:06.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:06.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:06.523017+0000) 2020-02-06T00:34:06.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:06.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:07.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:07.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:07.010844+0000) 2020-02-06T00:34:07.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:07.274+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:07.275736+0000) 2020-02-06T00:34:07.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:07.274+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:07.275845+0000) 2020-02-06T00:34:07.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:07.274+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:07.275966+0000) 2020-02-06T00:34:09.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:09.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:09.654810+0000) 2020-02-06T00:34:09.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:09.902+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:09.903656+0000) 2020-02-06T00:34:10.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:10.503+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:10.504273+0000) 2020-02-06T00:34:10.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:10.762+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:10.763310+0000) 2020-02-06T00:34:11.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:11.538+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.540343+0000) 2020-02-06T00:34:11.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:11.539+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.540686+0000) 2020-02-06T00:34:11.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:11.540+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.541231+0000) 2020-02-06T00:34:11.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:11.540+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.541465+0000) 2020-02-06T00:34:11.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:11.540+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.541902+0000) 2020-02-06T00:34:11.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:11.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.542403+0000) 2020-02-06T00:34:11.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:11.541+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.542777+0000) 2020-02-06T00:34:11.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:11.542+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.543578+0000) 2020-02-06T00:34:11.545 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:11.541+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.543114+0000) 2020-02-06T00:34:11.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:11.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.543897+0000) 2020-02-06T00:34:11.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:11.542+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.543909+0000) 2020-02-06T00:34:11.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:11.543+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.544866+0000) 2020-02-06T00:34:11.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:11.544+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.546200+0000) 2020-02-06T00:34:11.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:11.549+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:11.550701+0000) 2020-02-06T00:34:11.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:11.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177788 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:11.760 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:34:11.766 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:34:12.887 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:12.884+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:12.886044+0000) 2020-02-06T00:34:13.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:13.418+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:13.419531+0000) 2020-02-06T00:34:14.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:14.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:14.034855+0000) 2020-02-06T00:34:14.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:14.034+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:14.034819+0000) 2020-02-06T00:34:16.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:16.022+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.023036+0000) 2020-02-06T00:34:16.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:16.369+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.369863+0000) 2020-02-06T00:34:16.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:16.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.564752+0000) 2020-02-06T00:34:16.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:16.564+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.565471+0000) 2020-02-06T00:34:16.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:16.563+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.565277+0000) 2020-02-06T00:34:16.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:16.564+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.566189+0000) 2020-02-06T00:34:16.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:16.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.566673+0000) 2020-02-06T00:34:16.568 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:16.566+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.567162+0000) 2020-02-06T00:34:16.569 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:16.566+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.567509+0000) 2020-02-06T00:34:16.569 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:16.566+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.567643+0000) 2020-02-06T00:34:16.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:16.567+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.568763+0000) 2020-02-06T00:34:16.570 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:16.567+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.569190+0000) 2020-02-06T00:34:16.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:16.568+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.569798+0000) 2020-02-06T00:34:16.571 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:16.568+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.570154+0000) 2020-02-06T00:34:16.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:16.569+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.570426+0000) 2020-02-06T00:34:16.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:16.574+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:16.575797+0000) 2020-02-06T00:34:16.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:16.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:17.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:17.009+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:17.011055+0000) 2020-02-06T00:34:17.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:17.275+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:17.275948+0000) 2020-02-06T00:34:17.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:17.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:17.276076+0000) 2020-02-06T00:34:17.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:17.275+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:17.276333+0000) 2020-02-06T00:34:19.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:19.653+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:19.655041+0000) 2020-02-06T00:34:19.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:19.903+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:19.903887+0000) 2020-02-06T00:34:20.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:20.503+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:20.504529+0000) 2020-02-06T00:34:20.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:20.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:20.763542+0000) 2020-02-06T00:34:21.591 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:21.588+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.589528+0000) 2020-02-06T00:34:21.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:21.589+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.591132+0000) 2020-02-06T00:34:21.593 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:21.590+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.591626+0000) 2020-02-06T00:34:21.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:21.590+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.592000+0000) 2020-02-06T00:34:21.593 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:21.590+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.592357+0000) 2020-02-06T00:34:21.594 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:21.591+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.592875+0000) 2020-02-06T00:34:21.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:21.592+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.593324+0000) 2020-02-06T00:34:21.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:21.593+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.594286+0000) 2020-02-06T00:34:21.596 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:21.593+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.594989+0000) 2020-02-06T00:34:21.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:21.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.594809+0000) 2020-02-06T00:34:21.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:21.594+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.596052+0000) 2020-02-06T00:34:21.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:21.594+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.596411+0000) 2020-02-06T00:34:21.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:21.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.596551+0000) 2020-02-06T00:34:21.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:21.598+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:21.599659+0000) 2020-02-06T00:34:21.611 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:21.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:22.887 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:22.885+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:22.886316+0000) 2020-02-06T00:34:23.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:23.427+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:23.427427+0000) 2020-02-06T00:34:24.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:24.034+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:24.035071+0000) 2020-02-06T00:34:24.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:24.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:24.035107+0000) 2020-02-06T00:34:26.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:26.022+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.023298+0000) 2020-02-06T00:34:26.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:26.369+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.370135+0000) 2020-02-06T00:34:26.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:26.615+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.616681+0000) 2020-02-06T00:34:26.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:26.615+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.617154+0000) 2020-02-06T00:34:26.619 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:26.617+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.618062+0000) 2020-02-06T00:34:26.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:26.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.619215+0000) 2020-02-06T00:34:26.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:26.618+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.619833+0000) 2020-02-06T00:34:26.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:26.617+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.618686+0000) 2020-02-06T00:34:26.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:26.617+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.618753+0000) 2020-02-06T00:34:26.622 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:26.619+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.620280+0000) 2020-02-06T00:34:26.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:26.619+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.620578+0000) 2020-02-06T00:34:26.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:26.620+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.621715+0000) 2020-02-06T00:34:26.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:26.620+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.621917+0000) 2020-02-06T00:34:26.625 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:26.620+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.622214+0000) 2020-02-06T00:34:26.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:26.621+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.622476+0000) 2020-02-06T00:34:26.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:26.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:26.626625+0000) 2020-02-06T00:34:26.637 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:26.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:27.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:27.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:27.011319+0000) 2020-02-06T00:34:27.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:27.275+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:27.276208+0000) 2020-02-06T00:34:27.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:27.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:27.276316+0000) 2020-02-06T00:34:27.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:27.275+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:27.276522+0000) 2020-02-06T00:34:29.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:29.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:29.655314+0000) 2020-02-06T00:34:29.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:29.903+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:29.904140+0000) 2020-02-06T00:34:30.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:30.504+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:30.504775+0000) 2020-02-06T00:34:30.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:30.762+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:30.763770+0000) 2020-02-06T00:34:31.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:31.640+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.641442+0000) 2020-02-06T00:34:31.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:31.641+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.642257+0000) 2020-02-06T00:34:31.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:31.641+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.642408+0000) 2020-02-06T00:34:31.644 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:31.641+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.642823+0000) 2020-02-06T00:34:31.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:31.642+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.642867+0000) 2020-02-06T00:34:31.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:31.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.643446+0000) 2020-02-06T00:34:31.645 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:31.643+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.643834+0000) 2020-02-06T00:34:31.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:31.643+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.643993+0000) 2020-02-06T00:34:31.646 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:31.643+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.645043+0000) 2020-02-06T00:34:31.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:31.644+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.645341+0000) 2020-02-06T00:34:31.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:31.644+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.645654+0000) 2020-02-06T00:34:31.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:31.644+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.645389+0000) 2020-02-06T00:34:31.648 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:31.645+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.645757+0000) 2020-02-06T00:34:31.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:31.649+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:31.650341+0000) 2020-02-06T00:34:31.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:31.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:32.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:32.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:32.886574+0000) 2020-02-06T00:34:33.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:33.435+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:33.435683+0000) 2020-02-06T00:34:34.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:34.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:34.035360+0000) 2020-02-06T00:34:34.037 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:34.035+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:34.035360+0000) 2020-02-06T00:34:36.025 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:36.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.023554+0000) 2020-02-06T00:34:36.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:36.370+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.370363+0000) 2020-02-06T00:34:36.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:36.665+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.666881+0000) 2020-02-06T00:34:36.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:36.668+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.669365+0000) 2020-02-06T00:34:36.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:36.670+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.670897+0000) 2020-02-06T00:34:36.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:36.670+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.671154+0000) 2020-02-06T00:34:36.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:36.671+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.671934+0000) 2020-02-06T00:34:36.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:36.671+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.671765+0000) 2020-02-06T00:34:36.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:36.672+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.672834+0000) 2020-02-06T00:34:36.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:36.667+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.668141+0000) 2020-02-06T00:34:36.682 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:36.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.668516+0000) 2020-02-06T00:34:36.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:36.667+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.668828+0000) 2020-02-06T00:34:36.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:36.668+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.669716+0000) 2020-02-06T00:34:36.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:36.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.670662+0000) 2020-02-06T00:34:36.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:36.673+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.674332+0000) 2020-02-06T00:34:36.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:36.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:36.675264+0000) 2020-02-06T00:34:36.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:36.686+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 177993 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:37.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:37.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:37.011534+0000) 2020-02-06T00:34:37.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:37.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:37.276464+0000) 2020-02-06T00:34:37.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:37.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:37.276558+0000) 2020-02-06T00:34:37.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:37.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:37.276707+0000) 2020-02-06T00:34:39.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:39.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:39.655534+0000) 2020-02-06T00:34:39.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:39.904+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:39.904375+0000) 2020-02-06T00:34:40.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:40.504+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:40.505081+0000) 2020-02-06T00:34:40.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:40.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:40.764008+0000) 2020-02-06T00:34:41.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:41.692+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.692157+0000) 2020-02-06T00:34:41.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:41.692+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.692683+0000) 2020-02-06T00:34:41.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:41.693+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.694044+0000) 2020-02-06T00:34:41.707 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:41.694+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.694693+0000) 2020-02-06T00:34:41.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:41.695+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.696261+0000) 2020-02-06T00:34:41.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:41.697+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.697747+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:41.698+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.698462+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:41.694+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.694846+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:41.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.697538+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:41.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.703664+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:41.695+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.695267+0000) 2020-02-06T00:34:41.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:41.695+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.695548+0000) 2020-02-06T00:34:41.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:41.696+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.696639+0000) 2020-02-06T00:34:41.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:41.696+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:41.697016+0000) 2020-02-06T00:34:41.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:41.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178037 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:41.916 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:34:41.922 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:34:42.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:42.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:42.886854+0000) 2020-02-06T00:34:43.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:43.446+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:43.447951+0000) 2020-02-06T00:34:44.037 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:44.034+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:44.035606+0000) 2020-02-06T00:34:44.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:44.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:44.035601+0000) 2020-02-06T00:34:46.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:46.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.023811+0000) 2020-02-06T00:34:46.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:46.369+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.370613+0000) 2020-02-06T00:34:46.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:46.717+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.718346+0000) 2020-02-06T00:34:46.720 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:46.718+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.719359+0000) 2020-02-06T00:34:46.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:46.718+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.719726+0000) 2020-02-06T00:34:46.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:46.720+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.720616+0000) 2020-02-06T00:34:46.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:46.720+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.721114+0000) 2020-02-06T00:34:46.722 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:46.720+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.721497+0000) 2020-02-06T00:34:46.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:46.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.722328+0000) 2020-02-06T00:34:46.723 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:46.721+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.722577+0000) 2020-02-06T00:34:46.724 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:46.722+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.722911+0000) 2020-02-06T00:34:46.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:46.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.723208+0000) 2020-02-06T00:34:46.725 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:46.723+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.723876+0000) 2020-02-06T00:34:46.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:46.724+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.724596+0000) 2020-02-06T00:34:46.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:46.724+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.724663+0000) 2020-02-06T00:34:46.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:46.728+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:46.730043+0000) 2020-02-06T00:34:46.742 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:46.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178077 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:47.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:47.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:47.011750+0000) 2020-02-06T00:34:47.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:47.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:47.276772+0000) 2020-02-06T00:34:47.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:47.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:47.276710+0000) 2020-02-06T00:34:47.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:47.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:47.276934+0000) 2020-02-06T00:34:49.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:49.654+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:49.655757+0000) 2020-02-06T00:34:49.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:49.903+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:49.904602+0000) 2020-02-06T00:34:50.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:50.505+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:50.505315+0000) 2020-02-06T00:34:50.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:50.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:50.764209+0000) 2020-02-06T00:34:51.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:51.744+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.745536+0000) 2020-02-06T00:34:51.747 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:51.745+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.746256+0000) 2020-02-06T00:34:51.748 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:51.747+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.747489+0000) 2020-02-06T00:34:51.749 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:51.747+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.748170+0000) 2020-02-06T00:34:51.749 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:51.747+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.748279+0000) 2020-02-06T00:34:51.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:51.747+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.748618+0000) 2020-02-06T00:34:51.750 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:51.748+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.749056+0000) 2020-02-06T00:34:51.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:51.748+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.749134+0000) 2020-02-06T00:34:51.751 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:51.749+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.749456+0000) 2020-02-06T00:34:51.751 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:51.749+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.750143+0000) 2020-02-06T00:34:51.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:51.749+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.750658+0000) 2020-02-06T00:34:51.752 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:51.749+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.750187+0000) 2020-02-06T00:34:51.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:51.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.753759+0000) 2020-02-06T00:34:51.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:51.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:51.753967+0000) 2020-02-06T00:34:51.770 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:51.766+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178120 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:52.888 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:52.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:52.887128+0000) 2020-02-06T00:34:53.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:53.454+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:53.455984+0000) 2020-02-06T00:34:54.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:54.034+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:54.035841+0000) 2020-02-06T00:34:54.037 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:54.034+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:54.035832+0000) 2020-02-06T00:34:56.025 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:56.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.024072+0000) 2020-02-06T00:34:56.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:56.369+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.370856+0000) 2020-02-06T00:34:56.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:34:56.770+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.771915+0000) 2020-02-06T00:34:56.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:34:56.772+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.772631+0000) 2020-02-06T00:34:56.774 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:34:56.772+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.773372+0000) 2020-02-06T00:34:56.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:34:56.772+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.774000+0000) 2020-02-06T00:34:56.776 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:34:56.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.774492+0000) 2020-02-06T00:34:56.776 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:56.774+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.774927+0000) 2020-02-06T00:34:56.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:56.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.776056+0000) 2020-02-06T00:34:56.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:56.775+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.776341+0000) 2020-02-06T00:34:56.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:34:56.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.777152+0000) 2020-02-06T00:34:56.778 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:34:56.776+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.777504+0000) 2020-02-06T00:34:56.779 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:56.776+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.777732+0000) 2020-02-06T00:34:56.779 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:56.777+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.778301+0000) 2020-02-06T00:34:56.780 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:56.777+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.778384+0000) 2020-02-06T00:34:56.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:56.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:56.780963+0000) 2020-02-06T00:34:56.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:34:56.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:34:57.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:34:57.010+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:57.011977+0000) 2020-02-06T00:34:57.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:34:57.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:57.276989+0000) 2020-02-06T00:34:57.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:34:57.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:57.276972+0000) 2020-02-06T00:34:57.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:34:57.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:57.277118+0000) 2020-02-06T00:34:59.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:59.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:59.655984+0000) 2020-02-06T00:34:59.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:34:59.903+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:34:59.904831+0000) 2020-02-06T00:35:00.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:00.505+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:00.505565+0000) 2020-02-06T00:35:00.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:00.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:00.764420+0000) 2020-02-06T00:35:01.799 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:01.797+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.797978+0000) 2020-02-06T00:35:01.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:01.798+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.798862+0000) 2020-02-06T00:35:01.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:01.798+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.799060+0000) 2020-02-06T00:35:01.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:01.798+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.799910+0000) 2020-02-06T00:35:01.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:01.799+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.800377+0000) 2020-02-06T00:35:01.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:01.799+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.800888+0000) 2020-02-06T00:35:01.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:01.800+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.800662+0000) 2020-02-06T00:35:01.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:01.800+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.801546+0000) 2020-02-06T00:35:01.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:01.800+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.801638+0000) 2020-02-06T00:35:01.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:01.801+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.802671+0000) 2020-02-06T00:35:01.804 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:01.802+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.803016+0000) 2020-02-06T00:35:01.804 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:01.803+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.803664+0000) 2020-02-06T00:35:01.805 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:01.803+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.803814+0000) 2020-02-06T00:35:01.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:01.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:01.808667+0000) 2020-02-06T00:35:01.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:01.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:02.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:02.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:02.887482+0000) 2020-02-06T00:35:03.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:03.463+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:03.463742+0000) 2020-02-06T00:35:04.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:04.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:04.036120+0000) 2020-02-06T00:35:04.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:04.035+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:04.036134+0000) 2020-02-06T00:35:06.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:06.024+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.024325+0000) 2020-02-06T00:35:06.373 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:06.370+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.371130+0000) 2020-02-06T00:35:06.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:06.822+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.823524+0000) 2020-02-06T00:35:06.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:06.823+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.824257+0000) 2020-02-06T00:35:06.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:06.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.825497+0000) 2020-02-06T00:35:06.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:06.824+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.825837+0000) 2020-02-06T00:35:06.827 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:06.825+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.826210+0000) 2020-02-06T00:35:06.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:06.825+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.826515+0000) 2020-02-06T00:35:06.828 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:06.826+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.827131+0000) 2020-02-06T00:35:06.829 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:06.827+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.827816+0000) 2020-02-06T00:35:06.829 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:06.828+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.828498+0000) 2020-02-06T00:35:06.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:06.827+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.828884+0000) 2020-02-06T00:35:06.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:06.828+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.829295+0000) 2020-02-06T00:35:06.831 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:06.828+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.829364+0000) 2020-02-06T00:35:06.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:06.829+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.829706+0000) 2020-02-06T00:35:06.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:06.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:06.836065+0000) 2020-02-06T00:35:06.844 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:06.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:07.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:07.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:07.012186+0000) 2020-02-06T00:35:07.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:07.275+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:07.277225+0000) 2020-02-06T00:35:07.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:07.275+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:07.277225+0000) 2020-02-06T00:35:07.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:07.275+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:07.277342+0000) 2020-02-06T00:35:09.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:09.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:09.656221+0000) 2020-02-06T00:35:09.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:09.904+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:09.905056+0000) 2020-02-06T00:35:10.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:10.504+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:10.505803+0000) 2020-02-06T00:35:10.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:10.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:10.764691+0000) 2020-02-06T00:35:11.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:11.849+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.850742+0000) 2020-02-06T00:35:11.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:11.849+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.851228+0000) 2020-02-06T00:35:11.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:11.850+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.851726+0000) 2020-02-06T00:35:11.853 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:11.850+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.852177+0000) 2020-02-06T00:35:11.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:11.851+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.852631+0000) 2020-02-06T00:35:11.854 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:11.851+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.852808+0000) 2020-02-06T00:35:11.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:11.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.853235+0000) 2020-02-06T00:35:11.856 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:11.853+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.854599+0000) 2020-02-06T00:35:11.856 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:11.852+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.854380+0000) 2020-02-06T00:35:11.856 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:11.853+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.855018+0000) 2020-02-06T00:35:11.857 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:11.854+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.855907+0000) 2020-02-06T00:35:11.857 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:11.854+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.856109+0000) 2020-02-06T00:35:11.857 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:11.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.856251+0000) 2020-02-06T00:35:11.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:11.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:11.858386+0000) 2020-02-06T00:35:11.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:11.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178286 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:12.120 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:35:12.127 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:35:12.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:12.886+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:12.887751+0000) 2020-02-06T00:35:13.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:13.470+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:13.471989+0000) 2020-02-06T00:35:14.037 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:14.035+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:14.036313+0000) 2020-02-06T00:35:14.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:14.035+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:14.036313+0000) 2020-02-06T00:35:16.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:16.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.024656+0000) 2020-02-06T00:35:16.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:16.370+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.371385+0000) 2020-02-06T00:35:16.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:16.874+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.875917+0000) 2020-02-06T00:35:16.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:16.875+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.876578+0000) 2020-02-06T00:35:16.878 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:16.875+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.877232+0000) 2020-02-06T00:35:16.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:16.876+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.877612+0000) 2020-02-06T00:35:16.879 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:16.876+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.877944+0000) 2020-02-06T00:35:16.879 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:16.876+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.878088+0000) 2020-02-06T00:35:16.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:16.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.878791+0000) 2020-02-06T00:35:16.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:16.878+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.879281+0000) 2020-02-06T00:35:16.880 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:16.878+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.879705+0000) 2020-02-06T00:35:16.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:16.878+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.879369+0000) 2020-02-06T00:35:16.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:16.878+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.879864+0000) 2020-02-06T00:35:16.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:16.879+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.880698+0000) 2020-02-06T00:35:16.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:16.879+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.880954+0000) 2020-02-06T00:35:16.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:16.885+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:16.886668+0000) 2020-02-06T00:35:16.896 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:16.893+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:17.013 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:17.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:17.012399+0000) 2020-02-06T00:35:17.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:17.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:17.277504+0000) 2020-02-06T00:35:17.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:17.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:17.277483+0000) 2020-02-06T00:35:17.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:17.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:17.277592+0000) 2020-02-06T00:35:19.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:19.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:19.656472+0000) 2020-02-06T00:35:19.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:19.904+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:19.905275+0000) 2020-02-06T00:35:20.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:20.505+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:20.506060+0000) 2020-02-06T00:35:20.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:20.763+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:20.764870+0000) 2020-02-06T00:35:21.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:21.899+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.900558+0000) 2020-02-06T00:35:21.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:21.900+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.901406+0000) 2020-02-06T00:35:21.903 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:21.901+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.902285+0000) 2020-02-06T00:35:21.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:21.901+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.902542+0000) 2020-02-06T00:35:21.904 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:21.901+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.902990+0000) 2020-02-06T00:35:21.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:21.903+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.904181+0000) 2020-02-06T00:35:21.906 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:21.902+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.903661+0000) 2020-02-06T00:35:21.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:21.904+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.905585+0000) 2020-02-06T00:35:21.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:21.905+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.906308+0000) 2020-02-06T00:35:21.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:21.904+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.906260+0000) 2020-02-06T00:35:21.908 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:21.905+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.906652+0000) 2020-02-06T00:35:21.908 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:21.905+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.906890+0000) 2020-02-06T00:35:21.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:21.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.908005+0000) 2020-02-06T00:35:21.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:21.908+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:21.909256+0000) 2020-02-06T00:35:21.921 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:21.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:22.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:22.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:22.888006+0000) 2020-02-06T00:35:23.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:23.478+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:23.480006+0000) 2020-02-06T00:35:24.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:24.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:24.036553+0000) 2020-02-06T00:35:24.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:24.036+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:24.036551+0000) 2020-02-06T00:35:26.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:26.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.024893+0000) 2020-02-06T00:35:26.372 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:26.371+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.371614+0000) 2020-02-06T00:35:26.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:26.924+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.925415+0000) 2020-02-06T00:35:26.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:26.924+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.925600+0000) 2020-02-06T00:35:26.927 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:26.925+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.926229+0000) 2020-02-06T00:35:26.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:26.926+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.927481+0000) 2020-02-06T00:35:26.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:26.926+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.928126+0000) 2020-02-06T00:35:26.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:26.927+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.928672+0000) 2020-02-06T00:35:26.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:26.927+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.928744+0000) 2020-02-06T00:35:26.931 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:26.928+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.929644+0000) 2020-02-06T00:35:26.931 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:26.928+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.929813+0000) 2020-02-06T00:35:26.931 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:26.928+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.930238+0000) 2020-02-06T00:35:26.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:26.929+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.931163+0000) 2020-02-06T00:35:26.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:26.929+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.930915+0000) 2020-02-06T00:35:26.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:26.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.931547+0000) 2020-02-06T00:35:26.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:26.933+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:26.934282+0000) 2020-02-06T00:35:26.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:26.941+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:27.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:27.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:27.012649+0000) 2020-02-06T00:35:27.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:27.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:27.277727+0000) 2020-02-06T00:35:27.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:27.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:27.277707+0000) 2020-02-06T00:35:27.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:27.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:27.277835+0000) 2020-02-06T00:35:29.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:29.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:29.656700+0000) 2020-02-06T00:35:29.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:29.905+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:29.905529+0000) 2020-02-06T00:35:30.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:30.505+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:30.506317+0000) 2020-02-06T00:35:30.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:30.764+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:30.765129+0000) 2020-02-06T00:35:31.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:31.947+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.948941+0000) 2020-02-06T00:35:31.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:31.948+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.949380+0000) 2020-02-06T00:35:31.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:31.947+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.948509+0000) 2020-02-06T00:35:31.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:31.948+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.949711+0000) 2020-02-06T00:35:31.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:31.949+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.950323+0000) 2020-02-06T00:35:31.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:31.949+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.951372+0000) 2020-02-06T00:35:31.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:31.950+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.952332+0000) 2020-02-06T00:35:31.954 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:31.952+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.953341+0000) 2020-02-06T00:35:31.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:31.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.953888+0000) 2020-02-06T00:35:31.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:31.953+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.954718+0000) 2020-02-06T00:35:31.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:31.954+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.955135+0000) 2020-02-06T00:35:31.956 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:31.953+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.955016+0000) 2020-02-06T00:35:31.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:31.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.955655+0000) 2020-02-06T00:35:31.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:31.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:31.958270+0000) 2020-02-06T00:35:31.971 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:31.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:32.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:32.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:32.888171+0000) 2020-02-06T00:35:33.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:33.488+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:33.488109+0000) 2020-02-06T00:35:34.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:34.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:34.036807+0000) 2020-02-06T00:35:34.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:34.036+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:34.036809+0000) 2020-02-06T00:35:36.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:36.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.025154+0000) 2020-02-06T00:35:36.373 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:36.371+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.371849+0000) 2020-02-06T00:35:36.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:36.972+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.973468+0000) 2020-02-06T00:35:36.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:36.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.974270+0000) 2020-02-06T00:35:36.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:36.973+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.974714+0000) 2020-02-06T00:35:36.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:36.974+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.975215+0000) 2020-02-06T00:35:36.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:36.974+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.976057+0000) 2020-02-06T00:35:36.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:36.975+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.976613+0000) 2020-02-06T00:35:36.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:36.976+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.977904+0000) 2020-02-06T00:35:36.979 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:36.976+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.977579+0000) 2020-02-06T00:35:36.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:36.977+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.978619+0000) 2020-02-06T00:35:36.980 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:36.976+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.978384+0000) 2020-02-06T00:35:36.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:36.978+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.979663+0000) 2020-02-06T00:35:36.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:36.978+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.980160+0000) 2020-02-06T00:35:36.981 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:36.979+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.980682+0000) 2020-02-06T00:35:36.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:36.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:36.985513+0000) 2020-02-06T00:35:36.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:36.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:37.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:37.011+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:37.012864+0000) 2020-02-06T00:35:37.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:37.276+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:37.277949+0000) 2020-02-06T00:35:37.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:37.276+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:37.277971+0000) 2020-02-06T00:35:37.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:37.276+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:37.278036+0000) 2020-02-06T00:35:39.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:39.655+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:39.656897+0000) 2020-02-06T00:35:39.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:39.904+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:39.905784+0000) 2020-02-06T00:35:40.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:40.506+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:40.506706+0000) 2020-02-06T00:35:40.766 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:40.764+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:40.765335+0000) 2020-02-06T00:35:42.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:41.998+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:41.999519+0000) 2020-02-06T00:35:42.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:41.998+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:41.999932+0000) 2020-02-06T00:35:42.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:41.999+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.000515+0000) 2020-02-06T00:35:42.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:41.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.000470+0000) 2020-02-06T00:35:42.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:41.999+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.001014+0000) 2020-02-06T00:35:42.002 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:42.000+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.001646+0000) 2020-02-06T00:35:42.003 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:42.000+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.002138+0000) 2020-02-06T00:35:42.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:42.000+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.002386+0000) 2020-02-06T00:35:42.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:42.001+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.002798+0000) 2020-02-06T00:35:42.005 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:42.001+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.003304+0000) 2020-02-06T00:35:42.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:42.003+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.004243+0000) 2020-02-06T00:35:42.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:42.003+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.005259+0000) 2020-02-06T00:35:42.006 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:42.004+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.005393+0000) 2020-02-06T00:35:42.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:42.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.011514+0000) 2020-02-06T00:35:42.021 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:42.018+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178529 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:42.299 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:35:42.305 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:35:42.889 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:42.887+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:42.888422+0000) 2020-02-06T00:35:43.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:43.495+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:43.496219+0000) 2020-02-06T00:35:44.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:44.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:44.037048+0000) 2020-02-06T00:35:44.038 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:44.036+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:44.037105+0000) 2020-02-06T00:35:46.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:46.023+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:46.025407+0000) 2020-02-06T00:35:46.373 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:46.371+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:46.372134+0000) 2020-02-06T00:35:47.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:47.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.013105+0000) 2020-02-06T00:35:47.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:47.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.025819+0000) 2020-02-06T00:35:47.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:47.025+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.026456+0000) 2020-02-06T00:35:47.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:47.025+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.026875+0000) 2020-02-06T00:35:47.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:47.026+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.027688+0000) 2020-02-06T00:35:47.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:47.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.027981+0000) 2020-02-06T00:35:47.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:47.027+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.028336+0000) 2020-02-06T00:35:47.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:47.027+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.028817+0000) 2020-02-06T00:35:47.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:47.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.029844+0000) 2020-02-06T00:35:47.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:47.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.030556+0000) 2020-02-06T00:35:47.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:47.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.031358+0000) 2020-02-06T00:35:47.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:47.030+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.031756+0000) 2020-02-06T00:35:47.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:47.031+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.032269+0000) 2020-02-06T00:35:47.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:47.031+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.032421+0000) 2020-02-06T00:35:47.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:47.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.036298+0000) 2020-02-06T00:35:47.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:47.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:47.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:47.277+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.278206+0000) 2020-02-06T00:35:47.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:47.277+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.278232+0000) 2020-02-06T00:35:47.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:47.277+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:47.278218+0000) 2020-02-06T00:35:49.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:49.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:49.657154+0000) 2020-02-06T00:35:49.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:49.905+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:49.906021+0000) 2020-02-06T00:35:50.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:50.506+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:50.506960+0000) 2020-02-06T00:35:50.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:50.764+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:50.765578+0000) 2020-02-06T00:35:52.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:52.050+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.051756+0000) 2020-02-06T00:35:52.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:52.050+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.052114+0000) 2020-02-06T00:35:52.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:52.052+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.053196+0000) 2020-02-06T00:35:52.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:52.052+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.053644+0000) 2020-02-06T00:35:52.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:52.052+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.053887+0000) 2020-02-06T00:35:52.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:52.052+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.054391+0000) 2020-02-06T00:35:52.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:52.053+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.054598+0000) 2020-02-06T00:35:52.056 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:52.054+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.055362+0000) 2020-02-06T00:35:52.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:52.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.055588+0000) 2020-02-06T00:35:52.057 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:52.055+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.056538+0000) 2020-02-06T00:35:52.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:52.055+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.057407+0000) 2020-02-06T00:35:52.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:52.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.057665+0000) 2020-02-06T00:35:52.059 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:52.056+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.057641+0000) 2020-02-06T00:35:52.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:52.058+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.059097+0000) 2020-02-06T00:35:52.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:52.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:52.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:52.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:52.888714+0000) 2020-02-06T00:35:53.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:53.503+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:53.504325+0000) 2020-02-06T00:35:54.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:54.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:54.037319+0000) 2020-02-06T00:35:54.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:54.036+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:54.037352+0000) 2020-02-06T00:35:56.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:56.025+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:56.025683+0000) 2020-02-06T00:35:56.373 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:56.371+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:56.372364+0000) 2020-02-06T00:35:57.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:57.012+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.013354+0000) 2020-02-06T00:35:57.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:35:57.075+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.076890+0000) 2020-02-06T00:35:57.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:57.076+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.077929+0000) 2020-02-06T00:35:57.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:35:57.077+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.078915+0000) 2020-02-06T00:35:57.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:57.078+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.079645+0000) 2020-02-06T00:35:57.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:35:57.078+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.079716+0000) 2020-02-06T00:35:57.081 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:35:57.079+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.080677+0000) 2020-02-06T00:35:57.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:35:57.079+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.081226+0000) 2020-02-06T00:35:57.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:35:57.080+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.081146+0000) 2020-02-06T00:35:57.083 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:35:57.080+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.081620+0000) 2020-02-06T00:35:57.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:57.079+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.080577+0000) 2020-02-06T00:35:57.084 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:57.080+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.082232+0000) 2020-02-06T00:35:57.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:57.081+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.082319+0000) 2020-02-06T00:35:57.084 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:35:57.081+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.082363+0000) 2020-02-06T00:35:57.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:57.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.085525+0000) 2020-02-06T00:35:57.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:35:57.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178651 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:35:57.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:35:57.277+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.278510+0000) 2020-02-06T00:35:57.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:35:57.277+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.278504+0000) 2020-02-06T00:35:57.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:35:57.277+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:57.278505+0000) 2020-02-06T00:35:59.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:59.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:59.657400+0000) 2020-02-06T00:35:59.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:35:59.905+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:35:59.906282+0000) 2020-02-06T00:36:00.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:00.506+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:00.507234+0000) 2020-02-06T00:36:00.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:00.765+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:00.765810+0000) 2020-02-06T00:36:02.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:02.102+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.103396+0000) 2020-02-06T00:36:02.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:02.103+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.104185+0000) 2020-02-06T00:36:02.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:02.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.104981+0000) 2020-02-06T00:36:02.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:02.104+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.105317+0000) 2020-02-06T00:36:02.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:02.105+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.105542+0000) 2020-02-06T00:36:02.107 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:02.105+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.105788+0000) 2020-02-06T00:36:02.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:02.105+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.106171+0000) 2020-02-06T00:36:02.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:02.105+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.106726+0000) 2020-02-06T00:36:02.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:02.106+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.107273+0000) 2020-02-06T00:36:02.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:02.106+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.106609+0000) 2020-02-06T00:36:02.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:02.106+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.106793+0000) 2020-02-06T00:36:02.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:02.106+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.107405+0000) 2020-02-06T00:36:02.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:02.106+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.107867+0000) 2020-02-06T00:36:02.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:02.110+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.112063+0000) 2020-02-06T00:36:02.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:02.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:02.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:02.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:02.888967+0000) 2020-02-06T00:36:03.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:03.511+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:03.512479+0000) 2020-02-06T00:36:04.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:04.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:04.037559+0000) 2020-02-06T00:36:04.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:04.037+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:04.037590+0000) 2020-02-06T00:36:06.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:06.025+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:06.025946+0000) 2020-02-06T00:36:06.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:06.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:06.372633+0000) 2020-02-06T00:36:07.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:07.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.013584+0000) 2020-02-06T00:36:07.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:07.126+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.127514+0000) 2020-02-06T00:36:07.129 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:07.127+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.128067+0000) 2020-02-06T00:36:07.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:07.127+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.128367+0000) 2020-02-06T00:36:07.130 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:07.128+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.129062+0000) 2020-02-06T00:36:07.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:07.128+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.129255+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:07.130+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.130901+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:07.131+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.131645+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:07.129+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.130338+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:07.129+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.130601+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:07.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.131114+0000) 2020-02-06T00:36:07.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:07.131+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.132739+0000) 2020-02-06T00:36:07.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:07.132+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.132670+0000) 2020-02-06T00:36:07.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:07.132+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.133222+0000) 2020-02-06T00:36:07.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:07.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.137789+0000) 2020-02-06T00:36:07.146 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:07.144+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:07.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:07.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.278766+0000) 2020-02-06T00:36:07.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:07.278+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.278768+0000) 2020-02-06T00:36:07.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:07.278+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:07.278749+0000) 2020-02-06T00:36:09.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:09.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:09.657633+0000) 2020-02-06T00:36:09.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:09.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:09.906508+0000) 2020-02-06T00:36:10.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:10.507+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:10.507473+0000) 2020-02-06T00:36:10.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:10.765+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:10.766037+0000) 2020-02-06T00:36:12.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:12.149+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.150909+0000) 2020-02-06T00:36:12.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:12.150+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.151687+0000) 2020-02-06T00:36:12.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:12.151+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.152686+0000) 2020-02-06T00:36:12.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:12.152+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.152642+0000) 2020-02-06T00:36:12.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:12.153+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.153711+0000) 2020-02-06T00:36:12.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:12.152+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.153952+0000) 2020-02-06T00:36:12.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:12.153+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.154470+0000) 2020-02-06T00:36:12.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:12.153+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.154506+0000) 2020-02-06T00:36:12.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:12.154+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.155383+0000) 2020-02-06T00:36:12.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:12.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.156383+0000) 2020-02-06T00:36:12.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:12.155+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.156376+0000) 2020-02-06T00:36:12.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:12.155+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.156334+0000) 2020-02-06T00:36:12.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:12.156+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.156877+0000) 2020-02-06T00:36:12.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:12.159+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.160970+0000) 2020-02-06T00:36:12.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:12.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:12.463 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:36:12.469 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:36:12.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:12.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:12.889237+0000) 2020-02-06T00:36:13.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:13.520+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:13.520713+0000) 2020-02-06T00:36:14.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:14.036+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:14.037778+0000) 2020-02-06T00:36:14.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:14.036+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:14.037822+0000) 2020-02-06T00:36:16.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:16.025+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:16.026194+0000) 2020-02-06T00:36:16.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:16.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:16.372859+0000) 2020-02-06T00:36:17.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:17.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.013819+0000) 2020-02-06T00:36:17.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:17.176+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.177330+0000) 2020-02-06T00:36:17.180 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:17.178+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.178919+0000) 2020-02-06T00:36:17.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:17.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.179056+0000) 2020-02-06T00:36:17.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:17.178+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.179411+0000) 2020-02-06T00:36:17.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:17.178+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.179931+0000) 2020-02-06T00:36:17.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:17.180+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.180697+0000) 2020-02-06T00:36:17.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:17.180+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.180861+0000) 2020-02-06T00:36:17.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:17.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.181375+0000) 2020-02-06T00:36:17.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:17.180+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.181712+0000) 2020-02-06T00:36:17.183 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:17.181+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.181647+0000) 2020-02-06T00:36:17.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:17.181+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.182412+0000) 2020-02-06T00:36:17.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:17.181+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.182709+0000) 2020-02-06T00:36:17.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:17.183+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.183587+0000) 2020-02-06T00:36:17.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:17.187+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.188368+0000) 2020-02-06T00:36:17.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:17.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:17.280 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:17.278+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.279004+0000) 2020-02-06T00:36:17.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:17.278+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.278965+0000) 2020-02-06T00:36:17.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:17.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:17.279001+0000) 2020-02-06T00:36:19.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:19.656+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:19.657891+0000) 2020-02-06T00:36:19.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:19.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:19.906735+0000) 2020-02-06T00:36:20.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:20.507+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:20.507726+0000) 2020-02-06T00:36:20.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:20.766+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:20.766287+0000) 2020-02-06T00:36:22.206 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:22.205+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.205490+0000) 2020-02-06T00:36:22.207 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:22.205+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.206767+0000) 2020-02-06T00:36:22.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:22.206+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.207084+0000) 2020-02-06T00:36:22.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:22.207+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.207645+0000) 2020-02-06T00:36:22.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:22.206+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.207936+0000) 2020-02-06T00:36:22.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:22.207+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.208536+0000) 2020-02-06T00:36:22.210 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:22.207+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.208940+0000) 2020-02-06T00:36:22.210 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:22.208+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.209351+0000) 2020-02-06T00:36:22.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:22.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.209677+0000) 2020-02-06T00:36:22.211 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:22.209+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.210245+0000) 2020-02-06T00:36:22.212 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:22.209+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.210269+0000) 2020-02-06T00:36:22.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:22.210+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.210644+0000) 2020-02-06T00:36:22.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:22.210+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.210804+0000) 2020-02-06T00:36:22.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:22.213+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.214986+0000) 2020-02-06T00:36:22.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:22.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178853 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:22.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:22.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:22.889494+0000) 2020-02-06T00:36:23.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:23.528+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:23.528756+0000) 2020-02-06T00:36:24.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:24.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:24.038068+0000) 2020-02-06T00:36:24.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:24.037+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:24.038068+0000) 2020-02-06T00:36:26.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:26.026+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:26.026484+0000) 2020-02-06T00:36:26.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:26.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:26.373122+0000) 2020-02-06T00:36:27.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:27.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.014053+0000) 2020-02-06T00:36:27.234 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:27.233+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.233608+0000) 2020-02-06T00:36:27.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:27.233+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.233996+0000) 2020-02-06T00:36:27.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:27.234+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.234302+0000) 2020-02-06T00:36:27.236 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:27.234+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.234932+0000) 2020-02-06T00:36:27.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:27.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.236028+0000) 2020-02-06T00:36:27.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:27.236+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.236541+0000) 2020-02-06T00:36:27.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:27.236+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.237235+0000) 2020-02-06T00:36:27.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:27.237+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.238257+0000) 2020-02-06T00:36:27.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:27.238+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.238774+0000) 2020-02-06T00:36:27.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:27.237+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.237752+0000) 2020-02-06T00:36:27.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:27.238+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.238905+0000) 2020-02-06T00:36:27.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:27.239+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.239343+0000) 2020-02-06T00:36:27.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:27.239+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.239503+0000) 2020-02-06T00:36:27.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:27.241+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.241900+0000) 2020-02-06T00:36:27.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:27.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:27.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:27.278+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.279199+0000) 2020-02-06T00:36:27.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:27.278+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.279236+0000) 2020-02-06T00:36:27.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:27.278+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:27.279285+0000) 2020-02-06T00:36:29.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:29.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:29.658139+0000) 2020-02-06T00:36:29.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:29.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:29.906969+0000) 2020-02-06T00:36:30.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:30.507+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:30.507961+0000) 2020-02-06T00:36:30.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:30.766+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:30.766552+0000) 2020-02-06T00:36:32.257 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:32.255+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.256619+0000) 2020-02-06T00:36:32.258 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:32.256+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.257202+0000) 2020-02-06T00:36:32.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:32.256+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.257846+0000) 2020-02-06T00:36:32.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:32.257+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.258346+0000) 2020-02-06T00:36:32.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:32.258+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.259088+0000) 2020-02-06T00:36:32.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:32.259+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.259561+0000) 2020-02-06T00:36:32.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:32.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.260229+0000) 2020-02-06T00:36:32.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:32.260+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.260491+0000) 2020-02-06T00:36:32.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:32.259+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.260896+0000) 2020-02-06T00:36:32.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:32.261+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.262111+0000) 2020-02-06T00:36:32.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:32.262+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.262506+0000) 2020-02-06T00:36:32.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:32.262+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.262985+0000) 2020-02-06T00:36:32.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:32.262+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.263330+0000) 2020-02-06T00:36:32.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:32.265+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.266492+0000) 2020-02-06T00:36:32.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:32.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178934 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:32.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:32.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:32.889670+0000) 2020-02-06T00:36:33.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:33.536+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:33.537073+0000) 2020-02-06T00:36:34.039 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:34.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:34.038359+0000) 2020-02-06T00:36:34.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:34.037+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:34.038363+0000) 2020-02-06T00:36:36.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:36.026+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:36.026748+0000) 2020-02-06T00:36:36.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:36.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:36.373422+0000) 2020-02-06T00:36:37.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:37.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.014314+0000) 2020-02-06T00:36:37.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:37.278+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.279396+0000) 2020-02-06T00:36:37.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:37.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.280840+0000) 2020-02-06T00:36:37.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:37.279+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.279836+0000) 2020-02-06T00:36:37.286 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:37.283+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.284850+0000) 2020-02-06T00:36:37.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:37.284+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.285252+0000) 2020-02-06T00:36:37.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:37.285+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.285812+0000) 2020-02-06T00:36:37.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:37.284+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.285917+0000) 2020-02-06T00:36:37.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:37.285+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.286379+0000) 2020-02-06T00:36:37.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:37.287+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.287725+0000) 2020-02-06T00:36:37.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:37.287+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.287883+0000) 2020-02-06T00:36:37.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:37.287+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.288397+0000) 2020-02-06T00:36:37.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:37.288+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.289059+0000) 2020-02-06T00:36:37.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:37.288+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.289291+0000) 2020-02-06T00:36:37.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:37.288+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.289634+0000) 2020-02-06T00:36:37.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:37.288+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.289796+0000) 2020-02-06T00:36:37.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:37.289+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.289729+0000) 2020-02-06T00:36:37.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:37.293+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:37.294236+0000) 2020-02-06T00:36:37.304 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:37.303+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 178975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:39.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:39.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:39.658376+0000) 2020-02-06T00:36:39.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:39.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:39.907217+0000) 2020-02-06T00:36:40.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:40.508+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:40.508194+0000) 2020-02-06T00:36:40.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:40.765+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:40.766796+0000) 2020-02-06T00:36:42.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:42.307+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.308205+0000) 2020-02-06T00:36:42.310 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:42.307+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.309242+0000) 2020-02-06T00:36:42.311 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:42.309+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.310245+0000) 2020-02-06T00:36:42.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:42.309+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.310893+0000) 2020-02-06T00:36:42.312 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:42.310+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.311318+0000) 2020-02-06T00:36:42.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:42.310+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.311716+0000) 2020-02-06T00:36:42.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:42.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.312298+0000) 2020-02-06T00:36:42.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:42.311+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.312409+0000) 2020-02-06T00:36:42.314 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:42.311+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.313006+0000) 2020-02-06T00:36:42.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:42.312+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.313582+0000) 2020-02-06T00:36:42.315 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:42.312+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.313320+0000) 2020-02-06T00:36:42.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:42.312+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.314277+0000) 2020-02-06T00:36:42.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:42.312+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.314204+0000) 2020-02-06T00:36:42.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:42.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.318568+0000) 2020-02-06T00:36:42.329 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:42.327+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179016 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:42.660 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:36:42.667 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:36:42.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:42.888+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:42.889915+0000) 2020-02-06T00:36:43.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:43.547+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:43.548404+0000) 2020-02-06T00:36:44.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:44.037+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:44.038610+0000) 2020-02-06T00:36:44.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:44.037+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:44.038605+0000) 2020-02-06T00:36:46.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:46.025+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:46.027006+0000) 2020-02-06T00:36:46.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:46.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:46.373682+0000) 2020-02-06T00:36:47.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:47.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.014565+0000) 2020-02-06T00:36:47.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:47.278+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.279701+0000) 2020-02-06T00:36:47.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:47.278+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.280071+0000) 2020-02-06T00:36:47.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:47.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.281064+0000) 2020-02-06T00:36:47.335 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:47.333+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.334683+0000) 2020-02-06T00:36:47.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:47.333+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.335203+0000) 2020-02-06T00:36:47.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:47.334+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.335462+0000) 2020-02-06T00:36:47.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:47.334+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.336011+0000) 2020-02-06T00:36:47.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:47.335+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.336516+0000) 2020-02-06T00:36:47.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:47.336+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.337517+0000) 2020-02-06T00:36:47.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:47.336+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.337998+0000) 2020-02-06T00:36:47.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:47.337+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.338662+0000) 2020-02-06T00:36:47.340 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:47.338+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.339284+0000) 2020-02-06T00:36:47.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:47.337+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.339078+0000) 2020-02-06T00:36:47.341 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:47.338+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.339854+0000) 2020-02-06T00:36:47.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:47.338+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.340328+0000) 2020-02-06T00:36:47.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:47.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.343196+0000) 2020-02-06T00:36:47.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:47.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:47.343434+0000) 2020-02-06T00:36:47.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:47.353+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179056 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:49.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:49.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:49.658617+0000) 2020-02-06T00:36:49.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:49.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:49.907411+0000) 2020-02-06T00:36:50.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:50.507+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:50.508464+0000) 2020-02-06T00:36:50.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:50.766+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:50.767052+0000) 2020-02-06T00:36:52.362 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:52.360+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.361219+0000) 2020-02-06T00:36:52.363 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:52.360+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.361650+0000) 2020-02-06T00:36:52.363 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:52.360+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.362259+0000) 2020-02-06T00:36:52.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:52.361+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.362903+0000) 2020-02-06T00:36:52.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:52.361+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.363164+0000) 2020-02-06T00:36:52.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:52.362+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.363471+0000) 2020-02-06T00:36:52.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:52.362+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.363958+0000) 2020-02-06T00:36:52.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:52.362+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.364006+0000) 2020-02-06T00:36:52.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:52.362+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.364155+0000) 2020-02-06T00:36:52.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:52.363+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.364668+0000) 2020-02-06T00:36:52.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:52.364+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.365280+0000) 2020-02-06T00:36:52.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:52.364+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.365579+0000) 2020-02-06T00:36:52.367 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:52.364+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.365807+0000) 2020-02-06T00:36:52.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:52.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.372882+0000) 2020-02-06T00:36:52.381 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:52.379+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179096 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:52.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:52.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:52.890197+0000) 2020-02-06T00:36:53.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:53.559+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:53.560275+0000) 2020-02-06T00:36:54.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:54.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:54.038840+0000) 2020-02-06T00:36:54.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:54.038+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:54.038877+0000) 2020-02-06T00:36:56.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:56.026+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:56.027257+0000) 2020-02-06T00:36:56.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:56.372+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:56.373938+0000) 2020-02-06T00:36:57.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:57.013+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.014791+0000) 2020-02-06T00:36:57.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:57.279+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.279944+0000) 2020-02-06T00:36:57.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:57.279+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.280273+0000) 2020-02-06T00:36:57.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:57.279+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.281298+0000) 2020-02-06T00:36:57.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:36:57.385+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.386281+0000) 2020-02-06T00:36:57.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:36:57.385+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.386748+0000) 2020-02-06T00:36:57.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:36:57.386+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.387105+0000) 2020-02-06T00:36:57.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:36:57.386+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.388221+0000) 2020-02-06T00:36:57.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:36:57.387+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.388630+0000) 2020-02-06T00:36:57.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:57.388+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.389136+0000) 2020-02-06T00:36:57.390 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:36:57.388+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.389724+0000) 2020-02-06T00:36:57.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:36:57.389+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.390801+0000) 2020-02-06T00:36:57.393 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:36:57.390+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.391671+0000) 2020-02-06T00:36:57.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:36:57.389+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.391354+0000) 2020-02-06T00:36:57.393 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:36:57.391+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.392359+0000) 2020-02-06T00:36:57.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:57.391+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.392404+0000) 2020-02-06T00:36:57.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:36:57.390+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.392199+0000) 2020-02-06T00:36:57.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:57.395+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:57.396456+0000) 2020-02-06T00:36:57.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:36:57.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179137 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:36:59.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:59.657+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:59.658880+0000) 2020-02-06T00:36:59.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:36:59.906+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:36:59.907639+0000) 2020-02-06T00:37:00.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:00.507+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:00.508730+0000) 2020-02-06T00:37:00.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:00.766+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:00.767366+0000) 2020-02-06T00:37:02.414 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:02.412+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.413483+0000) 2020-02-06T00:37:02.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:02.411+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.413175+0000) 2020-02-06T00:37:02.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:02.413+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.414297+0000) 2020-02-06T00:37:02.416 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:02.413+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.414854+0000) 2020-02-06T00:37:02.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:02.414+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.415137+0000) 2020-02-06T00:37:02.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:02.414+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.415435+0000) 2020-02-06T00:37:02.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:02.415+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.416765+0000) 2020-02-06T00:37:02.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:02.415+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.416587+0000) 2020-02-06T00:37:02.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:02.415+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.416690+0000) 2020-02-06T00:37:02.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:02.416+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.417679+0000) 2020-02-06T00:37:02.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:02.416+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.417985+0000) 2020-02-06T00:37:02.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:02.417+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.418807+0000) 2020-02-06T00:37:02.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:02.417+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.419224+0000) 2020-02-06T00:37:02.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:02.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.423310+0000) 2020-02-06T00:37:02.434 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:02.431+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179178 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:02.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:02.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:02.890421+0000) 2020-02-06T00:37:03.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:03.567+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:03.567658+0000) 2020-02-06T00:37:04.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:04.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:04.039073+0000) 2020-02-06T00:37:04.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:04.038+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:04.039135+0000) 2020-02-06T00:37:06.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:06.026+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:06.027501+0000) 2020-02-06T00:37:06.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:06.373+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:06.374159+0000) 2020-02-06T00:37:07.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:07.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.015017+0000) 2020-02-06T00:37:07.281 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:07.279+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.280194+0000) 2020-02-06T00:37:07.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:07.279+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.280554+0000) 2020-02-06T00:37:07.282 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:07.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.281517+0000) 2020-02-06T00:37:07.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:07.438+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.439378+0000) 2020-02-06T00:37:07.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:07.438+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.439807+0000) 2020-02-06T00:37:07.441 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:07.439+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.440755+0000) 2020-02-06T00:37:07.442 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:07.439+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.441004+0000) 2020-02-06T00:37:07.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:07.440+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.441799+0000) 2020-02-06T00:37:07.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:07.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.442677+0000) 2020-02-06T00:37:07.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:07.441+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.443013+0000) 2020-02-06T00:37:07.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:07.442+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.443275+0000) 2020-02-06T00:37:07.444 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:07.442+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.443806+0000) 2020-02-06T00:37:07.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:07.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.444063+0000) 2020-02-06T00:37:07.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:07.443+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.444683+0000) 2020-02-06T00:37:07.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:07.443+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.445012+0000) 2020-02-06T00:37:07.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:07.444+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.445173+0000) 2020-02-06T00:37:07.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:07.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:07.451919+0000) 2020-02-06T00:37:07.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:07.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:09.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:09.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:09.659151+0000) 2020-02-06T00:37:09.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:09.907+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:09.907870+0000) 2020-02-06T00:37:10.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:10.508+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:10.508982+0000) 2020-02-06T00:37:10.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:10.767+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:10.767630+0000) 2020-02-06T00:37:12.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:12.462+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.463334+0000) 2020-02-06T00:37:12.465 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:12.462+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.464101+0000) 2020-02-06T00:37:12.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:12.463+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.464989+0000) 2020-02-06T00:37:12.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:12.464+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.465602+0000) 2020-02-06T00:37:12.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:12.464+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.466048+0000) 2020-02-06T00:37:12.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:12.465+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.466999+0000) 2020-02-06T00:37:12.468 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:12.466+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.467448+0000) 2020-02-06T00:37:12.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:12.466+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.467989+0000) 2020-02-06T00:37:12.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:12.467+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.468641+0000) 2020-02-06T00:37:12.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:12.466+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.468356+0000) 2020-02-06T00:37:12.470 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:12.468+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.469119+0000) 2020-02-06T00:37:12.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:12.468+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.470147+0000) 2020-02-06T00:37:12.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:12.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.470578+0000) 2020-02-06T00:37:12.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:12.477+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.478630+0000) 2020-02-06T00:37:12.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:12.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179258 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:12.889 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:37:12.896 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:37:12.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:12.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:12.890695+0000) 2020-02-06T00:37:13.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:13.575+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:13.575764+0000) 2020-02-06T00:37:14.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:14.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:14.039359+0000) 2020-02-06T00:37:14.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:14.039+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:14.039362+0000) 2020-02-06T00:37:16.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:16.026+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:16.027737+0000) 2020-02-06T00:37:16.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:16.374+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:16.374386+0000) 2020-02-06T00:37:17.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:17.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.015283+0000) 2020-02-06T00:37:17.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:17.279+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.280464+0000) 2020-02-06T00:37:17.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:17.279+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.280794+0000) 2020-02-06T00:37:17.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:17.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.281702+0000) 2020-02-06T00:37:17.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:17.490+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.491712+0000) 2020-02-06T00:37:17.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:17.490+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.491919+0000) 2020-02-06T00:37:17.494 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:17.491+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.492367+0000) 2020-02-06T00:37:17.494 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:17.491+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.492966+0000) 2020-02-06T00:37:17.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:17.492+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.493332+0000) 2020-02-06T00:37:17.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:17.492+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494018+0000) 2020-02-06T00:37:17.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:17.493+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494186+0000) 2020-02-06T00:37:17.496 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:17.492+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494157+0000) 2020-02-06T00:37:17.496 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:17.493+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494972+0000) 2020-02-06T00:37:17.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:17.493+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494806+0000) 2020-02-06T00:37:17.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:17.493+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.494658+0000) 2020-02-06T00:37:17.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:17.494+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.495794+0000) 2020-02-06T00:37:17.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:17.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.496279+0000) 2020-02-06T00:37:17.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:17.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:17.500634+0000) 2020-02-06T00:37:17.510 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:17.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:19.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:19.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:19.659382+0000) 2020-02-06T00:37:19.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:19.907+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:19.908127+0000) 2020-02-06T00:37:20.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:20.508+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:20.509213+0000) 2020-02-06T00:37:20.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:20.767+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:20.767879+0000) 2020-02-06T00:37:22.515 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:22.514+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.514379+0000) 2020-02-06T00:37:22.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:22.515+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.515211+0000) 2020-02-06T00:37:22.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:22.516+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.516194+0000) 2020-02-06T00:37:22.517 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:22.516+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.516497+0000) 2020-02-06T00:37:22.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:22.516+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.516845+0000) 2020-02-06T00:37:22.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:22.516+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.517594+0000) 2020-02-06T00:37:22.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:22.516+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.517885+0000) 2020-02-06T00:37:22.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:22.518+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.518853+0000) 2020-02-06T00:37:22.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:22.519+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.519357+0000) 2020-02-06T00:37:22.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:22.517+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.518825+0000) 2020-02-06T00:37:22.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:22.518+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.519549+0000) 2020-02-06T00:37:22.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:22.518+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.520166+0000) 2020-02-06T00:37:22.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:22.519+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.521084+0000) 2020-02-06T00:37:22.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:22.523+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.523418+0000) 2020-02-06T00:37:22.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:22.531+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:22.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:22.889+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:22.890908+0000) 2020-02-06T00:37:23.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:23.583+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:23.583895+0000) 2020-02-06T00:37:24.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:24.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:24.039633+0000) 2020-02-06T00:37:24.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:24.039+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:24.039633+0000) 2020-02-06T00:37:26.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:26.027+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:26.028020+0000) 2020-02-06T00:37:26.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:26.373+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:26.374625+0000) 2020-02-06T00:37:27.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:27.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.015522+0000) 2020-02-06T00:37:27.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:27.280+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.280712+0000) 2020-02-06T00:37:27.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:27.280+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.281027+0000) 2020-02-06T00:37:27.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:27.280+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.281939+0000) 2020-02-06T00:37:27.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:27.538+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.539603+0000) 2020-02-06T00:37:27.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:27.539+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.540243+0000) 2020-02-06T00:37:27.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:27.539+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.540864+0000) 2020-02-06T00:37:27.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:27.540+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.541460+0000) 2020-02-06T00:37:27.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:27.541+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.542214+0000) 2020-02-06T00:37:27.544 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:27.541+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.543015+0000) 2020-02-06T00:37:27.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:27.542+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.543582+0000) 2020-02-06T00:37:27.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:27.542+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.543694+0000) 2020-02-06T00:37:27.545 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:27.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.543984+0000) 2020-02-06T00:37:27.546 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:27.542+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.544294+0000) 2020-02-06T00:37:27.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:27.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.545019+0000) 2020-02-06T00:37:27.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:27.544+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.545421+0000) 2020-02-06T00:37:27.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:27.543+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.544768+0000) 2020-02-06T00:37:27.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:27.547+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:27.548088+0000) 2020-02-06T00:37:27.558 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:27.555+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:29.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:29.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:29.659608+0000) 2020-02-06T00:37:29.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:29.907+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:29.908359+0000) 2020-02-06T00:37:30.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:30.508+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:30.509469+0000) 2020-02-06T00:37:30.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:30.767+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:30.768154+0000) 2020-02-06T00:37:32.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:32.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.564427+0000) 2020-02-06T00:37:32.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:32.563+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.565132+0000) 2020-02-06T00:37:32.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:32.564+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.565518+0000) 2020-02-06T00:37:32.567 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:32.564+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.566196+0000) 2020-02-06T00:37:32.567 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:32.564+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.565999+0000) 2020-02-06T00:37:32.567 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:32.565+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.566441+0000) 2020-02-06T00:37:32.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:32.565+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.567221+0000) 2020-02-06T00:37:32.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:32.566+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.567900+0000) 2020-02-06T00:37:32.569 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:32.567+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.568351+0000) 2020-02-06T00:37:32.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:32.568+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.569606+0000) 2020-02-06T00:37:32.570 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:32.568+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.569496+0000) 2020-02-06T00:37:32.571 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:32.568+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.570004+0000) 2020-02-06T00:37:32.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:32.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.570157+0000) 2020-02-06T00:37:32.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:32.572+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.573127+0000) 2020-02-06T00:37:32.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:32.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:32.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:32.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:32.891176+0000) 2020-02-06T00:37:33.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:33.590+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:33.591790+0000) 2020-02-06T00:37:34.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:34.038+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:34.039907+0000) 2020-02-06T00:37:34.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:34.038+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:34.039896+0000) 2020-02-06T00:37:36.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:36.027+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:36.028305+0000) 2020-02-06T00:37:36.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:36.373+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:36.374823+0000) 2020-02-06T00:37:37.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:37.014+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.015758+0000) 2020-02-06T00:37:37.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:37.280+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.280956+0000) 2020-02-06T00:37:37.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:37.280+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.281275+0000) 2020-02-06T00:37:37.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:37.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.282130+0000) 2020-02-06T00:37:37.600 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:37.589+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.590898+0000) 2020-02-06T00:37:37.600 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:37.590+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.591390+0000) 2020-02-06T00:37:37.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:37.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.592283+0000) 2020-02-06T00:37:37.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:37.595+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.596243+0000) 2020-02-06T00:37:37.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:37.591+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.592618+0000) 2020-02-06T00:37:37.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:37.593+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.594412+0000) 2020-02-06T00:37:37.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:37.594+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.595665+0000) 2020-02-06T00:37:37.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:37.590+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.590597+0000) 2020-02-06T00:37:37.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:37.590+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.590543+0000) 2020-02-06T00:37:37.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:37.591+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.591715+0000) 2020-02-06T00:37:37.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:37.592+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.593337+0000) 2020-02-06T00:37:37.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:37.593+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.593969+0000) 2020-02-06T00:37:37.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:37.594+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.595398+0000) 2020-02-06T00:37:37.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:37.599+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:37.600424+0000) 2020-02-06T00:37:37.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:37.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:39.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:39.658+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:39.659862+0000) 2020-02-06T00:37:39.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:39.907+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:39.908603+0000) 2020-02-06T00:37:40.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:40.509+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:40.509757+0000) 2020-02-06T00:37:40.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:40.767+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:40.768361+0000) 2020-02-06T00:37:42.617 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:42.615+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.615557+0000) 2020-02-06T00:37:42.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:42.615+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.616318+0000) 2020-02-06T00:37:42.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:42.616+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.616797+0000) 2020-02-06T00:37:42.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:42.616+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.617309+0000) 2020-02-06T00:37:42.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:42.616+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.617989+0000) 2020-02-06T00:37:42.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:42.617+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.618692+0000) 2020-02-06T00:37:42.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:42.617+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.619047+0000) 2020-02-06T00:37:42.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:42.618+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.619299+0000) 2020-02-06T00:37:42.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:42.619+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.620641+0000) 2020-02-06T00:37:42.624 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:42.620+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.621144+0000) 2020-02-06T00:37:42.625 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:42.617+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.618022+0000) 2020-02-06T00:37:42.625 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:42.620+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.620662+0000) 2020-02-06T00:37:42.625 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:42.620+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.620717+0000) 2020-02-06T00:37:42.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:42.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.626631+0000) 2020-02-06T00:37:42.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:42.633+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:42.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:42.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:42.891393+0000) 2020-02-06T00:37:43.079 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:37:43.085 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:37:43.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:43.602+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:43.603705+0000) 2020-02-06T00:37:44.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:44.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:44.040144+0000) 2020-02-06T00:37:44.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:44.039+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:44.040136+0000) 2020-02-06T00:37:46.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:46.028+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:46.028575+0000) 2020-02-06T00:37:46.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:46.373+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:46.375049+0000) 2020-02-06T00:37:47.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:47.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.015980+0000) 2020-02-06T00:37:47.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:47.280+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.281214+0000) 2020-02-06T00:37:47.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:47.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.281512+0000) 2020-02-06T00:37:47.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:47.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.282315+0000) 2020-02-06T00:37:47.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:47.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.639454+0000) 2020-02-06T00:37:47.641 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:47.638+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.639879+0000) 2020-02-06T00:37:47.642 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:47.641+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.641511+0000) 2020-02-06T00:37:47.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:47.641+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.642192+0000) 2020-02-06T00:37:47.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:47.642+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.642496+0000) 2020-02-06T00:37:47.644 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:47.642+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.643154+0000) 2020-02-06T00:37:47.645 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:47.642+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.643844+0000) 2020-02-06T00:37:47.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:47.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.644501+0000) 2020-02-06T00:37:47.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:47.643+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.644715+0000) 2020-02-06T00:37:47.647 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:47.644+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.645785+0000) 2020-02-06T00:37:47.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:47.646+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.646478+0000) 2020-02-06T00:37:47.648 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:47.646+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.646630+0000) 2020-02-06T00:37:47.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:47.646+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.646119+0000) 2020-02-06T00:37:47.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:47.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:47.651258+0000) 2020-02-06T00:37:47.661 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:47.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:49.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:49.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:49.660131+0000) 2020-02-06T00:37:49.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:49.908+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:49.908838+0000) 2020-02-06T00:37:50.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:50.509+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:50.510005+0000) 2020-02-06T00:37:50.770 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:50.767+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:50.768592+0000) 2020-02-06T00:37:52.668 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:52.666+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.666467+0000) 2020-02-06T00:37:52.668 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:52.666+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.666983+0000) 2020-02-06T00:37:52.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:52.666+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.667900+0000) 2020-02-06T00:37:52.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:52.667+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.668068+0000) 2020-02-06T00:37:52.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:52.667+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.668317+0000) 2020-02-06T00:37:52.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:52.667+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.668797+0000) 2020-02-06T00:37:52.670 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:52.668+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.669294+0000) 2020-02-06T00:37:52.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:52.668+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.668652+0000) 2020-02-06T00:37:52.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:52.668+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.669594+0000) 2020-02-06T00:37:52.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:52.668+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.670059+0000) 2020-02-06T00:37:52.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:52.669+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.670188+0000) 2020-02-06T00:37:52.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:52.670+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.670888+0000) 2020-02-06T00:37:52.672 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:52.671+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.671417+0000) 2020-02-06T00:37:52.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:52.675+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.676525+0000) 2020-02-06T00:37:52.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:52.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179580 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:52.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:52.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:52.891653+0000) 2020-02-06T00:37:53.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:53.611+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:53.612063+0000) 2020-02-06T00:37:54.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:54.039+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:54.040377+0000) 2020-02-06T00:37:54.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:54.039+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:54.040402+0000) 2020-02-06T00:37:56.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:56.028+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:56.028821+0000) 2020-02-06T00:37:56.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:56.374+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:56.375297+0000) 2020-02-06T00:37:57.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:57.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.016203+0000) 2020-02-06T00:37:57.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:57.281+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.281527+0000) 2020-02-06T00:37:57.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:57.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.281755+0000) 2020-02-06T00:37:57.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:57.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.282590+0000) 2020-02-06T00:37:57.691 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:37:57.688+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.689841+0000) 2020-02-06T00:37:57.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:37:57.689+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.690545+0000) 2020-02-06T00:37:57.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:57.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.691554+0000) 2020-02-06T00:37:57.693 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:37:57.691+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.692381+0000) 2020-02-06T00:37:57.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:37:57.691+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.692033+0000) 2020-02-06T00:37:57.694 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:37:57.692+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.692887+0000) 2020-02-06T00:37:57.694 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:37:57.692+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.693091+0000) 2020-02-06T00:37:57.695 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:37:57.692+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.693328+0000) 2020-02-06T00:37:57.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:57.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.693883+0000) 2020-02-06T00:37:57.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:37:57.693+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.693499+0000) 2020-02-06T00:37:57.696 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:37:57.693+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.694331+0000) 2020-02-06T00:37:57.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:37:57.694+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.694948+0000) 2020-02-06T00:37:57.696 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:37:57.694+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.695132+0000) 2020-02-06T00:37:57.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:57.699+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:57.700857+0000) 2020-02-06T00:37:57.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:37:57.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:37:59.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:59.659+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:59.660330+0000) 2020-02-06T00:37:59.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:37:59.908+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:37:59.909103+0000) 2020-02-06T00:38:00.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:00.510+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:00.510260+0000) 2020-02-06T00:38:00.770 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:00.768+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:00.768858+0000) 2020-02-06T00:38:02.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:02.712+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.713231+0000) 2020-02-06T00:38:02.715 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:02.713+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.714143+0000) 2020-02-06T00:38:02.718 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:02.716+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.717036+0000) 2020-02-06T00:38:02.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:02.715+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.716442+0000) 2020-02-06T00:38:02.719 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:02.716+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.717387+0000) 2020-02-06T00:38:02.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:02.716+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.717896+0000) 2020-02-06T00:38:02.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:02.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.718400+0000) 2020-02-06T00:38:02.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:02.718+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.718750+0000) 2020-02-06T00:38:02.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:02.718+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.719298+0000) 2020-02-06T00:38:02.720 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:02.719+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.719573+0000) 2020-02-06T00:38:02.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:02.719+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.720247+0000) 2020-02-06T00:38:02.721 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:02.719+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.720285+0000) 2020-02-06T00:38:02.723 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:02.720+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.720646+0000) 2020-02-06T00:38:02.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:02.727+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.728114+0000) 2020-02-06T00:38:02.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:02.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179662 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:02.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:02.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:02.891870+0000) 2020-02-06T00:38:03.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:03.618+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:03.618602+0000) 2020-02-06T00:38:04.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:04.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:04.040600+0000) 2020-02-06T00:38:04.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:04.040+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:04.040677+0000) 2020-02-06T00:38:06.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:06.028+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:06.029064+0000) 2020-02-06T00:38:06.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:06.374+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:06.375551+0000) 2020-02-06T00:38:07.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:07.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.016463+0000) 2020-02-06T00:38:07.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:07.281+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.281834+0000) 2020-02-06T00:38:07.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:07.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.281987+0000) 2020-02-06T00:38:07.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:07.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.282767+0000) 2020-02-06T00:38:07.739 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:07.737+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.738212+0000) 2020-02-06T00:38:07.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:07.738+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.739161+0000) 2020-02-06T00:38:07.741 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:07.739+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.740193+0000) 2020-02-06T00:38:07.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:07.739+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.740803+0000) 2020-02-06T00:38:07.742 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:07.740+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.741389+0000) 2020-02-06T00:38:07.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:07.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.741478+0000) 2020-02-06T00:38:07.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:07.741+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.742247+0000) 2020-02-06T00:38:07.744 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:07.742+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.742956+0000) 2020-02-06T00:38:07.744 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:07.742+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.743192+0000) 2020-02-06T00:38:07.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:07.743+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.744087+0000) 2020-02-06T00:38:07.745 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:07.743+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.744387+0000) 2020-02-06T00:38:07.746 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:07.744+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.744888+0000) 2020-02-06T00:38:07.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:07.745+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.745607+0000) 2020-02-06T00:38:07.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:07.746+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:07.747679+0000) 2020-02-06T00:38:07.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:07.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179703 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:09.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:09.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:09.660565+0000) 2020-02-06T00:38:09.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:09.909+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:09.909287+0000) 2020-02-06T00:38:10.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:10.510+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:10.510507+0000) 2020-02-06T00:38:10.770 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:10.768+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:10.769124+0000) 2020-02-06T00:38:12.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:12.763+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.764196+0000) 2020-02-06T00:38:12.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:12.763+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.764779+0000) 2020-02-06T00:38:12.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:12.764+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.765248+0000) 2020-02-06T00:38:12.768 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:12.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.766924+0000) 2020-02-06T00:38:12.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:12.766+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.767673+0000) 2020-02-06T00:38:12.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:12.767+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.768382+0000) 2020-02-06T00:38:12.769 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:12.767+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.768479+0000) 2020-02-06T00:38:12.770 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:12.768+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.768731+0000) 2020-02-06T00:38:12.770 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:12.768+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.769276+0000) 2020-02-06T00:38:12.770 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:12.768+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.769912+0000) 2020-02-06T00:38:12.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:12.770+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.770524+0000) 2020-02-06T00:38:12.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:12.771+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.772254+0000) 2020-02-06T00:38:12.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:12.771+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.772476+0000) 2020-02-06T00:38:12.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:12.772+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.773483+0000) 2020-02-06T00:38:12.784 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:12.783+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179742 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:12.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:12.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:12.892083+0000) 2020-02-06T00:38:13.258 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:38:13.264 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:38:13.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:13.627+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:13.628135+0000) 2020-02-06T00:38:14.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:14.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:14.040884+0000) 2020-02-06T00:38:14.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:14.040+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:14.040916+0000) 2020-02-06T00:38:16.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:16.029+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:16.029329+0000) 2020-02-06T00:38:16.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:16.375+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:16.375782+0000) 2020-02-06T00:38:17.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:17.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.016694+0000) 2020-02-06T00:38:17.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:17.281+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.282072+0000) 2020-02-06T00:38:17.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:17.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.282233+0000) 2020-02-06T00:38:17.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:17.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.282957+0000) 2020-02-06T00:38:17.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:17.789+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.789465+0000) 2020-02-06T00:38:17.791 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:17.790+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.790125+0000) 2020-02-06T00:38:17.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:17.790+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.790775+0000) 2020-02-06T00:38:17.792 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:17.790+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.791066+0000) 2020-02-06T00:38:17.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:17.792+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.793040+0000) 2020-02-06T00:38:17.794 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:17.792+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.793020+0000) 2020-02-06T00:38:17.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:17.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.793612+0000) 2020-02-06T00:38:17.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:17.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.794151+0000) 2020-02-06T00:38:17.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:17.794+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.794667+0000) 2020-02-06T00:38:17.796 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:17.794+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.795053+0000) 2020-02-06T00:38:17.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:17.794+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.794935+0000) 2020-02-06T00:38:17.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:17.795+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.795348+0000) 2020-02-06T00:38:17.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:17.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.795972+0000) 2020-02-06T00:38:17.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:17.797+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:17.797634+0000) 2020-02-06T00:38:17.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:17.808+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179784 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:19.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:19.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:19.660799+0000) 2020-02-06T00:38:19.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:19.908+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:19.909526+0000) 2020-02-06T00:38:20.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:20.509+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:20.510758+0000) 2020-02-06T00:38:20.770 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:20.768+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:20.769359+0000) 2020-02-06T00:38:22.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:22.812+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.813342+0000) 2020-02-06T00:38:22.815 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:22.812+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.814363+0000) 2020-02-06T00:38:22.816 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:22.813+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.814554+0000) 2020-02-06T00:38:22.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:22.814+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.816157+0000) 2020-02-06T00:38:22.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:22.815+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.816671+0000) 2020-02-06T00:38:22.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:22.815+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.817097+0000) 2020-02-06T00:38:22.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:22.815+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.816966+0000) 2020-02-06T00:38:22.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:22.816+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.817688+0000) 2020-02-06T00:38:22.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:22.817+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.818152+0000) 2020-02-06T00:38:22.820 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:22.816+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.817682+0000) 2020-02-06T00:38:22.821 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:22.816+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.818361+0000) 2020-02-06T00:38:22.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:22.817+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.818704+0000) 2020-02-06T00:38:22.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:22.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.822269+0000) 2020-02-06T00:38:22.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:22.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.822477+0000) 2020-02-06T00:38:22.835 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:22.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179824 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:22.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:22.890+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:22.892272+0000) 2020-02-06T00:38:23.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:23.636+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:23.637918+0000) 2020-02-06T00:38:24.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:24.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:24.041114+0000) 2020-02-06T00:38:24.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:24.040+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:24.041221+0000) 2020-02-06T00:38:26.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:26.028+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:26.029572+0000) 2020-02-06T00:38:26.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:26.375+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:26.375984+0000) 2020-02-06T00:38:27.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:27.015+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.016926+0000) 2020-02-06T00:38:27.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:27.281+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.282309+0000) 2020-02-06T00:38:27.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:27.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.282505+0000) 2020-02-06T00:38:27.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:27.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.283137+0000) 2020-02-06T00:38:27.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:27.837+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.838205+0000) 2020-02-06T00:38:27.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:27.838+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.839381+0000) 2020-02-06T00:38:27.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:27.838+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.840160+0000) 2020-02-06T00:38:27.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:27.839+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.840644+0000) 2020-02-06T00:38:27.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:27.840+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.841772+0000) 2020-02-06T00:38:27.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:27.840+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.842303+0000) 2020-02-06T00:38:27.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:27.841+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.842641+0000) 2020-02-06T00:38:27.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:27.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.843252+0000) 2020-02-06T00:38:27.845 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:27.842+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.844034+0000) 2020-02-06T00:38:27.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:27.843+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.844455+0000) 2020-02-06T00:38:27.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:27.844+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.845093+0000) 2020-02-06T00:38:27.847 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:27.843+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.845262+0000) 2020-02-06T00:38:27.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:27.844+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.845667+0000) 2020-02-06T00:38:27.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:27.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:27.849525+0000) 2020-02-06T00:38:27.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:27.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:29.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:29.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:29.661029+0000) 2020-02-06T00:38:29.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:29.908+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:29.909783+0000) 2020-02-06T00:38:30.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:30.510+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:30.511019+0000) 2020-02-06T00:38:30.770 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:30.768+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:30.769602+0000) 2020-02-06T00:38:32.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:32.865+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.866107+0000) 2020-02-06T00:38:32.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:32.866+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.867557+0000) 2020-02-06T00:38:32.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:32.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.868566+0000) 2020-02-06T00:38:32.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:32.867+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.868805+0000) 2020-02-06T00:38:32.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:32.868+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.869353+0000) 2020-02-06T00:38:32.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:32.868+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.869395+0000) 2020-02-06T00:38:32.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:32.868+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.870110+0000) 2020-02-06T00:38:32.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:32.869+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.870636+0000) 2020-02-06T00:38:32.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:32.869+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.870889+0000) 2020-02-06T00:38:32.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:32.870+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.871438+0000) 2020-02-06T00:38:32.873 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:32.870+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.872386+0000) 2020-02-06T00:38:32.874 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:32.871+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.873056+0000) 2020-02-06T00:38:32.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:32.871+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.873144+0000) 2020-02-06T00:38:32.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:32.876+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.877555+0000) 2020-02-06T00:38:32.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:32.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179903 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:32.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:32.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:32.892486+0000) 2020-02-06T00:38:33.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:33.643+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:33.644865+0000) 2020-02-06T00:38:34.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:34.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:34.041335+0000) 2020-02-06T00:38:34.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:34.040+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:34.041430+0000) 2020-02-06T00:38:36.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:36.028+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:36.029770+0000) 2020-02-06T00:38:36.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:36.375+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:36.376212+0000) 2020-02-06T00:38:37.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:37.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.017153+0000) 2020-02-06T00:38:37.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:37.281+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.282527+0000) 2020-02-06T00:38:37.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:37.281+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.282733+0000) 2020-02-06T00:38:37.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:37.281+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.283331+0000) 2020-02-06T00:38:37.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:37.891+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.892550+0000) 2020-02-06T00:38:37.894 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:37.891+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.892469+0000) 2020-02-06T00:38:37.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:37.893+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.894804+0000) 2020-02-06T00:38:37.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:37.894+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.895252+0000) 2020-02-06T00:38:37.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:37.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.895766+0000) 2020-02-06T00:38:37.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:37.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.896618+0000) 2020-02-06T00:38:37.899 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:37.896+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.897550+0000) 2020-02-06T00:38:37.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:37.896+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.898040+0000) 2020-02-06T00:38:37.900 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:37.897+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.898129+0000) 2020-02-06T00:38:37.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:37.896+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.898213+0000) 2020-02-06T00:38:37.900 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:37.896+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.898157+0000) 2020-02-06T00:38:37.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:37.897+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.898753+0000) 2020-02-06T00:38:37.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:37.897+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.899134+0000) 2020-02-06T00:38:37.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:37.900+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:37.901675+0000) 2020-02-06T00:38:37.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:37.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:39.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:39.660+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:39.661302+0000) 2020-02-06T00:38:39.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:39.908+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:39.909989+0000) 2020-02-06T00:38:40.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:40.509+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:40.511261+0000) 2020-02-06T00:38:40.773 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:40.769+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:40.769828+0000) 2020-02-06T00:38:42.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:42.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.892696+0000) 2020-02-06T00:38:42.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:42.917+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.918237+0000) 2020-02-06T00:38:42.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:42.917+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.918651+0000) 2020-02-06T00:38:42.920 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:42.917+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.919048+0000) 2020-02-06T00:38:42.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:42.918+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.919593+0000) 2020-02-06T00:38:42.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:42.920+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.921463+0000) 2020-02-06T00:38:42.923 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:42.919+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.921146+0000) 2020-02-06T00:38:42.923 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:42.919+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.921310+0000) 2020-02-06T00:38:42.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:42.921+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.922276+0000) 2020-02-06T00:38:42.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:42.921+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.922802+0000) 2020-02-06T00:38:42.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:42.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.923236+0000) 2020-02-06T00:38:42.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:42.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.923632+0000) 2020-02-06T00:38:42.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:42.922+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.923875+0000) 2020-02-06T00:38:42.925 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:42.922+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.924333+0000) 2020-02-06T00:38:42.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:42.926+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:42.927740+0000) 2020-02-06T00:38:42.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:42.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 179985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:43.446 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:38:43.453 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:38:43.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:43.651+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:43.652607+0000) 2020-02-06T00:38:44.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:44.040+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:44.041585+0000) 2020-02-06T00:38:44.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:44.040+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:44.041683+0000) 2020-02-06T00:38:46.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:46.029+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:46.030037+0000) 2020-02-06T00:38:46.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:46.375+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:46.376396+0000) 2020-02-06T00:38:47.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:47.016+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.017331+0000) 2020-02-06T00:38:47.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:47.282+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.282770+0000) 2020-02-06T00:38:47.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:47.282+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.282978+0000) 2020-02-06T00:38:47.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:47.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.283496+0000) 2020-02-06T00:38:47.945 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:47.943+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.944415+0000) 2020-02-06T00:38:47.946 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:47.943+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.944773+0000) 2020-02-06T00:38:47.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:47.944+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.945288+0000) 2020-02-06T00:38:47.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:47.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.945484+0000) 2020-02-06T00:38:47.948 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:47.945+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.946766+0000) 2020-02-06T00:38:47.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:47.945+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.946604+0000) 2020-02-06T00:38:47.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:47.945+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.947022+0000) 2020-02-06T00:38:47.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:47.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.947436+0000) 2020-02-06T00:38:47.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:47.946+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.948067+0000) 2020-02-06T00:38:47.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:47.947+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.948351+0000) 2020-02-06T00:38:47.950 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:47.947+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.948723+0000) 2020-02-06T00:38:47.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:47.948+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.949416+0000) 2020-02-06T00:38:47.952 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:47.948+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.949781+0000) 2020-02-06T00:38:47.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:47.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:47.950984+0000) 2020-02-06T00:38:47.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:47.962+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180030 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:49.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:49.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:49.661540+0000) 2020-02-06T00:38:49.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:49.909+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:49.910244+0000) 2020-02-06T00:38:50.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:50.510+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:50.511510+0000) 2020-02-06T00:38:50.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:50.769+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:50.770067+0000) 2020-02-06T00:38:52.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:52.892+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.892925+0000) 2020-02-06T00:38:52.970 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:52.968+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.969500+0000) 2020-02-06T00:38:52.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:52.969+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.970683+0000) 2020-02-06T00:38:52.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:52.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.970910+0000) 2020-02-06T00:38:52.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:52.970+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.971290+0000) 2020-02-06T00:38:52.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:52.970+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.971923+0000) 2020-02-06T00:38:52.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:52.971+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.972324+0000) 2020-02-06T00:38:52.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:52.971+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.972895+0000) 2020-02-06T00:38:52.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:52.972+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.973439+0000) 2020-02-06T00:38:52.975 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:52.973+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.974823+0000) 2020-02-06T00:38:52.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:52.973+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.974738+0000) 2020-02-06T00:38:52.976 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:52.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.975334+0000) 2020-02-06T00:38:52.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:52.974+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.975884+0000) 2020-02-06T00:38:52.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:52.974+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.976356+0000) 2020-02-06T00:38:52.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:52.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:52.979380+0000) 2020-02-06T00:38:52.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:52.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:53.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:53.660+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:53.660862+0000) 2020-02-06T00:38:54.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:54.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:54.041821+0000) 2020-02-06T00:38:54.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:54.041+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:54.041919+0000) 2020-02-06T00:38:56.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:56.029+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:56.030279+0000) 2020-02-06T00:38:56.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:56.376+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:56.376660+0000) 2020-02-06T00:38:57.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:57.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.017563+0000) 2020-02-06T00:38:57.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:57.282+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.283005+0000) 2020-02-06T00:38:57.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:57.282+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.283253+0000) 2020-02-06T00:38:57.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:57.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.283708+0000) 2020-02-06T00:38:57.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:38:57.992+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.993787+0000) 2020-02-06T00:38:57.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:38:57.993+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.995001+0000) 2020-02-06T00:38:57.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:57.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.995752+0000) 2020-02-06T00:38:57.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:38:57.994+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.995948+0000) 2020-02-06T00:38:57.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:38:57.995+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.996478+0000) 2020-02-06T00:38:57.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:38:57.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.997692+0000) 2020-02-06T00:38:57.999 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:38:57.997+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.998269+0000) 2020-02-06T00:38:57.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:38:57.996+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.997857+0000) 2020-02-06T00:38:58.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:38:57.997+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.998415+0000) 2020-02-06T00:38:58.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:38:57.997+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.999326+0000) 2020-02-06T00:38:58.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:38:57.998+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.999543+0000) 2020-02-06T00:38:58.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:38:57.998+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:57.999960+0000) 2020-02-06T00:38:58.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:58.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:58.001317+0000) 2020-02-06T00:38:58.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:58.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:58.001628+0000) 2020-02-06T00:38:58.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:38:58.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:38:59.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:59.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:59.661827+0000) 2020-02-06T00:38:59.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:38:59.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:38:59.910487+0000) 2020-02-06T00:39:00.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:00.510+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:00.511697+0000) 2020-02-06T00:39:00.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:00.770+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:00.770379+0000) 2020-02-06T00:39:02.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:02.891+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:02.893172+0000) 2020-02-06T00:39:03.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:03.018+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.019525+0000) 2020-02-06T00:39:03.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:03.019+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.021072+0000) 2020-02-06T00:39:03.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:03.020+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.021403+0000) 2020-02-06T00:39:03.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:03.020+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.021926+0000) 2020-02-06T00:39:03.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:03.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.022488+0000) 2020-02-06T00:39:03.024 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:03.022+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.023381+0000) 2020-02-06T00:39:03.024 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:03.022+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.023882+0000) 2020-02-06T00:39:03.025 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:03.023+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.024928+0000) 2020-02-06T00:39:03.026 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:03.024+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.025466+0000) 2020-02-06T00:39:03.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:03.024+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.025420+0000) 2020-02-06T00:39:03.027 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:03.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.026043+0000) 2020-02-06T00:39:03.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:03.024+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.025857+0000) 2020-02-06T00:39:03.027 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:03.024+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.026277+0000) 2020-02-06T00:39:03.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:03.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.028693+0000) 2020-02-06T00:39:03.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:03.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180153 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:03.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:03.668+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:03.669130+0000) 2020-02-06T00:39:04.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:04.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:04.042055+0000) 2020-02-06T00:39:04.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:04.041+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:04.042190+0000) 2020-02-06T00:39:06.031 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:06.029+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:06.030541+0000) 2020-02-06T00:39:06.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:06.376+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:06.376923+0000) 2020-02-06T00:39:07.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:07.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:07.017805+0000) 2020-02-06T00:39:07.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:07.282+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:07.283250+0000) 2020-02-06T00:39:07.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:07.282+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:07.283475+0000) 2020-02-06T00:39:07.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:07.282+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:07.283927+0000) 2020-02-06T00:39:08.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:08.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.044950+0000) 2020-02-06T00:39:08.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:08.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.045569+0000) 2020-02-06T00:39:08.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:08.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.046058+0000) 2020-02-06T00:39:08.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:08.045+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.046422+0000) 2020-02-06T00:39:08.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:08.045+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.046605+0000) 2020-02-06T00:39:08.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:08.045+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.047054+0000) 2020-02-06T00:39:08.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:08.046+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.047439+0000) 2020-02-06T00:39:08.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:08.046+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.048226+0000) 2020-02-06T00:39:08.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:08.047+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.048960+0000) 2020-02-06T00:39:08.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:08.048+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.049716+0000) 2020-02-06T00:39:08.051 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:08.048+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.050148+0000) 2020-02-06T00:39:08.051 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:08.048+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.050415+0000) 2020-02-06T00:39:08.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:08.049+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.050478+0000) 2020-02-06T00:39:08.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:08.054+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:08.055351+0000) 2020-02-06T00:39:08.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:08.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180198 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:09.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:09.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:09.662104+0000) 2020-02-06T00:39:09.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:09.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:09.910718+0000) 2020-02-06T00:39:10.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:10.511+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:10.511954+0000) 2020-02-06T00:39:10.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:10.770+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:10.770616+0000) 2020-02-06T00:39:12.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:12.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:12.893469+0000) 2020-02-06T00:39:13.072 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:13.071+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.071336+0000) 2020-02-06T00:39:13.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:13.070+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.071168+0000) 2020-02-06T00:39:13.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:13.072+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.072539+0000) 2020-02-06T00:39:13.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:13.073+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.073579+0000) 2020-02-06T00:39:13.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:13.073+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.074385+0000) 2020-02-06T00:39:13.076 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:13.074+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.074736+0000) 2020-02-06T00:39:13.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:13.074+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.074933+0000) 2020-02-06T00:39:13.076 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:13.075+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.075746+0000) 2020-02-06T00:39:13.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:13.076+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.076144+0000) 2020-02-06T00:39:13.077 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:13.076+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.076785+0000) 2020-02-06T00:39:13.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:13.076+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.077284+0000) 2020-02-06T00:39:13.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:13.076+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.077386+0000) 2020-02-06T00:39:13.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:13.077+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.077218+0000) 2020-02-06T00:39:13.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:13.083+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.083671+0000) 2020-02-06T00:39:13.091 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:13.089+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:13.617 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:39:13.624 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:39:13.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:13.677+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:13.677178+0000) 2020-02-06T00:39:14.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:14.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:14.042379+0000) 2020-02-06T00:39:14.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:14.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:14.042426+0000) 2020-02-06T00:39:16.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:16.030+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:16.030797+0000) 2020-02-06T00:39:16.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:16.376+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:16.377188+0000) 2020-02-06T00:39:17.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:17.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:17.018041+0000) 2020-02-06T00:39:17.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:17.283+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:17.283549+0000) 2020-02-06T00:39:17.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:17.283+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:17.283723+0000) 2020-02-06T00:39:17.285 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:17.283+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:17.284144+0000) 2020-02-06T00:39:18.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:18.096+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.097765+0000) 2020-02-06T00:39:18.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:18.098+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.098810+0000) 2020-02-06T00:39:18.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:18.097+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.098810+0000) 2020-02-06T00:39:18.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:18.098+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.099214+0000) 2020-02-06T00:39:18.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:18.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.099639+0000) 2020-02-06T00:39:18.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:18.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.100254+0000) 2020-02-06T00:39:18.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:18.099+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.100118+0000) 2020-02-06T00:39:18.102 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:18.099+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.100397+0000) 2020-02-06T00:39:18.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:18.099+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.100817+0000) 2020-02-06T00:39:18.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:18.100+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.101368+0000) 2020-02-06T00:39:18.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:18.100+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.101136+0000) 2020-02-06T00:39:18.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:18.100+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.101318+0000) 2020-02-06T00:39:18.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:18.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.104836+0000) 2020-02-06T00:39:18.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:18.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:18.105114+0000) 2020-02-06T00:39:18.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:18.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180282 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:19.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:19.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:19.662329+0000) 2020-02-06T00:39:19.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:19.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:19.910978+0000) 2020-02-06T00:39:20.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:20.511+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:20.512207+0000) 2020-02-06T00:39:20.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:20.769+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:20.770851+0000) 2020-02-06T00:39:22.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:22.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:22.893794+0000) 2020-02-06T00:39:23.122 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:23.120+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.121441+0000) 2020-02-06T00:39:23.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:23.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.122225+0000) 2020-02-06T00:39:23.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:23.121+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.122812+0000) 2020-02-06T00:39:23.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:23.122+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.123440+0000) 2020-02-06T00:39:23.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:23.123+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.123888+0000) 2020-02-06T00:39:23.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:23.123+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.124109+0000) 2020-02-06T00:39:23.125 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:23.124+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.124804+0000) 2020-02-06T00:39:23.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:23.124+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.125199+0000) 2020-02-06T00:39:23.126 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:23.124+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.125622+0000) 2020-02-06T00:39:23.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:23.125+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.125742+0000) 2020-02-06T00:39:23.128 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:23.126+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.126741+0000) 2020-02-06T00:39:23.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:23.127+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.128135+0000) 2020-02-06T00:39:23.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:23.127+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.127976+0000) 2020-02-06T00:39:23.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:23.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.131632+0000) 2020-02-06T00:39:23.142 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:23.140+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180321 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:23.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:23.683+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:23.684568+0000) 2020-02-06T00:39:24.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:24.041+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:24.042592+0000) 2020-02-06T00:39:24.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:24.041+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:24.042650+0000) 2020-02-06T00:39:26.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:26.030+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:26.031088+0000) 2020-02-06T00:39:26.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:26.376+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:26.377414+0000) 2020-02-06T00:39:27.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:27.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:27.018285+0000) 2020-02-06T00:39:27.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:27.283+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:27.283803+0000) 2020-02-06T00:39:27.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:27.283+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:27.283935+0000) 2020-02-06T00:39:27.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:27.283+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:27.284376+0000) 2020-02-06T00:39:28.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:28.146+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.147331+0000) 2020-02-06T00:39:28.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:28.147+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.148243+0000) 2020-02-06T00:39:28.149 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:28.147+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.148431+0000) 2020-02-06T00:39:28.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:28.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.148999+0000) 2020-02-06T00:39:28.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:28.148+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.149346+0000) 2020-02-06T00:39:28.150 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:28.149+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.149708+0000) 2020-02-06T00:39:28.151 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:28.148+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.150008+0000) 2020-02-06T00:39:28.151 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:28.150+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.150903+0000) 2020-02-06T00:39:28.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:28.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.151332+0000) 2020-02-06T00:39:28.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:28.151+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.151659+0000) 2020-02-06T00:39:28.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:28.152+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.152656+0000) 2020-02-06T00:39:28.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:28.152+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.153002+0000) 2020-02-06T00:39:28.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:28.152+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.153286+0000) 2020-02-06T00:39:28.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:28.156+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:28.157636+0000) 2020-02-06T00:39:28.167 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:28.165+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180366 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:29.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:29.661+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:29.662566+0000) 2020-02-06T00:39:29.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:29.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:29.911222+0000) 2020-02-06T00:39:30.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:30.512+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:30.512478+0000) 2020-02-06T00:39:30.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:30.770+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:30.771179+0000) 2020-02-06T00:39:32.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:32.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:32.894047+0000) 2020-02-06T00:39:33.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:33.170+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.171840+0000) 2020-02-06T00:39:33.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:33.172+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.173545+0000) 2020-02-06T00:39:33.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:33.172+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.174029+0000) 2020-02-06T00:39:33.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:33.174+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.174818+0000) 2020-02-06T00:39:33.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:33.173+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.175002+0000) 2020-02-06T00:39:33.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:33.175+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.175415+0000) 2020-02-06T00:39:33.178 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:33.175+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.176402+0000) 2020-02-06T00:39:33.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:33.175+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.176773+0000) 2020-02-06T00:39:33.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:33.175+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.176215+0000) 2020-02-06T00:39:33.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:33.176+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.177868+0000) 2020-02-06T00:39:33.180 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:33.177+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.177843+0000) 2020-02-06T00:39:33.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:33.177+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.178258+0000) 2020-02-06T00:39:33.180 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:33.177+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.178299+0000) 2020-02-06T00:39:33.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:33.179+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.180195+0000) 2020-02-06T00:39:33.192 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:33.190+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:33.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:33.692+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:33.692794+0000) 2020-02-06T00:39:34.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:34.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:34.042802+0000) 2020-02-06T00:39:34.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:34.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:34.042886+0000) 2020-02-06T00:39:36.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:36.030+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:36.031335+0000) 2020-02-06T00:39:36.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:36.376+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:36.377644+0000) 2020-02-06T00:39:37.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:37.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:37.018519+0000) 2020-02-06T00:39:37.285 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:37.283+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:37.284048+0000) 2020-02-06T00:39:37.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:37.283+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:37.284142+0000) 2020-02-06T00:39:37.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:37.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:37.284624+0000) 2020-02-06T00:39:38.197 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:38.194+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.195970+0000) 2020-02-06T00:39:38.198 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:38.195+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.197043+0000) 2020-02-06T00:39:38.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:38.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.197963+0000) 2020-02-06T00:39:38.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:38.197+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.198551+0000) 2020-02-06T00:39:38.200 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:38.197+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.199059+0000) 2020-02-06T00:39:38.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:38.198+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.199267+0000) 2020-02-06T00:39:38.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:38.198+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.199519+0000) 2020-02-06T00:39:38.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:38.199+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.200524+0000) 2020-02-06T00:39:38.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:38.200+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.200894+0000) 2020-02-06T00:39:38.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:38.200+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.201315+0000) 2020-02-06T00:39:38.202 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:38.201+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.201524+0000) 2020-02-06T00:39:38.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:38.202+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.202817+0000) 2020-02-06T00:39:38.204 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:38.202+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.203226+0000) 2020-02-06T00:39:38.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:38.205+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:38.206805+0000) 2020-02-06T00:39:38.217 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:38.216+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:39.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:39.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:39.662816+0000) 2020-02-06T00:39:39.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:39.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:39.911447+0000) 2020-02-06T00:39:40.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:40.512+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:40.512736+0000) 2020-02-06T00:39:40.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:40.770+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:40.771358+0000) 2020-02-06T00:39:42.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:42.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:42.894306+0000) 2020-02-06T00:39:43.223 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:43.221+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.222497+0000) 2020-02-06T00:39:43.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:43.221+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.222556+0000) 2020-02-06T00:39:43.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:43.222+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.223049+0000) 2020-02-06T00:39:43.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:43.223+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.223854+0000) 2020-02-06T00:39:43.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:43.223+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.224273+0000) 2020-02-06T00:39:43.226 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:43.223+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.224678+0000) 2020-02-06T00:39:43.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:43.223+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.224876+0000) 2020-02-06T00:39:43.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:43.225+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.226143+0000) 2020-02-06T00:39:43.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:43.225+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.226527+0000) 2020-02-06T00:39:43.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:43.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.226858+0000) 2020-02-06T00:39:43.228 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:43.227+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.227891+0000) 2020-02-06T00:39:43.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:43.227+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.227976+0000) 2020-02-06T00:39:43.229 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:43.228+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.228494+0000) 2020-02-06T00:39:43.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:43.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.232869+0000) 2020-02-06T00:39:43.243 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:43.242+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:43.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:43.700+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:43.700902+0000) 2020-02-06T00:39:43.768 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:39:43.774 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:39:44.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:44.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:44.043012+0000) 2020-02-06T00:39:44.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:44.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:44.043148+0000) 2020-02-06T00:39:46.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:46.031+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:46.031654+0000) 2020-02-06T00:39:46.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:46.377+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:46.377874+0000) 2020-02-06T00:39:47.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:47.017+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:47.018767+0000) 2020-02-06T00:39:47.285 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:47.284+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:47.284328+0000) 2020-02-06T00:39:47.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:47.284+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:47.284402+0000) 2020-02-06T00:39:47.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:47.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:47.284875+0000) 2020-02-06T00:39:48.250 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:48.247+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.249296+0000) 2020-02-06T00:39:48.251 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:48.248+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.250056+0000) 2020-02-06T00:39:48.251 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:48.249+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.250446+0000) 2020-02-06T00:39:48.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:48.249+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.250847+0000) 2020-02-06T00:39:48.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:48.250+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.251256+0000) 2020-02-06T00:39:48.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:48.250+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.252054+0000) 2020-02-06T00:39:48.253 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:48.251+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.252570+0000) 2020-02-06T00:39:48.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:48.251+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.252919+0000) 2020-02-06T00:39:48.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:48.252+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.253119+0000) 2020-02-06T00:39:48.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:48.251+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.253271+0000) 2020-02-06T00:39:48.255 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:48.252+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.253565+0000) 2020-02-06T00:39:48.255 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:48.253+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.254101+0000) 2020-02-06T00:39:48.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:48.252+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.254290+0000) 2020-02-06T00:39:48.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:48.256+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:48.257664+0000) 2020-02-06T00:39:48.269 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:48.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:49.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:49.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:49.663109+0000) 2020-02-06T00:39:49.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:49.911+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:49.911684+0000) 2020-02-06T00:39:50.514 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:50.511+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:50.513029+0000) 2020-02-06T00:39:50.772 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:50.771+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:50.771590+0000) 2020-02-06T00:39:52.895 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:52.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:52.894568+0000) 2020-02-06T00:39:53.274 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:53.272+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.273524+0000) 2020-02-06T00:39:53.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:53.272+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.274050+0000) 2020-02-06T00:39:53.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:53.274+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.275218+0000) 2020-02-06T00:39:53.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:53.274+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.275776+0000) 2020-02-06T00:39:53.278 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:53.275+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.277273+0000) 2020-02-06T00:39:53.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:53.276+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.277586+0000) 2020-02-06T00:39:53.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:53.276+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.277728+0000) 2020-02-06T00:39:53.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:53.277+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.278641+0000) 2020-02-06T00:39:53.279 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:53.277+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.278543+0000) 2020-02-06T00:39:53.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:53.277+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.279348+0000) 2020-02-06T00:39:53.281 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:53.279+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.280474+0000) 2020-02-06T00:39:53.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:53.278+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.280033+0000) 2020-02-06T00:39:53.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:53.280+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.281882+0000) 2020-02-06T00:39:53.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:53.281+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.282141+0000) 2020-02-06T00:39:53.293 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:53.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:53.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:53.707+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:53.709042+0000) 2020-02-06T00:39:54.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:54.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:54.043260+0000) 2020-02-06T00:39:54.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:54.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:54.043395+0000) 2020-02-06T00:39:56.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:56.030+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:56.031923+0000) 2020-02-06T00:39:56.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:56.377+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:56.378122+0000) 2020-02-06T00:39:57.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:57.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:57.018995+0000) 2020-02-06T00:39:57.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:57.283+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:57.284677+0000) 2020-02-06T00:39:57.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:57.283+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:57.284665+0000) 2020-02-06T00:39:57.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:57.283+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:57.285126+0000) 2020-02-06T00:39:58.299 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:39:58.297+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.298577+0000) 2020-02-06T00:39:58.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:39:58.298+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.299728+0000) 2020-02-06T00:39:58.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:39:58.298+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.299530+0000) 2020-02-06T00:39:58.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:39:58.298+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.299981+0000) 2020-02-06T00:39:58.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:39:58.299+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.300450+0000) 2020-02-06T00:39:58.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:58.300+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.301383+0000) 2020-02-06T00:39:58.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:39:58.300+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.301512+0000) 2020-02-06T00:39:58.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:39:58.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.301797+0000) 2020-02-06T00:39:58.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:58.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.302428+0000) 2020-02-06T00:39:58.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:39:58.301+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.302566+0000) 2020-02-06T00:39:58.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:39:58.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.303935+0000) 2020-02-06T00:39:58.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:39:58.302+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.304217+0000) 2020-02-06T00:39:58.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:39:58.302+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.304316+0000) 2020-02-06T00:39:58.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:58.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:58.307375+0000) 2020-02-06T00:39:58.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:39:58.317+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180617 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:39:59.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:59.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:59.663368+0000) 2020-02-06T00:39:59.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:39:59.910+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:39:59.911935+0000) 2020-02-06T00:40:00.514 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:00.512+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:00.513275+0000) 2020-02-06T00:40:00.773 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:00.771+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:00.771829+0000) 2020-02-06T00:40:02.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:02.893+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:02.894812+0000) 2020-02-06T00:40:03.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:03.323+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.324861+0000) 2020-02-06T00:40:03.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:03.324+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.325650+0000) 2020-02-06T00:40:03.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:03.325+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.326605+0000) 2020-02-06T00:40:03.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:03.325+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.327083+0000) 2020-02-06T00:40:03.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:03.325+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.326989+0000) 2020-02-06T00:40:03.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:03.326+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.327306+0000) 2020-02-06T00:40:03.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:03.326+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.327715+0000) 2020-02-06T00:40:03.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:03.326+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.328113+0000) 2020-02-06T00:40:03.330 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:03.327+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.328529+0000) 2020-02-06T00:40:03.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:03.327+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.329119+0000) 2020-02-06T00:40:03.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:03.329+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.329349+0000) 2020-02-06T00:40:03.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:03.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.330045+0000) 2020-02-06T00:40:03.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:03.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.330748+0000) 2020-02-06T00:40:03.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:03.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.334864+0000) 2020-02-06T00:40:03.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:03.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:03.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:03.718+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:03.718839+0000) 2020-02-06T00:40:04.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:04.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:04.043475+0000) 2020-02-06T00:40:04.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:04.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:04.043633+0000) 2020-02-06T00:40:06.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:06.030+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:06.032170+0000) 2020-02-06T00:40:06.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:06.378+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:06.378351+0000) 2020-02-06T00:40:07.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:07.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:07.019297+0000) 2020-02-06T00:40:07.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:07.284+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:07.284944+0000) 2020-02-06T00:40:07.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:07.284+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:07.284944+0000) 2020-02-06T00:40:07.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:07.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:07.285392+0000) 2020-02-06T00:40:08.351 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:08.350+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.350642+0000) 2020-02-06T00:40:08.352 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:08.351+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.351239+0000) 2020-02-06T00:40:08.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:08.351+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.352520+0000) 2020-02-06T00:40:08.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:08.352+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.353563+0000) 2020-02-06T00:40:08.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:08.353+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.353788+0000) 2020-02-06T00:40:08.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:08.352+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.354312+0000) 2020-02-06T00:40:08.355 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:08.354+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.354928+0000) 2020-02-06T00:40:08.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:08.353+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.354779+0000) 2020-02-06T00:40:08.356 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:08.355+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.355374+0000) 2020-02-06T00:40:08.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:08.355+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.355843+0000) 2020-02-06T00:40:08.357 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:08.355+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.357036+0000) 2020-02-06T00:40:08.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:08.355+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.357368+0000) 2020-02-06T00:40:08.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:08.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.359151+0000) 2020-02-06T00:40:08.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:08.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:08.359364+0000) 2020-02-06T00:40:08.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:08.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:09.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:09.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:09.663614+0000) 2020-02-06T00:40:09.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:09.911+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:09.912154+0000) 2020-02-06T00:40:10.514 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:10.512+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:10.513529+0000) 2020-02-06T00:40:10.773 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:10.771+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:10.772146+0000) 2020-02-06T00:40:12.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:12.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:12.895107+0000) 2020-02-06T00:40:13.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:13.377+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.378516+0000) 2020-02-06T00:40:13.380 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:13.378+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.379220+0000) 2020-02-06T00:40:13.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:13.378+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.380058+0000) 2020-02-06T00:40:13.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:13.379+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.380204+0000) 2020-02-06T00:40:13.381 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:13.379+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.380542+0000) 2020-02-06T00:40:13.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:13.379+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.381223+0000) 2020-02-06T00:40:13.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:13.380+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.381779+0000) 2020-02-06T00:40:13.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:13.381+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.382280+0000) 2020-02-06T00:40:13.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:13.381+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.382963+0000) 2020-02-06T00:40:13.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:13.382+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.383400+0000) 2020-02-06T00:40:13.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:13.382+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.383818+0000) 2020-02-06T00:40:13.385 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:13.382+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.384370+0000) 2020-02-06T00:40:13.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:13.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.388106+0000) 2020-02-06T00:40:13.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:13.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.388395+0000) 2020-02-06T00:40:13.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:13.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180738 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:13.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:13.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:13.730250+0000) 2020-02-06T00:40:13.900 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:40:13.907 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:40:14.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:14.042+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:14.043689+0000) 2020-02-06T00:40:14.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:14.042+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:14.043863+0000) 2020-02-06T00:40:16.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:16.031+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:16.032461+0000) 2020-02-06T00:40:16.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:16.377+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:16.378575+0000) 2020-02-06T00:40:17.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:17.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:17.019529+0000) 2020-02-06T00:40:17.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:17.284+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:17.285249+0000) 2020-02-06T00:40:17.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:17.284+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:17.285251+0000) 2020-02-06T00:40:17.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:17.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:17.285686+0000) 2020-02-06T00:40:18.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:18.400+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.401867+0000) 2020-02-06T00:40:18.404 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:18.402+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.403794+0000) 2020-02-06T00:40:18.405 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:18.403+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.404714+0000) 2020-02-06T00:40:18.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:18.404+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.405248+0000) 2020-02-06T00:40:18.406 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:18.404+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.405555+0000) 2020-02-06T00:40:18.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:18.404+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.405993+0000) 2020-02-06T00:40:18.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:18.404+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.405568+0000) 2020-02-06T00:40:18.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:18.404+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.405834+0000) 2020-02-06T00:40:18.408 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:18.405+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.406479+0000) 2020-02-06T00:40:18.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:18.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.407166+0000) 2020-02-06T00:40:18.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:18.405+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.407291+0000) 2020-02-06T00:40:18.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:18.406+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.408090+0000) 2020-02-06T00:40:18.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:18.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.410086+0000) 2020-02-06T00:40:18.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:18.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:18.411187+0000) 2020-02-06T00:40:18.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:18.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180783 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:19.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:19.662+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:19.663842+0000) 2020-02-06T00:40:19.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:19.911+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:19.912330+0000) 2020-02-06T00:40:20.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:20.513+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:20.513801+0000) 2020-02-06T00:40:20.773 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:20.771+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:20.772448+0000) 2020-02-06T00:40:22.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:22.894+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:22.895367+0000) 2020-02-06T00:40:23.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:23.426+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.427521+0000) 2020-02-06T00:40:23.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:23.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.429717+0000) 2020-02-06T00:40:23.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:23.429+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.430585+0000) 2020-02-06T00:40:23.432 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:23.429+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.431250+0000) 2020-02-06T00:40:23.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:23.430+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.431635+0000) 2020-02-06T00:40:23.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:23.431+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.432171+0000) 2020-02-06T00:40:23.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:23.430+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.432107+0000) 2020-02-06T00:40:23.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:23.431+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.432829+0000) 2020-02-06T00:40:23.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:23.431+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.432922+0000) 2020-02-06T00:40:23.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:23.432+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.433616+0000) 2020-02-06T00:40:23.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:23.432+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.433684+0000) 2020-02-06T00:40:23.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:23.432+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.434276+0000) 2020-02-06T00:40:23.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:23.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.435812+0000) 2020-02-06T00:40:23.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:23.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.436018+0000) 2020-02-06T00:40:23.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:23.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:23.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:23.737+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:23.738989+0000) 2020-02-06T00:40:24.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:24.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:24.043948+0000) 2020-02-06T00:40:24.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:24.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:24.044157+0000) 2020-02-06T00:40:26.033 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:26.032+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:26.032728+0000) 2020-02-06T00:40:26.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:26.377+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:26.378824+0000) 2020-02-06T00:40:27.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:27.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:27.019765+0000) 2020-02-06T00:40:27.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:27.284+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:27.285549+0000) 2020-02-06T00:40:27.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:27.284+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:27.285549+0000) 2020-02-06T00:40:27.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:27.284+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:27.285959+0000) 2020-02-06T00:40:28.453 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:28.451+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.452681+0000) 2020-02-06T00:40:28.454 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:28.452+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.453537+0000) 2020-02-06T00:40:28.456 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:28.453+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.454718+0000) 2020-02-06T00:40:28.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:28.453+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.455338+0000) 2020-02-06T00:40:28.456 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:28.454+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.455670+0000) 2020-02-06T00:40:28.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:28.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.456503+0000) 2020-02-06T00:40:28.458 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:28.455+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.457326+0000) 2020-02-06T00:40:28.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:28.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.457926+0000) 2020-02-06T00:40:28.459 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:28.457+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.458516+0000) 2020-02-06T00:40:28.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:28.457+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.458783+0000) 2020-02-06T00:40:28.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:28.457+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.459173+0000) 2020-02-06T00:40:28.460 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:28.458+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.459148+0000) 2020-02-06T00:40:28.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:28.458+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.459662+0000) 2020-02-06T00:40:28.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:28.461+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:28.462759+0000) 2020-02-06T00:40:28.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:28.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180865 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:29.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:29.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:29.664112+0000) 2020-02-06T00:40:29.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:29.911+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:29.912558+0000) 2020-02-06T00:40:30.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:30.513+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:30.514132+0000) 2020-02-06T00:40:30.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:30.772+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:30.772681+0000) 2020-02-06T00:40:32.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:32.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:32.895683+0000) 2020-02-06T00:40:33.479 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:33.477+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.478065+0000) 2020-02-06T00:40:33.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:33.477+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.478641+0000) 2020-02-06T00:40:33.481 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:33.479+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.480545+0000) 2020-02-06T00:40:33.481 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:33.479+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.480728+0000) 2020-02-06T00:40:33.482 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:33.480+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.481655+0000) 2020-02-06T00:40:33.483 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:33.480+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.481899+0000) 2020-02-06T00:40:33.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:33.481+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.482232+0000) 2020-02-06T00:40:33.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:33.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.482654+0000) 2020-02-06T00:40:33.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:33.481+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.482796+0000) 2020-02-06T00:40:33.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:33.481+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.483146+0000) 2020-02-06T00:40:33.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:33.482+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.483209+0000) 2020-02-06T00:40:33.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:33.482+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.483627+0000) 2020-02-06T00:40:33.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:33.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.487343+0000) 2020-02-06T00:40:33.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:33.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.487588+0000) 2020-02-06T00:40:33.497 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:33.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180904 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:33.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:33.746+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:33.747247+0000) 2020-02-06T00:40:34.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:34.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:34.044162+0000) 2020-02-06T00:40:34.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:34.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:34.044360+0000) 2020-02-06T00:40:36.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:36.032+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:36.033017+0000) 2020-02-06T00:40:36.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:36.378+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:36.379053+0000) 2020-02-06T00:40:37.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:37.018+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:37.020000+0000) 2020-02-06T00:40:37.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:37.285+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:37.285825+0000) 2020-02-06T00:40:37.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:37.285+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:37.285825+0000) 2020-02-06T00:40:37.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:37.285+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:37.286195+0000) 2020-02-06T00:40:38.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:38.501+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.502620+0000) 2020-02-06T00:40:38.504 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:38.501+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.502967+0000) 2020-02-06T00:40:38.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:38.501+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.503259+0000) 2020-02-06T00:40:38.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:38.504+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.505729+0000) 2020-02-06T00:40:38.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:38.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.505783+0000) 2020-02-06T00:40:38.507 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:38.505+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.506201+0000) 2020-02-06T00:40:38.508 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:38.505+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.506737+0000) 2020-02-06T00:40:38.508 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:38.505+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.507136+0000) 2020-02-06T00:40:38.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:38.506+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.507543+0000) 2020-02-06T00:40:38.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:38.506+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.507844+0000) 2020-02-06T00:40:38.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:38.507+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.508862+0000) 2020-02-06T00:40:38.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:38.507+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.509048+0000) 2020-02-06T00:40:38.510 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:38.508+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.509436+0000) 2020-02-06T00:40:38.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:38.508+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:38.509895+0000) 2020-02-06T00:40:38.523 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:38.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180949 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:39.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:39.663+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:39.664384+0000) 2020-02-06T00:40:39.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:39.912+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:39.912789+0000) 2020-02-06T00:40:40.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:40.513+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:40.514372+0000) 2020-02-06T00:40:40.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:40.772+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:40.772909+0000) 2020-02-06T00:40:42.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:42.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:42.895932+0000) 2020-02-06T00:40:43.529 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:43.527+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.528273+0000) 2020-02-06T00:40:43.530 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:43.527+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.527964+0000) 2020-02-06T00:40:43.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:43.529+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.530131+0000) 2020-02-06T00:40:43.531 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:43.529+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.530847+0000) 2020-02-06T00:40:43.532 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:43.530+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.531226+0000) 2020-02-06T00:40:43.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:43.530+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.531912+0000) 2020-02-06T00:40:43.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:43.532+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.532965+0000) 2020-02-06T00:40:43.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:43.533+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.534088+0000) 2020-02-06T00:40:43.535 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:43.534+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.534643+0000) 2020-02-06T00:40:43.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:43.533+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.534812+0000) 2020-02-06T00:40:43.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:43.534+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.534962+0000) 2020-02-06T00:40:43.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:43.534+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.535153+0000) 2020-02-06T00:40:43.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:43.534+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.535674+0000) 2020-02-06T00:40:43.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:43.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.540318+0000) 2020-02-06T00:40:43.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:43.547+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 180988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:43.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:43.753+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:43.754737+0000) 2020-02-06T00:40:44.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:44.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:44.044402+0000) 2020-02-06T00:40:44.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:44.043+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:44.044599+0000) 2020-02-06T00:40:44.102 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:40:44.108 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:40:46.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:46.032+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:46.033260+0000) 2020-02-06T00:40:46.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:46.378+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:46.379339+0000) 2020-02-06T00:40:47.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:47.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:47.020310+0000) 2020-02-06T00:40:47.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:47.285+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:47.286141+0000) 2020-02-06T00:40:47.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:47.285+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:47.286141+0000) 2020-02-06T00:40:47.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:47.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:47.286409+0000) 2020-02-06T00:40:48.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:48.552+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.553551+0000) 2020-02-06T00:40:48.555 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:48.553+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.554181+0000) 2020-02-06T00:40:48.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:48.553+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.554454+0000) 2020-02-06T00:40:48.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:48.554+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.554864+0000) 2020-02-06T00:40:48.557 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:48.555+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.556232+0000) 2020-02-06T00:40:48.557 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:48.555+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.556385+0000) 2020-02-06T00:40:48.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:48.555+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.556927+0000) 2020-02-06T00:40:48.558 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:48.557+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.557571+0000) 2020-02-06T00:40:48.558 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:48.556+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.557360+0000) 2020-02-06T00:40:48.559 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:48.556+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.557160+0000) 2020-02-06T00:40:48.559 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:48.557+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.558151+0000) 2020-02-06T00:40:48.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:48.558+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.558951+0000) 2020-02-06T00:40:48.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:48.561+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.562956+0000) 2020-02-06T00:40:48.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:48.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:48.563232+0000) 2020-02-06T00:40:48.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:48.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:49.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:49.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:49.664615+0000) 2020-02-06T00:40:49.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:49.912+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:49.913017+0000) 2020-02-06T00:40:50.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:50.514+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:50.514643+0000) 2020-02-06T00:40:50.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:50.772+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:50.773171+0000) 2020-02-06T00:40:52.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:52.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:52.896222+0000) 2020-02-06T00:40:53.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:53.578+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.579261+0000) 2020-02-06T00:40:53.581 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:53.578+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.579891+0000) 2020-02-06T00:40:53.581 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:53.580+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.580989+0000) 2020-02-06T00:40:53.582 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:53.581+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.581628+0000) 2020-02-06T00:40:53.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:53.580+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.582024+0000) 2020-02-06T00:40:53.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:53.581+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.582313+0000) 2020-02-06T00:40:53.583 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:53.581+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.582610+0000) 2020-02-06T00:40:53.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:53.581+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.582951+0000) 2020-02-06T00:40:53.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:53.582+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.583258+0000) 2020-02-06T00:40:53.584 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:53.582+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.583391+0000) 2020-02-06T00:40:53.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:53.583+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.583707+0000) 2020-02-06T00:40:53.585 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:53.584+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.584651+0000) 2020-02-06T00:40:53.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:53.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.584978+0000) 2020-02-06T00:40:53.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:53.588+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.589907+0000) 2020-02-06T00:40:53.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:53.598+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181072 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:53.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:53.761+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:53.762894+0000) 2020-02-06T00:40:54.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:54.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:54.044614+0000) 2020-02-06T00:40:54.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:54.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:54.044806+0000) 2020-02-06T00:40:56.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:56.033+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:56.033497+0000) 2020-02-06T00:40:56.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:56.379+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:56.379566+0000) 2020-02-06T00:40:57.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:57.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:57.020548+0000) 2020-02-06T00:40:57.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:57.286+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:57.286478+0000) 2020-02-06T00:40:57.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:57.286+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:57.286478+0000) 2020-02-06T00:40:57.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:57.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:57.286711+0000) 2020-02-06T00:40:58.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:40:58.604+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.605133+0000) 2020-02-06T00:40:58.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:40:58.604+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.605183+0000) 2020-02-06T00:40:58.607 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:40:58.605+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.605730+0000) 2020-02-06T00:40:58.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:40:58.604+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.605980+0000) 2020-02-06T00:40:58.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:40:58.606+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.606637+0000) 2020-02-06T00:40:58.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:58.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.607520+0000) 2020-02-06T00:40:58.608 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:40:58.607+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.607706+0000) 2020-02-06T00:40:58.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:40:58.607+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.608914+0000) 2020-02-06T00:40:58.610 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:40:58.608+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.608744+0000) 2020-02-06T00:40:58.610 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:40:58.608+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.609339+0000) 2020-02-06T00:40:58.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:40:58.608+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.609407+0000) 2020-02-06T00:40:58.611 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:40:58.610+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.610577+0000) 2020-02-06T00:40:58.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:58.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.610402+0000) 2020-02-06T00:40:58.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:58.612+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:58.613193+0000) 2020-02-06T00:40:58.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:40:58.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:40:59.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:59.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:59.664871+0000) 2020-02-06T00:40:59.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:40:59.913+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:40:59.913276+0000) 2020-02-06T00:41:00.516 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:00.514+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:00.514911+0000) 2020-02-06T00:41:00.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:00.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:00.773400+0000) 2020-02-06T00:41:02.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:02.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:02.896489+0000) 2020-02-06T00:41:03.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:03.630+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.631180+0000) 2020-02-06T00:41:03.633 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:03.630+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.631915+0000) 2020-02-06T00:41:03.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:03.632+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.632697+0000) 2020-02-06T00:41:03.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:03.632+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.633153+0000) 2020-02-06T00:41:03.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:03.633+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.633659+0000) 2020-02-06T00:41:03.643 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:03.633+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.634158+0000) 2020-02-06T00:41:03.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:03.635+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.636313+0000) 2020-02-06T00:41:03.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:03.636+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.636881+0000) 2020-02-06T00:41:03.644 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:03.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.636969+0000) 2020-02-06T00:41:03.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:03.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.634494+0000) 2020-02-06T00:41:03.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:03.639+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.640955+0000) 2020-02-06T00:41:03.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:03.633+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.634796+0000) 2020-02-06T00:41:03.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:03.634+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.635589+0000) 2020-02-06T00:41:03.645 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:03.635+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.636639+0000) 2020-02-06T00:41:03.651 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:03.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181156 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:03.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:03.770+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:03.771131+0000) 2020-02-06T00:41:04.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:04.043+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:04.044838+0000) 2020-02-06T00:41:04.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:04.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:04.045071+0000) 2020-02-06T00:41:06.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:06.033+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:06.033765+0000) 2020-02-06T00:41:06.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:06.378+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:06.379808+0000) 2020-02-06T00:41:07.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:07.019+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:07.020787+0000) 2020-02-06T00:41:07.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:07.286+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:07.286788+0000) 2020-02-06T00:41:07.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:07.286+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:07.286789+0000) 2020-02-06T00:41:07.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:07.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:07.286990+0000) 2020-02-06T00:41:08.657 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:08.655+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.656134+0000) 2020-02-06T00:41:08.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:08.655+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.656602+0000) 2020-02-06T00:41:08.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:08.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.656971+0000) 2020-02-06T00:41:08.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:08.657+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.657747+0000) 2020-02-06T00:41:08.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:08.657+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.658112+0000) 2020-02-06T00:41:08.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:08.658+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.659396+0000) 2020-02-06T00:41:08.660 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:08.659+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.659737+0000) 2020-02-06T00:41:08.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:08.659+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.660276+0000) 2020-02-06T00:41:08.661 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:08.659+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.660316+0000) 2020-02-06T00:41:08.662 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:08.660+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.661117+0000) 2020-02-06T00:41:08.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:08.660+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.661370+0000) 2020-02-06T00:41:08.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:08.660+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.661752+0000) 2020-02-06T00:41:08.663 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:08.661+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.662314+0000) 2020-02-06T00:41:08.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:08.663+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:08.664605+0000) 2020-02-06T00:41:08.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:08.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:09.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:09.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:09.665133+0000) 2020-02-06T00:41:09.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:09.912+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:09.913506+0000) 2020-02-06T00:41:10.516 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:10.514+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:10.515134+0000) 2020-02-06T00:41:10.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:10.772+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:10.773639+0000) 2020-02-06T00:41:12.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:12.895+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:12.896743+0000) 2020-02-06T00:41:13.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:13.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.680429+0000) 2020-02-06T00:41:13.682 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:13.679+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.680868+0000) 2020-02-06T00:41:13.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:13.680+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.681169+0000) 2020-02-06T00:41:13.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:13.680+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.681969+0000) 2020-02-06T00:41:13.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:13.681+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.682693+0000) 2020-02-06T00:41:13.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:13.681+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.683310+0000) 2020-02-06T00:41:13.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:13.682+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.684229+0000) 2020-02-06T00:41:13.685 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:13.682+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.683720+0000) 2020-02-06T00:41:13.685 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:13.683+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.684588+0000) 2020-02-06T00:41:13.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:13.683+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.685034+0000) 2020-02-06T00:41:13.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:13.683+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.685135+0000) 2020-02-06T00:41:13.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:13.683+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.685265+0000) 2020-02-06T00:41:13.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:13.685+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.686409+0000) 2020-02-06T00:41:13.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:13.685+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.686621+0000) 2020-02-06T00:41:13.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:13.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:13.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:13.778+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:13.779637+0000) 2020-02-06T00:41:14.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:14.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:14.045049+0000) 2020-02-06T00:41:14.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:14.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:14.045305+0000) 2020-02-06T00:41:14.270 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:41:14.276 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:41:16.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:16.033+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:16.034056+0000) 2020-02-06T00:41:16.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:16.379+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:16.380015+0000) 2020-02-06T00:41:17.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:17.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:17.021022+0000) 2020-02-06T00:41:17.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:17.286+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:17.287074+0000) 2020-02-06T00:41:17.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:17.286+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:17.287074+0000) 2020-02-06T00:41:17.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:17.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:17.287266+0000) 2020-02-06T00:41:18.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:18.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.706191+0000) 2020-02-06T00:41:18.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:18.705+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.707393+0000) 2020-02-06T00:41:18.709 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:18.706+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.708041+0000) 2020-02-06T00:41:18.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:18.707+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.708642+0000) 2020-02-06T00:41:18.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:18.708+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.709431+0000) 2020-02-06T00:41:18.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:18.709+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.710343+0000) 2020-02-06T00:41:18.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:18.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.710669+0000) 2020-02-06T00:41:18.712 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:18.709+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.711103+0000) 2020-02-06T00:41:18.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:18.710+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.711485+0000) 2020-02-06T00:41:18.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:18.710+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.711695+0000) 2020-02-06T00:41:18.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:18.711+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.712482+0000) 2020-02-06T00:41:18.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:18.711+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.712591+0000) 2020-02-06T00:41:18.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:18.711+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.713198+0000) 2020-02-06T00:41:18.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:18.717+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:18.718466+0000) 2020-02-06T00:41:18.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:18.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:19.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:19.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:19.665372+0000) 2020-02-06T00:41:19.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:19.912+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:19.913737+0000) 2020-02-06T00:41:20.516 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:20.514+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:20.515394+0000) 2020-02-06T00:41:20.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:20.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:20.773874+0000) 2020-02-06T00:41:22.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:22.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:22.896988+0000) 2020-02-06T00:41:23.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:23.731+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.732410+0000) 2020-02-06T00:41:23.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:23.732+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.733838+0000) 2020-02-06T00:41:23.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:23.732+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.733859+0000) 2020-02-06T00:41:23.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:23.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.735052+0000) 2020-02-06T00:41:23.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:23.733+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.735038+0000) 2020-02-06T00:41:23.737 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:23.734+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.735655+0000) 2020-02-06T00:41:23.738 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:23.734+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.735801+0000) 2020-02-06T00:41:23.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:23.735+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.736291+0000) 2020-02-06T00:41:23.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:23.734+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.736210+0000) 2020-02-06T00:41:23.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:23.736+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.737651+0000) 2020-02-06T00:41:23.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:23.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.737515+0000) 2020-02-06T00:41:23.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:23.736+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.738139+0000) 2020-02-06T00:41:23.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:23.737+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.738846+0000) 2020-02-06T00:41:23.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:23.737+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.738936+0000) 2020-02-06T00:41:23.754 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:23.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:23.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:23.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:23.788464+0000) 2020-02-06T00:41:24.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:24.044+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:24.045300+0000) 2020-02-06T00:41:24.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:24.044+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:24.045474+0000) 2020-02-06T00:41:26.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:26.033+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:26.034317+0000) 2020-02-06T00:41:26.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:26.379+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:26.380301+0000) 2020-02-06T00:41:27.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:27.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:27.021297+0000) 2020-02-06T00:41:27.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:27.286+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:27.287379+0000) 2020-02-06T00:41:27.289 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:27.286+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:27.287379+0000) 2020-02-06T00:41:27.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:27.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:27.287537+0000) 2020-02-06T00:41:28.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:28.757+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.758848+0000) 2020-02-06T00:41:28.760 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:28.758+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.759798+0000) 2020-02-06T00:41:28.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:28.760+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.761698+0000) 2020-02-06T00:41:28.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:28.760+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.761756+0000) 2020-02-06T00:41:28.763 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:28.761+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.762156+0000) 2020-02-06T00:41:28.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:28.760+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.762085+0000) 2020-02-06T00:41:28.764 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:28.761+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.762536+0000) 2020-02-06T00:41:28.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:28.761+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.763033+0000) 2020-02-06T00:41:28.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:28.762+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.763413+0000) 2020-02-06T00:41:28.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:28.762+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.763698+0000) 2020-02-06T00:41:28.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:28.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.763699+0000) 2020-02-06T00:41:28.766 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:28.763+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.764606+0000) 2020-02-06T00:41:28.766 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:28.763+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.765241+0000) 2020-02-06T00:41:28.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:28.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:28.769237+0000) 2020-02-06T00:41:28.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:28.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:29.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:29.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:29.665605+0000) 2020-02-06T00:41:29.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:29.912+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:29.913962+0000) 2020-02-06T00:41:30.516 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:30.514+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:30.515697+0000) 2020-02-06T00:41:30.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:30.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:30.774134+0000) 2020-02-06T00:41:32.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:32.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:32.897267+0000) 2020-02-06T00:41:33.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:33.784+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.785258+0000) 2020-02-06T00:41:33.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:33.784+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.786011+0000) 2020-02-06T00:41:33.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:33.785+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.786301+0000) 2020-02-06T00:41:33.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:33.785+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.786849+0000) 2020-02-06T00:41:33.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:33.786+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.787173+0000) 2020-02-06T00:41:33.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:33.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.787376+0000) 2020-02-06T00:41:33.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:33.786+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.787864+0000) 2020-02-06T00:41:33.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:33.786+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.787963+0000) 2020-02-06T00:41:33.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:33.787+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.788513+0000) 2020-02-06T00:41:33.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:33.788+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.789907+0000) 2020-02-06T00:41:33.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:33.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.789993+0000) 2020-02-06T00:41:33.791 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:33.789+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.790539+0000) 2020-02-06T00:41:33.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:33.789+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.791061+0000) 2020-02-06T00:41:33.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:33.794+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.795195+0000) 2020-02-06T00:41:33.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:33.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:33.795229+0000) 2020-02-06T00:41:33.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:33.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:34.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:34.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:34.045507+0000) 2020-02-06T00:41:34.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:34.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:34.045636+0000) 2020-02-06T00:41:36.035 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:36.033+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:36.034713+0000) 2020-02-06T00:41:36.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:36.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:36.380529+0000) 2020-02-06T00:41:37.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:37.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:37.021548+0000) 2020-02-06T00:41:37.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:37.286+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:37.287725+0000) 2020-02-06T00:41:37.289 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:37.286+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:37.287725+0000) 2020-02-06T00:41:37.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:37.286+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:37.287779+0000) 2020-02-06T00:41:38.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:38.811+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.812930+0000) 2020-02-06T00:41:38.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:38.812+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.813159+0000) 2020-02-06T00:41:38.815 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:38.812+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.814268+0000) 2020-02-06T00:41:38.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:38.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.814586+0000) 2020-02-06T00:41:38.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:38.813+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.814859+0000) 2020-02-06T00:41:38.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:38.813+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.815018+0000) 2020-02-06T00:41:38.817 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:38.814+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.815284+0000) 2020-02-06T00:41:38.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:38.814+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.815536+0000) 2020-02-06T00:41:38.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:38.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.816050+0000) 2020-02-06T00:41:38.818 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:38.814+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.816142+0000) 2020-02-06T00:41:38.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:38.815+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.816597+0000) 2020-02-06T00:41:38.818 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:38.816+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.817549+0000) 2020-02-06T00:41:38.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:38.816+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.817869+0000) 2020-02-06T00:41:38.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:38.820+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:38.821473+0000) 2020-02-06T00:41:38.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:38.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181447 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:39.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:39.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:39.665816+0000) 2020-02-06T00:41:39.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:39.913+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:39.914207+0000) 2020-02-06T00:41:40.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:40.515+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:40.515989+0000) 2020-02-06T00:41:40.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:40.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:40.774365+0000) 2020-02-06T00:41:42.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:42.896+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:42.897515+0000) 2020-02-06T00:41:43.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:43.795+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.795421+0000) 2020-02-06T00:41:43.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:43.836+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.837444+0000) 2020-02-06T00:41:43.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:43.836+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.837961+0000) 2020-02-06T00:41:43.839 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:43.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.838707+0000) 2020-02-06T00:41:43.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:43.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.838996+0000) 2020-02-06T00:41:43.840 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:43.838+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.839510+0000) 2020-02-06T00:41:43.840 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:43.838+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.839919+0000) 2020-02-06T00:41:43.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:43.839+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.840342+0000) 2020-02-06T00:41:43.841 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:43.839+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.840678+0000) 2020-02-06T00:41:43.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:43.839+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.840650+0000) 2020-02-06T00:41:43.842 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:43.840+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.841665+0000) 2020-02-06T00:41:43.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:43.840+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.841363+0000) 2020-02-06T00:41:43.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:43.840+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.841447+0000) 2020-02-06T00:41:43.843 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:43.840+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.842084+0000) 2020-02-06T00:41:43.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:43.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:43.846800+0000) 2020-02-06T00:41:43.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:43.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:44.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:44.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:44.045716+0000) 2020-02-06T00:41:44.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:44.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:44.045847+0000) 2020-02-06T00:41:44.433 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:41:44.439 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:41:46.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:46.034+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:46.034979+0000) 2020-02-06T00:41:46.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:46.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:46.380763+0000) 2020-02-06T00:41:47.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:47.020+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:47.021796+0000) 2020-02-06T00:41:47.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:47.287+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:47.288053+0000) 2020-02-06T00:41:47.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:47.287+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:47.288041+0000) 2020-02-06T00:41:47.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:47.287+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:47.288037+0000) 2020-02-06T00:41:48.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:48.861+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.863005+0000) 2020-02-06T00:41:48.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:48.862+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.863521+0000) 2020-02-06T00:41:48.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:48.863+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.864584+0000) 2020-02-06T00:41:48.867 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:48.864+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.866020+0000) 2020-02-06T00:41:48.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:48.864+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.866017+0000) 2020-02-06T00:41:48.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:48.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.866674+0000) 2020-02-06T00:41:48.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:48.865+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.866501+0000) 2020-02-06T00:41:48.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:48.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.867170+0000) 2020-02-06T00:41:48.869 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:48.866+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.867436+0000) 2020-02-06T00:41:48.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:48.866+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.868264+0000) 2020-02-06T00:41:48.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:48.868+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.869318+0000) 2020-02-06T00:41:48.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:48.868+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.869644+0000) 2020-02-06T00:41:48.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:48.868+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.869730+0000) 2020-02-06T00:41:48.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:48.875+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:48.876394+0000) 2020-02-06T00:41:48.884 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:48.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181528 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:49.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:49.664+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:49.666026+0000) 2020-02-06T00:41:49.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:49.914+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:49.914439+0000) 2020-02-06T00:41:50.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:50.515+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:50.516228+0000) 2020-02-06T00:41:50.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:50.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:50.774614+0000) 2020-02-06T00:41:52.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:52.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:52.897767+0000) 2020-02-06T00:41:53.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:53.803+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.803332+0000) 2020-02-06T00:41:53.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:53.887+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.887398+0000) 2020-02-06T00:41:53.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:53.888+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.888626+0000) 2020-02-06T00:41:53.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:53.889+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.889437+0000) 2020-02-06T00:41:53.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:53.888+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.890333+0000) 2020-02-06T00:41:53.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:53.891+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.891120+0000) 2020-02-06T00:41:53.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:53.889+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.890993+0000) 2020-02-06T00:41:53.893 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:53.891+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.891421+0000) 2020-02-06T00:41:53.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:53.890+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.892185+0000) 2020-02-06T00:41:53.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:53.892+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.892836+0000) 2020-02-06T00:41:53.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:53.891+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.892558+0000) 2020-02-06T00:41:53.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:53.892+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.894005+0000) 2020-02-06T00:41:53.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:53.893+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.894820+0000) 2020-02-06T00:41:53.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:53.902+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.902459+0000) 2020-02-06T00:41:53.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:53.902+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:53.902803+0000) 2020-02-06T00:41:53.911 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:53.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:54.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:54.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:54.045945+0000) 2020-02-06T00:41:54.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:54.046+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:54.046133+0000) 2020-02-06T00:41:56.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:56.034+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:56.035235+0000) 2020-02-06T00:41:56.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:56.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:56.381041+0000) 2020-02-06T00:41:57.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:57.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:57.022032+0000) 2020-02-06T00:41:57.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:57.287+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:57.288312+0000) 2020-02-06T00:41:57.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:57.287+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:57.288329+0000) 2020-02-06T00:41:57.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:57.287+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:57.288332+0000) 2020-02-06T00:41:58.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:41:58.913+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.915027+0000) 2020-02-06T00:41:58.917 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:41:58.915+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.916511+0000) 2020-02-06T00:41:58.918 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:41:58.914+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.916263+0000) 2020-02-06T00:41:58.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:58.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.917160+0000) 2020-02-06T00:41:58.919 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:41:58.916+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.917387+0000) 2020-02-06T00:41:58.919 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:41:58.916+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.917673+0000) 2020-02-06T00:41:58.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:41:58.917+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.918578+0000) 2020-02-06T00:41:58.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:41:58.917+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.918569+0000) 2020-02-06T00:41:58.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:41:58.917+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.919057+0000) 2020-02-06T00:41:58.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:41:58.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.919499+0000) 2020-02-06T00:41:58.921 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:41:58.918+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.919707+0000) 2020-02-06T00:41:58.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:41:58.918+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.919930+0000) 2020-02-06T00:41:58.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:58.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.925035+0000) 2020-02-06T00:41:58.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:58.924+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:58.925316+0000) 2020-02-06T00:41:58.936 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:41:58.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:41:59.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:59.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:59.666265+0000) 2020-02-06T00:41:59.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:41:59.913+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:41:59.914686+0000) 2020-02-06T00:42:00.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:00.515+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:00.516467+0000) 2020-02-06T00:42:00.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:00.773+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:00.774861+0000) 2020-02-06T00:42:02.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:02.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:02.898030+0000) 2020-02-06T00:42:03.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:03.810+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.811471+0000) 2020-02-06T00:42:03.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:03.940+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.940776+0000) 2020-02-06T00:42:03.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:03.941+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.941995+0000) 2020-02-06T00:42:03.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:03.942+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.943164+0000) 2020-02-06T00:42:03.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:03.942+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.943462+0000) 2020-02-06T00:42:03.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:03.942+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.943953+0000) 2020-02-06T00:42:03.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:03.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.944384+0000) 2020-02-06T00:42:03.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:03.944+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.945786+0000) 2020-02-06T00:42:03.949 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:03.946+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.946689+0000) 2020-02-06T00:42:03.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:03.945+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.946660+0000) 2020-02-06T00:42:03.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:03.946+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.947534+0000) 2020-02-06T00:42:03.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:03.946+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.947229+0000) 2020-02-06T00:42:03.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:03.947+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.947659+0000) 2020-02-06T00:42:03.950 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:03.947+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.947999+0000) 2020-02-06T00:42:03.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:03.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:03.950211+0000) 2020-02-06T00:42:03.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:03.962+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:04.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:04.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:04.046161+0000) 2020-02-06T00:42:04.047 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:04.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:04.046334+0000) 2020-02-06T00:42:06.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:06.035+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:06.035477+0000) 2020-02-06T00:42:06.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:06.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:06.381303+0000) 2020-02-06T00:42:07.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:07.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:07.022295+0000) 2020-02-06T00:42:07.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:07.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:07.288578+0000) 2020-02-06T00:42:07.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:07.288+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:07.288649+0000) 2020-02-06T00:42:07.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:07.288+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:07.288649+0000) 2020-02-06T00:42:08.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:08.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.968383+0000) 2020-02-06T00:42:08.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:08.968+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.968861+0000) 2020-02-06T00:42:08.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:08.969+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.969933+0000) 2020-02-06T00:42:08.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:08.970+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.971301+0000) 2020-02-06T00:42:08.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:08.970+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.971843+0000) 2020-02-06T00:42:08.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:08.971+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.972297+0000) 2020-02-06T00:42:08.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:08.971+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.972615+0000) 2020-02-06T00:42:08.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:08.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.973031+0000) 2020-02-06T00:42:08.974 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:08.973+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.973840+0000) 2020-02-06T00:42:08.975 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:08.973+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.974216+0000) 2020-02-06T00:42:08.975 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:08.973+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.974399+0000) 2020-02-06T00:42:08.975 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:08.974+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.974964+0000) 2020-02-06T00:42:08.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:08.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.979072+0000) 2020-02-06T00:42:08.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:08.978+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:08.979339+0000) 2020-02-06T00:42:08.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:08.988+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:09.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:09.665+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:09.666432+0000) 2020-02-06T00:42:09.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:09.914+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:09.914888+0000) 2020-02-06T00:42:10.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:10.516+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:10.516699+0000) 2020-02-06T00:42:10.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:10.774+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:10.775128+0000) 2020-02-06T00:42:12.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:12.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:12.898270+0000) 2020-02-06T00:42:13.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:13.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.820138+0000) 2020-02-06T00:42:13.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:13.994+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.994664+0000) 2020-02-06T00:42:13.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:13.995+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.995733+0000) 2020-02-06T00:42:13.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:13.996+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.996658+0000) 2020-02-06T00:42:13.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:13.996+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.997436+0000) 2020-02-06T00:42:13.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:13.997+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.997933+0000) 2020-02-06T00:42:13.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:13.997+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.998311+0000) 2020-02-06T00:42:13.999 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:13.997+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.998952+0000) 2020-02-06T00:42:14.000 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:13.998+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.999466+0000) 2020-02-06T00:42:14.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:13.999+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:13.999870+0000) 2020-02-06T00:42:14.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:13.998+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.000065+0000) 2020-02-06T00:42:14.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:13.999+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.000562+0000) 2020-02-06T00:42:14.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:14.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.001245+0000) 2020-02-06T00:42:14.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:14.001+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.001568+0000) 2020-02-06T00:42:14.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:14.006+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.007101+0000) 2020-02-06T00:42:14.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:14.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:14.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:14.045+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.046340+0000) 2020-02-06T00:42:14.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:14.045+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:14.046484+0000) 2020-02-06T00:42:14.613 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:42:14.619 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:42:16.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:16.035+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:16.035749+0000) 2020-02-06T00:42:16.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:16.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:16.381535+0000) 2020-02-06T00:42:17.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:17.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:17.022537+0000) 2020-02-06T00:42:17.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:17.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:17.288820+0000) 2020-02-06T00:42:17.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:17.288+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:17.288933+0000) 2020-02-06T00:42:17.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:17.288+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:17.288933+0000) 2020-02-06T00:42:19.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:19.020+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.021865+0000) 2020-02-06T00:42:19.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:19.021+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.022386+0000) 2020-02-06T00:42:19.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:19.022+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.022851+0000) 2020-02-06T00:42:19.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:19.022+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.023940+0000) 2020-02-06T00:42:19.025 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:19.024+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.024724+0000) 2020-02-06T00:42:19.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:19.024+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.025382+0000) 2020-02-06T00:42:19.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:19.024+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.025627+0000) 2020-02-06T00:42:19.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:19.026+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.026622+0000) 2020-02-06T00:42:19.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:19.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.026671+0000) 2020-02-06T00:42:19.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:19.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.026962+0000) 2020-02-06T00:42:19.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:19.027+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.027676+0000) 2020-02-06T00:42:19.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:19.027+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.028111+0000) 2020-02-06T00:42:19.029 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:19.026+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.027536+0000) 2020-02-06T00:42:19.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:19.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.033756+0000) 2020-02-06T00:42:19.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:19.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:19.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:19.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.666667+0000) 2020-02-06T00:42:19.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:19.914+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:19.915140+0000) 2020-02-06T00:42:20.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:20.516+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:20.516922+0000) 2020-02-06T00:42:20.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:20.774+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:20.775309+0000) 2020-02-06T00:42:22.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:22.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:22.898510+0000) 2020-02-06T00:42:23.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:23.829+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:23.829683+0000) 2020-02-06T00:42:24.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:24.045+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.046376+0000) 2020-02-06T00:42:24.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:24.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.047534+0000) 2020-02-06T00:42:24.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:24.048+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.049081+0000) 2020-02-06T00:42:24.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:24.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.049560+0000) 2020-02-06T00:42:24.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:24.048+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.049969+0000) 2020-02-06T00:42:24.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:24.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.050346+0000) 2020-02-06T00:42:24.058 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:24.050+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.051222+0000) 2020-02-06T00:42:24.059 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:24.051+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.052311+0000) 2020-02-06T00:42:24.059 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:24.052+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.053164+0000) 2020-02-06T00:42:24.060 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:24.053+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.053575+0000) 2020-02-06T00:42:24.060 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:24.053+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.053557+0000) 2020-02-06T00:42:24.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:24.052+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.053686+0000) 2020-02-06T00:42:24.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:24.053+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.054036+0000) 2020-02-06T00:42:24.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:24.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:24.056840+0000) 2020-02-06T00:42:24.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:24.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:26.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:26.035+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:26.036026+0000) 2020-02-06T00:42:26.382 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:26.380+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:26.381770+0000) 2020-02-06T00:42:27.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:27.021+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:27.022818+0000) 2020-02-06T00:42:27.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:27.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:27.289063+0000) 2020-02-06T00:42:27.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:27.288+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:27.289211+0000) 2020-02-06T00:42:27.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:27.288+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:27.289224+0000) 2020-02-06T00:42:29.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:29.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.076848+0000) 2020-02-06T00:42:29.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:29.077+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.078063+0000) 2020-02-06T00:42:29.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:29.077+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.078492+0000) 2020-02-06T00:42:29.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:29.078+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.078790+0000) 2020-02-06T00:42:29.080 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:29.078+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.079330+0000) 2020-02-06T00:42:29.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:29.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.079501+0000) 2020-02-06T00:42:29.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:29.079+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.080267+0000) 2020-02-06T00:42:29.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:29.080+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.080417+0000) 2020-02-06T00:42:29.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:29.079+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.080654+0000) 2020-02-06T00:42:29.082 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:29.079+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.080888+0000) 2020-02-06T00:42:29.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:29.080+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.081129+0000) 2020-02-06T00:42:29.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:29.081+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.081798+0000) 2020-02-06T00:42:29.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:29.081+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.081993+0000) 2020-02-06T00:42:29.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:29.082+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.084061+0000) 2020-02-06T00:42:29.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:29.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:29.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:29.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.666879+0000) 2020-02-06T00:42:29.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:29.914+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:29.915359+0000) 2020-02-06T00:42:30.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:30.517+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:30.517179+0000) 2020-02-06T00:42:30.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:30.775+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:30.775550+0000) 2020-02-06T00:42:30.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:30.796+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:30.797288+0000) 2020-02-06T00:42:30.798 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:30.796+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:30.797419+0000) 2020-02-06T00:42:32.899 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:32.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:32.898766+0000) 2020-02-06T00:42:33.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:33.838+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:33.838997+0000) 2020-02-06T00:42:34.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:34.102+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.103033+0000) 2020-02-06T00:42:34.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:34.102+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.103584+0000) 2020-02-06T00:42:34.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:34.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.104165+0000) 2020-02-06T00:42:34.105 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:34.103+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.104418+0000) 2020-02-06T00:42:34.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:34.103+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.104653+0000) 2020-02-06T00:42:34.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:34.103+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.105057+0000) 2020-02-06T00:42:34.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:34.104+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.105561+0000) 2020-02-06T00:42:34.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:34.105+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.106220+0000) 2020-02-06T00:42:34.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:34.105+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.106222+0000) 2020-02-06T00:42:34.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:34.106+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.107159+0000) 2020-02-06T00:42:34.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:34.106+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.106618+0000) 2020-02-06T00:42:34.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:34.107+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.107882+0000) 2020-02-06T00:42:34.109 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:34.107+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.108226+0000) 2020-02-06T00:42:34.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:34.110+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:34.111402+0000) 2020-02-06T00:42:34.125 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:34.123+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:36.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:36.036+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:36.036294+0000) 2020-02-06T00:42:36.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:36.381+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:36.382066+0000) 2020-02-06T00:42:37.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:37.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:37.023046+0000) 2020-02-06T00:42:37.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:37.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:37.289311+0000) 2020-02-06T00:42:37.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:37.289+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:37.289491+0000) 2020-02-06T00:42:37.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:37.289+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:37.289504+0000) 2020-02-06T00:42:39.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:39.127+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.128993+0000) 2020-02-06T00:42:39.131 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:39.128+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.129449+0000) 2020-02-06T00:42:39.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:39.130+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.130599+0000) 2020-02-06T00:42:39.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:39.130+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.130648+0000) 2020-02-06T00:42:39.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:39.130+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.131065+0000) 2020-02-06T00:42:39.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:39.130+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.131597+0000) 2020-02-06T00:42:39.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:39.131+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.132326+0000) 2020-02-06T00:42:39.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:39.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.133039+0000) 2020-02-06T00:42:39.134 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:39.132+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.133706+0000) 2020-02-06T00:42:39.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:39.133+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.134818+0000) 2020-02-06T00:42:39.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:39.134+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.134815+0000) 2020-02-06T00:42:39.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:39.135+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.135478+0000) 2020-02-06T00:42:39.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:39.135+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.135640+0000) 2020-02-06T00:42:39.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:39.139+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.140422+0000) 2020-02-06T00:42:39.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:39.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:39.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:39.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.667136+0000) 2020-02-06T00:42:39.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:39.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:39.915548+0000) 2020-02-06T00:42:40.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:40.517+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:40.517465+0000) 2020-02-06T00:42:40.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:40.774+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:40.775777+0000) 2020-02-06T00:42:40.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:40.797+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:40.797500+0000) 2020-02-06T00:42:40.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:40.797+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:40.797649+0000) 2020-02-06T00:42:42.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:42.897+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:42.899024+0000) 2020-02-06T00:42:43.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:43.846+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:43.847275+0000) 2020-02-06T00:42:44.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:44.156+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.157629+0000) 2020-02-06T00:42:44.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:44.157+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.158341+0000) 2020-02-06T00:42:44.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:44.157+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.158997+0000) 2020-02-06T00:42:44.160 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:44.158+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.159484+0000) 2020-02-06T00:42:44.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:44.158+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.160092+0000) 2020-02-06T00:42:44.161 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:44.159+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.160410+0000) 2020-02-06T00:42:44.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:44.160+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.161954+0000) 2020-02-06T00:42:44.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:44.161+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.162165+0000) 2020-02-06T00:42:44.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:44.160+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.161987+0000) 2020-02-06T00:42:44.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:44.161+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.162691+0000) 2020-02-06T00:42:44.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:44.162+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.163224+0000) 2020-02-06T00:42:44.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:44.161+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.163095+0000) 2020-02-06T00:42:44.167 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:44.162+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.163901+0000) 2020-02-06T00:42:44.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:44.165+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:44.166382+0000) 2020-02-06T00:42:44.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:44.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 181980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:44.801 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:42:44.807 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:42:46.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:46.035+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:46.036545+0000) 2020-02-06T00:42:46.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:46.382+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:46.382381+0000) 2020-02-06T00:42:47.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:47.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:47.023289+0000) 2020-02-06T00:42:47.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:47.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:47.289545+0000) 2020-02-06T00:42:47.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:47.288+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:47.289795+0000) 2020-02-06T00:42:47.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:47.288+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:47.289790+0000) 2020-02-06T00:42:49.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:49.181+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.181953+0000) 2020-02-06T00:42:49.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:49.182+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.182538+0000) 2020-02-06T00:42:49.184 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:49.181+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.183332+0000) 2020-02-06T00:42:49.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:49.183+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.184034+0000) 2020-02-06T00:42:49.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:49.184+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.184542+0000) 2020-02-06T00:42:49.186 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:49.183+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.185100+0000) 2020-02-06T00:42:49.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:49.185+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.185393+0000) 2020-02-06T00:42:49.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:49.184+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.186189+0000) 2020-02-06T00:42:49.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:49.186+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.186314+0000) 2020-02-06T00:42:49.188 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:49.185+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.186884+0000) 2020-02-06T00:42:49.189 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:49.188+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.188250+0000) 2020-02-06T00:42:49.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:49.187+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.188556+0000) 2020-02-06T00:42:49.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:49.187+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.188723+0000) 2020-02-06T00:42:49.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:49.192+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.192845+0000) 2020-02-06T00:42:49.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:49.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:49.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:49.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.667345+0000) 2020-02-06T00:42:49.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:49.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:49.915752+0000) 2020-02-06T00:42:50.518 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:50.516+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:50.517723+0000) 2020-02-06T00:42:50.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:50.775+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:50.776043+0000) 2020-02-06T00:42:50.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:50.797+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:50.797770+0000) 2020-02-06T00:42:50.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:50.797+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:50.797922+0000) 2020-02-06T00:42:52.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:52.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:52.899281+0000) 2020-02-06T00:42:53.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:53.854+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:53.855530+0000) 2020-02-06T00:42:54.210 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:54.207+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.208288+0000) 2020-02-06T00:42:54.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:54.208+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.209846+0000) 2020-02-06T00:42:54.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:54.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.210426+0000) 2020-02-06T00:42:54.212 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:54.209+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.210655+0000) 2020-02-06T00:42:54.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:54.210+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.211116+0000) 2020-02-06T00:42:54.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:54.210+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.211515+0000) 2020-02-06T00:42:54.213 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:54.210+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.212328+0000) 2020-02-06T00:42:54.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:54.211+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.212780+0000) 2020-02-06T00:42:54.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:54.212+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.213733+0000) 2020-02-06T00:42:54.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:54.212+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.213939+0000) 2020-02-06T00:42:54.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:54.212+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.214055+0000) 2020-02-06T00:42:54.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:54.214+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.215656+0000) 2020-02-06T00:42:54.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:54.214+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.216142+0000) 2020-02-06T00:42:54.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:54.218+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:54.220018+0000) 2020-02-06T00:42:54.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:54.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:56.037 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:56.035+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:56.036795+0000) 2020-02-06T00:42:56.383 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:56.381+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:56.382643+0000) 2020-02-06T00:42:57.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:57.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:57.023537+0000) 2020-02-06T00:42:57.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:57.288+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:57.289776+0000) 2020-02-06T00:42:57.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:57.288+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:57.290010+0000) 2020-02-06T00:42:57.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:57.288+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:57.290043+0000) 2020-02-06T00:42:59.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:59.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.233977+0000) 2020-02-06T00:42:59.236 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:42:59.234+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.235561+0000) 2020-02-06T00:42:59.237 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:42:59.235+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.236630+0000) 2020-02-06T00:42:59.238 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:42:59.236+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.237513+0000) 2020-02-06T00:42:59.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:42:59.237+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.238762+0000) 2020-02-06T00:42:59.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:42:59.237+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.239082+0000) 2020-02-06T00:42:59.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:42:59.238+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.240047+0000) 2020-02-06T00:42:59.241 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:42:59.239+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.240147+0000) 2020-02-06T00:42:59.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:42:59.239+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.240658+0000) 2020-02-06T00:42:59.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:59.240+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.241184+0000) 2020-02-06T00:42:59.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:42:59.239+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.241077+0000) 2020-02-06T00:42:59.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:42:59.240+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.241627+0000) 2020-02-06T00:42:59.243 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:42:59.240+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.241957+0000) 2020-02-06T00:42:59.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:59.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.243922+0000) 2020-02-06T00:42:59.255 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:42:59.253+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182103 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:42:59.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:59.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.667551+0000) 2020-02-06T00:42:59.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:42:59.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:42:59.915967+0000) 2020-02-06T00:43:00.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:00.517+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:00.517972+0000) 2020-02-06T00:43:00.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:00.775+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:00.776303+0000) 2020-02-06T00:43:00.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:00.797+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:00.798001+0000) 2020-02-06T00:43:00.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:00.797+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:00.798196+0000) 2020-02-06T00:43:02.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:02.898+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:02.899534+0000) 2020-02-06T00:43:03.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:03.862+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:03.863319+0000) 2020-02-06T00:43:04.261 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:04.259+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.260318+0000) 2020-02-06T00:43:04.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:04.259+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.260541+0000) 2020-02-06T00:43:04.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:04.260+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.261296+0000) 2020-02-06T00:43:04.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:04.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.262490+0000) 2020-02-06T00:43:04.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:04.261+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.262810+0000) 2020-02-06T00:43:04.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:04.262+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.263117+0000) 2020-02-06T00:43:04.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:04.262+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.263800+0000) 2020-02-06T00:43:04.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:04.262+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.263779+0000) 2020-02-06T00:43:04.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:04.263+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.264132+0000) 2020-02-06T00:43:04.265 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:04.262+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.264373+0000) 2020-02-06T00:43:04.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:04.263+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.264798+0000) 2020-02-06T00:43:04.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:04.263+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.265258+0000) 2020-02-06T00:43:04.267 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:04.264+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.265763+0000) 2020-02-06T00:43:04.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:04.269+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:04.270869+0000) 2020-02-06T00:43:04.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:04.279+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:06.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:06.036+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:06.037066+0000) 2020-02-06T00:43:06.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:06.382+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:06.382945+0000) 2020-02-06T00:43:07.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:07.022+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:07.023767+0000) 2020-02-06T00:43:07.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:07.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:07.290026+0000) 2020-02-06T00:43:07.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:07.289+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:07.290301+0000) 2020-02-06T00:43:07.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:07.289+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:07.290301+0000) 2020-02-06T00:43:09.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:09.285+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.286404+0000) 2020-02-06T00:43:09.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:09.286+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.287589+0000) 2020-02-06T00:43:09.289 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:09.287+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.288414+0000) 2020-02-06T00:43:09.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:09.287+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.288892+0000) 2020-02-06T00:43:09.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:09.288+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.289168+0000) 2020-02-06T00:43:09.290 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:09.288+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.289645+0000) 2020-02-06T00:43:09.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:09.289+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.290142+0000) 2020-02-06T00:43:09.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:09.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.290759+0000) 2020-02-06T00:43:09.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:09.289+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.290944+0000) 2020-02-06T00:43:09.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:09.290+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.291381+0000) 2020-02-06T00:43:09.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:09.289+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.291251+0000) 2020-02-06T00:43:09.293 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:09.290+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.291881+0000) 2020-02-06T00:43:09.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:09.291+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.292675+0000) 2020-02-06T00:43:09.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:09.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.295942+0000) 2020-02-06T00:43:09.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:09.305+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:09.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:09.666+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.667742+0000) 2020-02-06T00:43:09.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:09.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:09.916140+0000) 2020-02-06T00:43:10.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:10.516+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:10.518228+0000) 2020-02-06T00:43:10.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:10.775+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:10.776533+0000) 2020-02-06T00:43:10.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:10.797+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:10.798272+0000) 2020-02-06T00:43:10.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:10.797+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:10.798441+0000) 2020-02-06T00:43:12.900 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:12.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:12.899803+0000) 2020-02-06T00:43:13.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:13.872+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:13.873122+0000) 2020-02-06T00:43:14.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:14.311+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.312245+0000) 2020-02-06T00:43:14.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:14.311+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.313144+0000) 2020-02-06T00:43:14.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:14.312+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.313197+0000) 2020-02-06T00:43:14.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:14.312+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.314013+0000) 2020-02-06T00:43:14.316 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:14.313+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.314245+0000) 2020-02-06T00:43:14.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:14.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.314709+0000) 2020-02-06T00:43:14.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:14.313+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.315154+0000) 2020-02-06T00:43:14.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:14.314+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.315757+0000) 2020-02-06T00:43:14.317 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:14.315+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.316199+0000) 2020-02-06T00:43:14.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:14.315+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.316654+0000) 2020-02-06T00:43:14.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:14.316+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.317739+0000) 2020-02-06T00:43:14.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:14.316+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.317479+0000) 2020-02-06T00:43:14.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:14.316+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.318079+0000) 2020-02-06T00:43:14.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:14.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:14.323086+0000) 2020-02-06T00:43:14.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:14.332+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182227 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:14.981 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:43:14.987 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:43:16.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:16.036+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:16.037286+0000) 2020-02-06T00:43:16.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:16.382+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:16.383203+0000) 2020-02-06T00:43:17.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:17.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:17.024007+0000) 2020-02-06T00:43:17.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:17.289+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:17.290272+0000) 2020-02-06T00:43:17.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:17.289+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:17.290593+0000) 2020-02-06T00:43:17.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:17.289+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:17.290643+0000) 2020-02-06T00:43:19.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:19.336+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.337910+0000) 2020-02-06T00:43:19.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:19.337+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.338719+0000) 2020-02-06T00:43:19.341 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:19.339+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.340612+0000) 2020-02-06T00:43:19.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:19.339+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.341256+0000) 2020-02-06T00:43:19.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:19.340+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.341476+0000) 2020-02-06T00:43:19.343 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:19.340+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.341911+0000) 2020-02-06T00:43:19.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:19.341+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.342569+0000) 2020-02-06T00:43:19.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:19.341+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.342791+0000) 2020-02-06T00:43:19.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:19.342+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.343416+0000) 2020-02-06T00:43:19.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:19.343+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.344731+0000) 2020-02-06T00:43:19.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:19.343+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.345149+0000) 2020-02-06T00:43:19.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:19.344+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.345448+0000) 2020-02-06T00:43:19.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:19.344+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.345705+0000) 2020-02-06T00:43:19.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:19.350+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.351717+0000) 2020-02-06T00:43:19.362 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:19.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:19.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:19.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.667951+0000) 2020-02-06T00:43:19.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:19.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:19.916359+0000) 2020-02-06T00:43:20.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:20.517+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:20.518492+0000) 2020-02-06T00:43:20.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:20.776+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:20.776765+0000) 2020-02-06T00:43:20.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:20.797+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:20.798508+0000) 2020-02-06T00:43:20.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:20.797+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:20.798685+0000) 2020-02-06T00:43:22.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:22.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:22.900047+0000) 2020-02-06T00:43:23.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:23.880+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:23.881327+0000) 2020-02-06T00:43:24.368 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:24.365+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.367333+0000) 2020-02-06T00:43:24.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:24.366+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.367713+0000) 2020-02-06T00:43:24.369 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:24.367+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.368350+0000) 2020-02-06T00:43:24.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:24.367+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.368826+0000) 2020-02-06T00:43:24.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:24.367+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.369356+0000) 2020-02-06T00:43:24.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:24.369+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.370137+0000) 2020-02-06T00:43:24.371 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:24.368+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.370243+0000) 2020-02-06T00:43:24.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:24.369+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.370690+0000) 2020-02-06T00:43:24.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:24.370+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.371536+0000) 2020-02-06T00:43:24.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:24.370+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.371900+0000) 2020-02-06T00:43:24.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:24.371+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.372591+0000) 2020-02-06T00:43:24.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:24.371+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.372322+0000) 2020-02-06T00:43:24.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:24.371+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.373224+0000) 2020-02-06T00:43:24.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:24.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:24.377604+0000) 2020-02-06T00:43:24.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:24.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:26.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:26.037+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:26.037645+0000) 2020-02-06T00:43:26.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:26.383+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:26.383436+0000) 2020-02-06T00:43:27.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:27.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:27.024335+0000) 2020-02-06T00:43:27.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:27.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:27.290535+0000) 2020-02-06T00:43:27.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:27.290+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:27.290844+0000) 2020-02-06T00:43:27.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:27.290+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:27.291018+0000) 2020-02-06T00:43:29.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:29.393+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.394473+0000) 2020-02-06T00:43:29.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:29.394+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.395360+0000) 2020-02-06T00:43:29.397 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:29.394+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.395794+0000) 2020-02-06T00:43:29.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:29.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.395658+0000) 2020-02-06T00:43:29.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:29.394+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.396029+0000) 2020-02-06T00:43:29.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:29.395+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.396637+0000) 2020-02-06T00:43:29.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:29.395+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.396337+0000) 2020-02-06T00:43:29.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:29.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.396918+0000) 2020-02-06T00:43:29.399 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:29.395+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.397049+0000) 2020-02-06T00:43:29.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:29.396+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.397420+0000) 2020-02-06T00:43:29.400 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:29.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.397759+0000) 2020-02-06T00:43:29.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:29.396+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.397972+0000) 2020-02-06T00:43:29.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:29.396+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.398330+0000) 2020-02-06T00:43:29.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:29.402+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.403918+0000) 2020-02-06T00:43:29.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:29.412+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:29.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:29.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.668164+0000) 2020-02-06T00:43:29.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:29.916+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:29.916563+0000) 2020-02-06T00:43:30.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:30.518+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:30.518746+0000) 2020-02-06T00:43:30.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:30.776+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:30.777058+0000) 2020-02-06T00:43:30.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:30.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:30.798738+0000) 2020-02-06T00:43:30.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:30.798+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:30.798920+0000) 2020-02-06T00:43:32.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:32.899+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:32.900289+0000) 2020-02-06T00:43:33.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:33.889+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:33.889559+0000) 2020-02-06T00:43:34.420 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:34.418+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.419334+0000) 2020-02-06T00:43:34.421 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:34.419+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.420282+0000) 2020-02-06T00:43:34.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:34.419+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.420481+0000) 2020-02-06T00:43:34.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:34.420+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.421070+0000) 2020-02-06T00:43:34.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:34.421+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.422588+0000) 2020-02-06T00:43:34.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:34.423+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.423716+0000) 2020-02-06T00:43:34.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:34.423+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.423583+0000) 2020-02-06T00:43:34.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:34.423+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.424216+0000) 2020-02-06T00:43:34.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:34.423+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.424381+0000) 2020-02-06T00:43:34.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:34.424+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.424473+0000) 2020-02-06T00:43:34.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:34.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.425166+0000) 2020-02-06T00:43:34.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:34.424+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.425351+0000) 2020-02-06T00:43:34.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:34.424+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.425567+0000) 2020-02-06T00:43:34.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:34.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:34.427850+0000) 2020-02-06T00:43:34.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:34.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:36.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:36.037+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:36.037928+0000) 2020-02-06T00:43:36.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:36.383+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:36.383665+0000) 2020-02-06T00:43:37.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:37.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:37.024607+0000) 2020-02-06T00:43:37.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:37.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:37.290737+0000) 2020-02-06T00:43:37.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:37.290+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:37.291078+0000) 2020-02-06T00:43:37.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:37.290+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:37.291253+0000) 2020-02-06T00:43:39.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:39.444+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.445288+0000) 2020-02-06T00:43:39.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:39.445+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.446187+0000) 2020-02-06T00:43:39.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:39.446+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.447417+0000) 2020-02-06T00:43:39.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:39.447+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.447487+0000) 2020-02-06T00:43:39.449 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:39.446+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.448042+0000) 2020-02-06T00:43:39.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:39.447+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.448453+0000) 2020-02-06T00:43:39.450 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:39.447+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.448982+0000) 2020-02-06T00:43:39.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:39.448+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.448950+0000) 2020-02-06T00:43:39.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:39.448+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.449954+0000) 2020-02-06T00:43:39.451 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:39.449+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.450346+0000) 2020-02-06T00:43:39.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:39.450+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.450883+0000) 2020-02-06T00:43:39.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:39.450+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.451289+0000) 2020-02-06T00:43:39.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:39.451+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.451443+0000) 2020-02-06T00:43:39.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:39.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.453081+0000) 2020-02-06T00:43:39.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:39.464+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:39.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:39.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.668369+0000) 2020-02-06T00:43:39.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:39.916+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:39.916793+0000) 2020-02-06T00:43:40.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:40.518+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:40.518997+0000) 2020-02-06T00:43:40.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:40.777+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:40.777366+0000) 2020-02-06T00:43:40.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:40.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:40.798966+0000) 2020-02-06T00:43:40.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:40.798+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:40.799142+0000) 2020-02-06T00:43:42.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:42.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:42.900593+0000) 2020-02-06T00:43:43.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:43.897+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:43.897930+0000) 2020-02-06T00:43:44.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:44.470+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.470554+0000) 2020-02-06T00:43:44.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:44.471+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.471640+0000) 2020-02-06T00:43:44.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:44.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.471992+0000) 2020-02-06T00:43:44.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:44.472+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.472966+0000) 2020-02-06T00:43:44.474 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:44.473+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.473463+0000) 2020-02-06T00:43:44.474 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:44.472+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.473337+0000) 2020-02-06T00:43:44.475 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:44.473+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.474048+0000) 2020-02-06T00:43:44.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:44.474+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.474595+0000) 2020-02-06T00:43:44.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:44.475+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.475665+0000) 2020-02-06T00:43:44.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:44.476+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.476331+0000) 2020-02-06T00:43:44.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:44.476+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.476410+0000) 2020-02-06T00:43:44.478 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:44.476+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.476856+0000) 2020-02-06T00:43:44.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:44.476+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.477046+0000) 2020-02-06T00:43:44.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:44.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:44.482164+0000) 2020-02-06T00:43:44.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:44.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:45.144 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:43:45.150 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:43:46.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:46.037+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:46.038194+0000) 2020-02-06T00:43:46.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:46.383+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:46.383921+0000) 2020-02-06T00:43:47.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:47.023+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:47.024853+0000) 2020-02-06T00:43:47.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:47.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:47.291008+0000) 2020-02-06T00:43:47.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:47.290+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:47.291308+0000) 2020-02-06T00:43:47.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:47.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:47.291536+0000) 2020-02-06T00:43:49.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:49.495+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.496331+0000) 2020-02-06T00:43:49.498 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:49.495+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.497032+0000) 2020-02-06T00:43:49.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:49.496+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.497059+0000) 2020-02-06T00:43:49.499 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:49.496+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.497317+0000) 2020-02-06T00:43:49.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:49.496+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.497838+0000) 2020-02-06T00:43:49.499 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:49.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.498067+0000) 2020-02-06T00:43:49.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:49.497+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.498486+0000) 2020-02-06T00:43:49.500 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:49.497+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.498563+0000) 2020-02-06T00:43:49.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:49.499+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.499603+0000) 2020-02-06T00:43:49.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:49.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.499756+0000) 2020-02-06T00:43:49.501 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:49.499+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.500186+0000) 2020-02-06T00:43:49.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:49.500+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.500745+0000) 2020-02-06T00:43:49.502 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:49.500+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.501332+0000) 2020-02-06T00:43:49.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:49.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.504866+0000) 2020-02-06T00:43:49.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:49.515+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:49.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:49.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.668579+0000) 2020-02-06T00:43:49.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:49.915+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:49.917036+0000) 2020-02-06T00:43:50.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:50.518+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:50.519232+0000) 2020-02-06T00:43:50.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:50.776+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:50.777598+0000) 2020-02-06T00:43:50.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:50.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:50.799194+0000) 2020-02-06T00:43:50.801 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:50.798+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:50.799344+0000) 2020-02-06T00:43:52.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:52.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:52.900856+0000) 2020-02-06T00:43:53.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:53.905+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:53.906281+0000) 2020-02-06T00:43:54.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:54.519+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.520860+0000) 2020-02-06T00:43:54.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:54.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.521815+0000) 2020-02-06T00:43:54.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:54.521+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.522324+0000) 2020-02-06T00:43:54.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:54.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.523134+0000) 2020-02-06T00:43:54.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:54.523+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.523944+0000) 2020-02-06T00:43:54.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:54.523+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.524397+0000) 2020-02-06T00:43:54.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:54.525+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.525861+0000) 2020-02-06T00:43:54.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:54.525+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.526298+0000) 2020-02-06T00:43:54.527 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:54.525+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.525594+0000) 2020-02-06T00:43:54.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:54.525+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.526886+0000) 2020-02-06T00:43:54.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:54.527+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.527755+0000) 2020-02-06T00:43:54.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:54.527+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.528153+0000) 2020-02-06T00:43:54.529 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:54.528+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.528507+0000) 2020-02-06T00:43:54.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:54.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:54.532673+0000) 2020-02-06T00:43:54.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:54.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:56.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:56.038+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:56.038480+0000) 2020-02-06T00:43:56.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:56.383+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:56.384152+0000) 2020-02-06T00:43:57.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:57.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:57.025114+0000) 2020-02-06T00:43:57.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:57.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:57.291259+0000) 2020-02-06T00:43:57.293 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:57.291+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:57.291571+0000) 2020-02-06T00:43:57.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:57.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:57.291782+0000) 2020-02-06T00:43:59.547 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:43:59.545+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.546828+0000) 2020-02-06T00:43:59.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:43:59.547+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.548854+0000) 2020-02-06T00:43:59.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:43:59.549+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.549604+0000) 2020-02-06T00:43:59.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:59.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.549889+0000) 2020-02-06T00:43:59.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:43:59.549+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.550282+0000) 2020-02-06T00:43:59.551 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:43:59.549+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.550854+0000) 2020-02-06T00:43:59.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:43:59.550+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.551201+0000) 2020-02-06T00:43:59.552 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:43:59.550+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.551868+0000) 2020-02-06T00:43:59.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:59.551+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.552858+0000) 2020-02-06T00:43:59.554 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:43:59.552+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.553113+0000) 2020-02-06T00:43:59.554 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:43:59.553+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.553762+0000) 2020-02-06T00:43:59.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:43:59.553+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.554319+0000) 2020-02-06T00:43:59.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:43:59.554+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.554589+0000) 2020-02-06T00:43:59.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:59.558+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.559286+0000) 2020-02-06T00:43:59.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:43:59.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182592 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:43:59.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:59.667+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.668807+0000) 2020-02-06T00:43:59.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:43:59.916+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:43:59.917214+0000) 2020-02-06T00:44:00.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:00.519+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:00.519474+0000) 2020-02-06T00:44:00.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:00.777+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:00.777933+0000) 2020-02-06T00:44:00.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:00.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:00.799424+0000) 2020-02-06T00:44:00.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:00.798+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:00.799608+0000) 2020-02-06T00:44:02.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:02.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:02.901127+0000) 2020-02-06T00:44:03.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:03.913+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:03.914319+0000) 2020-02-06T00:44:04.576 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:04.573+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.574855+0000) 2020-02-06T00:44:04.576 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:04.574+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.575471+0000) 2020-02-06T00:44:04.576 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:04.574+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.575857+0000) 2020-02-06T00:44:04.577 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:04.575+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.576394+0000) 2020-02-06T00:44:04.578 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:04.575+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.576876+0000) 2020-02-06T00:44:04.578 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:04.576+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.576703+0000) 2020-02-06T00:44:04.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:04.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.578005+0000) 2020-02-06T00:44:04.581 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:04.578+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.579013+0000) 2020-02-06T00:44:04.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:04.579+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.579723+0000) 2020-02-06T00:44:04.582 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:04.579+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.580043+0000) 2020-02-06T00:44:04.582 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:04.580+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.580499+0000) 2020-02-06T00:44:04.582 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:04.579+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.580183+0000) 2020-02-06T00:44:04.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:04.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.586064+0000) 2020-02-06T00:44:04.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:04.586+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:04.587225+0000) 2020-02-06T00:44:04.596 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:04.595+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:06.039 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:06.038+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:06.038781+0000) 2020-02-06T00:44:06.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:06.383+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:06.384400+0000) 2020-02-06T00:44:07.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:07.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:07.025354+0000) 2020-02-06T00:44:07.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:07.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:07.291515+0000) 2020-02-06T00:44:07.293 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:07.291+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:07.291815+0000) 2020-02-06T00:44:07.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:07.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:07.292050+0000) 2020-02-06T00:44:09.602 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:09.600+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.601574+0000) 2020-02-06T00:44:09.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:09.600+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.602288+0000) 2020-02-06T00:44:09.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:09.601+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.602871+0000) 2020-02-06T00:44:09.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:09.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.603240+0000) 2020-02-06T00:44:09.604 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:09.601+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.603406+0000) 2020-02-06T00:44:09.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:09.603+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.604262+0000) 2020-02-06T00:44:09.605 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:09.603+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.604470+0000) 2020-02-06T00:44:09.605 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:09.603+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.604798+0000) 2020-02-06T00:44:09.606 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:09.604+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.605508+0000) 2020-02-06T00:44:09.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:09.603+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.605171+0000) 2020-02-06T00:44:09.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:09.604+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.606208+0000) 2020-02-06T00:44:09.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:09.605+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.606647+0000) 2020-02-06T00:44:09.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:09.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.607059+0000) 2020-02-06T00:44:09.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:09.608+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.609340+0000) 2020-02-06T00:44:09.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:09.618+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182673 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:09.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:09.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.669028+0000) 2020-02-06T00:44:09.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:09.916+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:09.917516+0000) 2020-02-06T00:44:10.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:10.518+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:10.519760+0000) 2020-02-06T00:44:10.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:10.777+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:10.778165+0000) 2020-02-06T00:44:10.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:10.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:10.799674+0000) 2020-02-06T00:44:10.801 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:10.798+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:10.799857+0000) 2020-02-06T00:44:12.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:12.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:12.901361+0000) 2020-02-06T00:44:13.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:13.922+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:13.922839+0000) 2020-02-06T00:44:14.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:14.623+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.624407+0000) 2020-02-06T00:44:14.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:14.624+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.625529+0000) 2020-02-06T00:44:14.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:14.624+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.626039+0000) 2020-02-06T00:44:14.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:14.624+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.626266+0000) 2020-02-06T00:44:14.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:14.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.627189+0000) 2020-02-06T00:44:14.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:14.626+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.628396+0000) 2020-02-06T00:44:14.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:14.628+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.629256+0000) 2020-02-06T00:44:14.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:14.629+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.630196+0000) 2020-02-06T00:44:14.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:14.628+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.630295+0000) 2020-02-06T00:44:14.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:14.629+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.631113+0000) 2020-02-06T00:44:14.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:14.630+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.631507+0000) 2020-02-06T00:44:14.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:14.631+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.632655+0000) 2020-02-06T00:44:14.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:14.632+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.633386+0000) 2020-02-06T00:44:14.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:14.634+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:14.635140+0000) 2020-02-06T00:44:14.649 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:14.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182714 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:15.358 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:44:15.374 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:44:16.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:16.037+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:16.039012+0000) 2020-02-06T00:44:16.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:16.384+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:16.384630+0000) 2020-02-06T00:44:17.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:17.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:17.025629+0000) 2020-02-06T00:44:17.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:17.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:17.291862+0000) 2020-02-06T00:44:17.293 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:17.290+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:17.292033+0000) 2020-02-06T00:44:17.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:17.290+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:17.292285+0000) 2020-02-06T00:44:19.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:19.653+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.654830+0000) 2020-02-06T00:44:19.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:19.654+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.655282+0000) 2020-02-06T00:44:19.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:19.654+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.655645+0000) 2020-02-06T00:44:19.657 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:19.654+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.655903+0000) 2020-02-06T00:44:19.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:19.655+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.656366+0000) 2020-02-06T00:44:19.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:19.655+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.656954+0000) 2020-02-06T00:44:19.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:19.656+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.657562+0000) 2020-02-06T00:44:19.660 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:19.658+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.659156+0000) 2020-02-06T00:44:19.660 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:19.657+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.659116+0000) 2020-02-06T00:44:19.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:19.658+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.659928+0000) 2020-02-06T00:44:19.661 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:19.658+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.659994+0000) 2020-02-06T00:44:19.661 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:19.658+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.660243+0000) 2020-02-06T00:44:19.661 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:19.659+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.660850+0000) 2020-02-06T00:44:19.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:19.662+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.663138+0000) 2020-02-06T00:44:19.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:19.668+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.669220+0000) 2020-02-06T00:44:19.678 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:19.674+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:19.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:19.917+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:19.917724+0000) 2020-02-06T00:44:20.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:20.518+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:20.519959+0000) 2020-02-06T00:44:20.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:20.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:20.778437+0000) 2020-02-06T00:44:20.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:20.798+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:20.799847+0000) 2020-02-06T00:44:20.801 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:20.799+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:20.800129+0000) 2020-02-06T00:44:22.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:22.900+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:22.901676+0000) 2020-02-06T00:44:23.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:23.934+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:23.935265+0000) 2020-02-06T00:44:24.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:24.680+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.681444+0000) 2020-02-06T00:44:24.683 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:24.680+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.681965+0000) 2020-02-06T00:44:24.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:24.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.682468+0000) 2020-02-06T00:44:24.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:24.682+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.683235+0000) 2020-02-06T00:44:24.684 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:24.682+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.683926+0000) 2020-02-06T00:44:24.685 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:24.682+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.683730+0000) 2020-02-06T00:44:24.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:24.683+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.684779+0000) 2020-02-06T00:44:24.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:24.683+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.685181+0000) 2020-02-06T00:44:24.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:24.684+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.686022+0000) 2020-02-06T00:44:24.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:24.685+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.686369+0000) 2020-02-06T00:44:24.687 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:24.684+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.685729+0000) 2020-02-06T00:44:24.688 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:24.685+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.687133+0000) 2020-02-06T00:44:24.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:24.685+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.687170+0000) 2020-02-06T00:44:24.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:24.688+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:24.689755+0000) 2020-02-06T00:44:24.702 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:24.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182794 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:26.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:26.038+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:26.039263+0000) 2020-02-06T00:44:26.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:26.384+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:26.384893+0000) 2020-02-06T00:44:27.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:27.024+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:27.025877+0000) 2020-02-06T00:44:27.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:27.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:27.292130+0000) 2020-02-06T00:44:27.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:27.291+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:27.292302+0000) 2020-02-06T00:44:27.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:27.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:27.292661+0000) 2020-02-06T00:44:29.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:29.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.669465+0000) 2020-02-06T00:44:29.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:29.707+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.708219+0000) 2020-02-06T00:44:29.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:29.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.709015+0000) 2020-02-06T00:44:29.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:29.708+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.709600+0000) 2020-02-06T00:44:29.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:29.709+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.710247+0000) 2020-02-06T00:44:29.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:29.709+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.710883+0000) 2020-02-06T00:44:29.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:29.709+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.711238+0000) 2020-02-06T00:44:29.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:29.710+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.711547+0000) 2020-02-06T00:44:29.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:29.710+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.712146+0000) 2020-02-06T00:44:29.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:29.710+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.712381+0000) 2020-02-06T00:44:29.714 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:29.711+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.712834+0000) 2020-02-06T00:44:29.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:29.712+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.713451+0000) 2020-02-06T00:44:29.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:29.712+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.714199+0000) 2020-02-06T00:44:29.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:29.713+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.714164+0000) 2020-02-06T00:44:29.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:29.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.718779+0000) 2020-02-06T00:44:29.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:29.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:29.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:29.916+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:29.917941+0000) 2020-02-06T00:44:30.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:30.519+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:30.520185+0000) 2020-02-06T00:44:30.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:30.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:30.778672+0000) 2020-02-06T00:44:30.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:30.799+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:30.800180+0000) 2020-02-06T00:44:30.801 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:30.799+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:30.800368+0000) 2020-02-06T00:44:32.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:32.901+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:32.902024+0000) 2020-02-06T00:44:33.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:33.946+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:33.946790+0000) 2020-02-06T00:44:34.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:34.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.732341+0000) 2020-02-06T00:44:34.734 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:34.732+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.733313+0000) 2020-02-06T00:44:34.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:34.734+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.735127+0000) 2020-02-06T00:44:34.736 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:34.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.734961+0000) 2020-02-06T00:44:34.737 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:34.734+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.735848+0000) 2020-02-06T00:44:34.737 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:34.735+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.736173+0000) 2020-02-06T00:44:34.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:34.735+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.736873+0000) 2020-02-06T00:44:34.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:34.735+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.736987+0000) 2020-02-06T00:44:34.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:34.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.737527+0000) 2020-02-06T00:44:34.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:34.736+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.738180+0000) 2020-02-06T00:44:34.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:34.737+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.738666+0000) 2020-02-06T00:44:34.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:34.737+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.738607+0000) 2020-02-06T00:44:34.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:34.737+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.739089+0000) 2020-02-06T00:44:34.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:34.742+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:34.743872+0000) 2020-02-06T00:44:34.756 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:34.753+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:36.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:36.038+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:36.039506+0000) 2020-02-06T00:44:36.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:36.384+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:36.385129+0000) 2020-02-06T00:44:37.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:37.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:37.026176+0000) 2020-02-06T00:44:37.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:37.290+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:37.292365+0000) 2020-02-06T00:44:37.294 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:37.291+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:37.292551+0000) 2020-02-06T00:44:37.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:37.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:37.292918+0000) 2020-02-06T00:44:39.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:39.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.669711+0000) 2020-02-06T00:44:39.761 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:39.759+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.759859+0000) 2020-02-06T00:44:39.762 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:39.761+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.761302+0000) 2020-02-06T00:44:39.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:39.761+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.761585+0000) 2020-02-06T00:44:39.763 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:39.760+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.761992+0000) 2020-02-06T00:44:39.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:39.761+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.762489+0000) 2020-02-06T00:44:39.764 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:39.763+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.763098+0000) 2020-02-06T00:44:39.764 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:39.763+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.763293+0000) 2020-02-06T00:44:39.764 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:39.761+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.762676+0000) 2020-02-06T00:44:39.765 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:39.764+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.764239+0000) 2020-02-06T00:44:39.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:39.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.764262+0000) 2020-02-06T00:44:39.766 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:39.763+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.764417+0000) 2020-02-06T00:44:39.766 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:39.763+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.764825+0000) 2020-02-06T00:44:39.766 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:39.764+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.765723+0000) 2020-02-06T00:44:39.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:39.767+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.768026+0000) 2020-02-06T00:44:39.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:39.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182918 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:39.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:39.918+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:39.918170+0000) 2020-02-06T00:44:40.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:40.519+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:40.520422+0000) 2020-02-06T00:44:40.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:40.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:40.778902+0000) 2020-02-06T00:44:40.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:40.800+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:40.800487+0000) 2020-02-06T00:44:40.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:40.800+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:40.800627+0000) 2020-02-06T00:44:42.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:42.901+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:42.902297+0000) 2020-02-06T00:44:43.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:43.953+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:43.954920+0000) 2020-02-06T00:44:44.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:44.785+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.786169+0000) 2020-02-06T00:44:44.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:44.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.786711+0000) 2020-02-06T00:44:44.789 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:44.786+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.788341+0000) 2020-02-06T00:44:44.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:44.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.788755+0000) 2020-02-06T00:44:44.790 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:44.787+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.789031+0000) 2020-02-06T00:44:44.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:44.788+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.789522+0000) 2020-02-06T00:44:44.791 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:44.788+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.789857+0000) 2020-02-06T00:44:44.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:44.789+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.790736+0000) 2020-02-06T00:44:44.792 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:44.789+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.791215+0000) 2020-02-06T00:44:44.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:44.790+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.791508+0000) 2020-02-06T00:44:44.793 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:44.791+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.792132+0000) 2020-02-06T00:44:44.793 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:44.790+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.791825+0000) 2020-02-06T00:44:44.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:44.794+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.795211+0000) 2020-02-06T00:44:44.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:44.794+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:44.795449+0000) 2020-02-06T00:44:44.807 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:44.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 182959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:45.538 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:44:45.544 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:44:46.040 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:46.038+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:46.039706+0000) 2020-02-06T00:44:46.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:46.384+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:46.385369+0000) 2020-02-06T00:44:47.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:47.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:47.026404+0000) 2020-02-06T00:44:47.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:47.291+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:47.292645+0000) 2020-02-06T00:44:47.294 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:47.291+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:47.292772+0000) 2020-02-06T00:44:47.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:47.291+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:47.293158+0000) 2020-02-06T00:44:49.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:49.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.669968+0000) 2020-02-06T00:44:49.813 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:49.811+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.812563+0000) 2020-02-06T00:44:49.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:49.812+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.813117+0000) 2020-02-06T00:44:49.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:49.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.813478+0000) 2020-02-06T00:44:49.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:49.812+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.813845+0000) 2020-02-06T00:44:49.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:49.813+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.814530+0000) 2020-02-06T00:44:49.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:49.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.814366+0000) 2020-02-06T00:44:49.816 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:49.813+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.814847+0000) 2020-02-06T00:44:49.816 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:49.813+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.814958+0000) 2020-02-06T00:44:49.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:49.814+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.815528+0000) 2020-02-06T00:44:49.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:49.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.815875+0000) 2020-02-06T00:44:49.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:49.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.815261+0000) 2020-02-06T00:44:49.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:49.814+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.816309+0000) 2020-02-06T00:44:49.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:49.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.816308+0000) 2020-02-06T00:44:49.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:49.819+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.820136+0000) 2020-02-06T00:44:49.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:49.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:49.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:49.917+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:49.918384+0000) 2020-02-06T00:44:50.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:50.519+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:50.520677+0000) 2020-02-06T00:44:50.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:50.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:50.779132+0000) 2020-02-06T00:44:50.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:50.799+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:50.800730+0000) 2020-02-06T00:44:50.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:50.799+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:50.800871+0000) 2020-02-06T00:44:52.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:52.901+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:52.902563+0000) 2020-02-06T00:44:53.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:53.962+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:53.963034+0000) 2020-02-06T00:44:54.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:54.834+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.835986+0000) 2020-02-06T00:44:54.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:54.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.838396+0000) 2020-02-06T00:44:54.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:54.837+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.838785+0000) 2020-02-06T00:44:54.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:54.838+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.839060+0000) 2020-02-06T00:44:54.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:54.838+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.839954+0000) 2020-02-06T00:44:54.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:54.839+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.840564+0000) 2020-02-06T00:44:54.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:54.839+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.841061+0000) 2020-02-06T00:44:54.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:54.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.841645+0000) 2020-02-06T00:44:54.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:54.840+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.841957+0000) 2020-02-06T00:44:54.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:54.841+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.842648+0000) 2020-02-06T00:44:54.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:54.841+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.842627+0000) 2020-02-06T00:44:54.844 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:54.841+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.842839+0000) 2020-02-06T00:44:54.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:54.841+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.843227+0000) 2020-02-06T00:44:54.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:54.844+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:54.845593+0000) 2020-02-06T00:44:54.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:54.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:56.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:56.039+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:56.040011+0000) 2020-02-06T00:44:56.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:56.384+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:56.385598+0000) 2020-02-06T00:44:57.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:57.025+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:57.026637+0000) 2020-02-06T00:44:57.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:57.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:57.292918+0000) 2020-02-06T00:44:57.294 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:57.292+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:57.293029+0000) 2020-02-06T00:44:57.295 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:57.292+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:57.293372+0000) 2020-02-06T00:44:59.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:59.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.670277+0000) 2020-02-06T00:44:59.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:44:59.862+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.863707+0000) 2020-02-06T00:44:59.865 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:44:59.862+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.863153+0000) 2020-02-06T00:44:59.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:44:59.864+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.865064+0000) 2020-02-06T00:44:59.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:44:59.864+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.865157+0000) 2020-02-06T00:44:59.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:59.864+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.865665+0000) 2020-02-06T00:44:59.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:59.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.866164+0000) 2020-02-06T00:44:59.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:44:59.865+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.866448+0000) 2020-02-06T00:44:59.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:44:59.865+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.866645+0000) 2020-02-06T00:44:59.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:44:59.866+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.867147+0000) 2020-02-06T00:44:59.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:44:59.866+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.867364+0000) 2020-02-06T00:44:59.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:44:59.867+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.868130+0000) 2020-02-06T00:44:59.869 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:44:59.867+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.868241+0000) 2020-02-06T00:44:59.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:44:59.868+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.868724+0000) 2020-02-06T00:44:59.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:59.871+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.872380+0000) 2020-02-06T00:44:59.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:44:59.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:44:59.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:44:59.917+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:44:59.918594+0000) 2020-02-06T00:45:00.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:00.520+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:00.520913+0000) 2020-02-06T00:45:00.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:00.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:00.779393+0000) 2020-02-06T00:45:00.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:00.800+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:00.800940+0000) 2020-02-06T00:45:00.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:00.800+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:00.801132+0000) 2020-02-06T00:45:02.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:02.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:02.902830+0000) 2020-02-06T00:45:03.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:03.970+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:03.970667+0000) 2020-02-06T00:45:04.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:04.889+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.889931+0000) 2020-02-06T00:45:04.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:04.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.890461+0000) 2020-02-06T00:45:04.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:04.890+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.891268+0000) 2020-02-06T00:45:04.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:04.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.891484+0000) 2020-02-06T00:45:04.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:04.891+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.892161+0000) 2020-02-06T00:45:04.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:04.891+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.892289+0000) 2020-02-06T00:45:04.893 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:04.891+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.892362+0000) 2020-02-06T00:45:04.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:04.893+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.893506+0000) 2020-02-06T00:45:04.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:04.893+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.893726+0000) 2020-02-06T00:45:04.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:04.893+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.894262+0000) 2020-02-06T00:45:04.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:04.893+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.893927+0000) 2020-02-06T00:45:04.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:04.893+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.894158+0000) 2020-02-06T00:45:04.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:04.893+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.894579+0000) 2020-02-06T00:45:04.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:04.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:04.899958+0000) 2020-02-06T00:45:04.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:04.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:06.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:06.039+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:06.040298+0000) 2020-02-06T00:45:06.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:06.385+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:06.385824+0000) 2020-02-06T00:45:07.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:07.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:07.026873+0000) 2020-02-06T00:45:07.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:07.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:07.293166+0000) 2020-02-06T00:45:07.294 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:07.292+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:07.293278+0000) 2020-02-06T00:45:07.295 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:07.293+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:07.293694+0000) 2020-02-06T00:45:09.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:09.669+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.670522+0000) 2020-02-06T00:45:09.917 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:09.915+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.916564+0000) 2020-02-06T00:45:09.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:09.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.917911+0000) 2020-02-06T00:45:09.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:09.917+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.918760+0000) 2020-02-06T00:45:09.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:09.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.918867+0000) 2020-02-06T00:45:09.920 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:09.918+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.919295+0000) 2020-02-06T00:45:09.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:09.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.920750+0000) 2020-02-06T00:45:09.922 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:09.920+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.921238+0000) 2020-02-06T00:45:09.922 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:09.921+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.921816+0000) 2020-02-06T00:45:09.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:09.921+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.922389+0000) 2020-02-06T00:45:09.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:09.922+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.922525+0000) 2020-02-06T00:45:09.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:09.922+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.922519+0000) 2020-02-06T00:45:09.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:09.921+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.922019+0000) 2020-02-06T00:45:09.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:09.922+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.923312+0000) 2020-02-06T00:45:09.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:09.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.923840+0000) 2020-02-06T00:45:09.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:09.923+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:09.923622+0000) 2020-02-06T00:45:09.936 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:09.935+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183168 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:10.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:10.520+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:10.521234+0000) 2020-02-06T00:45:10.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:10.778+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:10.779595+0000) 2020-02-06T00:45:10.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:10.800+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:10.801146+0000) 2020-02-06T00:45:10.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:10.800+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:10.801365+0000) 2020-02-06T00:45:12.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:12.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:12.903071+0000) 2020-02-06T00:45:13.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:13.977+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:13.977553+0000) 2020-02-06T00:45:14.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:14.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.940828+0000) 2020-02-06T00:45:14.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:14.940+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.941850+0000) 2020-02-06T00:45:14.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:14.941+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.942485+0000) 2020-02-06T00:45:14.943 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:14.941+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.942955+0000) 2020-02-06T00:45:14.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:14.942+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.943461+0000) 2020-02-06T00:45:14.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:14.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.944113+0000) 2020-02-06T00:45:14.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:14.944+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.945150+0000) 2020-02-06T00:45:14.946 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:14.945+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.945924+0000) 2020-02-06T00:45:14.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:14.946+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.946560+0000) 2020-02-06T00:45:14.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:14.945+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.946329+0000) 2020-02-06T00:45:14.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:14.946+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.947223+0000) 2020-02-06T00:45:14.949 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:14.947+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.948270+0000) 2020-02-06T00:45:14.949 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:14.948+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.948629+0000) 2020-02-06T00:45:14.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:14.951+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:14.953007+0000) 2020-02-06T00:45:14.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:14.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183209 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:15.942 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:45:15.948 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:45:16.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:16.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:16.040484+0000) 2020-02-06T00:45:16.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:16.385+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:16.386063+0000) 2020-02-06T00:45:17.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:17.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:17.027143+0000) 2020-02-06T00:45:17.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:17.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:17.293407+0000) 2020-02-06T00:45:17.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:17.293+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:17.293548+0000) 2020-02-06T00:45:17.295 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:17.293+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:17.293922+0000) 2020-02-06T00:45:19.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:19.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.670781+0000) 2020-02-06T00:45:19.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:19.918+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.918977+0000) 2020-02-06T00:45:19.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:19.969+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.970723+0000) 2020-02-06T00:45:19.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:19.970+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.971262+0000) 2020-02-06T00:45:19.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:19.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.971919+0000) 2020-02-06T00:45:19.973 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:19.971+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.972357+0000) 2020-02-06T00:45:19.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:19.971+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.972980+0000) 2020-02-06T00:45:19.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:19.972+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.973494+0000) 2020-02-06T00:45:19.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:19.972+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.973617+0000) 2020-02-06T00:45:19.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:19.973+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.974319+0000) 2020-02-06T00:45:19.975 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:19.973+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.974641+0000) 2020-02-06T00:45:19.976 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:19.974+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.975329+0000) 2020-02-06T00:45:19.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:19.975+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.975565+0000) 2020-02-06T00:45:19.976 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:19.975+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.975467+0000) 2020-02-06T00:45:19.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:19.975+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.976152+0000) 2020-02-06T00:45:19.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:19.975+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:19.976996+0000) 2020-02-06T00:45:19.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:19.988+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:20.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:20.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:20.521511+0000) 2020-02-06T00:45:20.780 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:20.779+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:20.779812+0000) 2020-02-06T00:45:20.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:20.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:20.801397+0000) 2020-02-06T00:45:20.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:20.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:20.801554+0000) 2020-02-06T00:45:22.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:22.903+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:22.903328+0000) 2020-02-06T00:45:23.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:23.984+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:23.985553+0000) 2020-02-06T00:45:24.996 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:24.994+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.995436+0000) 2020-02-06T00:45:24.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:24.995+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.996636+0000) 2020-02-06T00:45:24.998 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:24.996+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.996693+0000) 2020-02-06T00:45:24.998 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:24.996+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.997457+0000) 2020-02-06T00:45:24.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:24.996+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.997858+0000) 2020-02-06T00:45:24.999 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:24.997+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.998474+0000) 2020-02-06T00:45:25.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:24.998+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.999031+0000) 2020-02-06T00:45:25.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:24.998+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.999174+0000) 2020-02-06T00:45:25.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:24.999+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:24.999629+0000) 2020-02-06T00:45:25.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:24.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:25.000218+0000) 2020-02-06T00:45:25.001 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:25.000+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:25.000548+0000) 2020-02-06T00:45:25.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:24.999+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:25.000932+0000) 2020-02-06T00:45:25.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:25.000+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:25.001287+0000) 2020-02-06T00:45:25.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:25.003+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:25.004730+0000) 2020-02-06T00:45:25.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:25.014+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:26.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:26.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:26.040727+0000) 2020-02-06T00:45:26.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:26.386+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:26.386314+0000) 2020-02-06T00:45:27.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:27.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:27.027372+0000) 2020-02-06T00:45:27.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:27.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:27.293712+0000) 2020-02-06T00:45:27.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:27.293+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:27.293805+0000) 2020-02-06T00:45:27.295 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:27.293+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:27.294146+0000) 2020-02-06T00:45:29.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:29.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:29.671038+0000) 2020-02-06T00:45:29.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:29.919+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:29.919203+0000) 2020-02-06T00:45:30.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:30.018+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.020054+0000) 2020-02-06T00:45:30.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:30.019+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.020736+0000) 2020-02-06T00:45:30.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:30.020+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.021336+0000) 2020-02-06T00:45:30.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:30.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.021681+0000) 2020-02-06T00:45:30.023 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:30.022+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.022544+0000) 2020-02-06T00:45:30.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:30.022+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.023239+0000) 2020-02-06T00:45:30.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:30.022+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.023669+0000) 2020-02-06T00:45:30.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:30.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.023718+0000) 2020-02-06T00:45:30.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:30.023+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.023811+0000) 2020-02-06T00:45:30.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:30.023+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.024596+0000) 2020-02-06T00:45:30.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:30.024+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.024659+0000) 2020-02-06T00:45:30.026 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:30.024+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.025147+0000) 2020-02-06T00:45:30.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:30.025+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.026078+0000) 2020-02-06T00:45:30.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:30.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.028130+0000) 2020-02-06T00:45:30.041 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:30.039+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183334 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:30.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:30.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.521742+0000) 2020-02-06T00:45:30.781 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:30.779+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.780056+0000) 2020-02-06T00:45:30.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:30.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.801598+0000) 2020-02-06T00:45:30.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:30.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:30.801729+0000) 2020-02-06T00:45:32.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:32.903+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:32.903594+0000) 2020-02-06T00:45:33.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:33.993+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:33.993353+0000) 2020-02-06T00:45:35.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:35.046+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.046402+0000) 2020-02-06T00:45:35.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:35.046+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.046920+0000) 2020-02-06T00:45:35.048 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:35.047+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.047613+0000) 2020-02-06T00:45:35.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:35.048+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.049190+0000) 2020-02-06T00:45:35.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:35.048+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.049127+0000) 2020-02-06T00:45:35.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:35.049+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.049678+0000) 2020-02-06T00:45:35.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:35.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.050220+0000) 2020-02-06T00:45:35.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:35.051+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.051389+0000) 2020-02-06T00:45:35.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:35.051+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.051577+0000) 2020-02-06T00:45:35.053 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:35.050+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.050802+0000) 2020-02-06T00:45:35.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:35.051+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.051931+0000) 2020-02-06T00:45:35.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:35.052+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.052974+0000) 2020-02-06T00:45:35.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:35.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.056775+0000) 2020-02-06T00:45:35.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:35.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:35.057025+0000) 2020-02-06T00:45:35.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:35.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:36.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:36.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:36.040984+0000) 2020-02-06T00:45:36.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:36.385+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:36.386571+0000) 2020-02-06T00:45:37.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:37.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:37.027604+0000) 2020-02-06T00:45:37.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:37.292+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:37.293951+0000) 2020-02-06T00:45:37.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:37.292+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:37.294089+0000) 2020-02-06T00:45:37.295 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:37.292+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:37.294379+0000) 2020-02-06T00:45:39.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:39.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:39.671308+0000) 2020-02-06T00:45:39.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:39.918+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:39.919431+0000) 2020-02-06T00:45:40.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:40.069+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.070425+0000) 2020-02-06T00:45:40.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:40.070+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.071938+0000) 2020-02-06T00:45:40.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:40.071+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.072422+0000) 2020-02-06T00:45:40.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:40.073+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.074243+0000) 2020-02-06T00:45:40.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:40.073+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.075055+0000) 2020-02-06T00:45:40.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:40.074+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.075389+0000) 2020-02-06T00:45:40.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:40.073+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.075257+0000) 2020-02-06T00:45:40.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:40.074+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.076297+0000) 2020-02-06T00:45:40.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:40.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.076851+0000) 2020-02-06T00:45:40.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:40.075+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.077077+0000) 2020-02-06T00:45:40.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:40.075+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.077092+0000) 2020-02-06T00:45:40.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:40.076+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.077474+0000) 2020-02-06T00:45:40.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:40.076+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.078085+0000) 2020-02-06T00:45:40.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:40.083+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.084507+0000) 2020-02-06T00:45:40.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:40.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:40.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:40.520+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.521937+0000) 2020-02-06T00:45:40.781 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:40.779+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.780311+0000) 2020-02-06T00:45:40.802 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:40.800+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.801801+0000) 2020-02-06T00:45:40.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:40.800+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:40.801915+0000) 2020-02-06T00:45:42.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:42.902+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:42.903892+0000) 2020-02-06T00:45:44.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:44.000+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:44.001698+0000) 2020-02-06T00:45:45.100 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:45.097+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.098964+0000) 2020-02-06T00:45:45.100 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:45.098+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.099546+0000) 2020-02-06T00:45:45.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:45.098+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.099205+0000) 2020-02-06T00:45:45.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:45.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.099803+0000) 2020-02-06T00:45:45.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:45.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.100237+0000) 2020-02-06T00:45:45.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:45.099+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.100939+0000) 2020-02-06T00:45:45.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:45.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.101041+0000) 2020-02-06T00:45:45.103 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:45.100+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.101500+0000) 2020-02-06T00:45:45.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:45.100+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.101792+0000) 2020-02-06T00:45:45.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:45.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.102163+0000) 2020-02-06T00:45:45.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:45.100+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.102287+0000) 2020-02-06T00:45:45.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:45.101+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.102699+0000) 2020-02-06T00:45:45.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:45.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.106540+0000) 2020-02-06T00:45:45.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:45.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:45.106743+0000) 2020-02-06T00:45:45.118 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:45.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183458 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:46.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:46.039+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:46.041187+0000) 2020-02-06T00:45:46.130 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:45:46.137 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:45:46.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:46.385+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:46.386809+0000) 2020-02-06T00:45:47.028 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:47.026+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:47.027834+0000) 2020-02-06T00:45:47.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:47.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:47.294217+0000) 2020-02-06T00:45:47.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:47.293+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:47.294318+0000) 2020-02-06T00:45:47.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:47.293+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:47.294746+0000) 2020-02-06T00:45:49.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:49.670+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:49.671557+0000) 2020-02-06T00:45:49.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:49.918+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:49.919655+0000) 2020-02-06T00:45:50.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:50.120+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.122017+0000) 2020-02-06T00:45:50.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:50.122+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.123417+0000) 2020-02-06T00:45:50.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:50.123+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.124208+0000) 2020-02-06T00:45:50.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:50.123+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.124740+0000) 2020-02-06T00:45:50.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:50.124+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.125521+0000) 2020-02-06T00:45:50.127 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:50.125+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.126250+0000) 2020-02-06T00:45:50.128 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:50.124+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.126179+0000) 2020-02-06T00:45:50.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:50.125+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.126764+0000) 2020-02-06T00:45:50.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:50.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.127056+0000) 2020-02-06T00:45:50.129 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:50.126+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.127371+0000) 2020-02-06T00:45:50.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:50.126+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.128139+0000) 2020-02-06T00:45:50.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:50.127+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.128869+0000) 2020-02-06T00:45:50.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:50.127+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.128924+0000) 2020-02-06T00:45:50.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:50.134+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.135182+0000) 2020-02-06T00:45:50.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:50.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:50.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:50.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.522171+0000) 2020-02-06T00:45:50.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:50.779+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.780566+0000) 2020-02-06T00:45:50.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:50.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.802020+0000) 2020-02-06T00:45:50.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:50.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:50.802140+0000) 2020-02-06T00:45:52.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:52.903+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:52.904135+0000) 2020-02-06T00:45:54.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:54.007+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:54.008664+0000) 2020-02-06T00:45:55.149 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:45:55.147+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.148417+0000) 2020-02-06T00:45:55.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:55.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.148990+0000) 2020-02-06T00:45:55.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:55.148+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.149686+0000) 2020-02-06T00:45:55.151 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:45:55.149+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.150526+0000) 2020-02-06T00:45:55.152 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:45:55.149+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.151297+0000) 2020-02-06T00:45:55.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:55.151+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.152212+0000) 2020-02-06T00:45:55.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:45:55.151+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.152643+0000) 2020-02-06T00:45:55.154 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:55.151+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.153143+0000) 2020-02-06T00:45:55.154 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:55.152+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.153888+0000) 2020-02-06T00:45:55.155 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:45:55.153+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.154411+0000) 2020-02-06T00:45:55.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:55.153+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.154656+0000) 2020-02-06T00:45:55.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:55.154+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.155494+0000) 2020-02-06T00:45:55.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:55.155+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.156560+0000) 2020-02-06T00:45:55.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:55.159+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:55.160586+0000) 2020-02-06T00:45:55.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:45:55.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:45:56.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:45:56.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:56.041412+0000) 2020-02-06T00:45:56.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:45:56.385+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:56.387028+0000) 2020-02-06T00:45:57.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:45:57.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:57.028069+0000) 2020-02-06T00:45:57.295 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:45:57.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:57.294504+0000) 2020-02-06T00:45:57.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:45:57.293+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:57.294825+0000) 2020-02-06T00:45:57.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:45:57.293+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:57.295088+0000) 2020-02-06T00:45:59.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:59.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:59.671781+0000) 2020-02-06T00:45:59.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:45:59.919+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:45:59.919941+0000) 2020-02-06T00:46:00.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:00.173+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.174150+0000) 2020-02-06T00:46:00.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:00.173+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.174574+0000) 2020-02-06T00:46:00.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:00.173+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.175118+0000) 2020-02-06T00:46:00.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:00.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.175524+0000) 2020-02-06T00:46:00.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:00.174+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.175851+0000) 2020-02-06T00:46:00.177 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:00.174+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.175988+0000) 2020-02-06T00:46:00.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:00.175+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.176608+0000) 2020-02-06T00:46:00.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:00.176+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.177679+0000) 2020-02-06T00:46:00.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:00.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.178235+0000) 2020-02-06T00:46:00.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:00.177+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.179026+0000) 2020-02-06T00:46:00.180 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:00.177+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.179015+0000) 2020-02-06T00:46:00.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:00.178+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.179465+0000) 2020-02-06T00:46:00.181 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:00.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.180094+0000) 2020-02-06T00:46:00.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:00.182+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.183516+0000) 2020-02-06T00:46:00.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:00.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:00.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:00.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.522403+0000) 2020-02-06T00:46:00.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:00.780+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.780778+0000) 2020-02-06T00:46:00.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:00.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.802287+0000) 2020-02-06T00:46:00.803 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:00.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:00.802335+0000) 2020-02-06T00:46:02.905 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:02.903+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:02.904461+0000) 2020-02-06T00:46:04.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:04.016+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:04.016866+0000) 2020-02-06T00:46:05.199 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:05.196+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.197990+0000) 2020-02-06T00:46:05.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:05.197+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.198564+0000) 2020-02-06T00:46:05.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:05.198+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.199660+0000) 2020-02-06T00:46:05.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:05.198+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.199989+0000) 2020-02-06T00:46:05.201 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:05.199+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.200533+0000) 2020-02-06T00:46:05.201 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:05.199+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.200958+0000) 2020-02-06T00:46:05.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:05.200+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.201702+0000) 2020-02-06T00:46:05.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:05.200+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.201586+0000) 2020-02-06T00:46:05.204 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:05.201+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.202881+0000) 2020-02-06T00:46:05.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:05.201+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.202687+0000) 2020-02-06T00:46:05.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:05.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.203619+0000) 2020-02-06T00:46:05.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:05.202+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.203953+0000) 2020-02-06T00:46:05.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:05.202+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.204092+0000) 2020-02-06T00:46:05.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:05.205+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:05.206763+0000) 2020-02-06T00:46:05.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:05.216+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:06.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:06.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:06.041696+0000) 2020-02-06T00:46:06.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:06.386+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:06.387297+0000) 2020-02-06T00:46:07.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:07.027+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:07.028359+0000) 2020-02-06T00:46:07.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:07.294+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:07.294792+0000) 2020-02-06T00:46:07.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:07.294+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:07.295052+0000) 2020-02-06T00:46:07.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:07.294+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:07.295318+0000) 2020-02-06T00:46:09.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:09.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:09.672039+0000) 2020-02-06T00:46:09.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:09.919+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:09.920181+0000) 2020-02-06T00:46:10.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:10.222+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.223418+0000) 2020-02-06T00:46:10.225 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:10.223+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.224131+0000) 2020-02-06T00:46:10.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:10.223+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.224704+0000) 2020-02-06T00:46:10.226 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:10.223+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.224938+0000) 2020-02-06T00:46:10.226 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:10.224+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.225473+0000) 2020-02-06T00:46:10.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:10.224+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.225792+0000) 2020-02-06T00:46:10.227 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:10.224+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.226042+0000) 2020-02-06T00:46:10.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:10.225+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.227356+0000) 2020-02-06T00:46:10.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:10.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.228015+0000) 2020-02-06T00:46:10.229 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:10.226+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.227661+0000) 2020-02-06T00:46:10.229 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:10.226+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.228370+0000) 2020-02-06T00:46:10.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:10.227+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.228917+0000) 2020-02-06T00:46:10.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:10.227+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.229042+0000) 2020-02-06T00:46:10.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:10.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.232276+0000) 2020-02-06T00:46:10.246 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:10.243+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:10.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:10.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.522624+0000) 2020-02-06T00:46:10.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:10.779+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.781046+0000) 2020-02-06T00:46:10.803 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:10.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.802471+0000) 2020-02-06T00:46:10.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:10.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:10.802523+0000) 2020-02-06T00:46:12.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:12.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:12.904723+0000) 2020-02-06T00:46:14.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:14.024+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:14.024878+0000) 2020-02-06T00:46:15.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:15.250+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.251467+0000) 2020-02-06T00:46:15.253 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:15.251+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.252160+0000) 2020-02-06T00:46:15.254 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:15.252+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.253145+0000) 2020-02-06T00:46:15.254 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:15.251+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.253037+0000) 2020-02-06T00:46:15.254 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:15.252+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.253683+0000) 2020-02-06T00:46:15.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:15.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.254373+0000) 2020-02-06T00:46:15.255 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:15.253+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.254446+0000) 2020-02-06T00:46:15.256 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:15.254+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.255527+0000) 2020-02-06T00:46:15.257 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:15.254+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.256315+0000) 2020-02-06T00:46:15.257 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:15.255+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.256343+0000) 2020-02-06T00:46:15.258 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:15.255+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.256833+0000) 2020-02-06T00:46:15.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:15.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.257240+0000) 2020-02-06T00:46:15.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:15.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.257239+0000) 2020-02-06T00:46:15.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:15.259+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:15.260525+0000) 2020-02-06T00:46:15.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:15.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:16.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:16.040+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:16.041908+0000) 2020-02-06T00:46:16.281 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:46:16.287 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:46:16.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:16.386+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:16.387505+0000) 2020-02-06T00:46:17.029 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:17.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:17.028635+0000) 2020-02-06T00:46:17.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:17.293+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:17.295006+0000) 2020-02-06T00:46:17.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:17.293+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:17.295295+0000) 2020-02-06T00:46:17.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:17.294+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:17.295591+0000) 2020-02-06T00:46:19.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:19.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:19.672304+0000) 2020-02-06T00:46:19.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:19.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:19.920407+0000) 2020-02-06T00:46:20.287 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:20.276+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.277454+0000) 2020-02-06T00:46:20.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:20.276+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.278134+0000) 2020-02-06T00:46:20.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:20.277+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.278525+0000) 2020-02-06T00:46:20.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:20.277+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.278588+0000) 2020-02-06T00:46:20.289 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:20.277+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.278987+0000) 2020-02-06T00:46:20.289 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:20.278+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.279368+0000) 2020-02-06T00:46:20.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:20.279+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.280535+0000) 2020-02-06T00:46:20.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:20.279+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.280945+0000) 2020-02-06T00:46:20.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:20.279+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.281363+0000) 2020-02-06T00:46:20.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:20.280+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.281872+0000) 2020-02-06T00:46:20.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:20.280+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.282100+0000) 2020-02-06T00:46:20.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:20.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.282063+0000) 2020-02-06T00:46:20.292 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:20.281+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.283239+0000) 2020-02-06T00:46:20.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:20.287+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.288479+0000) 2020-02-06T00:46:20.303 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:20.294+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183747 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:20.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:20.521+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.522793+0000) 2020-02-06T00:46:20.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:20.780+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.781251+0000) 2020-02-06T00:46:20.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:20.802+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.802709+0000) 2020-02-06T00:46:20.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:20.802+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:20.802709+0000) 2020-02-06T00:46:22.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:22.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:22.904970+0000) 2020-02-06T00:46:24.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:24.033+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:24.033654+0000) 2020-02-06T00:46:25.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:25.300+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.301897+0000) 2020-02-06T00:46:25.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:25.301+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.302917+0000) 2020-02-06T00:46:25.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:25.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.303195+0000) 2020-02-06T00:46:25.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:25.302+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.303618+0000) 2020-02-06T00:46:25.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:25.302+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.303847+0000) 2020-02-06T00:46:25.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:25.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.303993+0000) 2020-02-06T00:46:25.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:25.303+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.304884+0000) 2020-02-06T00:46:25.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:25.304+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.305819+0000) 2020-02-06T00:46:25.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:25.304+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.305942+0000) 2020-02-06T00:46:25.308 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:25.303+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.304723+0000) 2020-02-06T00:46:25.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:25.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.305450+0000) 2020-02-06T00:46:25.308 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:25.304+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.305569+0000) 2020-02-06T00:46:25.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:25.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.312652+0000) 2020-02-06T00:46:25.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:25.311+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:25.312970+0000) 2020-02-06T00:46:25.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:25.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183787 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:26.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:26.041+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:26.042165+0000) 2020-02-06T00:46:26.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:26.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:26.387747+0000) 2020-02-06T00:46:27.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:27.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:27.028866+0000) 2020-02-06T00:46:27.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:27.294+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:27.295289+0000) 2020-02-06T00:46:27.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:27.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:27.295548+0000) 2020-02-06T00:46:27.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:27.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:27.295835+0000) 2020-02-06T00:46:29.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:29.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:29.672573+0000) 2020-02-06T00:46:29.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:29.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:29.920596+0000) 2020-02-06T00:46:30.327 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:30.325+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.326966+0000) 2020-02-06T00:46:30.329 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:30.327+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.328340+0000) 2020-02-06T00:46:30.329 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:30.327+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.328023+0000) 2020-02-06T00:46:30.330 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:30.327+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.328973+0000) 2020-02-06T00:46:30.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:30.328+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.329373+0000) 2020-02-06T00:46:30.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:30.329+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.329745+0000) 2020-02-06T00:46:30.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:30.329+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.330220+0000) 2020-02-06T00:46:30.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:30.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.330325+0000) 2020-02-06T00:46:30.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:30.329+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.330555+0000) 2020-02-06T00:46:30.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:30.329+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.330616+0000) 2020-02-06T00:46:30.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:30.330+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.331010+0000) 2020-02-06T00:46:30.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:30.330+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.331313+0000) 2020-02-06T00:46:30.333 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:30.330+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.331187+0000) 2020-02-06T00:46:30.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:30.335+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.337020+0000) 2020-02-06T00:46:30.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:30.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:30.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:30.522+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.523014+0000) 2020-02-06T00:46:30.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:30.780+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.781481+0000) 2020-02-06T00:46:30.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:30.801+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.802967+0000) 2020-02-06T00:46:30.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:30.801+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:30.802965+0000) 2020-02-06T00:46:32.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:32.904+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:32.905220+0000) 2020-02-06T00:46:34.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:34.042+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:34.043678+0000) 2020-02-06T00:46:35.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:35.352+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.352726+0000) 2020-02-06T00:46:35.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:35.353+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.353614+0000) 2020-02-06T00:46:35.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:35.353+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.354135+0000) 2020-02-06T00:46:35.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:35.354+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.354491+0000) 2020-02-06T00:46:35.356 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:35.354+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.355259+0000) 2020-02-06T00:46:35.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:35.354+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.355800+0000) 2020-02-06T00:46:35.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:35.355+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.356560+0000) 2020-02-06T00:46:35.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:35.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.356675+0000) 2020-02-06T00:46:35.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:35.356+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.356980+0000) 2020-02-06T00:46:35.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:35.356+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.357493+0000) 2020-02-06T00:46:35.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:35.357+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.357552+0000) 2020-02-06T00:46:35.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:35.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.358090+0000) 2020-02-06T00:46:35.359 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:35.357+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.358272+0000) 2020-02-06T00:46:35.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:35.362+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:35.363211+0000) 2020-02-06T00:46:35.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:35.371+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183870 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:36.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:36.041+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:36.042379+0000) 2020-02-06T00:46:36.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:36.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:36.387978+0000) 2020-02-06T00:46:37.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:37.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:37.029126+0000) 2020-02-06T00:46:37.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:37.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:37.295525+0000) 2020-02-06T00:46:37.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:37.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:37.295836+0000) 2020-02-06T00:46:37.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:37.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:37.296094+0000) 2020-02-06T00:46:39.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:39.671+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:39.672809+0000) 2020-02-06T00:46:39.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:39.919+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:39.920846+0000) 2020-02-06T00:46:40.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:40.375+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.376175+0000) 2020-02-06T00:46:40.378 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:40.376+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.377184+0000) 2020-02-06T00:46:40.378 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:40.376+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.377713+0000) 2020-02-06T00:46:40.379 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:40.378+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.378635+0000) 2020-02-06T00:46:40.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:40.378+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.379107+0000) 2020-02-06T00:46:40.380 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:40.378+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.379214+0000) 2020-02-06T00:46:40.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:40.379+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.379764+0000) 2020-02-06T00:46:40.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:40.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.381049+0000) 2020-02-06T00:46:40.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:40.380+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.381286+0000) 2020-02-06T00:46:40.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:40.380+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.381552+0000) 2020-02-06T00:46:40.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:40.381+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.381913+0000) 2020-02-06T00:46:40.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:40.381+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.382191+0000) 2020-02-06T00:46:40.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:40.381+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.382814+0000) 2020-02-06T00:46:40.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:40.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.386135+0000) 2020-02-06T00:46:40.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:40.396+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183913 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:40.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:40.522+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.523246+0000) 2020-02-06T00:46:40.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:40.780+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.781720+0000) 2020-02-06T00:46:40.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:40.802+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.803195+0000) 2020-02-06T00:46:40.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:40.802+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:40.803192+0000) 2020-02-06T00:46:42.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:42.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:42.905510+0000) 2020-02-06T00:46:44.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:44.050+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:44.051796+0000) 2020-02-06T00:46:45.404 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:45.402+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.403377+0000) 2020-02-06T00:46:45.406 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:45.404+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.405021+0000) 2020-02-06T00:46:45.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:45.404+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.404811+0000) 2020-02-06T00:46:45.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:45.404+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.405523+0000) 2020-02-06T00:46:45.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:45.404+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.406056+0000) 2020-02-06T00:46:45.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:45.405+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.406161+0000) 2020-02-06T00:46:45.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:45.406+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.406566+0000) 2020-02-06T00:46:45.408 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:45.405+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.406928+0000) 2020-02-06T00:46:45.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:45.406+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.407166+0000) 2020-02-06T00:46:45.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:45.407+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.408139+0000) 2020-02-06T00:46:45.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:45.407+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.408228+0000) 2020-02-06T00:46:45.410 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:45.407+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.408410+0000) 2020-02-06T00:46:45.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:45.407+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.408867+0000) 2020-02-06T00:46:45.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:45.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:45.416045+0000) 2020-02-06T00:46:45.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:45.422+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183953 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:46.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:46.042+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:46.042592+0000) 2020-02-06T00:46:46.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:46.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:46.388222+0000) 2020-02-06T00:46:46.466 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:46:46.472 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:46:47.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:47.028+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:47.029360+0000) 2020-02-06T00:46:47.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:47.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:47.295764+0000) 2020-02-06T00:46:47.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:47.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:47.296056+0000) 2020-02-06T00:46:47.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:47.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:47.296358+0000) 2020-02-06T00:46:49.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:49.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:49.673067+0000) 2020-02-06T00:46:49.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:49.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:49.921124+0000) 2020-02-06T00:46:50.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:50.429+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.429667+0000) 2020-02-06T00:46:50.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:50.429+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.430178+0000) 2020-02-06T00:46:50.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:50.429+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.430491+0000) 2020-02-06T00:46:50.432 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:50.430+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.431121+0000) 2020-02-06T00:46:50.432 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:50.430+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.431421+0000) 2020-02-06T00:46:50.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:50.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.432407+0000) 2020-02-06T00:46:50.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:50.431+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.432629+0000) 2020-02-06T00:46:50.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:50.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.433051+0000) 2020-02-06T00:46:50.435 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:50.433+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.434077+0000) 2020-02-06T00:46:50.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:50.433+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.433711+0000) 2020-02-06T00:46:50.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:50.433+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.434344+0000) 2020-02-06T00:46:50.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:50.433+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.433721+0000) 2020-02-06T00:46:50.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:50.434+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.434588+0000) 2020-02-06T00:46:50.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:50.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.437704+0000) 2020-02-06T00:46:50.449 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:50.448+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 183997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:50.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:50.523+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.523495+0000) 2020-02-06T00:46:50.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:50.781+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.781953+0000) 2020-02-06T00:46:50.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:50.802+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.803427+0000) 2020-02-06T00:46:50.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:50.802+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:50.803427+0000) 2020-02-06T00:46:52.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:52.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:52.905752+0000) 2020-02-06T00:46:54.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:54.059+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:54.060044+0000) 2020-02-06T00:46:55.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:46:55.453+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.454374+0000) 2020-02-06T00:46:55.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:46:55.454+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.455324+0000) 2020-02-06T00:46:55.457 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:46:55.454+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.454912+0000) 2020-02-06T00:46:55.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:46:55.454+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.456009+0000) 2020-02-06T00:46:55.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:55.455+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.456622+0000) 2020-02-06T00:46:55.458 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:55.456+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.456988+0000) 2020-02-06T00:46:55.458 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:46:55.457+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.457762+0000) 2020-02-06T00:46:55.459 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:55.457+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.458240+0000) 2020-02-06T00:46:55.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:55.457+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.458500+0000) 2020-02-06T00:46:55.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:55.457+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.458889+0000) 2020-02-06T00:46:55.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:55.458+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.459172+0000) 2020-02-06T00:46:55.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:55.459+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.459844+0000) 2020-02-06T00:46:55.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:55.458+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.459326+0000) 2020-02-06T00:46:55.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:55.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:55.462836+0000) 2020-02-06T00:46:55.474 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:46:55.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184033 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:46:56.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:46:56.042+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:56.042835+0000) 2020-02-06T00:46:56.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:46:56.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:56.388385+0000) 2020-02-06T00:46:57.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:46:57.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:57.029623+0000) 2020-02-06T00:46:57.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:46:57.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:57.296021+0000) 2020-02-06T00:46:57.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:46:57.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:57.296300+0000) 2020-02-06T00:46:57.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:46:57.296+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:57.296614+0000) 2020-02-06T00:46:59.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:59.672+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:59.673371+0000) 2020-02-06T00:46:59.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:46:59.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:46:59.921355+0000) 2020-02-06T00:47:00.479 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:00.477+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.478823+0000) 2020-02-06T00:47:00.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:00.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.479505+0000) 2020-02-06T00:47:00.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:00.479+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.480371+0000) 2020-02-06T00:47:00.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:00.480+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.481208+0000) 2020-02-06T00:47:00.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:00.481+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.481833+0000) 2020-02-06T00:47:00.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:00.482+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.482424+0000) 2020-02-06T00:47:00.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:00.482+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.483135+0000) 2020-02-06T00:47:00.485 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:00.482+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.483300+0000) 2020-02-06T00:47:00.485 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:00.483+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.484246+0000) 2020-02-06T00:47:00.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:00.483+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.484423+0000) 2020-02-06T00:47:00.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:00.484+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.485037+0000) 2020-02-06T00:47:00.486 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:00.484+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.485235+0000) 2020-02-06T00:47:00.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:00.484+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.485538+0000) 2020-02-06T00:47:00.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:00.488+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.489645+0000) 2020-02-06T00:47:00.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:00.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184076 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:00.524 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:00.523+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.523725+0000) 2020-02-06T00:47:00.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:00.781+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.782221+0000) 2020-02-06T00:47:00.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:00.803+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.803668+0000) 2020-02-06T00:47:00.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:00.803+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:00.803668+0000) 2020-02-06T00:47:02.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:02.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:02.906005+0000) 2020-02-06T00:47:04.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:04.068+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:04.068476+0000) 2020-02-06T00:47:05.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:05.503+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.504737+0000) 2020-02-06T00:47:05.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:05.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.505219+0000) 2020-02-06T00:47:05.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:05.504+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.505751+0000) 2020-02-06T00:47:05.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:05.505+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.506437+0000) 2020-02-06T00:47:05.507 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:05.505+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.506571+0000) 2020-02-06T00:47:05.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:05.505+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.507017+0000) 2020-02-06T00:47:05.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:05.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.506823+0000) 2020-02-06T00:47:05.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:05.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.507562+0000) 2020-02-06T00:47:05.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:05.507+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.509040+0000) 2020-02-06T00:47:05.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:05.507+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.508732+0000) 2020-02-06T00:47:05.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:05.507+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.508539+0000) 2020-02-06T00:47:05.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:05.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.509563+0000) 2020-02-06T00:47:05.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:05.508+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.510110+0000) 2020-02-06T00:47:05.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:05.513+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:05.514605+0000) 2020-02-06T00:47:05.528 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:05.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184113 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:06.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:06.041+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:06.043050+0000) 2020-02-06T00:47:06.389 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:06.388+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:06.388590+0000) 2020-02-06T00:47:07.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:07.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:07.029900+0000) 2020-02-06T00:47:07.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:07.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:07.296312+0000) 2020-02-06T00:47:07.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:07.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:07.296528+0000) 2020-02-06T00:47:07.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:07.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:07.296866+0000) 2020-02-06T00:47:09.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:09.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:09.673602+0000) 2020-02-06T00:47:09.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:09.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:09.921659+0000) 2020-02-06T00:47:10.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:10.522+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.523922+0000) 2020-02-06T00:47:10.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:10.530+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.531380+0000) 2020-02-06T00:47:10.532 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:10.530+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.532033+0000) 2020-02-06T00:47:10.533 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:10.531+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.532916+0000) 2020-02-06T00:47:10.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:10.532+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.533848+0000) 2020-02-06T00:47:10.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:10.532+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.534011+0000) 2020-02-06T00:47:10.535 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:10.533+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.534910+0000) 2020-02-06T00:47:10.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:10.534+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.535462+0000) 2020-02-06T00:47:10.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:10.534+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.535895+0000) 2020-02-06T00:47:10.537 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:10.534+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.536095+0000) 2020-02-06T00:47:10.537 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:10.535+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.536310+0000) 2020-02-06T00:47:10.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:10.535+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.536549+0000) 2020-02-06T00:47:10.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:10.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.537027+0000) 2020-02-06T00:47:10.538 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:10.535+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.537286+0000) 2020-02-06T00:47:10.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:10.541+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.542463+0000) 2020-02-06T00:47:10.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:10.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:10.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:10.782+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.782435+0000) 2020-02-06T00:47:10.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:10.802+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.803915+0000) 2020-02-06T00:47:10.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:10.802+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:10.803917+0000) 2020-02-06T00:47:12.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:12.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:12.906254+0000) 2020-02-06T00:47:14.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:14.077+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:14.077434+0000) 2020-02-06T00:47:15.558 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:15.556+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.557650+0000) 2020-02-06T00:47:15.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:15.556+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.558019+0000) 2020-02-06T00:47:15.559 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:15.557+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.558464+0000) 2020-02-06T00:47:15.560 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:15.558+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.560059+0000) 2020-02-06T00:47:15.561 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:15.559+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.560329+0000) 2020-02-06T00:47:15.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:15.559+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.560865+0000) 2020-02-06T00:47:15.562 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:15.559+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.560887+0000) 2020-02-06T00:47:15.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:15.560+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.561218+0000) 2020-02-06T00:47:15.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:15.559+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.561308+0000) 2020-02-06T00:47:15.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:15.560+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.561878+0000) 2020-02-06T00:47:15.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:15.560+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.562239+0000) 2020-02-06T00:47:15.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:15.560+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.561926+0000) 2020-02-06T00:47:15.564 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:15.561+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.562821+0000) 2020-02-06T00:47:15.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:15.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:15.567784+0000) 2020-02-06T00:47:15.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:15.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:16.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:16.042+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:16.043294+0000) 2020-02-06T00:47:16.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:16.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:16.388770+0000) 2020-02-06T00:47:16.621 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:47:16.627 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:47:17.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:17.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:17.030155+0000) 2020-02-06T00:47:17.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:17.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:17.296525+0000) 2020-02-06T00:47:17.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:17.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:17.296737+0000) 2020-02-06T00:47:17.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:17.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:17.297105+0000) 2020-02-06T00:47:19.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:19.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:19.673844+0000) 2020-02-06T00:47:19.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:19.920+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:19.921923+0000) 2020-02-06T00:47:20.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:20.522+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.524167+0000) 2020-02-06T00:47:20.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:20.580+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.581652+0000) 2020-02-06T00:47:20.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:20.582+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.582163+0000) 2020-02-06T00:47:20.584 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:20.581+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.583382+0000) 2020-02-06T00:47:20.585 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:20.583+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.583792+0000) 2020-02-06T00:47:20.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:20.584+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.584389+0000) 2020-02-06T00:47:20.585 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:20.584+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.584320+0000) 2020-02-06T00:47:20.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:20.584+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.584780+0000) 2020-02-06T00:47:20.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:20.585+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.585353+0000) 2020-02-06T00:47:20.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:20.584+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.585763+0000) 2020-02-06T00:47:20.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:20.584+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.585867+0000) 2020-02-06T00:47:20.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:20.585+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.586657+0000) 2020-02-06T00:47:20.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:20.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.587334+0000) 2020-02-06T00:47:20.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:20.586+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.587940+0000) 2020-02-06T00:47:20.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:20.590+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.590345+0000) 2020-02-06T00:47:20.601 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:20.599+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:20.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:20.782+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.782647+0000) 2020-02-06T00:47:20.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:20.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.804161+0000) 2020-02-06T00:47:20.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:20.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:20.804160+0000) 2020-02-06T00:47:22.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:22.905+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:22.906519+0000) 2020-02-06T00:47:24.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:24.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:24.085159+0000) 2020-02-06T00:47:25.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:25.606+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.607304+0000) 2020-02-06T00:47:25.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:25.606+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.607703+0000) 2020-02-06T00:47:25.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:25.607+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.608447+0000) 2020-02-06T00:47:25.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:25.606+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.608247+0000) 2020-02-06T00:47:25.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:25.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.609176+0000) 2020-02-06T00:47:25.610 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:25.608+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.609423+0000) 2020-02-06T00:47:25.611 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:25.608+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.610391+0000) 2020-02-06T00:47:25.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:25.609+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.610565+0000) 2020-02-06T00:47:25.612 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:25.609+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.610859+0000) 2020-02-06T00:47:25.612 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:25.610+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.611407+0000) 2020-02-06T00:47:25.612 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:25.610+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.611585+0000) 2020-02-06T00:47:25.613 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:25.611+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.612588+0000) 2020-02-06T00:47:25.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:25.611+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.612826+0000) 2020-02-06T00:47:25.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:25.617+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:25.618591+0000) 2020-02-06T00:47:25.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:25.624+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184274 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:26.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:26.042+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:26.043530+0000) 2020-02-06T00:47:26.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:26.387+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:26.388994+0000) 2020-02-06T00:47:27.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:27.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:27.030397+0000) 2020-02-06T00:47:27.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:27.295+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:27.296792+0000) 2020-02-06T00:47:27.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:27.295+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:27.296975+0000) 2020-02-06T00:47:27.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:27.295+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:27.297321+0000) 2020-02-06T00:47:29.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:29.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:29.674132+0000) 2020-02-06T00:47:29.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:29.921+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:29.922150+0000) 2020-02-06T00:47:30.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:30.523+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.524399+0000) 2020-02-06T00:47:30.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:30.629+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.630573+0000) 2020-02-06T00:47:30.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:30.629+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.631353+0000) 2020-02-06T00:47:30.632 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:30.630+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.631983+0000) 2020-02-06T00:47:30.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:30.631+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.632933+0000) 2020-02-06T00:47:30.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:30.632+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.633392+0000) 2020-02-06T00:47:30.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:30.633+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.634199+0000) 2020-02-06T00:47:30.635 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:30.633+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.634783+0000) 2020-02-06T00:47:30.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:30.633+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.634986+0000) 2020-02-06T00:47:30.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:30.633+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.635057+0000) 2020-02-06T00:47:30.636 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:30.634+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.635603+0000) 2020-02-06T00:47:30.637 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:30.634+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.635810+0000) 2020-02-06T00:47:30.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:30.634+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.636372+0000) 2020-02-06T00:47:30.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:30.635+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.636577+0000) 2020-02-06T00:47:30.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:30.638+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.639208+0000) 2020-02-06T00:47:30.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:30.650+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184319 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:30.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:30.781+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.782873+0000) 2020-02-06T00:47:30.805 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:30.803+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.804400+0000) 2020-02-06T00:47:30.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:30.803+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:30.804397+0000) 2020-02-06T00:47:32.907 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:32.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:32.906815+0000) 2020-02-06T00:47:34.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:34.092+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:34.093968+0000) 2020-02-06T00:47:35.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:35.657+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.658516+0000) 2020-02-06T00:47:35.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:35.657+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.658944+0000) 2020-02-06T00:47:35.660 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:35.657+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.659366+0000) 2020-02-06T00:47:35.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:35.658+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.659729+0000) 2020-02-06T00:47:35.661 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:35.658+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.660021+0000) 2020-02-06T00:47:35.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:35.659+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.660355+0000) 2020-02-06T00:47:35.662 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:35.659+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.660735+0000) 2020-02-06T00:47:35.662 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:35.659+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.660930+0000) 2020-02-06T00:47:35.663 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:35.659+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.661258+0000) 2020-02-06T00:47:35.663 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:35.660+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.661803+0000) 2020-02-06T00:47:35.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:35.660+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.661597+0000) 2020-02-06T00:47:35.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:35.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.661469+0000) 2020-02-06T00:47:35.664 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:35.661+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.662645+0000) 2020-02-06T00:47:35.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:35.666+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:35.667706+0000) 2020-02-06T00:47:35.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:35.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:36.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:36.043+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:36.043771+0000) 2020-02-06T00:47:36.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:36.388+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:36.389224+0000) 2020-02-06T00:47:37.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:37.029+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:37.030632+0000) 2020-02-06T00:47:37.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:37.296+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:37.297052+0000) 2020-02-06T00:47:37.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:37.296+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:37.297216+0000) 2020-02-06T00:47:37.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:37.296+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:37.297612+0000) 2020-02-06T00:47:39.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:39.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:39.674364+0000) 2020-02-06T00:47:39.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:39.921+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:39.922382+0000) 2020-02-06T00:47:40.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:40.523+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.524613+0000) 2020-02-06T00:47:40.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:40.682+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.683608+0000) 2020-02-06T00:47:40.686 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:40.683+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.684654+0000) 2020-02-06T00:47:40.687 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:40.684+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.685353+0000) 2020-02-06T00:47:40.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:40.684+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.685716+0000) 2020-02-06T00:47:40.687 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:40.685+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.686349+0000) 2020-02-06T00:47:40.688 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:40.684+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.686082+0000) 2020-02-06T00:47:40.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:40.684+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.686375+0000) 2020-02-06T00:47:40.689 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:40.685+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.686884+0000) 2020-02-06T00:47:40.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:40.686+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.687504+0000) 2020-02-06T00:47:40.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:40.686+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.687945+0000) 2020-02-06T00:47:40.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:40.687+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.688776+0000) 2020-02-06T00:47:40.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:40.686+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.688400+0000) 2020-02-06T00:47:40.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:40.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.692866+0000) 2020-02-06T00:47:40.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:40.692+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.693221+0000) 2020-02-06T00:47:40.705 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:40.703+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184398 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:40.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:40.782+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.783120+0000) 2020-02-06T00:47:40.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:40.803+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.804658+0000) 2020-02-06T00:47:40.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:40.803+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:40.804658+0000) 2020-02-06T00:47:42.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:42.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:42.907066+0000) 2020-02-06T00:47:44.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:44.101+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:44.102314+0000) 2020-02-06T00:47:45.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:45.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.709605+0000) 2020-02-06T00:47:45.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:45.709+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.710547+0000) 2020-02-06T00:47:45.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:45.709+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.710586+0000) 2020-02-06T00:47:45.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:45.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.711518+0000) 2020-02-06T00:47:45.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:45.711+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.712152+0000) 2020-02-06T00:47:45.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:45.711+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.712794+0000) 2020-02-06T00:47:45.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:45.711+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.713151+0000) 2020-02-06T00:47:45.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:45.712+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.713610+0000) 2020-02-06T00:47:45.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:45.713+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.714321+0000) 2020-02-06T00:47:45.716 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:45.712+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.714155+0000) 2020-02-06T00:47:45.717 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:45.713+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.714573+0000) 2020-02-06T00:47:45.717 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:45.713+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.714966+0000) 2020-02-06T00:47:45.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:45.714+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.715946+0000) 2020-02-06T00:47:45.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:45.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:45.716452+0000) 2020-02-06T00:47:45.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:45.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184435 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:46.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:46.043+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:46.044016+0000) 2020-02-06T00:47:46.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:46.388+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:46.389427+0000) 2020-02-06T00:47:46.758 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:47:46.764 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:47:47.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:47.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:47.030866+0000) 2020-02-06T00:47:47.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:47.296+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:47.297305+0000) 2020-02-06T00:47:47.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:47.296+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:47.297478+0000) 2020-02-06T00:47:47.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:47.296+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:47.297836+0000) 2020-02-06T00:47:49.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:49.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:49.674601+0000) 2020-02-06T00:47:49.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:49.921+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:49.922571+0000) 2020-02-06T00:47:50.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:50.524+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.524845+0000) 2020-02-06T00:47:50.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:50.734+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.735661+0000) 2020-02-06T00:47:50.737 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:50.735+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.736380+0000) 2020-02-06T00:47:50.737 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:50.735+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.736907+0000) 2020-02-06T00:47:50.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:50.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.737480+0000) 2020-02-06T00:47:50.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:50.737+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.738432+0000) 2020-02-06T00:47:50.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:50.737+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.738789+0000) 2020-02-06T00:47:50.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:50.738+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.739376+0000) 2020-02-06T00:47:50.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:50.738+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.739709+0000) 2020-02-06T00:47:50.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:50.737+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.739315+0000) 2020-02-06T00:47:50.742 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:50.739+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.740858+0000) 2020-02-06T00:47:50.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:50.739+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.740759+0000) 2020-02-06T00:47:50.742 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:50.739+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.741123+0000) 2020-02-06T00:47:50.742 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:50.740+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.741712+0000) 2020-02-06T00:47:50.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:50.746+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.747140+0000) 2020-02-06T00:47:50.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:50.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:50.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:50.782+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.783376+0000) 2020-02-06T00:47:50.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:50.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.804929+0000) 2020-02-06T00:47:50.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:50.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:50.804929+0000) 2020-02-06T00:47:52.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:52.906+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:52.907305+0000) 2020-02-06T00:47:54.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:54.112+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:54.112955+0000) 2020-02-06T00:47:55.762 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:47:55.760+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.761318+0000) 2020-02-06T00:47:55.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:47:55.761+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.762252+0000) 2020-02-06T00:47:55.764 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:55.763+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.763673+0000) 2020-02-06T00:47:55.766 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:55.765+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.765405+0000) 2020-02-06T00:47:55.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:47:55.765+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.765916+0000) 2020-02-06T00:47:55.767 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:55.765+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.766417+0000) 2020-02-06T00:47:55.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:55.765+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.766661+0000) 2020-02-06T00:47:55.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:47:55.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.766708+0000) 2020-02-06T00:47:55.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:55.766+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.767245+0000) 2020-02-06T00:47:55.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:55.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.767495+0000) 2020-02-06T00:47:55.769 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:47:55.767+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.767703+0000) 2020-02-06T00:47:55.769 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:55.767+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.767561+0000) 2020-02-06T00:47:55.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:55.767+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.767519+0000) 2020-02-06T00:47:55.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:55.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:55.772645+0000) 2020-02-06T00:47:55.783 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:47:55.782+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184515 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:47:56.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:47:56.043+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:56.044258+0000) 2020-02-06T00:47:56.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:47:56.388+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:56.389616+0000) 2020-02-06T00:47:57.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:47:57.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:57.031132+0000) 2020-02-06T00:47:57.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:47:57.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:57.297493+0000) 2020-02-06T00:47:57.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:47:57.297+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:57.297747+0000) 2020-02-06T00:47:57.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:47:57.297+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:57.298048+0000) 2020-02-06T00:47:59.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:59.673+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:59.674845+0000) 2020-02-06T00:47:59.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:47:59.922+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:47:59.922810+0000) 2020-02-06T00:48:00.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:00.524+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.525073+0000) 2020-02-06T00:48:00.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:00.782+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.783562+0000) 2020-02-06T00:48:00.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:00.786+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.787038+0000) 2020-02-06T00:48:00.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:00.787+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.788712+0000) 2020-02-06T00:48:00.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:00.788+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.789104+0000) 2020-02-06T00:48:00.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:00.789+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.789906+0000) 2020-02-06T00:48:00.791 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:00.789+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.790348+0000) 2020-02-06T00:48:00.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:00.790+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.791419+0000) 2020-02-06T00:48:00.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:00.790+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.791907+0000) 2020-02-06T00:48:00.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:00.791+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.792334+0000) 2020-02-06T00:48:00.793 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:00.792+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.792616+0000) 2020-02-06T00:48:00.794 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:00.792+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.793008+0000) 2020-02-06T00:48:00.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:00.792+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.793279+0000) 2020-02-06T00:48:00.796 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:00.793+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.794042+0000) 2020-02-06T00:48:00.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:00.794+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.794719+0000) 2020-02-06T00:48:00.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:00.798+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.799589+0000) 2020-02-06T00:48:00.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:00.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.805151+0000) 2020-02-06T00:48:00.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:00.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:00.805151+0000) 2020-02-06T00:48:00.811 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:00.809+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:02.908 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:02.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:02.907675+0000) 2020-02-06T00:48:04.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:04.123+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:04.124747+0000) 2020-02-06T00:48:05.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:05.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.816878+0000) 2020-02-06T00:48:05.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:05.816+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.817507+0000) 2020-02-06T00:48:05.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:05.816+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.817409+0000) 2020-02-06T00:48:05.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:05.816+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.817898+0000) 2020-02-06T00:48:05.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:05.817+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.818271+0000) 2020-02-06T00:48:05.820 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:05.818+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.818895+0000) 2020-02-06T00:48:05.820 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:05.817+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.818503+0000) 2020-02-06T00:48:05.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:05.817+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.818814+0000) 2020-02-06T00:48:05.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:05.818+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.819230+0000) 2020-02-06T00:48:05.821 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:05.817+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.818119+0000) 2020-02-06T00:48:05.822 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:05.819+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.820109+0000) 2020-02-06T00:48:05.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:05.819+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.820125+0000) 2020-02-06T00:48:05.822 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:05.820+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.820866+0000) 2020-02-06T00:48:05.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:05.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:05.827649+0000) 2020-02-06T00:48:05.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:05.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184595 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:06.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:06.044+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:06.044489+0000) 2020-02-06T00:48:06.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:06.389+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:06.389843+0000) 2020-02-06T00:48:07.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:07.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:07.031371+0000) 2020-02-06T00:48:07.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:07.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:07.297788+0000) 2020-02-06T00:48:07.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:07.297+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:07.297989+0000) 2020-02-06T00:48:07.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:07.297+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:07.298288+0000) 2020-02-06T00:48:09.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:09.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:09.675123+0000) 2020-02-06T00:48:09.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:09.921+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:09.923059+0000) 2020-02-06T00:48:10.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:10.524+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.525322+0000) 2020-02-06T00:48:10.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:10.783+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.784331+0000) 2020-02-06T00:48:10.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:10.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.805374+0000) 2020-02-06T00:48:10.806 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:10.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.805372+0000) 2020-02-06T00:48:10.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:10.841+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.842375+0000) 2020-02-06T00:48:10.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:10.842+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.842850+0000) 2020-02-06T00:48:10.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:10.842+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.843072+0000) 2020-02-06T00:48:10.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:10.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.843492+0000) 2020-02-06T00:48:10.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:10.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.844054+0000) 2020-02-06T00:48:10.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:10.843+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.844204+0000) 2020-02-06T00:48:10.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:10.844+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.844538+0000) 2020-02-06T00:48:10.846 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:10.844+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.845046+0000) 2020-02-06T00:48:10.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:10.845+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.846409+0000) 2020-02-06T00:48:10.848 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:10.846+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.846677+0000) 2020-02-06T00:48:10.848 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:10.846+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.847070+0000) 2020-02-06T00:48:10.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:10.846+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.847155+0000) 2020-02-06T00:48:10.849 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:10.847+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.847628+0000) 2020-02-06T00:48:10.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:10.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:10.851320+0000) 2020-02-06T00:48:10.863 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:10.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184639 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:12.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:12.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:12.907931+0000) 2020-02-06T00:48:14.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:14.132+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:14.132630+0000) 2020-02-06T00:48:15.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:15.867+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.867693+0000) 2020-02-06T00:48:15.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:15.868+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.868590+0000) 2020-02-06T00:48:15.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:15.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.870023+0000) 2020-02-06T00:48:15.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:15.870+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.870499+0000) 2020-02-06T00:48:15.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:15.870+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.871007+0000) 2020-02-06T00:48:15.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:15.870+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.871004+0000) 2020-02-06T00:48:15.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:15.871+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.871547+0000) 2020-02-06T00:48:15.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:15.871+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.871968+0000) 2020-02-06T00:48:15.873 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:15.872+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.872623+0000) 2020-02-06T00:48:15.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:15.872+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.872871+0000) 2020-02-06T00:48:15.874 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:15.873+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.873980+0000) 2020-02-06T00:48:15.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:15.874+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.874223+0000) 2020-02-06T00:48:15.875 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:15.874+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.874644+0000) 2020-02-06T00:48:15.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:15.878+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:15.878852+0000) 2020-02-06T00:48:15.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:15.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:16.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:16.044+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:16.044896+0000) 2020-02-06T00:48:16.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:16.390+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:16.390135+0000) 2020-02-06T00:48:16.964 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:48:16.971 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:48:17.032 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:17.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:17.031589+0000) 2020-02-06T00:48:17.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:17.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:17.298047+0000) 2020-02-06T00:48:17.299 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:17.297+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:17.298247+0000) 2020-02-06T00:48:17.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:17.298+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:17.298598+0000) 2020-02-06T00:48:19.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:19.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:19.675351+0000) 2020-02-06T00:48:19.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:19.922+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:19.923318+0000) 2020-02-06T00:48:20.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:20.525+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.525619+0000) 2020-02-06T00:48:20.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:20.783+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.784583+0000) 2020-02-06T00:48:20.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:20.804+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.805642+0000) 2020-02-06T00:48:20.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:20.804+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.805639+0000) 2020-02-06T00:48:20.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:20.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.893512+0000) 2020-02-06T00:48:20.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:20.892+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.893724+0000) 2020-02-06T00:48:20.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:20.894+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.894492+0000) 2020-02-06T00:48:20.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:20.894+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.895244+0000) 2020-02-06T00:48:20.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:20.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.896091+0000) 2020-02-06T00:48:20.898 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:20.896+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.897122+0000) 2020-02-06T00:48:20.898 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:20.896+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.897469+0000) 2020-02-06T00:48:20.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:20.896+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.898015+0000) 2020-02-06T00:48:20.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:20.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.898524+0000) 2020-02-06T00:48:20.900 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:20.898+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.899020+0000) 2020-02-06T00:48:20.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:20.898+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.899143+0000) 2020-02-06T00:48:20.901 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:20.898+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.899524+0000) 2020-02-06T00:48:20.901 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:20.900+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.900549+0000) 2020-02-06T00:48:20.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:20.904+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:20.905590+0000) 2020-02-06T00:48:20.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:20.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:22.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:22.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:22.908178+0000) 2020-02-06T00:48:24.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:24.143+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:24.144376+0000) 2020-02-06T00:48:25.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:25.920+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.921421+0000) 2020-02-06T00:48:25.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:25.920+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.920917+0000) 2020-02-06T00:48:25.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:25.921+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.922507+0000) 2020-02-06T00:48:25.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:25.921+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.922662+0000) 2020-02-06T00:48:25.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:25.922+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.923240+0000) 2020-02-06T00:48:25.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:25.923+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.923639+0000) 2020-02-06T00:48:25.925 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:25.922+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.924016+0000) 2020-02-06T00:48:25.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:25.923+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.924450+0000) 2020-02-06T00:48:25.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:25.923+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.924724+0000) 2020-02-06T00:48:25.927 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:25.925+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.925827+0000) 2020-02-06T00:48:25.927 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:25.925+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.926096+0000) 2020-02-06T00:48:25.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:25.925+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.926207+0000) 2020-02-06T00:48:25.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:25.926+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.926829+0000) 2020-02-06T00:48:25.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:25.931+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:25.932997+0000) 2020-02-06T00:48:25.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:25.939+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:26.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:26.045+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:26.045149+0000) 2020-02-06T00:48:26.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:26.389+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:26.390380+0000) 2020-02-06T00:48:27.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:27.030+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:27.031852+0000) 2020-02-06T00:48:27.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:27.298+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:27.298310+0000) 2020-02-06T00:48:27.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:27.298+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:27.298531+0000) 2020-02-06T00:48:27.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:27.298+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:27.298825+0000) 2020-02-06T00:48:29.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:29.674+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:29.675589+0000) 2020-02-06T00:48:29.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:29.922+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:29.923551+0000) 2020-02-06T00:48:30.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:30.525+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.525882+0000) 2020-02-06T00:48:30.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:30.783+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.784831+0000) 2020-02-06T00:48:30.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:30.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.805917+0000) 2020-02-06T00:48:30.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:30.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.805917+0000) 2020-02-06T00:48:30.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:30.944+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.945685+0000) 2020-02-06T00:48:30.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:30.945+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.946770+0000) 2020-02-06T00:48:30.948 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:30.947+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.947560+0000) 2020-02-06T00:48:30.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:30.947+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.948667+0000) 2020-02-06T00:48:30.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:30.947+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.948292+0000) 2020-02-06T00:48:30.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:30.948+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.949317+0000) 2020-02-06T00:48:30.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:30.949+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.949680+0000) 2020-02-06T00:48:30.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:30.949+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.950558+0000) 2020-02-06T00:48:30.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:30.949+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.951030+0000) 2020-02-06T00:48:30.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:30.950+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.951196+0000) 2020-02-06T00:48:30.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:30.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.951616+0000) 2020-02-06T00:48:30.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:30.952+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.952439+0000) 2020-02-06T00:48:30.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:30.951+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.952370+0000) 2020-02-06T00:48:30.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:30.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:30.957185+0000) 2020-02-06T00:48:30.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:30.965+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:32.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:32.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:32.908468+0000) 2020-02-06T00:48:34.157 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:34.155+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:34.156269+0000) 2020-02-06T00:48:35.972 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:35.970+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.971232+0000) 2020-02-06T00:48:35.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:35.972+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.973892+0000) 2020-02-06T00:48:35.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:35.973+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.975004+0000) 2020-02-06T00:48:35.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:35.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.975340+0000) 2020-02-06T00:48:35.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:35.974+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.975568+0000) 2020-02-06T00:48:35.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:35.971+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.973035+0000) 2020-02-06T00:48:35.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:35.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.974382+0000) 2020-02-06T00:48:35.979 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:35.975+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.976584+0000) 2020-02-06T00:48:35.980 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:35.975+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.976676+0000) 2020-02-06T00:48:35.981 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:35.976+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.977231+0000) 2020-02-06T00:48:35.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:35.977+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.978458+0000) 2020-02-06T00:48:35.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:35.976+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.977811+0000) 2020-02-06T00:48:35.981 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:35.976+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.978314+0000) 2020-02-06T00:48:35.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:35.982+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:35.983969+0000) 2020-02-06T00:48:35.994 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:35.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:36.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:36.044+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:36.045361+0000) 2020-02-06T00:48:36.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:36.389+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:36.390566+0000) 2020-02-06T00:48:37.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:37.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:37.032113+0000) 2020-02-06T00:48:37.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:37.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:37.298645+0000) 2020-02-06T00:48:37.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:37.297+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:37.298826+0000) 2020-02-06T00:48:37.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:37.297+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:37.299082+0000) 2020-02-06T00:48:39.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:39.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:39.675824+0000) 2020-02-06T00:48:39.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:39.923+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:39.923800+0000) 2020-02-06T00:48:40.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:40.525+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.526128+0000) 2020-02-06T00:48:40.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:40.784+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.785057+0000) 2020-02-06T00:48:40.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:40.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.806187+0000) 2020-02-06T00:48:40.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:40.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.806187+0000) 2020-02-06T00:48:40.997 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:40.994+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.996004+0000) 2020-02-06T00:48:40.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:40.995+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.997102+0000) 2020-02-06T00:48:40.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:40.997+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.998118+0000) 2020-02-06T00:48:40.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:40.997+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.998747+0000) 2020-02-06T00:48:41.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:40.998+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.999242+0000) 2020-02-06T00:48:41.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:40.998+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.999444+0000) 2020-02-06T00:48:41.000 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:40.998+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.999743+0000) 2020-02-06T00:48:41.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:40.998+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:40.999885+0000) 2020-02-06T00:48:41.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:40.998+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.000312+0000) 2020-02-06T00:48:41.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:40.999+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.000631+0000) 2020-02-06T00:48:41.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:41.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.001416+0000) 2020-02-06T00:48:41.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:41.000+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.001421+0000) 2020-02-06T00:48:41.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:41.000+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.001777+0000) 2020-02-06T00:48:41.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:41.003+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:41.004630+0000) 2020-02-06T00:48:41.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:41.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184878 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:42.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:42.907+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:42.908678+0000) 2020-02-06T00:48:44.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:44.163+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:44.164492+0000) 2020-02-06T00:48:46.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:46.020+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.022049+0000) 2020-02-06T00:48:46.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:46.022+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.023070+0000) 2020-02-06T00:48:46.024 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:46.022+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.023481+0000) 2020-02-06T00:48:46.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:46.022+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.024011+0000) 2020-02-06T00:48:46.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:46.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.025568+0000) 2020-02-06T00:48:46.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:46.024+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.025923+0000) 2020-02-06T00:48:46.027 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:46.025+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.026503+0000) 2020-02-06T00:48:46.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:46.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.026826+0000) 2020-02-06T00:48:46.028 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:46.025+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.027219+0000) 2020-02-06T00:48:46.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:46.026+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.028221+0000) 2020-02-06T00:48:46.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:46.027+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.028425+0000) 2020-02-06T00:48:46.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:46.027+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.028604+0000) 2020-02-06T00:48:46.030 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:46.027+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.028793+0000) 2020-02-06T00:48:46.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:46.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.033252+0000) 2020-02-06T00:48:46.043 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:46.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:46.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:46.044+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.045487+0000) 2020-02-06T00:48:46.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:46.390+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:46.390772+0000) 2020-02-06T00:48:47.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:47.031+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:47.032353+0000) 2020-02-06T00:48:47.135 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:48:47.141 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:48:47.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:47.297+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:47.298887+0000) 2020-02-06T00:48:47.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:47.297+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:47.299089+0000) 2020-02-06T00:48:47.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:47.297+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:47.299350+0000) 2020-02-06T00:48:49.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:49.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:49.676052+0000) 2020-02-06T00:48:49.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:49.923+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:49.924030+0000) 2020-02-06T00:48:50.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:50.525+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:50.526362+0000) 2020-02-06T00:48:50.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:50.784+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:50.785324+0000) 2020-02-06T00:48:50.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:50.805+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:50.806466+0000) 2020-02-06T00:48:50.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:50.805+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:50.806466+0000) 2020-02-06T00:48:51.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:51.047+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.048475+0000) 2020-02-06T00:48:51.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:51.047+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.048791+0000) 2020-02-06T00:48:51.050 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:51.048+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.049947+0000) 2020-02-06T00:48:51.051 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:51.048+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.050313+0000) 2020-02-06T00:48:51.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:51.049+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.050471+0000) 2020-02-06T00:48:51.051 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:51.049+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.050914+0000) 2020-02-06T00:48:51.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:51.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.051558+0000) 2020-02-06T00:48:51.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:51.050+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.051861+0000) 2020-02-06T00:48:51.053 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:51.051+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.052628+0000) 2020-02-06T00:48:51.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:51.051+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.052972+0000) 2020-02-06T00:48:51.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:51.052+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.054008+0000) 2020-02-06T00:48:51.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:51.053+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.054159+0000) 2020-02-06T00:48:51.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:51.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.054283+0000) 2020-02-06T00:48:51.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:51.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:51.057655+0000) 2020-02-06T00:48:51.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:51.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:52.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:52.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:52.908947+0000) 2020-02-06T00:48:54.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:54.172+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:54.173056+0000) 2020-02-06T00:48:56.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:56.044+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.045733+0000) 2020-02-06T00:48:56.077 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:56.075+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.076305+0000) 2020-02-06T00:48:56.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:48:56.075+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.076825+0000) 2020-02-06T00:48:56.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:56.076+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.077268+0000) 2020-02-06T00:48:56.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:56.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.079008+0000) 2020-02-06T00:48:56.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:48:56.077+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.078676+0000) 2020-02-06T00:48:56.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:48:56.077+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.079272+0000) 2020-02-06T00:48:56.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:48:56.078+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.080007+0000) 2020-02-06T00:48:56.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:56.079+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.080473+0000) 2020-02-06T00:48:56.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:56.079+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.080561+0000) 2020-02-06T00:48:56.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:48:56.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.080737+0000) 2020-02-06T00:48:56.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:56.079+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.080964+0000) 2020-02-06T00:48:56.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:56.079+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.081123+0000) 2020-02-06T00:48:56.083 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:48:56.080+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.081836+0000) 2020-02-06T00:48:56.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:56.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.086057+0000) 2020-02-06T00:48:56.098 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:48:56.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 184995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:48:56.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:48:56.390+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:56.390971+0000) 2020-02-06T00:48:57.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:48:57.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:57.032637+0000) 2020-02-06T00:48:57.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:48:57.298+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:57.299159+0000) 2020-02-06T00:48:57.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:48:57.298+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:57.299359+0000) 2020-02-06T00:48:57.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:48:57.298+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:57.299659+0000) 2020-02-06T00:48:59.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:59.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:59.676295+0000) 2020-02-06T00:48:59.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:48:59.923+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:48:59.924302+0000) 2020-02-06T00:49:00.527 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:00.525+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:00.526684+0000) 2020-02-06T00:49:00.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:00.784+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:00.785587+0000) 2020-02-06T00:49:00.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:00.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:00.806733+0000) 2020-02-06T00:49:00.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:00.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:00.806736+0000) 2020-02-06T00:49:01.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:01.102+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.103183+0000) 2020-02-06T00:49:01.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:01.103+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.104383+0000) 2020-02-06T00:49:01.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:01.103+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.104627+0000) 2020-02-06T00:49:01.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:01.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.105135+0000) 2020-02-06T00:49:01.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:01.103+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.104939+0000) 2020-02-06T00:49:01.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:01.104+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.105585+0000) 2020-02-06T00:49:01.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:01.105+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.106577+0000) 2020-02-06T00:49:01.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:01.105+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.106470+0000) 2020-02-06T00:49:01.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:01.105+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.106882+0000) 2020-02-06T00:49:01.108 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:01.104+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.106154+0000) 2020-02-06T00:49:01.108 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:01.106+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.107507+0000) 2020-02-06T00:49:01.109 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:01.106+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.107898+0000) 2020-02-06T00:49:01.109 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:01.107+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.108477+0000) 2020-02-06T00:49:01.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:01.111+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:01.112841+0000) 2020-02-06T00:49:01.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:01.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185037 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:02.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:02.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:02.909212+0000) 2020-02-06T00:49:04.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:04.181+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:04.182214+0000) 2020-02-06T00:49:06.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:06.045+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.046031+0000) 2020-02-06T00:49:06.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:06.127+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.129060+0000) 2020-02-06T00:49:06.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:06.128+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.129487+0000) 2020-02-06T00:49:06.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:06.129+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.130321+0000) 2020-02-06T00:49:06.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:06.129+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.130824+0000) 2020-02-06T00:49:06.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:06.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.131161+0000) 2020-02-06T00:49:06.133 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:06.130+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.131554+0000) 2020-02-06T00:49:06.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:06.130+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.131784+0000) 2020-02-06T00:49:06.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:06.131+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.132786+0000) 2020-02-06T00:49:06.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:06.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.132041+0000) 2020-02-06T00:49:06.134 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:06.131+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.132659+0000) 2020-02-06T00:49:06.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:06.132+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.133522+0000) 2020-02-06T00:49:06.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:06.131+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.133369+0000) 2020-02-06T00:49:06.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:06.133+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.134560+0000) 2020-02-06T00:49:06.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:06.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.140604+0000) 2020-02-06T00:49:06.147 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:06.145+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:06.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:06.391+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:06.391211+0000) 2020-02-06T00:49:07.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:07.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:07.032869+0000) 2020-02-06T00:49:07.300 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:07.298+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:07.299489+0000) 2020-02-06T00:49:07.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:07.298+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:07.299688+0000) 2020-02-06T00:49:07.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:07.298+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:07.299866+0000) 2020-02-06T00:49:09.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:09.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:09.676500+0000) 2020-02-06T00:49:09.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:09.924+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:09.924555+0000) 2020-02-06T00:49:10.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:10.526+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:10.526939+0000) 2020-02-06T00:49:10.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:10.785+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:10.785820+0000) 2020-02-06T00:49:10.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:10.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:10.806999+0000) 2020-02-06T00:49:10.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:10.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:10.807000+0000) 2020-02-06T00:49:11.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:11.153+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.153222+0000) 2020-02-06T00:49:11.154 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:11.152+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.153659+0000) 2020-02-06T00:49:11.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:11.153+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.153921+0000) 2020-02-06T00:49:11.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:11.155+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.155673+0000) 2020-02-06T00:49:11.156 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:11.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.156065+0000) 2020-02-06T00:49:11.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:11.154+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.156340+0000) 2020-02-06T00:49:11.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:11.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.157113+0000) 2020-02-06T00:49:11.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:11.156+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.157544+0000) 2020-02-06T00:49:11.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:11.157+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.157427+0000) 2020-02-06T00:49:11.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:11.156+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.158005+0000) 2020-02-06T00:49:11.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:11.156+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.158321+0000) 2020-02-06T00:49:11.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:11.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.158861+0000) 2020-02-06T00:49:11.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:11.158+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.159478+0000) 2020-02-06T00:49:11.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:11.161+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:11.161745+0000) 2020-02-06T00:49:11.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:11.171+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:12.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:12.908+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:12.909498+0000) 2020-02-06T00:49:14.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:14.189+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:14.190527+0000) 2020-02-06T00:49:16.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:16.045+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.046323+0000) 2020-02-06T00:49:16.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:16.176+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.177769+0000) 2020-02-06T00:49:16.181 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:16.178+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.180315+0000) 2020-02-06T00:49:16.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:16.179+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.180672+0000) 2020-02-06T00:49:16.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:16.179+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.181256+0000) 2020-02-06T00:49:16.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:16.181+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.182411+0000) 2020-02-06T00:49:16.183 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:16.181+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.182833+0000) 2020-02-06T00:49:16.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:16.182+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.183343+0000) 2020-02-06T00:49:16.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:16.182+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.183755+0000) 2020-02-06T00:49:16.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:16.182+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.184057+0000) 2020-02-06T00:49:16.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:16.183+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.184070+0000) 2020-02-06T00:49:16.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:16.183+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.184663+0000) 2020-02-06T00:49:16.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:16.183+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.184770+0000) 2020-02-06T00:49:16.186 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:16.183+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.184729+0000) 2020-02-06T00:49:16.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:16.186+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.187775+0000) 2020-02-06T00:49:16.199 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:16.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:16.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:16.390+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:16.391379+0000) 2020-02-06T00:49:17.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:17.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:17.033122+0000) 2020-02-06T00:49:17.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:17.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:17.299723+0000) 2020-02-06T00:49:17.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:17.299+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:17.299930+0000) 2020-02-06T00:49:17.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:17.299+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:17.300130+0000) 2020-02-06T00:49:17.340 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:49:17.346 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:49:19.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:19.675+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:19.676760+0000) 2020-02-06T00:49:19.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:19.923+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:19.924814+0000) 2020-02-06T00:49:20.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:20.526+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:20.527192+0000) 2020-02-06T00:49:20.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:20.785+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:20.786071+0000) 2020-02-06T00:49:20.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:20.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:20.807287+0000) 2020-02-06T00:49:20.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:20.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:20.807290+0000) 2020-02-06T00:49:21.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:21.203+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.204477+0000) 2020-02-06T00:49:21.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:21.204+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.205390+0000) 2020-02-06T00:49:21.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:21.205+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.206064+0000) 2020-02-06T00:49:21.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:21.205+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.206381+0000) 2020-02-06T00:49:21.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:21.205+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.206854+0000) 2020-02-06T00:49:21.208 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:21.206+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.207446+0000) 2020-02-06T00:49:21.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:21.206+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.207769+0000) 2020-02-06T00:49:21.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:21.207+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.208717+0000) 2020-02-06T00:49:21.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:21.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.209347+0000) 2020-02-06T00:49:21.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:21.208+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.209740+0000) 2020-02-06T00:49:21.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:21.208+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.210069+0000) 2020-02-06T00:49:21.211 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:21.209+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.210801+0000) 2020-02-06T00:49:21.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:21.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.211188+0000) 2020-02-06T00:49:21.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:21.214+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:21.215887+0000) 2020-02-06T00:49:21.227 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:21.225+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:22.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:22.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:22.909766+0000) 2020-02-06T00:49:24.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:24.197+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:24.198241+0000) 2020-02-06T00:49:26.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:26.046+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.046589+0000) 2020-02-06T00:49:26.234 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:26.232+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.233505+0000) 2020-02-06T00:49:26.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:26.233+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.234124+0000) 2020-02-06T00:49:26.235 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:26.234+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.234762+0000) 2020-02-06T00:49:26.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:26.234+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.235026+0000) 2020-02-06T00:49:26.237 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:26.235+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.236471+0000) 2020-02-06T00:49:26.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:26.235+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.236305+0000) 2020-02-06T00:49:26.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:26.235+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.236797+0000) 2020-02-06T00:49:26.238 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:26.236+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.237163+0000) 2020-02-06T00:49:26.239 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:26.236+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.237312+0000) 2020-02-06T00:49:26.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:26.237+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.237734+0000) 2020-02-06T00:49:26.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:26.238+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.238786+0000) 2020-02-06T00:49:26.240 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:26.236+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.237945+0000) 2020-02-06T00:49:26.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:26.237+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.238769+0000) 2020-02-06T00:49:26.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:26.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.245231+0000) 2020-02-06T00:49:26.254 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:26.252+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:26.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:26.390+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:26.391638+0000) 2020-02-06T00:49:27.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:27.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:27.033338+0000) 2020-02-06T00:49:27.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:27.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:27.299951+0000) 2020-02-06T00:49:27.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:27.299+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:27.300160+0000) 2020-02-06T00:49:27.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:27.299+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:27.300400+0000) 2020-02-06T00:49:29.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:29.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:29.677015+0000) 2020-02-06T00:49:29.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:29.923+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:29.925056+0000) 2020-02-06T00:49:30.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:30.527+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:30.527454+0000) 2020-02-06T00:49:30.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:30.785+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:30.786294+0000) 2020-02-06T00:49:30.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:30.806+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:30.807545+0000) 2020-02-06T00:49:30.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:30.806+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:30.807545+0000) 2020-02-06T00:49:31.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:31.257+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.258817+0000) 2020-02-06T00:49:31.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:31.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.260423+0000) 2020-02-06T00:49:31.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:31.260+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.261399+0000) 2020-02-06T00:49:31.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:31.260+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.261562+0000) 2020-02-06T00:49:31.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:31.260+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.261862+0000) 2020-02-06T00:49:31.263 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:31.262+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.262615+0000) 2020-02-06T00:49:31.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:31.262+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.263194+0000) 2020-02-06T00:49:31.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:31.262+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.263363+0000) 2020-02-06T00:49:31.265 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:31.263+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.264301+0000) 2020-02-06T00:49:31.266 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:31.264+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.265045+0000) 2020-02-06T00:49:31.266 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:31.264+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.264980+0000) 2020-02-06T00:49:31.266 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:31.264+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.265151+0000) 2020-02-06T00:49:31.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:31.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.264768+0000) 2020-02-06T00:49:31.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:31.265+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:31.266892+0000) 2020-02-06T00:49:31.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:31.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:32.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:32.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:32.910014+0000) 2020-02-06T00:49:34.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:34.205+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:34.206433+0000) 2020-02-06T00:49:36.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:36.046+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.046882+0000) 2020-02-06T00:49:36.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:36.286+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.287334+0000) 2020-02-06T00:49:36.288 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:36.287+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.288052+0000) 2020-02-06T00:49:36.289 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:36.287+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.288375+0000) 2020-02-06T00:49:36.289 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:36.288+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.288924+0000) 2020-02-06T00:49:36.290 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:36.288+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.289401+0000) 2020-02-06T00:49:36.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:36.288+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.290051+0000) 2020-02-06T00:49:36.291 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:36.290+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.290481+0000) 2020-02-06T00:49:36.291 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:36.289+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.290743+0000) 2020-02-06T00:49:36.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:36.290+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.291126+0000) 2020-02-06T00:49:36.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:36.291+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.291522+0000) 2020-02-06T00:49:36.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:36.291+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.291822+0000) 2020-02-06T00:49:36.293 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:36.291+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.292063+0000) 2020-02-06T00:49:36.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:36.291+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.292569+0000) 2020-02-06T00:49:36.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:36.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.300282+0000) 2020-02-06T00:49:36.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:36.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:36.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:36.391+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:36.391892+0000) 2020-02-06T00:49:37.034 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:37.032+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:37.033592+0000) 2020-02-06T00:49:37.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:37.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:37.300188+0000) 2020-02-06T00:49:37.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:37.299+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:37.300406+0000) 2020-02-06T00:49:37.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:37.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:37.300712+0000) 2020-02-06T00:49:39.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:39.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:39.677287+0000) 2020-02-06T00:49:39.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:39.924+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:39.925338+0000) 2020-02-06T00:49:40.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:40.527+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:40.527650+0000) 2020-02-06T00:49:40.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:40.785+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:40.786581+0000) 2020-02-06T00:49:40.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:40.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:40.807815+0000) 2020-02-06T00:49:40.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:40.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:40.807814+0000) 2020-02-06T00:49:41.313 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:41.312+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.312827+0000) 2020-02-06T00:49:41.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:41.312+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.313433+0000) 2020-02-06T00:49:41.315 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:41.313+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.314439+0000) 2020-02-06T00:49:41.315 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:41.313+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.314768+0000) 2020-02-06T00:49:41.316 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:41.314+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.315405+0000) 2020-02-06T00:49:41.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:41.314+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.315314+0000) 2020-02-06T00:49:41.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:41.315+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.315803+0000) 2020-02-06T00:49:41.317 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:41.314+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.316020+0000) 2020-02-06T00:49:41.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:41.316+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.316424+0000) 2020-02-06T00:49:41.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:41.316+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.317002+0000) 2020-02-06T00:49:41.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:41.316+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.317199+0000) 2020-02-06T00:49:41.319 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:41.316+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.317815+0000) 2020-02-06T00:49:41.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:41.317+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.317472+0000) 2020-02-06T00:49:41.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:41.322+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:41.323205+0000) 2020-02-06T00:49:41.333 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:41.332+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:42.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:42.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:42.910285+0000) 2020-02-06T00:49:44.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:44.213+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:44.214777+0000) 2020-02-06T00:49:46.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:46.046+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.047143+0000) 2020-02-06T00:49:46.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:46.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.338580+0000) 2020-02-06T00:49:46.341 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:46.339+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.340448+0000) 2020-02-06T00:49:46.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:46.340+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.340759+0000) 2020-02-06T00:49:46.342 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:46.340+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.341335+0000) 2020-02-06T00:49:46.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:46.341+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.342394+0000) 2020-02-06T00:49:46.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:46.341+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.342879+0000) 2020-02-06T00:49:46.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:46.343+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.343808+0000) 2020-02-06T00:49:46.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:46.342+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.343967+0000) 2020-02-06T00:49:46.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:46.343+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.344444+0000) 2020-02-06T00:49:46.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:46.343+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.344584+0000) 2020-02-06T00:49:46.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:46.344+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.344999+0000) 2020-02-06T00:49:46.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:46.344+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.344832+0000) 2020-02-06T00:49:46.347 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:46.344+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.345229+0000) 2020-02-06T00:49:46.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:46.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.348812+0000) 2020-02-06T00:49:46.361 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:46.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:46.393 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:46.391+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:46.392138+0000) 2020-02-06T00:49:47.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:47.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:47.033803+0000) 2020-02-06T00:49:47.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:47.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:47.300481+0000) 2020-02-06T00:49:47.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:47.300+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:47.300719+0000) 2020-02-06T00:49:47.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:47.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:47.300923+0000) 2020-02-06T00:49:47.528 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:49:47.534 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:49:49.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:49.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:49.677530+0000) 2020-02-06T00:49:49.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:49.925+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:49.925542+0000) 2020-02-06T00:49:50.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:50.527+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:50.527887+0000) 2020-02-06T00:49:50.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:50.786+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:50.786876+0000) 2020-02-06T00:49:50.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:50.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:50.808117+0000) 2020-02-06T00:49:50.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:50.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:50.808115+0000) 2020-02-06T00:49:51.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:51.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.366346+0000) 2020-02-06T00:49:51.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:51.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.366701+0000) 2020-02-06T00:49:51.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:51.366+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.367217+0000) 2020-02-06T00:49:51.368 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:51.367+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.367713+0000) 2020-02-06T00:49:51.369 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:51.367+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.368576+0000) 2020-02-06T00:49:51.369 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:51.367+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.368484+0000) 2020-02-06T00:49:51.370 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:51.368+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.369113+0000) 2020-02-06T00:49:51.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:51.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.368968+0000) 2020-02-06T00:49:51.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:51.368+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.369863+0000) 2020-02-06T00:49:51.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:51.369+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.370298+0000) 2020-02-06T00:49:51.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:51.370+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.370797+0000) 2020-02-06T00:49:51.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:51.370+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.371351+0000) 2020-02-06T00:49:51.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:51.371+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.371885+0000) 2020-02-06T00:49:51.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:51.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:51.375249+0000) 2020-02-06T00:49:51.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:51.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185437 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:52.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:52.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:52.910550+0000) 2020-02-06T00:49:54.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:54.222+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:54.222933+0000) 2020-02-06T00:49:56.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:56.047+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.047382+0000) 2020-02-06T00:49:56.392 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:49:56.390+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.391404+0000) 2020-02-06T00:49:56.393 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:56.391+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.392928+0000) 2020-02-06T00:49:56.394 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:49:56.392+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.393729+0000) 2020-02-06T00:49:56.395 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:49:56.393+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.394265+0000) 2020-02-06T00:49:56.395 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:49:56.394+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.394708+0000) 2020-02-06T00:49:56.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:56.394+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.394876+0000) 2020-02-06T00:49:56.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:49:56.395+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.395661+0000) 2020-02-06T00:49:56.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:56.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.396010+0000) 2020-02-06T00:49:56.397 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:49:56.395+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.396891+0000) 2020-02-06T00:49:56.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:56.396+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.397375+0000) 2020-02-06T00:49:56.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:56.396+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.397161+0000) 2020-02-06T00:49:56.399 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:49:56.396+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.397821+0000) 2020-02-06T00:49:56.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:56.397+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.398016+0000) 2020-02-06T00:49:56.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:56.399+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:56.400892+0000) 2020-02-06T00:49:56.414 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:49:56.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:49:57.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:49:57.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:57.034032+0000) 2020-02-06T00:49:57.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:49:57.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:57.300710+0000) 2020-02-06T00:49:57.308 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:49:57.300+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:57.300900+0000) 2020-02-06T00:49:57.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:49:57.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:57.301099+0000) 2020-02-06T00:49:59.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:59.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:59.677781+0000) 2020-02-06T00:49:59.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:49:59.925+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:49:59.925776+0000) 2020-02-06T00:50:00.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:00.527+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:00.528138+0000) 2020-02-06T00:50:00.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:00.786+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:00.787152+0000) 2020-02-06T00:50:00.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:00.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:00.808389+0000) 2020-02-06T00:50:00.809 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:00.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:00.808390+0000) 2020-02-06T00:50:01.419 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:01.417+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.418441+0000) 2020-02-06T00:50:01.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:01.417+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.419310+0000) 2020-02-06T00:50:01.422 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:01.419+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.420594+0000) 2020-02-06T00:50:01.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:01.419+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.420941+0000) 2020-02-06T00:50:01.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:01.418+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.420153+0000) 2020-02-06T00:50:01.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:01.420+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.421514+0000) 2020-02-06T00:50:01.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:01.421+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.422207+0000) 2020-02-06T00:50:01.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:01.421+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.422923+0000) 2020-02-06T00:50:01.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:01.421+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.423111+0000) 2020-02-06T00:50:01.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:01.422+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.423533+0000) 2020-02-06T00:50:01.424 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:01.422+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.424053+0000) 2020-02-06T00:50:01.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:01.423+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.424541+0000) 2020-02-06T00:50:01.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:01.424+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.425831+0000) 2020-02-06T00:50:01.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:01.428+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:01.430045+0000) 2020-02-06T00:50:01.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:01.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:02.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:02.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:02.910806+0000) 2020-02-06T00:50:03.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:03.141+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:03.142720+0000) 2020-02-06T00:50:04.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:04.230+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:04.230970+0000) 2020-02-06T00:50:06.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:06.046+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.047671+0000) 2020-02-06T00:50:06.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:06.445+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.445333+0000) 2020-02-06T00:50:06.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:06.444+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.446316+0000) 2020-02-06T00:50:06.447 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:06.445+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.446496+0000) 2020-02-06T00:50:06.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:06.446+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.446833+0000) 2020-02-06T00:50:06.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:06.447+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.447254+0000) 2020-02-06T00:50:06.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:06.445+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.447147+0000) 2020-02-06T00:50:06.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:06.448+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.448202+0000) 2020-02-06T00:50:06.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:06.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.448500+0000) 2020-02-06T00:50:06.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:06.446+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.447926+0000) 2020-02-06T00:50:06.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:06.448+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.448728+0000) 2020-02-06T00:50:06.450 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:06.449+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.449233+0000) 2020-02-06T00:50:06.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:06.448+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.449979+0000) 2020-02-06T00:50:06.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:06.448+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.450073+0000) 2020-02-06T00:50:06.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:06.453+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:06.454054+0000) 2020-02-06T00:50:06.466 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:06.464+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:07.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:07.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:07.034273+0000) 2020-02-06T00:50:07.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:07.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:07.300943+0000) 2020-02-06T00:50:07.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:07.299+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:07.301213+0000) 2020-02-06T00:50:07.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:07.299+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:07.301340+0000) 2020-02-06T00:50:09.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:09.676+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:09.678010+0000) 2020-02-06T00:50:09.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:09.924+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:09.926009+0000) 2020-02-06T00:50:10.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:10.526+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:10.528377+0000) 2020-02-06T00:50:10.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:10.786+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:10.787333+0000) 2020-02-06T00:50:10.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:10.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:10.808664+0000) 2020-02-06T00:50:10.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:10.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:10.808664+0000) 2020-02-06T00:50:11.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:11.469+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.470638+0000) 2020-02-06T00:50:11.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:11.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.471453+0000) 2020-02-06T00:50:11.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:11.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.471982+0000) 2020-02-06T00:50:11.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:11.471+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.473347+0000) 2020-02-06T00:50:11.475 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:11.472+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.474026+0000) 2020-02-06T00:50:11.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:11.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.474473+0000) 2020-02-06T00:50:11.476 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:11.474+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.475067+0000) 2020-02-06T00:50:11.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:11.474+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.475494+0000) 2020-02-06T00:50:11.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:11.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.475769+0000) 2020-02-06T00:50:11.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:11.474+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.476133+0000) 2020-02-06T00:50:11.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:11.476+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.477505+0000) 2020-02-06T00:50:11.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:11.475+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.477079+0000) 2020-02-06T00:50:11.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:11.476+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.477519+0000) 2020-02-06T00:50:11.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:11.478+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:11.479274+0000) 2020-02-06T00:50:11.493 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:11.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185600 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:12.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:12.909+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:12.911039+0000) 2020-02-06T00:50:13.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:13.141+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:13.142967+0000) 2020-02-06T00:50:14.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:14.238+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:14.239261+0000) 2020-02-06T00:50:16.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:16.046+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.047887+0000) 2020-02-06T00:50:16.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:16.496+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.497633+0000) 2020-02-06T00:50:16.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:16.497+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.498509+0000) 2020-02-06T00:50:16.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:16.499+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.500183+0000) 2020-02-06T00:50:16.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:16.500+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.501509+0000) 2020-02-06T00:50:16.506 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:16.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.501176+0000) 2020-02-06T00:50:16.506 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:16.500+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.501672+0000) 2020-02-06T00:50:16.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:16.501+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.502316+0000) 2020-02-06T00:50:16.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:16.498+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.499569+0000) 2020-02-06T00:50:16.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:16.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.501015+0000) 2020-02-06T00:50:16.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:16.500+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.502207+0000) 2020-02-06T00:50:16.507 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:16.501+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.502849+0000) 2020-02-06T00:50:16.507 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:16.502+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.503885+0000) 2020-02-06T00:50:16.507 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:16.502+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.503663+0000) 2020-02-06T00:50:16.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:16.505+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:16.506782+0000) 2020-02-06T00:50:16.520 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:16.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:17.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:17.033+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:17.034487+0000) 2020-02-06T00:50:17.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:17.299+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:17.301111+0000) 2020-02-06T00:50:17.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:17.299+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:17.301405+0000) 2020-02-06T00:50:17.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:17.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:17.301640+0000) 2020-02-06T00:50:17.717 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:50:17.723 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:50:19.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:19.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:19.678221+0000) 2020-02-06T00:50:19.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:19.925+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:19.926307+0000) 2020-02-06T00:50:20.529 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:20.527+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:20.528724+0000) 2020-02-06T00:50:20.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:20.786+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:20.787565+0000) 2020-02-06T00:50:20.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:20.807+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:20.808878+0000) 2020-02-06T00:50:20.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:20.807+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:20.808917+0000) 2020-02-06T00:50:21.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:21.522+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.524017+0000) 2020-02-06T00:50:21.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:21.523+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.524942+0000) 2020-02-06T00:50:21.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:21.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.525561+0000) 2020-02-06T00:50:21.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:21.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.526012+0000) 2020-02-06T00:50:21.527 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:21.525+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.526277+0000) 2020-02-06T00:50:21.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:21.525+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.526864+0000) 2020-02-06T00:50:21.528 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:21.525+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.526867+0000) 2020-02-06T00:50:21.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:21.526+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.527357+0000) 2020-02-06T00:50:21.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:21.526+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.527687+0000) 2020-02-06T00:50:21.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:21.527+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.528593+0000) 2020-02-06T00:50:21.529 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:21.527+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.528782+0000) 2020-02-06T00:50:21.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:21.528+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.529761+0000) 2020-02-06T00:50:21.531 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:21.528+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.529961+0000) 2020-02-06T00:50:21.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:21.534+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:21.535169+0000) 2020-02-06T00:50:21.546 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:21.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185683 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:22.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:22.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:22.911296+0000) 2020-02-06T00:50:23.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:23.142+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:23.143195+0000) 2020-02-06T00:50:24.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:24.246+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:24.247267+0000) 2020-02-06T00:50:26.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:26.047+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.048221+0000) 2020-02-06T00:50:26.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:26.551+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.552249+0000) 2020-02-06T00:50:26.553 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:26.550+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.552167+0000) 2020-02-06T00:50:26.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:26.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.553259+0000) 2020-02-06T00:50:26.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:26.552+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.553906+0000) 2020-02-06T00:50:26.555 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:26.552+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.554303+0000) 2020-02-06T00:50:26.555 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:26.553+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.554974+0000) 2020-02-06T00:50:26.556 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:26.553+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.555283+0000) 2020-02-06T00:50:26.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:26.554+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.555789+0000) 2020-02-06T00:50:26.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:26.554+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.556178+0000) 2020-02-06T00:50:26.557 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:26.555+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.557019+0000) 2020-02-06T00:50:26.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:26.556+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.557542+0000) 2020-02-06T00:50:26.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:26.556+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.557690+0000) 2020-02-06T00:50:26.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:26.556+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.557895+0000) 2020-02-06T00:50:26.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:26.561+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:26.562133+0000) 2020-02-06T00:50:26.573 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:26.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:27.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:27.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:27.034729+0000) 2020-02-06T00:50:27.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:27.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:27.301382+0000) 2020-02-06T00:50:27.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:27.300+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:27.301629+0000) 2020-02-06T00:50:27.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:27.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:27.301824+0000) 2020-02-06T00:50:29.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:29.677+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:29.678457+0000) 2020-02-06T00:50:29.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:29.925+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:29.926544+0000) 2020-02-06T00:50:30.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:30.528+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:30.528995+0000) 2020-02-06T00:50:30.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:30.786+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:30.787850+0000) 2020-02-06T00:50:30.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:30.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:30.809145+0000) 2020-02-06T00:50:30.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:30.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:30.809164+0000) 2020-02-06T00:50:31.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:31.577+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.579196+0000) 2020-02-06T00:50:31.581 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:31.578+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.579707+0000) 2020-02-06T00:50:31.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:31.579+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.580415+0000) 2020-02-06T00:50:31.582 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:31.579+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.580775+0000) 2020-02-06T00:50:31.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:31.580+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.581387+0000) 2020-02-06T00:50:31.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:31.581+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.582624+0000) 2020-02-06T00:50:31.584 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:31.582+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.583705+0000) 2020-02-06T00:50:31.584 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:31.582+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.583847+0000) 2020-02-06T00:50:31.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:31.582+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.583811+0000) 2020-02-06T00:50:31.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:31.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.584431+0000) 2020-02-06T00:50:31.586 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:31.583+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.584175+0000) 2020-02-06T00:50:31.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:31.583+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.584489+0000) 2020-02-06T00:50:31.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:31.583+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.585416+0000) 2020-02-06T00:50:31.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:31.586+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:31.587381+0000) 2020-02-06T00:50:31.598 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:31.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:32.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:32.910+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:32.911545+0000) 2020-02-06T00:50:33.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:33.142+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:33.143414+0000) 2020-02-06T00:50:34.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:34.254+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:34.255389+0000) 2020-02-06T00:50:36.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:36.047+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.048517+0000) 2020-02-06T00:50:36.604 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:36.602+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.603415+0000) 2020-02-06T00:50:36.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:36.603+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.605214+0000) 2020-02-06T00:50:36.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:36.604+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.605745+0000) 2020-02-06T00:50:36.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:36.605+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.606256+0000) 2020-02-06T00:50:36.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:36.605+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.606835+0000) 2020-02-06T00:50:36.608 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:36.606+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.607303+0000) 2020-02-06T00:50:36.608 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:36.606+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.607483+0000) 2020-02-06T00:50:36.608 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:36.606+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.607891+0000) 2020-02-06T00:50:36.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:36.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.608252+0000) 2020-02-06T00:50:36.609 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:36.607+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.608759+0000) 2020-02-06T00:50:36.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:36.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.608970+0000) 2020-02-06T00:50:36.610 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:36.607+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.609190+0000) 2020-02-06T00:50:36.610 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:36.607+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.609302+0000) 2020-02-06T00:50:36.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:36.611+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:36.612477+0000) 2020-02-06T00:50:36.623 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:36.621+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:37.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:37.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:37.034940+0000) 2020-02-06T00:50:37.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:37.300+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:37.301636+0000) 2020-02-06T00:50:37.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:37.300+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:37.301868+0000) 2020-02-06T00:50:37.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:37.300+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:37.302020+0000) 2020-02-06T00:50:39.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:39.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:39.678693+0000) 2020-02-06T00:50:39.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:39.926+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:39.926794+0000) 2020-02-06T00:50:40.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:40.528+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:40.529228+0000) 2020-02-06T00:50:40.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:40.787+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:40.788115+0000) 2020-02-06T00:50:40.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:40.808+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:40.809374+0000) 2020-02-06T00:50:40.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:40.808+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:40.809444+0000) 2020-02-06T00:50:41.629 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:41.627+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.628546+0000) 2020-02-06T00:50:41.630 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:41.628+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.629183+0000) 2020-02-06T00:50:41.631 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:41.629+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.630828+0000) 2020-02-06T00:50:41.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:41.630+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.631211+0000) 2020-02-06T00:50:41.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:41.630+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.631800+0000) 2020-02-06T00:50:41.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:41.631+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.632465+0000) 2020-02-06T00:50:41.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:41.631+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.632463+0000) 2020-02-06T00:50:41.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:41.632+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.633661+0000) 2020-02-06T00:50:41.635 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:41.631+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.633285+0000) 2020-02-06T00:50:41.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:41.632+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.633688+0000) 2020-02-06T00:50:41.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:41.632+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.634278+0000) 2020-02-06T00:50:41.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:41.633+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.634701+0000) 2020-02-06T00:50:41.636 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:41.633+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.635070+0000) 2020-02-06T00:50:41.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:41.636+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:41.638015+0000) 2020-02-06T00:50:41.652 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:41.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185849 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:42.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:42.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:42.911817+0000) 2020-02-06T00:50:43.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:43.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:43.143648+0000) 2020-02-06T00:50:44.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:44.262+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:44.263401+0000) 2020-02-06T00:50:46.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:46.048+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.048789+0000) 2020-02-06T00:50:46.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:46.653+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.654563+0000) 2020-02-06T00:50:46.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:46.654+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.655107+0000) 2020-02-06T00:50:46.657 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:46.654+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.656009+0000) 2020-02-06T00:50:46.657 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:46.655+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.656814+0000) 2020-02-06T00:50:46.658 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:46.656+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.657331+0000) 2020-02-06T00:50:46.658 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:46.656+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.657703+0000) 2020-02-06T00:50:46.659 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:46.657+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.658639+0000) 2020-02-06T00:50:46.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:46.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.658470+0000) 2020-02-06T00:50:46.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:46.657+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.658958+0000) 2020-02-06T00:50:46.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:46.657+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.659096+0000) 2020-02-06T00:50:46.661 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:46.657+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.659286+0000) 2020-02-06T00:50:46.661 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:46.658+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.659658+0000) 2020-02-06T00:50:46.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:46.658+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.659286+0000) 2020-02-06T00:50:46.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:46.662+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:46.663153+0000) 2020-02-06T00:50:46.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:46.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:47.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:47.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:47.035180+0000) 2020-02-06T00:50:47.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:47.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:47.301868+0000) 2020-02-06T00:50:47.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:47.301+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:47.302089+0000) 2020-02-06T00:50:47.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:47.301+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:47.302213+0000) 2020-02-06T00:50:47.896 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:50:47.902 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:50:49.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:49.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:49.678924+0000) 2020-02-06T00:50:49.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:49.926+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:49.927019+0000) 2020-02-06T00:50:50.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:50.529+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:50.529492+0000) 2020-02-06T00:50:50.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:50.787+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:50.788304+0000) 2020-02-06T00:50:50.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:50.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:50.809621+0000) 2020-02-06T00:50:50.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:50.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:50.809696+0000) 2020-02-06T00:50:51.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:51.678+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.679711+0000) 2020-02-06T00:50:51.681 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:51.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.680318+0000) 2020-02-06T00:50:51.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:51.679+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.680726+0000) 2020-02-06T00:50:51.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:51.679+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.681054+0000) 2020-02-06T00:50:51.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:51.680+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.681660+0000) 2020-02-06T00:50:51.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:51.683+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.683470+0000) 2020-02-06T00:50:51.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:51.683+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.683919+0000) 2020-02-06T00:50:51.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:51.683+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.684422+0000) 2020-02-06T00:50:51.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:51.684+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.685317+0000) 2020-02-06T00:50:51.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:51.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.686059+0000) 2020-02-06T00:50:51.687 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:51.685+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.685898+0000) 2020-02-06T00:50:51.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:51.685+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.686237+0000) 2020-02-06T00:50:51.688 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:51.686+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.686441+0000) 2020-02-06T00:50:51.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:51.688+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:51.690039+0000) 2020-02-06T00:50:51.701 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:51.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185932 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:52.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:52.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:52.912089+0000) 2020-02-06T00:50:53.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:53.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:53.143875+0000) 2020-02-06T00:50:54.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:54.271+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:54.271569+0000) 2020-02-06T00:50:56.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:56.048+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.049056+0000) 2020-02-06T00:50:56.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:50:56.705+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.706247+0000) 2020-02-06T00:50:56.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:56.705+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.707045+0000) 2020-02-06T00:50:56.708 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:56.707+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.707734+0000) 2020-02-06T00:50:56.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:56.707+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.708549+0000) 2020-02-06T00:50:56.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:50:56.707+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.708759+0000) 2020-02-06T00:50:56.710 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:50:56.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.709531+0000) 2020-02-06T00:50:56.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:56.709+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.710071+0000) 2020-02-06T00:50:56.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:50:56.709+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.710231+0000) 2020-02-06T00:50:56.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:50:56.709+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.710203+0000) 2020-02-06T00:50:56.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:56.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.710674+0000) 2020-02-06T00:50:56.712 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:56.710+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.710987+0000) 2020-02-06T00:50:56.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:50:56.711+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.711678+0000) 2020-02-06T00:50:56.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:50:56.710+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.711396+0000) 2020-02-06T00:50:56.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:56.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:56.716712+0000) 2020-02-06T00:50:56.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:50:56.724+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 185970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:50:57.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:50:57.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:57.035389+0000) 2020-02-06T00:50:57.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:50:57.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:57.302108+0000) 2020-02-06T00:50:57.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:50:57.301+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:57.302297+0000) 2020-02-06T00:50:57.303 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:50:57.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:57.302414+0000) 2020-02-06T00:50:59.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:59.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:59.679194+0000) 2020-02-06T00:50:59.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:50:59.927+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:50:59.927276+0000) 2020-02-06T00:51:00.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:00.529+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:00.529739+0000) 2020-02-06T00:51:00.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:00.788+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:00.788547+0000) 2020-02-06T00:51:00.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:00.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:00.809916+0000) 2020-02-06T00:51:00.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:00.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:00.809942+0000) 2020-02-06T00:51:01.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:01.731+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.731968+0000) 2020-02-06T00:51:01.733 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:01.732+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.732519+0000) 2020-02-06T00:51:01.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:01.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.733359+0000) 2020-02-06T00:51:01.734 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:01.732+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.733007+0000) 2020-02-06T00:51:01.735 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:01.733+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.733681+0000) 2020-02-06T00:51:01.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:01.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.734256+0000) 2020-02-06T00:51:01.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:01.734+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.734640+0000) 2020-02-06T00:51:01.736 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:01.734+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.735119+0000) 2020-02-06T00:51:01.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:01.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.735892+0000) 2020-02-06T00:51:01.737 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:01.736+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.736546+0000) 2020-02-06T00:51:01.737 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:01.736+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.737121+0000) 2020-02-06T00:51:01.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:01.737+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.737602+0000) 2020-02-06T00:51:01.738 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:01.737+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.738018+0000) 2020-02-06T00:51:01.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:01.742+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:01.742589+0000) 2020-02-06T00:51:01.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:01.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186015 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:02.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:02.911+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:02.912360+0000) 2020-02-06T00:51:03.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:03.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:03.144139+0000) 2020-02-06T00:51:04.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:04.278+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:04.279750+0000) 2020-02-06T00:51:06.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:06.048+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.049308+0000) 2020-02-06T00:51:06.759 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:06.757+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.758409+0000) 2020-02-06T00:51:06.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:06.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.759035+0000) 2020-02-06T00:51:06.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:06.758+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.759685+0000) 2020-02-06T00:51:06.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:06.760+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.761373+0000) 2020-02-06T00:51:06.763 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:06.760+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.761859+0000) 2020-02-06T00:51:06.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:06.761+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.762124+0000) 2020-02-06T00:51:06.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:06.761+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.762376+0000) 2020-02-06T00:51:06.764 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:06.763+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.763732+0000) 2020-02-06T00:51:06.764 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:06.762+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.763233+0000) 2020-02-06T00:51:06.765 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:06.763+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.764296+0000) 2020-02-06T00:51:06.765 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:06.763+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.764362+0000) 2020-02-06T00:51:06.765 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:06.764+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.764603+0000) 2020-02-06T00:51:06.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:06.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.766580+0000) 2020-02-06T00:51:06.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:06.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:06.766826+0000) 2020-02-06T00:51:06.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:06.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:07.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:07.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:07.035598+0000) 2020-02-06T00:51:07.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:07.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:07.302330+0000) 2020-02-06T00:51:07.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:07.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:07.302498+0000) 2020-02-06T00:51:07.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:07.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:07.302680+0000) 2020-02-06T00:51:09.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:09.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:09.679426+0000) 2020-02-06T00:51:09.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:09.926+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:09.927505+0000) 2020-02-06T00:51:10.531 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:10.529+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:10.529988+0000) 2020-02-06T00:51:10.789 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:10.787+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:10.788816+0000) 2020-02-06T00:51:10.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:10.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:10.810163+0000) 2020-02-06T00:51:10.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:10.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:10.810176+0000) 2020-02-06T00:51:11.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:11.782+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.783831+0000) 2020-02-06T00:51:11.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:11.783+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.784551+0000) 2020-02-06T00:51:11.785 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:11.783+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.785013+0000) 2020-02-06T00:51:11.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:11.785+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.785805+0000) 2020-02-06T00:51:11.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:11.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.786264+0000) 2020-02-06T00:51:11.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:11.786+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.787001+0000) 2020-02-06T00:51:11.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:11.786+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.787402+0000) 2020-02-06T00:51:11.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:11.788+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.788626+0000) 2020-02-06T00:51:11.790 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:11.788+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.788798+0000) 2020-02-06T00:51:11.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:11.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.789056+0000) 2020-02-06T00:51:11.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:11.789+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.789974+0000) 2020-02-06T00:51:11.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:11.789+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.790258+0000) 2020-02-06T00:51:11.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:11.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.790711+0000) 2020-02-06T00:51:11.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:11.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:11.791716+0000) 2020-02-06T00:51:11.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:11.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:12.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:12.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:12.912667+0000) 2020-02-06T00:51:13.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:13.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:13.144315+0000) 2020-02-06T00:51:14.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:14.287+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:14.288190+0000) 2020-02-06T00:51:16.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:16.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.049604+0000) 2020-02-06T00:51:16.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:16.810+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.811361+0000) 2020-02-06T00:51:16.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:16.810+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.811905+0000) 2020-02-06T00:51:16.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:16.811+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.812239+0000) 2020-02-06T00:51:16.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:16.811+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.812499+0000) 2020-02-06T00:51:16.814 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:16.812+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.813082+0000) 2020-02-06T00:51:16.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:16.813+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.814399+0000) 2020-02-06T00:51:16.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:16.813+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.814163+0000) 2020-02-06T00:51:16.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:16.813+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.814738+0000) 2020-02-06T00:51:16.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:16.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.815036+0000) 2020-02-06T00:51:16.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:16.814+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.815624+0000) 2020-02-06T00:51:16.817 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:16.815+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.815522+0000) 2020-02-06T00:51:16.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:16.816+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.816858+0000) 2020-02-06T00:51:16.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:16.816+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.817174+0000) 2020-02-06T00:51:16.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:16.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:16.821655+0000) 2020-02-06T00:51:16.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:16.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186136 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:17.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:17.034+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:17.035758+0000) 2020-02-06T00:51:17.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:17.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:17.302547+0000) 2020-02-06T00:51:17.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:17.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:17.302712+0000) 2020-02-06T00:51:17.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:17.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:17.302861+0000) 2020-02-06T00:51:18.101 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:51:18.108 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:51:19.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:19.678+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:19.679669+0000) 2020-02-06T00:51:19.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:19.926+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:19.927736+0000) 2020-02-06T00:51:20.531 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:20.530+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:20.530234+0000) 2020-02-06T00:51:20.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:20.788+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:20.789135+0000) 2020-02-06T00:51:20.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:20.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:20.810404+0000) 2020-02-06T00:51:20.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:20.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:20.810413+0000) 2020-02-06T00:51:21.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:21.836+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.837546+0000) 2020-02-06T00:51:21.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:21.837+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.838364+0000) 2020-02-06T00:51:21.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:21.837+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.838480+0000) 2020-02-06T00:51:21.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:21.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.838942+0000) 2020-02-06T00:51:21.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:21.838+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.839169+0000) 2020-02-06T00:51:21.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:21.839+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.839596+0000) 2020-02-06T00:51:21.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:21.839+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.840122+0000) 2020-02-06T00:51:21.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:21.838+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.839542+0000) 2020-02-06T00:51:21.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:21.838+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.839332+0000) 2020-02-06T00:51:21.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:21.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.840424+0000) 2020-02-06T00:51:21.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:21.840+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.840765+0000) 2020-02-06T00:51:21.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:21.840+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.841405+0000) 2020-02-06T00:51:21.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:21.841+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.841757+0000) 2020-02-06T00:51:21.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:21.845+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:21.846521+0000) 2020-02-06T00:51:21.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:21.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186180 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:22.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:22.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:22.912927+0000) 2020-02-06T00:51:23.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:23.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:23.144538+0000) 2020-02-06T00:51:24.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:24.294+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:24.295289+0000) 2020-02-06T00:51:26.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:26.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.049954+0000) 2020-02-06T00:51:26.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:26.862+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.863842+0000) 2020-02-06T00:51:26.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:26.863+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.864893+0000) 2020-02-06T00:51:26.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:26.864+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.865515+0000) 2020-02-06T00:51:26.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:26.865+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.865543+0000) 2020-02-06T00:51:26.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:26.865+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.866340+0000) 2020-02-06T00:51:26.867 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:26.865+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.865815+0000) 2020-02-06T00:51:26.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:26.866+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.866502+0000) 2020-02-06T00:51:26.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:26.866+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.866815+0000) 2020-02-06T00:51:26.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:26.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.867273+0000) 2020-02-06T00:51:26.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:26.867+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.868075+0000) 2020-02-06T00:51:26.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:26.867+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.868439+0000) 2020-02-06T00:51:26.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:26.867+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.868714+0000) 2020-02-06T00:51:26.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:26.868+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.869131+0000) 2020-02-06T00:51:26.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:26.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:26.874175+0000) 2020-02-06T00:51:26.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:26.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:27.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:27.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:27.036005+0000) 2020-02-06T00:51:27.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:27.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:27.302779+0000) 2020-02-06T00:51:27.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:27.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:27.302959+0000) 2020-02-06T00:51:27.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:27.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:27.303071+0000) 2020-02-06T00:51:29.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:29.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:29.679903+0000) 2020-02-06T00:51:29.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:29.927+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:29.927969+0000) 2020-02-06T00:51:30.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:30.529+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:30.530484+0000) 2020-02-06T00:51:30.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:30.788+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:30.789360+0000) 2020-02-06T00:51:30.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:30.810+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:30.810666+0000) 2020-02-06T00:51:30.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:30.810+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:30.810671+0000) 2020-02-06T00:51:31.892 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:31.890+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.891152+0000) 2020-02-06T00:51:31.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:31.891+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.892788+0000) 2020-02-06T00:51:31.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:31.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.893434+0000) 2020-02-06T00:51:31.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:31.892+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.893603+0000) 2020-02-06T00:51:31.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:31.892+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.893817+0000) 2020-02-06T00:51:31.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:31.893+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.894524+0000) 2020-02-06T00:51:31.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:31.893+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.894768+0000) 2020-02-06T00:51:31.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:31.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.895436+0000) 2020-02-06T00:51:31.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:31.894+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.895466+0000) 2020-02-06T00:51:31.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:31.894+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.895873+0000) 2020-02-06T00:51:31.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:31.894+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.896306+0000) 2020-02-06T00:51:31.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:31.895+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.896954+0000) 2020-02-06T00:51:31.898 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:31.895+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.897302+0000) 2020-02-06T00:51:31.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:31.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:31.900148+0000) 2020-02-06T00:51:31.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:31.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186263 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:32.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:32.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:32.913220+0000) 2020-02-06T00:51:33.145 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:33.143+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:33.144788+0000) 2020-02-06T00:51:34.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:34.302+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:34.303681+0000) 2020-02-06T00:51:36.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:36.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.050227+0000) 2020-02-06T00:51:36.921 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:36.918+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.919870+0000) 2020-02-06T00:51:36.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:36.919+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.920118+0000) 2020-02-06T00:51:36.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:36.919+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.920899+0000) 2020-02-06T00:51:36.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:36.920+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.921199+0000) 2020-02-06T00:51:36.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:36.920+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.921868+0000) 2020-02-06T00:51:36.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:36.919+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.920572+0000) 2020-02-06T00:51:36.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:36.920+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.921712+0000) 2020-02-06T00:51:36.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:36.921+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.922394+0000) 2020-02-06T00:51:36.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:36.922+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.923192+0000) 2020-02-06T00:51:36.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:36.923+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.924166+0000) 2020-02-06T00:51:36.926 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:36.921+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.922740+0000) 2020-02-06T00:51:36.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:36.922+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.924274+0000) 2020-02-06T00:51:36.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:36.923+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.924738+0000) 2020-02-06T00:51:36.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:36.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:36.929624+0000) 2020-02-06T00:51:36.938 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:36.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:37.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:37.035+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:37.036204+0000) 2020-02-06T00:51:37.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:37.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:37.303019+0000) 2020-02-06T00:51:37.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:37.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:37.303186+0000) 2020-02-06T00:51:37.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:37.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:37.303273+0000) 2020-02-06T00:51:39.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:39.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:39.680141+0000) 2020-02-06T00:51:39.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:39.927+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:39.928194+0000) 2020-02-06T00:51:40.531 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:40.529+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:40.530672+0000) 2020-02-06T00:51:40.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:40.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:40.789603+0000) 2020-02-06T00:51:40.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:40.809+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:40.810932+0000) 2020-02-06T00:51:40.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:40.809+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:40.810954+0000) 2020-02-06T00:51:41.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:41.942+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.943164+0000) 2020-02-06T00:51:41.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:41.943+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.944233+0000) 2020-02-06T00:51:41.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:41.943+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.944444+0000) 2020-02-06T00:51:41.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:41.943+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.945160+0000) 2020-02-06T00:51:41.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:41.943+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.944806+0000) 2020-02-06T00:51:41.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:41.944+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.945217+0000) 2020-02-06T00:51:41.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:41.944+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.945591+0000) 2020-02-06T00:51:41.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:41.944+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.945962+0000) 2020-02-06T00:51:41.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:41.944+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.946196+0000) 2020-02-06T00:51:41.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:41.945+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.946720+0000) 2020-02-06T00:51:41.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:41.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.947187+0000) 2020-02-06T00:51:41.948 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:41.945+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.946932+0000) 2020-02-06T00:51:41.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:41.945+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.946737+0000) 2020-02-06T00:51:41.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:41.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:41.951827+0000) 2020-02-06T00:51:41.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:41.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:42.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:42.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:42.913541+0000) 2020-02-06T00:51:43.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:43.144+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:43.145023+0000) 2020-02-06T00:51:44.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:44.311+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:44.311850+0000) 2020-02-06T00:51:46.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:46.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.050499+0000) 2020-02-06T00:51:46.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:46.968+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.969198+0000) 2020-02-06T00:51:46.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:46.969+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.970371+0000) 2020-02-06T00:51:46.971 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:46.969+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.970837+0000) 2020-02-06T00:51:46.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:46.970+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.971146+0000) 2020-02-06T00:51:46.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:46.970+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.971528+0000) 2020-02-06T00:51:46.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:46.970+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.972028+0000) 2020-02-06T00:51:46.973 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:46.970+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.972303+0000) 2020-02-06T00:51:46.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:46.971+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.972871+0000) 2020-02-06T00:51:46.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:46.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.972816+0000) 2020-02-06T00:51:46.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:46.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.973858+0000) 2020-02-06T00:51:46.975 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:46.972+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.973590+0000) 2020-02-06T00:51:46.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:46.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.974152+0000) 2020-02-06T00:51:46.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:46.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.974792+0000) 2020-02-06T00:51:46.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:46.979+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:46.980318+0000) 2020-02-06T00:51:46.989 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:46.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186386 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:47.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:47.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:47.036464+0000) 2020-02-06T00:51:47.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:47.301+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:47.303200+0000) 2020-02-06T00:51:47.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:47.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:47.303497+0000) 2020-02-06T00:51:47.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:47.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:47.303522+0000) 2020-02-06T00:51:48.267 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:51:48.273 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:51:49.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:49.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:49.680348+0000) 2020-02-06T00:51:49.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:49.928+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:49.928449+0000) 2020-02-06T00:51:50.531 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:50.530+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:50.530903+0000) 2020-02-06T00:51:50.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:50.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:50.789835+0000) 2020-02-06T00:51:50.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:50.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:50.811200+0000) 2020-02-06T00:51:50.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:50.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:50.811251+0000) 2020-02-06T00:51:52.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:51.993+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.994913+0000) 2020-02-06T00:51:52.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:51.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.996252+0000) 2020-02-06T00:51:52.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:51.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.997315+0000) 2020-02-06T00:51:52.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:51.995+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.997202+0000) 2020-02-06T00:51:52.001 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:51.994+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.994630+0000) 2020-02-06T00:51:52.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:51.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.995590+0000) 2020-02-06T00:51:52.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:51.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.996461+0000) 2020-02-06T00:51:52.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:51.994+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.995923+0000) 2020-02-06T00:51:52.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:51.996+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.997997+0000) 2020-02-06T00:51:52.002 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:51.997+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.998545+0000) 2020-02-06T00:51:52.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:51.997+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.998996+0000) 2020-02-06T00:51:52.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:51.998+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:51.999459+0000) 2020-02-06T00:51:52.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:51.998+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:52.000127+0000) 2020-02-06T00:51:52.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:51.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:52.000669+0000) 2020-02-06T00:51:52.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:52.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186426 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:52.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:52.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:52.913782+0000) 2020-02-06T00:51:53.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:53.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:53.145303+0000) 2020-02-06T00:51:54.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:54.319+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:54.319775+0000) 2020-02-06T00:51:56.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:56.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:56.050744+0000) 2020-02-06T00:51:57.021 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:51:57.020+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.020290+0000) 2020-02-06T00:51:57.021 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:51:57.019+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.020789+0000) 2020-02-06T00:51:57.022 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:57.019+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.021379+0000) 2020-02-06T00:51:57.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:57.022+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.022235+0000) 2020-02-06T00:51:57.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:57.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.022634+0000) 2020-02-06T00:51:57.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:51:57.022+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.022917+0000) 2020-02-06T00:51:57.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:51:57.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.023366+0000) 2020-02-06T00:51:57.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:57.022+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.023673+0000) 2020-02-06T00:51:57.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:57.024+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.024333+0000) 2020-02-06T00:51:57.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:51:57.023+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.025302+0000) 2020-02-06T00:51:57.026 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:51:57.023+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.025124+0000) 2020-02-06T00:51:57.027 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:51:57.025+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.025774+0000) 2020-02-06T00:51:57.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:57.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.026072+0000) 2020-02-06T00:51:57.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:57.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.033414+0000) 2020-02-06T00:51:57.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:51:57.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.036657+0000) 2020-02-06T00:51:57.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:51:57.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:51:57.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:51:57.302+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.303465+0000) 2020-02-06T00:51:57.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:51:57.302+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.303733+0000) 2020-02-06T00:51:57.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:51:57.302+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:57.303741+0000) 2020-02-06T00:51:59.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:59.679+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:59.680607+0000) 2020-02-06T00:51:59.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:51:59.927+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:51:59.928673+0000) 2020-02-06T00:52:00.532 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:00.530+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:00.531144+0000) 2020-02-06T00:52:00.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:00.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:00.790051+0000) 2020-02-06T00:52:00.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:00.810+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:00.811479+0000) 2020-02-06T00:52:00.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:00.810+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:00.811492+0000) 2020-02-06T00:52:02.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:02.044+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.046013+0000) 2020-02-06T00:52:02.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:02.045+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.046333+0000) 2020-02-06T00:52:02.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:02.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.046740+0000) 2020-02-06T00:52:02.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:02.045+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.046936+0000) 2020-02-06T00:52:02.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:02.046+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.047354+0000) 2020-02-06T00:52:02.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:02.047+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.048447+0000) 2020-02-06T00:52:02.049 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:02.046+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.048176+0000) 2020-02-06T00:52:02.051 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:02.048+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.049992+0000) 2020-02-06T00:52:02.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:02.049+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.050152+0000) 2020-02-06T00:52:02.051 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:02.049+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.050674+0000) 2020-02-06T00:52:02.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:02.049+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.050799+0000) 2020-02-06T00:52:02.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:02.049+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.050990+0000) 2020-02-06T00:52:02.052 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:02.049+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.051028+0000) 2020-02-06T00:52:02.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:02.053+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.054969+0000) 2020-02-06T00:52:02.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:02.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186506 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:02.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:02.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:02.914013+0000) 2020-02-06T00:52:03.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:03.144+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:03.145538+0000) 2020-02-06T00:52:04.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:04.327+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:04.328573+0000) 2020-02-06T00:52:06.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:06.050+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:06.051006+0000) 2020-02-06T00:52:07.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:07.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.036898+0000) 2020-02-06T00:52:07.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:07.071+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.072491+0000) 2020-02-06T00:52:07.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:07.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.073525+0000) 2020-02-06T00:52:07.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:07.073+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.074196+0000) 2020-02-06T00:52:07.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:07.073+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.075302+0000) 2020-02-06T00:52:07.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:07.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.075596+0000) 2020-02-06T00:52:07.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:07.074+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.076285+0000) 2020-02-06T00:52:07.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:07.076+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.077143+0000) 2020-02-06T00:52:07.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:07.076+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.077431+0000) 2020-02-06T00:52:07.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:07.075+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.077149+0000) 2020-02-06T00:52:07.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:07.076+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.077923+0000) 2020-02-06T00:52:07.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:07.077+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.078990+0000) 2020-02-06T00:52:07.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:07.077+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.079208+0000) 2020-02-06T00:52:07.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:07.077+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.079241+0000) 2020-02-06T00:52:07.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:07.083+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.084639+0000) 2020-02-06T00:52:07.095 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:07.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:07.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:07.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.303677+0000) 2020-02-06T00:52:07.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:07.303+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.303940+0000) 2020-02-06T00:52:07.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:07.303+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:07.303953+0000) 2020-02-06T00:52:09.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:09.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:09.680844+0000) 2020-02-06T00:52:09.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:09.928+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:09.928904+0000) 2020-02-06T00:52:10.532 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:10.530+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:10.531390+0000) 2020-02-06T00:52:10.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:10.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:10.790312+0000) 2020-02-06T00:52:10.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:10.810+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:10.811734+0000) 2020-02-06T00:52:10.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:10.810+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:10.811737+0000) 2020-02-06T00:52:12.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:12.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.099385+0000) 2020-02-06T00:52:12.101 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:12.099+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.100138+0000) 2020-02-06T00:52:12.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:12.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.101982+0000) 2020-02-06T00:52:12.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:12.101+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.102528+0000) 2020-02-06T00:52:12.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:12.101+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.103266+0000) 2020-02-06T00:52:12.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:12.102+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.103730+0000) 2020-02-06T00:52:12.105 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:12.103+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.104252+0000) 2020-02-06T00:52:12.105 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:12.103+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.104582+0000) 2020-02-06T00:52:12.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:12.104+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.105760+0000) 2020-02-06T00:52:12.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:12.104+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.106167+0000) 2020-02-06T00:52:12.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:12.105+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.106225+0000) 2020-02-06T00:52:12.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:12.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.106426+0000) 2020-02-06T00:52:12.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:12.105+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.107278+0000) 2020-02-06T00:52:12.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:12.111+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.112314+0000) 2020-02-06T00:52:12.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:12.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:12.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:12.912+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:12.914183+0000) 2020-02-06T00:52:13.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:13.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:13.145786+0000) 2020-02-06T00:52:14.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:14.335+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:14.336454+0000) 2020-02-06T00:52:16.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:16.049+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:16.051254+0000) 2020-02-06T00:52:17.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:17.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.037135+0000) 2020-02-06T00:52:17.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:17.125+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.126846+0000) 2020-02-06T00:52:17.128 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:17.125+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.127224+0000) 2020-02-06T00:52:17.129 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:17.127+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.128959+0000) 2020-02-06T00:52:17.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:17.128+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.129341+0000) 2020-02-06T00:52:17.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:17.128+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.129591+0000) 2020-02-06T00:52:17.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:17.128+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.129932+0000) 2020-02-06T00:52:17.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:17.129+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.130667+0000) 2020-02-06T00:52:17.132 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:17.130+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.131194+0000) 2020-02-06T00:52:17.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:17.130+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.131599+0000) 2020-02-06T00:52:17.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:17.130+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.131976+0000) 2020-02-06T00:52:17.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:17.130+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.132161+0000) 2020-02-06T00:52:17.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:17.131+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.132848+0000) 2020-02-06T00:52:17.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:17.131+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.132568+0000) 2020-02-06T00:52:17.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:17.136+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.137623+0000) 2020-02-06T00:52:17.147 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:17.145+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186628 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:17.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:17.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.303880+0000) 2020-02-06T00:52:17.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:17.303+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.304164+0000) 2020-02-06T00:52:17.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:17.303+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:17.304167+0000) 2020-02-06T00:52:18.452 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:52:18.459 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:52:19.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:19.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:19.681071+0000) 2020-02-06T00:52:19.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:19.928+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:19.929138+0000) 2020-02-06T00:52:20.532 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:20.531+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:20.531679+0000) 2020-02-06T00:52:20.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:20.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:20.790545+0000) 2020-02-06T00:52:20.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:20.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:20.811962+0000) 2020-02-06T00:52:20.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:20.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:20.812007+0000) 2020-02-06T00:52:22.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:22.151+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.152646+0000) 2020-02-06T00:52:22.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:22.152+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.153319+0000) 2020-02-06T00:52:22.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:22.153+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.154059+0000) 2020-02-06T00:52:22.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:22.154+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.155882+0000) 2020-02-06T00:52:22.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:22.155+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.155549+0000) 2020-02-06T00:52:22.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:22.155+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.156437+0000) 2020-02-06T00:52:22.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:22.156+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.157219+0000) 2020-02-06T00:52:22.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:22.156+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.157269+0000) 2020-02-06T00:52:22.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:22.157+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.158050+0000) 2020-02-06T00:52:22.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:22.156+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.157557+0000) 2020-02-06T00:52:22.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:22.156+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.157898+0000) 2020-02-06T00:52:22.160 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:22.158+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.158844+0000) 2020-02-06T00:52:22.160 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:22.158+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.159043+0000) 2020-02-06T00:52:22.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:22.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.163538+0000) 2020-02-06T00:52:22.174 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:22.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:22.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:22.913+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:22.914408+0000) 2020-02-06T00:52:23.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:23.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:23.146002+0000) 2020-02-06T00:52:24.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:24.343+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:24.344451+0000) 2020-02-06T00:52:26.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:26.051+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:26.051492+0000) 2020-02-06T00:52:27.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:27.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.037358+0000) 2020-02-06T00:52:27.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:27.179+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.180214+0000) 2020-02-06T00:52:27.181 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:27.179+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.180563+0000) 2020-02-06T00:52:27.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:27.180+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.181119+0000) 2020-02-06T00:52:27.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:27.180+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.181445+0000) 2020-02-06T00:52:27.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:27.181+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.182594+0000) 2020-02-06T00:52:27.184 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:27.182+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.182993+0000) 2020-02-06T00:52:27.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:27.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.183205+0000) 2020-02-06T00:52:27.184 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:27.183+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.183963+0000) 2020-02-06T00:52:27.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:27.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.184447+0000) 2020-02-06T00:52:27.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:27.184+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.184458+0000) 2020-02-06T00:52:27.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:27.184+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.184778+0000) 2020-02-06T00:52:27.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:27.185+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.185567+0000) 2020-02-06T00:52:27.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:27.185+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.185510+0000) 2020-02-06T00:52:27.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:27.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.190369+0000) 2020-02-06T00:52:27.201 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:27.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:27.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:27.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.304106+0000) 2020-02-06T00:52:27.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:27.303+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.304344+0000) 2020-02-06T00:52:27.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:27.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:27.304407+0000) 2020-02-06T00:52:29.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:29.680+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:29.681371+0000) 2020-02-06T00:52:29.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:29.928+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:29.929369+0000) 2020-02-06T00:52:30.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:30.531+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:30.531932+0000) 2020-02-06T00:52:30.791 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:30.790+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:30.790772+0000) 2020-02-06T00:52:30.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:30.811+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:30.812207+0000) 2020-02-06T00:52:30.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:30.811+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:30.812325+0000) 2020-02-06T00:52:32.210 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:32.204+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.205246+0000) 2020-02-06T00:52:32.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:32.204+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.206037+0000) 2020-02-06T00:52:32.213 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:32.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.207250+0000) 2020-02-06T00:52:32.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:32.206+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.207554+0000) 2020-02-06T00:52:32.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:32.208+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.208799+0000) 2020-02-06T00:52:32.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:32.208+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.209800+0000) 2020-02-06T00:52:32.217 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:32.208+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.210032+0000) 2020-02-06T00:52:32.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:32.209+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.210549+0000) 2020-02-06T00:52:32.218 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:32.210+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.211154+0000) 2020-02-06T00:52:32.219 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:32.210+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.211773+0000) 2020-02-06T00:52:32.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:32.211+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.211732+0000) 2020-02-06T00:52:32.220 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:32.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.212477+0000) 2020-02-06T00:52:32.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:32.212+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.212901+0000) 2020-02-06T00:52:32.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:32.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.217382+0000) 2020-02-06T00:52:32.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:32.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:32.915 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:32.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:32.914627+0000) 2020-02-06T00:52:33.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:33.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:33.146206+0000) 2020-02-06T00:52:34.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:34.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:34.352678+0000) 2020-02-06T00:52:36.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:36.051+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:36.051695+0000) 2020-02-06T00:52:37.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:37.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.037584+0000) 2020-02-06T00:52:37.233 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:37.231+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.232860+0000) 2020-02-06T00:52:37.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:37.233+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.233944+0000) 2020-02-06T00:52:37.235 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:37.233+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.234352+0000) 2020-02-06T00:52:37.235 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:37.233+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.235035+0000) 2020-02-06T00:52:37.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:37.235+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.235780+0000) 2020-02-06T00:52:37.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:37.234+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.235778+0000) 2020-02-06T00:52:37.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:37.234+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.236021+0000) 2020-02-06T00:52:37.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:37.236+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.237121+0000) 2020-02-06T00:52:37.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:37.236+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.237021+0000) 2020-02-06T00:52:37.238 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:37.236+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.237354+0000) 2020-02-06T00:52:37.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:37.237+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.237963+0000) 2020-02-06T00:52:37.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:37.238+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.238508+0000) 2020-02-06T00:52:37.239 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:37.237+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.238925+0000) 2020-02-06T00:52:37.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:37.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.244748+0000) 2020-02-06T00:52:37.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:37.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:37.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:37.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.304322+0000) 2020-02-06T00:52:37.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:37.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.304478+0000) 2020-02-06T00:52:37.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:37.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:37.304650+0000) 2020-02-06T00:52:39.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:39.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:39.681642+0000) 2020-02-06T00:52:39.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:39.928+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:39.929596+0000) 2020-02-06T00:52:40.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:40.531+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:40.532169+0000) 2020-02-06T00:52:40.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:40.789+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:40.790992+0000) 2020-02-06T00:52:40.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:40.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:40.812420+0000) 2020-02-06T00:52:40.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:40.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:40.812551+0000) 2020-02-06T00:52:42.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:42.256+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.257993+0000) 2020-02-06T00:52:42.259 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:42.258+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.258862+0000) 2020-02-06T00:52:42.260 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:42.258+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.259650+0000) 2020-02-06T00:52:42.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:42.260+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.260781+0000) 2020-02-06T00:52:42.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:42.260+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.261118+0000) 2020-02-06T00:52:42.262 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:42.261+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.261465+0000) 2020-02-06T00:52:42.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:42.260+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.261739+0000) 2020-02-06T00:52:42.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:42.261+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.262151+0000) 2020-02-06T00:52:42.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:42.262+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.262787+0000) 2020-02-06T00:52:42.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:42.262+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.263207+0000) 2020-02-06T00:52:42.264 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:42.262+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.263872+0000) 2020-02-06T00:52:42.265 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:42.263+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.264199+0000) 2020-02-06T00:52:42.265 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:42.263+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.264419+0000) 2020-02-06T00:52:42.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:42.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.268681+0000) 2020-02-06T00:52:42.278 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:42.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:42.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:42.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:42.914900+0000) 2020-02-06T00:52:43.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:43.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:43.146378+0000) 2020-02-06T00:52:44.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:44.360+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:44.361032+0000) 2020-02-06T00:52:46.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:46.051+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:46.051916+0000) 2020-02-06T00:52:47.038 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:47.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.037821+0000) 2020-02-06T00:52:47.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:47.282+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.283310+0000) 2020-02-06T00:52:47.285 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:47.282+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.283852+0000) 2020-02-06T00:52:47.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:47.285+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.285104+0000) 2020-02-06T00:52:47.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:47.284+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.285385+0000) 2020-02-06T00:52:47.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:47.286+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.286505+0000) 2020-02-06T00:52:47.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:47.285+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.286040+0000) 2020-02-06T00:52:47.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:47.286+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.286859+0000) 2020-02-06T00:52:47.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:47.286+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.287214+0000) 2020-02-06T00:52:47.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:47.286+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.287476+0000) 2020-02-06T00:52:47.289 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:47.286+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.287985+0000) 2020-02-06T00:52:47.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:47.288+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.288599+0000) 2020-02-06T00:52:47.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:47.288+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.288665+0000) 2020-02-06T00:52:47.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:47.288+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.288785+0000) 2020-02-06T00:52:47.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:47.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.291123+0000) 2020-02-06T00:52:47.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:47.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.304514+0000) 2020-02-06T00:52:47.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:47.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.304623+0000) 2020-02-06T00:52:47.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:47.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:47.304793+0000) 2020-02-06T00:52:47.306 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:47.304+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186871 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:48.624 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:52:48.631 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:52:49.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:49.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:49.681864+0000) 2020-02-06T00:52:49.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:49.929+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:49.929825+0000) 2020-02-06T00:52:50.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:50.532+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:50.532408+0000) 2020-02-06T00:52:50.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:50.791+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:50.791280+0000) 2020-02-06T00:52:50.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:50.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:50.812689+0000) 2020-02-06T00:52:50.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:50.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:50.812774+0000) 2020-02-06T00:52:52.310 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:52.308+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.309905+0000) 2020-02-06T00:52:52.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:52.310+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.311205+0000) 2020-02-06T00:52:52.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:52.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.312557+0000) 2020-02-06T00:52:52.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:52.312+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.313281+0000) 2020-02-06T00:52:52.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:52.312+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.313916+0000) 2020-02-06T00:52:52.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:52.313+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.314062+0000) 2020-02-06T00:52:52.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:52.313+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.314289+0000) 2020-02-06T00:52:52.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:52.313+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.314645+0000) 2020-02-06T00:52:52.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:52.314+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.315165+0000) 2020-02-06T00:52:52.316 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:52.314+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.315378+0000) 2020-02-06T00:52:52.317 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:52.315+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.315821+0000) 2020-02-06T00:52:52.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:52.315+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.316352+0000) 2020-02-06T00:52:52.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:52.316+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.316964+0000) 2020-02-06T00:52:52.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:52.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.321473+0000) 2020-02-06T00:52:52.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:52.331+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:52.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:52.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:52.915129+0000) 2020-02-06T00:52:53.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:53.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:53.146579+0000) 2020-02-06T00:52:54.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:54.367+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:54.368940+0000) 2020-02-06T00:52:56.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:56.052+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:56.052170+0000) 2020-02-06T00:52:57.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:57.036+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.038044+0000) 2020-02-06T00:52:57.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:57.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.304744+0000) 2020-02-06T00:52:57.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:57.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.304794+0000) 2020-02-06T00:52:57.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:57.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.304954+0000) 2020-02-06T00:52:57.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:52:57.336+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.336843+0000) 2020-02-06T00:52:57.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:52:57.337+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.338238+0000) 2020-02-06T00:52:57.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:57.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.339156+0000) 2020-02-06T00:52:57.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:52:57.339+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.339571+0000) 2020-02-06T00:52:57.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:52:57.339+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.339960+0000) 2020-02-06T00:52:57.342 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:52:57.339+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.340904+0000) 2020-02-06T00:52:57.342 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:52:57.340+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.341401+0000) 2020-02-06T00:52:57.342 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:52:57.340+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.341903+0000) 2020-02-06T00:52:57.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:57.341+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.342316+0000) 2020-02-06T00:52:57.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:52:57.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.342618+0000) 2020-02-06T00:52:57.344 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:52:57.342+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.343245+0000) 2020-02-06T00:52:57.344 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:52:57.343+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.343637+0000) 2020-02-06T00:52:57.344 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:52:57.343+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.344107+0000) 2020-02-06T00:52:57.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:57.348+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:57.349277+0000) 2020-02-06T00:52:57.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:52:57.358+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186952 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:52:59.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:59.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:59.682133+0000) 2020-02-06T00:52:59.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:52:59.929+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:52:59.930069+0000) 2020-02-06T00:53:00.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:00.532+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:00.532694+0000) 2020-02-06T00:53:00.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:00.790+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:00.791563+0000) 2020-02-06T00:53:00.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:00.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:00.812960+0000) 2020-02-06T00:53:00.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:00.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:00.813051+0000) 2020-02-06T00:53:02.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:02.362+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.364330+0000) 2020-02-06T00:53:02.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:02.364+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.365099+0000) 2020-02-06T00:53:02.366 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:02.364+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.365916+0000) 2020-02-06T00:53:02.367 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:02.365+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.366722+0000) 2020-02-06T00:53:02.368 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:02.366+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.367345+0000) 2020-02-06T00:53:02.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:02.365+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.367101+0000) 2020-02-06T00:53:02.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:02.366+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.367596+0000) 2020-02-06T00:53:02.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:02.367+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.368420+0000) 2020-02-06T00:53:02.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:02.367+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.369178+0000) 2020-02-06T00:53:02.371 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:02.367+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.368985+0000) 2020-02-06T00:53:02.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:02.368+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.369726+0000) 2020-02-06T00:53:02.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:02.369+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.370475+0000) 2020-02-06T00:53:02.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:02.373+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.374616+0000) 2020-02-06T00:53:02.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:02.373+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.374824+0000) 2020-02-06T00:53:02.384 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:02.382+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 186994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:02.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:02.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:02.915365+0000) 2020-02-06T00:53:03.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:03.145+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:03.146796+0000) 2020-02-06T00:53:04.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:04.375+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:04.376318+0000) 2020-02-06T00:53:06.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:06.051+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:06.052457+0000) 2020-02-06T00:53:07.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:07.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.038323+0000) 2020-02-06T00:53:07.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:07.303+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.305030+0000) 2020-02-06T00:53:07.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:07.303+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.305030+0000) 2020-02-06T00:53:07.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:07.303+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.305215+0000) 2020-02-06T00:53:07.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:07.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.390163+0000) 2020-02-06T00:53:07.392 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:07.390+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.391316+0000) 2020-02-06T00:53:07.392 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:07.390+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.391720+0000) 2020-02-06T00:53:07.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:07.390+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.392301+0000) 2020-02-06T00:53:07.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:07.392+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.393758+0000) 2020-02-06T00:53:07.394 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:07.392+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.393967+0000) 2020-02-06T00:53:07.395 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:07.393+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.394752+0000) 2020-02-06T00:53:07.395 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:07.392+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.394405+0000) 2020-02-06T00:53:07.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:07.394+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.395456+0000) 2020-02-06T00:53:07.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:07.394+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.395565+0000) 2020-02-06T00:53:07.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:07.393+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.395303+0000) 2020-02-06T00:53:07.397 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:07.394+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.395653+0000) 2020-02-06T00:53:07.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:07.394+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.395843+0000) 2020-02-06T00:53:07.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:07.397+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:07.398163+0000) 2020-02-06T00:53:07.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:07.409+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187033 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:09.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:09.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:09.682377+0000) 2020-02-06T00:53:09.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:09.929+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:09.930335+0000) 2020-02-06T00:53:10.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:10.531+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:10.532915+0000) 2020-02-06T00:53:10.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:10.790+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:10.791741+0000) 2020-02-06T00:53:10.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:10.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:10.813187+0000) 2020-02-06T00:53:10.814 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:10.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:10.813318+0000) 2020-02-06T00:53:12.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:12.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.415922+0000) 2020-02-06T00:53:12.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:12.415+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.416613+0000) 2020-02-06T00:53:12.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:12.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.417346+0000) 2020-02-06T00:53:12.418 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:12.416+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.418005+0000) 2020-02-06T00:53:12.419 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:12.417+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.418204+0000) 2020-02-06T00:53:12.419 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:12.417+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.418741+0000) 2020-02-06T00:53:12.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:12.418+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.420118+0000) 2020-02-06T00:53:12.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:12.419+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.421060+0000) 2020-02-06T00:53:12.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:12.419+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.421408+0000) 2020-02-06T00:53:12.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:12.420+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.421836+0000) 2020-02-06T00:53:12.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:12.421+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.422758+0000) 2020-02-06T00:53:12.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:12.418+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.419376+0000) 2020-02-06T00:53:12.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:12.419+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.420424+0000) 2020-02-06T00:53:12.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:12.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.423867+0000) 2020-02-06T00:53:12.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:12.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:12.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:12.914+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:12.915645+0000) 2020-02-06T00:53:13.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:13.146+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:13.147005+0000) 2020-02-06T00:53:14.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:14.383+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:14.384503+0000) 2020-02-06T00:53:16.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:16.051+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:16.052697+0000) 2020-02-06T00:53:17.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:17.037+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.038553+0000) 2020-02-06T00:53:17.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:17.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.305337+0000) 2020-02-06T00:53:17.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:17.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.305335+0000) 2020-02-06T00:53:17.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:17.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.305695+0000) 2020-02-06T00:53:17.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:17.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.443240+0000) 2020-02-06T00:53:17.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:17.442+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.444112+0000) 2020-02-06T00:53:17.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:17.444+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.445071+0000) 2020-02-06T00:53:17.446 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:17.444+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.445686+0000) 2020-02-06T00:53:17.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:17.445+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.446158+0000) 2020-02-06T00:53:17.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:17.445+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.446801+0000) 2020-02-06T00:53:17.448 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:17.445+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.447025+0000) 2020-02-06T00:53:17.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:17.446+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.447529+0000) 2020-02-06T00:53:17.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:17.447+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.448586+0000) 2020-02-06T00:53:17.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:17.447+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.448769+0000) 2020-02-06T00:53:17.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:17.448+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.449792+0000) 2020-02-06T00:53:17.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:17.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.450132+0000) 2020-02-06T00:53:17.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:17.449+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.450147+0000) 2020-02-06T00:53:17.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:17.454+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:17.456046+0000) 2020-02-06T00:53:17.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:17.461+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:18.762 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:53:18.769 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:53:19.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:19.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:19.682636+0000) 2020-02-06T00:53:19.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:19.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:19.930592+0000) 2020-02-06T00:53:20.534 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:20.532+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:20.533191+0000) 2020-02-06T00:53:20.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:20.791+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:20.791970+0000) 2020-02-06T00:53:20.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:20.812+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:20.813440+0000) 2020-02-06T00:53:20.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:20.812+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:20.813581+0000) 2020-02-06T00:53:22.471 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:22.468+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.470068+0000) 2020-02-06T00:53:22.471 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:22.469+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.470392+0000) 2020-02-06T00:53:22.472 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:22.470+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.471543+0000) 2020-02-06T00:53:22.473 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:22.470+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.471554+0000) 2020-02-06T00:53:22.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:22.470+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.471932+0000) 2020-02-06T00:53:22.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:22.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.472218+0000) 2020-02-06T00:53:22.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:22.471+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.472683+0000) 2020-02-06T00:53:22.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:22.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.472819+0000) 2020-02-06T00:53:22.474 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:22.473+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.474070+0000) 2020-02-06T00:53:22.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:22.472+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.474339+0000) 2020-02-06T00:53:22.476 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:22.473+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.474991+0000) 2020-02-06T00:53:22.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:22.473+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.474743+0000) 2020-02-06T00:53:22.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:22.473+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.475220+0000) 2020-02-06T00:53:22.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:22.477+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.478287+0000) 2020-02-06T00:53:22.490 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:22.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187156 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:22.916 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:22.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:22.915925+0000) 2020-02-06T00:53:23.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:23.146+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:23.147201+0000) 2020-02-06T00:53:24.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:24.393+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:24.394199+0000) 2020-02-06T00:53:26.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:26.052+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:26.052953+0000) 2020-02-06T00:53:27.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:27.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.038783+0000) 2020-02-06T00:53:27.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:27.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.305690+0000) 2020-02-06T00:53:27.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:27.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.305692+0000) 2020-02-06T00:53:27.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:27.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.305975+0000) 2020-02-06T00:53:27.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:27.494+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.495687+0000) 2020-02-06T00:53:27.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:27.495+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.496192+0000) 2020-02-06T00:53:27.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:27.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.497010+0000) 2020-02-06T00:53:27.498 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:27.496+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.497159+0000) 2020-02-06T00:53:27.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:27.496+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.497869+0000) 2020-02-06T00:53:27.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:27.497+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.498526+0000) 2020-02-06T00:53:27.500 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:27.497+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.498945+0000) 2020-02-06T00:53:27.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:27.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.499166+0000) 2020-02-06T00:53:27.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:27.497+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.499121+0000) 2020-02-06T00:53:27.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:27.498+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.499877+0000) 2020-02-06T00:53:27.501 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:27.498+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.500287+0000) 2020-02-06T00:53:27.501 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:27.499+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.500615+0000) 2020-02-06T00:53:27.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:27.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.501484+0000) 2020-02-06T00:53:27.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:27.505+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:27.507019+0000) 2020-02-06T00:53:27.516 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:27.514+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187195 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:29.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:29.681+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:29.682901+0000) 2020-02-06T00:53:29.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:29.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:29.930820+0000) 2020-02-06T00:53:30.534 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:30.532+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:30.533489+0000) 2020-02-06T00:53:30.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:30.791+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:30.792287+0000) 2020-02-06T00:53:30.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:30.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:30.813678+0000) 2020-02-06T00:53:30.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:30.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:30.813852+0000) 2020-02-06T00:53:32.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:32.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.521274+0000) 2020-02-06T00:53:32.523 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:32.520+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.521727+0000) 2020-02-06T00:53:32.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:32.521+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.522211+0000) 2020-02-06T00:53:32.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:32.521+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.522805+0000) 2020-02-06T00:53:32.524 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:32.522+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.523067+0000) 2020-02-06T00:53:32.524 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:32.522+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.523413+0000) 2020-02-06T00:53:32.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:32.523+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.524657+0000) 2020-02-06T00:53:32.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:32.523+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.524793+0000) 2020-02-06T00:53:32.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:32.524+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.525579+0000) 2020-02-06T00:53:32.526 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:32.524+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.526007+0000) 2020-02-06T00:53:32.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:32.524+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.526333+0000) 2020-02-06T00:53:32.528 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:32.526+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.527619+0000) 2020-02-06T00:53:32.528 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:32.526+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.527967+0000) 2020-02-06T00:53:32.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:32.529+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.530647+0000) 2020-02-06T00:53:32.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:32.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:32.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:32.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:32.916166+0000) 2020-02-06T00:53:33.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:33.146+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:33.147376+0000) 2020-02-06T00:53:34.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:34.402+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:34.402737+0000) 2020-02-06T00:53:36.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:36.052+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:36.053210+0000) 2020-02-06T00:53:37.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:37.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.038976+0000) 2020-02-06T00:53:37.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:37.304+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.305961+0000) 2020-02-06T00:53:37.307 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:37.304+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.305965+0000) 2020-02-06T00:53:37.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:37.304+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.306224+0000) 2020-02-06T00:53:37.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:37.544+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.545707+0000) 2020-02-06T00:53:37.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:37.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.546220+0000) 2020-02-06T00:53:37.548 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:37.546+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.547589+0000) 2020-02-06T00:53:37.549 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:37.547+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.548528+0000) 2020-02-06T00:53:37.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:37.548+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.549256+0000) 2020-02-06T00:53:37.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:37.547+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.549405+0000) 2020-02-06T00:53:37.552 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:37.548+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.549642+0000) 2020-02-06T00:53:37.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:37.550+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.551338+0000) 2020-02-06T00:53:37.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:37.549+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.550545+0000) 2020-02-06T00:53:37.554 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:37.549+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.550817+0000) 2020-02-06T00:53:37.554 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:37.550+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.551788+0000) 2020-02-06T00:53:37.554 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:37.550+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.552305+0000) 2020-02-06T00:53:37.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:37.553+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.554965+0000) 2020-02-06T00:53:37.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:37.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:37.555174+0000) 2020-02-06T00:53:37.569 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:37.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187276 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:39.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:39.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:39.683137+0000) 2020-02-06T00:53:39.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:39.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:39.931116+0000) 2020-02-06T00:53:40.534 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:40.533+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:40.533742+0000) 2020-02-06T00:53:40.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:40.792+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:40.792543+0000) 2020-02-06T00:53:40.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:40.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:40.813909+0000) 2020-02-06T00:53:40.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:40.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:40.814100+0000) 2020-02-06T00:53:42.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:42.572+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.572692+0000) 2020-02-06T00:53:42.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:42.573+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.573446+0000) 2020-02-06T00:53:42.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:42.572+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.574054+0000) 2020-02-06T00:53:42.575 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:42.574+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.574511+0000) 2020-02-06T00:53:42.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:42.573+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.575114+0000) 2020-02-06T00:53:42.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:42.574+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.575733+0000) 2020-02-06T00:53:42.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:42.574+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.574994+0000) 2020-02-06T00:53:42.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:42.575+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.575873+0000) 2020-02-06T00:53:42.578 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:42.576+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.576394+0000) 2020-02-06T00:53:42.578 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:42.577+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.577264+0000) 2020-02-06T00:53:42.579 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:42.575+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.577209+0000) 2020-02-06T00:53:42.579 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:42.576+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.577680+0000) 2020-02-06T00:53:42.580 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:42.577+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.578558+0000) 2020-02-06T00:53:42.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:42.581+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.581234+0000) 2020-02-06T00:53:42.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:42.592+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:42.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:42.915+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:42.916455+0000) 2020-02-06T00:53:43.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:43.147+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:43.147607+0000) 2020-02-06T00:53:44.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:44.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:44.411246+0000) 2020-02-06T00:53:46.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:46.052+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:46.053404+0000) 2020-02-06T00:53:47.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:47.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.039206+0000) 2020-02-06T00:53:47.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:47.305+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.306248+0000) 2020-02-06T00:53:47.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:47.305+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.306248+0000) 2020-02-06T00:53:47.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:47.305+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.306495+0000) 2020-02-06T00:53:47.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:47.598+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.599785+0000) 2020-02-06T00:53:47.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:47.600+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.601954+0000) 2020-02-06T00:53:47.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:47.599+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.600943+0000) 2020-02-06T00:53:47.603 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:47.600+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.601365+0000) 2020-02-06T00:53:47.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:47.600+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.601761+0000) 2020-02-06T00:53:47.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:47.600+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.602019+0000) 2020-02-06T00:53:47.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:47.601+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.603389+0000) 2020-02-06T00:53:47.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:47.601+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.602759+0000) 2020-02-06T00:53:47.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:47.603+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.604120+0000) 2020-02-06T00:53:47.605 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:47.603+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.604499+0000) 2020-02-06T00:53:47.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:47.604+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.605521+0000) 2020-02-06T00:53:47.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:47.603+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.605266+0000) 2020-02-06T00:53:47.607 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:47.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.606057+0000) 2020-02-06T00:53:47.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:47.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:47.608552+0000) 2020-02-06T00:53:47.620 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:47.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187356 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:48.900 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:53:48.906 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:53:49.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:49.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:49.683361+0000) 2020-02-06T00:53:49.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:49.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:49.931378+0000) 2020-02-06T00:53:50.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:50.533+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:50.533999+0000) 2020-02-06T00:53:50.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:50.791+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:50.792771+0000) 2020-02-06T00:53:50.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:50.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:50.814179+0000) 2020-02-06T00:53:50.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:50.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:50.814340+0000) 2020-02-06T00:53:52.626 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:52.624+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.625484+0000) 2020-02-06T00:53:52.627 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:52.625+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.627202+0000) 2020-02-06T00:53:52.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:52.625+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.626975+0000) 2020-02-06T00:53:52.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:52.626+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.628023+0000) 2020-02-06T00:53:52.630 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:52.627+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.628725+0000) 2020-02-06T00:53:52.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:52.628+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.629983+0000) 2020-02-06T00:53:52.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:52.629+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.630412+0000) 2020-02-06T00:53:52.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:52.629+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.630856+0000) 2020-02-06T00:53:52.632 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:52.630+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.631296+0000) 2020-02-06T00:53:52.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:52.630+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.631498+0000) 2020-02-06T00:53:52.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:52.630+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.631838+0000) 2020-02-06T00:53:52.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:52.630+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.631908+0000) 2020-02-06T00:53:52.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:52.635+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.636405+0000) 2020-02-06T00:53:52.638 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:52.635+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.636739+0000) 2020-02-06T00:53:52.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:52.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187397 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:52.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:52.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:52.916688+0000) 2020-02-06T00:53:53.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:53.146+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:53.147829+0000) 2020-02-06T00:53:54.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:54.418+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:54.419900+0000) 2020-02-06T00:53:56.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:56.053+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:56.053684+0000) 2020-02-06T00:53:57.040 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:57.038+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.039453+0000) 2020-02-06T00:53:57.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:57.306+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.306559+0000) 2020-02-06T00:53:57.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:57.306+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.306742+0000) 2020-02-06T00:53:57.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:57.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.306684+0000) 2020-02-06T00:53:57.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:57.649+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.650225+0000) 2020-02-06T00:53:57.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:53:57.650+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.651114+0000) 2020-02-06T00:53:57.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:57.651+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.652440+0000) 2020-02-06T00:53:57.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:53:57.652+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.652689+0000) 2020-02-06T00:53:57.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:53:57.651+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.652946+0000) 2020-02-06T00:53:57.654 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:53:57.652+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.653619+0000) 2020-02-06T00:53:57.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:53:57.652+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.653747+0000) 2020-02-06T00:53:57.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:53:57.652+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.654092+0000) 2020-02-06T00:53:57.655 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:53:57.653+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.654725+0000) 2020-02-06T00:53:57.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:53:57.653+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.654055+0000) 2020-02-06T00:53:57.656 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:53:57.653+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.655276+0000) 2020-02-06T00:53:57.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:53:57.653+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.654923+0000) 2020-02-06T00:53:57.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:53:57.655+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.656165+0000) 2020-02-06T00:53:57.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:57.659+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:57.660395+0000) 2020-02-06T00:53:57.671 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:53:57.670+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:53:59.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:59.682+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:59.683593+0000) 2020-02-06T00:53:59.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:53:59.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:53:59.931618+0000) 2020-02-06T00:54:00.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:00.533+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:00.534247+0000) 2020-02-06T00:54:00.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:00.791+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:00.793055+0000) 2020-02-06T00:54:00.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:00.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:00.814512+0000) 2020-02-06T00:54:00.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:00.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:00.814613+0000) 2020-02-06T00:54:02.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:02.676+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.677047+0000) 2020-02-06T00:54:02.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:02.676+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.677712+0000) 2020-02-06T00:54:02.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:02.677+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.678097+0000) 2020-02-06T00:54:02.685 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:02.678+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.679975+0000) 2020-02-06T00:54:02.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:02.679+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.680818+0000) 2020-02-06T00:54:02.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:02.681+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.682212+0000) 2020-02-06T00:54:02.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:02.678+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.678990+0000) 2020-02-06T00:54:02.686 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:02.678+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.679352+0000) 2020-02-06T00:54:02.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:02.680+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.680619+0000) 2020-02-06T00:54:02.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:02.681+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.681473+0000) 2020-02-06T00:54:02.695 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:02.681+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.681772+0000) 2020-02-06T00:54:02.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:02.681+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.682155+0000) 2020-02-06T00:54:02.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:02.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.682675+0000) 2020-02-06T00:54:02.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:02.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.689055+0000) 2020-02-06T00:54:02.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:02.696+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:02.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:02.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:02.916931+0000) 2020-02-06T00:54:03.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:03.147+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:03.148101+0000) 2020-02-06T00:54:04.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:04.427+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:04.428632+0000) 2020-02-06T00:54:06.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:06.053+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:06.053938+0000) 2020-02-06T00:54:07.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:07.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.039686+0000) 2020-02-06T00:54:07.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:07.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.306896+0000) 2020-02-06T00:54:07.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:07.306+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.306861+0000) 2020-02-06T00:54:07.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:07.306+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.307025+0000) 2020-02-06T00:54:07.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:07.702+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.703313+0000) 2020-02-06T00:54:07.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:07.702+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.703788+0000) 2020-02-06T00:54:07.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:07.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.704366+0000) 2020-02-06T00:54:07.709 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:07.703+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.704914+0000) 2020-02-06T00:54:07.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:07.704+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.705351+0000) 2020-02-06T00:54:07.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:07.705+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.706038+0000) 2020-02-06T00:54:07.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:07.706+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.706715+0000) 2020-02-06T00:54:07.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:07.706+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.707510+0000) 2020-02-06T00:54:07.711 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:07.706+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.706498+0000) 2020-02-06T00:54:07.711 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:07.707+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.708112+0000) 2020-02-06T00:54:07.712 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:07.707+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.708237+0000) 2020-02-06T00:54:07.712 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:07.708+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.708617+0000) 2020-02-06T00:54:07.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:07.713+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.714228+0000) 2020-02-06T00:54:07.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:07.713+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:07.714548+0000) 2020-02-06T00:54:07.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:07.723+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:09.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:09.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:09.683809+0000) 2020-02-06T00:54:09.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:09.930+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:09.931874+0000) 2020-02-06T00:54:10.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:10.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:10.534484+0000) 2020-02-06T00:54:10.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:10.792+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:10.793336+0000) 2020-02-06T00:54:10.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:10.813+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:10.814737+0000) 2020-02-06T00:54:10.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:10.813+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:10.814821+0000) 2020-02-06T00:54:12.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:12.728+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.729365+0000) 2020-02-06T00:54:12.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:12.729+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.730389+0000) 2020-02-06T00:54:12.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:12.729+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.730949+0000) 2020-02-06T00:54:12.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:12.730+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.731343+0000) 2020-02-06T00:54:12.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:12.731+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.731688+0000) 2020-02-06T00:54:12.733 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:12.732+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.732778+0000) 2020-02-06T00:54:12.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:12.731+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.732838+0000) 2020-02-06T00:54:12.734 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:12.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.733483+0000) 2020-02-06T00:54:12.734 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:12.733+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.733955+0000) 2020-02-06T00:54:12.735 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:12.733+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.734973+0000) 2020-02-06T00:54:12.736 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:12.734+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.735174+0000) 2020-02-06T00:54:12.736 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:12.735+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.735754+0000) 2020-02-06T00:54:12.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:12.738+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.739362+0000) 2020-02-06T00:54:12.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:12.738+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.739726+0000) 2020-02-06T00:54:12.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:12.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:12.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:12.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:12.917115+0000) 2020-02-06T00:54:13.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:13.147+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:13.148326+0000) 2020-02-06T00:54:14.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:14.436+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:14.436882+0000) 2020-02-06T00:54:16.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:16.053+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:16.054239+0000) 2020-02-06T00:54:17.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:17.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.039956+0000) 2020-02-06T00:54:17.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:17.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.307174+0000) 2020-02-06T00:54:17.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:17.306+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.307148+0000) 2020-02-06T00:54:17.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:17.306+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.307296+0000) 2020-02-06T00:54:17.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:17.753+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.754682+0000) 2020-02-06T00:54:17.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:17.755+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.756314+0000) 2020-02-06T00:54:17.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:17.755+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.756756+0000) 2020-02-06T00:54:17.758 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:17.756+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.757129+0000) 2020-02-06T00:54:17.759 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:17.757+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.758310+0000) 2020-02-06T00:54:17.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:17.757+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.757916+0000) 2020-02-06T00:54:17.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:17.758+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.759121+0000) 2020-02-06T00:54:17.760 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:17.759+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.759706+0000) 2020-02-06T00:54:17.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:17.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.759891+0000) 2020-02-06T00:54:17.761 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:17.758+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.760034+0000) 2020-02-06T00:54:17.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:17.761+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.761867+0000) 2020-02-06T00:54:17.762 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:17.761+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.762073+0000) 2020-02-06T00:54:17.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:17.765+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.766086+0000) 2020-02-06T00:54:17.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:17.765+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:17.766364+0000) 2020-02-06T00:54:17.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:17.776+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:19.114 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:54:19.120 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:54:19.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:19.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:19.684050+0000) 2020-02-06T00:54:19.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:19.931+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:19.932125+0000) 2020-02-06T00:54:20.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:20.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:20.534733+0000) 2020-02-06T00:54:20.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:20.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:20.793577+0000) 2020-02-06T00:54:20.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:20.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:20.814964+0000) 2020-02-06T00:54:20.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:20.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:20.815113+0000) 2020-02-06T00:54:22.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:22.783+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.784272+0000) 2020-02-06T00:54:22.786 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:22.784+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.784796+0000) 2020-02-06T00:54:22.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:22.784+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.785169+0000) 2020-02-06T00:54:22.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:22.784+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.785399+0000) 2020-02-06T00:54:22.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:22.784+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.785851+0000) 2020-02-06T00:54:22.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:22.786+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.786696+0000) 2020-02-06T00:54:22.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:22.786+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.787099+0000) 2020-02-06T00:54:22.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:22.785+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.786281+0000) 2020-02-06T00:54:22.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:22.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.786469+0000) 2020-02-06T00:54:22.789 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:22.787+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.787586+0000) 2020-02-06T00:54:22.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:22.787+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.788094+0000) 2020-02-06T00:54:22.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:22.786+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.787399+0000) 2020-02-06T00:54:22.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:22.787+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.788495+0000) 2020-02-06T00:54:22.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:22.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.792889+0000) 2020-02-06T00:54:22.803 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:22.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:22.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:22.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:22.917342+0000) 2020-02-06T00:54:23.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:23.148+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:23.148540+0000) 2020-02-06T00:54:24.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:24.444+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:24.445131+0000) 2020-02-06T00:54:26.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:26.054+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:26.054519+0000) 2020-02-06T00:54:27.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:27.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.040192+0000) 2020-02-06T00:54:27.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:27.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.307411+0000) 2020-02-06T00:54:27.308 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:27.307+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.307411+0000) 2020-02-06T00:54:27.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:27.307+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.307618+0000) 2020-02-06T00:54:27.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:27.806+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.807827+0000) 2020-02-06T00:54:27.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:27.807+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.808832+0000) 2020-02-06T00:54:27.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:27.808+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.809376+0000) 2020-02-06T00:54:27.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:27.809+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.810532+0000) 2020-02-06T00:54:27.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:27.809+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.810972+0000) 2020-02-06T00:54:27.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:27.811+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.811836+0000) 2020-02-06T00:54:27.813 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:27.811+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.812174+0000) 2020-02-06T00:54:27.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:27.811+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.812425+0000) 2020-02-06T00:54:27.818 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:27.811+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.812491+0000) 2020-02-06T00:54:27.820 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:27.812+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.813281+0000) 2020-02-06T00:54:27.820 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:27.813+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.813489+0000) 2020-02-06T00:54:27.820 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:27.813+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.814128+0000) 2020-02-06T00:54:27.821 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:27.814+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.814528+0000) 2020-02-06T00:54:27.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:27.815+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:27.816790+0000) 2020-02-06T00:54:27.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:27.827+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:29.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:29.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:29.684319+0000) 2020-02-06T00:54:29.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:29.932+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:29.932352+0000) 2020-02-06T00:54:30.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:30.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:30.534993+0000) 2020-02-06T00:54:30.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:30.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:30.793836+0000) 2020-02-06T00:54:30.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:30.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:30.815189+0000) 2020-02-06T00:54:30.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:30.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:30.815311+0000) 2020-02-06T00:54:32.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:32.832+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.833113+0000) 2020-02-06T00:54:32.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:32.832+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.834028+0000) 2020-02-06T00:54:32.835 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:32.834+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.834776+0000) 2020-02-06T00:54:32.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:32.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.835534+0000) 2020-02-06T00:54:32.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:32.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.836119+0000) 2020-02-06T00:54:32.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:32.835+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.836695+0000) 2020-02-06T00:54:32.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:32.836+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.837353+0000) 2020-02-06T00:54:32.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:32.836+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.837136+0000) 2020-02-06T00:54:32.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:32.836+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.837888+0000) 2020-02-06T00:54:32.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:32.837+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.838211+0000) 2020-02-06T00:54:32.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:32.838+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.838816+0000) 2020-02-06T00:54:32.841 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:32.838+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.839017+0000) 2020-02-06T00:54:32.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:32.838+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.839248+0000) 2020-02-06T00:54:32.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:32.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.842902+0000) 2020-02-06T00:54:32.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:32.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:32.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:32.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:32.917575+0000) 2020-02-06T00:54:33.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:33.148+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:33.148754+0000) 2020-02-06T00:54:34.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:34.453+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:34.453213+0000) 2020-02-06T00:54:36.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:36.054+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:36.054758+0000) 2020-02-06T00:54:37.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:37.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.040462+0000) 2020-02-06T00:54:37.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:37.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.307707+0000) 2020-02-06T00:54:37.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:37.307+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.307720+0000) 2020-02-06T00:54:37.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:37.307+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.307858+0000) 2020-02-06T00:54:37.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:37.863+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.863098+0000) 2020-02-06T00:54:37.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:37.864+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.865193+0000) 2020-02-06T00:54:37.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:37.865+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.865685+0000) 2020-02-06T00:54:37.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:37.866+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.867280+0000) 2020-02-06T00:54:37.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:37.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.868292+0000) 2020-02-06T00:54:37.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:37.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.868784+0000) 2020-02-06T00:54:37.872 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:37.868+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.869230+0000) 2020-02-06T00:54:37.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:37.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.863749+0000) 2020-02-06T00:54:37.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:37.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.866640+0000) 2020-02-06T00:54:37.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:37.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.865919+0000) 2020-02-06T00:54:37.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:37.866+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.866166+0000) 2020-02-06T00:54:37.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:37.867+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.867329+0000) 2020-02-06T00:54:37.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:37.867+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.867759+0000) 2020-02-06T00:54:37.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:37.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:37.874999+0000) 2020-02-06T00:54:37.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:37.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:39.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:39.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:39.684554+0000) 2020-02-06T00:54:39.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:39.932+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:39.932586+0000) 2020-02-06T00:54:40.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:40.535+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:40.535223+0000) 2020-02-06T00:54:40.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:40.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:40.794071+0000) 2020-02-06T00:54:40.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:40.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:40.815472+0000) 2020-02-06T00:54:40.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:40.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:40.815586+0000) 2020-02-06T00:54:42.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:42.886+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.887593+0000) 2020-02-06T00:54:42.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:42.887+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.888618+0000) 2020-02-06T00:54:42.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:42.887+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.888967+0000) 2020-02-06T00:54:42.891 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:42.889+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.890697+0000) 2020-02-06T00:54:42.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:42.889+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.891347+0000) 2020-02-06T00:54:42.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:42.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.891800+0000) 2020-02-06T00:54:42.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:42.890+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.892373+0000) 2020-02-06T00:54:42.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:42.891+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.892661+0000) 2020-02-06T00:54:42.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:42.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.893508+0000) 2020-02-06T00:54:42.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:42.892+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.893875+0000) 2020-02-06T00:54:42.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:42.891+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.893179+0000) 2020-02-06T00:54:42.896 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:42.893+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.894445+0000) 2020-02-06T00:54:42.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:42.893+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.895077+0000) 2020-02-06T00:54:42.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:42.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.898417+0000) 2020-02-06T00:54:42.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:42.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187800 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:42.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:42.916+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:42.917753+0000) 2020-02-06T00:54:43.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:43.147+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:43.148962+0000) 2020-02-06T00:54:44.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:44.463+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:44.464991+0000) 2020-02-06T00:54:46.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:46.053+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:46.054995+0000) 2020-02-06T00:54:47.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:47.039+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.040704+0000) 2020-02-06T00:54:47.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:47.306+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.307939+0000) 2020-02-06T00:54:47.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:47.306+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.307916+0000) 2020-02-06T00:54:47.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:47.306+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.308088+0000) 2020-02-06T00:54:47.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:47.911+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.913032+0000) 2020-02-06T00:54:47.915 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:47.912+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.913693+0000) 2020-02-06T00:54:47.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:47.913+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.914738+0000) 2020-02-06T00:54:47.917 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:47.914+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.915577+0000) 2020-02-06T00:54:47.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:47.915+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.916408+0000) 2020-02-06T00:54:47.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:47.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.916904+0000) 2020-02-06T00:54:47.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:47.915+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.917372+0000) 2020-02-06T00:54:47.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:47.916+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.917864+0000) 2020-02-06T00:54:47.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:47.916+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.918140+0000) 2020-02-06T00:54:47.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:47.917+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.918501+0000) 2020-02-06T00:54:47.921 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:47.918+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.919462+0000) 2020-02-06T00:54:47.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:47.917+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.919199+0000) 2020-02-06T00:54:47.922 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:47.918+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.919730+0000) 2020-02-06T00:54:47.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:47.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:47.924807+0000) 2020-02-06T00:54:47.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:47.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187838 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:49.277 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:54:49.284 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:54:49.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:49.683+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:49.684780+0000) 2020-02-06T00:54:49.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:49.931+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:49.932819+0000) 2020-02-06T00:54:50.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:50.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:50.535490+0000) 2020-02-06T00:54:50.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:50.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:50.794346+0000) 2020-02-06T00:54:50.816 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:50.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:50.815726+0000) 2020-02-06T00:54:50.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:50.814+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:50.815860+0000) 2020-02-06T00:54:52.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:52.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.917987+0000) 2020-02-06T00:54:52.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:52.938+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.939706+0000) 2020-02-06T00:54:52.941 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:52.938+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.940339+0000) 2020-02-06T00:54:52.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:52.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.940491+0000) 2020-02-06T00:54:52.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:52.940+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.941834+0000) 2020-02-06T00:54:52.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:52.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.942138+0000) 2020-02-06T00:54:52.944 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:52.941+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.942854+0000) 2020-02-06T00:54:52.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:52.942+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.943212+0000) 2020-02-06T00:54:52.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:52.941+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.943113+0000) 2020-02-06T00:54:52.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:52.942+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.943715+0000) 2020-02-06T00:54:52.945 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:52.942+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.943930+0000) 2020-02-06T00:54:52.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:52.943+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.944397+0000) 2020-02-06T00:54:52.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:52.944+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.945754+0000) 2020-02-06T00:54:52.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:52.945+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.947039+0000) 2020-02-06T00:54:52.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:52.946+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:52.947307+0000) 2020-02-06T00:54:52.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:52.958+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187879 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:53.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:53.148+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:53.149174+0000) 2020-02-06T00:54:54.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:54.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:54.472841+0000) 2020-02-06T00:54:56.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:56.054+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:56.055265+0000) 2020-02-06T00:54:57.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:57.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.040931+0000) 2020-02-06T00:54:57.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:57.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.308187+0000) 2020-02-06T00:54:57.309 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:57.307+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.308160+0000) 2020-02-06T00:54:57.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:57.307+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.308340+0000) 2020-02-06T00:54:57.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:54:57.963+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.964715+0000) 2020-02-06T00:54:57.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:54:57.964+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.965344+0000) 2020-02-06T00:54:57.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:54:57.964+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.966213+0000) 2020-02-06T00:54:57.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:54:57.966+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.967479+0000) 2020-02-06T00:54:57.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:54:57.966+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.967630+0000) 2020-02-06T00:54:57.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:57.967+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.968387+0000) 2020-02-06T00:54:57.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:54:57.967+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.969374+0000) 2020-02-06T00:54:57.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:54:57.969+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.970719+0000) 2020-02-06T00:54:57.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:54:57.970+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.971120+0000) 2020-02-06T00:54:57.972 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:54:57.970+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.971409+0000) 2020-02-06T00:54:57.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:54:57.970+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.971901+0000) 2020-02-06T00:54:57.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:54:57.970+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.972072+0000) 2020-02-06T00:54:57.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:57.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.975290+0000) 2020-02-06T00:54:57.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:57.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:57.975549+0000) 2020-02-06T00:54:57.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:54:57.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:54:59.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:59.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:59.685103+0000) 2020-02-06T00:54:59.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:54:59.932+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:54:59.933069+0000) 2020-02-06T00:55:00.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:00.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:00.535730+0000) 2020-02-06T00:55:00.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:00.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:00.794532+0000) 2020-02-06T00:55:00.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:00.814+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:00.815954+0000) 2020-02-06T00:55:00.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:00.815+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:00.816168+0000) 2020-02-06T00:55:02.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:02.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.918248+0000) 2020-02-06T00:55:02.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:02.990+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.991310+0000) 2020-02-06T00:55:02.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:02.991+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.992410+0000) 2020-02-06T00:55:02.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:02.990+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.992029+0000) 2020-02-06T00:55:02.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:02.991+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.992846+0000) 2020-02-06T00:55:02.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:02.992+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.993632+0000) 2020-02-06T00:55:02.995 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:02.992+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.994382+0000) 2020-02-06T00:55:02.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:02.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.994238+0000) 2020-02-06T00:55:02.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:02.993+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.994602+0000) 2020-02-06T00:55:02.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:02.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.995573+0000) 2020-02-06T00:55:02.996 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:02.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.995584+0000) 2020-02-06T00:55:02.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:02.993+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.995389+0000) 2020-02-06T00:55:02.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:02.994+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.996367+0000) 2020-02-06T00:55:02.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:02.994+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:02.995957+0000) 2020-02-06T00:55:03.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:02.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:03.000672+0000) 2020-02-06T00:55:03.012 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:03.010+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 187963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:03.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:03.148+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:03.149331+0000) 2020-02-06T00:55:04.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:04.480+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:04.481408+0000) 2020-02-06T00:55:06.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:06.054+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:06.055522+0000) 2020-02-06T00:55:07.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:07.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:07.041200+0000) 2020-02-06T00:55:07.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:07.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:07.308500+0000) 2020-02-06T00:55:07.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:07.307+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:07.308500+0000) 2020-02-06T00:55:07.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:07.307+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:07.308637+0000) 2020-02-06T00:55:08.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:08.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.016830+0000) 2020-02-06T00:55:08.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:08.016+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.017603+0000) 2020-02-06T00:55:08.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:08.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.018155+0000) 2020-02-06T00:55:08.019 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:08.017+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.018660+0000) 2020-02-06T00:55:08.020 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:08.017+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.019187+0000) 2020-02-06T00:55:08.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:08.018+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.019518+0000) 2020-02-06T00:55:08.021 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:08.019+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.020548+0000) 2020-02-06T00:55:08.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:08.018+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.019976+0000) 2020-02-06T00:55:08.022 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:08.019+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.020606+0000) 2020-02-06T00:55:08.023 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:08.019+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.020391+0000) 2020-02-06T00:55:08.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:08.019+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.020417+0000) 2020-02-06T00:55:08.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:08.019+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.020849+0000) 2020-02-06T00:55:08.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:08.025+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.026500+0000) 2020-02-06T00:55:08.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:08.026+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:08.027602+0000) 2020-02-06T00:55:08.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:08.035+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188004 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:09.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:09.684+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:09.685372+0000) 2020-02-06T00:55:09.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:09.932+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:09.933380+0000) 2020-02-06T00:55:10.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:10.534+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:10.535944+0000) 2020-02-06T00:55:10.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:10.793+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:10.794782+0000) 2020-02-06T00:55:10.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:10.815+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:10.816181+0000) 2020-02-06T00:55:10.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:10.815+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:10.816434+0000) 2020-02-06T00:55:12.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:12.917+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:12.918491+0000) 2020-02-06T00:55:13.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:13.042+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.043441+0000) 2020-02-06T00:55:13.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:13.043+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.044188+0000) 2020-02-06T00:55:13.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:13.043+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.044694+0000) 2020-02-06T00:55:13.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:13.044+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.045500+0000) 2020-02-06T00:55:13.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:13.045+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.046176+0000) 2020-02-06T00:55:13.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:13.045+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.046398+0000) 2020-02-06T00:55:13.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:13.044+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.046126+0000) 2020-02-06T00:55:13.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:13.045+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.047246+0000) 2020-02-06T00:55:13.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:13.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.047238+0000) 2020-02-06T00:55:13.049 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:13.046+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.047488+0000) 2020-02-06T00:55:13.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:13.046+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.047819+0000) 2020-02-06T00:55:13.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:13.047+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.048594+0000) 2020-02-06T00:55:13.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:13.047+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.048778+0000) 2020-02-06T00:55:13.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:13.054+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.055616+0000) 2020-02-06T00:55:13.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:13.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:13.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:13.149+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:13.149540+0000) 2020-02-06T00:55:14.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:14.489+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:14.489675+0000) 2020-02-06T00:55:16.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:16.055+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:16.055782+0000) 2020-02-06T00:55:17.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:17.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:17.041429+0000) 2020-02-06T00:55:17.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:17.307+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:17.308761+0000) 2020-02-06T00:55:17.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:17.307+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:17.308722+0000) 2020-02-06T00:55:17.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:17.307+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:17.308911+0000) 2020-02-06T00:55:18.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:18.069+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.070327+0000) 2020-02-06T00:55:18.072 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:18.069+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.070829+0000) 2020-02-06T00:55:18.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:18.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.071336+0000) 2020-02-06T00:55:18.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:18.070+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.072117+0000) 2020-02-06T00:55:18.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:18.071+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.072676+0000) 2020-02-06T00:55:18.074 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:18.071+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.072704+0000) 2020-02-06T00:55:18.074 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:18.071+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.073053+0000) 2020-02-06T00:55:18.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:18.072+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.074040+0000) 2020-02-06T00:55:18.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:18.072+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.074384+0000) 2020-02-06T00:55:18.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:18.073+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.074498+0000) 2020-02-06T00:55:18.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:18.073+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.074833+0000) 2020-02-06T00:55:18.076 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:18.073+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.074949+0000) 2020-02-06T00:55:18.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:18.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.078487+0000) 2020-02-06T00:55:18.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:18.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:18.078707+0000) 2020-02-06T00:55:18.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:18.090+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188087 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:19.482 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:55:19.488 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:55:19.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:19.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:19.685618+0000) 2020-02-06T00:55:19.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:19.933+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:19.933614+0000) 2020-02-06T00:55:20.537 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:20.535+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:20.536193+0000) 2020-02-06T00:55:20.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:20.794+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:20.795014+0000) 2020-02-06T00:55:20.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:20.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:20.816402+0000) 2020-02-06T00:55:20.817 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:20.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:20.816666+0000) 2020-02-06T00:55:22.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:22.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:22.918748+0000) 2020-02-06T00:55:23.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:23.097+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.098341+0000) 2020-02-06T00:55:23.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:23.097+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.099254+0000) 2020-02-06T00:55:23.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:23.098+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.099366+0000) 2020-02-06T00:55:23.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:23.099+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.100377+0000) 2020-02-06T00:55:23.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:23.099+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.100455+0000) 2020-02-06T00:55:23.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:23.099+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.100569+0000) 2020-02-06T00:55:23.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:23.099+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.100927+0000) 2020-02-06T00:55:23.103 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:23.100+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.101731+0000) 2020-02-06T00:55:23.103 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:23.100+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.101864+0000) 2020-02-06T00:55:23.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:23.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.102822+0000) 2020-02-06T00:55:23.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:23.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.103225+0000) 2020-02-06T00:55:23.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:23.101+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.103261+0000) 2020-02-06T00:55:23.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:23.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.103737+0000) 2020-02-06T00:55:23.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:23.106+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.108056+0000) 2020-02-06T00:55:23.130 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:23.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:23.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:23.148+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:23.149813+0000) 2020-02-06T00:55:24.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:24.497+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:24.498050+0000) 2020-02-06T00:55:26.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:26.055+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:26.056027+0000) 2020-02-06T00:55:27.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:27.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:27.041615+0000) 2020-02-06T00:55:27.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:27.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:27.309016+0000) 2020-02-06T00:55:27.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:27.308+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:27.309021+0000) 2020-02-06T00:55:27.310 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:27.308+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:27.309119+0000) 2020-02-06T00:55:28.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:28.120+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.121824+0000) 2020-02-06T00:55:28.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:28.121+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.122795+0000) 2020-02-06T00:55:28.125 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:28.123+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.124363+0000) 2020-02-06T00:55:28.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:28.122+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.124345+0000) 2020-02-06T00:55:28.126 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:28.124+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.125261+0000) 2020-02-06T00:55:28.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:28.124+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.125676+0000) 2020-02-06T00:55:28.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:28.125+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.126519+0000) 2020-02-06T00:55:28.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:28.126+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.127347+0000) 2020-02-06T00:55:28.128 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:28.125+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.126928+0000) 2020-02-06T00:55:28.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:28.126+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.127597+0000) 2020-02-06T00:55:28.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:28.126+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.127637+0000) 2020-02-06T00:55:28.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:28.126+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.128336+0000) 2020-02-06T00:55:28.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:28.127+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.128762+0000) 2020-02-06T00:55:28.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:28.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:28.131090+0000) 2020-02-06T00:55:28.143 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:28.140+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:29.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:29.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:29.685874+0000) 2020-02-06T00:55:29.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:29.933+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:29.933845+0000) 2020-02-06T00:55:30.537 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:30.535+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:30.536464+0000) 2020-02-06T00:55:30.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:30.795+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:30.795282+0000) 2020-02-06T00:55:30.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:30.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:30.816637+0000) 2020-02-06T00:55:30.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:30.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:30.816854+0000) 2020-02-06T00:55:32.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:32.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:32.918990+0000) 2020-02-06T00:55:33.148 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:33.147+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.147574+0000) 2020-02-06T00:55:33.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:33.149+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.149120+0000) 2020-02-06T00:55:33.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:33.149+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.150053+0000) 2020-02-06T00:55:33.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:33.149+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.150018+0000) 2020-02-06T00:55:33.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:33.149+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.150333+0000) 2020-02-06T00:55:33.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:33.150+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.150706+0000) 2020-02-06T00:55:33.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:33.150+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.150833+0000) 2020-02-06T00:55:33.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:33.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.151517+0000) 2020-02-06T00:55:33.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:33.151+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.151810+0000) 2020-02-06T00:55:33.153 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:33.152+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.152542+0000) 2020-02-06T00:55:33.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:33.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.155282+0000) 2020-02-06T00:55:33.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:33.155+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.155496+0000) 2020-02-06T00:55:33.159 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:33.153+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.153636+0000) 2020-02-06T00:55:33.159 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:33.153+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.153904+0000) 2020-02-06T00:55:33.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:33.153+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:33.154205+0000) 2020-02-06T00:55:33.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:33.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188212 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:34.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:34.505+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:34.505820+0000) 2020-02-06T00:55:36.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:36.055+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:36.056273+0000) 2020-02-06T00:55:37.042 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:37.040+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:37.041836+0000) 2020-02-06T00:55:37.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:37.308+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:37.309279+0000) 2020-02-06T00:55:37.310 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:37.308+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:37.309291+0000) 2020-02-06T00:55:37.311 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:37.309+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:37.309499+0000) 2020-02-06T00:55:38.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:38.173+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.174438+0000) 2020-02-06T00:55:38.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:38.173+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.174880+0000) 2020-02-06T00:55:38.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:38.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.175436+0000) 2020-02-06T00:55:38.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:38.174+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.175747+0000) 2020-02-06T00:55:38.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:38.175+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.176190+0000) 2020-02-06T00:55:38.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:38.176+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.176936+0000) 2020-02-06T00:55:38.178 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:38.177+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.177498+0000) 2020-02-06T00:55:38.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:38.177+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.177971+0000) 2020-02-06T00:55:38.179 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:38.178+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.178702+0000) 2020-02-06T00:55:38.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:38.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.178626+0000) 2020-02-06T00:55:38.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:38.178+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.179218+0000) 2020-02-06T00:55:38.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:38.178+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.179644+0000) 2020-02-06T00:55:38.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:38.179+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.179802+0000) 2020-02-06T00:55:38.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:38.184+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:38.185187+0000) 2020-02-06T00:55:38.195 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:38.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:39.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:39.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:39.686136+0000) 2020-02-06T00:55:39.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:39.933+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:39.934069+0000) 2020-02-06T00:55:40.537 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:40.536+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:40.536730+0000) 2020-02-06T00:55:40.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:40.794+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:40.795525+0000) 2020-02-06T00:55:40.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:40.815+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:40.816896+0000) 2020-02-06T00:55:40.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:40.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:40.817073+0000) 2020-02-06T00:55:42.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:42.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:42.919250+0000) 2020-02-06T00:55:43.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:43.149+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.150274+0000) 2020-02-06T00:55:43.202 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:43.200+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.201307+0000) 2020-02-06T00:55:43.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:43.201+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.201932+0000) 2020-02-06T00:55:43.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:43.202+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.202992+0000) 2020-02-06T00:55:43.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:43.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.203495+0000) 2020-02-06T00:55:43.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:43.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.204456+0000) 2020-02-06T00:55:43.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:43.203+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.204808+0000) 2020-02-06T00:55:43.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:43.203+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.205054+0000) 2020-02-06T00:55:43.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:43.204+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.205384+0000) 2020-02-06T00:55:43.207 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:43.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.205323+0000) 2020-02-06T00:55:43.207 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:43.204+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.205823+0000) 2020-02-06T00:55:43.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:43.205+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.205884+0000) 2020-02-06T00:55:43.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:43.205+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.206102+0000) 2020-02-06T00:55:43.208 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:43.206+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.206516+0000) 2020-02-06T00:55:43.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:43.210+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:43.211820+0000) 2020-02-06T00:55:43.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:43.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188295 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:44.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:44.512+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:44.514008+0000) 2020-02-06T00:55:46.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:46.056+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:46.056516+0000) 2020-02-06T00:55:47.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:47.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:47.042138+0000) 2020-02-06T00:55:47.310 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:47.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:47.309651+0000) 2020-02-06T00:55:47.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:47.309+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:47.309654+0000) 2020-02-06T00:55:47.311 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:47.309+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:47.309975+0000) 2020-02-06T00:55:48.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:48.225+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.226307+0000) 2020-02-06T00:55:48.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:48.226+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.227067+0000) 2020-02-06T00:55:48.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:48.225+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.227002+0000) 2020-02-06T00:55:48.228 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:48.227+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.227856+0000) 2020-02-06T00:55:48.229 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:48.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.228061+0000) 2020-02-06T00:55:48.229 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:48.227+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.228460+0000) 2020-02-06T00:55:48.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:48.228+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.229189+0000) 2020-02-06T00:55:48.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:48.229+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.229833+0000) 2020-02-06T00:55:48.233 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:48.230+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.230896+0000) 2020-02-06T00:55:48.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:48.231+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.232356+0000) 2020-02-06T00:55:48.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:48.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.232233+0000) 2020-02-06T00:55:48.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:48.231+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.232461+0000) 2020-02-06T00:55:48.234 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:48.232+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.232840+0000) 2020-02-06T00:55:48.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:48.237+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:48.238654+0000) 2020-02-06T00:55:48.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:48.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:49.637 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:55:49.643 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:55:49.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:49.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:49.686335+0000) 2020-02-06T00:55:49.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:49.933+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:49.934411+0000) 2020-02-06T00:55:50.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:50.536+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:50.536986+0000) 2020-02-06T00:55:50.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:50.794+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:50.795757+0000) 2020-02-06T00:55:50.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:50.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:50.817126+0000) 2020-02-06T00:55:50.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:50.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:50.817315+0000) 2020-02-06T00:55:52.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:52.919+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:52.919500+0000) 2020-02-06T00:55:53.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:53.149+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.150506+0000) 2020-02-06T00:55:53.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:53.250+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.251987+0000) 2020-02-06T00:55:53.253 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:53.252+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.252825+0000) 2020-02-06T00:55:53.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:53.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.254341+0000) 2020-02-06T00:55:53.256 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:53.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.255679+0000) 2020-02-06T00:55:53.257 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:53.255+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.256405+0000) 2020-02-06T00:55:53.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:53.256+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.256722+0000) 2020-02-06T00:55:53.258 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:53.256+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.257383+0000) 2020-02-06T00:55:53.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:53.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.257889+0000) 2020-02-06T00:55:53.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:53.257+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.257564+0000) 2020-02-06T00:55:53.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:53.257+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.258040+0000) 2020-02-06T00:55:53.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:53.257+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.258355+0000) 2020-02-06T00:55:53.260 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:53.257+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.258885+0000) 2020-02-06T00:55:53.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:53.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.260554+0000) 2020-02-06T00:55:53.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:53.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:53.260749+0000) 2020-02-06T00:55:53.272 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:53.271+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:54.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:54.521+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:54.522252+0000) 2020-02-06T00:55:56.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:56.056+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:56.056819+0000) 2020-02-06T00:55:57.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:57.041+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:57.042374+0000) 2020-02-06T00:55:57.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:57.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:57.309913+0000) 2020-02-06T00:55:57.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:57.309+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:57.309928+0000) 2020-02-06T00:55:57.311 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:57.309+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:57.310225+0000) 2020-02-06T00:55:58.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:55:58.277+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.278092+0000) 2020-02-06T00:55:58.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:55:58.277+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.278288+0000) 2020-02-06T00:55:58.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:55:58.277+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.278536+0000) 2020-02-06T00:55:58.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:58.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.279054+0000) 2020-02-06T00:55:58.281 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:55:58.279+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.279576+0000) 2020-02-06T00:55:58.281 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:55:58.279+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.279888+0000) 2020-02-06T00:55:58.281 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:55:58.278+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.280008+0000) 2020-02-06T00:55:58.281 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:55:58.279+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.280443+0000) 2020-02-06T00:55:58.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:55:58.280+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.281288+0000) 2020-02-06T00:55:58.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:58.280+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.281409+0000) 2020-02-06T00:55:58.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:55:58.281+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.282337+0000) 2020-02-06T00:55:58.283 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:55:58.282+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.282871+0000) 2020-02-06T00:55:58.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:55:58.282+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.283202+0000) 2020-02-06T00:55:58.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:58.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:58.286978+0000) 2020-02-06T00:55:58.298 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:55:58.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:55:59.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:59.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:59.686593+0000) 2020-02-06T00:55:59.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:55:59.933+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:55:59.934687+0000) 2020-02-06T00:56:00.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:00.536+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:00.537312+0000) 2020-02-06T00:56:00.797 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:00.794+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:00.796028+0000) 2020-02-06T00:56:00.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:00.816+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:00.817355+0000) 2020-02-06T00:56:00.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:00.816+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:00.817521+0000) 2020-02-06T00:56:02.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:02.919+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:02.919768+0000) 2020-02-06T00:56:03.151 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:03.150+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.150737+0000) 2020-02-06T00:56:03.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:03.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.303338+0000) 2020-02-06T00:56:03.305 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:03.303+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.304106+0000) 2020-02-06T00:56:03.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:03.304+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.304513+0000) 2020-02-06T00:56:03.306 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:03.304+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.305158+0000) 2020-02-06T00:56:03.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:03.304+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.305364+0000) 2020-02-06T00:56:03.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:03.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.305807+0000) 2020-02-06T00:56:03.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:03.306+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.307291+0000) 2020-02-06T00:56:03.308 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:03.306+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.307657+0000) 2020-02-06T00:56:03.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:03.308+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.308796+0000) 2020-02-06T00:56:03.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:03.308+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.308917+0000) 2020-02-06T00:56:03.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:03.308+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.309266+0000) 2020-02-06T00:56:03.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:03.307+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.308291+0000) 2020-02-06T00:56:03.312 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:03.307+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.308457+0000) 2020-02-06T00:56:03.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:03.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:03.312380+0000) 2020-02-06T00:56:03.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:03.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:04.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:04.533+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:04.533768+0000) 2020-02-06T00:56:06.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:06.056+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:06.057063+0000) 2020-02-06T00:56:07.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:07.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:07.042609+0000) 2020-02-06T00:56:07.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:07.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:07.310181+0000) 2020-02-06T00:56:07.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:07.309+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:07.310173+0000) 2020-02-06T00:56:07.311 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:07.310+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:07.310507+0000) 2020-02-06T00:56:08.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:08.326+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.327815+0000) 2020-02-06T00:56:08.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:08.328+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.329429+0000) 2020-02-06T00:56:08.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:08.329+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.330356+0000) 2020-02-06T00:56:08.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:08.330+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.331191+0000) 2020-02-06T00:56:08.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:08.330+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.331434+0000) 2020-02-06T00:56:08.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:08.330+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.331891+0000) 2020-02-06T00:56:08.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:08.331+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.332397+0000) 2020-02-06T00:56:08.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:08.332+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.333392+0000) 2020-02-06T00:56:08.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:08.332+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.333874+0000) 2020-02-06T00:56:08.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:08.334+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.334504+0000) 2020-02-06T00:56:08.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:08.333+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.334598+0000) 2020-02-06T00:56:08.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:08.335+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.335557+0000) 2020-02-06T00:56:08.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:08.335+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.335689+0000) 2020-02-06T00:56:08.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:08.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:08.340025+0000) 2020-02-06T00:56:08.351 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:08.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:09.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:09.685+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:09.686821+0000) 2020-02-06T00:56:09.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:09.934+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:09.934920+0000) 2020-02-06T00:56:10.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:10.536+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:10.537560+0000) 2020-02-06T00:56:10.797 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:10.795+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:10.796296+0000) 2020-02-06T00:56:10.818 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:10.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:10.817584+0000) 2020-02-06T00:56:10.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:10.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:10.817744+0000) 2020-02-06T00:56:12.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:12.918+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:12.920024+0000) 2020-02-06T00:56:13.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:13.150+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.150972+0000) 2020-02-06T00:56:13.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:13.354+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.355600+0000) 2020-02-06T00:56:13.358 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:13.356+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.357639+0000) 2020-02-06T00:56:13.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:13.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.357853+0000) 2020-02-06T00:56:13.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:13.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.358249+0000) 2020-02-06T00:56:13.359 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:13.357+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.358201+0000) 2020-02-06T00:56:13.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:13.357+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.358549+0000) 2020-02-06T00:56:13.360 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:13.357+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.358911+0000) 2020-02-06T00:56:13.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:13.358+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.359918+0000) 2020-02-06T00:56:13.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:13.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.360877+0000) 2020-02-06T00:56:13.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:13.358+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.360187+0000) 2020-02-06T00:56:13.362 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:13.360+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.361861+0000) 2020-02-06T00:56:13.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:13.361+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.362339+0000) 2020-02-06T00:56:13.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:13.361+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.362864+0000) 2020-02-06T00:56:13.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:13.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:13.365023+0000) 2020-02-06T00:56:13.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:13.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188544 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:14.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:14.541+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:14.541959+0000) 2020-02-06T00:56:16.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:16.056+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:16.057327+0000) 2020-02-06T00:56:17.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:17.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:17.042884+0000) 2020-02-06T00:56:17.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:17.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:17.310423+0000) 2020-02-06T00:56:17.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:17.309+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:17.310421+0000) 2020-02-06T00:56:17.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:17.309+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:17.310773+0000) 2020-02-06T00:56:18.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:18.381+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.382219+0000) 2020-02-06T00:56:18.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:18.383+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.384193+0000) 2020-02-06T00:56:18.385 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:18.383+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.384853+0000) 2020-02-06T00:56:18.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:18.384+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.385643+0000) 2020-02-06T00:56:18.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:18.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.386109+0000) 2020-02-06T00:56:18.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:18.385+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.386517+0000) 2020-02-06T00:56:18.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:18.385+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.386777+0000) 2020-02-06T00:56:18.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:18.386+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.388171+0000) 2020-02-06T00:56:18.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:18.386+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.388370+0000) 2020-02-06T00:56:18.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:18.387+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.388770+0000) 2020-02-06T00:56:18.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:18.387+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.388845+0000) 2020-02-06T00:56:18.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:18.388+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.389212+0000) 2020-02-06T00:56:18.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:18.389+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.390838+0000) 2020-02-06T00:56:18.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:18.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:18.391901+0000) 2020-02-06T00:56:18.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:18.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:19.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:19.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:19.687110+0000) 2020-02-06T00:56:19.791 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:56:19.797 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:56:19.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:19.934+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:19.935154+0000) 2020-02-06T00:56:20.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:20.536+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:20.537812+0000) 2020-02-06T00:56:20.797 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:20.796+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:20.796532+0000) 2020-02-06T00:56:20.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:20.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:20.817826+0000) 2020-02-06T00:56:20.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:20.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:20.818015+0000) 2020-02-06T00:56:22.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:22.919+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:22.920266+0000) 2020-02-06T00:56:23.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:23.151+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.151289+0000) 2020-02-06T00:56:23.410 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:23.408+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.409649+0000) 2020-02-06T00:56:23.412 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:23.409+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.410767+0000) 2020-02-06T00:56:23.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:23.410+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.411620+0000) 2020-02-06T00:56:23.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:23.409+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.411323+0000) 2020-02-06T00:56:23.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:23.411+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.412320+0000) 2020-02-06T00:56:23.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:23.411+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.412944+0000) 2020-02-06T00:56:23.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:23.412+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.413635+0000) 2020-02-06T00:56:23.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:23.412+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.413560+0000) 2020-02-06T00:56:23.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:23.413+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.414730+0000) 2020-02-06T00:56:23.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:23.414+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.415687+0000) 2020-02-06T00:56:23.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:23.414+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.415995+0000) 2020-02-06T00:56:23.417 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:23.414+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.416283+0000) 2020-02-06T00:56:23.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:23.418+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.419871+0000) 2020-02-06T00:56:23.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:23.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:23.420132+0000) 2020-02-06T00:56:23.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:23.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188627 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:24.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:24.551+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:24.551638+0000) 2020-02-06T00:56:26.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:26.056+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:26.057671+0000) 2020-02-06T00:56:27.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:27.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:27.043119+0000) 2020-02-06T00:56:27.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:27.309+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:27.310717+0000) 2020-02-06T00:56:27.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:27.309+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:27.310715+0000) 2020-02-06T00:56:27.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:27.309+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:27.311022+0000) 2020-02-06T00:56:28.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:28.436+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.438320+0000) 2020-02-06T00:56:28.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:28.438+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.438361+0000) 2020-02-06T00:56:28.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:28.439+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.439948+0000) 2020-02-06T00:56:28.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:28.438+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.440112+0000) 2020-02-06T00:56:28.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:28.441+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.441263+0000) 2020-02-06T00:56:28.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:28.440+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.441973+0000) 2020-02-06T00:56:28.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:28.440+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.441814+0000) 2020-02-06T00:56:28.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:28.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.442342+0000) 2020-02-06T00:56:28.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:28.442+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.442807+0000) 2020-02-06T00:56:28.444 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:28.442+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.443032+0000) 2020-02-06T00:56:28.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:28.441+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.442947+0000) 2020-02-06T00:56:28.444 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:28.441+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.443359+0000) 2020-02-06T00:56:28.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:28.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.445671+0000) 2020-02-06T00:56:28.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:28.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:28.446589+0000) 2020-02-06T00:56:28.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:28.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:29.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:29.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:29.687347+0000) 2020-02-06T00:56:29.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:29.934+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:29.935407+0000) 2020-02-06T00:56:30.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:30.537+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:30.538069+0000) 2020-02-06T00:56:30.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:30.795+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:30.796758+0000) 2020-02-06T00:56:30.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:30.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:30.818136+0000) 2020-02-06T00:56:30.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:30.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:30.818312+0000) 2020-02-06T00:56:32.921 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:32.919+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:32.920517+0000) 2020-02-06T00:56:33.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:33.150+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.151522+0000) 2020-02-06T00:56:33.465 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:33.463+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.464566+0000) 2020-02-06T00:56:33.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:33.465+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.466699+0000) 2020-02-06T00:56:33.467 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:33.465+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.466770+0000) 2020-02-06T00:56:33.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:33.466+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.467341+0000) 2020-02-06T00:56:33.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:33.465+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.467385+0000) 2020-02-06T00:56:33.469 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:33.466+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.467828+0000) 2020-02-06T00:56:33.469 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:33.467+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.468433+0000) 2020-02-06T00:56:33.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:33.467+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.468755+0000) 2020-02-06T00:56:33.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:33.468+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.469204+0000) 2020-02-06T00:56:33.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:33.468+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.469549+0000) 2020-02-06T00:56:33.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:33.468+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.469914+0000) 2020-02-06T00:56:33.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:33.469+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.470707+0000) 2020-02-06T00:56:33.471 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:33.469+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.471028+0000) 2020-02-06T00:56:33.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:33.474+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:33.475071+0000) 2020-02-06T00:56:33.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:33.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:34.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:34.561+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:34.562223+0000) 2020-02-06T00:56:36.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:36.057+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:36.057933+0000) 2020-02-06T00:56:37.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:37.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:37.043301+0000) 2020-02-06T00:56:37.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:37.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:37.310969+0000) 2020-02-06T00:56:37.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:37.310+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:37.310968+0000) 2020-02-06T00:56:37.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:37.310+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:37.311266+0000) 2020-02-06T00:56:38.492 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:38.489+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.490929+0000) 2020-02-06T00:56:38.492 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:38.490+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.491539+0000) 2020-02-06T00:56:38.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:38.491+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.492947+0000) 2020-02-06T00:56:38.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:38.492+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.493960+0000) 2020-02-06T00:56:38.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:38.492+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.494409+0000) 2020-02-06T00:56:38.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:38.493+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.495382+0000) 2020-02-06T00:56:38.496 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:38.494+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.495456+0000) 2020-02-06T00:56:38.497 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:38.494+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.496173+0000) 2020-02-06T00:56:38.497 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:38.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.496561+0000) 2020-02-06T00:56:38.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:38.495+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.496910+0000) 2020-02-06T00:56:38.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:38.495+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.497244+0000) 2020-02-06T00:56:38.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:38.498+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.499769+0000) 2020-02-06T00:56:38.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:38.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.500102+0000) 2020-02-06T00:56:38.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:38.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:38.500303+0000) 2020-02-06T00:56:38.513 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:38.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188753 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:39.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:39.686+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:39.687587+0000) 2020-02-06T00:56:39.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:39.934+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:39.935670+0000) 2020-02-06T00:56:40.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:40.537+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:40.538301+0000) 2020-02-06T00:56:40.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:40.796+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:40.796984+0000) 2020-02-06T00:56:40.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:40.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:40.818379+0000) 2020-02-06T00:56:40.819 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:40.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:40.818612+0000) 2020-02-06T00:56:42.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:42.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:42.920761+0000) 2020-02-06T00:56:43.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:43.150+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.151750+0000) 2020-02-06T00:56:43.521 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:43.519+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.520459+0000) 2020-02-06T00:56:43.522 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:43.519+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.521306+0000) 2020-02-06T00:56:43.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:43.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.521420+0000) 2020-02-06T00:56:43.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:43.520+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.522134+0000) 2020-02-06T00:56:43.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:43.521+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.522449+0000) 2020-02-06T00:56:43.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:43.521+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.522900+0000) 2020-02-06T00:56:43.524 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:43.522+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.523708+0000) 2020-02-06T00:56:43.524 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:43.522+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.524001+0000) 2020-02-06T00:56:43.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:43.523+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.524475+0000) 2020-02-06T00:56:43.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:43.523+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.524901+0000) 2020-02-06T00:56:43.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:43.523+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.524675+0000) 2020-02-06T00:56:43.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:43.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.526386+0000) 2020-02-06T00:56:43.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:43.528+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.529574+0000) 2020-02-06T00:56:43.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:43.528+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:43.529848+0000) 2020-02-06T00:56:43.540 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:43.538+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:44.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:44.569+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:44.570507+0000) 2020-02-06T00:56:46.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:46.057+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:46.058203+0000) 2020-02-06T00:56:47.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:47.042+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:47.043539+0000) 2020-02-06T00:56:47.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:47.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:47.311236+0000) 2020-02-06T00:56:47.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:47.310+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:47.311239+0000) 2020-02-06T00:56:47.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:47.310+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:47.311484+0000) 2020-02-06T00:56:48.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:48.543+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.544454+0000) 2020-02-06T00:56:48.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:48.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.545139+0000) 2020-02-06T00:56:48.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:48.544+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.545572+0000) 2020-02-06T00:56:48.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:48.545+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.546563+0000) 2020-02-06T00:56:48.548 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:48.546+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.547211+0000) 2020-02-06T00:56:48.548 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:48.546+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.547514+0000) 2020-02-06T00:56:48.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:48.546+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.548202+0000) 2020-02-06T00:56:48.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:48.547+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.548319+0000) 2020-02-06T00:56:48.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:48.547+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.548979+0000) 2020-02-06T00:56:48.550 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:48.548+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.549329+0000) 2020-02-06T00:56:48.551 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:48.548+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.550107+0000) 2020-02-06T00:56:48.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:48.549+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.550591+0000) 2020-02-06T00:56:48.552 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:48.549+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.550945+0000) 2020-02-06T00:56:48.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:48.555+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:48.556981+0000) 2020-02-06T00:56:48.567 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:48.565+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:49.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:49.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:49.687843+0000) 2020-02-06T00:56:49.930 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:56:49.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:49.934+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:49.935860+0000) 2020-02-06T00:56:49.941 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:56:50.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:50.537+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:50.538535+0000) 2020-02-06T00:56:50.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:50.796+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:50.797290+0000) 2020-02-06T00:56:50.819 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:50.817+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:50.818652+0000) 2020-02-06T00:56:50.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:50.817+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:50.818849+0000) 2020-02-06T00:56:52.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:52.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:52.921033+0000) 2020-02-06T00:56:53.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:53.151+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.151982+0000) 2020-02-06T00:56:53.579 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:53.572+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.574077+0000) 2020-02-06T00:56:53.579 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:53.573+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.574713+0000) 2020-02-06T00:56:53.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:53.575+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.577138+0000) 2020-02-06T00:56:53.579 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:53.576+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.577812+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:53.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.572748+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:53.572+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.573376+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:53.573+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.574883+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:53.574+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.575711+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:53.575+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.576283+0000) 2020-02-06T00:56:53.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:53.576+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.577268+0000) 2020-02-06T00:56:53.581 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:53.576+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.577742+0000) 2020-02-06T00:56:53.581 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:53.576+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.578296+0000) 2020-02-06T00:56:53.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:53.577+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.578697+0000) 2020-02-06T00:56:53.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:53.581+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:53.582585+0000) 2020-02-06T00:56:53.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:53.591+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188877 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:54.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:54.581+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:54.582304+0000) 2020-02-06T00:56:56.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:56.057+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:56.058483+0000) 2020-02-06T00:56:57.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:57.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:57.043774+0000) 2020-02-06T00:56:57.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:57.310+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:57.311492+0000) 2020-02-06T00:56:57.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:57.310+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:57.311495+0000) 2020-02-06T00:56:57.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:57.310+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:57.311720+0000) 2020-02-06T00:56:58.599 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:56:58.597+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.598172+0000) 2020-02-06T00:56:58.600 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:56:58.598+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.599514+0000) 2020-02-06T00:56:58.601 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:56:58.598+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.600315+0000) 2020-02-06T00:56:58.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:58.600+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.601268+0000) 2020-02-06T00:56:58.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:58.600+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.601418+0000) 2020-02-06T00:56:58.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:56:58.600+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.601861+0000) 2020-02-06T00:56:58.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:56:58.601+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.602529+0000) 2020-02-06T00:56:58.605 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:56:58.601+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.603292+0000) 2020-02-06T00:56:58.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:56:58.602+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.603453+0000) 2020-02-06T00:56:58.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:56:58.603+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.604606+0000) 2020-02-06T00:56:58.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:56:58.603+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.604660+0000) 2020-02-06T00:56:58.606 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:56:58.602+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.603704+0000) 2020-02-06T00:56:58.606 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:56:58.603+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.604328+0000) 2020-02-06T00:56:58.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:58.608+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:58.609983+0000) 2020-02-06T00:56:58.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:56:58.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:56:59.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:59.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:59.688065+0000) 2020-02-06T00:56:59.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:56:59.935+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:56:59.936123+0000) 2020-02-06T00:57:00.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:00.538+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:00.538795+0000) 2020-02-06T00:57:00.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:00.796+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:00.797557+0000) 2020-02-06T00:57:00.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:00.818+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:00.818903+0000) 2020-02-06T00:57:00.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:00.818+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:00.819119+0000) 2020-02-06T00:57:02.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:02.920+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:02.921276+0000) 2020-02-06T00:57:03.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:03.151+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.152241+0000) 2020-02-06T00:57:03.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:03.622+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.623824+0000) 2020-02-06T00:57:03.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:03.624+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.625371+0000) 2020-02-06T00:57:03.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:03.624+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.625564+0000) 2020-02-06T00:57:03.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:03.624+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.625966+0000) 2020-02-06T00:57:03.627 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:03.625+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.626415+0000) 2020-02-06T00:57:03.627 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:03.625+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.626302+0000) 2020-02-06T00:57:03.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:03.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.627306+0000) 2020-02-06T00:57:03.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:03.626+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.627087+0000) 2020-02-06T00:57:03.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:03.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.627922+0000) 2020-02-06T00:57:03.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:03.628+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.628441+0000) 2020-02-06T00:57:03.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:03.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.628533+0000) 2020-02-06T00:57:03.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:03.628+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.628996+0000) 2020-02-06T00:57:03.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:03.629+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.629682+0000) 2020-02-06T00:57:03.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:03.631+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:03.632695+0000) 2020-02-06T00:57:03.648 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:03.647+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 188961 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:04.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:04.593+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:04.594234+0000) 2020-02-06T00:57:06.059 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:06.058+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:06.058723+0000) 2020-02-06T00:57:07.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:07.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:07.044007+0000) 2020-02-06T00:57:07.312 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:07.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:07.311721+0000) 2020-02-06T00:57:07.313 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:07.311+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:07.311809+0000) 2020-02-06T00:57:07.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:07.311+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:07.311968+0000) 2020-02-06T00:57:08.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:08.652+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.653139+0000) 2020-02-06T00:57:08.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:08.653+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.654268+0000) 2020-02-06T00:57:08.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:08.654+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.654831+0000) 2020-02-06T00:57:08.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:08.654+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.655350+0000) 2020-02-06T00:57:08.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:08.655+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.656013+0000) 2020-02-06T00:57:08.657 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:08.655+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.656393+0000) 2020-02-06T00:57:08.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:08.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.656791+0000) 2020-02-06T00:57:08.658 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:08.655+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.656974+0000) 2020-02-06T00:57:08.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:08.657+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.657572+0000) 2020-02-06T00:57:08.659 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:08.657+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.658133+0000) 2020-02-06T00:57:08.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:08.657+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.657732+0000) 2020-02-06T00:57:08.659 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:08.657+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.658158+0000) 2020-02-06T00:57:08.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:08.657+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.658573+0000) 2020-02-06T00:57:08.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:08.663+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:08.664632+0000) 2020-02-06T00:57:08.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:08.674+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:09.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:09.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:09.688386+0000) 2020-02-06T00:57:09.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:09.936+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:09.936354+0000) 2020-02-06T00:57:10.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:10.538+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:10.539033+0000) 2020-02-06T00:57:10.798 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:10.797+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:10.797808+0000) 2020-02-06T00:57:10.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:10.818+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:10.819138+0000) 2020-02-06T00:57:10.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:10.818+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:10.819369+0000) 2020-02-06T00:57:12.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:12.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:12.921514+0000) 2020-02-06T00:57:13.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:13.151+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.152460+0000) 2020-02-06T00:57:13.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:13.679+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.680388+0000) 2020-02-06T00:57:13.690 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:13.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.680850+0000) 2020-02-06T00:57:13.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:13.681+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.681916+0000) 2020-02-06T00:57:13.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:13.681+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.682533+0000) 2020-02-06T00:57:13.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:13.681+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.682925+0000) 2020-02-06T00:57:13.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:13.682+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.683779+0000) 2020-02-06T00:57:13.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:13.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.685427+0000) 2020-02-06T00:57:13.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:13.682+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.682651+0000) 2020-02-06T00:57:13.711 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:13.684+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.684408+0000) 2020-02-06T00:57:13.711 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:13.684+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.684732+0000) 2020-02-06T00:57:13.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:13.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.685592+0000) 2020-02-06T00:57:13.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:13.687+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.688455+0000) 2020-02-06T00:57:13.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:13.688+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.689680+0000) 2020-02-06T00:57:13.721 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:13.685+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:13.685689+0000) 2020-02-06T00:57:13.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:13.699+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:14.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:14.605+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:14.606410+0000) 2020-02-06T00:57:16.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:16.058+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:16.059003+0000) 2020-02-06T00:57:17.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:17.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:17.044230+0000) 2020-02-06T00:57:17.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:17.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:17.311962+0000) 2020-02-06T00:57:17.313 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:17.311+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:17.312070+0000) 2020-02-06T00:57:17.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:17.311+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:17.312184+0000) 2020-02-06T00:57:18.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:18.703+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.705011+0000) 2020-02-06T00:57:18.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:18.704+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.705834+0000) 2020-02-06T00:57:18.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:18.706+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.706722+0000) 2020-02-06T00:57:18.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:18.705+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.706771+0000) 2020-02-06T00:57:18.708 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:18.706+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.707586+0000) 2020-02-06T00:57:18.708 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:18.707+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.708065+0000) 2020-02-06T00:57:18.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:18.707+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.708132+0000) 2020-02-06T00:57:18.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:18.709+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.709495+0000) 2020-02-06T00:57:18.710 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:18.709+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.710081+0000) 2020-02-06T00:57:18.711 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:18.710+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.711095+0000) 2020-02-06T00:57:18.712 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:18.710+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.711371+0000) 2020-02-06T00:57:18.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:18.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.714325+0000) 2020-02-06T00:57:18.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:18.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.714544+0000) 2020-02-06T00:57:18.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:18.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:18.714726+0000) 2020-02-06T00:57:18.726 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:18.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:19.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:19.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:19.688620+0000) 2020-02-06T00:57:19.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:19.936+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:19.936592+0000) 2020-02-06T00:57:20.084 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:57:20.091 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:57:20.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:20.538+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:20.539270+0000) 2020-02-06T00:57:20.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:20.797+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:20.798037+0000) 2020-02-06T00:57:20.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:20.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:20.819367+0000) 2020-02-06T00:57:20.820 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:20.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:20.819612+0000) 2020-02-06T00:57:22.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:22.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:22.921773+0000) 2020-02-06T00:57:23.153 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:23.152+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.152684+0000) 2020-02-06T00:57:23.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:23.731+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.731220+0000) 2020-02-06T00:57:23.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:23.731+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.731656+0000) 2020-02-06T00:57:23.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:23.732+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.732614+0000) 2020-02-06T00:57:23.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:23.732+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.733346+0000) 2020-02-06T00:57:23.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:23.734+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.734180+0000) 2020-02-06T00:57:23.735 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:23.734+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.734778+0000) 2020-02-06T00:57:23.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:23.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.734605+0000) 2020-02-06T00:57:23.736 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:23.734+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.735244+0000) 2020-02-06T00:57:23.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:23.735+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.735442+0000) 2020-02-06T00:57:23.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:23.735+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.735897+0000) 2020-02-06T00:57:23.737 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:23.735+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.736204+0000) 2020-02-06T00:57:23.737 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:23.736+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.736742+0000) 2020-02-06T00:57:23.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:23.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.740509+0000) 2020-02-06T00:57:23.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:23.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:23.740805+0000) 2020-02-06T00:57:23.753 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:23.752+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:24.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:24.617+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:24.618390+0000) 2020-02-06T00:57:26.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:26.059+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:26.059251+0000) 2020-02-06T00:57:27.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:27.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:27.044506+0000) 2020-02-06T00:57:27.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:27.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:27.312189+0000) 2020-02-06T00:57:27.313 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:27.311+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:27.312305+0000) 2020-02-06T00:57:27.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:27.312+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:27.312499+0000) 2020-02-06T00:57:28.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:28.756+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.757664+0000) 2020-02-06T00:57:28.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:28.757+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.758447+0000) 2020-02-06T00:57:28.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:28.758+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.759227+0000) 2020-02-06T00:57:28.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:28.759+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.759866+0000) 2020-02-06T00:57:28.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:28.759+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.760209+0000) 2020-02-06T00:57:28.762 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:28.760+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.761389+0000) 2020-02-06T00:57:28.762 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:28.760+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.761693+0000) 2020-02-06T00:57:28.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:28.761+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.761548+0000) 2020-02-06T00:57:28.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:28.761+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.762328+0000) 2020-02-06T00:57:28.763 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:28.761+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.762280+0000) 2020-02-06T00:57:28.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:28.761+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.762772+0000) 2020-02-06T00:57:28.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:28.762+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.762996+0000) 2020-02-06T00:57:28.764 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:28.763+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.764348+0000) 2020-02-06T00:57:28.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:28.766+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:28.767218+0000) 2020-02-06T00:57:28.781 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:28.779+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189171 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:29.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:29.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:29.688826+0000) 2020-02-06T00:57:29.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:29.935+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:29.936840+0000) 2020-02-06T00:57:30.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:30.539+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:30.539547+0000) 2020-02-06T00:57:30.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:30.797+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:30.798313+0000) 2020-02-06T00:57:30.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:30.818+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:30.819601+0000) 2020-02-06T00:57:30.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:30.818+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:30.819804+0000) 2020-02-06T00:57:32.922 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:32.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:32.922030+0000) 2020-02-06T00:57:33.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:33.152+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.152904+0000) 2020-02-06T00:57:33.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:33.784+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.785348+0000) 2020-02-06T00:57:33.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:33.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.786320+0000) 2020-02-06T00:57:33.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:33.785+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.786807+0000) 2020-02-06T00:57:33.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:33.786+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.787409+0000) 2020-02-06T00:57:33.790 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:33.788+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.788913+0000) 2020-02-06T00:57:33.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:33.787+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.788944+0000) 2020-02-06T00:57:33.790 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:33.788+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.789492+0000) 2020-02-06T00:57:33.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:33.788+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.789720+0000) 2020-02-06T00:57:33.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:33.790+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.790563+0000) 2020-02-06T00:57:33.792 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:33.790+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.791015+0000) 2020-02-06T00:57:33.792 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:33.790+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.791413+0000) 2020-02-06T00:57:33.793 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:33.791+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.791455+0000) 2020-02-06T00:57:33.793 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:33.791+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.791765+0000) 2020-02-06T00:57:33.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:33.795+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:33.797066+0000) 2020-02-06T00:57:33.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:33.808+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:34.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:34.625+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:34.626785+0000) 2020-02-06T00:57:36.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:36.059+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:36.059574+0000) 2020-02-06T00:57:37.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:37.043+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:37.044713+0000) 2020-02-06T00:57:37.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:37.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:37.312484+0000) 2020-02-06T00:57:37.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:37.312+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:37.312639+0000) 2020-02-06T00:57:37.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:37.312+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:37.312754+0000) 2020-02-06T00:57:38.816 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:38.813+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.815044+0000) 2020-02-06T00:57:38.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:38.814+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.815410+0000) 2020-02-06T00:57:38.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:38.816+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.816719+0000) 2020-02-06T00:57:38.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:38.815+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.816991+0000) 2020-02-06T00:57:38.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:38.816+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.817341+0000) 2020-02-06T00:57:38.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:38.817+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.817718+0000) 2020-02-06T00:57:38.819 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:38.817+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.818982+0000) 2020-02-06T00:57:38.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:38.819+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.820197+0000) 2020-02-06T00:57:38.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:38.819+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.820327+0000) 2020-02-06T00:57:38.821 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:38.819+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.820344+0000) 2020-02-06T00:57:38.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:38.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.820678+0000) 2020-02-06T00:57:38.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:38.819+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.820958+0000) 2020-02-06T00:57:38.822 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:38.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.821360+0000) 2020-02-06T00:57:38.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:38.822+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:38.824029+0000) 2020-02-06T00:57:38.836 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:38.835+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:39.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:39.687+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:39.689023+0000) 2020-02-06T00:57:39.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:39.936+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:39.937108+0000) 2020-02-06T00:57:40.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:40.538+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:40.539734+0000) 2020-02-06T00:57:40.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:40.797+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:40.798534+0000) 2020-02-06T00:57:40.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:40.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:40.819829+0000) 2020-02-06T00:57:40.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:40.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:40.819986+0000) 2020-02-06T00:57:42.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:42.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:42.922262+0000) 2020-02-06T00:57:43.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:43.152+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.153133+0000) 2020-02-06T00:57:43.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:43.839+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.840803+0000) 2020-02-06T00:57:43.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:43.840+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.841158+0000) 2020-02-06T00:57:43.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:43.840+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.841590+0000) 2020-02-06T00:57:43.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:43.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.842530+0000) 2020-02-06T00:57:43.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:43.841+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.843120+0000) 2020-02-06T00:57:43.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:43.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.843567+0000) 2020-02-06T00:57:43.845 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:43.842+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.844368+0000) 2020-02-06T00:57:43.845 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:43.842+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.844412+0000) 2020-02-06T00:57:43.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:43.843+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.844805+0000) 2020-02-06T00:57:43.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:43.844+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.845302+0000) 2020-02-06T00:57:43.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:43.844+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.845597+0000) 2020-02-06T00:57:43.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:43.845+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.846062+0000) 2020-02-06T00:57:43.847 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:43.845+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.846955+0000) 2020-02-06T00:57:43.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:43.853+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:43.854663+0000) 2020-02-06T00:57:43.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:43.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:44.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:44.634+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:44.635061+0000) 2020-02-06T00:57:46.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:46.058+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:46.059773+0000) 2020-02-06T00:57:47.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:47.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:47.044946+0000) 2020-02-06T00:57:47.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:47.311+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:47.312748+0000) 2020-02-06T00:57:47.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:47.311+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:47.312910+0000) 2020-02-06T00:57:47.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:47.311+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:47.313004+0000) 2020-02-06T00:57:48.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:48.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.866126+0000) 2020-02-06T00:57:48.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:48.866+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.867180+0000) 2020-02-06T00:57:48.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:48.866+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.867415+0000) 2020-02-06T00:57:48.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:48.866+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.867936+0000) 2020-02-06T00:57:48.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:48.867+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.868291+0000) 2020-02-06T00:57:48.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:48.867+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.868655+0000) 2020-02-06T00:57:48.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:48.867+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.869225+0000) 2020-02-06T00:57:48.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:48.869+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.870410+0000) 2020-02-06T00:57:48.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:48.869+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.870933+0000) 2020-02-06T00:57:48.872 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:48.869+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.871305+0000) 2020-02-06T00:57:48.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:48.870+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.871616+0000) 2020-02-06T00:57:48.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:48.870+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.871954+0000) 2020-02-06T00:57:48.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:48.869+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.870652+0000) 2020-02-06T00:57:48.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:48.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:48.878242+0000) 2020-02-06T00:57:48.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:48.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:49.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:49.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:49.689292+0000) 2020-02-06T00:57:49.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:49.936+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:49.937341+0000) 2020-02-06T00:57:50.209 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:57:50.215 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:57:50.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:50.539+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:50.539993+0000) 2020-02-06T00:57:50.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:50.798+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:50.798772+0000) 2020-02-06T00:57:50.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:50.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:50.820065+0000) 2020-02-06T00:57:50.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:50.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:50.820178+0000) 2020-02-06T00:57:52.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:52.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:52.922500+0000) 2020-02-06T00:57:53.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:53.152+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.153354+0000) 2020-02-06T00:57:53.892 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:53.890+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.891806+0000) 2020-02-06T00:57:53.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:53.891+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.892459+0000) 2020-02-06T00:57:53.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:53.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.893496+0000) 2020-02-06T00:57:53.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:53.893+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.894338+0000) 2020-02-06T00:57:53.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:53.893+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.895276+0000) 2020-02-06T00:57:53.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:53.894+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.895927+0000) 2020-02-06T00:57:53.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:53.894+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.896328+0000) 2020-02-06T00:57:53.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:53.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.896716+0000) 2020-02-06T00:57:53.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:53.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.896957+0000) 2020-02-06T00:57:53.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:53.896+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.897324+0000) 2020-02-06T00:57:53.900 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:53.896+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.897578+0000) 2020-02-06T00:57:53.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:53.895+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.897103+0000) 2020-02-06T00:57:53.901 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:53.896+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.897831+0000) 2020-02-06T00:57:53.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:53.903+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:53.904679+0000) 2020-02-06T00:57:53.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:53.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:54.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:54.642+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:54.643361+0000) 2020-02-06T00:57:56.060 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:56.059+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:56.060032+0000) 2020-02-06T00:57:57.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:57.044+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:57.045207+0000) 2020-02-06T00:57:57.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:57.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:57.313004+0000) 2020-02-06T00:57:57.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:57.312+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:57.313127+0000) 2020-02-06T00:57:57.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:57.312+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:57.313251+0000) 2020-02-06T00:57:58.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:57:58.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.916964+0000) 2020-02-06T00:57:58.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:58.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.917704+0000) 2020-02-06T00:57:58.919 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:57:58.917+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.918707+0000) 2020-02-06T00:57:58.920 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:57:58.917+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.918501+0000) 2020-02-06T00:57:58.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:57:58.919+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.921028+0000) 2020-02-06T00:57:58.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:57:58.920+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.921314+0000) 2020-02-06T00:57:58.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:57:58.920+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.921922+0000) 2020-02-06T00:57:58.924 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:57:58.921+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.922586+0000) 2020-02-06T00:57:58.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:57:58.921+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.922977+0000) 2020-02-06T00:57:58.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:57:58.922+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.923582+0000) 2020-02-06T00:57:58.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:57:58.922+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.923732+0000) 2020-02-06T00:57:58.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:57:58.922+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.924209+0000) 2020-02-06T00:57:58.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:58.926+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.927322+0000) 2020-02-06T00:57:58.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:58.927+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:58.928658+0000) 2020-02-06T00:57:58.939 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:57:58.937+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189426 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:57:59.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:59.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:59.689509+0000) 2020-02-06T00:57:59.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:57:59.937+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:57:59.937573+0000) 2020-02-06T00:58:00.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:00.539+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:00.540255+0000) 2020-02-06T00:58:00.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:00.798+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:00.799006+0000) 2020-02-06T00:58:00.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:00.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:00.820350+0000) 2020-02-06T00:58:00.821 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:00.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:00.820369+0000) 2020-02-06T00:58:02.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:02.921+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:02.922700+0000) 2020-02-06T00:58:03.154 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:03.153+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.153598+0000) 2020-02-06T00:58:03.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:03.943+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.944527+0000) 2020-02-06T00:58:03.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:03.944+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.945803+0000) 2020-02-06T00:58:03.947 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:03.945+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.946598+0000) 2020-02-06T00:58:03.947 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:03.945+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.947014+0000) 2020-02-06T00:58:03.948 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:03.946+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.947856+0000) 2020-02-06T00:58:03.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:03.947+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.948405+0000) 2020-02-06T00:58:03.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:03.947+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.948699+0000) 2020-02-06T00:58:03.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:03.947+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.948901+0000) 2020-02-06T00:58:03.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:03.947+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.949157+0000) 2020-02-06T00:58:03.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:03.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.949570+0000) 2020-02-06T00:58:03.951 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:03.948+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.949917+0000) 2020-02-06T00:58:03.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:03.949+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.950963+0000) 2020-02-06T00:58:03.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:03.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.957308+0000) 2020-02-06T00:58:03.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:03.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:03.957630+0000) 2020-02-06T00:58:03.967 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:03.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:04.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:04.651+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:04.651374+0000) 2020-02-06T00:58:06.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:06.059+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:06.060302+0000) 2020-02-06T00:58:07.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:07.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:07.045473+0000) 2020-02-06T00:58:07.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:07.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:07.313270+0000) 2020-02-06T00:58:07.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:07.312+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:07.313529+0000) 2020-02-06T00:58:07.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:07.312+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:07.313558+0000) 2020-02-06T00:58:08.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:08.970+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.971319+0000) 2020-02-06T00:58:08.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:08.970+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.971723+0000) 2020-02-06T00:58:08.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:08.971+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.972950+0000) 2020-02-06T00:58:08.974 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:08.971+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.972899+0000) 2020-02-06T00:58:08.974 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:08.972+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.973547+0000) 2020-02-06T00:58:08.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:08.972+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.973924+0000) 2020-02-06T00:58:08.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:08.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.974140+0000) 2020-02-06T00:58:08.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:08.973+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.975372+0000) 2020-02-06T00:58:08.976 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:08.974+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.975705+0000) 2020-02-06T00:58:08.977 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:08.975+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.976625+0000) 2020-02-06T00:58:08.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:08.975+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.976561+0000) 2020-02-06T00:58:08.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:08.975+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.977237+0000) 2020-02-06T00:58:08.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:08.975+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.977223+0000) 2020-02-06T00:58:08.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:08.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:08.981707+0000) 2020-02-06T00:58:08.991 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:08.989+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:09.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:09.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:09.689739+0000) 2020-02-06T00:58:09.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:09.937+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:09.937820+0000) 2020-02-06T00:58:10.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:10.539+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:10.540558+0000) 2020-02-06T00:58:10.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:10.799+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:10.799289+0000) 2020-02-06T00:58:10.821 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:10.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:10.820594+0000) 2020-02-06T00:58:10.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:10.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:10.820591+0000) 2020-02-06T00:58:13.141 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:12.922+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:12.922959+0000) 2020-02-06T00:58:13.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:13.153+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.153845+0000) 2020-02-06T00:58:13.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:13.994+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.995959+0000) 2020-02-06T00:58:13.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:13.995+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.996886+0000) 2020-02-06T00:58:13.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:13.996+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.997240+0000) 2020-02-06T00:58:13.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:13.997+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.998131+0000) 2020-02-06T00:58:13.999 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:13.997+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.998876+0000) 2020-02-06T00:58:14.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:13.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:13.999390+0000) 2020-02-06T00:58:14.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:13.999+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.000704+0000) 2020-02-06T00:58:14.002 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:13.999+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.001100+0000) 2020-02-06T00:58:14.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:14.000+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.002142+0000) 2020-02-06T00:58:14.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:14.000+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.002378+0000) 2020-02-06T00:58:14.003 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:14.001+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.002221+0000) 2020-02-06T00:58:14.003 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:14.001+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.002768+0000) 2020-02-06T00:58:14.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:14.005+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.007018+0000) 2020-02-06T00:58:14.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:14.006+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.007355+0000) 2020-02-06T00:58:14.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:14.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:14.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:14.659+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:14.659579+0000) 2020-02-06T00:58:16.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:16.059+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:16.060561+0000) 2020-02-06T00:58:17.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:17.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:17.045703+0000) 2020-02-06T00:58:17.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:17.312+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:17.313522+0000) 2020-02-06T00:58:17.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:17.312+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:17.313745+0000) 2020-02-06T00:58:17.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:17.312+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:17.313843+0000) 2020-02-06T00:58:19.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:19.020+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.021730+0000) 2020-02-06T00:58:19.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:19.021+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.022257+0000) 2020-02-06T00:58:19.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:19.021+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.022883+0000) 2020-02-06T00:58:19.024 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:19.022+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.023287+0000) 2020-02-06T00:58:19.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:19.022+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.023898+0000) 2020-02-06T00:58:19.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:19.023+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.024517+0000) 2020-02-06T00:58:19.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:19.022+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.024335+0000) 2020-02-06T00:58:19.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:19.024+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.025870+0000) 2020-02-06T00:58:19.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:19.025+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.026462+0000) 2020-02-06T00:58:19.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:19.025+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.026944+0000) 2020-02-06T00:58:19.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:19.025+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.027376+0000) 2020-02-06T00:58:19.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:19.026+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.027768+0000) 2020-02-06T00:58:19.029 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:19.026+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.028210+0000) 2020-02-06T00:58:19.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:19.029+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.030924+0000) 2020-02-06T00:58:19.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:19.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:19.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:19.688+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.689970+0000) 2020-02-06T00:58:19.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:19.936+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:19.938017+0000) 2020-02-06T00:58:20.355 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:58:20.362 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:58:20.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:20.540+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:20.540906+0000) 2020-02-06T00:58:20.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:20.798+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:20.799542+0000) 2020-02-06T00:58:20.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:20.819+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:20.820845+0000) 2020-02-06T00:58:20.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:20.819+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:20.820841+0000) 2020-02-06T00:58:22.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:22.922+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:22.923210+0000) 2020-02-06T00:58:23.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:23.153+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:23.154115+0000) 2020-02-06T00:58:24.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:24.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.051163+0000) 2020-02-06T00:58:24.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:24.050+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.051946+0000) 2020-02-06T00:58:24.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:24.051+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.052284+0000) 2020-02-06T00:58:24.054 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:24.052+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.053492+0000) 2020-02-06T00:58:24.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:24.052+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.053644+0000) 2020-02-06T00:58:24.055 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:24.053+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.054354+0000) 2020-02-06T00:58:24.055 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:24.053+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.054912+0000) 2020-02-06T00:58:24.056 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:24.054+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.055664+0000) 2020-02-06T00:58:24.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:24.054+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.055847+0000) 2020-02-06T00:58:24.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:24.055+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.056337+0000) 2020-02-06T00:58:24.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:24.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.057107+0000) 2020-02-06T00:58:24.058 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:24.054+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.056193+0000) 2020-02-06T00:58:24.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:24.056+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.058268+0000) 2020-02-06T00:58:24.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:24.061+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.063058+0000) 2020-02-06T00:58:24.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:24.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:24.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:24.666+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:24.667556+0000) 2020-02-06T00:58:26.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:26.060+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:26.060846+0000) 2020-02-06T00:58:27.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:27.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:27.045934+0000) 2020-02-06T00:58:27.314 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:27.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:27.313758+0000) 2020-02-06T00:58:27.315 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:27.313+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:27.313963+0000) 2020-02-06T00:58:27.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:27.313+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:27.314103+0000) 2020-02-06T00:58:29.090 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:29.078+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.079656+0000) 2020-02-06T00:58:29.090 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:29.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.080457+0000) 2020-02-06T00:58:29.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:29.080+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.081317+0000) 2020-02-06T00:58:29.091 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:29.081+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.081933+0000) 2020-02-06T00:58:29.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:29.081+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.082062+0000) 2020-02-06T00:58:29.092 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:29.082+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.082493+0000) 2020-02-06T00:58:29.093 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:29.081+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.082652+0000) 2020-02-06T00:58:29.093 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:29.082+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.083236+0000) 2020-02-06T00:58:29.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:29.082+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.083636+0000) 2020-02-06T00:58:29.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:29.083+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.083780+0000) 2020-02-06T00:58:29.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:29.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.089420+0000) 2020-02-06T00:58:29.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:29.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.089659+0000) 2020-02-06T00:58:29.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:29.084+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.085122+0000) 2020-02-06T00:58:29.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:29.084+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.085290+0000) 2020-02-06T00:58:29.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:29.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:29.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:29.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.690175+0000) 2020-02-06T00:58:29.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:29.937+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:29.938260+0000) 2020-02-06T00:58:30.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:30.540+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:30.541160+0000) 2020-02-06T00:58:30.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:30.798+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:30.799776+0000) 2020-02-06T00:58:30.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:30.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:30.821145+0000) 2020-02-06T00:58:30.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:30.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:30.821151+0000) 2020-02-06T00:58:32.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:32.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:32.923492+0000) 2020-02-06T00:58:33.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:33.153+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:33.154382+0000) 2020-02-06T00:58:34.109 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:34.107+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.107565+0000) 2020-02-06T00:58:34.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:34.108+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.108895+0000) 2020-02-06T00:58:34.110 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:34.107+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.109018+0000) 2020-02-06T00:58:34.111 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:34.109+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.110076+0000) 2020-02-06T00:58:34.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:34.109+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.110697+0000) 2020-02-06T00:58:34.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:34.110+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.111129+0000) 2020-02-06T00:58:34.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:34.110+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.111697+0000) 2020-02-06T00:58:34.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:34.112+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.112453+0000) 2020-02-06T00:58:34.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:34.112+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.112954+0000) 2020-02-06T00:58:34.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:34.110+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.111804+0000) 2020-02-06T00:58:34.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:34.111+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.112320+0000) 2020-02-06T00:58:34.115 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:34.112+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.113163+0000) 2020-02-06T00:58:34.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:34.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.113688+0000) 2020-02-06T00:58:34.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:34.118+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.119739+0000) 2020-02-06T00:58:34.128 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:34.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189722 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:34.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:34.674+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:34.675529+0000) 2020-02-06T00:58:36.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:36.060+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:36.061090+0000) 2020-02-06T00:58:37.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:37.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:37.046166+0000) 2020-02-06T00:58:37.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:37.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:37.314014+0000) 2020-02-06T00:58:37.315 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:37.313+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:37.314174+0000) 2020-02-06T00:58:37.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:37.313+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:37.314337+0000) 2020-02-06T00:58:39.135 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:39.134+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.134707+0000) 2020-02-06T00:58:39.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:39.133+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.135019+0000) 2020-02-06T00:58:39.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:39.134+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.134475+0000) 2020-02-06T00:58:39.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:39.135+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.135882+0000) 2020-02-06T00:58:39.137 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:39.135+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.136196+0000) 2020-02-06T00:58:39.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:39.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.136771+0000) 2020-02-06T00:58:39.138 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:39.136+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.137293+0000) 2020-02-06T00:58:39.138 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:39.136+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.137394+0000) 2020-02-06T00:58:39.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:39.136+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.137887+0000) 2020-02-06T00:58:39.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:39.137+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.138314+0000) 2020-02-06T00:58:39.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:39.137+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.138920+0000) 2020-02-06T00:58:39.140 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:39.138+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.139265+0000) 2020-02-06T00:58:39.140 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:39.139+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.139929+0000) 2020-02-06T00:58:39.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:39.144+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.145275+0000) 2020-02-06T00:58:39.154 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:39.153+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:39.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:39.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.690370+0000) 2020-02-06T00:58:39.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:39.937+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:39.938464+0000) 2020-02-06T00:58:40.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:40.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:40.541495+0000) 2020-02-06T00:58:40.801 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:40.799+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:40.800009+0000) 2020-02-06T00:58:40.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:40.820+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:40.821391+0000) 2020-02-06T00:58:40.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:40.820+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:40.821413+0000) 2020-02-06T00:58:42.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:42.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:42.923748+0000) 2020-02-06T00:58:43.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:43.154+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:43.154621+0000) 2020-02-06T00:58:44.160 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:44.157+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.158951+0000) 2020-02-06T00:58:44.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:44.161+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.161878+0000) 2020-02-06T00:58:44.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:44.161+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.162370+0000) 2020-02-06T00:58:44.164 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:44.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.162016+0000) 2020-02-06T00:58:44.164 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:44.161+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.162632+0000) 2020-02-06T00:58:44.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:44.163+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.163544+0000) 2020-02-06T00:58:44.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:44.162+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.163307+0000) 2020-02-06T00:58:44.165 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:44.164+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.165122+0000) 2020-02-06T00:58:44.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:44.163+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.164940+0000) 2020-02-06T00:58:44.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:44.164+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.164940+0000) 2020-02-06T00:58:44.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:44.164+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.165661+0000) 2020-02-06T00:58:44.167 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:44.164+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.165670+0000) 2020-02-06T00:58:44.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:44.165+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.166100+0000) 2020-02-06T00:58:44.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:44.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.170743+0000) 2020-02-06T00:58:44.179 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:44.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189807 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:44.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:44.684+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:44.684595+0000) 2020-02-06T00:58:46.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:46.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:46.061328+0000) 2020-02-06T00:58:47.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:47.045+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:47.046391+0000) 2020-02-06T00:58:47.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:47.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:47.314263+0000) 2020-02-06T00:58:47.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:47.314+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:47.314413+0000) 2020-02-06T00:58:47.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:47.314+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:47.314628+0000) 2020-02-06T00:58:49.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:49.182+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.183893+0000) 2020-02-06T00:58:49.185 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:49.184+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.184625+0000) 2020-02-06T00:58:49.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:49.184+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.185199+0000) 2020-02-06T00:58:49.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:49.184+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.185858+0000) 2020-02-06T00:58:49.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:49.185+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.186121+0000) 2020-02-06T00:58:49.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:49.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.186890+0000) 2020-02-06T00:58:49.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:49.186+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.187073+0000) 2020-02-06T00:58:49.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:49.186+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.186581+0000) 2020-02-06T00:58:49.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:49.187+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.187958+0000) 2020-02-06T00:58:49.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:49.187+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.188405+0000) 2020-02-06T00:58:49.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:49.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.189053+0000) 2020-02-06T00:58:49.190 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:49.188+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.189244+0000) 2020-02-06T00:58:49.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:49.189+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.189551+0000) 2020-02-06T00:58:49.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:49.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.193027+0000) 2020-02-06T00:58:49.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:49.204+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189851 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:49.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:49.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.690578+0000) 2020-02-06T00:58:49.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:49.937+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:49.938625+0000) 2020-02-06T00:58:50.512 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:58:50.518 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:58:50.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:50.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:50.541699+0000) 2020-02-06T00:58:50.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:50.799+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:50.800282+0000) 2020-02-06T00:58:50.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:50.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:50.821625+0000) 2020-02-06T00:58:50.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:50.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:50.821684+0000) 2020-02-06T00:58:52.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:52.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:52.924015+0000) 2020-02-06T00:58:53.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:53.153+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:53.154874+0000) 2020-02-06T00:58:54.212 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:54.211+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.211411+0000) 2020-02-06T00:58:54.213 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:54.211+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.212365+0000) 2020-02-06T00:58:54.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:54.212+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.212954+0000) 2020-02-06T00:58:54.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:54.213+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.214322+0000) 2020-02-06T00:58:54.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:54.213+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.214411+0000) 2020-02-06T00:58:54.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:54.213+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.214909+0000) 2020-02-06T00:58:54.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:54.214+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.215568+0000) 2020-02-06T00:58:54.216 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:54.214+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.215651+0000) 2020-02-06T00:58:54.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:54.215+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.216399+0000) 2020-02-06T00:58:54.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:54.216+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.216870+0000) 2020-02-06T00:58:54.217 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:54.216+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.216799+0000) 2020-02-06T00:58:54.218 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:54.215+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.217033+0000) 2020-02-06T00:58:54.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:54.216+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.217354+0000) 2020-02-06T00:58:54.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:54.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.222003+0000) 2020-02-06T00:58:54.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:54.230+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189892 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:54.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:54.691+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:54.692682+0000) 2020-02-06T00:58:56.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:56.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:56.061591+0000) 2020-02-06T00:58:57.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:57.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:57.046627+0000) 2020-02-06T00:58:57.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:57.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:57.314540+0000) 2020-02-06T00:58:57.315 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:57.314+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:57.314745+0000) 2020-02-06T00:58:57.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:57.314+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:57.314883+0000) 2020-02-06T00:58:59.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:58:59.236+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.236560+0000) 2020-02-06T00:58:59.238 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:58:59.236+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.237055+0000) 2020-02-06T00:58:59.238 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:58:59.236+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.237850+0000) 2020-02-06T00:58:59.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:58:59.237+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.238533+0000) 2020-02-06T00:58:59.239 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:58:59.237+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.238844+0000) 2020-02-06T00:58:59.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:58:59.238+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.239284+0000) 2020-02-06T00:58:59.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:58:59.239+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.239651+0000) 2020-02-06T00:58:59.240 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:58:59.238+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.239312+0000) 2020-02-06T00:58:59.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:58:59.239+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.240070+0000) 2020-02-06T00:58:59.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:58:59.240+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.240516+0000) 2020-02-06T00:58:59.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:59.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.240209+0000) 2020-02-06T00:58:59.242 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:59.239+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.240438+0000) 2020-02-06T00:58:59.242 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:58:59.239+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.241039+0000) 2020-02-06T00:58:59.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:59.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.248829+0000) 2020-02-06T00:58:59.259 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:58:59.257+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189934 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:58:59.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:59.689+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.690792+0000) 2020-02-06T00:58:59.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:58:59.938+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:58:59.938844+0000) 2020-02-06T00:59:00.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:00.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:00.541965+0000) 2020-02-06T00:59:00.801 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:00.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:00.800530+0000) 2020-02-06T00:59:00.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:00.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:00.821879+0000) 2020-02-06T00:59:00.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:00.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:00.821959+0000) 2020-02-06T00:59:02.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:02.924+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:02.924280+0000) 2020-02-06T00:59:03.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:03.154+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:03.155137+0000) 2020-02-06T00:59:04.265 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:04.263+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.264065+0000) 2020-02-06T00:59:04.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:04.264+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.264937+0000) 2020-02-06T00:59:04.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:04.264+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.265912+0000) 2020-02-06T00:59:04.267 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:04.265+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.266732+0000) 2020-02-06T00:59:04.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:04.266+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.267522+0000) 2020-02-06T00:59:04.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:04.266+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.267335+0000) 2020-02-06T00:59:04.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:04.267+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.268177+0000) 2020-02-06T00:59:04.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:04.267+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.268691+0000) 2020-02-06T00:59:04.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:04.269+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.269308+0000) 2020-02-06T00:59:04.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:04.269+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.269977+0000) 2020-02-06T00:59:04.271 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:04.268+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.269895+0000) 2020-02-06T00:59:04.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:04.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.270272+0000) 2020-02-06T00:59:04.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:04.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.274119+0000) 2020-02-06T00:59:04.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:04.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.274380+0000) 2020-02-06T00:59:04.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:04.284+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 189975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:04.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:04.701+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:04.701363+0000) 2020-02-06T00:59:06.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:06.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:06.061896+0000) 2020-02-06T00:59:07.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:07.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:07.046881+0000) 2020-02-06T00:59:07.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:07.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:07.314768+0000) 2020-02-06T00:59:07.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:07.313+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:07.315174+0000) 2020-02-06T00:59:07.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:07.313+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:07.315023+0000) 2020-02-06T00:59:09.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:09.289+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.289297+0000) 2020-02-06T00:59:09.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:09.290+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.290975+0000) 2020-02-06T00:59:09.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:09.292+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.292333+0000) 2020-02-06T00:59:09.293 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:09.290+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.292310+0000) 2020-02-06T00:59:09.294 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:09.292+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.294132+0000) 2020-02-06T00:59:09.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:09.294+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.294217+0000) 2020-02-06T00:59:09.295 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:09.294+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.294928+0000) 2020-02-06T00:59:09.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:09.294+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.295416+0000) 2020-02-06T00:59:09.296 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:09.294+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.295638+0000) 2020-02-06T00:59:09.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:09.294+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.295789+0000) 2020-02-06T00:59:09.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:09.295+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.295939+0000) 2020-02-06T00:59:09.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:09.295+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.296542+0000) 2020-02-06T00:59:09.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:09.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.297278+0000) 2020-02-06T00:59:09.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:09.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.299848+0000) 2020-02-06T00:59:09.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:09.310+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:09.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:09.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.690996+0000) 2020-02-06T00:59:09.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:09.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:09.939112+0000) 2020-02-06T00:59:10.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:10.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:10.542211+0000) 2020-02-06T00:59:10.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:10.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:10.800779+0000) 2020-02-06T00:59:10.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:10.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:10.822158+0000) 2020-02-06T00:59:10.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:10.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:10.822182+0000) 2020-02-06T00:59:12.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:12.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:12.924541+0000) 2020-02-06T00:59:13.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:13.155+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:13.155368+0000) 2020-02-06T00:59:14.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:14.317+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.318805+0000) 2020-02-06T00:59:14.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:14.317+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.318913+0000) 2020-02-06T00:59:14.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:14.318+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.319863+0000) 2020-02-06T00:59:14.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:14.318+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.319937+0000) 2020-02-06T00:59:14.321 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:14.319+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.320394+0000) 2020-02-06T00:59:14.322 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:14.319+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.320503+0000) 2020-02-06T00:59:14.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:14.318+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.320397+0000) 2020-02-06T00:59:14.323 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:14.319+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.320846+0000) 2020-02-06T00:59:14.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:14.320+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.321063+0000) 2020-02-06T00:59:14.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:14.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.321420+0000) 2020-02-06T00:59:14.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:14.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.321641+0000) 2020-02-06T00:59:14.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:14.320+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.321743+0000) 2020-02-06T00:59:14.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:14.320+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.321829+0000) 2020-02-06T00:59:14.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:14.325+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.326923+0000) 2020-02-06T00:59:14.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:14.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:14.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:14.708+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:14.709215+0000) 2020-02-06T00:59:16.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:16.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:16.062140+0000) 2020-02-06T00:59:17.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:17.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:17.047150+0000) 2020-02-06T00:59:17.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:17.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:17.315008+0000) 2020-02-06T00:59:17.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:17.314+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:17.315302+0000) 2020-02-06T00:59:17.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:17.314+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:17.315423+0000) 2020-02-06T00:59:19.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:19.342+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.343683+0000) 2020-02-06T00:59:19.345 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:19.343+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.344350+0000) 2020-02-06T00:59:19.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:19.344+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.345709+0000) 2020-02-06T00:59:19.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:19.345+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.346140+0000) 2020-02-06T00:59:19.347 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:19.345+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.346374+0000) 2020-02-06T00:59:19.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:19.345+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.346778+0000) 2020-02-06T00:59:19.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:19.346+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.347569+0000) 2020-02-06T00:59:19.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:19.346+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.347810+0000) 2020-02-06T00:59:19.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:19.347+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.348304+0000) 2020-02-06T00:59:19.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:19.347+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.348692+0000) 2020-02-06T00:59:19.351 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:19.347+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.348980+0000) 2020-02-06T00:59:19.351 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:19.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.349388+0000) 2020-02-06T00:59:19.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:19.348+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.349717+0000) 2020-02-06T00:59:19.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:19.353+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.354496+0000) 2020-02-06T00:59:19.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:19.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:19.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:19.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.691213+0000) 2020-02-06T00:59:19.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:19.938+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:19.939319+0000) 2020-02-06T00:59:20.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:20.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:20.542472+0000) 2020-02-06T00:59:20.660 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:59:20.666 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:59:20.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:20.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:20.801123+0000) 2020-02-06T00:59:20.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:20.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:20.822404+0000) 2020-02-06T00:59:20.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:20.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:20.822418+0000) 2020-02-06T00:59:22.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:22.923+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:22.924809+0000) 2020-02-06T00:59:23.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:23.154+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:23.155614+0000) 2020-02-06T00:59:24.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:24.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.369861+0000) 2020-02-06T00:59:24.371 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:24.369+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.370403+0000) 2020-02-06T00:59:24.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:24.370+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.371234+0000) 2020-02-06T00:59:24.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:24.369+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.371225+0000) 2020-02-06T00:59:24.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:24.371+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.372146+0000) 2020-02-06T00:59:24.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:24.370+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.372057+0000) 2020-02-06T00:59:24.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:24.372+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.373658+0000) 2020-02-06T00:59:24.374 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:24.372+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.373713+0000) 2020-02-06T00:59:24.375 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:24.373+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.374365+0000) 2020-02-06T00:59:24.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:24.373+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.375358+0000) 2020-02-06T00:59:24.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:24.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.375187+0000) 2020-02-06T00:59:24.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:24.374+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.376329+0000) 2020-02-06T00:59:24.377 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:24.375+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.376471+0000) 2020-02-06T00:59:24.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:24.376+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.377628+0000) 2020-02-06T00:59:24.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:24.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:24.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:24.716+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:24.717206+0000) 2020-02-06T00:59:26.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:26.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:26.062381+0000) 2020-02-06T00:59:27.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:27.046+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:27.047427+0000) 2020-02-06T00:59:27.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:27.313+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:27.315263+0000) 2020-02-06T00:59:27.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:27.314+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:27.315540+0000) 2020-02-06T00:59:27.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:27.314+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:27.315748+0000) 2020-02-06T00:59:29.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:29.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.396119+0000) 2020-02-06T00:59:29.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:29.395+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.396755+0000) 2020-02-06T00:59:29.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:29.396+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.397791+0000) 2020-02-06T00:59:29.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:29.398+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.399493+0000) 2020-02-06T00:59:29.400 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:29.398+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.399676+0000) 2020-02-06T00:59:29.401 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:29.399+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.400206+0000) 2020-02-06T00:59:29.401 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:29.399+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.400899+0000) 2020-02-06T00:59:29.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:29.399+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.400876+0000) 2020-02-06T00:59:29.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:29.400+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.401558+0000) 2020-02-06T00:59:29.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:29.401+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.402485+0000) 2020-02-06T00:59:29.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:29.401+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.402766+0000) 2020-02-06T00:59:29.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:29.401+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.403010+0000) 2020-02-06T00:59:29.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:29.401+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.403149+0000) 2020-02-06T00:59:29.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:29.406+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.408066+0000) 2020-02-06T00:59:29.421 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:29.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:29.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:29.690+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.691429+0000) 2020-02-06T00:59:29.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:29.938+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:29.939558+0000) 2020-02-06T00:59:30.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:30.541+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:30.542720+0000) 2020-02-06T00:59:30.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:30.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:30.801412+0000) 2020-02-06T00:59:30.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:30.821+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:30.822650+0000) 2020-02-06T00:59:30.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:30.821+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:30.822704+0000) 2020-02-06T00:59:32.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:32.924+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:32.925071+0000) 2020-02-06T00:59:33.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:33.155+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:33.155843+0000) 2020-02-06T00:59:34.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:34.422+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.424046+0000) 2020-02-06T00:59:34.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:34.423+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.425181+0000) 2020-02-06T00:59:34.426 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:34.424+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.425387+0000) 2020-02-06T00:59:34.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:34.425+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.426696+0000) 2020-02-06T00:59:34.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:34.426+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.427545+0000) 2020-02-06T00:59:34.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:34.426+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.427841+0000) 2020-02-06T00:59:34.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:34.427+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.429142+0000) 2020-02-06T00:59:34.430 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:34.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.429394+0000) 2020-02-06T00:59:34.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:34.428+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.429488+0000) 2020-02-06T00:59:34.431 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:34.428+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.429819+0000) 2020-02-06T00:59:34.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:34.429+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.430285+0000) 2020-02-06T00:59:34.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:34.429+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.430503+0000) 2020-02-06T00:59:34.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:34.429+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.430680+0000) 2020-02-06T00:59:34.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:34.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.432897+0000) 2020-02-06T00:59:34.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:34.444+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190226 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:34.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:34.724+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:34.725351+0000) 2020-02-06T00:59:36.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:36.061+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:36.062651+0000) 2020-02-06T00:59:37.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:37.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:37.047713+0000) 2020-02-06T00:59:37.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:37.314+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:37.315527+0000) 2020-02-06T00:59:37.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:37.314+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:37.315824+0000) 2020-02-06T00:59:37.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:37.314+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:37.315974+0000) 2020-02-06T00:59:39.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:39.451+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.452527+0000) 2020-02-06T00:59:39.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:39.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.453033+0000) 2020-02-06T00:59:39.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:39.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.453467+0000) 2020-02-06T00:59:39.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:39.452+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.453918+0000) 2020-02-06T00:59:39.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:39.452+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.454042+0000) 2020-02-06T00:59:39.455 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:39.453+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.454539+0000) 2020-02-06T00:59:39.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:39.452+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.454000+0000) 2020-02-06T00:59:39.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:39.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.454471+0000) 2020-02-06T00:59:39.456 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:39.453+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.454965+0000) 2020-02-06T00:59:39.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:39.453+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.455301+0000) 2020-02-06T00:59:39.457 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:39.454+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.455189+0000) 2020-02-06T00:59:39.457 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:39.454+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.455844+0000) 2020-02-06T00:59:39.457 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:39.455+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.456263+0000) 2020-02-06T00:59:39.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:39.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.461694+0000) 2020-02-06T00:59:39.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:39.471+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190268 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:39.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:39.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.691640+0000) 2020-02-06T00:59:39.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:39.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:39.939763+0000) 2020-02-06T00:59:40.543 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:40.542+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:40.543006+0000) 2020-02-06T00:59:40.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:40.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:40.801641+0000) 2020-02-06T00:59:40.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:40.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:40.822970+0000) 2020-02-06T00:59:40.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:40.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:40.822974+0000) 2020-02-06T00:59:42.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:42.924+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:42.925304+0000) 2020-02-06T00:59:43.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:43.155+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:43.156070+0000) 2020-02-06T00:59:44.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:44.477+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.478274+0000) 2020-02-06T00:59:44.479 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:44.477+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.478736+0000) 2020-02-06T00:59:44.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:44.478+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.479733+0000) 2020-02-06T00:59:44.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:44.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.479834+0000) 2020-02-06T00:59:44.481 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:44.479+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.480742+0000) 2020-02-06T00:59:44.481 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:44.479+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.480823+0000) 2020-02-06T00:59:44.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:44.481+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.482161+0000) 2020-02-06T00:59:44.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:44.480+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.481850+0000) 2020-02-06T00:59:44.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:44.480+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.481851+0000) 2020-02-06T00:59:44.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:44.481+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.482533+0000) 2020-02-06T00:59:44.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:44.481+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.482710+0000) 2020-02-06T00:59:44.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:44.482+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.483050+0000) 2020-02-06T00:59:44.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:44.482+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.483161+0000) 2020-02-06T00:59:44.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:44.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.488985+0000) 2020-02-06T00:59:44.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:44.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:44.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:44.733+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:44.733705+0000) 2020-02-06T00:59:46.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:46.062+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:46.062899+0000) 2020-02-06T00:59:47.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:47.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:47.047971+0000) 2020-02-06T00:59:47.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:47.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:47.315786+0000) 2020-02-06T00:59:47.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:47.315+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:47.316231+0000) 2020-02-06T00:59:47.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:47.315+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:47.316118+0000) 2020-02-06T00:59:49.513 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:49.502+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.503259+0000) 2020-02-06T00:59:49.514 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:49.502+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.503854+0000) 2020-02-06T00:59:49.515 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:49.503+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.504525+0000) 2020-02-06T00:59:49.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:49.504+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.505298+0000) 2020-02-06T00:59:49.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:49.504+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.506399+0000) 2020-02-06T00:59:49.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:49.506+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.507246+0000) 2020-02-06T00:59:49.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:49.506+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.507525+0000) 2020-02-06T00:59:49.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:49.507+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.508229+0000) 2020-02-06T00:59:49.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:49.507+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.509038+0000) 2020-02-06T00:59:49.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:49.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.509699+0000) 2020-02-06T00:59:49.521 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:49.508+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.510312+0000) 2020-02-06T00:59:49.524 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:49.509+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.511111+0000) 2020-02-06T00:59:49.525 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:49.509+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.511284+0000) 2020-02-06T00:59:49.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:49.512+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.513788+0000) 2020-02-06T00:59:49.538 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:49.526+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:49.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:49.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.691806+0000) 2020-02-06T00:59:49.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:49.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:49.939975+0000) 2020-02-06T00:59:50.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:50.542+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:50.543238+0000) 2020-02-06T00:59:50.795 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:59:50.801 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T00:59:50.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:50.800+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:50.801933+0000) 2020-02-06T00:59:50.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:50.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:50.823237+0000) 2020-02-06T00:59:50.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:50.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:50.823191+0000) 2020-02-06T00:59:52.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:52.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:52.925641+0000) 2020-02-06T00:59:53.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:53.156+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:53.156357+0000) 2020-02-06T00:59:54.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:54.531+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.533036+0000) 2020-02-06T00:59:54.534 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:54.531+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.533043+0000) 2020-02-06T00:59:54.535 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:54.532+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.533731+0000) 2020-02-06T00:59:54.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:54.533+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.534331+0000) 2020-02-06T00:59:54.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:54.533+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.535048+0000) 2020-02-06T00:59:54.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:54.534+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.535708+0000) 2020-02-06T00:59:54.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:54.535+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.537000+0000) 2020-02-06T00:59:54.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:54.535+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.537306+0000) 2020-02-06T00:59:54.538 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:54.536+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.537532+0000) 2020-02-06T00:59:54.539 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:54.537+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.538599+0000) 2020-02-06T00:59:54.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:54.536+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.538054+0000) 2020-02-06T00:59:54.540 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:54.537+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.538508+0000) 2020-02-06T00:59:54.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:54.537+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.539150+0000) 2020-02-06T00:59:54.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:54.541+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.543024+0000) 2020-02-06T00:59:54.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:54.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190392 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:54.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:54.740+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:54.741596+0000) 2020-02-06T00:59:56.063 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:56.062+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:56.063163+0000) 2020-02-06T00:59:57.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:57.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:57.048228+0000) 2020-02-06T00:59:57.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:57.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:57.316041+0000) 2020-02-06T00:59:57.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:57.315+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:57.316407+0000) 2020-02-06T00:59:57.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:57.315+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:57.316491+0000) 2020-02-06T00:59:59.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T00:59:59.557+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.558159+0000) 2020-02-06T00:59:59.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:59.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.560759+0000) 2020-02-06T00:59:59.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T00:59:59.560+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.561159+0000) 2020-02-06T00:59:59.563 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T00:59:59.562+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.562412+0000) 2020-02-06T00:59:59.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T00:59:59.561+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.562459+0000) 2020-02-06T00:59:59.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T00:59:59.561+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.562927+0000) 2020-02-06T00:59:59.564 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T00:59:59.562+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.563288+0000) 2020-02-06T00:59:59.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:59.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.563845+0000) 2020-02-06T00:59:59.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T00:59:59.563+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.564332+0000) 2020-02-06T00:59:59.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T00:59:59.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.564272+0000) 2020-02-06T00:59:59.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T00:59:59.564+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.564770+0000) 2020-02-06T00:59:59.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T00:59:59.564+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.564947+0000) 2020-02-06T00:59:59.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T00:59:59.564+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.565298+0000) 2020-02-06T00:59:59.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:59.566+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.567200+0000) 2020-02-06T00:59:59.581 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T00:59:59.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T00:59:59.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:59.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.692013+0000) 2020-02-06T00:59:59.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T00:59:59.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-05T23:59:59.940205+0000) 2020-02-06T01:00:00.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:00.543+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:00.543490+0000) 2020-02-06T01:00:00.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:00.801+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:00.802171+0000) 2020-02-06T01:00:00.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:00.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:00.823494+0000) 2020-02-06T01:00:00.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:00.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:00.823488+0000) 2020-02-06T01:00:02.926 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:02.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:02.925894+0000) 2020-02-06T01:00:03.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:03.156+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:03.156601+0000) 2020-02-06T01:00:04.587 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:04.586+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.586556+0000) 2020-02-06T01:00:04.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:04.587+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.587424+0000) 2020-02-06T01:00:04.588 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:04.587+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.587891+0000) 2020-02-06T01:00:04.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:04.587+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.588242+0000) 2020-02-06T01:00:04.589 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:04.588+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.588259+0000) 2020-02-06T01:00:04.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:04.588+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.588856+0000) 2020-02-06T01:00:04.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:04.589+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.589186+0000) 2020-02-06T01:00:04.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:04.589+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.589675+0000) 2020-02-06T01:00:04.591 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:04.589+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.590077+0000) 2020-02-06T01:00:04.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:04.589+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.590275+0000) 2020-02-06T01:00:04.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:04.589+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.589865+0000) 2020-02-06T01:00:04.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:04.590+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.590840+0000) 2020-02-06T01:00:04.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:04.596+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.596965+0000) 2020-02-06T01:00:04.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:04.597+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.597332+0000) 2020-02-06T01:00:04.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:04.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:04.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:04.750+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:04.750196+0000) 2020-02-06T01:00:06.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:06.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:06.063422+0000) 2020-02-06T01:00:07.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:07.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:07.048463+0000) 2020-02-06T01:00:07.316 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:07.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:07.316300+0000) 2020-02-06T01:00:07.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:07.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:07.316678+0000) 2020-02-06T01:00:07.317 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:07.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:07.316750+0000) 2020-02-06T01:00:09.612 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:09.610+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.611467+0000) 2020-02-06T01:00:09.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:09.613+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.614108+0000) 2020-02-06T01:00:09.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:09.614+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.615265+0000) 2020-02-06T01:00:09.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:09.615+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.616025+0000) 2020-02-06T01:00:09.665 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:09.615+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.616126+0000) 2020-02-06T01:00:09.665 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:09.616+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.617256+0000) 2020-02-06T01:00:09.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:09.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.618477+0000) 2020-02-06T01:00:09.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:09.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:09.666 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:09.612+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.613322+0000) 2020-02-06T01:00:09.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:09.613+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.614265+0000) 2020-02-06T01:00:09.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:09.614+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.615115+0000) 2020-02-06T01:00:09.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:09.615+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.616511+0000) 2020-02-06T01:00:09.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:09.617+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.618711+0000) 2020-02-06T01:00:09.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:09.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.624783+0000) 2020-02-06T01:00:09.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:09.617+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.618318+0000) 2020-02-06T01:00:09.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:09.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.692265+0000) 2020-02-06T01:00:09.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:09.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:09.940409+0000) 2020-02-06T01:00:10.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:10.543+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:10.543721+0000) 2020-02-06T01:00:10.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:10.801+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:10.802417+0000) 2020-02-06T01:00:10.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:10.822+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:10.823760+0000) 2020-02-06T01:00:10.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:10.822+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:10.823753+0000) 2020-02-06T01:00:12.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:12.925+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:12.926161+0000) 2020-02-06T01:00:13.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:13.155+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:13.156857+0000) 2020-02-06T01:00:14.640 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:14.638+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.639284+0000) 2020-02-06T01:00:14.640 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:14.639+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.640179+0000) 2020-02-06T01:00:14.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:14.640+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.640974+0000) 2020-02-06T01:00:14.642 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:14.640+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.641154+0000) 2020-02-06T01:00:14.642 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:14.641+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.641587+0000) 2020-02-06T01:00:14.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:14.640+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.641872+0000) 2020-02-06T01:00:14.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:14.641+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.642281+0000) 2020-02-06T01:00:14.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:14.641+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.642858+0000) 2020-02-06T01:00:14.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:14.642+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.643273+0000) 2020-02-06T01:00:14.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:14.643+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.643750+0000) 2020-02-06T01:00:14.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:14.642+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.643761+0000) 2020-02-06T01:00:14.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:14.643+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.644530+0000) 2020-02-06T01:00:14.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:14.645+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.645419+0000) 2020-02-06T01:00:14.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:14.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.649768+0000) 2020-02-06T01:00:14.660 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:14.659+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:14.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:14.761+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:14.762737+0000) 2020-02-06T01:00:16.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:16.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:16.063709+0000) 2020-02-06T01:00:17.049 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:17.047+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:17.048708+0000) 2020-02-06T01:00:17.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:17.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:17.316586+0000) 2020-02-06T01:00:17.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:17.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:17.316917+0000) 2020-02-06T01:00:17.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:17.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:17.316983+0000) 2020-02-06T01:00:19.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:19.664+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.665409+0000) 2020-02-06T01:00:19.667 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:19.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.666423+0000) 2020-02-06T01:00:19.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:19.665+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.666672+0000) 2020-02-06T01:00:19.668 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:19.667+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.667533+0000) 2020-02-06T01:00:19.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:19.666+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.667756+0000) 2020-02-06T01:00:19.668 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:19.667+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.668067+0000) 2020-02-06T01:00:19.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:19.668+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.668776+0000) 2020-02-06T01:00:19.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:19.668+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.669400+0000) 2020-02-06T01:00:19.670 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:19.668+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.669685+0000) 2020-02-06T01:00:19.671 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:19.669+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.670341+0000) 2020-02-06T01:00:19.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:19.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.670324+0000) 2020-02-06T01:00:19.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:19.671+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.671846+0000) 2020-02-06T01:00:19.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:19.671+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.671608+0000) 2020-02-06T01:00:19.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:19.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.675686+0000) 2020-02-06T01:00:19.686 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:19.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190599 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:19.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:19.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.692423+0000) 2020-02-06T01:00:19.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:19.939+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:19.940631+0000) 2020-02-06T01:00:20.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:20.543+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:20.543956+0000) 2020-02-06T01:00:20.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:20.801+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:20.802689+0000) 2020-02-06T01:00:20.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:20.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:20.824010+0000) 2020-02-06T01:00:20.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:20.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:20.824011+0000) 2020-02-06T01:00:20.955 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:00:20.961 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:00:22.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:22.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:22.926466+0000) 2020-02-06T01:00:23.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:23.156+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:23.157115+0000) 2020-02-06T01:00:24.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:24.690+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.691510+0000) 2020-02-06T01:00:24.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:24.690+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.690642+0000) 2020-02-06T01:00:24.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:24.691+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.692383+0000) 2020-02-06T01:00:24.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:24.691+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.692875+0000) 2020-02-06T01:00:24.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:24.692+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.693435+0000) 2020-02-06T01:00:24.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:24.692+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.693831+0000) 2020-02-06T01:00:24.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:24.694+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.695275+0000) 2020-02-06T01:00:24.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:24.694+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.695165+0000) 2020-02-06T01:00:24.697 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:24.695+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.696254+0000) 2020-02-06T01:00:24.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:24.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.696425+0000) 2020-02-06T01:00:24.697 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:24.696+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.697034+0000) 2020-02-06T01:00:24.698 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:24.696+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.697400+0000) 2020-02-06T01:00:24.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:24.697+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.698273+0000) 2020-02-06T01:00:24.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:24.699+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.700143+0000) 2020-02-06T01:00:24.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:24.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:24.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:24.770+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:24.771061+0000) 2020-02-06T01:00:26.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:26.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:26.063949+0000) 2020-02-06T01:00:27.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:27.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:27.048938+0000) 2020-02-06T01:00:27.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:27.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:27.316817+0000) 2020-02-06T01:00:27.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:27.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:27.317134+0000) 2020-02-06T01:00:27.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:27.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:27.317292+0000) 2020-02-06T01:00:29.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:29.691+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.692635+0000) 2020-02-06T01:00:29.719 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:29.717+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.718394+0000) 2020-02-06T01:00:29.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:29.718+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.719315+0000) 2020-02-06T01:00:29.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:29.718+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.719372+0000) 2020-02-06T01:00:29.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:29.719+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.720202+0000) 2020-02-06T01:00:29.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:29.719+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.720513+0000) 2020-02-06T01:00:29.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:29.719+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.720733+0000) 2020-02-06T01:00:29.722 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:29.721+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.721504+0000) 2020-02-06T01:00:29.723 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:29.721+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.722197+0000) 2020-02-06T01:00:29.723 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:29.721+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.722627+0000) 2020-02-06T01:00:29.723 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:29.722+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.723351+0000) 2020-02-06T01:00:29.724 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:29.723+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.723690+0000) 2020-02-06T01:00:29.724 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:29.722+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.723850+0000) 2020-02-06T01:00:29.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:29.724+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.725959+0000) 2020-02-06T01:00:29.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:29.726+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.727190+0000) 2020-02-06T01:00:29.739 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:29.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:29.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:29.940+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:29.940838+0000) 2020-02-06T01:00:30.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:30.544+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:30.544232+0000) 2020-02-06T01:00:30.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:30.802+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:30.802915+0000) 2020-02-06T01:00:30.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:30.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:30.824291+0000) 2020-02-06T01:00:30.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:30.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:30.824270+0000) 2020-02-06T01:00:32.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:32.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:32.926759+0000) 2020-02-06T01:00:33.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:33.156+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:33.157352+0000) 2020-02-06T01:00:34.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:34.741+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.742996+0000) 2020-02-06T01:00:34.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:34.742+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.743666+0000) 2020-02-06T01:00:34.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:34.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.745413+0000) 2020-02-06T01:00:34.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:34.744+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.745354+0000) 2020-02-06T01:00:34.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:34.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.746023+0000) 2020-02-06T01:00:34.748 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:34.747+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.747966+0000) 2020-02-06T01:00:34.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:34.747+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.747977+0000) 2020-02-06T01:00:34.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:34.747+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.748302+0000) 2020-02-06T01:00:34.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:34.748+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.748693+0000) 2020-02-06T01:00:34.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:34.748+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.749437+0000) 2020-02-06T01:00:34.750 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:34.748+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.749040+0000) 2020-02-06T01:00:34.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:34.749+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.750649+0000) 2020-02-06T01:00:34.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:34.749+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.751039+0000) 2020-02-06T01:00:34.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:34.755+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.757006+0000) 2020-02-06T01:00:34.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:34.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:34.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:34.778+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:34.779445+0000) 2020-02-06T01:00:36.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:36.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:36.064206+0000) 2020-02-06T01:00:37.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:37.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:37.049174+0000) 2020-02-06T01:00:37.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:37.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:37.317060+0000) 2020-02-06T01:00:37.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:37.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:37.317400+0000) 2020-02-06T01:00:37.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:37.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:37.317612+0000) 2020-02-06T01:00:39.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:39.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.692871+0000) 2020-02-06T01:00:39.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:39.770+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.771362+0000) 2020-02-06T01:00:39.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:39.773+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.774459+0000) 2020-02-06T01:00:39.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:39.775+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776234+0000) 2020-02-06T01:00:39.779 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:39.773+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.775019+0000) 2020-02-06T01:00:39.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:39.775+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776800+0000) 2020-02-06T01:00:39.780 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:39.775+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776948+0000) 2020-02-06T01:00:39.780 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:39.775+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776603+0000) 2020-02-06T01:00:39.780 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:39.772+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.773585+0000) 2020-02-06T01:00:39.780 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:39.772+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.773789+0000) 2020-02-06T01:00:39.781 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:39.772+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.774372+0000) 2020-02-06T01:00:39.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:39.773+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.775217+0000) 2020-02-06T01:00:39.781 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:39.774+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776392+0000) 2020-02-06T01:00:39.781 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:39.775+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.776690+0000) 2020-02-06T01:00:39.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:39.780+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.781994+0000) 2020-02-06T01:00:39.793 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:39.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:39.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:39.940+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:39.941045+0000) 2020-02-06T01:00:40.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:40.543+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:40.544491+0000) 2020-02-06T01:00:40.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:40.802+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:40.803183+0000) 2020-02-06T01:00:40.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:40.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:40.824518+0000) 2020-02-06T01:00:40.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:40.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:40.824518+0000) 2020-02-06T01:00:42.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:42.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:42.927018+0000) 2020-02-06T01:00:43.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:43.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:43.157669+0000) 2020-02-06T01:00:44.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:44.786+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.787361+0000) 2020-02-06T01:00:44.797 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:44.795+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.796953+0000) 2020-02-06T01:00:44.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:44.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.798469+0000) 2020-02-06T01:00:44.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:44.797+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.798495+0000) 2020-02-06T01:00:44.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:44.798+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.799553+0000) 2020-02-06T01:00:44.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:44.798+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.800124+0000) 2020-02-06T01:00:44.801 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:44.799+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.800507+0000) 2020-02-06T01:00:44.801 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:44.799+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.800684+0000) 2020-02-06T01:00:44.802 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:44.800+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.801202+0000) 2020-02-06T01:00:44.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:44.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.801343+0000) 2020-02-06T01:00:44.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:44.800+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.801606+0000) 2020-02-06T01:00:44.803 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:44.799+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.801142+0000) 2020-02-06T01:00:44.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:44.801+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.802458+0000) 2020-02-06T01:00:44.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:44.800+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.802396+0000) 2020-02-06T01:00:44.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:44.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:44.808152+0000) 2020-02-06T01:00:44.820 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:44.818+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190804 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:46.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:46.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:46.064489+0000) 2020-02-06T01:00:47.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:47.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:47.049398+0000) 2020-02-06T01:00:47.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:47.315+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:47.317352+0000) 2020-02-06T01:00:47.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:47.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:47.317657+0000) 2020-02-06T01:00:47.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:47.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:47.317850+0000) 2020-02-06T01:00:49.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:49.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.693146+0000) 2020-02-06T01:00:49.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:49.825+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.826500+0000) 2020-02-06T01:00:49.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:49.825+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.826848+0000) 2020-02-06T01:00:49.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:49.826+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.827748+0000) 2020-02-06T01:00:49.829 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:49.827+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.828584+0000) 2020-02-06T01:00:49.829 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:49.827+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.828943+0000) 2020-02-06T01:00:49.830 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:49.828+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.829534+0000) 2020-02-06T01:00:49.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:49.828+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.829842+0000) 2020-02-06T01:00:49.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:49.829+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.830743+0000) 2020-02-06T01:00:49.837 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:49.830+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.831462+0000) 2020-02-06T01:00:49.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:49.829+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.831023+0000) 2020-02-06T01:00:49.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:49.830+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.831191+0000) 2020-02-06T01:00:49.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:49.830+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.831662+0000) 2020-02-06T01:00:49.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:49.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.835590+0000) 2020-02-06T01:00:49.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:49.830+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.831856+0000) 2020-02-06T01:00:49.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:49.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190845 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:49.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:49.940+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:49.941287+0000) 2020-02-06T01:00:50.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:50.543+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:50.544705+0000) 2020-02-06T01:00:50.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:50.802+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:50.803406+0000) 2020-02-06T01:00:50.825 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:50.823+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:50.824771+0000) 2020-02-06T01:00:50.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:50.823+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:50.824765+0000) 2020-02-06T01:00:51.120 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:00:51.126 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:00:52.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:52.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:52.927269+0000) 2020-02-06T01:00:53.158 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:53.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:53.157920+0000) 2020-02-06T01:00:54.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:54.797+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.797508+0000) 2020-02-06T01:00:54.850 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:54.848+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.849771+0000) 2020-02-06T01:00:54.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:54.850+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.851625+0000) 2020-02-06T01:00:54.852 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:54.850+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.851471+0000) 2020-02-06T01:00:54.853 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:54.849+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.851200+0000) 2020-02-06T01:00:54.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:54.851+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.852919+0000) 2020-02-06T01:00:54.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:54.852+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.853357+0000) 2020-02-06T01:00:54.854 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:54.852+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.853862+0000) 2020-02-06T01:00:54.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:54.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.854282+0000) 2020-02-06T01:00:54.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:54.854+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.855441+0000) 2020-02-06T01:00:54.856 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:54.853+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.855060+0000) 2020-02-06T01:00:54.857 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:54.854+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.856003+0000) 2020-02-06T01:00:54.857 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:54.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.856165+0000) 2020-02-06T01:00:54.857 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:54.854+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.856214+0000) 2020-02-06T01:00:54.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:54.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:54.862423+0000) 2020-02-06T01:00:54.874 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:54.872+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:56.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:56.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:56.064687+0000) 2020-02-06T01:00:57.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:57.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:57.049582+0000) 2020-02-06T01:00:57.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:57.316+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:57.317632+0000) 2020-02-06T01:00:57.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:57.316+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:57.317929+0000) 2020-02-06T01:00:57.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:57.316+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:57.318112+0000) 2020-02-06T01:00:59.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:59.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.693392+0000) 2020-02-06T01:00:59.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:59.880+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.880678+0000) 2020-02-06T01:00:59.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:00:59.880+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.881661+0000) 2020-02-06T01:00:59.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:59.882+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.882106+0000) 2020-02-06T01:00:59.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:00:59.882+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.882398+0000) 2020-02-06T01:00:59.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:00:59.882+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.882562+0000) 2020-02-06T01:00:59.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:00:59.882+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.883718+0000) 2020-02-06T01:00:59.884 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:00:59.883+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.883830+0000) 2020-02-06T01:00:59.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:00:59.884+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.884222+0000) 2020-02-06T01:00:59.886 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:00:59.884+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.885505+0000) 2020-02-06T01:00:59.886 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:00:59.883+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.885242+0000) 2020-02-06T01:00:59.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:00:59.886+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.886067+0000) 2020-02-06T01:00:59.887 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:00:59.884+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.886286+0000) 2020-02-06T01:00:59.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:00:59.885+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.886583+0000) 2020-02-06T01:00:59.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:59.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.889376+0000) 2020-02-06T01:00:59.900 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:00:59.897+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190927 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:00:59.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:00:59.941+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:00:59.941487+0000) 2020-02-06T01:01:00.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:00.544+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:00.544943+0000) 2020-02-06T01:01:00.804 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:00.803+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:00.803633+0000) 2020-02-06T01:01:00.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:00.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:00.824991+0000) 2020-02-06T01:01:00.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:00.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:00.824991+0000) 2020-02-06T01:01:02.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:02.926+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:02.927536+0000) 2020-02-06T01:01:03.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:03.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:03.158187+0000) 2020-02-06T01:01:04.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:04.804+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.805414+0000) 2020-02-06T01:01:04.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:04.904+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.905595+0000) 2020-02-06T01:01:04.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:04.905+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.906158+0000) 2020-02-06T01:01:04.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:04.905+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.906152+0000) 2020-02-06T01:01:04.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:04.905+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.906745+0000) 2020-02-06T01:01:04.909 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:04.906+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.907612+0000) 2020-02-06T01:01:04.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:04.906+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.908059+0000) 2020-02-06T01:01:04.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:04.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.907045+0000) 2020-02-06T01:01:04.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:04.906+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.907685+0000) 2020-02-06T01:01:04.911 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:04.906+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.907319+0000) 2020-02-06T01:01:04.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:04.906+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.907914+0000) 2020-02-06T01:01:04.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:04.907+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.908618+0000) 2020-02-06T01:01:04.912 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:04.907+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.908937+0000) 2020-02-06T01:01:04.913 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:04.907+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.909289+0000) 2020-02-06T01:01:04.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:04.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:04.914805+0000) 2020-02-06T01:01:04.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:04.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 190968 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:06.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:06.063+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:06.064927+0000) 2020-02-06T01:01:07.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:07.048+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:07.049811+0000) 2020-02-06T01:01:07.318 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:07.317+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:07.317921+0000) 2020-02-06T01:01:07.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:07.317+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:07.318155+0000) 2020-02-06T01:01:07.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:07.317+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:07.318507+0000) 2020-02-06T01:01:09.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:09.692+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.693686+0000) 2020-02-06T01:01:09.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:09.930+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.931898+0000) 2020-02-06T01:01:09.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:09.931+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.932630+0000) 2020-02-06T01:01:09.939 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:09.932+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.933792+0000) 2020-02-06T01:01:09.939 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:09.932+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.934217+0000) 2020-02-06T01:01:09.939 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:09.933+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.935250+0000) 2020-02-06T01:01:09.939 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:09.933+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.934842+0000) 2020-02-06T01:01:09.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:09.934+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.935452+0000) 2020-02-06T01:01:09.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:09.934+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.936292+0000) 2020-02-06T01:01:09.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:09.932+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.933187+0000) 2020-02-06T01:01:09.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:09.932+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.933522+0000) 2020-02-06T01:01:09.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:09.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.935775+0000) 2020-02-06T01:01:09.941 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:09.933+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.934424+0000) 2020-02-06T01:01:09.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:09.934+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.935420+0000) 2020-02-06T01:01:09.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:09.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.941396+0000) 2020-02-06T01:01:09.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:09.940+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:09.941622+0000) 2020-02-06T01:01:09.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:09.953+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:10.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:10.544+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:10.545176+0000) 2020-02-06T01:01:10.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:10.802+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:10.803853+0000) 2020-02-06T01:01:10.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:10.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:10.825196+0000) 2020-02-06T01:01:10.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:10.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:10.825198+0000) 2020-02-06T01:01:12.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:12.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:12.927784+0000) 2020-02-06T01:01:13.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:13.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:13.158402+0000) 2020-02-06T01:01:14.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:14.812+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.812885+0000) 2020-02-06T01:01:14.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:14.959+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.960844+0000) 2020-02-06T01:01:14.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:14.960+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.961524+0000) 2020-02-06T01:01:14.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:14.961+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.963003+0000) 2020-02-06T01:01:14.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:14.961+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.963203+0000) 2020-02-06T01:01:14.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:14.962+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.963517+0000) 2020-02-06T01:01:14.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:14.962+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.963834+0000) 2020-02-06T01:01:14.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:14.963+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.964554+0000) 2020-02-06T01:01:14.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:14.963+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.964416+0000) 2020-02-06T01:01:14.966 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:14.963+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.964770+0000) 2020-02-06T01:01:14.966 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:14.963+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.964913+0000) 2020-02-06T01:01:14.967 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:14.963+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.965110+0000) 2020-02-06T01:01:14.967 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:14.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.964753+0000) 2020-02-06T01:01:14.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:14.963+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.965300+0000) 2020-02-06T01:01:14.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:14.969+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:14.970158+0000) 2020-02-06T01:01:14.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:14.980+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:16.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:16.064+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:16.065169+0000) 2020-02-06T01:01:17.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:17.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:17.050063+0000) 2020-02-06T01:01:17.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:17.317+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:17.318159+0000) 2020-02-06T01:01:17.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:17.317+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:17.318481+0000) 2020-02-06T01:01:17.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:17.317+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:17.318744+0000) 2020-02-06T01:01:19.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:19.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.693897+0000) 2020-02-06T01:01:19.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:19.940+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.941804+0000) 2020-02-06T01:01:19.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:19.985+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.986375+0000) 2020-02-06T01:01:19.988 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:19.986+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.987272+0000) 2020-02-06T01:01:19.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:19.987+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.988209+0000) 2020-02-06T01:01:19.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:19.988+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.989800+0000) 2020-02-06T01:01:19.991 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:19.989+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.990217+0000) 2020-02-06T01:01:19.991 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:19.989+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.990778+0000) 2020-02-06T01:01:19.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:19.990+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.991267+0000) 2020-02-06T01:01:19.992 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:19.990+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.992070+0000) 2020-02-06T01:01:19.993 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:19.991+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.992755+0000) 2020-02-06T01:01:19.993 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:19.991+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.992980+0000) 2020-02-06T01:01:19.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:19.991+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.993292+0000) 2020-02-06T01:01:19.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:19.994+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.996015+0000) 2020-02-06T01:01:19.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:19.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.996254+0000) 2020-02-06T01:01:19.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:19.995+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:19.996445+0000) 2020-02-06T01:01:20.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:20.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:20.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:20.544+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:20.545411+0000) 2020-02-06T01:01:20.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:20.803+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:20.804059+0000) 2020-02-06T01:01:20.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:20.824+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:20.825358+0000) 2020-02-06T01:01:20.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:20.824+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:20.825425+0000) 2020-02-06T01:01:21.269 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:01:21.276 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:01:22.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:22.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:22.928048+0000) 2020-02-06T01:01:23.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:23.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:23.158636+0000) 2020-02-06T01:01:24.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:24.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:24.820855+0000) 2020-02-06T01:01:25.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:25.012+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.013482+0000) 2020-02-06T01:01:25.015 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:25.013+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.014397+0000) 2020-02-06T01:01:25.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:25.014+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.015420+0000) 2020-02-06T01:01:25.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:25.014+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.015695+0000) 2020-02-06T01:01:25.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:25.015+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.017328+0000) 2020-02-06T01:01:25.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:25.016+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.017304+0000) 2020-02-06T01:01:25.018 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:25.016+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.017788+0000) 2020-02-06T01:01:25.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:25.017+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.018510+0000) 2020-02-06T01:01:25.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:25.017+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.019373+0000) 2020-02-06T01:01:25.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:25.018+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.019667+0000) 2020-02-06T01:01:25.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:25.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.019685+0000) 2020-02-06T01:01:25.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:25.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.020282+0000) 2020-02-06T01:01:25.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:25.019+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.020702+0000) 2020-02-06T01:01:25.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:25.023+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:25.024326+0000) 2020-02-06T01:01:25.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:25.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:26.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:26.064+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:26.065399+0000) 2020-02-06T01:01:27.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:27.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:27.050364+0000) 2020-02-06T01:01:27.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:27.318+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:27.318418+0000) 2020-02-06T01:01:27.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:27.318+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:27.318707+0000) 2020-02-06T01:01:27.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:27.318+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:27.318953+0000) 2020-02-06T01:01:29.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:29.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:29.694160+0000) 2020-02-06T01:01:29.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:29.941+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:29.942028+0000) 2020-02-06T01:01:30.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:30.039+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.040362+0000) 2020-02-06T01:01:30.042 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:30.041+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.041998+0000) 2020-02-06T01:01:30.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:30.041+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.042240+0000) 2020-02-06T01:01:30.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:30.041+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.042845+0000) 2020-02-06T01:01:30.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:30.042+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.043331+0000) 2020-02-06T01:01:30.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:30.043+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.044165+0000) 2020-02-06T01:01:30.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:30.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.044373+0000) 2020-02-06T01:01:30.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:30.044+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.045182+0000) 2020-02-06T01:01:30.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:30.044+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.044949+0000) 2020-02-06T01:01:30.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:30.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.045931+0000) 2020-02-06T01:01:30.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:30.045+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.046010+0000) 2020-02-06T01:01:30.048 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:30.046+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.046899+0000) 2020-02-06T01:01:30.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:30.046+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.047008+0000) 2020-02-06T01:01:30.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:30.047+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.048674+0000) 2020-02-06T01:01:30.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:30.061+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:30.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:30.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.545631+0000) 2020-02-06T01:01:30.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:30.803+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.804312+0000) 2020-02-06T01:01:30.826 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:30.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.825609+0000) 2020-02-06T01:01:30.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:30.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:30.825641+0000) 2020-02-06T01:01:32.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:32.927+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:32.928302+0000) 2020-02-06T01:01:33.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:33.157+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:33.158860+0000) 2020-02-06T01:01:34.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:34.827+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:34.828709+0000) 2020-02-06T01:01:35.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:35.066+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.067237+0000) 2020-02-06T01:01:35.069 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:35.068+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.068642+0000) 2020-02-06T01:01:35.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:35.067+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.068759+0000) 2020-02-06T01:01:35.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:35.068+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.069473+0000) 2020-02-06T01:01:35.071 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:35.069+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.069918+0000) 2020-02-06T01:01:35.071 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:35.069+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.070345+0000) 2020-02-06T01:01:35.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:35.068+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.070000+0000) 2020-02-06T01:01:35.071 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:35.069+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.070477+0000) 2020-02-06T01:01:35.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:35.070+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.070741+0000) 2020-02-06T01:01:35.072 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:35.071+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.071646+0000) 2020-02-06T01:01:35.073 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:35.069+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.070815+0000) 2020-02-06T01:01:35.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:35.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.071329+0000) 2020-02-06T01:01:35.073 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:35.071+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.072204+0000) 2020-02-06T01:01:35.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:35.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:35.076571+0000) 2020-02-06T01:01:35.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:35.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:36.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:36.065+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:36.065680+0000) 2020-02-06T01:01:37.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:37.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:37.050612+0000) 2020-02-06T01:01:37.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:37.318+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:37.318649+0000) 2020-02-06T01:01:37.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:37.318+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:37.318986+0000) 2020-02-06T01:01:37.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:37.318+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:37.319180+0000) 2020-02-06T01:01:39.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:39.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:39.694393+0000) 2020-02-06T01:01:39.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:39.941+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:39.942318+0000) 2020-02-06T01:01:40.094 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:40.091+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.092935+0000) 2020-02-06T01:01:40.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:40.092+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.093480+0000) 2020-02-06T01:01:40.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:40.094+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.094681+0000) 2020-02-06T01:01:40.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:40.093+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.094940+0000) 2020-02-06T01:01:40.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:40.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.095289+0000) 2020-02-06T01:01:40.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:40.095+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.095565+0000) 2020-02-06T01:01:40.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:40.095+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.096220+0000) 2020-02-06T01:01:40.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:40.096+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.096461+0000) 2020-02-06T01:01:40.098 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:40.095+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.096952+0000) 2020-02-06T01:01:40.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:40.096+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.097481+0000) 2020-02-06T01:01:40.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:40.096+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.096859+0000) 2020-02-06T01:01:40.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:40.096+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.098025+0000) 2020-02-06T01:01:40.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:40.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.099654+0000) 2020-02-06T01:01:40.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:40.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.103795+0000) 2020-02-06T01:01:40.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:40.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:40.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:40.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.545827+0000) 2020-02-06T01:01:40.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:40.804+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.804527+0000) 2020-02-06T01:01:40.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:40.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.825828+0000) 2020-02-06T01:01:40.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:40.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:40.825880+0000) 2020-02-06T01:01:42.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:42.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:42.928630+0000) 2020-02-06T01:01:43.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:43.158+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:43.159142+0000) 2020-02-06T01:01:44.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:44.835+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:44.836705+0000) 2020-02-06T01:01:45.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:45.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.120231+0000) 2020-02-06T01:01:45.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:45.119+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.120433+0000) 2020-02-06T01:01:45.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:45.119+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.120914+0000) 2020-02-06T01:01:45.122 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:45.120+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.121217+0000) 2020-02-06T01:01:45.122 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:45.120+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.121388+0000) 2020-02-06T01:01:45.123 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:45.120+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.121830+0000) 2020-02-06T01:01:45.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:45.121+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.122120+0000) 2020-02-06T01:01:45.124 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:45.123+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.123616+0000) 2020-02-06T01:01:45.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:45.122+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.123319+0000) 2020-02-06T01:01:45.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:45.122+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.123406+0000) 2020-02-06T01:01:45.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:45.123+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.124289+0000) 2020-02-06T01:01:45.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:45.125+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.125527+0000) 2020-02-06T01:01:45.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:45.125+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.125872+0000) 2020-02-06T01:01:45.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:45.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:45.131734+0000) 2020-02-06T01:01:45.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:45.139+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:46.066 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:46.065+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:46.065930+0000) 2020-02-06T01:01:47.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:47.049+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:47.050862+0000) 2020-02-06T01:01:47.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:47.318+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:47.318900+0000) 2020-02-06T01:01:47.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:47.318+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:47.319259+0000) 2020-02-06T01:01:47.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:47.319+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:47.319415+0000) 2020-02-06T01:01:49.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:49.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:49.694600+0000) 2020-02-06T01:01:49.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:49.942+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:49.942562+0000) 2020-02-06T01:01:50.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:50.144+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.145637+0000) 2020-02-06T01:01:50.147 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:50.144+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.145536+0000) 2020-02-06T01:01:50.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:50.145+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.146115+0000) 2020-02-06T01:01:50.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:50.145+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.146964+0000) 2020-02-06T01:01:50.148 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:50.146+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.147505+0000) 2020-02-06T01:01:50.149 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:50.147+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.148107+0000) 2020-02-06T01:01:50.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:50.148+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.148730+0000) 2020-02-06T01:01:50.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:50.147+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.148871+0000) 2020-02-06T01:01:50.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:50.148+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.149269+0000) 2020-02-06T01:01:50.150 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:50.149+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.150014+0000) 2020-02-06T01:01:50.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:50.149+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.150377+0000) 2020-02-06T01:01:50.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:50.149+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.150538+0000) 2020-02-06T01:01:50.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:50.154+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.155595+0000) 2020-02-06T01:01:50.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:50.154+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.155910+0000) 2020-02-06T01:01:50.167 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:50.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:50.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:50.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.546066+0000) 2020-02-06T01:01:50.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:50.804+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.804761+0000) 2020-02-06T01:01:50.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:50.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.826023+0000) 2020-02-06T01:01:50.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:50.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:50.826149+0000) 2020-02-06T01:01:51.448 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:01:51.455 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:01:52.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:52.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:52.928884+0000) 2020-02-06T01:01:53.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:53.159+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:53.159374+0000) 2020-02-06T01:01:54.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:54.844+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:54.844877+0000) 2020-02-06T01:01:55.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:01:55.172+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.172199+0000) 2020-02-06T01:01:55.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:55.173+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.173458+0000) 2020-02-06T01:01:55.174 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:01:55.173+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.173931+0000) 2020-02-06T01:01:55.175 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:01:55.174+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.174483+0000) 2020-02-06T01:01:55.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:01:55.175+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.175488+0000) 2020-02-06T01:01:55.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:01:55.174+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.175113+0000) 2020-02-06T01:01:55.176 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:55.175+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.176027+0000) 2020-02-06T01:01:55.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:55.176+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.176510+0000) 2020-02-06T01:01:55.178 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:01:55.177+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.177554+0000) 2020-02-06T01:01:55.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:55.176+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.177307+0000) 2020-02-06T01:01:55.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:55.178+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.178164+0000) 2020-02-06T01:01:55.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:55.178+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.178675+0000) 2020-02-06T01:01:55.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:55.182+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.182411+0000) 2020-02-06T01:01:55.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:55.182+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:55.182671+0000) 2020-02-06T01:01:55.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:01:55.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:01:56.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:01:56.065+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:56.066202+0000) 2020-02-06T01:01:57.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:01:57.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:57.051120+0000) 2020-02-06T01:01:57.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:01:57.319+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:57.319145+0000) 2020-02-06T01:01:57.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:01:57.319+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:57.319632+0000) 2020-02-06T01:01:57.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:01:57.319+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:57.319800+0000) 2020-02-06T01:01:59.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:59.693+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:59.694831+0000) 2020-02-06T01:01:59.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:01:59.941+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:01:59.942793+0000) 2020-02-06T01:02:00.198 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:00.196+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.197285+0000) 2020-02-06T01:02:00.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:00.198+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.198450+0000) 2020-02-06T01:02:00.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:00.199+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.200158+0000) 2020-02-06T01:02:00.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:00.199+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.200379+0000) 2020-02-06T01:02:00.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:00.199+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.200686+0000) 2020-02-06T01:02:00.202 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:00.200+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.201354+0000) 2020-02-06T01:02:00.203 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:00.201+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.201900+0000) 2020-02-06T01:02:00.203 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:00.201+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.202232+0000) 2020-02-06T01:02:00.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:00.201+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.202587+0000) 2020-02-06T01:02:00.205 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:00.202+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.203096+0000) 2020-02-06T01:02:00.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:00.203+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.203862+0000) 2020-02-06T01:02:00.205 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:00.203+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.204151+0000) 2020-02-06T01:02:00.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:00.204+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.204703+0000) 2020-02-06T01:02:00.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:00.207+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.208811+0000) 2020-02-06T01:02:00.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:00.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:00.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:00.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.546310+0000) 2020-02-06T01:02:00.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:00.803+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.804978+0000) 2020-02-06T01:02:00.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:00.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.826271+0000) 2020-02-06T01:02:00.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:00.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:00.826366+0000) 2020-02-06T01:02:02.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:02.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:02.929167+0000) 2020-02-06T01:02:03.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:03.158+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:03.159600+0000) 2020-02-06T01:02:04.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:04.855+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:04.856646+0000) 2020-02-06T01:02:05.229 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:05.224+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.225893+0000) 2020-02-06T01:02:05.229 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:05.225+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.227016+0000) 2020-02-06T01:02:05.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:05.225+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.226424+0000) 2020-02-06T01:02:05.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:05.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.228036+0000) 2020-02-06T01:02:05.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:05.228+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.229252+0000) 2020-02-06T01:02:05.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:05.228+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.229751+0000) 2020-02-06T01:02:05.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:05.228+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.230288+0000) 2020-02-06T01:02:05.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:05.229+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.230657+0000) 2020-02-06T01:02:05.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:05.229+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.230959+0000) 2020-02-06T01:02:05.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:05.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.231320+0000) 2020-02-06T01:02:05.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:05.230+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.232056+0000) 2020-02-06T01:02:05.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:05.230+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.232415+0000) 2020-02-06T01:02:05.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:05.231+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.232251+0000) 2020-02-06T01:02:05.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:05.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:05.237176+0000) 2020-02-06T01:02:05.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:05.245+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:06.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:06.064+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:06.066384+0000) 2020-02-06T01:02:07.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:07.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:07.051357+0000) 2020-02-06T01:02:07.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:07.318+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:07.319394+0000) 2020-02-06T01:02:07.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:07.318+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:07.319869+0000) 2020-02-06T01:02:07.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:07.318+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:07.320043+0000) 2020-02-06T01:02:09.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:09.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:09.695063+0000) 2020-02-06T01:02:09.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:09.942+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:09.943047+0000) 2020-02-06T01:02:10.252 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:10.250+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.251539+0000) 2020-02-06T01:02:10.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:10.251+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.252902+0000) 2020-02-06T01:02:10.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:10.252+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.253689+0000) 2020-02-06T01:02:10.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:10.252+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.253716+0000) 2020-02-06T01:02:10.255 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:10.253+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.254499+0000) 2020-02-06T01:02:10.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:10.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.254601+0000) 2020-02-06T01:02:10.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:10.253+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.254644+0000) 2020-02-06T01:02:10.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:10.254+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.255443+0000) 2020-02-06T01:02:10.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:10.254+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.255651+0000) 2020-02-06T01:02:10.256 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:10.254+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.255625+0000) 2020-02-06T01:02:10.257 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:10.254+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.255610+0000) 2020-02-06T01:02:10.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:10.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.256724+0000) 2020-02-06T01:02:10.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:10.256+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.257793+0000) 2020-02-06T01:02:10.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:10.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.261754+0000) 2020-02-06T01:02:10.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:10.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:10.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:10.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.546550+0000) 2020-02-06T01:02:10.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:10.804+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.805178+0000) 2020-02-06T01:02:10.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:10.825+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.826488+0000) 2020-02-06T01:02:10.827 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:10.825+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:10.826587+0000) 2020-02-06T01:02:12.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:12.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:12.929454+0000) 2020-02-06T01:02:13.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:13.159+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:13.159833+0000) 2020-02-06T01:02:14.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:14.864+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:14.864938+0000) 2020-02-06T01:02:15.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:15.276+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.277460+0000) 2020-02-06T01:02:15.279 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:15.276+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.278332+0000) 2020-02-06T01:02:15.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:15.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.279611+0000) 2020-02-06T01:02:15.281 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:15.278+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.279689+0000) 2020-02-06T01:02:15.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:15.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.280093+0000) 2020-02-06T01:02:15.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:15.280+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.281332+0000) 2020-02-06T01:02:15.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:15.280+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.281756+0000) 2020-02-06T01:02:15.282 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:15.280+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.281827+0000) 2020-02-06T01:02:15.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:15.281+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.282573+0000) 2020-02-06T01:02:15.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:15.281+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.282598+0000) 2020-02-06T01:02:15.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:15.282+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.283612+0000) 2020-02-06T01:02:15.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:15.282+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.284292+0000) 2020-02-06T01:02:15.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:15.286+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.287925+0000) 2020-02-06T01:02:15.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:15.287+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:15.288301+0000) 2020-02-06T01:02:15.298 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:15.296+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:16.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:16.065+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:16.066661+0000) 2020-02-06T01:02:17.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:17.050+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:17.051582+0000) 2020-02-06T01:02:17.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:17.318+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:17.319688+0000) 2020-02-06T01:02:17.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:17.318+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:17.320111+0000) 2020-02-06T01:02:17.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:17.318+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:17.320272+0000) 2020-02-06T01:02:19.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:19.694+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:19.695372+0000) 2020-02-06T01:02:19.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:19.942+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:19.943305+0000) 2020-02-06T01:02:20.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:20.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.302564+0000) 2020-02-06T01:02:20.304 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:20.301+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.302817+0000) 2020-02-06T01:02:20.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:20.302+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.303549+0000) 2020-02-06T01:02:20.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:20.303+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.304200+0000) 2020-02-06T01:02:20.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:20.303+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.305123+0000) 2020-02-06T01:02:20.307 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:20.305+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.306413+0000) 2020-02-06T01:02:20.307 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:20.306+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.307158+0000) 2020-02-06T01:02:20.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:20.306+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.308262+0000) 2020-02-06T01:02:20.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:20.307+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.308458+0000) 2020-02-06T01:02:20.311 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:20.307+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.309011+0000) 2020-02-06T01:02:20.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:20.307+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.308921+0000) 2020-02-06T01:02:20.311 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:20.307+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.309275+0000) 2020-02-06T01:02:20.311 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:20.308+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.309705+0000) 2020-02-06T01:02:20.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:20.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.312136+0000) 2020-02-06T01:02:20.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:20.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191581 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:20.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:20.545+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.546762+0000) 2020-02-06T01:02:20.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:20.804+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.805420+0000) 2020-02-06T01:02:20.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:20.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.826706+0000) 2020-02-06T01:02:20.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:20.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:20.826799+0000) 2020-02-06T01:02:21.654 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:02:21.662 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:02:22.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:22.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:22.929739+0000) 2020-02-06T01:02:23.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:23.159+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:23.160061+0000) 2020-02-06T01:02:24.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:24.872+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:24.873121+0000) 2020-02-06T01:02:25.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:25.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.329678+0000) 2020-02-06T01:02:25.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:25.329+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.330222+0000) 2020-02-06T01:02:25.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:25.330+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.331670+0000) 2020-02-06T01:02:25.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:25.330+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.331901+0000) 2020-02-06T01:02:25.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:25.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.332517+0000) 2020-02-06T01:02:25.333 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:25.331+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.332675+0000) 2020-02-06T01:02:25.334 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:25.332+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.333342+0000) 2020-02-06T01:02:25.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:25.331+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.333404+0000) 2020-02-06T01:02:25.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:25.333+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.334645+0000) 2020-02-06T01:02:25.335 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:25.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.334962+0000) 2020-02-06T01:02:25.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:25.334+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.335540+0000) 2020-02-06T01:02:25.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:25.334+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.335700+0000) 2020-02-06T01:02:25.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:25.334+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.335965+0000) 2020-02-06T01:02:25.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:25.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:25.340229+0000) 2020-02-06T01:02:25.349 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:25.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191622 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:26.067 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:26.066+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:26.066916+0000) 2020-02-06T01:02:27.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:27.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:27.051810+0000) 2020-02-06T01:02:27.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:27.319+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:27.319950+0000) 2020-02-06T01:02:27.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:27.319+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:27.320378+0000) 2020-02-06T01:02:27.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:27.319+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:27.320513+0000) 2020-02-06T01:02:29.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:29.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:29.695606+0000) 2020-02-06T01:02:29.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:29.943+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:29.943525+0000) 2020-02-06T01:02:30.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:30.353+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.354674+0000) 2020-02-06T01:02:30.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:30.354+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.355110+0000) 2020-02-06T01:02:30.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:30.354+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.355465+0000) 2020-02-06T01:02:30.357 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:30.355+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.356114+0000) 2020-02-06T01:02:30.357 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:30.355+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.356388+0000) 2020-02-06T01:02:30.358 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:30.356+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.357670+0000) 2020-02-06T01:02:30.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:30.355+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.357304+0000) 2020-02-06T01:02:30.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:30.356+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.357985+0000) 2020-02-06T01:02:30.359 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:30.357+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.358432+0000) 2020-02-06T01:02:30.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:30.358+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.359379+0000) 2020-02-06T01:02:30.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:30.357+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.359154+0000) 2020-02-06T01:02:30.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:30.359+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.360674+0000) 2020-02-06T01:02:30.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:30.359+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.360690+0000) 2020-02-06T01:02:30.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:30.365+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.366282+0000) 2020-02-06T01:02:30.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:30.374+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191663 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:30.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:30.546+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.547016+0000) 2020-02-06T01:02:30.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:30.805+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.805641+0000) 2020-02-06T01:02:30.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:30.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.826916+0000) 2020-02-06T01:02:30.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:30.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:30.827035+0000) 2020-02-06T01:02:32.930 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:32.928+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:32.930000+0000) 2020-02-06T01:02:33.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:33.159+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:33.160367+0000) 2020-02-06T01:02:34.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:34.880+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:34.881228+0000) 2020-02-06T01:02:35.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:35.379+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.380918+0000) 2020-02-06T01:02:35.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:35.380+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.382025+0000) 2020-02-06T01:02:35.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:35.380+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.382192+0000) 2020-02-06T01:02:35.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:35.381+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.382515+0000) 2020-02-06T01:02:35.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:35.381+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.382819+0000) 2020-02-06T01:02:35.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:35.382+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.383732+0000) 2020-02-06T01:02:35.385 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:35.382+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.384272+0000) 2020-02-06T01:02:35.385 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:35.383+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.384701+0000) 2020-02-06T01:02:35.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:35.383+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.385357+0000) 2020-02-06T01:02:35.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:35.383+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.385346+0000) 2020-02-06T01:02:35.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:35.385+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.386434+0000) 2020-02-06T01:02:35.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:35.385+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.387162+0000) 2020-02-06T01:02:35.388 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:35.386+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.387096+0000) 2020-02-06T01:02:35.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:35.391+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:35.392829+0000) 2020-02-06T01:02:35.403 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:35.401+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:36.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:36.066+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:36.067155+0000) 2020-02-06T01:02:37.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:37.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:37.052037+0000) 2020-02-06T01:02:37.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:37.319+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:37.320184+0000) 2020-02-06T01:02:37.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:37.319+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:37.320618+0000) 2020-02-06T01:02:37.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:37.319+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:37.320779+0000) 2020-02-06T01:02:39.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:39.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:39.695852+0000) 2020-02-06T01:02:39.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:39.943+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:39.943779+0000) 2020-02-06T01:02:40.409 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:40.407+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.408495+0000) 2020-02-06T01:02:40.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:40.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.409608+0000) 2020-02-06T01:02:40.411 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:40.409+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.410167+0000) 2020-02-06T01:02:40.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:40.409+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.410828+0000) 2020-02-06T01:02:40.412 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:40.410+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.411530+0000) 2020-02-06T01:02:40.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:40.410+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.411977+0000) 2020-02-06T01:02:40.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:40.411+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.412694+0000) 2020-02-06T01:02:40.413 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:40.412+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.413144+0000) 2020-02-06T01:02:40.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:40.412+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.413976+0000) 2020-02-06T01:02:40.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:40.412+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.413830+0000) 2020-02-06T01:02:40.415 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:40.412+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.413837+0000) 2020-02-06T01:02:40.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:40.413+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.415017+0000) 2020-02-06T01:02:40.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:40.412+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.414359+0000) 2020-02-06T01:02:40.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:40.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.420324+0000) 2020-02-06T01:02:40.431 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:40.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:40.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:40.546+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.547254+0000) 2020-02-06T01:02:40.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:40.805+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.805852+0000) 2020-02-06T01:02:40.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:40.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.827126+0000) 2020-02-06T01:02:40.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:40.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:40.827292+0000) 2020-02-06T01:02:42.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:42.929+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:42.930253+0000) 2020-02-06T01:02:43.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:43.160+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:43.160597+0000) 2020-02-06T01:02:44.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:44.889+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:44.889251+0000) 2020-02-06T01:02:45.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:45.435+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.436938+0000) 2020-02-06T01:02:45.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:45.436+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.437472+0000) 2020-02-06T01:02:45.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:45.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.437890+0000) 2020-02-06T01:02:45.440 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:45.438+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.439181+0000) 2020-02-06T01:02:45.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:45.438+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.439517+0000) 2020-02-06T01:02:45.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:45.438+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.440135+0000) 2020-02-06T01:02:45.441 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:45.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.440673+0000) 2020-02-06T01:02:45.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:45.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.440929+0000) 2020-02-06T01:02:45.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:45.440+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.441514+0000) 2020-02-06T01:02:45.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:45.440+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.441877+0000) 2020-02-06T01:02:45.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:45.441+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.442950+0000) 2020-02-06T01:02:45.445 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:45.441+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.442700+0000) 2020-02-06T01:02:45.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:45.442+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.443255+0000) 2020-02-06T01:02:45.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:45.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:45.447393+0000) 2020-02-06T01:02:45.457 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:45.455+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191786 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:46.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:46.066+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:46.067417+0000) 2020-02-06T01:02:47.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:47.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:47.052289+0000) 2020-02-06T01:02:47.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:47.319+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:47.320465+0000) 2020-02-06T01:02:47.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:47.319+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:47.320828+0000) 2020-02-06T01:02:47.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:47.319+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:47.321035+0000) 2020-02-06T01:02:49.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:49.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:49.696035+0000) 2020-02-06T01:02:49.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:49.943+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:49.944014+0000) 2020-02-06T01:02:50.464 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:50.461+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.461841+0000) 2020-02-06T01:02:50.464 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:50.461+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.462599+0000) 2020-02-06T01:02:50.465 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:50.463+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.463514+0000) 2020-02-06T01:02:50.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:50.464+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.464312+0000) 2020-02-06T01:02:50.466 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:50.464+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.464767+0000) 2020-02-06T01:02:50.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:50.464+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.465651+0000) 2020-02-06T01:02:50.467 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:50.465+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.466529+0000) 2020-02-06T01:02:50.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:50.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.466557+0000) 2020-02-06T01:02:50.468 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:50.466+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.466812+0000) 2020-02-06T01:02:50.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:50.465+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.467390+0000) 2020-02-06T01:02:50.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:50.466+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.467822+0000) 2020-02-06T01:02:50.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:50.468+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.468278+0000) 2020-02-06T01:02:50.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:50.467+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.468750+0000) 2020-02-06T01:02:50.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:50.473+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.474009+0000) 2020-02-06T01:02:50.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:50.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:50.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:50.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.547644+0000) 2020-02-06T01:02:50.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:50.806+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.806132+0000) 2020-02-06T01:02:50.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:50.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.827318+0000) 2020-02-06T01:02:50.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:50.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:50.827503+0000) 2020-02-06T01:02:51.833 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:02:51.840 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:02:52.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:52.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:52.930510+0000) 2020-02-06T01:02:53.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:53.159+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:53.160805+0000) 2020-02-06T01:02:54.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:54.899+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:54.900758+0000) 2020-02-06T01:02:55.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:55.490+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.491281+0000) 2020-02-06T01:02:55.493 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:55.491+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.491374+0000) 2020-02-06T01:02:55.494 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:55.490+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.491497+0000) 2020-02-06T01:02:55.494 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:02:55.491+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.492426+0000) 2020-02-06T01:02:55.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:55.491+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.492800+0000) 2020-02-06T01:02:55.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:02:55.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.494141+0000) 2020-02-06T01:02:55.495 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:02:55.493+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.494653+0000) 2020-02-06T01:02:55.495 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:02:55.494+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.495225+0000) 2020-02-06T01:02:55.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:02:55.495+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.496209+0000) 2020-02-06T01:02:55.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:55.496+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.496759+0000) 2020-02-06T01:02:55.497 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:55.496+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.496803+0000) 2020-02-06T01:02:55.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:02:55.496+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.496938+0000) 2020-02-06T01:02:55.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:55.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.500941+0000) 2020-02-06T01:02:55.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:55.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:55.501237+0000) 2020-02-06T01:02:55.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:02:55.510+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191866 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:02:56.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:02:56.067+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:56.067679+0000) 2020-02-06T01:02:57.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:02:57.051+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:57.052543+0000) 2020-02-06T01:02:57.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:02:57.320+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:57.320715+0000) 2020-02-06T01:02:57.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:02:57.320+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:57.321108+0000) 2020-02-06T01:02:57.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:02:57.320+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:57.321365+0000) 2020-02-06T01:02:59.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:59.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:59.696325+0000) 2020-02-06T01:02:59.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:02:59.943+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:02:59.944305+0000) 2020-02-06T01:03:00.517 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:00.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.516693+0000) 2020-02-06T01:03:00.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:00.516+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.517751+0000) 2020-02-06T01:03:00.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:00.518+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.519061+0000) 2020-02-06T01:03:00.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:00.517+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.518979+0000) 2020-02-06T01:03:00.520 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:00.518+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.519385+0000) 2020-02-06T01:03:00.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:00.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.519699+0000) 2020-02-06T01:03:00.521 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:00.519+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.520935+0000) 2020-02-06T01:03:00.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:00.520+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.521178+0000) 2020-02-06T01:03:00.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:00.521+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.521699+0000) 2020-02-06T01:03:00.523 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:00.521+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.521816+0000) 2020-02-06T01:03:00.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:00.521+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.522268+0000) 2020-02-06T01:03:00.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:00.522+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.522668+0000) 2020-02-06T01:03:00.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:00.522+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.523271+0000) 2020-02-06T01:03:00.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:00.525+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.526683+0000) 2020-02-06T01:03:00.539 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:00.537+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191907 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:00.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:00.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.547832+0000) 2020-02-06T01:03:00.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:00.805+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.806349+0000) 2020-02-06T01:03:00.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:00.826+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.827523+0000) 2020-02-06T01:03:00.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:00.826+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:00.827751+0000) 2020-02-06T01:03:02.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:02.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:02.930785+0000) 2020-02-06T01:03:03.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:03.160+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:03.161060+0000) 2020-02-06T01:03:04.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:04.908+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:04.908982+0000) 2020-02-06T01:03:05.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:05.544+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.545275+0000) 2020-02-06T01:03:05.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:05.544+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.545953+0000) 2020-02-06T01:03:05.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:05.546+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.546773+0000) 2020-02-06T01:03:05.548 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:05.546+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.547129+0000) 2020-02-06T01:03:05.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:05.547+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.547758+0000) 2020-02-06T01:03:05.549 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:05.546+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.547774+0000) 2020-02-06T01:03:05.549 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:05.547+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.548367+0000) 2020-02-06T01:03:05.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:05.548+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.548797+0000) 2020-02-06T01:03:05.550 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:05.548+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.549059+0000) 2020-02-06T01:03:05.550 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:05.548+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.548919+0000) 2020-02-06T01:03:05.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:05.549+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.549579+0000) 2020-02-06T01:03:05.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:05.548+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.549439+0000) 2020-02-06T01:03:05.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:05.550+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.551474+0000) 2020-02-06T01:03:05.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:05.551+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:05.552584+0000) 2020-02-06T01:03:05.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:05.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:06.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:06.067+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:06.067924+0000) 2020-02-06T01:03:07.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:07.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:07.052802+0000) 2020-02-06T01:03:07.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:07.320+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:07.321004+0000) 2020-02-06T01:03:07.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:07.320+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:07.321332+0000) 2020-02-06T01:03:07.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:07.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:07.321630+0000) 2020-02-06T01:03:09.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:09.695+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:09.696556+0000) 2020-02-06T01:03:09.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:09.943+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:09.944544+0000) 2020-02-06T01:03:10.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:10.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.548048+0000) 2020-02-06T01:03:10.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:10.568+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.569119+0000) 2020-02-06T01:03:10.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:10.569+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.570424+0000) 2020-02-06T01:03:10.572 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:10.570+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.571331+0000) 2020-02-06T01:03:10.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:10.571+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.572082+0000) 2020-02-06T01:03:10.574 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:10.572+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.573378+0000) 2020-02-06T01:03:10.575 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:10.573+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.574248+0000) 2020-02-06T01:03:10.575 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:10.573+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.574315+0000) 2020-02-06T01:03:10.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:10.574+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.575100+0000) 2020-02-06T01:03:10.576 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:10.574+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.575267+0000) 2020-02-06T01:03:10.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:10.575+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.575801+0000) 2020-02-06T01:03:10.577 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:10.575+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.576095+0000) 2020-02-06T01:03:10.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:10.575+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.576283+0000) 2020-02-06T01:03:10.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:10.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.579844+0000) 2020-02-06T01:03:10.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:10.579+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.580120+0000) 2020-02-06T01:03:10.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:10.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 191988 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:10.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:10.805+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.806559+0000) 2020-02-06T01:03:10.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:10.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.827722+0000) 2020-02-06T01:03:10.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:10.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:10.827976+0000) 2020-02-06T01:03:12.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:12.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:12.931041+0000) 2020-02-06T01:03:13.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:13.160+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:13.161365+0000) 2020-02-06T01:03:14.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:14.916+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:14.916953+0000) 2020-02-06T01:03:15.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:15.594+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.595066+0000) 2020-02-06T01:03:15.596 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:15.595+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.596149+0000) 2020-02-06T01:03:15.597 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:15.595+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.596551+0000) 2020-02-06T01:03:15.597 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:15.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.596994+0000) 2020-02-06T01:03:15.598 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:15.597+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.597610+0000) 2020-02-06T01:03:15.598 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:15.597+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.597940+0000) 2020-02-06T01:03:15.599 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:15.596+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.598006+0000) 2020-02-06T01:03:15.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:15.597+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.598771+0000) 2020-02-06T01:03:15.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:15.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.600415+0000) 2020-02-06T01:03:15.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:15.600+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.601370+0000) 2020-02-06T01:03:15.602 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:15.601+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.602046+0000) 2020-02-06T01:03:15.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:15.601+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.602153+0000) 2020-02-06T01:03:15.603 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:15.600+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.601211+0000) 2020-02-06T01:03:15.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:15.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:15.604244+0000) 2020-02-06T01:03:15.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:15.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192026 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:16.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:16.067+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:16.068194+0000) 2020-02-06T01:03:17.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:17.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:17.053038+0000) 2020-02-06T01:03:17.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:17.321+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:17.321250+0000) 2020-02-06T01:03:17.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:17.321+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:17.321633+0000) 2020-02-06T01:03:17.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:17.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:17.321820+0000) 2020-02-06T01:03:19.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:19.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:19.696784+0000) 2020-02-06T01:03:19.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:19.944+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:19.944771+0000) 2020-02-06T01:03:20.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:20.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.548268+0000) 2020-02-06T01:03:20.623 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:20.620+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.621941+0000) 2020-02-06T01:03:20.624 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:20.622+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.623142+0000) 2020-02-06T01:03:20.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:20.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.624385+0000) 2020-02-06T01:03:20.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:20.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.624397+0000) 2020-02-06T01:03:20.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:20.623+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.624262+0000) 2020-02-06T01:03:20.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:20.624+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.625106+0000) 2020-02-06T01:03:20.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:20.625+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.625755+0000) 2020-02-06T01:03:20.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:20.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.625914+0000) 2020-02-06T01:03:20.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:20.625+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.626255+0000) 2020-02-06T01:03:20.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:20.626+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.627331+0000) 2020-02-06T01:03:20.629 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:20.626+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.626881+0000) 2020-02-06T01:03:20.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:20.627+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.627924+0000) 2020-02-06T01:03:20.630 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:20.628+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.628829+0000) 2020-02-06T01:03:20.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:20.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.635061+0000) 2020-02-06T01:03:20.643 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:20.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:20.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:20.806+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.806772+0000) 2020-02-06T01:03:20.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:20.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.827941+0000) 2020-02-06T01:03:20.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:20.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:20.828181+0000) 2020-02-06T01:03:22.048 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:03:22.055 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:03:22.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:22.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:22.931289+0000) 2020-02-06T01:03:23.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:23.160+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:23.161601+0000) 2020-02-06T01:03:24.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:24.928+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:24.928856+0000) 2020-02-06T01:03:25.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:25.647+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.648071+0000) 2020-02-06T01:03:25.649 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:25.648+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.648944+0000) 2020-02-06T01:03:25.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:25.649+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.650699+0000) 2020-02-06T01:03:25.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:25.650+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.651548+0000) 2020-02-06T01:03:25.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:25.651+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.652495+0000) 2020-02-06T01:03:25.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:25.651+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.652623+0000) 2020-02-06T01:03:25.654 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:25.652+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.652971+0000) 2020-02-06T01:03:25.654 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:25.652+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.653105+0000) 2020-02-06T01:03:25.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:25.652+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.653221+0000) 2020-02-06T01:03:25.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:25.654+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.654446+0000) 2020-02-06T01:03:25.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:25.654+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.654803+0000) 2020-02-06T01:03:25.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:25.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.654174+0000) 2020-02-06T01:03:25.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:25.656+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.657589+0000) 2020-02-06T01:03:25.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:25.656+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:25.657843+0000) 2020-02-06T01:03:25.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:25.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:26.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:26.068+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:26.068468+0000) 2020-02-06T01:03:27.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:27.052+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:27.053312+0000) 2020-02-06T01:03:27.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:27.321+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:27.321509+0000) 2020-02-06T01:03:27.322 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:27.321+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:27.321852+0000) 2020-02-06T01:03:27.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:27.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:27.322069+0000) 2020-02-06T01:03:29.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:29.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:29.697023+0000) 2020-02-06T01:03:29.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:29.944+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:29.945022+0000) 2020-02-06T01:03:30.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:30.548+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.548528+0000) 2020-02-06T01:03:30.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:30.673+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.674251+0000) 2020-02-06T01:03:30.676 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:30.674+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.675471+0000) 2020-02-06T01:03:30.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:30.675+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.676110+0000) 2020-02-06T01:03:30.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:30.675+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.676997+0000) 2020-02-06T01:03:30.678 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:30.676+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.677695+0000) 2020-02-06T01:03:30.678 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:30.676+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.677893+0000) 2020-02-06T01:03:30.679 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:30.678+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.678421+0000) 2020-02-06T01:03:30.679 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:30.678+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.678812+0000) 2020-02-06T01:03:30.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:30.678+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.679250+0000) 2020-02-06T01:03:30.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:30.679+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.679518+0000) 2020-02-06T01:03:30.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:30.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.679864+0000) 2020-02-06T01:03:30.681 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:30.680+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.680484+0000) 2020-02-06T01:03:30.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:30.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.680819+0000) 2020-02-06T01:03:30.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:30.681+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.682189+0000) 2020-02-06T01:03:30.696 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:30.694+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:30.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:30.805+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.807004+0000) 2020-02-06T01:03:30.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:30.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.828185+0000) 2020-02-06T01:03:30.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:30.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:30.828393+0000) 2020-02-06T01:03:32.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:32.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:32.931564+0000) 2020-02-06T01:03:33.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:33.161+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:33.161814+0000) 2020-02-06T01:03:34.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:34.935+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:34.935662+0000) 2020-02-06T01:03:35.701 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:35.699+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.700637+0000) 2020-02-06T01:03:35.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:35.700+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.701388+0000) 2020-02-06T01:03:35.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:35.700+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.701847+0000) 2020-02-06T01:03:35.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:35.701+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.702273+0000) 2020-02-06T01:03:35.703 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:35.701+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.703201+0000) 2020-02-06T01:03:35.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:35.702+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.703768+0000) 2020-02-06T01:03:35.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:35.702+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.703751+0000) 2020-02-06T01:03:35.705 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:35.702+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.704363+0000) 2020-02-06T01:03:35.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:35.703+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.704272+0000) 2020-02-06T01:03:35.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:35.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.704758+0000) 2020-02-06T01:03:35.706 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:35.703+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.704923+0000) 2020-02-06T01:03:35.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:35.704+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.706118+0000) 2020-02-06T01:03:35.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:35.705+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.706981+0000) 2020-02-06T01:03:35.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:35.709+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:35.710722+0000) 2020-02-06T01:03:35.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:35.720+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:36.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:36.067+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:36.068759+0000) 2020-02-06T01:03:37.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:37.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:37.053541+0000) 2020-02-06T01:03:37.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:37.320+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:37.321770+0000) 2020-02-06T01:03:37.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:37.320+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:37.322094+0000) 2020-02-06T01:03:37.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:37.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:37.322413+0000) 2020-02-06T01:03:39.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:39.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:39.697307+0000) 2020-02-06T01:03:39.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:39.944+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:39.945237+0000) 2020-02-06T01:03:40.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:40.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.548757+0000) 2020-02-06T01:03:40.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:40.726+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.727739+0000) 2020-02-06T01:03:40.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:40.727+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.728244+0000) 2020-02-06T01:03:40.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:40.727+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.728665+0000) 2020-02-06T01:03:40.730 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:40.727+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.729163+0000) 2020-02-06T01:03:40.731 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:40.728+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.729584+0000) 2020-02-06T01:03:40.731 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:40.728+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.730041+0000) 2020-02-06T01:03:40.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:40.729+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.730485+0000) 2020-02-06T01:03:40.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:40.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.730949+0000) 2020-02-06T01:03:40.732 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:40.730+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.731303+0000) 2020-02-06T01:03:40.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:40.730+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.732362+0000) 2020-02-06T01:03:40.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:40.731+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.732682+0000) 2020-02-06T01:03:40.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:40.731+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.733241+0000) 2020-02-06T01:03:40.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:40.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.737250+0000) 2020-02-06T01:03:40.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:40.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.737553+0000) 2020-02-06T01:03:40.749 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:40.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:40.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:40.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.807223+0000) 2020-02-06T01:03:40.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:40.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.828354+0000) 2020-02-06T01:03:40.829 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:40.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:40.828608+0000) 2020-02-06T01:03:42.932 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:42.930+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:42.931846+0000) 2020-02-06T01:03:43.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:43.161+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:43.162109+0000) 2020-02-06T01:03:44.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:44.944+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:44.944996+0000) 2020-02-06T01:03:45.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:45.752+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.753525+0000) 2020-02-06T01:03:45.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:45.753+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.754178+0000) 2020-02-06T01:03:45.756 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:45.754+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.755927+0000) 2020-02-06T01:03:45.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:45.755+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.756435+0000) 2020-02-06T01:03:45.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:45.756+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.757223+0000) 2020-02-06T01:03:45.758 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:45.756+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.757700+0000) 2020-02-06T01:03:45.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:45.756+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.758140+0000) 2020-02-06T01:03:45.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:45.757+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.758249+0000) 2020-02-06T01:03:45.761 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:45.757+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.758870+0000) 2020-02-06T01:03:45.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:45.757+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.759177+0000) 2020-02-06T01:03:45.762 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:45.758+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.759603+0000) 2020-02-06T01:03:45.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:45.758+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.759883+0000) 2020-02-06T01:03:45.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:45.758+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.760114+0000) 2020-02-06T01:03:45.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:45.761+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:45.762828+0000) 2020-02-06T01:03:45.774 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:45.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192268 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:46.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:46.068+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:46.068997+0000) 2020-02-06T01:03:47.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:47.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:47.053782+0000) 2020-02-06T01:03:47.322 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:47.321+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:47.322004+0000) 2020-02-06T01:03:47.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:47.321+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:47.322329+0000) 2020-02-06T01:03:47.323 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:47.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:47.322663+0000) 2020-02-06T01:03:49.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:49.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:49.697535+0000) 2020-02-06T01:03:49.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:49.945+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:49.945484+0000) 2020-02-06T01:03:50.549 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:50.548+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.548985+0000) 2020-02-06T01:03:50.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:50.779+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.779261+0000) 2020-02-06T01:03:50.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:50.779+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.779506+0000) 2020-02-06T01:03:50.781 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:50.781+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.781171+0000) 2020-02-06T01:03:50.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:50.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.781744+0000) 2020-02-06T01:03:50.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:50.782+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.782290+0000) 2020-02-06T01:03:50.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:50.782+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.782055+0000) 2020-02-06T01:03:50.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:50.782+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.782558+0000) 2020-02-06T01:03:50.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:50.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.783224+0000) 2020-02-06T01:03:50.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:50.781+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.783331+0000) 2020-02-06T01:03:50.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:50.782+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.783894+0000) 2020-02-06T01:03:50.785 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:50.783+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.784704+0000) 2020-02-06T01:03:50.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:50.783+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.784890+0000) 2020-02-06T01:03:50.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:50.784+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.785720+0000) 2020-02-06T01:03:50.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:50.790+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.790608+0000) 2020-02-06T01:03:50.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:50.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:50.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:50.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.807416+0000) 2020-02-06T01:03:50.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:50.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.828569+0000) 2020-02-06T01:03:50.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:50.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:50.828783+0000) 2020-02-06T01:03:52.258 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:03:52.271 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:03:52.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:52.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:52.932099+0000) 2020-02-06T01:03:53.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:53.161+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:53.162355+0000) 2020-02-06T01:03:54.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:54.951+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:54.952928+0000) 2020-02-06T01:03:55.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:03:55.805+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.806367+0000) 2020-02-06T01:03:55.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:55.806+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.807630+0000) 2020-02-06T01:03:55.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:03:55.807+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.808183+0000) 2020-02-06T01:03:55.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:55.807+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.808726+0000) 2020-02-06T01:03:55.810 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:55.807+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.808902+0000) 2020-02-06T01:03:55.810 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:03:55.807+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.809352+0000) 2020-02-06T01:03:55.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:03:55.809+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.810220+0000) 2020-02-06T01:03:55.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:03:55.809+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.810582+0000) 2020-02-06T01:03:55.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:55.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.811200+0000) 2020-02-06T01:03:55.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:55.809+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.811354+0000) 2020-02-06T01:03:55.813 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:03:55.810+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.811829+0000) 2020-02-06T01:03:55.813 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:55.811+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.812644+0000) 2020-02-06T01:03:55.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:55.811+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.812717+0000) 2020-02-06T01:03:55.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:55.818+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:55.819115+0000) 2020-02-06T01:03:55.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:03:55.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:03:56.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:03:56.068+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:56.069234+0000) 2020-02-06T01:03:57.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:03:57.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:57.054018+0000) 2020-02-06T01:03:57.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:03:57.321+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:57.322259+0000) 2020-02-06T01:03:57.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:03:57.321+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:57.322553+0000) 2020-02-06T01:03:57.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:03:57.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:57.322914+0000) 2020-02-06T01:03:59.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:59.696+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:59.697779+0000) 2020-02-06T01:03:59.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:03:59.944+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:03:59.945665+0000) 2020-02-06T01:04:00.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:00.547+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.549245+0000) 2020-02-06T01:04:00.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:00.806+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.807605+0000) 2020-02-06T01:04:00.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:00.827+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.828758+0000) 2020-02-06T01:04:00.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:00.827+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.829001+0000) 2020-02-06T01:04:00.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:00.832+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.833273+0000) 2020-02-06T01:04:00.834 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:00.832+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.834039+0000) 2020-02-06T01:04:00.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:00.833+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.834693+0000) 2020-02-06T01:04:00.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:00.834+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.835195+0000) 2020-02-06T01:04:00.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:00.834+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.835635+0000) 2020-02-06T01:04:00.836 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:00.834+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.836014+0000) 2020-02-06T01:04:00.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:00.835+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.836683+0000) 2020-02-06T01:04:00.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:00.835+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.836911+0000) 2020-02-06T01:04:00.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:00.835+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.837140+0000) 2020-02-06T01:04:00.838 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:00.836+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.837620+0000) 2020-02-06T01:04:00.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:00.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.837986+0000) 2020-02-06T01:04:00.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:00.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.837679+0000) 2020-02-06T01:04:00.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:00.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.838287+0000) 2020-02-06T01:04:00.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:00.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:00.842304+0000) 2020-02-06T01:04:00.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:00.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192389 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:02.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:02.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:02.932328+0000) 2020-02-06T01:04:03.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:03.161+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:03.162591+0000) 2020-02-06T01:04:04.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:04.960+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:04.960918+0000) 2020-02-06T01:04:05.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:05.857+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.858516+0000) 2020-02-06T01:04:05.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:05.857+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.859055+0000) 2020-02-06T01:04:05.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:05.859+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.860960+0000) 2020-02-06T01:04:05.862 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:05.860+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.861738+0000) 2020-02-06T01:04:05.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:05.861+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.862560+0000) 2020-02-06T01:04:05.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:05.861+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.862853+0000) 2020-02-06T01:04:05.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:05.862+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.863346+0000) 2020-02-06T01:04:05.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:05.862+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.863605+0000) 2020-02-06T01:04:05.865 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:05.863+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.864525+0000) 2020-02-06T01:04:05.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:05.863+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.864587+0000) 2020-02-06T01:04:05.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:05.863+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.864953+0000) 2020-02-06T01:04:05.867 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:05.863+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.865401+0000) 2020-02-06T01:04:05.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:05.864+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.865901+0000) 2020-02-06T01:04:05.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:05.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:05.870112+0000) 2020-02-06T01:04:05.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:05.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:06.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:06.068+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:06.069510+0000) 2020-02-06T01:04:07.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:07.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:07.054295+0000) 2020-02-06T01:04:07.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:07.321+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:07.322507+0000) 2020-02-06T01:04:07.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:07.321+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:07.322798+0000) 2020-02-06T01:04:07.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:07.321+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:07.323146+0000) 2020-02-06T01:04:09.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:09.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:09.698006+0000) 2020-02-06T01:04:09.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:09.944+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:09.945878+0000) 2020-02-06T01:04:10.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:10.548+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.549503+0000) 2020-02-06T01:04:10.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:10.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.807841+0000) 2020-02-06T01:04:10.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:10.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.829069+0000) 2020-02-06T01:04:10.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:10.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.829231+0000) 2020-02-06T01:04:10.888 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:10.885+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.887337+0000) 2020-02-06T01:04:10.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:10.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.887957+0000) 2020-02-06T01:04:10.889 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:10.886+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.888353+0000) 2020-02-06T01:04:10.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:10.888+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.889369+0000) 2020-02-06T01:04:10.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:10.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.889594+0000) 2020-02-06T01:04:10.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:10.889+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.890111+0000) 2020-02-06T01:04:10.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:10.889+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.890518+0000) 2020-02-06T01:04:10.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:10.890+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.891173+0000) 2020-02-06T01:04:10.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:10.890+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.892034+0000) 2020-02-06T01:04:10.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:10.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.891913+0000) 2020-02-06T01:04:10.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:10.891+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.892606+0000) 2020-02-06T01:04:10.893 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:10.891+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.892812+0000) 2020-02-06T01:04:10.894 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:10.892+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.893125+0000) 2020-02-06T01:04:10.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:10.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:10.897146+0000) 2020-02-06T01:04:10.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:10.905+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:12.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:12.931+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:12.932563+0000) 2020-02-06T01:04:13.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:13.161+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:13.162841+0000) 2020-02-06T01:04:14.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:14.971+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:14.972390+0000) 2020-02-06T01:04:15.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:15.910+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.912028+0000) 2020-02-06T01:04:15.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:15.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.912747+0000) 2020-02-06T01:04:15.913 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:15.912+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.913244+0000) 2020-02-06T01:04:15.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:15.912+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.913690+0000) 2020-02-06T01:04:15.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:15.913+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.914345+0000) 2020-02-06T01:04:15.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:15.913+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.915115+0000) 2020-02-06T01:04:15.917 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:15.914+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.915897+0000) 2020-02-06T01:04:15.917 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:15.914+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.916152+0000) 2020-02-06T01:04:15.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:15.915+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.916576+0000) 2020-02-06T01:04:15.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:15.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.917361+0000) 2020-02-06T01:04:15.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:15.916+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.917937+0000) 2020-02-06T01:04:15.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:15.916+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.918358+0000) 2020-02-06T01:04:15.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:15.917+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.918559+0000) 2020-02-06T01:04:15.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:15.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:15.923512+0000) 2020-02-06T01:04:15.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:15.930+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:16.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:16.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:16.069783+0000) 2020-02-06T01:04:17.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:17.053+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:17.054554+0000) 2020-02-06T01:04:17.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:17.322+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:17.322759+0000) 2020-02-06T01:04:17.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:17.322+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:17.323033+0000) 2020-02-06T01:04:17.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:17.322+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:17.323350+0000) 2020-02-06T01:04:19.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:19.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:19.698283+0000) 2020-02-06T01:04:19.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:19.945+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:19.946125+0000) 2020-02-06T01:04:20.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:20.549+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.549741+0000) 2020-02-06T01:04:20.808 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:20.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.808111+0000) 2020-02-06T01:04:20.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:20.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.829358+0000) 2020-02-06T01:04:20.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:20.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.829472+0000) 2020-02-06T01:04:20.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:20.935+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.936270+0000) 2020-02-06T01:04:20.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:20.937+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.938433+0000) 2020-02-06T01:04:20.940 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:20.938+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.939699+0000) 2020-02-06T01:04:20.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:20.938+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.939964+0000) 2020-02-06T01:04:20.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:20.939+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.940501+0000) 2020-02-06T01:04:20.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:20.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.940710+0000) 2020-02-06T01:04:20.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:20.939+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.941129+0000) 2020-02-06T01:04:20.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:20.939+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.941344+0000) 2020-02-06T01:04:20.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:20.940+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.942203+0000) 2020-02-06T01:04:20.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:20.942+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.943295+0000) 2020-02-06T01:04:20.944 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:20.942+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.943395+0000) 2020-02-06T01:04:20.944 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:20.942+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.943630+0000) 2020-02-06T01:04:20.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:20.942+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.943852+0000) 2020-02-06T01:04:20.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:20.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:20.949239+0000) 2020-02-06T01:04:20.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:20.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:22.434 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:04:22.441 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:04:22.933 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:22.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:22.932835+0000) 2020-02-06T01:04:23.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:23.162+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:23.163146+0000) 2020-02-06T01:04:24.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:24.980+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:24.980749+0000) 2020-02-06T01:04:25.964 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:25.962+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.963351+0000) 2020-02-06T01:04:25.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:25.963+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.964408+0000) 2020-02-06T01:04:25.966 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:25.964+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.965335+0000) 2020-02-06T01:04:25.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:25.965+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.965912+0000) 2020-02-06T01:04:25.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:25.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.966148+0000) 2020-02-06T01:04:25.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:25.966+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.967109+0000) 2020-02-06T01:04:25.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:25.966+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.967175+0000) 2020-02-06T01:04:25.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:25.967+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.968160+0000) 2020-02-06T01:04:25.969 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:25.968+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.968548+0000) 2020-02-06T01:04:25.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:25.968+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.969144+0000) 2020-02-06T01:04:25.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:25.968+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.968687+0000) 2020-02-06T01:04:25.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:25.969+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.969666+0000) 2020-02-06T01:04:25.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:25.973+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.975031+0000) 2020-02-06T01:04:25.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:25.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:25.975388+0000) 2020-02-06T01:04:25.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:25.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:26.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:26.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:26.070019+0000) 2020-02-06T01:04:27.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:27.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:27.054784+0000) 2020-02-06T01:04:27.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:27.322+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:27.323008+0000) 2020-02-06T01:04:27.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:27.322+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:27.323269+0000) 2020-02-06T01:04:27.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:27.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:27.323601+0000) 2020-02-06T01:04:29.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:29.697+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:29.698485+0000) 2020-02-06T01:04:29.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:29.945+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:29.946356+0000) 2020-02-06T01:04:30.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:30.549+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.549990+0000) 2020-02-06T01:04:30.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:30.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.808375+0000) 2020-02-06T01:04:30.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:30.828+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.829608+0000) 2020-02-06T01:04:30.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:30.828+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.829696+0000) 2020-02-06T01:04:30.991 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:30.989+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.990400+0000) 2020-02-06T01:04:30.992 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:30.990+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.990968+0000) 2020-02-06T01:04:30.992 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:30.991+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.992230+0000) 2020-02-06T01:04:30.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:30.991+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.992653+0000) 2020-02-06T01:04:30.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:30.991+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.992961+0000) 2020-02-06T01:04:30.994 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:30.992+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.993527+0000) 2020-02-06T01:04:30.994 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:30.993+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.993580+0000) 2020-02-06T01:04:30.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:30.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.994635+0000) 2020-02-06T01:04:30.995 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:30.994+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.995003+0000) 2020-02-06T01:04:30.996 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:30.994+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.995510+0000) 2020-02-06T01:04:30.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:30.994+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.995769+0000) 2020-02-06T01:04:30.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:30.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.996588+0000) 2020-02-06T01:04:30.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:30.996+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:30.996418+0000) 2020-02-06T01:04:31.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:30.999+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:31.001050+0000) 2020-02-06T01:04:31.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:31.009+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192631 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:32.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:32.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:32.933084+0000) 2020-02-06T01:04:33.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:33.163+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:33.163391+0000) 2020-02-06T01:04:34.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:34.991+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:34.991840+0000) 2020-02-06T01:04:36.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:36.016+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.016592+0000) 2020-02-06T01:04:36.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:36.016+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.017216+0000) 2020-02-06T01:04:36.018 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:36.016+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.017487+0000) 2020-02-06T01:04:36.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:36.016+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.017735+0000) 2020-02-06T01:04:36.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:36.016+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.018038+0000) 2020-02-06T01:04:36.019 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:36.018+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.018572+0000) 2020-02-06T01:04:36.019 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:36.017+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.018945+0000) 2020-02-06T01:04:36.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:36.018+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.019974+0000) 2020-02-06T01:04:36.021 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:36.020+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.020346+0000) 2020-02-06T01:04:36.021 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:36.019+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.020878+0000) 2020-02-06T01:04:36.022 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:36.020+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.021167+0000) 2020-02-06T01:04:36.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:36.020+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.021139+0000) 2020-02-06T01:04:36.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:36.021+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.021771+0000) 2020-02-06T01:04:36.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:36.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.028476+0000) 2020-02-06T01:04:36.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:36.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:36.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:36.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:36.070253+0000) 2020-02-06T01:04:37.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:37.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:37.055023+0000) 2020-02-06T01:04:37.324 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:37.322+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:37.323241+0000) 2020-02-06T01:04:37.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:37.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:37.323509+0000) 2020-02-06T01:04:37.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:37.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:37.323844+0000) 2020-02-06T01:04:39.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:39.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:39.698751+0000) 2020-02-06T01:04:39.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:39.946+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:39.946584+0000) 2020-02-06T01:04:40.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:40.550+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:40.550277+0000) 2020-02-06T01:04:40.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:40.808+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:40.808606+0000) 2020-02-06T01:04:40.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:40.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:40.829879+0000) 2020-02-06T01:04:40.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:40.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:40.829926+0000) 2020-02-06T01:04:41.042 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:41.040+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.041685+0000) 2020-02-06T01:04:41.043 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:41.041+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.042341+0000) 2020-02-06T01:04:41.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:41.041+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.042652+0000) 2020-02-06T01:04:41.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:41.042+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.043258+0000) 2020-02-06T01:04:41.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:41.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.043512+0000) 2020-02-06T01:04:41.044 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:41.042+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.043906+0000) 2020-02-06T01:04:41.045 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:41.043+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.044126+0000) 2020-02-06T01:04:41.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:41.043+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.044312+0000) 2020-02-06T01:04:41.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:41.044+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.044671+0000) 2020-02-06T01:04:41.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:41.044+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.045180+0000) 2020-02-06T01:04:41.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:41.045+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.045857+0000) 2020-02-06T01:04:41.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:41.043+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.044457+0000) 2020-02-06T01:04:41.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:41.046+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.046869+0000) 2020-02-06T01:04:41.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:41.050+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:41.052070+0000) 2020-02-06T01:04:41.060 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:41.059+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:42.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:42.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:42.933355+0000) 2020-02-06T01:04:43.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:43.163+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:43.163604+0000) 2020-02-06T01:04:45.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:44.999+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:45.000143+0000) 2020-02-06T01:04:46.066 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:46.065+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.065484+0000) 2020-02-06T01:04:46.066 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:46.065+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.065974+0000) 2020-02-06T01:04:46.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:46.067+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.067231+0000) 2020-02-06T01:04:46.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:46.066+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.066893+0000) 2020-02-06T01:04:46.069 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:46.067+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.067596+0000) 2020-02-06T01:04:46.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:46.068+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.068144+0000) 2020-02-06T01:04:46.070 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:46.068+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.069045+0000) 2020-02-06T01:04:46.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:46.068+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.069336+0000) 2020-02-06T01:04:46.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:46.068+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.068789+0000) 2020-02-06T01:04:46.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:46.068+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.068902+0000) 2020-02-06T01:04:46.072 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:46.069+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.069824+0000) 2020-02-06T01:04:46.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:46.069+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.069800+0000) 2020-02-06T01:04:46.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:46.069+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.069901+0000) 2020-02-06T01:04:46.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:46.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.070391+0000) 2020-02-06T01:04:46.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:46.076+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:46.076785+0000) 2020-02-06T01:04:46.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:46.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192748 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:47.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:47.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:47.055279+0000) 2020-02-06T01:04:47.324 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:47.323+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:47.323492+0000) 2020-02-06T01:04:47.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:47.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:47.323757+0000) 2020-02-06T01:04:47.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:47.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:47.324043+0000) 2020-02-06T01:04:49.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:49.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:49.699017+0000) 2020-02-06T01:04:49.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:49.946+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:49.946855+0000) 2020-02-06T01:04:50.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:50.550+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:50.550508+0000) 2020-02-06T01:04:50.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:50.807+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:50.808837+0000) 2020-02-06T01:04:50.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:50.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:50.830151+0000) 2020-02-06T01:04:50.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:50.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:50.830192+0000) 2020-02-06T01:04:51.092 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:51.089+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.091031+0000) 2020-02-06T01:04:51.092 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:51.090+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.091501+0000) 2020-02-06T01:04:51.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:51.091+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.092210+0000) 2020-02-06T01:04:51.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:51.092+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.092466+0000) 2020-02-06T01:04:51.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:51.091+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.092732+0000) 2020-02-06T01:04:51.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:51.092+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.093277+0000) 2020-02-06T01:04:51.094 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:51.092+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.093349+0000) 2020-02-06T01:04:51.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:51.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.093792+0000) 2020-02-06T01:04:51.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:51.093+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.094299+0000) 2020-02-06T01:04:51.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:51.095+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.095764+0000) 2020-02-06T01:04:51.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:51.095+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.096190+0000) 2020-02-06T01:04:51.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:51.096+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.096787+0000) 2020-02-06T01:04:51.097 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:51.096+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.097064+0000) 2020-02-06T01:04:51.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:51.099+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:51.100730+0000) 2020-02-06T01:04:51.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:51.109+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:52.617 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:04:52.622 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:04:52.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:52.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:52.933573+0000) 2020-02-06T01:04:53.164 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:53.162+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:53.163859+0000) 2020-02-06T01:04:55.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:55.007+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:55.008202+0000) 2020-02-06T01:04:56.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:56.070+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.070629+0000) 2020-02-06T01:04:56.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:56.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.115821+0000) 2020-02-06T01:04:56.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:04:56.115+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.116958+0000) 2020-02-06T01:04:56.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:56.116+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.116723+0000) 2020-02-06T01:04:56.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:04:56.116+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.117995+0000) 2020-02-06T01:04:56.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:04:56.117+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.118850+0000) 2020-02-06T01:04:56.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:56.118+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.119132+0000) 2020-02-06T01:04:56.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:56.118+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.119447+0000) 2020-02-06T01:04:56.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:04:56.119+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.119837+0000) 2020-02-06T01:04:56.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:04:56.119+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.120049+0000) 2020-02-06T01:04:56.121 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:04:56.119+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.120282+0000) 2020-02-06T01:04:56.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:56.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.120912+0000) 2020-02-06T01:04:56.122 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:04:56.120+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.121189+0000) 2020-02-06T01:04:56.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:56.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.125329+0000) 2020-02-06T01:04:56.127 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:56.124+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:56.125669+0000) 2020-02-06T01:04:56.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:04:56.137+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:04:57.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:04:57.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:57.055492+0000) 2020-02-06T01:04:57.324 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:04:57.323+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:57.323716+0000) 2020-02-06T01:04:57.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:04:57.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:57.323962+0000) 2020-02-06T01:04:57.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:04:57.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:57.324256+0000) 2020-02-06T01:04:59.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:59.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:59.699279+0000) 2020-02-06T01:04:59.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:04:59.946+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:04:59.947163+0000) 2020-02-06T01:05:00.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:00.549+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:00.550759+0000) 2020-02-06T01:05:00.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:00.808+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:00.809103+0000) 2020-02-06T01:05:00.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:00.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:00.830392+0000) 2020-02-06T01:05:00.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:00.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:00.830417+0000) 2020-02-06T01:05:01.152 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:01.142+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.144331+0000) 2020-02-06T01:05:01.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:01.143+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.145159+0000) 2020-02-06T01:05:01.153 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:01.146+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.147784+0000) 2020-02-06T01:05:01.153 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:01.147+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.149038+0000) 2020-02-06T01:05:01.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:01.147+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.149189+0000) 2020-02-06T01:05:01.153 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:01.147+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.149262+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:01.142+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.143844+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:01.144+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.145187+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:01.144+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.146002+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:01.145+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.146749+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:01.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.147506+0000) 2020-02-06T01:05:01.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:01.148+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.149697+0000) 2020-02-06T01:05:01.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:01.147+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.148446+0000) 2020-02-06T01:05:01.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:01.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:01.153364+0000) 2020-02-06T01:05:01.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:01.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192870 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:02.934 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:02.932+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:02.933894+0000) 2020-02-06T01:05:03.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:03.163+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:03.164128+0000) 2020-02-06T01:05:05.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:05.015+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:05.016209+0000) 2020-02-06T01:05:06.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:06.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.070859+0000) 2020-02-06T01:05:06.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:06.167+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.168834+0000) 2020-02-06T01:05:06.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:06.168+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.170005+0000) 2020-02-06T01:05:06.171 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:06.169+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.170424+0000) 2020-02-06T01:05:06.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:06.170+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.171412+0000) 2020-02-06T01:05:06.172 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:06.170+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.171629+0000) 2020-02-06T01:05:06.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:06.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.172210+0000) 2020-02-06T01:05:06.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:06.171+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.172884+0000) 2020-02-06T01:05:06.174 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:06.171+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.172511+0000) 2020-02-06T01:05:06.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:06.171+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.173384+0000) 2020-02-06T01:05:06.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:06.172+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.173537+0000) 2020-02-06T01:05:06.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:06.173+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.174070+0000) 2020-02-06T01:05:06.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:06.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.174728+0000) 2020-02-06T01:05:06.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:06.174+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.175748+0000) 2020-02-06T01:05:06.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:06.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:06.181853+0000) 2020-02-06T01:05:06.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:06.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:07.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:07.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:07.055696+0000) 2020-02-06T01:05:07.324 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:07.323+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:07.323991+0000) 2020-02-06T01:05:07.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:07.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:07.324179+0000) 2020-02-06T01:05:07.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:07.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:07.324524+0000) 2020-02-06T01:05:09.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:09.698+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:09.699509+0000) 2020-02-06T01:05:09.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:09.946+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:09.947413+0000) 2020-02-06T01:05:10.551 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:10.550+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:10.551025+0000) 2020-02-06T01:05:10.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:10.808+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:10.809340+0000) 2020-02-06T01:05:10.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:10.829+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:10.830637+0000) 2020-02-06T01:05:10.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:10.829+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:10.830661+0000) 2020-02-06T01:05:11.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:11.195+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.196769+0000) 2020-02-06T01:05:11.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:11.195+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.197291+0000) 2020-02-06T01:05:11.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:11.196+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.197943+0000) 2020-02-06T01:05:11.199 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:11.197+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.198891+0000) 2020-02-06T01:05:11.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:11.197+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.198701+0000) 2020-02-06T01:05:11.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:11.196+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.198245+0000) 2020-02-06T01:05:11.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:11.198+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.199623+0000) 2020-02-06T01:05:11.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:11.198+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.199993+0000) 2020-02-06T01:05:11.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:11.199+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.200838+0000) 2020-02-06T01:05:11.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:11.199+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.201266+0000) 2020-02-06T01:05:11.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:11.199+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.200878+0000) 2020-02-06T01:05:11.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:11.200+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.201800+0000) 2020-02-06T01:05:11.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:11.200+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.201980+0000) 2020-02-06T01:05:11.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:11.205+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:11.207058+0000) 2020-02-06T01:05:11.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:11.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192951 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:12.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:12.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:12.934160+0000) 2020-02-06T01:05:13.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:13.163+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:13.164359+0000) 2020-02-06T01:05:15.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:15.023+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:15.024266+0000) 2020-02-06T01:05:16.071 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:16.070+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.071114+0000) 2020-02-06T01:05:16.221 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:16.219+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.220820+0000) 2020-02-06T01:05:16.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:16.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.222608+0000) 2020-02-06T01:05:16.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:16.222+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.223151+0000) 2020-02-06T01:05:16.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:16.222+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.223647+0000) 2020-02-06T01:05:16.224 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:16.222+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.223770+0000) 2020-02-06T01:05:16.224 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:16.222+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.224133+0000) 2020-02-06T01:05:16.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:16.222+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.224342+0000) 2020-02-06T01:05:16.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:16.223+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.224903+0000) 2020-02-06T01:05:16.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:16.224+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.225789+0000) 2020-02-06T01:05:16.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:16.224+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.225485+0000) 2020-02-06T01:05:16.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:16.224+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.225737+0000) 2020-02-06T01:05:16.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:16.225+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.226473+0000) 2020-02-06T01:05:16.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:16.225+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.227312+0000) 2020-02-06T01:05:16.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:16.231+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:16.232516+0000) 2020-02-06T01:05:16.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:16.239+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 192989 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:17.056 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:17.054+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:17.055911+0000) 2020-02-06T01:05:17.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:17.323+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:17.324228+0000) 2020-02-06T01:05:17.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:17.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:17.324401+0000) 2020-02-06T01:05:17.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:17.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:17.324763+0000) 2020-02-06T01:05:19.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:19.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:19.699741+0000) 2020-02-06T01:05:19.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:19.947+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:19.947666+0000) 2020-02-06T01:05:20.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:20.550+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:20.551275+0000) 2020-02-06T01:05:20.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:20.809+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:20.809571+0000) 2020-02-06T01:05:20.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:20.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:20.830874+0000) 2020-02-06T01:05:20.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:20.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:20.830899+0000) 2020-02-06T01:05:21.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:21.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.246596+0000) 2020-02-06T01:05:21.247 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:21.246+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.247292+0000) 2020-02-06T01:05:21.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:21.246+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.247971+0000) 2020-02-06T01:05:21.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:21.247+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.249113+0000) 2020-02-06T01:05:21.250 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:21.248+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.249449+0000) 2020-02-06T01:05:21.251 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:21.248+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.250021+0000) 2020-02-06T01:05:21.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:21.248+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.250128+0000) 2020-02-06T01:05:21.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:21.248+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.249932+0000) 2020-02-06T01:05:21.251 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:21.249+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.250119+0000) 2020-02-06T01:05:21.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:21.249+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.250626+0000) 2020-02-06T01:05:21.252 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:21.249+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.251394+0000) 2020-02-06T01:05:21.252 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:21.250+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.251113+0000) 2020-02-06T01:05:21.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:21.250+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.251410+0000) 2020-02-06T01:05:21.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:21.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:21.254451+0000) 2020-02-06T01:05:21.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:21.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193032 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:22.743 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:05:22.749 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:05:22.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:22.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:22.934502+0000) 2020-02-06T01:05:23.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:23.164+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:23.164602+0000) 2020-02-06T01:05:25.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:25.031+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:25.031933+0000) 2020-02-06T01:05:26.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:26.069+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.071353+0000) 2020-02-06T01:05:26.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:26.273+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.274435+0000) 2020-02-06T01:05:26.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:26.274+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.275778+0000) 2020-02-06T01:05:26.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:26.275+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.276377+0000) 2020-02-06T01:05:26.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:26.275+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.276541+0000) 2020-02-06T01:05:26.277 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:26.275+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.276656+0000) 2020-02-06T01:05:26.278 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:26.276+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.277066+0000) 2020-02-06T01:05:26.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:26.276+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.277583+0000) 2020-02-06T01:05:26.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:26.276+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.278027+0000) 2020-02-06T01:05:26.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:26.277+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.278272+0000) 2020-02-06T01:05:26.279 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:26.276+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.278244+0000) 2020-02-06T01:05:26.280 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:26.278+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.279302+0000) 2020-02-06T01:05:26.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:26.277+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.279013+0000) 2020-02-06T01:05:26.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:26.278+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.279957+0000) 2020-02-06T01:05:26.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:26.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:26.284230+0000) 2020-02-06T01:05:26.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:26.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:27.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:27.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:27.056120+0000) 2020-02-06T01:05:27.325 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:27.323+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:27.324496+0000) 2020-02-06T01:05:27.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:27.323+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:27.324692+0000) 2020-02-06T01:05:27.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:27.323+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:27.325063+0000) 2020-02-06T01:05:29.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:29.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:29.699978+0000) 2020-02-06T01:05:29.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:29.947+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:29.947890+0000) 2020-02-06T01:05:30.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:30.550+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:30.551521+0000) 2020-02-06T01:05:30.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:30.808+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:30.809801+0000) 2020-02-06T01:05:30.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:30.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:30.831134+0000) 2020-02-06T01:05:30.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:30.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:30.831135+0000) 2020-02-06T01:05:31.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:31.299+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.300274+0000) 2020-02-06T01:05:31.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:31.299+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.301111+0000) 2020-02-06T01:05:31.302 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:31.300+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.301916+0000) 2020-02-06T01:05:31.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:31.301+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.302340+0000) 2020-02-06T01:05:31.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:31.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.303628+0000) 2020-02-06T01:05:31.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:31.303+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.304316+0000) 2020-02-06T01:05:31.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:31.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.304577+0000) 2020-02-06T01:05:31.305 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:31.303+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.304992+0000) 2020-02-06T01:05:31.306 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:31.304+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.305687+0000) 2020-02-06T01:05:31.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:31.304+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.306047+0000) 2020-02-06T01:05:31.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:31.304+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.305783+0000) 2020-02-06T01:05:31.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:31.304+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.306336+0000) 2020-02-06T01:05:31.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:31.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.309560+0000) 2020-02-06T01:05:31.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:31.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:31.309825+0000) 2020-02-06T01:05:31.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:31.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193113 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:32.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:32.933+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:32.934708+0000) 2020-02-06T01:05:33.165 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:33.164+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:33.164841+0000) 2020-02-06T01:05:35.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:35.038+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:35.039847+0000) 2020-02-06T01:05:36.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:36.070+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.071645+0000) 2020-02-06T01:05:36.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:36.324+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.325524+0000) 2020-02-06T01:05:36.327 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:36.326+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.327137+0000) 2020-02-06T01:05:36.328 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:36.326+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.327487+0000) 2020-02-06T01:05:36.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:36.326+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.327895+0000) 2020-02-06T01:05:36.328 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:36.327+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.328228+0000) 2020-02-06T01:05:36.329 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:36.327+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.328737+0000) 2020-02-06T01:05:36.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:36.328+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.329558+0000) 2020-02-06T01:05:36.331 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:36.328+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.330188+0000) 2020-02-06T01:05:36.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:36.329+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.330713+0000) 2020-02-06T01:05:36.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:36.329+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.330783+0000) 2020-02-06T01:05:36.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:36.330+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.331459+0000) 2020-02-06T01:05:36.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:36.330+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.332316+0000) 2020-02-06T01:05:36.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:36.336+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.337291+0000) 2020-02-06T01:05:36.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:36.336+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:36.337616+0000) 2020-02-06T01:05:36.349 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:36.346+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:37.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:37.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:37.056374+0000) 2020-02-06T01:05:37.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:37.324+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:37.324746+0000) 2020-02-06T01:05:37.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:37.324+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:37.324930+0000) 2020-02-06T01:05:37.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:37.324+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:37.325301+0000) 2020-02-06T01:05:39.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:39.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:39.700213+0000) 2020-02-06T01:05:39.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:39.947+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:39.948147+0000) 2020-02-06T01:05:40.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:40.551+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:40.551766+0000) 2020-02-06T01:05:40.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:40.810+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:40.810063+0000) 2020-02-06T01:05:40.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:40.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:40.831416+0000) 2020-02-06T01:05:40.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:40.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:40.831414+0000) 2020-02-06T01:05:41.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:41.354+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.354375+0000) 2020-02-06T01:05:41.356 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:41.355+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.355248+0000) 2020-02-06T01:05:41.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:41.355+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.355711+0000) 2020-02-06T01:05:41.357 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:41.354+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.356116+0000) 2020-02-06T01:05:41.357 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:41.356+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.356843+0000) 2020-02-06T01:05:41.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:41.355+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.356701+0000) 2020-02-06T01:05:41.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:41.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.357798+0000) 2020-02-06T01:05:41.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:41.356+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.357710+0000) 2020-02-06T01:05:41.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:41.358+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.358697+0000) 2020-02-06T01:05:41.360 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:41.359+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.359339+0000) 2020-02-06T01:05:41.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:41.357+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.359234+0000) 2020-02-06T01:05:41.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:41.358+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.359803+0000) 2020-02-06T01:05:41.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:41.358+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.360011+0000) 2020-02-06T01:05:41.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:41.365+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:41.365245+0000) 2020-02-06T01:05:41.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:41.374+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:42.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:42.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:42.934951+0000) 2020-02-06T01:05:43.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:43.165+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:43.165060+0000) 2020-02-06T01:05:45.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:45.046+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:45.047750+0000) 2020-02-06T01:05:46.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:46.071+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.071896+0000) 2020-02-06T01:05:46.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:46.380+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.381881+0000) 2020-02-06T01:05:46.383 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:46.381+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.382406+0000) 2020-02-06T01:05:46.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:46.381+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.383010+0000) 2020-02-06T01:05:46.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:46.382+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.383763+0000) 2020-02-06T01:05:46.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:46.382+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.384259+0000) 2020-02-06T01:05:46.385 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:46.383+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.384545+0000) 2020-02-06T01:05:46.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:46.384+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.385203+0000) 2020-02-06T01:05:46.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:46.384+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.385659+0000) 2020-02-06T01:05:46.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:46.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.386302+0000) 2020-02-06T01:05:46.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:46.385+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.387114+0000) 2020-02-06T01:05:46.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:46.386+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.387645+0000) 2020-02-06T01:05:46.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:46.386+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.387816+0000) 2020-02-06T01:05:46.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:46.386+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.388012+0000) 2020-02-06T01:05:46.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:46.390+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:46.391619+0000) 2020-02-06T01:05:46.404 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:46.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193233 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:47.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:47.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:47.056584+0000) 2020-02-06T01:05:47.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:47.324+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:47.324979+0000) 2020-02-06T01:05:47.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:47.324+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:47.325174+0000) 2020-02-06T01:05:47.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:47.324+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:47.325509+0000) 2020-02-06T01:05:49.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:49.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:49.700452+0000) 2020-02-06T01:05:49.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:49.947+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:49.948413+0000) 2020-02-06T01:05:50.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:50.551+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:50.552068+0000) 2020-02-06T01:05:50.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:50.809+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:50.810375+0000) 2020-02-06T01:05:50.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:50.830+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:50.831677+0000) 2020-02-06T01:05:50.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:50.830+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:50.831682+0000) 2020-02-06T01:05:51.410 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:51.409+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.409743+0000) 2020-02-06T01:05:51.411 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:51.410+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.411072+0000) 2020-02-06T01:05:51.412 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:51.410+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.411501+0000) 2020-02-06T01:05:51.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:51.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.411869+0000) 2020-02-06T01:05:51.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:51.411+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.412148+0000) 2020-02-06T01:05:51.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:51.412+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.412601+0000) 2020-02-06T01:05:51.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:51.413+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.413583+0000) 2020-02-06T01:05:51.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:51.412+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.413374+0000) 2020-02-06T01:05:51.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:51.414+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.415063+0000) 2020-02-06T01:05:51.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:51.414+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.415579+0000) 2020-02-06T01:05:51.416 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:51.415+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.416236+0000) 2020-02-06T01:05:51.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:51.414+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.416025+0000) 2020-02-06T01:05:51.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:51.417+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.418787+0000) 2020-02-06T01:05:51.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:51.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:51.419109+0000) 2020-02-06T01:05:51.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:51.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:52.883 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:05:52.890 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:05:52.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:52.934+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:52.935169+0000) 2020-02-06T01:05:53.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:53.164+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:53.165350+0000) 2020-02-06T01:05:55.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:55.055+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:55.055887+0000) 2020-02-06T01:05:56.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:56.071+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.072119+0000) 2020-02-06T01:05:56.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:56.434+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.434820+0000) 2020-02-06T01:05:56.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:56.435+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.435491+0000) 2020-02-06T01:05:56.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:05:56.436+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.437203+0000) 2020-02-06T01:05:56.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:05:56.436+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.437620+0000) 2020-02-06T01:05:56.440 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:05:56.436+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.437743+0000) 2020-02-06T01:05:56.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:56.437+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.438401+0000) 2020-02-06T01:05:56.441 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:05:56.437+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.438560+0000) 2020-02-06T01:05:56.442 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:56.438+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.439809+0000) 2020-02-06T01:05:56.442 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:05:56.439+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.440314+0000) 2020-02-06T01:05:56.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:56.440+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.440964+0000) 2020-02-06T01:05:56.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:56.440+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.441332+0000) 2020-02-06T01:05:56.443 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:05:56.441+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.441940+0000) 2020-02-06T01:05:56.444 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:05:56.441+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.442031+0000) 2020-02-06T01:05:56.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:56.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:56.446432+0000) 2020-02-06T01:05:56.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:05:56.456+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:05:57.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:05:57.055+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:57.056812+0000) 2020-02-06T01:05:57.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:05:57.324+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:57.325218+0000) 2020-02-06T01:05:57.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:05:57.324+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:57.325373+0000) 2020-02-06T01:05:57.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:05:57.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:57.325704+0000) 2020-02-06T01:05:59.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:59.699+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:59.700695+0000) 2020-02-06T01:05:59.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:05:59.947+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:05:59.948671+0000) 2020-02-06T01:06:00.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:00.551+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:00.552304+0000) 2020-02-06T01:06:00.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:00.809+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:00.810610+0000) 2020-02-06T01:06:00.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:00.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:00.831960+0000) 2020-02-06T01:06:00.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:00.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:00.831955+0000) 2020-02-06T01:06:01.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:01.462+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.462523+0000) 2020-02-06T01:06:01.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:01.463+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.464302+0000) 2020-02-06T01:06:01.465 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:01.463+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.464378+0000) 2020-02-06T01:06:01.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:01.463+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.464825+0000) 2020-02-06T01:06:01.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:01.464+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.465360+0000) 2020-02-06T01:06:01.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:01.464+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.465807+0000) 2020-02-06T01:06:01.466 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:01.465+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.466159+0000) 2020-02-06T01:06:01.467 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:01.465+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.466819+0000) 2020-02-06T01:06:01.468 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:01.467+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.467490+0000) 2020-02-06T01:06:01.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:01.467+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.467688+0000) 2020-02-06T01:06:01.469 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:01.468+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.468513+0000) 2020-02-06T01:06:01.469 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:01.468+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.469246+0000) 2020-02-06T01:06:01.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:01.468+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.469422+0000) 2020-02-06T01:06:01.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:01.470+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:01.471646+0000) 2020-02-06T01:06:01.483 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:01.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193357 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:02.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:02.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:02.935471+0000) 2020-02-06T01:06:03.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:03.164+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:03.165586+0000) 2020-02-06T01:06:05.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:05.062+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:05.063960+0000) 2020-02-06T01:06:06.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:06.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.072377+0000) 2020-02-06T01:06:06.490 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:06.488+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.488791+0000) 2020-02-06T01:06:06.490 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:06.488+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.488565+0000) 2020-02-06T01:06:06.491 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:06.488+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.489800+0000) 2020-02-06T01:06:06.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:06.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.490557+0000) 2020-02-06T01:06:06.492 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:06.490+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.491541+0000) 2020-02-06T01:06:06.493 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:06.491+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.492357+0000) 2020-02-06T01:06:06.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:06.491+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.492432+0000) 2020-02-06T01:06:06.494 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:06.492+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.493133+0000) 2020-02-06T01:06:06.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:06.493+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.493712+0000) 2020-02-06T01:06:06.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:06.491+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.493043+0000) 2020-02-06T01:06:06.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:06.493+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.493931+0000) 2020-02-06T01:06:06.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:06.494+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.494496+0000) 2020-02-06T01:06:06.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:06.493+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.495042+0000) 2020-02-06T01:06:06.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:06.496+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:06.497208+0000) 2020-02-06T01:06:06.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:06.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:07.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:07.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:07.057021+0000) 2020-02-06T01:06:07.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:07.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:07.325510+0000) 2020-02-06T01:06:07.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:07.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:07.325618+0000) 2020-02-06T01:06:07.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:07.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:07.325947+0000) 2020-02-06T01:06:09.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:09.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:09.700937+0000) 2020-02-06T01:06:09.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:09.948+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:09.948874+0000) 2020-02-06T01:06:10.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:10.552+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:10.552535+0000) 2020-02-06T01:06:10.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:10.810+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:10.810843+0000) 2020-02-06T01:06:10.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:10.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:10.832212+0000) 2020-02-06T01:06:10.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:10.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:10.832265+0000) 2020-02-06T01:06:11.515 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:11.513+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.513918+0000) 2020-02-06T01:06:11.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:11.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.515389+0000) 2020-02-06T01:06:11.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:11.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.516872+0000) 2020-02-06T01:06:11.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:11.516+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.517302+0000) 2020-02-06T01:06:11.518 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:11.515+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.516278+0000) 2020-02-06T01:06:11.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:11.516+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.517895+0000) 2020-02-06T01:06:11.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:11.516+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.517976+0000) 2020-02-06T01:06:11.519 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:11.517+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.518422+0000) 2020-02-06T01:06:11.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:11.519+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.519555+0000) 2020-02-06T01:06:11.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:11.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.520387+0000) 2020-02-06T01:06:11.521 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:11.520+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.520425+0000) 2020-02-06T01:06:11.521 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:11.519+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.520262+0000) 2020-02-06T01:06:11.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:11.520+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.520826+0000) 2020-02-06T01:06:11.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:11.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:11.525523+0000) 2020-02-06T01:06:11.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:11.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:12.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:12.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:12.935736+0000) 2020-02-06T01:06:13.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:13.164+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:13.165877+0000) 2020-02-06T01:06:15.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:15.071+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:15.071880+0000) 2020-02-06T01:06:16.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:16.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.072660+0000) 2020-02-06T01:06:16.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:16.541+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.541691+0000) 2020-02-06T01:06:16.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:16.541+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.542653+0000) 2020-02-06T01:06:16.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:16.542+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.543778+0000) 2020-02-06T01:06:16.544 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:16.543+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.544340+0000) 2020-02-06T01:06:16.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:16.544+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.545134+0000) 2020-02-06T01:06:16.546 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:16.545+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.546036+0000) 2020-02-06T01:06:16.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:16.545+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.546541+0000) 2020-02-06T01:06:16.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:16.546+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.547072+0000) 2020-02-06T01:06:16.548 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:16.547+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.547594+0000) 2020-02-06T01:06:16.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:16.547+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.548001+0000) 2020-02-06T01:06:16.549 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:16.547+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.548380+0000) 2020-02-06T01:06:16.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:16.547+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.548588+0000) 2020-02-06T01:06:16.549 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:16.547+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.548842+0000) 2020-02-06T01:06:16.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:16.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:16.551922+0000) 2020-02-06T01:06:16.563 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:16.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:17.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:17.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:17.057297+0000) 2020-02-06T01:06:17.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:17.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:17.325756+0000) 2020-02-06T01:06:17.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:17.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:17.325828+0000) 2020-02-06T01:06:17.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:17.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:17.326142+0000) 2020-02-06T01:06:19.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:19.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:19.701177+0000) 2020-02-06T01:06:19.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:19.948+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:19.949132+0000) 2020-02-06T01:06:20.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:20.552+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:20.552768+0000) 2020-02-06T01:06:20.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:20.810+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:20.811107+0000) 2020-02-06T01:06:20.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:20.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:20.832454+0000) 2020-02-06T01:06:20.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:20.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:20.832504+0000) 2020-02-06T01:06:21.568 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:21.566+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.567958+0000) 2020-02-06T01:06:21.569 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:21.567+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.568360+0000) 2020-02-06T01:06:21.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:21.568+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.569006+0000) 2020-02-06T01:06:21.570 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:21.568+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.570034+0000) 2020-02-06T01:06:21.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:21.569+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.570443+0000) 2020-02-06T01:06:21.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:21.570+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.570848+0000) 2020-02-06T01:06:21.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:21.570+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.571137+0000) 2020-02-06T01:06:21.573 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:21.571+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.572469+0000) 2020-02-06T01:06:21.574 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:21.571+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.571616+0000) 2020-02-06T01:06:21.574 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:21.572+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.572455+0000) 2020-02-06T01:06:21.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:21.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.572967+0000) 2020-02-06T01:06:21.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:21.572+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.573412+0000) 2020-02-06T01:06:21.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:21.573+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.574011+0000) 2020-02-06T01:06:21.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:21.577+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:21.578322+0000) 2020-02-06T01:06:21.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:21.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:22.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:22.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:22.935980+0000) 2020-02-06T01:06:23.044 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:06:23.051 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:06:23.166 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:23.165+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:23.166127+0000) 2020-02-06T01:06:25.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:25.079+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:25.080153+0000) 2020-02-06T01:06:26.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:26.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.072898+0000) 2020-02-06T01:06:26.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:26.593+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.594426+0000) 2020-02-06T01:06:26.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:26.595+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.595481+0000) 2020-02-06T01:06:26.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:26.597+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.597667+0000) 2020-02-06T01:06:26.602 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:26.594+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.595697+0000) 2020-02-06T01:06:26.602 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:26.596+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.597600+0000) 2020-02-06T01:06:26.603 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:26.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.598065+0000) 2020-02-06T01:06:26.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:26.597+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.598532+0000) 2020-02-06T01:06:26.604 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:26.598+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.599032+0000) 2020-02-06T01:06:26.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:26.599+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.600015+0000) 2020-02-06T01:06:26.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:26.600+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.600540+0000) 2020-02-06T01:06:26.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:26.600+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.600839+0000) 2020-02-06T01:06:26.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:26.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.599816+0000) 2020-02-06T01:06:26.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:26.603+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.604578+0000) 2020-02-06T01:06:26.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:26.603+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:26.604847+0000) 2020-02-06T01:06:26.617 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:26.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193564 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:27.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:27.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:27.057536+0000) 2020-02-06T01:06:27.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:27.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:27.325961+0000) 2020-02-06T01:06:27.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:27.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:27.326047+0000) 2020-02-06T01:06:27.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:27.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:27.326331+0000) 2020-02-06T01:06:29.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:29.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:29.701405+0000) 2020-02-06T01:06:29.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:29.949+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:29.949361+0000) 2020-02-06T01:06:30.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:30.551+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:30.553023+0000) 2020-02-06T01:06:30.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:30.811+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:30.811363+0000) 2020-02-06T01:06:30.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:30.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:30.832695+0000) 2020-02-06T01:06:30.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:30.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:30.832760+0000) 2020-02-06T01:06:31.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:31.619+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.620513+0000) 2020-02-06T01:06:31.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:31.620+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.621932+0000) 2020-02-06T01:06:31.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:31.621+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.622607+0000) 2020-02-06T01:06:31.623 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:31.622+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.623202+0000) 2020-02-06T01:06:31.624 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:31.622+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.623900+0000) 2020-02-06T01:06:31.625 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:31.623+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.624380+0000) 2020-02-06T01:06:31.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:31.623+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.624681+0000) 2020-02-06T01:06:31.626 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:31.624+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.625237+0000) 2020-02-06T01:06:31.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:31.624+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.626021+0000) 2020-02-06T01:06:31.627 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:31.625+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.626558+0000) 2020-02-06T01:06:31.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:31.625+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.626947+0000) 2020-02-06T01:06:31.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:31.625+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.627373+0000) 2020-02-06T01:06:31.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:31.628+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.630014+0000) 2020-02-06T01:06:31.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:31.629+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:31.630332+0000) 2020-02-06T01:06:31.641 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:31.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193609 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:32.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:32.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:32.936241+0000) 2020-02-06T01:06:33.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:33.166+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:33.166380+0000) 2020-02-06T01:06:35.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:35.088+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:35.088369+0000) 2020-02-06T01:06:36.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:36.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.073155+0000) 2020-02-06T01:06:36.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:36.646+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.646278+0000) 2020-02-06T01:06:36.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:36.646+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.647010+0000) 2020-02-06T01:06:36.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:36.646+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.647620+0000) 2020-02-06T01:06:36.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:36.647+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.648667+0000) 2020-02-06T01:06:36.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:36.649+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.649283+0000) 2020-02-06T01:06:36.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:36.647+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.649072+0000) 2020-02-06T01:06:36.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:36.650+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.650185+0000) 2020-02-06T01:06:36.651 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:36.650+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.650736+0000) 2020-02-06T01:06:36.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:36.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.651272+0000) 2020-02-06T01:06:36.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:36.650+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.651980+0000) 2020-02-06T01:06:36.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:36.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.651807+0000) 2020-02-06T01:06:36.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:36.651+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.652539+0000) 2020-02-06T01:06:36.653 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:36.651+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.652909+0000) 2020-02-06T01:06:36.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:36.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:36.658629+0000) 2020-02-06T01:06:36.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:36.665+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193647 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:37.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:37.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:37.057747+0000) 2020-02-06T01:06:37.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:37.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:37.326195+0000) 2020-02-06T01:06:37.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:37.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:37.326289+0000) 2020-02-06T01:06:37.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:37.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:37.326525+0000) 2020-02-06T01:06:39.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:39.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:39.701638+0000) 2020-02-06T01:06:39.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:39.948+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:39.949598+0000) 2020-02-06T01:06:40.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:40.552+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:40.553279+0000) 2020-02-06T01:06:40.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:40.810+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:40.811611+0000) 2020-02-06T01:06:40.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:40.831+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:40.832942+0000) 2020-02-06T01:06:40.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:40.831+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:40.833003+0000) 2020-02-06T01:06:41.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:41.670+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.671864+0000) 2020-02-06T01:06:41.673 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:41.671+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.672232+0000) 2020-02-06T01:06:41.673 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:41.671+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.673212+0000) 2020-02-06T01:06:41.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:41.672+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.673573+0000) 2020-02-06T01:06:41.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:41.673+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.674909+0000) 2020-02-06T01:06:41.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:41.674+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.675161+0000) 2020-02-06T01:06:41.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:41.674+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.676012+0000) 2020-02-06T01:06:41.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:41.674+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.675886+0000) 2020-02-06T01:06:41.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:41.675+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.676383+0000) 2020-02-06T01:06:41.677 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:41.675+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.676439+0000) 2020-02-06T01:06:41.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:41.675+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.676690+0000) 2020-02-06T01:06:41.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:41.675+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.677167+0000) 2020-02-06T01:06:41.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:41.676+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.678242+0000) 2020-02-06T01:06:41.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:41.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:41.683917+0000) 2020-02-06T01:06:41.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:41.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193693 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:42.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:42.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:42.936510+0000) 2020-02-06T01:06:43.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:43.165+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:43.166610+0000) 2020-02-06T01:06:45.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:45.095+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:45.096223+0000) 2020-02-06T01:06:46.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:46.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.073407+0000) 2020-02-06T01:06:46.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:46.698+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.700223+0000) 2020-02-06T01:06:46.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:46.699+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.700656+0000) 2020-02-06T01:06:46.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:46.700+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.701246+0000) 2020-02-06T01:06:46.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:46.700+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.701793+0000) 2020-02-06T01:06:46.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:46.701+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.702314+0000) 2020-02-06T01:06:46.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:46.701+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.702382+0000) 2020-02-06T01:06:46.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:46.702+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.703564+0000) 2020-02-06T01:06:46.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:46.702+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.703699+0000) 2020-02-06T01:06:46.705 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:46.703+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.704312+0000) 2020-02-06T01:06:46.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:46.703+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.704835+0000) 2020-02-06T01:06:46.706 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:46.704+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.705584+0000) 2020-02-06T01:06:46.706 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:46.704+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.705912+0000) 2020-02-06T01:06:46.706 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:46.704+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.706224+0000) 2020-02-06T01:06:46.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:46.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:46.709756+0000) 2020-02-06T01:06:46.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:46.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:47.058 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:47.056+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:47.057976+0000) 2020-02-06T01:06:47.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:47.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:47.326424+0000) 2020-02-06T01:06:47.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:47.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:47.326541+0000) 2020-02-06T01:06:47.327 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:47.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:47.326794+0000) 2020-02-06T01:06:49.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:49.700+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:49.701867+0000) 2020-02-06T01:06:49.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:49.948+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:49.949830+0000) 2020-02-06T01:06:50.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:50.552+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:50.553535+0000) 2020-02-06T01:06:50.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:50.810+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:50.811848+0000) 2020-02-06T01:06:50.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:50.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:50.833189+0000) 2020-02-06T01:06:50.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:50.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:50.833272+0000) 2020-02-06T01:06:51.730 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:51.726+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.727337+0000) 2020-02-06T01:06:51.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:51.726+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.727832+0000) 2020-02-06T01:06:51.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:51.727+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.728280+0000) 2020-02-06T01:06:51.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:51.728+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.730149+0000) 2020-02-06T01:06:51.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:51.728+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.729829+0000) 2020-02-06T01:06:51.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:51.729+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.730634+0000) 2020-02-06T01:06:51.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:51.729+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.730962+0000) 2020-02-06T01:06:51.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:51.729+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.731248+0000) 2020-02-06T01:06:51.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:51.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.731804+0000) 2020-02-06T01:06:51.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:51.731+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.732220+0000) 2020-02-06T01:06:51.733 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:51.731+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.732451+0000) 2020-02-06T01:06:51.733 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:51.730+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.732112+0000) 2020-02-06T01:06:51.734 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:51.732+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.733439+0000) 2020-02-06T01:06:51.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:51.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:51.734693+0000) 2020-02-06T01:06:51.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:51.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:52.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:52.935+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:52.936744+0000) 2020-02-06T01:06:53.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:53.165+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:53.166834+0000) 2020-02-06T01:06:53.176 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:06:53.182 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:06:55.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:55.103+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:55.104553+0000) 2020-02-06T01:06:56.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:56.072+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.073686+0000) 2020-02-06T01:06:56.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:06:56.751+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.752485+0000) 2020-02-06T01:06:56.753 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:06:56.751+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.753002+0000) 2020-02-06T01:06:56.754 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:56.752+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.753810+0000) 2020-02-06T01:06:56.754 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:06:56.752+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.754374+0000) 2020-02-06T01:06:56.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:56.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.755191+0000) 2020-02-06T01:06:56.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:06:56.755+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.756249+0000) 2020-02-06T01:06:56.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:56.754+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.755535+0000) 2020-02-06T01:06:56.757 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:06:56.756+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.757206+0000) 2020-02-06T01:06:56.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:56.756+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.757773+0000) 2020-02-06T01:06:56.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:56.757+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.758417+0000) 2020-02-06T01:06:56.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:56.757+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.758520+0000) 2020-02-06T01:06:56.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:06:56.757+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.758579+0000) 2020-02-06T01:06:56.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:06:56.757+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.759043+0000) 2020-02-06T01:06:56.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:56.758+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:56.759813+0000) 2020-02-06T01:06:56.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:06:56.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:06:57.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:06:57.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:57.058141+0000) 2020-02-06T01:06:57.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:06:57.325+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:57.326643+0000) 2020-02-06T01:06:57.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:06:57.325+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:57.326753+0000) 2020-02-06T01:06:57.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:06:57.325+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:57.327040+0000) 2020-02-06T01:06:59.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:59.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:59.702128+0000) 2020-02-06T01:06:59.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:06:59.949+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:06:59.950112+0000) 2020-02-06T01:07:00.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:00.553+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:00.553766+0000) 2020-02-06T01:07:00.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:00.811+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:00.812132+0000) 2020-02-06T01:07:00.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:00.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:00.833506+0000) 2020-02-06T01:07:00.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:00.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:00.833545+0000) 2020-02-06T01:07:01.781 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:01.779+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.780403+0000) 2020-02-06T01:07:01.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:01.780+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.781464+0000) 2020-02-06T01:07:01.782 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:01.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.781552+0000) 2020-02-06T01:07:01.783 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:01.780+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.782313+0000) 2020-02-06T01:07:01.783 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:01.781+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.782260+0000) 2020-02-06T01:07:01.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:01.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.782425+0000) 2020-02-06T01:07:01.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:01.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.782678+0000) 2020-02-06T01:07:01.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:01.781+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.783209+0000) 2020-02-06T01:07:01.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:01.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.783340+0000) 2020-02-06T01:07:01.785 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:01.781+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.782672+0000) 2020-02-06T01:07:01.785 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:01.783+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.784540+0000) 2020-02-06T01:07:01.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:01.783+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.784948+0000) 2020-02-06T01:07:01.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:01.783+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.784672+0000) 2020-02-06T01:07:01.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:01.788+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:01.789088+0000) 2020-02-06T01:07:01.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:01.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193860 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:02.937 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:02.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:02.937011+0000) 2020-02-06T01:07:03.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:03.166+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:03.167071+0000) 2020-02-06T01:07:05.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:05.111+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:05.112132+0000) 2020-02-06T01:07:06.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:06.073+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.073932+0000) 2020-02-06T01:07:06.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:06.804+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.805757+0000) 2020-02-06T01:07:06.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:06.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.806031+0000) 2020-02-06T01:07:06.807 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:06.805+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.806523+0000) 2020-02-06T01:07:06.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:06.805+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.806997+0000) 2020-02-06T01:07:06.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:06.806+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.807283+0000) 2020-02-06T01:07:06.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:06.806+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.808022+0000) 2020-02-06T01:07:06.809 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:06.806+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.808365+0000) 2020-02-06T01:07:06.809 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:06.807+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.808734+0000) 2020-02-06T01:07:06.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:06.807+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.809279+0000) 2020-02-06T01:07:06.811 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:06.809+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.810502+0000) 2020-02-06T01:07:06.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:06.809+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.811250+0000) 2020-02-06T01:07:06.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:06.811+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.812122+0000) 2020-02-06T01:07:06.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:06.815+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.816802+0000) 2020-02-06T01:07:06.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:06.816+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:06.817131+0000) 2020-02-06T01:07:06.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:06.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193900 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:07.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:07.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:07.058354+0000) 2020-02-06T01:07:07.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:07.326+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:07.326871+0000) 2020-02-06T01:07:07.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:07.326+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:07.326964+0000) 2020-02-06T01:07:07.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:07.326+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:07.327274+0000) 2020-02-06T01:07:09.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:09.701+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:09.702337+0000) 2020-02-06T01:07:09.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:09.949+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:09.950373+0000) 2020-02-06T01:07:10.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:10.553+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:10.554001+0000) 2020-02-06T01:07:10.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:10.811+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:10.812383+0000) 2020-02-06T01:07:10.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:10.832+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:10.833767+0000) 2020-02-06T01:07:10.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:10.832+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:10.833788+0000) 2020-02-06T01:07:11.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:11.829+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.830766+0000) 2020-02-06T01:07:11.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:11.830+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.831752+0000) 2020-02-06T01:07:11.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:11.831+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.832357+0000) 2020-02-06T01:07:11.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:11.831+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.832728+0000) 2020-02-06T01:07:11.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:11.831+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.833030+0000) 2020-02-06T01:07:11.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:11.832+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.833267+0000) 2020-02-06T01:07:11.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:11.832+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.833715+0000) 2020-02-06T01:07:11.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:11.833+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.834114+0000) 2020-02-06T01:07:11.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:11.833+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.834541+0000) 2020-02-06T01:07:11.836 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:11.833+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.834886+0000) 2020-02-06T01:07:11.836 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:11.833+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.835090+0000) 2020-02-06T01:07:11.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:11.833+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.835281+0000) 2020-02-06T01:07:11.836 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:11.834+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.836048+0000) 2020-02-06T01:07:11.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:11.839+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:11.840800+0000) 2020-02-06T01:07:11.852 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:11.850+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:12.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:12.936+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:12.937279+0000) 2020-02-06T01:07:13.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:13.166+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:13.167338+0000) 2020-02-06T01:07:15.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:15.119+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:15.120555+0000) 2020-02-06T01:07:16.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:16.073+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.074198+0000) 2020-02-06T01:07:16.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:16.856+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.857604+0000) 2020-02-06T01:07:16.859 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:16.857+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.858682+0000) 2020-02-06T01:07:16.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:16.857+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.858975+0000) 2020-02-06T01:07:16.860 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:16.858+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.859373+0000) 2020-02-06T01:07:16.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:16.860+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.861181+0000) 2020-02-06T01:07:16.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:16.860+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.861703+0000) 2020-02-06T01:07:16.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:16.858+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.860093+0000) 2020-02-06T01:07:16.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:16.859+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.860484+0000) 2020-02-06T01:07:16.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:16.860+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.861710+0000) 2020-02-06T01:07:16.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:16.861+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.862530+0000) 2020-02-06T01:07:16.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:16.861+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.863228+0000) 2020-02-06T01:07:16.864 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:16.861+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.863257+0000) 2020-02-06T01:07:16.865 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:16.862+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.864095+0000) 2020-02-06T01:07:16.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:16.866+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:16.867524+0000) 2020-02-06T01:07:16.880 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:16.878+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 193983 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:17.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:17.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:17.058548+0000) 2020-02-06T01:07:17.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:17.326+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:17.327083+0000) 2020-02-06T01:07:17.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:17.326+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:17.327185+0000) 2020-02-06T01:07:17.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:17.326+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:17.327539+0000) 2020-02-06T01:07:19.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:19.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:19.702587+0000) 2020-02-06T01:07:19.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:19.950+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:19.950595+0000) 2020-02-06T01:07:20.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:20.553+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:20.554248+0000) 2020-02-06T01:07:20.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:20.811+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:20.812633+0000) 2020-02-06T01:07:20.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:20.833+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:20.834049+0000) 2020-02-06T01:07:20.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:20.833+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:20.834057+0000) 2020-02-06T01:07:21.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:21.884+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.885635+0000) 2020-02-06T01:07:21.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:21.885+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.886066+0000) 2020-02-06T01:07:21.888 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:21.886+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.887509+0000) 2020-02-06T01:07:21.888 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:21.887+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.888174+0000) 2020-02-06T01:07:21.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:21.887+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.888506+0000) 2020-02-06T01:07:21.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:21.888+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.888900+0000) 2020-02-06T01:07:21.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:21.888+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.889240+0000) 2020-02-06T01:07:21.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:21.888+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.889633+0000) 2020-02-06T01:07:21.890 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:21.889+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.890126+0000) 2020-02-06T01:07:21.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:21.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.890796+0000) 2020-02-06T01:07:21.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:21.889+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.890941+0000) 2020-02-06T01:07:21.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:21.891+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.891622+0000) 2020-02-06T01:07:21.892 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:21.891+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.891511+0000) 2020-02-06T01:07:21.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:21.895+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:21.896907+0000) 2020-02-06T01:07:21.907 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:21.906+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:22.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:22.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:22.937538+0000) 2020-02-06T01:07:23.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:23.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:23.167588+0000) 2020-02-06T01:07:23.340 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:07:23.346 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:07:25.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:25.128+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:25.128618+0000) 2020-02-06T01:07:26.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:26.074+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.074499+0000) 2020-02-06T01:07:26.914 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:26.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.912958+0000) 2020-02-06T01:07:26.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:26.913+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.914525+0000) 2020-02-06T01:07:26.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:26.914+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.914896+0000) 2020-02-06T01:07:26.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:26.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.915492+0000) 2020-02-06T01:07:26.916 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:26.914+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.915709+0000) 2020-02-06T01:07:26.917 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:26.915+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.916221+0000) 2020-02-06T01:07:26.917 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:26.915+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.915950+0000) 2020-02-06T01:07:26.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:26.915+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.916787+0000) 2020-02-06T01:07:26.918 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:26.917+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.917860+0000) 2020-02-06T01:07:26.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:26.916+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.917400+0000) 2020-02-06T01:07:26.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:26.918+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.918548+0000) 2020-02-06T01:07:26.919 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:26.918+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.918908+0000) 2020-02-06T01:07:26.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:26.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.918991+0000) 2020-02-06T01:07:26.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:26.923+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:26.924263+0000) 2020-02-06T01:07:26.934 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:26.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:27.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:27.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:27.058763+0000) 2020-02-06T01:07:27.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:27.326+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:27.327312+0000) 2020-02-06T01:07:27.328 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:27.327+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:27.327478+0000) 2020-02-06T01:07:27.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:27.327+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:27.327770+0000) 2020-02-06T01:07:29.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:29.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:29.702810+0000) 2020-02-06T01:07:29.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:29.950+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:29.950825+0000) 2020-02-06T01:07:30.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:30.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:30.554501+0000) 2020-02-06T01:07:30.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:30.812+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:30.812869+0000) 2020-02-06T01:07:30.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:30.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:30.834308+0000) 2020-02-06T01:07:30.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:30.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:30.834360+0000) 2020-02-06T01:07:31.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:31.938+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.938673+0000) 2020-02-06T01:07:31.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:31.939+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.939741+0000) 2020-02-06T01:07:31.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:31.940+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.940131+0000) 2020-02-06T01:07:31.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:31.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.941398+0000) 2020-02-06T01:07:31.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:31.942+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.942364+0000) 2020-02-06T01:07:31.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:31.941+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.941852+0000) 2020-02-06T01:07:31.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:31.943+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.943444+0000) 2020-02-06T01:07:31.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:31.943+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.943714+0000) 2020-02-06T01:07:31.945 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:31.942+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.943117+0000) 2020-02-06T01:07:31.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:31.943+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.943893+0000) 2020-02-06T01:07:31.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:31.944+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.944719+0000) 2020-02-06T01:07:31.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:31.943+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.944335+0000) 2020-02-06T01:07:31.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:31.945+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.945556+0000) 2020-02-06T01:07:31.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:31.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:31.948360+0000) 2020-02-06T01:07:31.961 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:31.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194112 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:32.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:32.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:32.937831+0000) 2020-02-06T01:07:33.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:33.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:33.167828+0000) 2020-02-06T01:07:35.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:35.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:35.136801+0000) 2020-02-06T01:07:36.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:36.074+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.074784+0000) 2020-02-06T01:07:36.966 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:36.964+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.965834+0000) 2020-02-06T01:07:36.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:36.965+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.966659+0000) 2020-02-06T01:07:36.967 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:36.966+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.967282+0000) 2020-02-06T01:07:36.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:36.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.967817+0000) 2020-02-06T01:07:36.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:36.967+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.968370+0000) 2020-02-06T01:07:36.969 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:36.968+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.968801+0000) 2020-02-06T01:07:36.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:36.968+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.969089+0000) 2020-02-06T01:07:36.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:36.968+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.969389+0000) 2020-02-06T01:07:36.971 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:36.969+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.970245+0000) 2020-02-06T01:07:36.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:36.970+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.970829+0000) 2020-02-06T01:07:36.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:36.970+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.971253+0000) 2020-02-06T01:07:36.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:36.972+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.972480+0000) 2020-02-06T01:07:36.973 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:36.971+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.972359+0000) 2020-02-06T01:07:36.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:36.976+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:36.977606+0000) 2020-02-06T01:07:36.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:36.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:37.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:37.057+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:37.058971+0000) 2020-02-06T01:07:37.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:37.327+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:37.327534+0000) 2020-02-06T01:07:37.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:37.327+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:37.327708+0000) 2020-02-06T01:07:37.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:37.327+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:37.327959+0000) 2020-02-06T01:07:39.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:39.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:39.703124+0000) 2020-02-06T01:07:39.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:39.950+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:39.951106+0000) 2020-02-06T01:07:40.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:40.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:40.554728+0000) 2020-02-06T01:07:40.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:40.812+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:40.813129+0000) 2020-02-06T01:07:40.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:40.833+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:40.834556+0000) 2020-02-06T01:07:40.835 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:40.833+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:40.834585+0000) 2020-02-06T01:07:41.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:41.992+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.993653+0000) 2020-02-06T01:07:41.995 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:41.993+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.993979+0000) 2020-02-06T01:07:41.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:41.993+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.994376+0000) 2020-02-06T01:07:41.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:41.993+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.994680+0000) 2020-02-06T01:07:41.995 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:41.994+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.995150+0000) 2020-02-06T01:07:41.996 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:41.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.995577+0000) 2020-02-06T01:07:41.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:41.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.995617+0000) 2020-02-06T01:07:41.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:41.995+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.996227+0000) 2020-02-06T01:07:41.997 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:41.995+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.996539+0000) 2020-02-06T01:07:41.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:41.996+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.996966+0000) 2020-02-06T01:07:41.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:41.996+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.997382+0000) 2020-02-06T01:07:41.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:41.997+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.997886+0000) 2020-02-06T01:07:41.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:41.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:41.997322+0000) 2020-02-06T01:07:42.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:42.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:42.001329+0000) 2020-02-06T01:07:42.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:42.013+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:42.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:42.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:42.938065+0000) 2020-02-06T01:07:43.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:43.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:43.168121+0000) 2020-02-06T01:07:45.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:45.142+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:45.143665+0000) 2020-02-06T01:07:46.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:46.074+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:46.075017+0000) 2020-02-06T01:07:47.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:47.017+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.019018+0000) 2020-02-06T01:07:47.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:47.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.020796+0000) 2020-02-06T01:07:47.022 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:47.020+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.021231+0000) 2020-02-06T01:07:47.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:47.021+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.021524+0000) 2020-02-06T01:07:47.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:47.021+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.022294+0000) 2020-02-06T01:07:47.023 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:47.021+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.022375+0000) 2020-02-06T01:07:47.023 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:47.022+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.022741+0000) 2020-02-06T01:07:47.024 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:47.021+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.022770+0000) 2020-02-06T01:07:47.024 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:47.022+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.022999+0000) 2020-02-06T01:07:47.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:47.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.023508+0000) 2020-02-06T01:07:47.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:47.023+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.023964+0000) 2020-02-06T01:07:47.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:47.023+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.024298+0000) 2020-02-06T01:07:47.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:47.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.024571+0000) 2020-02-06T01:07:47.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:47.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.028276+0000) 2020-02-06T01:07:47.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:47.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194232 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:47.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:47.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.059190+0000) 2020-02-06T01:07:47.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:47.327+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.327737+0000) 2020-02-06T01:07:47.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:47.327+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.327909+0000) 2020-02-06T01:07:47.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:47.327+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:47.328180+0000) 2020-02-06T01:07:49.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:49.702+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:49.703362+0000) 2020-02-06T01:07:49.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:49.950+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:49.951385+0000) 2020-02-06T01:07:50.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:50.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:50.554986+0000) 2020-02-06T01:07:50.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:50.812+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:50.813302+0000) 2020-02-06T01:07:50.835 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:50.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:50.834793+0000) 2020-02-06T01:07:50.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:50.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:50.834823+0000) 2020-02-06T01:07:52.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:52.043+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.044397+0000) 2020-02-06T01:07:52.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:52.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.045151+0000) 2020-02-06T01:07:52.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:52.045+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.046358+0000) 2020-02-06T01:07:52.048 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:52.046+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.047408+0000) 2020-02-06T01:07:52.048 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:52.046+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.046837+0000) 2020-02-06T01:07:52.048 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:52.047+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.048219+0000) 2020-02-06T01:07:52.049 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:52.048+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.048737+0000) 2020-02-06T01:07:52.050 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:52.049+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.049790+0000) 2020-02-06T01:07:52.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:52.049+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.049535+0000) 2020-02-06T01:07:52.051 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:52.048+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.049848+0000) 2020-02-06T01:07:52.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:52.049+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.050729+0000) 2020-02-06T01:07:52.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:52.049+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.050949+0000) 2020-02-06T01:07:52.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:52.050+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.051288+0000) 2020-02-06T01:07:52.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:52.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.056244+0000) 2020-02-06T01:07:52.065 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:52.064+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194273 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:52.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:52.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:52.938342+0000) 2020-02-06T01:07:53.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:53.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:53.168349+0000) 2020-02-06T01:07:53.474 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:07:53.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:07:55.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:55.150+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:55.151892+0000) 2020-02-06T01:07:56.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:56.075+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:56.075287+0000) 2020-02-06T01:07:57.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:57.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.059406+0000) 2020-02-06T01:07:57.072 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:07:57.070+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.071519+0000) 2020-02-06T01:07:57.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:57.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.071901+0000) 2020-02-06T01:07:57.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:57.071+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.072650+0000) 2020-02-06T01:07:57.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:57.070+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.072382+0000) 2020-02-06T01:07:57.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:07:57.070+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.072220+0000) 2020-02-06T01:07:57.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:07:57.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.073133+0000) 2020-02-06T01:07:57.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:07:57.072+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.073578+0000) 2020-02-06T01:07:57.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:07:57.072+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.074242+0000) 2020-02-06T01:07:57.075 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:07:57.073+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.074146+0000) 2020-02-06T01:07:57.075 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:07:57.073+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.074650+0000) 2020-02-06T01:07:57.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:57.073+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.074873+0000) 2020-02-06T01:07:57.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:07:57.074+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.075403+0000) 2020-02-06T01:07:57.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:57.074+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.075894+0000) 2020-02-06T01:07:57.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:57.079+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.080768+0000) 2020-02-06T01:07:57.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:07:57.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:07:57.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:07:57.326+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.327985+0000) 2020-02-06T01:07:57.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:07:57.326+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.328124+0000) 2020-02-06T01:07:57.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:07:57.326+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:57.328378+0000) 2020-02-06T01:07:59.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:59.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:59.703623+0000) 2020-02-06T01:07:59.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:07:59.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:07:59.951665+0000) 2020-02-06T01:08:00.556 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:00.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:00.555266+0000) 2020-02-06T01:08:00.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:00.812+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:00.813531+0000) 2020-02-06T01:08:00.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:00.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:00.835037+0000) 2020-02-06T01:08:00.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:00.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:00.835114+0000) 2020-02-06T01:08:02.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:02.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.098622+0000) 2020-02-06T01:08:02.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:02.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.099227+0000) 2020-02-06T01:08:02.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:02.098+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.099621+0000) 2020-02-06T01:08:02.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:02.099+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.100398+0000) 2020-02-06T01:08:02.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:02.099+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.100795+0000) 2020-02-06T01:08:02.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:02.100+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.101410+0000) 2020-02-06T01:08:02.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:02.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.101599+0000) 2020-02-06T01:08:02.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:02.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.102011+0000) 2020-02-06T01:08:02.103 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:02.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.102445+0000) 2020-02-06T01:08:02.103 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:02.101+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.102804+0000) 2020-02-06T01:08:02.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:02.101+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.103042+0000) 2020-02-06T01:08:02.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:02.101+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.102874+0000) 2020-02-06T01:08:02.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:02.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.103296+0000) 2020-02-06T01:08:02.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:02.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.105299+0000) 2020-02-06T01:08:02.120 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:02.118+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194354 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:02.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:02.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:02.938592+0000) 2020-02-06T01:08:03.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:03.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:03.168553+0000) 2020-02-06T01:08:05.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:05.159+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:05.160317+0000) 2020-02-06T01:08:06.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:06.074+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:06.075594+0000) 2020-02-06T01:08:07.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:07.058+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.059643+0000) 2020-02-06T01:08:07.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:07.124+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.125879+0000) 2020-02-06T01:08:07.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:07.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.126589+0000) 2020-02-06T01:08:07.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:07.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.127514+0000) 2020-02-06T01:08:07.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:07.126+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.127255+0000) 2020-02-06T01:08:07.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:07.126+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.127848+0000) 2020-02-06T01:08:07.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:07.126+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.128224+0000) 2020-02-06T01:08:07.130 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:07.127+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.128340+0000) 2020-02-06T01:08:07.130 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:07.127+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.128576+0000) 2020-02-06T01:08:07.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:07.127+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.128949+0000) 2020-02-06T01:08:07.131 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:07.127+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.129199+0000) 2020-02-06T01:08:07.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:07.127+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.129386+0000) 2020-02-06T01:08:07.131 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:07.128+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.129754+0000) 2020-02-06T01:08:07.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:07.128+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.130314+0000) 2020-02-06T01:08:07.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:07.134+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.135580+0000) 2020-02-06T01:08:07.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:07.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:07.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:07.327+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.328233+0000) 2020-02-06T01:08:07.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:07.327+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.328347+0000) 2020-02-06T01:08:07.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:07.327+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:07.328627+0000) 2020-02-06T01:08:09.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:09.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:09.703877+0000) 2020-02-06T01:08:09.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:09.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:09.951896+0000) 2020-02-06T01:08:10.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:10.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:10.555514+0000) 2020-02-06T01:08:10.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:10.813+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:10.813762+0000) 2020-02-06T01:08:10.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:10.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:10.835306+0000) 2020-02-06T01:08:10.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:10.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:10.835396+0000) 2020-02-06T01:08:12.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:12.148+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.149808+0000) 2020-02-06T01:08:12.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:12.151+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.152227+0000) 2020-02-06T01:08:12.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:12.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.152556+0000) 2020-02-06T01:08:12.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:12.151+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.152990+0000) 2020-02-06T01:08:12.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:12.152+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.153324+0000) 2020-02-06T01:08:12.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:12.152+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.153793+0000) 2020-02-06T01:08:12.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:12.152+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.154405+0000) 2020-02-06T01:08:12.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:12.154+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.155410+0000) 2020-02-06T01:08:12.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:12.154+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.155745+0000) 2020-02-06T01:08:12.157 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:12.155+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.156716+0000) 2020-02-06T01:08:12.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:12.155+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.157176+0000) 2020-02-06T01:08:12.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:12.156+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.157715+0000) 2020-02-06T01:08:12.158 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:12.155+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.156741+0000) 2020-02-06T01:08:12.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:12.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.160004+0000) 2020-02-06T01:08:12.175 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:12.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194435 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:12.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:12.937+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:12.938829+0000) 2020-02-06T01:08:13.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:13.167+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:13.168780+0000) 2020-02-06T01:08:15.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:15.167+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:15.168856+0000) 2020-02-06T01:08:16.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:16.074+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:16.075934+0000) 2020-02-06T01:08:17.060 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:17.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.059868+0000) 2020-02-06T01:08:17.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:17.179+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.180053+0000) 2020-02-06T01:08:17.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:17.180+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.181794+0000) 2020-02-06T01:08:17.186 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:17.181+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.182410+0000) 2020-02-06T01:08:17.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:17.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.181970+0000) 2020-02-06T01:08:17.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:17.181+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.182913+0000) 2020-02-06T01:08:17.186 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:17.182+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.183653+0000) 2020-02-06T01:08:17.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:17.183+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.184305+0000) 2020-02-06T01:08:17.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:17.183+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.185287+0000) 2020-02-06T01:08:17.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:17.183+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.185070+0000) 2020-02-06T01:08:17.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:17.185+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.186773+0000) 2020-02-06T01:08:17.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:17.185+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.187031+0000) 2020-02-06T01:08:17.188 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:17.185+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.187155+0000) 2020-02-06T01:08:17.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:17.189+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.190189+0000) 2020-02-06T01:08:17.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:17.190+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.191471+0000) 2020-02-06T01:08:17.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:17.200+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194476 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:17.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:17.327+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.328493+0000) 2020-02-06T01:08:17.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:17.327+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.328607+0000) 2020-02-06T01:08:17.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:17.327+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:17.328815+0000) 2020-02-06T01:08:19.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:19.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:19.704145+0000) 2020-02-06T01:08:19.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:19.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:19.952131+0000) 2020-02-06T01:08:20.556 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:20.554+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:20.555762+0000) 2020-02-06T01:08:20.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:20.812+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:20.814013+0000) 2020-02-06T01:08:20.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:20.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:20.835529+0000) 2020-02-06T01:08:20.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:20.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:20.835639+0000) 2020-02-06T01:08:22.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:22.206+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.207443+0000) 2020-02-06T01:08:22.210 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:22.208+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.209321+0000) 2020-02-06T01:08:22.213 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:22.208+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.209693+0000) 2020-02-06T01:08:22.215 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:22.208+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.209997+0000) 2020-02-06T01:08:22.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:22.209+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.210490+0000) 2020-02-06T01:08:22.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:22.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.210749+0000) 2020-02-06T01:08:22.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:22.211+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.212485+0000) 2020-02-06T01:08:22.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:22.209+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.211318+0000) 2020-02-06T01:08:22.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:22.210+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.211506+0000) 2020-02-06T01:08:22.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:22.210+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.212206+0000) 2020-02-06T01:08:22.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:22.211+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.212988+0000) 2020-02-06T01:08:22.217 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:22.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.213554+0000) 2020-02-06T01:08:22.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:22.214+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.216006+0000) 2020-02-06T01:08:22.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:22.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.216445+0000) 2020-02-06T01:08:22.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:22.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:22.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:22.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:22.939045+0000) 2020-02-06T01:08:23.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:23.168+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:23.169020+0000) 2020-02-06T01:08:23.614 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:08:23.620 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:08:25.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:25.176+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:25.176817+0000) 2020-02-06T01:08:26.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:26.075+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:26.076196+0000) 2020-02-06T01:08:27.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:27.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.060150+0000) 2020-02-06T01:08:27.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:27.233+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.233504+0000) 2020-02-06T01:08:27.234 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:27.233+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.233916+0000) 2020-02-06T01:08:27.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:27.234+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.234380+0000) 2020-02-06T01:08:27.235 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:27.233+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.234869+0000) 2020-02-06T01:08:27.237 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:27.235+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.236488+0000) 2020-02-06T01:08:27.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:27.236+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.236880+0000) 2020-02-06T01:08:27.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:27.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.237418+0000) 2020-02-06T01:08:27.238 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:27.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.237762+0000) 2020-02-06T01:08:27.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:27.237+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.238509+0000) 2020-02-06T01:08:27.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:27.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.238777+0000) 2020-02-06T01:08:27.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:27.237+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.239162+0000) 2020-02-06T01:08:27.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:27.238+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.239649+0000) 2020-02-06T01:08:27.240 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:27.238+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.240346+0000) 2020-02-06T01:08:27.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:27.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.245004+0000) 2020-02-06T01:08:27.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:27.254+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:27.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:27.328+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.328719+0000) 2020-02-06T01:08:27.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:27.328+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.328839+0000) 2020-02-06T01:08:27.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:27.328+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:27.329056+0000) 2020-02-06T01:08:29.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:29.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:29.704325+0000) 2020-02-06T01:08:29.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:29.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:29.952357+0000) 2020-02-06T01:08:30.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:30.555+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:30.556011+0000) 2020-02-06T01:08:30.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:30.813+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:30.814281+0000) 2020-02-06T01:08:30.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:30.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:30.835759+0000) 2020-02-06T01:08:30.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:30.834+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:30.835879+0000) 2020-02-06T01:08:32.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:32.261+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.262777+0000) 2020-02-06T01:08:32.264 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:32.262+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.263249+0000) 2020-02-06T01:08:32.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:32.262+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.263630+0000) 2020-02-06T01:08:32.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:32.262+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.264058+0000) 2020-02-06T01:08:32.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:32.263+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.264727+0000) 2020-02-06T01:08:32.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:32.263+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.265356+0000) 2020-02-06T01:08:32.267 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:32.265+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.266596+0000) 2020-02-06T01:08:32.267 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:32.264+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.266283+0000) 2020-02-06T01:08:32.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:32.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.267344+0000) 2020-02-06T01:08:32.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:32.266+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.267675+0000) 2020-02-06T01:08:32.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:32.266+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.267557+0000) 2020-02-06T01:08:32.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:32.266+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.267985+0000) 2020-02-06T01:08:32.269 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:32.266+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.268216+0000) 2020-02-06T01:08:32.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:32.272+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.273241+0000) 2020-02-06T01:08:32.284 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:32.282+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194597 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:32.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:32.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:32.939270+0000) 2020-02-06T01:08:33.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:33.168+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:33.169258+0000) 2020-02-06T01:08:35.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:35.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:35.184440+0000) 2020-02-06T01:08:36.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:36.075+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:36.076489+0000) 2020-02-06T01:08:37.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:37.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.060388+0000) 2020-02-06T01:08:37.289 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:37.287+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.288496+0000) 2020-02-06T01:08:37.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:37.288+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.289458+0000) 2020-02-06T01:08:37.291 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:37.288+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.290377+0000) 2020-02-06T01:08:37.292 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:37.290+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.291821+0000) 2020-02-06T01:08:37.292 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:37.291+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.292426+0000) 2020-02-06T01:08:37.293 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:37.292+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.293233+0000) 2020-02-06T01:08:37.294 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:37.291+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.293284+0000) 2020-02-06T01:08:37.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:37.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.293904+0000) 2020-02-06T01:08:37.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:37.292+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.294265+0000) 2020-02-06T01:08:37.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:37.293+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.294504+0000) 2020-02-06T01:08:37.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:37.293+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.295114+0000) 2020-02-06T01:08:37.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:37.295+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.296362+0000) 2020-02-06T01:08:37.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:37.294+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.295834+0000) 2020-02-06T01:08:37.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:37.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.300217+0000) 2020-02-06T01:08:37.311 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:37.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194638 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:37.329 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:37.328+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.328921+0000) 2020-02-06T01:08:37.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:37.328+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.329049+0000) 2020-02-06T01:08:37.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:37.328+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:37.329218+0000) 2020-02-06T01:08:39.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:39.703+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:39.704559+0000) 2020-02-06T01:08:39.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:39.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:39.952585+0000) 2020-02-06T01:08:40.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:40.555+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:40.556337+0000) 2020-02-06T01:08:40.815 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:40.813+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:40.814504+0000) 2020-02-06T01:08:40.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:40.834+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:40.835982+0000) 2020-02-06T01:08:40.836 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:40.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:40.836167+0000) 2020-02-06T01:08:42.317 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:42.315+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.316553+0000) 2020-02-06T01:08:42.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:42.316+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.317402+0000) 2020-02-06T01:08:42.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:42.316+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.318321+0000) 2020-02-06T01:08:42.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:42.316+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.318242+0000) 2020-02-06T01:08:42.320 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:42.317+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.318870+0000) 2020-02-06T01:08:42.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:42.318+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.319598+0000) 2020-02-06T01:08:42.320 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:42.319+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.320212+0000) 2020-02-06T01:08:42.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:42.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.320876+0000) 2020-02-06T01:08:42.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:42.320+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.321455+0000) 2020-02-06T01:08:42.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:42.320+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.321840+0000) 2020-02-06T01:08:42.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:42.320+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.322313+0000) 2020-02-06T01:08:42.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:42.321+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.322483+0000) 2020-02-06T01:08:42.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:42.321+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.322343+0000) 2020-02-06T01:08:42.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:42.326+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.327363+0000) 2020-02-06T01:08:42.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:42.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:42.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:42.938+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:42.939513+0000) 2020-02-06T01:08:43.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:43.168+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:43.169473+0000) 2020-02-06T01:08:45.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:45.191+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:45.192418+0000) 2020-02-06T01:08:46.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:46.076+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:46.076739+0000) 2020-02-06T01:08:47.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:47.059+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.060625+0000) 2020-02-06T01:08:47.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:47.328+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.329155+0000) 2020-02-06T01:08:47.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:47.328+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.329381+0000) 2020-02-06T01:08:47.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:47.328+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.329417+0000) 2020-02-06T01:08:47.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:47.341+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.342588+0000) 2020-02-06T01:08:47.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:47.342+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.343867+0000) 2020-02-06T01:08:47.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:47.342+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.344144+0000) 2020-02-06T01:08:47.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:47.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.344672+0000) 2020-02-06T01:08:47.345 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:47.344+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.344952+0000) 2020-02-06T01:08:47.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:47.344+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.345178+0000) 2020-02-06T01:08:47.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:47.344+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.345650+0000) 2020-02-06T01:08:47.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:47.345+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.346170+0000) 2020-02-06T01:08:47.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:47.346+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.347182+0000) 2020-02-06T01:08:47.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:47.346+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.347504+0000) 2020-02-06T01:08:47.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:47.347+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.348619+0000) 2020-02-06T01:08:47.349 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:47.347+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.348795+0000) 2020-02-06T01:08:47.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:47.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.349005+0000) 2020-02-06T01:08:47.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:47.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:47.352964+0000) 2020-02-06T01:08:47.363 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:47.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:49.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:49.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:49.704820+0000) 2020-02-06T01:08:49.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:49.951+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:49.952818+0000) 2020-02-06T01:08:50.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:50.556+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:50.556636+0000) 2020-02-06T01:08:50.815 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:50.814+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:50.814754+0000) 2020-02-06T01:08:50.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:50.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:50.836298+0000) 2020-02-06T01:08:50.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:50.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:50.836412+0000) 2020-02-06T01:08:52.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:52.367+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.368737+0000) 2020-02-06T01:08:52.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:52.367+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.368810+0000) 2020-02-06T01:08:52.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:52.369+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.369764+0000) 2020-02-06T01:08:52.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:52.369+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.370310+0000) 2020-02-06T01:08:52.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:52.369+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.370805+0000) 2020-02-06T01:08:52.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:52.370+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.371076+0000) 2020-02-06T01:08:52.372 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:52.370+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.371261+0000) 2020-02-06T01:08:52.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:52.371+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.372154+0000) 2020-02-06T01:08:52.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:52.371+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.372258+0000) 2020-02-06T01:08:52.373 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:52.371+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.372697+0000) 2020-02-06T01:08:52.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:52.371+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.372985+0000) 2020-02-06T01:08:52.374 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:52.372+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.373028+0000) 2020-02-06T01:08:52.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:52.373+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.373744+0000) 2020-02-06T01:08:52.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:52.379+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.380615+0000) 2020-02-06T01:08:52.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:52.390+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194759 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:52.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:52.939+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:52.939845+0000) 2020-02-06T01:08:53.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:53.169+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:53.169681+0000) 2020-02-06T01:08:53.754 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:08:53.760 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:08:55.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:55.203+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:55.204187+0000) 2020-02-06T01:08:56.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:56.076+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:56.076981+0000) 2020-02-06T01:08:57.061 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:57.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.060879+0000) 2020-02-06T01:08:57.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:57.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.329412+0000) 2020-02-06T01:08:57.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:57.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.329717+0000) 2020-02-06T01:08:57.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:57.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.329635+0000) 2020-02-06T01:08:57.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:08:57.395+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.396123+0000) 2020-02-06T01:08:57.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:57.397+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.398350+0000) 2020-02-06T01:08:57.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:57.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.399023+0000) 2020-02-06T01:08:57.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:08:57.397+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.398743+0000) 2020-02-06T01:08:57.404 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:08:57.398+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.399324+0000) 2020-02-06T01:08:57.404 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:08:57.399+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.400455+0000) 2020-02-06T01:08:57.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:08:57.400+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.401232+0000) 2020-02-06T01:08:57.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:08:57.396+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.397041+0000) 2020-02-06T01:08:57.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:08:57.397+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.397761+0000) 2020-02-06T01:08:57.406 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:08:57.400+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.401270+0000) 2020-02-06T01:08:57.406 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:08:57.401+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.401632+0000) 2020-02-06T01:08:57.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:08:57.401+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.402331+0000) 2020-02-06T01:08:57.407 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:08:57.402+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.402677+0000) 2020-02-06T01:08:57.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:57.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:57.406004+0000) 2020-02-06T01:08:57.419 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:08:57.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194800 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:08:59.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:59.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:59.705063+0000) 2020-02-06T01:08:59.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:08:59.952+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:08:59.953051+0000) 2020-02-06T01:09:00.557 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:00.556+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:00.556878+0000) 2020-02-06T01:09:00.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:00.814+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:00.814993+0000) 2020-02-06T01:09:00.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:00.835+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:00.836526+0000) 2020-02-06T01:09:00.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:00.835+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:00.836627+0000) 2020-02-06T01:09:02.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:02.422+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.423697+0000) 2020-02-06T01:09:02.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:02.423+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.424239+0000) 2020-02-06T01:09:02.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:02.424+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.424711+0000) 2020-02-06T01:09:02.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:02.425+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.426404+0000) 2020-02-06T01:09:02.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:02.426+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.427245+0000) 2020-02-06T01:09:02.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:02.427+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.427835+0000) 2020-02-06T01:09:02.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:02.426+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.428016+0000) 2020-02-06T01:09:02.430 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:02.427+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.428358+0000) 2020-02-06T01:09:02.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:02.428+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.428968+0000) 2020-02-06T01:09:02.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:02.428+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.429280+0000) 2020-02-06T01:09:02.432 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:02.428+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.429705+0000) 2020-02-06T01:09:02.433 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:02.429+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.429987+0000) 2020-02-06T01:09:02.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:02.429+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.430347+0000) 2020-02-06T01:09:02.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:02.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.436474+0000) 2020-02-06T01:09:02.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:02.447+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:02.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:02.939+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:02.940085+0000) 2020-02-06T01:09:03.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:03.169+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:03.169969+0000) 2020-02-06T01:09:05.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:05.212+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:05.213321+0000) 2020-02-06T01:09:06.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:06.076+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:06.077233+0000) 2020-02-06T01:09:07.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:07.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.061119+0000) 2020-02-06T01:09:07.330 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:07.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.329748+0000) 2020-02-06T01:09:07.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:07.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.329868+0000) 2020-02-06T01:09:07.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:07.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.329934+0000) 2020-02-06T01:09:07.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:07.455+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.455866+0000) 2020-02-06T01:09:07.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:07.455+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.456305+0000) 2020-02-06T01:09:07.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:07.456+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.456828+0000) 2020-02-06T01:09:07.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:07.457+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.457503+0000) 2020-02-06T01:09:07.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:07.457+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.458016+0000) 2020-02-06T01:09:07.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:07.451+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.452649+0000) 2020-02-06T01:09:07.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:07.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.457052+0000) 2020-02-06T01:09:07.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:07.453+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.454244+0000) 2020-02-06T01:09:07.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:07.454+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.455157+0000) 2020-02-06T01:09:07.462 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:07.455+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.456498+0000) 2020-02-06T01:09:07.462 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:07.456+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.457339+0000) 2020-02-06T01:09:07.462 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:07.457+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.458157+0000) 2020-02-06T01:09:07.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:07.457+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.458396+0000) 2020-02-06T01:09:07.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:07.463+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:07.464435+0000) 2020-02-06T01:09:07.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:07.476+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194881 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:09.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:09.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:09.705335+0000) 2020-02-06T01:09:09.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:09.953+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:09.953348+0000) 2020-02-06T01:09:10.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:10.556+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:10.557083+0000) 2020-02-06T01:09:10.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:10.814+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:10.815266+0000) 2020-02-06T01:09:10.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:10.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:10.836794+0000) 2020-02-06T01:09:10.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:10.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:10.836883+0000) 2020-02-06T01:09:12.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:12.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.482844+0000) 2020-02-06T01:09:12.485 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:12.483+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.483990+0000) 2020-02-06T01:09:12.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:12.483+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.484279+0000) 2020-02-06T01:09:12.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:12.483+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.484751+0000) 2020-02-06T01:09:12.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:12.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.485155+0000) 2020-02-06T01:09:12.486 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:12.484+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.485559+0000) 2020-02-06T01:09:12.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:12.485+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.486335+0000) 2020-02-06T01:09:12.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:12.486+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.486857+0000) 2020-02-06T01:09:12.488 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:12.486+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.487402+0000) 2020-02-06T01:09:12.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:12.486+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.486518+0000) 2020-02-06T01:09:12.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:12.487+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.487860+0000) 2020-02-06T01:09:12.489 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:12.487+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.488264+0000) 2020-02-06T01:09:12.489 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:12.488+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.488630+0000) 2020-02-06T01:09:12.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:12.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.495145+0000) 2020-02-06T01:09:12.504 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:12.502+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:12.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:12.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:12.940310+0000) 2020-02-06T01:09:13.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:13.169+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:13.170182+0000) 2020-02-06T01:09:15.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:15.221+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:15.221634+0000) 2020-02-06T01:09:16.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:16.077+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:16.077524+0000) 2020-02-06T01:09:17.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:17.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.061354+0000) 2020-02-06T01:09:17.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:17.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.329973+0000) 2020-02-06T01:09:17.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:17.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.330117+0000) 2020-02-06T01:09:17.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:17.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.330179+0000) 2020-02-06T01:09:17.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:17.508+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.509188+0000) 2020-02-06T01:09:17.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:17.509+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.510252+0000) 2020-02-06T01:09:17.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:17.510+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.510955+0000) 2020-02-06T01:09:17.512 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:17.510+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.511161+0000) 2020-02-06T01:09:17.512 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:17.510+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.511772+0000) 2020-02-06T01:09:17.512 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:17.511+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.512217+0000) 2020-02-06T01:09:17.513 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:17.512+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.512757+0000) 2020-02-06T01:09:17.513 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:17.512+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.513302+0000) 2020-02-06T01:09:17.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:17.512+0000 7fdad5468700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.513945+0000) 2020-02-06T01:09:17.515 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:17.513+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.514188+0000) 2020-02-06T01:09:17.516 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:17.515+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.515409+0000) 2020-02-06T01:09:17.516 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:17.515+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.515631+0000) 2020-02-06T01:09:17.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:17.518+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.519553+0000) 2020-02-06T01:09:17.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:17.518+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:17.519821+0000) 2020-02-06T01:09:17.532 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:17.529+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 194962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:19.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:19.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:19.705569+0000) 2020-02-06T01:09:19.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:19.953+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:19.953577+0000) 2020-02-06T01:09:20.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:20.557+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:20.557356+0000) 2020-02-06T01:09:20.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:20.815+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:20.815526+0000) 2020-02-06T01:09:20.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:20.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:20.837030+0000) 2020-02-06T01:09:20.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:20.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:20.837145+0000) 2020-02-06T01:09:22.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:22.536+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.536740+0000) 2020-02-06T01:09:22.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:22.537+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.537518+0000) 2020-02-06T01:09:22.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:22.539+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.539344+0000) 2020-02-06T01:09:22.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:22.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.540438+0000) 2020-02-06T01:09:22.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:22.540+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.540746+0000) 2020-02-06T01:09:22.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:22.541+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.541260+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:22.541+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.541465+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:22.537+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.538136+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:22.539+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.539768+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:22.540+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.540865+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:22.540+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.541390+0000) 2020-02-06T01:09:22.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:22.541+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.541619+0000) 2020-02-06T01:09:22.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:22.545+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.545284+0000) 2020-02-06T01:09:22.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:22.545+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.545618+0000) 2020-02-06T01:09:22.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:22.556+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:22.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:22.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:22.940592+0000) 2020-02-06T01:09:23.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:23.170+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:23.170427+0000) 2020-02-06T01:09:23.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:09:23.898 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:09:25.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:25.228+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:25.229779+0000) 2020-02-06T01:09:26.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:26.076+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:26.077789+0000) 2020-02-06T01:09:27.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:27.060+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.061581+0000) 2020-02-06T01:09:27.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:27.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.330269+0000) 2020-02-06T01:09:28.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:27.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.330404+0000) 2020-02-06T01:09:28.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:27.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.330494+0000) 2020-02-06T01:09:28.119 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:27.560+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.561468+0000) 2020-02-06T01:09:28.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:27.561+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.562939+0000) 2020-02-06T01:09:28.119 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:27.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.563710+0000) 2020-02-06T01:09:28.119 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:27.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.564442+0000) 2020-02-06T01:09:28.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:27.564+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.565479+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:27.565+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.566954+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:27.573+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.574249+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:27.573+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.574515+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:27.561+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.562796+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:27.562+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.564165+0000) 2020-02-06T01:09:28.120 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:27.565+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.567369+0000) 2020-02-06T01:09:28.121 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:27.566+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.567810+0000) 2020-02-06T01:09:28.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:27.566+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.568196+0000) 2020-02-06T01:09:28.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:27.567+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:27.568916+0000) 2020-02-06T01:09:28.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:27.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:29.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:29.704+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:29.705803+0000) 2020-02-06T01:09:29.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:29.952+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:29.953807+0000) 2020-02-06T01:09:30.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:30.556+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:30.557620+0000) 2020-02-06T01:09:30.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:30.814+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:30.815747+0000) 2020-02-06T01:09:30.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:30.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:30.837294+0000) 2020-02-06T01:09:30.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:30.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:30.837393+0000) 2020-02-06T01:09:32.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:32.588+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.589366+0000) 2020-02-06T01:09:32.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:32.588+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.589984+0000) 2020-02-06T01:09:32.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:32.589+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.590618+0000) 2020-02-06T01:09:32.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:32.590+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.591258+0000) 2020-02-06T01:09:32.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:32.590+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.591653+0000) 2020-02-06T01:09:32.594 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:32.590+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.592189+0000) 2020-02-06T01:09:32.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:32.591+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.592780+0000) 2020-02-06T01:09:32.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:32.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.593661+0000) 2020-02-06T01:09:32.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:32.592+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.594294+0000) 2020-02-06T01:09:32.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:32.593+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.594748+0000) 2020-02-06T01:09:32.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:32.593+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.595144+0000) 2020-02-06T01:09:32.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:32.593+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.594949+0000) 2020-02-06T01:09:32.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:32.594+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.595907+0000) 2020-02-06T01:09:32.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:32.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.602137+0000) 2020-02-06T01:09:32.614 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:32.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195083 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:32.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:32.939+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:32.940864+0000) 2020-02-06T01:09:33.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:33.169+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:33.170637+0000) 2020-02-06T01:09:35.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:35.237+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:35.237872+0000) 2020-02-06T01:09:36.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:36.077+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:36.078057+0000) 2020-02-06T01:09:37.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:37.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.061836+0000) 2020-02-06T01:09:37.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:37.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.330559+0000) 2020-02-06T01:09:37.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:37.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.330774+0000) 2020-02-06T01:09:37.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:37.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.330710+0000) 2020-02-06T01:09:37.618 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:37.616+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.617135+0000) 2020-02-06T01:09:37.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:37.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.617514+0000) 2020-02-06T01:09:37.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:37.616+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.617739+0000) 2020-02-06T01:09:37.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:37.617+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.618352+0000) 2020-02-06T01:09:37.619 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:37.617+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.618970+0000) 2020-02-06T01:09:37.620 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:37.617+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.618917+0000) 2020-02-06T01:09:37.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:37.618+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.619464+0000) 2020-02-06T01:09:37.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:37.619+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.620875+0000) 2020-02-06T01:09:37.622 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:37.619+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.621119+0000) 2020-02-06T01:09:37.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:37.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.620101+0000) 2020-02-06T01:09:37.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:37.619+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.620275+0000) 2020-02-06T01:09:37.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:37.619+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.620456+0000) 2020-02-06T01:09:37.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:37.621+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.622387+0000) 2020-02-06T01:09:37.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:37.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:37.623488+0000) 2020-02-06T01:09:37.637 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:37.635+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:39.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:39.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:39.706061+0000) 2020-02-06T01:09:39.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:39.952+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:39.954006+0000) 2020-02-06T01:09:40.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:40.556+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:40.557873+0000) 2020-02-06T01:09:40.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:40.814+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:40.816041+0000) 2020-02-06T01:09:40.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:40.836+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:40.837566+0000) 2020-02-06T01:09:40.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:40.836+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:40.837616+0000) 2020-02-06T01:09:42.643 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:42.641+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.642852+0000) 2020-02-06T01:09:42.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:42.642+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.643850+0000) 2020-02-06T01:09:42.646 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:42.643+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.644304+0000) 2020-02-06T01:09:42.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:42.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.645125+0000) 2020-02-06T01:09:42.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:42.642+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.643682+0000) 2020-02-06T01:09:42.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:42.643+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.644592+0000) 2020-02-06T01:09:42.647 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:42.644+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.645727+0000) 2020-02-06T01:09:42.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:42.645+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.646509+0000) 2020-02-06T01:09:42.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:42.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.648113+0000) 2020-02-06T01:09:42.650 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:42.647+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.648512+0000) 2020-02-06T01:09:42.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:42.647+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.648310+0000) 2020-02-06T01:09:42.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:42.647+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.648769+0000) 2020-02-06T01:09:42.651 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:42.647+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.648955+0000) 2020-02-06T01:09:42.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:42.654+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.655910+0000) 2020-02-06T01:09:42.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:42.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195164 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:42.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:42.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:42.941095+0000) 2020-02-06T01:09:43.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:43.169+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:43.170867+0000) 2020-02-06T01:09:45.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:45.245+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:45.246121+0000) 2020-02-06T01:09:46.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:46.077+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:46.078318+0000) 2020-02-06T01:09:47.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:47.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.062121+0000) 2020-02-06T01:09:47.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:47.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.330740+0000) 2020-02-06T01:09:47.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:47.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.330926+0000) 2020-02-06T01:09:47.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:47.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.331048+0000) 2020-02-06T01:09:47.671 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:47.669+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.670639+0000) 2020-02-06T01:09:47.672 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:47.670+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.671665+0000) 2020-02-06T01:09:47.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:47.670+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.671829+0000) 2020-02-06T01:09:47.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:47.671+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.672602+0000) 2020-02-06T01:09:47.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:47.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.673326+0000) 2020-02-06T01:09:47.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:47.672+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.673652+0000) 2020-02-06T01:09:47.674 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:47.672+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.673767+0000) 2020-02-06T01:09:47.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:47.672+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.674321+0000) 2020-02-06T01:09:47.676 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:47.673+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.675044+0000) 2020-02-06T01:09:47.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:47.674+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.675622+0000) 2020-02-06T01:09:47.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:47.674+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.675526+0000) 2020-02-06T01:09:47.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:47.674+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.676223+0000) 2020-02-06T01:09:47.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:47.675+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.676170+0000) 2020-02-06T01:09:47.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:47.679+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:47.680623+0000) 2020-02-06T01:09:47.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:47.691+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:49.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:49.705+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:49.706350+0000) 2020-02-06T01:09:49.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:49.953+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:49.954259+0000) 2020-02-06T01:09:50.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:50.557+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:50.558136+0000) 2020-02-06T01:09:50.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:50.815+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:50.816301+0000) 2020-02-06T01:09:50.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:50.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:50.837818+0000) 2020-02-06T01:09:50.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:50.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:50.837909+0000) 2020-02-06T01:09:52.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:52.696+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.697353+0000) 2020-02-06T01:09:52.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:52.698+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.699453+0000) 2020-02-06T01:09:52.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:52.698+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.700395+0000) 2020-02-06T01:09:52.701 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:52.699+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.701174+0000) 2020-02-06T01:09:52.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:52.700+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.701654+0000) 2020-02-06T01:09:52.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:52.700+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.701492+0000) 2020-02-06T01:09:52.703 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:52.701+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.702516+0000) 2020-02-06T01:09:52.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:52.701+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.702660+0000) 2020-02-06T01:09:52.704 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:52.702+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.703311+0000) 2020-02-06T01:09:52.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:52.702+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.703635+0000) 2020-02-06T01:09:52.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:52.702+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.704115+0000) 2020-02-06T01:09:52.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:52.703+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.704132+0000) 2020-02-06T01:09:52.705 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:52.703+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.704857+0000) 2020-02-06T01:09:52.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:52.711+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.712526+0000) 2020-02-06T01:09:52.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:52.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:52.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:52.940+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:52.941381+0000) 2020-02-06T01:09:53.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:53.170+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:53.171103+0000) 2020-02-06T01:09:54.032 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:09:54.039 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:09:55.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:55.253+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:55.254392+0000) 2020-02-06T01:09:56.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:56.077+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:56.078579+0000) 2020-02-06T01:09:57.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:57.061+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.062385+0000) 2020-02-06T01:09:57.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:57.329+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.330971+0000) 2020-02-06T01:09:57.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:57.329+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.331182+0000) 2020-02-06T01:09:57.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:57.329+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.331384+0000) 2020-02-06T01:09:57.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:09:57.724+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.725803+0000) 2020-02-06T01:09:57.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:57.725+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.726223+0000) 2020-02-06T01:09:57.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:09:57.725+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.726649+0000) 2020-02-06T01:09:57.728 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:09:57.725+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.726899+0000) 2020-02-06T01:09:57.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:09:57.726+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.727205+0000) 2020-02-06T01:09:57.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:09:57.726+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.727622+0000) 2020-02-06T01:09:57.728 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:09:57.726+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.727748+0000) 2020-02-06T01:09:57.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:57.727+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.728421+0000) 2020-02-06T01:09:57.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:09:57.727+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.728890+0000) 2020-02-06T01:09:57.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:09:57.727+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.728734+0000) 2020-02-06T01:09:57.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:09:57.727+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.728913+0000) 2020-02-06T01:09:57.731 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:09:57.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.729534+0000) 2020-02-06T01:09:57.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:09:57.729+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.730715+0000) 2020-02-06T01:09:57.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:57.732+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:57.733844+0000) 2020-02-06T01:09:57.746 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:09:57.744+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:09:59.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:59.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:59.706582+0000) 2020-02-06T01:09:59.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:09:59.954+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:09:59.954504+0000) 2020-02-06T01:10:00.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:00.557+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:00.558358+0000) 2020-02-06T01:10:00.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:00.816+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:00.816532+0000) 2020-02-06T01:10:00.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:00.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:00.838133+0000) 2020-02-06T01:10:00.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:00.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:00.838139+0000) 2020-02-06T01:10:02.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:02.750+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.751470+0000) 2020-02-06T01:10:02.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:02.751+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.752659+0000) 2020-02-06T01:10:02.753 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:02.752+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.753115+0000) 2020-02-06T01:10:02.754 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:02.752+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.753379+0000) 2020-02-06T01:10:02.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:02.752+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.753845+0000) 2020-02-06T01:10:02.755 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:02.753+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.754534+0000) 2020-02-06T01:10:02.757 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:02.753+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.754824+0000) 2020-02-06T01:10:02.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:02.754+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.755602+0000) 2020-02-06T01:10:02.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:02.753+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.755109+0000) 2020-02-06T01:10:02.758 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:02.754+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.755974+0000) 2020-02-06T01:10:02.758 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:02.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.756831+0000) 2020-02-06T01:10:02.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:02.756+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.757582+0000) 2020-02-06T01:10:02.759 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:02.755+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.757291+0000) 2020-02-06T01:10:02.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:02.761+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.762531+0000) 2020-02-06T01:10:02.774 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:02.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:02.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:02.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:02.941652+0000) 2020-02-06T01:10:03.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:03.170+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:03.171321+0000) 2020-02-06T01:10:05.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:05.262+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:05.262682+0000) 2020-02-06T01:10:06.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:06.078+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:06.078883+0000) 2020-02-06T01:10:07.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:07.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.062655+0000) 2020-02-06T01:10:07.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:07.330+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.331224+0000) 2020-02-06T01:10:07.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:07.330+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.331422+0000) 2020-02-06T01:10:07.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:07.330+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.331670+0000) 2020-02-06T01:10:07.780 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:07.777+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.779205+0000) 2020-02-06T01:10:07.782 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:07.780+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.782036+0000) 2020-02-06T01:10:07.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:07.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.782481+0000) 2020-02-06T01:10:07.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:07.781+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.782700+0000) 2020-02-06T01:10:07.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:07.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.783277+0000) 2020-02-06T01:10:07.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:07.783+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.784219+0000) 2020-02-06T01:10:07.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:07.783+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.784603+0000) 2020-02-06T01:10:07.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:07.783+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.784987+0000) 2020-02-06T01:10:07.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:07.783+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.785153+0000) 2020-02-06T01:10:07.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:07.784+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.785547+0000) 2020-02-06T01:10:07.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:07.784+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.785957+0000) 2020-02-06T01:10:07.787 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:07.785+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.786165+0000) 2020-02-06T01:10:07.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:07.785+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.786563+0000) 2020-02-06T01:10:07.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:07.790+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:07.791861+0000) 2020-02-06T01:10:07.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:07.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195365 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:09.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:09.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:09.706818+0000) 2020-02-06T01:10:09.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:09.953+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:09.954739+0000) 2020-02-06T01:10:10.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:10.557+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:10.558619+0000) 2020-02-06T01:10:10.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:10.816+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:10.816766+0000) 2020-02-06T01:10:10.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:10.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:10.838367+0000) 2020-02-06T01:10:10.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:10.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:10.838349+0000) 2020-02-06T01:10:12.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:12.806+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.807282+0000) 2020-02-06T01:10:12.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:12.806+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.807417+0000) 2020-02-06T01:10:12.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:12.807+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.808375+0000) 2020-02-06T01:10:12.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:12.812+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.814025+0000) 2020-02-06T01:10:12.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:12.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.814254+0000) 2020-02-06T01:10:12.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:12.807+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.809049+0000) 2020-02-06T01:10:12.817 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:12.810+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.811385+0000) 2020-02-06T01:10:12.817 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:12.810+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.811599+0000) 2020-02-06T01:10:12.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:12.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.806342+0000) 2020-02-06T01:10:12.818 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:12.805+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.806945+0000) 2020-02-06T01:10:12.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:12.808+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.810026+0000) 2020-02-06T01:10:12.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:12.809+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.810595+0000) 2020-02-06T01:10:12.818 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:12.809+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.810784+0000) 2020-02-06T01:10:12.819 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:12.810+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.811770+0000) 2020-02-06T01:10:12.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:12.825+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:12.942 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:12.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:12.941917+0000) 2020-02-06T01:10:13.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:13.171+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:13.171575+0000) 2020-02-06T01:10:15.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:15.270+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:15.270534+0000) 2020-02-06T01:10:16.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:16.078+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:16.079142+0000) 2020-02-06T01:10:17.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:17.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.062890+0000) 2020-02-06T01:10:17.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:17.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.331473+0000) 2020-02-06T01:10:17.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:17.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.331685+0000) 2020-02-06T01:10:17.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:17.331+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.331994+0000) 2020-02-06T01:10:17.833 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:17.832+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.832934+0000) 2020-02-06T01:10:17.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:17.833+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.833474+0000) 2020-02-06T01:10:17.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:17.832+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.832412+0000) 2020-02-06T01:10:17.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:17.835+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.835214+0000) 2020-02-06T01:10:17.836 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:17.835+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.835491+0000) 2020-02-06T01:10:17.836 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:17.835+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.835891+0000) 2020-02-06T01:10:17.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:17.836+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.836432+0000) 2020-02-06T01:10:17.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:17.836+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.836772+0000) 2020-02-06T01:10:17.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:17.836+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.837183+0000) 2020-02-06T01:10:17.838 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:17.837+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.837779+0000) 2020-02-06T01:10:17.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:17.838+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.838066+0000) 2020-02-06T01:10:17.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:17.837+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.838380+0000) 2020-02-06T01:10:17.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:17.838+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.838500+0000) 2020-02-06T01:10:17.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:17.843+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:17.843855+0000) 2020-02-06T01:10:17.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:17.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:19.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:19.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:19.707057+0000) 2020-02-06T01:10:19.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:19.953+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:19.954993+0000) 2020-02-06T01:10:20.559 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:20.558+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:20.558875+0000) 2020-02-06T01:10:20.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:20.815+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:20.816999+0000) 2020-02-06T01:10:20.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:20.837+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:20.838659+0000) 2020-02-06T01:10:20.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:20.837+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:20.838642+0000) 2020-02-06T01:10:22.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:22.858+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.859328+0000) 2020-02-06T01:10:22.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:22.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.859879+0000) 2020-02-06T01:10:22.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:22.859+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.860287+0000) 2020-02-06T01:10:22.861 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:22.860+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.860862+0000) 2020-02-06T01:10:22.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:22.860+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.861468+0000) 2020-02-06T01:10:22.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:22.861+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.862134+0000) 2020-02-06T01:10:22.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:22.862+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.862538+0000) 2020-02-06T01:10:22.863 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:22.862+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.862913+0000) 2020-02-06T01:10:22.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:22.862+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.863373+0000) 2020-02-06T01:10:22.864 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:22.862+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.863983+0000) 2020-02-06T01:10:22.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:22.864+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.865457+0000) 2020-02-06T01:10:22.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:22.864+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.864923+0000) 2020-02-06T01:10:22.867 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:22.865+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.865991+0000) 2020-02-06T01:10:22.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:22.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.871324+0000) 2020-02-06T01:10:22.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:22.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195487 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:22.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:22.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:22.942128+0000) 2020-02-06T01:10:23.172 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:23.170+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:23.171740+0000) 2020-02-06T01:10:24.172 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:10:24.178 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:10:25.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:25.277+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:25.278635+0000) 2020-02-06T01:10:26.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:26.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:26.079403+0000) 2020-02-06T01:10:27.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:27.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.063121+0000) 2020-02-06T01:10:27.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:27.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.331714+0000) 2020-02-06T01:10:27.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:27.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.331931+0000) 2020-02-06T01:10:27.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:27.331+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.332259+0000) 2020-02-06T01:10:27.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:27.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.887250+0000) 2020-02-06T01:10:27.888 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:27.886+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.887887+0000) 2020-02-06T01:10:27.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:27.888+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.889687+0000) 2020-02-06T01:10:27.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:27.889+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.890174+0000) 2020-02-06T01:10:27.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:27.889+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.890639+0000) 2020-02-06T01:10:27.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:27.890+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.891395+0000) 2020-02-06T01:10:27.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:27.890+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.891285+0000) 2020-02-06T01:10:27.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:27.891+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.891877+0000) 2020-02-06T01:10:27.893 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:27.891+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.892240+0000) 2020-02-06T01:10:27.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:27.892+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.892728+0000) 2020-02-06T01:10:27.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:27.892+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.892853+0000) 2020-02-06T01:10:27.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:27.892+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.893500+0000) 2020-02-06T01:10:27.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:27.893+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.893887+0000) 2020-02-06T01:10:27.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:27.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:27.896581+0000) 2020-02-06T01:10:27.908 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:27.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195526 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:29.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:29.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:29.707367+0000) 2020-02-06T01:10:29.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:29.954+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:29.955215+0000) 2020-02-06T01:10:30.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:30.558+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:30.559140+0000) 2020-02-06T01:10:30.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:30.816+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:30.817266+0000) 2020-02-06T01:10:30.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:30.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:30.838902+0000) 2020-02-06T01:10:30.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:30.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:30.838922+0000) 2020-02-06T01:10:32.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:32.911+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.913047+0000) 2020-02-06T01:10:32.914 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:32.912+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.913940+0000) 2020-02-06T01:10:32.915 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:32.913+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.914739+0000) 2020-02-06T01:10:32.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:32.915+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.916358+0000) 2020-02-06T01:10:32.917 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:32.915+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.916284+0000) 2020-02-06T01:10:32.917 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:32.916+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.916975+0000) 2020-02-06T01:10:32.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:32.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.917940+0000) 2020-02-06T01:10:32.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:32.917+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.918561+0000) 2020-02-06T01:10:32.919 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:32.918+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.918639+0000) 2020-02-06T01:10:32.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:32.918+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.919273+0000) 2020-02-06T01:10:32.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:32.918+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.919698+0000) 2020-02-06T01:10:32.921 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:32.919+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.920088+0000) 2020-02-06T01:10:32.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:32.919+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.920187+0000) 2020-02-06T01:10:32.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:32.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.923052+0000) 2020-02-06T01:10:32.935 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:32.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:32.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:32.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:32.942370+0000) 2020-02-06T01:10:33.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:33.171+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:33.171952+0000) 2020-02-06T01:10:35.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:35.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:35.287226+0000) 2020-02-06T01:10:36.080 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:36.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:36.079765+0000) 2020-02-06T01:10:37.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:37.062+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.063353+0000) 2020-02-06T01:10:37.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:37.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.331957+0000) 2020-02-06T01:10:37.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:37.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.332158+0000) 2020-02-06T01:10:37.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:37.332+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.332529+0000) 2020-02-06T01:10:37.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:37.940+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.941395+0000) 2020-02-06T01:10:37.942 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:37.940+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.941812+0000) 2020-02-06T01:10:37.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:37.941+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.942344+0000) 2020-02-06T01:10:37.944 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:37.942+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.943441+0000) 2020-02-06T01:10:37.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:37.942+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.943413+0000) 2020-02-06T01:10:37.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:37.943+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.943809+0000) 2020-02-06T01:10:37.945 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:37.943+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.944023+0000) 2020-02-06T01:10:37.946 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:37.944+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.944515+0000) 2020-02-06T01:10:37.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:37.944+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.944972+0000) 2020-02-06T01:10:37.946 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:37.944+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.944787+0000) 2020-02-06T01:10:37.947 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:37.945+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.945719+0000) 2020-02-06T01:10:37.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:37.946+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.947197+0000) 2020-02-06T01:10:37.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:37.947+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.948672+0000) 2020-02-06T01:10:37.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:37.947+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:37.949028+0000) 2020-02-06T01:10:37.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:37.960+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195606 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:39.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:39.706+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:39.707623+0000) 2020-02-06T01:10:39.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:39.954+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:39.955442+0000) 2020-02-06T01:10:40.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:40.559+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:40.559371+0000) 2020-02-06T01:10:40.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:40.816+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:40.817496+0000) 2020-02-06T01:10:40.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:40.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:40.839128+0000) 2020-02-06T01:10:40.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:40.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:40.839119+0000) 2020-02-06T01:10:42.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:42.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.942600+0000) 2020-02-06T01:10:42.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:42.966+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.967061+0000) 2020-02-06T01:10:42.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:42.967+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.967986+0000) 2020-02-06T01:10:42.969 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:42.967+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.968295+0000) 2020-02-06T01:10:42.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:42.967+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.968803+0000) 2020-02-06T01:10:42.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:42.968+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.969189+0000) 2020-02-06T01:10:42.970 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:42.969+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.969802+0000) 2020-02-06T01:10:42.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:42.969+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.970223+0000) 2020-02-06T01:10:42.971 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:42.969+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.969519+0000) 2020-02-06T01:10:42.977 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:42.971+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.971992+0000) 2020-02-06T01:10:42.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:42.970+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.971461+0000) 2020-02-06T01:10:42.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:42.970+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.971723+0000) 2020-02-06T01:10:42.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:42.971+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.972257+0000) 2020-02-06T01:10:42.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:42.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.975809+0000) 2020-02-06T01:10:42.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:42.975+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:42.976115+0000) 2020-02-06T01:10:42.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:42.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195647 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:43.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:43.171+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:43.172185+0000) 2020-02-06T01:10:45.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:45.298+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:45.299400+0000) 2020-02-06T01:10:46.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:46.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:46.080017+0000) 2020-02-06T01:10:47.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:47.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.063580+0000) 2020-02-06T01:10:47.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:47.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.332205+0000) 2020-02-06T01:10:47.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:47.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.332369+0000) 2020-02-06T01:10:47.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:47.332+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.332796+0000) 2020-02-06T01:10:47.993 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:47.991+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.992578+0000) 2020-02-06T01:10:47.994 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:47.992+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.992944+0000) 2020-02-06T01:10:47.994 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:47.992+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.993381+0000) 2020-02-06T01:10:47.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:47.993+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.993505+0000) 2020-02-06T01:10:47.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:47.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.994511+0000) 2020-02-06T01:10:47.996 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:47.994+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.994908+0000) 2020-02-06T01:10:47.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:47.995+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.996345+0000) 2020-02-06T01:10:47.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:47.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.996528+0000) 2020-02-06T01:10:47.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:47.995+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.996744+0000) 2020-02-06T01:10:47.998 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:47.996+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.997162+0000) 2020-02-06T01:10:47.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:47.997+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.997539+0000) 2020-02-06T01:10:47.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:47.996+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.997117+0000) 2020-02-06T01:10:47.999 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:47.996+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.997515+0000) 2020-02-06T01:10:48.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:47.998+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:47.999984+0000) 2020-02-06T01:10:48.014 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:48.013+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195689 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:49.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:49.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:49.707859+0000) 2020-02-06T01:10:49.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:49.955+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:49.955674+0000) 2020-02-06T01:10:50.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:50.559+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:50.559632+0000) 2020-02-06T01:10:50.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:50.817+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:50.817758+0000) 2020-02-06T01:10:50.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:50.838+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:50.839348+0000) 2020-02-06T01:10:50.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:50.838+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:50.839361+0000) 2020-02-06T01:10:52.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:52.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:52.942896+0000) 2020-02-06T01:10:53.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:53.019+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.020102+0000) 2020-02-06T01:10:53.022 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:53.020+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.021279+0000) 2020-02-06T01:10:53.022 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:53.020+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.020872+0000) 2020-02-06T01:10:53.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:53.020+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.021672+0000) 2020-02-06T01:10:53.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:53.020+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.022035+0000) 2020-02-06T01:10:53.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:53.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.022728+0000) 2020-02-06T01:10:53.024 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:53.022+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.023303+0000) 2020-02-06T01:10:53.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:53.023+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.023664+0000) 2020-02-06T01:10:53.025 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:53.024+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.024633+0000) 2020-02-06T01:10:53.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:53.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.024736+0000) 2020-02-06T01:10:53.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:53.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.025462+0000) 2020-02-06T01:10:53.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:53.024+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.025699+0000) 2020-02-06T01:10:53.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:53.025+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.026030+0000) 2020-02-06T01:10:53.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:53.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.029991+0000) 2020-02-06T01:10:53.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:53.041+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:53.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:53.171+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:53.172426+0000) 2020-02-06T01:10:54.305 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:10:54.312 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:10:55.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:55.307+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:55.307662+0000) 2020-02-06T01:10:56.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:56.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:56.080257+0000) 2020-02-06T01:10:57.064 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:57.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:57.063811+0000) 2020-02-06T01:10:57.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:57.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:57.332474+0000) 2020-02-06T01:10:57.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:57.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:57.332685+0000) 2020-02-06T01:10:57.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:57.331+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:57.333001+0000) 2020-02-06T01:10:58.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:10:58.047+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.048105+0000) 2020-02-06T01:10:58.050 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:10:58.047+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.048424+0000) 2020-02-06T01:10:58.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:10:58.048+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.049481+0000) 2020-02-06T01:10:58.051 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:10:58.049+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.050175+0000) 2020-02-06T01:10:58.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:58.049+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.050540+0000) 2020-02-06T01:10:58.052 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:10:58.049+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.050712+0000) 2020-02-06T01:10:58.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:10:58.050+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.051153+0000) 2020-02-06T01:10:58.052 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:10:58.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.051488+0000) 2020-02-06T01:10:58.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:10:58.051+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.053172+0000) 2020-02-06T01:10:58.055 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:10:58.052+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.053716+0000) 2020-02-06T01:10:58.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:10:58.051+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.052684+0000) 2020-02-06T01:10:58.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:58.052+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.054061+0000) 2020-02-06T01:10:58.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:10:58.053+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.054955+0000) 2020-02-06T01:10:58.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:58.057+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:58.058818+0000) 2020-02-06T01:10:58.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:10:58.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:10:59.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:59.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:59.708112+0000) 2020-02-06T01:10:59.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:10:59.954+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:10:59.955901+0000) 2020-02-06T01:11:00.560 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:00.558+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:00.559879+0000) 2020-02-06T01:11:00.818 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:00.817+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:00.817987+0000) 2020-02-06T01:11:00.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:00.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:00.839611+0000) 2020-02-06T01:11:00.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:00.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:00.839613+0000) 2020-02-06T01:11:02.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:02.941+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:02.943135+0000) 2020-02-06T01:11:03.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:03.073+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.074551+0000) 2020-02-06T01:11:03.077 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:03.074+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.076327+0000) 2020-02-06T01:11:03.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:03.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.076802+0000) 2020-02-06T01:11:03.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:03.076+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.077803+0000) 2020-02-06T01:11:03.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:03.076+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.078181+0000) 2020-02-06T01:11:03.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:03.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.078775+0000) 2020-02-06T01:11:03.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:03.077+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.079224+0000) 2020-02-06T01:11:03.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:03.078+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.079329+0000) 2020-02-06T01:11:03.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:03.078+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.079680+0000) 2020-02-06T01:11:03.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:03.079+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.080717+0000) 2020-02-06T01:11:03.081 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:03.079+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.080795+0000) 2020-02-06T01:11:03.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:03.080+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.081460+0000) 2020-02-06T01:11:03.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:03.080+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.081631+0000) 2020-02-06T01:11:03.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:03.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.085251+0000) 2020-02-06T01:11:03.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:03.095+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:03.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:03.172+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:03.172659+0000) 2020-02-06T01:11:05.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:05.315+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:05.316042+0000) 2020-02-06T01:11:06.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:06.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:06.080514+0000) 2020-02-06T01:11:07.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:07.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:07.064042+0000) 2020-02-06T01:11:07.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:07.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:07.332713+0000) 2020-02-06T01:11:07.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:07.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:07.332919+0000) 2020-02-06T01:11:07.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:07.331+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:07.333244+0000) 2020-02-06T01:11:08.103 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:08.101+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.102598+0000) 2020-02-06T01:11:08.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:08.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.103533+0000) 2020-02-06T01:11:08.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:08.102+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.104291+0000) 2020-02-06T01:11:08.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:08.103+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.104709+0000) 2020-02-06T01:11:08.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:08.103+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.105200+0000) 2020-02-06T01:11:08.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:08.104+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.105147+0000) 2020-02-06T01:11:08.106 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:08.104+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.105747+0000) 2020-02-06T01:11:08.107 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:08.105+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.106610+0000) 2020-02-06T01:11:08.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:08.105+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.106998+0000) 2020-02-06T01:11:08.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:08.104+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.106276+0000) 2020-02-06T01:11:08.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:08.107+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.108186+0000) 2020-02-06T01:11:08.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:08.107+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.108141+0000) 2020-02-06T01:11:08.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:08.107+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.108598+0000) 2020-02-06T01:11:08.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:08.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:08.115157+0000) 2020-02-06T01:11:08.124 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:08.122+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:09.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:09.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:09.708404+0000) 2020-02-06T01:11:09.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:09.956+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:09.956130+0000) 2020-02-06T01:11:10.561 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:10.558+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:10.560119+0000) 2020-02-06T01:11:10.819 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:10.818+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:10.818208+0000) 2020-02-06T01:11:10.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:10.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:10.839832+0000) 2020-02-06T01:11:10.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:10.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:10.839889+0000) 2020-02-06T01:11:12.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:12.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:12.943369+0000) 2020-02-06T01:11:13.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:13.130+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.130750+0000) 2020-02-06T01:11:13.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:13.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.131217+0000) 2020-02-06T01:11:13.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:13.131+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.131731+0000) 2020-02-06T01:11:13.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:13.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.132271+0000) 2020-02-06T01:11:13.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:13.131+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.132473+0000) 2020-02-06T01:11:13.134 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:13.132+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.132859+0000) 2020-02-06T01:11:13.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:13.132+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.133667+0000) 2020-02-06T01:11:13.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:13.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.133982+0000) 2020-02-06T01:11:13.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:13.132+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.133692+0000) 2020-02-06T01:11:13.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:13.133+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.134681+0000) 2020-02-06T01:11:13.136 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:13.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.135519+0000) 2020-02-06T01:11:13.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:13.133+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.135056+0000) 2020-02-06T01:11:13.136 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:13.134+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.136144+0000) 2020-02-06T01:11:13.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:13.139+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.139861+0000) 2020-02-06T01:11:13.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:13.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:13.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:13.172+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:13.172879+0000) 2020-02-06T01:11:15.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:15.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:15.325329+0000) 2020-02-06T01:11:16.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:16.079+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:16.080749+0000) 2020-02-06T01:11:17.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:17.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:17.064287+0000) 2020-02-06T01:11:17.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:17.331+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:17.332957+0000) 2020-02-06T01:11:17.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:17.331+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:17.333164+0000) 2020-02-06T01:11:17.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:17.332+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:17.333471+0000) 2020-02-06T01:11:18.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:18.155+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.156738+0000) 2020-02-06T01:11:18.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:18.157+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.158065+0000) 2020-02-06T01:11:18.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:18.157+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.158561+0000) 2020-02-06T01:11:18.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:18.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.158610+0000) 2020-02-06T01:11:18.160 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:18.157+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.158858+0000) 2020-02-06T01:11:18.160 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:18.158+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.159478+0000) 2020-02-06T01:11:18.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:18.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.159732+0000) 2020-02-06T01:11:18.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:18.159+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.160829+0000) 2020-02-06T01:11:18.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:18.160+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.161068+0000) 2020-02-06T01:11:18.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:18.160+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.161662+0000) 2020-02-06T01:11:18.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:18.160+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.162246+0000) 2020-02-06T01:11:18.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:18.161+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.162657+0000) 2020-02-06T01:11:18.164 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:18.162+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.163578+0000) 2020-02-06T01:11:18.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:18.166+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:18.167901+0000) 2020-02-06T01:11:18.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:18.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195938 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:19.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:19.707+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:19.708628+0000) 2020-02-06T01:11:19.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:19.955+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:19.956389+0000) 2020-02-06T01:11:20.561 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:20.559+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:20.560381+0000) 2020-02-06T01:11:20.819 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:20.817+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:20.818437+0000) 2020-02-06T01:11:20.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:20.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:20.840059+0000) 2020-02-06T01:11:20.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:20.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:20.840131+0000) 2020-02-06T01:11:22.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:22.942+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:22.943666+0000) 2020-02-06T01:11:23.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:23.172+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.173130+0000) 2020-02-06T01:11:23.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:23.184+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.185492+0000) 2020-02-06T01:11:23.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:23.184+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.186057+0000) 2020-02-06T01:11:23.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:23.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.186749+0000) 2020-02-06T01:11:23.188 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:23.185+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.187228+0000) 2020-02-06T01:11:23.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:23.187+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.188258+0000) 2020-02-06T01:11:23.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:23.186+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.187837+0000) 2020-02-06T01:11:23.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:23.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.189355+0000) 2020-02-06T01:11:23.190 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:23.188+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.189600+0000) 2020-02-06T01:11:23.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:23.188+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.189876+0000) 2020-02-06T01:11:23.191 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:23.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.190041+0000) 2020-02-06T01:11:23.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:23.188+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.190308+0000) 2020-02-06T01:11:23.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:23.189+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.190995+0000) 2020-02-06T01:11:23.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:23.190+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.192147+0000) 2020-02-06T01:11:23.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:23.195+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:23.196764+0000) 2020-02-06T01:11:23.208 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:23.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 195979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:24.519 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:11:24.525 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:11:25.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:25.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:25.333288+0000) 2020-02-06T01:11:26.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:26.080+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:26.081012+0000) 2020-02-06T01:11:27.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:27.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:27.064520+0000) 2020-02-06T01:11:27.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:27.332+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:27.333211+0000) 2020-02-06T01:11:27.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:27.332+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:27.333468+0000) 2020-02-06T01:11:27.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:27.332+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:27.333700+0000) 2020-02-06T01:11:28.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:28.211+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.213029+0000) 2020-02-06T01:11:28.214 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:28.212+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.213468+0000) 2020-02-06T01:11:28.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:28.212+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.214245+0000) 2020-02-06T01:11:28.215 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:28.213+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.214784+0000) 2020-02-06T01:11:28.216 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:28.214+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.215402+0000) 2020-02-06T01:11:28.216 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:28.214+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.215722+0000) 2020-02-06T01:11:28.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:28.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.216536+0000) 2020-02-06T01:11:28.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:28.216+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.218060+0000) 2020-02-06T01:11:28.219 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:28.217+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.218424+0000) 2020-02-06T01:11:28.219 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:28.217+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.218879+0000) 2020-02-06T01:11:28.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:28.217+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.219157+0000) 2020-02-06T01:11:28.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:28.218+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.219555+0000) 2020-02-06T01:11:28.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:28.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.221348+0000) 2020-02-06T01:11:28.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:28.221+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:28.222260+0000) 2020-02-06T01:11:28.234 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:28.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196021 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:29.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:29.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:29.708850+0000) 2020-02-06T01:11:29.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:29.955+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:29.956612+0000) 2020-02-06T01:11:30.561 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:30.560+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:30.560726+0000) 2020-02-06T01:11:30.819 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:30.817+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:30.818670+0000) 2020-02-06T01:11:30.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:30.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:30.840367+0000) 2020-02-06T01:11:30.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:30.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:30.840344+0000) 2020-02-06T01:11:32.944 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:32.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:32.943942+0000) 2020-02-06T01:11:33.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:33.172+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.173359+0000) 2020-02-06T01:11:33.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:33.238+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.239411+0000) 2020-02-06T01:11:33.242 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:33.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.240886+0000) 2020-02-06T01:11:33.242 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:33.240+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.241270+0000) 2020-02-06T01:11:33.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:33.240+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.241599+0000) 2020-02-06T01:11:33.243 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:33.240+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.242001+0000) 2020-02-06T01:11:33.243 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:33.241+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.242449+0000) 2020-02-06T01:11:33.244 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:33.241+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.242780+0000) 2020-02-06T01:11:33.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:33.241+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.242461+0000) 2020-02-06T01:11:33.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:33.242+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.243587+0000) 2020-02-06T01:11:33.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:33.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.244154+0000) 2020-02-06T01:11:33.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:33.242+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.243648+0000) 2020-02-06T01:11:33.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:33.242+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.243770+0000) 2020-02-06T01:11:33.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:33.243+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.244641+0000) 2020-02-06T01:11:33.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:33.247+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:33.248895+0000) 2020-02-06T01:11:33.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:33.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:35.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:35.341+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:35.341873+0000) 2020-02-06T01:11:36.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:36.080+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:36.081272+0000) 2020-02-06T01:11:37.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:37.063+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:37.064772+0000) 2020-02-06T01:11:37.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:37.332+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:37.333496+0000) 2020-02-06T01:11:37.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:37.332+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:37.333717+0000) 2020-02-06T01:11:37.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:37.332+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:37.333930+0000) 2020-02-06T01:11:38.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:38.261+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.263080+0000) 2020-02-06T01:11:38.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:38.262+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.263458+0000) 2020-02-06T01:11:38.265 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:38.263+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.264220+0000) 2020-02-06T01:11:38.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:38.263+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.264932+0000) 2020-02-06T01:11:38.266 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:38.264+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.265245+0000) 2020-02-06T01:11:38.266 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:38.264+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.265681+0000) 2020-02-06T01:11:38.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:38.264+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.266027+0000) 2020-02-06T01:11:38.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:38.265+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.266476+0000) 2020-02-06T01:11:38.268 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:38.265+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.267214+0000) 2020-02-06T01:11:38.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:38.266+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.267405+0000) 2020-02-06T01:11:38.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:38.266+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.267756+0000) 2020-02-06T01:11:38.269 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:38.267+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.269067+0000) 2020-02-06T01:11:38.270 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:38.267+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.269327+0000) 2020-02-06T01:11:38.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:38.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:38.274576+0000) 2020-02-06T01:11:38.287 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:38.284+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:39.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:39.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:39.709130+0000) 2020-02-06T01:11:39.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:39.956+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:39.956846+0000) 2020-02-06T01:11:40.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:40.560+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:40.560983+0000) 2020-02-06T01:11:40.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:40.817+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:40.818893+0000) 2020-02-06T01:11:40.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:40.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:40.840618+0000) 2020-02-06T01:11:40.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:40.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:40.840609+0000) 2020-02-06T01:11:42.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:42.943+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:42.944245+0000) 2020-02-06T01:11:43.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:43.173+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.173598+0000) 2020-02-06T01:11:43.292 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:43.290+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.291760+0000) 2020-02-06T01:11:43.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:43.291+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.292852+0000) 2020-02-06T01:11:43.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:43.292+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.293367+0000) 2020-02-06T01:11:43.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:43.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.293918+0000) 2020-02-06T01:11:43.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:43.293+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.294123+0000) 2020-02-06T01:11:43.295 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:43.293+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.294573+0000) 2020-02-06T01:11:43.296 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:43.293+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.295377+0000) 2020-02-06T01:11:43.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:43.294+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.295660+0000) 2020-02-06T01:11:43.297 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:43.294+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.295799+0000) 2020-02-06T01:11:43.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:43.296+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.297647+0000) 2020-02-06T01:11:43.298 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:43.296+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.297832+0000) 2020-02-06T01:11:43.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:43.296+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.298031+0000) 2020-02-06T01:11:43.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:43.297+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.298703+0000) 2020-02-06T01:11:43.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:43.301+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:43.302117+0000) 2020-02-06T01:11:43.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:43.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196146 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:45.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:45.349+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:45.350363+0000) 2020-02-06T01:11:46.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:46.081+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:46.081532+0000) 2020-02-06T01:11:47.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:47.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:47.065000+0000) 2020-02-06T01:11:47.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:47.333+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:47.333741+0000) 2020-02-06T01:11:47.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:47.333+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:47.333929+0000) 2020-02-06T01:11:47.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:47.333+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:47.334204+0000) 2020-02-06T01:11:48.320 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:48.318+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.319729+0000) 2020-02-06T01:11:48.321 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:48.318+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.319111+0000) 2020-02-06T01:11:48.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:48.320+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.321498+0000) 2020-02-06T01:11:48.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:48.320+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.321144+0000) 2020-02-06T01:11:48.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:48.321+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.322223+0000) 2020-02-06T01:11:48.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:48.321+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.322988+0000) 2020-02-06T01:11:48.324 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:48.322+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.323046+0000) 2020-02-06T01:11:48.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:48.322+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.323868+0000) 2020-02-06T01:11:48.324 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:48.323+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.323510+0000) 2020-02-06T01:11:48.325 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:48.323+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.324607+0000) 2020-02-06T01:11:48.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:48.324+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.325307+0000) 2020-02-06T01:11:48.326 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:48.325+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.325584+0000) 2020-02-06T01:11:48.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:48.324+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.325680+0000) 2020-02-06T01:11:48.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:48.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:48.329920+0000) 2020-02-06T01:11:48.343 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:48.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:49.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:49.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:49.709366+0000) 2020-02-06T01:11:49.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:49.956+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:49.957048+0000) 2020-02-06T01:11:50.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:50.560+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:50.561232+0000) 2020-02-06T01:11:50.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:50.818+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:50.819122+0000) 2020-02-06T01:11:50.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:50.839+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:50.840871+0000) 2020-02-06T01:11:50.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:50.839+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:50.840866+0000) 2020-02-06T01:11:52.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:52.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:52.944502+0000) 2020-02-06T01:11:53.174 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:53.173+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.173820+0000) 2020-02-06T01:11:53.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:53.346+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.347694+0000) 2020-02-06T01:11:53.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:53.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.348656+0000) 2020-02-06T01:11:53.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:53.347+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.348896+0000) 2020-02-06T01:11:53.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:53.348+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.349390+0000) 2020-02-06T01:11:53.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:53.349+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.350214+0000) 2020-02-06T01:11:53.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:53.349+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.350412+0000) 2020-02-06T01:11:53.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:53.351+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.351745+0000) 2020-02-06T01:11:53.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:53.351+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.351512+0000) 2020-02-06T01:11:53.353 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:53.351+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.352095+0000) 2020-02-06T01:11:53.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:53.352+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.352640+0000) 2020-02-06T01:11:53.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:53.352+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.353364+0000) 2020-02-06T01:11:53.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:53.352+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.353804+0000) 2020-02-06T01:11:53.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:53.353+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.354260+0000) 2020-02-06T01:11:53.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:53.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:53.356687+0000) 2020-02-06T01:11:53.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:53.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:54.661 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:11:54.667 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:11:55.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:55.358+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:55.358841+0000) 2020-02-06T01:11:56.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:56.081+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:56.081798+0000) 2020-02-06T01:11:57.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:57.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:57.065265+0000) 2020-02-06T01:11:57.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:57.333+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:57.333991+0000) 2020-02-06T01:11:57.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:57.333+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:57.334168+0000) 2020-02-06T01:11:57.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:57.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:57.334512+0000) 2020-02-06T01:11:58.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:11:58.374+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.375503+0000) 2020-02-06T01:11:58.377 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:11:58.374+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.375993+0000) 2020-02-06T01:11:58.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:11:58.377+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.377796+0000) 2020-02-06T01:11:58.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:58.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.378256+0000) 2020-02-06T01:11:58.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:11:58.377+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.378480+0000) 2020-02-06T01:11:58.380 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:11:58.377+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.378662+0000) 2020-02-06T01:11:58.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:11:58.378+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.379019+0000) 2020-02-06T01:11:58.382 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:11:58.379+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.379851+0000) 2020-02-06T01:11:58.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:11:58.379+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.379591+0000) 2020-02-06T01:11:58.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:11:58.380+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.380424+0000) 2020-02-06T01:11:58.383 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:11:58.380+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.380586+0000) 2020-02-06T01:11:58.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:58.378+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.379759+0000) 2020-02-06T01:11:58.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:58.382+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.383620+0000) 2020-02-06T01:11:58.385 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:11:58.379+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:58.380298+0000) 2020-02-06T01:11:58.399 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:11:58.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196270 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:11:59.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:59.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:59.709634+0000) 2020-02-06T01:11:59.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:11:59.957+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:11:59.957314+0000) 2020-02-06T01:12:00.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:00.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:00.561490+0000) 2020-02-06T01:12:00.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:00.819+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:00.819367+0000) 2020-02-06T01:12:00.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:00.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:00.841130+0000) 2020-02-06T01:12:00.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:00.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:00.841141+0000) 2020-02-06T01:12:02.945 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:02.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:02.944764+0000) 2020-02-06T01:12:03.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:03.174+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.174133+0000) 2020-02-06T01:12:03.406 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:03.405+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.405193+0000) 2020-02-06T01:12:03.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:03.405+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.405652+0000) 2020-02-06T01:12:03.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:03.405+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.406439+0000) 2020-02-06T01:12:03.407 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:03.405+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.406713+0000) 2020-02-06T01:12:03.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:03.406+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.407600+0000) 2020-02-06T01:12:03.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:03.407+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.407712+0000) 2020-02-06T01:12:03.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:03.406+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.407960+0000) 2020-02-06T01:12:03.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:03.407+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.408338+0000) 2020-02-06T01:12:03.409 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:03.407+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.408831+0000) 2020-02-06T01:12:03.410 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:03.408+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.409165+0000) 2020-02-06T01:12:03.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:03.409+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.409644+0000) 2020-02-06T01:12:03.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:03.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.409883+0000) 2020-02-06T01:12:03.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:03.409+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.410390+0000) 2020-02-06T01:12:03.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:03.414+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:03.415163+0000) 2020-02-06T01:12:03.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:03.425+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196312 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:05.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:05.368+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:05.368305+0000) 2020-02-06T01:12:06.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:06.081+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:06.082064+0000) 2020-02-06T01:12:07.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:07.065+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:07.065500+0000) 2020-02-06T01:12:07.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:07.333+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:07.334246+0000) 2020-02-06T01:12:07.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:07.333+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:07.334404+0000) 2020-02-06T01:12:07.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:07.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:07.334723+0000) 2020-02-06T01:12:08.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:08.431+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.431983+0000) 2020-02-06T01:12:08.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:08.431+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.431781+0000) 2020-02-06T01:12:08.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:08.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.433430+0000) 2020-02-06T01:12:08.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:08.433+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.433611+0000) 2020-02-06T01:12:08.435 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:08.434+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.434391+0000) 2020-02-06T01:12:08.435 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:08.434+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.434981+0000) 2020-02-06T01:12:08.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:08.434+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.434818+0000) 2020-02-06T01:12:08.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:08.434+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.435362+0000) 2020-02-06T01:12:08.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:08.435+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.436301+0000) 2020-02-06T01:12:08.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:08.436+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.436453+0000) 2020-02-06T01:12:08.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:08.436+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.436855+0000) 2020-02-06T01:12:08.438 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:08.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.437291+0000) 2020-02-06T01:12:08.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:08.440+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.440895+0000) 2020-02-06T01:12:08.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:08.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:08.441124+0000) 2020-02-06T01:12:08.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:08.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196353 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:09.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:09.708+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:09.709845+0000) 2020-02-06T01:12:09.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:09.956+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:09.957548+0000) 2020-02-06T01:12:10.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:10.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:10.561714+0000) 2020-02-06T01:12:10.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:10.818+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:10.819622+0000) 2020-02-06T01:12:10.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:10.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:10.841368+0000) 2020-02-06T01:12:10.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:10.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:10.841411+0000) 2020-02-06T01:12:12.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:12.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:12.945019+0000) 2020-02-06T01:12:13.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:13.173+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.174362+0000) 2020-02-06T01:12:13.459 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:13.457+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.458289+0000) 2020-02-06T01:12:13.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:13.458+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.459122+0000) 2020-02-06T01:12:13.460 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:13.458+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.459645+0000) 2020-02-06T01:12:13.460 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:13.458+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.459853+0000) 2020-02-06T01:12:13.461 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:13.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.460011+0000) 2020-02-06T01:12:13.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:13.459+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.460352+0000) 2020-02-06T01:12:13.461 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:13.459+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.460786+0000) 2020-02-06T01:12:13.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:13.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.461266+0000) 2020-02-06T01:12:13.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:13.461+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.461965+0000) 2020-02-06T01:12:13.468 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:13.461+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.462254+0000) 2020-02-06T01:12:13.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:13.461+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.462237+0000) 2020-02-06T01:12:13.468 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:13.462+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.462923+0000) 2020-02-06T01:12:13.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:13.466+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.467901+0000) 2020-02-06T01:12:13.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:13.467+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:13.468182+0000) 2020-02-06T01:12:13.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:13.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196395 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:15.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:15.376+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:15.377127+0000) 2020-02-06T01:12:16.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:16.082+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:16.082333+0000) 2020-02-06T01:12:17.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:17.064+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:17.065810+0000) 2020-02-06T01:12:17.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:17.334+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:17.334479+0000) 2020-02-06T01:12:17.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:17.334+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:17.334703+0000) 2020-02-06T01:12:17.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:17.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:17.334993+0000) 2020-02-06T01:12:18.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:18.483+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.484434+0000) 2020-02-06T01:12:18.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:18.484+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.485758+0000) 2020-02-06T01:12:18.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:18.485+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.486228+0000) 2020-02-06T01:12:18.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:18.486+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.487234+0000) 2020-02-06T01:12:18.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:18.486+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.487509+0000) 2020-02-06T01:12:18.488 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:18.486+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.487984+0000) 2020-02-06T01:12:18.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:18.487+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.488351+0000) 2020-02-06T01:12:18.489 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:18.487+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.488627+0000) 2020-02-06T01:12:18.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:18.488+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.489401+0000) 2020-02-06T01:12:18.490 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:18.489+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.490124+0000) 2020-02-06T01:12:18.491 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:18.489+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.490245+0000) 2020-02-06T01:12:18.491 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:18.489+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.490433+0000) 2020-02-06T01:12:18.492 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:18.490+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.490911+0000) 2020-02-06T01:12:18.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:18.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:18.495905+0000) 2020-02-06T01:12:18.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:18.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196437 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:19.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:19.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:19.710129+0000) 2020-02-06T01:12:19.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:19.956+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:19.957809+0000) 2020-02-06T01:12:20.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:20.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:20.561947+0000) 2020-02-06T01:12:20.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:20.818+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:20.819867+0000) 2020-02-06T01:12:20.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:20.840+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:20.841602+0000) 2020-02-06T01:12:20.842 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:20.840+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:20.841633+0000) 2020-02-06T01:12:22.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:22.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:22.945345+0000) 2020-02-06T01:12:23.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:23.173+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.174530+0000) 2020-02-06T01:12:23.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:23.510+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.511836+0000) 2020-02-06T01:12:23.513 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:23.511+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.512190+0000) 2020-02-06T01:12:23.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:23.512+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.513170+0000) 2020-02-06T01:12:23.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:23.511+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.512771+0000) 2020-02-06T01:12:23.515 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:23.512+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.513769+0000) 2020-02-06T01:12:23.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:23.513+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.514405+0000) 2020-02-06T01:12:23.516 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:23.514+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.515222+0000) 2020-02-06T01:12:23.517 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:23.514+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.514916+0000) 2020-02-06T01:12:23.517 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:23.515+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.516149+0000) 2020-02-06T01:12:23.518 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:23.515+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.516429+0000) 2020-02-06T01:12:23.518 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:23.516+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.517152+0000) 2020-02-06T01:12:23.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:23.518+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.518493+0000) 2020-02-06T01:12:23.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:23.518+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.518595+0000) 2020-02-06T01:12:23.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:23.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:23.522688+0000) 2020-02-06T01:12:23.534 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:23.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:24.804 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:12:24.810 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:12:25.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:25.386+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:25.387040+0000) 2020-02-06T01:12:26.083 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:26.081+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:26.082609+0000) 2020-02-06T01:12:27.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:27.065+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:27.066105+0000) 2020-02-06T01:12:27.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:27.333+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:27.334718+0000) 2020-02-06T01:12:27.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:27.333+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:27.334927+0000) 2020-02-06T01:12:27.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:27.333+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:27.335249+0000) 2020-02-06T01:12:28.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:28.538+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.539469+0000) 2020-02-06T01:12:28.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:28.538+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.539995+0000) 2020-02-06T01:12:28.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:28.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.540435+0000) 2020-02-06T01:12:28.541 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:28.539+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.540700+0000) 2020-02-06T01:12:28.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:28.539+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.540978+0000) 2020-02-06T01:12:28.542 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:28.540+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.541597+0000) 2020-02-06T01:12:28.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:28.540+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.542205+0000) 2020-02-06T01:12:28.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:28.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.542950+0000) 2020-02-06T01:12:28.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:28.541+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.543235+0000) 2020-02-06T01:12:28.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:28.543+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.544282+0000) 2020-02-06T01:12:28.545 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:28.543+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.544434+0000) 2020-02-06T01:12:28.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:28.543+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.544758+0000) 2020-02-06T01:12:28.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:28.544+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.545593+0000) 2020-02-06T01:12:28.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:28.547+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:28.549001+0000) 2020-02-06T01:12:28.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:28.558+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:29.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:29.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:29.710367+0000) 2020-02-06T01:12:29.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:29.957+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:29.958032+0000) 2020-02-06T01:12:30.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:30.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:30.562202+0000) 2020-02-06T01:12:30.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:30.819+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:30.820144+0000) 2020-02-06T01:12:30.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:30.841+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:30.841833+0000) 2020-02-06T01:12:30.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:30.841+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:30.841911+0000) 2020-02-06T01:12:32.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:32.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:32.945650+0000) 2020-02-06T01:12:33.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:33.174+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.174763+0000) 2020-02-06T01:12:33.568 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:33.566+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.567406+0000) 2020-02-06T01:12:33.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:33.565+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.567337+0000) 2020-02-06T01:12:33.569 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:33.567+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.568203+0000) 2020-02-06T01:12:33.569 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:33.567+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.568864+0000) 2020-02-06T01:12:33.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:33.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.569232+0000) 2020-02-06T01:12:33.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:33.568+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.569657+0000) 2020-02-06T01:12:33.571 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:33.568+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.570231+0000) 2020-02-06T01:12:33.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:33.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.570940+0000) 2020-02-06T01:12:33.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:33.569+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.571140+0000) 2020-02-06T01:12:33.572 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:33.570+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.571295+0000) 2020-02-06T01:12:33.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:33.570+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.571727+0000) 2020-02-06T01:12:33.573 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:33.570+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.572254+0000) 2020-02-06T01:12:33.573 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:33.571+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.572226+0000) 2020-02-06T01:12:33.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:33.576+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:33.578067+0000) 2020-02-06T01:12:33.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:33.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196560 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:35.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:35.394+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:35.395227+0000) 2020-02-06T01:12:36.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:36.081+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:36.082889+0000) 2020-02-06T01:12:37.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:37.065+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:37.066400+0000) 2020-02-06T01:12:37.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:37.334+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:37.334960+0000) 2020-02-06T01:12:37.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:37.334+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:37.335148+0000) 2020-02-06T01:12:37.336 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:37.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:37.335522+0000) 2020-02-06T01:12:38.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:38.592+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.593754+0000) 2020-02-06T01:12:38.599 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:38.594+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.596261+0000) 2020-02-06T01:12:38.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:38.595+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.596945+0000) 2020-02-06T01:12:38.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:38.595+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.597412+0000) 2020-02-06T01:12:38.600 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:38.592+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.593276+0000) 2020-02-06T01:12:38.600 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:38.593+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.594228+0000) 2020-02-06T01:12:38.600 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:38.593+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.594897+0000) 2020-02-06T01:12:38.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:38.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.596401+0000) 2020-02-06T01:12:38.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:38.597+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.598158+0000) 2020-02-06T01:12:38.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:38.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.598489+0000) 2020-02-06T01:12:38.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:38.597+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.599041+0000) 2020-02-06T01:12:38.602 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:38.598+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.599459+0000) 2020-02-06T01:12:38.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:38.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.599101+0000) 2020-02-06T01:12:38.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:38.603+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:38.604841+0000) 2020-02-06T01:12:38.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:38.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:39.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:39.709+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:39.710575+0000) 2020-02-06T01:12:39.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:39.957+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:39.958291+0000) 2020-02-06T01:12:40.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:40.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:40.562481+0000) 2020-02-06T01:12:40.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:40.819+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:40.820368+0000) 2020-02-06T01:12:40.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:40.841+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:40.842120+0000) 2020-02-06T01:12:40.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:40.841+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:40.842171+0000) 2020-02-06T01:12:42.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:42.944+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:42.945861+0000) 2020-02-06T01:12:43.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:43.174+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.174999+0000) 2020-02-06T01:12:43.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:43.618+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.619935+0000) 2020-02-06T01:12:43.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:43.619+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.620834+0000) 2020-02-06T01:12:43.622 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:43.620+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.622005+0000) 2020-02-06T01:12:43.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:43.621+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.622253+0000) 2020-02-06T01:12:43.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:43.622+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.623506+0000) 2020-02-06T01:12:43.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:43.621+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.623202+0000) 2020-02-06T01:12:43.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:43.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.623745+0000) 2020-02-06T01:12:43.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:43.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.624338+0000) 2020-02-06T01:12:43.625 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:43.622+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.624168+0000) 2020-02-06T01:12:43.626 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:43.623+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.625033+0000) 2020-02-06T01:12:43.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:43.624+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.625365+0000) 2020-02-06T01:12:43.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:43.624+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.625569+0000) 2020-02-06T01:12:43.627 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:43.624+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.625661+0000) 2020-02-06T01:12:43.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:43.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:43.629049+0000) 2020-02-06T01:12:43.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:43.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196644 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:45.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:45.403+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:45.403548+0000) 2020-02-06T01:12:46.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:46.082+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:46.083142+0000) 2020-02-06T01:12:47.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:47.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:47.066649+0000) 2020-02-06T01:12:47.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:47.334+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:47.335205+0000) 2020-02-06T01:12:47.336 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:47.334+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:47.335384+0000) 2020-02-06T01:12:47.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:47.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:47.335775+0000) 2020-02-06T01:12:48.649 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:48.647+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.648861+0000) 2020-02-06T01:12:48.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:48.648+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.649510+0000) 2020-02-06T01:12:48.650 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:48.648+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.649972+0000) 2020-02-06T01:12:48.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:48.648+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.650367+0000) 2020-02-06T01:12:48.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:48.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.650887+0000) 2020-02-06T01:12:48.652 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:48.649+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.650923+0000) 2020-02-06T01:12:48.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:48.650+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.651402+0000) 2020-02-06T01:12:48.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:48.651+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.652689+0000) 2020-02-06T01:12:48.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:48.651+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.652870+0000) 2020-02-06T01:12:48.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:48.651+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.653104+0000) 2020-02-06T01:12:48.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:48.652+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.653503+0000) 2020-02-06T01:12:48.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:48.652+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.653624+0000) 2020-02-06T01:12:48.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:48.652+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.653735+0000) 2020-02-06T01:12:48.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:48.657+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:48.658896+0000) 2020-02-06T01:12:48.667 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:48.665+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:49.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:49.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:49.710810+0000) 2020-02-06T01:12:49.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:49.957+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:49.958518+0000) 2020-02-06T01:12:50.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:50.561+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:50.562731+0000) 2020-02-06T01:12:50.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:50.820+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:50.820607+0000) 2020-02-06T01:12:50.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:50.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:50.842354+0000) 2020-02-06T01:12:50.843 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:50.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:50.842427+0000) 2020-02-06T01:12:52.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:52.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:52.946107+0000) 2020-02-06T01:12:53.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:53.175+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.175308+0000) 2020-02-06T01:12:53.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:53.671+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.673035+0000) 2020-02-06T01:12:53.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:53.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.673466+0000) 2020-02-06T01:12:53.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:53.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.673724+0000) 2020-02-06T01:12:53.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:53.672+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.674121+0000) 2020-02-06T01:12:53.675 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:53.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.674701+0000) 2020-02-06T01:12:53.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:53.674+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.675418+0000) 2020-02-06T01:12:53.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:53.674+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.675815+0000) 2020-02-06T01:12:53.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:53.674+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.675531+0000) 2020-02-06T01:12:53.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:53.674+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.675597+0000) 2020-02-06T01:12:53.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:53.675+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.676190+0000) 2020-02-06T01:12:53.678 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:53.676+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.677469+0000) 2020-02-06T01:12:53.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:53.676+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.677852+0000) 2020-02-06T01:12:53.679 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:53.677+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.678173+0000) 2020-02-06T01:12:53.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:53.683+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:53.684851+0000) 2020-02-06T01:12:53.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:53.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196728 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:54.946 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:12:54.953 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:12:55.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:55.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:55.411648+0000) 2020-02-06T01:12:56.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:56.082+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:56.083417+0000) 2020-02-06T01:12:57.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:57.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:57.066897+0000) 2020-02-06T01:12:57.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:57.334+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:57.335485+0000) 2020-02-06T01:12:57.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:57.334+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:57.335702+0000) 2020-02-06T01:12:57.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:57.334+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:57.336004+0000) 2020-02-06T01:12:58.699 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:12:58.697+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.698826+0000) 2020-02-06T01:12:58.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:12:58.698+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.699419+0000) 2020-02-06T01:12:58.701 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:12:58.699+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.700373+0000) 2020-02-06T01:12:58.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:58.700+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.701256+0000) 2020-02-06T01:12:58.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:12:58.700+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.701523+0000) 2020-02-06T01:12:58.703 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:12:58.700+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.702182+0000) 2020-02-06T01:12:58.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:58.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.702845+0000) 2020-02-06T01:12:58.703 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:12:58.701+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.702857+0000) 2020-02-06T01:12:58.704 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:12:58.702+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.703681+0000) 2020-02-06T01:12:58.704 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:12:58.702+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.703997+0000) 2020-02-06T01:12:58.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:12:58.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.704831+0000) 2020-02-06T01:12:58.706 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:12:58.703+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.705376+0000) 2020-02-06T01:12:58.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:12:58.704+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.706260+0000) 2020-02-06T01:12:58.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:58.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:58.708886+0000) 2020-02-06T01:12:58.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:12:58.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:12:59.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:59.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:59.711068+0000) 2020-02-06T01:12:59.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:12:59.958+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:12:59.958767+0000) 2020-02-06T01:13:00.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:00.562+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:00.562977+0000) 2020-02-06T01:13:00.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:00.820+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:00.820859+0000) 2020-02-06T01:13:00.843 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:00.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:00.842605+0000) 2020-02-06T01:13:00.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:00.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:00.842710+0000) 2020-02-06T01:13:02.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:02.945+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:02.946361+0000) 2020-02-06T01:13:03.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:03.175+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.175524+0000) 2020-02-06T01:13:03.728 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:03.727+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.727072+0000) 2020-02-06T01:13:03.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:03.727+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.727511+0000) 2020-02-06T01:13:03.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:03.728+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.728190+0000) 2020-02-06T01:13:03.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:03.728+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.728723+0000) 2020-02-06T01:13:03.730 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:03.727+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.729289+0000) 2020-02-06T01:13:03.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:03.728+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.728961+0000) 2020-02-06T01:13:03.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:03.728+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.729829+0000) 2020-02-06T01:13:03.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:03.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.730606+0000) 2020-02-06T01:13:03.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:03.730+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.730263+0000) 2020-02-06T01:13:03.732 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:03.731+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.731249+0000) 2020-02-06T01:13:03.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:03.730+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.732125+0000) 2020-02-06T01:13:03.734 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:03.731+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.733191+0000) 2020-02-06T01:13:03.734 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:03.731+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.732651+0000) 2020-02-06T01:13:03.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:03.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:03.737629+0000) 2020-02-06T01:13:03.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:03.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:05.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:05.418+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:05.419156+0000) 2020-02-06T01:13:06.084 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:06.083+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:06.083707+0000) 2020-02-06T01:13:07.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:07.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:07.067123+0000) 2020-02-06T01:13:07.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:07.335+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:07.335720+0000) 2020-02-06T01:13:07.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:07.335+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:07.335939+0000) 2020-02-06T01:13:07.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:07.335+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:07.336246+0000) 2020-02-06T01:13:08.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:08.755+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.756141+0000) 2020-02-06T01:13:08.757 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:08.755+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.756997+0000) 2020-02-06T01:13:08.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:08.756+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.757697+0000) 2020-02-06T01:13:08.759 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:08.756+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.757566+0000) 2020-02-06T01:13:08.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:08.756+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.758032+0000) 2020-02-06T01:13:08.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:08.757+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.758424+0000) 2020-02-06T01:13:08.759 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:08.757+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.758818+0000) 2020-02-06T01:13:08.760 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:08.758+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.759420+0000) 2020-02-06T01:13:08.760 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:08.758+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.759711+0000) 2020-02-06T01:13:08.761 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:08.759+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.760835+0000) 2020-02-06T01:13:08.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:08.758+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.760161+0000) 2020-02-06T01:13:08.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:08.759+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.760946+0000) 2020-02-06T01:13:08.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:08.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.764646+0000) 2020-02-06T01:13:08.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:08.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:08.765031+0000) 2020-02-06T01:13:08.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:08.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:09.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:09.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:09.711331+0000) 2020-02-06T01:13:09.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:09.957+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:09.959018+0000) 2020-02-06T01:13:10.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:10.562+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:10.563224+0000) 2020-02-06T01:13:10.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:10.820+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:10.821117+0000) 2020-02-06T01:13:10.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:10.841+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:10.842833+0000) 2020-02-06T01:13:10.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:10.841+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:10.842937+0000) 2020-02-06T01:13:12.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:12.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:12.946679+0000) 2020-02-06T01:13:13.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:13.174+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.175741+0000) 2020-02-06T01:13:13.782 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:13.780+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.781161+0000) 2020-02-06T01:13:13.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:13.780+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.781563+0000) 2020-02-06T01:13:13.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:13.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.782180+0000) 2020-02-06T01:13:13.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:13.781+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.782635+0000) 2020-02-06T01:13:13.784 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:13.781+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.782616+0000) 2020-02-06T01:13:13.784 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:13.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.783277+0000) 2020-02-06T01:13:13.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:13.784+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.785818+0000) 2020-02-06T01:13:13.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:13.784+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.786031+0000) 2020-02-06T01:13:13.787 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:13.785+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.786824+0000) 2020-02-06T01:13:13.788 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:13.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.787050+0000) 2020-02-06T01:13:13.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:13.785+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.787391+0000) 2020-02-06T01:13:13.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:13.786+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.788113+0000) 2020-02-06T01:13:13.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:13.792+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.793518+0000) 2020-02-06T01:13:13.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:13.793+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:13.794899+0000) 2020-02-06T01:13:13.804 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:13.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:15.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:15.426+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:15.427317+0000) 2020-02-06T01:13:16.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:16.083+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:16.083953+0000) 2020-02-06T01:13:17.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:17.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:17.067353+0000) 2020-02-06T01:13:17.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:17.335+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:17.335975+0000) 2020-02-06T01:13:17.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:17.335+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:17.336217+0000) 2020-02-06T01:13:17.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:17.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:17.336503+0000) 2020-02-06T01:13:18.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:18.808+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.809123+0000) 2020-02-06T01:13:18.811 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:18.809+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.810859+0000) 2020-02-06T01:13:18.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:18.810+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.810617+0000) 2020-02-06T01:13:18.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:18.810+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.811313+0000) 2020-02-06T01:13:18.812 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:18.811+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.811866+0000) 2020-02-06T01:13:18.813 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:18.812+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.812664+0000) 2020-02-06T01:13:18.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:18.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.813539+0000) 2020-02-06T01:13:18.814 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:18.812+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.813959+0000) 2020-02-06T01:13:18.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:18.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.814794+0000) 2020-02-06T01:13:18.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:18.814+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.815031+0000) 2020-02-06T01:13:18.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:18.815+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.815600+0000) 2020-02-06T01:13:18.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:18.814+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.815702+0000) 2020-02-06T01:13:18.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:18.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.815849+0000) 2020-02-06T01:13:18.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:18.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:18.818411+0000) 2020-02-06T01:13:18.831 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:18.830+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:19.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:19.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:19.711528+0000) 2020-02-06T01:13:19.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:19.958+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:19.959311+0000) 2020-02-06T01:13:20.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:20.563+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:20.563519+0000) 2020-02-06T01:13:20.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:20.820+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:20.821345+0000) 2020-02-06T01:13:20.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:20.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:20.843063+0000) 2020-02-06T01:13:20.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:20.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:20.843161+0000) 2020-02-06T01:13:22.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:22.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:22.946911+0000) 2020-02-06T01:13:23.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:23.174+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.175990+0000) 2020-02-06T01:13:23.837 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:23.835+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.836824+0000) 2020-02-06T01:13:23.838 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:23.837+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.837726+0000) 2020-02-06T01:13:23.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:23.837+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.838157+0000) 2020-02-06T01:13:23.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:23.839+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.839857+0000) 2020-02-06T01:13:23.841 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:23.839+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.839735+0000) 2020-02-06T01:13:23.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:23.839+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.840399+0000) 2020-02-06T01:13:23.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:23.840+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.841162+0000) 2020-02-06T01:13:23.842 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:23.840+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.841383+0000) 2020-02-06T01:13:23.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:23.840+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.841772+0000) 2020-02-06T01:13:23.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:23.840+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.841887+0000) 2020-02-06T01:13:23.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:23.841+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.842040+0000) 2020-02-06T01:13:23.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:23.841+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.842665+0000) 2020-02-06T01:13:23.844 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:23.842+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.842588+0000) 2020-02-06T01:13:23.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:23.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:23.846987+0000) 2020-02-06T01:13:23.859 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:23.858+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 196981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:25.077 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:13:25.085 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:13:25.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:25.434+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:25.435319+0000) 2020-02-06T01:13:26.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:26.083+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:26.084224+0000) 2020-02-06T01:13:27.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:27.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:27.067582+0000) 2020-02-06T01:13:27.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:27.335+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:27.336230+0000) 2020-02-06T01:13:27.337 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:27.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:27.336490+0000) 2020-02-06T01:13:27.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:27.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:27.336736+0000) 2020-02-06T01:13:28.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:28.865+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.865829+0000) 2020-02-06T01:13:28.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:28.865+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.866171+0000) 2020-02-06T01:13:28.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:28.865+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.866457+0000) 2020-02-06T01:13:28.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:28.865+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.866795+0000) 2020-02-06T01:13:28.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:28.866+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.867212+0000) 2020-02-06T01:13:28.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:28.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.867474+0000) 2020-02-06T01:13:28.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:28.866+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.867732+0000) 2020-02-06T01:13:28.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:28.867+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.867918+0000) 2020-02-06T01:13:28.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:28.867+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.868025+0000) 2020-02-06T01:13:28.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:28.867+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.868371+0000) 2020-02-06T01:13:28.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:28.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.868722+0000) 2020-02-06T01:13:28.870 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:28.868+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.868817+0000) 2020-02-06T01:13:28.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:28.868+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.869169+0000) 2020-02-06T01:13:28.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:28.872+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:28.873540+0000) 2020-02-06T01:13:28.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:28.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:29.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:29.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:29.711741+0000) 2020-02-06T01:13:29.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:29.958+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:29.959547+0000) 2020-02-06T01:13:30.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:30.563+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:30.563715+0000) 2020-02-06T01:13:30.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:30.821+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:30.821574+0000) 2020-02-06T01:13:30.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:30.842+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:30.843323+0000) 2020-02-06T01:13:30.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:30.842+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:30.843381+0000) 2020-02-06T01:13:32.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:32.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:32.947166+0000) 2020-02-06T01:13:33.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:33.175+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.176229+0000) 2020-02-06T01:13:33.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:33.889+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.890132+0000) 2020-02-06T01:13:33.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:33.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.891421+0000) 2020-02-06T01:13:33.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:33.891+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.891552+0000) 2020-02-06T01:13:33.893 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:33.890+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.892004+0000) 2020-02-06T01:13:33.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:33.891+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.892119+0000) 2020-02-06T01:13:33.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:33.891+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.892560+0000) 2020-02-06T01:13:33.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:33.891+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.892707+0000) 2020-02-06T01:13:33.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:33.893+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.893639+0000) 2020-02-06T01:13:33.894 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:33.893+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.893419+0000) 2020-02-06T01:13:33.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:33.893+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.894450+0000) 2020-02-06T01:13:33.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:33.893+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.894222+0000) 2020-02-06T01:13:33.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:33.894+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.895111+0000) 2020-02-06T01:13:33.896 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:33.895+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.895646+0000) 2020-02-06T01:13:33.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:33.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:33.901827+0000) 2020-02-06T01:13:33.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:33.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197066 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:35.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:35.441+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:35.442249+0000) 2020-02-06T01:13:36.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:36.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:36.084501+0000) 2020-02-06T01:13:37.068 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:37.067+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:37.067813+0000) 2020-02-06T01:13:37.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:37.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:37.336487+0000) 2020-02-06T01:13:37.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:37.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:37.336725+0000) 2020-02-06T01:13:37.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:37.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:37.336976+0000) 2020-02-06T01:13:38.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:38.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.918471+0000) 2020-02-06T01:13:38.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:38.918+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.919529+0000) 2020-02-06T01:13:38.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:38.918+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.919618+0000) 2020-02-06T01:13:38.921 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:38.919+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.920312+0000) 2020-02-06T01:13:38.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:38.919+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.920612+0000) 2020-02-06T01:13:38.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:38.920+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.921122+0000) 2020-02-06T01:13:38.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:38.921+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.921731+0000) 2020-02-06T01:13:38.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:38.921+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.922115+0000) 2020-02-06T01:13:38.923 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:38.922+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.922552+0000) 2020-02-06T01:13:38.924 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:38.921+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.922894+0000) 2020-02-06T01:13:38.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:38.922+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.923303+0000) 2020-02-06T01:13:38.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:38.922+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.924023+0000) 2020-02-06T01:13:38.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:38.923+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.924355+0000) 2020-02-06T01:13:38.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:38.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:38.929803+0000) 2020-02-06T01:13:38.938 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:38.936+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:39.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:39.710+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:39.711969+0000) 2020-02-06T01:13:39.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:39.959+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:39.959781+0000) 2020-02-06T01:13:40.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:40.563+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:40.563949+0000) 2020-02-06T01:13:40.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:40.821+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:40.821811+0000) 2020-02-06T01:13:40.844 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:40.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:40.843567+0000) 2020-02-06T01:13:40.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:40.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:40.843607+0000) 2020-02-06T01:13:42.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:42.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:42.947478+0000) 2020-02-06T01:13:43.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:43.176+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.176436+0000) 2020-02-06T01:13:43.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:43.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.943041+0000) 2020-02-06T01:13:43.944 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:43.942+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.943413+0000) 2020-02-06T01:13:43.945 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:43.943+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.944361+0000) 2020-02-06T01:13:43.945 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:43.943+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.944568+0000) 2020-02-06T01:13:43.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:43.944+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.945183+0000) 2020-02-06T01:13:43.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:43.944+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.945629+0000) 2020-02-06T01:13:43.946 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:43.945+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.946104+0000) 2020-02-06T01:13:43.947 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:43.946+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.946888+0000) 2020-02-06T01:13:43.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:43.946+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.947416+0000) 2020-02-06T01:13:43.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:43.946+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.947981+0000) 2020-02-06T01:13:43.949 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:43.947+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.948300+0000) 2020-02-06T01:13:43.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:43.948+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.948729+0000) 2020-02-06T01:13:43.950 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:43.949+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.949645+0000) 2020-02-06T01:13:43.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:43.953+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:43.954817+0000) 2020-02-06T01:13:43.965 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:43.963+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197151 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:45.450 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:45.448+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:45.448930+0000) 2020-02-06T01:13:46.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:46.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:46.084742+0000) 2020-02-06T01:13:47.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:47.066+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:47.068058+0000) 2020-02-06T01:13:47.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:47.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:47.336795+0000) 2020-02-06T01:13:47.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:47.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:47.336981+0000) 2020-02-06T01:13:47.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:47.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:47.337223+0000) 2020-02-06T01:13:48.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:48.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.969882+0000) 2020-02-06T01:13:48.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:48.969+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.970517+0000) 2020-02-06T01:13:48.972 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:48.970+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.971462+0000) 2020-02-06T01:13:48.973 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:48.971+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.972055+0000) 2020-02-06T01:13:48.973 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:48.972+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.972610+0000) 2020-02-06T01:13:48.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:48.971+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.972900+0000) 2020-02-06T01:13:48.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:48.973+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.973683+0000) 2020-02-06T01:13:48.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:48.973+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.974474+0000) 2020-02-06T01:13:48.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:48.974+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.975178+0000) 2020-02-06T01:13:48.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:48.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.975523+0000) 2020-02-06T01:13:48.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:48.975+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.975548+0000) 2020-02-06T01:13:48.978 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:48.975+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.976020+0000) 2020-02-06T01:13:48.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:48.975+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.976459+0000) 2020-02-06T01:13:48.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:48.980+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:48.981858+0000) 2020-02-06T01:13:48.992 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:48.990+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197193 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:49.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:49.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:49.712185+0000) 2020-02-06T01:13:49.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:49.959+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:49.960008+0000) 2020-02-06T01:13:50.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:50.564+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:50.564211+0000) 2020-02-06T01:13:50.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:50.821+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:50.822106+0000) 2020-02-06T01:13:50.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:50.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:50.843844+0000) 2020-02-06T01:13:50.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:50.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:50.843847+0000) 2020-02-06T01:13:52.948 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:52.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:52.947721+0000) 2020-02-06T01:13:53.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:53.176+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:53.176660+0000) 2020-02-06T01:13:53.999 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:53.997+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:53.998477+0000) 2020-02-06T01:13:54.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:53.999+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.000031+0000) 2020-02-06T01:13:54.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:53.999+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.000161+0000) 2020-02-06T01:13:54.001 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:54.000+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.000579+0000) 2020-02-06T01:13:54.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:54.000+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.000895+0000) 2020-02-06T01:13:54.002 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:54.001+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.001208+0000) 2020-02-06T01:13:54.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:54.001+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.001577+0000) 2020-02-06T01:13:54.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:54.001+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.002012+0000) 2020-02-06T01:13:54.003 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:54.001+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.002124+0000) 2020-02-06T01:13:54.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:54.001+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.002680+0000) 2020-02-06T01:13:54.004 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:54.002+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.002837+0000) 2020-02-06T01:13:54.004 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:54.002+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.003035+0000) 2020-02-06T01:13:54.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:54.002+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.004018+0000) 2020-02-06T01:13:54.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:54.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:54.004948+0000) 2020-02-06T01:13:54.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:54.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:55.218 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:13:55.225 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:13:55.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:55.457+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:55.457177+0000) 2020-02-06T01:13:56.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:56.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:56.085000+0000) 2020-02-06T01:13:57.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:57.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:57.068311+0000) 2020-02-06T01:13:57.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:57.335+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:57.337047+0000) 2020-02-06T01:13:57.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:57.335+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:57.337218+0000) 2020-02-06T01:13:57.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:57.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:57.337507+0000) 2020-02-06T01:13:59.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:13:59.021+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.022466+0000) 2020-02-06T01:13:59.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:13:59.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.023716+0000) 2020-02-06T01:13:59.024 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:13:59.022+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.023587+0000) 2020-02-06T01:13:59.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:13:59.023+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.024199+0000) 2020-02-06T01:13:59.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:13:59.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.025418+0000) 2020-02-06T01:13:59.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:13:59.023+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.025292+0000) 2020-02-06T01:13:59.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:59.026+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.027305+0000) 2020-02-06T01:13:59.028 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:13:59.025+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.027372+0000) 2020-02-06T01:13:59.028 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:13:59.026+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.027686+0000) 2020-02-06T01:13:59.029 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:13:59.027+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.028203+0000) 2020-02-06T01:13:59.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:13:59.026+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.028117+0000) 2020-02-06T01:13:59.029 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:13:59.027+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.029249+0000) 2020-02-06T01:13:59.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:59.034+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.035139+0000) 2020-02-06T01:13:59.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:59.034+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.035461+0000) 2020-02-06T01:13:59.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:13:59.042+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:13:59.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:59.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.712421+0000) 2020-02-06T01:13:59.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:13:59.959+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:13:59.960229+0000) 2020-02-06T01:14:00.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:00.563+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:00.564478+0000) 2020-02-06T01:14:00.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:00.821+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:00.822352+0000) 2020-02-06T01:14:00.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:00.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:00.844131+0000) 2020-02-06T01:14:00.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:00.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:00.844129+0000) 2020-02-06T01:14:02.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:02.946+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:02.947968+0000) 2020-02-06T01:14:03.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:03.175+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:03.176890+0000) 2020-02-06T01:14:04.050 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:04.048+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.049673+0000) 2020-02-06T01:14:04.052 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:04.049+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.050542+0000) 2020-02-06T01:14:04.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:04.050+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.051680+0000) 2020-02-06T01:14:04.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:04.051+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.052286+0000) 2020-02-06T01:14:04.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:04.051+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.052722+0000) 2020-02-06T01:14:04.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:04.052+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.053430+0000) 2020-02-06T01:14:04.054 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:04.052+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.053487+0000) 2020-02-06T01:14:04.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:04.053+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.054361+0000) 2020-02-06T01:14:04.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:04.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.054621+0000) 2020-02-06T01:14:04.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:04.053+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.055366+0000) 2020-02-06T01:14:04.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:04.054+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.056005+0000) 2020-02-06T01:14:04.057 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:04.054+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.056345+0000) 2020-02-06T01:14:04.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:04.056+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.057232+0000) 2020-02-06T01:14:04.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:04.058+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:04.059305+0000) 2020-02-06T01:14:04.072 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:04.070+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197321 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:05.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:05.464+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:05.465349+0000) 2020-02-06T01:14:06.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:06.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:06.085319+0000) 2020-02-06T01:14:07.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:07.067+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:07.068539+0000) 2020-02-06T01:14:07.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:07.335+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:07.337293+0000) 2020-02-06T01:14:07.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:07.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:07.337499+0000) 2020-02-06T01:14:07.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:07.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:07.337775+0000) 2020-02-06T01:14:09.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:09.075+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.076965+0000) 2020-02-06T01:14:09.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:09.076+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.077376+0000) 2020-02-06T01:14:09.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:09.076+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.077825+0000) 2020-02-06T01:14:09.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:09.077+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.078461+0000) 2020-02-06T01:14:09.079 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:09.077+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.078172+0000) 2020-02-06T01:14:09.080 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:09.077+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.078683+0000) 2020-02-06T01:14:09.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:09.077+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.078950+0000) 2020-02-06T01:14:09.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:09.078+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.079197+0000) 2020-02-06T01:14:09.081 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:09.078+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.079782+0000) 2020-02-06T01:14:09.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:09.079+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.080578+0000) 2020-02-06T01:14:09.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:09.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.080907+0000) 2020-02-06T01:14:09.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:09.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.081229+0000) 2020-02-06T01:14:09.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:09.080+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.081658+0000) 2020-02-06T01:14:09.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:09.085+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.086151+0000) 2020-02-06T01:14:09.100 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:09.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:09.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:09.711+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.712625+0000) 2020-02-06T01:14:09.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:09.959+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:09.960461+0000) 2020-02-06T01:14:10.565 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:10.563+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:10.564742+0000) 2020-02-06T01:14:10.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:10.821+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:10.822590+0000) 2020-02-06T01:14:10.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:10.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:10.844389+0000) 2020-02-06T01:14:10.845 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:10.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:10.844387+0000) 2020-02-06T01:14:12.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:12.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:12.948216+0000) 2020-02-06T01:14:13.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:13.176+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:13.177130+0000) 2020-02-06T01:14:14.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:14.104+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.105445+0000) 2020-02-06T01:14:14.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:14.104+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.106173+0000) 2020-02-06T01:14:14.107 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:14.105+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.106497+0000) 2020-02-06T01:14:14.107 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:14.106+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.107244+0000) 2020-02-06T01:14:14.108 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:14.106+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.107615+0000) 2020-02-06T01:14:14.108 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:14.106+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.107964+0000) 2020-02-06T01:14:14.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:14.107+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.108358+0000) 2020-02-06T01:14:14.109 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:14.107+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.108736+0000) 2020-02-06T01:14:14.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:14.108+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.109228+0000) 2020-02-06T01:14:14.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:14.108+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.109793+0000) 2020-02-06T01:14:14.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:14.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.110308+0000) 2020-02-06T01:14:14.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:14.109+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.111216+0000) 2020-02-06T01:14:14.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:14.110+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.111623+0000) 2020-02-06T01:14:14.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:14.114+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:14.115696+0000) 2020-02-06T01:14:14.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:14.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:15.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:15.472+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:15.473873+0000) 2020-02-06T01:14:16.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:16.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:16.085563+0000) 2020-02-06T01:14:17.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:17.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:17.068784+0000) 2020-02-06T01:14:17.338 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:17.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:17.337513+0000) 2020-02-06T01:14:17.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:17.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:17.337748+0000) 2020-02-06T01:14:17.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:17.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:17.337973+0000) 2020-02-06T01:14:19.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:19.130+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.131915+0000) 2020-02-06T01:14:19.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:19.131+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.132312+0000) 2020-02-06T01:14:19.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:19.133+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.134179+0000) 2020-02-06T01:14:19.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:19.133+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.134705+0000) 2020-02-06T01:14:19.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:19.133+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.134670+0000) 2020-02-06T01:14:19.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:19.133+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.135338+0000) 2020-02-06T01:14:19.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:19.134+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.135655+0000) 2020-02-06T01:14:19.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:19.134+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.135968+0000) 2020-02-06T01:14:19.137 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:19.135+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.136682+0000) 2020-02-06T01:14:19.138 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:19.135+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.137150+0000) 2020-02-06T01:14:19.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:19.137+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.138201+0000) 2020-02-06T01:14:19.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:19.136+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.137796+0000) 2020-02-06T01:14:19.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:19.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.142202+0000) 2020-02-06T01:14:19.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:19.141+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.142458+0000) 2020-02-06T01:14:19.153 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:19.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:19.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:19.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.712833+0000) 2020-02-06T01:14:19.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:19.959+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:19.960666+0000) 2020-02-06T01:14:20.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:20.564+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:20.565000+0000) 2020-02-06T01:14:20.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:20.822+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:20.822824+0000) 2020-02-06T01:14:20.845 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:20.843+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:20.844648+0000) 2020-02-06T01:14:20.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:20.843+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:20.844645+0000) 2020-02-06T01:14:22.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:22.947+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:22.948424+0000) 2020-02-06T01:14:23.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:23.176+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:23.177386+0000) 2020-02-06T01:14:24.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:24.158+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.159449+0000) 2020-02-06T01:14:24.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:24.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.160211+0000) 2020-02-06T01:14:24.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:24.159+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.160744+0000) 2020-02-06T01:14:24.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:24.159+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.160949+0000) 2020-02-06T01:14:24.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:24.160+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.161298+0000) 2020-02-06T01:14:24.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:24.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.161472+0000) 2020-02-06T01:14:24.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:24.160+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.161556+0000) 2020-02-06T01:14:24.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:24.160+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.162182+0000) 2020-02-06T01:14:24.163 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:24.161+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.162498+0000) 2020-02-06T01:14:24.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:24.161+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.162193+0000) 2020-02-06T01:14:24.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:24.160+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.161833+0000) 2020-02-06T01:14:24.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:24.161+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.162845+0000) 2020-02-06T01:14:24.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:24.161+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.163280+0000) 2020-02-06T01:14:24.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:24.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:24.167618+0000) 2020-02-06T01:14:24.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:24.179+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197491 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:25.356 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:14:25.362 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:14:25.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:25.480+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:25.481568+0000) 2020-02-06T01:14:26.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:26.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:26.085804+0000) 2020-02-06T01:14:27.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:27.067+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:27.069052+0000) 2020-02-06T01:14:27.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:27.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:27.337740+0000) 2020-02-06T01:14:27.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:27.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:27.337960+0000) 2020-02-06T01:14:27.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:27.336+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:27.338211+0000) 2020-02-06T01:14:29.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:29.185+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.186223+0000) 2020-02-06T01:14:29.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:29.186+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.187843+0000) 2020-02-06T01:14:29.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:29.187+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.188638+0000) 2020-02-06T01:14:29.190 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:29.188+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.189410+0000) 2020-02-06T01:14:29.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:29.188+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.189444+0000) 2020-02-06T01:14:29.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:29.188+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.189917+0000) 2020-02-06T01:14:29.191 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:29.189+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.190213+0000) 2020-02-06T01:14:29.191 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:29.189+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.190704+0000) 2020-02-06T01:14:29.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:29.190+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.191315+0000) 2020-02-06T01:14:29.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:29.190+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.192032+0000) 2020-02-06T01:14:29.193 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:29.190+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.192032+0000) 2020-02-06T01:14:29.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:29.190+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.192192+0000) 2020-02-06T01:14:29.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:29.192+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.193562+0000) 2020-02-06T01:14:29.194 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:29.192+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.193751+0000) 2020-02-06T01:14:29.207 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:29.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197534 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:29.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:29.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.713059+0000) 2020-02-06T01:14:29.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:29.960+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:29.960893+0000) 2020-02-06T01:14:30.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:30.564+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:30.565236+0000) 2020-02-06T01:14:30.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:30.822+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:30.823109+0000) 2020-02-06T01:14:30.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:30.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:30.844881+0000) 2020-02-06T01:14:30.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:30.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:30.844886+0000) 2020-02-06T01:14:32.949 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:32.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:32.948722+0000) 2020-02-06T01:14:33.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:33.177+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:33.177616+0000) 2020-02-06T01:14:34.218 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:34.211+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.213279+0000) 2020-02-06T01:14:34.218 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:34.213+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.215209+0000) 2020-02-06T01:14:34.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:34.212+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.213488+0000) 2020-02-06T01:14:34.218 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:34.212+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.213968+0000) 2020-02-06T01:14:34.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:34.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.214442+0000) 2020-02-06T01:14:34.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:34.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.216770+0000) 2020-02-06T01:14:34.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:34.213+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.214667+0000) 2020-02-06T01:14:34.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:34.214+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.215501+0000) 2020-02-06T01:14:34.219 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:34.215+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.216234+0000) 2020-02-06T01:14:34.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:34.214+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.215933+0000) 2020-02-06T01:14:34.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:34.216+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.217785+0000) 2020-02-06T01:14:34.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:34.216+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.218057+0000) 2020-02-06T01:14:34.221 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:34.217+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.218980+0000) 2020-02-06T01:14:34.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:34.222+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:34.223585+0000) 2020-02-06T01:14:34.235 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:34.232+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:35.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:35.490+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:35.490552+0000) 2020-02-06T01:14:36.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:36.084+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:36.086065+0000) 2020-02-06T01:14:37.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:37.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:37.069310+0000) 2020-02-06T01:14:37.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:37.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:37.337967+0000) 2020-02-06T01:14:37.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:37.336+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:37.338190+0000) 2020-02-06T01:14:37.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:37.337+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:37.338543+0000) 2020-02-06T01:14:39.240 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:39.237+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.239017+0000) 2020-02-06T01:14:39.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:39.238+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.239759+0000) 2020-02-06T01:14:39.241 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:39.239+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.240729+0000) 2020-02-06T01:14:39.242 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:39.241+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.242186+0000) 2020-02-06T01:14:39.243 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:39.241+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.242641+0000) 2020-02-06T01:14:39.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:39.241+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.243003+0000) 2020-02-06T01:14:39.244 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:39.242+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.243268+0000) 2020-02-06T01:14:39.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:39.242+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.243597+0000) 2020-02-06T01:14:39.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:39.242+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.243760+0000) 2020-02-06T01:14:39.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:39.243+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.244138+0000) 2020-02-06T01:14:39.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:39.243+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.245108+0000) 2020-02-06T01:14:39.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:39.244+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.246025+0000) 2020-02-06T01:14:39.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:39.245+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.246422+0000) 2020-02-06T01:14:39.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:39.248+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.249242+0000) 2020-02-06T01:14:39.262 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:39.259+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197619 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:39.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:39.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.713319+0000) 2020-02-06T01:14:39.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:39.960+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:39.961130+0000) 2020-02-06T01:14:40.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:40.564+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:40.565492+0000) 2020-02-06T01:14:40.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:40.823+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:40.823338+0000) 2020-02-06T01:14:40.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:40.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:40.845139+0000) 2020-02-06T01:14:40.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:40.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:40.845144+0000) 2020-02-06T01:14:42.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:42.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:42.948966+0000) 2020-02-06T01:14:43.178 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:43.176+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:43.177906+0000) 2020-02-06T01:14:44.267 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:44.265+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.266830+0000) 2020-02-06T01:14:44.268 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:44.266+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.267565+0000) 2020-02-06T01:14:44.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:44.267+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.268278+0000) 2020-02-06T01:14:44.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:44.267+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.268963+0000) 2020-02-06T01:14:44.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:44.267+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.268609+0000) 2020-02-06T01:14:44.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:44.268+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.269713+0000) 2020-02-06T01:14:44.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:44.269+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.270329+0000) 2020-02-06T01:14:44.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:44.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.270820+0000) 2020-02-06T01:14:44.272 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:44.270+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.271130+0000) 2020-02-06T01:14:44.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:44.271+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.272410+0000) 2020-02-06T01:14:44.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:44.271+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.272660+0000) 2020-02-06T01:14:44.274 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:44.271+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.272660+0000) 2020-02-06T01:14:44.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:44.272+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.273507+0000) 2020-02-06T01:14:44.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:44.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:44.276155+0000) 2020-02-06T01:14:44.288 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:44.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:45.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:45.498+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:45.498652+0000) 2020-02-06T01:14:46.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:46.085+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:46.086357+0000) 2020-02-06T01:14:47.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:47.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:47.069586+0000) 2020-02-06T01:14:47.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:47.336+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:47.338187+0000) 2020-02-06T01:14:47.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:47.337+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:47.338410+0000) 2020-02-06T01:14:47.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:47.337+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:47.338812+0000) 2020-02-06T01:14:49.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:49.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.292638+0000) 2020-02-06T01:14:49.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:49.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.294839+0000) 2020-02-06T01:14:49.296 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:49.294+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.294959+0000) 2020-02-06T01:14:49.296 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:49.295+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.296011+0000) 2020-02-06T01:14:49.297 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:49.295+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.296169+0000) 2020-02-06T01:14:49.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:49.296+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.296566+0000) 2020-02-06T01:14:49.298 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:49.296+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.296964+0000) 2020-02-06T01:14:49.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:49.296+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.296767+0000) 2020-02-06T01:14:49.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:49.297+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.297493+0000) 2020-02-06T01:14:49.299 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:49.297+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.298150+0000) 2020-02-06T01:14:49.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:49.297+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.298272+0000) 2020-02-06T01:14:49.300 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:49.298+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.298470+0000) 2020-02-06T01:14:49.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:49.297+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.298035+0000) 2020-02-06T01:14:49.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:49.302+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.303330+0000) 2020-02-06T01:14:49.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:49.313+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:49.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:49.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.713538+0000) 2020-02-06T01:14:49.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:49.961+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:49.961358+0000) 2020-02-06T01:14:50.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:50.565+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:50.565742+0000) 2020-02-06T01:14:50.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:50.823+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:50.823573+0000) 2020-02-06T01:14:50.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:50.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:50.845323+0000) 2020-02-06T01:14:50.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:50.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:50.845413+0000) 2020-02-06T01:14:52.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:52.948+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:52.949187+0000) 2020-02-06T01:14:53.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:53.178+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:53.178132+0000) 2020-02-06T01:14:54.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:54.319+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.320417+0000) 2020-02-06T01:14:54.321 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:54.320+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.320878+0000) 2020-02-06T01:14:54.322 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:54.320+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.321341+0000) 2020-02-06T01:14:54.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:54.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.321795+0000) 2020-02-06T01:14:54.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:54.321+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.322160+0000) 2020-02-06T01:14:54.324 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:54.322+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.323275+0000) 2020-02-06T01:14:54.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:54.323+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.324522+0000) 2020-02-06T01:14:54.325 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:54.323+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.323745+0000) 2020-02-06T01:14:54.326 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:54.323+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.324064+0000) 2020-02-06T01:14:54.326 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:54.323+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.325003+0000) 2020-02-06T01:14:54.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:54.324+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.325277+0000) 2020-02-06T01:14:54.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:54.324+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.325366+0000) 2020-02-06T01:14:54.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:54.325+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.326173+0000) 2020-02-06T01:14:54.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:54.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:54.334925+0000) 2020-02-06T01:14:54.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:54.340+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:55.495 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:14:55.502 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:14:55.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:55.505+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:55.506486+0000) 2020-02-06T01:14:56.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:56.086+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:56.086640+0000) 2020-02-06T01:14:57.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:57.068+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:57.069821+0000) 2020-02-06T01:14:57.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:57.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:57.338422+0000) 2020-02-06T01:14:57.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:57.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:57.338774+0000) 2020-02-06T01:14:57.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:57.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:57.339071+0000) 2020-02-06T01:14:59.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:14:59.345+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.345628+0000) 2020-02-06T01:14:59.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:14:59.346+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.347176+0000) 2020-02-06T01:14:59.348 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:14:59.346+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.347677+0000) 2020-02-06T01:14:59.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:59.347+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.348181+0000) 2020-02-06T01:14:59.349 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:14:59.347+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.348684+0000) 2020-02-06T01:14:59.350 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:14:59.348+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.349344+0000) 2020-02-06T01:14:59.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:14:59.349+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.349957+0000) 2020-02-06T01:14:59.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:14:59.349+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.350616+0000) 2020-02-06T01:14:59.352 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:14:59.350+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.350908+0000) 2020-02-06T01:14:59.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:59.350+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.351536+0000) 2020-02-06T01:14:59.352 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:14:59.351+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.351764+0000) 2020-02-06T01:14:59.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:14:59.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.352371+0000) 2020-02-06T01:14:59.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:14:59.352+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.352656+0000) 2020-02-06T01:14:59.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:59.356+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.357375+0000) 2020-02-06T01:14:59.369 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:14:59.367+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:14:59.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:59.712+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.713728+0000) 2020-02-06T01:14:59.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:14:59.960+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:14:59.961569+0000) 2020-02-06T01:15:00.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:00.565+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:00.566001+0000) 2020-02-06T01:15:00.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:00.822+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:00.823808+0000) 2020-02-06T01:15:00.846 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:00.844+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:00.845550+0000) 2020-02-06T01:15:00.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:00.844+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:00.845705+0000) 2020-02-06T01:15:02.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:02.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:02.949471+0000) 2020-02-06T01:15:03.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:03.177+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:03.178358+0000) 2020-02-06T01:15:04.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:04.372+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.373946+0000) 2020-02-06T01:15:04.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:04.373+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.374321+0000) 2020-02-06T01:15:04.376 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:04.373+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.375003+0000) 2020-02-06T01:15:04.376 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:04.374+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.375935+0000) 2020-02-06T01:15:04.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:04.374+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.376058+0000) 2020-02-06T01:15:04.377 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:04.375+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.376816+0000) 2020-02-06T01:15:04.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:04.377+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.377476+0000) 2020-02-06T01:15:04.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:04.377+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.378197+0000) 2020-02-06T01:15:04.379 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:04.378+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.378509+0000) 2020-02-06T01:15:04.379 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:04.377+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.378673+0000) 2020-02-06T01:15:04.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:04.378+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.378953+0000) 2020-02-06T01:15:04.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:04.378+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.379349+0000) 2020-02-06T01:15:04.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:04.379+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.379491+0000) 2020-02-06T01:15:04.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:04.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:04.384983+0000) 2020-02-06T01:15:04.395 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:04.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:05.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:05.514+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:05.515701+0000) 2020-02-06T01:15:06.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:06.086+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:06.086912+0000) 2020-02-06T01:15:07.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:07.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:07.070060+0000) 2020-02-06T01:15:07.339 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:07.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:07.338658+0000) 2020-02-06T01:15:07.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:07.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:07.338994+0000) 2020-02-06T01:15:07.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:07.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:07.339287+0000) 2020-02-06T01:15:09.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:09.401+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.401789+0000) 2020-02-06T01:15:09.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:09.401+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.401948+0000) 2020-02-06T01:15:09.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:09.401+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.402232+0000) 2020-02-06T01:15:09.404 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:09.402+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.402784+0000) 2020-02-06T01:15:09.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:09.402+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.403388+0000) 2020-02-06T01:15:09.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:09.402+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.403618+0000) 2020-02-06T01:15:09.405 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:09.402+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.403899+0000) 2020-02-06T01:15:09.405 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:09.403+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.404307+0000) 2020-02-06T01:15:09.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:09.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.404782+0000) 2020-02-06T01:15:09.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:09.404+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.405262+0000) 2020-02-06T01:15:09.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:09.405+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.406082+0000) 2020-02-06T01:15:09.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:09.406+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.406465+0000) 2020-02-06T01:15:09.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:09.406+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.407186+0000) 2020-02-06T01:15:09.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:09.409+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.410979+0000) 2020-02-06T01:15:09.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:09.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197874 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:09.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:09.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.713923+0000) 2020-02-06T01:15:09.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:09.961+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:09.961798+0000) 2020-02-06T01:15:10.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:10.565+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:10.566238+0000) 2020-02-06T01:15:10.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:10.823+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:10.824072+0000) 2020-02-06T01:15:10.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:10.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:10.845827+0000) 2020-02-06T01:15:10.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:10.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:10.845972+0000) 2020-02-06T01:15:12.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:12.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:12.949711+0000) 2020-02-06T01:15:13.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:13.177+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:13.178632+0000) 2020-02-06T01:15:14.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:14.426+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.427685+0000) 2020-02-06T01:15:14.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:14.428+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.428703+0000) 2020-02-06T01:15:14.431 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:14.429+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.430276+0000) 2020-02-06T01:15:14.431 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:14.429+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.430753+0000) 2020-02-06T01:15:14.432 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:14.430+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.431383+0000) 2020-02-06T01:15:14.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:14.430+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.431405+0000) 2020-02-06T01:15:14.433 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:14.431+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.431673+0000) 2020-02-06T01:15:14.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:14.431+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.431857+0000) 2020-02-06T01:15:14.433 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:14.431+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.432379+0000) 2020-02-06T01:15:14.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:14.430+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.432040+0000) 2020-02-06T01:15:14.434 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:14.432+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.433274+0000) 2020-02-06T01:15:14.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:14.432+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.433244+0000) 2020-02-06T01:15:14.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:14.431+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.432810+0000) 2020-02-06T01:15:14.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:14.436+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:14.437189+0000) 2020-02-06T01:15:14.449 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:14.447+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:15.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:15.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:15.523987+0000) 2020-02-06T01:15:16.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:16.086+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:16.087171+0000) 2020-02-06T01:15:17.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:17.069+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:17.070364+0000) 2020-02-06T01:15:17.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:17.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:17.338876+0000) 2020-02-06T01:15:17.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:17.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:17.339212+0000) 2020-02-06T01:15:17.340 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:17.339+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:17.339507+0000) 2020-02-06T01:15:19.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:19.451+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.453019+0000) 2020-02-06T01:15:19.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:19.453+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.454123+0000) 2020-02-06T01:15:19.456 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:19.454+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.455147+0000) 2020-02-06T01:15:19.456 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:19.453+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.455004+0000) 2020-02-06T01:15:19.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:19.455+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.455691+0000) 2020-02-06T01:15:19.457 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:19.455+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.455933+0000) 2020-02-06T01:15:19.457 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:19.455+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.456399+0000) 2020-02-06T01:15:19.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:19.456+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.456730+0000) 2020-02-06T01:15:19.458 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:19.456+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.457431+0000) 2020-02-06T01:15:19.459 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:19.457+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.457826+0000) 2020-02-06T01:15:19.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:19.457+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.457866+0000) 2020-02-06T01:15:19.459 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:19.458+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.459024+0000) 2020-02-06T01:15:19.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:19.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.462323+0000) 2020-02-06T01:15:19.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:19.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.462641+0000) 2020-02-06T01:15:19.475 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:19.473+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:19.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:19.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.714160+0000) 2020-02-06T01:15:19.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:19.960+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:19.961953+0000) 2020-02-06T01:15:20.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:20.566+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:20.566489+0000) 2020-02-06T01:15:20.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:20.823+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:20.824378+0000) 2020-02-06T01:15:20.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:20.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:20.846127+0000) 2020-02-06T01:15:20.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:20.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:20.846206+0000) 2020-02-06T01:15:22.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:22.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:22.949936+0000) 2020-02-06T01:15:23.179 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:23.177+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:23.178878+0000) 2020-02-06T01:15:24.480 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:24.478+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.479510+0000) 2020-02-06T01:15:24.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:24.478+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.479995+0000) 2020-02-06T01:15:24.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:24.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.480482+0000) 2020-02-06T01:15:24.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:24.480+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.481066+0000) 2020-02-06T01:15:24.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:24.481+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.482027+0000) 2020-02-06T01:15:24.484 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:24.481+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.482882+0000) 2020-02-06T01:15:24.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:24.482+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.483325+0000) 2020-02-06T01:15:24.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:24.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.483657+0000) 2020-02-06T01:15:24.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:24.483+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.484134+0000) 2020-02-06T01:15:24.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:24.484+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.484611+0000) 2020-02-06T01:15:24.486 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:24.484+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.485030+0000) 2020-02-06T01:15:24.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:24.485+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.485834+0000) 2020-02-06T01:15:24.487 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:24.486+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.486555+0000) 2020-02-06T01:15:24.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:24.486+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:24.488009+0000) 2020-02-06T01:15:24.502 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:24.501+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 197999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:25.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:25.531+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:25.533040+0000) 2020-02-06T01:15:25.676 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:15:25.683 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:15:26.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:26.087+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:26.087473+0000) 2020-02-06T01:15:27.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:27.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:27.070623+0000) 2020-02-06T01:15:27.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:27.337+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:27.339084+0000) 2020-02-06T01:15:27.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:27.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:27.339412+0000) 2020-02-06T01:15:27.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:27.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:27.339711+0000) 2020-02-06T01:15:29.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:29.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.507314+0000) 2020-02-06T01:15:29.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:29.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.508466+0000) 2020-02-06T01:15:29.510 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:29.508+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.509456+0000) 2020-02-06T01:15:29.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:29.507+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.508889+0000) 2020-02-06T01:15:29.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:29.508+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.509923+0000) 2020-02-06T01:15:29.511 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:29.509+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.510265+0000) 2020-02-06T01:15:29.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:29.509+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.510986+0000) 2020-02-06T01:15:29.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:29.510+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.511490+0000) 2020-02-06T01:15:29.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:29.510+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.511936+0000) 2020-02-06T01:15:29.514 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:29.511+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.513305+0000) 2020-02-06T01:15:29.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:29.512+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.513434+0000) 2020-02-06T01:15:29.515 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:29.512+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.514045+0000) 2020-02-06T01:15:29.515 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:29.512+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.514074+0000) 2020-02-06T01:15:29.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:29.518+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.519226+0000) 2020-02-06T01:15:29.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:29.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198043 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:29.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:29.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.714360+0000) 2020-02-06T01:15:29.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:29.961+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:29.962164+0000) 2020-02-06T01:15:30.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:30.565+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:30.566725+0000) 2020-02-06T01:15:30.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:30.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:30.824610+0000) 2020-02-06T01:15:30.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:30.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:30.846360+0000) 2020-02-06T01:15:30.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:30.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:30.846461+0000) 2020-02-06T01:15:32.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:32.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:32.950198+0000) 2020-02-06T01:15:33.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:33.178+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:33.179142+0000) 2020-02-06T01:15:34.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:34.533+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.534847+0000) 2020-02-06T01:15:34.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:34.534+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.535351+0000) 2020-02-06T01:15:34.536 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:34.534+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.535912+0000) 2020-02-06T01:15:34.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:34.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.536433+0000) 2020-02-06T01:15:34.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:34.535+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.536581+0000) 2020-02-06T01:15:34.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:34.535+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.537296+0000) 2020-02-06T01:15:34.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:34.536+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.537348+0000) 2020-02-06T01:15:34.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:34.536+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.537736+0000) 2020-02-06T01:15:34.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:34.536+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.537943+0000) 2020-02-06T01:15:34.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:34.537+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.538725+0000) 2020-02-06T01:15:34.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:34.537+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.539138+0000) 2020-02-06T01:15:34.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:34.538+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.539628+0000) 2020-02-06T01:15:34.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:34.538+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.539873+0000) 2020-02-06T01:15:34.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:34.544+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:34.545255+0000) 2020-02-06T01:15:34.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:34.555+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198084 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:35.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:35.539+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:35.540910+0000) 2020-02-06T01:15:36.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:36.086+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:36.087761+0000) 2020-02-06T01:15:37.071 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:37.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:37.070856+0000) 2020-02-06T01:15:37.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:37.337+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:37.339309+0000) 2020-02-06T01:15:37.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:37.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:37.339752+0000) 2020-02-06T01:15:37.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:37.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:37.339959+0000) 2020-02-06T01:15:39.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:39.562+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.563172+0000) 2020-02-06T01:15:39.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:39.562+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.563513+0000) 2020-02-06T01:15:39.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:39.562+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.563883+0000) 2020-02-06T01:15:39.565 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:39.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.564416+0000) 2020-02-06T01:15:39.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:39.564+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.565267+0000) 2020-02-06T01:15:39.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:39.564+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.565760+0000) 2020-02-06T01:15:39.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:39.564+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.565836+0000) 2020-02-06T01:15:39.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:39.564+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.566207+0000) 2020-02-06T01:15:39.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:39.565+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.566518+0000) 2020-02-06T01:15:39.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:39.565+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.566746+0000) 2020-02-06T01:15:39.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:39.565+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.566816+0000) 2020-02-06T01:15:39.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:39.565+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.567259+0000) 2020-02-06T01:15:39.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:39.566+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.567530+0000) 2020-02-06T01:15:39.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:39.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.572753+0000) 2020-02-06T01:15:39.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:39.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:39.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:39.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.714574+0000) 2020-02-06T01:15:39.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:39.962+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:39.962370+0000) 2020-02-06T01:15:40.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:40.566+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:40.566975+0000) 2020-02-06T01:15:40.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:40.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:40.824836+0000) 2020-02-06T01:15:40.847 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:40.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:40.846638+0000) 2020-02-06T01:15:40.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:40.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:40.846695+0000) 2020-02-06T01:15:42.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:42.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:42.950492+0000) 2020-02-06T01:15:43.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:43.179+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:43.179471+0000) 2020-02-06T01:15:44.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:44.590+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.592128+0000) 2020-02-06T01:15:44.595 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:44.591+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.593102+0000) 2020-02-06T01:15:44.595 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:44.589+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.589685+0000) 2020-02-06T01:15:44.595 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:44.590+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.590279+0000) 2020-02-06T01:15:44.596 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:44.590+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.590651+0000) 2020-02-06T01:15:44.596 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:44.591+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.591671+0000) 2020-02-06T01:15:44.596 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:44.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.592489+0000) 2020-02-06T01:15:44.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:44.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.593699+0000) 2020-02-06T01:15:44.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:44.592+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.593971+0000) 2020-02-06T01:15:44.597 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:44.593+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.594798+0000) 2020-02-06T01:15:44.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:44.593+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.594898+0000) 2020-02-06T01:15:44.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:44.594+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.595547+0000) 2020-02-06T01:15:44.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:44.597+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.597701+0000) 2020-02-06T01:15:44.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:44.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:44.598590+0000) 2020-02-06T01:15:44.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:44.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198167 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:45.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:45.549+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:45.549218+0000) 2020-02-06T01:15:46.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:46.087+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:46.087982+0000) 2020-02-06T01:15:47.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:47.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:47.071125+0000) 2020-02-06T01:15:47.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:47.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:47.339557+0000) 2020-02-06T01:15:47.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:47.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:47.340000+0000) 2020-02-06T01:15:47.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:47.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:47.340167+0000) 2020-02-06T01:15:49.615 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:49.613+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.614735+0000) 2020-02-06T01:15:49.616 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:49.614+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.615828+0000) 2020-02-06T01:15:49.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:49.615+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.616492+0000) 2020-02-06T01:15:49.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:49.615+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.617171+0000) 2020-02-06T01:15:49.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:49.616+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.617540+0000) 2020-02-06T01:15:49.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:49.615+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.616898+0000) 2020-02-06T01:15:49.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:49.616+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.617910+0000) 2020-02-06T01:15:49.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:49.617+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.618509+0000) 2020-02-06T01:15:49.619 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:49.617+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.618867+0000) 2020-02-06T01:15:49.621 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:49.618+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.619521+0000) 2020-02-06T01:15:49.621 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:49.618+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.619868+0000) 2020-02-06T01:15:49.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:49.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.620001+0000) 2020-02-06T01:15:49.622 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:49.619+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.620889+0000) 2020-02-06T01:15:49.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:49.624+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.625903+0000) 2020-02-06T01:15:49.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:49.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:49.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:49.713+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.714787+0000) 2020-02-06T01:15:49.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:49.961+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:49.962539+0000) 2020-02-06T01:15:50.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:50.566+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:50.567204+0000) 2020-02-06T01:15:50.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:50.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:50.825067+0000) 2020-02-06T01:15:50.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:50.845+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:50.846919+0000) 2020-02-06T01:15:50.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:50.845+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:50.846883+0000) 2020-02-06T01:15:52.951 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:52.949+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:52.950739+0000) 2020-02-06T01:15:53.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:53.178+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:53.179705+0000) 2020-02-06T01:15:54.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:54.643+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.644069+0000) 2020-02-06T01:15:54.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:54.643+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.644675+0000) 2020-02-06T01:15:54.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:54.643+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.645020+0000) 2020-02-06T01:15:54.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:54.644+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.645254+0000) 2020-02-06T01:15:54.646 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:54.644+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.645859+0000) 2020-02-06T01:15:54.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:54.645+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.646443+0000) 2020-02-06T01:15:54.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:54.645+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.647353+0000) 2020-02-06T01:15:54.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:54.645+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.646896+0000) 2020-02-06T01:15:54.649 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:54.646+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.648123+0000) 2020-02-06T01:15:54.651 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:54.648+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.649466+0000) 2020-02-06T01:15:54.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:54.647+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.649169+0000) 2020-02-06T01:15:54.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:54.648+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.649565+0000) 2020-02-06T01:15:54.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:54.649+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.650422+0000) 2020-02-06T01:15:54.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:54.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:54.652474+0000) 2020-02-06T01:15:54.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:54.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:55.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:55.556+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:55.557523+0000) 2020-02-06T01:15:55.891 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:15:55.898 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:15:56.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:56.087+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:56.088284+0000) 2020-02-06T01:15:57.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:57.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:57.071386+0000) 2020-02-06T01:15:57.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:57.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:57.339791+0000) 2020-02-06T01:15:57.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:57.338+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:57.340248+0000) 2020-02-06T01:15:57.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:57.338+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:57.340396+0000) 2020-02-06T01:15:59.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:15:59.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.672117+0000) 2020-02-06T01:15:59.673 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:15:59.671+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.672498+0000) 2020-02-06T01:15:59.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:15:59.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.673281+0000) 2020-02-06T01:15:59.674 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:15:59.671+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.673327+0000) 2020-02-06T01:15:59.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:15:59.672+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.673805+0000) 2020-02-06T01:15:59.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:15:59.672+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.673797+0000) 2020-02-06T01:15:59.675 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:15:59.673+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.674357+0000) 2020-02-06T01:15:59.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:15:59.673+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.674837+0000) 2020-02-06T01:15:59.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:15:59.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.675340+0000) 2020-02-06T01:15:59.676 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:15:59.674+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.675636+0000) 2020-02-06T01:15:59.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:15:59.674+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.676072+0000) 2020-02-06T01:15:59.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:59.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.676630+0000) 2020-02-06T01:15:59.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:59.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.676955+0000) 2020-02-06T01:15:59.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:59.683+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.684405+0000) 2020-02-06T01:15:59.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:15:59.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:15:59.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:59.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.714993+0000) 2020-02-06T01:15:59.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:15:59.961+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:15:59.962771+0000) 2020-02-06T01:16:00.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:00.566+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:00.567424+0000) 2020-02-06T01:16:00.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:00.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:00.825347+0000) 2020-02-06T01:16:00.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:00.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:00.847127+0000) 2020-02-06T01:16:00.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:00.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:00.847155+0000) 2020-02-06T01:16:02.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:02.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:02.951025+0000) 2020-02-06T01:16:03.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:03.179+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:03.179977+0000) 2020-02-06T01:16:04.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:04.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.700137+0000) 2020-02-06T01:16:04.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:04.699+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.700985+0000) 2020-02-06T01:16:04.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:04.700+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.701433+0000) 2020-02-06T01:16:04.702 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:04.700+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.701894+0000) 2020-02-06T01:16:04.703 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:04.700+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.702165+0000) 2020-02-06T01:16:04.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:04.701+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.702593+0000) 2020-02-06T01:16:04.703 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:04.702+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703118+0000) 2020-02-06T01:16:04.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:04.701+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703156+0000) 2020-02-06T01:16:04.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:04.702+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703591+0000) 2020-02-06T01:16:04.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:04.702+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703813+0000) 2020-02-06T01:16:04.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:04.702+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703217+0000) 2020-02-06T01:16:04.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:04.702+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.703944+0000) 2020-02-06T01:16:04.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:04.703+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.704307+0000) 2020-02-06T01:16:04.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:04.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:04.710087+0000) 2020-02-06T01:16:04.721 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:04.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198333 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:05.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:05.568+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:05.569161+0000) 2020-02-06T01:16:06.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:06.087+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:06.088572+0000) 2020-02-06T01:16:07.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:07.070+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:07.071594+0000) 2020-02-06T01:16:07.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:07.338+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:07.339979+0000) 2020-02-06T01:16:07.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:07.339+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:07.340478+0000) 2020-02-06T01:16:07.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:07.339+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:07.340629+0000) 2020-02-06T01:16:09.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:09.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.715184+0000) 2020-02-06T01:16:09.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:09.725+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.726290+0000) 2020-02-06T01:16:09.727 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:09.725+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.726777+0000) 2020-02-06T01:16:09.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:09.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.728319+0000) 2020-02-06T01:16:09.729 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:09.726+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.727462+0000) 2020-02-06T01:16:09.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:09.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.728831+0000) 2020-02-06T01:16:09.730 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:09.728+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.729115+0000) 2020-02-06T01:16:09.730 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:09.728+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.730025+0000) 2020-02-06T01:16:09.731 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:09.729+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.730449+0000) 2020-02-06T01:16:09.731 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:09.729+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.730906+0000) 2020-02-06T01:16:09.732 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:09.729+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.731187+0000) 2020-02-06T01:16:09.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:09.730+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.732186+0000) 2020-02-06T01:16:09.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:09.730+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.731987+0000) 2020-02-06T01:16:09.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:09.731+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.732602+0000) 2020-02-06T01:16:09.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:09.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.735574+0000) 2020-02-06T01:16:09.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:09.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:09.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:09.962+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:09.962989+0000) 2020-02-06T01:16:10.568 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:10.567+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:10.567689+0000) 2020-02-06T01:16:10.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:10.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:10.825585+0000) 2020-02-06T01:16:10.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:10.846+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:10.847386+0000) 2020-02-06T01:16:10.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:10.846+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:10.847406+0000) 2020-02-06T01:16:12.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:12.950+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:12.951309+0000) 2020-02-06T01:16:13.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:13.179+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:13.180200+0000) 2020-02-06T01:16:14.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:14.753+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.754823+0000) 2020-02-06T01:16:14.756 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:14.754+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.755266+0000) 2020-02-06T01:16:14.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:14.754+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.755709+0000) 2020-02-06T01:16:14.757 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:14.755+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.756110+0000) 2020-02-06T01:16:14.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:14.755+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.756412+0000) 2020-02-06T01:16:14.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:14.755+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.756839+0000) 2020-02-06T01:16:14.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:14.756+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.757262+0000) 2020-02-06T01:16:14.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:14.755+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.756924+0000) 2020-02-06T01:16:14.759 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:14.756+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.757837+0000) 2020-02-06T01:16:14.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:14.756+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.757998+0000) 2020-02-06T01:16:14.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:14.757+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.758492+0000) 2020-02-06T01:16:14.760 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:14.758+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.759637+0000) 2020-02-06T01:16:14.760 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:14.758+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.759574+0000) 2020-02-06T01:16:14.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:14.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:14.764325+0000) 2020-02-06T01:16:14.777 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:14.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:15.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:15.576+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:15.577437+0000) 2020-02-06T01:16:16.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:16.088+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:16.088835+0000) 2020-02-06T01:16:17.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:17.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:17.071811+0000) 2020-02-06T01:16:17.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:17.339+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:17.340270+0000) 2020-02-06T01:16:17.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:17.339+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:17.340678+0000) 2020-02-06T01:16:17.342 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:17.339+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:17.340876+0000) 2020-02-06T01:16:19.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:19.714+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.715423+0000) 2020-02-06T01:16:19.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:19.781+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.782329+0000) 2020-02-06T01:16:19.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:19.782+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.783592+0000) 2020-02-06T01:16:19.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:19.783+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.783411+0000) 2020-02-06T01:16:19.792 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:19.783+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.784269+0000) 2020-02-06T01:16:19.792 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:19.784+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.785136+0000) 2020-02-06T01:16:19.793 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:19.787+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.787680+0000) 2020-02-06T01:16:19.793 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:19.787+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.788035+0000) 2020-02-06T01:16:19.793 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:19.787+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.788140+0000) 2020-02-06T01:16:19.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:19.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.784590+0000) 2020-02-06T01:16:19.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:19.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.792539+0000) 2020-02-06T01:16:19.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:19.784+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.785421+0000) 2020-02-06T01:16:19.795 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:19.784+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.785742+0000) 2020-02-06T01:16:19.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:19.785+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.786206+0000) 2020-02-06T01:16:19.795 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:19.785+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.786490+0000) 2020-02-06T01:16:19.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:19.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:19.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:19.962+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:19.963231+0000) 2020-02-06T01:16:20.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:20.567+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:20.567919+0000) 2020-02-06T01:16:20.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:20.824+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:20.825822+0000) 2020-02-06T01:16:20.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:20.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:20.847653+0000) 2020-02-06T01:16:20.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:20.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:20.847658+0000) 2020-02-06T01:16:22.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:22.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:22.951578+0000) 2020-02-06T01:16:23.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:23.180+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:23.180439+0000) 2020-02-06T01:16:24.811 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:24.809+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.810463+0000) 2020-02-06T01:16:24.811 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:24.809+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.810963+0000) 2020-02-06T01:16:24.813 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:24.811+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.812194+0000) 2020-02-06T01:16:24.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:24.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.813379+0000) 2020-02-06T01:16:24.814 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:24.812+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.813700+0000) 2020-02-06T01:16:24.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:24.812+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.813382+0000) 2020-02-06T01:16:24.815 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:24.813+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.814084+0000) 2020-02-06T01:16:24.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:24.814+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.815190+0000) 2020-02-06T01:16:24.816 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:24.814+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.814937+0000) 2020-02-06T01:16:24.817 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:24.814+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.815372+0000) 2020-02-06T01:16:24.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:24.814+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.815663+0000) 2020-02-06T01:16:24.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:24.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.816247+0000) 2020-02-06T01:16:24.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:24.815+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.816072+0000) 2020-02-06T01:16:24.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:24.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:24.818467+0000) 2020-02-06T01:16:24.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:24.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198499 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:25.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:25.584+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:25.585485+0000) 2020-02-06T01:16:26.079 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:16:26.085 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:16:26.090 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:26.088+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:26.089071+0000) 2020-02-06T01:16:27.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:27.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:27.072046+0000) 2020-02-06T01:16:27.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:27.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:27.340534+0000) 2020-02-06T01:16:27.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:27.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:27.340879+0000) 2020-02-06T01:16:27.342 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:27.340+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:27.341112+0000) 2020-02-06T01:16:29.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:29.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.715658+0000) 2020-02-06T01:16:29.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:29.837+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.838332+0000) 2020-02-06T01:16:29.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:29.839+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.840464+0000) 2020-02-06T01:16:29.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:29.839+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.840728+0000) 2020-02-06T01:16:29.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:29.840+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.841456+0000) 2020-02-06T01:16:29.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:29.841+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.841513+0000) 2020-02-06T01:16:29.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:29.840+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.841815+0000) 2020-02-06T01:16:29.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:29.841+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.842283+0000) 2020-02-06T01:16:29.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:29.841+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.842729+0000) 2020-02-06T01:16:29.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:29.842+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.843238+0000) 2020-02-06T01:16:29.849 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:29.843+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.843505+0000) 2020-02-06T01:16:29.849 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:29.843+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.844113+0000) 2020-02-06T01:16:29.850 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:29.844+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.844576+0000) 2020-02-06T01:16:29.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:29.844+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.844707+0000) 2020-02-06T01:16:29.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:29.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.849134+0000) 2020-02-06T01:16:29.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:29.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198543 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:29.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:29.963+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:29.963481+0000) 2020-02-06T01:16:30.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:30.567+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:30.568133+0000) 2020-02-06T01:16:30.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:30.825+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:30.826057+0000) 2020-02-06T01:16:30.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:30.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:30.847928+0000) 2020-02-06T01:16:30.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:30.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:30.847921+0000) 2020-02-06T01:16:32.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:32.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:32.951869+0000) 2020-02-06T01:16:33.181 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:33.180+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:33.180670+0000) 2020-02-06T01:16:34.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:34.864+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.865313+0000) 2020-02-06T01:16:34.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:34.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.866864+0000) 2020-02-06T01:16:34.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:34.866+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.867123+0000) 2020-02-06T01:16:34.868 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:34.866+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.867369+0000) 2020-02-06T01:16:34.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:34.866+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.867970+0000) 2020-02-06T01:16:34.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:34.868+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.868448+0000) 2020-02-06T01:16:34.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:34.867+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.868912+0000) 2020-02-06T01:16:34.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:34.869+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.869752+0000) 2020-02-06T01:16:34.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:34.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.870659+0000) 2020-02-06T01:16:34.871 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:34.870+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.870693+0000) 2020-02-06T01:16:34.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:34.870+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.871136+0000) 2020-02-06T01:16:34.872 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:34.871+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.871620+0000) 2020-02-06T01:16:34.872 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:34.871+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.872066+0000) 2020-02-06T01:16:34.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:34.873+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:34.874115+0000) 2020-02-06T01:16:34.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:34.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198583 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:35.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:35.593+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:35.593352+0000) 2020-02-06T01:16:36.090 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:36.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:36.089336+0000) 2020-02-06T01:16:37.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:37.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:37.072342+0000) 2020-02-06T01:16:37.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:37.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:37.340800+0000) 2020-02-06T01:16:37.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:37.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:37.341120+0000) 2020-02-06T01:16:37.342 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:37.340+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:37.341305+0000) 2020-02-06T01:16:39.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:39.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.715893+0000) 2020-02-06T01:16:39.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:39.893+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.893442+0000) 2020-02-06T01:16:39.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:39.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.894886+0000) 2020-02-06T01:16:39.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:39.895+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.895184+0000) 2020-02-06T01:16:39.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:39.895+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.895553+0000) 2020-02-06T01:16:39.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:39.895+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.896213+0000) 2020-02-06T01:16:39.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:39.896+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.896485+0000) 2020-02-06T01:16:39.898 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:39.896+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.897229+0000) 2020-02-06T01:16:39.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:39.897+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.897573+0000) 2020-02-06T01:16:39.899 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:39.897+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.898194+0000) 2020-02-06T01:16:39.900 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:39.898+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.898884+0000) 2020-02-06T01:16:39.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:39.898+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.899131+0000) 2020-02-06T01:16:39.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:39.899+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.899742+0000) 2020-02-06T01:16:39.901 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:39.899+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.899884+0000) 2020-02-06T01:16:39.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:39.901+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.902014+0000) 2020-02-06T01:16:39.915 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:39.914+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198626 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:39.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:39.963+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:39.963665+0000) 2020-02-06T01:16:40.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:40.568+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:40.568345+0000) 2020-02-06T01:16:40.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:40.826+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:40.826319+0000) 2020-02-06T01:16:40.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:40.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:40.848204+0000) 2020-02-06T01:16:40.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:40.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:40.848166+0000) 2020-02-06T01:16:42.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:42.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:42.952129+0000) 2020-02-06T01:16:43.182 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:43.180+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:43.180936+0000) 2020-02-06T01:16:44.921 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:44.919+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.920291+0000) 2020-02-06T01:16:44.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:44.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.920689+0000) 2020-02-06T01:16:44.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:44.920+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.921260+0000) 2020-02-06T01:16:44.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:44.920+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.921833+0000) 2020-02-06T01:16:44.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:44.922+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.923257+0000) 2020-02-06T01:16:44.924 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:44.923+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.923729+0000) 2020-02-06T01:16:44.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:44.923+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.924124+0000) 2020-02-06T01:16:44.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:44.923+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.924233+0000) 2020-02-06T01:16:44.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:44.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.925696+0000) 2020-02-06T01:16:44.927 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:44.924+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.924830+0000) 2020-02-06T01:16:44.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:44.925+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.925664+0000) 2020-02-06T01:16:44.927 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:44.925+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.925977+0000) 2020-02-06T01:16:44.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:44.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.928272+0000) 2020-02-06T01:16:44.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:44.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:44.928486+0000) 2020-02-06T01:16:44.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:44.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:45.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:45.600+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:45.601946+0000) 2020-02-06T01:16:46.090 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:46.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:46.089584+0000) 2020-02-06T01:16:47.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:47.071+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:47.072586+0000) 2020-02-06T01:16:47.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:47.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:47.341026+0000) 2020-02-06T01:16:47.342 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:47.341+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:47.341578+0000) 2020-02-06T01:16:47.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:47.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:47.341349+0000) 2020-02-06T01:16:49.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:49.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.716126+0000) 2020-02-06T01:16:49.948 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:49.946+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.947214+0000) 2020-02-06T01:16:49.949 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:49.947+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.948342+0000) 2020-02-06T01:16:49.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:49.948+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.948825+0000) 2020-02-06T01:16:49.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:49.948+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.949186+0000) 2020-02-06T01:16:49.951 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:49.949+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.949781+0000) 2020-02-06T01:16:49.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:49.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.949965+0000) 2020-02-06T01:16:49.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:49.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.951349+0000) 2020-02-06T01:16:49.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:49.949+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.950422+0000) 2020-02-06T01:16:49.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:49.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.950532+0000) 2020-02-06T01:16:49.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:49.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.950860+0000) 2020-02-06T01:16:49.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:49.949+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.950316+0000) 2020-02-06T01:16:49.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:49.950+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.951152+0000) 2020-02-06T01:16:49.955 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:49.951+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.951807+0000) 2020-02-06T01:16:49.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:49.956+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.957715+0000) 2020-02-06T01:16:49.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:49.962+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:49.963865+0000) 2020-02-06T01:16:49.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:49.967+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198709 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:50.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:50.568+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:50.568577+0000) 2020-02-06T01:16:50.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:50.825+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:50.826526+0000) 2020-02-06T01:16:50.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:50.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:50.848329+0000) 2020-02-06T01:16:50.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:50.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:50.848420+0000) 2020-02-06T01:16:52.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:52.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:52.952347+0000) 2020-02-06T01:16:53.182 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:53.180+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:53.181196+0000) 2020-02-06T01:16:54.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:54.972+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.973285+0000) 2020-02-06T01:16:54.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:54.973+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.974208+0000) 2020-02-06T01:16:54.975 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:54.973+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.975023+0000) 2020-02-06T01:16:54.976 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:54.975+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.975632+0000) 2020-02-06T01:16:54.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:16:54.975+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.976174+0000) 2020-02-06T01:16:54.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:54.976+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.976570+0000) 2020-02-06T01:16:54.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:54.975+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.976919+0000) 2020-02-06T01:16:54.978 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:16:54.976+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.977156+0000) 2020-02-06T01:16:54.979 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:16:54.976+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.977588+0000) 2020-02-06T01:16:54.980 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:54.977+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.978155+0000) 2020-02-06T01:16:54.980 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:16:54.978+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.978966+0000) 2020-02-06T01:16:54.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:54.977+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.978464+0000) 2020-02-06T01:16:54.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:16:54.979+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.979422+0000) 2020-02-06T01:16:54.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:54.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:54.984455+0000) 2020-02-06T01:16:54.995 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:16:54.994+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:16:55.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:55.609+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:55.610823+0000) 2020-02-06T01:16:56.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:16:56.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:56.089845+0000) 2020-02-06T01:16:56.258 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:16:56.264 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:16:57.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:16:57.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:57.072858+0000) 2020-02-06T01:16:57.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:16:57.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:57.341265+0000) 2020-02-06T01:16:57.342 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:16:57.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:57.341584+0000) 2020-02-06T01:16:57.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:16:57.340+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:57.341782+0000) 2020-02-06T01:16:59.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:59.715+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:59.716352+0000) 2020-02-06T01:16:59.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:16:59.963+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:16:59.964059+0000) 2020-02-06T01:17:00.001 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:16:59.999+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.000668+0000) 2020-02-06T01:17:00.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:00.000+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.001597+0000) 2020-02-06T01:17:00.002 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:00.000+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.002004+0000) 2020-02-06T01:17:00.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:00.001+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.002240+0000) 2020-02-06T01:17:00.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:00.001+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.002564+0000) 2020-02-06T01:17:00.003 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:00.001+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.002895+0000) 2020-02-06T01:17:00.004 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:00.001+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.003265+0000) 2020-02-06T01:17:00.011 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:00.003+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.004659+0000) 2020-02-06T01:17:00.011 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:00.004+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.005797+0000) 2020-02-06T01:17:00.011 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:00.004+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.006169+0000) 2020-02-06T01:17:00.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:00.003+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.005392+0000) 2020-02-06T01:17:00.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:00.005+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.006861+0000) 2020-02-06T01:17:00.012 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:00.003+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.004824+0000) 2020-02-06T01:17:00.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:00.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.009630+0000) 2020-02-06T01:17:00.022 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:00.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:00.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:00.567+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.568856+0000) 2020-02-06T01:17:00.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:00.825+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.826737+0000) 2020-02-06T01:17:00.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:00.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.848540+0000) 2020-02-06T01:17:00.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:00.847+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:00.848639+0000) 2020-02-06T01:17:02.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:02.951+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:02.952647+0000) 2020-02-06T01:17:03.182 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:03.180+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:03.181426+0000) 2020-02-06T01:17:05.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:05.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.027221+0000) 2020-02-06T01:17:05.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:05.026+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.028150+0000) 2020-02-06T01:17:05.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:05.028+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.029224+0000) 2020-02-06T01:17:05.030 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:05.027+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.029194+0000) 2020-02-06T01:17:05.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:05.028+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.029776+0000) 2020-02-06T01:17:05.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:05.029+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.030415+0000) 2020-02-06T01:17:05.031 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:05.029+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.030771+0000) 2020-02-06T01:17:05.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:05.029+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.031337+0000) 2020-02-06T01:17:05.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:05.029+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.031364+0000) 2020-02-06T01:17:05.032 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:05.030+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.031900+0000) 2020-02-06T01:17:05.033 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:05.031+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.032506+0000) 2020-02-06T01:17:05.033 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:05.031+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.033108+0000) 2020-02-06T01:17:05.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:05.032+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.033588+0000) 2020-02-06T01:17:05.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:05.035+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.036828+0000) 2020-02-06T01:17:05.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:05.050+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198831 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:05.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:05.617+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:05.618597+0000) 2020-02-06T01:17:06.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:06.088+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:06.090089+0000) 2020-02-06T01:17:07.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:07.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:07.073131+0000) 2020-02-06T01:17:07.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:07.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:07.341501+0000) 2020-02-06T01:17:07.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:07.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:07.341825+0000) 2020-02-06T01:17:07.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:07.340+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:07.342022+0000) 2020-02-06T01:17:09.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:09.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:09.716587+0000) 2020-02-06T01:17:09.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:09.963+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:09.964358+0000) 2020-02-06T01:17:10.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:10.056+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.057255+0000) 2020-02-06T01:17:10.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:10.057+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.058770+0000) 2020-02-06T01:17:10.060 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:10.058+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.059717+0000) 2020-02-06T01:17:10.061 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:10.059+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.060357+0000) 2020-02-06T01:17:10.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:10.059+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.060502+0000) 2020-02-06T01:17:10.062 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:10.059+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.061177+0000) 2020-02-06T01:17:10.062 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:10.060+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.061380+0000) 2020-02-06T01:17:10.062 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:10.060+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.061803+0000) 2020-02-06T01:17:10.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:10.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.062576+0000) 2020-02-06T01:17:10.064 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:10.060+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.062015+0000) 2020-02-06T01:17:10.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:10.060+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.062226+0000) 2020-02-06T01:17:10.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:10.061+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.062909+0000) 2020-02-06T01:17:10.065 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:10.062+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.063482+0000) 2020-02-06T01:17:10.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:10.064+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.065171+0000) 2020-02-06T01:17:10.080 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:10.078+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:10.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:10.568+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.569072+0000) 2020-02-06T01:17:10.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:10.825+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.826987+0000) 2020-02-06T01:17:10.849 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:10.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.848783+0000) 2020-02-06T01:17:10.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:10.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:10.848876+0000) 2020-02-06T01:17:12.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:12.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:12.952976+0000) 2020-02-06T01:17:13.182 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:13.181+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:13.181654+0000) 2020-02-06T01:17:15.085 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:15.083+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.084336+0000) 2020-02-06T01:17:15.086 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:15.084+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.085072+0000) 2020-02-06T01:17:15.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:15.084+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.085826+0000) 2020-02-06T01:17:15.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:15.085+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.086606+0000) 2020-02-06T01:17:15.087 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:15.085+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.086917+0000) 2020-02-06T01:17:15.088 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:15.085+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.087245+0000) 2020-02-06T01:17:15.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:15.086+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.087523+0000) 2020-02-06T01:17:15.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:15.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.087323+0000) 2020-02-06T01:17:15.089 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:15.087+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.088683+0000) 2020-02-06T01:17:15.090 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:15.087+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.088314+0000) 2020-02-06T01:17:15.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:15.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.088896+0000) 2020-02-06T01:17:15.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:15.087+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.088746+0000) 2020-02-06T01:17:15.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:15.087+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.089334+0000) 2020-02-06T01:17:15.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:15.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.093438+0000) 2020-02-06T01:17:15.107 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:15.105+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198915 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:15.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:15.626+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:15.626774+0000) 2020-02-06T01:17:16.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:16.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:16.090359+0000) 2020-02-06T01:17:17.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:17.072+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:17.073359+0000) 2020-02-06T01:17:17.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:17.340+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:17.341737+0000) 2020-02-06T01:17:17.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:17.340+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:17.342038+0000) 2020-02-06T01:17:17.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:17.340+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:17.342272+0000) 2020-02-06T01:17:19.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:19.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:19.716822+0000) 2020-02-06T01:17:19.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:19.964+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:19.964612+0000) 2020-02-06T01:17:20.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:20.110+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.111803+0000) 2020-02-06T01:17:20.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:20.112+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.113190+0000) 2020-02-06T01:17:20.114 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:20.112+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.113560+0000) 2020-02-06T01:17:20.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:20.112+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.113782+0000) 2020-02-06T01:17:20.115 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:20.113+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.114372+0000) 2020-02-06T01:17:20.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:20.113+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.115244+0000) 2020-02-06T01:17:20.116 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:20.114+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.115535+0000) 2020-02-06T01:17:20.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:20.114+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.115966+0000) 2020-02-06T01:17:20.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:20.115+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.116415+0000) 2020-02-06T01:17:20.118 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:20.115+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.117098+0000) 2020-02-06T01:17:20.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:20.115+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.117310+0000) 2020-02-06T01:17:20.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:20.116+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.118318+0000) 2020-02-06T01:17:20.119 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:20.116+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.118405+0000) 2020-02-06T01:17:20.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:20.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.123353+0000) 2020-02-06T01:17:20.134 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:20.132+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198958 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:20.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:20.568+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.569285+0000) 2020-02-06T01:17:20.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:20.826+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.827199+0000) 2020-02-06T01:17:20.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:20.847+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.849010+0000) 2020-02-06T01:17:20.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:20.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:20.849132+0000) 2020-02-06T01:17:22.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:22.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:22.953219+0000) 2020-02-06T01:17:23.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:23.181+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:23.181909+0000) 2020-02-06T01:17:25.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:25.138+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.139783+0000) 2020-02-06T01:17:25.141 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:25.139+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.140301+0000) 2020-02-06T01:17:25.142 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:25.139+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.140669+0000) 2020-02-06T01:17:25.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:25.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.141331+0000) 2020-02-06T01:17:25.143 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:25.140+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.141945+0000) 2020-02-06T01:17:25.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:25.140+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.141919+0000) 2020-02-06T01:17:25.144 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:25.141+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.142463+0000) 2020-02-06T01:17:25.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:25.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.143155+0000) 2020-02-06T01:17:25.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:25.142+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.143882+0000) 2020-02-06T01:17:25.145 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:25.142+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.143791+0000) 2020-02-06T01:17:25.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:25.143+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.145286+0000) 2020-02-06T01:17:25.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:25.143+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.145219+0000) 2020-02-06T01:17:25.146 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:25.144+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.145930+0000) 2020-02-06T01:17:25.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:25.148+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.149660+0000) 2020-02-06T01:17:25.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:25.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 198997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:25.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:25.637+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:25.638193+0000) 2020-02-06T01:17:26.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:26.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:26.090636+0000) 2020-02-06T01:17:26.438 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:17:26.445 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:17:27.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:27.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:27.073594+0000) 2020-02-06T01:17:27.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:27.341+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:27.341992+0000) 2020-02-06T01:17:27.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:27.341+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:27.342252+0000) 2020-02-06T01:17:27.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:27.341+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:27.342554+0000) 2020-02-06T01:17:29.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:29.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:29.717054+0000) 2020-02-06T01:17:29.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:29.964+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:29.964849+0000) 2020-02-06T01:17:30.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:30.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.166947+0000) 2020-02-06T01:17:30.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:30.166+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.167414+0000) 2020-02-06T01:17:30.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:30.166+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.167868+0000) 2020-02-06T01:17:30.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:30.167+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.168375+0000) 2020-02-06T01:17:30.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:30.167+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.168759+0000) 2020-02-06T01:17:30.171 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:30.167+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.169264+0000) 2020-02-06T01:17:30.171 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:30.168+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.169906+0000) 2020-02-06T01:17:30.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:30.169+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.170817+0000) 2020-02-06T01:17:30.172 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:30.170+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.171506+0000) 2020-02-06T01:17:30.173 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:30.170+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.171916+0000) 2020-02-06T01:17:30.173 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:30.170+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.172053+0000) 2020-02-06T01:17:30.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:30.171+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.172385+0000) 2020-02-06T01:17:30.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:30.176+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.177142+0000) 2020-02-06T01:17:30.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:30.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.178254+0000) 2020-02-06T01:17:30.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:30.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:30.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:30.568+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.569537+0000) 2020-02-06T01:17:30.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:30.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.827411+0000) 2020-02-06T01:17:30.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:30.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.849342+0000) 2020-02-06T01:17:30.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:30.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:30.849324+0000) 2020-02-06T01:17:32.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:32.952+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:32.953515+0000) 2020-02-06T01:17:33.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:33.182+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:33.182134+0000) 2020-02-06T01:17:35.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:35.190+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.192238+0000) 2020-02-06T01:17:35.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:35.193+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.193350+0000) 2020-02-06T01:17:35.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:35.193+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.193630+0000) 2020-02-06T01:17:35.196 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:35.195+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.195656+0000) 2020-02-06T01:17:35.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:35.195+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.195973+0000) 2020-02-06T01:17:35.197 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:35.194+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.196403+0000) 2020-02-06T01:17:35.199 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:35.196+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.198266+0000) 2020-02-06T01:17:35.199 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:35.197+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.198671+0000) 2020-02-06T01:17:35.200 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:35.198+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.198837+0000) 2020-02-06T01:17:35.200 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:35.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.199214+0000) 2020-02-06T01:17:35.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:35.197+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.199000+0000) 2020-02-06T01:17:35.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:35.197+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.199337+0000) 2020-02-06T01:17:35.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:35.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.204049+0000) 2020-02-06T01:17:35.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:35.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.204392+0000) 2020-02-06T01:17:35.214 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:35.212+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:35.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:35.645+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:35.645852+0000) 2020-02-06T01:17:36.091 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:36.089+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:36.090829+0000) 2020-02-06T01:17:37.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:37.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:37.073830+0000) 2020-02-06T01:17:37.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:37.341+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:37.342296+0000) 2020-02-06T01:17:37.343 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:37.341+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:37.342513+0000) 2020-02-06T01:17:37.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:37.341+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:37.342801+0000) 2020-02-06T01:17:39.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:39.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:39.717344+0000) 2020-02-06T01:17:39.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:39.964+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:39.965105+0000) 2020-02-06T01:17:40.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:40.217+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.218883+0000) 2020-02-06T01:17:40.220 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:40.218+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.219767+0000) 2020-02-06T01:17:40.221 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:40.219+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.220477+0000) 2020-02-06T01:17:40.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:40.219+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.220836+0000) 2020-02-06T01:17:40.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:40.219+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.221247+0000) 2020-02-06T01:17:40.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:40.220+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.221645+0000) 2020-02-06T01:17:40.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:40.220+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.221864+0000) 2020-02-06T01:17:40.223 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:40.221+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.222244+0000) 2020-02-06T01:17:40.223 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:40.221+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.222514+0000) 2020-02-06T01:17:40.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:40.221+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.222881+0000) 2020-02-06T01:17:40.224 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:40.222+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.223746+0000) 2020-02-06T01:17:40.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:40.222+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.223626+0000) 2020-02-06T01:17:40.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:40.227+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.228696+0000) 2020-02-06T01:17:40.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:40.227+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.228939+0000) 2020-02-06T01:17:40.241 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:40.239+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:40.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:40.569+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.569773+0000) 2020-02-06T01:17:40.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:40.826+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.827628+0000) 2020-02-06T01:17:40.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:40.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.849563+0000) 2020-02-06T01:17:40.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:40.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:40.849552+0000) 2020-02-06T01:17:42.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:42.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:42.953754+0000) 2020-02-06T01:17:43.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:43.181+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:43.182368+0000) 2020-02-06T01:17:45.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:45.245+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.246537+0000) 2020-02-06T01:17:45.249 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:45.247+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.248365+0000) 2020-02-06T01:17:45.249 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:45.247+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.248519+0000) 2020-02-06T01:17:45.250 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:45.247+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.248972+0000) 2020-02-06T01:17:45.250 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:45.248+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.249365+0000) 2020-02-06T01:17:45.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:45.248+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.249769+0000) 2020-02-06T01:17:45.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:45.249+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.250782+0000) 2020-02-06T01:17:45.252 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:45.249+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.251246+0000) 2020-02-06T01:17:45.253 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:45.250+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.252061+0000) 2020-02-06T01:17:45.253 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:45.251+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.252732+0000) 2020-02-06T01:17:45.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:45.251+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.252783+0000) 2020-02-06T01:17:45.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:45.251+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.252855+0000) 2020-02-06T01:17:45.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:45.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.255385+0000) 2020-02-06T01:17:45.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:45.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.255652+0000) 2020-02-06T01:17:45.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:45.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:45.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:45.652+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:45.653939+0000) 2020-02-06T01:17:46.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:46.090+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:46.091043+0000) 2020-02-06T01:17:47.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:47.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:47.074119+0000) 2020-02-06T01:17:47.343 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:47.342+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:47.342552+0000) 2020-02-06T01:17:47.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:47.342+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:47.342776+0000) 2020-02-06T01:17:47.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:47.342+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:47.343024+0000) 2020-02-06T01:17:49.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:49.716+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:49.717598+0000) 2020-02-06T01:17:49.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:49.964+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:49.965402+0000) 2020-02-06T01:17:50.275 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:50.272+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.274020+0000) 2020-02-06T01:17:50.275 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:50.273+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.275046+0000) 2020-02-06T01:17:50.276 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:50.274+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.275482+0000) 2020-02-06T01:17:50.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:50.275+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.275992+0000) 2020-02-06T01:17:50.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:50.275+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.276358+0000) 2020-02-06T01:17:50.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:50.276+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.277069+0000) 2020-02-06T01:17:50.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:50.277+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.277778+0000) 2020-02-06T01:17:50.279 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:50.277+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.278340+0000) 2020-02-06T01:17:50.279 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:50.277+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.278506+0000) 2020-02-06T01:17:50.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:50.278+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.278524+0000) 2020-02-06T01:17:50.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:50.278+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.278689+0000) 2020-02-06T01:17:50.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:50.278+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.279129+0000) 2020-02-06T01:17:50.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:50.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.279821+0000) 2020-02-06T01:17:50.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:50.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.284754+0000) 2020-02-06T01:17:50.296 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:50.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199204 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:50.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:50.569+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.569980+0000) 2020-02-06T01:17:50.828 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:50.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.827838+0000) 2020-02-06T01:17:50.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:50.848+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.849775+0000) 2020-02-06T01:17:50.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:50.848+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:50.849757+0000) 2020-02-06T01:17:52.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:52.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:52.954071+0000) 2020-02-06T01:17:53.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:53.181+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:53.182601+0000) 2020-02-06T01:17:55.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:17:55.299+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.300619+0000) 2020-02-06T01:17:55.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:17:55.302+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.302820+0000) 2020-02-06T01:17:55.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:55.302+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.303115+0000) 2020-02-06T01:17:55.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:55.302+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.303598+0000) 2020-02-06T01:17:55.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:17:55.303+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.304610+0000) 2020-02-06T01:17:55.305 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:17:55.303+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.304244+0000) 2020-02-06T01:17:55.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:17:55.304+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.305244+0000) 2020-02-06T01:17:55.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:55.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.306072+0000) 2020-02-06T01:17:55.307 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:17:55.305+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.306364+0000) 2020-02-06T01:17:55.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:55.306+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.307399+0000) 2020-02-06T01:17:55.308 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:55.307+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.307620+0000) 2020-02-06T01:17:55.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:55.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.307651+0000) 2020-02-06T01:17:55.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:55.307+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.308025+0000) 2020-02-06T01:17:55.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:55.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.311131+0000) 2020-02-06T01:17:55.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:17:55.321+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:17:55.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:55.659+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:55.660741+0000) 2020-02-06T01:17:56.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:17:56.090+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:56.091280+0000) 2020-02-06T01:17:56.627 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:17:56.633 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:17:57.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:17:57.073+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:57.074368+0000) 2020-02-06T01:17:57.344 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:17:57.342+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:57.342811+0000) 2020-02-06T01:17:57.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:17:57.342+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:57.342963+0000) 2020-02-06T01:17:57.344 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:17:57.342+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:57.343255+0000) 2020-02-06T01:17:59.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:59.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:59.717837+0000) 2020-02-06T01:17:59.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:17:59.964+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:17:59.965631+0000) 2020-02-06T01:18:00.329 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:00.327+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.328056+0000) 2020-02-06T01:18:00.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:00.328+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.328572+0000) 2020-02-06T01:18:00.330 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:00.328+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.329476+0000) 2020-02-06T01:18:00.331 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:00.329+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.330202+0000) 2020-02-06T01:18:00.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:00.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.331495+0000) 2020-02-06T01:18:00.332 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:00.331+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.331979+0000) 2020-02-06T01:18:00.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:00.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.332232+0000) 2020-02-06T01:18:00.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:00.332+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.333394+0000) 2020-02-06T01:18:00.335 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:00.333+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.333974+0000) 2020-02-06T01:18:00.335 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:00.333+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.333614+0000) 2020-02-06T01:18:00.335 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:00.333+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.334414+0000) 2020-02-06T01:18:00.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:00.334+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.334693+0000) 2020-02-06T01:18:00.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:00.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.334820+0000) 2020-02-06T01:18:00.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:00.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.338259+0000) 2020-02-06T01:18:00.350 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:00.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:00.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:00.569+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.570219+0000) 2020-02-06T01:18:00.829 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:00.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.828047+0000) 2020-02-06T01:18:00.850 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:00.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.849989+0000) 2020-02-06T01:18:00.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:00.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:00.849966+0000) 2020-02-06T01:18:02.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:02.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:02.954335+0000) 2020-02-06T01:18:03.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:03.182+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:03.182836+0000) 2020-02-06T01:18:05.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:05.353+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.354854+0000) 2020-02-06T01:18:05.356 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:05.354+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.355769+0000) 2020-02-06T01:18:05.357 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:05.355+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.356128+0000) 2020-02-06T01:18:05.357 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:05.355+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.356398+0000) 2020-02-06T01:18:05.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:05.355+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.357011+0000) 2020-02-06T01:18:05.358 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:05.356+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.357207+0000) 2020-02-06T01:18:05.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:05.357+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.357886+0000) 2020-02-06T01:18:05.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:05.357+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.357465+0000) 2020-02-06T01:18:05.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:05.357+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.358104+0000) 2020-02-06T01:18:05.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:05.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.358512+0000) 2020-02-06T01:18:05.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:05.358+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.359062+0000) 2020-02-06T01:18:05.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:05.359+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.359633+0000) 2020-02-06T01:18:05.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:05.359+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.360276+0000) 2020-02-06T01:18:05.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:05.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.364187+0000) 2020-02-06T01:18:05.377 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:05.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:05.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:05.667+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:05.668771+0000) 2020-02-06T01:18:06.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:06.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:06.091571+0000) 2020-02-06T01:18:07.075 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:07.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:07.074604+0000) 2020-02-06T01:18:07.344 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:07.342+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:07.343077+0000) 2020-02-06T01:18:07.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:07.342+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:07.343210+0000) 2020-02-06T01:18:07.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:07.343+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:07.343527+0000) 2020-02-06T01:18:09.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:09.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:09.718111+0000) 2020-02-06T01:18:09.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:09.965+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:09.965848+0000) 2020-02-06T01:18:10.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:10.382+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.383211+0000) 2020-02-06T01:18:10.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:10.382+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.383847+0000) 2020-02-06T01:18:10.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:10.383+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.384357+0000) 2020-02-06T01:18:10.386 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:10.383+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.384769+0000) 2020-02-06T01:18:10.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:10.384+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.385378+0000) 2020-02-06T01:18:10.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:10.385+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.386336+0000) 2020-02-06T01:18:10.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:10.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.386507+0000) 2020-02-06T01:18:10.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:10.386+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.387223+0000) 2020-02-06T01:18:10.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:10.386+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.387383+0000) 2020-02-06T01:18:10.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:10.387+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.388084+0000) 2020-02-06T01:18:10.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:10.387+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.388116+0000) 2020-02-06T01:18:10.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:10.388+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.388537+0000) 2020-02-06T01:18:10.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:10.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.389162+0000) 2020-02-06T01:18:10.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:10.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.391662+0000) 2020-02-06T01:18:10.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:10.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:10.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:10.570+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.570493+0000) 2020-02-06T01:18:10.829 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:10.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.828297+0000) 2020-02-06T01:18:10.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:10.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.850195+0000) 2020-02-06T01:18:10.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:10.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:10.850193+0000) 2020-02-06T01:18:12.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:12.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:12.954694+0000) 2020-02-06T01:18:13.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:13.182+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:13.183069+0000) 2020-02-06T01:18:15.411 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:15.409+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.410477+0000) 2020-02-06T01:18:15.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:15.409+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.410854+0000) 2020-02-06T01:18:15.412 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:15.411+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.411504+0000) 2020-02-06T01:18:15.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:15.410+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.411823+0000) 2020-02-06T01:18:15.413 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:15.412+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.412637+0000) 2020-02-06T01:18:15.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:15.414+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.414413+0000) 2020-02-06T01:18:15.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:15.414+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.415362+0000) 2020-02-06T01:18:15.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:15.415+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.416380+0000) 2020-02-06T01:18:15.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:15.415+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.416378+0000) 2020-02-06T01:18:15.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:15.415+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.416660+0000) 2020-02-06T01:18:15.418 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:15.416+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.416733+0000) 2020-02-06T01:18:15.418 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:15.416+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.416823+0000) 2020-02-06T01:18:15.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:15.416+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.417098+0000) 2020-02-06T01:18:15.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:15.418+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.419476+0000) 2020-02-06T01:18:15.432 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:15.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:15.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:15.680+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:15.680487+0000) 2020-02-06T01:18:16.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:16.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:16.091810+0000) 2020-02-06T01:18:17.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:17.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:17.074820+0000) 2020-02-06T01:18:17.344 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:17.343+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:17.343354+0000) 2020-02-06T01:18:17.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:17.343+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:17.343493+0000) 2020-02-06T01:18:17.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:17.343+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:17.343791+0000) 2020-02-06T01:18:19.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:19.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:19.718379+0000) 2020-02-06T01:18:19.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:19.965+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:19.966110+0000) 2020-02-06T01:18:20.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:20.435+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.436238+0000) 2020-02-06T01:18:20.438 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:20.436+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.436945+0000) 2020-02-06T01:18:20.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:20.436+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.437140+0000) 2020-02-06T01:18:20.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:20.437+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.438056+0000) 2020-02-06T01:18:20.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:20.437+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.438476+0000) 2020-02-06T01:18:20.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:20.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.439704+0000) 2020-02-06T01:18:20.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:20.440+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.440802+0000) 2020-02-06T01:18:20.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:20.440+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.441394+0000) 2020-02-06T01:18:20.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:20.441+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.442359+0000) 2020-02-06T01:18:20.443 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:20.442+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.442966+0000) 2020-02-06T01:18:20.444 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:20.442+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.443361+0000) 2020-02-06T01:18:20.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:20.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.443532+0000) 2020-02-06T01:18:20.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:20.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.446637+0000) 2020-02-06T01:18:20.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:20.445+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.446912+0000) 2020-02-06T01:18:20.459 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:20.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199449 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:20.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:20.570+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.570743+0000) 2020-02-06T01:18:20.829 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:20.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.828468+0000) 2020-02-06T01:18:20.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:20.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.850437+0000) 2020-02-06T01:18:20.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:20.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:20.850397+0000) 2020-02-06T01:18:22.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:22.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:22.954918+0000) 2020-02-06T01:18:23.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:23.183+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:23.183346+0000) 2020-02-06T01:18:25.465 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:25.463+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.463690+0000) 2020-02-06T01:18:25.465 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:25.462+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.464038+0000) 2020-02-06T01:18:25.466 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:25.464+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.464511+0000) 2020-02-06T01:18:25.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:25.464+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.465087+0000) 2020-02-06T01:18:25.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:25.465+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.466309+0000) 2020-02-06T01:18:25.467 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:25.465+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.466830+0000) 2020-02-06T01:18:25.468 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:25.465+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.467056+0000) 2020-02-06T01:18:25.468 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:25.466+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.467471+0000) 2020-02-06T01:18:25.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:25.467+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.468535+0000) 2020-02-06T01:18:25.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:25.468+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.469184+0000) 2020-02-06T01:18:25.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:25.468+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.469517+0000) 2020-02-06T01:18:25.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:25.469+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.469824+0000) 2020-02-06T01:18:25.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:25.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.470494+0000) 2020-02-06T01:18:25.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:25.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.472383+0000) 2020-02-06T01:18:25.487 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:25.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:25.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:25.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:25.688616+0000) 2020-02-06T01:18:26.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:26.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:26.092017+0000) 2020-02-06T01:18:26.763 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:18:26.770 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:18:27.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:27.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:27.075132+0000) 2020-02-06T01:18:27.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:27.342+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:27.343649+0000) 2020-02-06T01:18:27.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:27.342+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:27.343749+0000) 2020-02-06T01:18:27.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:27.342+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:27.344033+0000) 2020-02-06T01:18:29.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:29.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:29.718646+0000) 2020-02-06T01:18:29.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:29.966+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:29.966341+0000) 2020-02-06T01:18:30.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:30.492+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.492417+0000) 2020-02-06T01:18:30.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:30.492+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.493612+0000) 2020-02-06T01:18:30.495 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:30.493+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.494525+0000) 2020-02-06T01:18:30.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:30.495+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.495237+0000) 2020-02-06T01:18:30.496 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:30.494+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.495912+0000) 2020-02-06T01:18:30.497 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:30.496+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.496158+0000) 2020-02-06T01:18:30.497 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:30.496+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.496528+0000) 2020-02-06T01:18:30.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:30.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.497013+0000) 2020-02-06T01:18:30.498 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:30.496+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.497602+0000) 2020-02-06T01:18:30.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:30.497+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.498014+0000) 2020-02-06T01:18:30.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:30.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.498813+0000) 2020-02-06T01:18:30.500 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:30.497+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.499114+0000) 2020-02-06T01:18:30.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:30.497+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.499298+0000) 2020-02-06T01:18:30.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:30.504+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.504232+0000) 2020-02-06T01:18:30.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:30.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:30.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:30.569+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.570952+0000) 2020-02-06T01:18:30.829 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:30.828+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.828724+0000) 2020-02-06T01:18:30.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:30.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.850552+0000) 2020-02-06T01:18:30.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:30.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:30.850665+0000) 2020-02-06T01:18:32.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:32.953+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:32.955153+0000) 2020-02-06T01:18:33.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:33.182+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:33.183544+0000) 2020-02-06T01:18:35.521 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:35.519+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.520572+0000) 2020-02-06T01:18:35.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:35.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.521214+0000) 2020-02-06T01:18:35.522 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:35.520+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.521887+0000) 2020-02-06T01:18:35.523 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:35.521+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.522390+0000) 2020-02-06T01:18:35.523 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:35.521+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.522827+0000) 2020-02-06T01:18:35.524 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:35.522+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.523729+0000) 2020-02-06T01:18:35.525 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:35.522+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.524035+0000) 2020-02-06T01:18:35.525 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:35.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.524422+0000) 2020-02-06T01:18:35.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:35.523+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.525056+0000) 2020-02-06T01:18:35.526 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:35.524+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.525831+0000) 2020-02-06T01:18:35.527 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:35.524+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.526127+0000) 2020-02-06T01:18:35.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:35.525+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.526753+0000) 2020-02-06T01:18:35.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:35.525+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.526661+0000) 2020-02-06T01:18:35.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:35.531+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.532459+0000) 2020-02-06T01:18:35.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:35.541+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199569 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:35.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:35.695+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:35.696844+0000) 2020-02-06T01:18:36.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:36.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:36.092249+0000) 2020-02-06T01:18:37.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:37.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:37.075395+0000) 2020-02-06T01:18:37.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:37.342+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:37.343904+0000) 2020-02-06T01:18:37.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:37.342+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:37.344007+0000) 2020-02-06T01:18:37.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:37.342+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:37.344288+0000) 2020-02-06T01:18:39.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:39.717+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:39.718891+0000) 2020-02-06T01:18:39.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:39.965+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:39.966574+0000) 2020-02-06T01:18:40.550 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:40.548+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.549200+0000) 2020-02-06T01:18:40.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:40.548+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.549716+0000) 2020-02-06T01:18:40.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:40.549+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.550555+0000) 2020-02-06T01:18:40.552 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:40.549+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.551066+0000) 2020-02-06T01:18:40.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:40.550+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.551826+0000) 2020-02-06T01:18:40.553 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:40.550+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.551779+0000) 2020-02-06T01:18:40.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:40.551+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.552476+0000) 2020-02-06T01:18:40.554 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:40.552+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.553486+0000) 2020-02-06T01:18:40.554 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:40.551+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.553190+0000) 2020-02-06T01:18:40.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:40.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.554057+0000) 2020-02-06T01:18:40.555 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:40.553+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.554471+0000) 2020-02-06T01:18:40.556 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:40.553+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.554986+0000) 2020-02-06T01:18:40.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:40.553+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.554495+0000) 2020-02-06T01:18:40.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:40.559+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.561026+0000) 2020-02-06T01:18:40.571 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:40.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:40.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:40.569+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.571114+0000) 2020-02-06T01:18:40.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:40.827+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.828930+0000) 2020-02-06T01:18:40.851 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:40.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.850761+0000) 2020-02-06T01:18:40.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:40.849+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:40.850901+0000) 2020-02-06T01:18:42.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:42.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:42.955405+0000) 2020-02-06T01:18:43.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:43.182+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:43.183762+0000) 2020-02-06T01:18:45.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:45.573+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.575166+0000) 2020-02-06T01:18:45.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:45.574+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.575813+0000) 2020-02-06T01:18:45.577 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:45.575+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.576374+0000) 2020-02-06T01:18:45.577 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:45.575+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.576678+0000) 2020-02-06T01:18:45.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:45.576+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.577182+0000) 2020-02-06T01:18:45.578 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:45.576+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.577660+0000) 2020-02-06T01:18:45.581 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:45.578+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.579798+0000) 2020-02-06T01:18:45.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:45.578+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.580111+0000) 2020-02-06T01:18:45.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:45.580+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.581446+0000) 2020-02-06T01:18:45.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:45.580+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.582027+0000) 2020-02-06T01:18:45.583 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:45.581+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.582895+0000) 2020-02-06T01:18:45.584 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:45.581+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.583202+0000) 2020-02-06T01:18:45.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:45.582+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.583513+0000) 2020-02-06T01:18:45.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:45.586+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.587088+0000) 2020-02-06T01:18:45.599 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:45.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:45.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:45.703+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:45.704748+0000) 2020-02-06T01:18:46.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:46.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:46.092491+0000) 2020-02-06T01:18:47.076 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:47.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:47.075663+0000) 2020-02-06T01:18:47.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:47.343+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:47.344138+0000) 2020-02-06T01:18:47.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:47.343+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:47.344295+0000) 2020-02-06T01:18:47.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:47.343+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:47.344520+0000) 2020-02-06T01:18:49.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:49.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:49.719166+0000) 2020-02-06T01:18:49.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:49.966+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:49.966825+0000) 2020-02-06T01:18:50.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:50.570+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.571314+0000) 2020-02-06T01:18:50.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:50.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.603704+0000) 2020-02-06T01:18:50.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:50.603+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.604645+0000) 2020-02-06T01:18:50.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:50.603+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.605378+0000) 2020-02-06T01:18:50.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:50.604+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.605416+0000) 2020-02-06T01:18:50.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:50.604+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.605731+0000) 2020-02-06T01:18:50.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:50.605+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.606172+0000) 2020-02-06T01:18:50.607 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:50.605+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.606707+0000) 2020-02-06T01:18:50.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:50.605+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.606643+0000) 2020-02-06T01:18:50.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:50.605+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.607175+0000) 2020-02-06T01:18:50.608 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:50.606+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.607676+0000) 2020-02-06T01:18:50.609 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:50.606+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.607837+0000) 2020-02-06T01:18:50.609 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:50.606+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.608138+0000) 2020-02-06T01:18:50.609 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:50.607+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.608549+0000) 2020-02-06T01:18:50.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:50.611+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.612878+0000) 2020-02-06T01:18:50.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:50.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:50.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:50.828+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.829144+0000) 2020-02-06T01:18:50.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:50.849+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.850945+0000) 2020-02-06T01:18:50.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:50.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:50.851175+0000) 2020-02-06T01:18:52.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:52.954+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:52.955697+0000) 2020-02-06T01:18:53.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:53.183+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:53.183992+0000) 2020-02-06T01:18:55.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:55.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.634146+0000) 2020-02-06T01:18:55.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:55.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.634609+0000) 2020-02-06T01:18:55.636 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:18:55.633+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.634851+0000) 2020-02-06T01:18:55.636 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:18:55.634+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.635165+0000) 2020-02-06T01:18:55.636 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:18:55.634+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.635523+0000) 2020-02-06T01:18:55.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:18:55.634+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.635825+0000) 2020-02-06T01:18:55.637 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:55.634+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.636178+0000) 2020-02-06T01:18:55.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:18:55.635+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.636991+0000) 2020-02-06T01:18:55.638 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:55.635+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.637387+0000) 2020-02-06T01:18:55.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:18:55.636+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.637866+0000) 2020-02-06T01:18:55.639 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:55.636+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.638292+0000) 2020-02-06T01:18:55.640 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:55.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.638906+0000) 2020-02-06T01:18:55.640 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:55.637+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.639288+0000) 2020-02-06T01:18:55.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:55.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.644277+0000) 2020-02-06T01:18:55.654 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:18:55.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:18:55.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:55.711+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:55.711830+0000) 2020-02-06T01:18:56.093 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:18:56.091+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:56.092710+0000) 2020-02-06T01:18:56.971 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:18:56.977 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:18:57.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:18:57.074+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:57.075864+0000) 2020-02-06T01:18:57.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:18:57.343+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:57.344390+0000) 2020-02-06T01:18:57.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:18:57.343+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:57.344644+0000) 2020-02-06T01:18:57.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:18:57.343+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:57.344785+0000) 2020-02-06T01:18:59.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:59.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:59.719406+0000) 2020-02-06T01:18:59.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:18:59.965+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:18:59.967063+0000) 2020-02-06T01:19:00.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:00.570+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.571553+0000) 2020-02-06T01:19:00.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:00.657+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.658890+0000) 2020-02-06T01:19:00.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:00.658+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.659560+0000) 2020-02-06T01:19:00.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:00.658+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.660322+0000) 2020-02-06T01:19:00.661 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:00.659+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.660822+0000) 2020-02-06T01:19:00.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:00.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.661242+0000) 2020-02-06T01:19:00.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:00.660+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.662082+0000) 2020-02-06T01:19:00.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:00.661+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.662261+0000) 2020-02-06T01:19:00.663 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:00.661+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.662697+0000) 2020-02-06T01:19:00.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:00.661+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.663224+0000) 2020-02-06T01:19:00.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:00.662+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.663980+0000) 2020-02-06T01:19:00.665 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:00.662+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.664211+0000) 2020-02-06T01:19:00.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:00.664+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.665293+0000) 2020-02-06T01:19:00.666 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:00.664+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.666076+0000) 2020-02-06T01:19:00.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:00.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.670259+0000) 2020-02-06T01:19:00.683 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:00.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199771 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:00.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:00.828+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.829385+0000) 2020-02-06T01:19:00.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:00.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.851157+0000) 2020-02-06T01:19:00.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:00.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:00.851385+0000) 2020-02-06T01:19:02.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:02.955+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:02.956001+0000) 2020-02-06T01:19:03.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:03.183+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:03.184325+0000) 2020-02-06T01:19:05.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:05.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.687446+0000) 2020-02-06T01:19:05.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:05.687+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.688201+0000) 2020-02-06T01:19:05.689 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:05.687+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.688727+0000) 2020-02-06T01:19:05.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:05.688+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.689696+0000) 2020-02-06T01:19:05.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:05.688+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.690292+0000) 2020-02-06T01:19:05.693 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:05.691+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.692436+0000) 2020-02-06T01:19:05.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:05.690+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.692143+0000) 2020-02-06T01:19:05.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:05.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.693002+0000) 2020-02-06T01:19:05.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:05.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.693283+0000) 2020-02-06T01:19:05.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:05.692+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.693694+0000) 2020-02-06T01:19:05.695 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:05.692+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.693864+0000) 2020-02-06T01:19:05.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:05.692+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.694193+0000) 2020-02-06T01:19:05.695 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:05.692+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.694263+0000) 2020-02-06T01:19:05.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:05.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.696674+0000) 2020-02-06T01:19:05.710 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:05.707+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199809 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:05.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:05.722+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:05.723499+0000) 2020-02-06T01:19:06.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:06.092+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:06.092962+0000) 2020-02-06T01:19:07.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:07.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:07.076167+0000) 2020-02-06T01:19:07.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:07.344+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:07.344701+0000) 2020-02-06T01:19:07.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:07.344+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:07.344870+0000) 2020-02-06T01:19:07.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:07.344+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:07.345001+0000) 2020-02-06T01:19:09.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:09.718+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:09.719664+0000) 2020-02-06T01:19:09.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:09.966+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:09.967340+0000) 2020-02-06T01:19:10.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:10.571+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.571801+0000) 2020-02-06T01:19:10.716 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:10.714+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.715190+0000) 2020-02-06T01:19:10.718 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:10.715+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.716949+0000) 2020-02-06T01:19:10.718 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:10.715+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.717015+0000) 2020-02-06T01:19:10.718 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:10.716+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.717184+0000) 2020-02-06T01:19:10.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:10.716+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.717501+0000) 2020-02-06T01:19:10.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:10.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.718322+0000) 2020-02-06T01:19:10.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:10.717+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.718987+0000) 2020-02-06T01:19:10.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:10.717+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.719053+0000) 2020-02-06T01:19:10.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:10.717+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.719326+0000) 2020-02-06T01:19:10.721 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:10.718+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.720091+0000) 2020-02-06T01:19:10.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:10.719+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.720134+0000) 2020-02-06T01:19:10.721 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:10.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.721133+0000) 2020-02-06T01:19:10.722 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:10.720+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.721729+0000) 2020-02-06T01:19:10.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:10.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.723832+0000) 2020-02-06T01:19:10.736 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:10.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199852 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:10.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:10.829+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.829599+0000) 2020-02-06T01:19:10.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:10.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.851384+0000) 2020-02-06T01:19:10.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:10.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:10.851589+0000) 2020-02-06T01:19:12.957 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:12.955+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:12.956266+0000) 2020-02-06T01:19:13.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:13.184+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:13.184562+0000) 2020-02-06T01:19:15.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:15.730+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.731173+0000) 2020-02-06T01:19:15.741 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:15.739+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.740548+0000) 2020-02-06T01:19:15.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:15.740+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.741408+0000) 2020-02-06T01:19:15.743 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:15.740+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.741646+0000) 2020-02-06T01:19:15.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:15.741+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.742359+0000) 2020-02-06T01:19:15.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:15.741+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.742826+0000) 2020-02-06T01:19:15.745 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:15.741+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.742733+0000) 2020-02-06T01:19:15.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:15.742+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.743493+0000) 2020-02-06T01:19:15.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:15.742+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.743439+0000) 2020-02-06T01:19:15.746 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:15.742+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.743651+0000) 2020-02-06T01:19:15.747 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:15.742+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.744268+0000) 2020-02-06T01:19:15.747 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:15.743+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.745305+0000) 2020-02-06T01:19:15.747 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:15.744+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.746034+0000) 2020-02-06T01:19:15.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:15.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.745803+0000) 2020-02-06T01:19:15.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:15.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:15.749526+0000) 2020-02-06T01:19:15.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:15.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199889 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:16.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:16.092+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:16.093230+0000) 2020-02-06T01:19:17.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:17.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:17.076408+0000) 2020-02-06T01:19:17.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:17.344+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:17.344951+0000) 2020-02-06T01:19:17.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:17.344+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:17.345074+0000) 2020-02-06T01:19:17.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:17.344+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:17.345281+0000) 2020-02-06T01:19:19.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:19.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:19.719908+0000) 2020-02-06T01:19:19.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:19.967+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:19.967613+0000) 2020-02-06T01:19:20.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:20.571+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.572031+0000) 2020-02-06T01:19:20.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:20.766+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.767057+0000) 2020-02-06T01:19:20.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:20.766+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.767316+0000) 2020-02-06T01:19:20.769 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:20.767+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.768248+0000) 2020-02-06T01:19:20.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:20.767+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.768751+0000) 2020-02-06T01:19:20.770 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:20.768+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.769826+0000) 2020-02-06T01:19:20.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:20.769+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.770120+0000) 2020-02-06T01:19:20.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:20.770+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.770702+0000) 2020-02-06T01:19:20.771 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:20.769+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.770378+0000) 2020-02-06T01:19:20.772 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:20.770+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.770811+0000) 2020-02-06T01:19:20.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:20.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.771203+0000) 2020-02-06T01:19:20.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:20.771+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.772301+0000) 2020-02-06T01:19:20.773 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:20.772+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.772892+0000) 2020-02-06T01:19:20.774 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:20.773+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.773663+0000) 2020-02-06T01:19:20.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:20.777+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.778235+0000) 2020-02-06T01:19:20.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:20.788+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199931 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:20.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:20.828+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.829834+0000) 2020-02-06T01:19:20.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:20.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.851565+0000) 2020-02-06T01:19:20.852 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:20.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:20.851793+0000) 2020-02-06T01:19:22.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:22.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:22.956654+0000) 2020-02-06T01:19:23.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:23.184+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:23.184803+0000) 2020-02-06T01:19:25.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:25.739+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.739778+0000) 2020-02-06T01:19:25.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:25.794+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.794631+0000) 2020-02-06T01:19:25.796 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:25.795+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.795065+0000) 2020-02-06T01:19:25.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:25.795+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.795628+0000) 2020-02-06T01:19:25.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:25.796+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.796296+0000) 2020-02-06T01:19:25.797 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:25.796+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.796988+0000) 2020-02-06T01:19:25.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:25.797+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.797354+0000) 2020-02-06T01:19:25.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:25.798+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.798309+0000) 2020-02-06T01:19:25.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:25.797+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.798071+0000) 2020-02-06T01:19:25.800 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:25.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.798792+0000) 2020-02-06T01:19:25.800 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:25.799+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.799550+0000) 2020-02-06T01:19:25.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:25.799+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.800023+0000) 2020-02-06T01:19:25.801 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:25.799+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.800267+0000) 2020-02-06T01:19:25.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:25.801+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.801438+0000) 2020-02-06T01:19:25.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:25.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:25.806845+0000) 2020-02-06T01:19:25.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:25.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 199969 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:26.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:26.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:26.093488+0000) 2020-02-06T01:19:27.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:27.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:27.076647+0000) 2020-02-06T01:19:27.133 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:19:27.139 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:19:27.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:27.344+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:27.345216+0000) 2020-02-06T01:19:27.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:27.344+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:27.345287+0000) 2020-02-06T01:19:27.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:27.345+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:27.345558+0000) 2020-02-06T01:19:29.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:29.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:29.720150+0000) 2020-02-06T01:19:29.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:29.966+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:29.967839+0000) 2020-02-06T01:19:30.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:30.571+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.572272+0000) 2020-02-06T01:19:30.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:30.820+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.821524+0000) 2020-02-06T01:19:30.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:30.820+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.821899+0000) 2020-02-06T01:19:30.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:30.821+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.823020+0000) 2020-02-06T01:19:30.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:30.822+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.823693+0000) 2020-02-06T01:19:30.825 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:30.822+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.823386+0000) 2020-02-06T01:19:30.825 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:30.824+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.824822+0000) 2020-02-06T01:19:30.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:30.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.825426+0000) 2020-02-06T01:19:30.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:30.824+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.824936+0000) 2020-02-06T01:19:30.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:30.825+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.826183+0000) 2020-02-06T01:19:30.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:30.825+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.826937+0000) 2020-02-06T01:19:30.828 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:30.827+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.827731+0000) 2020-02-06T01:19:30.828 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:30.827+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.827691+0000) 2020-02-06T01:19:30.829 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:30.827+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.828299+0000) 2020-02-06T01:19:30.830 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:30.828+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.830016+0000) 2020-02-06T01:19:30.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:30.831+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.832970+0000) 2020-02-06T01:19:30.845 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:30.843+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200011 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:30.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:30.850+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.851726+0000) 2020-02-06T01:19:30.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:30.850+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:30.851945+0000) 2020-02-06T01:19:32.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:32.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:32.956907+0000) 2020-02-06T01:19:33.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:33.184+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:33.185126+0000) 2020-02-06T01:19:35.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:35.748+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.749666+0000) 2020-02-06T01:19:35.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:35.848+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.849133+0000) 2020-02-06T01:19:35.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:35.849+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.850271+0000) 2020-02-06T01:19:35.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:35.849+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.850569+0000) 2020-02-06T01:19:35.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:35.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.851415+0000) 2020-02-06T01:19:35.852 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:35.850+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.851475+0000) 2020-02-06T01:19:35.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:35.851+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.852293+0000) 2020-02-06T01:19:35.854 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:35.852+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.852978+0000) 2020-02-06T01:19:35.854 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:35.852+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.853288+0000) 2020-02-06T01:19:35.855 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:35.853+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.853715+0000) 2020-02-06T01:19:35.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:35.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.854776+0000) 2020-02-06T01:19:35.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:35.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.855263+0000) 2020-02-06T01:19:35.856 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:35.854+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.855215+0000) 2020-02-06T01:19:35.856 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:35.855+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.855637+0000) 2020-02-06T01:19:35.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:35.858+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:35.859927+0000) 2020-02-06T01:19:35.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:35.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200048 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:36.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:36.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:36.093722+0000) 2020-02-06T01:19:37.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:37.075+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:37.076885+0000) 2020-02-06T01:19:37.346 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:37.345+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:37.345528+0000) 2020-02-06T01:19:37.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:37.345+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:37.345533+0000) 2020-02-06T01:19:37.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:37.345+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:37.345826+0000) 2020-02-06T01:19:39.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:39.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:39.720389+0000) 2020-02-06T01:19:39.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:39.967+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:39.968106+0000) 2020-02-06T01:19:40.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:40.572+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.572563+0000) 2020-02-06T01:19:40.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:40.829+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.830228+0000) 2020-02-06T01:19:40.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:40.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.851924+0000) 2020-02-06T01:19:40.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:40.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.852157+0000) 2020-02-06T01:19:40.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:40.877+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.877494+0000) 2020-02-06T01:19:40.883 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:40.879+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.880065+0000) 2020-02-06T01:19:40.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:40.880+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.880492+0000) 2020-02-06T01:19:40.884 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:40.880+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.880987+0000) 2020-02-06T01:19:40.884 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:40.881+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.881790+0000) 2020-02-06T01:19:40.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:40.875+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.877057+0000) 2020-02-06T01:19:40.884 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:40.876+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.877994+0000) 2020-02-06T01:19:40.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:40.878+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.879238+0000) 2020-02-06T01:19:40.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:40.879+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.881001+0000) 2020-02-06T01:19:40.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:40.878+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.879408+0000) 2020-02-06T01:19:40.885 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:40.879+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.880386+0000) 2020-02-06T01:19:40.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:40.879+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.880827+0000) 2020-02-06T01:19:40.886 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:40.881+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.882259+0000) 2020-02-06T01:19:40.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:40.885+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:40.886186+0000) 2020-02-06T01:19:40.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:40.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200089 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:42.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:42.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:42.957179+0000) 2020-02-06T01:19:43.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:43.184+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:43.185357+0000) 2020-02-06T01:19:45.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:45.757+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.757960+0000) 2020-02-06T01:19:45.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:45.903+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.904853+0000) 2020-02-06T01:19:45.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:45.904+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.905318+0000) 2020-02-06T01:19:45.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:45.904+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.905491+0000) 2020-02-06T01:19:45.907 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:45.905+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.906503+0000) 2020-02-06T01:19:45.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:45.905+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.906988+0000) 2020-02-06T01:19:45.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:45.906+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.907694+0000) 2020-02-06T01:19:45.909 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:45.907+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.907925+0000) 2020-02-06T01:19:45.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:45.907+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.908739+0000) 2020-02-06T01:19:45.910 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:45.908+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.909486+0000) 2020-02-06T01:19:45.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:45.909+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.909927+0000) 2020-02-06T01:19:45.911 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:45.908+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.909176+0000) 2020-02-06T01:19:45.911 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:45.910+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.910718+0000) 2020-02-06T01:19:45.911 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:45.910+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.910766+0000) 2020-02-06T01:19:45.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:45.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:45.914134+0000) 2020-02-06T01:19:45.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:45.925+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:46.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:46.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:46.093966+0000) 2020-02-06T01:19:47.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:47.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:47.077118+0000) 2020-02-06T01:19:47.347 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:47.345+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:47.345807+0000) 2020-02-06T01:19:47.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:47.345+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:47.345807+0000) 2020-02-06T01:19:47.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:47.345+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:47.346075+0000) 2020-02-06T01:19:49.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:49.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:49.720636+0000) 2020-02-06T01:19:49.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:49.967+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:49.968362+0000) 2020-02-06T01:19:50.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:50.572+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.572810+0000) 2020-02-06T01:19:50.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:50.829+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.830457+0000) 2020-02-06T01:19:50.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:50.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.852159+0000) 2020-02-06T01:19:50.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:50.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.852378+0000) 2020-02-06T01:19:50.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:50.930+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.931468+0000) 2020-02-06T01:19:50.933 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:50.931+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.932276+0000) 2020-02-06T01:19:50.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:50.932+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.933070+0000) 2020-02-06T01:19:50.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:50.932+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.933549+0000) 2020-02-06T01:19:50.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:50.932+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.933990+0000) 2020-02-06T01:19:50.935 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:50.933+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.934134+0000) 2020-02-06T01:19:50.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:50.933+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.934590+0000) 2020-02-06T01:19:50.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:50.933+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.935028+0000) 2020-02-06T01:19:50.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:50.934+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.935373+0000) 2020-02-06T01:19:50.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:50.935+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.936321+0000) 2020-02-06T01:19:50.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:50.936+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.936666+0000) 2020-02-06T01:19:50.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:50.936+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.936894+0000) 2020-02-06T01:19:50.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:50.937+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.937573+0000) 2020-02-06T01:19:50.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:50.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:50.940753+0000) 2020-02-06T01:19:50.954 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:50.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:52.958 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:52.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:52.957413+0000) 2020-02-06T01:19:53.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:53.184+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:53.185594+0000) 2020-02-06T01:19:55.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:55.765+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.766115+0000) 2020-02-06T01:19:55.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:19:55.958+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.959882+0000) 2020-02-06T01:19:55.961 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:19:55.959+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.960613+0000) 2020-02-06T01:19:55.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:55.959+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.960800+0000) 2020-02-06T01:19:55.962 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:19:55.960+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.961287+0000) 2020-02-06T01:19:55.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:55.960+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.961759+0000) 2020-02-06T01:19:55.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:55.961+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.962788+0000) 2020-02-06T01:19:55.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:55.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.963400+0000) 2020-02-06T01:19:55.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:55.962+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.963821+0000) 2020-02-06T01:19:55.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:19:55.962+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.963946+0000) 2020-02-06T01:19:55.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:55.961+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.963083+0000) 2020-02-06T01:19:55.965 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:19:55.962+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.963623+0000) 2020-02-06T01:19:55.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:19:55.962+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.964222+0000) 2020-02-06T01:19:55.966 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:55.963+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.964809+0000) 2020-02-06T01:19:55.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:55.963+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:55.964782+0000) 2020-02-06T01:19:55.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:19:55.979+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200206 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:19:56.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:19:56.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:56.094209+0000) 2020-02-06T01:19:57.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:19:57.076+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:57.077354+0000) 2020-02-06T01:19:57.280 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:19:57.286 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:19:57.347 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:19:57.344+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:57.346074+0000) 2020-02-06T01:19:57.347 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:19:57.344+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:57.346073+0000) 2020-02-06T01:19:57.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:19:57.344+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:57.346330+0000) 2020-02-06T01:19:59.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:59.719+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:59.720881+0000) 2020-02-06T01:19:59.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:19:59.968+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:19:59.968625+0000) 2020-02-06T01:20:00.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:00.572+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.573065+0000) 2020-02-06T01:20:00.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:00.830+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.830688+0000) 2020-02-06T01:20:00.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:00.851+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.852426+0000) 2020-02-06T01:20:00.853 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:00.851+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.852614+0000) 2020-02-06T01:20:00.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:00.986+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.987731+0000) 2020-02-06T01:20:00.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:00.987+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.988248+0000) 2020-02-06T01:20:00.994 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:00.988+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.989557+0000) 2020-02-06T01:20:00.994 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:00.989+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.990591+0000) 2020-02-06T01:20:00.994 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:00.990+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.991555+0000) 2020-02-06T01:20:00.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:00.988+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.989124+0000) 2020-02-06T01:20:00.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:00.988+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.990044+0000) 2020-02-06T01:20:00.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:00.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.992413+0000) 2020-02-06T01:20:00.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:00.989+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.991007+0000) 2020-02-06T01:20:00.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:00.990+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.991903+0000) 2020-02-06T01:20:00.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:00.991+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.992787+0000) 2020-02-06T01:20:00.996 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:00.990+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.992168+0000) 2020-02-06T01:20:00.996 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:00.991+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.992916+0000) 2020-02-06T01:20:00.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:00.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:00.998365+0000) 2020-02-06T01:20:01.009 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:01.007+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200247 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:02.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:02.956+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:02.957675+0000) 2020-02-06T01:20:03.186 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:03.185+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:03.185843+0000) 2020-02-06T01:20:05.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:05.773+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:05.774361+0000) 2020-02-06T01:20:06.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:06.014+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.015504+0000) 2020-02-06T01:20:06.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:06.014+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.015835+0000) 2020-02-06T01:20:06.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:06.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.016629+0000) 2020-02-06T01:20:06.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:06.016+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.017642+0000) 2020-02-06T01:20:06.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:06.017+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.018645+0000) 2020-02-06T01:20:06.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:06.017+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.018866+0000) 2020-02-06T01:20:06.020 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:06.018+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.019528+0000) 2020-02-06T01:20:06.021 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:06.018+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.019983+0000) 2020-02-06T01:20:06.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:06.019+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.020462+0000) 2020-02-06T01:20:06.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:06.019+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.020759+0000) 2020-02-06T01:20:06.022 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:06.019+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.020619+0000) 2020-02-06T01:20:06.022 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:06.019+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.020833+0000) 2020-02-06T01:20:06.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:06.020+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.021474+0000) 2020-02-06T01:20:06.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:06.025+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.026223+0000) 2020-02-06T01:20:06.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:06.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:06.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:06.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:06.094476+0000) 2020-02-06T01:20:07.078 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:07.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:07.077574+0000) 2020-02-06T01:20:07.347 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:07.345+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:07.346382+0000) 2020-02-06T01:20:07.348 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:07.345+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:07.346382+0000) 2020-02-06T01:20:07.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:07.345+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:07.346627+0000) 2020-02-06T01:20:09.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:09.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:09.721143+0000) 2020-02-06T01:20:09.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:09.968+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:09.968854+0000) 2020-02-06T01:20:10.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:10.571+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:10.573309+0000) 2020-02-06T01:20:10.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:10.830+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:10.830923+0000) 2020-02-06T01:20:10.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:10.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:10.852665+0000) 2020-02-06T01:20:10.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:10.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:10.852880+0000) 2020-02-06T01:20:11.045 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:11.043+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.044492+0000) 2020-02-06T01:20:11.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:11.044+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.045066+0000) 2020-02-06T01:20:11.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:11.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.045610+0000) 2020-02-06T01:20:11.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:11.044+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.046055+0000) 2020-02-06T01:20:11.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:11.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.046027+0000) 2020-02-06T01:20:11.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:11.045+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.046794+0000) 2020-02-06T01:20:11.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:11.045+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.046511+0000) 2020-02-06T01:20:11.048 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:11.046+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.047184+0000) 2020-02-06T01:20:11.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:11.046+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.047561+0000) 2020-02-06T01:20:11.049 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:11.045+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.047351+0000) 2020-02-06T01:20:11.049 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:11.046+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.047586+0000) 2020-02-06T01:20:11.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:11.047+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.048420+0000) 2020-02-06T01:20:11.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:11.047+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.048937+0000) 2020-02-06T01:20:11.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:11.051+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:11.052532+0000) 2020-02-06T01:20:11.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:11.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200327 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:12.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:12.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:12.957933+0000) 2020-02-06T01:20:13.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:13.185+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:13.186122+0000) 2020-02-06T01:20:15.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:15.782+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:15.782674+0000) 2020-02-06T01:20:16.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:16.069+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.071063+0000) 2020-02-06T01:20:16.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:16.070+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.071863+0000) 2020-02-06T01:20:16.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:16.072+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.073502+0000) 2020-02-06T01:20:16.076 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:16.074+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.075328+0000) 2020-02-06T01:20:16.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:16.074+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.075570+0000) 2020-02-06T01:20:16.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:16.073+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.075040+0000) 2020-02-06T01:20:16.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:16.074+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.076029+0000) 2020-02-06T01:20:16.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:16.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.076948+0000) 2020-02-06T01:20:16.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:16.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.077889+0000) 2020-02-06T01:20:16.079 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:16.077+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.078804+0000) 2020-02-06T01:20:16.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:16.077+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.079352+0000) 2020-02-06T01:20:16.080 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:16.078+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.079864+0000) 2020-02-06T01:20:16.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:16.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.080568+0000) 2020-02-06T01:20:16.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:16.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.082474+0000) 2020-02-06T01:20:16.095 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:16.093+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200364 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:16.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:16.093+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:16.094620+0000) 2020-02-06T01:20:17.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:17.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:17.077808+0000) 2020-02-06T01:20:17.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:17.345+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:17.346695+0000) 2020-02-06T01:20:17.348 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:17.345+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:17.346695+0000) 2020-02-06T01:20:17.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:17.345+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:17.346884+0000) 2020-02-06T01:20:19.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:19.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:19.721384+0000) 2020-02-06T01:20:19.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:19.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:19.969122+0000) 2020-02-06T01:20:20.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:20.572+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:20.573560+0000) 2020-02-06T01:20:20.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:20.831+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:20.831149+0000) 2020-02-06T01:20:20.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:20.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:20.852897+0000) 2020-02-06T01:20:20.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:20.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:20.853134+0000) 2020-02-06T01:20:21.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:21.099+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.099664+0000) 2020-02-06T01:20:21.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:21.101+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.101145+0000) 2020-02-06T01:20:21.102 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:21.101+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.101320+0000) 2020-02-06T01:20:21.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:21.100+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.101928+0000) 2020-02-06T01:20:21.103 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:21.100+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.102267+0000) 2020-02-06T01:20:21.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:21.101+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.101799+0000) 2020-02-06T01:20:21.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:21.101+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.102612+0000) 2020-02-06T01:20:21.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:21.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.103068+0000) 2020-02-06T01:20:21.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:21.101+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.103081+0000) 2020-02-06T01:20:21.105 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:21.103+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.104017+0000) 2020-02-06T01:20:21.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:21.103+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.104601+0000) 2020-02-06T01:20:21.105 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:21.104+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.104955+0000) 2020-02-06T01:20:21.106 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:21.105+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.105413+0000) 2020-02-06T01:20:21.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:21.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:21.107461+0000) 2020-02-06T01:20:21.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:21.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200405 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:22.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:22.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:22.958181+0000) 2020-02-06T01:20:23.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:23.185+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:23.186347+0000) 2020-02-06T01:20:25.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:25.789+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:25.791016+0000) 2020-02-06T01:20:26.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:26.094+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.094822+0000) 2020-02-06T01:20:26.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:26.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.127703+0000) 2020-02-06T01:20:26.128 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:26.126+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.127746+0000) 2020-02-06T01:20:26.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:26.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.128327+0000) 2020-02-06T01:20:26.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:26.127+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.128847+0000) 2020-02-06T01:20:26.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:26.127+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.129213+0000) 2020-02-06T01:20:26.130 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:26.128+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.130079+0000) 2020-02-06T01:20:26.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:26.129+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.130785+0000) 2020-02-06T01:20:26.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:26.130+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.131057+0000) 2020-02-06T01:20:26.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:26.130+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.131440+0000) 2020-02-06T01:20:26.132 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:26.130+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.131549+0000) 2020-02-06T01:20:26.133 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:26.130+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.131864+0000) 2020-02-06T01:20:26.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:26.130+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.132341+0000) 2020-02-06T01:20:26.134 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:26.131+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.132661+0000) 2020-02-06T01:20:26.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:26.135+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:26.136284+0000) 2020-02-06T01:20:26.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:26.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200443 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:27.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:27.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:27.078065+0000) 2020-02-06T01:20:27.348 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:27.346+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:27.346996+0000) 2020-02-06T01:20:27.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:27.346+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:27.347000+0000) 2020-02-06T01:20:27.348 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:27.346+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:27.347163+0000) 2020-02-06T01:20:27.469 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:20:27.475 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:20:29.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:29.720+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:29.721617+0000) 2020-02-06T01:20:29.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:29.968+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:29.969370+0000) 2020-02-06T01:20:30.574 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:30.573+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:30.573818+0000) 2020-02-06T01:20:30.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:30.830+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:30.831381+0000) 2020-02-06T01:20:30.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:30.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:30.853124+0000) 2020-02-06T01:20:30.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:30.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:30.853363+0000) 2020-02-06T01:20:31.155 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:31.153+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.154547+0000) 2020-02-06T01:20:31.156 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:31.154+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.155420+0000) 2020-02-06T01:20:31.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:31.154+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.155774+0000) 2020-02-06T01:20:31.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:31.155+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.156743+0000) 2020-02-06T01:20:31.158 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:31.155+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.157188+0000) 2020-02-06T01:20:31.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:31.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.158204+0000) 2020-02-06T01:20:31.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:31.157+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.158547+0000) 2020-02-06T01:20:31.160 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:31.157+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.158491+0000) 2020-02-06T01:20:31.161 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:31.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.159274+0000) 2020-02-06T01:20:31.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:31.157+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.159207+0000) 2020-02-06T01:20:31.161 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:31.157+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.158910+0000) 2020-02-06T01:20:31.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:31.158+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.159255+0000) 2020-02-06T01:20:31.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:31.163+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.164105+0000) 2020-02-06T01:20:31.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:31.163+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:31.164430+0000) 2020-02-06T01:20:31.177 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:31.175+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200485 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:32.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:32.957+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:32.958478+0000) 2020-02-06T01:20:33.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:33.185+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:33.186596+0000) 2020-02-06T01:20:35.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:35.798+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:35.799570+0000) 2020-02-06T01:20:36.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:36.094+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.095082+0000) 2020-02-06T01:20:36.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:36.180+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.181511+0000) 2020-02-06T01:20:36.183 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:36.181+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.182481+0000) 2020-02-06T01:20:36.185 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:36.182+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.183811+0000) 2020-02-06T01:20:36.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:36.183+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.184408+0000) 2020-02-06T01:20:36.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:36.183+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.184865+0000) 2020-02-06T01:20:36.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:36.184+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.185481+0000) 2020-02-06T01:20:36.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:36.184+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.185808+0000) 2020-02-06T01:20:36.187 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:36.184+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.186222+0000) 2020-02-06T01:20:36.187 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:36.185+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.186308+0000) 2020-02-06T01:20:36.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:36.185+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.187219+0000) 2020-02-06T01:20:36.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:36.185+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.187405+0000) 2020-02-06T01:20:36.188 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:36.186+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.187852+0000) 2020-02-06T01:20:36.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:36.187+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.188501+0000) 2020-02-06T01:20:36.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:36.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:36.192207+0000) 2020-02-06T01:20:36.204 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:36.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200522 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:37.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:37.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:37.078333+0000) 2020-02-06T01:20:37.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:37.346+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:37.347296+0000) 2020-02-06T01:20:37.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:37.346+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:37.347294+0000) 2020-02-06T01:20:37.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:37.346+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:37.347550+0000) 2020-02-06T01:20:39.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:39.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:39.721854+0000) 2020-02-06T01:20:39.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:39.968+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:39.969624+0000) 2020-02-06T01:20:40.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:40.573+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:40.574067+0000) 2020-02-06T01:20:40.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:40.830+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:40.831642+0000) 2020-02-06T01:20:40.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:40.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:40.853401+0000) 2020-02-06T01:20:40.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:40.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:40.853607+0000) 2020-02-06T01:20:41.209 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:41.207+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.208303+0000) 2020-02-06T01:20:41.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:41.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.209131+0000) 2020-02-06T01:20:41.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:41.208+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.210190+0000) 2020-02-06T01:20:41.211 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:41.209+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.210517+0000) 2020-02-06T01:20:41.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:41.209+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.210943+0000) 2020-02-06T01:20:41.212 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:41.210+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.211679+0000) 2020-02-06T01:20:41.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:41.211+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.212417+0000) 2020-02-06T01:20:41.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:41.211+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.212945+0000) 2020-02-06T01:20:41.214 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:41.211+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.212802+0000) 2020-02-06T01:20:41.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:41.211+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.213284+0000) 2020-02-06T01:20:41.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:41.212+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.213859+0000) 2020-02-06T01:20:41.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:41.212+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.214244+0000) 2020-02-06T01:20:41.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:41.214+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.215144+0000) 2020-02-06T01:20:41.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:41.216+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:41.217742+0000) 2020-02-06T01:20:41.234 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:41.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:42.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:42.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:42.958740+0000) 2020-02-06T01:20:43.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:43.186+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:43.186826+0000) 2020-02-06T01:20:45.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:45.806+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:45.807597+0000) 2020-02-06T01:20:46.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:46.094+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.095317+0000) 2020-02-06T01:20:46.235 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:46.233+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.234464+0000) 2020-02-06T01:20:46.236 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:46.234+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.235843+0000) 2020-02-06T01:20:46.237 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:46.235+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.237054+0000) 2020-02-06T01:20:46.238 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:46.237+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.237491+0000) 2020-02-06T01:20:46.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:46.236+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.237566+0000) 2020-02-06T01:20:46.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:46.237+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.238058+0000) 2020-02-06T01:20:46.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:46.238+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.238635+0000) 2020-02-06T01:20:46.240 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:46.237+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.238358+0000) 2020-02-06T01:20:46.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:46.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.238545+0000) 2020-02-06T01:20:46.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:46.237+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.239021+0000) 2020-02-06T01:20:46.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:46.238+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.239081+0000) 2020-02-06T01:20:46.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:46.238+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.239380+0000) 2020-02-06T01:20:46.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:46.239+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.240083+0000) 2020-02-06T01:20:46.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:46.243+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:46.244301+0000) 2020-02-06T01:20:46.258 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:46.257+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200601 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:47.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:47.077+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:47.078585+0000) 2020-02-06T01:20:47.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:47.347+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:47.347568+0000) 2020-02-06T01:20:47.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:47.347+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:47.347596+0000) 2020-02-06T01:20:47.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:47.347+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:47.347807+0000) 2020-02-06T01:20:49.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:49.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:49.722135+0000) 2020-02-06T01:20:49.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:49.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:49.969859+0000) 2020-02-06T01:20:50.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:50.573+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:50.574316+0000) 2020-02-06T01:20:50.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:50.831+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:50.831866+0000) 2020-02-06T01:20:50.854 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:50.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:50.853629+0000) 2020-02-06T01:20:50.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:50.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:50.853831+0000) 2020-02-06T01:20:51.265 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:51.262+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.263846+0000) 2020-02-06T01:20:51.266 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:51.264+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.265095+0000) 2020-02-06T01:20:51.266 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:51.264+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.265431+0000) 2020-02-06T01:20:51.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:51.265+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.265956+0000) 2020-02-06T01:20:51.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:51.265+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.266706+0000) 2020-02-06T01:20:51.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:51.266+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.267232+0000) 2020-02-06T01:20:51.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:51.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.267622+0000) 2020-02-06T01:20:51.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:51.267+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.268417+0000) 2020-02-06T01:20:51.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:51.267+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.268935+0000) 2020-02-06T01:20:51.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:51.269+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.269585+0000) 2020-02-06T01:20:51.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:51.269+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.269875+0000) 2020-02-06T01:20:51.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:51.270+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.270573+0000) 2020-02-06T01:20:51.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:51.270+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.270931+0000) 2020-02-06T01:20:51.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:51.274+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:51.276065+0000) 2020-02-06T01:20:51.287 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:51.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200643 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:52.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:52.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:52.958991+0000) 2020-02-06T01:20:53.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:53.186+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:53.187029+0000) 2020-02-06T01:20:55.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:55.815+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:55.815838+0000) 2020-02-06T01:20:56.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:56.095+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.095685+0000) 2020-02-06T01:20:56.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:56.290+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.291514+0000) 2020-02-06T01:20:56.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:20:56.293+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.294614+0000) 2020-02-06T01:20:56.296 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:20:56.293+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.294281+0000) 2020-02-06T01:20:56.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:20:56.294+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.295704+0000) 2020-02-06T01:20:56.297 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:20:56.295+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.296150+0000) 2020-02-06T01:20:56.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:20:56.296+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.296689+0000) 2020-02-06T01:20:56.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:56.296+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.298008+0000) 2020-02-06T01:20:56.299 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:56.296+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.297000+0000) 2020-02-06T01:20:56.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:20:56.297+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.298503+0000) 2020-02-06T01:20:56.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:56.297+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.298657+0000) 2020-02-06T01:20:56.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:20:56.298+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.298627+0000) 2020-02-06T01:20:56.300 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:56.298+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.298424+0000) 2020-02-06T01:20:56.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:56.298+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.299402+0000) 2020-02-06T01:20:56.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:56.299+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:56.300607+0000) 2020-02-06T01:20:56.315 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:20:56.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200681 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:20:57.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:20:57.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:57.078784+0000) 2020-02-06T01:20:57.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:20:57.347+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:57.347873+0000) 2020-02-06T01:20:57.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:20:57.347+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:57.347875+0000) 2020-02-06T01:20:57.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:20:57.347+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:57.348054+0000) 2020-02-06T01:20:57.647 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:20:57.653 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:20:59.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:59.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:59.722392+0000) 2020-02-06T01:20:59.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:20:59.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:20:59.970122+0000) 2020-02-06T01:21:00.575 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:00.574+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:00.574695+0000) 2020-02-06T01:21:00.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:00.831+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:00.832175+0000) 2020-02-06T01:21:00.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:00.852+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:00.853878+0000) 2020-02-06T01:21:00.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:00.852+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:00.854052+0000) 2020-02-06T01:21:01.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:01.318+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.319960+0000) 2020-02-06T01:21:01.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:01.319+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.320878+0000) 2020-02-06T01:21:01.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:01.320+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.320694+0000) 2020-02-06T01:21:01.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:01.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.321568+0000) 2020-02-06T01:21:01.322 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:01.321+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.322160+0000) 2020-02-06T01:21:01.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:01.322+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.323258+0000) 2020-02-06T01:21:01.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:01.322+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.322821+0000) 2020-02-06T01:21:01.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:01.323+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.324011+0000) 2020-02-06T01:21:01.325 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:01.323+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.324845+0000) 2020-02-06T01:21:01.326 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:01.324+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.325095+0000) 2020-02-06T01:21:01.326 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:01.325+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.325864+0000) 2020-02-06T01:21:01.327 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:01.325+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.326378+0000) 2020-02-06T01:21:01.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:01.326+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.326965+0000) 2020-02-06T01:21:01.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:01.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:01.330999+0000) 2020-02-06T01:21:01.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:01.343+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:02.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:02.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:02.959252+0000) 2020-02-06T01:21:03.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:03.186+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:03.187246+0000) 2020-02-06T01:21:05.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:05.822+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:05.824045+0000) 2020-02-06T01:21:06.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:06.095+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.095928+0000) 2020-02-06T01:21:06.350 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:06.348+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.349317+0000) 2020-02-06T01:21:06.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:06.349+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.350071+0000) 2020-02-06T01:21:06.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:06.349+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.350548+0000) 2020-02-06T01:21:06.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:06.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.351158+0000) 2020-02-06T01:21:06.352 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:06.350+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.351431+0000) 2020-02-06T01:21:06.353 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:06.351+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.352243+0000) 2020-02-06T01:21:06.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:06.351+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.352710+0000) 2020-02-06T01:21:06.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:06.352+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.352682+0000) 2020-02-06T01:21:06.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:06.352+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.353078+0000) 2020-02-06T01:21:06.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:06.353+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.353718+0000) 2020-02-06T01:21:06.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:06.354+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.354707+0000) 2020-02-06T01:21:06.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:06.354+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.354670+0000) 2020-02-06T01:21:06.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:06.354+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.355206+0000) 2020-02-06T01:21:06.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:06.359+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:06.360325+0000) 2020-02-06T01:21:06.370 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:06.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:07.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:07.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:07.079003+0000) 2020-02-06T01:21:07.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:07.347+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:07.348164+0000) 2020-02-06T01:21:07.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:07.347+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:07.348166+0000) 2020-02-06T01:21:07.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:07.347+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:07.348339+0000) 2020-02-06T01:21:09.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:09.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:09.722693+0000) 2020-02-06T01:21:09.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:09.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:09.970349+0000) 2020-02-06T01:21:10.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:10.574+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:10.574951+0000) 2020-02-06T01:21:10.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:10.832+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:10.832420+0000) 2020-02-06T01:21:10.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:10.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:10.854123+0000) 2020-02-06T01:21:10.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:10.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:10.854343+0000) 2020-02-06T01:21:11.376 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:11.373+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.374824+0000) 2020-02-06T01:21:11.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:11.374+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.375231+0000) 2020-02-06T01:21:11.376 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:11.374+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.375642+0000) 2020-02-06T01:21:11.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:11.375+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.376775+0000) 2020-02-06T01:21:11.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:11.377+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.377607+0000) 2020-02-06T01:21:11.378 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:11.377+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.377917+0000) 2020-02-06T01:21:11.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:11.377+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.378281+0000) 2020-02-06T01:21:11.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:11.378+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.378625+0000) 2020-02-06T01:21:11.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:11.378+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.378792+0000) 2020-02-06T01:21:11.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:11.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.378923+0000) 2020-02-06T01:21:11.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:11.378+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.379364+0000) 2020-02-06T01:21:11.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:11.378+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.379850+0000) 2020-02-06T01:21:11.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:11.379+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.379936+0000) 2020-02-06T01:21:11.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:11.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:11.385564+0000) 2020-02-06T01:21:11.398 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:11.397+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:12.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:12.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:12.959510+0000) 2020-02-06T01:21:13.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:13.187+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:13.187504+0000) 2020-02-06T01:21:15.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:15.831+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:15.831938+0000) 2020-02-06T01:21:16.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:16.096+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.096191+0000) 2020-02-06T01:21:16.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:16.404+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.404633+0000) 2020-02-06T01:21:16.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:16.404+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.405263+0000) 2020-02-06T01:21:16.407 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:16.406+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.406284+0000) 2020-02-06T01:21:16.407 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:16.405+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.405976+0000) 2020-02-06T01:21:16.408 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:16.406+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.407292+0000) 2020-02-06T01:21:16.408 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:16.407+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.407641+0000) 2020-02-06T01:21:16.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:16.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.408183+0000) 2020-02-06T01:21:16.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:16.408+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.408450+0000) 2020-02-06T01:21:16.409 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:16.408+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.408802+0000) 2020-02-06T01:21:16.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:16.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.409404+0000) 2020-02-06T01:21:16.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:16.410+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.410129+0000) 2020-02-06T01:21:16.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:16.410+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.411088+0000) 2020-02-06T01:21:16.412 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:16.410+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.410699+0000) 2020-02-06T01:21:16.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:16.416+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:16.416125+0000) 2020-02-06T01:21:16.425 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:16.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:17.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:17.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:17.079194+0000) 2020-02-06T01:21:17.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:17.348+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:17.348471+0000) 2020-02-06T01:21:17.350 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:17.348+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:17.348471+0000) 2020-02-06T01:21:17.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:17.348+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:17.348601+0000) 2020-02-06T01:21:19.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:19.721+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:19.722924+0000) 2020-02-06T01:21:19.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:19.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:19.970582+0000) 2020-02-06T01:21:20.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:20.574+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:20.575203+0000) 2020-02-06T01:21:20.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:20.831+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:20.832660+0000) 2020-02-06T01:21:20.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:20.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:20.854353+0000) 2020-02-06T01:21:20.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:20.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:20.854601+0000) 2020-02-06T01:21:21.432 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:21.430+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.431516+0000) 2020-02-06T01:21:21.433 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:21.429+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.431173+0000) 2020-02-06T01:21:21.433 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:21.431+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.432141+0000) 2020-02-06T01:21:21.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:21.431+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.432462+0000) 2020-02-06T01:21:21.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:21.431+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.432703+0000) 2020-02-06T01:21:21.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:21.432+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.433804+0000) 2020-02-06T01:21:21.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:21.433+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.434454+0000) 2020-02-06T01:21:21.436 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:21.433+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.434902+0000) 2020-02-06T01:21:21.436 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:21.432+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.434034+0000) 2020-02-06T01:21:21.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:21.434+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.435654+0000) 2020-02-06T01:21:21.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:21.434+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.435895+0000) 2020-02-06T01:21:21.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:21.434+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.436020+0000) 2020-02-06T01:21:21.437 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:21.435+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.436854+0000) 2020-02-06T01:21:21.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:21.439+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:21.440595+0000) 2020-02-06T01:21:21.453 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:21.451+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:22.960 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:22.958+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:22.959748+0000) 2020-02-06T01:21:23.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:23.186+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:23.187733+0000) 2020-02-06T01:21:25.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:25.838+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:25.839567+0000) 2020-02-06T01:21:26.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:26.095+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.096487+0000) 2020-02-06T01:21:26.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:26.457+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.458311+0000) 2020-02-06T01:21:26.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:26.458+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.460084+0000) 2020-02-06T01:21:26.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:26.458+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.459852+0000) 2020-02-06T01:21:26.462 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:26.459+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.461022+0000) 2020-02-06T01:21:26.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:26.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.461763+0000) 2020-02-06T01:21:26.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:26.460+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.461842+0000) 2020-02-06T01:21:26.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:26.460+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.462396+0000) 2020-02-06T01:21:26.463 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:26.461+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.462940+0000) 2020-02-06T01:21:26.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:26.461+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.463296+0000) 2020-02-06T01:21:26.465 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:26.462+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.464050+0000) 2020-02-06T01:21:26.465 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:26.463+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.464484+0000) 2020-02-06T01:21:26.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:26.463+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.464757+0000) 2020-02-06T01:21:26.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:26.463+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.464791+0000) 2020-02-06T01:21:26.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:26.466+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:26.467529+0000) 2020-02-06T01:21:26.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:26.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:27.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:27.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:27.079396+0000) 2020-02-06T01:21:27.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:27.347+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:27.348761+0000) 2020-02-06T01:21:27.350 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:27.347+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:27.348762+0000) 2020-02-06T01:21:27.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:27.347+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:27.348858+0000) 2020-02-06T01:21:27.828 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:21:27.834 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:21:29.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:29.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:29.723185+0000) 2020-02-06T01:21:29.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:29.969+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:29.970805+0000) 2020-02-06T01:21:30.576 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:30.574+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:30.575486+0000) 2020-02-06T01:21:30.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:30.832+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:30.832895+0000) 2020-02-06T01:21:30.855 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:30.853+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:30.854592+0000) 2020-02-06T01:21:30.855 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:30.853+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:30.854822+0000) 2020-02-06T01:21:31.486 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:31.484+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.485391+0000) 2020-02-06T01:21:31.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:31.485+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.487242+0000) 2020-02-06T01:21:31.489 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:31.487+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.488223+0000) 2020-02-06T01:21:31.489 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:31.487+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.488413+0000) 2020-02-06T01:21:31.489 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:31.487+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.488583+0000) 2020-02-06T01:21:31.490 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:31.487+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.489232+0000) 2020-02-06T01:21:31.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:31.488+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.489071+0000) 2020-02-06T01:21:31.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:31.488+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.489059+0000) 2020-02-06T01:21:31.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:31.488+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.489617+0000) 2020-02-06T01:21:31.492 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:31.488+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.490203+0000) 2020-02-06T01:21:31.492 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:31.488+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.490401+0000) 2020-02-06T01:21:31.492 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:31.489+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.490641+0000) 2020-02-06T01:21:31.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:31.489+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.490939+0000) 2020-02-06T01:21:31.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:31.495+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:31.496766+0000) 2020-02-06T01:21:31.509 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:31.507+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 200964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:32.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:32.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:32.959992+0000) 2020-02-06T01:21:33.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:33.187+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:33.187968+0000) 2020-02-06T01:21:35.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:35.846+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:35.847677+0000) 2020-02-06T01:21:36.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:36.095+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.096732+0000) 2020-02-06T01:21:36.514 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:36.511+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.513014+0000) 2020-02-06T01:21:36.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:36.514+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.515153+0000) 2020-02-06T01:21:36.516 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:36.514+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.515738+0000) 2020-02-06T01:21:36.517 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:36.514+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.516323+0000) 2020-02-06T01:21:36.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:36.515+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.516893+0000) 2020-02-06T01:21:36.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:36.516+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.517691+0000) 2020-02-06T01:21:36.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:36.516+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.518037+0000) 2020-02-06T01:21:36.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:36.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.518312+0000) 2020-02-06T01:21:36.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:36.518+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.519233+0000) 2020-02-06T01:21:36.520 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:36.517+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.518931+0000) 2020-02-06T01:21:36.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:36.518+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.519794+0000) 2020-02-06T01:21:36.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:36.518+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.520070+0000) 2020-02-06T01:21:36.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:36.521+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.522887+0000) 2020-02-06T01:21:36.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:36.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:36.523219+0000) 2020-02-06T01:21:36.541 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:36.534+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:37.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:37.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:37.079626+0000) 2020-02-06T01:21:37.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:37.348+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:37.349029+0000) 2020-02-06T01:21:37.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:37.348+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:37.349124+0000) 2020-02-06T01:21:37.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:37.348+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:37.349026+0000) 2020-02-06T01:21:39.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:39.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:39.723444+0000) 2020-02-06T01:21:39.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:39.970+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:39.971052+0000) 2020-02-06T01:21:40.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:40.574+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:40.575712+0000) 2020-02-06T01:21:40.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:40.832+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:40.833123+0000) 2020-02-06T01:21:40.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:40.854+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:40.854801+0000) 2020-02-06T01:21:40.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:40.854+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:40.855010+0000) 2020-02-06T01:21:41.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:41.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.542053+0000) 2020-02-06T01:21:41.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:41.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.542732+0000) 2020-02-06T01:21:41.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:41.542+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.543361+0000) 2020-02-06T01:21:41.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:41.542+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.544204+0000) 2020-02-06T01:21:41.545 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:41.542+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.543951+0000) 2020-02-06T01:21:41.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:41.543+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.545105+0000) 2020-02-06T01:21:41.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:41.544+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.545390+0000) 2020-02-06T01:21:41.547 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:41.544+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.545620+0000) 2020-02-06T01:21:41.547 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:41.544+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.545780+0000) 2020-02-06T01:21:41.547 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:41.544+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.546006+0000) 2020-02-06T01:21:41.548 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:41.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.546289+0000) 2020-02-06T01:21:41.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:41.545+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.546502+0000) 2020-02-06T01:21:41.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:41.549+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.550296+0000) 2020-02-06T01:21:41.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:41.549+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:41.550615+0000) 2020-02-06T01:21:41.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:41.562+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201048 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:42.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:42.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:42.960263+0000) 2020-02-06T01:21:43.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:43.187+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:43.188277+0000) 2020-02-06T01:21:45.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:45.854+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:45.855592+0000) 2020-02-06T01:21:46.098 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:46.096+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.096973+0000) 2020-02-06T01:21:46.571 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:46.568+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.569863+0000) 2020-02-06T01:21:46.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:46.569+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.570340+0000) 2020-02-06T01:21:46.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:46.569+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.570874+0000) 2020-02-06T01:21:46.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:46.570+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.571577+0000) 2020-02-06T01:21:46.573 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:46.570+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.571834+0000) 2020-02-06T01:21:46.574 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:46.571+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.572717+0000) 2020-02-06T01:21:46.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:46.571+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.572494+0000) 2020-02-06T01:21:46.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:46.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.573503+0000) 2020-02-06T01:21:46.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:46.571+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.573387+0000) 2020-02-06T01:21:46.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:46.572+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.573700+0000) 2020-02-06T01:21:46.576 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:46.573+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.574411+0000) 2020-02-06T01:21:46.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:46.573+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.574747+0000) 2020-02-06T01:21:46.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:46.573+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.575329+0000) 2020-02-06T01:21:46.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:46.579+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:46.580646+0000) 2020-02-06T01:21:46.592 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:46.589+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201088 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:47.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:47.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:47.079788+0000) 2020-02-06T01:21:47.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:47.348+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:47.349276+0000) 2020-02-06T01:21:47.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:47.348+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:47.349276+0000) 2020-02-06T01:21:47.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:47.348+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:47.349373+0000) 2020-02-06T01:21:49.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:49.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:49.723682+0000) 2020-02-06T01:21:49.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:49.970+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:49.971307+0000) 2020-02-06T01:21:50.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:50.575+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:50.575972+0000) 2020-02-06T01:21:50.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:50.832+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:50.833377+0000) 2020-02-06T01:21:50.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:50.854+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:50.855068+0000) 2020-02-06T01:21:50.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:50.854+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:50.855262+0000) 2020-02-06T01:21:51.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:51.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.597782+0000) 2020-02-06T01:21:51.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:51.597+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.598943+0000) 2020-02-06T01:21:51.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:51.597+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.599402+0000) 2020-02-06T01:21:51.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:51.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.599802+0000) 2020-02-06T01:21:51.601 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:51.599+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.600169+0000) 2020-02-06T01:21:51.603 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:51.599+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.600809+0000) 2020-02-06T01:21:51.603 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:51.600+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.601308+0000) 2020-02-06T01:21:51.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:51.600+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.601715+0000) 2020-02-06T01:21:51.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:51.600+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.601855+0000) 2020-02-06T01:21:51.604 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:51.601+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.602355+0000) 2020-02-06T01:21:51.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:51.600+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.602277+0000) 2020-02-06T01:21:51.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:51.601+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.602616+0000) 2020-02-06T01:21:51.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:51.600+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.602409+0000) 2020-02-06T01:21:51.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:51.607+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:51.608712+0000) 2020-02-06T01:21:51.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:51.617+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:52.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:52.959+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:52.960510+0000) 2020-02-06T01:21:53.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:53.188+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:53.188546+0000) 2020-02-06T01:21:55.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:55.863+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:55.863408+0000) 2020-02-06T01:21:56.098 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:56.096+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.097216+0000) 2020-02-06T01:21:56.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:21:56.624+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.625110+0000) 2020-02-06T01:21:56.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:21:56.624+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.625530+0000) 2020-02-06T01:21:56.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:56.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.626960+0000) 2020-02-06T01:21:56.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:21:56.625+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.627266+0000) 2020-02-06T01:21:56.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:21:56.626+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.627580+0000) 2020-02-06T01:21:56.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:56.626+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.627896+0000) 2020-02-06T01:21:56.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:21:56.627+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.628734+0000) 2020-02-06T01:21:56.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:56.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.628820+0000) 2020-02-06T01:21:56.630 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:56.626+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.628361+0000) 2020-02-06T01:21:56.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:21:56.628+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.629948+0000) 2020-02-06T01:21:56.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:56.628+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.629685+0000) 2020-02-06T01:21:56.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:21:56.628+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.630388+0000) 2020-02-06T01:21:56.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:56.629+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.630345+0000) 2020-02-06T01:21:56.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:56.637+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:56.638299+0000) 2020-02-06T01:21:56.648 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:21:56.646+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201171 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:21:57.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:21:57.078+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:57.080031+0000) 2020-02-06T01:21:57.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:21:57.348+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:57.349527+0000) 2020-02-06T01:21:57.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:21:57.348+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:57.349527+0000) 2020-02-06T01:21:57.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:21:57.348+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:57.349640+0000) 2020-02-06T01:21:58.016 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:21:58.023 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:21:59.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:59.722+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:59.723966+0000) 2020-02-06T01:21:59.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:21:59.970+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:21:59.971534+0000) 2020-02-06T01:22:00.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:00.575+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:00.576221+0000) 2020-02-06T01:22:00.834 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:00.832+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:00.833641+0000) 2020-02-06T01:22:00.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:00.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:00.855338+0000) 2020-02-06T01:22:00.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:00.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:00.855609+0000) 2020-02-06T01:22:01.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:01.652+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.653220+0000) 2020-02-06T01:22:01.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:01.653+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.654317+0000) 2020-02-06T01:22:01.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:01.653+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.654847+0000) 2020-02-06T01:22:01.656 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:01.653+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.655187+0000) 2020-02-06T01:22:01.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:01.654+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.655527+0000) 2020-02-06T01:22:01.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:01.654+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.655842+0000) 2020-02-06T01:22:01.657 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:01.654+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.656127+0000) 2020-02-06T01:22:01.657 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:01.655+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.656582+0000) 2020-02-06T01:22:01.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:01.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.656944+0000) 2020-02-06T01:22:01.658 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:01.656+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.657541+0000) 2020-02-06T01:22:01.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:01.656+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.658217+0000) 2020-02-06T01:22:01.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:01.657+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.658381+0000) 2020-02-06T01:22:01.660 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:01.657+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.659411+0000) 2020-02-06T01:22:01.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:01.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:01.666138+0000) 2020-02-06T01:22:01.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:01.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201216 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:02.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:02.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:02.960758+0000) 2020-02-06T01:22:03.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:03.188+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:03.188779+0000) 2020-02-06T01:22:05.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:05.870+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:05.871994+0000) 2020-02-06T01:22:06.098 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:06.096+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.097496+0000) 2020-02-06T01:22:06.681 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:06.678+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.679681+0000) 2020-02-06T01:22:06.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:06.679+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.680523+0000) 2020-02-06T01:22:06.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:06.680+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.681060+0000) 2020-02-06T01:22:06.683 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:06.681+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.682470+0000) 2020-02-06T01:22:06.684 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:06.681+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.683024+0000) 2020-02-06T01:22:06.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:06.682+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.683455+0000) 2020-02-06T01:22:06.685 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:06.682+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.684054+0000) 2020-02-06T01:22:06.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:06.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.683938+0000) 2020-02-06T01:22:06.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:06.683+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.684778+0000) 2020-02-06T01:22:06.686 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:06.683+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.685058+0000) 2020-02-06T01:22:06.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:06.683+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.685172+0000) 2020-02-06T01:22:06.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:06.685+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.686270+0000) 2020-02-06T01:22:06.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:06.685+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.686608+0000) 2020-02-06T01:22:06.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:06.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:06.691239+0000) 2020-02-06T01:22:06.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:06.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:07.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:07.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:07.080267+0000) 2020-02-06T01:22:07.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:07.349+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:07.349780+0000) 2020-02-06T01:22:07.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:07.349+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:07.349782+0000) 2020-02-06T01:22:07.351 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:07.349+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:07.349878+0000) 2020-02-06T01:22:09.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:09.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:09.724196+0000) 2020-02-06T01:22:09.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:09.971+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:09.971760+0000) 2020-02-06T01:22:10.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:10.575+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:10.576487+0000) 2020-02-06T01:22:10.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:10.833+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:10.833863+0000) 2020-02-06T01:22:10.856 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:10.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:10.855614+0000) 2020-02-06T01:22:10.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:10.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:10.855830+0000) 2020-02-06T01:22:11.709 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:11.707+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.707959+0000) 2020-02-06T01:22:11.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:11.708+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.708757+0000) 2020-02-06T01:22:11.710 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:11.709+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.709289+0000) 2020-02-06T01:22:11.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:11.709+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.709896+0000) 2020-02-06T01:22:11.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:11.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.710593+0000) 2020-02-06T01:22:11.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:11.711+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.711515+0000) 2020-02-06T01:22:11.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:11.711+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.711859+0000) 2020-02-06T01:22:11.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:11.712+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.712766+0000) 2020-02-06T01:22:11.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:11.711+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.712029+0000) 2020-02-06T01:22:11.714 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:11.713+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.713524+0000) 2020-02-06T01:22:11.714 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:11.712+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.713368+0000) 2020-02-06T01:22:11.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:11.713+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.714055+0000) 2020-02-06T01:22:11.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:11.713+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.714165+0000) 2020-02-06T01:22:11.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:11.718+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:11.718099+0000) 2020-02-06T01:22:11.731 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:11.729+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:12.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:12.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:12.961043+0000) 2020-02-06T01:22:13.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:13.187+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:13.188999+0000) 2020-02-06T01:22:15.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:15.879+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:15.880150+0000) 2020-02-06T01:22:16.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:16.097+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.097700+0000) 2020-02-06T01:22:16.737 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:16.734+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.735907+0000) 2020-02-06T01:22:16.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:16.735+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.736333+0000) 2020-02-06T01:22:16.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:16.736+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.736536+0000) 2020-02-06T01:22:16.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:16.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.737554+0000) 2020-02-06T01:22:16.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:16.736+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.737554+0000) 2020-02-06T01:22:16.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:16.737+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.738825+0000) 2020-02-06T01:22:16.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:16.738+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.738520+0000) 2020-02-06T01:22:16.740 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:16.738+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.739225+0000) 2020-02-06T01:22:16.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:16.738+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.739820+0000) 2020-02-06T01:22:16.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:16.740+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.740468+0000) 2020-02-06T01:22:16.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:16.739+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.740706+0000) 2020-02-06T01:22:16.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:16.740+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.740827+0000) 2020-02-06T01:22:16.742 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:16.740+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.741249+0000) 2020-02-06T01:22:16.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:16.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:16.746487+0000) 2020-02-06T01:22:16.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:16.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:17.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:17.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:17.080480+0000) 2020-02-06T01:22:17.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:17.349+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:17.350046+0000) 2020-02-06T01:22:17.351 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:17.349+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:17.350047+0000) 2020-02-06T01:22:17.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:17.349+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:17.350117+0000) 2020-02-06T01:22:19.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:19.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:19.724433+0000) 2020-02-06T01:22:19.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:19.971+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:19.971983+0000) 2020-02-06T01:22:20.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:20.576+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:20.576726+0000) 2020-02-06T01:22:20.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:20.833+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:20.834127+0000) 2020-02-06T01:22:20.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:20.854+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:20.855844+0000) 2020-02-06T01:22:20.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:20.854+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:20.856019+0000) 2020-02-06T01:22:21.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:21.764+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.765419+0000) 2020-02-06T01:22:21.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:21.764+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.766033+0000) 2020-02-06T01:22:21.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:21.765+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.766310+0000) 2020-02-06T01:22:21.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:21.765+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.766811+0000) 2020-02-06T01:22:21.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:21.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.767089+0000) 2020-02-06T01:22:21.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:21.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.767870+0000) 2020-02-06T01:22:21.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:21.766+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.768043+0000) 2020-02-06T01:22:21.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:21.767+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.768580+0000) 2020-02-06T01:22:21.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:21.767+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.768774+0000) 2020-02-06T01:22:21.772 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:21.769+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.769419+0000) 2020-02-06T01:22:21.772 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:21.769+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.769643+0000) 2020-02-06T01:22:21.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:21.769+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.769870+0000) 2020-02-06T01:22:21.772 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:21.769+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.769956+0000) 2020-02-06T01:22:21.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:21.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:21.774689+0000) 2020-02-06T01:22:21.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:21.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201384 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:22.962 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:22.960+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:22.961300+0000) 2020-02-06T01:22:23.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:23.188+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:23.189303+0000) 2020-02-06T01:22:25.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:25.887+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:25.888191+0000) 2020-02-06T01:22:26.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:26.097+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.097963+0000) 2020-02-06T01:22:26.792 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:26.790+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.791524+0000) 2020-02-06T01:22:26.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:26.792+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.793317+0000) 2020-02-06T01:22:26.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:26.792+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.793936+0000) 2020-02-06T01:22:26.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:26.793+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.794126+0000) 2020-02-06T01:22:26.795 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:26.793+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.794778+0000) 2020-02-06T01:22:26.795 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:26.794+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.794510+0000) 2020-02-06T01:22:26.796 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:26.795+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.795578+0000) 2020-02-06T01:22:26.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:26.795+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.796329+0000) 2020-02-06T01:22:26.798 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:26.796+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.797059+0000) 2020-02-06T01:22:26.798 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:26.797+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.797719+0000) 2020-02-06T01:22:26.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:26.797+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.798029+0000) 2020-02-06T01:22:26.799 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:26.798+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.798962+0000) 2020-02-06T01:22:26.800 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:26.798+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.799317+0000) 2020-02-06T01:22:26.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:26.804+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:26.805782+0000) 2020-02-06T01:22:26.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:26.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201425 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:27.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:27.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:27.080714+0000) 2020-02-06T01:22:27.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:27.349+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:27.350327+0000) 2020-02-06T01:22:27.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:27.349+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:27.350341+0000) 2020-02-06T01:22:27.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:27.349+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:27.350327+0000) 2020-02-06T01:22:28.315 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:22:28.321 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:22:29.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:29.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:29.724714+0000) 2020-02-06T01:22:29.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:29.971+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:29.972188+0000) 2020-02-06T01:22:30.578 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:30.576+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:30.576984+0000) 2020-02-06T01:22:30.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:30.833+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:30.834354+0000) 2020-02-06T01:22:30.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:30.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:30.856106+0000) 2020-02-06T01:22:30.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:30.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:30.856256+0000) 2020-02-06T01:22:31.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:31.820+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.821551+0000) 2020-02-06T01:22:31.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:31.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.822037+0000) 2020-02-06T01:22:31.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:31.822+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.822953+0000) 2020-02-06T01:22:31.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:31.822+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.823528+0000) 2020-02-06T01:22:31.824 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:31.822+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.823893+0000) 2020-02-06T01:22:31.825 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:31.823+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.824419+0000) 2020-02-06T01:22:31.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:31.823+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.824694+0000) 2020-02-06T01:22:31.826 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:31.825+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.825476+0000) 2020-02-06T01:22:31.826 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:31.825+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.825542+0000) 2020-02-06T01:22:31.827 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:31.825+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.826274+0000) 2020-02-06T01:22:31.828 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:31.826+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.826877+0000) 2020-02-06T01:22:31.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:31.826+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.826776+0000) 2020-02-06T01:22:31.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:31.826+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.826991+0000) 2020-02-06T01:22:31.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:31.828+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:31.829211+0000) 2020-02-06T01:22:31.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:31.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201469 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:32.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:32.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:32.961654+0000) 2020-02-06T01:22:33.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:33.188+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:33.189534+0000) 2020-02-06T01:22:35.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:35.895+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:35.896579+0000) 2020-02-06T01:22:36.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:36.097+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.098207+0000) 2020-02-06T01:22:36.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:36.847+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.848650+0000) 2020-02-06T01:22:36.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:36.848+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.849319+0000) 2020-02-06T01:22:36.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:36.848+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.849777+0000) 2020-02-06T01:22:36.851 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:36.849+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.850325+0000) 2020-02-06T01:22:36.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:36.849+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.850578+0000) 2020-02-06T01:22:36.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:36.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.851116+0000) 2020-02-06T01:22:36.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:36.850+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.851309+0000) 2020-02-06T01:22:36.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:36.850+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.851622+0000) 2020-02-06T01:22:36.853 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:36.851+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.852158+0000) 2020-02-06T01:22:36.853 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:36.851+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.852369+0000) 2020-02-06T01:22:36.854 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:36.852+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.853473+0000) 2020-02-06T01:22:36.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:36.853+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.854339+0000) 2020-02-06T01:22:36.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:36.854+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.854444+0000) 2020-02-06T01:22:36.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:36.858+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:36.859705+0000) 2020-02-06T01:22:36.870 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:36.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:37.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:37.079+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:37.080875+0000) 2020-02-06T01:22:37.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:37.350+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:37.350545+0000) 2020-02-06T01:22:37.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:37.350+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:37.350558+0000) 2020-02-06T01:22:37.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:37.350+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:37.350584+0000) 2020-02-06T01:22:39.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:39.723+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:39.724967+0000) 2020-02-06T01:22:39.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:39.971+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:39.972416+0000) 2020-02-06T01:22:40.578 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:40.577+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:40.577236+0000) 2020-02-06T01:22:40.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:40.834+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:40.834587+0000) 2020-02-06T01:22:40.857 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:40.855+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:40.856372+0000) 2020-02-06T01:22:40.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:40.855+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:40.856486+0000) 2020-02-06T01:22:41.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:41.874+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.874843+0000) 2020-02-06T01:22:41.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:41.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.875709+0000) 2020-02-06T01:22:41.878 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:41.876+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.876970+0000) 2020-02-06T01:22:41.879 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:41.877+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.877695+0000) 2020-02-06T01:22:41.879 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:41.877+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.877790+0000) 2020-02-06T01:22:41.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:41.877+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.878444+0000) 2020-02-06T01:22:41.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:41.877+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.878054+0000) 2020-02-06T01:22:41.880 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:41.878+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.879287+0000) 2020-02-06T01:22:41.881 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:41.878+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.879846+0000) 2020-02-06T01:22:41.881 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:41.880+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.880624+0000) 2020-02-06T01:22:41.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:41.879+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.880827+0000) 2020-02-06T01:22:41.882 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:41.879+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.881020+0000) 2020-02-06T01:22:41.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:41.880+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.881250+0000) 2020-02-06T01:22:41.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:41.887+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:41.888633+0000) 2020-02-06T01:22:41.897 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:41.895+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:42.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:42.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:42.961902+0000) 2020-02-06T01:22:43.190 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:43.189+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:43.189769+0000) 2020-02-06T01:22:45.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:45.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:45.903178+0000) 2020-02-06T01:22:46.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:46.098+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.098500+0000) 2020-02-06T01:22:46.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:46.901+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.901788+0000) 2020-02-06T01:22:46.904 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:46.902+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.902954+0000) 2020-02-06T01:22:46.904 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:46.903+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.903575+0000) 2020-02-06T01:22:46.905 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:46.903+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.903903+0000) 2020-02-06T01:22:46.905 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:46.903+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.904177+0000) 2020-02-06T01:22:46.905 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:46.903+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.904895+0000) 2020-02-06T01:22:46.906 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:46.904+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.905174+0000) 2020-02-06T01:22:46.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:46.904+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.905357+0000) 2020-02-06T01:22:46.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:46.905+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.906196+0000) 2020-02-06T01:22:46.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:46.905+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.906485+0000) 2020-02-06T01:22:46.907 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:46.905+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.906744+0000) 2020-02-06T01:22:46.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:46.906+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.907399+0000) 2020-02-06T01:22:46.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:46.906+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.907938+0000) 2020-02-06T01:22:46.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:46.911+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:46.912895+0000) 2020-02-06T01:22:46.922 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:46.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201593 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:47.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:47.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:47.081123+0000) 2020-02-06T01:22:47.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:47.350+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:47.350798+0000) 2020-02-06T01:22:47.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:47.350+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:47.350800+0000) 2020-02-06T01:22:47.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:47.350+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:47.350864+0000) 2020-02-06T01:22:49.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:49.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:49.725202+0000) 2020-02-06T01:22:49.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:49.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:49.972646+0000) 2020-02-06T01:22:50.578 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:50.576+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:50.577503+0000) 2020-02-06T01:22:50.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:50.833+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:50.834838+0000) 2020-02-06T01:22:50.857 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:50.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:50.856692+0000) 2020-02-06T01:22:50.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:50.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:50.856626+0000) 2020-02-06T01:22:51.928 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:51.925+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.926850+0000) 2020-02-06T01:22:51.928 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:51.926+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.927381+0000) 2020-02-06T01:22:51.929 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:51.926+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.928363+0000) 2020-02-06T01:22:51.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:51.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.929010+0000) 2020-02-06T01:22:51.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:51.927+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.929271+0000) 2020-02-06T01:22:51.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:51.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.929635+0000) 2020-02-06T01:22:51.931 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:51.928+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.929984+0000) 2020-02-06T01:22:51.931 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:51.928+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.930318+0000) 2020-02-06T01:22:51.932 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:51.929+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.930581+0000) 2020-02-06T01:22:51.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:51.930+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.931148+0000) 2020-02-06T01:22:51.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:51.930+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.932182+0000) 2020-02-06T01:22:51.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:51.931+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.932273+0000) 2020-02-06T01:22:51.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:51.931+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.932900+0000) 2020-02-06T01:22:51.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:51.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:51.936795+0000) 2020-02-06T01:22:51.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:51.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201637 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:52.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:52.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:52.962162+0000) 2020-02-06T01:22:53.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:53.189+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:53.189998+0000) 2020-02-06T01:22:55.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:55.911+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:55.911618+0000) 2020-02-06T01:22:56.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:56.097+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.098762+0000) 2020-02-06T01:22:56.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:56.952+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.953173+0000) 2020-02-06T01:22:56.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:56.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.953546+0000) 2020-02-06T01:22:56.955 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:22:56.952+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.953893+0000) 2020-02-06T01:22:56.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:56.953+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.954275+0000) 2020-02-06T01:22:56.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:22:56.953+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.954742+0000) 2020-02-06T01:22:56.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:22:56.954+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.955563+0000) 2020-02-06T01:22:56.957 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:22:56.954+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.956053+0000) 2020-02-06T01:22:56.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:22:56.956+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.957344+0000) 2020-02-06T01:22:56.958 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:22:56.955+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.956902+0000) 2020-02-06T01:22:56.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:56.955+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.957009+0000) 2020-02-06T01:22:56.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:22:56.956+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.957661+0000) 2020-02-06T01:22:56.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:56.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.958999+0000) 2020-02-06T01:22:56.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:56.958+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.959584+0000) 2020-02-06T01:22:56.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:56.963+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:56.964238+0000) 2020-02-06T01:22:56.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:22:56.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:22:57.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:22:57.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:57.081371+0000) 2020-02-06T01:22:57.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:22:57.350+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:57.351041+0000) 2020-02-06T01:22:57.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:22:57.350+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:57.351034+0000) 2020-02-06T01:22:57.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:22:57.350+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:57.351064+0000) 2020-02-06T01:22:58.459 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:22:58.465 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:22:59.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:59.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:59.725439+0000) 2020-02-06T01:22:59.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:22:59.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:22:59.972874+0000) 2020-02-06T01:23:00.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:00.576+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:00.577776+0000) 2020-02-06T01:23:00.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:00.834+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:00.835061+0000) 2020-02-06T01:23:00.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:00.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:00.856914+0000) 2020-02-06T01:23:00.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:00.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:00.856914+0000) 2020-02-06T01:23:01.982 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:01.979+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.981131+0000) 2020-02-06T01:23:01.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:01.981+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.981963+0000) 2020-02-06T01:23:01.983 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:01.980+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.982342+0000) 2020-02-06T01:23:01.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:01.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.983067+0000) 2020-02-06T01:23:01.985 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:01.982+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.983991+0000) 2020-02-06T01:23:01.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:01.984+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.984390+0000) 2020-02-06T01:23:01.985 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:01.982+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.983947+0000) 2020-02-06T01:23:01.986 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:01.984+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.984913+0000) 2020-02-06T01:23:01.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:01.985+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.985126+0000) 2020-02-06T01:23:01.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:01.984+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.985598+0000) 2020-02-06T01:23:01.987 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:01.984+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.986055+0000) 2020-02-06T01:23:01.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:01.986+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.986509+0000) 2020-02-06T01:23:01.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:01.986+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.986707+0000) 2020-02-06T01:23:01.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:01.992+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:01.993038+0000) 2020-02-06T01:23:02.002 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:02.000+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201721 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:02.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:02.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:02.962412+0000) 2020-02-06T01:23:03.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:03.190+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:03.190304+0000) 2020-02-06T01:23:05.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:05.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:05.923414+0000) 2020-02-06T01:23:06.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:06.098+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:06.099052+0000) 2020-02-06T01:23:07.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:07.006+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.007476+0000) 2020-02-06T01:23:07.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:07.007+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.008745+0000) 2020-02-06T01:23:07.010 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:07.008+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.009134+0000) 2020-02-06T01:23:07.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:07.007+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.009400+0000) 2020-02-06T01:23:07.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:07.008+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.009942+0000) 2020-02-06T01:23:07.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:07.009+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.010554+0000) 2020-02-06T01:23:07.011 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:07.009+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.010706+0000) 2020-02-06T01:23:07.012 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:07.010+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.011601+0000) 2020-02-06T01:23:07.013 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:07.011+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.012458+0000) 2020-02-06T01:23:07.013 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:07.010+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.012031+0000) 2020-02-06T01:23:07.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:07.012+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.013135+0000) 2020-02-06T01:23:07.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:07.011+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.012989+0000) 2020-02-06T01:23:07.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:07.012+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.013467+0000) 2020-02-06T01:23:07.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:07.017+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.018136+0000) 2020-02-06T01:23:07.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:07.027+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:07.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:07.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.081590+0000) 2020-02-06T01:23:07.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:07.350+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.351292+0000) 2020-02-06T01:23:07.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:07.350+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.351253+0000) 2020-02-06T01:23:07.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:07.350+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:07.351287+0000) 2020-02-06T01:23:09.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:09.724+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:09.725696+0000) 2020-02-06T01:23:09.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:09.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:09.973135+0000) 2020-02-06T01:23:10.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:10.576+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:10.578099+0000) 2020-02-06T01:23:10.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:10.834+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:10.835338+0000) 2020-02-06T01:23:10.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:10.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:10.857185+0000) 2020-02-06T01:23:10.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:10.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:10.857190+0000) 2020-02-06T01:23:12.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:12.033+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.034466+0000) 2020-02-06T01:23:12.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:12.033+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.034781+0000) 2020-02-06T01:23:12.037 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:12.034+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.036049+0000) 2020-02-06T01:23:12.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:12.035+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.036911+0000) 2020-02-06T01:23:12.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:12.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.037352+0000) 2020-02-06T01:23:12.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:12.036+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.037804+0000) 2020-02-06T01:23:12.039 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:12.036+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.038038+0000) 2020-02-06T01:23:12.039 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:12.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.038900+0000) 2020-02-06T01:23:12.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:12.038+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.039324+0000) 2020-02-06T01:23:12.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:12.038+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.039431+0000) 2020-02-06T01:23:12.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:12.038+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.039777+0000) 2020-02-06T01:23:12.041 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:12.038+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.039629+0000) 2020-02-06T01:23:12.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:12.038+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.039625+0000) 2020-02-06T01:23:12.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:12.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.043568+0000) 2020-02-06T01:23:12.058 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:12.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:12.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:12.961+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:12.962730+0000) 2020-02-06T01:23:13.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:13.189+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:13.190547+0000) 2020-02-06T01:23:15.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:15.931+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:15.932120+0000) 2020-02-06T01:23:16.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:16.098+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:16.099330+0000) 2020-02-06T01:23:17.063 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:17.061+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.062774+0000) 2020-02-06T01:23:17.064 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:17.061+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.062597+0000) 2020-02-06T01:23:17.065 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:17.063+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.064268+0000) 2020-02-06T01:23:17.065 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:17.063+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.064955+0000) 2020-02-06T01:23:17.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:17.064+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.065468+0000) 2020-02-06T01:23:17.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:17.064+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.065775+0000) 2020-02-06T01:23:17.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:17.064+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.066247+0000) 2020-02-06T01:23:17.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:17.066+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.067625+0000) 2020-02-06T01:23:17.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:17.066+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.068203+0000) 2020-02-06T01:23:17.070 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:17.066+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.067515+0000) 2020-02-06T01:23:17.070 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:17.067+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.068609+0000) 2020-02-06T01:23:17.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:17.067+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.068840+0000) 2020-02-06T01:23:17.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:17.068+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.069432+0000) 2020-02-06T01:23:17.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:17.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.071126+0000) 2020-02-06T01:23:17.082 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:17.080+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.081812+0000) 2020-02-06T01:23:17.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:17.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201841 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:17.352 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:17.350+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.351538+0000) 2020-02-06T01:23:17.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:17.350+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.351510+0000) 2020-02-06T01:23:17.353 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:17.350+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:17.351536+0000) 2020-02-06T01:23:19.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:19.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:19.725937+0000) 2020-02-06T01:23:19.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:19.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:19.973371+0000) 2020-02-06T01:23:20.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:20.577+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:20.578349+0000) 2020-02-06T01:23:20.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:20.834+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:20.835612+0000) 2020-02-06T01:23:20.858 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:20.856+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:20.857457+0000) 2020-02-06T01:23:20.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:20.856+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:20.857457+0000) 2020-02-06T01:23:22.091 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:22.087+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.089367+0000) 2020-02-06T01:23:22.092 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:22.090+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.091767+0000) 2020-02-06T01:23:22.093 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:22.091+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.092729+0000) 2020-02-06T01:23:22.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:22.091+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.093253+0000) 2020-02-06T01:23:22.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:22.092+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.094005+0000) 2020-02-06T01:23:22.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:22.092+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.094314+0000) 2020-02-06T01:23:22.095 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:22.093+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.094869+0000) 2020-02-06T01:23:22.096 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:22.093+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.094714+0000) 2020-02-06T01:23:22.096 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:22.094+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.095651+0000) 2020-02-06T01:23:22.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:22.094+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.096151+0000) 2020-02-06T01:23:22.097 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:22.095+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.096409+0000) 2020-02-06T01:23:22.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:22.095+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.096630+0000) 2020-02-06T01:23:22.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:22.095+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.096979+0000) 2020-02-06T01:23:22.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:22.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.103142+0000) 2020-02-06T01:23:22.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:22.112+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:22.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:22.962+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:22.962962+0000) 2020-02-06T01:23:23.191 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:23.189+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:23.190770+0000) 2020-02-06T01:23:25.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:25.939+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:25.940140+0000) 2020-02-06T01:23:26.100 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:26.098+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:26.099584+0000) 2020-02-06T01:23:27.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:27.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.082007+0000) 2020-02-06T01:23:27.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:27.118+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.119550+0000) 2020-02-06T01:23:27.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:27.119+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.120302+0000) 2020-02-06T01:23:27.121 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:27.119+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.120418+0000) 2020-02-06T01:23:27.122 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:27.119+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.121078+0000) 2020-02-06T01:23:27.122 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:27.120+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.122031+0000) 2020-02-06T01:23:27.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:27.121+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.122815+0000) 2020-02-06T01:23:27.124 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:27.121+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.123037+0000) 2020-02-06T01:23:27.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:27.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.123419+0000) 2020-02-06T01:23:27.125 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:27.122+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.124274+0000) 2020-02-06T01:23:27.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:27.123+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.124770+0000) 2020-02-06T01:23:27.126 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:27.124+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.125594+0000) 2020-02-06T01:23:27.127 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:27.124+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.125588+0000) 2020-02-06T01:23:27.127 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:27.124+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.125731+0000) 2020-02-06T01:23:27.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:27.129+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.131024+0000) 2020-02-06T01:23:27.141 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:27.139+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201923 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:27.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:27.351+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.351797+0000) 2020-02-06T01:23:27.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:27.351+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.351746+0000) 2020-02-06T01:23:27.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:27.351+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:27.351797+0000) 2020-02-06T01:23:28.596 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:23:28.603 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:23:29.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:29.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:29.726175+0000) 2020-02-06T01:23:29.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:29.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:29.973604+0000) 2020-02-06T01:23:30.579 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:30.577+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:30.578565+0000) 2020-02-06T01:23:30.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:30.834+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:30.835856+0000) 2020-02-06T01:23:30.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:30.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:30.857729+0000) 2020-02-06T01:23:30.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:30.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:30.857729+0000) 2020-02-06T01:23:32.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:32.144+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.145652+0000) 2020-02-06T01:23:32.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:32.145+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.146308+0000) 2020-02-06T01:23:32.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:32.145+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.146845+0000) 2020-02-06T01:23:32.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:32.146+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.147832+0000) 2020-02-06T01:23:32.149 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:32.146+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.147642+0000) 2020-02-06T01:23:32.150 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:32.147+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.149191+0000) 2020-02-06T01:23:32.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:32.148+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.149995+0000) 2020-02-06T01:23:32.151 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:32.149+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.150944+0000) 2020-02-06T01:23:32.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:32.149+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.150991+0000) 2020-02-06T01:23:32.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:32.150+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.151479+0000) 2020-02-06T01:23:32.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:32.151+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.152158+0000) 2020-02-06T01:23:32.153 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:32.151+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.152331+0000) 2020-02-06T01:23:32.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:32.151+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.152606+0000) 2020-02-06T01:23:32.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:32.157+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.158130+0000) 2020-02-06T01:23:32.169 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:32.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 201964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:32.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:32.962+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:32.963201+0000) 2020-02-06T01:23:33.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:33.189+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:33.190997+0000) 2020-02-06T01:23:35.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:35.947+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:35.948510+0000) 2020-02-06T01:23:36.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:36.099+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:36.099852+0000) 2020-02-06T01:23:37.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:37.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.082302+0000) 2020-02-06T01:23:37.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:37.172+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.173957+0000) 2020-02-06T01:23:37.176 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:37.173+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.175186+0000) 2020-02-06T01:23:37.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:37.174+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.175510+0000) 2020-02-06T01:23:37.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:37.174+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.176018+0000) 2020-02-06T01:23:37.177 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:37.175+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.176493+0000) 2020-02-06T01:23:37.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:37.175+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.176564+0000) 2020-02-06T01:23:37.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:37.175+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.176792+0000) 2020-02-06T01:23:37.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:37.176+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.177583+0000) 2020-02-06T01:23:37.179 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:37.176+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.177792+0000) 2020-02-06T01:23:37.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:37.177+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.178453+0000) 2020-02-06T01:23:37.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:37.177+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.178879+0000) 2020-02-06T01:23:37.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:37.177+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.178516+0000) 2020-02-06T01:23:37.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:37.178+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.179678+0000) 2020-02-06T01:23:37.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:37.182+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.183155+0000) 2020-02-06T01:23:37.196 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:37.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:37.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:37.351+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.352063+0000) 2020-02-06T01:23:37.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:37.351+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.352047+0000) 2020-02-06T01:23:37.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:37.351+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:37.352067+0000) 2020-02-06T01:23:39.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:39.725+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:39.726416+0000) 2020-02-06T01:23:39.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:39.972+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:39.973833+0000) 2020-02-06T01:23:40.580 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:40.578+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:40.578820+0000) 2020-02-06T01:23:40.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:40.835+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:40.836117+0000) 2020-02-06T01:23:40.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:40.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:40.857998+0000) 2020-02-06T01:23:40.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:40.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:40.857998+0000) 2020-02-06T01:23:42.201 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:42.199+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.200313+0000) 2020-02-06T01:23:42.202 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:42.201+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.201751+0000) 2020-02-06T01:23:42.203 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:42.200+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.202050+0000) 2020-02-06T01:23:42.203 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:42.202+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.202837+0000) 2020-02-06T01:23:42.204 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:42.202+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.203716+0000) 2020-02-06T01:23:42.205 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:42.203+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.204955+0000) 2020-02-06T01:23:42.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:42.204+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.205521+0000) 2020-02-06T01:23:42.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:42.205+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.205832+0000) 2020-02-06T01:23:42.207 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:42.205+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.206102+0000) 2020-02-06T01:23:42.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:42.206+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.206604+0000) 2020-02-06T01:23:42.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:42.206+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.206776+0000) 2020-02-06T01:23:42.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:42.206+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.207364+0000) 2020-02-06T01:23:42.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:42.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.209167+0000) 2020-02-06T01:23:42.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:42.208+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.209411+0000) 2020-02-06T01:23:42.225 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:42.224+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:42.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:42.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:42.963473+0000) 2020-02-06T01:23:43.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:43.190+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:43.191292+0000) 2020-02-06T01:23:45.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:45.956+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:45.956359+0000) 2020-02-06T01:23:46.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:46.099+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:46.100092+0000) 2020-02-06T01:23:47.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:47.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.082534+0000) 2020-02-06T01:23:47.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:47.229+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.230238+0000) 2020-02-06T01:23:47.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:47.231+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.231921+0000) 2020-02-06T01:23:47.233 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:47.231+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.232242+0000) 2020-02-06T01:23:47.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:47.232+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.232579+0000) 2020-02-06T01:23:47.234 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:47.232+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.233128+0000) 2020-02-06T01:23:47.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:47.232+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.233293+0000) 2020-02-06T01:23:47.235 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:47.232+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.233442+0000) 2020-02-06T01:23:47.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:47.233+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.234257+0000) 2020-02-06T01:23:47.235 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:47.233+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.234383+0000) 2020-02-06T01:23:47.236 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:47.233+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.233701+0000) 2020-02-06T01:23:47.236 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:47.233+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.234592+0000) 2020-02-06T01:23:47.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:47.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.235227+0000) 2020-02-06T01:23:47.237 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:47.234+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.235400+0000) 2020-02-06T01:23:47.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:47.239+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.240782+0000) 2020-02-06T01:23:47.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:47.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202085 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:47.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:47.351+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.352307+0000) 2020-02-06T01:23:47.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:47.351+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.352276+0000) 2020-02-06T01:23:47.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:47.351+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:47.352307+0000) 2020-02-06T01:23:49.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:49.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:49.726653+0000) 2020-02-06T01:23:49.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:49.973+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:49.974067+0000) 2020-02-06T01:23:50.580 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:50.578+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:50.579060+0000) 2020-02-06T01:23:50.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:50.836+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:50.836354+0000) 2020-02-06T01:23:50.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:50.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:50.858329+0000) 2020-02-06T01:23:50.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:50.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:50.858329+0000) 2020-02-06T01:23:52.258 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:52.256+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.257343+0000) 2020-02-06T01:23:52.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:52.257+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.258137+0000) 2020-02-06T01:23:52.259 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:52.258+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.258607+0000) 2020-02-06T01:23:52.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:52.257+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.258706+0000) 2020-02-06T01:23:52.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:52.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.259403+0000) 2020-02-06T01:23:52.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:52.258+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.259472+0000) 2020-02-06T01:23:52.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:52.259+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.260070+0000) 2020-02-06T01:23:52.262 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:52.259+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.260261+0000) 2020-02-06T01:23:52.262 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:52.259+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.260694+0000) 2020-02-06T01:23:52.262 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:52.260+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.261597+0000) 2020-02-06T01:23:52.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:52.261+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.262197+0000) 2020-02-06T01:23:52.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:52.262+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.262545+0000) 2020-02-06T01:23:52.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:52.262+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.263106+0000) 2020-02-06T01:23:52.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:52.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.266517+0000) 2020-02-06T01:23:52.280 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:52.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202126 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:52.964 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:52.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:52.963746+0000) 2020-02-06T01:23:53.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:53.191+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:53.191498+0000) 2020-02-06T01:23:55.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:55.964+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:55.964135+0000) 2020-02-06T01:23:56.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:56.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:56.100355+0000) 2020-02-06T01:23:57.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:57.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.082804+0000) 2020-02-06T01:23:57.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:57.284+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.284105+0000) 2020-02-06T01:23:57.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:57.285+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.285805+0000) 2020-02-06T01:23:57.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:23:57.286+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.286391+0000) 2020-02-06T01:23:57.288 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:23:57.286+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.287026+0000) 2020-02-06T01:23:57.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:23:57.287+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.287559+0000) 2020-02-06T01:23:57.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:57.288+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.288312+0000) 2020-02-06T01:23:57.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:23:57.288+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.289399+0000) 2020-02-06T01:23:57.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:23:57.289+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.289801+0000) 2020-02-06T01:23:57.291 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:57.289+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.289999+0000) 2020-02-06T01:23:57.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:23:57.290+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.290305+0000) 2020-02-06T01:23:57.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:23:57.291+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.291298+0000) 2020-02-06T01:23:57.292 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:23:57.291+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.291574+0000) 2020-02-06T01:23:57.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:57.291+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.292038+0000) 2020-02-06T01:23:57.298 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:57.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.297231+0000) 2020-02-06T01:23:57.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:23:57.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202166 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:23:57.353 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:23:57.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.352528+0000) 2020-02-06T01:23:57.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:23:57.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.352502+0000) 2020-02-06T01:23:57.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:23:57.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:57.352527+0000) 2020-02-06T01:23:58.729 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:23:58.736 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:23:59.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:59.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:59.726915+0000) 2020-02-06T01:23:59.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:23:59.974+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:23:59.974316+0000) 2020-02-06T01:24:00.580 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:00.579+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:00.579365+0000) 2020-02-06T01:24:00.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:00.835+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:00.836588+0000) 2020-02-06T01:24:00.859 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:00.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:00.858610+0000) 2020-02-06T01:24:00.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:00.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:00.858605+0000) 2020-02-06T01:24:02.314 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:02.312+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.312992+0000) 2020-02-06T01:24:02.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:02.312+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.313448+0000) 2020-02-06T01:24:02.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:02.312+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.314050+0000) 2020-02-06T01:24:02.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:02.314+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.314642+0000) 2020-02-06T01:24:02.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:02.314+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.315498+0000) 2020-02-06T01:24:02.316 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:02.315+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.315781+0000) 2020-02-06T01:24:02.317 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:02.315+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.316321+0000) 2020-02-06T01:24:02.318 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:02.315+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.316980+0000) 2020-02-06T01:24:02.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:02.316+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.317328+0000) 2020-02-06T01:24:02.319 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:02.316+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.317630+0000) 2020-02-06T01:24:02.319 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:02.316+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.317992+0000) 2020-02-06T01:24:02.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:02.317+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.318308+0000) 2020-02-06T01:24:02.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:02.317+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.318500+0000) 2020-02-06T01:24:02.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:02.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.324453+0000) 2020-02-06T01:24:02.334 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:02.333+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202207 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:02.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:02.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:02.963976+0000) 2020-02-06T01:24:03.192 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:03.190+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:03.191707+0000) 2020-02-06T01:24:05.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:05.971+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:05.972615+0000) 2020-02-06T01:24:06.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:06.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:06.100591+0000) 2020-02-06T01:24:07.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:07.081+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.083000+0000) 2020-02-06T01:24:07.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:07.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.340396+0000) 2020-02-06T01:24:07.342 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:07.339+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.339881+0000) 2020-02-06T01:24:07.342 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:07.339+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.340873+0000) 2020-02-06T01:24:07.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:07.340+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.341303+0000) 2020-02-06T01:24:07.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:07.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.341820+0000) 2020-02-06T01:24:07.343 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:07.341+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.342226+0000) 2020-02-06T01:24:07.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:07.342+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.342597+0000) 2020-02-06T01:24:07.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:07.342+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.343122+0000) 2020-02-06T01:24:07.344 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:07.342+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.343373+0000) 2020-02-06T01:24:07.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:07.342+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.343538+0000) 2020-02-06T01:24:07.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:07.343+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.344129+0000) 2020-02-06T01:24:07.345 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:07.342+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.344007+0000) 2020-02-06T01:24:07.345 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:07.343+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.344229+0000) 2020-02-06T01:24:07.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:07.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.350402+0000) 2020-02-06T01:24:07.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:07.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.352693+0000) 2020-02-06T01:24:07.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:07.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.352744+0000) 2020-02-06T01:24:07.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:07.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:07.352748+0000) 2020-02-06T01:24:07.363 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:07.361+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202247 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:09.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:09.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:09.727180+0000) 2020-02-06T01:24:09.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:09.973+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:09.974544+0000) 2020-02-06T01:24:10.580 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:10.579+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:10.579635+0000) 2020-02-06T01:24:10.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:10.835+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:10.836819+0000) 2020-02-06T01:24:10.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:10.857+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:10.858864+0000) 2020-02-06T01:24:10.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:10.857+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:10.858864+0000) 2020-02-06T01:24:12.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:12.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.369171+0000) 2020-02-06T01:24:12.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:12.368+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.369602+0000) 2020-02-06T01:24:12.371 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:12.368+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.369350+0000) 2020-02-06T01:24:12.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:12.369+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.370389+0000) 2020-02-06T01:24:12.371 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:12.369+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.370859+0000) 2020-02-06T01:24:12.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:12.370+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.371078+0000) 2020-02-06T01:24:12.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:12.371+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.372317+0000) 2020-02-06T01:24:12.373 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:12.371+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.372507+0000) 2020-02-06T01:24:12.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:12.371+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.372970+0000) 2020-02-06T01:24:12.374 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:12.372+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.373273+0000) 2020-02-06T01:24:12.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:12.373+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.373759+0000) 2020-02-06T01:24:12.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:12.373+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.373646+0000) 2020-02-06T01:24:12.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:12.373+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.373671+0000) 2020-02-06T01:24:12.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:12.376+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.377798+0000) 2020-02-06T01:24:12.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:12.388+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202288 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:12.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:12.964+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:12.964194+0000) 2020-02-06T01:24:13.193 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:13.190+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:13.191944+0000) 2020-02-06T01:24:15.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:15.979+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:15.980549+0000) 2020-02-06T01:24:16.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:16.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:16.100862+0000) 2020-02-06T01:24:17.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:17.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.083301+0000) 2020-02-06T01:24:17.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:17.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.352912+0000) 2020-02-06T01:24:17.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:17.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.352956+0000) 2020-02-06T01:24:17.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:17.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.352955+0000) 2020-02-06T01:24:17.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:17.393+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.394765+0000) 2020-02-06T01:24:17.396 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:17.394+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.395553+0000) 2020-02-06T01:24:17.397 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:17.395+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.396243+0000) 2020-02-06T01:24:17.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:17.396+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.396673+0000) 2020-02-06T01:24:17.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:17.396+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.397206+0000) 2020-02-06T01:24:17.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:17.396+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.397379+0000) 2020-02-06T01:24:17.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:17.396+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.397794+0000) 2020-02-06T01:24:17.399 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:17.397+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.398088+0000) 2020-02-06T01:24:17.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:17.398+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.399199+0000) 2020-02-06T01:24:17.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:17.399+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.399452+0000) 2020-02-06T01:24:17.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:17.399+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.400747+0000) 2020-02-06T01:24:17.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:17.399+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.400397+0000) 2020-02-06T01:24:17.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:17.400+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.400653+0000) 2020-02-06T01:24:17.406 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:17.404+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:17.405639+0000) 2020-02-06T01:24:17.416 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:17.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202328 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:19.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:19.726+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:19.727419+0000) 2020-02-06T01:24:19.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:19.974+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:19.974803+0000) 2020-02-06T01:24:20.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:20.578+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:20.579888+0000) 2020-02-06T01:24:20.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:20.836+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:20.837052+0000) 2020-02-06T01:24:20.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:20.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:20.859129+0000) 2020-02-06T01:24:20.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:20.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:20.859129+0000) 2020-02-06T01:24:22.421 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:22.418+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.419788+0000) 2020-02-06T01:24:22.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:22.419+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.420553+0000) 2020-02-06T01:24:22.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:22.419+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.421238+0000) 2020-02-06T01:24:22.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:22.420+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.421653+0000) 2020-02-06T01:24:22.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:22.421+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.422916+0000) 2020-02-06T01:24:22.424 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:22.422+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.423397+0000) 2020-02-06T01:24:22.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:22.422+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.423670+0000) 2020-02-06T01:24:22.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:22.423+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.424483+0000) 2020-02-06T01:24:22.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:22.424+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.425149+0000) 2020-02-06T01:24:22.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:22.424+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.426300+0000) 2020-02-06T01:24:22.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:22.425+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.426351+0000) 2020-02-06T01:24:22.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:22.425+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.426447+0000) 2020-02-06T01:24:22.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:22.425+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.426737+0000) 2020-02-06T01:24:22.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:22.429+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.430723+0000) 2020-02-06T01:24:22.446 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:22.443+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:22.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:22.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:22.964458+0000) 2020-02-06T01:24:23.193 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:23.191+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:23.192149+0000) 2020-02-06T01:24:25.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:25.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:25.988642+0000) 2020-02-06T01:24:26.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:26.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:26.101120+0000) 2020-02-06T01:24:27.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:27.082+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.083527+0000) 2020-02-06T01:24:27.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:27.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.353198+0000) 2020-02-06T01:24:27.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:27.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.353161+0000) 2020-02-06T01:24:27.354 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:27.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.353201+0000) 2020-02-06T01:24:27.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:27.448+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.449532+0000) 2020-02-06T01:24:27.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:27.449+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.450852+0000) 2020-02-06T01:24:27.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:27.450+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.451605+0000) 2020-02-06T01:24:27.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:27.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.452298+0000) 2020-02-06T01:24:27.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:27.451+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.452667+0000) 2020-02-06T01:24:27.454 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:27.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.453386+0000) 2020-02-06T01:24:27.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:27.451+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.453197+0000) 2020-02-06T01:24:27.455 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:27.453+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.454378+0000) 2020-02-06T01:24:27.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:27.453+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.454867+0000) 2020-02-06T01:24:27.456 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:27.454+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.455926+0000) 2020-02-06T01:24:27.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:27.454+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.456206+0000) 2020-02-06T01:24:27.457 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:27.455+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.456798+0000) 2020-02-06T01:24:27.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:27.456+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.457120+0000) 2020-02-06T01:24:27.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:27.460+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:27.461521+0000) 2020-02-06T01:24:27.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:27.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:28.860 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:24:28.867 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:24:29.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:29.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:29.727651+0000) 2020-02-06T01:24:29.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:29.974+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:29.975038+0000) 2020-02-06T01:24:30.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:30.578+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:30.580126+0000) 2020-02-06T01:24:30.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:30.836+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:30.837313+0000) 2020-02-06T01:24:30.860 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:30.858+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:30.859412+0000) 2020-02-06T01:24:30.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:30.858+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:30.859407+0000) 2020-02-06T01:24:32.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:32.476+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.477839+0000) 2020-02-06T01:24:32.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:32.477+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.478677+0000) 2020-02-06T01:24:32.480 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:32.478+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.479182+0000) 2020-02-06T01:24:32.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:32.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.479487+0000) 2020-02-06T01:24:32.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:32.478+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.479681+0000) 2020-02-06T01:24:32.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:32.479+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.480453+0000) 2020-02-06T01:24:32.481 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:32.479+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.480861+0000) 2020-02-06T01:24:32.482 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:32.480+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.481144+0000) 2020-02-06T01:24:32.482 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:32.480+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.481409+0000) 2020-02-06T01:24:32.483 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:32.480+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.481839+0000) 2020-02-06T01:24:32.483 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:32.481+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.482733+0000) 2020-02-06T01:24:32.484 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:32.481+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.482957+0000) 2020-02-06T01:24:32.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:32.481+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.483365+0000) 2020-02-06T01:24:32.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:32.484+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.485975+0000) 2020-02-06T01:24:32.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:32.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:32.965 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:32.963+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:32.964722+0000) 2020-02-06T01:24:33.193 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:33.191+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:33.192363+0000) 2020-02-06T01:24:35.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:35.998+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:35.998516+0000) 2020-02-06T01:24:36.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:36.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:36.101388+0000) 2020-02-06T01:24:37.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:37.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.083753+0000) 2020-02-06T01:24:37.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:37.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.353481+0000) 2020-02-06T01:24:37.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:37.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.353484+0000) 2020-02-06T01:24:37.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:37.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.353482+0000) 2020-02-06T01:24:37.505 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:37.503+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.504514+0000) 2020-02-06T01:24:37.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:37.504+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.505372+0000) 2020-02-06T01:24:37.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:37.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.506042+0000) 2020-02-06T01:24:37.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:37.505+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.506412+0000) 2020-02-06T01:24:37.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:37.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.507821+0000) 2020-02-06T01:24:37.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:37.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.508452+0000) 2020-02-06T01:24:37.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:37.507+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.508770+0000) 2020-02-06T01:24:37.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:37.507+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.509098+0000) 2020-02-06T01:24:37.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:37.508+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.509489+0000) 2020-02-06T01:24:37.511 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:37.508+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.509940+0000) 2020-02-06T01:24:37.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:37.509+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.510808+0000) 2020-02-06T01:24:37.512 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:37.510+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.511449+0000) 2020-02-06T01:24:37.512 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:37.510+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.511776+0000) 2020-02-06T01:24:37.518 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:37.516+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:37.517217+0000) 2020-02-06T01:24:37.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:37.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202490 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:39.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:39.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:39.727896+0000) 2020-02-06T01:24:39.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:39.974+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:39.975254+0000) 2020-02-06T01:24:40.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:40.579+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:40.580459+0000) 2020-02-06T01:24:40.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:40.837+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:40.837562+0000) 2020-02-06T01:24:40.860 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:40.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:40.859678+0000) 2020-02-06T01:24:40.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:40.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:40.859680+0000) 2020-02-06T01:24:42.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:42.532+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.533866+0000) 2020-02-06T01:24:42.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:42.533+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.534488+0000) 2020-02-06T01:24:42.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:42.534+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.535318+0000) 2020-02-06T01:24:42.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:42.534+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.535973+0000) 2020-02-06T01:24:42.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:42.535+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.537046+0000) 2020-02-06T01:24:42.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:42.536+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.537454+0000) 2020-02-06T01:24:42.538 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:42.536+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.538029+0000) 2020-02-06T01:24:42.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:42.537+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.538451+0000) 2020-02-06T01:24:42.540 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:42.538+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.539405+0000) 2020-02-06T01:24:42.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:42.537+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.539117+0000) 2020-02-06T01:24:42.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:42.538+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.540074+0000) 2020-02-06T01:24:42.541 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:42.538+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.540068+0000) 2020-02-06T01:24:42.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:42.539+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.540644+0000) 2020-02-06T01:24:42.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:42.544+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.545784+0000) 2020-02-06T01:24:42.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:42.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:42.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:42.964+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:42.964955+0000) 2020-02-06T01:24:43.193 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:43.192+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:43.192576+0000) 2020-02-06T01:24:46.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:46.007+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:46.007300+0000) 2020-02-06T01:24:46.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:46.100+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:46.101684+0000) 2020-02-06T01:24:47.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:47.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.084066+0000) 2020-02-06T01:24:47.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:47.352+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.353746+0000) 2020-02-06T01:24:47.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:47.352+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.353768+0000) 2020-02-06T01:24:47.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:47.352+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.353773+0000) 2020-02-06T01:24:47.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:47.561+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.562819+0000) 2020-02-06T01:24:47.564 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:47.562+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.563304+0000) 2020-02-06T01:24:47.564 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:47.562+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.563490+0000) 2020-02-06T01:24:47.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:47.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.563810+0000) 2020-02-06T01:24:47.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:47.562+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.564056+0000) 2020-02-06T01:24:47.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:47.563+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.564391+0000) 2020-02-06T01:24:47.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:47.563+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.564852+0000) 2020-02-06T01:24:47.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:47.564+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.565189+0000) 2020-02-06T01:24:47.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:47.564+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.565532+0000) 2020-02-06T01:24:47.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:47.563+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.565389+0000) 2020-02-06T01:24:47.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:47.564+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.565995+0000) 2020-02-06T01:24:47.567 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:47.565+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.566605+0000) 2020-02-06T01:24:47.568 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:47.566+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.568053+0000) 2020-02-06T01:24:47.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:47.571+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:47.572383+0000) 2020-02-06T01:24:47.585 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:47.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:49.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:49.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:49.728160+0000) 2020-02-06T01:24:49.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:49.975+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:49.975528+0000) 2020-02-06T01:24:50.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:50.580+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:50.580735+0000) 2020-02-06T01:24:50.838 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:50.837+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:50.837793+0000) 2020-02-06T01:24:50.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:50.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:50.859952+0000) 2020-02-06T01:24:50.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:50.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:50.859954+0000) 2020-02-06T01:24:52.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:52.589+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.590746+0000) 2020-02-06T01:24:52.592 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:52.591+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.591204+0000) 2020-02-06T01:24:52.592 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:52.591+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.591630+0000) 2020-02-06T01:24:52.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:52.592+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.592757+0000) 2020-02-06T01:24:52.594 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:52.592+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.593035+0000) 2020-02-06T01:24:52.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:52.593+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.593605+0000) 2020-02-06T01:24:52.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:52.592+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.594162+0000) 2020-02-06T01:24:52.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:52.594+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.594676+0000) 2020-02-06T01:24:52.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:52.595+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.595215+0000) 2020-02-06T01:24:52.596 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:52.594+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.595565+0000) 2020-02-06T01:24:52.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:52.594+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.595866+0000) 2020-02-06T01:24:52.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:52.594+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.596067+0000) 2020-02-06T01:24:52.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:52.595+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.596709+0000) 2020-02-06T01:24:52.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:52.602+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.602354+0000) 2020-02-06T01:24:52.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:52.611+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:52.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:52.964+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:52.965182+0000) 2020-02-06T01:24:53.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:53.192+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:53.192781+0000) 2020-02-06T01:24:56.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:56.014+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:56.015666+0000) 2020-02-06T01:24:56.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:56.101+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:56.101932+0000) 2020-02-06T01:24:57.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:57.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.084359+0000) 2020-02-06T01:24:57.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:57.353+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.354056+0000) 2020-02-06T01:24:57.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:57.353+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.354036+0000) 2020-02-06T01:24:57.355 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:57.353+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.354056+0000) 2020-02-06T01:24:57.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:24:57.616+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.617960+0000) 2020-02-06T01:24:57.620 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:24:57.618+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.619185+0000) 2020-02-06T01:24:57.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:24:57.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.619880+0000) 2020-02-06T01:24:57.621 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:24:57.618+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.619681+0000) 2020-02-06T01:24:57.621 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:24:57.619+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.620481+0000) 2020-02-06T01:24:57.621 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:24:57.619+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.620826+0000) 2020-02-06T01:24:57.622 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:24:57.619+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.621358+0000) 2020-02-06T01:24:57.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:57.619+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.621002+0000) 2020-02-06T01:24:57.623 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:24:57.620+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.621630+0000) 2020-02-06T01:24:57.623 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:24:57.620+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.621714+0000) 2020-02-06T01:24:57.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:24:57.620+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.622019+0000) 2020-02-06T01:24:57.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:24:57.621+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.622814+0000) 2020-02-06T01:24:57.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:57.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.626167+0000) 2020-02-06T01:24:57.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:57.625+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:57.626454+0000) 2020-02-06T01:24:57.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:24:57.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:24:59.008 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:24:59.014 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:24:59.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:59.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:59.728390+0000) 2020-02-06T01:24:59.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:24:59.974+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:24:59.975758+0000) 2020-02-06T01:25:00.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:00.580+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:00.580975+0000) 2020-02-06T01:25:00.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:00.837+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:00.838044+0000) 2020-02-06T01:25:00.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:00.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:00.860216+0000) 2020-02-06T01:25:00.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:00.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:00.860219+0000) 2020-02-06T01:25:02.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:02.644+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.646295+0000) 2020-02-06T01:25:02.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:02.645+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.647022+0000) 2020-02-06T01:25:02.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:02.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.647482+0000) 2020-02-06T01:25:02.649 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:02.646+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.647843+0000) 2020-02-06T01:25:02.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:02.646+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.648003+0000) 2020-02-06T01:25:02.649 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:02.647+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.648422+0000) 2020-02-06T01:25:02.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:02.647+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.648989+0000) 2020-02-06T01:25:02.650 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:02.648+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.649516+0000) 2020-02-06T01:25:02.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:02.648+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.649759+0000) 2020-02-06T01:25:02.651 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:02.648+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.650096+0000) 2020-02-06T01:25:02.651 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:02.649+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.650778+0000) 2020-02-06T01:25:02.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:02.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.651105+0000) 2020-02-06T01:25:02.652 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:02.650+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.651453+0000) 2020-02-06T01:25:02.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:02.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.654132+0000) 2020-02-06T01:25:02.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:02.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:02.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:02.964+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:02.965460+0000) 2020-02-06T01:25:03.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:03.192+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:03.192993+0000) 2020-02-06T01:25:06.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:06.023+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:06.024252+0000) 2020-02-06T01:25:06.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:06.101+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:06.102180+0000) 2020-02-06T01:25:07.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:07.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.084595+0000) 2020-02-06T01:25:07.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:07.353+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.354332+0000) 2020-02-06T01:25:07.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:07.353+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.354349+0000) 2020-02-06T01:25:07.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:07.353+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.354349+0000) 2020-02-06T01:25:07.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:07.670+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.671291+0000) 2020-02-06T01:25:07.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:07.670+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.671769+0000) 2020-02-06T01:25:07.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:07.670+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.672035+0000) 2020-02-06T01:25:07.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:07.672+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.673073+0000) 2020-02-06T01:25:07.674 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:07.672+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.673466+0000) 2020-02-06T01:25:07.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:07.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.673759+0000) 2020-02-06T01:25:07.675 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:07.672+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.674151+0000) 2020-02-06T01:25:07.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:07.673+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.674858+0000) 2020-02-06T01:25:07.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:07.675+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.676438+0000) 2020-02-06T01:25:07.677 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:07.675+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.676622+0000) 2020-02-06T01:25:07.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:07.675+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.676474+0000) 2020-02-06T01:25:07.678 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:07.675+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.677091+0000) 2020-02-06T01:25:07.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:07.675+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.677350+0000) 2020-02-06T01:25:07.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:07.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:07.682106+0000) 2020-02-06T01:25:07.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:07.691+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:09.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:09.727+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:09.728643+0000) 2020-02-06T01:25:09.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:09.975+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:09.976010+0000) 2020-02-06T01:25:10.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:10.580+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:10.581230+0000) 2020-02-06T01:25:10.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:10.837+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:10.838312+0000) 2020-02-06T01:25:10.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:10.859+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:10.860434+0000) 2020-02-06T01:25:10.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:10.859+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:10.860433+0000) 2020-02-06T01:25:12.699 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:12.697+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.698788+0000) 2020-02-06T01:25:12.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:12.698+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.699654+0000) 2020-02-06T01:25:12.701 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:12.699+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.700200+0000) 2020-02-06T01:25:12.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:12.700+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.701442+0000) 2020-02-06T01:25:12.703 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:12.700+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.701344+0000) 2020-02-06T01:25:12.703 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:12.701+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.701765+0000) 2020-02-06T01:25:12.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:12.700+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.701998+0000) 2020-02-06T01:25:12.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:12.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.702716+0000) 2020-02-06T01:25:12.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:12.701+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.702316+0000) 2020-02-06T01:25:12.705 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:12.702+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.703263+0000) 2020-02-06T01:25:12.705 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:12.703+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.703798+0000) 2020-02-06T01:25:12.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:12.703+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.703936+0000) 2020-02-06T01:25:12.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:12.702+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.703404+0000) 2020-02-06T01:25:12.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:12.711+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.712145+0000) 2020-02-06T01:25:12.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:12.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:12.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:12.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:12.965716+0000) 2020-02-06T01:25:13.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:13.192+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:13.193215+0000) 2020-02-06T01:25:16.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:16.032+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:16.032845+0000) 2020-02-06T01:25:16.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:16.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:16.102482+0000) 2020-02-06T01:25:17.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:17.083+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.084823+0000) 2020-02-06T01:25:17.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:17.354+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.354686+0000) 2020-02-06T01:25:17.356 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:17.354+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.354682+0000) 2020-02-06T01:25:17.356 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:17.354+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.354900+0000) 2020-02-06T01:25:17.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:17.723+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.724321+0000) 2020-02-06T01:25:17.726 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:17.723+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.724303+0000) 2020-02-06T01:25:17.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:17.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.725400+0000) 2020-02-06T01:25:17.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:17.724+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.725782+0000) 2020-02-06T01:25:17.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:17.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.726261+0000) 2020-02-06T01:25:17.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:17.725+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.726614+0000) 2020-02-06T01:25:17.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:17.726+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.727307+0000) 2020-02-06T01:25:17.728 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:17.726+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.727774+0000) 2020-02-06T01:25:17.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:17.728+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.728712+0000) 2020-02-06T01:25:17.731 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:17.729+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.729597+0000) 2020-02-06T01:25:17.731 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:17.729+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.730039+0000) 2020-02-06T01:25:17.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:17.730+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.730713+0000) 2020-02-06T01:25:17.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:17.729+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.730347+0000) 2020-02-06T01:25:17.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:17.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:17.735268+0000) 2020-02-06T01:25:17.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:17.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:19.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:19.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:19.728893+0000) 2020-02-06T01:25:19.977 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:19.975+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:19.976273+0000) 2020-02-06T01:25:20.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:20.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:20.581486+0000) 2020-02-06T01:25:20.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:20.837+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:20.838568+0000) 2020-02-06T01:25:20.861 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:20.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:20.860720+0000) 2020-02-06T01:25:20.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:20.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:20.860720+0000) 2020-02-06T01:25:22.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:22.751+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.752504+0000) 2020-02-06T01:25:22.754 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:22.752+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.753263+0000) 2020-02-06T01:25:22.754 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:22.752+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.753749+0000) 2020-02-06T01:25:22.756 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:22.754+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.754888+0000) 2020-02-06T01:25:22.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:22.754+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.755268+0000) 2020-02-06T01:25:22.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:22.754+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.755758+0000) 2020-02-06T01:25:22.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:22.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.756317+0000) 2020-02-06T01:25:22.757 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:22.756+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.756668+0000) 2020-02-06T01:25:22.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:22.755+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.756980+0000) 2020-02-06T01:25:22.758 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:22.756+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.757438+0000) 2020-02-06T01:25:22.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:22.757+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.757769+0000) 2020-02-06T01:25:22.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:22.756+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.758045+0000) 2020-02-06T01:25:22.760 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:22.757+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.758347+0000) 2020-02-06T01:25:22.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:22.762+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.763773+0000) 2020-02-06T01:25:22.774 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:22.772+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202854 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:22.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:22.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:22.965953+0000) 2020-02-06T01:25:23.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:23.192+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:23.193436+0000) 2020-02-06T01:25:26.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:26.040+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:26.041190+0000) 2020-02-06T01:25:26.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:26.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:26.102735+0000) 2020-02-06T01:25:27.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:27.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.085061+0000) 2020-02-06T01:25:27.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:27.354+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.354988+0000) 2020-02-06T01:25:27.356 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:27.354+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.354988+0000) 2020-02-06T01:25:27.356 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:27.354+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.355185+0000) 2020-02-06T01:25:27.780 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:27.778+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.779302+0000) 2020-02-06T01:25:27.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:27.779+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.780028+0000) 2020-02-06T01:25:27.781 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:27.780+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.780797+0000) 2020-02-06T01:25:27.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:27.779+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.781052+0000) 2020-02-06T01:25:27.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:27.780+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.781529+0000) 2020-02-06T01:25:27.783 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:27.780+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.781942+0000) 2020-02-06T01:25:27.783 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:27.781+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.782243+0000) 2020-02-06T01:25:27.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:27.782+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.782756+0000) 2020-02-06T01:25:27.784 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:27.782+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.783268+0000) 2020-02-06T01:25:27.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:27.782+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.783575+0000) 2020-02-06T01:25:27.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:27.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.784187+0000) 2020-02-06T01:25:27.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:27.784+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.784854+0000) 2020-02-06T01:25:27.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:27.784+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.785231+0000) 2020-02-06T01:25:27.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:27.789+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:27.790244+0000) 2020-02-06T01:25:27.801 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:27.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202893 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:29.147 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:25:29.153 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:25:29.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:29.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:29.729147+0000) 2020-02-06T01:25:29.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:29.976+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:29.976506+0000) 2020-02-06T01:25:30.582 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:30.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:30.581736+0000) 2020-02-06T01:25:30.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:30.838+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:30.838745+0000) 2020-02-06T01:25:30.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:30.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:30.860973+0000) 2020-02-06T01:25:30.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:30.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:30.860973+0000) 2020-02-06T01:25:32.806 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:32.804+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.805099+0000) 2020-02-06T01:25:32.807 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:32.805+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.806040+0000) 2020-02-06T01:25:32.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:32.806+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.807250+0000) 2020-02-06T01:25:32.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:32.807+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.808410+0000) 2020-02-06T01:25:32.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:32.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.808925+0000) 2020-02-06T01:25:32.810 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:32.808+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.809237+0000) 2020-02-06T01:25:32.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:32.809+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.809643+0000) 2020-02-06T01:25:32.811 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:32.808+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.809729+0000) 2020-02-06T01:25:32.812 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:32.809+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.810303+0000) 2020-02-06T01:25:32.813 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:32.810+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.810985+0000) 2020-02-06T01:25:32.813 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:32.811+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.811677+0000) 2020-02-06T01:25:32.814 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:32.811+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.811893+0000) 2020-02-06T01:25:32.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:32.810+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.811263+0000) 2020-02-06T01:25:32.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:32.815+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.816271+0000) 2020-02-06T01:25:32.828 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:32.827+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202934 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:32.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:32.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:32.966187+0000) 2020-02-06T01:25:33.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:33.193+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:33.193664+0000) 2020-02-06T01:25:36.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:36.049+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:36.049725+0000) 2020-02-06T01:25:36.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:36.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:36.102990+0000) 2020-02-06T01:25:37.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:37.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.085334+0000) 2020-02-06T01:25:37.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:37.355+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.355272+0000) 2020-02-06T01:25:37.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:37.355+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.355272+0000) 2020-02-06T01:25:37.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:37.355+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.355494+0000) 2020-02-06T01:25:37.834 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:37.832+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.833120+0000) 2020-02-06T01:25:37.835 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:37.833+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.834703+0000) 2020-02-06T01:25:37.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:37.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.835318+0000) 2020-02-06T01:25:37.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:37.834+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.835461+0000) 2020-02-06T01:25:37.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:37.835+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.836300+0000) 2020-02-06T01:25:37.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:37.835+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.836493+0000) 2020-02-06T01:25:37.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:37.835+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.836880+0000) 2020-02-06T01:25:37.838 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:37.837+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.837512+0000) 2020-02-06T01:25:37.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:37.836+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.837946+0000) 2020-02-06T01:25:37.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:37.837+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.837856+0000) 2020-02-06T01:25:37.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:37.837+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.838884+0000) 2020-02-06T01:25:37.840 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:37.839+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.839487+0000) 2020-02-06T01:25:37.840 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:37.839+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.839622+0000) 2020-02-06T01:25:37.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:37.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:37.843826+0000) 2020-02-06T01:25:37.856 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:37.854+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 202974 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:39.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:39.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:39.729394+0000) 2020-02-06T01:25:39.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:39.976+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:39.976756+0000) 2020-02-06T01:25:40.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:40.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:40.581993+0000) 2020-02-06T01:25:40.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:40.838+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:40.838987+0000) 2020-02-06T01:25:40.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:40.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:40.861308+0000) 2020-02-06T01:25:40.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:40.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:40.861307+0000) 2020-02-06T01:25:42.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:42.859+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.860986+0000) 2020-02-06T01:25:42.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:42.860+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.861493+0000) 2020-02-06T01:25:42.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:42.861+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.862346+0000) 2020-02-06T01:25:42.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:42.861+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.862191+0000) 2020-02-06T01:25:42.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:42.861+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.862636+0000) 2020-02-06T01:25:42.864 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:42.862+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.863226+0000) 2020-02-06T01:25:42.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:42.862+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.863909+0000) 2020-02-06T01:25:42.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:42.863+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.864505+0000) 2020-02-06T01:25:42.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:42.864+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.864930+0000) 2020-02-06T01:25:42.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:42.865+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.865830+0000) 2020-02-06T01:25:42.867 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:42.865+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.865754+0000) 2020-02-06T01:25:42.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:42.866+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.866484+0000) 2020-02-06T01:25:42.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:42.866+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.866576+0000) 2020-02-06T01:25:42.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:42.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.871127+0000) 2020-02-06T01:25:42.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:42.880+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203016 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:42.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:42.966+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:42.966408+0000) 2020-02-06T01:25:43.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:43.193+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:43.193922+0000) 2020-02-06T01:25:46.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:46.057+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:46.057944+0000) 2020-02-06T01:25:46.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:46.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:46.103228+0000) 2020-02-06T01:25:47.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:47.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.085583+0000) 2020-02-06T01:25:47.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:47.355+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.355573+0000) 2020-02-06T01:25:47.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:47.355+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.355573+0000) 2020-02-06T01:25:47.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:47.355+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.355767+0000) 2020-02-06T01:25:47.887 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:47.885+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.886111+0000) 2020-02-06T01:25:47.887 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:47.886+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.887043+0000) 2020-02-06T01:25:47.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:47.888+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.888635+0000) 2020-02-06T01:25:47.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:47.890+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.890353+0000) 2020-02-06T01:25:47.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:47.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.890857+0000) 2020-02-06T01:25:47.892 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:47.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.891147+0000) 2020-02-06T01:25:47.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:47.891+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.891694+0000) 2020-02-06T01:25:47.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:47.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.892122+0000) 2020-02-06T01:25:47.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:47.891+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.892366+0000) 2020-02-06T01:25:47.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:47.892+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.892637+0000) 2020-02-06T01:25:47.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:47.892+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.892939+0000) 2020-02-06T01:25:47.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:47.893+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.893633+0000) 2020-02-06T01:25:47.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:47.894+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.894507+0000) 2020-02-06T01:25:47.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:47.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:47.898606+0000) 2020-02-06T01:25:47.909 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:47.908+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203054 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:49.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:49.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:49.729619+0000) 2020-02-06T01:25:49.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:49.976+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:49.976985+0000) 2020-02-06T01:25:50.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:50.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:50.582238+0000) 2020-02-06T01:25:50.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:50.838+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:50.839222+0000) 2020-02-06T01:25:50.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:50.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:50.861577+0000) 2020-02-06T01:25:50.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:50.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:50.861577+0000) 2020-02-06T01:25:52.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:52.912+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.913543+0000) 2020-02-06T01:25:52.918 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:52.913+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.914464+0000) 2020-02-06T01:25:52.918 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:52.913+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.914832+0000) 2020-02-06T01:25:52.919 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:52.915+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.916488+0000) 2020-02-06T01:25:52.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:52.915+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.916978+0000) 2020-02-06T01:25:52.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:52.916+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.917528+0000) 2020-02-06T01:25:52.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:52.916+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.917754+0000) 2020-02-06T01:25:52.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:52.916+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.918247+0000) 2020-02-06T01:25:52.920 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:52.916+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.918311+0000) 2020-02-06T01:25:52.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:52.915+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.917051+0000) 2020-02-06T01:25:52.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:52.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.917684+0000) 2020-02-06T01:25:52.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:52.916+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.918035+0000) 2020-02-06T01:25:52.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:52.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.922443+0000) 2020-02-06T01:25:52.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:52.922+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.923450+0000) 2020-02-06T01:25:52.936 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:52.934+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:52.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:52.965+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:52.966733+0000) 2020-02-06T01:25:53.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:53.193+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:53.194142+0000) 2020-02-06T01:25:56.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:56.066+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:56.067288+0000) 2020-02-06T01:25:56.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:56.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:56.103498+0000) 2020-02-06T01:25:57.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:57.084+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.085815+0000) 2020-02-06T01:25:57.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:57.354+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.355873+0000) 2020-02-06T01:25:57.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:57.354+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.356080+0000) 2020-02-06T01:25:57.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:57.354+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.355876+0000) 2020-02-06T01:25:57.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:25:57.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.941009+0000) 2020-02-06T01:25:57.942 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:25:57.940+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.941700+0000) 2020-02-06T01:25:57.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:25:57.940+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.942054+0000) 2020-02-06T01:25:57.943 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:25:57.941+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.942547+0000) 2020-02-06T01:25:57.944 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:25:57.942+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.943876+0000) 2020-02-06T01:25:57.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:57.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.944477+0000) 2020-02-06T01:25:57.946 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:25:57.943+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.945035+0000) 2020-02-06T01:25:57.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:57.944+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.945324+0000) 2020-02-06T01:25:57.947 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:25:57.944+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.945664+0000) 2020-02-06T01:25:57.947 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:25:57.944+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.945790+0000) 2020-02-06T01:25:57.947 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:25:57.944+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.946228+0000) 2020-02-06T01:25:57.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:25:57.944+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.946253+0000) 2020-02-06T01:25:57.948 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:25:57.945+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.947302+0000) 2020-02-06T01:25:57.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:57.950+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:57.951341+0000) 2020-02-06T01:25:57.964 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:25:57.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203134 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:25:59.285 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:25:59.291 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:25:59.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:59.728+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:59.729861+0000) 2020-02-06T01:25:59.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:25:59.976+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:25:59.977243+0000) 2020-02-06T01:26:00.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:00.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:00.582477+0000) 2020-02-06T01:26:00.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:00.838+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:00.839433+0000) 2020-02-06T01:26:00.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:00.860+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:00.861858+0000) 2020-02-06T01:26:00.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:00.860+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:00.861858+0000) 2020-02-06T01:26:02.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:02.969+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.970892+0000) 2020-02-06T01:26:02.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:02.970+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.971535+0000) 2020-02-06T01:26:02.973 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:02.970+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.972007+0000) 2020-02-06T01:26:02.973 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:02.971+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.972339+0000) 2020-02-06T01:26:02.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:02.971+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.972763+0000) 2020-02-06T01:26:02.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:02.972+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.973182+0000) 2020-02-06T01:26:02.974 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:02.972+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.973360+0000) 2020-02-06T01:26:02.975 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:02.972+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.973822+0000) 2020-02-06T01:26:02.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:02.972+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.974187+0000) 2020-02-06T01:26:02.976 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:02.973+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.974389+0000) 2020-02-06T01:26:02.976 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:02.974+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.975805+0000) 2020-02-06T01:26:02.977 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:02.974+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.975981+0000) 2020-02-06T01:26:02.977 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:02.974+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.976073+0000) 2020-02-06T01:26:02.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:02.979+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:02.981038+0000) 2020-02-06T01:26:02.993 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:02.991+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:03.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:03.193+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:03.194356+0000) 2020-02-06T01:26:06.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:06.074+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:06.075817+0000) 2020-02-06T01:26:06.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:06.102+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:06.103682+0000) 2020-02-06T01:26:07.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:07.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.086048+0000) 2020-02-06T01:26:07.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:07.355+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.356171+0000) 2020-02-06T01:26:07.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:07.355+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.356171+0000) 2020-02-06T01:26:07.358 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:07.355+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.356363+0000) 2020-02-06T01:26:08.000 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:07.997+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.998953+0000) 2020-02-06T01:26:08.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:07.998+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.999510+0000) 2020-02-06T01:26:08.001 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:07.998+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:07.999756+0000) 2020-02-06T01:26:08.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:07.998+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.000098+0000) 2020-02-06T01:26:08.001 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:07.999+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.000470+0000) 2020-02-06T01:26:08.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:08.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.001109+0000) 2020-02-06T01:26:08.002 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:07.999+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.001369+0000) 2020-02-06T01:26:08.002 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:08.000+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.001617+0000) 2020-02-06T01:26:08.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:08.000+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.002302+0000) 2020-02-06T01:26:08.003 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:08.000+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.002118+0000) 2020-02-06T01:26:08.004 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:08.001+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.003367+0000) 2020-02-06T01:26:08.005 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:08.003+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.004533+0000) 2020-02-06T01:26:08.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:08.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.012506+0000) 2020-02-06T01:26:08.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:08.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:08.014008+0000) 2020-02-06T01:26:08.022 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:08.020+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:09.718 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:09.716+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:09.717513+0000) 2020-02-06T01:26:09.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:09.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:09.730133+0000) 2020-02-06T01:26:09.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:09.976+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:09.977467+0000) 2020-02-06T01:26:10.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:10.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:10.582805+0000) 2020-02-06T01:26:10.840 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:10.838+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:10.839668+0000) 2020-02-06T01:26:10.863 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:10.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:10.862134+0000) 2020-02-06T01:26:10.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:10.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:10.862134+0000) 2020-02-06T01:26:13.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:13.025+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.026941+0000) 2020-02-06T01:26:13.029 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:13.026+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.027924+0000) 2020-02-06T01:26:13.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:13.028+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.029208+0000) 2020-02-06T01:26:13.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:13.028+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.029478+0000) 2020-02-06T01:26:13.032 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:13.029+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.030880+0000) 2020-02-06T01:26:13.032 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:13.029+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.031160+0000) 2020-02-06T01:26:13.033 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:13.030+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.031763+0000) 2020-02-06T01:26:13.033 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:13.030+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.031857+0000) 2020-02-06T01:26:13.033 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:13.029+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.030707+0000) 2020-02-06T01:26:13.034 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:13.030+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.031874+0000) 2020-02-06T01:26:13.034 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:13.031+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.032785+0000) 2020-02-06T01:26:13.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:13.032+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.033474+0000) 2020-02-06T01:26:13.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:13.034+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.035947+0000) 2020-02-06T01:26:13.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:13.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.037018+0000) 2020-02-06T01:26:13.050 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:13.047+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203255 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:13.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:13.193+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:13.194596+0000) 2020-02-06T01:26:16.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:16.084+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:16.085162+0000) 2020-02-06T01:26:16.104 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:16.103+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:16.103923+0000) 2020-02-06T01:26:17.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:17.085+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:17.086364+0000) 2020-02-06T01:26:17.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:17.355+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:17.356480+0000) 2020-02-06T01:26:17.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:17.355+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:17.356488+0000) 2020-02-06T01:26:17.358 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:17.355+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:17.356677+0000) 2020-02-06T01:26:18.056 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:18.054+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.055543+0000) 2020-02-06T01:26:18.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:18.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.056500+0000) 2020-02-06T01:26:18.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:18.055+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.057087+0000) 2020-02-06T01:26:18.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:18.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.057878+0000) 2020-02-06T01:26:18.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:18.057+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.058125+0000) 2020-02-06T01:26:18.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:18.057+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.058654+0000) 2020-02-06T01:26:18.060 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:18.057+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.058818+0000) 2020-02-06T01:26:18.060 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:18.057+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.059046+0000) 2020-02-06T01:26:18.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:18.057+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.059346+0000) 2020-02-06T01:26:18.061 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:18.058+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.059998+0000) 2020-02-06T01:26:18.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:18.059+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.060585+0000) 2020-02-06T01:26:18.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:18.058+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.060299+0000) 2020-02-06T01:26:18.062 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:18.059+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.061224+0000) 2020-02-06T01:26:18.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:18.065+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:18.066429+0000) 2020-02-06T01:26:18.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:18.074+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:19.718 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:19.717+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:19.717748+0000) 2020-02-06T01:26:19.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:19.729+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:19.730362+0000) 2020-02-06T01:26:19.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:19.977+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:19.977691+0000) 2020-02-06T01:26:20.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:20.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:20.583030+0000) 2020-02-06T01:26:20.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:20.839+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:20.839901+0000) 2020-02-06T01:26:20.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:20.861+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:20.862413+0000) 2020-02-06T01:26:20.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:20.861+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:20.862413+0000) 2020-02-06T01:26:23.082 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:23.080+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.081662+0000) 2020-02-06T01:26:23.083 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:23.080+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.082190+0000) 2020-02-06T01:26:23.083 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:23.081+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.082686+0000) 2020-02-06T01:26:23.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:23.082+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.083528+0000) 2020-02-06T01:26:23.084 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:23.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.084041+0000) 2020-02-06T01:26:23.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:23.083+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.084392+0000) 2020-02-06T01:26:23.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:23.083+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.084628+0000) 2020-02-06T01:26:23.086 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:23.083+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.085309+0000) 2020-02-06T01:26:23.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:23.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.085643+0000) 2020-02-06T01:26:23.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:23.084+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.086146+0000) 2020-02-06T01:26:23.087 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:23.085+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.086324+0000) 2020-02-06T01:26:23.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:23.085+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.086973+0000) 2020-02-06T01:26:23.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:23.085+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.087333+0000) 2020-02-06T01:26:23.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:23.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.093431+0000) 2020-02-06T01:26:23.104 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:23.102+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:23.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:23.194+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:23.194816+0000) 2020-02-06T01:26:26.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:26.093+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:26.094225+0000) 2020-02-06T01:26:26.105 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:26.103+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:26.104707+0000) 2020-02-06T01:26:27.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:27.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:27.086595+0000) 2020-02-06T01:26:27.358 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:27.356+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:27.356774+0000) 2020-02-06T01:26:27.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:27.356+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:27.356774+0000) 2020-02-06T01:26:27.358 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:27.356+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:27.356941+0000) 2020-02-06T01:26:28.110 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:28.108+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.109723+0000) 2020-02-06T01:26:28.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:28.109+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.110281+0000) 2020-02-06T01:26:28.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:28.109+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.110555+0000) 2020-02-06T01:26:28.112 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:28.109+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.110799+0000) 2020-02-06T01:26:28.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:28.110+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.111449+0000) 2020-02-06T01:26:28.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:28.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.112320+0000) 2020-02-06T01:26:28.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:28.111+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.112758+0000) 2020-02-06T01:26:28.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:28.112+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.113924+0000) 2020-02-06T01:26:28.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:28.113+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.115017+0000) 2020-02-06T01:26:28.116 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:28.113+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.114837+0000) 2020-02-06T01:26:28.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:28.113+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.115180+0000) 2020-02-06T01:26:28.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:28.114+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.115794+0000) 2020-02-06T01:26:28.117 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:28.114+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.115732+0000) 2020-02-06T01:26:28.120 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:28.118+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:28.119151+0000) 2020-02-06T01:26:28.132 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:28.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:29.456 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:26:29.462 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:26:29.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:29.717+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:29.718040+0000) 2020-02-06T01:26:29.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:29.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:29.730602+0000) 2020-02-06T01:26:29.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:29.977+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:29.977915+0000) 2020-02-06T01:26:30.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:30.581+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:30.583281+0000) 2020-02-06T01:26:30.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:30.839+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:30.840132+0000) 2020-02-06T01:26:30.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:30.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:30.862682+0000) 2020-02-06T01:26:30.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:30.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:30.862679+0000) 2020-02-06T01:26:33.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:33.134+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.135538+0000) 2020-02-06T01:26:33.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:33.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.136115+0000) 2020-02-06T01:26:33.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:33.135+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.136554+0000) 2020-02-06T01:26:33.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:33.136+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.137661+0000) 2020-02-06T01:26:33.140 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:33.137+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.139324+0000) 2020-02-06T01:26:33.141 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:33.138+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.139954+0000) 2020-02-06T01:26:33.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:33.139+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.140064+0000) 2020-02-06T01:26:33.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:33.140+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.141538+0000) 2020-02-06T01:26:33.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:33.140+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.141719+0000) 2020-02-06T01:26:33.143 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:33.141+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.142808+0000) 2020-02-06T01:26:33.143 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:33.141+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.142953+0000) 2020-02-06T01:26:33.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:33.141+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.143192+0000) 2020-02-06T01:26:33.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:33.142+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.143565+0000) 2020-02-06T01:26:33.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:33.145+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.146809+0000) 2020-02-06T01:26:33.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:33.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:33.196 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:33.194+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:33.195004+0000) 2020-02-06T01:26:36.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:36.102+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:36.103136+0000) 2020-02-06T01:26:36.105 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:36.104+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:36.104947+0000) 2020-02-06T01:26:37.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:37.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:37.086828+0000) 2020-02-06T01:26:37.358 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:37.356+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:37.357075+0000) 2020-02-06T01:26:37.358 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:37.356+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:37.357075+0000) 2020-02-06T01:26:37.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:37.356+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:37.357219+0000) 2020-02-06T01:26:38.165 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:38.163+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.164475+0000) 2020-02-06T01:26:38.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:38.164+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.165157+0000) 2020-02-06T01:26:38.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:38.164+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.165671+0000) 2020-02-06T01:26:38.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:38.165+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.166514+0000) 2020-02-06T01:26:38.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:38.165+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.167173+0000) 2020-02-06T01:26:38.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:38.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.167678+0000) 2020-02-06T01:26:38.169 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:38.166+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.168135+0000) 2020-02-06T01:26:38.169 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:38.166+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.168259+0000) 2020-02-06T01:26:38.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:38.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.168658+0000) 2020-02-06T01:26:38.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:38.168+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.169140+0000) 2020-02-06T01:26:38.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:38.167+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.168959+0000) 2020-02-06T01:26:38.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:38.168+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.170080+0000) 2020-02-06T01:26:38.171 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:38.169+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.171138+0000) 2020-02-06T01:26:38.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:38.174+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:38.175450+0000) 2020-02-06T01:26:38.188 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:38.186+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203464 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:39.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:39.717+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:39.718300+0000) 2020-02-06T01:26:39.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:39.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:39.730855+0000) 2020-02-06T01:26:39.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:39.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:39.978152+0000) 2020-02-06T01:26:40.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:40.583+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:40.583537+0000) 2020-02-06T01:26:40.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:40.840+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:40.840324+0000) 2020-02-06T01:26:40.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:40.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:40.862962+0000) 2020-02-06T01:26:40.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:40.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:40.862962+0000) 2020-02-06T01:26:43.195 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:43.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.194263+0000) 2020-02-06T01:26:43.195 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:43.194+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.194814+0000) 2020-02-06T01:26:43.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:43.194+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.195060+0000) 2020-02-06T01:26:43.196 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:43.195+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.195579+0000) 2020-02-06T01:26:43.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:43.195+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.195522+0000) 2020-02-06T01:26:43.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:43.195+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.195939+0000) 2020-02-06T01:26:43.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:43.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.195974+0000) 2020-02-06T01:26:43.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:43.196+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.196537+0000) 2020-02-06T01:26:43.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:43.196+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.196595+0000) 2020-02-06T01:26:43.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:43.197+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.197606+0000) 2020-02-06T01:26:43.199 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:43.197+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.197549+0000) 2020-02-06T01:26:43.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:43.198+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.198135+0000) 2020-02-06T01:26:43.200 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:43.199+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.199952+0000) 2020-02-06T01:26:43.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:43.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:43.204904+0000) 2020-02-06T01:26:43.215 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:43.213+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:46.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:46.104+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:46.105201+0000) 2020-02-06T01:26:46.112 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:46.110+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:46.111204+0000) 2020-02-06T01:26:47.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:47.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:47.087054+0000) 2020-02-06T01:26:47.358 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:47.356+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:47.357379+0000) 2020-02-06T01:26:47.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:47.356+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:47.357382+0000) 2020-02-06T01:26:47.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:47.357+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:47.357514+0000) 2020-02-06T01:26:48.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:48.218+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.219329+0000) 2020-02-06T01:26:48.221 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:48.219+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.220064+0000) 2020-02-06T01:26:48.222 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:48.221+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.221534+0000) 2020-02-06T01:26:48.222 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:48.220+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.220682+0000) 2020-02-06T01:26:48.223 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:48.221+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.222297+0000) 2020-02-06T01:26:48.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:48.222+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.222675+0000) 2020-02-06T01:26:48.224 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:48.222+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.223271+0000) 2020-02-06T01:26:48.225 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:48.223+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.224241+0000) 2020-02-06T01:26:48.225 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:48.223+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.224036+0000) 2020-02-06T01:26:48.226 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:48.223+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.225057+0000) 2020-02-06T01:26:48.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:48.225+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.225671+0000) 2020-02-06T01:26:48.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:48.224+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.225613+0000) 2020-02-06T01:26:48.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:48.225+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.225868+0000) 2020-02-06T01:26:48.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:48.229+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:48.230945+0000) 2020-02-06T01:26:48.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:48.241+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203547 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:49.719 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:49.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:49.718641+0000) 2020-02-06T01:26:49.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:49.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:49.731124+0000) 2020-02-06T01:26:49.946 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:49.944+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:49.945758+0000) 2020-02-06T01:26:49.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:49.977+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:49.978344+0000) 2020-02-06T01:26:50.584 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:50.583+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:50.583810+0000) 2020-02-06T01:26:50.841 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:50.839+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:50.840584+0000) 2020-02-06T01:26:50.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:50.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:50.863229+0000) 2020-02-06T01:26:50.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:50.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:50.863215+0000) 2020-02-06T01:26:53.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:53.246+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.247135+0000) 2020-02-06T01:26:53.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:53.246+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.247741+0000) 2020-02-06T01:26:53.249 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:53.247+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.248045+0000) 2020-02-06T01:26:53.249 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:53.247+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.248333+0000) 2020-02-06T01:26:53.250 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:53.248+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.249276+0000) 2020-02-06T01:26:53.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:53.249+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.249959+0000) 2020-02-06T01:26:53.251 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:53.249+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.250221+0000) 2020-02-06T01:26:53.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:53.249+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.250516+0000) 2020-02-06T01:26:53.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:53.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.251350+0000) 2020-02-06T01:26:53.252 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:53.251+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.251763+0000) 2020-02-06T01:26:53.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:53.251+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.252487+0000) 2020-02-06T01:26:53.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:53.252+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.253031+0000) 2020-02-06T01:26:53.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:53.252+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.253142+0000) 2020-02-06T01:26:53.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:53.256+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:53.257671+0000) 2020-02-06T01:26:53.273 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:53.271+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203588 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:56.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:56.105+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:56.105481+0000) 2020-02-06T01:26:56.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:56.119+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:56.120268+0000) 2020-02-06T01:26:57.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:57.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:57.087320+0000) 2020-02-06T01:26:57.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:57.357+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:57.357715+0000) 2020-02-06T01:26:57.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:57.357+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:57.357715+0000) 2020-02-06T01:26:57.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:57.357+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:57.357798+0000) 2020-02-06T01:26:58.279 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:58.277+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.278453+0000) 2020-02-06T01:26:58.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:26:58.278+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.279116+0000) 2020-02-06T01:26:58.280 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:26:58.278+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.279371+0000) 2020-02-06T01:26:58.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:58.279+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.279964+0000) 2020-02-06T01:26:58.282 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:58.279+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.280675+0000) 2020-02-06T01:26:58.282 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:26:58.280+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.280480+0000) 2020-02-06T01:26:58.282 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:26:58.280+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.280992+0000) 2020-02-06T01:26:58.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:26:58.280+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.281161+0000) 2020-02-06T01:26:58.283 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:26:58.280+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.281630+0000) 2020-02-06T01:26:58.283 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:26:58.280+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.281946+0000) 2020-02-06T01:26:58.284 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:26:58.281+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.282090+0000) 2020-02-06T01:26:58.284 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:26:58.281+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.282467+0000) 2020-02-06T01:26:58.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:58.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.283407+0000) 2020-02-06T01:26:58.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:58.287+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:58.288546+0000) 2020-02-06T01:26:58.301 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:26:58.299+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203630 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:26:59.595 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:26:59.602 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:26:59.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:26:59.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:59.718887+0000) 2020-02-06T01:26:59.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:59.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:59.731333+0000) 2020-02-06T01:26:59.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:26:59.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:59.945980+0000) 2020-02-06T01:26:59.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:26:59.977+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:26:59.978580+0000) 2020-02-06T01:27:00.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:00.583+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:00.584065+0000) 2020-02-06T01:27:00.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:00.840+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:00.840839+0000) 2020-02-06T01:27:00.864 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:00.862+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:00.863487+0000) 2020-02-06T01:27:00.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:00.862+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:00.863490+0000) 2020-02-06T01:27:03.306 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:03.304+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.305869+0000) 2020-02-06T01:27:03.308 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:03.306+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.307246+0000) 2020-02-06T01:27:03.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:03.307+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.308602+0000) 2020-02-06T01:27:03.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:03.308+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.309354+0000) 2020-02-06T01:27:03.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:03.308+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.309140+0000) 2020-02-06T01:27:03.311 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:03.308+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.309631+0000) 2020-02-06T01:27:03.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:03.309+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.310301+0000) 2020-02-06T01:27:03.312 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:03.310+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.310618+0000) 2020-02-06T01:27:03.312 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:03.310+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.310980+0000) 2020-02-06T01:27:03.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:03.310+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.311380+0000) 2020-02-06T01:27:03.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:03.310+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.311409+0000) 2020-02-06T01:27:03.313 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:03.311+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.311686+0000) 2020-02-06T01:27:03.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:03.311+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.312265+0000) 2020-02-06T01:27:03.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:03.316+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:03.317146+0000) 2020-02-06T01:27:03.329 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:03.327+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203671 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:06.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:06.105+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:06.105737+0000) 2020-02-06T01:27:06.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:06.127+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:06.128423+0000) 2020-02-06T01:27:07.088 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:07.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:07.087555+0000) 2020-02-06T01:27:07.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:07.357+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:07.358021+0000) 2020-02-06T01:27:07.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:07.357+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:07.358021+0000) 2020-02-06T01:27:07.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:07.357+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:07.358321+0000) 2020-02-06T01:27:08.335 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:08.333+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.334262+0000) 2020-02-06T01:27:08.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:08.333+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.334761+0000) 2020-02-06T01:27:08.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:08.334+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.335143+0000) 2020-02-06T01:27:08.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:08.334+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.335434+0000) 2020-02-06T01:27:08.337 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:08.334+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.335760+0000) 2020-02-06T01:27:08.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:08.335+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.336228+0000) 2020-02-06T01:27:08.338 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:08.335+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.336526+0000) 2020-02-06T01:27:08.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:08.335+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.336823+0000) 2020-02-06T01:27:08.338 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:08.336+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.337120+0000) 2020-02-06T01:27:08.339 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:08.338+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.338418+0000) 2020-02-06T01:27:08.340 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:08.338+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.339232+0000) 2020-02-06T01:27:08.340 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:08.339+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.340072+0000) 2020-02-06T01:27:08.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:08.339+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.340175+0000) 2020-02-06T01:27:08.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:08.343+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:08.344166+0000) 2020-02-06T01:27:08.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:08.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203713 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:09.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:09.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:09.719158+0000) 2020-02-06T01:27:09.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:09.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:09.731619+0000) 2020-02-06T01:27:09.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:09.945+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:09.946222+0000) 2020-02-06T01:27:09.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:09.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:09.978815+0000) 2020-02-06T01:27:10.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:10.584+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:10.584323+0000) 2020-02-06T01:27:10.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:10.840+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:10.841029+0000) 2020-02-06T01:27:10.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:10.863+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:10.863748+0000) 2020-02-06T01:27:10.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:10.863+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:10.863749+0000) 2020-02-06T01:27:13.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:13.358+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.359618+0000) 2020-02-06T01:27:13.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:13.361+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.361747+0000) 2020-02-06T01:27:13.363 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:13.361+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.362211+0000) 2020-02-06T01:27:13.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:13.361+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.361610+0000) 2020-02-06T01:27:13.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:13.362+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.363285+0000) 2020-02-06T01:27:13.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:13.363+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.363768+0000) 2020-02-06T01:27:13.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:13.362+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.363938+0000) 2020-02-06T01:27:13.365 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:13.363+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.364366+0000) 2020-02-06T01:27:13.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:13.363+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.365005+0000) 2020-02-06T01:27:13.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:13.364+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.365187+0000) 2020-02-06T01:27:13.368 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:13.366+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.367383+0000) 2020-02-06T01:27:13.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:13.367+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.367568+0000) 2020-02-06T01:27:13.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:13.372+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.373734+0000) 2020-02-06T01:27:13.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:13.372+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:13.374041+0000) 2020-02-06T01:27:13.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:13.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:16.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:16.105+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:16.105978+0000) 2020-02-06T01:27:16.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:16.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:16.136604+0000) 2020-02-06T01:27:17.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:17.086+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:17.087810+0000) 2020-02-06T01:27:17.359 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:17.357+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:17.358302+0000) 2020-02-06T01:27:17.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:17.357+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:17.358302+0000) 2020-02-06T01:27:17.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:17.358+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:17.358598+0000) 2020-02-06T01:27:18.390 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:18.388+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.389678+0000) 2020-02-06T01:27:18.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:18.390+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.390748+0000) 2020-02-06T01:27:18.393 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:18.391+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.392189+0000) 2020-02-06T01:27:18.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:18.391+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.392617+0000) 2020-02-06T01:27:18.394 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:18.392+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.393719+0000) 2020-02-06T01:27:18.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:18.393+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.393879+0000) 2020-02-06T01:27:18.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:18.393+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.394303+0000) 2020-02-06T01:27:18.396 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:18.394+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.395129+0000) 2020-02-06T01:27:18.397 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:18.395+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.395680+0000) 2020-02-06T01:27:18.397 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:18.395+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.395997+0000) 2020-02-06T01:27:18.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:18.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.396278+0000) 2020-02-06T01:27:18.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:18.396+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.397128+0000) 2020-02-06T01:27:18.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:18.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.399822+0000) 2020-02-06T01:27:18.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:18.399+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:18.400747+0000) 2020-02-06T01:27:18.413 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:18.412+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203796 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:19.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:19.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:19.719413+0000) 2020-02-06T01:27:19.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:19.730+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:19.731869+0000) 2020-02-06T01:27:19.947 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:19.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:19.946453+0000) 2020-02-06T01:27:19.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:19.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:19.979047+0000) 2020-02-06T01:27:20.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:20.583+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:20.584637+0000) 2020-02-06T01:27:20.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:20.840+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:20.841292+0000) 2020-02-06T01:27:20.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:20.863+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:20.864023+0000) 2020-02-06T01:27:20.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:20.863+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:20.864023+0000) 2020-02-06T01:27:23.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:23.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.419981+0000) 2020-02-06T01:27:23.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:23.419+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.420752+0000) 2020-02-06T01:27:23.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:23.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.417646+0000) 2020-02-06T01:27:23.423 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:23.416+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.418045+0000) 2020-02-06T01:27:23.423 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:23.418+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.419106+0000) 2020-02-06T01:27:23.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:23.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.419682+0000) 2020-02-06T01:27:23.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:23.419+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.420915+0000) 2020-02-06T01:27:23.424 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:23.420+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.421861+0000) 2020-02-06T01:27:23.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:23.420+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.422262+0000) 2020-02-06T01:27:23.425 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:23.422+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.423541+0000) 2020-02-06T01:27:23.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:23.423+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.424114+0000) 2020-02-06T01:27:23.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:23.422+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.424313+0000) 2020-02-06T01:27:23.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:23.423+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.424742+0000) 2020-02-06T01:27:23.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:23.426+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:23.427516+0000) 2020-02-06T01:27:23.441 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:23.439+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203837 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:26.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:26.104+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:26.106246+0000) 2020-02-06T01:27:26.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:26.144+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:26.144532+0000) 2020-02-06T01:27:27.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:27.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:27.088046+0000) 2020-02-06T01:27:27.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:27.357+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:27.358577+0000) 2020-02-06T01:27:27.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:27.357+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:27.358577+0000) 2020-02-06T01:27:27.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:27.357+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:27.358837+0000) 2020-02-06T01:27:28.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:28.444+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.445471+0000) 2020-02-06T01:27:28.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:28.445+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.446614+0000) 2020-02-06T01:27:28.448 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:28.446+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.447542+0000) 2020-02-06T01:27:28.449 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:28.447+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.448701+0000) 2020-02-06T01:27:28.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:28.447+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.449081+0000) 2020-02-06T01:27:28.450 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:28.448+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.449904+0000) 2020-02-06T01:27:28.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:28.448+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.449826+0000) 2020-02-06T01:27:28.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:28.448+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.450089+0000) 2020-02-06T01:27:28.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:28.448+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.450379+0000) 2020-02-06T01:27:28.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:28.449+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.450918+0000) 2020-02-06T01:27:28.452 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:28.449+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.450711+0000) 2020-02-06T01:27:28.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:28.450+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.451154+0000) 2020-02-06T01:27:28.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:28.450+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.451613+0000) 2020-02-06T01:27:28.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:28.453+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:28.454449+0000) 2020-02-06T01:27:28.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:28.465+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203879 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:29.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:29.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:29.719656+0000) 2020-02-06T01:27:29.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:29.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:29.732125+0000) 2020-02-06T01:27:29.734 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:27:29.740 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:27:29.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:29.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:29.946697+0000) 2020-02-06T01:27:29.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:29.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:29.979308+0000) 2020-02-06T01:27:30.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:30.583+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:30.584890+0000) 2020-02-06T01:27:30.842 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:30.841+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:30.841521+0000) 2020-02-06T01:27:30.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:30.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:30.864319+0000) 2020-02-06T01:27:30.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:30.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:30.864319+0000) 2020-02-06T01:27:33.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:33.471+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.472333+0000) 2020-02-06T01:27:33.474 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:33.472+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.473631+0000) 2020-02-06T01:27:33.475 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:33.472+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.473804+0000) 2020-02-06T01:27:33.475 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:33.473+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.474614+0000) 2020-02-06T01:27:33.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:33.473+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.474896+0000) 2020-02-06T01:27:33.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:33.474+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.475630+0000) 2020-02-06T01:27:33.478 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:33.476+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.477340+0000) 2020-02-06T01:27:33.479 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:33.475+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.476486+0000) 2020-02-06T01:27:33.479 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:33.475+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.476858+0000) 2020-02-06T01:27:33.479 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:33.475+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.477361+0000) 2020-02-06T01:27:33.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:33.476+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.477831+0000) 2020-02-06T01:27:33.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:33.476+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.477960+0000) 2020-02-06T01:27:33.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:33.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.481767+0000) 2020-02-06T01:27:33.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:33.480+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:33.481961+0000) 2020-02-06T01:27:33.496 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:33.494+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203920 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:36.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:36.105+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:36.106512+0000) 2020-02-06T01:27:36.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:36.153+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:36.153221+0000) 2020-02-06T01:27:37.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:37.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:37.088399+0000) 2020-02-06T01:27:37.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:37.357+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:37.358824+0000) 2020-02-06T01:27:37.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:37.357+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:37.358821+0000) 2020-02-06T01:27:37.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:37.357+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:37.359092+0000) 2020-02-06T01:27:38.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:38.501+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.501551+0000) 2020-02-06T01:27:38.503 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:38.501+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.501972+0000) 2020-02-06T01:27:38.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:38.502+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.502808+0000) 2020-02-06T01:27:38.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:38.501+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.502514+0000) 2020-02-06T01:27:38.504 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:38.503+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.503418+0000) 2020-02-06T01:27:38.505 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:38.503+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.503915+0000) 2020-02-06T01:27:38.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:38.504+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.504657+0000) 2020-02-06T01:27:38.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:38.503+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.504955+0000) 2020-02-06T01:27:38.506 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:38.503+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.505294+0000) 2020-02-06T01:27:38.506 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:38.504+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.505737+0000) 2020-02-06T01:27:38.507 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:38.504+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.506298+0000) 2020-02-06T01:27:38.507 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:38.505+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.506674+0000) 2020-02-06T01:27:38.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:38.513+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.513214+0000) 2020-02-06T01:27:38.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:38.513+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:38.513567+0000) 2020-02-06T01:27:38.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:38.522+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 203963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:39.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:39.718+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:39.719943+0000) 2020-02-06T01:27:39.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:39.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:39.732357+0000) 2020-02-06T01:27:39.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:39.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:39.947115+0000) 2020-02-06T01:27:39.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:39.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:39.979549+0000) 2020-02-06T01:27:40.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:40.584+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:40.585187+0000) 2020-02-06T01:27:40.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:40.840+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:40.841755+0000) 2020-02-06T01:27:40.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:40.863+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:40.864583+0000) 2020-02-06T01:27:40.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:40.863+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:40.864583+0000) 2020-02-06T01:27:43.532 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:43.529+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.530968+0000) 2020-02-06T01:27:43.532 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:43.530+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.531740+0000) 2020-02-06T01:27:43.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:43.531+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.532640+0000) 2020-02-06T01:27:43.533 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:43.531+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.532960+0000) 2020-02-06T01:27:43.534 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:43.532+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.533571+0000) 2020-02-06T01:27:43.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:43.531+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.533330+0000) 2020-02-06T01:27:43.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:43.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.534442+0000) 2020-02-06T01:27:43.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:43.532+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.534288+0000) 2020-02-06T01:27:43.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:43.533+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.534846+0000) 2020-02-06T01:27:43.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:43.534+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.535495+0000) 2020-02-06T01:27:43.537 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:43.534+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.535985+0000) 2020-02-06T01:27:43.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:43.535+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.536255+0000) 2020-02-06T01:27:43.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:43.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.536422+0000) 2020-02-06T01:27:43.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:43.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:43.540430+0000) 2020-02-06T01:27:43.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:43.549+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204003 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:46.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:46.106+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:46.106804+0000) 2020-02-06T01:27:46.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:46.159+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:46.160579+0000) 2020-02-06T01:27:47.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:47.087+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:47.088639+0000) 2020-02-06T01:27:47.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:47.358+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:47.359133+0000) 2020-02-06T01:27:47.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:47.358+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:47.359135+0000) 2020-02-06T01:27:47.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:47.358+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:47.359311+0000) 2020-02-06T01:27:48.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:48.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.556650+0000) 2020-02-06T01:27:48.559 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:48.557+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.558472+0000) 2020-02-06T01:27:48.560 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:48.557+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.559039+0000) 2020-02-06T01:27:48.560 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:48.558+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.559565+0000) 2020-02-06T01:27:48.560 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:48.557+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.559115+0000) 2020-02-06T01:27:48.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:48.559+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.561243+0000) 2020-02-06T01:27:48.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:48.560+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.561639+0000) 2020-02-06T01:27:48.563 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:48.560+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.561927+0000) 2020-02-06T01:27:48.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:48.561+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.562485+0000) 2020-02-06T01:27:48.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:48.561+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.562583+0000) 2020-02-06T01:27:48.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:48.560+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.562309+0000) 2020-02-06T01:27:48.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:48.562+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.563288+0000) 2020-02-06T01:27:48.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:48.565+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.566771+0000) 2020-02-06T01:27:48.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:48.565+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:48.566990+0000) 2020-02-06T01:27:48.580 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:48.577+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:49.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:49.719+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:49.720199+0000) 2020-02-06T01:27:49.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:49.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:49.732595+0000) 2020-02-06T01:27:49.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:49.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:49.947383+0000) 2020-02-06T01:27:49.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:49.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:49.979782+0000) 2020-02-06T01:27:50.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:50.584+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:50.585479+0000) 2020-02-06T01:27:50.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:50.841+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:50.841986+0000) 2020-02-06T01:27:50.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:50.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:50.864863+0000) 2020-02-06T01:27:50.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:50.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:50.864859+0000) 2020-02-06T01:27:53.586 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:53.584+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.585583+0000) 2020-02-06T01:27:53.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:53.585+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.587323+0000) 2020-02-06T01:27:53.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:53.586+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.587625+0000) 2020-02-06T01:27:53.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:53.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.589043+0000) 2020-02-06T01:27:53.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:53.587+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.589405+0000) 2020-02-06T01:27:53.591 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:53.588+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.589810+0000) 2020-02-06T01:27:53.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:53.588+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.589774+0000) 2020-02-06T01:27:53.592 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:53.589+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.590409+0000) 2020-02-06T01:27:53.592 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:53.589+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.590821+0000) 2020-02-06T01:27:53.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:53.589+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.590536+0000) 2020-02-06T01:27:53.592 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:53.589+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.591008+0000) 2020-02-06T01:27:53.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:53.590+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.591190+0000) 2020-02-06T01:27:53.593 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:53.590+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.591960+0000) 2020-02-06T01:27:53.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:53.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:53.597185+0000) 2020-02-06T01:27:53.608 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:53.606+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204086 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:56.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:56.106+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:56.107066+0000) 2020-02-06T01:27:56.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:56.168+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:56.169200+0000) 2020-02-06T01:27:57.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:57.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:57.088935+0000) 2020-02-06T01:27:57.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:57.358+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:57.359479+0000) 2020-02-06T01:27:57.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:57.358+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:57.359476+0000) 2020-02-06T01:27:57.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:57.358+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:57.359651+0000) 2020-02-06T01:27:58.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:58.612+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.613799+0000) 2020-02-06T01:27:58.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:58.613+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.614441+0000) 2020-02-06T01:27:58.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:27:58.613+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.614827+0000) 2020-02-06T01:27:58.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:27:58.614+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.615451+0000) 2020-02-06T01:27:58.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:27:58.615+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.616279+0000) 2020-02-06T01:27:58.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:58.615+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.616927+0000) 2020-02-06T01:27:58.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:58.616+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.617116+0000) 2020-02-06T01:27:58.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:27:58.615+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.617393+0000) 2020-02-06T01:27:58.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:27:58.617+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.618140+0000) 2020-02-06T01:27:58.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:27:58.617+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.618685+0000) 2020-02-06T01:27:58.620 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:27:58.617+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.618876+0000) 2020-02-06T01:27:58.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:27:58.617+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.619216+0000) 2020-02-06T01:27:58.620 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:27:58.618+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.619485+0000) 2020-02-06T01:27:58.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:58.621+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:58.622624+0000) 2020-02-06T01:27:58.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:27:58.633+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204129 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:27:59.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:27:59.719+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:59.720491+0000) 2020-02-06T01:27:59.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:59.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:59.732834+0000) 2020-02-06T01:27:59.880 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:27:59.887 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:27:59.948 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:27:59.946+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:59.947594+0000) 2020-02-06T01:27:59.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:27:59.978+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:27:59.979954+0000) 2020-02-06T01:28:00.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:00.585+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:00.585736+0000) 2020-02-06T01:28:00.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:00.841+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:00.842224+0000) 2020-02-06T01:28:00.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:00.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:00.865144+0000) 2020-02-06T01:28:00.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:00.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:00.865144+0000) 2020-02-06T01:28:03.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:03.640+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.641199+0000) 2020-02-06T01:28:03.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:03.641+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.642060+0000) 2020-02-06T01:28:03.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:03.641+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.642517+0000) 2020-02-06T01:28:03.644 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:03.641+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.642958+0000) 2020-02-06T01:28:03.644 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:03.642+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.643681+0000) 2020-02-06T01:28:03.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:03.643+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.644066+0000) 2020-02-06T01:28:03.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:03.642+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.644353+0000) 2020-02-06T01:28:03.646 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:03.643+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.644461+0000) 2020-02-06T01:28:03.646 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:03.643+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.644416+0000) 2020-02-06T01:28:03.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:03.643+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.644968+0000) 2020-02-06T01:28:03.647 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:03.644+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.645518+0000) 2020-02-06T01:28:03.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:03.644+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.646030+0000) 2020-02-06T01:28:03.648 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:03.646+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.647270+0000) 2020-02-06T01:28:03.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:03.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:03.651758+0000) 2020-02-06T01:28:03.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:03.661+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:06.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:06.106+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:06.107314+0000) 2020-02-06T01:28:06.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:06.176+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:06.177557+0000) 2020-02-06T01:28:07.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:07.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:07.089161+0000) 2020-02-06T01:28:07.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:07.359+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:07.359783+0000) 2020-02-06T01:28:07.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:07.359+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:07.359808+0000) 2020-02-06T01:28:07.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:07.359+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:07.359917+0000) 2020-02-06T01:28:08.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:08.667+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.668129+0000) 2020-02-06T01:28:08.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:08.668+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.668934+0000) 2020-02-06T01:28:08.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:08.669+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.670150+0000) 2020-02-06T01:28:08.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:08.669+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.670665+0000) 2020-02-06T01:28:08.671 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:08.669+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.670880+0000) 2020-02-06T01:28:08.672 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:08.671+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.671526+0000) 2020-02-06T01:28:08.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:08.670+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.671762+0000) 2020-02-06T01:28:08.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:08.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.672395+0000) 2020-02-06T01:28:08.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:08.672+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.673403+0000) 2020-02-06T01:28:08.674 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:08.673+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.673715+0000) 2020-02-06T01:28:08.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:08.673+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.674184+0000) 2020-02-06T01:28:08.675 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:08.674+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.674766+0000) 2020-02-06T01:28:08.675 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:08.673+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.674323+0000) 2020-02-06T01:28:08.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:08.676+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:08.677784+0000) 2020-02-06T01:28:08.690 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:08.689+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204211 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:09.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:09.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:09.720757+0000) 2020-02-06T01:28:09.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:09.731+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:09.733054+0000) 2020-02-06T01:28:09.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:09.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:09.947844+0000) 2020-02-06T01:28:09.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:09.979+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:09.980177+0000) 2020-02-06T01:28:10.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:10.585+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:10.585996+0000) 2020-02-06T01:28:10.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:10.842+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:10.842489+0000) 2020-02-06T01:28:10.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:10.864+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:10.865437+0000) 2020-02-06T01:28:10.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:10.864+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:10.865438+0000) 2020-02-06T01:28:13.695 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:13.693+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.694722+0000) 2020-02-06T01:28:13.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:13.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.696063+0000) 2020-02-06T01:28:13.697 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:13.695+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.696065+0000) 2020-02-06T01:28:13.698 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:13.696+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.696915+0000) 2020-02-06T01:28:13.698 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:13.696+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.697212+0000) 2020-02-06T01:28:13.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:13.696+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.697737+0000) 2020-02-06T01:28:13.699 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:13.697+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.698249+0000) 2020-02-06T01:28:13.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:13.698+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.699958+0000) 2020-02-06T01:28:13.701 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:13.699+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.700149+0000) 2020-02-06T01:28:13.701 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:13.699+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.700476+0000) 2020-02-06T01:28:13.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:13.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.700979+0000) 2020-02-06T01:28:13.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:13.700+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.701171+0000) 2020-02-06T01:28:13.702 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:13.701+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.701571+0000) 2020-02-06T01:28:13.706 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:13.704+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:13.705291+0000) 2020-02-06T01:28:13.719 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:13.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:16.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:16.107+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:16.107558+0000) 2020-02-06T01:28:16.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:16.185+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:16.185474+0000) 2020-02-06T01:28:17.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:17.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:17.089391+0000) 2020-02-06T01:28:17.361 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:17.359+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:17.360082+0000) 2020-02-06T01:28:17.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:17.359+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:17.360082+0000) 2020-02-06T01:28:17.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:17.359+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:17.360199+0000) 2020-02-06T01:28:18.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:18.723+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.724071+0000) 2020-02-06T01:28:18.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:18.723+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.724472+0000) 2020-02-06T01:28:18.726 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:18.724+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.724657+0000) 2020-02-06T01:28:18.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:18.724+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.724864+0000) 2020-02-06T01:28:18.727 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:18.724+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.725309+0000) 2020-02-06T01:28:18.727 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:18.724+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.725590+0000) 2020-02-06T01:28:18.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:18.724+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.725842+0000) 2020-02-06T01:28:18.728 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:18.725+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.726041+0000) 2020-02-06T01:28:18.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:18.725+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.726819+0000) 2020-02-06T01:28:18.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:18.726+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.727704+0000) 2020-02-06T01:28:18.729 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:18.727+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.728153+0000) 2020-02-06T01:28:18.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:18.729+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.729540+0000) 2020-02-06T01:28:18.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:18.729+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.730082+0000) 2020-02-06T01:28:18.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:18.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:18.734960+0000) 2020-02-06T01:28:18.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:18.745+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:19.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:19.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:19.720991+0000) 2020-02-06T01:28:19.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:19.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:19.733314+0000) 2020-02-06T01:28:19.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:19.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:19.948129+0000) 2020-02-06T01:28:19.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:19.980+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:19.980403+0000) 2020-02-06T01:28:20.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:20.585+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:20.586242+0000) 2020-02-06T01:28:20.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:20.842+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:20.842743+0000) 2020-02-06T01:28:20.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:20.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:20.865720+0000) 2020-02-06T01:28:20.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:20.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:20.865724+0000) 2020-02-06T01:28:23.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:23.751+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.752436+0000) 2020-02-06T01:28:23.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:23.751+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.752666+0000) 2020-02-06T01:28:23.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:23.751+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.753030+0000) 2020-02-06T01:28:23.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:23.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.753258+0000) 2020-02-06T01:28:23.755 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:23.752+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.753855+0000) 2020-02-06T01:28:23.755 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:23.753+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.754192+0000) 2020-02-06T01:28:23.756 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:23.754+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.754603+0000) 2020-02-06T01:28:23.756 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:23.754+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.755299+0000) 2020-02-06T01:28:23.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:23.754+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.755291+0000) 2020-02-06T01:28:23.757 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:23.754+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.755917+0000) 2020-02-06T01:28:23.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:23.755+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.755973+0000) 2020-02-06T01:28:23.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:23.756+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.757260+0000) 2020-02-06T01:28:23.758 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:23.757+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.757699+0000) 2020-02-06T01:28:23.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:23.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:23.761179+0000) 2020-02-06T01:28:23.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:23.774+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204337 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:26.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:26.107+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:26.107825+0000) 2020-02-06T01:28:26.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:26.197+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:26.197446+0000) 2020-02-06T01:28:27.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:27.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:27.089693+0000) 2020-02-06T01:28:27.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:27.360+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:27.360388+0000) 2020-02-06T01:28:27.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:27.360+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:27.360476+0000) 2020-02-06T01:28:27.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:27.360+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:27.360392+0000) 2020-02-06T01:28:28.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:28.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.779543+0000) 2020-02-06T01:28:28.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:28.779+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.780151+0000) 2020-02-06T01:28:28.782 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:28.780+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.781285+0000) 2020-02-06T01:28:28.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:28.781+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.782351+0000) 2020-02-06T01:28:28.783 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:28.781+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.782055+0000) 2020-02-06T01:28:28.784 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:28.782+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.783406+0000) 2020-02-06T01:28:28.784 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:28.782+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.782965+0000) 2020-02-06T01:28:28.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:28.784+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.784201+0000) 2020-02-06T01:28:28.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:28.783+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.784262+0000) 2020-02-06T01:28:28.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:28.784+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.784962+0000) 2020-02-06T01:28:28.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:28.784+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.785498+0000) 2020-02-06T01:28:28.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:28.784+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.785859+0000) 2020-02-06T01:28:28.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:28.785+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.786220+0000) 2020-02-06T01:28:28.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:28.789+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:28.789154+0000) 2020-02-06T01:28:28.803 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:28.801+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:29.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:29.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:29.721217+0000) 2020-02-06T01:28:29.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:29.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:29.733551+0000) 2020-02-06T01:28:29.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:29.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:29.948398+0000) 2020-02-06T01:28:29.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:29.980+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:29.980624+0000) 2020-02-06T01:28:30.093 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:28:30.100 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:28:30.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:30.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:30.586499+0000) 2020-02-06T01:28:30.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:30.842+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:30.842998+0000) 2020-02-06T01:28:30.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:30.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:30.865991+0000) 2020-02-06T01:28:30.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:30.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:30.865991+0000) 2020-02-06T01:28:33.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:33.805+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.806962+0000) 2020-02-06T01:28:33.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:33.807+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.808804+0000) 2020-02-06T01:28:33.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:33.808+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.809148+0000) 2020-02-06T01:28:33.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:33.808+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.809628+0000) 2020-02-06T01:28:33.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:33.809+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.810462+0000) 2020-02-06T01:28:33.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:33.809+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.810134+0000) 2020-02-06T01:28:33.812 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:33.810+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.811037+0000) 2020-02-06T01:28:33.812 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:33.811+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.812018+0000) 2020-02-06T01:28:33.813 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:33.811+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.812355+0000) 2020-02-06T01:28:33.814 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:33.812+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.812458+0000) 2020-02-06T01:28:33.814 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:33.812+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.812639+0000) 2020-02-06T01:28:33.814 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:33.812+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.813173+0000) 2020-02-06T01:28:33.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:33.812+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.813492+0000) 2020-02-06T01:28:33.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:33.816+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:33.817659+0000) 2020-02-06T01:28:33.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:33.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:36.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:36.107+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:36.108091+0000) 2020-02-06T01:28:36.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:36.204+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:36.205958+0000) 2020-02-06T01:28:37.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:37.088+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:37.089934+0000) 2020-02-06T01:28:37.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:37.360+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:37.360716+0000) 2020-02-06T01:28:37.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:37.360+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:37.360724+0000) 2020-02-06T01:28:37.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:37.360+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:37.360718+0000) 2020-02-06T01:28:38.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:38.836+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.837362+0000) 2020-02-06T01:28:38.839 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:38.837+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.838122+0000) 2020-02-06T01:28:38.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:38.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.838794+0000) 2020-02-06T01:28:38.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:38.838+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.839102+0000) 2020-02-06T01:28:38.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:38.840+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.840993+0000) 2020-02-06T01:28:38.842 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:38.840+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.841310+0000) 2020-02-06T01:28:38.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:38.841+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.841769+0000) 2020-02-06T01:28:38.843 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:38.841+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.842401+0000) 2020-02-06T01:28:38.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:38.841+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.842953+0000) 2020-02-06T01:28:38.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:38.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.842868+0000) 2020-02-06T01:28:38.844 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:38.842+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.843378+0000) 2020-02-06T01:28:38.845 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:38.843+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.843759+0000) 2020-02-06T01:28:38.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:38.843+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.843843+0000) 2020-02-06T01:28:38.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:38.846+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:38.847527+0000) 2020-02-06T01:28:38.860 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:38.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204463 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:39.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:39.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:39.721379+0000) 2020-02-06T01:28:39.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:39.732+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:39.733769+0000) 2020-02-06T01:28:39.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:39.947+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:39.948628+0000) 2020-02-06T01:28:39.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:39.979+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:39.980840+0000) 2020-02-06T01:28:40.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:40.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:40.586779+0000) 2020-02-06T01:28:40.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:40.842+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:40.843274+0000) 2020-02-06T01:28:40.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:40.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:40.866300+0000) 2020-02-06T01:28:40.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:40.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:40.866300+0000) 2020-02-06T01:28:43.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:43.865+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.866558+0000) 2020-02-06T01:28:43.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:43.866+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.867360+0000) 2020-02-06T01:28:43.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:43.866+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.867858+0000) 2020-02-06T01:28:43.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:43.867+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.868683+0000) 2020-02-06T01:28:43.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:43.867+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.868288+0000) 2020-02-06T01:28:43.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:43.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.869169+0000) 2020-02-06T01:28:43.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:43.868+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.869632+0000) 2020-02-06T01:28:43.871 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:43.868+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.869945+0000) 2020-02-06T01:28:43.871 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:43.869+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.870370+0000) 2020-02-06T01:28:43.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:43.870+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.871093+0000) 2020-02-06T01:28:43.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:43.870+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.870997+0000) 2020-02-06T01:28:43.873 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:43.870+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.871861+0000) 2020-02-06T01:28:43.873 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:43.871+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.871919+0000) 2020-02-06T01:28:43.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:43.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:43.876635+0000) 2020-02-06T01:28:43.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:43.887+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:46.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:46.106+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:46.108337+0000) 2020-02-06T01:28:46.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:46.213+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:46.214046+0000) 2020-02-06T01:28:47.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:47.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:47.090194+0000) 2020-02-06T01:28:47.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:47.359+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:47.360991+0000) 2020-02-06T01:28:47.362 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:47.359+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:47.361014+0000) 2020-02-06T01:28:47.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:47.359+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:47.361009+0000) 2020-02-06T01:28:48.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:48.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.893829+0000) 2020-02-06T01:28:48.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:48.893+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.894408+0000) 2020-02-06T01:28:48.896 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:48.893+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.894884+0000) 2020-02-06T01:28:48.896 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:48.894+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.895579+0000) 2020-02-06T01:28:48.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:48.894+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.895798+0000) 2020-02-06T01:28:48.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:48.894+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.896348+0000) 2020-02-06T01:28:48.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:48.895+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.897002+0000) 2020-02-06T01:28:48.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:48.895+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.896466+0000) 2020-02-06T01:28:48.899 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:48.897+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.898103+0000) 2020-02-06T01:28:48.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:48.897+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.898507+0000) 2020-02-06T01:28:48.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:48.897+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.898903+0000) 2020-02-06T01:28:48.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:48.898+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.899066+0000) 2020-02-06T01:28:48.901 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:48.898+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.900299+0000) 2020-02-06T01:28:48.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:48.904+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:48.905396+0000) 2020-02-06T01:28:48.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:48.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:49.722 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:49.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:49.721636+0000) 2020-02-06T01:28:49.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:49.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:49.733981+0000) 2020-02-06T01:28:49.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:49.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:49.948871+0000) 2020-02-06T01:28:49.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:49.980+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:49.981069+0000) 2020-02-06T01:28:50.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:50.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:50.587077+0000) 2020-02-06T01:28:50.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:50.842+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:50.843505+0000) 2020-02-06T01:28:50.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:50.865+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:50.866565+0000) 2020-02-06T01:28:50.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:50.865+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:50.866565+0000) 2020-02-06T01:28:53.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:53.920+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.921798+0000) 2020-02-06T01:28:53.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:53.920+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.922189+0000) 2020-02-06T01:28:53.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:53.921+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.922378+0000) 2020-02-06T01:28:53.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:53.921+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.922953+0000) 2020-02-06T01:28:53.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:53.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.923524+0000) 2020-02-06T01:28:53.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:53.922+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.924021+0000) 2020-02-06T01:28:53.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:53.923+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.924683+0000) 2020-02-06T01:28:53.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:53.924+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.925145+0000) 2020-02-06T01:28:53.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:53.924+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.925338+0000) 2020-02-06T01:28:53.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:53.924+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.925695+0000) 2020-02-06T01:28:53.927 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:53.925+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.926712+0000) 2020-02-06T01:28:53.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:53.925+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.926908+0000) 2020-02-06T01:28:53.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:53.925+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.927209+0000) 2020-02-06T01:28:53.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:53.928+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:53.930056+0000) 2020-02-06T01:28:53.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:53.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204589 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:56.109 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:56.107+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:56.108572+0000) 2020-02-06T01:28:56.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:56.221+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:56.222380+0000) 2020-02-06T01:28:57.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:57.089+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:57.090425+0000) 2020-02-06T01:28:57.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:57.360+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:57.361245+0000) 2020-02-06T01:28:57.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:57.360+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:57.361284+0000) 2020-02-06T01:28:57.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:57.360+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:57.361282+0000) 2020-02-06T01:28:58.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:28:58.946+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.947486+0000) 2020-02-06T01:28:58.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:28:58.948+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.949228+0000) 2020-02-06T01:28:58.951 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:28:58.948+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.949751+0000) 2020-02-06T01:28:58.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:58.949+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.950148+0000) 2020-02-06T01:28:58.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:28:58.949+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.950637+0000) 2020-02-06T01:28:58.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:58.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.950946+0000) 2020-02-06T01:28:58.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:28:58.950+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.951408+0000) 2020-02-06T01:28:58.953 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:58.950+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.951916+0000) 2020-02-06T01:28:58.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:28:58.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.952416+0000) 2020-02-06T01:28:58.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:58.952+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.953334+0000) 2020-02-06T01:28:58.954 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:28:58.951+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.953130+0000) 2020-02-06T01:28:58.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:28:58.952+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.953988+0000) 2020-02-06T01:28:58.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:28:58.952+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.954151+0000) 2020-02-06T01:28:58.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:58.959+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:58.960926+0000) 2020-02-06T01:28:58.970 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:28:58.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:28:59.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:28:59.720+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:59.721861+0000) 2020-02-06T01:28:59.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:59.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:59.734187+0000) 2020-02-06T01:28:59.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:28:59.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:59.949146+0000) 2020-02-06T01:28:59.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:28:59.980+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:28:59.981334+0000) 2020-02-06T01:29:00.274 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:29:00.280 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:29:00.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:00.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:00.587340+0000) 2020-02-06T01:29:00.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:00.843+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:00.843730+0000) 2020-02-06T01:29:00.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:00.866+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:00.866818+0000) 2020-02-06T01:29:00.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:00.866+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:00.866822+0000) 2020-02-06T01:29:03.976 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:03.974+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.975555+0000) 2020-02-06T01:29:03.977 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:03.975+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.976143+0000) 2020-02-06T01:29:03.977 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:03.975+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.976753+0000) 2020-02-06T01:29:03.978 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:03.976+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.977526+0000) 2020-02-06T01:29:03.978 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:03.976+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.977940+0000) 2020-02-06T01:29:03.979 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:03.977+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.978611+0000) 2020-02-06T01:29:03.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:03.977+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.978895+0000) 2020-02-06T01:29:03.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:03.978+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.979610+0000) 2020-02-06T01:29:03.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:03.979+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.980557+0000) 2020-02-06T01:29:03.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:03.979+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.980980+0000) 2020-02-06T01:29:03.982 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:03.980+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.981551+0000) 2020-02-06T01:29:03.983 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:03.980+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.982376+0000) 2020-02-06T01:29:03.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:03.984+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.985460+0000) 2020-02-06T01:29:03.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:03.984+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:03.985774+0000) 2020-02-06T01:29:03.997 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:03.995+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204674 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:06.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:06.107+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:06.108787+0000) 2020-02-06T01:29:06.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:06.230+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:06.230735+0000) 2020-02-06T01:29:07.091 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:07.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:07.090654+0000) 2020-02-06T01:29:07.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:07.360+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:07.361501+0000) 2020-02-06T01:29:07.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:07.360+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:07.361580+0000) 2020-02-06T01:29:07.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:07.360+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:07.361575+0000) 2020-02-06T01:29:09.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:09.000+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.001420+0000) 2020-02-06T01:29:09.003 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:09.001+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.002695+0000) 2020-02-06T01:29:09.004 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:09.001+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.002971+0000) 2020-02-06T01:29:09.004 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:09.002+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.003685+0000) 2020-02-06T01:29:09.005 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:09.003+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.004159+0000) 2020-02-06T01:29:09.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:09.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.004746+0000) 2020-02-06T01:29:09.006 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:09.003+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.005074+0000) 2020-02-06T01:29:09.007 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:09.004+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.005844+0000) 2020-02-06T01:29:09.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:09.005+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.006188+0000) 2020-02-06T01:29:09.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:09.004+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.006383+0000) 2020-02-06T01:29:09.008 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:09.005+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.007366+0000) 2020-02-06T01:29:09.008 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:09.006+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.007904+0000) 2020-02-06T01:29:09.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:09.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.011100+0000) 2020-02-06T01:29:09.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:09.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.011374+0000) 2020-02-06T01:29:09.024 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:09.022+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204716 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:09.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:09.721+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.722093+0000) 2020-02-06T01:29:09.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:09.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.734397+0000) 2020-02-06T01:29:09.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:09.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.949358+0000) 2020-02-06T01:29:09.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:09.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:09.981592+0000) 2020-02-06T01:29:10.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:10.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:10.587630+0000) 2020-02-06T01:29:10.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:10.843+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:10.843965+0000) 2020-02-06T01:29:10.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:10.866+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:10.867025+0000) 2020-02-06T01:29:10.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:10.866+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:10.867028+0000) 2020-02-06T01:29:14.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:14.028+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.029704+0000) 2020-02-06T01:29:14.031 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:14.029+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.030181+0000) 2020-02-06T01:29:14.031 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:14.029+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.030654+0000) 2020-02-06T01:29:14.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:14.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.031378+0000) 2020-02-06T01:29:14.033 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:14.030+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.032134+0000) 2020-02-06T01:29:14.034 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:14.031+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.032565+0000) 2020-02-06T01:29:14.035 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:14.032+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.033731+0000) 2020-02-06T01:29:14.035 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:14.031+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.033292+0000) 2020-02-06T01:29:14.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:14.033+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.034641+0000) 2020-02-06T01:29:14.036 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:14.034+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.035289+0000) 2020-02-06T01:29:14.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:14.034+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.035466+0000) 2020-02-06T01:29:14.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:14.034+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.035615+0000) 2020-02-06T01:29:14.037 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:14.034+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.035855+0000) 2020-02-06T01:29:14.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:14.039+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:14.040736+0000) 2020-02-06T01:29:14.052 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:14.049+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204758 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:16.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:16.108+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:16.109038+0000) 2020-02-06T01:29:16.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:16.238+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:16.239313+0000) 2020-02-06T01:29:17.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:17.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:17.090894+0000) 2020-02-06T01:29:17.363 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:17.361+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:17.361877+0000) 2020-02-06T01:29:17.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:17.361+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:17.361769+0000) 2020-02-06T01:29:17.363 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:17.361+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:17.361853+0000) 2020-02-06T01:29:19.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:19.055+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.056892+0000) 2020-02-06T01:29:19.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:19.056+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.057594+0000) 2020-02-06T01:29:19.059 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:19.056+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.057969+0000) 2020-02-06T01:29:19.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:19.057+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.058984+0000) 2020-02-06T01:29:19.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:19.058+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.059315+0000) 2020-02-06T01:29:19.061 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:19.058+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.060363+0000) 2020-02-06T01:29:19.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:19.059+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.060733+0000) 2020-02-06T01:29:19.062 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:19.059+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.061092+0000) 2020-02-06T01:29:19.062 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:19.060+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.061212+0000) 2020-02-06T01:29:19.063 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:19.059+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.061289+0000) 2020-02-06T01:29:19.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:19.060+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.061849+0000) 2020-02-06T01:29:19.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:19.061+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.062649+0000) 2020-02-06T01:29:19.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:19.061+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.062718+0000) 2020-02-06T01:29:19.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:19.069+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.070382+0000) 2020-02-06T01:29:19.078 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:19.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204802 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:19.723 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:19.721+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.722314+0000) 2020-02-06T01:29:19.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:19.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.734627+0000) 2020-02-06T01:29:19.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:19.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.949572+0000) 2020-02-06T01:29:19.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:19.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:19.981806+0000) 2020-02-06T01:29:20.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:20.587+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:20.587926+0000) 2020-02-06T01:29:20.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:20.844+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:20.844197+0000) 2020-02-06T01:29:20.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:20.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:20.867308+0000) 2020-02-06T01:29:20.868 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:20.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:20.867310+0000) 2020-02-06T01:29:24.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:24.084+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.084427+0000) 2020-02-06T01:29:24.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:24.083+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.084736+0000) 2020-02-06T01:29:24.086 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:24.085+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.085413+0000) 2020-02-06T01:29:24.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:24.085+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.085873+0000) 2020-02-06T01:29:24.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:24.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.086634+0000) 2020-02-06T01:29:24.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:24.085+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.086798+0000) 2020-02-06T01:29:24.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:24.087+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.087511+0000) 2020-02-06T01:29:24.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:24.086+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.087636+0000) 2020-02-06T01:29:24.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:24.086+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.087901+0000) 2020-02-06T01:29:24.089 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:24.088+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.088873+0000) 2020-02-06T01:29:24.090 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:24.087+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.089172+0000) 2020-02-06T01:29:24.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:24.088+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.089644+0000) 2020-02-06T01:29:24.091 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:24.089+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.089295+0000) 2020-02-06T01:29:24.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:24.093+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:24.093312+0000) 2020-02-06T01:29:24.109 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:24.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204843 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:26.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:26.108+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:26.109298+0000) 2020-02-06T01:29:26.251 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:26.249+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:26.249280+0000) 2020-02-06T01:29:27.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:27.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:27.091133+0000) 2020-02-06T01:29:27.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:27.361+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:27.362026+0000) 2020-02-06T01:29:27.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:27.361+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:27.362208+0000) 2020-02-06T01:29:27.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:27.361+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:27.362125+0000) 2020-02-06T01:29:29.115 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:29.111+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.112595+0000) 2020-02-06T01:29:29.115 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:29.113+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.114187+0000) 2020-02-06T01:29:29.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:29.114+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.115304+0000) 2020-02-06T01:29:29.116 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:29.114+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.115836+0000) 2020-02-06T01:29:29.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:29.115+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.116540+0000) 2020-02-06T01:29:29.118 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:29.115+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.117199+0000) 2020-02-06T01:29:29.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:29.116+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.118249+0000) 2020-02-06T01:29:29.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:29.117+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.119113+0000) 2020-02-06T01:29:29.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:29.118+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.119600+0000) 2020-02-06T01:29:29.121 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:29.117+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.119330+0000) 2020-02-06T01:29:29.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:29.118+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.120025+0000) 2020-02-06T01:29:29.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:29.118+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.120026+0000) 2020-02-06T01:29:29.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:29.120+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.121576+0000) 2020-02-06T01:29:29.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:29.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.122506+0000) 2020-02-06T01:29:29.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:29.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204886 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:29.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:29.721+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.722529+0000) 2020-02-06T01:29:29.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:29.733+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.734822+0000) 2020-02-06T01:29:29.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:29.948+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.949785+0000) 2020-02-06T01:29:29.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:29.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:29.982009+0000) 2020-02-06T01:29:30.437 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:29:30.443 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:29:30.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:30.586+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:30.588207+0000) 2020-02-06T01:29:30.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:30.843+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:30.844466+0000) 2020-02-06T01:29:30.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:30.866+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:30.867572+0000) 2020-02-06T01:29:30.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:30.866+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:30.867576+0000) 2020-02-06T01:29:34.141 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:34.139+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.140254+0000) 2020-02-06T01:29:34.142 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:34.140+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.141427+0000) 2020-02-06T01:29:34.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:34.141+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.142128+0000) 2020-02-06T01:29:34.143 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:34.141+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.142835+0000) 2020-02-06T01:29:34.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:34.142+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.142613+0000) 2020-02-06T01:29:34.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:34.142+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.143236+0000) 2020-02-06T01:29:34.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:34.143+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.143876+0000) 2020-02-06T01:29:34.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:34.143+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.144252+0000) 2020-02-06T01:29:34.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:34.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.145020+0000) 2020-02-06T01:29:34.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:34.145+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.146303+0000) 2020-02-06T01:29:34.148 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:34.146+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.146528+0000) 2020-02-06T01:29:34.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:34.146+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.146995+0000) 2020-02-06T01:29:34.148 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:34.146+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.147312+0000) 2020-02-06T01:29:34.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:34.149+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:34.150805+0000) 2020-02-06T01:29:34.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:34.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:36.110 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:36.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:36.109559+0000) 2020-02-06T01:29:36.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:36.256+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:36.257161+0000) 2020-02-06T01:29:37.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:37.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:37.091381+0000) 2020-02-06T01:29:37.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:37.361+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:37.362283+0000) 2020-02-06T01:29:37.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:37.361+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:37.362378+0000) 2020-02-06T01:29:37.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:37.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:37.362484+0000) 2020-02-06T01:29:39.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:39.169+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.170716+0000) 2020-02-06T01:29:39.172 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:39.170+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.171308+0000) 2020-02-06T01:29:39.172 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:39.170+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.171853+0000) 2020-02-06T01:29:39.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:39.171+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.172238+0000) 2020-02-06T01:29:39.174 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:39.172+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.173060+0000) 2020-02-06T01:29:39.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:39.172+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.173383+0000) 2020-02-06T01:29:39.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:39.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.173998+0000) 2020-02-06T01:29:39.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:39.173+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.174546+0000) 2020-02-06T01:29:39.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:39.174+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.175176+0000) 2020-02-06T01:29:39.177 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:39.175+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.175872+0000) 2020-02-06T01:29:39.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:39.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.175261+0000) 2020-02-06T01:29:39.178 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:39.175+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.175498+0000) 2020-02-06T01:29:39.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:39.174+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.175826+0000) 2020-02-06T01:29:39.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:39.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.178944+0000) 2020-02-06T01:29:39.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:39.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 204972 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:39.732 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:39.722+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.722737+0000) 2020-02-06T01:29:39.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:39.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.735002+0000) 2020-02-06T01:29:39.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:39.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.949988+0000) 2020-02-06T01:29:39.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:39.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:39.982206+0000) 2020-02-06T01:29:40.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:40.588+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:40.588490+0000) 2020-02-06T01:29:40.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:40.843+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:40.844704+0000) 2020-02-06T01:29:40.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:40.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:40.867838+0000) 2020-02-06T01:29:40.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:40.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:40.867839+0000) 2020-02-06T01:29:44.199 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:44.196+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.198032+0000) 2020-02-06T01:29:44.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:44.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.198716+0000) 2020-02-06T01:29:44.200 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:44.198+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.199227+0000) 2020-02-06T01:29:44.200 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:44.198+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.199815+0000) 2020-02-06T01:29:44.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:44.199+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.200689+0000) 2020-02-06T01:29:44.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:44.200+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.201339+0000) 2020-02-06T01:29:44.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:44.200+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.200572+0000) 2020-02-06T01:29:44.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:44.201+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.201802+0000) 2020-02-06T01:29:44.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:44.201+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.203017+0000) 2020-02-06T01:29:44.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:44.202+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.203322+0000) 2020-02-06T01:29:44.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:44.202+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.203386+0000) 2020-02-06T01:29:44.205 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:44.204+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.204928+0000) 2020-02-06T01:29:44.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:44.204+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.204671+0000) 2020-02-06T01:29:44.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:44.205+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:44.206885+0000) 2020-02-06T01:29:44.221 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:44.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205014 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:46.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:46.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:46.109820+0000) 2020-02-06T01:29:46.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:46.264+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:46.265563+0000) 2020-02-06T01:29:47.092 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:47.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:47.091626+0000) 2020-02-06T01:29:47.363 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:47.362+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:47.362542+0000) 2020-02-06T01:29:47.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:47.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:47.362677+0000) 2020-02-06T01:29:47.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:47.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:47.362751+0000) 2020-02-06T01:29:49.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:49.224+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.225371+0000) 2020-02-06T01:29:49.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:49.225+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.226259+0000) 2020-02-06T01:29:49.227 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:49.225+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.227007+0000) 2020-02-06T01:29:49.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:49.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.227678+0000) 2020-02-06T01:29:49.229 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:49.227+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.228201+0000) 2020-02-06T01:29:49.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:49.227+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.228696+0000) 2020-02-06T01:29:49.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:49.227+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.228763+0000) 2020-02-06T01:29:49.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:49.228+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.229319+0000) 2020-02-06T01:29:49.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:49.229+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.229892+0000) 2020-02-06T01:29:49.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:49.229+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.230030+0000) 2020-02-06T01:29:49.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:49.230+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.230703+0000) 2020-02-06T01:29:49.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:49.230+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.231358+0000) 2020-02-06T01:29:49.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:49.231+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.231744+0000) 2020-02-06T01:29:49.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:49.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.235497+0000) 2020-02-06T01:29:49.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:49.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:49.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:49.722+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.722954+0000) 2020-02-06T01:29:49.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:49.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.735211+0000) 2020-02-06T01:29:49.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:49.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.950207+0000) 2020-02-06T01:29:49.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:49.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:49.982369+0000) 2020-02-06T01:29:50.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:50.588+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:50.588786+0000) 2020-02-06T01:29:50.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:50.843+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:50.844961+0000) 2020-02-06T01:29:50.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:50.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:50.868151+0000) 2020-02-06T01:29:50.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:50.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:50.868151+0000) 2020-02-06T01:29:54.253 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:54.250+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.252026+0000) 2020-02-06T01:29:54.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:54.251+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.252912+0000) 2020-02-06T01:29:54.254 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:54.252+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.253401+0000) 2020-02-06T01:29:54.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:54.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.254237+0000) 2020-02-06T01:29:54.256 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:54.254+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.255066+0000) 2020-02-06T01:29:54.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:54.254+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.255404+0000) 2020-02-06T01:29:54.257 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:54.255+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.256274+0000) 2020-02-06T01:29:54.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:54.256+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.256660+0000) 2020-02-06T01:29:54.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:54.256+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.257245+0000) 2020-02-06T01:29:54.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:54.256+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.257531+0000) 2020-02-06T01:29:54.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:54.257+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.257859+0000) 2020-02-06T01:29:54.259 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:54.258+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.258737+0000) 2020-02-06T01:29:54.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:54.258+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.258912+0000) 2020-02-06T01:29:54.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:54.265+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:54.266369+0000) 2020-02-06T01:29:54.275 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:54.273+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205099 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:56.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:56.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:56.110081+0000) 2020-02-06T01:29:56.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:56.273+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:56.273773+0000) 2020-02-06T01:29:57.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:57.090+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:57.091870+0000) 2020-02-06T01:29:57.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:57.362+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:57.362799+0000) 2020-02-06T01:29:57.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:57.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:57.362946+0000) 2020-02-06T01:29:57.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:57.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:57.363026+0000) 2020-02-06T01:29:59.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:59.278+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.279910+0000) 2020-02-06T01:29:59.281 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:29:59.279+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.280762+0000) 2020-02-06T01:29:59.282 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:59.280+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.281132+0000) 2020-02-06T01:29:59.282 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:29:59.280+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.281570+0000) 2020-02-06T01:29:59.283 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:59.281+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.281675+0000) 2020-02-06T01:29:59.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:29:59.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.282272+0000) 2020-02-06T01:29:59.283 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:29:59.282+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.282680+0000) 2020-02-06T01:29:59.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:59.282+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.283252+0000) 2020-02-06T01:29:59.284 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:29:59.282+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.283264+0000) 2020-02-06T01:29:59.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:29:59.283+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.283657+0000) 2020-02-06T01:29:59.285 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:29:59.283+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.284035+0000) 2020-02-06T01:29:59.286 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:29:59.284+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.285085+0000) 2020-02-06T01:29:59.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:29:59.284+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.285286+0000) 2020-02-06T01:29:59.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:59.289+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.290158+0000) 2020-02-06T01:29:59.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:29:59.301+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:29:59.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:29:59.722+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.723183+0000) 2020-02-06T01:29:59.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:59.734+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.735418+0000) 2020-02-06T01:29:59.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:29:59.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.950365+0000) 2020-02-06T01:29:59.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:29:59.982+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:29:59.982576+0000) 2020-02-06T01:30:00.583 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:30:00.589 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:30:00.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:00.588+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:00.589008+0000) 2020-02-06T01:30:00.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:00.844+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:00.845229+0000) 2020-02-06T01:30:00.869 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:00.867+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:00.868427+0000) 2020-02-06T01:30:00.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:00.867+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:00.868427+0000) 2020-02-06T01:30:04.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:04.307+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.307724+0000) 2020-02-06T01:30:04.309 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:04.307+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.308380+0000) 2020-02-06T01:30:04.310 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:04.307+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.308843+0000) 2020-02-06T01:30:04.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:04.310+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.310663+0000) 2020-02-06T01:30:04.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:04.310+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.311634+0000) 2020-02-06T01:30:04.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:04.310+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.311974+0000) 2020-02-06T01:30:04.313 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:04.311+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.312002+0000) 2020-02-06T01:30:04.314 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:04.312+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.313071+0000) 2020-02-06T01:30:04.315 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:04.312+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.313401+0000) 2020-02-06T01:30:04.315 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:04.312+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.312509+0000) 2020-02-06T01:30:04.315 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:04.312+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.312701+0000) 2020-02-06T01:30:04.315 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:04.312+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.313319+0000) 2020-02-06T01:30:04.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:04.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.318132+0000) 2020-02-06T01:30:04.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:04.317+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:04.318474+0000) 2020-02-06T01:30:04.331 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:04.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205184 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:06.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:06.110+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:06.110321+0000) 2020-02-06T01:30:06.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:06.281+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:06.282146+0000) 2020-02-06T01:30:07.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:07.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:07.092131+0000) 2020-02-06T01:30:07.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:07.362+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:07.363049+0000) 2020-02-06T01:30:07.364 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:07.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:07.363213+0000) 2020-02-06T01:30:07.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:07.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:07.363310+0000) 2020-02-06T01:30:09.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:09.334+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.335171+0000) 2020-02-06T01:30:09.336 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:09.334+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.335691+0000) 2020-02-06T01:30:09.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:09.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.336708+0000) 2020-02-06T01:30:09.338 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:09.337+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.337476+0000) 2020-02-06T01:30:09.338 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:09.336+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.337721+0000) 2020-02-06T01:30:09.339 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:09.337+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.338439+0000) 2020-02-06T01:30:09.339 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:09.338+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.338726+0000) 2020-02-06T01:30:09.340 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:09.338+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.339287+0000) 2020-02-06T01:30:09.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:09.339+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.339664+0000) 2020-02-06T01:30:09.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:09.339+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.340138+0000) 2020-02-06T01:30:09.341 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:09.340+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.340946+0000) 2020-02-06T01:30:09.342 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:09.340+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.341293+0000) 2020-02-06T01:30:09.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:09.341+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.342132+0000) 2020-02-06T01:30:09.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:09.343+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.344638+0000) 2020-02-06T01:30:09.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:09.357+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205227 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:09.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:09.723+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.723418+0000) 2020-02-06T01:30:09.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:09.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.735650+0000) 2020-02-06T01:30:09.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:09.949+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.950566+0000) 2020-02-06T01:30:09.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:09.981+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:09.982780+0000) 2020-02-06T01:30:10.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:10.589+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:10.589279+0000) 2020-02-06T01:30:10.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:10.845+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:10.845435+0000) 2020-02-06T01:30:10.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:10.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:10.868713+0000) 2020-02-06T01:30:10.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:10.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:10.868713+0000) 2020-02-06T01:30:14.364 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:14.362+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.363456+0000) 2020-02-06T01:30:14.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:14.363+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.364600+0000) 2020-02-06T01:30:14.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:14.364+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.365528+0000) 2020-02-06T01:30:14.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:14.365+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.366142+0000) 2020-02-06T01:30:14.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:14.365+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.366518+0000) 2020-02-06T01:30:14.368 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:14.365+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.366929+0000) 2020-02-06T01:30:14.368 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:14.366+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.367236+0000) 2020-02-06T01:30:14.369 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:14.367+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.368463+0000) 2020-02-06T01:30:14.369 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:14.366+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.368149+0000) 2020-02-06T01:30:14.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:14.367+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.368836+0000) 2020-02-06T01:30:14.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:14.367+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.369202+0000) 2020-02-06T01:30:14.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:14.368+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.369680+0000) 2020-02-06T01:30:14.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:14.368+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.370416+0000) 2020-02-06T01:30:14.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:14.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:14.375552+0000) 2020-02-06T01:30:14.386 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:14.384+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205269 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:16.111 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:16.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:16.110574+0000) 2020-02-06T01:30:16.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:16.290+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:16.290190+0000) 2020-02-06T01:30:17.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:17.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:17.092361+0000) 2020-02-06T01:30:17.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:17.361+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:17.363287+0000) 2020-02-06T01:30:17.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:17.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:17.363480+0000) 2020-02-06T01:30:17.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:17.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:17.363619+0000) 2020-02-06T01:30:19.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:19.392+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.392276+0000) 2020-02-06T01:30:19.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:19.391+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.393144+0000) 2020-02-06T01:30:19.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:19.393+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.393524+0000) 2020-02-06T01:30:19.395 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:19.393+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.394038+0000) 2020-02-06T01:30:19.395 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:19.394+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.394690+0000) 2020-02-06T01:30:19.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:19.393+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.395140+0000) 2020-02-06T01:30:19.397 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:19.395+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.395499+0000) 2020-02-06T01:30:19.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:19.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.396023+0000) 2020-02-06T01:30:19.398 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:19.396+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.396249+0000) 2020-02-06T01:30:19.398 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:19.395+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.396824+0000) 2020-02-06T01:30:19.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:19.396+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.397487+0000) 2020-02-06T01:30:19.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:19.396+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.397872+0000) 2020-02-06T01:30:19.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:19.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.398122+0000) 2020-02-06T01:30:19.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:19.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.401767+0000) 2020-02-06T01:30:19.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:19.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205312 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:19.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:19.722+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.723683+0000) 2020-02-06T01:30:19.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:19.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.735852+0000) 2020-02-06T01:30:19.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:19.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.950823+0000) 2020-02-06T01:30:19.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:19.982+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:19.982988+0000) 2020-02-06T01:30:20.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:20.588+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:20.589531+0000) 2020-02-06T01:30:20.846 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:20.845+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:20.845666+0000) 2020-02-06T01:30:20.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:20.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:20.868970+0000) 2020-02-06T01:30:20.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:20.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:20.868968+0000) 2020-02-06T01:30:24.423 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:24.421+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.422433+0000) 2020-02-06T01:30:24.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:24.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.422958+0000) 2020-02-06T01:30:24.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:24.422+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.423417+0000) 2020-02-06T01:30:24.425 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:24.423+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.424335+0000) 2020-02-06T01:30:24.426 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:24.424+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.425040+0000) 2020-02-06T01:30:24.426 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:24.423+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.424945+0000) 2020-02-06T01:30:24.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:24.424+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.425651+0000) 2020-02-06T01:30:24.427 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:24.424+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.425596+0000) 2020-02-06T01:30:24.427 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:24.425+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.426304+0000) 2020-02-06T01:30:24.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:24.425+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.426713+0000) 2020-02-06T01:30:24.428 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:24.425+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.427197+0000) 2020-02-06T01:30:24.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:24.426+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.427552+0000) 2020-02-06T01:30:24.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:24.426+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.427734+0000) 2020-02-06T01:30:24.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:24.428+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:24.429122+0000) 2020-02-06T01:30:24.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:24.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205353 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:26.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:26.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:26.110835+0000) 2020-02-06T01:30:26.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:26.297+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:26.298469+0000) 2020-02-06T01:30:27.093 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:27.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:27.092591+0000) 2020-02-06T01:30:27.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:27.362+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:27.363528+0000) 2020-02-06T01:30:27.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:27.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:27.363743+0000) 2020-02-06T01:30:27.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:27.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:27.363945+0000) 2020-02-06T01:30:29.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:29.447+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.448644+0000) 2020-02-06T01:30:29.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:29.447+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.449400+0000) 2020-02-06T01:30:29.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:29.448+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.449761+0000) 2020-02-06T01:30:29.451 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:29.449+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.450263+0000) 2020-02-06T01:30:29.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:29.449+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.450573+0000) 2020-02-06T01:30:29.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:29.450+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.451169+0000) 2020-02-06T01:30:29.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:29.450+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.451553+0000) 2020-02-06T01:30:29.453 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:29.450+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.451968+0000) 2020-02-06T01:30:29.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:29.451+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.452470+0000) 2020-02-06T01:30:29.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:29.451+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.452793+0000) 2020-02-06T01:30:29.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:29.452+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.453626+0000) 2020-02-06T01:30:29.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:29.452+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.454232+0000) 2020-02-06T01:30:29.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:29.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.456564+0000) 2020-02-06T01:30:29.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:29.456+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.457558+0000) 2020-02-06T01:30:29.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:29.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:29.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:29.722+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.723909+0000) 2020-02-06T01:30:29.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:29.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.736120+0000) 2020-02-06T01:30:29.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:29.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.951040+0000) 2020-02-06T01:30:29.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:29.982+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:29.983191+0000) 2020-02-06T01:30:30.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:30.588+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:30.589722+0000) 2020-02-06T01:30:30.772 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:30:30.778 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:30:30.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:30.844+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:30.845959+0000) 2020-02-06T01:30:30.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:30.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:30.869225+0000) 2020-02-06T01:30:30.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:30.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:30.869218+0000) 2020-02-06T01:30:34.476 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:34.474+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.475571+0000) 2020-02-06T01:30:34.478 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:34.476+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.477318+0000) 2020-02-06T01:30:34.478 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:34.475+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.476860+0000) 2020-02-06T01:30:34.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:34.477+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.478477+0000) 2020-02-06T01:30:34.480 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:34.477+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479044+0000) 2020-02-06T01:30:34.480 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:34.477+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479338+0000) 2020-02-06T01:30:34.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:34.477+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479282+0000) 2020-02-06T01:30:34.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:34.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479663+0000) 2020-02-06T01:30:34.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:34.478+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479899+0000) 2020-02-06T01:30:34.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:34.478+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.479963+0000) 2020-02-06T01:30:34.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:34.478+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.480189+0000) 2020-02-06T01:30:34.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:34.478+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.480405+0000) 2020-02-06T01:30:34.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:34.483+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.484270+0000) 2020-02-06T01:30:34.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:34.484+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:34.485259+0000) 2020-02-06T01:30:34.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:34.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205437 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:36.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:36.110+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:36.111093+0000) 2020-02-06T01:30:36.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:36.305+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:36.306592+0000) 2020-02-06T01:30:37.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:37.091+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:37.092846+0000) 2020-02-06T01:30:37.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:37.362+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:37.363770+0000) 2020-02-06T01:30:37.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:37.362+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:37.363985+0000) 2020-02-06T01:30:37.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:37.362+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:37.364238+0000) 2020-02-06T01:30:39.506 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:39.503+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.504765+0000) 2020-02-06T01:30:39.507 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:39.505+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.506262+0000) 2020-02-06T01:30:39.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:39.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.506878+0000) 2020-02-06T01:30:39.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:39.506+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.507139+0000) 2020-02-06T01:30:39.508 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:39.506+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.507619+0000) 2020-02-06T01:30:39.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:39.506+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.508039+0000) 2020-02-06T01:30:39.509 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:39.507+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.508408+0000) 2020-02-06T01:30:39.510 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:39.507+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.508878+0000) 2020-02-06T01:30:39.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:39.507+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.509409+0000) 2020-02-06T01:30:39.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:39.508+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.509715+0000) 2020-02-06T01:30:39.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:39.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.509821+0000) 2020-02-06T01:30:39.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:39.507+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.509344+0000) 2020-02-06T01:30:39.512 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:39.508+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.509724+0000) 2020-02-06T01:30:39.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:39.513+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.514943+0000) 2020-02-06T01:30:39.527 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:39.525+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205479 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:39.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:39.723+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.724133+0000) 2020-02-06T01:30:39.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:39.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.736333+0000) 2020-02-06T01:30:39.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:39.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.951280+0000) 2020-02-06T01:30:39.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:39.982+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:39.983395+0000) 2020-02-06T01:30:40.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:40.589+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:40.589978+0000) 2020-02-06T01:30:40.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:40.845+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:40.846200+0000) 2020-02-06T01:30:40.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:40.868+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:40.869479+0000) 2020-02-06T01:30:40.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:40.868+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:40.869432+0000) 2020-02-06T01:30:44.533 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:44.531+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.532262+0000) 2020-02-06T01:30:44.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:44.531+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.533010+0000) 2020-02-06T01:30:44.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:44.532+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.533736+0000) 2020-02-06T01:30:44.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:44.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.534848+0000) 2020-02-06T01:30:44.536 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:44.533+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.534791+0000) 2020-02-06T01:30:44.536 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:44.533+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.535372+0000) 2020-02-06T01:30:44.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:44.534+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.535794+0000) 2020-02-06T01:30:44.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:44.535+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.536700+0000) 2020-02-06T01:30:44.538 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:44.535+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.536494+0000) 2020-02-06T01:30:44.538 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:44.535+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.537355+0000) 2020-02-06T01:30:44.539 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:44.537+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.538522+0000) 2020-02-06T01:30:44.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:44.537+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.538613+0000) 2020-02-06T01:30:44.540 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:44.537+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.539076+0000) 2020-02-06T01:30:44.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:44.542+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:44.543741+0000) 2020-02-06T01:30:44.556 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:44.553+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205519 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:46.112 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:46.109+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:46.111377+0000) 2020-02-06T01:30:46.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:46.314+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:46.314717+0000) 2020-02-06T01:30:47.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:47.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:47.093125+0000) 2020-02-06T01:30:47.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:47.363+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:47.364068+0000) 2020-02-06T01:30:47.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:47.363+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:47.364245+0000) 2020-02-06T01:30:47.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:47.363+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:47.364485+0000) 2020-02-06T01:30:49.561 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:49.558+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.559733+0000) 2020-02-06T01:30:49.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:49.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.561558+0000) 2020-02-06T01:30:49.565 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:49.562+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.563796+0000) 2020-02-06T01:30:49.565 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:49.562+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.563973+0000) 2020-02-06T01:30:49.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:49.562+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.564247+0000) 2020-02-06T01:30:49.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:49.563+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.564486+0000) 2020-02-06T01:30:49.567 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:49.563+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.564601+0000) 2020-02-06T01:30:49.567 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:49.563+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.564897+0000) 2020-02-06T01:30:49.567 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:49.563+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.565320+0000) 2020-02-06T01:30:49.568 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:49.564+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.565735+0000) 2020-02-06T01:30:49.568 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:49.564+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.565258+0000) 2020-02-06T01:30:49.568 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:49.564+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.565623+0000) 2020-02-06T01:30:49.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:49.564+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.565850+0000) 2020-02-06T01:30:49.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:49.567+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.568667+0000) 2020-02-06T01:30:49.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:49.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:49.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:49.723+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.724352+0000) 2020-02-06T01:30:49.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:49.735+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.736544+0000) 2020-02-06T01:30:49.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:49.950+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.951495+0000) 2020-02-06T01:30:49.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:49.983+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:49.983634+0000) 2020-02-06T01:30:50.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:50.589+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:50.590232+0000) 2020-02-06T01:30:50.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:50.845+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:50.846427+0000) 2020-02-06T01:30:50.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:50.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:50.869706+0000) 2020-02-06T01:30:50.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:50.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:50.869675+0000) 2020-02-06T01:30:54.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:54.587+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.588383+0000) 2020-02-06T01:30:54.590 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:54.588+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.589277+0000) 2020-02-06T01:30:54.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:54.588+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.589803+0000) 2020-02-06T01:30:54.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:54.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.590250+0000) 2020-02-06T01:30:54.591 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:54.588+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.590161+0000) 2020-02-06T01:30:54.592 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:54.589+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.590597+0000) 2020-02-06T01:30:54.592 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:54.590+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.591488+0000) 2020-02-06T01:30:54.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:54.590+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.591539+0000) 2020-02-06T01:30:54.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:54.591+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.592206+0000) 2020-02-06T01:30:54.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:54.590+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.592380+0000) 2020-02-06T01:30:54.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:54.590+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.592175+0000) 2020-02-06T01:30:54.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:54.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.592969+0000) 2020-02-06T01:30:54.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:54.591+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.593108+0000) 2020-02-06T01:30:54.597 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:54.595+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:54.596871+0000) 2020-02-06T01:30:54.609 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:54.607+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205603 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:56.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:56.111+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:56.111678+0000) 2020-02-06T01:30:56.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:56.322+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:56.322558+0000) 2020-02-06T01:30:57.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:57.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:57.093361+0000) 2020-02-06T01:30:57.365 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:57.363+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:57.364325+0000) 2020-02-06T01:30:57.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:57.363+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:57.364538+0000) 2020-02-06T01:30:57.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:57.363+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:57.364720+0000) 2020-02-06T01:30:59.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:59.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.614952+0000) 2020-02-06T01:30:59.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:30:59.613+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.615358+0000) 2020-02-06T01:30:59.617 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:59.615+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.616533+0000) 2020-02-06T01:30:59.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:30:59.616+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.617039+0000) 2020-02-06T01:30:59.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:30:59.616+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.617711+0000) 2020-02-06T01:30:59.619 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:30:59.617+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.618912+0000) 2020-02-06T01:30:59.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:30:59.616+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.618330+0000) 2020-02-06T01:30:59.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:59.618+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.619331+0000) 2020-02-06T01:30:59.621 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:30:59.618+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.619555+0000) 2020-02-06T01:30:59.621 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:30:59.618+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.619858+0000) 2020-02-06T01:30:59.621 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:30:59.619+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.620401+0000) 2020-02-06T01:30:59.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:59.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.620698+0000) 2020-02-06T01:30:59.622 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:30:59.619+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.620877+0000) 2020-02-06T01:30:59.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:59.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.624659+0000) 2020-02-06T01:30:59.636 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:30:59.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205645 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:30:59.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:30:59.724+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.724579+0000) 2020-02-06T01:30:59.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:59.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.736764+0000) 2020-02-06T01:30:59.952 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:30:59.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.951723+0000) 2020-02-06T01:30:59.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:30:59.982+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:30:59.983837+0000) 2020-02-06T01:31:00.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:00.589+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:00.590477+0000) 2020-02-06T01:31:00.847 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:00.846+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:00.846659+0000) 2020-02-06T01:31:00.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:00.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:00.869955+0000) 2020-02-06T01:31:00.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:00.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:00.869945+0000) 2020-02-06T01:31:00.934 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:31:00.941 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:31:04.642 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:04.640+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.641426+0000) 2020-02-06T01:31:04.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:04.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.642349+0000) 2020-02-06T01:31:04.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:04.641+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.642745+0000) 2020-02-06T01:31:04.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:04.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.643192+0000) 2020-02-06T01:31:04.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:04.642+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.643570+0000) 2020-02-06T01:31:04.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:04.643+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.644291+0000) 2020-02-06T01:31:04.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:04.643+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.644404+0000) 2020-02-06T01:31:04.646 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:04.644+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.645153+0000) 2020-02-06T01:31:04.646 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:04.643+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.644821+0000) 2020-02-06T01:31:04.647 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:04.645+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.645718+0000) 2020-02-06T01:31:04.647 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:04.645+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.645756+0000) 2020-02-06T01:31:04.648 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:04.647+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.647482+0000) 2020-02-06T01:31:04.649 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:04.647+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.648029+0000) 2020-02-06T01:31:04.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:04.651+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:04.652378+0000) 2020-02-06T01:31:04.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:04.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205684 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:06.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:06.111+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:06.111920+0000) 2020-02-06T01:31:06.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:06.329+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:06.329556+0000) 2020-02-06T01:31:07.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:07.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:07.093594+0000) 2020-02-06T01:31:07.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:07.364+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:07.364561+0000) 2020-02-06T01:31:07.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:07.364+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:07.364788+0000) 2020-02-06T01:31:07.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:07.364+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:07.364987+0000) 2020-02-06T01:31:09.672 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:09.669+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.670676+0000) 2020-02-06T01:31:09.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:09.670+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.671206+0000) 2020-02-06T01:31:09.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:09.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.672069+0000) 2020-02-06T01:31:09.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:09.672+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.672574+0000) 2020-02-06T01:31:09.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:09.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.673040+0000) 2020-02-06T01:31:09.674 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:09.672+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.673487+0000) 2020-02-06T01:31:09.675 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:09.672+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.673775+0000) 2020-02-06T01:31:09.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:09.673+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.674225+0000) 2020-02-06T01:31:09.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:09.673+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.674404+0000) 2020-02-06T01:31:09.676 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:09.674+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.674751+0000) 2020-02-06T01:31:09.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:09.675+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.676115+0000) 2020-02-06T01:31:09.677 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:09.676+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.676495+0000) 2020-02-06T01:31:09.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:09.676+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.677111+0000) 2020-02-06T01:31:09.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:09.681+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.682648+0000) 2020-02-06T01:31:09.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:09.693+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205727 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:09.725 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:09.724+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.724787+0000) 2020-02-06T01:31:09.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:09.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.736989+0000) 2020-02-06T01:31:09.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:09.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.951927+0000) 2020-02-06T01:31:09.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:09.983+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:09.984102+0000) 2020-02-06T01:31:10.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:10.590+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:10.590706+0000) 2020-02-06T01:31:10.848 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:10.846+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:10.846870+0000) 2020-02-06T01:31:10.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:10.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:10.870181+0000) 2020-02-06T01:31:10.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:10.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:10.870268+0000) 2020-02-06T01:31:14.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:14.699+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.699674+0000) 2020-02-06T01:31:14.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:14.700+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.700796+0000) 2020-02-06T01:31:14.702 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:14.700+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.701320+0000) 2020-02-06T01:31:14.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:14.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.701843+0000) 2020-02-06T01:31:14.703 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:14.701+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.702176+0000) 2020-02-06T01:31:14.704 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:14.703+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.703436+0000) 2020-02-06T01:31:14.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:14.703+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.703296+0000) 2020-02-06T01:31:14.705 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:14.703+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.703596+0000) 2020-02-06T01:31:14.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:14.703+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.703995+0000) 2020-02-06T01:31:14.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:14.704+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.704338+0000) 2020-02-06T01:31:14.706 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:14.704+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.704971+0000) 2020-02-06T01:31:14.706 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:14.704+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.705323+0000) 2020-02-06T01:31:14.707 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:14.705+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.705415+0000) 2020-02-06T01:31:14.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:14.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:14.709200+0000) 2020-02-06T01:31:14.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:14.721+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205766 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:16.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:16.111+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:16.112187+0000) 2020-02-06T01:31:16.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:16.336+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:16.337964+0000) 2020-02-06T01:31:17.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:17.092+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:17.093825+0000) 2020-02-06T01:31:17.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:17.364+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:17.364820+0000) 2020-02-06T01:31:17.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:17.364+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:17.365042+0000) 2020-02-06T01:31:17.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:17.364+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:17.365253+0000) 2020-02-06T01:31:19.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:19.727+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.728819+0000) 2020-02-06T01:31:19.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:19.728+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.729293+0000) 2020-02-06T01:31:19.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:19.728+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.729469+0000) 2020-02-06T01:31:19.731 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:19.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.729733+0000) 2020-02-06T01:31:19.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:19.729+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.729802+0000) 2020-02-06T01:31:19.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:19.729+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.730363+0000) 2020-02-06T01:31:19.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:19.730+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.730830+0000) 2020-02-06T01:31:19.732 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:19.729+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.730970+0000) 2020-02-06T01:31:19.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:19.731+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.731883+0000) 2020-02-06T01:31:19.734 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:19.732+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.733209+0000) 2020-02-06T01:31:19.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:19.732+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.733619+0000) 2020-02-06T01:31:19.735 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:19.733+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.734045+0000) 2020-02-06T01:31:19.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:19.733+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.734307+0000) 2020-02-06T01:31:19.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:19.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.737207+0000) 2020-02-06T01:31:19.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:19.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.739008+0000) 2020-02-06T01:31:19.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:19.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205808 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:19.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:19.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.952141+0000) 2020-02-06T01:31:19.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:19.983+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:19.984358+0000) 2020-02-06T01:31:20.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:20.590+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:20.590924+0000) 2020-02-06T01:31:20.848 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:20.846+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:20.847144+0000) 2020-02-06T01:31:20.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:20.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:20.870405+0000) 2020-02-06T01:31:20.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:20.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:20.870532+0000) 2020-02-06T01:31:24.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:24.753+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.754469+0000) 2020-02-06T01:31:24.756 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:24.754+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.755256+0000) 2020-02-06T01:31:24.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:24.755+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.756065+0000) 2020-02-06T01:31:24.757 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:24.755+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.756709+0000) 2020-02-06T01:31:24.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:24.756+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.757017+0000) 2020-02-06T01:31:24.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:24.756+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.757482+0000) 2020-02-06T01:31:24.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:24.756+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.757895+0000) 2020-02-06T01:31:24.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:24.757+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.758246+0000) 2020-02-06T01:31:24.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:24.758+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.758505+0000) 2020-02-06T01:31:24.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:24.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.758891+0000) 2020-02-06T01:31:24.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:24.758+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.759048+0000) 2020-02-06T01:31:24.761 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:24.759+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.760378+0000) 2020-02-06T01:31:24.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:24.760+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.760999+0000) 2020-02-06T01:31:24.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:24.766+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:24.767213+0000) 2020-02-06T01:31:24.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:24.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205846 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:26.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:26.112+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:26.112479+0000) 2020-02-06T01:31:26.350 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:26.348+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:26.349598+0000) 2020-02-06T01:31:26.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:26.475+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:26.475929+0000) 2020-02-06T01:31:27.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:27.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:27.094067+0000) 2020-02-06T01:31:27.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:27.364+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:27.365071+0000) 2020-02-06T01:31:27.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:27.364+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:27.365300+0000) 2020-02-06T01:31:27.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:27.365+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:27.365542+0000) 2020-02-06T01:31:29.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:29.736+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.737421+0000) 2020-02-06T01:31:29.784 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:29.782+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.783464+0000) 2020-02-06T01:31:29.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:29.783+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.784108+0000) 2020-02-06T01:31:29.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:29.783+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.784427+0000) 2020-02-06T01:31:29.786 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:29.784+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.785394+0000) 2020-02-06T01:31:29.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:29.784+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.785874+0000) 2020-02-06T01:31:29.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:29.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.786274+0000) 2020-02-06T01:31:29.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:29.785+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.786509+0000) 2020-02-06T01:31:29.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:29.785+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.786950+0000) 2020-02-06T01:31:29.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:29.786+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.787476+0000) 2020-02-06T01:31:29.789 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:29.787+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.788105+0000) 2020-02-06T01:31:29.789 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:29.788+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.788706+0000) 2020-02-06T01:31:29.790 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:29.788+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.789118+0000) 2020-02-06T01:31:29.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:29.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.789341+0000) 2020-02-06T01:31:29.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:29.790+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.791413+0000) 2020-02-06T01:31:29.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:29.804+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205889 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:29.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:29.951+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.952407+0000) 2020-02-06T01:31:29.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:29.983+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:29.984573+0000) 2020-02-06T01:31:30.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:30.590+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:30.591201+0000) 2020-02-06T01:31:30.848 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:30.846+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:30.847407+0000) 2020-02-06T01:31:30.871 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:30.869+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:30.870635+0000) 2020-02-06T01:31:30.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:30.869+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:30.870806+0000) 2020-02-06T01:31:31.106 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:31:31.112 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:31:34.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:34.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.811513+0000) 2020-02-06T01:31:34.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:34.811+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.812660+0000) 2020-02-06T01:31:34.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:34.812+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813102+0000) 2020-02-06T01:31:34.814 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:34.812+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813545+0000) 2020-02-06T01:31:34.814 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:34.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813659+0000) 2020-02-06T01:31:34.815 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:34.813+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813568+0000) 2020-02-06T01:31:34.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:34.813+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813724+0000) 2020-02-06T01:31:34.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:34.813+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.813883+0000) 2020-02-06T01:31:34.816 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:34.814+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.815296+0000) 2020-02-06T01:31:34.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:34.814+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.814695+0000) 2020-02-06T01:31:34.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:34.814+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.815026+0000) 2020-02-06T01:31:34.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:34.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.815367+0000) 2020-02-06T01:31:34.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:34.818+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.819100+0000) 2020-02-06T01:31:34.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:34.818+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:34.819476+0000) 2020-02-06T01:31:34.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:34.831+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205928 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:36.114 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:36.112+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:36.112719+0000) 2020-02-06T01:31:36.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:36.357+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:36.357813+0000) 2020-02-06T01:31:36.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:36.476+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:36.476193+0000) 2020-02-06T01:31:37.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:37.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:37.094325+0000) 2020-02-06T01:31:37.366 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:37.365+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:37.365327+0000) 2020-02-06T01:31:37.367 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:37.365+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:37.365615+0000) 2020-02-06T01:31:37.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:37.365+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:37.365789+0000) 2020-02-06T01:31:39.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:39.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.737660+0000) 2020-02-06T01:31:39.839 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:39.836+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.837975+0000) 2020-02-06T01:31:39.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:39.837+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.838466+0000) 2020-02-06T01:31:39.840 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:39.837+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.838771+0000) 2020-02-06T01:31:39.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:39.838+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.839177+0000) 2020-02-06T01:31:39.840 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:39.838+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.839569+0000) 2020-02-06T01:31:39.841 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:39.839+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.840375+0000) 2020-02-06T01:31:39.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:39.840+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.841429+0000) 2020-02-06T01:31:39.842 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:39.841+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.841729+0000) 2020-02-06T01:31:39.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:39.841+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.842270+0000) 2020-02-06T01:31:39.844 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:39.842+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.842967+0000) 2020-02-06T01:31:39.844 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:39.842+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.843108+0000) 2020-02-06T01:31:39.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:39.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.843757+0000) 2020-02-06T01:31:39.845 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:39.843+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.844205+0000) 2020-02-06T01:31:39.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:39.846+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.847383+0000) 2020-02-06T01:31:39.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:39.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 205970 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:39.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:39.952+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.952634+0000) 2020-02-06T01:31:39.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:39.983+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:39.984800+0000) 2020-02-06T01:31:40.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:40.591+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:40.591462+0000) 2020-02-06T01:31:40.848 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:40.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:40.847642+0000) 2020-02-06T01:31:40.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:40.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:40.870931+0000) 2020-02-06T01:31:40.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:40.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:40.871140+0000) 2020-02-06T01:31:44.875 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:44.865+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.866219+0000) 2020-02-06T01:31:44.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:44.865+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.866712+0000) 2020-02-06T01:31:44.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:44.866+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.867654+0000) 2020-02-06T01:31:44.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:44.866+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.867940+0000) 2020-02-06T01:31:44.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:44.867+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.868550+0000) 2020-02-06T01:31:44.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:44.867+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.868939+0000) 2020-02-06T01:31:44.878 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:44.868+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.869567+0000) 2020-02-06T01:31:44.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:44.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.870117+0000) 2020-02-06T01:31:44.880 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:44.870+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.871978+0000) 2020-02-06T01:31:44.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:44.871+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.872406+0000) 2020-02-06T01:31:44.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:44.870+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.872390+0000) 2020-02-06T01:31:44.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:44.871+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.872729+0000) 2020-02-06T01:31:44.881 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:44.871+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.872887+0000) 2020-02-06T01:31:44.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:44.876+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:44.877183+0000) 2020-02-06T01:31:44.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:44.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206008 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:46.114 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:46.111+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:46.112986+0000) 2020-02-06T01:31:46.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:46.365+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:46.365868+0000) 2020-02-06T01:31:46.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:46.475+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:46.476463+0000) 2020-02-06T01:31:47.095 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:47.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:47.094578+0000) 2020-02-06T01:31:47.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:47.364+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:47.365578+0000) 2020-02-06T01:31:47.367 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:47.364+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:47.365868+0000) 2020-02-06T01:31:47.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:47.364+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:47.366083+0000) 2020-02-06T01:31:49.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:49.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.737914+0000) 2020-02-06T01:31:49.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:49.892+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.893874+0000) 2020-02-06T01:31:49.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:49.893+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.894428+0000) 2020-02-06T01:31:49.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:49.894+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.896110+0000) 2020-02-06T01:31:49.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:49.894+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.896193+0000) 2020-02-06T01:31:49.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:49.895+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.896665+0000) 2020-02-06T01:31:49.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:49.896+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.897648+0000) 2020-02-06T01:31:49.899 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:49.896+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.898087+0000) 2020-02-06T01:31:49.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:49.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.898519+0000) 2020-02-06T01:31:49.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:49.897+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.898816+0000) 2020-02-06T01:31:49.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:49.898+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.899963+0000) 2020-02-06T01:31:49.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:49.899+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.900547+0000) 2020-02-06T01:31:49.902 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:49.899+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.900510+0000) 2020-02-06T01:31:49.903 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:49.899+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.900924+0000) 2020-02-06T01:31:49.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:49.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.901508+0000) 2020-02-06T01:31:49.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:49.915+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206051 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:49.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:49.952+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.952898+0000) 2020-02-06T01:31:49.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:49.984+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:49.985027+0000) 2020-02-06T01:31:50.592 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:50.590+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:50.591712+0000) 2020-02-06T01:31:50.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:50.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:50.847895+0000) 2020-02-06T01:31:50.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:50.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:50.871172+0000) 2020-02-06T01:31:50.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:50.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:50.871453+0000) 2020-02-06T01:31:54.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:54.922+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.923259+0000) 2020-02-06T01:31:54.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:54.922+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.924091+0000) 2020-02-06T01:31:54.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:54.922+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.923936+0000) 2020-02-06T01:31:54.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:54.923+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.924925+0000) 2020-02-06T01:31:54.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:54.924+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.925129+0000) 2020-02-06T01:31:54.927 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:54.924+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.925706+0000) 2020-02-06T01:31:54.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:54.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.925581+0000) 2020-02-06T01:31:54.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:54.924+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.925697+0000) 2020-02-06T01:31:54.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:54.925+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.926686+0000) 2020-02-06T01:31:54.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:54.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.926671+0000) 2020-02-06T01:31:54.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:54.925+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.926780+0000) 2020-02-06T01:31:54.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:54.925+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.926793+0000) 2020-02-06T01:31:54.929 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:54.925+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.927104+0000) 2020-02-06T01:31:54.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:54.930+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:54.931925+0000) 2020-02-06T01:31:54.945 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:54.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:56.114 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:56.112+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:56.113245+0000) 2020-02-06T01:31:56.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:56.373+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:56.374114+0000) 2020-02-06T01:31:56.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:56.475+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:56.476713+0000) 2020-02-06T01:31:57.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:57.093+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:57.094836+0000) 2020-02-06T01:31:57.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:57.364+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:57.365838+0000) 2020-02-06T01:31:57.367 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:57.364+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:57.366101+0000) 2020-02-06T01:31:57.367 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:57.364+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:57.366322+0000) 2020-02-06T01:31:59.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:59.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.738154+0000) 2020-02-06T01:31:59.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:31:59.949+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.950609+0000) 2020-02-06T01:31:59.952 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:31:59.950+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.951723+0000) 2020-02-06T01:31:59.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:59.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.952154+0000) 2020-02-06T01:31:59.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:31:59.951+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.952388+0000) 2020-02-06T01:31:59.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:31:59.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.953170+0000) 2020-02-06T01:31:59.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:31:59.951+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.953141+0000) 2020-02-06T01:31:59.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:31:59.952+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.953507+0000) 2020-02-06T01:31:59.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:31:59.953+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.954858+0000) 2020-02-06T01:31:59.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:59.954+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.955164+0000) 2020-02-06T01:31:59.956 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:31:59.954+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.955495+0000) 2020-02-06T01:31:59.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:31:59.954+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.956136+0000) 2020-02-06T01:31:59.957 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:31:59.955+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.956454+0000) 2020-02-06T01:31:59.961 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:31:59.956+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.957740+0000) 2020-02-06T01:31:59.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:59.961+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.962740+0000) 2020-02-06T01:31:59.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:31:59.971+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:31:59.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:31:59.984+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:31:59.985230+0000) 2020-02-06T01:32:00.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:00.591+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:00.591924+0000) 2020-02-06T01:32:00.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:00.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:00.848144+0000) 2020-02-06T01:32:00.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:00.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:00.871416+0000) 2020-02-06T01:32:00.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:00.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:00.871621+0000) 2020-02-06T01:32:01.235 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:32:01.242 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:32:04.980 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:04.978+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.979563+0000) 2020-02-06T01:32:04.981 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:04.978+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.979723+0000) 2020-02-06T01:32:04.981 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:04.979+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.980214+0000) 2020-02-06T01:32:04.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:04.979+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.980664+0000) 2020-02-06T01:32:04.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:04.980+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.981129+0000) 2020-02-06T01:32:04.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:04.980+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.981480+0000) 2020-02-06T01:32:04.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:04.980+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.981992+0000) 2020-02-06T01:32:04.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:04.982+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.983451+0000) 2020-02-06T01:32:04.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:04.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.983678+0000) 2020-02-06T01:32:04.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:04.982+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.984173+0000) 2020-02-06T01:32:04.986 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:04.983+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.984682+0000) 2020-02-06T01:32:04.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:04.983+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.985062+0000) 2020-02-06T01:32:04.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:04.983+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.985221+0000) 2020-02-06T01:32:04.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:04.987+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:04.988302+0000) 2020-02-06T01:32:05.002 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:05.000+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206170 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:06.114 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:06.112+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:06.113513+0000) 2020-02-06T01:32:06.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:06.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:06.382355+0000) 2020-02-06T01:32:06.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:06.476+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:06.476974+0000) 2020-02-06T01:32:06.704 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:06.703+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:06.703697+0000) 2020-02-06T01:32:07.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:07.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:07.095056+0000) 2020-02-06T01:32:07.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:07.365+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:07.366091+0000) 2020-02-06T01:32:07.367 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:07.365+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:07.366359+0000) 2020-02-06T01:32:07.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:07.365+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:07.366624+0000) 2020-02-06T01:32:09.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:09.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:09.738380+0000) 2020-02-06T01:32:09.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:09.984+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:09.985451+0000) 2020-02-06T01:32:10.007 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:10.005+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.006464+0000) 2020-02-06T01:32:10.008 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:10.006+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.007811+0000) 2020-02-06T01:32:10.009 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:10.007+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.008639+0000) 2020-02-06T01:32:10.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:10.008+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.009392+0000) 2020-02-06T01:32:10.010 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:10.008+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.009703+0000) 2020-02-06T01:32:10.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:10.009+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.010248+0000) 2020-02-06T01:32:10.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:10.009+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.010440+0000) 2020-02-06T01:32:10.012 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:10.009+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.011224+0000) 2020-02-06T01:32:10.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:10.010+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.011504+0000) 2020-02-06T01:32:10.013 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:10.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.011767+0000) 2020-02-06T01:32:10.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:10.012+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.013496+0000) 2020-02-06T01:32:10.015 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:10.012+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.014056+0000) 2020-02-06T01:32:10.015 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:10.012+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.014301+0000) 2020-02-06T01:32:10.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:10.018+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.019241+0000) 2020-02-06T01:32:10.031 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:10.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:10.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:10.591+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.592151+0000) 2020-02-06T01:32:10.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:10.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.848356+0000) 2020-02-06T01:32:10.872 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:10.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.871621+0000) 2020-02-06T01:32:10.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:10.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:10.871787+0000) 2020-02-06T01:32:15.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:15.034+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.035362+0000) 2020-02-06T01:32:15.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:15.034+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.036131+0000) 2020-02-06T01:32:15.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:15.036+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.037104+0000) 2020-02-06T01:32:15.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:15.036+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.037852+0000) 2020-02-06T01:32:15.039 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:15.037+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.038268+0000) 2020-02-06T01:32:15.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:15.037+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.038697+0000) 2020-02-06T01:32:15.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:15.038+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.039399+0000) 2020-02-06T01:32:15.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:15.039+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.040387+0000) 2020-02-06T01:32:15.041 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:15.038+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.040103+0000) 2020-02-06T01:32:15.042 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:15.040+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.041411+0000) 2020-02-06T01:32:15.042 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:15.040+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.041685+0000) 2020-02-06T01:32:15.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:15.040+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.041778+0000) 2020-02-06T01:32:15.043 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:15.040+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.042290+0000) 2020-02-06T01:32:15.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:15.044+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:15.046070+0000) 2020-02-06T01:32:15.058 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:15.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206252 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:16.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:16.113+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:16.113814+0000) 2020-02-06T01:32:16.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:16.389+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:16.390604+0000) 2020-02-06T01:32:16.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:16.476+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:16.477234+0000) 2020-02-06T01:32:16.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:16.703+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:16.703971+0000) 2020-02-06T01:32:17.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:17.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:17.095286+0000) 2020-02-06T01:32:17.367 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:17.365+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:17.366325+0000) 2020-02-06T01:32:17.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:17.365+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:17.366586+0000) 2020-02-06T01:32:17.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:17.365+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:17.366865+0000) 2020-02-06T01:32:19.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:19.737+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:19.738615+0000) 2020-02-06T01:32:19.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:19.984+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:19.985683+0000) 2020-02-06T01:32:20.065 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:20.062+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.064094+0000) 2020-02-06T01:32:20.065 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:20.063+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.064477+0000) 2020-02-06T01:32:20.066 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:20.064+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.065221+0000) 2020-02-06T01:32:20.066 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:20.064+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.065593+0000) 2020-02-06T01:32:20.067 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:20.063+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.065399+0000) 2020-02-06T01:32:20.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:20.065+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.066165+0000) 2020-02-06T01:32:20.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:20.065+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.066745+0000) 2020-02-06T01:32:20.068 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:20.065+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.066527+0000) 2020-02-06T01:32:20.068 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:20.066+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.067788+0000) 2020-02-06T01:32:20.069 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:20.066+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.067750+0000) 2020-02-06T01:32:20.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:20.066+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.068301+0000) 2020-02-06T01:32:20.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:20.067+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.068447+0000) 2020-02-06T01:32:20.070 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:20.067+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.068248+0000) 2020-02-06T01:32:20.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:20.071+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.072369+0000) 2020-02-06T01:32:20.085 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:20.082+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206294 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:20.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:20.591+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.592377+0000) 2020-02-06T01:32:20.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:20.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.848594+0000) 2020-02-06T01:32:20.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:20.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.871832+0000) 2020-02-06T01:32:20.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:20.870+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:20.872007+0000) 2020-02-06T01:32:25.089 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:25.087+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.088521+0000) 2020-02-06T01:32:25.090 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:25.087+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.088939+0000) 2020-02-06T01:32:25.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:25.089+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.090418+0000) 2020-02-06T01:32:25.092 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:25.090+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.091150+0000) 2020-02-06T01:32:25.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:25.090+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.091622+0000) 2020-02-06T01:32:25.093 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:25.090+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.092102+0000) 2020-02-06T01:32:25.093 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:25.091+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.092629+0000) 2020-02-06T01:32:25.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:25.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.093072+0000) 2020-02-06T01:32:25.094 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:25.092+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.093830+0000) 2020-02-06T01:32:25.095 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:25.092+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.093966+0000) 2020-02-06T01:32:25.096 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:25.093+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.095062+0000) 2020-02-06T01:32:25.096 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:25.094+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.095281+0000) 2020-02-06T01:32:25.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:25.094+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.095842+0000) 2020-02-06T01:32:25.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:25.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:25.101088+0000) 2020-02-06T01:32:25.114 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:25.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:26.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:26.113+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:26.114088+0000) 2020-02-06T01:32:26.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:26.400+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:26.401117+0000) 2020-02-06T01:32:26.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:26.476+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:26.477497+0000) 2020-02-06T01:32:26.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:26.703+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:26.704210+0000) 2020-02-06T01:32:27.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:27.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:27.095488+0000) 2020-02-06T01:32:27.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:27.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:27.366579+0000) 2020-02-06T01:32:27.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:27.366+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:27.366850+0000) 2020-02-06T01:32:27.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:27.366+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:27.367118+0000) 2020-02-06T01:32:29.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:29.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:29.738867+0000) 2020-02-06T01:32:29.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:29.985+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:29.985912+0000) 2020-02-06T01:32:30.121 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:30.118+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.119950+0000) 2020-02-06T01:32:30.121 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:30.119+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.120569+0000) 2020-02-06T01:32:30.122 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:30.120+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.121217+0000) 2020-02-06T01:32:30.122 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:30.120+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.121233+0000) 2020-02-06T01:32:30.123 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:30.120+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.121752+0000) 2020-02-06T01:32:30.123 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:30.120+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.122228+0000) 2020-02-06T01:32:30.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:30.121+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.122430+0000) 2020-02-06T01:32:30.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:30.122+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.123568+0000) 2020-02-06T01:32:30.124 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:30.122+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.123913+0000) 2020-02-06T01:32:30.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:30.122+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.124231+0000) 2020-02-06T01:32:30.125 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:30.123+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.124952+0000) 2020-02-06T01:32:30.126 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:30.124+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.125275+0000) 2020-02-06T01:32:30.126 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:30.123+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.125154+0000) 2020-02-06T01:32:30.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:30.128+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.129205+0000) 2020-02-06T01:32:30.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:30.138+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:30.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:30.592+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.592620+0000) 2020-02-06T01:32:30.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:30.848+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.848826+0000) 2020-02-06T01:32:30.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:30.870+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.872035+0000) 2020-02-06T01:32:30.873 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:30.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:30.872235+0000) 2020-02-06T01:32:31.369 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:32:31.375 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:32:35.146 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:35.144+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.145385+0000) 2020-02-06T01:32:35.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:35.145+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.146282+0000) 2020-02-06T01:32:35.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:35.145+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.146827+0000) 2020-02-06T01:32:35.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:35.146+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.147182+0000) 2020-02-06T01:32:35.148 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:35.147+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.147694+0000) 2020-02-06T01:32:35.149 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:35.147+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.148123+0000) 2020-02-06T01:32:35.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:35.147+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.148492+0000) 2020-02-06T01:32:35.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:35.148+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.149383+0000) 2020-02-06T01:32:35.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:35.148+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.149978+0000) 2020-02-06T01:32:35.151 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:35.150+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.150867+0000) 2020-02-06T01:32:35.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:35.150+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.151130+0000) 2020-02-06T01:32:35.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:35.150+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.151447+0000) 2020-02-06T01:32:35.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:35.151+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.151589+0000) 2020-02-06T01:32:35.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:35.153+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:35.154176+0000) 2020-02-06T01:32:35.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:35.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206414 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:36.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:36.113+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:36.114327+0000) 2020-02-06T01:32:36.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:36.408+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:36.409073+0000) 2020-02-06T01:32:36.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:36.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:36.477755+0000) 2020-02-06T01:32:36.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:36.703+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:36.704459+0000) 2020-02-06T01:32:37.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:37.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:37.095727+0000) 2020-02-06T01:32:37.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:37.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:37.366873+0000) 2020-02-06T01:32:37.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:37.366+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:37.367089+0000) 2020-02-06T01:32:37.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:37.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:37.367423+0000) 2020-02-06T01:32:39.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:39.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:39.739131+0000) 2020-02-06T01:32:39.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:39.985+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:39.986143+0000) 2020-02-06T01:32:40.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:40.171+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.172945+0000) 2020-02-06T01:32:40.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:40.172+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.173767+0000) 2020-02-06T01:32:40.175 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:40.173+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.174287+0000) 2020-02-06T01:32:40.175 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:40.173+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.174673+0000) 2020-02-06T01:32:40.176 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:40.175+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.175485+0000) 2020-02-06T01:32:40.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:40.175+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.176180+0000) 2020-02-06T01:32:40.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:40.176+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.177161+0000) 2020-02-06T01:32:40.179 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:40.177+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.177764+0000) 2020-02-06T01:32:40.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:40.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.178731+0000) 2020-02-06T01:32:40.180 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:40.178+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.178588+0000) 2020-02-06T01:32:40.180 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:40.178+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.179238+0000) 2020-02-06T01:32:40.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:40.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.179295+0000) 2020-02-06T01:32:40.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:40.179+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.179750+0000) 2020-02-06T01:32:40.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:40.181+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.182921+0000) 2020-02-06T01:32:40.196 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:40.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:40.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:40.592+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.592870+0000) 2020-02-06T01:32:40.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:40.847+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.849054+0000) 2020-02-06T01:32:40.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:40.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.872326+0000) 2020-02-06T01:32:40.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:40.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:40.872446+0000) 2020-02-06T01:32:45.202 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:45.199+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.200913+0000) 2020-02-06T01:32:45.202 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:45.200+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.201457+0000) 2020-02-06T01:32:45.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:45.201+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.202089+0000) 2020-02-06T01:32:45.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:45.201+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.202569+0000) 2020-02-06T01:32:45.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:45.202+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.202957+0000) 2020-02-06T01:32:45.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:45.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.203315+0000) 2020-02-06T01:32:45.205 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:45.203+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.203600+0000) 2020-02-06T01:32:45.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:45.203+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.204600+0000) 2020-02-06T01:32:45.206 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:45.204+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.205295+0000) 2020-02-06T01:32:45.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:45.204+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.205457+0000) 2020-02-06T01:32:45.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:45.205+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.205605+0000) 2020-02-06T01:32:45.207 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:45.205+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.206092+0000) 2020-02-06T01:32:45.208 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:45.206+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.207097+0000) 2020-02-06T01:32:45.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:45.210+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:45.211281+0000) 2020-02-06T01:32:45.225 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:45.223+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:46.115 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:46.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:46.114577+0000) 2020-02-06T01:32:46.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:46.418+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:46.419691+0000) 2020-02-06T01:32:46.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:46.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:46.478019+0000) 2020-02-06T01:32:46.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:46.704+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:46.704699+0000) 2020-02-06T01:32:47.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:47.094+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:47.095956+0000) 2020-02-06T01:32:47.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:47.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:47.367112+0000) 2020-02-06T01:32:47.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:47.366+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:47.367349+0000) 2020-02-06T01:32:47.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:47.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:47.367718+0000) 2020-02-06T01:32:49.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:49.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:49.739411+0000) 2020-02-06T01:32:49.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:49.986+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:49.986369+0000) 2020-02-06T01:32:50.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:50.228+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.229913+0000) 2020-02-06T01:32:50.231 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:50.230+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.230860+0000) 2020-02-06T01:32:50.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:50.231+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.231898+0000) 2020-02-06T01:32:50.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:50.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.232209+0000) 2020-02-06T01:32:50.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:50.231+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.232752+0000) 2020-02-06T01:32:50.234 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:50.232+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.233431+0000) 2020-02-06T01:32:50.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:50.232+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.233900+0000) 2020-02-06T01:32:50.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:50.234+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.234477+0000) 2020-02-06T01:32:50.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:50.234+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.235318+0000) 2020-02-06T01:32:50.236 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:50.234+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.235513+0000) 2020-02-06T01:32:50.237 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:50.235+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.236211+0000) 2020-02-06T01:32:50.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:50.235+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.236263+0000) 2020-02-06T01:32:50.237 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:50.236+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.236772+0000) 2020-02-06T01:32:50.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:50.238+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.239642+0000) 2020-02-06T01:32:50.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:50.252+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206537 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:50.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:50.592+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.593089+0000) 2020-02-06T01:32:50.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:50.848+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.849299+0000) 2020-02-06T01:32:50.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:50.871+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.872543+0000) 2020-02-06T01:32:50.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:50.871+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:50.872679+0000) 2020-02-06T01:32:55.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:55.257+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.257496+0000) 2020-02-06T01:32:55.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:55.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.259416+0000) 2020-02-06T01:32:55.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:55.258+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.258766+0000) 2020-02-06T01:32:55.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:32:55.258+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.260015+0000) 2020-02-06T01:32:55.261 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:55.259+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.260774+0000) 2020-02-06T01:32:55.262 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:55.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.261099+0000) 2020-02-06T01:32:55.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:55.261+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.261796+0000) 2020-02-06T01:32:55.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:55.260+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.262019+0000) 2020-02-06T01:32:55.263 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:32:55.261+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.263042+0000) 2020-02-06T01:32:55.264 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:32:55.262+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.263451+0000) 2020-02-06T01:32:55.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:55.263+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.263875+0000) 2020-02-06T01:32:55.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:55.263+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.264150+0000) 2020-02-06T01:32:55.265 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:32:55.264+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.264572+0000) 2020-02-06T01:32:55.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:55.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:55.269235+0000) 2020-02-06T01:32:55.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:32:55.280+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206576 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:32:56.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:32:56.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:56.114808+0000) 2020-02-06T01:32:56.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:56.427+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:56.427463+0000) 2020-02-06T01:32:56.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:32:56.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:56.478266+0000) 2020-02-06T01:32:56.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:32:56.704+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:56.705011+0000) 2020-02-06T01:32:57.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:32:57.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:57.096181+0000) 2020-02-06T01:32:57.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:32:57.367+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:57.367364+0000) 2020-02-06T01:32:57.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:32:57.367+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:57.367631+0000) 2020-02-06T01:32:57.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:32:57.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:57.367915+0000) 2020-02-06T01:32:59.740 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:59.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:59.739653+0000) 2020-02-06T01:32:59.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:32:59.986+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:32:59.986599+0000) 2020-02-06T01:33:00.288 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:00.287+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.287141+0000) 2020-02-06T01:33:00.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:00.287+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.287682+0000) 2020-02-06T01:33:00.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:00.287+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.288124+0000) 2020-02-06T01:33:00.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:00.288+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.288917+0000) 2020-02-06T01:33:00.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:00.288+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.289401+0000) 2020-02-06T01:33:00.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:00.289+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.289763+0000) 2020-02-06T01:33:00.291 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:00.289+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.290016+0000) 2020-02-06T01:33:00.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:00.290+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.290477+0000) 2020-02-06T01:33:00.292 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:00.291+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.291561+0000) 2020-02-06T01:33:00.293 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:00.291+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.292756+0000) 2020-02-06T01:33:00.294 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:00.292+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.292753+0000) 2020-02-06T01:33:00.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:00.292+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.293105+0000) 2020-02-06T01:33:00.295 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:00.292+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.293137+0000) 2020-02-06T01:33:00.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:00.295+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.296184+0000) 2020-02-06T01:33:00.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:00.308+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206618 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:00.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:00.593+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.593331+0000) 2020-02-06T01:33:00.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:00.849+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.849520+0000) 2020-02-06T01:33:00.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:00.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.872768+0000) 2020-02-06T01:33:00.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:00.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:00.872915+0000) 2020-02-06T01:33:01.512 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:33:01.518 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:33:05.315 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:05.313+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.314367+0000) 2020-02-06T01:33:05.316 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:05.313+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.314377+0000) 2020-02-06T01:33:05.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:05.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.314785+0000) 2020-02-06T01:33:05.316 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:05.314+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.315225+0000) 2020-02-06T01:33:05.317 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:05.315+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.316296+0000) 2020-02-06T01:33:05.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:05.315+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.317032+0000) 2020-02-06T01:33:05.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:05.317+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.317807+0000) 2020-02-06T01:33:05.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:05.317+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.318020+0000) 2020-02-06T01:33:05.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:05.318+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.318525+0000) 2020-02-06T01:33:05.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:05.318+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.319260+0000) 2020-02-06T01:33:05.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:05.319+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.319997+0000) 2020-02-06T01:33:05.321 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:05.319+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.320502+0000) 2020-02-06T01:33:05.322 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:05.320+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.321285+0000) 2020-02-06T01:33:05.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:05.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:05.324862+0000) 2020-02-06T01:33:05.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:05.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206656 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:06.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:06.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:06.115049+0000) 2020-02-06T01:33:06.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:06.434+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:06.435835+0000) 2020-02-06T01:33:06.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:06.478+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:06.478528+0000) 2020-02-06T01:33:06.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:06.704+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:06.705270+0000) 2020-02-06T01:33:07.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:07.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:07.096430+0000) 2020-02-06T01:33:07.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:07.367+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:07.367597+0000) 2020-02-06T01:33:07.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:07.367+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:07.367844+0000) 2020-02-06T01:33:07.369 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:07.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:07.368129+0000) 2020-02-06T01:33:09.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:09.738+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:09.739913+0000) 2020-02-06T01:33:09.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:09.985+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:09.986826+0000) 2020-02-06T01:33:10.342 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:10.340+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.341595+0000) 2020-02-06T01:33:10.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:10.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.342639+0000) 2020-02-06T01:33:10.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:10.342+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.343974+0000) 2020-02-06T01:33:10.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:10.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.344706+0000) 2020-02-06T01:33:10.346 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:10.344+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.345383+0000) 2020-02-06T01:33:10.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:10.345+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.345713+0000) 2020-02-06T01:33:10.347 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:10.346+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.346416+0000) 2020-02-06T01:33:10.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:10.345+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.346646+0000) 2020-02-06T01:33:10.349 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:10.347+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.348138+0000) 2020-02-06T01:33:10.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:10.348+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.348775+0000) 2020-02-06T01:33:10.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:10.348+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.349457+0000) 2020-02-06T01:33:10.351 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:10.349+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.350300+0000) 2020-02-06T01:33:10.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:10.349+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.350561+0000) 2020-02-06T01:33:10.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:10.350+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.352063+0000) 2020-02-06T01:33:10.369 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:10.367+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:10.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:10.593+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.593545+0000) 2020-02-06T01:33:10.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:10.848+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.849743+0000) 2020-02-06T01:33:10.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:10.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.872985+0000) 2020-02-06T01:33:10.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:10.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:10.873148+0000) 2020-02-06T01:33:15.374 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:15.372+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.373111+0000) 2020-02-06T01:33:15.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:15.372+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.373583+0000) 2020-02-06T01:33:15.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:15.372+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.374081+0000) 2020-02-06T01:33:15.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:15.373+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.374403+0000) 2020-02-06T01:33:15.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:15.373+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.374780+0000) 2020-02-06T01:33:15.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:15.373+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.375189+0000) 2020-02-06T01:33:15.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:15.374+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.375617+0000) 2020-02-06T01:33:15.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:15.375+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.377243+0000) 2020-02-06T01:33:15.379 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:15.377+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.378420+0000) 2020-02-06T01:33:15.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:15.377+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.378592+0000) 2020-02-06T01:33:15.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:15.378+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.379341+0000) 2020-02-06T01:33:15.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:15.377+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.378888+0000) 2020-02-06T01:33:15.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:15.378+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.379832+0000) 2020-02-06T01:33:15.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:15.383+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:15.384749+0000) 2020-02-06T01:33:15.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:15.394+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:16.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:16.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:16.115288+0000) 2020-02-06T01:33:16.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:16.442+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:16.443286+0000) 2020-02-06T01:33:16.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:16.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:16.478796+0000) 2020-02-06T01:33:16.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:16.704+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:16.705509+0000) 2020-02-06T01:33:17.097 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:17.095+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:17.096638+0000) 2020-02-06T01:33:17.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:17.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:17.367826+0000) 2020-02-06T01:33:17.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:17.366+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:17.368044+0000) 2020-02-06T01:33:17.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:17.366+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:17.368323+0000) 2020-02-06T01:33:19.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:19.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:19.740202+0000) 2020-02-06T01:33:19.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:19.986+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:19.987049+0000) 2020-02-06T01:33:20.402 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:20.400+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.401241+0000) 2020-02-06T01:33:20.404 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:20.402+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.403766+0000) 2020-02-06T01:33:20.405 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:20.403+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.404393+0000) 2020-02-06T01:33:20.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:20.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.404648+0000) 2020-02-06T01:33:20.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:20.403+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.404970+0000) 2020-02-06T01:33:20.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:20.403+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.404967+0000) 2020-02-06T01:33:20.406 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:20.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.405525+0000) 2020-02-06T01:33:20.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:20.404+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.405387+0000) 2020-02-06T01:33:20.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:20.404+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.405841+0000) 2020-02-06T01:33:20.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:20.404+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.406069+0000) 2020-02-06T01:33:20.409 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:20.405+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.406503+0000) 2020-02-06T01:33:20.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:20.404+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.406254+0000) 2020-02-06T01:33:20.410 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:20.405+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.406795+0000) 2020-02-06T01:33:20.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:20.410+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.411875+0000) 2020-02-06T01:33:20.423 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:20.421+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206779 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:20.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:20.592+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.593760+0000) 2020-02-06T01:33:20.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:20.849+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.849960+0000) 2020-02-06T01:33:20.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:20.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.873191+0000) 2020-02-06T01:33:20.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:20.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:20.873376+0000) 2020-02-06T01:33:25.429 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:25.427+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.428834+0000) 2020-02-06T01:33:25.431 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:25.428+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.429420+0000) 2020-02-06T01:33:25.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:25.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.430060+0000) 2020-02-06T01:33:25.432 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:25.430+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.431181+0000) 2020-02-06T01:33:25.432 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:25.430+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.431709+0000) 2020-02-06T01:33:25.433 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:25.430+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.431467+0000) 2020-02-06T01:33:25.433 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:25.430+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.431985+0000) 2020-02-06T01:33:25.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:25.430+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.432278+0000) 2020-02-06T01:33:25.434 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:25.430+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.432302+0000) 2020-02-06T01:33:25.434 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:25.432+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.433161+0000) 2020-02-06T01:33:25.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:25.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.433538+0000) 2020-02-06T01:33:25.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:25.432+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.434092+0000) 2020-02-06T01:33:25.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:25.432+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.434371+0000) 2020-02-06T01:33:25.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:25.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:25.439277+0000) 2020-02-06T01:33:25.452 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:25.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:26.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:26.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:26.115556+0000) 2020-02-06T01:33:26.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:26.450+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:26.451330+0000) 2020-02-06T01:33:26.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:26.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:26.479004+0000) 2020-02-06T01:33:26.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:26.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:26.705746+0000) 2020-02-06T01:33:27.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:27.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:27.096939+0000) 2020-02-06T01:33:27.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:27.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:27.368117+0000) 2020-02-06T01:33:27.369 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:27.367+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:27.368471+0000) 2020-02-06T01:33:27.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:27.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:27.368605+0000) 2020-02-06T01:33:29.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:29.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:29.740497+0000) 2020-02-06T01:33:29.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:29.986+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:29.987304+0000) 2020-02-06T01:33:30.457 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:30.455+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.456276+0000) 2020-02-06T01:33:30.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:30.456+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.457472+0000) 2020-02-06T01:33:30.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:30.456+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.457916+0000) 2020-02-06T01:33:30.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:30.456+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.458243+0000) 2020-02-06T01:33:30.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:30.457+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.458692+0000) 2020-02-06T01:33:30.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:30.458+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.459778+0000) 2020-02-06T01:33:30.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:30.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.460453+0000) 2020-02-06T01:33:30.461 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:30.459+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.460582+0000) 2020-02-06T01:33:30.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:30.460+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.461524+0000) 2020-02-06T01:33:30.463 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:30.461+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.462359+0000) 2020-02-06T01:33:30.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:30.460+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.462310+0000) 2020-02-06T01:33:30.463 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:30.461+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.462667+0000) 2020-02-06T01:33:30.464 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:30.461+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.462757+0000) 2020-02-06T01:33:30.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:30.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.466322+0000) 2020-02-06T01:33:30.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:30.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:30.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:30.593+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.593979+0000) 2020-02-06T01:33:30.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:30.849+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.850205+0000) 2020-02-06T01:33:30.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:30.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.873382+0000) 2020-02-06T01:33:30.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:30.872+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:30.873599+0000) 2020-02-06T01:33:31.659 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:33:31.665 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:33:35.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:35.483+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.484835+0000) 2020-02-06T01:33:35.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:35.484+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.485604+0000) 2020-02-06T01:33:35.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:35.484+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.485799+0000) 2020-02-06T01:33:35.487 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:35.485+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.486534+0000) 2020-02-06T01:33:35.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:35.486+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.487530+0000) 2020-02-06T01:33:35.491 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:35.487+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.488659+0000) 2020-02-06T01:33:35.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:35.487+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.488724+0000) 2020-02-06T01:33:35.491 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:35.488+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.490015+0000) 2020-02-06T01:33:35.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:35.487+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.488950+0000) 2020-02-06T01:33:35.492 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:35.488+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.489807+0000) 2020-02-06T01:33:35.492 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:35.489+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.490388+0000) 2020-02-06T01:33:35.493 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:35.489+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.490902+0000) 2020-02-06T01:33:35.493 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:35.489+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.491002+0000) 2020-02-06T01:33:35.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:35.493+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:35.494295+0000) 2020-02-06T01:33:35.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:35.505+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:36.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:36.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:36.115778+0000) 2020-02-06T01:33:36.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:36.462+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:36.463134+0000) 2020-02-06T01:33:36.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:36.477+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:36.479227+0000) 2020-02-06T01:33:36.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:36.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:36.705977+0000) 2020-02-06T01:33:37.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:37.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:37.097176+0000) 2020-02-06T01:33:37.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:37.366+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:37.368376+0000) 2020-02-06T01:33:37.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:37.367+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:37.368692+0000) 2020-02-06T01:33:37.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:37.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:37.368855+0000) 2020-02-06T01:33:39.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:39.739+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:39.740721+0000) 2020-02-06T01:33:39.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:39.987+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:39.987535+0000) 2020-02-06T01:33:40.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:40.510+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.511637+0000) 2020-02-06T01:33:40.514 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:40.511+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.513138+0000) 2020-02-06T01:33:40.515 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:40.513+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.514475+0000) 2020-02-06T01:33:40.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:40.514+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.515990+0000) 2020-02-06T01:33:40.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:40.515+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.516252+0000) 2020-02-06T01:33:40.517 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:40.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.516556+0000) 2020-02-06T01:33:40.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:40.515+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.516878+0000) 2020-02-06T01:33:40.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:40.515+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.517367+0000) 2020-02-06T01:33:40.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:40.516+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.517344+0000) 2020-02-06T01:33:40.519 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:40.516+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.518051+0000) 2020-02-06T01:33:40.519 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:40.516+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.518277+0000) 2020-02-06T01:33:40.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:40.517+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.518598+0000) 2020-02-06T01:33:40.520 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:40.517+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.518759+0000) 2020-02-06T01:33:40.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:40.521+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.522927+0000) 2020-02-06T01:33:40.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:40.532+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:40.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:40.593+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.594201+0000) 2020-02-06T01:33:40.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:40.850+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.850416+0000) 2020-02-06T01:33:40.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:40.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.873607+0000) 2020-02-06T01:33:40.874 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:40.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:40.873824+0000) 2020-02-06T01:33:45.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:45.537+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.538994+0000) 2020-02-06T01:33:45.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:45.540+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.541747+0000) 2020-02-06T01:33:45.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:45.541+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.542572+0000) 2020-02-06T01:33:45.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:45.540+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.542269+0000) 2020-02-06T01:33:45.544 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:45.541+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.542890+0000) 2020-02-06T01:33:45.545 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:45.542+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.543895+0000) 2020-02-06T01:33:45.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:45.543+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.544141+0000) 2020-02-06T01:33:45.545 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:45.543+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.544706+0000) 2020-02-06T01:33:45.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:45.544+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.545546+0000) 2020-02-06T01:33:45.547 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:45.544+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.545972+0000) 2020-02-06T01:33:45.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:45.544+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.546188+0000) 2020-02-06T01:33:45.547 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:45.545+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.546800+0000) 2020-02-06T01:33:45.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:45.547+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.548506+0000) 2020-02-06T01:33:45.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:45.547+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:45.548751+0000) 2020-02-06T01:33:45.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:45.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 206980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:46.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:46.115+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:46.116028+0000) 2020-02-06T01:33:46.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:46.473+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:46.474623+0000) 2020-02-06T01:33:46.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:46.478+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:46.479484+0000) 2020-02-06T01:33:46.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:46.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:46.706204+0000) 2020-02-06T01:33:47.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:47.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:47.097409+0000) 2020-02-06T01:33:47.369 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:47.367+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:47.368640+0000) 2020-02-06T01:33:47.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:47.367+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:47.368890+0000) 2020-02-06T01:33:47.370 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:47.367+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:47.369125+0000) 2020-02-06T01:33:49.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:49.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:49.740978+0000) 2020-02-06T01:33:49.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:49.987+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:49.987763+0000) 2020-02-06T01:33:50.571 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:50.568+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.569907+0000) 2020-02-06T01:33:50.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:50.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.570740+0000) 2020-02-06T01:33:50.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:50.570+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.571527+0000) 2020-02-06T01:33:50.573 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:50.570+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.572113+0000) 2020-02-06T01:33:50.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:50.570+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.571796+0000) 2020-02-06T01:33:50.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:50.571+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.572351+0000) 2020-02-06T01:33:50.574 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:50.571+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.572901+0000) 2020-02-06T01:33:50.574 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:50.572+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.573666+0000) 2020-02-06T01:33:50.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:50.572+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.573558+0000) 2020-02-06T01:33:50.575 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:50.572+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.574163+0000) 2020-02-06T01:33:50.576 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:50.574+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.575198+0000) 2020-02-06T01:33:50.576 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:50.574+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.575258+0000) 2020-02-06T01:33:50.576 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:50.574+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.575582+0000) 2020-02-06T01:33:50.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:50.583+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.584905+0000) 2020-02-06T01:33:50.593 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:50.590+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207026 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:50.595 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:50.592+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.594341+0000) 2020-02-06T01:33:50.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:50.850+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.850624+0000) 2020-02-06T01:33:50.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:50.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.873812+0000) 2020-02-06T01:33:50.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:50.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:50.874043+0000) 2020-02-06T01:33:55.598 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:33:55.597+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.597340+0000) 2020-02-06T01:33:55.599 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:55.596+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.597413+0000) 2020-02-06T01:33:55.599 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:55.596+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.598225+0000) 2020-02-06T01:33:55.601 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:33:55.600+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.600269+0000) 2020-02-06T01:33:55.602 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:55.601+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.601434+0000) 2020-02-06T01:33:55.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:55.601+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.601790+0000) 2020-02-06T01:33:55.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:55.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.602541+0000) 2020-02-06T01:33:55.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:33:55.602+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.602974+0000) 2020-02-06T01:33:55.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:55.601+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.603334+0000) 2020-02-06T01:33:55.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:55.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.603708+0000) 2020-02-06T01:33:55.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:55.602+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.604049+0000) 2020-02-06T01:33:55.605 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:55.602+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.604240+0000) 2020-02-06T01:33:55.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:33:55.602+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.604404+0000) 2020-02-06T01:33:55.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:55.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:55.610323+0000) 2020-02-06T01:33:55.622 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:33:55.620+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207059 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:33:56.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:33:56.114+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:56.116254+0000) 2020-02-06T01:33:56.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:33:56.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:56.479710+0000) 2020-02-06T01:33:56.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:56.486+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:56.486311+0000) 2020-02-06T01:33:56.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:33:56.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:56.706440+0000) 2020-02-06T01:33:57.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:33:57.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:57.097645+0000) 2020-02-06T01:33:57.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:33:57.368+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:57.368899+0000) 2020-02-06T01:33:57.370 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:33:57.368+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:57.369111+0000) 2020-02-06T01:33:57.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:33:57.368+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:57.369401+0000) 2020-02-06T01:33:59.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:59.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:59.741213+0000) 2020-02-06T01:33:59.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:33:59.986+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:33:59.987995+0000) 2020-02-06T01:34:00.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:00.594+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.595475+0000) 2020-02-06T01:34:00.627 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:00.624+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.625796+0000) 2020-02-06T01:34:00.628 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:00.626+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.627318+0000) 2020-02-06T01:34:00.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:00.626+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.628010+0000) 2020-02-06T01:34:00.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:00.627+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.628696+0000) 2020-02-06T01:34:00.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:00.628+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.629547+0000) 2020-02-06T01:34:00.632 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:00.629+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.631020+0000) 2020-02-06T01:34:00.632 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:00.629+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.631272+0000) 2020-02-06T01:34:00.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:00.630+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.632036+0000) 2020-02-06T01:34:00.633 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:00.630+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.631689+0000) 2020-02-06T01:34:00.633 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:00.631+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.633105+0000) 2020-02-06T01:34:00.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:00.632+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.633353+0000) 2020-02-06T01:34:00.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:00.632+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.633453+0000) 2020-02-06T01:34:00.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:00.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.634360+0000) 2020-02-06T01:34:00.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:00.637+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.638108+0000) 2020-02-06T01:34:00.651 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:00.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:00.851 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:00.849+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.850847+0000) 2020-02-06T01:34:00.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:00.872+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.874016+0000) 2020-02-06T01:34:00.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:00.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:00.874268+0000) 2020-02-06T01:34:01.809 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:34:01.815 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:34:05.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:05.655+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.656340+0000) 2020-02-06T01:34:05.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:05.655+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.656851+0000) 2020-02-06T01:34:05.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:05.656+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.657325+0000) 2020-02-06T01:34:05.659 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:05.657+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.657807+0000) 2020-02-06T01:34:05.659 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:05.656+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.657967+0000) 2020-02-06T01:34:05.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:05.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.658399+0000) 2020-02-06T01:34:05.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:05.657+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.658590+0000) 2020-02-06T01:34:05.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:05.658+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.659229+0000) 2020-02-06T01:34:05.662 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:05.659+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.660418+0000) 2020-02-06T01:34:05.662 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:05.660+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.660882+0000) 2020-02-06T01:34:05.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:05.661+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.661769+0000) 2020-02-06T01:34:05.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:05.661+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.662380+0000) 2020-02-06T01:34:05.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:05.661+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.662199+0000) 2020-02-06T01:34:05.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:05.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:05.666610+0000) 2020-02-06T01:34:05.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:05.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207138 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:06.117 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:06.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:06.116527+0000) 2020-02-06T01:34:06.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:06.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:06.479922+0000) 2020-02-06T01:34:06.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:06.493+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:06.494675+0000) 2020-02-06T01:34:06.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:06.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:06.706692+0000) 2020-02-06T01:34:07.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:07.096+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:07.097875+0000) 2020-02-06T01:34:07.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:07.368+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:07.369225+0000) 2020-02-06T01:34:07.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:07.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:07.369415+0000) 2020-02-06T01:34:07.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:07.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:07.369723+0000) 2020-02-06T01:34:09.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:09.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:09.741438+0000) 2020-02-06T01:34:09.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:09.987+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:09.988241+0000) 2020-02-06T01:34:10.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:10.595+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.595724+0000) 2020-02-06T01:34:10.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:10.682+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.683390+0000) 2020-02-06T01:34:10.685 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:10.683+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.684050+0000) 2020-02-06T01:34:10.685 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:10.684+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.684798+0000) 2020-02-06T01:34:10.686 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:10.684+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.685569+0000) 2020-02-06T01:34:10.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:10.684+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.685660+0000) 2020-02-06T01:34:10.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:10.685+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.686194+0000) 2020-02-06T01:34:10.687 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:10.685+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.686354+0000) 2020-02-06T01:34:10.688 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:10.685+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.686651+0000) 2020-02-06T01:34:10.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:10.686+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.687571+0000) 2020-02-06T01:34:10.689 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:10.686+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.687869+0000) 2020-02-06T01:34:10.689 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:10.687+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.688223+0000) 2020-02-06T01:34:10.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:10.687+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.688399+0000) 2020-02-06T01:34:10.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:10.688+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.689102+0000) 2020-02-06T01:34:10.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:10.692+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.693669+0000) 2020-02-06T01:34:10.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:10.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207186 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:10.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:10.850+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.851060+0000) 2020-02-06T01:34:10.875 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:10.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.874271+0000) 2020-02-06T01:34:10.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:10.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:10.874446+0000) 2020-02-06T01:34:15.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:15.710+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.711512+0000) 2020-02-06T01:34:15.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:15.711+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.712316+0000) 2020-02-06T01:34:15.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:15.712+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.712630+0000) 2020-02-06T01:34:15.714 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:15.712+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.713406+0000) 2020-02-06T01:34:15.715 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:15.712+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.713859+0000) 2020-02-06T01:34:15.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:15.713+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.714131+0000) 2020-02-06T01:34:15.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:15.713+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.714467+0000) 2020-02-06T01:34:15.716 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:15.714+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.715405+0000) 2020-02-06T01:34:15.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:15.715+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.716822+0000) 2020-02-06T01:34:15.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:15.716+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.717268+0000) 2020-02-06T01:34:15.718 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:15.716+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.717560+0000) 2020-02-06T01:34:15.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:15.715+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.716289+0000) 2020-02-06T01:34:15.719 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:15.717+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.717928+0000) 2020-02-06T01:34:15.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:15.719+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:15.720361+0000) 2020-02-06T01:34:15.734 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:15.733+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:16.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:16.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:16.116796+0000) 2020-02-06T01:34:16.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:16.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:16.480138+0000) 2020-02-06T01:34:16.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:16.506+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:16.507121+0000) 2020-02-06T01:34:16.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:16.705+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:16.706940+0000) 2020-02-06T01:34:17.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:17.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:17.098135+0000) 2020-02-06T01:34:17.370 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:17.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:17.369494+0000) 2020-02-06T01:34:17.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:17.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:17.369662+0000) 2020-02-06T01:34:17.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:17.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:17.369987+0000) 2020-02-06T01:34:19.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:19.740+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:19.741671+0000) 2020-02-06T01:34:19.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:19.987+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:19.988497+0000) 2020-02-06T01:34:20.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:20.595+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.595969+0000) 2020-02-06T01:34:20.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:20.738+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.739528+0000) 2020-02-06T01:34:20.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:20.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.739921+0000) 2020-02-06T01:34:20.741 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:20.740+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.740887+0000) 2020-02-06T01:34:20.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:20.740+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.741203+0000) 2020-02-06T01:34:20.743 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:20.741+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.742128+0000) 2020-02-06T01:34:20.743 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:20.741+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.742470+0000) 2020-02-06T01:34:20.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:20.741+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.742979+0000) 2020-02-06T01:34:20.744 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:20.742+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.743215+0000) 2020-02-06T01:34:20.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:20.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.743992+0000) 2020-02-06T01:34:20.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:20.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.744262+0000) 2020-02-06T01:34:20.746 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:20.744+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.744812+0000) 2020-02-06T01:34:20.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:20.744+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.745090+0000) 2020-02-06T01:34:20.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:20.744+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.745390+0000) 2020-02-06T01:34:20.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:20.747+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.748639+0000) 2020-02-06T01:34:20.764 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:20.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:20.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:20.850+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.851318+0000) 2020-02-06T01:34:20.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:20.873+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.874509+0000) 2020-02-06T01:34:20.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:20.873+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:20.874674+0000) 2020-02-06T01:34:25.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:25.768+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.769510+0000) 2020-02-06T01:34:25.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:25.768+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.769886+0000) 2020-02-06T01:34:25.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:25.769+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.770109+0000) 2020-02-06T01:34:25.771 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:25.769+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.770395+0000) 2020-02-06T01:34:25.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:25.769+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.771016+0000) 2020-02-06T01:34:25.772 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:25.770+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.771622+0000) 2020-02-06T01:34:25.773 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:25.771+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.772094+0000) 2020-02-06T01:34:25.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:25.771+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.772544+0000) 2020-02-06T01:34:25.774 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:25.772+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.773129+0000) 2020-02-06T01:34:25.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:25.773+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.774373+0000) 2020-02-06T01:34:25.777 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:25.773+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.773409+0000) 2020-02-06T01:34:25.777 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:25.774+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.774941+0000) 2020-02-06T01:34:25.778 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:25.774+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.775304+0000) 2020-02-06T01:34:25.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:25.777+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:25.778862+0000) 2020-02-06T01:34:25.792 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:25.791+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207300 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:26.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:26.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:26.117029+0000) 2020-02-06T01:34:26.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:26.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:26.480363+0000) 2020-02-06T01:34:26.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:26.514+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:26.515181+0000) 2020-02-06T01:34:26.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:26.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:26.707159+0000) 2020-02-06T01:34:27.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:27.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:27.098404+0000) 2020-02-06T01:34:27.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:27.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:27.369678+0000) 2020-02-06T01:34:27.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:27.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:27.369874+0000) 2020-02-06T01:34:27.371 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:27.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:27.370256+0000) 2020-02-06T01:34:29.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:29.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:29.741931+0000) 2020-02-06T01:34:29.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:29.988+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:29.988710+0000) 2020-02-06T01:34:30.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:30.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.596228+0000) 2020-02-06T01:34:30.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:30.795+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.796394+0000) 2020-02-06T01:34:30.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:30.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.797139+0000) 2020-02-06T01:34:30.799 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:30.797+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.798631+0000) 2020-02-06T01:34:30.800 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:30.798+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.799045+0000) 2020-02-06T01:34:30.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:30.798+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.799664+0000) 2020-02-06T01:34:30.801 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:30.799+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.800302+0000) 2020-02-06T01:34:30.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:30.800+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.800732+0000) 2020-02-06T01:34:30.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:30.800+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.801344+0000) 2020-02-06T01:34:30.803 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:30.801+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.802117+0000) 2020-02-06T01:34:30.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:30.801+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.802439+0000) 2020-02-06T01:34:30.804 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:30.802+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.803320+0000) 2020-02-06T01:34:30.804 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:30.802+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.803318+0000) 2020-02-06T01:34:30.805 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:30.803+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.803770+0000) 2020-02-06T01:34:30.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:30.809+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.810386+0000) 2020-02-06T01:34:30.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:30.822+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207348 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:30.852 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:30.851+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.851534+0000) 2020-02-06T01:34:30.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:30.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.874727+0000) 2020-02-06T01:34:30.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:30.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:30.874950+0000) 2020-02-06T01:34:32.035 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:34:32.041 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:34:35.829 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:35.827+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.828180+0000) 2020-02-06T01:34:35.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:35.827+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.829019+0000) 2020-02-06T01:34:35.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:35.829+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.830357+0000) 2020-02-06T01:34:35.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:35.829+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.830501+0000) 2020-02-06T01:34:35.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:35.830+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.831128+0000) 2020-02-06T01:34:35.832 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:35.830+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.831326+0000) 2020-02-06T01:34:35.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:35.830+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.831587+0000) 2020-02-06T01:34:35.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:35.830+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.831958+0000) 2020-02-06T01:34:35.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:35.831+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.832195+0000) 2020-02-06T01:34:35.834 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:35.831+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.832596+0000) 2020-02-06T01:34:35.834 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:35.832+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.832925+0000) 2020-02-06T01:34:35.835 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:35.834+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.834443+0000) 2020-02-06T01:34:35.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:35.834+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.834850+0000) 2020-02-06T01:34:35.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:35.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:35.838232+0000) 2020-02-06T01:34:35.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:35.849+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207381 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:36.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:36.117+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:36.117244+0000) 2020-02-06T01:34:36.481 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:36.480+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:36.480584+0000) 2020-02-06T01:34:36.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:36.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:36.523630+0000) 2020-02-06T01:34:36.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:36.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:36.707362+0000) 2020-02-06T01:34:37.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:37.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:37.098631+0000) 2020-02-06T01:34:37.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:37.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:37.369933+0000) 2020-02-06T01:34:37.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:37.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:37.370093+0000) 2020-02-06T01:34:37.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:37.370+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:37.370527+0000) 2020-02-06T01:34:39.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:39.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:39.742232+0000) 2020-02-06T01:34:39.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:39.988+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:39.988944+0000) 2020-02-06T01:34:40.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:40.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.596480+0000) 2020-02-06T01:34:40.856 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:40.854+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.855513+0000) 2020-02-06T01:34:40.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:40.855+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.856694+0000) 2020-02-06T01:34:40.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:40.856+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.857146+0000) 2020-02-06T01:34:40.858 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:40.857+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.857705+0000) 2020-02-06T01:34:40.859 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:40.856+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.857983+0000) 2020-02-06T01:34:40.859 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:40.857+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.858401+0000) 2020-02-06T01:34:40.859 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:40.857+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.858950+0000) 2020-02-06T01:34:40.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:40.859+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.859956+0000) 2020-02-06T01:34:40.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:40.860+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.860458+0000) 2020-02-06T01:34:40.862 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:40.860+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.860954+0000) 2020-02-06T01:34:40.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:40.860+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.861612+0000) 2020-02-06T01:34:40.862 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:40.861+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.861422+0000) 2020-02-06T01:34:40.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:40.861+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.861583+0000) 2020-02-06T01:34:40.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:40.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.866414+0000) 2020-02-06T01:34:40.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:40.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.874962+0000) 2020-02-06T01:34:40.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:40.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:40.875143+0000) 2020-02-06T01:34:40.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:40.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:45.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:45.879+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.880804+0000) 2020-02-06T01:34:45.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:45.880+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.882035+0000) 2020-02-06T01:34:45.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:45.882+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.883124+0000) 2020-02-06T01:34:45.885 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:45.883+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.884163+0000) 2020-02-06T01:34:45.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:45.883+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.884609+0000) 2020-02-06T01:34:45.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:45.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.884945+0000) 2020-02-06T01:34:45.886 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:45.884+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.885567+0000) 2020-02-06T01:34:45.887 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:45.884+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.886202+0000) 2020-02-06T01:34:45.888 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:45.886+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.887185+0000) 2020-02-06T01:34:45.888 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:45.885+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.887149+0000) 2020-02-06T01:34:45.889 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:45.886+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.887451+0000) 2020-02-06T01:34:45.889 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:45.886+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.887993+0000) 2020-02-06T01:34:45.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:45.888+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.889367+0000) 2020-02-06T01:34:45.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:45.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:45.890293+0000) 2020-02-06T01:34:45.904 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:45.901+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207461 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:46.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:46.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:46.117490+0000) 2020-02-06T01:34:46.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:46.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:46.480835+0000) 2020-02-06T01:34:46.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:46.531+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:46.531739+0000) 2020-02-06T01:34:46.708 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:46.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:46.707593+0000) 2020-02-06T01:34:47.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:47.097+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:47.098889+0000) 2020-02-06T01:34:47.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:47.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:47.370251+0000) 2020-02-06T01:34:47.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:47.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:47.370356+0000) 2020-02-06T01:34:47.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:47.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:47.370766+0000) 2020-02-06T01:34:47.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:47.602+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:47.602849+0000) 2020-02-06T01:34:49.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:49.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:49.742484+0000) 2020-02-06T01:34:49.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:49.989+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:49.989170+0000) 2020-02-06T01:34:50.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:50.595+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.596709+0000) 2020-02-06T01:34:50.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:50.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.875166+0000) 2020-02-06T01:34:50.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:50.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.875362+0000) 2020-02-06T01:34:50.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:50.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.907836+0000) 2020-02-06T01:34:50.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:50.908+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.908960+0000) 2020-02-06T01:34:50.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:50.911+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.911155+0000) 2020-02-06T01:34:50.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:50.911+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.911387+0000) 2020-02-06T01:34:50.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:50.910+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.911923+0000) 2020-02-06T01:34:50.913 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:50.912+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.912580+0000) 2020-02-06T01:34:50.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:50.912+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.912918+0000) 2020-02-06T01:34:50.914 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:50.911+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.913347+0000) 2020-02-06T01:34:50.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:50.912+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.913869+0000) 2020-02-06T01:34:50.915 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:50.912+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.914213+0000) 2020-02-06T01:34:50.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:50.914+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.914565+0000) 2020-02-06T01:34:50.916 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:50.913+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.914922+0000) 2020-02-06T01:34:50.916 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:50.913+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.915245+0000) 2020-02-06T01:34:50.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:50.917+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:50.917413+0000) 2020-02-06T01:34:50.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:50.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:55.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:55.936+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.937442+0000) 2020-02-06T01:34:55.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:55.936+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.938112+0000) 2020-02-06T01:34:55.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:55.937+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.938501+0000) 2020-02-06T01:34:55.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:55.938+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.939445+0000) 2020-02-06T01:34:55.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:55.938+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.939505+0000) 2020-02-06T01:34:55.941 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:34:55.938+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.939645+0000) 2020-02-06T01:34:55.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:34:55.938+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.939923+0000) 2020-02-06T01:34:55.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:55.938+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.940286+0000) 2020-02-06T01:34:55.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:55.938+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.940412+0000) 2020-02-06T01:34:55.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:55.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.940562+0000) 2020-02-06T01:34:55.943 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:34:55.939+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.941214+0000) 2020-02-06T01:34:55.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:55.940+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.941976+0000) 2020-02-06T01:34:55.943 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:55.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.941963+0000) 2020-02-06T01:34:55.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:55.945+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:55.946976+0000) 2020-02-06T01:34:55.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:34:55.956+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:34:56.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:34:56.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:56.117726+0000) 2020-02-06T01:34:56.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:34:56.479+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:56.481017+0000) 2020-02-06T01:34:56.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:56.539+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:56.541041+0000) 2020-02-06T01:34:56.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:34:56.706+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:56.707825+0000) 2020-02-06T01:34:57.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:34:57.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:57.099121+0000) 2020-02-06T01:34:57.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:34:57.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:57.370508+0000) 2020-02-06T01:34:57.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:34:57.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:57.370682+0000) 2020-02-06T01:34:57.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:34:57.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:57.371023+0000) 2020-02-06T01:34:57.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:34:57.602+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:57.603113+0000) 2020-02-06T01:34:59.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:59.741+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:59.742715+0000) 2020-02-06T01:34:59.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:34:59.988+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:34:59.989388+0000) 2020-02-06T01:35:00.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:00.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.597001+0000) 2020-02-06T01:35:00.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:00.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.875396+0000) 2020-02-06T01:35:00.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:00.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.875617+0000) 2020-02-06T01:35:00.965 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:00.962+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.963920+0000) 2020-02-06T01:35:00.965 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:00.963+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.964516+0000) 2020-02-06T01:35:00.966 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:00.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.964875+0000) 2020-02-06T01:35:00.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:00.964+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.965720+0000) 2020-02-06T01:35:00.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:00.965+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.966238+0000) 2020-02-06T01:35:00.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:00.965+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.966677+0000) 2020-02-06T01:35:00.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:00.965+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.966942+0000) 2020-02-06T01:35:00.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:00.968+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.969155+0000) 2020-02-06T01:35:00.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:00.967+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.969288+0000) 2020-02-06T01:35:00.971 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:00.968+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.969912+0000) 2020-02-06T01:35:00.971 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:00.968+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.969767+0000) 2020-02-06T01:35:00.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:00.968+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.970079+0000) 2020-02-06T01:35:00.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:00.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.975453+0000) 2020-02-06T01:35:00.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:00.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:00.975796+0000) 2020-02-06T01:35:00.986 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:00.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207587 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:02.223 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:35:02.229 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:35:05.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:05.989+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.990992+0000) 2020-02-06T01:35:05.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:05.990+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.991816+0000) 2020-02-06T01:35:05.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:05.992+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.993443+0000) 2020-02-06T01:35:05.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:05.993+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.994398+0000) 2020-02-06T01:35:05.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:05.992+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.994201+0000) 2020-02-06T01:35:05.996 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:05.993+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.994745+0000) 2020-02-06T01:35:05.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:05.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.995792+0000) 2020-02-06T01:35:05.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:05.994+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.996060+0000) 2020-02-06T01:35:05.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:05.995+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.996782+0000) 2020-02-06T01:35:05.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:05.995+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.996981+0000) 2020-02-06T01:35:05.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:05.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.997489+0000) 2020-02-06T01:35:05.999 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:05.996+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.997771+0000) 2020-02-06T01:35:05.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:05.996+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:05.997816+0000) 2020-02-06T01:35:06.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:06.000+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:06.001365+0000) 2020-02-06T01:35:06.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:06.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:06.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:06.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:06.117961+0000) 2020-02-06T01:35:06.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:06.480+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:06.481243+0000) 2020-02-06T01:35:06.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:06.548+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:06.549269+0000) 2020-02-06T01:35:06.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:06.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:06.708035+0000) 2020-02-06T01:35:07.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:07.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:07.099385+0000) 2020-02-06T01:35:07.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:07.369+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:07.370717+0000) 2020-02-06T01:35:07.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:07.369+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:07.370947+0000) 2020-02-06T01:35:07.372 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:07.369+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:07.371271+0000) 2020-02-06T01:35:07.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:07.602+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:07.603305+0000) 2020-02-06T01:35:09.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:09.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:09.742980+0000) 2020-02-06T01:35:09.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:09.988+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:09.989607+0000) 2020-02-06T01:35:10.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:10.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:10.597255+0000) 2020-02-06T01:35:10.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:10.874+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:10.875631+0000) 2020-02-06T01:35:10.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:10.874+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:10.875866+0000) 2020-02-06T01:35:11.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:11.018+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.019303+0000) 2020-02-06T01:35:11.021 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:11.019+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.020163+0000) 2020-02-06T01:35:11.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:11.019+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.021131+0000) 2020-02-06T01:35:11.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:11.021+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.022719+0000) 2020-02-06T01:35:11.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:11.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.023734+0000) 2020-02-06T01:35:11.025 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:11.022+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.023888+0000) 2020-02-06T01:35:11.025 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:11.022+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.024344+0000) 2020-02-06T01:35:11.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:11.023+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.024454+0000) 2020-02-06T01:35:11.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:11.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.024222+0000) 2020-02-06T01:35:11.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:11.023+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.025025+0000) 2020-02-06T01:35:11.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:11.023+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.025283+0000) 2020-02-06T01:35:11.027 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:11.023+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.025201+0000) 2020-02-06T01:35:11.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:11.023+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.025035+0000) 2020-02-06T01:35:11.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:11.027+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:11.028255+0000) 2020-02-06T01:35:11.045 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:11.043+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:16.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:16.050+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.051111+0000) 2020-02-06T01:35:16.053 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:16.050+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.051589+0000) 2020-02-06T01:35:16.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:16.050+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.051877+0000) 2020-02-06T01:35:16.053 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:16.051+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.052452+0000) 2020-02-06T01:35:16.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:16.051+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.052729+0000) 2020-02-06T01:35:16.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:16.052+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.053479+0000) 2020-02-06T01:35:16.054 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:16.052+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.053616+0000) 2020-02-06T01:35:16.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:16.052+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.054229+0000) 2020-02-06T01:35:16.056 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:16.053+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.055129+0000) 2020-02-06T01:35:16.056 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:16.054+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.055459+0000) 2020-02-06T01:35:16.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:16.054+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.055508+0000) 2020-02-06T01:35:16.057 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:16.055+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.056144+0000) 2020-02-06T01:35:16.057 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:16.054+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.056320+0000) 2020-02-06T01:35:16.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:16.061+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.062746+0000) 2020-02-06T01:35:16.074 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:16.071+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207701 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:16.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:16.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.118189+0000) 2020-02-06T01:35:16.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:16.480+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.481480+0000) 2020-02-06T01:35:16.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:16.556+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.556730+0000) 2020-02-06T01:35:16.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:16.707+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:16.708311+0000) 2020-02-06T01:35:17.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:17.098+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:17.099593+0000) 2020-02-06T01:35:17.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:17.370+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:17.370981+0000) 2020-02-06T01:35:17.372 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:17.370+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:17.371209+0000) 2020-02-06T01:35:17.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:17.370+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:17.371472+0000) 2020-02-06T01:35:17.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:17.602+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:17.603536+0000) 2020-02-06T01:35:19.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:19.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:19.743209+0000) 2020-02-06T01:35:19.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:19.989+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:19.989841+0000) 2020-02-06T01:35:20.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:20.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:20.597488+0000) 2020-02-06T01:35:20.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:20.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:20.875852+0000) 2020-02-06T01:35:20.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:20.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:20.876118+0000) 2020-02-06T01:35:21.079 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:21.076+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.077861+0000) 2020-02-06T01:35:21.079 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:21.077+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.078498+0000) 2020-02-06T01:35:21.079 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:21.078+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.079123+0000) 2020-02-06T01:35:21.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:21.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.079717+0000) 2020-02-06T01:35:21.081 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:21.078+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.080089+0000) 2020-02-06T01:35:21.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:21.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.081199+0000) 2020-02-06T01:35:21.082 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:21.080+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.081836+0000) 2020-02-06T01:35:21.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:21.080+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.081990+0000) 2020-02-06T01:35:21.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:21.081+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.082926+0000) 2020-02-06T01:35:21.084 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:21.082+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.083590+0000) 2020-02-06T01:35:21.085 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:21.082+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.083647+0000) 2020-02-06T01:35:21.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:21.082+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.084324+0000) 2020-02-06T01:35:21.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:21.083+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.084690+0000) 2020-02-06T01:35:21.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:21.087+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:21.088677+0000) 2020-02-06T01:35:21.102 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:21.099+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207747 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:26.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:26.104+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.105723+0000) 2020-02-06T01:35:26.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:26.106+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.107431+0000) 2020-02-06T01:35:26.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:26.107+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.108322+0000) 2020-02-06T01:35:26.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:26.107+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.108680+0000) 2020-02-06T01:35:26.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:26.107+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.109009+0000) 2020-02-06T01:35:26.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:26.107+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.109355+0000) 2020-02-06T01:35:26.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:26.108+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.109536+0000) 2020-02-06T01:35:26.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:26.109+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.110256+0000) 2020-02-06T01:35:26.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:26.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.110677+0000) 2020-02-06T01:35:26.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:26.109+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.110922+0000) 2020-02-06T01:35:26.114 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:26.109+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.111073+0000) 2020-02-06T01:35:26.114 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:26.109+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.111317+0000) 2020-02-06T01:35:26.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:26.110+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.112255+0000) 2020-02-06T01:35:26.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:26.116+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.117517+0000) 2020-02-06T01:35:26.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:26.116+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.118404+0000) 2020-02-06T01:35:26.130 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:26.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207781 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:26.482 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:26.480+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.481658+0000) 2020-02-06T01:35:26.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:26.567+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.568351+0000) 2020-02-06T01:35:26.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:26.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:26.708525+0000) 2020-02-06T01:35:27.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:27.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:27.099842+0000) 2020-02-06T01:35:27.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:27.370+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:27.371234+0000) 2020-02-06T01:35:27.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:27.370+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:27.371480+0000) 2020-02-06T01:35:27.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:27.370+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:27.371703+0000) 2020-02-06T01:35:27.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:27.603+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:27.603781+0000) 2020-02-06T01:35:29.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:29.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:29.743411+0000) 2020-02-06T01:35:29.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:29.989+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:29.990104+0000) 2020-02-06T01:35:30.598 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:30.596+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:30.597708+0000) 2020-02-06T01:35:30.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:30.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:30.876122+0000) 2020-02-06T01:35:30.877 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:30.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:30.876373+0000) 2020-02-06T01:35:31.134 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:31.132+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.133407+0000) 2020-02-06T01:35:31.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:31.133+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.134705+0000) 2020-02-06T01:35:31.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:31.134+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.135713+0000) 2020-02-06T01:35:31.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:31.135+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.136260+0000) 2020-02-06T01:35:31.138 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:31.135+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.137379+0000) 2020-02-06T01:35:31.139 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:31.136+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.137670+0000) 2020-02-06T01:35:31.139 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:31.137+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.138331+0000) 2020-02-06T01:35:31.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:31.137+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.138603+0000) 2020-02-06T01:35:31.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:31.137+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.138993+0000) 2020-02-06T01:35:31.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:31.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.139664+0000) 2020-02-06T01:35:31.140 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:31.138+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.140027+0000) 2020-02-06T01:35:31.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:31.139+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.141137+0000) 2020-02-06T01:35:31.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:31.140+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.141498+0000) 2020-02-06T01:35:31.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:31.146+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:31.147697+0000) 2020-02-06T01:35:31.159 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:31.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:32.394 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:35:32.401 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:35:36.119 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:36.118+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.118639+0000) 2020-02-06T01:35:36.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:36.162+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.163707+0000) 2020-02-06T01:35:36.165 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:36.163+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.164475+0000) 2020-02-06T01:35:36.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:36.164+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.165372+0000) 2020-02-06T01:35:36.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:36.165+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.166112+0000) 2020-02-06T01:35:36.167 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:36.165+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.166877+0000) 2020-02-06T01:35:36.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:36.166+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.167033+0000) 2020-02-06T01:35:36.168 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:36.166+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.167454+0000) 2020-02-06T01:35:36.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:36.166+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.167971+0000) 2020-02-06T01:35:36.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:36.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.168841+0000) 2020-02-06T01:35:36.170 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:36.168+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.168632+0000) 2020-02-06T01:35:36.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:36.169+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.169975+0000) 2020-02-06T01:35:36.171 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:36.170+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.170525+0000) 2020-02-06T01:35:36.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:36.170+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.171317+0000) 2020-02-06T01:35:36.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:36.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.176004+0000) 2020-02-06T01:35:36.189 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:36.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:36.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:36.481+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.481902+0000) 2020-02-06T01:35:36.578 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:36.575+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.576639+0000) 2020-02-06T01:35:36.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:36.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:36.708731+0000) 2020-02-06T01:35:37.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:37.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:37.100049+0000) 2020-02-06T01:35:37.372 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:37.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:37.371494+0000) 2020-02-06T01:35:37.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:37.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:37.371702+0000) 2020-02-06T01:35:37.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:37.371+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:37.371927+0000) 2020-02-06T01:35:37.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:37.602+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:37.604028+0000) 2020-02-06T01:35:39.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:39.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:39.743644+0000) 2020-02-06T01:35:39.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:39.990+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:39.990363+0000) 2020-02-06T01:35:40.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:40.597+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:40.597960+0000) 2020-02-06T01:35:40.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:40.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:40.876353+0000) 2020-02-06T01:35:40.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:40.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:40.876598+0000) 2020-02-06T01:35:41.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:41.192+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.193069+0000) 2020-02-06T01:35:41.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:41.194+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.194422+0000) 2020-02-06T01:35:41.196 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:41.195+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.195733+0000) 2020-02-06T01:35:41.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:41.194+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.195957+0000) 2020-02-06T01:35:41.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:41.195+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.196472+0000) 2020-02-06T01:35:41.198 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:41.196+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.196796+0000) 2020-02-06T01:35:41.198 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:41.196+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.197199+0000) 2020-02-06T01:35:41.198 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:41.196+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.197603+0000) 2020-02-06T01:35:41.199 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:41.197+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.198287+0000) 2020-02-06T01:35:41.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:41.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.198678+0000) 2020-02-06T01:35:41.200 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:41.198+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.199423+0000) 2020-02-06T01:35:41.200 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:41.198+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.199001+0000) 2020-02-06T01:35:41.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:41.199+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.200176+0000) 2020-02-06T01:35:41.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:41.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:41.205120+0000) 2020-02-06T01:35:41.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:41.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207905 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:46.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:46.118+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.118889+0000) 2020-02-06T01:35:46.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:46.223+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.223909+0000) 2020-02-06T01:35:46.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:46.224+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.224742+0000) 2020-02-06T01:35:46.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:46.225+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.225197+0000) 2020-02-06T01:35:46.226 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:46.225+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.225657+0000) 2020-02-06T01:35:46.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:46.226+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.226220+0000) 2020-02-06T01:35:46.227 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:46.226+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.226235+0000) 2020-02-06T01:35:46.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:46.226+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.226636+0000) 2020-02-06T01:35:46.228 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:46.226+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.226919+0000) 2020-02-06T01:35:46.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:46.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.227046+0000) 2020-02-06T01:35:46.229 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:46.227+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.227338+0000) 2020-02-06T01:35:46.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:46.228+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.228659+0000) 2020-02-06T01:35:46.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:46.228+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.228723+0000) 2020-02-06T01:35:46.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:46.228+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.228586+0000) 2020-02-06T01:35:46.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:46.232+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.233028+0000) 2020-02-06T01:35:46.247 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:46.246+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:46.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:46.481+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.482163+0000) 2020-02-06T01:35:46.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:46.585+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.585131+0000) 2020-02-06T01:35:46.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:46.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:46.708978+0000) 2020-02-06T01:35:47.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:47.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:47.100230+0000) 2020-02-06T01:35:47.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:47.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:47.371751+0000) 2020-02-06T01:35:47.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:47.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:47.371980+0000) 2020-02-06T01:35:47.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:47.371+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:47.372181+0000) 2020-02-06T01:35:47.605 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:47.604+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:47.604288+0000) 2020-02-06T01:35:49.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:49.742+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:49.743893+0000) 2020-02-06T01:35:49.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:49.989+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:49.990594+0000) 2020-02-06T01:35:50.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:50.597+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:50.598192+0000) 2020-02-06T01:35:50.877 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:50.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:50.876566+0000) 2020-02-06T01:35:50.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:50.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:50.876792+0000) 2020-02-06T01:35:51.253 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:51.250+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.251709+0000) 2020-02-06T01:35:51.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:51.252+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.253588+0000) 2020-02-06T01:35:51.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:51.253+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.253943+0000) 2020-02-06T01:35:51.255 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:51.253+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.254252+0000) 2020-02-06T01:35:51.255 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:51.253+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.254760+0000) 2020-02-06T01:35:51.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:51.253+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.255057+0000) 2020-02-06T01:35:51.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:51.254+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.255607+0000) 2020-02-06T01:35:51.257 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:51.255+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.255844+0000) 2020-02-06T01:35:51.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:51.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.256045+0000) 2020-02-06T01:35:51.257 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:51.254+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.255862+0000) 2020-02-06T01:35:51.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:51.256+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.256699+0000) 2020-02-06T01:35:51.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:51.256+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.256927+0000) 2020-02-06T01:35:51.258 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:51.257+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.257962+0000) 2020-02-06T01:35:51.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:51.260+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:51.261116+0000) 2020-02-06T01:35:51.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:51.275+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 207984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:56.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:56.118+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.119136+0000) 2020-02-06T01:35:56.281 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:35:56.279+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.280211+0000) 2020-02-06T01:35:56.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:56.281+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.282254+0000) 2020-02-06T01:35:56.283 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:35:56.281+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.282646+0000) 2020-02-06T01:35:56.284 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:56.282+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.283089+0000) 2020-02-06T01:35:56.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:56.282+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.283309+0000) 2020-02-06T01:35:56.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:35:56.284+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.284915+0000) 2020-02-06T01:35:56.286 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:56.284+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.285289+0000) 2020-02-06T01:35:56.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:56.285+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.285492+0000) 2020-02-06T01:35:56.288 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:56.285+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.286248+0000) 2020-02-06T01:35:56.290 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:56.285+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.287027+0000) 2020-02-06T01:35:56.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:56.285+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.287021+0000) 2020-02-06T01:35:56.290 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:35:56.286+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.287188+0000) 2020-02-06T01:35:56.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:56.286+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.287234+0000) 2020-02-06T01:35:56.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:56.289+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.290348+0000) 2020-02-06T01:35:56.303 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:35:56.302+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208017 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:35:56.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:35:56.482+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.482492+0000) 2020-02-06T01:35:56.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:56.592+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.593469+0000) 2020-02-06T01:35:56.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:35:56.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:56.709194+0000) 2020-02-06T01:35:57.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:35:57.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:57.100437+0000) 2020-02-06T01:35:57.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:35:57.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:57.372044+0000) 2020-02-06T01:35:57.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:35:57.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:57.372234+0000) 2020-02-06T01:35:57.373 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:35:57.372+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:57.372476+0000) 2020-02-06T01:35:57.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:35:57.603+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:57.604556+0000) 2020-02-06T01:35:59.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:59.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:59.744151+0000) 2020-02-06T01:35:59.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:35:59.989+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:35:59.990819+0000) 2020-02-06T01:36:00.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:00.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:00.598477+0000) 2020-02-06T01:36:00.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:00.875+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:00.876797+0000) 2020-02-06T01:36:00.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:00.875+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:00.876963+0000) 2020-02-06T01:36:01.310 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:01.308+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.309320+0000) 2020-02-06T01:36:01.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:01.309+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.309557+0000) 2020-02-06T01:36:01.311 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:01.309+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.310323+0000) 2020-02-06T01:36:01.311 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:01.309+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.310498+0000) 2020-02-06T01:36:01.312 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:01.310+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.310633+0000) 2020-02-06T01:36:01.312 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:01.309+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.310892+0000) 2020-02-06T01:36:01.312 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:01.310+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.311584+0000) 2020-02-06T01:36:01.313 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:01.310+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.311743+0000) 2020-02-06T01:36:01.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:01.311+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.311965+0000) 2020-02-06T01:36:01.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:01.312+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.312682+0000) 2020-02-06T01:36:01.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:01.311+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.312639+0000) 2020-02-06T01:36:01.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:01.313+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.313748+0000) 2020-02-06T01:36:01.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:01.317+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.318953+0000) 2020-02-06T01:36:01.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:01.318+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:01.319346+0000) 2020-02-06T01:36:01.330 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:01.329+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:02.575 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:36:02.581 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:36:06.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:06.119+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.119414+0000) 2020-02-06T01:36:06.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:06.334+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.336059+0000) 2020-02-06T01:36:06.337 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:06.335+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.336465+0000) 2020-02-06T01:36:06.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:06.335+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.337059+0000) 2020-02-06T01:36:06.338 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:06.336+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.337439+0000) 2020-02-06T01:36:06.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:06.337+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.337733+0000) 2020-02-06T01:36:06.339 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:06.338+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.338471+0000) 2020-02-06T01:36:06.340 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:06.338+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.339031+0000) 2020-02-06T01:36:06.340 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:06.338+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.339239+0000) 2020-02-06T01:36:06.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:06.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.339780+0000) 2020-02-06T01:36:06.341 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:06.339+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.339644+0000) 2020-02-06T01:36:06.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:06.340+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.341406+0000) 2020-02-06T01:36:06.342 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:06.340+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.340895+0000) 2020-02-06T01:36:06.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:06.340+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.341230+0000) 2020-02-06T01:36:06.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:06.341+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.342907+0000) 2020-02-06T01:36:06.360 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:06.359+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208096 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:06.483 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:06.482+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.482726+0000) 2020-02-06T01:36:06.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:06.600+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.601682+0000) 2020-02-06T01:36:06.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:06.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:06.709354+0000) 2020-02-06T01:36:07.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:07.099+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:07.100636+0000) 2020-02-06T01:36:07.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:07.372+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:07.372306+0000) 2020-02-06T01:36:07.373 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:07.372+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:07.372524+0000) 2020-02-06T01:36:07.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:07.372+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:07.372737+0000) 2020-02-06T01:36:07.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:07.604+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:07.604827+0000) 2020-02-06T01:36:09.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:09.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:09.744377+0000) 2020-02-06T01:36:09.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:09.990+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:09.991062+0000) 2020-02-06T01:36:10.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:10.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:10.598736+0000) 2020-02-06T01:36:10.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:10.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:10.877020+0000) 2020-02-06T01:36:10.878 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:10.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:10.877273+0000) 2020-02-06T01:36:11.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:11.364+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.365662+0000) 2020-02-06T01:36:11.367 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:11.364+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.366018+0000) 2020-02-06T01:36:11.367 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:11.365+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.366437+0000) 2020-02-06T01:36:11.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:11.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.367107+0000) 2020-02-06T01:36:11.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:11.367+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.367560+0000) 2020-02-06T01:36:11.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:11.367+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.367926+0000) 2020-02-06T01:36:11.369 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:11.367+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.368904+0000) 2020-02-06T01:36:11.370 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:11.368+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.369051+0000) 2020-02-06T01:36:11.371 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:11.369+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.370140+0000) 2020-02-06T01:36:11.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:11.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.371003+0000) 2020-02-06T01:36:11.372 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:11.370+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.371083+0000) 2020-02-06T01:36:11.373 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:11.370+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.370524+0000) 2020-02-06T01:36:11.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:11.371+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.371581+0000) 2020-02-06T01:36:11.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:11.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:11.375452+0000) 2020-02-06T01:36:11.388 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:11.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208142 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:16.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:16.118+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.119709+0000) 2020-02-06T01:36:16.394 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:16.391+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.393053+0000) 2020-02-06T01:36:16.394 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:16.391+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.392937+0000) 2020-02-06T01:36:16.395 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:16.392+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.393598+0000) 2020-02-06T01:36:16.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:16.393+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.394481+0000) 2020-02-06T01:36:16.397 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:16.394+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.396078+0000) 2020-02-06T01:36:16.397 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:16.395+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.396451+0000) 2020-02-06T01:36:16.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:16.396+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.397201+0000) 2020-02-06T01:36:16.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:16.396+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.397532+0000) 2020-02-06T01:36:16.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:16.396+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.397879+0000) 2020-02-06T01:36:16.399 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:16.397+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.398137+0000) 2020-02-06T01:36:16.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:16.397+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.398774+0000) 2020-02-06T01:36:16.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:16.397+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.399068+0000) 2020-02-06T01:36:16.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:16.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.402125+0000) 2020-02-06T01:36:16.403 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:16.402+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.403097+0000) 2020-02-06T01:36:16.415 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:16.413+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208175 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:16.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:16.481+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.482945+0000) 2020-02-06T01:36:16.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:16.608+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.609551+0000) 2020-02-06T01:36:16.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:16.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:16.709563+0000) 2020-02-06T01:36:17.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:17.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:17.100844+0000) 2020-02-06T01:36:17.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:17.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:17.372542+0000) 2020-02-06T01:36:17.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:17.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:17.372791+0000) 2020-02-06T01:36:17.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:17.371+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:17.372974+0000) 2020-02-06T01:36:17.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:17.604+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:17.605102+0000) 2020-02-06T01:36:19.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:19.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:19.744637+0000) 2020-02-06T01:36:19.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:19.990+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:19.991303+0000) 2020-02-06T01:36:20.600 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:20.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:20.598996+0000) 2020-02-06T01:36:20.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:20.876+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:20.877248+0000) 2020-02-06T01:36:20.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:20.876+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:20.877476+0000) 2020-02-06T01:36:21.422 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:21.419+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.420837+0000) 2020-02-06T01:36:21.422 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:21.419+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.421078+0000) 2020-02-06T01:36:21.422 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:21.420+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.421438+0000) 2020-02-06T01:36:21.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:21.420+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.422025+0000) 2020-02-06T01:36:21.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:21.421+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.422193+0000) 2020-02-06T01:36:21.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:21.421+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.422632+0000) 2020-02-06T01:36:21.424 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:21.422+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.423198+0000) 2020-02-06T01:36:21.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:21.422+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.423625+0000) 2020-02-06T01:36:21.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:21.421+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.423386+0000) 2020-02-06T01:36:21.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:21.423+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.424565+0000) 2020-02-06T01:36:21.426 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:21.424+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.425161+0000) 2020-02-06T01:36:21.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:21.424+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.425548+0000) 2020-02-06T01:36:21.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:21.424+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.425708+0000) 2020-02-06T01:36:21.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:21.428+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:21.429749+0000) 2020-02-06T01:36:21.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:21.441+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:26.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:26.119+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.119980+0000) 2020-02-06T01:36:26.449 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:26.447+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.448443+0000) 2020-02-06T01:36:26.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:26.448+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.449318+0000) 2020-02-06T01:36:26.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:26.448+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.449741+0000) 2020-02-06T01:36:26.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:26.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.450213+0000) 2020-02-06T01:36:26.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:26.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.451100+0000) 2020-02-06T01:36:26.452 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:26.449+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.451030+0000) 2020-02-06T01:36:26.452 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:26.450+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.451751+0000) 2020-02-06T01:36:26.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:26.451+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.452238+0000) 2020-02-06T01:36:26.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:26.451+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.452816+0000) 2020-02-06T01:36:26.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:26.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.453556+0000) 2020-02-06T01:36:26.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:26.451+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.453151+0000) 2020-02-06T01:36:26.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:26.452+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.453839+0000) 2020-02-06T01:36:26.455 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:26.453+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.454430+0000) 2020-02-06T01:36:26.461 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:26.459+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.460495+0000) 2020-02-06T01:36:26.472 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:26.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208253 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:26.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:26.481+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.483256+0000) 2020-02-06T01:36:26.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:26.617+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.617377+0000) 2020-02-06T01:36:26.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:26.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:26.709793+0000) 2020-02-06T01:36:27.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:27.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:27.101060+0000) 2020-02-06T01:36:27.373 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:27.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:27.372759+0000) 2020-02-06T01:36:27.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:27.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:27.373027+0000) 2020-02-06T01:36:27.374 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:27.371+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:27.373166+0000) 2020-02-06T01:36:27.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:27.604+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:27.605332+0000) 2020-02-06T01:36:29.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:29.743+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:29.744816+0000) 2020-02-06T01:36:29.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:29.991+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:29.991529+0000) 2020-02-06T01:36:30.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:30.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:30.599249+0000) 2020-02-06T01:36:30.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:30.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:30.877483+0000) 2020-02-06T01:36:30.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:30.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:30.877705+0000) 2020-02-06T01:36:31.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:31.475+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.476370+0000) 2020-02-06T01:36:31.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:31.475+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.476895+0000) 2020-02-06T01:36:31.479 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:31.477+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.478736+0000) 2020-02-06T01:36:31.480 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:31.478+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.479229+0000) 2020-02-06T01:36:31.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:31.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.479710+0000) 2020-02-06T01:36:31.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:31.478+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.479771+0000) 2020-02-06T01:36:31.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:31.479+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.480204+0000) 2020-02-06T01:36:31.481 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:31.479+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.480720+0000) 2020-02-06T01:36:31.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:31.480+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.481780+0000) 2020-02-06T01:36:31.483 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:31.479+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.481136+0000) 2020-02-06T01:36:31.483 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:31.480+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.481879+0000) 2020-02-06T01:36:31.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:31.481+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.482941+0000) 2020-02-06T01:36:31.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:31.482+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.483132+0000) 2020-02-06T01:36:31.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:31.485+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:31.486140+0000) 2020-02-06T01:36:31.501 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:31.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:32.763 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:36:32.769 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:36:36.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:36.119+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.120234+0000) 2020-02-06T01:36:36.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:36.482+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.483497+0000) 2020-02-06T01:36:36.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:36.503+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.505165+0000) 2020-02-06T01:36:36.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:36.505+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.506479+0000) 2020-02-06T01:36:36.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:36.505+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.507009+0000) 2020-02-06T01:36:36.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:36.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.508341+0000) 2020-02-06T01:36:36.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:36.507+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.508411+0000) 2020-02-06T01:36:36.510 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:36.507+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.509043+0000) 2020-02-06T01:36:36.510 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:36.508+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.509543+0000) 2020-02-06T01:36:36.511 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:36.508+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.509626+0000) 2020-02-06T01:36:36.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:36.508+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.509981+0000) 2020-02-06T01:36:36.511 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:36.509+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.510611+0000) 2020-02-06T01:36:36.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:36.509+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.510878+0000) 2020-02-06T01:36:36.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:36.509+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.511369+0000) 2020-02-06T01:36:36.512 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:36.510+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.511817+0000) 2020-02-06T01:36:36.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:36.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.515891+0000) 2020-02-06T01:36:36.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:36.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208331 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:36.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:36.628+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.629237+0000) 2020-02-06T01:36:36.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:36.708+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:36.710003+0000) 2020-02-06T01:36:37.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:37.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:37.101328+0000) 2020-02-06T01:36:37.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:37.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:37.372997+0000) 2020-02-06T01:36:37.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:37.371+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:37.373321+0000) 2020-02-06T01:36:37.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:37.371+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:37.373379+0000) 2020-02-06T01:36:37.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:37.605+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:37.605568+0000) 2020-02-06T01:36:39.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:39.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:39.745000+0000) 2020-02-06T01:36:39.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:39.991+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:39.991746+0000) 2020-02-06T01:36:40.600 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:40.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:40.599500+0000) 2020-02-06T01:36:40.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:40.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:40.877708+0000) 2020-02-06T01:36:40.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:40.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:40.877900+0000) 2020-02-06T01:36:41.534 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:41.533+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.533642+0000) 2020-02-06T01:36:41.536 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:41.533+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.534787+0000) 2020-02-06T01:36:41.536 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:41.534+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.535000+0000) 2020-02-06T01:36:41.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:41.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.535678+0000) 2020-02-06T01:36:41.537 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:41.536+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.536214+0000) 2020-02-06T01:36:41.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:41.536+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.536530+0000) 2020-02-06T01:36:41.538 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:41.536+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.536890+0000) 2020-02-06T01:36:41.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:41.537+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.537780+0000) 2020-02-06T01:36:41.540 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:41.537+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.538889+0000) 2020-02-06T01:36:41.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:41.537+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.539265+0000) 2020-02-06T01:36:41.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:41.538+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.539665+0000) 2020-02-06T01:36:41.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:41.538+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.540157+0000) 2020-02-06T01:36:41.541 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:41.539+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.540479+0000) 2020-02-06T01:36:41.544 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:41.543+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:41.543303+0000) 2020-02-06T01:36:41.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:41.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:46.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:46.119+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.120509+0000) 2020-02-06T01:36:46.484 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:46.483+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.483803+0000) 2020-02-06T01:36:46.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:46.564+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.565703+0000) 2020-02-06T01:36:46.567 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:46.564+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.566294+0000) 2020-02-06T01:36:46.568 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:46.565+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.566692+0000) 2020-02-06T01:36:46.568 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:46.566+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.567202+0000) 2020-02-06T01:36:46.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:46.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.568714+0000) 2020-02-06T01:36:46.570 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:46.567+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.569115+0000) 2020-02-06T01:36:46.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:46.567+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.569281+0000) 2020-02-06T01:36:46.571 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:46.568+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.569517+0000) 2020-02-06T01:36:46.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:46.568+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.569988+0000) 2020-02-06T01:36:46.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:46.569+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.570452+0000) 2020-02-06T01:36:46.572 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:46.568+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.570353+0000) 2020-02-06T01:36:46.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:46.569+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.570840+0000) 2020-02-06T01:36:46.572 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:46.570+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.571287+0000) 2020-02-06T01:36:46.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:46.575+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.576895+0000) 2020-02-06T01:36:46.588 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:46.585+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208412 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:46.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:46.636+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.637126+0000) 2020-02-06T01:36:46.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:46.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:46.710275+0000) 2020-02-06T01:36:47.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:47.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:47.101515+0000) 2020-02-06T01:36:47.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:47.371+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:47.373166+0000) 2020-02-06T01:36:47.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:47.372+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:47.373495+0000) 2020-02-06T01:36:47.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:47.372+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:47.373633+0000) 2020-02-06T01:36:47.606 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:47.604+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:47.605809+0000) 2020-02-06T01:36:49.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:49.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:49.745278+0000) 2020-02-06T01:36:49.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:49.991+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:49.991970+0000) 2020-02-06T01:36:50.600 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:50.598+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:50.599771+0000) 2020-02-06T01:36:50.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:50.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:50.877934+0000) 2020-02-06T01:36:50.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:50.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:50.878152+0000) 2020-02-06T01:36:51.593 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:51.591+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.592370+0000) 2020-02-06T01:36:51.594 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:51.591+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.592981+0000) 2020-02-06T01:36:51.594 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:51.592+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.593856+0000) 2020-02-06T01:36:51.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:51.593+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.594485+0000) 2020-02-06T01:36:51.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:51.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.595288+0000) 2020-02-06T01:36:51.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:51.593+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.595096+0000) 2020-02-06T01:36:51.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:51.594+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.595839+0000) 2020-02-06T01:36:51.597 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:51.594+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.595614+0000) 2020-02-06T01:36:51.598 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:51.595+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.596496+0000) 2020-02-06T01:36:51.598 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:51.595+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.596971+0000) 2020-02-06T01:36:51.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:51.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.597383+0000) 2020-02-06T01:36:51.599 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:51.596+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.597925+0000) 2020-02-06T01:36:51.599 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:51.597+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.598868+0000) 2020-02-06T01:36:51.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:51.602+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:51.603086+0000) 2020-02-06T01:36:51.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:51.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:56.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:56.120+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.120752+0000) 2020-02-06T01:36:56.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:56.482+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.484004+0000) 2020-02-06T01:36:56.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:56.621+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.623015+0000) 2020-02-06T01:36:56.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:36:56.620+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.622404+0000) 2020-02-06T01:36:56.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:56.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.624167+0000) 2020-02-06T01:36:56.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:36:56.623+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.624607+0000) 2020-02-06T01:36:56.626 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:56.624+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.625257+0000) 2020-02-06T01:36:56.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:56.625+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.626153+0000) 2020-02-06T01:36:56.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:36:56.625+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.626861+0000) 2020-02-06T01:36:56.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:56.625+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.626964+0000) 2020-02-06T01:36:56.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:56.625+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.627385+0000) 2020-02-06T01:36:56.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:56.626+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.627548+0000) 2020-02-06T01:36:56.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:36:56.626+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.627894+0000) 2020-02-06T01:36:56.629 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:56.626+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.628360+0000) 2020-02-06T01:36:56.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:36:56.627+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.628855+0000) 2020-02-06T01:36:56.634 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:56.632+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.633110+0000) 2020-02-06T01:36:56.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:56.644+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.645128+0000) 2020-02-06T01:36:56.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:36:56.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208492 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:36:56.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:36:56.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:56.710492+0000) 2020-02-06T01:36:57.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:36:57.100+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:57.101740+0000) 2020-02-06T01:36:57.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:36:57.372+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:57.373377+0000) 2020-02-06T01:36:57.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:36:57.372+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:57.373665+0000) 2020-02-06T01:36:57.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:36:57.372+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:57.373834+0000) 2020-02-06T01:36:57.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:36:57.605+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:57.606035+0000) 2020-02-06T01:36:59.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:59.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:59.745484+0000) 2020-02-06T01:36:59.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:36:59.991+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:36:59.992227+0000) 2020-02-06T01:37:00.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:00.599+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:00.600017+0000) 2020-02-06T01:37:00.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:00.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:00.878198+0000) 2020-02-06T01:37:00.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:00.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:00.878400+0000) 2020-02-06T01:37:01.653 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:01.651+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.652401+0000) 2020-02-06T01:37:01.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:01.650+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.652273+0000) 2020-02-06T01:37:01.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:01.652+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.653319+0000) 2020-02-06T01:37:01.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:01.652+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.654013+0000) 2020-02-06T01:37:01.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:01.653+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.654456+0000) 2020-02-06T01:37:01.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:01.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.654707+0000) 2020-02-06T01:37:01.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:01.653+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.655302+0000) 2020-02-06T01:37:01.657 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:01.654+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.655833+0000) 2020-02-06T01:37:01.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:01.655+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.656526+0000) 2020-02-06T01:37:01.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:01.656+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.657118+0000) 2020-02-06T01:37:01.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:01.656+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.657633+0000) 2020-02-06T01:37:01.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:01.656+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.657941+0000) 2020-02-06T01:37:01.659 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:01.656+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.658104+0000) 2020-02-06T01:37:01.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:01.661+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:01.662250+0000) 2020-02-06T01:37:01.674 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:01.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:02.935 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:37:02.941 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:37:06.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:06.120+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.121020+0000) 2020-02-06T01:37:06.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:06.483+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.484268+0000) 2020-02-06T01:37:06.651 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:06.649+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.650301+0000) 2020-02-06T01:37:06.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:06.677+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.678243+0000) 2020-02-06T01:37:06.679 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:06.677+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.678754+0000) 2020-02-06T01:37:06.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:06.678+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.679875+0000) 2020-02-06T01:37:06.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:06.679+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.680243+0000) 2020-02-06T01:37:06.681 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:06.680+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.680523+0000) 2020-02-06T01:37:06.682 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:06.680+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.681229+0000) 2020-02-06T01:37:06.682 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:06.680+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.681366+0000) 2020-02-06T01:37:06.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:06.681+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.681716+0000) 2020-02-06T01:37:06.683 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:06.681+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.682591+0000) 2020-02-06T01:37:06.683 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:06.681+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.682168+0000) 2020-02-06T01:37:06.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:06.682+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.683248+0000) 2020-02-06T01:37:06.684 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:06.682+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.682858+0000) 2020-02-06T01:37:06.685 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:06.683+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.683893+0000) 2020-02-06T01:37:06.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:06.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.687545+0000) 2020-02-06T01:37:06.701 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:06.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208574 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:06.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:06.709+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:06.710670+0000) 2020-02-06T01:37:07.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:07.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:07.101959+0000) 2020-02-06T01:37:07.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:07.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:07.373632+0000) 2020-02-06T01:37:07.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:07.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:07.373838+0000) 2020-02-06T01:37:07.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:07.373+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:07.374109+0000) 2020-02-06T01:37:07.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:07.605+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:07.606290+0000) 2020-02-06T01:37:09.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:09.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:09.745716+0000) 2020-02-06T01:37:09.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:09.991+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:09.992467+0000) 2020-02-06T01:37:10.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:10.599+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:10.600305+0000) 2020-02-06T01:37:10.879 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:10.877+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:10.878421+0000) 2020-02-06T01:37:10.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:10.877+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:10.878624+0000) 2020-02-06T01:37:11.707 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:11.705+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.706435+0000) 2020-02-06T01:37:11.708 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:11.706+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.706854+0000) 2020-02-06T01:37:11.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:11.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.708467+0000) 2020-02-06T01:37:11.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:11.708+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.708814+0000) 2020-02-06T01:37:11.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:11.707+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.709053+0000) 2020-02-06T01:37:11.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:11.708+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.709572+0000) 2020-02-06T01:37:11.711 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:11.709+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.709918+0000) 2020-02-06T01:37:11.711 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:11.709+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.710405+0000) 2020-02-06T01:37:11.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:11.709+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.711012+0000) 2020-02-06T01:37:11.712 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:11.711+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.711629+0000) 2020-02-06T01:37:11.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:11.710+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.711451+0000) 2020-02-06T01:37:11.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:11.711+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.712012+0000) 2020-02-06T01:37:11.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:11.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.711919+0000) 2020-02-06T01:37:11.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:11.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:11.716064+0000) 2020-02-06T01:37:11.729 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:11.728+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208619 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:16.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:16.120+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.121278+0000) 2020-02-06T01:37:16.485 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:16.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.484532+0000) 2020-02-06T01:37:16.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:16.658+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.658607+0000) 2020-02-06T01:37:16.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:16.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.710875+0000) 2020-02-06T01:37:16.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:16.734+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.735218+0000) 2020-02-06T01:37:16.736 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:16.734+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.735585+0000) 2020-02-06T01:37:16.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:16.735+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.736140+0000) 2020-02-06T01:37:16.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:16.735+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.736609+0000) 2020-02-06T01:37:16.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:16.736+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.736999+0000) 2020-02-06T01:37:16.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:16.736+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.737215+0000) 2020-02-06T01:37:16.738 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:16.736+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.737930+0000) 2020-02-06T01:37:16.740 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:16.738+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.739021+0000) 2020-02-06T01:37:16.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:16.739+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.739819+0000) 2020-02-06T01:37:16.741 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:16.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.740231+0000) 2020-02-06T01:37:16.741 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:16.739+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.740092+0000) 2020-02-06T01:37:16.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:16.739+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.740313+0000) 2020-02-06T01:37:16.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:16.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.743929+0000) 2020-02-06T01:37:16.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:16.743+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:16.744219+0000) 2020-02-06T01:37:16.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:16.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:17.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:17.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:17.102149+0000) 2020-02-06T01:37:17.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:17.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:17.373832+0000) 2020-02-06T01:37:17.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:17.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:17.374070+0000) 2020-02-06T01:37:17.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:17.373+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:17.374306+0000) 2020-02-06T01:37:17.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:17.606+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:17.606516+0000) 2020-02-06T01:37:19.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:19.744+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:19.745945+0000) 2020-02-06T01:37:19.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:19.992+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:19.992698+0000) 2020-02-06T01:37:20.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:20.600+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:20.600551+0000) 2020-02-06T01:37:20.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:20.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:20.878650+0000) 2020-02-06T01:37:20.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:20.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:20.878891+0000) 2020-02-06T01:37:21.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:21.761+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.762965+0000) 2020-02-06T01:37:21.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:21.762+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.764027+0000) 2020-02-06T01:37:21.766 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:21.764+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.764774+0000) 2020-02-06T01:37:21.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:21.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.766559+0000) 2020-02-06T01:37:21.768 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:21.766+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.766812+0000) 2020-02-06T01:37:21.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:21.766+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.767536+0000) 2020-02-06T01:37:21.768 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:21.766+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.767209+0000) 2020-02-06T01:37:21.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:21.767+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.768150+0000) 2020-02-06T01:37:21.770 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:21.768+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.769162+0000) 2020-02-06T01:37:21.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:21.769+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.769865+0000) 2020-02-06T01:37:21.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:21.769+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.769839+0000) 2020-02-06T01:37:21.771 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:21.769+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.770130+0000) 2020-02-06T01:37:21.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:21.774+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.775701+0000) 2020-02-06T01:37:21.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:21.775+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:21.776137+0000) 2020-02-06T01:37:21.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:21.785+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208700 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:26.122 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:26.121+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.121544+0000) 2020-02-06T01:37:26.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:26.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.484783+0000) 2020-02-06T01:37:26.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:26.668+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.668565+0000) 2020-02-06T01:37:26.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:26.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.711143+0000) 2020-02-06T01:37:26.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:26.789+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.790884+0000) 2020-02-06T01:37:26.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:26.790+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.791393+0000) 2020-02-06T01:37:26.793 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:26.791+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.792232+0000) 2020-02-06T01:37:26.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:26.791+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.792880+0000) 2020-02-06T01:37:26.795 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:26.793+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.794257+0000) 2020-02-06T01:37:26.796 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:26.793+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.794501+0000) 2020-02-06T01:37:26.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:26.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.794848+0000) 2020-02-06T01:37:26.797 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:26.795+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.796166+0000) 2020-02-06T01:37:26.797 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:26.795+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.796408+0000) 2020-02-06T01:37:26.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:26.796+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.797069+0000) 2020-02-06T01:37:26.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:26.796+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.797589+0000) 2020-02-06T01:37:26.799 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:26.796+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.797289+0000) 2020-02-06T01:37:26.799 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:26.797+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.797970+0000) 2020-02-06T01:37:26.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:26.799+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:26.800758+0000) 2020-02-06T01:37:26.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:26.814+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208736 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:27.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:27.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:27.102350+0000) 2020-02-06T01:37:27.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:27.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:27.373995+0000) 2020-02-06T01:37:27.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:27.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:27.374312+0000) 2020-02-06T01:37:27.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:27.374+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:27.374505+0000) 2020-02-06T01:37:27.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:27.606+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:27.606754+0000) 2020-02-06T01:37:29.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:29.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:29.746197+0000) 2020-02-06T01:37:29.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:29.992+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:29.992920+0000) 2020-02-06T01:37:30.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:30.600+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:30.600782+0000) 2020-02-06T01:37:30.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:30.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:30.878929+0000) 2020-02-06T01:37:30.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:30.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:30.879129+0000) 2020-02-06T01:37:31.826 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:31.823+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.823843+0000) 2020-02-06T01:37:31.827 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:31.824+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.824581+0000) 2020-02-06T01:37:31.827 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:31.819+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.820109+0000) 2020-02-06T01:37:31.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:31.820+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.821382+0000) 2020-02-06T01:37:31.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:31.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.823266+0000) 2020-02-06T01:37:31.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:31.820+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.821612+0000) 2020-02-06T01:37:31.828 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:31.821+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.822397+0000) 2020-02-06T01:37:31.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:31.821+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.822589+0000) 2020-02-06T01:37:31.828 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:31.823+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.824651+0000) 2020-02-06T01:37:31.829 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:31.824+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.825298+0000) 2020-02-06T01:37:31.829 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:31.825+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.825730+0000) 2020-02-06T01:37:31.830 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:31.825+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.825845+0000) 2020-02-06T01:37:31.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:31.825+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.826402+0000) 2020-02-06T01:37:31.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:31.827+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:31.828867+0000) 2020-02-06T01:37:31.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:31.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208780 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:33.148 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:37:33.154 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:37:36.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:36.121+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.121810+0000) 2020-02-06T01:37:36.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:36.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.485029+0000) 2020-02-06T01:37:36.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:36.676+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.676616+0000) 2020-02-06T01:37:36.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:36.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.711376+0000) 2020-02-06T01:37:36.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:36.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.848238+0000) 2020-02-06T01:37:36.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:36.849+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.849452+0000) 2020-02-06T01:37:36.851 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:36.849+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.850058+0000) 2020-02-06T01:37:36.851 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:36.850+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.850913+0000) 2020-02-06T01:37:36.852 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:36.851+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.851140+0000) 2020-02-06T01:37:36.853 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:36.852+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.852124+0000) 2020-02-06T01:37:36.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:36.852+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.852460+0000) 2020-02-06T01:37:36.854 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:36.852+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.852730+0000) 2020-02-06T01:37:36.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:36.852+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.853355+0000) 2020-02-06T01:37:36.855 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:36.853+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.853513+0000) 2020-02-06T01:37:36.855 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:36.853+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.854095+0000) 2020-02-06T01:37:36.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:36.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.853758+0000) 2020-02-06T01:37:36.856 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:36.854+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.854424+0000) 2020-02-06T01:37:36.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:36.857+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:36.857403+0000) 2020-02-06T01:37:36.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:36.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:37.103 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:37.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:37.102589+0000) 2020-02-06T01:37:37.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:37.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:37.374230+0000) 2020-02-06T01:37:37.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:37.374+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:37.374532+0000) 2020-02-06T01:37:37.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:37.374+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:37.374679+0000) 2020-02-06T01:37:37.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:37.606+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:37.606961+0000) 2020-02-06T01:37:39.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:39.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:39.746381+0000) 2020-02-06T01:37:39.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:39.992+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:39.993154+0000) 2020-02-06T01:37:40.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:40.600+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:40.601045+0000) 2020-02-06T01:37:40.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:40.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:40.879159+0000) 2020-02-06T01:37:40.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:40.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:40.879362+0000) 2020-02-06T01:37:41.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:41.876+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.877173+0000) 2020-02-06T01:37:41.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:41.877+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.878154+0000) 2020-02-06T01:37:41.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:41.879+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.879412+0000) 2020-02-06T01:37:41.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:41.879+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.879832+0000) 2020-02-06T01:37:41.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:41.879+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.880116+0000) 2020-02-06T01:37:41.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:41.879+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.880767+0000) 2020-02-06T01:37:41.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:41.880+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.881302+0000) 2020-02-06T01:37:41.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:41.880+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.881968+0000) 2020-02-06T01:37:41.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:41.881+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.882410+0000) 2020-02-06T01:37:41.883 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:41.882+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.883152+0000) 2020-02-06T01:37:41.884 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:41.882+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.883240+0000) 2020-02-06T01:37:41.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:41.882+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.883480+0000) 2020-02-06T01:37:41.885 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:41.883+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.883493+0000) 2020-02-06T01:37:41.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:41.888+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:41.889431+0000) 2020-02-06T01:37:41.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:41.899+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:46.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:46.121+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.122080+0000) 2020-02-06T01:37:46.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:46.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.485286+0000) 2020-02-06T01:37:46.686 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:46.683+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.684748+0000) 2020-02-06T01:37:46.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:46.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.711605+0000) 2020-02-06T01:37:46.907 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:46.904+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.905682+0000) 2020-02-06T01:37:46.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:46.905+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.906536+0000) 2020-02-06T01:37:46.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:46.905+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.906917+0000) 2020-02-06T01:37:46.909 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:46.906+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.907959+0000) 2020-02-06T01:37:46.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:46.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.908315+0000) 2020-02-06T01:37:46.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:46.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.908927+0000) 2020-02-06T01:37:46.910 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:46.907+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.909154+0000) 2020-02-06T01:37:46.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:46.908+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.909532+0000) 2020-02-06T01:37:46.911 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:46.908+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.909782+0000) 2020-02-06T01:37:46.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:46.908+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.910047+0000) 2020-02-06T01:37:46.912 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:46.909+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.910753+0000) 2020-02-06T01:37:46.912 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:46.909+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.911371+0000) 2020-02-06T01:37:46.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:46.909+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.911076+0000) 2020-02-06T01:37:46.915 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:46.913+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:46.914307+0000) 2020-02-06T01:37:46.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:46.926+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:47.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:47.101+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:47.102833+0000) 2020-02-06T01:37:47.375 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:47.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:47.374497+0000) 2020-02-06T01:37:47.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:47.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:47.374851+0000) 2020-02-06T01:37:47.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:47.373+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:47.374915+0000) 2020-02-06T01:37:47.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:47.606+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:47.607202+0000) 2020-02-06T01:37:49.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:49.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:49.746634+0000) 2020-02-06T01:37:49.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:49.992+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:49.993330+0000) 2020-02-06T01:37:50.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:50.600+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:50.601308+0000) 2020-02-06T01:37:50.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:50.878+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:50.879386+0000) 2020-02-06T01:37:50.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:50.878+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:50.879623+0000) 2020-02-06T01:37:51.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:51.932+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.934028+0000) 2020-02-06T01:37:51.935 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:51.933+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.934444+0000) 2020-02-06T01:37:51.936 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:51.933+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.935080+0000) 2020-02-06T01:37:51.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:51.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.935362+0000) 2020-02-06T01:37:51.937 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:51.934+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.936140+0000) 2020-02-06T01:37:51.937 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:51.935+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.936748+0000) 2020-02-06T01:37:51.938 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:51.936+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.937064+0000) 2020-02-06T01:37:51.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:51.936+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.937632+0000) 2020-02-06T01:37:51.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:51.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.938375+0000) 2020-02-06T01:37:51.939 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:51.936+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.937876+0000) 2020-02-06T01:37:51.940 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:51.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.938968+0000) 2020-02-06T01:37:51.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:51.938+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.939279+0000) 2020-02-06T01:37:51.941 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:51.938+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.940206+0000) 2020-02-06T01:37:51.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:51.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:51.941848+0000) 2020-02-06T01:37:51.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:51.955+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:56.123 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:56.121+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.122331+0000) 2020-02-06T01:37:56.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:56.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.485537+0000) 2020-02-06T01:37:56.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:56.691+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.692897+0000) 2020-02-06T01:37:56.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:56.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.711839+0000) 2020-02-06T01:37:56.962 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:37:56.959+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.961010+0000) 2020-02-06T01:37:56.964 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:56.961+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.962952+0000) 2020-02-06T01:37:56.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:37:56.963+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.964363+0000) 2020-02-06T01:37:56.966 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:37:56.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.964810+0000) 2020-02-06T01:37:56.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:56.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.965099+0000) 2020-02-06T01:37:56.966 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:56.964+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.965554+0000) 2020-02-06T01:37:56.971 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:56.965+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.966845+0000) 2020-02-06T01:37:56.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:56.966+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.967593+0000) 2020-02-06T01:37:56.971 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:37:56.964+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.966267+0000) 2020-02-06T01:37:56.971 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:56.965+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.967183+0000) 2020-02-06T01:37:56.971 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:56.966+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.968188+0000) 2020-02-06T01:37:56.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:37:56.967+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.968662+0000) 2020-02-06T01:37:56.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:37:56.967+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.968601+0000) 2020-02-06T01:37:56.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:56.974+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:56.975269+0000) 2020-02-06T01:37:56.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:37:56.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 208978 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:37:57.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:37:57.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:57.103063+0000) 2020-02-06T01:37:57.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:37:57.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:57.374773+0000) 2020-02-06T01:37:57.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:37:57.373+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:57.375146+0000) 2020-02-06T01:37:57.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:37:57.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:57.375153+0000) 2020-02-06T01:37:57.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:37:57.606+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:57.607401+0000) 2020-02-06T01:37:59.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:59.745+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:59.746892+0000) 2020-02-06T01:37:59.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:37:59.992+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:37:59.993575+0000) 2020-02-06T01:38:00.602 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:00.600+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:00.601577+0000) 2020-02-06T01:38:00.880 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:00.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:00.879662+0000) 2020-02-06T01:38:00.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:00.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:00.879838+0000) 2020-02-06T01:38:01.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:01.991+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.992070+0000) 2020-02-06T01:38:01.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:01.992+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.993883+0000) 2020-02-06T01:38:01.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:01.993+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.994465+0000) 2020-02-06T01:38:01.997 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:01.994+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.995233+0000) 2020-02-06T01:38:01.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:01.994+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.995424+0000) 2020-02-06T01:38:01.997 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:01.993+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.994834+0000) 2020-02-06T01:38:01.997 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:01.994+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.996120+0000) 2020-02-06T01:38:01.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:01.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.996251+0000) 2020-02-06T01:38:01.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:01.995+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.997010+0000) 2020-02-06T01:38:02.000 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:01.997+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.998828+0000) 2020-02-06T01:38:02.000 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:01.996+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.998142+0000) 2020-02-06T01:38:02.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:01.996+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.998189+0000) 2020-02-06T01:38:02.000 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:01.997+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:01.998883+0000) 2020-02-06T01:38:02.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:02.002+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:02.003629+0000) 2020-02-06T01:38:02.015 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:02.012+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209023 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:03.336 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:38:03.342 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:38:06.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:06.121+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:06.122559+0000) 2020-02-06T01:38:06.486 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:06.484+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:06.485771+0000) 2020-02-06T01:38:06.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:06.703+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:06.704058+0000) 2020-02-06T01:38:06.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:06.710+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:06.712050+0000) 2020-02-06T01:38:07.021 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:07.018+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.020044+0000) 2020-02-06T01:38:07.022 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:07.020+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.021490+0000) 2020-02-06T01:38:07.023 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:07.020+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.022309+0000) 2020-02-06T01:38:07.023 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:07.021+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.022609+0000) 2020-02-06T01:38:07.024 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:07.022+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.023167+0000) 2020-02-06T01:38:07.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:07.022+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.023959+0000) 2020-02-06T01:38:07.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:07.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.024278+0000) 2020-02-06T01:38:07.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:07.024+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.025560+0000) 2020-02-06T01:38:07.026 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:07.023+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.025034+0000) 2020-02-06T01:38:07.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:07.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.026037+0000) 2020-02-06T01:38:07.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:07.025+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.026343+0000) 2020-02-06T01:38:07.027 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:07.025+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.026445+0000) 2020-02-06T01:38:07.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:07.025+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.026700+0000) 2020-02-06T01:38:07.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:07.028+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.029749+0000) 2020-02-06T01:38:07.042 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:07.040+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:07.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:07.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.103327+0000) 2020-02-06T01:38:07.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:07.373+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.374977+0000) 2020-02-06T01:38:07.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:07.373+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.375376+0000) 2020-02-06T01:38:07.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:07.373+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.375353+0000) 2020-02-06T01:38:07.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:07.607+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:07.607608+0000) 2020-02-06T01:38:09.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:09.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:09.747125+0000) 2020-02-06T01:38:09.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:09.993+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:09.993773+0000) 2020-02-06T01:38:10.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:10.601+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:10.601874+0000) 2020-02-06T01:38:10.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:10.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:10.879897+0000) 2020-02-06T01:38:10.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:10.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:10.880135+0000) 2020-02-06T01:38:12.047 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:12.045+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.046394+0000) 2020-02-06T01:38:12.047 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:12.045+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.046954+0000) 2020-02-06T01:38:12.048 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:12.046+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.047469+0000) 2020-02-06T01:38:12.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:12.047+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.048711+0000) 2020-02-06T01:38:12.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:12.048+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.049606+0000) 2020-02-06T01:38:12.051 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:12.048+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.049681+0000) 2020-02-06T01:38:12.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:12.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.049451+0000) 2020-02-06T01:38:12.051 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:12.048+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.050225+0000) 2020-02-06T01:38:12.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:12.049+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.050557+0000) 2020-02-06T01:38:12.052 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:12.049+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.050343+0000) 2020-02-06T01:38:12.052 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:12.050+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.051471+0000) 2020-02-06T01:38:12.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:12.050+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.051632+0000) 2020-02-06T01:38:12.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:12.050+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.052235+0000) 2020-02-06T01:38:12.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:12.055+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:12.056352+0000) 2020-02-06T01:38:12.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:12.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:16.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:16.122+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:16.122798+0000) 2020-02-06T01:38:16.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:16.485+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:16.486024+0000) 2020-02-06T01:38:16.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:16.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:16.712333+0000) 2020-02-06T01:38:16.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:16.712+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:16.714028+0000) 2020-02-06T01:38:17.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:17.074+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.075948+0000) 2020-02-06T01:38:17.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:17.076+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.077407+0000) 2020-02-06T01:38:17.079 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:17.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.078016+0000) 2020-02-06T01:38:17.079 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:17.076+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.078058+0000) 2020-02-06T01:38:17.079 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:17.077+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.078333+0000) 2020-02-06T01:38:17.080 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:17.077+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.079248+0000) 2020-02-06T01:38:17.080 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:17.077+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.078845+0000) 2020-02-06T01:38:17.081 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:17.078+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.079371+0000) 2020-02-06T01:38:17.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:17.078+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.080003+0000) 2020-02-06T01:38:17.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:17.079+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.080450+0000) 2020-02-06T01:38:17.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:17.079+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.080829+0000) 2020-02-06T01:38:17.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:17.080+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.081477+0000) 2020-02-06T01:38:17.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:17.080+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.081903+0000) 2020-02-06T01:38:17.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:17.085+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.086722+0000) 2020-02-06T01:38:17.099 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:17.096+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:17.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:17.102+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.103496+0000) 2020-02-06T01:38:17.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:17.374+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.375211+0000) 2020-02-06T01:38:17.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:17.374+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.375635+0000) 2020-02-06T01:38:17.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:17.374+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.375688+0000) 2020-02-06T01:38:17.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:17.607+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:17.607838+0000) 2020-02-06T01:38:19.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:19.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:19.747320+0000) 2020-02-06T01:38:19.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:19.993+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:19.994007+0000) 2020-02-06T01:38:20.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:20.601+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:20.602116+0000) 2020-02-06T01:38:20.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:20.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:20.880126+0000) 2020-02-06T01:38:20.881 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:20.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:20.880358+0000) 2020-02-06T01:38:22.105 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:22.102+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.104088+0000) 2020-02-06T01:38:22.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:22.103+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.104394+0000) 2020-02-06T01:38:22.106 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:22.104+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.105671+0000) 2020-02-06T01:38:22.107 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:22.105+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.106677+0000) 2020-02-06T01:38:22.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:22.105+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.106983+0000) 2020-02-06T01:38:22.108 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:22.106+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.107422+0000) 2020-02-06T01:38:22.108 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:22.106+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.107780+0000) 2020-02-06T01:38:22.109 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:22.106+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.108289+0000) 2020-02-06T01:38:22.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:22.107+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.108551+0000) 2020-02-06T01:38:22.110 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:22.108+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.109463+0000) 2020-02-06T01:38:22.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:22.108+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.109742+0000) 2020-02-06T01:38:22.111 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:22.108+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.109695+0000) 2020-02-06T01:38:22.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:22.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.116689+0000) 2020-02-06T01:38:22.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:22.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:22.117002+0000) 2020-02-06T01:38:22.129 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:22.126+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:26.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:26.122+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:26.123059+0000) 2020-02-06T01:38:26.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:26.485+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:26.486271+0000) 2020-02-06T01:38:26.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:26.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:26.712559+0000) 2020-02-06T01:38:26.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:26.720+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:26.720748+0000) 2020-02-06T01:38:27.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:27.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.103727+0000) 2020-02-06T01:38:27.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:27.132+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.134060+0000) 2020-02-06T01:38:27.136 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:27.133+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.134777+0000) 2020-02-06T01:38:27.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:27.133+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.135225+0000) 2020-02-06T01:38:27.137 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:27.134+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.136329+0000) 2020-02-06T01:38:27.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:27.136+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.137761+0000) 2020-02-06T01:38:27.139 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:27.137+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.138789+0000) 2020-02-06T01:38:27.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:27.137+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.138649+0000) 2020-02-06T01:38:27.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:27.138+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.140070+0000) 2020-02-06T01:38:27.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:27.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.139558+0000) 2020-02-06T01:38:27.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:27.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.140029+0000) 2020-02-06T01:38:27.142 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:27.138+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.139716+0000) 2020-02-06T01:38:27.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:27.139+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.140335+0000) 2020-02-06T01:38:27.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:27.139+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.140455+0000) 2020-02-06T01:38:27.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:27.143+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.144437+0000) 2020-02-06T01:38:27.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:27.153+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209219 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:27.376 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:27.374+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.375462+0000) 2020-02-06T01:38:27.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:27.374+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.375832+0000) 2020-02-06T01:38:27.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:27.374+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.375932+0000) 2020-02-06T01:38:27.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:27.607+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:27.608129+0000) 2020-02-06T01:38:29.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:29.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:29.747596+0000) 2020-02-06T01:38:29.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:29.994+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:29.994271+0000) 2020-02-06T01:38:30.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:30.601+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:30.602411+0000) 2020-02-06T01:38:30.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:30.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:30.880382+0000) 2020-02-06T01:38:30.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:30.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:30.880609+0000) 2020-02-06T01:38:32.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:32.161+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.161621+0000) 2020-02-06T01:38:32.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:32.159+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.161281+0000) 2020-02-06T01:38:32.163 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:32.161+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.162793+0000) 2020-02-06T01:38:32.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:32.163+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.163480+0000) 2020-02-06T01:38:32.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:32.163+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.164061+0000) 2020-02-06T01:38:32.166 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:32.164+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.164991+0000) 2020-02-06T01:38:32.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:32.165+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.165225+0000) 2020-02-06T01:38:32.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:32.165+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.165421+0000) 2020-02-06T01:38:32.167 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:32.165+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.165799+0000) 2020-02-06T01:38:32.167 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:32.165+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.165825+0000) 2020-02-06T01:38:32.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:32.165+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.165603+0000) 2020-02-06T01:38:32.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:32.165+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.166663+0000) 2020-02-06T01:38:32.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:32.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.166825+0000) 2020-02-06T01:38:32.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:32.172+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:32.172757+0000) 2020-02-06T01:38:32.184 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:32.182+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:33.552 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:38:33.558 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:38:36.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:36.122+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:36.123309+0000) 2020-02-06T01:38:36.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:36.486+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:36.486497+0000) 2020-02-06T01:38:36.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:36.711+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:36.712806+0000) 2020-02-06T01:38:36.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:36.727+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:36.728866+0000) 2020-02-06T01:38:37.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:37.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.104001+0000) 2020-02-06T01:38:37.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:37.187+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.188514+0000) 2020-02-06T01:38:37.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:37.188+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.189780+0000) 2020-02-06T01:38:37.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:37.189+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.190333+0000) 2020-02-06T01:38:37.192 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:37.190+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.191449+0000) 2020-02-06T01:38:37.193 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:37.191+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.192223+0000) 2020-02-06T01:38:37.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:37.191+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.192409+0000) 2020-02-06T01:38:37.194 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:37.192+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.192930+0000) 2020-02-06T01:38:37.194 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:37.192+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.193221+0000) 2020-02-06T01:38:37.194 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:37.192+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.193488+0000) 2020-02-06T01:38:37.195 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:37.194+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.194794+0000) 2020-02-06T01:38:37.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:37.194+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.195277+0000) 2020-02-06T01:38:37.196 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:37.195+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.195654+0000) 2020-02-06T01:38:37.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:37.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.196049+0000) 2020-02-06T01:38:37.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:37.199+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.200501+0000) 2020-02-06T01:38:37.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:37.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209297 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:37.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:37.375+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.375698+0000) 2020-02-06T01:38:37.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:37.375+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.376048+0000) 2020-02-06T01:38:37.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:37.375+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.376193+0000) 2020-02-06T01:38:37.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:37.607+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:37.608342+0000) 2020-02-06T01:38:39.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:39.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:39.747820+0000) 2020-02-06T01:38:39.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:39.993+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:39.994520+0000) 2020-02-06T01:38:40.603 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:40.602+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:40.602694+0000) 2020-02-06T01:38:40.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:40.879+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:40.880617+0000) 2020-02-06T01:38:40.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:40.879+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:40.880819+0000) 2020-02-06T01:38:42.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:42.217+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.217683+0000) 2020-02-06T01:38:42.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:42.216+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.217923+0000) 2020-02-06T01:38:42.220 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:42.217+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.218171+0000) 2020-02-06T01:38:42.220 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:42.217+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.218806+0000) 2020-02-06T01:38:42.220 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:42.218+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.219281+0000) 2020-02-06T01:38:42.220 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:42.218+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.219475+0000) 2020-02-06T01:38:42.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:42.219+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.220416+0000) 2020-02-06T01:38:42.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:42.220+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.220422+0000) 2020-02-06T01:38:42.222 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:42.220+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.221132+0000) 2020-02-06T01:38:42.223 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:42.221+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.221636+0000) 2020-02-06T01:38:42.223 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:42.220+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.221141+0000) 2020-02-06T01:38:42.223 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:42.221+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.221891+0000) 2020-02-06T01:38:42.223 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:42.222+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.222549+0000) 2020-02-06T01:38:42.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:42.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:42.228401+0000) 2020-02-06T01:38:42.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:42.238+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:46.124 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:46.123+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:46.123561+0000) 2020-02-06T01:38:46.487 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:46.486+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:46.486731+0000) 2020-02-06T01:38:46.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:46.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:46.713037+0000) 2020-02-06T01:38:46.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:46.736+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:46.736985+0000) 2020-02-06T01:38:47.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:47.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.104294+0000) 2020-02-06T01:38:47.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:47.242+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.243853+0000) 2020-02-06T01:38:47.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:47.243+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.244510+0000) 2020-02-06T01:38:47.245 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:47.243+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.244923+0000) 2020-02-06T01:38:47.246 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:47.244+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.245422+0000) 2020-02-06T01:38:47.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:47.244+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.245828+0000) 2020-02-06T01:38:47.247 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:47.245+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.246390+0000) 2020-02-06T01:38:47.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:47.246+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.247391+0000) 2020-02-06T01:38:47.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:47.248+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.248483+0000) 2020-02-06T01:38:47.249 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:47.248+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.248504+0000) 2020-02-06T01:38:47.250 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:47.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.249253+0000) 2020-02-06T01:38:47.251 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:47.249+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.249833+0000) 2020-02-06T01:38:47.252 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:47.249+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.250092+0000) 2020-02-06T01:38:47.252 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:47.249+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.249471+0000) 2020-02-06T01:38:47.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:47.251+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.252983+0000) 2020-02-06T01:38:47.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:47.266+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209377 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:47.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:47.375+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.375928+0000) 2020-02-06T01:38:47.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:47.375+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.376250+0000) 2020-02-06T01:38:47.377 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:47.375+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.376382+0000) 2020-02-06T01:38:47.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:47.607+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:47.608633+0000) 2020-02-06T01:38:49.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:49.746+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:49.748043+0000) 2020-02-06T01:38:49.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:49.993+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:49.994762+0000) 2020-02-06T01:38:50.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:50.602+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:50.602953+0000) 2020-02-06T01:38:50.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:50.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:50.880887+0000) 2020-02-06T01:38:50.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:50.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:50.881021+0000) 2020-02-06T01:38:52.274 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:52.272+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.273564+0000) 2020-02-06T01:38:52.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:52.273+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.274875+0000) 2020-02-06T01:38:52.276 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:52.274+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.275149+0000) 2020-02-06T01:38:52.276 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:52.274+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.275543+0000) 2020-02-06T01:38:52.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:52.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.276462+0000) 2020-02-06T01:38:52.277 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:52.275+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.276629+0000) 2020-02-06T01:38:52.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:52.276+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.276975+0000) 2020-02-06T01:38:52.278 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:52.277+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.277594+0000) 2020-02-06T01:38:52.279 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:52.277+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.278306+0000) 2020-02-06T01:38:52.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:52.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.278728+0000) 2020-02-06T01:38:52.280 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:52.278+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.279114+0000) 2020-02-06T01:38:52.280 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:52.278+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.279382+0000) 2020-02-06T01:38:52.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:52.278+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.279225+0000) 2020-02-06T01:38:52.284 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:52.282+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:52.283906+0000) 2020-02-06T01:38:52.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:52.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209417 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:56.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:56.123+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:56.123853+0000) 2020-02-06T01:38:56.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:56.486+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:56.486991+0000) 2020-02-06T01:38:56.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:56.712+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:56.713307+0000) 2020-02-06T01:38:56.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:56.744+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:56.745215+0000) 2020-02-06T01:38:57.105 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:57.103+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.104534+0000) 2020-02-06T01:38:57.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:57.299+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.300251+0000) 2020-02-06T01:38:57.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:57.301+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.301651+0000) 2020-02-06T01:38:57.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:57.301+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.301557+0000) 2020-02-06T01:38:57.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:38:57.300+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.302046+0000) 2020-02-06T01:38:57.303 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:38:57.301+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.302278+0000) 2020-02-06T01:38:57.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:38:57.301+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.302544+0000) 2020-02-06T01:38:57.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:38:57.301+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.302348+0000) 2020-02-06T01:38:57.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:57.302+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.303212+0000) 2020-02-06T01:38:57.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:57.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.302900+0000) 2020-02-06T01:38:57.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:57.302+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.303668+0000) 2020-02-06T01:38:57.305 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:38:57.302+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.303459+0000) 2020-02-06T01:38:57.306 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:38:57.303+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.304274+0000) 2020-02-06T01:38:57.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:38:57.304+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.304604+0000) 2020-02-06T01:38:57.312 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:57.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.311016+0000) 2020-02-06T01:38:57.323 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:38:57.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209456 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:38:57.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:38:57.375+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.376163+0000) 2020-02-06T01:38:57.377 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:38:57.376+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.376512+0000) 2020-02-06T01:38:57.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:38:57.376+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.376628+0000) 2020-02-06T01:38:57.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:38:57.608+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:57.608878+0000) 2020-02-06T01:38:59.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:59.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:59.748317+0000) 2020-02-06T01:38:59.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:38:59.994+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:38:59.994984+0000) 2020-02-06T01:39:00.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:00.602+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:00.603199+0000) 2020-02-06T01:39:00.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:00.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:00.881067+0000) 2020-02-06T01:39:00.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:00.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:00.881269+0000) 2020-02-06T01:39:02.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:02.327+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.328592+0000) 2020-02-06T01:39:02.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:02.328+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.329390+0000) 2020-02-06T01:39:02.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:02.328+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.329692+0000) 2020-02-06T01:39:02.331 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:02.329+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.330380+0000) 2020-02-06T01:39:02.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:02.330+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.330745+0000) 2020-02-06T01:39:02.332 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:02.330+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.331429+0000) 2020-02-06T01:39:02.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:02.332+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.333208+0000) 2020-02-06T01:39:02.335 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:02.332+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.333562+0000) 2020-02-06T01:39:02.335 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:02.332+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.332898+0000) 2020-02-06T01:39:02.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:02.333+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.333824+0000) 2020-02-06T01:39:02.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:02.333+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.334044+0000) 2020-02-06T01:39:02.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:02.333+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.334451+0000) 2020-02-06T01:39:02.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:02.334+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.334500+0000) 2020-02-06T01:39:02.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:02.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:02.339185+0000) 2020-02-06T01:39:02.352 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:02.350+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209495 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:03.731 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:39:03.737 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:39:06.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:06.123+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:06.124116+0000) 2020-02-06T01:39:06.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:06.487+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:06.487314+0000) 2020-02-06T01:39:06.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:06.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:06.713535+0000) 2020-02-06T01:39:06.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:06.753+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:06.753664+0000) 2020-02-06T01:39:07.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:07.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.104801+0000) 2020-02-06T01:39:07.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:07.355+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.356935+0000) 2020-02-06T01:39:07.359 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:07.356+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.357313+0000) 2020-02-06T01:39:07.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:07.357+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.358128+0000) 2020-02-06T01:39:07.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:07.358+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.358667+0000) 2020-02-06T01:39:07.360 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:07.357+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.358942+0000) 2020-02-06T01:39:07.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:07.359+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.359541+0000) 2020-02-06T01:39:07.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:07.358+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.359891+0000) 2020-02-06T01:39:07.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:07.359+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.360384+0000) 2020-02-06T01:39:07.361 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:07.359+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.360611+0000) 2020-02-06T01:39:07.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:07.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.361309+0000) 2020-02-06T01:39:07.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:07.361+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.362018+0000) 2020-02-06T01:39:07.363 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:07.362+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.362555+0000) 2020-02-06T01:39:07.363 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:07.362+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.362720+0000) 2020-02-06T01:39:07.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:07.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.367353+0000) 2020-02-06T01:39:07.377 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:07.375+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.376382+0000) 2020-02-06T01:39:07.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:07.376+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.376720+0000) 2020-02-06T01:39:07.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:07.376+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.376833+0000) 2020-02-06T01:39:07.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:07.380+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209535 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:07.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:07.608+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:07.609121+0000) 2020-02-06T01:39:09.749 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:09.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:09.748534+0000) 2020-02-06T01:39:09.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:09.994+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:09.995204+0000) 2020-02-06T01:39:10.604 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:10.602+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:10.603478+0000) 2020-02-06T01:39:10.882 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:10.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:10.881352+0000) 2020-02-06T01:39:10.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:10.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:10.881474+0000) 2020-02-06T01:39:12.386 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:12.384+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.385531+0000) 2020-02-06T01:39:12.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:12.385+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.386165+0000) 2020-02-06T01:39:12.390 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:12.387+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.388942+0000) 2020-02-06T01:39:12.390 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:12.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.389362+0000) 2020-02-06T01:39:12.390 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:12.388+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.389894+0000) 2020-02-06T01:39:12.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:12.388+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.390061+0000) 2020-02-06T01:39:12.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:12.388+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.389944+0000) 2020-02-06T01:39:12.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:12.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.390434+0000) 2020-02-06T01:39:12.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:12.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.391530+0000) 2020-02-06T01:39:12.393 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:12.391+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.392618+0000) 2020-02-06T01:39:12.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:12.391+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.392556+0000) 2020-02-06T01:39:12.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:12.391+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.393022+0000) 2020-02-06T01:39:12.394 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:12.391+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.393336+0000) 2020-02-06T01:39:12.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:12.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:12.396798+0000) 2020-02-06T01:39:12.411 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:12.408+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209574 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:16.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:16.123+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:16.124359+0000) 2020-02-06T01:39:16.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:16.486+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:16.487571+0000) 2020-02-06T01:39:16.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:16.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:16.713767+0000) 2020-02-06T01:39:16.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:16.760+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:16.761961+0000) 2020-02-06T01:39:17.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:17.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.105126+0000) 2020-02-06T01:39:17.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:17.375+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.376635+0000) 2020-02-06T01:39:17.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:17.375+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.376897+0000) 2020-02-06T01:39:17.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:17.375+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.377060+0000) 2020-02-06T01:39:17.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:17.414+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.415997+0000) 2020-02-06T01:39:17.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:17.415+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.416830+0000) 2020-02-06T01:39:17.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:17.415+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.416678+0000) 2020-02-06T01:39:17.418 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:17.416+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.417571+0000) 2020-02-06T01:39:17.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:17.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.418120+0000) 2020-02-06T01:39:17.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:17.417+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.418621+0000) 2020-02-06T01:39:17.420 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:17.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.419568+0000) 2020-02-06T01:39:17.421 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:17.419+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.420159+0000) 2020-02-06T01:39:17.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:17.419+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.420246+0000) 2020-02-06T01:39:17.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:17.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.420378+0000) 2020-02-06T01:39:17.422 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:17.419+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.420611+0000) 2020-02-06T01:39:17.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:17.418+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.420332+0000) 2020-02-06T01:39:17.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:17.419+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.421245+0000) 2020-02-06T01:39:17.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:17.423+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.424803+0000) 2020-02-06T01:39:17.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:17.435+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:17.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:17.609+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:17.609387+0000) 2020-02-06T01:39:19.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:19.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:19.748776+0000) 2020-02-06T01:39:19.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:19.995+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:19.995427+0000) 2020-02-06T01:39:20.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:20.602+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:20.603732+0000) 2020-02-06T01:39:20.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:20.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:20.881584+0000) 2020-02-06T01:39:20.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:20.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:20.881674+0000) 2020-02-06T01:39:22.444 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:22.442+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.443156+0000) 2020-02-06T01:39:22.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:22.442+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.443621+0000) 2020-02-06T01:39:22.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:22.443+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.444494+0000) 2020-02-06T01:39:22.445 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:22.442+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.444380+0000) 2020-02-06T01:39:22.446 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:22.444+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.445340+0000) 2020-02-06T01:39:22.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:22.444+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.445700+0000) 2020-02-06T01:39:22.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:22.444+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.446004+0000) 2020-02-06T01:39:22.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:22.445+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.446244+0000) 2020-02-06T01:39:22.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:22.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.446511+0000) 2020-02-06T01:39:22.448 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:22.445+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.446983+0000) 2020-02-06T01:39:22.448 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:22.447+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.447510+0000) 2020-02-06T01:39:22.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:22.447+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.448502+0000) 2020-02-06T01:39:22.450 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:22.447+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.448909+0000) 2020-02-06T01:39:22.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:22.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:22.453538+0000) 2020-02-06T01:39:22.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:22.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209654 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:26.125 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:26.123+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:26.124664+0000) 2020-02-06T01:39:26.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:26.486+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:26.487794+0000) 2020-02-06T01:39:26.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:26.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:26.713990+0000) 2020-02-06T01:39:26.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:26.770+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:26.771013+0000) 2020-02-06T01:39:27.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:27.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.105373+0000) 2020-02-06T01:39:27.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:27.376+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.376886+0000) 2020-02-06T01:39:27.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:27.376+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.377162+0000) 2020-02-06T01:39:27.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:27.376+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.377311+0000) 2020-02-06T01:39:27.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:27.473+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.475171+0000) 2020-02-06T01:39:27.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:27.475+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.476496+0000) 2020-02-06T01:39:27.478 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:27.475+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.476957+0000) 2020-02-06T01:39:27.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:27.476+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.477832+0000) 2020-02-06T01:39:27.479 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:27.477+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.478326+0000) 2020-02-06T01:39:27.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:27.477+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.478972+0000) 2020-02-06T01:39:27.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:27.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.479599+0000) 2020-02-06T01:39:27.482 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:27.479+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.480884+0000) 2020-02-06T01:39:27.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:27.480+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.481497+0000) 2020-02-06T01:39:27.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:27.480+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.481153+0000) 2020-02-06T01:39:27.483 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:27.480+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.481216+0000) 2020-02-06T01:39:27.483 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:27.480+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.481689+0000) 2020-02-06T01:39:27.484 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:27.480+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.481844+0000) 2020-02-06T01:39:27.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:27.481+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.482584+0000) 2020-02-06T01:39:27.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:27.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:27.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:27.608+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:27.609604+0000) 2020-02-06T01:39:29.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:29.747+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:29.748995+0000) 2020-02-06T01:39:29.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:29.994+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:29.995654+0000) 2020-02-06T01:39:30.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:30.603+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:30.603961+0000) 2020-02-06T01:39:30.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:30.880+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:30.881834+0000) 2020-02-06T01:39:30.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:30.880+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:30.881883+0000) 2020-02-06T01:39:32.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:32.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.504441+0000) 2020-02-06T01:39:32.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:32.504+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.505115+0000) 2020-02-06T01:39:32.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:32.504+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.506140+0000) 2020-02-06T01:39:32.508 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:32.504+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.505478+0000) 2020-02-06T01:39:32.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:32.504+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.505895+0000) 2020-02-06T01:39:32.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:32.505+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.506384+0000) 2020-02-06T01:39:32.510 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:32.505+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.507278+0000) 2020-02-06T01:39:32.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:32.506+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.508365+0000) 2020-02-06T01:39:32.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:32.506+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.508185+0000) 2020-02-06T01:39:32.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:32.507+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.508818+0000) 2020-02-06T01:39:32.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:32.506+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.507838+0000) 2020-02-06T01:39:32.511 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:32.507+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.508547+0000) 2020-02-06T01:39:32.512 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:32.508+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.509562+0000) 2020-02-06T01:39:32.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:32.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:32.515342+0000) 2020-02-06T01:39:32.526 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:32.523+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:33.868 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:39:33.874 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:39:36.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:36.124+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:36.124910+0000) 2020-02-06T01:39:36.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:36.487+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:36.488064+0000) 2020-02-06T01:39:36.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:36.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:36.714209+0000) 2020-02-06T01:39:36.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:36.777+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:36.779027+0000) 2020-02-06T01:39:37.106 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:37.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.105589+0000) 2020-02-06T01:39:37.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:37.376+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.377139+0000) 2020-02-06T01:39:37.378 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:37.376+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.377394+0000) 2020-02-06T01:39:37.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:37.376+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.377535+0000) 2020-02-06T01:39:37.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:37.530+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.532264+0000) 2020-02-06T01:39:37.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:37.531+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.532915+0000) 2020-02-06T01:39:37.535 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:37.532+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.533689+0000) 2020-02-06T01:39:37.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:37.532+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.533850+0000) 2020-02-06T01:39:37.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:37.532+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.534040+0000) 2020-02-06T01:39:37.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:37.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.534629+0000) 2020-02-06T01:39:37.536 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:37.534+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.535059+0000) 2020-02-06T01:39:37.536 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:37.534+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.535113+0000) 2020-02-06T01:39:37.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:37.534+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.535720+0000) 2020-02-06T01:39:37.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:37.534+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.536228+0000) 2020-02-06T01:39:37.537 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:37.534+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.536249+0000) 2020-02-06T01:39:37.538 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:37.534+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.535930+0000) 2020-02-06T01:39:37.538 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:37.535+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.536717+0000) 2020-02-06T01:39:37.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:37.539+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.540566+0000) 2020-02-06T01:39:37.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:37.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209775 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:37.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:37.608+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:37.609815+0000) 2020-02-06T01:39:39.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:39.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:39.749271+0000) 2020-02-06T01:39:39.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:39.995+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:39.995876+0000) 2020-02-06T01:39:40.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:40.603+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:40.604215+0000) 2020-02-06T01:39:40.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:40.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:40.882128+0000) 2020-02-06T01:39:40.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:40.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:40.882123+0000) 2020-02-06T01:39:42.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:42.557+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.558744+0000) 2020-02-06T01:39:42.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:42.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.560471+0000) 2020-02-06T01:39:42.562 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:42.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.560881+0000) 2020-02-06T01:39:42.562 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:42.560+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.561605+0000) 2020-02-06T01:39:42.563 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:42.560+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.561918+0000) 2020-02-06T01:39:42.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:42.561+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.562549+0000) 2020-02-06T01:39:42.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:42.562+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.563187+0000) 2020-02-06T01:39:42.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:42.562+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.563424+0000) 2020-02-06T01:39:42.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:42.562+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.564242+0000) 2020-02-06T01:39:42.565 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:42.563+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.564754+0000) 2020-02-06T01:39:42.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:42.563+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.564742+0000) 2020-02-06T01:39:42.566 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:42.563+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.565023+0000) 2020-02-06T01:39:42.566 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:42.564+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.565737+0000) 2020-02-06T01:39:42.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:42.570+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:42.571445+0000) 2020-02-06T01:39:42.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:42.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:46.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:46.124+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:46.125151+0000) 2020-02-06T01:39:46.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:46.487+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:46.488494+0000) 2020-02-06T01:39:46.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:46.713+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:46.714468+0000) 2020-02-06T01:39:46.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:46.785+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:46.786886+0000) 2020-02-06T01:39:47.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:47.104+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.105839+0000) 2020-02-06T01:39:47.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:47.376+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.377394+0000) 2020-02-06T01:39:47.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:47.376+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.377728+0000) 2020-02-06T01:39:47.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:47.376+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.377794+0000) 2020-02-06T01:39:47.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:47.585+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.586138+0000) 2020-02-06T01:39:47.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:47.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.587682+0000) 2020-02-06T01:39:47.589 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:47.586+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.588013+0000) 2020-02-06T01:39:47.589 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:47.587+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.588564+0000) 2020-02-06T01:39:47.590 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:47.587+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.589271+0000) 2020-02-06T01:39:47.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:47.588+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.589916+0000) 2020-02-06T01:39:47.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:47.588+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.589990+0000) 2020-02-06T01:39:47.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:47.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.590344+0000) 2020-02-06T01:39:47.591 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:47.589+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.590630+0000) 2020-02-06T01:39:47.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:47.589+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.590966+0000) 2020-02-06T01:39:47.592 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:47.590+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.591298+0000) 2020-02-06T01:39:47.593 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:47.590+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.591706+0000) 2020-02-06T01:39:47.593 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:47.590+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.591911+0000) 2020-02-06T01:39:47.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:47.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.597109+0000) 2020-02-06T01:39:47.610 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:47.608+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:47.610 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:47.609+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:47.610012+0000) 2020-02-06T01:39:49.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:49.748+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:49.749467+0000) 2020-02-06T01:39:49.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:49.995+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:49.996130+0000) 2020-02-06T01:39:50.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:50.603+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:50.604470+0000) 2020-02-06T01:39:50.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:50.881+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:50.882392+0000) 2020-02-06T01:39:50.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:50.881+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:50.882390+0000) 2020-02-06T01:39:52.616 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:52.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.615059+0000) 2020-02-06T01:39:52.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:52.614+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.615737+0000) 2020-02-06T01:39:52.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:52.615+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.616149+0000) 2020-02-06T01:39:52.617 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:52.615+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.616683+0000) 2020-02-06T01:39:52.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:52.616+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.617680+0000) 2020-02-06T01:39:52.619 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:52.616+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.618315+0000) 2020-02-06T01:39:52.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:52.618+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.619301+0000) 2020-02-06T01:39:52.621 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:52.618+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.620196+0000) 2020-02-06T01:39:52.622 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:52.619+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.620890+0000) 2020-02-06T01:39:52.622 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:52.620+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.621278+0000) 2020-02-06T01:39:52.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:52.620+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.621223+0000) 2020-02-06T01:39:52.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:52.620+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.621648+0000) 2020-02-06T01:39:52.623 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:52.620+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.621964+0000) 2020-02-06T01:39:52.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:52.625+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:52.626321+0000) 2020-02-06T01:39:52.639 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:52.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:56.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:56.124+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:56.125397+0000) 2020-02-06T01:39:56.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:56.488+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:56.488735+0000) 2020-02-06T01:39:56.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:56.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:56.714694+0000) 2020-02-06T01:39:56.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:56.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:56.795922+0000) 2020-02-06T01:39:57.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:57.105+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.106069+0000) 2020-02-06T01:39:57.379 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:57.377+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.377740+0000) 2020-02-06T01:39:57.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:57.377+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.378045+0000) 2020-02-06T01:39:57.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:57.377+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.378069+0000) 2020-02-06T01:39:57.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:57.609+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.610193+0000) 2020-02-06T01:39:57.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:39:57.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.643719+0000) 2020-02-06T01:39:57.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:39:57.643+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.644259+0000) 2020-02-06T01:39:57.646 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:39:57.643+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.645044+0000) 2020-02-06T01:39:57.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:39:57.644+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.645434+0000) 2020-02-06T01:39:57.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:39:57.644+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.645910+0000) 2020-02-06T01:39:57.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:39:57.645+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.646495+0000) 2020-02-06T01:39:57.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:39:57.646+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.647562+0000) 2020-02-06T01:39:57.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:39:57.647+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.648993+0000) 2020-02-06T01:39:57.650 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:57.648+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.649676+0000) 2020-02-06T01:39:57.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:39:57.648+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.649715+0000) 2020-02-06T01:39:57.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:39:57.648+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.650303+0000) 2020-02-06T01:39:57.651 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:39:57.649+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.650485+0000) 2020-02-06T01:39:57.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:57.649+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.650348+0000) 2020-02-06T01:39:57.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:57.650+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:57.651967+0000) 2020-02-06T01:39:57.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:39:57.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209935 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:39:59.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:59.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:59.749700+0000) 2020-02-06T01:39:59.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:39:59.995+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:39:59.996348+0000) 2020-02-06T01:40:00.605 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:00.604+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:00.604656+0000) 2020-02-06T01:40:00.883 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:00.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:00.882664+0000) 2020-02-06T01:40:00.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:00.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:00.882661+0000) 2020-02-06T01:40:02.675 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:02.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.674264+0000) 2020-02-06T01:40:02.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:02.673+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.674733+0000) 2020-02-06T01:40:02.676 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:02.674+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.675285+0000) 2020-02-06T01:40:02.677 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:02.675+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.676532+0000) 2020-02-06T01:40:02.678 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:02.676+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.677215+0000) 2020-02-06T01:40:02.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:02.677+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.677958+0000) 2020-02-06T01:40:02.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:02.677+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.678258+0000) 2020-02-06T01:40:02.680 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:02.678+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.679251+0000) 2020-02-06T01:40:02.680 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:02.678+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.679103+0000) 2020-02-06T01:40:02.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:02.678+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.679910+0000) 2020-02-06T01:40:02.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:02.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.681018+0000) 2020-02-06T01:40:02.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:02.681+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.681754+0000) 2020-02-06T01:40:02.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:02.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.681900+0000) 2020-02-06T01:40:02.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:02.681+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:02.682135+0000) 2020-02-06T01:40:02.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:02.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 209975 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:04.003 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:40:04.009 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:40:06.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:06.125+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:06.125665+0000) 2020-02-06T01:40:06.490 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:06.488+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:06.488984+0000) 2020-02-06T01:40:06.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:06.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:06.714923+0000) 2020-02-06T01:40:06.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:06.806+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:06.807759+0000) 2020-02-06T01:40:07.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:07.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.106344+0000) 2020-02-06T01:40:07.379 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:07.377+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.377980+0000) 2020-02-06T01:40:07.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:07.377+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.378289+0000) 2020-02-06T01:40:07.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:07.377+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.378273+0000) 2020-02-06T01:40:07.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:07.610+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.610460+0000) 2020-02-06T01:40:07.704 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:07.702+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.702667+0000) 2020-02-06T01:40:07.705 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:07.703+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.704166+0000) 2020-02-06T01:40:07.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:07.703+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.704536+0000) 2020-02-06T01:40:07.706 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:07.703+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.704946+0000) 2020-02-06T01:40:07.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:07.704+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.705189+0000) 2020-02-06T01:40:07.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:07.705+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.705655+0000) 2020-02-06T01:40:07.707 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:07.705+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.706592+0000) 2020-02-06T01:40:07.708 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:07.707+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.707695+0000) 2020-02-06T01:40:07.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:07.707+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.708235+0000) 2020-02-06T01:40:07.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:07.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.708538+0000) 2020-02-06T01:40:07.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:07.708+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.709158+0000) 2020-02-06T01:40:07.710 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:07.709+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.709584+0000) 2020-02-06T01:40:07.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:07.708+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.709802+0000) 2020-02-06T01:40:07.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:07.714+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:07.715897+0000) 2020-02-06T01:40:07.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:07.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210015 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:09.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:09.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:09.749962+0000) 2020-02-06T01:40:09.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:09.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:09.996574+0000) 2020-02-06T01:40:10.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:10.604+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:10.604916+0000) 2020-02-06T01:40:10.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:10.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:10.882926+0000) 2020-02-06T01:40:10.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:10.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:10.882927+0000) 2020-02-06T01:40:12.734 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:12.731+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.732973+0000) 2020-02-06T01:40:12.734 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:12.732+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.733613+0000) 2020-02-06T01:40:12.735 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:12.734+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.734829+0000) 2020-02-06T01:40:12.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:12.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.735996+0000) 2020-02-06T01:40:12.737 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:12.735+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.736575+0000) 2020-02-06T01:40:12.737 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:12.736+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.737072+0000) 2020-02-06T01:40:12.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:12.736+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.737071+0000) 2020-02-06T01:40:12.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:12.737+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.737677+0000) 2020-02-06T01:40:12.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:12.736+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.737717+0000) 2020-02-06T01:40:12.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:12.736+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.737977+0000) 2020-02-06T01:40:12.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:12.738+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.738417+0000) 2020-02-06T01:40:12.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:12.738+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.738755+0000) 2020-02-06T01:40:12.740 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:12.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.739578+0000) 2020-02-06T01:40:12.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:12.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:12.741356+0000) 2020-02-06T01:40:12.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:12.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210055 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:16.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:16.125+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:16.125882+0000) 2020-02-06T01:40:16.490 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:16.488+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:16.489274+0000) 2020-02-06T01:40:16.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:16.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:16.715155+0000) 2020-02-06T01:40:16.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:16.815+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:16.815985+0000) 2020-02-06T01:40:17.107 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:17.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.106612+0000) 2020-02-06T01:40:17.379 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:17.377+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.378226+0000) 2020-02-06T01:40:17.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:17.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.378516+0000) 2020-02-06T01:40:17.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:17.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.378650+0000) 2020-02-06T01:40:17.611 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:17.610+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.610710+0000) 2020-02-06T01:40:17.764 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:17.762+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.762854+0000) 2020-02-06T01:40:17.765 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:17.764+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.764480+0000) 2020-02-06T01:40:17.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:17.764+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.764904+0000) 2020-02-06T01:40:17.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:17.765+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.765454+0000) 2020-02-06T01:40:17.767 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:17.765+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.765771+0000) 2020-02-06T01:40:17.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:17.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.765701+0000) 2020-02-06T01:40:17.767 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:17.765+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.766285+0000) 2020-02-06T01:40:17.768 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:17.766+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.767029+0000) 2020-02-06T01:40:17.768 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:17.766+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.766676+0000) 2020-02-06T01:40:17.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:17.767+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.767586+0000) 2020-02-06T01:40:17.769 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:17.767+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.767992+0000) 2020-02-06T01:40:17.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:17.767+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.768037+0000) 2020-02-06T01:40:17.769 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:17.768+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.768650+0000) 2020-02-06T01:40:17.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:17.773+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:17.773373+0000) 2020-02-06T01:40:17.786 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:17.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210095 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:19.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:19.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:19.750200+0000) 2020-02-06T01:40:19.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:19.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:19.996805+0000) 2020-02-06T01:40:20.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:20.604+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:20.605171+0000) 2020-02-06T01:40:20.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:20.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:20.883201+0000) 2020-02-06T01:40:20.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:20.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:20.883203+0000) 2020-02-06T01:40:22.792 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:22.789+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.790986+0000) 2020-02-06T01:40:22.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:22.790+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.791322+0000) 2020-02-06T01:40:22.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:22.791+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.792067+0000) 2020-02-06T01:40:22.793 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:22.791+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.792236+0000) 2020-02-06T01:40:22.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:22.791+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.792406+0000) 2020-02-06T01:40:22.794 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:22.791+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.792640+0000) 2020-02-06T01:40:22.794 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:22.792+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.793355+0000) 2020-02-06T01:40:22.795 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:22.792+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.793153+0000) 2020-02-06T01:40:22.795 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:22.791+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.792755+0000) 2020-02-06T01:40:22.795 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:22.793+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.794040+0000) 2020-02-06T01:40:22.796 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:22.794+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.795049+0000) 2020-02-06T01:40:22.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:22.795+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.795628+0000) 2020-02-06T01:40:22.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:22.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.801409+0000) 2020-02-06T01:40:22.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:22.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:22.801671+0000) 2020-02-06T01:40:22.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:22.812+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210136 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:26.127 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:26.125+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:26.126115+0000) 2020-02-06T01:40:26.490 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:26.489+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:26.489654+0000) 2020-02-06T01:40:26.716 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:26.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:26.715342+0000) 2020-02-06T01:40:26.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:26.823+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:26.824371+0000) 2020-02-06T01:40:27.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:27.105+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.106846+0000) 2020-02-06T01:40:27.379 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:27.378+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.378522+0000) 2020-02-06T01:40:27.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:27.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.378748+0000) 2020-02-06T01:40:27.380 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:27.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.378864+0000) 2020-02-06T01:40:27.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:27.609+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.610935+0000) 2020-02-06T01:40:27.820 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:27.817+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.818911+0000) 2020-02-06T01:40:27.821 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:27.819+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.820295+0000) 2020-02-06T01:40:27.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:27.820+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.821026+0000) 2020-02-06T01:40:27.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:27.820+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.821395+0000) 2020-02-06T01:40:27.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:27.821+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.822241+0000) 2020-02-06T01:40:27.823 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:27.821+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.822619+0000) 2020-02-06T01:40:27.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:27.822+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.823195+0000) 2020-02-06T01:40:27.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:27.823+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.823588+0000) 2020-02-06T01:40:27.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:27.822+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.824011+0000) 2020-02-06T01:40:27.825 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:27.823+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.824389+0000) 2020-02-06T01:40:27.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:27.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.825629+0000) 2020-02-06T01:40:27.827 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:27.825+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.826020+0000) 2020-02-06T01:40:27.827 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:27.825+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.826321+0000) 2020-02-06T01:40:27.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:27.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:27.827849+0000) 2020-02-06T01:40:27.842 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:27.841+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:29.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:29.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:29.750475+0000) 2020-02-06T01:40:29.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:29.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:29.997030+0000) 2020-02-06T01:40:30.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:30.605+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:30.605406+0000) 2020-02-06T01:40:30.884 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:30.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:30.883471+0000) 2020-02-06T01:40:30.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:30.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:30.883471+0000) 2020-02-06T01:40:32.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:32.846+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.847434+0000) 2020-02-06T01:40:32.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:32.848+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.849028+0000) 2020-02-06T01:40:32.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:32.848+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.849700+0000) 2020-02-06T01:40:32.851 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:32.849+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.850899+0000) 2020-02-06T01:40:32.852 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:32.850+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.851139+0000) 2020-02-06T01:40:32.852 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:32.850+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.851802+0000) 2020-02-06T01:40:32.853 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:32.851+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.852329+0000) 2020-02-06T01:40:32.854 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:32.852+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.852612+0000) 2020-02-06T01:40:32.854 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:32.851+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.852387+0000) 2020-02-06T01:40:32.854 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:32.852+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.853439+0000) 2020-02-06T01:40:32.855 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:32.852+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.853412+0000) 2020-02-06T01:40:32.855 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:32.853+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.854342+0000) 2020-02-06T01:40:32.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:32.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.854557+0000) 2020-02-06T01:40:32.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:32.856+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:32.857143+0000) 2020-02-06T01:40:32.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:32.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210216 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:34.178 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:40:34.184 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:40:36.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:36.126+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:36.126379+0000) 2020-02-06T01:40:36.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:36.489+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:36.489916+0000) 2020-02-06T01:40:36.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:36.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:36.715577+0000) 2020-02-06T01:40:36.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:36.831+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:36.832344+0000) 2020-02-06T01:40:37.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:37.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.107120+0000) 2020-02-06T01:40:37.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:37.378+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.378779+0000) 2020-02-06T01:40:37.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:37.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.379023+0000) 2020-02-06T01:40:37.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:37.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.379127+0000) 2020-02-06T01:40:37.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:37.610+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.611171+0000) 2020-02-06T01:40:37.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:37.875+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.876459+0000) 2020-02-06T01:40:37.878 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:37.876+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.877440+0000) 2020-02-06T01:40:37.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:37.876+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.877921+0000) 2020-02-06T01:40:37.879 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:37.877+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.878287+0000) 2020-02-06T01:40:37.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:37.880+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.881188+0000) 2020-02-06T01:40:37.884 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:37.880+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.881561+0000) 2020-02-06T01:40:37.884 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:37.880+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.881778+0000) 2020-02-06T01:40:37.884 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:37.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.880836+0000) 2020-02-06T01:40:37.884 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:37.880+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.880876+0000) 2020-02-06T01:40:37.884 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:37.881+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.881852+0000) 2020-02-06T01:40:37.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:37.881+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.882187+0000) 2020-02-06T01:40:37.885 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:37.882+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.882760+0000) 2020-02-06T01:40:37.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:37.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.884482+0000) 2020-02-06T01:40:37.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:37.883+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:37.884708+0000) 2020-02-06T01:40:37.899 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:37.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:39.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:39.749+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:39.750733+0000) 2020-02-06T01:40:39.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:39.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:39.997285+0000) 2020-02-06T01:40:40.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:40.604+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:40.605672+0000) 2020-02-06T01:40:40.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:40.882+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:40.883764+0000) 2020-02-06T01:40:40.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:40.882+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:40.883764+0000) 2020-02-06T01:40:42.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:42.902+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.904024+0000) 2020-02-06T01:40:42.905 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:42.903+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.904608+0000) 2020-02-06T01:40:42.906 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:42.903+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.905228+0000) 2020-02-06T01:40:42.906 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:42.904+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.905755+0000) 2020-02-06T01:40:42.907 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:42.905+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.906415+0000) 2020-02-06T01:40:42.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:42.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.906627+0000) 2020-02-06T01:40:42.908 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:42.905+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.907030+0000) 2020-02-06T01:40:42.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:42.906+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.907245+0000) 2020-02-06T01:40:42.909 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:42.906+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.907806+0000) 2020-02-06T01:40:42.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:42.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.907490+0000) 2020-02-06T01:40:42.909 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:42.906+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.907784+0000) 2020-02-06T01:40:42.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:42.907+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.908818+0000) 2020-02-06T01:40:42.910 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:42.907+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.909232+0000) 2020-02-06T01:40:42.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:42.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:42.916862+0000) 2020-02-06T01:40:42.927 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:42.924+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210298 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:46.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:46.125+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:46.126696+0000) 2020-02-06T01:40:46.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:46.489+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:46.490155+0000) 2020-02-06T01:40:46.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:46.714+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:46.715812+0000) 2020-02-06T01:40:46.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:46.839+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:46.840405+0000) 2020-02-06T01:40:47.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:47.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.107339+0000) 2020-02-06T01:40:47.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:47.378+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.379054+0000) 2020-02-06T01:40:47.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:47.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.379300+0000) 2020-02-06T01:40:47.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:47.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.379402+0000) 2020-02-06T01:40:47.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:47.610+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.611406+0000) 2020-02-06T01:40:47.932 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:47.930+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.931521+0000) 2020-02-06T01:40:47.933 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:47.931+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.932714+0000) 2020-02-06T01:40:47.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:47.932+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.933342+0000) 2020-02-06T01:40:47.934 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:47.932+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.933545+0000) 2020-02-06T01:40:47.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:47.933+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.934067+0000) 2020-02-06T01:40:47.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:47.932+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.934213+0000) 2020-02-06T01:40:47.936 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:47.933+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.935208+0000) 2020-02-06T01:40:47.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:47.934+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.935722+0000) 2020-02-06T01:40:47.937 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:47.934+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.935760+0000) 2020-02-06T01:40:47.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:47.935+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.936148+0000) 2020-02-06T01:40:47.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:47.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.936553+0000) 2020-02-06T01:40:47.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:47.936+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.937906+0000) 2020-02-06T01:40:47.939 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:47.936+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.938284+0000) 2020-02-06T01:40:47.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:47.942+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:47.943350+0000) 2020-02-06T01:40:47.954 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:47.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210338 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:49.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:49.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:49.751005+0000) 2020-02-06T01:40:49.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:49.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:49.997536+0000) 2020-02-06T01:40:50.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:50.604+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:50.605909+0000) 2020-02-06T01:40:50.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:50.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:50.884022+0000) 2020-02-06T01:40:50.885 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:50.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:50.884022+0000) 2020-02-06T01:40:52.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:52.959+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.960110+0000) 2020-02-06T01:40:52.962 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:52.959+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.960524+0000) 2020-02-06T01:40:52.962 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:52.960+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.961402+0000) 2020-02-06T01:40:52.963 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:52.960+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.961672+0000) 2020-02-06T01:40:52.963 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:52.961+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.962572+0000) 2020-02-06T01:40:52.964 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:52.961+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.962923+0000) 2020-02-06T01:40:52.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:52.962+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.963303+0000) 2020-02-06T01:40:52.966 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:52.962+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.963969+0000) 2020-02-06T01:40:52.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:52.962+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.964114+0000) 2020-02-06T01:40:52.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:52.963+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.964982+0000) 2020-02-06T01:40:52.967 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:52.963+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.964639+0000) 2020-02-06T01:40:52.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:52.964+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.965545+0000) 2020-02-06T01:40:52.968 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:52.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.967722+0000) 2020-02-06T01:40:52.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:52.966+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:52.967955+0000) 2020-02-06T01:40:52.982 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:52.980+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210378 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:56.128 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:56.126+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:56.126958+0000) 2020-02-06T01:40:56.491 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:56.488+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:56.490385+0000) 2020-02-06T01:40:56.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:56.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:56.716035+0000) 2020-02-06T01:40:56.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:56.848+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:56.848948+0000) 2020-02-06T01:40:57.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:57.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.107568+0000) 2020-02-06T01:40:57.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:57.378+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.379312+0000) 2020-02-06T01:40:57.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:57.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.379532+0000) 2020-02-06T01:40:57.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:57.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.379685+0000) 2020-02-06T01:40:57.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:57.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.611632+0000) 2020-02-06T01:40:57.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:40:57.985+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.986683+0000) 2020-02-06T01:40:57.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:57.986+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.987696+0000) 2020-02-06T01:40:57.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:40:57.987+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.989033+0000) 2020-02-06T01:40:57.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:40:57.988+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.989883+0000) 2020-02-06T01:40:57.991 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:40:57.988+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.990322+0000) 2020-02-06T01:40:57.991 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:40:57.989+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.990765+0000) 2020-02-06T01:40:57.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:40:57.990+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.991859+0000) 2020-02-06T01:40:57.993 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:40:57.990+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.991692+0000) 2020-02-06T01:40:57.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:40:57.991+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.992616+0000) 2020-02-06T01:40:57.994 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:40:57.991+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.993234+0000) 2020-02-06T01:40:57.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:57.992+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.993564+0000) 2020-02-06T01:40:57.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:40:57.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.994171+0000) 2020-02-06T01:40:57.995 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:40:57.992+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.993751+0000) 2020-02-06T01:40:57.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:57.996+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:57.997253+0000) 2020-02-06T01:40:58.010 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:40:58.008+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210419 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:40:59.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:59.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:59.751282+0000) 2020-02-06T01:40:59.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:40:59.996+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:40:59.997770+0000) 2020-02-06T01:41:00.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:00.605+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:00.606146+0000) 2020-02-06T01:41:00.885 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:00.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:00.884363+0000) 2020-02-06T01:41:00.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:00.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:00.884363+0000) 2020-02-06T01:41:03.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:03.013+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.014882+0000) 2020-02-06T01:41:03.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:03.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.016410+0000) 2020-02-06T01:41:03.018 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:03.015+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.016986+0000) 2020-02-06T01:41:03.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:03.015+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.017386+0000) 2020-02-06T01:41:03.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:03.016+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.017942+0000) 2020-02-06T01:41:03.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:03.017+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.019031+0000) 2020-02-06T01:41:03.020 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:03.018+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.019583+0000) 2020-02-06T01:41:03.021 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:03.019+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.020599+0000) 2020-02-06T01:41:03.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:03.020+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.021637+0000) 2020-02-06T01:41:03.023 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:03.020+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.021846+0000) 2020-02-06T01:41:03.023 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:03.020+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.022277+0000) 2020-02-06T01:41:03.024 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:03.020+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.021825+0000) 2020-02-06T01:41:03.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:03.021+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.022374+0000) 2020-02-06T01:41:03.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:03.024+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:03.025797+0000) 2020-02-06T01:41:03.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:03.038+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210460 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:04.450 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:41:04.456 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:41:06.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:06.126+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:06.127213+0000) 2020-02-06T01:41:06.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:06.489+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:06.490676+0000) 2020-02-06T01:41:06.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:06.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:06.716302+0000) 2020-02-06T01:41:06.860 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:06.858+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:06.858841+0000) 2020-02-06T01:41:07.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:07.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:07.107802+0000) 2020-02-06T01:41:07.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:07.378+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:07.379572+0000) 2020-02-06T01:41:07.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:07.378+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:07.379775+0000) 2020-02-06T01:41:07.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:07.378+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:07.379906+0000) 2020-02-06T01:41:07.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:07.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:07.611861+0000) 2020-02-06T01:41:08.045 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:08.043+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.044510+0000) 2020-02-06T01:41:08.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:08.044+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.045640+0000) 2020-02-06T01:41:08.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:08.044+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.045893+0000) 2020-02-06T01:41:08.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:08.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.047002+0000) 2020-02-06T01:41:08.048 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:08.046+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.047162+0000) 2020-02-06T01:41:08.048 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:08.046+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.047694+0000) 2020-02-06T01:41:08.049 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:08.046+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.048047+0000) 2020-02-06T01:41:08.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:08.047+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.048511+0000) 2020-02-06T01:41:08.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:08.047+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.048690+0000) 2020-02-06T01:41:08.050 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:08.048+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.049179+0000) 2020-02-06T01:41:08.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:08.048+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.049657+0000) 2020-02-06T01:41:08.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:08.049+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.050726+0000) 2020-02-06T01:41:08.052 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:08.050+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.051450+0000) 2020-02-06T01:41:08.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:08.053+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:08.054374+0000) 2020-02-06T01:41:08.068 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:08.065+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210500 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:09.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:09.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:09.751513+0000) 2020-02-06T01:41:09.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:09.997+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:09.998021+0000) 2020-02-06T01:41:10.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:10.605+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:10.606400+0000) 2020-02-06T01:41:10.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:10.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:10.884640+0000) 2020-02-06T01:41:10.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:10.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:10.884640+0000) 2020-02-06T01:41:13.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:13.073+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.073275+0000) 2020-02-06T01:41:13.075 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:13.072+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.074134+0000) 2020-02-06T01:41:13.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:13.074+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.074246+0000) 2020-02-06T01:41:13.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:13.073+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.074730+0000) 2020-02-06T01:41:13.076 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:13.074+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.074929+0000) 2020-02-06T01:41:13.076 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:13.075+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.075996+0000) 2020-02-06T01:41:13.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:13.076+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.076208+0000) 2020-02-06T01:41:13.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:13.075+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.076509+0000) 2020-02-06T01:41:13.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:13.075+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.077257+0000) 2020-02-06T01:41:13.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:13.077+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.077468+0000) 2020-02-06T01:41:13.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:13.078+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.078328+0000) 2020-02-06T01:41:13.079 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:13.076+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.078138+0000) 2020-02-06T01:41:13.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:13.076+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.078381+0000) 2020-02-06T01:41:13.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:13.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:13.084796+0000) 2020-02-06T01:41:13.095 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:13.092+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:16.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:16.126+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:16.127478+0000) 2020-02-06T01:41:16.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:16.490+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:16.490980+0000) 2020-02-06T01:41:16.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:16.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:16.716531+0000) 2020-02-06T01:41:16.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:16.865+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:16.866882+0000) 2020-02-06T01:41:17.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:17.106+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:17.108043+0000) 2020-02-06T01:41:17.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:17.379+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:17.379816+0000) 2020-02-06T01:41:17.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:17.379+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:17.379995+0000) 2020-02-06T01:41:17.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:17.379+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:17.380133+0000) 2020-02-06T01:41:17.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:17.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:17.612129+0000) 2020-02-06T01:41:18.100 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:18.098+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.099415+0000) 2020-02-06T01:41:18.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:18.099+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.100707+0000) 2020-02-06T01:41:18.102 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:18.100+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.101309+0000) 2020-02-06T01:41:18.103 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:18.101+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.102273+0000) 2020-02-06T01:41:18.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:18.100+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.101939+0000) 2020-02-06T01:41:18.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:18.101+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.102628+0000) 2020-02-06T01:41:18.104 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:18.102+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.103253+0000) 2020-02-06T01:41:18.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:18.102+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.103695+0000) 2020-02-06T01:41:18.105 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:18.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.104046+0000) 2020-02-06T01:41:18.105 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:18.102+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.104176+0000) 2020-02-06T01:41:18.106 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:18.103+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.104728+0000) 2020-02-06T01:41:18.106 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:18.104+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.105504+0000) 2020-02-06T01:41:18.106 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:18.104+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.105748+0000) 2020-02-06T01:41:18.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:18.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:18.110084+0000) 2020-02-06T01:41:18.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:18.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:19.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:19.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:19.751699+0000) 2020-02-06T01:41:19.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:19.997+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:19.998265+0000) 2020-02-06T01:41:20.607 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:20.606+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:20.606692+0000) 2020-02-06T01:41:20.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:20.883+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:20.884912+0000) 2020-02-06T01:41:20.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:20.883+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:20.884912+0000) 2020-02-06T01:41:23.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:23.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.127856+0000) 2020-02-06T01:41:23.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:23.128+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.129196+0000) 2020-02-06T01:41:23.131 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:23.129+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.130343+0000) 2020-02-06T01:41:23.131 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:23.129+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.130510+0000) 2020-02-06T01:41:23.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:23.129+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.130966+0000) 2020-02-06T01:41:23.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:23.130+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.131545+0000) 2020-02-06T01:41:23.133 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:23.130+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.131817+0000) 2020-02-06T01:41:23.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:23.130+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.131770+0000) 2020-02-06T01:41:23.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:23.131+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.132303+0000) 2020-02-06T01:41:23.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:23.131+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.132816+0000) 2020-02-06T01:41:23.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:23.132+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.133386+0000) 2020-02-06T01:41:23.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:23.131+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.133296+0000) 2020-02-06T01:41:23.135 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:23.132+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.134172+0000) 2020-02-06T01:41:23.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:23.135+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:23.136651+0000) 2020-02-06T01:41:23.151 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:23.148+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:26.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:26.127+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:26.127737+0000) 2020-02-06T01:41:26.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:26.490+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:26.491209+0000) 2020-02-06T01:41:26.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:26.715+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:26.716784+0000) 2020-02-06T01:41:26.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:26.874+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:26.875251+0000) 2020-02-06T01:41:27.109 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:27.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:27.108301+0000) 2020-02-06T01:41:27.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:27.379+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:27.380076+0000) 2020-02-06T01:41:27.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:27.379+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:27.380204+0000) 2020-02-06T01:41:27.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:27.379+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:27.380386+0000) 2020-02-06T01:41:27.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:27.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:27.612314+0000) 2020-02-06T01:41:28.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:28.155+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.156647+0000) 2020-02-06T01:41:28.158 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:28.156+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.157122+0000) 2020-02-06T01:41:28.158 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:28.156+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.157588+0000) 2020-02-06T01:41:28.159 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:28.156+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.157761+0000) 2020-02-06T01:41:28.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:28.158+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.159446+0000) 2020-02-06T01:41:28.161 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:28.157+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.158453+0000) 2020-02-06T01:41:28.161 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:28.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.159231+0000) 2020-02-06T01:41:28.161 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:28.158+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.159857+0000) 2020-02-06T01:41:28.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:28.159+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.160190+0000) 2020-02-06T01:41:28.162 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:28.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.160617+0000) 2020-02-06T01:41:28.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:28.159+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.161255+0000) 2020-02-06T01:41:28.163 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:28.159+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.161281+0000) 2020-02-06T01:41:28.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:28.160+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.161717+0000) 2020-02-06T01:41:28.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:28.165+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:28.166232+0000) 2020-02-06T01:41:28.179 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:28.176+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210668 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:29.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:29.750+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:29.751952+0000) 2020-02-06T01:41:29.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:29.997+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:29.998483+0000) 2020-02-06T01:41:30.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:30.606+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:30.606967+0000) 2020-02-06T01:41:30.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:30.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:30.885194+0000) 2020-02-06T01:41:30.886 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:30.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:30.885194+0000) 2020-02-06T01:41:33.185 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:33.183+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.183712+0000) 2020-02-06T01:41:33.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:33.183+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.184318+0000) 2020-02-06T01:41:33.186 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:33.183+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.184569+0000) 2020-02-06T01:41:33.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:33.183+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.184898+0000) 2020-02-06T01:41:33.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:33.184+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.185414+0000) 2020-02-06T01:41:33.187 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:33.185+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.185764+0000) 2020-02-06T01:41:33.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:33.185+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.186018+0000) 2020-02-06T01:41:33.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:33.185+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.187033+0000) 2020-02-06T01:41:33.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:33.187+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.187626+0000) 2020-02-06T01:41:33.189 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:33.187+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.188205+0000) 2020-02-06T01:41:33.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:33.188+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.189390+0000) 2020-02-06T01:41:33.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:33.189+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.190255+0000) 2020-02-06T01:41:33.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:33.192+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.193951+0000) 2020-02-06T01:41:33.197 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:33.193+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:33.194225+0000) 2020-02-06T01:41:33.208 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:33.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:34.604 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:41:34.610 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:41:36.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:36.127+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:36.128023+0000) 2020-02-06T01:41:36.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:36.491+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:36.491513+0000) 2020-02-06T01:41:36.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:36.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:36.717010+0000) 2020-02-06T01:41:36.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:36.882+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:36.883338+0000) 2020-02-06T01:41:37.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:37.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:37.108565+0000) 2020-02-06T01:41:37.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:37.379+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:37.380337+0000) 2020-02-06T01:41:37.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:37.380+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:37.380508+0000) 2020-02-06T01:41:37.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:37.380+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:37.380661+0000) 2020-02-06T01:41:37.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:37.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:37.612570+0000) 2020-02-06T01:41:38.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:38.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.213164+0000) 2020-02-06T01:41:38.215 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:38.214+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.214815+0000) 2020-02-06T01:41:38.216 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:38.214+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.215292+0000) 2020-02-06T01:41:38.217 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:38.214+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.215965+0000) 2020-02-06T01:41:38.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:38.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.216410+0000) 2020-02-06T01:41:38.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:38.216+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.216980+0000) 2020-02-06T01:41:38.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:38.217+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.217576+0000) 2020-02-06T01:41:38.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:38.216+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.217757+0000) 2020-02-06T01:41:38.219 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:38.217+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.218189+0000) 2020-02-06T01:41:38.219 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:38.218+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.218814+0000) 2020-02-06T01:41:38.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:38.217+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.218810+0000) 2020-02-06T01:41:38.220 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:38.218+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.219069+0000) 2020-02-06T01:41:38.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:38.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.222066+0000) 2020-02-06T01:41:38.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:38.221+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:38.222322+0000) 2020-02-06T01:41:38.236 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:38.235+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:39.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:39.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:39.752181+0000) 2020-02-06T01:41:39.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:39.998+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:39.998725+0000) 2020-02-06T01:41:40.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:40.606+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:40.607202+0000) 2020-02-06T01:41:40.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:40.884+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:40.885503+0000) 2020-02-06T01:41:40.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:40.884+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:40.885503+0000) 2020-02-06T01:41:43.243 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:43.240+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.241953+0000) 2020-02-06T01:41:43.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:43.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.242728+0000) 2020-02-06T01:41:43.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:43.242+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.242973+0000) 2020-02-06T01:41:43.244 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:43.242+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.243309+0000) 2020-02-06T01:41:43.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:43.243+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.243889+0000) 2020-02-06T01:41:43.246 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:43.244+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.245348+0000) 2020-02-06T01:41:43.246 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:43.245+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.245527+0000) 2020-02-06T01:41:43.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:43.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.245518+0000) 2020-02-06T01:41:43.247 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:43.244+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.245908+0000) 2020-02-06T01:41:43.248 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:43.246+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.246630+0000) 2020-02-06T01:41:43.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:43.245+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.246147+0000) 2020-02-06T01:41:43.248 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:43.245+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.246360+0000) 2020-02-06T01:41:43.248 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:43.245+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.246994+0000) 2020-02-06T01:41:43.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:43.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:43.251809+0000) 2020-02-06T01:41:43.264 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:43.262+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210793 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:46.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:46.127+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:46.128277+0000) 2020-02-06T01:41:46.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:46.491+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:46.491774+0000) 2020-02-06T01:41:46.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:46.716+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:46.717331+0000) 2020-02-06T01:41:46.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:46.890+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:46.891497+0000) 2020-02-06T01:41:47.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:47.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:47.108760+0000) 2020-02-06T01:41:47.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:47.380+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:47.380662+0000) 2020-02-06T01:41:47.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:47.380+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:47.380779+0000) 2020-02-06T01:41:47.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:47.380+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:47.380908+0000) 2020-02-06T01:41:47.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:47.611+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:47.612840+0000) 2020-02-06T01:41:48.270 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:48.268+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.269209+0000) 2020-02-06T01:41:48.271 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:48.268+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.270053+0000) 2020-02-06T01:41:48.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:48.269+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.271043+0000) 2020-02-06T01:41:48.272 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:48.270+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.271262+0000) 2020-02-06T01:41:48.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:48.271+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.272125+0000) 2020-02-06T01:41:48.273 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:48.271+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.272208+0000) 2020-02-06T01:41:48.273 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:48.271+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.272540+0000) 2020-02-06T01:41:48.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:48.273+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.273626+0000) 2020-02-06T01:41:48.275 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:48.272+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.273931+0000) 2020-02-06T01:41:48.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:48.272+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.273973+0000) 2020-02-06T01:41:48.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:48.273+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.274329+0000) 2020-02-06T01:41:48.276 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:48.274+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.274766+0000) 2020-02-06T01:41:48.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:48.273+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.274650+0000) 2020-02-06T01:41:48.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:48.279+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:48.280690+0000) 2020-02-06T01:41:48.292 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:48.291+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210836 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:49.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:49.751+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:49.752426+0000) 2020-02-06T01:41:50.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:49.998+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:49.998955+0000) 2020-02-06T01:41:50.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:50.607+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:50.607467+0000) 2020-02-06T01:41:50.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:50.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:50.885778+0000) 2020-02-06T01:41:50.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:50.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:50.885780+0000) 2020-02-06T01:41:53.300 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:53.296+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.297238+0000) 2020-02-06T01:41:53.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:53.297+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.298411+0000) 2020-02-06T01:41:53.301 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:53.298+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.298929+0000) 2020-02-06T01:41:53.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:53.299+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.299549+0000) 2020-02-06T01:41:53.301 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:53.299+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.300278+0000) 2020-02-06T01:41:53.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:53.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.301112+0000) 2020-02-06T01:41:53.302 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:53.300+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.301516+0000) 2020-02-06T01:41:53.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:53.301+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.302029+0000) 2020-02-06T01:41:53.303 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:53.301+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.301795+0000) 2020-02-06T01:41:53.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:53.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.302782+0000) 2020-02-06T01:41:53.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:53.302+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.302741+0000) 2020-02-06T01:41:53.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:53.302+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.303715+0000) 2020-02-06T01:41:53.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:53.303+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.303815+0000) 2020-02-06T01:41:53.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:53.308+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:53.309909+0000) 2020-02-06T01:41:53.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:53.321+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210877 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:56.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:56.128+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:56.128541+0000) 2020-02-06T01:41:56.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:56.491+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:56.491988+0000) 2020-02-06T01:41:56.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:56.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:56.717511+0000) 2020-02-06T01:41:56.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:56.899+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:56.899799+0000) 2020-02-06T01:41:57.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:57.107+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:57.109005+0000) 2020-02-06T01:41:57.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:57.380+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:57.380899+0000) 2020-02-06T01:41:57.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:57.380+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:57.381075+0000) 2020-02-06T01:41:57.382 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:57.380+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:57.381273+0000) 2020-02-06T01:41:57.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:57.612+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:57.613141+0000) 2020-02-06T01:41:58.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:41:58.327+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.328343+0000) 2020-02-06T01:41:58.330 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:41:58.328+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.329298+0000) 2020-02-06T01:41:58.331 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:41:58.328+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.329935+0000) 2020-02-06T01:41:58.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:41:58.329+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.330391+0000) 2020-02-06T01:41:58.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:58.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.331248+0000) 2020-02-06T01:41:58.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:58.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.331606+0000) 2020-02-06T01:41:58.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:41:58.331+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.331869+0000) 2020-02-06T01:41:58.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:41:58.331+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.332347+0000) 2020-02-06T01:41:58.334 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:41:58.332+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.333174+0000) 2020-02-06T01:41:58.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:41:58.332+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.333728+0000) 2020-02-06T01:41:58.335 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:41:58.333+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.334127+0000) 2020-02-06T01:41:58.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:41:58.333+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.334107+0000) 2020-02-06T01:41:58.335 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:41:58.333+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.334342+0000) 2020-02-06T01:41:58.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:58.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:58.339091+0000) 2020-02-06T01:41:58.349 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:41:58.348+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:41:59.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:59.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:59.752666+0000) 2020-02-06T01:42:00.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:41:59.998+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:41:59.999192+0000) 2020-02-06T01:42:00.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:00.607+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:00.607644+0000) 2020-02-06T01:42:00.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:00.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:00.886047+0000) 2020-02-06T01:42:00.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:00.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:00.886047+0000) 2020-02-06T01:42:03.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:03.353+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.354368+0000) 2020-02-06T01:42:03.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:03.354+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.355831+0000) 2020-02-06T01:42:03.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:03.355+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.356416+0000) 2020-02-06T01:42:03.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:03.355+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.356720+0000) 2020-02-06T01:42:03.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:03.356+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.356942+0000) 2020-02-06T01:42:03.358 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:03.356+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.357170+0000) 2020-02-06T01:42:03.359 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:03.357+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.357708+0000) 2020-02-06T01:42:03.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:03.357+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.358003+0000) 2020-02-06T01:42:03.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:03.358+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.358943+0000) 2020-02-06T01:42:03.360 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:03.358+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.359667+0000) 2020-02-06T01:42:03.360 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:03.358+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.360019+0000) 2020-02-06T01:42:03.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:03.360+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.360469+0000) 2020-02-06T01:42:03.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:03.359+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.360358+0000) 2020-02-06T01:42:03.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:03.363+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:03.365054+0000) 2020-02-06T01:42:03.377 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:03.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 210962 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:04.737 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:42:04.744 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:42:06.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:06.128+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:06.128747+0000) 2020-02-06T01:42:06.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:06.492+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:06.492241+0000) 2020-02-06T01:42:06.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:06.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:06.717745+0000) 2020-02-06T01:42:06.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:06.907+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:06.908144+0000) 2020-02-06T01:42:07.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:07.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:07.109265+0000) 2020-02-06T01:42:07.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:07.381+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:07.381138+0000) 2020-02-06T01:42:07.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:07.381+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:07.381350+0000) 2020-02-06T01:42:07.383 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:07.381+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:07.381511+0000) 2020-02-06T01:42:07.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:07.613+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:07.613346+0000) 2020-02-06T01:42:08.382 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:08.381+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.381344+0000) 2020-02-06T01:42:08.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:08.382+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.383179+0000) 2020-02-06T01:42:08.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:08.383+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.383528+0000) 2020-02-06T01:42:08.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:08.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.384244+0000) 2020-02-06T01:42:08.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:08.384+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.384813+0000) 2020-02-06T01:42:08.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:08.385+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.385580+0000) 2020-02-06T01:42:08.387 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:08.385+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.385820+0000) 2020-02-06T01:42:08.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:08.386+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.386781+0000) 2020-02-06T01:42:08.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:08.387+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.387414+0000) 2020-02-06T01:42:08.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:08.387+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.388048+0000) 2020-02-06T01:42:08.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:08.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.388431+0000) 2020-02-06T01:42:08.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:08.387+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.388346+0000) 2020-02-06T01:42:08.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:08.388+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.388900+0000) 2020-02-06T01:42:08.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:08.391+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:08.391694+0000) 2020-02-06T01:42:08.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:08.404+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211006 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:09.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:09.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:09.752917+0000) 2020-02-06T01:42:10.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:09.999+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:09.999428+0000) 2020-02-06T01:42:10.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:10.606+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:10.607921+0000) 2020-02-06T01:42:10.887 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:10.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:10.886365+0000) 2020-02-06T01:42:10.887 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:10.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:10.886365+0000) 2020-02-06T01:42:13.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:13.409+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.410893+0000) 2020-02-06T01:42:13.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:13.409+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.411057+0000) 2020-02-06T01:42:13.413 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:13.410+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.411503+0000) 2020-02-06T01:42:13.413 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:13.411+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.412154+0000) 2020-02-06T01:42:13.414 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:13.411+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.412648+0000) 2020-02-06T01:42:13.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:13.410+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.412359+0000) 2020-02-06T01:42:13.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:13.412+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.413562+0000) 2020-02-06T01:42:13.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:13.413+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.415020+0000) 2020-02-06T01:42:13.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:13.413+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.415178+0000) 2020-02-06T01:42:13.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:13.414+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.415467+0000) 2020-02-06T01:42:13.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:13.414+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.415726+0000) 2020-02-06T01:42:13.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:13.415+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.416396+0000) 2020-02-06T01:42:13.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:13.416+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.417869+0000) 2020-02-06T01:42:13.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:13.418+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:13.419908+0000) 2020-02-06T01:42:13.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:13.430+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211047 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:16.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:16.128+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:16.129042+0000) 2020-02-06T01:42:16.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:16.491+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:16.492500+0000) 2020-02-06T01:42:16.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:16.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:16.717979+0000) 2020-02-06T01:42:16.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:16.915+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:16.916413+0000) 2020-02-06T01:42:17.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:17.108+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:17.109493+0000) 2020-02-06T01:42:17.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:17.380+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:17.381399+0000) 2020-02-06T01:42:17.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:17.380+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:17.381684+0000) 2020-02-06T01:42:17.383 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:17.380+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:17.381769+0000) 2020-02-06T01:42:17.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:17.612+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:17.613559+0000) 2020-02-06T01:42:18.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:18.436+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.437687+0000) 2020-02-06T01:42:18.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:18.437+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.438639+0000) 2020-02-06T01:42:18.440 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:18.437+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.439110+0000) 2020-02-06T01:42:18.440 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:18.438+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.439785+0000) 2020-02-06T01:42:18.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:18.439+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.440879+0000) 2020-02-06T01:42:18.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:18.440+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.441693+0000) 2020-02-06T01:42:18.443 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:18.441+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.442312+0000) 2020-02-06T01:42:18.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:18.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.443296+0000) 2020-02-06T01:42:18.444 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:18.442+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.443482+0000) 2020-02-06T01:42:18.445 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:18.442+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.443898+0000) 2020-02-06T01:42:18.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:18.442+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.444119+0000) 2020-02-06T01:42:18.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:18.442+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.443573+0000) 2020-02-06T01:42:18.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:18.443+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.444110+0000) 2020-02-06T01:42:18.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:18.449+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:18.450709+0000) 2020-02-06T01:42:18.461 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:18.458+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:19.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:19.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:19.753176+0000) 2020-02-06T01:42:20.000 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:19.998+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:19.999669+0000) 2020-02-06T01:42:20.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:20.607+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:20.608171+0000) 2020-02-06T01:42:20.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:20.885+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:20.886567+0000) 2020-02-06T01:42:20.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:20.885+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:20.886587+0000) 2020-02-06T01:42:23.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:23.463+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.464958+0000) 2020-02-06T01:42:23.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:23.464+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.465492+0000) 2020-02-06T01:42:23.469 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:23.467+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.468188+0000) 2020-02-06T01:42:23.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:23.467+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.468627+0000) 2020-02-06T01:42:23.470 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:23.467+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.468959+0000) 2020-02-06T01:42:23.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:23.468+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.470177+0000) 2020-02-06T01:42:23.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:23.468+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.469732+0000) 2020-02-06T01:42:23.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:23.469+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.470507+0000) 2020-02-06T01:42:23.472 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:23.469+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.471360+0000) 2020-02-06T01:42:23.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:23.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.472285+0000) 2020-02-06T01:42:23.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:23.471+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.472305+0000) 2020-02-06T01:42:23.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:23.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.472508+0000) 2020-02-06T01:42:23.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:23.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.472722+0000) 2020-02-06T01:42:23.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:23.474+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:23.475394+0000) 2020-02-06T01:42:23.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:23.486+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211132 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:26.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:26.128+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:26.129313+0000) 2020-02-06T01:42:26.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:26.491+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:26.492734+0000) 2020-02-06T01:42:26.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:26.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:26.718209+0000) 2020-02-06T01:42:26.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:26.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:26.924553+0000) 2020-02-06T01:42:27.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:27.108+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:27.109713+0000) 2020-02-06T01:42:27.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:27.380+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:27.381747+0000) 2020-02-06T01:42:27.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:27.380+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:27.381933+0000) 2020-02-06T01:42:27.383 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:27.380+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:27.382031+0000) 2020-02-06T01:42:27.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:27.613+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:27.613777+0000) 2020-02-06T01:42:28.494 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:28.492+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.493472+0000) 2020-02-06T01:42:28.496 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:28.494+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.495177+0000) 2020-02-06T01:42:28.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:28.494+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.496219+0000) 2020-02-06T01:42:28.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:28.495+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.496871+0000) 2020-02-06T01:42:28.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:28.495+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.496529+0000) 2020-02-06T01:42:28.498 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:28.496+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.497424+0000) 2020-02-06T01:42:28.499 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:28.495+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.497376+0000) 2020-02-06T01:42:28.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:28.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.498039+0000) 2020-02-06T01:42:28.499 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:28.497+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.498698+0000) 2020-02-06T01:42:28.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:28.498+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.499407+0000) 2020-02-06T01:42:28.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:28.498+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.499732+0000) 2020-02-06T01:42:28.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:28.499+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.500283+0000) 2020-02-06T01:42:28.501 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:28.498+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.500168+0000) 2020-02-06T01:42:28.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:28.503+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:28.504679+0000) 2020-02-06T01:42:28.517 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:28.515+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:29.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:29.752+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:29.753453+0000) 2020-02-06T01:42:30.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:29.999+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:29.999892+0000) 2020-02-06T01:42:30.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:30.607+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:30.608406+0000) 2020-02-06T01:42:30.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:30.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:30.886850+0000) 2020-02-06T01:42:30.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:30.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:30.886848+0000) 2020-02-06T01:42:33.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:33.522+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.523383+0000) 2020-02-06T01:42:33.525 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:33.522+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.523819+0000) 2020-02-06T01:42:33.525 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:33.523+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.524657+0000) 2020-02-06T01:42:33.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:33.523+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.524941+0000) 2020-02-06T01:42:33.526 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:33.524+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.525267+0000) 2020-02-06T01:42:33.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:33.524+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.525944+0000) 2020-02-06T01:42:33.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:33.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.525893+0000) 2020-02-06T01:42:33.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:33.525+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.526489+0000) 2020-02-06T01:42:33.528 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:33.525+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.526775+0000) 2020-02-06T01:42:33.528 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:33.525+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.526835+0000) 2020-02-06T01:42:33.528 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:33.526+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.527649+0000) 2020-02-06T01:42:33.529 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:33.526+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.527728+0000) 2020-02-06T01:42:33.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:33.526+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.527645+0000) 2020-02-06T01:42:33.533 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:33.531+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:33.532332+0000) 2020-02-06T01:42:33.547 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:33.545+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211217 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:34.890 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:42:34.897 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:42:36.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:36.128+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:36.129626+0000) 2020-02-06T01:42:36.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:36.492+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:36.492977+0000) 2020-02-06T01:42:36.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:36.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:36.718435+0000) 2020-02-06T01:42:36.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:36.935+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:36.936348+0000) 2020-02-06T01:42:37.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:37.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:37.109932+0000) 2020-02-06T01:42:37.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:37.381+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:37.382008+0000) 2020-02-06T01:42:37.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:37.381+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:37.382173+0000) 2020-02-06T01:42:37.383 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:37.381+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:37.382291+0000) 2020-02-06T01:42:37.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:37.612+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:37.614016+0000) 2020-02-06T01:42:38.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:38.551+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.552946+0000) 2020-02-06T01:42:38.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:38.552+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.553806+0000) 2020-02-06T01:42:38.555 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:38.553+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.554505+0000) 2020-02-06T01:42:38.555 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:38.553+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.554811+0000) 2020-02-06T01:42:38.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:38.553+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.555051+0000) 2020-02-06T01:42:38.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:38.554+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.555530+0000) 2020-02-06T01:42:38.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:38.554+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.555682+0000) 2020-02-06T01:42:38.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:38.554+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.556044+0000) 2020-02-06T01:42:38.557 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:38.555+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.556638+0000) 2020-02-06T01:42:38.558 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:38.556+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.557509+0000) 2020-02-06T01:42:38.558 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:38.555+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.557280+0000) 2020-02-06T01:42:38.559 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:38.557+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.558644+0000) 2020-02-06T01:42:38.560 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:38.557+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.559219+0000) 2020-02-06T01:42:38.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:38.563+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:38.564467+0000) 2020-02-06T01:42:38.575 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:38.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:39.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:39.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:39.753726+0000) 2020-02-06T01:42:40.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:39.999+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:40.000122+0000) 2020-02-06T01:42:40.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:40.608+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:40.608707+0000) 2020-02-06T01:42:40.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:40.886+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:40.887144+0000) 2020-02-06T01:42:40.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:40.886+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:40.887144+0000) 2020-02-06T01:42:43.581 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:43.579+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.580142+0000) 2020-02-06T01:42:43.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:43.580+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.581462+0000) 2020-02-06T01:42:43.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:43.581+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.582511+0000) 2020-02-06T01:42:43.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:43.581+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.582973+0000) 2020-02-06T01:42:43.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:43.582+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.583387+0000) 2020-02-06T01:42:43.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:43.582+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.583715+0000) 2020-02-06T01:42:43.585 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:43.582+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.584078+0000) 2020-02-06T01:42:43.585 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:43.582+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.584326+0000) 2020-02-06T01:42:43.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:43.583+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.585005+0000) 2020-02-06T01:42:43.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:43.584+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.585659+0000) 2020-02-06T01:42:43.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:43.583+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.585104+0000) 2020-02-06T01:42:43.588 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:43.585+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.586454+0000) 2020-02-06T01:42:43.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:43.583+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.585038+0000) 2020-02-06T01:42:43.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:43.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:43.591243+0000) 2020-02-06T01:42:43.602 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:43.600+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211302 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:46.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:46.129+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:46.129890+0000) 2020-02-06T01:42:46.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:46.492+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:46.493219+0000) 2020-02-06T01:42:46.719 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:46.717+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:46.718662+0000) 2020-02-06T01:42:46.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:46.947+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:46.948339+0000) 2020-02-06T01:42:47.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:47.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:47.110165+0000) 2020-02-06T01:42:47.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:47.381+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:47.382262+0000) 2020-02-06T01:42:47.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:47.381+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:47.382422+0000) 2020-02-06T01:42:47.384 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:47.381+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:47.382590+0000) 2020-02-06T01:42:47.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:47.613+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:47.614278+0000) 2020-02-06T01:42:48.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:48.607+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.608759+0000) 2020-02-06T01:42:48.610 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:48.607+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.609025+0000) 2020-02-06T01:42:48.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:48.608+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.609419+0000) 2020-02-06T01:42:48.611 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:48.609+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.610491+0000) 2020-02-06T01:42:48.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:48.610+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.611649+0000) 2020-02-06T01:42:48.613 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:48.611+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.612606+0000) 2020-02-06T01:42:48.614 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:48.611+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.612958+0000) 2020-02-06T01:42:48.614 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:48.611+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.612881+0000) 2020-02-06T01:42:48.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:48.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.613436+0000) 2020-02-06T01:42:48.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:48.612+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.613718+0000) 2020-02-06T01:42:48.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:48.612+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.613813+0000) 2020-02-06T01:42:48.616 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:48.612+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.614055+0000) 2020-02-06T01:42:48.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:48.616+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.617773+0000) 2020-02-06T01:42:48.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:48.617+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:48.618115+0000) 2020-02-06T01:42:48.630 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:48.628+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:49.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:49.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:49.753964+0000) 2020-02-06T01:42:50.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:50.000+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:50.000375+0000) 2020-02-06T01:42:50.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:50.608+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:50.608958+0000) 2020-02-06T01:42:50.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:50.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:50.887412+0000) 2020-02-06T01:42:50.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:50.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:50.887412+0000) 2020-02-06T01:42:53.637 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:53.635+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.636612+0000) 2020-02-06T01:42:53.638 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:53.636+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.637525+0000) 2020-02-06T01:42:53.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:53.637+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.638532+0000) 2020-02-06T01:42:53.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:53.637+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.638652+0000) 2020-02-06T01:42:53.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:53.637+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.639054+0000) 2020-02-06T01:42:53.640 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:53.638+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.639380+0000) 2020-02-06T01:42:53.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:53.638+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.639777+0000) 2020-02-06T01:42:53.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:53.639+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.640543+0000) 2020-02-06T01:42:53.642 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:53.639+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.640869+0000) 2020-02-06T01:42:53.642 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:53.639+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.641277+0000) 2020-02-06T01:42:53.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:53.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.641503+0000) 2020-02-06T01:42:53.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:53.640+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.641553+0000) 2020-02-06T01:42:53.643 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:53.640+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.642018+0000) 2020-02-06T01:42:53.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:53.644+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:53.645578+0000) 2020-02-06T01:42:53.659 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:53.656+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:56.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:56.129+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:56.130163+0000) 2020-02-06T01:42:56.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:56.492+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:56.493487+0000) 2020-02-06T01:42:56.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:56.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:56.718928+0000) 2020-02-06T01:42:56.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:56.955+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:56.956067+0000) 2020-02-06T01:42:57.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:57.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:57.110387+0000) 2020-02-06T01:42:57.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:57.382+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:57.382522+0000) 2020-02-06T01:42:57.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:57.382+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:57.382694+0000) 2020-02-06T01:42:57.384 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:57.382+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:57.382816+0000) 2020-02-06T01:42:57.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:57.614+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:57.614509+0000) 2020-02-06T01:42:58.665 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:42:58.663+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.664218+0000) 2020-02-06T01:42:58.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:42:58.663+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.664541+0000) 2020-02-06T01:42:58.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:42:58.663+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.665031+0000) 2020-02-06T01:42:58.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:42:58.665+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.666063+0000) 2020-02-06T01:42:58.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:42:58.664+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.665795+0000) 2020-02-06T01:42:58.667 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:42:58.664+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.666185+0000) 2020-02-06T01:42:58.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:58.665+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.666647+0000) 2020-02-06T01:42:58.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:42:58.665+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.667051+0000) 2020-02-06T01:42:58.669 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:42:58.667+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.668197+0000) 2020-02-06T01:42:58.669 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:42:58.666+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.668121+0000) 2020-02-06T01:42:58.669 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:42:58.667+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.668676+0000) 2020-02-06T01:42:58.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:42:58.668+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.669639+0000) 2020-02-06T01:42:58.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:58.674+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.675631+0000) 2020-02-06T01:42:58.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:58.674+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:58.675982+0000) 2020-02-06T01:42:58.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:42:58.684+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211432 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:42:59.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:42:59.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:42:59.754202+0000) 2020-02-06T01:43:00.001 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:00.000+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:00.000610+0000) 2020-02-06T01:43:00.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:00.608+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:00.609201+0000) 2020-02-06T01:43:00.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:00.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:00.887709+0000) 2020-02-06T01:43:00.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:00.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:00.887709+0000) 2020-02-06T01:43:03.695 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:03.692+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.693028+0000) 2020-02-06T01:43:03.695 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:03.692+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.692188+0000) 2020-02-06T01:43:03.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:03.692+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.692386+0000) 2020-02-06T01:43:03.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:03.693+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.694296+0000) 2020-02-06T01:43:03.696 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:03.694+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.695205+0000) 2020-02-06T01:43:03.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:03.695+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.695840+0000) 2020-02-06T01:43:03.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:03.696+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.696626+0000) 2020-02-06T01:43:03.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:03.696+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.696999+0000) 2020-02-06T01:43:03.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:03.697+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.697737+0000) 2020-02-06T01:43:03.699 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:03.697+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.698016+0000) 2020-02-06T01:43:03.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:03.698+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.698399+0000) 2020-02-06T01:43:03.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:03.698+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.698530+0000) 2020-02-06T01:43:03.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:03.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.699225+0000) 2020-02-06T01:43:03.700 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:03.698+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:03.698720+0000) 2020-02-06T01:43:03.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:03.713+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:05.088 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:43:05.101 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:43:06.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:06.130+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:06.130465+0000) 2020-02-06T01:43:06.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:06.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:06.493683+0000) 2020-02-06T01:43:06.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:06.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:06.719276+0000) 2020-02-06T01:43:06.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:06.963+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:06.964632+0000) 2020-02-06T01:43:07.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:07.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:07.110619+0000) 2020-02-06T01:43:07.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:07.382+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:07.382777+0000) 2020-02-06T01:43:07.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:07.382+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:07.382957+0000) 2020-02-06T01:43:07.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:07.382+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:07.383063+0000) 2020-02-06T01:43:07.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:07.613+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:07.614744+0000) 2020-02-06T01:43:08.719 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:08.717+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.718522+0000) 2020-02-06T01:43:08.721 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:08.720+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.720710+0000) 2020-02-06T01:43:08.722 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:08.720+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.721359+0000) 2020-02-06T01:43:08.726 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:08.722+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.723454+0000) 2020-02-06T01:43:08.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:08.722+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.723876+0000) 2020-02-06T01:43:08.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:08.724+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.725432+0000) 2020-02-06T01:43:08.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:08.721+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.721876+0000) 2020-02-06T01:43:08.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:08.722+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.722949+0000) 2020-02-06T01:43:08.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:08.723+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.724155+0000) 2020-02-06T01:43:08.728 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:08.724+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.725178+0000) 2020-02-06T01:43:08.728 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:08.725+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.725460+0000) 2020-02-06T01:43:08.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:08.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.727924+0000) 2020-02-06T01:43:08.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:08.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.728190+0000) 2020-02-06T01:43:08.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:08.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:08.728359+0000) 2020-02-06T01:43:08.743 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:08.741+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211516 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:09.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:09.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:09.754380+0000) 2020-02-06T01:43:10.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:09.999+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:10.000875+0000) 2020-02-06T01:43:10.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:10.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:10.609471+0000) 2020-02-06T01:43:10.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:10.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:10.887983+0000) 2020-02-06T01:43:10.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:10.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:10.887983+0000) 2020-02-06T01:43:13.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:13.748+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.748668+0000) 2020-02-06T01:43:13.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:13.747+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.748989+0000) 2020-02-06T01:43:13.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:13.748+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.749863+0000) 2020-02-06T01:43:13.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:13.749+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.750340+0000) 2020-02-06T01:43:13.751 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:13.749+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.749744+0000) 2020-02-06T01:43:13.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:13.750+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.750706+0000) 2020-02-06T01:43:13.752 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:13.750+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.751822+0000) 2020-02-06T01:43:13.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:13.751+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.752272+0000) 2020-02-06T01:43:13.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:13.751+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.752668+0000) 2020-02-06T01:43:13.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:13.752+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.753621+0000) 2020-02-06T01:43:13.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:13.752+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.753780+0000) 2020-02-06T01:43:13.755 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:13.753+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.754158+0000) 2020-02-06T01:43:13.755 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:13.754+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.754639+0000) 2020-02-06T01:43:13.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:13.758+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:13.759416+0000) 2020-02-06T01:43:13.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:13.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:16.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:16.130+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:16.130740+0000) 2020-02-06T01:43:16.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:16.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:16.493932+0000) 2020-02-06T01:43:16.720 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:16.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:16.719513+0000) 2020-02-06T01:43:16.974 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:16.971+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:16.972889+0000) 2020-02-06T01:43:17.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:17.109+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:17.110835+0000) 2020-02-06T01:43:17.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:17.382+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:17.383031+0000) 2020-02-06T01:43:17.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:17.382+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:17.383214+0000) 2020-02-06T01:43:17.384 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:17.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:17.383331+0000) 2020-02-06T01:43:17.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:17.613+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:17.614998+0000) 2020-02-06T01:43:18.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:18.776+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.776577+0000) 2020-02-06T01:43:18.791 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:18.777+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.777864+0000) 2020-02-06T01:43:18.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:18.777+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.778263+0000) 2020-02-06T01:43:18.791 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:18.778+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.778849+0000) 2020-02-06T01:43:18.791 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:18.778+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.779397+0000) 2020-02-06T01:43:18.792 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:18.779+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.780283+0000) 2020-02-06T01:43:18.792 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:18.775+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.776145+0000) 2020-02-06T01:43:18.792 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:18.777+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.778197+0000) 2020-02-06T01:43:18.792 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:18.777+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.778637+0000) 2020-02-06T01:43:18.792 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:18.778+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.779276+0000) 2020-02-06T01:43:18.793 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:18.778+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.780022+0000) 2020-02-06T01:43:18.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:18.779+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.780116+0000) 2020-02-06T01:43:18.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:18.783+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.784539+0000) 2020-02-06T01:43:18.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:18.784+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:18.785943+0000) 2020-02-06T01:43:18.799 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:18.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211602 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:19.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:19.753+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:19.754641+0000) 2020-02-06T01:43:20.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:20.000+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:20.001145+0000) 2020-02-06T01:43:20.610 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:20.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:20.609723+0000) 2020-02-06T01:43:20.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:20.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:20.888296+0000) 2020-02-06T01:43:20.889 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:20.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:20.888298+0000) 2020-02-06T01:43:23.805 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:23.803+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.804330+0000) 2020-02-06T01:43:23.806 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:23.803+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.804841+0000) 2020-02-06T01:43:23.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:23.804+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.805536+0000) 2020-02-06T01:43:23.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:23.805+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.806782+0000) 2020-02-06T01:43:23.808 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:23.806+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.807307+0000) 2020-02-06T01:43:23.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:23.806+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.807560+0000) 2020-02-06T01:43:23.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:23.806+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.807746+0000) 2020-02-06T01:43:23.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:23.806+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.807259+0000) 2020-02-06T01:43:23.810 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:23.807+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.808296+0000) 2020-02-06T01:43:23.810 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:23.808+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.809084+0000) 2020-02-06T01:43:23.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:23.807+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.808912+0000) 2020-02-06T01:43:23.811 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:23.809+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.809767+0000) 2020-02-06T01:43:23.811 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:23.809+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.810221+0000) 2020-02-06T01:43:23.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:23.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:23.814619+0000) 2020-02-06T01:43:23.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:23.826+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211642 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:26.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:26.130+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:26.130999+0000) 2020-02-06T01:43:26.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:26.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:26.494172+0000) 2020-02-06T01:43:26.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:26.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:26.719750+0000) 2020-02-06T01:43:26.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:26.980+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:26.981129+0000) 2020-02-06T01:43:27.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:27.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:27.111137+0000) 2020-02-06T01:43:27.384 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:27.383+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:27.383269+0000) 2020-02-06T01:43:27.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:27.383+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:27.383490+0000) 2020-02-06T01:43:27.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:27.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:27.383786+0000) 2020-02-06T01:43:27.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:27.614+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:27.615282+0000) 2020-02-06T01:43:28.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:28.831+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.832817+0000) 2020-02-06T01:43:28.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:28.833+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.833586+0000) 2020-02-06T01:43:28.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:28.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.834049+0000) 2020-02-06T01:43:28.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:28.833+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.834509+0000) 2020-02-06T01:43:28.836 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:28.834+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.835218+0000) 2020-02-06T01:43:28.836 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:28.835+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.835836+0000) 2020-02-06T01:43:28.837 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:28.835+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.836138+0000) 2020-02-06T01:43:28.837 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:28.835+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.835923+0000) 2020-02-06T01:43:28.837 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:28.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.836984+0000) 2020-02-06T01:43:28.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:28.836+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.837323+0000) 2020-02-06T01:43:28.838 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:28.836+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.837627+0000) 2020-02-06T01:43:28.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:28.836+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.837802+0000) 2020-02-06T01:43:28.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:28.837+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.838336+0000) 2020-02-06T01:43:28.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:28.840+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:28.841165+0000) 2020-02-06T01:43:28.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:28.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211686 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:29.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:29.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:29.754894+0000) 2020-02-06T01:43:30.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:30.000+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:30.001381+0000) 2020-02-06T01:43:30.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:30.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:30.609959+0000) 2020-02-06T01:43:30.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:30.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:30.888579+0000) 2020-02-06T01:43:30.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:30.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:30.888579+0000) 2020-02-06T01:43:33.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:33.861+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.862018+0000) 2020-02-06T01:43:33.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:33.862+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.863692+0000) 2020-02-06T01:43:33.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:33.862+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.863765+0000) 2020-02-06T01:43:33.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:33.863+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.864171+0000) 2020-02-06T01:43:33.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:33.863+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.864626+0000) 2020-02-06T01:43:33.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:33.864+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.865262+0000) 2020-02-06T01:43:33.867 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:33.865+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.866189+0000) 2020-02-06T01:43:33.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:33.864+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.865745+0000) 2020-02-06T01:43:33.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:33.866+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.867248+0000) 2020-02-06T01:43:33.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:33.867+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.868166+0000) 2020-02-06T01:43:33.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:33.867+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.868364+0000) 2020-02-06T01:43:33.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:33.867+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.868987+0000) 2020-02-06T01:43:33.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:33.867+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.868393+0000) 2020-02-06T01:43:33.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:33.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:33.878195+0000) 2020-02-06T01:43:33.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:33.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211727 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:35.302 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:43:35.309 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:43:36.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:36.131+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:36.131271+0000) 2020-02-06T01:43:36.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:36.494+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:36.494406+0000) 2020-02-06T01:43:36.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:36.718+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:36.720004+0000) 2020-02-06T01:43:36.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:36.988+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:36.989780+0000) 2020-02-06T01:43:37.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:37.110+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:37.111390+0000) 2020-02-06T01:43:37.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:37.383+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:37.383523+0000) 2020-02-06T01:43:37.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:37.383+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:37.383706+0000) 2020-02-06T01:43:37.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:37.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:37.384035+0000) 2020-02-06T01:43:37.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:37.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:37.615517+0000) 2020-02-06T01:43:38.892 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:38.890+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.891317+0000) 2020-02-06T01:43:38.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:38.890+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.891976+0000) 2020-02-06T01:43:38.894 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:38.891+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.893260+0000) 2020-02-06T01:43:38.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:38.892+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.893786+0000) 2020-02-06T01:43:38.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:38.892+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.893816+0000) 2020-02-06T01:43:38.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:38.893+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.894297+0000) 2020-02-06T01:43:38.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:38.893+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.894855+0000) 2020-02-06T01:43:38.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:38.893+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.894395+0000) 2020-02-06T01:43:38.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:38.893+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.895012+0000) 2020-02-06T01:43:38.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:38.893+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.895332+0000) 2020-02-06T01:43:38.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:38.894+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.895550+0000) 2020-02-06T01:43:38.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:38.895+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.896146+0000) 2020-02-06T01:43:38.898 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:38.894+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.896359+0000) 2020-02-06T01:43:38.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:38.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:38.899370+0000) 2020-02-06T01:43:38.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:38.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211772 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:39.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:39.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:39.755139+0000) 2020-02-06T01:43:40.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:40.001+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:40.001624+0000) 2020-02-06T01:43:40.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:40.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:40.610214+0000) 2020-02-06T01:43:40.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:40.887+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:40.888866+0000) 2020-02-06T01:43:40.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:40.887+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:40.888869+0000) 2020-02-06T01:43:43.920 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:43.917+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.918801+0000) 2020-02-06T01:43:43.920 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:43.918+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.919329+0000) 2020-02-06T01:43:43.920 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:43.918+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.919976+0000) 2020-02-06T01:43:43.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:43.919+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.920206+0000) 2020-02-06T01:43:43.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:43.919+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.920556+0000) 2020-02-06T01:43:43.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:43.919+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.920986+0000) 2020-02-06T01:43:43.922 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:43.920+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.921672+0000) 2020-02-06T01:43:43.922 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:43.920+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.922049+0000) 2020-02-06T01:43:43.923 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:43.920+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.922231+0000) 2020-02-06T01:43:43.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:43.921+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.922682+0000) 2020-02-06T01:43:43.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:43.922+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.923167+0000) 2020-02-06T01:43:43.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:43.922+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.923728+0000) 2020-02-06T01:43:43.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:43.922+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.924139+0000) 2020-02-06T01:43:43.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:43.926+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:43.927117+0000) 2020-02-06T01:43:43.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:43.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211812 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:46.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:46.130+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:46.131529+0000) 2020-02-06T01:43:46.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:46.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:46.494638+0000) 2020-02-06T01:43:46.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:46.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:46.720309+0000) 2020-02-06T01:43:46.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:46.997+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:46.997868+0000) 2020-02-06T01:43:47.112 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:47.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:47.111621+0000) 2020-02-06T01:43:47.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:47.382+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:47.383783+0000) 2020-02-06T01:43:47.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:47.382+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:47.383985+0000) 2020-02-06T01:43:47.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:47.382+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:47.384237+0000) 2020-02-06T01:43:47.616 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:47.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:47.615700+0000) 2020-02-06T01:43:48.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:48.946+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.947316+0000) 2020-02-06T01:43:48.950 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:48.947+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.948875+0000) 2020-02-06T01:43:48.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:48.948+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.949657+0000) 2020-02-06T01:43:48.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:48.949+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.950428+0000) 2020-02-06T01:43:48.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:48.948+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.950312+0000) 2020-02-06T01:43:48.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:48.950+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.951361+0000) 2020-02-06T01:43:48.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:48.950+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.951567+0000) 2020-02-06T01:43:48.953 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:48.951+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.952586+0000) 2020-02-06T01:43:48.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:48.951+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.952770+0000) 2020-02-06T01:43:48.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:48.952+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.953393+0000) 2020-02-06T01:43:48.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:48.952+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.953496+0000) 2020-02-06T01:43:48.955 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:48.952+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.954118+0000) 2020-02-06T01:43:48.956 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:48.952+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.953866+0000) 2020-02-06T01:43:48.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:48.957+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:48.958772+0000) 2020-02-06T01:43:48.972 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:48.970+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211857 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:49.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:49.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:49.755354+0000) 2020-02-06T01:43:50.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:50.001+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:50.001851+0000) 2020-02-06T01:43:50.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:50.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:50.610499+0000) 2020-02-06T01:43:50.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:50.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:50.889136+0000) 2020-02-06T01:43:50.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:50.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:50.889136+0000) 2020-02-06T01:43:53.977 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:53.974+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.975872+0000) 2020-02-06T01:43:53.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:53.976+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.977752+0000) 2020-02-06T01:43:53.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:53.977+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.978601+0000) 2020-02-06T01:43:53.979 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:53.978+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.979071+0000) 2020-02-06T01:43:53.980 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:53.977+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.979281+0000) 2020-02-06T01:43:53.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:53.978+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.979777+0000) 2020-02-06T01:43:53.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:53.978+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.980206+0000) 2020-02-06T01:43:53.981 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:53.979+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.980978+0000) 2020-02-06T01:43:53.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:53.980+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.981367+0000) 2020-02-06T01:43:53.983 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:53.980+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.981675+0000) 2020-02-06T01:43:53.983 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:53.980+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.982025+0000) 2020-02-06T01:43:53.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:53.981+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.982273+0000) 2020-02-06T01:43:53.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:53.981+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.983242+0000) 2020-02-06T01:43:53.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:53.986+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:53.987889+0000) 2020-02-06T01:43:54.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:53.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211897 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:56.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:56.130+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:56.131786+0000) 2020-02-06T01:43:56.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:56.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:56.494864+0000) 2020-02-06T01:43:56.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:56.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:56.720533+0000) 2020-02-06T01:43:57.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:57.004+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.004751+0000) 2020-02-06T01:43:57.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:57.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.111865+0000) 2020-02-06T01:43:57.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:57.383+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.384035+0000) 2020-02-06T01:43:57.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:57.383+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.384209+0000) 2020-02-06T01:43:57.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:57.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.384500+0000) 2020-02-06T01:43:57.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:57.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:57.615996+0000) 2020-02-06T01:43:59.006 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:43:59.005+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.005156+0000) 2020-02-06T01:43:59.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:43:59.005+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.005421+0000) 2020-02-06T01:43:59.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:43:59.006+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.006186+0000) 2020-02-06T01:43:59.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:43:59.005+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.006407+0000) 2020-02-06T01:43:59.008 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:43:59.006+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.006952+0000) 2020-02-06T01:43:59.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:43:59.005+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.007124+0000) 2020-02-06T01:43:59.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:59.007+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.007949+0000) 2020-02-06T01:43:59.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:43:59.007+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.008011+0000) 2020-02-06T01:43:59.009 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:43:59.006+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.008006+0000) 2020-02-06T01:43:59.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:59.008+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.008409+0000) 2020-02-06T01:43:59.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:43:59.008+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.010083+0000) 2020-02-06T01:43:59.013 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:43:59.008+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.009724+0000) 2020-02-06T01:43:59.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:43:59.008+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.010095+0000) 2020-02-06T01:43:59.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:59.014+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.014683+0000) 2020-02-06T01:43:59.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:43:59.025+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:43:59.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:43:59.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:43:59.755567+0000) 2020-02-06T01:44:00.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:00.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:00.002123+0000) 2020-02-06T01:44:00.611 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:00.609+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:00.610739+0000) 2020-02-06T01:44:00.890 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:00.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:00.889397+0000) 2020-02-06T01:44:00.890 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:00.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:00.889397+0000) 2020-02-06T01:44:04.034 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:04.032+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.033236+0000) 2020-02-06T01:44:04.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:04.032+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.033795+0000) 2020-02-06T01:44:04.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:04.033+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.034736+0000) 2020-02-06T01:44:04.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:04.033+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.034953+0000) 2020-02-06T01:44:04.037 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:04.034+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.035402+0000) 2020-02-06T01:44:04.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:04.035+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.036543+0000) 2020-02-06T01:44:04.038 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:04.035+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.036747+0000) 2020-02-06T01:44:04.039 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:04.035+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.036713+0000) 2020-02-06T01:44:04.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:04.035+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.037339+0000) 2020-02-06T01:44:04.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:04.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.038253+0000) 2020-02-06T01:44:04.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:04.038+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.039193+0000) 2020-02-06T01:44:04.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:04.036+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.038037+0000) 2020-02-06T01:44:04.041 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:04.037+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.038573+0000) 2020-02-06T01:44:04.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:04.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:04.043798+0000) 2020-02-06T01:44:04.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:04.053+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 211982 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:05.473 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:44:05.479 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:44:06.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:06.131+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:06.132047+0000) 2020-02-06T01:44:06.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:06.493+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:06.495105+0000) 2020-02-06T01:44:06.721 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:06.719+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:06.720747+0000) 2020-02-06T01:44:07.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:07.011+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.012902+0000) 2020-02-06T01:44:07.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:07.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.112139+0000) 2020-02-06T01:44:07.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:07.383+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.384293+0000) 2020-02-06T01:44:07.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:07.383+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.384482+0000) 2020-02-06T01:44:07.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:07.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.384761+0000) 2020-02-06T01:44:07.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:07.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:07.616254+0000) 2020-02-06T01:44:09.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:09.059+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.060776+0000) 2020-02-06T01:44:09.063 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:09.060+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.062067+0000) 2020-02-06T01:44:09.063 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:09.061+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.062444+0000) 2020-02-06T01:44:09.064 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:09.061+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.063110+0000) 2020-02-06T01:44:09.064 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:09.062+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.063713+0000) 2020-02-06T01:44:09.065 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:09.062+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.063954+0000) 2020-02-06T01:44:09.065 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:09.062+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.064062+0000) 2020-02-06T01:44:09.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:09.063+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.064661+0000) 2020-02-06T01:44:09.066 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:09.063+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.065027+0000) 2020-02-06T01:44:09.066 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:09.064+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.066085+0000) 2020-02-06T01:44:09.067 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:09.065+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.066254+0000) 2020-02-06T01:44:09.067 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:09.065+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.066731+0000) 2020-02-06T01:44:09.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:09.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.071301+0000) 2020-02-06T01:44:09.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:09.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.071655+0000) 2020-02-06T01:44:09.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:09.083+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:09.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:09.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:09.755781+0000) 2020-02-06T01:44:10.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:10.001+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:10.002330+0000) 2020-02-06T01:44:10.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:10.610+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:10.611013+0000) 2020-02-06T01:44:10.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:10.888+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:10.889669+0000) 2020-02-06T01:44:10.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:10.888+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:10.889661+0000) 2020-02-06T01:44:14.093 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:14.091+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.092612+0000) 2020-02-06T01:44:14.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:14.091+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.092971+0000) 2020-02-06T01:44:14.094 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:14.092+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.093558+0000) 2020-02-06T01:44:14.095 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:14.092+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.093837+0000) 2020-02-06T01:44:14.095 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:14.092+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.093675+0000) 2020-02-06T01:44:14.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:14.093+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.094478+0000) 2020-02-06T01:44:14.096 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:14.093+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.094715+0000) 2020-02-06T01:44:14.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:14.093+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.094789+0000) 2020-02-06T01:44:14.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:14.093+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.095314+0000) 2020-02-06T01:44:14.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:14.093+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.095314+0000) 2020-02-06T01:44:14.097 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:14.094+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.095455+0000) 2020-02-06T01:44:14.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:14.094+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.095231+0000) 2020-02-06T01:44:14.098 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:14.094+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.095988+0000) 2020-02-06T01:44:14.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:14.101+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:14.102555+0000) 2020-02-06T01:44:14.113 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:14.111+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:16.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:16.131+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:16.132324+0000) 2020-02-06T01:44:16.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:16.494+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:16.495365+0000) 2020-02-06T01:44:16.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:16.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:16.720960+0000) 2020-02-06T01:44:17.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:17.020+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.021421+0000) 2020-02-06T01:44:17.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:17.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.112394+0000) 2020-02-06T01:44:17.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:17.383+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.384540+0000) 2020-02-06T01:44:17.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:17.383+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.384747+0000) 2020-02-06T01:44:17.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:17.383+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.384962+0000) 2020-02-06T01:44:17.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:17.615+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:17.616571+0000) 2020-02-06T01:44:19.118 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:19.116+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.117882+0000) 2020-02-06T01:44:19.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:19.117+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.119277+0000) 2020-02-06T01:44:19.120 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:19.118+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.119806+0000) 2020-02-06T01:44:19.121 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:19.119+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.120443+0000) 2020-02-06T01:44:19.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:19.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.122381+0000) 2020-02-06T01:44:19.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:19.121+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.122600+0000) 2020-02-06T01:44:19.124 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:19.121+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.122951+0000) 2020-02-06T01:44:19.124 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:19.122+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.123281+0000) 2020-02-06T01:44:19.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:19.122+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.124054+0000) 2020-02-06T01:44:19.125 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:19.122+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.124272+0000) 2020-02-06T01:44:19.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:19.122+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.124015+0000) 2020-02-06T01:44:19.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:19.123+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.124616+0000) 2020-02-06T01:44:19.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:19.123+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.124837+0000) 2020-02-06T01:44:19.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:19.127+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.128456+0000) 2020-02-06T01:44:19.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:19.141+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212113 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:19.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:19.754+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:19.756033+0000) 2020-02-06T01:44:20.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:20.001+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:20.002552+0000) 2020-02-06T01:44:20.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:20.610+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:20.611252+0000) 2020-02-06T01:44:20.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:20.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:20.889957+0000) 2020-02-06T01:44:20.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:20.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:20.889956+0000) 2020-02-06T01:44:24.150 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:24.147+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.148440+0000) 2020-02-06T01:44:24.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:24.147+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.149021+0000) 2020-02-06T01:44:24.150 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:24.148+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.149689+0000) 2020-02-06T01:44:24.151 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:24.148+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.150227+0000) 2020-02-06T01:44:24.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:24.149+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.150564+0000) 2020-02-06T01:44:24.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:24.149+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.150564+0000) 2020-02-06T01:44:24.152 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:24.149+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.150751+0000) 2020-02-06T01:44:24.153 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:24.149+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.151370+0000) 2020-02-06T01:44:24.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:24.151+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.152611+0000) 2020-02-06T01:44:24.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:24.151+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.152868+0000) 2020-02-06T01:44:24.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:24.152+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.153573+0000) 2020-02-06T01:44:24.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:24.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.153812+0000) 2020-02-06T01:44:24.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:24.152+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.153883+0000) 2020-02-06T01:44:24.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:24.156+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:24.157170+0000) 2020-02-06T01:44:24.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:24.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:26.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:26.132+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:26.132569+0000) 2020-02-06T01:44:26.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:26.495+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:26.495651+0000) 2020-02-06T01:44:26.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:26.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:26.721188+0000) 2020-02-06T01:44:27.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:27.032+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.033432+0000) 2020-02-06T01:44:27.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:27.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.112614+0000) 2020-02-06T01:44:27.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:27.384+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.384795+0000) 2020-02-06T01:44:27.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:27.384+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.384987+0000) 2020-02-06T01:44:27.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:27.384+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.385194+0000) 2020-02-06T01:44:27.617 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:27.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:27.616791+0000) 2020-02-06T01:44:29.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:29.176+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.177938+0000) 2020-02-06T01:44:29.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:29.177+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.178543+0000) 2020-02-06T01:44:29.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:29.178+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.179306+0000) 2020-02-06T01:44:29.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:29.178+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.179801+0000) 2020-02-06T01:44:29.181 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:29.179+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.180046+0000) 2020-02-06T01:44:29.182 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:29.180+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.180640+0000) 2020-02-06T01:44:29.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:29.181+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.181596+0000) 2020-02-06T01:44:29.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:29.181+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.182308+0000) 2020-02-06T01:44:29.183 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:29.181+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.182345+0000) 2020-02-06T01:44:29.184 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:29.182+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.183156+0000) 2020-02-06T01:44:29.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:29.182+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.182985+0000) 2020-02-06T01:44:29.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:29.183+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.184219+0000) 2020-02-06T01:44:29.185 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:29.183+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.184237+0000) 2020-02-06T01:44:29.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:29.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.189262+0000) 2020-02-06T01:44:29.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:29.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:29.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:29.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:29.756271+0000) 2020-02-06T01:44:30.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:30.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:30.002788+0000) 2020-02-06T01:44:30.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:30.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:30.611493+0000) 2020-02-06T01:44:30.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:30.889+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:30.890227+0000) 2020-02-06T01:44:30.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:30.889+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:30.890229+0000) 2020-02-06T01:44:34.208 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:34.206+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.207533+0000) 2020-02-06T01:44:34.209 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:34.207+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.208053+0000) 2020-02-06T01:44:34.209 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:34.207+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.208703+0000) 2020-02-06T01:44:34.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:34.208+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.209246+0000) 2020-02-06T01:44:34.211 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:34.209+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.210017+0000) 2020-02-06T01:44:34.211 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:34.209+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.209821+0000) 2020-02-06T01:44:34.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:34.209+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.210836+0000) 2020-02-06T01:44:34.212 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:34.210+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.211479+0000) 2020-02-06T01:44:34.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:34.211+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.211477+0000) 2020-02-06T01:44:34.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:34.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.212046+0000) 2020-02-06T01:44:34.214 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:34.212+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.213189+0000) 2020-02-06T01:44:34.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:34.212+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.213314+0000) 2020-02-06T01:44:34.214 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:34.212+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.213893+0000) 2020-02-06T01:44:34.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:34.216+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:34.217536+0000) 2020-02-06T01:44:34.231 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:34.229+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212237 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:35.637 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:44:35.644 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:44:36.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:36.132+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:36.132825+0000) 2020-02-06T01:44:36.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:36.495+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:36.495905+0000) 2020-02-06T01:44:36.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:36.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:36.721376+0000) 2020-02-06T01:44:37.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:37.044+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.045255+0000) 2020-02-06T01:44:37.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:37.111+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.112826+0000) 2020-02-06T01:44:37.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:37.384+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.385028+0000) 2020-02-06T01:44:37.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:37.384+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.385186+0000) 2020-02-06T01:44:37.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:37.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.385522+0000) 2020-02-06T01:44:37.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:37.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:37.617028+0000) 2020-02-06T01:44:39.236 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:39.234+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.235351+0000) 2020-02-06T01:44:39.237 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:39.235+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.236325+0000) 2020-02-06T01:44:39.237 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:39.235+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.236988+0000) 2020-02-06T01:44:39.238 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:39.236+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.237790+0000) 2020-02-06T01:44:39.239 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:39.238+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.238723+0000) 2020-02-06T01:44:39.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:39.237+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.239026+0000) 2020-02-06T01:44:39.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:39.238+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.239214+0000) 2020-02-06T01:44:39.240 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:39.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.239685+0000) 2020-02-06T01:44:39.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:39.238+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.240018+0000) 2020-02-06T01:44:39.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:39.240+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.240642+0000) 2020-02-06T01:44:39.242 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:39.240+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.241236+0000) 2020-02-06T01:44:39.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:39.241+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.241671+0000) 2020-02-06T01:44:39.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:39.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.245803+0000) 2020-02-06T01:44:39.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:39.244+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.246038+0000) 2020-02-06T01:44:39.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:39.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212283 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:39.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:39.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:39.756479+0000) 2020-02-06T01:44:40.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:40.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:40.002989+0000) 2020-02-06T01:44:40.612 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:40.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:40.611739+0000) 2020-02-06T01:44:40.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:40.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:40.890490+0000) 2020-02-06T01:44:40.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:40.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:40.890490+0000) 2020-02-06T01:44:44.266 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:44.264+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.265710+0000) 2020-02-06T01:44:44.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:44.265+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.266340+0000) 2020-02-06T01:44:44.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:44.266+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.267071+0000) 2020-02-06T01:44:44.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:44.266+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.267182+0000) 2020-02-06T01:44:44.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:44.266+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.267636+0000) 2020-02-06T01:44:44.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:44.267+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.268145+0000) 2020-02-06T01:44:44.269 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:44.268+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.268419+0000) 2020-02-06T01:44:44.270 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:44.267+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.268558+0000) 2020-02-06T01:44:44.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:44.268+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.269199+0000) 2020-02-06T01:44:44.271 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:44.269+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.270123+0000) 2020-02-06T01:44:44.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:44.270+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.271021+0000) 2020-02-06T01:44:44.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:44.270+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.271070+0000) 2020-02-06T01:44:44.272 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:44.270+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.271491+0000) 2020-02-06T01:44:44.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:44.276+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:44.277635+0000) 2020-02-06T01:44:44.291 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:44.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212322 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:46.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:46.132+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:46.133131+0000) 2020-02-06T01:44:46.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:46.495+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:46.496153+0000) 2020-02-06T01:44:46.722 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:46.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:46.721617+0000) 2020-02-06T01:44:47.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:47.053+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.053326+0000) 2020-02-06T01:44:47.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:47.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.113117+0000) 2020-02-06T01:44:47.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:47.384+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.385259+0000) 2020-02-06T01:44:47.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:47.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.385492+0000) 2020-02-06T01:44:47.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:47.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.385777+0000) 2020-02-06T01:44:47.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:47.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:47.617320+0000) 2020-02-06T01:44:49.295 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:49.293+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.294408+0000) 2020-02-06T01:44:49.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:49.294+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.295290+0000) 2020-02-06T01:44:49.297 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:49.295+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.296177+0000) 2020-02-06T01:44:49.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:49.296+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.296847+0000) 2020-02-06T01:44:49.298 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:49.296+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.297357+0000) 2020-02-06T01:44:49.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:49.297+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.297864+0000) 2020-02-06T01:44:49.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:49.297+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.298218+0000) 2020-02-06T01:44:49.300 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:49.298+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.299371+0000) 2020-02-06T01:44:49.301 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:49.299+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.300325+0000) 2020-02-06T01:44:49.301 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:49.299+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.300805+0000) 2020-02-06T01:44:49.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:49.300+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.301460+0000) 2020-02-06T01:44:49.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:49.300+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.301544+0000) 2020-02-06T01:44:49.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:49.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.306239+0000) 2020-02-06T01:44:49.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:49.305+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.306473+0000) 2020-02-06T01:44:49.320 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:49.318+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:49.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:49.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:49.756690+0000) 2020-02-06T01:44:50.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:50.003+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:50.003261+0000) 2020-02-06T01:44:50.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:50.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:50.611971+0000) 2020-02-06T01:44:50.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:50.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:50.890772+0000) 2020-02-06T01:44:50.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:50.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:50.890770+0000) 2020-02-06T01:44:54.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:54.326+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.326538+0000) 2020-02-06T01:44:54.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:54.327+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.327223+0000) 2020-02-06T01:44:54.328 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:54.327+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.327894+0000) 2020-02-06T01:44:54.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:54.328+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.328284+0000) 2020-02-06T01:44:54.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:54.328+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.328805+0000) 2020-02-06T01:44:54.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:54.329+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.329190+0000) 2020-02-06T01:44:54.331 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:54.329+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.329422+0000) 2020-02-06T01:44:54.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:54.329+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.329830+0000) 2020-02-06T01:44:54.331 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:54.330+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.330264+0000) 2020-02-06T01:44:54.332 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:54.330+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.330544+0000) 2020-02-06T01:44:54.332 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:54.330+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.330953+0000) 2020-02-06T01:44:54.332 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:54.330+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.331032+0000) 2020-02-06T01:44:54.332 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:54.331+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.331355+0000) 2020-02-06T01:44:54.338 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:54.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:54.337199+0000) 2020-02-06T01:44:54.348 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:54.347+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:56.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:56.133+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:56.133475+0000) 2020-02-06T01:44:56.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:56.496+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:56.496387+0000) 2020-02-06T01:44:56.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:56.720+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:56.721846+0000) 2020-02-06T01:44:57.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:57.060+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.061381+0000) 2020-02-06T01:44:57.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:57.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.113371+0000) 2020-02-06T01:44:57.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:57.385+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.385519+0000) 2020-02-06T01:44:57.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:57.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.385714+0000) 2020-02-06T01:44:57.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:57.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.386015+0000) 2020-02-06T01:44:57.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:57.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:57.617549+0000) 2020-02-06T01:44:59.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:44:59.351+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.352955+0000) 2020-02-06T01:44:59.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:59.352+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.353556+0000) 2020-02-06T01:44:59.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:44:59.354+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.354762+0000) 2020-02-06T01:44:59.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:44:59.354+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.355935+0000) 2020-02-06T01:44:59.357 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:44:59.355+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.356417+0000) 2020-02-06T01:44:59.357 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:44:59.355+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.356871+0000) 2020-02-06T01:44:59.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:59.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.357489+0000) 2020-02-06T01:44:59.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:44:59.356+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.357837+0000) 2020-02-06T01:44:59.359 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:44:59.357+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.358378+0000) 2020-02-06T01:44:59.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:44:59.358+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.358846+0000) 2020-02-06T01:44:59.360 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:44:59.358+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.359145+0000) 2020-02-06T01:44:59.360 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:44:59.358+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.359357+0000) 2020-02-06T01:44:59.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:44:59.359+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.359462+0000) 2020-02-06T01:44:59.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:59.360+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.361580+0000) 2020-02-06T01:44:59.379 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:44:59.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212453 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:44:59.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:44:59.755+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:44:59.756928+0000) 2020-02-06T01:45:00.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:00.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:00.003473+0000) 2020-02-06T01:45:00.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:00.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:00.612207+0000) 2020-02-06T01:45:00.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:00.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:00.891039+0000) 2020-02-06T01:45:00.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:00.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:00.891035+0000) 2020-02-06T01:45:04.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:04.382+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.383731+0000) 2020-02-06T01:45:04.385 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:04.383+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.384157+0000) 2020-02-06T01:45:04.386 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:04.384+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.384713+0000) 2020-02-06T01:45:04.386 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:04.384+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.385467+0000) 2020-02-06T01:45:04.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:04.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.385517+0000) 2020-02-06T01:45:04.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:04.385+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.386124+0000) 2020-02-06T01:45:04.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:04.385+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.386310+0000) 2020-02-06T01:45:04.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:04.386+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.386932+0000) 2020-02-06T01:45:04.388 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:04.386+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.387296+0000) 2020-02-06T01:45:04.389 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:04.386+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.387555+0000) 2020-02-06T01:45:04.389 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:04.387+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.388324+0000) 2020-02-06T01:45:04.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:04.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.388231+0000) 2020-02-06T01:45:04.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:04.388+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.389062+0000) 2020-02-06T01:45:04.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:04.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:04.391461+0000) 2020-02-06T01:45:04.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:04.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212492 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:05.784 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:45:05.791 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:45:06.134 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:06.133+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:06.133752+0000) 2020-02-06T01:45:06.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:06.496+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:06.496682+0000) 2020-02-06T01:45:06.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:06.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:06.722107+0000) 2020-02-06T01:45:07.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:07.068+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.069557+0000) 2020-02-06T01:45:07.114 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:07.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.113602+0000) 2020-02-06T01:45:07.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:07.384+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.385762+0000) 2020-02-06T01:45:07.388 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:07.384+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.385984+0000) 2020-02-06T01:45:07.388 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:07.384+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.386264+0000) 2020-02-06T01:45:07.618 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:07.616+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:07.617857+0000) 2020-02-06T01:45:09.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:09.411+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.412502+0000) 2020-02-06T01:45:09.414 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:09.412+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.413293+0000) 2020-02-06T01:45:09.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:09.413+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.414972+0000) 2020-02-06T01:45:09.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:09.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.414686+0000) 2020-02-06T01:45:09.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:09.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.415459+0000) 2020-02-06T01:45:09.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:09.414+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.416188+0000) 2020-02-06T01:45:09.418 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:09.415+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.416116+0000) 2020-02-06T01:45:09.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:09.415+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.416658+0000) 2020-02-06T01:45:09.418 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:09.416+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.417141+0000) 2020-02-06T01:45:09.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:09.417+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.418686+0000) 2020-02-06T01:45:09.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:09.417+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.418970+0000) 2020-02-06T01:45:09.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:09.417+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.419215+0000) 2020-02-06T01:45:09.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:09.418+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.419445+0000) 2020-02-06T01:45:09.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:09.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.421964+0000) 2020-02-06T01:45:09.436 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:09.434+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:09.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:09.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:09.757134+0000) 2020-02-06T01:45:10.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:10.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:10.003671+0000) 2020-02-06T01:45:10.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:10.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:10.612487+0000) 2020-02-06T01:45:10.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:10.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:10.891344+0000) 2020-02-06T01:45:10.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:10.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:10.891345+0000) 2020-02-06T01:45:14.442 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:14.439+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.440894+0000) 2020-02-06T01:45:14.442 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:14.440+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.441670+0000) 2020-02-06T01:45:14.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:14.441+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.442495+0000) 2020-02-06T01:45:14.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:14.441+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.442869+0000) 2020-02-06T01:45:14.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:14.442+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.443521+0000) 2020-02-06T01:45:14.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:14.443+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.444935+0000) 2020-02-06T01:45:14.446 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:14.444+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.445517+0000) 2020-02-06T01:45:14.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:14.445+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.446140+0000) 2020-02-06T01:45:14.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:14.446+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.447138+0000) 2020-02-06T01:45:14.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:14.446+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.447589+0000) 2020-02-06T01:45:14.449 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:14.446+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.447881+0000) 2020-02-06T01:45:14.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:14.446+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.448078+0000) 2020-02-06T01:45:14.449 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:14.446+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.447851+0000) 2020-02-06T01:45:14.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:14.453+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:14.454596+0000) 2020-02-06T01:45:14.465 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:14.463+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:16.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:16.133+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:16.134030+0000) 2020-02-06T01:45:16.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:16.495+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:16.496953+0000) 2020-02-06T01:45:16.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:16.721+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:16.722361+0000) 2020-02-06T01:45:17.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:17.077+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.078025+0000) 2020-02-06T01:45:17.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:17.112+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.113849+0000) 2020-02-06T01:45:17.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:17.385+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.385997+0000) 2020-02-06T01:45:17.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:17.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.386210+0000) 2020-02-06T01:45:17.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:17.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.386532+0000) 2020-02-06T01:45:17.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:17.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:17.618147+0000) 2020-02-06T01:45:19.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:19.469+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.470952+0000) 2020-02-06T01:45:19.472 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:19.470+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.471726+0000) 2020-02-06T01:45:19.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:19.470+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.472226+0000) 2020-02-06T01:45:19.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:19.472+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.473379+0000) 2020-02-06T01:45:19.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:19.472+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.473882+0000) 2020-02-06T01:45:19.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:19.473+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.474760+0000) 2020-02-06T01:45:19.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:19.473+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.474667+0000) 2020-02-06T01:45:19.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:19.473+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.475013+0000) 2020-02-06T01:45:19.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:19.474+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.475699+0000) 2020-02-06T01:45:19.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:19.474+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.475939+0000) 2020-02-06T01:45:19.478 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:19.475+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.476255+0000) 2020-02-06T01:45:19.478 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:19.474+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.476124+0000) 2020-02-06T01:45:19.478 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:19.475+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.476743+0000) 2020-02-06T01:45:19.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:19.478+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.479494+0000) 2020-02-06T01:45:19.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:19.492+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:19.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:19.756+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:19.757347+0000) 2020-02-06T01:45:20.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:20.002+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:20.003882+0000) 2020-02-06T01:45:20.613 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:20.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:20.612715+0000) 2020-02-06T01:45:20.892 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:20.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:20.891613+0000) 2020-02-06T01:45:20.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:20.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:20.891614+0000) 2020-02-06T01:45:24.501 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:24.498+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.499911+0000) 2020-02-06T01:45:24.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:24.498+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.500286+0000) 2020-02-06T01:45:24.502 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:24.499+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.501046+0000) 2020-02-06T01:45:24.502 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:24.499+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.500849+0000) 2020-02-06T01:45:24.503 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:24.500+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.501633+0000) 2020-02-06T01:45:24.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:24.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.501817+0000) 2020-02-06T01:45:24.503 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:24.500+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.502057+0000) 2020-02-06T01:45:24.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:24.501+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.502564+0000) 2020-02-06T01:45:24.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:24.501+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.502705+0000) 2020-02-06T01:45:24.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:24.501+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.503295+0000) 2020-02-06T01:45:24.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:24.501+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.503291+0000) 2020-02-06T01:45:24.505 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:24.503+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.504606+0000) 2020-02-06T01:45:24.505 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:24.503+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.504699+0000) 2020-02-06T01:45:24.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:24.511+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:24.512526+0000) 2020-02-06T01:45:24.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:24.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:26.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:26.133+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:26.134217+0000) 2020-02-06T01:45:26.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:26.495+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:26.497193+0000) 2020-02-06T01:45:26.723 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:26.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:26.722586+0000) 2020-02-06T01:45:27.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:27.085+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.086129+0000) 2020-02-06T01:45:27.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:27.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.114098+0000) 2020-02-06T01:45:27.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:27.385+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.386251+0000) 2020-02-06T01:45:27.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:27.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.386486+0000) 2020-02-06T01:45:27.388 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:27.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.386798+0000) 2020-02-06T01:45:27.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:27.617+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:27.618463+0000) 2020-02-06T01:45:29.528 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:29.526+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.527380+0000) 2020-02-06T01:45:29.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:29.526+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.527954+0000) 2020-02-06T01:45:29.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:29.526+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.528364+0000) 2020-02-06T01:45:29.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:29.528+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.529263+0000) 2020-02-06T01:45:29.530 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:29.528+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.529766+0000) 2020-02-06T01:45:29.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:29.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.529728+0000) 2020-02-06T01:45:29.531 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:29.529+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.530187+0000) 2020-02-06T01:45:29.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:29.529+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.530667+0000) 2020-02-06T01:45:29.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:29.529+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.531345+0000) 2020-02-06T01:45:29.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:29.530+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.531517+0000) 2020-02-06T01:45:29.533 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:29.530+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.531905+0000) 2020-02-06T01:45:29.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:29.531+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.532694+0000) 2020-02-06T01:45:29.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:29.531+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.532888+0000) 2020-02-06T01:45:29.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:29.535+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.536993+0000) 2020-02-06T01:45:29.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:29.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212707 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:29.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:29.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:29.757556+0000) 2020-02-06T01:45:30.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:30.003+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:30.004125+0000) 2020-02-06T01:45:30.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:30.612+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:30.612956+0000) 2020-02-06T01:45:30.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:30.890+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:30.891906+0000) 2020-02-06T01:45:30.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:30.890+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:30.891905+0000) 2020-02-06T01:45:34.557 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:34.555+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.556173+0000) 2020-02-06T01:45:34.558 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:34.555+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.556886+0000) 2020-02-06T01:45:34.558 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:34.556+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.557400+0000) 2020-02-06T01:45:34.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:34.556+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.557961+0000) 2020-02-06T01:45:34.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:34.557+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.558512+0000) 2020-02-06T01:45:34.560 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:34.557+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.558860+0000) 2020-02-06T01:45:34.561 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:34.557+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.559223+0000) 2020-02-06T01:45:34.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:34.559+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.560179+0000) 2020-02-06T01:45:34.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:34.559+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.560762+0000) 2020-02-06T01:45:34.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:34.559+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.560955+0000) 2020-02-06T01:45:34.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:34.559+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.561266+0000) 2020-02-06T01:45:34.564 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:34.559+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.560870+0000) 2020-02-06T01:45:34.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:34.560+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.561494+0000) 2020-02-06T01:45:34.564 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:34.562+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:34.563914+0000) 2020-02-06T01:45:34.578 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:34.576+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:35.939 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:45:35.946 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:45:36.135 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:36.133+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:36.134517+0000) 2020-02-06T01:45:36.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:36.496+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:36.497491+0000) 2020-02-06T01:45:36.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:36.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:36.722840+0000) 2020-02-06T01:45:37.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:37.094+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.094460+0000) 2020-02-06T01:45:37.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:37.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.114244+0000) 2020-02-06T01:45:37.387 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:37.385+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.386534+0000) 2020-02-06T01:45:37.388 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:37.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.386729+0000) 2020-02-06T01:45:37.388 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:37.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.387018+0000) 2020-02-06T01:45:37.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:37.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:37.618734+0000) 2020-02-06T01:45:39.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:39.584+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.585617+0000) 2020-02-06T01:45:39.587 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:39.584+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.585983+0000) 2020-02-06T01:45:39.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:39.584+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.585640+0000) 2020-02-06T01:45:39.588 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:39.585+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.586522+0000) 2020-02-06T01:45:39.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:39.585+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.586860+0000) 2020-02-06T01:45:39.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:39.586+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.587567+0000) 2020-02-06T01:45:39.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:39.587+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.588187+0000) 2020-02-06T01:45:39.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:39.587+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.588802+0000) 2020-02-06T01:45:39.590 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:39.587+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.589202+0000) 2020-02-06T01:45:39.590 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:39.588+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.589515+0000) 2020-02-06T01:45:39.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:39.588+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.589071+0000) 2020-02-06T01:45:39.591 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:39.588+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.590082+0000) 2020-02-06T01:45:39.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:39.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.590486+0000) 2020-02-06T01:45:39.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:39.592+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.593508+0000) 2020-02-06T01:45:39.606 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:39.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212791 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:39.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:39.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:39.757778+0000) 2020-02-06T01:45:40.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:40.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:40.004334+0000) 2020-02-06T01:45:40.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:40.611+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:40.613196+0000) 2020-02-06T01:45:40.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:40.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:40.892175+0000) 2020-02-06T01:45:40.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:40.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:40.892175+0000) 2020-02-06T01:45:44.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:44.609+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.610949+0000) 2020-02-06T01:45:44.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:44.610+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.611769+0000) 2020-02-06T01:45:44.613 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:44.610+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.612137+0000) 2020-02-06T01:45:44.614 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:44.611+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.612605+0000) 2020-02-06T01:45:44.614 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:44.611+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.612413+0000) 2020-02-06T01:45:44.615 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:44.612+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.613140+0000) 2020-02-06T01:45:44.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:44.612+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.613781+0000) 2020-02-06T01:45:44.616 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:44.612+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.613808+0000) 2020-02-06T01:45:44.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:44.613+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.614689+0000) 2020-02-06T01:45:44.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:44.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.615058+0000) 2020-02-06T01:45:44.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:44.614+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.615669+0000) 2020-02-06T01:45:44.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:44.614+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.615507+0000) 2020-02-06T01:45:44.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:44.614+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.616045+0000) 2020-02-06T01:45:44.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:44.622+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:44.623662+0000) 2020-02-06T01:45:44.635 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:44.633+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212829 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:46.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:46.134+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:46.134787+0000) 2020-02-06T01:45:46.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:46.497+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:46.497761+0000) 2020-02-06T01:45:46.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:46.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:46.723114+0000) 2020-02-06T01:45:47.107 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:47.106+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.106560+0000) 2020-02-06T01:45:47.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:47.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.114451+0000) 2020-02-06T01:45:47.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:47.385+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.386718+0000) 2020-02-06T01:45:47.388 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:47.385+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.386992+0000) 2020-02-06T01:45:47.388 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:47.385+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.387273+0000) 2020-02-06T01:45:47.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:47.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:47.618968+0000) 2020-02-06T01:45:49.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:49.640+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.640812+0000) 2020-02-06T01:45:49.643 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:49.640+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.641903+0000) 2020-02-06T01:45:49.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:49.642+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.642589+0000) 2020-02-06T01:45:49.645 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:49.641+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.642573+0000) 2020-02-06T01:45:49.645 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:49.641+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.643073+0000) 2020-02-06T01:45:49.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:49.642+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.643748+0000) 2020-02-06T01:45:49.647 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:49.643+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.643398+0000) 2020-02-06T01:45:49.647 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:49.644+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.644260+0000) 2020-02-06T01:45:49.647 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:49.644+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.644587+0000) 2020-02-06T01:45:49.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:49.645+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.645281+0000) 2020-02-06T01:45:49.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:49.645+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.645965+0000) 2020-02-06T01:45:49.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:49.644+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.646212+0000) 2020-02-06T01:45:49.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:49.644+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.646416+0000) 2020-02-06T01:45:49.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:49.651+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.651985+0000) 2020-02-06T01:45:49.664 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:49.662+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212875 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:49.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:49.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:49.758008+0000) 2020-02-06T01:45:50.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:50.003+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:50.004557+0000) 2020-02-06T01:45:50.614 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:50.612+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:50.613481+0000) 2020-02-06T01:45:50.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:50.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:50.892448+0000) 2020-02-06T01:45:50.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:50.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:50.892449+0000) 2020-02-06T01:45:54.670 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:54.668+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.669774+0000) 2020-02-06T01:45:54.671 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:54.669+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.670310+0000) 2020-02-06T01:45:54.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:54.669+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.670656+0000) 2020-02-06T01:45:54.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:54.669+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.670670+0000) 2020-02-06T01:45:54.672 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:54.670+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.671990+0000) 2020-02-06T01:45:54.673 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:54.670+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.671717+0000) 2020-02-06T01:45:54.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:54.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.672466+0000) 2020-02-06T01:45:54.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:54.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.672996+0000) 2020-02-06T01:45:54.674 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:54.672+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.673609+0000) 2020-02-06T01:45:54.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:54.672+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.674339+0000) 2020-02-06T01:45:54.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:54.673+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.674503+0000) 2020-02-06T01:45:54.680 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:54.673+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.675147+0000) 2020-02-06T01:45:54.680 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:54.674+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.675698+0000) 2020-02-06T01:45:54.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:54.678+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:54.680003+0000) 2020-02-06T01:45:54.693 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:54.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212914 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:56.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:56.134+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:56.135044+0000) 2020-02-06T01:45:56.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:56.497+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:56.498000+0000) 2020-02-06T01:45:56.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:56.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:56.723373+0000) 2020-02-06T01:45:57.115 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:57.113+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.114706+0000) 2020-02-06T01:45:57.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:57.115+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.116534+0000) 2020-02-06T01:45:57.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:57.386+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.386966+0000) 2020-02-06T01:45:57.388 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:57.386+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.387260+0000) 2020-02-06T01:45:57.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:57.386+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.387514+0000) 2020-02-06T01:45:57.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:57.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:57.619193+0000) 2020-02-06T01:45:59.699 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:45:59.698+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.698505+0000) 2020-02-06T01:45:59.700 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:45:59.697+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.698937+0000) 2020-02-06T01:45:59.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:45:59.698+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.699396+0000) 2020-02-06T01:45:59.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:59.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.700533+0000) 2020-02-06T01:45:59.701 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:45:59.699+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.699993+0000) 2020-02-06T01:45:59.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:59.699+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.701063+0000) 2020-02-06T01:45:59.705 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:45:59.700+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.701379+0000) 2020-02-06T01:45:59.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:45:59.700+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.701727+0000) 2020-02-06T01:45:59.705 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:45:59.701+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.702568+0000) 2020-02-06T01:45:59.705 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:45:59.701+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.702213+0000) 2020-02-06T01:45:59.706 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:45:59.702+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.703120+0000) 2020-02-06T01:45:59.706 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:45:59.703+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.703895+0000) 2020-02-06T01:45:59.707 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:45:59.704+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.704496+0000) 2020-02-06T01:45:59.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:59.707+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.708732+0000) 2020-02-06T01:45:59.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:45:59.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:45:59.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:45:59.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:45:59.758186+0000) 2020-02-06T01:46:00.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:00.003+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:00.004785+0000) 2020-02-06T01:46:00.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:00.613+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:00.613700+0000) 2020-02-06T01:46:00.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:00.891+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:00.892745+0000) 2020-02-06T01:46:00.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:00.891+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:00.892745+0000) 2020-02-06T01:46:04.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:04.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.726950+0000) 2020-02-06T01:46:04.728 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:04.726+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.727334+0000) 2020-02-06T01:46:04.729 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:04.726+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.728005+0000) 2020-02-06T01:46:04.729 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:04.727+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.728397+0000) 2020-02-06T01:46:04.730 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:04.728+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.728912+0000) 2020-02-06T01:46:04.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:04.728+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.729068+0000) 2020-02-06T01:46:04.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:04.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.729998+0000) 2020-02-06T01:46:04.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:04.729+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.730105+0000) 2020-02-06T01:46:04.732 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:04.730+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.730514+0000) 2020-02-06T01:46:04.732 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:04.729+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.730174+0000) 2020-02-06T01:46:04.732 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:04.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.730921+0000) 2020-02-06T01:46:04.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:04.729+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.730405+0000) 2020-02-06T01:46:04.733 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:04.730+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.731138+0000) 2020-02-06T01:46:04.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:04.733+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:04.734896+0000) 2020-02-06T01:46:04.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:04.746+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 212996 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:06.101 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:46:06.108 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:46:06.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:06.134+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:06.135246+0000) 2020-02-06T01:46:06.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:06.497+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:06.498249+0000) 2020-02-06T01:46:06.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:06.722+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:06.723618+0000) 2020-02-06T01:46:07.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:07.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.114981+0000) 2020-02-06T01:46:07.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:07.124+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.125569+0000) 2020-02-06T01:46:07.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:07.386+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.387270+0000) 2020-02-06T01:46:07.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:07.387+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.387483+0000) 2020-02-06T01:46:07.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:07.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.387808+0000) 2020-02-06T01:46:07.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:07.618+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:07.619434+0000) 2020-02-06T01:46:09.754 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:09.752+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.753104+0000) 2020-02-06T01:46:09.754 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:09.752+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.753666+0000) 2020-02-06T01:46:09.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:09.752+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.754057+0000) 2020-02-06T01:46:09.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:09.753+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.754532+0000) 2020-02-06T01:46:09.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:09.753+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.754961+0000) 2020-02-06T01:46:09.756 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:09.754+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.755134+0000) 2020-02-06T01:46:09.756 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:09.754+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.755522+0000) 2020-02-06T01:46:09.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:09.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.756075+0000) 2020-02-06T01:46:09.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:09.756+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.756504+0000) 2020-02-06T01:46:09.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:09.756+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.757039+0000) 2020-02-06T01:46:09.759 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:09.757+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.758009+0000) 2020-02-06T01:46:09.759 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:09.757+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.757743+0000) 2020-02-06T01:46:09.759 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:09.757+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.758305+0000) 2020-02-06T01:46:09.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:09.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.758343+0000) 2020-02-06T01:46:09.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:09.760+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:09.762061+0000) 2020-02-06T01:46:09.776 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:09.775+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213042 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:10.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:10.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:10.004992+0000) 2020-02-06T01:46:10.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:10.613+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:10.613936+0000) 2020-02-06T01:46:10.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:10.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:10.893020+0000) 2020-02-06T01:46:10.894 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:10.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:10.893017+0000) 2020-02-06T01:46:14.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:14.780+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.781536+0000) 2020-02-06T01:46:14.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:14.781+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.782126+0000) 2020-02-06T01:46:14.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:14.781+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.782649+0000) 2020-02-06T01:46:14.784 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:14.782+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.783152+0000) 2020-02-06T01:46:14.785 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:14.784+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.784779+0000) 2020-02-06T01:46:14.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:14.784+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.785394+0000) 2020-02-06T01:46:14.786 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:14.785+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.785499+0000) 2020-02-06T01:46:14.787 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:14.785+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.786359+0000) 2020-02-06T01:46:14.787 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:14.785+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.786331+0000) 2020-02-06T01:46:14.788 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:14.786+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.787419+0000) 2020-02-06T01:46:14.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:14.786+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.787748+0000) 2020-02-06T01:46:14.788 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:14.787+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.787962+0000) 2020-02-06T01:46:14.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:14.787+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.788335+0000) 2020-02-06T01:46:14.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:14.790+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:14.791761+0000) 2020-02-06T01:46:14.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:14.803+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:16.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:16.135+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:16.135509+0000) 2020-02-06T01:46:16.499 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:16.498+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:16.498546+0000) 2020-02-06T01:46:16.724 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:16.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:16.723839+0000) 2020-02-06T01:46:17.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:17.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.115217+0000) 2020-02-06T01:46:17.135 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:17.133+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.133780+0000) 2020-02-06T01:46:17.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:17.387+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.387522+0000) 2020-02-06T01:46:17.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:17.387+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.387725+0000) 2020-02-06T01:46:17.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:17.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.388062+0000) 2020-02-06T01:46:17.620 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:17.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:17.619683+0000) 2020-02-06T01:46:19.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:19.757+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.758530+0000) 2020-02-06T01:46:19.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:19.808+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.809630+0000) 2020-02-06T01:46:19.812 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:19.810+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.811242+0000) 2020-02-06T01:46:19.812 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:19.810+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.811065+0000) 2020-02-06T01:46:19.812 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:19.810+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.811585+0000) 2020-02-06T01:46:19.813 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:19.811+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.812448+0000) 2020-02-06T01:46:19.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:19.811+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.812909+0000) 2020-02-06T01:46:19.814 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:19.813+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.813471+0000) 2020-02-06T01:46:19.815 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:19.813+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.813764+0000) 2020-02-06T01:46:19.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:19.813+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.814647+0000) 2020-02-06T01:46:19.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:19.813+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.814971+0000) 2020-02-06T01:46:19.818 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:19.815+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.815959+0000) 2020-02-06T01:46:19.818 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:19.815+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.816052+0000) 2020-02-06T01:46:19.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:19.815+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.816374+0000) 2020-02-06T01:46:19.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:19.819+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:19.820505+0000) 2020-02-06T01:46:19.833 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:19.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:20.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:20.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:20.005203+0000) 2020-02-06T01:46:20.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:20.613+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:20.614173+0000) 2020-02-06T01:46:20.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:20.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:20.893333+0000) 2020-02-06T01:46:20.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:20.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:20.893331+0000) 2020-02-06T01:46:24.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:24.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.838455+0000) 2020-02-06T01:46:24.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:24.838+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.839259+0000) 2020-02-06T01:46:24.841 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:24.839+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.840288+0000) 2020-02-06T01:46:24.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:24.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.840630+0000) 2020-02-06T01:46:24.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:24.840+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.840906+0000) 2020-02-06T01:46:24.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:24.840+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.841768+0000) 2020-02-06T01:46:24.843 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:24.841+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.842234+0000) 2020-02-06T01:46:24.843 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:24.842+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.843182+0000) 2020-02-06T01:46:24.844 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:24.842+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.842951+0000) 2020-02-06T01:46:24.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:24.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.843957+0000) 2020-02-06T01:46:24.845 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:24.843+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.843955+0000) 2020-02-06T01:46:24.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:24.843+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.844331+0000) 2020-02-06T01:46:24.845 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:24.844+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.844473+0000) 2020-02-06T01:46:24.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:24.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:24.846870+0000) 2020-02-06T01:46:24.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:24.861+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213162 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:26.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:26.135+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:26.135769+0000) 2020-02-06T01:46:26.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:26.498+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:26.498778+0000) 2020-02-06T01:46:26.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:26.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:26.724065+0000) 2020-02-06T01:46:27.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:27.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.115451+0000) 2020-02-06T01:46:27.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:27.145+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.145473+0000) 2020-02-06T01:46:27.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:27.387+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.387756+0000) 2020-02-06T01:46:27.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:27.387+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.387949+0000) 2020-02-06T01:46:27.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:27.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.388292+0000) 2020-02-06T01:46:27.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:27.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:27.619899+0000) 2020-02-06T01:46:29.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:29.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.758790+0000) 2020-02-06T01:46:29.868 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:29.865+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.867027+0000) 2020-02-06T01:46:29.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:29.868+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.869173+0000) 2020-02-06T01:46:29.871 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:29.868+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.869823+0000) 2020-02-06T01:46:29.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:29.869+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.870263+0000) 2020-02-06T01:46:29.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:29.870+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.870837+0000) 2020-02-06T01:46:29.872 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:29.869+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.870888+0000) 2020-02-06T01:46:29.873 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:29.871+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.871580+0000) 2020-02-06T01:46:29.873 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:29.870+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.871853+0000) 2020-02-06T01:46:29.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:29.871+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.872380+0000) 2020-02-06T01:46:29.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:29.871+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.872663+0000) 2020-02-06T01:46:29.874 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:29.872+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.872812+0000) 2020-02-06T01:46:29.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:29.872+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.873012+0000) 2020-02-06T01:46:29.874 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:29.872+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.873343+0000) 2020-02-06T01:46:29.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:29.877+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:29.879061+0000) 2020-02-06T01:46:29.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:29.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213208 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:30.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:30.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:30.005369+0000) 2020-02-06T01:46:30.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:30.614+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:30.614389+0000) 2020-02-06T01:46:30.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:30.892+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:30.893600+0000) 2020-02-06T01:46:30.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:30.892+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:30.893598+0000) 2020-02-06T01:46:34.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:34.895+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.896996+0000) 2020-02-06T01:46:34.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:34.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.897762+0000) 2020-02-06T01:46:34.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:34.897+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.898305+0000) 2020-02-06T01:46:34.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:34.898+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.898947+0000) 2020-02-06T01:46:34.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:34.898+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.899922+0000) 2020-02-06T01:46:34.901 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:34.899+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.900336+0000) 2020-02-06T01:46:34.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:34.900+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.901257+0000) 2020-02-06T01:46:34.902 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:34.900+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.900768+0000) 2020-02-06T01:46:34.902 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:34.900+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.901200+0000) 2020-02-06T01:46:34.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:34.900+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.902014+0000) 2020-02-06T01:46:34.904 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:34.901+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.902502+0000) 2020-02-06T01:46:34.904 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:34.900+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.901362+0000) 2020-02-06T01:46:34.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:34.902+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.902454+0000) 2020-02-06T01:46:34.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:34.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:34.907624+0000) 2020-02-06T01:46:34.920 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:34.919+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213246 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:36.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:36.134+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:36.136010+0000) 2020-02-06T01:46:36.290 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:46:36.301 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:46:36.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:36.497+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:36.499046+0000) 2020-02-06T01:46:36.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:36.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:36.724341+0000) 2020-02-06T01:46:37.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:37.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.115703+0000) 2020-02-06T01:46:37.156 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:37.152+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.153638+0000) 2020-02-06T01:46:37.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:37.386+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.387953+0000) 2020-02-06T01:46:37.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:37.386+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.388170+0000) 2020-02-06T01:46:37.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:37.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.388528+0000) 2020-02-06T01:46:37.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:37.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:37.620132+0000) 2020-02-06T01:46:39.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:39.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.759054+0000) 2020-02-06T01:46:39.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:39.924+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.924596+0000) 2020-02-06T01:46:39.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:39.924+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.925889+0000) 2020-02-06T01:46:39.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:39.927+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.927319+0000) 2020-02-06T01:46:39.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:39.926+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.927535+0000) 2020-02-06T01:46:39.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:39.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.927803+0000) 2020-02-06T01:46:39.929 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:39.928+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.928125+0000) 2020-02-06T01:46:39.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:39.928+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.928433+0000) 2020-02-06T01:46:39.930 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:39.929+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.929339+0000) 2020-02-06T01:46:39.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:39.928+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.929759+0000) 2020-02-06T01:46:39.931 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:39.928+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.930240+0000) 2020-02-06T01:46:39.932 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:39.929+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.931093+0000) 2020-02-06T01:46:39.932 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:39.929+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.931246+0000) 2020-02-06T01:46:39.932 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:39.930+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.931637+0000) 2020-02-06T01:46:39.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:39.933+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:39.935003+0000) 2020-02-06T01:46:39.949 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:39.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213291 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:40.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:40.005+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:40.005590+0000) 2020-02-06T01:46:40.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:40.613+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:40.614634+0000) 2020-02-06T01:46:40.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:40.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:40.893874+0000) 2020-02-06T01:46:40.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:40.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:40.893871+0000) 2020-02-06T01:46:44.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:44.952+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.953397+0000) 2020-02-06T01:46:44.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:44.952+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.954027+0000) 2020-02-06T01:46:44.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:44.953+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.954950+0000) 2020-02-06T01:46:44.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:44.954+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.955290+0000) 2020-02-06T01:46:44.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:44.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.956243+0000) 2020-02-06T01:46:44.957 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:44.955+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.956877+0000) 2020-02-06T01:46:44.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:44.955+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.957294+0000) 2020-02-06T01:46:44.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:44.956+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.957347+0000) 2020-02-06T01:46:44.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:44.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.958672+0000) 2020-02-06T01:46:44.960 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:44.957+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.959238+0000) 2020-02-06T01:46:44.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:44.958+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.959589+0000) 2020-02-06T01:46:44.960 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:44.958+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.959616+0000) 2020-02-06T01:46:44.961 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:44.958+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.960183+0000) 2020-02-06T01:46:44.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:44.962+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:44.963496+0000) 2020-02-06T01:46:44.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:44.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213328 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:46.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:46.134+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:46.136251+0000) 2020-02-06T01:46:46.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:46.498+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:46.499296+0000) 2020-02-06T01:46:46.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:46.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:46.724586+0000) 2020-02-06T01:46:47.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:47.114+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.115942+0000) 2020-02-06T01:46:47.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:47.160+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.161753+0000) 2020-02-06T01:46:47.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:47.387+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.388188+0000) 2020-02-06T01:46:47.389 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:47.387+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.388503+0000) 2020-02-06T01:46:47.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:47.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.388760+0000) 2020-02-06T01:46:47.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:47.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:47.620412+0000) 2020-02-06T01:46:49.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:49.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.759326+0000) 2020-02-06T01:46:49.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:49.981+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.982327+0000) 2020-02-06T01:46:49.984 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:49.981+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.983062+0000) 2020-02-06T01:46:49.985 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:49.982+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.983571+0000) 2020-02-06T01:46:49.985 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:49.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.984448+0000) 2020-02-06T01:46:49.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:49.983+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.984712+0000) 2020-02-06T01:46:49.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:49.984+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.985414+0000) 2020-02-06T01:46:49.987 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:49.984+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.985612+0000) 2020-02-06T01:46:49.987 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:49.985+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.986213+0000) 2020-02-06T01:46:49.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:49.985+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.986351+0000) 2020-02-06T01:46:49.988 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:49.986+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.987500+0000) 2020-02-06T01:46:49.989 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:49.987+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.988735+0000) 2020-02-06T01:46:49.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:49.987+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.988925+0000) 2020-02-06T01:46:49.990 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:49.987+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.989316+0000) 2020-02-06T01:46:49.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:49.992+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:49.993608+0000) 2020-02-06T01:46:50.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:50.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:50.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:50.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:50.005746+0000) 2020-02-06T01:46:50.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:50.613+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:50.614881+0000) 2020-02-06T01:46:50.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:50.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:50.894135+0000) 2020-02-06T01:46:50.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:50.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:50.894149+0000) 2020-02-06T01:46:55.013 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:55.009+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.010877+0000) 2020-02-06T01:46:55.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:46:55.010+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.011504+0000) 2020-02-06T01:46:55.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:46:55.011+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.012231+0000) 2020-02-06T01:46:55.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:55.011+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.013148+0000) 2020-02-06T01:46:55.015 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:55.012+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.013885+0000) 2020-02-06T01:46:55.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:55.012+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.013873+0000) 2020-02-06T01:46:55.016 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:55.013+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.014541+0000) 2020-02-06T01:46:55.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:46:55.013+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.014632+0000) 2020-02-06T01:46:55.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:55.014+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.015572+0000) 2020-02-06T01:46:55.017 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:55.015+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.016056+0000) 2020-02-06T01:46:55.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:55.013+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.015176+0000) 2020-02-06T01:46:55.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:55.013+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.015286+0000) 2020-02-06T01:46:55.018 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:55.014+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.016028+0000) 2020-02-06T01:46:55.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:55.020+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:55.021170+0000) 2020-02-06T01:46:55.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:46:55.031+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213412 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:46:56.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:46:56.135+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:56.136516+0000) 2020-02-06T01:46:56.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:46:56.498+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:56.499548+0000) 2020-02-06T01:46:56.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:46:56.723+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:56.724818+0000) 2020-02-06T01:46:57.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:46:57.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.116175+0000) 2020-02-06T01:46:57.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:57.168+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.169757+0000) 2020-02-06T01:46:57.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:46:57.387+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.388469+0000) 2020-02-06T01:46:57.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:46:57.387+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.388781+0000) 2020-02-06T01:46:57.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:46:57.387+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.389010+0000) 2020-02-06T01:46:57.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:46:57.619+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:57.620624+0000) 2020-02-06T01:46:59.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:46:59.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:46:59.759518+0000) 2020-02-06T01:47:00.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:00.004+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.005956+0000) 2020-02-06T01:47:00.040 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:00.037+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.038932+0000) 2020-02-06T01:47:00.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:00.037+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.039146+0000) 2020-02-06T01:47:00.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:00.038+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.039764+0000) 2020-02-06T01:47:00.042 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:00.038+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.040260+0000) 2020-02-06T01:47:00.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:00.039+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.040511+0000) 2020-02-06T01:47:00.042 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:00.039+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.041048+0000) 2020-02-06T01:47:00.044 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:00.040+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.041565+0000) 2020-02-06T01:47:00.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:00.040+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.041995+0000) 2020-02-06T01:47:00.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:00.041+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.042409+0000) 2020-02-06T01:47:00.044 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:00.041+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.042620+0000) 2020-02-06T01:47:00.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:00.041+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.042569+0000) 2020-02-06T01:47:00.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:00.041+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.043349+0000) 2020-02-06T01:47:00.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:00.042+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.043973+0000) 2020-02-06T01:47:00.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:00.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.049132+0000) 2020-02-06T01:47:00.062 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:00.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213457 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:00.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:00.614+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.615125+0000) 2020-02-06T01:47:00.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:00.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.894335+0000) 2020-02-06T01:47:00.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:00.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:00.894334+0000) 2020-02-06T01:47:05.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:05.066+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.068113+0000) 2020-02-06T01:47:05.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:05.067+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.068737+0000) 2020-02-06T01:47:05.070 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:05.067+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.069394+0000) 2020-02-06T01:47:05.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:05.069+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.070341+0000) 2020-02-06T01:47:05.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:05.069+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.070718+0000) 2020-02-06T01:47:05.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:05.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.071208+0000) 2020-02-06T01:47:05.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:05.070+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.071469+0000) 2020-02-06T01:47:05.073 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:05.070+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.071788+0000) 2020-02-06T01:47:05.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:05.070+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.072008+0000) 2020-02-06T01:47:05.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:05.071+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.072949+0000) 2020-02-06T01:47:05.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:05.072+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.073478+0000) 2020-02-06T01:47:05.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:05.071+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.072719+0000) 2020-02-06T01:47:05.075 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:05.072+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.074180+0000) 2020-02-06T01:47:05.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:05.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:05.076771+0000) 2020-02-06T01:47:05.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:05.087+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:06.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:06.136+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:06.136797+0000) 2020-02-06T01:47:06.479 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:47:06.487 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:47:06.500 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:06.498+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:06.499771+0000) 2020-02-06T01:47:06.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:06.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:06.725062+0000) 2020-02-06T01:47:07.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:07.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.116414+0000) 2020-02-06T01:47:07.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:07.177+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.177732+0000) 2020-02-06T01:47:07.390 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:07.388+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.388728+0000) 2020-02-06T01:47:07.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:07.388+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.388985+0000) 2020-02-06T01:47:07.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:07.388+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.389238+0000) 2020-02-06T01:47:07.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:07.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:07.620863+0000) 2020-02-06T01:47:09.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:09.758+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:09.759765+0000) 2020-02-06T01:47:10.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:10.005+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.006192+0000) 2020-02-06T01:47:10.096 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:10.094+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.095501+0000) 2020-02-06T01:47:10.097 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:10.094+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.095868+0000) 2020-02-06T01:47:10.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:10.096+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.097313+0000) 2020-02-06T01:47:10.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:10.096+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.097923+0000) 2020-02-06T01:47:10.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:10.096+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.097740+0000) 2020-02-06T01:47:10.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:10.096+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.098243+0000) 2020-02-06T01:47:10.100 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:10.097+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.099103+0000) 2020-02-06T01:47:10.100 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:10.097+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.099093+0000) 2020-02-06T01:47:10.101 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:10.098+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.100080+0000) 2020-02-06T01:47:10.101 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:10.098+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.100273+0000) 2020-02-06T01:47:10.101 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:10.099+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.100465+0000) 2020-02-06T01:47:10.102 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:10.099+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.101041+0000) 2020-02-06T01:47:10.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:10.100+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.101416+0000) 2020-02-06T01:47:10.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:10.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.105195+0000) 2020-02-06T01:47:10.117 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:10.115+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213540 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:10.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:10.614+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.615357+0000) 2020-02-06T01:47:10.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:10.893+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.894547+0000) 2020-02-06T01:47:10.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:10.893+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:10.894550+0000) 2020-02-06T01:47:15.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:15.120+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.122023+0000) 2020-02-06T01:47:15.123 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:15.121+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.122846+0000) 2020-02-06T01:47:15.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:15.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.123707+0000) 2020-02-06T01:47:15.125 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:15.122+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.123983+0000) 2020-02-06T01:47:15.125 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:15.123+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.124753+0000) 2020-02-06T01:47:15.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:15.123+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.124576+0000) 2020-02-06T01:47:15.126 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:15.124+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.125656+0000) 2020-02-06T01:47:15.126 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:15.123+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.125362+0000) 2020-02-06T01:47:15.127 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:15.124+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.125417+0000) 2020-02-06T01:47:15.127 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:15.125+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.126605+0000) 2020-02-06T01:47:15.127 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:15.125+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.126507+0000) 2020-02-06T01:47:15.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:15.125+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.126947+0000) 2020-02-06T01:47:15.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:15.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.128234+0000) 2020-02-06T01:47:15.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:15.131+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:15.132812+0000) 2020-02-06T01:47:15.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:15.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213579 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:16.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:16.136+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:16.137048+0000) 2020-02-06T01:47:16.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:16.499+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:16.500075+0000) 2020-02-06T01:47:16.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:16.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:16.725366+0000) 2020-02-06T01:47:17.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:17.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.116686+0000) 2020-02-06T01:47:17.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:17.185+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.185996+0000) 2020-02-06T01:47:17.390 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:17.388+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.388992+0000) 2020-02-06T01:47:17.390 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:17.388+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.389213+0000) 2020-02-06T01:47:17.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:17.388+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.389525+0000) 2020-02-06T01:47:17.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:17.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:17.621163+0000) 2020-02-06T01:47:19.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:19.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:19.760018+0000) 2020-02-06T01:47:20.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:20.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.006451+0000) 2020-02-06T01:47:20.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:20.148+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.149775+0000) 2020-02-06T01:47:20.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:20.149+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.150655+0000) 2020-02-06T01:47:20.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:20.150+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.151249+0000) 2020-02-06T01:47:20.152 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:20.150+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.151510+0000) 2020-02-06T01:47:20.153 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:20.151+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.152341+0000) 2020-02-06T01:47:20.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:20.152+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.153412+0000) 2020-02-06T01:47:20.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:20.153+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.154762+0000) 2020-02-06T01:47:20.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:20.153+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.154311+0000) 2020-02-06T01:47:20.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:20.153+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.154493+0000) 2020-02-06T01:47:20.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:20.154+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.155351+0000) 2020-02-06T01:47:20.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:20.154+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.155860+0000) 2020-02-06T01:47:20.157 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:20.154+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.155963+0000) 2020-02-06T01:47:20.158 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:20.155+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.156712+0000) 2020-02-06T01:47:20.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:20.161+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.162359+0000) 2020-02-06T01:47:20.176 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:20.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:20.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:20.615+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.615632+0000) 2020-02-06T01:47:20.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:20.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.894806+0000) 2020-02-06T01:47:20.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:20.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:20.894806+0000) 2020-02-06T01:47:25.180 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:25.178+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.179820+0000) 2020-02-06T01:47:25.182 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:25.180+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.181228+0000) 2020-02-06T01:47:25.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:25.180+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.181912+0000) 2020-02-06T01:47:25.184 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:25.181+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.182310+0000) 2020-02-06T01:47:25.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:25.181+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.182219+0000) 2020-02-06T01:47:25.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:25.181+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.183021+0000) 2020-02-06T01:47:25.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:25.182+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.182738+0000) 2020-02-06T01:47:25.185 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:25.182+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.184058+0000) 2020-02-06T01:47:25.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:25.184+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.184482+0000) 2020-02-06T01:47:25.186 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:25.184+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.185106+0000) 2020-02-06T01:47:25.187 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:25.184+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.185181+0000) 2020-02-06T01:47:25.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:25.185+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.185885+0000) 2020-02-06T01:47:25.187 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:25.186+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.186713+0000) 2020-02-06T01:47:25.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:25.188+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:25.189866+0000) 2020-02-06T01:47:25.203 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:25.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:26.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:26.136+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:26.137225+0000) 2020-02-06T01:47:26.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:26.499+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:26.500331+0000) 2020-02-06T01:47:26.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:26.724+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:26.725661+0000) 2020-02-06T01:47:27.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:27.115+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.116932+0000) 2020-02-06T01:47:27.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:27.197+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.197717+0000) 2020-02-06T01:47:27.390 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:27.388+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.389292+0000) 2020-02-06T01:47:27.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:27.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.389483+0000) 2020-02-06T01:47:27.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:27.389+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.389854+0000) 2020-02-06T01:47:27.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:27.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:27.621346+0000) 2020-02-06T01:47:29.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:29.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:29.760331+0000) 2020-02-06T01:47:30.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:30.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.006685+0000) 2020-02-06T01:47:30.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:30.207+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.208189+0000) 2020-02-06T01:47:30.210 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:30.208+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.209178+0000) 2020-02-06T01:47:30.211 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:30.209+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.210286+0000) 2020-02-06T01:47:30.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:30.209+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.210583+0000) 2020-02-06T01:47:30.212 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:30.210+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.211143+0000) 2020-02-06T01:47:30.212 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:30.210+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.211568+0000) 2020-02-06T01:47:30.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:30.210+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.211767+0000) 2020-02-06T01:47:30.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:30.211+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.212408+0000) 2020-02-06T01:47:30.213 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:30.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.212523+0000) 2020-02-06T01:47:30.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:30.211+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.212898+0000) 2020-02-06T01:47:30.214 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:30.212+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.212980+0000) 2020-02-06T01:47:30.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:30.212+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.213383+0000) 2020-02-06T01:47:30.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:30.212+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.213816+0000) 2020-02-06T01:47:30.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:30.216+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.217825+0000) 2020-02-06T01:47:30.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:30.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:30.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:30.615+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.615863+0000) 2020-02-06T01:47:30.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:30.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.895052+0000) 2020-02-06T01:47:30.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:30.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:30.895052+0000) 2020-02-06T01:47:35.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:35.236+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.236629+0000) 2020-02-06T01:47:35.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:35.237+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.237984+0000) 2020-02-06T01:47:35.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:35.239+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.239521+0000) 2020-02-06T01:47:35.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:35.240+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.240679+0000) 2020-02-06T01:47:35.242 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:35.240+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.240924+0000) 2020-02-06T01:47:35.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:35.241+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.241416+0000) 2020-02-06T01:47:35.243 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:35.241+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.241811+0000) 2020-02-06T01:47:35.243 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:35.242+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.242159+0000) 2020-02-06T01:47:35.244 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:35.242+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.242659+0000) 2020-02-06T01:47:35.244 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:35.242+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.243296+0000) 2020-02-06T01:47:35.245 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:35.243+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.243515+0000) 2020-02-06T01:47:35.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:35.243+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.243926+0000) 2020-02-06T01:47:35.245 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:35.244+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.244270+0000) 2020-02-06T01:47:35.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:35.248+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:35.248424+0000) 2020-02-06T01:47:35.262 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:35.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:36.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:36.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:36.137490+0000) 2020-02-06T01:47:36.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:36.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:36.500528+0000) 2020-02-06T01:47:36.626 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:47:36.638 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:47:36.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:36.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:36.725856+0000) 2020-02-06T01:47:37.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:37.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.117164+0000) 2020-02-06T01:47:37.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:37.205+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.205964+0000) 2020-02-06T01:47:37.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:37.389+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.389547+0000) 2020-02-06T01:47:37.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:37.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.389747+0000) 2020-02-06T01:47:37.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:37.389+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.390096+0000) 2020-02-06T01:47:37.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:37.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:37.621573+0000) 2020-02-06T01:47:39.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:39.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:39.760564+0000) 2020-02-06T01:47:40.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:40.005+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.006920+0000) 2020-02-06T01:47:40.269 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:40.266+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.268021+0000) 2020-02-06T01:47:40.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:40.267+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.268566+0000) 2020-02-06T01:47:40.270 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:40.267+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.268856+0000) 2020-02-06T01:47:40.270 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:40.268+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.269370+0000) 2020-02-06T01:47:40.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:40.268+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.269985+0000) 2020-02-06T01:47:40.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:40.270+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.270483+0000) 2020-02-06T01:47:40.273 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:40.271+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.271551+0000) 2020-02-06T01:47:40.273 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:40.271+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.271934+0000) 2020-02-06T01:47:40.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:40.271+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.272371+0000) 2020-02-06T01:47:40.274 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:40.272+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.273266+0000) 2020-02-06T01:47:40.274 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:40.273+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.273726+0000) 2020-02-06T01:47:40.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:40.273+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.274022+0000) 2020-02-06T01:47:40.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:40.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.276870+0000) 2020-02-06T01:47:40.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:40.276+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.277199+0000) 2020-02-06T01:47:40.290 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:40.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:40.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:40.615+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.616138+0000) 2020-02-06T01:47:40.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:40.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.895352+0000) 2020-02-06T01:47:40.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:40.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:40.895352+0000) 2020-02-06T01:47:45.296 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:45.293+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.294900+0000) 2020-02-06T01:47:45.297 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:45.295+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.296275+0000) 2020-02-06T01:47:45.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:45.295+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.295958+0000) 2020-02-06T01:47:45.298 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:45.295+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.296672+0000) 2020-02-06T01:47:45.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:45.297+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.298929+0000) 2020-02-06T01:47:45.300 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:45.298+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.299188+0000) 2020-02-06T01:47:45.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:45.299+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.300138+0000) 2020-02-06T01:47:45.301 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:45.299+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.300458+0000) 2020-02-06T01:47:45.301 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:45.300+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.300903+0000) 2020-02-06T01:47:45.302 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:45.301+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.301652+0000) 2020-02-06T01:47:45.302 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:45.301+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.301778+0000) 2020-02-06T01:47:45.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:45.301+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.302257+0000) 2020-02-06T01:47:45.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:45.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.307361+0000) 2020-02-06T01:47:45.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:45.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:45.307658+0000) 2020-02-06T01:47:45.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:45.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213828 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:46.138 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:46.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:46.137735+0000) 2020-02-06T01:47:46.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:46.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:46.500719+0000) 2020-02-06T01:47:46.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:46.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:46.726140+0000) 2020-02-06T01:47:47.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:47.116+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.117441+0000) 2020-02-06T01:47:47.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:47.215+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.216285+0000) 2020-02-06T01:47:47.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:47.389+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.389801+0000) 2020-02-06T01:47:47.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:47.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.390008+0000) 2020-02-06T01:47:47.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:47.389+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.390301+0000) 2020-02-06T01:47:47.622 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:47.620+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:47.621804+0000) 2020-02-06T01:47:49.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:49.759+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:49.760818+0000) 2020-02-06T01:47:50.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:50.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.007119+0000) 2020-02-06T01:47:50.323 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:50.320+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.321797+0000) 2020-02-06T01:47:50.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:50.322+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.323251+0000) 2020-02-06T01:47:50.325 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:50.323+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.324122+0000) 2020-02-06T01:47:50.325 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:50.323+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.324516+0000) 2020-02-06T01:47:50.326 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:50.324+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.325083+0000) 2020-02-06T01:47:50.326 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:50.324+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.325143+0000) 2020-02-06T01:47:50.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:50.324+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.325676+0000) 2020-02-06T01:47:50.327 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:50.325+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.326101+0000) 2020-02-06T01:47:50.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:50.326+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.327006+0000) 2020-02-06T01:47:50.328 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:50.326+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.327345+0000) 2020-02-06T01:47:50.329 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:50.327+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.327642+0000) 2020-02-06T01:47:50.329 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:50.327+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.328078+0000) 2020-02-06T01:47:50.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:50.328+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.328686+0000) 2020-02-06T01:47:50.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:50.330+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.332036+0000) 2020-02-06T01:47:50.346 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:50.344+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:50.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:50.616+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.616374+0000) 2020-02-06T01:47:50.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:50.894+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.895571+0000) 2020-02-06T01:47:50.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:50.894+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:50.895572+0000) 2020-02-06T01:47:55.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:55.349+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.350642+0000) 2020-02-06T01:47:55.353 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:55.352+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.352623+0000) 2020-02-06T01:47:55.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:55.352+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.353067+0000) 2020-02-06T01:47:55.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:55.352+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.353392+0000) 2020-02-06T01:47:55.355 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:47:55.352+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.353828+0000) 2020-02-06T01:47:55.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:47:55.352+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.353989+0000) 2020-02-06T01:47:55.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:55.353+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.354352+0000) 2020-02-06T01:47:55.356 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:55.354+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.354750+0000) 2020-02-06T01:47:55.356 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:55.354+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.355004+0000) 2020-02-06T01:47:55.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:55.354+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.355535+0000) 2020-02-06T01:47:55.357 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:55.354+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.355854+0000) 2020-02-06T01:47:55.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:55.355+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.356252+0000) 2020-02-06T01:47:55.358 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:47:55.356+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.357227+0000) 2020-02-06T01:47:55.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:55.360+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:55.361887+0000) 2020-02-06T01:47:55.376 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:47:55.374+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:47:56.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:47:56.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:56.137938+0000) 2020-02-06T01:47:56.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:47:56.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:56.500993+0000) 2020-02-06T01:47:56.727 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:47:56.725+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:56.726373+0000) 2020-02-06T01:47:57.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:47:57.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.117699+0000) 2020-02-06T01:47:57.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:57.223+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.224381+0000) 2020-02-06T01:47:57.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:47:57.389+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.390068+0000) 2020-02-06T01:47:57.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:47:57.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.390290+0000) 2020-02-06T01:47:57.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:47:57.390+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.390521+0000) 2020-02-06T01:47:57.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:47:57.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:57.622055+0000) 2020-02-06T01:47:59.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:47:59.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:47:59.761103+0000) 2020-02-06T01:48:00.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:00.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.007361+0000) 2020-02-06T01:48:00.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:00.381+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.382239+0000) 2020-02-06T01:48:00.384 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:00.382+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.383113+0000) 2020-02-06T01:48:00.384 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:00.382+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.383958+0000) 2020-02-06T01:48:00.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:00.384+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.384539+0000) 2020-02-06T01:48:00.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:00.383+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.384764+0000) 2020-02-06T01:48:00.386 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:00.384+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.385172+0000) 2020-02-06T01:48:00.386 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:00.384+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.385345+0000) 2020-02-06T01:48:00.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:00.385+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.386334+0000) 2020-02-06T01:48:00.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:00.385+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.386379+0000) 2020-02-06T01:48:00.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:00.385+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.386434+0000) 2020-02-06T01:48:00.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:00.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.387003+0000) 2020-02-06T01:48:00.388 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:00.386+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.386888+0000) 2020-02-06T01:48:00.389 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:00.386+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.386871+0000) 2020-02-06T01:48:00.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:00.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.389123+0000) 2020-02-06T01:48:00.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:00.403+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:00.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:00.616+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.616653+0000) 2020-02-06T01:48:00.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:00.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.895812+0000) 2020-02-06T01:48:00.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:00.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:00.895812+0000) 2020-02-06T01:48:05.411 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:05.409+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.410596+0000) 2020-02-06T01:48:05.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:05.411+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.412605+0000) 2020-02-06T01:48:05.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:05.412+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.413270+0000) 2020-02-06T01:48:05.415 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:05.412+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.413963+0000) 2020-02-06T01:48:05.415 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:05.413+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.414633+0000) 2020-02-06T01:48:05.416 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:05.413+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.414552+0000) 2020-02-06T01:48:05.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:05.413+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.414934+0000) 2020-02-06T01:48:05.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:05.413+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.415281+0000) 2020-02-06T01:48:05.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:05.414+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.415817+0000) 2020-02-06T01:48:05.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:05.414+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.416040+0000) 2020-02-06T01:48:05.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:05.414+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.415709+0000) 2020-02-06T01:48:05.418 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:05.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.415920+0000) 2020-02-06T01:48:05.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:05.414+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.416396+0000) 2020-02-06T01:48:05.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:05.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:05.421854+0000) 2020-02-06T01:48:05.435 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:05.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 213992 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:06.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:06.136+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:06.138157+0000) 2020-02-06T01:48:06.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:06.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:06.501296+0000) 2020-02-06T01:48:06.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:06.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:06.726604+0000) 2020-02-06T01:48:06.785 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:48:06.791 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:48:07.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:07.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.117950+0000) 2020-02-06T01:48:07.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:07.235+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.236224+0000) 2020-02-06T01:48:07.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:07.389+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.390332+0000) 2020-02-06T01:48:07.391 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:07.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.390524+0000) 2020-02-06T01:48:07.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:07.389+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.390816+0000) 2020-02-06T01:48:07.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:07.621+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:07.622398+0000) 2020-02-06T01:48:09.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:09.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:09.761341+0000) 2020-02-06T01:48:10.008 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:10.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.007600+0000) 2020-02-06T01:48:10.440 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:10.438+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.439407+0000) 2020-02-06T01:48:10.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:10.438+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.440333+0000) 2020-02-06T01:48:10.442 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:10.439+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.440712+0000) 2020-02-06T01:48:10.442 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:10.440+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.441162+0000) 2020-02-06T01:48:10.443 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:10.441+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.442822+0000) 2020-02-06T01:48:10.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:10.442+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.443151+0000) 2020-02-06T01:48:10.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:10.442+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.443645+0000) 2020-02-06T01:48:10.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:10.442+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.443762+0000) 2020-02-06T01:48:10.446 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:10.443+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.444920+0000) 2020-02-06T01:48:10.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:10.444+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.445613+0000) 2020-02-06T01:48:10.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:10.443+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.444854+0000) 2020-02-06T01:48:10.447 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:10.445+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.446503+0000) 2020-02-06T01:48:10.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:10.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.450687+0000) 2020-02-06T01:48:10.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:10.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.450981+0000) 2020-02-06T01:48:10.463 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:10.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:10.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:10.615+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.616872+0000) 2020-02-06T01:48:10.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:10.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.896057+0000) 2020-02-06T01:48:10.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:10.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:10.896058+0000) 2020-02-06T01:48:15.468 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:15.466+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.467305+0000) 2020-02-06T01:48:15.469 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:15.466+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.467710+0000) 2020-02-06T01:48:15.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:15.467+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.468690+0000) 2020-02-06T01:48:15.471 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:15.468+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.469706+0000) 2020-02-06T01:48:15.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:15.468+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.469886+0000) 2020-02-06T01:48:15.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:15.468+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.470380+0000) 2020-02-06T01:48:15.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:15.469+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.470900+0000) 2020-02-06T01:48:15.472 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:15.470+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.471514+0000) 2020-02-06T01:48:15.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:15.471+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.472196+0000) 2020-02-06T01:48:15.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:15.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.472250+0000) 2020-02-06T01:48:15.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:15.471+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.472688+0000) 2020-02-06T01:48:15.474 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:15.472+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.473473+0000) 2020-02-06T01:48:15.474 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:15.472+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.474055+0000) 2020-02-06T01:48:15.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:15.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:15.477000+0000) 2020-02-06T01:48:15.495 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:15.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:16.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:16.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:16.138400+0000) 2020-02-06T01:48:16.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:16.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:16.501554+0000) 2020-02-06T01:48:16.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:16.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:16.726836+0000) 2020-02-06T01:48:17.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:17.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.118268+0000) 2020-02-06T01:48:17.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:17.243+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.244068+0000) 2020-02-06T01:48:17.391 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:17.389+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.390589+0000) 2020-02-06T01:48:17.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:17.389+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.390794+0000) 2020-02-06T01:48:17.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:17.389+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.391052+0000) 2020-02-06T01:48:17.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:17.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:17.622641+0000) 2020-02-06T01:48:19.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:19.760+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:19.761597+0000) 2020-02-06T01:48:20.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:20.006+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.007841+0000) 2020-02-06T01:48:20.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:20.497+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.498144+0000) 2020-02-06T01:48:20.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:20.497+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.498973+0000) 2020-02-06T01:48:20.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:20.498+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.499611+0000) 2020-02-06T01:48:20.501 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:20.499+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.500545+0000) 2020-02-06T01:48:20.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:20.499+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.500935+0000) 2020-02-06T01:48:20.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:20.500+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.501527+0000) 2020-02-06T01:48:20.502 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:20.500+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.502018+0000) 2020-02-06T01:48:20.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:20.501+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.502367+0000) 2020-02-06T01:48:20.503 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:20.501+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.502412+0000) 2020-02-06T01:48:20.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:20.501+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.502828+0000) 2020-02-06T01:48:20.504 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:20.501+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.503028+0000) 2020-02-06T01:48:20.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:20.502+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.503731+0000) 2020-02-06T01:48:20.505 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:20.503+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.504890+0000) 2020-02-06T01:48:20.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:20.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.508692+0000) 2020-02-06T01:48:20.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:20.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214119 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:20.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:20.616+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.617089+0000) 2020-02-06T01:48:20.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:20.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.896370+0000) 2020-02-06T01:48:20.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:20.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:20.896369+0000) 2020-02-06T01:48:25.528 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:25.526+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.527331+0000) 2020-02-06T01:48:25.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:25.527+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.528586+0000) 2020-02-06T01:48:25.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:25.528+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.529099+0000) 2020-02-06T01:48:25.530 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:25.528+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.529400+0000) 2020-02-06T01:48:25.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:25.528+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.529908+0000) 2020-02-06T01:48:25.532 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:25.529+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.530926+0000) 2020-02-06T01:48:25.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:25.529+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.531239+0000) 2020-02-06T01:48:25.533 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:25.530+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.532095+0000) 2020-02-06T01:48:25.533 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:25.531+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.532614+0000) 2020-02-06T01:48:25.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:25.530+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.531946+0000) 2020-02-06T01:48:25.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:25.531+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.532599+0000) 2020-02-06T01:48:25.534 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:25.531+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.532760+0000) 2020-02-06T01:48:25.534 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:25.532+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.533380+0000) 2020-02-06T01:48:25.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:25.535+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:25.536660+0000) 2020-02-06T01:48:25.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:25.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:26.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:26.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:26.138679+0000) 2020-02-06T01:48:26.502 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:26.500+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:26.501751+0000) 2020-02-06T01:48:26.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:26.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:26.727062+0000) 2020-02-06T01:48:27.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:27.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.118507+0000) 2020-02-06T01:48:27.253 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:27.252+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.252113+0000) 2020-02-06T01:48:27.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:27.390+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.390856+0000) 2020-02-06T01:48:27.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:27.390+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.391029+0000) 2020-02-06T01:48:27.392 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:27.390+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.391266+0000) 2020-02-06T01:48:27.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:27.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:27.622874+0000) 2020-02-06T01:48:29.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:29.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:29.761831+0000) 2020-02-06T01:48:30.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:30.007+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.008058+0000) 2020-02-06T01:48:30.556 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:30.555+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.555282+0000) 2020-02-06T01:48:30.557 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:30.556+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.556311+0000) 2020-02-06T01:48:30.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:30.557+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.557204+0000) 2020-02-06T01:48:30.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:30.559+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.559513+0000) 2020-02-06T01:48:30.571 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:30.565+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.565989+0000) 2020-02-06T01:48:30.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:30.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.559586+0000) 2020-02-06T01:48:30.572 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:30.560+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.560073+0000) 2020-02-06T01:48:30.572 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:30.560+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.560366+0000) 2020-02-06T01:48:30.572 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:30.556+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.558196+0000) 2020-02-06T01:48:30.572 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:30.557+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.558581+0000) 2020-02-06T01:48:30.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:30.557+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.559156+0000) 2020-02-06T01:48:30.573 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:30.558+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.560064+0000) 2020-02-06T01:48:30.573 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:30.559+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.561072+0000) 2020-02-06T01:48:30.573 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:30.559+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.561285+0000) 2020-02-06T01:48:30.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:30.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214203 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:30.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:30.616+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.617303+0000) 2020-02-06T01:48:30.897 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:30.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.896622+0000) 2020-02-06T01:48:30.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:30.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:30.896622+0000) 2020-02-06T01:48:35.582 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:35.578+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.580356+0000) 2020-02-06T01:48:35.582 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:35.580+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.581276+0000) 2020-02-06T01:48:35.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:35.582+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.583386+0000) 2020-02-06T01:48:35.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:35.582+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.583747+0000) 2020-02-06T01:48:35.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:35.583+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.584287+0000) 2020-02-06T01:48:35.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:35.583+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.584752+0000) 2020-02-06T01:48:35.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:35.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.585031+0000) 2020-02-06T01:48:35.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:35.584+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.585537+0000) 2020-02-06T01:48:35.586 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:35.584+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.585690+0000) 2020-02-06T01:48:35.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:35.584+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.586082+0000) 2020-02-06T01:48:35.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:35.585+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.586650+0000) 2020-02-06T01:48:35.588 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:35.586+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.587139+0000) 2020-02-06T01:48:35.588 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:35.586+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.587850+0000) 2020-02-06T01:48:35.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:35.588+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:35.589621+0000) 2020-02-06T01:48:35.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:35.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214240 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:36.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:36.137+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:36.138861+0000) 2020-02-06T01:48:36.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:36.501+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:36.501995+0000) 2020-02-06T01:48:36.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:36.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:36.727332+0000) 2020-02-06T01:48:36.936 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:48:36.942 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:48:37.119 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:37.117+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.118774+0000) 2020-02-06T01:48:37.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:37.259+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.260933+0000) 2020-02-06T01:48:37.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:37.390+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.391095+0000) 2020-02-06T01:48:37.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:37.390+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.391263+0000) 2020-02-06T01:48:37.393 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:37.390+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.391499+0000) 2020-02-06T01:48:37.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:37.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:37.623130+0000) 2020-02-06T01:48:39.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:39.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:39.762069+0000) 2020-02-06T01:48:40.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:40.007+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.008318+0000) 2020-02-06T01:48:40.611 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:40.608+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.609882+0000) 2020-02-06T01:48:40.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:40.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.610518+0000) 2020-02-06T01:48:40.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:40.610+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.611471+0000) 2020-02-06T01:48:40.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:40.611+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.612992+0000) 2020-02-06T01:48:40.614 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:40.612+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.613684+0000) 2020-02-06T01:48:40.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:40.612+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.614065+0000) 2020-02-06T01:48:40.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:40.612+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.613880+0000) 2020-02-06T01:48:40.615 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:40.613+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.614452+0000) 2020-02-06T01:48:40.616 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:40.614+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.615353+0000) 2020-02-06T01:48:40.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:40.614+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.615863+0000) 2020-02-06T01:48:40.617 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:40.614+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.616224+0000) 2020-02-06T01:48:40.617 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:40.614+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.616300+0000) 2020-02-06T01:48:40.617 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:40.615+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.616416+0000) 2020-02-06T01:48:40.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:40.616+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.617456+0000) 2020-02-06T01:48:40.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:40.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.617664+0000) 2020-02-06T01:48:40.633 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:40.630+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:40.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:40.895+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.896872+0000) 2020-02-06T01:48:40.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:40.895+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:40.896872+0000) 2020-02-06T01:48:45.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:45.637+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.638639+0000) 2020-02-06T01:48:45.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:45.638+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.639144+0000) 2020-02-06T01:48:45.640 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:45.638+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.639779+0000) 2020-02-06T01:48:45.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:45.639+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.641058+0000) 2020-02-06T01:48:45.642 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:45.640+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.641708+0000) 2020-02-06T01:48:45.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:45.640+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.642014+0000) 2020-02-06T01:48:45.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:45.641+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.642607+0000) 2020-02-06T01:48:45.643 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:45.641+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.642882+0000) 2020-02-06T01:48:45.644 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:45.642+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.643171+0000) 2020-02-06T01:48:45.644 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:45.642+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.643579+0000) 2020-02-06T01:48:45.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:45.642+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.644310+0000) 2020-02-06T01:48:45.645 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:45.643+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.644603+0000) 2020-02-06T01:48:45.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:45.642+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.644383+0000) 2020-02-06T01:48:45.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:45.648+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:45.649071+0000) 2020-02-06T01:48:45.663 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:45.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214321 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:46.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:46.138+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:46.139102+0000) 2020-02-06T01:48:46.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:46.501+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:46.502206+0000) 2020-02-06T01:48:46.728 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:46.726+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:46.727574+0000) 2020-02-06T01:48:47.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:47.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.119014+0000) 2020-02-06T01:48:47.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:47.268+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.269116+0000) 2020-02-06T01:48:47.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:47.390+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.391360+0000) 2020-02-06T01:48:47.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:47.390+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.391538+0000) 2020-02-06T01:48:47.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:47.390+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.391729+0000) 2020-02-06T01:48:47.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:47.622+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:47.623371+0000) 2020-02-06T01:48:49.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:49.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:49.762355+0000) 2020-02-06T01:48:50.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:50.007+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.008541+0000) 2020-02-06T01:48:50.618 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:50.617+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.617670+0000) 2020-02-06T01:48:50.671 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:50.667+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.668891+0000) 2020-02-06T01:48:50.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:50.668+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.669957+0000) 2020-02-06T01:48:50.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:50.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.670963+0000) 2020-02-06T01:48:50.672 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:50.670+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.671186+0000) 2020-02-06T01:48:50.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:50.670+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.671483+0000) 2020-02-06T01:48:50.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:50.670+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.671719+0000) 2020-02-06T01:48:50.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:50.671+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.672163+0000) 2020-02-06T01:48:50.674 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:50.671+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.672753+0000) 2020-02-06T01:48:50.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:50.671+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.672521+0000) 2020-02-06T01:48:50.674 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:50.671+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.672992+0000) 2020-02-06T01:48:50.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:50.671+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.673175+0000) 2020-02-06T01:48:50.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:50.672+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.674083+0000) 2020-02-06T01:48:50.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:50.672+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.674154+0000) 2020-02-06T01:48:50.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:50.674+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.675938+0000) 2020-02-06T01:48:50.691 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:50.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:50.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:50.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.897070+0000) 2020-02-06T01:48:50.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:50.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:50.897124+0000) 2020-02-06T01:48:55.696 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:55.694+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.695093+0000) 2020-02-06T01:48:55.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:55.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.696562+0000) 2020-02-06T01:48:55.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:48:55.696+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.697304+0000) 2020-02-06T01:48:55.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:48:55.697+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.697945+0000) 2020-02-06T01:48:55.699 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:55.697+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.698005+0000) 2020-02-06T01:48:55.699 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:55.697+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.698785+0000) 2020-02-06T01:48:55.700 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:55.698+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.699243+0000) 2020-02-06T01:48:55.700 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:55.698+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.699373+0000) 2020-02-06T01:48:55.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:55.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.699555+0000) 2020-02-06T01:48:55.701 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:48:55.698+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.699756+0000) 2020-02-06T01:48:55.701 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:55.699+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.700013+0000) 2020-02-06T01:48:55.701 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:55.699+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.700225+0000) 2020-02-06T01:48:55.702 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:55.700+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.701006+0000) 2020-02-06T01:48:55.709 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:55.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:55.708503+0000) 2020-02-06T01:48:55.719 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:48:55.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214402 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:48:56.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:48:56.138+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:56.139336+0000) 2020-02-06T01:48:56.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:48:56.502+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:56.502467+0000) 2020-02-06T01:48:56.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:48:56.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:56.727804+0000) 2020-02-06T01:48:57.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:48:57.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.119259+0000) 2020-02-06T01:48:57.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:57.275+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.276936+0000) 2020-02-06T01:48:57.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:48:57.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.391574+0000) 2020-02-06T01:48:57.393 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:48:57.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.391826+0000) 2020-02-06T01:48:57.393 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:48:57.391+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.392033+0000) 2020-02-06T01:48:57.624 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:48:57.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:57.623604+0000) 2020-02-06T01:48:59.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:48:59.761+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:48:59.762589+0000) 2020-02-06T01:49:00.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:00.008+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.008768+0000) 2020-02-06T01:49:00.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:00.617+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.617917+0000) 2020-02-06T01:49:00.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:00.722+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.723642+0000) 2020-02-06T01:49:00.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:00.723+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.724714+0000) 2020-02-06T01:49:00.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:00.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.725835+0000) 2020-02-06T01:49:00.727 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:00.726+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.726647+0000) 2020-02-06T01:49:00.728 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:00.727+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.727413+0000) 2020-02-06T01:49:00.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:00.726+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.727843+0000) 2020-02-06T01:49:00.729 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:00.727+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.728250+0000) 2020-02-06T01:49:00.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:00.727+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.728430+0000) 2020-02-06T01:49:00.730 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:00.728+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.728952+0000) 2020-02-06T01:49:00.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:00.728+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.729221+0000) 2020-02-06T01:49:00.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:00.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.729760+0000) 2020-02-06T01:49:00.731 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:00.729+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.730213+0000) 2020-02-06T01:49:00.731 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:00.729+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.730325+0000) 2020-02-06T01:49:00.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:00.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.735671+0000) 2020-02-06T01:49:00.748 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:00.747+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:00.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:00.896+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.897377+0000) 2020-02-06T01:49:00.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:00.896+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:00.897369+0000) 2020-02-06T01:49:05.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:05.752+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.753538+0000) 2020-02-06T01:49:05.755 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:05.753+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.754291+0000) 2020-02-06T01:49:05.755 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:05.754+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.754928+0000) 2020-02-06T01:49:05.756 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:05.755+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.755596+0000) 2020-02-06T01:49:05.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:05.754+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.755686+0000) 2020-02-06T01:49:05.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:05.755+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.755909+0000) 2020-02-06T01:49:05.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:05.755+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.756286+0000) 2020-02-06T01:49:05.758 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:05.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.756736+0000) 2020-02-06T01:49:05.758 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:05.755+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.756883+0000) 2020-02-06T01:49:05.759 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:05.756+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.757044+0000) 2020-02-06T01:49:05.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:05.756+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.757477+0000) 2020-02-06T01:49:05.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:05.757+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.758270+0000) 2020-02-06T01:49:05.760 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:05.758+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.758632+0000) 2020-02-06T01:49:05.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:05.764+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:05.765596+0000) 2020-02-06T01:49:05.778 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:05.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:06.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:06.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:06.139581+0000) 2020-02-06T01:49:06.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:06.502+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:06.502710+0000) 2020-02-06T01:49:06.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:06.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:06.728041+0000) 2020-02-06T01:49:07.110 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:49:07.116 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:49:07.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:07.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.119467+0000) 2020-02-06T01:49:07.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:07.284+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.285491+0000) 2020-02-06T01:49:07.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:07.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.391813+0000) 2020-02-06T01:49:07.393 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:07.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.392083+0000) 2020-02-06T01:49:07.393 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:07.391+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.392332+0000) 2020-02-06T01:49:07.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:07.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:07.623837+0000) 2020-02-06T01:49:09.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:09.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:09.762828+0000) 2020-02-06T01:49:10.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:10.008+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.009003+0000) 2020-02-06T01:49:10.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:10.617+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.618245+0000) 2020-02-06T01:49:10.783 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:10.781+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.782460+0000) 2020-02-06T01:49:10.784 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:10.782+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.783476+0000) 2020-02-06T01:49:10.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:10.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.784225+0000) 2020-02-06T01:49:10.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:10.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.784829+0000) 2020-02-06T01:49:10.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:10.783+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.784889+0000) 2020-02-06T01:49:10.786 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:10.784+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.785676+0000) 2020-02-06T01:49:10.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:10.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.785491+0000) 2020-02-06T01:49:10.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:10.785+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.786294+0000) 2020-02-06T01:49:10.788 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:10.786+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.786717+0000) 2020-02-06T01:49:10.788 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:10.786+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.786985+0000) 2020-02-06T01:49:10.788 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:10.786+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.787282+0000) 2020-02-06T01:49:10.789 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:10.788+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.788524+0000) 2020-02-06T01:49:10.789 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:10.788+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.788768+0000) 2020-02-06T01:49:10.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:10.792+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.793580+0000) 2020-02-06T01:49:10.806 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:10.805+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214531 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:10.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:10.897+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.897604+0000) 2020-02-06T01:49:10.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:10.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:10.897596+0000) 2020-02-06T01:49:15.812 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:15.810+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.811446+0000) 2020-02-06T01:49:15.813 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:15.810+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.811691+0000) 2020-02-06T01:49:15.813 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:15.811+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.812196+0000) 2020-02-06T01:49:15.813 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:15.811+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.812506+0000) 2020-02-06T01:49:15.814 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:15.813+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.813469+0000) 2020-02-06T01:49:15.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:15.812+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.813856+0000) 2020-02-06T01:49:15.816 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:15.813+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.814637+0000) 2020-02-06T01:49:15.816 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:15.813+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.814314+0000) 2020-02-06T01:49:15.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:15.814+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.814614+0000) 2020-02-06T01:49:15.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:15.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.815002+0000) 2020-02-06T01:49:15.816 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:15.814+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.815239+0000) 2020-02-06T01:49:15.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:15.815+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.815469+0000) 2020-02-06T01:49:15.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:15.817+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.818856+0000) 2020-02-06T01:49:15.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:15.818+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:15.819758+0000) 2020-02-06T01:49:15.834 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:15.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:16.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:16.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:16.139811+0000) 2020-02-06T01:49:16.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:16.502+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:16.502917+0000) 2020-02-06T01:49:16.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:16.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:16.728283+0000) 2020-02-06T01:49:17.120 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:17.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.119700+0000) 2020-02-06T01:49:17.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:17.292+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.293884+0000) 2020-02-06T01:49:17.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:17.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.392068+0000) 2020-02-06T01:49:17.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:17.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.392284+0000) 2020-02-06T01:49:17.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:17.392+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.392636+0000) 2020-02-06T01:49:17.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:17.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:17.624117+0000) 2020-02-06T01:49:19.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:19.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:19.763060+0000) 2020-02-06T01:49:20.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:20.009+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.009257+0000) 2020-02-06T01:49:20.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:20.618+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.618536+0000) 2020-02-06T01:49:20.841 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:20.839+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.840579+0000) 2020-02-06T01:49:20.842 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:20.840+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.841278+0000) 2020-02-06T01:49:20.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:20.841+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.842298+0000) 2020-02-06T01:49:20.843 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:20.841+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.842013+0000) 2020-02-06T01:49:20.844 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:20.841+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.842875+0000) 2020-02-06T01:49:20.844 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:20.842+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.842986+0000) 2020-02-06T01:49:20.844 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:20.843+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.843614+0000) 2020-02-06T01:49:20.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:20.843+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.843900+0000) 2020-02-06T01:49:20.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:20.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.844475+0000) 2020-02-06T01:49:20.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:20.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.844590+0000) 2020-02-06T01:49:20.846 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:20.844+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.844790+0000) 2020-02-06T01:49:20.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:20.844+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.845188+0000) 2020-02-06T01:49:20.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:20.843+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.844886+0000) 2020-02-06T01:49:20.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:20.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.849997+0000) 2020-02-06T01:49:20.862 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:20.860+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:20.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:20.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.897803+0000) 2020-02-06T01:49:20.899 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:20.897+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:20.897860+0000) 2020-02-06T01:49:25.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:25.866+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.866986+0000) 2020-02-06T01:49:25.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:25.867+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.868087+0000) 2020-02-06T01:49:25.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:25.868+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.869445+0000) 2020-02-06T01:49:25.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:25.868+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.869900+0000) 2020-02-06T01:49:25.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:25.869+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.870374+0000) 2020-02-06T01:49:25.871 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:25.869+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.870788+0000) 2020-02-06T01:49:25.872 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:25.869+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.870947+0000) 2020-02-06T01:49:25.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:25.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.871443+0000) 2020-02-06T01:49:25.873 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:25.871+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.871944+0000) 2020-02-06T01:49:25.873 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:25.871+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.872403+0000) 2020-02-06T01:49:25.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:25.871+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.871950+0000) 2020-02-06T01:49:25.874 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:25.872+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.873006+0000) 2020-02-06T01:49:25.874 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:25.872+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.872988+0000) 2020-02-06T01:49:25.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:25.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:25.878925+0000) 2020-02-06T01:49:25.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:25.890+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:26.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:26.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:26.140054+0000) 2020-02-06T01:49:26.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:26.503+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:26.503166+0000) 2020-02-06T01:49:26.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:26.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:26.728507+0000) 2020-02-06T01:49:27.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:27.118+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.119935+0000) 2020-02-06T01:49:27.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:27.300+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.301739+0000) 2020-02-06T01:49:27.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:27.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.392295+0000) 2020-02-06T01:49:27.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:27.392+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.392505+0000) 2020-02-06T01:49:27.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:27.392+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.392830+0000) 2020-02-06T01:49:27.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:27.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:27.624307+0000) 2020-02-06T01:49:29.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:29.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:29.763372+0000) 2020-02-06T01:49:30.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:30.008+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.009473+0000) 2020-02-06T01:49:30.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:30.617+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.618784+0000) 2020-02-06T01:49:30.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:30.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.896453+0000) 2020-02-06T01:49:30.898 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:30.896+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.897506+0000) 2020-02-06T01:49:30.899 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:30.896+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.897968+0000) 2020-02-06T01:49:30.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:30.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.898125+0000) 2020-02-06T01:49:30.900 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:30.898+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.899663+0000) 2020-02-06T01:49:30.901 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:30.898+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.900007+0000) 2020-02-06T01:49:30.902 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:30.900+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.901458+0000) 2020-02-06T01:49:30.902 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:30.899+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.901057+0000) 2020-02-06T01:49:30.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:30.900+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.901815+0000) 2020-02-06T01:49:30.903 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:30.900+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.902372+0000) 2020-02-06T01:49:30.904 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:30.901+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.902893+0000) 2020-02-06T01:49:30.904 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:30.901+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.902976+0000) 2020-02-06T01:49:30.904 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:30.902+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.903244+0000) 2020-02-06T01:49:30.905 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:30.902+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.903785+0000) 2020-02-06T01:49:30.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:30.905+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:30.906112+0000) 2020-02-06T01:49:30.920 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:30.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:35.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:35.924+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.925401+0000) 2020-02-06T01:49:35.927 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:35.924+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.925981+0000) 2020-02-06T01:49:35.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:35.925+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.926576+0000) 2020-02-06T01:49:35.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:35.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.927026+0000) 2020-02-06T01:49:35.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:35.925+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.926734+0000) 2020-02-06T01:49:35.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:35.926+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.927851+0000) 2020-02-06T01:49:35.929 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:35.927+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.928203+0000) 2020-02-06T01:49:35.929 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:35.927+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.929051+0000) 2020-02-06T01:49:35.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:35.928+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.929499+0000) 2020-02-06T01:49:35.930 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:35.928+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.929953+0000) 2020-02-06T01:49:35.931 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:35.928+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.930102+0000) 2020-02-06T01:49:35.931 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:35.929+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.930621+0000) 2020-02-06T01:49:35.932 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:35.930+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.931571+0000) 2020-02-06T01:49:35.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:35.936+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:35.937719+0000) 2020-02-06T01:49:35.949 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:35.947+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214729 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:36.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:36.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:36.140302+0000) 2020-02-06T01:49:36.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:36.502+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:36.503409+0000) 2020-02-06T01:49:36.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:36.727+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:36.728722+0000) 2020-02-06T01:49:37.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:37.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.120178+0000) 2020-02-06T01:49:37.272 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:49:37.279 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:49:37.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:37.308+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.309850+0000) 2020-02-06T01:49:37.393 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:37.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.392521+0000) 2020-02-06T01:49:37.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:37.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.392707+0000) 2020-02-06T01:49:37.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:37.391+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.393075+0000) 2020-02-06T01:49:37.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:37.623+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.624545+0000) 2020-02-06T01:49:37.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:37.647+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:37.648663+0000) 2020-02-06T01:49:39.764 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:39.762+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:39.763605+0000) 2020-02-06T01:49:40.010 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:40.008+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.009703+0000) 2020-02-06T01:49:40.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:40.618+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.619041+0000) 2020-02-06T01:49:40.899 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:40.897+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.898392+0000) 2020-02-06T01:49:40.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:40.952+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.953949+0000) 2020-02-06T01:49:40.955 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:40.953+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.954310+0000) 2020-02-06T01:49:40.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:40.953+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.954953+0000) 2020-02-06T01:49:40.956 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:40.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.954825+0000) 2020-02-06T01:49:40.957 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:40.954+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.956133+0000) 2020-02-06T01:49:40.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:40.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.956760+0000) 2020-02-06T01:49:40.958 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:40.956+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.957151+0000) 2020-02-06T01:49:40.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:40.956+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.957480+0000) 2020-02-06T01:49:40.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:40.956+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.958002+0000) 2020-02-06T01:49:40.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:40.956+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.958118+0000) 2020-02-06T01:49:40.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:40.957+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.958898+0000) 2020-02-06T01:49:40.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:40.958+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.959471+0000) 2020-02-06T01:49:40.960 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:40.958+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.959750+0000) 2020-02-06T01:49:40.964 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:40.962+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:40.963908+0000) 2020-02-06T01:49:40.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:40.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:45.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:45.980+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.981717+0000) 2020-02-06T01:49:45.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:45.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.982381+0000) 2020-02-06T01:49:45.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:45.981+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.983014+0000) 2020-02-06T01:49:45.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:45.983+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.984370+0000) 2020-02-06T01:49:45.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:45.983+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.984594+0000) 2020-02-06T01:49:45.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:45.983+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.984726+0000) 2020-02-06T01:49:45.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:45.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.985229+0000) 2020-02-06T01:49:45.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:45.983+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.984978+0000) 2020-02-06T01:49:45.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:45.984+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.985709+0000) 2020-02-06T01:49:45.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:45.984+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.985948+0000) 2020-02-06T01:49:45.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:45.984+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.986331+0000) 2020-02-06T01:49:45.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:45.986+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.987115+0000) 2020-02-06T01:49:45.988 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:45.986+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.988012+0000) 2020-02-06T01:49:45.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:45.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:45.992690+0000) 2020-02-06T01:49:46.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:46.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214809 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:46.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:46.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:46.140531+0000) 2020-02-06T01:49:46.504 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:46.502+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:46.503692+0000) 2020-02-06T01:49:46.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:46.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:46.728942+0000) 2020-02-06T01:49:47.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:47.119+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.120423+0000) 2020-02-06T01:49:47.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:47.313+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.314999+0000) 2020-02-06T01:49:47.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:47.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.392722+0000) 2020-02-06T01:49:47.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:47.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.392968+0000) 2020-02-06T01:49:47.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:47.391+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.393298+0000) 2020-02-06T01:49:47.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:47.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.624781+0000) 2020-02-06T01:49:47.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:47.648+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:47.648902+0000) 2020-02-06T01:49:49.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:49.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:49.763846+0000) 2020-02-06T01:49:50.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:50.009+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:50.009928+0000) 2020-02-06T01:49:50.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:50.617+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:50.619288+0000) 2020-02-06T01:49:50.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:50.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:50.898621+0000) 2020-02-06T01:49:51.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:51.010+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.011148+0000) 2020-02-06T01:49:51.012 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:51.010+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.011666+0000) 2020-02-06T01:49:51.013 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:51.011+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.012303+0000) 2020-02-06T01:49:51.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:51.011+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.012684+0000) 2020-02-06T01:49:51.014 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:51.011+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.013133+0000) 2020-02-06T01:49:51.014 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:51.012+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.013426+0000) 2020-02-06T01:49:51.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:51.013+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.014848+0000) 2020-02-06T01:49:51.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:51.013+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.015277+0000) 2020-02-06T01:49:51.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:51.014+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.015912+0000) 2020-02-06T01:49:51.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:51.013+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.015400+0000) 2020-02-06T01:49:51.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:51.015+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.016803+0000) 2020-02-06T01:49:51.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:51.015+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.016974+0000) 2020-02-06T01:49:51.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:51.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.022315+0000) 2020-02-06T01:49:51.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:51.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:51.022623+0000) 2020-02-06T01:49:51.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:51.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214856 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:56.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:56.038+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.039071+0000) 2020-02-06T01:49:56.041 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:49:56.039+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.040214+0000) 2020-02-06T01:49:56.042 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:56.038+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.040254+0000) 2020-02-06T01:49:56.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:56.041+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.042189+0000) 2020-02-06T01:49:56.043 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:56.040+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.041947+0000) 2020-02-06T01:49:56.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:56.041+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.042634+0000) 2020-02-06T01:49:56.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:56.041+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.042989+0000) 2020-02-06T01:49:56.044 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:56.042+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.043726+0000) 2020-02-06T01:49:56.045 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:56.043+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.044818+0000) 2020-02-06T01:49:56.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:56.044+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.045099+0000) 2020-02-06T01:49:56.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:49:56.043+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.045190+0000) 2020-02-06T01:49:56.046 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:56.044+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.045470+0000) 2020-02-06T01:49:56.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:56.047+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.048898+0000) 2020-02-06T01:49:56.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:56.048+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.049186+0000) 2020-02-06T01:49:56.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:49:56.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214890 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:49:56.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:49:56.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.140767+0000) 2020-02-06T01:49:56.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:49:56.503+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.503906+0000) 2020-02-06T01:49:56.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:49:56.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:56.729159+0000) 2020-02-06T01:49:57.121 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:49:57.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.120657+0000) 2020-02-06T01:49:57.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:57.325+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.326579+0000) 2020-02-06T01:49:57.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:49:57.391+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.392964+0000) 2020-02-06T01:49:57.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:49:57.391+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.393172+0000) 2020-02-06T01:49:57.394 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:49:57.392+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.393484+0000) 2020-02-06T01:49:57.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:49:57.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.625006+0000) 2020-02-06T01:49:57.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:49:57.648+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:57.649128+0000) 2020-02-06T01:49:59.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:49:59.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:49:59.764117+0000) 2020-02-06T01:50:00.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:00.009+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:00.010161+0000) 2020-02-06T01:50:00.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:00.618+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:00.619615+0000) 2020-02-06T01:50:00.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:00.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:00.898843+0000) 2020-02-06T01:50:01.068 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:01.066+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.067353+0000) 2020-02-06T01:50:01.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:01.068+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.069470+0000) 2020-02-06T01:50:01.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:01.069+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.070425+0000) 2020-02-06T01:50:01.071 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:01.069+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.070830+0000) 2020-02-06T01:50:01.072 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:01.070+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.071669+0000) 2020-02-06T01:50:01.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:01.070+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.071980+0000) 2020-02-06T01:50:01.073 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:01.070+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.072144+0000) 2020-02-06T01:50:01.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:01.071+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.072317+0000) 2020-02-06T01:50:01.074 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:01.071+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.072840+0000) 2020-02-06T01:50:01.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:01.071+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.073318+0000) 2020-02-06T01:50:01.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:01.072+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.073555+0000) 2020-02-06T01:50:01.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:01.072+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.073929+0000) 2020-02-06T01:50:01.075 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:01.073+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.074741+0000) 2020-02-06T01:50:01.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:01.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:01.079392+0000) 2020-02-06T01:50:01.090 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:01.088+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:06.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:06.095+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.096588+0000) 2020-02-06T01:50:06.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:06.095+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.096624+0000) 2020-02-06T01:50:06.099 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:06.095+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.097027+0000) 2020-02-06T01:50:06.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:06.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.097584+0000) 2020-02-06T01:50:06.099 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:06.096+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.098013+0000) 2020-02-06T01:50:06.100 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:06.097+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.098609+0000) 2020-02-06T01:50:06.100 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:06.097+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.099126+0000) 2020-02-06T01:50:06.101 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:06.097+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.099205+0000) 2020-02-06T01:50:06.101 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:06.097+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.098684+0000) 2020-02-06T01:50:06.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:06.098+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.099179+0000) 2020-02-06T01:50:06.101 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:06.098+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.099700+0000) 2020-02-06T01:50:06.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:06.098+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.099972+0000) 2020-02-06T01:50:06.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:06.099+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.100616+0000) 2020-02-06T01:50:06.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:06.106+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.107482+0000) 2020-02-06T01:50:06.119 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:06.117+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 214971 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:06.141 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:06.139+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.140955+0000) 2020-02-06T01:50:06.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:06.503+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.504172+0000) 2020-02-06T01:50:06.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:06.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:06.729374+0000) 2020-02-06T01:50:07.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:07.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.120913+0000) 2020-02-06T01:50:07.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:07.334+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.334874+0000) 2020-02-06T01:50:07.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:07.392+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.393200+0000) 2020-02-06T01:50:07.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:07.392+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.393483+0000) 2020-02-06T01:50:07.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:07.392+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.393701+0000) 2020-02-06T01:50:07.461 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:50:07.467 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:50:07.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:07.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.625264+0000) 2020-02-06T01:50:07.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:07.648+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:07.649335+0000) 2020-02-06T01:50:09.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:09.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:09.764380+0000) 2020-02-06T01:50:10.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:10.010+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:10.010400+0000) 2020-02-06T01:50:10.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:10.619+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:10.619891+0000) 2020-02-06T01:50:10.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:10.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:10.899068+0000) 2020-02-06T01:50:11.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:11.123+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.125216+0000) 2020-02-06T01:50:11.127 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:11.125+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.126257+0000) 2020-02-06T01:50:11.127 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:11.125+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.126654+0000) 2020-02-06T01:50:11.128 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:11.125+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.127060+0000) 2020-02-06T01:50:11.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:11.126+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.127624+0000) 2020-02-06T01:50:11.129 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:11.127+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.128458+0000) 2020-02-06T01:50:11.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:11.127+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.129017+0000) 2020-02-06T01:50:11.130 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:11.127+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.129032+0000) 2020-02-06T01:50:11.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:11.128+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.129563+0000) 2020-02-06T01:50:11.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:11.128+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.130160+0000) 2020-02-06T01:50:11.131 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:11.128+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.130220+0000) 2020-02-06T01:50:11.132 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:11.129+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.130751+0000) 2020-02-06T01:50:11.132 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:11.129+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.131102+0000) 2020-02-06T01:50:11.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:11.132+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:11.134034+0000) 2020-02-06T01:50:11.148 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:11.146+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:16.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:16.140+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.141182+0000) 2020-02-06T01:50:16.154 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:16.153+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.153473+0000) 2020-02-06T01:50:16.154 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:16.153+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.153807+0000) 2020-02-06T01:50:16.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:16.154+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.154495+0000) 2020-02-06T01:50:16.155 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:16.154+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.154789+0000) 2020-02-06T01:50:16.156 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:16.153+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.155027+0000) 2020-02-06T01:50:16.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:16.155+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.155429+0000) 2020-02-06T01:50:16.156 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:16.154+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.155682+0000) 2020-02-06T01:50:16.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:16.156+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.156359+0000) 2020-02-06T01:50:16.157 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:16.156+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.156621+0000) 2020-02-06T01:50:16.157 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:16.154+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.156066+0000) 2020-02-06T01:50:16.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:16.155+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.156957+0000) 2020-02-06T01:50:16.158 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:16.156+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.157993+0000) 2020-02-06T01:50:16.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:16.156+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.158360+0000) 2020-02-06T01:50:16.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:16.162+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.162333+0000) 2020-02-06T01:50:16.176 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:16.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215052 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:16.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:16.503+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.504359+0000) 2020-02-06T01:50:16.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:16.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:16.729600+0000) 2020-02-06T01:50:17.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:17.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.121181+0000) 2020-02-06T01:50:17.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:17.342+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.342735+0000) 2020-02-06T01:50:17.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:17.392+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.393492+0000) 2020-02-06T01:50:17.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:17.392+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.393700+0000) 2020-02-06T01:50:17.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:17.392+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.393924+0000) 2020-02-06T01:50:17.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:17.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.625519+0000) 2020-02-06T01:50:17.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:17.649+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:17.649514+0000) 2020-02-06T01:50:19.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:19.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:19.764614+0000) 2020-02-06T01:50:20.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:20.010+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:20.010641+0000) 2020-02-06T01:50:20.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:20.619+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:20.620207+0000) 2020-02-06T01:50:20.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:20.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:20.899375+0000) 2020-02-06T01:50:21.182 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:21.179+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.180758+0000) 2020-02-06T01:50:21.182 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:21.180+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.181973+0000) 2020-02-06T01:50:21.183 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:21.181+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.181696+0000) 2020-02-06T01:50:21.183 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:21.181+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.182731+0000) 2020-02-06T01:50:21.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:21.182+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.183322+0000) 2020-02-06T01:50:21.184 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:21.182+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.183474+0000) 2020-02-06T01:50:21.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:21.183+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.183822+0000) 2020-02-06T01:50:21.185 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:21.183+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.184280+0000) 2020-02-06T01:50:21.186 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:21.184+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.185476+0000) 2020-02-06T01:50:21.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:21.184+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.185186+0000) 2020-02-06T01:50:21.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:21.185+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.186214+0000) 2020-02-06T01:50:21.187 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:21.186+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.186489+0000) 2020-02-06T01:50:21.187 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:21.186+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.187056+0000) 2020-02-06T01:50:21.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:21.190+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:21.191763+0000) 2020-02-06T01:50:21.206 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:21.205+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215098 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:26.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:26.141+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.141468+0000) 2020-02-06T01:50:26.213 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:26.210+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.211977+0000) 2020-02-06T01:50:26.213 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:26.211+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.212440+0000) 2020-02-06T01:50:26.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:26.212+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.213049+0000) 2020-02-06T01:50:26.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:26.212+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.213243+0000) 2020-02-06T01:50:26.214 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:26.212+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.213729+0000) 2020-02-06T01:50:26.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:26.213+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.214820+0000) 2020-02-06T01:50:26.216 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:26.214+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.214411+0000) 2020-02-06T01:50:26.217 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:26.216+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.216517+0000) 2020-02-06T01:50:26.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:26.215+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.216828+0000) 2020-02-06T01:50:26.218 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:26.216+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.217046+0000) 2020-02-06T01:50:26.218 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:26.216+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.217196+0000) 2020-02-06T01:50:26.219 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:26.216+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.217853+0000) 2020-02-06T01:50:26.219 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:26.218+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.218419+0000) 2020-02-06T01:50:26.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:26.220+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.221506+0000) 2020-02-06T01:50:26.236 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:26.234+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215133 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:26.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:26.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.504685+0000) 2020-02-06T01:50:26.730 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:26.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:26.729814+0000) 2020-02-06T01:50:27.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:27.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.121394+0000) 2020-02-06T01:50:27.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:27.353+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.354401+0000) 2020-02-06T01:50:27.395 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:27.393+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.393694+0000) 2020-02-06T01:50:27.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:27.393+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.393945+0000) 2020-02-06T01:50:27.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:27.393+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.394213+0000) 2020-02-06T01:50:27.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:27.624+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.625750+0000) 2020-02-06T01:50:27.650 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:27.648+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:27.649744+0000) 2020-02-06T01:50:29.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:29.763+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:29.764852+0000) 2020-02-06T01:50:30.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:30.009+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:30.010872+0000) 2020-02-06T01:50:30.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:30.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:30.620495+0000) 2020-02-06T01:50:30.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:30.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:30.899588+0000) 2020-02-06T01:50:31.242 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:31.240+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.241381+0000) 2020-02-06T01:50:31.243 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:31.241+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.241555+0000) 2020-02-06T01:50:31.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:31.242+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.243400+0000) 2020-02-06T01:50:31.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:31.243+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.243874+0000) 2020-02-06T01:50:31.245 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:31.243+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.244372+0000) 2020-02-06T01:50:31.246 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:31.244+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.245420+0000) 2020-02-06T01:50:31.246 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:31.244+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.245833+0000) 2020-02-06T01:50:31.247 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:31.245+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.246277+0000) 2020-02-06T01:50:31.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:31.246+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.246522+0000) 2020-02-06T01:50:31.248 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:31.246+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.247052+0000) 2020-02-06T01:50:31.248 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:31.245+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.246776+0000) 2020-02-06T01:50:31.248 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:31.247+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.247726+0000) 2020-02-06T01:50:31.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:31.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.248826+0000) 2020-02-06T01:50:31.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:31.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:31.249036+0000) 2020-02-06T01:50:31.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:31.264+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215179 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:36.142 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:36.141+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.141669+0000) 2020-02-06T01:50:36.271 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:36.269+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.270195+0000) 2020-02-06T01:50:36.271 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:36.269+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.270471+0000) 2020-02-06T01:50:36.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:36.270+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.271152+0000) 2020-02-06T01:50:36.272 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:36.270+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.271555+0000) 2020-02-06T01:50:36.273 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:36.271+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.272167+0000) 2020-02-06T01:50:36.275 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:36.273+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.274047+0000) 2020-02-06T01:50:36.275 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:36.274+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.274694+0000) 2020-02-06T01:50:36.276 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:36.274+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.275289+0000) 2020-02-06T01:50:36.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:36.274+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.275142+0000) 2020-02-06T01:50:36.277 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:36.275+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.276051+0000) 2020-02-06T01:50:36.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:36.275+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.276016+0000) 2020-02-06T01:50:36.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:36.276+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.276614+0000) 2020-02-06T01:50:36.278 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:36.276+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.277320+0000) 2020-02-06T01:50:36.283 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:36.280+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.281941+0000) 2020-02-06T01:50:36.295 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:36.293+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215213 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:36.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:36.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.504924+0000) 2020-02-06T01:50:36.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:36.728+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:36.730041+0000) 2020-02-06T01:50:37.122 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:37.120+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.121602+0000) 2020-02-06T01:50:37.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:37.362+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.362953+0000) 2020-02-06T01:50:37.395 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:37.393+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.393932+0000) 2020-02-06T01:50:37.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:37.393+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.394166+0000) 2020-02-06T01:50:37.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:37.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.394487+0000) 2020-02-06T01:50:37.624 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:50:37.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:37.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.625975+0000) 2020-02-06T01:50:37.630 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:50:37.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:37.648+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:37.649966+0000) 2020-02-06T01:50:39.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:39.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:39.765141+0000) 2020-02-06T01:50:40.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:40.010+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:40.011145+0000) 2020-02-06T01:50:40.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:40.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:40.620724+0000) 2020-02-06T01:50:40.900 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:40.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:40.899815+0000) 2020-02-06T01:50:41.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:41.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.300553+0000) 2020-02-06T01:50:41.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:41.300+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.301546+0000) 2020-02-06T01:50:41.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:41.301+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.302286+0000) 2020-02-06T01:50:41.303 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:41.301+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.302538+0000) 2020-02-06T01:50:41.304 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:41.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.303168+0000) 2020-02-06T01:50:41.304 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:41.302+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.302829+0000) 2020-02-06T01:50:41.305 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:41.303+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.304906+0000) 2020-02-06T01:50:41.306 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:41.304+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.305165+0000) 2020-02-06T01:50:41.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:41.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.305775+0000) 2020-02-06T01:50:41.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:41.305+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.305957+0000) 2020-02-06T01:50:41.307 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:41.305+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.306152+0000) 2020-02-06T01:50:41.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:41.306+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.306514+0000) 2020-02-06T01:50:41.308 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:41.306+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.306902+0000) 2020-02-06T01:50:41.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:41.307+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:41.308611+0000) 2020-02-06T01:50:41.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:41.322+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:46.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:46.141+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.141930+0000) 2020-02-06T01:50:46.328 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:46.326+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.327675+0000) 2020-02-06T01:50:46.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:46.328+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.328545+0000) 2020-02-06T01:50:46.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:46.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.329540+0000) 2020-02-06T01:50:46.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:46.329+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.330379+0000) 2020-02-06T01:50:46.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:46.329+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.330755+0000) 2020-02-06T01:50:46.332 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:46.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.331542+0000) 2020-02-06T01:50:46.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:46.330+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.331894+0000) 2020-02-06T01:50:46.333 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:46.332+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.332772+0000) 2020-02-06T01:50:46.334 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:46.332+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.333591+0000) 2020-02-06T01:50:46.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:46.333+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.333869+0000) 2020-02-06T01:50:46.335 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:46.334+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.334578+0000) 2020-02-06T01:50:46.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:46.334+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.335080+0000) 2020-02-06T01:50:46.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:46.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.338425+0000) 2020-02-06T01:50:46.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:46.337+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.338671+0000) 2020-02-06T01:50:46.354 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:46.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215293 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:46.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:46.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.505131+0000) 2020-02-06T01:50:46.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:46.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:46.730287+0000) 2020-02-06T01:50:47.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:47.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.121808+0000) 2020-02-06T01:50:47.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:47.370+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.371307+0000) 2020-02-06T01:50:47.395 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:47.393+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.394165+0000) 2020-02-06T01:50:47.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:47.393+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.394393+0000) 2020-02-06T01:50:47.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:47.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.394721+0000) 2020-02-06T01:50:47.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:47.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.626178+0000) 2020-02-06T01:50:47.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:47.649+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:47.650201+0000) 2020-02-06T01:50:49.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:49.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:49.765378+0000) 2020-02-06T01:50:50.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:50.010+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:50.011387+0000) 2020-02-06T01:50:50.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:50.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:50.620972+0000) 2020-02-06T01:50:50.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:50.898+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:50.900055+0000) 2020-02-06T01:50:51.360 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:51.358+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.359474+0000) 2020-02-06T01:50:51.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:51.359+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.359572+0000) 2020-02-06T01:50:51.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:51.358+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.360052+0000) 2020-02-06T01:50:51.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:51.359+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.360440+0000) 2020-02-06T01:50:51.362 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:51.359+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.360723+0000) 2020-02-06T01:50:51.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:51.359+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.360976+0000) 2020-02-06T01:50:51.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:51.361+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.361630+0000) 2020-02-06T01:50:51.363 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:51.361+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.362183+0000) 2020-02-06T01:50:51.363 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:51.361+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.362777+0000) 2020-02-06T01:50:51.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:51.362+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.363095+0000) 2020-02-06T01:50:51.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:51.362+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.363359+0000) 2020-02-06T01:50:51.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:51.362+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.363867+0000) 2020-02-06T01:50:51.365 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:51.363+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.363920+0000) 2020-02-06T01:50:51.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:51.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:51.368565+0000) 2020-02-06T01:50:51.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:51.381+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215341 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:56.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:56.142+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.142176+0000) 2020-02-06T01:50:56.386 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:56.384+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.385671+0000) 2020-02-06T01:50:56.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:56.385+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.386815+0000) 2020-02-06T01:50:56.388 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:56.386+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.387577+0000) 2020-02-06T01:50:56.389 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:56.387+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.388550+0000) 2020-02-06T01:50:56.390 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:56.388+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.389305+0000) 2020-02-06T01:50:56.391 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:56.388+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.389598+0000) 2020-02-06T01:50:56.391 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:56.389+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.390289+0000) 2020-02-06T01:50:56.392 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:50:56.390+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.390601+0000) 2020-02-06T01:50:56.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:56.390+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.391378+0000) 2020-02-06T01:50:56.393 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:50:56.392+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.392899+0000) 2020-02-06T01:50:56.394 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:50:56.392+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.393641+0000) 2020-02-06T01:50:56.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:56.392+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.393329+0000) 2020-02-06T01:50:56.395 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:56.393+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.393935+0000) 2020-02-06T01:50:56.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:56.397+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.398391+0000) 2020-02-06T01:50:56.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:50:56.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215374 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:50:56.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:50:56.505+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.505352+0000) 2020-02-06T01:50:56.731 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:50:56.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:56.730495+0000) 2020-02-06T01:50:57.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:50:57.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.122015+0000) 2020-02-06T01:50:57.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:57.382+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.382965+0000) 2020-02-06T01:50:57.395 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:50:57.394+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.394397+0000) 2020-02-06T01:50:57.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:50:57.394+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.394647+0000) 2020-02-06T01:50:57.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:50:57.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.394962+0000) 2020-02-06T01:50:57.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:50:57.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.626457+0000) 2020-02-06T01:50:57.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:50:57.650+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:57.650469+0000) 2020-02-06T01:50:59.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:50:59.764+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:50:59.765635+0000) 2020-02-06T01:51:00.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:00.011+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:00.011637+0000) 2020-02-06T01:51:00.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:00.619+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:00.621217+0000) 2020-02-06T01:51:00.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:00.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:00.900334+0000) 2020-02-06T01:51:01.417 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:01.415+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.416233+0000) 2020-02-06T01:51:01.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:01.415+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.416649+0000) 2020-02-06T01:51:01.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:01.417+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.418654+0000) 2020-02-06T01:51:01.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:01.418+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.419264+0000) 2020-02-06T01:51:01.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:01.418+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.419419+0000) 2020-02-06T01:51:01.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:01.419+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.420578+0000) 2020-02-06T01:51:01.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:01.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.420862+0000) 2020-02-06T01:51:01.422 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:01.420+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.421373+0000) 2020-02-06T01:51:01.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:01.420+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.421513+0000) 2020-02-06T01:51:01.423 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:01.420+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.421837+0000) 2020-02-06T01:51:01.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:01.421+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.422141+0000) 2020-02-06T01:51:01.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:01.421+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.422752+0000) 2020-02-06T01:51:01.424 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:01.422+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.423479+0000) 2020-02-06T01:51:01.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:01.427+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:01.428140+0000) 2020-02-06T01:51:01.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:01.438+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215422 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:06.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:06.141+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.142471+0000) 2020-02-06T01:51:06.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:06.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.444638+0000) 2020-02-06T01:51:06.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:06.444+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.445331+0000) 2020-02-06T01:51:06.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:06.445+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.446407+0000) 2020-02-06T01:51:06.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:06.445+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.446909+0000) 2020-02-06T01:51:06.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:06.446+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.447232+0000) 2020-02-06T01:51:06.449 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:06.446+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.448118+0000) 2020-02-06T01:51:06.451 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:06.448+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.449261+0000) 2020-02-06T01:51:06.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:06.448+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.449627+0000) 2020-02-06T01:51:06.452 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:06.448+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.450030+0000) 2020-02-06T01:51:06.452 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:06.449+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.450773+0000) 2020-02-06T01:51:06.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:06.449+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.451239+0000) 2020-02-06T01:51:06.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:06.450+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.451421+0000) 2020-02-06T01:51:06.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:06.450+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.451348+0000) 2020-02-06T01:51:06.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:06.454+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.455087+0000) 2020-02-06T01:51:06.470 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:06.468+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215455 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:06.506 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:06.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.505584+0000) 2020-02-06T01:51:06.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:06.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:06.730720+0000) 2020-02-06T01:51:07.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:07.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.122252+0000) 2020-02-06T01:51:07.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:07.394+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.394372+0000) 2020-02-06T01:51:07.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:07.393+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.394695+0000) 2020-02-06T01:51:07.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:07.393+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.394859+0000) 2020-02-06T01:51:07.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:07.393+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.395203+0000) 2020-02-06T01:51:07.627 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:07.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.626686+0000) 2020-02-06T01:51:07.651 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:07.650+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:07.650718+0000) 2020-02-06T01:51:07.812 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:51:07.818 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:51:09.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:09.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:09.765877+0000) 2020-02-06T01:51:10.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:10.011+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:10.011834+0000) 2020-02-06T01:51:10.622 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:10.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:10.621481+0000) 2020-02-06T01:51:10.901 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:10.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:10.900574+0000) 2020-02-06T01:51:11.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:11.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.474590+0000) 2020-02-06T01:51:11.477 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:11.474+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.476068+0000) 2020-02-06T01:51:11.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:11.476+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.476540+0000) 2020-02-06T01:51:11.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:11.476+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.477836+0000) 2020-02-06T01:51:11.479 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:11.478+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.478198+0000) 2020-02-06T01:51:11.480 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:11.478+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.479505+0000) 2020-02-06T01:51:11.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:11.478+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.480311+0000) 2020-02-06T01:51:11.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:11.480+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.480706+0000) 2020-02-06T01:51:11.482 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:11.480+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.481575+0000) 2020-02-06T01:51:11.482 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:11.481+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.481523+0000) 2020-02-06T01:51:11.483 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:11.480+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.481643+0000) 2020-02-06T01:51:11.483 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:11.481+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.481934+0000) 2020-02-06T01:51:11.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:11.484+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.484480+0000) 2020-02-06T01:51:11.485 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:11.484+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:11.484714+0000) 2020-02-06T01:51:11.498 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:11.495+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215502 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:16.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:16.141+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.142664+0000) 2020-02-06T01:51:16.502 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:16.500+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.501382+0000) 2020-02-06T01:51:16.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:16.503+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.504312+0000) 2020-02-06T01:51:16.506 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:16.504+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.505454+0000) 2020-02-06T01:51:16.506 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:16.505+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.506187+0000) 2020-02-06T01:51:16.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:16.504+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.505793+0000) 2020-02-06T01:51:16.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:16.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.505940+0000) 2020-02-06T01:51:16.508 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:16.505+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.506916+0000) 2020-02-06T01:51:16.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:16.506+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.507212+0000) 2020-02-06T01:51:16.508 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:16.506+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.507665+0000) 2020-02-06T01:51:16.509 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:16.506+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.507873+0000) 2020-02-06T01:51:16.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:16.506+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.508184+0000) 2020-02-06T01:51:16.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:16.507+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.508143+0000) 2020-02-06T01:51:16.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:16.507+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.508627+0000) 2020-02-06T01:51:16.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:16.507+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.509283+0000) 2020-02-06T01:51:16.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:16.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.514367+0000) 2020-02-06T01:51:16.529 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:16.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215535 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:16.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:16.729+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:16.730929+0000) 2020-02-06T01:51:17.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:17.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.122480+0000) 2020-02-06T01:51:17.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:17.394+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.395066+0000) 2020-02-06T01:51:17.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:17.394+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.395003+0000) 2020-02-06T01:51:17.396 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:17.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.395382+0000) 2020-02-06T01:51:17.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:17.401+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.402743+0000) 2020-02-06T01:51:17.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:17.625+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.626941+0000) 2020-02-06T01:51:17.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:17.649+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:17.650950+0000) 2020-02-06T01:51:19.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:19.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:19.766137+0000) 2020-02-06T01:51:20.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:20.011+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:20.012060+0000) 2020-02-06T01:51:20.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:20.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:20.621708+0000) 2020-02-06T01:51:20.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:20.899+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:20.900811+0000) 2020-02-06T01:51:21.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:21.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.534795+0000) 2020-02-06T01:51:21.537 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:21.534+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.536336+0000) 2020-02-06T01:51:21.538 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:21.535+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.536678+0000) 2020-02-06T01:51:21.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:21.535+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.537216+0000) 2020-02-06T01:51:21.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:21.536+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.537493+0000) 2020-02-06T01:51:21.539 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:21.536+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.537966+0000) 2020-02-06T01:51:21.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:21.537+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.538770+0000) 2020-02-06T01:51:21.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:21.537+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.539230+0000) 2020-02-06T01:51:21.541 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:21.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.539861+0000) 2020-02-06T01:51:21.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:21.539+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.540251+0000) 2020-02-06T01:51:21.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:21.539+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.540347+0000) 2020-02-06T01:51:21.542 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:21.539+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.540803+0000) 2020-02-06T01:51:21.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:21.540+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.541282+0000) 2020-02-06T01:51:21.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:21.545+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:21.546310+0000) 2020-02-06T01:51:21.557 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:21.554+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215582 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:26.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:26.142+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.142912+0000) 2020-02-06T01:51:26.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:26.504+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.506173+0000) 2020-02-06T01:51:26.563 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:26.560+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.562052+0000) 2020-02-06T01:51:26.563 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:26.561+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.562238+0000) 2020-02-06T01:51:26.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:26.561+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.562855+0000) 2020-02-06T01:51:26.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:26.561+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.563253+0000) 2020-02-06T01:51:26.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:26.563+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.564581+0000) 2020-02-06T01:51:26.566 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:26.564+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.565219+0000) 2020-02-06T01:51:26.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:26.564+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.565796+0000) 2020-02-06T01:51:26.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:26.565+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.566972+0000) 2020-02-06T01:51:26.568 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:26.565+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.567185+0000) 2020-02-06T01:51:26.569 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:26.566+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.567736+0000) 2020-02-06T01:51:26.569 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:26.566+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.567846+0000) 2020-02-06T01:51:26.569 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:26.566+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.568108+0000) 2020-02-06T01:51:26.573 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:26.571+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.572521+0000) 2020-02-06T01:51:26.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:26.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.573697+0000) 2020-02-06T01:51:26.586 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:26.583+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215616 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:26.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:26.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:26.731166+0000) 2020-02-06T01:51:27.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:27.121+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.122657+0000) 2020-02-06T01:51:27.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:27.393+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.395238+0000) 2020-02-06T01:51:27.396 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:27.393+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.395279+0000) 2020-02-06T01:51:27.397 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:27.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.395610+0000) 2020-02-06T01:51:27.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:27.410+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.411025+0000) 2020-02-06T01:51:27.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:27.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.627177+0000) 2020-02-06T01:51:27.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:27.650+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:27.651213+0000) 2020-02-06T01:51:29.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:29.765+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:29.766371+0000) 2020-02-06T01:51:30.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:30.011+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:30.012330+0000) 2020-02-06T01:51:30.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:30.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:30.621937+0000) 2020-02-06T01:51:30.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:30.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:30.901041+0000) 2020-02-06T01:51:31.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:31.589+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.591071+0000) 2020-02-06T01:51:31.592 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:31.590+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.591827+0000) 2020-02-06T01:51:31.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:31.591+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.592841+0000) 2020-02-06T01:51:31.594 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:31.592+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.593520+0000) 2020-02-06T01:51:31.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:31.592+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.593974+0000) 2020-02-06T01:51:31.595 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:31.593+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.594673+0000) 2020-02-06T01:51:31.596 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:31.593+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.594952+0000) 2020-02-06T01:51:31.596 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:31.594+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.595239+0000) 2020-02-06T01:51:31.596 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:31.594+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.595434+0000) 2020-02-06T01:51:31.597 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:31.594+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.595720+0000) 2020-02-06T01:51:31.597 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:31.594+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.596091+0000) 2020-02-06T01:51:31.597 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:31.595+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.596457+0000) 2020-02-06T01:51:31.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:31.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.597418+0000) 2020-02-06T01:51:31.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:31.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:31.602362+0000) 2020-02-06T01:51:31.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:31.615+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215662 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:36.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:36.142+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.143162+0000) 2020-02-06T01:51:36.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:36.505+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.506493+0000) 2020-02-06T01:51:36.622 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:36.620+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.621717+0000) 2020-02-06T01:51:36.623 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:36.621+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.622310+0000) 2020-02-06T01:51:36.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:36.621+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.622724+0000) 2020-02-06T01:51:36.624 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:36.622+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.623743+0000) 2020-02-06T01:51:36.625 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:36.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.624548+0000) 2020-02-06T01:51:36.626 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:36.624+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.625256+0000) 2020-02-06T01:51:36.626 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:36.624+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.625667+0000) 2020-02-06T01:51:36.626 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:36.624+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.626025+0000) 2020-02-06T01:51:36.627 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:36.625+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.626850+0000) 2020-02-06T01:51:36.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:36.625+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.626649+0000) 2020-02-06T01:51:36.628 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:36.626+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.627155+0000) 2020-02-06T01:51:36.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:36.625+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.626790+0000) 2020-02-06T01:51:36.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:36.627+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.628765+0000) 2020-02-06T01:51:36.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:36.631+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.633005+0000) 2020-02-06T01:51:36.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:36.642+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:36.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:36.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:36.731382+0000) 2020-02-06T01:51:37.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:37.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.122932+0000) 2020-02-06T01:51:37.396 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:37.394+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.395468+0000) 2020-02-06T01:51:37.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:37.394+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.395518+0000) 2020-02-06T01:51:37.397 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:37.394+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.395824+0000) 2020-02-06T01:51:37.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:37.421+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.422525+0000) 2020-02-06T01:51:37.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:37.626+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.627414+0000) 2020-02-06T01:51:37.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:37.650+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:37.651440+0000) 2020-02-06T01:51:37.968 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:51:37.975 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:51:39.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:39.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:39.766621+0000) 2020-02-06T01:51:40.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:40.011+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:40.012535+0000) 2020-02-06T01:51:40.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:40.620+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:40.622230+0000) 2020-02-06T01:51:40.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:40.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:40.901310+0000) 2020-02-06T01:51:41.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:41.649+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.650139+0000) 2020-02-06T01:51:41.651 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:41.648+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.650361+0000) 2020-02-06T01:51:41.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:41.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.651364+0000) 2020-02-06T01:51:41.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:41.650+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.651639+0000) 2020-02-06T01:51:41.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:41.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.651953+0000) 2020-02-06T01:51:41.653 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:41.650+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.652358+0000) 2020-02-06T01:51:41.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:41.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.652839+0000) 2020-02-06T01:51:41.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:41.651+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.653394+0000) 2020-02-06T01:51:41.654 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:41.652+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.653676+0000) 2020-02-06T01:51:41.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:41.652+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.654000+0000) 2020-02-06T01:51:41.655 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:41.652+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.654247+0000) 2020-02-06T01:51:41.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:41.652+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.653414+0000) 2020-02-06T01:51:41.656 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:41.653+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.654453+0000) 2020-02-06T01:51:41.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:41.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:41.660028+0000) 2020-02-06T01:51:41.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:41.672+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:46.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:46.142+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.143407+0000) 2020-02-06T01:51:46.507 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:46.506+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.506680+0000) 2020-02-06T01:51:46.680 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:46.678+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.679139+0000) 2020-02-06T01:51:46.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:46.678+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.679942+0000) 2020-02-06T01:51:46.681 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:46.679+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.680518+0000) 2020-02-06T01:51:46.682 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:46.679+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.681251+0000) 2020-02-06T01:51:46.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:46.680+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.681559+0000) 2020-02-06T01:51:46.684 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:46.681+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.683357+0000) 2020-02-06T01:51:46.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:46.682+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.684027+0000) 2020-02-06T01:51:46.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:46.682+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.684384+0000) 2020-02-06T01:51:46.686 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:46.684+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.685250+0000) 2020-02-06T01:51:46.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:46.684+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.685886+0000) 2020-02-06T01:51:46.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:46.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.686079+0000) 2020-02-06T01:51:46.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:46.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.688426+0000) 2020-02-06T01:51:46.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:46.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.688670+0000) 2020-02-06T01:51:46.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:46.688+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.689699+0000) 2020-02-06T01:51:46.703 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:46.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:46.732 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:46.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:46.731642+0000) 2020-02-06T01:51:47.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:47.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.123177+0000) 2020-02-06T01:51:47.397 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:47.395+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.395687+0000) 2020-02-06T01:51:47.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:47.395+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.395767+0000) 2020-02-06T01:51:47.397 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:47.395+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.395985+0000) 2020-02-06T01:51:47.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:47.429+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.430341+0000) 2020-02-06T01:51:47.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:47.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.627645+0000) 2020-02-06T01:51:47.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:47.651+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:47.651675+0000) 2020-02-06T01:51:49.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:49.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:49.766857+0000) 2020-02-06T01:51:50.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:50.012+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:50.012763+0000) 2020-02-06T01:51:50.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:50.622+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:50.622475+0000) 2020-02-06T01:51:50.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:50.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:50.901528+0000) 2020-02-06T01:51:51.709 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:51.707+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.708010+0000) 2020-02-06T01:51:51.709 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:51.707+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.708169+0000) 2020-02-06T01:51:51.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:51.707+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.708581+0000) 2020-02-06T01:51:51.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:51.707+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.708992+0000) 2020-02-06T01:51:51.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:51.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.710448+0000) 2020-02-06T01:51:51.712 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:51.710+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.711524+0000) 2020-02-06T01:51:51.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:51.710+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.711665+0000) 2020-02-06T01:51:51.713 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:51.711+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.712343+0000) 2020-02-06T01:51:51.713 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:51.712+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.712687+0000) 2020-02-06T01:51:51.714 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:51.713+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.713458+0000) 2020-02-06T01:51:51.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:51.713+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.714142+0000) 2020-02-06T01:51:51.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:51.713+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.714189+0000) 2020-02-06T01:51:51.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:51.714+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.714570+0000) 2020-02-06T01:51:51.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:51.715+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:51.716713+0000) 2020-02-06T01:51:51.732 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:51.730+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:56.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:56.143+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.143633+0000) 2020-02-06T01:51:56.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:56.506+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.506937+0000) 2020-02-06T01:51:56.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:56.730+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.731824+0000) 2020-02-06T01:51:56.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:56.735+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.736205+0000) 2020-02-06T01:51:56.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:51:56.736+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.737284+0000) 2020-02-06T01:51:56.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:56.736+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.737617+0000) 2020-02-06T01:51:56.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:51:56.738+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.738531+0000) 2020-02-06T01:51:56.739 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:56.737+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.739017+0000) 2020-02-06T01:51:56.740 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:51:56.738+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.739884+0000) 2020-02-06T01:51:56.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:56.739+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.739794+0000) 2020-02-06T01:51:56.741 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:56.739+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.740489+0000) 2020-02-06T01:51:56.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:56.740+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.741131+0000) 2020-02-06T01:51:56.742 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:51:56.741+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.741949+0000) 2020-02-06T01:51:56.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:56.741+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.742114+0000) 2020-02-06T01:51:56.743 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:51:56.742+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.742743+0000) 2020-02-06T01:51:56.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:56.742+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.742751+0000) 2020-02-06T01:51:56.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:56.746+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:56.747325+0000) 2020-02-06T01:51:56.760 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:51:56.758+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:51:57.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:51:57.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.123377+0000) 2020-02-06T01:51:57.397 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:51:57.395+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.395921+0000) 2020-02-06T01:51:57.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:51:57.395+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.395974+0000) 2020-02-06T01:51:57.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:51:57.395+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.396168+0000) 2020-02-06T01:51:57.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:57.437+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.438251+0000) 2020-02-06T01:51:57.628 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:51:57.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.627846+0000) 2020-02-06T01:51:57.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:51:57.650+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:57.651893+0000) 2020-02-06T01:51:59.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:51:59.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:51:59.767125+0000) 2020-02-06T01:52:00.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:00.012+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:00.012991+0000) 2020-02-06T01:52:00.623 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:00.622+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:00.622707+0000) 2020-02-06T01:52:00.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:00.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:00.901765+0000) 2020-02-06T01:52:01.765 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:01.763+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.764446+0000) 2020-02-06T01:52:01.766 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:01.764+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.765362+0000) 2020-02-06T01:52:01.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:01.764+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.765917+0000) 2020-02-06T01:52:01.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:01.765+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.766545+0000) 2020-02-06T01:52:01.767 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:01.766+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.766871+0000) 2020-02-06T01:52:01.768 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:01.766+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.767330+0000) 2020-02-06T01:52:01.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:01.767+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.767565+0000) 2020-02-06T01:52:01.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:01.766+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.767916+0000) 2020-02-06T01:52:01.769 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:01.767+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.768357+0000) 2020-02-06T01:52:01.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:01.767+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.768339+0000) 2020-02-06T01:52:01.770 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:01.769+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.770122+0000) 2020-02-06T01:52:01.771 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:01.770+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.770873+0000) 2020-02-06T01:52:01.772 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:01.769+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.770917+0000) 2020-02-06T01:52:01.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:01.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:01.774718+0000) 2020-02-06T01:52:01.788 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:01.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:06.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:06.143+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.143867+0000) 2020-02-06T01:52:06.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:06.506+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.507183+0000) 2020-02-06T01:52:06.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:06.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.732122+0000) 2020-02-06T01:52:06.793 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:06.792+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.792422+0000) 2020-02-06T01:52:06.794 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:06.793+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.793910+0000) 2020-02-06T01:52:06.795 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:06.794+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.794618+0000) 2020-02-06T01:52:06.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:06.794+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.794777+0000) 2020-02-06T01:52:06.796 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:06.795+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.795421+0000) 2020-02-06T01:52:06.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:06.796+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.796747+0000) 2020-02-06T01:52:06.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:06.795+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.796407+0000) 2020-02-06T01:52:06.798 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:06.797+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.797586+0000) 2020-02-06T01:52:06.799 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:06.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.798245+0000) 2020-02-06T01:52:06.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:06.798+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.798504+0000) 2020-02-06T01:52:06.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:06.798+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.799041+0000) 2020-02-06T01:52:06.800 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:06.798+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.799202+0000) 2020-02-06T01:52:06.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:06.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.802601+0000) 2020-02-06T01:52:06.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:06.802+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:06.802838+0000) 2020-02-06T01:52:06.817 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:06.815+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215942 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:07.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:07.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.123591+0000) 2020-02-06T01:52:07.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:07.395+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.396198+0000) 2020-02-06T01:52:07.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:07.395+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.396167+0000) 2020-02-06T01:52:07.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:07.395+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.396374+0000) 2020-02-06T01:52:07.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:07.446+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.446664+0000) 2020-02-06T01:52:07.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:07.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.628054+0000) 2020-02-06T01:52:07.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:07.652+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:07.652136+0000) 2020-02-06T01:52:08.122 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:52:08.129 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:52:09.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:09.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:09.767356+0000) 2020-02-06T01:52:10.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:10.012+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:10.013217+0000) 2020-02-06T01:52:10.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:10.622+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:10.622987+0000) 2020-02-06T01:52:10.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:10.900+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:10.902003+0000) 2020-02-06T01:52:11.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:11.821+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.822303+0000) 2020-02-06T01:52:11.824 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:11.821+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.822725+0000) 2020-02-06T01:52:11.825 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:11.823+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.823817+0000) 2020-02-06T01:52:11.825 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:11.823+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.824565+0000) 2020-02-06T01:52:11.826 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:11.823+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.824965+0000) 2020-02-06T01:52:11.826 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:11.825+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.825771+0000) 2020-02-06T01:52:11.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:11.825+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.826046+0000) 2020-02-06T01:52:11.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:11.825+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.826168+0000) 2020-02-06T01:52:11.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:11.826+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.826594+0000) 2020-02-06T01:52:11.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:11.826+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.826716+0000) 2020-02-06T01:52:11.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:11.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.827322+0000) 2020-02-06T01:52:11.829 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:11.827+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.827807+0000) 2020-02-06T01:52:11.829 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:11.828+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.828445+0000) 2020-02-06T01:52:11.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:11.828+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:11.829873+0000) 2020-02-06T01:52:11.846 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:11.845+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 215990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:16.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:16.143+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.144104+0000) 2020-02-06T01:52:16.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:16.506+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.507403+0000) 2020-02-06T01:52:16.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:16.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.732362+0000) 2020-02-06T01:52:16.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:16.851+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.852324+0000) 2020-02-06T01:52:16.860 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:16.853+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.854230+0000) 2020-02-06T01:52:16.860 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:16.854+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.854780+0000) 2020-02-06T01:52:16.860 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:16.854+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.854854+0000) 2020-02-06T01:52:16.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:16.855+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.855956+0000) 2020-02-06T01:52:16.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:16.856+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.856619+0000) 2020-02-06T01:52:16.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:16.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.851248+0000) 2020-02-06T01:52:16.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:16.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.853333+0000) 2020-02-06T01:52:16.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:16.858+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.859354+0000) 2020-02-06T01:52:16.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:16.851+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.852494+0000) 2020-02-06T01:52:16.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:16.852+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.853575+0000) 2020-02-06T01:52:16.862 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:16.852+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.853892+0000) 2020-02-06T01:52:16.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:16.853+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.854544+0000) 2020-02-06T01:52:16.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:16.854+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:16.856030+0000) 2020-02-06T01:52:16.877 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:16.875+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:17.124 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:17.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.123795+0000) 2020-02-06T01:52:17.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:17.396+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.396420+0000) 2020-02-06T01:52:17.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:17.396+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.396416+0000) 2020-02-06T01:52:17.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:17.396+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.396617+0000) 2020-02-06T01:52:17.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:17.453+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.454783+0000) 2020-02-06T01:52:17.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:17.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.628301+0000) 2020-02-06T01:52:17.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:17.651+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:17.652344+0000) 2020-02-06T01:52:19.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:19.766+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:19.767587+0000) 2020-02-06T01:52:20.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:20.012+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:20.013441+0000) 2020-02-06T01:52:20.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:20.623+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:20.623238+0000) 2020-02-06T01:52:20.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:20.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:20.902272+0000) 2020-02-06T01:52:21.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:21.880+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.881472+0000) 2020-02-06T01:52:21.883 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:21.881+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.882528+0000) 2020-02-06T01:52:21.883 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:21.881+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.882905+0000) 2020-02-06T01:52:21.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:21.882+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.883320+0000) 2020-02-06T01:52:21.884 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:21.883+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.883613+0000) 2020-02-06T01:52:21.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:21.882+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.883773+0000) 2020-02-06T01:52:21.885 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:21.883+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.884134+0000) 2020-02-06T01:52:21.885 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:21.884+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.884543+0000) 2020-02-06T01:52:21.886 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:21.884+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.884822+0000) 2020-02-06T01:52:21.886 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:21.884+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.885514+0000) 2020-02-06T01:52:21.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:21.884+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.885955+0000) 2020-02-06T01:52:21.887 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:21.885+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.886000+0000) 2020-02-06T01:52:21.887 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:21.886+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.886418+0000) 2020-02-06T01:52:21.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:21.889+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:21.890954+0000) 2020-02-06T01:52:21.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:21.903+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216072 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:26.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:26.144+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.144493+0000) 2020-02-06T01:52:26.513 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:26.507+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.507670+0000) 2020-02-06T01:52:26.733 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:26.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.732593+0000) 2020-02-06T01:52:26.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:26.907+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.908879+0000) 2020-02-06T01:52:26.911 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:26.909+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.910931+0000) 2020-02-06T01:52:26.912 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:26.911+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.911598+0000) 2020-02-06T01:52:26.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:26.911+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.912361+0000) 2020-02-06T01:52:26.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:26.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.912807+0000) 2020-02-06T01:52:26.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:26.912+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.913231+0000) 2020-02-06T01:52:26.914 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:26.912+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.913240+0000) 2020-02-06T01:52:26.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:26.912+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.913499+0000) 2020-02-06T01:52:26.915 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:26.913+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.913553+0000) 2020-02-06T01:52:26.915 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:26.913+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.914033+0000) 2020-02-06T01:52:26.916 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:26.913+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.914305+0000) 2020-02-06T01:52:26.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:26.913+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.914328+0000) 2020-02-06T01:52:26.916 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:26.915+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.915964+0000) 2020-02-06T01:52:26.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:26.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:26.919920+0000) 2020-02-06T01:52:26.933 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:26.932+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:27.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:27.122+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.124031+0000) 2020-02-06T01:52:27.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:27.396+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.396641+0000) 2020-02-06T01:52:27.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:27.396+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.396756+0000) 2020-02-06T01:52:27.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:27.396+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.396870+0000) 2020-02-06T01:52:27.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:27.461+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.462663+0000) 2020-02-06T01:52:27.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:27.627+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.628526+0000) 2020-02-06T01:52:27.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:27.651+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:27.652573+0000) 2020-02-06T01:52:29.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:29.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:29.767819+0000) 2020-02-06T01:52:30.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:30.013+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:30.013677+0000) 2020-02-06T01:52:30.624 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:30.622+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:30.623515+0000) 2020-02-06T01:52:30.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:30.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:30.902507+0000) 2020-02-06T01:52:31.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:31.936+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.937975+0000) 2020-02-06T01:52:31.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:31.937+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.938508+0000) 2020-02-06T01:52:31.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:31.938+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.939246+0000) 2020-02-06T01:52:31.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:31.938+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.939687+0000) 2020-02-06T01:52:31.941 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:31.938+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.939433+0000) 2020-02-06T01:52:31.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:31.938+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.940019+0000) 2020-02-06T01:52:31.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:31.940+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.941848+0000) 2020-02-06T01:52:31.943 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:31.940+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.941743+0000) 2020-02-06T01:52:31.943 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:31.940+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.942408+0000) 2020-02-06T01:52:31.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:31.941+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.942694+0000) 2020-02-06T01:52:31.944 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:31.941+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.942562+0000) 2020-02-06T01:52:31.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:31.942+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.943509+0000) 2020-02-06T01:52:31.945 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:31.942+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.943926+0000) 2020-02-06T01:52:31.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:31.948+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:31.950064+0000) 2020-02-06T01:52:31.962 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:31.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216154 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:36.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:36.143+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.144749+0000) 2020-02-06T01:52:36.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:36.506+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.507920+0000) 2020-02-06T01:52:36.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:36.731+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.732941+0000) 2020-02-06T01:52:36.968 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:36.966+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.967428+0000) 2020-02-06T01:52:36.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:36.967+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.968144+0000) 2020-02-06T01:52:36.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:36.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.968939+0000) 2020-02-06T01:52:36.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:36.968+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.969630+0000) 2020-02-06T01:52:36.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:36.968+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.969878+0000) 2020-02-06T01:52:36.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:36.969+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.970296+0000) 2020-02-06T01:52:36.971 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:36.969+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.970489+0000) 2020-02-06T01:52:36.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:36.969+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.970891+0000) 2020-02-06T01:52:36.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:36.970+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.971934+0000) 2020-02-06T01:52:36.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:36.970+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.971913+0000) 2020-02-06T01:52:36.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:36.971+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.972440+0000) 2020-02-06T01:52:36.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:36.971+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.973340+0000) 2020-02-06T01:52:36.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:36.972+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.973538+0000) 2020-02-06T01:52:36.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:36.976+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:36.977238+0000) 2020-02-06T01:52:36.990 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:36.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:37.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:37.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.124268+0000) 2020-02-06T01:52:37.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:37.395+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.396863+0000) 2020-02-06T01:52:37.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:37.395+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.396997+0000) 2020-02-06T01:52:37.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:37.395+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.397126+0000) 2020-02-06T01:52:37.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:37.471+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.471643+0000) 2020-02-06T01:52:37.629 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:37.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.628795+0000) 2020-02-06T01:52:37.653 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:37.651+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:37.652839+0000) 2020-02-06T01:52:38.293 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:52:38.300 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:52:39.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:39.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:39.768135+0000) 2020-02-06T01:52:40.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:40.013+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:40.013913+0000) 2020-02-06T01:52:40.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:40.622+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:40.623769+0000) 2020-02-06T01:52:40.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:40.901+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:40.902755+0000) 2020-02-06T01:52:41.995 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:41.993+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.994674+0000) 2020-02-06T01:52:41.996 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:41.994+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.995148+0000) 2020-02-06T01:52:41.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:41.994+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.995519+0000) 2020-02-06T01:52:41.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:41.994+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.995645+0000) 2020-02-06T01:52:41.997 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:41.995+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.996103+0000) 2020-02-06T01:52:41.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:41.995+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.996576+0000) 2020-02-06T01:52:41.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:41.995+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.996937+0000) 2020-02-06T01:52:41.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:41.996+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.997299+0000) 2020-02-06T01:52:41.999 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:41.996+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.997949+0000) 2020-02-06T01:52:41.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:41.996+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.998001+0000) 2020-02-06T01:52:41.999 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:41.996+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.998283+0000) 2020-02-06T01:52:42.000 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:41.997+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:41.998850+0000) 2020-02-06T01:52:42.001 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:41.998+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:42.000378+0000) 2020-02-06T01:52:42.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:42.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:42.004091+0000) 2020-02-06T01:52:42.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:42.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216236 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:46.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:46.144+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:46.145007+0000) 2020-02-06T01:52:46.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:46.507+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:46.508177+0000) 2020-02-06T01:52:46.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:46.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:46.733216+0000) 2020-02-06T01:52:47.024 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:47.021+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.022767+0000) 2020-02-06T01:52:47.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:47.022+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.023566+0000) 2020-02-06T01:52:47.024 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:47.022+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.023661+0000) 2020-02-06T01:52:47.025 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:47.022+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.024027+0000) 2020-02-06T01:52:47.025 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:47.022+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.024356+0000) 2020-02-06T01:52:47.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:47.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.025048+0000) 2020-02-06T01:52:47.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:47.024+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.025380+0000) 2020-02-06T01:52:47.026 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:47.023+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.025277+0000) 2020-02-06T01:52:47.027 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:47.025+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.026942+0000) 2020-02-06T01:52:47.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:47.026+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.027595+0000) 2020-02-06T01:52:47.028 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:47.026+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.027740+0000) 2020-02-06T01:52:47.029 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:47.026+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.027939+0000) 2020-02-06T01:52:47.029 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:47.026+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.028335+0000) 2020-02-06T01:52:47.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:47.033+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.034305+0000) 2020-02-06T01:52:47.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:47.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216271 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:47.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:47.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.124490+0000) 2020-02-06T01:52:47.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:47.396+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.397107+0000) 2020-02-06T01:52:47.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:47.396+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.397193+0000) 2020-02-06T01:52:47.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:47.396+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.397342+0000) 2020-02-06T01:52:47.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:47.479+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.479960+0000) 2020-02-06T01:52:47.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:47.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.629002+0000) 2020-02-06T01:52:47.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:47.652+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:47.653051+0000) 2020-02-06T01:52:49.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:49.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:49.768427+0000) 2020-02-06T01:52:50.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:50.013+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:50.014144+0000) 2020-02-06T01:52:50.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:50.623+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:50.624002+0000) 2020-02-06T01:52:50.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:50.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:50.902987+0000) 2020-02-06T01:52:52.052 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:52.050+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.051498+0000) 2020-02-06T01:52:52.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:52.051+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.052308+0000) 2020-02-06T01:52:52.054 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:52.052+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.053181+0000) 2020-02-06T01:52:52.054 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:52.052+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.053727+0000) 2020-02-06T01:52:52.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:52.053+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.054581+0000) 2020-02-06T01:52:52.055 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:52.053+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.054817+0000) 2020-02-06T01:52:52.056 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:52.053+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.054791+0000) 2020-02-06T01:52:52.056 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:52.054+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.056035+0000) 2020-02-06T01:52:52.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:52.054+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.056180+0000) 2020-02-06T01:52:52.057 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:52.054+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.056335+0000) 2020-02-06T01:52:52.058 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:52.055+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.056764+0000) 2020-02-06T01:52:52.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:52.055+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.057041+0000) 2020-02-06T01:52:52.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:52.056+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.057559+0000) 2020-02-06T01:52:52.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:52.060+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:52.061275+0000) 2020-02-06T01:52:52.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:52.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:56.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:56.144+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:56.145270+0000) 2020-02-06T01:52:56.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:56.507+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:56.508458+0000) 2020-02-06T01:52:56.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:56.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:56.733431+0000) 2020-02-06T01:52:57.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:57.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.081336+0000) 2020-02-06T01:52:57.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:57.080+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.081683+0000) 2020-02-06T01:52:57.083 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:57.080+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.081985+0000) 2020-02-06T01:52:57.084 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:57.081+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.082393+0000) 2020-02-06T01:52:57.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:57.081+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.082502+0000) 2020-02-06T01:52:57.084 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:57.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.082621+0000) 2020-02-06T01:52:57.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:52:57.081+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.082986+0000) 2020-02-06T01:52:57.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:52:57.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.083301+0000) 2020-02-06T01:52:57.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:52:57.082+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.083909+0000) 2020-02-06T01:52:57.086 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:57.083+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.084668+0000) 2020-02-06T01:52:57.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:57.083+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.084728+0000) 2020-02-06T01:52:57.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:52:57.084+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.086099+0000) 2020-02-06T01:52:57.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:52:57.085+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.086464+0000) 2020-02-06T01:52:57.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:57.088+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.089561+0000) 2020-02-06T01:52:57.104 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:52:57.101+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216352 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:52:57.125 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:52:57.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.124735+0000) 2020-02-06T01:52:57.398 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:52:57.396+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.397323+0000) 2020-02-06T01:52:57.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:52:57.396+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.397416+0000) 2020-02-06T01:52:57.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:52:57.396+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.397645+0000) 2020-02-06T01:52:57.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:57.488+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.488192+0000) 2020-02-06T01:52:57.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:52:57.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.629310+0000) 2020-02-06T01:52:57.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:52:57.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:57.653320+0000) 2020-02-06T01:52:59.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:52:59.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:52:59.768681+0000) 2020-02-06T01:53:00.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:00.014+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:00.014368+0000) 2020-02-06T01:53:00.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:00.623+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:00.624280+0000) 2020-02-06T01:53:00.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:00.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:00.903190+0000) 2020-02-06T01:53:02.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:02.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.109377+0000) 2020-02-06T01:53:02.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:02.109+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.109839+0000) 2020-02-06T01:53:02.111 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:02.110+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.110226+0000) 2020-02-06T01:53:02.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:02.110+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.110720+0000) 2020-02-06T01:53:02.112 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:02.109+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.111354+0000) 2020-02-06T01:53:02.112 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:02.111+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.111709+0000) 2020-02-06T01:53:02.113 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:02.110+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.112195+0000) 2020-02-06T01:53:02.115 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:02.114+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.114241+0000) 2020-02-06T01:53:02.115 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:02.113+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.113776+0000) 2020-02-06T01:53:02.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:02.113+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.114631+0000) 2020-02-06T01:53:02.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:02.113+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.114716+0000) 2020-02-06T01:53:02.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:02.113+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.115109+0000) 2020-02-06T01:53:02.117 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:02.113+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.115233+0000) 2020-02-06T01:53:02.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:02.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:02.116970+0000) 2020-02-06T01:53:02.132 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:02.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216399 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:06.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:06.144+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:06.145527+0000) 2020-02-06T01:53:06.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:06.508+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:06.508705+0000) 2020-02-06T01:53:06.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:06.732+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:06.733684+0000) 2020-02-06T01:53:07.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:07.123+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.124978+0000) 2020-02-06T01:53:07.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:07.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.136638+0000) 2020-02-06T01:53:07.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:07.136+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.137877+0000) 2020-02-06T01:53:07.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:07.136+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.138186+0000) 2020-02-06T01:53:07.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:07.137+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.138755+0000) 2020-02-06T01:53:07.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:07.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.139276+0000) 2020-02-06T01:53:07.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:07.138+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.139466+0000) 2020-02-06T01:53:07.141 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:07.138+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.139836+0000) 2020-02-06T01:53:07.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:07.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.139868+0000) 2020-02-06T01:53:07.141 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:07.138+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.140178+0000) 2020-02-06T01:53:07.142 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:07.139+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.140631+0000) 2020-02-06T01:53:07.142 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:07.140+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.141334+0000) 2020-02-06T01:53:07.142 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:07.139+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.141032+0000) 2020-02-06T01:53:07.143 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:07.140+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.142103+0000) 2020-02-06T01:53:07.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:07.146+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.147171+0000) 2020-02-06T01:53:07.162 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:07.160+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:07.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:07.396+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.397554+0000) 2020-02-06T01:53:07.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:07.396+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.397655+0000) 2020-02-06T01:53:07.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:07.396+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.397953+0000) 2020-02-06T01:53:07.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:07.495+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.496307+0000) 2020-02-06T01:53:07.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:07.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.629529+0000) 2020-02-06T01:53:07.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:07.652+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:07.653547+0000) 2020-02-06T01:53:08.473 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:53:08.480 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:53:09.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:09.767+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:09.768897+0000) 2020-02-06T01:53:10.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:10.013+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:10.014600+0000) 2020-02-06T01:53:10.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:10.623+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:10.624529+0000) 2020-02-06T01:53:10.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:10.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:10.903426+0000) 2020-02-06T01:53:12.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:12.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.167887+0000) 2020-02-06T01:53:12.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:12.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.168378+0000) 2020-02-06T01:53:12.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:12.167+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.169176+0000) 2020-02-06T01:53:12.170 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:12.168+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.169833+0000) 2020-02-06T01:53:12.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:12.168+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.170289+0000) 2020-02-06T01:53:12.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:12.169+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.170488+0000) 2020-02-06T01:53:12.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:12.169+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.171009+0000) 2020-02-06T01:53:12.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:12.170+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.171619+0000) 2020-02-06T01:53:12.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:12.171+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.172431+0000) 2020-02-06T01:53:12.174 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:12.170+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.172081+0000) 2020-02-06T01:53:12.174 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:12.171+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.172992+0000) 2020-02-06T01:53:12.174 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:12.172+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.173860+0000) 2020-02-06T01:53:12.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:12.173+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.174207+0000) 2020-02-06T01:53:12.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:12.175+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:12.176521+0000) 2020-02-06T01:53:12.191 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:12.189+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216477 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:16.146 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:16.145+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:16.145758+0000) 2020-02-06T01:53:16.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:16.508+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:16.508980+0000) 2020-02-06T01:53:16.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:16.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:16.733953+0000) 2020-02-06T01:53:17.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:17.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.125294+0000) 2020-02-06T01:53:17.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:17.194+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.196277+0000) 2020-02-06T01:53:17.198 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:17.196+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.197744+0000) 2020-02-06T01:53:17.199 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:17.197+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.198755+0000) 2020-02-06T01:53:17.200 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:17.197+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.199184+0000) 2020-02-06T01:53:17.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:17.198+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.199896+0000) 2020-02-06T01:53:17.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:17.198+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.199971+0000) 2020-02-06T01:53:17.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:17.199+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.200681+0000) 2020-02-06T01:53:17.201 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:17.199+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.200356+0000) 2020-02-06T01:53:17.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:17.200+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.201069+0000) 2020-02-06T01:53:17.202 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:17.200+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.201475+0000) 2020-02-06T01:53:17.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:17.200+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.202138+0000) 2020-02-06T01:53:17.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:17.201+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.202439+0000) 2020-02-06T01:53:17.203 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:17.201+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.202607+0000) 2020-02-06T01:53:17.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:17.206+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.207749+0000) 2020-02-06T01:53:17.220 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:17.217+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216511 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:17.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:17.397+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.397759+0000) 2020-02-06T01:53:17.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:17.397+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.397875+0000) 2020-02-06T01:53:17.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:17.397+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.398177+0000) 2020-02-06T01:53:17.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:17.503+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.504287+0000) 2020-02-06T01:53:17.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:17.628+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.629783+0000) 2020-02-06T01:53:17.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:17.652+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:17.653768+0000) 2020-02-06T01:53:19.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:19.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:19.769121+0000) 2020-02-06T01:53:20.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:20.014+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:20.014831+0000) 2020-02-06T01:53:20.626 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:20.624+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:20.624774+0000) 2020-02-06T01:53:20.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:20.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:20.903661+0000) 2020-02-06T01:53:22.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:22.223+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.223905+0000) 2020-02-06T01:53:22.225 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:22.223+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.224164+0000) 2020-02-06T01:53:22.227 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:22.225+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.226461+0000) 2020-02-06T01:53:22.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:22.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.227501+0000) 2020-02-06T01:53:22.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:22.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.228822+0000) 2020-02-06T01:53:22.230 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:22.228+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.228982+0000) 2020-02-06T01:53:22.230 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:22.229+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.229669+0000) 2020-02-06T01:53:22.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:22.229+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.230166+0000) 2020-02-06T01:53:22.232 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:22.229+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.230325+0000) 2020-02-06T01:53:22.232 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:22.229+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.230620+0000) 2020-02-06T01:53:22.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:22.228+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.229402+0000) 2020-02-06T01:53:22.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:22.229+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.230326+0000) 2020-02-06T01:53:22.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:22.230+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.230766+0000) 2020-02-06T01:53:22.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:22.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:22.236059+0000) 2020-02-06T01:53:22.248 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:22.247+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216555 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:26.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:26.145+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:26.146015+0000) 2020-02-06T01:53:26.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:26.508+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:26.509219+0000) 2020-02-06T01:53:26.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:26.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:26.734177+0000) 2020-02-06T01:53:27.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:27.124+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.125524+0000) 2020-02-06T01:53:27.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:27.253+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.253506+0000) 2020-02-06T01:53:27.255 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:27.253+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.254391+0000) 2020-02-06T01:53:27.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:27.254+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.255782+0000) 2020-02-06T01:53:27.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:27.255+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.255805+0000) 2020-02-06T01:53:27.257 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:27.255+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.256244+0000) 2020-02-06T01:53:27.258 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:27.255+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.256874+0000) 2020-02-06T01:53:27.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:27.257+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.257794+0000) 2020-02-06T01:53:27.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:27.257+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.258596+0000) 2020-02-06T01:53:27.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:27.258+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.259137+0000) 2020-02-06T01:53:27.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:27.258+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.259072+0000) 2020-02-06T01:53:27.261 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:27.258+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.259345+0000) 2020-02-06T01:53:27.261 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:27.258+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.259488+0000) 2020-02-06T01:53:27.261 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:27.259+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.259913+0000) 2020-02-06T01:53:27.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:27.262+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.263525+0000) 2020-02-06T01:53:27.278 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:27.276+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216591 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:27.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:27.397+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.398030+0000) 2020-02-06T01:53:27.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:27.397+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.398106+0000) 2020-02-06T01:53:27.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:27.397+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.398380+0000) 2020-02-06T01:53:27.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:27.511+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.512020+0000) 2020-02-06T01:53:27.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:27.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.630040+0000) 2020-02-06T01:53:27.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:27.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:27.653980+0000) 2020-02-06T01:53:29.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:29.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:29.769383+0000) 2020-02-06T01:53:30.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:30.014+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:30.015068+0000) 2020-02-06T01:53:30.626 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:30.624+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:30.625031+0000) 2020-02-06T01:53:30.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:30.902+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:30.903848+0000) 2020-02-06T01:53:32.286 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:32.284+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.285316+0000) 2020-02-06T01:53:32.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:32.284+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.285571+0000) 2020-02-06T01:53:32.287 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:32.285+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.286034+0000) 2020-02-06T01:53:32.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:32.285+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.286076+0000) 2020-02-06T01:53:32.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:32.285+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.286424+0000) 2020-02-06T01:53:32.288 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:32.286+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.287139+0000) 2020-02-06T01:53:32.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:32.286+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.287319+0000) 2020-02-06T01:53:32.289 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:32.286+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.287091+0000) 2020-02-06T01:53:32.289 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:32.286+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.287691+0000) 2020-02-06T01:53:32.290 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:32.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.288165+0000) 2020-02-06T01:53:32.290 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:32.287+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.288530+0000) 2020-02-06T01:53:32.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:32.287+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.288181+0000) 2020-02-06T01:53:32.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:32.288+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.288720+0000) 2020-02-06T01:53:32.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:32.293+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:32.294318+0000) 2020-02-06T01:53:32.308 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:32.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:36.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:36.145+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:36.146257+0000) 2020-02-06T01:53:36.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:36.509+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:36.509500+0000) 2020-02-06T01:53:36.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:36.733+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:36.734420+0000) 2020-02-06T01:53:37.126 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:37.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.125757+0000) 2020-02-06T01:53:37.314 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:37.312+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.313317+0000) 2020-02-06T01:53:37.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:37.313+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.314451+0000) 2020-02-06T01:53:37.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:37.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.314813+0000) 2020-02-06T01:53:37.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:37.314+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.315567+0000) 2020-02-06T01:53:37.316 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:37.314+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.315200+0000) 2020-02-06T01:53:37.317 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:37.315+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.316701+0000) 2020-02-06T01:53:37.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:37.316+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.316825+0000) 2020-02-06T01:53:37.318 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:37.317+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.317700+0000) 2020-02-06T01:53:37.318 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:37.317+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.317621+0000) 2020-02-06T01:53:37.319 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:37.317+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.318534+0000) 2020-02-06T01:53:37.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:37.317+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.317787+0000) 2020-02-06T01:53:37.320 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:37.317+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.318089+0000) 2020-02-06T01:53:37.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:37.318+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.319188+0000) 2020-02-06T01:53:37.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:37.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.321829+0000) 2020-02-06T01:53:37.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:37.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216670 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:37.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:37.397+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.398260+0000) 2020-02-06T01:53:37.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:37.397+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.398340+0000) 2020-02-06T01:53:37.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:37.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.398542+0000) 2020-02-06T01:53:37.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:37.519+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.520151+0000) 2020-02-06T01:53:37.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:37.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.630294+0000) 2020-02-06T01:53:37.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:37.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:37.654201+0000) 2020-02-06T01:53:38.620 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:53:38.627 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:53:39.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:39.768+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:39.769629+0000) 2020-02-06T01:53:40.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:40.014+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:40.015373+0000) 2020-02-06T01:53:40.626 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:40.624+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:40.625282+0000) 2020-02-06T01:53:40.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:40.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:40.904142+0000) 2020-02-06T01:53:42.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:42.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.341471+0000) 2020-02-06T01:53:42.343 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:42.341+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.342072+0000) 2020-02-06T01:53:42.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:42.342+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.342634+0000) 2020-02-06T01:53:42.344 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:42.342+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.343293+0000) 2020-02-06T01:53:42.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:42.342+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.343542+0000) 2020-02-06T01:53:42.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:42.343+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.344523+0000) 2020-02-06T01:53:42.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:42.343+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.344207+0000) 2020-02-06T01:53:42.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:42.344+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.345569+0000) 2020-02-06T01:53:42.346 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:42.345+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.345783+0000) 2020-02-06T01:53:42.347 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:42.345+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.346277+0000) 2020-02-06T01:53:42.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:42.346+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.346555+0000) 2020-02-06T01:53:42.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:42.346+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.347119+0000) 2020-02-06T01:53:42.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:42.345+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.346895+0000) 2020-02-06T01:53:42.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:42.347+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:42.348573+0000) 2020-02-06T01:53:42.364 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:42.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216713 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:46.147 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:46.146+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:46.146520+0000) 2020-02-06T01:53:46.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:46.509+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:46.509752+0000) 2020-02-06T01:53:46.735 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:46.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:46.734653+0000) 2020-02-06T01:53:47.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:47.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.125993+0000) 2020-02-06T01:53:47.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:47.367+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.368725+0000) 2020-02-06T01:53:47.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:47.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.369430+0000) 2020-02-06T01:53:47.370 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:47.368+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.369998+0000) 2020-02-06T01:53:47.371 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:47.369+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.370440+0000) 2020-02-06T01:53:47.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:47.369+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.370939+0000) 2020-02-06T01:53:47.372 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:47.370+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.371429+0000) 2020-02-06T01:53:47.373 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:47.371+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.372435+0000) 2020-02-06T01:53:47.373 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:47.371+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.372330+0000) 2020-02-06T01:53:47.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:47.373+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.373683+0000) 2020-02-06T01:53:47.375 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:47.373+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.374106+0000) 2020-02-06T01:53:47.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:47.374+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.374810+0000) 2020-02-06T01:53:47.376 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:47.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.375172+0000) 2020-02-06T01:53:47.376 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:47.375+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.375815+0000) 2020-02-06T01:53:47.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:47.379+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.380832+0000) 2020-02-06T01:53:47.392 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:47.391+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216749 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:47.399 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:47.398+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.398450+0000) 2020-02-06T01:53:47.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:47.398+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.398524+0000) 2020-02-06T01:53:47.399 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:47.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.398705+0000) 2020-02-06T01:53:47.529 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:47.527+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.528131+0000) 2020-02-06T01:53:47.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:47.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.630609+0000) 2020-02-06T01:53:47.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:47.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:47.654352+0000) 2020-02-06T01:53:49.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:49.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:49.769864+0000) 2020-02-06T01:53:50.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:50.015+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:50.015633+0000) 2020-02-06T01:53:50.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:50.625+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:50.625542+0000) 2020-02-06T01:53:50.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:50.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:50.904385+0000) 2020-02-06T01:53:52.397 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:52.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.395880+0000) 2020-02-06T01:53:52.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:52.397+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.397922+0000) 2020-02-06T01:53:52.399 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:52.398+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.398331+0000) 2020-02-06T01:53:52.399 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:52.398+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.398626+0000) 2020-02-06T01:53:52.400 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:52.399+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.399638+0000) 2020-02-06T01:53:52.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:52.400+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.400372+0000) 2020-02-06T01:53:52.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:52.400+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.401028+0000) 2020-02-06T01:53:52.402 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:52.401+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.401772+0000) 2020-02-06T01:53:52.403 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:52.401+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.401421+0000) 2020-02-06T01:53:52.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:52.401+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.402333+0000) 2020-02-06T01:53:52.403 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:52.402+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.402746+0000) 2020-02-06T01:53:52.404 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:52.403+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.403446+0000) 2020-02-06T01:53:52.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:52.403+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.403729+0000) 2020-02-06T01:53:52.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:52.408+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:52.409157+0000) 2020-02-06T01:53:52.421 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:52.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:56.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:56.146+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:56.146732+0000) 2020-02-06T01:53:56.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:56.509+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:56.509997+0000) 2020-02-06T01:53:56.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:56.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:56.734891+0000) 2020-02-06T01:53:57.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:57.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.126318+0000) 2020-02-06T01:53:57.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:57.398+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.398659+0000) 2020-02-06T01:53:57.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:57.398+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.398705+0000) 2020-02-06T01:53:57.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:57.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.398873+0000) 2020-02-06T01:53:57.427 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:53:57.425+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.426803+0000) 2020-02-06T01:53:57.428 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:57.426+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.427245+0000) 2020-02-06T01:53:57.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:53:57.426+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.427877+0000) 2020-02-06T01:53:57.429 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:53:57.427+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.428305+0000) 2020-02-06T01:53:57.429 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:53:57.428+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.428679+0000) 2020-02-06T01:53:57.430 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:57.428+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.429047+0000) 2020-02-06T01:53:57.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:53:57.429+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.429832+0000) 2020-02-06T01:53:57.431 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:53:57.430+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.430664+0000) 2020-02-06T01:53:57.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:53:57.431+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.431488+0000) 2020-02-06T01:53:57.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:53:57.431+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.431564+0000) 2020-02-06T01:53:57.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:53:57.430+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.430755+0000) 2020-02-06T01:53:57.433 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:57.431+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.432643+0000) 2020-02-06T01:53:57.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:57.434+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.435838+0000) 2020-02-06T01:53:57.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:57.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.437025+0000) 2020-02-06T01:53:57.450 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:53:57.449+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216826 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:53:57.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:57.536+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.537654+0000) 2020-02-06T01:53:57.631 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:53:57.629+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.630817+0000) 2020-02-06T01:53:57.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:53:57.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:57.654570+0000) 2020-02-06T01:53:59.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:53:59.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:53:59.770135+0000) 2020-02-06T01:54:00.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:00.014+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:00.015888+0000) 2020-02-06T01:54:00.626 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:00.624+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:00.625801+0000) 2020-02-06T01:54:00.905 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:00.903+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:00.904631+0000) 2020-02-06T01:54:02.456 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:02.454+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.455841+0000) 2020-02-06T01:54:02.457 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:02.455+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.456775+0000) 2020-02-06T01:54:02.458 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:02.456+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.457063+0000) 2020-02-06T01:54:02.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:02.455+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.457363+0000) 2020-02-06T01:54:02.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:02.456+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.457622+0000) 2020-02-06T01:54:02.459 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:02.457+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.458738+0000) 2020-02-06T01:54:02.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:02.457+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.458980+0000) 2020-02-06T01:54:02.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:02.458+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.459420+0000) 2020-02-06T01:54:02.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:02.458+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.459982+0000) 2020-02-06T01:54:02.461 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:02.459+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.460308+0000) 2020-02-06T01:54:02.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:02.459+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.460745+0000) 2020-02-06T01:54:02.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:02.460+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.461668+0000) 2020-02-06T01:54:02.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:02.460+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.461799+0000) 2020-02-06T01:54:02.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:02.465+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:02.466280+0000) 2020-02-06T01:54:02.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:02.477+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216867 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:06.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:06.146+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:06.147015+0000) 2020-02-06T01:54:06.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:06.509+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:06.510235+0000) 2020-02-06T01:54:06.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:06.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:06.735181+0000) 2020-02-06T01:54:07.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:07.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.126545+0000) 2020-02-06T01:54:07.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:07.398+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.398967+0000) 2020-02-06T01:54:07.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:07.398+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.398968+0000) 2020-02-06T01:54:07.400 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:07.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.399268+0000) 2020-02-06T01:54:07.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:07.483+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.484312+0000) 2020-02-06T01:54:07.485 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:07.483+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.484695+0000) 2020-02-06T01:54:07.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:07.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.485554+0000) 2020-02-06T01:54:07.486 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:07.484+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.486017+0000) 2020-02-06T01:54:07.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:07.485+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.486506+0000) 2020-02-06T01:54:07.487 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:07.485+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.486615+0000) 2020-02-06T01:54:07.488 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:07.485+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.487031+0000) 2020-02-06T01:54:07.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:07.485+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.486942+0000) 2020-02-06T01:54:07.488 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:07.486+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.487756+0000) 2020-02-06T01:54:07.489 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:07.487+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.488569+0000) 2020-02-06T01:54:07.489 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:07.487+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.488896+0000) 2020-02-06T01:54:07.490 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:07.488+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.489641+0000) 2020-02-06T01:54:07.490 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:07.488+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.490141+0000) 2020-02-06T01:54:07.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:07.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.495429+0000) 2020-02-06T01:54:07.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:07.504+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:07.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:07.545+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.545912+0000) 2020-02-06T01:54:07.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:07.630+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.631035+0000) 2020-02-06T01:54:07.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:07.653+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:07.654717+0000) 2020-02-06T01:54:08.750 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:54:08.756 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:54:09.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:09.769+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:09.770434+0000) 2020-02-06T01:54:10.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:10.015+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:10.016125+0000) 2020-02-06T01:54:10.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:10.625+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:10.626132+0000) 2020-02-06T01:54:10.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:10.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:10.904864+0000) 2020-02-06T01:54:12.513 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:12.511+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.512767+0000) 2020-02-06T01:54:12.514 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:12.512+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.513584+0000) 2020-02-06T01:54:12.514 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:12.512+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.514041+0000) 2020-02-06T01:54:12.515 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:12.513+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.514536+0000) 2020-02-06T01:54:12.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:12.513+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.514866+0000) 2020-02-06T01:54:12.516 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:12.514+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.515155+0000) 2020-02-06T01:54:12.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:12.514+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.516025+0000) 2020-02-06T01:54:12.517 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:12.515+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.516603+0000) 2020-02-06T01:54:12.517 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:12.515+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.516783+0000) 2020-02-06T01:54:12.518 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:12.516+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.517903+0000) 2020-02-06T01:54:12.519 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:12.516+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.518412+0000) 2020-02-06T01:54:12.519 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:12.517+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.518690+0000) 2020-02-06T01:54:12.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:12.517+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.519384+0000) 2020-02-06T01:54:12.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:12.521+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:12.522843+0000) 2020-02-06T01:54:12.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:12.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:16.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:16.146+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:16.147277+0000) 2020-02-06T01:54:16.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:16.509+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:16.510491+0000) 2020-02-06T01:54:16.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:16.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:16.735434+0000) 2020-02-06T01:54:17.127 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:17.126+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.126779+0000) 2020-02-06T01:54:17.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:17.398+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.399224+0000) 2020-02-06T01:54:17.400 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:17.398+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.399230+0000) 2020-02-06T01:54:17.401 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:17.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.399539+0000) 2020-02-06T01:54:17.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:17.540+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.541142+0000) 2020-02-06T01:54:17.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:17.540+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.541976+0000) 2020-02-06T01:54:17.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:17.541+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.542396+0000) 2020-02-06T01:54:17.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:17.542+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.543262+0000) 2020-02-06T01:54:17.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:17.542+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.543625+0000) 2020-02-06T01:54:17.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:17.543+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.544334+0000) 2020-02-06T01:54:17.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:17.543+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.544575+0000) 2020-02-06T01:54:17.545 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:17.543+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.544818+0000) 2020-02-06T01:54:17.546 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:17.543+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.545395+0000) 2020-02-06T01:54:17.547 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:17.545+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.546364+0000) 2020-02-06T01:54:17.547 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:17.544+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.545818+0000) 2020-02-06T01:54:17.548 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:17.545+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.546839+0000) 2020-02-06T01:54:17.548 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:17.545+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.547334+0000) 2020-02-06T01:54:17.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:17.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.553065+0000) 2020-02-06T01:54:17.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:17.553+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.554132+0000) 2020-02-06T01:54:17.565 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:17.562+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 216984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:17.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:17.630+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.631302+0000) 2020-02-06T01:54:17.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:17.654+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:17.654928+0000) 2020-02-06T01:54:19.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:19.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:19.770665+0000) 2020-02-06T01:54:20.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:20.015+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:20.016398+0000) 2020-02-06T01:54:20.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:20.625+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:20.626364+0000) 2020-02-06T01:54:20.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:20.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:20.905159+0000) 2020-02-06T01:54:22.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:22.569+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.570558+0000) 2020-02-06T01:54:22.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:22.569+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.571325+0000) 2020-02-06T01:54:22.572 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:22.570+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.571696+0000) 2020-02-06T01:54:22.573 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:22.571+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.572389+0000) 2020-02-06T01:54:22.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:22.571+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.572801+0000) 2020-02-06T01:54:22.575 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:22.573+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.574495+0000) 2020-02-06T01:54:22.575 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:22.573+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.574405+0000) 2020-02-06T01:54:22.576 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:22.573+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.574742+0000) 2020-02-06T01:54:22.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:22.574+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.575201+0000) 2020-02-06T01:54:22.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:22.574+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.575562+0000) 2020-02-06T01:54:22.577 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:22.574+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.576133+0000) 2020-02-06T01:54:22.577 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:22.574+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.576287+0000) 2020-02-06T01:54:22.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:22.575+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.577078+0000) 2020-02-06T01:54:22.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:22.579+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:22.580952+0000) 2020-02-06T01:54:22.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:22.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:26.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:26.146+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:26.147555+0000) 2020-02-06T01:54:26.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:26.510+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:26.510742+0000) 2020-02-06T01:54:26.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:26.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:26.735660+0000) 2020-02-06T01:54:27.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:27.125+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.127033+0000) 2020-02-06T01:54:27.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:27.398+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.399483+0000) 2020-02-06T01:54:27.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:27.398+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.399478+0000) 2020-02-06T01:54:27.401 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:27.398+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.399787+0000) 2020-02-06T01:54:27.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:27.558+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.559265+0000) 2020-02-06T01:54:27.602 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:27.599+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.600686+0000) 2020-02-06T01:54:27.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:27.600+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.601105+0000) 2020-02-06T01:54:27.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:27.601+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.602536+0000) 2020-02-06T01:54:27.604 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:27.601+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.603117+0000) 2020-02-06T01:54:27.604 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:27.602+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.603473+0000) 2020-02-06T01:54:27.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:27.602+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.603844+0000) 2020-02-06T01:54:27.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:27.603+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.604492+0000) 2020-02-06T01:54:27.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:27.602+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.604375+0000) 2020-02-06T01:54:27.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:27.603+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.604325+0000) 2020-02-06T01:54:27.608 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:27.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.605581+0000) 2020-02-06T01:54:27.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:27.604+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.605724+0000) 2020-02-06T01:54:27.609 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:27.604+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.605789+0000) 2020-02-06T01:54:27.610 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:27.607+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.608954+0000) 2020-02-06T01:54:27.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:27.609+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.610181+0000) 2020-02-06T01:54:27.625 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:27.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217062 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:27.632 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:27.630+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.631462+0000) 2020-02-06T01:54:27.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:27.654+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:27.655154+0000) 2020-02-06T01:54:29.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:29.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:29.770891+0000) 2020-02-06T01:54:30.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:30.016+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:30.016622+0000) 2020-02-06T01:54:30.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:30.625+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:30.626646+0000) 2020-02-06T01:54:30.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:30.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:30.905432+0000) 2020-02-06T01:54:32.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:32.628+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.629466+0000) 2020-02-06T01:54:32.631 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:32.628+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.629854+0000) 2020-02-06T01:54:32.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:32.628+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.630130+0000) 2020-02-06T01:54:32.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:32.631+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.633015+0000) 2020-02-06T01:54:32.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:32.632+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.633777+0000) 2020-02-06T01:54:32.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:32.633+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.634123+0000) 2020-02-06T01:54:32.635 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:32.633+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.634482+0000) 2020-02-06T01:54:32.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:32.633+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.634862+0000) 2020-02-06T01:54:32.636 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:32.633+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.635162+0000) 2020-02-06T01:54:32.636 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:32.633+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.635333+0000) 2020-02-06T01:54:32.637 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:32.634+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.636033+0000) 2020-02-06T01:54:32.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:32.635+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.636301+0000) 2020-02-06T01:54:32.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:32.635+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.636504+0000) 2020-02-06T01:54:32.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:32.637+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:32.638303+0000) 2020-02-06T01:54:32.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:32.653+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:36.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:36.147+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:36.147820+0000) 2020-02-06T01:54:36.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:36.510+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:36.510993+0000) 2020-02-06T01:54:36.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:36.734+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:36.735906+0000) 2020-02-06T01:54:37.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:37.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.127333+0000) 2020-02-06T01:54:37.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:37.399+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.399732+0000) 2020-02-06T01:54:37.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:37.399+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.399719+0000) 2020-02-06T01:54:37.401 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:37.399+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.400037+0000) 2020-02-06T01:54:37.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:37.566+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.566619+0000) 2020-02-06T01:54:37.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:37.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.631671+0000) 2020-02-06T01:54:37.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:37.654+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.655362+0000) 2020-02-06T01:54:37.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:37.660+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.662152+0000) 2020-02-06T01:54:37.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:37.661+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.662411+0000) 2020-02-06T01:54:37.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:37.661+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.662865+0000) 2020-02-06T01:54:37.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:37.662+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.663130+0000) 2020-02-06T01:54:37.664 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:37.662+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.663682+0000) 2020-02-06T01:54:37.665 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:37.662+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.663713+0000) 2020-02-06T01:54:37.665 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:37.663+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.664290+0000) 2020-02-06T01:54:37.665 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:37.663+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.664687+0000) 2020-02-06T01:54:37.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:37.664+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.665209+0000) 2020-02-06T01:54:37.666 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:37.664+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.665598+0000) 2020-02-06T01:54:37.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:37.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.666181+0000) 2020-02-06T01:54:37.667 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:37.665+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.666498+0000) 2020-02-06T01:54:37.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:37.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.672240+0000) 2020-02-06T01:54:37.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:37.671+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:37.672542+0000) 2020-02-06T01:54:37.685 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:37.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217140 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:38.890 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:54:38.896 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:54:39.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:39.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:39.771126+0000) 2020-02-06T01:54:40.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:40.016+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:40.016859+0000) 2020-02-06T01:54:40.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:40.625+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:40.626876+0000) 2020-02-06T01:54:40.906 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:40.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:40.905663+0000) 2020-02-06T01:54:42.690 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:42.687+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.689250+0000) 2020-02-06T01:54:42.690 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:42.688+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.689959+0000) 2020-02-06T01:54:42.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:42.689+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.690383+0000) 2020-02-06T01:54:42.692 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:42.689+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.691205+0000) 2020-02-06T01:54:42.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:42.689+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.691038+0000) 2020-02-06T01:54:42.692 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:42.690+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.691602+0000) 2020-02-06T01:54:42.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:42.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.691955+0000) 2020-02-06T01:54:42.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:42.691+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.692659+0000) 2020-02-06T01:54:42.694 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:42.692+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.693070+0000) 2020-02-06T01:54:42.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:42.692+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.693296+0000) 2020-02-06T01:54:42.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:42.692+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.693476+0000) 2020-02-06T01:54:42.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:42.692+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.693802+0000) 2020-02-06T01:54:42.695 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:42.692+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.694418+0000) 2020-02-06T01:54:42.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:42.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:42.699313+0000) 2020-02-06T01:54:42.713 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:42.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217180 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:46.149 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:46.147+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:46.148089+0000) 2020-02-06T01:54:46.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:46.510+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:46.511246+0000) 2020-02-06T01:54:46.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:46.736+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:46.736135+0000) 2020-02-06T01:54:47.128 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:47.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.127582+0000) 2020-02-06T01:54:47.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:47.399+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.399995+0000) 2020-02-06T01:54:47.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:47.399+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.399990+0000) 2020-02-06T01:54:47.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:47.399+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.400272+0000) 2020-02-06T01:54:47.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:47.574+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.574610+0000) 2020-02-06T01:54:47.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:47.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.631899+0000) 2020-02-06T01:54:47.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:47.655+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.655637+0000) 2020-02-06T01:54:47.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:47.716+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.717000+0000) 2020-02-06T01:54:47.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:47.716+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.718077+0000) 2020-02-06T01:54:47.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:47.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.718802+0000) 2020-02-06T01:54:47.720 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:47.719+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.719835+0000) 2020-02-06T01:54:47.721 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:47.718+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.719557+0000) 2020-02-06T01:54:47.721 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:47.720+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.720411+0000) 2020-02-06T01:54:47.721 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:47.720+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.720508+0000) 2020-02-06T01:54:47.723 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:47.720+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.721424+0000) 2020-02-06T01:54:47.723 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:47.720+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.721680+0000) 2020-02-06T01:54:47.723 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:47.720+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.721741+0000) 2020-02-06T01:54:47.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:47.720+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.721031+0000) 2020-02-06T01:54:47.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:47.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.721610+0000) 2020-02-06T01:54:47.724 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:47.720+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.722362+0000) 2020-02-06T01:54:47.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:47.727+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:47.727620+0000) 2020-02-06T01:54:47.742 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:47.739+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217218 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:49.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:49.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:49.771355+0000) 2020-02-06T01:54:50.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:50.017+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:50.017124+0000) 2020-02-06T01:54:50.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:50.626+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:50.627131+0000) 2020-02-06T01:54:50.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:50.904+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:50.905934+0000) 2020-02-06T01:54:52.747 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:52.746+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.746881+0000) 2020-02-06T01:54:52.748 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:52.746+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.747461+0000) 2020-02-06T01:54:52.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:52.746+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.747907+0000) 2020-02-06T01:54:52.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:52.748+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.748754+0000) 2020-02-06T01:54:52.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:52.747+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.748897+0000) 2020-02-06T01:54:52.750 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:52.748+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.749299+0000) 2020-02-06T01:54:52.751 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:52.749+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.749560+0000) 2020-02-06T01:54:52.751 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:52.748+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.749880+0000) 2020-02-06T01:54:52.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:52.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.751426+0000) 2020-02-06T01:54:52.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:52.751+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.751655+0000) 2020-02-06T01:54:52.753 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:52.751+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.751879+0000) 2020-02-06T01:54:52.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:52.751+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.752500+0000) 2020-02-06T01:54:52.754 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:52.752+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.752940+0000) 2020-02-06T01:54:52.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:52.756+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:52.757301+0000) 2020-02-06T01:54:52.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:52.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217257 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:56.149 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:56.147+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:56.148330+0000) 2020-02-06T01:54:56.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:56.511+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:56.511540+0000) 2020-02-06T01:54:56.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:56.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:56.736360+0000) 2020-02-06T01:54:57.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:57.126+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.127829+0000) 2020-02-06T01:54:57.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:57.399+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.400250+0000) 2020-02-06T01:54:57.401 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:57.399+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.400243+0000) 2020-02-06T01:54:57.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:57.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.400512+0000) 2020-02-06T01:54:57.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:57.581+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.582732+0000) 2020-02-06T01:54:57.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:57.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.632132+0000) 2020-02-06T01:54:57.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:57.654+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.655868+0000) 2020-02-06T01:54:57.778 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:54:57.775+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.776395+0000) 2020-02-06T01:54:57.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:54:57.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.777407+0000) 2020-02-06T01:54:57.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:54:57.777+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.778388+0000) 2020-02-06T01:54:57.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:54:57.778+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.778791+0000) 2020-02-06T01:54:57.780 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:54:57.778+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.779067+0000) 2020-02-06T01:54:57.781 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:54:57.779+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.780250+0000) 2020-02-06T01:54:57.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:57.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.780871+0000) 2020-02-06T01:54:57.782 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:54:57.780+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.781105+0000) 2020-02-06T01:54:57.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:54:57.781+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.781746+0000) 2020-02-06T01:54:57.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:54:57.781+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.782011+0000) 2020-02-06T01:54:57.783 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:54:57.781+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.782171+0000) 2020-02-06T01:54:57.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:54:57.782+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.782519+0000) 2020-02-06T01:54:57.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:57.781+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.782976+0000) 2020-02-06T01:54:57.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:57.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:57.786146+0000) 2020-02-06T01:54:57.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:54:57.798+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217296 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:54:59.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:54:59.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:54:59.771595+0000) 2020-02-06T01:55:00.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:00.016+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:00.017366+0000) 2020-02-06T01:55:00.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:00.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:00.627372+0000) 2020-02-06T01:55:00.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:00.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:00.906189+0000) 2020-02-06T01:55:02.805 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:02.803+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.804397+0000) 2020-02-06T01:55:02.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:02.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.805219+0000) 2020-02-06T01:55:02.806 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:02.804+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.805925+0000) 2020-02-06T01:55:02.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:02.805+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.806616+0000) 2020-02-06T01:55:02.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:02.805+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.806840+0000) 2020-02-06T01:55:02.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:02.806+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.807145+0000) 2020-02-06T01:55:02.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:02.807+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.808103+0000) 2020-02-06T01:55:02.809 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:02.806+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.807976+0000) 2020-02-06T01:55:02.810 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:02.807+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.808516+0000) 2020-02-06T01:55:02.810 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:02.808+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.809203+0000) 2020-02-06T01:55:02.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:02.808+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.809333+0000) 2020-02-06T01:55:02.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:02.809+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.810071+0000) 2020-02-06T01:55:02.811 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:02.810+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.810892+0000) 2020-02-06T01:55:02.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:02.814+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:02.815488+0000) 2020-02-06T01:55:02.829 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:02.827+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217336 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:06.149 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:06.148+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:06.148581+0000) 2020-02-06T01:55:06.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:06.511+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:06.511819+0000) 2020-02-06T01:55:06.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:06.735+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:06.736631+0000) 2020-02-06T01:55:07.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:07.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.128049+0000) 2020-02-06T01:55:07.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:07.400+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.400493+0000) 2020-02-06T01:55:07.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:07.400+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.400482+0000) 2020-02-06T01:55:07.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:07.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.400750+0000) 2020-02-06T01:55:07.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:07.593+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.594579+0000) 2020-02-06T01:55:07.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:07.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.632359+0000) 2020-02-06T01:55:07.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:07.655+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.656121+0000) 2020-02-06T01:55:07.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:07.832+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.833441+0000) 2020-02-06T01:55:07.836 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:07.834+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.835122+0000) 2020-02-06T01:55:07.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:07.834+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.834803+0000) 2020-02-06T01:55:07.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:07.835+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.836535+0000) 2020-02-06T01:55:07.838 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:07.836+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.836907+0000) 2020-02-06T01:55:07.838 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:07.836+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.837215+0000) 2020-02-06T01:55:07.839 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:07.837+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.838071+0000) 2020-02-06T01:55:07.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:07.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.838497+0000) 2020-02-06T01:55:07.839 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:07.838+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.838842+0000) 2020-02-06T01:55:07.840 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:07.839+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.839530+0000) 2020-02-06T01:55:07.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:07.839+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.839741+0000) 2020-02-06T01:55:07.841 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:07.839+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.840301+0000) 2020-02-06T01:55:07.842 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:07.839+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.839851+0000) 2020-02-06T01:55:07.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:07.843+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:07.844741+0000) 2020-02-06T01:55:07.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:07.856+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217375 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:09.076 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:55:09.083 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:55:09.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:09.770+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:09.771847+0000) 2020-02-06T01:55:10.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:10.016+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:10.017612+0000) 2020-02-06T01:55:10.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:10.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:10.627637+0000) 2020-02-06T01:55:10.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:10.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:10.906412+0000) 2020-02-06T01:55:12.867 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:12.862+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.863490+0000) 2020-02-06T01:55:12.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:12.863+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.864153+0000) 2020-02-06T01:55:12.867 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:12.860+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.861415+0000) 2020-02-06T01:55:12.868 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:12.863+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.863908+0000) 2020-02-06T01:55:12.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:12.864+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.864455+0000) 2020-02-06T01:55:12.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:12.864+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.865011+0000) 2020-02-06T01:55:12.868 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:12.865+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.865918+0000) 2020-02-06T01:55:12.869 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:12.864+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.865966+0000) 2020-02-06T01:55:12.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:12.865+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.866223+0000) 2020-02-06T01:55:12.870 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:12.865+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.866337+0000) 2020-02-06T01:55:12.870 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:12.866+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.867522+0000) 2020-02-06T01:55:12.870 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:12.867+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.868122+0000) 2020-02-06T01:55:12.870 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:12.867+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.867937+0000) 2020-02-06T01:55:12.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:12.872+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:12.873601+0000) 2020-02-06T01:55:12.883 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:12.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217414 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:16.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:16.148+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:16.148841+0000) 2020-02-06T01:55:16.513 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:16.511+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:16.512085+0000) 2020-02-06T01:55:16.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:16.736+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:16.736866+0000) 2020-02-06T01:55:17.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:17.127+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.128311+0000) 2020-02-06T01:55:17.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:17.400+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.400739+0000) 2020-02-06T01:55:17.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:17.400+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.400739+0000) 2020-02-06T01:55:17.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:17.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.401021+0000) 2020-02-06T01:55:17.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:17.601+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.602691+0000) 2020-02-06T01:55:17.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:17.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.632583+0000) 2020-02-06T01:55:17.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:17.655+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.656358+0000) 2020-02-06T01:55:17.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:17.887+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.888170+0000) 2020-02-06T01:55:17.890 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:17.888+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.889343+0000) 2020-02-06T01:55:17.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:17.888+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.890033+0000) 2020-02-06T01:55:17.891 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:17.889+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.890313+0000) 2020-02-06T01:55:17.891 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:17.890+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.891016+0000) 2020-02-06T01:55:17.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:17.890+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.891345+0000) 2020-02-06T01:55:17.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:17.891+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.891581+0000) 2020-02-06T01:55:17.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:17.890+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.892027+0000) 2020-02-06T01:55:17.893 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:17.891+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.892167+0000) 2020-02-06T01:55:17.893 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:17.891+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.892254+0000) 2020-02-06T01:55:17.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:17.892+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.892670+0000) 2020-02-06T01:55:17.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:17.892+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.893171+0000) 2020-02-06T01:55:17.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:17.892+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.893078+0000) 2020-02-06T01:55:17.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:17.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:17.898599+0000) 2020-02-06T01:55:17.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:17.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:19.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:19.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:19.772112+0000) 2020-02-06T01:55:20.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:20.016+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:20.017854+0000) 2020-02-06T01:55:20.628 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:20.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:20.627898+0000) 2020-02-06T01:55:20.907 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:20.905+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:20.906631+0000) 2020-02-06T01:55:22.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:22.916+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.917405+0000) 2020-02-06T01:55:22.918 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:22.916+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.917621+0000) 2020-02-06T01:55:22.919 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:22.917+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.918240+0000) 2020-02-06T01:55:22.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:22.917+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.918809+0000) 2020-02-06T01:55:22.920 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:22.919+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.919680+0000) 2020-02-06T01:55:22.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:22.918+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.919946+0000) 2020-02-06T01:55:22.922 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:22.920+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.921132+0000) 2020-02-06T01:55:22.923 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:22.921+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.922192+0000) 2020-02-06T01:55:22.923 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:22.921+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.921587+0000) 2020-02-06T01:55:22.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:22.922+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.922878+0000) 2020-02-06T01:55:22.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:22.922+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.922920+0000) 2020-02-06T01:55:22.924 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:22.922+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.923770+0000) 2020-02-06T01:55:22.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:22.923+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.923969+0000) 2020-02-06T01:55:22.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:22.926+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:22.927299+0000) 2020-02-06T01:55:22.942 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:22.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217494 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:26.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:26.149+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:26.149091+0000) 2020-02-06T01:55:26.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:26.511+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:26.512379+0000) 2020-02-06T01:55:26.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:26.736+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:26.737135+0000) 2020-02-06T01:55:27.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:27.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.128542+0000) 2020-02-06T01:55:27.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:27.399+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.400997+0000) 2020-02-06T01:55:27.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:27.399+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.400994+0000) 2020-02-06T01:55:27.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:27.399+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.401292+0000) 2020-02-06T01:55:27.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:27.609+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.610877+0000) 2020-02-06T01:55:27.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:27.631+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.632837+0000) 2020-02-06T01:55:27.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:27.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.656616+0000) 2020-02-06T01:55:27.948 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:27.946+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.947151+0000) 2020-02-06T01:55:27.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:27.946+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.948023+0000) 2020-02-06T01:55:27.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:27.947+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.948846+0000) 2020-02-06T01:55:27.950 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:27.947+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.949305+0000) 2020-02-06T01:55:27.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:27.948+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.949618+0000) 2020-02-06T01:55:27.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:27.948+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.949691+0000) 2020-02-06T01:55:27.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:27.948+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.950062+0000) 2020-02-06T01:55:27.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:27.949+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.950385+0000) 2020-02-06T01:55:27.952 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:27.949+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.951018+0000) 2020-02-06T01:55:27.952 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:27.949+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.951278+0000) 2020-02-06T01:55:27.952 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:27.949+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.951377+0000) 2020-02-06T01:55:27.953 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:27.950+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.952080+0000) 2020-02-06T01:55:27.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:27.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.953024+0000) 2020-02-06T01:55:27.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:27.955+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:27.956693+0000) 2020-02-06T01:55:27.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:27.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217533 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:29.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:29.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:29.772316+0000) 2020-02-06T01:55:30.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:30.017+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:30.018121+0000) 2020-02-06T01:55:30.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:30.626+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:30.628134+0000) 2020-02-06T01:55:30.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:30.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:30.906870+0000) 2020-02-06T01:55:32.974 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:32.972+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.973660+0000) 2020-02-06T01:55:32.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:32.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.974988+0000) 2020-02-06T01:55:32.976 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:32.974+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.975562+0000) 2020-02-06T01:55:32.977 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:32.975+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.976872+0000) 2020-02-06T01:55:32.978 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:32.976+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.977686+0000) 2020-02-06T01:55:32.979 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:32.975+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.977410+0000) 2020-02-06T01:55:32.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:32.976+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.978367+0000) 2020-02-06T01:55:32.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:32.977+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.978915+0000) 2020-02-06T01:55:32.980 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:32.978+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.979365+0000) 2020-02-06T01:55:32.980 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:32.978+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.979685+0000) 2020-02-06T01:55:32.986 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:32.980+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.981442+0000) 2020-02-06T01:55:32.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:32.979+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.980899+0000) 2020-02-06T01:55:32.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:32.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.984999+0000) 2020-02-06T01:55:32.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:32.985+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:32.986225+0000) 2020-02-06T01:55:32.999 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:32.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217572 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:36.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:36.148+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:36.149353+0000) 2020-02-06T01:55:36.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:36.511+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:36.512664+0000) 2020-02-06T01:55:36.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:36.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:36.737396+0000) 2020-02-06T01:55:37.129 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:37.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.128775+0000) 2020-02-06T01:55:37.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:37.400+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.401272+0000) 2020-02-06T01:55:37.402 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:37.400+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.401231+0000) 2020-02-06T01:55:37.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:37.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.401517+0000) 2020-02-06T01:55:37.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:37.618+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.619041+0000) 2020-02-06T01:55:37.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:37.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.633125+0000) 2020-02-06T01:55:37.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:37.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:37.656846+0000) 2020-02-06T01:55:38.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:38.003+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.004543+0000) 2020-02-06T01:55:38.007 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:38.004+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.006048+0000) 2020-02-06T01:55:38.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:38.005+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.006483+0000) 2020-02-06T01:55:38.008 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:38.005+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.007011+0000) 2020-02-06T01:55:38.008 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:38.006+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.007542+0000) 2020-02-06T01:55:38.008 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:38.005+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.007368+0000) 2020-02-06T01:55:38.010 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:38.007+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.009307+0000) 2020-02-06T01:55:38.010 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:38.008+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.009684+0000) 2020-02-06T01:55:38.011 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:38.007+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.009387+0000) 2020-02-06T01:55:38.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:38.008+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.010178+0000) 2020-02-06T01:55:38.011 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:38.009+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.010067+0000) 2020-02-06T01:55:38.012 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:38.009+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.010520+0000) 2020-02-06T01:55:38.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:38.013+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.014418+0000) 2020-02-06T01:55:38.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:38.013+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:38.014730+0000) 2020-02-06T01:55:38.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:38.026+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:39.282 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:55:39.290 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:55:39.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:39.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:39.772577+0000) 2020-02-06T01:55:40.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:40.018+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:40.018354+0000) 2020-02-06T01:55:40.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:40.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:40.628365+0000) 2020-02-06T01:55:40.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:40.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:40.907131+0000) 2020-02-06T01:55:43.035 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:43.033+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.033799+0000) 2020-02-06T01:55:43.035 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:43.031+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.033224+0000) 2020-02-06T01:55:43.035 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:43.034+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.034635+0000) 2020-02-06T01:55:43.036 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:43.034+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.035016+0000) 2020-02-06T01:55:43.036 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:43.034+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.035856+0000) 2020-02-06T01:55:43.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:43.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.036258+0000) 2020-02-06T01:55:43.037 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:43.034+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.036199+0000) 2020-02-06T01:55:43.038 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:43.036+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.036655+0000) 2020-02-06T01:55:43.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:43.036+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.037035+0000) 2020-02-06T01:55:43.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:43.037+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.037652+0000) 2020-02-06T01:55:43.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:43.036+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.038310+0000) 2020-02-06T01:55:43.040 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:43.037+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.039131+0000) 2020-02-06T01:55:43.040 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:43.038+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.039719+0000) 2020-02-06T01:55:43.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:43.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:43.046357+0000) 2020-02-06T01:55:43.056 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:43.054+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:46.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:46.148+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:46.149635+0000) 2020-02-06T01:55:46.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:46.512+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:46.512915+0000) 2020-02-06T01:55:46.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:46.736+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:46.737642+0000) 2020-02-06T01:55:47.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:47.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.129012+0000) 2020-02-06T01:55:47.402 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:47.400+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.401515+0000) 2020-02-06T01:55:47.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:47.400+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.401507+0000) 2020-02-06T01:55:47.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:47.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.401764+0000) 2020-02-06T01:55:47.628 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:47.625+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.626999+0000) 2020-02-06T01:55:47.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:47.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.633348+0000) 2020-02-06T01:55:47.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:47.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:47.657072+0000) 2020-02-06T01:55:48.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:48.060+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.061362+0000) 2020-02-06T01:55:48.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:48.061+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.062157+0000) 2020-02-06T01:55:48.063 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:48.061+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.062562+0000) 2020-02-06T01:55:48.063 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:48.061+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.063034+0000) 2020-02-06T01:55:48.064 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:48.063+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.064143+0000) 2020-02-06T01:55:48.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:48.062+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.064051+0000) 2020-02-06T01:55:48.067 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:48.064+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.065778+0000) 2020-02-06T01:55:48.067 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:48.065+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.066402+0000) 2020-02-06T01:55:48.068 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:48.064+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.066065+0000) 2020-02-06T01:55:48.068 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:48.065+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.066765+0000) 2020-02-06T01:55:48.068 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:48.065+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.067043+0000) 2020-02-06T01:55:48.068 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:48.066+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.067477+0000) 2020-02-06T01:55:48.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:48.068+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.069994+0000) 2020-02-06T01:55:48.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:48.069+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:48.070310+0000) 2020-02-06T01:55:48.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:48.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:49.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:49.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:49.772802+0000) 2020-02-06T01:55:50.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:50.017+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:50.018582+0000) 2020-02-06T01:55:50.629 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:50.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:50.628640+0000) 2020-02-06T01:55:50.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:50.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:50.907390+0000) 2020-02-06T01:55:53.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:53.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.089776+0000) 2020-02-06T01:55:53.094 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:53.092+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.093121+0000) 2020-02-06T01:55:53.095 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:53.092+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.093622+0000) 2020-02-06T01:55:53.095 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:53.093+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.094344+0000) 2020-02-06T01:55:53.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:53.093+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.094766+0000) 2020-02-06T01:55:53.096 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:53.093+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.095015+0000) 2020-02-06T01:55:53.096 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:53.094+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.095520+0000) 2020-02-06T01:55:53.096 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:53.094+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.095777+0000) 2020-02-06T01:55:53.097 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:53.094+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.095705+0000) 2020-02-06T01:55:53.097 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:53.094+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.096364+0000) 2020-02-06T01:55:53.097 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:53.095+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.096510+0000) 2020-02-06T01:55:53.098 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:53.095+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.097355+0000) 2020-02-06T01:55:53.098 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:53.096+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.097636+0000) 2020-02-06T01:55:53.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:53.102+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:53.103952+0000) 2020-02-06T01:55:53.116 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:53.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217730 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:56.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:56.149+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:56.149897+0000) 2020-02-06T01:55:56.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:56.512+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:56.513148+0000) 2020-02-06T01:55:56.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:56.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:56.737881+0000) 2020-02-06T01:55:57.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:57.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.129234+0000) 2020-02-06T01:55:57.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:57.400+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.401712+0000) 2020-02-06T01:55:57.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:57.400+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.401750+0000) 2020-02-06T01:55:57.403 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:57.400+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.402058+0000) 2020-02-06T01:55:57.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:57.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.633580+0000) 2020-02-06T01:55:57.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:57.634+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.635583+0000) 2020-02-06T01:55:57.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:57.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:57.657327+0000) 2020-02-06T01:55:58.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:55:58.120+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.122008+0000) 2020-02-06T01:55:58.129 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:55:58.121+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.122810+0000) 2020-02-06T01:55:58.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:55:58.123+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.125149+0000) 2020-02-06T01:55:58.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:55:58.124+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.125721+0000) 2020-02-06T01:55:58.129 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:55:58.124+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.126279+0000) 2020-02-06T01:55:58.130 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:55:58.125+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.126925+0000) 2020-02-06T01:55:58.130 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:55:58.125+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.127381+0000) 2020-02-06T01:55:58.130 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:55:58.121+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.122962+0000) 2020-02-06T01:55:58.130 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:55:58.122+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.123750+0000) 2020-02-06T01:55:58.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:58.123+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.124307+0000) 2020-02-06T01:55:58.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:58.125+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.126958+0000) 2020-02-06T01:55:58.131 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:55:58.123+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.124543+0000) 2020-02-06T01:55:58.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:55:58.126+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.127116+0000) 2020-02-06T01:55:58.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:58.129+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:58.130671+0000) 2020-02-06T01:55:58.144 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:55:58.142+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217773 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:55:59.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:55:59.771+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:55:59.773041+0000) 2020-02-06T01:56:00.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:00.018+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:00.018817+0000) 2020-02-06T01:56:00.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:00.628+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:00.628919+0000) 2020-02-06T01:56:00.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:00.906+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:00.907663+0000) 2020-02-06T01:56:03.151 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:03.149+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.150421+0000) 2020-02-06T01:56:03.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:03.150+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.151193+0000) 2020-02-06T01:56:03.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:03.150+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.151499+0000) 2020-02-06T01:56:03.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:03.150+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.151884+0000) 2020-02-06T01:56:03.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:03.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.152455+0000) 2020-02-06T01:56:03.154 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:03.151+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.153117+0000) 2020-02-06T01:56:03.155 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:03.152+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.153970+0000) 2020-02-06T01:56:03.155 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:03.152+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.154269+0000) 2020-02-06T01:56:03.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:03.153+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.154697+0000) 2020-02-06T01:56:03.156 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:03.153+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.154941+0000) 2020-02-06T01:56:03.156 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:03.154+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.155475+0000) 2020-02-06T01:56:03.157 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:03.153+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.155364+0000) 2020-02-06T01:56:03.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:03.154+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.155905+0000) 2020-02-06T01:56:03.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:03.156+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:03.157955+0000) 2020-02-06T01:56:03.173 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:03.170+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217813 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:06.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:06.148+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:06.150154+0000) 2020-02-06T01:56:06.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:06.512+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:06.513474+0000) 2020-02-06T01:56:06.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:06.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:06.738151+0000) 2020-02-06T01:56:07.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:07.128+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.129417+0000) 2020-02-06T01:56:07.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:07.401+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.401979+0000) 2020-02-06T01:56:07.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:07.401+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.401971+0000) 2020-02-06T01:56:07.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:07.401+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.402304+0000) 2020-02-06T01:56:07.634 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:07.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.633797+0000) 2020-02-06T01:56:07.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:07.642+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.643669+0000) 2020-02-06T01:56:07.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:07.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:07.657510+0000) 2020-02-06T01:56:08.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:08.177+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.178164+0000) 2020-02-06T01:56:08.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:08.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.178607+0000) 2020-02-06T01:56:08.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:08.177+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.178885+0000) 2020-02-06T01:56:08.188 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:08.179+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.181265+0000) 2020-02-06T01:56:08.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:08.180+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.181865+0000) 2020-02-06T01:56:08.188 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:08.181+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.182525+0000) 2020-02-06T01:56:08.188 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:08.181+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.182643+0000) 2020-02-06T01:56:08.189 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:08.181+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.182960+0000) 2020-02-06T01:56:08.189 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:08.181+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.183398+0000) 2020-02-06T01:56:08.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:08.178+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.179703+0000) 2020-02-06T01:56:08.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:08.179+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.180263+0000) 2020-02-06T01:56:08.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:08.182+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.183774+0000) 2020-02-06T01:56:08.190 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:08.183+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.184169+0000) 2020-02-06T01:56:08.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:08.189+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:08.190193+0000) 2020-02-06T01:56:08.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:08.198+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:09.486 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:56:09.492 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:56:09.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:09.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:09.773254+0000) 2020-02-06T01:56:10.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:10.018+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:10.019048+0000) 2020-02-06T01:56:10.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:10.627+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:10.629140+0000) 2020-02-06T01:56:10.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:10.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:10.907897+0000) 2020-02-06T01:56:13.206 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:13.204+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.205358+0000) 2020-02-06T01:56:13.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:13.204+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.205874+0000) 2020-02-06T01:56:13.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:13.205+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.206073+0000) 2020-02-06T01:56:13.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:13.205+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.206620+0000) 2020-02-06T01:56:13.208 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:13.206+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.207640+0000) 2020-02-06T01:56:13.209 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:13.206+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.207915+0000) 2020-02-06T01:56:13.210 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:13.206+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.208102+0000) 2020-02-06T01:56:13.210 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:13.207+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.208671+0000) 2020-02-06T01:56:13.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:13.207+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.209101+0000) 2020-02-06T01:56:13.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:13.208+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.209414+0000) 2020-02-06T01:56:13.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:13.208+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.209603+0000) 2020-02-06T01:56:13.211 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:13.208+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.209324+0000) 2020-02-06T01:56:13.212 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:13.208+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.209719+0000) 2020-02-06T01:56:13.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:13.214+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:13.215949+0000) 2020-02-06T01:56:13.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:13.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217894 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:16.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:16.149+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:16.150409+0000) 2020-02-06T01:56:16.514 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:16.513+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:16.513723+0000) 2020-02-06T01:56:16.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:16.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:16.738414+0000) 2020-02-06T01:56:17.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:17.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.129660+0000) 2020-02-06T01:56:17.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:17.401+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.402254+0000) 2020-02-06T01:56:17.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:17.401+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.402252+0000) 2020-02-06T01:56:17.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:17.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.402565+0000) 2020-02-06T01:56:17.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:17.632+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.634047+0000) 2020-02-06T01:56:17.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:17.650+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.651143+0000) 2020-02-06T01:56:17.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:17.656+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:17.657683+0000) 2020-02-06T01:56:18.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:18.233+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.234766+0000) 2020-02-06T01:56:18.237 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:18.235+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.236791+0000) 2020-02-06T01:56:18.238 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:18.236+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.236623+0000) 2020-02-06T01:56:18.238 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:18.236+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.237685+0000) 2020-02-06T01:56:18.238 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:18.236+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.237388+0000) 2020-02-06T01:56:18.239 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:18.237+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.238177+0000) 2020-02-06T01:56:18.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:18.237+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.238740+0000) 2020-02-06T01:56:18.240 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:18.238+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.239389+0000) 2020-02-06T01:56:18.240 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:18.238+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.239989+0000) 2020-02-06T01:56:18.241 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:18.239+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.240252+0000) 2020-02-06T01:56:18.241 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:18.239+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.240310+0000) 2020-02-06T01:56:18.241 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:18.240+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.240702+0000) 2020-02-06T01:56:18.242 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:18.240+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.240884+0000) 2020-02-06T01:56:18.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:18.245+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:18.246669+0000) 2020-02-06T01:56:18.259 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:18.257+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217937 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:19.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:19.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:19.773492+0000) 2020-02-06T01:56:20.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:20.018+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:20.019384+0000) 2020-02-06T01:56:20.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:20.628+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:20.629393+0000) 2020-02-06T01:56:20.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:20.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:20.908072+0000) 2020-02-06T01:56:23.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:23.262+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.263453+0000) 2020-02-06T01:56:23.266 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:23.264+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.265459+0000) 2020-02-06T01:56:23.267 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:23.265+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.266273+0000) 2020-02-06T01:56:23.267 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:23.265+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.266003+0000) 2020-02-06T01:56:23.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:23.265+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.266873+0000) 2020-02-06T01:56:23.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:23.267+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.267631+0000) 2020-02-06T01:56:23.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:23.266+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.267844+0000) 2020-02-06T01:56:23.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:23.267+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.268510+0000) 2020-02-06T01:56:23.269 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:23.267+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.268895+0000) 2020-02-06T01:56:23.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:23.268+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.269375+0000) 2020-02-06T01:56:23.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:23.269+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.269650+0000) 2020-02-06T01:56:23.271 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:23.270+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.270571+0000) 2020-02-06T01:56:23.271 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:23.270+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.270567+0000) 2020-02-06T01:56:23.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:23.274+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:23.275551+0000) 2020-02-06T01:56:23.287 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:23.286+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 217977 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:26.151 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:26.150+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:26.150651+0000) 2020-02-06T01:56:26.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:26.513+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:26.514018+0000) 2020-02-06T01:56:26.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:26.737+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:26.738666+0000) 2020-02-06T01:56:27.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:27.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.129891+0000) 2020-02-06T01:56:27.403 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:27.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.402517+0000) 2020-02-06T01:56:27.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:27.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.402510+0000) 2020-02-06T01:56:27.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:27.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.402794+0000) 2020-02-06T01:56:27.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:27.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.634330+0000) 2020-02-06T01:56:27.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:27.657+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.657946+0000) 2020-02-06T01:56:27.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:27.658+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:27.659210+0000) 2020-02-06T01:56:28.292 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:28.290+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.291465+0000) 2020-02-06T01:56:28.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:28.292+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.293215+0000) 2020-02-06T01:56:28.294 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:28.292+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.293637+0000) 2020-02-06T01:56:28.301 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:28.293+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.294068+0000) 2020-02-06T01:56:28.302 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:28.293+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.294419+0000) 2020-02-06T01:56:28.302 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:28.294+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.296018+0000) 2020-02-06T01:56:28.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:28.296+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.297685+0000) 2020-02-06T01:56:28.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:28.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.299487+0000) 2020-02-06T01:56:28.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:28.296+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.296489+0000) 2020-02-06T01:56:28.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:28.296+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.296873+0000) 2020-02-06T01:56:28.304 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:28.297+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.298163+0000) 2020-02-06T01:56:28.304 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:28.298+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.298641+0000) 2020-02-06T01:56:28.305 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:28.298+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.299130+0000) 2020-02-06T01:56:28.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:28.302+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:28.303386+0000) 2020-02-06T01:56:28.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:28.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:29.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:29.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:29.773729+0000) 2020-02-06T01:56:30.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:30.019+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:30.019643+0000) 2020-02-06T01:56:30.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:30.629+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:30.629681+0000) 2020-02-06T01:56:30.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:30.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:30.908319+0000) 2020-02-06T01:56:33.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:33.321+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.321994+0000) 2020-02-06T01:56:33.323 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:33.321+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.322344+0000) 2020-02-06T01:56:33.324 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:33.321+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.322993+0000) 2020-02-06T01:56:33.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:33.322+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.322682+0000) 2020-02-06T01:56:33.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:33.322+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.323845+0000) 2020-02-06T01:56:33.325 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:33.322+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.324041+0000) 2020-02-06T01:56:33.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:33.323+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.323735+0000) 2020-02-06T01:56:33.325 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:33.324+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.324442+0000) 2020-02-06T01:56:33.326 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:33.323+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.324302+0000) 2020-02-06T01:56:33.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:33.323+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.324864+0000) 2020-02-06T01:56:33.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:33.324+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.324500+0000) 2020-02-06T01:56:33.327 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:33.325+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.325592+0000) 2020-02-06T01:56:33.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:33.324+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.325905+0000) 2020-02-06T01:56:33.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:33.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:33.330044+0000) 2020-02-06T01:56:33.343 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:33.341+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:36.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:36.150+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:36.150913+0000) 2020-02-06T01:56:36.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:36.514+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:36.514316+0000) 2020-02-06T01:56:36.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:36.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:36.738897+0000) 2020-02-06T01:56:37.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:37.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.130125+0000) 2020-02-06T01:56:37.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:37.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.402747+0000) 2020-02-06T01:56:37.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:37.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.402807+0000) 2020-02-06T01:56:37.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:37.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.403046+0000) 2020-02-06T01:56:37.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:37.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.634557+0000) 2020-02-06T01:56:37.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:37.658+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.658148+0000) 2020-02-06T01:56:37.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:37.667+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:37.667962+0000) 2020-02-06T01:56:38.348 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:38.346+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.346993+0000) 2020-02-06T01:56:38.350 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:38.348+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.349173+0000) 2020-02-06T01:56:38.350 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:38.349+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.349450+0000) 2020-02-06T01:56:38.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:38.349+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.350199+0000) 2020-02-06T01:56:38.352 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:38.350+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.350655+0000) 2020-02-06T01:56:38.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:38.351+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.351361+0000) 2020-02-06T01:56:38.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:38.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.351482+0000) 2020-02-06T01:56:38.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:38.351+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.351953+0000) 2020-02-06T01:56:38.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:38.352+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.352723+0000) 2020-02-06T01:56:38.355 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:38.352+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.352561+0000) 2020-02-06T01:56:38.355 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:38.352+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.353349+0000) 2020-02-06T01:56:38.355 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:38.353+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.353141+0000) 2020-02-06T01:56:38.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:38.353+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.353250+0000) 2020-02-06T01:56:38.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:38.359+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:38.359441+0000) 2020-02-06T01:56:38.371 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:38.370+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:39.710 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:56:39.716 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:56:39.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:39.772+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:39.773967+0000) 2020-02-06T01:56:40.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:40.018+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:40.019923+0000) 2020-02-06T01:56:40.630 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:40.629+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:40.629875+0000) 2020-02-06T01:56:40.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:40.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:40.908531+0000) 2020-02-06T01:56:43.377 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:43.375+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.376281+0000) 2020-02-06T01:56:43.378 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:43.376+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.377253+0000) 2020-02-06T01:56:43.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:43.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.378131+0000) 2020-02-06T01:56:43.379 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:43.377+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.378479+0000) 2020-02-06T01:56:43.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:43.377+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.378864+0000) 2020-02-06T01:56:43.380 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:43.378+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.379183+0000) 2020-02-06T01:56:43.380 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:43.379+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.379620+0000) 2020-02-06T01:56:43.382 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:43.380+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.381400+0000) 2020-02-06T01:56:43.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:43.380+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.381910+0000) 2020-02-06T01:56:43.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:43.381+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.382303+0000) 2020-02-06T01:56:43.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:43.381+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.382331+0000) 2020-02-06T01:56:43.383 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:43.382+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.382806+0000) 2020-02-06T01:56:43.384 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:43.382+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.382894+0000) 2020-02-06T01:56:43.388 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:43.387+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:43.388139+0000) 2020-02-06T01:56:43.399 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:43.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218144 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:46.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:46.150+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:46.151167+0000) 2020-02-06T01:56:46.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:46.514+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:46.514546+0000) 2020-02-06T01:56:46.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:46.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:46.739159+0000) 2020-02-06T01:56:47.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:47.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.130344+0000) 2020-02-06T01:56:47.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:47.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.402990+0000) 2020-02-06T01:56:47.404 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:47.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.403043+0000) 2020-02-06T01:56:47.404 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:47.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.403279+0000) 2020-02-06T01:56:47.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:47.633+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.634784+0000) 2020-02-06T01:56:47.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:47.657+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.658378+0000) 2020-02-06T01:56:47.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:47.674+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:47.675863+0000) 2020-02-06T01:56:48.405 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:48.403+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.404569+0000) 2020-02-06T01:56:48.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:48.405+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.405477+0000) 2020-02-06T01:56:48.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:48.405+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.406113+0000) 2020-02-06T01:56:48.407 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:48.405+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.406569+0000) 2020-02-06T01:56:48.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:48.406+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.407576+0000) 2020-02-06T01:56:48.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:48.406+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.407962+0000) 2020-02-06T01:56:48.414 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:48.408+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.408472+0000) 2020-02-06T01:56:48.415 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:48.409+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.409841+0000) 2020-02-06T01:56:48.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:48.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.410372+0000) 2020-02-06T01:56:48.415 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:48.410+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.410993+0000) 2020-02-06T01:56:48.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:48.407+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.408498+0000) 2020-02-06T01:56:48.415 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:48.407+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.408999+0000) 2020-02-06T01:56:48.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:48.409+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.410211+0000) 2020-02-06T01:56:48.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:48.413+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:48.414407+0000) 2020-02-06T01:56:48.429 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:48.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218189 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:49.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:49.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:49.774183+0000) 2020-02-06T01:56:50.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:50.019+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:50.020180+0000) 2020-02-06T01:56:50.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:50.630+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:50.630139+0000) 2020-02-06T01:56:50.909 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:50.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:50.908758+0000) 2020-02-06T01:56:53.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:53.431+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.432960+0000) 2020-02-06T01:56:53.435 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:53.433+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.434575+0000) 2020-02-06T01:56:53.436 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:53.434+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.435365+0000) 2020-02-06T01:56:53.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:53.434+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.435804+0000) 2020-02-06T01:56:53.437 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:53.435+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.436320+0000) 2020-02-06T01:56:53.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:53.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.436773+0000) 2020-02-06T01:56:53.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:53.436+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.437041+0000) 2020-02-06T01:56:53.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:53.437+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.437932+0000) 2020-02-06T01:56:53.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:53.437+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.438344+0000) 2020-02-06T01:56:53.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:53.438+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.438905+0000) 2020-02-06T01:56:53.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:53.438+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.438725+0000) 2020-02-06T01:56:53.440 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:53.438+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.440051+0000) 2020-02-06T01:56:53.441 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:53.439+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.440184+0000) 2020-02-06T01:56:53.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:53.442+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:53.443192+0000) 2020-02-06T01:56:53.457 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:53.456+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:56.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:56.150+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:56.151424+0000) 2020-02-06T01:56:56.515 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:56.513+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:56.514770+0000) 2020-02-06T01:56:56.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:56.738+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:56.739391+0000) 2020-02-06T01:56:57.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:57.129+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.130550+0000) 2020-02-06T01:56:57.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:57.401+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.403241+0000) 2020-02-06T01:56:57.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:57.401+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.403297+0000) 2020-02-06T01:56:57.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:57.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.403504+0000) 2020-02-06T01:56:57.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:57.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.635005+0000) 2020-02-06T01:56:57.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:57.657+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.658602+0000) 2020-02-06T01:56:57.688 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:57.686+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:57.687295+0000) 2020-02-06T01:56:58.463 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:56:58.461+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.462153+0000) 2020-02-06T01:56:58.464 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:56:58.461+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.462618+0000) 2020-02-06T01:56:58.464 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:56:58.462+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.463241+0000) 2020-02-06T01:56:58.465 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:58.462+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.463756+0000) 2020-02-06T01:56:58.465 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:56:58.462+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.464284+0000) 2020-02-06T01:56:58.466 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:56:58.463+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.464963+0000) 2020-02-06T01:56:58.466 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:58.464+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.465378+0000) 2020-02-06T01:56:58.467 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:56:58.464+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.466099+0000) 2020-02-06T01:56:58.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:56:58.465+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.466371+0000) 2020-02-06T01:56:58.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:56:58.465+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.467261+0000) 2020-02-06T01:56:58.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:56:58.466+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.467736+0000) 2020-02-06T01:56:58.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:56:58.466+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.468065+0000) 2020-02-06T01:56:58.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:56:58.467+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.468873+0000) 2020-02-06T01:56:58.473 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:58.471+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:58.472381+0000) 2020-02-06T01:56:58.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:56:58.484+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218274 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:56:59.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:56:59.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:56:59.774444+0000) 2020-02-06T01:57:00.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:00.019+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:00.020435+0000) 2020-02-06T01:57:00.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:00.629+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:00.630371+0000) 2020-02-06T01:57:00.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:00.907+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:00.909012+0000) 2020-02-06T01:57:03.493 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:03.490+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.492017+0000) 2020-02-06T01:57:03.493 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:03.491+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.493003+0000) 2020-02-06T01:57:03.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:03.492+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.493068+0000) 2020-02-06T01:57:03.495 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:03.492+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.493674+0000) 2020-02-06T01:57:03.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:03.492+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.493731+0000) 2020-02-06T01:57:03.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:03.493+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.494110+0000) 2020-02-06T01:57:03.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:03.493+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.494420+0000) 2020-02-06T01:57:03.496 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:03.493+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.494597+0000) 2020-02-06T01:57:03.496 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:03.494+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.495240+0000) 2020-02-06T01:57:03.496 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:03.493+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.495007+0000) 2020-02-06T01:57:03.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:03.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.496297+0000) 2020-02-06T01:57:03.497 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:03.495+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.496789+0000) 2020-02-06T01:57:03.498 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:03.495+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.496904+0000) 2020-02-06T01:57:03.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:03.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:03.500105+0000) 2020-02-06T01:57:03.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:03.512+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218314 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:06.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:06.150+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:06.151690+0000) 2020-02-06T01:57:06.516 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:06.514+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:06.515005+0000) 2020-02-06T01:57:06.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:06.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:06.739615+0000) 2020-02-06T01:57:07.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:07.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.130810+0000) 2020-02-06T01:57:07.404 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:07.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.403488+0000) 2020-02-06T01:57:07.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:07.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.403686+0000) 2020-02-06T01:57:07.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:07.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.403758+0000) 2020-02-06T01:57:07.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:07.634+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.635230+0000) 2020-02-06T01:57:07.659 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:07.658+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.658828+0000) 2020-02-06T01:57:07.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:07.694+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:07.695372+0000) 2020-02-06T01:57:08.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:08.517+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.518823+0000) 2020-02-06T01:57:08.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:08.518+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.519990+0000) 2020-02-06T01:57:08.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:08.520+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.522030+0000) 2020-02-06T01:57:08.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:08.521+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.522644+0000) 2020-02-06T01:57:08.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:08.521+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.522965+0000) 2020-02-06T01:57:08.524 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:08.522+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.523702+0000) 2020-02-06T01:57:08.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:08.522+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.523517+0000) 2020-02-06T01:57:08.525 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:08.523+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.524453+0000) 2020-02-06T01:57:08.525 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:08.523+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.524615+0000) 2020-02-06T01:57:08.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:08.523+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.524855+0000) 2020-02-06T01:57:08.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:08.523+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.524962+0000) 2020-02-06T01:57:08.527 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:08.524+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.525109+0000) 2020-02-06T01:57:08.527 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:08.523+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.525161+0000) 2020-02-06T01:57:08.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:08.525+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:08.526668+0000) 2020-02-06T01:57:08.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:08.540+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218359 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:09.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:09.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:09.774713+0000) 2020-02-06T01:57:09.840 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:57:09.848 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:57:10.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:10.019+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:10.020680+0000) 2020-02-06T01:57:10.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:10.629+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:10.630634+0000) 2020-02-06T01:57:10.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:10.908+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:10.909268+0000) 2020-02-06T01:57:13.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:13.545+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.547039+0000) 2020-02-06T01:57:13.548 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:13.544+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.546300+0000) 2020-02-06T01:57:13.549 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:13.546+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.548067+0000) 2020-02-06T01:57:13.550 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:13.547+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.548555+0000) 2020-02-06T01:57:13.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:13.548+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.550155+0000) 2020-02-06T01:57:13.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:13.549+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.550361+0000) 2020-02-06T01:57:13.551 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:13.549+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.550878+0000) 2020-02-06T01:57:13.552 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:13.550+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.551552+0000) 2020-02-06T01:57:13.553 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:13.551+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.552553+0000) 2020-02-06T01:57:13.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:13.552+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.553233+0000) 2020-02-06T01:57:13.554 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:13.552+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.553471+0000) 2020-02-06T01:57:13.555 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:13.552+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.553942+0000) 2020-02-06T01:57:13.555 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:13.552+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.553504+0000) 2020-02-06T01:57:13.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:13.559+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:13.560967+0000) 2020-02-06T01:57:13.571 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:13.568+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218399 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:16.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:16.151+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:16.151930+0000) 2020-02-06T01:57:16.516 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:16.514+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:16.515238+0000) 2020-02-06T01:57:16.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:16.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:16.739866+0000) 2020-02-06T01:57:17.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:17.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.131034+0000) 2020-02-06T01:57:17.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:17.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.403728+0000) 2020-02-06T01:57:17.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:17.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.403932+0000) 2020-02-06T01:57:17.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:17.402+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.404018+0000) 2020-02-06T01:57:17.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:17.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.635459+0000) 2020-02-06T01:57:17.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:17.658+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.659059+0000) 2020-02-06T01:57:17.704 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:17.702+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:17.703256+0000) 2020-02-06T01:57:18.576 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:18.573+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.575011+0000) 2020-02-06T01:57:18.577 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:18.575+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.576665+0000) 2020-02-06T01:57:18.578 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:18.576+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.577343+0000) 2020-02-06T01:57:18.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:18.576+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.577899+0000) 2020-02-06T01:57:18.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:18.577+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.578264+0000) 2020-02-06T01:57:18.580 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:18.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.578517+0000) 2020-02-06T01:57:18.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:18.577+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.578909+0000) 2020-02-06T01:57:18.581 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:18.577+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.579157+0000) 2020-02-06T01:57:18.582 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:18.579+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.580448+0000) 2020-02-06T01:57:18.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:18.579+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.580412+0000) 2020-02-06T01:57:18.582 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:18.578+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.579994+0000) 2020-02-06T01:57:18.583 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:18.579+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.581218+0000) 2020-02-06T01:57:18.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:18.580+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.581184+0000) 2020-02-06T01:57:18.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:18.585+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:18.586634+0000) 2020-02-06T01:57:18.600 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:18.598+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218445 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:19.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:19.773+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:19.774979+0000) 2020-02-06T01:57:20.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:20.020+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:20.020930+0000) 2020-02-06T01:57:20.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:20.630+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:20.630889+0000) 2020-02-06T01:57:20.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:20.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:20.909503+0000) 2020-02-06T01:57:23.607 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:23.605+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.606150+0000) 2020-02-06T01:57:23.607 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:23.605+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.606417+0000) 2020-02-06T01:57:23.608 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:23.606+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.607429+0000) 2020-02-06T01:57:23.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:23.606+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.607662+0000) 2020-02-06T01:57:23.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:23.606+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.607869+0000) 2020-02-06T01:57:23.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:23.607+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.608530+0000) 2020-02-06T01:57:23.610 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:23.607+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.608623+0000) 2020-02-06T01:57:23.610 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:23.607+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.608803+0000) 2020-02-06T01:57:23.610 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:23.607+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.608985+0000) 2020-02-06T01:57:23.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:23.608+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.609532+0000) 2020-02-06T01:57:23.611 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:23.608+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.610281+0000) 2020-02-06T01:57:23.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:23.609+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.610738+0000) 2020-02-06T01:57:23.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:23.609+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.611113+0000) 2020-02-06T01:57:23.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:23.612+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:23.613155+0000) 2020-02-06T01:57:23.628 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:23.625+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:26.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:26.151+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:26.152190+0000) 2020-02-06T01:57:26.516 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:26.514+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:26.515526+0000) 2020-02-06T01:57:26.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:26.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:26.740137+0000) 2020-02-06T01:57:27.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:27.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.131261+0000) 2020-02-06T01:57:27.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:27.403+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.403977+0000) 2020-02-06T01:57:27.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:27.403+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.404153+0000) 2020-02-06T01:57:27.405 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:27.403+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.404264+0000) 2020-02-06T01:57:27.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:27.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.635723+0000) 2020-02-06T01:57:27.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:27.659+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.659319+0000) 2020-02-06T01:57:27.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:27.713+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:27.713201+0000) 2020-02-06T01:57:28.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:28.631+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.632700+0000) 2020-02-06T01:57:28.634 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:28.632+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.633168+0000) 2020-02-06T01:57:28.635 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:28.633+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.634508+0000) 2020-02-06T01:57:28.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:28.633+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.634678+0000) 2020-02-06T01:57:28.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:28.635+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.636424+0000) 2020-02-06T01:57:28.637 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:28.636+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.636719+0000) 2020-02-06T01:57:28.638 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:28.636+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.637521+0000) 2020-02-06T01:57:28.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:28.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.637488+0000) 2020-02-06T01:57:28.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:28.636+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.637964+0000) 2020-02-06T01:57:28.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:28.637+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.638505+0000) 2020-02-06T01:57:28.639 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:28.637+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.638897+0000) 2020-02-06T01:57:28.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:28.637+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.639354+0000) 2020-02-06T01:57:28.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:28.641+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.642088+0000) 2020-02-06T01:57:28.643 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:28.642+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:28.643124+0000) 2020-02-06T01:57:28.655 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:28.652+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218529 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:29.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:29.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:29.775215+0000) 2020-02-06T01:57:30.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:30.021+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:30.021161+0000) 2020-02-06T01:57:30.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:30.630+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:30.631129+0000) 2020-02-06T01:57:30.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:30.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:30.909738+0000) 2020-02-06T01:57:33.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:33.659+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.659402+0000) 2020-02-06T01:57:33.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:33.659+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.659932+0000) 2020-02-06T01:57:33.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:33.660+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.660876+0000) 2020-02-06T01:57:33.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:33.661+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.661250+0000) 2020-02-06T01:57:33.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:33.660+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.661647+0000) 2020-02-06T01:57:33.663 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:33.661+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.661854+0000) 2020-02-06T01:57:33.664 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:33.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.663329+0000) 2020-02-06T01:57:33.664 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:33.661+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.663359+0000) 2020-02-06T01:57:33.665 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:33.662+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.663833+0000) 2020-02-06T01:57:33.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:33.665+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.665487+0000) 2020-02-06T01:57:33.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:33.664+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.665834+0000) 2020-02-06T01:57:33.667 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:33.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.666086+0000) 2020-02-06T01:57:33.667 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:33.664+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.666235+0000) 2020-02-06T01:57:33.670 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:33.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:33.669739+0000) 2020-02-06T01:57:33.684 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:33.682+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:36.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:36.151+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:36.152487+0000) 2020-02-06T01:57:36.516 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:36.515+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:36.515767+0000) 2020-02-06T01:57:36.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:36.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:36.740370+0000) 2020-02-06T01:57:37.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:37.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.131498+0000) 2020-02-06T01:57:37.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:37.402+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.404212+0000) 2020-02-06T01:57:37.405 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:37.402+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.404372+0000) 2020-02-06T01:57:37.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:37.403+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.404529+0000) 2020-02-06T01:57:37.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:37.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.636045+0000) 2020-02-06T01:57:37.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:37.658+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.659546+0000) 2020-02-06T01:57:37.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:37.720+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:37.721247+0000) 2020-02-06T01:57:38.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:38.688+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.689586+0000) 2020-02-06T01:57:38.691 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:38.689+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.690564+0000) 2020-02-06T01:57:38.692 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:38.690+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.691293+0000) 2020-02-06T01:57:38.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:38.690+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.691901+0000) 2020-02-06T01:57:38.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:38.691+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.692325+0000) 2020-02-06T01:57:38.693 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:38.691+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.692398+0000) 2020-02-06T01:57:38.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:38.691+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.692991+0000) 2020-02-06T01:57:38.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:38.691+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.693022+0000) 2020-02-06T01:57:38.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:38.692+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.693562+0000) 2020-02-06T01:57:38.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:38.692+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.694015+0000) 2020-02-06T01:57:38.696 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:38.692+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.693858+0000) 2020-02-06T01:57:38.696 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:38.692+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.694085+0000) 2020-02-06T01:57:38.696 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:38.692+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.693768+0000) 2020-02-06T01:57:38.699 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:38.698+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:38.699086+0000) 2020-02-06T01:57:38.715 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:38.712+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218615 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:39.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:39.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:39.775476+0000) 2020-02-06T01:57:40.019 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:57:40.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:40.020+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:40.021419+0000) 2020-02-06T01:57:40.025 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:57:40.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:40.630+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:40.631396+0000) 2020-02-06T01:57:40.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:40.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:40.909967+0000) 2020-02-06T01:57:43.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:43.718+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.720032+0000) 2020-02-06T01:57:43.721 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:43.719+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.720953+0000) 2020-02-06T01:57:43.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:43.721+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.722834+0000) 2020-02-06T01:57:43.724 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:43.721+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.722752+0000) 2020-02-06T01:57:43.724 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:43.721+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.723083+0000) 2020-02-06T01:57:43.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:43.722+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.723761+0000) 2020-02-06T01:57:43.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:43.722+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.723936+0000) 2020-02-06T01:57:43.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:43.723+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.724269+0000) 2020-02-06T01:57:43.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:43.721+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.723400+0000) 2020-02-06T01:57:43.726 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:43.722+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.723948+0000) 2020-02-06T01:57:43.726 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:43.722+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.724385+0000) 2020-02-06T01:57:43.726 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:43.722+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.724217+0000) 2020-02-06T01:57:43.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:43.723+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.724739+0000) 2020-02-06T01:57:43.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:43.727+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:43.728549+0000) 2020-02-06T01:57:43.742 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:43.740+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218655 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:46.153 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:46.152+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:46.152747+0000) 2020-02-06T01:57:46.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:46.515+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:46.516091+0000) 2020-02-06T01:57:46.741 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:46.739+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:46.740660+0000) 2020-02-06T01:57:47.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:47.130+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.131752+0000) 2020-02-06T01:57:47.405 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:47.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.404471+0000) 2020-02-06T01:57:47.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:47.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.404660+0000) 2020-02-06T01:57:47.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:47.404+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.404792+0000) 2020-02-06T01:57:47.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:47.635+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.636316+0000) 2020-02-06T01:57:47.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:47.658+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.659767+0000) 2020-02-06T01:57:47.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:47.732+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:47.733271+0000) 2020-02-06T01:57:48.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:48.745+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.747043+0000) 2020-02-06T01:57:48.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:48.747+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.748657+0000) 2020-02-06T01:57:48.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:48.747+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.748978+0000) 2020-02-06T01:57:48.750 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:48.749+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.749708+0000) 2020-02-06T01:57:48.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:48.749+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.750199+0000) 2020-02-06T01:57:48.751 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:48.749+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.750617+0000) 2020-02-06T01:57:48.752 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:48.750+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.750808+0000) 2020-02-06T01:57:48.752 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:48.750+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.751227+0000) 2020-02-06T01:57:48.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:48.750+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.751602+0000) 2020-02-06T01:57:48.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:48.752+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.752895+0000) 2020-02-06T01:57:48.754 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:48.752+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.753368+0000) 2020-02-06T01:57:48.754 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:48.753+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.753810+0000) 2020-02-06T01:57:48.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:48.752+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.753689+0000) 2020-02-06T01:57:48.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:48.758+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:48.759125+0000) 2020-02-06T01:57:48.771 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:48.770+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218699 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:49.776 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:49.774+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:49.775709+0000) 2020-02-06T01:57:50.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:50.020+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:50.021645+0000) 2020-02-06T01:57:50.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:50.631+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:50.631678+0000) 2020-02-06T01:57:50.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:50.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:50.910198+0000) 2020-02-06T01:57:53.776 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:53.773+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.774898+0000) 2020-02-06T01:57:53.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:53.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.776562+0000) 2020-02-06T01:57:53.778 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:53.776+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.777508+0000) 2020-02-06T01:57:53.779 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:53.777+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.778169+0000) 2020-02-06T01:57:53.779 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:53.777+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.778342+0000) 2020-02-06T01:57:53.779 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:53.778+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.778696+0000) 2020-02-06T01:57:53.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:53.778+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.779443+0000) 2020-02-06T01:57:53.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:53.778+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.779963+0000) 2020-02-06T01:57:53.781 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:53.779+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.780261+0000) 2020-02-06T01:57:53.782 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:53.781+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.781857+0000) 2020-02-06T01:57:53.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:53.781+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.782068+0000) 2020-02-06T01:57:53.783 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:53.782+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.782529+0000) 2020-02-06T01:57:53.783 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:53.782+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.782689+0000) 2020-02-06T01:57:53.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:53.785+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:53.786245+0000) 2020-02-06T01:57:53.800 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:53.799+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218740 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:56.154 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:56.152+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:56.153007+0000) 2020-02-06T01:57:56.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:56.515+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:56.516324+0000) 2020-02-06T01:57:56.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:56.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:56.740890+0000) 2020-02-06T01:57:57.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:57.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.131987+0000) 2020-02-06T01:57:57.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:57.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.404707+0000) 2020-02-06T01:57:57.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:57.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.404878+0000) 2020-02-06T01:57:57.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:57.404+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.405051+0000) 2020-02-06T01:57:57.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:57.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.636623+0000) 2020-02-06T01:57:57.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:57.659+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.659999+0000) 2020-02-06T01:57:57.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:57.740+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:57.741113+0000) 2020-02-06T01:57:58.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:57:58.803+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.804240+0000) 2020-02-06T01:57:58.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:58.804+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.805114+0000) 2020-02-06T01:57:58.807 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:57:58.805+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.806269+0000) 2020-02-06T01:57:58.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:57:58.805+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.807007+0000) 2020-02-06T01:57:58.808 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:57:58.806+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.807567+0000) 2020-02-06T01:57:58.809 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:57:58.807+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.808268+0000) 2020-02-06T01:57:58.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:58.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.808948+0000) 2020-02-06T01:57:58.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:57:58.808+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.809157+0000) 2020-02-06T01:57:58.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:57:58.808+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.809559+0000) 2020-02-06T01:57:58.811 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:57:58.809+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.809523+0000) 2020-02-06T01:57:58.811 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:57:58.809+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.810003+0000) 2020-02-06T01:57:58.812 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:57:58.808+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.809319+0000) 2020-02-06T01:57:58.812 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:57:58.810+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.810969+0000) 2020-02-06T01:57:58.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:58.812+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:58.813426+0000) 2020-02-06T01:57:58.830 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:57:58.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:57:59.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:57:59.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:57:59.775950+0000) 2020-02-06T01:58:00.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:00.020+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:00.021898+0000) 2020-02-06T01:58:00.632 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:00.631+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:00.631910+0000) 2020-02-06T01:58:00.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:00.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:00.910425+0000) 2020-02-06T01:58:03.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:03.835+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.835675+0000) 2020-02-06T01:58:03.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:03.834+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.836024+0000) 2020-02-06T01:58:03.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:03.835+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.836634+0000) 2020-02-06T01:58:03.838 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:03.836+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.837267+0000) 2020-02-06T01:58:03.839 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:03.837+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.838202+0000) 2020-02-06T01:58:03.839 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:03.837+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.838281+0000) 2020-02-06T01:58:03.839 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:03.838+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.838659+0000) 2020-02-06T01:58:03.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:03.837+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.838759+0000) 2020-02-06T01:58:03.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:03.838+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.839124+0000) 2020-02-06T01:58:03.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:03.838+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.839650+0000) 2020-02-06T01:58:03.841 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:03.839+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.839958+0000) 2020-02-06T01:58:03.842 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:03.840+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.841160+0000) 2020-02-06T01:58:03.842 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:03.839+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.840207+0000) 2020-02-06T01:58:03.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:03.842+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:03.843723+0000) 2020-02-06T01:58:03.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:03.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218825 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:06.154 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:06.152+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:06.153282+0000) 2020-02-06T01:58:06.517 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:06.516+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:06.516547+0000) 2020-02-06T01:58:06.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:06.740+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:06.741159+0000) 2020-02-06T01:58:07.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:07.131+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.132260+0000) 2020-02-06T01:58:07.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:07.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.404949+0000) 2020-02-06T01:58:07.406 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:07.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.405106+0000) 2020-02-06T01:58:07.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:07.405+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.405486+0000) 2020-02-06T01:58:07.637 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:07.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.636847+0000) 2020-02-06T01:58:07.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:07.659+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.660219+0000) 2020-02-06T01:58:07.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:07.748+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:07.749248+0000) 2020-02-06T01:58:08.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:08.864+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.865552+0000) 2020-02-06T01:58:08.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:08.865+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.866468+0000) 2020-02-06T01:58:08.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:08.865+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.866852+0000) 2020-02-06T01:58:08.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:08.866+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.867272+0000) 2020-02-06T01:58:08.868 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:08.866+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.867335+0000) 2020-02-06T01:58:08.869 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:08.867+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.868053+0000) 2020-02-06T01:58:08.869 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:08.868+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.868532+0000) 2020-02-06T01:58:08.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:08.868+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.869150+0000) 2020-02-06T01:58:08.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:08.868+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.869633+0000) 2020-02-06T01:58:08.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:08.869+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.870043+0000) 2020-02-06T01:58:08.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:08.869+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.870713+0000) 2020-02-06T01:58:08.871 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:08.869+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.869821+0000) 2020-02-06T01:58:08.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:08.870+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.870987+0000) 2020-02-06T01:58:08.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:08.876+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:08.877870+0000) 2020-02-06T01:58:08.889 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:08.888+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218869 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:09.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:09.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:09.776175+0000) 2020-02-06T01:58:10.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:10.021+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:10.022131+0000) 2020-02-06T01:58:10.199 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:58:10.206 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:58:10.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:10.631+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:10.632148+0000) 2020-02-06T01:58:10.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:10.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:10.910672+0000) 2020-02-06T01:58:13.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:13.893+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.894961+0000) 2020-02-06T01:58:13.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:13.894+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.895640+0000) 2020-02-06T01:58:13.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:13.896+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.896545+0000) 2020-02-06T01:58:13.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:13.896+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.896880+0000) 2020-02-06T01:58:13.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:13.897+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.898365+0000) 2020-02-06T01:58:13.899 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:13.897+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.898551+0000) 2020-02-06T01:58:13.899 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:13.897+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.898217+0000) 2020-02-06T01:58:13.900 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:13.898+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.899039+0000) 2020-02-06T01:58:13.900 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:13.898+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.899774+0000) 2020-02-06T01:58:13.901 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:13.899+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.899598+0000) 2020-02-06T01:58:13.901 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:13.899+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.900196+0000) 2020-02-06T01:58:13.902 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:13.899+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.900198+0000) 2020-02-06T01:58:13.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:13.899+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.900579+0000) 2020-02-06T01:58:13.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:13.906+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:13.907568+0000) 2020-02-06T01:58:13.919 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:13.918+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:16.154 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:16.153+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:16.153545+0000) 2020-02-06T01:58:16.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:16.516+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:16.516776+0000) 2020-02-06T01:58:16.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:16.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:16.741457+0000) 2020-02-06T01:58:17.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:17.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.132492+0000) 2020-02-06T01:58:17.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:17.405+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.405213+0000) 2020-02-06T01:58:17.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:17.405+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.405368+0000) 2020-02-06T01:58:17.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:17.405+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.405785+0000) 2020-02-06T01:58:17.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:17.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.637114+0000) 2020-02-06T01:58:17.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:17.659+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.660407+0000) 2020-02-06T01:58:17.759 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:17.757+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:17.757852+0000) 2020-02-06T01:58:18.924 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:18.922+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.923819+0000) 2020-02-06T01:58:18.926 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:18.924+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.924878+0000) 2020-02-06T01:58:18.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:18.924+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.925255+0000) 2020-02-06T01:58:18.926 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:18.924+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.926050+0000) 2020-02-06T01:58:18.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:18.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.926599+0000) 2020-02-06T01:58:18.928 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:18.926+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.927276+0000) 2020-02-06T01:58:18.928 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:18.926+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.927058+0000) 2020-02-06T01:58:18.928 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:18.926+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.927504+0000) 2020-02-06T01:58:18.929 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:18.926+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.927961+0000) 2020-02-06T01:58:18.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:18.927+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.928157+0000) 2020-02-06T01:58:18.930 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:18.928+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.928490+0000) 2020-02-06T01:58:18.930 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:18.928+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.929154+0000) 2020-02-06T01:58:18.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:18.927+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.928405+0000) 2020-02-06T01:58:18.936 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:18.934+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:18.935309+0000) 2020-02-06T01:58:18.948 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:18.946+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218955 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:19.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:19.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:19.776375+0000) 2020-02-06T01:58:20.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:20.022+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:20.022407+0000) 2020-02-06T01:58:20.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:20.632+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:20.632398+0000) 2020-02-06T01:58:20.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:20.909+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:20.910920+0000) 2020-02-06T01:58:23.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:23.951+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.951253+0000) 2020-02-06T01:58:23.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:23.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.952779+0000) 2020-02-06T01:58:23.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:23.953+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.953264+0000) 2020-02-06T01:58:23.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:23.953+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.953886+0000) 2020-02-06T01:58:23.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:23.952+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.954014+0000) 2020-02-06T01:58:23.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:23.954+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.954537+0000) 2020-02-06T01:58:23.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:23.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.954902+0000) 2020-02-06T01:58:23.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:23.953+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.954948+0000) 2020-02-06T01:58:23.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:23.953+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.954936+0000) 2020-02-06T01:58:23.957 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:23.955+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.955279+0000) 2020-02-06T01:58:23.957 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:23.954+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.955809+0000) 2020-02-06T01:58:23.957 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:23.955+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.957032+0000) 2020-02-06T01:58:23.958 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:23.957+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.957105+0000) 2020-02-06T01:58:23.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:23.960+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:23.960865+0000) 2020-02-06T01:58:23.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:23.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 218995 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:26.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:26.152+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:26.153807+0000) 2020-02-06T01:58:26.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:26.516+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:26.517079+0000) 2020-02-06T01:58:26.742 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:26.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:26.741708+0000) 2020-02-06T01:58:27.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:27.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.132771+0000) 2020-02-06T01:58:27.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:27.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.405471+0000) 2020-02-06T01:58:27.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:27.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.405660+0000) 2020-02-06T01:58:27.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:27.404+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.406014+0000) 2020-02-06T01:58:27.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:27.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.637360+0000) 2020-02-06T01:58:27.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:27.660+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.660637+0000) 2020-02-06T01:58:27.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:27.769+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:27.769624+0000) 2020-02-06T01:58:28.984 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:28.981+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.982913+0000) 2020-02-06T01:58:28.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:28.982+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.983238+0000) 2020-02-06T01:58:28.984 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:28.982+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.983621+0000) 2020-02-06T01:58:28.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:28.982+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.983985+0000) 2020-02-06T01:58:28.985 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:28.982+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.984367+0000) 2020-02-06T01:58:28.986 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:28.984+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.985877+0000) 2020-02-06T01:58:28.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:28.985+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.987046+0000) 2020-02-06T01:58:28.988 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:28.985+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.986415+0000) 2020-02-06T01:58:28.989 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:28.986+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.987822+0000) 2020-02-06T01:58:28.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:28.987+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.988567+0000) 2020-02-06T01:58:28.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:28.986+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.988075+0000) 2020-02-06T01:58:28.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:28.987+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.988467+0000) 2020-02-06T01:58:28.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:28.988+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.989363+0000) 2020-02-06T01:58:28.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:28.990+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:28.991937+0000) 2020-02-06T01:58:29.007 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:29.005+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219039 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:29.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:29.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:29.776576+0000) 2020-02-06T01:58:30.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:30.021+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:30.022634+0000) 2020-02-06T01:58:30.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:30.631+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:30.632658+0000) 2020-02-06T01:58:30.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:30.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:30.911151+0000) 2020-02-06T01:58:34.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:34.011+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.012775+0000) 2020-02-06T01:58:34.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:34.011+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.012836+0000) 2020-02-06T01:58:34.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:34.012+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.013659+0000) 2020-02-06T01:58:34.015 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:34.012+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.013186+0000) 2020-02-06T01:58:34.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:34.012+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.013736+0000) 2020-02-06T01:58:34.015 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:34.013+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.014431+0000) 2020-02-06T01:58:34.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:34.014+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.015185+0000) 2020-02-06T01:58:34.017 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:34.014+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.016226+0000) 2020-02-06T01:58:34.018 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:34.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.016895+0000) 2020-02-06T01:58:34.018 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:34.016+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.017910+0000) 2020-02-06T01:58:34.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:34.017+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.018571+0000) 2020-02-06T01:58:34.020 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:34.017+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.018511+0000) 2020-02-06T01:58:34.020 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:34.017+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.018510+0000) 2020-02-06T01:58:34.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:34.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:34.020751+0000) 2020-02-06T01:58:34.035 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:34.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:36.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:36.153+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:36.154063+0000) 2020-02-06T01:58:36.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:36.516+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:36.517333+0000) 2020-02-06T01:58:36.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:36.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:36.741944+0000) 2020-02-06T01:58:37.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:37.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.133011+0000) 2020-02-06T01:58:37.406 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:37.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.405725+0000) 2020-02-06T01:58:37.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:37.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.405904+0000) 2020-02-06T01:58:37.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:37.404+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.406236+0000) 2020-02-06T01:58:37.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:37.636+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.637606+0000) 2020-02-06T01:58:37.661 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:37.659+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.660868+0000) 2020-02-06T01:58:37.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:37.776+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:37.777591+0000) 2020-02-06T01:58:39.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:39.039+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.040061+0000) 2020-02-06T01:58:39.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:39.039+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.040882+0000) 2020-02-06T01:58:39.042 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:39.040+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.041471+0000) 2020-02-06T01:58:39.042 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:39.040+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.041955+0000) 2020-02-06T01:58:39.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:39.040+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.041980+0000) 2020-02-06T01:58:39.043 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:39.041+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.042816+0000) 2020-02-06T01:58:39.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:39.041+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.042656+0000) 2020-02-06T01:58:39.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:39.042+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.044068+0000) 2020-02-06T01:58:39.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:39.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.043893+0000) 2020-02-06T01:58:39.046 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:39.042+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.044331+0000) 2020-02-06T01:58:39.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:39.043+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.045226+0000) 2020-02-06T01:58:39.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:39.044+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.045238+0000) 2020-02-06T01:58:39.047 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:39.043+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.045364+0000) 2020-02-06T01:58:39.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:39.048+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.049445+0000) 2020-02-06T01:58:39.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:39.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:39.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:39.775+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:39.776787+0000) 2020-02-06T01:58:40.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:40.022+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:40.022873+0000) 2020-02-06T01:58:40.387 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:58:40.393 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:58:40.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:40.632+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:40.632916+0000) 2020-02-06T01:58:40.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:40.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:40.911384+0000) 2020-02-06T01:58:44.069 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:44.067+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.068496+0000) 2020-02-06T01:58:44.070 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:44.068+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.069747+0000) 2020-02-06T01:58:44.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:44.069+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.070715+0000) 2020-02-06T01:58:44.072 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:44.070+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.071231+0000) 2020-02-06T01:58:44.072 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:44.070+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.071638+0000) 2020-02-06T01:58:44.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:44.070+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.071922+0000) 2020-02-06T01:58:44.073 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:44.070+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.072351+0000) 2020-02-06T01:58:44.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:44.071+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.073049+0000) 2020-02-06T01:58:44.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:44.072+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.073547+0000) 2020-02-06T01:58:44.074 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:44.072+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.073875+0000) 2020-02-06T01:58:44.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:44.072+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.074338+0000) 2020-02-06T01:58:44.076 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:44.073+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.075254+0000) 2020-02-06T01:58:44.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:44.073+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.075242+0000) 2020-02-06T01:58:44.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:44.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:44.078216+0000) 2020-02-06T01:58:44.094 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:44.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219165 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:46.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:46.153+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:46.154306+0000) 2020-02-06T01:58:46.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:46.516+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:46.517596+0000) 2020-02-06T01:58:46.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:46.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:46.742211+0000) 2020-02-06T01:58:47.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:47.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.133272+0000) 2020-02-06T01:58:47.407 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:47.405+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.405979+0000) 2020-02-06T01:58:47.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:47.405+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.406124+0000) 2020-02-06T01:58:47.407 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:47.405+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.406515+0000) 2020-02-06T01:58:47.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:47.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.637835+0000) 2020-02-06T01:58:47.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:47.660+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.661140+0000) 2020-02-06T01:58:47.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:47.788+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:47.789317+0000) 2020-02-06T01:58:49.099 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:49.097+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.098288+0000) 2020-02-06T01:58:49.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:49.097+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.099048+0000) 2020-02-06T01:58:49.101 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:49.099+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.100213+0000) 2020-02-06T01:58:49.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:49.099+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.101154+0000) 2020-02-06T01:58:49.102 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:49.099+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.101296+0000) 2020-02-06T01:58:49.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:49.100+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.101756+0000) 2020-02-06T01:58:49.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:49.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.102469+0000) 2020-02-06T01:58:49.103 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:49.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.102641+0000) 2020-02-06T01:58:49.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:49.102+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.103497+0000) 2020-02-06T01:58:49.105 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:49.102+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.104406+0000) 2020-02-06T01:58:49.106 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:49.104+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.105181+0000) 2020-02-06T01:58:49.106 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:49.104+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.105222+0000) 2020-02-06T01:58:49.106 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:49.103+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.105388+0000) 2020-02-06T01:58:49.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:49.108+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.109491+0000) 2020-02-06T01:58:49.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:49.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219209 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:49.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:49.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:49.776990+0000) 2020-02-06T01:58:50.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:50.022+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:50.023114+0000) 2020-02-06T01:58:50.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:50.632+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:50.633176+0000) 2020-02-06T01:58:50.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:50.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:50.911607+0000) 2020-02-06T01:58:54.130 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:54.128+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.129580+0000) 2020-02-06T01:58:54.131 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:54.127+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.129349+0000) 2020-02-06T01:58:54.132 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:54.130+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.131589+0000) 2020-02-06T01:58:54.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:54.130+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.132017+0000) 2020-02-06T01:58:54.133 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:54.130+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.132278+0000) 2020-02-06T01:58:54.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:54.131+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.132647+0000) 2020-02-06T01:58:54.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:54.132+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.133532+0000) 2020-02-06T01:58:54.135 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:54.132+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.133893+0000) 2020-02-06T01:58:54.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:54.132+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.133872+0000) 2020-02-06T01:58:54.135 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:54.133+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.134183+0000) 2020-02-06T01:58:54.136 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:54.132+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.133959+0000) 2020-02-06T01:58:54.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:54.132+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.134211+0000) 2020-02-06T01:58:54.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:54.132+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.134370+0000) 2020-02-06T01:58:54.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:54.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:54.139645+0000) 2020-02-06T01:58:54.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:54.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219250 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:56.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:56.153+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:56.154551+0000) 2020-02-06T01:58:56.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:56.517+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:56.517823+0000) 2020-02-06T01:58:56.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:56.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:56.742442+0000) 2020-02-06T01:58:57.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:57.132+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.133442+0000) 2020-02-06T01:58:57.407 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:57.404+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.406221+0000) 2020-02-06T01:58:57.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:57.404+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.406393+0000) 2020-02-06T01:58:57.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:57.405+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.406790+0000) 2020-02-06T01:58:57.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:57.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.638070+0000) 2020-02-06T01:58:57.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:57.660+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.661368+0000) 2020-02-06T01:58:57.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:57.796+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:57.796543+0000) 2020-02-06T01:58:59.158 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:58:59.156+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.157308+0000) 2020-02-06T01:58:59.159 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:58:59.157+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.158293+0000) 2020-02-06T01:58:59.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:58:59.156+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.158129+0000) 2020-02-06T01:58:59.159 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:58:59.157+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.158916+0000) 2020-02-06T01:58:59.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:59.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.159210+0000) 2020-02-06T01:58:59.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:59.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.159772+0000) 2020-02-06T01:58:59.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:58:59.158+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.160001+0000) 2020-02-06T01:58:59.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:58:59.159+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.160783+0000) 2020-02-06T01:58:59.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:58:59.159+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.161010+0000) 2020-02-06T01:58:59.162 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:58:59.160+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.161446+0000) 2020-02-06T01:58:59.162 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:58:59.160+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.162284+0000) 2020-02-06T01:58:59.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:58:59.161+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.162267+0000) 2020-02-06T01:58:59.164 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:58:59.161+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.163334+0000) 2020-02-06T01:58:59.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:59.167+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.168212+0000) 2020-02-06T01:58:59.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:58:59.178+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219295 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:58:59.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:58:59.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:58:59.777198+0000) 2020-02-06T01:59:00.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:00.022+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:00.023351+0000) 2020-02-06T01:59:00.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:00.632+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:00.633408+0000) 2020-02-06T01:59:00.912 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:00.910+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:00.911795+0000) 2020-02-06T01:59:04.187 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:04.185+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.186342+0000) 2020-02-06T01:59:04.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:04.186+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.187789+0000) 2020-02-06T01:59:04.190 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:04.187+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.189248+0000) 2020-02-06T01:59:04.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:04.187+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.189178+0000) 2020-02-06T01:59:04.191 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:04.188+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.189419+0000) 2020-02-06T01:59:04.191 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:04.188+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.190200+0000) 2020-02-06T01:59:04.191 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:04.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.190726+0000) 2020-02-06T01:59:04.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:04.190+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.191591+0000) 2020-02-06T01:59:04.192 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:04.190+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.191729+0000) 2020-02-06T01:59:04.193 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:04.191+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.192448+0000) 2020-02-06T01:59:04.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:04.191+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.192149+0000) 2020-02-06T01:59:04.194 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:04.191+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.192414+0000) 2020-02-06T01:59:04.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:04.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.198420+0000) 2020-02-06T01:59:04.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:04.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:04.198749+0000) 2020-02-06T01:59:04.214 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:04.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219335 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:06.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:06.154+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:06.154807+0000) 2020-02-06T01:59:06.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:06.517+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:06.518068+0000) 2020-02-06T01:59:06.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:06.741+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:06.742663+0000) 2020-02-06T01:59:07.134 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:07.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.133679+0000) 2020-02-06T01:59:07.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:07.405+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.406499+0000) 2020-02-06T01:59:07.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:07.405+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.406660+0000) 2020-02-06T01:59:07.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:07.405+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.407087+0000) 2020-02-06T01:59:07.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:07.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.638329+0000) 2020-02-06T01:59:07.662 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:07.661+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.661593+0000) 2020-02-06T01:59:07.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:07.804+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:07.804957+0000) 2020-02-06T01:59:09.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:09.212+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.213568+0000) 2020-02-06T01:59:09.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:09.213+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.214353+0000) 2020-02-06T01:59:09.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:09.214+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.215504+0000) 2020-02-06T01:59:09.216 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:09.214+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.215574+0000) 2020-02-06T01:59:09.218 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:09.216+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.217242+0000) 2020-02-06T01:59:09.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:09.216+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.217397+0000) 2020-02-06T01:59:09.218 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:09.216+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.217727+0000) 2020-02-06T01:59:09.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:09.216+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.218328+0000) 2020-02-06T01:59:09.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:09.217+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.218768+0000) 2020-02-06T01:59:09.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:09.217+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.219029+0000) 2020-02-06T01:59:09.220 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:09.217+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.219049+0000) 2020-02-06T01:59:09.220 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:09.217+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.219358+0000) 2020-02-06T01:59:09.221 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:09.218+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.219588+0000) 2020-02-06T01:59:09.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:09.220+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.221145+0000) 2020-02-06T01:59:09.239 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:09.237+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219379 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:09.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:09.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:09.777432+0000) 2020-02-06T01:59:10.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:10.023+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:10.023561+0000) 2020-02-06T01:59:10.584 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:59:10.590 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:59:10.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:10.632+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:10.633674+0000) 2020-02-06T01:59:10.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:10.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:10.912033+0000) 2020-02-06T01:59:14.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:14.242+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.243732+0000) 2020-02-06T01:59:14.245 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:14.243+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.244538+0000) 2020-02-06T01:59:14.245 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:14.243+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.245018+0000) 2020-02-06T01:59:14.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:14.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.246418+0000) 2020-02-06T01:59:14.247 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:14.246+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.246718+0000) 2020-02-06T01:59:14.248 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:14.245+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.247056+0000) 2020-02-06T01:59:14.248 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:14.247+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.247492+0000) 2020-02-06T01:59:14.248 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:14.247+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.247790+0000) 2020-02-06T01:59:14.249 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:14.248+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.248702+0000) 2020-02-06T01:59:14.250 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:14.248+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.249516+0000) 2020-02-06T01:59:14.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:14.249+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.249868+0000) 2020-02-06T01:59:14.251 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:14.249+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.250042+0000) 2020-02-06T01:59:14.251 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:14.249+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.250854+0000) 2020-02-06T01:59:14.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:14.252+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:14.253916+0000) 2020-02-06T01:59:14.268 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:14.267+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:16.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:16.154+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:16.155041+0000) 2020-02-06T01:59:16.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:16.517+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:16.518318+0000) 2020-02-06T01:59:16.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:16.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:16.742896+0000) 2020-02-06T01:59:17.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:17.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.133899+0000) 2020-02-06T01:59:17.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:17.406+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.406805+0000) 2020-02-06T01:59:17.408 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:17.406+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.406918+0000) 2020-02-06T01:59:17.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:17.406+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.407342+0000) 2020-02-06T01:59:17.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:17.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.638558+0000) 2020-02-06T01:59:17.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:17.661+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.661820+0000) 2020-02-06T01:59:17.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:17.813+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:17.813362+0000) 2020-02-06T01:59:19.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:19.273+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.274002+0000) 2020-02-06T01:59:19.275 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:19.274+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.274512+0000) 2020-02-06T01:59:19.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:19.275+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.275517+0000) 2020-02-06T01:59:19.277 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:19.275+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.276187+0000) 2020-02-06T01:59:19.277 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:19.276+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.276510+0000) 2020-02-06T01:59:19.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:19.276+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.276932+0000) 2020-02-06T01:59:19.278 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:19.277+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.277731+0000) 2020-02-06T01:59:19.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:19.277+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.277642+0000) 2020-02-06T01:59:19.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:19.277+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.277389+0000) 2020-02-06T01:59:19.279 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:19.277+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.277804+0000) 2020-02-06T01:59:19.279 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:19.278+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.278554+0000) 2020-02-06T01:59:19.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:19.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.277614+0000) 2020-02-06T01:59:19.280 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:19.278+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.278905+0000) 2020-02-06T01:59:19.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:19.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.284255+0000) 2020-02-06T01:59:19.296 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:19.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:19.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:19.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:19.777675+0000) 2020-02-06T01:59:20.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:20.023+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:20.023784+0000) 2020-02-06T01:59:20.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:20.633+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:20.633880+0000) 2020-02-06T01:59:20.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:20.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:20.912313+0000) 2020-02-06T01:59:24.302 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:24.299+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.301056+0000) 2020-02-06T01:59:24.302 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:24.300+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.301857+0000) 2020-02-06T01:59:24.303 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:24.302+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.302668+0000) 2020-02-06T01:59:24.304 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:24.302+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.303317+0000) 2020-02-06T01:59:24.305 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:24.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.304050+0000) 2020-02-06T01:59:24.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:24.303+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.304466+0000) 2020-02-06T01:59:24.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:24.303+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.304757+0000) 2020-02-06T01:59:24.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:24.304+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.305133+0000) 2020-02-06T01:59:24.306 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:24.305+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.305718+0000) 2020-02-06T01:59:24.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:24.305+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.306377+0000) 2020-02-06T01:59:24.307 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:24.305+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.306838+0000) 2020-02-06T01:59:24.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:24.306+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.307257+0000) 2020-02-06T01:59:24.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:24.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.307632+0000) 2020-02-06T01:59:24.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:24.311+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:24.312676+0000) 2020-02-06T01:59:24.324 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:24.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219505 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:26.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:26.154+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:26.155281+0000) 2020-02-06T01:59:26.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:26.518+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:26.518535+0000) 2020-02-06T01:59:26.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:26.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:26.743158+0000) 2020-02-06T01:59:27.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:27.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.134128+0000) 2020-02-06T01:59:27.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:27.406+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.407063+0000) 2020-02-06T01:59:27.408 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:27.406+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.407146+0000) 2020-02-06T01:59:27.408 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:27.407+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.407623+0000) 2020-02-06T01:59:27.639 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:27.637+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.638793+0000) 2020-02-06T01:59:27.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:27.661+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.662045+0000) 2020-02-06T01:59:27.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:27.820+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:27.821344+0000) 2020-02-06T01:59:29.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:29.328+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.329927+0000) 2020-02-06T01:59:29.332 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:29.329+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.330745+0000) 2020-02-06T01:59:29.333 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:29.330+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.331096+0000) 2020-02-06T01:59:29.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:29.330+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.331506+0000) 2020-02-06T01:59:29.334 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:29.331+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.331742+0000) 2020-02-06T01:59:29.334 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:29.331+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.332001+0000) 2020-02-06T01:59:29.335 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:29.332+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.332470+0000) 2020-02-06T01:59:29.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:29.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.332226+0000) 2020-02-06T01:59:29.335 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:29.332+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.332890+0000) 2020-02-06T01:59:29.336 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:29.331+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.331966+0000) 2020-02-06T01:59:29.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:29.332+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.333953+0000) 2020-02-06T01:59:29.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:29.333+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.334313+0000) 2020-02-06T01:59:29.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:29.333+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.334672+0000) 2020-02-06T01:59:29.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:29.340+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.341377+0000) 2020-02-06T01:59:29.356 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:29.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219550 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:29.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:29.776+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:29.777904+0000) 2020-02-06T01:59:30.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:30.022+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:30.024033+0000) 2020-02-06T01:59:30.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:30.633+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:30.634129+0000) 2020-02-06T01:59:30.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:30.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:30.912549+0000) 2020-02-06T01:59:34.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:34.356+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.357877+0000) 2020-02-06T01:59:34.359 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:34.357+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.358441+0000) 2020-02-06T01:59:34.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:34.359+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.360083+0000) 2020-02-06T01:59:34.361 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:34.359+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.360430+0000) 2020-02-06T01:59:34.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:34.359+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.361007+0000) 2020-02-06T01:59:34.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:34.361+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.362024+0000) 2020-02-06T01:59:34.364 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:34.362+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.362706+0000) 2020-02-06T01:59:34.364 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:34.361+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.362999+0000) 2020-02-06T01:59:34.364 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:34.363+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.363783+0000) 2020-02-06T01:59:34.365 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:34.363+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.364302+0000) 2020-02-06T01:59:34.366 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:34.363+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.364618+0000) 2020-02-06T01:59:34.366 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:34.364+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.365096+0000) 2020-02-06T01:59:34.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:34.366+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.367679+0000) 2020-02-06T01:59:34.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:34.367+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:34.368735+0000) 2020-02-06T01:59:34.384 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:34.382+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219590 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:36.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:36.155+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:36.155534+0000) 2020-02-06T01:59:36.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:36.518+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:36.518784+0000) 2020-02-06T01:59:36.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:36.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:36.743406+0000) 2020-02-06T01:59:37.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:37.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.134360+0000) 2020-02-06T01:59:37.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:37.407+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.407323+0000) 2020-02-06T01:59:37.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:37.407+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.407380+0000) 2020-02-06T01:59:37.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:37.407+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.407849+0000) 2020-02-06T01:59:37.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:37.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.639048+0000) 2020-02-06T01:59:37.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:37.661+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.662299+0000) 2020-02-06T01:59:37.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:37.828+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:37.829721+0000) 2020-02-06T01:59:39.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:39.387+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.388183+0000) 2020-02-06T01:59:39.390 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:39.388+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.389985+0000) 2020-02-06T01:59:39.391 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:39.389+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.390004+0000) 2020-02-06T01:59:39.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:39.389+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.390442+0000) 2020-02-06T01:59:39.392 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:39.390+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.390985+0000) 2020-02-06T01:59:39.392 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:39.389+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.391015+0000) 2020-02-06T01:59:39.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:39.390+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.391625+0000) 2020-02-06T01:59:39.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:39.391+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.392116+0000) 2020-02-06T01:59:39.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:39.391+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.392622+0000) 2020-02-06T01:59:39.394 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:39.392+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.392827+0000) 2020-02-06T01:59:39.394 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:39.392+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.393239+0000) 2020-02-06T01:59:39.394 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:39.393+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.393706+0000) 2020-02-06T01:59:39.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:39.393+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.394295+0000) 2020-02-06T01:59:39.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:39.397+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.398981+0000) 2020-02-06T01:59:39.412 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:39.411+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219635 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:39.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:39.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:39.778136+0000) 2020-02-06T01:59:40.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:40.023+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:40.024278+0000) 2020-02-06T01:59:40.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:40.634+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:40.634366+0000) 2020-02-06T01:59:40.757 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:59:40.769 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T01:59:40.913 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:40.911+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:40.912778+0000) 2020-02-06T01:59:44.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:44.416+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.417637+0000) 2020-02-06T01:59:44.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:44.416+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.417224+0000) 2020-02-06T01:59:44.419 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:44.417+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.418199+0000) 2020-02-06T01:59:44.420 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:44.417+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.418952+0000) 2020-02-06T01:59:44.421 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:44.419+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.420270+0000) 2020-02-06T01:59:44.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:44.419+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.420812+0000) 2020-02-06T01:59:44.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:44.420+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.420626+0000) 2020-02-06T01:59:44.422 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:44.420+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.421278+0000) 2020-02-06T01:59:44.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:44.420+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.421398+0000) 2020-02-06T01:59:44.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:44.420+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.422008+0000) 2020-02-06T01:59:44.424 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:44.421+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.422396+0000) 2020-02-06T01:59:44.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:44.422+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.422818+0000) 2020-02-06T01:59:44.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:44.422+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.422706+0000) 2020-02-06T01:59:44.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:44.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:44.425586+0000) 2020-02-06T01:59:44.442 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:44.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:46.156 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:46.155+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:46.155825+0000) 2020-02-06T01:59:46.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:46.518+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:46.519072+0000) 2020-02-06T01:59:46.744 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:46.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:46.743674+0000) 2020-02-06T01:59:47.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:47.133+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.134599+0000) 2020-02-06T01:59:47.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:47.407+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.407576+0000) 2020-02-06T01:59:47.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:47.407+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.407697+0000) 2020-02-06T01:59:47.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:47.407+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.408036+0000) 2020-02-06T01:59:47.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:47.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.639338+0000) 2020-02-06T01:59:47.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:47.661+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.662523+0000) 2020-02-06T01:59:47.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:47.837+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:47.837869+0000) 2020-02-06T01:59:49.447 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:49.445+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.446285+0000) 2020-02-06T01:59:49.448 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:49.446+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.446475+0000) 2020-02-06T01:59:49.448 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:49.446+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.448051+0000) 2020-02-06T01:59:49.450 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:49.448+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.449040+0000) 2020-02-06T01:59:49.450 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:49.448+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.449274+0000) 2020-02-06T01:59:49.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:49.448+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.449495+0000) 2020-02-06T01:59:49.452 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:49.449+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.450147+0000) 2020-02-06T01:59:49.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:49.450+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.450727+0000) 2020-02-06T01:59:49.452 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:49.449+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.450310+0000) 2020-02-06T01:59:49.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:49.450+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.451520+0000) 2020-02-06T01:59:49.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:49.451+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.451986+0000) 2020-02-06T01:59:49.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:49.452+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.452741+0000) 2020-02-06T01:59:49.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:49.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.457047+0000) 2020-02-06T01:59:49.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:49.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.457298+0000) 2020-02-06T01:59:49.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:49.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:49.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:49.777+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:49.778374+0000) 2020-02-06T01:59:50.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:50.023+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:50.024476+0000) 2020-02-06T01:59:50.635 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:50.634+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:50.634628+0000) 2020-02-06T01:59:50.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:50.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:50.913104+0000) 2020-02-06T01:59:54.477 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:54.475+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.476222+0000) 2020-02-06T01:59:54.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:54.475+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.476737+0000) 2020-02-06T01:59:54.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:54.476+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.477468+0000) 2020-02-06T01:59:54.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:54.476+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.478070+0000) 2020-02-06T01:59:54.479 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:54.477+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.478488+0000) 2020-02-06T01:59:54.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:54.478+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.479257+0000) 2020-02-06T01:59:54.480 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:54.478+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.479723+0000) 2020-02-06T01:59:54.480 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:54.478+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.480127+0000) 2020-02-06T01:59:54.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:54.479+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.480769+0000) 2020-02-06T01:59:54.482 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:54.480+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.481513+0000) 2020-02-06T01:59:54.482 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:54.480+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.481800+0000) 2020-02-06T01:59:54.483 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:54.480+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.481943+0000) 2020-02-06T01:59:54.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:54.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.482423+0000) 2020-02-06T01:59:54.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:54.485+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:54.487041+0000) 2020-02-06T01:59:54.500 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:54.498+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:56.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:56.155+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:56.156121+0000) 2020-02-06T01:59:56.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:56.517+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:56.519310+0000) 2020-02-06T01:59:56.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:56.742+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:56.743920+0000) 2020-02-06T01:59:57.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:57.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.134827+0000) 2020-02-06T01:59:57.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:57.406+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.407833+0000) 2020-02-06T01:59:57.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:57.406+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.407962+0000) 2020-02-06T01:59:57.409 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:57.406+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.408367+0000) 2020-02-06T01:59:57.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:57.638+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.639561+0000) 2020-02-06T01:59:57.663 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:57.662+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.662734+0000) 2020-02-06T01:59:57.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:57.845+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:57.846121+0000) 2020-02-06T01:59:59.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T01:59:59.504+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.505626+0000) 2020-02-06T01:59:59.507 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T01:59:59.505+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.506616+0000) 2020-02-06T01:59:59.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T01:59:59.506+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.507362+0000) 2020-02-06T01:59:59.508 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T01:59:59.506+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.507571+0000) 2020-02-06T01:59:59.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T01:59:59.507+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.508056+0000) 2020-02-06T01:59:59.509 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T01:59:59.507+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.508572+0000) 2020-02-06T01:59:59.509 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T01:59:59.507+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.508681+0000) 2020-02-06T01:59:59.510 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T01:59:59.507+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.508921+0000) 2020-02-06T01:59:59.510 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:59.508+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.509612+0000) 2020-02-06T01:59:59.511 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T01:59:59.508+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.509851+0000) 2020-02-06T01:59:59.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:59.509+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.510419+0000) 2020-02-06T01:59:59.511 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T01:59:59.508+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.510378+0000) 2020-02-06T01:59:59.512 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T01:59:59.509+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.510787+0000) 2020-02-06T01:59:59.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:59.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.516047+0000) 2020-02-06T01:59:59.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T01:59:59.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219805 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T01:59:59.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T01:59:59.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T00:59:59.778606+0000) 2020-02-06T02:00:00.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:00.024+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:00.024680+0000) 2020-02-06T02:00:00.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:00.633+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:00.634879+0000) 2020-02-06T02:00:00.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:00.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:00.913321+0000) 2020-02-06T02:00:04.535 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:04.533+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.534105+0000) 2020-02-06T02:00:04.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:04.535+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.536808+0000) 2020-02-06T02:00:04.538 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:04.535+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.537117+0000) 2020-02-06T02:00:04.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:04.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.537517+0000) 2020-02-06T02:00:04.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:04.536+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.537971+0000) 2020-02-06T02:00:04.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:04.536+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.538295+0000) 2020-02-06T02:00:04.539 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:04.537+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.538425+0000) 2020-02-06T02:00:04.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:04.538+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.539537+0000) 2020-02-06T02:00:04.540 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:04.538+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.539692+0000) 2020-02-06T02:00:04.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:04.538+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.540156+0000) 2020-02-06T02:00:04.541 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:04.539+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.540564+0000) 2020-02-06T02:00:04.542 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:04.539+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.540749+0000) 2020-02-06T02:00:04.542 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:04.539+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.540834+0000) 2020-02-06T02:00:04.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:04.543+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:04.544629+0000) 2020-02-06T02:00:04.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:04.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:06.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:06.155+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:06.156411+0000) 2020-02-06T02:00:06.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:06.518+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:06.519541+0000) 2020-02-06T02:00:06.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:06.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:06.744216+0000) 2020-02-06T02:00:07.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:07.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.135060+0000) 2020-02-06T02:00:07.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:07.407+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.408076+0000) 2020-02-06T02:00:07.409 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:07.407+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.408192+0000) 2020-02-06T02:00:07.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:07.407+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.408642+0000) 2020-02-06T02:00:07.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:07.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.639833+0000) 2020-02-06T02:00:07.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:07.662+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.662993+0000) 2020-02-06T02:00:07.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:07.854+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:07.854677+0000) 2020-02-06T02:00:09.566 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:09.563+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.564867+0000) 2020-02-06T02:00:09.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:09.564+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.565781+0000) 2020-02-06T02:00:09.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:09.565+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.566273+0000) 2020-02-06T02:00:09.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:09.565+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.567024+0000) 2020-02-06T02:00:09.568 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:09.565+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.566985+0000) 2020-02-06T02:00:09.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:09.566+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.567571+0000) 2020-02-06T02:00:09.569 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:09.567+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.568455+0000) 2020-02-06T02:00:09.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:09.567+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.568725+0000) 2020-02-06T02:00:09.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:09.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.569391+0000) 2020-02-06T02:00:09.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:09.567+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.569096+0000) 2020-02-06T02:00:09.571 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:09.568+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.570246+0000) 2020-02-06T02:00:09.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:09.569+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.570912+0000) 2020-02-06T02:00:09.572 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:09.570+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.571245+0000) 2020-02-06T02:00:09.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:09.575+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.576613+0000) 2020-02-06T02:00:09.589 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:09.587+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219889 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:09.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:09.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:09.778826+0000) 2020-02-06T02:00:10.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:10.024+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:10.024901+0000) 2020-02-06T02:00:10.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:10.634+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:10.635141+0000) 2020-02-06T02:00:10.894 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:00:10.901 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:00:10.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:10.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:10.913537+0000) 2020-02-06T02:00:14.595 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:14.593+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.594015+0000) 2020-02-06T02:00:14.595 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:14.594+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.594813+0000) 2020-02-06T02:00:14.596 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:14.595+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.595479+0000) 2020-02-06T02:00:14.598 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:14.595+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.596614+0000) 2020-02-06T02:00:14.598 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:14.595+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.596731+0000) 2020-02-06T02:00:14.598 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:14.597+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.597060+0000) 2020-02-06T02:00:14.599 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:14.596+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.598056+0000) 2020-02-06T02:00:14.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:14.598+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.598327+0000) 2020-02-06T02:00:14.599 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:14.597+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.598837+0000) 2020-02-06T02:00:14.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:14.598+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.599784+0000) 2020-02-06T02:00:14.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:14.598+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.600329+0000) 2020-02-06T02:00:14.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:14.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.602410+0000) 2020-02-06T02:00:14.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:14.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.602616+0000) 2020-02-06T02:00:14.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:14.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:14.602788+0000) 2020-02-06T02:00:14.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:14.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219929 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:16.157 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:16.155+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:16.156674+0000) 2020-02-06T02:00:16.520 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:16.519+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:16.519797+0000) 2020-02-06T02:00:16.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:16.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:16.744494+0000) 2020-02-06T02:00:17.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:17.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.135334+0000) 2020-02-06T02:00:17.409 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:17.407+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.408389+0000) 2020-02-06T02:00:17.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:17.407+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.408474+0000) 2020-02-06T02:00:17.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:17.407+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.408849+0000) 2020-02-06T02:00:17.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:17.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.640070+0000) 2020-02-06T02:00:17.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:17.662+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.663273+0000) 2020-02-06T02:00:17.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:17.861+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:17.862679+0000) 2020-02-06T02:00:19.624 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:19.622+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.623413+0000) 2020-02-06T02:00:19.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:19.623+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.624330+0000) 2020-02-06T02:00:19.626 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:19.624+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.625163+0000) 2020-02-06T02:00:19.626 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:19.624+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.625853+0000) 2020-02-06T02:00:19.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:19.624+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.626128+0000) 2020-02-06T02:00:19.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:19.625+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.626754+0000) 2020-02-06T02:00:19.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:19.625+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.627199+0000) 2020-02-06T02:00:19.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:19.626+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.627544+0000) 2020-02-06T02:00:19.629 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:19.627+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.628782+0000) 2020-02-06T02:00:19.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:19.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.628684+0000) 2020-02-06T02:00:19.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:19.627+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.629331+0000) 2020-02-06T02:00:19.630 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:19.628+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.630018+0000) 2020-02-06T02:00:19.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:19.630+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.631877+0000) 2020-02-06T02:00:19.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:19.631+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.632105+0000) 2020-02-06T02:00:19.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:19.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 219973 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:19.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:19.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:19.779035+0000) 2020-02-06T02:00:20.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:20.024+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:20.025129+0000) 2020-02-06T02:00:20.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:20.633+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:20.635363+0000) 2020-02-06T02:00:20.914 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:20.912+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:20.913796+0000) 2020-02-06T02:00:24.652 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:24.650+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.651773+0000) 2020-02-06T02:00:24.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:24.651+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.652572+0000) 2020-02-06T02:00:24.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:24.652+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.653299+0000) 2020-02-06T02:00:24.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:24.652+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.653810+0000) 2020-02-06T02:00:24.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:24.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.654604+0000) 2020-02-06T02:00:24.655 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:24.653+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.654556+0000) 2020-02-06T02:00:24.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:24.654+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.655574+0000) 2020-02-06T02:00:24.656 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:24.653+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.655337+0000) 2020-02-06T02:00:24.657 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:24.655+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.656352+0000) 2020-02-06T02:00:24.658 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:24.656+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.657265+0000) 2020-02-06T02:00:24.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:24.655+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.657404+0000) 2020-02-06T02:00:24.658 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:24.656+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.657511+0000) 2020-02-06T02:00:24.659 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:24.656+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.658028+0000) 2020-02-06T02:00:24.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:24.661+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:24.662896+0000) 2020-02-06T02:00:24.678 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:24.675+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220012 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:26.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:26.156+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:26.156956+0000) 2020-02-06T02:00:26.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:26.519+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:26.520037+0000) 2020-02-06T02:00:26.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:26.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:26.744759+0000) 2020-02-06T02:00:27.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:27.134+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.135567+0000) 2020-02-06T02:00:27.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:27.408+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.408717+0000) 2020-02-06T02:00:27.410 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:27.408+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.408713+0000) 2020-02-06T02:00:27.410 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:27.408+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.409059+0000) 2020-02-06T02:00:27.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:27.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.640343+0000) 2020-02-06T02:00:27.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:27.662+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.663506+0000) 2020-02-06T02:00:27.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:27.870+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:27.870977+0000) 2020-02-06T02:00:29.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:29.682+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.683267+0000) 2020-02-06T02:00:29.684 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:29.682+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.684038+0000) 2020-02-06T02:00:29.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:29.683+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.684528+0000) 2020-02-06T02:00:29.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:29.683+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.685105+0000) 2020-02-06T02:00:29.687 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:29.684+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.686120+0000) 2020-02-06T02:00:29.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:29.684+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.686196+0000) 2020-02-06T02:00:29.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:29.685+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.686593+0000) 2020-02-06T02:00:29.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:29.685+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.686944+0000) 2020-02-06T02:00:29.688 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:29.687+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.688129+0000) 2020-02-06T02:00:29.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:29.687+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.688368+0000) 2020-02-06T02:00:29.690 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:29.687+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.688654+0000) 2020-02-06T02:00:29.690 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:29.687+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.688720+0000) 2020-02-06T02:00:29.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:29.688+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.689215+0000) 2020-02-06T02:00:29.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:29.695+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.696903+0000) 2020-02-06T02:00:29.707 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:29.705+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220057 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:29.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:29.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:29.779285+0000) 2020-02-06T02:00:30.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:30.024+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:30.025352+0000) 2020-02-06T02:00:30.636 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:30.635+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:30.635648+0000) 2020-02-06T02:00:30.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:30.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:30.914028+0000) 2020-02-06T02:00:34.713 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:34.711+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.712478+0000) 2020-02-06T02:00:34.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:34.712+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.713578+0000) 2020-02-06T02:00:34.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:34.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.714175+0000) 2020-02-06T02:00:34.715 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:34.712+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.714286+0000) 2020-02-06T02:00:34.715 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:34.713+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.714734+0000) 2020-02-06T02:00:34.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:34.714+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.715344+0000) 2020-02-06T02:00:34.716 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:34.714+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.715898+0000) 2020-02-06T02:00:34.717 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:34.714+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.715934+0000) 2020-02-06T02:00:34.718 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:34.715+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.717149+0000) 2020-02-06T02:00:34.719 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:34.716+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.717810+0000) 2020-02-06T02:00:34.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:34.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.718499+0000) 2020-02-06T02:00:34.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:34.717+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.719005+0000) 2020-02-06T02:00:34.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:34.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.722225+0000) 2020-02-06T02:00:34.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:34.721+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:34.722551+0000) 2020-02-06T02:00:34.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:34.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220097 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:36.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:36.156+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:36.157188+0000) 2020-02-06T02:00:36.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:36.519+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:36.520288+0000) 2020-02-06T02:00:36.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:36.743+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:36.744992+0000) 2020-02-06T02:00:37.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:37.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.135793+0000) 2020-02-06T02:00:37.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:37.408+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.408982+0000) 2020-02-06T02:00:37.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:37.408+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.408985+0000) 2020-02-06T02:00:37.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:37.408+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.409275+0000) 2020-02-06T02:00:37.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:37.639+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.640578+0000) 2020-02-06T02:00:37.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:37.663+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.663761+0000) 2020-02-06T02:00:37.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:37.878+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:37.879181+0000) 2020-02-06T02:00:39.741 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:39.739+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.740668+0000) 2020-02-06T02:00:39.742 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:39.740+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.741686+0000) 2020-02-06T02:00:39.743 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:39.741+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.742263+0000) 2020-02-06T02:00:39.743 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:39.741+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.742535+0000) 2020-02-06T02:00:39.744 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:39.741+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.743059+0000) 2020-02-06T02:00:39.745 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:39.743+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.744226+0000) 2020-02-06T02:00:39.745 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:39.743+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.744748+0000) 2020-02-06T02:00:39.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:39.744+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.745481+0000) 2020-02-06T02:00:39.746 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:39.744+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.745499+0000) 2020-02-06T02:00:39.747 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:39.745+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.746123+0000) 2020-02-06T02:00:39.747 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:39.744+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.745936+0000) 2020-02-06T02:00:39.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:39.745+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.746701+0000) 2020-02-06T02:00:39.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:39.748+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.749925+0000) 2020-02-06T02:00:39.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:39.750+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.751322+0000) 2020-02-06T02:00:39.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:39.762+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220141 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:39.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:39.778+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:39.779534+0000) 2020-02-06T02:00:40.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:40.024+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:40.025555+0000) 2020-02-06T02:00:40.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:40.635+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:40.635920+0000) 2020-02-06T02:00:40.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:40.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:40.914301+0000) 2020-02-06T02:00:41.038 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:00:41.046 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:00:44.771 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:44.769+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.770380+0000) 2020-02-06T02:00:44.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:44.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.771317+0000) 2020-02-06T02:00:44.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:44.771+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.771797+0000) 2020-02-06T02:00:44.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:44.771+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.772663+0000) 2020-02-06T02:00:44.774 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:44.772+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.773304+0000) 2020-02-06T02:00:44.774 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:44.772+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.773278+0000) 2020-02-06T02:00:44.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:44.772+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.773940+0000) 2020-02-06T02:00:44.775 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:44.772+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.774041+0000) 2020-02-06T02:00:44.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:44.773+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.774377+0000) 2020-02-06T02:00:44.775 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:44.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.774637+0000) 2020-02-06T02:00:44.776 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:44.774+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.775060+0000) 2020-02-06T02:00:44.776 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:44.774+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.774816+0000) 2020-02-06T02:00:44.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:44.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.775613+0000) 2020-02-06T02:00:44.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:44.780+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:44.781417+0000) 2020-02-06T02:00:44.794 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:44.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220179 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:46.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:46.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:46.157478+0000) 2020-02-06T02:00:46.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:46.520+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:46.520546+0000) 2020-02-06T02:00:46.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:46.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:46.745224+0000) 2020-02-06T02:00:47.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:47.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.136026+0000) 2020-02-06T02:00:47.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:47.408+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.409227+0000) 2020-02-06T02:00:47.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:47.408+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.409255+0000) 2020-02-06T02:00:47.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:47.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.409496+0000) 2020-02-06T02:00:47.641 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:47.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.640805+0000) 2020-02-06T02:00:47.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:47.663+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.663986+0000) 2020-02-06T02:00:47.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:47.885+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:47.886065+0000) 2020-02-06T02:00:49.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:49.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.779731+0000) 2020-02-06T02:00:49.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:49.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.799426+0000) 2020-02-06T02:00:49.801 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:49.799+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.800156+0000) 2020-02-06T02:00:49.801 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:49.799+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.800449+0000) 2020-02-06T02:00:49.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:49.800+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.800865+0000) 2020-02-06T02:00:49.802 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:49.801+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.802165+0000) 2020-02-06T02:00:49.803 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:49.802+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.802643+0000) 2020-02-06T02:00:49.804 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:49.802+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.803066+0000) 2020-02-06T02:00:49.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:49.803+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.804278+0000) 2020-02-06T02:00:49.805 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:49.803+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.803908+0000) 2020-02-06T02:00:49.805 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:49.804+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.804620+0000) 2020-02-06T02:00:49.806 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:49.804+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.805216+0000) 2020-02-06T02:00:49.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:49.804+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.805403+0000) 2020-02-06T02:00:49.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:49.806+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.807567+0000) 2020-02-06T02:00:49.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:49.807+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:49.808663+0000) 2020-02-06T02:00:49.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:49.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:50.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:50.025+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:50.025763+0000) 2020-02-06T02:00:50.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:50.635+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:50.636136+0000) 2020-02-06T02:00:50.915 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:50.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:50.914593+0000) 2020-02-06T02:00:54.831 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:54.829+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.830208+0000) 2020-02-06T02:00:54.831 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:54.829+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.830640+0000) 2020-02-06T02:00:54.832 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:54.830+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.831112+0000) 2020-02-06T02:00:54.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:54.830+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.831889+0000) 2020-02-06T02:00:54.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:54.830+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.831986+0000) 2020-02-06T02:00:54.833 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:54.831+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.832335+0000) 2020-02-06T02:00:54.833 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:54.831+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.832498+0000) 2020-02-06T02:00:54.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:54.832+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.832769+0000) 2020-02-06T02:00:54.834 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:54.832+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.832896+0000) 2020-02-06T02:00:54.835 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:54.832+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.833274+0000) 2020-02-06T02:00:54.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:54.833+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.833720+0000) 2020-02-06T02:00:54.835 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:54.833+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.833679+0000) 2020-02-06T02:00:54.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:54.834+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.834802+0000) 2020-02-06T02:00:54.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:54.838+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:54.840058+0000) 2020-02-06T02:00:54.854 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:54.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:00:56.158 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:56.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:56.157743+0000) 2020-02-06T02:00:56.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:56.520+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:56.520802+0000) 2020-02-06T02:00:56.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:56.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:56.745423+0000) 2020-02-06T02:00:57.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:57.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.136395+0000) 2020-02-06T02:00:57.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:57.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.409481+0000) 2020-02-06T02:00:57.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:57.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.409488+0000) 2020-02-06T02:00:57.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:57.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.409692+0000) 2020-02-06T02:00:57.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:57.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.641008+0000) 2020-02-06T02:00:57.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:57.663+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.664238+0000) 2020-02-06T02:00:57.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:57.894+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:57.894324+0000) 2020-02-06T02:00:59.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:59.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.779967+0000) 2020-02-06T02:00:59.860 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:00:59.858+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.859231+0000) 2020-02-06T02:00:59.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:00:59.859+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.860294+0000) 2020-02-06T02:00:59.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:59.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.860072+0000) 2020-02-06T02:00:59.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:00:59.859+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.860670+0000) 2020-02-06T02:00:59.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:00:59.861+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.861745+0000) 2020-02-06T02:00:59.863 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:00:59.861+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.862335+0000) 2020-02-06T02:00:59.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:00:59.861+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.862372+0000) 2020-02-06T02:00:59.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:00:59.862+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.863235+0000) 2020-02-06T02:00:59.866 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:00:59.862+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.863405+0000) 2020-02-06T02:00:59.866 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:00:59.863+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.864157+0000) 2020-02-06T02:00:59.866 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:00:59.864+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.864897+0000) 2020-02-06T02:00:59.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:59.862+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.863669+0000) 2020-02-06T02:00:59.867 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:00:59.863+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.864289+0000) 2020-02-06T02:00:59.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:00:59.867+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:00:59.868951+0000) 2020-02-06T02:00:59.882 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:00:59.881+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:00.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:00.025+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:00.026020+0000) 2020-02-06T02:01:00.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:00.636+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:00.636345+0000) 2020-02-06T02:01:00.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:00.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:00.914817+0000) 2020-02-06T02:01:04.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:04.886+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.887644+0000) 2020-02-06T02:01:04.889 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:04.887+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.888182+0000) 2020-02-06T02:01:04.889 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:04.887+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.888519+0000) 2020-02-06T02:01:04.890 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:04.888+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.889341+0000) 2020-02-06T02:01:04.891 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:04.889+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.890172+0000) 2020-02-06T02:01:04.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:04.890+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.890787+0000) 2020-02-06T02:01:04.892 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:04.890+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.891088+0000) 2020-02-06T02:01:04.893 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:04.891+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.892528+0000) 2020-02-06T02:01:04.893 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:04.891+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.892672+0000) 2020-02-06T02:01:04.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:04.892+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.893220+0000) 2020-02-06T02:01:04.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:04.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.893469+0000) 2020-02-06T02:01:04.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:04.893+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.893640+0000) 2020-02-06T02:01:04.895 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:04.893+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.893904+0000) 2020-02-06T02:01:04.896 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:04.894+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:04.895491+0000) 2020-02-06T02:01:04.913 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:04.911+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220344 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:06.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:06.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:06.158007+0000) 2020-02-06T02:01:06.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:06.520+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:06.521044+0000) 2020-02-06T02:01:06.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:06.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:06.745653+0000) 2020-02-06T02:01:07.137 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:07.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.136627+0000) 2020-02-06T02:01:07.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:07.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.409666+0000) 2020-02-06T02:01:07.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:07.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.409681+0000) 2020-02-06T02:01:07.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:07.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.409948+0000) 2020-02-06T02:01:07.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:07.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.641267+0000) 2020-02-06T02:01:07.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:07.664+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.664456+0000) 2020-02-06T02:01:07.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:07.902+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:07.902481+0000) 2020-02-06T02:01:09.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:09.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.780200+0000) 2020-02-06T02:01:09.919 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:09.917+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.918244+0000) 2020-02-06T02:01:09.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:09.919+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.919221+0000) 2020-02-06T02:01:09.920 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:09.918+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.919136+0000) 2020-02-06T02:01:09.921 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:09.920+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.920202+0000) 2020-02-06T02:01:09.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:09.921+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.921599+0000) 2020-02-06T02:01:09.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:09.921+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.921809+0000) 2020-02-06T02:01:09.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:09.921+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.921813+0000) 2020-02-06T02:01:09.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:09.922+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.922317+0000) 2020-02-06T02:01:09.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:09.921+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.921773+0000) 2020-02-06T02:01:09.924 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:09.922+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.922422+0000) 2020-02-06T02:01:09.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:09.921+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.922298+0000) 2020-02-06T02:01:09.925 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:09.922+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.922885+0000) 2020-02-06T02:01:09.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:09.923+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.923170+0000) 2020-02-06T02:01:09.929 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:09.928+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:09.928312+0000) 2020-02-06T02:01:09.941 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:09.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:10.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:10.026+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:10.026253+0000) 2020-02-06T02:01:10.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:10.636+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:10.636575+0000) 2020-02-06T02:01:10.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:10.913+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:10.915032+0000) 2020-02-06T02:01:11.186 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:01:11.193 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:01:14.947 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:14.945+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.945537+0000) 2020-02-06T02:01:14.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:14.945+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.947053+0000) 2020-02-06T02:01:14.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:14.946+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.947556+0000) 2020-02-06T02:01:14.949 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:14.947+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.948372+0000) 2020-02-06T02:01:14.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:14.948+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.949350+0000) 2020-02-06T02:01:14.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:14.948+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.949071+0000) 2020-02-06T02:01:14.951 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:14.948+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.949858+0000) 2020-02-06T02:01:14.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:14.949+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.950282+0000) 2020-02-06T02:01:14.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:14.950+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.951021+0000) 2020-02-06T02:01:14.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:14.950+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.951407+0000) 2020-02-06T02:01:14.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:14.951+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.951974+0000) 2020-02-06T02:01:14.953 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:14.951+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.952383+0000) 2020-02-06T02:01:14.954 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:14.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.952890+0000) 2020-02-06T02:01:14.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:14.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:14.958600+0000) 2020-02-06T02:01:14.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:14.968+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220428 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:16.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:16.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:16.158288+0000) 2020-02-06T02:01:16.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:16.521+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:16.521296+0000) 2020-02-06T02:01:16.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:16.744+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:16.745885+0000) 2020-02-06T02:01:17.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:17.135+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.136879+0000) 2020-02-06T02:01:17.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:17.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.409915+0000) 2020-02-06T02:01:17.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:17.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.409929+0000) 2020-02-06T02:01:17.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:17.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.410263+0000) 2020-02-06T02:01:17.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:17.640+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.641501+0000) 2020-02-06T02:01:17.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:17.663+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.664682+0000) 2020-02-06T02:01:17.914 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:17.909+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:17.910597+0000) 2020-02-06T02:01:19.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:19.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.780393+0000) 2020-02-06T02:01:19.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:19.972+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.973089+0000) 2020-02-06T02:01:19.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:19.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.975030+0000) 2020-02-06T02:01:19.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:19.974+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.975535+0000) 2020-02-06T02:01:19.976 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:19.974+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.975665+0000) 2020-02-06T02:01:19.977 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:19.975+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.976038+0000) 2020-02-06T02:01:19.978 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:19.976+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.976857+0000) 2020-02-06T02:01:19.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:19.976+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.977314+0000) 2020-02-06T02:01:19.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:19.976+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.977653+0000) 2020-02-06T02:01:19.979 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:19.977+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.978606+0000) 2020-02-06T02:01:19.979 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:19.978+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.978524+0000) 2020-02-06T02:01:19.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:19.977+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.978920+0000) 2020-02-06T02:01:19.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:19.978+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.979223+0000) 2020-02-06T02:01:19.980 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:19.979+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.979764+0000) 2020-02-06T02:01:19.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:19.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:19.984600+0000) 2020-02-06T02:01:19.998 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:19.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220474 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:20.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:20.025+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:20.026539+0000) 2020-02-06T02:01:20.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:20.635+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:20.636800+0000) 2020-02-06T02:01:20.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:20.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:20.915333+0000) 2020-02-06T02:01:25.003 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:25.001+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.002651+0000) 2020-02-06T02:01:25.004 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:25.001+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.002965+0000) 2020-02-06T02:01:25.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:25.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.004630+0000) 2020-02-06T02:01:25.006 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:25.004+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.005174+0000) 2020-02-06T02:01:25.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:25.004+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.005348+0000) 2020-02-06T02:01:25.006 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:25.004+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.005887+0000) 2020-02-06T02:01:25.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:25.005+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.006572+0000) 2020-02-06T02:01:25.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:25.006+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.007480+0000) 2020-02-06T02:01:25.008 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:25.006+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.007837+0000) 2020-02-06T02:01:25.009 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:25.007+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.008418+0000) 2020-02-06T02:01:25.010 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:25.007+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.009388+0000) 2020-02-06T02:01:25.010 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:25.008+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.010095+0000) 2020-02-06T02:01:25.011 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:25.009+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.010515+0000) 2020-02-06T02:01:25.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:25.010+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:25.011935+0000) 2020-02-06T02:01:25.028 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:25.025+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220510 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:26.159 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:26.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:26.158555+0000) 2020-02-06T02:01:26.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:26.520+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:26.521645+0000) 2020-02-06T02:01:26.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:26.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:26.746136+0000) 2020-02-06T02:01:27.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:27.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.137133+0000) 2020-02-06T02:01:27.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:27.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.410214+0000) 2020-02-06T02:01:27.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:27.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.410214+0000) 2020-02-06T02:01:27.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:27.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.410490+0000) 2020-02-06T02:01:27.642 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:27.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.641807+0000) 2020-02-06T02:01:27.665 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:27.664+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.664910+0000) 2020-02-06T02:01:27.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:27.918+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:27.918939+0000) 2020-02-06T02:01:29.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:29.779+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:29.780631+0000) 2020-02-06T02:01:30.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:30.025+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.026798+0000) 2020-02-06T02:01:30.033 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:30.031+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.032398+0000) 2020-02-06T02:01:30.034 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:30.032+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.033330+0000) 2020-02-06T02:01:30.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:30.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.033896+0000) 2020-02-06T02:01:30.035 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:30.033+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.034566+0000) 2020-02-06T02:01:30.036 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:30.034+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.035207+0000) 2020-02-06T02:01:30.036 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:30.034+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.035724+0000) 2020-02-06T02:01:30.037 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:30.034+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.036311+0000) 2020-02-06T02:01:30.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:30.035+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.036754+0000) 2020-02-06T02:01:30.038 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:30.035+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.037263+0000) 2020-02-06T02:01:30.038 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:30.036+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.037592+0000) 2020-02-06T02:01:30.039 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:30.036+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.038019+0000) 2020-02-06T02:01:30.039 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:30.037+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.038419+0000) 2020-02-06T02:01:30.039 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:30.037+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.039244+0000) 2020-02-06T02:01:30.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:30.041+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.042966+0000) 2020-02-06T02:01:30.058 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:30.056+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:30.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:30.636+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.637080+0000) 2020-02-06T02:01:30.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:30.914+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:30.915635+0000) 2020-02-06T02:01:35.063 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:35.060+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.061842+0000) 2020-02-06T02:01:35.064 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:35.062+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.063879+0000) 2020-02-06T02:01:35.065 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:35.062+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.063960+0000) 2020-02-06T02:01:35.065 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:35.063+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.064461+0000) 2020-02-06T02:01:35.066 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:35.064+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.065628+0000) 2020-02-06T02:01:35.066 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:35.064+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.066043+0000) 2020-02-06T02:01:35.067 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:35.065+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.066721+0000) 2020-02-06T02:01:35.067 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:35.065+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.066830+0000) 2020-02-06T02:01:35.068 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:35.066+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.067590+0000) 2020-02-06T02:01:35.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:35.066+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.067418+0000) 2020-02-06T02:01:35.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:35.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.067518+0000) 2020-02-06T02:01:35.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:35.066+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.067929+0000) 2020-02-06T02:01:35.069 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:35.066+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.067776+0000) 2020-02-06T02:01:35.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:35.069+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:35.070510+0000) 2020-02-06T02:01:35.086 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:35.084+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220594 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:36.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:36.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:36.158818+0000) 2020-02-06T02:01:36.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:36.521+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:36.521900+0000) 2020-02-06T02:01:36.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:36.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:36.746386+0000) 2020-02-06T02:01:37.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:37.136+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.137394+0000) 2020-02-06T02:01:37.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:37.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.410519+0000) 2020-02-06T02:01:37.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:37.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.410496+0000) 2020-02-06T02:01:37.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:37.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.410759+0000) 2020-02-06T02:01:37.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:37.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.642036+0000) 2020-02-06T02:01:37.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:37.664+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.665179+0000) 2020-02-06T02:01:37.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:37.929+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:37.930386+0000) 2020-02-06T02:01:39.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:39.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:39.780866+0000) 2020-02-06T02:01:40.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:40.026+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.027031+0000) 2020-02-06T02:01:40.093 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:40.090+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.092010+0000) 2020-02-06T02:01:40.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:40.091+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.093008+0000) 2020-02-06T02:01:40.094 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:40.092+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.093401+0000) 2020-02-06T02:01:40.095 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:40.093+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.094484+0000) 2020-02-06T02:01:40.096 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:40.093+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.094640+0000) 2020-02-06T02:01:40.096 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:40.093+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.095087+0000) 2020-02-06T02:01:40.097 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:40.094+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.095741+0000) 2020-02-06T02:01:40.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:40.094+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.095985+0000) 2020-02-06T02:01:40.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:40.095+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.096348+0000) 2020-02-06T02:01:40.098 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:40.096+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.097408+0000) 2020-02-06T02:01:40.099 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:40.095+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.096808+0000) 2020-02-06T02:01:40.099 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:40.095+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.097154+0000) 2020-02-06T02:01:40.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:40.095+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.097381+0000) 2020-02-06T02:01:40.102 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:40.100+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.101628+0000) 2020-02-06T02:01:40.115 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:40.113+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220640 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:40.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:40.636+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.637295+0000) 2020-02-06T02:01:40.916 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:40.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:40.915837+0000) 2020-02-06T02:01:41.349 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:01:41.355 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:01:45.121 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:45.119+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.120245+0000) 2020-02-06T02:01:45.122 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:45.120+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.121102+0000) 2020-02-06T02:01:45.122 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:45.120+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.121640+0000) 2020-02-06T02:01:45.123 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:45.120+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.122026+0000) 2020-02-06T02:01:45.123 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:45.121+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.122288+0000) 2020-02-06T02:01:45.124 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:45.121+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.123216+0000) 2020-02-06T02:01:45.124 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:45.121+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.123210+0000) 2020-02-06T02:01:45.125 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:45.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.124005+0000) 2020-02-06T02:01:45.125 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:45.123+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.124160+0000) 2020-02-06T02:01:45.125 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:45.123+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.124666+0000) 2020-02-06T02:01:45.126 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:45.123+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.124698+0000) 2020-02-06T02:01:45.126 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:45.124+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.125252+0000) 2020-02-06T02:01:45.127 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:45.124+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.125813+0000) 2020-02-06T02:01:45.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:45.130+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:45.132051+0000) 2020-02-06T02:01:45.145 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:45.143+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220676 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:46.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:46.157+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:46.159065+0000) 2020-02-06T02:01:46.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:46.520+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:46.522145+0000) 2020-02-06T02:01:46.747 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:46.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:46.746638+0000) 2020-02-06T02:01:47.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:47.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.137625+0000) 2020-02-06T02:01:47.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:47.409+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.410689+0000) 2020-02-06T02:01:47.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:47.409+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.410720+0000) 2020-02-06T02:01:47.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:47.409+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.410968+0000) 2020-02-06T02:01:47.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:47.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.642303+0000) 2020-02-06T02:01:47.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:47.665+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.665407+0000) 2020-02-06T02:01:47.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:47.937+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:47.938692+0000) 2020-02-06T02:01:49.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:49.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:49.781128+0000) 2020-02-06T02:01:50.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:50.026+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.027246+0000) 2020-02-06T02:01:50.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:50.149+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.150195+0000) 2020-02-06T02:01:50.151 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:50.149+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.150875+0000) 2020-02-06T02:01:50.152 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:50.150+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.151180+0000) 2020-02-06T02:01:50.152 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:50.150+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.151448+0000) 2020-02-06T02:01:50.153 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:50.150+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.151684+0000) 2020-02-06T02:01:50.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:50.150+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.151998+0000) 2020-02-06T02:01:50.154 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:50.151+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.152969+0000) 2020-02-06T02:01:50.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:50.151+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.152596+0000) 2020-02-06T02:01:50.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:50.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.153101+0000) 2020-02-06T02:01:50.155 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:50.151+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.153358+0000) 2020-02-06T02:01:50.155 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:50.152+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.153791+0000) 2020-02-06T02:01:50.155 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:50.153+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.154817+0000) 2020-02-06T02:01:50.156 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:50.153+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.154767+0000) 2020-02-06T02:01:50.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:50.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.159217+0000) 2020-02-06T02:01:50.175 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:50.173+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220723 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:50.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:50.636+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.637566+0000) 2020-02-06T02:01:50.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:50.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:50.916040+0000) 2020-02-06T02:01:55.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:55.178+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.179971+0000) 2020-02-06T02:01:55.182 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:55.180+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.181606+0000) 2020-02-06T02:01:55.182 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:55.180+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.181512+0000) 2020-02-06T02:01:55.183 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:55.181+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.182187+0000) 2020-02-06T02:01:55.183 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:01:55.181+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.182521+0000) 2020-02-06T02:01:55.184 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:01:55.181+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.183036+0000) 2020-02-06T02:01:55.184 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:55.182+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.183485+0000) 2020-02-06T02:01:55.184 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:55.182+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.183710+0000) 2020-02-06T02:01:55.185 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:55.182+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.184253+0000) 2020-02-06T02:01:55.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:55.183+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.184776+0000) 2020-02-06T02:01:55.186 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:55.184+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.185545+0000) 2020-02-06T02:01:55.186 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:55.184+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.185755+0000) 2020-02-06T02:01:55.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:55.184+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.186297+0000) 2020-02-06T02:01:55.192 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:55.190+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:55.191124+0000) 2020-02-06T02:01:55.205 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:01:55.202+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220760 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:01:56.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:01:56.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:56.159327+0000) 2020-02-06T02:01:56.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:01:56.521+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:56.522396+0000) 2020-02-06T02:01:56.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:01:56.745+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:56.746871+0000) 2020-02-06T02:01:57.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:01:57.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.137843+0000) 2020-02-06T02:01:57.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:01:57.410+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.410956+0000) 2020-02-06T02:01:57.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:01:57.410+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.410957+0000) 2020-02-06T02:01:57.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:01:57.410+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.411203+0000) 2020-02-06T02:01:57.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:01:57.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.642517+0000) 2020-02-06T02:01:57.666 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:01:57.665+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.665663+0000) 2020-02-06T02:01:57.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:57.946+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:57.947026+0000) 2020-02-06T02:01:59.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:01:59.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:01:59.781388+0000) 2020-02-06T02:02:00.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:00.027+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.027496+0000) 2020-02-06T02:02:00.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:00.209+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.210239+0000) 2020-02-06T02:02:00.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:00.210+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.211158+0000) 2020-02-06T02:02:00.213 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:00.210+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.212012+0000) 2020-02-06T02:02:00.213 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:00.211+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.212717+0000) 2020-02-06T02:02:00.213 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:00.211+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.213055+0000) 2020-02-06T02:02:00.214 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:00.212+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.213684+0000) 2020-02-06T02:02:00.214 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:00.212+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.213802+0000) 2020-02-06T02:02:00.215 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:00.212+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.213706+0000) 2020-02-06T02:02:00.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:00.213+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.214533+0000) 2020-02-06T02:02:00.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:00.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.214599+0000) 2020-02-06T02:02:00.216 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:00.213+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.214878+0000) 2020-02-06T02:02:00.216 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:00.213+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.215085+0000) 2020-02-06T02:02:00.216 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:00.214+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.215848+0000) 2020-02-06T02:02:00.221 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:00.219+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.220606+0000) 2020-02-06T02:02:00.234 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:00.231+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220806 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:00.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:00.637+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.637803+0000) 2020-02-06T02:02:00.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:00.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:00.916285+0000) 2020-02-06T02:02:05.239 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:05.238+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.238703+0000) 2020-02-06T02:02:05.240 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:05.239+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.239685+0000) 2020-02-06T02:02:05.241 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:05.240+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.240254+0000) 2020-02-06T02:02:05.241 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:05.240+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.240768+0000) 2020-02-06T02:02:05.243 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:05.240+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.241723+0000) 2020-02-06T02:02:05.243 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:05.240+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.242127+0000) 2020-02-06T02:02:05.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:05.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.241833+0000) 2020-02-06T02:02:05.244 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:05.241+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.242772+0000) 2020-02-06T02:02:05.244 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:05.241+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.242893+0000) 2020-02-06T02:02:05.244 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:05.243+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.243660+0000) 2020-02-06T02:02:05.245 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:05.243+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.244472+0000) 2020-02-06T02:02:05.245 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:05.243+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.245091+0000) 2020-02-06T02:02:05.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:05.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.246468+0000) 2020-02-06T02:02:05.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:05.247+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:05.247343+0000) 2020-02-06T02:02:05.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:05.260+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:06.160 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:06.158+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:06.159559+0000) 2020-02-06T02:02:06.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:06.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:06.522686+0000) 2020-02-06T02:02:06.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:06.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:06.747135+0000) 2020-02-06T02:02:07.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:07.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.138068+0000) 2020-02-06T02:02:07.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:07.410+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.411229+0000) 2020-02-06T02:02:07.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:07.410+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.411199+0000) 2020-02-06T02:02:07.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:07.410+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.411478+0000) 2020-02-06T02:02:07.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:07.641+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.642732+0000) 2020-02-06T02:02:07.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:07.664+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.665915+0000) 2020-02-06T02:02:07.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:07.953+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:07.954386+0000) 2020-02-06T02:02:09.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:09.780+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:09.781651+0000) 2020-02-06T02:02:10.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:10.026+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.027721+0000) 2020-02-06T02:02:10.268 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:10.265+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.267144+0000) 2020-02-06T02:02:10.269 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:10.267+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.268926+0000) 2020-02-06T02:02:10.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:10.267+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.268814+0000) 2020-02-06T02:02:10.271 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:10.269+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.270530+0000) 2020-02-06T02:02:10.272 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:10.269+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.271250+0000) 2020-02-06T02:02:10.272 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:10.270+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.272017+0000) 2020-02-06T02:02:10.273 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:10.270+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.272286+0000) 2020-02-06T02:02:10.273 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:10.271+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.272921+0000) 2020-02-06T02:02:10.274 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:10.272+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.273526+0000) 2020-02-06T02:02:10.274 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:10.272+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.273801+0000) 2020-02-06T02:02:10.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:10.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.273767+0000) 2020-02-06T02:02:10.275 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:10.272+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.274125+0000) 2020-02-06T02:02:10.276 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:10.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.276261+0000) 2020-02-06T02:02:10.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:10.275+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.276471+0000) 2020-02-06T02:02:10.292 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:10.290+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220889 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:10.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:10.637+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.638104+0000) 2020-02-06T02:02:10.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:10.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:10.916535+0000) 2020-02-06T02:02:11.524 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:02:11.530 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:02:15.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:15.297+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.297695+0000) 2020-02-06T02:02:15.300 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:15.297+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.298060+0000) 2020-02-06T02:02:15.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:15.297+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.298808+0000) 2020-02-06T02:02:15.300 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:15.299+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.299766+0000) 2020-02-06T02:02:15.301 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:15.299+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.300401+0000) 2020-02-06T02:02:15.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:15.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.301205+0000) 2020-02-06T02:02:15.302 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:15.300+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.301063+0000) 2020-02-06T02:02:15.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:15.301+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.302367+0000) 2020-02-06T02:02:15.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:15.302+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.302722+0000) 2020-02-06T02:02:15.304 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:15.301+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.302587+0000) 2020-02-06T02:02:15.304 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:15.302+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.303370+0000) 2020-02-06T02:02:15.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:15.302+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.303275+0000) 2020-02-06T02:02:15.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:15.303+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.303941+0000) 2020-02-06T02:02:15.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:15.306+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:15.307695+0000) 2020-02-06T02:02:15.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:15.320+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220926 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:16.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:16.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:16.159785+0000) 2020-02-06T02:02:16.524 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:16.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:16.522918+0000) 2020-02-06T02:02:16.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:16.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:16.747402+0000) 2020-02-06T02:02:17.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:17.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.138344+0000) 2020-02-06T02:02:17.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:17.411+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.411494+0000) 2020-02-06T02:02:17.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:17.411+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.411486+0000) 2020-02-06T02:02:17.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:17.411+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.411687+0000) 2020-02-06T02:02:17.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:17.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.642964+0000) 2020-02-06T02:02:17.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:17.665+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.666140+0000) 2020-02-06T02:02:17.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:17.961+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:17.962725+0000) 2020-02-06T02:02:19.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:19.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:19.781904+0000) 2020-02-06T02:02:20.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:20.027+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.027984+0000) 2020-02-06T02:02:20.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:20.324+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.325683+0000) 2020-02-06T02:02:20.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:20.326+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.327268+0000) 2020-02-06T02:02:20.329 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:20.327+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.328269+0000) 2020-02-06T02:02:20.330 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:20.327+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.328747+0000) 2020-02-06T02:02:20.330 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:20.328+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.329663+0000) 2020-02-06T02:02:20.332 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:20.329+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.330664+0000) 2020-02-06T02:02:20.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:20.330+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.330697+0000) 2020-02-06T02:02:20.332 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:20.330+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.330883+0000) 2020-02-06T02:02:20.333 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:20.330+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.331140+0000) 2020-02-06T02:02:20.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:20.330+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.331375+0000) 2020-02-06T02:02:20.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:20.330+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.331331+0000) 2020-02-06T02:02:20.334 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:20.330+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.332011+0000) 2020-02-06T02:02:20.334 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:20.332+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.332890+0000) 2020-02-06T02:02:20.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:20.333+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.334223+0000) 2020-02-06T02:02:20.350 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:20.348+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 220972 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:20.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:20.637+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.638319+0000) 2020-02-06T02:02:20.917 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:20.915+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:20.916747+0000) 2020-02-06T02:02:25.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:25.353+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.354197+0000) 2020-02-06T02:02:25.355 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:25.353+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.354967+0000) 2020-02-06T02:02:25.356 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:25.354+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.355596+0000) 2020-02-06T02:02:25.357 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:25.355+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.356628+0000) 2020-02-06T02:02:25.358 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:25.356+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.357130+0000) 2020-02-06T02:02:25.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:25.356+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.357590+0000) 2020-02-06T02:02:25.359 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:25.357+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.358067+0000) 2020-02-06T02:02:25.359 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:25.357+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.358394+0000) 2020-02-06T02:02:25.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:25.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.359050+0000) 2020-02-06T02:02:25.360 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:25.359+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.359470+0000) 2020-02-06T02:02:25.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:25.359+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.359862+0000) 2020-02-06T02:02:25.362 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:25.360+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.361049+0000) 2020-02-06T02:02:25.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:25.360+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.360931+0000) 2020-02-06T02:02:25.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:25.361+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:25.362506+0000) 2020-02-06T02:02:25.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:25.377+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:26.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:26.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:26.160030+0000) 2020-02-06T02:02:26.524 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:26.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:26.523165+0000) 2020-02-06T02:02:26.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:26.746+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:26.747632+0000) 2020-02-06T02:02:27.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:27.137+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.138577+0000) 2020-02-06T02:02:27.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:27.411+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.411775+0000) 2020-02-06T02:02:27.413 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:27.411+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.411750+0000) 2020-02-06T02:02:27.413 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:27.411+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.411929+0000) 2020-02-06T02:02:27.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:27.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.643203+0000) 2020-02-06T02:02:27.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:27.665+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.666373+0000) 2020-02-06T02:02:27.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:27.970+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:27.971043+0000) 2020-02-06T02:02:29.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:29.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:29.782168+0000) 2020-02-06T02:02:30.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:30.027+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.028212+0000) 2020-02-06T02:02:30.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:30.381+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.382868+0000) 2020-02-06T02:02:30.385 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:30.383+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.384279+0000) 2020-02-06T02:02:30.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:30.384+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.385429+0000) 2020-02-06T02:02:30.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:30.384+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.385945+0000) 2020-02-06T02:02:30.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:30.385+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.386138+0000) 2020-02-06T02:02:30.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:30.386+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.386536+0000) 2020-02-06T02:02:30.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:30.385+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.386718+0000) 2020-02-06T02:02:30.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:30.386+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.387136+0000) 2020-02-06T02:02:30.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:30.386+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.386967+0000) 2020-02-06T02:02:30.389 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:30.386+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.387775+0000) 2020-02-06T02:02:30.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:30.387+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.387547+0000) 2020-02-06T02:02:30.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:30.387+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.387866+0000) 2020-02-06T02:02:30.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:30.388+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.388490+0000) 2020-02-06T02:02:30.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:30.391+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.392610+0000) 2020-02-06T02:02:30.406 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:30.405+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221055 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:30.639 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:30.638+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.638605+0000) 2020-02-06T02:02:30.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:30.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:30.916960+0000) 2020-02-06T02:02:35.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:35.411+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.412194+0000) 2020-02-06T02:02:35.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:35.411+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.412595+0000) 2020-02-06T02:02:35.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:35.412+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.413241+0000) 2020-02-06T02:02:35.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:35.413+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.413554+0000) 2020-02-06T02:02:35.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:35.413+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.413935+0000) 2020-02-06T02:02:35.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:35.413+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.414401+0000) 2020-02-06T02:02:35.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:35.414+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.414578+0000) 2020-02-06T02:02:35.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:35.414+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.415019+0000) 2020-02-06T02:02:35.417 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:35.414+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.415315+0000) 2020-02-06T02:02:35.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:35.414+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.415429+0000) 2020-02-06T02:02:35.418 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:35.415+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.416059+0000) 2020-02-06T02:02:35.418 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:35.415+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.416728+0000) 2020-02-06T02:02:35.419 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:35.416+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.417110+0000) 2020-02-06T02:02:35.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:35.420+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:35.421448+0000) 2020-02-06T02:02:35.435 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:35.434+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221091 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:36.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:36.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:36.160265+0000) 2020-02-06T02:02:36.524 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:36.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:36.523367+0000) 2020-02-06T02:02:36.748 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:36.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:36.747859+0000) 2020-02-06T02:02:37.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:37.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.138807+0000) 2020-02-06T02:02:37.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:37.411+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.412105+0000) 2020-02-06T02:02:37.413 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:37.411+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.412035+0000) 2020-02-06T02:02:37.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:37.411+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.412208+0000) 2020-02-06T02:02:37.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:37.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.643391+0000) 2020-02-06T02:02:37.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:37.666+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.666592+0000) 2020-02-06T02:02:37.979 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:37.978+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:37.978487+0000) 2020-02-06T02:02:39.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:39.781+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:39.782428+0000) 2020-02-06T02:02:40.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:40.028+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.028447+0000) 2020-02-06T02:02:40.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:40.438+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.440059+0000) 2020-02-06T02:02:40.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:40.439+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.440570+0000) 2020-02-06T02:02:40.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:40.441+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.442116+0000) 2020-02-06T02:02:40.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:40.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.441009+0000) 2020-02-06T02:02:40.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:40.440+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.441473+0000) 2020-02-06T02:02:40.451 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:40.441+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.443048+0000) 2020-02-06T02:02:40.451 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:40.443+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.443647+0000) 2020-02-06T02:02:40.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:40.443+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.444088+0000) 2020-02-06T02:02:40.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:40.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.444572+0000) 2020-02-06T02:02:40.453 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:40.444+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.445143+0000) 2020-02-06T02:02:40.453 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:40.445+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.445632+0000) 2020-02-06T02:02:40.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:40.445+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.445879+0000) 2020-02-06T02:02:40.455 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:40.445+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.446184+0000) 2020-02-06T02:02:40.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:40.450+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.451229+0000) 2020-02-06T02:02:40.471 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:40.464+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221137 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:40.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:40.638+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.638837+0000) 2020-02-06T02:02:40.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:40.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:40.917158+0000) 2020-02-06T02:02:41.668 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:02:41.675 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:02:45.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:45.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.471387+0000) 2020-02-06T02:02:45.473 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:45.470+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.472012+0000) 2020-02-06T02:02:45.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:45.471+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.472372+0000) 2020-02-06T02:02:45.474 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:45.471+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.473039+0000) 2020-02-06T02:02:45.474 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:45.472+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.473485+0000) 2020-02-06T02:02:45.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:45.473+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.474596+0000) 2020-02-06T02:02:45.476 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:45.474+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.474876+0000) 2020-02-06T02:02:45.476 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:45.474+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.474942+0000) 2020-02-06T02:02:45.476 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:45.474+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.475365+0000) 2020-02-06T02:02:45.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:45.474+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.476024+0000) 2020-02-06T02:02:45.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:45.475+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.476349+0000) 2020-02-06T02:02:45.477 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:45.475+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.476279+0000) 2020-02-06T02:02:45.478 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:45.476+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.477058+0000) 2020-02-06T02:02:45.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:45.479+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:45.480828+0000) 2020-02-06T02:02:45.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:45.493+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221173 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:46.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:46.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:46.160481+0000) 2020-02-06T02:02:46.524 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:46.523+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:46.523698+0000) 2020-02-06T02:02:46.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:46.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:46.748134+0000) 2020-02-06T02:02:47.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:47.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.139042+0000) 2020-02-06T02:02:47.413 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:47.412+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.412361+0000) 2020-02-06T02:02:47.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:47.412+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.412369+0000) 2020-02-06T02:02:47.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:47.412+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.412522+0000) 2020-02-06T02:02:47.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:47.642+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.643632+0000) 2020-02-06T02:02:47.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:47.666+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.666820+0000) 2020-02-06T02:02:47.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:47.986+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:47.986774+0000) 2020-02-06T02:02:49.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:49.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:49.782676+0000) 2020-02-06T02:02:50.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:50.028+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.028671+0000) 2020-02-06T02:02:50.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:50.498+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.499402+0000) 2020-02-06T02:02:50.501 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:50.499+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.500285+0000) 2020-02-06T02:02:50.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:50.500+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.500724+0000) 2020-02-06T02:02:50.502 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:50.501+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.501961+0000) 2020-02-06T02:02:50.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:50.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.502229+0000) 2020-02-06T02:02:50.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:50.502+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.503434+0000) 2020-02-06T02:02:50.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:50.503+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.504571+0000) 2020-02-06T02:02:50.508 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:50.503+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.504821+0000) 2020-02-06T02:02:50.508 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:50.504+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.505238+0000) 2020-02-06T02:02:50.508 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:50.502+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.503023+0000) 2020-02-06T02:02:50.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:50.504+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.505295+0000) 2020-02-06T02:02:50.508 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:50.505+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.505641+0000) 2020-02-06T02:02:50.509 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:50.505+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.505915+0000) 2020-02-06T02:02:50.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:50.508+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.509720+0000) 2020-02-06T02:02:50.524 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:50.523+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221220 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:50.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:50.639+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.639078+0000) 2020-02-06T02:02:50.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:50.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:50.917372+0000) 2020-02-06T02:02:55.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:55.529+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.529294+0000) 2020-02-06T02:02:55.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:55.529+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.529897+0000) 2020-02-06T02:02:55.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:55.530+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.530557+0000) 2020-02-06T02:02:55.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:55.529+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.531131+0000) 2020-02-06T02:02:55.532 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:55.530+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.531972+0000) 2020-02-06T02:02:55.533 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:55.530+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.532368+0000) 2020-02-06T02:02:55.533 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:02:55.532+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.532679+0000) 2020-02-06T02:02:55.534 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:55.533+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.533621+0000) 2020-02-06T02:02:55.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:55.533+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.533337+0000) 2020-02-06T02:02:55.535 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:55.532+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.534128+0000) 2020-02-06T02:02:55.535 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:55.533+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.534613+0000) 2020-02-06T02:02:55.536 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:02:55.533+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.534914+0000) 2020-02-06T02:02:55.536 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:55.533+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.535101+0000) 2020-02-06T02:02:55.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:55.537+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:55.538427+0000) 2020-02-06T02:02:55.553 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:02:55.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221256 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:02:56.161 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:02:56.159+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:56.160706+0000) 2020-02-06T02:02:56.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:02:56.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:56.523962+0000) 2020-02-06T02:02:56.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:02:56.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:56.748352+0000) 2020-02-06T02:02:57.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:02:57.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.139358+0000) 2020-02-06T02:02:57.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:02:57.411+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.412660+0000) 2020-02-06T02:02:57.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:02:57.411+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.412660+0000) 2020-02-06T02:02:57.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:02:57.411+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.412768+0000) 2020-02-06T02:02:57.644 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:02:57.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.643864+0000) 2020-02-06T02:02:57.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:02:57.667+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.667137+0000) 2020-02-06T02:02:57.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:57.995+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:57.995224+0000) 2020-02-06T02:02:59.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:02:59.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:02:59.782877+0000) 2020-02-06T02:03:00.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:00.027+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.028898+0000) 2020-02-06T02:03:00.558 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:00.556+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.557561+0000) 2020-02-06T02:03:00.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:00.557+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.558073+0000) 2020-02-06T02:03:00.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:00.557+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.558644+0000) 2020-02-06T02:03:00.564 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:00.560+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.562161+0000) 2020-02-06T02:03:00.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:00.560+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.562242+0000) 2020-02-06T02:03:00.564 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:00.561+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.562665+0000) 2020-02-06T02:03:00.564 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:00.558+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.559612+0000) 2020-02-06T02:03:00.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:00.559+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.560396+0000) 2020-02-06T02:03:00.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:00.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.560952+0000) 2020-02-06T02:03:00.565 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:00.560+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.561375+0000) 2020-02-06T02:03:00.566 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:00.562+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.563927+0000) 2020-02-06T02:03:00.566 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:00.562+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.564385+0000) 2020-02-06T02:03:00.566 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:00.562+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.563521+0000) 2020-02-06T02:03:00.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:00.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.568489+0000) 2020-02-06T02:03:00.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:00.580+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221303 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:00.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:00.638+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.639294+0000) 2020-02-06T02:03:00.918 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:00.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:00.917582+0000) 2020-02-06T02:03:05.588 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:05.585+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.586427+0000) 2020-02-06T02:03:05.589 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:05.587+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.588342+0000) 2020-02-06T02:03:05.589 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:05.586+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.588289+0000) 2020-02-06T02:03:05.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:05.587+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.588879+0000) 2020-02-06T02:03:05.590 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:05.587+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.589399+0000) 2020-02-06T02:03:05.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:05.589+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.590357+0000) 2020-02-06T02:03:05.591 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:05.589+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.590651+0000) 2020-02-06T02:03:05.592 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:05.590+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.591522+0000) 2020-02-06T02:03:05.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:05.591+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.592729+0000) 2020-02-06T02:03:05.594 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:05.590+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.592338+0000) 2020-02-06T02:03:05.594 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:05.591+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.593281+0000) 2020-02-06T02:03:05.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:05.591+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.592964+0000) 2020-02-06T02:03:05.595 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:05.592+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.593604+0000) 2020-02-06T02:03:05.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:05.594+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:05.595524+0000) 2020-02-06T02:03:05.612 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:05.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221339 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:06.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:06.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:06.160940+0000) 2020-02-06T02:03:06.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:06.522+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:06.524170+0000) 2020-02-06T02:03:06.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:06.747+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:06.748623+0000) 2020-02-06T02:03:07.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:07.138+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.139582+0000) 2020-02-06T02:03:07.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:07.411+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.412906+0000) 2020-02-06T02:03:07.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:07.411+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.412917+0000) 2020-02-06T02:03:07.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:07.411+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.413053+0000) 2020-02-06T02:03:07.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:07.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.644135+0000) 2020-02-06T02:03:07.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:07.666+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:07.667388+0000) 2020-02-06T02:03:08.004 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:08.002+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:08.003445+0000) 2020-02-06T02:03:09.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:09.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:09.783140+0000) 2020-02-06T02:03:10.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:10.028+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.029126+0000) 2020-02-06T02:03:10.618 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:10.616+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.617385+0000) 2020-02-06T02:03:10.619 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:10.617+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.618424+0000) 2020-02-06T02:03:10.620 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:10.617+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.619125+0000) 2020-02-06T02:03:10.620 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:10.618+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.619964+0000) 2020-02-06T02:03:10.621 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:10.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.620223+0000) 2020-02-06T02:03:10.622 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:10.619+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.620920+0000) 2020-02-06T02:03:10.623 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:10.620+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.621291+0000) 2020-02-06T02:03:10.623 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:10.620+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.621758+0000) 2020-02-06T02:03:10.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:10.621+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.622095+0000) 2020-02-06T02:03:10.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:10.621+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.622762+0000) 2020-02-06T02:03:10.625 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:10.621+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.623214+0000) 2020-02-06T02:03:10.625 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:10.622+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.623241+0000) 2020-02-06T02:03:10.625 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:10.622+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.623846+0000) 2020-02-06T02:03:10.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:10.627+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.628832+0000) 2020-02-06T02:03:10.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:10.638+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.639471+0000) 2020-02-06T02:03:10.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:10.639+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:10.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:10.916+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:10.917850+0000) 2020-02-06T02:03:11.807 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:03:11.815 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:03:15.648 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:15.646+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.647502+0000) 2020-02-06T02:03:15.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:15.646+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.648031+0000) 2020-02-06T02:03:15.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:15.646+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.647581+0000) 2020-02-06T02:03:15.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:15.648+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.649906+0000) 2020-02-06T02:03:15.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:15.649+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.650410+0000) 2020-02-06T02:03:15.655 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:15.652+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.653099+0000) 2020-02-06T02:03:15.655 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:15.647+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.649291+0000) 2020-02-06T02:03:15.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:15.649+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.651151+0000) 2020-02-06T02:03:15.655 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:15.650+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.651817+0000) 2020-02-06T02:03:15.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:15.650+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.652336+0000) 2020-02-06T02:03:15.656 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:15.651+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.652979+0000) 2020-02-06T02:03:15.656 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:15.652+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.653901+0000) 2020-02-06T02:03:15.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:15.653+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.653633+0000) 2020-02-06T02:03:15.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:15.655+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:15.657003+0000) 2020-02-06T02:03:15.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:15.668+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221421 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:16.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:16.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:16.161157+0000) 2020-02-06T02:03:16.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:16.523+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:16.524408+0000) 2020-02-06T02:03:16.749 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:16.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:16.748853+0000) 2020-02-06T02:03:17.140 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:17.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.139814+0000) 2020-02-06T02:03:17.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:17.412+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.413140+0000) 2020-02-06T02:03:17.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:17.412+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.413159+0000) 2020-02-06T02:03:17.414 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:17.412+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.413301+0000) 2020-02-06T02:03:17.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:17.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.644367+0000) 2020-02-06T02:03:17.668 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:17.666+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:17.667613+0000) 2020-02-06T02:03:18.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:18.010+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:18.011117+0000) 2020-02-06T02:03:19.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:19.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:19.783362+0000) 2020-02-06T02:03:20.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:20.028+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.029359+0000) 2020-02-06T02:03:20.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:20.638+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.639701+0000) 2020-02-06T02:03:20.677 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:20.674+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.675642+0000) 2020-02-06T02:03:20.677 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:20.675+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.676572+0000) 2020-02-06T02:03:20.678 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:20.675+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.677177+0000) 2020-02-06T02:03:20.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:20.675+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.677390+0000) 2020-02-06T02:03:20.678 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:20.676+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.677823+0000) 2020-02-06T02:03:20.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:20.677+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.678592+0000) 2020-02-06T02:03:20.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:20.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.679283+0000) 2020-02-06T02:03:20.680 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:20.678+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.679684+0000) 2020-02-06T02:03:20.681 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:20.679+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.680511+0000) 2020-02-06T02:03:20.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:20.680+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.681289+0000) 2020-02-06T02:03:20.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:20.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.681428+0000) 2020-02-06T02:03:20.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:20.680+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.681601+0000) 2020-02-06T02:03:20.683 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:20.680+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.681635+0000) 2020-02-06T02:03:20.687 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:20.685+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.686424+0000) 2020-02-06T02:03:20.699 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:20.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221468 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:20.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:20.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:20.918057+0000) 2020-02-06T02:03:25.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:25.703+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.704643+0000) 2020-02-06T02:03:25.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:25.702+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.704187+0000) 2020-02-06T02:03:25.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:25.704+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.705131+0000) 2020-02-06T02:03:25.707 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:25.704+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.705887+0000) 2020-02-06T02:03:25.707 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:25.705+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.706402+0000) 2020-02-06T02:03:25.708 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:25.706+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.707801+0000) 2020-02-06T02:03:25.709 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:25.707+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.708668+0000) 2020-02-06T02:03:25.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:25.707+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.708738+0000) 2020-02-06T02:03:25.710 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:25.708+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.709616+0000) 2020-02-06T02:03:25.710 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:25.708+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.709562+0000) 2020-02-06T02:03:25.711 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:25.709+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.710195+0000) 2020-02-06T02:03:25.711 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:25.708+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.710367+0000) 2020-02-06T02:03:25.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:25.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.711137+0000) 2020-02-06T02:03:25.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:25.716+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:25.717382+0000) 2020-02-06T02:03:25.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:25.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221503 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:26.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:26.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:26.161390+0000) 2020-02-06T02:03:26.525 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:26.524+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:26.524676+0000) 2020-02-06T02:03:26.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:26.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:26.749134+0000) 2020-02-06T02:03:27.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:27.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.140040+0000) 2020-02-06T02:03:27.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:27.412+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.413395+0000) 2020-02-06T02:03:27.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:27.412+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.413396+0000) 2020-02-06T02:03:27.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:27.412+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.413527+0000) 2020-02-06T02:03:27.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:27.643+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.644627+0000) 2020-02-06T02:03:27.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:27.666+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:27.667881+0000) 2020-02-06T02:03:28.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:28.017+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:28.018503+0000) 2020-02-06T02:03:29.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:29.782+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:29.783619+0000) 2020-02-06T02:03:30.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:30.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.029617+0000) 2020-02-06T02:03:30.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:30.639+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.639949+0000) 2020-02-06T02:03:30.734 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:30.731+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.732628+0000) 2020-02-06T02:03:30.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:30.732+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.733396+0000) 2020-02-06T02:03:30.735 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:30.732+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.734302+0000) 2020-02-06T02:03:30.735 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:30.733+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.734793+0000) 2020-02-06T02:03:30.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:30.734+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.735453+0000) 2020-02-06T02:03:30.736 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:30.734+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.735962+0000) 2020-02-06T02:03:30.738 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:30.736+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.737640+0000) 2020-02-06T02:03:30.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:30.736+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.737780+0000) 2020-02-06T02:03:30.739 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:30.736+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.738156+0000) 2020-02-06T02:03:30.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:30.737+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.738549+0000) 2020-02-06T02:03:30.740 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:30.737+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.738661+0000) 2020-02-06T02:03:30.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:30.737+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.739191+0000) 2020-02-06T02:03:30.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:30.741+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.742478+0000) 2020-02-06T02:03:30.748 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:30.741+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.742744+0000) 2020-02-06T02:03:30.757 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:30.755+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:30.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:30.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:30.918318+0000) 2020-02-06T02:03:35.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:35.761+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.762395+0000) 2020-02-06T02:03:35.764 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:35.762+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.763487+0000) 2020-02-06T02:03:35.764 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:35.762+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.763770+0000) 2020-02-06T02:03:35.765 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:35.763+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.764345+0000) 2020-02-06T02:03:35.765 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:35.763+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.764558+0000) 2020-02-06T02:03:35.766 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:35.763+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.765011+0000) 2020-02-06T02:03:35.766 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:35.764+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.765433+0000) 2020-02-06T02:03:35.766 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:35.764+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.765717+0000) 2020-02-06T02:03:35.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:35.765+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.766545+0000) 2020-02-06T02:03:35.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:35.766+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.767621+0000) 2020-02-06T02:03:35.768 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:35.766+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.767643+0000) 2020-02-06T02:03:35.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:35.767+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.768423+0000) 2020-02-06T02:03:35.769 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:35.767+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.768506+0000) 2020-02-06T02:03:35.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:35.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:35.771091+0000) 2020-02-06T02:03:35.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:35.784+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221586 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:36.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:36.160+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:36.161600+0000) 2020-02-06T02:03:36.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:36.524+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:36.524905+0000) 2020-02-06T02:03:36.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:36.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:36.749397+0000) 2020-02-06T02:03:37.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:37.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.140307+0000) 2020-02-06T02:03:37.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:37.413+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.413688+0000) 2020-02-06T02:03:37.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:37.413+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.413693+0000) 2020-02-06T02:03:37.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:37.413+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.413787+0000) 2020-02-06T02:03:37.645 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:37.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.644866+0000) 2020-02-06T02:03:37.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:37.667+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:37.668132+0000) 2020-02-06T02:03:38.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:38.026+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:38.026719+0000) 2020-02-06T02:03:39.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:39.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:39.783862+0000) 2020-02-06T02:03:40.030 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:40.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.029864+0000) 2020-02-06T02:03:40.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:40.639+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.640172+0000) 2020-02-06T02:03:40.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:40.791+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.792414+0000) 2020-02-06T02:03:40.794 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:40.792+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.793708+0000) 2020-02-06T02:03:40.795 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:40.792+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.794097+0000) 2020-02-06T02:03:40.795 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:40.793+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.794442+0000) 2020-02-06T02:03:40.795 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:40.793+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.794867+0000) 2020-02-06T02:03:40.796 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:40.794+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.795459+0000) 2020-02-06T02:03:40.796 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:40.794+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.795747+0000) 2020-02-06T02:03:40.797 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:40.794+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.796319+0000) 2020-02-06T02:03:40.797 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:40.795+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.796670+0000) 2020-02-06T02:03:40.798 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:40.795+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.796850+0000) 2020-02-06T02:03:40.798 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:40.795+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.797121+0000) 2020-02-06T02:03:40.798 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:40.796+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.797815+0000) 2020-02-06T02:03:40.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:40.798+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.799149+0000) 2020-02-06T02:03:40.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:40.799+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.800363+0000) 2020-02-06T02:03:40.816 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:40.813+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:40.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:40.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:40.918528+0000) 2020-02-06T02:03:41.945 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:03:41.992 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:03:45.822 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:45.820+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.821532+0000) 2020-02-06T02:03:45.823 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:45.821+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.822601+0000) 2020-02-06T02:03:45.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:45.821+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.822836+0000) 2020-02-06T02:03:45.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:45.823+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.823490+0000) 2020-02-06T02:03:45.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:45.822+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.823858+0000) 2020-02-06T02:03:45.825 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:45.823+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.824181+0000) 2020-02-06T02:03:45.825 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:45.822+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.823090+0000) 2020-02-06T02:03:45.825 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:45.824+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.824783+0000) 2020-02-06T02:03:45.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:45.823+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.825010+0000) 2020-02-06T02:03:45.826 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:45.825+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.825855+0000) 2020-02-06T02:03:45.827 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:45.825+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.826553+0000) 2020-02-06T02:03:45.827 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:45.826+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.826693+0000) 2020-02-06T02:03:45.828 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:45.826+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.827213+0000) 2020-02-06T02:03:45.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:45.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:45.830931+0000) 2020-02-06T02:03:45.843 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:45.842+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221669 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:46.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:46.161+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:46.161840+0000) 2020-02-06T02:03:46.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:46.524+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:46.525122+0000) 2020-02-06T02:03:46.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:46.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:46.749688+0000) 2020-02-06T02:03:47.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:47.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.140543+0000) 2020-02-06T02:03:47.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:47.413+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.413966+0000) 2020-02-06T02:03:47.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:47.413+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.413965+0000) 2020-02-06T02:03:47.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:47.413+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.414042+0000) 2020-02-06T02:03:47.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:47.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.645130+0000) 2020-02-06T02:03:47.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:47.668+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:47.668361+0000) 2020-02-06T02:03:48.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:48.037+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:48.038121+0000) 2020-02-06T02:03:49.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:49.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:49.784129+0000) 2020-02-06T02:03:50.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:50.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.030133+0000) 2020-02-06T02:03:50.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:50.640+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.640423+0000) 2020-02-06T02:03:50.849 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:50.848+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.848060+0000) 2020-02-06T02:03:50.850 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:50.849+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.849411+0000) 2020-02-06T02:03:50.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:50.849+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.850049+0000) 2020-02-06T02:03:50.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:50.850+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.850981+0000) 2020-02-06T02:03:50.852 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:50.851+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.851881+0000) 2020-02-06T02:03:50.853 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:50.851+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.851972+0000) 2020-02-06T02:03:50.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:50.852+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.852533+0000) 2020-02-06T02:03:50.854 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:50.853+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.853673+0000) 2020-02-06T02:03:50.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:50.853+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.854112+0000) 2020-02-06T02:03:50.855 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:50.854+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.854204+0000) 2020-02-06T02:03:50.856 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:50.854+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.854429+0000) 2020-02-06T02:03:50.860 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:50.854+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.854731+0000) 2020-02-06T02:03:50.861 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:50.854+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.854813+0000) 2020-02-06T02:03:50.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:50.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.858891+0000) 2020-02-06T02:03:50.872 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:50.871+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221718 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:50.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:50.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:50.918774+0000) 2020-02-06T02:03:55.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:55.876+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.877580+0000) 2020-02-06T02:03:55.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:55.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.878111+0000) 2020-02-06T02:03:55.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:55.877+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.878299+0000) 2020-02-06T02:03:55.880 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:03:55.878+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.879555+0000) 2020-02-06T02:03:55.881 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:55.879+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.879593+0000) 2020-02-06T02:03:55.881 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:55.879+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.880070+0000) 2020-02-06T02:03:55.881 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:55.878+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.879891+0000) 2020-02-06T02:03:55.882 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:03:55.880+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.880537+0000) 2020-02-06T02:03:55.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:55.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.880662+0000) 2020-02-06T02:03:55.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:55.879+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.880541+0000) 2020-02-06T02:03:55.883 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:55.880+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.881407+0000) 2020-02-06T02:03:55.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:55.881+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.881660+0000) 2020-02-06T02:03:55.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:55.881+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.881756+0000) 2020-02-06T02:03:55.885 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:55.883+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:55.884538+0000) 2020-02-06T02:03:55.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:03:55.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221752 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:03:56.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:03:56.161+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:56.162085+0000) 2020-02-06T02:03:56.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:03:56.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:56.525366+0000) 2020-02-06T02:03:56.750 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:03:56.748+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:56.749889+0000) 2020-02-06T02:03:57.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:03:57.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.140778+0000) 2020-02-06T02:03:57.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:03:57.413+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.414224+0000) 2020-02-06T02:03:57.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:03:57.413+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.414230+0000) 2020-02-06T02:03:57.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:03:57.413+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.414331+0000) 2020-02-06T02:03:57.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:03:57.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.645307+0000) 2020-02-06T02:03:57.669 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:03:57.667+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:57.668595+0000) 2020-02-06T02:03:58.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:58.045+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:58.046464+0000) 2020-02-06T02:03:59.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:03:59.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:03:59.784328+0000) 2020-02-06T02:04:00.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:00.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.030361+0000) 2020-02-06T02:04:00.641 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:00.640+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.640756+0000) 2020-02-06T02:04:00.907 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:00.904+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.905897+0000) 2020-02-06T02:04:00.911 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:00.908+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.909320+0000) 2020-02-06T02:04:00.911 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:00.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.908618+0000) 2020-02-06T02:04:00.911 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:00.908+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.909769+0000) 2020-02-06T02:04:00.912 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:00.909+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.910125+0000) 2020-02-06T02:04:00.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:00.910+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.910566+0000) 2020-02-06T02:04:00.913 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:00.910+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.911209+0000) 2020-02-06T02:04:00.913 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:00.911+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.911658+0000) 2020-02-06T02:04:00.913 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:00.911+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.911922+0000) 2020-02-06T02:04:00.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:00.910+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.911406+0000) 2020-02-06T02:04:00.913 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:00.910+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.911487+0000) 2020-02-06T02:04:00.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:00.912+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.912682+0000) 2020-02-06T02:04:00.914 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:00.911+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.912836+0000) 2020-02-06T02:04:00.917 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:00.915+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.916805+0000) 2020-02-06T02:04:00.919 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:00.917+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:00.918960+0000) 2020-02-06T02:04:00.931 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:00.929+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:05.936 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:05.934+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.935324+0000) 2020-02-06T02:04:05.937 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:05.935+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.936325+0000) 2020-02-06T02:04:05.938 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:05.937+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.937530+0000) 2020-02-06T02:04:05.938 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:05.936+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.937928+0000) 2020-02-06T02:04:05.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:05.937+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.938173+0000) 2020-02-06T02:04:05.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:05.937+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.938569+0000) 2020-02-06T02:04:05.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:05.937+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.938839+0000) 2020-02-06T02:04:05.940 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:05.938+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.939291+0000) 2020-02-06T02:04:05.940 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:05.938+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.939552+0000) 2020-02-06T02:04:05.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:05.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.940776+0000) 2020-02-06T02:04:05.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:05.940+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.940827+0000) 2020-02-06T02:04:05.942 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:05.940+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.940482+0000) 2020-02-06T02:04:05.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:05.941+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.941491+0000) 2020-02-06T02:04:05.946 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:05.944+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:05.945307+0000) 2020-02-06T02:04:05.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:05.957+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221834 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:06.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:06.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:06.162354+0000) 2020-02-06T02:04:06.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:06.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:06.525630+0000) 2020-02-06T02:04:06.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:06.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:06.750156+0000) 2020-02-06T02:04:07.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:07.139+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.141012+0000) 2020-02-06T02:04:07.415 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:07.414+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.414489+0000) 2020-02-06T02:04:07.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:07.414+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.414488+0000) 2020-02-06T02:04:07.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:07.414+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.414641+0000) 2020-02-06T02:04:07.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:07.644+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.645595+0000) 2020-02-06T02:04:07.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:07.668+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:07.668844+0000) 2020-02-06T02:04:08.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:08.053+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:08.054535+0000) 2020-02-06T02:04:09.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:09.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:09.784577+0000) 2020-02-06T02:04:10.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:10.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.030587+0000) 2020-02-06T02:04:10.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:10.640+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.641011+0000) 2020-02-06T02:04:10.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:10.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.919120+0000) 2020-02-06T02:04:10.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:10.963+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.964253+0000) 2020-02-06T02:04:10.965 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:10.963+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.964793+0000) 2020-02-06T02:04:10.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:10.964+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.965496+0000) 2020-02-06T02:04:10.966 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:10.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.965901+0000) 2020-02-06T02:04:10.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:10.966+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.966489+0000) 2020-02-06T02:04:10.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:10.965+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.966721+0000) 2020-02-06T02:04:10.968 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:10.966+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.967361+0000) 2020-02-06T02:04:10.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:10.967+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.967874+0000) 2020-02-06T02:04:10.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:10.967+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.968262+0000) 2020-02-06T02:04:10.969 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:10.967+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.968277+0000) 2020-02-06T02:04:10.969 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:10.967+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.968588+0000) 2020-02-06T02:04:10.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:10.968+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.969193+0000) 2020-02-06T02:04:10.970 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:10.969+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.969976+0000) 2020-02-06T02:04:10.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:10.973+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:10.974789+0000) 2020-02-06T02:04:10.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:10.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221882 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:12.117 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:04:12.123 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:04:15.992 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:15.990+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.991556+0000) 2020-02-06T02:04:15.993 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:15.990+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.991864+0000) 2020-02-06T02:04:15.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:15.992+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.993269+0000) 2020-02-06T02:04:15.995 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:15.993+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.993821+0000) 2020-02-06T02:04:15.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:15.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.995060+0000) 2020-02-06T02:04:15.996 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:15.994+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.995257+0000) 2020-02-06T02:04:15.997 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:15.995+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.996460+0000) 2020-02-06T02:04:15.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:15.995+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.996229+0000) 2020-02-06T02:04:15.998 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:15.996+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.997198+0000) 2020-02-06T02:04:15.998 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:15.997+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.998153+0000) 2020-02-06T02:04:15.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:15.997+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.998526+0000) 2020-02-06T02:04:15.999 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:15.998+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.998720+0000) 2020-02-06T02:04:16.000 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:15.998+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:15.998918+0000) 2020-02-06T02:04:16.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:16.000+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:16.001577+0000) 2020-02-06T02:04:16.016 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:16.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221917 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:16.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:16.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:16.162598+0000) 2020-02-06T02:04:16.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:16.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:16.525849+0000) 2020-02-06T02:04:16.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:16.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:16.750371+0000) 2020-02-06T02:04:17.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:17.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.141294+0000) 2020-02-06T02:04:17.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:17.414+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.414745+0000) 2020-02-06T02:04:17.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:17.414+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.414743+0000) 2020-02-06T02:04:17.416 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:17.414+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.414896+0000) 2020-02-06T02:04:17.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:17.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.645828+0000) 2020-02-06T02:04:17.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:17.668+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:17.669112+0000) 2020-02-06T02:04:18.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:18.065+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:18.066458+0000) 2020-02-06T02:04:19.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:19.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:19.784817+0000) 2020-02-06T02:04:20.031 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:20.030+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:20.030809+0000) 2020-02-06T02:04:20.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:20.641+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:20.641269+0000) 2020-02-06T02:04:20.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:20.918+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:20.919401+0000) 2020-02-06T02:04:21.024 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:21.021+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.022960+0000) 2020-02-06T02:04:21.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:21.022+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.023434+0000) 2020-02-06T02:04:21.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:21.022+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.024020+0000) 2020-02-06T02:04:21.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:21.023+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.024128+0000) 2020-02-06T02:04:21.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:21.023+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.024659+0000) 2020-02-06T02:04:21.026 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:21.024+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.025146+0000) 2020-02-06T02:04:21.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:21.024+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.025409+0000) 2020-02-06T02:04:21.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:21.024+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.025931+0000) 2020-02-06T02:04:21.027 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:21.025+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.026150+0000) 2020-02-06T02:04:21.027 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:21.025+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.026864+0000) 2020-02-06T02:04:21.028 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:21.026+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.027310+0000) 2020-02-06T02:04:21.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:21.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.027361+0000) 2020-02-06T02:04:21.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:21.026+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.027939+0000) 2020-02-06T02:04:21.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:21.032+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:21.033616+0000) 2020-02-06T02:04:21.046 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:21.044+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:26.052 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:26.050+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.051267+0000) 2020-02-06T02:04:26.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:26.050+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.051705+0000) 2020-02-06T02:04:26.053 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:26.050+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.052310+0000) 2020-02-06T02:04:26.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:26.051+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.052678+0000) 2020-02-06T02:04:26.054 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:26.052+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.053308+0000) 2020-02-06T02:04:26.054 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:26.051+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.053315+0000) 2020-02-06T02:04:26.054 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:26.052+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.053906+0000) 2020-02-06T02:04:26.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:26.053+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.054817+0000) 2020-02-06T02:04:26.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:26.054+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.055359+0000) 2020-02-06T02:04:26.058 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:26.054+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.056233+0000) 2020-02-06T02:04:26.058 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:26.054+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.056371+0000) 2020-02-06T02:04:26.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:26.055+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.056871+0000) 2020-02-06T02:04:26.058 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:26.054+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.055811+0000) 2020-02-06T02:04:26.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:26.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.060829+0000) 2020-02-06T02:04:26.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:26.073+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 221999 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:26.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:26.161+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.162836+0000) 2020-02-06T02:04:26.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:26.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.526126+0000) 2020-02-06T02:04:26.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:26.749+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:26.750609+0000) 2020-02-06T02:04:27.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:27.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.141561+0000) 2020-02-06T02:04:27.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:27.413+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.414987+0000) 2020-02-06T02:04:27.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:27.413+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.414983+0000) 2020-02-06T02:04:27.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:27.413+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.415151+0000) 2020-02-06T02:04:27.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:27.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.646132+0000) 2020-02-06T02:04:27.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:27.668+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:27.669381+0000) 2020-02-06T02:04:28.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:28.073+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:28.074329+0000) 2020-02-06T02:04:29.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:29.783+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:29.785051+0000) 2020-02-06T02:04:30.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:30.029+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:30.031053+0000) 2020-02-06T02:04:30.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:30.640+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:30.641501+0000) 2020-02-06T02:04:30.920 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:30.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:30.919626+0000) 2020-02-06T02:04:31.080 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:31.078+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.079272+0000) 2020-02-06T02:04:31.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:31.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.081842+0000) 2020-02-06T02:04:31.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:31.081+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.082202+0000) 2020-02-06T02:04:31.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:31.081+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.082518+0000) 2020-02-06T02:04:31.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:31.081+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.082566+0000) 2020-02-06T02:04:31.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:31.083+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.084184+0000) 2020-02-06T02:04:31.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:31.082+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.084238+0000) 2020-02-06T02:04:31.085 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:31.082+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.084103+0000) 2020-02-06T02:04:31.086 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:31.083+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.085165+0000) 2020-02-06T02:04:31.086 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:31.084+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.085433+0000) 2020-02-06T02:04:31.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:31.084+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.085896+0000) 2020-02-06T02:04:31.087 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:31.084+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.086094+0000) 2020-02-06T02:04:31.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:31.084+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.085966+0000) 2020-02-06T02:04:31.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:31.087+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:31.088224+0000) 2020-02-06T02:04:31.105 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:31.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222046 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:36.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:36.109+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.110318+0000) 2020-02-06T02:04:36.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:36.109+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.110930+0000) 2020-02-06T02:04:36.113 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:36.110+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.112027+0000) 2020-02-06T02:04:36.114 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:36.112+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.113318+0000) 2020-02-06T02:04:36.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:36.112+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.113527+0000) 2020-02-06T02:04:36.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:36.113+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.114033+0000) 2020-02-06T02:04:36.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:36.113+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.114549+0000) 2020-02-06T02:04:36.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:36.113+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.115254+0000) 2020-02-06T02:04:36.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:36.114+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.115897+0000) 2020-02-06T02:04:36.117 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:36.115+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.116421+0000) 2020-02-06T02:04:36.117 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:36.114+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.116231+0000) 2020-02-06T02:04:36.118 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:36.115+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.117220+0000) 2020-02-06T02:04:36.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:36.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.118471+0000) 2020-02-06T02:04:36.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:36.117+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.118696+0000) 2020-02-06T02:04:36.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:36.133+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:36.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:36.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.163063+0000) 2020-02-06T02:04:36.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:36.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.526362+0000) 2020-02-06T02:04:36.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:36.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:36.750811+0000) 2020-02-06T02:04:37.142 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:37.140+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.141814+0000) 2020-02-06T02:04:37.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:37.414+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.415234+0000) 2020-02-06T02:04:37.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:37.414+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.415228+0000) 2020-02-06T02:04:37.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:37.414+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.415405+0000) 2020-02-06T02:04:37.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:37.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.646310+0000) 2020-02-06T02:04:37.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:37.669+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:37.669629+0000) 2020-02-06T02:04:38.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:38.081+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:38.082938+0000) 2020-02-06T02:04:39.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:39.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:39.785315+0000) 2020-02-06T02:04:40.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:40.031+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:40.031333+0000) 2020-02-06T02:04:40.642 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:40.640+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:40.641733+0000) 2020-02-06T02:04:40.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:40.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:40.919865+0000) 2020-02-06T02:04:41.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:41.140+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.142021+0000) 2020-02-06T02:04:41.144 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:41.141+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.143189+0000) 2020-02-06T02:04:41.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:41.142+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.143770+0000) 2020-02-06T02:04:41.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:41.143+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.144202+0000) 2020-02-06T02:04:41.145 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:41.142+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.144225+0000) 2020-02-06T02:04:41.145 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:41.143+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.144447+0000) 2020-02-06T02:04:41.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:41.144+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.145302+0000) 2020-02-06T02:04:41.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:41.144+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.145603+0000) 2020-02-06T02:04:41.147 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:41.144+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.145720+0000) 2020-02-06T02:04:41.147 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:41.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.145984+0000) 2020-02-06T02:04:41.147 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:41.145+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.146160+0000) 2020-02-06T02:04:41.148 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:41.144+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.146215+0000) 2020-02-06T02:04:41.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:41.145+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.147249+0000) 2020-02-06T02:04:41.152 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:41.150+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:41.151842+0000) 2020-02-06T02:04:41.164 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:41.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222127 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:42.254 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:04:42.261 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:04:46.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:46.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.163297+0000) 2020-02-06T02:04:46.170 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:46.169+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.169425+0000) 2020-02-06T02:04:46.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:46.170+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.170821+0000) 2020-02-06T02:04:46.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:46.169+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.170850+0000) 2020-02-06T02:04:46.172 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:46.169+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.171162+0000) 2020-02-06T02:04:46.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:46.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.171523+0000) 2020-02-06T02:04:46.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:46.171+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.172028+0000) 2020-02-06T02:04:46.173 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:46.170+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.171719+0000) 2020-02-06T02:04:46.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:46.171+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.173059+0000) 2020-02-06T02:04:46.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:46.172+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.173991+0000) 2020-02-06T02:04:46.176 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:46.173+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.175007+0000) 2020-02-06T02:04:46.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:46.174+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.174913+0000) 2020-02-06T02:04:46.176 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:46.174+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.175016+0000) 2020-02-06T02:04:46.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:46.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.174742+0000) 2020-02-06T02:04:46.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:46.177+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.177100+0000) 2020-02-06T02:04:46.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:46.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222163 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:46.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:46.525+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.526608+0000) 2020-02-06T02:04:46.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:46.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:46.751032+0000) 2020-02-06T02:04:47.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:47.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.142051+0000) 2020-02-06T02:04:47.416 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:47.414+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.415485+0000) 2020-02-06T02:04:47.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:47.414+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.415492+0000) 2020-02-06T02:04:47.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:47.414+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.415704+0000) 2020-02-06T02:04:47.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:47.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.646545+0000) 2020-02-06T02:04:47.670 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:47.669+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:47.669852+0000) 2020-02-06T02:04:48.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:48.091+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:48.091723+0000) 2020-02-06T02:04:49.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:49.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:49.785549+0000) 2020-02-06T02:04:50.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:50.030+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:50.031558+0000) 2020-02-06T02:04:50.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:50.641+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:50.641992+0000) 2020-02-06T02:04:50.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:50.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:50.920122+0000) 2020-02-06T02:04:51.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:51.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.198256+0000) 2020-02-06T02:04:51.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:51.197+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.199203+0000) 2020-02-06T02:04:51.201 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:51.198+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.199527+0000) 2020-02-06T02:04:51.201 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:51.198+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.200037+0000) 2020-02-06T02:04:51.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:51.199+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.201039+0000) 2020-02-06T02:04:51.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:51.199+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.201134+0000) 2020-02-06T02:04:51.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:51.200+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.201481+0000) 2020-02-06T02:04:51.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:51.200+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.202257+0000) 2020-02-06T02:04:51.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:51.201+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.202995+0000) 2020-02-06T02:04:51.204 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:51.201+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.203396+0000) 2020-02-06T02:04:51.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:51.202+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.203869+0000) 2020-02-06T02:04:51.205 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:51.203+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.204542+0000) 2020-02-06T02:04:51.206 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:51.204+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.205470+0000) 2020-02-06T02:04:51.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:51.206+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:51.207905+0000) 2020-02-06T02:04:51.223 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:51.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222210 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:56.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:56.162+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.163561+0000) 2020-02-06T02:04:56.230 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:56.227+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.228845+0000) 2020-02-06T02:04:56.230 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:56.228+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.229423+0000) 2020-02-06T02:04:56.230 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:56.228+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.229677+0000) 2020-02-06T02:04:56.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:04:56.229+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.230484+0000) 2020-02-06T02:04:56.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:56.229+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.230944+0000) 2020-02-06T02:04:56.232 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:56.230+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.231412+0000) 2020-02-06T02:04:56.232 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:56.230+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.231671+0000) 2020-02-06T02:04:56.233 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:56.231+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.232226+0000) 2020-02-06T02:04:56.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:56.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.232734+0000) 2020-02-06T02:04:56.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:56.231+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.232960+0000) 2020-02-06T02:04:56.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:04:56.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.233381+0000) 2020-02-06T02:04:56.234 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:56.232+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.234087+0000) 2020-02-06T02:04:56.235 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:04:56.233+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.234467+0000) 2020-02-06T02:04:56.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:56.237+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.238295+0000) 2020-02-06T02:04:56.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:04:56.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222245 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:04:56.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:04:56.526+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.526841+0000) 2020-02-06T02:04:56.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:04:56.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:56.751253+0000) 2020-02-06T02:04:57.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:04:57.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.142321+0000) 2020-02-06T02:04:57.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:04:57.415+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.415738+0000) 2020-02-06T02:04:57.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:04:57.415+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.415733+0000) 2020-02-06T02:04:57.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:04:57.415+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.415995+0000) 2020-02-06T02:04:57.647 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:04:57.645+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.646787+0000) 2020-02-06T02:04:57.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:04:57.669+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:57.670113+0000) 2020-02-06T02:04:58.101 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:58.099+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:58.100315+0000) 2020-02-06T02:04:59.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:04:59.784+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:04:59.785782+0000) 2020-02-06T02:05:00.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:00.030+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:00.031784+0000) 2020-02-06T02:05:00.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:00.641+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:00.642221+0000) 2020-02-06T02:05:00.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:00.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:00.920306+0000) 2020-02-06T02:05:01.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:01.257+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.258258+0000) 2020-02-06T02:05:01.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:01.257+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.258586+0000) 2020-02-06T02:05:01.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:01.258+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.259221+0000) 2020-02-06T02:05:01.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:01.258+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.259984+0000) 2020-02-06T02:05:01.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:01.259+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.260446+0000) 2020-02-06T02:05:01.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:01.260+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.261643+0000) 2020-02-06T02:05:01.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:01.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.262128+0000) 2020-02-06T02:05:01.263 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:01.260+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.262370+0000) 2020-02-06T02:05:01.263 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:01.261+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.262849+0000) 2020-02-06T02:05:01.264 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:01.261+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.263143+0000) 2020-02-06T02:05:01.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:01.261+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.263413+0000) 2020-02-06T02:05:01.265 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:01.262+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.263968+0000) 2020-02-06T02:05:01.265 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:01.263+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.264458+0000) 2020-02-06T02:05:01.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:01.266+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:01.267309+0000) 2020-02-06T02:05:01.281 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:01.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222290 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:06.164 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:06.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.163805+0000) 2020-02-06T02:05:06.287 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:06.285+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.286225+0000) 2020-02-06T02:05:06.287 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:06.285+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.286655+0000) 2020-02-06T02:05:06.288 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:06.286+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.287255+0000) 2020-02-06T02:05:06.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:06.286+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.287594+0000) 2020-02-06T02:05:06.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:06.286+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.288006+0000) 2020-02-06T02:05:06.290 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:06.287+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.289230+0000) 2020-02-06T02:05:06.291 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:06.289+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.290189+0000) 2020-02-06T02:05:06.291 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:06.289+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.290481+0000) 2020-02-06T02:05:06.292 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:06.290+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.291345+0000) 2020-02-06T02:05:06.292 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:06.289+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.291142+0000) 2020-02-06T02:05:06.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:06.290+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.291838+0000) 2020-02-06T02:05:06.293 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:06.290+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.291931+0000) 2020-02-06T02:05:06.293 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:06.290+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.292397+0000) 2020-02-06T02:05:06.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:06.293+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.294899+0000) 2020-02-06T02:05:06.308 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:06.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222326 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:06.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:06.526+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.527072+0000) 2020-02-06T02:05:06.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:06.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:06.751470+0000) 2020-02-06T02:05:07.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:07.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.142524+0000) 2020-02-06T02:05:07.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:07.415+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.416032+0000) 2020-02-06T02:05:07.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:07.415+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.416030+0000) 2020-02-06T02:05:07.417 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:07.415+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.416216+0000) 2020-02-06T02:05:07.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:07.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.647012+0000) 2020-02-06T02:05:07.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:07.669+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:07.670346+0000) 2020-02-06T02:05:08.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:08.107+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:08.108270+0000) 2020-02-06T02:05:09.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:09.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:09.786056+0000) 2020-02-06T02:05:10.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:10.031+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:10.032018+0000) 2020-02-06T02:05:10.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:10.641+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:10.642471+0000) 2020-02-06T02:05:10.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:10.919+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:10.920540+0000) 2020-02-06T02:05:11.313 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:11.311+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.312560+0000) 2020-02-06T02:05:11.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:11.312+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.313306+0000) 2020-02-06T02:05:11.315 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:11.313+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.314296+0000) 2020-02-06T02:05:11.315 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:11.313+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.314894+0000) 2020-02-06T02:05:11.316 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:11.314+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.315160+0000) 2020-02-06T02:05:11.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:11.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.316129+0000) 2020-02-06T02:05:11.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:11.315+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.316452+0000) 2020-02-06T02:05:11.318 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:11.315+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.317135+0000) 2020-02-06T02:05:11.318 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:11.316+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.317516+0000) 2020-02-06T02:05:11.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:11.316+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.317782+0000) 2020-02-06T02:05:11.319 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:11.317+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.318224+0000) 2020-02-06T02:05:11.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:11.316+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.318276+0000) 2020-02-06T02:05:11.320 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:11.317+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.318625+0000) 2020-02-06T02:05:11.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:11.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:11.320729+0000) 2020-02-06T02:05:11.337 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:11.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222372 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:12.403 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:05:12.409 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:05:16.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:16.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.164049+0000) 2020-02-06T02:05:16.343 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:16.341+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.342711+0000) 2020-02-06T02:05:16.344 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:16.342+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.343326+0000) 2020-02-06T02:05:16.344 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:16.342+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.343713+0000) 2020-02-06T02:05:16.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:16.343+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.344522+0000) 2020-02-06T02:05:16.345 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:16.343+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.344741+0000) 2020-02-06T02:05:16.346 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:16.344+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.345338+0000) 2020-02-06T02:05:16.346 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:16.345+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.345662+0000) 2020-02-06T02:05:16.347 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:16.345+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.346128+0000) 2020-02-06T02:05:16.347 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:16.346+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.347058+0000) 2020-02-06T02:05:16.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:16.346+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.347299+0000) 2020-02-06T02:05:16.349 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:16.347+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.348133+0000) 2020-02-06T02:05:16.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:16.347+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.348176+0000) 2020-02-06T02:05:16.349 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:16.347+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.347789+0000) 2020-02-06T02:05:16.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:16.352+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.353232+0000) 2020-02-06T02:05:16.369 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:16.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:16.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:16.526+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.527307+0000) 2020-02-06T02:05:16.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:16.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:16.751668+0000) 2020-02-06T02:05:17.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:17.141+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.142710+0000) 2020-02-06T02:05:17.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:17.415+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.416274+0000) 2020-02-06T02:05:17.417 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:17.415+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.416312+0000) 2020-02-06T02:05:17.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:17.416+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.416509+0000) 2020-02-06T02:05:17.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:17.646+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.647368+0000) 2020-02-06T02:05:17.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:17.670+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:17.670577+0000) 2020-02-06T02:05:18.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:18.114+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:18.115434+0000) 2020-02-06T02:05:19.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:19.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:19.786334+0000) 2020-02-06T02:05:20.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:20.031+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:20.032312+0000) 2020-02-06T02:05:20.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:20.642+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:20.642722+0000) 2020-02-06T02:05:20.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:20.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:20.920777+0000) 2020-02-06T02:05:21.374 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:21.372+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.373261+0000) 2020-02-06T02:05:21.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:21.372+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.373889+0000) 2020-02-06T02:05:21.375 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:21.373+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.374300+0000) 2020-02-06T02:05:21.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:21.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.375725+0000) 2020-02-06T02:05:21.377 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:21.374+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.375926+0000) 2020-02-06T02:05:21.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:21.376+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.376975+0000) 2020-02-06T02:05:21.379 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:21.376+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.377975+0000) 2020-02-06T02:05:21.379 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:21.377+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.377878+0000) 2020-02-06T02:05:21.379 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:21.378+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.378632+0000) 2020-02-06T02:05:21.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:21.378+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.378982+0000) 2020-02-06T02:05:21.380 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:21.378+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.379337+0000) 2020-02-06T02:05:21.380 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:21.379+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.379708+0000) 2020-02-06T02:05:21.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:21.379+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.380004+0000) 2020-02-06T02:05:21.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:21.384+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:21.385382+0000) 2020-02-06T02:05:21.399 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:21.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222454 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:26.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:26.163+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.164292+0000) 2020-02-06T02:05:26.407 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:26.403+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.404571+0000) 2020-02-06T02:05:26.408 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:26.405+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.405851+0000) 2020-02-06T02:05:26.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:26.405+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.406292+0000) 2020-02-06T02:05:26.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:26.405+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.406674+0000) 2020-02-06T02:05:26.409 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:26.406+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.407226+0000) 2020-02-06T02:05:26.410 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:26.407+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.407531+0000) 2020-02-06T02:05:26.410 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:26.407+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.408007+0000) 2020-02-06T02:05:26.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:26.407+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.408309+0000) 2020-02-06T02:05:26.410 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:26.407+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.408605+0000) 2020-02-06T02:05:26.411 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:26.408+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.409201+0000) 2020-02-06T02:05:26.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:26.409+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.409856+0000) 2020-02-06T02:05:26.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:26.409+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.410751+0000) 2020-02-06T02:05:26.412 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:26.410+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.410881+0000) 2020-02-06T02:05:26.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:26.414+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.415964+0000) 2020-02-06T02:05:26.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:26.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222489 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:26.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:26.527+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.527546+0000) 2020-02-06T02:05:26.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:26.750+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:26.751901+0000) 2020-02-06T02:05:27.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:27.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.142952+0000) 2020-02-06T02:05:27.417 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:27.416+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.416536+0000) 2020-02-06T02:05:27.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:27.416+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.416553+0000) 2020-02-06T02:05:27.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:27.416+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.416769+0000) 2020-02-06T02:05:27.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:27.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.647597+0000) 2020-02-06T02:05:27.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:27.669+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:27.670832+0000) 2020-02-06T02:05:28.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:28.123+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:28.123758+0000) 2020-02-06T02:05:29.787 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:29.785+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:29.786595+0000) 2020-02-06T02:05:30.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:30.031+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:30.032529+0000) 2020-02-06T02:05:30.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:30.642+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:30.642956+0000) 2020-02-06T02:05:30.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:30.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:30.921004+0000) 2020-02-06T02:05:31.435 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:31.432+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.433910+0000) 2020-02-06T02:05:31.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:31.433+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.433646+0000) 2020-02-06T02:05:31.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:31.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.436277+0000) 2020-02-06T02:05:31.437 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:31.435+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.436735+0000) 2020-02-06T02:05:31.437 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:31.436+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.436828+0000) 2020-02-06T02:05:31.439 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:31.436+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.438022+0000) 2020-02-06T02:05:31.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:31.437+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.438391+0000) 2020-02-06T02:05:31.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:31.437+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.437892+0000) 2020-02-06T02:05:31.440 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:31.439+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.439641+0000) 2020-02-06T02:05:31.440 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:31.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.439904+0000) 2020-02-06T02:05:31.441 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:31.438+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.440032+0000) 2020-02-06T02:05:31.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:31.438+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.438958+0000) 2020-02-06T02:05:31.443 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:31.441+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.443025+0000) 2020-02-06T02:05:31.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:31.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:31.443345+0000) 2020-02-06T02:05:31.458 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:31.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222536 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:36.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:36.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.164528+0000) 2020-02-06T02:05:36.466 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:36.464+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.464671+0000) 2020-02-06T02:05:36.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:36.464+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.465249+0000) 2020-02-06T02:05:36.467 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:36.464+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.465651+0000) 2020-02-06T02:05:36.467 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:36.464+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.465997+0000) 2020-02-06T02:05:36.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:36.465+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.466616+0000) 2020-02-06T02:05:36.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:36.466+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.467127+0000) 2020-02-06T02:05:36.468 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:36.466+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.467609+0000) 2020-02-06T02:05:36.469 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:36.467+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.468067+0000) 2020-02-06T02:05:36.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:36.467+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.469040+0000) 2020-02-06T02:05:36.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:36.468+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.468768+0000) 2020-02-06T02:05:36.471 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:36.468+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.468836+0000) 2020-02-06T02:05:36.471 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:36.468+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.469356+0000) 2020-02-06T02:05:36.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:36.469+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.469790+0000) 2020-02-06T02:05:36.474 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:36.472+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.473908+0000) 2020-02-06T02:05:36.489 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:36.487+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:36.528 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:36.527+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.527803+0000) 2020-02-06T02:05:36.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:36.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:36.752142+0000) 2020-02-06T02:05:37.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:37.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.143156+0000) 2020-02-06T02:05:37.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:37.416+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.416771+0000) 2020-02-06T02:05:37.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:37.416+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.416857+0000) 2020-02-06T02:05:37.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:37.417+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.417502+0000) 2020-02-06T02:05:37.648 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:37.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.647837+0000) 2020-02-06T02:05:37.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:37.670+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:37.671071+0000) 2020-02-06T02:05:38.133 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:38.131+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:38.131860+0000) 2020-02-06T02:05:39.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:39.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:39.786837+0000) 2020-02-06T02:05:40.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:40.032+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:40.032770+0000) 2020-02-06T02:05:40.644 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:40.642+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:40.643199+0000) 2020-02-06T02:05:40.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:40.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:40.921220+0000) 2020-02-06T02:05:41.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:41.492+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.492528+0000) 2020-02-06T02:05:41.495 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:41.494+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.494440+0000) 2020-02-06T02:05:41.495 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:41.494+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.494604+0000) 2020-02-06T02:05:41.496 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:41.494+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.494805+0000) 2020-02-06T02:05:41.496 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:41.494+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.494903+0000) 2020-02-06T02:05:41.496 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:41.494+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.495256+0000) 2020-02-06T02:05:41.497 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:41.495+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.495613+0000) 2020-02-06T02:05:41.497 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:41.495+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.495948+0000) 2020-02-06T02:05:41.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:41.495+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.496232+0000) 2020-02-06T02:05:41.498 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:41.496+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.496618+0000) 2020-02-06T02:05:41.498 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:41.496+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.497155+0000) 2020-02-06T02:05:41.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:41.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.496824+0000) 2020-02-06T02:05:41.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:41.496+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.496930+0000) 2020-02-06T02:05:41.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:41.502+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:41.502781+0000) 2020-02-06T02:05:41.519 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:41.517+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222617 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:42.550 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:05:42.557 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:05:46.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:46.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.164787+0000) 2020-02-06T02:05:46.526 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:46.523+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.524920+0000) 2020-02-06T02:05:46.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:46.524+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.525354+0000) 2020-02-06T02:05:46.526 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:46.524+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.524629+0000) 2020-02-06T02:05:46.527 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:46.524+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.526017+0000) 2020-02-06T02:05:46.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:46.525+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.525870+0000) 2020-02-06T02:05:46.528 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:46.525+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.526705+0000) 2020-02-06T02:05:46.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:46.526+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.527379+0000) 2020-02-06T02:05:46.528 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:46.526+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.527621+0000) 2020-02-06T02:05:46.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:46.527+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.528068+0000) 2020-02-06T02:05:46.529 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:46.528+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.528649+0000) 2020-02-06T02:05:46.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:46.528+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.528854+0000) 2020-02-06T02:05:46.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:46.528+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.529291+0000) 2020-02-06T02:05:46.530 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:46.529+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.529762+0000) 2020-02-06T02:05:46.531 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:46.530+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.530533+0000) 2020-02-06T02:05:46.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:46.532+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.533880+0000) 2020-02-06T02:05:46.548 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:46.547+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:46.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:46.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:46.752342+0000) 2020-02-06T02:05:47.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:47.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.143368+0000) 2020-02-06T02:05:47.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:47.416+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.417071+0000) 2020-02-06T02:05:47.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:47.416+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.417023+0000) 2020-02-06T02:05:47.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:47.417+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.417695+0000) 2020-02-06T02:05:47.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:47.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.648065+0000) 2020-02-06T02:05:47.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:47.670+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:47.671351+0000) 2020-02-06T02:05:48.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:48.139+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:48.140373+0000) 2020-02-06T02:05:49.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:49.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:49.787131+0000) 2020-02-06T02:05:50.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:50.032+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:50.033024+0000) 2020-02-06T02:05:50.644 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:50.642+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:50.643470+0000) 2020-02-06T02:05:50.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:50.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:50.921432+0000) 2020-02-06T02:05:51.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:51.551+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.552931+0000) 2020-02-06T02:05:51.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:51.552+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.553929+0000) 2020-02-06T02:05:51.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:51.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.554036+0000) 2020-02-06T02:05:51.555 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:51.553+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.554461+0000) 2020-02-06T02:05:51.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:51.553+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.554596+0000) 2020-02-06T02:05:51.556 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:51.553+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.554771+0000) 2020-02-06T02:05:51.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:51.553+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.554986+0000) 2020-02-06T02:05:51.557 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:51.554+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.555204+0000) 2020-02-06T02:05:51.557 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:51.554+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.555627+0000) 2020-02-06T02:05:51.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:51.554+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.556074+0000) 2020-02-06T02:05:51.558 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:51.555+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.556819+0000) 2020-02-06T02:05:51.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:51.555+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.556410+0000) 2020-02-06T02:05:51.558 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:51.555+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.557382+0000) 2020-02-06T02:05:51.565 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:51.563+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:51.564365+0000) 2020-02-06T02:05:51.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:51.575+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222698 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:56.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:56.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.165059+0000) 2020-02-06T02:05:56.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:56.527+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.528333+0000) 2020-02-06T02:05:56.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:05:56.580+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.582280+0000) 2020-02-06T02:05:56.584 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:05:56.582+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.583214+0000) 2020-02-06T02:05:56.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:56.583+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.584957+0000) 2020-02-06T02:05:56.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:05:56.583+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.585021+0000) 2020-02-06T02:05:56.586 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:56.584+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.585421+0000) 2020-02-06T02:05:56.587 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:05:56.585+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.586854+0000) 2020-02-06T02:05:56.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:56.586+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.587315+0000) 2020-02-06T02:05:56.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:56.586+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.587695+0000) 2020-02-06T02:05:56.589 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:56.586+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.587747+0000) 2020-02-06T02:05:56.589 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:56.586+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.588103+0000) 2020-02-06T02:05:56.589 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:56.587+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.588574+0000) 2020-02-06T02:05:56.590 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:56.589+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.590121+0000) 2020-02-06T02:05:56.591 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:56.588+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.590232+0000) 2020-02-06T02:05:56.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:56.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.597521+0000) 2020-02-06T02:05:56.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:05:56.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222733 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:05:56.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:05:56.751+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:56.752542+0000) 2020-02-06T02:05:57.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:05:57.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.143582+0000) 2020-02-06T02:05:57.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:05:57.415+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.417259+0000) 2020-02-06T02:05:57.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:05:57.415+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.417250+0000) 2020-02-06T02:05:57.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:05:57.416+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.417865+0000) 2020-02-06T02:05:57.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:05:57.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.648338+0000) 2020-02-06T02:05:57.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:05:57.670+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:57.671578+0000) 2020-02-06T02:05:58.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:58.147+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:58.148370+0000) 2020-02-06T02:05:59.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:05:59.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:05:59.787357+0000) 2020-02-06T02:06:00.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:00.032+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:00.033292+0000) 2020-02-06T02:06:00.644 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:00.642+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:00.643762+0000) 2020-02-06T02:06:00.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:00.920+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:00.921642+0000) 2020-02-06T02:06:01.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:01.611+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.612434+0000) 2020-02-06T02:06:01.614 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:01.611+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.612983+0000) 2020-02-06T02:06:01.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:01.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.613023+0000) 2020-02-06T02:06:01.614 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:01.612+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.613739+0000) 2020-02-06T02:06:01.615 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:01.613+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.614192+0000) 2020-02-06T02:06:01.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:01.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.614603+0000) 2020-02-06T02:06:01.615 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:01.613+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.614861+0000) 2020-02-06T02:06:01.616 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:01.614+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.615463+0000) 2020-02-06T02:06:01.616 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:01.614+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.615977+0000) 2020-02-06T02:06:01.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:01.614+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.615964+0000) 2020-02-06T02:06:01.618 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:01.615+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.617378+0000) 2020-02-06T02:06:01.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:01.616+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.617528+0000) 2020-02-06T02:06:01.619 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:01.617+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.618566+0000) 2020-02-06T02:06:01.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:01.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:01.623734+0000) 2020-02-06T02:06:01.637 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:01.634+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222779 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:06.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:06.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.165318+0000) 2020-02-06T02:06:06.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:06.527+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.528685+0000) 2020-02-06T02:06:06.643 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:06.640+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.642070+0000) 2020-02-06T02:06:06.643 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:06.641+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.642932+0000) 2020-02-06T02:06:06.644 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:06.642+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.643507+0000) 2020-02-06T02:06:06.644 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:06.642+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.643833+0000) 2020-02-06T02:06:06.645 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:06.643+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.644139+0000) 2020-02-06T02:06:06.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:06.643+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.644798+0000) 2020-02-06T02:06:06.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:06.644+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.645250+0000) 2020-02-06T02:06:06.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:06.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.645946+0000) 2020-02-06T02:06:06.647 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:06.643+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.644942+0000) 2020-02-06T02:06:06.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:06.644+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.645538+0000) 2020-02-06T02:06:06.647 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:06.644+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.645832+0000) 2020-02-06T02:06:06.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:06.644+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.646129+0000) 2020-02-06T02:06:06.648 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:06.645+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.647280+0000) 2020-02-06T02:06:06.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:06.650+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.651951+0000) 2020-02-06T02:06:06.665 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:06.663+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:06.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:06.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:06.752783+0000) 2020-02-06T02:06:07.144 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:07.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.143801+0000) 2020-02-06T02:06:07.418 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:07.416+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.417488+0000) 2020-02-06T02:06:07.418 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:07.416+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.417481+0000) 2020-02-06T02:06:07.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:07.416+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.418035+0000) 2020-02-06T02:06:07.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:07.647+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.648570+0000) 2020-02-06T02:06:07.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:07.671+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:07.671804+0000) 2020-02-06T02:06:08.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:08.156+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:08.157382+0000) 2020-02-06T02:06:09.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:09.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:09.787600+0000) 2020-02-06T02:06:10.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:10.032+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:10.033517+0000) 2020-02-06T02:06:10.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:10.643+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:10.644027+0000) 2020-02-06T02:06:10.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:10.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:10.921904+0000) 2020-02-06T02:06:11.671 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:11.668+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.669893+0000) 2020-02-06T02:06:11.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:11.669+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.670816+0000) 2020-02-06T02:06:11.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:11.670+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.672036+0000) 2020-02-06T02:06:11.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:11.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.672774+0000) 2020-02-06T02:06:11.674 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:11.672+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.673401+0000) 2020-02-06T02:06:11.674 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:11.672+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.673616+0000) 2020-02-06T02:06:11.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:11.672+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.673965+0000) 2020-02-06T02:06:11.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:11.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.675126+0000) 2020-02-06T02:06:11.676 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:11.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.675033+0000) 2020-02-06T02:06:11.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:11.674+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.675748+0000) 2020-02-06T02:06:11.677 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:11.674+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.676098+0000) 2020-02-06T02:06:11.677 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:11.674+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.676407+0000) 2020-02-06T02:06:11.677 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:11.675+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.676845+0000) 2020-02-06T02:06:11.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:11.680+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:11.681736+0000) 2020-02-06T02:06:11.694 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:11.692+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:12.690 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:06:12.697 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:06:16.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:16.164+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.165568+0000) 2020-02-06T02:06:16.529 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:16.528+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.528934+0000) 2020-02-06T02:06:16.700 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:16.698+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.699832+0000) 2020-02-06T02:06:16.701 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:16.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.700065+0000) 2020-02-06T02:06:16.702 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:16.699+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.700787+0000) 2020-02-06T02:06:16.702 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:16.699+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.700815+0000) 2020-02-06T02:06:16.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:16.700+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.701173+0000) 2020-02-06T02:06:16.703 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:16.701+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.702728+0000) 2020-02-06T02:06:16.703 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:16.701+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.703014+0000) 2020-02-06T02:06:16.704 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:16.702+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.703823+0000) 2020-02-06T02:06:16.705 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:16.703+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.704463+0000) 2020-02-06T02:06:16.706 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:16.704+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.705501+0000) 2020-02-06T02:06:16.707 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:16.704+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.705686+0000) 2020-02-06T02:06:16.707 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:16.704+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.705499+0000) 2020-02-06T02:06:16.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:16.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.710652+0000) 2020-02-06T02:06:16.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:16.709+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.710935+0000) 2020-02-06T02:06:16.723 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:16.721+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222896 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:16.753 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:16.752+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:16.752972+0000) 2020-02-06T02:06:17.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:17.142+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.144032+0000) 2020-02-06T02:06:17.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:17.416+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.417726+0000) 2020-02-06T02:06:17.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:17.416+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.417716+0000) 2020-02-06T02:06:17.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:17.416+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.418201+0000) 2020-02-06T02:06:17.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:17.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.648801+0000) 2020-02-06T02:06:17.672 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:17.671+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:17.672033+0000) 2020-02-06T02:06:18.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:18.165+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:18.166121+0000) 2020-02-06T02:06:19.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:19.786+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:19.787849+0000) 2020-02-06T02:06:20.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:20.033+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:20.033746+0000) 2020-02-06T02:06:20.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:20.643+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:20.644278+0000) 2020-02-06T02:06:20.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:20.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:20.922182+0000) 2020-02-06T02:06:21.729 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:21.727+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.728326+0000) 2020-02-06T02:06:21.729 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:21.727+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.728788+0000) 2020-02-06T02:06:21.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:21.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.729348+0000) 2020-02-06T02:06:21.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:21.728+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.729699+0000) 2020-02-06T02:06:21.733 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:21.729+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.730595+0000) 2020-02-06T02:06:21.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:21.730+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.732204+0000) 2020-02-06T02:06:21.734 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:21.730+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.731413+0000) 2020-02-06T02:06:21.734 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:21.731+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.732283+0000) 2020-02-06T02:06:21.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:21.731+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.732933+0000) 2020-02-06T02:06:21.735 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:21.730+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.732168+0000) 2020-02-06T02:06:21.735 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:21.731+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.733205+0000) 2020-02-06T02:06:21.735 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:21.732+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.733743+0000) 2020-02-06T02:06:21.736 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:21.732+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.733613+0000) 2020-02-06T02:06:21.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:21.737+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:21.738913+0000) 2020-02-06T02:06:21.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:21.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:26.166 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:26.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.165813+0000) 2020-02-06T02:06:26.530 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:26.528+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.529189+0000) 2020-02-06T02:06:26.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:26.755+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.756616+0000) 2020-02-06T02:06:26.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:26.755+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.757036+0000) 2020-02-06T02:06:26.758 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:26.756+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.757711+0000) 2020-02-06T02:06:26.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:26.757+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.758066+0000) 2020-02-06T02:06:26.759 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:26.757+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.758413+0000) 2020-02-06T02:06:26.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:26.757+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.759042+0000) 2020-02-06T02:06:26.760 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:26.758+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.759180+0000) 2020-02-06T02:06:26.760 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:26.758+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.759717+0000) 2020-02-06T02:06:26.761 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:26.758+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.760118+0000) 2020-02-06T02:06:26.761 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:26.758+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.760236+0000) 2020-02-06T02:06:26.762 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:26.759+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.760470+0000) 2020-02-06T02:06:26.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:26.759+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.760942+0000) 2020-02-06T02:06:26.762 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:26.760+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.761968+0000) 2020-02-06T02:06:26.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:26.764+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:26.765231+0000) 2020-02-06T02:06:26.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:26.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 222979 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:27.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:27.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.144296+0000) 2020-02-06T02:06:27.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:27.417+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.417950+0000) 2020-02-06T02:06:27.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:27.417+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.417954+0000) 2020-02-06T02:06:27.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:27.417+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.418393+0000) 2020-02-06T02:06:27.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:27.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.649039+0000) 2020-02-06T02:06:27.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:27.672+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:27.672287+0000) 2020-02-06T02:06:28.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:28.174+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:28.174897+0000) 2020-02-06T02:06:29.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:29.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:29.788123+0000) 2020-02-06T02:06:30.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:30.033+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:30.033971+0000) 2020-02-06T02:06:30.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:30.643+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:30.644514+0000) 2020-02-06T02:06:30.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:30.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:30.922434+0000) 2020-02-06T02:06:31.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:31.783+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.785054+0000) 2020-02-06T02:06:31.786 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:31.784+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.785619+0000) 2020-02-06T02:06:31.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:31.784+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.786014+0000) 2020-02-06T02:06:31.787 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:31.785+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.786673+0000) 2020-02-06T02:06:31.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:31.785+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.786923+0000) 2020-02-06T02:06:31.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:31.786+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.787743+0000) 2020-02-06T02:06:31.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:31.786+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.787928+0000) 2020-02-06T02:06:31.789 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:31.787+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.788358+0000) 2020-02-06T02:06:31.790 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:31.788+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.789368+0000) 2020-02-06T02:06:31.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:31.787+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.788858+0000) 2020-02-06T02:06:31.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:31.787+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.788916+0000) 2020-02-06T02:06:31.791 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:31.788+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.789948+0000) 2020-02-06T02:06:31.791 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:31.788+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.790242+0000) 2020-02-06T02:06:31.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:31.792+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:31.793894+0000) 2020-02-06T02:06:31.810 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:31.807+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223025 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:33.504 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:33.503+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:33.503748+0000) 2020-02-06T02:06:36.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:36.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.166068+0000) 2020-02-06T02:06:36.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:36.528+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.529481+0000) 2020-02-06T02:06:36.816 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:36.813+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.814710+0000) 2020-02-06T02:06:36.817 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:36.814+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.815859+0000) 2020-02-06T02:06:36.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:36.816+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.816663+0000) 2020-02-06T02:06:36.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:36.817+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.817306+0000) 2020-02-06T02:06:36.818 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:36.817+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.817751+0000) 2020-02-06T02:06:36.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:36.818+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.818722+0000) 2020-02-06T02:06:36.820 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:36.819+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.819708+0000) 2020-02-06T02:06:36.820 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:36.819+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.819876+0000) 2020-02-06T02:06:36.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:36.820+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.820424+0000) 2020-02-06T02:06:36.822 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:36.819+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.820705+0000) 2020-02-06T02:06:36.822 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:36.819+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.820971+0000) 2020-02-06T02:06:36.822 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:36.819+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.821026+0000) 2020-02-06T02:06:36.822 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:36.820+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.821658+0000) 2020-02-06T02:06:36.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:36.824+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:36.824628+0000) 2020-02-06T02:06:36.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:36.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223060 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:37.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:37.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.144505+0000) 2020-02-06T02:06:37.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:37.417+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.418182+0000) 2020-02-06T02:06:37.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:37.417+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.418181+0000) 2020-02-06T02:06:37.419 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:37.417+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.418601+0000) 2020-02-06T02:06:37.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:37.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.649272+0000) 2020-02-06T02:06:37.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:37.672+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:37.672490+0000) 2020-02-06T02:06:38.184 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:38.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:38.183825+0000) 2020-02-06T02:06:39.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:39.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:39.788370+0000) 2020-02-06T02:06:40.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:40.033+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:40.034217+0000) 2020-02-06T02:06:40.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:40.644+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:40.644745+0000) 2020-02-06T02:06:40.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:40.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:40.922665+0000) 2020-02-06T02:06:41.845 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:41.843+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.844452+0000) 2020-02-06T02:06:41.847 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:41.845+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.846202+0000) 2020-02-06T02:06:41.848 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:41.846+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.846714+0000) 2020-02-06T02:06:41.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:41.846+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.847644+0000) 2020-02-06T02:06:41.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:41.847+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.848122+0000) 2020-02-06T02:06:41.849 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:41.847+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.848071+0000) 2020-02-06T02:06:41.849 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:41.848+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.848979+0000) 2020-02-06T02:06:41.851 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:41.847+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.848912+0000) 2020-02-06T02:06:41.851 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:41.848+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.849485+0000) 2020-02-06T02:06:41.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:41.848+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.849500+0000) 2020-02-06T02:06:41.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:41.848+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.849398+0000) 2020-02-06T02:06:41.851 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:41.849+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.850096+0000) 2020-02-06T02:06:41.852 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:41.849+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.849590+0000) 2020-02-06T02:06:41.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:41.851+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:41.852220+0000) 2020-02-06T02:06:41.869 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:41.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:42.852 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:06:42.859 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:06:43.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:43.503+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:43.504003+0000) 2020-02-06T02:06:46.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:46.165+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.166318+0000) 2020-02-06T02:06:46.530 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:46.529+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.529778+0000) 2020-02-06T02:06:46.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:46.873+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.874180+0000) 2020-02-06T02:06:46.876 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:46.874+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.875414+0000) 2020-02-06T02:06:46.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:46.874+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.875158+0000) 2020-02-06T02:06:46.877 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:46.875+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.876681+0000) 2020-02-06T02:06:46.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:46.875+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.877028+0000) 2020-02-06T02:06:46.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:46.876+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.877841+0000) 2020-02-06T02:06:46.879 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:46.877+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.878382+0000) 2020-02-06T02:06:46.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:46.877+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.878413+0000) 2020-02-06T02:06:46.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:46.878+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.879207+0000) 2020-02-06T02:06:46.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:46.879+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.879664+0000) 2020-02-06T02:06:46.880 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:46.879+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.879956+0000) 2020-02-06T02:06:46.881 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:46.880+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.880890+0000) 2020-02-06T02:06:46.882 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:46.880+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.881097+0000) 2020-02-06T02:06:46.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:46.882+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:46.883799+0000) 2020-02-06T02:06:46.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:46.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:47.145 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:47.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.144671+0000) 2020-02-06T02:06:47.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:47.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.418420+0000) 2020-02-06T02:06:47.419 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:47.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.418475+0000) 2020-02-06T02:06:47.420 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:47.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.418826+0000) 2020-02-06T02:06:47.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:47.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.649471+0000) 2020-02-06T02:06:47.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:47.671+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:47.672719+0000) 2020-02-06T02:06:48.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:48.191+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:48.192521+0000) 2020-02-06T02:06:49.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:49.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:49.788627+0000) 2020-02-06T02:06:50.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:50.033+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:50.034468+0000) 2020-02-06T02:06:50.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:50.644+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:50.644995+0000) 2020-02-06T02:06:50.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:50.921+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:50.922920+0000) 2020-02-06T02:06:51.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:51.906+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.907348+0000) 2020-02-06T02:06:51.908 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:51.905+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.907037+0000) 2020-02-06T02:06:51.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:51.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.907906+0000) 2020-02-06T02:06:51.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:51.908+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.908797+0000) 2020-02-06T02:06:51.910 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:51.909+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.909804+0000) 2020-02-06T02:06:51.910 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:51.909+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.909867+0000) 2020-02-06T02:06:51.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:51.909+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.910797+0000) 2020-02-06T02:06:51.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:51.912+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.913827+0000) 2020-02-06T02:06:51.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:51.911+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.912445+0000) 2020-02-06T02:06:51.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:51.911+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.912680+0000) 2020-02-06T02:06:51.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:51.911+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.912813+0000) 2020-02-06T02:06:51.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:51.910+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.910579+0000) 2020-02-06T02:06:51.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:51.911+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.911580+0000) 2020-02-06T02:06:51.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:51.911+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:51.912112+0000) 2020-02-06T02:06:51.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:51.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223188 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:53.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:53.503+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:53.504350+0000) 2020-02-06T02:06:56.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:56.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.166568+0000) 2020-02-06T02:06:56.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:56.529+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.530032+0000) 2020-02-06T02:06:56.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:06:56.934+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.935230+0000) 2020-02-06T02:06:56.937 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:06:56.935+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.936136+0000) 2020-02-06T02:06:56.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:06:56.937+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.937523+0000) 2020-02-06T02:06:56.939 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:56.936+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.937719+0000) 2020-02-06T02:06:56.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:56.937+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.938400+0000) 2020-02-06T02:06:56.939 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:56.938+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.938942+0000) 2020-02-06T02:06:56.940 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:56.938+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.939650+0000) 2020-02-06T02:06:56.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:56.938+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.940050+0000) 2020-02-06T02:06:56.941 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:06:56.940+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.940488+0000) 2020-02-06T02:06:56.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:56.939+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.940883+0000) 2020-02-06T02:06:56.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:06:56.940+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.940957+0000) 2020-02-06T02:06:56.942 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:56.940+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.941295+0000) 2020-02-06T02:06:56.942 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:56.941+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.941515+0000) 2020-02-06T02:06:56.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:56.942+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:56.943426+0000) 2020-02-06T02:06:56.960 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:06:56.959+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223224 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:06:57.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:06:57.143+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.144931+0000) 2020-02-06T02:06:57.419 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:06:57.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.418644+0000) 2020-02-06T02:06:57.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:06:57.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.418742+0000) 2020-02-06T02:06:57.420 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:06:57.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.419044+0000) 2020-02-06T02:06:57.650 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:06:57.648+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.649700+0000) 2020-02-06T02:06:57.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:06:57.672+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:57.673004+0000) 2020-02-06T02:06:58.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:58.200+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:58.201003+0000) 2020-02-06T02:06:59.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:06:59.787+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:06:59.788879+0000) 2020-02-06T02:07:00.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:00.034+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:00.034705+0000) 2020-02-06T02:07:00.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:00.644+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:00.645254+0000) 2020-02-06T02:07:00.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:00.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:00.923150+0000) 2020-02-06T02:07:01.966 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:01.964+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.965243+0000) 2020-02-06T02:07:01.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:01.964+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.965855+0000) 2020-02-06T02:07:01.967 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:01.965+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.966344+0000) 2020-02-06T02:07:01.968 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:01.965+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.966665+0000) 2020-02-06T02:07:01.968 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:01.966+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.967080+0000) 2020-02-06T02:07:01.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:01.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.968227+0000) 2020-02-06T02:07:01.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:01.967+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.968879+0000) 2020-02-06T02:07:01.970 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:01.968+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.969400+0000) 2020-02-06T02:07:01.970 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:01.968+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.969500+0000) 2020-02-06T02:07:01.971 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:01.969+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.970268+0000) 2020-02-06T02:07:01.972 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:01.970+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.970845+0000) 2020-02-06T02:07:01.972 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:01.970+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.970631+0000) 2020-02-06T02:07:01.972 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:01.970+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.971152+0000) 2020-02-06T02:07:01.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:01.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:01.975191+0000) 2020-02-06T02:07:01.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:01.987+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223270 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:03.505 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:03.503+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:03.504577+0000) 2020-02-06T02:07:06.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:06.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.166807+0000) 2020-02-06T02:07:06.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:06.530+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.530264+0000) 2020-02-06T02:07:06.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:06.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.992858+0000) 2020-02-06T02:07:06.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:06.992+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.993590+0000) 2020-02-06T02:07:06.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:06.993+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.994215+0000) 2020-02-06T02:07:06.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:06.993+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.994652+0000) 2020-02-06T02:07:06.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:06.993+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.995053+0000) 2020-02-06T02:07:06.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:06.995+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.995711+0000) 2020-02-06T02:07:06.997 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:06.995+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.996272+0000) 2020-02-06T02:07:06.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:06.996+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.996789+0000) 2020-02-06T02:07:06.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:06.997+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.997792+0000) 2020-02-06T02:07:06.999 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:06.997+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.998314+0000) 2020-02-06T02:07:06.999 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:06.997+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.998557+0000) 2020-02-06T02:07:07.000 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:06.998+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.999578+0000) 2020-02-06T02:07:07.000 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:06.999+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:06.999836+0000) 2020-02-06T02:07:07.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:07.001+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.002798+0000) 2020-02-06T02:07:07.018 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:07.016+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:07.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:07.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.145131+0000) 2020-02-06T02:07:07.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:07.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.418897+0000) 2020-02-06T02:07:07.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:07.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.418988+0000) 2020-02-06T02:07:07.420 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:07.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.419234+0000) 2020-02-06T02:07:07.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:07.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.649940+0000) 2020-02-06T02:07:07.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:07.672+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:07.673277+0000) 2020-02-06T02:07:08.211 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:08.209+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:08.210421+0000) 2020-02-06T02:07:09.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:09.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:09.789139+0000) 2020-02-06T02:07:10.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:10.033+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:10.034962+0000) 2020-02-06T02:07:10.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:10.645+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:10.645491+0000) 2020-02-06T02:07:10.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:10.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:10.923406+0000) 2020-02-06T02:07:12.023 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:12.021+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.022682+0000) 2020-02-06T02:07:12.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:12.022+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.023206+0000) 2020-02-06T02:07:12.024 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:12.023+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.023830+0000) 2020-02-06T02:07:12.025 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:12.022+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.023973+0000) 2020-02-06T02:07:12.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:12.024+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.024763+0000) 2020-02-06T02:07:12.026 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:12.024+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.025287+0000) 2020-02-06T02:07:12.026 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:12.024+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.025757+0000) 2020-02-06T02:07:12.027 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:12.024+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.026057+0000) 2020-02-06T02:07:12.028 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:12.026+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.027193+0000) 2020-02-06T02:07:12.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:12.026+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.027615+0000) 2020-02-06T02:07:12.029 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:12.027+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.028202+0000) 2020-02-06T02:07:12.029 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:12.028+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.028585+0000) 2020-02-06T02:07:12.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:12.028+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.029082+0000) 2020-02-06T02:07:12.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:12.030+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:12.031716+0000) 2020-02-06T02:07:12.048 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:12.047+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223352 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:12.989 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:07:12.996 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:07:13.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:13.503+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:13.504849+0000) 2020-02-06T02:07:16.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:16.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:16.167061+0000) 2020-02-06T02:07:16.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:16.530+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:16.530517+0000) 2020-02-06T02:07:17.055 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:17.052+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.053972+0000) 2020-02-06T02:07:17.055 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:17.053+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.054757+0000) 2020-02-06T02:07:17.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:17.054+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.055342+0000) 2020-02-06T02:07:17.056 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:17.054+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.055637+0000) 2020-02-06T02:07:17.056 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:17.055+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.055970+0000) 2020-02-06T02:07:17.057 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:17.055+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.056403+0000) 2020-02-06T02:07:17.058 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:17.056+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.056935+0000) 2020-02-06T02:07:17.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:17.056+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.057379+0000) 2020-02-06T02:07:17.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:17.057+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.058343+0000) 2020-02-06T02:07:17.059 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:17.058+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.058802+0000) 2020-02-06T02:07:17.060 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:17.059+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.059847+0000) 2020-02-06T02:07:17.061 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:17.059+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.059732+0000) 2020-02-06T02:07:17.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:17.059+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.060340+0000) 2020-02-06T02:07:17.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:17.065+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.066671+0000) 2020-02-06T02:07:17.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:17.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223388 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:17.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:17.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.145375+0000) 2020-02-06T02:07:17.420 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:17.419+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.419230+0000) 2020-02-06T02:07:17.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:17.419+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.419113+0000) 2020-02-06T02:07:17.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:17.419+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.419420+0000) 2020-02-06T02:07:17.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:17.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.650140+0000) 2020-02-06T02:07:17.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:17.673+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:17.673507+0000) 2020-02-06T02:07:18.220 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:18.218+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:18.219001+0000) 2020-02-06T02:07:19.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:19.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:19.789379+0000) 2020-02-06T02:07:20.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:20.034+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:20.035195+0000) 2020-02-06T02:07:20.646 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:20.644+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:20.645734+0000) 2020-02-06T02:07:20.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:20.922+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:20.923634+0000) 2020-02-06T02:07:22.084 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:22.081+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.082996+0000) 2020-02-06T02:07:22.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:22.082+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.083264+0000) 2020-02-06T02:07:22.085 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:22.082+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.083710+0000) 2020-02-06T02:07:22.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:22.082+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.083940+0000) 2020-02-06T02:07:22.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:22.083+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.084545+0000) 2020-02-06T02:07:22.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:22.083+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.084676+0000) 2020-02-06T02:07:22.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:22.084+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.085254+0000) 2020-02-06T02:07:22.087 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:22.083+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.085275+0000) 2020-02-06T02:07:22.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:22.084+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.085808+0000) 2020-02-06T02:07:22.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:22.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.085972+0000) 2020-02-06T02:07:22.088 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:22.086+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.087602+0000) 2020-02-06T02:07:22.089 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:22.087+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.088277+0000) 2020-02-06T02:07:22.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:22.087+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.088593+0000) 2020-02-06T02:07:22.094 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:22.092+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:22.093708+0000) 2020-02-06T02:07:22.106 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:22.104+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223434 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:23.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:23.504+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:23.505136+0000) 2020-02-06T02:07:26.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:26.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:26.167323+0000) 2020-02-06T02:07:26.531 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:26.529+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:26.530750+0000) 2020-02-06T02:07:27.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:27.110+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.111171+0000) 2020-02-06T02:07:27.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:27.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.112602+0000) 2020-02-06T02:07:27.114 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:27.111+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.112779+0000) 2020-02-06T02:07:27.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:27.112+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.113255+0000) 2020-02-06T02:07:27.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:27.112+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.113625+0000) 2020-02-06T02:07:27.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:27.113+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.114192+0000) 2020-02-06T02:07:27.115 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:27.113+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.114654+0000) 2020-02-06T02:07:27.116 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:27.113+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.115128+0000) 2020-02-06T02:07:27.116 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:27.114+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.115859+0000) 2020-02-06T02:07:27.117 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:27.115+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.116647+0000) 2020-02-06T02:07:27.118 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:27.115+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.116811+0000) 2020-02-06T02:07:27.118 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:27.115+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.117008+0000) 2020-02-06T02:07:27.118 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:27.114+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.116314+0000) 2020-02-06T02:07:27.123 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:27.121+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.122427+0000) 2020-02-06T02:07:27.136 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:27.134+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223471 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:27.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:27.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.145523+0000) 2020-02-06T02:07:27.420 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:27.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.419450+0000) 2020-02-06T02:07:27.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:27.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.419476+0000) 2020-02-06T02:07:27.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:27.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.419680+0000) 2020-02-06T02:07:27.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:27.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.650342+0000) 2020-02-06T02:07:27.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:27.673+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:27.673721+0000) 2020-02-06T02:07:28.228 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:28.227+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:28.227696+0000) 2020-02-06T02:07:29.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:29.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:29.789630+0000) 2020-02-06T02:07:30.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:30.035+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:30.035377+0000) 2020-02-06T02:07:30.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:30.645+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:30.645981+0000) 2020-02-06T02:07:30.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:30.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:30.923893+0000) 2020-02-06T02:07:32.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:32.140+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.140700+0000) 2020-02-06T02:07:32.143 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:32.142+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.142618+0000) 2020-02-06T02:07:32.144 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:32.141+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.143247+0000) 2020-02-06T02:07:32.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:32.143+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.143958+0000) 2020-02-06T02:07:32.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:32.142+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.143992+0000) 2020-02-06T02:07:32.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:32.144+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.144346+0000) 2020-02-06T02:07:32.146 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:32.143+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.144773+0000) 2020-02-06T02:07:32.146 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:32.145+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.145227+0000) 2020-02-06T02:07:32.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:32.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.145551+0000) 2020-02-06T02:07:32.147 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:32.145+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.145734+0000) 2020-02-06T02:07:32.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:32.145+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.146869+0000) 2020-02-06T02:07:32.148 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:32.146+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.147453+0000) 2020-02-06T02:07:32.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:32.146+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.147604+0000) 2020-02-06T02:07:32.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:32.152+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:32.152482+0000) 2020-02-06T02:07:32.165 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:32.162+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223517 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:33.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:33.504+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:33.505406+0000) 2020-02-06T02:07:36.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:36.166+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:36.167590+0000) 2020-02-06T02:07:36.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:36.529+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:36.530979+0000) 2020-02-06T02:07:37.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:37.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.145730+0000) 2020-02-06T02:07:37.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:37.170+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.171389+0000) 2020-02-06T02:07:37.172 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:37.170+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.171356+0000) 2020-02-06T02:07:37.173 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:37.170+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.171630+0000) 2020-02-06T02:07:37.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:37.171+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.172068+0000) 2020-02-06T02:07:37.174 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:37.171+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.172503+0000) 2020-02-06T02:07:37.174 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:37.171+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.172714+0000) 2020-02-06T02:07:37.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:37.172+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.173595+0000) 2020-02-06T02:07:37.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:37.172+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.173335+0000) 2020-02-06T02:07:37.175 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:37.172+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.173909+0000) 2020-02-06T02:07:37.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:37.173+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.174569+0000) 2020-02-06T02:07:37.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:37.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.175297+0000) 2020-02-06T02:07:37.177 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:37.174+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.175768+0000) 2020-02-06T02:07:37.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:37.178+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.179392+0000) 2020-02-06T02:07:37.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:37.178+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.179627+0000) 2020-02-06T02:07:37.194 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:37.192+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:37.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:37.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.419686+0000) 2020-02-06T02:07:37.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:37.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.419675+0000) 2020-02-06T02:07:37.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:37.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.419930+0000) 2020-02-06T02:07:37.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:37.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.650563+0000) 2020-02-06T02:07:37.674 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:37.672+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:37.673926+0000) 2020-02-06T02:07:38.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:38.235+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:38.236313+0000) 2020-02-06T02:07:39.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:39.788+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:39.789864+0000) 2020-02-06T02:07:40.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:40.034+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:40.035604+0000) 2020-02-06T02:07:40.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:40.645+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:40.646230+0000) 2020-02-06T02:07:40.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:40.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:40.924135+0000) 2020-02-06T02:07:42.200 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:42.198+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.199336+0000) 2020-02-06T02:07:42.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:42.198+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.200133+0000) 2020-02-06T02:07:42.201 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:42.199+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.200858+0000) 2020-02-06T02:07:42.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:42.199+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.201196+0000) 2020-02-06T02:07:42.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:42.200+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.201604+0000) 2020-02-06T02:07:42.203 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:42.200+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.201678+0000) 2020-02-06T02:07:42.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:42.201+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.202239+0000) 2020-02-06T02:07:42.204 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:42.201+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.202799+0000) 2020-02-06T02:07:42.204 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:42.201+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.203015+0000) 2020-02-06T02:07:42.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:42.201+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.202421+0000) 2020-02-06T02:07:42.205 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:42.202+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.203716+0000) 2020-02-06T02:07:42.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:42.203+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.204549+0000) 2020-02-06T02:07:42.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:42.203+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.205405+0000) 2020-02-06T02:07:42.209 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:42.207+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:42.208998+0000) 2020-02-06T02:07:42.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:42.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223596 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:43.123 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:07:43.129 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:07:43.506 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:43.504+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:43.505645+0000) 2020-02-06T02:07:46.168 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:46.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:46.167854+0000) 2020-02-06T02:07:46.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:46.529+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:46.531217+0000) 2020-02-06T02:07:47.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:47.144+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.145910+0000) 2020-02-06T02:07:47.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:47.227+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.228811+0000) 2020-02-06T02:07:47.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:47.227+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.228507+0000) 2020-02-06T02:07:47.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:47.228+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.229289+0000) 2020-02-06T02:07:47.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:47.228+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.229636+0000) 2020-02-06T02:07:47.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:47.228+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.229708+0000) 2020-02-06T02:07:47.231 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:47.229+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.230179+0000) 2020-02-06T02:07:47.231 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:47.229+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.230385+0000) 2020-02-06T02:07:47.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:47.229+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.231027+0000) 2020-02-06T02:07:47.232 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:47.229+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.230984+0000) 2020-02-06T02:07:47.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:47.231+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.232411+0000) 2020-02-06T02:07:47.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:47.231+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.232458+0000) 2020-02-06T02:07:47.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:47.231+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.233088+0000) 2020-02-06T02:07:47.234 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:47.232+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.233818+0000) 2020-02-06T02:07:47.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:47.234+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.235569+0000) 2020-02-06T02:07:47.251 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:47.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223632 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:47.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:47.419+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.419909+0000) 2020-02-06T02:07:47.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:47.419+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.419914+0000) 2020-02-06T02:07:47.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:47.419+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.420110+0000) 2020-02-06T02:07:47.651 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:47.649+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.650795+0000) 2020-02-06T02:07:47.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:47.673+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:47.674130+0000) 2020-02-06T02:07:48.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:48.244+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:48.244971+0000) 2020-02-06T02:07:49.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:49.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:49.790125+0000) 2020-02-06T02:07:50.036 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:50.034+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:50.035829+0000) 2020-02-06T02:07:50.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:50.645+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:50.646481+0000) 2020-02-06T02:07:50.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:50.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:50.924424+0000) 2020-02-06T02:07:52.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:52.254+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.255443+0000) 2020-02-06T02:07:52.256 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:52.253+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.255106+0000) 2020-02-06T02:07:52.258 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:52.256+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.257251+0000) 2020-02-06T02:07:52.258 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:52.256+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.257715+0000) 2020-02-06T02:07:52.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:52.257+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.258464+0000) 2020-02-06T02:07:52.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:52.257+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.258632+0000) 2020-02-06T02:07:52.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:52.258+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.259698+0000) 2020-02-06T02:07:52.260 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:52.258+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.259692+0000) 2020-02-06T02:07:52.261 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:52.258+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.259999+0000) 2020-02-06T02:07:52.261 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:52.259+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.260236+0000) 2020-02-06T02:07:52.261 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:52.259+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.261069+0000) 2020-02-06T02:07:52.262 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:52.259+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.261223+0000) 2020-02-06T02:07:52.262 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:52.260+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.261680+0000) 2020-02-06T02:07:52.266 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:52.264+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:52.265856+0000) 2020-02-06T02:07:52.280 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:52.277+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223675 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:53.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:53.505+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:53.505896+0000) 2020-02-06T02:07:56.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:56.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:56.168111+0000) 2020-02-06T02:07:56.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:56.530+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:56.531491+0000) 2020-02-06T02:07:57.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:57.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.146131+0000) 2020-02-06T02:07:57.285 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:07:57.282+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.283907+0000) 2020-02-06T02:07:57.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:07:57.283+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.284438+0000) 2020-02-06T02:07:57.286 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:07:57.284+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.285840+0000) 2020-02-06T02:07:57.287 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:57.285+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.286530+0000) 2020-02-06T02:07:57.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:57.285+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.286635+0000) 2020-02-06T02:07:57.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:57.286+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.287118+0000) 2020-02-06T02:07:57.288 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:07:57.286+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.287767+0000) 2020-02-06T02:07:57.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:07:57.286+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.288302+0000) 2020-02-06T02:07:57.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:57.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.288884+0000) 2020-02-06T02:07:57.290 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:07:57.287+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.288977+0000) 2020-02-06T02:07:57.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:57.288+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.289513+0000) 2020-02-06T02:07:57.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:57.289+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.290897+0000) 2020-02-06T02:07:57.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:57.290+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.291347+0000) 2020-02-06T02:07:57.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:57.294+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.295898+0000) 2020-02-06T02:07:57.309 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:07:57.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223711 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:07:57.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:07:57.418+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.420135+0000) 2020-02-06T02:07:57.421 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:07:57.418+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.420138+0000) 2020-02-06T02:07:57.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:07:57.418+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.420293+0000) 2020-02-06T02:07:57.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:07:57.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.651011+0000) 2020-02-06T02:07:57.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:07:57.673+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:57.674333+0000) 2020-02-06T02:07:58.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:58.253+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:58.253864+0000) 2020-02-06T02:07:59.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:07:59.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:07:59.790319+0000) 2020-02-06T02:08:00.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:00.035+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:00.036111+0000) 2020-02-06T02:08:00.647 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:00.646+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:00.646764+0000) 2020-02-06T02:08:00.925 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:00.923+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:00.924670+0000) 2020-02-06T02:08:02.315 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:02.313+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.314554+0000) 2020-02-06T02:08:02.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:02.314+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.315413+0000) 2020-02-06T02:08:02.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:02.314+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.315778+0000) 2020-02-06T02:08:02.316 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:02.314+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.315844+0000) 2020-02-06T02:08:02.317 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:02.314+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.315934+0000) 2020-02-06T02:08:02.317 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:02.315+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.316605+0000) 2020-02-06T02:08:02.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:02.315+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.316516+0000) 2020-02-06T02:08:02.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:02.315+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.317135+0000) 2020-02-06T02:08:02.319 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:02.316+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.317817+0000) 2020-02-06T02:08:02.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:02.316+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.317803+0000) 2020-02-06T02:08:02.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:02.316+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.318010+0000) 2020-02-06T02:08:02.319 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:02.315+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.316967+0000) 2020-02-06T02:08:02.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:02.316+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.317535+0000) 2020-02-06T02:08:02.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:02.320+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:02.322009+0000) 2020-02-06T02:08:02.336 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:02.334+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223755 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:03.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:03.505+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:03.506160+0000) 2020-02-06T02:08:06.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:06.167+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:06.168370+0000) 2020-02-06T02:08:06.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:06.531+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:06.531774+0000) 2020-02-06T02:08:07.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:07.145+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.146364+0000) 2020-02-06T02:08:07.343 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:07.341+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.342250+0000) 2020-02-06T02:08:07.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:07.341+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.342919+0000) 2020-02-06T02:08:07.344 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:07.342+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.343587+0000) 2020-02-06T02:08:07.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:07.341+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.343259+0000) 2020-02-06T02:08:07.345 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:07.343+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.344472+0000) 2020-02-06T02:08:07.346 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:07.344+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.345339+0000) 2020-02-06T02:08:07.346 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:07.344+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.345723+0000) 2020-02-06T02:08:07.348 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:07.344+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.346200+0000) 2020-02-06T02:08:07.348 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:07.345+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.346781+0000) 2020-02-06T02:08:07.348 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:07.345+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.346871+0000) 2020-02-06T02:08:07.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:07.345+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.346271+0000) 2020-02-06T02:08:07.349 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:07.345+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.347268+0000) 2020-02-06T02:08:07.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:07.346+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.347454+0000) 2020-02-06T02:08:07.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:07.352+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.353342+0000) 2020-02-06T02:08:07.365 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:07.363+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223792 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:07.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:07.419+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.420364+0000) 2020-02-06T02:08:07.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:07.419+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.420363+0000) 2020-02-06T02:08:07.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:07.419+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.420544+0000) 2020-02-06T02:08:07.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:07.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.651261+0000) 2020-02-06T02:08:07.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:07.674+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:07.674567+0000) 2020-02-06T02:08:08.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:08.262+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:08.262763+0000) 2020-02-06T02:08:09.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:09.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:09.790576+0000) 2020-02-06T02:08:10.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:10.036+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:10.036383+0000) 2020-02-06T02:08:10.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:10.646+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:10.647014+0000) 2020-02-06T02:08:10.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:10.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:10.924904+0000) 2020-02-06T02:08:12.371 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:12.369+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.370652+0000) 2020-02-06T02:08:12.372 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:12.369+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.370257+0000) 2020-02-06T02:08:12.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:12.370+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.371424+0000) 2020-02-06T02:08:12.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:12.371+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.371911+0000) 2020-02-06T02:08:12.374 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:12.371+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.372648+0000) 2020-02-06T02:08:12.374 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:12.372+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.373264+0000) 2020-02-06T02:08:12.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:12.373+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.373560+0000) 2020-02-06T02:08:12.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:12.372+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.373961+0000) 2020-02-06T02:08:12.376 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:12.375+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.375532+0000) 2020-02-06T02:08:12.377 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:12.373+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.375020+0000) 2020-02-06T02:08:12.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:12.375+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.376409+0000) 2020-02-06T02:08:12.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:12.376+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.376520+0000) 2020-02-06T02:08:12.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:12.374+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.376032+0000) 2020-02-06T02:08:12.379 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:12.377+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:12.378714+0000) 2020-02-06T02:08:12.397 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:12.395+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:13.265 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:08:13.271 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:08:13.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:13.506+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:13.506397+0000) 2020-02-06T02:08:16.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:16.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:16.168640+0000) 2020-02-06T02:08:16.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:16.531+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:16.532072+0000) 2020-02-06T02:08:17.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:17.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.146600+0000) 2020-02-06T02:08:17.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:17.400+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.401724+0000) 2020-02-06T02:08:17.403 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:17.401+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.402291+0000) 2020-02-06T02:08:17.403 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:17.401+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.402828+0000) 2020-02-06T02:08:17.404 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:17.403+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.403630+0000) 2020-02-06T02:08:17.405 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:17.403+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.404151+0000) 2020-02-06T02:08:17.405 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:17.403+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.404287+0000) 2020-02-06T02:08:17.406 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:17.403+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.404945+0000) 2020-02-06T02:08:17.406 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:17.404+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.405346+0000) 2020-02-06T02:08:17.406 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:17.405+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.405787+0000) 2020-02-06T02:08:17.407 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:17.405+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.406421+0000) 2020-02-06T02:08:17.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:17.405+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.406114+0000) 2020-02-06T02:08:17.408 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:17.406+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.407006+0000) 2020-02-06T02:08:17.408 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:17.406+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.407153+0000) 2020-02-06T02:08:17.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:17.411+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.412085+0000) 2020-02-06T02:08:17.421 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:17.420+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.420621+0000) 2020-02-06T02:08:17.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:17.420+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.420606+0000) 2020-02-06T02:08:17.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:17.420+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.420742+0000) 2020-02-06T02:08:17.426 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:17.424+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:17.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:17.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.651474+0000) 2020-02-06T02:08:17.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:17.674+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:17.674773+0000) 2020-02-06T02:08:18.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:18.271+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:18.271393+0000) 2020-02-06T02:08:19.791 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:19.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:19.790769+0000) 2020-02-06T02:08:20.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:20.036+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:20.036610+0000) 2020-02-06T02:08:20.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:20.646+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:20.647313+0000) 2020-02-06T02:08:20.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:20.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:20.925132+0000) 2020-02-06T02:08:22.432 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:22.430+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.431142+0000) 2020-02-06T02:08:22.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:22.430+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.431913+0000) 2020-02-06T02:08:22.433 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:22.431+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.432526+0000) 2020-02-06T02:08:22.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:22.432+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.433302+0000) 2020-02-06T02:08:22.434 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:22.432+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.433402+0000) 2020-02-06T02:08:22.434 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:22.432+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.433718+0000) 2020-02-06T02:08:22.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:22.433+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.434290+0000) 2020-02-06T02:08:22.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:22.433+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.434308+0000) 2020-02-06T02:08:22.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:22.433+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.434776+0000) 2020-02-06T02:08:22.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:22.434+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.435131+0000) 2020-02-06T02:08:22.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:22.434+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.435246+0000) 2020-02-06T02:08:22.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:22.436+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.436533+0000) 2020-02-06T02:08:22.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:22.435+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.436855+0000) 2020-02-06T02:08:22.441 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:22.440+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:22.441141+0000) 2020-02-06T02:08:22.455 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:22.454+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223913 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:23.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:23.506+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:23.506651+0000) 2020-02-06T02:08:26.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:26.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:26.168881+0000) 2020-02-06T02:08:26.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:26.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:26.532326+0000) 2020-02-06T02:08:27.147 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:27.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.146834+0000) 2020-02-06T02:08:27.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:27.420+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.420860+0000) 2020-02-06T02:08:27.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:27.420+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.420872+0000) 2020-02-06T02:08:27.422 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:27.420+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.420900+0000) 2020-02-06T02:08:27.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:27.460+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.460245+0000) 2020-02-06T02:08:27.462 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:27.459+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.460389+0000) 2020-02-06T02:08:27.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:27.460+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.460845+0000) 2020-02-06T02:08:27.462 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:27.461+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.461273+0000) 2020-02-06T02:08:27.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:27.462+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.462268+0000) 2020-02-06T02:08:27.464 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:27.462+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.462174+0000) 2020-02-06T02:08:27.464 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:27.462+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.463387+0000) 2020-02-06T02:08:27.464 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:27.463+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.463861+0000) 2020-02-06T02:08:27.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:27.463+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.464333+0000) 2020-02-06T02:08:27.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:27.465+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.465618+0000) 2020-02-06T02:08:27.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:27.465+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.465834+0000) 2020-02-06T02:08:27.467 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:27.466+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.466240+0000) 2020-02-06T02:08:27.470 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:27.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.469955+0000) 2020-02-06T02:08:27.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:27.471+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.471093+0000) 2020-02-06T02:08:27.485 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:27.483+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223950 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:27.652 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:27.650+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.651692+0000) 2020-02-06T02:08:27.675 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:27.673+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:27.674979+0000) 2020-02-06T02:08:28.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:28.278+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:28.279825+0000) 2020-02-06T02:08:29.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:29.789+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:29.791026+0000) 2020-02-06T02:08:30.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:30.035+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:30.036842+0000) 2020-02-06T02:08:30.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:30.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:30.647587+0000) 2020-02-06T02:08:30.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:30.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:30.925371+0000) 2020-02-06T02:08:32.489 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:32.487+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.488438+0000) 2020-02-06T02:08:32.492 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:32.490+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.491071+0000) 2020-02-06T02:08:32.492 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:32.491+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.491616+0000) 2020-02-06T02:08:32.493 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:32.490+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.491930+0000) 2020-02-06T02:08:32.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:32.491+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.492459+0000) 2020-02-06T02:08:32.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:32.492+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.492748+0000) 2020-02-06T02:08:32.494 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:32.492+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.493473+0000) 2020-02-06T02:08:32.494 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:32.492+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.493927+0000) 2020-02-06T02:08:32.495 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:32.494+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.494643+0000) 2020-02-06T02:08:32.495 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:32.494+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.495173+0000) 2020-02-06T02:08:32.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:32.495+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.496077+0000) 2020-02-06T02:08:32.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:32.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.496247+0000) 2020-02-06T02:08:32.497 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:32.495+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.496313+0000) 2020-02-06T02:08:32.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:32.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:32.499103+0000) 2020-02-06T02:08:32.514 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:32.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 223991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:33.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:33.505+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:33.506879+0000) 2020-02-06T02:08:36.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:36.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:36.169132+0000) 2020-02-06T02:08:36.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:36.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:36.532556+0000) 2020-02-06T02:08:37.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:37.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.147067+0000) 2020-02-06T02:08:37.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:37.420+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.421161+0000) 2020-02-06T02:08:37.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:37.420+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.421161+0000) 2020-02-06T02:08:37.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:37.420+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.421159+0000) 2020-02-06T02:08:37.519 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:37.517+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.518665+0000) 2020-02-06T02:08:37.520 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:37.518+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.519124+0000) 2020-02-06T02:08:37.521 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:37.519+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.520329+0000) 2020-02-06T02:08:37.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:37.519+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.521049+0000) 2020-02-06T02:08:37.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:37.521+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.522071+0000) 2020-02-06T02:08:37.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:37.522+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.522848+0000) 2020-02-06T02:08:37.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:37.522+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.523443+0000) 2020-02-06T02:08:37.524 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:37.523+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.523861+0000) 2020-02-06T02:08:37.525 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:37.523+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.524256+0000) 2020-02-06T02:08:37.525 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:37.524+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.525060+0000) 2020-02-06T02:08:37.526 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:37.525+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.525483+0000) 2020-02-06T02:08:37.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:37.524+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.525446+0000) 2020-02-06T02:08:37.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:37.526+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.527714+0000) 2020-02-06T02:08:37.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:37.526+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.527977+0000) 2020-02-06T02:08:37.544 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:37.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224029 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:37.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:37.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.651915+0000) 2020-02-06T02:08:37.676 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:37.674+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:37.675190+0000) 2020-02-06T02:08:38.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:38.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:38.287721+0000) 2020-02-06T02:08:39.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:39.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:39.791320+0000) 2020-02-06T02:08:40.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:40.036+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:40.037071+0000) 2020-02-06T02:08:40.648 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:40.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:40.647839+0000) 2020-02-06T02:08:40.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:40.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:40.925622+0000) 2020-02-06T02:08:42.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:42.548+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.549340+0000) 2020-02-06T02:08:42.550 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:42.548+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.549347+0000) 2020-02-06T02:08:42.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:42.549+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.550407+0000) 2020-02-06T02:08:42.551 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:42.549+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.550911+0000) 2020-02-06T02:08:42.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:42.550+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.550777+0000) 2020-02-06T02:08:42.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:42.551+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.551970+0000) 2020-02-06T02:08:42.553 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:42.551+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.551715+0000) 2020-02-06T02:08:42.553 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:42.551+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.552304+0000) 2020-02-06T02:08:42.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:42.552+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.552873+0000) 2020-02-06T02:08:42.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:42.551+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.552946+0000) 2020-02-06T02:08:42.554 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:42.553+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.554011+0000) 2020-02-06T02:08:42.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:42.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.553688+0000) 2020-02-06T02:08:42.555 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:42.553+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.554453+0000) 2020-02-06T02:08:42.559 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:42.557+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:42.558449+0000) 2020-02-06T02:08:42.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:42.573+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224070 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:43.397 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:08:43.403 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:08:43.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:43.506+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:43.507162+0000) 2020-02-06T02:08:46.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:46.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:46.169367+0000) 2020-02-06T02:08:46.533 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:46.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:46.532835+0000) 2020-02-06T02:08:47.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:47.146+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.147349+0000) 2020-02-06T02:08:47.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:47.420+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.421459+0000) 2020-02-06T02:08:47.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:47.420+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.421423+0000) 2020-02-06T02:08:47.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:47.420+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.421473+0000) 2020-02-06T02:08:47.579 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:47.577+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.578590+0000) 2020-02-06T02:08:47.580 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:47.578+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.579382+0000) 2020-02-06T02:08:47.580 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:47.578+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.579628+0000) 2020-02-06T02:08:47.582 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:47.580+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.581545+0000) 2020-02-06T02:08:47.582 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:47.580+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.582019+0000) 2020-02-06T02:08:47.583 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:47.581+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.582802+0000) 2020-02-06T02:08:47.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:47.582+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.584059+0000) 2020-02-06T02:08:47.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:47.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.584269+0000) 2020-02-06T02:08:47.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:47.583+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.584495+0000) 2020-02-06T02:08:47.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:47.583+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.585045+0000) 2020-02-06T02:08:47.586 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:47.583+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.585205+0000) 2020-02-06T02:08:47.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:47.584+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.585611+0000) 2020-02-06T02:08:47.587 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:47.584+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.585258+0000) 2020-02-06T02:08:47.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:47.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.590234+0000) 2020-02-06T02:08:47.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:47.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224108 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:47.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:47.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.652172+0000) 2020-02-06T02:08:47.676 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:47.674+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:47.675429+0000) 2020-02-06T02:08:48.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:48.295+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:48.296008+0000) 2020-02-06T02:08:49.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:49.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:49.791559+0000) 2020-02-06T02:08:50.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:50.036+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:50.037373+0000) 2020-02-06T02:08:50.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:50.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:50.648117+0000) 2020-02-06T02:08:50.926 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:50.924+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:50.925880+0000) 2020-02-06T02:08:52.610 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:52.608+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.609422+0000) 2020-02-06T02:08:52.611 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:52.610+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.611057+0000) 2020-02-06T02:08:52.612 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:52.610+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.611416+0000) 2020-02-06T02:08:52.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:52.610+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.611956+0000) 2020-02-06T02:08:52.613 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:52.611+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.612197+0000) 2020-02-06T02:08:52.613 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:52.611+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.612598+0000) 2020-02-06T02:08:52.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:52.612+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.613390+0000) 2020-02-06T02:08:52.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:52.612+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.613182+0000) 2020-02-06T02:08:52.615 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:52.612+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.613650+0000) 2020-02-06T02:08:52.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:52.612+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.613941+0000) 2020-02-06T02:08:52.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:52.613+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.615007+0000) 2020-02-06T02:08:52.616 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:52.613+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.615002+0000) 2020-02-06T02:08:52.616 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:52.614+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.615945+0000) 2020-02-06T02:08:52.619 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:52.617+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:52.619055+0000) 2020-02-06T02:08:52.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:52.631+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224149 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:53.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:53.506+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:53.507378+0000) 2020-02-06T02:08:56.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:56.168+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:56.169695+0000) 2020-02-06T02:08:56.534 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:56.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:56.533075+0000) 2020-02-06T02:08:57.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:57.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.147623+0000) 2020-02-06T02:08:57.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:57.420+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.421726+0000) 2020-02-06T02:08:57.423 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:57.420+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.421738+0000) 2020-02-06T02:08:57.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:57.420+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.421737+0000) 2020-02-06T02:08:57.639 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:08:57.637+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.638389+0000) 2020-02-06T02:08:57.639 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:57.637+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.638811+0000) 2020-02-06T02:08:57.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:57.638+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.639277+0000) 2020-02-06T02:08:57.640 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:08:57.638+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.639707+0000) 2020-02-06T02:08:57.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:57.638+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.639979+0000) 2020-02-06T02:08:57.641 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:08:57.638+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.640269+0000) 2020-02-06T02:08:57.641 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:08:57.639+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.640522+0000) 2020-02-06T02:08:57.642 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:08:57.639+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.641391+0000) 2020-02-06T02:08:57.642 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:08:57.640+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.641638+0000) 2020-02-06T02:08:57.643 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:57.640+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.641959+0000) 2020-02-06T02:08:57.643 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:08:57.640+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.642356+0000) 2020-02-06T02:08:57.643 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:08:57.641+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.642437+0000) 2020-02-06T02:08:57.644 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:08:57.642+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.643530+0000) 2020-02-06T02:08:57.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:57.645+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.647018+0000) 2020-02-06T02:08:57.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:08:57.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.652374+0000) 2020-02-06T02:08:57.662 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:08:57.660+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224187 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:08:57.676 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:08:57.674+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:57.675616+0000) 2020-02-06T02:08:58.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:58.303+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:58.304434+0000) 2020-02-06T02:08:59.792 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:08:59.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:08:59.791773+0000) 2020-02-06T02:09:00.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:00.037+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:00.037602+0000) 2020-02-06T02:09:00.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:00.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:00.648374+0000) 2020-02-06T02:09:00.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:00.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:00.926131+0000) 2020-02-06T02:09:02.668 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:02.666+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.667544+0000) 2020-02-06T02:09:02.670 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:02.667+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.669047+0000) 2020-02-06T02:09:02.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:02.667+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.668835+0000) 2020-02-06T02:09:02.671 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:02.668+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.669302+0000) 2020-02-06T02:09:02.671 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:02.668+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.669449+0000) 2020-02-06T02:09:02.672 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:02.668+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.670269+0000) 2020-02-06T02:09:02.673 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:02.669+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.671312+0000) 2020-02-06T02:09:02.673 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:02.669+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.670545+0000) 2020-02-06T02:09:02.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:02.669+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.670643+0000) 2020-02-06T02:09:02.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:02.670+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.671181+0000) 2020-02-06T02:09:02.675 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:02.670+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.671853+0000) 2020-02-06T02:09:02.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:02.670+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.671980+0000) 2020-02-06T02:09:02.675 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:02.671+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.672810+0000) 2020-02-06T02:09:02.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:02.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:02.675743+0000) 2020-02-06T02:09:02.692 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:02.690+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224229 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:03.509 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:03.507+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:03.507596+0000) 2020-02-06T02:09:06.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:06.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:06.170024+0000) 2020-02-06T02:09:06.534 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:06.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:06.533327+0000) 2020-02-06T02:09:07.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:07.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.147881+0000) 2020-02-06T02:09:07.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:07.421+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.421983+0000) 2020-02-06T02:09:07.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:07.421+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.422029+0000) 2020-02-06T02:09:07.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:07.421+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.422051+0000) 2020-02-06T02:09:07.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:07.651+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.652601+0000) 2020-02-06T02:09:07.676 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:07.675+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.675860+0000) 2020-02-06T02:09:07.698 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:07.696+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.697517+0000) 2020-02-06T02:09:07.699 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:07.697+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.698310+0000) 2020-02-06T02:09:07.700 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:07.697+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.699194+0000) 2020-02-06T02:09:07.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:07.698+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.699465+0000) 2020-02-06T02:09:07.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:07.699+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.700256+0000) 2020-02-06T02:09:07.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:07.699+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.700680+0000) 2020-02-06T02:09:07.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:07.700+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.701547+0000) 2020-02-06T02:09:07.702 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:07.699+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.701301+0000) 2020-02-06T02:09:07.703 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:07.700+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.702023+0000) 2020-02-06T02:09:07.703 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:07.701+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.702464+0000) 2020-02-06T02:09:07.704 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:07.701+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.703163+0000) 2020-02-06T02:09:07.704 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:07.701+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.703357+0000) 2020-02-06T02:09:07.704 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:07.702+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.703332+0000) 2020-02-06T02:09:07.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:07.707+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:07.708095+0000) 2020-02-06T02:09:07.722 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:07.719+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224266 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:08.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:08.312+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:08.312814+0000) 2020-02-06T02:09:09.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:09.790+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:09.792033+0000) 2020-02-06T02:09:10.038 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:10.036+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:10.037836+0000) 2020-02-06T02:09:10.649 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:10.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:10.648654+0000) 2020-02-06T02:09:10.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:10.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:10.926360+0000) 2020-02-06T02:09:12.727 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:12.725+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.726822+0000) 2020-02-06T02:09:12.728 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:12.726+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.727968+0000) 2020-02-06T02:09:12.729 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:12.727+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.728375+0000) 2020-02-06T02:09:12.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:12.727+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.728753+0000) 2020-02-06T02:09:12.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:12.728+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.729643+0000) 2020-02-06T02:09:12.730 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:12.729+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.730140+0000) 2020-02-06T02:09:12.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:12.728+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.729992+0000) 2020-02-06T02:09:12.731 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:12.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.730523+0000) 2020-02-06T02:09:12.732 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:12.730+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.731310+0000) 2020-02-06T02:09:12.732 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:12.730+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.731721+0000) 2020-02-06T02:09:12.733 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:12.730+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.732176+0000) 2020-02-06T02:09:12.733 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:12.731+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.733050+0000) 2020-02-06T02:09:12.734 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:12.731+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.732889+0000) 2020-02-06T02:09:12.738 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:12.736+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:12.737452+0000) 2020-02-06T02:09:12.750 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:12.748+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:13.508 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:13.507+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:13.507788+0000) 2020-02-06T02:09:13.527 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:09:13.533 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:09:16.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:16.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:16.170272+0000) 2020-02-06T02:09:16.534 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:16.532+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:16.533585+0000) 2020-02-06T02:09:17.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:17.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.148116+0000) 2020-02-06T02:09:17.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:17.421+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.422244+0000) 2020-02-06T02:09:17.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:17.421+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.422277+0000) 2020-02-06T02:09:17.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:17.421+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.422359+0000) 2020-02-06T02:09:17.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:17.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.652843+0000) 2020-02-06T02:09:17.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:17.675+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.676136+0000) 2020-02-06T02:09:17.755 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:17.753+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.754923+0000) 2020-02-06T02:09:17.756 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:17.754+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.755835+0000) 2020-02-06T02:09:17.757 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:17.754+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.756306+0000) 2020-02-06T02:09:17.757 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:17.756+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.756948+0000) 2020-02-06T02:09:17.758 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:17.756+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.757339+0000) 2020-02-06T02:09:17.758 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:17.756+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.757832+0000) 2020-02-06T02:09:17.759 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:17.757+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.758375+0000) 2020-02-06T02:09:17.760 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:17.757+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.758775+0000) 2020-02-06T02:09:17.760 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:17.758+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.759326+0000) 2020-02-06T02:09:17.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:17.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.760814+0000) 2020-02-06T02:09:17.762 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:17.759+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.760780+0000) 2020-02-06T02:09:17.762 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:17.760+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.761731+0000) 2020-02-06T02:09:17.762 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:17.760+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.761717+0000) 2020-02-06T02:09:17.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:17.765+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:17.765417+0000) 2020-02-06T02:09:17.780 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:17.777+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224345 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:18.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:18.321+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:18.321304+0000) 2020-02-06T02:09:19.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:19.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:19.792295+0000) 2020-02-06T02:09:20.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:20.038+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:20.038064+0000) 2020-02-06T02:09:20.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:20.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:20.648907+0000) 2020-02-06T02:09:20.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:20.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:20.926624+0000) 2020-02-06T02:09:22.786 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:22.784+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.785299+0000) 2020-02-06T02:09:22.786 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:22.784+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.786058+0000) 2020-02-06T02:09:22.787 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:22.785+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.786580+0000) 2020-02-06T02:09:22.787 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:22.785+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.786761+0000) 2020-02-06T02:09:22.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:22.785+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.787058+0000) 2020-02-06T02:09:22.788 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:22.786+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.787229+0000) 2020-02-06T02:09:22.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:22.786+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.787541+0000) 2020-02-06T02:09:22.789 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:22.786+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.787788+0000) 2020-02-06T02:09:22.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:22.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.788474+0000) 2020-02-06T02:09:22.790 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:22.786+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.788393+0000) 2020-02-06T02:09:22.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:22.787+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.788680+0000) 2020-02-06T02:09:22.790 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:22.787+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.788806+0000) 2020-02-06T02:09:22.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:22.787+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.789002+0000) 2020-02-06T02:09:22.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:22.792+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:22.793718+0000) 2020-02-06T02:09:22.809 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:22.806+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224387 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:23.509 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:23.506+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:23.507987+0000) 2020-02-06T02:09:26.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:26.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:26.170539+0000) 2020-02-06T02:09:26.534 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:26.533+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:26.533816+0000) 2020-02-06T02:09:27.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:27.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.148345+0000) 2020-02-06T02:09:27.423 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:27.421+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.422479+0000) 2020-02-06T02:09:27.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:27.421+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.422516+0000) 2020-02-06T02:09:27.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:27.421+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.422651+0000) 2020-02-06T02:09:27.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:27.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.653125+0000) 2020-02-06T02:09:27.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:27.675+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.676402+0000) 2020-02-06T02:09:27.814 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:27.812+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.813112+0000) 2020-02-06T02:09:27.816 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:27.813+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.815359+0000) 2020-02-06T02:09:27.816 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:27.814+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.815437+0000) 2020-02-06T02:09:27.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:27.814+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.815852+0000) 2020-02-06T02:09:27.817 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:27.814+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.816367+0000) 2020-02-06T02:09:27.817 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:27.815+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.816702+0000) 2020-02-06T02:09:27.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:27.816+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.817321+0000) 2020-02-06T02:09:27.818 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:27.816+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.817805+0000) 2020-02-06T02:09:27.818 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:27.816+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.818021+0000) 2020-02-06T02:09:27.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:27.817+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.819064+0000) 2020-02-06T02:09:27.820 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:27.818+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.819516+0000) 2020-02-06T02:09:27.820 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:27.818+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.819424+0000) 2020-02-06T02:09:27.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:27.818+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.819799+0000) 2020-02-06T02:09:27.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:27.824+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:27.825644+0000) 2020-02-06T02:09:27.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:27.837+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224424 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:28.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:28.328+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:28.329755+0000) 2020-02-06T02:09:29.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:29.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:29.792531+0000) 2020-02-06T02:09:30.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:30.037+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:30.038357+0000) 2020-02-06T02:09:30.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:30.647+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:30.649145+0000) 2020-02-06T02:09:30.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:30.925+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:30.926848+0000) 2020-02-06T02:09:32.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:32.844+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.845666+0000) 2020-02-06T02:09:32.847 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:32.845+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.846536+0000) 2020-02-06T02:09:32.847 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:32.845+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.846946+0000) 2020-02-06T02:09:32.848 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:32.846+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.847350+0000) 2020-02-06T02:09:32.848 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:32.846+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.847372+0000) 2020-02-06T02:09:32.849 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:32.846+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.847707+0000) 2020-02-06T02:09:32.849 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:32.846+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.847862+0000) 2020-02-06T02:09:32.849 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:32.846+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.848217+0000) 2020-02-06T02:09:32.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:32.846+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.848038+0000) 2020-02-06T02:09:32.850 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:32.847+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.848737+0000) 2020-02-06T02:09:32.850 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:32.848+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.849634+0000) 2020-02-06T02:09:32.850 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:32.848+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.849891+0000) 2020-02-06T02:09:32.851 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:32.848+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.850365+0000) 2020-02-06T02:09:32.855 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:32.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:32.854310+0000) 2020-02-06T02:09:32.869 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:32.867+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224465 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:33.509 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:33.507+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:33.508185+0000) 2020-02-06T02:09:36.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:36.169+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:36.170819+0000) 2020-02-06T02:09:36.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:36.533+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:36.534080+0000) 2020-02-06T02:09:37.149 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:37.147+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.148589+0000) 2020-02-06T02:09:37.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:37.422+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.422733+0000) 2020-02-06T02:09:37.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:37.422+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.422762+0000) 2020-02-06T02:09:37.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:37.422+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.422910+0000) 2020-02-06T02:09:37.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:37.652+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.653385+0000) 2020-02-06T02:09:37.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:37.675+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.676653+0000) 2020-02-06T02:09:37.875 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:37.873+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.874167+0000) 2020-02-06T02:09:37.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:37.873+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.874676+0000) 2020-02-06T02:09:37.876 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:37.874+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.875857+0000) 2020-02-06T02:09:37.877 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:37.875+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.876781+0000) 2020-02-06T02:09:37.878 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:37.876+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.877310+0000) 2020-02-06T02:09:37.878 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:37.875+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.876900+0000) 2020-02-06T02:09:37.879 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:37.876+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.877539+0000) 2020-02-06T02:09:37.879 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:37.876+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.877960+0000) 2020-02-06T02:09:37.880 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:37.877+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.878219+0000) 2020-02-06T02:09:37.880 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:37.877+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.878556+0000) 2020-02-06T02:09:37.880 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:37.876+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.878103+0000) 2020-02-06T02:09:37.880 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:37.876+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.878387+0000) 2020-02-06T02:09:37.880 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:37.877+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.878941+0000) 2020-02-06T02:09:37.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:37.882+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:37.883367+0000) 2020-02-06T02:09:37.898 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:37.896+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224504 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:38.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:38.337+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:38.337909+0000) 2020-02-06T02:09:39.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:39.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:39.792765+0000) 2020-02-06T02:09:40.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:40.037+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:40.038587+0000) 2020-02-06T02:09:40.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:40.649+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:40.649462+0000) 2020-02-06T02:09:40.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:40.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:40.927133+0000) 2020-02-06T02:09:42.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:42.901+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.902665+0000) 2020-02-06T02:09:42.904 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:42.901+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.903043+0000) 2020-02-06T02:09:42.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:42.904+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.904503+0000) 2020-02-06T02:09:42.906 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:42.904+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.905487+0000) 2020-02-06T02:09:42.906 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:42.904+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.905372+0000) 2020-02-06T02:09:42.907 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:42.905+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.906065+0000) 2020-02-06T02:09:42.907 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:42.906+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.906477+0000) 2020-02-06T02:09:42.907 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:42.905+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.906562+0000) 2020-02-06T02:09:42.908 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:42.906+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.907163+0000) 2020-02-06T02:09:42.908 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:42.906+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.907390+0000) 2020-02-06T02:09:42.908 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:42.905+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.906917+0000) 2020-02-06T02:09:42.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:42.907+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.907960+0000) 2020-02-06T02:09:42.909 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:42.907+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.908323+0000) 2020-02-06T02:09:42.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:42.910+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:42.911476+0000) 2020-02-06T02:09:42.929 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:42.927+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224546 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:43.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:43.507+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:43.508379+0000) 2020-02-06T02:09:43.667 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:09:43.673 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:09:46.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:46.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:46.171099+0000) 2020-02-06T02:09:46.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:46.533+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:46.534313+0000) 2020-02-06T02:09:47.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:47.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.148821+0000) 2020-02-06T02:09:47.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:47.422+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.422984+0000) 2020-02-06T02:09:47.424 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:47.422+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.423020+0000) 2020-02-06T02:09:47.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:47.422+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.423175+0000) 2020-02-06T02:09:47.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:47.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.653609+0000) 2020-02-06T02:09:47.677 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:47.676+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.676866+0000) 2020-02-06T02:09:47.934 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:47.932+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.933659+0000) 2020-02-06T02:09:47.935 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:47.933+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.934346+0000) 2020-02-06T02:09:47.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:47.934+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.935266+0000) 2020-02-06T02:09:47.936 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:47.934+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.934960+0000) 2020-02-06T02:09:47.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:47.935+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.936148+0000) 2020-02-06T02:09:47.937 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:47.935+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.936675+0000) 2020-02-06T02:09:47.937 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:47.936+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.936996+0000) 2020-02-06T02:09:47.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:47.937+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.937539+0000) 2020-02-06T02:09:47.938 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:47.937+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.938199+0000) 2020-02-06T02:09:47.939 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:47.937+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.938938+0000) 2020-02-06T02:09:47.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:47.938+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.939550+0000) 2020-02-06T02:09:47.940 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:47.939+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.939882+0000) 2020-02-06T02:09:47.944 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:47.942+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.943532+0000) 2020-02-06T02:09:47.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:47.943+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:47.944530+0000) 2020-02-06T02:09:47.958 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:47.957+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224584 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:48.346 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:48.343+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:48.344937+0000) 2020-02-06T02:09:49.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:49.791+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:49.793015+0000) 2020-02-06T02:09:50.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:50.038+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:50.038837+0000) 2020-02-06T02:09:50.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:50.649+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:50.649718+0000) 2020-02-06T02:09:50.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:50.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:50.927367+0000) 2020-02-06T02:09:52.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:52.961+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.962256+0000) 2020-02-06T02:09:52.963 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:52.961+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.963006+0000) 2020-02-06T02:09:52.964 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:52.962+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.963724+0000) 2020-02-06T02:09:52.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:52.964+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.964476+0000) 2020-02-06T02:09:52.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:52.964+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.965144+0000) 2020-02-06T02:09:52.966 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:52.964+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.965620+0000) 2020-02-06T02:09:52.966 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:52.965+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.965486+0000) 2020-02-06T02:09:52.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:52.966+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.966608+0000) 2020-02-06T02:09:52.968 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:52.966+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.967055+0000) 2020-02-06T02:09:52.968 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:52.967+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.967656+0000) 2020-02-06T02:09:52.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:52.966+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.967925+0000) 2020-02-06T02:09:52.969 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:52.967+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.968206+0000) 2020-02-06T02:09:52.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:52.968+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.968463+0000) 2020-02-06T02:09:52.975 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:52.973+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:52.974931+0000) 2020-02-06T02:09:52.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:52.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:53.509 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:53.508+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:53.508612+0000) 2020-02-06T02:09:56.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:56.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:56.171376+0000) 2020-02-06T02:09:56.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:56.534+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:56.534562+0000) 2020-02-06T02:09:57.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:57.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.149060+0000) 2020-02-06T02:09:57.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:57.422+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.423248+0000) 2020-02-06T02:09:57.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:57.422+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.423255+0000) 2020-02-06T02:09:57.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:57.423+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.423411+0000) 2020-02-06T02:09:57.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:57.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.653848+0000) 2020-02-06T02:09:57.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:57.676+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.677125+0000) 2020-02-06T02:09:57.993 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:09:57.991+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.992238+0000) 2020-02-06T02:09:57.994 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:09:57.992+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.993706+0000) 2020-02-06T02:09:57.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:09:57.993+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.993599+0000) 2020-02-06T02:09:57.995 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:09:57.992+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.994060+0000) 2020-02-06T02:09:57.996 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:09:57.994+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.994761+0000) 2020-02-06T02:09:57.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:09:57.994+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.994782+0000) 2020-02-06T02:09:57.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:09:57.994+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.995322+0000) 2020-02-06T02:09:57.997 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:09:57.995+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.996353+0000) 2020-02-06T02:09:57.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:09:57.996+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.996703+0000) 2020-02-06T02:09:57.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:57.996+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.997365+0000) 2020-02-06T02:09:57.998 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:09:57.996+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.997868+0000) 2020-02-06T02:09:57.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:57.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.998244+0000) 2020-02-06T02:09:57.999 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:09:57.998+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:57.998773+0000) 2020-02-06T02:09:58.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:58.003+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:58.004491+0000) 2020-02-06T02:09:58.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:09:58.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224664 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:09:58.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:58.352+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:58.353218+0000) 2020-02-06T02:09:59.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:09:59.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:09:59.793351+0000) 2020-02-06T02:10:00.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:00.038+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:00.039066+0000) 2020-02-06T02:10:00.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:00.649+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:00.649948+0000) 2020-02-06T02:10:00.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:00.926+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:00.927618+0000) 2020-02-06T02:10:03.023 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:03.021+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.022366+0000) 2020-02-06T02:10:03.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:03.021+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.022882+0000) 2020-02-06T02:10:03.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:03.022+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.023196+0000) 2020-02-06T02:10:03.025 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:03.023+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.023558+0000) 2020-02-06T02:10:03.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:03.023+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.024271+0000) 2020-02-06T02:10:03.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:03.024+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.024671+0000) 2020-02-06T02:10:03.027 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:03.024+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.025469+0000) 2020-02-06T02:10:03.027 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:03.025+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.025409+0000) 2020-02-06T02:10:03.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:03.025+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.025820+0000) 2020-02-06T02:10:03.028 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:03.026+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.027373+0000) 2020-02-06T02:10:03.028 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:03.027+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.027998+0000) 2020-02-06T02:10:03.029 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:03.027+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.028155+0000) 2020-02-06T02:10:03.029 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:03.027+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.028192+0000) 2020-02-06T02:10:03.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:03.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.032243+0000) 2020-02-06T02:10:03.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:03.045+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224705 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:03.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:03.508+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:03.508863+0000) 2020-02-06T02:10:06.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:06.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:06.171653+0000) 2020-02-06T02:10:06.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:06.534+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:06.534899+0000) 2020-02-06T02:10:07.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:07.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.149380+0000) 2020-02-06T02:10:07.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:07.423+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.423504+0000) 2020-02-06T02:10:07.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:07.423+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.423527+0000) 2020-02-06T02:10:07.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:07.423+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.423692+0000) 2020-02-06T02:10:07.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:07.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.654052+0000) 2020-02-06T02:10:07.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:07.677+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:07.677353+0000) 2020-02-06T02:10:08.052 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:08.050+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.051572+0000) 2020-02-06T02:10:08.053 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:08.050+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.051995+0000) 2020-02-06T02:10:08.053 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:08.051+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.052608+0000) 2020-02-06T02:10:08.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:08.052+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.053538+0000) 2020-02-06T02:10:08.055 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:08.053+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.054113+0000) 2020-02-06T02:10:08.055 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:08.054+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.054789+0000) 2020-02-06T02:10:08.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:08.055+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.055953+0000) 2020-02-06T02:10:08.057 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:08.056+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.056869+0000) 2020-02-06T02:10:08.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:08.055+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.056407+0000) 2020-02-06T02:10:08.058 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:08.055+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.056789+0000) 2020-02-06T02:10:08.058 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:08.056+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.057244+0000) 2020-02-06T02:10:08.059 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:08.057+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.057896+0000) 2020-02-06T02:10:08.059 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:08.056+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.057888+0000) 2020-02-06T02:10:08.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:08.061+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.062315+0000) 2020-02-06T02:10:08.075 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:08.074+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224743 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:08.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:08.361+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:08.361243+0000) 2020-02-06T02:10:09.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:09.793+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:09.793584+0000) 2020-02-06T02:10:10.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:10.039+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:10.039411+0000) 2020-02-06T02:10:10.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:10.650+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:10.650203+0000) 2020-02-06T02:10:10.928 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:10.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:10.927873+0000) 2020-02-06T02:10:13.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:13.080+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.080383+0000) 2020-02-06T02:10:13.081 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:13.080+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.080952+0000) 2020-02-06T02:10:13.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:13.080+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.081328+0000) 2020-02-06T02:10:13.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:13.081+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.081781+0000) 2020-02-06T02:10:13.084 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:13.083+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.083526+0000) 2020-02-06T02:10:13.085 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:13.084+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.084419+0000) 2020-02-06T02:10:13.085 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:13.084+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.084831+0000) 2020-02-06T02:10:13.086 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:13.085+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.085294+0000) 2020-02-06T02:10:13.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:13.085+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.085554+0000) 2020-02-06T02:10:13.086 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:13.085+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.085661+0000) 2020-02-06T02:10:13.087 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:13.085+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.086275+0000) 2020-02-06T02:10:13.087 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:13.085+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.086067+0000) 2020-02-06T02:10:13.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:13.086+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.086784+0000) 2020-02-06T02:10:13.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:13.088+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.088387+0000) 2020-02-06T02:10:13.104 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:13.103+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224783 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:13.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:13.509+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:13.509154+0000) 2020-02-06T02:10:13.797 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:10:13.803 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:10:16.172 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:16.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:16.171902+0000) 2020-02-06T02:10:16.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:16.535+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:16.535152+0000) 2020-02-06T02:10:17.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:17.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.149626+0000) 2020-02-06T02:10:17.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:17.422+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.423727+0000) 2020-02-06T02:10:17.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:17.422+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.423731+0000) 2020-02-06T02:10:17.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:17.422+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.423934+0000) 2020-02-06T02:10:17.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:17.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.654334+0000) 2020-02-06T02:10:17.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:17.676+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:17.677581+0000) 2020-02-06T02:10:18.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:18.108+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.109526+0000) 2020-02-06T02:10:18.111 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:18.108+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.110028+0000) 2020-02-06T02:10:18.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:18.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.110545+0000) 2020-02-06T02:10:18.112 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:18.110+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.111337+0000) 2020-02-06T02:10:18.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:18.109+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.110983+0000) 2020-02-06T02:10:18.113 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:18.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.112405+0000) 2020-02-06T02:10:18.113 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:18.111+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.112679+0000) 2020-02-06T02:10:18.113 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:18.111+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.113038+0000) 2020-02-06T02:10:18.115 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:18.112+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.114287+0000) 2020-02-06T02:10:18.115 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:18.113+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.114689+0000) 2020-02-06T02:10:18.116 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:18.113+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.115125+0000) 2020-02-06T02:10:18.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:18.113+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.115245+0000) 2020-02-06T02:10:18.116 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:18.114+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.115606+0000) 2020-02-06T02:10:18.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:18.116+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.117902+0000) 2020-02-06T02:10:18.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:18.131+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:18.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:18.367+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:18.368556+0000) 2020-02-06T02:10:19.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:19.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:19.793828+0000) 2020-02-06T02:10:20.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:20.038+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:20.039640+0000) 2020-02-06T02:10:20.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:20.649+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:20.650497+0000) 2020-02-06T02:10:20.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:20.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:20.928144+0000) 2020-02-06T02:10:23.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:23.139+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.140476+0000) 2020-02-06T02:10:23.141 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:23.139+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.140778+0000) 2020-02-06T02:10:23.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:23.139+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.141118+0000) 2020-02-06T02:10:23.142 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:23.140+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.141548+0000) 2020-02-06T02:10:23.143 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:23.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.142225+0000) 2020-02-06T02:10:23.143 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:23.141+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.142513+0000) 2020-02-06T02:10:23.144 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:23.141+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.142654+0000) 2020-02-06T02:10:23.144 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:23.142+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.143479+0000) 2020-02-06T02:10:23.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:23.142+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.144005+0000) 2020-02-06T02:10:23.145 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:23.143+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.144490+0000) 2020-02-06T02:10:23.145 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:23.142+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.144227+0000) 2020-02-06T02:10:23.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:23.143+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.144740+0000) 2020-02-06T02:10:23.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:23.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.145656+0000) 2020-02-06T02:10:23.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:23.147+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.148614+0000) 2020-02-06T02:10:23.163 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:23.161+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224863 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:23.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:23.508+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:23.509404+0000) 2020-02-06T02:10:26.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:26.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:26.172138+0000) 2020-02-06T02:10:26.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:26.534+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:26.535418+0000) 2020-02-06T02:10:27.150 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:27.148+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.149846+0000) 2020-02-06T02:10:27.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:27.423+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.423990+0000) 2020-02-06T02:10:27.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:27.423+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.423994+0000) 2020-02-06T02:10:27.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:27.423+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.424154+0000) 2020-02-06T02:10:27.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:27.653+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.654606+0000) 2020-02-06T02:10:27.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:27.676+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:27.677808+0000) 2020-02-06T02:10:28.169 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:28.167+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.168645+0000) 2020-02-06T02:10:28.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:28.168+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.169187+0000) 2020-02-06T02:10:28.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:28.168+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.169529+0000) 2020-02-06T02:10:28.170 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:28.168+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.169939+0000) 2020-02-06T02:10:28.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:28.169+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.170323+0000) 2020-02-06T02:10:28.171 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:28.169+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.170806+0000) 2020-02-06T02:10:28.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:28.170+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.171798+0000) 2020-02-06T02:10:28.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:28.170+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.172044+0000) 2020-02-06T02:10:28.173 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:28.171+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.172973+0000) 2020-02-06T02:10:28.174 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:28.172+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.173706+0000) 2020-02-06T02:10:28.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:28.172+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.174154+0000) 2020-02-06T02:10:28.175 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:28.173+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.174629+0000) 2020-02-06T02:10:28.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:28.173+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.174941+0000) 2020-02-06T02:10:28.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:28.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.179038+0000) 2020-02-06T02:10:28.193 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:28.191+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224904 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:28.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:28.375+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:28.376718+0000) 2020-02-06T02:10:29.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:29.792+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:29.794100+0000) 2020-02-06T02:10:30.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:30.039+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:30.039875+0000) 2020-02-06T02:10:30.651 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:30.649+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:30.650822+0000) 2020-02-06T02:10:30.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:30.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:30.928372+0000) 2020-02-06T02:10:33.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:33.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.198485+0000) 2020-02-06T02:10:33.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:33.197+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.198656+0000) 2020-02-06T02:10:33.200 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:33.198+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.199257+0000) 2020-02-06T02:10:33.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:33.199+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.200756+0000) 2020-02-06T02:10:33.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:33.200+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.201301+0000) 2020-02-06T02:10:33.202 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:33.200+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.201543+0000) 2020-02-06T02:10:33.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:33.200+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.202299+0000) 2020-02-06T02:10:33.203 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:33.201+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.202774+0000) 2020-02-06T02:10:33.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:33.202+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.203903+0000) 2020-02-06T02:10:33.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:33.202+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.203636+0000) 2020-02-06T02:10:33.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:33.203+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.204611+0000) 2020-02-06T02:10:33.206 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:33.203+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.204566+0000) 2020-02-06T02:10:33.206 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:33.203+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.204639+0000) 2020-02-06T02:10:33.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:33.206+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.207785+0000) 2020-02-06T02:10:33.223 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:33.220+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224944 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:33.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:33.508+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:33.509609+0000) 2020-02-06T02:10:36.173 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:36.170+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:36.172376+0000) 2020-02-06T02:10:36.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:36.534+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:36.535702+0000) 2020-02-06T02:10:37.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:37.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.150124+0000) 2020-02-06T02:10:37.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:37.423+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.424267+0000) 2020-02-06T02:10:37.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:37.423+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.424272+0000) 2020-02-06T02:10:37.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:37.423+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.424391+0000) 2020-02-06T02:10:37.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:37.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.654877+0000) 2020-02-06T02:10:37.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:37.677+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:37.678047+0000) 2020-02-06T02:10:38.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:38.227+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.228445+0000) 2020-02-06T02:10:38.230 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:38.227+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.229088+0000) 2020-02-06T02:10:38.230 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:38.228+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.229103+0000) 2020-02-06T02:10:38.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:38.228+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.230092+0000) 2020-02-06T02:10:38.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:38.229+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.230718+0000) 2020-02-06T02:10:38.232 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:38.229+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.231191+0000) 2020-02-06T02:10:38.232 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:38.230+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.231517+0000) 2020-02-06T02:10:38.233 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:38.231+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.232343+0000) 2020-02-06T02:10:38.233 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:38.231+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.232792+0000) 2020-02-06T02:10:38.234 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:38.232+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.233384+0000) 2020-02-06T02:10:38.234 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:38.231+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.233117+0000) 2020-02-06T02:10:38.235 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:38.232+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.233792+0000) 2020-02-06T02:10:38.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:38.237+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.238752+0000) 2020-02-06T02:10:38.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:38.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.239762+0000) 2020-02-06T02:10:38.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:38.251+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 224986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:38.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:38.384+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:38.385317+0000) 2020-02-06T02:10:39.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:39.793+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:39.794336+0000) 2020-02-06T02:10:40.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:40.039+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:40.040133+0000) 2020-02-06T02:10:40.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:40.650+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:40.651044+0000) 2020-02-06T02:10:40.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:40.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:40.928622+0000) 2020-02-06T02:10:43.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:43.257+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.258381+0000) 2020-02-06T02:10:43.259 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:43.257+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.258704+0000) 2020-02-06T02:10:43.260 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:43.257+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.259272+0000) 2020-02-06T02:10:43.260 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:43.257+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.259009+0000) 2020-02-06T02:10:43.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:43.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.259678+0000) 2020-02-06T02:10:43.261 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:43.258+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.260156+0000) 2020-02-06T02:10:43.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:43.259+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.260953+0000) 2020-02-06T02:10:43.262 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:43.260+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.261364+0000) 2020-02-06T02:10:43.262 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:43.259+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.260967+0000) 2020-02-06T02:10:43.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:43.261+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.262471+0000) 2020-02-06T02:10:43.264 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:43.261+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.262930+0000) 2020-02-06T02:10:43.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:43.260+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.261584+0000) 2020-02-06T02:10:43.264 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:43.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.262140+0000) 2020-02-06T02:10:43.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:43.268+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.269256+0000) 2020-02-06T02:10:43.285 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:43.283+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225027 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:43.510 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:43.509+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:43.509850+0000) 2020-02-06T02:10:43.932 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:10:43.938 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:10:46.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:46.171+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:46.172725+0000) 2020-02-06T02:10:46.536 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:46.535+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:46.535962+0000) 2020-02-06T02:10:47.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:47.149+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.150356+0000) 2020-02-06T02:10:47.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:47.423+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.424516+0000) 2020-02-06T02:10:47.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:47.423+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.424598+0000) 2020-02-06T02:10:47.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:47.423+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.424757+0000) 2020-02-06T02:10:47.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:47.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.655145+0000) 2020-02-06T02:10:47.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:47.677+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:47.678304+0000) 2020-02-06T02:10:48.291 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:48.289+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.290409+0000) 2020-02-06T02:10:48.292 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:48.289+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.291285+0000) 2020-02-06T02:10:48.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:48.291+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.292288+0000) 2020-02-06T02:10:48.293 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:48.291+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.292588+0000) 2020-02-06T02:10:48.294 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:48.292+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.293301+0000) 2020-02-06T02:10:48.294 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:48.292+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.294021+0000) 2020-02-06T02:10:48.295 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:48.292+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.293896+0000) 2020-02-06T02:10:48.295 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:48.293+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.294519+0000) 2020-02-06T02:10:48.296 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:48.293+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.295081+0000) 2020-02-06T02:10:48.296 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:48.294+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.295527+0000) 2020-02-06T02:10:48.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:48.294+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.296169+0000) 2020-02-06T02:10:48.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:48.295+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.296568+0000) 2020-02-06T02:10:48.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:48.295+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.297109+0000) 2020-02-06T02:10:48.301 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:48.299+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.300270+0000) 2020-02-06T02:10:48.314 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:48.312+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:48.393 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:48.392+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:48.392674+0000) 2020-02-06T02:10:49.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:49.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:49.794574+0000) 2020-02-06T02:10:50.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:50.039+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:50.040392+0000) 2020-02-06T02:10:50.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:50.650+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:50.651291+0000) 2020-02-06T02:10:50.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:50.927+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:50.928872+0000) 2020-02-06T02:10:53.320 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:53.318+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.319806+0000) 2020-02-06T02:10:53.321 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:53.319+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.320339+0000) 2020-02-06T02:10:53.321 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:53.319+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.320689+0000) 2020-02-06T02:10:53.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:53.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.321165+0000) 2020-02-06T02:10:53.323 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:53.320+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.322220+0000) 2020-02-06T02:10:53.323 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:53.321+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.322659+0000) 2020-02-06T02:10:53.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:53.321+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.322573+0000) 2020-02-06T02:10:53.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:53.321+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.323210+0000) 2020-02-06T02:10:53.325 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:53.323+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.324889+0000) 2020-02-06T02:10:53.326 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:53.324+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.325320+0000) 2020-02-06T02:10:53.326 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:53.324+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.325462+0000) 2020-02-06T02:10:53.326 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:53.323+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.325144+0000) 2020-02-06T02:10:53.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:53.324+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.325886+0000) 2020-02-06T02:10:53.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:53.326+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.327238+0000) 2020-02-06T02:10:53.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:53.342+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225107 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:53.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:53.508+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:53.510022+0000) 2020-02-06T02:10:56.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:56.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:56.172976+0000) 2020-02-06T02:10:56.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:56.535+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:56.536220+0000) 2020-02-06T02:10:57.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:57.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.150595+0000) 2020-02-06T02:10:57.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:57.424+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.424832+0000) 2020-02-06T02:10:57.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:57.424+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.424833+0000) 2020-02-06T02:10:57.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:57.424+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.424997+0000) 2020-02-06T02:10:57.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:57.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.655371+0000) 2020-02-06T02:10:57.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:57.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:57.678579+0000) 2020-02-06T02:10:58.350 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:10:58.348+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.349472+0000) 2020-02-06T02:10:58.351 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:10:58.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.349924+0000) 2020-02-06T02:10:58.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:58.349+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.350559+0000) 2020-02-06T02:10:58.351 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:10:58.349+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.350772+0000) 2020-02-06T02:10:58.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:58.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.351515+0000) 2020-02-06T02:10:58.354 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:10:58.353+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.354063+0000) 2020-02-06T02:10:58.355 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:10:58.353+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.354488+0000) 2020-02-06T02:10:58.356 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:10:58.353+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.355068+0000) 2020-02-06T02:10:58.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:10:58.354+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.355834+0000) 2020-02-06T02:10:58.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:10:58.354+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.355981+0000) 2020-02-06T02:10:58.357 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:10:58.354+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.355833+0000) 2020-02-06T02:10:58.357 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:10:58.355+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.356496+0000) 2020-02-06T02:10:58.358 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:10:58.355+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.356564+0000) 2020-02-06T02:10:58.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:58.360+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.361463+0000) 2020-02-06T02:10:58.375 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:10:58.373+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:10:58.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:58.401+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:58.401448+0000) 2020-02-06T02:10:59.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:10:59.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:10:59.794809+0000) 2020-02-06T02:11:00.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:00.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:00.040625+0000) 2020-02-06T02:11:00.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:00.650+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:00.651546+0000) 2020-02-06T02:11:00.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:00.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:00.929136+0000) 2020-02-06T02:11:03.382 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:03.380+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.381520+0000) 2020-02-06T02:11:03.383 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:03.381+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.382730+0000) 2020-02-06T02:11:03.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:03.382+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.383778+0000) 2020-02-06T02:11:03.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:03.382+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.384359+0000) 2020-02-06T02:11:03.385 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:03.383+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.384820+0000) 2020-02-06T02:11:03.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:03.384+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.385502+0000) 2020-02-06T02:11:03.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:03.384+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.385883+0000) 2020-02-06T02:11:03.387 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:03.385+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.386242+0000) 2020-02-06T02:11:03.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:03.385+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.386269+0000) 2020-02-06T02:11:03.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:03.385+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.386371+0000) 2020-02-06T02:11:03.388 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:03.385+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.387210+0000) 2020-02-06T02:11:03.388 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:03.386+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.387288+0000) 2020-02-06T02:11:03.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:03.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.389288+0000) 2020-02-06T02:11:03.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:03.388+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.389479+0000) 2020-02-06T02:11:03.405 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:03.402+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225189 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:03.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:03.510+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:03.510271+0000) 2020-02-06T02:11:06.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:06.172+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:06.173227+0000) 2020-02-06T02:11:06.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:06.535+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:06.536489+0000) 2020-02-06T02:11:07.151 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:07.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.150818+0000) 2020-02-06T02:11:07.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:07.424+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.425069+0000) 2020-02-06T02:11:07.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:07.424+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.425147+0000) 2020-02-06T02:11:07.426 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:07.424+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.425274+0000) 2020-02-06T02:11:07.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:07.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.655632+0000) 2020-02-06T02:11:07.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:07.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:07.678804+0000) 2020-02-06T02:11:08.409 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:08.408+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.408792+0000) 2020-02-06T02:11:08.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:08.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.410281+0000) 2020-02-06T02:11:08.411 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:08.410+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.410833+0000) 2020-02-06T02:11:08.412 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:08.411+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.411757+0000) 2020-02-06T02:11:08.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:08.412+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.412297+0000) 2020-02-06T02:11:08.413 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:08.412+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.412919+0000) 2020-02-06T02:11:08.414 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:08.413+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.413164+0000) 2020-02-06T02:11:08.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:08.413+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.413588+0000) 2020-02-06T02:11:08.414 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:08.413+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.414191+0000) 2020-02-06T02:11:08.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:08.414+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.414245+0000) 2020-02-06T02:11:08.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:08.413+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.413927+0000) 2020-02-06T02:11:08.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:08.414+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.414619+0000) 2020-02-06T02:11:08.416 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:08.415+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.415214+0000) 2020-02-06T02:11:08.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:08.419+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:08.419736+0000) 2020-02-06T02:11:08.433 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:08.432+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225233 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:09.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:09.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:09.795042+0000) 2020-02-06T02:11:10.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:10.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:10.040851+0000) 2020-02-06T02:11:10.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:10.651+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:10.651798+0000) 2020-02-06T02:11:10.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:10.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:10.929324+0000) 2020-02-06T02:11:13.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:13.439+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.439674+0000) 2020-02-06T02:11:13.441 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:13.439+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.440520+0000) 2020-02-06T02:11:13.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:13.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.440074+0000) 2020-02-06T02:11:13.442 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:13.439+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.440190+0000) 2020-02-06T02:11:13.442 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:13.440+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.441304+0000) 2020-02-06T02:11:13.443 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:13.440+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.441311+0000) 2020-02-06T02:11:13.443 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:13.440+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.441649+0000) 2020-02-06T02:11:13.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:13.441+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.441675+0000) 2020-02-06T02:11:13.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:13.441+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.442294+0000) 2020-02-06T02:11:13.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:13.441+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.442607+0000) 2020-02-06T02:11:13.444 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:13.441+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.442709+0000) 2020-02-06T02:11:13.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:13.441+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.441910+0000) 2020-02-06T02:11:13.445 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:13.443+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.443420+0000) 2020-02-06T02:11:13.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:13.446+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.447812+0000) 2020-02-06T02:11:13.462 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:13.460+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225272 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:13.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:13.509+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:13.510433+0000) 2020-02-06T02:11:14.072 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:11:14.078 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:11:15.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:15.158+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:15.159543+0000) 2020-02-06T02:11:16.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:16.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:16.173476+0000) 2020-02-06T02:11:16.537 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:16.536+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:16.536757+0000) 2020-02-06T02:11:17.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:17.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.151011+0000) 2020-02-06T02:11:17.426 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:17.424+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.425330+0000) 2020-02-06T02:11:17.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:17.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.425465+0000) 2020-02-06T02:11:17.427 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:17.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.425531+0000) 2020-02-06T02:11:17.656 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:17.654+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.655887+0000) 2020-02-06T02:11:17.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:17.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:17.679051+0000) 2020-02-06T02:11:18.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:18.467+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.467573+0000) 2020-02-06T02:11:18.469 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:18.467+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.468132+0000) 2020-02-06T02:11:18.469 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:18.467+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.468559+0000) 2020-02-06T02:11:18.470 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:18.469+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.469481+0000) 2020-02-06T02:11:18.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:18.469+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.470228+0000) 2020-02-06T02:11:18.471 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:18.470+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.470530+0000) 2020-02-06T02:11:18.472 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:18.470+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.471397+0000) 2020-02-06T02:11:18.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:18.472+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.472748+0000) 2020-02-06T02:11:18.474 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:18.472+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.472617+0000) 2020-02-06T02:11:18.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:18.471+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.472364+0000) 2020-02-06T02:11:18.474 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:18.471+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.473031+0000) 2020-02-06T02:11:18.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:18.472+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.473587+0000) 2020-02-06T02:11:18.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:18.475+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.477026+0000) 2020-02-06T02:11:18.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:18.476+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:18.477277+0000) 2020-02-06T02:11:18.492 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:18.490+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225316 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:19.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:19.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:19.795306+0000) 2020-02-06T02:11:20.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:20.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:20.041065+0000) 2020-02-06T02:11:20.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:20.651+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:20.652062+0000) 2020-02-06T02:11:20.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:20.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:20.929536+0000) 2020-02-06T02:11:23.497 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:23.495+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.496743+0000) 2020-02-06T02:11:23.498 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:23.495+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.496188+0000) 2020-02-06T02:11:23.498 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:23.496+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.497372+0000) 2020-02-06T02:11:23.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:23.496+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.497747+0000) 2020-02-06T02:11:23.500 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:23.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.499137+0000) 2020-02-06T02:11:23.500 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:23.499+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.499511+0000) 2020-02-06T02:11:23.501 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:23.499+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.500182+0000) 2020-02-06T02:11:23.501 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:23.499+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.500194+0000) 2020-02-06T02:11:23.501 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:23.500+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.500833+0000) 2020-02-06T02:11:23.502 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:23.500+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.501274+0000) 2020-02-06T02:11:23.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:23.501+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.502646+0000) 2020-02-06T02:11:23.503 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:23.502+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.502497+0000) 2020-02-06T02:11:23.503 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:23.502+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.503326+0000) 2020-02-06T02:11:23.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:23.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.507458+0000) 2020-02-06T02:11:23.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:23.509+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:23.510651+0000) 2020-02-06T02:11:23.521 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:23.520+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:25.169 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:25.167+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:25.168127+0000) 2020-02-06T02:11:26.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:26.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:26.173710+0000) 2020-02-06T02:11:26.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:26.536+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:26.537016+0000) 2020-02-06T02:11:27.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:27.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.151294+0000) 2020-02-06T02:11:27.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:27.425+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.425634+0000) 2020-02-06T02:11:27.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:27.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.425751+0000) 2020-02-06T02:11:27.427 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:27.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.425823+0000) 2020-02-06T02:11:27.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:27.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.656145+0000) 2020-02-06T02:11:27.680 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:27.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:27.679305+0000) 2020-02-06T02:11:28.529 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:28.527+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.527791+0000) 2020-02-06T02:11:28.530 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:28.527+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.528211+0000) 2020-02-06T02:11:28.530 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:28.525+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.526479+0000) 2020-02-06T02:11:28.530 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:28.526+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.527254+0000) 2020-02-06T02:11:28.530 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:28.527+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.528660+0000) 2020-02-06T02:11:28.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:28.529+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.529644+0000) 2020-02-06T02:11:28.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:28.528+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.529483+0000) 2020-02-06T02:11:28.532 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:28.528+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.530045+0000) 2020-02-06T02:11:28.532 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:28.529+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.530365+0000) 2020-02-06T02:11:28.532 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:28.529+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.530713+0000) 2020-02-06T02:11:28.532 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:28.530+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.531250+0000) 2020-02-06T02:11:28.533 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:28.531+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.531413+0000) 2020-02-06T02:11:28.533 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:28.532+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.532593+0000) 2020-02-06T02:11:28.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:28.536+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:28.537433+0000) 2020-02-06T02:11:28.552 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:28.550+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225399 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:29.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:29.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:29.795539+0000) 2020-02-06T02:11:30.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:30.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:30.041366+0000) 2020-02-06T02:11:30.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:30.652+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:30.652296+0000) 2020-02-06T02:11:30.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:30.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:30.929788+0000) 2020-02-06T02:11:33.511 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:33.509+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.510849+0000) 2020-02-06T02:11:33.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:33.554+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.555956+0000) 2020-02-06T02:11:33.559 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:33.558+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.558544+0000) 2020-02-06T02:11:33.559 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:33.557+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.559055+0000) 2020-02-06T02:11:33.560 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:33.557+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.558397+0000) 2020-02-06T02:11:33.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:33.558+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.559564+0000) 2020-02-06T02:11:33.561 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:33.558+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.560021+0000) 2020-02-06T02:11:33.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:33.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.560352+0000) 2020-02-06T02:11:33.561 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:33.560+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.560802+0000) 2020-02-06T02:11:33.562 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:33.560+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.561312+0000) 2020-02-06T02:11:33.562 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:33.560+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.561127+0000) 2020-02-06T02:11:33.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:33.561+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.562018+0000) 2020-02-06T02:11:33.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:33.561+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.562344+0000) 2020-02-06T02:11:33.563 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:33.562+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.562727+0000) 2020-02-06T02:11:33.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:33.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:33.568499+0000) 2020-02-06T02:11:33.583 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:33.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:35.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:35.175+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:35.176147+0000) 2020-02-06T02:11:36.174 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:36.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:36.173914+0000) 2020-02-06T02:11:36.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:36.537+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:36.537261+0000) 2020-02-06T02:11:37.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:37.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.151527+0000) 2020-02-06T02:11:37.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:37.425+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.425892+0000) 2020-02-06T02:11:37.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:37.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.425989+0000) 2020-02-06T02:11:37.427 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:37.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.426123+0000) 2020-02-06T02:11:37.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:37.655+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.656380+0000) 2020-02-06T02:11:37.680 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:37.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:37.679531+0000) 2020-02-06T02:11:38.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:38.586+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.588010+0000) 2020-02-06T02:11:38.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:38.587+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.588396+0000) 2020-02-06T02:11:38.590 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:38.588+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.589440+0000) 2020-02-06T02:11:38.591 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:38.589+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.589986+0000) 2020-02-06T02:11:38.591 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:38.590+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.590459+0000) 2020-02-06T02:11:38.591 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:38.590+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.591056+0000) 2020-02-06T02:11:38.592 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:38.590+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.591171+0000) 2020-02-06T02:11:38.592 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:38.590+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.591388+0000) 2020-02-06T02:11:38.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:38.591+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.591880+0000) 2020-02-06T02:11:38.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:38.591+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.592364+0000) 2020-02-06T02:11:38.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:38.591+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.592838+0000) 2020-02-06T02:11:38.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:38.591+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.592711+0000) 2020-02-06T02:11:38.594 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:38.592+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.593107+0000) 2020-02-06T02:11:38.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:38.598+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:38.599241+0000) 2020-02-06T02:11:38.613 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:38.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:39.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:39.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:39.795764+0000) 2020-02-06T02:11:40.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:40.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:40.041618+0000) 2020-02-06T02:11:40.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:40.652+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:40.652549+0000) 2020-02-06T02:11:40.930 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:40.928+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:40.929973+0000) 2020-02-06T02:11:43.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:43.510+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.511119+0000) 2020-02-06T02:11:43.618 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:43.616+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.617577+0000) 2020-02-06T02:11:43.620 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:43.618+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.619034+0000) 2020-02-06T02:11:43.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:43.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.620576+0000) 2020-02-06T02:11:43.621 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:43.620+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.621219+0000) 2020-02-06T02:11:43.622 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:43.620+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.621195+0000) 2020-02-06T02:11:43.622 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:43.620+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.621660+0000) 2020-02-06T02:11:43.623 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:43.621+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.622483+0000) 2020-02-06T02:11:43.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:43.622+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.622549+0000) 2020-02-06T02:11:43.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:43.621+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.622857+0000) 2020-02-06T02:11:43.624 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:43.623+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.623690+0000) 2020-02-06T02:11:43.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:43.622+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.623495+0000) 2020-02-06T02:11:43.625 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:43.623+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.623943+0000) 2020-02-06T02:11:43.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:43.624+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.624961+0000) 2020-02-06T02:11:43.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:43.628+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:43.629893+0000) 2020-02-06T02:11:43.644 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:43.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:44.225 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:11:44.232 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:11:45.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:45.183+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:45.184026+0000) 2020-02-06T02:11:46.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:46.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:46.174198+0000) 2020-02-06T02:11:46.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:46.536+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:46.537525+0000) 2020-02-06T02:11:47.152 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:47.150+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.151785+0000) 2020-02-06T02:11:47.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:47.425+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.426126+0000) 2020-02-06T02:11:47.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:47.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.426219+0000) 2020-02-06T02:11:47.428 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:47.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.426394+0000) 2020-02-06T02:11:47.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:47.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.656600+0000) 2020-02-06T02:11:47.680 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:47.678+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:47.679819+0000) 2020-02-06T02:11:48.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:48.649+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.650541+0000) 2020-02-06T02:11:48.652 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:48.649+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.651088+0000) 2020-02-06T02:11:48.653 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:48.650+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.651539+0000) 2020-02-06T02:11:48.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:48.650+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.651907+0000) 2020-02-06T02:11:48.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:48.651+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.652967+0000) 2020-02-06T02:11:48.654 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:48.652+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.653369+0000) 2020-02-06T02:11:48.654 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:48.652+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.653640+0000) 2020-02-06T02:11:48.654 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:48.652+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.653923+0000) 2020-02-06T02:11:48.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:48.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.654509+0000) 2020-02-06T02:11:48.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:48.653+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.654818+0000) 2020-02-06T02:11:48.656 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:48.653+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.654888+0000) 2020-02-06T02:11:48.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:48.654+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.655634+0000) 2020-02-06T02:11:48.657 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:48.654+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.655761+0000) 2020-02-06T02:11:48.661 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:48.659+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:48.660412+0000) 2020-02-06T02:11:48.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:48.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225565 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:49.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:49.794+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:49.796036+0000) 2020-02-06T02:11:50.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:50.040+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:50.041863+0000) 2020-02-06T02:11:50.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:50.651+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:50.652802+0000) 2020-02-06T02:11:50.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:50.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:50.930192+0000) 2020-02-06T02:11:53.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:53.510+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.511307+0000) 2020-02-06T02:11:53.681 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:53.679+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.680755+0000) 2020-02-06T02:11:53.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:53.681+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.682393+0000) 2020-02-06T02:11:53.683 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:53.680+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.682121+0000) 2020-02-06T02:11:53.685 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:53.682+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.684120+0000) 2020-02-06T02:11:53.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:53.683+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.684633+0000) 2020-02-06T02:11:53.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:53.684+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.685069+0000) 2020-02-06T02:11:53.686 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:53.684+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.685519+0000) 2020-02-06T02:11:53.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:53.685+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.686103+0000) 2020-02-06T02:11:53.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:53.685+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.686330+0000) 2020-02-06T02:11:53.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:53.685+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.686640+0000) 2020-02-06T02:11:53.689 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:53.685+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.686882+0000) 2020-02-06T02:11:53.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:53.685+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.687323+0000) 2020-02-06T02:11:53.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:53.685+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.686722+0000) 2020-02-06T02:11:53.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:53.693+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:53.694447+0000) 2020-02-06T02:11:53.706 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:53.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225604 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:55.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:55.190+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:55.192006+0000) 2020-02-06T02:11:56.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:56.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:56.174488+0000) 2020-02-06T02:11:56.538 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:56.536+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:56.537785+0000) 2020-02-06T02:11:57.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:57.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.152021+0000) 2020-02-06T02:11:57.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:57.425+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.426363+0000) 2020-02-06T02:11:57.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:57.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.426521+0000) 2020-02-06T02:11:57.428 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:57.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.426688+0000) 2020-02-06T02:11:57.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:57.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.656829+0000) 2020-02-06T02:11:57.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:57.679+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:57.680051+0000) 2020-02-06T02:11:58.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:11:58.709+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.710433+0000) 2020-02-06T02:11:58.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:11:58.710+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.711057+0000) 2020-02-06T02:11:58.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:58.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.711826+0000) 2020-02-06T02:11:58.713 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:11:58.711+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.712541+0000) 2020-02-06T02:11:58.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:58.712+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.713458+0000) 2020-02-06T02:11:58.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:11:58.712+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.713564+0000) 2020-02-06T02:11:58.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:11:58.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.714114+0000) 2020-02-06T02:11:58.715 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:11:58.712+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.714074+0000) 2020-02-06T02:11:58.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:11:58.713+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.715213+0000) 2020-02-06T02:11:58.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:11:58.714+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.715567+0000) 2020-02-06T02:11:58.717 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:11:58.714+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.716161+0000) 2020-02-06T02:11:58.717 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:11:58.715+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.716123+0000) 2020-02-06T02:11:58.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:11:58.715+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.716989+0000) 2020-02-06T02:11:58.723 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:58.721+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:58.722417+0000) 2020-02-06T02:11:58.737 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:11:58.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225649 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:11:59.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:11:59.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:11:59.796311+0000) 2020-02-06T02:12:00.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:00.041+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:00.042143+0000) 2020-02-06T02:12:00.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:00.652+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:00.653056+0000) 2020-02-06T02:12:00.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:00.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:00.930432+0000) 2020-02-06T02:12:03.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:03.511+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.511535+0000) 2020-02-06T02:12:03.744 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:03.741+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.742999+0000) 2020-02-06T02:12:03.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:03.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.743348+0000) 2020-02-06T02:12:03.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:03.743+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.744004+0000) 2020-02-06T02:12:03.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:03.744+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.744227+0000) 2020-02-06T02:12:03.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:03.744+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.744811+0000) 2020-02-06T02:12:03.746 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:03.745+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.745098+0000) 2020-02-06T02:12:03.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:03.745+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.745380+0000) 2020-02-06T02:12:03.747 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:03.745+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.745691+0000) 2020-02-06T02:12:03.748 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:03.744+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.746219+0000) 2020-02-06T02:12:03.748 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:03.745+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.746577+0000) 2020-02-06T02:12:03.748 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:03.744+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.745977+0000) 2020-02-06T02:12:03.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:03.745+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.747050+0000) 2020-02-06T02:12:03.748 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:03.745+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.747193+0000) 2020-02-06T02:12:03.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:03.752+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:03.752577+0000) 2020-02-06T02:12:03.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:03.764+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225687 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:05.201 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:05.200+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:05.200658+0000) 2020-02-06T02:12:06.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:06.173+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:06.174722+0000) 2020-02-06T02:12:06.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:06.537+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:06.538036+0000) 2020-02-06T02:12:07.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:07.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.152282+0000) 2020-02-06T02:12:07.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:07.425+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.426584+0000) 2020-02-06T02:12:07.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:07.425+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.426795+0000) 2020-02-06T02:12:07.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:07.425+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.426952+0000) 2020-02-06T02:12:07.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:07.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.657111+0000) 2020-02-06T02:12:07.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:07.679+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:07.680316+0000) 2020-02-06T02:12:08.772 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:08.769+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.771006+0000) 2020-02-06T02:12:08.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:08.771+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.772249+0000) 2020-02-06T02:12:08.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:08.771+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.772590+0000) 2020-02-06T02:12:08.774 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:08.771+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.773379+0000) 2020-02-06T02:12:08.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:08.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.773710+0000) 2020-02-06T02:12:08.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:08.773+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.774769+0000) 2020-02-06T02:12:08.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:08.773+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.775003+0000) 2020-02-06T02:12:08.776 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:08.773+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.774518+0000) 2020-02-06T02:12:08.776 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:08.774+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.775754+0000) 2020-02-06T02:12:08.776 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:08.773+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.775172+0000) 2020-02-06T02:12:08.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:08.775+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.776464+0000) 2020-02-06T02:12:08.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:08.775+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.776893+0000) 2020-02-06T02:12:08.778 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:08.775+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.777256+0000) 2020-02-06T02:12:08.782 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:08.780+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:08.781593+0000) 2020-02-06T02:12:08.796 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:08.794+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225731 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:09.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:09.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:09.796573+0000) 2020-02-06T02:12:10.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:10.041+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:10.042378+0000) 2020-02-06T02:12:10.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:10.652+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:10.653291+0000) 2020-02-06T02:12:10.931 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:10.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:10.930697+0000) 2020-02-06T02:12:13.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:13.510+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.511766+0000) 2020-02-06T02:12:13.802 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:13.800+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.801616+0000) 2020-02-06T02:12:13.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:13.801+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.802546+0000) 2020-02-06T02:12:13.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:13.802+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.803304+0000) 2020-02-06T02:12:13.804 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:13.802+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.803912+0000) 2020-02-06T02:12:13.805 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:13.803+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.804683+0000) 2020-02-06T02:12:13.806 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:13.803+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.804903+0000) 2020-02-06T02:12:13.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:13.804+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.806011+0000) 2020-02-06T02:12:13.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:13.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.806488+0000) 2020-02-06T02:12:13.808 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:13.805+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.806705+0000) 2020-02-06T02:12:13.808 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:13.805+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.807375+0000) 2020-02-06T02:12:13.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:13.806+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.807702+0000) 2020-02-06T02:12:13.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:13.805+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.807055+0000) 2020-02-06T02:12:13.810 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:13.806+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.808122+0000) 2020-02-06T02:12:13.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:13.811+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:13.812645+0000) 2020-02-06T02:12:13.827 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:13.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225770 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:14.364 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:12:14.371 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:12:15.210 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:15.208+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:15.209221+0000) 2020-02-06T02:12:16.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:16.174+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:16.175028+0000) 2020-02-06T02:12:16.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:16.537+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:16.538275+0000) 2020-02-06T02:12:17.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:17.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.152519+0000) 2020-02-06T02:12:17.427 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:17.426+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.426814+0000) 2020-02-06T02:12:17.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:17.426+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.427036+0000) 2020-02-06T02:12:17.428 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:17.426+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.427294+0000) 2020-02-06T02:12:17.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:17.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.657357+0000) 2020-02-06T02:12:17.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:17.679+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:17.680548+0000) 2020-02-06T02:12:18.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:18.831+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.833201+0000) 2020-02-06T02:12:18.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:18.832+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.833764+0000) 2020-02-06T02:12:18.835 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:18.832+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.834200+0000) 2020-02-06T02:12:18.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:18.833+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.834762+0000) 2020-02-06T02:12:18.836 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:18.834+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.835484+0000) 2020-02-06T02:12:18.836 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:18.833+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.835133+0000) 2020-02-06T02:12:18.837 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:18.834+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.835893+0000) 2020-02-06T02:12:18.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:18.835+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.836045+0000) 2020-02-06T02:12:18.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:18.835+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.836561+0000) 2020-02-06T02:12:18.838 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:18.836+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.837922+0000) 2020-02-06T02:12:18.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:18.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.838304+0000) 2020-02-06T02:12:18.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:18.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.838373+0000) 2020-02-06T02:12:18.839 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:18.837+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.838541+0000) 2020-02-06T02:12:18.843 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:18.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:18.842712+0000) 2020-02-06T02:12:18.857 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:18.855+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225815 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:19.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:19.795+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:19.796790+0000) 2020-02-06T02:12:20.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:20.041+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:20.042609+0000) 2020-02-06T02:12:20.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:20.652+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:20.653531+0000) 2020-02-06T02:12:20.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:20.929+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:20.930945+0000) 2020-02-06T02:12:23.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:23.511+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.511983+0000) 2020-02-06T02:12:23.863 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:23.861+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.862833+0000) 2020-02-06T02:12:23.864 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:23.861+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.863175+0000) 2020-02-06T02:12:23.865 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:23.863+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.864315+0000) 2020-02-06T02:12:23.865 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:23.863+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.865024+0000) 2020-02-06T02:12:23.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:23.864+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.865231+0000) 2020-02-06T02:12:23.866 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:23.864+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.865607+0000) 2020-02-06T02:12:23.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:23.865+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.866611+0000) 2020-02-06T02:12:23.868 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:23.864+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.866304+0000) 2020-02-06T02:12:23.868 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:23.864+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.866352+0000) 2020-02-06T02:12:23.868 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:23.865+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.867079+0000) 2020-02-06T02:12:23.868 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:23.866+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.867613+0000) 2020-02-06T02:12:23.870 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:23.868+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.869377+0000) 2020-02-06T02:12:23.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:23.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.872757+0000) 2020-02-06T02:12:23.873 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:23.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:23.873066+0000) 2020-02-06T02:12:23.888 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:23.886+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225853 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:25.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:25.216+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:25.217423+0000) 2020-02-06T02:12:26.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:26.174+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:26.175315+0000) 2020-02-06T02:12:26.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:26.537+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:26.538530+0000) 2020-02-06T02:12:27.153 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:27.151+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.152775+0000) 2020-02-06T02:12:27.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:27.426+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.427062+0000) 2020-02-06T02:12:27.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:27.426+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.427296+0000) 2020-02-06T02:12:27.428 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:27.426+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.427619+0000) 2020-02-06T02:12:27.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:27.656+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.657581+0000) 2020-02-06T02:12:27.681 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:27.680+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:27.680782+0000) 2020-02-06T02:12:28.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:28.893+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.894402+0000) 2020-02-06T02:12:28.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:28.893+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.895020+0000) 2020-02-06T02:12:28.896 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:28.894+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.895483+0000) 2020-02-06T02:12:28.896 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:28.894+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.895772+0000) 2020-02-06T02:12:28.897 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:28.894+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.896270+0000) 2020-02-06T02:12:28.897 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:28.895+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.896542+0000) 2020-02-06T02:12:28.898 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:28.895+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.897093+0000) 2020-02-06T02:12:28.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:28.896+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.897555+0000) 2020-02-06T02:12:28.899 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:28.895+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.897276+0000) 2020-02-06T02:12:28.899 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:28.896+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.898114+0000) 2020-02-06T02:12:28.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:28.898+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.899171+0000) 2020-02-06T02:12:28.900 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:28.897+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.899215+0000) 2020-02-06T02:12:28.900 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:28.898+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.899525+0000) 2020-02-06T02:12:28.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:28.903+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:28.905017+0000) 2020-02-06T02:12:28.918 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:28.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225897 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:29.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:29.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:29.797029+0000) 2020-02-06T02:12:30.043 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:30.042+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:30.042852+0000) 2020-02-06T02:12:30.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:30.653+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:30.653783+0000) 2020-02-06T02:12:30.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:30.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:30.931182+0000) 2020-02-06T02:12:33.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:33.511+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.512184+0000) 2020-02-06T02:12:33.923 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:33.921+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.922859+0000) 2020-02-06T02:12:33.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:33.922+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.923951+0000) 2020-02-06T02:12:33.925 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:33.923+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.924445+0000) 2020-02-06T02:12:33.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:33.923+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.924974+0000) 2020-02-06T02:12:33.927 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:33.925+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.926389+0000) 2020-02-06T02:12:33.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:33.925+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.926830+0000) 2020-02-06T02:12:33.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:33.926+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.927122+0000) 2020-02-06T02:12:33.928 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:33.925+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.927360+0000) 2020-02-06T02:12:33.929 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:33.926+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.928126+0000) 2020-02-06T02:12:33.930 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:33.927+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.929027+0000) 2020-02-06T02:12:33.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:33.927+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.929204+0000) 2020-02-06T02:12:33.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:33.928+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.929824+0000) 2020-02-06T02:12:33.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:33.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.931619+0000) 2020-02-06T02:12:33.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:33.930+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:33.931868+0000) 2020-02-06T02:12:33.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:33.944+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225936 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:35.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:35.224+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:35.225517+0000) 2020-02-06T02:12:36.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:36.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:36.175558+0000) 2020-02-06T02:12:36.539 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:36.538+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:36.538785+0000) 2020-02-06T02:12:37.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:37.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.153005+0000) 2020-02-06T02:12:37.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:37.426+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.427284+0000) 2020-02-06T02:12:37.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:37.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.427596+0000) 2020-02-06T02:12:37.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:37.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.427838+0000) 2020-02-06T02:12:37.658 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:37.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.657813+0000) 2020-02-06T02:12:37.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:37.679+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:37.681007+0000) 2020-02-06T02:12:38.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:38.949+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.951063+0000) 2020-02-06T02:12:38.952 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:38.950+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.951705+0000) 2020-02-06T02:12:38.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:38.952+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.952995+0000) 2020-02-06T02:12:38.954 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:38.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.953315+0000) 2020-02-06T02:12:38.954 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:38.953+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.953752+0000) 2020-02-06T02:12:38.955 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:38.953+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.954473+0000) 2020-02-06T02:12:38.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:38.954+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.954648+0000) 2020-02-06T02:12:38.956 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:38.954+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.955087+0000) 2020-02-06T02:12:38.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:38.954+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.955741+0000) 2020-02-06T02:12:38.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:38.956+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.957099+0000) 2020-02-06T02:12:38.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:38.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.957335+0000) 2020-02-06T02:12:38.958 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:38.956+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.957812+0000) 2020-02-06T02:12:38.959 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:38.957+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.958302+0000) 2020-02-06T02:12:38.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:38.960+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:38.962053+0000) 2020-02-06T02:12:38.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:38.975+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 225981 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:39.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:39.796+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:39.797278+0000) 2020-02-06T02:12:40.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:40.042+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:40.043109+0000) 2020-02-06T02:12:40.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:40.653+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:40.654014+0000) 2020-02-06T02:12:40.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:40.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:40.931450+0000) 2020-02-06T02:12:43.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:43.512+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.512419+0000) 2020-02-06T02:12:43.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:43.979+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.980978+0000) 2020-02-06T02:12:43.982 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:43.981+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.982044+0000) 2020-02-06T02:12:43.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:43.981+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.982545+0000) 2020-02-06T02:12:43.983 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:43.981+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.983000+0000) 2020-02-06T02:12:43.984 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:43.982+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.983369+0000) 2020-02-06T02:12:43.984 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:43.982+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.983735+0000) 2020-02-06T02:12:43.985 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:43.984+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.984627+0000) 2020-02-06T02:12:43.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:43.983+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.984247+0000) 2020-02-06T02:12:43.986 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:43.984+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.985200+0000) 2020-02-06T02:12:43.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:43.984+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.985832+0000) 2020-02-06T02:12:43.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:43.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.986173+0000) 2020-02-06T02:12:43.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:43.986+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.986518+0000) 2020-02-06T02:12:43.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:43.986+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.986867+0000) 2020-02-06T02:12:43.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:43.989+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:43.990967+0000) 2020-02-06T02:12:44.005 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:44.004+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226019 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:44.512 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:12:44.518 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:12:45.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:45.233+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:45.234114+0000) 2020-02-06T02:12:46.176 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:46.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:46.175763+0000) 2020-02-06T02:12:46.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:46.538+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:46.539009+0000) 2020-02-06T02:12:47.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:47.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.153227+0000) 2020-02-06T02:12:47.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:47.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.427515+0000) 2020-02-06T02:12:47.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:47.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.427833+0000) 2020-02-06T02:12:47.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:47.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.428088+0000) 2020-02-06T02:12:47.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:47.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.658045+0000) 2020-02-06T02:12:47.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:47.680+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:47.681286+0000) 2020-02-06T02:12:49.011 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:49.009+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.010476+0000) 2020-02-06T02:12:49.012 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:49.010+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.011976+0000) 2020-02-06T02:12:49.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:49.012+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.012941+0000) 2020-02-06T02:12:49.014 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:49.013+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.013493+0000) 2020-02-06T02:12:49.014 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:49.013+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.013644+0000) 2020-02-06T02:12:49.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:49.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.014503+0000) 2020-02-06T02:12:49.015 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:49.013+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.014243+0000) 2020-02-06T02:12:49.015 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:49.013+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.014822+0000) 2020-02-06T02:12:49.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:49.014+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.015551+0000) 2020-02-06T02:12:49.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:49.014+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.016028+0000) 2020-02-06T02:12:49.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:49.015+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.016267+0000) 2020-02-06T02:12:49.017 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:49.016+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.016811+0000) 2020-02-06T02:12:49.017 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:49.015+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.016366+0000) 2020-02-06T02:12:49.021 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:49.019+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.020651+0000) 2020-02-06T02:12:49.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:49.033+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:49.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:49.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:49.797494+0000) 2020-02-06T02:12:50.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:50.043+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:50.043341+0000) 2020-02-06T02:12:50.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:50.653+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:50.654248+0000) 2020-02-06T02:12:50.932 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:50.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:50.931710+0000) 2020-02-06T02:12:53.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:53.512+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:53.512642+0000) 2020-02-06T02:12:54.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:54.037+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.038962+0000) 2020-02-06T02:12:54.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:54.039+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.040157+0000) 2020-02-06T02:12:54.041 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:54.039+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.040497+0000) 2020-02-06T02:12:54.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:54.040+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.041217+0000) 2020-02-06T02:12:54.042 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:54.041+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.041618+0000) 2020-02-06T02:12:54.043 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:54.041+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.042256+0000) 2020-02-06T02:12:54.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:54.041+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.042579+0000) 2020-02-06T02:12:54.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:54.042+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.042941+0000) 2020-02-06T02:12:54.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:54.043+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.043616+0000) 2020-02-06T02:12:54.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:54.044+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.045188+0000) 2020-02-06T02:12:54.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:54.044+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.045816+0000) 2020-02-06T02:12:54.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:54.044+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.045935+0000) 2020-02-06T02:12:54.047 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:54.045+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.046079+0000) 2020-02-06T02:12:54.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:54.050+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:54.051184+0000) 2020-02-06T02:12:54.064 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:54.062+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226102 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:55.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:55.245+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:55.245558+0000) 2020-02-06T02:12:56.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:56.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:56.176038+0000) 2020-02-06T02:12:56.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:56.539+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:56.539266+0000) 2020-02-06T02:12:57.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:57.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.153464+0000) 2020-02-06T02:12:57.428 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:57.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.427752+0000) 2020-02-06T02:12:57.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:57.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.428077+0000) 2020-02-06T02:12:57.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:57.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.428353+0000) 2020-02-06T02:12:57.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:57.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.658339+0000) 2020-02-06T02:12:57.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:57.681+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:57.681516+0000) 2020-02-06T02:12:59.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:12:59.069+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.069990+0000) 2020-02-06T02:12:59.071 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:12:59.070+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.070319+0000) 2020-02-06T02:12:59.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:12:59.070+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.070695+0000) 2020-02-06T02:12:59.072 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:12:59.070+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.071285+0000) 2020-02-06T02:12:59.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:12:59.071+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.071395+0000) 2020-02-06T02:12:59.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:12:59.071+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.071802+0000) 2020-02-06T02:12:59.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:59.072+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.072366+0000) 2020-02-06T02:12:59.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:12:59.072+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.073115+0000) 2020-02-06T02:12:59.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:59.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.073545+0000) 2020-02-06T02:12:59.075 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:12:59.073+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.074206+0000) 2020-02-06T02:12:59.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:12:59.074+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.074657+0000) 2020-02-06T02:12:59.075 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:12:59.074+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.074850+0000) 2020-02-06T02:12:59.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:12:59.074+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.075404+0000) 2020-02-06T02:12:59.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:59.077+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.077991+0000) 2020-02-06T02:12:59.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:12:59.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226147 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:12:59.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:12:59.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:12:59.797703+0000) 2020-02-06T02:13:00.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:00.043+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:00.043543+0000) 2020-02-06T02:13:00.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:00.654+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:00.654519+0000) 2020-02-06T02:13:00.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:00.930+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:00.931942+0000) 2020-02-06T02:13:03.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:03.511+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:03.512877+0000) 2020-02-06T02:13:04.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:04.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.097831+0000) 2020-02-06T02:13:04.100 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:04.098+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.099509+0000) 2020-02-06T02:13:04.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:04.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.099920+0000) 2020-02-06T02:13:04.101 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:04.099+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.100509+0000) 2020-02-06T02:13:04.101 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:04.099+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.100371+0000) 2020-02-06T02:13:04.101 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:04.100+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.100737+0000) 2020-02-06T02:13:04.102 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:04.099+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.100915+0000) 2020-02-06T02:13:04.102 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:04.100+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.101218+0000) 2020-02-06T02:13:04.103 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:04.101+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.101578+0000) 2020-02-06T02:13:04.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:04.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.101965+0000) 2020-02-06T02:13:04.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:04.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.102151+0000) 2020-02-06T02:13:04.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:04.101+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.102634+0000) 2020-02-06T02:13:04.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:04.102+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.102493+0000) 2020-02-06T02:13:04.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:04.103+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:04.105020+0000) 2020-02-06T02:13:04.121 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:04.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226185 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:05.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:05.254+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:05.255300+0000) 2020-02-06T02:13:06.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:06.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:06.176299+0000) 2020-02-06T02:13:06.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:06.539+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:06.539515+0000) 2020-02-06T02:13:07.154 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:07.152+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.153700+0000) 2020-02-06T02:13:07.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:07.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.428006+0000) 2020-02-06T02:13:07.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:07.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.428313+0000) 2020-02-06T02:13:07.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:07.428+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.428600+0000) 2020-02-06T02:13:07.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:07.657+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.658578+0000) 2020-02-06T02:13:07.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:07.680+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:07.681747+0000) 2020-02-06T02:13:09.128 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:09.126+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.127623+0000) 2020-02-06T02:13:09.129 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:09.127+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.128266+0000) 2020-02-06T02:13:09.129 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:09.127+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.128554+0000) 2020-02-06T02:13:09.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:09.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.129176+0000) 2020-02-06T02:13:09.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:09.128+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.129603+0000) 2020-02-06T02:13:09.131 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:09.129+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.130714+0000) 2020-02-06T02:13:09.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:09.131+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.131417+0000) 2020-02-06T02:13:09.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:09.130+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.131667+0000) 2020-02-06T02:13:09.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:09.130+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.131948+0000) 2020-02-06T02:13:09.133 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:09.131+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.132068+0000) 2020-02-06T02:13:09.133 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:09.132+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.132682+0000) 2020-02-06T02:13:09.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:09.132+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.132627+0000) 2020-02-06T02:13:09.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:09.133+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.133461+0000) 2020-02-06T02:13:09.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:09.135+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.136227+0000) 2020-02-06T02:13:09.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:09.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:09.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:09.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:09.797915+0000) 2020-02-06T02:13:10.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:10.042+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:10.043774+0000) 2020-02-06T02:13:10.655 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:10.654+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:10.654742+0000) 2020-02-06T02:13:10.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:10.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:10.932209+0000) 2020-02-06T02:13:13.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:13.512+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:13.513128+0000) 2020-02-06T02:13:14.157 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:14.155+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.156417+0000) 2020-02-06T02:13:14.158 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:14.157+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.157474+0000) 2020-02-06T02:13:14.159 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:14.157+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.158169+0000) 2020-02-06T02:13:14.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:14.157+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.158441+0000) 2020-02-06T02:13:14.159 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:14.157+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.158973+0000) 2020-02-06T02:13:14.160 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:14.158+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.159435+0000) 2020-02-06T02:13:14.160 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:14.158+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.159549+0000) 2020-02-06T02:13:14.161 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:14.159+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.160024+0000) 2020-02-06T02:13:14.161 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:14.159+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.160490+0000) 2020-02-06T02:13:14.162 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:14.160+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.160991+0000) 2020-02-06T02:13:14.162 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:14.160+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.161154+0000) 2020-02-06T02:13:14.162 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:14.160+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.160946+0000) 2020-02-06T02:13:14.163 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:14.161+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.162380+0000) 2020-02-06T02:13:14.165 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:14.163+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:14.164711+0000) 2020-02-06T02:13:14.181 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:14.180+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226267 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:14.643 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:13:14.649 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:13:15.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:15.262+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:15.263568+0000) 2020-02-06T02:13:16.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:16.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:16.176563+0000) 2020-02-06T02:13:16.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:16.538+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:16.539798+0000) 2020-02-06T02:13:17.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:17.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.153916+0000) 2020-02-06T02:13:17.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:17.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.428260+0000) 2020-02-06T02:13:17.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:17.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.428496+0000) 2020-02-06T02:13:17.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:17.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.428881+0000) 2020-02-06T02:13:17.659 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:17.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.658853+0000) 2020-02-06T02:13:17.682 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:17.680+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:17.681996+0000) 2020-02-06T02:13:19.187 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:19.185+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.186366+0000) 2020-02-06T02:13:19.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:19.185+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.186915+0000) 2020-02-06T02:13:19.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:19.186+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.187423+0000) 2020-02-06T02:13:19.188 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:19.186+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.188054+0000) 2020-02-06T02:13:19.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:19.187+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.188525+0000) 2020-02-06T02:13:19.189 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:19.187+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.188549+0000) 2020-02-06T02:13:19.190 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:19.187+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.189261+0000) 2020-02-06T02:13:19.191 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:19.189+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.190070+0000) 2020-02-06T02:13:19.191 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:19.189+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.190699+0000) 2020-02-06T02:13:19.191 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:19.189+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.190508+0000) 2020-02-06T02:13:19.192 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:19.189+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.191264+0000) 2020-02-06T02:13:19.192 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:19.190+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.191629+0000) 2020-02-06T02:13:19.192 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:19.190+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.192306+0000) 2020-02-06T02:13:19.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:19.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.195860+0000) 2020-02-06T02:13:19.209 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:19.206+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226313 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:19.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:19.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:19.798126+0000) 2020-02-06T02:13:20.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:20.043+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:20.043975+0000) 2020-02-06T02:13:20.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:20.654+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:20.655000+0000) 2020-02-06T02:13:20.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:20.931+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:20.932489+0000) 2020-02-06T02:13:23.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:23.512+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:23.513359+0000) 2020-02-06T02:13:24.215 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:24.213+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.214574+0000) 2020-02-06T02:13:24.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:24.214+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.215257+0000) 2020-02-06T02:13:24.217 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:24.215+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.216656+0000) 2020-02-06T02:13:24.217 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:24.215+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.216948+0000) 2020-02-06T02:13:24.218 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:24.216+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.217502+0000) 2020-02-06T02:13:24.218 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:24.215+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.217234+0000) 2020-02-06T02:13:24.219 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:24.216+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.217822+0000) 2020-02-06T02:13:24.219 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:24.217+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.218163+0000) 2020-02-06T02:13:24.219 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:24.217+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.218629+0000) 2020-02-06T02:13:24.220 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:24.216+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.218316+0000) 2020-02-06T02:13:24.220 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:24.218+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.219141+0000) 2020-02-06T02:13:24.220 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:24.217+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.219261+0000) 2020-02-06T02:13:24.221 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:24.218+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.219779+0000) 2020-02-06T02:13:24.223 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:24.221+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:24.222394+0000) 2020-02-06T02:13:24.238 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:24.236+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:25.272 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:25.271+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:25.271727+0000) 2020-02-06T02:13:26.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:26.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:26.176758+0000) 2020-02-06T02:13:26.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:26.539+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:26.540103+0000) 2020-02-06T02:13:27.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:27.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.154171+0000) 2020-02-06T02:13:27.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:27.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.428517+0000) 2020-02-06T02:13:27.429 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:27.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.428682+0000) 2020-02-06T02:13:27.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:27.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.429119+0000) 2020-02-06T02:13:27.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:27.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.659121+0000) 2020-02-06T02:13:27.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:27.681+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:27.682264+0000) 2020-02-06T02:13:29.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:29.242+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.243568+0000) 2020-02-06T02:13:29.246 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:29.243+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.245294+0000) 2020-02-06T02:13:29.246 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:29.244+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.246135+0000) 2020-02-06T02:13:29.247 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:29.245+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.246341+0000) 2020-02-06T02:13:29.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:29.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.247220+0000) 2020-02-06T02:13:29.249 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:29.246+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.247636+0000) 2020-02-06T02:13:29.249 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:29.246+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.247839+0000) 2020-02-06T02:13:29.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:29.246+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.248278+0000) 2020-02-06T02:13:29.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:29.247+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.248932+0000) 2020-02-06T02:13:29.250 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:29.247+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.248987+0000) 2020-02-06T02:13:29.250 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:29.247+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.249308+0000) 2020-02-06T02:13:29.251 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:29.247+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.248514+0000) 2020-02-06T02:13:29.251 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:29.248+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.249572+0000) 2020-02-06T02:13:29.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:29.253+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.254356+0000) 2020-02-06T02:13:29.267 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:29.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226396 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:29.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:29.797+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:29.798342+0000) 2020-02-06T02:13:30.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:30.043+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:30.044183+0000) 2020-02-06T02:13:30.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:30.654+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:30.655251+0000) 2020-02-06T02:13:30.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:30.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:30.932744+0000) 2020-02-06T02:13:33.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:33.513+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:33.513579+0000) 2020-02-06T02:13:34.272 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:34.270+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.271455+0000) 2020-02-06T02:13:34.274 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:34.272+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.273367+0000) 2020-02-06T02:13:34.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:34.272+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.273825+0000) 2020-02-06T02:13:34.275 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:34.272+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.273941+0000) 2020-02-06T02:13:34.275 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:34.273+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.274361+0000) 2020-02-06T02:13:34.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:34.274+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.275072+0000) 2020-02-06T02:13:34.276 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:34.273+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.274548+0000) 2020-02-06T02:13:34.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:34.273+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.275047+0000) 2020-02-06T02:13:34.277 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:34.274+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.276024+0000) 2020-02-06T02:13:34.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:34.275+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.276583+0000) 2020-02-06T02:13:34.277 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:34.275+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.276419+0000) 2020-02-06T02:13:34.278 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:34.275+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.276884+0000) 2020-02-06T02:13:34.278 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:34.275+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.277117+0000) 2020-02-06T02:13:34.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:34.279+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:34.280797+0000) 2020-02-06T02:13:34.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:34.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226436 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:35.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:35.278+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:35.279968+0000) 2020-02-06T02:13:36.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:36.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:36.177001+0000) 2020-02-06T02:13:36.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:36.539+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:36.540360+0000) 2020-02-06T02:13:37.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:37.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.154413+0000) 2020-02-06T02:13:37.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:37.427+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.428768+0000) 2020-02-06T02:13:37.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:37.427+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.428906+0000) 2020-02-06T02:13:37.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:37.427+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.429323+0000) 2020-02-06T02:13:37.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:37.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.659381+0000) 2020-02-06T02:13:37.683 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:37.682+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:37.682491+0000) 2020-02-06T02:13:39.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:39.300+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.301990+0000) 2020-02-06T02:13:39.303 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:39.301+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.302564+0000) 2020-02-06T02:13:39.305 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:39.302+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.303990+0000) 2020-02-06T02:13:39.305 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:39.303+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.304725+0000) 2020-02-06T02:13:39.306 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:39.303+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.304876+0000) 2020-02-06T02:13:39.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:39.304+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.305106+0000) 2020-02-06T02:13:39.307 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:39.304+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.305403+0000) 2020-02-06T02:13:39.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:39.304+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.305768+0000) 2020-02-06T02:13:39.307 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:39.303+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.305146+0000) 2020-02-06T02:13:39.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:39.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.305844+0000) 2020-02-06T02:13:39.308 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:39.305+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.306122+0000) 2020-02-06T02:13:39.308 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:39.305+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.306938+0000) 2020-02-06T02:13:39.308 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:39.304+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.306229+0000) 2020-02-06T02:13:39.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:39.314+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.315773+0000) 2020-02-06T02:13:39.325 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:39.323+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:39.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:39.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:39.798579+0000) 2020-02-06T02:13:40.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:40.044+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:40.044411+0000) 2020-02-06T02:13:40.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:40.654+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:40.655508+0000) 2020-02-06T02:13:40.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:40.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:40.932984+0000) 2020-02-06T02:13:43.514 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:43.513+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:43.513806+0000) 2020-02-06T02:13:44.331 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:44.329+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.330724+0000) 2020-02-06T02:13:44.332 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:44.329+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.331287+0000) 2020-02-06T02:13:44.333 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:44.330+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.332147+0000) 2020-02-06T02:13:44.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:44.331+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.332322+0000) 2020-02-06T02:13:44.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:44.332+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.333617+0000) 2020-02-06T02:13:44.334 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:44.332+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.334113+0000) 2020-02-06T02:13:44.335 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:44.333+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.334849+0000) 2020-02-06T02:13:44.336 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:44.333+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.334635+0000) 2020-02-06T02:13:44.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:44.334+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.335044+0000) 2020-02-06T02:13:44.336 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:44.334+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.335935+0000) 2020-02-06T02:13:44.337 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:44.334+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.335702+0000) 2020-02-06T02:13:44.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:44.334+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.336277+0000) 2020-02-06T02:13:44.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:44.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.339394+0000) 2020-02-06T02:13:44.340 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:44.338+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:44.339610+0000) 2020-02-06T02:13:44.355 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:44.352+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:44.783 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:13:44.789 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:13:45.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:45.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:45.287918+0000) 2020-02-06T02:13:46.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:46.175+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:46.177250+0000) 2020-02-06T02:13:46.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:46.539+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:46.540647+0000) 2020-02-06T02:13:47.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:47.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.154648+0000) 2020-02-06T02:13:47.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:47.428+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.429018+0000) 2020-02-06T02:13:47.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:47.428+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.429136+0000) 2020-02-06T02:13:47.430 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:47.428+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.429553+0000) 2020-02-06T02:13:47.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:47.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.659607+0000) 2020-02-06T02:13:47.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:47.682+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:47.682743+0000) 2020-02-06T02:13:49.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:49.358+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.359627+0000) 2020-02-06T02:13:49.361 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:49.358+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.360247+0000) 2020-02-06T02:13:49.362 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:49.359+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.361142+0000) 2020-02-06T02:13:49.362 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:49.361+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.361490+0000) 2020-02-06T02:13:49.363 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:49.361+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.362506+0000) 2020-02-06T02:13:49.364 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:49.362+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.363262+0000) 2020-02-06T02:13:49.364 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:49.362+0000 7f5581a2e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.363546+0000) 2020-02-06T02:13:49.365 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:49.363+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.364760+0000) 2020-02-06T02:13:49.366 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:49.363+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.364579+0000) 2020-02-06T02:13:49.366 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:49.364+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.365443+0000) 2020-02-06T02:13:49.366 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:49.364+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.365758+0000) 2020-02-06T02:13:49.367 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:49.364+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.365794+0000) 2020-02-06T02:13:49.367 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:49.364+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.365970+0000) 2020-02-06T02:13:49.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:49.368+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.369282+0000) 2020-02-06T02:13:49.387 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:49.385+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:49.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:49.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:49.798803+0000) 2020-02-06T02:13:50.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:50.044+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:50.044617+0000) 2020-02-06T02:13:50.656 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:50.655+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:50.655777+0000) 2020-02-06T02:13:50.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:50.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:50.933233+0000) 2020-02-06T02:13:53.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:53.513+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:53.514029+0000) 2020-02-06T02:13:54.392 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:54.391+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.391299+0000) 2020-02-06T02:13:54.393 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:54.392+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.392969+0000) 2020-02-06T02:13:54.394 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:54.392+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.393529+0000) 2020-02-06T02:13:54.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:54.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.394159+0000) 2020-02-06T02:13:54.395 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:54.394+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.394511+0000) 2020-02-06T02:13:54.396 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:54.393+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.395252+0000) 2020-02-06T02:13:54.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:54.395+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.396008+0000) 2020-02-06T02:13:54.397 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:54.395+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.396590+0000) 2020-02-06T02:13:54.397 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:54.395+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.396625+0000) 2020-02-06T02:13:54.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:54.395+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.397153+0000) 2020-02-06T02:13:54.398 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:54.397+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.397590+0000) 2020-02-06T02:13:54.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:54.395+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.397326+0000) 2020-02-06T02:13:54.401 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:54.400+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.400972+0000) 2020-02-06T02:13:54.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:54.401+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:54.401267+0000) 2020-02-06T02:13:54.417 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:54.414+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226606 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:55.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:55.296+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:55.297697+0000) 2020-02-06T02:13:56.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:56.176+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:56.177541+0000) 2020-02-06T02:13:56.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:56.540+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:56.540937+0000) 2020-02-06T02:13:57.155 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:57.153+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.154912+0000) 2020-02-06T02:13:57.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:57.428+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.429253+0000) 2020-02-06T02:13:57.430 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:57.428+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.429416+0000) 2020-02-06T02:13:57.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:57.428+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.429769+0000) 2020-02-06T02:13:57.660 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:57.658+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.659841+0000) 2020-02-06T02:13:57.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:57.681+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:57.682978+0000) 2020-02-06T02:13:59.423 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:13:59.421+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.422258+0000) 2020-02-06T02:13:59.423 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:13:59.421+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.422877+0000) 2020-02-06T02:13:59.424 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:13:59.422+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.423386+0000) 2020-02-06T02:13:59.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:13:59.421+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.423167+0000) 2020-02-06T02:13:59.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:59.423+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.424235+0000) 2020-02-06T02:13:59.425 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:13:59.423+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.424973+0000) 2020-02-06T02:13:59.426 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:13:59.423+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.425366+0000) 2020-02-06T02:13:59.426 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:13:59.424+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.425980+0000) 2020-02-06T02:13:59.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:59.425+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.426205+0000) 2020-02-06T02:13:59.427 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:13:59.426+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.427055+0000) 2020-02-06T02:13:59.428 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:13:59.426+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.427662+0000) 2020-02-06T02:13:59.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:13:59.427+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.428185+0000) 2020-02-06T02:13:59.429 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:13:59.427+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.428473+0000) 2020-02-06T02:13:59.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:59.430+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.431568+0000) 2020-02-06T02:13:59.448 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:13:59.445+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:13:59.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:13:59.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:13:59.799017+0000) 2020-02-06T02:14:00.045 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:00.043+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:00.044835+0000) 2020-02-06T02:14:00.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:00.655+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:00.656040+0000) 2020-02-06T02:14:00.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:00.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:00.933455+0000) 2020-02-06T02:14:03.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:03.513+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:03.514318+0000) 2020-02-06T02:14:04.454 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:04.451+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.452939+0000) 2020-02-06T02:14:04.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:04.452+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.453514+0000) 2020-02-06T02:14:04.455 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:04.452+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.453971+0000) 2020-02-06T02:14:04.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:04.453+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.454391+0000) 2020-02-06T02:14:04.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:04.453+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.455024+0000) 2020-02-06T02:14:04.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:04.455+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.456334+0000) 2020-02-06T02:14:04.457 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:04.455+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.456468+0000) 2020-02-06T02:14:04.458 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:04.456+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.457709+0000) 2020-02-06T02:14:04.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:04.456+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.457826+0000) 2020-02-06T02:14:04.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:04.457+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.458142+0000) 2020-02-06T02:14:04.459 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:04.457+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.458745+0000) 2020-02-06T02:14:04.460 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:04.457+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.458982+0000) 2020-02-06T02:14:04.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:04.457+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.459205+0000) 2020-02-06T02:14:04.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:04.460+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:04.462038+0000) 2020-02-06T02:14:04.477 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:04.475+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226692 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:05.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:05.305+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:05.305920+0000) 2020-02-06T02:14:06.178 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:06.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:06.177748+0000) 2020-02-06T02:14:06.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:06.540+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:06.541175+0000) 2020-02-06T02:14:07.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:07.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.155143+0000) 2020-02-06T02:14:07.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:07.429+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.429551+0000) 2020-02-06T02:14:07.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:07.429+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.429729+0000) 2020-02-06T02:14:07.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:07.429+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.429978+0000) 2020-02-06T02:14:07.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:07.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.660110+0000) 2020-02-06T02:14:07.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:07.682+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:07.683208+0000) 2020-02-06T02:14:09.483 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:09.480+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.481348+0000) 2020-02-06T02:14:09.484 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:09.483+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.483661+0000) 2020-02-06T02:14:09.485 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:09.483+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.484488+0000) 2020-02-06T02:14:09.485 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:09.483+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.484572+0000) 2020-02-06T02:14:09.486 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:09.483+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.485064+0000) 2020-02-06T02:14:09.486 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:09.484+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.485214+0000) 2020-02-06T02:14:09.486 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:09.484+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.485534+0000) 2020-02-06T02:14:09.487 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:09.485+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.486099+0000) 2020-02-06T02:14:09.487 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:09.486+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.486546+0000) 2020-02-06T02:14:09.487 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:09.486+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.487149+0000) 2020-02-06T02:14:09.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:09.487+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.487670+0000) 2020-02-06T02:14:09.489 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:09.488+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.488537+0000) 2020-02-06T02:14:09.490 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:09.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.490224+0000) 2020-02-06T02:14:09.491 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:09.490+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.491233+0000) 2020-02-06T02:14:09.507 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:09.506+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226737 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:09.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:09.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:09.799237+0000) 2020-02-06T02:14:10.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:10.044+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:10.045039+0000) 2020-02-06T02:14:10.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:10.655+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:10.656280+0000) 2020-02-06T02:14:10.934 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:10.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:10.933683+0000) 2020-02-06T02:14:13.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:13.513+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:13.514568+0000) 2020-02-06T02:14:14.514 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:14.511+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.512851+0000) 2020-02-06T02:14:14.515 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:14.513+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.513590+0000) 2020-02-06T02:14:14.515 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:14.513+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.514273+0000) 2020-02-06T02:14:14.516 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:14.514+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.515251+0000) 2020-02-06T02:14:14.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:14.514+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.515218+0000) 2020-02-06T02:14:14.517 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:14.515+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.515846+0000) 2020-02-06T02:14:14.518 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:14.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.516211+0000) 2020-02-06T02:14:14.518 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:14.515+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.516540+0000) 2020-02-06T02:14:14.518 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:14.517+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.517975+0000) 2020-02-06T02:14:14.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:14.516+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.517531+0000) 2020-02-06T02:14:14.519 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:14.517+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.518418+0000) 2020-02-06T02:14:14.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:14.517+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.518115+0000) 2020-02-06T02:14:14.520 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:14.518+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.519177+0000) 2020-02-06T02:14:14.523 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:14.519+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:14.520867+0000) 2020-02-06T02:14:14.537 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:14.536+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226777 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:14.923 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:14:14.929 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:14:15.315 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:15.312+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:15.313950+0000) 2020-02-06T02:14:16.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:16.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:16.178006+0000) 2020-02-06T02:14:16.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:16.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:16.541457+0000) 2020-02-06T02:14:17.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:17.154+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.155376+0000) 2020-02-06T02:14:17.430 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:17.429+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.429806+0000) 2020-02-06T02:14:17.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:17.429+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.430004+0000) 2020-02-06T02:14:17.431 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:17.429+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.430223+0000) 2020-02-06T02:14:17.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:17.659+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.660358+0000) 2020-02-06T02:14:17.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:17.682+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:17.683436+0000) 2020-02-06T02:14:19.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:19.540+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.541287+0000) 2020-02-06T02:14:19.543 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:19.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.542932+0000) 2020-02-06T02:14:19.544 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:19.542+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.543280+0000) 2020-02-06T02:14:19.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:19.542+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.543473+0000) 2020-02-06T02:14:19.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:19.543+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.544616+0000) 2020-02-06T02:14:19.545 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:19.543+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.544226+0000) 2020-02-06T02:14:19.546 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:19.544+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.545666+0000) 2020-02-06T02:14:19.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:19.544+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.545991+0000) 2020-02-06T02:14:19.547 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:19.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.546043+0000) 2020-02-06T02:14:19.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:19.545+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.546362+0000) 2020-02-06T02:14:19.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:19.545+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.546347+0000) 2020-02-06T02:14:19.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:19.546+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.546921+0000) 2020-02-06T02:14:19.548 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:19.546+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.547077+0000) 2020-02-06T02:14:19.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:19.550+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.551697+0000) 2020-02-06T02:14:19.568 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:19.566+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226822 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:19.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:19.798+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:19.799448+0000) 2020-02-06T02:14:20.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:20.044+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:20.045313+0000) 2020-02-06T02:14:20.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:20.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:20.656533+0000) 2020-02-06T02:14:20.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:20.932+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:20.933934+0000) 2020-02-06T02:14:23.515 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:23.514+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:23.514793+0000) 2020-02-06T02:14:24.572 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:24.570+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.571755+0000) 2020-02-06T02:14:24.573 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:24.572+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.573104+0000) 2020-02-06T02:14:24.574 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:24.572+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.573610+0000) 2020-02-06T02:14:24.575 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:24.574+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.574561+0000) 2020-02-06T02:14:24.576 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:24.573+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.575015+0000) 2020-02-06T02:14:24.576 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:24.574+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.575346+0000) 2020-02-06T02:14:24.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:24.575+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.575794+0000) 2020-02-06T02:14:24.577 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:24.576+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.576683+0000) 2020-02-06T02:14:24.577 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:24.576+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.577040+0000) 2020-02-06T02:14:24.578 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:24.577+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.577626+0000) 2020-02-06T02:14:24.580 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:24.578+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.578925+0000) 2020-02-06T02:14:24.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:24.577+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.578454+0000) 2020-02-06T02:14:24.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:24.581+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.582284+0000) 2020-02-06T02:14:24.583 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:24.581+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:24.582534+0000) 2020-02-06T02:14:24.597 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:24.596+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:25.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:25.325+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:25.326354+0000) 2020-02-06T02:14:26.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:26.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:26.178269+0000) 2020-02-06T02:14:26.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:26.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:26.541738+0000) 2020-02-06T02:14:27.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:27.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.155613+0000) 2020-02-06T02:14:27.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:27.429+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.430056+0000) 2020-02-06T02:14:27.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:27.429+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.430249+0000) 2020-02-06T02:14:27.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:27.430+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.430479+0000) 2020-02-06T02:14:27.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:27.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.660590+0000) 2020-02-06T02:14:27.684 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:27.683+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:27.683702+0000) 2020-02-06T02:14:29.603 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:29.601+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.602448+0000) 2020-02-06T02:14:29.604 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:29.602+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.603597+0000) 2020-02-06T02:14:29.605 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:29.603+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.604332+0000) 2020-02-06T02:14:29.605 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:29.604+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.604730+0000) 2020-02-06T02:14:29.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:29.604+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.605854+0000) 2020-02-06T02:14:29.607 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:29.606+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.606608+0000) 2020-02-06T02:14:29.608 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:29.606+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.606989+0000) 2020-02-06T02:14:29.608 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:29.606+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.607050+0000) 2020-02-06T02:14:29.609 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:29.607+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.608101+0000) 2020-02-06T02:14:29.609 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:29.607+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.607887+0000) 2020-02-06T02:14:29.609 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:29.608+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.608636+0000) 2020-02-06T02:14:29.610 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:29.607+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.608888+0000) 2020-02-06T02:14:29.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:29.610+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.611089+0000) 2020-02-06T02:14:29.612 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:29.610+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.611411+0000) 2020-02-06T02:14:29.629 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:29.627+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226906 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:29.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:29.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:29.799660+0000) 2020-02-06T02:14:30.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:30.045+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:30.045513+0000) 2020-02-06T02:14:30.657 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:30.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:30.656806+0000) 2020-02-06T02:14:30.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:30.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:30.934172+0000) 2020-02-06T02:14:33.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:33.514+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:33.515017+0000) 2020-02-06T02:14:34.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:34.631+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.632475+0000) 2020-02-06T02:14:34.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:34.633+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.634356+0000) 2020-02-06T02:14:34.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:34.635+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.636062+0000) 2020-02-06T02:14:34.637 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:34.636+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.636987+0000) 2020-02-06T02:14:34.638 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:34.636+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.637343+0000) 2020-02-06T02:14:34.638 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:34.636+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.637806+0000) 2020-02-06T02:14:34.639 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:34.636+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.637879+0000) 2020-02-06T02:14:34.639 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:34.637+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.637934+0000) 2020-02-06T02:14:34.640 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:34.638+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.638857+0000) 2020-02-06T02:14:34.640 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:34.637+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.638320+0000) 2020-02-06T02:14:34.640 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:34.637+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.638695+0000) 2020-02-06T02:14:34.641 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:34.638+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.639167+0000) 2020-02-06T02:14:34.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:34.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.643463+0000) 2020-02-06T02:14:34.644 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:34.642+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:34.643656+0000) 2020-02-06T02:14:34.658 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:34.657+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226946 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:35.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:35.334+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:35.334588+0000) 2020-02-06T02:14:36.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:36.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:36.178548+0000) 2020-02-06T02:14:36.542 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:36.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:36.541965+0000) 2020-02-06T02:14:37.156 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:37.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.155845+0000) 2020-02-06T02:14:37.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:37.430+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.430308+0000) 2020-02-06T02:14:37.431 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:37.430+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.430505+0000) 2020-02-06T02:14:37.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:37.430+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.430715+0000) 2020-02-06T02:14:37.661 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:37.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.660821+0000) 2020-02-06T02:14:37.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:37.683+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:37.683927+0000) 2020-02-06T02:14:39.663 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:39.661+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.662623+0000) 2020-02-06T02:14:39.664 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:39.662+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.663548+0000) 2020-02-06T02:14:39.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:39.663+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.664269+0000) 2020-02-06T02:14:39.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:39.664+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.664912+0000) 2020-02-06T02:14:39.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:39.664+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.665116+0000) 2020-02-06T02:14:39.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:39.664+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.665457+0000) 2020-02-06T02:14:39.667 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:39.664+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.665921+0000) 2020-02-06T02:14:39.667 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:39.666+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.666736+0000) 2020-02-06T02:14:39.668 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:39.665+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.666913+0000) 2020-02-06T02:14:39.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:39.667+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.667771+0000) 2020-02-06T02:14:39.669 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:39.667+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.668064+0000) 2020-02-06T02:14:39.670 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:39.667+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.668326+0000) 2020-02-06T02:14:39.670 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:39.668+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.668850+0000) 2020-02-06T02:14:39.673 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:39.671+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.672626+0000) 2020-02-06T02:14:39.689 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:39.688+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 226990 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:39.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:39.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:39.799873+0000) 2020-02-06T02:14:40.046 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:40.045+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:40.045727+0000) 2020-02-06T02:14:40.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:40.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:40.657045+0000) 2020-02-06T02:14:40.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:40.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:40.934360+0000) 2020-02-06T02:14:43.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:43.515+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:43.515277+0000) 2020-02-06T02:14:44.696 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:44.694+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.695432+0000) 2020-02-06T02:14:44.696 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:44.694+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.695860+0000) 2020-02-06T02:14:44.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:44.695+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.696349+0000) 2020-02-06T02:14:44.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:44.696+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.697113+0000) 2020-02-06T02:14:44.698 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:44.695+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.697320+0000) 2020-02-06T02:14:44.699 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:44.696+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.698167+0000) 2020-02-06T02:14:44.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:44.699+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.699197+0000) 2020-02-06T02:14:44.700 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:44.697+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.699249+0000) 2020-02-06T02:14:44.700 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:44.699+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.699496+0000) 2020-02-06T02:14:44.701 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:44.698+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.700401+0000) 2020-02-06T02:14:44.701 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:44.699+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.700458+0000) 2020-02-06T02:14:44.702 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:44.699+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.700226+0000) 2020-02-06T02:14:44.702 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:44.699+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.700944+0000) 2020-02-06T02:14:44.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:44.703+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:44.704978+0000) 2020-02-06T02:14:44.720 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:44.718+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227029 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:45.071 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:14:45.077 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:14:45.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:45.344+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:45.344260+0000) 2020-02-06T02:14:46.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:46.177+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:46.178778+0000) 2020-02-06T02:14:46.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:46.540+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:46.542231+0000) 2020-02-06T02:14:47.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:47.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.156120+0000) 2020-02-06T02:14:47.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:47.429+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.430570+0000) 2020-02-06T02:14:47.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:47.429+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.430756+0000) 2020-02-06T02:14:47.432 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:47.429+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.430963+0000) 2020-02-06T02:14:47.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:47.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.661067+0000) 2020-02-06T02:14:47.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:47.684+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:47.684169+0000) 2020-02-06T02:14:49.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:49.724+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.725360+0000) 2020-02-06T02:14:49.726 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:49.724+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.725874+0000) 2020-02-06T02:14:49.727 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:49.725+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.726143+0000) 2020-02-06T02:14:49.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:49.725+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.726651+0000) 2020-02-06T02:14:49.729 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:49.727+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.728680+0000) 2020-02-06T02:14:49.729 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:49.727+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.728750+0000) 2020-02-06T02:14:49.730 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:49.727+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.729337+0000) 2020-02-06T02:14:49.730 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:49.728+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.729605+0000) 2020-02-06T02:14:49.731 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:49.728+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.729363+0000) 2020-02-06T02:14:49.731 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:49.728+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.729429+0000) 2020-02-06T02:14:49.731 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:49.728+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.730109+0000) 2020-02-06T02:14:49.732 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:49.729+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.730541+0000) 2020-02-06T02:14:49.732 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:49.729+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.730274+0000) 2020-02-06T02:14:49.736 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:49.734+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.735527+0000) 2020-02-06T02:14:49.751 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:49.749+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:49.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:49.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:49.800112+0000) 2020-02-06T02:14:50.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:50.044+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:50.045951+0000) 2020-02-06T02:14:50.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:50.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:50.657304+0000) 2020-02-06T02:14:50.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:50.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:50.934621+0000) 2020-02-06T02:14:53.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:53.514+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:53.515501+0000) 2020-02-06T02:14:54.757 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:54.755+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.756105+0000) 2020-02-06T02:14:54.757 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:54.755+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.756533+0000) 2020-02-06T02:14:54.758 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:54.756+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.757818+0000) 2020-02-06T02:14:54.759 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:54.757+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.758432+0000) 2020-02-06T02:14:54.759 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:54.757+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.758988+0000) 2020-02-06T02:14:54.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:54.758+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.759389+0000) 2020-02-06T02:14:54.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:54.758+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.759897+0000) 2020-02-06T02:14:54.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:54.758+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.759718+0000) 2020-02-06T02:14:54.761 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:54.758+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.760288+0000) 2020-02-06T02:14:54.761 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:54.759+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.760935+0000) 2020-02-06T02:14:54.762 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:54.759+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.761188+0000) 2020-02-06T02:14:54.762 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:54.760+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.761566+0000) 2020-02-06T02:14:54.763 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:54.761+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.762232+0000) 2020-02-06T02:14:54.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:54.766+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:54.767647+0000) 2020-02-06T02:14:54.782 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:54.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:14:55.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:55.355+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:55.356383+0000) 2020-02-06T02:14:56.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:56.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:56.179079+0000) 2020-02-06T02:14:56.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:56.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:56.542493+0000) 2020-02-06T02:14:57.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:57.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.156365+0000) 2020-02-06T02:14:57.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:57.429+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.430823+0000) 2020-02-06T02:14:57.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:57.429+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.431044+0000) 2020-02-06T02:14:57.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:57.430+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.431494+0000) 2020-02-06T02:14:57.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:57.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.661363+0000) 2020-02-06T02:14:57.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:57.683+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:57.684460+0000) 2020-02-06T02:14:59.786 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:14:59.784+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.785225+0000) 2020-02-06T02:14:59.788 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:14:59.786+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.787393+0000) 2020-02-06T02:14:59.788 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:14:59.785+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.787127+0000) 2020-02-06T02:14:59.788 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:14:59.786+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.787885+0000) 2020-02-06T02:14:59.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:59.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.788404+0000) 2020-02-06T02:14:59.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:59.788+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.789487+0000) 2020-02-06T02:14:59.790 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:14:59.788+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.789534+0000) 2020-02-06T02:14:59.790 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:14:59.788+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.789794+0000) 2020-02-06T02:14:59.791 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:14:59.789+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.790400+0000) 2020-02-06T02:14:59.791 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:14:59.789+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.790725+0000) 2020-02-06T02:14:59.792 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:14:59.789+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.790468+0000) 2020-02-06T02:14:59.792 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:14:59.790+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.791608+0000) 2020-02-06T02:14:59.792 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:14:59.790+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.791504+0000) 2020-02-06T02:14:59.800 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:59.793+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.794150+0000) 2020-02-06T02:14:59.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:14:59.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:14:59.800363+0000) 2020-02-06T02:14:59.810 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:14:59.808+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227158 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:00.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:00.045+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:00.046154+0000) 2020-02-06T02:15:00.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:00.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:00.657563+0000) 2020-02-06T02:15:00.935 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:00.933+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:00.934825+0000) 2020-02-06T02:15:03.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:03.514+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:03.515753+0000) 2020-02-06T02:15:04.815 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:04.813+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.814623+0000) 2020-02-06T02:15:04.817 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:04.814+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.816341+0000) 2020-02-06T02:15:04.818 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:04.816+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.817241+0000) 2020-02-06T02:15:04.821 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:04.817+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.819091+0000) 2020-02-06T02:15:04.822 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:04.818+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.820133+0000) 2020-02-06T02:15:04.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:04.816+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.817842+0000) 2020-02-06T02:15:04.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:04.817+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.818419+0000) 2020-02-06T02:15:04.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:04.818+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.819385+0000) 2020-02-06T02:15:04.822 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:04.818+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.819766+0000) 2020-02-06T02:15:04.823 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:04.819+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.820866+0000) 2020-02-06T02:15:04.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:04.819+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.821130+0000) 2020-02-06T02:15:04.823 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:04.820+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.821424+0000) 2020-02-06T02:15:04.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:04.819+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.821294+0000) 2020-02-06T02:15:04.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:04.824+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:04.825637+0000) 2020-02-06T02:15:04.840 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:04.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227196 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:05.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:05.363+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:05.364595+0000) 2020-02-06T02:15:06.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:06.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:06.179367+0000) 2020-02-06T02:15:06.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:06.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:06.542752+0000) 2020-02-06T02:15:07.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:07.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.156585+0000) 2020-02-06T02:15:07.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:07.430+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.431071+0000) 2020-02-06T02:15:07.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:07.430+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.431306+0000) 2020-02-06T02:15:07.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:07.430+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.431685+0000) 2020-02-06T02:15:07.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:07.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.661564+0000) 2020-02-06T02:15:07.685 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:07.684+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:07.684729+0000) 2020-02-06T02:15:09.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:09.799+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.800540+0000) 2020-02-06T02:15:09.845 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:09.843+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.844231+0000) 2020-02-06T02:15:09.845 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:09.843+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.844872+0000) 2020-02-06T02:15:09.847 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:09.844+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.846061+0000) 2020-02-06T02:15:09.847 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:09.844+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.846248+0000) 2020-02-06T02:15:09.847 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:09.845+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.846525+0000) 2020-02-06T02:15:09.849 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:09.847+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.848318+0000) 2020-02-06T02:15:09.849 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:09.847+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.848593+0000) 2020-02-06T02:15:09.849 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:09.847+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.848866+0000) 2020-02-06T02:15:09.851 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:09.848+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.850263+0000) 2020-02-06T02:15:09.851 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:09.849+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.850754+0000) 2020-02-06T02:15:09.852 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:09.850+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.851314+0000) 2020-02-06T02:15:09.852 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:09.850+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.851486+0000) 2020-02-06T02:15:09.852 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:09.850+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.851922+0000) 2020-02-06T02:15:09.858 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:09.856+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:09.857359+0000) 2020-02-06T02:15:09.871 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:09.868+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227241 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:10.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:10.045+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:10.046367+0000) 2020-02-06T02:15:10.658 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:10.657+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:10.657788+0000) 2020-02-06T02:15:10.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:10.934+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:10.935068+0000) 2020-02-06T02:15:13.516 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:13.515+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:13.515981+0000) 2020-02-06T02:15:14.875 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:14.873+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.874993+0000) 2020-02-06T02:15:14.876 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:14.874+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.875627+0000) 2020-02-06T02:15:14.876 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:14.875+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.876224+0000) 2020-02-06T02:15:14.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:14.876+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.877226+0000) 2020-02-06T02:15:14.878 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:14.876+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.877986+0000) 2020-02-06T02:15:14.879 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:14.877+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.878556+0000) 2020-02-06T02:15:14.880 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:14.878+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.879598+0000) 2020-02-06T02:15:14.881 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:14.879+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.880479+0000) 2020-02-06T02:15:14.881 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:14.878+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.880065+0000) 2020-02-06T02:15:14.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:14.879+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.880937+0000) 2020-02-06T02:15:14.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:14.880+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.881567+0000) 2020-02-06T02:15:14.883 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:14.881+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.882422+0000) 2020-02-06T02:15:14.883 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:14.881+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.882371+0000) 2020-02-06T02:15:14.884 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:14.883+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:14.884169+0000) 2020-02-06T02:15:14.901 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:14.898+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:15.211 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:15:15.217 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:15:15.373 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:15.371+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:15.372562+0000) 2020-02-06T02:15:16.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:16.178+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:16.179673+0000) 2020-02-06T02:15:16.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:16.541+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:16.542952+0000) 2020-02-06T02:15:17.157 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:17.155+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.156833+0000) 2020-02-06T02:15:17.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:17.430+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.431307+0000) 2020-02-06T02:15:17.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:17.430+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.431616+0000) 2020-02-06T02:15:17.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:17.430+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.431945+0000) 2020-02-06T02:15:17.662 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:17.660+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.661739+0000) 2020-02-06T02:15:17.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:17.684+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:17.685004+0000) 2020-02-06T02:15:19.801 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:19.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.800779+0000) 2020-02-06T02:15:19.909 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:19.907+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.908320+0000) 2020-02-06T02:15:19.909 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:19.906+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.908253+0000) 2020-02-06T02:15:19.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:19.907+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.908917+0000) 2020-02-06T02:15:19.910 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:19.907+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.909337+0000) 2020-02-06T02:15:19.911 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:19.908+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.910047+0000) 2020-02-06T02:15:19.911 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:19.909+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.910344+0000) 2020-02-06T02:15:19.912 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:19.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.910596+0000) 2020-02-06T02:15:19.912 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:19.909+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.910941+0000) 2020-02-06T02:15:19.912 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:19.910+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.911469+0000) 2020-02-06T02:15:19.913 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:19.910+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.911861+0000) 2020-02-06T02:15:19.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:19.909+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.911388+0000) 2020-02-06T02:15:19.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:19.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.912962+0000) 2020-02-06T02:15:19.914 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:19.910+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.912037+0000) 2020-02-06T02:15:19.919 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:19.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:19.919095+0000) 2020-02-06T02:15:19.930 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:19.928+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:20.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:20.046+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:20.046579+0000) 2020-02-06T02:15:20.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:20.656+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:20.658003+0000) 2020-02-06T02:15:20.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:20.934+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:20.935370+0000) 2020-02-06T02:15:23.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:23.515+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:23.516213+0000) 2020-02-06T02:15:24.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:24.934+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.935207+0000) 2020-02-06T02:15:24.936 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:24.934+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.935928+0000) 2020-02-06T02:15:24.937 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:24.935+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.936893+0000) 2020-02-06T02:15:24.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:24.935+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.936711+0000) 2020-02-06T02:15:24.938 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:24.937+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.938212+0000) 2020-02-06T02:15:24.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:24.936+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.938387+0000) 2020-02-06T02:15:24.939 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:24.937+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.938675+0000) 2020-02-06T02:15:24.940 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:24.938+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.939388+0000) 2020-02-06T02:15:24.940 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:24.937+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.939211+0000) 2020-02-06T02:15:24.941 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:24.939+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.940616+0000) 2020-02-06T02:15:24.942 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:24.939+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.941099+0000) 2020-02-06T02:15:24.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:24.940+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.941448+0000) 2020-02-06T02:15:24.942 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:24.940+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.941803+0000) 2020-02-06T02:15:24.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:24.945+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:24.946833+0000) 2020-02-06T02:15:24.959 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:24.957+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227362 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:25.381 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:25.380+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:25.380589+0000) 2020-02-06T02:15:26.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:26.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:26.179980+0000) 2020-02-06T02:15:26.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:26.542+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:26.543190+0000) 2020-02-06T02:15:27.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:27.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.157138+0000) 2020-02-06T02:15:27.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:27.431+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.431565+0000) 2020-02-06T02:15:27.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:27.431+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.431834+0000) 2020-02-06T02:15:27.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:27.431+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.432175+0000) 2020-02-06T02:15:27.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:27.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.661957+0000) 2020-02-06T02:15:27.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:27.684+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:27.685309+0000) 2020-02-06T02:15:29.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:29.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.801063+0000) 2020-02-06T02:15:29.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:29.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.964416+0000) 2020-02-06T02:15:29.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:29.964+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.966213+0000) 2020-02-06T02:15:29.967 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:29.965+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.966469+0000) 2020-02-06T02:15:29.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:29.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.966954+0000) 2020-02-06T02:15:29.968 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:29.966+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.967450+0000) 2020-02-06T02:15:29.969 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:29.966+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.967815+0000) 2020-02-06T02:15:29.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:29.967+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.968128+0000) 2020-02-06T02:15:29.969 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:29.966+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.968362+0000) 2020-02-06T02:15:29.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:29.967+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.968794+0000) 2020-02-06T02:15:29.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:29.968+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.969580+0000) 2020-02-06T02:15:29.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:29.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.969880+0000) 2020-02-06T02:15:29.971 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:29.968+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.969766+0000) 2020-02-06T02:15:29.971 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:29.969+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.970337+0000) 2020-02-06T02:15:29.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:29.974+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:29.975842+0000) 2020-02-06T02:15:29.988 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:29.986+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227407 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:30.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:30.046+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:30.046794+0000) 2020-02-06T02:15:30.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:30.657+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:30.658219+0000) 2020-02-06T02:15:30.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:30.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:30.935572+0000) 2020-02-06T02:15:33.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:33.516+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:33.516439+0000) 2020-02-06T02:15:34.995 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:34.993+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.994942+0000) 2020-02-06T02:15:34.996 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:34.995+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.995524+0000) 2020-02-06T02:15:34.997 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:34.994+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.995599+0000) 2020-02-06T02:15:34.998 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:34.995+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.996118+0000) 2020-02-06T02:15:34.998 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:34.995+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.996674+0000) 2020-02-06T02:15:34.998 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:34.996+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.997203+0000) 2020-02-06T02:15:34.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:34.996+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.997827+0000) 2020-02-06T02:15:34.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:34.997+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.998240+0000) 2020-02-06T02:15:35.000 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:34.997+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.998210+0000) 2020-02-06T02:15:35.000 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:34.998+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.999200+0000) 2020-02-06T02:15:35.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:34.998+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.999676+0000) 2020-02-06T02:15:35.001 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:34.999+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.999765+0000) 2020-02-06T02:15:35.001 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:34.999+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:34.999918+0000) 2020-02-06T02:15:35.005 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:35.003+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:35.004613+0000) 2020-02-06T02:15:35.016 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:35.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:35.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:35.389+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:35.389506+0000) 2020-02-06T02:15:36.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:36.179+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:36.180357+0000) 2020-02-06T02:15:36.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:36.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:36.543419+0000) 2020-02-06T02:15:37.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:37.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.157376+0000) 2020-02-06T02:15:37.432 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:37.431+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.431798+0000) 2020-02-06T02:15:37.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:37.431+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.432049+0000) 2020-02-06T02:15:37.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:37.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.432414+0000) 2020-02-06T02:15:37.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:37.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.662186+0000) 2020-02-06T02:15:37.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:37.685+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:37.685545+0000) 2020-02-06T02:15:39.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:39.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:39.801415+0000) 2020-02-06T02:15:40.022 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:40.020+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.020976+0000) 2020-02-06T02:15:40.022 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:40.021+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.021531+0000) 2020-02-06T02:15:40.023 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:40.021+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.022042+0000) 2020-02-06T02:15:40.024 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:40.022+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.022763+0000) 2020-02-06T02:15:40.024 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:40.023+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.023201+0000) 2020-02-06T02:15:40.025 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:40.022+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.022724+0000) 2020-02-06T02:15:40.025 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:40.023+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.023419+0000) 2020-02-06T02:15:40.025 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:40.024+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.024609+0000) 2020-02-06T02:15:40.026 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:40.023+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.024125+0000) 2020-02-06T02:15:40.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:40.025+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.025328+0000) 2020-02-06T02:15:40.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:40.030+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.030804+0000) 2020-02-06T02:15:40.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:40.025+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.025817+0000) 2020-02-06T02:15:40.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:40.026+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.026951+0000) 2020-02-06T02:15:40.032 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:40.026+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.026991+0000) 2020-02-06T02:15:40.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:40.046+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.047008+0000) 2020-02-06T02:15:40.048 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:40.047+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:40.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:40.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.658480+0000) 2020-02-06T02:15:40.936 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:40.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:40.935802+0000) 2020-02-06T02:15:43.517 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:43.516+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:43.516620+0000) 2020-02-06T02:15:45.054 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:45.052+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.053717+0000) 2020-02-06T02:15:45.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:45.053+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.054676+0000) 2020-02-06T02:15:45.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:45.054+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.055123+0000) 2020-02-06T02:15:45.056 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:45.054+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.056060+0000) 2020-02-06T02:15:45.057 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:45.055+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.056484+0000) 2020-02-06T02:15:45.057 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:45.056+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.056653+0000) 2020-02-06T02:15:45.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:45.056+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.056930+0000) 2020-02-06T02:15:45.058 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:45.056+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.057546+0000) 2020-02-06T02:15:45.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:45.057+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.058377+0000) 2020-02-06T02:15:45.059 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:45.057+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.058403+0000) 2020-02-06T02:15:45.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:45.058+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.058758+0000) 2020-02-06T02:15:45.060 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:45.058+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.059011+0000) 2020-02-06T02:15:45.060 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:45.058+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.059312+0000) 2020-02-06T02:15:45.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:45.064+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.065520+0000) 2020-02-06T02:15:45.077 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:45.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227527 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:45.362 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:15:45.368 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:15:45.402 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:45.400+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:45.401158+0000) 2020-02-06T02:15:46.181 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:46.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:46.180659+0000) 2020-02-06T02:15:46.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:46.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:46.543723+0000) 2020-02-06T02:15:47.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:47.156+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.157607+0000) 2020-02-06T02:15:47.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:47.431+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.432037+0000) 2020-02-06T02:15:47.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:47.431+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.432281+0000) 2020-02-06T02:15:47.433 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:47.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.432689+0000) 2020-02-06T02:15:47.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:47.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.662417+0000) 2020-02-06T02:15:47.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:47.684+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:47.685784+0000) 2020-02-06T02:15:49.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:49.800+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:49.801663+0000) 2020-02-06T02:15:50.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:50.046+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.047190+0000) 2020-02-06T02:15:50.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:50.080+0000 7f2efc2ac700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.081250+0000) 2020-02-06T02:15:50.083 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:50.081+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.082728+0000) 2020-02-06T02:15:50.085 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:50.082+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.083395+0000) 2020-02-06T02:15:50.085 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:50.082+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.083947+0000) 2020-02-06T02:15:50.086 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:50.084+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.084727+0000) 2020-02-06T02:15:50.086 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:50.084+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.085048+0000) 2020-02-06T02:15:50.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:50.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.085612+0000) 2020-02-06T02:15:50.087 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:50.086+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.086515+0000) 2020-02-06T02:15:50.088 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:50.086+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.087206+0000) 2020-02-06T02:15:50.088 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:50.087+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.087788+0000) 2020-02-06T02:15:50.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:50.087+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.088286+0000) 2020-02-06T02:15:50.089 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:50.087+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.088193+0000) 2020-02-06T02:15:50.089 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:50.088+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.088465+0000) 2020-02-06T02:15:50.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:50.096+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.097160+0000) 2020-02-06T02:15:50.107 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:50.105+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227570 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:50.659 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:50.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.658717+0000) 2020-02-06T02:15:50.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:50.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:50.936009+0000) 2020-02-06T02:15:53.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:53.515+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:53.516854+0000) 2020-02-06T02:15:55.128 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:15:55.110+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.111876+0000) 2020-02-06T02:15:55.128 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:55.111+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.112714+0000) 2020-02-06T02:15:55.129 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:55.112+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.113300+0000) 2020-02-06T02:15:55.129 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:55.113+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.113629+0000) 2020-02-06T02:15:55.130 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:55.113+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.114039+0000) 2020-02-06T02:15:55.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:15:55.114+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.114624+0000) 2020-02-06T02:15:55.130 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:55.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.115782+0000) 2020-02-06T02:15:55.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:55.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.116428+0000) 2020-02-06T02:15:55.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:15:55.116+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.117403+0000) 2020-02-06T02:15:55.132 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:55.116+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.117556+0000) 2020-02-06T02:15:55.133 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:55.117+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.117739+0000) 2020-02-06T02:15:55.133 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:55.117+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.117994+0000) 2020-02-06T02:15:55.133 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:55.117+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.118412+0000) 2020-02-06T02:15:55.141 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:55.122+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.123458+0000) 2020-02-06T02:15:55.155 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:15:55.135+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:15:55.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:55.406+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:55.407597+0000) 2020-02-06T02:15:56.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:15:56.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:56.180934+0000) 2020-02-06T02:15:56.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:15:56.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:56.543964+0000) 2020-02-06T02:15:57.158 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:15:57.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.157882+0000) 2020-02-06T02:15:57.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:15:57.432+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.432273+0000) 2020-02-06T02:15:57.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:15:57.432+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.432518+0000) 2020-02-06T02:15:57.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:15:57.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.432906+0000) 2020-02-06T02:15:57.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:15:57.661+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.662711+0000) 2020-02-06T02:15:57.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:15:57.685+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:57.686011+0000) 2020-02-06T02:15:59.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:15:59.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:15:59.801945+0000) 2020-02-06T02:16:00.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:00.046+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.047494+0000) 2020-02-06T02:16:00.142 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:00.140+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.141811+0000) 2020-02-06T02:16:00.143 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:00.141+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.142417+0000) 2020-02-06T02:16:00.144 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:00.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.143442+0000) 2020-02-06T02:16:00.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:00.143+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.144804+0000) 2020-02-06T02:16:00.146 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:00.144+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.144585+0000) 2020-02-06T02:16:00.146 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:00.144+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.145106+0000) 2020-02-06T02:16:00.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:00.145+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.145744+0000) 2020-02-06T02:16:00.147 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:00.144+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.146021+0000) 2020-02-06T02:16:00.147 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:00.145+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.146306+0000) 2020-02-06T02:16:00.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:00.145+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.146728+0000) 2020-02-06T02:16:00.148 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:00.147+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.147580+0000) 2020-02-06T02:16:00.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:00.146+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.147116+0000) 2020-02-06T02:16:00.149 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:00.148+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.148415+0000) 2020-02-06T02:16:00.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:00.152+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.153769+0000) 2020-02-06T02:16:00.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:00.167+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227652 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:00.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:00.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.658969+0000) 2020-02-06T02:16:00.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:00.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:00.936164+0000) 2020-02-06T02:16:03.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:03.516+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:03.517112+0000) 2020-02-06T02:16:05.173 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:05.171+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.172715+0000) 2020-02-06T02:16:05.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:05.172+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.173863+0000) 2020-02-06T02:16:05.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:05.174+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.174565+0000) 2020-02-06T02:16:05.176 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:05.174+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.175204+0000) 2020-02-06T02:16:05.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:05.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.175676+0000) 2020-02-06T02:16:05.177 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:05.175+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.176321+0000) 2020-02-06T02:16:05.177 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:05.176+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.176773+0000) 2020-02-06T02:16:05.178 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:05.176+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.177110+0000) 2020-02-06T02:16:05.178 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:05.176+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.177374+0000) 2020-02-06T02:16:05.179 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:05.177+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.178244+0000) 2020-02-06T02:16:05.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:05.178+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.179056+0000) 2020-02-06T02:16:05.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:05.178+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.179463+0000) 2020-02-06T02:16:05.180 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:05.179+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.179650+0000) 2020-02-06T02:16:05.182 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:05.180+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.181492+0000) 2020-02-06T02:16:05.196 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:05.195+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227691 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:05.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:05.414+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:05.415815+0000) 2020-02-06T02:16:06.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:06.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:06.181147+0000) 2020-02-06T02:16:06.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:06.544+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:06.544214+0000) 2020-02-06T02:16:07.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:07.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.158177+0000) 2020-02-06T02:16:07.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:07.432+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.432557+0000) 2020-02-06T02:16:07.433 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:07.432+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.432734+0000) 2020-02-06T02:16:07.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:07.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.433119+0000) 2020-02-06T02:16:07.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:07.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.662937+0000) 2020-02-06T02:16:07.687 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:07.685+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:07.686262+0000) 2020-02-06T02:16:09.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:09.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:09.802143+0000) 2020-02-06T02:16:10.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:10.047+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.047728+0000) 2020-02-06T02:16:10.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:10.200+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.201630+0000) 2020-02-06T02:16:10.203 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:10.201+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.202336+0000) 2020-02-06T02:16:10.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:10.201+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.202684+0000) 2020-02-06T02:16:10.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:10.202+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.203145+0000) 2020-02-06T02:16:10.204 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:10.203+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.203698+0000) 2020-02-06T02:16:10.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:10.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.204150+0000) 2020-02-06T02:16:10.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:10.204+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.204616+0000) 2020-02-06T02:16:10.206 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:10.205+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.205705+0000) 2020-02-06T02:16:10.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:10.205+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.206238+0000) 2020-02-06T02:16:10.214 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:10.206+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.207637+0000) 2020-02-06T02:16:10.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:10.211+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.212250+0000) 2020-02-06T02:16:10.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:10.206+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.207305+0000) 2020-02-06T02:16:10.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:10.207+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.207470+0000) 2020-02-06T02:16:10.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:10.207+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.207769+0000) 2020-02-06T02:16:10.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:10.227+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227734 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:10.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:10.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.659222+0000) 2020-02-06T02:16:10.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:10.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:10.936404+0000) 2020-02-06T02:16:13.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:13.516+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:13.517355+0000) 2020-02-06T02:16:15.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:15.232+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.233604+0000) 2020-02-06T02:16:15.238 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:15.234+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.235497+0000) 2020-02-06T02:16:15.238 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:15.233+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.234796+0000) 2020-02-06T02:16:15.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:15.234+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.235888+0000) 2020-02-06T02:16:15.238 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:15.235+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.236462+0000) 2020-02-06T02:16:15.238 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:15.236+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.237147+0000) 2020-02-06T02:16:15.239 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:15.236+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.237302+0000) 2020-02-06T02:16:15.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:15.236+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.238007+0000) 2020-02-06T02:16:15.240 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:15.237+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.238751+0000) 2020-02-06T02:16:15.240 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:15.237+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.239038+0000) 2020-02-06T02:16:15.240 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:15.236+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.237699+0000) 2020-02-06T02:16:15.241 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:15.238+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.239646+0000) 2020-02-06T02:16:15.241 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:15.238+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.240197+0000) 2020-02-06T02:16:15.246 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:15.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.246099+0000) 2020-02-06T02:16:15.257 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:15.255+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227774 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:15.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:15.422+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:15.423215+0000) 2020-02-06T02:16:15.505 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:16:15.512 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:16:16.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:16.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:16.181488+0000) 2020-02-06T02:16:16.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:16.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:16.544502+0000) 2020-02-06T02:16:17.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:17.157+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.158406+0000) 2020-02-06T02:16:17.433 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:17.431+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.432798+0000) 2020-02-06T02:16:17.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:17.431+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.432986+0000) 2020-02-06T02:16:17.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:17.431+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.433352+0000) 2020-02-06T02:16:17.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:17.662+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.663170+0000) 2020-02-06T02:16:17.687 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:17.686+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:17.686474+0000) 2020-02-06T02:16:19.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:19.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:19.802374+0000) 2020-02-06T02:16:20.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:20.047+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.047989+0000) 2020-02-06T02:16:20.263 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:20.261+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.262873+0000) 2020-02-06T02:16:20.264 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:20.261+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.263363+0000) 2020-02-06T02:16:20.264 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:20.262+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.263897+0000) 2020-02-06T02:16:20.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:20.262+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.264264+0000) 2020-02-06T02:16:20.267 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:20.264+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.265069+0000) 2020-02-06T02:16:20.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:20.264+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.265534+0000) 2020-02-06T02:16:20.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:20.264+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.265942+0000) 2020-02-06T02:16:20.268 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:20.264+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.266180+0000) 2020-02-06T02:16:20.268 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:20.265+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.267046+0000) 2020-02-06T02:16:20.269 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:20.266+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.267217+0000) 2020-02-06T02:16:20.269 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:20.266+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.267820+0000) 2020-02-06T02:16:20.270 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:20.267+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.268389+0000) 2020-02-06T02:16:20.270 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:20.267+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.268750+0000) 2020-02-06T02:16:20.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:20.273+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.274140+0000) 2020-02-06T02:16:20.286 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:20.284+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:20.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:20.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.659495+0000) 2020-02-06T02:16:20.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:20.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:20.936610+0000) 2020-02-06T02:16:23.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:23.516+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:23.517611+0000) 2020-02-06T02:16:25.292 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:25.290+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.291413+0000) 2020-02-06T02:16:25.292 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:25.291+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.292055+0000) 2020-02-06T02:16:25.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:25.291+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.292837+0000) 2020-02-06T02:16:25.294 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:25.292+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.293351+0000) 2020-02-06T02:16:25.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:25.292+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.293786+0000) 2020-02-06T02:16:25.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:25.293+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.294764+0000) 2020-02-06T02:16:25.296 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:25.294+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.295589+0000) 2020-02-06T02:16:25.297 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:25.294+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.296191+0000) 2020-02-06T02:16:25.298 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:25.295+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.297406+0000) 2020-02-06T02:16:25.298 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:25.296+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.297583+0000) 2020-02-06T02:16:25.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:25.296+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.297459+0000) 2020-02-06T02:16:25.299 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:25.296+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.297892+0000) 2020-02-06T02:16:25.299 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:25.297+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.298529+0000) 2020-02-06T02:16:25.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:25.300+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.301796+0000) 2020-02-06T02:16:25.318 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:25.316+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227855 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:25.432 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:25.431+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:25.431869+0000) 2020-02-06T02:16:26.182 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:26.180+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:26.181739+0000) 2020-02-06T02:16:26.545 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:26.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:26.544785+0000) 2020-02-06T02:16:27.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:27.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.158637+0000) 2020-02-06T02:16:27.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:27.432+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.433048+0000) 2020-02-06T02:16:27.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:27.432+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.433225+0000) 2020-02-06T02:16:27.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:27.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.433598+0000) 2020-02-06T02:16:27.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:27.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.663401+0000) 2020-02-06T02:16:27.687 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:27.686+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:27.686698+0000) 2020-02-06T02:16:29.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:29.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:29.802608+0000) 2020-02-06T02:16:30.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:30.048+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.048252+0000) 2020-02-06T02:16:30.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:30.321+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.322879+0000) 2020-02-06T02:16:30.325 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:30.322+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.323842+0000) 2020-02-06T02:16:30.325 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:30.323+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.324879+0000) 2020-02-06T02:16:30.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:30.324+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.325644+0000) 2020-02-06T02:16:30.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:30.325+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.326143+0000) 2020-02-06T02:16:30.327 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:30.325+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.326554+0000) 2020-02-06T02:16:30.327 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:30.325+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.327211+0000) 2020-02-06T02:16:30.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:30.326+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.327504+0000) 2020-02-06T02:16:30.329 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:30.327+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.328303+0000) 2020-02-06T02:16:30.329 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:30.327+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.328942+0000) 2020-02-06T02:16:30.330 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:30.327+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.329284+0000) 2020-02-06T02:16:30.330 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:30.328+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.329989+0000) 2020-02-06T02:16:30.331 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:30.328+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.329767+0000) 2020-02-06T02:16:30.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:30.333+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.334294+0000) 2020-02-06T02:16:30.348 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:30.346+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:30.660 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:30.658+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.659728+0000) 2020-02-06T02:16:30.937 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:30.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:30.936842+0000) 2020-02-06T02:16:33.518 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:33.517+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:33.517878+0000) 2020-02-06T02:16:35.355 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:35.354+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.354564+0000) 2020-02-06T02:16:35.356 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:35.353+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.355308+0000) 2020-02-06T02:16:35.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:35.355+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.355670+0000) 2020-02-06T02:16:35.357 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:35.355+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.356046+0000) 2020-02-06T02:16:35.357 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:35.355+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.356836+0000) 2020-02-06T02:16:35.358 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:35.355+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.356840+0000) 2020-02-06T02:16:35.358 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:35.356+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.356490+0000) 2020-02-06T02:16:35.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:35.356+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.356526+0000) 2020-02-06T02:16:35.359 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:35.356+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.357807+0000) 2020-02-06T02:16:35.359 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:35.357+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.357828+0000) 2020-02-06T02:16:35.359 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:35.358+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.358352+0000) 2020-02-06T02:16:35.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:35.357+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.358511+0000) 2020-02-06T02:16:35.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:35.356+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.358062+0000) 2020-02-06T02:16:35.366 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:35.365+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.365707+0000) 2020-02-06T02:16:35.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:35.375+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227939 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:35.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:35.438+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:35.438990+0000) 2020-02-06T02:16:36.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:36.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:36.181955+0000) 2020-02-06T02:16:36.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:36.544+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:36.545045+0000) 2020-02-06T02:16:37.159 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:37.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.158868+0000) 2020-02-06T02:16:37.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:37.432+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.433397+0000) 2020-02-06T02:16:37.434 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:37.432+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.433488+0000) 2020-02-06T02:16:37.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:37.432+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.433824+0000) 2020-02-06T02:16:37.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:37.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.663660+0000) 2020-02-06T02:16:37.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:37.686+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:37.686936+0000) 2020-02-06T02:16:39.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:39.801+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:39.802838+0000) 2020-02-06T02:16:40.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:40.047+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.048480+0000) 2020-02-06T02:16:40.384 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:40.382+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.383424+0000) 2020-02-06T02:16:40.384 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:40.381+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.382866+0000) 2020-02-06T02:16:40.385 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:40.382+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.383974+0000) 2020-02-06T02:16:40.385 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:40.383+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.385003+0000) 2020-02-06T02:16:40.386 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:40.384+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.385878+0000) 2020-02-06T02:16:40.387 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:40.384+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.385909+0000) 2020-02-06T02:16:40.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:40.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.386429+0000) 2020-02-06T02:16:40.387 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:40.385+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.386755+0000) 2020-02-06T02:16:40.388 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:40.386+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.387378+0000) 2020-02-06T02:16:40.388 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:40.386+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.387920+0000) 2020-02-06T02:16:40.389 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:40.387+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.388575+0000) 2020-02-06T02:16:40.389 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:40.387+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.389058+0000) 2020-02-06T02:16:40.390 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:40.387+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.389276+0000) 2020-02-06T02:16:40.394 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:40.392+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.393657+0000) 2020-02-06T02:16:40.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:40.407+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 227980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:40.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:40.659+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.659955+0000) 2020-02-06T02:16:40.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:40.935+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:40.937048+0000) 2020-02-06T02:16:43.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:43.517+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:43.518134+0000) 2020-02-06T02:16:45.415 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:45.413+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.414116+0000) 2020-02-06T02:16:45.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:45.413+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.414665+0000) 2020-02-06T02:16:45.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:45.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.418565+0000) 2020-02-06T02:16:45.423 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:45.418+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.419122+0000) 2020-02-06T02:16:45.423 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:45.418+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.419585+0000) 2020-02-06T02:16:45.424 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:45.419+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.420337+0000) 2020-02-06T02:16:45.424 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:45.419+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.420812+0000) 2020-02-06T02:16:45.424 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:45.414+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.415797+0000) 2020-02-06T02:16:45.424 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:45.413+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.415338+0000) 2020-02-06T02:16:45.425 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:45.415+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.416752+0000) 2020-02-06T02:16:45.425 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:45.415+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.417389+0000) 2020-02-06T02:16:45.425 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:45.416+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.417772+0000) 2020-02-06T02:16:45.425 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:45.418+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.419972+0000) 2020-02-06T02:16:45.428 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:45.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.426026+0000) 2020-02-06T02:16:45.440 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:45.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228020 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:45.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:45.443+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:45.444096+0000) 2020-02-06T02:16:45.678 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:16:45.685 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:16:46.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:46.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:46.182262+0000) 2020-02-06T02:16:46.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:46.543+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:46.545304+0000) 2020-02-06T02:16:47.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:47.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.159150+0000) 2020-02-06T02:16:47.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:47.433+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.433750+0000) 2020-02-06T02:16:47.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:47.433+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.433753+0000) 2020-02-06T02:16:47.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:47.433+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.434071+0000) 2020-02-06T02:16:47.664 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:47.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.663892+0000) 2020-02-06T02:16:47.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:47.686+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:47.687166+0000) 2020-02-06T02:16:49.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:49.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:49.803109+0000) 2020-02-06T02:16:50.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:50.047+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.048734+0000) 2020-02-06T02:16:50.445 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:50.442+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.444261+0000) 2020-02-06T02:16:50.446 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:50.443+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.444934+0000) 2020-02-06T02:16:50.446 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:50.444+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.445879+0000) 2020-02-06T02:16:50.447 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:50.445+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.446516+0000) 2020-02-06T02:16:50.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:50.446+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.447623+0000) 2020-02-06T02:16:50.449 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:50.446+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.447986+0000) 2020-02-06T02:16:50.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:50.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.448212+0000) 2020-02-06T02:16:50.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:50.446+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.448339+0000) 2020-02-06T02:16:50.450 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:50.447+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.449044+0000) 2020-02-06T02:16:50.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:50.448+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.449497+0000) 2020-02-06T02:16:50.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:50.448+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.449944+0000) 2020-02-06T02:16:50.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:50.449+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.450441+0000) 2020-02-06T02:16:50.455 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:50.453+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.455047+0000) 2020-02-06T02:16:50.456 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:50.454+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.455414+0000) 2020-02-06T02:16:50.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:50.466+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228061 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:50.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:50.659+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.660208+0000) 2020-02-06T02:16:50.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:50.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:50.937302+0000) 2020-02-06T02:16:53.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:53.517+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:53.518363+0000) 2020-02-06T02:16:55.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:55.450+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.451145+0000) 2020-02-06T02:16:55.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:55.471+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.472669+0000) 2020-02-06T02:16:55.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:55.472+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.474368+0000) 2020-02-06T02:16:55.475 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:55.473+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.474730+0000) 2020-02-06T02:16:55.476 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:55.473+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.474923+0000) 2020-02-06T02:16:55.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:55.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.475430+0000) 2020-02-06T02:16:55.477 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:16:55.474+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.475801+0000) 2020-02-06T02:16:55.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:55.474+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.476051+0000) 2020-02-06T02:16:55.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:55.475+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.476232+0000) 2020-02-06T02:16:55.478 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:16:55.475+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.476970+0000) 2020-02-06T02:16:55.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:16:55.475+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.476838+0000) 2020-02-06T02:16:55.478 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:55.476+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.477857+0000) 2020-02-06T02:16:55.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:55.476+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.477504+0000) 2020-02-06T02:16:55.479 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:55.477+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.479298+0000) 2020-02-06T02:16:55.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:55.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:55.482553+0000) 2020-02-06T02:16:55.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:16:55.496+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:16:56.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:16:56.181+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:56.182499+0000) 2020-02-06T02:16:56.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:16:56.544+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:56.545553+0000) 2020-02-06T02:16:57.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:16:57.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.159410+0000) 2020-02-06T02:16:57.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:16:57.433+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.434048+0000) 2020-02-06T02:16:57.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:16:57.433+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.434040+0000) 2020-02-06T02:16:57.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:16:57.433+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.434319+0000) 2020-02-06T02:16:57.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:16:57.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.664123+0000) 2020-02-06T02:16:57.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:16:57.686+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:57.687415+0000) 2020-02-06T02:16:59.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:16:59.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:16:59.803347+0000) 2020-02-06T02:17:00.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:00.047+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.048983+0000) 2020-02-06T02:17:00.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:00.503+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.504569+0000) 2020-02-06T02:17:00.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:00.506+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.507981+0000) 2020-02-06T02:17:00.512 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:00.504+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.505328+0000) 2020-02-06T02:17:00.512 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:00.506+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.507518+0000) 2020-02-06T02:17:00.512 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:00.506+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.508034+0000) 2020-02-06T02:17:00.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:00.507+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.508533+0000) 2020-02-06T02:17:00.512 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:00.507+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.508789+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:00.502+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.504032+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:00.503+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.505033+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:00.505+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.507345+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:00.506+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.508032+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:00.506+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.508278+0000) 2020-02-06T02:17:00.513 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:00.507+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.509297+0000) 2020-02-06T02:17:00.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:00.514+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.515459+0000) 2020-02-06T02:17:00.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:00.527+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:00.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:00.659+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.660420+0000) 2020-02-06T02:17:00.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:00.936+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:00.937506+0000) 2020-02-06T02:17:03.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:03.517+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:03.518597+0000) 2020-02-06T02:17:05.460 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:05.458+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.459171+0000) 2020-02-06T02:17:05.534 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:05.532+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.533667+0000) 2020-02-06T02:17:05.535 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:05.533+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.534317+0000) 2020-02-06T02:17:05.536 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:05.533+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.534712+0000) 2020-02-06T02:17:05.536 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:05.534+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.535721+0000) 2020-02-06T02:17:05.537 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:05.535+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.535785+0000) 2020-02-06T02:17:05.537 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:05.535+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.537033+0000) 2020-02-06T02:17:05.538 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:05.537+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.538122+0000) 2020-02-06T02:17:05.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:05.538+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.538565+0000) 2020-02-06T02:17:05.540 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:05.538+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.539414+0000) 2020-02-06T02:17:05.540 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:05.538+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.539405+0000) 2020-02-06T02:17:05.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:05.538+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.539951+0000) 2020-02-06T02:17:05.541 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:05.540+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.540631+0000) 2020-02-06T02:17:05.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:05.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.545321+0000) 2020-02-06T02:17:05.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:05.544+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:05.545562+0000) 2020-02-06T02:17:05.559 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:05.557+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:06.183 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:06.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:06.182742+0000) 2020-02-06T02:17:06.546 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:06.545+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:06.545792+0000) 2020-02-06T02:17:07.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:07.158+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.159672+0000) 2020-02-06T02:17:07.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:07.433+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.434285+0000) 2020-02-06T02:17:07.435 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:07.433+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.434289+0000) 2020-02-06T02:17:07.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:07.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.434624+0000) 2020-02-06T02:17:07.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:07.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.664359+0000) 2020-02-06T02:17:07.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:07.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:07.687640+0000) 2020-02-06T02:17:09.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:09.802+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:09.803570+0000) 2020-02-06T02:17:10.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:10.048+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.049212+0000) 2020-02-06T02:17:10.564 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:10.562+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.563827+0000) 2020-02-06T02:17:10.565 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:10.564+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.564476+0000) 2020-02-06T02:17:10.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:10.564+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.564657+0000) 2020-02-06T02:17:10.566 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:10.564+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.565102+0000) 2020-02-06T02:17:10.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:10.566+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.567340+0000) 2020-02-06T02:17:10.568 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:10.566+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.567724+0000) 2020-02-06T02:17:10.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:10.567+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.567803+0000) 2020-02-06T02:17:10.569 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:10.567+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.568262+0000) 2020-02-06T02:17:10.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:10.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.569233+0000) 2020-02-06T02:17:10.570 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:10.568+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.568894+0000) 2020-02-06T02:17:10.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:10.569+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.569420+0000) 2020-02-06T02:17:10.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:10.568+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.569598+0000) 2020-02-06T02:17:10.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:10.569+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.570185+0000) 2020-02-06T02:17:10.574 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:10.572+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.573896+0000) 2020-02-06T02:17:10.587 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:10.586+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228225 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:10.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:10.660+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.660691+0000) 2020-02-06T02:17:10.938 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:10.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:10.937727+0000) 2020-02-06T02:17:13.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:13.517+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:13.518844+0000) 2020-02-06T02:17:15.472 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:15.471+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.471525+0000) 2020-02-06T02:17:15.593 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:15.591+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.592239+0000) 2020-02-06T02:17:15.594 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:15.592+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.593222+0000) 2020-02-06T02:17:15.594 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:15.592+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.593624+0000) 2020-02-06T02:17:15.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:15.592+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.593940+0000) 2020-02-06T02:17:15.595 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:15.593+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.594542+0000) 2020-02-06T02:17:15.595 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:15.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.595002+0000) 2020-02-06T02:17:15.596 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:15.594+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.595259+0000) 2020-02-06T02:17:15.596 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:15.595+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.595508+0000) 2020-02-06T02:17:15.597 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:15.595+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.595819+0000) 2020-02-06T02:17:15.597 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:15.596+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.596479+0000) 2020-02-06T02:17:15.598 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:15.595+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.596903+0000) 2020-02-06T02:17:15.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:15.598+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.598941+0000) 2020-02-06T02:17:15.600 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:15.598+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.599054+0000) 2020-02-06T02:17:15.605 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:15.603+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:15.604289+0000) 2020-02-06T02:17:15.618 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:15.616+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228264 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:15.820 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:17:15.826 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:17:16.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:16.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:16.182991+0000) 2020-02-06T02:17:16.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:16.545+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:16.546014+0000) 2020-02-06T02:17:17.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:17.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.159904+0000) 2020-02-06T02:17:17.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:17.434+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.434535+0000) 2020-02-06T02:17:17.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:17.434+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.434554+0000) 2020-02-06T02:17:17.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:17.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.434848+0000) 2020-02-06T02:17:17.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:17.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.664576+0000) 2020-02-06T02:17:17.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:17.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:17.687867+0000) 2020-02-06T02:17:19.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:19.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:19.803812+0000) 2020-02-06T02:17:20.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:20.048+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.049400+0000) 2020-02-06T02:17:20.625 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:20.623+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.624370+0000) 2020-02-06T02:17:20.626 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:20.624+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.624953+0000) 2020-02-06T02:17:20.626 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:20.624+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.625306+0000) 2020-02-06T02:17:20.626 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:20.624+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.625625+0000) 2020-02-06T02:17:20.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:20.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.626427+0000) 2020-02-06T02:17:20.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:20.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.626820+0000) 2020-02-06T02:17:20.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:20.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.627460+0000) 2020-02-06T02:17:20.631 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:20.628+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.629305+0000) 2020-02-06T02:17:20.631 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:20.629+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.629587+0000) 2020-02-06T02:17:20.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:20.626+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.627525+0000) 2020-02-06T02:17:20.631 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:20.627+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.628396+0000) 2020-02-06T02:17:20.631 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:20.627+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.628861+0000) 2020-02-06T02:17:20.632 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:20.629+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.630293+0000) 2020-02-06T02:17:20.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:20.633+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.634598+0000) 2020-02-06T02:17:20.647 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:20.645+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228307 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:20.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:20.660+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.660878+0000) 2020-02-06T02:17:20.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:20.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:20.937958+0000) 2020-02-06T02:17:23.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:23.518+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:23.519120+0000) 2020-02-06T02:17:25.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:25.479+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.479520+0000) 2020-02-06T02:17:25.652 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:25.651+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.651415+0000) 2020-02-06T02:17:25.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:25.652+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.652533+0000) 2020-02-06T02:17:25.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:25.653+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.653842+0000) 2020-02-06T02:17:25.655 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:25.653+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.654152+0000) 2020-02-06T02:17:25.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:25.653+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.654771+0000) 2020-02-06T02:17:25.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:25.653+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.654725+0000) 2020-02-06T02:17:25.656 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:25.654+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.655517+0000) 2020-02-06T02:17:25.656 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:25.655+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.655680+0000) 2020-02-06T02:17:25.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:25.654+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.656053+0000) 2020-02-06T02:17:25.657 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:25.655+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.656380+0000) 2020-02-06T02:17:25.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:25.656+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.656527+0000) 2020-02-06T02:17:25.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:25.655+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.657010+0000) 2020-02-06T02:17:25.658 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:25.656+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.657516+0000) 2020-02-06T02:17:25.666 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:25.664+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:25.665407+0000) 2020-02-06T02:17:25.679 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:25.677+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228346 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:26.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:26.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:26.183223+0000) 2020-02-06T02:17:26.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:26.545+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:26.546284+0000) 2020-02-06T02:17:27.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:27.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.160138+0000) 2020-02-06T02:17:27.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:27.434+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.434796+0000) 2020-02-06T02:17:27.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:27.434+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.434799+0000) 2020-02-06T02:17:27.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:27.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.435106+0000) 2020-02-06T02:17:27.665 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:27.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.664807+0000) 2020-02-06T02:17:27.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:27.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:27.688131+0000) 2020-02-06T02:17:29.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:29.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:29.804063+0000) 2020-02-06T02:17:30.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:30.049+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.049628+0000) 2020-02-06T02:17:30.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:30.660+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.661093+0000) 2020-02-06T02:17:30.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:30.683+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.683773+0000) 2020-02-06T02:17:30.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:30.683+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.684383+0000) 2020-02-06T02:17:30.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:30.684+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.684681+0000) 2020-02-06T02:17:30.686 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:30.684+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.685216+0000) 2020-02-06T02:17:30.686 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:30.685+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.685758+0000) 2020-02-06T02:17:30.687 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:30.686+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.686249+0000) 2020-02-06T02:17:30.687 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:30.686+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.686687+0000) 2020-02-06T02:17:30.688 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:30.686+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.686966+0000) 2020-02-06T02:17:30.689 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:30.686+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.687391+0000) 2020-02-06T02:17:30.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:30.688+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.688895+0000) 2020-02-06T02:17:30.690 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:30.689+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.690028+0000) 2020-02-06T02:17:30.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:30.690+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.690238+0000) 2020-02-06T02:17:30.691 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:30.690+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.690262+0000) 2020-02-06T02:17:30.695 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:30.694+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.694644+0000) 2020-02-06T02:17:30.709 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:30.708+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228390 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:30.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:30.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:30.938157+0000) 2020-02-06T02:17:33.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:33.518+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:33.519352+0000) 2020-02-06T02:17:35.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:35.486+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.487746+0000) 2020-02-06T02:17:35.715 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:35.713+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.714140+0000) 2020-02-06T02:17:35.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:35.714+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.714769+0000) 2020-02-06T02:17:35.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:35.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.715337+0000) 2020-02-06T02:17:35.716 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:35.714+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.715742+0000) 2020-02-06T02:17:35.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:35.715+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.716320+0000) 2020-02-06T02:17:35.717 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:35.715+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.716691+0000) 2020-02-06T02:17:35.718 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:35.716+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.717335+0000) 2020-02-06T02:17:35.719 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:35.717+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.718259+0000) 2020-02-06T02:17:35.720 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:35.718+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.719582+0000) 2020-02-06T02:17:35.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:35.718+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.718752+0000) 2020-02-06T02:17:35.721 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:35.719+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.720140+0000) 2020-02-06T02:17:35.721 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:35.719+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.720126+0000) 2020-02-06T02:17:35.722 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:35.719+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.720324+0000) 2020-02-06T02:17:35.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:35.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:35.725717+0000) 2020-02-06T02:17:35.738 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:35.737+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228427 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:36.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:36.183+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:36.183411+0000) 2020-02-06T02:17:36.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:36.546+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:36.546502+0000) 2020-02-06T02:17:37.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:37.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.160332+0000) 2020-02-06T02:17:37.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:37.434+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.435075+0000) 2020-02-06T02:17:37.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:37.434+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.435070+0000) 2020-02-06T02:17:37.436 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:37.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.435298+0000) 2020-02-06T02:17:37.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:37.663+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.665052+0000) 2020-02-06T02:17:37.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:37.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:37.688319+0000) 2020-02-06T02:17:39.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:39.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:39.804347+0000) 2020-02-06T02:17:40.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:40.048+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.049890+0000) 2020-02-06T02:17:40.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:40.661+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.661455+0000) 2020-02-06T02:17:40.744 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:40.742+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.743448+0000) 2020-02-06T02:17:40.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:40.743+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.744988+0000) 2020-02-06T02:17:40.746 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:40.744+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.745374+0000) 2020-02-06T02:17:40.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:40.744+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.745672+0000) 2020-02-06T02:17:40.747 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:40.745+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.746457+0000) 2020-02-06T02:17:40.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:40.745+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.746924+0000) 2020-02-06T02:17:40.748 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:40.746+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.747265+0000) 2020-02-06T02:17:40.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:40.746+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.747360+0000) 2020-02-06T02:17:40.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:40.747+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.747812+0000) 2020-02-06T02:17:40.749 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:40.747+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.747886+0000) 2020-02-06T02:17:40.749 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:40.747+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.748321+0000) 2020-02-06T02:17:40.750 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:40.748+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.748834+0000) 2020-02-06T02:17:40.750 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:40.749+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.749730+0000) 2020-02-06T02:17:40.756 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:40.754+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.755883+0000) 2020-02-06T02:17:40.768 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:40.767+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228470 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:40.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:40.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:40.938344+0000) 2020-02-06T02:17:43.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:43.518+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:43.519588+0000) 2020-02-06T02:17:45.495 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:45.493+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.494706+0000) 2020-02-06T02:17:45.774 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:45.772+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.773280+0000) 2020-02-06T02:17:45.775 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:45.773+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.774539+0000) 2020-02-06T02:17:45.775 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:45.773+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.774735+0000) 2020-02-06T02:17:45.776 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:45.773+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.775054+0000) 2020-02-06T02:17:45.776 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:45.774+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.775510+0000) 2020-02-06T02:17:45.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:45.774+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.775849+0000) 2020-02-06T02:17:45.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:45.774+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.775953+0000) 2020-02-06T02:17:45.777 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:45.775+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.776070+0000) 2020-02-06T02:17:45.778 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:45.774+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.776247+0000) 2020-02-06T02:17:45.778 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:45.775+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.776481+0000) 2020-02-06T02:17:45.779 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:45.775+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.777019+0000) 2020-02-06T02:17:45.779 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:45.775+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.777213+0000) 2020-02-06T02:17:45.779 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:45.775+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.776882+0000) 2020-02-06T02:17:45.783 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:45.781+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:45.782637+0000) 2020-02-06T02:17:45.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:45.795+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228508 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:45.967 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:17:45.974 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:17:46.184 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:46.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:46.183680+0000) 2020-02-06T02:17:46.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:46.545+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:46.546742+0000) 2020-02-06T02:17:47.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:47.159+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.160560+0000) 2020-02-06T02:17:47.436 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:47.434+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.435329+0000) 2020-02-06T02:17:47.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:47.434+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.435324+0000) 2020-02-06T02:17:47.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:47.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.435581+0000) 2020-02-06T02:17:47.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:47.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.665323+0000) 2020-02-06T02:17:47.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:47.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:47.688549+0000) 2020-02-06T02:17:49.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:49.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:49.804588+0000) 2020-02-06T02:17:50.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:50.049+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.050149+0000) 2020-02-06T02:17:50.662 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:50.660+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.661737+0000) 2020-02-06T02:17:50.805 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:50.801+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.802768+0000) 2020-02-06T02:17:50.805 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:50.802+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.803504+0000) 2020-02-06T02:17:50.806 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:50.803+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.804209+0000) 2020-02-06T02:17:50.806 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:50.803+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.804999+0000) 2020-02-06T02:17:50.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:50.804+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.805277+0000) 2020-02-06T02:17:50.807 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:50.804+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.805547+0000) 2020-02-06T02:17:50.807 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:50.804+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.806185+0000) 2020-02-06T02:17:50.808 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:50.805+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.806849+0000) 2020-02-06T02:17:50.808 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:50.805+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.806645+0000) 2020-02-06T02:17:50.808 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:50.806+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.807465+0000) 2020-02-06T02:17:50.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:50.806+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.807907+0000) 2020-02-06T02:17:50.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:50.806+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.807558+0000) 2020-02-06T02:17:50.809 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:50.807+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.808900+0000) 2020-02-06T02:17:50.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:50.812+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.813744+0000) 2020-02-06T02:17:50.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:50.823+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228553 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:50.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:50.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:50.938589+0000) 2020-02-06T02:17:53.520 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:53.519+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:53.519817+0000) 2020-02-06T02:17:55.505 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:55.502+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.503213+0000) 2020-02-06T02:17:55.831 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:55.828+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.829887+0000) 2020-02-06T02:17:55.832 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:17:55.830+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.831299+0000) 2020-02-06T02:17:55.832 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:55.830+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.831699+0000) 2020-02-06T02:17:55.833 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:55.830+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.831654+0000) 2020-02-06T02:17:55.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:55.830+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.832123+0000) 2020-02-06T02:17:55.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:55.831+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.832530+0000) 2020-02-06T02:17:55.834 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:55.831+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.832758+0000) 2020-02-06T02:17:55.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:55.832+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.833367+0000) 2020-02-06T02:17:55.834 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:55.832+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.833907+0000) 2020-02-06T02:17:55.835 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:17:55.832+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.834094+0000) 2020-02-06T02:17:55.835 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:17:55.833+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.834430+0000) 2020-02-06T02:17:55.836 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:55.833+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.834896+0000) 2020-02-06T02:17:55.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:55.834+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.835276+0000) 2020-02-06T02:17:55.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:55.837+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:55.838563+0000) 2020-02-06T02:17:55.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:17:55.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228590 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:17:56.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:17:56.183+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:56.183914+0000) 2020-02-06T02:17:56.547 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:17:56.545+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:56.546941+0000) 2020-02-06T02:17:57.161 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:17:57.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.160799+0000) 2020-02-06T02:17:57.436 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:17:57.434+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.435595+0000) 2020-02-06T02:17:57.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:17:57.434+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.435590+0000) 2020-02-06T02:17:57.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:17:57.434+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.435851+0000) 2020-02-06T02:17:57.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:17:57.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.665560+0000) 2020-02-06T02:17:57.689 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:17:57.687+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:57.688796+0000) 2020-02-06T02:17:59.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:17:59.803+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:17:59.804818+0000) 2020-02-06T02:18:00.051 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:00.049+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.050398+0000) 2020-02-06T02:18:00.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:00.661+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.661979+0000) 2020-02-06T02:18:00.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:00.859+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.861161+0000) 2020-02-06T02:18:00.869 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:00.860+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.861404+0000) 2020-02-06T02:18:00.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:00.860+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.862134+0000) 2020-02-06T02:18:00.871 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:00.861+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.862756+0000) 2020-02-06T02:18:00.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:00.861+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.862672+0000) 2020-02-06T02:18:00.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:00.862+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.863222+0000) 2020-02-06T02:18:00.872 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:00.862+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.863656+0000) 2020-02-06T02:18:00.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:00.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.864219+0000) 2020-02-06T02:18:00.874 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:00.864+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.865141+0000) 2020-02-06T02:18:00.875 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:00.864+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.865574+0000) 2020-02-06T02:18:00.875 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:00.864+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.866125+0000) 2020-02-06T02:18:00.876 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:00.864+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.866365+0000) 2020-02-06T02:18:00.876 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:00.865+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.866675+0000) 2020-02-06T02:18:00.882 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:00.870+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.871874+0000) 2020-02-06T02:18:00.896 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:00.882+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228634 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:00.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:00.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:00.938813+0000) 2020-02-06T02:18:03.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:03.519+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:03.520104+0000) 2020-02-06T02:18:05.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:05.511+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.512460+0000) 2020-02-06T02:18:05.890 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:05.888+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.889798+0000) 2020-02-06T02:18:05.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:05.889+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.890829+0000) 2020-02-06T02:18:05.892 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:05.890+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.891878+0000) 2020-02-06T02:18:05.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:05.891+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.892492+0000) 2020-02-06T02:18:05.894 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:05.891+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.892932+0000) 2020-02-06T02:18:05.895 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:05.893+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.894477+0000) 2020-02-06T02:18:05.895 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:05.893+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.894996+0000) 2020-02-06T02:18:05.896 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:05.894+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.895577+0000) 2020-02-06T02:18:05.896 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:05.894+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.895772+0000) 2020-02-06T02:18:05.897 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:05.894+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.895930+0000) 2020-02-06T02:18:05.897 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:05.894+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.895993+0000) 2020-02-06T02:18:05.897 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:05.895+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.896476+0000) 2020-02-06T02:18:05.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:05.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.898612+0000) 2020-02-06T02:18:05.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:05.897+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:05.898879+0000) 2020-02-06T02:18:05.914 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:05.912+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:06.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:06.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:06.184116+0000) 2020-02-06T02:18:06.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:06.546+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:06.547179+0000) 2020-02-06T02:18:07.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:07.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.161033+0000) 2020-02-06T02:18:07.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:07.435+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.435896+0000) 2020-02-06T02:18:07.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:07.435+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.435853+0000) 2020-02-06T02:18:07.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:07.435+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.436071+0000) 2020-02-06T02:18:07.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:07.664+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.665808+0000) 2020-02-06T02:18:07.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:07.688+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:07.689030+0000) 2020-02-06T02:18:09.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:09.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:09.805071+0000) 2020-02-06T02:18:10.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:10.050+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.050652+0000) 2020-02-06T02:18:10.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:10.661+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.662235+0000) 2020-02-06T02:18:10.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:10.919+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.920913+0000) 2020-02-06T02:18:10.922 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:10.920+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.921620+0000) 2020-02-06T02:18:10.922 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:10.921+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.922115+0000) 2020-02-06T02:18:10.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:10.921+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.922528+0000) 2020-02-06T02:18:10.923 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:10.921+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.923036+0000) 2020-02-06T02:18:10.924 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:10.922+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.923380+0000) 2020-02-06T02:18:10.924 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:10.922+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.923905+0000) 2020-02-06T02:18:10.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:10.923+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.924405+0000) 2020-02-06T02:18:10.926 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:10.923+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.925205+0000) 2020-02-06T02:18:10.926 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:10.924+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.925534+0000) 2020-02-06T02:18:10.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:10.924+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.925578+0000) 2020-02-06T02:18:10.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:10.924+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.925986+0000) 2020-02-06T02:18:10.927 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:10.924+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.925905+0000) 2020-02-06T02:18:10.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:10.931+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.932397+0000) 2020-02-06T02:18:10.939 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:10.937+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:10.939028+0000) 2020-02-06T02:18:10.943 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:10.940+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228717 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:13.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:13.520+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:13.520347+0000) 2020-02-06T02:18:15.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:15.523+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.523701+0000) 2020-02-06T02:18:15.947 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:15.945+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.946439+0000) 2020-02-06T02:18:15.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:15.947+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.948919+0000) 2020-02-06T02:18:15.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:15.948+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.949460+0000) 2020-02-06T02:18:15.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:15.948+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.949926+0000) 2020-02-06T02:18:15.951 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:15.949+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.950694+0000) 2020-02-06T02:18:15.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:15.950+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.951966+0000) 2020-02-06T02:18:15.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:15.949+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.951341+0000) 2020-02-06T02:18:15.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:15.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.952363+0000) 2020-02-06T02:18:15.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:15.950+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.951986+0000) 2020-02-06T02:18:15.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:15.951+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.952856+0000) 2020-02-06T02:18:15.954 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:15.951+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.953348+0000) 2020-02-06T02:18:15.955 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:15.952+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.953770+0000) 2020-02-06T02:18:15.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:15.952+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.954031+0000) 2020-02-06T02:18:15.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:15.955+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:15.956798+0000) 2020-02-06T02:18:15.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:15.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228754 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:16.141 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:18:16.147 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:18:16.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:16.182+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:16.184337+0000) 2020-02-06T02:18:16.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:16.546+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:16.547410+0000) 2020-02-06T02:18:17.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:17.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.161285+0000) 2020-02-06T02:18:17.437 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:17.435+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.436157+0000) 2020-02-06T02:18:17.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:17.435+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.436163+0000) 2020-02-06T02:18:17.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:17.435+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.436316+0000) 2020-02-06T02:18:17.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:17.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.666132+0000) 2020-02-06T02:18:17.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:17.688+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:17.689281+0000) 2020-02-06T02:18:19.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:19.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:19.805351+0000) 2020-02-06T02:18:20.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:20.050+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.050884+0000) 2020-02-06T02:18:20.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:20.661+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.662474+0000) 2020-02-06T02:18:20.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:20.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.939162+0000) 2020-02-06T02:18:20.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:20.979+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.980947+0000) 2020-02-06T02:18:20.982 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:20.980+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.981635+0000) 2020-02-06T02:18:20.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:20.981+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.982307+0000) 2020-02-06T02:18:20.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:20.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.982495+0000) 2020-02-06T02:18:20.983 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:20.981+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.982812+0000) 2020-02-06T02:18:20.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:20.981+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.983347+0000) 2020-02-06T02:18:20.985 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:20.983+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.984840+0000) 2020-02-06T02:18:20.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:20.982+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.984177+0000) 2020-02-06T02:18:20.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:20.984+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.985632+0000) 2020-02-06T02:18:20.987 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:20.984+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.985821+0000) 2020-02-06T02:18:20.987 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:20.984+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.986199+0000) 2020-02-06T02:18:20.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:20.985+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.986597+0000) 2020-02-06T02:18:20.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:20.988+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.989830+0000) 2020-02-06T02:18:20.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:20.988+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:20.990031+0000) 2020-02-06T02:18:21.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:21.004+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228797 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:23.521 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:23.520+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:23.520574+0000) 2020-02-06T02:18:25.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:25.535+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:25.535574+0000) 2020-02-06T02:18:26.013 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:26.010+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.012054+0000) 2020-02-06T02:18:26.013 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:26.012+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.012659+0000) 2020-02-06T02:18:26.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:26.013+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.013779+0000) 2020-02-06T02:18:26.015 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:26.014+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.014356+0000) 2020-02-06T02:18:26.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:26.012+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.014050+0000) 2020-02-06T02:18:26.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:26.013+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.015060+0000) 2020-02-06T02:18:26.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:26.013+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.015232+0000) 2020-02-06T02:18:26.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:26.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.015767+0000) 2020-02-06T02:18:26.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:26.016+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.016224+0000) 2020-02-06T02:18:26.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:26.016+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.016603+0000) 2020-02-06T02:18:26.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:26.015+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.016693+0000) 2020-02-06T02:18:26.018 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:26.016+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.016990+0000) 2020-02-06T02:18:26.018 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:26.016+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.017840+0000) 2020-02-06T02:18:26.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:26.024+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.024573+0000) 2020-02-06T02:18:26.037 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:26.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228835 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:26.185 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:26.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.184647+0000) 2020-02-06T02:18:26.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:26.547+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:26.547696+0000) 2020-02-06T02:18:27.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:27.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.161503+0000) 2020-02-06T02:18:27.437 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:27.435+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.436482+0000) 2020-02-06T02:18:27.438 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:27.435+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.436464+0000) 2020-02-06T02:18:27.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:27.435+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.436593+0000) 2020-02-06T02:18:27.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:27.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.666365+0000) 2020-02-06T02:18:27.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:27.688+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:27.689468+0000) 2020-02-06T02:18:29.806 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:29.804+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:29.805635+0000) 2020-02-06T02:18:30.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:30.050+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:30.051123+0000) 2020-02-06T02:18:30.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:30.662+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:30.662704+0000) 2020-02-06T02:18:30.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:30.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:30.939392+0000) 2020-02-06T02:18:31.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:31.040+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.041561+0000) 2020-02-06T02:18:31.044 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:31.041+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.043104+0000) 2020-02-06T02:18:31.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:31.042+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.043507+0000) 2020-02-06T02:18:31.045 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:31.042+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.043955+0000) 2020-02-06T02:18:31.045 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:31.042+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.043769+0000) 2020-02-06T02:18:31.046 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:31.043+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.044412+0000) 2020-02-06T02:18:31.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:31.043+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.044796+0000) 2020-02-06T02:18:31.046 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:31.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.045059+0000) 2020-02-06T02:18:31.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:31.044+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.045759+0000) 2020-02-06T02:18:31.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:31.044+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.045554+0000) 2020-02-06T02:18:31.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:31.045+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.046676+0000) 2020-02-06T02:18:31.048 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:31.045+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.046938+0000) 2020-02-06T02:18:31.048 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:31.045+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.046951+0000) 2020-02-06T02:18:31.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:31.051+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:31.052790+0000) 2020-02-06T02:18:31.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:31.068+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228879 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:33.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:33.519+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:33.520798+0000) 2020-02-06T02:18:35.546 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:35.544+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:35.545588+0000) 2020-02-06T02:18:36.076 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:36.074+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.074926+0000) 2020-02-06T02:18:36.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:36.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.075174+0000) 2020-02-06T02:18:36.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:36.075+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.076352+0000) 2020-02-06T02:18:36.078 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:36.077+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.077645+0000) 2020-02-06T02:18:36.079 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:36.077+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.077780+0000) 2020-02-06T02:18:36.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:36.078+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.079027+0000) 2020-02-06T02:18:36.080 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:36.078+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.079400+0000) 2020-02-06T02:18:36.081 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:36.079+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.080311+0000) 2020-02-06T02:18:36.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:36.080+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.081240+0000) 2020-02-06T02:18:36.082 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:36.080+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.081119+0000) 2020-02-06T02:18:36.082 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:36.080+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.081271+0000) 2020-02-06T02:18:36.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:36.080+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.081443+0000) 2020-02-06T02:18:36.083 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:36.081+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.081777+0000) 2020-02-06T02:18:36.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:36.084+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.085155+0000) 2020-02-06T02:18:36.101 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:36.100+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228916 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:36.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:36.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.184860+0000) 2020-02-06T02:18:36.548 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:36.547+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:36.547932+0000) 2020-02-06T02:18:37.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:37.160+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.161739+0000) 2020-02-06T02:18:37.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:37.436+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.436725+0000) 2020-02-06T02:18:37.438 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:37.436+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.436765+0000) 2020-02-06T02:18:37.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:37.436+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.436870+0000) 2020-02-06T02:18:37.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:37.665+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.666594+0000) 2020-02-06T02:18:37.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:37.688+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:37.689693+0000) 2020-02-06T02:18:39.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:39.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:39.805895+0000) 2020-02-06T02:18:40.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:40.050+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:40.051356+0000) 2020-02-06T02:18:40.663 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:40.662+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:40.662987+0000) 2020-02-06T02:18:40.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:40.938+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:40.939619+0000) 2020-02-06T02:18:41.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:41.106+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.106842+0000) 2020-02-06T02:18:41.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:41.106+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.107623+0000) 2020-02-06T02:18:41.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:41.107+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.108250+0000) 2020-02-06T02:18:41.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:41.107+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.108809+0000) 2020-02-06T02:18:41.110 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:41.108+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.109148+0000) 2020-02-06T02:18:41.110 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:41.109+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.109673+0000) 2020-02-06T02:18:41.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:41.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.110788+0000) 2020-02-06T02:18:41.112 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:41.109+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.111035+0000) 2020-02-06T02:18:41.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:41.111+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.111787+0000) 2020-02-06T02:18:41.112 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:41.110+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.111521+0000) 2020-02-06T02:18:41.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:41.112+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.112507+0000) 2020-02-06T02:18:41.113 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:41.111+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.112310+0000) 2020-02-06T02:18:41.114 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:41.112+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.113099+0000) 2020-02-06T02:18:41.116 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:41.115+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:41.116092+0000) 2020-02-06T02:18:41.131 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:41.130+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228959 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:43.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:43.519+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:43.521026+0000) 2020-02-06T02:18:45.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:45.553+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:45.553826+0000) 2020-02-06T02:18:46.136 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:46.134+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.135949+0000) 2020-02-06T02:18:46.137 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:46.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.136773+0000) 2020-02-06T02:18:46.137 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:46.136+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.137157+0000) 2020-02-06T02:18:46.138 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:46.136+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.137445+0000) 2020-02-06T02:18:46.139 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:46.137+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.138059+0000) 2020-02-06T02:18:46.139 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:46.137+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.138432+0000) 2020-02-06T02:18:46.140 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:46.138+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.139289+0000) 2020-02-06T02:18:46.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:46.138+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.139829+0000) 2020-02-06T02:18:46.141 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:46.139+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.140261+0000) 2020-02-06T02:18:46.141 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:46.140+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.140694+0000) 2020-02-06T02:18:46.142 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:46.140+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.141226+0000) 2020-02-06T02:18:46.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:46.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.141845+0000) 2020-02-06T02:18:46.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:46.141+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.142182+0000) 2020-02-06T02:18:46.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:46.147+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.148276+0000) 2020-02-06T02:18:46.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:46.159+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 228997 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:46.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:46.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.185116+0000) 2020-02-06T02:18:46.338 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:18:46.344 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:18:46.549 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:46.547+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:46.548176+0000) 2020-02-06T02:18:47.162 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:47.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.161957+0000) 2020-02-06T02:18:47.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:47.436+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.436961+0000) 2020-02-06T02:18:47.438 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:47.436+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.437021+0000) 2020-02-06T02:18:47.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:47.436+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.437126+0000) 2020-02-06T02:18:47.667 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:47.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.666867+0000) 2020-02-06T02:18:47.690 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:47.689+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:47.689920+0000) 2020-02-06T02:18:49.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:49.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:49.806109+0000) 2020-02-06T02:18:50.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:50.050+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:50.051606+0000) 2020-02-06T02:18:50.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:50.662+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:50.663238+0000) 2020-02-06T02:18:50.940 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:50.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:50.939845+0000) 2020-02-06T02:18:51.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:51.164+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.165496+0000) 2020-02-06T02:18:51.167 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:51.164+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.165302+0000) 2020-02-06T02:18:51.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:51.167+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.168346+0000) 2020-02-06T02:18:51.169 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:51.167+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.168884+0000) 2020-02-06T02:18:51.170 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:51.168+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.169112+0000) 2020-02-06T02:18:51.170 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:51.169+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.169954+0000) 2020-02-06T02:18:51.171 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:51.169+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.170357+0000) 2020-02-06T02:18:51.171 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:51.170+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.170828+0000) 2020-02-06T02:18:51.172 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:51.170+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.171000+0000) 2020-02-06T02:18:51.172 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:51.171+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.171528+0000) 2020-02-06T02:18:51.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:51.171+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.171950+0000) 2020-02-06T02:18:51.173 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:51.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.172061+0000) 2020-02-06T02:18:51.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:51.175+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.176126+0000) 2020-02-06T02:18:51.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:51.175+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:51.176441+0000) 2020-02-06T02:18:51.190 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:51.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229041 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:53.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:53.520+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:53.521355+0000) 2020-02-06T02:18:55.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:55.561+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:55.562479+0000) 2020-02-06T02:18:56.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:56.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.185350+0000) 2020-02-06T02:18:56.211 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:56.195+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.196040+0000) 2020-02-06T02:18:56.211 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:56.195+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.196380+0000) 2020-02-06T02:18:56.211 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:56.197+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.197577+0000) 2020-02-06T02:18:56.212 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:18:56.198+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.198652+0000) 2020-02-06T02:18:56.213 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:18:56.198+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.199359+0000) 2020-02-06T02:18:56.214 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:56.199+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.200214+0000) 2020-02-06T02:18:56.223 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:56.193+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.194962+0000) 2020-02-06T02:18:56.224 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:18:56.195+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.196699+0000) 2020-02-06T02:18:56.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:56.196+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.198054+0000) 2020-02-06T02:18:56.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:56.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.205653+0000) 2020-02-06T02:18:56.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:56.204+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.205929+0000) 2020-02-06T02:18:56.226 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:56.197+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.198244+0000) 2020-02-06T02:18:56.226 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:56.198+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.199124+0000) 2020-02-06T02:18:56.227 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:18:56.198+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.199933+0000) 2020-02-06T02:18:56.232 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:18:56.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229079 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:18:56.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:18:56.548+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:56.548418+0000) 2020-02-06T02:18:57.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:18:57.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.162194+0000) 2020-02-06T02:18:57.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:18:57.436+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.437192+0000) 2020-02-06T02:18:57.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:18:57.436+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.437384+0000) 2020-02-06T02:18:57.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:18:57.436+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.437301+0000) 2020-02-06T02:18:57.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:18:57.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.667126+0000) 2020-02-06T02:18:57.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:18:57.689+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:57.690159+0000) 2020-02-06T02:18:59.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:18:59.805+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:18:59.806343+0000) 2020-02-06T02:19:00.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:00.051+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:00.051834+0000) 2020-02-06T02:19:00.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:00.663+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:00.663422+0000) 2020-02-06T02:19:00.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:00.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:00.940106+0000) 2020-02-06T02:19:01.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:01.224+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.225793+0000) 2020-02-06T02:19:01.227 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:01.225+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.226211+0000) 2020-02-06T02:19:01.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:01.225+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.226596+0000) 2020-02-06T02:19:01.228 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:01.226+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.227239+0000) 2020-02-06T02:19:01.228 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:01.227+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.228067+0000) 2020-02-06T02:19:01.230 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:01.229+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.229753+0000) 2020-02-06T02:19:01.231 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:01.229+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.230231+0000) 2020-02-06T02:19:01.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:01.229+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.230597+0000) 2020-02-06T02:19:01.232 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:01.230+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.231037+0000) 2020-02-06T02:19:01.232 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:01.230+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.231400+0000) 2020-02-06T02:19:01.232 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:01.231+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.231667+0000) 2020-02-06T02:19:01.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:01.230+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.231158+0000) 2020-02-06T02:19:01.233 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:01.230+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.231394+0000) 2020-02-06T02:19:01.236 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:01.234+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:01.235460+0000) 2020-02-06T02:19:01.249 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:01.248+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229122 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:03.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:03.521+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:03.521583+0000) 2020-02-06T02:19:05.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:05.569+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:05.569458+0000) 2020-02-06T02:19:06.186 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:06.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.185621+0000) 2020-02-06T02:19:06.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:06.253+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.254259+0000) 2020-02-06T02:19:06.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:06.253+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.254604+0000) 2020-02-06T02:19:06.256 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:06.254+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.255146+0000) 2020-02-06T02:19:06.256 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:06.254+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.255646+0000) 2020-02-06T02:19:06.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:06.255+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.256171+0000) 2020-02-06T02:19:06.258 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:06.257+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.257454+0000) 2020-02-06T02:19:06.258 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:06.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.257688+0000) 2020-02-06T02:19:06.259 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:06.257+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.257949+0000) 2020-02-06T02:19:06.259 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:06.257+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.258280+0000) 2020-02-06T02:19:06.259 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:06.258+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.259031+0000) 2020-02-06T02:19:06.260 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:06.258+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.259151+0000) 2020-02-06T02:19:06.260 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:06.258+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.259341+0000) 2020-02-06T02:19:06.260 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:06.259+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.259848+0000) 2020-02-06T02:19:06.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:06.262+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.263555+0000) 2020-02-06T02:19:06.279 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:06.278+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229161 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:06.549 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:06.548+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:06.548712+0000) 2020-02-06T02:19:07.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:07.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.162407+0000) 2020-02-06T02:19:07.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:07.437+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.437422+0000) 2020-02-06T02:19:07.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:07.437+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.437590+0000) 2020-02-06T02:19:07.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:07.437+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.437657+0000) 2020-02-06T02:19:07.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:07.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.667342+0000) 2020-02-06T02:19:07.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:07.690+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:07.690393+0000) 2020-02-06T02:19:09.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:09.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:09.806574+0000) 2020-02-06T02:19:10.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:10.051+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:10.052067+0000) 2020-02-06T02:19:10.664 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:10.663+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:10.663691+0000) 2020-02-06T02:19:10.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:10.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:10.940346+0000) 2020-02-06T02:19:11.285 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:11.283+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.284227+0000) 2020-02-06T02:19:11.285 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:11.283+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.284929+0000) 2020-02-06T02:19:11.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:11.284+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.285613+0000) 2020-02-06T02:19:11.287 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:11.285+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.286300+0000) 2020-02-06T02:19:11.288 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:11.285+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.287174+0000) 2020-02-06T02:19:11.288 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:11.285+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.286858+0000) 2020-02-06T02:19:11.288 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:11.286+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.287494+0000) 2020-02-06T02:19:11.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:11.286+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.288007+0000) 2020-02-06T02:19:11.289 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:11.287+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.288591+0000) 2020-02-06T02:19:11.290 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:11.288+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.289649+0000) 2020-02-06T02:19:11.291 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:11.289+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.290170+0000) 2020-02-06T02:19:11.291 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:11.288+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.290198+0000) 2020-02-06T02:19:11.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:11.288+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.290294+0000) 2020-02-06T02:19:11.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:11.294+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:11.295672+0000) 2020-02-06T02:19:11.312 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:11.309+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229205 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:13.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:13.521+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:13.521820+0000) 2020-02-06T02:19:15.582 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:15.581+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:15.581364+0000) 2020-02-06T02:19:16.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:16.184+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.185883+0000) 2020-02-06T02:19:16.318 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:16.316+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.317241+0000) 2020-02-06T02:19:16.318 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:16.316+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.317912+0000) 2020-02-06T02:19:16.319 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:16.317+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.318416+0000) 2020-02-06T02:19:16.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:16.317+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.319003+0000) 2020-02-06T02:19:16.320 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:16.318+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.319478+0000) 2020-02-06T02:19:16.321 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:16.318+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.320033+0000) 2020-02-06T02:19:16.321 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:16.319+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.320503+0000) 2020-02-06T02:19:16.321 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:16.319+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.320854+0000) 2020-02-06T02:19:16.322 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:16.319+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.321373+0000) 2020-02-06T02:19:16.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:16.320+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.322111+0000) 2020-02-06T02:19:16.323 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:16.320+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.322003+0000) 2020-02-06T02:19:16.323 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:16.320+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.322406+0000) 2020-02-06T02:19:16.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:16.321+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.322637+0000) 2020-02-06T02:19:16.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:16.325+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.326926+0000) 2020-02-06T02:19:16.341 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:16.339+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229243 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:16.493 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:19:16.499 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:19:16.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:16.547+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:16.548928+0000) 2020-02-06T02:19:17.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:17.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.162623+0000) 2020-02-06T02:19:17.438 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:17.436+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.437720+0000) 2020-02-06T02:19:17.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:17.436+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.437870+0000) 2020-02-06T02:19:17.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:17.436+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.437910+0000) 2020-02-06T02:19:17.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:17.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.667571+0000) 2020-02-06T02:19:17.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:17.690+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:17.690622+0000) 2020-02-06T02:19:19.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:19.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:19.806814+0000) 2020-02-06T02:19:20.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:20.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:20.052362+0000) 2020-02-06T02:19:20.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:20.663+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:20.663955+0000) 2020-02-06T02:19:20.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:20.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:20.940573+0000) 2020-02-06T02:19:21.348 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:21.347+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.347958+0000) 2020-02-06T02:19:21.349 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:21.347+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.348663+0000) 2020-02-06T02:19:21.350 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:21.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.348985+0000) 2020-02-06T02:19:21.350 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:21.349+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.349330+0000) 2020-02-06T02:19:21.350 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:21.349+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.349505+0000) 2020-02-06T02:19:21.351 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:21.349+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.350511+0000) 2020-02-06T02:19:21.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:21.350+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.350973+0000) 2020-02-06T02:19:21.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:21.348+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.349921+0000) 2020-02-06T02:19:21.352 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:21.350+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.351423+0000) 2020-02-06T02:19:21.353 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:21.351+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.352621+0000) 2020-02-06T02:19:21.354 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:21.353+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.353364+0000) 2020-02-06T02:19:21.354 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:21.352+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.353454+0000) 2020-02-06T02:19:21.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:21.358+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.358713+0000) 2020-02-06T02:19:21.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:21.358+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:21.359022+0000) 2020-02-06T02:19:21.372 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:21.369+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229285 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:23.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:23.521+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:23.522112+0000) 2020-02-06T02:19:25.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:25.588+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:25.589659+0000) 2020-02-06T02:19:26.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:26.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.186172+0000) 2020-02-06T02:19:26.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:26.376+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.377587+0000) 2020-02-06T02:19:26.379 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:26.376+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.378367+0000) 2020-02-06T02:19:26.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:26.376+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.378101+0000) 2020-02-06T02:19:26.380 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:26.378+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.379172+0000) 2020-02-06T02:19:26.381 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:26.379+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.380345+0000) 2020-02-06T02:19:26.381 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:26.379+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.380726+0000) 2020-02-06T02:19:26.382 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:26.379+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.381263+0000) 2020-02-06T02:19:26.382 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:26.380+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.381426+0000) 2020-02-06T02:19:26.383 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:26.381+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.382284+0000) 2020-02-06T02:19:26.383 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:26.380+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.382264+0000) 2020-02-06T02:19:26.383 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:26.381+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.382410+0000) 2020-02-06T02:19:26.384 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:26.381+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.382814+0000) 2020-02-06T02:19:26.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:26.382+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.383258+0000) 2020-02-06T02:19:26.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:26.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.386135+0000) 2020-02-06T02:19:26.401 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:26.399+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229324 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:26.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:26.548+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:26.549209+0000) 2020-02-06T02:19:27.163 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:27.161+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.162834+0000) 2020-02-06T02:19:27.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:27.437+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.437954+0000) 2020-02-06T02:19:27.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:27.437+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.438120+0000) 2020-02-06T02:19:27.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:27.437+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.438198+0000) 2020-02-06T02:19:27.668 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:27.666+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.667798+0000) 2020-02-06T02:19:27.691 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:27.689+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:27.690853+0000) 2020-02-06T02:19:29.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:29.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:29.807042+0000) 2020-02-06T02:19:30.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:30.051+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:30.052618+0000) 2020-02-06T02:19:30.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:30.663+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:30.664190+0000) 2020-02-06T02:19:30.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:30.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:30.940835+0000) 2020-02-06T02:19:31.407 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:31.405+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.406497+0000) 2020-02-06T02:19:31.408 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:31.406+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.407199+0000) 2020-02-06T02:19:31.409 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:31.407+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.408775+0000) 2020-02-06T02:19:31.410 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:31.407+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.408926+0000) 2020-02-06T02:19:31.410 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:31.408+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.409237+0000) 2020-02-06T02:19:31.410 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:31.408+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.409667+0000) 2020-02-06T02:19:31.411 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:31.409+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.410574+0000) 2020-02-06T02:19:31.412 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:31.409+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.411268+0000) 2020-02-06T02:19:31.412 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:31.410+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.411470+0000) 2020-02-06T02:19:31.412 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:31.410+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.411799+0000) 2020-02-06T02:19:31.413 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:31.410+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.411957+0000) 2020-02-06T02:19:31.414 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:31.410+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.412107+0000) 2020-02-06T02:19:31.414 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:31.411+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.412414+0000) 2020-02-06T02:19:31.422 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:31.420+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:31.421220+0000) 2020-02-06T02:19:31.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:31.428+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229368 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:33.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:33.521+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:33.522351+0000) 2020-02-06T02:19:35.602 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:35.600+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:35.601571+0000) 2020-02-06T02:19:36.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:36.185+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.186459+0000) 2020-02-06T02:19:36.435 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:36.433+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.434390+0000) 2020-02-06T02:19:36.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:36.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.435540+0000) 2020-02-06T02:19:36.438 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:36.435+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.436760+0000) 2020-02-06T02:19:36.438 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:36.435+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.436870+0000) 2020-02-06T02:19:36.438 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:36.435+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.437175+0000) 2020-02-06T02:19:36.439 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:36.436+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.437439+0000) 2020-02-06T02:19:36.439 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:36.437+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.438148+0000) 2020-02-06T02:19:36.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:36.437+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.438220+0000) 2020-02-06T02:19:36.440 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:36.437+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.438693+0000) 2020-02-06T02:19:36.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:36.437+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.439008+0000) 2020-02-06T02:19:36.441 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:36.439+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.440358+0000) 2020-02-06T02:19:36.441 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:36.439+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.440658+0000) 2020-02-06T02:19:36.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:36.442+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.443970+0000) 2020-02-06T02:19:36.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:36.443+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.444227+0000) 2020-02-06T02:19:36.460 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:36.457+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229406 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:36.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:36.548+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:36.549485+0000) 2020-02-06T02:19:37.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:37.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.163068+0000) 2020-02-06T02:19:37.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:37.437+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.438195+0000) 2020-02-06T02:19:37.439 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:37.437+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.438332+0000) 2020-02-06T02:19:37.439 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:37.437+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.438486+0000) 2020-02-06T02:19:37.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:37.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.668114+0000) 2020-02-06T02:19:37.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:37.690+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:37.691113+0000) 2020-02-06T02:19:39.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:39.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:39.807271+0000) 2020-02-06T02:19:40.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:40.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:40.052879+0000) 2020-02-06T02:19:40.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:40.663+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:40.664473+0000) 2020-02-06T02:19:40.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:40.939+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:40.941049+0000) 2020-02-06T02:19:41.466 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:41.464+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.465890+0000) 2020-02-06T02:19:41.467 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:41.465+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.466464+0000) 2020-02-06T02:19:41.467 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:41.465+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.466874+0000) 2020-02-06T02:19:41.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:41.465+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.467335+0000) 2020-02-06T02:19:41.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:41.467+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.468666+0000) 2020-02-06T02:19:41.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:41.467+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.468999+0000) 2020-02-06T02:19:41.470 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:41.468+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.469507+0000) 2020-02-06T02:19:41.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:41.467+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.469155+0000) 2020-02-06T02:19:41.471 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:41.468+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.469902+0000) 2020-02-06T02:19:41.475 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:41.470+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.471682+0000) 2020-02-06T02:19:41.475 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:41.470+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.471183+0000) 2020-02-06T02:19:41.475 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:41.470+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.471073+0000) 2020-02-06T02:19:41.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:41.470+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.471623+0000) 2020-02-06T02:19:41.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:41.476+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:41.477351+0000) 2020-02-06T02:19:41.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:41.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229450 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:43.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:43.521+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:43.522655+0000) 2020-02-06T02:19:45.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:45.612+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:45.613761+0000) 2020-02-06T02:19:46.187 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:46.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.186710+0000) 2020-02-06T02:19:46.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:46.495+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.496684+0000) 2020-02-06T02:19:46.498 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:46.495+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.497375+0000) 2020-02-06T02:19:46.499 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:46.496+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.497702+0000) 2020-02-06T02:19:46.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:46.497+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.498668+0000) 2020-02-06T02:19:46.500 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:46.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.499009+0000) 2020-02-06T02:19:46.501 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:46.498+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.499717+0000) 2020-02-06T02:19:46.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:46.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.499883+0000) 2020-02-06T02:19:46.502 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:46.499+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.500195+0000) 2020-02-06T02:19:46.503 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:46.500+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.501588+0000) 2020-02-06T02:19:46.503 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:46.499+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.501401+0000) 2020-02-06T02:19:46.503 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:46.499+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.500985+0000) 2020-02-06T02:19:46.504 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:46.500+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.502016+0000) 2020-02-06T02:19:46.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:46.501+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.503046+0000) 2020-02-06T02:19:46.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:46.506+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.507525+0000) 2020-02-06T02:19:46.523 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:46.521+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229488 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:46.550 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:46.548+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:46.549687+0000) 2020-02-06T02:19:46.663 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:19:46.670 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:19:47.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:47.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.163312+0000) 2020-02-06T02:19:47.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:47.437+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.438413+0000) 2020-02-06T02:19:47.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:47.437+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.438530+0000) 2020-02-06T02:19:47.440 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:47.437+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.438704+0000) 2020-02-06T02:19:47.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:47.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.668368+0000) 2020-02-06T02:19:47.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:47.690+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:47.691407+0000) 2020-02-06T02:19:49.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:49.806+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:49.807505+0000) 2020-02-06T02:19:50.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:50.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:50.053124+0000) 2020-02-06T02:19:50.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:50.664+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:50.664729+0000) 2020-02-06T02:19:50.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:50.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:50.941311+0000) 2020-02-06T02:19:51.541 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:51.526+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.528069+0000) 2020-02-06T02:19:51.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:51.527+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.528629+0000) 2020-02-06T02:19:51.543 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:51.529+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.530362+0000) 2020-02-06T02:19:51.543 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:51.529+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.530740+0000) 2020-02-06T02:19:51.544 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:51.530+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.531220+0000) 2020-02-06T02:19:51.544 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:51.530+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.531689+0000) 2020-02-06T02:19:51.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:51.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.532556+0000) 2020-02-06T02:19:51.546 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:51.531+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.532694+0000) 2020-02-06T02:19:51.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:51.532+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.533065+0000) 2020-02-06T02:19:51.546 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:51.531+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.533001+0000) 2020-02-06T02:19:51.547 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:51.531+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.533385+0000) 2020-02-06T02:19:51.547 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:51.532+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.534293+0000) 2020-02-06T02:19:51.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:51.533+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.534436+0000) 2020-02-06T02:19:51.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:51.538+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:51.539584+0000) 2020-02-06T02:19:51.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:51.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229532 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:53.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:53.522+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:53.522885+0000) 2020-02-06T02:19:55.624 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:55.623+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:55.623822+0000) 2020-02-06T02:19:56.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:56.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.186973+0000) 2020-02-06T02:19:56.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:56.549+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.549901+0000) 2020-02-06T02:19:56.579 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:19:56.557+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.558436+0000) 2020-02-06T02:19:56.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:56.557+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.559036+0000) 2020-02-06T02:19:56.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:56.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.567344+0000) 2020-02-06T02:19:56.579 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:56.566+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.567710+0000) 2020-02-06T02:19:56.579 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:56.558+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.559596+0000) 2020-02-06T02:19:56.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:56.560+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.561137+0000) 2020-02-06T02:19:56.580 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:19:56.560+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.561630+0000) 2020-02-06T02:19:56.580 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:56.560+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.561778+0000) 2020-02-06T02:19:56.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:19:56.558+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.560281+0000) 2020-02-06T02:19:56.581 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:56.559+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.561390+0000) 2020-02-06T02:19:56.581 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:56.560+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.562043+0000) 2020-02-06T02:19:56.581 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:19:56.561+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.563269+0000) 2020-02-06T02:19:56.582 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:56.561+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.563396+0000) 2020-02-06T02:19:56.582 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:19:56.562+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:56.563967+0000) 2020-02-06T02:19:56.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:19:56.581+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229571 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:19:57.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:19:57.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.163519+0000) 2020-02-06T02:19:57.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:19:57.438+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.438697+0000) 2020-02-06T02:19:57.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:19:57.438+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.438737+0000) 2020-02-06T02:19:57.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:19:57.438+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.438888+0000) 2020-02-06T02:19:57.669 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:19:57.667+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.668581+0000) 2020-02-06T02:19:57.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:19:57.691+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:57.691640+0000) 2020-02-06T02:19:59.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:19:59.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:19:59.807747+0000) 2020-02-06T02:20:00.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:00.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:00.053398+0000) 2020-02-06T02:20:00.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:00.664+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:00.664966+0000) 2020-02-06T02:20:00.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:00.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:00.941582+0000) 2020-02-06T02:20:01.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:01.590+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.590998+0000) 2020-02-06T02:20:01.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:01.590+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.591210+0000) 2020-02-06T02:20:01.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:01.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.589326+0000) 2020-02-06T02:20:01.593 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:01.589+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.590254+0000) 2020-02-06T02:20:01.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:01.592+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.592702+0000) 2020-02-06T02:20:01.594 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:01.593+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.593518+0000) 2020-02-06T02:20:01.594 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:01.592+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.593490+0000) 2020-02-06T02:20:01.595 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:01.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.593635+0000) 2020-02-06T02:20:01.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:01.593+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.594190+0000) 2020-02-06T02:20:01.596 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:01.593+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.594211+0000) 2020-02-06T02:20:01.596 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:01.593+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.594708+0000) 2020-02-06T02:20:01.596 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:01.593+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.594841+0000) 2020-02-06T02:20:01.596 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:01.594+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.595130+0000) 2020-02-06T02:20:01.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:01.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:01.600447+0000) 2020-02-06T02:20:01.615 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:01.613+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229614 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:03.524 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:03.522+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:03.523207+0000) 2020-02-06T02:20:05.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:05.630+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:05.631807+0000) 2020-02-06T02:20:06.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:06.186+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.187233+0000) 2020-02-06T02:20:06.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:06.549+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.550195+0000) 2020-02-06T02:20:06.620 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:06.618+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.619028+0000) 2020-02-06T02:20:06.621 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:06.619+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.620799+0000) 2020-02-06T02:20:06.622 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:06.620+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.621216+0000) 2020-02-06T02:20:06.626 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:06.623+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.623421+0000) 2020-02-06T02:20:06.627 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:06.623+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.624298+0000) 2020-02-06T02:20:06.627 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:06.620+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.621943+0000) 2020-02-06T02:20:06.627 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:06.621+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.622248+0000) 2020-02-06T02:20:06.627 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:06.621+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.622636+0000) 2020-02-06T02:20:06.627 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:06.623+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.624416+0000) 2020-02-06T02:20:06.628 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:06.623+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.624907+0000) 2020-02-06T02:20:06.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:06.625+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.625699+0000) 2020-02-06T02:20:06.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:06.626+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.626531+0000) 2020-02-06T02:20:06.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:06.625+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.626690+0000) 2020-02-06T02:20:06.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:06.628+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:06.629193+0000) 2020-02-06T02:20:06.646 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:06.644+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229653 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:07.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:07.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.163727+0000) 2020-02-06T02:20:07.440 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:07.438+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.438929+0000) 2020-02-06T02:20:07.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:07.438+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.438943+0000) 2020-02-06T02:20:07.440 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:07.438+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.439095+0000) 2020-02-06T02:20:07.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:07.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.668868+0000) 2020-02-06T02:20:07.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:07.691+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:07.691898+0000) 2020-02-06T02:20:09.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:09.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:09.807993+0000) 2020-02-06T02:20:10.054 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:10.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:10.053629+0000) 2020-02-06T02:20:10.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:10.664+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:10.665202+0000) 2020-02-06T02:20:10.942 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:10.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:10.941822+0000) 2020-02-06T02:20:11.653 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:11.652+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.652698+0000) 2020-02-06T02:20:11.654 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:11.653+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.653222+0000) 2020-02-06T02:20:11.654 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:11.653+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.653600+0000) 2020-02-06T02:20:11.655 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:11.653+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.653907+0000) 2020-02-06T02:20:11.655 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:11.654+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.654711+0000) 2020-02-06T02:20:11.655 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:11.655+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.655241+0000) 2020-02-06T02:20:11.656 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:11.655+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.655503+0000) 2020-02-06T02:20:11.656 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:11.655+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.655811+0000) 2020-02-06T02:20:11.657 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:11.656+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.656421+0000) 2020-02-06T02:20:11.657 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:11.656+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.656624+0000) 2020-02-06T02:20:11.658 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:11.656+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.656931+0000) 2020-02-06T02:20:11.658 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:11.656+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.657063+0000) 2020-02-06T02:20:11.658 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:11.656+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.657397+0000) 2020-02-06T02:20:11.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:11.660+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:11.661468+0000) 2020-02-06T02:20:11.676 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:11.674+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229696 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:13.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:13.523+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:13.523417+0000) 2020-02-06T02:20:15.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:15.642+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:15.642310+0000) 2020-02-06T02:20:16.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:16.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.187481+0000) 2020-02-06T02:20:16.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:16.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.550492+0000) 2020-02-06T02:20:16.682 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:16.680+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.681325+0000) 2020-02-06T02:20:16.683 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:16.682+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.682417+0000) 2020-02-06T02:20:16.684 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:16.682+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.683153+0000) 2020-02-06T02:20:16.684 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:16.683+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.683975+0000) 2020-02-06T02:20:16.685 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:16.683+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.684169+0000) 2020-02-06T02:20:16.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:16.683+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.684581+0000) 2020-02-06T02:20:16.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:16.684+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.684969+0000) 2020-02-06T02:20:16.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:16.684+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.685387+0000) 2020-02-06T02:20:16.686 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:16.684+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.685825+0000) 2020-02-06T02:20:16.687 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:16.685+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.686296+0000) 2020-02-06T02:20:16.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:16.685+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.686750+0000) 2020-02-06T02:20:16.689 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:16.685+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.686660+0000) 2020-02-06T02:20:16.689 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:16.686+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.687303+0000) 2020-02-06T02:20:16.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:16.690+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:16.691607+0000) 2020-02-06T02:20:16.705 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:16.704+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229735 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:16.838 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:20:16.854 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:20:17.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:17.162+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.163958+0000) 2020-02-06T02:20:17.440 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:17.438+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.439184+0000) 2020-02-06T02:20:17.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:17.438+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.439180+0000) 2020-02-06T02:20:17.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:17.438+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.439339+0000) 2020-02-06T02:20:17.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:17.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.669127+0000) 2020-02-06T02:20:17.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:17.691+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:17.692144+0000) 2020-02-06T02:20:19.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:19.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:19.808239+0000) 2020-02-06T02:20:20.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:20.052+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:20.053878+0000) 2020-02-06T02:20:20.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:20.665+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:20.665484+0000) 2020-02-06T02:20:20.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:20.940+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:20.942043+0000) 2020-02-06T02:20:21.711 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:21.709+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.710650+0000) 2020-02-06T02:20:21.712 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:21.709+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.710916+0000) 2020-02-06T02:20:21.713 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:21.711+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.711650+0000) 2020-02-06T02:20:21.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:21.710+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.711323+0000) 2020-02-06T02:20:21.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:21.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.711486+0000) 2020-02-06T02:20:21.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:21.710+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.711832+0000) 2020-02-06T02:20:21.714 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:21.712+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.712505+0000) 2020-02-06T02:20:21.715 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:21.713+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.714313+0000) 2020-02-06T02:20:21.715 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:21.713+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.714795+0000) 2020-02-06T02:20:21.716 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:21.714+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.714466+0000) 2020-02-06T02:20:21.716 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:21.714+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.715159+0000) 2020-02-06T02:20:21.716 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:21.714+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.714970+0000) 2020-02-06T02:20:21.717 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:21.715+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.715946+0000) 2020-02-06T02:20:21.720 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:21.719+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:21.720242+0000) 2020-02-06T02:20:21.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:21.734+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229779 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:23.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:23.522+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:23.523716+0000) 2020-02-06T02:20:25.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:25.650+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:25.650965+0000) 2020-02-06T02:20:26.188 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:26.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.187728+0000) 2020-02-06T02:20:26.551 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:26.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.550691+0000) 2020-02-06T02:20:26.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:26.738+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.739484+0000) 2020-02-06T02:20:26.741 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:26.739+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.740926+0000) 2020-02-06T02:20:26.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:26.741+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.741488+0000) 2020-02-06T02:20:26.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:26.741+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.742437+0000) 2020-02-06T02:20:26.744 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:26.742+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.743257+0000) 2020-02-06T02:20:26.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:26.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.743387+0000) 2020-02-06T02:20:26.744 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:26.742+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.743691+0000) 2020-02-06T02:20:26.745 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:26.743+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.744177+0000) 2020-02-06T02:20:26.745 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:26.743+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.744312+0000) 2020-02-06T02:20:26.746 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:26.743+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.744937+0000) 2020-02-06T02:20:26.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:26.744+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.745317+0000) 2020-02-06T02:20:26.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:26.744+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.745539+0000) 2020-02-06T02:20:26.747 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:26.745+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.746074+0000) 2020-02-06T02:20:26.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:26.748+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:26.749724+0000) 2020-02-06T02:20:26.766 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:26.765+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229819 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:27.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:27.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.164181+0000) 2020-02-06T02:20:27.440 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:27.439+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.439419+0000) 2020-02-06T02:20:27.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:27.439+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.439422+0000) 2020-02-06T02:20:27.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:27.439+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.439609+0000) 2020-02-06T02:20:27.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:27.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.669372+0000) 2020-02-06T02:20:27.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:27.691+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:27.692379+0000) 2020-02-06T02:20:29.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:29.807+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:29.808484+0000) 2020-02-06T02:20:30.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:30.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:30.054147+0000) 2020-02-06T02:20:30.666 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:30.665+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:30.665746+0000) 2020-02-06T02:20:30.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:30.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:30.942314+0000) 2020-02-06T02:20:31.772 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:31.769+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.770984+0000) 2020-02-06T02:20:31.773 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:31.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.771868+0000) 2020-02-06T02:20:31.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:31.771+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.772362+0000) 2020-02-06T02:20:31.774 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:31.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.773279+0000) 2020-02-06T02:20:31.774 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:31.772+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.773855+0000) 2020-02-06T02:20:31.775 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:31.774+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.774581+0000) 2020-02-06T02:20:31.777 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:31.774+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.775481+0000) 2020-02-06T02:20:31.777 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:31.774+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.774804+0000) 2020-02-06T02:20:31.777 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:31.774+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.775342+0000) 2020-02-06T02:20:31.778 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:31.775+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.776202+0000) 2020-02-06T02:20:31.779 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:31.775+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.776425+0000) 2020-02-06T02:20:31.779 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:31.776+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.777072+0000) 2020-02-06T02:20:31.780 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:31.777+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.777804+0000) 2020-02-06T02:20:31.785 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:31.783+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:31.785032+0000) 2020-02-06T02:20:31.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:31.796+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229862 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:33.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:33.523+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:33.523981+0000) 2020-02-06T02:20:35.663 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:35.660+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:35.662027+0000) 2020-02-06T02:20:36.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:36.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.187995+0000) 2020-02-06T02:20:36.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:36.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.550963+0000) 2020-02-06T02:20:36.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:36.800+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.801617+0000) 2020-02-06T02:20:36.803 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:36.800+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.802024+0000) 2020-02-06T02:20:36.804 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:36.803+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.803629+0000) 2020-02-06T02:20:36.805 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:36.803+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.804603+0000) 2020-02-06T02:20:36.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:36.804+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.804509+0000) 2020-02-06T02:20:36.806 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:36.804+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.805729+0000) 2020-02-06T02:20:36.807 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:36.805+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.806150+0000) 2020-02-06T02:20:36.807 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:36.806+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.806822+0000) 2020-02-06T02:20:36.808 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:36.807+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.807611+0000) 2020-02-06T02:20:36.809 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:36.807+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.808177+0000) 2020-02-06T02:20:36.809 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:36.807+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.808305+0000) 2020-02-06T02:20:36.810 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:36.807+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.808276+0000) 2020-02-06T02:20:36.810 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:36.807+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.808540+0000) 2020-02-06T02:20:36.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:36.810+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:36.811986+0000) 2020-02-06T02:20:36.825 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:36.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229901 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:37.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:37.163+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.164392+0000) 2020-02-06T02:20:37.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:37.439+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.439647+0000) 2020-02-06T02:20:37.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:37.439+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.439818+0000) 2020-02-06T02:20:37.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:37.439+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.439734+0000) 2020-02-06T02:20:37.670 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:37.668+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.669581+0000) 2020-02-06T02:20:37.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:37.691+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:37.692618+0000) 2020-02-06T02:20:39.809 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:39.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:39.808712+0000) 2020-02-06T02:20:40.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:40.053+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:40.054390+0000) 2020-02-06T02:20:40.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:40.665+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:40.666040+0000) 2020-02-06T02:20:40.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:40.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:40.942548+0000) 2020-02-06T02:20:41.834 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:41.829+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.830765+0000) 2020-02-06T02:20:41.946 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:41.829+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.831278+0000) 2020-02-06T02:20:41.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:41.828+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.829581+0000) 2020-02-06T02:20:41.947 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:41.831+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.832539+0000) 2020-02-06T02:20:41.948 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:41.832+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.833531+0000) 2020-02-06T02:20:41.948 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:41.832+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.834142+0000) 2020-02-06T02:20:41.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:41.833+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.835305+0000) 2020-02-06T02:20:41.949 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:41.835+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.836576+0000) 2020-02-06T02:20:41.949 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:41.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229945 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:41.949 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:41.832+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.833737+0000) 2020-02-06T02:20:41.949 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:41.833+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.834636+0000) 2020-02-06T02:20:41.950 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:41.840+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.841544+0000) 2020-02-06T02:20:41.950 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:41.834+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.835624+0000) 2020-02-06T02:20:41.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:41.835+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.836237+0000) 2020-02-06T02:20:41.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:41.835+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:41.836132+0000) 2020-02-06T02:20:43.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:43.523+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:43.524204+0000) 2020-02-06T02:20:45.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:45.673+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:45.673757+0000) 2020-02-06T02:20:46.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:46.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.188263+0000) 2020-02-06T02:20:46.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:46.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.551220+0000) 2020-02-06T02:20:46.861 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:46.859+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.860413+0000) 2020-02-06T02:20:46.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:46.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.860930+0000) 2020-02-06T02:20:46.862 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:46.860+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.861370+0000) 2020-02-06T02:20:46.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:46.860+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.861560+0000) 2020-02-06T02:20:46.863 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:46.860+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.861999+0000) 2020-02-06T02:20:46.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:46.861+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.862388+0000) 2020-02-06T02:20:46.864 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:46.861+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.862888+0000) 2020-02-06T02:20:46.864 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:46.862+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.863523+0000) 2020-02-06T02:20:46.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:46.863+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.864791+0000) 2020-02-06T02:20:46.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:46.863+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.865169+0000) 2020-02-06T02:20:46.866 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:46.864+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.865797+0000) 2020-02-06T02:20:46.866 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:46.865+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.866053+0000) 2020-02-06T02:20:46.867 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:46.865+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.866653+0000) 2020-02-06T02:20:46.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:46.867+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:46.868837+0000) 2020-02-06T02:20:46.885 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:46.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 229985 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:47.033 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:20:47.039 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:20:47.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:47.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.164682+0000) 2020-02-06T02:20:47.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:47.438+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.439865+0000) 2020-02-06T02:20:47.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:47.438+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.440053+0000) 2020-02-06T02:20:47.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:47.438+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.439958+0000) 2020-02-06T02:20:47.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:47.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.669786+0000) 2020-02-06T02:20:47.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:47.692+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:47.692828+0000) 2020-02-06T02:20:49.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:49.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:49.808952+0000) 2020-02-06T02:20:50.055 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:50.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:50.054643+0000) 2020-02-06T02:20:50.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:50.665+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:50.666322+0000) 2020-02-06T02:20:50.943 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:50.941+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:50.942795+0000) 2020-02-06T02:20:51.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:51.889+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.890445+0000) 2020-02-06T02:20:51.891 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:51.889+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.891008+0000) 2020-02-06T02:20:51.892 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:51.890+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.891724+0000) 2020-02-06T02:20:51.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:51.890+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.891972+0000) 2020-02-06T02:20:51.893 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:51.891+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.892543+0000) 2020-02-06T02:20:51.893 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:51.891+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.892548+0000) 2020-02-06T02:20:51.894 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:51.891+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.892900+0000) 2020-02-06T02:20:51.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:51.891+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.893128+0000) 2020-02-06T02:20:51.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:51.892+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.893818+0000) 2020-02-06T02:20:51.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:51.892+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.893847+0000) 2020-02-06T02:20:51.895 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:51.893+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.894181+0000) 2020-02-06T02:20:51.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:51.893+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.894463+0000) 2020-02-06T02:20:51.896 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:51.893+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.895078+0000) 2020-02-06T02:20:51.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:51.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:51.899765+0000) 2020-02-06T02:20:51.915 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:51.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230028 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:53.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:53.524+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:53.524469+0000) 2020-02-06T02:20:55.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:55.681+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:55.681910+0000) 2020-02-06T02:20:56.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:56.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.188501+0000) 2020-02-06T02:20:56.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:56.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.551490+0000) 2020-02-06T02:20:56.923 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:20:56.920+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.921130+0000) 2020-02-06T02:20:56.923 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:56.920+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.921633+0000) 2020-02-06T02:20:56.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:20:56.920+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.922050+0000) 2020-02-06T02:20:56.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:20:56.921+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.922643+0000) 2020-02-06T02:20:56.924 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:56.921+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.922957+0000) 2020-02-06T02:20:56.924 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:56.922+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.923414+0000) 2020-02-06T02:20:56.925 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:20:56.922+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.923979+0000) 2020-02-06T02:20:56.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:56.923+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.924451+0000) 2020-02-06T02:20:56.925 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:56.923+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.924535+0000) 2020-02-06T02:20:56.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:56.923+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.924743+0000) 2020-02-06T02:20:56.926 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:56.924+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.925502+0000) 2020-02-06T02:20:56.926 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:56.924+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.925859+0000) 2020-02-06T02:20:56.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:20:56.924+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.926226+0000) 2020-02-06T02:20:56.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:56.931+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:56.932507+0000) 2020-02-06T02:20:56.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:20:56.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:20:57.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:20:57.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.164903+0000) 2020-02-06T02:20:57.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:20:57.439+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.440098+0000) 2020-02-06T02:20:57.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:20:57.439+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.440168+0000) 2020-02-06T02:20:57.441 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:20:57.439+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.440292+0000) 2020-02-06T02:20:57.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:20:57.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.670001+0000) 2020-02-06T02:20:57.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:20:57.692+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:57.693045+0000) 2020-02-06T02:20:59.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:20:59.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:20:59.809194+0000) 2020-02-06T02:21:00.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:00.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:00.054867+0000) 2020-02-06T02:21:00.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:00.665+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:00.666578+0000) 2020-02-06T02:21:00.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:00.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:00.943027+0000) 2020-02-06T02:21:01.949 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:01.947+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.948573+0000) 2020-02-06T02:21:01.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:01.948+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.949768+0000) 2020-02-06T02:21:01.951 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:01.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.950299+0000) 2020-02-06T02:21:01.952 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:01.950+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.951227+0000) 2020-02-06T02:21:01.952 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:01.950+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.951688+0000) 2020-02-06T02:21:01.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:01.951+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.952735+0000) 2020-02-06T02:21:01.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:01.951+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.953232+0000) 2020-02-06T02:21:01.954 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:01.952+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.953850+0000) 2020-02-06T02:21:01.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:01.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.954041+0000) 2020-02-06T02:21:01.955 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:01.952+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.954390+0000) 2020-02-06T02:21:01.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:01.953+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.954619+0000) 2020-02-06T02:21:01.956 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:01.951+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.952917+0000) 2020-02-06T02:21:01.956 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:01.953+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.954561+0000) 2020-02-06T02:21:01.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:01.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:01.957289+0000) 2020-02-06T02:21:01.974 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:01.972+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230111 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:03.525 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:03.524+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:03.524661+0000) 2020-02-06T02:21:05.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:05.693+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:05.693526+0000) 2020-02-06T02:21:06.189 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:06.187+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.188737+0000) 2020-02-06T02:21:06.552 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:06.550+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.551728+0000) 2020-02-06T02:21:06.979 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:06.978+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.978510+0000) 2020-02-06T02:21:06.980 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:06.979+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.979206+0000) 2020-02-06T02:21:06.981 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:06.979+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.980766+0000) 2020-02-06T02:21:06.982 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:06.980+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.981041+0000) 2020-02-06T02:21:06.982 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:06.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.981352+0000) 2020-02-06T02:21:06.982 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:06.980+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.981897+0000) 2020-02-06T02:21:06.983 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:06.982+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.982442+0000) 2020-02-06T02:21:06.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:06.982+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.982785+0000) 2020-02-06T02:21:06.984 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:06.982+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.983025+0000) 2020-02-06T02:21:06.984 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:06.982+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.983906+0000) 2020-02-06T02:21:06.985 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:06.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.984622+0000) 2020-02-06T02:21:06.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:06.983+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.984985+0000) 2020-02-06T02:21:06.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:06.984+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.985476+0000) 2020-02-06T02:21:06.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:06.991+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:06.991119+0000) 2020-02-06T02:21:07.004 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:07.002+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:07.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:07.165+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.165186+0000) 2020-02-06T02:21:07.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:07.439+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.440331+0000) 2020-02-06T02:21:07.441 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:07.439+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.440468+0000) 2020-02-06T02:21:07.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:07.439+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.440531+0000) 2020-02-06T02:21:07.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:07.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.670204+0000) 2020-02-06T02:21:07.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:07.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:07.693282+0000) 2020-02-06T02:21:09.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:09.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:09.809408+0000) 2020-02-06T02:21:10.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:10.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:10.055129+0000) 2020-02-06T02:21:10.667 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:10.666+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:10.666831+0000) 2020-02-06T02:21:10.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:10.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:10.943289+0000) 2020-02-06T02:21:12.009 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:12.007+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.008502+0000) 2020-02-06T02:21:12.010 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:12.008+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.009377+0000) 2020-02-06T02:21:12.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:12.009+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.010617+0000) 2020-02-06T02:21:12.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:12.009+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.010674+0000) 2020-02-06T02:21:12.012 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:12.010+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.011472+0000) 2020-02-06T02:21:12.012 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:12.010+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.011472+0000) 2020-02-06T02:21:12.013 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:12.011+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.012167+0000) 2020-02-06T02:21:12.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:12.010+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.012075+0000) 2020-02-06T02:21:12.014 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:12.012+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.013449+0000) 2020-02-06T02:21:12.014 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:12.011+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.013133+0000) 2020-02-06T02:21:12.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:12.012+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.014093+0000) 2020-02-06T02:21:12.016 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:12.013+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.015033+0000) 2020-02-06T02:21:12.016 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:12.013+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.015143+0000) 2020-02-06T02:21:12.019 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:12.017+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:12.018688+0000) 2020-02-06T02:21:12.034 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:12.032+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230194 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:13.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:13.523+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:13.524939+0000) 2020-02-06T02:21:15.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:15.700+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:15.701618+0000) 2020-02-06T02:21:16.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:16.188+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:16.188986+0000) 2020-02-06T02:21:16.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:16.551+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:16.551989+0000) 2020-02-06T02:21:17.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:17.037+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.038948+0000) 2020-02-06T02:21:17.040 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:17.038+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.039459+0000) 2020-02-06T02:21:17.041 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:17.039+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.040383+0000) 2020-02-06T02:21:17.041 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:17.039+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.040716+0000) 2020-02-06T02:21:17.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:17.040+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.041733+0000) 2020-02-06T02:21:17.043 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:17.040+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.042046+0000) 2020-02-06T02:21:17.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:17.041+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.042482+0000) 2020-02-06T02:21:17.044 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:17.040+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.042384+0000) 2020-02-06T02:21:17.045 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:17.042+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.043920+0000) 2020-02-06T02:21:17.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:17.042+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.044090+0000) 2020-02-06T02:21:17.046 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:17.043+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.044871+0000) 2020-02-06T02:21:17.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:17.043+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.045020+0000) 2020-02-06T02:21:17.047 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:17.043+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.045258+0000) 2020-02-06T02:21:17.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:17.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.048569+0000) 2020-02-06T02:21:17.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:17.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:17.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:17.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.165395+0000) 2020-02-06T02:21:17.289 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:21:17.295 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:21:17.441 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:17.439+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.440546+0000) 2020-02-06T02:21:17.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:17.439+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.440725+0000) 2020-02-06T02:21:17.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:17.439+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.440704+0000) 2020-02-06T02:21:17.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:17.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.670410+0000) 2020-02-06T02:21:17.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:17.692+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:17.693487+0000) 2020-02-06T02:21:19.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:19.808+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:19.809644+0000) 2020-02-06T02:21:20.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:20.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:20.055354+0000) 2020-02-06T02:21:20.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:20.666+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:20.667123+0000) 2020-02-06T02:21:20.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:20.942+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:20.943517+0000) 2020-02-06T02:21:22.070 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:22.068+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.069484+0000) 2020-02-06T02:21:22.071 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:22.069+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.070446+0000) 2020-02-06T02:21:22.071 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:22.069+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.070684+0000) 2020-02-06T02:21:22.072 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:22.069+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.070998+0000) 2020-02-06T02:21:22.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:22.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.071532+0000) 2020-02-06T02:21:22.072 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:22.070+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.071787+0000) 2020-02-06T02:21:22.073 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:22.071+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.072287+0000) 2020-02-06T02:21:22.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:22.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.072591+0000) 2020-02-06T02:21:22.074 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:22.072+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.073543+0000) 2020-02-06T02:21:22.074 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:22.072+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.073814+0000) 2020-02-06T02:21:22.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:22.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.074079+0000) 2020-02-06T02:21:22.078 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:22.073+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.074475+0000) 2020-02-06T02:21:22.078 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:22.072+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.074352+0000) 2020-02-06T02:21:22.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:22.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:22.079007+0000) 2020-02-06T02:21:22.093 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:22.091+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230277 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:23.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:23.524+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:23.525192+0000) 2020-02-06T02:21:25.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:25.710+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:25.711591+0000) 2020-02-06T02:21:26.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:26.188+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:26.189237+0000) 2020-02-06T02:21:26.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:26.551+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:26.552246+0000) 2020-02-06T02:21:27.099 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:27.097+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.098885+0000) 2020-02-06T02:21:27.100 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:27.099+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.100119+0000) 2020-02-06T02:21:27.101 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:27.100+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.100553+0000) 2020-02-06T02:21:27.102 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:27.100+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.101325+0000) 2020-02-06T02:21:27.102 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:27.100+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.101363+0000) 2020-02-06T02:21:27.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:27.100+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.101491+0000) 2020-02-06T02:21:27.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:27.100+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.101727+0000) 2020-02-06T02:21:27.103 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:27.100+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.101964+0000) 2020-02-06T02:21:27.104 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:27.101+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.102647+0000) 2020-02-06T02:21:27.104 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:27.101+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.102369+0000) 2020-02-06T02:21:27.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:27.101+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.102416+0000) 2020-02-06T02:21:27.104 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:27.101+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.103206+0000) 2020-02-06T02:21:27.105 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:27.102+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.104403+0000) 2020-02-06T02:21:27.109 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:27.107+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.108431+0000) 2020-02-06T02:21:27.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:27.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:27.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:27.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.165659+0000) 2020-02-06T02:21:27.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:27.440+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.440798+0000) 2020-02-06T02:21:27.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:27.440+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.440988+0000) 2020-02-06T02:21:27.442 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:27.440+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.440995+0000) 2020-02-06T02:21:27.671 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:27.669+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.670683+0000) 2020-02-06T02:21:27.694 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:27.692+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:27.693692+0000) 2020-02-06T02:21:29.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:29.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:29.809886+0000) 2020-02-06T02:21:30.056 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:30.054+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:30.055588+0000) 2020-02-06T02:21:30.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:30.666+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:30.667376+0000) 2020-02-06T02:21:30.944 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:30.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:30.943753+0000) 2020-02-06T02:21:32.129 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:32.127+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.128322+0000) 2020-02-06T02:21:32.129 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:32.128+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.128837+0000) 2020-02-06T02:21:32.130 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:32.128+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.129406+0000) 2020-02-06T02:21:32.131 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:32.129+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.130701+0000) 2020-02-06T02:21:32.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:32.130+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.130543+0000) 2020-02-06T02:21:32.132 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:32.130+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.131239+0000) 2020-02-06T02:21:32.132 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:32.131+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.131638+0000) 2020-02-06T02:21:32.133 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:32.130+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.132058+0000) 2020-02-06T02:21:32.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:32.131+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.133047+0000) 2020-02-06T02:21:32.134 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:32.132+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.133120+0000) 2020-02-06T02:21:32.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:32.132+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.133384+0000) 2020-02-06T02:21:32.135 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:32.133+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.133814+0000) 2020-02-06T02:21:32.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:32.134+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.134520+0000) 2020-02-06T02:21:32.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:32.137+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:32.138752+0000) 2020-02-06T02:21:32.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:32.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230360 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:33.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:33.524+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:33.525422+0000) 2020-02-06T02:21:35.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:35.719+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:35.719951+0000) 2020-02-06T02:21:36.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:36.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:36.189478+0000) 2020-02-06T02:21:36.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:36.552+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:36.552496+0000) 2020-02-06T02:21:37.158 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:37.156+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.157641+0000) 2020-02-06T02:21:37.159 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:37.158+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.158851+0000) 2020-02-06T02:21:37.160 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:37.158+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.159098+0000) 2020-02-06T02:21:37.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:37.159+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.160412+0000) 2020-02-06T02:21:37.161 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:37.159+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.160631+0000) 2020-02-06T02:21:37.162 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:37.160+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.161339+0000) 2020-02-06T02:21:37.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:37.161+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.162249+0000) 2020-02-06T02:21:37.163 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:37.161+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.162043+0000) 2020-02-06T02:21:37.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:37.162+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.162546+0000) 2020-02-06T02:21:37.164 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:37.162+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.163460+0000) 2020-02-06T02:21:37.164 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:37.162+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.163587+0000) 2020-02-06T02:21:37.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:37.163+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.163720+0000) 2020-02-06T02:21:37.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:37.163+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.163918+0000) 2020-02-06T02:21:37.166 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:37.164+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.165805+0000) 2020-02-06T02:21:37.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:37.165+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.166731+0000) 2020-02-06T02:21:37.182 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:37.180+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:37.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:37.440+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.441075+0000) 2020-02-06T02:21:37.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:37.440+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.441255+0000) 2020-02-06T02:21:37.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:37.440+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.441261+0000) 2020-02-06T02:21:37.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:37.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.670886+0000) 2020-02-06T02:21:37.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:37.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:37.693934+0000) 2020-02-06T02:21:39.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:39.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:39.810133+0000) 2020-02-06T02:21:40.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:40.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:40.055835+0000) 2020-02-06T02:21:40.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:40.667+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:40.667706+0000) 2020-02-06T02:21:40.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:40.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:40.943989+0000) 2020-02-06T02:21:42.188 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:42.186+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.187123+0000) 2020-02-06T02:21:42.188 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:42.187+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.187588+0000) 2020-02-06T02:21:42.189 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:42.187+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.188147+0000) 2020-02-06T02:21:42.189 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:42.187+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.188400+0000) 2020-02-06T02:21:42.190 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:42.188+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.189712+0000) 2020-02-06T02:21:42.191 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:42.190+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.190602+0000) 2020-02-06T02:21:42.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:42.190+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.190833+0000) 2020-02-06T02:21:42.192 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:42.190+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.191006+0000) 2020-02-06T02:21:42.193 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:42.190+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.191598+0000) 2020-02-06T02:21:42.193 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:42.191+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.191772+0000) 2020-02-06T02:21:42.193 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:42.191+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.192208+0000) 2020-02-06T02:21:42.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:42.191+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.192399+0000) 2020-02-06T02:21:42.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:42.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.192834+0000) 2020-02-06T02:21:42.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:42.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:42.195882+0000) 2020-02-06T02:21:42.212 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:42.210+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230441 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:43.526 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:43.525+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:43.525660+0000) 2020-02-06T02:21:45.729 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:45.727+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:45.728229+0000) 2020-02-06T02:21:46.190 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:46.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:46.189731+0000) 2020-02-06T02:21:46.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:46.552+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:46.552758+0000) 2020-02-06T02:21:47.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:47.165+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.165949+0000) 2020-02-06T02:21:47.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:47.214+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.216024+0000) 2020-02-06T02:21:47.217 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:47.215+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.216645+0000) 2020-02-06T02:21:47.218 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:47.216+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.217594+0000) 2020-02-06T02:21:47.218 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:47.216+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.218059+0000) 2020-02-06T02:21:47.219 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:47.217+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.218477+0000) 2020-02-06T02:21:47.221 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:47.218+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.218888+0000) 2020-02-06T02:21:47.221 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:47.219+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.219960+0000) 2020-02-06T02:21:47.222 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:47.220+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.220893+0000) 2020-02-06T02:21:47.222 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:47.220+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.221365+0000) 2020-02-06T02:21:47.222 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:47.221+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.221820+0000) 2020-02-06T02:21:47.223 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:47.222+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.222709+0000) 2020-02-06T02:21:47.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:47.222+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.223045+0000) 2020-02-06T02:21:47.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:47.223+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.224989+0000) 2020-02-06T02:21:47.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:47.224+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.225250+0000) 2020-02-06T02:21:47.242 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:47.240+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230482 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:47.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:47.441+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.441288+0000) 2020-02-06T02:21:47.442 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:47.441+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.441472+0000) 2020-02-06T02:21:47.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:47.441+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.441491+0000) 2020-02-06T02:21:47.449 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:21:47.456 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:21:47.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:47.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.671140+0000) 2020-02-06T02:21:47.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:47.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:47.694175+0000) 2020-02-06T02:21:49.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:49.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:49.810388+0000) 2020-02-06T02:21:50.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:50.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:50.056018+0000) 2020-02-06T02:21:50.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:50.667+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:50.667937+0000) 2020-02-06T02:21:50.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:50.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:50.944233+0000) 2020-02-06T02:21:52.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:52.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.246447+0000) 2020-02-06T02:21:52.248 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:52.246+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.247814+0000) 2020-02-06T02:21:52.249 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:52.247+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.248375+0000) 2020-02-06T02:21:52.250 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:52.248+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.249334+0000) 2020-02-06T02:21:52.250 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:52.249+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.249668+0000) 2020-02-06T02:21:52.251 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:52.249+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.250087+0000) 2020-02-06T02:21:52.251 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:52.249+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.250580+0000) 2020-02-06T02:21:52.252 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:52.251+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.251456+0000) 2020-02-06T02:21:52.253 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:52.250+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.251462+0000) 2020-02-06T02:21:52.253 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:52.250+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.251682+0000) 2020-02-06T02:21:52.253 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:52.251+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.252099+0000) 2020-02-06T02:21:52.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:52.251+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.252337+0000) 2020-02-06T02:21:52.254 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:52.252+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.252892+0000) 2020-02-06T02:21:52.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:52.254+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:52.255763+0000) 2020-02-06T02:21:52.271 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:52.270+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230521 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:53.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:53.525+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:53.525895+0000) 2020-02-06T02:21:55.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:55.735+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:55.736739+0000) 2020-02-06T02:21:56.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:56.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:56.190018+0000) 2020-02-06T02:21:56.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:56.552+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:56.553107+0000) 2020-02-06T02:21:57.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:57.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.166210+0000) 2020-02-06T02:21:57.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:57.275+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.276224+0000) 2020-02-06T02:21:57.279 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:57.277+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.278330+0000) 2020-02-06T02:21:57.279 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:21:57.277+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.278741+0000) 2020-02-06T02:21:57.281 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:21:57.279+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.280261+0000) 2020-02-06T02:21:57.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:57.279+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.280948+0000) 2020-02-06T02:21:57.282 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:57.280+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.281181+0000) 2020-02-06T02:21:57.282 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:21:57.281+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.281679+0000) 2020-02-06T02:21:57.283 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:21:57.280+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.281926+0000) 2020-02-06T02:21:57.283 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:21:57.281+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.282464+0000) 2020-02-06T02:21:57.284 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:21:57.282+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.282560+0000) 2020-02-06T02:21:57.284 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:57.282+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.283043+0000) 2020-02-06T02:21:57.284 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:57.281+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.282738+0000) 2020-02-06T02:21:57.284 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:57.283+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.283620+0000) 2020-02-06T02:21:57.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:57.284+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.285722+0000) 2020-02-06T02:21:57.302 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:21:57.300+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230562 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:21:57.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:21:57.441+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.441515+0000) 2020-02-06T02:21:57.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:21:57.441+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.441819+0000) 2020-02-06T02:21:57.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:21:57.441+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.441879+0000) 2020-02-06T02:21:57.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:21:57.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.671344+0000) 2020-02-06T02:21:57.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:21:57.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:57.694341+0000) 2020-02-06T02:21:59.811 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:21:59.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:21:59.810638+0000) 2020-02-06T02:22:00.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:00.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:00.056269+0000) 2020-02-06T02:22:00.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:00.667+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:00.668173+0000) 2020-02-06T02:22:00.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:00.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:00.944483+0000) 2020-02-06T02:22:02.307 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:02.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.306124+0000) 2020-02-06T02:22:02.308 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:02.306+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.307216+0000) 2020-02-06T02:22:02.309 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:02.308+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.308526+0000) 2020-02-06T02:22:02.310 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:02.309+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.309304+0000) 2020-02-06T02:22:02.310 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:02.309+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.309623+0000) 2020-02-06T02:22:02.311 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:02.309+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.310313+0000) 2020-02-06T02:22:02.311 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:02.310+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.310648+0000) 2020-02-06T02:22:02.312 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:02.311+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.311506+0000) 2020-02-06T02:22:02.312 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:02.311+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.311715+0000) 2020-02-06T02:22:02.313 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:02.312+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.312131+0000) 2020-02-06T02:22:02.313 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:02.311+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.312408+0000) 2020-02-06T02:22:02.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:02.312+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.312689+0000) 2020-02-06T02:22:02.314 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:02.313+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.313472+0000) 2020-02-06T02:22:02.314 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:02.313+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:02.314041+0000) 2020-02-06T02:22:02.332 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:02.331+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230602 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:03.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:03.526+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:03.526169+0000) 2020-02-06T02:22:05.746 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:05.744+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:05.745153+0000) 2020-02-06T02:22:06.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:06.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:06.190284+0000) 2020-02-06T02:22:06.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:06.552+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:06.553362+0000) 2020-02-06T02:22:07.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:07.165+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.166450+0000) 2020-02-06T02:22:07.337 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:07.335+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.336648+0000) 2020-02-06T02:22:07.338 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:07.336+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.337914+0000) 2020-02-06T02:22:07.339 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:07.338+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.339148+0000) 2020-02-06T02:22:07.341 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:07.339+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.340639+0000) 2020-02-06T02:22:07.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:07.340+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.341126+0000) 2020-02-06T02:22:07.343 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:07.340+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.341462+0000) 2020-02-06T02:22:07.343 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:07.340+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.341727+0000) 2020-02-06T02:22:07.343 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:07.340+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.341859+0000) 2020-02-06T02:22:07.344 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:07.340+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.341616+0000) 2020-02-06T02:22:07.344 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:07.340+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.342009+0000) 2020-02-06T02:22:07.345 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:07.340+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.342207+0000) 2020-02-06T02:22:07.345 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:07.341+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.342650+0000) 2020-02-06T02:22:07.345 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:07.341+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.342970+0000) 2020-02-06T02:22:07.349 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:07.347+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.349028+0000) 2020-02-06T02:22:07.362 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:07.360+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230644 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:07.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:07.440+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.441693+0000) 2020-02-06T02:22:07.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:07.440+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.442091+0000) 2020-02-06T02:22:07.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:07.440+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.442088+0000) 2020-02-06T02:22:07.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:07.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.671569+0000) 2020-02-06T02:22:07.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:07.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:07.694548+0000) 2020-02-06T02:22:09.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:09.809+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:09.810894+0000) 2020-02-06T02:22:10.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:10.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:10.056493+0000) 2020-02-06T02:22:10.669 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:10.667+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:10.668415+0000) 2020-02-06T02:22:10.945 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:10.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:10.944724+0000) 2020-02-06T02:22:12.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:12.365+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.366758+0000) 2020-02-06T02:22:12.369 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:12.367+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.368464+0000) 2020-02-06T02:22:12.374 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:12.368+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.369523+0000) 2020-02-06T02:22:12.374 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:12.368+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.370355+0000) 2020-02-06T02:22:12.374 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:12.369+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.371251+0000) 2020-02-06T02:22:12.375 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:12.369+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.371287+0000) 2020-02-06T02:22:12.375 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:12.370+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.371578+0000) 2020-02-06T02:22:12.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:12.370+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.371759+0000) 2020-02-06T02:22:12.375 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:12.368+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.369411+0000) 2020-02-06T02:22:12.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:12.369+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.370215+0000) 2020-02-06T02:22:12.376 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:12.369+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.370442+0000) 2020-02-06T02:22:12.376 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:12.369+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.370655+0000) 2020-02-06T02:22:12.376 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:12.371+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.372278+0000) 2020-02-06T02:22:12.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:12.374+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:12.375962+0000) 2020-02-06T02:22:12.391 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:12.389+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230683 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:13.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:13.525+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:13.526395+0000) 2020-02-06T02:22:15.754 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:15.752+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:15.753637+0000) 2020-02-06T02:22:16.191 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:16.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:16.190550+0000) 2020-02-06T02:22:16.554 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:16.552+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:16.553650+0000) 2020-02-06T02:22:17.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:17.165+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.166683+0000) 2020-02-06T02:22:17.398 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:17.395+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.396497+0000) 2020-02-06T02:22:17.398 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:17.396+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.397361+0000) 2020-02-06T02:22:17.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:17.396+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.397902+0000) 2020-02-06T02:22:17.399 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:17.397+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.398553+0000) 2020-02-06T02:22:17.399 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:17.397+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.398858+0000) 2020-02-06T02:22:17.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:17.398+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.399457+0000) 2020-02-06T02:22:17.401 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:17.399+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.400213+0000) 2020-02-06T02:22:17.401 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:17.399+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.400515+0000) 2020-02-06T02:22:17.402 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:17.399+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.400985+0000) 2020-02-06T02:22:17.402 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:17.399+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.401302+0000) 2020-02-06T02:22:17.403 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:17.400+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.401907+0000) 2020-02-06T02:22:17.403 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:17.399+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.401184+0000) 2020-02-06T02:22:17.403 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:17.401+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.402397+0000) 2020-02-06T02:22:17.408 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:17.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.408096+0000) 2020-02-06T02:22:17.422 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:17.420+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230724 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:17.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:17.440+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.441874+0000) 2020-02-06T02:22:17.443 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:17.440+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.442280+0000) 2020-02-06T02:22:17.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:17.440+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.442279+0000) 2020-02-06T02:22:17.614 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:22:17.620 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:22:17.672 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:17.670+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.671739+0000) 2020-02-06T02:22:17.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:17.693+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:17.694755+0000) 2020-02-06T02:22:19.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:19.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:19.811130+0000) 2020-02-06T02:22:20.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:20.055+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:20.056743+0000) 2020-02-06T02:22:20.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:20.667+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:20.668702+0000) 2020-02-06T02:22:20.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:20.943+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:20.944977+0000) 2020-02-06T02:22:22.429 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:22.426+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.427846+0000) 2020-02-06T02:22:22.430 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:22.427+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.428839+0000) 2020-02-06T02:22:22.430 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:22.427+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.429380+0000) 2020-02-06T02:22:22.431 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:22.428+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.429612+0000) 2020-02-06T02:22:22.431 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:22.428+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.429912+0000) 2020-02-06T02:22:22.432 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:22.429+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.430355+0000) 2020-02-06T02:22:22.432 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:22.429+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.430781+0000) 2020-02-06T02:22:22.432 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:22.429+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.430590+0000) 2020-02-06T02:22:22.433 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:22.430+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.431162+0000) 2020-02-06T02:22:22.433 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:22.430+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.431753+0000) 2020-02-06T02:22:22.434 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:22.430+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.431903+0000) 2020-02-06T02:22:22.434 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:22.430+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.432075+0000) 2020-02-06T02:22:22.434 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:22.431+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.432999+0000) 2020-02-06T02:22:22.439 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:22.436+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:22.437900+0000) 2020-02-06T02:22:22.457 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:22.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230763 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:23.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:23.525+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:23.526611+0000) 2020-02-06T02:22:25.763 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:25.760+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:25.761886+0000) 2020-02-06T02:22:26.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:26.189+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:26.190822+0000) 2020-02-06T02:22:26.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:26.553+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:26.553949+0000) 2020-02-06T02:22:27.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:27.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.166928+0000) 2020-02-06T02:22:27.443 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:27.441+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.442110+0000) 2020-02-06T02:22:27.443 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:27.441+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.442475+0000) 2020-02-06T02:22:27.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:27.441+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.442495+0000) 2020-02-06T02:22:27.457 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:27.455+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.456893+0000) 2020-02-06T02:22:27.458 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:27.456+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.457472+0000) 2020-02-06T02:22:27.459 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:27.457+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.458286+0000) 2020-02-06T02:22:27.460 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:27.458+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.459217+0000) 2020-02-06T02:22:27.461 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:27.459+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.460534+0000) 2020-02-06T02:22:27.461 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:27.458+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.460302+0000) 2020-02-06T02:22:27.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:27.460+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.461214+0000) 2020-02-06T02:22:27.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:27.460+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.461723+0000) 2020-02-06T02:22:27.463 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:27.460+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.461731+0000) 2020-02-06T02:22:27.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:27.460+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.462115+0000) 2020-02-06T02:22:27.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:27.460+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.462209+0000) 2020-02-06T02:22:27.464 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:27.461+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.462994+0000) 2020-02-06T02:22:27.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:27.462+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.463462+0000) 2020-02-06T02:22:27.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:27.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.467126+0000) 2020-02-06T02:22:27.486 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:27.482+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230805 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:27.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:27.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.671967+0000) 2020-02-06T02:22:27.695 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:27.694+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:27.694973+0000) 2020-02-06T02:22:29.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:29.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:29.811347+0000) 2020-02-06T02:22:30.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:30.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:30.056978+0000) 2020-02-06T02:22:30.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:30.668+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:30.668896+0000) 2020-02-06T02:22:30.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:30.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:30.945208+0000) 2020-02-06T02:22:32.490 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:32.487+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.488800+0000) 2020-02-06T02:22:32.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:32.488+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.489447+0000) 2020-02-06T02:22:32.491 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:32.488+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.489974+0000) 2020-02-06T02:22:32.491 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:32.489+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.490532+0000) 2020-02-06T02:22:32.492 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:32.490+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.491304+0000) 2020-02-06T02:22:32.494 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:32.492+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.493605+0000) 2020-02-06T02:22:32.495 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:32.493+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.494765+0000) 2020-02-06T02:22:32.496 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:32.493+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.495116+0000) 2020-02-06T02:22:32.496 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:32.494+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.495995+0000) 2020-02-06T02:22:32.497 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:32.494+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.496067+0000) 2020-02-06T02:22:32.497 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:32.495+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.496760+0000) 2020-02-06T02:22:32.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:32.495+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.497038+0000) 2020-02-06T02:22:32.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:32.498+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.499163+0000) 2020-02-06T02:22:32.501 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:32.499+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:32.500244+0000) 2020-02-06T02:22:32.515 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:32.513+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230842 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:33.527 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:33.526+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:33.526844+0000) 2020-02-06T02:22:35.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:35.769+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:35.770397+0000) 2020-02-06T02:22:36.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:36.190+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:36.191094+0000) 2020-02-06T02:22:36.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:36.553+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:36.554202+0000) 2020-02-06T02:22:37.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:37.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.167162+0000) 2020-02-06T02:22:37.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:37.441+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.442365+0000) 2020-02-06T02:22:37.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:37.441+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.442704+0000) 2020-02-06T02:22:37.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:37.441+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.442762+0000) 2020-02-06T02:22:37.522 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:37.519+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.520585+0000) 2020-02-06T02:22:37.522 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:37.520+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.521455+0000) 2020-02-06T02:22:37.523 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:37.520+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.521899+0000) 2020-02-06T02:22:37.523 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:37.521+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.522470+0000) 2020-02-06T02:22:37.523 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:37.521+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.522863+0000) 2020-02-06T02:22:37.524 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:37.522+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.523524+0000) 2020-02-06T02:22:37.525 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:37.522+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.523846+0000) 2020-02-06T02:22:37.526 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:37.523+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.524849+0000) 2020-02-06T02:22:37.526 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:37.524+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.525361+0000) 2020-02-06T02:22:37.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:37.524+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.525945+0000) 2020-02-06T02:22:37.527 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:37.524+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.526269+0000) 2020-02-06T02:22:37.527 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:37.525+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.526743+0000) 2020-02-06T02:22:37.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:37.525+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.526694+0000) 2020-02-06T02:22:37.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:37.529+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.530668+0000) 2020-02-06T02:22:37.549 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:37.544+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230884 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:37.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:37.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.672187+0000) 2020-02-06T02:22:37.696 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:37.694+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:37.695183+0000) 2020-02-06T02:22:39.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:39.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:39.811596+0000) 2020-02-06T02:22:40.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:40.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:40.057307+0000) 2020-02-06T02:22:40.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:40.668+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:40.669144+0000) 2020-02-06T02:22:40.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:40.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:40.945445+0000) 2020-02-06T02:22:42.552 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:42.549+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.550972+0000) 2020-02-06T02:22:42.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:42.550+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.551419+0000) 2020-02-06T02:22:42.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:42.550+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.552404+0000) 2020-02-06T02:22:42.554 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:42.551+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.552912+0000) 2020-02-06T02:22:42.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:42.552+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.553367+0000) 2020-02-06T02:22:42.554 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:42.552+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.554064+0000) 2020-02-06T02:22:42.555 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:42.553+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.554877+0000) 2020-02-06T02:22:42.556 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:42.553+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.555237+0000) 2020-02-06T02:22:42.557 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:42.554+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.556315+0000) 2020-02-06T02:22:42.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:42.555+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.556544+0000) 2020-02-06T02:22:42.558 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:42.555+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.556792+0000) 2020-02-06T02:22:42.558 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:42.555+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.557148+0000) 2020-02-06T02:22:42.559 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:42.556+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.557670+0000) 2020-02-06T02:22:42.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:42.559+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:42.560685+0000) 2020-02-06T02:22:42.577 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:42.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230921 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:43.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:43.526+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:43.527129+0000) 2020-02-06T02:22:45.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:45.778+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:45.779301+0000) 2020-02-06T02:22:46.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:46.190+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:46.191383+0000) 2020-02-06T02:22:46.555 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:46.553+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:46.554477+0000) 2020-02-06T02:22:47.168 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:47.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.167395+0000) 2020-02-06T02:22:47.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:47.442+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.442649+0000) 2020-02-06T02:22:47.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:47.442+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.442977+0000) 2020-02-06T02:22:47.444 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:47.442+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.442974+0000) 2020-02-06T02:22:47.582 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:47.580+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.581486+0000) 2020-02-06T02:22:47.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:47.580+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.581610+0000) 2020-02-06T02:22:47.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:47.580+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.581992+0000) 2020-02-06T02:22:47.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:47.581+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.582912+0000) 2020-02-06T02:22:47.584 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:47.581+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.583207+0000) 2020-02-06T02:22:47.584 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:47.582+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.583742+0000) 2020-02-06T02:22:47.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:47.583+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.584111+0000) 2020-02-06T02:22:47.585 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:47.583+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.584591+0000) 2020-02-06T02:22:47.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:47.583+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.584861+0000) 2020-02-06T02:22:47.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:47.583+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.584969+0000) 2020-02-06T02:22:47.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:47.584+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.586311+0000) 2020-02-06T02:22:47.587 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:47.585+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.586475+0000) 2020-02-06T02:22:47.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:47.585+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.586921+0000) 2020-02-06T02:22:47.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:47.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.590558+0000) 2020-02-06T02:22:47.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:47.604+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 230963 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:47.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:47.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.672406+0000) 2020-02-06T02:22:47.696 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:47.695+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:47.695392+0000) 2020-02-06T02:22:47.801 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:22:47.808 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:22:49.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:49.810+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:49.811855+0000) 2020-02-06T02:22:50.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:50.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:50.057534+0000) 2020-02-06T02:22:50.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:50.668+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:50.669394+0000) 2020-02-06T02:22:50.946 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:50.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:50.945680+0000) 2020-02-06T02:22:52.613 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:52.611+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.612210+0000) 2020-02-06T02:22:52.613 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:52.611+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.612888+0000) 2020-02-06T02:22:52.614 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:52.612+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.613416+0000) 2020-02-06T02:22:52.614 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:52.612+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.613727+0000) 2020-02-06T02:22:52.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:52.613+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.614186+0000) 2020-02-06T02:22:52.615 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:52.613+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.614899+0000) 2020-02-06T02:22:52.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:52.613+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.615268+0000) 2020-02-06T02:22:52.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:52.615+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.616483+0000) 2020-02-06T02:22:52.618 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:52.615+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.616785+0000) 2020-02-06T02:22:52.618 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:52.616+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.617518+0000) 2020-02-06T02:22:52.619 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:52.616+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.617966+0000) 2020-02-06T02:22:52.619 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:52.616+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.617994+0000) 2020-02-06T02:22:52.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:52.616+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.618320+0000) 2020-02-06T02:22:52.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:52.620+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:52.621435+0000) 2020-02-06T02:22:52.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:52.636+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231001 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:53.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:53.527+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:53.527336+0000) 2020-02-06T02:22:55.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:55.787+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:55.787817+0000) 2020-02-06T02:22:56.192 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:56.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:56.191648+0000) 2020-02-06T02:22:56.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:56.554+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:56.554674+0000) 2020-02-06T02:22:57.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:57.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.167625+0000) 2020-02-06T02:22:57.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:57.442+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.442909+0000) 2020-02-06T02:22:57.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:57.442+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.443198+0000) 2020-02-06T02:22:57.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:57.442+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.443224+0000) 2020-02-06T02:22:57.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:57.644+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.644434+0000) 2020-02-06T02:22:57.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:57.644+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.644937+0000) 2020-02-06T02:22:57.646 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:22:57.645+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.645113+0000) 2020-02-06T02:22:57.646 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:57.645+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.645523+0000) 2020-02-06T02:22:57.647 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:22:57.645+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.645799+0000) 2020-02-06T02:22:57.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:57.646+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.646269+0000) 2020-02-06T02:22:57.647 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:22:57.646+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.646512+0000) 2020-02-06T02:22:57.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:22:57.646+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.646854+0000) 2020-02-06T02:22:57.648 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:22:57.647+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.647412+0000) 2020-02-06T02:22:57.648 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:22:57.647+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.647459+0000) 2020-02-06T02:22:57.649 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:22:57.647+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.647951+0000) 2020-02-06T02:22:57.649 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:22:57.648+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.648577+0000) 2020-02-06T02:22:57.650 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:22:57.648+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.648686+0000) 2020-02-06T02:22:57.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:57.652+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.652332+0000) 2020-02-06T02:22:57.669 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:22:57.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231042 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:22:57.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:22:57.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.672600+0000) 2020-02-06T02:22:57.696 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:22:57.695+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:57.695586+0000) 2020-02-06T02:22:59.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:22:59.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:22:59.812121+0000) 2020-02-06T02:23:00.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:00.056+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:00.057767+0000) 2020-02-06T02:23:00.670 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:00.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:00.669650+0000) 2020-02-06T02:23:00.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:00.944+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:00.945917+0000) 2020-02-06T02:23:02.674 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:02.673+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.673785+0000) 2020-02-06T02:23:02.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:02.673+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.674160+0000) 2020-02-06T02:23:02.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:02.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.675122+0000) 2020-02-06T02:23:02.676 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:02.674+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.675742+0000) 2020-02-06T02:23:02.677 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:02.675+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.676210+0000) 2020-02-06T02:23:02.677 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:02.676+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.676419+0000) 2020-02-06T02:23:02.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:02.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.676559+0000) 2020-02-06T02:23:02.678 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:02.676+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.676624+0000) 2020-02-06T02:23:02.678 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:02.676+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.677119+0000) 2020-02-06T02:23:02.679 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:02.676+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.677438+0000) 2020-02-06T02:23:02.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:02.677+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.678323+0000) 2020-02-06T02:23:02.679 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:02.676+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.677933+0000) 2020-02-06T02:23:02.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:02.678+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.678933+0000) 2020-02-06T02:23:02.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:02.682+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:02.683402+0000) 2020-02-06T02:23:02.698 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:02.696+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231081 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:03.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:03.526+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:03.527545+0000) 2020-02-06T02:23:05.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:05.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:05.795833+0000) 2020-02-06T02:23:06.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:06.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:06.191909+0000) 2020-02-06T02:23:06.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:06.554+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:06.554939+0000) 2020-02-06T02:23:07.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:07.166+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.167863+0000) 2020-02-06T02:23:07.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:07.442+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.443146+0000) 2020-02-06T02:23:07.444 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:07.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.443415+0000) 2020-02-06T02:23:07.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:07.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.443519+0000) 2020-02-06T02:23:07.673 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:07.671+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.672803+0000) 2020-02-06T02:23:07.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:07.694+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.695807+0000) 2020-02-06T02:23:07.704 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:07.702+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.703576+0000) 2020-02-06T02:23:07.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:07.702+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.704013+0000) 2020-02-06T02:23:07.705 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:07.703+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.704379+0000) 2020-02-06T02:23:07.705 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:07.703+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.704904+0000) 2020-02-06T02:23:07.706 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:07.704+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.705581+0000) 2020-02-06T02:23:07.707 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:07.705+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.706195+0000) 2020-02-06T02:23:07.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:07.706+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.706672+0000) 2020-02-06T02:23:07.708 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:07.706+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.707372+0000) 2020-02-06T02:23:07.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:07.706+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.707531+0000) 2020-02-06T02:23:07.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:07.707+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.708364+0000) 2020-02-06T02:23:07.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:07.708+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.709012+0000) 2020-02-06T02:23:07.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:07.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.709502+0000) 2020-02-06T02:23:07.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:07.708+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.708775+0000) 2020-02-06T02:23:07.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:07.708+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:07.709776+0000) 2020-02-06T02:23:07.727 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:07.725+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231121 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:09.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:09.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:09.812372+0000) 2020-02-06T02:23:10.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:10.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:10.058042+0000) 2020-02-06T02:23:10.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:10.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:10.669894+0000) 2020-02-06T02:23:10.947 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:10.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:10.946153+0000) 2020-02-06T02:23:12.734 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:12.731+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.732765+0000) 2020-02-06T02:23:12.734 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:12.731+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.733050+0000) 2020-02-06T02:23:12.734 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:12.733+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.733593+0000) 2020-02-06T02:23:12.735 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:12.733+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.734136+0000) 2020-02-06T02:23:12.735 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:12.733+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.734662+0000) 2020-02-06T02:23:12.736 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:12.734+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.735324+0000) 2020-02-06T02:23:12.737 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:12.735+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.736437+0000) 2020-02-06T02:23:12.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:12.735+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.736645+0000) 2020-02-06T02:23:12.738 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:12.735+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.736001+0000) 2020-02-06T02:23:12.738 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:12.736+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.737280+0000) 2020-02-06T02:23:12.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:12.736+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.737803+0000) 2020-02-06T02:23:12.739 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:12.736+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.737309+0000) 2020-02-06T02:23:12.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:12.737+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.738259+0000) 2020-02-06T02:23:12.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:12.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:12.741420+0000) 2020-02-06T02:23:12.758 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:12.756+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231159 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:13.528 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:13.526+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:13.527751+0000) 2020-02-06T02:23:15.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:15.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:15.803558+0000) 2020-02-06T02:23:16.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:16.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:16.192152+0000) 2020-02-06T02:23:16.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:16.554+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:16.555176+0000) 2020-02-06T02:23:17.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:17.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.168108+0000) 2020-02-06T02:23:17.444 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:17.443+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.443377+0000) 2020-02-06T02:23:17.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:17.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.443702+0000) 2020-02-06T02:23:17.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:17.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.443760+0000) 2020-02-06T02:23:17.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:17.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.673038+0000) 2020-02-06T02:23:17.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:17.695+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.696070+0000) 2020-02-06T02:23:17.763 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:17.761+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.762908+0000) 2020-02-06T02:23:17.764 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:17.762+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.763460+0000) 2020-02-06T02:23:17.765 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:17.763+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.764216+0000) 2020-02-06T02:23:17.766 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:17.764+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.765338+0000) 2020-02-06T02:23:17.766 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:17.764+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.765011+0000) 2020-02-06T02:23:17.766 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:17.764+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.765745+0000) 2020-02-06T02:23:17.767 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:17.765+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.766069+0000) 2020-02-06T02:23:17.767 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:17.766+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.766906+0000) 2020-02-06T02:23:17.768 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:17.765+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.767011+0000) 2020-02-06T02:23:17.768 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:17.766+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.767098+0000) 2020-02-06T02:23:17.768 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:17.767+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.767425+0000) 2020-02-06T02:23:17.769 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:17.766+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.767922+0000) 2020-02-06T02:23:17.769 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:17.767+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.768037+0000) 2020-02-06T02:23:17.772 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:17.770+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:17.771396+0000) 2020-02-06T02:23:17.787 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:17.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231200 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:17.966 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:23:17.972 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:23:19.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:19.811+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:19.812615+0000) 2020-02-06T02:23:20.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:20.057+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:20.058335+0000) 2020-02-06T02:23:20.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:20.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:20.670141+0000) 2020-02-06T02:23:20.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:20.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:20.946387+0000) 2020-02-06T02:23:22.794 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:22.791+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.793060+0000) 2020-02-06T02:23:22.794 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:22.792+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.792525+0000) 2020-02-06T02:23:22.795 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:22.793+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.794947+0000) 2020-02-06T02:23:22.796 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:22.794+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.794848+0000) 2020-02-06T02:23:22.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:22.794+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.795501+0000) 2020-02-06T02:23:22.796 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:22.794+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.795776+0000) 2020-02-06T02:23:22.797 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:22.795+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.796402+0000) 2020-02-06T02:23:22.797 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:22.796+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.796671+0000) 2020-02-06T02:23:22.799 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:22.797+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.797851+0000) 2020-02-06T02:23:22.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:22.797+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.798350+0000) 2020-02-06T02:23:22.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:22.797+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.798623+0000) 2020-02-06T02:23:22.800 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:22.798+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.798485+0000) 2020-02-06T02:23:22.800 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:22.798+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.798798+0000) 2020-02-06T02:23:22.803 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:22.801+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:22.802318+0000) 2020-02-06T02:23:22.818 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:22.816+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231239 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:23.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:23.527+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:23.527957+0000) 2020-02-06T02:23:25.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:25.810+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:25.811432+0000) 2020-02-06T02:23:26.193 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:26.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:26.192403+0000) 2020-02-06T02:23:26.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:26.555+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:26.555467+0000) 2020-02-06T02:23:27.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:27.167+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.168356+0000) 2020-02-06T02:23:27.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:27.443+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.443635+0000) 2020-02-06T02:23:27.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:27.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.443903+0000) 2020-02-06T02:23:27.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:27.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.444062+0000) 2020-02-06T02:23:27.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:27.672+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.673316+0000) 2020-02-06T02:23:27.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:27.695+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.696372+0000) 2020-02-06T02:23:27.824 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:27.822+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.823226+0000) 2020-02-06T02:23:27.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:27.822+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.822922+0000) 2020-02-06T02:23:27.825 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:27.824+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.824973+0000) 2020-02-06T02:23:27.827 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:27.825+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.826361+0000) 2020-02-06T02:23:27.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:27.826+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.826750+0000) 2020-02-06T02:23:27.828 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:27.826+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.827420+0000) 2020-02-06T02:23:27.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:27.827+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.828406+0000) 2020-02-06T02:23:27.830 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:27.827+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.828243+0000) 2020-02-06T02:23:27.830 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:27.828+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.828584+0000) 2020-02-06T02:23:27.830 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:27.827+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.828595+0000) 2020-02-06T02:23:27.830 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:27.827+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.828927+0000) 2020-02-06T02:23:27.831 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:27.828+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.829255+0000) 2020-02-06T02:23:27.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:27.829+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.830232+0000) 2020-02-06T02:23:27.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:27.830+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:27.831134+0000) 2020-02-06T02:23:27.849 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:27.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231279 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:29.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:29.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:29.812849+0000) 2020-02-06T02:23:30.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:30.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:30.058582+0000) 2020-02-06T02:23:30.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:30.670+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:30.670388+0000) 2020-02-06T02:23:30.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:30.945+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:30.946622+0000) 2020-02-06T02:23:32.854 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:32.851+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.853277+0000) 2020-02-06T02:23:32.855 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:32.852+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.854365+0000) 2020-02-06T02:23:32.855 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:32.853+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.854631+0000) 2020-02-06T02:23:32.857 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:32.854+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.855861+0000) 2020-02-06T02:23:32.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:32.855+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.856695+0000) 2020-02-06T02:23:32.859 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:32.857+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.858794+0000) 2020-02-06T02:23:32.860 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:32.857+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.859391+0000) 2020-02-06T02:23:32.860 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:32.858+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.859611+0000) 2020-02-06T02:23:32.861 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:32.858+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.859939+0000) 2020-02-06T02:23:32.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:32.859+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.860432+0000) 2020-02-06T02:23:32.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:32.859+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.860850+0000) 2020-02-06T02:23:32.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:32.859+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.860990+0000) 2020-02-06T02:23:32.862 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:32.859+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.861174+0000) 2020-02-06T02:23:32.866 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:32.864+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:32.866065+0000) 2020-02-06T02:23:32.880 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:32.877+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231317 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:33.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:33.527+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:33.528169+0000) 2020-02-06T02:23:35.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:35.819+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:35.819676+0000) 2020-02-06T02:23:36.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:36.191+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:36.192739+0000) 2020-02-06T02:23:36.556 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:36.554+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:36.555710+0000) 2020-02-06T02:23:37.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:37.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.168606+0000) 2020-02-06T02:23:37.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:37.442+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.443898+0000) 2020-02-06T02:23:37.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:37.442+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.444162+0000) 2020-02-06T02:23:37.445 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:37.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.444468+0000) 2020-02-06T02:23:37.674 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:37.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.673543+0000) 2020-02-06T02:23:37.697 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:37.696+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.696643+0000) 2020-02-06T02:23:37.885 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:37.883+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.884297+0000) 2020-02-06T02:23:37.886 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:37.883+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.885169+0000) 2020-02-06T02:23:37.886 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:37.884+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.885752+0000) 2020-02-06T02:23:37.887 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:37.885+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.886836+0000) 2020-02-06T02:23:37.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:37.886+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.887218+0000) 2020-02-06T02:23:37.888 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:37.886+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.887752+0000) 2020-02-06T02:23:37.889 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:37.887+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.888133+0000) 2020-02-06T02:23:37.889 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:37.887+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.888523+0000) 2020-02-06T02:23:37.891 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:37.889+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.890418+0000) 2020-02-06T02:23:37.892 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:37.887+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.889125+0000) 2020-02-06T02:23:37.892 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:37.888+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.889904+0000) 2020-02-06T02:23:37.892 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:37.889+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.891264+0000) 2020-02-06T02:23:37.893 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:37.890+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.891816+0000) 2020-02-06T02:23:37.900 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:37.897+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:37.898124+0000) 2020-02-06T02:23:37.910 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:37.907+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231358 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:39.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:39.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:39.813158+0000) 2020-02-06T02:23:40.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:40.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:40.058819+0000) 2020-02-06T02:23:40.671 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:40.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:40.670685+0000) 2020-02-06T02:23:40.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:40.946+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:40.946850+0000) 2020-02-06T02:23:42.917 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:42.915+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.916147+0000) 2020-02-06T02:23:42.917 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:42.915+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.916649+0000) 2020-02-06T02:23:42.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:42.916+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.917445+0000) 2020-02-06T02:23:42.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:42.916+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.917790+0000) 2020-02-06T02:23:42.919 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:42.916+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.918332+0000) 2020-02-06T02:23:42.919 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:42.917+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.918412+0000) 2020-02-06T02:23:42.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:42.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.919223+0000) 2020-02-06T02:23:42.920 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:42.918+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.919834+0000) 2020-02-06T02:23:42.921 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:42.918+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.919533+0000) 2020-02-06T02:23:42.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:42.919+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.920308+0000) 2020-02-06T02:23:42.921 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:42.919+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.920449+0000) 2020-02-06T02:23:42.922 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:42.920+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.921541+0000) 2020-02-06T02:23:42.922 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:42.920+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.921789+0000) 2020-02-06T02:23:42.930 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:42.928+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:42.929946+0000) 2020-02-06T02:23:42.940 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:42.937+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231397 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:43.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:43.528+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:43.528389+0000) 2020-02-06T02:23:45.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:45.827+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:45.828163+0000) 2020-02-06T02:23:46.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:46.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:46.192993+0000) 2020-02-06T02:23:46.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:46.555+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:46.555960+0000) 2020-02-06T02:23:47.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:47.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.168840+0000) 2020-02-06T02:23:47.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:47.443+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.444149+0000) 2020-02-06T02:23:47.445 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:47.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.444359+0000) 2020-02-06T02:23:47.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:47.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.444740+0000) 2020-02-06T02:23:47.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:47.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.673768+0000) 2020-02-06T02:23:47.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:47.696+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.696876+0000) 2020-02-06T02:23:47.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:47.944+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.944669+0000) 2020-02-06T02:23:47.946 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:47.945+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.945448+0000) 2020-02-06T02:23:47.946 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:47.945+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.946000+0000) 2020-02-06T02:23:47.947 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:47.945+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.946853+0000) 2020-02-06T02:23:47.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:47.947+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.947411+0000) 2020-02-06T02:23:47.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:47.947+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.947296+0000) 2020-02-06T02:23:47.949 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:47.946+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.948030+0000) 2020-02-06T02:23:47.950 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:47.948+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.948974+0000) 2020-02-06T02:23:47.950 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:47.947+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.948947+0000) 2020-02-06T02:23:47.950 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:47.947+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.949177+0000) 2020-02-06T02:23:47.951 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:47.949+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.949686+0000) 2020-02-06T02:23:47.951 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:47.948+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.949589+0000) 2020-02-06T02:23:47.952 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:47.949+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.950503+0000) 2020-02-06T02:23:47.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:47.953+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:47.953869+0000) 2020-02-06T02:23:47.969 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:47.966+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231438 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:48.128 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:23:48.134 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:23:49.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:49.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:49.813424+0000) 2020-02-06T02:23:50.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:50.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:50.059056+0000) 2020-02-06T02:23:50.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:50.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:50.670918+0000) 2020-02-06T02:23:50.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:50.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:50.947113+0000) 2020-02-06T02:23:52.976 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:52.974+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.975070+0000) 2020-02-06T02:23:52.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:52.974+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.975948+0000) 2020-02-06T02:23:52.977 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:52.975+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.976323+0000) 2020-02-06T02:23:52.977 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:52.975+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.976695+0000) 2020-02-06T02:23:52.978 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:52.975+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.976928+0000) 2020-02-06T02:23:52.978 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:52.976+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.977405+0000) 2020-02-06T02:23:52.978 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:52.976+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.977611+0000) 2020-02-06T02:23:52.979 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:52.976+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.978338+0000) 2020-02-06T02:23:52.980 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:52.977+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.978752+0000) 2020-02-06T02:23:52.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:52.978+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.980011+0000) 2020-02-06T02:23:52.981 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:52.978+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.979804+0000) 2020-02-06T02:23:52.981 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:52.978+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.980298+0000) 2020-02-06T02:23:52.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:52.984+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.985795+0000) 2020-02-06T02:23:52.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:52.985+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:52.986224+0000) 2020-02-06T02:23:53.000 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:52.997+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231476 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:53.529 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:53.527+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:53.528600+0000) 2020-02-06T02:23:55.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:55.835+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:55.836364+0000) 2020-02-06T02:23:56.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:56.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:56.193256+0000) 2020-02-06T02:23:56.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:56.555+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:56.556193+0000) 2020-02-06T02:23:57.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:57.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.169113+0000) 2020-02-06T02:23:57.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:57.443+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.444408+0000) 2020-02-06T02:23:57.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:57.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.444626+0000) 2020-02-06T02:23:57.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:57.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.444942+0000) 2020-02-06T02:23:57.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:57.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.673998+0000) 2020-02-06T02:23:57.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:57.696+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:57.697122+0000) 2020-02-06T02:23:58.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:23:58.003+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.004592+0000) 2020-02-06T02:23:58.006 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:23:58.004+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.005356+0000) 2020-02-06T02:23:58.006 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:23:58.004+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.005889+0000) 2020-02-06T02:23:58.007 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:58.005+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.006431+0000) 2020-02-06T02:23:58.007 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:23:58.005+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.006489+0000) 2020-02-06T02:23:58.008 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:23:58.006+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.007580+0000) 2020-02-06T02:23:58.008 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:23:58.006+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.008000+0000) 2020-02-06T02:23:58.009 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:23:58.006+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.008044+0000) 2020-02-06T02:23:58.009 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:58.007+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.008546+0000) 2020-02-06T02:23:58.010 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:23:58.007+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.008667+0000) 2020-02-06T02:23:58.010 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:23:58.007+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.008819+0000) 2020-02-06T02:23:58.010 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:23:58.007+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.009393+0000) 2020-02-06T02:23:58.011 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:23:58.008+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.009787+0000) 2020-02-06T02:23:58.012 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:58.010+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:58.011969+0000) 2020-02-06T02:23:58.030 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:23:58.028+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231518 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:23:59.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:23:59.812+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:23:59.813699+0000) 2020-02-06T02:24:00.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:00.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:00.059321+0000) 2020-02-06T02:24:00.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:00.669+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:00.671173+0000) 2020-02-06T02:24:00.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:00.946+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:00.947351+0000) 2020-02-06T02:24:03.039 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:03.034+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.035149+0000) 2020-02-06T02:24:03.040 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:03.034+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.035596+0000) 2020-02-06T02:24:03.040 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:03.035+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.036513+0000) 2020-02-06T02:24:03.040 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:03.035+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.036998+0000) 2020-02-06T02:24:03.041 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:03.036+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.038165+0000) 2020-02-06T02:24:03.041 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:03.037+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.038815+0000) 2020-02-06T02:24:03.042 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:03.037+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.039294+0000) 2020-02-06T02:24:03.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:03.036+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.038037+0000) 2020-02-06T02:24:03.042 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:03.039+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.040291+0000) 2020-02-06T02:24:03.043 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:03.039+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.040872+0000) 2020-02-06T02:24:03.043 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:03.038+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.039837+0000) 2020-02-06T02:24:03.043 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:03.039+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.041239+0000) 2020-02-06T02:24:03.044 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:03.040+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.041485+0000) 2020-02-06T02:24:03.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:03.045+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.046830+0000) 2020-02-06T02:24:03.063 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:03.060+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231557 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:03.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:03.528+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:03.528856+0000) 2020-02-06T02:24:05.845 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:05.843+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:05.844402+0000) 2020-02-06T02:24:06.194 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:06.192+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:06.193514+0000) 2020-02-06T02:24:06.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:06.555+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:06.556472+0000) 2020-02-06T02:24:07.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:07.168+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.169303+0000) 2020-02-06T02:24:07.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:07.444+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.444695+0000) 2020-02-06T02:24:07.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:07.444+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.444867+0000) 2020-02-06T02:24:07.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:07.444+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.445186+0000) 2020-02-06T02:24:07.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:07.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.674264+0000) 2020-02-06T02:24:07.698 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:07.696+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:07.697362+0000) 2020-02-06T02:24:08.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:08.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.067897+0000) 2020-02-06T02:24:08.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:08.067+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.068708+0000) 2020-02-06T02:24:08.070 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:08.068+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.069566+0000) 2020-02-06T02:24:08.071 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:08.069+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.070902+0000) 2020-02-06T02:24:08.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:08.070+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.071287+0000) 2020-02-06T02:24:08.072 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:08.070+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.071881+0000) 2020-02-06T02:24:08.073 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:08.070+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.072148+0000) 2020-02-06T02:24:08.073 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:08.071+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.072339+0000) 2020-02-06T02:24:08.074 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:08.071+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.072758+0000) 2020-02-06T02:24:08.074 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:08.071+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.073068+0000) 2020-02-06T02:24:08.075 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:08.072+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.073461+0000) 2020-02-06T02:24:08.075 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:08.072+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.073864+0000) 2020-02-06T02:24:08.075 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:08.072+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.073823+0000) 2020-02-06T02:24:08.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:08.078+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:08.079702+0000) 2020-02-06T02:24:08.092 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:08.090+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231598 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:09.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:09.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:09.813955+0000) 2020-02-06T02:24:10.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:10.058+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:10.059554+0000) 2020-02-06T02:24:10.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:10.670+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:10.671413+0000) 2020-02-06T02:24:10.948 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:10.946+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:10.947531+0000) 2020-02-06T02:24:13.097 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:13.094+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.096090+0000) 2020-02-06T02:24:13.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:13.095+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.096934+0000) 2020-02-06T02:24:13.100 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:13.098+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.099066+0000) 2020-02-06T02:24:13.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:13.098+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.099561+0000) 2020-02-06T02:24:13.101 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:13.099+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.100115+0000) 2020-02-06T02:24:13.101 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:13.099+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.101014+0000) 2020-02-06T02:24:13.102 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:13.100+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.101605+0000) 2020-02-06T02:24:13.103 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:13.100+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.101901+0000) 2020-02-06T02:24:13.103 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:13.101+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.102437+0000) 2020-02-06T02:24:13.103 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:13.100+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.102225+0000) 2020-02-06T02:24:13.104 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:13.101+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.102831+0000) 2020-02-06T02:24:13.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:13.101+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.102764+0000) 2020-02-06T02:24:13.104 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:13.101+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.103276+0000) 2020-02-06T02:24:13.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:13.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.105556+0000) 2020-02-06T02:24:13.123 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:13.121+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231636 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:13.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:13.528+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:13.529130+0000) 2020-02-06T02:24:15.853 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:15.852+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:15.852668+0000) 2020-02-06T02:24:16.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:16.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:16.193749+0000) 2020-02-06T02:24:16.557 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:16.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:16.556710+0000) 2020-02-06T02:24:17.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:17.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.169582+0000) 2020-02-06T02:24:17.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:17.443+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.444959+0000) 2020-02-06T02:24:17.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:17.443+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.445130+0000) 2020-02-06T02:24:17.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:17.443+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.445391+0000) 2020-02-06T02:24:17.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:17.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.674492+0000) 2020-02-06T02:24:17.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:17.697+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:17.697597+0000) 2020-02-06T02:24:18.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:18.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.128039+0000) 2020-02-06T02:24:18.130 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:18.128+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.129156+0000) 2020-02-06T02:24:18.131 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:18.128+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.129703+0000) 2020-02-06T02:24:18.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:18.128+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.130112+0000) 2020-02-06T02:24:18.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:18.129+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.130391+0000) 2020-02-06T02:24:18.132 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:18.129+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.131022+0000) 2020-02-06T02:24:18.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:18.131+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.132416+0000) 2020-02-06T02:24:18.133 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:18.131+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.132872+0000) 2020-02-06T02:24:18.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:18.132+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.133561+0000) 2020-02-06T02:24:18.135 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:18.132+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.134178+0000) 2020-02-06T02:24:18.135 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:18.133+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.134569+0000) 2020-02-06T02:24:18.136 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:18.133+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.135265+0000) 2020-02-06T02:24:18.136 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:18.134+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.135486+0000) 2020-02-06T02:24:18.140 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:18.138+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:18.139318+0000) 2020-02-06T02:24:18.154 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:18.151+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231678 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:18.299 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:24:18.305 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:24:19.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:19.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:19.814188+0000) 2020-02-06T02:24:20.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:20.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:20.059782+0000) 2020-02-06T02:24:20.672 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:20.671+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:20.671697+0000) 2020-02-06T02:24:20.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:20.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:20.947758+0000) 2020-02-06T02:24:23.160 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:23.157+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.159050+0000) 2020-02-06T02:24:23.160 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:23.158+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.159446+0000) 2020-02-06T02:24:23.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:23.158+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.159873+0000) 2020-02-06T02:24:23.161 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:23.159+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.160292+0000) 2020-02-06T02:24:23.161 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:23.159+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.160158+0000) 2020-02-06T02:24:23.162 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:23.159+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.160766+0000) 2020-02-06T02:24:23.162 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:23.159+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.160818+0000) 2020-02-06T02:24:23.162 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:23.161+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.161545+0000) 2020-02-06T02:24:23.164 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:23.162+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.162473+0000) 2020-02-06T02:24:23.165 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:23.162+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.162739+0000) 2020-02-06T02:24:23.165 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:23.163+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.163670+0000) 2020-02-06T02:24:23.165 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:23.162+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.163351+0000) 2020-02-06T02:24:23.166 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:23.163+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.163894+0000) 2020-02-06T02:24:23.168 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:23.166+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.168013+0000) 2020-02-06T02:24:23.187 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:23.185+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231717 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:23.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:23.528+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:23.529349+0000) 2020-02-06T02:24:25.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:25.860+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:25.860792+0000) 2020-02-06T02:24:26.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:26.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:26.193946+0000) 2020-02-06T02:24:26.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:26.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:26.556948+0000) 2020-02-06T02:24:27.170 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:27.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.169820+0000) 2020-02-06T02:24:27.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:27.444+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.445153+0000) 2020-02-06T02:24:27.446 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:27.444+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.445406+0000) 2020-02-06T02:24:27.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:27.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.445653+0000) 2020-02-06T02:24:27.675 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:27.673+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.674722+0000) 2020-02-06T02:24:27.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:27.696+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:27.697859+0000) 2020-02-06T02:24:28.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:28.191+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.192737+0000) 2020-02-06T02:24:28.195 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:28.193+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.194559+0000) 2020-02-06T02:24:28.196 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:28.194+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.195324+0000) 2020-02-06T02:24:28.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:28.195+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.195784+0000) 2020-02-06T02:24:28.197 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:28.194+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.195989+0000) 2020-02-06T02:24:28.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:28.195+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.196457+0000) 2020-02-06T02:24:28.198 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:28.196+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.196662+0000) 2020-02-06T02:24:28.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:28.196+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.197725+0000) 2020-02-06T02:24:28.199 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:28.197+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.197876+0000) 2020-02-06T02:24:28.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:28.197+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.197700+0000) 2020-02-06T02:24:28.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:28.197+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.198531+0000) 2020-02-06T02:24:28.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:28.198+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.198591+0000) 2020-02-06T02:24:28.200 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:28.198+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.198624+0000) 2020-02-06T02:24:28.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:28.202+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:28.203615+0000) 2020-02-06T02:24:28.216 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:28.215+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231761 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:29.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:29.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:29.814445+0000) 2020-02-06T02:24:30.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:30.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:30.060019+0000) 2020-02-06T02:24:30.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:30.671+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:30.671926+0000) 2020-02-06T02:24:30.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:30.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:30.947982+0000) 2020-02-06T02:24:33.223 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:33.220+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.222027+0000) 2020-02-06T02:24:33.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:33.221+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.222493+0000) 2020-02-06T02:24:33.224 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:33.222+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.223147+0000) 2020-02-06T02:24:33.224 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:33.223+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.223634+0000) 2020-02-06T02:24:33.224 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:33.222+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.223532+0000) 2020-02-06T02:24:33.225 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:33.223+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.224332+0000) 2020-02-06T02:24:33.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:33.223+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.224679+0000) 2020-02-06T02:24:33.226 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:33.224+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.225190+0000) 2020-02-06T02:24:33.226 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:33.224+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.225284+0000) 2020-02-06T02:24:33.226 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:33.223+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.225028+0000) 2020-02-06T02:24:33.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:33.224+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.225627+0000) 2020-02-06T02:24:33.227 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:33.225+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.225649+0000) 2020-02-06T02:24:33.227 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:33.225+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.226056+0000) 2020-02-06T02:24:33.231 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:33.229+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.230603+0000) 2020-02-06T02:24:33.246 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:33.244+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231799 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:33.530 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:33.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:33.529602+0000) 2020-02-06T02:24:35.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:35.868+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:35.868995+0000) 2020-02-06T02:24:36.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:36.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:36.194205+0000) 2020-02-06T02:24:36.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:36.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:36.557184+0000) 2020-02-06T02:24:37.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:37.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.170123+0000) 2020-02-06T02:24:37.446 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:37.445+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.445476+0000) 2020-02-06T02:24:37.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:37.445+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.445637+0000) 2020-02-06T02:24:37.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:37.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.445899+0000) 2020-02-06T02:24:37.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:37.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.674954+0000) 2020-02-06T02:24:37.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:37.697+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:37.698133+0000) 2020-02-06T02:24:38.252 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:38.250+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.251411+0000) 2020-02-06T02:24:38.252 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:38.250+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.251700+0000) 2020-02-06T02:24:38.253 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:38.251+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.252197+0000) 2020-02-06T02:24:38.253 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:38.251+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.252440+0000) 2020-02-06T02:24:38.254 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:38.252+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.253381+0000) 2020-02-06T02:24:38.254 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:38.252+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.253677+0000) 2020-02-06T02:24:38.255 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:38.253+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.254254+0000) 2020-02-06T02:24:38.255 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:38.253+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.254864+0000) 2020-02-06T02:24:38.256 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:38.254+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.254484+0000) 2020-02-06T02:24:38.256 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:38.254+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.255237+0000) 2020-02-06T02:24:38.257 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:38.255+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.255844+0000) 2020-02-06T02:24:38.257 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:38.255+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.255919+0000) 2020-02-06T02:24:38.258 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:38.256+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.257090+0000) 2020-02-06T02:24:38.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:38.262+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:38.263748+0000) 2020-02-06T02:24:38.276 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:38.274+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231844 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:39.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:39.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:39.814674+0000) 2020-02-06T02:24:40.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:40.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:40.060305+0000) 2020-02-06T02:24:40.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:40.671+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:40.672164+0000) 2020-02-06T02:24:40.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:40.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:40.948208+0000) 2020-02-06T02:24:43.282 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:43.280+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.280799+0000) 2020-02-06T02:24:43.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:43.281+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.281944+0000) 2020-02-06T02:24:43.283 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:43.281+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.281887+0000) 2020-02-06T02:24:43.283 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:43.280+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.281183+0000) 2020-02-06T02:24:43.284 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:43.282+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.282878+0000) 2020-02-06T02:24:43.284 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:43.283+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.283571+0000) 2020-02-06T02:24:43.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:43.283+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.283880+0000) 2020-02-06T02:24:43.285 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:43.284+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.284435+0000) 2020-02-06T02:24:43.285 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:43.283+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.283982+0000) 2020-02-06T02:24:43.286 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:43.284+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.284971+0000) 2020-02-06T02:24:43.286 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:43.284+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.285279+0000) 2020-02-06T02:24:43.286 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:43.284+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.285415+0000) 2020-02-06T02:24:43.287 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:43.285+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.286028+0000) 2020-02-06T02:24:43.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:43.293+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.293564+0000) 2020-02-06T02:24:43.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:43.305+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231883 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:43.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:43.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:43.529831+0000) 2020-02-06T02:24:45.878 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:45.876+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:45.877167+0000) 2020-02-06T02:24:46.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:46.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:46.194512+0000) 2020-02-06T02:24:46.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:46.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:46.557407+0000) 2020-02-06T02:24:47.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:47.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.170379+0000) 2020-02-06T02:24:47.447 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:47.445+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.445730+0000) 2020-02-06T02:24:47.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:47.445+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.445898+0000) 2020-02-06T02:24:47.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:47.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.446133+0000) 2020-02-06T02:24:47.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:47.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.675174+0000) 2020-02-06T02:24:47.699 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:47.697+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:47.698409+0000) 2020-02-06T02:24:48.313 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:48.310+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.311935+0000) 2020-02-06T02:24:48.313 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:48.311+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.312730+0000) 2020-02-06T02:24:48.314 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:48.312+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.313125+0000) 2020-02-06T02:24:48.314 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:48.313+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.313791+0000) 2020-02-06T02:24:48.315 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:48.313+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.314694+0000) 2020-02-06T02:24:48.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:48.315+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.315608+0000) 2020-02-06T02:24:48.317 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:48.315+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.316148+0000) 2020-02-06T02:24:48.317 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:48.315+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.316063+0000) 2020-02-06T02:24:48.318 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:48.315+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.316250+0000) 2020-02-06T02:24:48.318 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:48.316+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.316863+0000) 2020-02-06T02:24:48.319 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:48.317+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.317477+0000) 2020-02-06T02:24:48.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:48.316+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.317400+0000) 2020-02-06T02:24:48.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:48.316+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.317628+0000) 2020-02-06T02:24:48.320 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:48.318+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:48.319697+0000) 2020-02-06T02:24:48.336 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:48.335+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231926 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:48.487 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:24:48.494 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:24:49.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:49.813+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:49.814934+0000) 2020-02-06T02:24:50.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:50.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:50.060537+0000) 2020-02-06T02:24:50.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:50.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:50.672399+0000) 2020-02-06T02:24:50.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:50.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:50.948452+0000) 2020-02-06T02:24:53.347 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:53.339+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.340557+0000) 2020-02-06T02:24:53.348 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:53.340+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.341045+0000) 2020-02-06T02:24:53.348 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:53.340+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.341898+0000) 2020-02-06T02:24:53.349 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:53.341+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.342247+0000) 2020-02-06T02:24:53.351 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:53.343+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.344221+0000) 2020-02-06T02:24:53.351 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:53.343+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.344696+0000) 2020-02-06T02:24:53.352 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:53.343+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.344922+0000) 2020-02-06T02:24:53.352 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:53.344+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.345114+0000) 2020-02-06T02:24:53.353 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:53.344+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.345978+0000) 2020-02-06T02:24:53.354 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:53.345+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.346388+0000) 2020-02-06T02:24:53.354 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:53.345+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.345635+0000) 2020-02-06T02:24:53.354 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:53.346+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.346979+0000) 2020-02-06T02:24:53.355 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:53.347+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.347613+0000) 2020-02-06T02:24:53.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:53.348+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.350092+0000) 2020-02-06T02:24:53.382 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:53.365+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 231964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:53.745 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:53.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:53.530053+0000) 2020-02-06T02:24:55.887 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:55.885+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:55.886276+0000) 2020-02-06T02:24:56.195 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:56.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:56.194756+0000) 2020-02-06T02:24:56.558 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:56.557+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:56.557677+0000) 2020-02-06T02:24:57.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:57.169+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.170626+0000) 2020-02-06T02:24:57.447 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:57.445+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.446040+0000) 2020-02-06T02:24:57.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:57.445+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.446161+0000) 2020-02-06T02:24:57.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:57.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.446359+0000) 2020-02-06T02:24:57.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:57.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.675408+0000) 2020-02-06T02:24:57.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:57.697+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:57.698634+0000) 2020-02-06T02:24:58.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:58.371+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.372980+0000) 2020-02-06T02:24:58.374 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:24:58.372+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.373669+0000) 2020-02-06T02:24:58.375 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:24:58.373+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.374153+0000) 2020-02-06T02:24:58.375 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:24:58.374+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.374753+0000) 2020-02-06T02:24:58.376 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:24:58.374+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.374988+0000) 2020-02-06T02:24:58.376 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:24:58.374+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.375692+0000) 2020-02-06T02:24:58.377 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:24:58.374+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.376028+0000) 2020-02-06T02:24:58.378 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:24:58.376+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.377332+0000) 2020-02-06T02:24:58.378 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:24:58.376+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.377361+0000) 2020-02-06T02:24:58.378 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:24:58.377+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.377632+0000) 2020-02-06T02:24:58.379 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:24:58.376+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.377860+0000) 2020-02-06T02:24:58.379 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:24:58.377+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.378151+0000) 2020-02-06T02:24:58.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:58.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.381326+0000) 2020-02-06T02:24:58.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:58.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:58.381604+0000) 2020-02-06T02:24:58.402 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:24:58.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232008 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:24:59.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:24:59.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:24:59.815184+0000) 2020-02-06T02:25:00.061 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:00.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:00.060772+0000) 2020-02-06T02:25:00.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:00.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:00.672671+0000) 2020-02-06T02:25:00.949 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:00.947+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:00.948694+0000) 2020-02-06T02:25:03.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:03.403+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.404325+0000) 2020-02-06T02:25:03.406 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:03.403+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.404897+0000) 2020-02-06T02:25:03.406 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:03.404+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.405418+0000) 2020-02-06T02:25:03.407 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:03.404+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.405956+0000) 2020-02-06T02:25:03.407 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:03.405+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.406529+0000) 2020-02-06T02:25:03.408 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:03.406+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.407524+0000) 2020-02-06T02:25:03.409 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:03.407+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.408224+0000) 2020-02-06T02:25:03.409 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:03.406+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.408362+0000) 2020-02-06T02:25:03.411 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:03.408+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.410033+0000) 2020-02-06T02:25:03.411 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:03.408+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.410272+0000) 2020-02-06T02:25:03.412 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:03.408+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.410043+0000) 2020-02-06T02:25:03.412 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:03.409+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.410880+0000) 2020-02-06T02:25:03.412 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:03.409+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.410763+0000) 2020-02-06T02:25:03.415 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:03.413+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.414899+0000) 2020-02-06T02:25:03.430 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:03.427+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232047 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:03.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:03.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:03.530298+0000) 2020-02-06T02:25:05.895 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:05.893+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:05.894329+0000) 2020-02-06T02:25:06.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:06.193+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:06.195008+0000) 2020-02-06T02:25:06.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:06.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:06.557915+0000) 2020-02-06T02:25:07.171 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:07.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.170854+0000) 2020-02-06T02:25:07.447 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:07.444+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.446288+0000) 2020-02-06T02:25:07.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:07.444+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.446393+0000) 2020-02-06T02:25:07.448 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:07.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.446644+0000) 2020-02-06T02:25:07.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:07.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.675637+0000) 2020-02-06T02:25:07.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:07.698+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:07.698861+0000) 2020-02-06T02:25:08.435 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:08.433+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.434236+0000) 2020-02-06T02:25:08.436 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:08.434+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.435359+0000) 2020-02-06T02:25:08.437 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:08.434+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.435826+0000) 2020-02-06T02:25:08.437 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:08.434+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.436131+0000) 2020-02-06T02:25:08.437 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:08.435+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.436745+0000) 2020-02-06T02:25:08.438 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:08.435+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.436926+0000) 2020-02-06T02:25:08.438 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:08.436+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.437499+0000) 2020-02-06T02:25:08.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:08.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.438014+0000) 2020-02-06T02:25:08.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:08.437+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.438589+0000) 2020-02-06T02:25:08.440 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:08.437+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.439142+0000) 2020-02-06T02:25:08.440 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:08.438+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.439474+0000) 2020-02-06T02:25:08.440 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:08.438+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.439933+0000) 2020-02-06T02:25:08.442 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:08.440+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.441852+0000) 2020-02-06T02:25:08.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:08.446+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:08.447920+0000) 2020-02-06T02:25:08.461 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:08.458+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232092 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:09.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:09.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:09.815438+0000) 2020-02-06T02:25:10.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:10.059+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:10.061028+0000) 2020-02-06T02:25:10.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:10.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:10.672988+0000) 2020-02-06T02:25:10.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:10.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:10.948871+0000) 2020-02-06T02:25:13.466 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:13.464+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.465162+0000) 2020-02-06T02:25:13.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:13.464+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.466238+0000) 2020-02-06T02:25:13.468 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:13.466+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.467410+0000) 2020-02-06T02:25:13.468 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:13.466+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.468029+0000) 2020-02-06T02:25:13.469 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:13.467+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.468411+0000) 2020-02-06T02:25:13.469 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:13.467+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.468631+0000) 2020-02-06T02:25:13.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:13.467+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.469234+0000) 2020-02-06T02:25:13.471 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:13.468+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.469989+0000) 2020-02-06T02:25:13.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:13.469+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.470579+0000) 2020-02-06T02:25:13.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:13.469+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.471166+0000) 2020-02-06T02:25:13.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:13.470+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.471779+0000) 2020-02-06T02:25:13.473 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:13.470+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.471751+0000) 2020-02-06T02:25:13.476 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:13.474+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.475700+0000) 2020-02-06T02:25:13.477 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:13.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.476669+0000) 2020-02-06T02:25:13.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:13.488+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232131 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:13.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:13.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:13.530519+0000) 2020-02-06T02:25:15.908 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:15.906+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:15.907025+0000) 2020-02-06T02:25:16.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:16.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:16.195268+0000) 2020-02-06T02:25:16.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:16.556+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:16.558155+0000) 2020-02-06T02:25:17.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:17.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.171104+0000) 2020-02-06T02:25:17.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:17.445+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.446653+0000) 2020-02-06T02:25:17.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:17.445+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.446536+0000) 2020-02-06T02:25:17.448 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:17.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.446899+0000) 2020-02-06T02:25:17.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:17.674+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.675826+0000) 2020-02-06T02:25:17.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:17.698+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:17.699116+0000) 2020-02-06T02:25:18.496 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:18.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.495385+0000) 2020-02-06T02:25:18.497 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:18.494+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.495940+0000) 2020-02-06T02:25:18.498 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:18.495+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.496939+0000) 2020-02-06T02:25:18.498 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:18.496+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.497541+0000) 2020-02-06T02:25:18.499 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:18.497+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.498175+0000) 2020-02-06T02:25:18.500 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:18.497+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.498662+0000) 2020-02-06T02:25:18.500 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:18.498+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.499195+0000) 2020-02-06T02:25:18.501 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:18.498+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.499830+0000) 2020-02-06T02:25:18.502 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:18.499+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.501118+0000) 2020-02-06T02:25:18.503 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:18.500+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.501419+0000) 2020-02-06T02:25:18.503 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:18.500+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.501408+0000) 2020-02-06T02:25:18.503 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:18.500+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.501651+0000) 2020-02-06T02:25:18.504 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:18.500+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.502043+0000) 2020-02-06T02:25:18.506 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:18.504+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:18.505120+0000) 2020-02-06T02:25:18.521 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:18.518+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232176 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:18.651 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:25:18.657 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:25:19.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:19.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:19.815700+0000) 2020-02-06T02:25:20.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:20.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:20.061288+0000) 2020-02-06T02:25:20.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:20.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:20.673244+0000) 2020-02-06T02:25:20.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:20.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:20.949131+0000) 2020-02-06T02:25:23.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:23.523+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.524873+0000) 2020-02-06T02:25:23.527 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:23.524+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.526272+0000) 2020-02-06T02:25:23.527 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:23.525+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.526937+0000) 2020-02-06T02:25:23.528 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:23.526+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.527461+0000) 2020-02-06T02:25:23.528 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:23.526+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.527831+0000) 2020-02-06T02:25:23.530 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:23.527+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.529086+0000) 2020-02-06T02:25:23.530 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:23.527+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.529411+0000) 2020-02-06T02:25:23.531 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:23.527+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.528538+0000) 2020-02-06T02:25:23.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:23.528+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.530384+0000) 2020-02-06T02:25:23.532 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:23.529+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.530680+0000) 2020-02-06T02:25:23.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:23.529+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.530674+0000) 2020-02-06T02:25:23.532 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:23.529+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.531368+0000) 2020-02-06T02:25:23.533 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:23.530+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.532295+0000) 2020-02-06T02:25:23.533 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:23.530+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.531844+0000) 2020-02-06T02:25:23.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:23.536+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:23.537275+0000) 2020-02-06T02:25:23.550 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:23.548+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232215 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:25.916 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:25.915+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:25.915338+0000) 2020-02-06T02:25:26.196 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:26.194+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:26.195540+0000) 2020-02-06T02:25:26.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:26.557+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:26.558406+0000) 2020-02-06T02:25:27.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:27.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.171391+0000) 2020-02-06T02:25:27.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:27.445+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.446851+0000) 2020-02-06T02:25:27.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:27.445+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.446869+0000) 2020-02-06T02:25:27.448 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:27.445+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.447255+0000) 2020-02-06T02:25:27.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:27.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.676113+0000) 2020-02-06T02:25:27.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:27.699+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:27.699354+0000) 2020-02-06T02:25:28.556 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:28.554+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.555213+0000) 2020-02-06T02:25:28.557 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:28.555+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.556212+0000) 2020-02-06T02:25:28.557 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:28.555+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.556638+0000) 2020-02-06T02:25:28.558 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:28.556+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.557178+0000) 2020-02-06T02:25:28.560 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:28.556+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.558252+0000) 2020-02-06T02:25:28.560 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:28.556+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.557515+0000) 2020-02-06T02:25:28.560 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:28.557+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.558526+0000) 2020-02-06T02:25:28.561 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:28.557+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.558880+0000) 2020-02-06T02:25:28.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:28.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.559917+0000) 2020-02-06T02:25:28.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:28.558+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.560402+0000) 2020-02-06T02:25:28.562 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:28.559+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.560958+0000) 2020-02-06T02:25:28.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:28.559+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.561170+0000) 2020-02-06T02:25:28.562 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:28.559+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.560744+0000) 2020-02-06T02:25:28.567 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:28.565+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:28.566778+0000) 2020-02-06T02:25:28.582 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:28.579+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232260 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:29.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:29.814+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:29.815937+0000) 2020-02-06T02:25:30.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:30.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:30.061524+0000) 2020-02-06T02:25:30.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:30.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:30.673454+0000) 2020-02-06T02:25:30.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:30.949+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:30.949371+0000) 2020-02-06T02:25:33.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:33.530+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.530905+0000) 2020-02-06T02:25:33.587 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:33.585+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.586344+0000) 2020-02-06T02:25:33.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:33.586+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.587399+0000) 2020-02-06T02:25:33.589 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:33.587+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.588298+0000) 2020-02-06T02:25:33.589 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:33.586+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.587758+0000) 2020-02-06T02:25:33.590 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:33.587+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.588860+0000) 2020-02-06T02:25:33.590 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:33.588+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.589370+0000) 2020-02-06T02:25:33.591 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:33.588+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.590026+0000) 2020-02-06T02:25:33.591 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:33.589+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.590595+0000) 2020-02-06T02:25:33.591 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:33.589+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.590908+0000) 2020-02-06T02:25:33.593 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:33.590+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.591540+0000) 2020-02-06T02:25:33.593 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:33.590+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.592119+0000) 2020-02-06T02:25:33.593 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:33.590+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.592135+0000) 2020-02-06T02:25:33.594 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:33.590+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.591988+0000) 2020-02-06T02:25:33.599 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:33.597+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:33.598093+0000) 2020-02-06T02:25:33.612 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:33.610+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232299 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:35.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:35.923+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:35.923614+0000) 2020-02-06T02:25:36.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:36.195+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:36.195809+0000) 2020-02-06T02:25:36.559 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:36.557+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:36.558662+0000) 2020-02-06T02:25:37.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:37.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.171624+0000) 2020-02-06T02:25:37.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:37.446+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.447056+0000) 2020-02-06T02:25:37.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:37.446+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.447083+0000) 2020-02-06T02:25:37.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:37.446+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.447546+0000) 2020-02-06T02:25:37.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:37.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.676303+0000) 2020-02-06T02:25:37.700 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:37.699+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:37.699598+0000) 2020-02-06T02:25:38.619 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:38.618+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.618247+0000) 2020-02-06T02:25:38.620 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:38.618+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.618943+0000) 2020-02-06T02:25:38.620 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:38.619+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.619148+0000) 2020-02-06T02:25:38.620 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:38.619+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.619640+0000) 2020-02-06T02:25:38.621 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:38.620+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.620357+0000) 2020-02-06T02:25:38.623 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:38.619+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.620961+0000) 2020-02-06T02:25:38.623 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:38.620+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.621002+0000) 2020-02-06T02:25:38.623 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:38.620+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.621909+0000) 2020-02-06T02:25:38.624 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:38.621+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.622544+0000) 2020-02-06T02:25:38.624 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:38.621+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.622898+0000) 2020-02-06T02:25:38.624 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:38.621+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.623068+0000) 2020-02-06T02:25:38.625 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:38.623+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.623525+0000) 2020-02-06T02:25:38.625 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:38.622+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.623735+0000) 2020-02-06T02:25:38.630 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:38.629+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:38.629310+0000) 2020-02-06T02:25:38.642 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:38.640+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232344 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:39.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:39.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:39.816172+0000) 2020-02-06T02:25:40.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:40.060+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:40.061754+0000) 2020-02-06T02:25:40.674 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:40.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:40.673700+0000) 2020-02-06T02:25:40.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:40.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:40.949577+0000) 2020-02-06T02:25:43.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:43.530+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.531173+0000) 2020-02-06T02:25:43.649 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:43.646+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.647913+0000) 2020-02-06T02:25:43.649 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:43.647+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.648830+0000) 2020-02-06T02:25:43.650 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:43.648+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.649451+0000) 2020-02-06T02:25:43.650 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:43.648+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.649928+0000) 2020-02-06T02:25:43.651 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:43.649+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.650451+0000) 2020-02-06T02:25:43.651 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:43.649+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.650954+0000) 2020-02-06T02:25:43.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:43.650+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.651526+0000) 2020-02-06T02:25:43.652 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:43.650+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.651863+0000) 2020-02-06T02:25:43.653 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:43.651+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.652372+0000) 2020-02-06T02:25:43.653 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:43.650+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.652294+0000) 2020-02-06T02:25:43.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:43.651+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.652990+0000) 2020-02-06T02:25:43.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:43.651+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.652848+0000) 2020-02-06T02:25:43.654 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:43.651+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.653365+0000) 2020-02-06T02:25:43.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:43.656+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:43.658022+0000) 2020-02-06T02:25:43.673 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:43.670+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232383 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:45.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:45.934+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:45.935655+0000) 2020-02-06T02:25:46.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:46.195+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:46.196056+0000) 2020-02-06T02:25:46.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:46.558+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:46.558901+0000) 2020-02-06T02:25:47.172 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:47.170+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.171810+0000) 2020-02-06T02:25:47.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:47.446+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.447305+0000) 2020-02-06T02:25:47.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:47.446+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.447310+0000) 2020-02-06T02:25:47.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:47.446+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.447762+0000) 2020-02-06T02:25:47.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:47.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.676535+0000) 2020-02-06T02:25:47.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:47.698+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:47.699801+0000) 2020-02-06T02:25:48.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:48.676+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.677379+0000) 2020-02-06T02:25:48.679 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:48.676+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.678007+0000) 2020-02-06T02:25:48.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:48.678+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.679102+0000) 2020-02-06T02:25:48.680 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:48.678+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.679582+0000) 2020-02-06T02:25:48.681 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:48.678+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.680092+0000) 2020-02-06T02:25:48.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:48.679+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.680280+0000) 2020-02-06T02:25:48.682 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:48.680+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.681115+0000) 2020-02-06T02:25:48.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:48.679+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.681371+0000) 2020-02-06T02:25:48.683 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:48.681+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.682777+0000) 2020-02-06T02:25:48.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:48.681+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.682997+0000) 2020-02-06T02:25:48.684 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:48.682+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.683846+0000) 2020-02-06T02:25:48.685 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:48.683+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.684249+0000) 2020-02-06T02:25:48.685 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:48.682+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.684272+0000) 2020-02-06T02:25:48.689 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:48.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:48.688711+0000) 2020-02-06T02:25:48.703 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:48.700+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232429 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:48.781 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:25:48.787 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:25:49.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:49.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:49.816409+0000) 2020-02-06T02:25:50.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:50.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:50.061997+0000) 2020-02-06T02:25:50.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:50.672+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:50.673935+0000) 2020-02-06T02:25:50.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:50.948+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:50.949829+0000) 2020-02-06T02:25:53.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:53.530+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.531403+0000) 2020-02-06T02:25:53.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:53.706+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.707910+0000) 2020-02-06T02:25:53.709 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:53.707+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.708326+0000) 2020-02-06T02:25:53.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:53.708+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.709268+0000) 2020-02-06T02:25:53.710 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:53.708+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.709351+0000) 2020-02-06T02:25:53.711 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:53.709+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.710035+0000) 2020-02-06T02:25:53.711 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:53.709+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.710275+0000) 2020-02-06T02:25:53.711 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:53.710+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.710540+0000) 2020-02-06T02:25:53.712 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:53.710+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.711215+0000) 2020-02-06T02:25:53.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:53.710+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.711275+0000) 2020-02-06T02:25:53.712 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:53.710+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.711672+0000) 2020-02-06T02:25:53.713 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:53.710+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.711868+0000) 2020-02-06T02:25:53.713 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:53.711+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.712409+0000) 2020-02-06T02:25:53.714 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:53.711+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.712918+0000) 2020-02-06T02:25:53.719 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:53.717+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:53.718633+0000) 2020-02-06T02:25:53.732 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:53.730+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232467 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:55.945 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:55.943+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:55.943824+0000) 2020-02-06T02:25:56.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:56.195+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:56.196255+0000) 2020-02-06T02:25:56.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:56.558+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:56.559145+0000) 2020-02-06T02:25:57.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:57.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.172130+0000) 2020-02-06T02:25:57.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:57.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.447574+0000) 2020-02-06T02:25:57.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:57.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.447576+0000) 2020-02-06T02:25:57.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:57.447+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.447986+0000) 2020-02-06T02:25:57.677 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:57.675+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.676798+0000) 2020-02-06T02:25:57.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:57.698+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:57.700006+0000) 2020-02-06T02:25:58.738 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:25:58.736+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.736899+0000) 2020-02-06T02:25:58.738 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:25:58.736+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.737471+0000) 2020-02-06T02:25:58.739 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:25:58.737+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.738199+0000) 2020-02-06T02:25:58.739 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:25:58.738+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.738654+0000) 2020-02-06T02:25:58.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:25:58.737+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.738999+0000) 2020-02-06T02:25:58.741 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:25:58.739+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.740360+0000) 2020-02-06T02:25:58.742 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:25:58.740+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.741132+0000) 2020-02-06T02:25:58.742 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:58.740+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.741799+0000) 2020-02-06T02:25:58.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:25:58.741+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.741784+0000) 2020-02-06T02:25:58.743 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:25:58.741+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.742232+0000) 2020-02-06T02:25:58.744 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:25:58.741+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.742652+0000) 2020-02-06T02:25:58.744 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:25:58.742+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.742973+0000) 2020-02-06T02:25:58.744 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:58.742+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.743352+0000) 2020-02-06T02:25:58.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:58.742+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:58.743443+0000) 2020-02-06T02:25:58.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:25:58.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232514 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:25:59.817 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:25:59.815+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:25:59.816651+0000) 2020-02-06T02:26:00.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:00.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:00.062202+0000) 2020-02-06T02:26:00.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:00.673+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:00.674161+0000) 2020-02-06T02:26:00.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:00.949+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:00.950116+0000) 2020-02-06T02:26:03.532 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:03.530+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.531604+0000) 2020-02-06T02:26:03.769 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:03.766+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.767122+0000) 2020-02-06T02:26:03.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:03.767+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.768574+0000) 2020-02-06T02:26:03.770 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:03.768+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.769107+0000) 2020-02-06T02:26:03.771 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:03.768+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.769415+0000) 2020-02-06T02:26:03.772 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:03.769+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.770345+0000) 2020-02-06T02:26:03.772 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:03.769+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.769929+0000) 2020-02-06T02:26:03.772 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:03.769+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.770687+0000) 2020-02-06T02:26:03.773 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:03.770+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.771116+0000) 2020-02-06T02:26:03.773 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:03.770+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.771691+0000) 2020-02-06T02:26:03.774 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:03.772+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.772931+0000) 2020-02-06T02:26:03.774 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:03.772+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.773245+0000) 2020-02-06T02:26:03.775 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:03.772+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.773644+0000) 2020-02-06T02:26:03.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:03.772+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.773700+0000) 2020-02-06T02:26:03.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:03.776+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:03.777914+0000) 2020-02-06T02:26:03.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:03.790+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232551 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:05.953 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:05.951+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:05.952167+0000) 2020-02-06T02:26:06.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:06.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:06.196508+0000) 2020-02-06T02:26:06.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:06.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:06.559405+0000) 2020-02-06T02:26:07.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:07.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.172378+0000) 2020-02-06T02:26:07.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:07.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.447761+0000) 2020-02-06T02:26:07.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:07.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.447768+0000) 2020-02-06T02:26:07.449 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:07.447+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.448230+0000) 2020-02-06T02:26:07.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:07.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.677055+0000) 2020-02-06T02:26:07.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:07.699+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:07.700256+0000) 2020-02-06T02:26:08.798 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:08.795+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.797007+0000) 2020-02-06T02:26:08.798 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:08.797+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.797517+0000) 2020-02-06T02:26:08.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:08.797+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.798351+0000) 2020-02-06T02:26:08.799 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:08.797+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.798523+0000) 2020-02-06T02:26:08.800 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:08.797+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.798690+0000) 2020-02-06T02:26:08.800 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:08.797+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.798998+0000) 2020-02-06T02:26:08.800 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:08.798+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.799324+0000) 2020-02-06T02:26:08.801 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:08.798+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.799766+0000) 2020-02-06T02:26:08.802 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:08.799+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.800219+0000) 2020-02-06T02:26:08.802 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:08.800+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.801011+0000) 2020-02-06T02:26:08.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:08.799+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.800706+0000) 2020-02-06T02:26:08.803 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:08.801+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.801740+0000) 2020-02-06T02:26:08.803 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:08.801+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.802203+0000) 2020-02-06T02:26:08.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:08.802+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:08.803130+0000) 2020-02-06T02:26:08.823 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:08.821+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232597 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:09.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:09.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:09.816887+0000) 2020-02-06T02:26:10.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:10.061+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:10.062427+0000) 2020-02-06T02:26:10.675 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:10.674+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:10.674488+0000) 2020-02-06T02:26:10.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:10.949+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:10.950368+0000) 2020-02-06T02:26:13.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:13.531+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.531838+0000) 2020-02-06T02:26:13.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:13.827+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.828826+0000) 2020-02-06T02:26:13.830 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:13.828+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.829280+0000) 2020-02-06T02:26:13.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:13.829+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.830077+0000) 2020-02-06T02:26:13.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:13.829+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.830259+0000) 2020-02-06T02:26:13.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:13.829+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.830386+0000) 2020-02-06T02:26:13.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:13.829+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.830698+0000) 2020-02-06T02:26:13.832 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:13.830+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.831148+0000) 2020-02-06T02:26:13.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:13.829+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.831017+0000) 2020-02-06T02:26:13.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:13.831+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.832098+0000) 2020-02-06T02:26:13.833 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:13.830+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.831973+0000) 2020-02-06T02:26:13.834 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:13.831+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.832604+0000) 2020-02-06T02:26:13.834 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:13.831+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.832378+0000) 2020-02-06T02:26:13.834 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:13.832+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.833159+0000) 2020-02-06T02:26:13.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:13.837+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:13.838300+0000) 2020-02-06T02:26:13.858 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:13.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232635 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:15.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:15.959+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:15.960362+0000) 2020-02-06T02:26:16.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:16.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:16.196780+0000) 2020-02-06T02:26:16.560 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:16.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:16.559630+0000) 2020-02-06T02:26:17.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:17.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.172587+0000) 2020-02-06T02:26:17.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:17.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.447989+0000) 2020-02-06T02:26:17.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:17.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.448010+0000) 2020-02-06T02:26:17.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:17.448+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.448479+0000) 2020-02-06T02:26:17.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:17.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.677333+0000) 2020-02-06T02:26:17.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:17.699+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:17.700457+0000) 2020-02-06T02:26:18.861 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:18.858+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.859947+0000) 2020-02-06T02:26:18.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:18.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.860461+0000) 2020-02-06T02:26:18.863 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:18.861+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.861835+0000) 2020-02-06T02:26:18.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:18.861+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.862159+0000) 2020-02-06T02:26:18.863 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:18.862+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.862509+0000) 2020-02-06T02:26:18.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:18.862+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.863146+0000) 2020-02-06T02:26:18.865 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:18.863+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.864181+0000) 2020-02-06T02:26:18.866 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:18.863+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.863883+0000) 2020-02-06T02:26:18.866 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:18.864+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.865209+0000) 2020-02-06T02:26:18.866 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:18.864+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.865577+0000) 2020-02-06T02:26:18.867 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:18.865+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.866224+0000) 2020-02-06T02:26:18.867 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:18.865+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.866285+0000) 2020-02-06T02:26:18.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:18.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.871280+0000) 2020-02-06T02:26:18.872 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:18.870+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:18.871605+0000) 2020-02-06T02:26:18.884 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:18.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232682 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:18.913 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:26:18.918 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:26:19.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:19.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:19.817166+0000) 2020-02-06T02:26:20.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:20.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:20.062651+0000) 2020-02-06T02:26:20.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:20.674+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:20.674746+0000) 2020-02-06T02:26:20.951 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:20.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:20.950606+0000) 2020-02-06T02:26:23.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:23.531+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.532072+0000) 2020-02-06T02:26:23.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:23.888+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.889473+0000) 2020-02-06T02:26:23.891 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:23.890+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.890760+0000) 2020-02-06T02:26:23.893 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:23.891+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.892157+0000) 2020-02-06T02:26:23.893 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:23.892+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.892781+0000) 2020-02-06T02:26:23.894 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:23.892+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.893370+0000) 2020-02-06T02:26:23.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:23.892+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.893642+0000) 2020-02-06T02:26:23.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:23.893+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.893677+0000) 2020-02-06T02:26:23.897 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:23.893+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.894942+0000) 2020-02-06T02:26:23.898 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:23.894+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.895537+0000) 2020-02-06T02:26:23.898 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:23.894+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.895306+0000) 2020-02-06T02:26:23.898 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:23.893+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.894133+0000) 2020-02-06T02:26:23.898 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:23.894+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.894786+0000) 2020-02-06T02:26:23.898 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:23.894+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.895024+0000) 2020-02-06T02:26:23.899 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:23.897+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:23.898204+0000) 2020-02-06T02:26:23.915 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:23.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232719 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:25.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:25.968+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:25.968705+0000) 2020-02-06T02:26:26.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:26.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:26.197047+0000) 2020-02-06T02:26:26.561 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:26.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:26.559923+0000) 2020-02-06T02:26:27.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:27.171+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.172818+0000) 2020-02-06T02:26:27.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:27.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.448215+0000) 2020-02-06T02:26:27.449 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:27.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.448224+0000) 2020-02-06T02:26:27.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:27.448+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.448716+0000) 2020-02-06T02:26:27.678 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:27.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.677583+0000) 2020-02-06T02:26:27.701 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:27.700+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:27.700683+0000) 2020-02-06T02:26:28.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:28.918+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.919823+0000) 2020-02-06T02:26:28.921 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:28.919+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.920490+0000) 2020-02-06T02:26:28.921 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:28.919+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.920786+0000) 2020-02-06T02:26:28.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:28.920+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.921072+0000) 2020-02-06T02:26:28.922 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:28.920+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.921436+0000) 2020-02-06T02:26:28.923 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:28.921+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.921952+0000) 2020-02-06T02:26:28.923 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:28.922+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.922745+0000) 2020-02-06T02:26:28.924 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:28.922+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.923115+0000) 2020-02-06T02:26:28.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:28.923+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.924032+0000) 2020-02-06T02:26:28.925 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:28.924+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.924937+0000) 2020-02-06T02:26:28.926 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:28.923+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.925034+0000) 2020-02-06T02:26:28.926 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:28.924+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.925208+0000) 2020-02-06T02:26:28.927 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:28.925+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.925846+0000) 2020-02-06T02:26:28.928 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:28.926+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:28.927964+0000) 2020-02-06T02:26:28.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:28.943+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232765 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:29.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:29.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:29.817385+0000) 2020-02-06T02:26:30.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:30.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:30.062876+0000) 2020-02-06T02:26:30.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:30.673+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:30.674988+0000) 2020-02-06T02:26:30.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:30.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:30.950859+0000) 2020-02-06T02:26:33.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:33.532+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.532345+0000) 2020-02-06T02:26:33.951 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:33.948+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.950063+0000) 2020-02-06T02:26:33.952 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:33.949+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.950970+0000) 2020-02-06T02:26:33.952 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:33.950+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.951648+0000) 2020-02-06T02:26:33.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:33.951+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.952395+0000) 2020-02-06T02:26:33.953 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:33.951+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.952800+0000) 2020-02-06T02:26:33.954 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:33.951+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.953311+0000) 2020-02-06T02:26:33.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:33.952+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.953599+0000) 2020-02-06T02:26:33.955 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:33.953+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.954105+0000) 2020-02-06T02:26:33.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:33.953+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.954579+0000) 2020-02-06T02:26:33.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:33.954+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.955202+0000) 2020-02-06T02:26:33.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:33.954+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.955695+0000) 2020-02-06T02:26:33.957 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:33.954+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.955657+0000) 2020-02-06T02:26:33.957 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:33.954+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.955571+0000) 2020-02-06T02:26:33.965 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:33.962+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:33.964051+0000) 2020-02-06T02:26:33.975 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:33.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232803 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:35.978 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:35.975+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:35.977027+0000) 2020-02-06T02:26:36.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:36.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:36.197317+0000) 2020-02-06T02:26:36.561 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:36.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:36.560180+0000) 2020-02-06T02:26:37.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:37.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.173118+0000) 2020-02-06T02:26:37.449 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:37.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.448496+0000) 2020-02-06T02:26:37.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:37.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.448498+0000) 2020-02-06T02:26:37.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:37.447+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.448953+0000) 2020-02-06T02:26:37.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:37.676+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.677850+0000) 2020-02-06T02:26:37.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:37.699+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:37.700913+0000) 2020-02-06T02:26:38.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:38.980+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.981193+0000) 2020-02-06T02:26:38.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:38.981+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.982114+0000) 2020-02-06T02:26:38.983 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:38.981+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.982559+0000) 2020-02-06T02:26:38.984 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:38.981+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.982927+0000) 2020-02-06T02:26:38.984 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:38.982+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.983852+0000) 2020-02-06T02:26:38.985 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:38.983+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.984669+0000) 2020-02-06T02:26:38.985 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:38.982+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.984359+0000) 2020-02-06T02:26:38.986 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:38.983+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.984414+0000) 2020-02-06T02:26:38.986 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:38.983+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.985054+0000) 2020-02-06T02:26:38.987 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:38.984+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.985370+0000) 2020-02-06T02:26:38.987 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:38.984+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.985725+0000) 2020-02-06T02:26:38.987 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:38.984+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.986229+0000) 2020-02-06T02:26:38.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:38.985+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.986182+0000) 2020-02-06T02:26:38.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:38.989+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:38.990992+0000) 2020-02-06T02:26:39.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:39.004+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232850 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:39.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:39.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:39.817607+0000) 2020-02-06T02:26:40.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:40.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:40.063136+0000) 2020-02-06T02:26:40.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:40.673+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:40.675243+0000) 2020-02-06T02:26:40.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:40.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:40.951137+0000) 2020-02-06T02:26:43.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:43.531+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:43.532577+0000) 2020-02-06T02:26:44.012 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:44.009+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.010910+0000) 2020-02-06T02:26:44.013 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:44.011+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.012432+0000) 2020-02-06T02:26:44.014 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:44.012+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.013105+0000) 2020-02-06T02:26:44.014 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:44.012+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.013469+0000) 2020-02-06T02:26:44.014 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:44.012+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.013682+0000) 2020-02-06T02:26:44.015 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:44.013+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.014206+0000) 2020-02-06T02:26:44.015 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:44.013+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.014551+0000) 2020-02-06T02:26:44.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:44.013+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.015009+0000) 2020-02-06T02:26:44.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:44.013+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.015389+0000) 2020-02-06T02:26:44.016 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:44.014+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.015751+0000) 2020-02-06T02:26:44.017 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:44.014+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.016307+0000) 2020-02-06T02:26:44.017 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:44.015+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.016640+0000) 2020-02-06T02:26:44.018 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:44.015+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.017207+0000) 2020-02-06T02:26:44.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:44.020+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:44.021996+0000) 2020-02-06T02:26:44.036 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:44.034+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232887 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:45.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:45.984+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:45.984843+0000) 2020-02-06T02:26:46.198 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:46.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:46.197583+0000) 2020-02-06T02:26:46.561 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:46.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:46.560477+0000) 2020-02-06T02:26:47.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:47.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.173354+0000) 2020-02-06T02:26:47.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:47.447+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.448737+0000) 2020-02-06T02:26:47.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:47.447+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.448734+0000) 2020-02-06T02:26:47.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:47.447+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.449181+0000) 2020-02-06T02:26:47.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:47.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.678032+0000) 2020-02-06T02:26:47.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:47.700+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:47.701164+0000) 2020-02-06T02:26:49.042 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:49.040+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.041714+0000) 2020-02-06T02:26:49.043 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:49.041+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.042232+0000) 2020-02-06T02:26:49.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:49.041+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.042618+0000) 2020-02-06T02:26:49.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:49.042+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.043218+0000) 2020-02-06T02:26:49.044 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:49.041+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.043355+0000) 2020-02-06T02:26:49.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:49.042+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.043987+0000) 2020-02-06T02:26:49.045 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:49.043+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.044297+0000) 2020-02-06T02:26:49.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:49.043+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.044453+0000) 2020-02-06T02:26:49.046 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:49.043+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.044785+0000) 2020-02-06T02:26:49.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:49.043+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.044994+0000) 2020-02-06T02:26:49.047 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:49.044+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.045500+0000) 2020-02-06T02:26:49.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:49.044+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.045559+0000) 2020-02-06T02:26:49.047 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:49.044+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.046258+0000) 2020-02-06T02:26:49.050 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:49.048+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.049633+0000) 2020-02-06T02:26:49.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:49.063+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232933 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:49.083 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:26:49.089 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:26:49.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:49.816+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:49.817797+0000) 2020-02-06T02:26:50.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:50.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:50.063354+0000) 2020-02-06T02:26:50.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:50.674+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:50.675518+0000) 2020-02-06T02:26:50.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:50.950+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:50.951361+0000) 2020-02-06T02:26:53.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:53.531+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:53.532855+0000) 2020-02-06T02:26:54.072 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:54.069+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.071044+0000) 2020-02-06T02:26:54.072 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:54.070+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.071529+0000) 2020-02-06T02:26:54.073 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:54.070+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.072168+0000) 2020-02-06T02:26:54.073 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:54.071+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.072361+0000) 2020-02-06T02:26:54.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:54.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.073188+0000) 2020-02-06T02:26:54.074 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:54.071+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.073322+0000) 2020-02-06T02:26:54.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:54.073+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.074407+0000) 2020-02-06T02:26:54.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:54.074+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.075211+0000) 2020-02-06T02:26:54.077 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:54.072+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.073734+0000) 2020-02-06T02:26:54.077 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:54.072+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.074401+0000) 2020-02-06T02:26:54.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:54.073+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.075190+0000) 2020-02-06T02:26:54.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:54.075+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.076496+0000) 2020-02-06T02:26:54.078 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:54.076+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.077597+0000) 2020-02-06T02:26:54.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:54.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:54.082272+0000) 2020-02-06T02:26:54.096 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:54.093+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 232971 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:55.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:55.992+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:55.993332+0000) 2020-02-06T02:26:56.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:56.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:56.197775+0000) 2020-02-06T02:26:56.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:56.559+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:56.560711+0000) 2020-02-06T02:26:57.174 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:57.172+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.173594+0000) 2020-02-06T02:26:57.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:57.448+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.448999+0000) 2020-02-06T02:26:57.450 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:57.448+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.449002+0000) 2020-02-06T02:26:57.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:57.448+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.449378+0000) 2020-02-06T02:26:57.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:57.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.678326+0000) 2020-02-06T02:26:57.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:57.700+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:57.701351+0000) 2020-02-06T02:26:59.102 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:26:59.100+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.101152+0000) 2020-02-06T02:26:59.103 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:26:59.100+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.101547+0000) 2020-02-06T02:26:59.103 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:26:59.100+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.102045+0000) 2020-02-06T02:26:59.104 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:26:59.101+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.102598+0000) 2020-02-06T02:26:59.104 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:26:59.102+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.103288+0000) 2020-02-06T02:26:59.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:26:59.102+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.103583+0000) 2020-02-06T02:26:59.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:26:59.102+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.104118+0000) 2020-02-06T02:26:59.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:59.104+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.105909+0000) 2020-02-06T02:26:59.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:59.105+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.106999+0000) 2020-02-06T02:26:59.119 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:59.111+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.112739+0000) 2020-02-06T02:26:59.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:26:59.103+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.104807+0000) 2020-02-06T02:26:59.119 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:26:59.103+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.105150+0000) 2020-02-06T02:26:59.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:26:59.104+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.105608+0000) 2020-02-06T02:26:59.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:26:59.105+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.107043+0000) 2020-02-06T02:26:59.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:26:59.124+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233018 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:26:59.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:26:59.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:26:59.818028+0000) 2020-02-06T02:27:00.064 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:00.062+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:00.063575+0000) 2020-02-06T02:27:00.676 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:00.675+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:00.675756+0000) 2020-02-06T02:27:00.952 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:00.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:00.951585+0000) 2020-02-06T02:27:03.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:03.532+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:03.533125+0000) 2020-02-06T02:27:04.132 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:04.129+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.130827+0000) 2020-02-06T02:27:04.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:04.130+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.131419+0000) 2020-02-06T02:27:04.133 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:04.131+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.132200+0000) 2020-02-06T02:27:04.134 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:04.132+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.133494+0000) 2020-02-06T02:27:04.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:04.133+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.134248+0000) 2020-02-06T02:27:04.136 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:04.133+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.134909+0000) 2020-02-06T02:27:04.136 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:04.133+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.135229+0000) 2020-02-06T02:27:04.137 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:04.134+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.135723+0000) 2020-02-06T02:27:04.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:04.134+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.136288+0000) 2020-02-06T02:27:04.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:04.135+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.136329+0000) 2020-02-06T02:27:04.138 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:04.135+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.136869+0000) 2020-02-06T02:27:04.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:04.136+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.137138+0000) 2020-02-06T02:27:04.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:04.136+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.137801+0000) 2020-02-06T02:27:04.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:04.141+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:04.142561+0000) 2020-02-06T02:27:04.156 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:04.154+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233055 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:06.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:06.000+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:06.001401+0000) 2020-02-06T02:27:06.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:06.197+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:06.198017+0000) 2020-02-06T02:27:06.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:06.560+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:06.560969+0000) 2020-02-06T02:27:07.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:07.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.173837+0000) 2020-02-06T02:27:07.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:07.448+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.449239+0000) 2020-02-06T02:27:07.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:07.448+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.449246+0000) 2020-02-06T02:27:07.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:07.448+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.449654+0000) 2020-02-06T02:27:07.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:07.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.678562+0000) 2020-02-06T02:27:07.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:07.700+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:07.701584+0000) 2020-02-06T02:27:09.163 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:09.160+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.161491+0000) 2020-02-06T02:27:09.163 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:09.160+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.161993+0000) 2020-02-06T02:27:09.163 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:09.161+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.162558+0000) 2020-02-06T02:27:09.164 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:09.161+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.163198+0000) 2020-02-06T02:27:09.164 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:09.162+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.163641+0000) 2020-02-06T02:27:09.165 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:09.163+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.164108+0000) 2020-02-06T02:27:09.165 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:09.163+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.164462+0000) 2020-02-06T02:27:09.166 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:09.163+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.165126+0000) 2020-02-06T02:27:09.166 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:09.164+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.165469+0000) 2020-02-06T02:27:09.167 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:09.164+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.166109+0000) 2020-02-06T02:27:09.167 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:09.165+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.166518+0000) 2020-02-06T02:27:09.168 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:09.165+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.167274+0000) 2020-02-06T02:27:09.174 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:09.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.173052+0000) 2020-02-06T02:27:09.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:09.173+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.174591+0000) 2020-02-06T02:27:09.185 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:09.183+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233101 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:09.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:09.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:09.818266+0000) 2020-02-06T02:27:10.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:10.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:10.063781+0000) 2020-02-06T02:27:10.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:10.675+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:10.676004+0000) 2020-02-06T02:27:10.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:10.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:10.951817+0000) 2020-02-06T02:27:13.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:13.532+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:13.533318+0000) 2020-02-06T02:27:14.192 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:14.189+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.190736+0000) 2020-02-06T02:27:14.192 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:14.190+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.191307+0000) 2020-02-06T02:27:14.193 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:14.190+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.191715+0000) 2020-02-06T02:27:14.193 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:14.191+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.192153+0000) 2020-02-06T02:27:14.202 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:14.191+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.192642+0000) 2020-02-06T02:27:14.202 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:14.191+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.192775+0000) 2020-02-06T02:27:14.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:14.192+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.193460+0000) 2020-02-06T02:27:14.203 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:14.192+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.194040+0000) 2020-02-06T02:27:14.203 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:14.193+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.194681+0000) 2020-02-06T02:27:14.203 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:14.194+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.195248+0000) 2020-02-06T02:27:14.204 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:14.197+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.199014+0000) 2020-02-06T02:27:14.204 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:14.192+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.193813+0000) 2020-02-06T02:27:14.204 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:14.193+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.194587+0000) 2020-02-06T02:27:14.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:14.195+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:14.196942+0000) 2020-02-06T02:27:14.218 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:14.214+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233139 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:16.011 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:16.009+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:16.010146+0000) 2020-02-06T02:27:16.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:16.196+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:16.198263+0000) 2020-02-06T02:27:16.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:16.560+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:16.561204+0000) 2020-02-06T02:27:17.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:17.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.174136+0000) 2020-02-06T02:27:17.450 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:17.448+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.449489+0000) 2020-02-06T02:27:17.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:17.448+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.449521+0000) 2020-02-06T02:27:17.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:17.448+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.449894+0000) 2020-02-06T02:27:17.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:17.677+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.678803+0000) 2020-02-06T02:27:17.702 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:17.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:17.701814+0000) 2020-02-06T02:27:19.223 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:19.221+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.222672+0000) 2020-02-06T02:27:19.224 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:19.222+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.223557+0000) 2020-02-06T02:27:19.225 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:19.222+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.223821+0000) 2020-02-06T02:27:19.225 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:19.223+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.224309+0000) 2020-02-06T02:27:19.226 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:19.223+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.224620+0000) 2020-02-06T02:27:19.226 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:19.223+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.224980+0000) 2020-02-06T02:27:19.227 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:19.224+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.225523+0000) 2020-02-06T02:27:19.227 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:19.224+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.225873+0000) 2020-02-06T02:27:19.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:19.225+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.226447+0000) 2020-02-06T02:27:19.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:19.225+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.226429+0000) 2020-02-06T02:27:19.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:19.225+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.227127+0000) 2020-02-06T02:27:19.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:19.226+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.228122+0000) 2020-02-06T02:27:19.231 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:19.226+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.228267+0000) 2020-02-06T02:27:19.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:19.233+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.234623+0000) 2020-02-06T02:27:19.246 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:19.244+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233186 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:19.255 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:27:19.260 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:27:19.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:19.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:19.818477+0000) 2020-02-06T02:27:20.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:20.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:20.063990+0000) 2020-02-06T02:27:20.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:20.675+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:20.676262+0000) 2020-02-06T02:27:20.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:20.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:20.952069+0000) 2020-02-06T02:27:23.534 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:23.533+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:23.533594+0000) 2020-02-06T02:27:24.253 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:24.252+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.252518+0000) 2020-02-06T02:27:24.254 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:24.252+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.253022+0000) 2020-02-06T02:27:24.254 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:24.252+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.253315+0000) 2020-02-06T02:27:24.255 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:24.254+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.254252+0000) 2020-02-06T02:27:24.255 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:24.253+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.254236+0000) 2020-02-06T02:27:24.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:24.254+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.254681+0000) 2020-02-06T02:27:24.256 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:24.255+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.255304+0000) 2020-02-06T02:27:24.256 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:24.254+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.254630+0000) 2020-02-06T02:27:24.256 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:24.255+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.255067+0000) 2020-02-06T02:27:24.257 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:24.254+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.255357+0000) 2020-02-06T02:27:24.257 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:24.255+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.255724+0000) 2020-02-06T02:27:24.257 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:24.256+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.256193+0000) 2020-02-06T02:27:24.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:24.256+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.256606+0000) 2020-02-06T02:27:24.262 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:24.261+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:24.261409+0000) 2020-02-06T02:27:24.277 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:24.275+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233223 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:26.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:26.018+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:26.019046+0000) 2020-02-06T02:27:26.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:26.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:26.198503+0000) 2020-02-06T02:27:26.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:26.561+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:26.561497+0000) 2020-02-06T02:27:27.175 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:27.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.174354+0000) 2020-02-06T02:27:27.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:27.449+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.449792+0000) 2020-02-06T02:27:27.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:27.449+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.449776+0000) 2020-02-06T02:27:27.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:27.449+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.450148+0000) 2020-02-06T02:27:27.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:27.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.679017+0000) 2020-02-06T02:27:27.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:27.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:27.702054+0000) 2020-02-06T02:27:29.282 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:29.280+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.281364+0000) 2020-02-06T02:27:29.285 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:29.282+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.282965+0000) 2020-02-06T02:27:29.285 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:29.283+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.283911+0000) 2020-02-06T02:27:29.286 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:29.281+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.282381+0000) 2020-02-06T02:27:29.286 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:29.283+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.284527+0000) 2020-02-06T02:27:29.287 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:29.284+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.284925+0000) 2020-02-06T02:27:29.287 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:29.285+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.285635+0000) 2020-02-06T02:27:29.287 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:29.283+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.284904+0000) 2020-02-06T02:27:29.288 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:29.284+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.285231+0000) 2020-02-06T02:27:29.288 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:29.284+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.285721+0000) 2020-02-06T02:27:29.288 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:29.285+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.286546+0000) 2020-02-06T02:27:29.289 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:29.285+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.286091+0000) 2020-02-06T02:27:29.289 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:29.285+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.286133+0000) 2020-02-06T02:27:29.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:29.290+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.291642+0000) 2020-02-06T02:27:29.307 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:29.306+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233269 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:29.819 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:29.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:29.818688+0000) 2020-02-06T02:27:30.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:30.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:30.064198+0000) 2020-02-06T02:27:30.677 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:30.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:30.676525+0000) 2020-02-06T02:27:30.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:30.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:30.952332+0000) 2020-02-06T02:27:33.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:33.532+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:33.533833+0000) 2020-02-06T02:27:34.314 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:34.312+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.313394+0000) 2020-02-06T02:27:34.316 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:34.313+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.314744+0000) 2020-02-06T02:27:34.316 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:34.314+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.314424+0000) 2020-02-06T02:27:34.316 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:34.315+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.315565+0000) 2020-02-06T02:27:34.317 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:34.315+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.316193+0000) 2020-02-06T02:27:34.317 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:34.316+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.316622+0000) 2020-02-06T02:27:34.318 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:34.315+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.316867+0000) 2020-02-06T02:27:34.318 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:34.316+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.317526+0000) 2020-02-06T02:27:34.319 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:34.317+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.317844+0000) 2020-02-06T02:27:34.319 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:34.317+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.318171+0000) 2020-02-06T02:27:34.319 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:34.316+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.317987+0000) 2020-02-06T02:27:34.319 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:34.317+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.318535+0000) 2020-02-06T02:27:34.320 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:34.318+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.319301+0000) 2020-02-06T02:27:34.325 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:34.323+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:34.324579+0000) 2020-02-06T02:27:34.339 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:34.337+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233308 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:36.028 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:36.026+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:36.027614+0000) 2020-02-06T02:27:36.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:36.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:36.198744+0000) 2020-02-06T02:27:36.562 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:36.561+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:36.561687+0000) 2020-02-06T02:27:37.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:37.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.174627+0000) 2020-02-06T02:27:37.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:37.449+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.449996+0000) 2020-02-06T02:27:37.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:37.449+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.450038+0000) 2020-02-06T02:27:37.451 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:37.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.450422+0000) 2020-02-06T02:27:37.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:37.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.679283+0000) 2020-02-06T02:27:37.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:37.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:37.702312+0000) 2020-02-06T02:27:39.345 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:39.343+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.344747+0000) 2020-02-06T02:27:39.346 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:39.344+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.344477+0000) 2020-02-06T02:27:39.346 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:39.344+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.345164+0000) 2020-02-06T02:27:39.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:39.344+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.345557+0000) 2020-02-06T02:27:39.347 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:39.344+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.345686+0000) 2020-02-06T02:27:39.347 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:39.345+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.346335+0000) 2020-02-06T02:27:39.348 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:39.345+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.346567+0000) 2020-02-06T02:27:39.349 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:39.346+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.346990+0000) 2020-02-06T02:27:39.349 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:39.346+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.347336+0000) 2020-02-06T02:27:39.349 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:39.347+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.347746+0000) 2020-02-06T02:27:39.349 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:39.347+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.348229+0000) 2020-02-06T02:27:39.350 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:39.347+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.348173+0000) 2020-02-06T02:27:39.350 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:39.348+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.348951+0000) 2020-02-06T02:27:39.353 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:39.351+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.352306+0000) 2020-02-06T02:27:39.369 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:39.368+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233355 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:39.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:39.817+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:39.818849+0000) 2020-02-06T02:27:40.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:40.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:40.064402+0000) 2020-02-06T02:27:40.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:40.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:40.676767+0000) 2020-02-06T02:27:40.953 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:40.951+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:40.952562+0000) 2020-02-06T02:27:43.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:43.533+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:43.534068+0000) 2020-02-06T02:27:44.375 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:44.372+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.373847+0000) 2020-02-06T02:27:44.375 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:44.373+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.374673+0000) 2020-02-06T02:27:44.376 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:44.374+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.375598+0000) 2020-02-06T02:27:44.377 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:44.375+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.376338+0000) 2020-02-06T02:27:44.379 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:44.377+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.378369+0000) 2020-02-06T02:27:44.380 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:44.377+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.377951+0000) 2020-02-06T02:27:44.380 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:44.376+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.377033+0000) 2020-02-06T02:27:44.381 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:44.377+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.378255+0000) 2020-02-06T02:27:44.381 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:44.376+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.377242+0000) 2020-02-06T02:27:44.381 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:44.376+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.377180+0000) 2020-02-06T02:27:44.381 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:44.378+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.378480+0000) 2020-02-06T02:27:44.382 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:44.378+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.379210+0000) 2020-02-06T02:27:44.382 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:44.379+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.380103+0000) 2020-02-06T02:27:44.383 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:44.380+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:44.381970+0000) 2020-02-06T02:27:44.399 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:44.398+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233393 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:46.037 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:46.034+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:46.035986+0000) 2020-02-06T02:27:46.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:46.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:46.199007+0000) 2020-02-06T02:27:46.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:46.561+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:46.561924+0000) 2020-02-06T02:27:47.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:47.173+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.174853+0000) 2020-02-06T02:27:47.451 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:47.450+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.450262+0000) 2020-02-06T02:27:47.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:47.450+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.450310+0000) 2020-02-06T02:27:47.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:47.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.450737+0000) 2020-02-06T02:27:47.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:47.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.679505+0000) 2020-02-06T02:27:47.703 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:47.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:47.702543+0000) 2020-02-06T02:27:49.390 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:27:49.410 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:27:49.411 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:49.403+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.404324+0000) 2020-02-06T02:27:49.412 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:49.404+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.405614+0000) 2020-02-06T02:27:49.412 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:49.405+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.406365+0000) 2020-02-06T02:27:49.412 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:49.405+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.406788+0000) 2020-02-06T02:27:49.413 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:49.405+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.406327+0000) 2020-02-06T02:27:49.413 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:49.406+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.407267+0000) 2020-02-06T02:27:49.414 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:49.408+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.408901+0000) 2020-02-06T02:27:49.415 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:49.409+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.410397+0000) 2020-02-06T02:27:49.416 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:49.410+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.410586+0000) 2020-02-06T02:27:49.416 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:49.406+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.407976+0000) 2020-02-06T02:27:49.417 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:49.408+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.409694+0000) 2020-02-06T02:27:49.417 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:49.410+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.411115+0000) 2020-02-06T02:27:49.418 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:49.410+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.411073+0000) 2020-02-06T02:27:49.420 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:49.413+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.414774+0000) 2020-02-06T02:27:49.437 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:49.429+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233439 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:49.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:49.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:49.819127+0000) 2020-02-06T02:27:50.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:50.063+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:50.064608+0000) 2020-02-06T02:27:50.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:50.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:50.677028+0000) 2020-02-06T02:27:50.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:50.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:50.952790+0000) 2020-02-06T02:27:53.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:53.533+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:53.534350+0000) 2020-02-06T02:27:54.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:54.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.436399+0000) 2020-02-06T02:27:54.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:54.436+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.437193+0000) 2020-02-06T02:27:54.438 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:54.436+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.437298+0000) 2020-02-06T02:27:54.439 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:54.436+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.437786+0000) 2020-02-06T02:27:54.439 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:54.437+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.438275+0000) 2020-02-06T02:27:54.439 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:54.438+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.438707+0000) 2020-02-06T02:27:54.446 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:54.440+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.440862+0000) 2020-02-06T02:27:54.447 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:54.440+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.440796+0000) 2020-02-06T02:27:54.447 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:54.441+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.441619+0000) 2020-02-06T02:27:54.447 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:54.441+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.441956+0000) 2020-02-06T02:27:54.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:54.438+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.439200+0000) 2020-02-06T02:27:54.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:54.438+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.439530+0000) 2020-02-06T02:27:54.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:54.439+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.440887+0000) 2020-02-06T02:27:54.448 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:54.446+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:54.447496+0000) 2020-02-06T02:27:54.461 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:54.459+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233478 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:56.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:56.047+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:56.047892+0000) 2020-02-06T02:27:56.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:56.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:56.199278+0000) 2020-02-06T02:27:56.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:56.562+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:56.562150+0000) 2020-02-06T02:27:57.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:57.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.175113+0000) 2020-02-06T02:27:57.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:57.450+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.450498+0000) 2020-02-06T02:27:57.452 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:57.450+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.450569+0000) 2020-02-06T02:27:57.452 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:57.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.450956+0000) 2020-02-06T02:27:57.680 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:57.678+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.679727+0000) 2020-02-06T02:27:57.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:57.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:57.702790+0000) 2020-02-06T02:27:59.466 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:27:59.463+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.464834+0000) 2020-02-06T02:27:59.467 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:59.465+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.466276+0000) 2020-02-06T02:27:59.468 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:27:59.465+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.466945+0000) 2020-02-06T02:27:59.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:59.466+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.467467+0000) 2020-02-06T02:27:59.469 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:27:59.467+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.468421+0000) 2020-02-06T02:27:59.470 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:27:59.467+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.468531+0000) 2020-02-06T02:27:59.470 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:27:59.467+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.468941+0000) 2020-02-06T02:27:59.470 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:27:59.467+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.469316+0000) 2020-02-06T02:27:59.471 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:27:59.468+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.469821+0000) 2020-02-06T02:27:59.471 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:27:59.469+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.470833+0000) 2020-02-06T02:27:59.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:27:59.469+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.471199+0000) 2020-02-06T02:27:59.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:27:59.469+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.471296+0000) 2020-02-06T02:27:59.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:27:59.470+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.472106+0000) 2020-02-06T02:27:59.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:59.476+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.477232+0000) 2020-02-06T02:27:59.491 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:27:59.489+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233525 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:27:59.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:27:59.818+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:27:59.819336+0000) 2020-02-06T02:28:00.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:00.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:00.064813+0000) 2020-02-06T02:28:00.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:00.675+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:00.677277+0000) 2020-02-06T02:28:00.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:00.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:00.953026+0000) 2020-02-06T02:28:03.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:03.534+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:03.534585+0000) 2020-02-06T02:28:04.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:04.496+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.497172+0000) 2020-02-06T02:28:04.500 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:04.497+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.498994+0000) 2020-02-06T02:28:04.500 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:04.498+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.499192+0000) 2020-02-06T02:28:04.501 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:04.498+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.499624+0000) 2020-02-06T02:28:04.501 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:04.498+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.499899+0000) 2020-02-06T02:28:04.502 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:04.499+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.500372+0000) 2020-02-06T02:28:04.502 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:04.499+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.500565+0000) 2020-02-06T02:28:04.503 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:04.500+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.501217+0000) 2020-02-06T02:28:04.504 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:04.499+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.501151+0000) 2020-02-06T02:28:04.504 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:04.500+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.501741+0000) 2020-02-06T02:28:04.504 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:04.500+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.502147+0000) 2020-02-06T02:28:04.505 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:04.500+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.502274+0000) 2020-02-06T02:28:04.505 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:04.500+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.501919+0000) 2020-02-06T02:28:04.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:04.507+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:04.508615+0000) 2020-02-06T02:28:04.522 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:04.519+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233563 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:06.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:06.055+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:06.056288+0000) 2020-02-06T02:28:06.200 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:06.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:06.199585+0000) 2020-02-06T02:28:06.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:06.561+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:06.562400+0000) 2020-02-06T02:28:07.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:07.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.175345+0000) 2020-02-06T02:28:07.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:07.449+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.450762+0000) 2020-02-06T02:28:07.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:07.449+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.450824+0000) 2020-02-06T02:28:07.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:07.449+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.451258+0000) 2020-02-06T02:28:07.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:07.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.679963+0000) 2020-02-06T02:28:07.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:07.701+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:07.703009+0000) 2020-02-06T02:28:09.529 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:09.526+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.527938+0000) 2020-02-06T02:28:09.529 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:09.527+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.528785+0000) 2020-02-06T02:28:09.531 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:09.528+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.530006+0000) 2020-02-06T02:28:09.531 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:09.529+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.530559+0000) 2020-02-06T02:28:09.531 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:09.529+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.530936+0000) 2020-02-06T02:28:09.532 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:09.530+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.531132+0000) 2020-02-06T02:28:09.533 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:09.530+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.531626+0000) 2020-02-06T02:28:09.533 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:09.530+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.531835+0000) 2020-02-06T02:28:09.533 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:09.531+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.532191+0000) 2020-02-06T02:28:09.534 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:09.530+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.532400+0000) 2020-02-06T02:28:09.534 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:09.531+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.532970+0000) 2020-02-06T02:28:09.535 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:09.531+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.533109+0000) 2020-02-06T02:28:09.535 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:09.531+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.532764+0000) 2020-02-06T02:28:09.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:09.538+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.539355+0000) 2020-02-06T02:28:09.554 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:09.551+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233610 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:09.820 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:09.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:09.819609+0000) 2020-02-06T02:28:10.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:10.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:10.065016+0000) 2020-02-06T02:28:10.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:10.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:10.677522+0000) 2020-02-06T02:28:10.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:10.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:10.953261+0000) 2020-02-06T02:28:13.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:13.534+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:13.534824+0000) 2020-02-06T02:28:14.560 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:14.558+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.559103+0000) 2020-02-06T02:28:14.561 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:14.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.560015+0000) 2020-02-06T02:28:14.561 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:14.559+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.560685+0000) 2020-02-06T02:28:14.562 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:14.560+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.561533+0000) 2020-02-06T02:28:14.562 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:14.560+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.561475+0000) 2020-02-06T02:28:14.563 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:14.560+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.562134+0000) 2020-02-06T02:28:14.563 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:14.561+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.562260+0000) 2020-02-06T02:28:14.563 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:14.561+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.562427+0000) 2020-02-06T02:28:14.564 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:14.562+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.563325+0000) 2020-02-06T02:28:14.564 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:14.562+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.563666+0000) 2020-02-06T02:28:14.565 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:14.562+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.563814+0000) 2020-02-06T02:28:14.565 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:14.563+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.564945+0000) 2020-02-06T02:28:14.566 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:14.563+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.564982+0000) 2020-02-06T02:28:14.570 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:14.568+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:14.569562+0000) 2020-02-06T02:28:14.584 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:14.582+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233648 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:16.065 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:16.064+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:16.064605+0000) 2020-02-06T02:28:16.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:16.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:16.199835+0000) 2020-02-06T02:28:16.563 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:16.561+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:16.562690+0000) 2020-02-06T02:28:17.176 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:17.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.175599+0000) 2020-02-06T02:28:17.452 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:17.450+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.451013+0000) 2020-02-06T02:28:17.452 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:17.450+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.451093+0000) 2020-02-06T02:28:17.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:17.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.451511+0000) 2020-02-06T02:28:17.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:17.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.680187+0000) 2020-02-06T02:28:17.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:17.703+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:17.703268+0000) 2020-02-06T02:28:19.590 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:19.589+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.589175+0000) 2020-02-06T02:28:19.590 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:19.589+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.589633+0000) 2020-02-06T02:28:19.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:19.590+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.590900+0000) 2020-02-06T02:28:19.592 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:19.590+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.591770+0000) 2020-02-06T02:28:19.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:19.591+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.591988+0000) 2020-02-06T02:28:19.593 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:19.592+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.592406+0000) 2020-02-06T02:28:19.594 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:19.591+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.592832+0000) 2020-02-06T02:28:19.595 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:19.592+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.593916+0000) 2020-02-06T02:28:19.595 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:19.594+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.594155+0000) 2020-02-06T02:28:19.595 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:19.594+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.594243+0000) 2020-02-06T02:28:19.596 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:19.593+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.595227+0000) 2020-02-06T02:28:19.596 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:19.593+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.595233+0000) 2020-02-06T02:28:19.597 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:19.594+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.595986+0000) 2020-02-06T02:28:19.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:19.599+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.599118+0000) 2020-02-06T02:28:19.619 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:19.612+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:19.630 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:28:19.634 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:28:19.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:19.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:19.819830+0000) 2020-02-06T02:28:20.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:20.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:20.065227+0000) 2020-02-06T02:28:20.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:20.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:20.677745+0000) 2020-02-06T02:28:20.954 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:20.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:20.953486+0000) 2020-02-06T02:28:23.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:23.534+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:23.535051+0000) 2020-02-06T02:28:24.627 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:24.619+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.621200+0000) 2020-02-06T02:28:24.628 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:24.620+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.622244+0000) 2020-02-06T02:28:24.628 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:24.622+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.623622+0000) 2020-02-06T02:28:24.628 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:24.623+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.624633+0000) 2020-02-06T02:28:24.628 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:24.623+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.625210+0000) 2020-02-06T02:28:24.628 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:24.624+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.626062+0000) 2020-02-06T02:28:24.629 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:24.618+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.619348+0000) 2020-02-06T02:28:24.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:24.619+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.620282+0000) 2020-02-06T02:28:24.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:24.622+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.623759+0000) 2020-02-06T02:28:24.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:24.619+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.620651+0000) 2020-02-06T02:28:24.630 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:24.621+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.622578+0000) 2020-02-06T02:28:24.630 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:24.621+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.623028+0000) 2020-02-06T02:28:24.630 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:24.623+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.624830+0000) 2020-02-06T02:28:24.632 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:24.630+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:24.631295+0000) 2020-02-06T02:28:24.645 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:24.643+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:26.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:26.072+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:26.073135+0000) 2020-02-06T02:28:26.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:26.198+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:26.200053+0000) 2020-02-06T02:28:26.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:26.562+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:26.562949+0000) 2020-02-06T02:28:27.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:27.174+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.175846+0000) 2020-02-06T02:28:27.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:27.450+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.451274+0000) 2020-02-06T02:28:27.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:27.450+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.451333+0000) 2020-02-06T02:28:27.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:27.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.451714+0000) 2020-02-06T02:28:27.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:27.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.680406+0000) 2020-02-06T02:28:27.704 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:27.702+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:27.703518+0000) 2020-02-06T02:28:29.652 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:29.649+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.651083+0000) 2020-02-06T02:28:29.653 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:29.650+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.651732+0000) 2020-02-06T02:28:29.653 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:29.651+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.652207+0000) 2020-02-06T02:28:29.654 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:29.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.652856+0000) 2020-02-06T02:28:29.654 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:29.651+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.653254+0000) 2020-02-06T02:28:29.655 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:29.653+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.654204+0000) 2020-02-06T02:28:29.655 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:29.653+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.654556+0000) 2020-02-06T02:28:29.656 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:29.653+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.654854+0000) 2020-02-06T02:28:29.657 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:29.655+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.656255+0000) 2020-02-06T02:28:29.658 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:29.654+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.656315+0000) 2020-02-06T02:28:29.658 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:29.655+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.656598+0000) 2020-02-06T02:28:29.658 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:29.655+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.656742+0000) 2020-02-06T02:28:29.659 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:29.655+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.657272+0000) 2020-02-06T02:28:29.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:29.662+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.663395+0000) 2020-02-06T02:28:29.675 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:29.673+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233779 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:29.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:29.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:29.820066+0000) 2020-02-06T02:28:30.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:30.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:30.065450+0000) 2020-02-06T02:28:30.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:30.676+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:30.677963+0000) 2020-02-06T02:28:30.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:30.952+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:30.953728+0000) 2020-02-06T02:28:33.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:33.534+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:33.535340+0000) 2020-02-06T02:28:34.681 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:34.678+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.679999+0000) 2020-02-06T02:28:34.682 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:34.679+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.681054+0000) 2020-02-06T02:28:34.682 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:34.679+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.680964+0000) 2020-02-06T02:28:34.683 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:34.680+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.681516+0000) 2020-02-06T02:28:34.683 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:34.681+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.682583+0000) 2020-02-06T02:28:34.684 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:34.681+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.682685+0000) 2020-02-06T02:28:34.685 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:34.682+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.683710+0000) 2020-02-06T02:28:34.685 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:34.683+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.684544+0000) 2020-02-06T02:28:34.686 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:34.684+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.685189+0000) 2020-02-06T02:28:34.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:34.684+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.685257+0000) 2020-02-06T02:28:34.687 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:34.684+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.685608+0000) 2020-02-06T02:28:34.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:34.684+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.685529+0000) 2020-02-06T02:28:34.688 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:34.684+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.686239+0000) 2020-02-06T02:28:34.692 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:34.690+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:34.691266+0000) 2020-02-06T02:28:34.704 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:34.701+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:36.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:36.081+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:36.082473+0000) 2020-02-06T02:28:36.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:36.199+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:36.200294+0000) 2020-02-06T02:28:36.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:36.562+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:36.563202+0000) 2020-02-06T02:28:37.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:37.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.176108+0000) 2020-02-06T02:28:37.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:37.450+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.451528+0000) 2020-02-06T02:28:37.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:37.450+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.451647+0000) 2020-02-06T02:28:37.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:37.450+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.451921+0000) 2020-02-06T02:28:37.681 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:37.679+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.680642+0000) 2020-02-06T02:28:37.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:37.702+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:37.703762+0000) 2020-02-06T02:28:39.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:39.708+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.709150+0000) 2020-02-06T02:28:39.711 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:39.708+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.710056+0000) 2020-02-06T02:28:39.711 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:39.709+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.710760+0000) 2020-02-06T02:28:39.712 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:39.710+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.711537+0000) 2020-02-06T02:28:39.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:39.711+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.712151+0000) 2020-02-06T02:28:39.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:39.710+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.712324+0000) 2020-02-06T02:28:39.714 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:39.711+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.712620+0000) 2020-02-06T02:28:39.714 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:39.711+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.713298+0000) 2020-02-06T02:28:39.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:39.712+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.713936+0000) 2020-02-06T02:28:39.715 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:39.712+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.714372+0000) 2020-02-06T02:28:39.715 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:39.713+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.714603+0000) 2020-02-06T02:28:39.716 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:39.713+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.714990+0000) 2020-02-06T02:28:39.716 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:39.714+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.715272+0000) 2020-02-06T02:28:39.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:39.719+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.720554+0000) 2020-02-06T02:28:39.735 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:39.732+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233863 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:39.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:39.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:39.820298+0000) 2020-02-06T02:28:40.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:40.064+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:40.065696+0000) 2020-02-06T02:28:40.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:40.677+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:40.678214+0000) 2020-02-06T02:28:40.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:40.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:40.953924+0000) 2020-02-06T02:28:43.536 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:43.534+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:43.535555+0000) 2020-02-06T02:28:44.741 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:44.739+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.740416+0000) 2020-02-06T02:28:44.742 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:44.739+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.740777+0000) 2020-02-06T02:28:44.742 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:44.740+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.741287+0000) 2020-02-06T02:28:44.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:44.740+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.741987+0000) 2020-02-06T02:28:44.743 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:44.740+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.742034+0000) 2020-02-06T02:28:44.743 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:44.741+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.742959+0000) 2020-02-06T02:28:44.745 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:44.742+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.743867+0000) 2020-02-06T02:28:44.745 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:44.742+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.743990+0000) 2020-02-06T02:28:44.746 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:44.743+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.744644+0000) 2020-02-06T02:28:44.746 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:44.743+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.744523+0000) 2020-02-06T02:28:44.746 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:44.743+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.745002+0000) 2020-02-06T02:28:44.746 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:44.744+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.745829+0000) 2020-02-06T02:28:44.747 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:44.744+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.745940+0000) 2020-02-06T02:28:44.752 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:44.750+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:44.751299+0000) 2020-02-06T02:28:44.765 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:44.763+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233901 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:46.092 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:46.090+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:46.090778+0000) 2020-02-06T02:28:46.201 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:46.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:46.200560+0000) 2020-02-06T02:28:46.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:46.562+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:46.563401+0000) 2020-02-06T02:28:47.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:47.175+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.176365+0000) 2020-02-06T02:28:47.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:47.451+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.451765+0000) 2020-02-06T02:28:47.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:47.451+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.451889+0000) 2020-02-06T02:28:47.453 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:47.451+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.452155+0000) 2020-02-06T02:28:47.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:47.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.680908+0000) 2020-02-06T02:28:47.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:47.703+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:47.704034+0000) 2020-02-06T02:28:49.761 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:28:49.768 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:28:49.770 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:49.768+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.769177+0000) 2020-02-06T02:28:49.773 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:49.770+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.771788+0000) 2020-02-06T02:28:49.773 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:49.771+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.772537+0000) 2020-02-06T02:28:49.774 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:49.771+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.772600+0000) 2020-02-06T02:28:49.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:49.771+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.772895+0000) 2020-02-06T02:28:49.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:49.770+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.771320+0000) 2020-02-06T02:28:49.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:49.771+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.772067+0000) 2020-02-06T02:28:49.775 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:49.769+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.770580+0000) 2020-02-06T02:28:49.776 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:49.769+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.770167+0000) 2020-02-06T02:28:49.776 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:49.768+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.769673+0000) 2020-02-06T02:28:49.776 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:49.772+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.773430+0000) 2020-02-06T02:28:49.777 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:49.772+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.773653+0000) 2020-02-06T02:28:49.777 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:49.772+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.773903+0000) 2020-02-06T02:28:49.780 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:49.778+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.779386+0000) 2020-02-06T02:28:49.797 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:49.793+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233947 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:49.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:49.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:49.820474+0000) 2020-02-06T02:28:50.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:50.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:50.065913+0000) 2020-02-06T02:28:50.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:50.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:50.678469+0000) 2020-02-06T02:28:50.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:50.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:50.954117+0000) 2020-02-06T02:28:53.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:53.535+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:53.535784+0000) 2020-02-06T02:28:54.802 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:54.799+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.800777+0000) 2020-02-06T02:28:54.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:54.801+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.802588+0000) 2020-02-06T02:28:54.804 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:54.801+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.802351+0000) 2020-02-06T02:28:54.804 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:54.802+0000 7f72af4f5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.803225+0000) 2020-02-06T02:28:54.804 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:54.802+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.803639+0000) 2020-02-06T02:28:54.805 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:54.803+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.804284+0000) 2020-02-06T02:28:54.806 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:54.804+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.804803+0000) 2020-02-06T02:28:54.806 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:54.804+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.805225+0000) 2020-02-06T02:28:54.806 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:54.805+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.805845+0000) 2020-02-06T02:28:54.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:54.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.806300+0000) 2020-02-06T02:28:54.808 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:54.806+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.806518+0000) 2020-02-06T02:28:54.808 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:54.806+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.806757+0000) 2020-02-06T02:28:54.808 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:54.805+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.806833+0000) 2020-02-06T02:28:54.810 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:54.808+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:54.809767+0000) 2020-02-06T02:28:54.826 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:54.824+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 233986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:28:56.099 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:56.097+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:56.098061+0000) 2020-02-06T02:28:56.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:56.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:56.200813+0000) 2020-02-06T02:28:56.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:56.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:56.563624+0000) 2020-02-06T02:28:57.177 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:57.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.176607+0000) 2020-02-06T02:28:57.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:57.451+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.452019+0000) 2020-02-06T02:28:57.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:57.451+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.452135+0000) 2020-02-06T02:28:57.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:57.451+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.452373+0000) 2020-02-06T02:28:57.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:57.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.681134+0000) 2020-02-06T02:28:57.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:57.703+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:57.704289+0000) 2020-02-06T02:28:59.821 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:59.819+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.820662+0000) 2020-02-06T02:28:59.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:28:59.829+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.829967+0000) 2020-02-06T02:28:59.833 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:28:59.832+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.832594+0000) 2020-02-06T02:28:59.834 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:28:59.832+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.833139+0000) 2020-02-06T02:28:59.834 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:28:59.832+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.833647+0000) 2020-02-06T02:28:59.834 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:28:59.832+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.834032+0000) 2020-02-06T02:28:59.835 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:28:59.833+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.834470+0000) 2020-02-06T02:28:59.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:59.833+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.834495+0000) 2020-02-06T02:28:59.836 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:28:59.833+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.834569+0000) 2020-02-06T02:28:59.836 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:28:59.834+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.835162+0000) 2020-02-06T02:28:59.836 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:28:59.834+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.835167+0000) 2020-02-06T02:28:59.837 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:28:59.834+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.834901+0000) 2020-02-06T02:28:59.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:59.834+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.835349+0000) 2020-02-06T02:28:59.837 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:28:59.835+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.836070+0000) 2020-02-06T02:28:59.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:28:59.842+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:28:59.843990+0000) 2020-02-06T02:28:59.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:28:59.853+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234031 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:00.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:00.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:00.066132+0000) 2020-02-06T02:29:00.679 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:00.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:00.678642+0000) 2020-02-06T02:29:00.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:00.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:00.954343+0000) 2020-02-06T02:29:03.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:03.535+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:03.536018+0000) 2020-02-06T02:29:04.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:04.858+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.859069+0000) 2020-02-06T02:29:04.861 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:04.859+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.860427+0000) 2020-02-06T02:29:04.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:04.860+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.861144+0000) 2020-02-06T02:29:04.862 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:04.860+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.861297+0000) 2020-02-06T02:29:04.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:04.860+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.862014+0000) 2020-02-06T02:29:04.863 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:04.861+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.862647+0000) 2020-02-06T02:29:04.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:04.862+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.862786+0000) 2020-02-06T02:29:04.864 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:04.862+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.863445+0000) 2020-02-06T02:29:04.865 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:04.863+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.863780+0000) 2020-02-06T02:29:04.865 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:04.863+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.864185+0000) 2020-02-06T02:29:04.865 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:04.864+0000 7fd456e20700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.864938+0000) 2020-02-06T02:29:04.866 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:04.864+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.865370+0000) 2020-02-06T02:29:04.867 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:04.864+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.866039+0000) 2020-02-06T02:29:04.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:04.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:04.869084+0000) 2020-02-06T02:29:04.887 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:04.885+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234068 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:06.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:06.103+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:06.104945+0000) 2020-02-06T02:29:06.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:06.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:06.201079+0000) 2020-02-06T02:29:06.564 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:06.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:06.563846+0000) 2020-02-06T02:29:07.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:07.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.176842+0000) 2020-02-06T02:29:07.453 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:07.451+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.452255+0000) 2020-02-06T02:29:07.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:07.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.452451+0000) 2020-02-06T02:29:07.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:07.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.452665+0000) 2020-02-06T02:29:07.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:07.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.681373+0000) 2020-02-06T02:29:07.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:07.704+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:07.704523+0000) 2020-02-06T02:29:09.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:09.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.820889+0000) 2020-02-06T02:29:09.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:09.890+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.891419+0000) 2020-02-06T02:29:09.894 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:09.891+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.892977+0000) 2020-02-06T02:29:09.894 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:09.892+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.893562+0000) 2020-02-06T02:29:09.895 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:09.893+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.894143+0000) 2020-02-06T02:29:09.895 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:09.893+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.894501+0000) 2020-02-06T02:29:09.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:09.893+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.893977+0000) 2020-02-06T02:29:09.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:09.895+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.895656+0000) 2020-02-06T02:29:09.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:09.894+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.895350+0000) 2020-02-06T02:29:09.897 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:09.894+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.895567+0000) 2020-02-06T02:29:09.897 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:09.896+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.896507+0000) 2020-02-06T02:29:09.898 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:09.897+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.897596+0000) 2020-02-06T02:29:09.898 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:09.897+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.897635+0000) 2020-02-06T02:29:09.899 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:09.897+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.897671+0000) 2020-02-06T02:29:09.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:09.901+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:09.902860+0000) 2020-02-06T02:29:09.917 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:09.916+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234114 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:10.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:10.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:10.066363+0000) 2020-02-06T02:29:10.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:10.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:10.678878+0000) 2020-02-06T02:29:10.955 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:10.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:10.954566+0000) 2020-02-06T02:29:13.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:13.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:13.536282+0000) 2020-02-06T02:29:14.923 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:14.921+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.921854+0000) 2020-02-06T02:29:14.924 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:14.923+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.923878+0000) 2020-02-06T02:29:14.925 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:14.924+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.924296+0000) 2020-02-06T02:29:14.925 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:14.924+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.924808+0000) 2020-02-06T02:29:14.927 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:14.925+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.925610+0000) 2020-02-06T02:29:14.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:14.924+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.925272+0000) 2020-02-06T02:29:14.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:14.925+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.926207+0000) 2020-02-06T02:29:14.928 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:14.927+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.927299+0000) 2020-02-06T02:29:14.928 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:14.926+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.927272+0000) 2020-02-06T02:29:14.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:14.928+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.928305+0000) 2020-02-06T02:29:14.929 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:14.927+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.927960+0000) 2020-02-06T02:29:14.930 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:14.927+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.928092+0000) 2020-02-06T02:29:14.930 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:14.928+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.928521+0000) 2020-02-06T02:29:14.933 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:14.932+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:14.932519+0000) 2020-02-06T02:29:14.950 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:14.947+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234152 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:16.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:16.116+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:16.116464+0000) 2020-02-06T02:29:16.202 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:16.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:16.201338+0000) 2020-02-06T02:29:16.565 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:16.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:16.564068+0000) 2020-02-06T02:29:17.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:17.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.177116+0000) 2020-02-06T02:29:17.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:17.452+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.452512+0000) 2020-02-06T02:29:17.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:17.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.452723+0000) 2020-02-06T02:29:17.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:17.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.452906+0000) 2020-02-06T02:29:17.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:17.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.681595+0000) 2020-02-06T02:29:17.705 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:17.703+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:17.704755+0000) 2020-02-06T02:29:19.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:19.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.821126+0000) 2020-02-06T02:29:19.924 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:29:19.931 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:29:19.953 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:19.951+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.952582+0000) 2020-02-06T02:29:19.954 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:19.952+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.953133+0000) 2020-02-06T02:29:19.955 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:19.953+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.954305+0000) 2020-02-06T02:29:19.956 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:19.953+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.954846+0000) 2020-02-06T02:29:19.956 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:19.954+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.955409+0000) 2020-02-06T02:29:19.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:19.955+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.956420+0000) 2020-02-06T02:29:19.957 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:19.955+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.956455+0000) 2020-02-06T02:29:19.959 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:19.956+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.956666+0000) 2020-02-06T02:29:19.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:19.956+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.956994+0000) 2020-02-06T02:29:19.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:19.956+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.957287+0000) 2020-02-06T02:29:19.959 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:19.954+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.955244+0000) 2020-02-06T02:29:19.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:19.957+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.957660+0000) 2020-02-06T02:29:19.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:19.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.958677+0000) 2020-02-06T02:29:19.960 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:19.957+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:19.958907+0000) 2020-02-06T02:29:19.977 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:19.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234197 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:20.067 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:20.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:20.066619+0000) 2020-02-06T02:29:20.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:20.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:20.679107+0000) 2020-02-06T02:29:20.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:20.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:20.954808+0000) 2020-02-06T02:29:23.537 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:23.535+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:23.536550+0000) 2020-02-06T02:29:24.984 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:24.982+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.983543+0000) 2020-02-06T02:29:24.985 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:24.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.984365+0000) 2020-02-06T02:29:24.985 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:24.983+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.984419+0000) 2020-02-06T02:29:24.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:24.985+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.985521+0000) 2020-02-06T02:29:24.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:24.985+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.986103+0000) 2020-02-06T02:29:24.987 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:24.985+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.986265+0000) 2020-02-06T02:29:24.987 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:24.985+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.987007+0000) 2020-02-06T02:29:24.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:24.986+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.987654+0000) 2020-02-06T02:29:24.989 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:24.987+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.987618+0000) 2020-02-06T02:29:24.989 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:24.987+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.987940+0000) 2020-02-06T02:29:24.990 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:24.987+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.988282+0000) 2020-02-06T02:29:24.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:24.987+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.988849+0000) 2020-02-06T02:29:24.990 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:24.988+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.988970+0000) 2020-02-06T02:29:24.995 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:24.993+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:24.994072+0000) 2020-02-06T02:29:25.008 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:25.007+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234234 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:26.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:26.127+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:26.128476+0000) 2020-02-06T02:29:26.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:26.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:26.201671+0000) 2020-02-06T02:29:26.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:26.564+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:26.564318+0000) 2020-02-06T02:29:27.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:27.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.177305+0000) 2020-02-06T02:29:27.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:27.452+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.452808+0000) 2020-02-06T02:29:27.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:27.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.452962+0000) 2020-02-06T02:29:27.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:27.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.453152+0000) 2020-02-06T02:29:27.682 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:27.680+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.681829+0000) 2020-02-06T02:29:27.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:27.703+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:27.704982+0000) 2020-02-06T02:29:29.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:29.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:29.821327+0000) 2020-02-06T02:29:30.015 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:30.012+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.013955+0000) 2020-02-06T02:29:30.016 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:30.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.014938+0000) 2020-02-06T02:29:30.016 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:30.014+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.015426+0000) 2020-02-06T02:29:30.016 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:30.014+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.015795+0000) 2020-02-06T02:29:30.017 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:30.014+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.016009+0000) 2020-02-06T02:29:30.017 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:30.015+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.016519+0000) 2020-02-06T02:29:30.018 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:30.015+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.017194+0000) 2020-02-06T02:29:30.019 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:30.016+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.017865+0000) 2020-02-06T02:29:30.019 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:30.017+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.018422+0000) 2020-02-06T02:29:30.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:30.018+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.019159+0000) 2020-02-06T02:29:30.020 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:30.018+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.019311+0000) 2020-02-06T02:29:30.021 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:30.017+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.018897+0000) 2020-02-06T02:29:30.021 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:30.018+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.019555+0000) 2020-02-06T02:29:30.026 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:30.024+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.025242+0000) 2020-02-06T02:29:30.040 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:30.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234280 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:30.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:30.065+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.066856+0000) 2020-02-06T02:29:30.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:30.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.679337+0000) 2020-02-06T02:29:30.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:30.953+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:30.955027+0000) 2020-02-06T02:29:33.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:33.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:33.536782+0000) 2020-02-06T02:29:35.045 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:35.043+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.044488+0000) 2020-02-06T02:29:35.046 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:35.044+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.045506+0000) 2020-02-06T02:29:35.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:35.044+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.046019+0000) 2020-02-06T02:29:35.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:35.045+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.046569+0000) 2020-02-06T02:29:35.048 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:35.045+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.047144+0000) 2020-02-06T02:29:35.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:35.046+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.047812+0000) 2020-02-06T02:29:35.049 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:35.047+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.048736+0000) 2020-02-06T02:29:35.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:35.047+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.048847+0000) 2020-02-06T02:29:35.050 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:35.048+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.049489+0000) 2020-02-06T02:29:35.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:35.048+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.050041+0000) 2020-02-06T02:29:35.051 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:35.049+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.050348+0000) 2020-02-06T02:29:35.052 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:35.049+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.050762+0000) 2020-02-06T02:29:35.052 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:35.049+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.051079+0000) 2020-02-06T02:29:35.058 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:35.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:35.057705+0000) 2020-02-06T02:29:35.070 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:35.067+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234318 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:36.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:36.135+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:36.136483+0000) 2020-02-06T02:29:36.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:36.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:36.201938+0000) 2020-02-06T02:29:36.565 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:36.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:36.564574+0000) 2020-02-06T02:29:37.178 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:37.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.177541+0000) 2020-02-06T02:29:37.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:37.452+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.453058+0000) 2020-02-06T02:29:37.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:37.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.453192+0000) 2020-02-06T02:29:37.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:37.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.453384+0000) 2020-02-06T02:29:37.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:37.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.682071+0000) 2020-02-06T02:29:37.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:37.704+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:37.705258+0000) 2020-02-06T02:29:39.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:39.820+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:39.821567+0000) 2020-02-06T02:29:40.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:40.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.067059+0000) 2020-02-06T02:29:40.075 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:40.072+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.073967+0000) 2020-02-06T02:29:40.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:40.074+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.075677+0000) 2020-02-06T02:29:40.077 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:40.075+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.076894+0000) 2020-02-06T02:29:40.078 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:40.076+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.077953+0000) 2020-02-06T02:29:40.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:40.077+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.078093+0000) 2020-02-06T02:29:40.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:40.077+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.078625+0000) 2020-02-06T02:29:40.080 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:40.077+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.078954+0000) 2020-02-06T02:29:40.080 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:40.078+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.079339+0000) 2020-02-06T02:29:40.080 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:40.078+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.080014+0000) 2020-02-06T02:29:40.081 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:40.079+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.080579+0000) 2020-02-06T02:29:40.082 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:40.079+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.080800+0000) 2020-02-06T02:29:40.082 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:40.079+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.080911+0000) 2020-02-06T02:29:40.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:40.079+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.081216+0000) 2020-02-06T02:29:40.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:40.081+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.082332+0000) 2020-02-06T02:29:40.100 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:40.097+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234363 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:40.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:40.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.679526+0000) 2020-02-06T02:29:40.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:40.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:40.955267+0000) 2020-02-06T02:29:43.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:43.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:43.537009+0000) 2020-02-06T02:29:45.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:45.105+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.106558+0000) 2020-02-06T02:29:45.108 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:45.105+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.106810+0000) 2020-02-06T02:29:45.109 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:45.106+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.107327+0000) 2020-02-06T02:29:45.109 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:45.106+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.107543+0000) 2020-02-06T02:29:45.110 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:45.106+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.108057+0000) 2020-02-06T02:29:45.111 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:45.106+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.108042+0000) 2020-02-06T02:29:45.111 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:45.107+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.108522+0000) 2020-02-06T02:29:45.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:45.107+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.108882+0000) 2020-02-06T02:29:45.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:45.108+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.109325+0000) 2020-02-06T02:29:45.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:45.108+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.109841+0000) 2020-02-06T02:29:45.112 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:45.108+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.110160+0000) 2020-02-06T02:29:45.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:45.108+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.110156+0000) 2020-02-06T02:29:45.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:45.109+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.111308+0000) 2020-02-06T02:29:45.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:45.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:45.116652+0000) 2020-02-06T02:29:45.130 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:45.127+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234400 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:46.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:46.144+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:46.144921+0000) 2020-02-06T02:29:46.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:46.200+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:46.202174+0000) 2020-02-06T02:29:46.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:46.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:46.564819+0000) 2020-02-06T02:29:47.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:47.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.177796+0000) 2020-02-06T02:29:47.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:47.452+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.453329+0000) 2020-02-06T02:29:47.454 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:47.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.453500+0000) 2020-02-06T02:29:47.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:47.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.453660+0000) 2020-02-06T02:29:47.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:47.681+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.682293+0000) 2020-02-06T02:29:47.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:47.705+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:47.705486+0000) 2020-02-06T02:29:49.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:49.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:49.821792+0000) 2020-02-06T02:29:50.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:50.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.067320+0000) 2020-02-06T02:29:50.122 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:29:50.129 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:29:50.137 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:50.134+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.135554+0000) 2020-02-06T02:29:50.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:50.135+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.136213+0000) 2020-02-06T02:29:50.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:50.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.136425+0000) 2020-02-06T02:29:50.138 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:50.135+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.136851+0000) 2020-02-06T02:29:50.138 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:50.136+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.137213+0000) 2020-02-06T02:29:50.139 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:50.136+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.137523+0000) 2020-02-06T02:29:50.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:50.136+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.137820+0000) 2020-02-06T02:29:50.140 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:50.137+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.138522+0000) 2020-02-06T02:29:50.140 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:50.137+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.138896+0000) 2020-02-06T02:29:50.140 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:50.138+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.139665+0000) 2020-02-06T02:29:50.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:50.138+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.140251+0000) 2020-02-06T02:29:50.142 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:50.139+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.140812+0000) 2020-02-06T02:29:50.142 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:50.139+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.140925+0000) 2020-02-06T02:29:50.146 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:50.144+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.145325+0000) 2020-02-06T02:29:50.161 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:50.158+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234446 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:50.680 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:50.678+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.679742+0000) 2020-02-06T02:29:50.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:50.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:50.955476+0000) 2020-02-06T02:29:53.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:53.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:53.537266+0000) 2020-02-06T02:29:55.166 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:55.164+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.165423+0000) 2020-02-06T02:29:55.167 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:55.165+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.166185+0000) 2020-02-06T02:29:55.167 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:55.165+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.166639+0000) 2020-02-06T02:29:55.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:29:55.165+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.166716+0000) 2020-02-06T02:29:55.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:55.166+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.167644+0000) 2020-02-06T02:29:55.169 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:29:55.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.168387+0000) 2020-02-06T02:29:55.169 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:29:55.166+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.168268+0000) 2020-02-06T02:29:55.170 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:55.167+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.169318+0000) 2020-02-06T02:29:55.171 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:55.168+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.169909+0000) 2020-02-06T02:29:55.171 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:55.168+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.170283+0000) 2020-02-06T02:29:55.171 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:55.169+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.170826+0000) 2020-02-06T02:29:55.172 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:55.170+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.171573+0000) 2020-02-06T02:29:55.172 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:55.170+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.172051+0000) 2020-02-06T02:29:55.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:55.173+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:55.174459+0000) 2020-02-06T02:29:55.191 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:29:55.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234484 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:29:56.154 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:56.152+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:56.153188+0000) 2020-02-06T02:29:56.203 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:29:56.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:56.202465+0000) 2020-02-06T02:29:56.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:29:56.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:56.565088+0000) 2020-02-06T02:29:57.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:29:57.176+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.178024+0000) 2020-02-06T02:29:57.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:29:57.452+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.453565+0000) 2020-02-06T02:29:57.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:29:57.452+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.453752+0000) 2020-02-06T02:29:57.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:29:57.452+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.453908+0000) 2020-02-06T02:29:57.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:29:57.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.682603+0000) 2020-02-06T02:29:57.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:29:57.704+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:57.705711+0000) 2020-02-06T02:29:59.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:29:59.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:29:59.822018+0000) 2020-02-06T02:30:00.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:00.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.067548+0000) 2020-02-06T02:30:00.197 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:00.194+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.196406+0000) 2020-02-06T02:30:00.198 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:00.195+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.197180+0000) 2020-02-06T02:30:00.198 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:00.196+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.197509+0000) 2020-02-06T02:30:00.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:00.196+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.198017+0000) 2020-02-06T02:30:00.199 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:00.197+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.198375+0000) 2020-02-06T02:30:00.200 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:00.198+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.199258+0000) 2020-02-06T02:30:00.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:00.198+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.199717+0000) 2020-02-06T02:30:00.201 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:00.198+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.199962+0000) 2020-02-06T02:30:00.202 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:00.199+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.201016+0000) 2020-02-06T02:30:00.202 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:00.200+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.201503+0000) 2020-02-06T02:30:00.202 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:00.200+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.201533+0000) 2020-02-06T02:30:00.203 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:00.200+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.202344+0000) 2020-02-06T02:30:00.203 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:00.201+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.202594+0000) 2020-02-06T02:30:00.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:00.206+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.207259+0000) 2020-02-06T02:30:00.222 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:00.219+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234529 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:00.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:00.679+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.679973+0000) 2020-02-06T02:30:00.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:00.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:00.955690+0000) 2020-02-06T02:30:03.538 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:03.537+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:03.537515+0000) 2020-02-06T02:30:05.228 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:05.225+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.226698+0000) 2020-02-06T02:30:05.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:05.225+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.227062+0000) 2020-02-06T02:30:05.229 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:05.227+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.228564+0000) 2020-02-06T02:30:05.231 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:05.227+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.229038+0000) 2020-02-06T02:30:05.231 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:05.228+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.229706+0000) 2020-02-06T02:30:05.232 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:05.229+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.230406+0000) 2020-02-06T02:30:05.233 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:05.229+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.230769+0000) 2020-02-06T02:30:05.233 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:05.230+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.231876+0000) 2020-02-06T02:30:05.234 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:05.230+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.231177+0000) 2020-02-06T02:30:05.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:05.230+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.231543+0000) 2020-02-06T02:30:05.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:05.231+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.232126+0000) 2020-02-06T02:30:05.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:05.231+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.232347+0000) 2020-02-06T02:30:05.235 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:05.231+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.232484+0000) 2020-02-06T02:30:05.239 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:05.237+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:05.238351+0000) 2020-02-06T02:30:05.253 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:05.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234566 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:06.162 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:06.161+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:06.161166+0000) 2020-02-06T02:30:06.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:06.201+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:06.202752+0000) 2020-02-06T02:30:06.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:06.563+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:06.565318+0000) 2020-02-06T02:30:07.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:07.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.178308+0000) 2020-02-06T02:30:07.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:07.453+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.453803+0000) 2020-02-06T02:30:07.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:07.453+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.453961+0000) 2020-02-06T02:30:07.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:07.453+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.454138+0000) 2020-02-06T02:30:07.683 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:07.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.682812+0000) 2020-02-06T02:30:07.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:07.705+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:07.705942+0000) 2020-02-06T02:30:09.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:09.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:09.822292+0000) 2020-02-06T02:30:10.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:10.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.067771+0000) 2020-02-06T02:30:10.258 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:10.256+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.257557+0000) 2020-02-06T02:30:10.259 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:10.257+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.257905+0000) 2020-02-06T02:30:10.259 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:10.258+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.258728+0000) 2020-02-06T02:30:10.260 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:10.259+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.259114+0000) 2020-02-06T02:30:10.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:10.257+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.259366+0000) 2020-02-06T02:30:10.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:10.258+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.260245+0000) 2020-02-06T02:30:10.261 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:10.260+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.260463+0000) 2020-02-06T02:30:10.262 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:10.260+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.261502+0000) 2020-02-06T02:30:10.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:10.261+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.261402+0000) 2020-02-06T02:30:10.263 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:10.260+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.262199+0000) 2020-02-06T02:30:10.263 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:10.262+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.262432+0000) 2020-02-06T02:30:10.264 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:10.261+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.263061+0000) 2020-02-06T02:30:10.264 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:10.263+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.263566+0000) 2020-02-06T02:30:10.268 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:10.267+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.267646+0000) 2020-02-06T02:30:10.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:10.281+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234612 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:10.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:10.679+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.680207+0000) 2020-02-06T02:30:10.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:10.954+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:10.955896+0000) 2020-02-06T02:30:13.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:13.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:13.537758+0000) 2020-02-06T02:30:15.288 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:15.286+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.287377+0000) 2020-02-06T02:30:15.290 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:15.287+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.288845+0000) 2020-02-06T02:30:15.290 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:15.288+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.289302+0000) 2020-02-06T02:30:15.291 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:15.289+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.290111+0000) 2020-02-06T02:30:15.291 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:15.288+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.290185+0000) 2020-02-06T02:30:15.291 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:15.288+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.290400+0000) 2020-02-06T02:30:15.292 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:15.290+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.292000+0000) 2020-02-06T02:30:15.293 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:15.291+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.292440+0000) 2020-02-06T02:30:15.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:15.291+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.292963+0000) 2020-02-06T02:30:15.294 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:15.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.293169+0000) 2020-02-06T02:30:15.294 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:15.291+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.292948+0000) 2020-02-06T02:30:15.295 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:15.292+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.293722+0000) 2020-02-06T02:30:15.295 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:15.292+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.293966+0000) 2020-02-06T02:30:15.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:15.294+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:15.295573+0000) 2020-02-06T02:30:15.313 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:15.311+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234650 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:16.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:16.168+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:16.169422+0000) 2020-02-06T02:30:16.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:16.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:16.202995+0000) 2020-02-06T02:30:16.566 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:16.565+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:16.565583+0000) 2020-02-06T02:30:17.179 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:17.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.178560+0000) 2020-02-06T02:30:17.455 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:17.453+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.454049+0000) 2020-02-06T02:30:17.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:17.453+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.454182+0000) 2020-02-06T02:30:17.455 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:17.453+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.454343+0000) 2020-02-06T02:30:17.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:17.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.683128+0000) 2020-02-06T02:30:17.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:17.705+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:17.706216+0000) 2020-02-06T02:30:19.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:19.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:19.822510+0000) 2020-02-06T02:30:20.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:20.066+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.067992+0000) 2020-02-06T02:30:20.319 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:20.317+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.318555+0000) 2020-02-06T02:30:20.320 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:20.318+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.319463+0000) 2020-02-06T02:30:20.320 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:20.318+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.319819+0000) 2020-02-06T02:30:20.321 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:20.319+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.320342+0000) 2020-02-06T02:30:20.321 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:20.319+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.320227+0000) 2020-02-06T02:30:20.322 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:20.319+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.320818+0000) 2020-02-06T02:30:20.322 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:20.320+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.321255+0000) 2020-02-06T02:30:20.322 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:20.320+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.321404+0000) 2020-02-06T02:30:20.323 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:20.321+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.322150+0000) 2020-02-06T02:30:20.324 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:20.322+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.322466+0000) 2020-02-06T02:30:20.324 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:20.322+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.323380+0000) 2020-02-06T02:30:20.324 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:20.322+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.323018+0000) 2020-02-06T02:30:20.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:20.322+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.324027+0000) 2020-02-06T02:30:20.327 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:20.325+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.326677+0000) 2020-02-06T02:30:20.336 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:30:20.340 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:30:20.344 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:20.342+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234695 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:20.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:20.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.680479+0000) 2020-02-06T02:30:20.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:20.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:20.956135+0000) 2020-02-06T02:30:23.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:23.536+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:23.538005+0000) 2020-02-06T02:30:25.355 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:25.347+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.348735+0000) 2020-02-06T02:30:25.356 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:25.348+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.349146+0000) 2020-02-06T02:30:25.356 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:25.348+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.349646+0000) 2020-02-06T02:30:25.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:25.349+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.349527+0000) 2020-02-06T02:30:25.357 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:25.348+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.349897+0000) 2020-02-06T02:30:25.357 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:25.349+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.350268+0000) 2020-02-06T02:30:25.358 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:25.349+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.350795+0000) 2020-02-06T02:30:25.360 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:25.351+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.351853+0000) 2020-02-06T02:30:25.361 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:25.351+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.352169+0000) 2020-02-06T02:30:25.361 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:25.351+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.352315+0000) 2020-02-06T02:30:25.361 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:25.351+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.351809+0000) 2020-02-06T02:30:25.363 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:25.353+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.354266+0000) 2020-02-06T02:30:25.365 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:25.350+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.351165+0000) 2020-02-06T02:30:25.367 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:25.357+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:25.358782+0000) 2020-02-06T02:30:25.383 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:25.374+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234732 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:26.179 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:26.176+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:26.177897+0000) 2020-02-06T02:30:26.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:26.202+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:26.203225+0000) 2020-02-06T02:30:26.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:26.565+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:26.565842+0000) 2020-02-06T02:30:27.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:27.177+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.178808+0000) 2020-02-06T02:30:27.455 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:27.453+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.454362+0000) 2020-02-06T02:30:27.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:27.453+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.454321+0000) 2020-02-06T02:30:27.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:27.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.454647+0000) 2020-02-06T02:30:27.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:27.682+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.683364+0000) 2020-02-06T02:30:27.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:27.705+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:27.706457+0000) 2020-02-06T02:30:29.823 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:29.821+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:29.822744+0000) 2020-02-06T02:30:30.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:30.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.068217+0000) 2020-02-06T02:30:30.380 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:30.378+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.379274+0000) 2020-02-06T02:30:30.381 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:30.379+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.380425+0000) 2020-02-06T02:30:30.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:30.380+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.381289+0000) 2020-02-06T02:30:30.382 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:30.380+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.381633+0000) 2020-02-06T02:30:30.383 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:30.381+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.382282+0000) 2020-02-06T02:30:30.383 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:30.382+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.382748+0000) 2020-02-06T02:30:30.384 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:30.382+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.383632+0000) 2020-02-06T02:30:30.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:30.382+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.383884+0000) 2020-02-06T02:30:30.385 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:30.383+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.384056+0000) 2020-02-06T02:30:30.385 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:30.383+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.384395+0000) 2020-02-06T02:30:30.386 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:30.384+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.384881+0000) 2020-02-06T02:30:30.387 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:30.385+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.386286+0000) 2020-02-06T02:30:30.387 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:30.386+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.386649+0000) 2020-02-06T02:30:30.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:30.393+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.394560+0000) 2020-02-06T02:30:30.408 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:30.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234778 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:30.681 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:30.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.680720+0000) 2020-02-06T02:30:30.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:30.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:30.956308+0000) 2020-02-06T02:30:33.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:33.537+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:33.538279+0000) 2020-02-06T02:30:35.413 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:35.410+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.411255+0000) 2020-02-06T02:30:35.414 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:35.412+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.413144+0000) 2020-02-06T02:30:35.414 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:35.413+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.413583+0000) 2020-02-06T02:30:35.415 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:35.413+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.413946+0000) 2020-02-06T02:30:35.415 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:35.414+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.414827+0000) 2020-02-06T02:30:35.416 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:35.414+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.415870+0000) 2020-02-06T02:30:35.417 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:35.415+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.416713+0000) 2020-02-06T02:30:35.418 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:35.417+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.417629+0000) 2020-02-06T02:30:35.420 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:35.417+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.418064+0000) 2020-02-06T02:30:35.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:35.417+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.418187+0000) 2020-02-06T02:30:35.421 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:35.417+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.418701+0000) 2020-02-06T02:30:35.421 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:35.417+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.418175+0000) 2020-02-06T02:30:35.421 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:35.418+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.419024+0000) 2020-02-06T02:30:35.423 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:35.421+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:35.422310+0000) 2020-02-06T02:30:35.438 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:35.437+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234816 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:36.187 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:36.184+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:36.185284+0000) 2020-02-06T02:30:36.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:36.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:36.203562+0000) 2020-02-06T02:30:36.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:36.565+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:36.566089+0000) 2020-02-06T02:30:37.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:37.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.179052+0000) 2020-02-06T02:30:37.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:37.454+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.454589+0000) 2020-02-06T02:30:37.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:37.454+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.454694+0000) 2020-02-06T02:30:37.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:37.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.454894+0000) 2020-02-06T02:30:37.684 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:37.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.683594+0000) 2020-02-06T02:30:37.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:37.706+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:37.706685+0000) 2020-02-06T02:30:39.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:39.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:39.822979+0000) 2020-02-06T02:30:40.069 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:40.067+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.068516+0000) 2020-02-06T02:30:40.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:40.442+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.443211+0000) 2020-02-06T02:30:40.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:40.442+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.443748+0000) 2020-02-06T02:30:40.445 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:40.443+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.444162+0000) 2020-02-06T02:30:40.445 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:40.444+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.444617+0000) 2020-02-06T02:30:40.447 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:40.444+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.445881+0000) 2020-02-06T02:30:40.447 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:40.445+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.446319+0000) 2020-02-06T02:30:40.447 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:40.445+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.446822+0000) 2020-02-06T02:30:40.448 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:40.446+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.446992+0000) 2020-02-06T02:30:40.448 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:40.446+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.447270+0000) 2020-02-06T02:30:40.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:40.447+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.447804+0000) 2020-02-06T02:30:40.449 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:40.447+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.448211+0000) 2020-02-06T02:30:40.450 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:40.448+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.449139+0000) 2020-02-06T02:30:40.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:40.452+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.453089+0000) 2020-02-06T02:30:40.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:40.452+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.453390+0000) 2020-02-06T02:30:40.468 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:40.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234861 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:40.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:40.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.680961+0000) 2020-02-06T02:30:40.957 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:40.955+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:40.956479+0000) 2020-02-06T02:30:43.539 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:43.538+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:43.538508+0000) 2020-02-06T02:30:45.474 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:45.472+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.473703+0000) 2020-02-06T02:30:45.475 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:45.473+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.474278+0000) 2020-02-06T02:30:45.476 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:45.474+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.475227+0000) 2020-02-06T02:30:45.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:45.475+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.475797+0000) 2020-02-06T02:30:45.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:45.475+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.476182+0000) 2020-02-06T02:30:45.477 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:45.475+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.476333+0000) 2020-02-06T02:30:45.477 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:45.475+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.476510+0000) 2020-02-06T02:30:45.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:45.475+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.476781+0000) 2020-02-06T02:30:45.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:45.477+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.477729+0000) 2020-02-06T02:30:45.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:45.477+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.477978+0000) 2020-02-06T02:30:45.479 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:45.477+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.478196+0000) 2020-02-06T02:30:45.479 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:45.477+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.478878+0000) 2020-02-06T02:30:45.480 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:45.478+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.479267+0000) 2020-02-06T02:30:45.483 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:45.481+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:45.482977+0000) 2020-02-06T02:30:45.499 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:45.497+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234898 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:46.198 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:46.196+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:46.197402+0000) 2020-02-06T02:30:46.204 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:46.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:46.203806+0000) 2020-02-06T02:30:46.567 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:46.566+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:46.566386+0000) 2020-02-06T02:30:47.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:47.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.179272+0000) 2020-02-06T02:30:47.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:47.454+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.454824+0000) 2020-02-06T02:30:47.456 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:47.454+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.454914+0000) 2020-02-06T02:30:47.456 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:47.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.455129+0000) 2020-02-06T02:30:47.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:47.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.683819+0000) 2020-02-06T02:30:47.707 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:47.706+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:47.706913+0000) 2020-02-06T02:30:49.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:49.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:49.823278+0000) 2020-02-06T02:30:50.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:50.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.068753+0000) 2020-02-06T02:30:50.503 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:50.501+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.502408+0000) 2020-02-06T02:30:50.506 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:50.504+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.505068+0000) 2020-02-06T02:30:50.506 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:50.504+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.505411+0000) 2020-02-06T02:30:50.507 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:50.504+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.505828+0000) 2020-02-06T02:30:50.508 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:50.506+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.507039+0000) 2020-02-06T02:30:50.508 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:50.507+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.507617+0000) 2020-02-06T02:30:50.509 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:50.507+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.507855+0000) 2020-02-06T02:30:50.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:50.507+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.508394+0000) 2020-02-06T02:30:50.509 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:50.508+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.508486+0000) 2020-02-06T02:30:50.510 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:50.508+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.508722+0000) 2020-02-06T02:30:50.510 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:50.508+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.509328+0000) 2020-02-06T02:30:50.510 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:50.509+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.509453+0000) 2020-02-06T02:30:50.511 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:50.508+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.509992+0000) 2020-02-06T02:30:50.513 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:50.511+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.512743+0000) 2020-02-06T02:30:50.530 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:50.528+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234943 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:50.540 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:30:50.545 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:30:50.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:50.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.681178+0000) 2020-02-06T02:30:50.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:50.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:50.956691+0000) 2020-02-06T02:30:53.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:53.537+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:53.538755+0000) 2020-02-06T02:30:55.535 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:30:55.533+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.534336+0000) 2020-02-06T02:30:55.536 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:55.534+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.535253+0000) 2020-02-06T02:30:55.536 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:55.534+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.535799+0000) 2020-02-06T02:30:55.537 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:55.535+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.536128+0000) 2020-02-06T02:30:55.537 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:55.535+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.536461+0000) 2020-02-06T02:30:55.538 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:55.535+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.537009+0000) 2020-02-06T02:30:55.538 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:55.537+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.537544+0000) 2020-02-06T02:30:55.539 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:30:55.537+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.537712+0000) 2020-02-06T02:30:55.539 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:55.537+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.538237+0000) 2020-02-06T02:30:55.540 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:55.538+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.538925+0000) 2020-02-06T02:30:55.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:55.538+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.539317+0000) 2020-02-06T02:30:55.540 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:30:55.538+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.539673+0000) 2020-02-06T02:30:55.541 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:55.540+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.540684+0000) 2020-02-06T02:30:55.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:55.542+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:55.544045+0000) 2020-02-06T02:30:55.560 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:30:55.559+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 234980 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:30:56.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:30:56.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:56.204014+0000) 2020-02-06T02:30:56.219 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:56.209+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:56.209382+0000) 2020-02-06T02:30:56.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:30:56.566+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:56.566646+0000) 2020-02-06T02:30:57.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:30:57.179+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.179522+0000) 2020-02-06T02:30:57.456 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:30:57.455+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.455068+0000) 2020-02-06T02:30:57.457 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:30:57.455+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.455188+0000) 2020-02-06T02:30:57.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:30:57.455+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.455384+0000) 2020-02-06T02:30:57.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:30:57.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.684052+0000) 2020-02-06T02:30:57.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:30:57.706+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:57.707146+0000) 2020-02-06T02:30:59.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:30:59.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:30:59.823514+0000) 2020-02-06T02:31:00.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:00.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.068992+0000) 2020-02-06T02:31:00.566 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:00.564+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.564876+0000) 2020-02-06T02:31:00.566 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:00.565+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.565669+0000) 2020-02-06T02:31:00.567 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:00.565+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.566007+0000) 2020-02-06T02:31:00.567 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:00.566+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.566513+0000) 2020-02-06T02:31:00.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:00.567+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.567292+0000) 2020-02-06T02:31:00.568 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:00.567+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.567535+0000) 2020-02-06T02:31:00.569 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:00.566+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.567639+0000) 2020-02-06T02:31:00.569 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:00.568+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.568414+0000) 2020-02-06T02:31:00.569 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:00.566+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.568320+0000) 2020-02-06T02:31:00.570 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:00.567+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.568885+0000) 2020-02-06T02:31:00.570 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:00.567+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.568803+0000) 2020-02-06T02:31:00.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:00.568+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.569775+0000) 2020-02-06T02:31:00.572 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:00.569+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.571142+0000) 2020-02-06T02:31:00.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:00.573+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.574859+0000) 2020-02-06T02:31:00.590 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:00.588+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235026 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:00.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:00.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.681458+0000) 2020-02-06T02:31:00.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:00.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:00.957021+0000) 2020-02-06T02:31:03.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:03.538+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:03.538992+0000) 2020-02-06T02:31:05.597 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:05.594+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.595828+0000) 2020-02-06T02:31:05.598 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:05.595+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.596807+0000) 2020-02-06T02:31:05.598 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:05.596+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.597483+0000) 2020-02-06T02:31:05.599 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:05.596+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.597921+0000) 2020-02-06T02:31:05.599 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:05.596+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.598007+0000) 2020-02-06T02:31:05.600 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:05.597+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.598461+0000) 2020-02-06T02:31:05.601 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:05.597+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.599075+0000) 2020-02-06T02:31:05.601 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:05.597+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.598734+0000) 2020-02-06T02:31:05.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:05.598+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.599779+0000) 2020-02-06T02:31:05.602 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:05.598+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.599566+0000) 2020-02-06T02:31:05.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:05.598+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.600018+0000) 2020-02-06T02:31:05.602 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:05.599+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.600350+0000) 2020-02-06T02:31:05.603 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:05.599+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.600997+0000) 2020-02-06T02:31:05.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:05.607+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:05.608435+0000) 2020-02-06T02:31:05.621 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:05.619+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235063 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:06.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:06.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:06.204241+0000) 2020-02-06T02:31:06.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:06.216+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:06.217508+0000) 2020-02-06T02:31:06.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:06.565+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:06.566912+0000) 2020-02-06T02:31:07.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:07.178+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.179751+0000) 2020-02-06T02:31:07.457 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:07.453+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.455308+0000) 2020-02-06T02:31:07.457 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:07.454+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.455486+0000) 2020-02-06T02:31:07.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:07.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.455671+0000) 2020-02-06T02:31:07.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:07.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.684310+0000) 2020-02-06T02:31:07.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:07.706+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:07.707369+0000) 2020-02-06T02:31:09.824 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:09.822+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:09.823748+0000) 2020-02-06T02:31:10.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:10.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.069218+0000) 2020-02-06T02:31:10.628 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:10.626+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.627554+0000) 2020-02-06T02:31:10.629 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:10.626+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.628108+0000) 2020-02-06T02:31:10.629 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:10.627+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.628272+0000) 2020-02-06T02:31:10.630 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:10.627+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.628771+0000) 2020-02-06T02:31:10.631 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:10.628+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.629880+0000) 2020-02-06T02:31:10.631 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:10.628+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.629674+0000) 2020-02-06T02:31:10.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:10.628+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.630105+0000) 2020-02-06T02:31:10.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:10.629+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.630362+0000) 2020-02-06T02:31:10.633 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:10.629+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.630617+0000) 2020-02-06T02:31:10.633 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:10.630+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.631620+0000) 2020-02-06T02:31:10.633 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:10.630+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.631828+0000) 2020-02-06T02:31:10.634 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:10.629+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.631098+0000) 2020-02-06T02:31:10.634 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:10.630+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.632222+0000) 2020-02-06T02:31:10.637 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:10.635+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.636371+0000) 2020-02-06T02:31:10.652 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:10.649+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235109 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:10.682 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:10.680+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.681709+0000) 2020-02-06T02:31:10.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:10.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:10.957275+0000) 2020-02-06T02:31:13.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:13.538+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:13.539268+0000) 2020-02-06T02:31:15.656 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:15.654+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.655297+0000) 2020-02-06T02:31:15.657 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:15.655+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.656165+0000) 2020-02-06T02:31:15.657 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:15.655+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.656675+0000) 2020-02-06T02:31:15.659 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:15.656+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.657569+0000) 2020-02-06T02:31:15.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:15.657+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.658232+0000) 2020-02-06T02:31:15.659 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:15.657+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.658764+0000) 2020-02-06T02:31:15.661 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:15.658+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.660006+0000) 2020-02-06T02:31:15.661 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:15.658+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.659650+0000) 2020-02-06T02:31:15.662 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:15.659+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.660419+0000) 2020-02-06T02:31:15.662 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:15.659+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.660764+0000) 2020-02-06T02:31:15.662 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:15.660+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.661128+0000) 2020-02-06T02:31:15.662 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:15.660+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.661655+0000) 2020-02-06T02:31:15.663 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:15.660+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.661957+0000) 2020-02-06T02:31:15.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:15.666+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:15.667134+0000) 2020-02-06T02:31:15.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:15.680+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235146 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:16.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:16.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:16.204488+0000) 2020-02-06T02:31:16.225 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:16.223+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:16.224621+0000) 2020-02-06T02:31:16.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:16.566+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:16.567145+0000) 2020-02-06T02:31:17.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:17.179+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.179991+0000) 2020-02-06T02:31:17.457 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:17.454+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.455643+0000) 2020-02-06T02:31:17.457 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:17.454+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.455744+0000) 2020-02-06T02:31:17.457 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:17.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.455929+0000) 2020-02-06T02:31:17.685 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:17.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.684545+0000) 2020-02-06T02:31:17.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:17.706+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:17.707597+0000) 2020-02-06T02:31:19.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:19.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:19.823999+0000) 2020-02-06T02:31:20.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:20.068+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.069451+0000) 2020-02-06T02:31:20.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:20.681+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.681934+0000) 2020-02-06T02:31:20.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:20.686+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.687560+0000) 2020-02-06T02:31:20.689 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:20.686+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.687699+0000) 2020-02-06T02:31:20.691 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:20.688+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.689856+0000) 2020-02-06T02:31:20.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:20.688+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.689555+0000) 2020-02-06T02:31:20.691 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:20.688+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.690219+0000) 2020-02-06T02:31:20.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:20.689+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.690535+0000) 2020-02-06T02:31:20.692 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:20.689+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.690973+0000) 2020-02-06T02:31:20.693 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:20.690+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.691529+0000) 2020-02-06T02:31:20.693 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:20.689+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.691168+0000) 2020-02-06T02:31:20.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:20.691+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.692514+0000) 2020-02-06T02:31:20.694 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:20.690+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.692175+0000) 2020-02-06T02:31:20.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:20.691+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.692979+0000) 2020-02-06T02:31:20.694 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:20.692+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.693309+0000) 2020-02-06T02:31:20.698 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:20.696+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.697873+0000) 2020-02-06T02:31:20.711 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:20.709+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235192 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:20.747 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:31:20.752 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:31:20.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:20.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:20.957476+0000) 2020-02-06T02:31:23.540 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:23.538+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:23.539516+0000) 2020-02-06T02:31:25.719 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:25.715+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.716776+0000) 2020-02-06T02:31:25.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:25.715+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.716849+0000) 2020-02-06T02:31:25.720 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:25.717+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.718490+0000) 2020-02-06T02:31:25.720 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:25.717+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.719101+0000) 2020-02-06T02:31:25.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:25.718+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.719504+0000) 2020-02-06T02:31:25.722 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:25.718+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.719791+0000) 2020-02-06T02:31:25.722 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:25.719+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.720366+0000) 2020-02-06T02:31:25.722 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:25.718+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.719791+0000) 2020-02-06T02:31:25.724 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:25.719+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.720793+0000) 2020-02-06T02:31:25.724 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:25.720+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.721495+0000) 2020-02-06T02:31:25.725 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:25.720+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.722056+0000) 2020-02-06T02:31:25.725 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:25.719+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.721136+0000) 2020-02-06T02:31:25.725 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:25.720+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.721670+0000) 2020-02-06T02:31:25.726 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:25.724+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:25.725407+0000) 2020-02-06T02:31:25.741 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:25.738+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235228 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:26.205 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:26.203+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:26.204707+0000) 2020-02-06T02:31:26.234 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:26.231+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:26.232658+0000) 2020-02-06T02:31:26.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:26.566+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:26.567382+0000) 2020-02-06T02:31:27.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:27.179+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.180247+0000) 2020-02-06T02:31:27.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:27.454+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.455866+0000) 2020-02-06T02:31:27.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:27.454+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.455984+0000) 2020-02-06T02:31:27.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:27.454+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.456174+0000) 2020-02-06T02:31:27.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:27.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.684766+0000) 2020-02-06T02:31:27.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:27.707+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:27.707819+0000) 2020-02-06T02:31:29.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:29.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:29.824307+0000) 2020-02-06T02:31:30.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:30.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.069707+0000) 2020-02-06T02:31:30.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:30.681+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.682193+0000) 2020-02-06T02:31:30.746 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:30.744+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.745489+0000) 2020-02-06T02:31:30.747 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:30.745+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.746498+0000) 2020-02-06T02:31:30.747 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:30.745+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.746807+0000) 2020-02-06T02:31:30.748 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:30.746+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.747465+0000) 2020-02-06T02:31:30.748 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:30.746+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.747687+0000) 2020-02-06T02:31:30.749 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:30.746+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.748404+0000) 2020-02-06T02:31:30.750 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:30.747+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.748603+0000) 2020-02-06T02:31:30.750 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:30.747+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.748948+0000) 2020-02-06T02:31:30.751 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:30.748+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.749800+0000) 2020-02-06T02:31:30.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:30.749+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.750419+0000) 2020-02-06T02:31:30.751 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:30.749+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.750829+0000) 2020-02-06T02:31:30.753 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:30.750+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.751600+0000) 2020-02-06T02:31:30.753 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:30.750+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.751873+0000) 2020-02-06T02:31:30.758 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:30.756+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.757198+0000) 2020-02-06T02:31:30.772 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:30.769+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235275 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:30.958 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:30.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:30.957699+0000) 2020-02-06T02:31:33.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:33.539+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:33.539751+0000) 2020-02-06T02:31:35.778 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:35.776+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.777154+0000) 2020-02-06T02:31:35.778 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:35.776+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.777698+0000) 2020-02-06T02:31:35.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:35.777+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.778670+0000) 2020-02-06T02:31:35.780 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:35.777+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.779044+0000) 2020-02-06T02:31:35.780 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:35.778+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.779563+0000) 2020-02-06T02:31:35.781 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:35.778+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.779915+0000) 2020-02-06T02:31:35.781 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:35.778+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.780114+0000) 2020-02-06T02:31:35.781 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:35.779+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.780538+0000) 2020-02-06T02:31:35.782 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:35.779+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.780939+0000) 2020-02-06T02:31:35.782 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:35.780+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.781562+0000) 2020-02-06T02:31:35.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:35.780+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.781911+0000) 2020-02-06T02:31:35.783 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:35.780+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.782364+0000) 2020-02-06T02:31:35.783 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:35.781+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.782525+0000) 2020-02-06T02:31:35.789 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:35.787+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:35.788355+0000) 2020-02-06T02:31:35.802 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:35.800+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:36.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:36.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:36.204931+0000) 2020-02-06T02:31:36.241 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:36.239+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:36.239986+0000) 2020-02-06T02:31:36.568 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:36.567+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:36.567646+0000) 2020-02-06T02:31:37.181 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:37.180+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.180502+0000) 2020-02-06T02:31:37.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:37.455+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.456127+0000) 2020-02-06T02:31:37.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:37.455+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.456248+0000) 2020-02-06T02:31:37.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:37.455+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.456470+0000) 2020-02-06T02:31:37.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:37.683+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.685007+0000) 2020-02-06T02:31:37.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:37.707+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:37.708043+0000) 2020-02-06T02:31:39.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:39.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:39.824569+0000) 2020-02-06T02:31:40.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:40.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.069944+0000) 2020-02-06T02:31:40.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:40.681+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.682482+0000) 2020-02-06T02:31:40.807 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:40.805+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.806345+0000) 2020-02-06T02:31:40.808 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:40.806+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.807178+0000) 2020-02-06T02:31:40.809 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:40.807+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.808265+0000) 2020-02-06T02:31:40.809 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:40.807+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.808593+0000) 2020-02-06T02:31:40.810 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:40.808+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.809743+0000) 2020-02-06T02:31:40.812 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:40.809+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.810855+0000) 2020-02-06T02:31:40.812 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:40.809+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.810649+0000) 2020-02-06T02:31:40.812 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:40.810+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.811542+0000) 2020-02-06T02:31:40.813 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:40.809+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.811233+0000) 2020-02-06T02:31:40.814 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:40.812+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.813217+0000) 2020-02-06T02:31:40.814 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:40.811+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.813275+0000) 2020-02-06T02:31:40.814 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:40.812+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.813491+0000) 2020-02-06T02:31:40.815 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:40.812+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.813758+0000) 2020-02-06T02:31:40.815 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:40.813+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.814950+0000) 2020-02-06T02:31:40.832 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:40.829+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235358 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:40.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:40.956+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:40.957876+0000) 2020-02-06T02:31:43.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:43.539+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:43.540062+0000) 2020-02-06T02:31:45.837 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:45.834+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.835710+0000) 2020-02-06T02:31:45.837 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:45.835+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.836291+0000) 2020-02-06T02:31:45.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:45.837+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.838607+0000) 2020-02-06T02:31:45.840 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:45.838+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.839205+0000) 2020-02-06T02:31:45.840 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:45.838+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.839906+0000) 2020-02-06T02:31:45.841 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:45.839+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.840767+0000) 2020-02-06T02:31:45.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:45.840+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.841646+0000) 2020-02-06T02:31:45.843 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:45.840+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.842060+0000) 2020-02-06T02:31:45.843 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:45.841+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.842342+0000) 2020-02-06T02:31:45.844 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:45.841+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.842804+0000) 2020-02-06T02:31:45.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:45.842+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.843560+0000) 2020-02-06T02:31:45.845 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:45.841+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.843060+0000) 2020-02-06T02:31:45.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:45.841+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.843255+0000) 2020-02-06T02:31:45.848 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:45.846+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:45.847875+0000) 2020-02-06T02:31:45.861 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:45.859+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235393 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:46.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:46.204+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:46.205225+0000) 2020-02-06T02:31:46.249 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:46.247+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:46.248128+0000) 2020-02-06T02:31:46.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:46.567+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:46.567922+0000) 2020-02-06T02:31:47.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:47.180+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.180722+0000) 2020-02-06T02:31:47.457 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:47.455+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.456384+0000) 2020-02-06T02:31:47.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:47.455+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.456518+0000) 2020-02-06T02:31:47.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:47.455+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.456684+0000) 2020-02-06T02:31:47.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:47.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.685279+0000) 2020-02-06T02:31:47.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:47.708+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:47.708310+0000) 2020-02-06T02:31:49.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:49.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:49.824807+0000) 2020-02-06T02:31:50.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:50.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.070168+0000) 2020-02-06T02:31:50.683 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:50.682+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.682736+0000) 2020-02-06T02:31:50.868 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:50.865+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.866885+0000) 2020-02-06T02:31:50.868 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:50.866+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.867343+0000) 2020-02-06T02:31:50.869 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:50.867+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.867960+0000) 2020-02-06T02:31:50.869 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:50.867+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.868148+0000) 2020-02-06T02:31:50.869 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:50.867+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.868497+0000) 2020-02-06T02:31:50.870 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:50.868+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.869115+0000) 2020-02-06T02:31:50.870 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:50.868+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.869155+0000) 2020-02-06T02:31:50.870 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:50.868+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.869534+0000) 2020-02-06T02:31:50.871 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:50.869+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.869995+0000) 2020-02-06T02:31:50.872 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:50.870+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.870827+0000) 2020-02-06T02:31:50.872 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:50.870+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.871474+0000) 2020-02-06T02:31:50.873 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:50.871+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.871618+0000) 2020-02-06T02:31:50.873 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:50.870+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.871261+0000) 2020-02-06T02:31:50.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:50.874+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.875754+0000) 2020-02-06T02:31:50.892 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:50.891+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235442 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:50.926 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:31:50.931 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:31:50.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:50.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:50.958104+0000) 2020-02-06T02:31:53.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:53.540+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:53.540328+0000) 2020-02-06T02:31:55.897 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:55.895+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.895951+0000) 2020-02-06T02:31:55.898 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:55.897+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.897369+0000) 2020-02-06T02:31:55.898 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:31:55.897+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.897751+0000) 2020-02-06T02:31:55.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:55.899+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.899474+0000) 2020-02-06T02:31:55.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:55.900+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.900152+0000) 2020-02-06T02:31:55.902 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:55.900+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.900587+0000) 2020-02-06T02:31:55.902 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:55.900+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.900438+0000) 2020-02-06T02:31:55.902 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:31:55.900+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.900886+0000) 2020-02-06T02:31:55.903 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:55.901+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.901399+0000) 2020-02-06T02:31:55.903 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:55.901+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.901440+0000) 2020-02-06T02:31:55.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:55.902+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.902505+0000) 2020-02-06T02:31:55.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:55.901+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.902066+0000) 2020-02-06T02:31:55.904 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:31:55.902+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.903022+0000) 2020-02-06T02:31:55.906 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:55.904+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:55.905046+0000) 2020-02-06T02:31:55.923 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:31:55.921+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235476 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:31:56.206 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:31:56.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:56.205501+0000) 2020-02-06T02:31:56.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:56.256+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:56.256067+0000) 2020-02-06T02:31:56.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:31:56.567+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:56.568147+0000) 2020-02-06T02:31:57.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:31:57.180+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.180996+0000) 2020-02-06T02:31:57.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:31:57.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.456652+0000) 2020-02-06T02:31:57.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:31:57.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.456741+0000) 2020-02-06T02:31:57.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:31:57.456+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.456915+0000) 2020-02-06T02:31:57.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:31:57.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.685514+0000) 2020-02-06T02:31:57.709 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:31:57.708+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:57.708542+0000) 2020-02-06T02:31:59.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:31:59.823+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:31:59.825049+0000) 2020-02-06T02:32:00.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:00.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.070405+0000) 2020-02-06T02:32:00.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:00.682+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.682995+0000) 2020-02-06T02:32:00.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:00.926+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.927602+0000) 2020-02-06T02:32:00.929 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:00.927+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.928469+0000) 2020-02-06T02:32:00.929 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:00.928+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.928592+0000) 2020-02-06T02:32:00.930 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:00.927+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.929036+0000) 2020-02-06T02:32:00.930 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:00.928+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.929419+0000) 2020-02-06T02:32:00.931 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:00.928+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.929964+0000) 2020-02-06T02:32:00.932 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:00.929+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.930882+0000) 2020-02-06T02:32:00.933 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:00.931+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.932387+0000) 2020-02-06T02:32:00.933 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:00.931+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.932198+0000) 2020-02-06T02:32:00.934 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:00.931+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.932617+0000) 2020-02-06T02:32:00.934 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:00.932+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.933242+0000) 2020-02-06T02:32:00.935 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:00.933+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.933860+0000) 2020-02-06T02:32:00.935 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:00.933+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.934083+0000) 2020-02-06T02:32:00.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:00.935+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.936754+0000) 2020-02-06T02:32:00.954 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:00.952+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235523 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:00.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:00.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:00.958295+0000) 2020-02-06T02:32:03.541 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:03.539+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:03.540558+0000) 2020-02-06T02:32:05.958 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:05.956+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.957781+0000) 2020-02-06T02:32:05.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:05.957+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.958513+0000) 2020-02-06T02:32:05.960 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:05.958+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.959687+0000) 2020-02-06T02:32:05.961 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:05.959+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.960475+0000) 2020-02-06T02:32:05.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:05.959+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.960472+0000) 2020-02-06T02:32:05.962 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:05.959+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.960611+0000) 2020-02-06T02:32:05.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:05.960+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.961195+0000) 2020-02-06T02:32:05.962 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:05.960+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.961767+0000) 2020-02-06T02:32:05.963 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:05.961+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.962238+0000) 2020-02-06T02:32:05.963 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:05.962+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.962652+0000) 2020-02-06T02:32:05.965 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:05.963+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.963894+0000) 2020-02-06T02:32:05.965 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:05.963+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.964049+0000) 2020-02-06T02:32:05.966 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:05.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.965577+0000) 2020-02-06T02:32:05.969 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:05.966+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:05.968027+0000) 2020-02-06T02:32:05.985 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:05.983+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235558 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:06.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:06.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:06.205799+0000) 2020-02-06T02:32:06.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:06.263+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:06.264278+0000) 2020-02-06T02:32:06.569 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:06.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:06.568365+0000) 2020-02-06T02:32:07.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:07.180+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.181248+0000) 2020-02-06T02:32:07.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:07.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.456880+0000) 2020-02-06T02:32:07.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:07.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.456993+0000) 2020-02-06T02:32:07.458 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:07.456+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.457128+0000) 2020-02-06T02:32:07.686 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:07.684+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.685743+0000) 2020-02-06T02:32:07.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:07.707+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:07.708823+0000) 2020-02-06T02:32:09.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:09.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:09.825311+0000) 2020-02-06T02:32:10.071 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:10.069+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.070638+0000) 2020-02-06T02:32:10.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:10.683+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.683289+0000) 2020-02-06T02:32:10.959 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:10.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.958518+0000) 2020-02-06T02:32:10.990 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:10.987+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.989030+0000) 2020-02-06T02:32:10.991 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:10.989+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.990252+0000) 2020-02-06T02:32:10.992 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:10.990+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.991204+0000) 2020-02-06T02:32:10.992 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:10.990+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.991798+0000) 2020-02-06T02:32:10.993 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:10.991+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.992773+0000) 2020-02-06T02:32:10.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:10.992+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.993228+0000) 2020-02-06T02:32:10.994 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:10.992+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.993901+0000) 2020-02-06T02:32:10.995 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:10.994+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.994488+0000) 2020-02-06T02:32:10.995 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:10.994+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.994577+0000) 2020-02-06T02:32:10.996 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:10.993+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.994993+0000) 2020-02-06T02:32:10.996 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:10.994+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.995391+0000) 2020-02-06T02:32:10.997 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:10.995+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.995939+0000) 2020-02-06T02:32:10.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:10.995+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.996186+0000) 2020-02-06T02:32:10.998 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:10.996+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:10.997436+0000) 2020-02-06T02:32:11.017 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:11.015+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235605 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:13.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:13.540+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:13.540822+0000) 2020-02-06T02:32:16.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:16.020+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.021424+0000) 2020-02-06T02:32:16.023 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:16.021+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.022327+0000) 2020-02-06T02:32:16.024 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:16.022+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.023138+0000) 2020-02-06T02:32:16.024 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:16.022+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.023134+0000) 2020-02-06T02:32:16.025 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:16.022+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.024013+0000) 2020-02-06T02:32:16.025 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:16.024+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.024516+0000) 2020-02-06T02:32:16.026 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:16.023+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.024512+0000) 2020-02-06T02:32:16.026 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:16.024+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.025203+0000) 2020-02-06T02:32:16.026 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:16.023+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.024976+0000) 2020-02-06T02:32:16.027 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:16.025+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.025845+0000) 2020-02-06T02:32:16.028 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:16.026+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.027130+0000) 2020-02-06T02:32:16.028 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:16.026+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.027696+0000) 2020-02-06T02:32:16.029 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:16.027+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.028128+0000) 2020-02-06T02:32:16.033 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:16.031+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.032398+0000) 2020-02-06T02:32:16.047 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:16.045+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235639 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:16.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:16.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.206029+0000) 2020-02-06T02:32:16.273 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:16.271+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.272415+0000) 2020-02-06T02:32:16.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:16.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:16.568613+0000) 2020-02-06T02:32:17.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:17.180+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.181441+0000) 2020-02-06T02:32:17.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:17.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.457149+0000) 2020-02-06T02:32:17.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:17.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.457253+0000) 2020-02-06T02:32:17.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:17.457+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.457412+0000) 2020-02-06T02:32:17.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:17.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.685967+0000) 2020-02-06T02:32:17.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:17.708+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:17.709053+0000) 2020-02-06T02:32:19.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:19.824+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:19.825544+0000) 2020-02-06T02:32:20.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:20.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:20.070867+0000) 2020-02-06T02:32:20.684 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:20.683+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:20.683522+0000) 2020-02-06T02:32:20.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:20.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:20.958750+0000) 2020-02-06T02:32:21.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:21.050+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.051792+0000) 2020-02-06T02:32:21.053 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:21.051+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.052736+0000) 2020-02-06T02:32:21.054 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:21.052+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.053442+0000) 2020-02-06T02:32:21.055 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:21.053+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.054231+0000) 2020-02-06T02:32:21.055 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:21.053+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.054731+0000) 2020-02-06T02:32:21.056 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:21.054+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.055217+0000) 2020-02-06T02:32:21.056 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:21.055+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.055608+0000) 2020-02-06T02:32:21.057 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:21.055+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.056179+0000) 2020-02-06T02:32:21.057 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:21.055+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.056422+0000) 2020-02-06T02:32:21.057 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:21.055+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.056728+0000) 2020-02-06T02:32:21.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:21.056+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.057062+0000) 2020-02-06T02:32:21.058 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:21.056+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.057429+0000) 2020-02-06T02:32:21.059 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:21.057+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.057973+0000) 2020-02-06T02:32:21.063 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:21.061+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:21.062448+0000) 2020-02-06T02:32:21.078 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:21.076+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235685 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:21.106 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:32:21.111 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:32:23.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:23.540+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:23.541138+0000) 2020-02-06T02:32:26.084 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:26.082+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.083281+0000) 2020-02-06T02:32:26.084 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:26.082+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.083838+0000) 2020-02-06T02:32:26.085 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:26.083+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.084242+0000) 2020-02-06T02:32:26.086 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:26.084+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.085085+0000) 2020-02-06T02:32:26.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:26.083+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.085061+0000) 2020-02-06T02:32:26.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:26.084+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.085392+0000) 2020-02-06T02:32:26.087 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:26.085+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.085866+0000) 2020-02-06T02:32:26.087 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:26.085+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.086434+0000) 2020-02-06T02:32:26.088 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:26.085+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.086285+0000) 2020-02-06T02:32:26.088 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:26.085+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.086946+0000) 2020-02-06T02:32:26.089 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:26.087+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.088159+0000) 2020-02-06T02:32:26.089 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:26.088+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.088692+0000) 2020-02-06T02:32:26.090 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:26.088+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.088983+0000) 2020-02-06T02:32:26.095 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:26.092+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.093901+0000) 2020-02-06T02:32:26.108 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:26.106+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235720 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:26.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:26.206+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.206246+0000) 2020-02-06T02:32:26.281 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:26.279+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.280597+0000) 2020-02-06T02:32:26.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:26.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:26.568822+0000) 2020-02-06T02:32:27.182 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:27.181+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.181679+0000) 2020-02-06T02:32:27.458 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:27.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.457406+0000) 2020-02-06T02:32:27.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:27.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.457519+0000) 2020-02-06T02:32:27.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:27.456+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.457697+0000) 2020-02-06T02:32:27.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:27.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.686200+0000) 2020-02-06T02:32:27.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:27.708+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:27.709323+0000) 2020-02-06T02:32:29.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:29.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:29.825777+0000) 2020-02-06T02:32:30.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:30.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:30.071154+0000) 2020-02-06T02:32:30.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:30.682+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:30.683758+0000) 2020-02-06T02:32:30.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:30.957+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:30.958993+0000) 2020-02-06T02:32:31.114 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:31.112+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.113149+0000) 2020-02-06T02:32:31.115 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:31.112+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.114175+0000) 2020-02-06T02:32:31.116 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:31.114+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.115188+0000) 2020-02-06T02:32:31.116 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:31.114+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.115717+0000) 2020-02-06T02:32:31.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:31.115+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.116304+0000) 2020-02-06T02:32:31.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:31.115+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.116804+0000) 2020-02-06T02:32:31.118 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:31.115+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.116994+0000) 2020-02-06T02:32:31.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:31.116+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.117481+0000) 2020-02-06T02:32:31.118 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:31.116+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.117664+0000) 2020-02-06T02:32:31.119 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:31.117+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.118582+0000) 2020-02-06T02:32:31.120 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:31.117+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.118878+0000) 2020-02-06T02:32:31.120 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:31.117+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.119105+0000) 2020-02-06T02:32:31.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:31.118+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.119863+0000) 2020-02-06T02:32:31.124 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:31.122+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:31.123854+0000) 2020-02-06T02:32:31.138 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:31.135+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235767 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:33.542 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:33.540+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:33.541402+0000) 2020-02-06T02:32:36.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:36.141+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.143270+0000) 2020-02-06T02:32:36.145 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:36.142+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.143945+0000) 2020-02-06T02:32:36.146 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:36.143+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.144917+0000) 2020-02-06T02:32:36.146 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:36.144+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.145495+0000) 2020-02-06T02:32:36.147 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:36.144+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.145880+0000) 2020-02-06T02:32:36.148 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:36.145+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.146373+0000) 2020-02-06T02:32:36.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:36.145+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.146781+0000) 2020-02-06T02:32:36.149 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:36.145+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.146758+0000) 2020-02-06T02:32:36.149 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:36.146+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.147727+0000) 2020-02-06T02:32:36.149 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:36.146+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.147291+0000) 2020-02-06T02:32:36.149 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:36.146+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.147965+0000) 2020-02-06T02:32:36.150 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:36.146+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.148375+0000) 2020-02-06T02:32:36.150 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:36.147+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.148641+0000) 2020-02-06T02:32:36.153 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:36.151+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.152730+0000) 2020-02-06T02:32:36.168 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:36.166+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235801 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:36.207 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:36.205+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.206508+0000) 2020-02-06T02:32:36.287 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:36.286+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.286693+0000) 2020-02-06T02:32:36.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:36.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:36.569077+0000) 2020-02-06T02:32:37.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:37.181+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.181931+0000) 2020-02-06T02:32:37.459 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:37.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.457697+0000) 2020-02-06T02:32:37.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:37.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.457751+0000) 2020-02-06T02:32:37.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:37.456+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.458129+0000) 2020-02-06T02:32:37.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:37.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.686431+0000) 2020-02-06T02:32:37.710 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:37.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:37.709576+0000) 2020-02-06T02:32:39.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:39.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:39.826019+0000) 2020-02-06T02:32:40.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:40.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:40.071345+0000) 2020-02-06T02:32:40.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:40.682+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:40.684002+0000) 2020-02-06T02:32:40.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:40.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:40.959253+0000) 2020-02-06T02:32:41.175 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:41.172+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.174336+0000) 2020-02-06T02:32:41.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:41.173+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.174915+0000) 2020-02-06T02:32:41.176 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:41.174+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.175523+0000) 2020-02-06T02:32:41.177 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:41.174+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.176061+0000) 2020-02-06T02:32:41.177 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:41.175+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.176316+0000) 2020-02-06T02:32:41.178 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:41.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.176533+0000) 2020-02-06T02:32:41.178 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:41.175+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.177314+0000) 2020-02-06T02:32:41.178 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:41.176+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.177063+0000) 2020-02-06T02:32:41.179 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:41.176+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.177908+0000) 2020-02-06T02:32:41.180 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:41.177+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.178403+0000) 2020-02-06T02:32:41.180 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:41.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.178486+0000) 2020-02-06T02:32:41.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:41.177+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.178901+0000) 2020-02-06T02:32:41.180 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:41.178+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.179564+0000) 2020-02-06T02:32:41.185 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:41.183+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:41.184444+0000) 2020-02-06T02:32:41.200 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:41.197+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235847 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:43.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:43.541+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:43.541659+0000) 2020-02-06T02:32:46.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:46.203+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.204351+0000) 2020-02-06T02:32:46.206 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:46.203+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.205172+0000) 2020-02-06T02:32:46.206 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:46.204+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.205427+0000) 2020-02-06T02:32:46.207 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:46.205+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.206284+0000) 2020-02-06T02:32:46.208 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:46.206+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.207761+0000) 2020-02-06T02:32:46.209 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:46.205+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.207387+0000) 2020-02-06T02:32:46.209 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:46.207+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.208412+0000) 2020-02-06T02:32:46.210 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:46.207+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.209035+0000) 2020-02-06T02:32:46.210 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:46.208+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.210066+0000) 2020-02-06T02:32:46.211 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:46.209+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.210631+0000) 2020-02-06T02:32:46.212 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:46.209+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.210940+0000) 2020-02-06T02:32:46.212 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:46.210+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.211237+0000) 2020-02-06T02:32:46.212 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:46.210+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.211717+0000) 2020-02-06T02:32:46.215 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:46.213+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.214125+0000) 2020-02-06T02:32:46.230 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:46.228+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235882 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:46.296 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:46.294+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.294976+0000) 2020-02-06T02:32:46.570 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:46.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:46.569319+0000) 2020-02-06T02:32:47.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:47.182+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.182204+0000) 2020-02-06T02:32:47.459 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:47.456+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.457939+0000) 2020-02-06T02:32:47.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:47.456+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.458045+0000) 2020-02-06T02:32:47.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:47.456+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.458372+0000) 2020-02-06T02:32:47.687 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:47.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.686638+0000) 2020-02-06T02:32:47.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:47.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:47.709807+0000) 2020-02-06T02:32:49.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:49.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:49.826244+0000) 2020-02-06T02:32:50.072 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:50.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:50.071572+0000) 2020-02-06T02:32:50.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:50.683+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:50.684258+0000) 2020-02-06T02:32:50.960 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:50.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:50.959514+0000) 2020-02-06T02:32:51.236 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:51.234+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.235556+0000) 2020-02-06T02:32:51.237 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:51.236+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.236303+0000) 2020-02-06T02:32:51.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:51.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.236934+0000) 2020-02-06T02:32:51.239 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:51.236+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.237691+0000) 2020-02-06T02:32:51.239 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:51.236+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.238009+0000) 2020-02-06T02:32:51.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:51.237+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.238808+0000) 2020-02-06T02:32:51.240 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:51.237+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.239178+0000) 2020-02-06T02:32:51.241 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:51.239+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.239654+0000) 2020-02-06T02:32:51.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:51.240+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.240038+0000) 2020-02-06T02:32:51.242 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:51.240+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.240337+0000) 2020-02-06T02:32:51.242 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:51.240+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.240521+0000) 2020-02-06T02:32:51.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:51.239+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.241138+0000) 2020-02-06T02:32:51.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:51.240+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.240981+0000) 2020-02-06T02:32:51.244 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:51.243+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:51.243645+0000) 2020-02-06T02:32:51.263 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:51.261+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235929 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:51.269 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:32:51.273 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:32:52.958 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:52.956+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:52.957552+0000) 2020-02-06T02:32:53.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:53.541+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:53.541911+0000) 2020-02-06T02:32:56.270 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:56.267+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.268827+0000) 2020-02-06T02:32:56.270 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:56.268+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.269615+0000) 2020-02-06T02:32:56.271 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:56.269+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.271003+0000) 2020-02-06T02:32:56.272 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:32:56.269+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.271338+0000) 2020-02-06T02:32:56.273 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:32:56.270+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.271625+0000) 2020-02-06T02:32:56.273 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:56.270+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.272370+0000) 2020-02-06T02:32:56.274 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:56.272+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.273397+0000) 2020-02-06T02:32:56.275 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:32:56.272+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.273437+0000) 2020-02-06T02:32:56.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:56.271+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.272894+0000) 2020-02-06T02:32:56.275 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:32:56.271+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.273337+0000) 2020-02-06T02:32:56.276 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:56.273+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.274537+0000) 2020-02-06T02:32:56.276 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:56.272+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.274230+0000) 2020-02-06T02:32:56.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:56.273+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.274824+0000) 2020-02-06T02:32:56.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:56.278+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.279929+0000) 2020-02-06T02:32:56.294 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:32:56.292+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 235964 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:32:56.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:56.302+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.303068+0000) 2020-02-06T02:32:56.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:32:56.568+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:56.569548+0000) 2020-02-06T02:32:57.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:32:57.181+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.182431+0000) 2020-02-06T02:32:57.459 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:32:57.457+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.458237+0000) 2020-02-06T02:32:57.459 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:32:57.457+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.458333+0000) 2020-02-06T02:32:57.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:32:57.457+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.458662+0000) 2020-02-06T02:32:57.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:32:57.685+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.686876+0000) 2020-02-06T02:32:57.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:32:57.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:57.710037+0000) 2020-02-06T02:32:59.827 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:32:59.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:32:59.826479+0000) 2020-02-06T02:33:00.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:00.070+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:00.071802+0000) 2020-02-06T02:33:00.685 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:00.683+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:00.684544+0000) 2020-02-06T02:33:00.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:00.958+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:00.959741+0000) 2020-02-06T02:33:01.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:01.298+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.299466+0000) 2020-02-06T02:33:01.302 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:01.299+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.300905+0000) 2020-02-06T02:33:01.303 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:01.300+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.301665+0000) 2020-02-06T02:33:01.303 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:01.300+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.302020+0000) 2020-02-06T02:33:01.303 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:01.301+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.302288+0000) 2020-02-06T02:33:01.304 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:01.301+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.302942+0000) 2020-02-06T02:33:01.304 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:01.302+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.303506+0000) 2020-02-06T02:33:01.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:01.302+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.303503+0000) 2020-02-06T02:33:01.305 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:01.302+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.304281+0000) 2020-02-06T02:33:01.305 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:01.303+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.304439+0000) 2020-02-06T02:33:01.306 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:01.303+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.305214+0000) 2020-02-06T02:33:01.306 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:01.303+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.305323+0000) 2020-02-06T02:33:01.306 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:01.304+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.305677+0000) 2020-02-06T02:33:01.311 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:01.309+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:01.310243+0000) 2020-02-06T02:33:01.326 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:01.324+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236009 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:02.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:02.957+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:02.957795+0000) 2020-02-06T02:33:03.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:03.541+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:03.542155+0000) 2020-02-06T02:33:06.310 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:06.308+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.309044+0000) 2020-02-06T02:33:06.333 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:06.331+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.332554+0000) 2020-02-06T02:33:06.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:06.332+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.333154+0000) 2020-02-06T02:33:06.334 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:06.332+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.333577+0000) 2020-02-06T02:33:06.334 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:06.332+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.333705+0000) 2020-02-06T02:33:06.335 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:06.332+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.333767+0000) 2020-02-06T02:33:06.335 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:06.332+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.334051+0000) 2020-02-06T02:33:06.336 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:06.333+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.334589+0000) 2020-02-06T02:33:06.336 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:06.333+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.334841+0000) 2020-02-06T02:33:06.337 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:06.334+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.335130+0000) 2020-02-06T02:33:06.337 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:06.334+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.335320+0000) 2020-02-06T02:33:06.337 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:06.334+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.335496+0000) 2020-02-06T02:33:06.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:06.334+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.336234+0000) 2020-02-06T02:33:06.338 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:06.335+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.336587+0000) 2020-02-06T02:33:06.342 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:06.339+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.341040+0000) 2020-02-06T02:33:06.357 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:06.354+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236045 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:06.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:06.569+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:06.569826+0000) 2020-02-06T02:33:07.183 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:07.181+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.182665+0000) 2020-02-06T02:33:07.459 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:07.457+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.458489+0000) 2020-02-06T02:33:07.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:07.457+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.458601+0000) 2020-02-06T02:33:07.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:07.457+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.458913+0000) 2020-02-06T02:33:07.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:07.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.687130+0000) 2020-02-06T02:33:07.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:07.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:07.710329+0000) 2020-02-06T02:33:09.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:09.825+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:09.826713+0000) 2020-02-06T02:33:10.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:10.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:10.072060+0000) 2020-02-06T02:33:10.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:10.684+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:10.684798+0000) 2020-02-06T02:33:10.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:10.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:10.960009+0000) 2020-02-06T02:33:11.363 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:11.361+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.362135+0000) 2020-02-06T02:33:11.363 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:11.361+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.362628+0000) 2020-02-06T02:33:11.364 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:11.362+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.363418+0000) 2020-02-06T02:33:11.365 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:11.362+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.363649+0000) 2020-02-06T02:33:11.365 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:11.362+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.364381+0000) 2020-02-06T02:33:11.366 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:11.364+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.365301+0000) 2020-02-06T02:33:11.366 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:11.364+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.365521+0000) 2020-02-06T02:33:11.367 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:11.363+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.365350+0000) 2020-02-06T02:33:11.367 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:11.365+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.366070+0000) 2020-02-06T02:33:11.368 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:11.365+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.366904+0000) 2020-02-06T02:33:11.368 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:11.365+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.366770+0000) 2020-02-06T02:33:11.368 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:11.366+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.367281+0000) 2020-02-06T02:33:11.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:11.365+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.367253+0000) 2020-02-06T02:33:11.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:11.372+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:11.373691+0000) 2020-02-06T02:33:11.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:11.386+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236090 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:12.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:12.957+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:12.958043+0000) 2020-02-06T02:33:13.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:13.541+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:13.542408+0000) 2020-02-06T02:33:16.318 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:16.316+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.317164+0000) 2020-02-06T02:33:16.395 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:16.393+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.394504+0000) 2020-02-06T02:33:16.396 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:16.393+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.394926+0000) 2020-02-06T02:33:16.396 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:16.394+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.395290+0000) 2020-02-06T02:33:16.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:16.394+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.395864+0000) 2020-02-06T02:33:16.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:16.394+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.395619+0000) 2020-02-06T02:33:16.399 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:16.395+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.396977+0000) 2020-02-06T02:33:16.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:16.395+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.397346+0000) 2020-02-06T02:33:16.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:16.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.397597+0000) 2020-02-06T02:33:16.399 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:16.396+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.398087+0000) 2020-02-06T02:33:16.400 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:16.395+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.396407+0000) 2020-02-06T02:33:16.400 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:16.395+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.396229+0000) 2020-02-06T02:33:16.400 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:16.395+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.396902+0000) 2020-02-06T02:33:16.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:16.397+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.399241+0000) 2020-02-06T02:33:16.405 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:16.403+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.404125+0000) 2020-02-06T02:33:16.419 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:16.416+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236125 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:16.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:16.569+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:16.570063+0000) 2020-02-06T02:33:17.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:17.182+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.182909+0000) 2020-02-06T02:33:17.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:17.458+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.458722+0000) 2020-02-06T02:33:17.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:17.458+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.458829+0000) 2020-02-06T02:33:17.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:17.458+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.459120+0000) 2020-02-06T02:33:17.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:17.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.687352+0000) 2020-02-06T02:33:17.711 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:17.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:17.710557+0000) 2020-02-06T02:33:19.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:19.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:19.826967+0000) 2020-02-06T02:33:20.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:20.071+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:20.072418+0000) 2020-02-06T02:33:20.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:20.684+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:20.685035+0000) 2020-02-06T02:33:20.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:20.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:20.960272+0000) 2020-02-06T02:33:21.425 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:21.423+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.424799+0000) 2020-02-06T02:33:21.426 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:21.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.425567+0000) 2020-02-06T02:33:21.427 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:21.425+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.425747+0000) 2020-02-06T02:33:21.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:21.425+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.425683+0000) 2020-02-06T02:33:21.427 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:21.424+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.426031+0000) 2020-02-06T02:33:21.427 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:21.425+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.426391+0000) 2020-02-06T02:33:21.428 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:21.425+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.426665+0000) 2020-02-06T02:33:21.428 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:21.426+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.427456+0000) 2020-02-06T02:33:21.429 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:21.427+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.427619+0000) 2020-02-06T02:33:21.429 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:21.427+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.428005+0000) 2020-02-06T02:33:21.430 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:21.428+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.429382+0000) 2020-02-06T02:33:21.430 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:21.429+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.429696+0000) 2020-02-06T02:33:21.431 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:21.430+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.430457+0000) 2020-02-06T02:33:21.437 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:21.435+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:21.436420+0000) 2020-02-06T02:33:21.451 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:21.450+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236169 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:21.493 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:33:21.499 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:33:22.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:22.957+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:22.958281+0000) 2020-02-06T02:33:23.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:23.542+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:23.542638+0000) 2020-02-06T02:33:26.326 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:26.324+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.325666+0000) 2020-02-06T02:33:26.457 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:26.454+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.455938+0000) 2020-02-06T02:33:26.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:26.456+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.457241+0000) 2020-02-06T02:33:26.459 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:26.457+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.458268+0000) 2020-02-06T02:33:26.459 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:26.457+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.458582+0000) 2020-02-06T02:33:26.460 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:26.458+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.459137+0000) 2020-02-06T02:33:26.461 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:26.459+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.459536+0000) 2020-02-06T02:33:26.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:26.459+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.459642+0000) 2020-02-06T02:33:26.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:26.458+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.459381+0000) 2020-02-06T02:33:26.462 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:26.459+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.460244+0000) 2020-02-06T02:33:26.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:26.460+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.461136+0000) 2020-02-06T02:33:26.463 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:26.460+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.460790+0000) 2020-02-06T02:33:26.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:26.460+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.460468+0000) 2020-02-06T02:33:26.463 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:26.461+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.462003+0000) 2020-02-06T02:33:26.468 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:26.465+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.467000+0000) 2020-02-06T02:33:26.494 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:26.479+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236205 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:26.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:26.569+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:26.570304+0000) 2020-02-06T02:33:27.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:27.182+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.183125+0000) 2020-02-06T02:33:27.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:27.458+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.458973+0000) 2020-02-06T02:33:27.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:27.458+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.459113+0000) 2020-02-06T02:33:27.460 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:27.458+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.459319+0000) 2020-02-06T02:33:27.688 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:27.686+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.687592+0000) 2020-02-06T02:33:27.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:27.709+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:27.710807+0000) 2020-02-06T02:33:29.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:29.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:29.827200+0000) 2020-02-06T02:33:30.073 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:30.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:30.072645+0000) 2020-02-06T02:33:30.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:30.684+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:30.685268+0000) 2020-02-06T02:33:30.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:30.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:30.960498+0000) 2020-02-06T02:33:31.487 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:31.485+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.486241+0000) 2020-02-06T02:33:31.488 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:31.485+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.486782+0000) 2020-02-06T02:33:31.488 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:31.487+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.487508+0000) 2020-02-06T02:33:31.489 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:31.487+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.488424+0000) 2020-02-06T02:33:31.490 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:31.488+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.489269+0000) 2020-02-06T02:33:31.490 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:31.488+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.489606+0000) 2020-02-06T02:33:31.491 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:31.489+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.490192+0000) 2020-02-06T02:33:31.491 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:31.489+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.489245+0000) 2020-02-06T02:33:31.493 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:31.491+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.491424+0000) 2020-02-06T02:33:31.493 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:31.491+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.491954+0000) 2020-02-06T02:33:31.494 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:31.491+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.492059+0000) 2020-02-06T02:33:31.494 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:31.492+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.492448+0000) 2020-02-06T02:33:31.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:31.490+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.491181+0000) 2020-02-06T02:33:31.498 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:31.494+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:31.496061+0000) 2020-02-06T02:33:31.512 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:31.511+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236251 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:32.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:32.958+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:32.958519+0000) 2020-02-06T02:33:33.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:33.542+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:33.542866+0000) 2020-02-06T02:33:36.335 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:36.332+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.334013+0000) 2020-02-06T02:33:36.518 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:36.515+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.516709+0000) 2020-02-06T02:33:36.518 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:36.517+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.517340+0000) 2020-02-06T02:33:36.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:36.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.518448+0000) 2020-02-06T02:33:36.520 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:36.518+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.519561+0000) 2020-02-06T02:33:36.520 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:36.518+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.520046+0000) 2020-02-06T02:33:36.521 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:36.519+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.520621+0000) 2020-02-06T02:33:36.522 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:36.520+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.521164+0000) 2020-02-06T02:33:36.522 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:36.521+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.521519+0000) 2020-02-06T02:33:36.523 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:36.520+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.521844+0000) 2020-02-06T02:33:36.523 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:36.522+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.522623+0000) 2020-02-06T02:33:36.523 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:36.522+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.522833+0000) 2020-02-06T02:33:36.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:36.522+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.523186+0000) 2020-02-06T02:33:36.524 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:36.522+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.523409+0000) 2020-02-06T02:33:36.526 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:36.524+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.525275+0000) 2020-02-06T02:33:36.543 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:36.542+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236286 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:36.571 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:36.570+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:36.570584+0000) 2020-02-06T02:33:37.184 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:37.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.183336+0000) 2020-02-06T02:33:37.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:37.458+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.459183+0000) 2020-02-06T02:33:37.460 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:37.458+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.459331+0000) 2020-02-06T02:33:37.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:37.459+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.459522+0000) 2020-02-06T02:33:37.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:37.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.687852+0000) 2020-02-06T02:33:37.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:37.710+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:37.711123+0000) 2020-02-06T02:33:39.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:39.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:39.827432+0000) 2020-02-06T02:33:40.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:40.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:40.072872+0000) 2020-02-06T02:33:40.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:40.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:40.685525+0000) 2020-02-06T02:33:40.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:40.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:40.960729+0000) 2020-02-06T02:33:41.549 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:41.546+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.547962+0000) 2020-02-06T02:33:41.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:41.548+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.549474+0000) 2020-02-06T02:33:41.551 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:41.548+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.550028+0000) 2020-02-06T02:33:41.551 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:41.549+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.550541+0000) 2020-02-06T02:33:41.552 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:41.550+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.551172+0000) 2020-02-06T02:33:41.553 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:41.551+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.551868+0000) 2020-02-06T02:33:41.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:41.551+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.552057+0000) 2020-02-06T02:33:41.554 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:41.551+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.552317+0000) 2020-02-06T02:33:41.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:41.551+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.552796+0000) 2020-02-06T02:33:41.554 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:41.552+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.553319+0000) 2020-02-06T02:33:41.555 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:41.553+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.553779+0000) 2020-02-06T02:33:41.556 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:41.554+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.555374+0000) 2020-02-06T02:33:41.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:41.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.559188+0000) 2020-02-06T02:33:41.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:41.558+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:41.559514+0000) 2020-02-06T02:33:41.576 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:41.574+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:42.959 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:42.958+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:42.958760+0000) 2020-02-06T02:33:43.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:43.542+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:43.543144+0000) 2020-02-06T02:33:46.343 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:46.342+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.342275+0000) 2020-02-06T02:33:46.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:46.570+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.570745+0000) 2020-02-06T02:33:46.580 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:46.579+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.579950+0000) 2020-02-06T02:33:46.583 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:46.582+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.582301+0000) 2020-02-06T02:33:46.583 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:46.582+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.582769+0000) 2020-02-06T02:33:46.584 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:46.583+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.583402+0000) 2020-02-06T02:33:46.585 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:46.583+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.583906+0000) 2020-02-06T02:33:46.585 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:46.584+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.584315+0000) 2020-02-06T02:33:46.586 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:46.584+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.584293+0000) 2020-02-06T02:33:46.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:46.584+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.585049+0000) 2020-02-06T02:33:46.587 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:46.584+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.584985+0000) 2020-02-06T02:33:46.588 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:46.584+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.585137+0000) 2020-02-06T02:33:46.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:46.585+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.586156+0000) 2020-02-06T02:33:46.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:46.585+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.586026+0000) 2020-02-06T02:33:46.589 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:46.586+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.587080+0000) 2020-02-06T02:33:46.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:46.591+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:46.591275+0000) 2020-02-06T02:33:46.607 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:46.605+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236367 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:47.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:47.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.183551+0000) 2020-02-06T02:33:47.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:47.459+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.459485+0000) 2020-02-06T02:33:47.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:47.459+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.459549+0000) 2020-02-06T02:33:47.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:47.459+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.459713+0000) 2020-02-06T02:33:47.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:47.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.688040+0000) 2020-02-06T02:33:47.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:47.711+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:47.711352+0000) 2020-02-06T02:33:49.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:49.826+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:49.827676+0000) 2020-02-06T02:33:50.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:50.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:50.073134+0000) 2020-02-06T02:33:50.686 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:50.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:50.685764+0000) 2020-02-06T02:33:50.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:50.959+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:50.960959+0000) 2020-02-06T02:33:51.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:51.611+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.612431+0000) 2020-02-06T02:33:51.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:51.612+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.613201+0000) 2020-02-06T02:33:51.614 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:51.612+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.613365+0000) 2020-02-06T02:33:51.614 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:51.612+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.613935+0000) 2020-02-06T02:33:51.615 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:51.613+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.614577+0000) 2020-02-06T02:33:51.615 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:51.614+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.614557+0000) 2020-02-06T02:33:51.616 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:51.613+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.614998+0000) 2020-02-06T02:33:51.616 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:51.615+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.615730+0000) 2020-02-06T02:33:51.617 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:51.615+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.616303+0000) 2020-02-06T02:33:51.618 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:51.616+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.616730+0000) 2020-02-06T02:33:51.618 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:51.616+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.617069+0000) 2020-02-06T02:33:51.618 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:51.617+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.617843+0000) 2020-02-06T02:33:51.619 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:51.617+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.618342+0000) 2020-02-06T02:33:51.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:51.620+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:51.621823+0000) 2020-02-06T02:33:51.631 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:33:51.635 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:33:51.638 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:51.637+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236413 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:52.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:52.958+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:52.959032+0000) 2020-02-06T02:33:53.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:53.542+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:53.543406+0000) 2020-02-06T02:33:56.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:56.350+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.350953+0000) 2020-02-06T02:33:56.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:56.569+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.570992+0000) 2020-02-06T02:33:56.645 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:33:56.642+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.643864+0000) 2020-02-06T02:33:56.645 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:56.643+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.644205+0000) 2020-02-06T02:33:56.646 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:33:56.643+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.644894+0000) 2020-02-06T02:33:56.646 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:56.644+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.645398+0000) 2020-02-06T02:33:56.646 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:33:56.644+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.645977+0000) 2020-02-06T02:33:56.647 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:56.645+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.646560+0000) 2020-02-06T02:33:56.648 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:33:56.646+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.647186+0000) 2020-02-06T02:33:56.648 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:33:56.646+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.647812+0000) 2020-02-06T02:33:56.649 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:56.647+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.648351+0000) 2020-02-06T02:33:56.650 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:56.647+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.648560+0000) 2020-02-06T02:33:56.650 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:56.647+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.648446+0000) 2020-02-06T02:33:56.650 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:56.647+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.649047+0000) 2020-02-06T02:33:56.650 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:56.647+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.649160+0000) 2020-02-06T02:33:56.653 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:56.651+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:56.652945+0000) 2020-02-06T02:33:56.670 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:33:56.667+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236448 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:33:57.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:33:57.182+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.183759+0000) 2020-02-06T02:33:57.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:33:57.458+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.459695+0000) 2020-02-06T02:33:57.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:33:57.458+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.459776+0000) 2020-02-06T02:33:57.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:33:57.458+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.459943+0000) 2020-02-06T02:33:57.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:33:57.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.688338+0000) 2020-02-06T02:33:57.712 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:33:57.710+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:57.711585+0000) 2020-02-06T02:33:59.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:33:59.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:33:59.827906+0000) 2020-02-06T02:34:00.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:00.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:00.073361+0000) 2020-02-06T02:34:00.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:00.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:00.686018+0000) 2020-02-06T02:34:00.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:00.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:00.961262+0000) 2020-02-06T02:34:01.675 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:01.673+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.674370+0000) 2020-02-06T02:34:01.676 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:01.674+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.675372+0000) 2020-02-06T02:34:01.678 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:01.674+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.676046+0000) 2020-02-06T02:34:01.678 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:01.674+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.676038+0000) 2020-02-06T02:34:01.678 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:01.675+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.676549+0000) 2020-02-06T02:34:01.679 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:01.675+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.676592+0000) 2020-02-06T02:34:01.680 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:01.675+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.677414+0000) 2020-02-06T02:34:01.680 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:01.676+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.678056+0000) 2020-02-06T02:34:01.680 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:01.676+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.678348+0000) 2020-02-06T02:34:01.681 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:01.677+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.679084+0000) 2020-02-06T02:34:01.681 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:01.676+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.677751+0000) 2020-02-06T02:34:01.681 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:01.677+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.678785+0000) 2020-02-06T02:34:01.682 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:01.677+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.679245+0000) 2020-02-06T02:34:01.684 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:01.682+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:01.683231+0000) 2020-02-06T02:34:01.699 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:01.697+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236493 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:02.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:02.957+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:02.959327+0000) 2020-02-06T02:34:03.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:03.542+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:03.543646+0000) 2020-02-06T02:34:06.361 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:06.358+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.359957+0000) 2020-02-06T02:34:06.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:06.569+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.571238+0000) 2020-02-06T02:34:06.705 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:06.703+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.704741+0000) 2020-02-06T02:34:06.706 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:06.704+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.705507+0000) 2020-02-06T02:34:06.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:06.704+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.705841+0000) 2020-02-06T02:34:06.707 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:06.704+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.706362+0000) 2020-02-06T02:34:06.708 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:06.706+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.707541+0000) 2020-02-06T02:34:06.709 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:06.706+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.708290+0000) 2020-02-06T02:34:06.709 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:06.707+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.708787+0000) 2020-02-06T02:34:06.710 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:06.708+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.709181+0000) 2020-02-06T02:34:06.710 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:06.708+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.709422+0000) 2020-02-06T02:34:06.710 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:06.708+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.709924+0000) 2020-02-06T02:34:06.711 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:06.709+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.710314+0000) 2020-02-06T02:34:06.711 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:06.709+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.710388+0000) 2020-02-06T02:34:06.713 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:06.708+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.710300+0000) 2020-02-06T02:34:06.717 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:06.715+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:06.716185+0000) 2020-02-06T02:34:06.730 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:06.727+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236530 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:07.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:07.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.183976+0000) 2020-02-06T02:34:07.461 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:07.458+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.459923+0000) 2020-02-06T02:34:07.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:07.458+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.459992+0000) 2020-02-06T02:34:07.461 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:07.458+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.460156+0000) 2020-02-06T02:34:07.689 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:07.687+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.688561+0000) 2020-02-06T02:34:07.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:07.710+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:07.711832+0000) 2020-02-06T02:34:09.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:09.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:09.828170+0000) 2020-02-06T02:34:10.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:10.072+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:10.073588+0000) 2020-02-06T02:34:10.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:10.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:10.686267+0000) 2020-02-06T02:34:10.962 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:10.960+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:10.961495+0000) 2020-02-06T02:34:11.736 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:11.732+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.733967+0000) 2020-02-06T02:34:11.736 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:11.733+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.734656+0000) 2020-02-06T02:34:11.737 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:11.734+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.736059+0000) 2020-02-06T02:34:11.737 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:11.735+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.736786+0000) 2020-02-06T02:34:11.738 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:11.736+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.737502+0000) 2020-02-06T02:34:11.739 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:11.737+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.738437+0000) 2020-02-06T02:34:11.739 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:11.736+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.738295+0000) 2020-02-06T02:34:11.740 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:11.738+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.739143+0000) 2020-02-06T02:34:11.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:11.737+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.739163+0000) 2020-02-06T02:34:11.740 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:11.738+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.739738+0000) 2020-02-06T02:34:11.743 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:11.739+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.740454+0000) 2020-02-06T02:34:11.754 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:11.739+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.740600+0000) 2020-02-06T02:34:11.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:11.742+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.743616+0000) 2020-02-06T02:34:11.755 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:11.743+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:11.744716+0000) 2020-02-06T02:34:11.762 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:11.760+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236575 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:12.960 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:12.958+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:12.959576+0000) 2020-02-06T02:34:13.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:13.543+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:13.543871+0000) 2020-02-06T02:34:16.370 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:16.368+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.368830+0000) 2020-02-06T02:34:16.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:16.570+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.571477+0000) 2020-02-06T02:34:16.769 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:16.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.767778+0000) 2020-02-06T02:34:16.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:16.766+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.767712+0000) 2020-02-06T02:34:16.770 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:16.767+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.768380+0000) 2020-02-06T02:34:16.770 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:16.768+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.769423+0000) 2020-02-06T02:34:16.770 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:16.768+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.769621+0000) 2020-02-06T02:34:16.771 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:16.769+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.770260+0000) 2020-02-06T02:34:16.773 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:16.769+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.770483+0000) 2020-02-06T02:34:16.773 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:16.770+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.771167+0000) 2020-02-06T02:34:16.774 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:16.770+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.772225+0000) 2020-02-06T02:34:16.774 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:16.771+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.773162+0000) 2020-02-06T02:34:16.774 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:16.772+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.773490+0000) 2020-02-06T02:34:16.775 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:16.771+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.772790+0000) 2020-02-06T02:34:16.775 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:16.771+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.773405+0000) 2020-02-06T02:34:16.777 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:16.775+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:16.776955+0000) 2020-02-06T02:34:16.791 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:16.789+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236611 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:17.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:17.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.184129+0000) 2020-02-06T02:34:17.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:17.459+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.460195+0000) 2020-02-06T02:34:17.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:17.459+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.460162+0000) 2020-02-06T02:34:17.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:17.459+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.460361+0000) 2020-02-06T02:34:17.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:17.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.688769+0000) 2020-02-06T02:34:17.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:17.710+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:17.712036+0000) 2020-02-06T02:34:19.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:19.827+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:19.828410+0000) 2020-02-06T02:34:20.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:20.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:20.073827+0000) 2020-02-06T02:34:20.687 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:20.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:20.686515+0000) 2020-02-06T02:34:20.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:20.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:20.961802+0000) 2020-02-06T02:34:21.754 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:34:21.760 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:34:21.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:21.794+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.795475+0000) 2020-02-06T02:34:21.797 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:21.795+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.796217+0000) 2020-02-06T02:34:21.799 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:21.795+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.797011+0000) 2020-02-06T02:34:21.799 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:21.796+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.797310+0000) 2020-02-06T02:34:21.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:21.796+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.797668+0000) 2020-02-06T02:34:21.799 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:21.797+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.798163+0000) 2020-02-06T02:34:21.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:21.796+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.797938+0000) 2020-02-06T02:34:21.801 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:21.796+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.797517+0000) 2020-02-06T02:34:21.801 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:21.796+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.798354+0000) 2020-02-06T02:34:21.801 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:21.797+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.799376+0000) 2020-02-06T02:34:21.802 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:21.798+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.799950+0000) 2020-02-06T02:34:21.803 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:21.798+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.799844+0000) 2020-02-06T02:34:21.804 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:21.799+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.800512+0000) 2020-02-06T02:34:21.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:21.800+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:21.801930+0000) 2020-02-06T02:34:21.822 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:21.820+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236657 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:22.961 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:22.959+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:22.959875+0000) 2020-02-06T02:34:23.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:23.543+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:23.544122+0000) 2020-02-06T02:34:26.378 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:26.376+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.377147+0000) 2020-02-06T02:34:26.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:26.571+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.571769+0000) 2020-02-06T02:34:26.828 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:26.826+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.827375+0000) 2020-02-06T02:34:26.829 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:26.826+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.827889+0000) 2020-02-06T02:34:26.829 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:26.827+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.828916+0000) 2020-02-06T02:34:26.830 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:26.828+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.829667+0000) 2020-02-06T02:34:26.831 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:26.829+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.830047+0000) 2020-02-06T02:34:26.831 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:26.829+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.830421+0000) 2020-02-06T02:34:26.831 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:26.829+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.830593+0000) 2020-02-06T02:34:26.832 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:26.829+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.830949+0000) 2020-02-06T02:34:26.832 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:26.830+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.831166+0000) 2020-02-06T02:34:26.832 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:26.830+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.831426+0000) 2020-02-06T02:34:26.833 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:26.830+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.831743+0000) 2020-02-06T02:34:26.833 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:26.830+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.831893+0000) 2020-02-06T02:34:26.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:26.831+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.832720+0000) 2020-02-06T02:34:26.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:26.832+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:26.833781+0000) 2020-02-06T02:34:26.855 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:26.852+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236694 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:27.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:27.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.184310+0000) 2020-02-06T02:34:27.461 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:27.459+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.460420+0000) 2020-02-06T02:34:27.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:27.459+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.460420+0000) 2020-02-06T02:34:27.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:27.459+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.460650+0000) 2020-02-06T02:34:27.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:27.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.688976+0000) 2020-02-06T02:34:27.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:27.711+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:27.712264+0000) 2020-02-06T02:34:29.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:29.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:29.828657+0000) 2020-02-06T02:34:30.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:30.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:30.074129+0000) 2020-02-06T02:34:30.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:30.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:30.686807+0000) 2020-02-06T02:34:30.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:30.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:30.962070+0000) 2020-02-06T02:34:31.862 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:31.860+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.861271+0000) 2020-02-06T02:34:31.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:31.859+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.861191+0000) 2020-02-06T02:34:31.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:31.859+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.860707+0000) 2020-02-06T02:34:31.863 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:31.860+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.861702+0000) 2020-02-06T02:34:31.864 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:31.860+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.862007+0000) 2020-02-06T02:34:31.864 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:31.861+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.863089+0000) 2020-02-06T02:34:31.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:31.862+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.863327+0000) 2020-02-06T02:34:31.865 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:31.862+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.863935+0000) 2020-02-06T02:34:31.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:31.863+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.864510+0000) 2020-02-06T02:34:31.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:31.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.865020+0000) 2020-02-06T02:34:31.866 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:31.864+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.865412+0000) 2020-02-06T02:34:31.870 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:31.864+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.866002+0000) 2020-02-06T02:34:31.870 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:31.864+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.866253+0000) 2020-02-06T02:34:31.871 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:31.869+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:31.870580+0000) 2020-02-06T02:34:31.886 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:31.883+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236739 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:32.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:32.959+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:32.960127+0000) 2020-02-06T02:34:33.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:33.544+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:33.544342+0000) 2020-02-06T02:34:36.386 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:36.385+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.385569+0000) 2020-02-06T02:34:36.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:36.570+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.572089+0000) 2020-02-06T02:34:36.892 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:36.890+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.891308+0000) 2020-02-06T02:34:36.893 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:36.891+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.892519+0000) 2020-02-06T02:34:36.894 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:36.890+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.892328+0000) 2020-02-06T02:34:36.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:36.892+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.893954+0000) 2020-02-06T02:34:36.903 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:36.894+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.895274+0000) 2020-02-06T02:34:36.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:36.895+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.896511+0000) 2020-02-06T02:34:36.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:36.895+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.896917+0000) 2020-02-06T02:34:36.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:36.898+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.899273+0000) 2020-02-06T02:34:36.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:36.895+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.896921+0000) 2020-02-06T02:34:36.904 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:36.892+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.893649+0000) 2020-02-06T02:34:36.904 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:36.893+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.894689+0000) 2020-02-06T02:34:36.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:36.894+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.895796+0000) 2020-02-06T02:34:36.905 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:36.895+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.896781+0000) 2020-02-06T02:34:36.905 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:36.895+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:36.897052+0000) 2020-02-06T02:34:36.916 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:36.913+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236776 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:37.185 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:37.184+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.184521+0000) 2020-02-06T02:34:37.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:37.460+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.460720+0000) 2020-02-06T02:34:37.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:37.460+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.460719+0000) 2020-02-06T02:34:37.462 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:37.460+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.460875+0000) 2020-02-06T02:34:37.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:37.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.689185+0000) 2020-02-06T02:34:37.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:37.712+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:37.712471+0000) 2020-02-06T02:34:39.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:39.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:39.828912+0000) 2020-02-06T02:34:40.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:40.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:40.074391+0000) 2020-02-06T02:34:40.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:40.685+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:40.687087+0000) 2020-02-06T02:34:40.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:40.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:40.962367+0000) 2020-02-06T02:34:41.922 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:41.920+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.920831+0000) 2020-02-06T02:34:41.923 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:41.921+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.922660+0000) 2020-02-06T02:34:41.924 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:41.922+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.922920+0000) 2020-02-06T02:34:41.924 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:41.923+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.923864+0000) 2020-02-06T02:34:41.925 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:41.924+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.924269+0000) 2020-02-06T02:34:41.925 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:41.923+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.924920+0000) 2020-02-06T02:34:41.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:41.925+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.925337+0000) 2020-02-06T02:34:41.926 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:41.923+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.924732+0000) 2020-02-06T02:34:41.927 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:41.925+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.925918+0000) 2020-02-06T02:34:41.927 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:41.924+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.926247+0000) 2020-02-06T02:34:41.927 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:41.924+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.926243+0000) 2020-02-06T02:34:41.928 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:41.925+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.926888+0000) 2020-02-06T02:34:41.928 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:41.926+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.926904+0000) 2020-02-06T02:34:41.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:41.934+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:41.934122+0000) 2020-02-06T02:34:41.947 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:41.945+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236821 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:42.961 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:42.959+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:42.960410+0000) 2020-02-06T02:34:43.545 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:43.544+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:43.544581+0000) 2020-02-06T02:34:46.395 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:46.393+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.394522+0000) 2020-02-06T02:34:46.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:46.571+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.572341+0000) 2020-02-06T02:34:46.953 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:46.951+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.952223+0000) 2020-02-06T02:34:46.954 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:46.952+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.953344+0000) 2020-02-06T02:34:46.955 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:46.952+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.954030+0000) 2020-02-06T02:34:46.956 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:46.954+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.955125+0000) 2020-02-06T02:34:46.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:46.955+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.955552+0000) 2020-02-06T02:34:46.956 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:46.954+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.955628+0000) 2020-02-06T02:34:46.957 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:46.956+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.956755+0000) 2020-02-06T02:34:46.958 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:46.956+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.956757+0000) 2020-02-06T02:34:46.958 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:46.957+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.957622+0000) 2020-02-06T02:34:46.959 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:46.956+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.957757+0000) 2020-02-06T02:34:46.959 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:46.958+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.958556+0000) 2020-02-06T02:34:46.959 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:46.958+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.958469+0000) 2020-02-06T02:34:46.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:46.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.962396+0000) 2020-02-06T02:34:46.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:46.961+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:46.962684+0000) 2020-02-06T02:34:46.978 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:46.976+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236859 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:47.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:47.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.184729+0000) 2020-02-06T02:34:47.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:47.460+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.460942+0000) 2020-02-06T02:34:47.462 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:47.460+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.460942+0000) 2020-02-06T02:34:47.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:47.460+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.461108+0000) 2020-02-06T02:34:47.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:47.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.689384+0000) 2020-02-06T02:34:47.713 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:47.711+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:47.712674+0000) 2020-02-06T02:34:49.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:49.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:49.829139+0000) 2020-02-06T02:34:50.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:50.073+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:50.074624+0000) 2020-02-06T02:34:50.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:50.686+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:50.687375+0000) 2020-02-06T02:34:50.963 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:50.961+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:50.962592+0000) 2020-02-06T02:34:51.915 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:34:51.922 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:34:51.982 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:51.980+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.981337+0000) 2020-02-06T02:34:51.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:51.980+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.981652+0000) 2020-02-06T02:34:51.983 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:51.981+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.982379+0000) 2020-02-06T02:34:51.984 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:51.983+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.983476+0000) 2020-02-06T02:34:51.985 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:51.982+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.983977+0000) 2020-02-06T02:34:51.985 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:51.983+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.984297+0000) 2020-02-06T02:34:51.986 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:51.983+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.984976+0000) 2020-02-06T02:34:51.986 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:51.984+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.985193+0000) 2020-02-06T02:34:51.986 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:51.984+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.985312+0000) 2020-02-06T02:34:51.987 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:51.985+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.986215+0000) 2020-02-06T02:34:51.988 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:51.986+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.986466+0000) 2020-02-06T02:34:51.988 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:51.985+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.986201+0000) 2020-02-06T02:34:51.992 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:51.989+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.990966+0000) 2020-02-06T02:34:51.993 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:51.991+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:51.992775+0000) 2020-02-06T02:34:52.008 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:52.006+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236902 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:52.961 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:52.960+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:52.960675+0000) 2020-02-06T02:34:53.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:53.543+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:53.544806+0000) 2020-02-06T02:34:56.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:56.402+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:56.403156+0000) 2020-02-06T02:34:56.573 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:56.572+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:56.572626+0000) 2020-02-06T02:34:57.014 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:57.012+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.013178+0000) 2020-02-06T02:34:57.016 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:34:57.015+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.015607+0000) 2020-02-06T02:34:57.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:57.015+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.016137+0000) 2020-02-06T02:34:57.017 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:57.015+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.016381+0000) 2020-02-06T02:34:57.018 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:57.015+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.016620+0000) 2020-02-06T02:34:57.018 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:34:57.016+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.016845+0000) 2020-02-06T02:34:57.019 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:57.016+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.017266+0000) 2020-02-06T02:34:57.019 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:34:57.016+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.017126+0000) 2020-02-06T02:34:57.019 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:34:57.016+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.017293+0000) 2020-02-06T02:34:57.020 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:34:57.017+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.018208+0000) 2020-02-06T02:34:57.020 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:57.017+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.018350+0000) 2020-02-06T02:34:57.020 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:57.016+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.018004+0000) 2020-02-06T02:34:57.020 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:57.016+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.017970+0000) 2020-02-06T02:34:57.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:57.020+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.021464+0000) 2020-02-06T02:34:57.039 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:34:57.037+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236940 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:34:57.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:34:57.183+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.184965+0000) 2020-02-06T02:34:57.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:34:57.460+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.461180+0000) 2020-02-06T02:34:57.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:34:57.460+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.461168+0000) 2020-02-06T02:34:57.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:34:57.460+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.461347+0000) 2020-02-06T02:34:57.690 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:34:57.688+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.689555+0000) 2020-02-06T02:34:57.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:34:57.712+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:57.712881+0000) 2020-02-06T02:34:59.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:34:59.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:34:59.829333+0000) 2020-02-06T02:35:00.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:00.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:00.074898+0000) 2020-02-06T02:35:00.688 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:00.687+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:00.687650+0000) 2020-02-06T02:35:00.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:00.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:00.962824+0000) 2020-02-06T02:35:02.044 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:02.042+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.043439+0000) 2020-02-06T02:35:02.045 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:02.043+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.043747+0000) 2020-02-06T02:35:02.045 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:02.044+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.044522+0000) 2020-02-06T02:35:02.046 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:02.044+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.045250+0000) 2020-02-06T02:35:02.047 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:02.044+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.045900+0000) 2020-02-06T02:35:02.047 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:02.045+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.046377+0000) 2020-02-06T02:35:02.048 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:02.046+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.047214+0000) 2020-02-06T02:35:02.049 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:02.047+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.047875+0000) 2020-02-06T02:35:02.049 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:02.047+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.048066+0000) 2020-02-06T02:35:02.050 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:02.048+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.048590+0000) 2020-02-06T02:35:02.051 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:02.048+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.049080+0000) 2020-02-06T02:35:02.051 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:02.049+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.049552+0000) 2020-02-06T02:35:02.052 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:02.047+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.048754+0000) 2020-02-06T02:35:02.059 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:02.056+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.058007+0000) 2020-02-06T02:35:02.071 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:02.069+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 236984 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:02.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:02.960+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:02.960896+0000) 2020-02-06T02:35:03.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:03.544+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:03.545036+0000) 2020-02-06T02:35:06.413 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:06.411+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:06.411901+0000) 2020-02-06T02:35:06.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:06.572+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:06.572906+0000) 2020-02-06T02:35:07.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:07.075+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.076133+0000) 2020-02-06T02:35:07.077 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:07.075+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.076490+0000) 2020-02-06T02:35:07.078 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:07.076+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.077217+0000) 2020-02-06T02:35:07.078 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:07.077+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.077835+0000) 2020-02-06T02:35:07.080 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:07.078+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.079104+0000) 2020-02-06T02:35:07.081 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:07.079+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.080306+0000) 2020-02-06T02:35:07.082 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:07.079+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.080283+0000) 2020-02-06T02:35:07.082 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:07.080+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.080517+0000) 2020-02-06T02:35:07.082 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:07.080+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.080660+0000) 2020-02-06T02:35:07.082 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:07.080+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.081122+0000) 2020-02-06T02:35:07.083 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:07.079+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.080872+0000) 2020-02-06T02:35:07.083 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:07.080+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.081425+0000) 2020-02-06T02:35:07.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:07.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.087301+0000) 2020-02-06T02:35:07.088 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:07.086+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.087593+0000) 2020-02-06T02:35:07.100 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:07.099+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:07.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:07.184+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.185196+0000) 2020-02-06T02:35:07.462 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:07.461+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.461397+0000) 2020-02-06T02:35:07.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:07.461+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.461393+0000) 2020-02-06T02:35:07.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:07.461+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.461564+0000) 2020-02-06T02:35:07.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:07.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.689807+0000) 2020-02-06T02:35:07.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:07.712+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:07.713124+0000) 2020-02-06T02:35:09.830 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:09.828+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:09.829576+0000) 2020-02-06T02:35:10.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:10.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:10.075151+0000) 2020-02-06T02:35:10.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:10.687+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:10.687897+0000) 2020-02-06T02:35:10.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:10.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:10.963058+0000) 2020-02-06T02:35:12.110 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:12.104+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.105351+0000) 2020-02-06T02:35:12.110 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:12.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.105820+0000) 2020-02-06T02:35:12.111 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:12.105+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.106504+0000) 2020-02-06T02:35:12.111 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:12.105+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.106812+0000) 2020-02-06T02:35:12.111 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:12.106+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.107511+0000) 2020-02-06T02:35:12.112 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:12.107+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.107633+0000) 2020-02-06T02:35:12.112 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:12.107+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.108094+0000) 2020-02-06T02:35:12.113 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:12.109+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.109581+0000) 2020-02-06T02:35:12.113 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:12.109+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.109723+0000) 2020-02-06T02:35:12.113 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:12.110+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.110607+0000) 2020-02-06T02:35:12.113 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:12.110+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.110736+0000) 2020-02-06T02:35:12.114 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:12.109+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.110211+0000) 2020-02-06T02:35:12.114 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:12.110+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.111312+0000) 2020-02-06T02:35:12.115 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:12.113+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.114229+0000) 2020-02-06T02:35:12.133 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:12.132+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237067 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:12.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:12.960+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:12.961120+0000) 2020-02-06T02:35:13.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:13.544+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:13.545300+0000) 2020-02-06T02:35:16.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:16.419+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:16.420322+0000) 2020-02-06T02:35:16.574 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:16.573+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:16.573164+0000) 2020-02-06T02:35:17.138 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:17.136+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.137216+0000) 2020-02-06T02:35:17.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:17.136+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.137926+0000) 2020-02-06T02:35:17.139 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:17.137+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.138732+0000) 2020-02-06T02:35:17.140 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:17.138+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.139789+0000) 2020-02-06T02:35:17.141 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:17.139+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.140074+0000) 2020-02-06T02:35:17.141 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:17.140+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.140725+0000) 2020-02-06T02:35:17.142 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:17.141+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.141663+0000) 2020-02-06T02:35:17.143 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:17.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.141936+0000) 2020-02-06T02:35:17.143 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:17.142+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.142621+0000) 2020-02-06T02:35:17.144 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:17.142+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.143122+0000) 2020-02-06T02:35:17.144 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:17.142+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.143410+0000) 2020-02-06T02:35:17.144 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:17.143+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.143766+0000) 2020-02-06T02:35:17.145 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:17.143+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.144677+0000) 2020-02-06T02:35:17.151 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:17.149+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.150196+0000) 2020-02-06T02:35:17.165 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:17.164+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237106 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:17.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:17.184+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.185416+0000) 2020-02-06T02:35:17.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:17.461+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.461679+0000) 2020-02-06T02:35:17.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:17.461+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.461671+0000) 2020-02-06T02:35:17.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:17.461+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.461811+0000) 2020-02-06T02:35:17.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:17.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.690113+0000) 2020-02-06T02:35:17.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:17.712+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:17.713288+0000) 2020-02-06T02:35:19.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:19.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:19.829820+0000) 2020-02-06T02:35:20.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:20.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:20.075384+0000) 2020-02-06T02:35:20.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:20.688+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:20.688130+0000) 2020-02-06T02:35:20.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:20.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:20.963287+0000) 2020-02-06T02:35:22.070 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:35:22.083 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:35:22.173 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:22.169+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.170762+0000) 2020-02-06T02:35:22.173 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:22.170+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.171219+0000) 2020-02-06T02:35:22.173 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:22.170+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.171424+0000) 2020-02-06T02:35:22.173 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:22.169+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.169927+0000) 2020-02-06T02:35:22.174 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:22.170+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.170526+0000) 2020-02-06T02:35:22.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:22.171+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.172633+0000) 2020-02-06T02:35:22.175 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:22.171+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.173069+0000) 2020-02-06T02:35:22.175 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:22.171+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.172022+0000) 2020-02-06T02:35:22.175 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:22.172+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.172579+0000) 2020-02-06T02:35:22.175 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:22.173+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.173669+0000) 2020-02-06T02:35:22.176 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:22.173+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.174298+0000) 2020-02-06T02:35:22.176 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:22.174+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.175136+0000) 2020-02-06T02:35:22.176 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:22.174+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.175128+0000) 2020-02-06T02:35:22.180 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:22.177+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.178853+0000) 2020-02-06T02:35:22.199 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:22.194+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237150 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:22.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:22.961+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:22.961350+0000) 2020-02-06T02:35:23.546 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:23.545+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:23.545510+0000) 2020-02-06T02:35:26.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:26.428+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:26.428692+0000) 2020-02-06T02:35:26.575 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:26.572+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:26.573413+0000) 2020-02-06T02:35:27.186 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:27.185+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.185652+0000) 2020-02-06T02:35:27.203 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:27.201+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.202254+0000) 2020-02-06T02:35:27.203 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:27.201+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.203022+0000) 2020-02-06T02:35:27.204 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:27.202+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.203592+0000) 2020-02-06T02:35:27.205 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:27.203+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.204402+0000) 2020-02-06T02:35:27.205 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:27.202+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.204381+0000) 2020-02-06T02:35:27.206 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:27.204+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.205123+0000) 2020-02-06T02:35:27.206 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:27.204+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.205660+0000) 2020-02-06T02:35:27.207 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:27.204+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.205863+0000) 2020-02-06T02:35:27.207 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:27.204+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.206230+0000) 2020-02-06T02:35:27.207 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:27.205+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.206801+0000) 2020-02-06T02:35:27.208 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:27.205+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.206807+0000) 2020-02-06T02:35:27.208 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:27.205+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.207045+0000) 2020-02-06T02:35:27.208 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:27.206+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.207775+0000) 2020-02-06T02:35:27.214 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:27.212+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.213550+0000) 2020-02-06T02:35:27.228 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:27.226+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237190 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:27.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:27.460+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.461909+0000) 2020-02-06T02:35:27.463 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:27.460+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.462033+0000) 2020-02-06T02:35:27.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:27.460+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.461908+0000) 2020-02-06T02:35:27.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:27.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.690276+0000) 2020-02-06T02:35:27.714 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:27.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:27.713504+0000) 2020-02-06T02:35:29.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:29.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:29.830131+0000) 2020-02-06T02:35:30.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:30.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:30.075590+0000) 2020-02-06T02:35:30.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:30.687+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:30.688392+0000) 2020-02-06T02:35:30.964 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:30.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:30.963523+0000) 2020-02-06T02:35:32.233 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:32.231+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.232447+0000) 2020-02-06T02:35:32.234 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:32.231+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.232898+0000) 2020-02-06T02:35:32.234 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:32.232+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.233446+0000) 2020-02-06T02:35:32.235 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:32.232+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.233961+0000) 2020-02-06T02:35:32.236 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:32.234+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.235597+0000) 2020-02-06T02:35:32.237 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:32.234+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.235750+0000) 2020-02-06T02:35:32.237 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:32.234+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.235940+0000) 2020-02-06T02:35:32.238 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:32.235+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.236427+0000) 2020-02-06T02:35:32.239 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:32.235+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.237037+0000) 2020-02-06T02:35:32.239 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:32.235+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.236761+0000) 2020-02-06T02:35:32.240 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:32.237+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.238393+0000) 2020-02-06T02:35:32.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:32.237+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.238691+0000) 2020-02-06T02:35:32.240 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:32.237+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.238813+0000) 2020-02-06T02:35:32.243 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:32.241+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.242418+0000) 2020-02-06T02:35:32.260 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:32.258+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237230 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:32.962 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:32.960+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:32.961647+0000) 2020-02-06T02:35:33.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:33.545+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:33.545738+0000) 2020-02-06T02:35:36.438 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:36.436+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:36.436871+0000) 2020-02-06T02:35:36.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:36.572+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:36.573728+0000) 2020-02-06T02:35:37.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:37.185+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.185882+0000) 2020-02-06T02:35:37.265 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:37.264+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.264214+0000) 2020-02-06T02:35:37.265 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:37.263+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.264814+0000) 2020-02-06T02:35:37.266 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:37.263+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.265325+0000) 2020-02-06T02:35:37.267 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:37.265+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.266015+0000) 2020-02-06T02:35:37.267 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:37.266+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.266159+0000) 2020-02-06T02:35:37.268 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:37.266+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.266644+0000) 2020-02-06T02:35:37.268 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:37.266+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.266666+0000) 2020-02-06T02:35:37.268 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:37.266+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.267544+0000) 2020-02-06T02:35:37.269 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:37.267+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.267940+0000) 2020-02-06T02:35:37.269 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:37.267+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.267942+0000) 2020-02-06T02:35:37.270 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:37.267+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.268733+0000) 2020-02-06T02:35:37.270 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:37.267+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.269019+0000) 2020-02-06T02:35:37.270 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:37.267+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.269214+0000) 2020-02-06T02:35:37.277 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:37.272+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.272984+0000) 2020-02-06T02:35:37.291 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:37.288+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237269 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:37.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:37.460+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.462148+0000) 2020-02-06T02:35:37.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:37.460+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.462176+0000) 2020-02-06T02:35:37.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:37.460+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.462248+0000) 2020-02-06T02:35:37.691 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:37.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.690485+0000) 2020-02-06T02:35:37.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:37.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:37.713714+0000) 2020-02-06T02:35:39.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:39.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:39.830395+0000) 2020-02-06T02:35:40.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:40.074+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:40.075846+0000) 2020-02-06T02:35:40.689 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:40.687+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:40.688680+0000) 2020-02-06T02:35:40.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:40.962+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:40.963758+0000) 2020-02-06T02:35:42.297 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:42.294+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.296141+0000) 2020-02-06T02:35:42.298 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:42.295+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.296938+0000) 2020-02-06T02:35:42.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:42.296+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.297148+0000) 2020-02-06T02:35:42.299 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:42.297+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.298226+0000) 2020-02-06T02:35:42.299 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:42.297+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.298657+0000) 2020-02-06T02:35:42.300 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:42.297+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.299043+0000) 2020-02-06T02:35:42.300 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:42.298+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.299495+0000) 2020-02-06T02:35:42.301 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:42.298+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.299995+0000) 2020-02-06T02:35:42.301 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:42.299+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.300740+0000) 2020-02-06T02:35:42.302 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:42.299+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.300850+0000) 2020-02-06T02:35:42.303 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:42.299+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.301026+0000) 2020-02-06T02:35:42.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:42.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.301546+0000) 2020-02-06T02:35:42.304 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:42.301+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.302677+0000) 2020-02-06T02:35:42.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:42.306+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.307457+0000) 2020-02-06T02:35:42.322 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:42.319+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237310 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:42.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:42.961+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:42.961905+0000) 2020-02-06T02:35:43.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:43.544+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:43.545977+0000) 2020-02-06T02:35:46.445 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:46.443+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:46.444006+0000) 2020-02-06T02:35:46.575 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:46.572+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:46.573978+0000) 2020-02-06T02:35:47.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:47.185+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.186146+0000) 2020-02-06T02:35:47.327 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:47.324+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.325914+0000) 2020-02-06T02:35:47.328 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:47.325+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.327240+0000) 2020-02-06T02:35:47.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:47.325+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.327313+0000) 2020-02-06T02:35:47.330 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:47.327+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.328788+0000) 2020-02-06T02:35:47.330 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:47.328+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.329579+0000) 2020-02-06T02:35:47.331 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:47.328+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.330161+0000) 2020-02-06T02:35:47.331 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:47.329+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.330434+0000) 2020-02-06T02:35:47.331 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:47.329+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.330791+0000) 2020-02-06T02:35:47.332 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:47.330+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.331314+0000) 2020-02-06T02:35:47.332 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:47.330+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.331541+0000) 2020-02-06T02:35:47.333 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:47.331+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.332536+0000) 2020-02-06T02:35:47.333 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:47.331+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.332603+0000) 2020-02-06T02:35:47.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:47.334+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.335965+0000) 2020-02-06T02:35:47.337 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:47.335+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.336227+0000) 2020-02-06T02:35:47.353 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:47.349+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237351 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:47.463 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:47.461+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.462366+0000) 2020-02-06T02:35:47.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:47.461+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.462401+0000) 2020-02-06T02:35:47.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:47.461+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.462515+0000) 2020-02-06T02:35:47.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:47.689+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.690710+0000) 2020-02-06T02:35:47.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:47.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:47.713943+0000) 2020-02-06T02:35:49.831 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:49.829+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:49.830595+0000) 2020-02-06T02:35:50.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:50.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:50.076134+0000) 2020-02-06T02:35:50.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:50.688+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:50.688931+0000) 2020-02-06T02:35:50.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:50.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:50.963985+0000) 2020-02-06T02:35:52.231 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:35:52.239 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:35:52.358 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:52.355+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.356742+0000) 2020-02-06T02:35:52.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:52.355+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.357269+0000) 2020-02-06T02:35:52.359 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:52.357+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.358589+0000) 2020-02-06T02:35:52.360 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:52.356+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.357794+0000) 2020-02-06T02:35:52.360 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:52.356+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.358144+0000) 2020-02-06T02:35:52.360 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:52.358+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.359267+0000) 2020-02-06T02:35:52.361 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:52.358+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.359461+0000) 2020-02-06T02:35:52.361 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:52.359+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.360336+0000) 2020-02-06T02:35:52.361 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:52.359+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.360597+0000) 2020-02-06T02:35:52.362 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:52.359+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.360588+0000) 2020-02-06T02:35:52.362 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:52.360+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.361225+0000) 2020-02-06T02:35:52.362 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:52.360+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.361642+0000) 2020-02-06T02:35:52.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:52.366+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.367509+0000) 2020-02-06T02:35:52.368 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:52.366+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.367733+0000) 2020-02-06T02:35:52.381 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:52.378+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237391 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:52.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:52.961+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:52.962120+0000) 2020-02-06T02:35:53.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:53.545+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:53.546191+0000) 2020-02-06T02:35:56.453 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:56.451+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:56.452540+0000) 2020-02-06T02:35:56.575 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:56.573+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:56.574208+0000) 2020-02-06T02:35:57.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:57.185+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.186340+0000) 2020-02-06T02:35:57.386 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:57.384+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.385382+0000) 2020-02-06T02:35:57.387 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:35:57.384+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.385952+0000) 2020-02-06T02:35:57.389 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:35:57.385+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.387229+0000) 2020-02-06T02:35:57.389 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:35:57.385+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.386519+0000) 2020-02-06T02:35:57.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:57.385+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.386884+0000) 2020-02-06T02:35:57.390 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:57.386+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.387681+0000) 2020-02-06T02:35:57.391 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:57.387+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.388570+0000) 2020-02-06T02:35:57.391 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:35:57.387+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.388681+0000) 2020-02-06T02:35:57.391 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:35:57.388+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.389811+0000) 2020-02-06T02:35:57.391 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:35:57.387+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.388984+0000) 2020-02-06T02:35:57.391 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:57.388+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.390012+0000) 2020-02-06T02:35:57.392 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:57.389+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.390616+0000) 2020-02-06T02:35:57.392 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:57.389+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.391125+0000) 2020-02-06T02:35:57.396 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:57.394+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.395683+0000) 2020-02-06T02:35:57.418 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:35:57.410+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237431 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:35:57.464 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:35:57.461+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.462595+0000) 2020-02-06T02:35:57.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:35:57.461+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.462613+0000) 2020-02-06T02:35:57.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:35:57.461+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.462701+0000) 2020-02-06T02:35:57.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:35:57.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.690967+0000) 2020-02-06T02:35:57.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:35:57.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:57.714155+0000) 2020-02-06T02:35:59.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:35:59.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:35:59.830916+0000) 2020-02-06T02:36:00.077 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:00.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:00.076380+0000) 2020-02-06T02:36:00.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:00.688+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:00.689182+0000) 2020-02-06T02:36:00.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:00.963+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:00.964218+0000) 2020-02-06T02:36:02.417 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:02.414+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.415998+0000) 2020-02-06T02:36:02.418 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:02.415+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.416900+0000) 2020-02-06T02:36:02.419 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:02.417+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.418494+0000) 2020-02-06T02:36:02.420 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:02.418+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.419517+0000) 2020-02-06T02:36:02.421 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:02.419+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.420470+0000) 2020-02-06T02:36:02.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:02.419+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.420968+0000) 2020-02-06T02:36:02.422 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:02.420+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.421768+0000) 2020-02-06T02:36:02.423 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:02.421+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.422410+0000) 2020-02-06T02:36:02.424 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:02.421+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.422776+0000) 2020-02-06T02:36:02.424 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:02.422+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.423260+0000) 2020-02-06T02:36:02.424 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:02.421+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.422956+0000) 2020-02-06T02:36:02.425 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:02.421+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.423133+0000) 2020-02-06T02:36:02.425 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:02.421+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.423409+0000) 2020-02-06T02:36:02.425 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:02.423+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.424696+0000) 2020-02-06T02:36:02.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:02.440+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237472 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:02.963 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:02.960+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:02.962364+0000) 2020-02-06T02:36:03.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:03.545+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:03.546411+0000) 2020-02-06T02:36:06.462 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:06.460+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:06.460907+0000) 2020-02-06T02:36:06.575 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:06.573+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:06.574472+0000) 2020-02-06T02:36:07.187 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:07.186+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.186567+0000) 2020-02-06T02:36:07.448 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:07.446+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.447373+0000) 2020-02-06T02:36:07.450 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:07.448+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.449432+0000) 2020-02-06T02:36:07.451 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:07.449+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.450072+0000) 2020-02-06T02:36:07.451 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:07.449+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.450524+0000) 2020-02-06T02:36:07.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:07.449+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.451060+0000) 2020-02-06T02:36:07.452 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:07.450+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.451402+0000) 2020-02-06T02:36:07.452 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:07.450+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.451811+0000) 2020-02-06T02:36:07.453 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:07.451+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.452470+0000) 2020-02-06T02:36:07.453 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:07.451+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.452548+0000) 2020-02-06T02:36:07.454 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:07.451+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.453164+0000) 2020-02-06T02:36:07.454 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:07.452+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.453603+0000) 2020-02-06T02:36:07.454 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:07.452+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.453560+0000) 2020-02-06T02:36:07.455 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:07.452+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.453861+0000) 2020-02-06T02:36:07.458 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:07.456+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.458060+0000) 2020-02-06T02:36:07.463 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:07.461+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.462779+0000) 2020-02-06T02:36:07.464 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:07.461+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.462839+0000) 2020-02-06T02:36:07.464 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:07.461+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.462854+0000) 2020-02-06T02:36:07.473 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:07.470+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237513 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:07.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:07.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.691183+0000) 2020-02-06T02:36:07.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:07.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:07.714362+0000) 2020-02-06T02:36:09.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:09.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:09.831157+0000) 2020-02-06T02:36:10.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:10.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:10.076631+0000) 2020-02-06T02:36:10.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:10.688+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:10.689458+0000) 2020-02-06T02:36:10.965 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:10.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:10.964536+0000) 2020-02-06T02:36:12.479 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:12.476+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.477750+0000) 2020-02-06T02:36:12.480 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:12.477+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.478204+0000) 2020-02-06T02:36:12.480 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:12.477+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.478754+0000) 2020-02-06T02:36:12.481 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:12.478+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.480039+0000) 2020-02-06T02:36:12.481 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:12.479+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.480331+0000) 2020-02-06T02:36:12.482 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:12.479+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.481034+0000) 2020-02-06T02:36:12.482 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:12.480+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.481336+0000) 2020-02-06T02:36:12.483 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:12.480+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.482303+0000) 2020-02-06T02:36:12.483 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:12.481+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.482648+0000) 2020-02-06T02:36:12.484 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:12.481+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.482992+0000) 2020-02-06T02:36:12.484 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:12.482+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.483389+0000) 2020-02-06T02:36:12.485 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:12.481+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.483416+0000) 2020-02-06T02:36:12.485 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:12.482+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.483728+0000) 2020-02-06T02:36:12.489 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:12.487+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.488456+0000) 2020-02-06T02:36:12.503 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:12.501+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237552 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:12.965 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:12.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:12.962641+0000) 2020-02-06T02:36:13.547 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:13.545+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:13.546596+0000) 2020-02-06T02:36:16.471 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:16.469+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:16.469941+0000) 2020-02-06T02:36:16.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:16.574+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:16.574728+0000) 2020-02-06T02:36:17.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:17.186+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.186838+0000) 2020-02-06T02:36:17.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:17.462+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.462987+0000) 2020-02-06T02:36:17.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:17.462+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.463026+0000) 2020-02-06T02:36:17.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:17.462+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.463076+0000) 2020-02-06T02:36:17.509 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:17.506+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.507984+0000) 2020-02-06T02:36:17.509 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:17.507+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.508937+0000) 2020-02-06T02:36:17.510 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:17.509+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.509529+0000) 2020-02-06T02:36:17.511 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:17.509+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.510345+0000) 2020-02-06T02:36:17.512 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:17.509+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.510890+0000) 2020-02-06T02:36:17.512 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:17.510+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.511136+0000) 2020-02-06T02:36:17.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:17.511+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.511533+0000) 2020-02-06T02:36:17.513 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:17.510+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.511437+0000) 2020-02-06T02:36:17.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:17.511+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.512075+0000) 2020-02-06T02:36:17.513 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:17.511+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.512469+0000) 2020-02-06T02:36:17.514 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:17.512+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.513397+0000) 2020-02-06T02:36:17.514 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:17.512+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.513576+0000) 2020-02-06T02:36:17.515 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:17.513+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.514158+0000) 2020-02-06T02:36:17.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:17.517+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.518126+0000) 2020-02-06T02:36:17.535 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:17.533+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237593 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:17.692 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:17.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.691399+0000) 2020-02-06T02:36:17.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:17.714+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:17.714576+0000) 2020-02-06T02:36:19.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:19.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:19.831404+0000) 2020-02-06T02:36:20.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:20.075+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:20.076860+0000) 2020-02-06T02:36:20.690 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:20.689+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:20.689697+0000) 2020-02-06T02:36:20.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:20.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:20.964768+0000) 2020-02-06T02:36:22.405 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:36:22.412 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:36:22.540 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:22.537+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.539047+0000) 2020-02-06T02:36:22.540 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:22.538+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.538903+0000) 2020-02-06T02:36:22.541 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:22.538+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.539536+0000) 2020-02-06T02:36:22.541 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:22.538+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.539807+0000) 2020-02-06T02:36:22.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:22.540+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.541130+0000) 2020-02-06T02:36:22.542 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:22.540+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.541453+0000) 2020-02-06T02:36:22.543 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:22.541+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.541964+0000) 2020-02-06T02:36:22.543 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:22.541+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.542145+0000) 2020-02-06T02:36:22.543 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:22.541+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.542445+0000) 2020-02-06T02:36:22.544 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:22.541+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.542874+0000) 2020-02-06T02:36:22.544 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:22.542+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.542912+0000) 2020-02-06T02:36:22.545 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:22.542+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.542960+0000) 2020-02-06T02:36:22.545 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:22.543+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.543960+0000) 2020-02-06T02:36:22.549 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:22.547+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.548085+0000) 2020-02-06T02:36:22.564 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:22.563+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237631 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:22.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:22.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:22.962855+0000) 2020-02-06T02:36:23.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:23.546+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:23.546857+0000) 2020-02-06T02:36:26.478 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:26.476+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:26.477781+0000) 2020-02-06T02:36:26.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:26.574+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:26.574980+0000) 2020-02-06T02:36:27.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:27.186+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.187129+0000) 2020-02-06T02:36:27.464 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:27.462+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.463242+0000) 2020-02-06T02:36:27.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:27.462+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.463300+0000) 2020-02-06T02:36:27.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:27.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.463360+0000) 2020-02-06T02:36:27.570 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:27.568+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.569903+0000) 2020-02-06T02:36:27.571 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:27.569+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.570385+0000) 2020-02-06T02:36:27.572 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:27.570+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.570787+0000) 2020-02-06T02:36:27.572 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:27.570+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.571712+0000) 2020-02-06T02:36:27.573 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:27.571+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.571646+0000) 2020-02-06T02:36:27.573 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:27.572+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.572640+0000) 2020-02-06T02:36:27.574 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:27.572+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.573582+0000) 2020-02-06T02:36:27.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:27.572+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.573349+0000) 2020-02-06T02:36:27.575 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:27.572+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.573517+0000) 2020-02-06T02:36:27.575 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:27.573+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.574401+0000) 2020-02-06T02:36:27.576 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:27.574+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.574957+0000) 2020-02-06T02:36:27.576 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:27.575+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.575858+0000) 2020-02-06T02:36:27.577 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:27.575+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.576144+0000) 2020-02-06T02:36:27.581 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:27.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.580335+0000) 2020-02-06T02:36:27.595 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:27.593+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237672 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:27.694 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:27.691+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.691612+0000) 2020-02-06T02:36:27.715 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:27.714+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:27.714814+0000) 2020-02-06T02:36:29.832 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:29.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:29.831659+0000) 2020-02-06T02:36:30.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:30.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:30.077122+0000) 2020-02-06T02:36:30.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:30.689+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:30.689953+0000) 2020-02-06T02:36:30.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:30.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:30.965000+0000) 2020-02-06T02:36:32.601 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:32.599+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.599729+0000) 2020-02-06T02:36:32.602 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:32.600+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.601309+0000) 2020-02-06T02:36:32.603 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:32.602+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.602372+0000) 2020-02-06T02:36:32.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:32.602+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.602156+0000) 2020-02-06T02:36:32.604 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:32.602+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.602665+0000) 2020-02-06T02:36:32.606 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:32.602+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.602418+0000) 2020-02-06T02:36:32.606 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:32.602+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.603314+0000) 2020-02-06T02:36:32.606 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:32.604+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.604469+0000) 2020-02-06T02:36:32.607 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:32.603+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.603414+0000) 2020-02-06T02:36:32.607 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:32.603+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.603945+0000) 2020-02-06T02:36:32.607 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:32.604+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.604565+0000) 2020-02-06T02:36:32.608 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:32.604+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.605086+0000) 2020-02-06T02:36:32.608 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:32.605+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.605440+0000) 2020-02-06T02:36:32.609 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:32.607+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.607408+0000) 2020-02-06T02:36:32.624 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:32.622+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237710 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:32.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:32.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:32.963085+0000) 2020-02-06T02:36:33.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:33.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:33.547115+0000) 2020-02-06T02:36:36.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:36.485+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:36.486352+0000) 2020-02-06T02:36:36.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:36.574+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:36.575232+0000) 2020-02-06T02:36:37.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:37.186+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.187312+0000) 2020-02-06T02:36:37.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:37.463+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.463534+0000) 2020-02-06T02:36:37.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:37.463+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.463512+0000) 2020-02-06T02:36:37.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:37.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.463742+0000) 2020-02-06T02:36:37.629 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:37.627+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.628439+0000) 2020-02-06T02:36:37.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:37.628+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.629923+0000) 2020-02-06T02:36:37.631 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:37.629+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.630070+0000) 2020-02-06T02:36:37.632 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:37.631+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.631498+0000) 2020-02-06T02:36:37.633 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:37.631+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.631990+0000) 2020-02-06T02:36:37.633 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:37.631+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.632405+0000) 2020-02-06T02:36:37.633 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:37.631+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.632998+0000) 2020-02-06T02:36:37.635 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:37.633+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.634310+0000) 2020-02-06T02:36:37.635 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:37.633+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.634516+0000) 2020-02-06T02:36:37.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:37.633+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.633618+0000) 2020-02-06T02:36:37.636 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:37.634+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.634717+0000) 2020-02-06T02:36:37.636 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:37.634+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.635316+0000) 2020-02-06T02:36:37.637 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:37.634+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.635266+0000) 2020-02-06T02:36:37.640 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:37.638+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.639623+0000) 2020-02-06T02:36:37.653 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:37.651+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237751 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:37.693 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:37.690+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.691823+0000) 2020-02-06T02:36:37.716 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:37.713+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:37.714995+0000) 2020-02-06T02:36:39.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:39.830+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:39.831913+0000) 2020-02-06T02:36:40.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:40.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:40.077352+0000) 2020-02-06T02:36:40.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:40.689+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:40.690194+0000) 2020-02-06T02:36:40.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:40.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:40.965269+0000) 2020-02-06T02:36:42.660 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:42.657+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.658457+0000) 2020-02-06T02:36:42.660 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:42.657+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.658646+0000) 2020-02-06T02:36:42.660 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:42.657+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.658980+0000) 2020-02-06T02:36:42.662 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:42.659+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.660093+0000) 2020-02-06T02:36:42.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:42.658+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.659484+0000) 2020-02-06T02:36:42.662 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:42.659+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.660767+0000) 2020-02-06T02:36:42.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:42.658+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.659972+0000) 2020-02-06T02:36:42.663 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:42.660+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.661251+0000) 2020-02-06T02:36:42.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:42.660+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.660646+0000) 2020-02-06T02:36:42.665 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:42.661+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.661478+0000) 2020-02-06T02:36:42.665 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:42.661+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.661679+0000) 2020-02-06T02:36:42.665 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:42.661+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.662044+0000) 2020-02-06T02:36:42.666 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:42.661+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.662364+0000) 2020-02-06T02:36:42.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:42.666+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.667753+0000) 2020-02-06T02:36:42.682 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:42.681+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:42.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:42.963+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:42.963302+0000) 2020-02-06T02:36:43.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:43.546+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:43.547339+0000) 2020-02-06T02:36:46.499 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:46.497+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:46.498172+0000) 2020-02-06T02:36:46.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:46.575+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:46.575473+0000) 2020-02-06T02:36:47.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:47.186+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.187547+0000) 2020-02-06T02:36:47.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:47.463+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.463807+0000) 2020-02-06T02:36:47.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:47.463+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.463762+0000) 2020-02-06T02:36:47.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:47.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.463986+0000) 2020-02-06T02:36:47.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:47.686+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.687537+0000) 2020-02-06T02:36:47.691 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:47.686+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.686881+0000) 2020-02-06T02:36:47.691 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:47.687+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.688934+0000) 2020-02-06T02:36:47.691 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:47.688+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.689605+0000) 2020-02-06T02:36:47.692 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:47.689+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.689579+0000) 2020-02-06T02:36:47.692 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:47.688+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.690026+0000) 2020-02-06T02:36:47.693 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:47.690+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.690689+0000) 2020-02-06T02:36:47.693 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:47.691+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.691728+0000) 2020-02-06T02:36:47.693 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:47.691+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.692387+0000) 2020-02-06T02:36:47.694 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:47.691+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.692399+0000) 2020-02-06T02:36:47.694 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:47.692+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.692686+0000) 2020-02-06T02:36:47.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:47.693+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.693668+0000) 2020-02-06T02:36:47.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:47.692+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.694030+0000) 2020-02-06T02:36:47.700 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:47.697+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.698846+0000) 2020-02-06T02:36:47.714 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:47.711+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237830 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:47.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:47.714+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:47.715182+0000) 2020-02-06T02:36:49.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:49.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:49.832152+0000) 2020-02-06T02:36:50.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:50.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:50.077615+0000) 2020-02-06T02:36:50.691 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:50.690+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:50.690481+0000) 2020-02-06T02:36:50.966 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:50.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:50.965559+0000) 2020-02-06T02:36:52.562 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:36:52.568 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:36:52.718 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:52.715+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.716594+0000) 2020-02-06T02:36:52.718 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:52.716+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.717795+0000) 2020-02-06T02:36:52.719 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:52.717+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.717539+0000) 2020-02-06T02:36:52.720 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:52.717+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.718178+0000) 2020-02-06T02:36:52.720 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:52.717+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.718555+0000) 2020-02-06T02:36:52.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:52.717+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.719059+0000) 2020-02-06T02:36:52.721 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:52.719+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.719822+0000) 2020-02-06T02:36:52.721 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:52.718+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.719544+0000) 2020-02-06T02:36:52.722 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:52.719+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.720232+0000) 2020-02-06T02:36:52.723 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:52.720+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.720860+0000) 2020-02-06T02:36:52.723 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:52.720+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.721232+0000) 2020-02-06T02:36:52.723 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:52.721+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.721881+0000) 2020-02-06T02:36:52.723 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:52.719+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.720961+0000) 2020-02-06T02:36:52.728 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:52.725+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.726700+0000) 2020-02-06T02:36:52.744 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:52.742+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:52.964 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:52.963+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:52.963534+0000) 2020-02-06T02:36:53.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:53.546+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:53.547544+0000) 2020-02-06T02:36:54.443 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:54.441+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:54.442489+0000) 2020-02-06T02:36:56.507 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:56.505+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:56.506533+0000) 2020-02-06T02:36:56.576 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:56.574+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:56.575695+0000) 2020-02-06T02:36:57.188 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:57.187+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.187809+0000) 2020-02-06T02:36:57.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:57.463+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.464057+0000) 2020-02-06T02:36:57.465 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:57.463+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.464039+0000) 2020-02-06T02:36:57.465 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:57.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.464224+0000) 2020-02-06T02:36:57.716 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:57.714+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.715437+0000) 2020-02-06T02:36:57.750 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:36:57.748+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.749533+0000) 2020-02-06T02:36:57.751 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:36:57.748+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.750117+0000) 2020-02-06T02:36:57.751 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:57.749+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.750429+0000) 2020-02-06T02:36:57.752 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:36:57.750+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.751132+0000) 2020-02-06T02:36:57.752 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:36:57.750+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.751445+0000) 2020-02-06T02:36:57.753 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:36:57.750+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.751713+0000) 2020-02-06T02:36:57.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:57.751+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.752679+0000) 2020-02-06T02:36:57.754 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:36:57.751+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.753000+0000) 2020-02-06T02:36:57.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:36:57.752+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.753645+0000) 2020-02-06T02:36:57.755 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:36:57.752+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.754129+0000) 2020-02-06T02:36:57.755 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:36:57.752+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.754239+0000) 2020-02-06T02:36:57.755 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:36:57.753+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.754895+0000) 2020-02-06T02:36:57.756 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:36:57.753+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.755271+0000) 2020-02-06T02:36:57.761 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:57.759+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:57.760483+0000) 2020-02-06T02:36:57.775 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:36:57.771+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:36:59.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:36:59.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:36:59.832390+0000) 2020-02-06T02:37:00.078 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:00.076+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:00.077828+0000) 2020-02-06T02:37:00.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:00.689+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:00.690734+0000) 2020-02-06T02:37:00.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:00.964+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:00.965817+0000) 2020-02-06T02:37:02.780 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:02.777+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.778808+0000) 2020-02-06T02:37:02.780 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:02.778+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.779540+0000) 2020-02-06T02:37:02.781 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:02.779+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.780585+0000) 2020-02-06T02:37:02.781 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:02.778+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.780332+0000) 2020-02-06T02:37:02.782 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:02.779+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.780985+0000) 2020-02-06T02:37:02.782 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:02.780+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.781485+0000) 2020-02-06T02:37:02.783 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:02.780+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.782178+0000) 2020-02-06T02:37:02.783 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:02.780+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.782158+0000) 2020-02-06T02:37:02.784 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:02.782+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.783181+0000) 2020-02-06T02:37:02.785 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:02.782+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.783959+0000) 2020-02-06T02:37:02.785 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:02.783+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.784404+0000) 2020-02-06T02:37:02.786 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:02.783+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.784561+0000) 2020-02-06T02:37:02.786 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:02.783+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.784769+0000) 2020-02-06T02:37:02.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:02.788+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.789239+0000) 2020-02-06T02:37:02.805 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:02.802+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237947 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:02.965 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:02.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:02.963775+0000) 2020-02-06T02:37:03.548 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:03.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:03.547771+0000) 2020-02-06T02:37:04.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:04.442+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:04.442757+0000) 2020-02-06T02:37:06.519 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:06.517+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:06.518398+0000) 2020-02-06T02:37:06.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:06.574+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:06.575934+0000) 2020-02-06T02:37:07.189 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:07.187+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.188041+0000) 2020-02-06T02:37:07.465 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:07.463+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.464303+0000) 2020-02-06T02:37:07.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:07.463+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.464303+0000) 2020-02-06T02:37:07.466 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:07.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.464492+0000) 2020-02-06T02:37:07.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:07.715+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.715661+0000) 2020-02-06T02:37:07.812 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:07.809+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.810297+0000) 2020-02-06T02:37:07.813 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:07.809+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.811055+0000) 2020-02-06T02:37:07.814 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:07.810+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.811351+0000) 2020-02-06T02:37:07.815 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:07.810+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.811776+0000) 2020-02-06T02:37:07.822 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:07.811+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.812184+0000) 2020-02-06T02:37:07.822 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:07.811+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.812749+0000) 2020-02-06T02:37:07.823 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:07.811+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.813359+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:07.812+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.814353+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:07.813+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.814744+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:07.813+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.815159+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:07.814+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.815519+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:07.812+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.813748+0000) 2020-02-06T02:37:07.824 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:07.814+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.815168+0000) 2020-02-06T02:37:07.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:07.819+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:07.820501+0000) 2020-02-06T02:37:07.839 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:07.832+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 237986 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:09.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:09.831+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:09.832634+0000) 2020-02-06T02:37:10.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:10.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:10.078109+0000) 2020-02-06T02:37:10.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:10.690+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:10.691015+0000) 2020-02-06T02:37:10.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:10.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:10.966048+0000) 2020-02-06T02:37:12.841 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:12.838+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.840227+0000) 2020-02-06T02:37:12.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:12.839+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.840784+0000) 2020-02-06T02:37:12.842 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:12.840+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.841313+0000) 2020-02-06T02:37:12.842 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:12.840+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.841757+0000) 2020-02-06T02:37:12.843 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:12.841+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.842688+0000) 2020-02-06T02:37:12.844 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:12.841+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.842457+0000) 2020-02-06T02:37:12.844 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:12.841+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.842739+0000) 2020-02-06T02:37:12.844 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:12.842+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.843145+0000) 2020-02-06T02:37:12.844 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:12.842+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.843443+0000) 2020-02-06T02:37:12.845 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:12.842+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.844046+0000) 2020-02-06T02:37:12.845 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:12.843+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.844534+0000) 2020-02-06T02:37:12.846 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:12.844+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.845217+0000) 2020-02-06T02:37:12.846 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:12.843+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.844758+0000) 2020-02-06T02:37:12.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:12.848+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.849825+0000) 2020-02-06T02:37:12.865 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:12.862+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238024 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:12.965 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:12.963+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:12.964011+0000) 2020-02-06T02:37:13.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:13.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:13.547975+0000) 2020-02-06T02:37:14.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:14.442+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:14.442985+0000) 2020-02-06T02:37:16.527 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:16.526+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:16.526714+0000) 2020-02-06T02:37:16.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:16.575+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:16.576234+0000) 2020-02-06T02:37:17.189 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:17.187+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.188308+0000) 2020-02-06T02:37:17.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:17.463+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.464555+0000) 2020-02-06T02:37:17.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:17.463+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.464625+0000) 2020-02-06T02:37:17.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:17.463+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.464754+0000) 2020-02-06T02:37:17.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:17.714+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.715877+0000) 2020-02-06T02:37:17.871 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:17.868+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.869851+0000) 2020-02-06T02:37:17.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:17.869+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.871259+0000) 2020-02-06T02:37:17.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:17.870+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.871892+0000) 2020-02-06T02:37:17.873 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:17.871+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.872482+0000) 2020-02-06T02:37:17.874 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:17.871+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.872937+0000) 2020-02-06T02:37:17.874 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:17.872+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.873235+0000) 2020-02-06T02:37:17.874 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:17.872+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.873502+0000) 2020-02-06T02:37:17.875 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:17.873+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.874142+0000) 2020-02-06T02:37:17.875 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:17.872+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.874181+0000) 2020-02-06T02:37:17.876 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:17.873+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.874312+0000) 2020-02-06T02:37:17.876 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:17.873+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.874760+0000) 2020-02-06T02:37:17.876 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:17.873+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.875303+0000) 2020-02-06T02:37:17.876 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:17.874+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.875749+0000) 2020-02-06T02:37:17.890 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:17.880+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:17.881384+0000) 2020-02-06T02:37:17.895 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:17.893+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238064 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:19.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:19.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:19.832866+0000) 2020-02-06T02:37:20.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:20.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:20.078329+0000) 2020-02-06T02:37:20.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:20.690+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:20.691254+0000) 2020-02-06T02:37:20.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:20.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:20.966321+0000) 2020-02-06T02:37:22.710 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:37:22.717 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:37:22.900 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:22.898+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.899351+0000) 2020-02-06T02:37:22.901 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:22.899+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.900295+0000) 2020-02-06T02:37:22.902 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:22.900+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.901246+0000) 2020-02-06T02:37:22.902 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:22.899+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.901053+0000) 2020-02-06T02:37:22.903 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:22.900+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.901838+0000) 2020-02-06T02:37:22.903 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:22.901+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.902173+0000) 2020-02-06T02:37:22.904 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:22.902+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.903456+0000) 2020-02-06T02:37:22.904 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:22.902+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.903999+0000) 2020-02-06T02:37:22.905 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:22.902+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.904206+0000) 2020-02-06T02:37:22.905 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:22.903+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.904868+0000) 2020-02-06T02:37:22.906 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:22.903+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.905110+0000) 2020-02-06T02:37:22.906 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:22.904+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.906037+0000) 2020-02-06T02:37:22.907 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:22.904+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.906250+0000) 2020-02-06T02:37:22.909 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:22.907+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.908802+0000) 2020-02-06T02:37:22.926 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:22.923+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238104 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:22.965 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:22.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:22.964211+0000) 2020-02-06T02:37:23.572 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:23.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:23.548218+0000) 2020-02-06T02:37:24.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:24.443+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:24.443220+0000) 2020-02-06T02:37:26.539 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:26.534+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:26.534701+0000) 2020-02-06T02:37:26.577 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:26.575+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:26.576482+0000) 2020-02-06T02:37:27.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:27.188+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.188542+0000) 2020-02-06T02:37:27.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:27.464+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.464859+0000) 2020-02-06T02:37:27.466 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:27.464+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.464900+0000) 2020-02-06T02:37:27.466 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:27.464+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.465027+0000) 2020-02-06T02:37:27.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:27.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.716121+0000) 2020-02-06T02:37:27.932 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:27.930+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.930841+0000) 2020-02-06T02:37:27.933 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:27.931+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.931997+0000) 2020-02-06T02:37:27.934 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:27.930+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.931905+0000) 2020-02-06T02:37:27.934 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:27.931+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.933017+0000) 2020-02-06T02:37:27.935 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:27.932+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.932587+0000) 2020-02-06T02:37:27.936 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:27.932+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.933549+0000) 2020-02-06T02:37:27.936 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:27.934+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.934189+0000) 2020-02-06T02:37:27.937 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:27.935+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.935166+0000) 2020-02-06T02:37:27.937 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:27.935+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.935352+0000) 2020-02-06T02:37:27.938 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:27.934+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.936071+0000) 2020-02-06T02:37:27.938 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:27.935+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.936629+0000) 2020-02-06T02:37:27.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:27.936+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.936454+0000) 2020-02-06T02:37:27.939 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:27.935+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.937287+0000) 2020-02-06T02:37:27.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:27.941+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:27.941363+0000) 2020-02-06T02:37:27.956 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:27.954+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238143 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:29.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:29.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:29.833127+0000) 2020-02-06T02:37:30.079 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:30.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:30.078542+0000) 2020-02-06T02:37:30.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:30.690+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:30.691501+0000) 2020-02-06T02:37:30.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:30.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:30.966555+0000) 2020-02-06T02:37:32.961 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:32.959+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.960244+0000) 2020-02-06T02:37:32.961 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:32.959+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.960685+0000) 2020-02-06T02:37:32.963 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:32.961+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.962485+0000) 2020-02-06T02:37:32.964 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:32.962+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.963326+0000) 2020-02-06T02:37:32.964 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:32.962+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.963757+0000) 2020-02-06T02:37:32.965 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:32.963+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.964647+0000) 2020-02-06T02:37:32.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:32.962+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.964317+0000) 2020-02-06T02:37:32.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:32.962+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.964374+0000) 2020-02-06T02:37:32.966 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:32.963+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.965028+0000) 2020-02-06T02:37:32.966 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:32.963+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.965279+0000) 2020-02-06T02:37:32.967 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:32.964+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.965356+0000) 2020-02-06T02:37:32.967 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:32.965+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.966480+0000) 2020-02-06T02:37:32.968 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:32.965+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.966986+0000) 2020-02-06T02:37:32.968 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:32.965+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.967329+0000) 2020-02-06T02:37:32.972 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:32.970+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:32.971379+0000) 2020-02-06T02:37:32.987 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:32.984+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238182 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:33.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:33.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:33.548428+0000) 2020-02-06T02:37:34.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:34.442+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:34.443489+0000) 2020-02-06T02:37:36.542 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:36.540+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:36.541644+0000) 2020-02-06T02:37:36.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:36.576+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:36.576787+0000) 2020-02-06T02:37:37.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:37.187+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.188812+0000) 2020-02-06T02:37:37.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:37.464+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.465115+0000) 2020-02-06T02:37:37.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:37.464+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.465151+0000) 2020-02-06T02:37:37.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:37.464+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.465312+0000) 2020-02-06T02:37:37.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:37.715+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.716367+0000) 2020-02-06T02:37:37.993 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:37.991+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.992422+0000) 2020-02-06T02:37:37.994 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:37.992+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.993121+0000) 2020-02-06T02:37:37.994 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:37.992+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.993295+0000) 2020-02-06T02:37:37.994 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:37.992+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.993831+0000) 2020-02-06T02:37:37.995 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:37.992+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.994235+0000) 2020-02-06T02:37:37.995 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:37.993+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.994801+0000) 2020-02-06T02:37:37.996 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:37.994+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.995393+0000) 2020-02-06T02:37:37.996 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:37.994+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.995700+0000) 2020-02-06T02:37:37.997 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:37.994+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.995850+0000) 2020-02-06T02:37:37.997 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:37.995+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.996741+0000) 2020-02-06T02:37:37.998 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:37.995+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.996817+0000) 2020-02-06T02:37:37.998 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:37.996+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.997886+0000) 2020-02-06T02:37:37.999 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:37.997+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:37.998253+0000) 2020-02-06T02:37:38.002 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:38.000+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:38.001544+0000) 2020-02-06T02:37:38.020 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:38.017+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238222 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:39.842 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:39.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:39.833388+0000) 2020-02-06T02:37:40.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:40.077+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:40.078772+0000) 2020-02-06T02:37:40.692 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:40.691+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:40.691753+0000) 2020-02-06T02:37:40.967 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:40.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:40.966797+0000) 2020-02-06T02:37:42.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:42.964+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:42.964738+0000) 2020-02-06T02:37:43.027 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:43.025+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.026494+0000) 2020-02-06T02:37:43.028 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:43.025+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.027041+0000) 2020-02-06T02:37:43.028 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:43.026+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.027571+0000) 2020-02-06T02:37:43.029 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:43.026+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.028363+0000) 2020-02-06T02:37:43.030 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:43.027+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.028868+0000) 2020-02-06T02:37:43.030 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:43.028+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.029261+0000) 2020-02-06T02:37:43.030 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:43.028+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.029458+0000) 2020-02-06T02:37:43.031 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:43.028+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.030009+0000) 2020-02-06T02:37:43.031 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:43.028+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.030349+0000) 2020-02-06T02:37:43.031 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:43.029+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.030687+0000) 2020-02-06T02:37:43.032 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:43.030+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.031216+0000) 2020-02-06T02:37:43.032 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:43.029+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.031181+0000) 2020-02-06T02:37:43.034 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:43.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.033782+0000) 2020-02-06T02:37:43.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:43.032+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.034001+0000) 2020-02-06T02:37:43.051 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:43.048+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238262 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:43.549 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:43.547+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:43.548640+0000) 2020-02-06T02:37:44.444 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:44.442+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:44.443702+0000) 2020-02-06T02:37:46.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:46.548+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:46.549113+0000) 2020-02-06T02:37:46.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:46.576+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:46.577041+0000) 2020-02-06T02:37:47.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:47.188+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:47.189114+0000) 2020-02-06T02:37:47.466 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:47.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:47.465407+0000) 2020-02-06T02:37:47.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:47.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:47.465414+0000) 2020-02-06T02:37:47.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:47.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:47.465605+0000) 2020-02-06T02:37:47.717 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:47.715+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:47.716600+0000) 2020-02-06T02:37:48.057 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:48.055+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.056326+0000) 2020-02-06T02:37:48.058 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:48.056+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.056925+0000) 2020-02-06T02:37:48.058 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:48.056+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.057395+0000) 2020-02-06T02:37:48.058 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:48.057+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.057756+0000) 2020-02-06T02:37:48.059 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:48.057+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.058442+0000) 2020-02-06T02:37:48.059 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:48.057+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.058681+0000) 2020-02-06T02:37:48.060 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:48.058+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.059141+0000) 2020-02-06T02:37:48.060 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:48.058+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.059437+0000) 2020-02-06T02:37:48.060 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:48.058+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.059614+0000) 2020-02-06T02:37:48.061 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:48.059+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.060144+0000) 2020-02-06T02:37:48.061 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:48.058+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.059867+0000) 2020-02-06T02:37:48.062 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:48.059+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.060819+0000) 2020-02-06T02:37:48.062 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:48.060+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.061350+0000) 2020-02-06T02:37:48.066 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:48.064+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:48.065712+0000) 2020-02-06T02:37:48.081 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:48.080+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238301 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:49.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:49.832+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:49.833636+0000) 2020-02-06T02:37:50.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:50.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:50.079009+0000) 2020-02-06T02:37:50.693 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:50.691+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:50.692018+0000) 2020-02-06T02:37:50.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:50.965+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:50.967049+0000) 2020-02-06T02:37:52.864 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:37:52.870 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:37:52.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:52.964+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:52.964955+0000) 2020-02-06T02:37:53.087 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:53.085+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.086423+0000) 2020-02-06T02:37:53.088 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:53.086+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.087185+0000) 2020-02-06T02:37:53.089 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:53.087+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.088185+0000) 2020-02-06T02:37:53.089 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:53.087+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.088283+0000) 2020-02-06T02:37:53.090 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:53.088+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.088707+0000) 2020-02-06T02:37:53.090 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:53.087+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.088633+0000) 2020-02-06T02:37:53.091 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:53.088+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.089117+0000) 2020-02-06T02:37:53.091 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:53.089+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.089807+0000) 2020-02-06T02:37:53.091 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:53.088+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.089719+0000) 2020-02-06T02:37:53.091 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:53.089+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.090130+0000) 2020-02-06T02:37:53.092 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:53.089+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.090350+0000) 2020-02-06T02:37:53.092 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:53.090+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.091251+0000) 2020-02-06T02:37:53.092 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:53.090+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.091190+0000) 2020-02-06T02:37:53.097 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:53.095+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.096874+0000) 2020-02-06T02:37:53.110 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:53.109+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238340 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:53.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:53.548+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:53.548854+0000) 2020-02-06T02:37:54.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:54.442+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:54.444014+0000) 2020-02-06T02:37:56.560 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:56.558+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:56.558842+0000) 2020-02-06T02:37:56.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:56.576+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:56.577298+0000) 2020-02-06T02:37:57.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:57.188+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:57.189327+0000) 2020-02-06T02:37:57.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:57.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:57.465756+0000) 2020-02-06T02:37:57.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:57.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:57.465768+0000) 2020-02-06T02:37:57.467 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:57.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:57.465879+0000) 2020-02-06T02:37:57.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:57.715+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:57.716827+0000) 2020-02-06T02:37:58.116 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:37:58.115+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.115479+0000) 2020-02-06T02:37:58.117 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:37:58.115+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.116170+0000) 2020-02-06T02:37:58.117 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:37:58.115+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.116333+0000) 2020-02-06T02:37:58.117 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:58.115+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.116778+0000) 2020-02-06T02:37:58.118 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:58.116+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.117324+0000) 2020-02-06T02:37:58.118 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:37:58.116+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.117510+0000) 2020-02-06T02:37:58.119 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:37:58.116+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.117559+0000) 2020-02-06T02:37:58.119 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:37:58.118+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.118417+0000) 2020-02-06T02:37:58.119 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:37:58.118+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.118896+0000) 2020-02-06T02:37:58.120 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:37:58.118+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.119304+0000) 2020-02-06T02:37:58.120 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:37:58.118+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.119007+0000) 2020-02-06T02:37:58.120 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:37:58.118+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.119342+0000) 2020-02-06T02:37:58.121 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:37:58.119+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.120256+0000) 2020-02-06T02:37:58.126 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:58.124+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:58.125523+0000) 2020-02-06T02:37:58.140 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:37:58.139+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238380 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:37:59.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:37:59.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:37:59.833872+0000) 2020-02-06T02:38:00.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:00.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:00.079264+0000) 2020-02-06T02:38:00.693 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:00.691+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:00.692256+0000) 2020-02-06T02:38:00.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:00.966+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:00.967327+0000) 2020-02-06T02:38:02.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:02.964+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:02.965262+0000) 2020-02-06T02:38:03.147 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:03.145+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.145822+0000) 2020-02-06T02:38:03.147 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:03.146+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.146788+0000) 2020-02-06T02:38:03.148 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:03.145+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.146902+0000) 2020-02-06T02:38:03.148 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:03.146+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.147497+0000) 2020-02-06T02:38:03.148 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:03.147+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.147775+0000) 2020-02-06T02:38:03.149 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:03.147+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.148214+0000) 2020-02-06T02:38:03.149 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:03.147+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.148581+0000) 2020-02-06T02:38:03.150 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:03.147+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.148833+0000) 2020-02-06T02:38:03.150 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:03.148+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.149335+0000) 2020-02-06T02:38:03.150 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:03.149+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.149768+0000) 2020-02-06T02:38:03.152 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:03.151+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.151460+0000) 2020-02-06T02:38:03.152 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:03.151+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.151467+0000) 2020-02-06T02:38:03.153 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:03.149+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.150352+0000) 2020-02-06T02:38:03.158 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:03.156+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.157660+0000) 2020-02-06T02:38:03.171 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:03.169+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238420 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:03.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:03.548+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:03.549101+0000) 2020-02-06T02:38:04.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:04.443+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:04.444276+0000) 2020-02-06T02:38:06.568 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:06.566+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:06.567278+0000) 2020-02-06T02:38:06.578 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:06.577+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:06.577508+0000) 2020-02-06T02:38:07.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:07.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:07.189555+0000) 2020-02-06T02:38:07.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:07.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:07.466024+0000) 2020-02-06T02:38:07.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:07.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:07.466020+0000) 2020-02-06T02:38:07.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:07.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:07.466167+0000) 2020-02-06T02:38:07.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:07.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:07.717125+0000) 2020-02-06T02:38:08.178 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:08.176+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.177888+0000) 2020-02-06T02:38:08.179 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:08.177+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.178582+0000) 2020-02-06T02:38:08.179 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:08.177+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.178363+0000) 2020-02-06T02:38:08.180 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:08.178+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.179049+0000) 2020-02-06T02:38:08.180 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:08.178+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.179556+0000) 2020-02-06T02:38:08.180 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:08.179+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.179842+0000) 2020-02-06T02:38:08.188 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:08.179+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.181040+0000) 2020-02-06T02:38:08.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:08.180+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.181494+0000) 2020-02-06T02:38:08.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:08.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.184164+0000) 2020-02-06T02:38:08.188 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:08.183+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.184390+0000) 2020-02-06T02:38:08.189 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:08.180+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.181615+0000) 2020-02-06T02:38:08.189 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:08.180+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.180724+0000) 2020-02-06T02:38:08.189 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:08.181+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.181936+0000) 2020-02-06T02:38:08.189 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:08.181+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:08.181941+0000) 2020-02-06T02:38:08.202 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:08.201+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:09.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:09.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:09.834139+0000) 2020-02-06T02:38:10.080 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:10.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:10.079484+0000) 2020-02-06T02:38:10.693 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:10.692+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:10.692489+0000) 2020-02-06T02:38:10.968 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:10.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:10.967583+0000) 2020-02-06T02:38:12.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:12.965+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:12.965526+0000) 2020-02-06T02:38:13.208 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:13.206+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.207346+0000) 2020-02-06T02:38:13.209 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:13.207+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.208741+0000) 2020-02-06T02:38:13.210 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:13.207+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.208299+0000) 2020-02-06T02:38:13.215 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:13.209+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.209413+0000) 2020-02-06T02:38:13.215 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:13.209+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.209408+0000) 2020-02-06T02:38:13.215 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:13.210+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.210453+0000) 2020-02-06T02:38:13.215 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:13.212+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.212720+0000) 2020-02-06T02:38:13.215 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:13.212+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.212488+0000) 2020-02-06T02:38:13.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:13.208+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.210010+0000) 2020-02-06T02:38:13.216 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:13.211+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.212554+0000) 2020-02-06T02:38:13.217 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:13.209+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.210950+0000) 2020-02-06T02:38:13.217 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:13.211+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.212136+0000) 2020-02-06T02:38:13.217 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:13.211+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.212381+0000) 2020-02-06T02:38:13.218 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:13.216+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.217989+0000) 2020-02-06T02:38:13.234 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:13.233+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:13.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:13.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:13.549314+0000) 2020-02-06T02:38:14.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:14.444+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:14.444506+0000) 2020-02-06T02:38:16.576 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:16.575+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:16.575370+0000) 2020-02-06T02:38:16.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:16.577+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:16.577775+0000) 2020-02-06T02:38:17.190 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:17.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:17.189793+0000) 2020-02-06T02:38:17.467 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:17.466+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:17.466260+0000) 2020-02-06T02:38:17.467 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:17.466+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:17.466259+0000) 2020-02-06T02:38:17.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:17.466+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:17.466480+0000) 2020-02-06T02:38:17.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:17.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:17.717308+0000) 2020-02-06T02:38:18.240 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:18.238+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.239162+0000) 2020-02-06T02:38:18.241 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:18.239+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.240186+0000) 2020-02-06T02:38:18.242 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:18.240+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.240690+0000) 2020-02-06T02:38:18.242 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:18.240+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.240907+0000) 2020-02-06T02:38:18.243 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:18.240+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.241336+0000) 2020-02-06T02:38:18.243 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:18.240+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.241706+0000) 2020-02-06T02:38:18.243 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:18.240+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.241929+0000) 2020-02-06T02:38:18.244 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:18.242+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.243126+0000) 2020-02-06T02:38:18.245 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:18.243+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.244148+0000) 2020-02-06T02:38:18.245 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:18.243+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.243841+0000) 2020-02-06T02:38:18.246 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:18.244+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.244745+0000) 2020-02-06T02:38:18.246 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:18.244+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.245141+0000) 2020-02-06T02:38:18.247 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:18.245+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.246150+0000) 2020-02-06T02:38:18.252 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:18.250+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:18.251399+0000) 2020-02-06T02:38:18.266 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:18.265+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238541 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:19.835 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:19.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:19.834364+0000) 2020-02-06T02:38:20.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:20.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:20.079749+0000) 2020-02-06T02:38:20.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:20.692+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:20.692738+0000) 2020-02-06T02:38:20.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:20.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:20.967861+0000) 2020-02-06T02:38:22.966 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:22.965+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:22.965781+0000) 2020-02-06T02:38:23.036 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:38:23.043 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:38:23.273 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:23.272+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.272125+0000) 2020-02-06T02:38:23.273 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:23.272+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.272472+0000) 2020-02-06T02:38:23.274 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:23.273+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.273523+0000) 2020-02-06T02:38:23.274 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:23.273+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.273705+0000) 2020-02-06T02:38:23.275 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:23.273+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.273894+0000) 2020-02-06T02:38:23.275 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:23.273+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.273822+0000) 2020-02-06T02:38:23.275 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:23.274+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.274427+0000) 2020-02-06T02:38:23.276 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:23.274+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.274560+0000) 2020-02-06T02:38:23.276 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:23.274+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.274918+0000) 2020-02-06T02:38:23.277 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:23.274+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.275200+0000) 2020-02-06T02:38:23.277 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:23.275+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.275446+0000) 2020-02-06T02:38:23.277 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:23.276+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.276656+0000) 2020-02-06T02:38:23.278 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:23.276+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.277005+0000) 2020-02-06T02:38:23.280 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:23.279+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.279555+0000) 2020-02-06T02:38:23.297 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:23.295+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238581 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:23.550 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:23.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:23.549530+0000) 2020-02-06T02:38:24.445 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:24.444+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:24.444734+0000) 2020-02-06T02:38:26.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:26.576+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:26.578010+0000) 2020-02-06T02:38:26.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:26.582+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:26.583352+0000) 2020-02-06T02:38:27.202 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:27.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:27.190067+0000) 2020-02-06T02:38:27.478 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:27.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:27.466510+0000) 2020-02-06T02:38:27.478 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:27.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:27.466507+0000) 2020-02-06T02:38:27.479 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:27.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:27.466734+0000) 2020-02-06T02:38:27.727 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:27.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:27.717488+0000) 2020-02-06T02:38:28.303 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:28.300+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.301863+0000) 2020-02-06T02:38:28.303 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:28.301+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.302805+0000) 2020-02-06T02:38:28.304 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:28.302+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.303742+0000) 2020-02-06T02:38:28.305 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:28.303+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.304248+0000) 2020-02-06T02:38:28.306 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:28.303+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.304896+0000) 2020-02-06T02:38:28.306 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:28.304+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.305273+0000) 2020-02-06T02:38:28.306 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:28.304+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.305713+0000) 2020-02-06T02:38:28.307 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:28.304+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.306081+0000) 2020-02-06T02:38:28.307 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:28.305+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.306494+0000) 2020-02-06T02:38:28.308 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:28.305+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.307060+0000) 2020-02-06T02:38:28.309 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:28.306+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.308109+0000) 2020-02-06T02:38:28.309 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:28.306+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.308261+0000) 2020-02-06T02:38:28.309 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:28.306+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.307962+0000) 2020-02-06T02:38:28.313 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:28.310+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:28.312037+0000) 2020-02-06T02:38:28.328 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:28.326+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238623 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:29.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:29.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:29.834594+0000) 2020-02-06T02:38:30.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:30.078+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:30.079997+0000) 2020-02-06T02:38:30.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:30.691+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:30.692995+0000) 2020-02-06T02:38:30.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:30.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:30.968141+0000) 2020-02-06T02:38:32.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:32.965+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:32.966024+0000) 2020-02-06T02:38:33.336 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:33.333+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.334846+0000) 2020-02-06T02:38:33.336 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:33.334+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.335558+0000) 2020-02-06T02:38:33.337 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:33.334+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.336283+0000) 2020-02-06T02:38:33.337 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:33.335+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.336609+0000) 2020-02-06T02:38:33.338 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:33.335+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.337024+0000) 2020-02-06T02:38:33.338 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:33.336+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.337599+0000) 2020-02-06T02:38:33.339 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:33.336+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.337756+0000) 2020-02-06T02:38:33.339 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:33.337+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.338226+0000) 2020-02-06T02:38:33.339 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:33.337+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.339010+0000) 2020-02-06T02:38:33.340 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:33.338+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.339232+0000) 2020-02-06T02:38:33.340 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:33.337+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.339374+0000) 2020-02-06T02:38:33.341 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:33.338+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.339581+0000) 2020-02-06T02:38:33.341 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:33.338+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.339886+0000) 2020-02-06T02:38:33.345 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:33.343+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.344237+0000) 2020-02-06T02:38:33.359 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:33.356+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238662 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:33.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:33.548+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:33.549777+0000) 2020-02-06T02:38:34.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:34.444+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:34.444970+0000) 2020-02-06T02:38:36.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:36.577+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:36.578248+0000) 2020-02-06T02:38:36.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:36.590+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:36.591439+0000) 2020-02-06T02:38:37.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:37.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:37.190285+0000) 2020-02-06T02:38:37.468 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:37.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:37.466757+0000) 2020-02-06T02:38:37.468 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:37.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:37.466756+0000) 2020-02-06T02:38:37.468 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:37.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:37.466951+0000) 2020-02-06T02:38:37.718 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:37.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:37.717712+0000) 2020-02-06T02:38:38.365 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:38.363+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.364807+0000) 2020-02-06T02:38:38.366 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:38.363+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.364592+0000) 2020-02-06T02:38:38.366 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:38.364+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.365490+0000) 2020-02-06T02:38:38.367 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:38.365+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.366952+0000) 2020-02-06T02:38:38.368 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:38.365+0000 7f63cd3d7700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.366693+0000) 2020-02-06T02:38:38.369 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:38.367+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.368605+0000) 2020-02-06T02:38:38.370 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:38.367+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.368979+0000) 2020-02-06T02:38:38.370 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:38.367+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.369043+0000) 2020-02-06T02:38:38.370 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:38.368+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.369399+0000) 2020-02-06T02:38:38.371 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:38.368+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.370226+0000) 2020-02-06T02:38:38.371 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:38.368+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.370345+0000) 2020-02-06T02:38:38.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:38.369+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.370190+0000) 2020-02-06T02:38:38.372 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:38.369+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.370205+0000) 2020-02-06T02:38:38.374 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:38.372+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:38.374000+0000) 2020-02-06T02:38:38.389 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:38.387+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238704 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:39.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:39.833+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:39.834834+0000) 2020-02-06T02:38:40.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:40.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:40.080265+0000) 2020-02-06T02:38:40.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:40.692+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:40.693253+0000) 2020-02-06T02:38:40.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:40.967+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:40.968393+0000) 2020-02-06T02:38:42.967 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:42.965+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:42.966278+0000) 2020-02-06T02:38:43.395 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:43.392+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.393901+0000) 2020-02-06T02:38:43.395 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:43.393+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.394983+0000) 2020-02-06T02:38:43.396 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:43.394+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.395570+0000) 2020-02-06T02:38:43.396 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:43.394+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.395905+0000) 2020-02-06T02:38:43.397 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:43.395+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.396385+0000) 2020-02-06T02:38:43.398 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:43.395+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.396652+0000) 2020-02-06T02:38:43.398 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:43.395+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.396818+0000) 2020-02-06T02:38:43.398 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:43.395+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.396647+0000) 2020-02-06T02:38:43.398 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:43.396+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.397458+0000) 2020-02-06T02:38:43.399 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:43.396+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.397450+0000) 2020-02-06T02:38:43.399 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:43.396+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.398370+0000) 2020-02-06T02:38:43.400 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:43.397+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.398829+0000) 2020-02-06T02:38:43.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:43.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.402725+0000) 2020-02-06T02:38:43.404 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:43.401+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.402976+0000) 2020-02-06T02:38:43.420 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:43.417+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238744 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:43.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:43.548+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:43.549988+0000) 2020-02-06T02:38:44.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:44.444+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:44.445320+0000) 2020-02-06T02:38:46.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:46.577+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:46.578480+0000) 2020-02-06T02:38:46.604 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:46.602+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:46.603357+0000) 2020-02-06T02:38:47.191 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:47.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:47.190514+0000) 2020-02-06T02:38:47.468 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:47.465+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:47.467022+0000) 2020-02-06T02:38:47.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:47.465+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:47.467011+0000) 2020-02-06T02:38:47.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:47.465+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:47.467199+0000) 2020-02-06T02:38:47.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:47.716+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:47.717955+0000) 2020-02-06T02:38:48.426 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:48.424+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.425164+0000) 2020-02-06T02:38:48.426 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:48.424+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.425960+0000) 2020-02-06T02:38:48.427 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:48.425+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.426597+0000) 2020-02-06T02:38:48.428 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:48.425+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.426945+0000) 2020-02-06T02:38:48.428 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:48.426+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.427545+0000) 2020-02-06T02:38:48.428 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:48.426+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.427857+0000) 2020-02-06T02:38:48.435 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:48.428+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.429621+0000) 2020-02-06T02:38:48.435 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:48.429+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.430471+0000) 2020-02-06T02:38:48.435 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:48.429+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.430790+0000) 2020-02-06T02:38:48.435 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:48.429+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.431223+0000) 2020-02-06T02:38:48.435 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:48.428+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.429201+0000) 2020-02-06T02:38:48.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:48.429+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.430668+0000) 2020-02-06T02:38:48.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:48.432+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.433846+0000) 2020-02-06T02:38:48.436 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:48.433+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:48.434136+0000) 2020-02-06T02:38:48.449 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:48.447+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238785 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:49.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:49.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:49.835105+0000) 2020-02-06T02:38:50.081 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:50.079+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:50.080452+0000) 2020-02-06T02:38:50.694 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:50.692+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:50.693480+0000) 2020-02-06T02:38:50.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:50.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:50.968622+0000) 2020-02-06T02:38:52.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:52.965+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:52.966534+0000) 2020-02-06T02:38:53.207 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:38:53.213 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:38:53.454 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:53.452+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.453632+0000) 2020-02-06T02:38:53.455 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:53.452+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.453918+0000) 2020-02-06T02:38:53.455 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:53.453+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.454735+0000) 2020-02-06T02:38:53.456 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:53.454+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.455828+0000) 2020-02-06T02:38:53.457 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:53.455+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.456256+0000) 2020-02-06T02:38:53.458 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:53.455+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.456671+0000) 2020-02-06T02:38:53.458 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:53.455+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.456691+0000) 2020-02-06T02:38:53.459 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:53.457+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.458254+0000) 2020-02-06T02:38:53.459 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:53.457+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.458513+0000) 2020-02-06T02:38:53.460 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:53.457+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.459000+0000) 2020-02-06T02:38:53.460 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:53.458+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.460059+0000) 2020-02-06T02:38:53.462 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:53.459+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.461008+0000) 2020-02-06T02:38:53.463 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:53.461+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.462828+0000) 2020-02-06T02:38:53.464 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:53.462+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.463761+0000) 2020-02-06T02:38:53.480 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:53.478+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238824 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:53.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:53.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:53.550144+0000) 2020-02-06T02:38:54.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:54.445+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:54.445546+0000) 2020-02-06T02:38:56.579 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:56.577+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:56.578659+0000) 2020-02-06T02:38:56.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:56.611+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:56.611927+0000) 2020-02-06T02:38:57.191 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:57.189+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:57.190694+0000) 2020-02-06T02:38:57.468 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:57.466+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:57.467259+0000) 2020-02-06T02:38:57.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:57.466+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:57.467253+0000) 2020-02-06T02:38:57.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:57.466+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:57.467482+0000) 2020-02-06T02:38:57.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:57.717+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:57.718191+0000) 2020-02-06T02:38:58.487 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:38:58.485+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.486528+0000) 2020-02-06T02:38:58.488 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:38:58.485+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.486943+0000) 2020-02-06T02:38:58.489 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:38:58.486+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.487956+0000) 2020-02-06T02:38:58.490 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:38:58.487+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.488540+0000) 2020-02-06T02:38:58.490 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:38:58.487+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.488624+0000) 2020-02-06T02:38:58.491 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:38:58.488+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.489194+0000) 2020-02-06T02:38:58.491 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:38:58.489+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.490168+0000) 2020-02-06T02:38:58.492 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:38:58.488+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.489489+0000) 2020-02-06T02:38:58.492 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:38:58.488+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.489504+0000) 2020-02-06T02:38:58.493 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:38:58.489+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.490391+0000) 2020-02-06T02:38:58.493 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:58.489+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.490755+0000) 2020-02-06T02:38:58.493 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:38:58.489+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.490946+0000) 2020-02-06T02:38:58.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:58.492+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.493396+0000) 2020-02-06T02:38:58.494 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:58.492+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:58.493608+0000) 2020-02-06T02:38:58.511 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:38:58.508+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238868 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:38:59.850 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:38:59.834+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:38:59.835371+0000) 2020-02-06T02:39:00.096 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:00.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:00.080687+0000) 2020-02-06T02:39:00.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:00.693+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:00.693735+0000) 2020-02-06T02:39:00.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:00.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:00.968847+0000) 2020-02-06T02:39:02.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:02.966+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:02.966789+0000) 2020-02-06T02:39:03.516 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:03.514+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.515740+0000) 2020-02-06T02:39:03.517 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:03.515+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.516682+0000) 2020-02-06T02:39:03.518 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:03.516+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.517401+0000) 2020-02-06T02:39:03.519 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:03.517+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.518164+0000) 2020-02-06T02:39:03.519 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:03.517+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.518680+0000) 2020-02-06T02:39:03.520 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:03.517+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.519083+0000) 2020-02-06T02:39:03.520 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:03.518+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.519855+0000) 2020-02-06T02:39:03.521 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:03.519+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.520358+0000) 2020-02-06T02:39:03.521 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:03.519+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.520603+0000) 2020-02-06T02:39:03.522 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:03.519+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.520915+0000) 2020-02-06T02:39:03.522 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:03.519+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.521361+0000) 2020-02-06T02:39:03.522 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:03.520+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.521230+0000) 2020-02-06T02:39:03.523 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:03.520+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.521994+0000) 2020-02-06T02:39:03.524 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:03.522+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.523486+0000) 2020-02-06T02:39:03.542 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:03.539+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238908 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:03.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:03.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:03.550317+0000) 2020-02-06T02:39:04.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:04.445+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:04.445753+0000) 2020-02-06T02:39:06.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:06.578+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:06.578890+0000) 2020-02-06T02:39:06.622 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:06.621+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:06.621395+0000) 2020-02-06T02:39:07.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:07.190+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:07.190923+0000) 2020-02-06T02:39:07.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:07.466+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:07.467517+0000) 2020-02-06T02:39:07.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:07.466+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:07.467517+0000) 2020-02-06T02:39:07.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:07.466+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:07.467734+0000) 2020-02-06T02:39:07.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:07.717+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:07.718402+0000) 2020-02-06T02:39:08.547 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:08.545+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.546275+0000) 2020-02-06T02:39:08.548 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:08.545+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.547320+0000) 2020-02-06T02:39:08.549 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:08.547+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.548266+0000) 2020-02-06T02:39:08.549 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:08.547+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.548908+0000) 2020-02-06T02:39:08.550 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:08.548+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.549372+0000) 2020-02-06T02:39:08.551 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:08.548+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.550361+0000) 2020-02-06T02:39:08.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:08.549+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.550887+0000) 2020-02-06T02:39:08.552 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:08.550+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.551330+0000) 2020-02-06T02:39:08.553 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:08.550+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.551781+0000) 2020-02-06T02:39:08.553 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:08.550+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.552124+0000) 2020-02-06T02:39:08.553 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:08.550+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.552348+0000) 2020-02-06T02:39:08.554 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:08.551+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.553385+0000) 2020-02-06T02:39:08.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:08.552+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.553533+0000) 2020-02-06T02:39:08.554 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:08.552+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:08.553574+0000) 2020-02-06T02:39:08.574 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:08.571+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238951 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:09.836 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:09.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:09.835607+0000) 2020-02-06T02:39:10.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:10.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:10.080878+0000) 2020-02-06T02:39:10.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:10.693+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:10.693987+0000) 2020-02-06T02:39:10.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:10.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:10.969033+0000) 2020-02-06T02:39:12.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:12.966+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:12.967034+0000) 2020-02-06T02:39:13.551 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:13.550+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.550537+0000) 2020-02-06T02:39:13.583 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:13.577+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.578482+0000) 2020-02-06T02:39:13.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:13.578+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.579966+0000) 2020-02-06T02:39:13.584 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:13.579+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.580417+0000) 2020-02-06T02:39:13.584 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:13.580+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.581537+0000) 2020-02-06T02:39:13.584 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:13.580+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.581679+0000) 2020-02-06T02:39:13.585 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:13.581+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.582125+0000) 2020-02-06T02:39:13.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:13.579+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.580577+0000) 2020-02-06T02:39:13.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:13.579+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.581164+0000) 2020-02-06T02:39:13.587 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:13.580+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.582167+0000) 2020-02-06T02:39:13.588 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:13.582+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.583563+0000) 2020-02-06T02:39:13.588 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:13.582+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.583930+0000) 2020-02-06T02:39:13.588 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:13.582+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.584243+0000) 2020-02-06T02:39:13.589 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:13.581+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.582910+0000) 2020-02-06T02:39:13.589 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:13.587+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:13.588687+0000) 2020-02-06T02:39:13.605 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:13.603+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 238991 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:14.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:14.445+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:14.445962+0000) 2020-02-06T02:39:16.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:16.578+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:16.579122+0000) 2020-02-06T02:39:16.631 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:16.629+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:16.629815+0000) 2020-02-06T02:39:17.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:17.191+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:17.191159+0000) 2020-02-06T02:39:17.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:17.467+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:17.467759+0000) 2020-02-06T02:39:17.469 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:17.467+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:17.467759+0000) 2020-02-06T02:39:17.469 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:17.467+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:17.467980+0000) 2020-02-06T02:39:17.719 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:17.718+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:17.718635+0000) 2020-02-06T02:39:18.611 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:18.609+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.609963+0000) 2020-02-06T02:39:18.612 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:18.610+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.610972+0000) 2020-02-06T02:39:18.612 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:18.611+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.611294+0000) 2020-02-06T02:39:18.612 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:18.611+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.611944+0000) 2020-02-06T02:39:18.613 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:18.612+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.612406+0000) 2020-02-06T02:39:18.614 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:18.612+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.612939+0000) 2020-02-06T02:39:18.614 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:18.613+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.613341+0000) 2020-02-06T02:39:18.615 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:18.613+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.613867+0000) 2020-02-06T02:39:18.615 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:18.614+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.614647+0000) 2020-02-06T02:39:18.616 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:18.614+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.615092+0000) 2020-02-06T02:39:18.616 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:18.615+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.615488+0000) 2020-02-06T02:39:18.617 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:18.615+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.615897+0000) 2020-02-06T02:39:18.617 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:18.615+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.615905+0000) 2020-02-06T02:39:18.621 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:18.620+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:18.620374+0000) 2020-02-06T02:39:18.634 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:18.632+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239034 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:19.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:19.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:19.835868+0000) 2020-02-06T02:39:20.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:20.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:20.081135+0000) 2020-02-06T02:39:20.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:20.693+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:20.694239+0000) 2020-02-06T02:39:20.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:20.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:20.969299+0000) 2020-02-06T02:39:22.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:22.966+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:22.967306+0000) 2020-02-06T02:39:23.414 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:39:23.420 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:39:23.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:23.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.550778+0000) 2020-02-06T02:39:23.639 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:23.637+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.638327+0000) 2020-02-06T02:39:23.640 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:23.638+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.639067+0000) 2020-02-06T02:39:23.640 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:23.638+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.639375+0000) 2020-02-06T02:39:23.641 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:23.638+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.639955+0000) 2020-02-06T02:39:23.643 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:23.641+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.642190+0000) 2020-02-06T02:39:23.644 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:23.642+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643169+0000) 2020-02-06T02:39:23.644 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:23.642+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643283+0000) 2020-02-06T02:39:23.645 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:23.642+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643309+0000) 2020-02-06T02:39:23.645 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:23.641+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.642985+0000) 2020-02-06T02:39:23.645 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:23.642+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643900+0000) 2020-02-06T02:39:23.646 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:23.643+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643510+0000) 2020-02-06T02:39:23.646 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:23.643+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.643923+0000) 2020-02-06T02:39:23.646 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:23.643+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.644451+0000) 2020-02-06T02:39:23.648 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:23.646+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:23.647236+0000) 2020-02-06T02:39:23.666 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:23.664+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239074 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:24.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:24.445+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:24.446289+0000) 2020-02-06T02:39:26.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:26.578+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:26.579359+0000) 2020-02-06T02:39:26.641 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:26.638+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:26.639760+0000) 2020-02-06T02:39:27.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:27.190+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:27.191437+0000) 2020-02-06T02:39:27.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:27.467+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:27.467981+0000) 2020-02-06T02:39:27.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:27.467+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:27.468016+0000) 2020-02-06T02:39:27.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:27.467+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:27.468230+0000) 2020-02-06T02:39:27.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:27.717+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:27.718888+0000) 2020-02-06T02:39:28.672 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:28.669+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.671038+0000) 2020-02-06T02:39:28.673 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:28.671+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.672257+0000) 2020-02-06T02:39:28.673 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:28.672+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.672478+0000) 2020-02-06T02:39:28.673 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:28.671+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.671975+0000) 2020-02-06T02:39:28.674 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:28.671+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.672969+0000) 2020-02-06T02:39:28.674 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:28.672+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.673076+0000) 2020-02-06T02:39:28.675 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:28.673+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.673816+0000) 2020-02-06T02:39:28.675 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:28.673+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.674014+0000) 2020-02-06T02:39:28.676 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:28.673+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.674708+0000) 2020-02-06T02:39:28.676 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:28.673+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.674836+0000) 2020-02-06T02:39:28.676 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:28.673+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.675021+0000) 2020-02-06T02:39:28.676 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:28.675+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.676096+0000) 2020-02-06T02:39:28.677 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:28.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.675656+0000) 2020-02-06T02:39:28.680 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:28.678+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:28.679906+0000) 2020-02-06T02:39:28.697 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:28.695+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239117 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:29.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:29.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:29.836138+0000) 2020-02-06T02:39:30.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:30.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:30.081362+0000) 2020-02-06T02:39:30.695 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:30.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:30.694493+0000) 2020-02-06T02:39:30.970 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:30.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:30.969526+0000) 2020-02-06T02:39:32.968 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:32.967+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:32.967544+0000) 2020-02-06T02:39:33.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:33.549+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.550987+0000) 2020-02-06T02:39:33.701 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:33.699+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.700699+0000) 2020-02-06T02:39:33.704 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:33.702+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.702120+0000) 2020-02-06T02:39:33.704 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:33.702+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.703586+0000) 2020-02-06T02:39:33.705 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:33.704+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.704488+0000) 2020-02-06T02:39:33.706 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:33.704+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.705858+0000) 2020-02-06T02:39:33.707 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:33.705+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.705514+0000) 2020-02-06T02:39:33.707 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:33.705+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.706419+0000) 2020-02-06T02:39:33.707 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:33.706+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.706632+0000) 2020-02-06T02:39:33.708 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:33.706+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.707330+0000) 2020-02-06T02:39:33.708 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:33.706+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.707575+0000) 2020-02-06T02:39:33.709 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:33.707+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.707708+0000) 2020-02-06T02:39:33.709 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:33.707+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.708062+0000) 2020-02-06T02:39:33.709 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:33.706+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.707996+0000) 2020-02-06T02:39:33.712 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:33.710+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:33.711529+0000) 2020-02-06T02:39:33.728 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:33.726+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239157 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:34.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:34.445+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:34.446498+0000) 2020-02-06T02:39:36.580 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:36.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:36.579639+0000) 2020-02-06T02:39:36.652 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:36.650+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:36.651440+0000) 2020-02-06T02:39:37.192 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:37.190+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:37.191705+0000) 2020-02-06T02:39:37.469 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:37.467+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:37.468231+0000) 2020-02-06T02:39:37.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:37.467+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:37.468237+0000) 2020-02-06T02:39:37.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:37.468+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:37.468424+0000) 2020-02-06T02:39:37.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:37.718+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:37.719116+0000) 2020-02-06T02:39:38.733 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:38.731+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.732585+0000) 2020-02-06T02:39:38.734 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:38.732+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.733713+0000) 2020-02-06T02:39:38.735 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:38.733+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.734317+0000) 2020-02-06T02:39:38.735 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:38.734+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.734833+0000) 2020-02-06T02:39:38.736 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:38.734+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.735241+0000) 2020-02-06T02:39:38.736 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:38.735+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.735802+0000) 2020-02-06T02:39:38.737 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:38.735+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.736325+0000) 2020-02-06T02:39:38.738 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:38.736+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.737229+0000) 2020-02-06T02:39:38.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:38.737+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.738241+0000) 2020-02-06T02:39:38.739 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:38.737+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.738396+0000) 2020-02-06T02:39:38.740 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:38.738+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.738883+0000) 2020-02-06T02:39:38.740 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:38.738+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.739409+0000) 2020-02-06T02:39:38.740 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:38.739+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.739779+0000) 2020-02-06T02:39:38.743 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:38.741+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:38.742821+0000) 2020-02-06T02:39:38.759 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:38.757+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239201 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:39.837 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:39.835+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:39.836383+0000) 2020-02-06T02:39:40.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:40.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:40.081610+0000) 2020-02-06T02:39:40.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:40.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:40.694731+0000) 2020-02-06T02:39:40.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:40.968+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:40.969741+0000) 2020-02-06T02:39:42.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:42.967+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:42.967811+0000) 2020-02-06T02:39:43.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:43.550+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.551286+0000) 2020-02-06T02:39:43.766 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:43.763+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.764942+0000) 2020-02-06T02:39:43.766 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:43.764+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.765344+0000) 2020-02-06T02:39:43.767 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:43.764+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.765729+0000) 2020-02-06T02:39:43.767 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:43.765+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.766068+0000) 2020-02-06T02:39:43.767 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:43.766+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.766588+0000) 2020-02-06T02:39:43.768 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:43.766+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.767267+0000) 2020-02-06T02:39:43.768 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:43.766+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.766926+0000) 2020-02-06T02:39:43.769 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:43.767+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.768514+0000) 2020-02-06T02:39:43.769 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:43.767+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.768244+0000) 2020-02-06T02:39:43.771 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:43.769+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.769467+0000) 2020-02-06T02:39:43.771 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:43.769+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.770117+0000) 2020-02-06T02:39:43.771 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:43.769+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.770191+0000) 2020-02-06T02:39:43.772 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:43.768+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.769766+0000) 2020-02-06T02:39:43.790 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:43.786+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239241 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:43.790 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:43.776+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:43.777626+0000) 2020-02-06T02:39:44.447 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:44.446+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:44.446713+0000) 2020-02-06T02:39:46.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:46.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:46.579891+0000) 2020-02-06T02:39:46.660 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:46.659+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:46.659688+0000) 2020-02-06T02:39:47.203 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:47.191+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:47.191938+0000) 2020-02-06T02:39:47.484 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:47.468+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:47.468480+0000) 2020-02-06T02:39:47.484 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:47.468+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:47.468481+0000) 2020-02-06T02:39:47.485 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:47.468+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:47.468698+0000) 2020-02-06T02:39:47.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:47.718+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:47.719361+0000) 2020-02-06T02:39:48.794 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:48.792+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.793355+0000) 2020-02-06T02:39:48.795 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:48.792+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.793997+0000) 2020-02-06T02:39:48.795 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:48.793+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.794507+0000) 2020-02-06T02:39:48.796 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:48.794+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.794808+0000) 2020-02-06T02:39:48.796 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:48.794+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.795029+0000) 2020-02-06T02:39:48.796 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:48.794+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.795339+0000) 2020-02-06T02:39:48.797 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:48.795+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.796163+0000) 2020-02-06T02:39:48.797 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:48.795+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.796305+0000) 2020-02-06T02:39:48.800 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:48.795+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.796905+0000) 2020-02-06T02:39:48.801 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:48.796+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.797170+0000) 2020-02-06T02:39:48.801 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:48.797+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.797976+0000) 2020-02-06T02:39:48.802 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:48.798+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.798854+0000) 2020-02-06T02:39:48.802 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:48.796+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.797402+0000) 2020-02-06T02:39:48.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:48.802+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:48.803885+0000) 2020-02-06T02:39:48.819 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:48.817+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239284 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:49.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:49.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:49.836616+0000) 2020-02-06T02:39:50.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:50.080+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:50.081861+0000) 2020-02-06T02:39:50.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:50.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:50.694979+0000) 2020-02-06T02:39:50.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:50.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:50.969976+0000) 2020-02-06T02:39:52.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:52.968+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:52.968112+0000) 2020-02-06T02:39:53.552 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:53.551+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.551535+0000) 2020-02-06T02:39:53.576 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:39:53.582 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:39:53.826 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:53.821+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.822463+0000) 2020-02-06T02:39:53.827 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:53.824+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.824547+0000) 2020-02-06T02:39:53.827 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:53.823+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.824722+0000) 2020-02-06T02:39:53.827 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:53.824+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.825404+0000) 2020-02-06T02:39:53.828 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:53.825+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.826515+0000) 2020-02-06T02:39:53.828 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:53.825+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.826282+0000) 2020-02-06T02:39:53.828 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:53.825+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.826400+0000) 2020-02-06T02:39:53.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:53.826+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.827561+0000) 2020-02-06T02:39:53.829 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:53.826+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.827752+0000) 2020-02-06T02:39:53.830 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:53.827+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.828408+0000) 2020-02-06T02:39:53.830 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:53.828+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.829323+0000) 2020-02-06T02:39:53.831 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:53.828+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.829412+0000) 2020-02-06T02:39:53.831 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:53.829+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.829716+0000) 2020-02-06T02:39:53.834 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:53.832+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:53.833428+0000) 2020-02-06T02:39:53.850 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:53.848+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239325 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:54.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:54.446+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:54.446925+0000) 2020-02-06T02:39:56.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:56.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:56.580184+0000) 2020-02-06T02:39:56.669 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:56.667+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:56.668360+0000) 2020-02-06T02:39:57.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:57.191+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:57.192181+0000) 2020-02-06T02:39:57.470 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:57.467+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:57.468724+0000) 2020-02-06T02:39:57.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:57.467+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:57.468717+0000) 2020-02-06T02:39:57.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:57.467+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:57.468950+0000) 2020-02-06T02:39:57.720 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:57.719+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:57.719595+0000) 2020-02-06T02:39:58.856 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:39:58.854+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.855598+0000) 2020-02-06T02:39:58.857 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:58.855+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.856269+0000) 2020-02-06T02:39:58.857 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:39:58.855+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.856505+0000) 2020-02-06T02:39:58.858 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:39:58.855+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.856877+0000) 2020-02-06T02:39:58.859 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:39:58.856+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.857877+0000) 2020-02-06T02:39:58.860 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:39:58.857+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.858470+0000) 2020-02-06T02:39:58.860 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:39:58.857+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.859335+0000) 2020-02-06T02:39:58.861 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:39:58.858+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.859637+0000) 2020-02-06T02:39:58.861 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:39:58.858+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.860028+0000) 2020-02-06T02:39:58.862 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:58.858+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.859882+0000) 2020-02-06T02:39:58.862 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:39:58.859+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.860375+0000) 2020-02-06T02:39:58.863 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:39:58.859+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.860652+0000) 2020-02-06T02:39:58.863 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:39:58.859+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.860742+0000) 2020-02-06T02:39:58.865 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:58.863+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:58.864587+0000) 2020-02-06T02:39:58.881 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:39:58.879+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239369 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:39:59.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:39:59.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:39:59.836845+0000) 2020-02-06T02:40:00.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:00.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:00.082124+0000) 2020-02-06T02:40:00.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:00.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:00.695214+0000) 2020-02-06T02:40:00.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:00.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:00.970230+0000) 2020-02-06T02:40:02.969 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:02.966+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:02.968347+0000) 2020-02-06T02:40:03.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:03.550+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.551829+0000) 2020-02-06T02:40:03.887 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:03.884+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.885789+0000) 2020-02-06T02:40:03.888 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:03.885+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.886993+0000) 2020-02-06T02:40:03.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:03.886+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.887613+0000) 2020-02-06T02:40:03.889 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:03.887+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.888150+0000) 2020-02-06T02:40:03.890 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:03.887+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.888998+0000) 2020-02-06T02:40:03.890 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:03.887+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.888790+0000) 2020-02-06T02:40:03.891 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:03.887+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.888709+0000) 2020-02-06T02:40:03.891 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:03.888+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.889595+0000) 2020-02-06T02:40:03.891 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:03.889+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.890531+0000) 2020-02-06T02:40:03.894 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:03.890+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.891580+0000) 2020-02-06T02:40:03.894 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:03.890+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.891997+0000) 2020-02-06T02:40:03.895 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:03.890+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.892283+0000) 2020-02-06T02:40:03.895 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:03.890+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.892358+0000) 2020-02-06T02:40:03.897 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:03.894+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:03.895705+0000) 2020-02-06T02:40:03.912 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:03.910+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239409 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:04.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:04.446+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:04.447133+0000) 2020-02-06T02:40:06.581 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:06.578+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:06.580384+0000) 2020-02-06T02:40:06.681 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:06.680+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:06.680236+0000) 2020-02-06T02:40:07.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:07.192+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:07.192454+0000) 2020-02-06T02:40:07.470 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:07.468+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:07.468951+0000) 2020-02-06T02:40:07.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:07.468+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:07.468946+0000) 2020-02-06T02:40:07.470 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:07.468+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:07.469171+0000) 2020-02-06T02:40:07.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:07.718+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:07.719882+0000) 2020-02-06T02:40:08.917 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:08.915+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.916642+0000) 2020-02-06T02:40:08.918 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:08.916+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.917496+0000) 2020-02-06T02:40:08.919 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:08.917+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.918634+0000) 2020-02-06T02:40:08.920 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:08.917+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.919192+0000) 2020-02-06T02:40:08.920 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:08.918+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.919531+0000) 2020-02-06T02:40:08.921 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:08.919+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.920450+0000) 2020-02-06T02:40:08.922 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:08.920+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.921160+0000) 2020-02-06T02:40:08.923 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:08.921+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.922264+0000) 2020-02-06T02:40:08.924 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:08.920+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.922313+0000) 2020-02-06T02:40:08.925 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:08.921+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.922631+0000) 2020-02-06T02:40:08.925 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:08.921+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.922832+0000) 2020-02-06T02:40:08.925 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:08.921+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.923341+0000) 2020-02-06T02:40:08.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:08.924+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.925606+0000) 2020-02-06T02:40:08.926 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:08.924+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:08.925849+0000) 2020-02-06T02:40:08.944 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:08.942+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239452 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:09.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:09.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:09.837059+0000) 2020-02-06T02:40:10.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:10.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:10.082352+0000) 2020-02-06T02:40:10.696 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:10.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:10.695471+0000) 2020-02-06T02:40:10.971 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:10.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:10.970481+0000) 2020-02-06T02:40:12.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:12.967+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:12.968635+0000) 2020-02-06T02:40:13.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:13.552+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.552068+0000) 2020-02-06T02:40:13.950 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:13.949+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.949220+0000) 2020-02-06T02:40:13.950 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:13.949+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.949853+0000) 2020-02-06T02:40:13.951 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:13.948+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.950347+0000) 2020-02-06T02:40:13.952 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:13.950+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.950612+0000) 2020-02-06T02:40:13.952 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:13.949+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.951128+0000) 2020-02-06T02:40:13.953 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:13.951+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.952054+0000) 2020-02-06T02:40:13.953 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:13.952+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.952460+0000) 2020-02-06T02:40:13.954 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:13.951+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.952666+0000) 2020-02-06T02:40:13.954 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:13.953+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.953381+0000) 2020-02-06T02:40:13.955 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:13.953+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.953587+0000) 2020-02-06T02:40:13.955 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:13.953+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.954580+0000) 2020-02-06T02:40:13.955 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:13.953+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.954937+0000) 2020-02-06T02:40:13.956 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:13.953+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.955226+0000) 2020-02-06T02:40:13.958 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:13.957+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:13.957111+0000) 2020-02-06T02:40:13.976 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:13.973+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239493 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:14.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:14.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:14.447293+0000) 2020-02-06T02:40:16.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:16.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:16.580710+0000) 2020-02-06T02:40:16.690 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:16.687+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:16.688659+0000) 2020-02-06T02:40:17.193 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:17.191+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:17.192689+0000) 2020-02-06T02:40:17.470 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:17.467+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:17.469211+0000) 2020-02-06T02:40:17.470 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:17.467+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:17.469214+0000) 2020-02-06T02:40:17.471 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:17.467+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:17.469365+0000) 2020-02-06T02:40:17.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:17.719+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:17.720114+0000) 2020-02-06T02:40:18.981 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:18.979+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.980462+0000) 2020-02-06T02:40:18.982 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:18.980+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.981122+0000) 2020-02-06T02:40:18.982 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:18.980+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.981843+0000) 2020-02-06T02:40:18.988 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:18.983+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.984190+0000) 2020-02-06T02:40:18.988 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:18.983+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.984743+0000) 2020-02-06T02:40:18.989 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:18.983+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.985025+0000) 2020-02-06T02:40:18.989 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:18.984+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.986025+0000) 2020-02-06T02:40:18.989 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:18.985+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.986595+0000) 2020-02-06T02:40:18.989 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:18.981+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.982982+0000) 2020-02-06T02:40:18.990 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:18.983+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.985337+0000) 2020-02-06T02:40:18.990 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:18.983+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.985339+0000) 2020-02-06T02:40:18.990 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:18.984+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.986125+0000) 2020-02-06T02:40:18.990 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:18.985+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.986641+0000) 2020-02-06T02:40:18.991 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:18.988+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:18.989863+0000) 2020-02-06T02:40:19.006 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:19.003+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239538 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:19.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:19.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:19.837337+0000) 2020-02-06T02:40:20.083 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:20.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:20.082611+0000) 2020-02-06T02:40:20.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:20.694+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:20.695724+0000) 2020-02-06T02:40:20.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:20.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:20.970708+0000) 2020-02-06T02:40:22.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:22.968+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:22.968898+0000) 2020-02-06T02:40:23.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:23.551+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:23.552308+0000) 2020-02-06T02:40:23.740 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:40:23.747 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:40:24.013 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:24.010+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.011838+0000) 2020-02-06T02:40:24.014 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:24.011+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.013124+0000) 2020-02-06T02:40:24.014 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:24.012+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.013657+0000) 2020-02-06T02:40:24.015 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:24.013+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.014616+0000) 2020-02-06T02:40:24.016 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:24.014+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.015061+0000) 2020-02-06T02:40:24.016 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:24.014+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.015691+0000) 2020-02-06T02:40:24.017 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:24.014+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.016079+0000) 2020-02-06T02:40:24.017 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:24.015+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.016242+0000) 2020-02-06T02:40:24.018 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:24.015+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.016580+0000) 2020-02-06T02:40:24.018 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:24.015+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.017118+0000) 2020-02-06T02:40:24.018 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:24.016+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.017565+0000) 2020-02-06T02:40:24.019 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:24.017+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.018250+0000) 2020-02-06T02:40:24.019 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:24.017+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.018594+0000) 2020-02-06T02:40:24.022 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:24.020+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.021549+0000) 2020-02-06T02:40:24.038 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:24.035+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239577 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:24.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:24.446+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:24.447502+0000) 2020-02-06T02:40:26.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:26.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:26.580904+0000) 2020-02-06T02:40:26.697 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:26.695+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:26.696766+0000) 2020-02-06T02:40:27.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:27.192+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:27.192929+0000) 2020-02-06T02:40:27.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:27.468+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:27.469494+0000) 2020-02-06T02:40:27.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:27.468+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:27.469512+0000) 2020-02-06T02:40:27.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:27.468+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:27.469657+0000) 2020-02-06T02:40:27.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:27.719+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:27.720339+0000) 2020-02-06T02:40:29.043 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:29.041+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.042325+0000) 2020-02-06T02:40:29.044 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:29.042+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.043278+0000) 2020-02-06T02:40:29.044 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:29.042+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.043580+0000) 2020-02-06T02:40:29.045 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:29.043+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.044552+0000) 2020-02-06T02:40:29.046 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:29.043+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.045064+0000) 2020-02-06T02:40:29.047 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:29.044+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.045771+0000) 2020-02-06T02:40:29.047 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:29.044+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.045913+0000) 2020-02-06T02:40:29.048 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:29.045+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.046319+0000) 2020-02-06T02:40:29.048 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:29.045+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.047122+0000) 2020-02-06T02:40:29.049 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:29.047+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.048073+0000) 2020-02-06T02:40:29.050 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:29.047+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.048377+0000) 2020-02-06T02:40:29.050 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:29.046+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.048081+0000) 2020-02-06T02:40:29.050 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:29.046+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.047992+0000) 2020-02-06T02:40:29.053 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:29.051+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.052938+0000) 2020-02-06T02:40:29.067 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:29.064+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239621 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:29.838 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:29.836+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:29.837550+0000) 2020-02-06T02:40:30.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:30.081+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:30.082860+0000) 2020-02-06T02:40:30.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:30.695+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:30.695976+0000) 2020-02-06T02:40:30.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:30.969+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:30.970934+0000) 2020-02-06T02:40:32.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:32.968+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:32.969141+0000) 2020-02-06T02:40:33.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:33.551+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:33.552543+0000) 2020-02-06T02:40:34.074 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:34.071+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.072890+0000) 2020-02-06T02:40:34.074 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:34.072+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.073166+0000) 2020-02-06T02:40:34.075 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:34.072+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.074000+0000) 2020-02-06T02:40:34.075 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:34.073+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.074547+0000) 2020-02-06T02:40:34.076 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:34.073+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.074863+0000) 2020-02-06T02:40:34.076 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:34.073+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.075002+0000) 2020-02-06T02:40:34.076 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:34.074+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.075518+0000) 2020-02-06T02:40:34.077 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:34.075+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.076562+0000) 2020-02-06T02:40:34.077 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:34.075+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.076536+0000) 2020-02-06T02:40:34.078 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:34.075+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.076996+0000) 2020-02-06T02:40:34.078 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:34.076+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.077460+0000) 2020-02-06T02:40:34.078 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:34.076+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.077836+0000) 2020-02-06T02:40:34.079 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:34.076+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.077660+0000) 2020-02-06T02:40:34.082 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:34.080+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.081968+0000) 2020-02-06T02:40:34.097 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:34.094+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239661 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:34.448 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:34.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:34.447716+0000) 2020-02-06T02:40:36.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:36.579+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:36.581138+0000) 2020-02-06T02:40:36.710 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:36.703+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:36.704358+0000) 2020-02-06T02:40:37.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:37.192+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:37.193182+0000) 2020-02-06T02:40:37.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:37.469+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:37.469734+0000) 2020-02-06T02:40:37.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:37.469+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:37.469764+0000) 2020-02-06T02:40:37.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:37.469+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:37.469892+0000) 2020-02-06T02:40:37.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:37.719+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:37.720581+0000) 2020-02-06T02:40:39.103 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:39.101+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.102406+0000) 2020-02-06T02:40:39.104 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:39.101+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.102948+0000) 2020-02-06T02:40:39.104 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:39.102+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.103601+0000) 2020-02-06T02:40:39.105 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:39.102+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.104085+0000) 2020-02-06T02:40:39.105 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:39.103+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.104684+0000) 2020-02-06T02:40:39.106 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:39.104+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.105489+0000) 2020-02-06T02:40:39.107 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:39.104+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.105707+0000) 2020-02-06T02:40:39.107 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:39.104+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.105916+0000) 2020-02-06T02:40:39.108 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:39.105+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.106694+0000) 2020-02-06T02:40:39.108 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:39.105+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.106567+0000) 2020-02-06T02:40:39.108 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:39.105+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.106834+0000) 2020-02-06T02:40:39.108 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:39.106+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.107233+0000) 2020-02-06T02:40:39.109 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:39.105+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.107170+0000) 2020-02-06T02:40:39.111 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:39.109+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.110144+0000) 2020-02-06T02:40:39.127 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:39.125+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239706 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:39.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:39.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:39.837761+0000) 2020-02-06T02:40:40.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:40.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:40.083103+0000) 2020-02-06T02:40:40.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:40.695+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:40.696231+0000) 2020-02-06T02:40:40.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:40.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:40.971178+0000) 2020-02-06T02:40:42.978 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:42.968+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:42.969394+0000) 2020-02-06T02:40:43.553 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:43.552+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:43.552803+0000) 2020-02-06T02:40:44.134 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:44.131+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.132901+0000) 2020-02-06T02:40:44.134 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:44.131+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.133005+0000) 2020-02-06T02:40:44.135 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:44.132+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.133454+0000) 2020-02-06T02:40:44.135 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:44.133+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.134702+0000) 2020-02-06T02:40:44.135 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:44.133+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.134794+0000) 2020-02-06T02:40:44.136 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:44.134+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.135544+0000) 2020-02-06T02:40:44.137 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:44.135+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.136178+0000) 2020-02-06T02:40:44.137 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:44.134+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.136291+0000) 2020-02-06T02:40:44.138 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:44.135+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.136503+0000) 2020-02-06T02:40:44.138 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:44.135+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.137003+0000) 2020-02-06T02:40:44.138 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:44.136+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.137487+0000) 2020-02-06T02:40:44.139 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:44.136+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.138071+0000) 2020-02-06T02:40:44.139 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:44.137+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.138548+0000) 2020-02-06T02:40:44.142 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:44.140+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.141834+0000) 2020-02-06T02:40:44.160 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:44.157+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239745 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:44.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:44.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:44.447931+0000) 2020-02-06T02:40:46.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:46.580+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:46.581393+0000) 2020-02-06T02:40:46.713 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:46.712+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:46.712613+0000) 2020-02-06T02:40:47.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:47.192+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:47.193385+0000) 2020-02-06T02:40:47.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:47.469+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:47.469992+0000) 2020-02-06T02:40:47.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:47.469+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:47.470021+0000) 2020-02-06T02:40:47.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:47.469+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:47.470171+0000) 2020-02-06T02:40:47.721 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:47.720+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:47.720811+0000) 2020-02-06T02:40:49.166 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:49.164+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.165699+0000) 2020-02-06T02:40:49.167 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:49.165+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.166310+0000) 2020-02-06T02:40:49.168 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:49.166+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.166537+0000) 2020-02-06T02:40:49.168 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:49.165+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.166862+0000) 2020-02-06T02:40:49.168 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:49.166+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.167193+0000) 2020-02-06T02:40:49.169 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:49.166+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.168043+0000) 2020-02-06T02:40:49.169 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:49.167+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.168239+0000) 2020-02-06T02:40:49.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:49.167+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.168525+0000) 2020-02-06T02:40:49.170 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:49.168+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.169354+0000) 2020-02-06T02:40:49.170 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:49.167+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.168796+0000) 2020-02-06T02:40:49.171 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:49.169+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.169678+0000) 2020-02-06T02:40:49.171 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:49.169+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.169772+0000) 2020-02-06T02:40:49.171 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:49.169+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.170351+0000) 2020-02-06T02:40:49.177 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:49.175+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.176995+0000) 2020-02-06T02:40:49.189 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:49.188+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239789 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:49.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:49.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:49.837972+0000) 2020-02-06T02:40:50.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:50.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:50.083309+0000) 2020-02-06T02:40:50.697 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:50.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:50.696482+0000) 2020-02-06T02:40:50.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:50.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:50.971457+0000) 2020-02-06T02:40:52.970 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:52.969+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:52.969675+0000) 2020-02-06T02:40:53.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:53.552+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:53.553031+0000) 2020-02-06T02:40:53.885 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:40:53.891 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:40:54.196 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:54.194+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.195649+0000) 2020-02-06T02:40:54.197 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:54.194+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.196011+0000) 2020-02-06T02:40:54.197 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:54.195+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.196379+0000) 2020-02-06T02:40:54.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:54.195+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.196519+0000) 2020-02-06T02:40:54.198 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:54.195+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.196751+0000) 2020-02-06T02:40:54.199 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:54.197+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.197753+0000) 2020-02-06T02:40:54.199 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:54.197+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.197834+0000) 2020-02-06T02:40:54.199 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:54.196+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.197201+0000) 2020-02-06T02:40:54.199 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:54.196+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.197681+0000) 2020-02-06T02:40:54.200 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:54.198+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.198520+0000) 2020-02-06T02:40:54.200 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:54.198+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.198710+0000) 2020-02-06T02:40:54.200 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:54.198+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.199087+0000) 2020-02-06T02:40:54.201 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:54.199+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.199712+0000) 2020-02-06T02:40:54.205 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:54.203+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.204161+0000) 2020-02-06T02:40:54.220 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:54.218+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239829 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:54.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:54.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:54.448181+0000) 2020-02-06T02:40:56.582 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:56.581+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:56.581680+0000) 2020-02-06T02:40:56.722 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:56.720+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:56.721223+0000) 2020-02-06T02:40:57.194 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:57.193+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:57.193619+0000) 2020-02-06T02:40:57.471 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:57.469+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:57.470264+0000) 2020-02-06T02:40:57.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:57.469+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:57.470264+0000) 2020-02-06T02:40:57.472 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:57.470+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:57.470470+0000) 2020-02-06T02:40:57.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:57.720+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:57.721033+0000) 2020-02-06T02:40:59.227 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:59.224+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.225750+0000) 2020-02-06T02:40:59.227 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:40:59.225+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.226481+0000) 2020-02-06T02:40:59.227 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:40:59.225+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.226586+0000) 2020-02-06T02:40:59.228 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:40:59.226+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.227288+0000) 2020-02-06T02:40:59.228 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:40:59.227+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.227809+0000) 2020-02-06T02:40:59.229 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:59.227+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.228201+0000) 2020-02-06T02:40:59.229 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:40:59.228+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.228657+0000) 2020-02-06T02:40:59.230 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:40:59.228+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.229403+0000) 2020-02-06T02:40:59.231 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:40:59.228+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.229364+0000) 2020-02-06T02:40:59.231 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:40:59.229+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.229940+0000) 2020-02-06T02:40:59.231 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:40:59.230+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.230420+0000) 2020-02-06T02:40:59.232 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:40:59.230+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.231898+0000) 2020-02-06T02:40:59.233 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:40:59.231+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.232188+0000) 2020-02-06T02:40:59.238 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:59.236+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.237101+0000) 2020-02-06T02:40:59.252 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:40:59.250+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239873 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:40:59.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:40:59.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:40:59.838201+0000) 2020-02-06T02:41:00.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:00.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:00.083488+0000) 2020-02-06T02:41:00.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:00.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:00.696721+0000) 2020-02-06T02:41:00.972 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:00.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:00.971682+0000) 2020-02-06T02:41:02.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:02.969+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:02.969932+0000) 2020-02-06T02:41:03.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:03.553+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:03.553304+0000) 2020-02-06T02:41:04.257 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:04.254+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.255913+0000) 2020-02-06T02:41:04.258 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:04.257+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.257695+0000) 2020-02-06T02:41:04.259 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:04.257+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.258266+0000) 2020-02-06T02:41:04.259 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:04.258+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.258697+0000) 2020-02-06T02:41:04.260 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:04.258+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.259048+0000) 2020-02-06T02:41:04.261 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:04.259+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.260069+0000) 2020-02-06T02:41:04.261 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:04.259+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.260171+0000) 2020-02-06T02:41:04.261 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:04.259+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.260358+0000) 2020-02-06T02:41:04.262 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:04.259+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.260660+0000) 2020-02-06T02:41:04.262 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:04.259+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.260933+0000) 2020-02-06T02:41:04.263 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:04.260+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.261292+0000) 2020-02-06T02:41:04.263 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:04.260+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.261661+0000) 2020-02-06T02:41:04.263 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:04.261+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.262374+0000) 2020-02-06T02:41:04.267 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:04.265+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.266812+0000) 2020-02-06T02:41:04.283 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:04.282+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239912 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:04.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:04.448+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:04.448341+0000) 2020-02-06T02:41:06.583 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:06.581+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:06.581929+0000) 2020-02-06T02:41:06.730 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:06.729+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:06.729520+0000) 2020-02-06T02:41:07.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:07.193+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:07.193872+0000) 2020-02-06T02:41:07.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:07.470+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:07.470508+0000) 2020-02-06T02:41:07.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:07.470+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:07.470512+0000) 2020-02-06T02:41:07.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:07.470+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:07.470691+0000) 2020-02-06T02:41:07.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:07.721+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:07.721300+0000) 2020-02-06T02:41:09.294 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:09.288+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.288414+0000) 2020-02-06T02:41:09.294 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:09.289+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.289168+0000) 2020-02-06T02:41:09.295 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:09.289+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.290136+0000) 2020-02-06T02:41:09.296 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:09.290+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.291179+0000) 2020-02-06T02:41:09.296 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:09.291+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.291838+0000) 2020-02-06T02:41:09.297 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:09.292+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.292380+0000) 2020-02-06T02:41:09.297 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:09.291+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.292209+0000) 2020-02-06T02:41:09.297 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:09.292+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.293135+0000) 2020-02-06T02:41:09.298 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:09.293+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.293850+0000) 2020-02-06T02:41:09.298 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:09.293+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.294082+0000) 2020-02-06T02:41:09.299 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:09.294+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.294545+0000) 2020-02-06T02:41:09.299 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:09.295+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.295271+0000) 2020-02-06T02:41:09.299 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:09.294+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.294726+0000) 2020-02-06T02:41:09.305 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:09.301+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.301734+0000) 2020-02-06T02:41:09.321 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:09.314+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239955 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:09.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:09.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:09.838416+0000) 2020-02-06T02:41:10.084 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:10.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:10.083698+0000) 2020-02-06T02:41:10.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:10.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:10.696956+0000) 2020-02-06T02:41:10.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:10.970+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:10.971915+0000) 2020-02-06T02:41:12.971 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:12.970+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:12.970214+0000) 2020-02-06T02:41:13.554 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:13.552+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:13.553547+0000) 2020-02-06T02:41:14.322 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:14.320+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.321068+0000) 2020-02-06T02:41:14.323 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:14.321+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.322331+0000) 2020-02-06T02:41:14.323 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:14.321+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.321801+0000) 2020-02-06T02:41:14.324 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:14.322+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.323227+0000) 2020-02-06T02:41:14.324 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:14.322+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.323609+0000) 2020-02-06T02:41:14.325 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:14.322+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.323185+0000) 2020-02-06T02:41:14.326 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:14.323+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.324666+0000) 2020-02-06T02:41:14.326 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:14.324+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.324424+0000) 2020-02-06T02:41:14.327 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:14.325+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.326127+0000) 2020-02-06T02:41:14.328 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:14.325+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.326506+0000) 2020-02-06T02:41:14.328 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:14.325+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.325772+0000) 2020-02-06T02:41:14.328 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:14.325+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.326357+0000) 2020-02-06T02:41:14.328 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:14.326+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.326613+0000) 2020-02-06T02:41:14.333 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:14.331+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.332396+0000) 2020-02-06T02:41:14.347 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:14.345+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 239996 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:14.449 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:14.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:14.448559+0000) 2020-02-06T02:41:16.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:16.582+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:16.582176+0000) 2020-02-06T02:41:16.739 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:16.736+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:16.737932+0000) 2020-02-06T02:41:17.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:17.193+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:17.194137+0000) 2020-02-06T02:41:17.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:17.470+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:17.470746+0000) 2020-02-06T02:41:17.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:17.470+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:17.470748+0000) 2020-02-06T02:41:17.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:17.470+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:17.470950+0000) 2020-02-06T02:41:17.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:17.720+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:17.721550+0000) 2020-02-06T02:41:19.352 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:19.350+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.351510+0000) 2020-02-06T02:41:19.353 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:19.351+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.352492+0000) 2020-02-06T02:41:19.354 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:19.352+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.353124+0000) 2020-02-06T02:41:19.354 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:19.352+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.353825+0000) 2020-02-06T02:41:19.355 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:19.353+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.354400+0000) 2020-02-06T02:41:19.355 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:19.353+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.354550+0000) 2020-02-06T02:41:19.356 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:19.354+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.355210+0000) 2020-02-06T02:41:19.356 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:19.355+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.355603+0000) 2020-02-06T02:41:19.357 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:19.355+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.356123+0000) 2020-02-06T02:41:19.357 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:19.356+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.356657+0000) 2020-02-06T02:41:19.358 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:19.357+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.357629+0000) 2020-02-06T02:41:19.359 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:19.357+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.357708+0000) 2020-02-06T02:41:19.359 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:19.357+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.358438+0000) 2020-02-06T02:41:19.362 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:19.360+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.361573+0000) 2020-02-06T02:41:19.378 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:19.376+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240040 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:19.839 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:19.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:19.838642+0000) 2020-02-06T02:41:20.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:20.082+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:20.083918+0000) 2020-02-06T02:41:20.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:20.697+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:20.697206+0000) 2020-02-06T02:41:20.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:20.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:20.972145+0000) 2020-02-06T02:41:22.971 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:22.970+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:22.970490+0000) 2020-02-06T02:41:23.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:23.553+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:23.553817+0000) 2020-02-06T02:41:24.025 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:41:24.032 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:41:24.382 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:24.380+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.381837+0000) 2020-02-06T02:41:24.383 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:24.381+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.382326+0000) 2020-02-06T02:41:24.384 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:24.382+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.383202+0000) 2020-02-06T02:41:24.384 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:24.382+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.383830+0000) 2020-02-06T02:41:24.385 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:24.382+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.384045+0000) 2020-02-06T02:41:24.385 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:24.383+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.384680+0000) 2020-02-06T02:41:24.386 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:24.383+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.385249+0000) 2020-02-06T02:41:24.386 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:24.383+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.385413+0000) 2020-02-06T02:41:24.387 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:24.384+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.386022+0000) 2020-02-06T02:41:24.387 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:24.384+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.385999+0000) 2020-02-06T02:41:24.387 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:24.384+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.386327+0000) 2020-02-06T02:41:24.388 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:24.385+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.386814+0000) 2020-02-06T02:41:24.389 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:24.386+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.387890+0000) 2020-02-06T02:41:24.392 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:24.390+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.391552+0000) 2020-02-06T02:41:24.409 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:24.406+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240080 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:24.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:24.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:24.448777+0000) 2020-02-06T02:41:26.590 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:26.580+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:26.582360+0000) 2020-02-06T02:41:26.747 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:26.745+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:26.746358+0000) 2020-02-06T02:41:27.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:27.193+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:27.194328+0000) 2020-02-06T02:41:27.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:27.469+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:27.471009+0000) 2020-02-06T02:41:27.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:27.469+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:27.471001+0000) 2020-02-06T02:41:27.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:27.469+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:27.471207+0000) 2020-02-06T02:41:27.722 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:27.720+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:27.721813+0000) 2020-02-06T02:41:29.416 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:29.413+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.414974+0000) 2020-02-06T02:41:29.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:29.414+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.415927+0000) 2020-02-06T02:41:29.419 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:29.415+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.416587+0000) 2020-02-06T02:41:29.419 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:29.416+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.417064+0000) 2020-02-06T02:41:29.419 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:29.416+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.417736+0000) 2020-02-06T02:41:29.420 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:29.414+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.415515+0000) 2020-02-06T02:41:29.420 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:29.416+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.417541+0000) 2020-02-06T02:41:29.421 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:29.418+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.419888+0000) 2020-02-06T02:41:29.421 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:29.418+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.420015+0000) 2020-02-06T02:41:29.421 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:29.417+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.418424+0000) 2020-02-06T02:41:29.422 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:29.417+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.419026+0000) 2020-02-06T02:41:29.422 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:29.418+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.419828+0000) 2020-02-06T02:41:29.422 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:29.418+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.420322+0000) 2020-02-06T02:41:29.429 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:29.424+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.425801+0000) 2020-02-06T02:41:29.443 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:29.436+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240123 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:29.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:29.837+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:29.838891+0000) 2020-02-06T02:41:30.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:30.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:30.084162+0000) 2020-02-06T02:41:30.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:30.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:30.697492+0000) 2020-02-06T02:41:30.973 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:30.971+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:30.972376+0000) 2020-02-06T02:41:32.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:32.969+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:32.970740+0000) 2020-02-06T02:41:33.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:33.553+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:33.554037+0000) 2020-02-06T02:41:34.443 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:34.441+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.442789+0000) 2020-02-06T02:41:34.444 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:34.442+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.443653+0000) 2020-02-06T02:41:34.445 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:34.443+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.444327+0000) 2020-02-06T02:41:34.446 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:34.444+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.445166+0000) 2020-02-06T02:41:34.446 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:34.444+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.445733+0000) 2020-02-06T02:41:34.447 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:34.445+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.446539+0000) 2020-02-06T02:41:34.447 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:34.445+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.446608+0000) 2020-02-06T02:41:34.448 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:34.446+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.447327+0000) 2020-02-06T02:41:34.448 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:34.445+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.447023+0000) 2020-02-06T02:41:34.449 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:34.446+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.447734+0000) 2020-02-06T02:41:34.449 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:34.446+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.447880+0000) 2020-02-06T02:41:34.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:34.447+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.448946+0000) 2020-02-06T02:41:34.450 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:34.448+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.449384+0000) 2020-02-06T02:41:34.451 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:34.447+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.448633+0000) 2020-02-06T02:41:34.454 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:34.452+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:34.453725+0000) 2020-02-06T02:41:34.469 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:34.467+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240164 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:36.592 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:36.581+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:36.582634+0000) 2020-02-06T02:41:36.760 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:36.758+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:36.758976+0000) 2020-02-06T02:41:37.195 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:37.193+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:37.194564+0000) 2020-02-06T02:41:37.472 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:37.470+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:37.471266+0000) 2020-02-06T02:41:37.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:37.470+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:37.471269+0000) 2020-02-06T02:41:37.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:37.470+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:37.471489+0000) 2020-02-06T02:41:37.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:37.721+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:37.722037+0000) 2020-02-06T02:41:39.476 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:39.473+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.475325+0000) 2020-02-06T02:41:39.477 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:39.474+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.475467+0000) 2020-02-06T02:41:39.477 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:39.474+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.476227+0000) 2020-02-06T02:41:39.478 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:39.476+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.477112+0000) 2020-02-06T02:41:39.478 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:39.476+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.477353+0000) 2020-02-06T02:41:39.478 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:39.476+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.477697+0000) 2020-02-06T02:41:39.479 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:39.477+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.478199+0000) 2020-02-06T02:41:39.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:39.477+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.478860+0000) 2020-02-06T02:41:39.481 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:39.478+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.479980+0000) 2020-02-06T02:41:39.481 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:39.478+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.479283+0000) 2020-02-06T02:41:39.481 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:39.477+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.478649+0000) 2020-02-06T02:41:39.482 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:39.478+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.480301+0000) 2020-02-06T02:41:39.482 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:39.479+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.480735+0000) 2020-02-06T02:41:39.487 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:39.485+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.486257+0000) 2020-02-06T02:41:39.501 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:39.499+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240208 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:39.846 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:39.838+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:39.839133+0000) 2020-02-06T02:41:40.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:40.083+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:40.084384+0000) 2020-02-06T02:41:40.698 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:40.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:40.697802+0000) 2020-02-06T02:41:40.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:40.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:40.972626+0000) 2020-02-06T02:41:42.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:42.970+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:42.970979+0000) 2020-02-06T02:41:43.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:43.553+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:43.554315+0000) 2020-02-06T02:41:44.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:44.448+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.449142+0000) 2020-02-06T02:41:44.507 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:44.505+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.506160+0000) 2020-02-06T02:41:44.507 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:44.505+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.506926+0000) 2020-02-06T02:41:44.508 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:44.506+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.507808+0000) 2020-02-06T02:41:44.509 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:44.506+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.507956+0000) 2020-02-06T02:41:44.509 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:44.507+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.508545+0000) 2020-02-06T02:41:44.510 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:44.508+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.509992+0000) 2020-02-06T02:41:44.511 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:44.508+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.509740+0000) 2020-02-06T02:41:44.511 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:44.509+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.510476+0000) 2020-02-06T02:41:44.512 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:44.509+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.511180+0000) 2020-02-06T02:41:44.512 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:44.510+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.511607+0000) 2020-02-06T02:41:44.512 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:44.510+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.511853+0000) 2020-02-06T02:41:44.513 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:44.511+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.512293+0000) 2020-02-06T02:41:44.513 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:44.511+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.512557+0000) 2020-02-06T02:41:44.516 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:44.514+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:44.515109+0000) 2020-02-06T02:41:44.533 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:44.530+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240248 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:46.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:46.582+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:46.582888+0000) 2020-02-06T02:41:46.771 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:46.770+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:46.770455+0000) 2020-02-06T02:41:47.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:47.194+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:47.194792+0000) 2020-02-06T02:41:47.472 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:47.470+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:47.471527+0000) 2020-02-06T02:41:47.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:47.470+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:47.471527+0000) 2020-02-06T02:41:47.473 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:47.470+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:47.471715+0000) 2020-02-06T02:41:47.723 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:47.721+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:47.722306+0000) 2020-02-06T02:41:49.537 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:49.535+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.536476+0000) 2020-02-06T02:41:49.542 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:49.538+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.539584+0000) 2020-02-06T02:41:49.542 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:49.538+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.540266+0000) 2020-02-06T02:41:49.542 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:49.537+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.538255+0000) 2020-02-06T02:41:49.542 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:49.537+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.538706+0000) 2020-02-06T02:41:49.543 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:49.538+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.539062+0000) 2020-02-06T02:41:49.543 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:49.538+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.539636+0000) 2020-02-06T02:41:49.543 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:49.540+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.541228+0000) 2020-02-06T02:41:49.544 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:49.539+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.540487+0000) 2020-02-06T02:41:49.544 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:49.539+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.540683+0000) 2020-02-06T02:41:49.544 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:49.540+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.541827+0000) 2020-02-06T02:41:49.544 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:49.541+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.543148+0000) 2020-02-06T02:41:49.545 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:49.542+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.543768+0000) 2020-02-06T02:41:49.550 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:49.548+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.549316+0000) 2020-02-06T02:41:49.566 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:49.561+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240292 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:49.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:49.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:49.839341+0000) 2020-02-06T02:41:50.085 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:50.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:50.084593+0000) 2020-02-06T02:41:50.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:50.696+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:50.698024+0000) 2020-02-06T02:41:50.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:50.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:50.972858+0000) 2020-02-06T02:41:52.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:52.969+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:52.971221+0000) 2020-02-06T02:41:53.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:53.553+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:53.554545+0000) 2020-02-06T02:41:54.173 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:41:54.179 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:41:54.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:54.449+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.449413+0000) 2020-02-06T02:41:54.577 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:54.567+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.568114+0000) 2020-02-06T02:41:54.586 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:54.570+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.572205+0000) 2020-02-06T02:41:54.586 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:54.571+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.572504+0000) 2020-02-06T02:41:54.586 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:54.572+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.573993+0000) 2020-02-06T02:41:54.586 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:54.573+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.574780+0000) 2020-02-06T02:41:54.587 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:54.573+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.574975+0000) 2020-02-06T02:41:54.587 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:54.573+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.575225+0000) 2020-02-06T02:41:54.587 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:54.567+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.568735+0000) 2020-02-06T02:41:54.587 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:54.567+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.568922+0000) 2020-02-06T02:41:54.587 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:54.569+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.570569+0000) 2020-02-06T02:41:54.588 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:54.569+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.570912+0000) 2020-02-06T02:41:54.588 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:54.570+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.571583+0000) 2020-02-06T02:41:54.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:54.571+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.572844+0000) 2020-02-06T02:41:54.588 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:54.578+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:54.579149+0000) 2020-02-06T02:41:54.594 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:54.591+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240332 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:56.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:56.582+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:56.583141+0000) 2020-02-06T02:41:56.779 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:56.777+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:56.778790+0000) 2020-02-06T02:41:57.216 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:57.194+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:57.195031+0000) 2020-02-06T02:41:57.485 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:57.471+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:57.471783+0000) 2020-02-06T02:41:57.486 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:57.471+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:57.471802+0000) 2020-02-06T02:41:57.486 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:57.471+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:57.471952+0000) 2020-02-06T02:41:57.730 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:57.722+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:57.722535+0000) 2020-02-06T02:41:59.599 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:41:59.597+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.598046+0000) 2020-02-06T02:41:59.600 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:41:59.599+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.599845+0000) 2020-02-06T02:41:59.601 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:41:59.600+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.600395+0000) 2020-02-06T02:41:59.602 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:41:59.601+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.601098+0000) 2020-02-06T02:41:59.602 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:41:59.600+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.601590+0000) 2020-02-06T02:41:59.603 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:59.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.602684+0000) 2020-02-06T02:41:59.604 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:41:59.601+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.602777+0000) 2020-02-06T02:41:59.604 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:41:59.601+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.603029+0000) 2020-02-06T02:41:59.605 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:41:59.602+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.602947+0000) 2020-02-06T02:41:59.605 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:41:59.602+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.603606+0000) 2020-02-06T02:41:59.606 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:41:59.603+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.604507+0000) 2020-02-06T02:41:59.606 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:41:59.603+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.604555+0000) 2020-02-06T02:41:59.606 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:59.602+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.603988+0000) 2020-02-06T02:41:59.611 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:59.609+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.610546+0000) 2020-02-06T02:41:59.626 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:41:59.623+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240376 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:41:59.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:41:59.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:41:59.839557+0000) 2020-02-06T02:42:00.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:00.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:00.084847+0000) 2020-02-06T02:42:00.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:00.697+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:00.698297+0000) 2020-02-06T02:42:00.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:00.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:00.973117+0000) 2020-02-06T02:42:02.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:02.970+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:02.971483+0000) 2020-02-06T02:42:03.555 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:03.554+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:03.554802+0000) 2020-02-06T02:42:04.450 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:04.448+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.449611+0000) 2020-02-06T02:42:04.633 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:04.628+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.629943+0000) 2020-02-06T02:42:04.634 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:04.629+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.630776+0000) 2020-02-06T02:42:04.634 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:04.631+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.632158+0000) 2020-02-06T02:42:04.635 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:04.631+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.633357+0000) 2020-02-06T02:42:04.635 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:04.632+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.633660+0000) 2020-02-06T02:42:04.635 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:04.631+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.632977+0000) 2020-02-06T02:42:04.635 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:04.633+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.634058+0000) 2020-02-06T02:42:04.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:04.633+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.634970+0000) 2020-02-06T02:42:04.636 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:04.633+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.635035+0000) 2020-02-06T02:42:04.636 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:04.634+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.635548+0000) 2020-02-06T02:42:04.637 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:04.634+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.635976+0000) 2020-02-06T02:42:04.637 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:04.635+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.636793+0000) 2020-02-06T02:42:04.638 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:04.635+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.637084+0000) 2020-02-06T02:42:04.642 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:04.640+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:04.641132+0000) 2020-02-06T02:42:04.656 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:04.654+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240415 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:06.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:06.582+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:06.583466+0000) 2020-02-06T02:42:06.788 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:06.785+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:06.786919+0000) 2020-02-06T02:42:07.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:07.194+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:07.195300+0000) 2020-02-06T02:42:07.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:07.471+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:07.472027+0000) 2020-02-06T02:42:07.473 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:07.471+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:07.472115+0000) 2020-02-06T02:42:07.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:07.471+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:07.472204+0000) 2020-02-06T02:42:07.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:07.721+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:07.722767+0000) 2020-02-06T02:42:09.663 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:09.660+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.662035+0000) 2020-02-06T02:42:09.664 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:09.661+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.663012+0000) 2020-02-06T02:42:09.664 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:09.662+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.663605+0000) 2020-02-06T02:42:09.665 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:09.662+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.663752+0000) 2020-02-06T02:42:09.665 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:09.662+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.664226+0000) 2020-02-06T02:42:09.666 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:09.663+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.664600+0000) 2020-02-06T02:42:09.666 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:09.663+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.664940+0000) 2020-02-06T02:42:09.666 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:09.664+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.665614+0000) 2020-02-06T02:42:09.667 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:09.664+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.666040+0000) 2020-02-06T02:42:09.667 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:09.665+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.666847+0000) 2020-02-06T02:42:09.668 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:09.666+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.667475+0000) 2020-02-06T02:42:09.668 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:09.666+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.667454+0000) 2020-02-06T02:42:09.669 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:09.666+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.667844+0000) 2020-02-06T02:42:09.676 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:09.674+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.675794+0000) 2020-02-06T02:42:09.687 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:09.685+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240459 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:09.840 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:09.838+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:09.839791+0000) 2020-02-06T02:42:10.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:10.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:10.085131+0000) 2020-02-06T02:42:10.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:10.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:10.698537+0000) 2020-02-06T02:42:10.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:10.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:10.973345+0000) 2020-02-06T02:42:12.981 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:12.971+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:12.971753+0000) 2020-02-06T02:42:13.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:13.554+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:13.555063+0000) 2020-02-06T02:42:14.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:14.448+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.449846+0000) 2020-02-06T02:42:14.692 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:14.690+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.691584+0000) 2020-02-06T02:42:14.693 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:14.691+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.692855+0000) 2020-02-06T02:42:14.694 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:14.692+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.693560+0000) 2020-02-06T02:42:14.695 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:14.693+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.694174+0000) 2020-02-06T02:42:14.695 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:14.694+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.694541+0000) 2020-02-06T02:42:14.695 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:14.694+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.694665+0000) 2020-02-06T02:42:14.696 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:14.694+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.695359+0000) 2020-02-06T02:42:14.696 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:14.695+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.695614+0000) 2020-02-06T02:42:14.697 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:14.695+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.696481+0000) 2020-02-06T02:42:14.698 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:14.696+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.697105+0000) 2020-02-06T02:42:14.698 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:14.697+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.697991+0000) 2020-02-06T02:42:14.699 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:14.697+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.698337+0000) 2020-02-06T02:42:14.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:14.700+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.701319+0000) 2020-02-06T02:42:14.702 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:14.700+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:14.701585+0000) 2020-02-06T02:42:14.718 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:14.717+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240498 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:16.584 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:16.583+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:16.583734+0000) 2020-02-06T02:42:16.796 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:16.794+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:16.795328+0000) 2020-02-06T02:42:17.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:17.194+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:17.195554+0000) 2020-02-06T02:42:17.473 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:17.471+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:17.472342+0000) 2020-02-06T02:42:17.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:17.471+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:17.472349+0000) 2020-02-06T02:42:17.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:17.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:17.472531+0000) 2020-02-06T02:42:17.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:17.722+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:17.722993+0000) 2020-02-06T02:42:19.724 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:19.721+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.723008+0000) 2020-02-06T02:42:19.725 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:19.722+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.723859+0000) 2020-02-06T02:42:19.725 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:19.723+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.724313+0000) 2020-02-06T02:42:19.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:19.723+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.724612+0000) 2020-02-06T02:42:19.726 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:19.723+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.724839+0000) 2020-02-06T02:42:19.726 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:19.724+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.725349+0000) 2020-02-06T02:42:19.726 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:19.724+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.725282+0000) 2020-02-06T02:42:19.727 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:19.725+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.725879+0000) 2020-02-06T02:42:19.727 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:19.726+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.726590+0000) 2020-02-06T02:42:19.728 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:19.726+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.726890+0000) 2020-02-06T02:42:19.728 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:19.727+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.727719+0000) 2020-02-06T02:42:19.729 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:19.728+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.728468+0000) 2020-02-06T02:42:19.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:19.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.732678+0000) 2020-02-06T02:42:19.733 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:19.731+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.732921+0000) 2020-02-06T02:42:19.747 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:19.746+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240542 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:19.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:19.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:19.839996+0000) 2020-02-06T02:42:20.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:20.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:20.085319+0000) 2020-02-06T02:42:20.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:20.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:20.698799+0000) 2020-02-06T02:42:20.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:20.972+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:20.973506+0000) 2020-02-06T02:42:22.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:22.971+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:22.971985+0000) 2020-02-06T02:42:23.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:23.554+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:23.555338+0000) 2020-02-06T02:42:24.344 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:42:24.351 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:42:24.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:24.449+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.450065+0000) 2020-02-06T02:42:24.752 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:24.750+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.751722+0000) 2020-02-06T02:42:24.753 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:24.751+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.752313+0000) 2020-02-06T02:42:24.754 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:24.751+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.752656+0000) 2020-02-06T02:42:24.755 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:24.753+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.754641+0000) 2020-02-06T02:42:24.756 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:24.754+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.755321+0000) 2020-02-06T02:42:24.757 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:24.755+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.755710+0000) 2020-02-06T02:42:24.757 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:24.755+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.756101+0000) 2020-02-06T02:42:24.758 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:24.755+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.756632+0000) 2020-02-06T02:42:24.758 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:24.756+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.756986+0000) 2020-02-06T02:42:24.759 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:24.757+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.757510+0000) 2020-02-06T02:42:24.760 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:24.757+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.758092+0000) 2020-02-06T02:42:24.760 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:24.758+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.758614+0000) 2020-02-06T02:42:24.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:24.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.761097+0000) 2020-02-06T02:42:24.762 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:24.760+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:24.761334+0000) 2020-02-06T02:42:24.779 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:24.778+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240581 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:26.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:26.583+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:26.583984+0000) 2020-02-06T02:42:26.804 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:26.802+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:26.803559+0000) 2020-02-06T02:42:27.196 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:27.195+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:27.195785+0000) 2020-02-06T02:42:27.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:27.472+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:27.472633+0000) 2020-02-06T02:42:27.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:27.472+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:27.472633+0000) 2020-02-06T02:42:27.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:27.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:27.472802+0000) 2020-02-06T02:42:27.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:27.722+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:27.723246+0000) 2020-02-06T02:42:29.785 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:29.783+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.784721+0000) 2020-02-06T02:42:29.786 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:29.784+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.785590+0000) 2020-02-06T02:42:29.787 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:29.785+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.786137+0000) 2020-02-06T02:42:29.788 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:29.786+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.787289+0000) 2020-02-06T02:42:29.788 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:29.786+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.787405+0000) 2020-02-06T02:42:29.789 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:29.787+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.787782+0000) 2020-02-06T02:42:29.789 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:29.787+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.788320+0000) 2020-02-06T02:42:29.789 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:29.787+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.788913+0000) 2020-02-06T02:42:29.790 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:29.788+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.789267+0000) 2020-02-06T02:42:29.790 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:29.789+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.789698+0000) 2020-02-06T02:42:29.791 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:29.789+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.790266+0000) 2020-02-06T02:42:29.791 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:29.790+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.790663+0000) 2020-02-06T02:42:29.793 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:29.791+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.792860+0000) 2020-02-06T02:42:29.794 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:29.792+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.793139+0000) 2020-02-06T02:42:29.811 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:29.809+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240625 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:29.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:29.839+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:29.840213+0000) 2020-02-06T02:42:30.086 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:30.084+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:30.085564+0000) 2020-02-06T02:42:30.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:30.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:30.699008+0000) 2020-02-06T02:42:30.974 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:30.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:30.973721+0000) 2020-02-06T02:42:32.973 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:32.971+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:32.972257+0000) 2020-02-06T02:42:33.556 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:33.554+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:33.555582+0000) 2020-02-06T02:42:34.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:34.449+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.450290+0000) 2020-02-06T02:42:34.815 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:34.813+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.814701+0000) 2020-02-06T02:42:34.818 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:34.816+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.817453+0000) 2020-02-06T02:42:34.819 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:34.817+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.818144+0000) 2020-02-06T02:42:34.819 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:34.818+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.818767+0000) 2020-02-06T02:42:34.820 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:34.818+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.818991+0000) 2020-02-06T02:42:34.820 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:34.818+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.818304+0000) 2020-02-06T02:42:34.820 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:34.818+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.819943+0000) 2020-02-06T02:42:34.821 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:34.819+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.820321+0000) 2020-02-06T02:42:34.822 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:34.820+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.820625+0000) 2020-02-06T02:42:34.822 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:34.819+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.820904+0000) 2020-02-06T02:42:34.822 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:34.820+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.821571+0000) 2020-02-06T02:42:34.823 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:34.821+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.821961+0000) 2020-02-06T02:42:34.823 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:34.821+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.822179+0000) 2020-02-06T02:42:34.826 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:34.824+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:34.825160+0000) 2020-02-06T02:42:34.839 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:34.838+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240665 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:36.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:36.583+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:36.584219+0000) 2020-02-06T02:42:36.816 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:36.815+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:36.815672+0000) 2020-02-06T02:42:37.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:37.195+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:37.196006+0000) 2020-02-06T02:42:37.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:37.472+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:37.472881+0000) 2020-02-06T02:42:37.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:37.472+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:37.472893+0000) 2020-02-06T02:42:37.474 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:37.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:37.473067+0000) 2020-02-06T02:42:37.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:37.722+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:37.723516+0000) 2020-02-06T02:42:39.860 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:39.843+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.844185+0000) 2020-02-06T02:42:39.861 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:39.843+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.844916+0000) 2020-02-06T02:42:39.861 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:39.845+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.845624+0000) 2020-02-06T02:42:39.862 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:39.845+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.846002+0000) 2020-02-06T02:42:39.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:39.845+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.846390+0000) 2020-02-06T02:42:39.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:39.846+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.847139+0000) 2020-02-06T02:42:39.864 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:39.853+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.854927+0000) 2020-02-06T02:42:39.865 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:39.846+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.847296+0000) 2020-02-06T02:42:39.865 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:39.847+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.848121+0000) 2020-02-06T02:42:39.865 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:39.847+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.848346+0000) 2020-02-06T02:42:39.872 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:39.847+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.848603+0000) 2020-02-06T02:42:39.872 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:39.846+0000 7f52f6ae3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.847351+0000) 2020-02-06T02:42:39.872 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:39.847+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.847970+0000) 2020-02-06T02:42:39.873 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:39.848+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:39.849218+0000) 2020-02-06T02:42:39.873 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:39.869+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240708 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:40.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:40.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:40.085792+0000) 2020-02-06T02:42:40.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:40.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:40.699234+0000) 2020-02-06T02:42:40.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:40.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:40.973942+0000) 2020-02-06T02:42:42.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:42.972+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:42.972547+0000) 2020-02-06T02:42:43.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:43.554+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:43.555826+0000) 2020-02-06T02:42:44.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:44.450+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.450523+0000) 2020-02-06T02:42:44.877 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:44.875+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.876259+0000) 2020-02-06T02:42:44.877 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:44.875+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.876721+0000) 2020-02-06T02:42:44.878 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:44.876+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.877449+0000) 2020-02-06T02:42:44.879 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:44.877+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.878620+0000) 2020-02-06T02:42:44.880 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:44.877+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.878263+0000) 2020-02-06T02:42:44.880 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:44.878+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.879256+0000) 2020-02-06T02:42:44.880 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:44.879+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.879623+0000) 2020-02-06T02:42:44.881 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:44.879+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.880150+0000) 2020-02-06T02:42:44.882 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:44.880+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.881227+0000) 2020-02-06T02:42:44.882 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:44.880+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.881365+0000) 2020-02-06T02:42:44.883 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:44.880+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.881622+0000) 2020-02-06T02:42:44.883 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:44.881+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.881849+0000) 2020-02-06T02:42:44.883 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:44.881+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.882125+0000) 2020-02-06T02:42:44.888 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:44.886+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:44.887522+0000) 2020-02-06T02:42:44.902 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:44.900+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240746 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:46.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:46.584+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:46.584475+0000) 2020-02-06T02:42:46.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:46.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:46.591135+0000) 2020-02-06T02:42:46.825 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:46.823+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:46.823853+0000) 2020-02-06T02:42:47.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:47.196+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:47.196287+0000) 2020-02-06T02:42:47.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:47.472+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:47.473135+0000) 2020-02-06T02:42:47.474 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:47.472+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:47.473156+0000) 2020-02-06T02:42:47.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:47.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:47.473304+0000) 2020-02-06T02:42:47.724 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:47.723+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:47.723745+0000) 2020-02-06T02:42:49.908 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:49.906+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.907416+0000) 2020-02-06T02:42:49.909 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:49.907+0000 7f7c70794700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.908269+0000) 2020-02-06T02:42:49.910 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:49.907+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.908932+0000) 2020-02-06T02:42:49.910 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:49.908+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.909315+0000) 2020-02-06T02:42:49.910 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:49.908+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.909534+0000) 2020-02-06T02:42:49.911 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:49.908+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.909987+0000) 2020-02-06T02:42:49.911 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:49.909+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.910366+0000) 2020-02-06T02:42:49.912 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:49.910+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.911194+0000) 2020-02-06T02:42:49.912 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:49.910+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.911482+0000) 2020-02-06T02:42:49.913 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:49.911+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.912346+0000) 2020-02-06T02:42:49.913 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:49.912+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.912563+0000) 2020-02-06T02:42:49.914 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:49.912+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.913225+0000) 2020-02-06T02:42:49.914 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:49.913+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.913673+0000) 2020-02-06T02:42:49.918 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:49.916+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:49.917574+0000) 2020-02-06T02:42:49.932 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:49.931+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240790 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:50.089 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:50.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:50.086007+0000) 2020-02-06T02:42:50.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:50.699+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:50.699477+0000) 2020-02-06T02:42:50.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:50.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:50.974176+0000) 2020-02-06T02:42:52.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:52.971+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:52.972833+0000) 2020-02-06T02:42:53.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:53.556+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:53.556134+0000) 2020-02-06T02:42:54.451 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:54.450+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.450785+0000) 2020-02-06T02:42:54.557 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:42:54.563 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:42:54.939 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:54.937+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.937754+0000) 2020-02-06T02:42:54.939 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:54.938+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.938407+0000) 2020-02-06T02:42:54.940 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:54.937+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.939139+0000) 2020-02-06T02:42:54.940 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:54.937+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.939312+0000) 2020-02-06T02:42:54.941 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:54.939+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.939769+0000) 2020-02-06T02:42:54.941 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:54.939+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.940031+0000) 2020-02-06T02:42:54.941 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:54.938+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.940224+0000) 2020-02-06T02:42:54.942 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:54.941+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.941193+0000) 2020-02-06T02:42:54.942 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:54.941+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.941545+0000) 2020-02-06T02:42:54.943 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:54.940+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.941594+0000) 2020-02-06T02:42:54.943 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:54.940+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.941609+0000) 2020-02-06T02:42:54.943 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:54.942+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.942272+0000) 2020-02-06T02:42:54.944 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:54.941+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.942821+0000) 2020-02-06T02:42:54.948 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:54.947+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:54.947575+0000) 2020-02-06T02:42:54.963 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:54.961+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240829 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:42:56.585 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:56.583+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:56.584711+0000) 2020-02-06T02:42:56.600 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:56.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:56.591372+0000) 2020-02-06T02:42:56.833 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:56.832+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:56.832189+0000) 2020-02-06T02:42:57.197 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:57.196+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:57.196557+0000) 2020-02-06T02:42:57.474 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:57.472+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:57.473401+0000) 2020-02-06T02:42:57.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:57.472+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:57.473398+0000) 2020-02-06T02:42:57.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:57.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:57.473510+0000) 2020-02-06T02:42:57.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:57.723+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:57.724000+0000) 2020-02-06T02:42:59.969 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:42:59.966+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.967940+0000) 2020-02-06T02:42:59.969 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:42:59.967+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.968457+0000) 2020-02-06T02:42:59.970 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:59.968+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.969431+0000) 2020-02-06T02:42:59.970 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:42:59.968+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.969791+0000) 2020-02-06T02:42:59.971 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:42:59.969+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.970068+0000) 2020-02-06T02:42:59.971 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:42:59.969+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.970677+0000) 2020-02-06T02:42:59.972 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:42:59.969+0000 7f1577ec2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.971047+0000) 2020-02-06T02:42:59.972 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:42:59.969+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.971301+0000) 2020-02-06T02:42:59.972 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:42:59.970+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.971885+0000) 2020-02-06T02:42:59.973 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:59.971+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.972541+0000) 2020-02-06T02:42:59.973 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:42:59.971+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.972875+0000) 2020-02-06T02:42:59.974 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:42:59.971+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.973040+0000) 2020-02-06T02:42:59.974 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:42:59.972+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.973838+0000) 2020-02-06T02:42:59.980 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:42:59.978+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:42:59.979566+0000) 2020-02-06T02:42:59.995 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:42:59.992+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240872 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:00.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:00.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:00.086288+0000) 2020-02-06T02:43:00.700 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:00.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:00.699698+0000) 2020-02-06T02:43:00.975 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:00.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:00.974389+0000) 2020-02-06T02:43:02.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:02.972+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:02.973100+0000) 2020-02-06T02:43:03.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:03.555+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:03.556368+0000) 2020-02-06T02:43:04.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:04.449+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:04.451019+0000) 2020-02-06T02:43:05.001 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:04.999+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.000698+0000) 2020-02-06T02:43:05.002 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:04.999+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.001040+0000) 2020-02-06T02:43:05.002 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:05.000+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.001477+0000) 2020-02-06T02:43:05.003 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:05.001+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.002362+0000) 2020-02-06T02:43:05.004 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:05.001+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.002965+0000) 2020-02-06T02:43:05.004 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:05.001+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.003066+0000) 2020-02-06T02:43:05.004 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:05.002+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.003365+0000) 2020-02-06T02:43:05.005 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:05.003+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.004150+0000) 2020-02-06T02:43:05.005 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:05.003+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.004692+0000) 2020-02-06T02:43:05.006 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:05.004+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.005484+0000) 2020-02-06T02:43:05.006 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:05.004+0000 7f5582a30700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.005590+0000) 2020-02-06T02:43:05.007 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:05.004+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.005775+0000) 2020-02-06T02:43:05.007 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:05.005+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.006229+0000) 2020-02-06T02:43:05.013 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:05.011+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:05.012252+0000) 2020-02-06T02:43:05.026 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:43:05.023+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240910 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:06.591 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:06.584+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:06.584968+0000) 2020-02-06T02:43:06.601 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:06.590+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:06.591609+0000) 2020-02-06T02:43:06.841 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:06.839+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:06.840381+0000) 2020-02-06T02:43:07.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:07.195+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:07.196788+0000) 2020-02-06T02:43:07.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:07.472+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:07.473702+0000) 2020-02-06T02:43:07.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:07.472+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:07.473703+0000) 2020-02-06T02:43:07.476 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:07.472+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:07.473805+0000) 2020-02-06T02:43:07.725 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:07.723+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:07.724203+0000) 2020-02-06T02:43:10.030 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:10.027+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.029123+0000) 2020-02-06T02:43:10.032 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:10.029+0000 7fd15d2fa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.030769+0000) 2020-02-06T02:43:10.032 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:10.030+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.031476+0000) 2020-02-06T02:43:10.032 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:10.030+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.032004+0000) 2020-02-06T02:43:10.033 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:10.031+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.032502+0000) 2020-02-06T02:43:10.034 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:10.032+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.033073+0000) 2020-02-06T02:43:10.035 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:10.032+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.033826+0000) 2020-02-06T02:43:10.038 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:10.033+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.034479+0000) 2020-02-06T02:43:10.038 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:10.032+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.034206+0000) 2020-02-06T02:43:10.039 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:10.034+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.035970+0000) 2020-02-06T02:43:10.039 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:10.034+0000 7f63cc3d5700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.036098+0000) 2020-02-06T02:43:10.039 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:10.035+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.036445+0000) 2020-02-06T02:43:10.039 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:10.034+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.035866+0000) 2020-02-06T02:43:10.041 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:10.039+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.040194+0000) 2020-02-06T02:43:10.066 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:43:10.054+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240954 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:10.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:10.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.086521+0000) 2020-02-06T02:43:10.701 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:10.698+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.699913+0000) 2020-02-06T02:43:10.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:10.973+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:10.974618+0000) 2020-02-06T02:43:12.974 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:12.972+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:12.973342+0000) 2020-02-06T02:43:13.557 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:13.555+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:13.556599+0000) 2020-02-06T02:43:14.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:14.450+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:14.451274+0000) 2020-02-06T02:43:15.062 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:15.060+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.061183+0000) 2020-02-06T02:43:15.063 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:15.061+0000 7fcd8a025700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.062656+0000) 2020-02-06T02:43:15.068 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:15.062+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.064005+0000) 2020-02-06T02:43:15.068 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:15.064+0000 7f2efbaab700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.065496+0000) 2020-02-06T02:43:15.068 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:15.065+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.066297+0000) 2020-02-06T02:43:15.069 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:15.061+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.063165+0000) 2020-02-06T02:43:15.069 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:15.063+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.064498+0000) 2020-02-06T02:43:15.069 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:15.065+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.066516+0000) 2020-02-06T02:43:15.069 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:15.065+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.066803+0000) 2020-02-06T02:43:15.069 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:15.065+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.066929+0000) 2020-02-06T02:43:15.070 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:15.066+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.067475+0000) 2020-02-06T02:43:15.070 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:15.066+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.067143+0000) 2020-02-06T02:43:15.070 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:15.066+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.067786+0000) 2020-02-06T02:43:15.074 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:15.071+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:15.072214+0000) 2020-02-06T02:43:15.088 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:43:15.085+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 240994 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:16.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:16.584+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:16.585219+0000) 2020-02-06T02:43:16.592 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:16.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:16.591846+0000) 2020-02-06T02:43:16.851 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:16.849+0000 7f9ed8517700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:16.850139+0000) 2020-02-06T02:43:17.198 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:17.196+0000 7fcd7b16d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:17.197020+0000) 2020-02-06T02:43:17.475 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:17.473+0000 7f5573377700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:17.473993+0000) 2020-02-06T02:43:17.475 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:17.473+0000 7f63bdd1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:17.474036+0000) 2020-02-06T02:43:17.475 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:17.473+0000 7fd447767700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:17.473989+0000) 2020-02-06T02:43:17.738 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:17.723+0000 7f1568809700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:17.724440+0000) 2020-02-06T02:43:20.087 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:20.085+0000 7f9eabc5d700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.086757+0000) 2020-02-06T02:43:20.093 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:20.091+0000 7fd45661f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.092501+0000) 2020-02-06T02:43:20.094 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:20.091+0000 7f52f62e2700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.092617+0000) 2020-02-06T02:43:20.095 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:20.093+0000 7f72aecf4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.094989+0000) 2020-02-06T02:43:20.096 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:20.094+0000 7f7c70f95700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.095567+0000) 2020-02-06T02:43:20.097 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:20.095+0000 7fd15e2fc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.096474+0000) 2020-02-06T02:43:20.097 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:20.095+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.096618+0000) 2020-02-06T02:43:20.098 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:20.095+0000 7fcd89824700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.096933+0000) 2020-02-06T02:43:20.098 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:20.096+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.097441+0000) 2020-02-06T02:43:20.099 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:20.096+0000 7fdad646a700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.097751+0000) 2020-02-06T02:43:20.099 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:20.097+0000 7f15776c1700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.098065+0000) 2020-02-06T02:43:20.099 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:20.097+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.098458+0000) 2020-02-06T02:43:20.100 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:20.097+0000 7f9ede523700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.098664+0000) 2020-02-06T02:43:20.100 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:20.097+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.099129+0000) 2020-02-06T02:43:20.105 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:20.103+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.104502+0000) 2020-02-06T02:43:20.122 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:43:20.120+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 241036 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:20.701 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:20.699+0000 7f7c618dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.700144+0000) 2020-02-06T02:43:20.976 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:20.974+0000 7f729fe3c700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:20.974828+0000) 2020-02-06T02:43:22.975 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:22.972+0000 7fd1582f0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:22.973628+0000) 2020-02-06T02:43:23.565 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:23.556+0000 7fdad2462700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:23.556829+0000) 2020-02-06T02:43:24.452 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:24.450+0000 7f2eecbf3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:24.451539+0000) 2020-02-06T02:43:24.730 INFO:teuthology.orchestra.run.smithi008:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:43:24.746 INFO:teuthology.orchestra.run.smithi164:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2020-02-06T02:43:25.129 INFO:tasks.ceph.osd.2.smithi008.stderr:2020-02-06T02:43:25.126+0000 7f558222f700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.128144+0000) 2020-02-06T02:43:25.129 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:25.126+0000 7f9edd521700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.128049+0000) 2020-02-06T02:43:25.130 INFO:tasks.ceph.osd.0.smithi008.stderr:2020-02-06T02:43:25.127+0000 7f7c6ff93700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.129040+0000) 2020-02-06T02:43:25.130 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:25.128+0000 7f52f72e4700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.129577+0000) 2020-02-06T02:43:25.131 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:25.128+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.129505+0000) 2020-02-06T02:43:25.131 INFO:tasks.ceph.osd.6.smithi164.stderr:2020-02-06T02:43:25.128+0000 7f1576ec0700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.129832+0000) 2020-02-06T02:43:25.131 INFO:tasks.ceph.osd.1.smithi008.stderr:2020-02-06T02:43:25.128+0000 7fd455e1e700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.130375+0000) 2020-02-06T02:43:25.132 INFO:tasks.ceph.osd.4.smithi164.stderr:2020-02-06T02:43:25.130+0000 7fcd8a826700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.131463+0000) 2020-02-06T02:43:25.132 INFO:tasks.ceph.mgr.y.smithi008.stderr:2020-02-06T02:43:25.129+0000 7fd15dafb700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.131321+0000) 2020-02-06T02:43:25.133 INFO:tasks.ceph.mds.b.smithi164.stderr:2020-02-06T02:43:25.131+0000 7fdad5c69700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.132133+0000) 2020-02-06T02:43:25.133 INFO:tasks.ceph.osd.7.smithi164.stderr:2020-02-06T02:43:25.131+0000 7f72ae4f3700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.132707+0000) 2020-02-06T02:43:25.134 INFO:tasks.ceph.osd.3.smithi008.stderr:2020-02-06T02:43:25.131+0000 7f63ccbd6700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.132964+0000) 2020-02-06T02:43:25.134 INFO:tasks.ceph.osd.5.smithi164.stderr:2020-02-06T02:43:25.132+0000 7f2efb2aa700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.133970+0000) 2020-02-06T02:43:25.139 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:25.137+0000 7f9eddd22700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:25.138934+0000) 2020-02-06T02:43:25.152 INFO:tasks.ceph.mon.a.smithi008.stderr:2020-02-06T02:43:25.150+0000 7f9e898e3700 -1 mon.a@0(probing) e1 get_health_metrics reporting 241075 slow ops, oldest is monmgrreport(0 checks, 0 progress events) 2020-02-06T02:43:26.586 INFO:tasks.ceph.mds.a.smithi008.stderr:2020-02-06T02:43:26.584+0000 7f52f32dc700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:26.585481+0000) 2020-02-06T02:43:26.593 INFO:tasks.ceph.mgr.x.smithi164.stderr:2020-02-06T02:43:26.591+0000 7f9eb2c6b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2020-02-06T01:43:26.592068+0000) 2020-02-06T02:43:26.704 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-02-06T02:43:26.704 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2020-02-06T02:43:26.704 DEBUG:teuthology.task.console_log:Killing console logger for smithi008 2020-02-06T02:43:26.705 DEBUG:teuthology.task.console_log:Killing console logger for smithi164 2020-02-06T02:43:26.705 DEBUG:teuthology.task.console_log:Killing console logger for smithi008 2020-02-06T02:43:26.705 DEBUG:teuthology.exit:Finished running handlers